A feeder bridges one external source into the MetaMesh network. It implements FeederPlugin — find records, resolve them to content-addressed outcomes, optionally fetch bytes — and serve_feeders exposes any set of plugins over the feeder HTTP contract. The gateway core consumes that contract and keeps the libp2p wire, the bitswap blockstore, the hashing-into-the-blockstore and the meta-core store-back to itself.

Deliberately libp2p-free and blockstore-free: a feeder finds and fetches bytes; it never joins the swarm.

Using it

[dependencies]
meta-feeder-sdk = { git = "https://github.com/worph/meta-feeder-sdk", tag = "v1.3.0" }

Not on crates.io yet — the contract is still moving, so consumers pin a tag.

use meta_feeder_sdk::{serve_feeders, FeederPlugin};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let plugins: Vec<Box<dyn FeederPlugin>> = vec![Box::new(MyPlugin::new())];
    serve_feeders(plugins, "/data/meta-feeder".into(), "0.0.0.0:8080".parse()?).await
}

A service is a Vec — grouping plugins into one binary is a deployment choice, not an architectural one.

Discovery (1.6.0, scan 1.7.0)

serve_feeders and serve_transport advertise the process over beacon v2 (UDP multicast 239.255.99.1:9099, spec: meta-root docs/project-architecture/beacon-v2.md) as soon as the port is bound — a feeder as metamesh.feeder/<upstream_id> (one cap per hosted plugin), a transport as metamesh.transport/<id>@<contract> — with endpoints.http, endpoints.manifest = /manifest and rev = a hash of the served manifest (re-advertised when it changes). Nothing to call. Env:

Env	Default
`ENABLE_UDP_DISCOVERY`	`true`	`false` turns advertising off
`BEACON_ADVERTISE_URL`	`http://$HOSTNAME:<port>`	`endpoints.http`
`BEACON_BINDS`	unset	the one consumer instance allowed to list this plugin
`BEACON_GROUP` / `BEACON_PORT` / `BEACON_INTERVAL_MS`	`239.255.99.1` / `9099` / `10000`

beacon::scan(node, pattern, configured, wait) (1.7.0) is what a consumer's Scan button runs: probe everyone, wait, then classify every resource matching pattern as configured / addable / bound-elsewhere against the consumer's (name, url) list — the ScanReport JSON the shared <meta-beacon-scan> element renders. Transports advertise as meta-transport-<id> (SERVICE_NAME overrides), feeders as meta-feeder-<first upstream>.

The beacon module is also usable on its own — the meta-* services depend on the SDK for nothing else:

meta-feeder-sdk = { git = "https://github.com/worph/meta-feeder-sdk", tag = "v1.7.0", default-features = false, features = ["beacon"] }

Features: feeder (default — the whole feeder harness; implies beacon), beacon (tokio/serde/tracing/libc only), transport (implies feeder), testkit.

Writing a plugin

Four required methods; everything else has a default:

method	you must	notes
`upstream_id`	return a stable `&'static str`	identifies your records
`configure`	set up a cache dir; return `MissingConfig` if unconfigured	the harness soft-skips a plugin that returns `MissingConfig`, so an unconfigured plugin degrades instead of killing the sidecar
`handle_query`	answer a `GatewayQuery` with `DiscoveryRecord`s	open with `query_eval::query_accepts_plugin` — see below
`compute_outcomes`	resolve a `record_id` to `HashOutcome`s	`bytes: None` for metadata-only sources

Defaults you can override: handle_query_stream, handle_fetch, health, served_file_types, served_content_kinds, package, redeems, config_schema, config_values, get_blob.

Redeeming locators (1.3.0)

A plugin can redeem a locator family — turn a locator cid into the bytes it names, usually spending a metered quota (a Newznab .nzb grab, an OpenSubtitles download):

Return a RedeemClaim from redeems() — RedeemClaim::nzb_release(hosts) or RedeemClaim::provider_file(sources) — reflecting the current config (no credential → no claim). Served per plugin in GET /manifest and, for polling, at GET /redeems → {"redeems": {"<upstream_id>": [claim…]}}.
Name a stable package() — the /files/plugin/<package>/ folder the gateway stores the bytes under.
In compute_outcomes, accept the locator cid as record_id and return ONE Sha2_256 outcome: hash = hash::compute_ipfs_cid(&bytes), bytes, file_extension, and optionally a record whose fields the gateway merges onto the locator record. Return NotFound when the cid is not yours (e.g. no key for that host) so the gateway can try the next claimer; RateLimited when the quota is spent.

The gateway calls this only on a real play, one claimer at a time, and writes the bytes' cid onto the locator record under the claim's field (manifest / file), so each file's quota is spent once. Decode helpers: hash::decode_nzb_release_cid, hash::decode_provider_file_cid, hash::codec_of.

The routing gate

The gateway dispatches on (fileType, contentKind), never on a service name. Advertise your axes and gate on them, or you will be handed queries you cannot answer:

fn served_file_types(&self)    -> &'static [&'static str] { &["video"] }
fn served_content_kinds(&self) -> &'static [&'static str] { &["movie", "series"] }

async fn handle_query(&self, q: &GatewayQuery, n: usize) -> Result<Vec<DiscoveryRecord>, GatewayError> {
    if !meta_feeder_sdk::query_eval::query_accepts_plugin(
        q, self.served_file_types(), self.served_content_kinds()) {
        return Ok(Vec::new());
    }
    // ...
}

&["*"] on an axis means "I serve every value on it".

Three rules that are not obvious

Records must carry a type filter. A record with no fileType / contentKind never reaches a client wall, and the symptom looks like a transport bug rather than a metadata one.
filters / ranges / negations are authoritative; raw_text is informational. Consume one source per filter or you will double-filter and under-match.
Never degrade an unresolvable anchor into a free-text query. free_text_or_star() exists for upstreams that 4xx on empty input — reaching for it when an identity filter failed to resolve dumps the upstream's recent feed into someone's library.

Hash families

HashKind selects how the core routes an outcome. Only Sha2_256 is seeded into the bitswap blockstore.

kind	codec	means
`Sha2_256`	standard	real bytes, retrievable via bitswap
`Midhash256`	`0x1000`	size + middle-sample; MetaMesh-internal
`BtV1File`	`0x1001`	a file inside a torrent — locator
`NzbRelease`	`0x1005`	a Usenet release — locator, redeemed by a credentialed peer
`CardLocator`	`0x1007`	a work, no bytes anywhere
`YtVideo`	`0x1008`	delegated playback — bytes are permanently someone else's

The last two carry an identity multihash. Anything that dispatches on the multihash instead of the codec will silently mis-rank them.

hash also encodes/decodes the redeemable locators: nzb-release (0x1005) and provider-file (0x100A, (source, id) — docs/cid-formats.md §8).

Layout

module	what
`plugin`	the `FeederPlugin` trait, `HashKind`, `HashOutcome`, `ConfigError`
`types`	`DiscoveryRecord`, `GatewayError`, `PluginHealth`, `ByteStream`
`query` / `query_eval`	the structured query and its evaluator
`serve`	the axum harness and the HTTP contract DTOs
`config`	self-describing config schema — the dashboard renders a form from it
`cache`	per-plugin redb cache
`hash`	the content-addressing families above
`filename_meta` / `lang`	release-tag parsing helpers
`enrich`, `meta_core`, `common`	MetaMesh-internal; not part of the plugin contract
`beacon`	beacon v2 local discovery: `BeaconNode`, `Resource`, `cap_matches`, the MetaMesh `caps` vocabulary

Development

cargo build && cargo test

To test a feeder against an SDK change before its tag is pushed: cargo will not resolve an unpushed tag, not even under a [patch] override ("failed to find tag"). Build a scratch copy of the feeder (outside its git tree) with the dependency swapped for a path, and run cargo test there:

mkdir -p /tmp/feeder && (cd meta-feeder-x && tar cf - --exclude=./target --exclude=./.git .) | (cd /tmp/feeder && tar xf -)
sed -i 's#meta-feeder-sdk = { git = .*#meta-feeder-sdk = { path = "/abs/path/to/meta-feeder-sdk" }#' /tmp/feeder/feeder-plugin/*/Cargo.toml
(cd /tmp/feeder && cargo test)

Never commit the path form: each feeder repo is its own Docker build context.

Context

Design rationale, the feeder roster, and the naming rules live in the meta-root: docs/project-architecture/feeder-architecture.md.

License

MIT