meta-feeder-sdk/README.md

meta-feeder-sdk

The source-agnostic foundation shared by the MetaMesh gateway core and every feeder sidecar.

A feeder bridges one external source into the MetaMesh network. It implements FeederPlugin — find records, resolve them to content-addressed outcomes, optionally fetch bytes — and serve_feeders exposes any set of plugins over the feeder HTTP contract. The gateway core consumes that contract and keeps the libp2p wire, the bitswap blockstore, the hashing-into-the-blockstore and the meta-core store-back to itself.

Deliberately libp2p-free and blockstore-free: a feeder finds and fetches bytes; it never joins the swarm.

Using it

[dependencies]
meta-feeder-sdk = { git = "https://github.com/worph/meta-feeder-sdk", tag = "v1.3.0" }

Not on crates.io yet — the contract is still moving, so consumers pin a tag.

use meta_feeder_sdk::{serve_feeders, FeederPlugin};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let plugins: Vec<Box<dyn FeederPlugin>> = vec![Box::new(MyPlugin::new())];
    serve_feeders(plugins, "/data/meta-feeder".into(), "0.0.0.0:8080".parse()?).await
}

A service is a Vec — grouping plugins into one binary is a deployment choice, not an architectural one.

Discovery (1.6.0, scan 1.7.0)

serve_feeders and serve_transport advertise the process over beacon v2 (UDP multicast 239.255.99.1:9099, spec: meta-root docs/project-architecture/beacon-v2.md) as soon as the port is bound — a feeder as metamesh.feeder/<upstream_id> (one cap per hosted plugin), a transport as metamesh.transport/<id>@<contract> — with endpoints.http, endpoints.manifest = /manifest and rev = a hash of the served manifest (re-advertised when it changes). Nothing to call. Env:

Env	Default
ENABLE_UDP_DISCOVERY	true	false turns advertising off
BEACON_ADVERTISE_URL	http://$HOSTNAME:<port>	endpoints.http
BEACON_BINDS	unset	the one consumer instance allowed to list this plugin
BEACON_GROUP / BEACON_PORT / BEACON_INTERVAL_MS	239.255.99.1 / 9099 / 10000

beacon::scan(node, pattern, configured, wait) (1.7.0) is what a consumer's Scan button runs: probe everyone, wait, then classify every resource matching pattern as configured / addable / bound-elsewhere against the consumer's (name, url) list — the ScanReport JSON the shared <meta-beacon-scan> element renders. Transports advertise as meta-transport-<id> (SERVICE_NAME overrides), feeders as meta-feeder-<first upstream>.

The beacon module is also usable on its own — the meta-* services depend on the SDK for nothing else:

meta-feeder-sdk = { git = "https://github.com/worph/meta-feeder-sdk", tag = "v1.7.0", default-features = false, features = ["beacon"] }

Features: feeder (default — the whole feeder harness; implies beacon), beacon (tokio/serde/tracing/libc only), transport (implies feeder), testkit.

Writing a plugin

Four required methods; everything else has a default:

method	you must	notes
upstream_id	return a stable &'static str	identifies your records
configure	set up a cache dir; return MissingConfig if unconfigured	the harness soft-skips a plugin that returns MissingConfig, so an unconfigured plugin degrades instead of killing the sidecar
handle_query	answer a GatewayQuery with DiscoveryRecords	open with query_eval::query_accepts_plugin — see below
compute_outcomes	resolve a record_id to HashOutcomes	bytes: None for metadata-only sources

Defaults you can override: handle_query_stream, handle_fetch, health, served_file_types, served_content_kinds, package, redeems, config_schema, config_values, get_blob.

Redeeming locators (1.3.0)

A plugin can redeem a locator family — turn a locator cid into the bytes it names, usually spending a metered quota (a Newznab .nzb grab, an OpenSubtitles download):

1. Return a RedeemClaim from redeems() — RedeemClaim::nzb_release(hosts) or RedeemClaim::provider_file(sources) — reflecting the current config (no credential → no claim). Served per plugin in GET /manifest and, for polling, at GET /redeems → {"redeems": {"<upstream_id>": [claim…]}}.
2. Name a stable package() — the /files/plugin/<package>/ folder the gateway stores the bytes under.
3. In compute_outcomes, accept the locator cid as record_id and return ONE Sha2_256 outcome: hash = hash::compute_ipfs_cid(&bytes), bytes, file_extension, and optionally a record whose fields the gateway merges onto the locator record. Return NotFound when the cid is not yours (e.g. no key for that host) so the gateway can try the next claimer; RateLimited when the quota is spent.

The gateway calls this only on a real play, one claimer at a time, and writes the bytes' cid onto the locator record under the claim's field (manifest / file), so each file's quota is spent once. Decode helpers: hash::decode_nzb_release_cid, hash::decode_provider_file_cid, hash::codec_of.

The routing gate

The gateway dispatches on (fileType, contentKind), never on a service name. Advertise your axes and gate on them, or you will be handed queries you cannot answer:

fn served_file_types(&self)    -> &'static [&'static str] { &["video"] }
fn served_content_kinds(&self) -> &'static [&'static str] { &["movie", "series"] }

async fn handle_query(&self, q: &GatewayQuery, n: usize) -> Result<Vec<DiscoveryRecord>, GatewayError> {
    if !meta_feeder_sdk::query_eval::query_accepts_plugin(
        q, self.served_file_types(), self.served_content_kinds()) {
        return Ok(Vec::new());
    }
    // ...
}

&["*"] on an axis means "I serve every value on it".

Three rules that are not obvious

1. Records must carry a type filter. A record with no fileType / contentKind never reaches a client wall, and the symptom looks like a transport bug rather than a metadata one.
2. filters / ranges / negations are authoritative; raw_text is informational. Consume one source per filter or you will double-filter and under-match.
3. Never degrade an unresolvable anchor into a free-text query. free_text_or_star() exists for upstreams that 4xx on empty input — reaching for it when an identity filter failed to resolve dumps the upstream's recent feed into someone's library.

Hash families

HashKind selects how the core routes an outcome. Only Sha2_256 is seeded into the bitswap blockstore.

kind	codec	means
Sha2_256	standard	real bytes, retrievable via bitswap
Midhash256	0x1000	size + middle-sample; MetaMesh-internal
BtV1File	0x1001	a file inside a torrent — locator
NzbRelease	0x1005	a Usenet release — locator, redeemed by a credentialed peer
CardLocator	0x1007	a work, no bytes anywhere
YtVideo	0x1008	delegated playback — bytes are permanently someone else's

The last two carry an identity multihash. Anything that dispatches on the multihash instead of the codec will silently mis-rank them.

hash also encodes/decodes the redeemable locators: nzb-release (0x1005) and provider-file (0x100A, (source, id) — docs/cid-formats.md §8).

Layout

module	what
plugin	the FeederPlugin trait, HashKind, HashOutcome, ConfigError
types	DiscoveryRecord, GatewayError, PluginHealth, ByteStream
query / query_eval	the structured query and its evaluator
serve	the axum harness and the HTTP contract DTOs
config	self-describing config schema — the dashboard renders a form from it
cache	per-plugin redb cache
hash	the content-addressing families above
filename_meta / lang	release-tag parsing helpers
enrich, meta_core, common	MetaMesh-internal; not part of the plugin contract
beacon	beacon v2 local discovery: BeaconNode, Resource, cap_matches, the MetaMesh caps vocabulary

Development

cargo build && cargo test

To test a feeder against an SDK change before its tag is pushed: cargo will not resolve an unpushed tag, not even under a [patch] override ("failed to find tag"). Build a scratch copy of the feeder (outside its git tree) with the dependency swapped for a path, and run cargo test there:

mkdir -p /tmp/feeder && (cd meta-feeder-x && tar cf - --exclude=./target --exclude=./.git .) | (cd /tmp/feeder && tar xf -)
sed -i 's#meta-feeder-sdk = { git = .*#meta-feeder-sdk = { path = "/abs/path/to/meta-feeder-sdk" }#' /tmp/feeder/feeder-plugin/*/Cargo.toml
(cd /tmp/feeder && cargo test)

Never commit the path form: each feeder repo is its own Docker build context.

Context

Design rationale, the feeder roster, and the naming rules live in the meta-root: docs/project-architecture/feeder-architecture.md.

License

MIT