meta-feeder-sdk/README.md

meta-feeder-sdk

The source-agnostic foundation shared by the MetaMesh gateway core and every feeder sidecar.

A feeder bridges one external source into the MetaMesh network. It implements FeederPlugin — find records, resolve them to content-addressed outcomes, optionally fetch bytes — and serve_feeders exposes any set of plugins over the feeder HTTP contract. The gateway core consumes that contract and keeps the libp2p wire, the bitswap blockstore, the hashing-into-the-blockstore and the meta-core store-back to itself.

Deliberately libp2p-free and blockstore-free: a feeder finds and fetches bytes; it never joins the swarm.

Using it

[dependencies]
meta-feeder-sdk = { git = "https://github.com/worph/meta-feeder-sdk", tag = "v0.1.0" }

Not on crates.io yet — the contract is still moving, so consumers pin a tag.

use meta_feeder_sdk::{serve_feeders, FeederPlugin};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let plugins: Vec<Box<dyn FeederPlugin>> = vec![Box::new(MyPlugin::new())];
    serve_feeders(plugins, "/data/meta-feeder".into(), "0.0.0.0:8080".parse()?).await
}

A service is a Vec — grouping plugins into one binary is a deployment choice, not an architectural one.

Writing a plugin

Four required methods; everything else has a default:

method	you must	notes
upstream_id	return a stable &'static str	identifies your records
configure	set up a cache dir; return MissingConfig if unconfigured	the harness soft-skips a plugin that returns MissingConfig, so an unconfigured plugin degrades instead of killing the sidecar
handle_query	answer a GatewayQuery with DiscoveryRecords	open with query_eval::query_accepts_plugin — see below
compute_outcomes	resolve a record_id to HashOutcomes	bytes: None for metadata-only sources

Defaults you can override: handle_query_stream, handle_fetch, health, served_file_types, served_content_kinds, config_schema, config_values, get_blob.

The routing gate

The gateway dispatches on (fileType, contentKind), never on a service name. Advertise your axes and gate on them, or you will be handed queries you cannot answer:

fn served_file_types(&self)    -> &'static [&'static str] { &["video"] }
fn served_content_kinds(&self) -> &'static [&'static str] { &["movie", "series"] }

async fn handle_query(&self, q: &GatewayQuery, n: usize) -> Result<Vec<DiscoveryRecord>, GatewayError> {
    if !meta_feeder_sdk::query_eval::query_accepts_plugin(
        q, self.served_file_types(), self.served_content_kinds()) {
        return Ok(Vec::new());
    }
    // ...
}

&["*"] on an axis means "I serve every value on it".

Three rules that are not obvious

1. Records must carry a type filter. A record with no fileType / contentKind never reaches a client wall, and the symptom looks like a transport bug rather than a metadata one.
2. filters / ranges / negations are authoritative; raw_text is informational. Consume one source per filter or you will double-filter and under-match.
3. Never degrade an unresolvable anchor into a free-text query. free_text_or_star() exists for upstreams that 4xx on empty input — reaching for it when an identity filter failed to resolve dumps the upstream's recent feed into someone's library.

Hash families

HashKind selects how the core routes an outcome. Only Sha2_256 is seeded into the bitswap blockstore.

kind	codec	means
Sha2_256	standard	real bytes, retrievable via bitswap
Midhash256	0x1000	size + middle-sample; MetaMesh-internal
BtV1File	0x1001	a file inside a torrent — locator
NzbRelease	0x1005	a Usenet release — locator, redeemed by a credentialed peer
CardLocator	0x1007	a work, no bytes anywhere
YtVideo	0x1008	delegated playback — bytes are permanently someone else's

The last two carry an identity multihash. Anything that dispatches on the multihash instead of the codec will silently mis-rank them.

Layout

module	what
plugin	the FeederPlugin trait, HashKind, HashOutcome, ConfigError
types	DiscoveryRecord, GatewayError, PluginHealth, ByteStream
query / query_eval	the structured query and its evaluator
serve	the axum harness and the HTTP contract DTOs
config	self-describing config schema — the dashboard renders a form from it
cache	per-plugin redb cache
hash	the content-addressing families above
filename_meta / lang	release-tag parsing helpers
enrich, meta_core, common	MetaMesh-internal; not part of the plugin contract

Development

cargo build && cargo test

To work on the SDK from inside a feeder checkout without pushing a tag, add a gitignored .cargo/config.toml in the feeder:

paths = ["../meta-feeder-sdk"]

Context

Design rationale, the feeder roster, and the naming rules live in the meta-root: docs/project-architecture/feeder-architecture.md.

License

MIT