meta-feeder-usenet

Usenet feeder sidecar for the MetaMesh gateway tier. Bridges a Usenet header scan we run ourselves — the NNTmux app (nntmux, worph/AppStore) — into the MetaMesh network, over nntmux's Newznab API.

Why this exists

meta-feeder-indexer proxies an external Newznab indexer and mints an nzb-release (0x1005) cid that embeds that indexer's host. Only a peer holding that indexer's credential can ever redeem it — so the metadata half of Usenet is neither self-owned nor portable.

Scanning Usenet ourselves gives us the article Message-IDs for free at collation time (no grab, no scrape). That lets this feeder mint a portable nzb-posting (0x1003) cid — a digest over the Message-ID set, embedding no host — which any peer with a plain NNTP provider can redeem, with no indexer credential anywhere in the path.

This feeder adds a Usenet metadata source alongside meta-feeder-indexer. It replaces nothing; a gateway query fans out to both.

It also redeems external releases. The torznab feeder mints nzb-release (0x1005) locators from Prowlarr/Newznab search hits; when a viewer plays one, the gateway asks this feeder (POST /compute with the locator cid) to grab its .nzb with the key configured here for that indexer host. The Newznab keys live on this feeder — meta-share keeps only the NNTP provider. See Redeeming nzb-release locators.

Design record: meta-gateway/docs/others/self-hosted-usenet-indexer-study.md.

Shape

[NNTmux app: PHP + MariaDB + Redis]          [usenet-feeder: Rust, FeederPlugin]
  header-scan groups (few conns)   ◄── t=search&q=…  (Newznab API, `metamesh` key)
  collate → releases               ◄── /getnzb?id=<guid>  once per release → cache
  writes <guid>.nzb.gz                  mints 0x1003 from the Message-ID set
  (CBP purged; the .nzb is the only     emits the record + a relative
   durable home of the msg-ids)          `manifest_url` onto its own /blob route
                                                    │
                                                    ▼
   gateway core: seeds the .nzb as an ORDINARY sha2-256 IPFS cid, publishes it
   as a fileType=nzb meta-core record, rewrites the field to that cid
   (the same path posters take — `remote_feeder.rs` SEEDABLE_FIELDS)
                                                    │
                                                    ▼
   meta-share: resolve `manifest` cid (blockstore / bitswap / ANY holder) →
   NzbManifest::from_nzb_xml → BODY-fetch each Message-ID from ANY plain NNTP
   provider → materialise → play

nntmux over its API, not its database (1.1)

Until 1.1 the feeder read nntmux's MariaDB with SQL and gunzipped <guid>.nzb.gz off nntmux's storage volume. That made the two a single unit: the feeder held nntmux's DB password, depended on its table layout and its NZB shard depth, and had to mount its storage. nntmux is now its own app, and the feeder uses the same contract any Newznab client uses (the torznab feeder does the same with Prowlarr):

Need	Call	nntmux counter
search	`GET {nntmux_url}/api/v1/api?t=search&q=…&extended=1&apikey=…`	`apirequests`
a release's `.nzb` (Message-IDs)	`GET {nntmux_url}/getnzb?id=<guid>.nzb&r=<key>`	`downloadrequests`
compute (by guid)	`t=details&id=<guid>`	`apirequests`
reachability (`/health`)	`t=caps` (no key)	—

⚠ The API is under /api/v1/api; /api is a 404. t=get redirects to /getnzb on the public APP_URL host, so the feeder calls /getnzb on the internal host directly.
Every search hit must carry its 0x1003 cid (else meta-watch shows "Unavailable"), so a hit costs one /getnzb the first time it is seen. Each .nzb is cached under <state>/gateway/usenet/nntmux-nzb/<guid>.nzb (a collated release never changes) and reused by repeat searches, /compute and /blob.
nntmux caps every role per 24 h (Admin 1000/1000, User 10/10) and throttles /api to 60/min. Use the NNTmux app's dedicated metamesh user (its role has raised caps); the feeder paces itself to ≤ 50 requests/min, and stops a query's downloads once nntmux reports a cap (500/501).

Like every feeder this one is meta-core-free and blockstore-free: it finds records and serves bytes; the core owns hashing-into-the-blockstore, the meta-core store-back, and the libp2p wire (gateway invariant 10).

Configuration

Operator config lives on the feeder, not the gateway (invariant 12). Set it on the plugin's config page in the gateway dashboard; the env vars below are a first-boot seed only — the persisted config.json wins on the next restart, and there is no hot reload.

Field	Env seed	What
`nntmux_url`	`NNTMUX_URL`	Base URL of the NNTmux app as this container reaches it (default `http://nntmux`); the API path is appended.
`nntmux_api_key`	`NNTMUX_API_KEY`	API key of the nntmux user the feeder searches as (the `metamesh` user). Blank → self-scan search is off.
`indexers`	— (config page only)	`[{host, api_key}]` Newznab keys for redeeming `nzb-release` locators. `host` is the bare authority (`api.nzbgeek.info` — no scheme, no `/api`).

Both halves are optional; with neither configured the plugin reports Degraded, as it does when nntmux is unreachable or refuses the key. A grab-only deployment (keys, no nntmux) is valid. A pre-1.1 config.json (db_url/nzb_root) is ignored with a warning.

Infra env: META_FEEDER_HTTP_LISTEN (default 0.0.0.0:8080), META_FEEDER_STATE_DIR (default /data/meta-feeder), RUST_LOG.

Scan policy — which groups, backfill depth, scan interval, connection budget — is the gateway maintainer's call and is configured in the NNTmux app. Keep the connection budget conservative: the scanner and playback draw on the same provider slots, so a greedy value does not merely over-scan, it stalls playback mid-stream.

Redeeming nzb-release locators

POST /compute {upstream_id: "usenet", record_id: <0x1005 cid>}:

Decode {api_base, id} from the cid; find the key for its host. No key → 404 (not ours — the gateway tries another claimer, nothing is spent).
GET https://{api_base}/api?t=get&id=…&apikey=… with a named User-Agent (nzbgeek answers 109 Invalid User Agent without one). This spends the indexer's daily download quota — the gateway only calls it on a real play, and never again for a release whose manifest it already stored.
Strip the .nzb <head> except <meta type="password"> (the file is seeded to the whole mesh; the password is needed to unpack a protected RAR set).
Mint the portable nzb-posting (0x1003) cid from its Message-IDs.
Answer ONE sha2_256 outcome: the .nzb bytes (file_extension: nzb) and a record carrying cids/<nzb-posting> + segmentCount. The gateway stores the file under /files/plugin/meta-feeder-usenet/, writes its cid onto the release record as manifest, and merges the posting cid — after which any peer with a plain NNTP provider can play the release.

Errors: an indexer credential/account refusal (100–102, 910) or an HTML page → 422; request/download limit (429, 500, 501, or HTTP 429) → 429; no such item (300) → 404.

GET /redeems and /manifest advertise {codec: nzb-release, field: manifest, hosts: [...]} for the hosts that have a key; nothing without keys.

Build

cargo build --release --bin usenet-feeder
cargo test --workspace
cargo clippy --workspace --all-targets -- -D warnings

docker build -f feeder-plugin/usenet-feeder/Dockerfile -t ghcr.io/worph/meta-feeder-usenet:dev .

meta-feeder-sdk is a git dependency pinned by tag (v1.2.2; this repo used to vendor a copy). An SDK change is invisible here until it is tagged and pushed, and cargo cannot resolve an unpushed tag — not even under a [patch] override. To verify against a local SDK checkout, build a scratch copy with the dependency swapped for a path (never commit that form — this repo is the Docker build context):

SCRATCH=/d/workspace/tmp-claude/usenet-feeder; SDK=/d/workspace/MetaMesh/meta-root-v2/packages/plugins/meta-feeder-sdk
rm -rf $SCRATCH && mkdir -p $SCRATCH && tar cf - --exclude=./target --exclude=./.git . | (cd $SCRATCH && tar xf -)
sed -i "s#meta-feeder-sdk = { git = .*#meta-feeder-sdk = { path = \"$SDK\" }#" $SCRATCH/feeder-plugin/*/Cargo.toml
docker run --rm -e RUSTUP_TOOLCHAIN=1.89.0 -e CARGO_TARGET_DIR=/target -v /d/workspace:/d/workspace \
  -v feeder-cargo-registry:/usr/local/cargo/registry -v usenet-feeder-target:/target -w $SCRATCH \
  rust:1.89 sh -c 'apt-get install -y -qq pkg-config libssl-dev >/dev/null 2>&1; cargo test'

After the tag is pushed, run cargo update -p meta-feeder-sdk here so Cargo.lock stops pointing at the removed vendored path.

compute_nzb_posting_cid's normalisation rule is the dedup contract and is mirrored on the meta-share side — change them together.

Gotchas

An empty catalog is the visible symptom of both nntmux silent-zero traps: an empty PATH_TO_NZBS (set-but-empty beats the default, so createNZBs() aborts and you get 0 releases with no error), and seeded groups left active=0. The plugin logs a loud warning when releases is empty at boot — believe it.
The Message-IDs are not in the database. nntmux purges collections/binaries/parts right after writing each NZB, so the on-disk .nzb.gz is the only durable home. A wrong nzb_root therefore fails at compute_outcomes, not at query time.

9.6 KiB Raw Permalink Blame History Unescape Escape