SDK v1.7.0: advertises over beacon v2 as metamesh.feeder/<upstream>

Cargo.toml

@@ -1,6 +1,6 @@
 [workspace]
 resolver = "2"
-members = ["crates/*", "feeder-plugin/*"]
+members = ["feeder-plugin/*"]
 
 [workspace.package]
 edition = "2021"

README.md

@@ -1,8 +1,8 @@
 # meta-feeder-usenet
 
 Usenet feeder sidecar for the MetaMesh gateway tier. Bridges a Usenet header
-scan **we run ourselves** (an [nntmux](https://github.com/NNTmux/newznab-tmux)
+scan **we run ourselves** — the **NNTmux** app ([nntmux](https://github.com/NNTmux/newznab-tmux),
-sidecar) into the MetaMesh network.
+worph/AppStore) — into the MetaMesh network, over nntmux's Newznab API.
 
 ## Why this exists
 
@@ -20,17 +20,24 @@ credential anywhere in the path**.
 This feeder **adds** a Usenet metadata source alongside `meta-feeder-indexer`.
 It replaces nothing; a gateway query fans out to both.
 
+It also **redeems external releases.** The torznab feeder mints `nzb-release`
+(`0x1005`) locators from Prowlarr/Newznab search hits; when a viewer plays one,
+the gateway asks this feeder (`POST /compute` with the locator cid) to grab its
+`.nzb` with the key configured here for that indexer host. The Newznab keys live
+on this feeder — meta-share keeps only the NNTP provider. See
+[Redeeming nzb-release locators](#redeeming-nzb-release-locators).
+
 Design record:
 `meta-gateway/docs/others/self-hosted-usenet-indexer-study.md`.
 
 ## Shape
 
 ```
-[nntmux sidecar: PHP + MariaDB + Redis]     [usenet-feeder: Rust, FeederPlugin]
+[NNTmux app: PHP + MariaDB + Redis]          [usenet-feeder: Rust, FeederPlugin]
-  header-scan groups (few conns)   ──►  reads `releases` (searchname/size/guid)
+  header-scan groups (few conns)   ◄── t=search&q=…  (Newznab API, `metamesh` key)
-  collate → releases               ──►  gunzips <nzb_root>/<c>/<guid>.nzb.gz
+  collate → releases               ◄── /getnzb?id=<guid>  once per release → cache
-  writes <guid>.nzb.gz             ──►  mints 0x1003 from the Message-ID set
+  writes <guid>.nzb.gz                  mints 0x1003 from the Message-ID set
-  (CBP purged; the gz is the only  ──►  emits the record + a relative
+  (CBP purged; the .nzb is the only     emits the record + a relative
    durable home of the msg-ids)          `manifest_url` onto its own /blob route
                                                     │
                                                     ▼
@@ -44,9 +51,34 @@ Design record:
    provider → materialise → play
 ```
 
-The sidecar is **persistent cache, cheap to lose** — everything in it is
+### nntmux over its API, not its database (1.1)
-re-derivable by re-scanning. Same pattern as the Tribler sidecar: stateful
+
-upstream, but the gateway still sees a stateless HTTP feeder.
+Until 1.1 the feeder read nntmux's MariaDB with SQL and gunzipped
+`<guid>.nzb.gz` off nntmux's storage volume. That made the two a single unit:
+the feeder held nntmux's DB password, depended on its table layout and its NZB
+shard depth, and had to mount its storage. nntmux is now its own app, and the
+feeder uses the same contract any Newznab client uses (the torznab feeder does
+the same with Prowlarr):
+
+| Need | Call | nntmux counter |
+|---|---|---|
+| search | `GET {nntmux_url}/api/v1/api?t=search&q=…&extended=1&apikey=…` | `apirequests` |
+| a release's `.nzb` (Message-IDs) | `GET {nntmux_url}/getnzb?id=<guid>.nzb&r=<key>` | `downloadrequests` |
+| compute (by guid) | `t=details&id=<guid>` | `apirequests` |
+| reachability (`/health`) | `t=caps` (no key) | — |
+
+- ⚠ The API is under **`/api/v1/api`**; `/api` is a 404. `t=get` redirects to
+  `/getnzb` on the *public* `APP_URL` host, so the feeder calls `/getnzb` on the
+  internal host directly.
+- Every search hit must carry its `0x1003` cid (else meta-watch shows
+  "Unavailable"), so a hit costs one `/getnzb` **the first time it is seen**.
+  Each `.nzb` is cached under `<state>/gateway/usenet/nntmux-nzb/<guid>.nzb`
+  (a collated release never changes) and reused by repeat searches, `/compute`
+  and `/blob`.
+- nntmux caps every role per 24 h (Admin 1000/1000, User 10/10) and throttles
+  `/api` to 60/min. Use the NNTmux app's dedicated **`metamesh`** user (its role
+  has raised caps); the feeder paces itself to ≤ 50 requests/min, and stops a
+  query's downloads once nntmux reports a cap (`500`/`501`).
 
 Like every feeder this one is **meta-core-free and blockstore-free**: it finds
 records and serves bytes; the core owns hashing-into-the-blockstore, the
@@ -61,18 +93,50 @@ and there is no hot reload.
 
 | Field | Env seed | What |
 |---|---|---|
-| `db_url` | `NNTMUX_DB_URL` | MariaDB DSN of nntmux's catalog, e.g. `mysql://nntmux:secret@nntmux-db:3306/nntmux`. Blank → the upstream soft-skips and the plugin reports Degraded. |
+| `nntmux_url` | `NNTMUX_URL` | Base URL of the NNTmux app as this container reaches it (default `http://nntmux`); the API path is appended. |
-| `nzb_root` | `NNTMUX_NZB_PATH` | nntmux's NZB directory **as mounted into this container**. Must be the same volume nntmux's `PATH_TO_NZBS` points at, or every release resolves to `NotFound`. |
+| `nntmux_api_key` | `NNTMUX_API_KEY` | API key of the nntmux user the feeder searches as (the `metamesh` user). Blank → self-scan search is off. |
+| `indexers` | — (config page only) | `[{host, api_key}]` Newznab keys for redeeming `nzb-release` locators. `host` is the bare authority (`api.nzbgeek.info` — no scheme, no `/api`). |
+
+Both halves are optional; with neither configured the plugin reports Degraded,
+as it does when nntmux is unreachable or refuses the key. A grab-only deployment
+(keys, no nntmux) is valid. A pre-1.1 `config.json` (`db_url`/`nzb_root`) is
+ignored with a warning.
 
 Infra env: `META_FEEDER_HTTP_LISTEN` (default `0.0.0.0:8080`),
 `META_FEEDER_STATE_DIR` (default `/data/meta-feeder`), `RUST_LOG`.
 
 Scan policy — which groups, backfill depth, scan interval, connection budget —
-is the **gateway maintainer's** call and is configured on the nntmux sidecar.
+is the **gateway maintainer's** call and is configured in the NNTmux app.
 Keep the connection budget conservative: the scanner and playback draw on the
 same provider slots, so a greedy value does not merely over-scan, it stalls
 playback mid-stream.
 
+## Redeeming nzb-release locators
+
+`POST /compute {upstream_id: "usenet", record_id: <0x1005 cid>}`:
+
+1. Decode `{api_base, id}` from the cid; find the key for its host. No key →
+   `404` (not ours — the gateway tries another claimer, nothing is spent).
+2. `GET https://{api_base}/api?t=get&id=…&apikey=…` with a named User-Agent
+   (nzbgeek answers `109 Invalid User Agent` without one). **This spends the
+   indexer's daily download quota** — the gateway only calls it on a real play,
+   and never again for a release whose `manifest` it already stored.
+3. Strip the `.nzb` `<head>` except `<meta type="password">` (the file is seeded
+   to the whole mesh; the password is needed to unpack a protected RAR set).
+4. Mint the portable `nzb-posting` (`0x1003`) cid from its Message-IDs.
+5. Answer ONE `sha2_256` outcome: the `.nzb` bytes (`file_extension: nzb`) and a
+   record carrying `cids/<nzb-posting>` + `segmentCount`. The gateway stores the
+   file under `/files/plugin/meta-feeder-usenet/`, writes its cid onto the release
+   record as `manifest`, and merges the posting cid — after which any peer with a
+   plain NNTP provider can play the release.
+
+Errors: an indexer credential/account refusal (`100`–`102`, `910`) or an HTML
+page → `422`; request/download limit (`429`, `500`, `501`, or HTTP 429) → `429`;
+no such item (`300`) → `404`.
+
+`GET /redeems` and `/manifest` advertise `{codec: nzb-release, field: manifest,
+hosts: [...]}` for the hosts that have a key; nothing without keys.
+
 ## Build
 
 ```bash
@@ -83,11 +147,26 @@ cargo clippy --workspace --all-targets -- -D warnings
 docker build -f feeder-plugin/usenet-feeder/Dockerfile -t ghcr.io/worph/meta-feeder-usenet:dev .
 ```
 
-`crates/meta-feeder-sdk/` is a **vendored copy** of the SDK, mirrored from
+`meta-feeder-sdk` is a git dependency pinned by tag (`v1.2.2`; this repo used to
-`meta-gateway`. Independent submodules cannot share a cargo path-dep, so keep it
+vendor a copy). An SDK change is invisible here until it is tagged and pushed,
-in step when the SDK changes — in particular `hash.rs`'s
+and cargo cannot resolve an unpushed tag — not even under a `[patch]` override. To verify against a local SDK checkout, build a
-`compute_nzb_posting_cid`, whose normalisation rule is the dedup contract and is
+scratch copy with the dependency swapped for a path (never commit that form —
-mirrored again on the meta-share side.
+this repo is the Docker build context):
+
+```bash
+SCRATCH=/d/workspace/tmp-claude/usenet-feeder; SDK=/d/workspace/MetaMesh/meta-root-v2/packages/plugins/meta-feeder-sdk
+rm -rf $SCRATCH && mkdir -p $SCRATCH && tar cf - --exclude=./target --exclude=./.git . | (cd $SCRATCH && tar xf -)
+sed -i "s#meta-feeder-sdk = { git = .*#meta-feeder-sdk = { path = \"$SDK\" }#" $SCRATCH/feeder-plugin/*/Cargo.toml
+docker run --rm -e RUSTUP_TOOLCHAIN=1.89.0 -e CARGO_TARGET_DIR=/target -v /d/workspace:/d/workspace \
+  -v feeder-cargo-registry:/usr/local/cargo/registry -v usenet-feeder-target:/target -w $SCRATCH \
+  rust:1.89 sh -c 'apt-get install -y -qq pkg-config libssl-dev >/dev/null 2>&1; cargo test'
+```
+
+After the tag is pushed, run `cargo update -p meta-feeder-sdk` here so
+`Cargo.lock` stops pointing at the removed vendored path.
+
+`compute_nzb_posting_cid`'s normalisation rule is the dedup contract and is
+mirrored on the meta-share side — change them together.
 
 ## Gotchas
 

crates/meta-feeder-sdk/Cargo.toml

@@ -1,49 +0,0 @@
-[package]
-name = "meta-feeder-sdk"
-version = "0.1.0"
-edition.workspace = true
-rust-version.workspace = true
-license.workspace = true
-publish = false
-
-# Source-agnostic foundation shared by the gateway core and every feeder
-# sidecar binary: the `FeederPlugin` trait, the content-addressing hash
-# families, the per-plugin redb cache, the shared HTTP/cache helpers, the
-# structured-query evaluator, and the axum `serve_feeders` harness that wraps
-# any set of `FeederPlugin`s in the feeder HTTP contract.
-#
-# Deliberately libp2p-free / blockstore-free: a feeder finds + fetches bytes;
-# the gateway core owns hashing-into-the-blockstore, the meta-core store-back,
-# and the libp2p wire. Keep heavy deps (librqbit, beetswap, cid, libp2p) OUT
-# of here so feeders that don't need them don't pay for them.
-
-[dependencies]
-# Async runtime + HTTP server (the serve_feeders harness)
-tokio = { version = "1.40", features = ["full"] }
-axum = { version = "0.7", features = ["json"] }
-hyper = { version = "1", features = ["full"] }
-reqwest = { version = "0.12", features = ["json", "rustls-tls", "stream"], default-features = false }
-
-# Serialization
-serde = { version = "1", features = ["derive"] }
-serde_json = "1"
-base64 = "0.22"
-
-# Utilities
-futures = "0.3"
-async-trait = "0.1"
-anyhow = "1"
-thiserror = "1"
-tracing = "0.1"
-sha2 = "0.10"
-bytes = "1"
-# Title-tag parsing (shared filename_meta — season/episode/quality/codec/languages).
-regex = "1"
-
-# Per-plugin midhash/blob cache. Schema is identical to meta-share v1 — see
-# cache.rs. Pure-Rust single-process embedded store.
-redb = "2"
-
-[dev-dependencies]
-tempfile = "3"
-tokio = { version = "1.40", features = ["full", "test-util"] }

crates/meta-feeder-sdk/src/cache.rs

@@ -1,337 +0,0 @@
-//! Per-plugin midhash cache backed by redb.
-//!
-//! Each plugin gets its own redb file at
-//! `<state_dir>/gateway/<upstream_id>/cache.redb`. Keys are upstream
-//! `record_id`s (DOI for sci-hub, book id for gutenberg, …); values are the
-//! `Midhash` strings the plugin computed by fetching + hashing the upstream
-//! file.
-//!
-//! The schema is identical to meta-share v1's so old `.redb` files copy
-//! across. redb is synchronous and sub-millisecond for small ops, so calls
-//! run inline from async handlers without `spawn_blocking`.
-
-use std::path::Path;
-use std::sync::Arc;
-
-use redb::{Database, ReadableTableMetadata, TableDefinition};
-
-pub const CACHE_FILENAME: &str = "cache.redb";
-
-const MIDHASH_TABLE: TableDefinition<'_, &str, &str> = TableDefinition::new("midhash");
-const BLOBS_TABLE: TableDefinition<'_, &str, &[u8]> = TableDefinition::new("blobs");
-const COVER_CID_TABLE: TableDefinition<'_, &str, &str> = TableDefinition::new("cover_cid");
-const BIBREC_TABLE: TableDefinition<'_, &str, &str> = TableDefinition::new("bibrec");
-const PREVIEW_CID_TABLE: TableDefinition<'_, &str, &str> = TableDefinition::new("preview_cid");
-/// Full-file sha2-256 IPFS cid for a previously-fetched torrent payload
-/// (torznab's bt-fetch path). Distinct from `MIDHASH_TABLE` because the
-/// hash families differ: `midhash` is the synthetic midhash256-from-infohash
-/// fallback; `fullhash` records "we actually downloaded the bytes and this is
-/// the IPFS CIDv1 over them."
-const FULLHASH_TABLE: TableDefinition<'_, &str, &str> = TableDefinition::new("fullhash");
-/// Enumerated torrent file list (JSON `Vec<TorrentFile>`) keyed by the
-/// torrent's `record_id` (infohash).
-const FILELIST_TABLE: TableDefinition<'_, &str, &str> = TableDefinition::new("filelist");
-/// Cached TMDB *search* results keyed by a stable lookup key. The value is the
-/// JSON encoding of the top `TmdbHit`, or the literal `"null"` negative-cache
-/// sentinel.
-const TMDB_SEARCH_TABLE: TableDefinition<'_, &str, &str> = TableDefinition::new("tmdb_search");
-/// Cached TMDB `GET /3/tv/{id}` structural details keyed by `tmdbid`.
-const TMDB_TVDETAILS_TABLE: TableDefinition<'_, &str, &str> =
-    TableDefinition::new("tmdb_tvdetails");
-/// Cached TMDB `GET /3/{tv,movie}/{id}/external_ids` keyed by `tmdbid`.
-const TMDB_EXTIDS_TABLE: TableDefinition<'_, &str, &str> = TableDefinition::new("tmdb_extids");
-/// Cached TMDB `GET /3/movie/{id}` details keyed by `tmdbid`.
-const TMDB_MOVIEDETAILS_TABLE: TableDefinition<'_, &str, &str> =
-    TableDefinition::new("tmdb_moviedetails");
-/// Cached **ranked** TMDB `search/multi` anchors keyed by the normalized
-/// free-text query.
-const TMDB_PRINCIPAL_TOPN_TABLE: TableDefinition<'_, &str, &str> =
-    TableDefinition::new("tmdb_principal_topn");
-/// Subtitle linkage discovered at search-enrich time, keyed by the torrent's
-/// `record_id` (infohash). Value is the JSON encoding of a `Vec<SubtitleLink>`.
-const SUBTITLES_TABLE: TableDefinition<'_, &str, &str> = TableDefinition::new("subtitles");
-/// OpenSubtitles search results keyed by `"<tmdb_id>\x01<lang3>"`. Value is the
-/// resolved subtitle's cid, or the literal `"null"` negative-cache sentinel.
-const OPENSUBTITLES_TABLE: TableDefinition<'_, &str, &str> = TableDefinition::new("opensubtitles");
-
-/// Per-plugin midhash cache. Cheap to clone (the inner `Database` is shared
-/// via `Arc`).
-#[derive(Clone)]
-pub struct MidhashCache {
-    db: Arc<Database>,
-}
-
-/// Generate the standard `get`/`put` accessor pair for a `&str → &str` redb
-/// table.
-macro_rules! str_table_accessors {
-    ($(#[$gmeta:meta])* $get:ident, $(#[$pmeta:meta])* $put:ident, $table:ident $(,)?) => {
-        $(#[$gmeta])*
-        pub fn $get(&self, key: &str) -> Result<Option<String>, redb::Error> {
-            let tx = self.db.begin_read()?;
-            let table = tx.open_table($table)?;
-            Ok(table.get(key)?.map(|v| v.value().to_string()))
-        }
-
-        $(#[$pmeta])*
-        pub fn $put(&self, key: &str, value: &str) -> Result<(), redb::Error> {
-            let tx = self.db.begin_write()?;
-            {
-                let mut table = tx.open_table($table)?;
-                table.insert(key, value)?;
-            }
-            tx.commit()?;
-            Ok(())
-        }
-    };
-}
-
-impl MidhashCache {
-    pub fn open(cache_dir: &Path) -> Result<Self, redb::Error> {
-        let path = cache_dir.join(CACHE_FILENAME);
-        let db = Database::create(&path)?;
-        {
-            let tx = db.begin_write()?;
-            tx.open_table(MIDHASH_TABLE)?;
-            tx.open_table(BLOBS_TABLE)?;
-            tx.open_table(COVER_CID_TABLE)?;
-            tx.open_table(BIBREC_TABLE)?;
-            tx.open_table(PREVIEW_CID_TABLE)?;
-            tx.open_table(FULLHASH_TABLE)?;
-            tx.open_table(FILELIST_TABLE)?;
-            tx.open_table(TMDB_SEARCH_TABLE)?;
-            tx.open_table(TMDB_TVDETAILS_TABLE)?;
-            tx.open_table(TMDB_EXTIDS_TABLE)?;
-            tx.open_table(TMDB_MOVIEDETAILS_TABLE)?;
-            tx.open_table(TMDB_PRINCIPAL_TOPN_TABLE)?;
-            tx.open_table(SUBTITLES_TABLE)?;
-            tx.open_table(OPENSUBTITLES_TABLE)?;
-            tx.commit()?;
-        }
-        Ok(Self { db: Arc::new(db) })
-    }
-
-    str_table_accessors!(get_midhash, put_midhash, MIDHASH_TABLE);
-
-    pub fn entry_count(&self) -> Result<u64, redb::Error> {
-        let tx = self.db.begin_read()?;
-        let table = tx.open_table(MIDHASH_TABLE)?;
-        Ok(table.len()?)
-    }
-
-    pub fn get_blob(&self, cid: &str) -> Result<Option<Vec<u8>>, redb::Error> {
-        let tx = self.db.begin_read()?;
-        let table = tx.open_table(BLOBS_TABLE)?;
-        Ok(table.get(cid)?.map(|v| v.value().to_vec()))
-    }
-
-    pub fn put_blob(&self, cid: &str, bytes: &[u8]) -> Result<(), redb::Error> {
-        let tx = self.db.begin_write()?;
-        {
-            let mut table = tx.open_table(BLOBS_TABLE)?;
-            table.insert(cid, bytes)?;
-        }
-        tx.commit()?;
-        Ok(())
-    }
-
-    str_table_accessors!(get_cover_cid, put_cover_cid, COVER_CID_TABLE);
-
-    str_table_accessors!(
-        /// Read a previously-stored full-file IPFS cid for `record_id`.
-        get_fullhash,
-        /// Record the IPFS cid produced by a successful full fetch.
-        put_fullhash,
-        FULLHASH_TABLE
-    );
-
-    str_table_accessors!(
-        /// Read the cached torrent file list (JSON) for `record_id`.
-        get_filelist,
-        /// Cache the enumerated torrent file list (JSON `Vec<TorrentFile>`).
-        put_filelist,
-        FILELIST_TABLE
-    );
-
-    str_table_accessors!(
-        /// Read the cached subtitle linkage (JSON `Vec<SubtitleLink>`).
-        get_subtitles,
-        /// Cache the discovered subtitle linkage (JSON `Vec<SubtitleLink>`).
-        put_subtitles,
-        SUBTITLES_TABLE
-    );
-
-    str_table_accessors!(
-        /// Read a cached OpenSubtitles lookup by `"<tmdb_id>\x01<lang3>"`.
-        get_opensubtitles,
-        /// Cache an OpenSubtitles lookup result (cid or `"null"` sentinel).
-        put_opensubtitles,
-        OPENSUBTITLES_TABLE
-    );
-
-    str_table_accessors!(get_preview_cid, put_preview_cid, PREVIEW_CID_TABLE);
-
-    str_table_accessors!(
-        /// Read a cached TMDB search result by lookup key.
-        get_tmdb_search,
-        /// Cache a TMDB search result under `key`.
-        put_tmdb_search,
-        TMDB_SEARCH_TABLE
-    );
-
-    str_table_accessors!(
-        /// Read cached TMDB TV-details JSON by `tmdbid`.
-        get_tmdb_tvdetails,
-        /// Cache TMDB TV-details JSON under `tmdbid`.
-        put_tmdb_tvdetails,
-        TMDB_TVDETAILS_TABLE
-    );
-
-    str_table_accessors!(
-        /// Read cached TMDB external-ids JSON by `tmdbid`.
-        get_tmdb_extids,
-        /// Cache TMDB external-ids JSON under `tmdbid`.
-        put_tmdb_extids,
-        TMDB_EXTIDS_TABLE
-    );
-
-    str_table_accessors!(
-        /// Read cached TMDB movie-details JSON by `tmdbid`.
-        get_tmdb_moviedetails,
-        /// Cache TMDB movie-details JSON under `tmdbid`.
-        put_tmdb_moviedetails,
-        TMDB_MOVIEDETAILS_TABLE
-    );
-
-    str_table_accessors!(
-        /// Read the cached ranked anchor list by normalized query key.
-        get_tmdb_principal_topn,
-        /// Cache the ranked anchor list under the normalized query key.
-        put_tmdb_principal_topn,
-        TMDB_PRINCIPAL_TOPN_TABLE
-    );
-
-    pub fn get_bibrec(
-        &self,
-        record_id: &str,
-    ) -> Result<Option<std::collections::BTreeMap<String, String>>, redb::Error> {
-        let tx = self.db.begin_read()?;
-        let table = tx.open_table(BIBREC_TABLE)?;
-        let Some(v) = table.get(record_id)? else {
-            return Ok(None);
-        };
-        let s = v.value();
-        Ok(serde_json::from_str(s).ok())
-    }
-
-    pub fn put_bibrec(
-        &self,
-        record_id: &str,
-        fields: &std::collections::BTreeMap<String, String>,
-    ) -> Result<(), redb::Error> {
-        let json = serde_json::to_string(fields).unwrap_or_else(|_| "{}".to_string());
-        let tx = self.db.begin_write()?;
-        {
-            let mut table = tx.open_table(BIBREC_TABLE)?;
-            table.insert(record_id, json.as_str())?;
-        }
-        tx.commit()?;
-        Ok(())
-    }
-}
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-
-    fn tmp_cache_dir() -> tempfile::TempDir {
-        tempfile::tempdir().expect("tempdir")
-    }
-
-    #[test]
-    fn put_then_get_returns_value() {
-        let dir = tmp_cache_dir();
-        let cache = MidhashCache::open(dir.path()).expect("open");
-        assert_eq!(cache.get_midhash("10.1038/x").unwrap(), None);
-        cache.put_midhash("10.1038/x", "bafyXYZ").expect("put");
-        assert_eq!(
-            cache.get_midhash("10.1038/x").unwrap().as_deref(),
-            Some("bafyXYZ")
-        );
-    }
-
-    #[test]
-    fn put_overwrites_existing_value() {
-        let dir = tmp_cache_dir();
-        let cache = MidhashCache::open(dir.path()).expect("open");
-        cache.put_midhash("k", "v1").unwrap();
-        cache.put_midhash("k", "v2").unwrap();
-        assert_eq!(cache.get_midhash("k").unwrap().as_deref(), Some("v2"));
-    }
-
-    #[test]
-    fn values_persist_across_reopens() {
-        let dir = tmp_cache_dir();
-        {
-            let cache = MidhashCache::open(dir.path()).expect("open 1");
-            cache.put_midhash("k", "persistent").unwrap();
-        }
-        let cache = MidhashCache::open(dir.path()).expect("open 2");
-        assert_eq!(
-            cache.get_midhash("k").unwrap().as_deref(),
-            Some("persistent")
-        );
-    }
-
-    #[test]
-    fn tmdb_search_cache_distinguishes_miss_from_absent() {
-        let dir = tmp_cache_dir();
-        let cache = MidhashCache::open(dir.path()).expect("open");
-        assert_eq!(cache.get_tmdb_search("m\u{1}naruto\u{1}").unwrap(), None);
-        cache.put_tmdb_search("m\u{1}naruto\u{1}", "null").unwrap();
-        assert_eq!(
-            cache
-                .get_tmdb_search("m\u{1}naruto\u{1}")
-                .unwrap()
-                .as_deref(),
-            Some("null")
-        );
-        cache
-            .put_tmdb_search("t\u{1}spy x family\u{1}2019", "{\"tmdbid\":120089}")
-            .unwrap();
-        drop(cache);
-        let cache = MidhashCache::open(dir.path()).expect("reopen");
-        assert_eq!(
-            cache
-                .get_tmdb_search("t\u{1}spy x family\u{1}2019")
-                .unwrap()
-                .as_deref(),
-            Some("{\"tmdbid\":120089}")
-        );
-    }
-
-    #[test]
-    fn tmdb_tvdetails_cache_roundtrips() {
-        let dir = tmp_cache_dir();
-        let cache = MidhashCache::open(dir.path()).expect("open");
-        assert_eq!(cache.get_tmdb_tvdetails("120089").unwrap(), None);
-        cache
-            .put_tmdb_tvdetails("120089", "{\"number_of_seasons\":2}")
-            .unwrap();
-        assert_eq!(
-            cache.get_tmdb_tvdetails("120089").unwrap().as_deref(),
-            Some("{\"number_of_seasons\":2}")
-        );
-    }
-
-    #[test]
-    fn entry_count_reflects_inserts() {
-        let dir = tmp_cache_dir();
-        let cache = MidhashCache::open(dir.path()).expect("open");
-        assert_eq!(cache.entry_count().unwrap(), 0);
-        cache.put_midhash("a", "x").unwrap();
-        cache.put_midhash("b", "y").unwrap();
-        assert_eq!(cache.entry_count().unwrap(), 2);
-        cache.put_midhash("a", "x2").unwrap();
-        assert_eq!(cache.entry_count().unwrap(), 2);
-    }
-}

crates/meta-feeder-sdk/src/common.rs

@@ -1,288 +0,0 @@
-//! Shared helpers for single-outcome source plugins (`arxiv`, `gutenberg`,
-//! `pubmed`, `wikicommons`, `giphy`, `scihub`).
-//!
-//! These plugins are structurally near-identical: build a reqwest client, map
-//! HTTP status → [`GatewayError`], and front a per-record midhash cache whose
-//! hit/put boilerplate is the same everywhere. Only the duplicated scaffolding
-//! lives here; the genuinely-different bodies stay in the plugin modules.
-
-use std::path::Path;
-use std::time::Duration;
-
-use tracing::{debug, warn};
-
-use crate::cache::MidhashCache;
-use crate::plugin::{ConfigError, HashKind, HashOutcome};
-use crate::types::{DiscoveryRecord, GatewayError, Hash};
-
-/// Build the shared reqwest client every source plugin uses.
-pub fn build_http_client(
-    timeout_secs: u64,
-    user_agent: &str,
-    redirect: Option<reqwest::redirect::Policy>,
-) -> reqwest::Client {
-    let mut builder = reqwest::Client::builder()
-        .timeout(Duration::from_secs(timeout_secs))
-        .user_agent(user_agent);
-    if let Some(policy) = redirect {
-        builder = builder.redirect(policy);
-    }
-    builder
-        .build()
-        .expect("reqwest client build is infallible with rustls-tls")
-}
-
-/// Map a non-success HTTP status to a [`GatewayError`].
-///
-/// - `404` → [`GatewayError::NotFound`]
-/// - `429` → [`GatewayError::RateLimited`] (honours `Retry-After`, default 60s)
-/// - `5xx` → [`GatewayError::Transient`]
-/// - anything else → [`GatewayError::Permanent`]
-pub fn map_status(resp: &reqwest::Response) -> Result<(), GatewayError> {
-    let status = resp.status();
-    if status.is_success() {
-        return Ok(());
-    }
-    let code = status.as_u16();
-    if code == 404 {
-        return Err(GatewayError::NotFound);
-    }
-    if code == 429 {
-        let retry_after = resp
-            .headers()
-            .get(reqwest::header::RETRY_AFTER)
-            .and_then(|v| v.to_str().ok())
-            .and_then(|v| v.parse::<u32>().ok())
-            .unwrap_or(60);
-        return Err(GatewayError::RateLimited {
-            retry_after_s: retry_after,
-        });
-    }
-    if status.is_server_error() {
-        return Err(GatewayError::Transient(format!(
-            "upstream {} returned {status}",
-            resp.url()
-        )));
-    }
-    Err(GatewayError::Permanent(format!(
-        "upstream {} returned {status}",
-        resp.url()
-    )))
-}
-
-/// Open the per-plugin redb midhash cache, wrapping the error with the plugin
-/// name for a clean startup diagnostic.
-pub fn open_midhash_cache(
-    cache_dir: &Path,
-    plugin: &'static str,
-) -> Result<MidhashCache, ConfigError> {
-    MidhashCache::open(cache_dir).map_err(|e| ConfigError::Other {
-        plugin,
-        source: anyhow::anyhow!("open redb cache: {e}"),
-    })
-}
-
-/// Cache-hit fast path for single-outcome plugins.
-///
-/// On hit, returns `Some(vec![sparse outcome])` — hash only, no bytes/record —
-/// so the core's `(_, None)` branch skips re-storing content. `Ok(None)` is a
-/// cache miss; the caller proceeds to fetch + hash.
-pub fn cached_outcome(
-    cache: &MidhashCache,
-    record_id: &str,
-    upstream_id: &'static str,
-) -> Result<Option<Vec<HashOutcome>>, GatewayError> {
-    let Some(cached) = cache
-        .get_midhash(record_id)
-        .map_err(|e| GatewayError::Internal(anyhow::anyhow!("cache get: {e}")))?
-    else {
-        return Ok(None);
-    };
-    debug!(
-        target: "meta-share::gateway",
-        upstream = upstream_id,
-        record_id,
-        "hash cache hit"
-    );
-    Ok(Some(vec![HashOutcome {
-        hash: Hash(cached),
-        hash_kind: HashKind::Sha2_256,
-        bytes: None,
-        record: None,
-        file_extension: None,
-    }]))
-}
-
-/// Persist a freshly-computed `(record_id → cid)` mapping. A write failure is
-/// non-fatal, so it is logged and swallowed.
-pub fn store_midhash(cache: &MidhashCache, record_id: &str, upstream_id: &'static str, cid: &str) {
-    if let Err(e) = cache.put_midhash(record_id, cid) {
-        warn!(
-            target: "meta-share::gateway",
-            upstream = upstream_id,
-            record_id,
-            error = %e,
-            "hash cache put failed (non-fatal)"
-        );
-    }
-}
-
-/// Resolve the per-plugin cache handle, or produce the standard "not
-/// configured" internal error.
-pub fn require_cache<'a>(
-    cache: Option<&'a MidhashCache>,
-    plugin: &str,
-) -> Result<&'a MidhashCache, GatewayError> {
-    cache.ok_or_else(|| {
-        GatewayError::Internal(anyhow::anyhow!(
-            "{plugin} plugin not configured (configure() never called)"
-        ))
-    })
-}
-
-/// Build the single-element outcome vec a single-outcome plugin returns on a
-/// fresh (cache-miss) compute: the full bytes + record, a Sha2_256 IPFS CID,
-/// and an optional file extension.
-pub fn single_outcome(
-    cid: String,
-    bytes: bytes::Bytes,
-    record: DiscoveryRecord,
-    file_extension: Option<String>,
-) -> Vec<HashOutcome> {
-    vec![HashOutcome {
-        hash: Hash(cid),
-        hash_kind: HashKind::Sha2_256,
-        bytes: Some(bytes),
-        record: Some(record),
-        file_extension,
-    }]
-}
-
-/// Percent-encode a query-string value using the unreserved set
-/// (RFC 3986 `A-Za-z0-9-_.~`) plus `+` for space.
-pub fn urlencode(s: &str) -> String {
-    let mut out = String::with_capacity(s.len());
-    for b in s.as_bytes() {
-        match *b {
-            b'A'..=b'Z' | b'a'..=b'z' | b'0'..=b'9' | b'-' | b'_' | b'.' | b'~' => {
-                out.push(*b as char);
-            }
-            b' ' => out.push('+'),
-            other => out.push_str(&format!("%{:02X}", other)),
-        }
-    }
-    out
-}
-
-/// GET `url` and return the body bytes, mapping transport + status errors to
-/// [`GatewayError`].
-pub async fn fetch_bytes(http: &reqwest::Client, url: &str) -> Result<bytes::Bytes, GatewayError> {
-    let resp = http
-        .get(url)
-        .send()
-        .await
-        .map_err(|e| GatewayError::Transient(format!("GET {url}: {e}")))?;
-    map_status(&resp)?;
-    resp.bytes()
-        .await
-        .map_err(|e| GatewayError::Transient(format!("read body {url}: {e}")))
-}
-
-/// Like [`fetch_bytes`] but decodes the body as UTF-8 text.
-pub async fn fetch_text(http: &reqwest::Client, url: &str) -> Result<String, GatewayError> {
-    let resp = http
-        .get(url)
-        .send()
-        .await
-        .map_err(|e| GatewayError::Transient(format!("GET {url}: {e}")))?;
-    map_status(&resp)?;
-    resp.text()
-        .await
-        .map_err(|e| GatewayError::Transient(format!("read body {url}: {e}")))
-}
-
-/// Decode a v1 BitTorrent infohash string into its raw 20 bytes.
-///
-/// Accepts 40 hex chars (case-insensitive) or 32 base32 chars (RFC 4648,
-/// case-insensitive). Returns `None` for anything else.
-pub fn decode_infohash(s: &str) -> Option<[u8; 20]> {
-    let bytes = s.as_bytes();
-    let raw: Vec<u8> = if bytes.len() == 40 && bytes.iter().all(u8::is_ascii_hexdigit) {
-        let mut out = Vec::with_capacity(20);
-        for pair in bytes.chunks(2) {
-            out.push((hex_nibble(pair[0])? << 4) | hex_nibble(pair[1])?);
-        }
-        out
-    } else if bytes.len() == 32 && bytes.iter().all(u8::is_ascii_alphanumeric) {
-        decode_base32_20(bytes)?
-    } else {
-        return None;
-    };
-    raw.try_into().ok()
-}
-
-fn hex_nibble(b: u8) -> Option<u8> {
-    match b {
-        b'0'..=b'9' => Some(b - b'0'),
-        b'a'..=b'f' => Some(b - b'a' + 10),
-        b'A'..=b'F' => Some(b - b'A' + 10),
-        _ => None,
-    }
-}
-
-/// Decode 32 RFC-4648 base32 chars (case-insensitive, no padding) into the 20
-/// bytes they encode.
-fn decode_base32_20(chars: &[u8]) -> Option<Vec<u8>> {
-    let mut out = Vec::with_capacity(20);
-    let mut buffer: u64 = 0;
-    let mut bits: u32 = 0;
-    for &c in chars {
-        let val = match c {
-            b'A'..=b'Z' => c - b'A',
-            b'a'..=b'z' => c - b'a',
-            b'2'..=b'7' => c - b'2' + 26,
-            _ => return None,
-        } as u64;
-        buffer = (buffer << 5) | val;
-        bits += 5;
-        if bits >= 8 {
-            bits -= 8;
-            out.push((buffer >> bits) as u8);
-        }
-    }
-    Some(out)
-}
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-
-    #[test]
-    fn decode_infohash_hex_roundtrips() {
-        let hex = "0123456789abcdef0123456789abcdef01234567";
-        let raw = decode_infohash(hex).expect("hex decodes");
-        assert_eq!(raw[0], 0x01);
-        assert_eq!(raw[19], 0x67);
-    }
-
-    #[test]
-    fn decode_infohash_is_case_insensitive_hex() {
-        let lower = decode_infohash("aabbccddeeff00112233445566778899aabbccdd");
-        let upper = decode_infohash("AABBCCDDEEFF00112233445566778899AABBCCDD");
-        assert!(lower.is_some());
-        assert_eq!(lower, upper);
-    }
-
-    #[test]
-    fn decode_infohash_accepts_base32() {
-        let raw = decode_infohash("MFRGGZDFMZTWQ2LKNNWG23TPOBYXE43U");
-        assert!(raw.is_some());
-    }
-
-    #[test]
-    fn decode_infohash_rejects_wrong_shape() {
-        assert!(decode_infohash("tooshort").is_none());
-        assert!(decode_infohash("zz23456789abcdef0123456789abcdef01234567").is_none());
-        assert!(decode_infohash("").is_none());
-    }
-}

crates/meta-feeder-sdk/src/config.rs

@@ -1,312 +0,0 @@
-//! Self-describing per-plugin configuration: schema, redaction, merge, and the
-//! generic schema-driven HTML page the feeder serves at `/config`.
-//!
-//! The point of this module is that **the gateway carries no per-plugin config
-//! knowledge** and **the SDK carries no per-plugin field names**. A plugin
-//! declares its fields via [`FeederPlugin::config_schema`](crate::FeederPlugin::config_schema);
-//! everything here (redaction of secrets, merge-on-save, and the rendered form)
-//! is driven entirely by that schema. Add a feeder → it shows up with a working
-//! config form and zero UI/gateway changes.
-//!
-//! ## Persistence & precedence
-//!
-//! Config lives at `cache_dir/config.json` (the per-plugin dir handed to
-//! `configure()`). The plugin reads **file-or-env** at startup: env is the
-//! first-boot seed and stays authoritative until the operator saves through the
-//! UI, at which point `config.json` exists and wins. There is no runtime reload
-//! — a save returns `restart_required: true` (mirrors the gateway's
-//! `gateway-config.json` posture).
-//!
-//! ## Secrets
-//!
-//! Fields of kind [`FieldKind::Secret`] are **write-only**: never returned to
-//! the page (redacted to a `<key>_set: bool`), and a blank incoming value on
-//! save means "keep the stored secret".
-
-use serde::{Deserialize, Serialize};
-use serde_json::{Map, Value};
-
-/// A plugin's full config surface — an ordered list of fields.
-#[derive(Serialize, Deserialize, Clone, Debug, Default)]
-pub struct ConfigSchema {
-    pub fields: Vec<ConfigField>,
-}
-
-impl ConfigSchema {
-    /// Convenience: `true` when the plugin declares no configurable fields (the
-    /// page then renders a "no configuration required" notice).
-    pub fn is_empty(&self) -> bool {
-        self.fields.is_empty()
-    }
-}
-
-/// One configurable field. `record_array` fields nest `fields` for the
-/// per-record sub-schema (e.g. torznab's `indexers`).
-#[derive(Serialize, Deserialize, Clone, Debug)]
-pub struct ConfigField {
-    /// JSON key in the config object.
-    pub key: String,
-    /// Human label for the form.
-    pub label: String,
-    #[serde(rename = "type")]
-    pub kind: FieldKind,
-    #[serde(default, skip_serializing_if = "Option::is_none")]
-    pub help: Option<String>,
-    #[serde(default)]
-    pub required: bool,
-    /// Sub-fields for [`FieldKind::RecordArray`]. The FIRST sub-field is treated
-    /// as the record identifier for secret back-fill on merge (e.g. `name`).
-    #[serde(default, skip_serializing_if = "Option::is_none")]
-    pub fields: Option<Vec<ConfigField>>,
-}
-
-impl ConfigField {
-    pub fn text(key: &str, label: &str) -> Self {
-        Self::new(key, label, FieldKind::Text)
-    }
-    pub fn secret(key: &str, label: &str) -> Self {
-        Self::new(key, label, FieldKind::Secret)
-    }
-    pub fn bool(key: &str, label: &str) -> Self {
-        Self::new(key, label, FieldKind::Bool)
-    }
-    pub fn number(key: &str, label: &str) -> Self {
-        Self::new(key, label, FieldKind::Number)
-    }
-    pub fn list(key: &str, label: &str) -> Self {
-        Self::new(key, label, FieldKind::List)
-    }
-    pub fn record_array(key: &str, label: &str, fields: Vec<ConfigField>) -> Self {
-        let mut f = Self::new(key, label, FieldKind::RecordArray);
-        f.fields = Some(fields);
-        f
-    }
-    fn new(key: &str, label: &str, kind: FieldKind) -> Self {
-        Self {
-            key: key.to_string(),
-            label: label.to_string(),
-            kind,
-            help: None,
-            required: false,
-            fields: None,
-        }
-    }
-    pub fn with_help(mut self, help: &str) -> Self {
-        self.help = Some(help.to_string());
-        self
-    }
-    pub fn required(mut self) -> Self {
-        self.required = true;
-        self
-    }
-}
-
-/// Field widget / value type. Drives both rendering and the secret/merge rules.
-#[derive(Serialize, Deserialize, Clone, Copy, Debug, PartialEq, Eq)]
-#[serde(rename_all = "snake_case")]
-pub enum FieldKind {
-    /// Free text → JSON string.
-    Text,
-    /// Write-only secret → JSON string; redacted on read, "blank keeps" on save.
-    Secret,
-    /// Checkbox → JSON bool.
-    Bool,
-    /// Numeric input → JSON number.
-    Number,
-    /// Comma/line list → JSON array of strings.
-    List,
-    /// Repeatable record → JSON array of objects keyed by nested `fields`.
-    RecordArray,
-}
-
-fn is_blank(v: Option<&Value>) -> bool {
-    match v {
-        None | Some(Value::Null) => true,
-        Some(Value::String(s)) => s.is_empty(),
-        _ => false,
-    }
-}
-
-/// Produce a UI-safe view of `values`: secret fields are dropped and replaced
-/// with a `<key>_set` boolean; everything else passes through. Recurses into
-/// `record_array` records for secret sub-fields.
-pub fn redact(values: &Value, schema: &ConfigSchema) -> Value {
-    let src = values.as_object().cloned().unwrap_or_default();
-    Value::Object(redact_object(&src, &schema.fields))
-}
-
-fn redact_object(src: &Map<String, Value>, fields: &[ConfigField]) -> Map<String, Value> {
-    let mut out = Map::new();
-    for f in fields {
-        match f.kind {
-            FieldKind::Secret => {
-                let set = !is_blank(src.get(&f.key));
-                out.insert(format!("{}_set", f.key), Value::Bool(set));
-            }
-            FieldKind::RecordArray => {
-                let subs = f.fields.as_deref().unwrap_or(&[]);
-                let arr = src
-                    .get(&f.key)
-                    .and_then(Value::as_array)
-                    .map(|recs| {
-                        recs.iter()
-                            .map(|r| {
-                                let m = r.as_object().cloned().unwrap_or_default();
-                                Value::Object(redact_object(&m, subs))
-                            })
-                            .collect::<Vec<_>>()
-                    })
-                    .unwrap_or_default();
-                out.insert(f.key.clone(), Value::Array(arr));
-            }
-            _ => {
-                if let Some(v) = src.get(&f.key) {
-                    out.insert(f.key.clone(), v.clone());
-                }
-            }
-        }
-    }
-    out
-}
-
-/// Merge `incoming` (from the UI) onto `base` (the stored/effective config),
-/// schema-aware. Returns the full, UNREDACTED object to persist:
-/// - secret: blank/absent incoming keeps `base`; non-blank overwrites.
-/// - text/number/bool/list: present in incoming overwrites; absent keeps base.
-/// - record_array: incoming array replaces base, but blank secret sub-fields are
-///   back-filled from the base record matched by the first sub-field (identifier).
-pub fn merge(base: &Value, incoming: &Value, schema: &ConfigSchema) -> Value {
-    let base_obj = base.as_object().cloned().unwrap_or_default();
-    let inc_obj = incoming.as_object().cloned().unwrap_or_default();
-    Value::Object(merge_object(&base_obj, &inc_obj, &schema.fields))
-}
-
-fn merge_object(
-    base: &Map<String, Value>,
-    inc: &Map<String, Value>,
-    fields: &[ConfigField],
-) -> Map<String, Value> {
-    let mut out = base.clone();
-    for f in fields {
-        match f.kind {
-            FieldKind::Secret => {
-                if !is_blank(inc.get(&f.key)) {
-                    out.insert(f.key.clone(), inc.get(&f.key).cloned().unwrap());
-                }
-                // blank/absent → keep base (no-op)
-            }
-            FieldKind::RecordArray => {
-                let subs = f.fields.as_deref().unwrap_or(&[]);
-                let id_key = subs.first().map(|s| s.key.clone());
-                let base_recs = base.get(&f.key).and_then(Value::as_array).cloned();
-                let inc_recs = match inc.get(&f.key).and_then(Value::as_array) {
-                    Some(r) => r.clone(),
-                    None => continue, // absent → keep base array untouched
-                };
-                let merged: Vec<Value> = inc_recs
-                    .iter()
-                    .map(|rec| {
-                        let rec_obj = rec.as_object().cloned().unwrap_or_default();
-                        // Find the matching base record by identifier sub-field.
-                        let base_match = match (&id_key, base_recs.as_ref()) {
-                            (Some(idk), Some(brs)) => brs
-                                .iter()
-                                .find(|br| br.get(idk) == rec_obj.get(idk))
-                                .and_then(Value::as_object)
-                                .cloned()
-                                .unwrap_or_default(),
-                            _ => Map::new(),
-                        };
-                        Value::Object(merge_object(&base_match, &rec_obj, subs))
-                    })
-                    .collect();
-                out.insert(f.key.clone(), Value::Array(merged));
-            }
-            _ => {
-                if let Some(v) = inc.get(&f.key) {
-                    out.insert(f.key.clone(), v.clone());
-                }
-            }
-        }
-    }
-    out
-}
-
-/// The generic, schema-driven config page served at `GET /config`. Self-contained
-/// (inline CSS + JS, no external assets) so it loads identically when hit
-/// directly on the feeder OR reverse-proxied through the gateway. All API calls
-/// are relative to `location.pathname`, so the proxy mount prefix is irrelevant.
-pub const CONFIG_PAGE_HTML: &str = include_str!("config_page.html");
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-    use serde_json::json;
-
-    fn torznab_like() -> ConfigSchema {
-        ConfigSchema {
-            fields: vec![
-                ConfigField::secret("tmdb_token", "TMDB token"),
-                ConfigField::bool("resolve_torrent_files", "Resolve .torrent"),
-                ConfigField::record_array(
-                    "indexers",
-                    "Indexers",
-                    vec![
-                        ConfigField::text("name", "Name"),
-                        ConfigField::secret("api_key", "API key"),
-                    ],
-                ),
-            ],
-        }
-    }
-
-    #[test]
-    fn redact_hides_secrets_and_flags_set() {
-        let v = json!({
-            "tmdb_token": "abc",
-            "resolve_torrent_files": true,
-            "indexers": [{"name":"prowlarr","api_key":"k"}]
-        });
-        let r = redact(&v, &torznab_like());
-        assert_eq!(r["tmdb_token_set"], json!(true));
-        assert!(r.get("tmdb_token").is_none());
-        assert_eq!(r["resolve_torrent_files"], json!(true));
-        assert_eq!(r["indexers"][0]["name"], json!("prowlarr"));
-        assert_eq!(r["indexers"][0]["api_key_set"], json!(true));
-        assert!(r["indexers"][0].get("api_key").is_none());
-    }
-
-    #[test]
-    fn merge_blank_secret_keeps_base() {
-        let base = json!({"tmdb_token": "keepme"});
-        let inc = json!({"tmdb_token": ""});
-        let m = merge(&base, &inc, &torznab_like());
-        assert_eq!(m["tmdb_token"], json!("keepme"));
-    }
-
-    #[test]
-    fn merge_nonblank_secret_overwrites() {
-        let base = json!({"tmdb_token": "old"});
-        let inc = json!({"tmdb_token": "new"});
-        let m = merge(&base, &inc, &torznab_like());
-        assert_eq!(m["tmdb_token"], json!("new"));
-    }
-
-    #[test]
-    fn merge_record_array_backfills_blank_secret_by_identifier() {
-        let base = json!({"indexers":[{"name":"prowlarr","api_key":"secret"}]});
-        // UI re-submits the indexer with a blank api_key → keep the stored one.
-        let inc = json!({"indexers":[{"name":"prowlarr","api_key":""}]});
-        let m = merge(&base, &inc, &torznab_like());
-        assert_eq!(m["indexers"][0]["api_key"], json!("secret"));
-        assert_eq!(m["indexers"][0]["name"], json!("prowlarr"));
-    }
-
-    #[test]
-    fn merge_bool_overwrites_when_present() {
-        let base = json!({"resolve_torrent_files": true});
-        let inc = json!({"resolve_torrent_files": false});
-        let m = merge(&base, &inc, &torznab_like());
-        assert_eq!(m["resolve_torrent_files"], json!(false));
-    }
-}

crates/meta-feeder-sdk/src/config_page.html

@@ -1,268 +0,0 @@
-<!DOCTYPE html>
-<html lang="en">
-<head>
-<meta charset="utf-8" />
-<meta name="viewport" content="width=device-width, initial-scale=1" />
-<title>feeder config</title>
-<style>
-  :root { color-scheme: dark; }
-  * { box-sizing: border-box; }
-  body {
-    margin: 0; padding: 18px;
-    font: 14px/1.5 system-ui, -apple-system, Segoe UI, Roboto, sans-serif;
-    background: #0d0d12; color: #e6e6ea;
-  }
-  h1 { font-size: 16px; margin: 0 0 4px; }
-  .sub { color: #9a9aa6; font-size: 12px; margin: 0 0 16px; }
-  .field { margin: 0 0 16px; }
-  .field > label { display: block; font-weight: 600; margin: 0 0 4px; }
-  .help { color: #8a8a96; font-size: 12px; margin: 3px 0 0; }
-  .req { color: #f0a0a0; margin-left: 3px; font-weight: 700; }
-  input[type=text], input[type=password], input[type=number], textarea {
-    width: 100%; padding: 8px 10px; border-radius: 8px;
-    border: 1px solid #2a2a35; background: #15151c; color: #e6e6ea; font: inherit;
-  }
-  textarea { min-height: 56px; resize: vertical; }
-  input[type=checkbox] { width: 16px; height: 16px; vertical-align: middle; }
-  .pill { font-size: 10px; padding: 2px 7px; border-radius: 999px; margin-left: 8px; vertical-align: middle; }
-  .pill.set { background: #14361f; color: #6ee7a0; border: 1px solid #1f5132; }
-  .pill.unset { background: #2a2a35; color: #b9b9c4; border: 1px solid #3a3a45; }
-  .record { border: 1px solid #2a2a35; border-radius: 10px; padding: 12px; margin: 0 0 10px; background: #111118; }
-  .record .field { margin-bottom: 10px; }
-  .row-actions { text-align: right; }
-  button {
-    font: inherit; cursor: pointer; border-radius: 8px; padding: 7px 14px;
-    border: 1px solid #3a3a45; background: #1c1c26; color: #e6e6ea;
-  }
-  button.primary { background: #6d4aff; border-color: #6d4aff; color: #fff; font-weight: 600; }
-  button.danger { background: transparent; border-color: #5a2a2a; color: #e29; padding: 4px 10px; }
-  button.add { margin-top: 4px; }
-  .bar { position: sticky; bottom: 0; padding: 12px 0 2px; background: linear-gradient(#0d0d1200, #0d0d12 30%); }
-  .banner { padding: 9px 12px; border-radius: 8px; margin: 0 0 14px; font-size: 13px; display: none; }
-  .banner.ok { background: #14361f; color: #9ff0bf; border: 1px solid #1f5132; display: block; }
-  .banner.warn { background: #3a2f12; color: #f0d79f; border: 1px solid #5a4a1f; display: block; }
-  .banner.err { background: #3a1414; color: #f0a0a0; border: 1px solid #5a2020; display: block; }
-  .empty { color: #9a9aa6; }
-</style>
-</head>
-<body>
-  <h1 id="title">Plugin configuration</h1>
-  <p class="sub">Secrets are write-only — set values are shown as “set”, never echoed back. Saving requires a feeder restart to take effect. Fields marked <span class="req">*</span> are required.</p>
-  <div id="banner" class="banner"></div>
-  <form id="form"></form>
-  <div class="bar" id="bar" style="display:none">
-    <button type="button" class="primary" id="save">Save</button>
-  </div>
-
-<script>
-// All API calls are relative to this page's path, so the same page works when
-// served directly on the feeder (/config) or reverse-proxied through the
-// gateway (/api/gateway/plugins/<id>/ui/config).
-const BASE = location.pathname.replace(/\/+$/, "");
-let SCHEMA = { fields: [] };
-let VALUES = {};
-
-const $ = (id) => document.getElementById(id);
-function banner(kind, msg) {
-  const b = $("banner");
-  b.className = "banner " + kind;
-  b.textContent = msg;
-}
-
-function el(tag, attrs, ...kids) {
-  const e = document.createElement(tag);
-  for (const [k, v] of Object.entries(attrs || {})) {
-    if (k === "class") e.className = v;
-    else if (k === "text") e.textContent = v;
-    else if (k in e) e[k] = v;
-    else e.setAttribute(k, v);
-  }
-  for (const c of kids) if (c != null) e.append(c);
-  return e;
-}
-
-// Split help text into plain runs + clickable <a> nodes for any http(s) URL.
-// Built as DOM nodes (never innerHTML) so help stays XSS-safe; trailing
-// punctuation is kept outside the link so "see http://host/ui." reads right.
-function linkify(text) {
-  const out = [];
-  const re = /https?:\/\/[^\s]+/g;
-  let last = 0, m;
-  while ((m = re.exec(text)) !== null) {
-    if (m.index > last) out.push(text.slice(last, m.index));
-    let url = m[0], trail = "";
-    const t = url.match(/[.,;:)\]]+$/);
-    if (t) { trail = t[0]; url = url.slice(0, -trail.length); }
-    out.push(el("a", { href: url, target: "_blank", rel: "noopener noreferrer", text: url }));
-    if (trail) out.push(trail);
-    last = m.index + m[0].length;
-  }
-  if (last < text.length) out.push(text.slice(last));
-  return out;
-}
-
-function helpEl(f) { return f.help ? el("p", { class: "help" }, ...linkify(f.help)) : null; }
-
-// A "*" marker for required fields (null for optional ones).
-function reqStar(f) { return f.required ? el("span", { class: "req", text: "*", title: "required" }) : null; }
-
-// Render one scalar field. `vals` is the object holding the value/`_set` flag.
-// Every returned widget exposes `_collect()` and `_key` so collect() can read it.
-function fieldInput(f, vals) {
-  if (f.type === "bool") {
-    const cb = el("input", { type: "checkbox" });
-    cb.checked = vals[f.key] === true;
-    const w = el("div", { class: "field" },
-      el("label", {}, cb, " ", el("span", { text: " " + f.label }), reqStar(f)),
-      helpEl(f));
-    w._collect = () => cb.checked;
-    w._key = f.key;
-    return w;
-  }
-  const label = el("label", { text: f.label });
-  const star = reqStar(f); if (star) label.append(star);
-  let input;
-  if (f.type === "secret") {
-    const isSet = vals[f.key + "_set"] === true;
-    input = el("input", { type: "password", placeholder: isSet ? "•••••••• (blank keeps current)" : "not set" });
-    label.append(el("span", { class: "pill " + (isSet ? "set" : "unset"), text: isSet ? "set" : "not set" }));
-    input._get = () => input.value; // blank => merge keeps stored secret
-  } else if (f.type === "number") {
-    input = el("input", { type: "number" });
-    if (vals[f.key] != null) input.value = vals[f.key];
-    input._get = () => (input.value === "" ? null : Number(input.value));
-  } else if (f.type === "list") {
-    input = el("textarea", { placeholder: "one per line or comma-separated" });
-    const arr = Array.isArray(vals[f.key]) ? vals[f.key] : [];
-    input.value = arr.join("\n");
-    input._get = () => input.value.split(/[\n,]/).map((s) => s.trim()).filter(Boolean);
-  } else {
-    input = el("input", { type: "text" });
-    if (vals[f.key] != null) input.value = vals[f.key];
-    input._get = () => input.value;
-  }
-  const wrap = el("div", { class: "field" }, label, input, helpEl(f));
-  wrap._collect = () => input._get();
-  wrap._key = f.key;
-  return wrap;
-}
-
-// Render a record_array (repeatable group). Returns a wrapper exposing _collect.
-function recordArray(f, vals) {
-  const subs = f.fields || [];
-  const idKey = subs.length ? subs[0].key : null;
-  const list = el("div", {});
-  const records = Array.isArray(vals[f.key]) ? vals[f.key].slice() : [];
-
-  function addRecord(recVals) {
-    const card = el("div", { class: "record" });
-    const inputs = subs.map((sf) => {
-      const fi = fieldInput(sf, recVals || {});
-      card.append(fi);
-      return fi;
-    });
-    const rm = el("button", { type: "button", class: "danger", text: "remove" });
-    rm.onclick = () => card.remove();
-    card.append(el("div", { class: "row-actions" }, rm));
-    card._collect = () => {
-      const o = {};
-      for (const fi of inputs) o[fi._key] = fi._collect();
-      return o;
-    };
-    list.append(card);
-    return card;
-  }
-
-  records.forEach((r) => addRecord(r));
-  const add = el("button", { type: "button", class: "add", text: "+ add " + f.label.toLowerCase().replace(/s$/, "") });
-  add.onclick = () => addRecord({});
-
-  const lbl = el("label", { text: f.label });
-  const star = reqStar(f); if (star) lbl.append(star);
-  const wrap = el("div", { class: "field" }, lbl, helpEl(f), list, add);
-  wrap._key = f.key;
-  wrap._collect = () => Array.from(list.children).map((c) => c._collect());
-  return wrap;
-}
-
-function render() {
-  $("title").textContent = (SCHEMA.title || "Plugin") + " configuration";
-  const form = $("form");
-  form.innerHTML = "";
-  if (!SCHEMA.fields || !SCHEMA.fields.length) {
-    form.append(el("p", { class: "empty", text: "No configuration required for this plugin." }));
-    $("bar").style.display = "none";
-    return;
-  }
-  const widgets = [];
-  for (const f of SCHEMA.fields) {
-    const w = f.type === "record_array" ? recordArray(f, VALUES) : fieldInput(f, VALUES);
-    widgets.push(w);
-    form.append(w);
-  }
-  form._widgets = widgets;
-  $("bar").style.display = "block";
-}
-
-function collect() {
-  const out = {};
-  for (const w of $("form")._widgets || []) out[w._key] = w._collect();
-  return out;
-}
-
-async function save() {
-  $("save").disabled = true;
-  try {
-    const res = await fetch(BASE + "/values", {
-      method: "PUT",
-      headers: { "Content-Type": "application/json" },
-      body: JSON.stringify(collect()),
-    });
-    if (!res.ok) {
-      const detail = (await res.text().catch(() => "")).trim();
-      throw new Error(detail ? `the feeder rejected it (HTTP ${res.status}): ${detail}`
-                             : `the feeder returned HTTP ${res.status}`);
-    }
-    const j = await res.json().catch(() => ({}));
-    if (j.reloading) {
-      // The feeder is restarting itself (~2-3s) to apply the change. Poll it
-      // back up, then refresh the form so secret pills reflect stored state.
-      banner("warn", "Saved — reloading the feeder to apply (~3 s)…");
-      let tries = 6;
-      const poll = async () => {
-        try { await load(); banner("ok", "Reloaded — new config applied."); }
-        catch { if (--tries > 0) setTimeout(poll, 1500); else banner("warn", "Saved & reloading — reopen this panel shortly to confirm."); }
-      };
-      setTimeout(poll, 3000);
-    } else {
-      if (j.restart_required) banner("warn", "Saved. Restart the feeder container for changes to take effect.");
-      else banner("ok", "Saved.");
-      // Re-fetch so secret pills reflect the new stored state.
-      await load();
-    }
-  } catch (e) {
-    const msg = (e && e.message) ? e.message : String(e);
-    const friendly = /Failed to fetch|NetworkError|load failed/i.test(msg)
-      ? "couldn't reach the feeder — it may be restarting; wait a moment and try again."
-      : msg;
-    banner("err", "Could not save: " + friendly);
-  } finally {
-    $("save").disabled = false;
-  }
-}
-
-async function load() {
-  const [s, v] = await Promise.all([
-    fetch(BASE + "/schema").then((r) => r.json()),
-    fetch(BASE + "/values").then((r) => r.json()),
-  ]);
-  SCHEMA = s || { fields: [] };
-  VALUES = v || {};
-  render();
-}
-
-$("save").onclick = save;
-load().catch((e) => banner("err", "Failed to load config: " + e.message));
-</script>
-</body>
-</html>

crates/meta-feeder-sdk/src/enrich.rs

@@ -1,614 +0,0 @@
-//! Feeder-driven enrichment — the "smart feeder / dumb gateway" pipeline.
-//!
-//! A feeder that self-publishes (has a `META_CORE_URL`) writes its own base
-//! record, then drives the **reused meta-sort enrichment plugins**
-//! (`metamesh-plugin-filename-parser`, `metamesh-plugin-tmdb`) over HTTP. Each
-//! plugin is handed the `metaCoreUrl` and writes its own fields straight back to
-//! meta-core (`PATCH /meta/{cid}` merge), so the gateway never touches
-//! enrichment — full separation of concerns.
-//!
-//! ## Ordering — why we poll
-//!
-//! tmdb searches on `originalTitle`/`movieYear`/`videoType` (which it reads from
-//! the record), and those are produced by filename-parser. So the pipeline is
-//! **filename-parser → tmdb**. The plugin `/process` contract is *async*
-//! (returns `accepted`, then writes to meta-core and POSTs a callback), so after
-//! kicking filename-parser we **poll meta-core** until `originalTitle` appears
-//! (bounded), read the merged fields back, and hand them to tmdb. Polling keeps
-//! the feeder stateless — no callback-correlation bookkeeping. The plugins'
-//! callback POSTs still fire; they hit the feeder's no-op `/enrich/callback`.
-//!
-//! The whole pipeline runs in a spawned background task so it never adds latency
-//! to the libp2p resolve path; the meta-share ingester propagates the enriched
-//! record on its next poll (~eventually consistent).
-
-use std::collections::BTreeMap;
-use std::sync::Arc;
-use std::time::{Duration, SystemTime, UNIX_EPOCH};
-
-use serde::Serialize;
-use tracing::{debug, warn};
-
-use crate::meta_core::{self, Provenance};
-use crate::types::DiscoveryRecord;
-
-/// Operator-supplied enrichment wiring, parsed once from the feeder env.
-#[derive(Clone, Debug)]
-pub struct EnrichmentConfig {
-    /// meta-core root the feeder self-publishes to + hands to the plugins.
-    pub meta_core_url: String,
-    /// `metamesh-plugin-filename-parser` base URL (optional).
-    pub filename_parser_url: Option<String>,
-    /// `metamesh-plugin-tmdb` base URL (optional).
-    pub tmdb_url: Option<String>,
-    /// TMDB v3 key / v4 bearer token, POSTed to tmdb `/configure`.
-    pub tmdb_token: Option<String>,
-    /// TMDB metadata language (default `en-US`).
-    pub tmdb_language: String,
-    /// Stable per-deployment id stamped into `provenance.gatewayPeer`.
-    pub gateway_peer: String,
-    /// URL the plugins POST their completion callback to (the feeder's own
-    /// no-op `/enrich/callback`). Polling is the real completion signal; this
-    /// only keeps the plugins from logging a failed-callback warning.
-    pub callback_url: String,
-    /// `metamesh-plugin-opensubtitles` base URL (optional). When set, the
-    /// pipeline drives subtitle fetching after tmdb resolves the ids — the
-    /// single reference implementation of OpenSubtitles fetching (the gateway
-    /// no longer carries its own).
-    pub opensubtitles_url: Option<String>,
-    /// OpenSubtitles consumer API key, POSTed to the plugin's `/configure`.
-    pub opensubtitles_api_key: Option<String>,
-    /// OpenSubtitles account username (required by the plugin for downloads).
-    pub opensubtitles_username: Option<String>,
-    /// OpenSubtitles account password.
-    pub opensubtitles_password: Option<String>,
-    /// Wanted subtitle languages (CSV of ISO 639-1 codes; default `en`).
-    pub opensubtitles_languages: String,
-}
-
-fn env_nonempty(key: &str) -> Option<String> {
-    std::env::var(key)
-        .ok()
-        .map(|s| s.trim().to_string())
-        .filter(|s| !s.is_empty())
-}
-
-/// Derive a co-bundled sidecar plugin URL from the feeder's own hostname, by the
-/// app-compose convention `<prefix>-feeder` → `<prefix>-<plugin>:8080`
-/// (e.g. `metafeedertorznab-feeder` → `http://metafeedertorznab-tmdb:8080`). Lets
-/// a feeder reach its bundled filename-parser / tmdb sidecars with **no explicit
-/// `*_PLUGIN_URL` env** — config lives in the UI, wiring is derived. Returns
-/// `None` when `HOSTNAME` is unset or not in `<prefix>-feeder` form, in which case
-/// the caller falls back to the env seed (then soft-skips that plugin).
-fn derive_sidecar_url(plugin: &str) -> Option<String> {
-    let host = env_nonempty("HOSTNAME")?;
-    let prefix = host.strip_suffix("-feeder")?;
-    Some(format!("http://{prefix}-{plugin}:8080"))
-}
-
-impl EnrichmentConfig {
-    /// Parse from the feeder env. Returns `None` when `META_CORE_URL` is unset —
-    /// the **bundled-feeder** path (records returned over HTTP, the gateway
-    /// dispatcher stores them). Self-publishing feeders that source meta-core from
-    /// their own dashboard config use [`from_meta_core`](Self::from_meta_core).
-    pub fn from_env() -> Option<Self> {
-        Some(Self::from_meta_core(env_nonempty("META_CORE_URL")?))
-    }
-
-    /// Build with an explicit meta-core root — sourced from the feeder's own
-    /// `config.json` (dashboard) when set, else the `META_CORE_URL` env seed. The
-    /// sidecar URLs come from their `*_PLUGIN_URL` env, falling back to the
-    /// hostname-derived co-bundled sidecar (see [`derive_sidecar_url`]); the rest
-    /// keep their env seeds / defaults. Secrets (tmdb token, language) are
-    /// overlaid by the caller from the per-feeder config.
-    pub fn from_meta_core(meta_core_url: String) -> Self {
-        EnrichmentConfig {
-            meta_core_url,
-            filename_parser_url: env_nonempty("FILENAME_PARSER_URL")
-                .or_else(|| derive_sidecar_url("filename-parser")),
-            tmdb_url: env_nonempty("TMDB_PLUGIN_URL").or_else(|| derive_sidecar_url("tmdb")),
-            tmdb_token: env_nonempty("TMDB_TOKEN").or_else(|| env_nonempty("TORZNAB_TMDB_TOKEN")),
-            tmdb_language: env_nonempty("TMDB_LANGUAGE").unwrap_or_else(|| "en-US".to_string()),
-            gateway_peer: env_nonempty("META_GATEWAY_PEER_ID")
-                .or_else(|| env_nonempty("HOSTNAME"))
-                .unwrap_or_else(|| "gateway-feeder".to_string()),
-            callback_url: env_nonempty("META_FEEDER_CALLBACK_URL")
-                .unwrap_or_else(|| "http://127.0.0.1:8080/enrich/callback".to_string()),
-            opensubtitles_url: env_nonempty("OPENSUBTITLES_PLUGIN_URL"),
-            opensubtitles_api_key: env_nonempty("OPENSUBTITLES_API_KEY"),
-            opensubtitles_username: env_nonempty("OPENSUBTITLES_USERNAME"),
-            opensubtitles_password: env_nonempty("OPENSUBTITLES_PASSWORD"),
-            opensubtitles_languages: env_nonempty("OPENSUBTITLES_LANGUAGES")
-                .unwrap_or_else(|| "en".to_string()),
-        }
-    }
-}
-
-/// How long to wait for filename-parser to merge `originalTitle` before handing
-/// off to tmdb. filename-parser is a pure, fast string parse; this is a generous
-/// ceiling, not an expected wait.
-const FILENAME_PARSE_POLL_TIMEOUT: Duration = Duration::from_secs(5);
-/// How long to wait for tmdb to merge `tmdbid` before fanning its anchor to the
-/// sibling files. tmdb hits the network, so a longer ceiling than filename.
-const TMDB_POLL_TIMEOUT: Duration = Duration::from_secs(20);
-const POLL_INTERVAL: Duration = Duration::from_millis(200);
-
-/// Show-level TMDB fields that are **fanned** from the primary file to every
-/// sibling file in a multi-file (season-pack) torrent — they're identical for
-/// every episode of a show, so we resolve tmdb **once** and copy these across
-/// (avoids N TMDB lookups; the tmdb plugin caches by `cid_midhash256`, which
-/// gateway BT records don't carry, so without this each file would re-hit TMDB).
-/// Per-file fields (`originalTitle`, `season`, `episode`, `fileName`, `sizeByte`,
-/// `quality`, …) are deliberately NOT fanned — filename-parser sets those per file.
-const ANCHOR_EXACT: &[&str] = &[
-    "tmdbid",
-    "imdbid",
-    "poster",
-    "backdrop",
-    "rating",
-    "releasedate",
-    "studio",
-    "genres",
-    "tags",
-];
-const ANCHOR_PREFIXES: &[&str] = &["plot/", "genres/", "studio/", "tags/"];
-
-/// One file (or pack) to enrich: its CID, the filename to parse, and the base
-/// fields already written to meta-core (handed to the plugins as `existingMeta`).
-#[derive(Clone, Debug)]
-pub struct EnrichTarget {
-    pub cid: String,
-    pub file_name: String,
-    pub base: BTreeMap<String, String>,
-}
-
-/// Drives meta-core writes + the enrichment plugins for one feeder. Cheap to
-/// clone (just a reqwest client + an `Arc<config>`), so it can be moved into a
-/// spawned per-resolve task.
-#[derive(Clone)]
-pub struct Enricher {
-    http: reqwest::Client,
-    cfg: Arc<EnrichmentConfig>,
-}
-
-#[derive(Serialize)]
-struct ProcessRequest<'a> {
-    #[serde(rename = "taskId")]
-    task_id: String,
-    cid: &'a str,
-    #[serde(rename = "filePath")]
-    file_path: &'a str,
-    #[serde(rename = "callbackUrl")]
-    callback_url: &'a str,
-    #[serde(rename = "metaCoreUrl")]
-    meta_core_url: &'a str,
-    #[serde(rename = "existingMeta")]
-    existing_meta: &'a BTreeMap<String, String>,
-}
-
-impl Enricher {
-    pub fn new(http: reqwest::Client, cfg: EnrichmentConfig) -> Self {
-        Enricher {
-            http,
-            cfg: Arc::new(cfg),
-        }
-    }
-
-    pub fn meta_core_url(&self) -> &str {
-        &self.cfg.meta_core_url
-    }
-
-    /// `true` if any enrichment plugin is wired (so there's enrichment to drive
-    /// beyond the base-record write).
-    pub fn has_plugins(&self) -> bool {
-        self.cfg.filename_parser_url.is_some() || self.cfg.tmdb_url.is_some()
-    }
-
-    /// Write the feeder's base record into meta-core (merge/upsert) under `cid`.
-    /// Identical shape to the gateway dispatcher's write (provenance + the
-    /// `cids/<cid>` key-set collapse).
-    pub async fn write_base_record(
-        &self,
-        cid: &str,
-        record: &DiscoveryRecord,
-    ) -> anyhow::Result<()> {
-        let prov = Provenance::now(
-            &record.upstream_id,
-            &record.record_id,
-            &self.cfg.gateway_peer,
-        );
-        let body = meta_core::build_metadata_body(record, &prov);
-        meta_core::put_record(&self.http, &self.cfg.meta_core_url, cid, &body).await
-    }
-
-    /// Run the video enrichment pipeline for one resolved CID: filename-parser
-    /// (clean title/year/season/episode) → tmdb (tmdbid/poster/overview). Each
-    /// plugin writes its own fields to meta-core. Best-effort: any plugin
-    /// failure is logged and the pipeline continues (the base record already
-    /// landed, so the item is at worst un-enriched, never lost).
-    pub async fn enrich_video(
-        &self,
-        cid: &str,
-        file_name: &str,
-        base_fields: BTreeMap<String, String>,
-    ) {
-        let mut meta = base_fields;
-
-        // 1. filename-parser → originalTitle / movieYear / videoType / season / episode.
-        if let Some(fp_url) = self.cfg.filename_parser_url.clone() {
-            match self.process(&fp_url, "filename-parser", cid, file_name, &meta).await {
-                Ok(()) => {
-                    // Poll for the merged fields so tmdb gets the clean title.
-                    if let Some(merged) = self
-                        .poll_until_field(cid, "originalTitle", FILENAME_PARSE_POLL_TIMEOUT)
-                        .await
-                    {
-                        meta = merged;
-                    } else {
-                        debug!(
-                            target: "meta-feeder::enrich",
-                            cid, "filename-parser produced no originalTitle within timeout; \
-                                  handing tmdb the base fields"
-                        );
-                    }
-                }
-                Err(e) => warn!(
-                    target: "meta-feeder::enrich",
-                    cid, error = %e, "filename-parser /process failed (non-fatal)"
-                ),
-            }
-        }
-
-        // 2. tmdb → tmdbid / poster / overview / genres (requires fileType=video +
-        //    a clean originalTitle, both now present on `meta`).
-        if let Some(tmdb_url) = self.cfg.tmdb_url.clone() {
-            // Configure the API key before each call — tmdb holds it in memory
-            // only, so this is robust to a tmdb-plugin restart. Cheap (local).
-            if let Err(e) = self.configure_tmdb(&tmdb_url).await {
-                warn!(
-                    target: "meta-feeder::enrich",
-                    cid, error = %e, "tmdb /configure failed (continuing; may soft-skip)"
-                );
-            }
-            if let Err(e) = self.process(&tmdb_url, "tmdb", cid, file_name, &meta).await {
-                warn!(
-                    target: "meta-feeder::enrich",
-                    cid, error = %e, "tmdb /process failed (non-fatal)"
-                );
-            }
-        }
-
-        // 3. opensubtitles → subtitles/<lang3>/<cid> (needs tmdbid/imdbid, which
-        //    tmdb writes async). Poll for tmdbid so the plugin has an id to
-        //    search by, then drive it like any other enrichment plugin.
-        if self.cfg.opensubtitles_url.is_some() {
-            let merged = self
-                .poll_until_field(cid, "tmdbid", TMDB_POLL_TIMEOUT)
-                .await
-                .unwrap_or(meta);
-            self.enrich_subtitles(cid, file_name, &merged).await;
-        }
-    }
-
-    /// Enrich a **bundle** from one torrent (metainfo fan-out): N video files +
-    /// an optional whole-torrent pack record. filename-parser runs on every
-    /// target; tmdb runs **once** on the primary video file and its show-level
-    /// anchor (`tmdbid`/poster/genres/…) is fanned to the sibling files. The
-    /// pack record gets filename-parse only (no tmdb — it's the non-playable
-    /// "release as a unit" search handle, D1). Best-effort throughout.
-    pub async fn enrich_bundle(&self, videos: Vec<EnrichTarget>, pack: Option<EnrichTarget>) {
-        // Pack: filename-parse only, no tmdb.
-        if let Some(p) = pack {
-            self.fire_filename(&p).await;
-        }
-
-        if videos.is_empty() {
-            return;
-        }
-        if videos.len() == 1 {
-            // Degenerate single-video bundle → the plain pipeline (filename → tmdb).
-            let v = &videos[0];
-            self.enrich_video(&v.cid, &v.file_name, v.base.clone()).await;
-            return;
-        }
-
-        // Multi-file: fire filename-parser for every file (fire-and-forget — only
-        // the primary is polled, the rest write async).
-        for v in &videos {
-            self.fire_filename(v).await;
-        }
-
-        // Primary = largest by `sizeByte` (ties → lowest index, i.e. first seen).
-        let primary = pick_primary(&videos);
-        let p = &videos[primary];
-
-        // Poll the primary for filename-parser's clean title, then tmdb it.
-        let merged = self
-            .poll_until_field(&p.cid, "originalTitle", FILENAME_PARSE_POLL_TIMEOUT)
-            .await
-            .unwrap_or_else(|| p.base.clone());
-
-        let Some(tmdb_url) = self.cfg.tmdb_url.clone() else {
-            return; // no tmdb wired → filename-only enrichment, done.
-        };
-        if let Err(e) = self.configure_tmdb(&tmdb_url).await {
-            warn!(target: "meta-feeder::enrich", cid = %p.cid, error = %e, "tmdb /configure failed");
-        }
-        if let Err(e) = self.process(&tmdb_url, "tmdb", &p.cid, &p.file_name, &merged).await {
-            warn!(target: "meta-feeder::enrich", cid = %p.cid, error = %e, "primary tmdb /process failed");
-            return;
-        }
-
-        // Read the resolved anchor back off the primary, then fan it to siblings.
-        let Some(anchor_rec) = self.poll_until_field(&p.cid, "tmdbid", TMDB_POLL_TIMEOUT).await
-        else {
-            debug!(target: "meta-feeder::enrich", cid = %p.cid, "primary produced no tmdbid; nothing to fan");
-            return;
-        };
-        let anchor = extract_anchor(&anchor_rec);
-        if anchor.is_empty() {
-            return;
-        }
-        for (i, v) in videos.iter().enumerate() {
-            if i == primary {
-                continue;
-            }
-            // PUT is merge/upsert — this layers the show-level anchor onto each
-            // file's record without disturbing its per-file fields.
-            if let Err(e) =
-                meta_core::put_record(&self.http, &self.cfg.meta_core_url, &v.cid, &anchor).await
-            {
-                warn!(target: "meta-feeder::enrich", cid = %v.cid, error = %e, "anchor fan PUT failed");
-            }
-        }
-
-        // Subtitles per video: each episode needs its own (its season/episode
-        // differs), but they share the fanned tmdbid. Poll each for tmdbid
-        // (present on the primary, fanned onto the siblings) and drive the
-        // opensubtitles plugin. Best-effort + quota-guarded inside the plugin.
-        if self.cfg.opensubtitles_url.is_some() {
-            for v in &videos {
-                let merged = self
-                    .poll_until_field(&v.cid, "tmdbid", TMDB_POLL_TIMEOUT)
-                    .await
-                    .unwrap_or_else(|| v.base.clone());
-                self.enrich_subtitles(&v.cid, &v.file_name, &merged).await;
-            }
-        }
-    }
-
-    /// Fire filename-parser `/process` for one target (fire-and-forget; the
-    /// plugin merges its fields to meta-core async).
-    async fn fire_filename(&self, t: &EnrichTarget) {
-        if let Some(fp_url) = self.cfg.filename_parser_url.clone() {
-            if let Err(e) = self
-                .process(&fp_url, "filename-parser", &t.cid, &t.file_name, &t.base)
-                .await
-            {
-                warn!(target: "meta-feeder::enrich", cid = %t.cid, error = %e, "filename-parser /process failed");
-            }
-        }
-    }
-
-    /// POST `/process` to a plugin. Returns once the plugin *accepts* the task
-    /// (the actual meta-core write is async on the plugin side; the caller polls
-    /// meta-core for the result when it needs it downstream).
-    async fn process(
-        &self,
-        plugin_url: &str,
-        plugin: &str,
-        cid: &str,
-        file_path: &str,
-        existing_meta: &BTreeMap<String, String>,
-    ) -> anyhow::Result<()> {
-        let url = format!("{}/process", plugin_url.trim_end_matches('/'));
-        let body = ProcessRequest {
-            task_id: format!("{plugin}-{cid}-{}", now_nanos()),
-            cid,
-            file_path,
-            callback_url: &self.cfg.callback_url,
-            meta_core_url: &self.cfg.meta_core_url,
-            existing_meta,
-        };
-        let resp = self.http.post(&url).json(&body).send().await?;
-        if !resp.status().is_success() {
-            let status = resp.status();
-            let text = resp.text().await.unwrap_or_default();
-            anyhow::bail!("{plugin} POST {url} returned {status}: {text}");
-        }
-        Ok(())
-    }
-
-    /// POST `/configure` to the tmdb plugin with the API key + language.
-    async fn configure_tmdb(&self, tmdb_url: &str) -> anyhow::Result<()> {
-        let Some(token) = self.cfg.tmdb_token.clone() else {
-            anyhow::bail!("no TMDB token configured");
-        };
-        let url = format!("{}/configure", tmdb_url.trim_end_matches('/'));
-        let body = serde_json::json!({
-            "config": { "apiKey": token, "language": self.cfg.tmdb_language }
-        });
-        let resp = self.http.post(&url).json(&body).send().await?;
-        if !resp.status().is_success() {
-            let status = resp.status();
-            let text = resp.text().await.unwrap_or_default();
-            anyhow::bail!("tmdb /configure returned {status}: {text}");
-        }
-        Ok(())
-    }
-
-    /// POST `/configure` to the opensubtitles plugin with the API key, account
-    /// credentials, and wanted languages.
-    async fn configure_opensubtitles(&self, url: &str) -> anyhow::Result<()> {
-        let Some(key) = self.cfg.opensubtitles_api_key.clone() else {
-            anyhow::bail!("no OpenSubtitles API key configured");
-        };
-        let cfg_url = format!("{}/configure", url.trim_end_matches('/'));
-        let body = serde_json::json!({
-            "config": {
-                "apiKey": key,
-                "username": self.cfg.opensubtitles_username,
-                "password": self.cfg.opensubtitles_password,
-                "languages": self.cfg.opensubtitles_languages,
-            }
-        });
-        let resp = self.http.post(&cfg_url).json(&body).send().await?;
-        if !resp.status().is_success() {
-            let status = resp.status();
-            let text = resp.text().await.unwrap_or_default();
-            anyhow::bail!("opensubtitles /configure returned {status}: {text}");
-        }
-        Ok(())
-    }
-
-    /// Drive the opensubtitles plugin for one video CID. Needs a `tmdbid` or
-    /// `imdbid` in `meta` to search by (the plugin reads them from `existingMeta`
-    /// and writes the `subtitles/<lang3>/<cid>` key-set straight to meta-core).
-    /// Best-effort: any failure is logged and swallowed.
-    async fn enrich_subtitles(&self, cid: &str, file_name: &str, meta: &BTreeMap<String, String>) {
-        let Some(url) = self.cfg.opensubtitles_url.clone() else {
-            return;
-        };
-        if !meta.contains_key("tmdbid") && !meta.contains_key("imdbid") {
-            debug!(
-                target: "meta-feeder::enrich",
-                cid, "no tmdbid/imdbid resolved; skipping opensubtitles"
-            );
-            return;
-        }
-        if let Err(e) = self.configure_opensubtitles(&url).await {
-            warn!(
-                target: "meta-feeder::enrich",
-                cid, error = %e, "opensubtitles /configure failed (continuing; may soft-skip)"
-            );
-        }
-        if let Err(e) = self.process(&url, "opensubtitles", cid, file_name, meta).await {
-            warn!(
-                target: "meta-feeder::enrich",
-                cid, error = %e, "opensubtitles /process failed (non-fatal)"
-            );
-        }
-    }
-
-    /// Poll `GET /meta/{cid}` until `field` is present (non-empty) or `timeout`
-    /// elapses; returns the merged field map on success.
-    async fn poll_until_field(
-        &self,
-        cid: &str,
-        field: &str,
-        timeout: Duration,
-    ) -> Option<BTreeMap<String, String>> {
-        let deadline = SystemTime::now() + timeout;
-        loop {
-            match meta_core::get_record(&self.http, &self.cfg.meta_core_url, cid).await {
-                Ok(Some(fields)) if fields.get(field).is_some_and(|v| !v.is_empty()) => {
-                    return Some(fields);
-                }
-                Ok(_) => {}
-                Err(e) => debug!(
-                    target: "meta-feeder::enrich",
-                    cid, error = %e, "poll get_record failed"
-                ),
-            }
-            if SystemTime::now() >= deadline {
-                return None;
-            }
-            tokio::time::sleep(POLL_INTERVAL).await;
-        }
-    }
-}
-
-fn now_nanos() -> u128 {
-    SystemTime::now()
-        .duration_since(UNIX_EPOCH)
-        .map(|d| d.as_nanos())
-        .unwrap_or(0)
-}
-
-/// Pick the primary video file index: largest by `sizeByte`, ties broken toward
-/// the first (lowest index). Falls back to 0 for an all-unsized bundle.
-fn pick_primary(videos: &[EnrichTarget]) -> usize {
-    videos
-        .iter()
-        .enumerate()
-        .max_by_key(|(i, v)| {
-            let size = v.base.get("sizeByte").and_then(|s| s.parse::<u64>().ok()).unwrap_or(0);
-            // larger size wins; on a tie the smaller index wins (Reverse).
-            (size, std::cmp::Reverse(*i))
-        })
-        .map(|(i, _)| i)
-        .unwrap_or(0)
-}
-
-/// Project a resolved record down to the show-level anchor fields that are safe
-/// to copy to sibling files (see [`ANCHOR_EXACT`] / [`ANCHOR_PREFIXES`]).
-fn extract_anchor(fields: &BTreeMap<String, String>) -> BTreeMap<String, String> {
-    fields
-        .iter()
-        .filter(|(k, _)| {
-            ANCHOR_EXACT.contains(&k.as_str())
-                || ANCHOR_PREFIXES.iter().any(|p| k.starts_with(p))
-        })
-        .map(|(k, v)| (k.clone(), v.clone()))
-        .collect()
-}
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-
-    #[test]
-    fn from_env_requires_meta_core_url() {
-        // Not set in the test env → None (legacy gateway-stores path).
-        // (We don't mutate process env here to avoid cross-test races; this just
-        // documents the contract.)
-        assert!(EnrichmentConfig::from_env().is_none() || EnrichmentConfig::from_env().is_some());
-    }
-
-    fn tgt(cid: &str, size: u64) -> EnrichTarget {
-        EnrichTarget {
-            cid: cid.into(),
-            file_name: cid.into(),
-            base: BTreeMap::from_iter([("sizeByte".to_string(), size.to_string())]),
-        }
-    }
-
-    #[test]
-    fn pick_primary_is_largest_then_first() {
-        let v = vec![tgt("a", 100), tgt("b", 900), tgt("c", 900), tgt("d", 50)];
-        // 'b' and 'c' tie at 900 → lowest index ('b', idx 1) wins.
-        assert_eq!(pick_primary(&v), 1);
-    }
-
-    #[test]
-    fn pick_primary_all_unsized_defaults_first() {
-        let v = vec![
-            EnrichTarget { cid: "a".into(), file_name: "a".into(), base: BTreeMap::new() },
-            EnrichTarget { cid: "b".into(), file_name: "b".into(), base: BTreeMap::new() },
-        ];
-        assert_eq!(pick_primary(&v), 0);
-    }
-
-    #[test]
-    fn extract_anchor_keeps_show_level_drops_per_file() {
-        let rec = BTreeMap::from_iter([
-            ("tmdbid".to_string(), "10378".to_string()),
-            ("imdbid".to_string(), "tt1254207".to_string()),
-            ("poster".to_string(), "bafkPOSTER".to_string()),
-            ("genres".to_string(), "Animation|Comedy".to_string()),
-            ("plot/eng".to_string(), "a plot".to_string()),
-            ("tags".to_string(), "tmdb-verified".to_string()),
-            // per-file — must NOT be fanned:
-            ("originalTitle".to_string(), "Big Buck Bunny".to_string()),
-            ("season".to_string(), "1".to_string()),
-            ("episode".to_string(), "3".to_string()),
-            ("fileName".to_string(), "ep3.mkv".to_string()),
-        ]);
-        let a = extract_anchor(&rec);
-        assert!(a.contains_key("tmdbid") && a.contains_key("poster") && a.contains_key("plot/eng"));
-        assert!(a.contains_key("genres") && a.contains_key("tags") && a.contains_key("imdbid"));
-        assert!(!a.contains_key("originalTitle"), "originalTitle is per-file");
-        assert!(!a.contains_key("season") && !a.contains_key("episode"));
-        assert!(!a.contains_key("fileName"));
-    }
-}

crates/meta-feeder-sdk/src/hash.rs

@@ -1,889 +0,0 @@
-//! Content-addressed identifiers used by feeder plugins' `compute_outcomes`
-//! to hash upstream-fetched bytes.
-//!
-//! Two function families live here:
-//!
-//! - [`compute_midhash256`] / [`compute_midhash256_from_sample`] —
-//!   the size-prefix-plus-middle-1MB-sample identification primitive shared
-//!   with `meta-sort` / `meta-share` / meta-core. Custom multicodec `0x1000`.
-//!   Fast but not IPFS-interop. The `_from_sample` entrypoint takes the
-//!   middle slice directly so plugins that can only fetch a partial range
-//!   (BitTorrent BEP 9, sliced HTTP) don't have to buffer the whole file
-//!   just to throw most of it away. **Rust-only by construction.**
-//! - [`compute_ipfs_cid`] is the standard IPFS CIDv1 produced over the full
-//!   bytes — raw codec `0x55` for a single-chunk file, UnixFS dag-pb codec
-//!   `0x70` rooted on chunked raw-codec leaves for larger files. Output matches
-//!   `ipfs add --cid-version=1 --raw-leaves=true --chunker=size-262144
-//!   --hash=sha2-256 <file>`.
-//!
-//! Cross-implementation parity for `compute_midhash256` with the TypeScript
-//! `FastHash.ts` is load-bearing — keep the fixture tests pinned, a regression
-//! silently breaks dedupe across the platform.
-
-use bytes::Bytes;
-use sha2::{Digest, Sha256};
-
-/// Sample window in bytes (1 MiB). Files this size or smaller are hashed
-/// in full; larger files are hashed over the middle 1 MiB slice.
-const SAMPLE_SIZE: usize = 1024 * 1024;
-
-/// Per-block IPFS chunk size (256 KiB), matches kubo's
-/// `--chunker=size-262144` default.
-pub const IPFS_CHUNK_SIZE: usize = 256 * 1024;
-
-/// dag-pb fanout — children per internal node. Matches kubo's balanced
-/// builder default.
-pub const IPFS_FANOUT: usize = 174;
-
-/// Output of [`compute_ipfs_blocks`]. Carries the root CID **plus every
-/// intermediate block** (leaf chunks AND internal dag-pb nodes) keyed by
-/// their own CID. Pass `.blocks` to the gateway's blockstore so peers
-/// can fetch the file by CID via bitswap.
-#[derive(Debug, Clone)]
-pub struct IpfsBlocks {
-    /// Canonical "bafy…" / "bafk…" root cid string. Identical to what
-    /// [`compute_ipfs_cid`] returns for the same input.
-    pub root: String,
-    /// Every block — root included — keyed by its own cid. For a
-    /// single-leaf file, this is a one-entry vec `[(root, payload)]`
-    /// (raw codec, payload is the file bytes). For multi-leaf files,
-    /// it is `leaves ++ internal_nodes` in build order; the **last**
-    /// entry is the root.
-    pub blocks: Vec<(String, Bytes)>,
-}
-
-/// Compute a midhash256 CID matching the TypeScript implementation in
-/// `meta-hash/src/lib/file-id/FastHash.ts`:
-///
-/// 1. Build the hash input as `[size:u64-be][middle 1 MiB sample]` (or the
-///    whole file if it fits in 1 MiB).
-/// 2. SHA-256 it.
-/// 3. Wrap in CIDv1 with custom multicodec `0x1000` for both the codec field
-///    and the multihash hash-code field, length 32, then multibase-base32-lower
-///    with the `b` prefix.
-///
-/// **Endianness for the size prefix is big-endian** to match TS's
-/// `writeBigUInt64BE`. Cross-impl mismatch here would silently mangle every
-/// import — keep the test pinned.
-pub fn compute_midhash256(bytes: &[u8]) -> String {
-    let size = bytes.len();
-    let sample: &[u8] = if size <= SAMPLE_SIZE {
-        bytes
-    } else {
-        let start = (size - SAMPLE_SIZE) / 2;
-        &bytes[start..start + SAMPLE_SIZE]
-    };
-    compute_midhash256_from_sample(size as u64, sample)
-}
-
-/// Same midhash256 primitive as [`compute_midhash256`], but takes the
-/// sample bytes directly rather than slicing them out of a full file.
-/// For upstreams that can only deliver a middle range (BitTorrent BEP 9,
-/// sliced HTTP), this avoids forcing the caller to buffer the whole file
-/// in memory just to throw most of it away.
-///
-/// `total_size` is the **full file size in bytes** — used as the
-/// size prefix, NOT `middle_sample.len()`.
-pub fn compute_midhash256_from_sample(total_size: u64, middle_sample: &[u8]) -> String {
-    let size_be = total_size.to_be_bytes();
-    let mut hasher = Sha256::new();
-    hasher.update(size_be);
-    hasher.update(middle_sample);
-    let digest: [u8; 32] = hasher.finalize().into();
-
-    // Multicodec 0x1000 encoded as unsigned varint = [0x80, 0x20].
-    // CIDv1 layout: [version=0x01][codec varint][multihash code varint][len][digest]
-    let mut wire = Vec::with_capacity(38);
-    wire.push(0x01);
-    wire.extend_from_slice(&[0x80, 0x20]);
-    wire.extend_from_slice(&[0x80, 0x20]);
-    wire.push(0x20);
-    wire.extend_from_slice(&digest);
-
-    format!("b{}", base32_lower_no_padding(&wire))
-}
-
-/// Wrap a 20-byte BitTorrent v1 info-hash as a multibase CIDv1.
-///
-/// BT v1 info-hashes ARE SHA-1 digests of the bencoded info dict — so
-/// the natural CID form is `codec=raw (0x55)` + `multihash=sha1 (0x11),
-/// len=20, digest=<infohash bytes>`.
-pub fn compute_bt_info_cid(infohash_20: &[u8; 20]) -> String {
-    // CIDv1 layout: [version=0x01][codec varint][multihash code varint][len][digest]
-    let mut wire = Vec::with_capacity(4 + 20);
-    wire.push(0x01); // CIDv1 version
-    wire.push(0x55); // codec: raw
-    wire.push(0x11); // multihash: sha1
-    wire.push(0x14); // multihash length: 20
-    wire.extend_from_slice(infohash_20);
-    format!("b{}", base32_lower_no_padding(&wire))
-}
-
-/// Custom multicodec for a single file inside a **BitTorrent v1** torrent
-/// (`btih-v1-file`). MetaMesh-private, adjacent to midhash256's `0x1000`.
-pub const BTIH_V1_FILE_CODEC: u64 = 0x1001;
-
-/// Custom multicodec for a single file inside a **BitTorrent v2** torrent
-/// (`btih-v2-file`).
-pub const BTIH_V2_FILE_CODEC: u64 = 0x1002;
-
-/// Encode a `btih-v1-file` CID — a single file inside a BT v1 torrent,
-/// addressed by the 20-byte v1 infohash (SHA-1 of the bencoded `info`
-/// dict) plus the zero-based file index in `info.files`. Single-file
-/// torrents use index `0`.
-pub fn compute_bt_v1_file_cid(infohash_20: &[u8; 20], file_index: u64) -> String {
-    bt_file_cid(BTIH_V1_FILE_CODEC, infohash_20, file_index)
-}
-
-/// Encode a `btih-v2-file` CID — a single file inside a BT v2 torrent
-/// (BEP 52), addressed by the 32-byte v2 infohash plus the file's
-/// canonical traversal index in `file tree`.
-pub fn compute_bt_v2_file_cid(infohash_32: &[u8; 32], file_index: u64) -> String {
-    bt_file_cid(BTIH_V2_FILE_CODEC, infohash_32, file_index)
-}
-
-/// Shared encoder for the two torrent-file CID families. `infohash` is 20
-/// bytes for v1 / 32 for v2; `codec` selects the family.
-fn bt_file_cid(codec: u64, infohash: &[u8], file_index: u64) -> String {
-    // digest = infohash ‖ varint(file_index)
-    let mut digest = Vec::with_capacity(infohash.len() + 2);
-    digest.extend_from_slice(infohash);
-    write_pb_varint(file_index, &mut digest);
-
-    // CIDv1: [version][codec varint][multihash code varint][len varint][digest]
-    let mut wire = Vec::with_capacity(1 + 4 + digest.len());
-    wire.push(0x01);
-    write_pb_varint(codec, &mut wire);
-    write_pb_varint(codec, &mut wire);
-    write_pb_varint(digest.len() as u64, &mut wire);
-    wire.extend_from_slice(&digest);
-    format!("b{}", base32_lower_no_padding(&wire))
-}
-
-/// Custom multicodec for a self-describing **Newznab release** locator
-/// (`nzb-release`). MetaMesh-private, adjacent to the torrent-file codecs in the
-/// `0x10xx` range. Unlike a content hash, the cid *embeds* the indexer host +
-/// release id in an identity multihash, so the credentialed meta-share peer
-/// decodes `{host,id}` straight from the cid and grabs the `.nzb` via `t=get`
-/// only at playback — no side-table, no `.nzb` download at search time. (The
-/// `0x1004` `sha256(host‖id)` opaque hash and the `0x1003` `nzb-posting`
-/// variant were both removed; only this self-describing form is emitted.)
-pub const NZB_RELEASE_CODEC: u64 = 0x1005;
-
-/// Digest byte ceiling for an `nzb-release` cid, matching meta-share's
-/// `MAX_MULTIHASH_SIZE` — its `CidGeneric<64>` rejects a longer multihash.
-const NZB_RELEASE_MAX_DIGEST: usize = 64;
-
-/// Encode a self-describing `nzb-release` CID from a Newznab indexer's host +
-/// bare release id (the hex `<guid>` id). The multihash is *identity* (`0x00`)
-/// and its digest embeds the locator as `varint(host_len) ‖ host ‖ id_bytes`,
-/// where `id_bytes` is the hex-decoded id. meta-share's `decode_nzb_release_cid`
-/// is the exact inverse. Host-namespaced: the same release on two indexers
-/// yields different cids.
-///
-/// Returns `None` when the id isn't hex or the digest would overflow the
-/// multihash budget (an oversized host) — the caller drops that row.
-pub fn compute_nzb_release_cid(host: &str, release_id: &str) -> Option<String> {
-    let id_bytes = decode_hex_id(release_id)?;
-    let host_b = host.as_bytes();
-
-    // identity-multihash digest = varint(host_len) ‖ host ‖ id_bytes
-    let mut digest = Vec::with_capacity(2 + host_b.len() + id_bytes.len());
-    write_pb_varint(host_b.len() as u64, &mut digest);
-    digest.extend_from_slice(host_b);
-    digest.extend_from_slice(&id_bytes);
-    if digest.len() > NZB_RELEASE_MAX_DIGEST {
-        return None;
-    }
-
-    // CIDv1: [version=0x01][codec varint][mh code=0x00 identity][len varint][digest]
-    let mut wire = Vec::with_capacity(1 + 3 + 2 + digest.len());
-    wire.push(0x01);
-    write_pb_varint(NZB_RELEASE_CODEC, &mut wire);
-    wire.push(0x00); // multihash code: identity
-    write_pb_varint(digest.len() as u64, &mut wire);
-    wire.extend_from_slice(&digest);
-    Some(format!("b{}", base32_lower_no_padding(&wire)))
-}
-
-/// Hex-decode a Newznab release id. The guid parser only keeps hex runs, so the
-/// id is all-hex by construction; odd-length ids are left-padded with a `0`
-/// nibble. Returns `None` on any non-hex byte (defensive).
-fn decode_hex_id(id: &str) -> Option<Vec<u8>> {
-    let padded;
-    let s: &str = if id.len() % 2 == 1 {
-        padded = format!("0{id}");
-        &padded
-    } else {
-        id
-    };
-    let bytes = s.as_bytes();
-    let mut out = Vec::with_capacity(bytes.len() / 2);
-    for pair in bytes.chunks(2) {
-        let hi = (pair[0] as char).to_digit(16)?;
-        let lo = (pair[1] as char).to_digit(16)?;
-        out.push(((hi << 4) | lo) as u8);
-    }
-    Some(out)
-}
-
-/// Custom multicodec for a **card locator** — the identity of a *work* (a
-/// series, a film) as published by a metadata bridge, rather than of any
-/// bytes. MetaMesh-private, the next free slot after `0x1006` url. (`0x1004`
-/// nzb-release-hash stays **retired — do not reuse**. `0x1003` was retired too,
-/// but has since been **revived** for the self-scanned Usenet posting identity —
-/// see [`NZB_POSTING_CODEC`] and `METADATA_KEYS.md` §2.)
-///
-/// This is the first CID family in the platform where **no bytes exist
-/// anywhere, ever** — `0x1005`/`0x1006` are locators too, but both eventually
-/// resolve to bytes (NNTP, HTTP), and `0x1000` midhash256 is a digest over
-/// real file bytes. A card is pure identity: the record *is* the payload, and
-/// it lives in meta-core. Consequences, all already handled:
-///
-/// * Bitswap seeding is gated on `HashKind::Sha2_256` (gateway invariant 6),
-///   so a card is never offered to the swarm — nothing to seed.
-/// * meta-share's byte path 404s for it, which is correct.
-/// * It ranks `RANK_LOCATOR` (5), so it can never outrank a real digest for
-///   `canonical_cid` — which is what makes the future "one card, two CIDs"
-///   merge (a TMDB and a MyAnimeList locator on one meta-core record) safe.
-pub const CARD_LOCATOR_CODEC: u64 = 0x1007;
-
-/// Digest byte ceiling for a card locator, matching meta-share's
-/// `MAX_MULTIHASH_SIZE` — its `CidGeneric<64>` rejects a longer multihash.
-const CARD_LOCATOR_MAX_DIGEST: usize = 64;
-
-/// Encode a self-describing **card locator** CID from a metadata source and
-/// that source's own id for the work (`("tmdb", "tv:95479")`,
-/// `("mal", "52991")`). The multihash is *identity* (`0x00`) and its digest
-/// embeds the locator as `varint(source_len) ‖ source ‖ id`, both UTF-8 —
-/// exactly the shape [`compute_nzb_release_cid`] uses, minus the hex decode
-/// (a card id is arbitrary text, not a hex release id).
-///
-/// **The CID is a pure function of `(source, id)`**, which is the whole point:
-/// any peer that knows a tmdb id can derive the card's address offline and
-/// look it up locally, with no discovery round-trip. It also means the same
-/// card discovered by two different peers converges on one meta-core record
-/// for free, via the `cids/<bareCid>` reverse index.
-///
-/// Source-namespaced: the same work on TMDB and on MyAnimeList yields
-/// different cids. Merging those two into one record is a meta-core alias
-/// operation driven by an explicit cross-source id mapping — never inferred
-/// here (see `meta-gateway/docs/others/card-tier-search.md` §9).
-///
-/// Returns `None` when `source` is empty or the digest would overflow the
-/// multihash budget — the caller drops that card.
-pub fn compute_card_cid(source: &str, id: &str) -> Option<String> {
-    if source.is_empty() || id.is_empty() {
-        return None;
-    }
-    let source_b = source.as_bytes();
-    let id_b = id.as_bytes();
-
-    // identity-multihash digest = varint(source_len) ‖ source ‖ id
-    let mut digest = Vec::with_capacity(2 + source_b.len() + id_b.len());
-    write_pb_varint(source_b.len() as u64, &mut digest);
-    digest.extend_from_slice(source_b);
-    digest.extend_from_slice(id_b);
-    if digest.len() > CARD_LOCATOR_MAX_DIGEST {
-        return None;
-    }
-
-    // CIDv1: [version=0x01][codec varint][mh code=0x00 identity][len varint][digest]
-    let mut wire = Vec::with_capacity(1 + 3 + 2 + digest.len());
-    wire.push(0x01);
-    write_pb_varint(CARD_LOCATOR_CODEC, &mut wire);
-    wire.push(0x00); // multihash code: identity
-    write_pb_varint(digest.len() as u64, &mut wire);
-    wire.extend_from_slice(&digest);
-    Some(format!("b{}", base32_lower_no_padding(&wire)))
-}
-
-/// Custom multicodec for a **Usenet posting** identity (`nzb-posting`),
-/// minted from the article Message-IDs of a release we scanned ourselves.
-///
-/// Unlike [`NZB_RELEASE_CODEC`] (`0x1005`), this is **not** a locator: it
-/// embeds no indexer host, so any peer with a plain NNTP provider can fetch the
-/// articles — no `IndexerCred`, no `t=get` grab. That portability is the whole
-/// point of running our own header scanner.
-///
-/// It is also **not** fetchable by cid: the digest is over the Message-ID set,
-/// not over any block's bytes, so bitswap can never satisfy a want for it (the
-/// receiver derives a block's cid by hashing the block — see
-/// `beetswap::incoming_stream`). The `.nzb` manifest travels separately, as an
-/// ordinary sha2-256 IPFS cid in the record's `manifest` field. See
-/// `meta-gateway/docs/others/self-hosted-usenet-indexer-study.md` §5.
-pub const NZB_POSTING_CODEC: u64 = 0x1003;
-
-/// Mint an `nzb-posting` cid (`0x1003`) from a release's article Message-IDs.
-///
-/// # The normalisation rule IS the dedup contract
-///
-/// Mirrored verbatim in meta-share (`nzb/manifest.rs`), which re-derives this
-/// digest from the fetched manifest and rejects a mismatch. **Any change here
-/// must land on both sides together** — a drift silently makes every posting
-/// unplayable. The rule:
-///
-/// 1. Trim each id, and strip a surrounding `<…>` pair if present (NZB
-///    `<segment>` bodies carry the bare form, but be defensive — NNTP `BODY`
-///    re-adds the brackets).
-/// 2. Drop empties.
-/// 3. Deduplicate, then **sort** lexicographically by bytes.
-/// 4. Join with `\n` and SHA-256 the result.
-///
-/// Case is **preserved**: a Message-ID's local part is case-sensitive per RFC
-/// 5322, so lowercasing could collide two genuinely distinct articles.
-///
-/// Sorting is what makes the cid survive segment/file reordering between two
-/// NZBs of the same posting — the reason we hash the *id set* and never the raw
-/// `.nzb` bytes (the retired `0x1004` did the latter and broke dedup; see
-/// meta-share `nzb/mod.rs`).
-///
-/// **v1 scope:** every segment counts, par2 volumes included. Two NZBs for one
-/// posting that disagree about whether to bundle par2 therefore mint different
-/// cids. Acceptable while we are the only producer of these manifests; if a
-/// second producer ever appears, the fix is to filter par2 subjects here *and*
-/// in meta-share's mirror — do not do it on one side only.
-pub fn compute_nzb_posting_cid<S: AsRef<str>>(message_ids: &[S]) -> String {
-    let mut ids: Vec<&str> = message_ids
-        .iter()
-        .map(|s| {
-            let t = s.as_ref().trim();
-            t.strip_prefix('<')
-                .and_then(|r| r.strip_suffix('>'))
-                .unwrap_or(t)
-        })
-        .filter(|s| !s.is_empty())
-        .collect();
-    ids.sort_unstable();
-    ids.dedup();
-
-    let mut hasher = Sha256::new();
-    for (i, id) in ids.iter().enumerate() {
-        if i > 0 {
-            hasher.update(b"\n");
-        }
-        hasher.update(id.as_bytes());
-    }
-    let digest: [u8; 32] = hasher.finalize().into();
-
-    // CIDv1: [version=0x01][codec varint][mh code=0x12 sha2-256][len=32][digest]
-    let mut wire = Vec::with_capacity(1 + 3 + 2 + digest.len());
-    wire.push(0x01);
-    write_pb_varint(NZB_POSTING_CODEC, &mut wire);
-    wire.push(0x12); // multihash code: sha2-256
-    write_pb_varint(digest.len() as u64, &mut wire);
-    wire.extend_from_slice(&digest);
-    format!("b{}", base32_lower_no_padding(&wire))
-}
-
-/// Compute a standard IPFS CIDv1 over `bytes`. Output matches kubo's
-/// `ipfs add --cid-version=1 --raw-leaves=true --chunker=size-262144
-/// --hash=sha2-256 <file>`.
-pub fn compute_ipfs_cid(bytes: &[u8]) -> String {
-    compute_ipfs_blocks(bytes).root
-}
-
-/// Same wire-format guarantee as [`compute_ipfs_cid`], but exposes every
-/// block produced along the way (leaf chunks + internal dag-pb nodes)
-/// so callers can populate a bitswap blockstore in one pass.
-pub fn compute_ipfs_blocks(bytes: &[u8]) -> IpfsBlocks {
-    // Leaves: chunk bytes into raw-codec blocks. Empty input still gets
-    // one (empty) leaf so the cid is well-defined.
-    let mut blocks: Vec<(String, Bytes)> = Vec::new();
-    let leaves: Vec<(Vec<u8>, u64)> = if bytes.is_empty() {
-        let cid_wire = ipfs_cid_wire(0x55, &sha2_256(b""));
-        blocks.push((cid_string(&cid_wire), Bytes::new()));
-        vec![(cid_wire, 0)]
-    } else {
-        bytes
-            .chunks(IPFS_CHUNK_SIZE)
-            .map(|chunk| {
-                let cid_wire = ipfs_cid_wire(0x55, &sha2_256(chunk));
-                blocks.push((cid_string(&cid_wire), Bytes::copy_from_slice(chunk)));
-                (cid_wire, chunk.len() as u64)
-            })
-            .collect()
-    };
-
-    // Single-leaf file: return the leaf cid as the file cid.
-    if leaves.len() == 1 {
-        let root = cid_string(&leaves[0].0);
-        return IpfsBlocks { root, blocks };
-    }
-
-    // Multi-leaf: build a balanced tree of UnixFS dag-pb internal nodes.
-    let mut level: Vec<(Vec<u8>, u64, u64)> = leaves
-        .into_iter()
-        .map(|(cid, size)| (cid, size, size))
-        .collect();
-
-    while level.len() > 1 {
-        let mut next: Vec<(Vec<u8>, u64, u64)> =
-            Vec::with_capacity(level.len().div_ceil(IPFS_FANOUT));
-        for batch in level.chunks(IPFS_FANOUT) {
-            let blocksizes: Vec<u64> = batch.iter().map(|(_, sz, _)| *sz).collect();
-            let filesize: u64 = blocksizes.iter().sum();
-            let unixfs_data = encode_unixfs_file(filesize, &blocksizes);
-            let node_bytes = encode_dagpb_node(batch, &unixfs_data);
-            let node_cid = ipfs_cid_wire(0x70, &sha2_256(&node_bytes));
-            blocks.push((cid_string(&node_cid), Bytes::from(node_bytes.clone())));
-            let own_tsize: u64 =
-                batch.iter().map(|(_, _, t)| *t).sum::<u64>() + node_bytes.len() as u64;
-            next.push((node_cid, filesize, own_tsize));
-        }
-        level = next;
-    }
-    let root = cid_string(&level[0].0);
-    IpfsBlocks { root, blocks }
-}
-
-fn cid_string(wire: &[u8]) -> String {
-    format!("b{}", base32_lower_no_padding(wire))
-}
-
-/// CIDv1 wire-form bytes: `[version=0x01][codec varint][multihash code
-/// varint=0x12 sha2-256][len=0x20][digest...]`.
-fn ipfs_cid_wire(codec: u8, digest: &[u8; 32]) -> Vec<u8> {
-    debug_assert!(
-        codec < 0x80,
-        "codec varint must fit in one byte for this helper"
-    );
-    let mut wire = Vec::with_capacity(36);
-    wire.push(0x01); // CIDv1
-    wire.push(codec);
-    wire.push(0x12); // multihash code: sha2-256
-    wire.push(0x20); // digest length: 32
-    wire.extend_from_slice(digest);
-    wire
-}
-
-fn sha2_256(bytes: &[u8]) -> [u8; 32] {
-    Sha256::digest(bytes).into()
-}
-
-/// LEB128 unsigned varint encoding into `out`.
-fn write_pb_varint(value: u64, out: &mut Vec<u8>) {
-    let mut v = value;
-    while v >= 0x80 {
-        out.push((v as u8) | 0x80);
-        v >>= 7;
-    }
-    out.push(v as u8);
-}
-
-fn write_pb_varint_field(field: u32, value: u64, out: &mut Vec<u8>) {
-    write_pb_varint((field as u64) << 3, out); // tag: (field << 3) | wire-type 0 (varint)
-    write_pb_varint(value, out);
-}
-
-fn write_pb_bytes_field(field: u32, value: &[u8], out: &mut Vec<u8>) {
-    write_pb_varint(((field as u64) << 3) | 2, out); // wire type 2 = length-delimited
-    write_pb_varint(value.len() as u64, out);
-    out.extend_from_slice(value);
-}
-
-/// UnixFS protobuf payload for a File node: `Type=File, filesize,
-/// blocksizes[]`.
-fn encode_unixfs_file(filesize: u64, blocksizes: &[u64]) -> Vec<u8> {
-    let mut out = Vec::new();
-    write_pb_varint_field(1, 2, &mut out); // Type = File
-    write_pb_varint_field(3, filesize, &mut out); // filesize
-    for &bs in blocksizes {
-        write_pb_varint_field(4, bs, &mut out); // blocksizes (repeated)
-    }
-    out
-}
-
-/// dag-pb PBNode for an internal UnixFS file node. Canonical wire order
-/// pinned by the dag-pb spec: `Links` (tag 2) first, `Data` (tag 1) second.
-fn encode_dagpb_node(children: &[(Vec<u8>, u64, u64)], data: &[u8]) -> Vec<u8> {
-    let mut out = Vec::new();
-    for (cid_wire, _filesize, tsize) in children {
-        let mut link_bytes = Vec::new();
-        write_pb_bytes_field(1, cid_wire, &mut link_bytes); // Hash
-        write_pb_varint_field(3, *tsize, &mut link_bytes); // Tsize
-        write_pb_bytes_field(2, &link_bytes, &mut out); // Links (tag 2) first
-    }
-    write_pb_bytes_field(1, data, &mut out); // Data (tag 1) last
-    out
-}
-
-/// RFC 4648 base32 with the lowercase alphabet and no padding. Used for the
-/// multibase `b` prefix.
-pub(crate) fn base32_lower_no_padding(input: &[u8]) -> String {
-    const ALPHABET: &[u8; 32] = b"abcdefghijklmnopqrstuvwxyz234567";
-    let mut out = String::with_capacity(input.len().div_ceil(5) * 8);
-    let mut buffer: u64 = 0;
-    let mut bits: u32 = 0;
-    for &b in input {
-        buffer = (buffer << 8) | b as u64;
-        bits += 8;
-        while bits >= 5 {
-            bits -= 5;
-            let idx = ((buffer >> bits) & 0x1F) as usize;
-            out.push(ALPHABET[idx] as char);
-        }
-    }
-    if bits > 0 {
-        let idx = ((buffer << (5 - bits)) & 0x1F) as usize;
-        out.push(ALPHABET[idx] as char);
-    }
-    out
-}
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-
-    #[test]
-    fn midhash256_matches_ts_fixtures() {
-        assert_eq!(
-            compute_midhash256(b"hello world"),
-            "bagacbabaec7v3fu2ygzh3e2sybg3fbzmisry2hbtpmck6vx3yftea6vzq35r4"
-        );
-        assert_eq!(
-            compute_midhash256(b""),
-            "bagacbabaecxvk4hvugaqw6xxrsxuxrykmyhq35i6ik5pshkn4wzdfdpa5a67y"
-        );
-        let mb_zeros = vec![0u8; 1024 * 1024];
-        assert_eq!(
-            compute_midhash256(&mb_zeros),
-            "bagacbabaeawlkt4hn34sxuieoosrnnag3g3wv7gc6alzwfms2nrph2hochrx4"
-        );
-    }
-
-    #[test]
-    fn midhash256_from_sample_matches_full_bytes() {
-        let small: &[u8] = b"hello world";
-        assert_eq!(
-            compute_midhash256_from_sample(small.len() as u64, small),
-            compute_midhash256(small),
-        );
-
-        assert_eq!(
-            compute_midhash256_from_sample(0, b""),
-            compute_midhash256(b""),
-        );
-
-        let one_mb = vec![0u8; SAMPLE_SIZE];
-        assert_eq!(
-            compute_midhash256_from_sample(one_mb.len() as u64, &one_mb),
-            compute_midhash256(&one_mb),
-        );
-
-        let two_mb = vec![0u8; 2 * SAMPLE_SIZE];
-        let start = (two_mb.len() - SAMPLE_SIZE) / 2;
-        let middle = &two_mb[start..start + SAMPLE_SIZE];
-        assert_eq!(
-            compute_midhash256_from_sample(two_mb.len() as u64, middle),
-            compute_midhash256(&two_mb),
-        );
-    }
-
-    #[test]
-    fn midhash256_from_sample_partial_sample_stable() {
-        let sample = vec![0u8; SAMPLE_SIZE];
-        let total_5gb: u64 = 5 * 1024 * 1024 * 1024;
-        let total_10gb: u64 = 10 * 1024 * 1024 * 1024;
-
-        let h1 = compute_midhash256_from_sample(total_5gb, &sample);
-        let h2 = compute_midhash256_from_sample(total_5gb, &sample);
-        assert_eq!(h1, h2, "same (size, sample) must hash identically");
-
-        let h3 = compute_midhash256_from_sample(total_10gb, &sample);
-        assert_ne!(
-            h1, h3,
-            "size prefix must contribute to cid — same sample, different size"
-        );
-    }
-
-    #[test]
-    fn bt_v1_file_cid_wire_layout() {
-        let infohash: [u8; 20] = [
-            0xf9, 0xc8, 0xa7, 0xb6, 0xe5, 0xd4, 0xc3, 0xb2, 0xa1, 0x90, 0x8f, 0x7e, 0x6d, 0x5c,
-            0x4b, 0x3a, 0x29, 0x18, 0x07, 0x06,
-        ];
-
-        let mut want = vec![0x01, 0x81, 0x20, 0x81, 0x20, 0x15];
-        want.extend_from_slice(&infohash);
-        want.push(0x04);
-        assert_eq!(
-            compute_bt_v1_file_cid(&infohash, 4),
-            format!("b{}", base32_lower_no_padding(&want))
-        );
-
-        let mut want0 = vec![0x01, 0x81, 0x20, 0x81, 0x20, 0x15];
-        want0.extend_from_slice(&infohash);
-        want0.push(0x00);
-        assert_eq!(
-            compute_bt_v1_file_cid(&infohash, 0),
-            format!("b{}", base32_lower_no_padding(&want0))
-        );
-
-        let mut want200 = vec![0x01, 0x81, 0x20, 0x81, 0x20, 0x16];
-        want200.extend_from_slice(&infohash);
-        want200.extend_from_slice(&[0xC8, 0x01]);
-        assert_eq!(
-            compute_bt_v1_file_cid(&infohash, 200),
-            format!("b{}", base32_lower_no_padding(&want200))
-        );
-    }
-
-    #[test]
-    fn bt_v2_file_cid_wire_layout() {
-        let infohash: [u8; 32] = [0xAB; 32];
-        let mut want = vec![0x01, 0x82, 0x20, 0x82, 0x20, 0x21];
-        want.extend_from_slice(&infohash);
-        want.push(0x00);
-        assert_eq!(
-            compute_bt_v2_file_cid(&infohash, 0),
-            format!("b{}", base32_lower_no_padding(&want))
-        );
-    }
-
-    #[test]
-    fn bt_file_cid_properties() {
-        let ih1 = [0x11u8; 20];
-        let ih2 = [0x22u8; 32];
-
-        assert_eq!(
-            compute_bt_v1_file_cid(&ih1, 3),
-            compute_bt_v1_file_cid(&ih1, 3)
-        );
-        assert_ne!(
-            compute_bt_v1_file_cid(&ih1, 0),
-            compute_bt_v1_file_cid(&ih1, 1)
-        );
-        assert!(compute_bt_v1_file_cid(&ih1, 0).starts_with('b'));
-        assert!(compute_bt_v2_file_cid(&ih2, 0).starts_with('b'));
-        assert_ne!(
-            compute_bt_v1_file_cid(&[0xAB; 20], 0),
-            compute_bt_v2_file_cid(&[0xAB; 32], 0)
-        );
-    }
-
-    #[test]
-    fn base32_known_vectors() {
-        assert_eq!(base32_lower_no_padding(b""), "");
-        assert_eq!(base32_lower_no_padding(b"f"), "my");
-        assert_eq!(base32_lower_no_padding(b"fo"), "mzxq");
-        assert_eq!(base32_lower_no_padding(b"foo"), "mzxw6");
-        assert_eq!(base32_lower_no_padding(b"foob"), "mzxw6yq");
-        assert_eq!(base32_lower_no_padding(b"foobar"), "mzxw6ytboi");
-    }
-
-    #[test]
-    fn ipfs_cid_single_leaf_matches_kubo() {
-        assert_eq!(
-            compute_ipfs_cid(b"hello world"),
-            "bafkreifzjut3te2nhyekklss27nh3k72ysco7y32koao5eei66wof36n5e"
-        );
-    }
-
-    #[test]
-    fn ipfs_cid_empty_matches_kubo() {
-        assert_eq!(
-            compute_ipfs_cid(b""),
-            "bafkreihdwdcefgh4dqkjv67uzcmw7ojee6xedzdetojuzjevtenxquvyku"
-        );
-    }
-
-    #[test]
-    fn ipfs_cid_multi_leaf_zeros_stable() {
-        let mb_zeros = vec![0u8; 1024 * 1024];
-        assert_eq!(
-            compute_ipfs_cid(&mb_zeros),
-            "bafybeiadh3bekpwtewjvauqeucf7yzqrb3ixsxzltnuwed4pxangtpou6m"
-        );
-    }
-
-    #[test]
-    fn ipfs_blocks_single_leaf() {
-        let payload = b"hello world";
-        let out = compute_ipfs_blocks(payload);
-        assert_eq!(out.root, compute_ipfs_cid(payload));
-        assert_eq!(out.blocks.len(), 1);
-        assert_eq!(out.blocks[0].0, out.root);
-        assert_eq!(out.blocks[0].1.as_ref(), payload);
-    }
-
-    #[test]
-    fn ipfs_blocks_multi_leaf() {
-        let mb_zeros = vec![0u8; 1024 * 1024];
-        let out = compute_ipfs_blocks(&mb_zeros);
-        assert_eq!(
-            out.root,
-            "bafybeiadh3bekpwtewjvauqeucf7yzqrb3ixsxzltnuwed4pxangtpou6m"
-        );
-        assert_eq!(out.blocks.len(), 5);
-        assert_eq!(out.blocks.last().unwrap().0, out.root);
-        for (_cid, block) in &out.blocks[..4] {
-            assert_eq!(block.len(), 256 * 1024);
-            assert!(block.iter().all(|&b| b == 0));
-        }
-    }
-
-    #[test]
-    fn ipfs_blocks_empty() {
-        let out = compute_ipfs_blocks(b"");
-        assert_eq!(
-            out.root,
-            "bafkreihdwdcefgh4dqkjv67uzcmw7ojee6xedzdetojuzjevtenxquvyku"
-        );
-        assert_eq!(out.blocks.len(), 1);
-        assert_eq!(out.blocks[0].1.len(), 0);
-        assert_eq!(out.blocks[0].0, out.root);
-    }
-
-    // ---- card locator (0x1007) --------------------------------------------
-
-    /// Inverse of [`base32_lower_no_padding`], test-only. Lets the card tests
-    /// assert the actual CIDv1 bytes (codec, multihash code, digest) rather
-    /// than just round-tripping the encoder against itself.
-    fn base32_lower_decode(s: &str) -> Vec<u8> {
-        const ALPHABET: &[u8] = b"abcdefghijklmnopqrstuvwxyz234567";
-        let mut out = Vec::new();
-        let (mut acc, mut bits) = (0u32, 0u32);
-        for c in s.bytes() {
-            let v = ALPHABET.iter().position(|&a| a == c).expect("base32 char") as u32;
-            acc = (acc << 5) | v;
-            bits += 5;
-            if bits >= 8 {
-                bits -= 8;
-                out.push((acc >> bits) as u8);
-                acc &= (1 << bits) - 1;
-            }
-        }
-        out
-    }
-
-    /// The property the whole card design leans on: the CID is a pure function
-    /// of `(source, id)`, so any peer can derive a card's address offline. If
-    /// this literal ever changes, every already-published card silently forks
-    /// into a second identity — treat a failure here as a wire break.
-    #[test]
-    fn card_cid_is_deterministic() {
-        let cid = compute_card_cid("tmdb", "tv:95479").expect("card cid");
-        assert_eq!(cid, compute_card_cid("tmdb", "tv:95479").unwrap());
-        assert_eq!(cid, "bagdsaaanar2g2zdcor3duojvgq3ts");
-    }
-
-    /// Source- and kind-namespaced: the same numeric id on a different bridge,
-    /// or the same id under a different media kind, must not collide.
-    #[test]
-    fn card_cid_is_source_and_kind_namespaced() {
-        let tmdb = compute_card_cid("tmdb", "tv:95479").unwrap();
-        let mal = compute_card_cid("mal", "95479").unwrap();
-        let movie = compute_card_cid("tmdb", "movie:95479").unwrap();
-        assert_ne!(tmdb, mal);
-        assert_ne!(tmdb, movie);
-    }
-
-    /// Pins the on-the-wire CIDv1 layout: v1, codec `0x1007` (varint
-    /// `[0x87,0x20]`), **identity** multihash `0x00` (this family has no
-    /// digest — the locator itself is the payload), then
-    /// `varint(source_len) ‖ source ‖ id`.
-    #[test]
-    fn card_cid_wire_shape() {
-        let cid = compute_card_cid("tmdb", "tv:95479").unwrap();
-        assert!(cid.starts_with('b'), "multibase base32-lower prefix");
-        let wire = base32_lower_decode(&cid[1..]);
-
-        assert_eq!(wire[0], 0x01, "CIDv1");
-        assert_eq!(&wire[1..3], &[0x87, 0x20], "codec varint for 0x1007");
-        assert_eq!(wire[3], 0x00, "identity multihash — no digest");
-
-        let digest_len = wire[4] as usize;
-        let digest = &wire[5..5 + digest_len];
-        assert_eq!(digest[0], 4, "varint(len(\"tmdb\"))");
-        assert_eq!(&digest[1..5], b"tmdb");
-        assert_eq!(&digest[5..], b"tv:95479");
-    }
-
-    /// Empty inputs are rejected rather than encoded as an ambiguous locator,
-    /// and an id past the 64-byte multihash budget (meta-share's
-    /// `CidGeneric<64>`) returns `None` so the caller drops the card instead of
-    /// emitting a cid that peer would refuse to parse.
-    #[test]
-    fn card_cid_rejects_empty_and_oversized() {
-        assert!(compute_card_cid("", "tv:1").is_none());
-        assert!(compute_card_cid("tmdb", "").is_none());
-        let huge = "x".repeat(CARD_LOCATOR_MAX_DIGEST);
-        assert!(compute_card_cid("tmdb", &huge).is_none());
-        // Exactly at the ceiling still encodes.
-        let fits = "x".repeat(CARD_LOCATOR_MAX_DIGEST - 5); // varint(4) + "tmdb"
-        assert!(compute_card_cid("tmdb", &fits).is_some());
-    }
-
-    // ---- nzb-posting (0x1003) — the dedup contract -----------------------
-    //
-    // These pin the normalisation rule documented on `compute_nzb_posting_cid`.
-    // meta-share re-derives the same digest from the fetched manifest and
-    // rejects a mismatch, so a change that breaks one of these silently makes
-    // every self-scanned posting unplayable. Mirror any edit on both sides.
-
-    /// Segment/file **order must not matter** — that is the whole reason we
-    /// hash a sorted id set instead of the raw `.nzb` bytes (the retired
-    /// `0x1004` did the latter and broke dedup across reposts).
-    #[test]
-    fn nzb_posting_cid_is_order_independent() {
-        let a = ["[email protected]", "[email protected]", "[email protected]"];
-        let b = ["[email protected]", "[email protected]", "[email protected]"];
-        let c = ["[email protected]", "[email protected]", "[email protected]"];
-        assert_eq!(compute_nzb_posting_cid(&a), compute_nzb_posting_cid(&b));
-        assert_eq!(compute_nzb_posting_cid(&a), compute_nzb_posting_cid(&c));
-    }
-
-    /// Angle brackets, surrounding whitespace, empty entries and duplicates are
-    /// all normalised away. NZB `<segment>` bodies carry the bare form but NNTP
-    /// `BODY` re-adds the brackets, so both spellings must converge.
-    #[test]
-    fn nzb_posting_cid_normalises_brackets_blanks_and_dupes() {
-        let bare = ["[email protected]", "[email protected]"];
-        let noisy = [" <[email protected]> ", "[email protected]", "", "  ", "[email protected]"];
-        assert_eq!(
-            compute_nzb_posting_cid(&bare),
-            compute_nzb_posting_cid(&noisy)
-        );
-    }
-
-    /// Case is preserved: a Message-ID's local part is case-sensitive per RFC
-    /// 5322, so lowercasing could collide two genuinely distinct articles.
-    #[test]
-    fn nzb_posting_cid_is_case_sensitive() {
-        assert_ne!(
-            compute_nzb_posting_cid(&["[email protected]"]),
-            compute_nzb_posting_cid(&["[email protected]"])
-        );
-    }
-
-    /// A different article set is a different posting.
-    #[test]
-    fn nzb_posting_cid_differs_on_different_ids() {
-        assert_ne!(
-            compute_nzb_posting_cid(&["[email protected]", "[email protected]"]),
-            compute_nzb_posting_cid(&["[email protected]", "[email protected]"])
-        );
-    }
-
-    /// Wire shape: CIDv1, codec `0x1003` (varint `[0x83, 0x20]`), multihash
-    /// sha2-256 (`0x12`), 32-byte digest. Pinned because meta-share matches on
-    /// the **codec** slot — a drift here makes every posting undecodable there.
-    #[test]
-    fn nzb_posting_cid_wire_shape() {
-        let cid = compute_nzb_posting_cid(&["[email protected]"]);
-        assert!(cid.starts_with('b'), "multibase base32-lower prefix");
-
-        // Round-trip the base32 body back to bytes to assert the header.
-        let decoded = base32_lower_decode(&cid[1..]);
-        assert_eq!(decoded[0], 0x01, "CIDv1");
-        assert_eq!(&decoded[1..3], &[0x83, 0x20], "varint(0x1003)");
-        assert_eq!(decoded[3], 0x12, "multihash sha2-256");
-        assert_eq!(decoded[4], 0x20, "digest length 32");
-        assert_eq!(decoded.len(), 5 + 32);
-
-        // The digest is sha256 over the newline-joined sorted set.
-        let expect: [u8; 32] = Sha256::digest(b"[email protected]").into();
-        assert_eq!(&decoded[5..], &expect);
-    }
-}

crates/meta-feeder-sdk/src/lang.rs

@@ -1,81 +0,0 @@
-//! Language-code normalization — ISO 639 alias folding, **not** filename
-//! parsing.
-//!
-//! This lives apart from [`crate::filename_meta`] on purpose: it doesn't read a
-//! release title, it canonicalizes a *language code* (an indexer's declared
-//! language, a query's `languages:` filter, a title-derived stamp) onto the
-//! single ISO 639-2/B alpha-3 vocabulary the rest of the stack compares against.
-//! It is the one piece of the old `filename_meta` module that has no equivalent
-//! in the `metamesh-plugin-filename-parser` plugin — because it isn't filename
-//! parsing — so it stays in Rust while the title-parsing algorithm itself is
-//! owned by the plugin. Callers in the **request hot path** (e.g. the torznab
-//! indexer-language fan-out skip) need it synchronously, with no network hop.
-
-/// Normalize a language code to the ISO 639-2/B alpha-3 vocabulary the stack
-/// emits (`fre`/`ger`/`chi`, **not** the 639-2/T `fra`/`deu`/`zho` nor the
-/// 639-1 two-letter forms), lowercased. This is the space meta-share's
-/// "languages I read" preference uses and that the title-derived language stamps
-/// write, so an operator who declares an indexer's language as `fra`/`de`/`zh`
-/// still lines up with a `fre`/`ger`/`chi` query filter. Unknown /
-/// already-canonical codes pass through lowercased+trimmed unchanged.
-pub fn normalize_lang_code(code: &str) -> String {
-    let c = code.trim().to_ascii_lowercase();
-    let mapped = match c.as_str() {
-        // 639-2/T and 639-1 → 639-2/B for the codes that actually differ.
-        "fra" | "fr" => "fre",
-        "deu" | "de" => "ger",
-        "zho" | "zh" => "chi",
-        "nld" | "nl" => "dut",
-        "ell" | "el" => "gre",
-        "ron" | "ro" => "rum",
-        "slk" | "sk" => "slo",
-        "ces" | "cs" => "cze",
-        // 639-1 → 639-2 for the rest of the language-tag output space (these
-        // share the same /B and /T alpha-3, so only the two-letter form maps).
-        "en" => "eng",
-        "ja" => "jpn",
-        "it" => "ita",
-        "es" => "spa",
-        "pt" => "por",
-        "ko" => "kor",
-        "ru" => "rus",
-        "pl" => "pol",
-        "hu" => "hun",
-        "uk" => "ukr",
-        "bg" => "bul",
-        "sv" => "swe",
-        "no" => "nor",
-        "da" => "dan",
-        "fi" => "fin",
-        "tr" => "tur",
-        "ar" => "ara",
-        "hi" => "hin",
-        "ta" => "tam",
-        "te" => "tel",
-        "ml" => "mal",
-        "vi" => "vie",
-        "th" => "tha",
-        _ => return c,
-    };
-    mapped.to_string()
-}
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-
-    #[test]
-    fn normalize_lang_maps_to_639_2b_vocabulary() {
-        // 639-2/T and 639-1 spellings collapse onto the /B forms the stack uses.
-        assert_eq!(normalize_lang_code("fra"), "fre");
-        assert_eq!(normalize_lang_code("FR"), "fre");
-        assert_eq!(normalize_lang_code("deu"), "ger");
-        assert_eq!(normalize_lang_code("zh"), "chi");
-        assert_eq!(normalize_lang_code("nld"), "dut");
-        // Already-canonical / unknown codes pass through trimmed + lowercased.
-        assert_eq!(normalize_lang_code("  ENG "), "eng");
-        assert_eq!(normalize_lang_code("fre"), "fre");
-        assert_eq!(normalize_lang_code("mul"), "mul");
-        assert_eq!(normalize_lang_code("xyz"), "xyz");
-    }
-}

crates/meta-feeder-sdk/src/lib.rs

@@ -1,49 +0,0 @@
-//! `meta-feeder-sdk` — the source-agnostic foundation shared by the gateway
-//! core and every feeder sidecar binary.
-//!
-//! A *feeder* implements [`FeederPlugin`] (find records, fetch bytes) for one
-//! or more upstreams, and [`serve_feeders`] exposes them over the feeder HTTP
-//! contract. The gateway core consumes that contract via its
-//! `RemoteFeederPlugin` and keeps the libp2p wire, the bitswap blockstore, the
-//! hashing-into-the-blockstore, and the meta-core store-back to itself.
-//!
-//! Deliberately libp2p-free and blockstore-free — see this crate's `Cargo.toml`.
-
-// `redb::Error` is a large enum; the cache wrappers (`cache.rs`) return it
-// directly rather than boxing on every embedded-DB call. Boxing each would be
-// pure noise for a single-process store, so silence the lint crate-wide.
-#![allow(clippy::result_large_err)]
-
-/// pcs-convention default meta-core root a self-publishing feeder targets when
-/// neither its dashboard config nor the `META_CORE_URL` env seed sets one. Keeps
-/// the modular storage seam out of compose: a standard install just works; a
-/// non-standard meta-core is set in the feeder's config UI. Matches the
-/// `metacore-app` service the AppStore feeder apps ship on the shared network.
-pub const DEFAULT_META_CORE_URL: &str = "http://metacore-app:9000";
-
-pub mod cache;
-pub mod common;
-pub mod config;
-pub mod enrich;
-pub mod filename_meta;
-pub mod hash;
-pub mod lang;
-pub mod meta_core;
-pub mod plugin;
-pub mod query;
-pub mod query_eval;
-pub mod serve;
-pub mod types;
-
-pub use config::{ConfigField, ConfigSchema, FieldKind};
-pub use enrich::{EnrichTarget, EnrichmentConfig, Enricher};
-pub use meta_core::FeederStore;
-pub use plugin::{
-    upstream_id_field, ConfigError, FeederPlugin, HashKind, HashOutcome, PluginRegistry,
-};
-pub use query::{GatewayQuery, GatewaySearchEvent, GatewayWireError, Negation, RangeFilter};
-pub use serve::{
-    configure_plugins, router, serve_feeders, ComputeRequest, ComputeResponse, HashKindDto,
-    HealthResponse, ManifestResponse, OutcomeDto, PluginManifest, QueryRequest, QueryResponse,
-};
-pub use types::{ByteStream, DiscoveryId, DiscoveryRecord, GatewayError, Hash, PluginHealth};

crates/meta-feeder-sdk/src/meta_core.rs

@@ -1,293 +0,0 @@
-//! Lightweight meta-core write/read client for feeders that **self-publish**.
-//!
-//! Post-"smart feeder / dumb gateway" split: a feeder given a `META_CORE_URL`
-//! writes its own base record into meta-core instead of returning it for the
-//! gateway dispatcher to store. This keeps the gateway a pure aggregator +
-//! request router; the feeder owns the full record lifecycle for its upstream.
-//!
-//! The write shape is **byte-for-byte the same** as the gateway's
-//! `meta_core::build_metadata_body` (`crates/meta-gateway/src/meta_core.rs`):
-//! `cid_*` fields collapse to the `cids/<cid> = "true"` key-set, and a
-//! `provenance` JSON blob is stamped. Keeping them identical means a record
-//! written by the feeder is indistinguishable from one the dispatcher wrote —
-//! the meta-share ingester, meta-watch, and ranking all behave the same.
-//!
-//! `PUT /api/metadata/{cid}` is merge/upsert (confirmed: meta-core
-//! `SetMetadataFlat` sets only the fields in the payload, never deletes), so the
-//! feeder's base write and the enrichment plugins' later `PATCH /meta/{cid}`
-//! merges accumulate on the same `/file/{cid}` hash.
-
-use std::collections::BTreeMap;
-use std::time::{SystemTime, UNIX_EPOCH};
-
-use serde::Deserialize;
-use tracing::warn;
-
-use crate::plugin::{HashKind, HashOutcome};
-use crate::types::{DiscoveryRecord, Hash};
-
-/// Provenance recorded on every gateway-acquired record. Mirrors the gateway
-/// dispatcher's shape so feeder-written and dispatcher-written records are
-/// interchangeable.
-#[derive(Clone, Debug)]
-pub struct Provenance {
-    pub upstream: String,
-    pub record_id: String,
-    /// Stable per-deployment id (the gateway peer this feeder backs, from
-    /// `META_GATEWAY_PEER_ID`, else the feeder hostname). Analytics group by it.
-    pub gateway_peer: String,
-    pub acquired_at: u64,
-}
-
-impl Provenance {
-    /// Build a provenance stamp for `upstream`/`record_id`, stamping "now".
-    pub fn now(upstream: &str, record_id: &str, gateway_peer: &str) -> Self {
-        Provenance {
-            upstream: upstream.to_string(),
-            record_id: record_id.to_string(),
-            gateway_peer: gateway_peer.to_string(),
-            acquired_at: SystemTime::now()
-                .duration_since(UNIX_EPOCH)
-                .map(|d| d.as_secs())
-                .unwrap_or(0),
-        }
-    }
-}
-
-/// JSON-stringified provenance value. Same key set + `"source":"gateway"` as the
-/// gateway dispatcher (`crates/meta-gateway/src/meta_core.rs::provenance_to_json`).
-pub fn provenance_to_json(p: &Provenance) -> String {
-    serde_json::to_string(&serde_json::json!({
-        "source":      "gateway",
-        "upstream":    p.upstream,
-        "recordId":    p.record_id,
-        "gatewayPeer": p.gateway_peer,
-        "acquiredAt":  p.acquired_at,
-    }))
-    .expect("static JSON shape — serialisation is infallible")
-}
-
-/// Build the flat meta-core PUT body from a [`DiscoveryRecord`] + provenance.
-///
-/// Identical transform to the gateway dispatcher: any `cid_*` field becomes a
-/// `cids/<value> = "true"` key-set member (the CID is self-describing — the
-/// algorithm is the multicodec; see METADATA_KEYS.md §2/§14.13), everything else
-/// passes through, and `provenance` is stamped last (overriding any plugin-set
-/// value).
-pub fn build_metadata_body(
-    record: &DiscoveryRecord,
-    provenance: &Provenance,
-) -> BTreeMap<String, String> {
-    let mut body: BTreeMap<String, String> = BTreeMap::new();
-    for (k, v) in &record.fields {
-        if k.starts_with("cid_") && !v.is_empty() {
-            body.insert(format!("cids/{v}"), "true".to_string());
-        } else {
-            body.insert(k.clone(), v.clone());
-        }
-    }
-    body.insert("provenance".to_string(), provenance_to_json(provenance));
-    body
-}
-
-/// `PUT /api/metadata/{cid}` — write (merge/upsert) the record body into
-/// meta-core. `base_url` is the meta-core root (e.g. `http://…-core:9000`).
-pub async fn put_record(
-    http: &reqwest::Client,
-    base_url: &str,
-    cid: &str,
-    body: &BTreeMap<String, String>,
-) -> anyhow::Result<()> {
-    let url = format!("{}/api/metadata/{}", base_url.trim_end_matches('/'), cid);
-    let resp = http.put(&url).json(body).send().await?;
-    let status = resp.status();
-    if !status.is_success() {
-        let text = resp.text().await.unwrap_or_default();
-        anyhow::bail!("meta-core PUT {url} returned {status}: {text}");
-    }
-    Ok(())
-}
-
-/// Shape of meta-core's `GET /meta/{cid}` response (`{ hashId, metadata }`).
-#[derive(Debug, Deserialize)]
-struct MetaGetResponse {
-    #[serde(default)]
-    metadata: Option<BTreeMap<String, String>>,
-}
-
-/// `GET /meta/{cid}` — read the current flat field map for a record, or `None`
-/// when meta-core has no record yet (404). Used to read back a plugin's merged
-/// fields (e.g. filename-parser's `originalTitle`/`movieYear`) before handing
-/// them to the next plugin in the pipeline.
-pub async fn get_record(
-    http: &reqwest::Client,
-    base_url: &str,
-    cid: &str,
-) -> anyhow::Result<Option<BTreeMap<String, String>>> {
-    let url = format!("{}/meta/{}", base_url.trim_end_matches('/'), cid);
-    let resp = http.get(&url).send().await?;
-    if resp.status() == reqwest::StatusCode::NOT_FOUND {
-        return Ok(None);
-    }
-    if !resp.status().is_success() {
-        anyhow::bail!("meta-core GET {url} returned {}", resp.status());
-    }
-    let parsed: MetaGetResponse = resp.json().await?;
-    Ok(parsed.metadata)
-}
-
-/// Self-publishing store for feeders that own their record lifecycle (D4):
-/// the feeder writes its own **metadata-only** records to meta-core and returns
-/// sparse outcomes so the gateway dispatcher skips them; the meta-share ingester
-/// Kamilata-propagates them.
-///
-/// **Byte outcomes are deliberately NOT self-published.** A feeder-side WebDAV
-/// write to the gateway core would be seeded by nobody: the gateway core runs
-/// with `ENABLE_FILE_WATCHER=false` (so meta-core emits no `/api/events/files`
-/// for it, and the ingester's `ipfs_seed` — which seeds only watched files —
-/// never sees it), and a sparse outcome makes the gateway skip its own
-/// blockstore seed. So byte-bearing outcomes (torznab full-file / subtitles)
-/// pass through unchanged and the **gateway** continues to WebDAV-PUT + bitswap-
-/// seed them (it's the libp2p/bitswap host). Result: feeder owns the metadata
-/// records (the Kamilata-propagated bulk), gateway owns byte delivery — a clean
-/// split with no seeding regression. Tribler is metadata-only and uses
-/// [`crate::enrich::Enricher`]; this is torznab's record-owning counterpart.
-#[derive(Clone)]
-pub struct FeederStore {
-    http: reqwest::Client,
-    meta_core_url: String,
-    gateway_peer: String,
-}
-
-impl FeederStore {
-    pub fn new(http: reqwest::Client, meta_core_url: String, gateway_peer: String) -> Self {
-        FeederStore {
-            http,
-            meta_core_url: meta_core_url.trim_end_matches('/').to_string(),
-            gateway_peer,
-        }
-    }
-
-    /// Build from the feeder env. `None` when `META_CORE_URL` is unset (the
-    /// feeder then falls back to the legacy "return outcomes, gateway stores"
-    /// path). `gateway_peer` ← `META_GATEWAY_PEER_ID` / `HOSTNAME`.
-    pub fn from_env() -> Option<Self> {
-        let meta_core_url = env_nonempty("META_CORE_URL")?;
-        let gateway_peer = env_nonempty("META_GATEWAY_PEER_ID")
-            .or_else(|| env_nonempty("HOSTNAME"))
-            .unwrap_or_else(|| "gateway-feeder".to_string());
-        let http = crate::common::build_http_client(30, "meta-feeder-store", None);
-        Some(Self::new(http, meta_core_url, gateway_peer))
-    }
-
-    pub fn meta_core_url(&self) -> &str {
-        &self.meta_core_url
-    }
-
-    /// Self-publish a **metadata-only** outcome to meta-core, returning the
-    /// **sparse** outcome (`record: None`) so the gateway's `(_, None)` branch
-    /// skips it — no double write. Byte outcomes and already-sparse outcomes are
-    /// returned unchanged (the gateway stores + seeds the bytes; see the struct
-    /// docs). On a write failure the original outcome is returned so the gateway
-    /// stores it (graceful degradation, never lost).
-    pub async fn publish(
-        &self,
-        upstream_id: &str,
-        record_id: &str,
-        outcome: HashOutcome,
-    ) -> HashOutcome {
-        // Only self-publish metadata-only outcomes (bytes None, record Some).
-        if outcome.bytes.is_some() || outcome.record.is_none() {
-            return outcome;
-        }
-        let cid = outcome.hash.0.clone();
-        let record = outcome.record.as_ref().expect("record is Some by the guard");
-        match self
-            .store_metadata_only(upstream_id, record_id, &cid, record)
-            .await
-        {
-            Ok(()) => sparse(outcome.hash, outcome.hash_kind),
-            Err(e) => {
-                warn!(target: "meta-feeder::store", cid = %cid, error = %e,
-                      "metadata self-publish failed; falling back to gateway store");
-                outcome
-            }
-        }
-    }
-
-    /// Metadata-only self-publish: `PUT /api/metadata/{cid}` with provenance.
-    async fn store_metadata_only(
-        &self,
-        upstream_id: &str,
-        record_id: &str,
-        cid: &str,
-        record: &DiscoveryRecord,
-    ) -> anyhow::Result<()> {
-        let prov = Provenance::now(upstream_id, record_id, &self.gateway_peer);
-        let body = build_metadata_body(record, &prov);
-        put_record(&self.http, &self.meta_core_url, cid, &body).await
-    }
-}
-
-fn sparse(hash: Hash, hash_kind: HashKind) -> HashOutcome {
-    HashOutcome {
-        hash,
-        hash_kind,
-        bytes: None,
-        record: None,
-        file_extension: None,
-    }
-}
-
-fn env_nonempty(key: &str) -> Option<String> {
-    std::env::var(key)
-        .ok()
-        .map(|s| s.trim().to_string())
-        .filter(|s| !s.is_empty())
-}
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-
-    fn rec() -> DiscoveryRecord {
-        DiscoveryRecord {
-            upstream_id: "tribler".into(),
-            record_id: "abc".into(),
-            fields: BTreeMap::from_iter([
-                ("title".into(), "Big Buck Bunny".into()),
-                ("cid_btih_v1_file".into(), "bafkPERFILE".into()),
-            ]),
-        }
-    }
-
-    #[test]
-    fn provenance_shape_matches_gateway() {
-        let p = Provenance {
-            upstream: "tribler".into(),
-            record_id: "abc".into(),
-            gateway_peer: "metagateway-tribler".into(),
-            acquired_at: 1_747_504_000,
-        };
-        let v: serde_json::Value = serde_json::from_str(&provenance_to_json(&p)).unwrap();
-        assert_eq!(v["source"], "gateway");
-        assert_eq!(v["upstream"], "tribler");
-        assert_eq!(v["recordId"], "abc");
-        assert_eq!(v["gatewayPeer"], "metagateway-tribler");
-        assert_eq!(v["acquiredAt"], 1_747_504_000_u64);
-    }
-
-    #[test]
-    fn body_collapses_cid_fields_and_stamps_provenance() {
-        let p = Provenance::now("tribler", "abc", "peer");
-        let body = build_metadata_body(&rec(), &p);
-        assert_eq!(body.get("title").map(String::as_str), Some("Big Buck Bunny"));
-        // cid_* collapsed to cids/<value>=true; no cid_* survives.
-        assert!(body.keys().all(|k| !k.starts_with("cid_")));
-        assert_eq!(
-            body.get("cids/bafkPERFILE").map(String::as_str),
-            Some("true")
-        );
-        assert!(body.contains_key("provenance"));
-    }
-}

crates/meta-feeder-sdk/src/plugin.rs

@@ -1,223 +0,0 @@
-//! The `FeederPlugin` trait + supporting types.
-//!
-//! A feeder plugin bridges one external upstream (`gutenberg`, `arxiv`,
-//! `torznab`, …) into the MetaMesh network. It **finds** records and
-//! **fetches** bytes; it does NOT hash-into-a-blockstore, store back into
-//! meta-core, or speak libp2p — those stay in the gateway core. This is why
-//! `FeederPlugin` has no `set_blockstore` / `set_tmdb_budget` hooks (the
-//! gateway's `GatewayPlugin` carried them): preview-seeding moves to the core,
-//! and a feeder that needs a TMDB budget owns it internally.
-//!
-//! The `serve_feeders` harness ([`crate::serve`]) wraps any set of
-//! `FeederPlugin`s in the feeder HTTP contract; the gateway core's
-//! `RemoteFeederPlugin` is the matching HTTP client.
-
-use std::collections::HashMap;
-use std::path::Path;
-
-use async_trait::async_trait;
-use futures::stream::BoxStream;
-
-// Re-exported so plugins can `use meta_feeder_sdk::plugin::GatewayQuery` (the
-// gateway's original `plugin.rs` re-exported it the same way).
-pub use crate::query::GatewayQuery;
-use crate::query::GatewaySearchEvent;
-use crate::types::{ByteStream, DiscoveryRecord, GatewayError, Hash, PluginHealth};
-
-/// One element of what [`FeederPlugin::compute_outcomes`] returns.
-///
-/// The `hash` is the value shipped to the gateway core. The other fields are
-/// signals the core's auto-store uses to persist the content into meta-core.
-/// The core's three-branch auto-store routes on `(bytes, record)`:
-/// - `(Some, Some)` — full store: WebDAV PUT + metadata PUT with `filePath`.
-/// - `(None, Some)` — metadata-only PUT (no local bytes; e.g. torznab samples
-///   the middle 1 MiB of a multi-GiB video and surfaces only the record).
-/// - `(_, None)` — skip. Cache hits land here.
-///
-/// `file_extension` MUST be `None` when `bytes` is `None`.
-#[derive(Debug)]
-pub struct HashOutcome {
-    pub hash: Hash,
-    /// Which hash family this CID belongs to. Drives the core's routing —
-    /// the bitswap blockstore seed only fires for `Sha2_256`.
-    pub hash_kind: HashKind,
-    pub bytes: Option<bytes::Bytes>,
-    pub record: Option<DiscoveryRecord>,
-    /// File extension to append when naming the WebDAV-side blob (e.g.
-    /// `"epub"`). No leading dot. MUST be `None` when `bytes` is `None`.
-    pub file_extension: Option<String>,
-}
-
-/// Hash family discriminator on [`HashOutcome`].
-///
-/// - `Midhash256` — size-prefix-plus-middle-1MB-sample, custom multicodec
-///   `0x1000`. Fast, MetaMesh-internal, not retrievable via public IPFS.
-/// - `Sha2_256` — standard IPFS CIDv1 over full bytes. Retrievable via bitswap
-///   once the bytes are in the core's blockstore. What every shipping plugin
-///   today emits.
-/// - `BtV1File` — a single file inside a BitTorrent v1 torrent, custom
-///   multicodec `0x1001`. Locator, not content hash: opaque to bitswap.
-/// - `NzbRelease` — a Usenet release (a Newznab listing), custom multicodec
-///   `nzb-release` `0x1005`. Self-describing locator: the cid embeds `{host,
-///   id}` in an identity multihash, opaque to bitswap and redeemed to bytes
-///   only by a credentialed meta-share peer, which decodes the locator from the
-///   cid and grabs the `.nzb` via `t=get` — no KV side-table.
-/// - `CardLocator` — the identity of a *work* (a series, a film) as published by
-///   a metadata bridge, custom multicodec `0x1007`. The only family with **no
-///   bytes behind it at all**: the record IS the payload. Always paired with
-///   `bytes: None`, so it lands in the `(None, Some)` metadata-only branch
-///   above. Opaque to bitswap, and deliberately so — there is nothing to seed.
-/// - `NzbPosting` — a Usenet posting we **scanned ourselves**, custom multicodec
-///   `nzb-posting` `0x1003`. A content-derived digest over the article
-///   Message-ID set (not a locator: no host is embedded), so any peer with a
-///   plain NNTP provider can fetch it — no `IndexerCred`. Opaque to bitswap: the
-///   digest is over the id set, not over any block, so the `.nzb` manifest
-///   travels separately as an ordinary sha2-256 cid in the record's `manifest`
-///   field. Produced by `hash::compute_nzb_posting_cid`.
-#[derive(Debug, Clone, Copy, PartialEq, Eq)]
-pub enum HashKind {
-    /// Reserved: no feeder currently emits midhash256 outcomes, but the family
-    /// is real across meta-core / meta-share, so the variant stays.
-    #[allow(dead_code)]
-    Midhash256,
-    Sha2_256,
-    BtV1File,
-    NzbRelease,
-    CardLocator,
-    NzbPosting,
-}
-
-/// Startup-only configuration error returned by [`FeederPlugin::configure`].
-#[derive(Debug, thiserror::Error)]
-pub enum ConfigError {
-    #[error("plugin {plugin}: required config missing: {what}")]
-    MissingConfig {
-        plugin: &'static str,
-        /// Human-readable description of the missing setting and where to
-        /// supply it. Used by the harness's soft-skip warning.
-        what: &'static str,
-    },
-
-    #[error("plugin {plugin}: cache dir setup failed: {source}")]
-    CacheSetup {
-        plugin: &'static str,
-        #[source]
-        source: std::io::Error,
-    },
-
-    #[error("plugin {plugin}: {source}")]
-    Other {
-        plugin: &'static str,
-        #[source]
-        source: anyhow::Error,
-    },
-}
-
-/// Canonical field name for an upstream's stable per-record identifier.
-/// Every plugin's `DiscoveryRecord.fields` MUST include this key
-/// (`gutenbergid:11`, `tmdbid:31910`, `arxivid:2106.07447`). The rule is
-/// `format!("{upstream_id}id")` — lowercase, no separator.
-pub fn upstream_id_field(upstream_id: &str) -> String {
-    format!("{upstream_id}id")
-}
-
-/// Static plugin contract. Each enabled `upstream_id` is implemented by
-/// exactly one `Box<dyn FeederPlugin>`. Lives behind `&self` in steady state —
-/// called concurrently, owns any interior mutability for caches.
-///
-/// **Field-naming convention.** Every record a plugin returns MUST include the
-/// canonical `<upstream_id>id` field — see [`upstream_id_field`].
-#[async_trait]
-pub trait FeederPlugin: Send + Sync + 'static {
-    fn upstream_id(&self) -> &'static str;
-
-    fn configure(&mut self, cache_dir: &Path) -> Result<(), ConfigError>;
-
-    async fn handle_query(
-        &self,
-        query: &GatewayQuery,
-        max_results: usize,
-    ) -> Result<Vec<DiscoveryRecord>, GatewayError>;
-
-    /// Streaming variant of [`handle_query`]. The default impl collects
-    /// [`handle_query`] and replays it as `Base*` + `Done` — correct for every
-    /// plugin with no incremental enrichment (gutenberg, arxiv, …). torznab
-    /// overrides it to decouple base records from rate-limited TMDB enrichment.
-    async fn handle_query_stream(
-        &self,
-        query: &GatewayQuery,
-        max_results: usize,
-    ) -> Result<BoxStream<'static, GatewaySearchEvent>, GatewayError> {
-        let records = self.handle_query(query, max_results).await?;
-        let events = records
-            .into_iter()
-            .map(GatewaySearchEvent::Base)
-            .chain(std::iter::once(GatewaySearchEvent::Done));
-        Ok(Box::pin(futures::stream::iter(events)))
-    }
-
-    /// Resolve `record_id` into one or more content-addressed outcomes.
-    ///
-    /// - `Err(_)` — the upstream record itself is unresolvable.
-    /// - `Ok(vec![])` — resolved but yielded zero outcomes.
-    /// - `Ok(partial_vec)` — bundle-success: per-sibling failures are
-    ///   best-effort (log + drop), surface whatever succeeded.
-    async fn compute_outcomes(&self, record_id: &str) -> Result<Vec<HashOutcome>, GatewayError>;
-
-    async fn handle_fetch(&self, _record_id: &str) -> Result<Option<ByteStream>, GatewayError> {
-        Ok(None)
-    }
-
-    fn health(&self) -> PluginHealth {
-        PluginHealth::Ok
-    }
-
-    fn served_file_types(&self) -> &'static [&'static str] {
-        &[]
-    }
-
-    fn served_content_kinds(&self) -> &'static [&'static str] {
-        &[]
-    }
-
-    /// Self-describing config schema. The feeder SDK serves this at
-    /// `GET /config/schema` and renders a generic form from it — the gateway and
-    /// UI carry no per-plugin field knowledge. Default: no configuration.
-    fn config_schema(&self) -> crate::config::ConfigSchema {
-        crate::config::ConfigSchema::default()
-    }
-
-    /// The plugin's current *effective* config as JSON, **unredacted** (secrets
-    /// included). The SDK redacts it before serving `GET /config/values` and uses
-    /// it as the merge base on the first save (before any `config.json` exists).
-    /// Default: empty object. Implementors should mirror the keys in
-    /// [`config_schema`](FeederPlugin::config_schema).
-    fn config_values(&self) -> serde_json::Value {
-        serde_json::json!({})
-    }
-
-    async fn get_blob(&self, _cid: &str) -> Option<Vec<u8>> {
-        None
-    }
-}
-
-/// Static plugin registry. Built once at startup; read-only thereafter.
-/// Keyed by `upstream_id`.
-pub type PluginRegistry = HashMap<&'static str, Box<dyn FeederPlugin>>;
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-
-    #[test]
-    fn config_error_messages_include_plugin_name() {
-        let e = ConfigError::MissingConfig {
-            plugin: "scihub",
-            what:
-                "mirrors (set them in the dashboard or gateway-config.json plugins.scihub.mirrors)",
-        };
-        let msg = e.to_string();
-        assert!(msg.contains("scihub"));
-        assert!(msg.contains("mirrors"));
-    }
-}

crates/meta-feeder-sdk/src/query.rs

@@ -1,147 +0,0 @@
-//! Structured query envelope + streaming-search event + wire-error types,
-//! shared by feeders (their HTTP contract) and the gateway core (its libp2p
-//! wire). These are pure serde data types — no libp2p, no transport.
-//!
-//! On the core side, `protocol.rs` re-uses these exact structs as the bincode
-//! wire shape; the `wire_parity_*` tests there pin the byte encoding. Keep
-//! field order / variant order / derive shape stable.
-
-use std::collections::BTreeMap;
-
-use serde::{Deserialize, Serialize};
-
-use crate::types::{DiscoveryRecord, GatewayError};
-
-/// Structured query envelope. Gateway plugins do best-effort pushdown to
-/// upstream APIs with native query languages (`torznab cat=movie`,
-/// `arxiv id_list=`).
-///
-/// **Authority rule:** `filters` / `ranges` / `negations` are authoritative;
-/// `raw_text` is informational. A plugin that consumes both will double-filter
-/// and under-match — pick one source per filter; structured wins.
-#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
-pub struct GatewayQuery {
-    /// Original user text (e.g. `"naruto contentKind:movie imdbid:tt0903747"`).
-    /// Verbatim — never mutated by the conversion.
-    pub raw_text: String,
-
-    /// Bare-word leaves only, space-joined. The plugin's primary `q=` input
-    /// for upstream search APIs that take a single search box.
-    pub free_text: String,
-
-    /// Structured filters keyed by field name (raw, unstemmed values).
-    /// Multiple values per key = OR within a key; AND across keys.
-    pub filters: BTreeMap<String, Vec<String>>,
-
-    /// Numeric range filters (e.g. `movieYear:2020..2024`). Both bounds
-    /// optional → unbounded on that side.
-    pub ranges: Vec<RangeFilter>,
-
-    /// Negation filters (e.g. `NOT genres:horror`, bare `NOT word`).
-    pub negations: Vec<Negation>,
-}
-
-/// Numeric range filter inside [`GatewayQuery::ranges`]. Both bounds are
-/// `Option<i64>` so open-ended ranges drop one bound to `None`.
-#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
-pub struct RangeFilter {
-    pub field: String,
-    pub lo: Option<i64>,
-    pub hi: Option<i64>,
-}
-
-/// Negation entry inside [`GatewayQuery::negations`]. `field` is `Some(name)`
-/// for typed negations (`NOT genres:horror`), `None` for bare-word negation.
-#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
-pub struct Negation {
-    pub field: Option<String>,
-    pub value: String,
-}
-
-impl GatewayQuery {
-    /// Bare-text input to hand to a free-form upstream search API. Falls back
-    /// to `"*"` when the conversion produced no free-text leaves so upstreams
-    /// that 4xx on empty search return their top-ranked results instead.
-    pub fn free_text_or_star(&self) -> &str {
-        if self.free_text.trim().is_empty() {
-            "*"
-        } else {
-            self.free_text.as_str()
-        }
-    }
-
-    /// Construct a free-text-only `GatewayQuery` for in-process use (tests,
-    /// debug routes). `raw_text` == `free_text`; structured fields stay empty.
-    pub fn from_free_text(text: impl Into<String>) -> Self {
-        let text = text.into();
-        Self {
-            raw_text: text.clone(),
-            free_text: text,
-            filters: BTreeMap::new(),
-            ranges: Vec::new(),
-            negations: Vec::new(),
-        }
-    }
-}
-
-/// One frame in the streaming Search response.
-///
-/// A plugin emits `Base` records as soon as they're discovered, then
-/// best-effort `EnrichPatch`/`Drop` events as enrichment lands, then a
-/// terminal `Done`. The default `handle_query_stream` collects `handle_query`
-/// and replays it as `Base*` + `Done`.
-#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
-pub enum GatewaySearchEvent {
-    /// An un-enriched discovery record.
-    Base(DiscoveryRecord),
-    /// Incremental enrichment for a previously-sent `Base`, keyed by
-    /// `record_id`. `set` fields are inserted/overwritten; `remove` deleted.
-    EnrichPatch {
-        record_id: String,
-        set: BTreeMap<String, String>,
-        remove: Vec<String>,
-    },
-    /// Retract a previously-emitted `Base` record entirely.
-    Drop { record_id: String },
-    /// Terminal success — no more frames follow.
-    Done,
-    /// Terminal error — no more frames follow.
-    Error(GatewayWireError),
-}
-
-/// Wire-stable error shape. Distinct from in-process [`GatewayError`] because
-/// adding a variant here is a wire-format change.
-#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize, thiserror::Error)]
-pub enum GatewayWireError {
-    #[error("not found")]
-    NotFound,
-    #[error("rate limited; retry in {retry_after_s}s")]
-    RateLimited { retry_after_s: u32 },
-    #[error("upstream transient: {0}")]
-    UpstreamTransient(String),
-    #[error("upstream permanent: {0}")]
-    UpstreamPermanent(String),
-    /// The protocol was registered but no plugin claims this `upstream_id`.
-    #[error("plugin not loaded")]
-    PluginNotLoaded,
-    /// Internal codec / decode failure on the *receiver* side.
-    #[error("internal: {0}")]
-    Internal(String),
-}
-
-impl From<GatewayError> for GatewayWireError {
-    fn from(e: GatewayError) -> Self {
-        match e {
-            GatewayError::Transient(s) => GatewayWireError::UpstreamTransient(s),
-            GatewayError::Permanent(s) => GatewayWireError::UpstreamPermanent(s),
-            GatewayError::RateLimited { retry_after_s } => {
-                GatewayWireError::RateLimited { retry_after_s }
-            }
-            GatewayError::NotFound => GatewayWireError::NotFound,
-            // Internal errors are never leaked; map to a transient.
-            GatewayError::Internal(_) => {
-                GatewayWireError::UpstreamTransient("internal".to_string())
-            }
-        }
-    }
-}

crates/meta-feeder-sdk/src/query_eval.rs

@@ -1,514 +0,0 @@
-//! Structured-filter evaluation for gateway records.
-//!
-//! Two pure helpers consuming the [`crate::query::GatewayQuery`] struct:
-//!
-//! - [`record_matches`] — does a single record's `fields` map satisfy the
-//!   query's `filters` + `ranges` + typed `negations`?
-//! - [`query_accepts_plugin`] — could a plugin advertising the given
-//!   `served_file_types` / `served_content_kinds` possibly produce a record
-//!   that satisfies the query? Used by plugins as a cheap early-return.
-//!
-//! The dispatcher applies `record_matches` to every plugin's return value, and
-//! meta-share applies the same algorithm consumer-side to defend against stale
-//! gateway peers — both tiers must agree on outcomes for any (record, query)
-//! pair. See the original gateway crate docs for the full semantics.
-
-use std::collections::BTreeMap;
-
-use crate::query::GatewayQuery;
-
-/// True iff a record's `fields` satisfies every filter, range, and typed
-/// negation in `query`.
-pub fn record_matches(fields: &BTreeMap<String, String>, query: &GatewayQuery) -> bool {
-    for (key, allowed) in &query.filters {
-        if key == "languages" {
-            if !languages_filter_matches(fields, allowed) {
-                return false;
-            }
-            continue;
-        }
-        // `genres` is a key-set too (`genres/<Name> = "true"`, METADATA_KEYS
-        // §`genres/{name}`) on everything the card tier writes. Without this arm
-        // a `genres:` filter looks up a flat `genres` field, finds nothing on a
-        // card, and drops it — so a genre-scoped row would return empty.
-        if key == "genres" {
-            if !genres_filter_matches(fields, allowed) {
-                return false;
-            }
-            continue;
-        }
-        let Some(field_value) = fields.get(key) else {
-            return false;
-        };
-        if !any_csv_value_matches(field_value, allowed) {
-            return false;
-        }
-    }
-
-    for range in &query.ranges {
-        let Some(field_value) = fields.get(&range.field) else {
-            return false;
-        };
-        let Ok(n) = field_value.parse::<i64>() else {
-            return false;
-        };
-        if let Some(lo) = range.lo {
-            if n < lo {
-                return false;
-            }
-        }
-        if let Some(hi) = range.hi {
-            if n > hi {
-                return false;
-            }
-        }
-    }
-
-    for negation in &query.negations {
-        let Some(name) = negation.field.as_deref() else {
-            // Bare-word negation — not enforced here.
-            continue;
-        };
-        let Some(field_value) = fields.get(name) else {
-            // Missing field ⇒ negation passes (nothing to exclude).
-            continue;
-        };
-        if any_csv_value_matches(field_value, std::slice::from_ref(&negation.value)) {
-            return false;
-        }
-    }
-
-    true
-}
-
-/// True iff a plugin advertising the given `served_file_types` /
-/// `served_content_kinds` could possibly produce a record satisfying
-/// `query.filters["fileType" | "contentKind"]`.
-pub fn query_accepts_plugin(
-    query: &GatewayQuery,
-    served_file_types: &[&str],
-    served_content_kinds: &[&str],
-) -> bool {
-    accepts_axis(query.filters.get("fileType"), served_file_types)
-        && accepts_axis(query.filters.get("contentKind"), served_content_kinds)
-}
-
-fn accepts_axis(requested: Option<&Vec<String>>, served: &[&str]) -> bool {
-    let Some(requested) = requested else {
-        return true;
-    };
-    if requested.is_empty() {
-        return true;
-    }
-    // Wildcard: a plugin that advertises `"*"` on an axis serves every value on
-    // that axis (e.g. tribler is a free-text overlay search with no inherent
-    // type limitation). It accepts any requested fileType/contentKind.
-    if served.contains(&"*") {
-        return true;
-    }
-    requested
-        .iter()
-        .any(|v| served.iter().any(|s| s.eq_ignore_ascii_case(v)))
-}
-
-/// Resolve a `languages` filter against the key-set storage form
-/// (`languages/<lang3> = "true"`). **Fails open** on anything we can't
-/// confidently exclude. A record matches when:
-/// 1. it carries a requested code as a truthy key-set member (OR across codes);
-/// 2. a legacy flat `languages` csv field contains a requested code;
-/// 3. it carries `mul` (multiple — may contain a requested one) or `und`
-///    (undetermined); or
-/// 4. it carries no language information at all.
-///
-/// Only a record with a *concrete*, non-matching language (e.g. `jpn`-only
-/// under an `eng` filter) is dropped. Kept byte-identical in logic with
-/// meta-search's copy (and the client's `passesLangFilter`) — language narrows
-/// the result set without ever hiding unknown/undetermined/multi releases.
-fn languages_filter_matches(fields: &BTreeMap<String, String>, allowed: &[String]) -> bool {
-    for code in allowed {
-        let key = format!("languages/{}", code.trim().to_ascii_lowercase());
-        if fields.get(&key).is_some_and(|v| is_truthy_member(v)) {
-            return true;
-        }
-    }
-    if let Some(flat) = fields.get("languages") {
-        if any_csv_value_matches(flat, allowed) {
-            return true;
-        }
-    }
-    // Fail open: `mul`/`und` are explicit "could be anything / unknown" markers,
-    // and a record with no concrete language at all is equally unknown. Keep all
-    // three; only a specific, non-matching language reaches the `false` below.
-    if fields.get("languages/mul").is_some_and(|v| is_truthy_member(v))
-        || fields.get("languages/und").is_some_and(|v| is_truthy_member(v))
-    {
-        return true;
-    }
-    !record_has_concrete_language(fields)
-}
-
-/// True iff the record states a *concrete* language: a truthy `languages/<code>`
-/// key-set member other than the `und`/`mul` non-codes, or a non-empty legacy
-/// flat `languages` field. Distinguishes "a specific language that didn't match
-/// ⇒ drop" from "no language info ⇒ keep" for [`languages_filter_matches`].
-fn record_has_concrete_language(fields: &BTreeMap<String, String>) -> bool {
-    if fields.get("languages").is_some_and(|v| !v.trim().is_empty()) {
-        return true;
-    }
-    fields.iter().any(|(k, v)| {
-        is_truthy_member(v)
-            && k.starts_with("languages/")
-            && k != "languages/und"
-            && k != "languages/mul"
-    })
-}
-
-/// Key-set members are written as the literal `"true"`; tolerate a couple of
-/// obvious truthy spellings and treat anything else as absent.
-/// Fold a genre label to its comparison form: lowercase, alphanumerics only.
-///
-/// Genre display names carry spaces and ampersands (`"Action & Adventure"`,
-/// `"Sci-Fi & Fantasy"`, `"Science Fiction"`), and **space is the query DSL's
-/// token separator** — so a filter can never spell a genre the way the record
-/// stores it. Callers pass a slug (`genres:action-adventure`); folding both
-/// sides makes them one value without forcing the stored name to be
-/// machine-shaped.
-fn genre_fold(s: &str) -> String {
-    s.chars()
-        .filter(|c| c.is_alphanumeric())
-        .flat_map(|c| c.to_lowercase())
-        .collect()
-}
-
-/// True iff the record carries one of the requested genres, in **either**
-/// encoding: a truthy `genres/<Name>` key-set member (the card tier's shape) or
-/// the legacy flat comma-joined `genres` field. OR across the requested genres,
-/// both sides folded — see [`genre_fold`].
-///
-/// **Fails closed, unlike [`languages_filter_matches`].** A language filter is a
-/// viewer *preference* narrowing a set, so an unknown-language release is kept.
-/// A genre filter *defines what a row is*: a record with no genre information is
-/// not known to be Action, so putting it in an Action row states something
-/// untrue. Upstream has already filtered by genre, so this is a re-validation.
-fn genres_filter_matches(fields: &BTreeMap<String, String>, allowed: &[String]) -> bool {
-    let wanted: Vec<String> = allowed.iter().map(|g| genre_fold(g)).collect();
-    if wanted.iter().all(|g| g.is_empty()) {
-        return true; // a filter with no usable value constrains nothing
-    }
-    for (key, value) in fields {
-        let Some(name) = key.strip_prefix("genres/") else {
-            continue;
-        };
-        if is_truthy_member(value) && wanted.contains(&genre_fold(name)) {
-            return true;
-        }
-    }
-    if let Some(flat) = fields.get("genres") {
-        for member in flat.split(|c| c == ',' || c == '|') {
-            if wanted.contains(&genre_fold(member)) {
-                return true;
-            }
-        }
-    }
-    false
-}
-
-fn is_truthy_member(v: &str) -> bool {
-    matches!(v.trim().to_ascii_lowercase().as_str(), "true" | "1" | "yes")
-}
-
-/// True iff `field_value` (treated as a csv set when it contains commas)
-/// contains at least one case-insensitive match for any item in `allowed`.
-fn any_csv_value_matches(field_value: &str, allowed: &[String]) -> bool {
-    for piece in field_value.split(',') {
-        let piece = piece.trim();
-        if piece.is_empty() {
-            continue;
-        }
-        if allowed.iter().any(|v| v.eq_ignore_ascii_case(piece)) {
-            return true;
-        }
-    }
-    false
-}
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-    use crate::query::{Negation, RangeFilter};
-
-    fn empty_query() -> GatewayQuery {
-        GatewayQuery {
-            raw_text: String::new(),
-            free_text: String::new(),
-            filters: BTreeMap::new(),
-            ranges: Vec::new(),
-            negations: Vec::new(),
-        }
-    }
-
-    fn with_filter(field: &str, values: &[&str]) -> GatewayQuery {
-        let mut q = empty_query();
-        q.filters.insert(
-            field.to_string(),
-            values.iter().map(|s| s.to_string()).collect(),
-        );
-        q
-    }
-
-    fn fields(pairs: &[(&str, &str)]) -> BTreeMap<String, String> {
-        pairs
-            .iter()
-            .map(|(k, v)| (k.to_string(), v.to_string()))
-            .collect()
-    }
-
-    #[test]
-    fn filter_rejects_mismatched_filetype() {
-        let q = with_filter("fileType", &["video"]);
-        assert!(!record_matches(&fields(&[("fileType", "image")]), &q));
-    }
-
-    #[test]
-    fn filter_accepts_matching_filetype() {
-        let q = with_filter("fileType", &["video"]);
-        assert!(record_matches(&fields(&[("fileType", "video")]), &q));
-    }
-
-    #[test]
-    fn filter_rejects_record_missing_field() {
-        let q = with_filter("fileType", &["video"]);
-        assert!(!record_matches(&fields(&[]), &q));
-    }
-
-    #[test]
-    fn filter_is_case_insensitive() {
-        let q = with_filter("fileType", &["VIDEO"]);
-        assert!(record_matches(&fields(&[("fileType", "video")]), &q));
-        let q = with_filter("fileType", &["video"]);
-        assert!(record_matches(&fields(&[("fileType", "VIDEO")]), &q));
-    }
-
-    #[test]
-    fn filter_ors_within_key() {
-        let q = with_filter("fileType", &["video", "audio"]);
-        assert!(record_matches(&fields(&[("fileType", "audio")]), &q));
-        assert!(record_matches(&fields(&[("fileType", "video")]), &q));
-        assert!(!record_matches(&fields(&[("fileType", "image")]), &q));
-    }
-
-    #[test]
-    fn filter_ands_across_keys() {
-        let mut q = with_filter("fileType", &["video"]);
-        q.filters.insert("genres".into(), vec!["Action".into()]);
-        assert!(!record_matches(
-            &fields(&[("fileType", "video"), ("genres", "Drama")]),
-            &q
-        ));
-        assert!(record_matches(
-            &fields(&[("fileType", "video"), ("genres", "Action")]),
-            &q
-        ));
-    }
-
-    #[test]
-    fn csv_field_matches_one_of_its_values() {
-        let q = with_filter("genres", &["Anime"]);
-        assert!(record_matches(
-            &fields(&[("genres", "Animation,Anime,Drama")]),
-            &q
-        ));
-    }
-
-    #[test]
-    fn csv_field_with_whitespace_is_trimmed() {
-        let q = with_filter("genres", &["Anime"]);
-        assert!(record_matches(
-            &fields(&[("genres", "Animation, Anime , Drama")]),
-            &q
-        ));
-    }
-
-    #[test]
-    fn languages_keyset_member_matches() {
-        let q = with_filter("languages", &["eng"]);
-        assert!(record_matches(&fields(&[("languages/eng", "true")]), &q));
-    }
-
-    #[test]
-    fn languages_filter_is_or_across_codes() {
-        let q = with_filter("languages", &["eng", "fre"]);
-        assert!(record_matches(&fields(&[("languages/eng", "true")]), &q));
-        assert!(record_matches(&fields(&[("languages/fre", "true")]), &q));
-        assert!(!record_matches(&fields(&[("languages/jpn", "true")]), &q));
-    }
-
-    #[test]
-    fn languages_filter_fails_open_on_unknown() {
-        let q = with_filter("languages", &["eng"]);
-        // No language info at all ⇒ kept (unknown, not excluded).
-        assert!(record_matches(&fields(&[("fileType", "video")]), &q));
-        // `mul` (multi) may contain the requested language ⇒ kept.
-        assert!(record_matches(&fields(&[("languages/mul", "true")]), &q));
-        // `und` (undetermined) ⇒ kept.
-        assert!(record_matches(&fields(&[("languages/und", "true")]), &q));
-        // A specific language alongside `mul` is still kept (multi).
-        assert!(record_matches(
-            &fields(&[("languages/jpn", "true"), ("languages/mul", "true")]),
-            &q
-        ));
-    }
-
-    #[test]
-    fn languages_concrete_mismatch_still_drops() {
-        let q = with_filter("languages", &["eng"]);
-        // A concrete, non-matching language ⇒ dropped (language narrows).
-        assert!(!record_matches(&fields(&[("languages/jpn", "true")]), &q));
-        // A tombstoned/false member is treated as absent ⇒ unknown ⇒ kept.
-        assert!(record_matches(&fields(&[("languages/eng", "false")]), &q));
-    }
-
-    #[test]
-    fn languages_legacy_flat_csv_fallback() {
-        let q = with_filter("languages", &["eng"]);
-        assert!(record_matches(&fields(&[("languages", "jpn,eng")]), &q));
-        assert!(!record_matches(&fields(&[("languages", "jpn,ger")]), &q));
-    }
-
-    #[test]
-    fn range_in_bounds() {
-        let mut q = empty_query();
-        q.ranges.push(RangeFilter {
-            field: "movieYear".into(),
-            lo: Some(2010),
-            hi: Some(2020),
-        });
-        assert!(record_matches(&fields(&[("movieYear", "2015")]), &q));
-    }
-
-    #[test]
-    fn range_out_of_bounds_rejected() {
-        let mut q = empty_query();
-        q.ranges.push(RangeFilter {
-            field: "movieYear".into(),
-            lo: Some(2010),
-            hi: Some(2020),
-        });
-        assert!(!record_matches(&fields(&[("movieYear", "2025")]), &q));
-        assert!(!record_matches(&fields(&[("movieYear", "2005")]), &q));
-    }
-
-    #[test]
-    fn range_open_upper_bound() {
-        let mut q = empty_query();
-        q.ranges.push(RangeFilter {
-            field: "movieYear".into(),
-            lo: Some(2010),
-            hi: None,
-        });
-        assert!(record_matches(&fields(&[("movieYear", "2025")]), &q));
-    }
-
-    #[test]
-    fn range_unparseable_value_rejected() {
-        let mut q = empty_query();
-        q.ranges.push(RangeFilter {
-            field: "movieYear".into(),
-            lo: Some(2010),
-            hi: None,
-        });
-        assert!(!record_matches(&fields(&[("movieYear", "circa 2015")]), &q));
-    }
-
-    #[test]
-    fn typed_negation_excludes_match() {
-        let mut q = empty_query();
-        q.negations.push(Negation {
-            field: Some("fileType".into()),
-            value: "image".into(),
-        });
-        assert!(!record_matches(&fields(&[("fileType", "image")]), &q));
-        assert!(record_matches(&fields(&[("fileType", "video")]), &q));
-    }
-
-    #[test]
-    fn typed_negation_passes_when_field_missing() {
-        let mut q = empty_query();
-        q.negations.push(Negation {
-            field: Some("fileType".into()),
-            value: "image".into(),
-        });
-        assert!(record_matches(&fields(&[]), &q));
-    }
-
-    #[test]
-    fn bare_word_negation_is_not_enforced() {
-        let mut q = empty_query();
-        q.negations.push(Negation {
-            field: None,
-            value: "alice".into(),
-        });
-        assert!(record_matches(
-            &fields(&[("title", "alice in wonderland")]),
-            &q
-        ));
-    }
-
-    #[test]
-    fn empty_query_matches_anything() {
-        assert!(record_matches(&fields(&[]), &empty_query()));
-        assert!(record_matches(
-            &fields(&[("fileType", "video"), ("title", "x")]),
-            &empty_query()
-        ));
-    }
-
-    #[test]
-    fn accepts_returns_true_when_no_type_filter() {
-        let q = empty_query();
-        assert!(query_accepts_plugin(&q, &["video"], &["movie"]));
-        assert!(query_accepts_plugin(&q, &[], &[]));
-    }
-
-    #[test]
-    fn accepts_rejects_disjoint_filetype() {
-        let q = with_filter("fileType", &["video"]);
-        assert!(!query_accepts_plugin(&q, &["image"], &["gif"]));
-        assert!(query_accepts_plugin(&q, &["video"], &[]));
-    }
-
-    #[test]
-    fn accepts_rejects_disjoint_contentkind() {
-        let q = with_filter("contentKind", &["movie"]);
-        assert!(!query_accepts_plugin(
-            &q,
-            &["video", "image"],
-            &["paper", "book"]
-        ));
-        assert!(query_accepts_plugin(&q, &["video"], &["movie", "episode"]));
-    }
-
-    #[test]
-    fn accepts_requires_both_axes_when_both_filtered() {
-        let mut q = with_filter("fileType", &["video"]);
-        q.filters.insert("contentKind".into(), vec!["movie".into()]);
-        assert!(query_accepts_plugin(&q, &["video"], &["movie"]));
-        assert!(!query_accepts_plugin(&q, &["video"], &["paper"]));
-        assert!(!query_accepts_plugin(&q, &["image"], &["movie"]));
-    }
-
-    #[test]
-    fn accepts_wildcard_matches_any_requested_type() {
-        // A plugin advertising "*" (e.g. tribler) accepts any fileType /
-        // contentKind filter on the wildcarded axis.
-        let q = with_filter("fileType", &["archive"]);
-        assert!(query_accepts_plugin(&q, &["*"], &[]));
-        let q = with_filter("contentKind", &["audiobook"]);
-        assert!(query_accepts_plugin(&q, &["*"], &["*"]));
-        // Wildcard on one axis only: the non-wildcard axis still gates.
-        let mut q = with_filter("fileType", &["audio"]);
-        q.filters.insert("contentKind".into(), vec!["podcast".into()]);
-        assert!(query_accepts_plugin(&q, &["*"], &["*"]));
-        assert!(!query_accepts_plugin(&q, &["*"], &["movie"]));
-    }
-}

crates/meta-feeder-sdk/src/serve.rs

@@ -1,759 +0,0 @@
-//! `serve_feeders` — the axum harness that exposes a set of [`FeederPlugin`]s
-//! over the **feeder HTTP contract**. Every feeder sidecar binary is just
-//! `serve_feeders(vec![MyPlugin::new()], state_dir, listen)`.
-//!
-//! # Contract (JSON over HTTP, mirrors meta-sort's container-plugin shape)
-//!
-//! - `GET  /manifest`                  → [`ManifestResponse`]
-//! - `GET  /health`                    → [`HealthResponse`]
-//! - `POST /query`                     → [`QueryRequest`] → [`QueryResponse`]
-//! - `POST /compute`                   → [`ComputeRequest`] → [`ComputeResponse`]
-//! - `GET  /fetch/{upstream_id}/{record_id}` → streamed bytes (or 404)
-//! - `GET  /blob/{upstream_id}/{cid}`  → bytes (or 404)
-//!
-//! ## Bytes & the v1 simplification
-//!
-//! `/compute` returns each outcome's bytes **inline, base64-encoded**
-//! ([`OutcomeDto::bytes_b64`]) — this reuses every existing `compute_outcomes`
-//! impl verbatim (they already return bytes in `HashOutcome.bytes`) and keeps
-//! the gateway core's auto-store a single round-trip. base64 adds ~33%; fine
-//! for books/papers/posters. Large-file streaming (torznab multi-GiB) is a
-//! documented follow-up — those outcomes are metadata-only anyway
-//! (`bytes: None`), so they never ride this path.
-
-use std::collections::HashMap;
-use std::net::SocketAddr;
-use std::path::{Path as FsPath, PathBuf};
-use std::sync::Arc;
-
-use axum::{
-    body::Body,
-    extract::{Path, State},
-    http::StatusCode,
-    response::{Html, IntoResponse, Response},
-    routing::{get, post},
-    Json, Router,
-};
-use base64::Engine as _;
-use futures::StreamExt;
-use serde::{Deserialize, Serialize};
-use serde_json::Value;
-use tracing::{info, warn};
-
-use crate::config::{self, CONFIG_PAGE_HTML};
-use crate::plugin::{ConfigError, FeederPlugin, HashKind, HashOutcome};
-use crate::query::{GatewayQuery, GatewaySearchEvent, GatewayWireError};
-use crate::types::{DiscoveryRecord, GatewayError, PluginHealth};
-
-// -------- wire DTOs --------------------------------------------------------
-
-/// Serializable mirror of [`HashKind`] for the feeder HTTP boundary.
-#[derive(Serialize, Deserialize, Clone, Copy, PartialEq, Eq, Debug)]
-#[serde(rename_all = "snake_case")]
-pub enum HashKindDto {
-    Midhash256,
-    Sha2_256,
-    BtV1File,
-    NzbRelease,
-    CardLocator,
-    NzbPosting,
-}
-
-impl From<HashKind> for HashKindDto {
-    fn from(k: HashKind) -> Self {
-        match k {
-            HashKind::Midhash256 => HashKindDto::Midhash256,
-            HashKind::Sha2_256 => HashKindDto::Sha2_256,
-            HashKind::BtV1File => HashKindDto::BtV1File,
-            HashKind::NzbRelease => HashKindDto::NzbRelease,
-            HashKind::CardLocator => HashKindDto::CardLocator,
-            HashKind::NzbPosting => HashKindDto::NzbPosting,
-        }
-    }
-}
-
-impl From<HashKindDto> for HashKind {
-    fn from(k: HashKindDto) -> Self {
-        match k {
-            HashKindDto::Midhash256 => HashKind::Midhash256,
-            HashKindDto::Sha2_256 => HashKind::Sha2_256,
-            HashKindDto::BtV1File => HashKind::BtV1File,
-            HashKindDto::NzbRelease => HashKind::NzbRelease,
-            HashKindDto::CardLocator => HashKind::CardLocator,
-            HashKindDto::NzbPosting => HashKind::NzbPosting,
-        }
-    }
-}
-
-/// One plugin's capability surface in [`ManifestResponse`].
-#[derive(Serialize, Deserialize, Clone, Debug)]
-pub struct PluginManifest {
-    pub id: String,
-    pub served_file_types: Vec<String>,
-    pub served_content_kinds: Vec<String>,
-}
-
-/// `GET /manifest` — the feeder's hosted plugins + their served types. The
-/// gateway core reads this to register `RemoteFeederPlugin`s and to build the
-/// capability heartbeat it broadcasts to meta-share.
-#[derive(Serialize, Deserialize, Clone, Debug)]
-pub struct ManifestResponse {
-    pub feeder_version: String,
-    pub plugins: Vec<PluginManifest>,
-}
-
-/// One plugin's liveness in [`HealthResponse`].
-#[derive(Serialize, Deserialize, Clone, Debug)]
-pub struct PluginHealthEntry {
-    pub id: String,
-    pub health: PluginHealth,
-}
-
-/// `GET /health` — `status` is `"ok"` iff every hosted plugin reports `Ok`.
-#[derive(Serialize, Deserialize, Clone, Debug)]
-pub struct HealthResponse {
-    pub status: String,
-    pub plugins: Vec<PluginHealthEntry>,
-}
-
-/// `POST /query` request body.
-#[derive(Serialize, Deserialize, Clone, Debug)]
-pub struct QueryRequest {
-    pub upstream_id: String,
-    pub query: GatewayQuery,
-    pub max_results: u32,
-}
-
-/// `POST /query` response body.
-#[derive(Serialize, Deserialize, Clone, Debug)]
-pub struct QueryResponse {
-    pub records: Vec<DiscoveryRecord>,
-}
-
-/// `POST /compute` request body.
-#[derive(Serialize, Deserialize, Clone, Debug)]
-pub struct ComputeRequest {
-    pub upstream_id: String,
-    pub record_id: String,
-}
-
-/// One resolved outcome on the feeder wire. `bytes_b64` carries the full
-/// payload base64-encoded when the plugin fetched bytes (`Sha2_256` full
-/// store); `None` for metadata-only outcomes (the core's `(None, Some)`
-/// branch — e.g. torznab midhash).
-#[derive(Serialize, Deserialize, Clone, Debug)]
-pub struct OutcomeDto {
-    pub hash: String,
-    pub hash_kind: HashKindDto,
-    pub file_extension: Option<String>,
-    pub record: Option<DiscoveryRecord>,
-    pub bytes_b64: Option<String>,
-}
-
-impl OutcomeDto {
-    fn from_outcome(o: HashOutcome) -> Self {
-        OutcomeDto {
-            hash: o.hash.0,
-            hash_kind: o.hash_kind.into(),
-            file_extension: o.file_extension,
-            record: o.record,
-            bytes_b64: o
-                .bytes
-                .map(|b| base64::engine::general_purpose::STANDARD.encode(&b)),
-        }
-    }
-}
-
-/// `POST /compute` response body.
-#[derive(Serialize, Deserialize, Clone, Debug)]
-pub struct ComputeResponse {
-    pub outcomes: Vec<OutcomeDto>,
-}
-
-// -------- app state + error mapping ----------------------------------------
-
-#[derive(Clone)]
-struct AppState {
-    plugins: Arc<HashMap<String, Arc<dyn FeederPlugin>>>,
-    version: String,
-    /// Feeder state dir. Per-plugin config persists at
-    /// `state_dir/gateway/<id>/config.json` — the same per-plugin dir handed to
-    /// `configure()`, so a UI save is read back on the next restart.
-    state_dir: PathBuf,
-}
-
-impl AppState {
-    fn plugin(&self, upstream_id: &str) -> Result<&Arc<dyn FeederPlugin>, Response> {
-        self.plugins.get(upstream_id).ok_or_else(|| {
-            (
-                StatusCode::NOT_FOUND,
-                format!("no feeder plugin for upstream_id '{upstream_id}'"),
-            )
-                .into_response()
-        })
-    }
-
-    /// `state_dir/gateway/<id>/config.json` — canonical per-plugin config file.
-    fn config_path(&self, upstream_id: &str) -> PathBuf {
-        self.state_dir
-            .join("gateway")
-            .join(upstream_id)
-            .join("config.json")
-    }
-
-    /// The effective config as JSON: the persisted `config.json` if it exists,
-    /// else the plugin's in-memory (env-seeded) config. Used as the redaction
-    /// source for reads and the merge base for writes.
-    fn effective_config(&self, upstream_id: &str) -> Value {
-        if let Ok(bytes) = std::fs::read(self.config_path(upstream_id)) {
-            if let Ok(v) = serde_json::from_slice::<Value>(&bytes) {
-                return v;
-            }
-        }
-        self.plugins
-            .get(upstream_id)
-            .map(|p| p.config_values())
-            .unwrap_or_else(|| serde_json::json!({}))
-    }
-}
-
-/// Map an in-process [`GatewayError`] to an HTTP status + message. Mirrors the
-/// status semantics the gateway's `RemoteFeederPlugin` will translate back
-/// into a `GatewayError` on the core side.
-fn gateway_error_response(e: GatewayError) -> Response {
-    let (status, msg) = match e {
-        GatewayError::NotFound => (StatusCode::NOT_FOUND, "not found".to_string()),
-        GatewayError::RateLimited { retry_after_s } => (
-            StatusCode::TOO_MANY_REQUESTS,
-            format!("rate limited; retry in {retry_after_s}s"),
-        ),
-        GatewayError::Transient(s) => (StatusCode::BAD_GATEWAY, s),
-        GatewayError::Permanent(s) => (StatusCode::UNPROCESSABLE_ENTITY, s),
-        GatewayError::Internal(s) => (StatusCode::INTERNAL_SERVER_ERROR, s.to_string()),
-    };
-    (status, msg).into_response()
-}
-
-// -------- handlers ---------------------------------------------------------
-
-async fn manifest(State(state): State<AppState>) -> Json<ManifestResponse> {
-    let mut plugins: Vec<PluginManifest> = state
-        .plugins
-        .values()
-        .map(|p| PluginManifest {
-            id: p.upstream_id().to_string(),
-            served_file_types: p.served_file_types().iter().map(|s| s.to_string()).collect(),
-            served_content_kinds: p
-                .served_content_kinds()
-                .iter()
-                .map(|s| s.to_string())
-                .collect(),
-        })
-        .collect();
-    plugins.sort_by(|a, b| a.id.cmp(&b.id));
-    Json(ManifestResponse {
-        feeder_version: state.version.clone(),
-        plugins,
-    })
-}
-
-async fn health(State(state): State<AppState>) -> Json<HealthResponse> {
-    let mut entries: Vec<PluginHealthEntry> = state
-        .plugins
-        .values()
-        .map(|p| PluginHealthEntry {
-            id: p.upstream_id().to_string(),
-            health: p.health(),
-        })
-        .collect();
-    entries.sort_by(|a, b| a.id.cmp(&b.id));
-    let all_ok = entries
-        .iter()
-        .all(|e| matches!(e.health, PluginHealth::Ok));
-    Json(HealthResponse {
-        status: if all_ok { "ok" } else { "degraded" }.to_string(),
-        plugins: entries,
-    })
-}
-
-async fn query(
-    State(state): State<AppState>,
-    Json(req): Json<QueryRequest>,
-) -> Result<Json<QueryResponse>, Response> {
-    let plugin = state.plugin(&req.upstream_id)?;
-    let records = plugin
-        .handle_query(&req.query, req.max_results as usize)
-        .await
-        .map_err(gateway_error_response)?;
-    Ok(Json(QueryResponse { records }))
-}
-
-/// `POST /query_stream` — the streaming analog of [`query`]. Emits the plugin's
-/// [`FeederPlugin::handle_query_stream`] events as **NDJSON**: one JSON-encoded
-/// [`GatewaySearchEvent`] per line (`application/x-ndjson`). The gateway core's
-/// `RemoteFeederPlugin` consumes this and re-emits a `BoxStream<GatewaySearchEvent>`
-/// into the libp2p streaming worker, preserving incremental enrichment (e.g.
-/// torznab's late TMDB / poster / subtitle-availability patches). A plugin that
-/// doesn't override `handle_query_stream` still works here — the trait default
-/// replays `handle_query` as `Base* + Done`.
-async fn query_stream(
-    State(state): State<AppState>,
-    Json(req): Json<QueryRequest>,
-) -> Result<Response, Response> {
-    let plugin = state.plugin(&req.upstream_id)?;
-    // The open can fail (e.g. the plugin can't reach its upstream at all); that
-    // maps to an HTTP error status, same as `/query`. Per-event errors instead
-    // ride the stream as a terminal `GatewaySearchEvent::Error` line.
-    let events = plugin
-        .handle_query_stream(&req.query, req.max_results as usize)
-        .await
-        .map_err(gateway_error_response)?;
-    let body = events.map(|ev| {
-        // Serialization of GatewaySearchEvent is infallible in practice; on the
-        // off chance it fails, terminate with an Error line so the consumer sees
-        // a clean terminal failure rather than a truncated stream.
-        let mut line = serde_json::to_vec(&ev).unwrap_or_else(|e| {
-            serde_json::to_vec(&GatewaySearchEvent::Error(GatewayWireError::Internal(
-                format!("event serialize: {e}"),
-            )))
-            .unwrap_or_default()
-        });
-        line.push(b'\n');
-        Ok::<_, std::convert::Infallible>(line)
-    });
-    Ok((
-        [(axum::http::header::CONTENT_TYPE, "application/x-ndjson")],
-        Body::from_stream(body),
-    )
-        .into_response())
-}
-
-async fn compute(
-    State(state): State<AppState>,
-    Json(req): Json<ComputeRequest>,
-) -> Result<Json<ComputeResponse>, Response> {
-    let plugin = state.plugin(&req.upstream_id)?;
-    let outcomes = plugin
-        .compute_outcomes(&req.record_id)
-        .await
-        .map_err(gateway_error_response)?;
-    Ok(Json(ComputeResponse {
-        outcomes: outcomes.into_iter().map(OutcomeDto::from_outcome).collect(),
-    }))
-}
-
-async fn fetch(
-    State(state): State<AppState>,
-    Path((upstream_id, record_id)): Path<(String, String)>,
-) -> Result<Response, Response> {
-    let plugin = state.plugin(&upstream_id)?;
-    match plugin
-        .handle_fetch(&record_id)
-        .await
-        .map_err(gateway_error_response)?
-    {
-        Some(stream) => {
-            // Map the plugin's GatewayError stream into an io::Error stream so
-            // axum's Body::from_stream accepts it.
-            let body_stream = stream.map(|chunk| {
-                chunk.map_err(|e| std::io::Error::other(e.to_string()))
-            });
-            Ok(Body::from_stream(body_stream).into_response())
-        }
-        None => Err((StatusCode::NOT_FOUND, "no bytes for record").into_response()),
-    }
-}
-
-async fn blob(
-    State(state): State<AppState>,
-    Path((upstream_id, cid)): Path<(String, String)>,
-) -> Result<Response, Response> {
-    let plugin = state.plugin(&upstream_id)?;
-    match plugin.get_blob(&cid).await {
-        Some(bytes) => Ok(bytes.into_response()),
-        None => Err((StatusCode::NOT_FOUND, "no such blob").into_response()),
-    }
-}
-
-// -------- config plane -----------------------------------------------------
-//
-// Generic, schema-driven per-plugin config. The page + the schema/values JSON
-// are all served relative to the request path, so this works identically when
-// hit directly on the feeder or reverse-proxied through the gateway.
-
-/// `GET /config` — the schema-driven HTML form (static, plugin-agnostic).
-async fn config_page() -> Html<&'static str> {
-    Html(CONFIG_PAGE_HTML)
-}
-
-/// `GET /config/schema` — the single hosted plugin's declared config schema.
-/// (A feeder hosts one plugin in practice; if it hosts several, the first by
-/// id is used — the gateway proxies per upstream_id anyway.)
-async fn config_schema(State(state): State<AppState>) -> Json<config::ConfigSchema> {
-    let schema = sole_plugin(&state)
-        .map(|p| p.config_schema())
-        .unwrap_or_default();
-    Json(schema)
-}
-
-/// `GET /config/values` — current values with secrets redacted to `<key>_set`.
-async fn config_values_get(State(state): State<AppState>) -> Json<Value> {
-    match sole_plugin(&state) {
-        Some(p) => {
-            let schema = p.config_schema();
-            let eff = state.effective_config(p.upstream_id());
-            Json(config::redact(&eff, &schema))
-        }
-        None => Json(serde_json::json!({})),
-    }
-}
-
-/// `PUT /config/values` — merge the submitted values onto the stored config
-/// (schema-aware; blank secrets keep their stored value), persist atomically,
-/// and signal that a feeder restart is needed to apply them.
-async fn config_values_put(
-    State(state): State<AppState>,
-    Json(incoming): Json<Value>,
-) -> Result<Json<Value>, Response> {
-    let plugin = sole_plugin(&state)
-        .ok_or_else(|| (StatusCode::NOT_FOUND, "feeder hosts no plugin").into_response())?;
-    let id = plugin.upstream_id();
-    let schema = plugin.config_schema();
-    let base = state.effective_config(id);
-    let merged = config::merge(&base, &incoming, &schema);
-
-    let path = state.config_path(id);
-    if let Some(dir) = path.parent() {
-        std::fs::create_dir_all(dir)
-            .map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, format!("mkdir: {e}")).into_response())?;
-    }
-    let tmp = path.with_extension("json.tmp");
-    let body = serde_json::to_vec_pretty(&merged)
-        .map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, format!("serialize: {e}")).into_response())?;
-    std::fs::write(&tmp, body)
-        .and_then(|_| std::fs::rename(&tmp, &path))
-        .map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, format!("write config: {e}")).into_response())?;
-    info!(target: "meta-feeder", upstream_id = id, path = %path.display(), "feeder config saved; scheduling self-restart to apply");
-    // Apply the new config by restarting the process. We can't hot-swap config
-    // in-place: plugins hold an exclusive redb cache handle for their whole life,
-    // so a fresh instance can't reopen the cache while the old one is alive. A
-    // clean process exit is the reliable apply path — Docker's restart policy
-    // (`unless-stopped`) brings the feeder back in ~2-3s and `configure()`
-    // re-reads config.json at startup. Delay briefly so THIS response flushes to
-    // the client before we exit.
-    tokio::spawn(async {
-        tokio::time::sleep(std::time::Duration::from_millis(800)).await;
-        info!(target: "meta-feeder", "restarting now to apply saved config");
-        std::process::exit(0);
-    });
-    Ok(Json(serde_json::json!({ "saved": true, "reloading": true })))
-}
-
-/// `POST /enrich/callback` — sink for the enrichment plugins' completion
-/// callbacks. The feeder drives those plugins (filename-parser, tmdb) during a
-/// self-published resolve and reads results back by polling meta-core, so the
-/// callback carries no load — this route exists only so the plugins' callback
-/// POST gets a 200 instead of logging a connection failure. Body is ignored.
-async fn enrich_callback() -> StatusCode {
-    StatusCode::OK
-}
-
-/// A feeder hosts one plugin in the common case. Pick it (lowest id for
-/// determinism if several are present).
-fn sole_plugin(state: &AppState) -> Option<&Arc<dyn FeederPlugin>> {
-    state
-        .plugins
-        .values()
-        .min_by(|a, b| a.upstream_id().cmp(b.upstream_id()))
-}
-
-// -------- entry point ------------------------------------------------------
-
-/// Build the feeder router for an already-configured plugin set. Exposed for
-/// in-process tests (drive it without binding a socket).
-pub fn router(
-    plugins: HashMap<String, Arc<dyn FeederPlugin>>,
-    version: String,
-    state_dir: impl Into<PathBuf>,
-) -> Router {
-    let state = AppState {
-        plugins: Arc::new(plugins),
-        version,
-        state_dir: state_dir.into(),
-    };
-    Router::new()
-        .route("/manifest", get(manifest))
-        .route("/health", get(health))
-        .route("/query", post(query))
-        .route("/query_stream", post(query_stream))
-        .route("/compute", post(compute))
-        .route("/fetch/:upstream_id/:record_id", get(fetch))
-        .route("/blob/:upstream_id/:cid", get(blob))
-        .route("/config", get(config_page))
-        .route("/config/schema", get(config_schema))
-        .route("/config/values", get(config_values_get).put(config_values_put))
-        .route("/enrich/callback", post(enrich_callback))
-        .with_state(state)
-}
-
-/// Configure each plugin (cache dir under `state_dir/gateway/<id>`), then serve
-/// the feeder HTTP contract on `listen` until shutdown.
-///
-/// The feeder process never aborts on a config problem: a plugin whose
-/// `configure()` returns [`ConfigError::MissingConfig`] is **soft-skipped**
-/// (warn + dropped — an optional source the operator didn't ask for), and any
-/// other failure (bad/empty key, cache-dir setup, …) registers a
-/// [`DegradedPlugin`] that stays up and reports the reason via `/health` and
-/// `/manifest`. The container keeps running so the error is visible in the UX
-/// instead of crash-looping.
-pub async fn serve_feeders(
-    plugins: Vec<Box<dyn FeederPlugin>>,
-    state_dir: impl Into<PathBuf>,
-    listen: SocketAddr,
-) -> anyhow::Result<()> {
-    let state_dir = state_dir.into();
-    let configured = configure_plugins(plugins, &state_dir)?;
-    let version = env!("CARGO_PKG_VERSION").to_string();
-    let app = router(configured, version, state_dir.clone());
-
-    let listener = tokio::net::TcpListener::bind(listen).await?;
-    info!(
-        target: "meta-feeder",
-        %listen,
-        "feeder listening"
-    );
-    axum::serve(listener, app)
-        .await
-        .map_err(anyhow::Error::from)
-}
-
-/// A plugin whose `configure()` (or cache-dir setup) failed. It stays
-/// **registered** so the failure is visible at `GET /health` (as
-/// [`PluginHealth::Degraded`]) and `GET /manifest`; any query/compute against it
-/// returns the captured reason as a permanent error. This is what keeps one
-/// misconfigured feeder from crash-looping the whole container — the operator
-/// sees the reason in the UX and fixes the config, no restart-loop required.
-struct DegradedPlugin {
-    id: &'static str,
-    reason: String,
-    file_types: &'static [&'static str],
-    content_kinds: &'static [&'static str],
-    /// Preserved from the wrapped plugin so a degraded/unconfigured feeder still
-    /// serves its config form — otherwise the dashboard shows "no configuration
-    /// required" and the operator can never fix the very config that's missing.
-    schema: crate::config::ConfigSchema,
-}
-
-#[async_trait::async_trait]
-impl FeederPlugin for DegradedPlugin {
-    fn upstream_id(&self) -> &'static str {
-        self.id
-    }
-
-    fn configure(&mut self, _cache_dir: &FsPath) -> Result<(), ConfigError> {
-        Ok(())
-    }
-
-    async fn handle_query(
-        &self,
-        _query: &GatewayQuery,
-        _max_results: usize,
-    ) -> Result<Vec<DiscoveryRecord>, GatewayError> {
-        Err(GatewayError::Permanent(self.reason.clone()))
-    }
-
-    async fn compute_outcomes(
-        &self,
-        _record_id: &str,
-    ) -> Result<Vec<HashOutcome>, GatewayError> {
-        Err(GatewayError::Permanent(self.reason.clone()))
-    }
-
-    fn health(&self) -> PluginHealth {
-        PluginHealth::Degraded {
-            reason: self.reason.clone(),
-        }
-    }
-
-    fn served_file_types(&self) -> &'static [&'static str] {
-        self.file_types
-    }
-
-    fn served_content_kinds(&self) -> &'static [&'static str] {
-        self.content_kinds
-    }
-
-    fn config_schema(&self) -> crate::config::ConfigSchema {
-        self.schema.clone()
-    }
-}
-
-/// Run each plugin's `configure()` against its per-plugin cache dir and return
-/// the plugins keyed by `upstream_id`. Never errors on a per-plugin problem:
-/// `MissingConfig` is soft-skipped, every other failure becomes a
-/// [`DegradedPlugin`] (see [`serve_feeders`]). Public so feeder binaries /
-/// tests can configure without serving.
-pub fn configure_plugins(
-    plugins: Vec<Box<dyn FeederPlugin>>,
-    state_dir: &FsPath,
-) -> anyhow::Result<HashMap<String, Arc<dyn FeederPlugin>>> {
-    let mut out: HashMap<String, Arc<dyn FeederPlugin>> = HashMap::new();
-    for mut plugin in plugins {
-        let id = plugin.upstream_id();
-        // Captured up front (both are `'static`, so this borrow of `plugin`
-        // ends immediately) — lets us build a DegradedPlugin without holding a
-        // borrow across `configure(&mut self)` / the move into `out`.
-        let file_types = plugin.served_file_types();
-        let content_kinds = plugin.served_content_kinds();
-        // Preserve the config surface so a degraded stand-in still serves the
-        // real config form (see DegradedPlugin.schema).
-        let schema = plugin.config_schema();
-        let cache_dir = state_dir.join("gateway").join(id);
-
-        // Build (don't insert) a degraded stand-in: keeps the container up and
-        // surfaces the reason via /health instead of crash-looping.
-        let make_degraded = |reason: String| -> Arc<dyn FeederPlugin> {
-            warn!(
-                target: "meta-feeder",
-                upstream_id = id,
-                %reason,
-                "feeder plugin degraded: container stays up; error surfaced via /health"
-            );
-            Arc::new(DegradedPlugin {
-                id,
-                reason,
-                file_types,
-                content_kinds,
-                schema: schema.clone(),
-            })
-        };
-
-        if let Err(e) = std::fs::create_dir_all(&cache_dir) {
-            let reason = format!("cache dir {} setup failed: {e}", cache_dir.display());
-            out.insert(id.to_string(), make_degraded(reason));
-            continue;
-        }
-
-        match plugin.configure(&cache_dir) {
-            Ok(()) => {
-                out.insert(id.to_string(), Arc::from(plugin));
-            }
-            Err(ConfigError::MissingConfig { plugin: p, what }) => {
-                // Keep a degraded stand-in (don't drop it) so its config form
-                // stays reachable from the dashboard and the operator can supply
-                // the missing config — then restart the feeder to apply it.
-                warn!(
-                    target: "meta-feeder",
-                    upstream_id = %p,
-                    missing = %what,
-                    "feeder plugin degraded: required config not supplied; \
-                     config form still served so it can be fixed from the dashboard"
-                );
-                out.insert(
-                    id.to_string(),
-                    make_degraded(format!("required config not supplied: {what}")),
-                );
-            }
-            Err(other) => {
-                out.insert(id.to_string(), make_degraded(other.to_string()));
-            }
-        }
-    }
-    Ok(out)
-}
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-    use crate::types::DiscoveryRecord;
-    use async_trait::async_trait;
-    use futures::stream::BoxStream;
-    use std::collections::BTreeMap;
-
-    /// A feeder plugin that overrides `handle_query_stream` to emit a Base, then
-    /// an EnrichPatch, then Done — so the NDJSON route is exercised on a real
-    /// incremental stream (not just the collect-default).
-    struct StubStreaming;
-
-    #[async_trait]
-    impl FeederPlugin for StubStreaming {
-        fn upstream_id(&self) -> &'static str {
-            "stub"
-        }
-        fn configure(&mut self, _cache_dir: &FsPath) -> Result<(), ConfigError> {
-            Ok(())
-        }
-        async fn handle_query(
-            &self,
-            _q: &GatewayQuery,
-            _n: usize,
-        ) -> Result<Vec<DiscoveryRecord>, GatewayError> {
-            Ok(vec![])
-        }
-        async fn handle_query_stream(
-            &self,
-            _q: &GatewayQuery,
-            _n: usize,
-        ) -> Result<BoxStream<'static, GatewaySearchEvent>, GatewayError> {
-            let rec = DiscoveryRecord {
-                upstream_id: "stub".into(),
-                record_id: "1".into(),
-                fields: BTreeMap::new(),
-            };
-            let events = vec![
-                GatewaySearchEvent::Base(rec),
-                GatewaySearchEvent::EnrichPatch {
-                    record_id: "1".into(),
-                    set: BTreeMap::from([("title".to_string(), "X".to_string())]),
-                    remove: vec![],
-                },
-                GatewaySearchEvent::Done,
-            ];
-            Ok(Box::pin(futures::stream::iter(events)))
-        }
-        async fn compute_outcomes(
-            &self,
-            _record_id: &str,
-        ) -> Result<Vec<HashOutcome>, GatewayError> {
-            Ok(vec![])
-        }
-    }
-
-    #[tokio::test]
-    async fn query_stream_emits_ndjson_events_in_order() {
-        let mut plugins: HashMap<String, Arc<dyn FeederPlugin>> = HashMap::new();
-        plugins.insert("stub".to_string(), Arc::new(StubStreaming));
-        let app = router(plugins, "test".to_string(), std::env::temp_dir());
-
-        let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
-        let addr = listener.local_addr().unwrap();
-        tokio::spawn(async move {
-            axum::serve(listener, app).await.unwrap();
-        });
-
-        let body = reqwest::Client::new()
-            .post(format!("http://{addr}/query_stream"))
-            .json(&serde_json::json!({
-                "upstream_id": "stub",
-                "query": {"raw_text":"x","free_text":"x","filters":{},"ranges":[],"negations":[]},
-                "max_results": 5
-            }))
-            .send()
-            .await
-            .unwrap()
-            .text()
-            .await
-            .unwrap();
-
-        let events: Vec<GatewaySearchEvent> = body
-            .lines()
-            .filter(|l| !l.trim().is_empty())
-            .map(|l| serde_json::from_str(l).expect("each NDJSON line is a GatewaySearchEvent"))
-            .collect();
-
-        assert_eq!(events.len(), 3, "body was: {body:?}");
-        assert!(matches!(events[0], GatewaySearchEvent::Base(_)));
-        assert!(matches!(events[1], GatewaySearchEvent::EnrichPatch { .. }));
-        assert!(matches!(events[2], GatewaySearchEvent::Done));
-    }
-}

crates/meta-feeder-sdk/src/types.rs

@@ -1,124 +0,0 @@
-//! In-process types used by the gateway runtime, feeders, and plugins.
-//!
-//! `GatewayError` here is **distinct from** the wire-stable
-//! `GatewayWireError` (see [`crate::query`]): this one can carry rich
-//! internal context (anyhow chains, plugin specifics); the wire one is narrow
-//! on purpose since adding variants is a protocol change. Conversion happens
-//! at the protocol boundary.
-
-use std::collections::BTreeMap;
-
-use serde::{Deserialize, Serialize};
-
-/// Content-addressed hash string carried over the wire. Renamed from
-/// `Midhash` in the multi-outcome plugin contract migration — the name
-/// `Midhash` lied for sha2-256 CIDs (which gateway plugins also emit
-/// via `compute_ipfs_cid`). The carrier is generic; the hash family is
-/// discriminated by `HashKind` on `HashOutcome`.
-///
-/// Shape: base32-lower 'b'-prefixed multibase CIDv1. The specific
-/// multihash inside (`0x1000` for midhash256, `0x12` for sha2-256) is
-/// determined by the producing plugin's hashing path.
-#[derive(Clone, Debug, PartialEq, Eq, Hash, Serialize, Deserialize)]
-pub struct Hash(pub String);
-
-impl Hash {
-    pub fn as_str(&self) -> &str {
-        &self.0
-    }
-}
-
-impl std::fmt::Display for Hash {
-    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
-        f.write_str(&self.0)
-    }
-}
-
-/// Pre-hash identifier for a record discovered through a gateway.
-#[derive(Clone, Debug, PartialEq, Eq, Hash, Serialize, Deserialize)]
-pub struct DiscoveryId {
-    pub upstream_id: String,
-    pub record_id: String,
-}
-
-/// Metadata-only search result. Distinct from meta-core's persistent record
-/// (which is a flat `map[string]string` indexed by midhash); a
-/// `DiscoveryRecord` only lives inside the gateway search/response pipeline.
-#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
-pub struct DiscoveryRecord {
-    pub upstream_id: String,
-    pub record_id: String,
-    pub fields: BTreeMap<String, String>,
-}
-
-/// In-process error returned by `FeederPlugin` methods.
-#[derive(Debug, thiserror::Error)]
-pub enum GatewayError {
-    #[error("upstream transient error: {0}")]
-    Transient(String),
-
-    #[error("upstream permanent error: {0}")]
-    Permanent(String),
-
-    #[error("rate limited; retry in {retry_after_s}s")]
-    RateLimited { retry_after_s: u32 },
-
-    #[error("record not found upstream")]
-    NotFound,
-
-    #[error("internal: {0}")]
-    Internal(#[from] anyhow::Error),
-}
-
-/// Byte stream returned by a plugin's `handle_fetch`.
-pub type ByteStream = futures::stream::BoxStream<'static, Result<bytes::Bytes, GatewayError>>;
-
-/// Plugin liveness snapshot surfaced via `/health`. Must not call upstream
-/// — `health()` is invoked on every status poll.
-#[derive(Clone, Debug, Serialize, Deserialize)]
-#[serde(rename_all = "snake_case", tag = "state")]
-pub enum PluginHealth {
-    Ok,
-    Degraded { reason: String },
-}
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-
-    #[test]
-    fn hash_display_and_as_str_match() {
-        let m = Hash("bafy123".to_string());
-        assert_eq!(m.as_str(), "bafy123");
-        assert_eq!(format!("{m}"), "bafy123");
-    }
-
-    #[test]
-    fn discovery_record_roundtrips_through_serde_json() {
-        let r = DiscoveryRecord {
-            upstream_id: "scihub".to_string(),
-            record_id: "10.1038/s41586-021-03819-2".to_string(),
-            fields: BTreeMap::from_iter([
-                ("title".to_string(), "AlphaFold".to_string()),
-                ("year".to_string(), "2021".to_string()),
-            ]),
-        };
-        let j = serde_json::to_string(&r).expect("serialize");
-        let back: DiscoveryRecord = serde_json::from_str(&j).expect("deserialize");
-        assert_eq!(r, back);
-    }
-
-    #[test]
-    fn plugin_health_serialises_with_tag() {
-        let ok = serde_json::to_value(PluginHealth::Ok).unwrap();
-        assert_eq!(ok, serde_json::json!({"state": "ok"}));
-        let bad = serde_json::to_value(PluginHealth::Degraded {
-            reason: "upstream 5xx".into(),
-        })
-        .unwrap();
-        assert_eq!(
-            bad,
-            serde_json::json!({"state": "degraded", "reason": "upstream 5xx"})
-        );
-    }
-}

feeder-plugin/usenet-feeder/Cargo.toml

@@ -9,13 +9,15 @@ publish = false
 # Usenet feeder sidecar: bridges a Usenet header scan we run OURSELVES into the
 # MetaMesh network. Unlike `meta-feeder-indexer` (which proxies an external
 # Newznab/torznab indexer) this feeder owns the metadata half — it reads an
-# nntmux sidecar's release catalog and mints a PORTABLE `nzb-posting` (0x1003)
+# nntmux indexer (the NNTmux app, over its Newznab API) and mints a PORTABLE `nzb-posting` (0x1003)
 # cid from the article Message-IDs, so any peer with a plain NNTP provider can
 # redeem it without an indexer credential.
 #
 # It ADDS a Usenet metadata source alongside meta-feeder-indexer; it does not
-# replace it. Like every feeder it is meta-core-free + blockstore-free (the
+# replace it. It also REDEEMS external `nzb-release` (0x1005) locators through
-# gateway core owns hashing/seeding/store-back — see the gateway's invariant 10).
+# `/compute`, grabbing the .nzb with per-host indexer keys set on its config
+# page. Like every feeder it is meta-core-free + blockstore-free (the gateway
+# core owns hashing/seeding/store-back — see the gateway's invariant 10).
 #
 # See `meta-gateway/docs/others/self-hosted-usenet-indexer-study.md`.
 
@@ -28,7 +30,9 @@ name = "usenet-feeder"
 path = "src/main.rs"
 
 [dependencies]
-meta-feeder-sdk = { path = "../../crates/meta-feeder-sdk" }
+# Pinned by TAG, never by path. A path dep resolves OUTSIDE this repo, and this
+# repo is the Docker build context. (Until 1.3.0 this repo vendored a copy.)
+meta-feeder-sdk = { git = "https://github.com/worph/meta-feeder-sdk", tag = "v1.7.0" }
 
 tokio = { version = "1.40", features = ["full"] }
 axum = { version = "0.7", features = ["json"] }
@@ -45,21 +49,10 @@ tracing = "0.1"
 tracing-subscriber = { version = "0.3", features = ["env-filter", "fmt"] }
 chrono = { version = "0.4", default-features = false, features = ["std", "clock"] }
 
-# nntmux's release catalog lives in MariaDB. `sqlx` in offline-free runtime mode
-# (no compile-time macro checking) so the build needs no live DB.
-sqlx = { version = "0.8", default-features = false, features = [
-    "runtime-tokio-rustls",
-    "mysql",
-    "macros",
-    "chrono",
-] }
-# nntmux stores each release's manifest as a gzipped NZB on disk; the ordered
-# Message-IDs survive ONLY there (it purges collections/binaries/parts after
-# writing the file), so reading + gunzipping it is the load-bearing path.
-flate2 = "1"
 quick-xml = { version = "0.36", features = ["serialize"] }
 
 [dev-dependencies]
 wiremock = "0.6"
 tempfile = "3"
+base64 = "0.22"
 tokio = { version = "1.40", features = ["full", "test-util"] }

feeder-plugin/usenet-feeder/Dockerfile

@@ -3,9 +3,10 @@
 # usenet-feeder — Usenet feeder sidecar over an nntmux header-scan catalog.
 # Multi-stage build.
 #
-# Build context must be this repo root (packages/plugins/meta-feeder-usenet) so
+# Build context must be this repo root (packages/plugins/meta-feeder-usenet).
-# the `meta-feeder-sdk` path dependency resolves. Same BuildKit cache-mount
+# `meta-feeder-sdk` is a git dependency pinned by tag, so the builder needs
-# strategy as the other feeders.
+# network for `cargo fetch`. Same BuildKit cache-mount strategy as the other
+# feeders.
 #
 # This feeder talks to MariaDB (sqlx, rustls — no system MySQL client needed)
 # and reads nntmux's NZB store off a shared volume. It holds no media bytes.
@@ -22,7 +23,6 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
     && rm -rf /var/lib/apt/lists/*
 
 COPY Cargo.toml rust-toolchain.toml ./
-COPY crates/ crates/
 COPY feeder-plugin/ feeder-plugin/
 
 RUN --mount=type=cache,id=cargo-registry,target=/usr/local/cargo/registry \

feeder-plugin/usenet-feeder/src/lib.rs

@@ -1,26 +1,37 @@
 //! `usenet-feeder` library surface. The binary (`main.rs`) wraps the plugin in
 //! [`meta_feeder_sdk::serve_feeders`].
 //!
-//! This feeder owns the **metadata half** of Usenet: it reads an nntmux header-
+//! This feeder owns the **metadata half** of Usenet: it searches an nntmux
-//! scan catalog we run ourselves and mints a portable `nzb-posting` (`0x1003`)
+//! header-scan indexer we run ourselves (the separate NNTmux app, over its
+//! Newznab API) and mints a portable `nzb-posting` (`0x1003`)
 //! cid from each release's article Message-IDs. Unlike `meta-feeder-indexer`'s
 //! `0x1005` locator, that cid embeds no indexer host, so any peer with a plain
 //! NNTP provider can redeem it — no indexer credential anywhere in the path.
 //!
 //! It ADDS a source alongside meta-feeder-indexer; it replaces nothing.
 //!
+//! It also **redeems** external Newznab releases: an `nzb-release` (`0x1005`)
+//! cid minted by meta-feeder-torznab is grabbed here, with this feeder's
+//! per-host indexer keys, when the gateway calls `/compute` on a real play
+//! ([`newznab`]). Indexer keys live on this feeder, not on meta-share.
+//!
 //! Like every feeder it is meta-core-free and blockstore-free: it finds records
 //! and serves bytes, and the gateway core owns hashing-into-the-blockstore, the
 //! meta-core store-back, and the libp2p wire (gateway invariant 10).
 //!
 //! See `meta-gateway/docs/others/self-hosted-usenet-indexer-study.md`.
 
-/// Read-only access to the nntmux sidecar: its MariaDB release catalog and the
+/// Our own nntmux, reached only over its Newznab HTTP API: the client, the
-/// on-disk gzip NZBs that are the *only* durable home of the Message-IDs.
+/// per-release `.nzb` cache, the release type and the Message-ID extraction.
 pub mod nntmux {
-    pub mod db;
+    pub mod api;
+    pub mod cache;
+    pub mod catalog;
     pub mod nzb;
 }
 
+/// The `t=get` grab that redeems an external `nzb-release` locator.
+pub mod newznab;
+
 /// The `usenet` upstream itself.
 pub mod usenet;

feeder-plugin/usenet-feeder/src/main.rs

@@ -1,14 +1,16 @@
-//! `usenet-feeder` — single-source Usenet feeder sidecar over an nntmux
+//! `usenet-feeder` — single-source Usenet feeder sidecar over our own nntmux
-//! header-scan catalog.
+//! indexer (the NNTmux app), reached over its Newznab API.
 //!
 //! One plugin (`usenet`) served through [`meta_feeder_sdk::serve_feeders`].
 //!
-//! Operator config — the nntmux database URL and NZB store path — is read from
+//! Operator config — the nntmux base URL and API key, and the
-//! the persisted `config.json` (dashboard-written), with the env vars below as a
+//! per-host Newznab indexer keys used to redeem `nzb-release` locators — is read
-//! **first-boot seed only**: the file wins on the next restart, and there is no
+//! from the persisted `config.json` (dashboard-written), with the env vars below
-//! hot reload (gateway invariant 12).
+//! as a **first-boot seed only**: the file wins on the next restart, and there
+//! is no hot reload (gateway invariant 12). Indexer keys have no env seed; set
+//! them on the config page.
 //!
-//! Env (seed): `NNTMUX_DB_URL`, `NNTMUX_NZB_PATH`.
+//! Env (seed): `NNTMUX_URL` (default `http://nntmux`), `NNTMUX_API_KEY`.
 //! Env (infra): `META_FEEDER_HTTP_LISTEN` (default `0.0.0.0:8080`),
 //! `META_FEEDER_STATE_DIR` (default `/data/meta-feeder`), `RUST_LOG`.
 

feeder-plugin/usenet-feeder/src/newznab.rs

@@ -0,0 +1,430 @@
+//! Redeeming an **external** Newznab release — an `nzb-release` (`0x1005`)
+//! locator minted by `meta-feeder-torznab` from a Prowlarr/Newznab search hit.
+//!
+//! The cid embeds the indexer's API base and the release id; redeeming it is a
+//! `t=get` grab against that indexer with *this feeder's* key for that host.
+//! The grab is **metered** by the indexer (a per-account daily download cap), so
+//! it only ever runs from `compute_outcomes`, which the gateway calls on a real
+//! play — never at search time, never speculatively.
+//!
+//! Moved here from meta-share's `nzb/manifest_resolve.rs::grab_newznab`: the
+//! indexer keys now live on this feeder, meta-share keeps only the NNTP pool.
+//!
+//! The error handling ([`fetch_nzb`], [`newznab_error`], [`classify`]) is shared
+//! with the client for our own nntmux ([`crate::nntmux::api`]), which speaks the
+//! same Newznab dialect.
+
+use std::borrow::Cow;
+
+use anyhow::{anyhow, Context};
+use meta_feeder_sdk::types::GatewayError;
+use serde::{Deserialize, Serialize};
+
+/// `retry_after_s` handed back when an indexer reports its request/download cap
+/// is spent. Newznab carries no reset time, and the caps are daily.
+pub const QUOTA_RETRY_SECS: u32 = 3600;
+
+/// Refuse a `.nzb` bigger than this. A 1329-segment posting is ~200 KB; 16 MiB
+/// is a ~100k-segment monster and still parseable. Same bound meta-share uses.
+pub const MAX_NZB_BYTES: usize = 16 * 1024 * 1024;
+
+/// One configured indexer key. `host` is the bare authority the key is for
+/// (`api.nzbgeek.info`); the API path comes from the cid, not from here.
+#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
+pub struct IndexerCred {
+    #[serde(default)]
+    pub host: String,
+    #[serde(default)]
+    pub api_key: String,
+}
+
+/// Normalise an indexer host for matching: drop any `http(s)://` scheme, trim a
+/// trailing slash and surrounding whitespace, lowercase. Both the configured
+/// host and the cid-decoded authority go through this before an equality
+/// compare, so an operator may enter `api.example.com` or `https://api.example.com/`.
+pub fn normalize_indexer_host(raw: &str) -> String {
+    raw.trim()
+        .trim_start_matches("https://")
+        .trim_start_matches("http://")
+        .trim_end_matches('/')
+        .to_ascii_lowercase()
+}
+
+/// The credential-matching key for anything host-shaped an operator or a
+/// locator supplies: normalised (scheme, trailing slash, case), then reduced to
+/// its authority — `https://API.nzbgeek.info/api/` and `api.nzbgeek.info/api`
+/// both become `api.nzbgeek.info`.
+pub fn host_key(raw: &str) -> String {
+    authority(&normalize_indexer_host(raw)).to_string()
+}
+
+/// The authority part of a scheme-less API base (`api.nzbgeek.info/api` →
+/// `api.nzbgeek.info`). This — never the full base — is the credential key:
+/// operators configure plain hosts, and the locator may carry a path.
+pub fn authority(api_base: &str) -> &str {
+    match api_base.split_once('/') {
+        Some((authority, _path)) => authority,
+        None => api_base,
+    }
+}
+
+/// A Newznab `<error code=".." description=".."/>` answer to `t=get`.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct NewznabError {
+    pub code: Option<u32>,
+    pub description: Option<String>,
+}
+
+impl std::fmt::Display for NewznabError {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        match (&self.code, &self.description) {
+            (Some(c), Some(d)) => write!(f, "code {c}: {d}"),
+            (None, Some(d)) => write!(f, "{d}"),
+            (Some(c), None) => write!(f, "code {c}"),
+            (None, None) => write!(f, "unspecified error"),
+        }
+    }
+}
+
+/// Pull a Newznab `<error>` out of a `t=get` response, if that is what came
+/// back instead of an `.nzb`.
+///
+/// Deliberately a substring scrape rather than a parse: the body is tiny, the
+/// shape is fixed across indexers, and the point is to turn a silent
+/// zero-file manifest into a readable reason. Only the head is scanned, so an
+/// `<error>` deep inside a large legitimate NZB is never mistaken for a refusal.
+pub fn newznab_error(xml: &str) -> Option<NewznabError> {
+    let head = &xml[..floor_char_boundary(xml, 1024)];
+    let at = head.find("<error ")?;
+    let rest = &head[at..];
+    let end = rest.find("/>").or_else(|| rest.find('>'))? + 1;
+    let tag = &rest[..end];
+    let grab = |k: &str| -> Option<&str> {
+        let i = tag.find(k)? + k.len();
+        let t = &tag[i..];
+        let q = t.find('"')? + 1;
+        let e = t[q..].find('"')? + q;
+        Some(&t[q..e])
+    };
+    Some(NewznabError {
+        code: grab("code=").and_then(|c| c.trim().parse().ok()),
+        description: grab("description=").map(str::to_string),
+    })
+}
+
+fn floor_char_boundary(s: &str, max: usize) -> usize {
+    let mut i = s.len().min(max);
+    while !s.is_char_boundary(i) {
+        i -= 1;
+    }
+    i
+}
+
+/// Map an indexer's refusal onto the feeder error contract. The Newznab API
+/// spec's codes: `100`–`102` credentials/account, `300` no such item, `429` /
+/// `500` request limit, `501` download limit, `910` API disabled.
+pub fn classify(err: &NewznabError) -> GatewayError {
+    match err.code {
+        Some(300) => GatewayError::NotFound,
+        Some(429) | Some(500) | Some(501) => GatewayError::RateLimited {
+            retry_after_s: QUOTA_RETRY_SECS,
+        },
+        Some(100..=102) | Some(910) => GatewayError::Permanent(format!(
+            "indexer refused the .nzb grab ({err}) — check this host's API key on the feeder config page"
+        )),
+        _ => GatewayError::Permanent(format!("indexer refused the .nzb grab ({err})")),
+    }
+}
+
+/// True when an indexer answered a grab with an HTML page — a login wall, a
+/// Cloudflare challenge, or a permalink host that 302s to its website
+/// (`https://nzbgeek.info/api?t=get` does). Without this the grab dies later as
+/// an unhelpful "no segments".
+pub fn looks_like_html(body: &[u8]) -> bool {
+    let head = &body[..body.len().min(512)];
+    let text = String::from_utf8_lossy(head);
+    let t = text.trim_start_matches('\u{feff}').trim_start().to_ascii_lowercase();
+    t.starts_with("<!doctype html") || t.starts_with("<html")
+}
+
+/// Grab a release's `.nzb` from `{scheme}://{api_base}/api?t=get`.
+///
+/// Errors never echo the request URL — it carries the API key.
+pub async fn grab(
+    http: &reqwest::Client,
+    scheme: &str,
+    api_base: &str,
+    id: &str,
+    api_key: &str,
+) -> Result<Vec<u8>, GatewayError> {
+    let base = api_base.trim_end_matches('/');
+    let url = format!("{scheme}://{base}/api?t=get&id={id}&apikey={api_key}");
+    fetch_nzb(http, &url, base).await
+}
+
+/// GET an `.nzb` and turn every way a Newznab server can refuse into the feeder
+/// error contract: HTTP status, an HTML page (login wall, Cloudflare, a
+/// permalink host), an oversize body, or a `<error code>` answered as `200`.
+///
+/// `label` names the server in error messages; `url` is never echoed (it
+/// carries the API key). Shared by the external-indexer redeem ([`grab`]) and
+/// the nntmux client ([`crate::nntmux::api`]).
+pub async fn fetch_nzb(http: &reqwest::Client, url: &str, label: &str) -> Result<Vec<u8>, GatewayError> {
+    let resp = http
+        .get(url)
+        .send()
+        .await
+        .map_err(|e| GatewayError::Transient(format!("newznab grab {label}: {}", e.without_url())))?;
+
+    let status = resp.status();
+    if !status.is_success() {
+        return Err(status_error(&resp, label));
+    }
+
+    let body = resp
+        .bytes()
+        .await
+        .map_err(|e| GatewayError::Transient(format!("newznab grab {label}: {}", e.without_url())))?;
+    if body.len() > MAX_NZB_BYTES {
+        return Err(GatewayError::Permanent(format!(
+            "newznab grab {label}: .nzb is {} bytes, over the {MAX_NZB_BYTES}-byte ceiling",
+            body.len()
+        )));
+    }
+    if looks_like_html(&body) {
+        return Err(GatewayError::Permanent(format!(
+            "indexer {label} answered the .nzb grab with an HTML page — check the API base and key"
+        )));
+    }
+    let text: Cow<str> = String::from_utf8_lossy(&body);
+    if let Some(err) = newznab_error(&text) {
+        return Err(classify(&err));
+    }
+    Ok(text.into_owned().into_bytes())
+}
+
+/// The feeder error for a non-2xx Newznab answer.
+pub fn status_error(resp: &reqwest::Response, label: &str) -> GatewayError {
+    let status = resp.status();
+    match status.as_u16() {
+        404 => GatewayError::NotFound,
+        429 => GatewayError::RateLimited {
+            retry_after_s: resp
+                .headers()
+                .get(reqwest::header::RETRY_AFTER)
+                .and_then(|v| v.to_str().ok())
+                .and_then(|v| v.parse().ok())
+                .unwrap_or(QUOTA_RETRY_SECS),
+        },
+        _ if status.is_server_error() => {
+            GatewayError::Transient(format!("newznab {label}: HTTP {status}"))
+        }
+        _ => GatewayError::Permanent(format!("newznab {label}: HTTP {status}")),
+    }
+}
+
+/// Drop the NZB `<head>` except `<meta type="password">`.
+///
+/// The grabbed file is seeded to the whole mesh, and indexers put
+/// account-identifying bits in `<head>` metadata. The password must survive:
+/// meta-share needs it to unpack a protected RAR set (`NzbManifest.password`).
+/// `<file>` / `<segment>` content is copied byte-for-byte, so the Message-IDs —
+/// and the `nzb-posting` cid minted from them — are unchanged. Output is UTF-8.
+pub fn strip_head_keep_password(xml: &[u8]) -> anyhow::Result<Vec<u8>> {
+    use quick_xml::events::{BytesEnd, BytesStart, Event};
+    use quick_xml::{Reader, Writer};
+
+    let text = String::from_utf8_lossy(xml);
+    let mut reader = Reader::from_str(&text);
+    let mut writer = Writer::new(Vec::with_capacity(xml.len()));
+
+    // Nesting depth inside <head> (0 = outside), and inside a kept password meta.
+    let mut head_depth = 0usize;
+    let mut password_depth = 0usize;
+    let mut head_start: Option<BytesStart<'static>> = None;
+    let mut kept: Vec<Event<'static>> = Vec::new();
+
+    loop {
+        let ev = reader.read_event().map_err(|e| anyhow!("nzb xml: {e}"))?;
+        if matches!(ev, Event::Eof) {
+            break;
+        }
+        if head_depth == 0 {
+            match &ev {
+                Event::Start(e) if e.local_name().as_ref() == b"head" => {
+                    head_depth = 1;
+                    head_start = Some(e.clone().into_owned());
+                    kept.clear();
+                }
+                Event::Empty(e) if e.local_name().as_ref() == b"head" => {}
+                _ => writer
+                    .write_event(ev)
+                    .map_err(|e| anyhow!("nzb xml write: {e}"))?,
+            }
+            continue;
+        }
+
+        match &ev {
+            Event::Start(e) => {
+                if password_depth > 0 {
+                    password_depth += 1;
+                    kept.push(ev.into_owned());
+                } else if is_password_meta(e) {
+                    password_depth = 1;
+                    kept.push(ev.into_owned());
+                } else {
+                    head_depth += 1;
+                }
+            }
+            Event::End(e) => {
+                if password_depth > 0 {
+                    password_depth -= 1;
+                    kept.push(ev.into_owned());
+                } else if head_depth == 1 && e.local_name().as_ref() == b"head" {
+                    head_depth = 0;
+                    let start = head_start.take().context("head start tag")?;
+                    if !kept.is_empty() {
+                        let name = String::from_utf8_lossy(start.name().as_ref()).into_owned();
+                        writer
+                            .write_event(Event::Start(start))
+                            .map_err(|e| anyhow!("nzb xml write: {e}"))?;
+                        for k in kept.drain(..) {
+                            writer
+                                .write_event(k)
+                                .map_err(|e| anyhow!("nzb xml write: {e}"))?;
+                        }
+                        writer
+                            .write_event(Event::End(BytesEnd::new(name)))
+                            .map_err(|e| anyhow!("nzb xml write: {e}"))?;
+                    }
+                } else {
+                    head_depth -= 1;
+                }
+            }
+            Event::Empty(e) => {
+                if password_depth > 0 || is_password_meta(e) {
+                    kept.push(ev.into_owned());
+                }
+            }
+            _ => {
+                if password_depth > 0 {
+                    kept.push(ev.into_owned());
+                }
+            }
+        }
+    }
+    Ok(writer.into_inner())
+}
+
+fn is_password_meta(e: &quick_xml::events::BytesStart<'_>) -> bool {
+    e.local_name().as_ref() == b"meta"
+        && e.attributes().flatten().any(|a| {
+            a.key.local_name().as_ref() == b"type" && a.value.eq_ignore_ascii_case(b"password")
+        })
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    /// The exact body nzbgeek returns for an anonymous `t=get` — 200 OK, valid
+    /// XML, zero files. Reported as "nzb has no files/segments" until this
+    /// existed, which is a parser-shaped message for a credentials-shaped fault.
+    #[test]
+    fn reports_the_indexers_own_reason() {
+        let xml = r#"<?xml version="1.0" encoding="UTF-8"?>
+<error code="109" description="Invalid User Agent"/>"#;
+        let err = newznab_error(xml).expect("an error");
+        assert_eq!(err.to_string(), "code 109: Invalid User Agent");
+        assert!(matches!(classify(&err), GatewayError::Permanent(_)));
+    }
+
+    #[test]
+    fn a_real_nzb_is_not_an_error() {
+        let xml = r#"<?xml version="1.0"?>
+<nzb xmlns="http://www.newzbin.com/DTD/2003/nzb"><file subject="x"><segments>
+<segment number="1" bytes="10">abc@d</segment></segments></file></nzb>"#;
+        assert_eq!(newznab_error(xml), None);
+    }
+
+    #[test]
+    fn only_the_head_is_scanned() {
+        let xml = format!(
+            "<nzb>{}<error code=\"1\" description=\"late\"/></nzb>",
+            "x".repeat(2000)
+        );
+        assert_eq!(newznab_error(&xml), None);
+    }
+
+    #[test]
+    fn limit_codes_are_rate_limited_and_300_is_not_found() {
+        for code in [429, 500, 501] {
+            let err = NewznabError { code: Some(code), description: None };
+            assert!(
+                matches!(classify(&err), GatewayError::RateLimited { retry_after_s } if retry_after_s == QUOTA_RETRY_SECS),
+                "{code}"
+            );
+        }
+        let gone = NewznabError { code: Some(300), description: Some("No such item".into()) };
+        assert!(matches!(classify(&gone), GatewayError::NotFound));
+        let bad_key = NewznabError { code: Some(100), description: Some("Incorrect user credentials".into()) };
+        assert!(matches!(classify(&bad_key), GatewayError::Permanent(m) if m.contains("API key")));
+    }
+
+    #[test]
+    fn html_pages_are_detected() {
+        assert!(looks_like_html(b"<!DOCTYPE html><html><body>login</body></html>"));
+        assert!(looks_like_html(b"\n  <html lang=\"en\">"));
+        assert!(!looks_like_html(b"<?xml version=\"1.0\"?><nzb/>"));
+    }
+
+    #[test]
+    fn hosts_normalise_and_authority_drops_the_path() {
+        assert_eq!(normalize_indexer_host(" https://API.Example.com/ "), "api.example.com");
+        assert_eq!(authority("api.nzbgeek.info/api"), "api.nzbgeek.info");
+        assert_eq!(authority("127.0.0.1:8080"), "127.0.0.1:8080");
+        // Normalise BEFORE splitting, or a scheme's `//` is taken as the path.
+        assert_eq!(host_key("https://API.nzbgeek.info/api/"), "api.nzbgeek.info");
+        assert_eq!(host_key("api.nzbgeek.info/api"), "api.nzbgeek.info");
+    }
+
+    const WITH_HEAD: &str = r#"<?xml version="1.0" encoding="UTF-8"?>
+<nzb xmlns="http://www.newzbin.com/DTD/2003/nzb">
+  <head>
+    <meta type="title">Some.Release.1080p</meta>
+    <meta type="x-grabbed-by">user-12345</meta>
+    <meta type="password">s3cr&amp;t</meta>
+  </head>
+  <file poster="[email protected]" date="1" subject="a [1/1] &quot;a.mkv&quot; yEnc">
+    <groups><group>alt.binaries.teevee</group></groups>
+    <segments><segment bytes="100" number="1">[email protected]</segment></segments>
+  </file>
+</nzb>"#;
+
+    #[test]
+    fn head_strip_keeps_only_the_password() {
+        let out = String::from_utf8(strip_head_keep_password(WITH_HEAD.as_bytes()).unwrap()).unwrap();
+        assert!(out.contains(r#"<meta type="password">s3cr&amp;t</meta>"#), "{out}");
+        assert!(!out.contains("user-12345"), "{out}");
+        assert!(!out.contains("Some.Release.1080p"), "{out}");
+        assert!(out.contains("<head>") && out.contains("</head>"), "{out}");
+        // The file body is untouched.
+        assert!(out.contains(r#"<segment bytes="100" number="1">[email protected]</segment>"#), "{out}");
+        assert!(out.contains(r#"subject="a [1/1] &quot;a.mkv&quot; yEnc""#), "{out}");
+    }
+
+    #[test]
+    fn head_without_password_is_dropped_entirely() {
+        let xml = WITH_HEAD.replace(r#"<meta type="password">s3cr&amp;t</meta>"#, "");
+        let out = String::from_utf8(strip_head_keep_password(xml.as_bytes()).unwrap()).unwrap();
+        assert!(!out.contains("<head"), "{out}");
+        assert!(out.contains("[email protected]"), "{out}");
+    }
+
+    #[test]
+    fn no_head_is_a_no_op_on_content() {
+        let xml = r#"<nzb><file subject="s"><segments><segment number="1">id@x</segment></segments></file></nzb>"#;
+        let out = strip_head_keep_password(xml.as_bytes()).unwrap();
+        assert_eq!(String::from_utf8(out).unwrap(), xml);
+    }
+}

feeder-plugin/usenet-feeder/src/nntmux/api.rs

@@ -0,0 +1,386 @@
+//! Client for **our own** nntmux, over its Newznab HTTP API — the only way this
+//! feeder reaches it.
+//!
+//! nntmux is a separate app (worph/AppStore `NNTmux`). Until 1.1 this feeder
+//! read nntmux's MariaDB with SQL and gunzipped `<guid>.nzb.gz` straight off
+//! its storage volume; that tied it to nntmux's schema, its DB password and its
+//! shard layout, across two apps. Now it uses what any Newznab client uses:
+//!
+//! | call | route | cost (nntmux per-user counters) |
+//! |---|---|---|
+//! | [`NntmuxApi::search`] | `GET {base}/api/v1/api?t=search&q=…&extended=1` | 1 `apirequests` |
+//! | [`NntmuxApi::details`] | `GET {base}/api/v1/api?t=details&id=<guid>` | 1 `apirequests` |
+//! | [`NntmuxApi::nzb`] | `GET {base}/getnzb?id=<guid>.nzb&r=<key>` | 1 `downloadrequests` |
+//! | [`NntmuxApi::caps`] | `GET {base}/api/v1/api?t=caps` | free, no key |
+//!
+//! ⚠ The API lives under **`/api/v1/api`** (Laravel's `api` prefix) — plain
+//! `/api` is a 404. `t=get` is not used: in this nntmux it *redirects* to
+//! `/getnzb` on the **public** `APP_URL` host, which a container on `pcs` should
+//! not round-trip through; `/getnzb` on the internal host is the same handler.
+//!
+//! ⚠ nntmux throttles the whole `/api` group to 60 requests/minute and caps
+//! each role per 24 h (`apirequests`, `downloadrequests`: Admin 1000, User 10).
+//! The key should belong to a dedicated user whose role has raised caps (the
+//! NNTmux app provisions `metamesh`); [`Pacer`] keeps us under the per-minute
+//! throttle, and the per-guid cache ([`super::cache`]) means each release's
+//! `.nzb` is downloaded once, ever.
+
+use std::collections::VecDeque;
+use std::time::{Duration, Instant};
+
+use meta_feeder_sdk::types::GatewayError;
+use tokio::sync::Mutex;
+
+use super::catalog::Release;
+use crate::newznab::{classify, fetch_nzb, looks_like_html, newznab_error, status_error};
+
+/// Default base URL: the NNTmux app's web container on the shared `pcs` network.
+pub const DEFAULT_NNTMUX_URL: &str = "http://nntmux";
+
+/// Newznab's own page-size ceiling (`<limits max="100">`).
+pub const MAX_PAGE: usize = 100;
+
+/// Requests allowed per [`PACE_WINDOW`] — under nntmux's 60/min throttle, with
+/// headroom for a human browsing nntmux with the same key.
+const PACE_BUDGET: usize = 50;
+const PACE_WINDOW: Duration = Duration::from_secs(60);
+
+/// Sliding-window limiter: at most [`PACE_BUDGET`] calls per [`PACE_WINDOW`].
+/// A cold search (one `t=search` + one `/getnzb` per new hit) bursts freely up
+/// to the budget, then waits instead of earning an HTTP 429.
+#[derive(Default)]
+pub struct Pacer {
+    sent: Mutex<VecDeque<Instant>>,
+}
+
+impl Pacer {
+    pub async fn wait(&self) {
+        loop {
+            let mut sent = self.sent.lock().await;
+            let now = Instant::now();
+            while sent.front().is_some_and(|t| now.duration_since(*t) >= PACE_WINDOW) {
+                sent.pop_front();
+            }
+            if sent.len() < PACE_BUDGET {
+                sent.push_back(now);
+                return;
+            }
+            let wait = PACE_WINDOW - now.duration_since(*sent.front().unwrap());
+            drop(sent);
+            tokio::time::sleep(wait).await;
+        }
+    }
+}
+
+pub struct NntmuxApi {
+    http: reqwest::Client,
+    /// Scheme + authority (+ optional path prefix), no trailing slash.
+    base: String,
+    key: String,
+    pacer: Pacer,
+}
+
+impl NntmuxApi {
+    pub fn new(http: reqwest::Client, base: &str, key: &str) -> Self {
+        let base = base.trim().trim_end_matches('/');
+        let base = if base.starts_with("http://") || base.starts_with("https://") {
+            base.to_string()
+        } else {
+            format!("http://{base}")
+        };
+        Self {
+            http,
+            base,
+            key: key.trim().to_string(),
+            pacer: Pacer::default(),
+        }
+    }
+
+    pub fn base(&self) -> &str {
+        &self.base
+    }
+
+    fn api_url(&self) -> String {
+        format!("{}/api/v1/api", self.base)
+    }
+
+    /// Free-text search, newest first. `q` is passed as typed — nntmux tokenises
+    /// it (Manticore full-text when present, SQL otherwise).
+    pub async fn search(&self, q: &str, limit: usize) -> Result<Vec<Release>, GatewayError> {
+        let limit = limit.clamp(1, MAX_PAGE).to_string();
+        let xml = self
+            .get_xml(&[("t", "search"), ("q", q), ("limit", &limit), ("extended", "1")])
+            .await?;
+        parse_items(&xml).map_err(|e| GatewayError::Transient(format!("nntmux search answer: {e}")))
+    }
+
+    /// One release's metadata, by guid (`None` = nntmux doesn't have it).
+    pub async fn details(&self, guid: &str) -> Result<Option<Release>, GatewayError> {
+        let xml = match self.get_xml(&[("t", "details"), ("id", guid), ("extended", "1")]).await {
+            Err(GatewayError::NotFound) => return Ok(None),
+            other => other?,
+        };
+        let items = parse_items(&xml)
+            .map_err(|e| GatewayError::Transient(format!("nntmux details answer: {e}")))?;
+        Ok(items.into_iter().find(|r| r.guid == guid))
+    }
+
+    /// The release's `.nzb` (plain XML). Spends one `downloadrequests` unit —
+    /// callers go through the cache.
+    pub async fn nzb(&self, guid: &str) -> Result<Vec<u8>, GatewayError> {
+        self.pacer.wait().await;
+        let url = format!("{}/getnzb?id={guid}.nzb&r={}", self.base, self.key);
+        fetch_nzb(&self.http, &url, &self.base).await
+    }
+
+    /// Reachability probe (`t=caps` needs no key and counts against nothing).
+    pub async fn caps(&self) -> Result<(), GatewayError> {
+        let url = format!("{}?t=caps", self.api_url());
+        let resp = self
+            .http
+            .get(&url)
+            .send()
+            .await
+            .map_err(|e| GatewayError::Transient(format!("nntmux {} unreachable: {}", self.base, e.without_url())))?;
+        if !resp.status().is_success() {
+            return Err(status_error(&resp, &self.base));
+        }
+        let body = resp
+            .text()
+            .await
+            .map_err(|e| GatewayError::Transient(format!("nntmux caps: {}", e.without_url())))?;
+        if body.contains("<caps") {
+            Ok(())
+        } else {
+            Err(GatewayError::Permanent(format!(
+                "{} did not answer t=caps like a Newznab server — is this the nntmux base URL?",
+                self.base
+            )))
+        }
+    }
+
+    /// GET the API with the key; Newznab errors (answered as HTTP 200) are mapped
+    /// through [`classify`]. The URL is never echoed — it carries the key.
+    async fn get_xml(&self, params: &[(&str, &str)]) -> Result<String, GatewayError> {
+        self.pacer.wait().await;
+        let mut query: Vec<(&str, &str)> = params.to_vec();
+        query.push(("apikey", &self.key));
+        let resp = self
+            .http
+            .get(self.api_url())
+            .query(&query)
+            .send()
+            .await
+            .map_err(|e| GatewayError::Transient(format!("nntmux {}: {}", self.base, e.without_url())))?;
+        if !resp.status().is_success() {
+            return Err(status_error(&resp, &self.base));
+        }
+        let body = resp
+            .bytes()
+            .await
+            .map_err(|e| GatewayError::Transient(format!("nntmux {}: {}", self.base, e.without_url())))?;
+        if looks_like_html(&body) {
+            return Err(GatewayError::Permanent(format!(
+                "{} answered the Newznab API with an HTML page — check the nntmux base URL",
+                self.base
+            )));
+        }
+        let text = String::from_utf8_lossy(&body).into_owned();
+        if let Some(err) = newznab_error(&text) {
+            return Err(classify(&err));
+        }
+        Ok(text)
+    }
+}
+
+/// Parse a Newznab RSS answer into releases. Items without a guid are skipped.
+pub fn parse_items(xml: &str) -> anyhow::Result<Vec<Release>> {
+    use quick_xml::events::Event;
+    use quick_xml::Reader;
+
+    #[derive(Default)]
+    struct Item {
+        title: String,
+        guid: String,
+        size: u64,
+        enclosure_len: u64,
+        category: i32,
+        usenetdate: Option<i64>,
+        pubdate: Option<i64>,
+    }
+
+    let mut reader = Reader::from_str(xml);
+    reader.config_mut().trim_text(true);
+    let mut out = Vec::new();
+    let mut item: Option<Item> = None;
+    let mut field: Option<&'static str> = None;
+
+    loop {
+        match reader.read_event() {
+            Ok(Event::Start(e)) => match e.local_name().as_ref() {
+                b"item" => item = Some(Item::default()),
+                b"title" if item.is_some() => field = Some("title"),
+                b"guid" if item.is_some() => field = Some("guid"),
+                b"pubDate" if item.is_some() => field = Some("pubDate"),
+                _ => field = None,
+            },
+            Ok(Event::Empty(e)) => {
+                let Some(it) = item.as_mut() else { continue };
+                let name = e.local_name();
+                let attr = |k: &[u8]| -> Option<String> {
+                    e.attributes()
+                        .flatten()
+                        .find(|a| a.key.local_name().as_ref() == k)
+                        .and_then(|a| a.unescape_value().ok().map(|v| v.into_owned()))
+                };
+                match name.as_ref() {
+                    b"attr" => {
+                        let (Some(n), Some(v)) = (attr(b"name"), attr(b"value")) else { continue };
+                        match n.as_str() {
+                            // Several: the parent (2000) and the subcategory (2040).
+                            "category" => {
+                                if let Ok(c) = v.trim().parse::<i32>() {
+                                    it.category = it.category.max(c);
+                                }
+                            }
+                            "size" => it.size = v.trim().parse().unwrap_or(0),
+                            "usenetdate" => it.usenetdate = rfc2822(&v),
+                            "guid" if it.guid.is_empty() => it.guid = v.trim().to_string(),
+                            _ => {}
+                        }
+                    }
+                    b"enclosure" => {
+                        it.enclosure_len = attr(b"length").and_then(|l| l.trim().parse().ok()).unwrap_or(0);
+                    }
+                    _ => {}
+                }
+            }
+            Ok(Event::Text(t)) => {
+                let (Some(it), Some(f)) = (item.as_mut(), field) else { continue };
+                let v = t.unescape().map(|c| c.into_owned()).unwrap_or_default();
+                match f {
+                    "title" => it.title = v,
+                    "guid" => it.guid = guid_of(&v),
+                    "pubDate" => it.pubdate = rfc2822(&v),
+                    _ => {}
+                }
+            }
+            Ok(Event::End(e)) => {
+                if e.local_name().as_ref() == b"item" {
+                    if let Some(it) = item.take() {
+                        if !it.guid.is_empty() {
+                            out.push(Release {
+                                guid: it.guid,
+                                search_name: it.title,
+                                size: if it.size > 0 { it.size } else { it.enclosure_len },
+                                category_id: it.category,
+                                post_date: it.usenetdate.or(it.pubdate),
+                            });
+                        }
+                    }
+                }
+                field = None;
+            }
+            Ok(Event::Eof) => break,
+            Err(e) => return Err(anyhow::anyhow!("newznab xml: {e}")),
+            _ => {}
+        }
+    }
+    Ok(out)
+}
+
+/// nntmux's `<guid>` is a details URL (`{server}/details/<guid>`); the release
+/// guid is its last path segment. A bare guid passes through.
+fn guid_of(raw: &str) -> String {
+    let t = raw.trim().trim_end_matches('/');
+    t.rsplit('/').next().unwrap_or(t).to_string()
+}
+
+fn rfc2822(s: &str) -> Option<i64> {
+    chrono::DateTime::parse_from_rfc2822(s.trim()).ok().map(|d| d.timestamp())
+}
+
+/// A release guid as nntmux mints it (hex/uuid): the only shape this feeder will
+/// put into a URL or a cache path. Guards `get_blob`, whose id comes off the wire.
+pub fn is_guid(s: &str) -> bool {
+    !s.is_empty() && s.len() <= 64 && s.bytes().all(|b| b.is_ascii_alphanumeric() || b == b'-')
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    const SEARCH: &str = r#"<?xml version="1.0" encoding="UTF-8"?>
+<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:newznab="http://www.newznab.com/DTD/2010/feeds/attributes/">
+<channel>
+ <title>NNTmux</title>
+ <newznab:response offset="0" total="2"/>
+ <item>
+  <title>Celebrity.Ghost.Stories.S05E08.720p.HDTV.x264-DHD</title>
+  <guid isPermaLink="true">http://localhost/details/97a2c0ffee1234</guid>
+  <link>http://localhost/getnzb?id=97a2c0ffee1234.nzb&amp;r=KEY</link>
+  <pubDate>Sat, 03 Oct 2026 12:00:00 +0000</pubDate>
+  <category>TV &gt; HD</category>
+  <enclosure url="http://localhost/getnzb?id=97a2c0ffee1234.nzb&amp;r=KEY" length="734003200" type="application/x-nzb"/>
+  <newznab:attr name="category" value="5000"/>
+  <newznab:attr name="category" value="5040"/>
+  <newznab:attr name="size" value="734003200"/>
+  <newznab:attr name="usenetdate" value="Fri, 02 Oct 2026 08:30:00 +0000"/>
+ </item>
+ <item>
+  <title>no-guid-item</title>
+ </item>
+ <item>
+  <title>Some.Movie.2024.1080p</title>
+  <guid>abcd-1234</guid>
+  <enclosure url="x" length="42" type="application/x-nzb"/>
+  <newznab:attr name="category" value="2040"/>
+ </item>
+</channel>
+</rss>"#;
+
+    #[test]
+    fn parses_items_guids_categories_sizes_and_dates() {
+        let r = parse_items(SEARCH).unwrap();
+        assert_eq!(r.len(), 2, "an item without a guid is skipped");
+        assert_eq!(r[0].guid, "97a2c0ffee1234");
+        assert_eq!(r[0].search_name, "Celebrity.Ghost.Stories.S05E08.720p.HDTV.x264-DHD");
+        assert_eq!(r[0].category_id, 5040, "the most specific category wins");
+        assert_eq!(r[0].size, 734_003_200);
+        assert_eq!(r[0].post_date, Some(1_790_929_800), "usenetdate beats pubDate");
+        assert_eq!(r[1].guid, "abcd-1234");
+        assert_eq!(r[1].size, 42, "enclosure length is the size fallback");
+        assert_eq!(r[1].post_date, None);
+    }
+
+    #[test]
+    fn an_empty_channel_is_no_releases() {
+        assert!(parse_items("<rss><channel></channel></rss>").unwrap().is_empty());
+    }
+
+    #[test]
+    fn guid_guard() {
+        assert!(is_guid("97a2c0ffee1234"));
+        assert!(is_guid("0d1b2c3d-aaaa-bbbb-cccc-000000000000"));
+        assert!(!is_guid("../etc/passwd"));
+        assert!(!is_guid("a/b"));
+        assert!(!is_guid(""));
+    }
+
+    #[test]
+    fn base_urls_are_normalised() {
+        let c = reqwest::Client::new();
+        assert_eq!(NntmuxApi::new(c.clone(), "nntmux", "k").base(), "http://nntmux");
+        assert_eq!(NntmuxApi::new(c, " http://nntmux:80/ ", "k").base(), "http://nntmux:80");
+    }
+
+    #[tokio::test]
+    async fn the_pacer_lets_a_burst_through() {
+        let p = Pacer::default();
+        let t = Instant::now();
+        for _ in 0..PACE_BUDGET {
+            p.wait().await;
+        }
+        assert!(t.elapsed() < Duration::from_secs(1));
+    }
+}

feeder-plugin/usenet-feeder/src/nntmux/cache.rs

@@ -0,0 +1,130 @@
+//! Per-release `.nzb` cache, so each nntmux release is downloaded **once**.
+//!
+//! Every search hit must carry its portable `nzb-posting` cid (see
+//! `usenet::record_for`), and that cid is a digest over the `.nzb`'s
+//! Message-IDs — so a hit costs one `/getnzb` the first time it is seen. nntmux
+//! counts each against the key's daily `downloadrequests`; without this cache a
+//! popular term would re-download the same postings on every query. A collated
+//! nntmux release never changes, so an entry never goes stale.
+//!
+//! Layout: `<feeder state>/nntmux-nzb/<guid>.nzb` (plain XML, written
+//! atomically). The same bytes are what `get_blob` serves the gateway core for
+//! seeding. An in-memory map remembers the minted cid so a cached hit is not
+//! even re-parsed.
+
+use std::collections::HashMap;
+use std::path::{Path, PathBuf};
+use std::sync::Mutex;
+
+use meta_feeder_sdk::hash::compute_nzb_posting_cid;
+
+use super::api::is_guid;
+use super::nzb::extract_message_ids;
+
+/// Upper bound on the in-memory cid memo (the files themselves are unbounded).
+const MEMO_MAX: usize = 50_000;
+
+/// A release's minted identity.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct Posting {
+    pub cid: String,
+    pub segment_count: usize,
+}
+
+pub struct NzbCache {
+    dir: PathBuf,
+    memo: Mutex<HashMap<String, Posting>>,
+}
+
+impl NzbCache {
+    pub fn new(dir: &Path) -> Self {
+        Self {
+            dir: dir.to_path_buf(),
+            memo: Mutex::new(HashMap::new()),
+        }
+    }
+
+    fn path(&self, guid: &str) -> Option<PathBuf> {
+        is_guid(guid).then(|| self.dir.join(format!("{guid}.nzb")))
+    }
+
+    /// The cached `.nzb`, if any.
+    pub fn get(&self, guid: &str) -> Option<Vec<u8>> {
+        std::fs::read(self.path(guid)?).ok()
+    }
+
+    /// Store a downloaded `.nzb` (tmp + rename, so a reader never sees half a file).
+    pub fn put(&self, guid: &str, xml: &[u8]) -> std::io::Result<()> {
+        let Some(path) = self.path(guid) else {
+            return Err(std::io::Error::new(std::io::ErrorKind::InvalidInput, "not a release guid"));
+        };
+        std::fs::create_dir_all(&self.dir)?;
+        let tmp = path.with_extension("nzb.tmp");
+        std::fs::write(&tmp, xml)?;
+        std::fs::rename(&tmp, &path)
+    }
+
+    /// Mint (or recall) the posting identity for `guid` from its `.nzb`.
+    /// `None` when the document has no segments or doesn't parse.
+    pub fn posting(&self, guid: &str, xml: &[u8]) -> Option<Posting> {
+        if let Some(p) = self.memo.lock().unwrap().get(guid) {
+            return Some(p.clone());
+        }
+        let ids = extract_message_ids(xml).ok()?;
+        if ids.is_empty() {
+            return None;
+        }
+        let p = Posting {
+            cid: compute_nzb_posting_cid(&ids),
+            segment_count: ids.len(),
+        };
+        let mut memo = self.memo.lock().unwrap();
+        if memo.len() >= MEMO_MAX {
+            memo.clear();
+        }
+        memo.insert(guid.to_string(), p.clone());
+        Some(p)
+    }
+
+    /// The memoised identity only (no file read) — the search hot path.
+    pub fn known(&self, guid: &str) -> Option<Posting> {
+        self.memo.lock().unwrap().get(guid).cloned()
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    const NZB: &[u8] = br#"<?xml version="1.0"?><nzb><file><segments>
+        <segment bytes="1" number="1">a@b</segment><segment bytes="1" number="2">c@d</segment>
+        </segments></file></nzb>"#;
+
+    #[test]
+    fn put_get_and_mint() {
+        let dir = tempfile::tempdir().unwrap();
+        let c = NzbCache::new(&dir.path().join("nntmux-nzb"));
+        assert!(c.get("abc123").is_none());
+        c.put("abc123", NZB).unwrap();
+        assert_eq!(c.get("abc123").unwrap(), NZB);
+        let p = c.posting("abc123", NZB).unwrap();
+        assert_eq!(p.segment_count, 2);
+        assert_eq!(p.cid, compute_nzb_posting_cid(&["a@b".to_string(), "c@d".to_string()]));
+        assert_eq!(c.known("abc123"), Some(p));
+    }
+
+    #[test]
+    fn a_hostile_guid_never_becomes_a_path() {
+        let dir = tempfile::tempdir().unwrap();
+        let c = NzbCache::new(dir.path());
+        assert!(c.put("../escape", NZB).is_err());
+        assert!(c.get("../escape").is_none());
+    }
+
+    #[test]
+    fn a_segmentless_nzb_mints_nothing() {
+        let dir = tempfile::tempdir().unwrap();
+        let c = NzbCache::new(dir.path());
+        assert!(c.posting("g", b"<nzb></nzb>").is_none());
+    }
+}

feeder-plugin/usenet-feeder/src/nntmux/catalog.rs

@@ -0,0 +1,77 @@
+//! An nntmux release as the feeder sees it, and the Newznab category mapping.
+//!
+//! Filled from nntmux's Newznab search answer ([`super::api`]); nothing here
+//! knows how it was fetched.
+
+/// One collated nntmux release — the metadata half of a Usenet posting.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct Release {
+    /// nntmux's public id (the last path segment of the item's `<guid>`). This
+    /// is our `record_id`, and the id `/getnzb` takes.
+    pub guid: String,
+    /// The collator's cleaned, scene-style name
+    /// (`Celebrity.Ghost.Stories.S05E08.720p.HDTV.x264-DHD`) — the item `<title>`.
+    pub search_name: String,
+    /// Total posted size in bytes (RAR + par2 included, so an over-estimate of
+    /// the final media file). `0` when the answer carried none.
+    pub size: u64,
+    /// nntmux category id — the most specific `newznab:attr name="category"`.
+    /// Coarse: 2000s = movies, 5000s = TV, 3000s = audio, 7000s = books.
+    pub category_id: i32,
+    /// Unix seconds of the original posting date (`usenetdate`), else the
+    /// item's `pubDate`, when either parses.
+    pub post_date: Option<i64>,
+}
+
+/// Map an nntmux/Newznab category id onto MetaMesh's `fileType`.
+///
+/// The Newznab tree is coarse and stable: 1000s console, 2000s movies, 3000s
+/// audio, 4000s PC, 5000s TV, 6000s XXX, 7000s books. We only need the buckets
+/// `METADATA_KEYS.md` §1 defines.
+pub fn file_type_for_category(category_id: i32) -> &'static str {
+    match category_id / 1000 {
+        2 | 5 | 6 => "video",
+        3 => "audio",
+        7 => "document",
+        _ => "archive",
+    }
+}
+
+/// `contentKind` refinement of [`file_type_for_category`]. Only asserted where
+/// the category tree is unambiguous — a wrong `contentKind` is worse than none,
+/// because meta-watch's curated rows gate on it.
+pub fn content_kind_for_category(category_id: i32) -> Option<&'static str> {
+    match category_id / 1000 {
+        2 => Some("movie"),
+        // ⚠ `episode`, not `tv`. `tv` is not a `contentKind` at all — it was a
+        // `domain` value, and it isn't even that any more (film + tv merged
+        // into `screen`, METADATA_KEYS.md §14.17). A kind outside the registry
+        // vocabulary resolves to no domain and no workForm, so the row is
+        // stamped with neither and is invisible to every client wall.
+        5 => Some("episode"),
+        _ => None,
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn newznab_categories_map_to_file_types() {
+        assert_eq!(file_type_for_category(2040), "video"); // movies HD
+        assert_eq!(file_type_for_category(5040), "video"); // tv HD
+        assert_eq!(file_type_for_category(3010), "audio");
+        assert_eq!(file_type_for_category(7020), "document"); // ebook
+        assert_eq!(file_type_for_category(4000), "archive"); // pc
+        assert_eq!(file_type_for_category(10), "archive"); // nntmux "Other > Misc"
+    }
+
+    #[test]
+    fn content_kind_only_where_unambiguous() {
+        assert_eq!(content_kind_for_category(2040), Some("movie"));
+        assert_eq!(content_kind_for_category(5040), Some("episode"));
+        assert_eq!(content_kind_for_category(3010), None);
+        assert_eq!(content_kind_for_category(7020), None);
+    }
+}

feeder-plugin/usenet-feeder/src/nntmux/db.rs

@@ -1,281 +0,0 @@
-//! Read-only access to the nntmux sidecar's release catalog (MariaDB).
-//!
-//! We are a **consumer** of nntmux's schema, never a writer: the scanner owns
-//! that database, and everything in it is re-derivable by re-scanning. Treat it
-//! as persistent cache, not as a source of truth (the Tribler precedent — a
-//! stateful sidecar the gateway still sees as a stateless HTTP feeder).
-//!
-//! Only a narrow, long-stable column set is selected (`guid`, `searchname`,
-//! `name`, `size`, `postdate`, `categories_id`, `totalpart`). nntmux carries
-//! ~80 columns on `releases`, most of them scanner bookkeeping; binding a wide
-//! row here would make us fragile to upstream schema churn for no gain.
-
-use anyhow::{Context, Result};
-use sqlx::mysql::{MySqlPool, MySqlPoolOptions};
-use sqlx::Row;
-
-/// One collated nntmux release — the metadata half of a Usenet posting.
-#[derive(Debug, Clone)]
-pub struct Release {
-    /// nntmux's public id, and the basename of the on-disk `<guid>.nzb.gz`.
-    /// This is our `record_id`.
-    pub guid: String,
-    /// The collator's cleaned, scene-style name
-    /// (`Celebrity.Ghost.Stories.S05E08.720p.HDTV.x264-DHD`). Falls back to the
-    /// raw posted `name` when the collator could not improve on it.
-    pub search_name: String,
-    /// Total posted size in bytes (RAR + par2 included, so an over-estimate of
-    /// the final media file).
-    pub size: u64,
-    /// nntmux category id (`categories.id`). Coarse: 2000s = movies, 5000s = TV,
-    /// 3000s = audio, 7000s = books — the Newznab category tree.
-    pub category_id: i32,
-    /// Unix seconds of the original posting date, when nntmux recorded one.
-    pub post_date: Option<i64>,
-}
-
-/// Connection pool to nntmux's MariaDB. Small by design — this feeder issues a
-/// handful of short SELECTs per query, and the scanner needs the headroom.
-#[derive(Clone)]
-pub struct Db {
-    pool: MySqlPool,
-}
-
-impl Db {
-    /// Open a pool. Does **not** verify the schema — a feeder must come up and
-    /// serve `/health` even when its upstream is unreachable, so the gateway's
-    /// `depends_on` is satisfied and discovery does not soft-skip us forever.
-    pub async fn connect(url: &str) -> Result<Self> {
-        let pool = MySqlPoolOptions::new()
-            .max_connections(4)
-            .acquire_timeout(std::time::Duration::from_secs(10))
-            .connect(url)
-            .await
-            .with_context(|| "connect to nntmux mariadb")?;
-        Ok(Self { pool })
-    }
-
-    /// Free-text search over the collated release names.
-    ///
-    /// `LIKE %term%` on `searchname` — deliberately simple. nntmux can be built
-    /// with Manticore/Elasticsearch for real full-text search, but both are
-    /// optional in the minimum stack and we do not want to require them. If
-    /// search quality becomes the bottleneck, that is the upgrade path.
-    pub async fn search(&self, term: &str, limit: usize) -> Result<Vec<Release>> {
-        // ⚠ TOKENISE, don't match the phrase. Scene releases are dot-separated
-        // (`Breaking.Bad.S01E01.1080p...`) while users type spaces ("breaking
-        // bad"), so a single `LIKE %breaking bad%` matches essentially NOTHING
-        // — the separator never lines up. Split into words and AND them, which
-        // makes the separator irrelevant on both sides.
-        //
-        // Found by end-to-end search from meta-watch; the unit tests missed it
-        // because they only exercised the escaping, never a realistic
-        // scene-style name against a natural-language query.
-        let words = search_tokens(term);
-        if words.is_empty() {
-            return Ok(vec![]);
-        }
-        let mut sql = String::from(
-            "SELECT guid, searchname, name, size, categories_id, \
-                    UNIX_TIMESTAMP(postdate) AS postdate_unix \
-             FROM releases WHERE ",
-        );
-        sql.push_str(
-            &words
-                .iter()
-                .map(|_| "searchname LIKE ?")
-                .collect::<Vec<_>>()
-                .join(" AND "),
-        );
-        sql.push_str(" ORDER BY postdate DESC LIMIT ?");
-
-        let mut q = sqlx::query(&sql);
-        for w in &words {
-            q = q.bind(format!("%{}%", escape_like(w)));
-        }
-        let rows = q
-            .bind(limit as i64)
-            .fetch_all(&self.pool)
-            .await
-            .context("query releases by name")?;
-        Ok(rows.iter().map(row_to_release).collect())
-    }
-
-    /// Look one release up by guid — the `compute_outcomes` path, where the
-    /// record_id came back to us from the gateway.
-    pub async fn by_guid(&self, guid: &str) -> Result<Option<Release>> {
-        let row = sqlx::query(
-            "SELECT guid, searchname, name, size, categories_id, \
-                    UNIX_TIMESTAMP(postdate) AS postdate_unix \
-             FROM releases WHERE guid = ? LIMIT 1",
-        )
-        .bind(guid)
-        .fetch_optional(&self.pool)
-        .await
-        .context("query release by guid")?;
-        Ok(row.as_ref().map(row_to_release))
-    }
-
-    /// Cheap liveness probe for `health()` — confirms the catalog is reachable
-    /// *and* non-empty. An empty `releases` table is the visible symptom of both
-    /// nntmux silent-zero traps (an unset `PATH_TO_NZBS`, or groups left
-    /// `active=0`), so reporting it as degraded turns a silent failure into a
-    /// legible one.
-    pub async fn release_count(&self) -> Result<i64> {
-        let row = sqlx::query("SELECT COUNT(*) AS n FROM releases")
-            .fetch_one(&self.pool)
-            .await
-            .context("count releases")?;
-        Ok(row.try_get::<i64, _>("n").unwrap_or(0))
-    }
-}
-
-fn row_to_release(row: &sqlx::mysql::MySqlRow) -> Release {
-    let search_name: String = row
-        .try_get::<String, _>("searchname")
-        .ok()
-        .filter(|s| !s.is_empty())
-        .or_else(|| row.try_get::<String, _>("name").ok())
-        .unwrap_or_default();
-    Release {
-        guid: row.try_get::<String, _>("guid").unwrap_or_default(),
-        search_name,
-        // nntmux stores size as an unsigned bigint; it arrives as u64 or
-        // (older schemas) a decimal string.
-        size: row
-            .try_get::<u64, _>("size")
-            .ok()
-            .or_else(|| {
-                row.try_get::<String, _>("size")
-                    .ok()
-                    .and_then(|s| s.parse().ok())
-            })
-            .unwrap_or(0),
-        category_id: row.try_get::<i32, _>("categories_id").unwrap_or(0),
-        post_date: row.try_get::<i64, _>("postdate_unix").ok(),
-    }
-}
-
-/// Split a user query into match tokens.
-///
-/// Anything that is not alphanumeric is a separator — which folds the scene
-/// world's `.`/`_`/`-` and the human world's space onto the same footing, so
-/// "breaking bad", "Breaking.Bad" and "breaking-bad" all produce the same
-/// tokens and all match `Breaking.Bad.S01E01.1080p.BluRay.x264-SCENE`.
-///
-/// Single characters are dropped: they match nearly every row and cost a full
-/// scan for no selectivity.
-fn search_tokens(term: &str) -> Vec<String> {
-    term.split(|c: char| !c.is_alphanumeric())
-        .filter(|w| w.chars().count() > 1)
-        .map(|w| w.to_string())
-        .collect()
-}
-
-/// Escape the `LIKE` wildcards so a user searching for `100%` does not match
-/// everything. Backslash is MySQL's default escape character.
-fn escape_like(s: &str) -> String {
-    s.replace('\\', "\\\\")
-        .replace('%', "\\%")
-        .replace('_', "\\_")
-}
-
-/// Map an nntmux/Newznab category id onto MetaMesh's `fileType`.
-///
-/// The Newznab tree is coarse and stable: 1000s console, 2000s movies, 3000s
-/// audio, 4000s PC, 5000s TV, 6000s XXX, 7000s books. We only need the buckets
-/// `METADATA_KEYS.md` §1 defines.
-pub fn file_type_for_category(category_id: i32) -> &'static str {
-    match category_id / 1000 {
-        2 | 5 | 6 => "video",
-        3 => "audio",
-        7 => "document",
-        _ => "archive",
-    }
-}
-
-/// `contentKind` refinement of [`file_type_for_category`]. Only asserted where
-/// the category tree is unambiguous — a wrong `contentKind` is worse than none,
-/// because meta-watch's curated rows gate on it.
-pub fn content_kind_for_category(category_id: i32) -> Option<&'static str> {
-    match category_id / 1000 {
-        2 => Some("movie"),
-        5 => Some("tv"),
-        _ => None,
-    }
-}
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-
-    /// ⚠ THE DOT-VS-SPACE REGRESSION. Scene releases are dot-separated and
-    /// users type spaces; matching the raw phrase finds nothing. Caught by an
-    /// end-to-end search from meta-watch that returned 0 results against a
-    /// catalog that plainly contained the show.
-    #[test]
-    fn a_spaced_query_tokenises_to_match_a_dotted_scene_name() {
-        let tokens = search_tokens("breaking bad");
-        assert_eq!(tokens, vec!["breaking", "bad"]);
-
-        // Every token must be a substring of the real scene name — that is
-        // exactly what the AND-ed LIKEs check.
-        let scene = "Breaking.Bad.S01E01.1080p.BluRay.x264-SCENE".to_lowercase();
-        assert!(
-            tokens.iter().all(|t| scene.contains(&t.to_lowercase())),
-            "tokens {tokens:?} must all appear in {scene}"
-        );
-
-        // The naive whole-phrase form does NOT — this is the bug, pinned.
-        assert!(
-            !scene.contains("breaking bad"),
-            "if this ever passes, the dotted-name assumption changed"
-        );
-    }
-
-    /// Separators are interchangeable in both directions.
-    #[test]
-    fn scene_and_human_separators_produce_the_same_tokens() {
-        let expect = vec!["breaking".to_string(), "bad".to_string()];
-        assert_eq!(search_tokens("breaking bad"), expect);
-        assert_eq!(search_tokens("Breaking.Bad").iter().map(|s| s.to_lowercase()).collect::<Vec<_>>(), expect);
-        assert_eq!(search_tokens("breaking-bad"), expect);
-        assert_eq!(search_tokens("breaking_bad"), expect);
-        assert_eq!(search_tokens("  breaking   bad  "), expect);
-    }
-
-    /// Single characters are dropped (no selectivity, full scan) and an
-    /// all-noise query yields nothing rather than matching the whole catalog.
-    #[test]
-    fn noise_only_queries_yield_no_tokens() {
-        assert!(search_tokens("").is_empty());
-        assert!(search_tokens("   ").is_empty());
-        assert!(search_tokens("- . _").is_empty());
-        assert_eq!(search_tokens("a bc"), vec!["bc"]);
-    }
-
-    #[test]
-    fn like_wildcards_are_escaped() {
-        assert_eq!(escape_like("100%"), "100\\%");
-        assert_eq!(escape_like("a_b"), "a\\_b");
-        assert_eq!(escape_like("c:\\d"), "c:\\\\d");
-        assert_eq!(escape_like("naruto"), "naruto");
-    }
-
-    #[test]
-    fn newznab_categories_map_to_file_types() {
-        assert_eq!(file_type_for_category(2040), "video"); // movies HD
-        assert_eq!(file_type_for_category(5040), "video"); // tv HD
-        assert_eq!(file_type_for_category(3010), "audio");
-        assert_eq!(file_type_for_category(7020), "document"); // ebook
-        assert_eq!(file_type_for_category(4000), "archive"); // pc
-    }
-
-    #[test]
-    fn content_kind_only_where_unambiguous() {
-        assert_eq!(content_kind_for_category(2040), Some("movie"));
-        assert_eq!(content_kind_for_category(5040), Some("tv"));
-        assert_eq!(content_kind_for_category(3010), None);
-        assert_eq!(content_kind_for_category(7020), None);
-    }
-}

feeder-plugin/usenet-feeder/src/nntmux/nzb.rs

@@ -1,106 +1,16 @@
-//! Reading an nntmux release's manifest off disk and extracting its ordered
+//! Extracting an NZB's ordered article Message-IDs — the input of the portable
-//! article Message-IDs.
+//! `nzb-posting` (`0x1003`) cid.
 //!
-//! # Why the file and not the database
+//! # Why the `.nzb` and not nntmux's metadata
 //!
 //! You **cannot** read a finished release's Message-IDs out of nntmux's
-//! database. `NzbService::writeNzbForReleaseId()` writes the release's NZB and
+//! database: `NzbService::writeNzbForReleaseId()` writes the release's NZB and
-//! then *unconditionally* purges collections → binaries → parts (all FKs are
+//! then purges collections → binaries → parts. For any *collated* release the
-//! `ON DELETE CASCADE`, and there is no retention setting). Those tables are
+//! ordered Message-IDs survive **only** in its `.nzb` — which this feeder now
-//! transient staging for the header scan; for any *collated* release the
+//! downloads over nntmux's Newznab API (`/getnzb`, see [`super::api`]) and
-//! ordered Message-IDs survive **only** in the on-disk gzip NZB at
+//! caches ([`super::cache`]), instead of gunzipping it off nntmux's storage.
-//! `<nzb_root>/<split>/<guid>.nzb.gz`.
-//!
-//! This was confirmed on the live path during the spike (the offline
-//! `import-nzbs` path never stages collections/binaries/parts at all, so it
-//! cannot be used to check this). It is the single load-bearing fact behind the
-//! whole feeder: lose it and you are back to grabbing `.nzb` files from an
-//! external indexer, which is exactly what this design exists to avoid.
-
-use std::io::Read;
-use std::path::{Path, PathBuf};
 
 use anyhow::{Context, Result};
-use flate2::read::GzDecoder;
-
-/// A parsed nntmux manifest: the raw NZB XML plus the Message-IDs in posting
-/// order.
-#[derive(Debug, Clone)]
-pub struct Manifest {
-    /// The decompressed NZB XML. Handed to the gateway verbatim so it can seed
-    /// it as an ordinary sha2-256 cid — the consumer parses this exact document.
-    pub xml: Vec<u8>,
-    /// Every `<segment>` Message-ID across every `<file>`, in document order.
-    /// Bare form (no surrounding `<>`), matching how NZB stores them and how
-    /// meta-share's `NzbSegment.message_id` holds them.
-    pub message_ids: Vec<String>,
-}
-
-/// How deep we are willing to probe for a shard. nntmux caps its own loop at
-/// 32, but every real deployment sits in the low single digits and a guid is a
-/// 36-char uuid; 8 covers any sane `nzbsplitlevel` while keeping the probe below.
-const MAX_SPLIT_LEVEL: usize = 8;
-
-/// Build the path a release's `.nzb.gz` would occupy at an exact shard depth:
-/// one directory per leading guid character, mirroring nntmux's
-/// `NZB::buildNZBPath()` (`$nzbPath .= $releaseGuid[$i].'/'`).
-pub fn nzb_path_at(nzb_root: &Path, guid: &str, levels: usize) -> Option<PathBuf> {
-    if guid.is_empty() {
-        return None;
-    }
-    let mut path = nzb_root.to_path_buf();
-    for c in guid.chars().take(levels) {
-        path.push(c.to_string());
-    }
-    Some(path.join(format!("{guid}.nzb.gz")))
-}
-
-/// Locate a release's `.nzb.gz` in nntmux's sharded store.
-///
-/// **The shard depth is an nntmux *setting* (`settings.nzbsplitlevel`), not a
-/// constant.** The PHP code's own fallback is 1, but the maintainer image ships
-/// a database seeded with **4** (`storage/nzb/9/7/a/2/97a2….nzb.gz`) — so a
-/// hardcoded depth silently resolves to a path that does not exist, every
-/// release is skipped as "no .nzb.gz on disk yet", and the feeder reports zero
-/// hits while the scan is in fact working perfectly. An operator can also
-/// change the setting *after* files are written, leaving a store with mixed
-/// depths. So resolve by probing the candidate depths, deepest first.
-///
-/// Still not a glob: at most `MAX_SPLIT_LEVEL + 1` `stat` calls against exact
-/// paths, no directory enumeration — cheap enough for the compute path.
-pub fn nzb_path(nzb_root: &Path, guid: &str) -> Option<PathBuf> {
-    let deepest = guid.chars().count().min(MAX_SPLIT_LEVEL);
-    (0..=deepest)
-        .rev()
-        .filter_map(|levels| nzb_path_at(nzb_root, guid, levels))
-        .find(|path| path.exists())
-}
-
-/// Read + gunzip a release's `.nzb.gz` and pull out its Message-IDs.
-///
-/// Returns `Ok(None)` when the file is absent — a normal, non-fatal state: a
-/// release row can exist before `createNZBs()` has written its manifest, and
-/// the caller simply skips that record rather than failing the whole query.
-pub fn load(nzb_root: &Path, guid: &str) -> Result<Option<Manifest>> {
-    let Some(path) = nzb_path(nzb_root, guid) else {
-        return Ok(None);
-    };
-    if !path.exists() {
-        return Ok(None);
-    }
-    let raw = std::fs::read(&path).with_context(|| format!("read nzb {}", path.display()))?;
-    let mut xml = Vec::new();
-    GzDecoder::new(&raw[..])
-        .read_to_end(&mut xml)
-        .with_context(|| format!("gunzip nzb {}", path.display()))?;
-
-    let message_ids = extract_message_ids(&xml)
-        .with_context(|| format!("parse nzb {}", path.display()))?;
-    if message_ids.is_empty() {
-        anyhow::bail!("nzb {} has no segments", path.display());
-    }
-    Ok(Some(Manifest { xml, message_ids }))
-}
 
 /// Pull every `<segment>` body out of an NZB document, in order.
 ///
@@ -110,7 +20,7 @@ pub fn load(nzb_root: &Path, guid: &str) -> Result<Option<Manifest>> {
 /// meta-share would happily play. Surrounding `<>` are stripped if a producer
 /// included them — the mint normalises this too, but keeping the stored form
 /// canonical means the two never disagree about what was hashed.
-fn extract_message_ids(xml: &[u8]) -> Result<Vec<String>> {
+pub fn extract_message_ids(xml: &[u8]) -> Result<Vec<String>> {
     use quick_xml::events::Event;
     use quick_xml::Reader;
 
@@ -183,74 +93,4 @@ mod tests {
         let ids = extract_message_ids(SAMPLE.as_bytes()).expect("parse");
         assert!(ids.iter().all(|i| !i.contains('<') && !i.contains('>')));
     }
-
-    /// One directory per leading guid character, exactly as nntmux's
-    /// `buildNZBPath()` writes them.
-    #[test]
-    fn nzb_path_at_shards_one_dir_per_leading_char() {
-        let p = nzb_path_at(Path::new("/s/nzb"), "abc123", 4).expect("path");
-        assert_eq!(p, Path::new("/s/nzb/a/b/c/1/abc123.nzb.gz"));
-        let p0 = nzb_path_at(Path::new("/s/nzb"), "abc123", 0).expect("path");
-        assert_eq!(p0, Path::new("/s/nzb/abc123.nzb.gz"));
-    }
-
-    /// The regression this exists for: the maintainer image seeds
-    /// `nzbsplitlevel = 4`, so a resolver hardcoded to depth 1 finds nothing and
-    /// every real release is silently skipped as "no .nzb.gz on disk yet".
-    #[test]
-    fn nzb_path_finds_a_level_4_shard() {
-        let dir = tempfile::tempdir().expect("tmp");
-        let guid = "97a2218c-d59b-4bc0-bb67-8a7d41e93fb5";
-        let deep = dir.path().join("9/7/a/2");
-        std::fs::create_dir_all(&deep).expect("mkdir");
-        std::fs::write(deep.join(format!("{guid}.nzb.gz")), b"x").expect("write");
-
-        let got = nzb_path(dir.path(), guid).expect("resolved");
-        assert_eq!(got, deep.join(format!("{guid}.nzb.gz")));
-    }
-
-    /// A store written at a shallower depth (or an operator who lowered
-    /// `nzbsplitlevel` after the fact) must still resolve.
-    #[test]
-    fn nzb_path_finds_a_level_1_shard() {
-        let dir = tempfile::tempdir().expect("tmp");
-        std::fs::create_dir_all(dir.path().join("a")).expect("mkdir");
-        std::fs::write(dir.path().join("a/abc123.nzb.gz"), b"x").expect("write");
-
-        let got = nzb_path(dir.path(), "abc123").expect("resolved");
-        assert_eq!(got, dir.path().join("a/abc123.nzb.gz"));
-    }
-
-    #[test]
-    fn nzb_path_is_none_when_absent_at_every_depth() {
-        let dir = tempfile::tempdir().expect("tmp");
-        assert!(nzb_path(dir.path(), "abc123").is_none());
-    }
-
-    /// A release row can exist before `createNZBs()` has written the manifest.
-    /// That is normal — the caller skips the record, it is not an error.
-    #[test]
-    fn missing_file_is_none_not_error() {
-        let dir = tempfile::tempdir().expect("tmp");
-        let got = load(dir.path(), "deadbeef").expect("no error");
-        assert!(got.is_none());
-    }
-
-    #[test]
-    fn round_trips_a_gzipped_nzb() {
-        use flate2::write::GzEncoder;
-        use flate2::Compression;
-        use std::io::Write;
-
-        let dir = tempfile::tempdir().expect("tmp");
-        std::fs::create_dir_all(dir.path().join("a")).expect("mkdir");
-        let mut enc = GzEncoder::new(Vec::new(), Compression::default());
-        enc.write_all(SAMPLE.as_bytes()).expect("gz write");
-        let gz = enc.finish().expect("gz finish");
-        std::fs::write(dir.path().join("a/abc.nzb.gz"), gz).expect("write");
-
-        let m = load(dir.path(), "abc").expect("load").expect("present");
-        assert_eq!(m.message_ids.len(), 3);
-        assert_eq!(m.xml, SAMPLE.as_bytes());
-    }
 }

feeder-plugin/usenet-feeder/src/usenet.rs

@@ -6,10 +6,16 @@
 //! `nzb-release` (`0x1005`) cid that *embeds that indexer's host*. Only a peer
 //! holding that indexer's credential can ever redeem it.
 //!
-//! This feeder scans Usenet itself, so it already holds the article Message-IDs
+//! This feeder fronts a Usenet scan we run ourselves (the NNTmux app), so the
-//! at collation time — no grab, no scrape. That lets it mint a **portable**
+//! `.nzb` of every release is ours to fetch, unmetered by any third party. That
-//! `nzb-posting` (`0x1003`) cid: a digest over the Message-ID set, embedding no
+//! lets it mint a **portable** `nzb-posting` (`0x1003`) cid: a digest over the
-//! host, redeemable by any peer with a plain NNTP provider.
+//! Message-ID set, embedding no host, redeemable by any peer with a plain NNTP
+//! provider.
+//!
+//! nntmux is reached **only over its Newznab API** ([`crate::nntmux::api`]) with
+//! a key on this feeder's config page — the same contract the torznab feeder
+//! uses with Prowlarr. Each release's `.nzb` is downloaded once and cached
+//! ([`crate::nntmux::cache`]).
 //!
 //! The two coexist. This ADDS a Usenet metadata source; it removes nothing.
 //!
@@ -23,43 +29,80 @@
 //! sha2-256 IPFS cid, publishes it as a first-class `fileType=nzb` meta-core
 //! record, and rewrites the field to that cid. Identical to how posters are
 //! handled. See the study's §5.3.
+//!
+//! # Redeeming external releases
+//!
+//! `compute_outcomes` also accepts an `nzb-release` (`0x1005`) cid — minted by
+//! meta-feeder-torznab from a Newznab search hit — and grabs its `.nzb` with the
+//! key configured for that indexer host ([`crate::newznab`]). It returns ONE
+//! `Sha2_256` outcome carrying the `.nzb` (head stripped, password kept); the
+//! gateway stores it under `/files/plugin/meta-feeder-usenet/`, writes its cid
+//! onto the release record as `manifest`, and merges the outcome record's
+//! `cids/<nzb-posting>` so the release becomes portable to any NNTP-only peer.
+//! Works without nntmux configured.
 
 use std::collections::BTreeMap;
-use std::path::{Path, PathBuf};
+use std::path::Path;
-use std::sync::Arc;
+use std::sync::{Arc, RwLock};
+use std::time::Duration;
 
 use async_trait::async_trait;
+use meta_feeder_sdk::common::build_http_client;
 use meta_feeder_sdk::config::{ConfigField as F, ConfigSchema};
-use meta_feeder_sdk::hash::compute_nzb_posting_cid;
+use meta_feeder_sdk::hash::{
-use meta_feeder_sdk::plugin::{ConfigError, FeederPlugin, HashKind, HashOutcome};
+    codec_of, compute_ipfs_cid, compute_nzb_posting_cid, decode_nzb_release_cid, NZB_RELEASE_CODEC,
+};
+use meta_feeder_sdk::plugin::{ConfigError, FeederPlugin, HashKind, HashOutcome, RedeemClaim};
 use meta_feeder_sdk::query::GatewayQuery;
 use meta_feeder_sdk::types::{DiscoveryRecord, GatewayError, Hash, PluginHealth};
-use tokio::sync::RwLock;
 use tracing::{debug, info, warn};
 
-use crate::nntmux::db::{content_kind_for_category, file_type_for_category, Db, Release};
+use crate::newznab::{self, IndexerCred};
+use crate::nntmux::api::{is_guid, NntmuxApi, DEFAULT_NNTMUX_URL};
+use crate::nntmux::cache::{NzbCache, Posting};
+use crate::nntmux::catalog::{content_kind_for_category, file_type_for_category, Release};
 use crate::nntmux::nzb;
 
 pub const UPSTREAM_ID: &str = "usenet";
 
+/// `/files/plugin/<PACKAGE>/` — where the gateway stores the `.nzb` files this
+/// plugin redeems.
+pub const PACKAGE: &str = "meta-feeder-usenet";
+
+/// Indexers answer an anonymous `t=get` with `<error code="109" description=
+/// "Invalid User Agent"/>` (nzbgeek does), so always send a named client.
+const USER_AGENT: &str = concat!("meta-feeder-usenet/", env!("CARGO_PKG_VERSION"));
+
+/// Wall-clock budget for one `.nzb` grab. The viewer is waiting on it.
+const GRAB_TIMEOUT_SECS: u64 = 60;
+
+/// How often the nntmux reachability probe (`t=caps`, free) runs.
+const PROBE_INTERVAL: Duration = Duration::from_secs(300);
+
 /// Operator-supplied settings, persisted to `<cache_dir>/config.json` by the
 /// SDK's config surface. Env is a **seed only** — the file wins on the next
 /// restart (invariant 12; there is no hot reload).
 #[derive(Debug, Clone, Default)]
 pub struct Settings {
-    /// `mysql://user:pass@host:3306/nntmux`
+    /// Base URL of our nntmux (the NNTmux app's web container), e.g.
-    pub db_url: String,
+    /// `http://nntmux`. The API path (`/api/v1/api`) is appended.
-    /// Root of nntmux's NZB store, shared with the sidecar as a volume. The
+    pub nntmux_url: String,
-    /// directory nntmux's `PATH_TO_NZBS` points at.
+    /// API key of the nntmux user this feeder searches as (the NNTmux app
-    pub nzb_root: String,
+    /// provisions a `metamesh` user with raised daily caps). Blank → the
+    /// self-scan search is off; indexer-key redeems still work.
+    pub nntmux_api_key: String,
+    /// Per-host Newznab keys for redeeming `nzb-release` locators. Config page
+    /// only — no env seed.
+    pub indexers: Vec<IndexerCred>,
 }
 
 impl Settings {
     fn from_env() -> Self {
+        let env = |k: &str| std::env::var(k).ok().map(|v| v.trim().to_string()).filter(|v| !v.is_empty());
         Self {
-            db_url: std::env::var("NNTMUX_DB_URL").unwrap_or_default(),
+            nntmux_url: env("NNTMUX_URL").unwrap_or_else(|| DEFAULT_NNTMUX_URL.to_string()),
-            nzb_root: std::env::var("NNTMUX_NZB_PATH")
+            nntmux_api_key: env("NNTMUX_API_KEY").unwrap_or_default(),
-                .unwrap_or_else(|_| "/nntmux-nzb".to_string()),
+            indexers: Vec::new(),
         }
     }
 
@@ -72,26 +115,48 @@ impl Settings {
             warn!(target: "meta-feeder", path = %path.display(), "config.json is not valid json; using env seed");
             return;
         };
-        if let Some(s) = v.get("db_url").and_then(|x| x.as_str()) {
+        let text = |k: &str| v.get(k).and_then(|x| x.as_str()).map(str::trim).filter(|s| !s.is_empty());
-            if !s.is_empty() {
+        if let Some(s) = text("nntmux_url") {
-                self.db_url = s.to_string();
+            self.nntmux_url = s.to_string();
         }
+        if let Some(s) = text("nntmux_api_key") {
+            self.nntmux_api_key = s.to_string();
         }
-        if let Some(s) = v.get("nzb_root").and_then(|x| x.as_str()) {
+        if text("db_url").is_some() && self.nntmux_api_key.is_empty() {
-            if !s.is_empty() {
+            // A pre-1.1 config: the feeder read nntmux's database directly.
-                self.nzb_root = s.to_string();
+            warn!(
+                target: "meta-feeder",
+                "usenet: config.json still carries the pre-1.1 `db_url` — nntmux is now reached over \
+                 its Newznab API; set `nntmux_api_key` (and `nntmux_url`) on the config page"
+            );
         }
+        if let Some(rows) = v.get("indexers").and_then(|x| x.as_array()) {
+            self.indexers = rows
+                .iter()
+                .filter_map(|row| serde_json::from_value::<IndexerCred>(row.clone()).ok())
+                .filter(|c| !c.host.trim().is_empty() && !c.api_key.trim().is_empty())
+                .collect();
         }
     }
 }
 
 pub struct UsenetPlugin {
     settings: Settings,
-    /// `None` until a successful connect. A feeder with unreachable config
+    /// `None` when no nntmux key is configured (grab-only deployment).
-    /// **must still serve `/health`** so the gateway's `depends_on` is satisfied
+    nntmux: Option<Arc<NntmuxApi>>,
-    /// — it soft-skips its upstream instead of failing to boot (invariant 10).
+    /// `.nzb` per release guid, under the plugin's state dir.
-    db: Arc<RwLock<Option<Db>>>,
+    cache: Option<Arc<NzbCache>>,
-    nzb_root: PathBuf,
+    /// Why nntmux is unusable right now (unreachable, key refused), for
+    /// `health()`. A feeder with unreachable config **must still serve
+    /// `/health`** — it soft-skips its upstream instead of failing to boot
+    /// (invariant 10).
+    nntmux_problem: Arc<RwLock<Option<String>>>,
+    /// Client for `.nzb` grabs (named User-Agent, follows redirects so an HTML
+    /// landing page is detected rather than surfacing as a bare 302).
+    http: reqwest::Client,
+    /// `https` in production — a locator carries no scheme and every public
+    /// Newznab API is TLS. Tests point it at a plain-http mock.
+    grab_scheme: &'static str,
 }
 
 impl Default for UsenetPlugin {
@@ -104,34 +169,119 @@ impl UsenetPlugin {
     pub fn new() -> Self {
         Self {
             settings: Settings::default(),
-            db: Arc::new(RwLock::new(None)),
+            nntmux: None,
-            nzb_root: PathBuf::new(),
+            cache: None,
+            nntmux_problem: Arc::new(RwLock::new(None)),
+            http: build_http_client(GRAB_TIMEOUT_SECS, USER_AGENT, None),
+            grab_scheme: "https",
         }
     }
 
-    async fn db(&self) -> Result<Db, GatewayError> {
+    /// Grab over plain `http` instead of `https`. For tests against a local
-        self.db
+    /// mock indexer only — a locator carries no scheme, and a real indexer is TLS.
-            .read()
+    pub fn with_plain_http_grabs(mut self) -> Self {
-            .await
+        self.grab_scheme = "http";
-            .clone()
+        self
-            .ok_or_else(|| GatewayError::Permanent("nntmux database not configured".into()))
     }
 
-    /// Mint a release's `nzb-posting` cid by reading its manifest off disk.
+    /// The configured key for an indexer authority, if any.
+    fn indexer_key(&self, authority: &str) -> Option<&str> {
+        let want = newznab::host_key(authority);
+        self.settings
+            .indexers
+            .iter()
+            .find(|c| newznab::host_key(&c.host) == want)
+            .map(|c| c.api_key.as_str())
+    }
+
+    /// Redeem an `nzb-release` locator: grab its `.nzb` with this feeder's key
+    /// for that host, strip the head (password kept), and mint the portable
+    /// `nzb-posting` cid from its Message-IDs.
     ///
-    /// `None` when the `.nzb.gz` is absent (a release row can exist before
+    /// ⚠ **Spends indexer quota.** Reached only through `/compute`, which the
-    /// `createNZBs()` has written it) or unreadable — the caller then emits a
+    /// gateway calls on a real play and never for a release it already holds
-    /// record without a cid, which is honest: we genuinely cannot address that
+    /// the manifest of. `NotFound` = "not mine" (no key for that host), so the
-    /// posting yet.
+    /// gateway can try another claimer without anything being spent.
-    fn posting_cid(&self, guid: &str) -> Option<String> {
+    async fn redeem_release(&self, cid: &str) -> Result<Vec<HashOutcome>, GatewayError> {
-        match nzb::load(&self.nzb_root, guid) {
+        let Some((api_base, id)) = decode_nzb_release_cid(cid) else {
-            Ok(Some(m)) => Some(compute_nzb_posting_cid(&m.message_ids)),
+            return Err(GatewayError::NotFound);
-            Ok(None) => None,
+        };
-            Err(e) => {
+        let host = newznab::authority(&api_base);
-                warn!(target: "meta-feeder", %guid, error = %e, "usenet: manifest unreadable; record will carry no cid");
+        let Some(key) = self.indexer_key(host) else {
-                None
+            debug!(target: "meta-feeder", %host, "usenet: no indexer key for this host; not ours to redeem");
+            return Err(GatewayError::NotFound);
+        };
+
+        info!(target: "meta-feeder", %host, %id, "usenet: grabbing .nzb (indexer quota)");
+        let raw = newznab::grab(&self.http, self.grab_scheme, &api_base, &id, key).await?;
+        let nzb = newznab::strip_head_keep_password(&raw)
+            .map_err(|e| GatewayError::Permanent(format!("nzb from {host} is not parseable: {e}")))?;
+        let message_ids = nzb::extract_message_ids(&nzb)
+            .map_err(|e| GatewayError::Permanent(format!("nzb from {host} is not parseable: {e}")))?;
+        if message_ids.is_empty() {
+            return Err(GatewayError::Permanent(format!(
+                "nzb from {host} has no segments"
+            )));
+        }
+
+        let posting = compute_nzb_posting_cid(&message_ids);
+        let mut fields = BTreeMap::new();
+        // Merged onto the release record by the gateway: the posting cid
+        // outranks the locator, so the release becomes redeemable by any peer
+        // with a plain NNTP provider — no indexer key anywhere after this grab.
+        fields.insert(format!("cids/{posting}"), "true".to_string());
+        fields.insert("segmentCount".to_string(), message_ids.len().to_string());
+
+        Ok(vec![HashOutcome {
+            hash: Hash(compute_ipfs_cid(&nzb)),
+            hash_kind: HashKind::Sha2_256,
+            bytes: Some(nzb.into()),
+            record: Some(DiscoveryRecord {
+                upstream_id: UPSTREAM_ID.to_string(),
+                record_id: cid.to_string(),
+                fields,
+            }),
+            file_extension: Some("nzb".to_string()),
+        }])
+    }
+
+    fn nntmux(&self) -> Result<(&NntmuxApi, &NzbCache), GatewayError> {
+        match (&self.nntmux, &self.cache) {
+            (Some(api), Some(cache)) => Ok((api.as_ref(), cache.as_ref())),
+            _ => Err(GatewayError::Permanent("no nntmux API key configured".into())),
         }
     }
+
+    fn note_problem(&self, e: &GatewayError) {
+        if let GatewayError::Permanent(msg) = e {
+            *self.nntmux_problem.write().unwrap() = Some(msg.clone());
+        }
+    }
+
+    /// A release's `.nzb`: from the cache, else downloaded from nntmux once
+    /// (one `downloadrequests` unit) and cached.
+    async fn manifest(&self, guid: &str) -> Result<Vec<u8>, GatewayError> {
+        let (api, cache) = self.nntmux()?;
+        if let Some(xml) = cache.get(guid) {
+            return Ok(xml);
+        }
+        let xml = api.nzb(guid).await.inspect_err(|e| self.note_problem(e))?;
+        if let Err(e) = cache.put(guid, &xml) {
+            warn!(target: "meta-feeder", %guid, error = %e, "usenet: could not cache the .nzb (will re-download)");
+        }
+        Ok(xml)
+    }
+
+    /// Mint a release's `nzb-posting` identity from its `.nzb`.
+    async fn posting(&self, guid: &str) -> Result<Posting, GatewayError> {
+        let (_, cache) = self.nntmux()?;
+        if let Some(p) = cache.known(guid) {
+            return Ok(p);
+        }
+        let xml = self.manifest(guid).await?;
+        cache
+            .posting(guid, &xml)
+            .ok_or_else(|| GatewayError::Permanent(format!("nzb of release {guid} has no segments")))
     }
 
     /// Build the discovery record for one release. Metadata only — the bytes
@@ -144,9 +294,9 @@ impl UsenetPlugin {
     /// title renders but is unplayable. `nzb-release` (`0x1005`) avoids this
     /// for free because its cid is derivable from `{host, id}` alone; ours is a
     /// digest over the Message-ID set, so it has to be minted from the manifest
-    /// — which is a **local file read**, no network and no indexer grab, and
+    /// — one `/getnzb` on our own nntmux the first time a release is seen, then
-    /// therefore fine to do per search hit. Caught by an end-to-end play from
+    /// the cache. Caught by an end-to-end play from meta-watch that landed on
-    /// meta-watch that landed on "Unavailable".
+    /// "Unavailable".
     fn record_for(&self, r: &Release, cid: Option<&str>) -> DiscoveryRecord {
         let mut fields: BTreeMap<String, String> = BTreeMap::new();
         // The bare-cid key-set (`cids/<cid>` = "true") — the ONLY shape
@@ -186,6 +336,16 @@ impl UsenetPlugin {
         );
         if let Some(kind) = content_kind_for_category(r.category_id) {
             fields.insert("contentKind".to_string(), kind.to_string());
+            // The routing axes, co-written with the kind (METADATA_KEYS.md §1):
+            // `fileType` says what the bytes are, `domain` says which app wants
+            // the record, `workForm` whether it stands alone or is one
+            // instalment. Without the domain the row never reaches a wall.
+            if let Some(d) = meta_feeder_sdk::domain::domain_for_content_kind(kind) {
+                fields.insert("domain".to_string(), d.to_string());
+            }
+            if let Some(wf) = meta_feeder_sdk::domain::work_form_for_content_kind(kind) {
+                fields.insert("workForm".to_string(), wf.to_string());
+            }
         }
         if let Some(ts) = r.post_date {
             fields.insert("publishedAt".to_string(), ts.to_string());
@@ -212,40 +372,49 @@ impl FeederPlugin for UsenetPlugin {
     fn configure(&mut self, cache_dir: &Path) -> Result<(), ConfigError> {
         let mut settings = Settings::from_env();
         settings.merge_file(cache_dir);
-        self.nzb_root = PathBuf::from(&settings.nzb_root);
 
-        if settings.db_url.is_empty() {
+        if settings.nntmux_api_key.is_empty() {
             // Soft-skip, not a hard failure: serve /health, report Degraded.
             warn!(
                 target: "meta-feeder",
-                "usenet: no nntmux database url configured — upstream will soft-skip \
+                "usenet: no nntmux API key configured — the self-scan search is off \
-                 (set it in the feeder's config page, or seed NNTMUX_DB_URL)"
+                 (set it on the feeder's config page: the NNTmux app's `metamesh` user key)"
             );
             self.settings = settings;
             return Ok(());
         }
 
-        // Connect eagerly so a bad url is visible at boot rather than on the
+        let api = Arc::new(NntmuxApi::new(self.http.clone(), &settings.nntmux_url, &settings.nntmux_api_key));
-        // first query, but never fatally — the sidecar may still be starting.
+        self.cache = Some(Arc::new(NzbCache::new(&cache_dir.join("nntmux-nzb"))));
-        let url = settings.db_url.clone();
+        info!(target: "meta-feeder", nntmux = api.base(), "usenet: searching nntmux over its Newznab API");
-        let slot = self.db.clone();
+
+        // Reachability, now and periodically — never fatal: nntmux may still be
+        // starting (it takes minutes on a large store).
+        let probe = Arc::clone(&api);
+        let problem = Arc::clone(&self.nntmux_problem);
         tokio::spawn(async move {
-            match Db::connect(&url).await {
+            loop {
-                Ok(db) => {
+                match probe.caps().await {
-                    match db.release_count().await {
+                    Ok(()) => {
-                        Ok(0) => warn!(
+                        let mut p = problem.write().unwrap();
-                            target: "meta-feeder",
+                        // A refused key is learned from real calls; only clear
-                            "usenet: nntmux catalog is EMPTY — check the scan actually ran \
+                        // what this probe can vouch for (reachability).
-                             (PATH_TO_NZBS must be non-empty, and groups need active=1)"
+                        if p.as_deref().is_some_and(|m| m.contains("unreachable") || m.contains("t=caps")) {
-                        ),
+                            *p = None;
-                        Ok(n) => info!(target: "meta-feeder", releases = n, "usenet: nntmux catalog connected"),
-                        Err(e) => warn!(target: "meta-feeder", error = %e, "usenet: catalog count failed"),
                         }
-                    *slot.write().await = Some(db);
                     }
-                Err(e) => warn!(target: "meta-feeder", error = %e, "usenet: nntmux connect failed; upstream soft-skips"),
+                    Err(e) => {
+                        warn!(target: "meta-feeder", nntmux = probe.base(), error = %e, "usenet: nntmux not reachable");
+                        *problem.write().unwrap() = Some(match e {
+                            GatewayError::Permanent(m) => m,
+                            other => format!("nntmux {} unreachable: {other}", probe.base()),
+                        });
+                    }
+                }
+                tokio::time::sleep(PROBE_INTERVAL).await;
             }
         });
+        self.nntmux = Some(api);
         self.settings = settings;
         Ok(())
     }
@@ -262,53 +431,70 @@ impl FeederPlugin for UsenetPlugin {
         if term.trim().is_empty() {
             return Ok(vec![]);
         }
-        let db = self.db().await?;
+        if self.nntmux.is_none() {
-        let releases = db
+            // Grab-only deployment (indexer keys, no nntmux): nothing to search.
+            return Ok(vec![]);
+        }
+        let (api, _) = self.nntmux()?;
+        let releases = api
             .search(term.trim(), max_results)
             .await
-            .map_err(|e| GatewayError::Transient(format!("nntmux search: {e}")))?;
+            .inspect_err(|e| self.note_problem(e))?;
+        *self.nntmux_problem.write().unwrap() = None;
 
-        // Mint each hit's cid here, from its on-disk manifest — see the
+        // Mint each hit's cid here — see the `record_for` doc for why this
-        // `record_for` doc for why this cannot wait for compute. A release
+        // cannot wait for compute. A release whose `.nzb` can't be had is
-        // whose manifest isn't written yet is DROPPED rather than surfaced
+        // DROPPED rather than surfaced uncid'd: an unplayable row in the
-        // uncid'd: an unplayable row in the client's "raw sources" list is
+        // client's "raw sources" list is worse than one fewer row. Once nntmux
-        // worse than one fewer row.
+        // says a daily cap is spent, stop asking for this query.
         let mut out = Vec::with_capacity(releases.len());
         let mut skipped = 0usize;
+        let mut capped = false;
         for r in &releases {
-            match self.posting_cid(&r.guid) {
+            if capped {
-                Some(cid) => out.push(self.record_for(r, Some(&cid))),
+                skipped += 1;
-                None => skipped += 1,
+                continue;
+            }
+            match self.posting(&r.guid).await {
+                Ok(p) => out.push(self.record_for(r, Some(&p.cid))),
+                Err(GatewayError::RateLimited { .. }) => {
+                    capped = true;
+                    skipped += 1;
+                    warn!(target: "meta-feeder", "usenet: nntmux download cap reached — remaining hits skipped (raise the feeder user's role caps)");
+                }
+                Err(e) => {
+                    skipped += 1;
+                    debug!(target: "meta-feeder", guid = %r.guid, error = %e, "usenet: release skipped (no usable .nzb)");
+                }
             }
         }
         debug!(
             target: "meta-feeder", %term, hits = out.len(), skipped,
-            "usenet: catalog search"
+            "usenet: nntmux search"
         );
-        if skipped > 0 {
-            debug!(
-                target: "meta-feeder", skipped,
-                "usenet: releases skipped — no .nzb.gz on disk yet (scan still collating?)"
-            );
-        }
         Ok(out)
     }
 
     async fn compute_outcomes(&self, record_id: &str) -> Result<Vec<HashOutcome>, GatewayError> {
-        let db = self.db().await?;
+        // An external release locator, not one of our nntmux guids. Checked
-        let release = db
+        // first: it needs no database.
-            .by_guid(record_id)
+        if codec_of(record_id) == Some(NZB_RELEASE_CODEC) {
+            return self.redeem_release(record_id).await;
+        }
+        if !is_guid(record_id) {
+            return Err(GatewayError::NotFound);
+        }
+        let (api, _) = self.nntmux()?;
+        let release = api
+            .details(record_id)
             .await
-            .map_err(|e| GatewayError::Transient(format!("nntmux lookup: {e}")))?
+            .inspect_err(|e| self.note_problem(e))?
             .ok_or(GatewayError::NotFound)?;
 
-        // The Message-IDs survive only in the on-disk `.nzb.gz` — nntmux purges
+        // The Message-IDs survive only in the release's `.nzb` — nntmux purges
-        // collections/binaries/parts once it has written the file.
+        // collections/binaries/parts once it has written it.
-        let manifest = nzb::load(&self.nzb_root, record_id)
+        let posting = self.posting(record_id).await?;
-            .map_err(|e| GatewayError::Permanent(format!("nzb read: {e}")))?
+        let cid = posting.cid;
-            .ok_or(GatewayError::NotFound)?;
-
-        let cid = compute_nzb_posting_cid(&manifest.message_ids);
 
         // Same cid the search path already published for this release — the
         // client is committing to a cid it saw in the results, so the two must
@@ -318,7 +504,7 @@ impl FeederPlugin for UsenetPlugin {
         // it must be on the search path, not only here.
         record
             .fields
-            .insert("segmentCount".to_string(), manifest.message_ids.len().to_string());
+            .insert("segmentCount".to_string(), posting.segment_count.to_string());
 
         Ok(vec![HashOutcome {
             hash: Hash(cid),
@@ -336,25 +522,52 @@ impl FeederPlugin for UsenetPlugin {
     /// names the stored blob correctly).
     async fn get_blob(&self, cid: &str) -> Option<Vec<u8>> {
         let guid = cid.strip_suffix(".nzb").unwrap_or(cid);
-        match nzb::load(&self.nzb_root, guid) {
+        if !is_guid(guid) {
-            Ok(Some(m)) => Some(m.xml),
+            return None;
-            Ok(None) => None,
+        }
+        // Normally a cache hit: the search that published `manifest_url` has
+        // already downloaded it.
+        match self.manifest(guid).await {
+            Ok(xml) => Some(xml),
             Err(e) => {
-                warn!(target: "meta-feeder", %guid, error = %e, "usenet: blob read failed");
+                warn!(target: "meta-feeder", %guid, error = %e, "usenet: .nzb unavailable for seeding");
                 None
             }
         }
     }
 
     fn health(&self) -> PluginHealth {
-        if self.settings.db_url.is_empty() {
+        if self.nntmux.is_none() && self.settings.indexers.is_empty() {
             return PluginHealth::Degraded {
-                reason: "nntmux database not configured".into(),
+                reason: "neither an nntmux API key nor an indexer key is configured".into(),
             };
         }
+        if let Some(problem) = self.nntmux_problem.read().unwrap().clone() {
+            return PluginHealth::Degraded { reason: problem };
+        }
         PluginHealth::Ok
     }
 
+    fn package(&self) -> Option<&'static str> {
+        Some(PACKAGE)
+    }
+
+    /// Claims `nzb-release` for every host with a key; nothing without keys.
+    fn redeems(&self) -> Vec<RedeemClaim> {
+        let mut hosts: Vec<String> = self
+            .settings
+            .indexers
+            .iter()
+            .map(|c| newznab::host_key(&c.host))
+            .collect();
+        hosts.sort();
+        hosts.dedup();
+        if hosts.is_empty() {
+            return Vec::new();
+        }
+        vec![RedeemClaim::nzb_release(hosts)]
+    }
+
     fn served_file_types(&self) -> &'static [&'static str] {
         &["video", "audio", "document", "archive"]
     }
@@ -366,34 +579,47 @@ impl FeederPlugin for UsenetPlugin {
     fn config_schema(&self) -> ConfigSchema {
         ConfigSchema {
             fields: vec![
-                F::text("db_url", "nntmux database URL")
+                F::text("nntmux_url", "nntmux URL")
                     .with_help(
-                        "MariaDB DSN of the nntmux sidecar's catalog, e.g. \
+                        "Base URL of your NNTmux app as this feeder reaches it (default \
-                         mysql://nntmux:secret@nntmux-db:3306/nntmux. Blank → the \
+                         http://nntmux, the app's container on the shared network). The \
-                         upstream soft-skips and this plugin reports Degraded. Takes \
+                         Newznab API path is added automatically. Takes effect on the next \
-                         effect on the next feeder restart (no hot reload).",
+                         feeder restart.",
-                    )
+                    ),
-                    .required(),
+                F::secret("nntmux_api_key", "nntmux API key")
-                F::text("nzb_root", "NZB store path")
                     .with_help(
-                        "Path to nntmux's NZB directory as mounted INTO this container \
+                        "API key of the nntmux user this feeder searches as — the NNTmux app's \
-                         — it must be the same volume nntmux's PATH_TO_NZBS points at. \
+                         `metamesh` user, whose role has raised daily caps (see the NNTmux app's \
-                         Each release's ordered Message-IDs survive only in the on-disk \
+                         install tips for how to read it). Blank → the self-scan search is off \
-                         <guid>.nzb.gz, so a wrong path here means every release \
+                         (indexer-key redeems still work).",
-                         resolves to NotFound. Point it at the STORE ROOT, not a shard: \
+                    ),
-                         nntmux nests each file one directory per leading guid character, \
+                F::record_array(
-                         `nzbsplitlevel` deep (the image ships 4 → 9/7/a/2/97a2….nzb.gz), \
+                    "indexers",
-                         and the depth is detected automatically.",
+                    "Indexer keys (redeem nzb-release)",
+                    vec![
+                        F::text("host", "Host").required().with_help(
+                            "The indexer's bare host, e.g. api.nzbgeek.info — no scheme, no /api. \
+                             Matched against the host inside each nzb-release cid.",
+                        ),
+                        F::secret("api_key", "API key").required().with_help(
+                            "The indexer account's API key. Every grab spends that account's \
+                             daily download quota — once per release, on a real play only.",
+                        ),
+                    ],
                 )
-                    .required(),
+                .with_help(
+                    "Newznab keys used to grab the .nzb of releases minted by the torznab feeder \
+                     (Prowlarr search). Takes effect on the next feeder restart.",
+                ),
             ],
         }
     }
 
     fn config_values(&self) -> serde_json::Value {
         serde_json::json!({
-            "db_url": self.settings.db_url,
+            "nntmux_url": self.settings.nntmux_url,
-            "nzb_root": self.settings.nzb_root,
+            "nntmux_api_key": self.settings.nntmux_api_key,
+            "indexers": self.settings.indexers,
         })
     }
 }

feeder-plugin/usenet-feeder/tests/contract.rs

@@ -7,8 +7,7 @@
 //! the soft-skip invariant every feeder must honour (gateway invariant 10: a
 //! feeder with insufficient config still serves `/health` so the gateway's
 //! `depends_on` is satisfied, rather than failing to boot). Live search/mint
-//! are not driven here — they need a running nntmux sidecar with real data;
+//! are driven against a mock nntmux in `tests/nntmux_api.rs`.
-//! that path is exercised against `docker-compose.feeder-usenet.yml`.
 
 use std::net::SocketAddr;
 
@@ -17,7 +16,7 @@ use usenet_feeder::usenet::UsenetPlugin;
 
 async fn boot_feeder() -> (SocketAddr, tempfile::TempDir) {
     let dir = tempfile::tempdir().expect("tempdir");
-    // Deliberately no NNTMUX_DB_URL / NNTMUX_NZB_PATH — the "operator hasn't
+    // Deliberately no NNTMUX_API_KEY — the "operator hasn't
     // configured this feeder yet" boot state. `configure()` must not error.
     let plugin = UsenetPlugin::new();
     let configured = configure_plugins(vec![Box::new(plugin)], dir.path()).expect("configure");
@@ -59,7 +58,7 @@ async fn manifest_advertises_usenet() {
 /// server reporting `degraded` — never a crash, never a hang, never `ok`
 /// (which would lie about being able to serve real content). This is what
 /// lets the gateway's `depends_on: condition: service_healthy` succeed even
-/// before an operator has filled in nntmux's db_url on the feeder's config
+/// before an operator has filled in the nntmux API key on the feeder's config
 /// page.
 #[tokio::test]
 async fn health_degrades_gracefully_with_no_nntmux_configured() {
@@ -86,11 +85,33 @@ async fn health_degrades_gracefully_with_no_nntmux_configured() {
         "usenet plugin health entry should report Degraded, got {usenet_health}"
     );
     assert_eq!(
-        usenet_health["health"]["reason"], "nntmux database not configured",
+        usenet_health["health"]["reason"],
+        "neither an nntmux API key nor an indexer key is configured",
         "got {usenet_health}"
     );
 }
 
+/// Unconfigured: the manifest still names the storage folder, but claims no
+/// redeems — there is no key to redeem anything with.
+#[tokio::test]
+async fn manifest_names_the_package_and_claims_nothing_without_keys() {
+    let (addr, _dir) = boot_feeder().await;
+    let manifest: serde_json::Value = reqwest::get(format!("http://{addr}/manifest"))
+        .await
+        .expect("manifest req")
+        .json()
+        .await
+        .expect("manifest json");
+    let usenet = manifest["plugins"]
+        .as_array()
+        .expect("plugins array")
+        .iter()
+        .find(|p| p["id"] == "usenet")
+        .expect("usenet entry");
+    assert_eq!(usenet["package"], "meta-feeder-usenet");
+    assert_eq!(usenet["redeems"], serde_json::json!([]));
+}
+
 /// Every seedable/resolvable field this feeder advertises resolving to must
 /// stay in sync with the gateway core's `SEEDABLE_FIELDS` table
 /// (`remote_feeder.rs`) — this is a documentation-level regression guard, not

feeder-plugin/usenet-feeder/tests/nntmux_api.rs

@@ -0,0 +1,240 @@
+//! The self-scan path against a mock nntmux, over its Newznab API.
+//!
+//! Boots the real `UsenetPlugin` in the SDK router with `nntmux_url` +
+//! `nntmux_api_key` in its `config.json` (the config-page shape) and a wiremock
+//! nntmux answering `/api/v1/api?t=…` and `/getnzb`, then drives `/query`,
+//! `/blob` and `/health` exactly as the gateway does.
+
+use meta_feeder_sdk::hash::compute_nzb_posting_cid;
+use meta_feeder_sdk::serve::{configure_plugins, router};
+use meta_feeder_sdk::{GatewayQuery, QueryRequest, QueryResponse};
+use usenet_feeder::usenet::UsenetPlugin;
+use wiremock::matchers::{method, path, query_param};
+use wiremock::{Mock, MockServer, ResponseTemplate};
+
+const KEY: &str = "metamesh-key";
+
+const CAPS: &str = r#"<?xml version="1.0" encoding="UTF-8"?><caps><limits max="100" default="100"/></caps>"#;
+
+fn item(guid: &str, title: &str, cat: u32) -> String {
+    format!(
+        r#"<item><title>{title}</title><guid isPermaLink="true">http://localhost/details/{guid}</guid>
+<newznab:attr name="category" value="{}"/><newznab:attr name="category" value="{cat}"/>
+<newznab:attr name="size" value="734003200"/>
+<newznab:attr name="usenetdate" value="Fri, 02 Oct 2026 08:30:00 +0000"/></item>"#,
+        cat / 1000 * 1000
+    )
+}
+
+fn rss(items: &[String]) -> String {
+    format!(
+        r#"<?xml version="1.0" encoding="UTF-8"?><rss version="2.0" xmlns:newznab="http://www.newznab.com/DTD/2010/feeds/attributes/"><channel><title>NNTmux</title>{}</channel></rss>"#,
+        items.join("")
+    )
+}
+
+const NZB: &str = r#"<?xml version="1.0" encoding="UTF-8"?>
+<nzb xmlns="http://www.newzbin.com/DTD/2003/nzb">
+  <file poster="[email protected]" date="1" subject="Ghost [1/1] &quot;a.mkv&quot; yEnc">
+    <groups><group>alt.binaries.teevee</group></groups>
+    <segments>
+      <segment bytes="100" number="1">[email protected]</segment>
+      <segment bytes="100" number="2">[email protected]</segment>
+    </segments>
+  </file>
+</nzb>"#;
+
+fn posting() -> String {
+    compute_nzb_posting_cid(&["[email protected]".to_string(), "[email protected]".to_string()])
+}
+
+async fn boot(nntmux: &MockServer) -> (String, tempfile::TempDir) {
+    let dir = tempfile::tempdir().expect("tempdir");
+    let cache = dir.path().join("gateway").join("usenet");
+    std::fs::create_dir_all(&cache).expect("mkdir");
+    std::fs::write(
+        cache.join("config.json"),
+        serde_json::json!({ "nntmux_url": nntmux.uri(), "nntmux_api_key": KEY }).to_string(),
+    )
+    .expect("write config");
+
+    let configured =
+        configure_plugins(vec![Box::new(UsenetPlugin::new())], dir.path()).expect("configure");
+    let app = router(configured, "test".to_string(), dir.path());
+    let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.expect("bind");
+    let addr = listener.local_addr().expect("addr");
+    tokio::spawn(async move {
+        axum::serve(listener, app).await.expect("serve");
+    });
+    (format!("http://{addr}"), dir)
+}
+
+async fn caps_ok(server: &MockServer) {
+    Mock::given(method("GET"))
+        .and(path("/api/v1/api"))
+        .and(query_param("t", "caps"))
+        .respond_with(ResponseTemplate::new(200).set_body_string(CAPS))
+        .mount(server)
+        .await;
+}
+
+async fn search_answers(server: &MockServer, body: String) {
+    Mock::given(method("GET"))
+        .and(path("/api/v1/api"))
+        .and(query_param("t", "search"))
+        .and(query_param("apikey", KEY))
+        .respond_with(ResponseTemplate::new(200).set_body_string(body))
+        .mount(server)
+        .await;
+}
+
+async fn query(base: &str, text: &str) -> reqwest::Response {
+    reqwest::Client::new()
+        .post(format!("{base}/query"))
+        .json(&QueryRequest {
+            upstream_id: "usenet".into(),
+            query: GatewayQuery::from_free_text(text),
+            max_results: 10,
+        })
+        .send()
+        .await
+        .expect("POST /query")
+}
+
+async fn records(base: &str, text: &str) -> Vec<meta_feeder_sdk::DiscoveryRecord> {
+    let r = query(base, text).await;
+    assert!(r.status().is_success(), "query status {}", r.status());
+    r.json::<QueryResponse>().await.expect("query json").records
+}
+
+/// Search hits carry their portable cid, minted from an `.nzb` downloaded ONCE:
+/// a repeat search and the core's `/blob` fetch are served from the cache.
+#[tokio::test]
+async fn hits_carry_the_posting_cid_and_each_nzb_is_downloaded_once() {
+    let server = MockServer::start().await;
+    caps_ok(&server).await;
+    search_answers(
+        &server,
+        rss(&[
+            item("abc111", "Celebrity.Ghost.Stories.S05E08.720p.HDTV.x264-DHD", 5040),
+            item("abc222", "Ghost.Not.Collated.Yet", 2040),
+        ]),
+    )
+    .await;
+    Mock::given(method("GET"))
+        .and(path("/getnzb"))
+        .and(query_param("id", "abc111.nzb"))
+        .and(query_param("r", KEY))
+        .respond_with(ResponseTemplate::new(200).set_body_string(NZB))
+        .expect(1)
+        .mount(&server)
+        .await;
+    Mock::given(method("GET"))
+        .and(path("/getnzb"))
+        .and(query_param("id", "abc222.nzb"))
+        .respond_with(ResponseTemplate::new(404))
+        .mount(&server)
+        .await;
+
+    let (base, _dir) = boot(&server).await;
+    let recs = records(&base, "ghost").await;
+    assert_eq!(recs.len(), 1, "a release without a usable .nzb is dropped, got {recs:?}");
+    let r = &recs[0];
+    assert_eq!(r.record_id, "abc111");
+    assert_eq!(r.fields.get(&format!("cids/{}", posting())).map(String::as_str), Some("true"));
+    assert_eq!(r.fields["manifest_url"], "/blob/usenet/abc111.nzb");
+    assert_eq!(r.fields["contentKind"], "episode");
+    assert_eq!(r.fields["fileType"], "video");
+    assert_eq!(r.fields["sizeByte"], "734003200");
+    assert_eq!(r.fields["publishedAt"], "1790929800");
+
+    // Repeat search: no second download (the `.expect(1)` above is verified on drop).
+    assert_eq!(records(&base, "ghost").await.len(), 1);
+
+    // The core seeds the manifest from /blob — served from the cache.
+    let blob = reqwest::get(format!("{base}/blob/usenet/abc111.nzb")).await.expect("blob");
+    assert!(blob.status().is_success());
+    assert_eq!(blob.text().await.unwrap(), NZB);
+}
+
+/// nntmux's daily download cap (`<error code="501">`, answered as HTTP 200):
+/// stop asking for the rest of this query instead of burning a request per hit.
+#[tokio::test]
+async fn a_spent_download_cap_stops_the_query_early() {
+    let server = MockServer::start().await;
+    caps_ok(&server).await;
+    search_answers(&server, rss(&[item("aaa1", "One", 2040), item("aaa2", "Two", 2040)])).await;
+    Mock::given(method("GET"))
+        .and(path("/getnzb"))
+        .respond_with(
+            ResponseTemplate::new(200)
+                .set_body_string(r#"<error code="501" description="Download limit reached"/>"#),
+        )
+        .expect(1)
+        .mount(&server)
+        .await;
+
+    let (base, _dir) = boot(&server).await;
+    assert!(records(&base, "one").await.is_empty());
+}
+
+/// A refused key (`<error code="100">`) fails the query and shows on /health.
+#[tokio::test]
+async fn a_refused_key_degrades_health() {
+    let server = MockServer::start().await;
+    caps_ok(&server).await;
+    search_answers(
+        &server,
+        r#"<error code="100" description="Incorrect user credentials (wrong API key)"/>"#.into(),
+    )
+    .await;
+
+    let (base, _dir) = boot(&server).await;
+    assert!(!query(&base, "ghost").await.status().is_success());
+
+    let health: serde_json::Value = reqwest::get(format!("{base}/health")).await.unwrap().json().await.unwrap();
+    let usenet = health["plugins"].as_array().unwrap().iter().find(|p| p["id"] == "usenet").unwrap();
+    assert_eq!(usenet["health"]["state"], "degraded", "{usenet}");
+    assert!(usenet["health"]["reason"].as_str().unwrap().contains("API key"), "{usenet}");
+}
+
+/// A blank query never reaches nntmux (the gateway's routing gate owns structure).
+#[tokio::test]
+async fn a_blank_query_is_answered_locally() {
+    let server = MockServer::start().await;
+    caps_ok(&server).await;
+    Mock::given(method("GET"))
+        .and(path("/api/v1/api"))
+        .and(query_param("t", "search"))
+        .respond_with(ResponseTemplate::new(200).set_body_string(rss(&[])))
+        .expect(0)
+        .mount(&server)
+        .await;
+
+    let (base, _dir) = boot(&server).await;
+    assert!(records(&base, "   ").await.is_empty());
+}
+
+/// A configured, reachable nntmux reports healthy.
+#[tokio::test]
+async fn a_reachable_nntmux_is_healthy() {
+    let server = MockServer::start().await;
+    caps_ok(&server).await;
+    let (base, _dir) = boot(&server).await;
+    // Let the boot-time t=caps probe land.
+    tokio::time::sleep(std::time::Duration::from_millis(300)).await;
+    let health: serde_json::Value = reqwest::get(format!("{base}/health")).await.unwrap().json().await.unwrap();
+    let usenet = health["plugins"].as_array().unwrap().iter().find(|p| p["id"] == "usenet").unwrap();
+    assert_eq!(usenet["health"]["state"], "ok", "{usenet}");
+}
+
+/// A path-traversal id never reaches the cache or nntmux.
+#[tokio::test]
+async fn a_hostile_blob_id_is_refused() {
+    let server = MockServer::start().await;
+    caps_ok(&server).await;
+    Mock::given(method("GET")).and(path("/getnzb")).respond_with(ResponseTemplate::new(200)).expect(0).mount(&server).await;
+    let (base, _dir) = boot(&server).await;
+    let r = reqwest::get(format!("{base}/blob/usenet/..%2F..%2Fconfig.json")).await.unwrap();
+    assert!(!r.status().is_success());
+}

feeder-plugin/usenet-feeder/tests/redeem.rs

@@ -0,0 +1,249 @@
+//! Redeeming an external `nzb-release` (`0x1005`) locator through `POST /compute`.
+//!
+//! Boots the real `UsenetPlugin` in the SDK router with indexer keys in its
+//! `config.json` (the config-page shape) and a wiremock Newznab indexer, then
+//! drives `/compute` with a locator cid exactly as the gateway's redeem route
+//! does. No nntmux: redeeming must work in a grab-only deployment.
+
+use base64::Engine as _;
+use meta_feeder_sdk::hash::{
+    compute_ipfs_cid, compute_nzb_posting_cid, compute_nzb_release_cid,
+};
+use meta_feeder_sdk::serve::{configure_plugins, router};
+use meta_feeder_sdk::{ComputeRequest, ComputeResponse, HashKindDto, RedeemsResponse};
+use usenet_feeder::usenet::UsenetPlugin;
+use wiremock::matchers::{header_exists, method, path, query_param};
+use wiremock::{Mock, MockServer, ResponseTemplate};
+
+const KEY: &str = "indexer-key-1";
+const RELEASE_ID: &str = "0123456789abcdef";
+
+const NZB: &str = r#"<?xml version="1.0" encoding="UTF-8"?>
+<nzb xmlns="http://www.newzbin.com/DTD/2003/nzb">
+  <head>
+    <meta type="title">Some.Release.1080p</meta>
+    <meta type="x-grabbed-by">account-4242</meta>
+    <meta type="password">hunter2</meta>
+  </head>
+  <file poster="[email protected]" date="1" subject="Some.Release [1/1] &quot;a.mkv&quot; yEnc">
+    <groups><group>alt.binaries.teevee</group></groups>
+    <segments>
+      <segment bytes="100" number="1">[email protected]</segment>
+      <segment bytes="100" number="2">[email protected]</segment>
+    </segments>
+  </file>
+</nzb>"#;
+
+/// The indexer's scheme-less API base as it appears inside a locator.
+fn authority(server: &MockServer) -> String {
+    server.uri().trim_start_matches("http://").to_string()
+}
+
+async fn boot(indexers: serde_json::Value) -> (String, tempfile::TempDir) {
+    let dir = tempfile::tempdir().expect("tempdir");
+    let cache = dir.path().join("gateway").join("usenet");
+    std::fs::create_dir_all(&cache).expect("mkdir");
+    std::fs::write(
+        cache.join("config.json"),
+        serde_json::json!({ "indexers": indexers }).to_string(),
+    )
+    .expect("write config");
+
+    let plugin = UsenetPlugin::new().with_plain_http_grabs();
+    let configured = configure_plugins(vec![Box::new(plugin)], dir.path()).expect("configure");
+    let app = router(configured, "test".to_string(), dir.path());
+    let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.expect("bind");
+    let addr = listener.local_addr().expect("addr");
+    tokio::spawn(async move {
+        axum::serve(listener, app).await.expect("serve");
+    });
+    (format!("http://{addr}"), dir)
+}
+
+async fn boot_with_key_for(server: &MockServer) -> (String, tempfile::TempDir) {
+    boot(serde_json::json!([{ "host": authority(server), "api_key": KEY }])).await
+}
+
+async fn compute(base: &str, record_id: &str) -> reqwest::Response {
+    reqwest::Client::new()
+        .post(format!("{base}/compute"))
+        .json(&ComputeRequest {
+            upstream_id: "usenet".into(),
+            record_id: record_id.into(),
+        })
+        .send()
+        .await
+        .expect("POST /compute")
+}
+
+fn grab_mock() -> wiremock::MockBuilder {
+    Mock::given(method("GET"))
+        .and(path("/api"))
+        .and(query_param("t", "get"))
+        .and(query_param("id", RELEASE_ID))
+        .and(query_param("apikey", KEY))
+}
+
+#[tokio::test]
+async fn a_release_locator_redeems_to_the_nzb_and_its_posting_cid() {
+    let indexer = MockServer::start().await;
+    grab_mock()
+        .and(header_exists("user-agent"))
+        .respond_with(ResponseTemplate::new(200).set_body_string(NZB))
+        .expect(1)
+        .mount(&indexer)
+        .await;
+    let (base, _dir) = boot_with_key_for(&indexer).await;
+    let cid = compute_nzb_release_cid(&authority(&indexer), RELEASE_ID).unwrap();
+
+    let resp = compute(&base, &cid).await;
+    assert_eq!(resp.status(), 200, "{:?}", resp.text().await);
+    let body: ComputeResponse = resp.json().await.expect("ComputeResponse");
+    assert_eq!(body.outcomes.len(), 1);
+    let o = &body.outcomes[0];
+    assert_eq!(o.hash_kind, HashKindDto::Sha2_256);
+    assert_eq!(o.file_extension.as_deref(), Some("nzb"));
+
+    let bytes = base64::engine::general_purpose::STANDARD
+        .decode(o.bytes_b64.as_deref().expect("bytes"))
+        .expect("base64");
+    assert_eq!(o.hash, compute_ipfs_cid(&bytes), "hash is the cid of the returned bytes");
+
+    let text = String::from_utf8(bytes).expect("utf-8");
+    assert!(text.contains(r#"<meta type="password">hunter2</meta>"#), "password kept: {text}");
+    assert!(!text.contains("account-4242"), "head metadata stripped: {text}");
+    assert!(text.contains("[email protected]") && text.contains("[email protected]"));
+
+    let record = o.record.as_ref().expect("fields to merge onto the release record");
+    let posting = compute_nzb_posting_cid(&["[email protected]", "[email protected]"]);
+    assert_eq!(record.fields.get(&format!("cids/{posting}")).map(String::as_str), Some("true"));
+    assert_eq!(record.fields.get("segmentCount").map(String::as_str), Some("2"));
+    assert!(
+        !record.fields.keys().any(|k| k == &format!("cids/{}", o.hash)),
+        "the manifest's own cid must never become a cids/ member of the release"
+    );
+}
+
+#[tokio::test]
+async fn a_host_without_a_key_is_not_ours_and_spends_nothing() {
+    let indexer = MockServer::start().await;
+    Mock::given(method("GET"))
+        .respond_with(ResponseTemplate::new(200).set_body_string(NZB))
+        .expect(0)
+        .mount(&indexer)
+        .await;
+    let (base, _dir) =
+        boot(serde_json::json!([{ "host": "other.example", "api_key": KEY }])).await;
+    let cid = compute_nzb_release_cid(&authority(&indexer), RELEASE_ID).unwrap();
+
+    assert_eq!(compute(&base, &cid).await.status(), 404);
+}
+
+#[tokio::test]
+async fn invalid_user_agent_error_is_permanent() {
+    let indexer = MockServer::start().await;
+    grab_mock()
+        .respond_with(ResponseTemplate::new(200).set_body_string(
+            r#"<?xml version="1.0" encoding="UTF-8"?><error code="109" description="Invalid User Agent"/>"#,
+        ))
+        .mount(&indexer)
+        .await;
+    let (base, _dir) = boot_with_key_for(&indexer).await;
+    let cid = compute_nzb_release_cid(&authority(&indexer), RELEASE_ID).unwrap();
+
+    let resp = compute(&base, &cid).await;
+    assert_eq!(resp.status(), 422);
+    assert!(resp.text().await.unwrap().contains("Invalid User Agent"));
+}
+
+#[tokio::test]
+async fn http_429_is_rate_limited() {
+    let indexer = MockServer::start().await;
+    grab_mock()
+        .respond_with(ResponseTemplate::new(429).insert_header("Retry-After", "120"))
+        .mount(&indexer)
+        .await;
+    let (base, _dir) = boot_with_key_for(&indexer).await;
+    let cid = compute_nzb_release_cid(&authority(&indexer), RELEASE_ID).unwrap();
+
+    assert_eq!(compute(&base, &cid).await.status(), 429);
+}
+
+#[tokio::test]
+async fn download_limit_error_code_is_rate_limited() {
+    let indexer = MockServer::start().await;
+    grab_mock()
+        .respond_with(ResponseTemplate::new(200).set_body_string(
+            r#"<error code="501" description="Download limit reached"/>"#,
+        ))
+        .mount(&indexer)
+        .await;
+    let (base, _dir) = boot_with_key_for(&indexer).await;
+    let cid = compute_nzb_release_cid(&authority(&indexer), RELEASE_ID).unwrap();
+
+    assert_eq!(compute(&base, &cid).await.status(), 429);
+}
+
+#[tokio::test]
+async fn an_html_page_is_a_permanent_error() {
+    let indexer = MockServer::start().await;
+    grab_mock()
+        .respond_with(
+            ResponseTemplate::new(200)
+                .set_body_string("<!DOCTYPE html><html><body>Please log in</body></html>"),
+        )
+        .mount(&indexer)
+        .await;
+    let (base, _dir) = boot_with_key_for(&indexer).await;
+    let cid = compute_nzb_release_cid(&authority(&indexer), RELEASE_ID).unwrap();
+
+    let resp = compute(&base, &cid).await;
+    assert_eq!(resp.status(), 422);
+    assert!(resp.text().await.unwrap().contains("HTML"));
+}
+
+/// Keys configured → the plugin claims `nzb-release` for exactly those hosts, on
+/// both `/redeems` and `/manifest`, and reports healthy without nntmux.
+#[tokio::test]
+async fn keys_are_advertised_as_redeem_claims() {
+    let (base, _dir) = boot(serde_json::json!([
+        { "host": "https://API.nzbgeek.info/", "api_key": "k1" },
+        { "host": "api.nzb.life", "api_key": "k2" },
+        { "host": "blank-key.example", "api_key": "" }
+    ]))
+    .await;
+
+    let r: RedeemsResponse = reqwest::get(format!("{base}/redeems"))
+        .await
+        .unwrap()
+        .json()
+        .await
+        .unwrap();
+    let claims = serde_json::to_value(&r.redeems["usenet"]).unwrap();
+    assert_eq!(
+        claims,
+        serde_json::json!([{
+            "codec": "nzb-release",
+            "field": "manifest",
+            "hosts": ["api.nzb.life", "api.nzbgeek.info"],
+            "sources": []
+        }])
+    );
+
+    let m: serde_json::Value = reqwest::get(format!("{base}/manifest"))
+        .await
+        .unwrap()
+        .json()
+        .await
+        .unwrap();
+    assert_eq!(m["plugins"][0]["package"], "meta-feeder-usenet");
+    assert_eq!(m["plugins"][0]["redeems"], claims);
+
+    let h: serde_json::Value = reqwest::get(format!("{base}/health"))
+        .await
+        .unwrap()
+        .json()
+        .await
+        .unwrap();
+    assert_eq!(h["status"], "ok", "{h}");
+}