change default

.github/workflows/publish.yml

@@ -5,7 +5,7 @@ name: Publish image
 #   * every `v*` tag       -> semver tags (e.g. v1.0.0 -> 1.0.0, 1.0, 1)
 # The build context is the repo root (the workspace) so the vendored
 # `meta-feeder-sdk` path dependency resolves; the Dockerfile lives under
-# feeder-plugin/torrent-feeder/.
+# feeder-plugin/tribler-feeder/.
 on:
   push:
     branches: [main]
@@ -51,7 +51,7 @@ jobs:
         uses: docker/build-push-action@v6
         with:
           context: .
-          file: ./feeder-plugin/torrent-feeder/Dockerfile
+          file: ./feeder-plugin/tribler-feeder/Dockerfile
           push: true
           tags: ${{ steps.meta.outputs.tags }}
           labels: ${{ steps.meta.outputs.labels }}

README.md

@@ -1 +1,128 @@
 # meta-feeder-tribler
+
+**Tribler feeder sidecar** for
+[MetaMesh](https://github.com/worph/meta-gateway) — bridges the
+**decentralized [Tribler](https://www.tribler.org/) network** (its IPv8
+peer-to-peer overlay search) into a meta-gateway as a search source.
+
+Unlike a Torznab/Prowlarr feeder that talks to centralized indexer sites,
+Tribler has **no central index**: it runs its own anonymous overlay and each
+peer contributes the torrents it has seen. This feeder does not reimplement
+that overlay — it runs a **headless Tribler core as a sidecar container**
+(`tribler-instance`) and queries its REST API, exactly the way a Torznab feeder
+treats an external indexer. It bridges Tribler's *content corpus* into the
+mesh so those torrents become discoverable and playable via meta-share.
+
+This feeder was **split out of** [`meta-feeder-torrent`](../meta-feeder-torrent)
+so that Tribler is a **single-plugin** feeder. A feeder exposes one `/config`
+surface; a multi-plugin feeder (Torznab + Tribler in one binary) can only route
+that surface to one plugin, so its dashboard config panel could not resolve to
+the Tribler schema. Splitting gives Tribler its own clean config card.
+
+## Metadata-only
+
+This is the **light** feeder. It resolves a Tribler search hit's infohash
+directly into a `btih-v1-file` CID and **advertises** the record — it does
+**not** download or seed the bytes. There is no `librqbit` here: file lists
+come from the Tribler core's own `torrentinfo` metainfo response, not from a
+BitTorrent fetch. So the cold Docker build is much lighter than the torrent
+feeder's, and full-file fetch/seed is left to meta-share's on-demand fetcher
+(the feeder synthesizes magnets with a set of well-known public trackers so
+that fetcher can find peers quickly, since Tribler torrents are often DHT-only).
+
+It carries a **copy** of the source-agnostic "torrent-core" TMDB
+catalog-discovery + enrichment stack (`tmdb`, `tmdb_budget`, `discovery`,
+`enrich`, `title`, `consts`, `filename_meta`). That stack cannot live in a
+container plugin — a container plugin enriches a *known* CID; it cannot answer
+"what's popular" for the home-row browse — so the browse/catalog primitive is
+inherently in-process and shared verbatim by every torrent source. Like every
+feeder it stays **meta-core-free at the transport layer and blockstore-free**:
+it finds records and enriches them; publishing resolved records + TMDB posters
+goes to the meta-core configured in the dashboard.
+
+## Role in MetaMesh
+
+A feeder is a stateless HTTP sidecar. The gateway registers it as a remote
+feeder plugin pointing at its `/` and drives the SDK contract
+([`meta_feeder_sdk::serve_feeders`](crates/meta-feeder-sdk/src/serve.rs)):
+
+| Endpoint | Purpose |
+|----------|---------|
+| `GET /manifest` | feeder identity + served types (`*` / `*` — Tribler is free-text, any query can return anything) |
+| `GET /health` | liveness (`Degraded` until `configure()` runs) |
+| `POST /query`, `POST /query_stream` | structured / streaming search over the Tribler local + remote (IPv8) overlay |
+| `POST /compute` | outcome compute + TMDB enrichment |
+| `GET /fetch/:upstream_id/:record_id`, `GET /blob/:upstream_id/:cid` | SDK byte routes (metadata-only here — no local bytes to serve) |
+| `POST /enrich/callback` | async enrichment callback sink |
+| `GET /config`, `GET /config/schema`, `GET\|PUT /config/values` | schema-driven config UI + API |
+
+## Configuration
+
+All Tribler config is **dashboard config only** — it lives in the persisted
+`config.json` (written via the gateway dashboard) and has **no env var**. This
+is the MetaMesh rule: a field in the web-UI config schema must not also be an
+env seed (config from JSON / web UI is the single source of truth).
+
+Configured on the plugin's card in the gateway dashboard:
+
+| Field | Notes |
+|-------|-------|
+| `meta_core_url` | meta-core this feeder publishes resolved records + TMDB posters to (e.g. `http://metacore-app:9000`). Blank → publish soft-skips. |
+| `sidecar_url` | Tribler core REST base (default `http://tribler-instance:8085`). |
+| `api_key` | Tribler REST `X-Api-Key` (secret); the dev sidecar uses `changeme`. Blank → no auth header. |
+| `tmdb_api_key` | TMDB v3 key / v4 bearer (secret) for poster / overview / tmdbid enrichment. Blank → TMDB enrichment soft-skips. |
+| `tmdb_language` | TMDB metadata language tag (e.g. `en-US`, `fr-FR`, `ja-JP`). Blank → `en-US`. |
+
+Config changes take effect on the next feeder restart (no hot reload). Open the
+Tribler web UI via the "↗ Tribler UI" link on the plugin's card — it opens the
+gated `/ui/` behind single sign-on (put `?key=` **after** the `#`; the UI is a
+HashRouter).
+
+Env vars are **infra only**:
+
+| Env var | Default | Notes |
+|---------|---------|-------|
+| `META_FEEDER_HTTP_LISTEN` | `0.0.0.0:8080` | HTTP listen address |
+| `META_FEEDER_STATE_DIR` | `/data/meta-feeder` | redb midhash cache + state |
+| `RUST_LOG` | `info` | tracing filter |
+
+## Image
+
+```
+ghcr.io/worph/meta-feeder-tribler
+```
+
+Exposes `8080`. Built and pushed by CI on every push to `main` (the moving
+`main` tag) and on `v*` tags (semver tags). Metadata-only (no `librqbit`), so
+the cold build is lighter than `meta-feeder-torrent`.
+
+## Build locally
+
+The build context is the **repo root** (the Cargo workspace) so the vendored
+`meta-feeder-sdk` path dependency resolves:
+
+```bash
+docker build -f feeder-plugin/tribler-feeder/Dockerfile -t ghcr.io/worph/meta-feeder-tribler:dev .
+```
+
+## Repo layout
+
+This repo is a self-contained Cargo workspace vendored out of the
+`meta-gateway` monorepo:
+
+```
+Cargo.toml                      # workspace: members = crates/*, feeder-plugin/*
+                                # (keeps the core2 [patch.crates-io] safety net)
+crates/meta-feeder-sdk/         # vendored shared feeder SDK
+feeder-plugin/tribler-feeder/   # this feeder's crate + Dockerfile
+    src/tribler/                # the Tribler source (IPv8-overlay REST client)
+    src/{tmdb,discovery,enrich,title,filename_meta,...}.rs
+                                # copied source-agnostic "torrent-core" catalog stack
+```
+
+> The workspace keeps the `core2` `[patch.crates-io]` override (a yanked
+> transitive dep) as a safety net, matching the `meta-gateway` workspace.
+
+Upstream source of truth for the SDK and the Tribler source is
+[`worph/meta-gateway`](https://github.com/worph/meta-gateway); changes there are
+vendored back into this repo.

crates/meta-feeder-sdk/src/hash.rs

@@ -160,33 +160,71 @@ fn bt_file_cid(codec: u64, infohash: &[u8], file_index: u64) -> String {
     format!("b{}", base32_lower_no_padding(&wire))
 }
 
-/// Custom multicodec for a generic **Newznab release** locator (`nzb-release`).
+/// Custom multicodec for a self-describing **Newznab release** locator
-/// MetaMesh-private, adjacent to the torrent-file codecs (`0x1002` is taken by
+/// (`nzb-release`). MetaMesh-private, adjacent to the torrent-file codecs in the
-/// [`BTIH_V2_FILE_CODEC`]). It identifies an indexer *listing* (host + release
+/// `0x10xx` range. Unlike a content hash, the cid *embeds* the indexer host +
-/// id), so it is derivable at search time with **no `.nzb` download**. See
+/// release id in an identity multihash, so the credentialed meta-share peer
-/// `USENET-GATEWAY-STUDY.md` §13. (The v1 content-stable `nzb-posting` `0x1003`
+/// decodes `{host,id}` straight from the cid and grabs the `.nzb` via `t=get`
-/// variant — `sha256(message-ids)` — was removed; only this one is emitted.)
+/// only at playback — no side-table, no `.nzb` download at search time. (The
-pub const NZB_RELEASE_CODEC: u64 = 0x1004;
+/// `0x1004` `sha256(host‖id)` opaque hash and the `0x1003` `nzb-posting`
+/// variant were both removed; only this self-describing form is emitted.)
+pub const NZB_RELEASE_CODEC: u64 = 0x1005;
 
-/// Encode an `nzb-release` CID from a Newznab indexer's host + bare release id
+/// Digest byte ceiling for an `nzb-release` cid, matching meta-share's
-/// (the 32-hex `<guid>` id). `sha2-256(host ‖ "\n" ‖ id)` — host-namespaced so
+/// `MAX_MULTIHASH_SIZE` — its `CidGeneric<64>` rejects a longer multihash.
-/// the same movie on two indexers (different postings) yields different cids.
+const NZB_RELEASE_MAX_DIGEST: usize = 64;
-/// Deterministic + opaque, same wire house-style as the other locator codecs.
-pub fn compute_nzb_release_cid(host: &str, release_id: &str) -> String {
-    let mut hasher = Sha256::new();
-    hasher.update(host.as_bytes());
-    hasher.update(b"\n");
-    hasher.update(release_id.as_bytes());
-    let digest: [u8; 32] = hasher.finalize().into();
 
-    // CIDv1: [version=0x01][codec varint][multihash code varint][len=0x20][digest]
+/// Encode a self-describing `nzb-release` CID from a Newznab indexer's host +
-    let mut wire = Vec::with_capacity(1 + 4 + 1 + 32);
+/// bare release id (the hex `<guid>` id). The multihash is *identity* (`0x00`)
+/// and its digest embeds the locator as `varint(host_len) ‖ host ‖ id_bytes`,
+/// where `id_bytes` is the hex-decoded id. meta-share's `decode_nzb_release_cid`
+/// is the exact inverse. Host-namespaced: the same release on two indexers
+/// yields different cids.
+///
+/// Returns `None` when the id isn't hex or the digest would overflow the
+/// multihash budget (an oversized host) — the caller drops that row.
+pub fn compute_nzb_release_cid(host: &str, release_id: &str) -> Option<String> {
+    let id_bytes = decode_hex_id(release_id)?;
+    let host_b = host.as_bytes();
+
+    // identity-multihash digest = varint(host_len) ‖ host ‖ id_bytes
+    let mut digest = Vec::with_capacity(2 + host_b.len() + id_bytes.len());
+    write_pb_varint(host_b.len() as u64, &mut digest);
+    digest.extend_from_slice(host_b);
+    digest.extend_from_slice(&id_bytes);
+    if digest.len() > NZB_RELEASE_MAX_DIGEST {
+        return None;
+    }
+
+    // CIDv1: [version=0x01][codec varint][mh code=0x00 identity][len varint][digest]
+    let mut wire = Vec::with_capacity(1 + 3 + 2 + digest.len());
     wire.push(0x01);
     write_pb_varint(NZB_RELEASE_CODEC, &mut wire);
-    write_pb_varint(NZB_RELEASE_CODEC, &mut wire);
+    wire.push(0x00); // multihash code: identity
-    wire.push(0x20);
+    write_pb_varint(digest.len() as u64, &mut wire);
     wire.extend_from_slice(&digest);
-    format!("b{}", base32_lower_no_padding(&wire))
+    Some(format!("b{}", base32_lower_no_padding(&wire)))
+}
+
+/// Hex-decode a Newznab release id. The guid parser only keeps hex runs, so the
+/// id is all-hex by construction; odd-length ids are left-padded with a `0`
+/// nibble. Returns `None` on any non-hex byte (defensive).
+fn decode_hex_id(id: &str) -> Option<Vec<u8>> {
+    let padded;
+    let s: &str = if id.len() % 2 == 1 {
+        padded = format!("0{id}");
+        &padded
+    } else {
+        id
+    };
+    let bytes = s.as_bytes();
+    let mut out = Vec::with_capacity(bytes.len() / 2);
+    for pair in bytes.chunks(2) {
+        let hi = (pair[0] as char).to_digit(16)?;
+        let lo = (pair[1] as char).to_digit(16)?;
+        out.push(((hi << 4) | lo) as u8);
+    }
+    Some(out)
 }
 
 /// Compute a standard IPFS CIDv1 over `bytes`. Output matches kubo's
@@ -477,26 +515,34 @@ mod tests {
 
     #[test]
     fn nzb_release_cid_wire_and_properties() {
-        let cid = compute_nzb_release_cid("api.nzb.life", "485c52f078b580667a01ac34a1cef6c2");
+        let cid =
+            compute_nzb_release_cid("api.nzb.life", "485c52f078b580667a01ac34a1cef6c2").unwrap();
         assert!(cid.starts_with('b'));
         // Deterministic.
         assert_eq!(
             cid,
-            compute_nzb_release_cid("api.nzb.life", "485c52f078b580667a01ac34a1cef6c2")
+            compute_nzb_release_cid("api.nzb.life", "485c52f078b580667a01ac34a1cef6c2").unwrap()
         );
         // Host-namespaced: same id, different host → different cid.
         assert_ne!(
             cid,
             compute_nzb_release_cid("api.nzbgeek.info", "485c52f078b580667a01ac34a1cef6c2")
+                .unwrap()
         );
-        // Explicit wire form: [01][84 20][84 20][20][sha256("host\nid")].
+        // Explicit self-describing wire form:
-        let digest: [u8; 32] =
+        //   [01][85 20][00][len][varint(host_len) ‖ host ‖ hex-decoded id].
-            Sha256::digest(b"api.nzb.life\n485c52f078b580667a01ac34a1cef6c2").into();
+        let mut digest = Vec::new();
-        let mut want = vec![0x01, 0x84, 0x20, 0x84, 0x20, 0x20];
+        write_pb_varint("api.nzb.life".len() as u64, &mut digest);
+        digest.extend_from_slice(b"api.nzb.life");
+        digest.extend_from_slice(&decode_hex_id("485c52f078b580667a01ac34a1cef6c2").unwrap());
+        let mut want = vec![0x01, 0x85, 0x20, 0x00];
+        write_pb_varint(digest.len() as u64, &mut want);
         want.extend_from_slice(&digest);
         assert_eq!(cid, format!("b{}", base32_lower_no_padding(&want)));
         // Distinct codec from the bt-file codecs in the 0x100x neighborhood.
         assert_ne!(cid, compute_bt_v1_file_cid(&[0xAB; 20], 0));
+        // An oversized host overflows the 64-byte digest budget → None.
+        assert!(compute_nzb_release_cid(&"h".repeat(80), "485c52f0").is_none());
     }
 
     #[test]

crates/meta-feeder-sdk/src/plugin.rs

@@ -58,9 +58,10 @@ pub struct HashOutcome {
 /// - `BtV1File` — a single file inside a BitTorrent v1 torrent, custom
 ///   multicodec `0x1001`. Locator, not content hash: opaque to bitswap.
 /// - `NzbRelease` — a Usenet release (a Newznab listing), custom multicodec
-///   `nzb-release` `0x1004`. Locator like `BtV1File`: opaque to bitswap,
+///   `nzb-release` `0x1005`. Self-describing locator: the cid embeds `{host,
-///   redeemed to bytes only by a credentialed meta-share peer, which grabs the
+///   id}` in an identity multihash, opaque to bitswap and redeemed to bytes
-///   `.nzb` via the `{base, id}` descriptor stored at `nzb:<cid>` KV.
+///   only by a credentialed meta-share peer, which decodes the locator from the
+///   cid and grabs the `.nzb` via `t=get` — no KV side-table.
 #[derive(Debug, Clone, Copy, PartialEq, Eq)]
 pub enum HashKind {
     /// Reserved: no feeder currently emits midhash256 outcomes, but the family

feeder-plugin/tribler-feeder/src/consts.rs

@@ -81,8 +81,8 @@ pub(crate) const TORRENT_RESOLVE_MAX_PER_INDEXER: usize = 30;
 /// it arrives until the gateway sends `Done`. (The 5 s `GATEWAY_CALL_TIMEOUT`
 /// is only the one-shot fallback for gateways that don't speak the streaming
 /// protocol.) So this can be generous: the binding constraint is meta-share's
-/// `SEARCH_TIMEOUT_SECS` silence window (set to match in the dev compose), not
+/// `SEARCH_TIMEOUT_SECS` silence window (300 s, the meta-search code default),
-/// a hard call timeout. 5 min is a safety ceiling on a pathological page;
+/// not a hard call timeout. 5 min is a safety ceiling on a pathological page;
 /// a normal XXX page (≤ `TORRENT_RESOLVE_MAX_PER_INDEXER` items at ~1.9 s
 /// each, 8-wide) finishes in well under 10 s and the gateway returns then.
 pub(crate) const TORRENT_RESOLVE_BUDGET: Duration = Duration::from_secs(300);