metastar/meta-feeder-usenet
4
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

improve compute API
Publish image / publish (push) Failing after 13s
Details

Browse Source
This commit is contained in:
Worph
2026-09-13 14:30:35 +02:00
parent 32d89e8354
commit b0c5a221a7
28 changed files with 935 additions and 7118 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Cargo.toml
+1 -1
View File
@@ -1,6 +1,6 @@
[workspace] [workspace]
resolver = "2" resolver = "2"
members = ["crates/*", "feeder-plugin/*"] members = ["feeder-plugin/*"]
[workspace.package] [workspace.package]
edition = "2021" edition = "2021"
README.md
+58 -6
View File
@@ -20,6 +20,13 @@ credential anywhere in the path**.
This feeder **adds** a Usenet metadata source alongside `meta-feeder-indexer`. This feeder **adds** a Usenet metadata source alongside `meta-feeder-indexer`.
It replaces nothing; a gateway query fans out to both. It replaces nothing; a gateway query fans out to both.
It also **redeems external releases.** The torznab feeder mints `nzb-release`
(`0x1005`) locators from Prowlarr/Newznab search hits; when a viewer plays one,
the gateway asks this feeder (`POST /compute` with the locator cid) to grab its
`.nzb` with the key configured here for that indexer host. The Newznab keys live
on this feeder — meta-share keeps only the NNTP provider. See
[Redeeming nzb-release locators](#redeeming-nzb-release-locators).
Design record: Design record:
`meta-gateway/docs/others/self-hosted-usenet-indexer-study.md`. `meta-gateway/docs/others/self-hosted-usenet-indexer-study.md`.
@@ -61,8 +68,12 @@ and there is no hot reload.
| Field | Env seed | What | | Field | Env seed | What |
|---|---|---| |---|---|---|
| `db_url` | `NNTMUX_DB_URL` | MariaDB DSN of nntmux's catalog, e.g. `mysql://nntmux:secret@nntmux-db:3306/nntmux`. Blank → the upstream soft-skips and the plugin reports Degraded. | | `db_url` | `NNTMUX_DB_URL` | MariaDB DSN of nntmux's catalog, e.g. `mysql://nntmux:secret@nntmux-db:3306/nntmux`. Blank → self-scan search is off. |
| `nzb_root` | `NNTMUX_NZB_PATH` | nntmux's NZB directory **as mounted into this container**. Must be the same volume nntmux's `PATH_TO_NZBS` points at, or every release resolves to `NotFound`. | | `nzb_root` | `NNTMUX_NZB_PATH` | nntmux's NZB directory **as mounted into this container**. Must be the same volume nntmux's `PATH_TO_NZBS` points at, or every release resolves to `NotFound`. |
| `indexers` | — (config page only) | `[{host, api_key}]` Newznab keys for redeeming `nzb-release` locators. `host` is the bare authority (`api.nzbgeek.info` — no scheme, no `/api`). |
Both halves are optional; with neither configured the plugin reports Degraded.
A grab-only deployment (keys, no nntmux) is valid.
Infra env: `META_FEEDER_HTTP_LISTEN` (default `0.0.0.0:8080`), Infra env: `META_FEEDER_HTTP_LISTEN` (default `0.0.0.0:8080`),
`META_FEEDER_STATE_DIR` (default `/data/meta-feeder`), `RUST_LOG`. `META_FEEDER_STATE_DIR` (default `/data/meta-feeder`), `RUST_LOG`.
@@ -73,6 +84,32 @@ Keep the connection budget conservative: the scanner and playback draw on the
same provider slots, so a greedy value does not merely over-scan, it stalls same provider slots, so a greedy value does not merely over-scan, it stalls
playback mid-stream. playback mid-stream.
## Redeeming nzb-release locators
`POST /compute {upstream_id: "usenet", record_id: <0x1005 cid>}`:
1. Decode `{api_base, id}` from the cid; find the key for its host. No key →
`404` (not ours — the gateway tries another claimer, nothing is spent).
2. `GET https://{api_base}/api?t=get&id=…&apikey=…` with a named User-Agent
(nzbgeek answers `109 Invalid User Agent` without one). **This spends the
indexer's daily download quota** — the gateway only calls it on a real play,
and never again for a release whose `manifest` it already stored.
3. Strip the `.nzb` `<head>` except `<meta type="password">` (the file is seeded
to the whole mesh; the password is needed to unpack a protected RAR set).
4. Mint the portable `nzb-posting` (`0x1003`) cid from its Message-IDs.
5. Answer ONE `sha2_256` outcome: the `.nzb` bytes (`file_extension: nzb`) and a
record carrying `cids/<nzb-posting>` + `segmentCount`. The gateway stores the
file under `/files/plugin/meta-feeder-usenet/`, writes its cid onto the release
record as `manifest`, and merges the posting cid — after which any peer with a
plain NNTP provider can play the release.
Errors: an indexer credential/account refusal (`100`–`102`, `910`) or an HTML
page → `422`; request/download limit (`429`, `500`, `501`, or HTTP 429) → `429`;
no such item (`300`) → `404`.
`GET /redeems` and `/manifest` advertise `{codec: nzb-release, field: manifest,
hosts: [...]}` for the hosts that have a key; nothing without keys.
## Build ## Build
```bash ```bash
@@ -83,11 +120,26 @@ cargo clippy --workspace --all-targets -- -D warnings
docker build -f feeder-plugin/usenet-feeder/Dockerfile -t ghcr.io/worph/meta-feeder-usenet:dev . docker build -f feeder-plugin/usenet-feeder/Dockerfile -t ghcr.io/worph/meta-feeder-usenet:dev .
``` ```
`crates/meta-feeder-sdk/` is a **vendored copy** of the SDK, mirrored from `meta-feeder-sdk` is a git dependency pinned by tag (`v1.3.0`; this repo used to
`meta-gateway`. Independent submodules cannot share a cargo path-dep, so keep it vendor a copy). Until that tag is pushed, cargo cannot resolve it — not even
in step when the SDK changes — in particular `hash.rs`'s under a `[patch]` override. To verify against a local SDK checkout, build a
`compute_nzb_posting_cid`, whose normalisation rule is the dedup contract and is scratch copy with the dependency swapped for a path (never commit that form —
mirrored again on the meta-share side. this repo is the Docker build context):
```bash
SCRATCH=/d/workspace/tmp-claude/usenet-feeder; SDK=/d/workspace/MetaMesh/meta-root-v2/packages/plugins/meta-feeder-sdk
rm -rf $SCRATCH && mkdir -p $SCRATCH && tar cf - --exclude=./target --exclude=./.git . | (cd $SCRATCH && tar xf -)
sed -i "s#meta-feeder-sdk = { git = .*#meta-feeder-sdk = { path = \"$SDK\" }#" $SCRATCH/feeder-plugin/*/Cargo.toml
docker run --rm -e RUSTUP_TOOLCHAIN=1.89.0 -e CARGO_TARGET_DIR=/target -v /d/workspace:/d/workspace \
-v feeder-cargo-registry:/usr/local/cargo/registry -v usenet-feeder-target:/target -w $SCRATCH \
rust:1.89 sh -c 'apt-get install -y -qq pkg-config libssl-dev >/dev/null 2>&1; cargo test'
```
After the tag is pushed, run `cargo update -p meta-feeder-sdk` here so
`Cargo.lock` stops pointing at the removed vendored path.
`compute_nzb_posting_cid`'s normalisation rule is the dedup contract and is
mirrored on the meta-share side — change them together.
## Gotchas ## Gotchas
crates/meta-feeder-sdk/Cargo.toml
-49
View File
@@ -1,49 +0,0 @@
[package]
name = "meta-feeder-sdk"
version = "0.1.0"
edition.workspace = true
rust-version.workspace = true
license.workspace = true
publish = false
# Source-agnostic foundation shared by the gateway core and every feeder
# sidecar binary: the `FeederPlugin` trait, the content-addressing hash
# families, the per-plugin redb cache, the shared HTTP/cache helpers, the
# structured-query evaluator, and the axum `serve_feeders` harness that wraps
# any set of `FeederPlugin`s in the feeder HTTP contract.
#
# Deliberately libp2p-free / blockstore-free: a feeder finds + fetches bytes;
# the gateway core owns hashing-into-the-blockstore, the meta-core store-back,
# and the libp2p wire. Keep heavy deps (librqbit, beetswap, cid, libp2p) OUT
# of here so feeders that don't need them don't pay for them.
[dependencies]
# Async runtime + HTTP server (the serve_feeders harness)
tokio = { version = "1.40", features = ["full"] }
axum = { version = "0.7", features = ["json"] }
hyper = { version = "1", features = ["full"] }
reqwest = { version = "0.12", features = ["json", "rustls-tls", "stream"], default-features = false }
# Serialization
serde = { version = "1", features = ["derive"] }
serde_json = "1"
base64 = "0.22"
# Utilities
futures = "0.3"
async-trait = "0.1"
anyhow = "1"
thiserror = "1"
tracing = "0.1"
sha2 = "0.10"
bytes = "1"
# Title-tag parsing (shared filename_meta — season/episode/quality/codec/languages).
regex = "1"
# Per-plugin midhash/blob cache. Schema is identical to meta-share v1 — see
# cache.rs. Pure-Rust single-process embedded store.
redb = "2"
[dev-dependencies]
tempfile = "3"
tokio = { version = "1.40", features = ["full", "test-util"] }
crates/meta-feeder-sdk/src/cache.rs
-353
View File
@@ -1,353 +0,0 @@
//! Per-plugin midhash cache backed by redb.
//!
//! Each plugin gets its own redb file at
//! `<state_dir>/gateway/<upstream_id>/cache.redb`. Keys are upstream
//! `record_id`s (DOI for sci-hub, book id for gutenberg, …); values are the
//! `Midhash` strings the plugin computed by fetching + hashing the upstream
//! file.
//!
//! The schema is identical to meta-share v1's so old `.redb` files copy
//! across. redb is synchronous and sub-millisecond for small ops, so calls
//! run inline from async handlers without `spawn_blocking`.
use std::path::Path;
use std::sync::Arc;
use redb::{Database, ReadableTableMetadata, TableDefinition};
pub const CACHE_FILENAME: &str = "cache.redb";
const MIDHASH_TABLE: TableDefinition<'_, &str, &str> = TableDefinition::new("midhash");
const BLOBS_TABLE: TableDefinition<'_, &str, &[u8]> = TableDefinition::new("blobs");
const COVER_CID_TABLE: TableDefinition<'_, &str, &str> = TableDefinition::new("cover_cid");
const BIBREC_TABLE: TableDefinition<'_, &str, &str> = TableDefinition::new("bibrec");
const PREVIEW_CID_TABLE: TableDefinition<'_, &str, &str> = TableDefinition::new("preview_cid");
/// Full-file sha2-256 IPFS cid for a previously-fetched torrent payload
/// (torznab's bt-fetch path). Distinct from `MIDHASH_TABLE` because the
/// hash families differ: `midhash` is the synthetic midhash256-from-infohash
/// fallback; `fullhash` records "we actually downloaded the bytes and this is
/// the IPFS CIDv1 over them."
const FULLHASH_TABLE: TableDefinition<'_, &str, &str> = TableDefinition::new("fullhash");
/// Enumerated torrent file list (JSON `Vec<TorrentFile>`) keyed by the
/// torrent's `record_id` (infohash).
const FILELIST_TABLE: TableDefinition<'_, &str, &str> = TableDefinition::new("filelist");
/// Cached TMDB *search* results keyed by a stable lookup key. The value is the
/// JSON encoding of the top `TmdbHit`, or the literal `"null"` negative-cache
/// sentinel.
const TMDB_SEARCH_TABLE: TableDefinition<'_, &str, &str> = TableDefinition::new("tmdb_search");
/// Cached TMDB `GET /3/tv/{id}` structural details keyed by `tmdbid`.
const TMDB_TVDETAILS_TABLE: TableDefinition<'_, &str, &str> =
TableDefinition::new("tmdb_tvdetails");
/// Cached TMDB `GET /3/{tv,movie}/{id}/external_ids` keyed by `tmdbid`.
const TMDB_EXTIDS_TABLE: TableDefinition<'_, &str, &str> = TableDefinition::new("tmdb_extids");
/// Cached TMDB `GET /3/movie/{id}` details keyed by `tmdbid`.
const TMDB_MOVIEDETAILS_TABLE: TableDefinition<'_, &str, &str> =
TableDefinition::new("tmdb_moviedetails");
/// Cached **ranked** TMDB `search/multi` anchors keyed by the normalized
/// free-text query.
const TMDB_PRINCIPAL_TOPN_TABLE: TableDefinition<'_, &str, &str> =
TableDefinition::new("tmdb_principal_topn");
/// Subtitle linkage discovered at search-enrich time, keyed by the torrent's
/// `record_id` (infohash). Value is the JSON encoding of a `Vec<SubtitleLink>`.
const SUBTITLES_TABLE: TableDefinition<'_, &str, &str> = TableDefinition::new("subtitles");
/// OpenSubtitles search results keyed by `"<tmdb_id>\x01<lang3>"`. Value is the
/// resolved subtitle's cid, or the literal `"null"` negative-cache sentinel.
const OPENSUBTITLES_TABLE: TableDefinition<'_, &str, &str> = TableDefinition::new("opensubtitles");
/// Small durable plugin scratch values that must survive a restart but aren't
/// content-derived — today, the torznab plugin's Prowlarr indexer-set
/// fingerprint, so a set that changed *while the feeder was down* is still
/// detected as a change on the next boot. Keys are plugin-namespaced
/// (`prowlarr:indexer-set`). redb creates the table on open, so an older
/// `.redb` file gains it transparently.
const MISC_TABLE: TableDefinition<'_, &str, &str> = TableDefinition::new("misc");
/// Per-plugin midhash cache. Cheap to clone (the inner `Database` is shared
/// via `Arc`).
#[derive(Clone)]
pub struct MidhashCache {
db: Arc<Database>,
}
/// Generate the standard `get`/`put` accessor pair for a `&str → &str` redb
/// table.
macro_rules! str_table_accessors {
($(#[$gmeta:meta])* $get:ident, $(#[$pmeta:meta])* $put:ident, $table:ident $(,)?) => {
$(#[$gmeta])*
pub fn $get(&self, key: &str) -> Result<Option<String>, redb::Error> {
let tx = self.db.begin_read()?;
let table = tx.open_table($table)?;
Ok(table.get(key)?.map(|v| v.value().to_string()))
}
$(#[$pmeta])*
pub fn $put(&self, key: &str, value: &str) -> Result<(), redb::Error> {
let tx = self.db.begin_write()?;
{
let mut table = tx.open_table($table)?;
table.insert(key, value)?;
}
tx.commit()?;
Ok(())
}
};
}
impl MidhashCache {
pub fn open(cache_dir: &Path) -> Result<Self, redb::Error> {
let path = cache_dir.join(CACHE_FILENAME);
let db = Database::create(&path)?;
{
let tx = db.begin_write()?;
tx.open_table(MIDHASH_TABLE)?;
tx.open_table(BLOBS_TABLE)?;
tx.open_table(COVER_CID_TABLE)?;
tx.open_table(BIBREC_TABLE)?;
tx.open_table(PREVIEW_CID_TABLE)?;
tx.open_table(FULLHASH_TABLE)?;
tx.open_table(FILELIST_TABLE)?;
tx.open_table(TMDB_SEARCH_TABLE)?;
tx.open_table(TMDB_TVDETAILS_TABLE)?;
tx.open_table(TMDB_EXTIDS_TABLE)?;
tx.open_table(TMDB_MOVIEDETAILS_TABLE)?;
tx.open_table(TMDB_PRINCIPAL_TOPN_TABLE)?;
tx.open_table(SUBTITLES_TABLE)?;
tx.open_table(OPENSUBTITLES_TABLE)?;
tx.open_table(MISC_TABLE)?;
tx.commit()?;
}
Ok(Self { db: Arc::new(db) })
}
str_table_accessors!(get_midhash, put_midhash, MIDHASH_TABLE);
str_table_accessors!(
/// Read a durable plugin scratch value (see [`MISC_TABLE`]).
get_misc,
/// Write a durable plugin scratch value (see [`MISC_TABLE`]).
put_misc,
MISC_TABLE,
);
pub fn entry_count(&self) -> Result<u64, redb::Error> {
let tx = self.db.begin_read()?;
let table = tx.open_table(MIDHASH_TABLE)?;
Ok(table.len()?)
}
pub fn get_blob(&self, cid: &str) -> Result<Option<Vec<u8>>, redb::Error> {
let tx = self.db.begin_read()?;
let table = tx.open_table(BLOBS_TABLE)?;
Ok(table.get(cid)?.map(|v| v.value().to_vec()))
}
pub fn put_blob(&self, cid: &str, bytes: &[u8]) -> Result<(), redb::Error> {
let tx = self.db.begin_write()?;
{
let mut table = tx.open_table(BLOBS_TABLE)?;
table.insert(cid, bytes)?;
}
tx.commit()?;
Ok(())
}
str_table_accessors!(get_cover_cid, put_cover_cid, COVER_CID_TABLE);
str_table_accessors!(
/// Read a previously-stored full-file IPFS cid for `record_id`.
get_fullhash,
/// Record the IPFS cid produced by a successful full fetch.
put_fullhash,
FULLHASH_TABLE
);
str_table_accessors!(
/// Read the cached torrent file list (JSON) for `record_id`.
get_filelist,
/// Cache the enumerated torrent file list (JSON `Vec<TorrentFile>`).
put_filelist,
FILELIST_TABLE
);
str_table_accessors!(
/// Read the cached subtitle linkage (JSON `Vec<SubtitleLink>`).
get_subtitles,
/// Cache the discovered subtitle linkage (JSON `Vec<SubtitleLink>`).
put_subtitles,
SUBTITLES_TABLE
);
str_table_accessors!(
/// Read a cached OpenSubtitles lookup by `"<tmdb_id>\x01<lang3>"`.
get_opensubtitles,
/// Cache an OpenSubtitles lookup result (cid or `"null"` sentinel).
put_opensubtitles,
OPENSUBTITLES_TABLE
);
str_table_accessors!(get_preview_cid, put_preview_cid, PREVIEW_CID_TABLE);
str_table_accessors!(
/// Read a cached TMDB search result by lookup key.
get_tmdb_search,
/// Cache a TMDB search result under `key`.
put_tmdb_search,
TMDB_SEARCH_TABLE
);
str_table_accessors!(
/// Read cached TMDB TV-details JSON by `tmdbid`.
get_tmdb_tvdetails,
/// Cache TMDB TV-details JSON under `tmdbid`.
put_tmdb_tvdetails,
TMDB_TVDETAILS_TABLE
);
str_table_accessors!(
/// Read cached TMDB external-ids JSON by `tmdbid`.
get_tmdb_extids,
/// Cache TMDB external-ids JSON under `tmdbid`.
put_tmdb_extids,
TMDB_EXTIDS_TABLE
);
str_table_accessors!(
/// Read cached TMDB movie-details JSON by `tmdbid`.
get_tmdb_moviedetails,
/// Cache TMDB movie-details JSON under `tmdbid`.
put_tmdb_moviedetails,
TMDB_MOVIEDETAILS_TABLE
);
str_table_accessors!(
/// Read the cached ranked anchor list by normalized query key.
get_tmdb_principal_topn,
/// Cache the ranked anchor list under the normalized query key.
put_tmdb_principal_topn,
TMDB_PRINCIPAL_TOPN_TABLE
);
pub fn get_bibrec(
&self,
record_id: &str,
) -> Result<Option<std::collections::BTreeMap<String, String>>, redb::Error> {
let tx = self.db.begin_read()?;
let table = tx.open_table(BIBREC_TABLE)?;
let Some(v) = table.get(record_id)? else {
return Ok(None);
};
let s = v.value();
Ok(serde_json::from_str(s).ok())
}
pub fn put_bibrec(
&self,
record_id: &str,
fields: &std::collections::BTreeMap<String, String>,
) -> Result<(), redb::Error> {
let json = serde_json::to_string(fields).unwrap_or_else(|_| "{}".to_string());
let tx = self.db.begin_write()?;
{
let mut table = tx.open_table(BIBREC_TABLE)?;
table.insert(record_id, json.as_str())?;
}
tx.commit()?;
Ok(())
}
}
#[cfg(test)]
mod tests {
use super::*;
fn tmp_cache_dir() -> tempfile::TempDir {
tempfile::tempdir().expect("tempdir")
}
#[test]
fn put_then_get_returns_value() {
let dir = tmp_cache_dir();
let cache = MidhashCache::open(dir.path()).expect("open");
assert_eq!(cache.get_midhash("10.1038/x").unwrap(), None);
cache.put_midhash("10.1038/x", "bafyXYZ").expect("put");
assert_eq!(
cache.get_midhash("10.1038/x").unwrap().as_deref(),
Some("bafyXYZ")
);
}
#[test]
fn put_overwrites_existing_value() {
let dir = tmp_cache_dir();
let cache = MidhashCache::open(dir.path()).expect("open");
cache.put_midhash("k", "v1").unwrap();
cache.put_midhash("k", "v2").unwrap();
assert_eq!(cache.get_midhash("k").unwrap().as_deref(), Some("v2"));
}
#[test]
fn values_persist_across_reopens() {
let dir = tmp_cache_dir();
{
let cache = MidhashCache::open(dir.path()).expect("open 1");
cache.put_midhash("k", "persistent").unwrap();
}
let cache = MidhashCache::open(dir.path()).expect("open 2");
assert_eq!(
cache.get_midhash("k").unwrap().as_deref(),
Some("persistent")
);
}
#[test]
fn tmdb_search_cache_distinguishes_miss_from_absent() {
let dir = tmp_cache_dir();
let cache = MidhashCache::open(dir.path()).expect("open");
assert_eq!(cache.get_tmdb_search("m\u{1}naruto\u{1}").unwrap(), None);
cache.put_tmdb_search("m\u{1}naruto\u{1}", "null").unwrap();
assert_eq!(
cache
.get_tmdb_search("m\u{1}naruto\u{1}")
.unwrap()
.as_deref(),
Some("null")
);
cache
.put_tmdb_search("t\u{1}spy x family\u{1}2019", "{\"tmdbid\":120089}")
.unwrap();
drop(cache);
let cache = MidhashCache::open(dir.path()).expect("reopen");
assert_eq!(
cache
.get_tmdb_search("t\u{1}spy x family\u{1}2019")
.unwrap()
.as_deref(),
Some("{\"tmdbid\":120089}")
);
}
#[test]
fn tmdb_tvdetails_cache_roundtrips() {
let dir = tmp_cache_dir();
let cache = MidhashCache::open(dir.path()).expect("open");
assert_eq!(cache.get_tmdb_tvdetails("120089").unwrap(), None);
cache
.put_tmdb_tvdetails("120089", "{\"number_of_seasons\":2}")
.unwrap();
assert_eq!(
cache.get_tmdb_tvdetails("120089").unwrap().as_deref(),
Some("{\"number_of_seasons\":2}")
);
}
#[test]
fn entry_count_reflects_inserts() {
let dir = tmp_cache_dir();
let cache = MidhashCache::open(dir.path()).expect("open");
assert_eq!(cache.entry_count().unwrap(), 0);
cache.put_midhash("a", "x").unwrap();
cache.put_midhash("b", "y").unwrap();
assert_eq!(cache.entry_count().unwrap(), 2);
cache.put_midhash("a", "x2").unwrap();
assert_eq!(cache.entry_count().unwrap(), 2);
}
}
crates/meta-feeder-sdk/src/common.rs
-288
View File
@@ -1,288 +0,0 @@
//! Shared helpers for single-outcome source plugins (`arxiv`, `gutenberg`,
//! `pubmed`, `wikicommons`, `giphy`, `scihub`).
//!
//! These plugins are structurally near-identical: build a reqwest client, map
//! HTTP status → [`GatewayError`], and front a per-record midhash cache whose
//! hit/put boilerplate is the same everywhere. Only the duplicated scaffolding
//! lives here; the genuinely-different bodies stay in the plugin modules.
use std::path::Path;
use std::time::Duration;
use tracing::{debug, warn};
use crate::cache::MidhashCache;
use crate::plugin::{ConfigError, HashKind, HashOutcome};
use crate::types::{DiscoveryRecord, GatewayError, Hash};
/// Build the shared reqwest client every source plugin uses.
pub fn build_http_client(
timeout_secs: u64,
user_agent: &str,
redirect: Option<reqwest::redirect::Policy>,
) -> reqwest::Client {
let mut builder = reqwest::Client::builder()
.timeout(Duration::from_secs(timeout_secs))
.user_agent(user_agent);
if let Some(policy) = redirect {
builder = builder.redirect(policy);
}
builder
.build()
.expect("reqwest client build is infallible with rustls-tls")
}
/// Map a non-success HTTP status to a [`GatewayError`].
///
/// - `404` → [`GatewayError::NotFound`]
/// - `429` → [`GatewayError::RateLimited`] (honours `Retry-After`, default 60s)
/// - `5xx` → [`GatewayError::Transient`]
/// - anything else → [`GatewayError::Permanent`]
pub fn map_status(resp: &reqwest::Response) -> Result<(), GatewayError> {
let status = resp.status();
if status.is_success() {
return Ok(());
}
let code = status.as_u16();
if code == 404 {
return Err(GatewayError::NotFound);
}
if code == 429 {
let retry_after = resp
.headers()
.get(reqwest::header::RETRY_AFTER)
.and_then(|v| v.to_str().ok())
.and_then(|v| v.parse::<u32>().ok())
.unwrap_or(60);
return Err(GatewayError::RateLimited {
retry_after_s: retry_after,
});
}
if status.is_server_error() {
return Err(GatewayError::Transient(format!(
"upstream {} returned {status}",
resp.url()
)));
}
Err(GatewayError::Permanent(format!(
"upstream {} returned {status}",
resp.url()
)))
}
/// Open the per-plugin redb midhash cache, wrapping the error with the plugin
/// name for a clean startup diagnostic.
pub fn open_midhash_cache(
cache_dir: &Path,
plugin: &'static str,
) -> Result<MidhashCache, ConfigError> {
MidhashCache::open(cache_dir).map_err(|e| ConfigError::Other {
plugin,
source: anyhow::anyhow!("open redb cache: {e}"),
})
}
/// Cache-hit fast path for single-outcome plugins.
///
/// On hit, returns `Some(vec![sparse outcome])` — hash only, no bytes/record —
/// so the core's `(_, None)` branch skips re-storing content. `Ok(None)` is a
/// cache miss; the caller proceeds to fetch + hash.
pub fn cached_outcome(
cache: &MidhashCache,
record_id: &str,
upstream_id: &'static str,
) -> Result<Option<Vec<HashOutcome>>, GatewayError> {
let Some(cached) = cache
.get_midhash(record_id)
.map_err(|e| GatewayError::Internal(anyhow::anyhow!("cache get: {e}")))?
else {
return Ok(None);
};
debug!(
target: "meta-share::gateway",
upstream = upstream_id,
record_id,
"hash cache hit"
);
Ok(Some(vec![HashOutcome {
hash: Hash(cached),
hash_kind: HashKind::Sha2_256,
bytes: None,
record: None,
file_extension: None,
}]))
}
/// Persist a freshly-computed `(record_id → cid)` mapping. A write failure is
/// non-fatal, so it is logged and swallowed.
pub fn store_midhash(cache: &MidhashCache, record_id: &str, upstream_id: &'static str, cid: &str) {
if let Err(e) = cache.put_midhash(record_id, cid) {
warn!(
target: "meta-share::gateway",
upstream = upstream_id,
record_id,
error = %e,
"hash cache put failed (non-fatal)"
);
}
}
/// Resolve the per-plugin cache handle, or produce the standard "not
/// configured" internal error.
pub fn require_cache<'a>(
cache: Option<&'a MidhashCache>,
plugin: &str,
) -> Result<&'a MidhashCache, GatewayError> {
cache.ok_or_else(|| {
GatewayError::Internal(anyhow::anyhow!(
"{plugin} plugin not configured (configure() never called)"
))
})
}
/// Build the single-element outcome vec a single-outcome plugin returns on a
/// fresh (cache-miss) compute: the full bytes + record, a Sha2_256 IPFS CID,
/// and an optional file extension.
pub fn single_outcome(
cid: String,
bytes: bytes::Bytes,
record: DiscoveryRecord,
file_extension: Option<String>,
) -> Vec<HashOutcome> {
vec![HashOutcome {
hash: Hash(cid),
hash_kind: HashKind::Sha2_256,
bytes: Some(bytes),
record: Some(record),
file_extension,
}]
}
/// Percent-encode a query-string value using the unreserved set
/// (RFC 3986 `A-Za-z0-9-_.~`) plus `+` for space.
pub fn urlencode(s: &str) -> String {
let mut out = String::with_capacity(s.len());
for b in s.as_bytes() {
match *b {
b'A'..=b'Z' | b'a'..=b'z' | b'0'..=b'9' | b'-' | b'_' | b'.' | b'~' => {
out.push(*b as char);
}
b' ' => out.push('+'),
other => out.push_str(&format!("%{:02X}", other)),
}
}
out
}
/// GET `url` and return the body bytes, mapping transport + status errors to
/// [`GatewayError`].
pub async fn fetch_bytes(http: &reqwest::Client, url: &str) -> Result<bytes::Bytes, GatewayError> {
let resp = http
.get(url)
.send()
.await
.map_err(|e| GatewayError::Transient(format!("GET {url}: {e}")))?;
map_status(&resp)?;
resp.bytes()
.await
.map_err(|e| GatewayError::Transient(format!("read body {url}: {e}")))
}
/// Like [`fetch_bytes`] but decodes the body as UTF-8 text.
pub async fn fetch_text(http: &reqwest::Client, url: &str) -> Result<String, GatewayError> {
let resp = http
.get(url)
.send()
.await
.map_err(|e| GatewayError::Transient(format!("GET {url}: {e}")))?;
map_status(&resp)?;
resp.text()
.await
.map_err(|e| GatewayError::Transient(format!("read body {url}: {e}")))
}
/// Decode a v1 BitTorrent infohash string into its raw 20 bytes.
///
/// Accepts 40 hex chars (case-insensitive) or 32 base32 chars (RFC 4648,
/// case-insensitive). Returns `None` for anything else.
pub fn decode_infohash(s: &str) -> Option<[u8; 20]> {
let bytes = s.as_bytes();
let raw: Vec<u8> = if bytes.len() == 40 && bytes.iter().all(u8::is_ascii_hexdigit) {
let mut out = Vec::with_capacity(20);
for pair in bytes.chunks(2) {
out.push((hex_nibble(pair[0])? << 4) | hex_nibble(pair[1])?);
}
out
} else if bytes.len() == 32 && bytes.iter().all(u8::is_ascii_alphanumeric) {
decode_base32_20(bytes)?
} else {
return None;
};
raw.try_into().ok()
}
fn hex_nibble(b: u8) -> Option<u8> {
match b {
b'0'..=b'9' => Some(b - b'0'),
b'a'..=b'f' => Some(b - b'a' + 10),
b'A'..=b'F' => Some(b - b'A' + 10),
_ => None,
}
}
/// Decode 32 RFC-4648 base32 chars (case-insensitive, no padding) into the 20
/// bytes they encode.
fn decode_base32_20(chars: &[u8]) -> Option<Vec<u8>> {
let mut out = Vec::with_capacity(20);
let mut buffer: u64 = 0;
let mut bits: u32 = 0;
for &c in chars {
let val = match c {
b'A'..=b'Z' => c - b'A',
b'a'..=b'z' => c - b'a',
b'2'..=b'7' => c - b'2' + 26,
_ => return None,
} as u64;
buffer = (buffer << 5) | val;
bits += 5;
if bits >= 8 {
bits -= 8;
out.push((buffer >> bits) as u8);
}
}
Some(out)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn decode_infohash_hex_roundtrips() {
let hex = "0123456789abcdef0123456789abcdef01234567";
let raw = decode_infohash(hex).expect("hex decodes");
assert_eq!(raw[0], 0x01);
assert_eq!(raw[19], 0x67);
}
#[test]
fn decode_infohash_is_case_insensitive_hex() {
let lower = decode_infohash("aabbccddeeff00112233445566778899aabbccdd");
let upper = decode_infohash("AABBCCDDEEFF00112233445566778899AABBCCDD");
assert!(lower.is_some());
assert_eq!(lower, upper);
}
#[test]
fn decode_infohash_accepts_base32() {
let raw = decode_infohash("MFRGGZDFMZTWQ2LKNNWG23TPOBYXE43U");
assert!(raw.is_some());
}
#[test]
fn decode_infohash_rejects_wrong_shape() {
assert!(decode_infohash("tooshort").is_none());
assert!(decode_infohash("zz23456789abcdef0123456789abcdef01234567").is_none());
assert!(decode_infohash("").is_none());
}
}
crates/meta-feeder-sdk/src/config.rs
-312
View File
@@ -1,312 +0,0 @@
//! Self-describing per-plugin configuration: schema, redaction, merge, and the
//! generic schema-driven HTML page the feeder serves at `/config`.
//!
//! The point of this module is that **the gateway carries no per-plugin config
//! knowledge** and **the SDK carries no per-plugin field names**. A plugin
//! declares its fields via [`FeederPlugin::config_schema`](crate::FeederPlugin::config_schema);
//! everything here (redaction of secrets, merge-on-save, and the rendered form)
//! is driven entirely by that schema. Add a feeder → it shows up with a working
//! config form and zero UI/gateway changes.
//!
//! ## Persistence & precedence
//!
//! Config lives at `cache_dir/config.json` (the per-plugin dir handed to
//! `configure()`). The plugin reads **file-or-env** at startup: env is the
//! first-boot seed and stays authoritative until the operator saves through the
//! UI, at which point `config.json` exists and wins. There is no runtime reload
//! — a save returns `restart_required: true` (mirrors the gateway's
//! `gateway-config.json` posture).
//!
//! ## Secrets
//!
//! Fields of kind [`FieldKind::Secret`] are **write-only**: never returned to
//! the page (redacted to a `<key>_set: bool`), and a blank incoming value on
//! save means "keep the stored secret".
use serde::{Deserialize, Serialize};
use serde_json::{Map, Value};
/// A plugin's full config surface — an ordered list of fields.
#[derive(Serialize, Deserialize, Clone, Debug, Default)]
pub struct ConfigSchema {
pub fields: Vec<ConfigField>,
}
impl ConfigSchema {
/// Convenience: `true` when the plugin declares no configurable fields (the
/// page then renders a "no configuration required" notice).
pub fn is_empty(&self) -> bool {
self.fields.is_empty()
}
}
/// One configurable field. `record_array` fields nest `fields` for the
/// per-record sub-schema (e.g. torznab's `indexers`).
#[derive(Serialize, Deserialize, Clone, Debug)]
pub struct ConfigField {
/// JSON key in the config object.
pub key: String,
/// Human label for the form.
pub label: String,
#[serde(rename = "type")]
pub kind: FieldKind,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub help: Option<String>,
#[serde(default)]
pub required: bool,
/// Sub-fields for [`FieldKind::RecordArray`]. The FIRST sub-field is treated
/// as the record identifier for secret back-fill on merge (e.g. `name`).
#[serde(default, skip_serializing_if = "Option::is_none")]
pub fields: Option<Vec<ConfigField>>,
}
impl ConfigField {
pub fn text(key: &str, label: &str) -> Self {
Self::new(key, label, FieldKind::Text)
}
pub fn secret(key: &str, label: &str) -> Self {
Self::new(key, label, FieldKind::Secret)
}
pub fn bool(key: &str, label: &str) -> Self {
Self::new(key, label, FieldKind::Bool)
}
pub fn number(key: &str, label: &str) -> Self {
Self::new(key, label, FieldKind::Number)
}
pub fn list(key: &str, label: &str) -> Self {
Self::new(key, label, FieldKind::List)
}
pub fn record_array(key: &str, label: &str, fields: Vec<ConfigField>) -> Self {
let mut f = Self::new(key, label, FieldKind::RecordArray);
f.fields = Some(fields);
f
}
fn new(key: &str, label: &str, kind: FieldKind) -> Self {
Self {
key: key.to_string(),
label: label.to_string(),
kind,
help: None,
required: false,
fields: None,
}
}
pub fn with_help(mut self, help: &str) -> Self {
self.help = Some(help.to_string());
self
}
pub fn required(mut self) -> Self {
self.required = true;
self
}
}
/// Field widget / value type. Drives both rendering and the secret/merge rules.
#[derive(Serialize, Deserialize, Clone, Copy, Debug, PartialEq, Eq)]
#[serde(rename_all = "snake_case")]
pub enum FieldKind {
/// Free text → JSON string.
Text,
/// Write-only secret → JSON string; redacted on read, "blank keeps" on save.
Secret,
/// Checkbox → JSON bool.
Bool,
/// Numeric input → JSON number.
Number,
/// Comma/line list → JSON array of strings.
List,
/// Repeatable record → JSON array of objects keyed by nested `fields`.
RecordArray,
}
fn is_blank(v: Option<&Value>) -> bool {
match v {
None | Some(Value::Null) => true,
Some(Value::String(s)) => s.is_empty(),
_ => false,
}
}
/// Produce a UI-safe view of `values`: secret fields are dropped and replaced
/// with a `<key>_set` boolean; everything else passes through. Recurses into
/// `record_array` records for secret sub-fields.
pub fn redact(values: &Value, schema: &ConfigSchema) -> Value {
let src = values.as_object().cloned().unwrap_or_default();
Value::Object(redact_object(&src, &schema.fields))
}
fn redact_object(src: &Map<String, Value>, fields: &[ConfigField]) -> Map<String, Value> {
let mut out = Map::new();
for f in fields {
match f.kind {
FieldKind::Secret => {
let set = !is_blank(src.get(&f.key));
out.insert(format!("{}_set", f.key), Value::Bool(set));
}
FieldKind::RecordArray => {
let subs = f.fields.as_deref().unwrap_or(&[]);
let arr = src
.get(&f.key)
.and_then(Value::as_array)
.map(|recs| {
recs.iter()
.map(|r| {
let m = r.as_object().cloned().unwrap_or_default();
Value::Object(redact_object(&m, subs))
})
.collect::<Vec<_>>()
})
.unwrap_or_default();
out.insert(f.key.clone(), Value::Array(arr));
}
_ => {
if let Some(v) = src.get(&f.key) {
out.insert(f.key.clone(), v.clone());
}
}
}
}
out
}
/// Merge `incoming` (from the UI) onto `base` (the stored/effective config),
/// schema-aware. Returns the full, UNREDACTED object to persist:
/// - secret: blank/absent incoming keeps `base`; non-blank overwrites.
/// - text/number/bool/list: present in incoming overwrites; absent keeps base.
/// - record_array: incoming array replaces base, but blank secret sub-fields are
/// back-filled from the base record matched by the first sub-field (identifier).
pub fn merge(base: &Value, incoming: &Value, schema: &ConfigSchema) -> Value {
let base_obj = base.as_object().cloned().unwrap_or_default();
let inc_obj = incoming.as_object().cloned().unwrap_or_default();
Value::Object(merge_object(&base_obj, &inc_obj, &schema.fields))
}
fn merge_object(
base: &Map<String, Value>,
inc: &Map<String, Value>,
fields: &[ConfigField],
) -> Map<String, Value> {
let mut out = base.clone();
for f in fields {
match f.kind {
FieldKind::Secret => {
if !is_blank(inc.get(&f.key)) {
out.insert(f.key.clone(), inc.get(&f.key).cloned().unwrap());
}
// blank/absent → keep base (no-op)
}
FieldKind::RecordArray => {
let subs = f.fields.as_deref().unwrap_or(&[]);
let id_key = subs.first().map(|s| s.key.clone());
let base_recs = base.get(&f.key).and_then(Value::as_array).cloned();
let inc_recs = match inc.get(&f.key).and_then(Value::as_array) {
Some(r) => r.clone(),
None => continue, // absent → keep base array untouched
};
let merged: Vec<Value> = inc_recs
.iter()
.map(|rec| {
let rec_obj = rec.as_object().cloned().unwrap_or_default();
// Find the matching base record by identifier sub-field.
let base_match = match (&id_key, base_recs.as_ref()) {
(Some(idk), Some(brs)) => brs
.iter()
.find(|br| br.get(idk) == rec_obj.get(idk))
.and_then(Value::as_object)
.cloned()
.unwrap_or_default(),
_ => Map::new(),
};
Value::Object(merge_object(&base_match, &rec_obj, subs))
})
.collect();
out.insert(f.key.clone(), Value::Array(merged));
}
_ => {
if let Some(v) = inc.get(&f.key) {
out.insert(f.key.clone(), v.clone());
}
}
}
}
out
}
/// The generic, schema-driven config page served at `GET /config`. Self-contained
/// (inline CSS + JS, no external assets) so it loads identically when hit
/// directly on the feeder OR reverse-proxied through the gateway. All API calls
/// are relative to `location.pathname`, so the proxy mount prefix is irrelevant.
pub const CONFIG_PAGE_HTML: &str = include_str!("config_page.html");
#[cfg(test)]
mod tests {
use super::*;
use serde_json::json;
fn torznab_like() -> ConfigSchema {
ConfigSchema {
fields: vec![
ConfigField::secret("tmdb_token", "TMDB token"),
ConfigField::bool("resolve_torrent_files", "Resolve .torrent"),
ConfigField::record_array(
"indexers",
"Indexers",
vec![
ConfigField::text("name", "Name"),
ConfigField::secret("api_key", "API key"),
],
),
],
}
}
#[test]
fn redact_hides_secrets_and_flags_set() {
let v = json!({
"tmdb_token": "abc",
"resolve_torrent_files": true,
"indexers": [{"name":"prowlarr","api_key":"k"}]
});
let r = redact(&v, &torznab_like());
assert_eq!(r["tmdb_token_set"], json!(true));
assert!(r.get("tmdb_token").is_none());
assert_eq!(r["resolve_torrent_files"], json!(true));
assert_eq!(r["indexers"][0]["name"], json!("prowlarr"));
assert_eq!(r["indexers"][0]["api_key_set"], json!(true));
assert!(r["indexers"][0].get("api_key").is_none());
}
#[test]
fn merge_blank_secret_keeps_base() {
let base = json!({"tmdb_token": "keepme"});
let inc = json!({"tmdb_token": ""});
let m = merge(&base, &inc, &torznab_like());
assert_eq!(m["tmdb_token"], json!("keepme"));
}
#[test]
fn merge_nonblank_secret_overwrites() {
let base = json!({"tmdb_token": "old"});
let inc = json!({"tmdb_token": "new"});
let m = merge(&base, &inc, &torznab_like());
assert_eq!(m["tmdb_token"], json!("new"));
}
#[test]
fn merge_record_array_backfills_blank_secret_by_identifier() {
let base = json!({"indexers":[{"name":"prowlarr","api_key":"secret"}]});
// UI re-submits the indexer with a blank api_key → keep the stored one.
let inc = json!({"indexers":[{"name":"prowlarr","api_key":""}]});
let m = merge(&base, &inc, &torznab_like());
assert_eq!(m["indexers"][0]["api_key"], json!("secret"));
assert_eq!(m["indexers"][0]["name"], json!("prowlarr"));
}
#[test]
fn merge_bool_overwrites_when_present() {
let base = json!({"resolve_torrent_files": true});
let inc = json!({"resolve_torrent_files": false});
let m = merge(&base, &inc, &torznab_like());
assert_eq!(m["resolve_torrent_files"], json!(false));
}
}
crates/meta-feeder-sdk/src/config_page.html
-268
View File
@@ -1,268 +0,0 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>feeder config</title>
<style>
:root { color-scheme: dark; }
* { box-sizing: border-box; }
body {
margin: 0; padding: 18px;
font: 14px/1.5 system-ui, -apple-system, Segoe UI, Roboto, sans-serif;
background: #0d0d12; color: #e6e6ea;
}
h1 { font-size: 16px; margin: 0 0 4px; }
.sub { color: #9a9aa6; font-size: 12px; margin: 0 0 16px; }
.field { margin: 0 0 16px; }
.field > label { display: block; font-weight: 600; margin: 0 0 4px; }
.help { color: #8a8a96; font-size: 12px; margin: 3px 0 0; }
.req { color: #f0a0a0; margin-left: 3px; font-weight: 700; }
input[type=text], input[type=password], input[type=number], textarea {
width: 100%; padding: 8px 10px; border-radius: 8px;
border: 1px solid #2a2a35; background: #15151c; color: #e6e6ea; font: inherit;
}
textarea { min-height: 56px; resize: vertical; }
input[type=checkbox] { width: 16px; height: 16px; vertical-align: middle; }
.pill { font-size: 10px; padding: 2px 7px; border-radius: 999px; margin-left: 8px; vertical-align: middle; }
.pill.set { background: #14361f; color: #6ee7a0; border: 1px solid #1f5132; }
.pill.unset { background: #2a2a35; color: #b9b9c4; border: 1px solid #3a3a45; }
.record { border: 1px solid #2a2a35; border-radius: 10px; padding: 12px; margin: 0 0 10px; background: #111118; }
.record .field { margin-bottom: 10px; }
.row-actions { text-align: right; }
button {
font: inherit; cursor: pointer; border-radius: 8px; padding: 7px 14px;
border: 1px solid #3a3a45; background: #1c1c26; color: #e6e6ea;
}
button.primary { background: #6d4aff; border-color: #6d4aff; color: #fff; font-weight: 600; }
button.danger { background: transparent; border-color: #5a2a2a; color: #e29; padding: 4px 10px; }
button.add { margin-top: 4px; }
.bar { position: sticky; bottom: 0; padding: 12px 0 2px; background: linear-gradient(#0d0d1200, #0d0d12 30%); }
.banner { padding: 9px 12px; border-radius: 8px; margin: 0 0 14px; font-size: 13px; display: none; }
.banner.ok { background: #14361f; color: #9ff0bf; border: 1px solid #1f5132; display: block; }
.banner.warn { background: #3a2f12; color: #f0d79f; border: 1px solid #5a4a1f; display: block; }
.banner.err { background: #3a1414; color: #f0a0a0; border: 1px solid #5a2020; display: block; }
.empty { color: #9a9aa6; }
</style>
</head>
<body>
<h1 id="title">Plugin configuration</h1>
<p class="sub">Secrets are write-only — set values are shown as “set”, never echoed back. Saving requires a feeder restart to take effect. Fields marked <span class="req">*</span> are required.</p>
<div id="banner" class="banner"></div>
<form id="form"></form>
<div class="bar" id="bar" style="display:none">
<button type="button" class="primary" id="save">Save</button>
</div>
<script>
// All API calls are relative to this page's path, so the same page works when
// served directly on the feeder (/config) or reverse-proxied through the
// gateway (/api/gateway/plugins/<id>/ui/config).
const BASE = location.pathname.replace(/\/+$/, "");
let SCHEMA = { fields: [] };
let VALUES = {};
const $ = (id) => document.getElementById(id);
function banner(kind, msg) {
const b = $("banner");
b.className = "banner " + kind;
b.textContent = msg;
}
function el(tag, attrs, ...kids) {
const e = document.createElement(tag);
for (const [k, v] of Object.entries(attrs || {})) {
if (k === "class") e.className = v;
else if (k === "text") e.textContent = v;
else if (k in e) e[k] = v;
else e.setAttribute(k, v);
}
for (const c of kids) if (c != null) e.append(c);
return e;
}
// Split help text into plain runs + clickable <a> nodes for any http(s) URL.
// Built as DOM nodes (never innerHTML) so help stays XSS-safe; trailing
// punctuation is kept outside the link so "see http://host/ui." reads right.
function linkify(text) {
const out = [];
const re = /https?:\/\/[^\s]+/g;
let last = 0, m;
while ((m = re.exec(text)) !== null) {
if (m.index > last) out.push(text.slice(last, m.index));
let url = m[0], trail = "";
const t = url.match(/[.,;:)\]]+$/);
if (t) { trail = t[0]; url = url.slice(0, -trail.length); }
out.push(el("a", { href: url, target: "_blank", rel: "noopener noreferrer", text: url }));
if (trail) out.push(trail);
last = m.index + m[0].length;
}
if (last < text.length) out.push(text.slice(last));
return out;
}
function helpEl(f) { return f.help ? el("p", { class: "help" }, ...linkify(f.help)) : null; }
// A "*" marker for required fields (null for optional ones).
function reqStar(f) { return f.required ? el("span", { class: "req", text: "*", title: "required" }) : null; }
// Render one scalar field. `vals` is the object holding the value/`_set` flag.
// Every returned widget exposes `_collect()` and `_key` so collect() can read it.
function fieldInput(f, vals) {
if (f.type === "bool") {
const cb = el("input", { type: "checkbox" });
cb.checked = vals[f.key] === true;
const w = el("div", { class: "field" },
el("label", {}, cb, " ", el("span", { text: " " + f.label }), reqStar(f)),
helpEl(f));
w._collect = () => cb.checked;
w._key = f.key;
return w;
}
const label = el("label", { text: f.label });
const star = reqStar(f); if (star) label.append(star);
let input;
if (f.type === "secret") {
const isSet = vals[f.key + "_set"] === true;
input = el("input", { type: "password", placeholder: isSet ? "•••••••• (blank keeps current)" : "not set" });
label.append(el("span", { class: "pill " + (isSet ? "set" : "unset"), text: isSet ? "set" : "not set" }));
input._get = () => input.value; // blank => merge keeps stored secret
} else if (f.type === "number") {
input = el("input", { type: "number" });
if (vals[f.key] != null) input.value = vals[f.key];
input._get = () => (input.value === "" ? null : Number(input.value));
} else if (f.type === "list") {
input = el("textarea", { placeholder: "one per line or comma-separated" });
const arr = Array.isArray(vals[f.key]) ? vals[f.key] : [];
input.value = arr.join("\n");
input._get = () => input.value.split(/[\n,]/).map((s) => s.trim()).filter(Boolean);
} else {
input = el("input", { type: "text" });
if (vals[f.key] != null) input.value = vals[f.key];
input._get = () => input.value;
}
const wrap = el("div", { class: "field" }, label, input, helpEl(f));
wrap._collect = () => input._get();
wrap._key = f.key;
return wrap;
}
// Render a record_array (repeatable group). Returns a wrapper exposing _collect.
function recordArray(f, vals) {
const subs = f.fields || [];
const idKey = subs.length ? subs[0].key : null;
const list = el("div", {});
const records = Array.isArray(vals[f.key]) ? vals[f.key].slice() : [];
function addRecord(recVals) {
const card = el("div", { class: "record" });
const inputs = subs.map((sf) => {
const fi = fieldInput(sf, recVals || {});
card.append(fi);
return fi;
});
const rm = el("button", { type: "button", class: "danger", text: "remove" });
rm.onclick = () => card.remove();
card.append(el("div", { class: "row-actions" }, rm));
card._collect = () => {
const o = {};
for (const fi of inputs) o[fi._key] = fi._collect();
return o;
};
list.append(card);
return card;
}
records.forEach((r) => addRecord(r));
const add = el("button", { type: "button", class: "add", text: "+ add " + f.label.toLowerCase().replace(/s$/, "") });
add.onclick = () => addRecord({});
const lbl = el("label", { text: f.label });
const star = reqStar(f); if (star) lbl.append(star);
const wrap = el("div", { class: "field" }, lbl, helpEl(f), list, add);
wrap._key = f.key;
wrap._collect = () => Array.from(list.children).map((c) => c._collect());
return wrap;
}
function render() {
$("title").textContent = (SCHEMA.title || "Plugin") + " configuration";
const form = $("form");
form.innerHTML = "";
if (!SCHEMA.fields || !SCHEMA.fields.length) {
form.append(el("p", { class: "empty", text: "No configuration required for this plugin." }));
$("bar").style.display = "none";
return;
}
const widgets = [];
for (const f of SCHEMA.fields) {
const w = f.type === "record_array" ? recordArray(f, VALUES) : fieldInput(f, VALUES);
widgets.push(w);
form.append(w);
}
form._widgets = widgets;
$("bar").style.display = "block";
}
function collect() {
const out = {};
for (const w of $("form")._widgets || []) out[w._key] = w._collect();
return out;
}
async function save() {
$("save").disabled = true;
try {
const res = await fetch(BASE + "/values", {
method: "PUT",
headers: { "Content-Type": "application/json" },
body: JSON.stringify(collect()),
});
if (!res.ok) {
const detail = (await res.text().catch(() => "")).trim();
throw new Error(detail ? `the feeder rejected it (HTTP ${res.status}): ${detail}`
: `the feeder returned HTTP ${res.status}`);
}
const j = await res.json().catch(() => ({}));
if (j.reloading) {
// The feeder is restarting itself (~2-3s) to apply the change. Poll it
// back up, then refresh the form so secret pills reflect stored state.
banner("warn", "Saved — reloading the feeder to apply (~3 s)…");
let tries = 6;
const poll = async () => {
try { await load(); banner("ok", "Reloaded — new config applied."); }
catch { if (--tries > 0) setTimeout(poll, 1500); else banner("warn", "Saved & reloading — reopen this panel shortly to confirm."); }
};
setTimeout(poll, 3000);
} else {
if (j.restart_required) banner("warn", "Saved. Restart the feeder container for changes to take effect.");
else banner("ok", "Saved.");
// Re-fetch so secret pills reflect the new stored state.
await load();
}
} catch (e) {
const msg = (e && e.message) ? e.message : String(e);
const friendly = /Failed to fetch|NetworkError|load failed/i.test(msg)
? "couldn't reach the feeder — it may be restarting; wait a moment and try again."
: msg;
banner("err", "Could not save: " + friendly);
} finally {
$("save").disabled = false;
}
}
async function load() {
const [s, v] = await Promise.all([
fetch(BASE + "/schema").then((r) => r.json()),
fetch(BASE + "/values").then((r) => r.json()),
]);
SCHEMA = s || { fields: [] };
VALUES = v || {};
render();
}
$("save").onclick = save;
load().catch((e) => banner("err", "Failed to load config: " + e.message));
</script>
</body>
</html>
crates/meta-feeder-sdk/src/domain.rs
-549
View File
@@ -1,549 +0,0 @@
//! `contentKind` → `domain` / `workForm` — the routing tables from
//! METADATA_KEYS.md §1.
//!
//! `domain` answers **"which application is this record destined for?"**. It
//! is the coarsest facet in the registry and the key every client filters its
//! catalogue on — **one value per app**, no alternation:
//!
//! ```text
//! meta-watch → domain:screen
//! meta-listen → domain:music
//! meta-read → domain:literature
//! ```
//!
//! `workForm` answers the *other* question the old `film`/`tv` split was
//! secretly encoding: **is this work complete in itself, or one instalment of
//! a run?** (`standalone` | `serial`). It is cross-domain — a comic issue is
//! `serial` in `literature` exactly as an episode is `serial` in `screen` —
//! which is what lets a client ask for it independently of the wall.
//!
//! # Why they are stored and not derived on read
//!
//! `contentKind`'s vocabulary is deliberately open — a new feeder may mint a
//! kind and ship without a fleet-wide update. That only stays safe because the
//! domain travels *with* the record: a peer that has never heard of
//! `podcastEpisode` still routes it correctly, because it never has to
//! interpret the token. Deriving on read would silently drop every record
//! whose kind the reader predates. The same argument applies to `workForm`,
//! plus a second one: the two kinds that most need it (`pack` and `card`) are
//! exactly the two the kind alone cannot decide.
//!
//! # ⚠ The binding is permanent and one-way
//!
//! Add kinds freely; **never re-map an existing kind to a different domain**.
//! Because the value is stored, two peers on different versions that disagree
//! about where `audiobook` belongs produce a split-brain that redeploying
//! cannot heal — the wrong domain is already written into records across the
//! mesh. If a kind genuinely has to move, mint a new kind.
//!
//! The `film`+`tv` → `screen` merge (METADATA_KEYS.md §14.17) is the one
//! deliberate exception, taken once and early and paid for with a one-shot
//! meta-core corpus sweep. It is not a precedent.
//!
//! # Why `domain` and not one kind-key per medium
//!
//! `record_matches` is AND-across-keys / OR-within-key, and a non-matching key
//! fails the record outright. A per-medium split (`videoKind` / `audioKind` /
//! `documentKind`) would force meta-listen to express "music video **or**
//! audio track" as an OR across two *different* keys — which the query
//! language cannot represent at all. One domain key, OR-ed within itself,
//! always can. `workForm` is one key everywhere for the same reason.
/// The `contentKind` → `domain` table. A kind absent from this list has no
/// domain and is **not independently routable** — correct for sidecars
/// (subtitles, artwork) and for un-anchored hits, a producer bug for anything
/// else.
///
/// ⚠ `pack` is deliberately absent. A season pack and an album release are the
/// same structural thing, so the kind alone cannot tell `screen` from `music`;
/// the writer stamps `pack`'s domain from its own context (the same reason
/// `derive_file_type` cannot have a second `pack` arm).
///
/// ⚠ `podcast` / `podcastEpisode` are absent too: the `spoken` domain was
/// retired unused (METADATA_KEYS.md §14.17), and the PR that first *writes*
/// those kinds picks their domain.
const DOMAIN_BY_CONTENT_KIND: &[(&str, &str)] = &[
// screen — everything meta-watch consumes, films and serials alike. The
// standalone/serial split lives in `workForm`, not here.
("movie", "screen"),
("series", "screen"),
("episode", "screen"),
// music
("track", "music"),
("album", "music"),
("artist", "music"),
("musicVideo", "music"),
("djMix", "music"),
("liveSet", "music"),
// literature — an audiobook's identity is the book (same author, same ISBN
// family, same series as the ebook), so meta-read groups the two editions
// on one page. The player does not decide the domain.
("book", "literature"),
("audiobook", "literature"),
("comic", "literature"),
("manga", "literature"),
("magazine", "literature"),
// science
("paper", "science"),
];
/// The `contentKind` → `workForm` table (METADATA_KEYS.md §1).
///
/// `standalone` = a work complete in itself. `serial` = an instalment of an
/// ongoing run, **or the container or headline for one** — which is why a
/// `series` card and an `episode` are both `serial`.
///
/// ⚠ `pack` is absent for the same reason it is absent above: a season pack is
/// `serial`, an album release is `standalone`, and only the writer knows which.
const WORK_FORM_BY_CONTENT_KIND: &[(&str, &str)] = &[
// screen
("movie", "standalone"),
("series", "serial"),
("episode", "serial"),
// music — an album is a closed work, not an ongoing run, so every rung of
// the music ladder is standalone.
("track", "standalone"),
("album", "standalone"),
("artist", "standalone"),
("musicVideo", "standalone"),
("djMix", "standalone"),
("liveSet", "standalone"),
// literature — the split that proves `workForm` is not a screen-only axis.
("book", "standalone"),
("audiobook", "standalone"),
("comic", "serial"),
("manga", "serial"),
("magazine", "serial"),
// science
("paper", "standalone"),
];
/// Resolve the domain a `contentKind` belongs to.
///
/// Returns `None` for `pack` (not derivable from the kind — the writer supplies
/// it), for format/sidecar kinds that route nowhere, and for any kind this
/// build has never heard of. A caller that already knows the domain from its
/// own context must prefer that over this table.
pub fn domain_for_content_kind(content_kind: &str) -> Option<&'static str> {
lookup(DOMAIN_BY_CONTENT_KIND, content_kind)
}
/// Resolve the work form a `contentKind` sits on. Same contract as
/// [`domain_for_content_kind`]: `None` means "not derivable", never a guess.
pub fn work_form_for_content_kind(content_kind: &str) -> Option<&'static str> {
lookup(WORK_FORM_BY_CONTENT_KIND, content_kind)
}
fn lookup(table: &'static [(&'static str, &'static str)], key: &str) -> Option<&'static str> {
let k = key.trim();
table
.iter()
.find(|(kind, _)| kind.eq_ignore_ascii_case(k))
.map(|(_, value)| *value)
}
/// Kinds whose `domain` the kind→domain table deliberately cannot decide, so
/// the **writer** supplies it (METADATA_KEYS.md §1). Today that is `pack`
/// alone: a season pack and an album release are the same structural thing, and
/// only the producer knows which — indexer-feeder's `bt.rs` reads the domain off
/// the per-file kind *before* overwriting it with `pack` precisely so the value
/// survives.
///
/// They must ride along in every `domain:` → `contentKind:` expansion. The
/// expansion is only ever a **pre-filter** — the real `domain:` term is enforced
/// afterwards by [`crate::query_eval::record_matches`] — so widening it is free,
/// while omitting these kinds would drop exactly the records whose `domain` is
/// the most authoritative one on the wire.
pub const WRITER_SUPPLIED_DOMAIN_KINDS: &[&str] = &["pack"];
/// The `workForm` twin of [`WRITER_SUPPLIED_DOMAIN_KINDS`], and for the same
/// reason: a season pack is `serial`, an album release is `standalone`.
pub const WRITER_SUPPLIED_WORK_FORM_KINDS: &[&str] = &["pack"];
/// The inverse: every `contentKind` that belongs to `domain`.
///
/// Used at the gateway-routing boundary. A gateway declares the *content kinds*
/// it serves (`served_content_kinds`), not domains, so a `domain:` filter is
/// matched by expanding it here rather than by adding a `served_domains` field
/// to the heartbeat — which would be a wire-protocol change requiring every
/// feeder manifest and every peer to move at once.
///
/// ⚠ Since the `film`+`tv` merge this returns **several** kinds for `screen`
/// (`movie`, `series`, `episode`) where the two old values were disjoint. A
/// caller that reads only the first element silently means "movie" — scan the
/// whole list, or narrow with a `workForm:` term first.
pub fn content_kinds_for_domain(domain: &str) -> Vec<&'static str> {
inverse(DOMAIN_BY_CONTENT_KIND, domain)
}
/// The inverse of the `workForm` table. Same routing-boundary role, same
/// scan-the-whole-list warning as [`content_kinds_for_domain`].
pub fn content_kinds_for_work_form(work_form: &str) -> Vec<&'static str> {
inverse(WORK_FORM_BY_CONTENT_KIND, work_form)
}
fn inverse(table: &'static [(&'static str, &'static str)], value: &str) -> Vec<&'static str> {
let v = value.trim();
table
.iter()
.filter(|(_, val)| val.eq_ignore_ascii_case(v))
.map(|(kind, _)| *kind)
.collect()
}
/// Rewrite a query so a `domain:` / `workForm:` filter can reach a plugin that
/// has never heard of either key.
///
/// Returns `None` when the query carries neither — the common path, and cheaper
/// than cloning.
///
/// # Why translate instead of teaching every plugin
///
/// `record_matches` fails a record outright when a filter key is missing from
/// its fields, and `domain` is missing from every record a not-yet-updated
/// feeder emits (`workForm` from every record written before it existed).
/// Most feeders pin the SDK by git tag, so they cannot pick up a new key on our
/// schedule. Translating `domain:literature` into the `contentKind:`
/// alternation it covers means the wire query a plugin sees never mentions
/// either key — old plugins keep working unchanged, while the dispatcher still
/// enforces the real filters afterwards, against records it has just stamped.
///
/// # The two axes intersect
///
/// `domain:screen workForm:serial` covers `series` and `episode` — the
/// **intersection**, not the union. Widening to the union would hand a serial
/// query the whole film corpus as a pre-filter; the post-stamp check would then
/// throw it away, having already paid for it upstream.
///
/// An explicit `contentKind:` filter already in the query is left alone: it is
/// the narrower of the three, and the `domain`/`workForm` terms are still
/// enforced post-stamp, so the AND semantics hold either way.
pub fn expand_routing_filters(
query: &crate::query::GatewayQuery,
) -> Option<crate::query::GatewayQuery> {
let domains = query.filters.get("domain");
let work_forms = query.filters.get("workForm");
if domains.is_none() && work_forms.is_none() {
return None;
}
let by_domain = domains.map(|vs| expand_values(vs, content_kinds_for_domain));
let by_work_form = work_forms.map(|vs| expand_values(vs, content_kinds_for_work_form));
let mut out = query.clone();
out.filters.remove("domain");
out.filters.remove("workForm");
if out.filters.contains_key("contentKind") {
return Some(out);
}
let mut kinds: Vec<&'static str> = match (by_domain, by_work_form) {
(Some(d), Some(w)) => d.into_iter().filter(|k| w.contains(k)).collect(),
(Some(d), None) => d,
(None, Some(w)) => w,
(None, None) => unreachable!("guarded above"),
};
// Domain-ambiguous kinds come along whenever the expansion is non-empty
// — see [`WRITER_SUPPLIED_DOMAIN_KINDS`]. Skipped when nothing mapped,
// so an unknown value still expands to nothing and leaves the filter
// out entirely (below) rather than narrowing to `contentKind:pack`.
if !kinds.is_empty() {
for k in WRITER_SUPPLIED_DOMAIN_KINDS
.iter()
.chain(WRITER_SUPPLIED_WORK_FORM_KINDS.iter())
{
if !kinds.contains(k) {
kinds.push(k);
}
}
}
// An unknown value — or two axes with no kind in common — expands to
// nothing. Inserting an empty alternation would drop every record at the
// plugin; leaving the filter out lets the plugin answer and the post-stamp
// check do the (correct, empty) filtering.
if !kinds.is_empty() {
out.filters.insert(
"contentKind".to_string(),
kinds.into_iter().map(|k| k.to_string()).collect(),
);
}
Some(out)
}
/// Deprecated alias for [`expand_routing_filters`], kept so a caller pinned to
/// an older SDK keeps compiling. It expands `workForm:` too — the name is the
/// only thing that is stale.
#[deprecated(note = "renamed to expand_routing_filters; it now expands workForm: as well")]
pub fn expand_domain_filter(
query: &crate::query::GatewayQuery,
) -> Option<crate::query::GatewayQuery> {
expand_routing_filters(query)
}
fn expand_values(
values: &[String],
expand: fn(&str) -> Vec<&'static str>,
) -> Vec<&'static str> {
let mut out: Vec<&'static str> = Vec::new();
for v in values {
for k in expand(v) {
if !out.contains(&k) {
out.push(k);
}
}
}
out
}
/// Stamp `domain` on a field map that already carries a `contentKind`.
///
/// The convention from METADATA_KEYS.md §1: **whoever writes `contentKind`
/// writes `domain` in the same breath**. An existing `domain` is never
/// overwritten — a writer that knows better than the table (a `pack`, say) has
/// already set it.
pub fn stamp_domain(fields: &mut std::collections::BTreeMap<String, String>) {
stamp(fields, "domain", domain_for_content_kind);
}
/// Stamp `workForm` the same way, from the same `contentKind`. Call it
/// wherever [`stamp_domain`] is called — the registry treats the two as one
/// breath.
pub fn stamp_work_form(fields: &mut std::collections::BTreeMap<String, String>) {
stamp(fields, "workForm", work_form_for_content_kind);
}
fn stamp(
fields: &mut std::collections::BTreeMap<String, String>,
key: &str,
derive: fn(&str) -> Option<&'static str>,
) {
if fields.contains_key(key) {
return;
}
let Some(kind) = fields.get("contentKind").cloned() else {
return;
};
if let Some(value) = derive(&kind) {
fields.insert(key.to_string(), value.to_string());
}
}
#[cfg(test)]
mod tests {
use super::*;
use std::collections::BTreeMap;
#[test]
fn kinds_map_to_their_domain() {
assert_eq!(domain_for_content_kind("movie"), Some("screen"));
assert_eq!(domain_for_content_kind("episode"), Some("screen"));
assert_eq!(domain_for_content_kind("track"), Some("music"));
// The case that proves medium and domain are different axes.
assert_eq!(domain_for_content_kind("musicVideo"), Some("music"));
// The case that proves the player does not decide the domain.
assert_eq!(domain_for_content_kind("audiobook"), Some("literature"));
}
#[test]
fn film_and_tv_are_one_domain_now() {
// The merge (METADATA_KEYS.md §14.17): meta-watch's wall is one term.
assert_eq!(
domain_for_content_kind("movie"),
domain_for_content_kind("series")
);
// ...and the split it used to carry lives here instead.
assert_eq!(work_form_for_content_kind("movie"), Some("standalone"));
assert_eq!(work_form_for_content_kind("series"), Some("serial"));
assert_eq!(work_form_for_content_kind("episode"), Some("serial"));
}
#[test]
fn work_form_is_not_a_screen_only_axis() {
// The point of the key: it means the same thing in every domain.
assert_eq!(work_form_for_content_kind("comic"), Some("serial"));
assert_eq!(work_form_for_content_kind("book"), Some("standalone"));
assert_eq!(work_form_for_content_kind("album"), Some("standalone"));
}
#[test]
fn the_spoken_domain_is_gone() {
// Retired unused. The kinds stay reserved with no domain until the PR
// that writes them picks one.
assert_eq!(domain_for_content_kind("podcast"), None);
assert_eq!(domain_for_content_kind("podcastEpisode"), None);
assert!(content_kinds_for_domain("spoken").is_empty());
}
#[test]
fn pack_has_no_derivable_domain_or_work_form() {
assert_eq!(domain_for_content_kind("pack"), None);
assert_eq!(work_form_for_content_kind("pack"), None);
}
#[test]
fn unknown_kind_is_none_not_a_guess() {
assert_eq!(domain_for_content_kind("hologram"), None);
assert_eq!(work_form_for_content_kind("hologram"), None);
}
#[test]
fn expansion_is_the_inverse() {
let music = content_kinds_for_domain("music");
assert!(music.contains(&"track"));
assert!(music.contains(&"musicVideo"));
assert!(!music.contains(&"audiobook"));
for kind in content_kinds_for_domain("literature") {
assert_eq!(domain_for_content_kind(kind), Some("literature"));
}
for kind in content_kinds_for_work_form("serial") {
assert_eq!(work_form_for_content_kind(kind), Some("serial"));
}
}
#[test]
fn screen_expands_to_every_rung_not_just_the_first() {
// The regression the merge creates for any caller reading `.first()`.
let screen = content_kinds_for_domain("screen");
assert!(screen.contains(&"movie"));
assert!(screen.contains(&"series"));
assert!(screen.contains(&"episode"));
}
fn q(filters: &[(&str, &[&str])]) -> crate::query::GatewayQuery {
crate::query::GatewayQuery {
raw_text: String::new(),
free_text: String::new(),
filters: filters
.iter()
.map(|(k, vs)| {
(
k.to_string(),
vs.iter().map(|v| v.to_string()).collect::<Vec<_>>(),
)
})
.collect(),
ranges: Vec::new(),
negations: Vec::new(),
}
}
#[test]
fn no_routing_filter_is_left_untouched() {
assert!(expand_routing_filters(&q(&[("fileType", &["video"])])).is_none());
}
#[test]
fn domain_becomes_the_content_kinds_it_covers() {
let out = expand_routing_filters(&q(&[("domain", &["literature"])])).unwrap();
// The plugin never sees `domain:` — that is the whole point.
assert!(!out.filters.contains_key("domain"));
let kinds = out.filters.get("contentKind").unwrap();
assert!(kinds.contains(&"book".to_string()));
assert!(kinds.contains(&"audiobook".to_string()));
assert!(!kinds.contains(&"track".to_string()));
}
#[test]
fn work_form_alone_also_expands() {
let out = expand_routing_filters(&q(&[("workForm", &["serial"])])).unwrap();
assert!(!out.filters.contains_key("workForm"));
let kinds = out.filters.get("contentKind").unwrap();
assert!(kinds.contains(&"episode".to_string()));
assert!(kinds.contains(&"comic".to_string()));
assert!(!kinds.contains(&"movie".to_string()));
}
#[test]
fn the_two_axes_intersect_they_do_not_union() {
let out = expand_routing_filters(&q(&[
("domain", &["screen"][..]),
("workForm", &["serial"][..]),
]))
.unwrap();
let kinds = out.filters.get("contentKind").unwrap();
assert!(kinds.contains(&"series".to_string()));
assert!(kinds.contains(&"episode".to_string()));
// The union would have dragged the whole film corpus in.
assert!(!kinds.contains(&"movie".to_string()));
// A `serial` comic is in `literature`, not on meta-watch's wall.
assert!(!kinds.contains(&"comic".to_string()));
}
#[test]
fn an_explicit_content_kind_filter_wins() {
let out = expand_routing_filters(&q(&[
("domain", &["music"][..]),
("contentKind", &["track"][..]),
]))
.unwrap();
assert_eq!(
out.filters.get("contentKind").map(Vec::as_slice),
Some(&["track".to_string()][..])
);
}
#[test]
fn expansion_carries_the_writer_supplied_kinds() {
// `pack` has no domain in the table (a season pack and an album are the
// same structural thing), so the writer stamps one. Leaving it out of
// the alternation would make a `domain:screen` pre-filter drop the whole
// torrent-pack corpus, which `record_matches` would then never see.
let out = expand_routing_filters(&q(&[("domain", &["screen"])])).unwrap();
let kinds = out.filters.get("contentKind").unwrap();
assert!(kinds.contains(&"episode".to_string()));
assert!(kinds.contains(&"pack".to_string()));
// ...and it rides along on the `workForm` axis too, exactly once.
let out = expand_routing_filters(&q(&[("workForm", &["serial"])])).unwrap();
let kinds = out.filters.get("contentKind").unwrap();
assert_eq!(kinds.iter().filter(|k| *k == "pack").count(), 1);
}
#[test]
fn unknown_value_adds_no_alternation() {
// An empty alternation would drop every record at the plugin; the
// post-stamp check does the (correct, empty) filtering.
let out = expand_routing_filters(&q(&[("domain", &["hologram"])])).unwrap();
// Not even `pack`: a lone `contentKind:pack` would be a NARROWING of a
// filter that should widen to nothing.
assert!(!out.filters.contains_key("contentKind"));
}
#[test]
fn two_axes_with_nothing_in_common_widen_rather_than_narrow() {
// No music kind is `serial`. The pre-filter drops out; the post-stamp
// check still enforces both terms.
let out = expand_routing_filters(&q(&[
("domain", &["music"][..]),
("workForm", &["serial"][..]),
]))
.unwrap();
assert!(!out.filters.contains_key("contentKind"));
}
#[test]
fn stamp_fills_in_and_never_overwrites() {
let mut f: BTreeMap<String, String> = BTreeMap::new();
f.insert("contentKind".into(), "episode".into());
stamp_domain(&mut f);
stamp_work_form(&mut f);
assert_eq!(f.get("domain").map(String::as_str), Some("screen"));
assert_eq!(f.get("workForm").map(String::as_str), Some("serial"));
// A pack writer that already knows its context keeps it.
let mut p: BTreeMap<String, String> = BTreeMap::new();
p.insert("contentKind".into(), "pack".into());
p.insert("domain".into(), "music".into());
p.insert("workForm".into(), "standalone".into());
stamp_domain(&mut p);
stamp_work_form(&mut p);
assert_eq!(p.get("domain").map(String::as_str), Some("music"));
assert_eq!(p.get("workForm").map(String::as_str), Some("standalone"));
// No contentKind → nothing invented.
let mut n: BTreeMap<String, String> = BTreeMap::new();
stamp_domain(&mut n);
stamp_work_form(&mut n);
assert!(n.get("domain").is_none());
assert!(n.get("workForm").is_none());
}
}
crates/meta-feeder-sdk/src/enrich.rs
-614
View File
@@ -1,614 +0,0 @@
//! Feeder-driven enrichment — the "smart feeder / dumb gateway" pipeline.
//!
//! A feeder that self-publishes (has a `META_CORE_URL`) writes its own base
//! record, then drives the **reused meta-sort enrichment plugins**
//! (`metamesh-plugin-filename-parser`, `metamesh-plugin-tmdb`) over HTTP. Each
//! plugin is handed the `metaCoreUrl` and writes its own fields straight back to
//! meta-core (`PATCH /meta/{cid}` merge), so the gateway never touches
//! enrichment — full separation of concerns.
//!
//! ## Ordering — why we poll
//!
//! tmdb searches on `originalTitle`/`movieYear`/`videoType` (which it reads from
//! the record), and those are produced by filename-parser. So the pipeline is
//! **filename-parser → tmdb**. The plugin `/process` contract is *async*
//! (returns `accepted`, then writes to meta-core and POSTs a callback), so after
//! kicking filename-parser we **poll meta-core** until `originalTitle` appears
//! (bounded), read the merged fields back, and hand them to tmdb. Polling keeps
//! the feeder stateless — no callback-correlation bookkeeping. The plugins'
//! callback POSTs still fire; they hit the feeder's no-op `/enrich/callback`.
//!
//! The whole pipeline runs in a spawned background task so it never adds latency
//! to the libp2p resolve path; the meta-share ingester propagates the enriched
//! record on its next poll (~eventually consistent).
use std::collections::BTreeMap;
use std::sync::Arc;
use std::time::{Duration, SystemTime, UNIX_EPOCH};
use serde::Serialize;
use tracing::{debug, warn};
use crate::meta_core::{self, Provenance};
use crate::types::DiscoveryRecord;
/// Operator-supplied enrichment wiring, parsed once from the feeder env.
#[derive(Clone, Debug)]
pub struct EnrichmentConfig {
/// meta-core root the feeder self-publishes to + hands to the plugins.
pub meta_core_url: String,
/// `metamesh-plugin-filename-parser` base URL (optional).
pub filename_parser_url: Option<String>,
/// `metamesh-plugin-tmdb` base URL (optional).
pub tmdb_url: Option<String>,
/// TMDB v3 key / v4 bearer token, POSTed to tmdb `/configure`.
pub tmdb_token: Option<String>,
/// TMDB metadata language (default `en-US`).
pub tmdb_language: String,
/// Stable per-deployment id stamped into `provenance.gatewayPeer`.
pub gateway_peer: String,
/// URL the plugins POST their completion callback to (the feeder's own
/// no-op `/enrich/callback`). Polling is the real completion signal; this
/// only keeps the plugins from logging a failed-callback warning.
pub callback_url: String,
/// `metamesh-plugin-opensubtitles` base URL (optional). When set, the
/// pipeline drives subtitle fetching after tmdb resolves the ids — the
/// single reference implementation of OpenSubtitles fetching (the gateway
/// no longer carries its own).
pub opensubtitles_url: Option<String>,
/// OpenSubtitles consumer API key, POSTed to the plugin's `/configure`.
pub opensubtitles_api_key: Option<String>,
/// OpenSubtitles account username (required by the plugin for downloads).
pub opensubtitles_username: Option<String>,
/// OpenSubtitles account password.
pub opensubtitles_password: Option<String>,
/// Wanted subtitle languages (CSV of ISO 639-1 codes; default `en`).
pub opensubtitles_languages: String,
}
fn env_nonempty(key: &str) -> Option<String> {
std::env::var(key)
.ok()
.map(|s| s.trim().to_string())
.filter(|s| !s.is_empty())
}
/// Derive a co-bundled sidecar plugin URL from the feeder's own hostname, by the
/// app-compose convention `<prefix>-feeder` → `<prefix>-<plugin>:8080`
/// (e.g. `metafeedertorznab-feeder` → `http://metafeedertorznab-tmdb:8080`). Lets
/// a feeder reach its bundled filename-parser / tmdb sidecars with **no explicit
/// `*_PLUGIN_URL` env** — config lives in the UI, wiring is derived. Returns
/// `None` when `HOSTNAME` is unset or not in `<prefix>-feeder` form, in which case
/// the caller falls back to the env seed (then soft-skips that plugin).
fn derive_sidecar_url(plugin: &str) -> Option<String> {
let host = env_nonempty("HOSTNAME")?;
let prefix = host.strip_suffix("-feeder")?;
Some(format!("http://{prefix}-{plugin}:8080"))
}
impl EnrichmentConfig {
/// Parse from the feeder env. Returns `None` when `META_CORE_URL` is unset —
/// the **bundled-feeder** path (records returned over HTTP, the gateway
/// dispatcher stores them). Self-publishing feeders that source meta-core from
/// their own dashboard config use [`from_meta_core`](Self::from_meta_core).
pub fn from_env() -> Option<Self> {
Some(Self::from_meta_core(env_nonempty("META_CORE_URL")?))
}
/// Build with an explicit meta-core root — sourced from the feeder's own
/// `config.json` (dashboard) when set, else the `META_CORE_URL` env seed. The
/// sidecar URLs come from their `*_PLUGIN_URL` env, falling back to the
/// hostname-derived co-bundled sidecar (see [`derive_sidecar_url`]); the rest
/// keep their env seeds / defaults. Secrets (tmdb token, language) are
/// overlaid by the caller from the per-feeder config.
pub fn from_meta_core(meta_core_url: String) -> Self {
EnrichmentConfig {
meta_core_url,
filename_parser_url: env_nonempty("FILENAME_PARSER_URL")
.or_else(|| derive_sidecar_url("filename-parser")),
tmdb_url: env_nonempty("TMDB_PLUGIN_URL").or_else(|| derive_sidecar_url("tmdb")),
tmdb_token: env_nonempty("TMDB_TOKEN").or_else(|| env_nonempty("TORZNAB_TMDB_TOKEN")),
tmdb_language: env_nonempty("TMDB_LANGUAGE").unwrap_or_else(|| "en-US".to_string()),
gateway_peer: env_nonempty("META_GATEWAY_PEER_ID")
.or_else(|| env_nonempty("HOSTNAME"))
.unwrap_or_else(|| "gateway-feeder".to_string()),
callback_url: env_nonempty("META_FEEDER_CALLBACK_URL")
.unwrap_or_else(|| "http://127.0.0.1:8080/enrich/callback".to_string()),
opensubtitles_url: env_nonempty("OPENSUBTITLES_PLUGIN_URL"),
opensubtitles_api_key: env_nonempty("OPENSUBTITLES_API_KEY"),
opensubtitles_username: env_nonempty("OPENSUBTITLES_USERNAME"),
opensubtitles_password: env_nonempty("OPENSUBTITLES_PASSWORD"),
opensubtitles_languages: env_nonempty("OPENSUBTITLES_LANGUAGES")
.unwrap_or_else(|| "en".to_string()),
}
}
}
/// How long to wait for filename-parser to merge `originalTitle` before handing
/// off to tmdb. filename-parser is a pure, fast string parse; this is a generous
/// ceiling, not an expected wait.
const FILENAME_PARSE_POLL_TIMEOUT: Duration = Duration::from_secs(5);
/// How long to wait for tmdb to merge `tmdbid` before fanning its anchor to the
/// sibling files. tmdb hits the network, so a longer ceiling than filename.
const TMDB_POLL_TIMEOUT: Duration = Duration::from_secs(20);
const POLL_INTERVAL: Duration = Duration::from_millis(200);
/// Show-level TMDB fields that are **fanned** from the primary file to every
/// sibling file in a multi-file (season-pack) torrent — they're identical for
/// every episode of a show, so we resolve tmdb **once** and copy these across
/// (avoids N TMDB lookups; the tmdb plugin caches by `cid_midhash256`, which
/// gateway BT records don't carry, so without this each file would re-hit TMDB).
/// Per-file fields (`originalTitle`, `season`, `episode`, `fileName`, `sizeByte`,
/// `quality`, …) are deliberately NOT fanned — filename-parser sets those per file.
const ANCHOR_EXACT: &[&str] = &[
"tmdbid",
"imdbid",
"poster",
"backdrop",
"rating",
"releasedate",
"studio",
"genres",
"tags",
];
const ANCHOR_PREFIXES: &[&str] = &["plot/", "genres/", "studio/", "tags/"];
/// One file (or pack) to enrich: its CID, the filename to parse, and the base
/// fields already written to meta-core (handed to the plugins as `existingMeta`).
#[derive(Clone, Debug)]
pub struct EnrichTarget {
pub cid: String,
pub file_name: String,
pub base: BTreeMap<String, String>,
}
/// Drives meta-core writes + the enrichment plugins for one feeder. Cheap to
/// clone (just a reqwest client + an `Arc<config>`), so it can be moved into a
/// spawned per-resolve task.
#[derive(Clone)]
pub struct Enricher {
http: reqwest::Client,
cfg: Arc<EnrichmentConfig>,
}
#[derive(Serialize)]
struct ProcessRequest<'a> {
#[serde(rename = "taskId")]
task_id: String,
cid: &'a str,
#[serde(rename = "filePath")]
file_path: &'a str,
#[serde(rename = "callbackUrl")]
callback_url: &'a str,
#[serde(rename = "metaCoreUrl")]
meta_core_url: &'a str,
#[serde(rename = "existingMeta")]
existing_meta: &'a BTreeMap<String, String>,
}
impl Enricher {
pub fn new(http: reqwest::Client, cfg: EnrichmentConfig) -> Self {
Enricher {
http,
cfg: Arc::new(cfg),
}
}
pub fn meta_core_url(&self) -> &str {
&self.cfg.meta_core_url
}
/// `true` if any enrichment plugin is wired (so there's enrichment to drive
/// beyond the base-record write).
pub fn has_plugins(&self) -> bool {
self.cfg.filename_parser_url.is_some() || self.cfg.tmdb_url.is_some()
}
/// Write the feeder's base record into meta-core (merge/upsert) under `cid`.
/// Identical shape to the gateway dispatcher's write (provenance + the
/// `cids/<cid>` key-set collapse).
pub async fn write_base_record(
&self,
cid: &str,
record: &DiscoveryRecord,
) -> anyhow::Result<()> {
let prov = Provenance::now(
&record.upstream_id,
&record.record_id,
&self.cfg.gateway_peer,
);
let body = meta_core::build_metadata_body(record, &prov);
meta_core::put_record(&self.http, &self.cfg.meta_core_url, cid, &body).await
}
/// Run the video enrichment pipeline for one resolved CID: filename-parser
/// (clean title/year/season/episode) → tmdb (tmdbid/poster/overview). Each
/// plugin writes its own fields to meta-core. Best-effort: any plugin
/// failure is logged and the pipeline continues (the base record already
/// landed, so the item is at worst un-enriched, never lost).
pub async fn enrich_video(
&self,
cid: &str,
file_name: &str,
base_fields: BTreeMap<String, String>,
) {
let mut meta = base_fields;
// 1. filename-parser → originalTitle / movieYear / videoType / season / episode.
if let Some(fp_url) = self.cfg.filename_parser_url.clone() {
match self.process(&fp_url, "filename-parser", cid, file_name, &meta).await {
Ok(()) => {
// Poll for the merged fields so tmdb gets the clean title.
if let Some(merged) = self
.poll_until_field(cid, "originalTitle", FILENAME_PARSE_POLL_TIMEOUT)
.await
{
meta = merged;
} else {
debug!(
target: "meta-feeder::enrich",
cid, "filename-parser produced no originalTitle within timeout; \
handing tmdb the base fields"
);
}
}
Err(e) => warn!(
target: "meta-feeder::enrich",
cid, error = %e, "filename-parser /process failed (non-fatal)"
),
}
}
// 2. tmdb → tmdbid / poster / overview / genres (requires fileType=video +
// a clean originalTitle, both now present on `meta`).
if let Some(tmdb_url) = self.cfg.tmdb_url.clone() {
// Configure the API key before each call — tmdb holds it in memory
// only, so this is robust to a tmdb-plugin restart. Cheap (local).
if let Err(e) = self.configure_tmdb(&tmdb_url).await {
warn!(
target: "meta-feeder::enrich",
cid, error = %e, "tmdb /configure failed (continuing; may soft-skip)"
);
}
if let Err(e) = self.process(&tmdb_url, "tmdb", cid, file_name, &meta).await {
warn!(
target: "meta-feeder::enrich",
cid, error = %e, "tmdb /process failed (non-fatal)"
);
}
}
// 3. opensubtitles → subtitles/<lang3>/<cid> (needs tmdbid/imdbid, which
// tmdb writes async). Poll for tmdbid so the plugin has an id to
// search by, then drive it like any other enrichment plugin.
if self.cfg.opensubtitles_url.is_some() {
let merged = self
.poll_until_field(cid, "tmdbid", TMDB_POLL_TIMEOUT)
.await
.unwrap_or(meta);
self.enrich_subtitles(cid, file_name, &merged).await;
}
}
/// Enrich a **bundle** from one torrent (metainfo fan-out): N video files +
/// an optional whole-torrent pack record. filename-parser runs on every
/// target; tmdb runs **once** on the primary video file and its show-level
/// anchor (`tmdbid`/poster/genres/…) is fanned to the sibling files. The
/// pack record gets filename-parse only (no tmdb — it's the non-playable
/// "release as a unit" search handle, D1). Best-effort throughout.
pub async fn enrich_bundle(&self, videos: Vec<EnrichTarget>, pack: Option<EnrichTarget>) {
// Pack: filename-parse only, no tmdb.
if let Some(p) = pack {
self.fire_filename(&p).await;
}
if videos.is_empty() {
return;
}
if videos.len() == 1 {
// Degenerate single-video bundle → the plain pipeline (filename → tmdb).
let v = &videos[0];
self.enrich_video(&v.cid, &v.file_name, v.base.clone()).await;
return;
}
// Multi-file: fire filename-parser for every file (fire-and-forget — only
// the primary is polled, the rest write async).
for v in &videos {
self.fire_filename(v).await;
}
// Primary = largest by `sizeByte` (ties → lowest index, i.e. first seen).
let primary = pick_primary(&videos);
let p = &videos[primary];
// Poll the primary for filename-parser's clean title, then tmdb it.
let merged = self
.poll_until_field(&p.cid, "originalTitle", FILENAME_PARSE_POLL_TIMEOUT)
.await
.unwrap_or_else(|| p.base.clone());
let Some(tmdb_url) = self.cfg.tmdb_url.clone() else {
return; // no tmdb wired → filename-only enrichment, done.
};
if let Err(e) = self.configure_tmdb(&tmdb_url).await {
warn!(target: "meta-feeder::enrich", cid = %p.cid, error = %e, "tmdb /configure failed");
}
if let Err(e) = self.process(&tmdb_url, "tmdb", &p.cid, &p.file_name, &merged).await {
warn!(target: "meta-feeder::enrich", cid = %p.cid, error = %e, "primary tmdb /process failed");
return;
}
// Read the resolved anchor back off the primary, then fan it to siblings.
let Some(anchor_rec) = self.poll_until_field(&p.cid, "tmdbid", TMDB_POLL_TIMEOUT).await
else {
debug!(target: "meta-feeder::enrich", cid = %p.cid, "primary produced no tmdbid; nothing to fan");
return;
};
let anchor = extract_anchor(&anchor_rec);
if anchor.is_empty() {
return;
}
for (i, v) in videos.iter().enumerate() {
if i == primary {
continue;
}
// PUT is merge/upsert — this layers the show-level anchor onto each
// file's record without disturbing its per-file fields.
if let Err(e) =
meta_core::put_record(&self.http, &self.cfg.meta_core_url, &v.cid, &anchor).await
{
warn!(target: "meta-feeder::enrich", cid = %v.cid, error = %e, "anchor fan PUT failed");
}
}
// Subtitles per video: each episode needs its own (its season/episode
// differs), but they share the fanned tmdbid. Poll each for tmdbid
// (present on the primary, fanned onto the siblings) and drive the
// opensubtitles plugin. Best-effort + quota-guarded inside the plugin.
if self.cfg.opensubtitles_url.is_some() {
for v in &videos {
let merged = self
.poll_until_field(&v.cid, "tmdbid", TMDB_POLL_TIMEOUT)
.await
.unwrap_or_else(|| v.base.clone());
self.enrich_subtitles(&v.cid, &v.file_name, &merged).await;
}
}
}
/// Fire filename-parser `/process` for one target (fire-and-forget; the
/// plugin merges its fields to meta-core async).
async fn fire_filename(&self, t: &EnrichTarget) {
if let Some(fp_url) = self.cfg.filename_parser_url.clone() {
if let Err(e) = self
.process(&fp_url, "filename-parser", &t.cid, &t.file_name, &t.base)
.await
{
warn!(target: "meta-feeder::enrich", cid = %t.cid, error = %e, "filename-parser /process failed");
}
}
}
/// POST `/process` to a plugin. Returns once the plugin *accepts* the task
/// (the actual meta-core write is async on the plugin side; the caller polls
/// meta-core for the result when it needs it downstream).
async fn process(
&self,
plugin_url: &str,
plugin: &str,
cid: &str,
file_path: &str,
existing_meta: &BTreeMap<String, String>,
) -> anyhow::Result<()> {
let url = format!("{}/process", plugin_url.trim_end_matches('/'));
let body = ProcessRequest {
task_id: format!("{plugin}-{cid}-{}", now_nanos()),
cid,
file_path,
callback_url: &self.cfg.callback_url,
meta_core_url: &self.cfg.meta_core_url,
existing_meta,
};
let resp = self.http.post(&url).json(&body).send().await?;
if !resp.status().is_success() {
let status = resp.status();
let text = resp.text().await.unwrap_or_default();
anyhow::bail!("{plugin} POST {url} returned {status}: {text}");
}
Ok(())
}
/// POST `/configure` to the tmdb plugin with the API key + language.
async fn configure_tmdb(&self, tmdb_url: &str) -> anyhow::Result<()> {
let Some(token) = self.cfg.tmdb_token.clone() else {
anyhow::bail!("no TMDB token configured");
};
let url = format!("{}/configure", tmdb_url.trim_end_matches('/'));
let body = serde_json::json!({
"config": { "apiKey": token, "language": self.cfg.tmdb_language }
});
let resp = self.http.post(&url).json(&body).send().await?;
if !resp.status().is_success() {
let status = resp.status();
let text = resp.text().await.unwrap_or_default();
anyhow::bail!("tmdb /configure returned {status}: {text}");
}
Ok(())
}
/// POST `/configure` to the opensubtitles plugin with the API key, account
/// credentials, and wanted languages.
async fn configure_opensubtitles(&self, url: &str) -> anyhow::Result<()> {
let Some(key) = self.cfg.opensubtitles_api_key.clone() else {
anyhow::bail!("no OpenSubtitles API key configured");
};
let cfg_url = format!("{}/configure", url.trim_end_matches('/'));
let body = serde_json::json!({
"config": {
"apiKey": key,
"username": self.cfg.opensubtitles_username,
"password": self.cfg.opensubtitles_password,
"languages": self.cfg.opensubtitles_languages,
}
});
let resp = self.http.post(&cfg_url).json(&body).send().await?;
if !resp.status().is_success() {
let status = resp.status();
let text = resp.text().await.unwrap_or_default();
anyhow::bail!("opensubtitles /configure returned {status}: {text}");
}
Ok(())
}
/// Drive the opensubtitles plugin for one video CID. Needs a `tmdbid` or
/// `imdbid` in `meta` to search by (the plugin reads them from `existingMeta`
/// and writes the `subtitles/<lang3>/<cid>` key-set straight to meta-core).
/// Best-effort: any failure is logged and swallowed.
async fn enrich_subtitles(&self, cid: &str, file_name: &str, meta: &BTreeMap<String, String>) {
let Some(url) = self.cfg.opensubtitles_url.clone() else {
return;
};
if !meta.contains_key("tmdbid") && !meta.contains_key("imdbid") {
debug!(
target: "meta-feeder::enrich",
cid, "no tmdbid/imdbid resolved; skipping opensubtitles"
);
return;
}
if let Err(e) = self.configure_opensubtitles(&url).await {
warn!(
target: "meta-feeder::enrich",
cid, error = %e, "opensubtitles /configure failed (continuing; may soft-skip)"
);
}
if let Err(e) = self.process(&url, "opensubtitles", cid, file_name, meta).await {
warn!(
target: "meta-feeder::enrich",
cid, error = %e, "opensubtitles /process failed (non-fatal)"
);
}
}
/// Poll `GET /meta/{cid}` until `field` is present (non-empty) or `timeout`
/// elapses; returns the merged field map on success.
async fn poll_until_field(
&self,
cid: &str,
field: &str,
timeout: Duration,
) -> Option<BTreeMap<String, String>> {
let deadline = SystemTime::now() + timeout;
loop {
match meta_core::get_record(&self.http, &self.cfg.meta_core_url, cid).await {
Ok(Some(fields)) if fields.get(field).is_some_and(|v| !v.is_empty()) => {
return Some(fields);
}
Ok(_) => {}
Err(e) => debug!(
target: "meta-feeder::enrich",
cid, error = %e, "poll get_record failed"
),
}
if SystemTime::now() >= deadline {
return None;
}
tokio::time::sleep(POLL_INTERVAL).await;
}
}
}
fn now_nanos() -> u128 {
SystemTime::now()
.duration_since(UNIX_EPOCH)
.map(|d| d.as_nanos())
.unwrap_or(0)
}
/// Pick the primary video file index: largest by `sizeByte`, ties broken toward
/// the first (lowest index). Falls back to 0 for an all-unsized bundle.
fn pick_primary(videos: &[EnrichTarget]) -> usize {
videos
.iter()
.enumerate()
.max_by_key(|(i, v)| {
let size = v.base.get("sizeByte").and_then(|s| s.parse::<u64>().ok()).unwrap_or(0);
// larger size wins; on a tie the smaller index wins (Reverse).
(size, std::cmp::Reverse(*i))
})
.map(|(i, _)| i)
.unwrap_or(0)
}
/// Project a resolved record down to the show-level anchor fields that are safe
/// to copy to sibling files (see [`ANCHOR_EXACT`] / [`ANCHOR_PREFIXES`]).
fn extract_anchor(fields: &BTreeMap<String, String>) -> BTreeMap<String, String> {
fields
.iter()
.filter(|(k, _)| {
ANCHOR_EXACT.contains(&k.as_str())
|| ANCHOR_PREFIXES.iter().any(|p| k.starts_with(p))
})
.map(|(k, v)| (k.clone(), v.clone()))
.collect()
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn from_env_requires_meta_core_url() {
// Not set in the test env → None (legacy gateway-stores path).
// (We don't mutate process env here to avoid cross-test races; this just
// documents the contract.)
assert!(EnrichmentConfig::from_env().is_none() || EnrichmentConfig::from_env().is_some());
}
fn tgt(cid: &str, size: u64) -> EnrichTarget {
EnrichTarget {
cid: cid.into(),
file_name: cid.into(),
base: BTreeMap::from_iter([("sizeByte".to_string(), size.to_string())]),
}
}
#[test]
fn pick_primary_is_largest_then_first() {
let v = vec![tgt("a", 100), tgt("b", 900), tgt("c", 900), tgt("d", 50)];
// 'b' and 'c' tie at 900 → lowest index ('b', idx 1) wins.
assert_eq!(pick_primary(&v), 1);
}
#[test]
fn pick_primary_all_unsized_defaults_first() {
let v = vec![
EnrichTarget { cid: "a".into(), file_name: "a".into(), base: BTreeMap::new() },
EnrichTarget { cid: "b".into(), file_name: "b".into(), base: BTreeMap::new() },
];
assert_eq!(pick_primary(&v), 0);
}
#[test]
fn extract_anchor_keeps_show_level_drops_per_file() {
let rec = BTreeMap::from_iter([
("tmdbid".to_string(), "10378".to_string()),
("imdbid".to_string(), "tt1254207".to_string()),
("poster".to_string(), "bafkPOSTER".to_string()),
("genres".to_string(), "Animation|Comedy".to_string()),
("plot/eng".to_string(), "a plot".to_string()),
("tags".to_string(), "tmdb-verified".to_string()),
// per-file — must NOT be fanned:
("originalTitle".to_string(), "Big Buck Bunny".to_string()),
("season".to_string(), "1".to_string()),
("episode".to_string(), "3".to_string()),
("fileName".to_string(), "ep3.mkv".to_string()),
]);
let a = extract_anchor(&rec);
assert!(a.contains_key("tmdbid") && a.contains_key("poster") && a.contains_key("plot/eng"));
assert!(a.contains_key("genres") && a.contains_key("tags") && a.contains_key("imdbid"));
assert!(!a.contains_key("originalTitle"), "originalTitle is per-file");
assert!(!a.contains_key("season") && !a.contains_key("episode"));
assert!(!a.contains_key("fileName"));
}
}
crates/meta-feeder-sdk/src/filename_meta.rs
-1309
View File
File diff suppressed because it is too large Load Diff
crates/meta-feeder-sdk/src/hash.rs
-1076
View File
File diff suppressed because it is too large Load Diff
crates/meta-feeder-sdk/src/lang.rs
-81
View File
@@ -1,81 +0,0 @@
//! Language-code normalization — ISO 639 alias folding, **not** filename
//! parsing.
//!
//! This lives apart from [`crate::filename_meta`] on purpose: it doesn't read a
//! release title, it canonicalizes a *language code* (an indexer's declared
//! language, a query's `languages:` filter, a title-derived stamp) onto the
//! single ISO 639-2/B alpha-3 vocabulary the rest of the stack compares against.
//! It is the one piece of the old `filename_meta` module that has no equivalent
//! in the `metamesh-plugin-filename-parser` plugin — because it isn't filename
//! parsing — so it stays in Rust while the title-parsing algorithm itself is
//! owned by the plugin. Callers in the **request hot path** (e.g. the torznab
//! indexer-language fan-out skip) need it synchronously, with no network hop.
/// Normalize a language code to the ISO 639-2/B alpha-3 vocabulary the stack
/// emits (`fre`/`ger`/`chi`, **not** the 639-2/T `fra`/`deu`/`zho` nor the
/// 639-1 two-letter forms), lowercased. This is the space meta-share's
/// "languages I read" preference uses and that the title-derived language stamps
/// write, so an operator who declares an indexer's language as `fra`/`de`/`zh`
/// still lines up with a `fre`/`ger`/`chi` query filter. Unknown /
/// already-canonical codes pass through lowercased+trimmed unchanged.
pub fn normalize_lang_code(code: &str) -> String {
let c = code.trim().to_ascii_lowercase();
let mapped = match c.as_str() {
// 639-2/T and 639-1 → 639-2/B for the codes that actually differ.
"fra" | "fr" => "fre",
"deu" | "de" => "ger",
"zho" | "zh" => "chi",
"nld" | "nl" => "dut",
"ell" | "el" => "gre",
"ron" | "ro" => "rum",
"slk" | "sk" => "slo",
"ces" | "cs" => "cze",
// 639-1 → 639-2 for the rest of the language-tag output space (these
// share the same /B and /T alpha-3, so only the two-letter form maps).
"en" => "eng",
"ja" => "jpn",
"it" => "ita",
"es" => "spa",
"pt" => "por",
"ko" => "kor",
"ru" => "rus",
"pl" => "pol",
"hu" => "hun",
"uk" => "ukr",
"bg" => "bul",
"sv" => "swe",
"no" => "nor",
"da" => "dan",
"fi" => "fin",
"tr" => "tur",
"ar" => "ara",
"hi" => "hin",
"ta" => "tam",
"te" => "tel",
"ml" => "mal",
"vi" => "vie",
"th" => "tha",
_ => return c,
};
mapped.to_string()
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn normalize_lang_maps_to_639_2b_vocabulary() {
// 639-2/T and 639-1 spellings collapse onto the /B forms the stack uses.
assert_eq!(normalize_lang_code("fra"), "fre");
assert_eq!(normalize_lang_code("FR"), "fre");
assert_eq!(normalize_lang_code("deu"), "ger");
assert_eq!(normalize_lang_code("zh"), "chi");
assert_eq!(normalize_lang_code("nld"), "dut");
// Already-canonical / unknown codes pass through trimmed + lowercased.
assert_eq!(normalize_lang_code(" ENG "), "eng");
assert_eq!(normalize_lang_code("fre"), "fre");
assert_eq!(normalize_lang_code("mul"), "mul");
assert_eq!(normalize_lang_code("xyz"), "xyz");
}
}
crates/meta-feeder-sdk/src/lib.rs
-50
View File
@@ -1,50 +0,0 @@
//! `meta-feeder-sdk` — the source-agnostic foundation shared by the gateway
//! core and every feeder sidecar binary.
//!
//! A *feeder* implements [`FeederPlugin`] (find records, fetch bytes) for one
//! or more upstreams, and [`serve_feeders`] exposes them over the feeder HTTP
//! contract. The gateway core consumes that contract via its
//! `RemoteFeederPlugin` and keeps the libp2p wire, the bitswap blockstore, the
//! hashing-into-the-blockstore, and the meta-core store-back to itself.
//!
//! Deliberately libp2p-free and blockstore-free — see this crate's `Cargo.toml`.
// `redb::Error` is a large enum; the cache wrappers (`cache.rs`) return it
// directly rather than boxing on every embedded-DB call. Boxing each would be
// pure noise for a single-process store, so silence the lint crate-wide.
#![allow(clippy::result_large_err)]
/// pcs-convention default meta-core root a self-publishing feeder targets when
/// neither its dashboard config nor the `META_CORE_URL` env seed sets one. Keeps
/// the modular storage seam out of compose: a standard install just works; a
/// non-standard meta-core is set in the feeder's config UI. Matches the
/// `metacore-app` service the AppStore feeder apps ship on the shared network.
pub const DEFAULT_META_CORE_URL: &str = "http://metacore-app:9000";
pub mod cache;
pub mod common;
pub mod config;
pub mod domain;
pub mod enrich;
pub mod filename_meta;
pub mod hash;
pub mod lang;
pub mod meta_core;
pub mod plugin;
pub mod query;
pub mod query_eval;
pub mod serve;
pub mod types;
pub use config::{ConfigField, ConfigSchema, FieldKind};
pub use enrich::{EnrichTarget, EnrichmentConfig, Enricher};
pub use meta_core::FeederStore;
pub use plugin::{
upstream_id_field, ConfigError, FeederPlugin, HashKind, HashOutcome, PluginRegistry,
};
pub use query::{GatewayQuery, GatewaySearchEvent, GatewayWireError, Negation, RangeFilter};
pub use serve::{
configure_plugins, router, serve_feeders, ComputeRequest, ComputeResponse, HashKindDto,
HealthResponse, ManifestResponse, OutcomeDto, PluginManifest, QueryRequest, QueryResponse,
};
pub use types::{ByteStream, DiscoveryId, DiscoveryRecord, GatewayError, Hash, PluginHealth};
crates/meta-feeder-sdk/src/meta_core.rs
-335
View File
@@ -1,335 +0,0 @@
//! Lightweight meta-core write/read client for feeders that **self-publish**.
//!
//! Post-"smart feeder / dumb gateway" split: a feeder given a `META_CORE_URL`
//! writes its own base record into meta-core instead of returning it for the
//! gateway dispatcher to store. This keeps the gateway a pure aggregator +
//! request router; the feeder owns the full record lifecycle for its upstream.
//!
//! The write shape is **byte-for-byte the same** as the gateway's
//! `meta_core::build_metadata_body` (`crates/meta-gateway/src/meta_core.rs`):
//! `cid_*` fields collapse to the `cids/<cid> = "true"` key-set, and a
//! `provenance` JSON blob is stamped. Keeping them identical means a record
//! written by the feeder is indistinguishable from one the dispatcher wrote —
//! the meta-share ingester, meta-watch, and ranking all behave the same.
//!
//! `PUT /api/metadata/{cid}` is merge/upsert (confirmed: meta-core
//! `SetMetadataFlat` sets only the fields in the payload, never deletes), so the
//! feeder's base write and the enrichment plugins' later `PATCH /meta/{cid}`
//! merges accumulate on the same `/file/{cid}` hash.
use std::collections::BTreeMap;
use std::time::{SystemTime, UNIX_EPOCH};
use serde::Deserialize;
use tracing::warn;
use crate::plugin::{HashKind, HashOutcome};
use crate::types::{DiscoveryRecord, Hash};
/// Provenance recorded on every gateway-acquired record. Mirrors the gateway
/// dispatcher's shape so feeder-written and dispatcher-written records are
/// interchangeable.
#[derive(Clone, Debug)]
pub struct Provenance {
pub upstream: String,
pub record_id: String,
/// Stable per-deployment id (the gateway peer this feeder backs, from
/// `META_GATEWAY_PEER_ID`, else the feeder hostname). Analytics group by it.
pub gateway_peer: String,
pub acquired_at: u64,
}
impl Provenance {
/// Build a provenance stamp for `upstream`/`record_id`, stamping "now".
pub fn now(upstream: &str, record_id: &str, gateway_peer: &str) -> Self {
Provenance {
upstream: upstream.to_string(),
record_id: record_id.to_string(),
gateway_peer: gateway_peer.to_string(),
acquired_at: SystemTime::now()
.duration_since(UNIX_EPOCH)
.map(|d| d.as_secs())
.unwrap_or(0),
}
}
}
/// JSON-stringified provenance value. Same key set + `"source":"gateway"` as the
/// gateway dispatcher (`crates/meta-gateway/src/meta_core.rs::provenance_to_json`).
pub fn provenance_to_json(p: &Provenance) -> String {
serde_json::to_string(&serde_json::json!({
"source": "gateway",
"upstream": p.upstream,
"recordId": p.record_id,
"gatewayPeer": p.gateway_peer,
"acquiredAt": p.acquired_at,
}))
.expect("static JSON shape — serialisation is infallible")
}
/// Build the flat meta-core PUT body from a [`DiscoveryRecord`] + provenance.
///
/// Identical transform to the gateway dispatcher: any `cid_*` field becomes a
/// `cids/<value> = "true"` key-set member (the CID is self-describing — the
/// algorithm is the multicodec; see METADATA_KEYS.md §2/§14.13), everything else
/// passes through, and `provenance` is stamped last (overriding any plugin-set
/// value).
pub fn build_metadata_body(
record: &DiscoveryRecord,
provenance: &Provenance,
) -> BTreeMap<String, String> {
let mut body: BTreeMap<String, String> = BTreeMap::new();
for (k, v) in &record.fields {
if k.starts_with("cid_") && !v.is_empty() {
body.insert(format!("cids/{v}"), "true".to_string());
} else {
body.insert(k.clone(), v.clone());
}
}
body.insert("provenance".to_string(), provenance_to_json(provenance));
body
}
/// `PUT /api/metadata/{cid}` — write (merge/upsert) the record body into
/// meta-core. `base_url` is the meta-core root (e.g. `http://…-core:9000`).
pub async fn put_record(
http: &reqwest::Client,
base_url: &str,
cid: &str,
body: &BTreeMap<String, String>,
) -> anyhow::Result<()> {
let url = format!("{}/api/metadata/{}", base_url.trim_end_matches('/'), cid);
let resp = http.put(&url).json(body).send().await?;
let status = resp.status();
if !status.is_success() {
let text = resp.text().await.unwrap_or_default();
anyhow::bail!("meta-core PUT {url} returned {status}: {text}");
}
Ok(())
}
/// Shape of meta-core's `GET /meta/{cid}` response (`{ hashId, metadata }`).
#[derive(Debug, Deserialize)]
struct MetaGetResponse {
#[serde(default)]
metadata: Option<BTreeMap<String, String>>,
}
/// `GET /meta/{cid}` — read the current flat field map for a record, or `None`
/// when meta-core has no record yet (404). Used to read back a plugin's merged
/// fields (e.g. filename-parser's `originalTitle`/`movieYear`) before handing
/// them to the next plugin in the pipeline.
pub async fn get_record(
http: &reqwest::Client,
base_url: &str,
cid: &str,
) -> anyhow::Result<Option<BTreeMap<String, String>>> {
let url = format!("{}/meta/{}", base_url.trim_end_matches('/'), cid);
let resp = http.get(&url).send().await?;
if resp.status() == reqwest::StatusCode::NOT_FOUND {
return Ok(None);
}
if !resp.status().is_success() {
anyhow::bail!("meta-core GET {url} returned {}", resp.status());
}
let parsed: MetaGetResponse = resp.json().await?;
Ok(parsed.metadata)
}
/// Self-publishing store for feeders that own their record lifecycle (D4):
/// the feeder writes its own **metadata-only** records to meta-core and returns
/// sparse outcomes so the gateway dispatcher skips them; the meta-share ingester
/// Kamilata-propagates them.
///
/// **Byte outcomes are deliberately NOT self-published.** A feeder-side WebDAV
/// write to the gateway core would be seeded by nobody: the gateway core runs
/// with `ENABLE_FILE_WATCHER=false` (so meta-core emits no `/api/events/files`
/// for it, and the ingester's `ipfs_seed` — which seeds only watched files —
/// never sees it), and a sparse outcome makes the gateway skip its own
/// blockstore seed. So byte-bearing outcomes (torznab full-file / subtitles)
/// pass through unchanged and the **gateway** continues to WebDAV-PUT + bitswap-
/// seed them (it's the libp2p/bitswap host). Result: feeder owns the metadata
/// records (the Kamilata-propagated bulk), gateway owns byte delivery — a clean
/// split with no seeding regression. Tribler is metadata-only and uses
/// [`crate::enrich::Enricher`]; this is torznab's record-owning counterpart.
#[derive(Clone)]
pub struct FeederStore {
http: reqwest::Client,
meta_core_url: String,
gateway_peer: String,
}
impl FeederStore {
pub fn new(http: reqwest::Client, meta_core_url: String, gateway_peer: String) -> Self {
FeederStore {
http,
meta_core_url: meta_core_url.trim_end_matches('/').to_string(),
gateway_peer,
}
}
/// Build from the feeder env. `None` when `META_CORE_URL` is unset (the
/// feeder then falls back to the legacy "return outcomes, gateway stores"
/// path). `gateway_peer` ← `META_GATEWAY_PEER_ID` / `HOSTNAME`.
pub fn from_env() -> Option<Self> {
let meta_core_url = env_nonempty("META_CORE_URL")?;
let gateway_peer = env_nonempty("META_GATEWAY_PEER_ID")
.or_else(|| env_nonempty("HOSTNAME"))
.unwrap_or_else(|| "gateway-feeder".to_string());
let http = crate::common::build_http_client(30, "meta-feeder-store", None);
Some(Self::new(http, meta_core_url, gateway_peer))
}
pub fn meta_core_url(&self) -> &str {
&self.meta_core_url
}
/// Raw keys containing `contains` (`GET /api/kv/search`). Returns
/// `(keys, truncated)`; `truncated` means the server capped the result and
/// there may be more.
///
/// This is a keyspace scan, so it belongs on rare control-plane paths only —
/// today, invalidating the gateway's search-coverage markers when a feeder's
/// upstream source set changes. Never call it per search.
pub async fn search_keys(&self, contains: &str, limit: usize) -> anyhow::Result<(Vec<String>, bool)> {
let url = format!("{}/api/kv/search", self.meta_core_url);
let resp = self
.http
.get(&url)
.query(&[("contains", contains), ("limit", &limit.to_string())])
.send()
.await?;
let status = resp.status();
if !status.is_success() {
anyhow::bail!("meta-core KV search `{contains}` returned {status}");
}
#[derive(Deserialize)]
struct KvSearchResponse {
#[serde(default)]
keys: Vec<String>,
#[serde(default)]
truncated: bool,
}
let body: KvSearchResponse = resp.json().await?;
Ok((body.keys, body.truncated))
}
/// Delete one raw KV key (`DELETE /api/kv/value?key=`). A missing key is not
/// an error.
pub async fn delete_kv(&self, key: &str) -> anyhow::Result<()> {
let url = format!("{}/api/kv/value", self.meta_core_url);
let resp = self.http.delete(&url).query(&[("key", key)]).send().await?;
let status = resp.status();
if status.is_success() || status == reqwest::StatusCode::NOT_FOUND {
return Ok(());
}
anyhow::bail!("meta-core KV DELETE {key} returned {status}");
}
/// Self-publish a **metadata-only** outcome to meta-core, returning the
/// **sparse** outcome (`record: None`) so the gateway's `(_, None)` branch
/// skips it — no double write. Byte outcomes and already-sparse outcomes are
/// returned unchanged (the gateway stores + seeds the bytes; see the struct
/// docs). On a write failure the original outcome is returned so the gateway
/// stores it (graceful degradation, never lost).
pub async fn publish(
&self,
upstream_id: &str,
record_id: &str,
outcome: HashOutcome,
) -> HashOutcome {
// Only self-publish metadata-only outcomes (bytes None, record Some).
if outcome.bytes.is_some() || outcome.record.is_none() {
return outcome;
}
let cid = outcome.hash.0.clone();
let record = outcome.record.as_ref().expect("record is Some by the guard");
match self
.store_metadata_only(upstream_id, record_id, &cid, record)
.await
{
Ok(()) => sparse(outcome.hash, outcome.hash_kind),
Err(e) => {
warn!(target: "meta-feeder::store", cid = %cid, error = %e,
"metadata self-publish failed; falling back to gateway store");
outcome
}
}
}
/// Metadata-only self-publish: `PUT /api/metadata/{cid}` with provenance.
async fn store_metadata_only(
&self,
upstream_id: &str,
record_id: &str,
cid: &str,
record: &DiscoveryRecord,
) -> anyhow::Result<()> {
let prov = Provenance::now(upstream_id, record_id, &self.gateway_peer);
let body = build_metadata_body(record, &prov);
put_record(&self.http, &self.meta_core_url, cid, &body).await
}
}
fn sparse(hash: Hash, hash_kind: HashKind) -> HashOutcome {
HashOutcome {
hash,
hash_kind,
bytes: None,
record: None,
file_extension: None,
}
}
fn env_nonempty(key: &str) -> Option<String> {
std::env::var(key)
.ok()
.map(|s| s.trim().to_string())
.filter(|s| !s.is_empty())
}
#[cfg(test)]
mod tests {
use super::*;
fn rec() -> DiscoveryRecord {
DiscoveryRecord {
upstream_id: "tribler".into(),
record_id: "abc".into(),
fields: BTreeMap::from_iter([
("title".into(), "Big Buck Bunny".into()),
("cid_btih_v1_file".into(), "bafkPERFILE".into()),
]),
}
}
#[test]
fn provenance_shape_matches_gateway() {
let p = Provenance {
upstream: "tribler".into(),
record_id: "abc".into(),
gateway_peer: "metagateway-tribler".into(),
acquired_at: 1_747_504_000,
};
let v: serde_json::Value = serde_json::from_str(&provenance_to_json(&p)).unwrap();
assert_eq!(v["source"], "gateway");
assert_eq!(v["upstream"], "tribler");
assert_eq!(v["recordId"], "abc");
assert_eq!(v["gatewayPeer"], "metagateway-tribler");
assert_eq!(v["acquiredAt"], 1_747_504_000_u64);
}
#[test]
fn body_collapses_cid_fields_and_stamps_provenance() {
let p = Provenance::now("tribler", "abc", "peer");
let body = build_metadata_body(&rec(), &p);
assert_eq!(body.get("title").map(String::as_str), Some("Big Buck Bunny"));
// cid_* collapsed to cids/<value>=true; no cid_* survives.
assert!(body.keys().all(|k| !k.starts_with("cid_")));
assert_eq!(
body.get("cids/bafkPERFILE").map(String::as_str),
Some("true")
);
assert!(body.contains_key("provenance"));
}
}
crates/meta-feeder-sdk/src/plugin.rs
-258
View File
@@ -1,258 +0,0 @@
//! The `FeederPlugin` trait + supporting types.
//!
//! A feeder plugin bridges one external upstream (`gutenberg`, `arxiv`,
//! `torznab`, …) into the MetaMesh network. It **finds** records and
//! **fetches** bytes; it does NOT hash-into-a-blockstore, store back into
//! meta-core, or speak libp2p — those stay in the gateway core. This is why
//! `FeederPlugin` has no `set_blockstore` / `set_tmdb_budget` hooks (the
//! gateway's `GatewayPlugin` carried them): preview-seeding moves to the core,
//! and a feeder that needs a TMDB budget owns it internally.
//!
//! The `serve_feeders` harness ([`crate::serve`]) wraps any set of
//! `FeederPlugin`s in the feeder HTTP contract; the gateway core's
//! `RemoteFeederPlugin` is the matching HTTP client.
use std::collections::HashMap;
use std::path::Path;
use async_trait::async_trait;
use futures::stream::BoxStream;
// Re-exported so plugins can `use meta_feeder_sdk::plugin::GatewayQuery` (the
// gateway's original `plugin.rs` re-exported it the same way).
pub use crate::query::GatewayQuery;
use crate::query::GatewaySearchEvent;
use crate::types::{ByteStream, DiscoveryRecord, GatewayError, Hash, PluginHealth};
/// One element of what [`FeederPlugin::compute_outcomes`] returns.
///
/// The `hash` is the value shipped to the gateway core. The other fields are
/// signals the core's auto-store uses to persist the content into meta-core.
/// The core's three-branch auto-store routes on `(bytes, record)`:
/// - `(Some, Some)` — full store: WebDAV PUT + metadata PUT with `filePath`.
/// - `(None, Some)` — metadata-only PUT (no local bytes; e.g. torznab samples
/// the middle 1 MiB of a multi-GiB video and surfaces only the record).
/// - `(_, None)` — skip. Cache hits land here.
///
/// `file_extension` MUST be `None` when `bytes` is `None`.
#[derive(Debug)]
pub struct HashOutcome {
pub hash: Hash,
/// Which hash family this CID belongs to. Drives the core's routing —
/// the bitswap blockstore seed only fires for `Sha2_256`.
pub hash_kind: HashKind,
pub bytes: Option<bytes::Bytes>,
pub record: Option<DiscoveryRecord>,
/// File extension to append when naming the WebDAV-side blob (e.g.
/// `"epub"`). No leading dot. MUST be `None` when `bytes` is `None`.
pub file_extension: Option<String>,
}
/// Hash family discriminator on [`HashOutcome`].
///
/// - `Midhash256` — size-prefix-plus-middle-1MB-sample, custom multicodec
/// `0x1000`. Fast, MetaMesh-internal, not retrievable via public IPFS.
/// - `Sha2_256` — standard IPFS CIDv1 over full bytes. Retrievable via bitswap
/// once the bytes are in the core's blockstore. What every shipping plugin
/// today emits.
/// - `BtV1File` — a single file inside a BitTorrent v1 torrent, custom
/// multicodec `0x1001`. Locator, not content hash: opaque to bitswap.
/// - `NzbRelease` — a Usenet release (a Newznab listing), custom multicodec
/// `nzb-release` `0x1005`. Self-describing locator: the cid embeds `{host,
/// id}` in an identity multihash, opaque to bitswap and redeemed to bytes
/// only by a credentialed meta-share peer, which decodes the locator from the
/// cid and grabs the `.nzb` via `t=get` — no KV side-table.
/// - `CardLocator` — the identity of a *work* (a series, a film) as published by
/// a metadata bridge, custom multicodec `0x1007`. The only family with **no
/// bytes behind it at all**: the record IS the payload. Always paired with
/// `bytes: None`, so it lands in the `(None, Some)` metadata-only branch
/// above. Opaque to bitswap, and deliberately so — there is nothing to seed.
/// - `NzbPosting` — a Usenet posting we **scanned ourselves**, custom multicodec
/// `nzb-posting` `0x1003`. A content-derived digest over the article
/// Message-ID set (not a locator: no host is embedded), so any peer with a
/// plain NNTP provider can fetch it — no `IndexerCred`. Opaque to bitswap: the
/// digest is over the id set, not over any block, so the `.nzb` manifest
/// travels separately as an ordinary sha2-256 cid in the record's `manifest`
/// field. Produced by `hash::compute_nzb_posting_cid`.
/// - `YtVideo` — a **delegated-playback** reference, custom multicodec
/// `yt-video` `0x1008`. Like `CardLocator` it is always paired with
/// `bytes: None` and lands in the `(None, Some)` metadata-only branch, but it
/// means something stronger than "no bytes yet": the bytes **exist and are
/// permanently someone else's**. Nothing is fetched, seeded or
/// content-addressed, and meta-share's byte path REFUSES the cid with a `400`
/// rather than 404ing. Produced by `hash::compute_yt_video_cid`.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum HashKind {
/// Reserved: no feeder currently emits midhash256 outcomes, but the family
/// is real across meta-core / meta-share, so the variant stays.
#[allow(dead_code)]
Midhash256,
Sha2_256,
BtV1File,
NzbRelease,
CardLocator,
NzbPosting,
YtVideo,
}
/// Startup-only configuration error returned by [`FeederPlugin::configure`].
#[derive(Debug, thiserror::Error)]
pub enum ConfigError {
#[error("plugin {plugin}: required config missing: {what}")]
MissingConfig {
plugin: &'static str,
/// Human-readable description of the missing setting and where to
/// supply it. Used by the harness's soft-skip warning.
what: &'static str,
},
#[error("plugin {plugin}: cache dir setup failed: {source}")]
CacheSetup {
plugin: &'static str,
#[source]
source: std::io::Error,
},
#[error("plugin {plugin}: {source}")]
Other {
plugin: &'static str,
#[source]
source: anyhow::Error,
},
}
/// Canonical field name for an upstream's stable per-record identifier.
/// Every plugin's `DiscoveryRecord.fields` MUST include this key
/// (`gutenbergid:11`, `tmdbid:31910`, `arxivid:2106.07447`). The rule is
/// `format!("{upstream_id}id")` — lowercase, no separator.
pub fn upstream_id_field(upstream_id: &str) -> String {
format!("{upstream_id}id")
}
/// Key-set prefix for the provenance `source/<label>` field (METADATA_KEYS §5).
/// Each member is its own hash field (`source/gateway:nyaa.si = "true"`) so two
/// producers that reach the same content by different routes union without a
/// last-writer-wins clobber. The `<label>` uses `:` internally (not `/`, which
/// is the key-set path separator) — e.g. `gateway:nyaa.si`, `gateway:tribler`.
pub const SOURCE_KEYSET_PREFIX: &str = "source/";
/// Stamp the SDK's default provenance member — `source/gateway:<upstream_id>` —
/// onto a record's fields, but **only if it carries no `source/*` member yet**.
/// This gives every gateway hit a provenance label for free (so meta-watch never
/// falls back to "unknown" for a feeder record), while a plugin that knows a
/// finer origin (torznab → the specific Prowlarr indexer) stamps its own flat
/// `source/*` facets first (`source/gateway`, `source/<protocol>`,
/// `source/<indexer-slug>`) and this default then no-ops.
pub fn stamp_default_source(
fields: &mut std::collections::BTreeMap<String, String>,
upstream_id: &str,
) {
if fields.keys().any(|k| k.starts_with(SOURCE_KEYSET_PREFIX)) {
return;
}
fields.insert(
format!("{SOURCE_KEYSET_PREFIX}gateway:{upstream_id}"),
"true".to_string(),
);
}
/// Static plugin contract. Each enabled `upstream_id` is implemented by
/// exactly one `Box<dyn FeederPlugin>`. Lives behind `&self` in steady state —
/// called concurrently, owns any interior mutability for caches.
///
/// **Field-naming convention.** Every record a plugin returns MUST include the
/// canonical `<upstream_id>id` field — see [`upstream_id_field`].
#[async_trait]
pub trait FeederPlugin: Send + Sync + 'static {
fn upstream_id(&self) -> &'static str;
fn configure(&mut self, cache_dir: &Path) -> Result<(), ConfigError>;
async fn handle_query(
&self,
query: &GatewayQuery,
max_results: usize,
) -> Result<Vec<DiscoveryRecord>, GatewayError>;
/// Streaming variant of [`handle_query`]. The default impl collects
/// [`handle_query`] and replays it as `Base*` + `Done` — correct for every
/// plugin with no incremental enrichment (gutenberg, arxiv, …). torznab
/// overrides it to decouple base records from rate-limited TMDB enrichment.
async fn handle_query_stream(
&self,
query: &GatewayQuery,
max_results: usize,
) -> Result<BoxStream<'static, GatewaySearchEvent>, GatewayError> {
let records = self.handle_query(query, max_results).await?;
let events = records
.into_iter()
.map(GatewaySearchEvent::Base)
.chain(std::iter::once(GatewaySearchEvent::Done));
Ok(Box::pin(futures::stream::iter(events)))
}
/// Resolve `record_id` into one or more content-addressed outcomes.
///
/// - `Err(_)` — the upstream record itself is unresolvable.
/// - `Ok(vec![])` — resolved but yielded zero outcomes.
/// - `Ok(partial_vec)` — bundle-success: per-sibling failures are
/// best-effort (log + drop), surface whatever succeeded.
async fn compute_outcomes(&self, record_id: &str) -> Result<Vec<HashOutcome>, GatewayError>;
async fn handle_fetch(&self, _record_id: &str) -> Result<Option<ByteStream>, GatewayError> {
Ok(None)
}
fn health(&self) -> PluginHealth {
PluginHealth::Ok
}
fn served_file_types(&self) -> &'static [&'static str] {
&[]
}
fn served_content_kinds(&self) -> &'static [&'static str] {
&[]
}
/// Self-describing config schema. The feeder SDK serves this at
/// `GET /config/schema` and renders a generic form from it — the gateway and
/// UI carry no per-plugin field knowledge. Default: no configuration.
fn config_schema(&self) -> crate::config::ConfigSchema {
crate::config::ConfigSchema::default()
}
/// The plugin's current *effective* config as JSON, **unredacted** (secrets
/// included). The SDK redacts it before serving `GET /config/values` and uses
/// it as the merge base on the first save (before any `config.json` exists).
/// Default: empty object. Implementors should mirror the keys in
/// [`config_schema`](FeederPlugin::config_schema).
fn config_values(&self) -> serde_json::Value {
serde_json::json!({})
}
async fn get_blob(&self, _cid: &str) -> Option<Vec<u8>> {
None
}
}
/// Static plugin registry. Built once at startup; read-only thereafter.
/// Keyed by `upstream_id`.
pub type PluginRegistry = HashMap<&'static str, Box<dyn FeederPlugin>>;
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn config_error_messages_include_plugin_name() {
let e = ConfigError::MissingConfig {
plugin: "scihub",
what:
"mirrors (set them in the dashboard or gateway-config.json plugins.scihub.mirrors)",
};
let msg = e.to_string();
assert!(msg.contains("scihub"));
assert!(msg.contains("mirrors"));
}
}
crates/meta-feeder-sdk/src/query.rs
-147
View File
@@ -1,147 +0,0 @@
//! Structured query envelope + streaming-search event + wire-error types,
//! shared by feeders (their HTTP contract) and the gateway core (its libp2p
//! wire). These are pure serde data types — no libp2p, no transport.
//!
//! On the core side, `protocol.rs` re-uses these exact structs as the bincode
//! wire shape; the `wire_parity_*` tests there pin the byte encoding. Keep
//! field order / variant order / derive shape stable.
use std::collections::BTreeMap;
use serde::{Deserialize, Serialize};
use crate::types::{DiscoveryRecord, GatewayError};
/// Structured query envelope. Gateway plugins do best-effort pushdown to
/// upstream APIs with native query languages (`torznab cat=movie`,
/// `arxiv id_list=`).
///
/// **Authority rule:** `filters` / `ranges` / `negations` are authoritative;
/// `raw_text` is informational. A plugin that consumes both will double-filter
/// and under-match — pick one source per filter; structured wins.
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
pub struct GatewayQuery {
/// Original user text (e.g. `"naruto contentKind:movie imdbid:tt0903747"`).
/// Verbatim — never mutated by the conversion.
pub raw_text: String,
/// Bare-word leaves only, space-joined. The plugin's primary `q=` input
/// for upstream search APIs that take a single search box.
pub free_text: String,
/// Structured filters keyed by field name (raw, unstemmed values).
/// Multiple values per key = OR within a key; AND across keys.
pub filters: BTreeMap<String, Vec<String>>,
/// Numeric range filters (e.g. `movieYear:2020..2024`). Both bounds
/// optional → unbounded on that side.
pub ranges: Vec<RangeFilter>,
/// Negation filters (e.g. `NOT genres:horror`, bare `NOT word`).
pub negations: Vec<Negation>,
}
/// Numeric range filter inside [`GatewayQuery::ranges`]. Both bounds are
/// `Option<i64>` so open-ended ranges drop one bound to `None`.
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
pub struct RangeFilter {
pub field: String,
pub lo: Option<i64>,
pub hi: Option<i64>,
}
/// Negation entry inside [`GatewayQuery::negations`]. `field` is `Some(name)`
/// for typed negations (`NOT genres:horror`), `None` for bare-word negation.
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
pub struct Negation {
pub field: Option<String>,
pub value: String,
}
impl GatewayQuery {
/// Bare-text input to hand to a free-form upstream search API. Falls back
/// to `"*"` when the conversion produced no free-text leaves so upstreams
/// that 4xx on empty search return their top-ranked results instead.
pub fn free_text_or_star(&self) -> &str {
if self.free_text.trim().is_empty() {
"*"
} else {
self.free_text.as_str()
}
}
/// Construct a free-text-only `GatewayQuery` for in-process use (tests,
/// debug routes). `raw_text` == `free_text`; structured fields stay empty.
pub fn from_free_text(text: impl Into<String>) -> Self {
let text = text.into();
Self {
raw_text: text.clone(),
free_text: text,
filters: BTreeMap::new(),
ranges: Vec::new(),
negations: Vec::new(),
}
}
}
/// One frame in the streaming Search response.
///
/// A plugin emits `Base` records as soon as they're discovered, then
/// best-effort `EnrichPatch`/`Drop` events as enrichment lands, then a
/// terminal `Done`. The default `handle_query_stream` collects `handle_query`
/// and replays it as `Base*` + `Done`.
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
pub enum GatewaySearchEvent {
/// An un-enriched discovery record.
Base(DiscoveryRecord),
/// Incremental enrichment for a previously-sent `Base`, keyed by
/// `record_id`. `set` fields are inserted/overwritten; `remove` deleted.
EnrichPatch {
record_id: String,
set: BTreeMap<String, String>,
remove: Vec<String>,
},
/// Retract a previously-emitted `Base` record entirely.
Drop { record_id: String },
/// Terminal success — no more frames follow.
Done,
/// Terminal error — no more frames follow.
Error(GatewayWireError),
}
/// Wire-stable error shape. Distinct from in-process [`GatewayError`] because
/// adding a variant here is a wire-format change.
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize, thiserror::Error)]
pub enum GatewayWireError {
#[error("not found")]
NotFound,
#[error("rate limited; retry in {retry_after_s}s")]
RateLimited { retry_after_s: u32 },
#[error("upstream transient: {0}")]
UpstreamTransient(String),
#[error("upstream permanent: {0}")]
UpstreamPermanent(String),
/// The protocol was registered but no plugin claims this `upstream_id`.
#[error("plugin not loaded")]
PluginNotLoaded,
/// Internal codec / decode failure on the *receiver* side.
#[error("internal: {0}")]
Internal(String),
}
impl From<GatewayError> for GatewayWireError {
fn from(e: GatewayError) -> Self {
match e {
GatewayError::Transient(s) => GatewayWireError::UpstreamTransient(s),
GatewayError::Permanent(s) => GatewayWireError::UpstreamPermanent(s),
GatewayError::RateLimited { retry_after_s } => {
GatewayWireError::RateLimited { retry_after_s }
}
GatewayError::NotFound => GatewayWireError::NotFound,
// Internal errors are never leaked; map to a transient.
GatewayError::Internal(_) => {
GatewayWireError::UpstreamTransient("internal".to_string())
}
}
}
}
crates/meta-feeder-sdk/src/query_eval.rs
-514
View File
@@ -1,514 +0,0 @@
//! Structured-filter evaluation for gateway records.
//!
//! Two pure helpers consuming the [`crate::query::GatewayQuery`] struct:
//!
//! - [`record_matches`] — does a single record's `fields` map satisfy the
//! query's `filters` + `ranges` + typed `negations`?
//! - [`query_accepts_plugin`] — could a plugin advertising the given
//! `served_file_types` / `served_content_kinds` possibly produce a record
//! that satisfies the query? Used by plugins as a cheap early-return.
//!
//! The dispatcher applies `record_matches` to every plugin's return value, and
//! meta-share applies the same algorithm consumer-side to defend against stale
//! gateway peers — both tiers must agree on outcomes for any (record, query)
//! pair. See the original gateway crate docs for the full semantics.
use std::collections::BTreeMap;
use crate::query::GatewayQuery;
/// True iff a record's `fields` satisfies every filter, range, and typed
/// negation in `query`.
pub fn record_matches(fields: &BTreeMap<String, String>, query: &GatewayQuery) -> bool {
for (key, allowed) in &query.filters {
if key == "languages" {
if !languages_filter_matches(fields, allowed) {
return false;
}
continue;
}
// `genres` is a key-set too (`genres/<Name> = "true"`, METADATA_KEYS
// §`genres/{name}`) on everything the card tier writes. Without this arm
// a `genres:` filter looks up a flat `genres` field, finds nothing on a
// card, and drops it — so a genre-scoped row would return empty.
if key == "genres" {
if !genres_filter_matches(fields, allowed) {
return false;
}
continue;
}
let Some(field_value) = fields.get(key) else {
return false;
};
if !any_csv_value_matches(field_value, allowed) {
return false;
}
}
for range in &query.ranges {
let Some(field_value) = fields.get(&range.field) else {
return false;
};
let Ok(n) = field_value.parse::<i64>() else {
return false;
};
if let Some(lo) = range.lo {
if n < lo {
return false;
}
}
if let Some(hi) = range.hi {
if n > hi {
return false;
}
}
}
for negation in &query.negations {
let Some(name) = negation.field.as_deref() else {
// Bare-word negation — not enforced here.
continue;
};
let Some(field_value) = fields.get(name) else {
// Missing field ⇒ negation passes (nothing to exclude).
continue;
};
if any_csv_value_matches(field_value, std::slice::from_ref(&negation.value)) {
return false;
}
}
true
}
/// True iff a plugin advertising the given `served_file_types` /
/// `served_content_kinds` could possibly produce a record satisfying
/// `query.filters["fileType" | "contentKind"]`.
pub fn query_accepts_plugin(
query: &GatewayQuery,
served_file_types: &[&str],
served_content_kinds: &[&str],
) -> bool {
accepts_axis(query.filters.get("fileType"), served_file_types)
&& accepts_axis(query.filters.get("contentKind"), served_content_kinds)
}
fn accepts_axis(requested: Option<&Vec<String>>, served: &[&str]) -> bool {
let Some(requested) = requested else {
return true;
};
if requested.is_empty() {
return true;
}
// Wildcard: a plugin that advertises `"*"` on an axis serves every value on
// that axis (e.g. tribler is a free-text overlay search with no inherent
// type limitation). It accepts any requested fileType/contentKind.
if served.contains(&"*") {
return true;
}
requested
.iter()
.any(|v| served.iter().any(|s| s.eq_ignore_ascii_case(v)))
}
/// Resolve a `languages` filter against the key-set storage form
/// (`languages/<lang3> = "true"`). **Fails open** on anything we can't
/// confidently exclude. A record matches when:
/// 1. it carries a requested code as a truthy key-set member (OR across codes);
/// 2. a legacy flat `languages` csv field contains a requested code;
/// 3. it carries `mul` (multiple — may contain a requested one) or `und`
/// (undetermined); or
/// 4. it carries no language information at all.
///
/// Only a record with a *concrete*, non-matching language (e.g. `jpn`-only
/// under an `eng` filter) is dropped. Kept byte-identical in logic with
/// meta-search's copy (and the client's `passesLangFilter`) — language narrows
/// the result set without ever hiding unknown/undetermined/multi releases.
fn languages_filter_matches(fields: &BTreeMap<String, String>, allowed: &[String]) -> bool {
for code in allowed {
let key = format!("languages/{}", code.trim().to_ascii_lowercase());
if fields.get(&key).is_some_and(|v| is_truthy_member(v)) {
return true;
}
}
if let Some(flat) = fields.get("languages") {
if any_csv_value_matches(flat, allowed) {
return true;
}
}
// Fail open: `mul`/`und` are explicit "could be anything / unknown" markers,
// and a record with no concrete language at all is equally unknown. Keep all
// three; only a specific, non-matching language reaches the `false` below.
if fields.get("languages/mul").is_some_and(|v| is_truthy_member(v))
|| fields.get("languages/und").is_some_and(|v| is_truthy_member(v))
{
return true;
}
!record_has_concrete_language(fields)
}
/// True iff the record states a *concrete* language: a truthy `languages/<code>`
/// key-set member other than the `und`/`mul` non-codes, or a non-empty legacy
/// flat `languages` field. Distinguishes "a specific language that didn't match
/// ⇒ drop" from "no language info ⇒ keep" for [`languages_filter_matches`].
fn record_has_concrete_language(fields: &BTreeMap<String, String>) -> bool {
if fields.get("languages").is_some_and(|v| !v.trim().is_empty()) {
return true;
}
fields.iter().any(|(k, v)| {
is_truthy_member(v)
&& k.starts_with("languages/")
&& k != "languages/und"
&& k != "languages/mul"
})
}
/// Key-set members are written as the literal `"true"`; tolerate a couple of
/// obvious truthy spellings and treat anything else as absent.
/// Fold a genre label to its comparison form: lowercase, alphanumerics only.
///
/// Genre display names carry spaces and ampersands (`"Action & Adventure"`,
/// `"Sci-Fi & Fantasy"`, `"Science Fiction"`), and **space is the query DSL's
/// token separator** — so a filter can never spell a genre the way the record
/// stores it. Callers pass a slug (`genres:action-adventure`); folding both
/// sides makes them one value without forcing the stored name to be
/// machine-shaped.
fn genre_fold(s: &str) -> String {
s.chars()
.filter(|c| c.is_alphanumeric())
.flat_map(|c| c.to_lowercase())
.collect()
}
/// True iff the record carries one of the requested genres, in **either**
/// encoding: a truthy `genres/<Name>` key-set member (the card tier's shape) or
/// the legacy flat comma-joined `genres` field. OR across the requested genres,
/// both sides folded — see [`genre_fold`].
///
/// **Fails closed, unlike [`languages_filter_matches`].** A language filter is a
/// viewer *preference* narrowing a set, so an unknown-language release is kept.
/// A genre filter *defines what a row is*: a record with no genre information is
/// not known to be Action, so putting it in an Action row states something
/// untrue. Upstream has already filtered by genre, so this is a re-validation.
fn genres_filter_matches(fields: &BTreeMap<String, String>, allowed: &[String]) -> bool {
let wanted: Vec<String> = allowed.iter().map(|g| genre_fold(g)).collect();
if wanted.iter().all(|g| g.is_empty()) {
return true; // a filter with no usable value constrains nothing
}
for (key, value) in fields {
let Some(name) = key.strip_prefix("genres/") else {
continue;
};
if is_truthy_member(value) && wanted.contains(&genre_fold(name)) {
return true;
}
}
if let Some(flat) = fields.get("genres") {
for member in flat.split(|c| c == ',' || c == '|') {
if wanted.contains(&genre_fold(member)) {
return true;
}
}
}
false
}
fn is_truthy_member(v: &str) -> bool {
matches!(v.trim().to_ascii_lowercase().as_str(), "true" | "1" | "yes")
}
/// True iff `field_value` (treated as a csv set when it contains commas)
/// contains at least one case-insensitive match for any item in `allowed`.
fn any_csv_value_matches(field_value: &str, allowed: &[String]) -> bool {
for piece in field_value.split(',') {
let piece = piece.trim();
if piece.is_empty() {
continue;
}
if allowed.iter().any(|v| v.eq_ignore_ascii_case(piece)) {
return true;
}
}
false
}
#[cfg(test)]
mod tests {
use super::*;
use crate::query::{Negation, RangeFilter};
fn empty_query() -> GatewayQuery {
GatewayQuery {
raw_text: String::new(),
free_text: String::new(),
filters: BTreeMap::new(),
ranges: Vec::new(),
negations: Vec::new(),
}
}
fn with_filter(field: &str, values: &[&str]) -> GatewayQuery {
let mut q = empty_query();
q.filters.insert(
field.to_string(),
values.iter().map(|s| s.to_string()).collect(),
);
q
}
fn fields(pairs: &[(&str, &str)]) -> BTreeMap<String, String> {
pairs
.iter()
.map(|(k, v)| (k.to_string(), v.to_string()))
.collect()
}
#[test]
fn filter_rejects_mismatched_filetype() {
let q = with_filter("fileType", &["video"]);
assert!(!record_matches(&fields(&[("fileType", "image")]), &q));
}
#[test]
fn filter_accepts_matching_filetype() {
let q = with_filter("fileType", &["video"]);
assert!(record_matches(&fields(&[("fileType", "video")]), &q));
}
#[test]
fn filter_rejects_record_missing_field() {
let q = with_filter("fileType", &["video"]);
assert!(!record_matches(&fields(&[]), &q));
}
#[test]
fn filter_is_case_insensitive() {
let q = with_filter("fileType", &["VIDEO"]);
assert!(record_matches(&fields(&[("fileType", "video")]), &q));
let q = with_filter("fileType", &["video"]);
assert!(record_matches(&fields(&[("fileType", "VIDEO")]), &q));
}
#[test]
fn filter_ors_within_key() {
let q = with_filter("fileType", &["video", "audio"]);
assert!(record_matches(&fields(&[("fileType", "audio")]), &q));
assert!(record_matches(&fields(&[("fileType", "video")]), &q));
assert!(!record_matches(&fields(&[("fileType", "image")]), &q));
}
#[test]
fn filter_ands_across_keys() {
let mut q = with_filter("fileType", &["video"]);
q.filters.insert("genres".into(), vec!["Action".into()]);
assert!(!record_matches(
&fields(&[("fileType", "video"), ("genres", "Drama")]),
&q
));
assert!(record_matches(
&fields(&[("fileType", "video"), ("genres", "Action")]),
&q
));
}
#[test]
fn csv_field_matches_one_of_its_values() {
let q = with_filter("genres", &["Anime"]);
assert!(record_matches(
&fields(&[("genres", "Animation,Anime,Drama")]),
&q
));
}
#[test]
fn csv_field_with_whitespace_is_trimmed() {
let q = with_filter("genres", &["Anime"]);
assert!(record_matches(
&fields(&[("genres", "Animation, Anime , Drama")]),
&q
));
}
#[test]
fn languages_keyset_member_matches() {
let q = with_filter("languages", &["eng"]);
assert!(record_matches(&fields(&[("languages/eng", "true")]), &q));
}
#[test]
fn languages_filter_is_or_across_codes() {
let q = with_filter("languages", &["eng", "fre"]);
assert!(record_matches(&fields(&[("languages/eng", "true")]), &q));
assert!(record_matches(&fields(&[("languages/fre", "true")]), &q));
assert!(!record_matches(&fields(&[("languages/jpn", "true")]), &q));
}
#[test]
fn languages_filter_fails_open_on_unknown() {
let q = with_filter("languages", &["eng"]);
// No language info at all ⇒ kept (unknown, not excluded).
assert!(record_matches(&fields(&[("fileType", "video")]), &q));
// `mul` (multi) may contain the requested language ⇒ kept.
assert!(record_matches(&fields(&[("languages/mul", "true")]), &q));
// `und` (undetermined) ⇒ kept.
assert!(record_matches(&fields(&[("languages/und", "true")]), &q));
// A specific language alongside `mul` is still kept (multi).
assert!(record_matches(
&fields(&[("languages/jpn", "true"), ("languages/mul", "true")]),
&q
));
}
#[test]
fn languages_concrete_mismatch_still_drops() {
let q = with_filter("languages", &["eng"]);
// A concrete, non-matching language ⇒ dropped (language narrows).
assert!(!record_matches(&fields(&[("languages/jpn", "true")]), &q));
// A tombstoned/false member is treated as absent ⇒ unknown ⇒ kept.
assert!(record_matches(&fields(&[("languages/eng", "false")]), &q));
}
#[test]
fn languages_legacy_flat_csv_fallback() {
let q = with_filter("languages", &["eng"]);
assert!(record_matches(&fields(&[("languages", "jpn,eng")]), &q));
assert!(!record_matches(&fields(&[("languages", "jpn,ger")]), &q));
}
#[test]
fn range_in_bounds() {
let mut q = empty_query();
q.ranges.push(RangeFilter {
field: "movieYear".into(),
lo: Some(2010),
hi: Some(2020),
});
assert!(record_matches(&fields(&[("movieYear", "2015")]), &q));
}
#[test]
fn range_out_of_bounds_rejected() {
let mut q = empty_query();
q.ranges.push(RangeFilter {
field: "movieYear".into(),
lo: Some(2010),
hi: Some(2020),
});
assert!(!record_matches(&fields(&[("movieYear", "2025")]), &q));
assert!(!record_matches(&fields(&[("movieYear", "2005")]), &q));
}
#[test]
fn range_open_upper_bound() {
let mut q = empty_query();
q.ranges.push(RangeFilter {
field: "movieYear".into(),
lo: Some(2010),
hi: None,
});
assert!(record_matches(&fields(&[("movieYear", "2025")]), &q));
}
#[test]
fn range_unparseable_value_rejected() {
let mut q = empty_query();
q.ranges.push(RangeFilter {
field: "movieYear".into(),
lo: Some(2010),
hi: None,
});
assert!(!record_matches(&fields(&[("movieYear", "circa 2015")]), &q));
}
#[test]
fn typed_negation_excludes_match() {
let mut q = empty_query();
q.negations.push(Negation {
field: Some("fileType".into()),
value: "image".into(),
});
assert!(!record_matches(&fields(&[("fileType", "image")]), &q));
assert!(record_matches(&fields(&[("fileType", "video")]), &q));
}
#[test]
fn typed_negation_passes_when_field_missing() {
let mut q = empty_query();
q.negations.push(Negation {
field: Some("fileType".into()),
value: "image".into(),
});
assert!(record_matches(&fields(&[]), &q));
}
#[test]
fn bare_word_negation_is_not_enforced() {
let mut q = empty_query();
q.negations.push(Negation {
field: None,
value: "alice".into(),
});
assert!(record_matches(
&fields(&[("title", "alice in wonderland")]),
&q
));
}
#[test]
fn empty_query_matches_anything() {
assert!(record_matches(&fields(&[]), &empty_query()));
assert!(record_matches(
&fields(&[("fileType", "video"), ("title", "x")]),
&empty_query()
));
}
#[test]
fn accepts_returns_true_when_no_type_filter() {
let q = empty_query();
assert!(query_accepts_plugin(&q, &["video"], &["movie"]));
assert!(query_accepts_plugin(&q, &[], &[]));
}
#[test]
fn accepts_rejects_disjoint_filetype() {
let q = with_filter("fileType", &["video"]);
assert!(!query_accepts_plugin(&q, &["image"], &["gif"]));
assert!(query_accepts_plugin(&q, &["video"], &[]));
}
#[test]
fn accepts_rejects_disjoint_contentkind() {
let q = with_filter("contentKind", &["movie"]);
assert!(!query_accepts_plugin(
&q,
&["video", "image"],
&["paper", "book"]
));
assert!(query_accepts_plugin(&q, &["video"], &["movie", "episode"]));
}
#[test]
fn accepts_requires_both_axes_when_both_filtered() {
let mut q = with_filter("fileType", &["video"]);
q.filters.insert("contentKind".into(), vec!["movie".into()]);
assert!(query_accepts_plugin(&q, &["video"], &["movie"]));
assert!(!query_accepts_plugin(&q, &["video"], &["paper"]));
assert!(!query_accepts_plugin(&q, &["image"], &["movie"]));
}
#[test]
fn accepts_wildcard_matches_any_requested_type() {
// A plugin advertising "*" (e.g. tribler) accepts any fileType /
// contentKind filter on the wildcarded axis.
let q = with_filter("fileType", &["archive"]);
assert!(query_accepts_plugin(&q, &["*"], &[]));
let q = with_filter("contentKind", &["audiobook"]);
assert!(query_accepts_plugin(&q, &["*"], &["*"]));
// Wildcard on one axis only: the non-wildcard axis still gates.
let mut q = with_filter("fileType", &["audio"]);
q.filters.insert("contentKind".into(), vec!["podcast".into()]);
assert!(query_accepts_plugin(&q, &["*"], &["*"]));
assert!(!query_accepts_plugin(&q, &["*"], &["movie"]));
}
}
crates/meta-feeder-sdk/src/serve.rs
-762
View File
@@ -1,762 +0,0 @@
//! `serve_feeders` — the axum harness that exposes a set of [`FeederPlugin`]s
//! over the **feeder HTTP contract**. Every feeder sidecar binary is just
//! `serve_feeders(vec![MyPlugin::new()], state_dir, listen)`.
//!
//! # Contract (JSON over HTTP, mirrors meta-sort's container-plugin shape)
//!
//! - `GET /manifest` → [`ManifestResponse`]
//! - `GET /health` → [`HealthResponse`]
//! - `POST /query` → [`QueryRequest`] → [`QueryResponse`]
//! - `POST /compute` → [`ComputeRequest`] → [`ComputeResponse`]
//! - `GET /fetch/{upstream_id}/{record_id}` → streamed bytes (or 404)
//! - `GET /blob/{upstream_id}/{cid}` → bytes (or 404)
//!
//! ## Bytes & the v1 simplification
//!
//! `/compute` returns each outcome's bytes **inline, base64-encoded**
//! ([`OutcomeDto::bytes_b64`]) — this reuses every existing `compute_outcomes`
//! impl verbatim (they already return bytes in `HashOutcome.bytes`) and keeps
//! the gateway core's auto-store a single round-trip. base64 adds ~33%; fine
//! for books/papers/posters. Large-file streaming (torznab multi-GiB) is a
//! documented follow-up — those outcomes are metadata-only anyway
//! (`bytes: None`), so they never ride this path.
use std::collections::HashMap;
use std::net::SocketAddr;
use std::path::{Path as FsPath, PathBuf};
use std::sync::Arc;
use axum::{
body::Body,
extract::{Path, State},
http::StatusCode,
response::{Html, IntoResponse, Response},
routing::{get, post},
Json, Router,
};
use base64::Engine as _;
use futures::StreamExt;
use serde::{Deserialize, Serialize};
use serde_json::Value;
use tracing::{info, warn};
use crate::config::{self, CONFIG_PAGE_HTML};
use crate::plugin::{ConfigError, FeederPlugin, HashKind, HashOutcome};
use crate::query::{GatewayQuery, GatewaySearchEvent, GatewayWireError};
use crate::types::{DiscoveryRecord, GatewayError, PluginHealth};
// -------- wire DTOs --------------------------------------------------------
/// Serializable mirror of [`HashKind`] for the feeder HTTP boundary.
#[derive(Serialize, Deserialize, Clone, Copy, PartialEq, Eq, Debug)]
#[serde(rename_all = "snake_case")]
pub enum HashKindDto {
Midhash256,
Sha2_256,
BtV1File,
NzbRelease,
CardLocator,
NzbPosting,
YtVideo,
}
impl From<HashKind> for HashKindDto {
fn from(k: HashKind) -> Self {
match k {
HashKind::Midhash256 => HashKindDto::Midhash256,
HashKind::Sha2_256 => HashKindDto::Sha2_256,
HashKind::BtV1File => HashKindDto::BtV1File,
HashKind::NzbRelease => HashKindDto::NzbRelease,
HashKind::CardLocator => HashKindDto::CardLocator,
HashKind::NzbPosting => HashKindDto::NzbPosting,
HashKind::YtVideo => HashKindDto::YtVideo,
}
}
}
impl From<HashKindDto> for HashKind {
fn from(k: HashKindDto) -> Self {
match k {
HashKindDto::Midhash256 => HashKind::Midhash256,
HashKindDto::Sha2_256 => HashKind::Sha2_256,
HashKindDto::BtV1File => HashKind::BtV1File,
HashKindDto::NzbRelease => HashKind::NzbRelease,
HashKindDto::CardLocator => HashKind::CardLocator,
HashKindDto::NzbPosting => HashKind::NzbPosting,
HashKindDto::YtVideo => HashKind::YtVideo,
}
}
}
/// One plugin's capability surface in [`ManifestResponse`].
#[derive(Serialize, Deserialize, Clone, Debug)]
pub struct PluginManifest {
pub id: String,
pub served_file_types: Vec<String>,
pub served_content_kinds: Vec<String>,
}
/// `GET /manifest` — the feeder's hosted plugins + their served types. The
/// gateway core reads this to register `RemoteFeederPlugin`s and to build the
/// capability heartbeat it broadcasts to meta-share.
#[derive(Serialize, Deserialize, Clone, Debug)]
pub struct ManifestResponse {
pub feeder_version: String,
pub plugins: Vec<PluginManifest>,
}
/// One plugin's liveness in [`HealthResponse`].
#[derive(Serialize, Deserialize, Clone, Debug)]
pub struct PluginHealthEntry {
pub id: String,
pub health: PluginHealth,
}
/// `GET /health` — `status` is `"ok"` iff every hosted plugin reports `Ok`.
#[derive(Serialize, Deserialize, Clone, Debug)]
pub struct HealthResponse {
pub status: String,
pub plugins: Vec<PluginHealthEntry>,
}
/// `POST /query` request body.
#[derive(Serialize, Deserialize, Clone, Debug)]
pub struct QueryRequest {
pub upstream_id: String,
pub query: GatewayQuery,
pub max_results: u32,
}
/// `POST /query` response body.
#[derive(Serialize, Deserialize, Clone, Debug)]
pub struct QueryResponse {
pub records: Vec<DiscoveryRecord>,
}
/// `POST /compute` request body.
#[derive(Serialize, Deserialize, Clone, Debug)]
pub struct ComputeRequest {
pub upstream_id: String,
pub record_id: String,
}
/// One resolved outcome on the feeder wire. `bytes_b64` carries the full
/// payload base64-encoded when the plugin fetched bytes (`Sha2_256` full
/// store); `None` for metadata-only outcomes (the core's `(None, Some)`
/// branch — e.g. torznab midhash).
#[derive(Serialize, Deserialize, Clone, Debug)]
pub struct OutcomeDto {
pub hash: String,
pub hash_kind: HashKindDto,
pub file_extension: Option<String>,
pub record: Option<DiscoveryRecord>,
pub bytes_b64: Option<String>,
}
impl OutcomeDto {
fn from_outcome(o: HashOutcome) -> Self {
OutcomeDto {
hash: o.hash.0,
hash_kind: o.hash_kind.into(),
file_extension: o.file_extension,
record: o.record,
bytes_b64: o
.bytes
.map(|b| base64::engine::general_purpose::STANDARD.encode(&b)),
}
}
}
/// `POST /compute` response body.
#[derive(Serialize, Deserialize, Clone, Debug)]
pub struct ComputeResponse {
pub outcomes: Vec<OutcomeDto>,
}
// -------- app state + error mapping ----------------------------------------
#[derive(Clone)]
struct AppState {
plugins: Arc<HashMap<String, Arc<dyn FeederPlugin>>>,
version: String,
/// Feeder state dir. Per-plugin config persists at
/// `state_dir/gateway/<id>/config.json` — the same per-plugin dir handed to
/// `configure()`, so a UI save is read back on the next restart.
state_dir: PathBuf,
}
impl AppState {
fn plugin(&self, upstream_id: &str) -> Result<&Arc<dyn FeederPlugin>, Response> {
self.plugins.get(upstream_id).ok_or_else(|| {
(
StatusCode::NOT_FOUND,
format!("no feeder plugin for upstream_id '{upstream_id}'"),
)
.into_response()
})
}
/// `state_dir/gateway/<id>/config.json` — canonical per-plugin config file.
fn config_path(&self, upstream_id: &str) -> PathBuf {
self.state_dir
.join("gateway")
.join(upstream_id)
.join("config.json")
}
/// The effective config as JSON: the persisted `config.json` if it exists,
/// else the plugin's in-memory (env-seeded) config. Used as the redaction
/// source for reads and the merge base for writes.
fn effective_config(&self, upstream_id: &str) -> Value {
if let Ok(bytes) = std::fs::read(self.config_path(upstream_id)) {
if let Ok(v) = serde_json::from_slice::<Value>(&bytes) {
return v;
}
}
self.plugins
.get(upstream_id)
.map(|p| p.config_values())
.unwrap_or_else(|| serde_json::json!({}))
}
}
/// Map an in-process [`GatewayError`] to an HTTP status + message. Mirrors the
/// status semantics the gateway's `RemoteFeederPlugin` will translate back
/// into a `GatewayError` on the core side.
fn gateway_error_response(e: GatewayError) -> Response {
let (status, msg) = match e {
GatewayError::NotFound => (StatusCode::NOT_FOUND, "not found".to_string()),
GatewayError::RateLimited { retry_after_s } => (
StatusCode::TOO_MANY_REQUESTS,
format!("rate limited; retry in {retry_after_s}s"),
),
GatewayError::Transient(s) => (StatusCode::BAD_GATEWAY, s),
GatewayError::Permanent(s) => (StatusCode::UNPROCESSABLE_ENTITY, s),
GatewayError::Internal(s) => (StatusCode::INTERNAL_SERVER_ERROR, s.to_string()),
};
(status, msg).into_response()
}
// -------- handlers ---------------------------------------------------------
async fn manifest(State(state): State<AppState>) -> Json<ManifestResponse> {
let mut plugins: Vec<PluginManifest> = state
.plugins
.values()
.map(|p| PluginManifest {
id: p.upstream_id().to_string(),
served_file_types: p.served_file_types().iter().map(|s| s.to_string()).collect(),
served_content_kinds: p
.served_content_kinds()
.iter()
.map(|s| s.to_string())
.collect(),
})
.collect();
plugins.sort_by(|a, b| a.id.cmp(&b.id));
Json(ManifestResponse {
feeder_version: state.version.clone(),
plugins,
})
}
async fn health(State(state): State<AppState>) -> Json<HealthResponse> {
let mut entries: Vec<PluginHealthEntry> = state
.plugins
.values()
.map(|p| PluginHealthEntry {
id: p.upstream_id().to_string(),
health: p.health(),
})
.collect();
entries.sort_by(|a, b| a.id.cmp(&b.id));
let all_ok = entries
.iter()
.all(|e| matches!(e.health, PluginHealth::Ok));
Json(HealthResponse {
status: if all_ok { "ok" } else { "degraded" }.to_string(),
plugins: entries,
})
}
async fn query(
State(state): State<AppState>,
Json(req): Json<QueryRequest>,
) -> Result<Json<QueryResponse>, Response> {
let plugin = state.plugin(&req.upstream_id)?;
let records = plugin
.handle_query(&req.query, req.max_results as usize)
.await
.map_err(gateway_error_response)?;
Ok(Json(QueryResponse { records }))
}
/// `POST /query_stream` — the streaming analog of [`query`]. Emits the plugin's
/// [`FeederPlugin::handle_query_stream`] events as **NDJSON**: one JSON-encoded
/// [`GatewaySearchEvent`] per line (`application/x-ndjson`). The gateway core's
/// `RemoteFeederPlugin` consumes this and re-emits a `BoxStream<GatewaySearchEvent>`
/// into the libp2p streaming worker, preserving incremental enrichment (e.g.
/// torznab's late TMDB / poster / subtitle-availability patches). A plugin that
/// doesn't override `handle_query_stream` still works here — the trait default
/// replays `handle_query` as `Base* + Done`.
async fn query_stream(
State(state): State<AppState>,
Json(req): Json<QueryRequest>,
) -> Result<Response, Response> {
let plugin = state.plugin(&req.upstream_id)?;
// The open can fail (e.g. the plugin can't reach its upstream at all); that
// maps to an HTTP error status, same as `/query`. Per-event errors instead
// ride the stream as a terminal `GatewaySearchEvent::Error` line.
let events = plugin
.handle_query_stream(&req.query, req.max_results as usize)
.await
.map_err(gateway_error_response)?;
let body = events.map(|ev| {
// Serialization of GatewaySearchEvent is infallible in practice; on the
// off chance it fails, terminate with an Error line so the consumer sees
// a clean terminal failure rather than a truncated stream.
let mut line = serde_json::to_vec(&ev).unwrap_or_else(|e| {
serde_json::to_vec(&GatewaySearchEvent::Error(GatewayWireError::Internal(
format!("event serialize: {e}"),
)))
.unwrap_or_default()
});
line.push(b'\n');
Ok::<_, std::convert::Infallible>(line)
});
Ok((
[(axum::http::header::CONTENT_TYPE, "application/x-ndjson")],
Body::from_stream(body),
)
.into_response())
}
async fn compute(
State(state): State<AppState>,
Json(req): Json<ComputeRequest>,
) -> Result<Json<ComputeResponse>, Response> {
let plugin = state.plugin(&req.upstream_id)?;
let outcomes = plugin
.compute_outcomes(&req.record_id)
.await
.map_err(gateway_error_response)?;
Ok(Json(ComputeResponse {
outcomes: outcomes.into_iter().map(OutcomeDto::from_outcome).collect(),
}))
}
async fn fetch(
State(state): State<AppState>,
Path((upstream_id, record_id)): Path<(String, String)>,
) -> Result<Response, Response> {
let plugin = state.plugin(&upstream_id)?;
match plugin
.handle_fetch(&record_id)
.await
.map_err(gateway_error_response)?
{
Some(stream) => {
// Map the plugin's GatewayError stream into an io::Error stream so
// axum's Body::from_stream accepts it.
let body_stream = stream.map(|chunk| {
chunk.map_err(|e| std::io::Error::other(e.to_string()))
});
Ok(Body::from_stream(body_stream).into_response())
}
None => Err((StatusCode::NOT_FOUND, "no bytes for record").into_response()),
}
}
async fn blob(
State(state): State<AppState>,
Path((upstream_id, cid)): Path<(String, String)>,
) -> Result<Response, Response> {
let plugin = state.plugin(&upstream_id)?;
match plugin.get_blob(&cid).await {
Some(bytes) => Ok(bytes.into_response()),
None => Err((StatusCode::NOT_FOUND, "no such blob").into_response()),
}
}
// -------- config plane -----------------------------------------------------
//
// Generic, schema-driven per-plugin config. The page + the schema/values JSON
// are all served relative to the request path, so this works identically when
// hit directly on the feeder or reverse-proxied through the gateway.
/// `GET /config` — the schema-driven HTML form (static, plugin-agnostic).
async fn config_page() -> Html<&'static str> {
Html(CONFIG_PAGE_HTML)
}
/// `GET /config/schema` — the single hosted plugin's declared config schema.
/// (A feeder hosts one plugin in practice; if it hosts several, the first by
/// id is used — the gateway proxies per upstream_id anyway.)
async fn config_schema(State(state): State<AppState>) -> Json<config::ConfigSchema> {
let schema = sole_plugin(&state)
.map(|p| p.config_schema())
.unwrap_or_default();
Json(schema)
}
/// `GET /config/values` — current values with secrets redacted to `<key>_set`.
async fn config_values_get(State(state): State<AppState>) -> Json<Value> {
match sole_plugin(&state) {
Some(p) => {
let schema = p.config_schema();
let eff = state.effective_config(p.upstream_id());
Json(config::redact(&eff, &schema))
}
None => Json(serde_json::json!({})),
}
}
/// `PUT /config/values` — merge the submitted values onto the stored config
/// (schema-aware; blank secrets keep their stored value), persist atomically,
/// and signal that a feeder restart is needed to apply them.
async fn config_values_put(
State(state): State<AppState>,
Json(incoming): Json<Value>,
) -> Result<Json<Value>, Response> {
let plugin = sole_plugin(&state)
.ok_or_else(|| (StatusCode::NOT_FOUND, "feeder hosts no plugin").into_response())?;
let id = plugin.upstream_id();
let schema = plugin.config_schema();
let base = state.effective_config(id);
let merged = config::merge(&base, &incoming, &schema);
let path = state.config_path(id);
if let Some(dir) = path.parent() {
std::fs::create_dir_all(dir)
.map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, format!("mkdir: {e}")).into_response())?;
}
let tmp = path.with_extension("json.tmp");
let body = serde_json::to_vec_pretty(&merged)
.map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, format!("serialize: {e}")).into_response())?;
std::fs::write(&tmp, body)
.and_then(|_| std::fs::rename(&tmp, &path))
.map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, format!("write config: {e}")).into_response())?;
info!(target: "meta-feeder", upstream_id = id, path = %path.display(), "feeder config saved; scheduling self-restart to apply");
// Apply the new config by restarting the process. We can't hot-swap config
// in-place: plugins hold an exclusive redb cache handle for their whole life,
// so a fresh instance can't reopen the cache while the old one is alive. A
// clean process exit is the reliable apply path — Docker's restart policy
// (`unless-stopped`) brings the feeder back in ~2-3s and `configure()`
// re-reads config.json at startup. Delay briefly so THIS response flushes to
// the client before we exit.
tokio::spawn(async {
tokio::time::sleep(std::time::Duration::from_millis(800)).await;
info!(target: "meta-feeder", "restarting now to apply saved config");
std::process::exit(0);
});
Ok(Json(serde_json::json!({ "saved": true, "reloading": true })))
}
/// `POST /enrich/callback` — sink for the enrichment plugins' completion
/// callbacks. The feeder drives those plugins (filename-parser, tmdb) during a
/// self-published resolve and reads results back by polling meta-core, so the
/// callback carries no load — this route exists only so the plugins' callback
/// POST gets a 200 instead of logging a connection failure. Body is ignored.
async fn enrich_callback() -> StatusCode {
StatusCode::OK
}
/// A feeder hosts one plugin in the common case. Pick it (lowest id for
/// determinism if several are present).
fn sole_plugin(state: &AppState) -> Option<&Arc<dyn FeederPlugin>> {
state
.plugins
.values()
.min_by(|a, b| a.upstream_id().cmp(b.upstream_id()))
}
// -------- entry point ------------------------------------------------------
/// Build the feeder router for an already-configured plugin set. Exposed for
/// in-process tests (drive it without binding a socket).
pub fn router(
plugins: HashMap<String, Arc<dyn FeederPlugin>>,
version: String,
state_dir: impl Into<PathBuf>,
) -> Router {
let state = AppState {
plugins: Arc::new(plugins),
version,
state_dir: state_dir.into(),
};
Router::new()
.route("/manifest", get(manifest))
.route("/health", get(health))
.route("/query", post(query))
.route("/query_stream", post(query_stream))
.route("/compute", post(compute))
.route("/fetch/:upstream_id/:record_id", get(fetch))
.route("/blob/:upstream_id/:cid", get(blob))
.route("/config", get(config_page))
.route("/config/schema", get(config_schema))
.route("/config/values", get(config_values_get).put(config_values_put))
.route("/enrich/callback", post(enrich_callback))
.with_state(state)
}
/// Configure each plugin (cache dir under `state_dir/gateway/<id>`), then serve
/// the feeder HTTP contract on `listen` until shutdown.
///
/// The feeder process never aborts on a config problem: a plugin whose
/// `configure()` returns [`ConfigError::MissingConfig`] is **soft-skipped**
/// (warn + dropped — an optional source the operator didn't ask for), and any
/// other failure (bad/empty key, cache-dir setup, …) registers a
/// [`DegradedPlugin`] that stays up and reports the reason via `/health` and
/// `/manifest`. The container keeps running so the error is visible in the UX
/// instead of crash-looping.
pub async fn serve_feeders(
plugins: Vec<Box<dyn FeederPlugin>>,
state_dir: impl Into<PathBuf>,
listen: SocketAddr,
) -> anyhow::Result<()> {
let state_dir = state_dir.into();
let configured = configure_plugins(plugins, &state_dir)?;
let version = env!("CARGO_PKG_VERSION").to_string();
let app = router(configured, version, state_dir.clone());
let listener = tokio::net::TcpListener::bind(listen).await?;
info!(
target: "meta-feeder",
%listen,
"feeder listening"
);
axum::serve(listener, app)
.await
.map_err(anyhow::Error::from)
}
/// A plugin whose `configure()` (or cache-dir setup) failed. It stays
/// **registered** so the failure is visible at `GET /health` (as
/// [`PluginHealth::Degraded`]) and `GET /manifest`; any query/compute against it
/// returns the captured reason as a permanent error. This is what keeps one
/// misconfigured feeder from crash-looping the whole container — the operator
/// sees the reason in the UX and fixes the config, no restart-loop required.
struct DegradedPlugin {
id: &'static str,
reason: String,
file_types: &'static [&'static str],
content_kinds: &'static [&'static str],
/// Preserved from the wrapped plugin so a degraded/unconfigured feeder still
/// serves its config form — otherwise the dashboard shows "no configuration
/// required" and the operator can never fix the very config that's missing.
schema: crate::config::ConfigSchema,
}
#[async_trait::async_trait]
impl FeederPlugin for DegradedPlugin {
fn upstream_id(&self) -> &'static str {
self.id
}
fn configure(&mut self, _cache_dir: &FsPath) -> Result<(), ConfigError> {
Ok(())
}
async fn handle_query(
&self,
_query: &GatewayQuery,
_max_results: usize,
) -> Result<Vec<DiscoveryRecord>, GatewayError> {
Err(GatewayError::Permanent(self.reason.clone()))
}
async fn compute_outcomes(
&self,
_record_id: &str,
) -> Result<Vec<HashOutcome>, GatewayError> {
Err(GatewayError::Permanent(self.reason.clone()))
}
fn health(&self) -> PluginHealth {
PluginHealth::Degraded {
reason: self.reason.clone(),
}
}
fn served_file_types(&self) -> &'static [&'static str] {
self.file_types
}
fn served_content_kinds(&self) -> &'static [&'static str] {
self.content_kinds
}
fn config_schema(&self) -> crate::config::ConfigSchema {
self.schema.clone()
}
}
/// Run each plugin's `configure()` against its per-plugin cache dir and return
/// the plugins keyed by `upstream_id`. Never errors on a per-plugin problem:
/// `MissingConfig` is soft-skipped, every other failure becomes a
/// [`DegradedPlugin`] (see [`serve_feeders`]). Public so feeder binaries /
/// tests can configure without serving.
pub fn configure_plugins(
plugins: Vec<Box<dyn FeederPlugin>>,
state_dir: &FsPath,
) -> anyhow::Result<HashMap<String, Arc<dyn FeederPlugin>>> {
let mut out: HashMap<String, Arc<dyn FeederPlugin>> = HashMap::new();
for mut plugin in plugins {
let id = plugin.upstream_id();
// Captured up front (both are `'static`, so this borrow of `plugin`
// ends immediately) — lets us build a DegradedPlugin without holding a
// borrow across `configure(&mut self)` / the move into `out`.
let file_types = plugin.served_file_types();
let content_kinds = plugin.served_content_kinds();
// Preserve the config surface so a degraded stand-in still serves the
// real config form (see DegradedPlugin.schema).
let schema = plugin.config_schema();
let cache_dir = state_dir.join("gateway").join(id);
// Build (don't insert) a degraded stand-in: keeps the container up and
// surfaces the reason via /health instead of crash-looping.
let make_degraded = |reason: String| -> Arc<dyn FeederPlugin> {
warn!(
target: "meta-feeder",
upstream_id = id,
%reason,
"feeder plugin degraded: container stays up; error surfaced via /health"
);
Arc::new(DegradedPlugin {
id,
reason,
file_types,
content_kinds,
schema: schema.clone(),
})
};
if let Err(e) = std::fs::create_dir_all(&cache_dir) {
let reason = format!("cache dir {} setup failed: {e}", cache_dir.display());
out.insert(id.to_string(), make_degraded(reason));
continue;
}
match plugin.configure(&cache_dir) {
Ok(()) => {
out.insert(id.to_string(), Arc::from(plugin));
}
Err(ConfigError::MissingConfig { plugin: p, what }) => {
// Keep a degraded stand-in (don't drop it) so its config form
// stays reachable from the dashboard and the operator can supply
// the missing config — then restart the feeder to apply it.
warn!(
target: "meta-feeder",
upstream_id = %p,
missing = %what,
"feeder plugin degraded: required config not supplied; \
config form still served so it can be fixed from the dashboard"
);
out.insert(
id.to_string(),
make_degraded(format!("required config not supplied: {what}")),
);
}
Err(other) => {
out.insert(id.to_string(), make_degraded(other.to_string()));
}
}
}
Ok(out)
}
#[cfg(test)]
mod tests {
use super::*;
use crate::types::DiscoveryRecord;
use async_trait::async_trait;
use futures::stream::BoxStream;
use std::collections::BTreeMap;
/// A feeder plugin that overrides `handle_query_stream` to emit a Base, then
/// an EnrichPatch, then Done — so the NDJSON route is exercised on a real
/// incremental stream (not just the collect-default).
struct StubStreaming;
#[async_trait]
impl FeederPlugin for StubStreaming {
fn upstream_id(&self) -> &'static str {
"stub"
}
fn configure(&mut self, _cache_dir: &FsPath) -> Result<(), ConfigError> {
Ok(())
}
async fn handle_query(
&self,
_q: &GatewayQuery,
_n: usize,
) -> Result<Vec<DiscoveryRecord>, GatewayError> {
Ok(vec![])
}
async fn handle_query_stream(
&self,
_q: &GatewayQuery,
_n: usize,
) -> Result<BoxStream<'static, GatewaySearchEvent>, GatewayError> {
let rec = DiscoveryRecord {
upstream_id: "stub".into(),
record_id: "1".into(),
fields: BTreeMap::new(),
};
let events = vec![
GatewaySearchEvent::Base(rec),
GatewaySearchEvent::EnrichPatch {
record_id: "1".into(),
set: BTreeMap::from([("title".to_string(), "X".to_string())]),
remove: vec![],
},
GatewaySearchEvent::Done,
];
Ok(Box::pin(futures::stream::iter(events)))
}
async fn compute_outcomes(
&self,
_record_id: &str,
) -> Result<Vec<HashOutcome>, GatewayError> {
Ok(vec![])
}
}
#[tokio::test]
async fn query_stream_emits_ndjson_events_in_order() {
let mut plugins: HashMap<String, Arc<dyn FeederPlugin>> = HashMap::new();
plugins.insert("stub".to_string(), Arc::new(StubStreaming));
let app = router(plugins, "test".to_string(), std::env::temp_dir());
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
tokio::spawn(async move {
axum::serve(listener, app).await.unwrap();
});
let body = reqwest::Client::new()
.post(format!("http://{addr}/query_stream"))
.json(&serde_json::json!({
"upstream_id": "stub",
"query": {"raw_text":"x","free_text":"x","filters":{},"ranges":[],"negations":[]},
"max_results": 5
}))
.send()
.await
.unwrap()
.text()
.await
.unwrap();
let events: Vec<GatewaySearchEvent> = body
.lines()
.filter(|l| !l.trim().is_empty())
.map(|l| serde_json::from_str(l).expect("each NDJSON line is a GatewaySearchEvent"))
.collect();
assert_eq!(events.len(), 3, "body was: {body:?}");
assert!(matches!(events[0], GatewaySearchEvent::Base(_)));
assert!(matches!(events[1], GatewaySearchEvent::EnrichPatch { .. }));
assert!(matches!(events[2], GatewaySearchEvent::Done));
}
}
crates/meta-feeder-sdk/src/types.rs
-124
View File
@@ -1,124 +0,0 @@
//! In-process types used by the gateway runtime, feeders, and plugins.
//!
//! `GatewayError` here is **distinct from** the wire-stable
//! `GatewayWireError` (see [`crate::query`]): this one can carry rich
//! internal context (anyhow chains, plugin specifics); the wire one is narrow
//! on purpose since adding variants is a protocol change. Conversion happens
//! at the protocol boundary.
use std::collections::BTreeMap;
use serde::{Deserialize, Serialize};
/// Content-addressed hash string carried over the wire. Renamed from
/// `Midhash` in the multi-outcome plugin contract migration — the name
/// `Midhash` lied for sha2-256 CIDs (which gateway plugins also emit
/// via `compute_ipfs_cid`). The carrier is generic; the hash family is
/// discriminated by `HashKind` on `HashOutcome`.
///
/// Shape: base32-lower 'b'-prefixed multibase CIDv1. The specific
/// multihash inside (`0x1000` for midhash256, `0x12` for sha2-256) is
/// determined by the producing plugin's hashing path.
#[derive(Clone, Debug, PartialEq, Eq, Hash, Serialize, Deserialize)]
pub struct Hash(pub String);
impl Hash {
pub fn as_str(&self) -> &str {
&self.0
}
}
impl std::fmt::Display for Hash {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.write_str(&self.0)
}
}
/// Pre-hash identifier for a record discovered through a gateway.
#[derive(Clone, Debug, PartialEq, Eq, Hash, Serialize, Deserialize)]
pub struct DiscoveryId {
pub upstream_id: String,
pub record_id: String,
}
/// Metadata-only search result. Distinct from meta-core's persistent record
/// (which is a flat `map[string]string` indexed by midhash); a
/// `DiscoveryRecord` only lives inside the gateway search/response pipeline.
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
pub struct DiscoveryRecord {
pub upstream_id: String,
pub record_id: String,
pub fields: BTreeMap<String, String>,
}
/// In-process error returned by `FeederPlugin` methods.
#[derive(Debug, thiserror::Error)]
pub enum GatewayError {
#[error("upstream transient error: {0}")]
Transient(String),
#[error("upstream permanent error: {0}")]
Permanent(String),
#[error("rate limited; retry in {retry_after_s}s")]
RateLimited { retry_after_s: u32 },
#[error("record not found upstream")]
NotFound,
#[error("internal: {0}")]
Internal(#[from] anyhow::Error),
}
/// Byte stream returned by a plugin's `handle_fetch`.
pub type ByteStream = futures::stream::BoxStream<'static, Result<bytes::Bytes, GatewayError>>;
/// Plugin liveness snapshot surfaced via `/health`. Must not call upstream
/// — `health()` is invoked on every status poll.
#[derive(Clone, Debug, Serialize, Deserialize)]
#[serde(rename_all = "snake_case", tag = "state")]
pub enum PluginHealth {
Ok,
Degraded { reason: String },
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn hash_display_and_as_str_match() {
let m = Hash("bafy123".to_string());
assert_eq!(m.as_str(), "bafy123");
assert_eq!(format!("{m}"), "bafy123");
}
#[test]
fn discovery_record_roundtrips_through_serde_json() {
let r = DiscoveryRecord {
upstream_id: "scihub".to_string(),
record_id: "10.1038/s41586-021-03819-2".to_string(),
fields: BTreeMap::from_iter([
("title".to_string(), "AlphaFold".to_string()),
("year".to_string(), "2021".to_string()),
]),
};
let j = serde_json::to_string(&r).expect("serialize");
let back: DiscoveryRecord = serde_json::from_str(&j).expect("deserialize");
assert_eq!(r, back);
}
#[test]
fn plugin_health_serialises_with_tag() {
let ok = serde_json::to_value(PluginHealth::Ok).unwrap();
assert_eq!(ok, serde_json::json!({"state": "ok"}));
let bad = serde_json::to_value(PluginHealth::Degraded {
reason: "upstream 5xx".into(),
})
.unwrap();
assert_eq!(
bad,
serde_json::json!({"state": "degraded", "reason": "upstream 5xx"})
);
}
}
feeder-plugin/usenet-feeder/Cargo.toml
+8 -3
View File
@@ -14,8 +14,10 @@ publish = false
# redeem it without an indexer credential. # redeem it without an indexer credential.
# #
# It ADDS a Usenet metadata source alongside meta-feeder-indexer; it does not # It ADDS a Usenet metadata source alongside meta-feeder-indexer; it does not
# replace it. Like every feeder it is meta-core-free + blockstore-free (the # replace it. It also REDEEMS external `nzb-release` (0x1005) locators through
# gateway core owns hashing/seeding/store-back — see the gateway's invariant 10). # `/compute`, grabbing the .nzb with per-host indexer keys set on its config
# page. Like every feeder it is meta-core-free + blockstore-free (the gateway
# core owns hashing/seeding/store-back — see the gateway's invariant 10).
# #
# See `meta-gateway/docs/others/self-hosted-usenet-indexer-study.md`. # See `meta-gateway/docs/others/self-hosted-usenet-indexer-study.md`.
@@ -28,7 +30,9 @@ name = "usenet-feeder"
path = "src/main.rs" path = "src/main.rs"
[dependencies] [dependencies]
meta-feeder-sdk = { path = "../../crates/meta-feeder-sdk" } # Pinned by TAG, never by path. A path dep resolves OUTSIDE this repo, and this
# repo is the Docker build context. (Until 1.3.0 this repo vendored a copy.)
meta-feeder-sdk = { git = "https://github.com/worph/meta-feeder-sdk", tag = "v1.3.0" }
tokio = { version = "1.40", features = ["full"] } tokio = { version = "1.40", features = ["full"] }
axum = { version = "0.7", features = ["json"] } axum = { version = "0.7", features = ["json"] }
@@ -62,4 +66,5 @@ quick-xml = { version = "0.36", features = ["serialize"] }
[dev-dependencies] [dev-dependencies]
wiremock = "0.6" wiremock = "0.6"
tempfile = "3" tempfile = "3"
base64 = "0.22"
tokio = { version = "1.40", features = ["full", "test-util"] } tokio = { version = "1.40", features = ["full", "test-util"] }
feeder-plugin/usenet-feeder/Dockerfile
+4 -4
View File
@@ -3,9 +3,10 @@
# usenet-feeder — Usenet feeder sidecar over an nntmux header-scan catalog. # usenet-feeder — Usenet feeder sidecar over an nntmux header-scan catalog.
# Multi-stage build. # Multi-stage build.
# #
# Build context must be this repo root (packages/plugins/meta-feeder-usenet) so # Build context must be this repo root (packages/plugins/meta-feeder-usenet).
# the `meta-feeder-sdk` path dependency resolves. Same BuildKit cache-mount # `meta-feeder-sdk` is a git dependency pinned by tag, so the builder needs
# strategy as the other feeders. # network for `cargo fetch`. Same BuildKit cache-mount strategy as the other
# feeders.
# #
# This feeder talks to MariaDB (sqlx, rustls — no system MySQL client needed) # This feeder talks to MariaDB (sqlx, rustls — no system MySQL client needed)
# and reads nntmux's NZB store off a shared volume. It holds no media bytes. # and reads nntmux's NZB store off a shared volume. It holds no media bytes.
@@ -22,7 +23,6 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
&& rm -rf /var/lib/apt/lists/* && rm -rf /var/lib/apt/lists/*
COPY Cargo.toml rust-toolchain.toml ./ COPY Cargo.toml rust-toolchain.toml ./
COPY crates/ crates/
COPY feeder-plugin/ feeder-plugin/ COPY feeder-plugin/ feeder-plugin/
RUN --mount=type=cache,id=cargo-registry,target=/usr/local/cargo/registry \ RUN --mount=type=cache,id=cargo-registry,target=/usr/local/cargo/registry \
feeder-plugin/usenet-feeder/src/lib.rs
+8
View File
@@ -9,6 +9,11 @@
//! //!
//! It ADDS a source alongside meta-feeder-indexer; it replaces nothing. //! It ADDS a source alongside meta-feeder-indexer; it replaces nothing.
//! //!
//! It also **redeems** external Newznab releases: an `nzb-release` (`0x1005`)
//! cid minted by meta-feeder-torznab is grabbed here, with this feeder's
//! per-host indexer keys, when the gateway calls `/compute` on a real play
//! ([`newznab`]). Indexer keys live on this feeder, not on meta-share.
//!
//! Like every feeder it is meta-core-free and blockstore-free: it finds records //! Like every feeder it is meta-core-free and blockstore-free: it finds records
//! and serves bytes, and the gateway core owns hashing-into-the-blockstore, the //! and serves bytes, and the gateway core owns hashing-into-the-blockstore, the
//! meta-core store-back, and the libp2p wire (gateway invariant 10). //! meta-core store-back, and the libp2p wire (gateway invariant 10).
@@ -22,5 +27,8 @@ pub mod nntmux {
pub mod nzb; pub mod nzb;
} }
/// The `t=get` grab that redeems an external `nzb-release` locator.
pub mod newznab;
/// The `usenet` upstream itself. /// The `usenet` upstream itself.
pub mod usenet; pub mod usenet;
feeder-plugin/usenet-feeder/src/main.rs
+6 -4
View File
@@ -3,10 +3,12 @@
//! //!
//! One plugin (`usenet`) served through [`meta_feeder_sdk::serve_feeders`]. //! One plugin (`usenet`) served through [`meta_feeder_sdk::serve_feeders`].
//! //!
//! Operator config — the nntmux database URL and NZB store path — is read from //! Operator config — the nntmux database URL and NZB store path, and the
//! the persisted `config.json` (dashboard-written), with the env vars below as a //! per-host Newznab indexer keys used to redeem `nzb-release` locators — is read
//! **first-boot seed only**: the file wins on the next restart, and there is no //! from the persisted `config.json` (dashboard-written), with the env vars below
//! hot reload (gateway invariant 12). //! as a **first-boot seed only**: the file wins on the next restart, and there
//! is no hot reload (gateway invariant 12). Indexer keys have no env seed; set
//! them on the config page.
//! //!
//! Env (seed): `NNTMUX_DB_URL`, `NNTMUX_NZB_PATH`. //! Env (seed): `NNTMUX_DB_URL`, `NNTMUX_NZB_PATH`.
//! Env (infra): `META_FEEDER_HTTP_LISTEN` (default `0.0.0.0:8080`), //! Env (infra): `META_FEEDER_HTTP_LISTEN` (default `0.0.0.0:8080`),
feeder-plugin/usenet-feeder/src/newznab.rs
+409
View File
@@ -0,0 +1,409 @@
//! Redeeming an **external** Newznab release — an `nzb-release` (`0x1005`)
//! locator minted by `meta-feeder-torznab` from a Prowlarr/Newznab search hit.
//!
//! The cid embeds the indexer's API base and the release id; redeeming it is a
//! `t=get` grab against that indexer with *this feeder's* key for that host.
//! The grab is **metered** by the indexer (a per-account daily download cap), so
//! it only ever runs from `compute_outcomes`, which the gateway calls on a real
//! play — never at search time, never speculatively.
//!
//! Moved here from meta-share's `nzb/manifest_resolve.rs::grab_newznab`: the
//! indexer keys now live on this feeder, meta-share keeps only the NNTP pool.
use std::borrow::Cow;
use anyhow::{anyhow, Context};
use meta_feeder_sdk::types::GatewayError;
use serde::{Deserialize, Serialize};
/// `retry_after_s` handed back when an indexer reports its request/download cap
/// is spent. Newznab carries no reset time, and the caps are daily.
pub const QUOTA_RETRY_SECS: u32 = 3600;
/// Refuse a `.nzb` bigger than this. A 1329-segment posting is ~200 KB; 16 MiB
/// is a ~100k-segment monster and still parseable. Same bound meta-share uses.
pub const MAX_NZB_BYTES: usize = 16 * 1024 * 1024;
/// One configured indexer key. `host` is the bare authority the key is for
/// (`api.nzbgeek.info`); the API path comes from the cid, not from here.
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
pub struct IndexerCred {
#[serde(default)]
pub host: String,
#[serde(default)]
pub api_key: String,
}
/// Normalise an indexer host for matching: drop any `http(s)://` scheme, trim a
/// trailing slash and surrounding whitespace, lowercase. Both the configured
/// host and the cid-decoded authority go through this before an equality
/// compare, so an operator may enter `api.example.com` or `https://api.example.com/`.
pub fn normalize_indexer_host(raw: &str) -> String {
raw.trim()
.trim_start_matches("https://")
.trim_start_matches("http://")
.trim_end_matches('/')
.to_ascii_lowercase()
}
/// The credential-matching key for anything host-shaped an operator or a
/// locator supplies: normalised (scheme, trailing slash, case), then reduced to
/// its authority — `https://API.nzbgeek.info/api/` and `api.nzbgeek.info/api`
/// both become `api.nzbgeek.info`.
pub fn host_key(raw: &str) -> String {
authority(&normalize_indexer_host(raw)).to_string()
}
/// The authority part of a scheme-less API base (`api.nzbgeek.info/api` →
/// `api.nzbgeek.info`). This — never the full base — is the credential key:
/// operators configure plain hosts, and the locator may carry a path.
pub fn authority(api_base: &str) -> &str {
match api_base.split_once('/') {
Some((authority, _path)) => authority,
None => api_base,
}
}
/// A Newznab `<error code=".." description=".."/>` answer to `t=get`.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct NewznabError {
pub code: Option<u32>,
pub description: Option<String>,
}
impl std::fmt::Display for NewznabError {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match (&self.code, &self.description) {
(Some(c), Some(d)) => write!(f, "code {c}: {d}"),
(None, Some(d)) => write!(f, "{d}"),
(Some(c), None) => write!(f, "code {c}"),
(None, None) => write!(f, "unspecified error"),
}
}
}
/// Pull a Newznab `<error>` out of a `t=get` response, if that is what came
/// back instead of an `.nzb`.
///
/// Deliberately a substring scrape rather than a parse: the body is tiny, the
/// shape is fixed across indexers, and the point is to turn a silent
/// zero-file manifest into a readable reason. Only the head is scanned, so an
/// `<error>` deep inside a large legitimate NZB is never mistaken for a refusal.
pub fn newznab_error(xml: &str) -> Option<NewznabError> {
let head = &xml[..floor_char_boundary(xml, 1024)];
let at = head.find("<error ")?;
let rest = &head[at..];
let end = rest.find("/>").or_else(|| rest.find('>'))? + 1;
let tag = &rest[..end];
let grab = |k: &str| -> Option<&str> {
let i = tag.find(k)? + k.len();
let t = &tag[i..];
let q = t.find('"')? + 1;
let e = t[q..].find('"')? + q;
Some(&t[q..e])
};
Some(NewznabError {
code: grab("code=").and_then(|c| c.trim().parse().ok()),
description: grab("description=").map(str::to_string),
})
}
fn floor_char_boundary(s: &str, max: usize) -> usize {
let mut i = s.len().min(max);
while !s.is_char_boundary(i) {
i -= 1;
}
i
}
/// Map an indexer's refusal onto the feeder error contract. The Newznab API
/// spec's codes: `100`–`102` credentials/account, `300` no such item, `429` /
/// `500` request limit, `501` download limit, `910` API disabled.
pub fn classify(err: &NewznabError) -> GatewayError {
match err.code {
Some(300) => GatewayError::NotFound,
Some(429) | Some(500) | Some(501) => GatewayError::RateLimited {
retry_after_s: QUOTA_RETRY_SECS,
},
Some(100..=102) | Some(910) => GatewayError::Permanent(format!(
"indexer refused the .nzb grab ({err}) — check this host's API key on the feeder config page"
)),
_ => GatewayError::Permanent(format!("indexer refused the .nzb grab ({err})")),
}
}
/// True when an indexer answered a grab with an HTML page — a login wall, a
/// Cloudflare challenge, or a permalink host that 302s to its website
/// (`https://nzbgeek.info/api?t=get` does). Without this the grab dies later as
/// an unhelpful "no segments".
pub fn looks_like_html(body: &[u8]) -> bool {
let head = &body[..body.len().min(512)];
let text = String::from_utf8_lossy(head);
let t = text.trim_start_matches('\u{feff}').trim_start().to_ascii_lowercase();
t.starts_with("<!doctype html") || t.starts_with("<html")
}
/// Grab a release's `.nzb` from `{scheme}://{api_base}/api?t=get`.
///
/// Errors never echo the request URL — it carries the API key.
pub async fn grab(
http: &reqwest::Client,
scheme: &str,
api_base: &str,
id: &str,
api_key: &str,
) -> Result<Vec<u8>, GatewayError> {
let base = api_base.trim_end_matches('/');
let url = format!("{scheme}://{base}/api?t=get&id={id}&apikey={api_key}");
let resp = http
.get(&url)
.send()
.await
.map_err(|e| GatewayError::Transient(format!("newznab grab {base}: {}", e.without_url())))?;
let status = resp.status();
if !status.is_success() {
return Err(match status.as_u16() {
404 => GatewayError::NotFound,
429 => GatewayError::RateLimited {
retry_after_s: resp
.headers()
.get(reqwest::header::RETRY_AFTER)
.and_then(|v| v.to_str().ok())
.and_then(|v| v.parse().ok())
.unwrap_or(QUOTA_RETRY_SECS),
},
_ if status.is_server_error() => {
GatewayError::Transient(format!("newznab grab {base}: HTTP {status}"))
}
_ => GatewayError::Permanent(format!("newznab grab {base}: HTTP {status}")),
});
}
let body = resp
.bytes()
.await
.map_err(|e| GatewayError::Transient(format!("newznab grab {base}: {}", e.without_url())))?;
if body.len() > MAX_NZB_BYTES {
return Err(GatewayError::Permanent(format!(
"newznab grab {base}: .nzb is {} bytes, over the {MAX_NZB_BYTES}-byte ceiling",
body.len()
)));
}
if looks_like_html(&body) {
return Err(GatewayError::Permanent(format!(
"indexer {base} answered the .nzb grab with an HTML page — check the API base and key"
)));
}
let text: Cow<str> = String::from_utf8_lossy(&body);
if let Some(err) = newznab_error(&text) {
return Err(classify(&err));
}
Ok(text.into_owned().into_bytes())
}
/// Drop the NZB `<head>` except `<meta type="password">`.
///
/// The grabbed file is seeded to the whole mesh, and indexers put
/// account-identifying bits in `<head>` metadata. The password must survive:
/// meta-share needs it to unpack a protected RAR set (`NzbManifest.password`).
/// `<file>` / `<segment>` content is copied byte-for-byte, so the Message-IDs —
/// and the `nzb-posting` cid minted from them — are unchanged. Output is UTF-8.
pub fn strip_head_keep_password(xml: &[u8]) -> anyhow::Result<Vec<u8>> {
use quick_xml::events::{BytesEnd, BytesStart, Event};
use quick_xml::{Reader, Writer};
let text = String::from_utf8_lossy(xml);
let mut reader = Reader::from_str(&text);
let mut writer = Writer::new(Vec::with_capacity(xml.len()));
// Nesting depth inside <head> (0 = outside), and inside a kept password meta.
let mut head_depth = 0usize;
let mut password_depth = 0usize;
let mut head_start: Option<BytesStart<'static>> = None;
let mut kept: Vec<Event<'static>> = Vec::new();
loop {
let ev = reader.read_event().map_err(|e| anyhow!("nzb xml: {e}"))?;
if matches!(ev, Event::Eof) {
break;
}
if head_depth == 0 {
match &ev {
Event::Start(e) if e.local_name().as_ref() == b"head" => {
head_depth = 1;
head_start = Some(e.clone().into_owned());
kept.clear();
}
Event::Empty(e) if e.local_name().as_ref() == b"head" => {}
_ => writer
.write_event(ev)
.map_err(|e| anyhow!("nzb xml write: {e}"))?,
}
continue;
}
match &ev {
Event::Start(e) => {
if password_depth > 0 {
password_depth += 1;
kept.push(ev.into_owned());
} else if is_password_meta(e) {
password_depth = 1;
kept.push(ev.into_owned());
} else {
head_depth += 1;
}
}
Event::End(e) => {
if password_depth > 0 {
password_depth -= 1;
kept.push(ev.into_owned());
} else if head_depth == 1 && e.local_name().as_ref() == b"head" {
head_depth = 0;
let start = head_start.take().context("head start tag")?;
if !kept.is_empty() {
let name = String::from_utf8_lossy(start.name().as_ref()).into_owned();
writer
.write_event(Event::Start(start))
.map_err(|e| anyhow!("nzb xml write: {e}"))?;
for k in kept.drain(..) {
writer
.write_event(k)
.map_err(|e| anyhow!("nzb xml write: {e}"))?;
}
writer
.write_event(Event::End(BytesEnd::new(name)))
.map_err(|e| anyhow!("nzb xml write: {e}"))?;
}
} else {
head_depth -= 1;
}
}
Event::Empty(e) => {
if password_depth > 0 || is_password_meta(e) {
kept.push(ev.into_owned());
}
}
_ => {
if password_depth > 0 {
kept.push(ev.into_owned());
}
}
}
}
Ok(writer.into_inner())
}
fn is_password_meta(e: &quick_xml::events::BytesStart<'_>) -> bool {
e.local_name().as_ref() == b"meta"
&& e.attributes().flatten().any(|a| {
a.key.local_name().as_ref() == b"type" && a.value.eq_ignore_ascii_case(b"password")
})
}
#[cfg(test)]
mod tests {
use super::*;
/// The exact body nzbgeek returns for an anonymous `t=get` — 200 OK, valid
/// XML, zero files. Reported as "nzb has no files/segments" until this
/// existed, which is a parser-shaped message for a credentials-shaped fault.
#[test]
fn reports_the_indexers_own_reason() {
let xml = r#"<?xml version="1.0" encoding="UTF-8"?>
<error code="109" description="Invalid User Agent"/>"#;
let err = newznab_error(xml).expect("an error");
assert_eq!(err.to_string(), "code 109: Invalid User Agent");
assert!(matches!(classify(&err), GatewayError::Permanent(_)));
}
#[test]
fn a_real_nzb_is_not_an_error() {
let xml = r#"<?xml version="1.0"?>
<nzb xmlns="http://www.newzbin.com/DTD/2003/nzb"><file subject="x"><segments>
<segment number="1" bytes="10">abc@d</segment></segments></file></nzb>"#;
assert_eq!(newznab_error(xml), None);
}
#[test]
fn only_the_head_is_scanned() {
let xml = format!(
"<nzb>{}<error code=\"1\" description=\"late\"/></nzb>",
"x".repeat(2000)
);
assert_eq!(newznab_error(&xml), None);
}
#[test]
fn limit_codes_are_rate_limited_and_300_is_not_found() {
for code in [429, 500, 501] {
let err = NewznabError { code: Some(code), description: None };
assert!(
matches!(classify(&err), GatewayError::RateLimited { retry_after_s } if retry_after_s == QUOTA_RETRY_SECS),
"{code}"
);
}
let gone = NewznabError { code: Some(300), description: Some("No such item".into()) };
assert!(matches!(classify(&gone), GatewayError::NotFound));
let bad_key = NewznabError { code: Some(100), description: Some("Incorrect user credentials".into()) };
assert!(matches!(classify(&bad_key), GatewayError::Permanent(m) if m.contains("API key")));
}
#[test]
fn html_pages_are_detected() {
assert!(looks_like_html(b"<!DOCTYPE html><html><body>login</body></html>"));
assert!(looks_like_html(b"\n <html lang=\"en\">"));
assert!(!looks_like_html(b"<?xml version=\"1.0\"?><nzb/>"));
}
#[test]
fn hosts_normalise_and_authority_drops_the_path() {
assert_eq!(normalize_indexer_host(" https://API.Example.com/ "), "api.example.com");
assert_eq!(authority("api.nzbgeek.info/api"), "api.nzbgeek.info");
assert_eq!(authority("127.0.0.1:8080"), "127.0.0.1:8080");
// Normalise BEFORE splitting, or a scheme's `//` is taken as the path.
assert_eq!(host_key("https://API.nzbgeek.info/api/"), "api.nzbgeek.info");
assert_eq!(host_key("api.nzbgeek.info/api"), "api.nzbgeek.info");
}
const WITH_HEAD: &str = r#"<?xml version="1.0" encoding="UTF-8"?>
<nzb xmlns="http://www.newzbin.com/DTD/2003/nzb">
<head>
<meta type="title">Some.Release.1080p</meta>
<meta type="x-grabbed-by">user-12345</meta>
<meta type="password">s3cr&amp;t</meta>
</head>
<file poster="[email protected]" date="1" subject="a [1/1] &quot;a.mkv&quot; yEnc">
<groups><group>alt.binaries.teevee</group></groups>
<segments><segment bytes="100" number="1">[email protected]</segment></segments>
</file>
</nzb>"#;
#[test]
fn head_strip_keeps_only_the_password() {
let out = String::from_utf8(strip_head_keep_password(WITH_HEAD.as_bytes()).unwrap()).unwrap();
assert!(out.contains(r#"<meta type="password">s3cr&amp;t</meta>"#), "{out}");
assert!(!out.contains("user-12345"), "{out}");
assert!(!out.contains("Some.Release.1080p"), "{out}");
assert!(out.contains("<head>") && out.contains("</head>"), "{out}");
// The file body is untouched.
assert!(out.contains(r#"<segment bytes="100" number="1">[email protected]</segment>"#), "{out}");
assert!(out.contains(r#"subject="a [1/1] &quot;a.mkv&quot; yEnc""#), "{out}");
}
#[test]
fn head_without_password_is_dropped_entirely() {
let xml = WITH_HEAD.replace(r#"<meta type="password">s3cr&amp;t</meta>"#, "");
let out = String::from_utf8(strip_head_keep_password(xml.as_bytes()).unwrap()).unwrap();
assert!(!out.contains("<head"), "{out}");
assert!(out.contains("[email protected]"), "{out}");
}
#[test]
fn no_head_is_a_no_op_on_content() {
let xml = r#"<nzb><file subject="s"><segments><segment number="1">id@x</segment></segments></file></nzb>"#;
let out = strip_head_keep_password(xml.as_bytes()).unwrap();
assert_eq!(String::from_utf8(out).unwrap(), xml);
}
}
feeder-plugin/usenet-feeder/src/nntmux/nzb.rs
+1 -1
View File
@@ -110,7 +110,7 @@ pub fn load(nzb_root: &Path, guid: &str) -> Result<Option<Manifest>> {
/// meta-share would happily play. Surrounding `<>` are stripped if a producer /// meta-share would happily play. Surrounding `<>` are stripped if a producer
/// included them — the mint normalises this too, but keeping the stored form /// included them — the mint normalises this too, but keeping the stored form
/// canonical means the two never disagree about what was hashed. /// canonical means the two never disagree about what was hashed.
fn extract_message_ids(xml: &[u8]) -> Result<Vec<String>> { pub(crate) fn extract_message_ids(xml: &[u8]) -> Result<Vec<String>> {
use quick_xml::events::Event; use quick_xml::events::Event;
use quick_xml::Reader; use quick_xml::Reader;
feeder-plugin/usenet-feeder/src/usenet.rs
+168 -9
View File
@@ -23,25 +23,51 @@
//! sha2-256 IPFS cid, publishes it as a first-class `fileType=nzb` meta-core //! sha2-256 IPFS cid, publishes it as a first-class `fileType=nzb` meta-core
//! record, and rewrites the field to that cid. Identical to how posters are //! record, and rewrites the field to that cid. Identical to how posters are
//! handled. See the study's §5.3. //! handled. See the study's §5.3.
//!
//! # Redeeming external releases
//!
//! `compute_outcomes` also accepts an `nzb-release` (`0x1005`) cid — minted by
//! meta-feeder-torznab from a Newznab search hit — and grabs its `.nzb` with the
//! key configured for that indexer host ([`crate::newznab`]). It returns ONE
//! `Sha2_256` outcome carrying the `.nzb` (head stripped, password kept); the
//! gateway stores it under `/files/plugin/meta-feeder-usenet/`, writes its cid
//! onto the release record as `manifest`, and merges the outcome record's
//! `cids/<nzb-posting>` so the release becomes portable to any NNTP-only peer.
//! Works without nntmux configured.
use std::collections::BTreeMap; use std::collections::BTreeMap;
use std::path::{Path, PathBuf}; use std::path::{Path, PathBuf};
use std::sync::Arc; use std::sync::Arc;
use async_trait::async_trait; use async_trait::async_trait;
use meta_feeder_sdk::common::build_http_client;
use meta_feeder_sdk::config::{ConfigField as F, ConfigSchema}; use meta_feeder_sdk::config::{ConfigField as F, ConfigSchema};
use meta_feeder_sdk::hash::compute_nzb_posting_cid; use meta_feeder_sdk::hash::{
use meta_feeder_sdk::plugin::{ConfigError, FeederPlugin, HashKind, HashOutcome}; codec_of, compute_ipfs_cid, compute_nzb_posting_cid, decode_nzb_release_cid, NZB_RELEASE_CODEC,
};
use meta_feeder_sdk::plugin::{ConfigError, FeederPlugin, HashKind, HashOutcome, RedeemClaim};
use meta_feeder_sdk::query::GatewayQuery; use meta_feeder_sdk::query::GatewayQuery;
use meta_feeder_sdk::types::{DiscoveryRecord, GatewayError, Hash, PluginHealth}; use meta_feeder_sdk::types::{DiscoveryRecord, GatewayError, Hash, PluginHealth};
use tokio::sync::RwLock; use tokio::sync::RwLock;
use tracing::{debug, info, warn}; use tracing::{debug, info, warn};
use crate::newznab::{self, IndexerCred};
use crate::nntmux::db::{content_kind_for_category, file_type_for_category, Db, Release}; use crate::nntmux::db::{content_kind_for_category, file_type_for_category, Db, Release};
use crate::nntmux::nzb; use crate::nntmux::nzb;
pub const UPSTREAM_ID: &str = "usenet"; pub const UPSTREAM_ID: &str = "usenet";
/// `/files/plugin/<PACKAGE>/` — where the gateway stores the `.nzb` files this
/// plugin redeems.
pub const PACKAGE: &str = "meta-feeder-usenet";
/// Indexers answer an anonymous `t=get` with `<error code="109" description=
/// "Invalid User Agent"/>` (nzbgeek does), so always send a named client.
const USER_AGENT: &str = concat!("meta-feeder-usenet/", env!("CARGO_PKG_VERSION"));
/// Wall-clock budget for one `.nzb` grab. The viewer is waiting on it.
const GRAB_TIMEOUT_SECS: u64 = 60;
/// Operator-supplied settings, persisted to `<cache_dir>/config.json` by the /// Operator-supplied settings, persisted to `<cache_dir>/config.json` by the
/// SDK's config surface. Env is a **seed only** — the file wins on the next /// SDK's config surface. Env is a **seed only** — the file wins on the next
/// restart (invariant 12; there is no hot reload). /// restart (invariant 12; there is no hot reload).
@@ -52,6 +78,9 @@ pub struct Settings {
/// Root of nntmux's NZB store, shared with the sidecar as a volume. The /// Root of nntmux's NZB store, shared with the sidecar as a volume. The
/// directory nntmux's `PATH_TO_NZBS` points at. /// directory nntmux's `PATH_TO_NZBS` points at.
pub nzb_root: String, pub nzb_root: String,
/// Per-host Newznab keys for redeeming `nzb-release` locators. Config page
/// only — no env seed.
pub indexers: Vec<IndexerCred>,
} }
impl Settings { impl Settings {
@@ -60,6 +89,7 @@ impl Settings {
db_url: std::env::var("NNTMUX_DB_URL").unwrap_or_default(), db_url: std::env::var("NNTMUX_DB_URL").unwrap_or_default(),
nzb_root: std::env::var("NNTMUX_NZB_PATH") nzb_root: std::env::var("NNTMUX_NZB_PATH")
.unwrap_or_else(|_| "/nntmux-nzb".to_string()), .unwrap_or_else(|_| "/nntmux-nzb".to_string()),
indexers: Vec::new(),
} }
} }
@@ -82,6 +112,13 @@ impl Settings {
self.nzb_root = s.to_string(); self.nzb_root = s.to_string();
} }
} }
if let Some(rows) = v.get("indexers").and_then(|x| x.as_array()) {
self.indexers = rows
.iter()
.filter_map(|row| serde_json::from_value::<IndexerCred>(row.clone()).ok())
.filter(|c| !c.host.trim().is_empty() && !c.api_key.trim().is_empty())
.collect();
}
} }
} }
@@ -92,6 +129,12 @@ pub struct UsenetPlugin {
/// — it soft-skips its upstream instead of failing to boot (invariant 10). /// — it soft-skips its upstream instead of failing to boot (invariant 10).
db: Arc<RwLock<Option<Db>>>, db: Arc<RwLock<Option<Db>>>,
nzb_root: PathBuf, nzb_root: PathBuf,
/// Client for `.nzb` grabs (named User-Agent, follows redirects so an HTML
/// landing page is detected rather than surfacing as a bare 302).
http: reqwest::Client,
/// `https` in production — a locator carries no scheme and every public
/// Newznab API is TLS. Tests point it at a plain-http mock.
grab_scheme: &'static str,
} }
impl Default for UsenetPlugin { impl Default for UsenetPlugin {
@@ -106,9 +149,79 @@ impl UsenetPlugin {
settings: Settings::default(), settings: Settings::default(),
db: Arc::new(RwLock::new(None)), db: Arc::new(RwLock::new(None)),
nzb_root: PathBuf::new(), nzb_root: PathBuf::new(),
http: build_http_client(GRAB_TIMEOUT_SECS, USER_AGENT, None),
grab_scheme: "https",
} }
} }
/// Grab over plain `http` instead of `https`. For tests against a local
/// mock indexer only — a locator carries no scheme, and a real indexer is TLS.
pub fn with_plain_http_grabs(mut self) -> Self {
self.grab_scheme = "http";
self
}
/// The configured key for an indexer authority, if any.
fn indexer_key(&self, authority: &str) -> Option<&str> {
let want = newznab::host_key(authority);
self.settings
.indexers
.iter()
.find(|c| newznab::host_key(&c.host) == want)
.map(|c| c.api_key.as_str())
}
/// Redeem an `nzb-release` locator: grab its `.nzb` with this feeder's key
/// for that host, strip the head (password kept), and mint the portable
/// `nzb-posting` cid from its Message-IDs.
///
/// ⚠ **Spends indexer quota.** Reached only through `/compute`, which the
/// gateway calls on a real play and never for a release it already holds
/// the manifest of. `NotFound` = "not mine" (no key for that host), so the
/// gateway can try another claimer without anything being spent.
async fn redeem_release(&self, cid: &str) -> Result<Vec<HashOutcome>, GatewayError> {
let Some((api_base, id)) = decode_nzb_release_cid(cid) else {
return Err(GatewayError::NotFound);
};
let host = newznab::authority(&api_base);
let Some(key) = self.indexer_key(host) else {
debug!(target: "meta-feeder", %host, "usenet: no indexer key for this host; not ours to redeem");
return Err(GatewayError::NotFound);
};
info!(target: "meta-feeder", %host, %id, "usenet: grabbing .nzb (indexer quota)");
let raw = newznab::grab(&self.http, self.grab_scheme, &api_base, &id, key).await?;
let nzb = newznab::strip_head_keep_password(&raw)
.map_err(|e| GatewayError::Permanent(format!("nzb from {host} is not parseable: {e}")))?;
let message_ids = nzb::extract_message_ids(&nzb)
.map_err(|e| GatewayError::Permanent(format!("nzb from {host} is not parseable: {e}")))?;
if message_ids.is_empty() {
return Err(GatewayError::Permanent(format!(
"nzb from {host} has no segments"
)));
}
let posting = compute_nzb_posting_cid(&message_ids);
let mut fields = BTreeMap::new();
// Merged onto the release record by the gateway: the posting cid
// outranks the locator, so the release becomes redeemable by any peer
// with a plain NNTP provider — no indexer key anywhere after this grab.
fields.insert(format!("cids/{posting}"), "true".to_string());
fields.insert("segmentCount".to_string(), message_ids.len().to_string());
Ok(vec![HashOutcome {
hash: Hash(compute_ipfs_cid(&nzb)),
hash_kind: HashKind::Sha2_256,
bytes: Some(nzb.into()),
record: Some(DiscoveryRecord {
upstream_id: UPSTREAM_ID.to_string(),
record_id: cid.to_string(),
fields,
}),
file_extension: Some("nzb".to_string()),
}])
}
async fn db(&self) -> Result<Db, GatewayError> { async fn db(&self) -> Result<Db, GatewayError> {
self.db self.db
.read() .read()
@@ -272,6 +385,10 @@ impl FeederPlugin for UsenetPlugin {
if term.trim().is_empty() { if term.trim().is_empty() {
return Ok(vec![]); return Ok(vec![]);
} }
if self.settings.db_url.is_empty() {
// Grab-only deployment (indexer keys, no nntmux): nothing to search.
return Ok(vec![]);
}
let db = self.db().await?; let db = self.db().await?;
let releases = db let releases = db
.search(term.trim(), max_results) .search(term.trim(), max_results)
@@ -305,6 +422,11 @@ impl FeederPlugin for UsenetPlugin {
} }
async fn compute_outcomes(&self, record_id: &str) -> Result<Vec<HashOutcome>, GatewayError> { async fn compute_outcomes(&self, record_id: &str) -> Result<Vec<HashOutcome>, GatewayError> {
// An external release locator, not one of our nntmux guids. Checked
// first: it needs no database.
if codec_of(record_id) == Some(NZB_RELEASE_CODEC) {
return self.redeem_release(record_id).await;
}
let db = self.db().await?; let db = self.db().await?;
let release = db let release = db
.by_guid(record_id) .by_guid(record_id)
@@ -357,14 +479,34 @@ impl FeederPlugin for UsenetPlugin {
} }
fn health(&self) -> PluginHealth { fn health(&self) -> PluginHealth {
if self.settings.db_url.is_empty() { if self.settings.db_url.is_empty() && self.settings.indexers.is_empty() {
return PluginHealth::Degraded { return PluginHealth::Degraded {
reason: "nntmux database not configured".into(), reason: "neither an nntmux database nor an indexer key is configured".into(),
}; };
} }
PluginHealth::Ok PluginHealth::Ok
} }
fn package(&self) -> Option<&'static str> {
Some(PACKAGE)
}
/// Claims `nzb-release` for every host with a key; nothing without keys.
fn redeems(&self) -> Vec<RedeemClaim> {
let mut hosts: Vec<String> = self
.settings
.indexers
.iter()
.map(|c| newznab::host_key(&c.host))
.collect();
hosts.sort();
hosts.dedup();
if hosts.is_empty() {
return Vec::new();
}
vec![RedeemClaim::nzb_release(hosts)]
}
fn served_file_types(&self) -> &'static [&'static str] { fn served_file_types(&self) -> &'static [&'static str] {
&["video", "audio", "document", "archive"] &["video", "audio", "document", "archive"]
} }
@@ -380,10 +522,9 @@ impl FeederPlugin for UsenetPlugin {
.with_help( .with_help(
"MariaDB DSN of the nntmux sidecar's catalog, e.g. \ "MariaDB DSN of the nntmux sidecar's catalog, e.g. \
mysql://nntmux:secret@nntmux-db:3306/nntmux. Blank → the \ mysql://nntmux:secret@nntmux-db:3306/nntmux. Blank → the \
upstream soft-skips and this plugin reports Degraded. Takes \ self-scan search is off (indexer-key redeems still work). Takes \
effect on the next feeder restart (no hot reload).", effect on the next feeder restart (no hot reload).",
) ),
.required(),
F::text("nzb_root", "NZB store path") F::text("nzb_root", "NZB store path")
.with_help( .with_help(
"Path to nntmux's NZB directory as mounted INTO this container \ "Path to nntmux's NZB directory as mounted INTO this container \
@@ -394,8 +535,25 @@ impl FeederPlugin for UsenetPlugin {
nntmux nests each file one directory per leading guid character, \ nntmux nests each file one directory per leading guid character, \
`nzbsplitlevel` deep (the image ships 4 → 9/7/a/2/97a2….nzb.gz), \ `nzbsplitlevel` deep (the image ships 4 → 9/7/a/2/97a2….nzb.gz), \
and the depth is detected automatically.", and the depth is detected automatically.",
) ),
.required(), F::record_array(
"indexers",
"Indexer keys (redeem nzb-release)",
vec![
F::text("host", "Host").required().with_help(
"The indexer's bare host, e.g. api.nzbgeek.info — no scheme, no /api. \
Matched against the host inside each nzb-release cid.",
),
F::secret("api_key", "API key").required().with_help(
"The indexer account's API key. Every grab spends that account's \
daily download quota — once per release, on a real play only.",
),
],
)
.with_help(
"Newznab keys used to grab the .nzb of releases minted by the torznab feeder \
(Prowlarr search). Takes effect on the next feeder restart.",
),
], ],
} }
} }
@@ -404,6 +562,7 @@ impl FeederPlugin for UsenetPlugin {
serde_json::json!({ serde_json::json!({
"db_url": self.settings.db_url, "db_url": self.settings.db_url,
"nzb_root": self.settings.nzb_root, "nzb_root": self.settings.nzb_root,
"indexers": self.settings.indexers,
}) })
} }
} }
feeder-plugin/usenet-feeder/tests/contract.rs
+23 -1
View File
@@ -86,11 +86,33 @@ async fn health_degrades_gracefully_with_no_nntmux_configured() {
"usenet plugin health entry should report Degraded, got {usenet_health}" "usenet plugin health entry should report Degraded, got {usenet_health}"
); );
assert_eq!( assert_eq!(
usenet_health["health"]["reason"], "nntmux database not configured", usenet_health["health"]["reason"],
"neither an nntmux database nor an indexer key is configured",
"got {usenet_health}" "got {usenet_health}"
); );
} }
/// Unconfigured: the manifest still names the storage folder, but claims no
/// redeems — there is no key to redeem anything with.
#[tokio::test]
async fn manifest_names_the_package_and_claims_nothing_without_keys() {
let (addr, _dir) = boot_feeder().await;
let manifest: serde_json::Value = reqwest::get(format!("http://{addr}/manifest"))
.await
.expect("manifest req")
.json()
.await
.expect("manifest json");
let usenet = manifest["plugins"]
.as_array()
.expect("plugins array")
.iter()
.find(|p| p["id"] == "usenet")
.expect("usenet entry");
assert_eq!(usenet["package"], "meta-feeder-usenet");
assert_eq!(usenet["redeems"], serde_json::json!([]));
}
/// Every seedable/resolvable field this feeder advertises resolving to must /// Every seedable/resolvable field this feeder advertises resolving to must
/// stay in sync with the gateway core's `SEEDABLE_FIELDS` table /// stay in sync with the gateway core's `SEEDABLE_FIELDS` table
/// (`remote_feeder.rs`) — this is a documentation-level regression guard, not /// (`remote_feeder.rs`) — this is a documentation-level regression guard, not
feeder-plugin/usenet-feeder/tests/redeem.rs
+249
View File
@@ -0,0 +1,249 @@
//! Redeeming an external `nzb-release` (`0x1005`) locator through `POST /compute`.
//!
//! Boots the real `UsenetPlugin` in the SDK router with indexer keys in its
//! `config.json` (the config-page shape) and a wiremock Newznab indexer, then
//! drives `/compute` with a locator cid exactly as the gateway's redeem route
//! does. No nntmux: redeeming must work in a grab-only deployment.
use base64::Engine as _;
use meta_feeder_sdk::hash::{
compute_ipfs_cid, compute_nzb_posting_cid, compute_nzb_release_cid,
};
use meta_feeder_sdk::serve::{configure_plugins, router};
use meta_feeder_sdk::{ComputeRequest, ComputeResponse, HashKindDto, RedeemsResponse};
use usenet_feeder::usenet::UsenetPlugin;
use wiremock::matchers::{header_exists, method, path, query_param};
use wiremock::{Mock, MockServer, ResponseTemplate};
const KEY: &str = "indexer-key-1";
const RELEASE_ID: &str = "0123456789abcdef";
const NZB: &str = r#"<?xml version="1.0" encoding="UTF-8"?>
<nzb xmlns="http://www.newzbin.com/DTD/2003/nzb">
<head>
<meta type="title">Some.Release.1080p</meta>
<meta type="x-grabbed-by">account-4242</meta>
<meta type="password">hunter2</meta>
</head>
<file poster="[email protected]" date="1" subject="Some.Release [1/1] &quot;a.mkv&quot; yEnc">
<groups><group>alt.binaries.teevee</group></groups>
<segments>
<segment bytes="100" number="1">aaa@example.com</segment>
<segment bytes="100" number="2">bbb@example.com</segment>
</segments>
</file>
</nzb>"#;
/// The indexer's scheme-less API base as it appears inside a locator.
fn authority(server: &MockServer) -> String {
server.uri().trim_start_matches("http://").to_string()
}
async fn boot(indexers: serde_json::Value) -> (String, tempfile::TempDir) {
let dir = tempfile::tempdir().expect("tempdir");
let cache = dir.path().join("gateway").join("usenet");
std::fs::create_dir_all(&cache).expect("mkdir");
std::fs::write(
cache.join("config.json"),
serde_json::json!({ "indexers": indexers }).to_string(),
)
.expect("write config");
let plugin = UsenetPlugin::new().with_plain_http_grabs();
let configured = configure_plugins(vec![Box::new(plugin)], dir.path()).expect("configure");
let app = router(configured, "test".to_string(), dir.path());
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.expect("bind");
let addr = listener.local_addr().expect("addr");
tokio::spawn(async move {
axum::serve(listener, app).await.expect("serve");
});
(format!("http://{addr}"), dir)
}
async fn boot_with_key_for(server: &MockServer) -> (String, tempfile::TempDir) {
boot(serde_json::json!([{ "host": authority(server), "api_key": KEY }])).await
}
async fn compute(base: &str, record_id: &str) -> reqwest::Response {
reqwest::Client::new()
.post(format!("{base}/compute"))
.json(&ComputeRequest {
upstream_id: "usenet".into(),
record_id: record_id.into(),
})
.send()
.await
.expect("POST /compute")
}
fn grab_mock() -> wiremock::MockBuilder {
Mock::given(method("GET"))
.and(path("/api"))
.and(query_param("t", "get"))
.and(query_param("id", RELEASE_ID))
.and(query_param("apikey", KEY))
}
#[tokio::test]
async fn a_release_locator_redeems_to_the_nzb_and_its_posting_cid() {
let indexer = MockServer::start().await;
grab_mock()
.and(header_exists("user-agent"))
.respond_with(ResponseTemplate::new(200).set_body_string(NZB))
.expect(1)
.mount(&indexer)
.await;
let (base, _dir) = boot_with_key_for(&indexer).await;
let cid = compute_nzb_release_cid(&authority(&indexer), RELEASE_ID).unwrap();
let resp = compute(&base, &cid).await;
assert_eq!(resp.status(), 200, "{:?}", resp.text().await);
let body: ComputeResponse = resp.json().await.expect("ComputeResponse");
assert_eq!(body.outcomes.len(), 1);
let o = &body.outcomes[0];
assert_eq!(o.hash_kind, HashKindDto::Sha2_256);
assert_eq!(o.file_extension.as_deref(), Some("nzb"));
let bytes = base64::engine::general_purpose::STANDARD
.decode(o.bytes_b64.as_deref().expect("bytes"))
.expect("base64");
assert_eq!(o.hash, compute_ipfs_cid(&bytes), "hash is the cid of the returned bytes");
let text = String::from_utf8(bytes).expect("utf-8");
assert!(text.contains(r#"<meta type="password">hunter2</meta>"#), "password kept: {text}");
assert!(!text.contains("account-4242"), "head metadata stripped: {text}");
assert!(text.contains("[email protected]") && text.contains("[email protected]"));
let record = o.record.as_ref().expect("fields to merge onto the release record");
let posting = compute_nzb_posting_cid(&["[email protected]", "[email protected]"]);
assert_eq!(record.fields.get(&format!("cids/{posting}")).map(String::as_str), Some("true"));
assert_eq!(record.fields.get("segmentCount").map(String::as_str), Some("2"));
assert!(
!record.fields.keys().any(|k| k == &format!("cids/{}", o.hash)),
"the manifest's own cid must never become a cids/ member of the release"
);
}
#[tokio::test]
async fn a_host_without_a_key_is_not_ours_and_spends_nothing() {
let indexer = MockServer::start().await;
Mock::given(method("GET"))
.respond_with(ResponseTemplate::new(200).set_body_string(NZB))
.expect(0)
.mount(&indexer)
.await;
let (base, _dir) =
boot(serde_json::json!([{ "host": "other.example", "api_key": KEY }])).await;
let cid = compute_nzb_release_cid(&authority(&indexer), RELEASE_ID).unwrap();
assert_eq!(compute(&base, &cid).await.status(), 404);
}
#[tokio::test]
async fn invalid_user_agent_error_is_permanent() {
let indexer = MockServer::start().await;
grab_mock()
.respond_with(ResponseTemplate::new(200).set_body_string(
r#"<?xml version="1.0" encoding="UTF-8"?><error code="109" description="Invalid User Agent"/>"#,
))
.mount(&indexer)
.await;
let (base, _dir) = boot_with_key_for(&indexer).await;
let cid = compute_nzb_release_cid(&authority(&indexer), RELEASE_ID).unwrap();
let resp = compute(&base, &cid).await;
assert_eq!(resp.status(), 422);
assert!(resp.text().await.unwrap().contains("Invalid User Agent"));
}
#[tokio::test]
async fn http_429_is_rate_limited() {
let indexer = MockServer::start().await;
grab_mock()
.respond_with(ResponseTemplate::new(429).insert_header("Retry-After", "120"))
.mount(&indexer)
.await;
let (base, _dir) = boot_with_key_for(&indexer).await;
let cid = compute_nzb_release_cid(&authority(&indexer), RELEASE_ID).unwrap();
assert_eq!(compute(&base, &cid).await.status(), 429);
}
#[tokio::test]
async fn download_limit_error_code_is_rate_limited() {
let indexer = MockServer::start().await;
grab_mock()
.respond_with(ResponseTemplate::new(200).set_body_string(
r#"<error code="501" description="Download limit reached"/>"#,
))
.mount(&indexer)
.await;
let (base, _dir) = boot_with_key_for(&indexer).await;
let cid = compute_nzb_release_cid(&authority(&indexer), RELEASE_ID).unwrap();
assert_eq!(compute(&base, &cid).await.status(), 429);
}
#[tokio::test]
async fn an_html_page_is_a_permanent_error() {
let indexer = MockServer::start().await;
grab_mock()
.respond_with(
ResponseTemplate::new(200)
.set_body_string("<!DOCTYPE html><html><body>Please log in</body></html>"),
)
.mount(&indexer)
.await;
let (base, _dir) = boot_with_key_for(&indexer).await;
let cid = compute_nzb_release_cid(&authority(&indexer), RELEASE_ID).unwrap();
let resp = compute(&base, &cid).await;
assert_eq!(resp.status(), 422);
assert!(resp.text().await.unwrap().contains("HTML"));
}
/// Keys configured → the plugin claims `nzb-release` for exactly those hosts, on
/// both `/redeems` and `/manifest`, and reports healthy without nntmux.
#[tokio::test]
async fn keys_are_advertised_as_redeem_claims() {
let (base, _dir) = boot(serde_json::json!([
{ "host": "https://API.nzbgeek.info/", "api_key": "k1" },
{ "host": "api.nzb.life", "api_key": "k2" },
{ "host": "blank-key.example", "api_key": "" }
]))
.await;
let r: RedeemsResponse = reqwest::get(format!("{base}/redeems"))
.await
.unwrap()
.json()
.await
.unwrap();
let claims = serde_json::to_value(&r.redeems["usenet"]).unwrap();
assert_eq!(
claims,
serde_json::json!([{
"codec": "nzb-release",
"field": "manifest",
"hosts": ["api.nzb.life", "api.nzbgeek.info"],
"sources": []
}])
);
let m: serde_json::Value = reqwest::get(format!("{base}/manifest"))
.await
.unwrap()
.json()
.await
.unwrap();
assert_eq!(m["plugins"][0]["package"], "meta-feeder-usenet");
assert_eq!(m["plugins"][0]["redeems"], claims);
let h: serde_json::Value = reqwest::get(format!("{base}/health"))
.await
.unwrap()
.json()
.await
.unwrap();
assert_eq!(h["status"], "ok", "{h}");
}