Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Worph
|
4b956bc259 | ||
|
Worph
|
a7cb27984b | ||
|
Worph
|
48b0e52146 | ||
|
Worph
|
aea258c803 | ||
|
Worph
|
b823654b9a | ||
|
Worph
|
fadfebbf9b | ||
|
Worph
|
1e919d89c7 | ||
|
Worph
|
a8df97ac1c | ||
|
Worph
|
d1b2ebef11 | ||
|
Worph
|
d6a1d8090e | ||
|
Worph
|
3852d3375c | ||
|
Worph
|
2ce28c1cbd | ||
|
Worph
|
f70bb52aab |
@@ -0,0 +1,6 @@
|
||||
# FIX CRLF always when developer has not set
|
||||
# Linux/Mac: git config --global core.autocrlf input
|
||||
# Windows: git config --global core.autocrlf true
|
||||
# Auto detect text files and perform LF normalization
|
||||
* text=auto
|
||||
* eol=lf
|
||||
@@ -3,9 +3,9 @@ name: Publish image
|
||||
# Build and push the feeder image to the GitHub Container Registry (ghcr.io):
|
||||
# * every push to main -> the moving `main` tag
|
||||
# * every `v*` tag -> semver tags (e.g. v1.0.0 -> 1.0.0, 1.0, 1)
|
||||
# The build context is the repo root (the workspace) so the vendored
|
||||
# `meta-feeder-sdk` path dependency resolves; the Dockerfile lives under
|
||||
# feeder-plugin/torrent-feeder/.
|
||||
# The build context is the repo root (the workspace; `meta-feeder-sdk` is a
|
||||
# git dependency pinned by tag); the Dockerfile lives under
|
||||
# feeder-plugin/tribler-feeder/.
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
@@ -51,7 +51,7 @@ jobs:
|
||||
uses: docker/build-push-action@v6
|
||||
with:
|
||||
context: .
|
||||
file: ./feeder-plugin/torrent-feeder/Dockerfile
|
||||
file: ./feeder-plugin/tribler-feeder/Dockerfile
|
||||
push: true
|
||||
tags: ${{ steps.meta.outputs.tags }}
|
||||
labels: ${{ steps.meta.outputs.labels }}
|
||||
|
||||
Generated
+3
-2
@@ -788,7 +788,8 @@ checksum = "88904434abc2901f197fe8cc55f0445e7ded921dba5911dad2e2b39b48e663c4"
|
||||
|
||||
[[package]]
|
||||
name = "meta-feeder-sdk"
|
||||
version = "0.1.0"
|
||||
version = "1.7.0"
|
||||
source = "git+https://github.com/worph/meta-feeder-sdk?tag=v1.7.0#027917c8cffe7ca43dadcf3b426cc5f69ecc443a"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"async-trait",
|
||||
@@ -797,13 +798,13 @@ dependencies = [
|
||||
"bytes",
|
||||
"futures",
|
||||
"hyper",
|
||||
"libc",
|
||||
"redb",
|
||||
"regex",
|
||||
"reqwest",
|
||||
"serde",
|
||||
"serde_json",
|
||||
"sha2",
|
||||
"tempfile",
|
||||
"thiserror 1.0.69",
|
||||
"tokio",
|
||||
"tracing",
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
[workspace]
|
||||
resolver = "2"
|
||||
members = ["crates/*", "feeder-plugin/*"]
|
||||
members = ["feeder-plugin/*"]
|
||||
|
||||
[workspace.package]
|
||||
edition = "2021"
|
||||
|
||||
@@ -1 +1,128 @@
|
||||
# meta-feeder-tribler
|
||||
|
||||
**Tribler feeder sidecar** for
|
||||
[MetaMesh](https://github.com/worph/meta-gateway) — bridges the
|
||||
**decentralized [Tribler](https://www.tribler.org/) network** (its IPv8
|
||||
peer-to-peer overlay search) into a meta-gateway as a search source.
|
||||
|
||||
Unlike a Torznab/Prowlarr feeder that talks to centralized indexer sites,
|
||||
Tribler has **no central index**: it runs its own anonymous overlay and each
|
||||
peer contributes the torrents it has seen. This feeder does not reimplement
|
||||
that overlay — it runs a **headless Tribler core as a sidecar container**
|
||||
(`tribler-instance`) and queries its REST API, exactly the way a Torznab feeder
|
||||
treats an external indexer. It bridges Tribler's *content corpus* into the
|
||||
mesh so those torrents become discoverable and playable via meta-share.
|
||||
|
||||
This feeder was **split out of** [`meta-feeder-torrent`](../meta-feeder-torrent)
|
||||
so that Tribler is a **single-plugin** feeder. A feeder exposes one `/config`
|
||||
surface; a multi-plugin feeder (Torznab + Tribler in one binary) can only route
|
||||
that surface to one plugin, so its dashboard config panel could not resolve to
|
||||
the Tribler schema. Splitting gives Tribler its own clean config card.
|
||||
|
||||
## Metadata-only
|
||||
|
||||
This is the **light** feeder. It resolves a Tribler search hit's infohash
|
||||
directly into a `btih-v1-file` CID and **advertises** the record — it does
|
||||
**not** download or seed the bytes. There is no `librqbit` here: file lists
|
||||
come from the Tribler core's own `torrentinfo` metainfo response, not from a
|
||||
BitTorrent fetch. So the cold Docker build is much lighter than the torrent
|
||||
feeder's, and full-file fetch/seed is left to meta-share's on-demand fetcher
|
||||
(the feeder synthesizes magnets with a set of well-known public trackers so
|
||||
that fetcher can find peers quickly, since Tribler torrents are often DHT-only).
|
||||
|
||||
It carries a **copy** of the source-agnostic "torrent-core" TMDB
|
||||
catalog-discovery + enrichment stack (`tmdb`, `tmdb_budget`, `discovery`,
|
||||
`enrich`, `title`, `consts`, `filename_meta`). That stack cannot live in a
|
||||
container plugin — a container plugin enriches a *known* CID; it cannot answer
|
||||
"what's popular" for the home-row browse — so the browse/catalog primitive is
|
||||
inherently in-process and shared verbatim by every torrent source. Like every
|
||||
feeder it stays **meta-core-free at the transport layer and blockstore-free**:
|
||||
it finds records and enriches them; publishing resolved records + TMDB posters
|
||||
goes to the meta-core configured in the dashboard.
|
||||
|
||||
## Role in MetaMesh
|
||||
|
||||
A feeder is a stateless HTTP sidecar. The gateway registers it as a remote
|
||||
feeder plugin pointing at its `/` and drives the SDK contract
|
||||
([`meta_feeder_sdk::serve_feeders`](https://github.com/worph/meta-feeder-sdk/blob/main/src/serve.rs)):
|
||||
|
||||
| Endpoint | Purpose |
|
||||
|----------|---------|
|
||||
| `GET /manifest` | feeder identity + served types (`*` / `*` — Tribler is free-text, any query can return anything) |
|
||||
| `GET /health` | liveness (`Degraded` until `configure()` runs) |
|
||||
| `POST /query`, `POST /query_stream` | structured / streaming search over the Tribler local + remote (IPv8) overlay |
|
||||
| `POST /compute` | outcome compute + TMDB enrichment |
|
||||
| `GET /fetch/:upstream_id/:record_id`, `GET /blob/:upstream_id/:cid` | SDK byte routes (metadata-only here — no local bytes to serve) |
|
||||
| `POST /enrich/callback` | async enrichment callback sink |
|
||||
| `GET /config`, `GET /config/schema`, `GET\|PUT /config/values` | schema-driven config UI + API |
|
||||
|
||||
## Configuration
|
||||
|
||||
All Tribler config is **dashboard config only** — it lives in the persisted
|
||||
`config.json` (written via the gateway dashboard) and has **no env var**. This
|
||||
is the MetaMesh rule: a field in the web-UI config schema must not also be an
|
||||
env seed (config from JSON / web UI is the single source of truth).
|
||||
|
||||
Configured on the plugin's card in the gateway dashboard:
|
||||
|
||||
| Field | Notes |
|
||||
|-------|-------|
|
||||
| `meta_core_url` | meta-core this feeder publishes resolved records + TMDB posters to (e.g. `http://metacore-app:9000`). Blank → publish soft-skips. |
|
||||
| `sidecar_url` | Tribler core REST base (default `http://tribler-instance:8085`). |
|
||||
| `api_key` | Tribler REST `X-Api-Key` (secret); the dev sidecar uses `changeme`. Blank → no auth header. |
|
||||
| `tmdb_api_key` | TMDB v3 key / v4 bearer (secret) for poster / overview / tmdbid enrichment. Blank → TMDB enrichment soft-skips. |
|
||||
| `tmdb_language` | TMDB metadata language tag (e.g. `en-US`, `fr-FR`, `ja-JP`). Blank → `en-US`. |
|
||||
|
||||
Config changes take effect on the next feeder restart (no hot reload). Open the
|
||||
Tribler web UI via the "↗ Tribler UI" link on the plugin's card — it opens the
|
||||
gated `/ui/` behind single sign-on (put `?key=` **after** the `#`; the UI is a
|
||||
HashRouter).
|
||||
|
||||
Env vars are **infra only**:
|
||||
|
||||
| Env var | Default | Notes |
|
||||
|---------|---------|-------|
|
||||
| `META_FEEDER_HTTP_LISTEN` | `0.0.0.0:8080` | HTTP listen address |
|
||||
| `META_FEEDER_STATE_DIR` | `/data/meta-feeder` | redb midhash cache + state |
|
||||
| `RUST_LOG` | `info` | tracing filter |
|
||||
|
||||
## Image
|
||||
|
||||
```
|
||||
ghcr.io/worph/meta-feeder-tribler
|
||||
```
|
||||
|
||||
Exposes `8080`. Built and pushed by CI on every push to `main` (the moving
|
||||
`main` tag) and on `v*` tags (semver tags). Metadata-only (no `librqbit`), so
|
||||
the cold build is lighter than `meta-feeder-torrent`.
|
||||
|
||||
## Build locally
|
||||
|
||||
The build context is the **repo root** (the Cargo workspace) so the vendored
|
||||
`meta-feeder-sdk` path dependency resolves:
|
||||
|
||||
```bash
|
||||
docker build -f feeder-plugin/tribler-feeder/Dockerfile -t ghcr.io/worph/meta-feeder-tribler:dev .
|
||||
```
|
||||
|
||||
## Repo layout
|
||||
|
||||
This repo is a self-contained Cargo workspace vendored out of the
|
||||
`meta-gateway` monorepo:
|
||||
|
||||
```
|
||||
Cargo.toml # workspace: members = crates/*, feeder-plugin/*
|
||||
# (keeps the core2 [patch.crates-io] safety net)
|
||||
crates/meta-feeder-sdk/ # vendored shared feeder SDK
|
||||
feeder-plugin/tribler-feeder/ # this feeder's crate + Dockerfile
|
||||
src/tribler/ # the Tribler source (IPv8-overlay REST client)
|
||||
src/{tmdb,discovery,enrich,title,filename_meta,...}.rs
|
||||
# copied source-agnostic "torrent-core" catalog stack
|
||||
```
|
||||
|
||||
> The workspace keeps the `core2` `[patch.crates-io]` override (a yanked
|
||||
> transitive dep) as a safety net, matching the `meta-gateway` workspace.
|
||||
|
||||
Upstream source of truth for the SDK and the Tribler source is
|
||||
[`worph/meta-gateway`](https://github.com/worph/meta-gateway); changes there are
|
||||
vendored back into this repo.
|
||||
|
||||
@@ -1,49 +0,0 @@
|
||||
[package]
|
||||
name = "meta-feeder-sdk"
|
||||
version = "0.1.0"
|
||||
edition.workspace = true
|
||||
rust-version.workspace = true
|
||||
license.workspace = true
|
||||
publish = false
|
||||
|
||||
# Source-agnostic foundation shared by the gateway core and every feeder
|
||||
# sidecar binary: the `FeederPlugin` trait, the content-addressing hash
|
||||
# families, the per-plugin redb cache, the shared HTTP/cache helpers, the
|
||||
# structured-query evaluator, and the axum `serve_feeders` harness that wraps
|
||||
# any set of `FeederPlugin`s in the feeder HTTP contract.
|
||||
#
|
||||
# Deliberately libp2p-free / blockstore-free: a feeder finds + fetches bytes;
|
||||
# the gateway core owns hashing-into-the-blockstore, the meta-core store-back,
|
||||
# and the libp2p wire. Keep heavy deps (librqbit, beetswap, cid, libp2p) OUT
|
||||
# of here so feeders that don't need them don't pay for them.
|
||||
|
||||
[dependencies]
|
||||
# Async runtime + HTTP server (the serve_feeders harness)
|
||||
tokio = { version = "1.40", features = ["full"] }
|
||||
axum = { version = "0.7", features = ["json"] }
|
||||
hyper = { version = "1", features = ["full"] }
|
||||
reqwest = { version = "0.12", features = ["json", "rustls-tls", "stream"], default-features = false }
|
||||
|
||||
# Serialization
|
||||
serde = { version = "1", features = ["derive"] }
|
||||
serde_json = "1"
|
||||
base64 = "0.22"
|
||||
|
||||
# Utilities
|
||||
futures = "0.3"
|
||||
async-trait = "0.1"
|
||||
anyhow = "1"
|
||||
thiserror = "1"
|
||||
tracing = "0.1"
|
||||
sha2 = "0.10"
|
||||
bytes = "1"
|
||||
# Title-tag parsing (shared filename_meta — season/episode/quality/codec/languages).
|
||||
regex = "1"
|
||||
|
||||
# Per-plugin midhash/blob cache. Schema is identical to meta-share v1 — see
|
||||
# cache.rs. Pure-Rust single-process embedded store.
|
||||
redb = "2"
|
||||
|
||||
[dev-dependencies]
|
||||
tempfile = "3"
|
||||
tokio = { version = "1.40", features = ["full", "test-util"] }
|
||||
@@ -1,337 +0,0 @@
|
||||
//! Per-plugin midhash cache backed by redb.
|
||||
//!
|
||||
//! Each plugin gets its own redb file at
|
||||
//! `<state_dir>/gateway/<upstream_id>/cache.redb`. Keys are upstream
|
||||
//! `record_id`s (DOI for sci-hub, book id for gutenberg, …); values are the
|
||||
//! `Midhash` strings the plugin computed by fetching + hashing the upstream
|
||||
//! file.
|
||||
//!
|
||||
//! The schema is identical to meta-share v1's so old `.redb` files copy
|
||||
//! across. redb is synchronous and sub-millisecond for small ops, so calls
|
||||
//! run inline from async handlers without `spawn_blocking`.
|
||||
|
||||
use std::path::Path;
|
||||
use std::sync::Arc;
|
||||
|
||||
use redb::{Database, ReadableTableMetadata, TableDefinition};
|
||||
|
||||
pub const CACHE_FILENAME: &str = "cache.redb";
|
||||
|
||||
const MIDHASH_TABLE: TableDefinition<'_, &str, &str> = TableDefinition::new("midhash");
|
||||
const BLOBS_TABLE: TableDefinition<'_, &str, &[u8]> = TableDefinition::new("blobs");
|
||||
const COVER_CID_TABLE: TableDefinition<'_, &str, &str> = TableDefinition::new("cover_cid");
|
||||
const BIBREC_TABLE: TableDefinition<'_, &str, &str> = TableDefinition::new("bibrec");
|
||||
const PREVIEW_CID_TABLE: TableDefinition<'_, &str, &str> = TableDefinition::new("preview_cid");
|
||||
/// Full-file sha2-256 IPFS cid for a previously-fetched torrent payload
|
||||
/// (torznab's bt-fetch path). Distinct from `MIDHASH_TABLE` because the
|
||||
/// hash families differ: `midhash` is the synthetic midhash256-from-infohash
|
||||
/// fallback; `fullhash` records "we actually downloaded the bytes and this is
|
||||
/// the IPFS CIDv1 over them."
|
||||
const FULLHASH_TABLE: TableDefinition<'_, &str, &str> = TableDefinition::new("fullhash");
|
||||
/// Enumerated torrent file list (JSON `Vec<TorrentFile>`) keyed by the
|
||||
/// torrent's `record_id` (infohash).
|
||||
const FILELIST_TABLE: TableDefinition<'_, &str, &str> = TableDefinition::new("filelist");
|
||||
/// Cached TMDB *search* results keyed by a stable lookup key. The value is the
|
||||
/// JSON encoding of the top `TmdbHit`, or the literal `"null"` negative-cache
|
||||
/// sentinel.
|
||||
const TMDB_SEARCH_TABLE: TableDefinition<'_, &str, &str> = TableDefinition::new("tmdb_search");
|
||||
/// Cached TMDB `GET /3/tv/{id}` structural details keyed by `tmdbid`.
|
||||
const TMDB_TVDETAILS_TABLE: TableDefinition<'_, &str, &str> =
|
||||
TableDefinition::new("tmdb_tvdetails");
|
||||
/// Cached TMDB `GET /3/{tv,movie}/{id}/external_ids` keyed by `tmdbid`.
|
||||
const TMDB_EXTIDS_TABLE: TableDefinition<'_, &str, &str> = TableDefinition::new("tmdb_extids");
|
||||
/// Cached TMDB `GET /3/movie/{id}` details keyed by `tmdbid`.
|
||||
const TMDB_MOVIEDETAILS_TABLE: TableDefinition<'_, &str, &str> =
|
||||
TableDefinition::new("tmdb_moviedetails");
|
||||
/// Cached **ranked** TMDB `search/multi` anchors keyed by the normalized
|
||||
/// free-text query.
|
||||
const TMDB_PRINCIPAL_TOPN_TABLE: TableDefinition<'_, &str, &str> =
|
||||
TableDefinition::new("tmdb_principal_topn");
|
||||
/// Subtitle linkage discovered at search-enrich time, keyed by the torrent's
|
||||
/// `record_id` (infohash). Value is the JSON encoding of a `Vec<SubtitleLink>`.
|
||||
const SUBTITLES_TABLE: TableDefinition<'_, &str, &str> = TableDefinition::new("subtitles");
|
||||
/// OpenSubtitles search results keyed by `"<tmdb_id>\x01<lang3>"`. Value is the
|
||||
/// resolved subtitle's cid, or the literal `"null"` negative-cache sentinel.
|
||||
const OPENSUBTITLES_TABLE: TableDefinition<'_, &str, &str> = TableDefinition::new("opensubtitles");
|
||||
|
||||
/// Per-plugin midhash cache. Cheap to clone (the inner `Database` is shared
|
||||
/// via `Arc`).
|
||||
#[derive(Clone)]
|
||||
pub struct MidhashCache {
|
||||
db: Arc<Database>,
|
||||
}
|
||||
|
||||
/// Generate the standard `get`/`put` accessor pair for a `&str → &str` redb
|
||||
/// table.
|
||||
macro_rules! str_table_accessors {
|
||||
($(#[$gmeta:meta])* $get:ident, $(#[$pmeta:meta])* $put:ident, $table:ident $(,)?) => {
|
||||
$(#[$gmeta])*
|
||||
pub fn $get(&self, key: &str) -> Result<Option<String>, redb::Error> {
|
||||
let tx = self.db.begin_read()?;
|
||||
let table = tx.open_table($table)?;
|
||||
Ok(table.get(key)?.map(|v| v.value().to_string()))
|
||||
}
|
||||
|
||||
$(#[$pmeta])*
|
||||
pub fn $put(&self, key: &str, value: &str) -> Result<(), redb::Error> {
|
||||
let tx = self.db.begin_write()?;
|
||||
{
|
||||
let mut table = tx.open_table($table)?;
|
||||
table.insert(key, value)?;
|
||||
}
|
||||
tx.commit()?;
|
||||
Ok(())
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
impl MidhashCache {
|
||||
pub fn open(cache_dir: &Path) -> Result<Self, redb::Error> {
|
||||
let path = cache_dir.join(CACHE_FILENAME);
|
||||
let db = Database::create(&path)?;
|
||||
{
|
||||
let tx = db.begin_write()?;
|
||||
tx.open_table(MIDHASH_TABLE)?;
|
||||
tx.open_table(BLOBS_TABLE)?;
|
||||
tx.open_table(COVER_CID_TABLE)?;
|
||||
tx.open_table(BIBREC_TABLE)?;
|
||||
tx.open_table(PREVIEW_CID_TABLE)?;
|
||||
tx.open_table(FULLHASH_TABLE)?;
|
||||
tx.open_table(FILELIST_TABLE)?;
|
||||
tx.open_table(TMDB_SEARCH_TABLE)?;
|
||||
tx.open_table(TMDB_TVDETAILS_TABLE)?;
|
||||
tx.open_table(TMDB_EXTIDS_TABLE)?;
|
||||
tx.open_table(TMDB_MOVIEDETAILS_TABLE)?;
|
||||
tx.open_table(TMDB_PRINCIPAL_TOPN_TABLE)?;
|
||||
tx.open_table(SUBTITLES_TABLE)?;
|
||||
tx.open_table(OPENSUBTITLES_TABLE)?;
|
||||
tx.commit()?;
|
||||
}
|
||||
Ok(Self { db: Arc::new(db) })
|
||||
}
|
||||
|
||||
str_table_accessors!(get_midhash, put_midhash, MIDHASH_TABLE);
|
||||
|
||||
pub fn entry_count(&self) -> Result<u64, redb::Error> {
|
||||
let tx = self.db.begin_read()?;
|
||||
let table = tx.open_table(MIDHASH_TABLE)?;
|
||||
Ok(table.len()?)
|
||||
}
|
||||
|
||||
pub fn get_blob(&self, cid: &str) -> Result<Option<Vec<u8>>, redb::Error> {
|
||||
let tx = self.db.begin_read()?;
|
||||
let table = tx.open_table(BLOBS_TABLE)?;
|
||||
Ok(table.get(cid)?.map(|v| v.value().to_vec()))
|
||||
}
|
||||
|
||||
pub fn put_blob(&self, cid: &str, bytes: &[u8]) -> Result<(), redb::Error> {
|
||||
let tx = self.db.begin_write()?;
|
||||
{
|
||||
let mut table = tx.open_table(BLOBS_TABLE)?;
|
||||
table.insert(cid, bytes)?;
|
||||
}
|
||||
tx.commit()?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
str_table_accessors!(get_cover_cid, put_cover_cid, COVER_CID_TABLE);
|
||||
|
||||
str_table_accessors!(
|
||||
/// Read a previously-stored full-file IPFS cid for `record_id`.
|
||||
get_fullhash,
|
||||
/// Record the IPFS cid produced by a successful full fetch.
|
||||
put_fullhash,
|
||||
FULLHASH_TABLE
|
||||
);
|
||||
|
||||
str_table_accessors!(
|
||||
/// Read the cached torrent file list (JSON) for `record_id`.
|
||||
get_filelist,
|
||||
/// Cache the enumerated torrent file list (JSON `Vec<TorrentFile>`).
|
||||
put_filelist,
|
||||
FILELIST_TABLE
|
||||
);
|
||||
|
||||
str_table_accessors!(
|
||||
/// Read the cached subtitle linkage (JSON `Vec<SubtitleLink>`).
|
||||
get_subtitles,
|
||||
/// Cache the discovered subtitle linkage (JSON `Vec<SubtitleLink>`).
|
||||
put_subtitles,
|
||||
SUBTITLES_TABLE
|
||||
);
|
||||
|
||||
str_table_accessors!(
|
||||
/// Read a cached OpenSubtitles lookup by `"<tmdb_id>\x01<lang3>"`.
|
||||
get_opensubtitles,
|
||||
/// Cache an OpenSubtitles lookup result (cid or `"null"` sentinel).
|
||||
put_opensubtitles,
|
||||
OPENSUBTITLES_TABLE
|
||||
);
|
||||
|
||||
str_table_accessors!(get_preview_cid, put_preview_cid, PREVIEW_CID_TABLE);
|
||||
|
||||
str_table_accessors!(
|
||||
/// Read a cached TMDB search result by lookup key.
|
||||
get_tmdb_search,
|
||||
/// Cache a TMDB search result under `key`.
|
||||
put_tmdb_search,
|
||||
TMDB_SEARCH_TABLE
|
||||
);
|
||||
|
||||
str_table_accessors!(
|
||||
/// Read cached TMDB TV-details JSON by `tmdbid`.
|
||||
get_tmdb_tvdetails,
|
||||
/// Cache TMDB TV-details JSON under `tmdbid`.
|
||||
put_tmdb_tvdetails,
|
||||
TMDB_TVDETAILS_TABLE
|
||||
);
|
||||
|
||||
str_table_accessors!(
|
||||
/// Read cached TMDB external-ids JSON by `tmdbid`.
|
||||
get_tmdb_extids,
|
||||
/// Cache TMDB external-ids JSON under `tmdbid`.
|
||||
put_tmdb_extids,
|
||||
TMDB_EXTIDS_TABLE
|
||||
);
|
||||
|
||||
str_table_accessors!(
|
||||
/// Read cached TMDB movie-details JSON by `tmdbid`.
|
||||
get_tmdb_moviedetails,
|
||||
/// Cache TMDB movie-details JSON under `tmdbid`.
|
||||
put_tmdb_moviedetails,
|
||||
TMDB_MOVIEDETAILS_TABLE
|
||||
);
|
||||
|
||||
str_table_accessors!(
|
||||
/// Read the cached ranked anchor list by normalized query key.
|
||||
get_tmdb_principal_topn,
|
||||
/// Cache the ranked anchor list under the normalized query key.
|
||||
put_tmdb_principal_topn,
|
||||
TMDB_PRINCIPAL_TOPN_TABLE
|
||||
);
|
||||
|
||||
pub fn get_bibrec(
|
||||
&self,
|
||||
record_id: &str,
|
||||
) -> Result<Option<std::collections::BTreeMap<String, String>>, redb::Error> {
|
||||
let tx = self.db.begin_read()?;
|
||||
let table = tx.open_table(BIBREC_TABLE)?;
|
||||
let Some(v) = table.get(record_id)? else {
|
||||
return Ok(None);
|
||||
};
|
||||
let s = v.value();
|
||||
Ok(serde_json::from_str(s).ok())
|
||||
}
|
||||
|
||||
pub fn put_bibrec(
|
||||
&self,
|
||||
record_id: &str,
|
||||
fields: &std::collections::BTreeMap<String, String>,
|
||||
) -> Result<(), redb::Error> {
|
||||
let json = serde_json::to_string(fields).unwrap_or_else(|_| "{}".to_string());
|
||||
let tx = self.db.begin_write()?;
|
||||
{
|
||||
let mut table = tx.open_table(BIBREC_TABLE)?;
|
||||
table.insert(record_id, json.as_str())?;
|
||||
}
|
||||
tx.commit()?;
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
fn tmp_cache_dir() -> tempfile::TempDir {
|
||||
tempfile::tempdir().expect("tempdir")
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn put_then_get_returns_value() {
|
||||
let dir = tmp_cache_dir();
|
||||
let cache = MidhashCache::open(dir.path()).expect("open");
|
||||
assert_eq!(cache.get_midhash("10.1038/x").unwrap(), None);
|
||||
cache.put_midhash("10.1038/x", "bafyXYZ").expect("put");
|
||||
assert_eq!(
|
||||
cache.get_midhash("10.1038/x").unwrap().as_deref(),
|
||||
Some("bafyXYZ")
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn put_overwrites_existing_value() {
|
||||
let dir = tmp_cache_dir();
|
||||
let cache = MidhashCache::open(dir.path()).expect("open");
|
||||
cache.put_midhash("k", "v1").unwrap();
|
||||
cache.put_midhash("k", "v2").unwrap();
|
||||
assert_eq!(cache.get_midhash("k").unwrap().as_deref(), Some("v2"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn values_persist_across_reopens() {
|
||||
let dir = tmp_cache_dir();
|
||||
{
|
||||
let cache = MidhashCache::open(dir.path()).expect("open 1");
|
||||
cache.put_midhash("k", "persistent").unwrap();
|
||||
}
|
||||
let cache = MidhashCache::open(dir.path()).expect("open 2");
|
||||
assert_eq!(
|
||||
cache.get_midhash("k").unwrap().as_deref(),
|
||||
Some("persistent")
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn tmdb_search_cache_distinguishes_miss_from_absent() {
|
||||
let dir = tmp_cache_dir();
|
||||
let cache = MidhashCache::open(dir.path()).expect("open");
|
||||
assert_eq!(cache.get_tmdb_search("m\u{1}naruto\u{1}").unwrap(), None);
|
||||
cache.put_tmdb_search("m\u{1}naruto\u{1}", "null").unwrap();
|
||||
assert_eq!(
|
||||
cache
|
||||
.get_tmdb_search("m\u{1}naruto\u{1}")
|
||||
.unwrap()
|
||||
.as_deref(),
|
||||
Some("null")
|
||||
);
|
||||
cache
|
||||
.put_tmdb_search("t\u{1}spy x family\u{1}2019", "{\"tmdbid\":120089}")
|
||||
.unwrap();
|
||||
drop(cache);
|
||||
let cache = MidhashCache::open(dir.path()).expect("reopen");
|
||||
assert_eq!(
|
||||
cache
|
||||
.get_tmdb_search("t\u{1}spy x family\u{1}2019")
|
||||
.unwrap()
|
||||
.as_deref(),
|
||||
Some("{\"tmdbid\":120089}")
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn tmdb_tvdetails_cache_roundtrips() {
|
||||
let dir = tmp_cache_dir();
|
||||
let cache = MidhashCache::open(dir.path()).expect("open");
|
||||
assert_eq!(cache.get_tmdb_tvdetails("120089").unwrap(), None);
|
||||
cache
|
||||
.put_tmdb_tvdetails("120089", "{\"number_of_seasons\":2}")
|
||||
.unwrap();
|
||||
assert_eq!(
|
||||
cache.get_tmdb_tvdetails("120089").unwrap().as_deref(),
|
||||
Some("{\"number_of_seasons\":2}")
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn entry_count_reflects_inserts() {
|
||||
let dir = tmp_cache_dir();
|
||||
let cache = MidhashCache::open(dir.path()).expect("open");
|
||||
assert_eq!(cache.entry_count().unwrap(), 0);
|
||||
cache.put_midhash("a", "x").unwrap();
|
||||
cache.put_midhash("b", "y").unwrap();
|
||||
assert_eq!(cache.entry_count().unwrap(), 2);
|
||||
cache.put_midhash("a", "x2").unwrap();
|
||||
assert_eq!(cache.entry_count().unwrap(), 2);
|
||||
}
|
||||
}
|
||||
@@ -1,288 +0,0 @@
|
||||
//! Shared helpers for single-outcome source plugins (`arxiv`, `gutenberg`,
|
||||
//! `pubmed`, `wikicommons`, `giphy`, `scihub`).
|
||||
//!
|
||||
//! These plugins are structurally near-identical: build a reqwest client, map
|
||||
//! HTTP status → [`GatewayError`], and front a per-record midhash cache whose
|
||||
//! hit/put boilerplate is the same everywhere. Only the duplicated scaffolding
|
||||
//! lives here; the genuinely-different bodies stay in the plugin modules.
|
||||
|
||||
use std::path::Path;
|
||||
use std::time::Duration;
|
||||
|
||||
use tracing::{debug, warn};
|
||||
|
||||
use crate::cache::MidhashCache;
|
||||
use crate::plugin::{ConfigError, HashKind, HashOutcome};
|
||||
use crate::types::{DiscoveryRecord, GatewayError, Hash};
|
||||
|
||||
/// Build the shared reqwest client every source plugin uses.
|
||||
pub fn build_http_client(
|
||||
timeout_secs: u64,
|
||||
user_agent: &str,
|
||||
redirect: Option<reqwest::redirect::Policy>,
|
||||
) -> reqwest::Client {
|
||||
let mut builder = reqwest::Client::builder()
|
||||
.timeout(Duration::from_secs(timeout_secs))
|
||||
.user_agent(user_agent);
|
||||
if let Some(policy) = redirect {
|
||||
builder = builder.redirect(policy);
|
||||
}
|
||||
builder
|
||||
.build()
|
||||
.expect("reqwest client build is infallible with rustls-tls")
|
||||
}
|
||||
|
||||
/// Map a non-success HTTP status to a [`GatewayError`].
|
||||
///
|
||||
/// - `404` → [`GatewayError::NotFound`]
|
||||
/// - `429` → [`GatewayError::RateLimited`] (honours `Retry-After`, default 60s)
|
||||
/// - `5xx` → [`GatewayError::Transient`]
|
||||
/// - anything else → [`GatewayError::Permanent`]
|
||||
pub fn map_status(resp: &reqwest::Response) -> Result<(), GatewayError> {
|
||||
let status = resp.status();
|
||||
if status.is_success() {
|
||||
return Ok(());
|
||||
}
|
||||
let code = status.as_u16();
|
||||
if code == 404 {
|
||||
return Err(GatewayError::NotFound);
|
||||
}
|
||||
if code == 429 {
|
||||
let retry_after = resp
|
||||
.headers()
|
||||
.get(reqwest::header::RETRY_AFTER)
|
||||
.and_then(|v| v.to_str().ok())
|
||||
.and_then(|v| v.parse::<u32>().ok())
|
||||
.unwrap_or(60);
|
||||
return Err(GatewayError::RateLimited {
|
||||
retry_after_s: retry_after,
|
||||
});
|
||||
}
|
||||
if status.is_server_error() {
|
||||
return Err(GatewayError::Transient(format!(
|
||||
"upstream {} returned {status}",
|
||||
resp.url()
|
||||
)));
|
||||
}
|
||||
Err(GatewayError::Permanent(format!(
|
||||
"upstream {} returned {status}",
|
||||
resp.url()
|
||||
)))
|
||||
}
|
||||
|
||||
/// Open the per-plugin redb midhash cache, wrapping the error with the plugin
|
||||
/// name for a clean startup diagnostic.
|
||||
pub fn open_midhash_cache(
|
||||
cache_dir: &Path,
|
||||
plugin: &'static str,
|
||||
) -> Result<MidhashCache, ConfigError> {
|
||||
MidhashCache::open(cache_dir).map_err(|e| ConfigError::Other {
|
||||
plugin,
|
||||
source: anyhow::anyhow!("open redb cache: {e}"),
|
||||
})
|
||||
}
|
||||
|
||||
/// Cache-hit fast path for single-outcome plugins.
|
||||
///
|
||||
/// On hit, returns `Some(vec![sparse outcome])` — hash only, no bytes/record —
|
||||
/// so the core's `(_, None)` branch skips re-storing content. `Ok(None)` is a
|
||||
/// cache miss; the caller proceeds to fetch + hash.
|
||||
pub fn cached_outcome(
|
||||
cache: &MidhashCache,
|
||||
record_id: &str,
|
||||
upstream_id: &'static str,
|
||||
) -> Result<Option<Vec<HashOutcome>>, GatewayError> {
|
||||
let Some(cached) = cache
|
||||
.get_midhash(record_id)
|
||||
.map_err(|e| GatewayError::Internal(anyhow::anyhow!("cache get: {e}")))?
|
||||
else {
|
||||
return Ok(None);
|
||||
};
|
||||
debug!(
|
||||
target: "meta-share::gateway",
|
||||
upstream = upstream_id,
|
||||
record_id,
|
||||
"hash cache hit"
|
||||
);
|
||||
Ok(Some(vec![HashOutcome {
|
||||
hash: Hash(cached),
|
||||
hash_kind: HashKind::Sha2_256,
|
||||
bytes: None,
|
||||
record: None,
|
||||
file_extension: None,
|
||||
}]))
|
||||
}
|
||||
|
||||
/// Persist a freshly-computed `(record_id → cid)` mapping. A write failure is
|
||||
/// non-fatal, so it is logged and swallowed.
|
||||
pub fn store_midhash(cache: &MidhashCache, record_id: &str, upstream_id: &'static str, cid: &str) {
|
||||
if let Err(e) = cache.put_midhash(record_id, cid) {
|
||||
warn!(
|
||||
target: "meta-share::gateway",
|
||||
upstream = upstream_id,
|
||||
record_id,
|
||||
error = %e,
|
||||
"hash cache put failed (non-fatal)"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/// Resolve the per-plugin cache handle, or produce the standard "not
|
||||
/// configured" internal error.
|
||||
pub fn require_cache<'a>(
|
||||
cache: Option<&'a MidhashCache>,
|
||||
plugin: &str,
|
||||
) -> Result<&'a MidhashCache, GatewayError> {
|
||||
cache.ok_or_else(|| {
|
||||
GatewayError::Internal(anyhow::anyhow!(
|
||||
"{plugin} plugin not configured (configure() never called)"
|
||||
))
|
||||
})
|
||||
}
|
||||
|
||||
/// Build the single-element outcome vec a single-outcome plugin returns on a
|
||||
/// fresh (cache-miss) compute: the full bytes + record, a Sha2_256 IPFS CID,
|
||||
/// and an optional file extension.
|
||||
pub fn single_outcome(
|
||||
cid: String,
|
||||
bytes: bytes::Bytes,
|
||||
record: DiscoveryRecord,
|
||||
file_extension: Option<String>,
|
||||
) -> Vec<HashOutcome> {
|
||||
vec![HashOutcome {
|
||||
hash: Hash(cid),
|
||||
hash_kind: HashKind::Sha2_256,
|
||||
bytes: Some(bytes),
|
||||
record: Some(record),
|
||||
file_extension,
|
||||
}]
|
||||
}
|
||||
|
||||
/// Percent-encode a query-string value using the unreserved set
|
||||
/// (RFC 3986 `A-Za-z0-9-_.~`) plus `+` for space.
|
||||
pub fn urlencode(s: &str) -> String {
|
||||
let mut out = String::with_capacity(s.len());
|
||||
for b in s.as_bytes() {
|
||||
match *b {
|
||||
b'A'..=b'Z' | b'a'..=b'z' | b'0'..=b'9' | b'-' | b'_' | b'.' | b'~' => {
|
||||
out.push(*b as char);
|
||||
}
|
||||
b' ' => out.push('+'),
|
||||
other => out.push_str(&format!("%{:02X}", other)),
|
||||
}
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
/// GET `url` and return the body bytes, mapping transport + status errors to
|
||||
/// [`GatewayError`].
|
||||
pub async fn fetch_bytes(http: &reqwest::Client, url: &str) -> Result<bytes::Bytes, GatewayError> {
|
||||
let resp = http
|
||||
.get(url)
|
||||
.send()
|
||||
.await
|
||||
.map_err(|e| GatewayError::Transient(format!("GET {url}: {e}")))?;
|
||||
map_status(&resp)?;
|
||||
resp.bytes()
|
||||
.await
|
||||
.map_err(|e| GatewayError::Transient(format!("read body {url}: {e}")))
|
||||
}
|
||||
|
||||
/// Like [`fetch_bytes`] but decodes the body as UTF-8 text.
|
||||
pub async fn fetch_text(http: &reqwest::Client, url: &str) -> Result<String, GatewayError> {
|
||||
let resp = http
|
||||
.get(url)
|
||||
.send()
|
||||
.await
|
||||
.map_err(|e| GatewayError::Transient(format!("GET {url}: {e}")))?;
|
||||
map_status(&resp)?;
|
||||
resp.text()
|
||||
.await
|
||||
.map_err(|e| GatewayError::Transient(format!("read body {url}: {e}")))
|
||||
}
|
||||
|
||||
/// Decode a v1 BitTorrent infohash string into its raw 20 bytes.
|
||||
///
|
||||
/// Accepts 40 hex chars (case-insensitive) or 32 base32 chars (RFC 4648,
|
||||
/// case-insensitive). Returns `None` for anything else.
|
||||
pub fn decode_infohash(s: &str) -> Option<[u8; 20]> {
|
||||
let bytes = s.as_bytes();
|
||||
let raw: Vec<u8> = if bytes.len() == 40 && bytes.iter().all(u8::is_ascii_hexdigit) {
|
||||
let mut out = Vec::with_capacity(20);
|
||||
for pair in bytes.chunks(2) {
|
||||
out.push((hex_nibble(pair[0])? << 4) | hex_nibble(pair[1])?);
|
||||
}
|
||||
out
|
||||
} else if bytes.len() == 32 && bytes.iter().all(u8::is_ascii_alphanumeric) {
|
||||
decode_base32_20(bytes)?
|
||||
} else {
|
||||
return None;
|
||||
};
|
||||
raw.try_into().ok()
|
||||
}
|
||||
|
||||
fn hex_nibble(b: u8) -> Option<u8> {
|
||||
match b {
|
||||
b'0'..=b'9' => Some(b - b'0'),
|
||||
b'a'..=b'f' => Some(b - b'a' + 10),
|
||||
b'A'..=b'F' => Some(b - b'A' + 10),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
/// Decode 32 RFC-4648 base32 chars (case-insensitive, no padding) into the 20
|
||||
/// bytes they encode.
|
||||
fn decode_base32_20(chars: &[u8]) -> Option<Vec<u8>> {
|
||||
let mut out = Vec::with_capacity(20);
|
||||
let mut buffer: u64 = 0;
|
||||
let mut bits: u32 = 0;
|
||||
for &c in chars {
|
||||
let val = match c {
|
||||
b'A'..=b'Z' => c - b'A',
|
||||
b'a'..=b'z' => c - b'a',
|
||||
b'2'..=b'7' => c - b'2' + 26,
|
||||
_ => return None,
|
||||
} as u64;
|
||||
buffer = (buffer << 5) | val;
|
||||
bits += 5;
|
||||
if bits >= 8 {
|
||||
bits -= 8;
|
||||
out.push((buffer >> bits) as u8);
|
||||
}
|
||||
}
|
||||
Some(out)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn decode_infohash_hex_roundtrips() {
|
||||
let hex = "0123456789abcdef0123456789abcdef01234567";
|
||||
let raw = decode_infohash(hex).expect("hex decodes");
|
||||
assert_eq!(raw[0], 0x01);
|
||||
assert_eq!(raw[19], 0x67);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn decode_infohash_is_case_insensitive_hex() {
|
||||
let lower = decode_infohash("aabbccddeeff00112233445566778899aabbccdd");
|
||||
let upper = decode_infohash("AABBCCDDEEFF00112233445566778899AABBCCDD");
|
||||
assert!(lower.is_some());
|
||||
assert_eq!(lower, upper);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn decode_infohash_accepts_base32() {
|
||||
let raw = decode_infohash("MFRGGZDFMZTWQ2LKNNWG23TPOBYXE43U");
|
||||
assert!(raw.is_some());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn decode_infohash_rejects_wrong_shape() {
|
||||
assert!(decode_infohash("tooshort").is_none());
|
||||
assert!(decode_infohash("zz23456789abcdef0123456789abcdef01234567").is_none());
|
||||
assert!(decode_infohash("").is_none());
|
||||
}
|
||||
}
|
||||
@@ -1,312 +0,0 @@
|
||||
//! Self-describing per-plugin configuration: schema, redaction, merge, and the
|
||||
//! generic schema-driven HTML page the feeder serves at `/config`.
|
||||
//!
|
||||
//! The point of this module is that **the gateway carries no per-plugin config
|
||||
//! knowledge** and **the SDK carries no per-plugin field names**. A plugin
|
||||
//! declares its fields via [`FeederPlugin::config_schema`](crate::FeederPlugin::config_schema);
|
||||
//! everything here (redaction of secrets, merge-on-save, and the rendered form)
|
||||
//! is driven entirely by that schema. Add a feeder → it shows up with a working
|
||||
//! config form and zero UI/gateway changes.
|
||||
//!
|
||||
//! ## Persistence & precedence
|
||||
//!
|
||||
//! Config lives at `cache_dir/config.json` (the per-plugin dir handed to
|
||||
//! `configure()`). The plugin reads **file-or-env** at startup: env is the
|
||||
//! first-boot seed and stays authoritative until the operator saves through the
|
||||
//! UI, at which point `config.json` exists and wins. There is no runtime reload
|
||||
//! — a save returns `restart_required: true` (mirrors the gateway's
|
||||
//! `gateway-config.json` posture).
|
||||
//!
|
||||
//! ## Secrets
|
||||
//!
|
||||
//! Fields of kind [`FieldKind::Secret`] are **write-only**: never returned to
|
||||
//! the page (redacted to a `<key>_set: bool`), and a blank incoming value on
|
||||
//! save means "keep the stored secret".
|
||||
|
||||
use serde::{Deserialize, Serialize};
|
||||
use serde_json::{Map, Value};
|
||||
|
||||
/// A plugin's full config surface — an ordered list of fields.
|
||||
#[derive(Serialize, Deserialize, Clone, Debug, Default)]
|
||||
pub struct ConfigSchema {
|
||||
pub fields: Vec<ConfigField>,
|
||||
}
|
||||
|
||||
impl ConfigSchema {
|
||||
/// Convenience: `true` when the plugin declares no configurable fields (the
|
||||
/// page then renders a "no configuration required" notice).
|
||||
pub fn is_empty(&self) -> bool {
|
||||
self.fields.is_empty()
|
||||
}
|
||||
}
|
||||
|
||||
/// One configurable field. `record_array` fields nest `fields` for the
|
||||
/// per-record sub-schema (e.g. torznab's `indexers`).
|
||||
#[derive(Serialize, Deserialize, Clone, Debug)]
|
||||
pub struct ConfigField {
|
||||
/// JSON key in the config object.
|
||||
pub key: String,
|
||||
/// Human label for the form.
|
||||
pub label: String,
|
||||
#[serde(rename = "type")]
|
||||
pub kind: FieldKind,
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub help: Option<String>,
|
||||
#[serde(default)]
|
||||
pub required: bool,
|
||||
/// Sub-fields for [`FieldKind::RecordArray`]. The FIRST sub-field is treated
|
||||
/// as the record identifier for secret back-fill on merge (e.g. `name`).
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub fields: Option<Vec<ConfigField>>,
|
||||
}
|
||||
|
||||
impl ConfigField {
|
||||
pub fn text(key: &str, label: &str) -> Self {
|
||||
Self::new(key, label, FieldKind::Text)
|
||||
}
|
||||
pub fn secret(key: &str, label: &str) -> Self {
|
||||
Self::new(key, label, FieldKind::Secret)
|
||||
}
|
||||
pub fn bool(key: &str, label: &str) -> Self {
|
||||
Self::new(key, label, FieldKind::Bool)
|
||||
}
|
||||
pub fn number(key: &str, label: &str) -> Self {
|
||||
Self::new(key, label, FieldKind::Number)
|
||||
}
|
||||
pub fn list(key: &str, label: &str) -> Self {
|
||||
Self::new(key, label, FieldKind::List)
|
||||
}
|
||||
pub fn record_array(key: &str, label: &str, fields: Vec<ConfigField>) -> Self {
|
||||
let mut f = Self::new(key, label, FieldKind::RecordArray);
|
||||
f.fields = Some(fields);
|
||||
f
|
||||
}
|
||||
fn new(key: &str, label: &str, kind: FieldKind) -> Self {
|
||||
Self {
|
||||
key: key.to_string(),
|
||||
label: label.to_string(),
|
||||
kind,
|
||||
help: None,
|
||||
required: false,
|
||||
fields: None,
|
||||
}
|
||||
}
|
||||
pub fn with_help(mut self, help: &str) -> Self {
|
||||
self.help = Some(help.to_string());
|
||||
self
|
||||
}
|
||||
pub fn required(mut self) -> Self {
|
||||
self.required = true;
|
||||
self
|
||||
}
|
||||
}
|
||||
|
||||
/// Field widget / value type. Drives both rendering and the secret/merge rules.
|
||||
#[derive(Serialize, Deserialize, Clone, Copy, Debug, PartialEq, Eq)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum FieldKind {
|
||||
/// Free text → JSON string.
|
||||
Text,
|
||||
/// Write-only secret → JSON string; redacted on read, "blank keeps" on save.
|
||||
Secret,
|
||||
/// Checkbox → JSON bool.
|
||||
Bool,
|
||||
/// Numeric input → JSON number.
|
||||
Number,
|
||||
/// Comma/line list → JSON array of strings.
|
||||
List,
|
||||
/// Repeatable record → JSON array of objects keyed by nested `fields`.
|
||||
RecordArray,
|
||||
}
|
||||
|
||||
fn is_blank(v: Option<&Value>) -> bool {
|
||||
match v {
|
||||
None | Some(Value::Null) => true,
|
||||
Some(Value::String(s)) => s.is_empty(),
|
||||
_ => false,
|
||||
}
|
||||
}
|
||||
|
||||
/// Produce a UI-safe view of `values`: secret fields are dropped and replaced
|
||||
/// with a `<key>_set` boolean; everything else passes through. Recurses into
|
||||
/// `record_array` records for secret sub-fields.
|
||||
pub fn redact(values: &Value, schema: &ConfigSchema) -> Value {
|
||||
let src = values.as_object().cloned().unwrap_or_default();
|
||||
Value::Object(redact_object(&src, &schema.fields))
|
||||
}
|
||||
|
||||
fn redact_object(src: &Map<String, Value>, fields: &[ConfigField]) -> Map<String, Value> {
|
||||
let mut out = Map::new();
|
||||
for f in fields {
|
||||
match f.kind {
|
||||
FieldKind::Secret => {
|
||||
let set = !is_blank(src.get(&f.key));
|
||||
out.insert(format!("{}_set", f.key), Value::Bool(set));
|
||||
}
|
||||
FieldKind::RecordArray => {
|
||||
let subs = f.fields.as_deref().unwrap_or(&[]);
|
||||
let arr = src
|
||||
.get(&f.key)
|
||||
.and_then(Value::as_array)
|
||||
.map(|recs| {
|
||||
recs.iter()
|
||||
.map(|r| {
|
||||
let m = r.as_object().cloned().unwrap_or_default();
|
||||
Value::Object(redact_object(&m, subs))
|
||||
})
|
||||
.collect::<Vec<_>>()
|
||||
})
|
||||
.unwrap_or_default();
|
||||
out.insert(f.key.clone(), Value::Array(arr));
|
||||
}
|
||||
_ => {
|
||||
if let Some(v) = src.get(&f.key) {
|
||||
out.insert(f.key.clone(), v.clone());
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
/// Merge `incoming` (from the UI) onto `base` (the stored/effective config),
|
||||
/// schema-aware. Returns the full, UNREDACTED object to persist:
|
||||
/// - secret: blank/absent incoming keeps `base`; non-blank overwrites.
|
||||
/// - text/number/bool/list: present in incoming overwrites; absent keeps base.
|
||||
/// - record_array: incoming array replaces base, but blank secret sub-fields are
|
||||
/// back-filled from the base record matched by the first sub-field (identifier).
|
||||
pub fn merge(base: &Value, incoming: &Value, schema: &ConfigSchema) -> Value {
|
||||
let base_obj = base.as_object().cloned().unwrap_or_default();
|
||||
let inc_obj = incoming.as_object().cloned().unwrap_or_default();
|
||||
Value::Object(merge_object(&base_obj, &inc_obj, &schema.fields))
|
||||
}
|
||||
|
||||
fn merge_object(
|
||||
base: &Map<String, Value>,
|
||||
inc: &Map<String, Value>,
|
||||
fields: &[ConfigField],
|
||||
) -> Map<String, Value> {
|
||||
let mut out = base.clone();
|
||||
for f in fields {
|
||||
match f.kind {
|
||||
FieldKind::Secret => {
|
||||
if !is_blank(inc.get(&f.key)) {
|
||||
out.insert(f.key.clone(), inc.get(&f.key).cloned().unwrap());
|
||||
}
|
||||
// blank/absent → keep base (no-op)
|
||||
}
|
||||
FieldKind::RecordArray => {
|
||||
let subs = f.fields.as_deref().unwrap_or(&[]);
|
||||
let id_key = subs.first().map(|s| s.key.clone());
|
||||
let base_recs = base.get(&f.key).and_then(Value::as_array).cloned();
|
||||
let inc_recs = match inc.get(&f.key).and_then(Value::as_array) {
|
||||
Some(r) => r.clone(),
|
||||
None => continue, // absent → keep base array untouched
|
||||
};
|
||||
let merged: Vec<Value> = inc_recs
|
||||
.iter()
|
||||
.map(|rec| {
|
||||
let rec_obj = rec.as_object().cloned().unwrap_or_default();
|
||||
// Find the matching base record by identifier sub-field.
|
||||
let base_match = match (&id_key, base_recs.as_ref()) {
|
||||
(Some(idk), Some(brs)) => brs
|
||||
.iter()
|
||||
.find(|br| br.get(idk) == rec_obj.get(idk))
|
||||
.and_then(Value::as_object)
|
||||
.cloned()
|
||||
.unwrap_or_default(),
|
||||
_ => Map::new(),
|
||||
};
|
||||
Value::Object(merge_object(&base_match, &rec_obj, subs))
|
||||
})
|
||||
.collect();
|
||||
out.insert(f.key.clone(), Value::Array(merged));
|
||||
}
|
||||
_ => {
|
||||
if let Some(v) = inc.get(&f.key) {
|
||||
out.insert(f.key.clone(), v.clone());
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
/// The generic, schema-driven config page served at `GET /config`. Self-contained
|
||||
/// (inline CSS + JS, no external assets) so it loads identically when hit
|
||||
/// directly on the feeder OR reverse-proxied through the gateway. All API calls
|
||||
/// are relative to `location.pathname`, so the proxy mount prefix is irrelevant.
|
||||
pub const CONFIG_PAGE_HTML: &str = include_str!("config_page.html");
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use serde_json::json;
|
||||
|
||||
fn torznab_like() -> ConfigSchema {
|
||||
ConfigSchema {
|
||||
fields: vec![
|
||||
ConfigField::secret("tmdb_token", "TMDB token"),
|
||||
ConfigField::bool("resolve_torrent_files", "Resolve .torrent"),
|
||||
ConfigField::record_array(
|
||||
"indexers",
|
||||
"Indexers",
|
||||
vec![
|
||||
ConfigField::text("name", "Name"),
|
||||
ConfigField::secret("api_key", "API key"),
|
||||
],
|
||||
),
|
||||
],
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn redact_hides_secrets_and_flags_set() {
|
||||
let v = json!({
|
||||
"tmdb_token": "abc",
|
||||
"resolve_torrent_files": true,
|
||||
"indexers": [{"name":"prowlarr","api_key":"k"}]
|
||||
});
|
||||
let r = redact(&v, &torznab_like());
|
||||
assert_eq!(r["tmdb_token_set"], json!(true));
|
||||
assert!(r.get("tmdb_token").is_none());
|
||||
assert_eq!(r["resolve_torrent_files"], json!(true));
|
||||
assert_eq!(r["indexers"][0]["name"], json!("prowlarr"));
|
||||
assert_eq!(r["indexers"][0]["api_key_set"], json!(true));
|
||||
assert!(r["indexers"][0].get("api_key").is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn merge_blank_secret_keeps_base() {
|
||||
let base = json!({"tmdb_token": "keepme"});
|
||||
let inc = json!({"tmdb_token": ""});
|
||||
let m = merge(&base, &inc, &torznab_like());
|
||||
assert_eq!(m["tmdb_token"], json!("keepme"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn merge_nonblank_secret_overwrites() {
|
||||
let base = json!({"tmdb_token": "old"});
|
||||
let inc = json!({"tmdb_token": "new"});
|
||||
let m = merge(&base, &inc, &torznab_like());
|
||||
assert_eq!(m["tmdb_token"], json!("new"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn merge_record_array_backfills_blank_secret_by_identifier() {
|
||||
let base = json!({"indexers":[{"name":"prowlarr","api_key":"secret"}]});
|
||||
// UI re-submits the indexer with a blank api_key → keep the stored one.
|
||||
let inc = json!({"indexers":[{"name":"prowlarr","api_key":""}]});
|
||||
let m = merge(&base, &inc, &torznab_like());
|
||||
assert_eq!(m["indexers"][0]["api_key"], json!("secret"));
|
||||
assert_eq!(m["indexers"][0]["name"], json!("prowlarr"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn merge_bool_overwrites_when_present() {
|
||||
let base = json!({"resolve_torrent_files": true});
|
||||
let inc = json!({"resolve_torrent_files": false});
|
||||
let m = merge(&base, &inc, &torznab_like());
|
||||
assert_eq!(m["resolve_torrent_files"], json!(false));
|
||||
}
|
||||
}
|
||||
@@ -1,268 +0,0 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
||||
<title>feeder config</title>
|
||||
<style>
|
||||
:root { color-scheme: dark; }
|
||||
* { box-sizing: border-box; }
|
||||
body {
|
||||
margin: 0; padding: 18px;
|
||||
font: 14px/1.5 system-ui, -apple-system, Segoe UI, Roboto, sans-serif;
|
||||
background: #0d0d12; color: #e6e6ea;
|
||||
}
|
||||
h1 { font-size: 16px; margin: 0 0 4px; }
|
||||
.sub { color: #9a9aa6; font-size: 12px; margin: 0 0 16px; }
|
||||
.field { margin: 0 0 16px; }
|
||||
.field > label { display: block; font-weight: 600; margin: 0 0 4px; }
|
||||
.help { color: #8a8a96; font-size: 12px; margin: 3px 0 0; }
|
||||
.req { color: #f0a0a0; margin-left: 3px; font-weight: 700; }
|
||||
input[type=text], input[type=password], input[type=number], textarea {
|
||||
width: 100%; padding: 8px 10px; border-radius: 8px;
|
||||
border: 1px solid #2a2a35; background: #15151c; color: #e6e6ea; font: inherit;
|
||||
}
|
||||
textarea { min-height: 56px; resize: vertical; }
|
||||
input[type=checkbox] { width: 16px; height: 16px; vertical-align: middle; }
|
||||
.pill { font-size: 10px; padding: 2px 7px; border-radius: 999px; margin-left: 8px; vertical-align: middle; }
|
||||
.pill.set { background: #14361f; color: #6ee7a0; border: 1px solid #1f5132; }
|
||||
.pill.unset { background: #2a2a35; color: #b9b9c4; border: 1px solid #3a3a45; }
|
||||
.record { border: 1px solid #2a2a35; border-radius: 10px; padding: 12px; margin: 0 0 10px; background: #111118; }
|
||||
.record .field { margin-bottom: 10px; }
|
||||
.row-actions { text-align: right; }
|
||||
button {
|
||||
font: inherit; cursor: pointer; border-radius: 8px; padding: 7px 14px;
|
||||
border: 1px solid #3a3a45; background: #1c1c26; color: #e6e6ea;
|
||||
}
|
||||
button.primary { background: #6d4aff; border-color: #6d4aff; color: #fff; font-weight: 600; }
|
||||
button.danger { background: transparent; border-color: #5a2a2a; color: #e29; padding: 4px 10px; }
|
||||
button.add { margin-top: 4px; }
|
||||
.bar { position: sticky; bottom: 0; padding: 12px 0 2px; background: linear-gradient(#0d0d1200, #0d0d12 30%); }
|
||||
.banner { padding: 9px 12px; border-radius: 8px; margin: 0 0 14px; font-size: 13px; display: none; }
|
||||
.banner.ok { background: #14361f; color: #9ff0bf; border: 1px solid #1f5132; display: block; }
|
||||
.banner.warn { background: #3a2f12; color: #f0d79f; border: 1px solid #5a4a1f; display: block; }
|
||||
.banner.err { background: #3a1414; color: #f0a0a0; border: 1px solid #5a2020; display: block; }
|
||||
.empty { color: #9a9aa6; }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<h1 id="title">Plugin configuration</h1>
|
||||
<p class="sub">Secrets are write-only — set values are shown as “set”, never echoed back. Saving requires a feeder restart to take effect. Fields marked <span class="req">*</span> are required.</p>
|
||||
<div id="banner" class="banner"></div>
|
||||
<form id="form"></form>
|
||||
<div class="bar" id="bar" style="display:none">
|
||||
<button type="button" class="primary" id="save">Save</button>
|
||||
</div>
|
||||
|
||||
<script>
|
||||
// All API calls are relative to this page's path, so the same page works when
|
||||
// served directly on the feeder (/config) or reverse-proxied through the
|
||||
// gateway (/api/gateway/plugins/<id>/ui/config).
|
||||
const BASE = location.pathname.replace(/\/+$/, "");
|
||||
let SCHEMA = { fields: [] };
|
||||
let VALUES = {};
|
||||
|
||||
const $ = (id) => document.getElementById(id);
|
||||
function banner(kind, msg) {
|
||||
const b = $("banner");
|
||||
b.className = "banner " + kind;
|
||||
b.textContent = msg;
|
||||
}
|
||||
|
||||
function el(tag, attrs, ...kids) {
|
||||
const e = document.createElement(tag);
|
||||
for (const [k, v] of Object.entries(attrs || {})) {
|
||||
if (k === "class") e.className = v;
|
||||
else if (k === "text") e.textContent = v;
|
||||
else if (k in e) e[k] = v;
|
||||
else e.setAttribute(k, v);
|
||||
}
|
||||
for (const c of kids) if (c != null) e.append(c);
|
||||
return e;
|
||||
}
|
||||
|
||||
// Split help text into plain runs + clickable <a> nodes for any http(s) URL.
|
||||
// Built as DOM nodes (never innerHTML) so help stays XSS-safe; trailing
|
||||
// punctuation is kept outside the link so "see http://host/ui." reads right.
|
||||
function linkify(text) {
|
||||
const out = [];
|
||||
const re = /https?:\/\/[^\s]+/g;
|
||||
let last = 0, m;
|
||||
while ((m = re.exec(text)) !== null) {
|
||||
if (m.index > last) out.push(text.slice(last, m.index));
|
||||
let url = m[0], trail = "";
|
||||
const t = url.match(/[.,;:)\]]+$/);
|
||||
if (t) { trail = t[0]; url = url.slice(0, -trail.length); }
|
||||
out.push(el("a", { href: url, target: "_blank", rel: "noopener noreferrer", text: url }));
|
||||
if (trail) out.push(trail);
|
||||
last = m.index + m[0].length;
|
||||
}
|
||||
if (last < text.length) out.push(text.slice(last));
|
||||
return out;
|
||||
}
|
||||
|
||||
function helpEl(f) { return f.help ? el("p", { class: "help" }, ...linkify(f.help)) : null; }
|
||||
|
||||
// A "*" marker for required fields (null for optional ones).
|
||||
function reqStar(f) { return f.required ? el("span", { class: "req", text: "*", title: "required" }) : null; }
|
||||
|
||||
// Render one scalar field. `vals` is the object holding the value/`_set` flag.
|
||||
// Every returned widget exposes `_collect()` and `_key` so collect() can read it.
|
||||
function fieldInput(f, vals) {
|
||||
if (f.type === "bool") {
|
||||
const cb = el("input", { type: "checkbox" });
|
||||
cb.checked = vals[f.key] === true;
|
||||
const w = el("div", { class: "field" },
|
||||
el("label", {}, cb, " ", el("span", { text: " " + f.label }), reqStar(f)),
|
||||
helpEl(f));
|
||||
w._collect = () => cb.checked;
|
||||
w._key = f.key;
|
||||
return w;
|
||||
}
|
||||
const label = el("label", { text: f.label });
|
||||
const star = reqStar(f); if (star) label.append(star);
|
||||
let input;
|
||||
if (f.type === "secret") {
|
||||
const isSet = vals[f.key + "_set"] === true;
|
||||
input = el("input", { type: "password", placeholder: isSet ? "•••••••• (blank keeps current)" : "not set" });
|
||||
label.append(el("span", { class: "pill " + (isSet ? "set" : "unset"), text: isSet ? "set" : "not set" }));
|
||||
input._get = () => input.value; // blank => merge keeps stored secret
|
||||
} else if (f.type === "number") {
|
||||
input = el("input", { type: "number" });
|
||||
if (vals[f.key] != null) input.value = vals[f.key];
|
||||
input._get = () => (input.value === "" ? null : Number(input.value));
|
||||
} else if (f.type === "list") {
|
||||
input = el("textarea", { placeholder: "one per line or comma-separated" });
|
||||
const arr = Array.isArray(vals[f.key]) ? vals[f.key] : [];
|
||||
input.value = arr.join("\n");
|
||||
input._get = () => input.value.split(/[\n,]/).map((s) => s.trim()).filter(Boolean);
|
||||
} else {
|
||||
input = el("input", { type: "text" });
|
||||
if (vals[f.key] != null) input.value = vals[f.key];
|
||||
input._get = () => input.value;
|
||||
}
|
||||
const wrap = el("div", { class: "field" }, label, input, helpEl(f));
|
||||
wrap._collect = () => input._get();
|
||||
wrap._key = f.key;
|
||||
return wrap;
|
||||
}
|
||||
|
||||
// Render a record_array (repeatable group). Returns a wrapper exposing _collect.
|
||||
function recordArray(f, vals) {
|
||||
const subs = f.fields || [];
|
||||
const idKey = subs.length ? subs[0].key : null;
|
||||
const list = el("div", {});
|
||||
const records = Array.isArray(vals[f.key]) ? vals[f.key].slice() : [];
|
||||
|
||||
function addRecord(recVals) {
|
||||
const card = el("div", { class: "record" });
|
||||
const inputs = subs.map((sf) => {
|
||||
const fi = fieldInput(sf, recVals || {});
|
||||
card.append(fi);
|
||||
return fi;
|
||||
});
|
||||
const rm = el("button", { type: "button", class: "danger", text: "remove" });
|
||||
rm.onclick = () => card.remove();
|
||||
card.append(el("div", { class: "row-actions" }, rm));
|
||||
card._collect = () => {
|
||||
const o = {};
|
||||
for (const fi of inputs) o[fi._key] = fi._collect();
|
||||
return o;
|
||||
};
|
||||
list.append(card);
|
||||
return card;
|
||||
}
|
||||
|
||||
records.forEach((r) => addRecord(r));
|
||||
const add = el("button", { type: "button", class: "add", text: "+ add " + f.label.toLowerCase().replace(/s$/, "") });
|
||||
add.onclick = () => addRecord({});
|
||||
|
||||
const lbl = el("label", { text: f.label });
|
||||
const star = reqStar(f); if (star) lbl.append(star);
|
||||
const wrap = el("div", { class: "field" }, lbl, helpEl(f), list, add);
|
||||
wrap._key = f.key;
|
||||
wrap._collect = () => Array.from(list.children).map((c) => c._collect());
|
||||
return wrap;
|
||||
}
|
||||
|
||||
function render() {
|
||||
$("title").textContent = (SCHEMA.title || "Plugin") + " configuration";
|
||||
const form = $("form");
|
||||
form.innerHTML = "";
|
||||
if (!SCHEMA.fields || !SCHEMA.fields.length) {
|
||||
form.append(el("p", { class: "empty", text: "No configuration required for this plugin." }));
|
||||
$("bar").style.display = "none";
|
||||
return;
|
||||
}
|
||||
const widgets = [];
|
||||
for (const f of SCHEMA.fields) {
|
||||
const w = f.type === "record_array" ? recordArray(f, VALUES) : fieldInput(f, VALUES);
|
||||
widgets.push(w);
|
||||
form.append(w);
|
||||
}
|
||||
form._widgets = widgets;
|
||||
$("bar").style.display = "block";
|
||||
}
|
||||
|
||||
function collect() {
|
||||
const out = {};
|
||||
for (const w of $("form")._widgets || []) out[w._key] = w._collect();
|
||||
return out;
|
||||
}
|
||||
|
||||
async function save() {
|
||||
$("save").disabled = true;
|
||||
try {
|
||||
const res = await fetch(BASE + "/values", {
|
||||
method: "PUT",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify(collect()),
|
||||
});
|
||||
if (!res.ok) {
|
||||
const detail = (await res.text().catch(() => "")).trim();
|
||||
throw new Error(detail ? `the feeder rejected it (HTTP ${res.status}): ${detail}`
|
||||
: `the feeder returned HTTP ${res.status}`);
|
||||
}
|
||||
const j = await res.json().catch(() => ({}));
|
||||
if (j.reloading) {
|
||||
// The feeder is restarting itself (~2-3s) to apply the change. Poll it
|
||||
// back up, then refresh the form so secret pills reflect stored state.
|
||||
banner("warn", "Saved — reloading the feeder to apply (~3 s)…");
|
||||
let tries = 6;
|
||||
const poll = async () => {
|
||||
try { await load(); banner("ok", "Reloaded — new config applied."); }
|
||||
catch { if (--tries > 0) setTimeout(poll, 1500); else banner("warn", "Saved & reloading — reopen this panel shortly to confirm."); }
|
||||
};
|
||||
setTimeout(poll, 3000);
|
||||
} else {
|
||||
if (j.restart_required) banner("warn", "Saved. Restart the feeder container for changes to take effect.");
|
||||
else banner("ok", "Saved.");
|
||||
// Re-fetch so secret pills reflect the new stored state.
|
||||
await load();
|
||||
}
|
||||
} catch (e) {
|
||||
const msg = (e && e.message) ? e.message : String(e);
|
||||
const friendly = /Failed to fetch|NetworkError|load failed/i.test(msg)
|
||||
? "couldn't reach the feeder — it may be restarting; wait a moment and try again."
|
||||
: msg;
|
||||
banner("err", "Could not save: " + friendly);
|
||||
} finally {
|
||||
$("save").disabled = false;
|
||||
}
|
||||
}
|
||||
|
||||
async function load() {
|
||||
const [s, v] = await Promise.all([
|
||||
fetch(BASE + "/schema").then((r) => r.json()),
|
||||
fetch(BASE + "/values").then((r) => r.json()),
|
||||
]);
|
||||
SCHEMA = s || { fields: [] };
|
||||
VALUES = v || {};
|
||||
render();
|
||||
}
|
||||
|
||||
$("save").onclick = save;
|
||||
load().catch((e) => banner("err", "Failed to load config: " + e.message));
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -1,614 +0,0 @@
|
||||
//! Feeder-driven enrichment — the "smart feeder / dumb gateway" pipeline.
|
||||
//!
|
||||
//! A feeder that self-publishes (has a `META_CORE_URL`) writes its own base
|
||||
//! record, then drives the **reused meta-sort enrichment plugins**
|
||||
//! (`metamesh-plugin-filename-parser`, `metamesh-plugin-tmdb`) over HTTP. Each
|
||||
//! plugin is handed the `metaCoreUrl` and writes its own fields straight back to
|
||||
//! meta-core (`PATCH /meta/{cid}` merge), so the gateway never touches
|
||||
//! enrichment — full separation of concerns.
|
||||
//!
|
||||
//! ## Ordering — why we poll
|
||||
//!
|
||||
//! tmdb searches on `originalTitle`/`movieYear`/`videoType` (which it reads from
|
||||
//! the record), and those are produced by filename-parser. So the pipeline is
|
||||
//! **filename-parser → tmdb**. The plugin `/process` contract is *async*
|
||||
//! (returns `accepted`, then writes to meta-core and POSTs a callback), so after
|
||||
//! kicking filename-parser we **poll meta-core** until `originalTitle` appears
|
||||
//! (bounded), read the merged fields back, and hand them to tmdb. Polling keeps
|
||||
//! the feeder stateless — no callback-correlation bookkeeping. The plugins'
|
||||
//! callback POSTs still fire; they hit the feeder's no-op `/enrich/callback`.
|
||||
//!
|
||||
//! The whole pipeline runs in a spawned background task so it never adds latency
|
||||
//! to the libp2p resolve path; the meta-share ingester propagates the enriched
|
||||
//! record on its next poll (~eventually consistent).
|
||||
|
||||
use std::collections::BTreeMap;
|
||||
use std::sync::Arc;
|
||||
use std::time::{Duration, SystemTime, UNIX_EPOCH};
|
||||
|
||||
use serde::Serialize;
|
||||
use tracing::{debug, warn};
|
||||
|
||||
use crate::meta_core::{self, Provenance};
|
||||
use crate::types::DiscoveryRecord;
|
||||
|
||||
/// Operator-supplied enrichment wiring, parsed once from the feeder env.
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct EnrichmentConfig {
|
||||
/// meta-core root the feeder self-publishes to + hands to the plugins.
|
||||
pub meta_core_url: String,
|
||||
/// `metamesh-plugin-filename-parser` base URL (optional).
|
||||
pub filename_parser_url: Option<String>,
|
||||
/// `metamesh-plugin-tmdb` base URL (optional).
|
||||
pub tmdb_url: Option<String>,
|
||||
/// TMDB v3 key / v4 bearer token, POSTed to tmdb `/configure`.
|
||||
pub tmdb_token: Option<String>,
|
||||
/// TMDB metadata language (default `en-US`).
|
||||
pub tmdb_language: String,
|
||||
/// Stable per-deployment id stamped into `provenance.gatewayPeer`.
|
||||
pub gateway_peer: String,
|
||||
/// URL the plugins POST their completion callback to (the feeder's own
|
||||
/// no-op `/enrich/callback`). Polling is the real completion signal; this
|
||||
/// only keeps the plugins from logging a failed-callback warning.
|
||||
pub callback_url: String,
|
||||
/// `metamesh-plugin-opensubtitles` base URL (optional). When set, the
|
||||
/// pipeline drives subtitle fetching after tmdb resolves the ids — the
|
||||
/// single reference implementation of OpenSubtitles fetching (the gateway
|
||||
/// no longer carries its own).
|
||||
pub opensubtitles_url: Option<String>,
|
||||
/// OpenSubtitles consumer API key, POSTed to the plugin's `/configure`.
|
||||
pub opensubtitles_api_key: Option<String>,
|
||||
/// OpenSubtitles account username (required by the plugin for downloads).
|
||||
pub opensubtitles_username: Option<String>,
|
||||
/// OpenSubtitles account password.
|
||||
pub opensubtitles_password: Option<String>,
|
||||
/// Wanted subtitle languages (CSV of ISO 639-1 codes; default `en`).
|
||||
pub opensubtitles_languages: String,
|
||||
}
|
||||
|
||||
fn env_nonempty(key: &str) -> Option<String> {
|
||||
std::env::var(key)
|
||||
.ok()
|
||||
.map(|s| s.trim().to_string())
|
||||
.filter(|s| !s.is_empty())
|
||||
}
|
||||
|
||||
/// Derive a co-bundled sidecar plugin URL from the feeder's own hostname, by the
|
||||
/// app-compose convention `<prefix>-feeder` → `<prefix>-<plugin>:8080`
|
||||
/// (e.g. `metafeedertorznab-feeder` → `http://metafeedertorznab-tmdb:8080`). Lets
|
||||
/// a feeder reach its bundled filename-parser / tmdb sidecars with **no explicit
|
||||
/// `*_PLUGIN_URL` env** — config lives in the UI, wiring is derived. Returns
|
||||
/// `None` when `HOSTNAME` is unset or not in `<prefix>-feeder` form, in which case
|
||||
/// the caller falls back to the env seed (then soft-skips that plugin).
|
||||
fn derive_sidecar_url(plugin: &str) -> Option<String> {
|
||||
let host = env_nonempty("HOSTNAME")?;
|
||||
let prefix = host.strip_suffix("-feeder")?;
|
||||
Some(format!("http://{prefix}-{plugin}:8080"))
|
||||
}
|
||||
|
||||
impl EnrichmentConfig {
|
||||
/// Parse from the feeder env. Returns `None` when `META_CORE_URL` is unset —
|
||||
/// the **bundled-feeder** path (records returned over HTTP, the gateway
|
||||
/// dispatcher stores them). Self-publishing feeders that source meta-core from
|
||||
/// their own dashboard config use [`from_meta_core`](Self::from_meta_core).
|
||||
pub fn from_env() -> Option<Self> {
|
||||
Some(Self::from_meta_core(env_nonempty("META_CORE_URL")?))
|
||||
}
|
||||
|
||||
/// Build with an explicit meta-core root — sourced from the feeder's own
|
||||
/// `config.json` (dashboard) when set, else the `META_CORE_URL` env seed. The
|
||||
/// sidecar URLs come from their `*_PLUGIN_URL` env, falling back to the
|
||||
/// hostname-derived co-bundled sidecar (see [`derive_sidecar_url`]); the rest
|
||||
/// keep their env seeds / defaults. Secrets (tmdb token, language) are
|
||||
/// overlaid by the caller from the per-feeder config.
|
||||
pub fn from_meta_core(meta_core_url: String) -> Self {
|
||||
EnrichmentConfig {
|
||||
meta_core_url,
|
||||
filename_parser_url: env_nonempty("FILENAME_PARSER_URL")
|
||||
.or_else(|| derive_sidecar_url("filename-parser")),
|
||||
tmdb_url: env_nonempty("TMDB_PLUGIN_URL").or_else(|| derive_sidecar_url("tmdb")),
|
||||
tmdb_token: env_nonempty("TMDB_TOKEN").or_else(|| env_nonempty("PROWLARR_TMDB_TOKEN")),
|
||||
tmdb_language: env_nonempty("TMDB_LANGUAGE").unwrap_or_else(|| "en-US".to_string()),
|
||||
gateway_peer: env_nonempty("META_GATEWAY_PEER_ID")
|
||||
.or_else(|| env_nonempty("HOSTNAME"))
|
||||
.unwrap_or_else(|| "gateway-feeder".to_string()),
|
||||
callback_url: env_nonempty("META_FEEDER_CALLBACK_URL")
|
||||
.unwrap_or_else(|| "http://127.0.0.1:8080/enrich/callback".to_string()),
|
||||
opensubtitles_url: env_nonempty("OPENSUBTITLES_PLUGIN_URL"),
|
||||
opensubtitles_api_key: env_nonempty("OPENSUBTITLES_API_KEY"),
|
||||
opensubtitles_username: env_nonempty("OPENSUBTITLES_USERNAME"),
|
||||
opensubtitles_password: env_nonempty("OPENSUBTITLES_PASSWORD"),
|
||||
opensubtitles_languages: env_nonempty("OPENSUBTITLES_LANGUAGES")
|
||||
.unwrap_or_else(|| "en".to_string()),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// How long to wait for filename-parser to merge `originalTitle` before handing
|
||||
/// off to tmdb. filename-parser is a pure, fast string parse; this is a generous
|
||||
/// ceiling, not an expected wait.
|
||||
const FILENAME_PARSE_POLL_TIMEOUT: Duration = Duration::from_secs(5);
|
||||
/// How long to wait for tmdb to merge `tmdbid` before fanning its anchor to the
|
||||
/// sibling files. tmdb hits the network, so a longer ceiling than filename.
|
||||
const TMDB_POLL_TIMEOUT: Duration = Duration::from_secs(20);
|
||||
const POLL_INTERVAL: Duration = Duration::from_millis(200);
|
||||
|
||||
/// Show-level TMDB fields that are **fanned** from the primary file to every
|
||||
/// sibling file in a multi-file (season-pack) torrent — they're identical for
|
||||
/// every episode of a show, so we resolve tmdb **once** and copy these across
|
||||
/// (avoids N TMDB lookups; the tmdb plugin caches by `cid_midhash256`, which
|
||||
/// gateway BT records don't carry, so without this each file would re-hit TMDB).
|
||||
/// Per-file fields (`originalTitle`, `season`, `episode`, `fileName`, `sizeByte`,
|
||||
/// `quality`, …) are deliberately NOT fanned — filename-parser sets those per file.
|
||||
const ANCHOR_EXACT: &[&str] = &[
|
||||
"tmdbid",
|
||||
"imdbid",
|
||||
"poster",
|
||||
"backdrop",
|
||||
"rating",
|
||||
"releasedate",
|
||||
"studio",
|
||||
"genres",
|
||||
"tags",
|
||||
];
|
||||
const ANCHOR_PREFIXES: &[&str] = &["plot/", "genres/", "studio/", "tags/"];
|
||||
|
||||
/// One file (or pack) to enrich: its CID, the filename to parse, and the base
|
||||
/// fields already written to meta-core (handed to the plugins as `existingMeta`).
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct EnrichTarget {
|
||||
pub cid: String,
|
||||
pub file_name: String,
|
||||
pub base: BTreeMap<String, String>,
|
||||
}
|
||||
|
||||
/// Drives meta-core writes + the enrichment plugins for one feeder. Cheap to
|
||||
/// clone (just a reqwest client + an `Arc<config>`), so it can be moved into a
|
||||
/// spawned per-resolve task.
|
||||
#[derive(Clone)]
|
||||
pub struct Enricher {
|
||||
http: reqwest::Client,
|
||||
cfg: Arc<EnrichmentConfig>,
|
||||
}
|
||||
|
||||
#[derive(Serialize)]
|
||||
struct ProcessRequest<'a> {
|
||||
#[serde(rename = "taskId")]
|
||||
task_id: String,
|
||||
cid: &'a str,
|
||||
#[serde(rename = "filePath")]
|
||||
file_path: &'a str,
|
||||
#[serde(rename = "callbackUrl")]
|
||||
callback_url: &'a str,
|
||||
#[serde(rename = "metaCoreUrl")]
|
||||
meta_core_url: &'a str,
|
||||
#[serde(rename = "existingMeta")]
|
||||
existing_meta: &'a BTreeMap<String, String>,
|
||||
}
|
||||
|
||||
impl Enricher {
|
||||
pub fn new(http: reqwest::Client, cfg: EnrichmentConfig) -> Self {
|
||||
Enricher {
|
||||
http,
|
||||
cfg: Arc::new(cfg),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn meta_core_url(&self) -> &str {
|
||||
&self.cfg.meta_core_url
|
||||
}
|
||||
|
||||
/// `true` if any enrichment plugin is wired (so there's enrichment to drive
|
||||
/// beyond the base-record write).
|
||||
pub fn has_plugins(&self) -> bool {
|
||||
self.cfg.filename_parser_url.is_some() || self.cfg.tmdb_url.is_some()
|
||||
}
|
||||
|
||||
/// Write the feeder's base record into meta-core (merge/upsert) under `cid`.
|
||||
/// Identical shape to the gateway dispatcher's write (provenance + the
|
||||
/// `cids/<cid>` key-set collapse).
|
||||
pub async fn write_base_record(
|
||||
&self,
|
||||
cid: &str,
|
||||
record: &DiscoveryRecord,
|
||||
) -> anyhow::Result<()> {
|
||||
let prov = Provenance::now(
|
||||
&record.upstream_id,
|
||||
&record.record_id,
|
||||
&self.cfg.gateway_peer,
|
||||
);
|
||||
let body = meta_core::build_metadata_body(record, &prov);
|
||||
meta_core::put_record(&self.http, &self.cfg.meta_core_url, cid, &body).await
|
||||
}
|
||||
|
||||
/// Run the video enrichment pipeline for one resolved CID: filename-parser
|
||||
/// (clean title/year/season/episode) → tmdb (tmdbid/poster/overview). Each
|
||||
/// plugin writes its own fields to meta-core. Best-effort: any plugin
|
||||
/// failure is logged and the pipeline continues (the base record already
|
||||
/// landed, so the item is at worst un-enriched, never lost).
|
||||
pub async fn enrich_video(
|
||||
&self,
|
||||
cid: &str,
|
||||
file_name: &str,
|
||||
base_fields: BTreeMap<String, String>,
|
||||
) {
|
||||
let mut meta = base_fields;
|
||||
|
||||
// 1. filename-parser → originalTitle / movieYear / videoType / season / episode.
|
||||
if let Some(fp_url) = self.cfg.filename_parser_url.clone() {
|
||||
match self.process(&fp_url, "filename-parser", cid, file_name, &meta).await {
|
||||
Ok(()) => {
|
||||
// Poll for the merged fields so tmdb gets the clean title.
|
||||
if let Some(merged) = self
|
||||
.poll_until_field(cid, "originalTitle", FILENAME_PARSE_POLL_TIMEOUT)
|
||||
.await
|
||||
{
|
||||
meta = merged;
|
||||
} else {
|
||||
debug!(
|
||||
target: "meta-feeder::enrich",
|
||||
cid, "filename-parser produced no originalTitle within timeout; \
|
||||
handing tmdb the base fields"
|
||||
);
|
||||
}
|
||||
}
|
||||
Err(e) => warn!(
|
||||
target: "meta-feeder::enrich",
|
||||
cid, error = %e, "filename-parser /process failed (non-fatal)"
|
||||
),
|
||||
}
|
||||
}
|
||||
|
||||
// 2. tmdb → tmdbid / poster / overview / genres (requires fileType=video +
|
||||
// a clean originalTitle, both now present on `meta`).
|
||||
if let Some(tmdb_url) = self.cfg.tmdb_url.clone() {
|
||||
// Configure the API key before each call — tmdb holds it in memory
|
||||
// only, so this is robust to a tmdb-plugin restart. Cheap (local).
|
||||
if let Err(e) = self.configure_tmdb(&tmdb_url).await {
|
||||
warn!(
|
||||
target: "meta-feeder::enrich",
|
||||
cid, error = %e, "tmdb /configure failed (continuing; may soft-skip)"
|
||||
);
|
||||
}
|
||||
if let Err(e) = self.process(&tmdb_url, "tmdb", cid, file_name, &meta).await {
|
||||
warn!(
|
||||
target: "meta-feeder::enrich",
|
||||
cid, error = %e, "tmdb /process failed (non-fatal)"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
// 3. opensubtitles → subtitles/<lang3>/<cid> (needs tmdbid/imdbid, which
|
||||
// tmdb writes async). Poll for tmdbid so the plugin has an id to
|
||||
// search by, then drive it like any other enrichment plugin.
|
||||
if self.cfg.opensubtitles_url.is_some() {
|
||||
let merged = self
|
||||
.poll_until_field(cid, "tmdbid", TMDB_POLL_TIMEOUT)
|
||||
.await
|
||||
.unwrap_or(meta);
|
||||
self.enrich_subtitles(cid, file_name, &merged).await;
|
||||
}
|
||||
}
|
||||
|
||||
/// Enrich a **bundle** from one torrent (metainfo fan-out): N video files +
|
||||
/// an optional whole-torrent pack record. filename-parser runs on every
|
||||
/// target; tmdb runs **once** on the primary video file and its show-level
|
||||
/// anchor (`tmdbid`/poster/genres/…) is fanned to the sibling files. The
|
||||
/// pack record gets filename-parse only (no tmdb — it's the non-playable
|
||||
/// "release as a unit" search handle, D1). Best-effort throughout.
|
||||
pub async fn enrich_bundle(&self, videos: Vec<EnrichTarget>, pack: Option<EnrichTarget>) {
|
||||
// Pack: filename-parse only, no tmdb.
|
||||
if let Some(p) = pack {
|
||||
self.fire_filename(&p).await;
|
||||
}
|
||||
|
||||
if videos.is_empty() {
|
||||
return;
|
||||
}
|
||||
if videos.len() == 1 {
|
||||
// Degenerate single-video bundle → the plain pipeline (filename → tmdb).
|
||||
let v = &videos[0];
|
||||
self.enrich_video(&v.cid, &v.file_name, v.base.clone()).await;
|
||||
return;
|
||||
}
|
||||
|
||||
// Multi-file: fire filename-parser for every file (fire-and-forget — only
|
||||
// the primary is polled, the rest write async).
|
||||
for v in &videos {
|
||||
self.fire_filename(v).await;
|
||||
}
|
||||
|
||||
// Primary = largest by `sizeByte` (ties → lowest index, i.e. first seen).
|
||||
let primary = pick_primary(&videos);
|
||||
let p = &videos[primary];
|
||||
|
||||
// Poll the primary for filename-parser's clean title, then tmdb it.
|
||||
let merged = self
|
||||
.poll_until_field(&p.cid, "originalTitle", FILENAME_PARSE_POLL_TIMEOUT)
|
||||
.await
|
||||
.unwrap_or_else(|| p.base.clone());
|
||||
|
||||
let Some(tmdb_url) = self.cfg.tmdb_url.clone() else {
|
||||
return; // no tmdb wired → filename-only enrichment, done.
|
||||
};
|
||||
if let Err(e) = self.configure_tmdb(&tmdb_url).await {
|
||||
warn!(target: "meta-feeder::enrich", cid = %p.cid, error = %e, "tmdb /configure failed");
|
||||
}
|
||||
if let Err(e) = self.process(&tmdb_url, "tmdb", &p.cid, &p.file_name, &merged).await {
|
||||
warn!(target: "meta-feeder::enrich", cid = %p.cid, error = %e, "primary tmdb /process failed");
|
||||
return;
|
||||
}
|
||||
|
||||
// Read the resolved anchor back off the primary, then fan it to siblings.
|
||||
let Some(anchor_rec) = self.poll_until_field(&p.cid, "tmdbid", TMDB_POLL_TIMEOUT).await
|
||||
else {
|
||||
debug!(target: "meta-feeder::enrich", cid = %p.cid, "primary produced no tmdbid; nothing to fan");
|
||||
return;
|
||||
};
|
||||
let anchor = extract_anchor(&anchor_rec);
|
||||
if anchor.is_empty() {
|
||||
return;
|
||||
}
|
||||
for (i, v) in videos.iter().enumerate() {
|
||||
if i == primary {
|
||||
continue;
|
||||
}
|
||||
// PUT is merge/upsert — this layers the show-level anchor onto each
|
||||
// file's record without disturbing its per-file fields.
|
||||
if let Err(e) =
|
||||
meta_core::put_record(&self.http, &self.cfg.meta_core_url, &v.cid, &anchor).await
|
||||
{
|
||||
warn!(target: "meta-feeder::enrich", cid = %v.cid, error = %e, "anchor fan PUT failed");
|
||||
}
|
||||
}
|
||||
|
||||
// Subtitles per video: each episode needs its own (its season/episode
|
||||
// differs), but they share the fanned tmdbid. Poll each for tmdbid
|
||||
// (present on the primary, fanned onto the siblings) and drive the
|
||||
// opensubtitles plugin. Best-effort + quota-guarded inside the plugin.
|
||||
if self.cfg.opensubtitles_url.is_some() {
|
||||
for v in &videos {
|
||||
let merged = self
|
||||
.poll_until_field(&v.cid, "tmdbid", TMDB_POLL_TIMEOUT)
|
||||
.await
|
||||
.unwrap_or_else(|| v.base.clone());
|
||||
self.enrich_subtitles(&v.cid, &v.file_name, &merged).await;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Fire filename-parser `/process` for one target (fire-and-forget; the
|
||||
/// plugin merges its fields to meta-core async).
|
||||
async fn fire_filename(&self, t: &EnrichTarget) {
|
||||
if let Some(fp_url) = self.cfg.filename_parser_url.clone() {
|
||||
if let Err(e) = self
|
||||
.process(&fp_url, "filename-parser", &t.cid, &t.file_name, &t.base)
|
||||
.await
|
||||
{
|
||||
warn!(target: "meta-feeder::enrich", cid = %t.cid, error = %e, "filename-parser /process failed");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// POST `/process` to a plugin. Returns once the plugin *accepts* the task
|
||||
/// (the actual meta-core write is async on the plugin side; the caller polls
|
||||
/// meta-core for the result when it needs it downstream).
|
||||
async fn process(
|
||||
&self,
|
||||
plugin_url: &str,
|
||||
plugin: &str,
|
||||
cid: &str,
|
||||
file_path: &str,
|
||||
existing_meta: &BTreeMap<String, String>,
|
||||
) -> anyhow::Result<()> {
|
||||
let url = format!("{}/process", plugin_url.trim_end_matches('/'));
|
||||
let body = ProcessRequest {
|
||||
task_id: format!("{plugin}-{cid}-{}", now_nanos()),
|
||||
cid,
|
||||
file_path,
|
||||
callback_url: &self.cfg.callback_url,
|
||||
meta_core_url: &self.cfg.meta_core_url,
|
||||
existing_meta,
|
||||
};
|
||||
let resp = self.http.post(&url).json(&body).send().await?;
|
||||
if !resp.status().is_success() {
|
||||
let status = resp.status();
|
||||
let text = resp.text().await.unwrap_or_default();
|
||||
anyhow::bail!("{plugin} POST {url} returned {status}: {text}");
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// POST `/configure` to the tmdb plugin with the API key + language.
|
||||
async fn configure_tmdb(&self, tmdb_url: &str) -> anyhow::Result<()> {
|
||||
let Some(token) = self.cfg.tmdb_token.clone() else {
|
||||
anyhow::bail!("no TMDB token configured");
|
||||
};
|
||||
let url = format!("{}/configure", tmdb_url.trim_end_matches('/'));
|
||||
let body = serde_json::json!({
|
||||
"config": { "apiKey": token, "language": self.cfg.tmdb_language }
|
||||
});
|
||||
let resp = self.http.post(&url).json(&body).send().await?;
|
||||
if !resp.status().is_success() {
|
||||
let status = resp.status();
|
||||
let text = resp.text().await.unwrap_or_default();
|
||||
anyhow::bail!("tmdb /configure returned {status}: {text}");
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// POST `/configure` to the opensubtitles plugin with the API key, account
|
||||
/// credentials, and wanted languages.
|
||||
async fn configure_opensubtitles(&self, url: &str) -> anyhow::Result<()> {
|
||||
let Some(key) = self.cfg.opensubtitles_api_key.clone() else {
|
||||
anyhow::bail!("no OpenSubtitles API key configured");
|
||||
};
|
||||
let cfg_url = format!("{}/configure", url.trim_end_matches('/'));
|
||||
let body = serde_json::json!({
|
||||
"config": {
|
||||
"apiKey": key,
|
||||
"username": self.cfg.opensubtitles_username,
|
||||
"password": self.cfg.opensubtitles_password,
|
||||
"languages": self.cfg.opensubtitles_languages,
|
||||
}
|
||||
});
|
||||
let resp = self.http.post(&cfg_url).json(&body).send().await?;
|
||||
if !resp.status().is_success() {
|
||||
let status = resp.status();
|
||||
let text = resp.text().await.unwrap_or_default();
|
||||
anyhow::bail!("opensubtitles /configure returned {status}: {text}");
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Drive the opensubtitles plugin for one video CID. Needs a `tmdbid` or
|
||||
/// `imdbid` in `meta` to search by (the plugin reads them from `existingMeta`
|
||||
/// and writes the `subtitles/<lang3>/<cid>` key-set straight to meta-core).
|
||||
/// Best-effort: any failure is logged and swallowed.
|
||||
async fn enrich_subtitles(&self, cid: &str, file_name: &str, meta: &BTreeMap<String, String>) {
|
||||
let Some(url) = self.cfg.opensubtitles_url.clone() else {
|
||||
return;
|
||||
};
|
||||
if !meta.contains_key("tmdbid") && !meta.contains_key("imdbid") {
|
||||
debug!(
|
||||
target: "meta-feeder::enrich",
|
||||
cid, "no tmdbid/imdbid resolved; skipping opensubtitles"
|
||||
);
|
||||
return;
|
||||
}
|
||||
if let Err(e) = self.configure_opensubtitles(&url).await {
|
||||
warn!(
|
||||
target: "meta-feeder::enrich",
|
||||
cid, error = %e, "opensubtitles /configure failed (continuing; may soft-skip)"
|
||||
);
|
||||
}
|
||||
if let Err(e) = self.process(&url, "opensubtitles", cid, file_name, meta).await {
|
||||
warn!(
|
||||
target: "meta-feeder::enrich",
|
||||
cid, error = %e, "opensubtitles /process failed (non-fatal)"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/// Poll `GET /meta/{cid}` until `field` is present (non-empty) or `timeout`
|
||||
/// elapses; returns the merged field map on success.
|
||||
async fn poll_until_field(
|
||||
&self,
|
||||
cid: &str,
|
||||
field: &str,
|
||||
timeout: Duration,
|
||||
) -> Option<BTreeMap<String, String>> {
|
||||
let deadline = SystemTime::now() + timeout;
|
||||
loop {
|
||||
match meta_core::get_record(&self.http, &self.cfg.meta_core_url, cid).await {
|
||||
Ok(Some(fields)) if fields.get(field).is_some_and(|v| !v.is_empty()) => {
|
||||
return Some(fields);
|
||||
}
|
||||
Ok(_) => {}
|
||||
Err(e) => debug!(
|
||||
target: "meta-feeder::enrich",
|
||||
cid, error = %e, "poll get_record failed"
|
||||
),
|
||||
}
|
||||
if SystemTime::now() >= deadline {
|
||||
return None;
|
||||
}
|
||||
tokio::time::sleep(POLL_INTERVAL).await;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn now_nanos() -> u128 {
|
||||
SystemTime::now()
|
||||
.duration_since(UNIX_EPOCH)
|
||||
.map(|d| d.as_nanos())
|
||||
.unwrap_or(0)
|
||||
}
|
||||
|
||||
/// Pick the primary video file index: largest by `sizeByte`, ties broken toward
|
||||
/// the first (lowest index). Falls back to 0 for an all-unsized bundle.
|
||||
fn pick_primary(videos: &[EnrichTarget]) -> usize {
|
||||
videos
|
||||
.iter()
|
||||
.enumerate()
|
||||
.max_by_key(|(i, v)| {
|
||||
let size = v.base.get("sizeByte").and_then(|s| s.parse::<u64>().ok()).unwrap_or(0);
|
||||
// larger size wins; on a tie the smaller index wins (Reverse).
|
||||
(size, std::cmp::Reverse(*i))
|
||||
})
|
||||
.map(|(i, _)| i)
|
||||
.unwrap_or(0)
|
||||
}
|
||||
|
||||
/// Project a resolved record down to the show-level anchor fields that are safe
|
||||
/// to copy to sibling files (see [`ANCHOR_EXACT`] / [`ANCHOR_PREFIXES`]).
|
||||
fn extract_anchor(fields: &BTreeMap<String, String>) -> BTreeMap<String, String> {
|
||||
fields
|
||||
.iter()
|
||||
.filter(|(k, _)| {
|
||||
ANCHOR_EXACT.contains(&k.as_str())
|
||||
|| ANCHOR_PREFIXES.iter().any(|p| k.starts_with(p))
|
||||
})
|
||||
.map(|(k, v)| (k.clone(), v.clone()))
|
||||
.collect()
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn from_env_requires_meta_core_url() {
|
||||
// Not set in the test env → None (legacy gateway-stores path).
|
||||
// (We don't mutate process env here to avoid cross-test races; this just
|
||||
// documents the contract.)
|
||||
assert!(EnrichmentConfig::from_env().is_none() || EnrichmentConfig::from_env().is_some());
|
||||
}
|
||||
|
||||
fn tgt(cid: &str, size: u64) -> EnrichTarget {
|
||||
EnrichTarget {
|
||||
cid: cid.into(),
|
||||
file_name: cid.into(),
|
||||
base: BTreeMap::from_iter([("sizeByte".to_string(), size.to_string())]),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn pick_primary_is_largest_then_first() {
|
||||
let v = vec![tgt("a", 100), tgt("b", 900), tgt("c", 900), tgt("d", 50)];
|
||||
// 'b' and 'c' tie at 900 → lowest index ('b', idx 1) wins.
|
||||
assert_eq!(pick_primary(&v), 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn pick_primary_all_unsized_defaults_first() {
|
||||
let v = vec![
|
||||
EnrichTarget { cid: "a".into(), file_name: "a".into(), base: BTreeMap::new() },
|
||||
EnrichTarget { cid: "b".into(), file_name: "b".into(), base: BTreeMap::new() },
|
||||
];
|
||||
assert_eq!(pick_primary(&v), 0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn extract_anchor_keeps_show_level_drops_per_file() {
|
||||
let rec = BTreeMap::from_iter([
|
||||
("tmdbid".to_string(), "10378".to_string()),
|
||||
("imdbid".to_string(), "tt1254207".to_string()),
|
||||
("poster".to_string(), "bafkPOSTER".to_string()),
|
||||
("genres".to_string(), "Animation|Comedy".to_string()),
|
||||
("plot/eng".to_string(), "a plot".to_string()),
|
||||
("tags".to_string(), "tmdb-verified".to_string()),
|
||||
// per-file — must NOT be fanned:
|
||||
("originalTitle".to_string(), "Big Buck Bunny".to_string()),
|
||||
("season".to_string(), "1".to_string()),
|
||||
("episode".to_string(), "3".to_string()),
|
||||
("fileName".to_string(), "ep3.mkv".to_string()),
|
||||
]);
|
||||
let a = extract_anchor(&rec);
|
||||
assert!(a.contains_key("tmdbid") && a.contains_key("poster") && a.contains_key("plot/eng"));
|
||||
assert!(a.contains_key("genres") && a.contains_key("tags") && a.contains_key("imdbid"));
|
||||
assert!(!a.contains_key("originalTitle"), "originalTitle is per-file");
|
||||
assert!(!a.contains_key("season") && !a.contains_key("episode"));
|
||||
assert!(!a.contains_key("fileName"));
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,574 +0,0 @@
|
||||
//! Content-addressed identifiers used by feeder plugins' `compute_outcomes`
|
||||
//! to hash upstream-fetched bytes.
|
||||
//!
|
||||
//! Two function families live here:
|
||||
//!
|
||||
//! - [`compute_midhash256`] / [`compute_midhash256_from_sample`] —
|
||||
//! the size-prefix-plus-middle-1MB-sample identification primitive shared
|
||||
//! with `meta-sort` / `meta-share` / meta-core. Custom multicodec `0x1000`.
|
||||
//! Fast but not IPFS-interop. The `_from_sample` entrypoint takes the
|
||||
//! middle slice directly so plugins that can only fetch a partial range
|
||||
//! (BitTorrent BEP 9, sliced HTTP) don't have to buffer the whole file
|
||||
//! just to throw most of it away. **Rust-only by construction.**
|
||||
//! - [`compute_ipfs_cid`] is the standard IPFS CIDv1 produced over the full
|
||||
//! bytes — raw codec `0x55` for a single-chunk file, UnixFS dag-pb codec
|
||||
//! `0x70` rooted on chunked raw-codec leaves for larger files. Output matches
|
||||
//! `ipfs add --cid-version=1 --raw-leaves=true --chunker=size-262144
|
||||
//! --hash=sha2-256 <file>`.
|
||||
//!
|
||||
//! Cross-implementation parity for `compute_midhash256` with the TypeScript
|
||||
//! `FastHash.ts` is load-bearing — keep the fixture tests pinned, a regression
|
||||
//! silently breaks dedupe across the platform.
|
||||
|
||||
use bytes::Bytes;
|
||||
use sha2::{Digest, Sha256};
|
||||
|
||||
/// Sample window in bytes (1 MiB). Files this size or smaller are hashed
|
||||
/// in full; larger files are hashed over the middle 1 MiB slice.
|
||||
const SAMPLE_SIZE: usize = 1024 * 1024;
|
||||
|
||||
/// Per-block IPFS chunk size (256 KiB), matches kubo's
|
||||
/// `--chunker=size-262144` default.
|
||||
pub const IPFS_CHUNK_SIZE: usize = 256 * 1024;
|
||||
|
||||
/// dag-pb fanout — children per internal node. Matches kubo's balanced
|
||||
/// builder default.
|
||||
pub const IPFS_FANOUT: usize = 174;
|
||||
|
||||
/// Output of [`compute_ipfs_blocks`]. Carries the root CID **plus every
|
||||
/// intermediate block** (leaf chunks AND internal dag-pb nodes) keyed by
|
||||
/// their own CID. Pass `.blocks` to the gateway's blockstore so peers
|
||||
/// can fetch the file by CID via bitswap.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct IpfsBlocks {
|
||||
/// Canonical "bafy…" / "bafk…" root cid string. Identical to what
|
||||
/// [`compute_ipfs_cid`] returns for the same input.
|
||||
pub root: String,
|
||||
/// Every block — root included — keyed by its own cid. For a
|
||||
/// single-leaf file, this is a one-entry vec `[(root, payload)]`
|
||||
/// (raw codec, payload is the file bytes). For multi-leaf files,
|
||||
/// it is `leaves ++ internal_nodes` in build order; the **last**
|
||||
/// entry is the root.
|
||||
pub blocks: Vec<(String, Bytes)>,
|
||||
}
|
||||
|
||||
/// Compute a midhash256 CID matching the TypeScript implementation in
|
||||
/// `meta-hash/src/lib/file-id/FastHash.ts`:
|
||||
///
|
||||
/// 1. Build the hash input as `[size:u64-be][middle 1 MiB sample]` (or the
|
||||
/// whole file if it fits in 1 MiB).
|
||||
/// 2. SHA-256 it.
|
||||
/// 3. Wrap in CIDv1 with custom multicodec `0x1000` for both the codec field
|
||||
/// and the multihash hash-code field, length 32, then multibase-base32-lower
|
||||
/// with the `b` prefix.
|
||||
///
|
||||
/// **Endianness for the size prefix is big-endian** to match TS's
|
||||
/// `writeBigUInt64BE`. Cross-impl mismatch here would silently mangle every
|
||||
/// import — keep the test pinned.
|
||||
pub fn compute_midhash256(bytes: &[u8]) -> String {
|
||||
let size = bytes.len();
|
||||
let sample: &[u8] = if size <= SAMPLE_SIZE {
|
||||
bytes
|
||||
} else {
|
||||
let start = (size - SAMPLE_SIZE) / 2;
|
||||
&bytes[start..start + SAMPLE_SIZE]
|
||||
};
|
||||
compute_midhash256_from_sample(size as u64, sample)
|
||||
}
|
||||
|
||||
/// Same midhash256 primitive as [`compute_midhash256`], but takes the
|
||||
/// sample bytes directly rather than slicing them out of a full file.
|
||||
/// For upstreams that can only deliver a middle range (BitTorrent BEP 9,
|
||||
/// sliced HTTP), this avoids forcing the caller to buffer the whole file
|
||||
/// in memory just to throw most of it away.
|
||||
///
|
||||
/// `total_size` is the **full file size in bytes** — used as the
|
||||
/// size prefix, NOT `middle_sample.len()`.
|
||||
pub fn compute_midhash256_from_sample(total_size: u64, middle_sample: &[u8]) -> String {
|
||||
let size_be = total_size.to_be_bytes();
|
||||
let mut hasher = Sha256::new();
|
||||
hasher.update(size_be);
|
||||
hasher.update(middle_sample);
|
||||
let digest: [u8; 32] = hasher.finalize().into();
|
||||
|
||||
// Multicodec 0x1000 encoded as unsigned varint = [0x80, 0x20].
|
||||
// CIDv1 layout: [version=0x01][codec varint][multihash code varint][len][digest]
|
||||
let mut wire = Vec::with_capacity(38);
|
||||
wire.push(0x01);
|
||||
wire.extend_from_slice(&[0x80, 0x20]);
|
||||
wire.extend_from_slice(&[0x80, 0x20]);
|
||||
wire.push(0x20);
|
||||
wire.extend_from_slice(&digest);
|
||||
|
||||
format!("b{}", base32_lower_no_padding(&wire))
|
||||
}
|
||||
|
||||
/// Wrap a 20-byte BitTorrent v1 info-hash as a multibase CIDv1.
|
||||
///
|
||||
/// BT v1 info-hashes ARE SHA-1 digests of the bencoded info dict — so
|
||||
/// the natural CID form is `codec=raw (0x55)` + `multihash=sha1 (0x11),
|
||||
/// len=20, digest=<infohash bytes>`.
|
||||
pub fn compute_bt_info_cid(infohash_20: &[u8; 20]) -> String {
|
||||
// CIDv1 layout: [version=0x01][codec varint][multihash code varint][len][digest]
|
||||
let mut wire = Vec::with_capacity(4 + 20);
|
||||
wire.push(0x01); // CIDv1 version
|
||||
wire.push(0x55); // codec: raw
|
||||
wire.push(0x11); // multihash: sha1
|
||||
wire.push(0x14); // multihash length: 20
|
||||
wire.extend_from_slice(infohash_20);
|
||||
format!("b{}", base32_lower_no_padding(&wire))
|
||||
}
|
||||
|
||||
/// Custom multicodec for a single file inside a **BitTorrent v1** torrent
|
||||
/// (`btih-v1-file`). MetaMesh-private, adjacent to midhash256's `0x1000`.
|
||||
pub const BTIH_V1_FILE_CODEC: u64 = 0x1001;
|
||||
|
||||
/// Custom multicodec for a single file inside a **BitTorrent v2** torrent
|
||||
/// (`btih-v2-file`).
|
||||
pub const BTIH_V2_FILE_CODEC: u64 = 0x1002;
|
||||
|
||||
/// Encode a `btih-v1-file` CID — a single file inside a BT v1 torrent,
|
||||
/// addressed by the 20-byte v1 infohash (SHA-1 of the bencoded `info`
|
||||
/// dict) plus the zero-based file index in `info.files`. Single-file
|
||||
/// torrents use index `0`.
|
||||
pub fn compute_bt_v1_file_cid(infohash_20: &[u8; 20], file_index: u64) -> String {
|
||||
bt_file_cid(BTIH_V1_FILE_CODEC, infohash_20, file_index)
|
||||
}
|
||||
|
||||
/// Encode a `btih-v2-file` CID — a single file inside a BT v2 torrent
|
||||
/// (BEP 52), addressed by the 32-byte v2 infohash plus the file's
|
||||
/// canonical traversal index in `file tree`.
|
||||
pub fn compute_bt_v2_file_cid(infohash_32: &[u8; 32], file_index: u64) -> String {
|
||||
bt_file_cid(BTIH_V2_FILE_CODEC, infohash_32, file_index)
|
||||
}
|
||||
|
||||
/// Shared encoder for the two torrent-file CID families. `infohash` is 20
|
||||
/// bytes for v1 / 32 for v2; `codec` selects the family.
|
||||
fn bt_file_cid(codec: u64, infohash: &[u8], file_index: u64) -> String {
|
||||
// digest = infohash ‖ varint(file_index)
|
||||
let mut digest = Vec::with_capacity(infohash.len() + 2);
|
||||
digest.extend_from_slice(infohash);
|
||||
write_pb_varint(file_index, &mut digest);
|
||||
|
||||
// CIDv1: [version][codec varint][multihash code varint][len varint][digest]
|
||||
let mut wire = Vec::with_capacity(1 + 4 + digest.len());
|
||||
wire.push(0x01);
|
||||
write_pb_varint(codec, &mut wire);
|
||||
write_pb_varint(codec, &mut wire);
|
||||
write_pb_varint(digest.len() as u64, &mut wire);
|
||||
wire.extend_from_slice(&digest);
|
||||
format!("b{}", base32_lower_no_padding(&wire))
|
||||
}
|
||||
|
||||
/// Custom multicodec for a generic **Newznab release** locator (`nzb-release`).
|
||||
/// MetaMesh-private, adjacent to the torrent-file codecs (`0x1002` is taken by
|
||||
/// [`BTIH_V2_FILE_CODEC`]). It identifies an indexer *listing* (host + release
|
||||
/// id), so it is derivable at search time with **no `.nzb` download**. See
|
||||
/// `USENET-GATEWAY-STUDY.md` §13. (The v1 content-stable `nzb-posting` `0x1003`
|
||||
/// variant — `sha256(message-ids)` — was removed; only this one is emitted.)
|
||||
pub const NZB_RELEASE_CODEC: u64 = 0x1004;
|
||||
|
||||
/// Encode an `nzb-release` CID from a Newznab indexer's host + bare release id
|
||||
/// (the 32-hex `<guid>` id). `sha2-256(host ‖ "\n" ‖ id)` — host-namespaced so
|
||||
/// the same movie on two indexers (different postings) yields different cids.
|
||||
/// Deterministic + opaque, same wire house-style as the other locator codecs.
|
||||
pub fn compute_nzb_release_cid(host: &str, release_id: &str) -> String {
|
||||
let mut hasher = Sha256::new();
|
||||
hasher.update(host.as_bytes());
|
||||
hasher.update(b"\n");
|
||||
hasher.update(release_id.as_bytes());
|
||||
let digest: [u8; 32] = hasher.finalize().into();
|
||||
|
||||
// CIDv1: [version=0x01][codec varint][multihash code varint][len=0x20][digest]
|
||||
let mut wire = Vec::with_capacity(1 + 4 + 1 + 32);
|
||||
wire.push(0x01);
|
||||
write_pb_varint(NZB_RELEASE_CODEC, &mut wire);
|
||||
write_pb_varint(NZB_RELEASE_CODEC, &mut wire);
|
||||
wire.push(0x20);
|
||||
wire.extend_from_slice(&digest);
|
||||
format!("b{}", base32_lower_no_padding(&wire))
|
||||
}
|
||||
|
||||
/// Compute a standard IPFS CIDv1 over `bytes`. Output matches kubo's
|
||||
/// `ipfs add --cid-version=1 --raw-leaves=true --chunker=size-262144
|
||||
/// --hash=sha2-256 <file>`.
|
||||
pub fn compute_ipfs_cid(bytes: &[u8]) -> String {
|
||||
compute_ipfs_blocks(bytes).root
|
||||
}
|
||||
|
||||
/// Same wire-format guarantee as [`compute_ipfs_cid`], but exposes every
|
||||
/// block produced along the way (leaf chunks + internal dag-pb nodes)
|
||||
/// so callers can populate a bitswap blockstore in one pass.
|
||||
pub fn compute_ipfs_blocks(bytes: &[u8]) -> IpfsBlocks {
|
||||
// Leaves: chunk bytes into raw-codec blocks. Empty input still gets
|
||||
// one (empty) leaf so the cid is well-defined.
|
||||
let mut blocks: Vec<(String, Bytes)> = Vec::new();
|
||||
let leaves: Vec<(Vec<u8>, u64)> = if bytes.is_empty() {
|
||||
let cid_wire = ipfs_cid_wire(0x55, &sha2_256(b""));
|
||||
blocks.push((cid_string(&cid_wire), Bytes::new()));
|
||||
vec![(cid_wire, 0)]
|
||||
} else {
|
||||
bytes
|
||||
.chunks(IPFS_CHUNK_SIZE)
|
||||
.map(|chunk| {
|
||||
let cid_wire = ipfs_cid_wire(0x55, &sha2_256(chunk));
|
||||
blocks.push((cid_string(&cid_wire), Bytes::copy_from_slice(chunk)));
|
||||
(cid_wire, chunk.len() as u64)
|
||||
})
|
||||
.collect()
|
||||
};
|
||||
|
||||
// Single-leaf file: return the leaf cid as the file cid.
|
||||
if leaves.len() == 1 {
|
||||
let root = cid_string(&leaves[0].0);
|
||||
return IpfsBlocks { root, blocks };
|
||||
}
|
||||
|
||||
// Multi-leaf: build a balanced tree of UnixFS dag-pb internal nodes.
|
||||
let mut level: Vec<(Vec<u8>, u64, u64)> = leaves
|
||||
.into_iter()
|
||||
.map(|(cid, size)| (cid, size, size))
|
||||
.collect();
|
||||
|
||||
while level.len() > 1 {
|
||||
let mut next: Vec<(Vec<u8>, u64, u64)> =
|
||||
Vec::with_capacity(level.len().div_ceil(IPFS_FANOUT));
|
||||
for batch in level.chunks(IPFS_FANOUT) {
|
||||
let blocksizes: Vec<u64> = batch.iter().map(|(_, sz, _)| *sz).collect();
|
||||
let filesize: u64 = blocksizes.iter().sum();
|
||||
let unixfs_data = encode_unixfs_file(filesize, &blocksizes);
|
||||
let node_bytes = encode_dagpb_node(batch, &unixfs_data);
|
||||
let node_cid = ipfs_cid_wire(0x70, &sha2_256(&node_bytes));
|
||||
blocks.push((cid_string(&node_cid), Bytes::from(node_bytes.clone())));
|
||||
let own_tsize: u64 =
|
||||
batch.iter().map(|(_, _, t)| *t).sum::<u64>() + node_bytes.len() as u64;
|
||||
next.push((node_cid, filesize, own_tsize));
|
||||
}
|
||||
level = next;
|
||||
}
|
||||
let root = cid_string(&level[0].0);
|
||||
IpfsBlocks { root, blocks }
|
||||
}
|
||||
|
||||
fn cid_string(wire: &[u8]) -> String {
|
||||
format!("b{}", base32_lower_no_padding(wire))
|
||||
}
|
||||
|
||||
/// CIDv1 wire-form bytes: `[version=0x01][codec varint][multihash code
|
||||
/// varint=0x12 sha2-256][len=0x20][digest...]`.
|
||||
fn ipfs_cid_wire(codec: u8, digest: &[u8; 32]) -> Vec<u8> {
|
||||
debug_assert!(
|
||||
codec < 0x80,
|
||||
"codec varint must fit in one byte for this helper"
|
||||
);
|
||||
let mut wire = Vec::with_capacity(36);
|
||||
wire.push(0x01); // CIDv1
|
||||
wire.push(codec);
|
||||
wire.push(0x12); // multihash code: sha2-256
|
||||
wire.push(0x20); // digest length: 32
|
||||
wire.extend_from_slice(digest);
|
||||
wire
|
||||
}
|
||||
|
||||
fn sha2_256(bytes: &[u8]) -> [u8; 32] {
|
||||
Sha256::digest(bytes).into()
|
||||
}
|
||||
|
||||
/// LEB128 unsigned varint encoding into `out`.
|
||||
fn write_pb_varint(value: u64, out: &mut Vec<u8>) {
|
||||
let mut v = value;
|
||||
while v >= 0x80 {
|
||||
out.push((v as u8) | 0x80);
|
||||
v >>= 7;
|
||||
}
|
||||
out.push(v as u8);
|
||||
}
|
||||
|
||||
fn write_pb_varint_field(field: u32, value: u64, out: &mut Vec<u8>) {
|
||||
write_pb_varint((field as u64) << 3, out); // tag: (field << 3) | wire-type 0 (varint)
|
||||
write_pb_varint(value, out);
|
||||
}
|
||||
|
||||
fn write_pb_bytes_field(field: u32, value: &[u8], out: &mut Vec<u8>) {
|
||||
write_pb_varint(((field as u64) << 3) | 2, out); // wire type 2 = length-delimited
|
||||
write_pb_varint(value.len() as u64, out);
|
||||
out.extend_from_slice(value);
|
||||
}
|
||||
|
||||
/// UnixFS protobuf payload for a File node: `Type=File, filesize,
|
||||
/// blocksizes[]`.
|
||||
fn encode_unixfs_file(filesize: u64, blocksizes: &[u64]) -> Vec<u8> {
|
||||
let mut out = Vec::new();
|
||||
write_pb_varint_field(1, 2, &mut out); // Type = File
|
||||
write_pb_varint_field(3, filesize, &mut out); // filesize
|
||||
for &bs in blocksizes {
|
||||
write_pb_varint_field(4, bs, &mut out); // blocksizes (repeated)
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
/// dag-pb PBNode for an internal UnixFS file node. Canonical wire order
|
||||
/// pinned by the dag-pb spec: `Links` (tag 2) first, `Data` (tag 1) second.
|
||||
fn encode_dagpb_node(children: &[(Vec<u8>, u64, u64)], data: &[u8]) -> Vec<u8> {
|
||||
let mut out = Vec::new();
|
||||
for (cid_wire, _filesize, tsize) in children {
|
||||
let mut link_bytes = Vec::new();
|
||||
write_pb_bytes_field(1, cid_wire, &mut link_bytes); // Hash
|
||||
write_pb_varint_field(3, *tsize, &mut link_bytes); // Tsize
|
||||
write_pb_bytes_field(2, &link_bytes, &mut out); // Links (tag 2) first
|
||||
}
|
||||
write_pb_bytes_field(1, data, &mut out); // Data (tag 1) last
|
||||
out
|
||||
}
|
||||
|
||||
/// RFC 4648 base32 with the lowercase alphabet and no padding. Used for the
|
||||
/// multibase `b` prefix.
|
||||
pub(crate) fn base32_lower_no_padding(input: &[u8]) -> String {
|
||||
const ALPHABET: &[u8; 32] = b"abcdefghijklmnopqrstuvwxyz234567";
|
||||
let mut out = String::with_capacity(input.len().div_ceil(5) * 8);
|
||||
let mut buffer: u64 = 0;
|
||||
let mut bits: u32 = 0;
|
||||
for &b in input {
|
||||
buffer = (buffer << 8) | b as u64;
|
||||
bits += 8;
|
||||
while bits >= 5 {
|
||||
bits -= 5;
|
||||
let idx = ((buffer >> bits) & 0x1F) as usize;
|
||||
out.push(ALPHABET[idx] as char);
|
||||
}
|
||||
}
|
||||
if bits > 0 {
|
||||
let idx = ((buffer << (5 - bits)) & 0x1F) as usize;
|
||||
out.push(ALPHABET[idx] as char);
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn midhash256_matches_ts_fixtures() {
|
||||
assert_eq!(
|
||||
compute_midhash256(b"hello world"),
|
||||
"bagacbabaec7v3fu2ygzh3e2sybg3fbzmisry2hbtpmck6vx3yftea6vzq35r4"
|
||||
);
|
||||
assert_eq!(
|
||||
compute_midhash256(b""),
|
||||
"bagacbabaecxvk4hvugaqw6xxrsxuxrykmyhq35i6ik5pshkn4wzdfdpa5a67y"
|
||||
);
|
||||
let mb_zeros = vec![0u8; 1024 * 1024];
|
||||
assert_eq!(
|
||||
compute_midhash256(&mb_zeros),
|
||||
"bagacbabaeawlkt4hn34sxuieoosrnnag3g3wv7gc6alzwfms2nrph2hochrx4"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn midhash256_from_sample_matches_full_bytes() {
|
||||
let small: &[u8] = b"hello world";
|
||||
assert_eq!(
|
||||
compute_midhash256_from_sample(small.len() as u64, small),
|
||||
compute_midhash256(small),
|
||||
);
|
||||
|
||||
assert_eq!(
|
||||
compute_midhash256_from_sample(0, b""),
|
||||
compute_midhash256(b""),
|
||||
);
|
||||
|
||||
let one_mb = vec![0u8; SAMPLE_SIZE];
|
||||
assert_eq!(
|
||||
compute_midhash256_from_sample(one_mb.len() as u64, &one_mb),
|
||||
compute_midhash256(&one_mb),
|
||||
);
|
||||
|
||||
let two_mb = vec![0u8; 2 * SAMPLE_SIZE];
|
||||
let start = (two_mb.len() - SAMPLE_SIZE) / 2;
|
||||
let middle = &two_mb[start..start + SAMPLE_SIZE];
|
||||
assert_eq!(
|
||||
compute_midhash256_from_sample(two_mb.len() as u64, middle),
|
||||
compute_midhash256(&two_mb),
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn midhash256_from_sample_partial_sample_stable() {
|
||||
let sample = vec![0u8; SAMPLE_SIZE];
|
||||
let total_5gb: u64 = 5 * 1024 * 1024 * 1024;
|
||||
let total_10gb: u64 = 10 * 1024 * 1024 * 1024;
|
||||
|
||||
let h1 = compute_midhash256_from_sample(total_5gb, &sample);
|
||||
let h2 = compute_midhash256_from_sample(total_5gb, &sample);
|
||||
assert_eq!(h1, h2, "same (size, sample) must hash identically");
|
||||
|
||||
let h3 = compute_midhash256_from_sample(total_10gb, &sample);
|
||||
assert_ne!(
|
||||
h1, h3,
|
||||
"size prefix must contribute to cid — same sample, different size"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn bt_v1_file_cid_wire_layout() {
|
||||
let infohash: [u8; 20] = [
|
||||
0xf9, 0xc8, 0xa7, 0xb6, 0xe5, 0xd4, 0xc3, 0xb2, 0xa1, 0x90, 0x8f, 0x7e, 0x6d, 0x5c,
|
||||
0x4b, 0x3a, 0x29, 0x18, 0x07, 0x06,
|
||||
];
|
||||
|
||||
let mut want = vec![0x01, 0x81, 0x20, 0x81, 0x20, 0x15];
|
||||
want.extend_from_slice(&infohash);
|
||||
want.push(0x04);
|
||||
assert_eq!(
|
||||
compute_bt_v1_file_cid(&infohash, 4),
|
||||
format!("b{}", base32_lower_no_padding(&want))
|
||||
);
|
||||
|
||||
let mut want0 = vec![0x01, 0x81, 0x20, 0x81, 0x20, 0x15];
|
||||
want0.extend_from_slice(&infohash);
|
||||
want0.push(0x00);
|
||||
assert_eq!(
|
||||
compute_bt_v1_file_cid(&infohash, 0),
|
||||
format!("b{}", base32_lower_no_padding(&want0))
|
||||
);
|
||||
|
||||
let mut want200 = vec![0x01, 0x81, 0x20, 0x81, 0x20, 0x16];
|
||||
want200.extend_from_slice(&infohash);
|
||||
want200.extend_from_slice(&[0xC8, 0x01]);
|
||||
assert_eq!(
|
||||
compute_bt_v1_file_cid(&infohash, 200),
|
||||
format!("b{}", base32_lower_no_padding(&want200))
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn bt_v2_file_cid_wire_layout() {
|
||||
let infohash: [u8; 32] = [0xAB; 32];
|
||||
let mut want = vec![0x01, 0x82, 0x20, 0x82, 0x20, 0x21];
|
||||
want.extend_from_slice(&infohash);
|
||||
want.push(0x00);
|
||||
assert_eq!(
|
||||
compute_bt_v2_file_cid(&infohash, 0),
|
||||
format!("b{}", base32_lower_no_padding(&want))
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn bt_file_cid_properties() {
|
||||
let ih1 = [0x11u8; 20];
|
||||
let ih2 = [0x22u8; 32];
|
||||
|
||||
assert_eq!(
|
||||
compute_bt_v1_file_cid(&ih1, 3),
|
||||
compute_bt_v1_file_cid(&ih1, 3)
|
||||
);
|
||||
assert_ne!(
|
||||
compute_bt_v1_file_cid(&ih1, 0),
|
||||
compute_bt_v1_file_cid(&ih1, 1)
|
||||
);
|
||||
assert!(compute_bt_v1_file_cid(&ih1, 0).starts_with('b'));
|
||||
assert!(compute_bt_v2_file_cid(&ih2, 0).starts_with('b'));
|
||||
assert_ne!(
|
||||
compute_bt_v1_file_cid(&[0xAB; 20], 0),
|
||||
compute_bt_v2_file_cid(&[0xAB; 32], 0)
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn nzb_release_cid_wire_and_properties() {
|
||||
let cid = compute_nzb_release_cid("api.nzb.life", "485c52f078b580667a01ac34a1cef6c2");
|
||||
assert!(cid.starts_with('b'));
|
||||
// Deterministic.
|
||||
assert_eq!(
|
||||
cid,
|
||||
compute_nzb_release_cid("api.nzb.life", "485c52f078b580667a01ac34a1cef6c2")
|
||||
);
|
||||
// Host-namespaced: same id, different host → different cid.
|
||||
assert_ne!(
|
||||
cid,
|
||||
compute_nzb_release_cid("api.nzbgeek.info", "485c52f078b580667a01ac34a1cef6c2")
|
||||
);
|
||||
// Explicit wire form: [01][84 20][84 20][20][sha256("host\nid")].
|
||||
let digest: [u8; 32] =
|
||||
Sha256::digest(b"api.nzb.life\n485c52f078b580667a01ac34a1cef6c2").into();
|
||||
let mut want = vec![0x01, 0x84, 0x20, 0x84, 0x20, 0x20];
|
||||
want.extend_from_slice(&digest);
|
||||
assert_eq!(cid, format!("b{}", base32_lower_no_padding(&want)));
|
||||
// Distinct codec from the bt-file codecs in the 0x100x neighborhood.
|
||||
assert_ne!(cid, compute_bt_v1_file_cid(&[0xAB; 20], 0));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn base32_known_vectors() {
|
||||
assert_eq!(base32_lower_no_padding(b""), "");
|
||||
assert_eq!(base32_lower_no_padding(b"f"), "my");
|
||||
assert_eq!(base32_lower_no_padding(b"fo"), "mzxq");
|
||||
assert_eq!(base32_lower_no_padding(b"foo"), "mzxw6");
|
||||
assert_eq!(base32_lower_no_padding(b"foob"), "mzxw6yq");
|
||||
assert_eq!(base32_lower_no_padding(b"foobar"), "mzxw6ytboi");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn ipfs_cid_single_leaf_matches_kubo() {
|
||||
assert_eq!(
|
||||
compute_ipfs_cid(b"hello world"),
|
||||
"bafkreifzjut3te2nhyekklss27nh3k72ysco7y32koao5eei66wof36n5e"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn ipfs_cid_empty_matches_kubo() {
|
||||
assert_eq!(
|
||||
compute_ipfs_cid(b""),
|
||||
"bafkreihdwdcefgh4dqkjv67uzcmw7ojee6xedzdetojuzjevtenxquvyku"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn ipfs_cid_multi_leaf_zeros_stable() {
|
||||
let mb_zeros = vec![0u8; 1024 * 1024];
|
||||
assert_eq!(
|
||||
compute_ipfs_cid(&mb_zeros),
|
||||
"bafybeiadh3bekpwtewjvauqeucf7yzqrb3ixsxzltnuwed4pxangtpou6m"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn ipfs_blocks_single_leaf() {
|
||||
let payload = b"hello world";
|
||||
let out = compute_ipfs_blocks(payload);
|
||||
assert_eq!(out.root, compute_ipfs_cid(payload));
|
||||
assert_eq!(out.blocks.len(), 1);
|
||||
assert_eq!(out.blocks[0].0, out.root);
|
||||
assert_eq!(out.blocks[0].1.as_ref(), payload);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn ipfs_blocks_multi_leaf() {
|
||||
let mb_zeros = vec![0u8; 1024 * 1024];
|
||||
let out = compute_ipfs_blocks(&mb_zeros);
|
||||
assert_eq!(
|
||||
out.root,
|
||||
"bafybeiadh3bekpwtewjvauqeucf7yzqrb3ixsxzltnuwed4pxangtpou6m"
|
||||
);
|
||||
assert_eq!(out.blocks.len(), 5);
|
||||
assert_eq!(out.blocks.last().unwrap().0, out.root);
|
||||
for (_cid, block) in &out.blocks[..4] {
|
||||
assert_eq!(block.len(), 256 * 1024);
|
||||
assert!(block.iter().all(|&b| b == 0));
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn ipfs_blocks_empty() {
|
||||
let out = compute_ipfs_blocks(b"");
|
||||
assert_eq!(
|
||||
out.root,
|
||||
"bafkreihdwdcefgh4dqkjv67uzcmw7ojee6xedzdetojuzjevtenxquvyku"
|
||||
);
|
||||
assert_eq!(out.blocks.len(), 1);
|
||||
assert_eq!(out.blocks[0].1.len(), 0);
|
||||
assert_eq!(out.blocks[0].0, out.root);
|
||||
}
|
||||
}
|
||||
@@ -1,81 +0,0 @@
|
||||
//! Language-code normalization — ISO 639 alias folding, **not** filename
|
||||
//! parsing.
|
||||
//!
|
||||
//! This lives apart from [`crate::filename_meta`] on purpose: it doesn't read a
|
||||
//! release title, it canonicalizes a *language code* (an indexer's declared
|
||||
//! language, a query's `languages:` filter, a title-derived stamp) onto the
|
||||
//! single ISO 639-2/B alpha-3 vocabulary the rest of the stack compares against.
|
||||
//! It is the one piece of the old `filename_meta` module that has no equivalent
|
||||
//! in the `metamesh-plugin-filename-parser` plugin — because it isn't filename
|
||||
//! parsing — so it stays in Rust while the title-parsing algorithm itself is
|
||||
//! owned by the plugin. Callers in the **request hot path** (e.g. the torznab
|
||||
//! indexer-language fan-out skip) need it synchronously, with no network hop.
|
||||
|
||||
/// Normalize a language code to the ISO 639-2/B alpha-3 vocabulary the stack
|
||||
/// emits (`fre`/`ger`/`chi`, **not** the 639-2/T `fra`/`deu`/`zho` nor the
|
||||
/// 639-1 two-letter forms), lowercased. This is the space meta-share's
|
||||
/// "languages I read" preference uses and that the title-derived language stamps
|
||||
/// write, so an operator who declares an indexer's language as `fra`/`de`/`zh`
|
||||
/// still lines up with a `fre`/`ger`/`chi` query filter. Unknown /
|
||||
/// already-canonical codes pass through lowercased+trimmed unchanged.
|
||||
pub fn normalize_lang_code(code: &str) -> String {
|
||||
let c = code.trim().to_ascii_lowercase();
|
||||
let mapped = match c.as_str() {
|
||||
// 639-2/T and 639-1 → 639-2/B for the codes that actually differ.
|
||||
"fra" | "fr" => "fre",
|
||||
"deu" | "de" => "ger",
|
||||
"zho" | "zh" => "chi",
|
||||
"nld" | "nl" => "dut",
|
||||
"ell" | "el" => "gre",
|
||||
"ron" | "ro" => "rum",
|
||||
"slk" | "sk" => "slo",
|
||||
"ces" | "cs" => "cze",
|
||||
// 639-1 → 639-2 for the rest of the language-tag output space (these
|
||||
// share the same /B and /T alpha-3, so only the two-letter form maps).
|
||||
"en" => "eng",
|
||||
"ja" => "jpn",
|
||||
"it" => "ita",
|
||||
"es" => "spa",
|
||||
"pt" => "por",
|
||||
"ko" => "kor",
|
||||
"ru" => "rus",
|
||||
"pl" => "pol",
|
||||
"hu" => "hun",
|
||||
"uk" => "ukr",
|
||||
"bg" => "bul",
|
||||
"sv" => "swe",
|
||||
"no" => "nor",
|
||||
"da" => "dan",
|
||||
"fi" => "fin",
|
||||
"tr" => "tur",
|
||||
"ar" => "ara",
|
||||
"hi" => "hin",
|
||||
"ta" => "tam",
|
||||
"te" => "tel",
|
||||
"ml" => "mal",
|
||||
"vi" => "vie",
|
||||
"th" => "tha",
|
||||
_ => return c,
|
||||
};
|
||||
mapped.to_string()
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn normalize_lang_maps_to_639_2b_vocabulary() {
|
||||
// 639-2/T and 639-1 spellings collapse onto the /B forms the stack uses.
|
||||
assert_eq!(normalize_lang_code("fra"), "fre");
|
||||
assert_eq!(normalize_lang_code("FR"), "fre");
|
||||
assert_eq!(normalize_lang_code("deu"), "ger");
|
||||
assert_eq!(normalize_lang_code("zh"), "chi");
|
||||
assert_eq!(normalize_lang_code("nld"), "dut");
|
||||
// Already-canonical / unknown codes pass through trimmed + lowercased.
|
||||
assert_eq!(normalize_lang_code(" ENG "), "eng");
|
||||
assert_eq!(normalize_lang_code("fre"), "fre");
|
||||
assert_eq!(normalize_lang_code("mul"), "mul");
|
||||
assert_eq!(normalize_lang_code("xyz"), "xyz");
|
||||
}
|
||||
}
|
||||
@@ -1,49 +0,0 @@
|
||||
//! `meta-feeder-sdk` — the source-agnostic foundation shared by the gateway
|
||||
//! core and every feeder sidecar binary.
|
||||
//!
|
||||
//! A *feeder* implements [`FeederPlugin`] (find records, fetch bytes) for one
|
||||
//! or more upstreams, and [`serve_feeders`] exposes them over the feeder HTTP
|
||||
//! contract. The gateway core consumes that contract via its
|
||||
//! `RemoteFeederPlugin` and keeps the libp2p wire, the bitswap blockstore, the
|
||||
//! hashing-into-the-blockstore, and the meta-core store-back to itself.
|
||||
//!
|
||||
//! Deliberately libp2p-free and blockstore-free — see this crate's `Cargo.toml`.
|
||||
|
||||
// `redb::Error` is a large enum; the cache wrappers (`cache.rs`) return it
|
||||
// directly rather than boxing on every embedded-DB call. Boxing each would be
|
||||
// pure noise for a single-process store, so silence the lint crate-wide.
|
||||
#![allow(clippy::result_large_err)]
|
||||
|
||||
/// pcs-convention default meta-core root a self-publishing feeder targets when
|
||||
/// neither its dashboard config nor the `META_CORE_URL` env seed sets one. Keeps
|
||||
/// the modular storage seam out of compose: a standard install just works; a
|
||||
/// non-standard meta-core is set in the feeder's config UI. Matches the
|
||||
/// `metacore-app` service the AppStore feeder apps ship on the shared network.
|
||||
pub const DEFAULT_META_CORE_URL: &str = "http://metacore-app:9000";
|
||||
|
||||
pub mod cache;
|
||||
pub mod common;
|
||||
pub mod config;
|
||||
pub mod enrich;
|
||||
pub mod filename_meta;
|
||||
pub mod hash;
|
||||
pub mod lang;
|
||||
pub mod meta_core;
|
||||
pub mod plugin;
|
||||
pub mod query;
|
||||
pub mod query_eval;
|
||||
pub mod serve;
|
||||
pub mod types;
|
||||
|
||||
pub use config::{ConfigField, ConfigSchema, FieldKind};
|
||||
pub use enrich::{EnrichTarget, EnrichmentConfig, Enricher};
|
||||
pub use meta_core::FeederStore;
|
||||
pub use plugin::{
|
||||
upstream_id_field, ConfigError, FeederPlugin, HashKind, HashOutcome, PluginRegistry,
|
||||
};
|
||||
pub use query::{GatewayQuery, GatewaySearchEvent, GatewayWireError, Negation, RangeFilter};
|
||||
pub use serve::{
|
||||
configure_plugins, router, serve_feeders, ComputeRequest, ComputeResponse, HashKindDto,
|
||||
HealthResponse, ManifestResponse, OutcomeDto, PluginManifest, QueryRequest, QueryResponse,
|
||||
};
|
||||
pub use types::{ByteStream, DiscoveryId, DiscoveryRecord, GatewayError, Hash, PluginHealth};
|
||||
@@ -1,293 +0,0 @@
|
||||
//! Lightweight meta-core write/read client for feeders that **self-publish**.
|
||||
//!
|
||||
//! Post-"smart feeder / dumb gateway" split: a feeder given a `META_CORE_URL`
|
||||
//! writes its own base record into meta-core instead of returning it for the
|
||||
//! gateway dispatcher to store. This keeps the gateway a pure aggregator +
|
||||
//! request router; the feeder owns the full record lifecycle for its upstream.
|
||||
//!
|
||||
//! The write shape is **byte-for-byte the same** as the gateway's
|
||||
//! `meta_core::build_metadata_body` (`crates/meta-gateway/src/meta_core.rs`):
|
||||
//! `cid_*` fields collapse to the `cids/<cid> = "true"` key-set, and a
|
||||
//! `provenance` JSON blob is stamped. Keeping them identical means a record
|
||||
//! written by the feeder is indistinguishable from one the dispatcher wrote —
|
||||
//! the meta-share ingester, meta-watch, and ranking all behave the same.
|
||||
//!
|
||||
//! `PUT /api/metadata/{cid}` is merge/upsert (confirmed: meta-core
|
||||
//! `SetMetadataFlat` sets only the fields in the payload, never deletes), so the
|
||||
//! feeder's base write and the enrichment plugins' later `PATCH /meta/{cid}`
|
||||
//! merges accumulate on the same `/file/{cid}` hash.
|
||||
|
||||
use std::collections::BTreeMap;
|
||||
use std::time::{SystemTime, UNIX_EPOCH};
|
||||
|
||||
use serde::Deserialize;
|
||||
use tracing::warn;
|
||||
|
||||
use crate::plugin::{HashKind, HashOutcome};
|
||||
use crate::types::{DiscoveryRecord, Hash};
|
||||
|
||||
/// Provenance recorded on every gateway-acquired record. Mirrors the gateway
|
||||
/// dispatcher's shape so feeder-written and dispatcher-written records are
|
||||
/// interchangeable.
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct Provenance {
|
||||
pub upstream: String,
|
||||
pub record_id: String,
|
||||
/// Stable per-deployment id (the gateway peer this feeder backs, from
|
||||
/// `META_GATEWAY_PEER_ID`, else the feeder hostname). Analytics group by it.
|
||||
pub gateway_peer: String,
|
||||
pub acquired_at: u64,
|
||||
}
|
||||
|
||||
impl Provenance {
|
||||
/// Build a provenance stamp for `upstream`/`record_id`, stamping "now".
|
||||
pub fn now(upstream: &str, record_id: &str, gateway_peer: &str) -> Self {
|
||||
Provenance {
|
||||
upstream: upstream.to_string(),
|
||||
record_id: record_id.to_string(),
|
||||
gateway_peer: gateway_peer.to_string(),
|
||||
acquired_at: SystemTime::now()
|
||||
.duration_since(UNIX_EPOCH)
|
||||
.map(|d| d.as_secs())
|
||||
.unwrap_or(0),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// JSON-stringified provenance value. Same key set + `"source":"gateway"` as the
|
||||
/// gateway dispatcher (`crates/meta-gateway/src/meta_core.rs::provenance_to_json`).
|
||||
pub fn provenance_to_json(p: &Provenance) -> String {
|
||||
serde_json::to_string(&serde_json::json!({
|
||||
"source": "gateway",
|
||||
"upstream": p.upstream,
|
||||
"recordId": p.record_id,
|
||||
"gatewayPeer": p.gateway_peer,
|
||||
"acquiredAt": p.acquired_at,
|
||||
}))
|
||||
.expect("static JSON shape — serialisation is infallible")
|
||||
}
|
||||
|
||||
/// Build the flat meta-core PUT body from a [`DiscoveryRecord`] + provenance.
|
||||
///
|
||||
/// Identical transform to the gateway dispatcher: any `cid_*` field becomes a
|
||||
/// `cids/<value> = "true"` key-set member (the CID is self-describing — the
|
||||
/// algorithm is the multicodec; see METADATA_KEYS.md §2/§14.13), everything else
|
||||
/// passes through, and `provenance` is stamped last (overriding any plugin-set
|
||||
/// value).
|
||||
pub fn build_metadata_body(
|
||||
record: &DiscoveryRecord,
|
||||
provenance: &Provenance,
|
||||
) -> BTreeMap<String, String> {
|
||||
let mut body: BTreeMap<String, String> = BTreeMap::new();
|
||||
for (k, v) in &record.fields {
|
||||
if k.starts_with("cid_") && !v.is_empty() {
|
||||
body.insert(format!("cids/{v}"), "true".to_string());
|
||||
} else {
|
||||
body.insert(k.clone(), v.clone());
|
||||
}
|
||||
}
|
||||
body.insert("provenance".to_string(), provenance_to_json(provenance));
|
||||
body
|
||||
}
|
||||
|
||||
/// `PUT /api/metadata/{cid}` — write (merge/upsert) the record body into
|
||||
/// meta-core. `base_url` is the meta-core root (e.g. `http://…-core:9000`).
|
||||
pub async fn put_record(
|
||||
http: &reqwest::Client,
|
||||
base_url: &str,
|
||||
cid: &str,
|
||||
body: &BTreeMap<String, String>,
|
||||
) -> anyhow::Result<()> {
|
||||
let url = format!("{}/api/metadata/{}", base_url.trim_end_matches('/'), cid);
|
||||
let resp = http.put(&url).json(body).send().await?;
|
||||
let status = resp.status();
|
||||
if !status.is_success() {
|
||||
let text = resp.text().await.unwrap_or_default();
|
||||
anyhow::bail!("meta-core PUT {url} returned {status}: {text}");
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Shape of meta-core's `GET /meta/{cid}` response (`{ hashId, metadata }`).
|
||||
#[derive(Debug, Deserialize)]
|
||||
struct MetaGetResponse {
|
||||
#[serde(default)]
|
||||
metadata: Option<BTreeMap<String, String>>,
|
||||
}
|
||||
|
||||
/// `GET /meta/{cid}` — read the current flat field map for a record, or `None`
|
||||
/// when meta-core has no record yet (404). Used to read back a plugin's merged
|
||||
/// fields (e.g. filename-parser's `originalTitle`/`movieYear`) before handing
|
||||
/// them to the next plugin in the pipeline.
|
||||
pub async fn get_record(
|
||||
http: &reqwest::Client,
|
||||
base_url: &str,
|
||||
cid: &str,
|
||||
) -> anyhow::Result<Option<BTreeMap<String, String>>> {
|
||||
let url = format!("{}/meta/{}", base_url.trim_end_matches('/'), cid);
|
||||
let resp = http.get(&url).send().await?;
|
||||
if resp.status() == reqwest::StatusCode::NOT_FOUND {
|
||||
return Ok(None);
|
||||
}
|
||||
if !resp.status().is_success() {
|
||||
anyhow::bail!("meta-core GET {url} returned {}", resp.status());
|
||||
}
|
||||
let parsed: MetaGetResponse = resp.json().await?;
|
||||
Ok(parsed.metadata)
|
||||
}
|
||||
|
||||
/// Self-publishing store for feeders that own their record lifecycle (D4):
|
||||
/// the feeder writes its own **metadata-only** records to meta-core and returns
|
||||
/// sparse outcomes so the gateway dispatcher skips them; the meta-share ingester
|
||||
/// Kamilata-propagates them.
|
||||
///
|
||||
/// **Byte outcomes are deliberately NOT self-published.** A feeder-side WebDAV
|
||||
/// write to the gateway core would be seeded by nobody: the gateway core runs
|
||||
/// with `ENABLE_FILE_WATCHER=false` (so meta-core emits no `/api/events/files`
|
||||
/// for it, and the ingester's `ipfs_seed` — which seeds only watched files —
|
||||
/// never sees it), and a sparse outcome makes the gateway skip its own
|
||||
/// blockstore seed. So byte-bearing outcomes (torznab full-file / subtitles)
|
||||
/// pass through unchanged and the **gateway** continues to WebDAV-PUT + bitswap-
|
||||
/// seed them (it's the libp2p/bitswap host). Result: feeder owns the metadata
|
||||
/// records (the Kamilata-propagated bulk), gateway owns byte delivery — a clean
|
||||
/// split with no seeding regression. Tribler is metadata-only and uses
|
||||
/// [`crate::enrich::Enricher`]; this is torznab's record-owning counterpart.
|
||||
#[derive(Clone)]
|
||||
pub struct FeederStore {
|
||||
http: reqwest::Client,
|
||||
meta_core_url: String,
|
||||
gateway_peer: String,
|
||||
}
|
||||
|
||||
impl FeederStore {
|
||||
pub fn new(http: reqwest::Client, meta_core_url: String, gateway_peer: String) -> Self {
|
||||
FeederStore {
|
||||
http,
|
||||
meta_core_url: meta_core_url.trim_end_matches('/').to_string(),
|
||||
gateway_peer,
|
||||
}
|
||||
}
|
||||
|
||||
/// Build from the feeder env. `None` when `META_CORE_URL` is unset (the
|
||||
/// feeder then falls back to the legacy "return outcomes, gateway stores"
|
||||
/// path). `gateway_peer` ← `META_GATEWAY_PEER_ID` / `HOSTNAME`.
|
||||
pub fn from_env() -> Option<Self> {
|
||||
let meta_core_url = env_nonempty("META_CORE_URL")?;
|
||||
let gateway_peer = env_nonempty("META_GATEWAY_PEER_ID")
|
||||
.or_else(|| env_nonempty("HOSTNAME"))
|
||||
.unwrap_or_else(|| "gateway-feeder".to_string());
|
||||
let http = crate::common::build_http_client(30, "meta-feeder-store", None);
|
||||
Some(Self::new(http, meta_core_url, gateway_peer))
|
||||
}
|
||||
|
||||
pub fn meta_core_url(&self) -> &str {
|
||||
&self.meta_core_url
|
||||
}
|
||||
|
||||
/// Self-publish a **metadata-only** outcome to meta-core, returning the
|
||||
/// **sparse** outcome (`record: None`) so the gateway's `(_, None)` branch
|
||||
/// skips it — no double write. Byte outcomes and already-sparse outcomes are
|
||||
/// returned unchanged (the gateway stores + seeds the bytes; see the struct
|
||||
/// docs). On a write failure the original outcome is returned so the gateway
|
||||
/// stores it (graceful degradation, never lost).
|
||||
pub async fn publish(
|
||||
&self,
|
||||
upstream_id: &str,
|
||||
record_id: &str,
|
||||
outcome: HashOutcome,
|
||||
) -> HashOutcome {
|
||||
// Only self-publish metadata-only outcomes (bytes None, record Some).
|
||||
if outcome.bytes.is_some() || outcome.record.is_none() {
|
||||
return outcome;
|
||||
}
|
||||
let cid = outcome.hash.0.clone();
|
||||
let record = outcome.record.as_ref().expect("record is Some by the guard");
|
||||
match self
|
||||
.store_metadata_only(upstream_id, record_id, &cid, record)
|
||||
.await
|
||||
{
|
||||
Ok(()) => sparse(outcome.hash, outcome.hash_kind),
|
||||
Err(e) => {
|
||||
warn!(target: "meta-feeder::store", cid = %cid, error = %e,
|
||||
"metadata self-publish failed; falling back to gateway store");
|
||||
outcome
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Metadata-only self-publish: `PUT /api/metadata/{cid}` with provenance.
|
||||
async fn store_metadata_only(
|
||||
&self,
|
||||
upstream_id: &str,
|
||||
record_id: &str,
|
||||
cid: &str,
|
||||
record: &DiscoveryRecord,
|
||||
) -> anyhow::Result<()> {
|
||||
let prov = Provenance::now(upstream_id, record_id, &self.gateway_peer);
|
||||
let body = build_metadata_body(record, &prov);
|
||||
put_record(&self.http, &self.meta_core_url, cid, &body).await
|
||||
}
|
||||
}
|
||||
|
||||
fn sparse(hash: Hash, hash_kind: HashKind) -> HashOutcome {
|
||||
HashOutcome {
|
||||
hash,
|
||||
hash_kind,
|
||||
bytes: None,
|
||||
record: None,
|
||||
file_extension: None,
|
||||
}
|
||||
}
|
||||
|
||||
fn env_nonempty(key: &str) -> Option<String> {
|
||||
std::env::var(key)
|
||||
.ok()
|
||||
.map(|s| s.trim().to_string())
|
||||
.filter(|s| !s.is_empty())
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
fn rec() -> DiscoveryRecord {
|
||||
DiscoveryRecord {
|
||||
upstream_id: "tribler".into(),
|
||||
record_id: "abc".into(),
|
||||
fields: BTreeMap::from_iter([
|
||||
("title".into(), "Big Buck Bunny".into()),
|
||||
("cid_btih_v1_file".into(), "bafkPERFILE".into()),
|
||||
]),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn provenance_shape_matches_gateway() {
|
||||
let p = Provenance {
|
||||
upstream: "tribler".into(),
|
||||
record_id: "abc".into(),
|
||||
gateway_peer: "metagateway-tribler".into(),
|
||||
acquired_at: 1_747_504_000,
|
||||
};
|
||||
let v: serde_json::Value = serde_json::from_str(&provenance_to_json(&p)).unwrap();
|
||||
assert_eq!(v["source"], "gateway");
|
||||
assert_eq!(v["upstream"], "tribler");
|
||||
assert_eq!(v["recordId"], "abc");
|
||||
assert_eq!(v["gatewayPeer"], "metagateway-tribler");
|
||||
assert_eq!(v["acquiredAt"], 1_747_504_000_u64);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn body_collapses_cid_fields_and_stamps_provenance() {
|
||||
let p = Provenance::now("tribler", "abc", "peer");
|
||||
let body = build_metadata_body(&rec(), &p);
|
||||
assert_eq!(body.get("title").map(String::as_str), Some("Big Buck Bunny"));
|
||||
// cid_* collapsed to cids/<value>=true; no cid_* survives.
|
||||
assert!(body.keys().all(|k| !k.starts_with("cid_")));
|
||||
assert_eq!(
|
||||
body.get("cids/bafkPERFILE").map(String::as_str),
|
||||
Some("true")
|
||||
);
|
||||
assert!(body.contains_key("provenance"));
|
||||
}
|
||||
}
|
||||
@@ -1,234 +0,0 @@
|
||||
//! The `FeederPlugin` trait + supporting types.
|
||||
//!
|
||||
//! A feeder plugin bridges one external upstream (`gutenberg`, `arxiv`,
|
||||
//! `torznab`, …) into the MetaMesh network. It **finds** records and
|
||||
//! **fetches** bytes; it does NOT hash-into-a-blockstore, store back into
|
||||
//! meta-core, or speak libp2p — those stay in the gateway core. This is why
|
||||
//! `FeederPlugin` has no `set_blockstore` / `set_tmdb_budget` hooks (the
|
||||
//! gateway's `GatewayPlugin` carried them): preview-seeding moves to the core,
|
||||
//! and a feeder that needs a TMDB budget owns it internally.
|
||||
//!
|
||||
//! The `serve_feeders` harness ([`crate::serve`]) wraps any set of
|
||||
//! `FeederPlugin`s in the feeder HTTP contract; the gateway core's
|
||||
//! `RemoteFeederPlugin` is the matching HTTP client.
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::path::Path;
|
||||
|
||||
use async_trait::async_trait;
|
||||
use futures::stream::BoxStream;
|
||||
|
||||
// Re-exported so plugins can `use meta_feeder_sdk::plugin::GatewayQuery` (the
|
||||
// gateway's original `plugin.rs` re-exported it the same way).
|
||||
pub use crate::query::GatewayQuery;
|
||||
use crate::query::GatewaySearchEvent;
|
||||
use crate::types::{ByteStream, DiscoveryRecord, GatewayError, Hash, PluginHealth};
|
||||
|
||||
/// One element of what [`FeederPlugin::compute_outcomes`] returns.
|
||||
///
|
||||
/// The `hash` is the value shipped to the gateway core. The other fields are
|
||||
/// signals the core's auto-store uses to persist the content into meta-core.
|
||||
/// The core's three-branch auto-store routes on `(bytes, record)`:
|
||||
/// - `(Some, Some)` — full store: WebDAV PUT + metadata PUT with `filePath`.
|
||||
/// - `(None, Some)` — metadata-only PUT (no local bytes; e.g. torznab samples
|
||||
/// the middle 1 MiB of a multi-GiB video and surfaces only the record).
|
||||
/// - `(_, None)` — skip. Cache hits land here.
|
||||
///
|
||||
/// `file_extension` MUST be `None` when `bytes` is `None`.
|
||||
#[derive(Debug)]
|
||||
pub struct HashOutcome {
|
||||
pub hash: Hash,
|
||||
/// Which hash family this CID belongs to. Drives the core's routing —
|
||||
/// the bitswap blockstore seed only fires for `Sha2_256`.
|
||||
pub hash_kind: HashKind,
|
||||
pub bytes: Option<bytes::Bytes>,
|
||||
pub record: Option<DiscoveryRecord>,
|
||||
/// File extension to append when naming the WebDAV-side blob (e.g.
|
||||
/// `"epub"`). No leading dot. MUST be `None` when `bytes` is `None`.
|
||||
pub file_extension: Option<String>,
|
||||
}
|
||||
|
||||
/// Hash family discriminator on [`HashOutcome`].
|
||||
///
|
||||
/// - `Midhash256` — size-prefix-plus-middle-1MB-sample, custom multicodec
|
||||
/// `0x1000`. Fast, MetaMesh-internal, not retrievable via public IPFS.
|
||||
/// - `Sha2_256` — standard IPFS CIDv1 over full bytes. Retrievable via bitswap
|
||||
/// once the bytes are in the core's blockstore. What every shipping plugin
|
||||
/// today emits.
|
||||
/// - `BtV1File` — a single file inside a BitTorrent v1 torrent, custom
|
||||
/// multicodec `0x1001`. Locator, not content hash: opaque to bitswap.
|
||||
/// - `NzbRelease` — a Usenet release (a Newznab listing), custom multicodec
|
||||
/// `nzb-release` `0x1004`. Locator like `BtV1File`: opaque to bitswap,
|
||||
/// redeemed to bytes only by a credentialed meta-share peer, which grabs the
|
||||
/// `.nzb` via the `{base, id}` descriptor stored at `nzb:<cid>` KV.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub enum HashKind {
|
||||
/// Reserved: no feeder currently emits midhash256 outcomes, but the family
|
||||
/// is real across meta-core / meta-share, so the variant stays.
|
||||
#[allow(dead_code)]
|
||||
Midhash256,
|
||||
Sha2_256,
|
||||
BtV1File,
|
||||
NzbRelease,
|
||||
}
|
||||
|
||||
/// Startup-only configuration error returned by [`FeederPlugin::configure`].
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum ConfigError {
|
||||
#[error("plugin {plugin}: required config missing: {what}")]
|
||||
MissingConfig {
|
||||
plugin: &'static str,
|
||||
/// Human-readable description of the missing setting and where to
|
||||
/// supply it. Used by the harness's soft-skip warning.
|
||||
what: &'static str,
|
||||
},
|
||||
|
||||
#[error("plugin {plugin}: cache dir setup failed: {source}")]
|
||||
CacheSetup {
|
||||
plugin: &'static str,
|
||||
#[source]
|
||||
source: std::io::Error,
|
||||
},
|
||||
|
||||
#[error("plugin {plugin}: {source}")]
|
||||
Other {
|
||||
plugin: &'static str,
|
||||
#[source]
|
||||
source: anyhow::Error,
|
||||
},
|
||||
}
|
||||
|
||||
/// Canonical field name for an upstream's stable per-record identifier.
|
||||
/// Every plugin's `DiscoveryRecord.fields` MUST include this key
|
||||
/// (`gutenbergid:11`, `tmdbid:31910`, `arxivid:2106.07447`). The rule is
|
||||
/// `format!("{upstream_id}id")` — lowercase, no separator.
|
||||
pub fn upstream_id_field(upstream_id: &str) -> String {
|
||||
format!("{upstream_id}id")
|
||||
}
|
||||
|
||||
/// Key-set prefix for the provenance `source/<label>` field (METADATA_KEYS §5).
|
||||
/// Each member is its own hash field (`source/gateway:nyaa.si = "true"`) so two
|
||||
/// producers that reach the same content by different routes union without a
|
||||
/// last-writer-wins clobber. The `<label>` uses `:` internally (not `/`, which
|
||||
/// is the key-set path separator) — e.g. `gateway:nyaa.si`, `gateway:tribler`.
|
||||
pub const SOURCE_KEYSET_PREFIX: &str = "source/";
|
||||
|
||||
/// Stamp the SDK's default provenance member — `source/gateway:<upstream_id>` —
|
||||
/// onto a record's fields, but **only if it carries no `source/*` member yet**.
|
||||
/// This gives every gateway hit a provenance label for free (so meta-watch never
|
||||
/// falls back to "unknown" for a feeder record), while a plugin that knows a
|
||||
/// finer origin (torznab → the specific Prowlarr indexer) stamps its own
|
||||
/// `source/gateway:<indexer>` first and this default then no-ops.
|
||||
pub fn stamp_default_source(
|
||||
fields: &mut std::collections::BTreeMap<String, String>,
|
||||
upstream_id: &str,
|
||||
) {
|
||||
if fields.keys().any(|k| k.starts_with(SOURCE_KEYSET_PREFIX)) {
|
||||
return;
|
||||
}
|
||||
fields.insert(
|
||||
format!("{SOURCE_KEYSET_PREFIX}gateway:{upstream_id}"),
|
||||
"true".to_string(),
|
||||
);
|
||||
}
|
||||
|
||||
/// Static plugin contract. Each enabled `upstream_id` is implemented by
|
||||
/// exactly one `Box<dyn FeederPlugin>`. Lives behind `&self` in steady state —
|
||||
/// called concurrently, owns any interior mutability for caches.
|
||||
///
|
||||
/// **Field-naming convention.** Every record a plugin returns MUST include the
|
||||
/// canonical `<upstream_id>id` field — see [`upstream_id_field`].
|
||||
#[async_trait]
|
||||
pub trait FeederPlugin: Send + Sync + 'static {
|
||||
fn upstream_id(&self) -> &'static str;
|
||||
|
||||
fn configure(&mut self, cache_dir: &Path) -> Result<(), ConfigError>;
|
||||
|
||||
async fn handle_query(
|
||||
&self,
|
||||
query: &GatewayQuery,
|
||||
max_results: usize,
|
||||
) -> Result<Vec<DiscoveryRecord>, GatewayError>;
|
||||
|
||||
/// Streaming variant of [`handle_query`]. The default impl collects
|
||||
/// [`handle_query`] and replays it as `Base*` + `Done` — correct for every
|
||||
/// plugin with no incremental enrichment (gutenberg, arxiv, …). torznab
|
||||
/// overrides it to decouple base records from rate-limited TMDB enrichment.
|
||||
async fn handle_query_stream(
|
||||
&self,
|
||||
query: &GatewayQuery,
|
||||
max_results: usize,
|
||||
) -> Result<BoxStream<'static, GatewaySearchEvent>, GatewayError> {
|
||||
let records = self.handle_query(query, max_results).await?;
|
||||
let events = records
|
||||
.into_iter()
|
||||
.map(GatewaySearchEvent::Base)
|
||||
.chain(std::iter::once(GatewaySearchEvent::Done));
|
||||
Ok(Box::pin(futures::stream::iter(events)))
|
||||
}
|
||||
|
||||
/// Resolve `record_id` into one or more content-addressed outcomes.
|
||||
///
|
||||
/// - `Err(_)` — the upstream record itself is unresolvable.
|
||||
/// - `Ok(vec![])` — resolved but yielded zero outcomes.
|
||||
/// - `Ok(partial_vec)` — bundle-success: per-sibling failures are
|
||||
/// best-effort (log + drop), surface whatever succeeded.
|
||||
async fn compute_outcomes(&self, record_id: &str) -> Result<Vec<HashOutcome>, GatewayError>;
|
||||
|
||||
async fn handle_fetch(&self, _record_id: &str) -> Result<Option<ByteStream>, GatewayError> {
|
||||
Ok(None)
|
||||
}
|
||||
|
||||
fn health(&self) -> PluginHealth {
|
||||
PluginHealth::Ok
|
||||
}
|
||||
|
||||
fn served_file_types(&self) -> &'static [&'static str] {
|
||||
&[]
|
||||
}
|
||||
|
||||
fn served_content_kinds(&self) -> &'static [&'static str] {
|
||||
&[]
|
||||
}
|
||||
|
||||
/// Self-describing config schema. The feeder SDK serves this at
|
||||
/// `GET /config/schema` and renders a generic form from it — the gateway and
|
||||
/// UI carry no per-plugin field knowledge. Default: no configuration.
|
||||
fn config_schema(&self) -> crate::config::ConfigSchema {
|
||||
crate::config::ConfigSchema::default()
|
||||
}
|
||||
|
||||
/// The plugin's current *effective* config as JSON, **unredacted** (secrets
|
||||
/// included). The SDK redacts it before serving `GET /config/values` and uses
|
||||
/// it as the merge base on the first save (before any `config.json` exists).
|
||||
/// Default: empty object. Implementors should mirror the keys in
|
||||
/// [`config_schema`](FeederPlugin::config_schema).
|
||||
fn config_values(&self) -> serde_json::Value {
|
||||
serde_json::json!({})
|
||||
}
|
||||
|
||||
async fn get_blob(&self, _cid: &str) -> Option<Vec<u8>> {
|
||||
None
|
||||
}
|
||||
}
|
||||
|
||||
/// Static plugin registry. Built once at startup; read-only thereafter.
|
||||
/// Keyed by `upstream_id`.
|
||||
pub type PluginRegistry = HashMap<&'static str, Box<dyn FeederPlugin>>;
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn config_error_messages_include_plugin_name() {
|
||||
let e = ConfigError::MissingConfig {
|
||||
plugin: "scihub",
|
||||
what:
|
||||
"mirrors (set them in the dashboard or gateway-config.json plugins.scihub.mirrors)",
|
||||
};
|
||||
let msg = e.to_string();
|
||||
assert!(msg.contains("scihub"));
|
||||
assert!(msg.contains("mirrors"));
|
||||
}
|
||||
}
|
||||
@@ -1,147 +0,0 @@
|
||||
//! Structured query envelope + streaming-search event + wire-error types,
|
||||
//! shared by feeders (their HTTP contract) and the gateway core (its libp2p
|
||||
//! wire). These are pure serde data types — no libp2p, no transport.
|
||||
//!
|
||||
//! On the core side, `protocol.rs` re-uses these exact structs as the bincode
|
||||
//! wire shape; the `wire_parity_*` tests there pin the byte encoding. Keep
|
||||
//! field order / variant order / derive shape stable.
|
||||
|
||||
use std::collections::BTreeMap;
|
||||
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
use crate::types::{DiscoveryRecord, GatewayError};
|
||||
|
||||
/// Structured query envelope. Gateway plugins do best-effort pushdown to
|
||||
/// upstream APIs with native query languages (`torznab cat=movie`,
|
||||
/// `arxiv id_list=`).
|
||||
///
|
||||
/// **Authority rule:** `filters` / `ranges` / `negations` are authoritative;
|
||||
/// `raw_text` is informational. A plugin that consumes both will double-filter
|
||||
/// and under-match — pick one source per filter; structured wins.
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub struct GatewayQuery {
|
||||
/// Original user text (e.g. `"naruto contentKind:movie imdbid:tt0903747"`).
|
||||
/// Verbatim — never mutated by the conversion.
|
||||
pub raw_text: String,
|
||||
|
||||
/// Bare-word leaves only, space-joined. The plugin's primary `q=` input
|
||||
/// for upstream search APIs that take a single search box.
|
||||
pub free_text: String,
|
||||
|
||||
/// Structured filters keyed by field name (raw, unstemmed values).
|
||||
/// Multiple values per key = OR within a key; AND across keys.
|
||||
pub filters: BTreeMap<String, Vec<String>>,
|
||||
|
||||
/// Numeric range filters (e.g. `movieYear:2020..2024`). Both bounds
|
||||
/// optional → unbounded on that side.
|
||||
pub ranges: Vec<RangeFilter>,
|
||||
|
||||
/// Negation filters (e.g. `NOT genres:horror`, bare `NOT word`).
|
||||
pub negations: Vec<Negation>,
|
||||
}
|
||||
|
||||
/// Numeric range filter inside [`GatewayQuery::ranges`]. Both bounds are
|
||||
/// `Option<i64>` so open-ended ranges drop one bound to `None`.
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub struct RangeFilter {
|
||||
pub field: String,
|
||||
pub lo: Option<i64>,
|
||||
pub hi: Option<i64>,
|
||||
}
|
||||
|
||||
/// Negation entry inside [`GatewayQuery::negations`]. `field` is `Some(name)`
|
||||
/// for typed negations (`NOT genres:horror`), `None` for bare-word negation.
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub struct Negation {
|
||||
pub field: Option<String>,
|
||||
pub value: String,
|
||||
}
|
||||
|
||||
impl GatewayQuery {
|
||||
/// Bare-text input to hand to a free-form upstream search API. Falls back
|
||||
/// to `"*"` when the conversion produced no free-text leaves so upstreams
|
||||
/// that 4xx on empty search return their top-ranked results instead.
|
||||
pub fn free_text_or_star(&self) -> &str {
|
||||
if self.free_text.trim().is_empty() {
|
||||
"*"
|
||||
} else {
|
||||
self.free_text.as_str()
|
||||
}
|
||||
}
|
||||
|
||||
/// Construct a free-text-only `GatewayQuery` for in-process use (tests,
|
||||
/// debug routes). `raw_text` == `free_text`; structured fields stay empty.
|
||||
pub fn from_free_text(text: impl Into<String>) -> Self {
|
||||
let text = text.into();
|
||||
Self {
|
||||
raw_text: text.clone(),
|
||||
free_text: text,
|
||||
filters: BTreeMap::new(),
|
||||
ranges: Vec::new(),
|
||||
negations: Vec::new(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// One frame in the streaming Search response.
|
||||
///
|
||||
/// A plugin emits `Base` records as soon as they're discovered, then
|
||||
/// best-effort `EnrichPatch`/`Drop` events as enrichment lands, then a
|
||||
/// terminal `Done`. The default `handle_query_stream` collects `handle_query`
|
||||
/// and replays it as `Base*` + `Done`.
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub enum GatewaySearchEvent {
|
||||
/// An un-enriched discovery record.
|
||||
Base(DiscoveryRecord),
|
||||
/// Incremental enrichment for a previously-sent `Base`, keyed by
|
||||
/// `record_id`. `set` fields are inserted/overwritten; `remove` deleted.
|
||||
EnrichPatch {
|
||||
record_id: String,
|
||||
set: BTreeMap<String, String>,
|
||||
remove: Vec<String>,
|
||||
},
|
||||
/// Retract a previously-emitted `Base` record entirely.
|
||||
Drop { record_id: String },
|
||||
/// Terminal success — no more frames follow.
|
||||
Done,
|
||||
/// Terminal error — no more frames follow.
|
||||
Error(GatewayWireError),
|
||||
}
|
||||
|
||||
/// Wire-stable error shape. Distinct from in-process [`GatewayError`] because
|
||||
/// adding a variant here is a wire-format change.
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize, thiserror::Error)]
|
||||
pub enum GatewayWireError {
|
||||
#[error("not found")]
|
||||
NotFound,
|
||||
#[error("rate limited; retry in {retry_after_s}s")]
|
||||
RateLimited { retry_after_s: u32 },
|
||||
#[error("upstream transient: {0}")]
|
||||
UpstreamTransient(String),
|
||||
#[error("upstream permanent: {0}")]
|
||||
UpstreamPermanent(String),
|
||||
/// The protocol was registered but no plugin claims this `upstream_id`.
|
||||
#[error("plugin not loaded")]
|
||||
PluginNotLoaded,
|
||||
/// Internal codec / decode failure on the *receiver* side.
|
||||
#[error("internal: {0}")]
|
||||
Internal(String),
|
||||
}
|
||||
|
||||
impl From<GatewayError> for GatewayWireError {
|
||||
fn from(e: GatewayError) -> Self {
|
||||
match e {
|
||||
GatewayError::Transient(s) => GatewayWireError::UpstreamTransient(s),
|
||||
GatewayError::Permanent(s) => GatewayWireError::UpstreamPermanent(s),
|
||||
GatewayError::RateLimited { retry_after_s } => {
|
||||
GatewayWireError::RateLimited { retry_after_s }
|
||||
}
|
||||
GatewayError::NotFound => GatewayWireError::NotFound,
|
||||
// Internal errors are never leaked; map to a transient.
|
||||
GatewayError::Internal(_) => {
|
||||
GatewayWireError::UpstreamTransient("internal".to_string())
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,456 +0,0 @@
|
||||
//! Structured-filter evaluation for gateway records.
|
||||
//!
|
||||
//! Two pure helpers consuming the [`crate::query::GatewayQuery`] struct:
|
||||
//!
|
||||
//! - [`record_matches`] — does a single record's `fields` map satisfy the
|
||||
//! query's `filters` + `ranges` + typed `negations`?
|
||||
//! - [`query_accepts_plugin`] — could a plugin advertising the given
|
||||
//! `served_file_types` / `served_content_kinds` possibly produce a record
|
||||
//! that satisfies the query? Used by plugins as a cheap early-return.
|
||||
//!
|
||||
//! The dispatcher applies `record_matches` to every plugin's return value, and
|
||||
//! meta-share applies the same algorithm consumer-side to defend against stale
|
||||
//! gateway peers — both tiers must agree on outcomes for any (record, query)
|
||||
//! pair. See the original gateway crate docs for the full semantics.
|
||||
|
||||
use std::collections::BTreeMap;
|
||||
|
||||
use crate::query::GatewayQuery;
|
||||
|
||||
/// True iff a record's `fields` satisfies every filter, range, and typed
|
||||
/// negation in `query`.
|
||||
pub fn record_matches(fields: &BTreeMap<String, String>, query: &GatewayQuery) -> bool {
|
||||
for (key, allowed) in &query.filters {
|
||||
if key == "languages" {
|
||||
if !languages_filter_matches(fields, allowed) {
|
||||
return false;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
let Some(field_value) = fields.get(key) else {
|
||||
return false;
|
||||
};
|
||||
if !any_csv_value_matches(field_value, allowed) {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
for range in &query.ranges {
|
||||
let Some(field_value) = fields.get(&range.field) else {
|
||||
return false;
|
||||
};
|
||||
let Ok(n) = field_value.parse::<i64>() else {
|
||||
return false;
|
||||
};
|
||||
if let Some(lo) = range.lo {
|
||||
if n < lo {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
if let Some(hi) = range.hi {
|
||||
if n > hi {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
for negation in &query.negations {
|
||||
let Some(name) = negation.field.as_deref() else {
|
||||
// Bare-word negation — not enforced here.
|
||||
continue;
|
||||
};
|
||||
let Some(field_value) = fields.get(name) else {
|
||||
// Missing field ⇒ negation passes (nothing to exclude).
|
||||
continue;
|
||||
};
|
||||
if any_csv_value_matches(field_value, std::slice::from_ref(&negation.value)) {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
true
|
||||
}
|
||||
|
||||
/// True iff a plugin advertising the given `served_file_types` /
|
||||
/// `served_content_kinds` could possibly produce a record satisfying
|
||||
/// `query.filters["fileType" | "contentKind"]`.
|
||||
pub fn query_accepts_plugin(
|
||||
query: &GatewayQuery,
|
||||
served_file_types: &[&str],
|
||||
served_content_kinds: &[&str],
|
||||
) -> bool {
|
||||
accepts_axis(query.filters.get("fileType"), served_file_types)
|
||||
&& accepts_axis(query.filters.get("contentKind"), served_content_kinds)
|
||||
}
|
||||
|
||||
fn accepts_axis(requested: Option<&Vec<String>>, served: &[&str]) -> bool {
|
||||
let Some(requested) = requested else {
|
||||
return true;
|
||||
};
|
||||
if requested.is_empty() {
|
||||
return true;
|
||||
}
|
||||
// Wildcard: a plugin that advertises `"*"` on an axis serves every value on
|
||||
// that axis (e.g. tribler is a free-text overlay search with no inherent
|
||||
// type limitation). It accepts any requested fileType/contentKind.
|
||||
if served.contains(&"*") {
|
||||
return true;
|
||||
}
|
||||
requested
|
||||
.iter()
|
||||
.any(|v| served.iter().any(|s| s.eq_ignore_ascii_case(v)))
|
||||
}
|
||||
|
||||
/// Resolve a `languages` filter against the key-set storage form
|
||||
/// (`languages/<lang3> = "true"`). **Fails open** on anything we can't
|
||||
/// confidently exclude. A record matches when:
|
||||
/// 1. it carries a requested code as a truthy key-set member (OR across codes);
|
||||
/// 2. a legacy flat `languages` csv field contains a requested code;
|
||||
/// 3. it carries `mul` (multiple — may contain a requested one) or `und`
|
||||
/// (undetermined); or
|
||||
/// 4. it carries no language information at all.
|
||||
///
|
||||
/// Only a record with a *concrete*, non-matching language (e.g. `jpn`-only
|
||||
/// under an `eng` filter) is dropped. Kept byte-identical in logic with
|
||||
/// meta-search's copy (and the client's `passesLangFilter`) — language narrows
|
||||
/// the result set without ever hiding unknown/undetermined/multi releases.
|
||||
fn languages_filter_matches(fields: &BTreeMap<String, String>, allowed: &[String]) -> bool {
|
||||
for code in allowed {
|
||||
let key = format!("languages/{}", code.trim().to_ascii_lowercase());
|
||||
if fields.get(&key).is_some_and(|v| is_truthy_member(v)) {
|
||||
return true;
|
||||
}
|
||||
}
|
||||
if let Some(flat) = fields.get("languages") {
|
||||
if any_csv_value_matches(flat, allowed) {
|
||||
return true;
|
||||
}
|
||||
}
|
||||
// Fail open: `mul`/`und` are explicit "could be anything / unknown" markers,
|
||||
// and a record with no concrete language at all is equally unknown. Keep all
|
||||
// three; only a specific, non-matching language reaches the `false` below.
|
||||
if fields.get("languages/mul").is_some_and(|v| is_truthy_member(v))
|
||||
|| fields.get("languages/und").is_some_and(|v| is_truthy_member(v))
|
||||
{
|
||||
return true;
|
||||
}
|
||||
!record_has_concrete_language(fields)
|
||||
}
|
||||
|
||||
/// True iff the record states a *concrete* language: a truthy `languages/<code>`
|
||||
/// key-set member other than the `und`/`mul` non-codes, or a non-empty legacy
|
||||
/// flat `languages` field. Distinguishes "a specific language that didn't match
|
||||
/// ⇒ drop" from "no language info ⇒ keep" for [`languages_filter_matches`].
|
||||
fn record_has_concrete_language(fields: &BTreeMap<String, String>) -> bool {
|
||||
if fields.get("languages").is_some_and(|v| !v.trim().is_empty()) {
|
||||
return true;
|
||||
}
|
||||
fields.iter().any(|(k, v)| {
|
||||
is_truthy_member(v)
|
||||
&& k.starts_with("languages/")
|
||||
&& k != "languages/und"
|
||||
&& k != "languages/mul"
|
||||
})
|
||||
}
|
||||
|
||||
/// Key-set members are written as the literal `"true"`; tolerate a couple of
|
||||
/// obvious truthy spellings and treat anything else as absent.
|
||||
fn is_truthy_member(v: &str) -> bool {
|
||||
matches!(v.trim().to_ascii_lowercase().as_str(), "true" | "1" | "yes")
|
||||
}
|
||||
|
||||
/// True iff `field_value` (treated as a csv set when it contains commas)
|
||||
/// contains at least one case-insensitive match for any item in `allowed`.
|
||||
fn any_csv_value_matches(field_value: &str, allowed: &[String]) -> bool {
|
||||
for piece in field_value.split(',') {
|
||||
let piece = piece.trim();
|
||||
if piece.is_empty() {
|
||||
continue;
|
||||
}
|
||||
if allowed.iter().any(|v| v.eq_ignore_ascii_case(piece)) {
|
||||
return true;
|
||||
}
|
||||
}
|
||||
false
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::query::{Negation, RangeFilter};
|
||||
|
||||
fn empty_query() -> GatewayQuery {
|
||||
GatewayQuery {
|
||||
raw_text: String::new(),
|
||||
free_text: String::new(),
|
||||
filters: BTreeMap::new(),
|
||||
ranges: Vec::new(),
|
||||
negations: Vec::new(),
|
||||
}
|
||||
}
|
||||
|
||||
fn with_filter(field: &str, values: &[&str]) -> GatewayQuery {
|
||||
let mut q = empty_query();
|
||||
q.filters.insert(
|
||||
field.to_string(),
|
||||
values.iter().map(|s| s.to_string()).collect(),
|
||||
);
|
||||
q
|
||||
}
|
||||
|
||||
fn fields(pairs: &[(&str, &str)]) -> BTreeMap<String, String> {
|
||||
pairs
|
||||
.iter()
|
||||
.map(|(k, v)| (k.to_string(), v.to_string()))
|
||||
.collect()
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn filter_rejects_mismatched_filetype() {
|
||||
let q = with_filter("fileType", &["video"]);
|
||||
assert!(!record_matches(&fields(&[("fileType", "image")]), &q));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn filter_accepts_matching_filetype() {
|
||||
let q = with_filter("fileType", &["video"]);
|
||||
assert!(record_matches(&fields(&[("fileType", "video")]), &q));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn filter_rejects_record_missing_field() {
|
||||
let q = with_filter("fileType", &["video"]);
|
||||
assert!(!record_matches(&fields(&[]), &q));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn filter_is_case_insensitive() {
|
||||
let q = with_filter("fileType", &["VIDEO"]);
|
||||
assert!(record_matches(&fields(&[("fileType", "video")]), &q));
|
||||
let q = with_filter("fileType", &["video"]);
|
||||
assert!(record_matches(&fields(&[("fileType", "VIDEO")]), &q));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn filter_ors_within_key() {
|
||||
let q = with_filter("fileType", &["video", "audio"]);
|
||||
assert!(record_matches(&fields(&[("fileType", "audio")]), &q));
|
||||
assert!(record_matches(&fields(&[("fileType", "video")]), &q));
|
||||
assert!(!record_matches(&fields(&[("fileType", "image")]), &q));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn filter_ands_across_keys() {
|
||||
let mut q = with_filter("fileType", &["video"]);
|
||||
q.filters.insert("genres".into(), vec!["Action".into()]);
|
||||
assert!(!record_matches(
|
||||
&fields(&[("fileType", "video"), ("genres", "Drama")]),
|
||||
&q
|
||||
));
|
||||
assert!(record_matches(
|
||||
&fields(&[("fileType", "video"), ("genres", "Action")]),
|
||||
&q
|
||||
));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn csv_field_matches_one_of_its_values() {
|
||||
let q = with_filter("genres", &["Anime"]);
|
||||
assert!(record_matches(
|
||||
&fields(&[("genres", "Animation,Anime,Drama")]),
|
||||
&q
|
||||
));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn csv_field_with_whitespace_is_trimmed() {
|
||||
let q = with_filter("genres", &["Anime"]);
|
||||
assert!(record_matches(
|
||||
&fields(&[("genres", "Animation, Anime , Drama")]),
|
||||
&q
|
||||
));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn languages_keyset_member_matches() {
|
||||
let q = with_filter("languages", &["eng"]);
|
||||
assert!(record_matches(&fields(&[("languages/eng", "true")]), &q));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn languages_filter_is_or_across_codes() {
|
||||
let q = with_filter("languages", &["eng", "fre"]);
|
||||
assert!(record_matches(&fields(&[("languages/eng", "true")]), &q));
|
||||
assert!(record_matches(&fields(&[("languages/fre", "true")]), &q));
|
||||
assert!(!record_matches(&fields(&[("languages/jpn", "true")]), &q));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn languages_filter_fails_open_on_unknown() {
|
||||
let q = with_filter("languages", &["eng"]);
|
||||
// No language info at all ⇒ kept (unknown, not excluded).
|
||||
assert!(record_matches(&fields(&[("fileType", "video")]), &q));
|
||||
// `mul` (multi) may contain the requested language ⇒ kept.
|
||||
assert!(record_matches(&fields(&[("languages/mul", "true")]), &q));
|
||||
// `und` (undetermined) ⇒ kept.
|
||||
assert!(record_matches(&fields(&[("languages/und", "true")]), &q));
|
||||
// A specific language alongside `mul` is still kept (multi).
|
||||
assert!(record_matches(
|
||||
&fields(&[("languages/jpn", "true"), ("languages/mul", "true")]),
|
||||
&q
|
||||
));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn languages_concrete_mismatch_still_drops() {
|
||||
let q = with_filter("languages", &["eng"]);
|
||||
// A concrete, non-matching language ⇒ dropped (language narrows).
|
||||
assert!(!record_matches(&fields(&[("languages/jpn", "true")]), &q));
|
||||
// A tombstoned/false member is treated as absent ⇒ unknown ⇒ kept.
|
||||
assert!(record_matches(&fields(&[("languages/eng", "false")]), &q));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn languages_legacy_flat_csv_fallback() {
|
||||
let q = with_filter("languages", &["eng"]);
|
||||
assert!(record_matches(&fields(&[("languages", "jpn,eng")]), &q));
|
||||
assert!(!record_matches(&fields(&[("languages", "jpn,ger")]), &q));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn range_in_bounds() {
|
||||
let mut q = empty_query();
|
||||
q.ranges.push(RangeFilter {
|
||||
field: "movieYear".into(),
|
||||
lo: Some(2010),
|
||||
hi: Some(2020),
|
||||
});
|
||||
assert!(record_matches(&fields(&[("movieYear", "2015")]), &q));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn range_out_of_bounds_rejected() {
|
||||
let mut q = empty_query();
|
||||
q.ranges.push(RangeFilter {
|
||||
field: "movieYear".into(),
|
||||
lo: Some(2010),
|
||||
hi: Some(2020),
|
||||
});
|
||||
assert!(!record_matches(&fields(&[("movieYear", "2025")]), &q));
|
||||
assert!(!record_matches(&fields(&[("movieYear", "2005")]), &q));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn range_open_upper_bound() {
|
||||
let mut q = empty_query();
|
||||
q.ranges.push(RangeFilter {
|
||||
field: "movieYear".into(),
|
||||
lo: Some(2010),
|
||||
hi: None,
|
||||
});
|
||||
assert!(record_matches(&fields(&[("movieYear", "2025")]), &q));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn range_unparseable_value_rejected() {
|
||||
let mut q = empty_query();
|
||||
q.ranges.push(RangeFilter {
|
||||
field: "movieYear".into(),
|
||||
lo: Some(2010),
|
||||
hi: None,
|
||||
});
|
||||
assert!(!record_matches(&fields(&[("movieYear", "circa 2015")]), &q));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn typed_negation_excludes_match() {
|
||||
let mut q = empty_query();
|
||||
q.negations.push(Negation {
|
||||
field: Some("fileType".into()),
|
||||
value: "image".into(),
|
||||
});
|
||||
assert!(!record_matches(&fields(&[("fileType", "image")]), &q));
|
||||
assert!(record_matches(&fields(&[("fileType", "video")]), &q));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn typed_negation_passes_when_field_missing() {
|
||||
let mut q = empty_query();
|
||||
q.negations.push(Negation {
|
||||
field: Some("fileType".into()),
|
||||
value: "image".into(),
|
||||
});
|
||||
assert!(record_matches(&fields(&[]), &q));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn bare_word_negation_is_not_enforced() {
|
||||
let mut q = empty_query();
|
||||
q.negations.push(Negation {
|
||||
field: None,
|
||||
value: "alice".into(),
|
||||
});
|
||||
assert!(record_matches(
|
||||
&fields(&[("title", "alice in wonderland")]),
|
||||
&q
|
||||
));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn empty_query_matches_anything() {
|
||||
assert!(record_matches(&fields(&[]), &empty_query()));
|
||||
assert!(record_matches(
|
||||
&fields(&[("fileType", "video"), ("title", "x")]),
|
||||
&empty_query()
|
||||
));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn accepts_returns_true_when_no_type_filter() {
|
||||
let q = empty_query();
|
||||
assert!(query_accepts_plugin(&q, &["video"], &["movie"]));
|
||||
assert!(query_accepts_plugin(&q, &[], &[]));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn accepts_rejects_disjoint_filetype() {
|
||||
let q = with_filter("fileType", &["video"]);
|
||||
assert!(!query_accepts_plugin(&q, &["image"], &["gif"]));
|
||||
assert!(query_accepts_plugin(&q, &["video"], &[]));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn accepts_rejects_disjoint_contentkind() {
|
||||
let q = with_filter("contentKind", &["movie"]);
|
||||
assert!(!query_accepts_plugin(
|
||||
&q,
|
||||
&["video", "image"],
|
||||
&["paper", "book"]
|
||||
));
|
||||
assert!(query_accepts_plugin(&q, &["video"], &["movie", "episode"]));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn accepts_requires_both_axes_when_both_filtered() {
|
||||
let mut q = with_filter("fileType", &["video"]);
|
||||
q.filters.insert("contentKind".into(), vec!["movie".into()]);
|
||||
assert!(query_accepts_plugin(&q, &["video"], &["movie"]));
|
||||
assert!(!query_accepts_plugin(&q, &["video"], &["paper"]));
|
||||
assert!(!query_accepts_plugin(&q, &["image"], &["movie"]));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn accepts_wildcard_matches_any_requested_type() {
|
||||
// A plugin advertising "*" (e.g. tribler) accepts any fileType /
|
||||
// contentKind filter on the wildcarded axis.
|
||||
let q = with_filter("fileType", &["archive"]);
|
||||
assert!(query_accepts_plugin(&q, &["*"], &[]));
|
||||
let q = with_filter("contentKind", &["audiobook"]);
|
||||
assert!(query_accepts_plugin(&q, &["*"], &["*"]));
|
||||
// Wildcard on one axis only: the non-wildcard axis still gates.
|
||||
let mut q = with_filter("fileType", &["audio"]);
|
||||
q.filters.insert("contentKind".into(), vec!["podcast".into()]);
|
||||
assert!(query_accepts_plugin(&q, &["*"], &["*"]));
|
||||
assert!(!query_accepts_plugin(&q, &["*"], &["movie"]));
|
||||
}
|
||||
}
|
||||
@@ -1,773 +0,0 @@
|
||||
//! `serve_feeders` — the axum harness that exposes a set of [`FeederPlugin`]s
|
||||
//! over the **feeder HTTP contract**. Every feeder sidecar binary is just
|
||||
//! `serve_feeders(vec![MyPlugin::new()], state_dir, listen)`.
|
||||
//!
|
||||
//! # Contract (JSON over HTTP, mirrors meta-sort's container-plugin shape)
|
||||
//!
|
||||
//! - `GET /manifest` → [`ManifestResponse`]
|
||||
//! - `GET /health` → [`HealthResponse`]
|
||||
//! - `POST /query` → [`QueryRequest`] → [`QueryResponse`]
|
||||
//! - `POST /compute` → [`ComputeRequest`] → [`ComputeResponse`]
|
||||
//! - `GET /fetch/{upstream_id}/{record_id}` → streamed bytes (or 404)
|
||||
//! - `GET /blob/{upstream_id}/{cid}` → bytes (or 404)
|
||||
//!
|
||||
//! ## Bytes & the v1 simplification
|
||||
//!
|
||||
//! `/compute` returns each outcome's bytes **inline, base64-encoded**
|
||||
//! ([`OutcomeDto::bytes_b64`]) — this reuses every existing `compute_outcomes`
|
||||
//! impl verbatim (they already return bytes in `HashOutcome.bytes`) and keeps
|
||||
//! the gateway core's auto-store a single round-trip. base64 adds ~33%; fine
|
||||
//! for books/papers/posters. Large-file streaming (torznab multi-GiB) is a
|
||||
//! documented follow-up — those outcomes are metadata-only anyway
|
||||
//! (`bytes: None`), so they never ride this path.
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::net::SocketAddr;
|
||||
use std::path::{Path as FsPath, PathBuf};
|
||||
use std::sync::Arc;
|
||||
|
||||
use axum::{
|
||||
body::Body,
|
||||
extract::{Path, State},
|
||||
http::StatusCode,
|
||||
response::{Html, IntoResponse, Response},
|
||||
routing::{get, post},
|
||||
Json, Router,
|
||||
};
|
||||
use base64::Engine as _;
|
||||
use futures::StreamExt;
|
||||
use serde::{Deserialize, Serialize};
|
||||
use serde_json::Value;
|
||||
use tracing::{info, warn};
|
||||
|
||||
use crate::config::{self, CONFIG_PAGE_HTML};
|
||||
use crate::plugin::{stamp_default_source, ConfigError, FeederPlugin, HashKind, HashOutcome};
|
||||
use crate::query::{GatewayQuery, GatewaySearchEvent, GatewayWireError};
|
||||
use crate::types::{DiscoveryRecord, GatewayError, PluginHealth};
|
||||
|
||||
// -------- wire DTOs --------------------------------------------------------
|
||||
|
||||
/// Serializable mirror of [`HashKind`] for the feeder HTTP boundary.
|
||||
#[derive(Serialize, Deserialize, Clone, Copy, PartialEq, Eq, Debug)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum HashKindDto {
|
||||
Midhash256,
|
||||
Sha2_256,
|
||||
BtV1File,
|
||||
NzbRelease,
|
||||
}
|
||||
|
||||
impl From<HashKind> for HashKindDto {
|
||||
fn from(k: HashKind) -> Self {
|
||||
match k {
|
||||
HashKind::Midhash256 => HashKindDto::Midhash256,
|
||||
HashKind::Sha2_256 => HashKindDto::Sha2_256,
|
||||
HashKind::BtV1File => HashKindDto::BtV1File,
|
||||
HashKind::NzbRelease => HashKindDto::NzbRelease,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl From<HashKindDto> for HashKind {
|
||||
fn from(k: HashKindDto) -> Self {
|
||||
match k {
|
||||
HashKindDto::Midhash256 => HashKind::Midhash256,
|
||||
HashKindDto::Sha2_256 => HashKind::Sha2_256,
|
||||
HashKindDto::BtV1File => HashKind::BtV1File,
|
||||
HashKindDto::NzbRelease => HashKind::NzbRelease,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// One plugin's capability surface in [`ManifestResponse`].
|
||||
#[derive(Serialize, Deserialize, Clone, Debug)]
|
||||
pub struct PluginManifest {
|
||||
pub id: String,
|
||||
pub served_file_types: Vec<String>,
|
||||
pub served_content_kinds: Vec<String>,
|
||||
}
|
||||
|
||||
/// `GET /manifest` — the feeder's hosted plugins + their served types. The
|
||||
/// gateway core reads this to register `RemoteFeederPlugin`s and to build the
|
||||
/// capability heartbeat it broadcasts to meta-share.
|
||||
#[derive(Serialize, Deserialize, Clone, Debug)]
|
||||
pub struct ManifestResponse {
|
||||
pub feeder_version: String,
|
||||
pub plugins: Vec<PluginManifest>,
|
||||
}
|
||||
|
||||
/// One plugin's liveness in [`HealthResponse`].
|
||||
#[derive(Serialize, Deserialize, Clone, Debug)]
|
||||
pub struct PluginHealthEntry {
|
||||
pub id: String,
|
||||
pub health: PluginHealth,
|
||||
}
|
||||
|
||||
/// `GET /health` — `status` is `"ok"` iff every hosted plugin reports `Ok`.
|
||||
#[derive(Serialize, Deserialize, Clone, Debug)]
|
||||
pub struct HealthResponse {
|
||||
pub status: String,
|
||||
pub plugins: Vec<PluginHealthEntry>,
|
||||
}
|
||||
|
||||
/// `POST /query` request body.
|
||||
#[derive(Serialize, Deserialize, Clone, Debug)]
|
||||
pub struct QueryRequest {
|
||||
pub upstream_id: String,
|
||||
pub query: GatewayQuery,
|
||||
pub max_results: u32,
|
||||
}
|
||||
|
||||
/// `POST /query` response body.
|
||||
#[derive(Serialize, Deserialize, Clone, Debug)]
|
||||
pub struct QueryResponse {
|
||||
pub records: Vec<DiscoveryRecord>,
|
||||
}
|
||||
|
||||
/// `POST /compute` request body.
|
||||
#[derive(Serialize, Deserialize, Clone, Debug)]
|
||||
pub struct ComputeRequest {
|
||||
pub upstream_id: String,
|
||||
pub record_id: String,
|
||||
}
|
||||
|
||||
/// One resolved outcome on the feeder wire. `bytes_b64` carries the full
|
||||
/// payload base64-encoded when the plugin fetched bytes (`Sha2_256` full
|
||||
/// store); `None` for metadata-only outcomes (the core's `(None, Some)`
|
||||
/// branch — e.g. torznab midhash).
|
||||
#[derive(Serialize, Deserialize, Clone, Debug)]
|
||||
pub struct OutcomeDto {
|
||||
pub hash: String,
|
||||
pub hash_kind: HashKindDto,
|
||||
pub file_extension: Option<String>,
|
||||
pub record: Option<DiscoveryRecord>,
|
||||
pub bytes_b64: Option<String>,
|
||||
}
|
||||
|
||||
impl OutcomeDto {
|
||||
fn from_outcome(o: HashOutcome) -> Self {
|
||||
OutcomeDto {
|
||||
hash: o.hash.0,
|
||||
hash_kind: o.hash_kind.into(),
|
||||
file_extension: o.file_extension,
|
||||
record: o.record,
|
||||
bytes_b64: o
|
||||
.bytes
|
||||
.map(|b| base64::engine::general_purpose::STANDARD.encode(&b)),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// `POST /compute` response body.
|
||||
#[derive(Serialize, Deserialize, Clone, Debug)]
|
||||
pub struct ComputeResponse {
|
||||
pub outcomes: Vec<OutcomeDto>,
|
||||
}
|
||||
|
||||
// -------- app state + error mapping ----------------------------------------
|
||||
|
||||
#[derive(Clone)]
|
||||
struct AppState {
|
||||
plugins: Arc<HashMap<String, Arc<dyn FeederPlugin>>>,
|
||||
version: String,
|
||||
/// Feeder state dir. Per-plugin config persists at
|
||||
/// `state_dir/gateway/<id>/config.json` — the same per-plugin dir handed to
|
||||
/// `configure()`, so a UI save is read back on the next restart.
|
||||
state_dir: PathBuf,
|
||||
}
|
||||
|
||||
impl AppState {
|
||||
fn plugin(&self, upstream_id: &str) -> Result<&Arc<dyn FeederPlugin>, Response> {
|
||||
self.plugins.get(upstream_id).ok_or_else(|| {
|
||||
(
|
||||
StatusCode::NOT_FOUND,
|
||||
format!("no feeder plugin for upstream_id '{upstream_id}'"),
|
||||
)
|
||||
.into_response()
|
||||
})
|
||||
}
|
||||
|
||||
/// `state_dir/gateway/<id>/config.json` — canonical per-plugin config file.
|
||||
fn config_path(&self, upstream_id: &str) -> PathBuf {
|
||||
self.state_dir
|
||||
.join("gateway")
|
||||
.join(upstream_id)
|
||||
.join("config.json")
|
||||
}
|
||||
|
||||
/// The effective config as JSON: the persisted `config.json` if it exists,
|
||||
/// else the plugin's in-memory (env-seeded) config. Used as the redaction
|
||||
/// source for reads and the merge base for writes.
|
||||
fn effective_config(&self, upstream_id: &str) -> Value {
|
||||
if let Ok(bytes) = std::fs::read(self.config_path(upstream_id)) {
|
||||
if let Ok(v) = serde_json::from_slice::<Value>(&bytes) {
|
||||
return v;
|
||||
}
|
||||
}
|
||||
self.plugins
|
||||
.get(upstream_id)
|
||||
.map(|p| p.config_values())
|
||||
.unwrap_or_else(|| serde_json::json!({}))
|
||||
}
|
||||
}
|
||||
|
||||
/// Map an in-process [`GatewayError`] to an HTTP status + message. Mirrors the
|
||||
/// status semantics the gateway's `RemoteFeederPlugin` will translate back
|
||||
/// into a `GatewayError` on the core side.
|
||||
fn gateway_error_response(e: GatewayError) -> Response {
|
||||
let (status, msg) = match e {
|
||||
GatewayError::NotFound => (StatusCode::NOT_FOUND, "not found".to_string()),
|
||||
GatewayError::RateLimited { retry_after_s } => (
|
||||
StatusCode::TOO_MANY_REQUESTS,
|
||||
format!("rate limited; retry in {retry_after_s}s"),
|
||||
),
|
||||
GatewayError::Transient(s) => (StatusCode::BAD_GATEWAY, s),
|
||||
GatewayError::Permanent(s) => (StatusCode::UNPROCESSABLE_ENTITY, s),
|
||||
GatewayError::Internal(s) => (StatusCode::INTERNAL_SERVER_ERROR, s.to_string()),
|
||||
};
|
||||
(status, msg).into_response()
|
||||
}
|
||||
|
||||
// -------- handlers ---------------------------------------------------------
|
||||
|
||||
async fn manifest(State(state): State<AppState>) -> Json<ManifestResponse> {
|
||||
let mut plugins: Vec<PluginManifest> = state
|
||||
.plugins
|
||||
.values()
|
||||
.map(|p| PluginManifest {
|
||||
id: p.upstream_id().to_string(),
|
||||
served_file_types: p.served_file_types().iter().map(|s| s.to_string()).collect(),
|
||||
served_content_kinds: p
|
||||
.served_content_kinds()
|
||||
.iter()
|
||||
.map(|s| s.to_string())
|
||||
.collect(),
|
||||
})
|
||||
.collect();
|
||||
plugins.sort_by(|a, b| a.id.cmp(&b.id));
|
||||
Json(ManifestResponse {
|
||||
feeder_version: state.version.clone(),
|
||||
plugins,
|
||||
})
|
||||
}
|
||||
|
||||
async fn health(State(state): State<AppState>) -> Json<HealthResponse> {
|
||||
let mut entries: Vec<PluginHealthEntry> = state
|
||||
.plugins
|
||||
.values()
|
||||
.map(|p| PluginHealthEntry {
|
||||
id: p.upstream_id().to_string(),
|
||||
health: p.health(),
|
||||
})
|
||||
.collect();
|
||||
entries.sort_by(|a, b| a.id.cmp(&b.id));
|
||||
let all_ok = entries
|
||||
.iter()
|
||||
.all(|e| matches!(e.health, PluginHealth::Ok));
|
||||
Json(HealthResponse {
|
||||
status: if all_ok { "ok" } else { "degraded" }.to_string(),
|
||||
plugins: entries,
|
||||
})
|
||||
}
|
||||
|
||||
async fn query(
|
||||
State(state): State<AppState>,
|
||||
Json(req): Json<QueryRequest>,
|
||||
) -> Result<Json<QueryResponse>, Response> {
|
||||
let plugin = state.plugin(&req.upstream_id)?;
|
||||
let mut records = plugin
|
||||
.handle_query(&req.query, req.max_results as usize)
|
||||
.await
|
||||
.map_err(gateway_error_response)?;
|
||||
for r in &mut records {
|
||||
stamp_default_source(&mut r.fields, &req.upstream_id);
|
||||
}
|
||||
Ok(Json(QueryResponse { records }))
|
||||
}
|
||||
|
||||
/// `POST /query_stream` — the streaming analog of [`query`]. Emits the plugin's
|
||||
/// [`FeederPlugin::handle_query_stream`] events as **NDJSON**: one JSON-encoded
|
||||
/// [`GatewaySearchEvent`] per line (`application/x-ndjson`). The gateway core's
|
||||
/// `RemoteFeederPlugin` consumes this and re-emits a `BoxStream<GatewaySearchEvent>`
|
||||
/// into the libp2p streaming worker, preserving incremental enrichment (e.g.
|
||||
/// torznab's late TMDB / poster / subtitle-availability patches). A plugin that
|
||||
/// doesn't override `handle_query_stream` still works here — the trait default
|
||||
/// replays `handle_query` as `Base* + Done`.
|
||||
async fn query_stream(
|
||||
State(state): State<AppState>,
|
||||
Json(req): Json<QueryRequest>,
|
||||
) -> Result<Response, Response> {
|
||||
let plugin = state.plugin(&req.upstream_id)?;
|
||||
// The open can fail (e.g. the plugin can't reach its upstream at all); that
|
||||
// maps to an HTTP error status, same as `/query`. Per-event errors instead
|
||||
// ride the stream as a terminal `GatewaySearchEvent::Error` line.
|
||||
let events = plugin
|
||||
.handle_query_stream(&req.query, req.max_results as usize)
|
||||
.await
|
||||
.map_err(gateway_error_response)?;
|
||||
// Stamp the default provenance member on each streamed `Base` record (a
|
||||
// no-op when the plugin already carries a finer `source/*`, e.g. torznab's
|
||||
// per-indexer label). `EnrichPatch`/`Drop`/`Done` frames pass through: the
|
||||
// base they refer to was already stamped.
|
||||
let upstream_id = req.upstream_id.clone();
|
||||
let events = events.map(move |ev| match ev {
|
||||
GatewaySearchEvent::Base(mut rec) => {
|
||||
stamp_default_source(&mut rec.fields, &upstream_id);
|
||||
GatewaySearchEvent::Base(rec)
|
||||
}
|
||||
other => other,
|
||||
});
|
||||
let body = events.map(|ev| {
|
||||
// Serialization of GatewaySearchEvent is infallible in practice; on the
|
||||
// off chance it fails, terminate with an Error line so the consumer sees
|
||||
// a clean terminal failure rather than a truncated stream.
|
||||
let mut line = serde_json::to_vec(&ev).unwrap_or_else(|e| {
|
||||
serde_json::to_vec(&GatewaySearchEvent::Error(GatewayWireError::Internal(
|
||||
format!("event serialize: {e}"),
|
||||
)))
|
||||
.unwrap_or_default()
|
||||
});
|
||||
line.push(b'\n');
|
||||
Ok::<_, std::convert::Infallible>(line)
|
||||
});
|
||||
Ok((
|
||||
[(axum::http::header::CONTENT_TYPE, "application/x-ndjson")],
|
||||
Body::from_stream(body),
|
||||
)
|
||||
.into_response())
|
||||
}
|
||||
|
||||
async fn compute(
|
||||
State(state): State<AppState>,
|
||||
Json(req): Json<ComputeRequest>,
|
||||
) -> Result<Json<ComputeResponse>, Response> {
|
||||
let plugin = state.plugin(&req.upstream_id)?;
|
||||
let mut outcomes = plugin
|
||||
.compute_outcomes(&req.record_id)
|
||||
.await
|
||||
.map_err(gateway_error_response)?;
|
||||
for o in &mut outcomes {
|
||||
if let Some(rec) = o.record.as_mut() {
|
||||
stamp_default_source(&mut rec.fields, &req.upstream_id);
|
||||
}
|
||||
}
|
||||
Ok(Json(ComputeResponse {
|
||||
outcomes: outcomes.into_iter().map(OutcomeDto::from_outcome).collect(),
|
||||
}))
|
||||
}
|
||||
|
||||
async fn fetch(
|
||||
State(state): State<AppState>,
|
||||
Path((upstream_id, record_id)): Path<(String, String)>,
|
||||
) -> Result<Response, Response> {
|
||||
let plugin = state.plugin(&upstream_id)?;
|
||||
match plugin
|
||||
.handle_fetch(&record_id)
|
||||
.await
|
||||
.map_err(gateway_error_response)?
|
||||
{
|
||||
Some(stream) => {
|
||||
// Map the plugin's GatewayError stream into an io::Error stream so
|
||||
// axum's Body::from_stream accepts it.
|
||||
let body_stream = stream.map(|chunk| {
|
||||
chunk.map_err(|e| std::io::Error::other(e.to_string()))
|
||||
});
|
||||
Ok(Body::from_stream(body_stream).into_response())
|
||||
}
|
||||
None => Err((StatusCode::NOT_FOUND, "no bytes for record").into_response()),
|
||||
}
|
||||
}
|
||||
|
||||
async fn blob(
|
||||
State(state): State<AppState>,
|
||||
Path((upstream_id, cid)): Path<(String, String)>,
|
||||
) -> Result<Response, Response> {
|
||||
let plugin = state.plugin(&upstream_id)?;
|
||||
match plugin.get_blob(&cid).await {
|
||||
Some(bytes) => Ok(bytes.into_response()),
|
||||
None => Err((StatusCode::NOT_FOUND, "no such blob").into_response()),
|
||||
}
|
||||
}
|
||||
|
||||
// -------- config plane -----------------------------------------------------
|
||||
//
|
||||
// Generic, schema-driven per-plugin config. The page + the schema/values JSON
|
||||
// are all served relative to the request path, so this works identically when
|
||||
// hit directly on the feeder or reverse-proxied through the gateway.
|
||||
|
||||
/// `GET /config` — the schema-driven HTML form (static, plugin-agnostic).
|
||||
async fn config_page() -> Html<&'static str> {
|
||||
Html(CONFIG_PAGE_HTML)
|
||||
}
|
||||
|
||||
/// `GET /config/schema` — the single hosted plugin's declared config schema.
|
||||
/// (A feeder hosts one plugin in practice; if it hosts several, the first by
|
||||
/// id is used — the gateway proxies per upstream_id anyway.)
|
||||
async fn config_schema(State(state): State<AppState>) -> Json<config::ConfigSchema> {
|
||||
let schema = sole_plugin(&state)
|
||||
.map(|p| p.config_schema())
|
||||
.unwrap_or_default();
|
||||
Json(schema)
|
||||
}
|
||||
|
||||
/// `GET /config/values` — current values with secrets redacted to `<key>_set`.
|
||||
async fn config_values_get(State(state): State<AppState>) -> Json<Value> {
|
||||
match sole_plugin(&state) {
|
||||
Some(p) => {
|
||||
let schema = p.config_schema();
|
||||
let eff = state.effective_config(p.upstream_id());
|
||||
Json(config::redact(&eff, &schema))
|
||||
}
|
||||
None => Json(serde_json::json!({})),
|
||||
}
|
||||
}
|
||||
|
||||
/// `PUT /config/values` — merge the submitted values onto the stored config
|
||||
/// (schema-aware; blank secrets keep their stored value), persist atomically,
|
||||
/// and signal that a feeder restart is needed to apply them.
|
||||
async fn config_values_put(
|
||||
State(state): State<AppState>,
|
||||
Json(incoming): Json<Value>,
|
||||
) -> Result<Json<Value>, Response> {
|
||||
let plugin = sole_plugin(&state)
|
||||
.ok_or_else(|| (StatusCode::NOT_FOUND, "feeder hosts no plugin").into_response())?;
|
||||
let id = plugin.upstream_id();
|
||||
let schema = plugin.config_schema();
|
||||
let base = state.effective_config(id);
|
||||
let merged = config::merge(&base, &incoming, &schema);
|
||||
|
||||
let path = state.config_path(id);
|
||||
if let Some(dir) = path.parent() {
|
||||
std::fs::create_dir_all(dir)
|
||||
.map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, format!("mkdir: {e}")).into_response())?;
|
||||
}
|
||||
let tmp = path.with_extension("json.tmp");
|
||||
let body = serde_json::to_vec_pretty(&merged)
|
||||
.map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, format!("serialize: {e}")).into_response())?;
|
||||
std::fs::write(&tmp, body)
|
||||
.and_then(|_| std::fs::rename(&tmp, &path))
|
||||
.map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, format!("write config: {e}")).into_response())?;
|
||||
info!(target: "meta-feeder", upstream_id = id, path = %path.display(), "feeder config saved; scheduling self-restart to apply");
|
||||
// Apply the new config by restarting the process. We can't hot-swap config
|
||||
// in-place: plugins hold an exclusive redb cache handle for their whole life,
|
||||
// so a fresh instance can't reopen the cache while the old one is alive. A
|
||||
// clean process exit is the reliable apply path — Docker's restart policy
|
||||
// (`unless-stopped`) brings the feeder back in ~2-3s and `configure()`
|
||||
// re-reads config.json at startup. Delay briefly so THIS response flushes to
|
||||
// the client before we exit.
|
||||
tokio::spawn(async {
|
||||
tokio::time::sleep(std::time::Duration::from_millis(800)).await;
|
||||
info!(target: "meta-feeder", "restarting now to apply saved config");
|
||||
std::process::exit(0);
|
||||
});
|
||||
Ok(Json(serde_json::json!({ "saved": true, "reloading": true })))
|
||||
}
|
||||
|
||||
/// `POST /enrich/callback` — sink for the enrichment plugins' completion
|
||||
/// callbacks. The feeder drives those plugins (filename-parser, tmdb) during a
|
||||
/// self-published resolve and reads results back by polling meta-core, so the
|
||||
/// callback carries no load — this route exists only so the plugins' callback
|
||||
/// POST gets a 200 instead of logging a connection failure. Body is ignored.
|
||||
async fn enrich_callback() -> StatusCode {
|
||||
StatusCode::OK
|
||||
}
|
||||
|
||||
/// A feeder hosts one plugin in the common case. Pick it (lowest id for
|
||||
/// determinism if several are present).
|
||||
fn sole_plugin(state: &AppState) -> Option<&Arc<dyn FeederPlugin>> {
|
||||
state
|
||||
.plugins
|
||||
.values()
|
||||
.min_by(|a, b| a.upstream_id().cmp(b.upstream_id()))
|
||||
}
|
||||
|
||||
// -------- entry point ------------------------------------------------------
|
||||
|
||||
/// Build the feeder router for an already-configured plugin set. Exposed for
|
||||
/// in-process tests (drive it without binding a socket).
|
||||
pub fn router(
|
||||
plugins: HashMap<String, Arc<dyn FeederPlugin>>,
|
||||
version: String,
|
||||
state_dir: impl Into<PathBuf>,
|
||||
) -> Router {
|
||||
let state = AppState {
|
||||
plugins: Arc::new(plugins),
|
||||
version,
|
||||
state_dir: state_dir.into(),
|
||||
};
|
||||
Router::new()
|
||||
.route("/manifest", get(manifest))
|
||||
.route("/health", get(health))
|
||||
.route("/query", post(query))
|
||||
.route("/query_stream", post(query_stream))
|
||||
.route("/compute", post(compute))
|
||||
.route("/fetch/:upstream_id/:record_id", get(fetch))
|
||||
.route("/blob/:upstream_id/:cid", get(blob))
|
||||
.route("/config", get(config_page))
|
||||
.route("/config/schema", get(config_schema))
|
||||
.route("/config/values", get(config_values_get).put(config_values_put))
|
||||
.route("/enrich/callback", post(enrich_callback))
|
||||
.with_state(state)
|
||||
}
|
||||
|
||||
/// Configure each plugin (cache dir under `state_dir/gateway/<id>`), then serve
|
||||
/// the feeder HTTP contract on `listen` until shutdown.
|
||||
///
|
||||
/// The feeder process never aborts on a config problem: a plugin whose
|
||||
/// `configure()` returns [`ConfigError::MissingConfig`] is **soft-skipped**
|
||||
/// (warn + dropped — an optional source the operator didn't ask for), and any
|
||||
/// other failure (bad/empty key, cache-dir setup, …) registers a
|
||||
/// [`DegradedPlugin`] that stays up and reports the reason via `/health` and
|
||||
/// `/manifest`. The container keeps running so the error is visible in the UX
|
||||
/// instead of crash-looping.
|
||||
pub async fn serve_feeders(
|
||||
plugins: Vec<Box<dyn FeederPlugin>>,
|
||||
state_dir: impl Into<PathBuf>,
|
||||
listen: SocketAddr,
|
||||
) -> anyhow::Result<()> {
|
||||
let state_dir = state_dir.into();
|
||||
let configured = configure_plugins(plugins, &state_dir)?;
|
||||
let version = env!("CARGO_PKG_VERSION").to_string();
|
||||
let app = router(configured, version, state_dir.clone());
|
||||
|
||||
let listener = tokio::net::TcpListener::bind(listen).await?;
|
||||
info!(
|
||||
target: "meta-feeder",
|
||||
%listen,
|
||||
"feeder listening"
|
||||
);
|
||||
axum::serve(listener, app)
|
||||
.await
|
||||
.map_err(anyhow::Error::from)
|
||||
}
|
||||
|
||||
/// A plugin whose `configure()` (or cache-dir setup) failed. It stays
|
||||
/// **registered** so the failure is visible at `GET /health` (as
|
||||
/// [`PluginHealth::Degraded`]) and `GET /manifest`; any query/compute against it
|
||||
/// returns the captured reason as a permanent error. This is what keeps one
|
||||
/// misconfigured feeder from crash-looping the whole container — the operator
|
||||
/// sees the reason in the UX and fixes the config, no restart-loop required.
|
||||
struct DegradedPlugin {
|
||||
id: &'static str,
|
||||
reason: String,
|
||||
file_types: &'static [&'static str],
|
||||
content_kinds: &'static [&'static str],
|
||||
/// Preserved from the wrapped plugin so a degraded/unconfigured feeder still
|
||||
/// serves its config form — otherwise the dashboard shows "no configuration
|
||||
/// required" and the operator can never fix the very config that's missing.
|
||||
schema: crate::config::ConfigSchema,
|
||||
}
|
||||
|
||||
#[async_trait::async_trait]
|
||||
impl FeederPlugin for DegradedPlugin {
|
||||
fn upstream_id(&self) -> &'static str {
|
||||
self.id
|
||||
}
|
||||
|
||||
fn configure(&mut self, _cache_dir: &FsPath) -> Result<(), ConfigError> {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn handle_query(
|
||||
&self,
|
||||
_query: &GatewayQuery,
|
||||
_max_results: usize,
|
||||
) -> Result<Vec<DiscoveryRecord>, GatewayError> {
|
||||
Err(GatewayError::Permanent(self.reason.clone()))
|
||||
}
|
||||
|
||||
async fn compute_outcomes(
|
||||
&self,
|
||||
_record_id: &str,
|
||||
) -> Result<Vec<HashOutcome>, GatewayError> {
|
||||
Err(GatewayError::Permanent(self.reason.clone()))
|
||||
}
|
||||
|
||||
fn health(&self) -> PluginHealth {
|
||||
PluginHealth::Degraded {
|
||||
reason: self.reason.clone(),
|
||||
}
|
||||
}
|
||||
|
||||
fn served_file_types(&self) -> &'static [&'static str] {
|
||||
self.file_types
|
||||
}
|
||||
|
||||
fn served_content_kinds(&self) -> &'static [&'static str] {
|
||||
self.content_kinds
|
||||
}
|
||||
|
||||
fn config_schema(&self) -> crate::config::ConfigSchema {
|
||||
self.schema.clone()
|
||||
}
|
||||
}
|
||||
|
||||
/// Run each plugin's `configure()` against its per-plugin cache dir and return
|
||||
/// the plugins keyed by `upstream_id`. Never errors on a per-plugin problem:
|
||||
/// `MissingConfig` is soft-skipped, every other failure becomes a
|
||||
/// [`DegradedPlugin`] (see [`serve_feeders`]). Public so feeder binaries /
|
||||
/// tests can configure without serving.
|
||||
pub fn configure_plugins(
|
||||
plugins: Vec<Box<dyn FeederPlugin>>,
|
||||
state_dir: &FsPath,
|
||||
) -> anyhow::Result<HashMap<String, Arc<dyn FeederPlugin>>> {
|
||||
let mut out: HashMap<String, Arc<dyn FeederPlugin>> = HashMap::new();
|
||||
for mut plugin in plugins {
|
||||
let id = plugin.upstream_id();
|
||||
// Captured up front (both are `'static`, so this borrow of `plugin`
|
||||
// ends immediately) — lets us build a DegradedPlugin without holding a
|
||||
// borrow across `configure(&mut self)` / the move into `out`.
|
||||
let file_types = plugin.served_file_types();
|
||||
let content_kinds = plugin.served_content_kinds();
|
||||
// Preserve the config surface so a degraded stand-in still serves the
|
||||
// real config form (see DegradedPlugin.schema).
|
||||
let schema = plugin.config_schema();
|
||||
let cache_dir = state_dir.join("gateway").join(id);
|
||||
|
||||
// Build (don't insert) a degraded stand-in: keeps the container up and
|
||||
// surfaces the reason via /health instead of crash-looping.
|
||||
let make_degraded = |reason: String| -> Arc<dyn FeederPlugin> {
|
||||
warn!(
|
||||
target: "meta-feeder",
|
||||
upstream_id = id,
|
||||
%reason,
|
||||
"feeder plugin degraded: container stays up; error surfaced via /health"
|
||||
);
|
||||
Arc::new(DegradedPlugin {
|
||||
id,
|
||||
reason,
|
||||
file_types,
|
||||
content_kinds,
|
||||
schema: schema.clone(),
|
||||
})
|
||||
};
|
||||
|
||||
if let Err(e) = std::fs::create_dir_all(&cache_dir) {
|
||||
let reason = format!("cache dir {} setup failed: {e}", cache_dir.display());
|
||||
out.insert(id.to_string(), make_degraded(reason));
|
||||
continue;
|
||||
}
|
||||
|
||||
match plugin.configure(&cache_dir) {
|
||||
Ok(()) => {
|
||||
out.insert(id.to_string(), Arc::from(plugin));
|
||||
}
|
||||
Err(ConfigError::MissingConfig { plugin: p, what }) => {
|
||||
// Keep a degraded stand-in (don't drop it) so its config form
|
||||
// stays reachable from the dashboard and the operator can supply
|
||||
// the missing config — then restart the feeder to apply it.
|
||||
warn!(
|
||||
target: "meta-feeder",
|
||||
upstream_id = %p,
|
||||
missing = %what,
|
||||
"feeder plugin degraded: required config not supplied; \
|
||||
config form still served so it can be fixed from the dashboard"
|
||||
);
|
||||
out.insert(
|
||||
id.to_string(),
|
||||
make_degraded(format!("required config not supplied: {what}")),
|
||||
);
|
||||
}
|
||||
Err(other) => {
|
||||
out.insert(id.to_string(), make_degraded(other.to_string()));
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::types::DiscoveryRecord;
|
||||
use async_trait::async_trait;
|
||||
use futures::stream::BoxStream;
|
||||
use std::collections::BTreeMap;
|
||||
|
||||
/// A feeder plugin that overrides `handle_query_stream` to emit a Base, then
|
||||
/// an EnrichPatch, then Done — so the NDJSON route is exercised on a real
|
||||
/// incremental stream (not just the collect-default).
|
||||
struct StubStreaming;
|
||||
|
||||
#[async_trait]
|
||||
impl FeederPlugin for StubStreaming {
|
||||
fn upstream_id(&self) -> &'static str {
|
||||
"stub"
|
||||
}
|
||||
fn configure(&mut self, _cache_dir: &FsPath) -> Result<(), ConfigError> {
|
||||
Ok(())
|
||||
}
|
||||
async fn handle_query(
|
||||
&self,
|
||||
_q: &GatewayQuery,
|
||||
_n: usize,
|
||||
) -> Result<Vec<DiscoveryRecord>, GatewayError> {
|
||||
Ok(vec![])
|
||||
}
|
||||
async fn handle_query_stream(
|
||||
&self,
|
||||
_q: &GatewayQuery,
|
||||
_n: usize,
|
||||
) -> Result<BoxStream<'static, GatewaySearchEvent>, GatewayError> {
|
||||
let rec = DiscoveryRecord {
|
||||
upstream_id: "stub".into(),
|
||||
record_id: "1".into(),
|
||||
fields: BTreeMap::new(),
|
||||
};
|
||||
let events = vec![
|
||||
GatewaySearchEvent::Base(rec),
|
||||
GatewaySearchEvent::EnrichPatch {
|
||||
record_id: "1".into(),
|
||||
set: BTreeMap::from([("title".to_string(), "X".to_string())]),
|
||||
remove: vec![],
|
||||
},
|
||||
GatewaySearchEvent::Done,
|
||||
];
|
||||
Ok(Box::pin(futures::stream::iter(events)))
|
||||
}
|
||||
async fn compute_outcomes(
|
||||
&self,
|
||||
_record_id: &str,
|
||||
) -> Result<Vec<HashOutcome>, GatewayError> {
|
||||
Ok(vec![])
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn query_stream_emits_ndjson_events_in_order() {
|
||||
let mut plugins: HashMap<String, Arc<dyn FeederPlugin>> = HashMap::new();
|
||||
plugins.insert("stub".to_string(), Arc::new(StubStreaming));
|
||||
let app = router(plugins, "test".to_string(), std::env::temp_dir());
|
||||
|
||||
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
|
||||
let addr = listener.local_addr().unwrap();
|
||||
tokio::spawn(async move {
|
||||
axum::serve(listener, app).await.unwrap();
|
||||
});
|
||||
|
||||
let body = reqwest::Client::new()
|
||||
.post(format!("http://{addr}/query_stream"))
|
||||
.json(&serde_json::json!({
|
||||
"upstream_id": "stub",
|
||||
"query": {"raw_text":"x","free_text":"x","filters":{},"ranges":[],"negations":[]},
|
||||
"max_results": 5
|
||||
}))
|
||||
.send()
|
||||
.await
|
||||
.unwrap()
|
||||
.text()
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
let events: Vec<GatewaySearchEvent> = body
|
||||
.lines()
|
||||
.filter(|l| !l.trim().is_empty())
|
||||
.map(|l| serde_json::from_str(l).expect("each NDJSON line is a GatewaySearchEvent"))
|
||||
.collect();
|
||||
|
||||
assert_eq!(events.len(), 3, "body was: {body:?}");
|
||||
assert!(matches!(events[0], GatewaySearchEvent::Base(_)));
|
||||
assert!(matches!(events[1], GatewaySearchEvent::EnrichPatch { .. }));
|
||||
assert!(matches!(events[2], GatewaySearchEvent::Done));
|
||||
}
|
||||
}
|
||||
@@ -1,124 +0,0 @@
|
||||
//! In-process types used by the gateway runtime, feeders, and plugins.
|
||||
//!
|
||||
//! `GatewayError` here is **distinct from** the wire-stable
|
||||
//! `GatewayWireError` (see [`crate::query`]): this one can carry rich
|
||||
//! internal context (anyhow chains, plugin specifics); the wire one is narrow
|
||||
//! on purpose since adding variants is a protocol change. Conversion happens
|
||||
//! at the protocol boundary.
|
||||
|
||||
use std::collections::BTreeMap;
|
||||
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
/// Content-addressed hash string carried over the wire. Renamed from
|
||||
/// `Midhash` in the multi-outcome plugin contract migration — the name
|
||||
/// `Midhash` lied for sha2-256 CIDs (which gateway plugins also emit
|
||||
/// via `compute_ipfs_cid`). The carrier is generic; the hash family is
|
||||
/// discriminated by `HashKind` on `HashOutcome`.
|
||||
///
|
||||
/// Shape: base32-lower 'b'-prefixed multibase CIDv1. The specific
|
||||
/// multihash inside (`0x1000` for midhash256, `0x12` for sha2-256) is
|
||||
/// determined by the producing plugin's hashing path.
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Hash, Serialize, Deserialize)]
|
||||
pub struct Hash(pub String);
|
||||
|
||||
impl Hash {
|
||||
pub fn as_str(&self) -> &str {
|
||||
&self.0
|
||||
}
|
||||
}
|
||||
|
||||
impl std::fmt::Display for Hash {
|
||||
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
f.write_str(&self.0)
|
||||
}
|
||||
}
|
||||
|
||||
/// Pre-hash identifier for a record discovered through a gateway.
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Hash, Serialize, Deserialize)]
|
||||
pub struct DiscoveryId {
|
||||
pub upstream_id: String,
|
||||
pub record_id: String,
|
||||
}
|
||||
|
||||
/// Metadata-only search result. Distinct from meta-core's persistent record
|
||||
/// (which is a flat `map[string]string` indexed by midhash); a
|
||||
/// `DiscoveryRecord` only lives inside the gateway search/response pipeline.
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub struct DiscoveryRecord {
|
||||
pub upstream_id: String,
|
||||
pub record_id: String,
|
||||
pub fields: BTreeMap<String, String>,
|
||||
}
|
||||
|
||||
/// In-process error returned by `FeederPlugin` methods.
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum GatewayError {
|
||||
#[error("upstream transient error: {0}")]
|
||||
Transient(String),
|
||||
|
||||
#[error("upstream permanent error: {0}")]
|
||||
Permanent(String),
|
||||
|
||||
#[error("rate limited; retry in {retry_after_s}s")]
|
||||
RateLimited { retry_after_s: u32 },
|
||||
|
||||
#[error("record not found upstream")]
|
||||
NotFound,
|
||||
|
||||
#[error("internal: {0}")]
|
||||
Internal(#[from] anyhow::Error),
|
||||
}
|
||||
|
||||
/// Byte stream returned by a plugin's `handle_fetch`.
|
||||
pub type ByteStream = futures::stream::BoxStream<'static, Result<bytes::Bytes, GatewayError>>;
|
||||
|
||||
/// Plugin liveness snapshot surfaced via `/health`. Must not call upstream
|
||||
/// — `health()` is invoked on every status poll.
|
||||
#[derive(Clone, Debug, Serialize, Deserialize)]
|
||||
#[serde(rename_all = "snake_case", tag = "state")]
|
||||
pub enum PluginHealth {
|
||||
Ok,
|
||||
Degraded { reason: String },
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn hash_display_and_as_str_match() {
|
||||
let m = Hash("bafy123".to_string());
|
||||
assert_eq!(m.as_str(), "bafy123");
|
||||
assert_eq!(format!("{m}"), "bafy123");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn discovery_record_roundtrips_through_serde_json() {
|
||||
let r = DiscoveryRecord {
|
||||
upstream_id: "scihub".to_string(),
|
||||
record_id: "10.1038/s41586-021-03819-2".to_string(),
|
||||
fields: BTreeMap::from_iter([
|
||||
("title".to_string(), "AlphaFold".to_string()),
|
||||
("year".to_string(), "2021".to_string()),
|
||||
]),
|
||||
};
|
||||
let j = serde_json::to_string(&r).expect("serialize");
|
||||
let back: DiscoveryRecord = serde_json::from_str(&j).expect("deserialize");
|
||||
assert_eq!(r, back);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn plugin_health_serialises_with_tag() {
|
||||
let ok = serde_json::to_value(PluginHealth::Ok).unwrap();
|
||||
assert_eq!(ok, serde_json::json!({"state": "ok"}));
|
||||
let bad = serde_json::to_value(PluginHealth::Degraded {
|
||||
reason: "upstream 5xx".into(),
|
||||
})
|
||||
.unwrap();
|
||||
assert_eq!(
|
||||
bad,
|
||||
serde_json::json!({"state": "degraded", "reason": "upstream 5xx"})
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -25,7 +25,7 @@ name = "tribler-feeder"
|
||||
path = "src/main.rs"
|
||||
|
||||
[dependencies]
|
||||
meta-feeder-sdk = { path = "../../crates/meta-feeder-sdk" }
|
||||
meta-feeder-sdk = { git = "https://github.com/worph/meta-feeder-sdk", tag = "v1.7.0" }
|
||||
|
||||
tokio = { version = "1.40", features = ["full"] }
|
||||
axum = { version = "0.7", features = ["json"] }
|
||||
|
||||
@@ -2,8 +2,8 @@
|
||||
# =============================================================================
|
||||
# tribler-feeder — Tribler (IPv8 overlay) feeder sidecar. Multi-stage build.
|
||||
#
|
||||
# Build context must be this repo root (packages/plugins/meta-feeder-tribler) so
|
||||
# the `meta-feeder-sdk` path dependency resolves. Same BuildKit cache-mount
|
||||
# Build context must be this repo root (packages/plugins/meta-feeder-tribler).
|
||||
# `meta-feeder-sdk` is a git dependency pinned by tag. Same BuildKit cache-mount
|
||||
# strategy as the other feeders. No librqbit here (tribler enumerates files via
|
||||
# the Tribler core's metainfo), so the cold build is lighter than the torrent
|
||||
# feeder's.
|
||||
@@ -20,7 +20,6 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
COPY Cargo.toml Cargo.lock rust-toolchain.toml ./
|
||||
COPY crates/ crates/
|
||||
COPY feeder-plugin/ feeder-plugin/
|
||||
|
||||
RUN --mount=type=cache,id=cargo-registry,target=/usr/local/cargo/registry \
|
||||
|
||||
@@ -81,8 +81,8 @@ pub(crate) const TORRENT_RESOLVE_MAX_PER_INDEXER: usize = 30;
|
||||
/// it arrives until the gateway sends `Done`. (The 5 s `GATEWAY_CALL_TIMEOUT`
|
||||
/// is only the one-shot fallback for gateways that don't speak the streaming
|
||||
/// protocol.) So this can be generous: the binding constraint is meta-share's
|
||||
/// `SEARCH_TIMEOUT_SECS` silence window (set to match in the dev compose), not
|
||||
/// a hard call timeout. 5 min is a safety ceiling on a pathological page;
|
||||
/// `SEARCH_TIMEOUT_SECS` silence window (300 s, the meta-search code default),
|
||||
/// not a hard call timeout. 5 min is a safety ceiling on a pathological page;
|
||||
/// a normal XXX page (≤ `TORRENT_RESOLVE_MAX_PER_INDEXER` items at ~1.9 s
|
||||
/// each, 8-wide) finishes in well under 10 s and the gateway returns then.
|
||||
pub(crate) const TORRENT_RESOLVE_BUDGET: Duration = Duration::from_secs(300);
|
||||
|
||||
@@ -178,6 +178,16 @@ fn build_seed_record(
|
||||
// mode marker, contentKind, and optionally anime.
|
||||
fields.insert(mode.marker().to_string(), "true".to_string());
|
||||
fields.insert("contentKind".to_string(), content_kind.to_string());
|
||||
// The routing axes, co-written with the kind they derive from
|
||||
// (METADATA_KEYS.md §1): `fileType` says what the bytes are, `domain` says
|
||||
// which app wants the record, `workForm` whether the work stands alone or
|
||||
// is one instalment of a run.
|
||||
if let Some(d) = meta_feeder_sdk::domain::domain_for_content_kind(content_kind) {
|
||||
fields.insert("domain".to_string(), d.to_string());
|
||||
}
|
||||
if let Some(wf) = meta_feeder_sdk::domain::work_form_for_content_kind(content_kind) {
|
||||
fields.insert("workForm".to_string(), wf.to_string());
|
||||
}
|
||||
fields.insert("fileType".to_string(), "video".to_string());
|
||||
if anime {
|
||||
fields.insert("anime".to_string(), "true".to_string());
|
||||
|
||||
@@ -13,8 +13,8 @@ use tracing::{debug, warn};
|
||||
|
||||
use crate::title::clean_torrent_title;
|
||||
use crate::tmdb::{
|
||||
principal_top_n, season_episode_bounds, SeasonEpisodeBounds, TmdbCall, TmdbClient,
|
||||
TmdbExternalIds, TmdbHit, TmdbKind, TmdbTvDetails,
|
||||
principal_top_n, season_episode_bounds, tmdb_title_member_keys, SeasonEpisodeBounds, TmdbCall,
|
||||
TmdbClient, TmdbExternalIds, TmdbHit, TmdbKind, TmdbTvDetails,
|
||||
};
|
||||
use crate::tmdb_budget::TmdbBudget;
|
||||
use meta_feeder_sdk::cache::MidhashCache;
|
||||
@@ -548,6 +548,8 @@ fn build_stub_patch(f: &BTreeMap<String, String>) -> Option<EnrichOutcome> {
|
||||
set.insert("fileType".to_string(), "video".to_string());
|
||||
set.insert("videoType".to_string(), "movie".to_string());
|
||||
set.insert("contentKind".to_string(), "movie".to_string());
|
||||
set.insert("domain".to_string(), "screen".to_string());
|
||||
set.insert("workForm".to_string(), "standalone".to_string());
|
||||
}
|
||||
if !has_poster {
|
||||
// Non-resolvable marker — the image 404s (broken tile), but the card
|
||||
@@ -667,7 +669,7 @@ async fn compute_enrichment_inner(
|
||||
Some(resolved) => resolved,
|
||||
None => return EnrichOutcome::Noop,
|
||||
};
|
||||
build_enrichment_patch(record, enricher, hit, matched_kind, raw_title).await
|
||||
build_enrichment_patch(record, enricher, hit, matched_kind).await
|
||||
}
|
||||
|
||||
/// Pick the canonical TMDB entry for `record`. Prefers the **anchored** path —
|
||||
@@ -720,35 +722,32 @@ async fn resolve_enrichment_hit(
|
||||
/// content-kind reconciliation (movie clears bogus season/episode; tv normalises
|
||||
/// to `episode` and bounds-checks the parsed season/episode — an out-of-bounds
|
||||
/// verdict drops the record), and the transient `posterPath` the poster store
|
||||
/// consumes. `raw_title` is preserved as `originalTitle`.
|
||||
/// consumes. The raw release string is never copied into a title key — it
|
||||
/// stays in `fileName`.
|
||||
async fn build_enrichment_patch(
|
||||
record: &DiscoveryRecord,
|
||||
enricher: &TmdbEnricher,
|
||||
hit: TmdbHit,
|
||||
matched_kind: TmdbKind,
|
||||
raw_title: Option<String>,
|
||||
) -> EnrichOutcome {
|
||||
let mut set: BTreeMap<String, String> = BTreeMap::new();
|
||||
let mut remove: Vec<String> = Vec::new();
|
||||
set.insert("title".to_string(), hit.title.clone());
|
||||
// Preserve the raw tracker release title (quality tags / release-group
|
||||
// info) in `originalTitle`. This is the canonical home now that the
|
||||
// bespoke `releaseTitle` key is retired — reuse-before-invent
|
||||
// (METADATA_KEYS.md rule #1, §14.10 title sprawl).
|
||||
if let Some(raw_title) = raw_title {
|
||||
set.insert("originalTitle".to_string(), raw_title);
|
||||
}
|
||||
// TMDB's original-language title (e.g. "ナルト") is a genuine localized
|
||||
// title — file it under the namespaced `titles/{lang3}` (METADATA_KEYS.md
|
||||
// §3), keyed by TMDB's `original_language`, instead of clobbering
|
||||
// `originalTitle`. Skipped when the language is outside the common 639-1
|
||||
// set or the title is blank (originalTitle still carries the raw title).
|
||||
// The raw tracker release string is NOT stored here — it is emitted as
|
||||
// `fileName` at ingest (METADATA_KEYS.md §1). `originalTitle` carries TMDB's
|
||||
// *original-language* title (e.g. "ナルト" / "Sousou no Frieren"), clean —
|
||||
// never the release string.
|
||||
if let Some(orig) = hit.original_title.clone() {
|
||||
if !orig.trim().is_empty() {
|
||||
if let Some(lang3) = hit.original_lang3() {
|
||||
set.insert(format!("titles/{lang3}"), orig);
|
||||
set.insert("originalTitle".to_string(), orig);
|
||||
}
|
||||
}
|
||||
// Whoever writes `title`/`originalTitle` also files them as members of the
|
||||
// language-nested `titles/{lang3}/{name}` key-set (METADATA_KEYS.md): the
|
||||
// original under TMDB's `original_language` (`und` when unmapped), the title
|
||||
// under `eng` when it differs from the original.
|
||||
for key in tmdb_title_member_keys(&hit) {
|
||||
set.insert(key, "true".to_string());
|
||||
}
|
||||
if let Some(o) = hit.overview.clone() {
|
||||
if !o.trim().is_empty() {
|
||||
@@ -773,6 +772,13 @@ async fn build_enrichment_patch(
|
||||
match matched_kind {
|
||||
TmdbKind::Movie => {
|
||||
set.insert("contentKind".to_string(), "movie".to_string());
|
||||
// The routing axes move with the kind. `domain` is `screen` for
|
||||
// films and serials alike since the merge (METADATA_KEYS.md
|
||||
// §14.17), so the film-vs-serial half is `workForm` — and
|
||||
// reconciling the kind without it would leave the record shaped as
|
||||
// a series.
|
||||
set.insert("domain".to_string(), "screen".to_string());
|
||||
set.insert("workForm".to_string(), "standalone".to_string());
|
||||
// Co-write the narrower `videoType` facet (METADATA_KEYS.md §14.11
|
||||
// — contentKind=movie ↔ videoType=movie). Overwrites any
|
||||
// category-derived videoType from `into_discovery_record` with the
|
||||
@@ -784,6 +790,8 @@ async fn build_enrichment_patch(
|
||||
}
|
||||
TmdbKind::Tv => {
|
||||
set.insert("contentKind".to_string(), "episode".to_string());
|
||||
set.insert("domain".to_string(), "screen".to_string());
|
||||
set.insert("workForm".to_string(), "serial".to_string());
|
||||
// contentKind=episode ↔ videoType=tvshow (METADATA_KEYS.md §14.11).
|
||||
set.insert("videoType".to_string(), "tvshow".to_string());
|
||||
// Season/episode boundary validation. Only a parsed `season`
|
||||
|
||||
@@ -7,12 +7,16 @@
|
||||
//! torrent-core TMDB catalog-discovery stack. Split out of `meta-feeder-torrent`
|
||||
//! so its dashboard config panel resolves to the tribler schema.
|
||||
//!
|
||||
//! Env (shared): `META_FEEDER_HTTP_LISTEN` (default `0.0.0.0:8080`),
|
||||
//! `META_FEEDER_STATE_DIR` (default `/data/meta-feeder`), `META_CORE_URL` (D4
|
||||
//! self-publish), `RUST_LOG`.
|
||||
//! Env (tribler source): `TRIBLER_SIDECAR_URL`, `TRIBLER_API_KEY`,
|
||||
//! `TMDB_TOKEN`/`TMDB_LANGUAGE`, plus the enrichment-plugin URLs
|
||||
//! (`FILENAME_PARSER_URL`, `TMDB_PLUGIN_URL`, `META_FEEDER_CALLBACK_URL`).
|
||||
//! ALL tribler config — sidecar URL, api key, tmdb token/language, meta-core
|
||||
//! URL — is read from the persisted `config.json` (dashboard-written) ONLY.
|
||||
//! There is NO env seed for these: a field in the dashboard config schema must
|
||||
//! not have an env var (config from JSON file / web UI is the single source of
|
||||
//! truth).
|
||||
//!
|
||||
//! Env (infra only): `META_FEEDER_HTTP_LISTEN` (default `0.0.0.0:8080`),
|
||||
//! `META_FEEDER_STATE_DIR` (default `/data/meta-feeder`), `META_GATEWAY_PEER_ID`,
|
||||
//! the enrichment-plugin URLs (`FILENAME_PARSER_URL`, `TMDB_PLUGIN_URL`,
|
||||
//! `META_FEEDER_CALLBACK_URL`), `RUST_LOG`.
|
||||
|
||||
use std::net::SocketAddr;
|
||||
|
||||
@@ -21,7 +25,7 @@ use meta_feeder_sdk::serve_feeders;
|
||||
use tracing::info;
|
||||
use tracing_subscriber::EnvFilter;
|
||||
|
||||
use tribler_feeder::tribler::{TriblerConfigFile, TriblerPlugin};
|
||||
use tribler_feeder::tribler::TriblerPlugin;
|
||||
|
||||
#[tokio::main]
|
||||
async fn main() -> anyhow::Result<()> {
|
||||
@@ -35,11 +39,10 @@ async fn main() -> anyhow::Result<()> {
|
||||
let state_dir =
|
||||
std::env::var("META_FEEDER_STATE_DIR").unwrap_or_else(|_| "/data/meta-feeder".to_string());
|
||||
|
||||
// Env is the first-boot seed; the persisted config.json (under the
|
||||
// per-source cache dir) wins once saved through the dashboard. The tribler
|
||||
// source builds its enrichment driver in configure().
|
||||
let mut tr = TriblerPlugin::new();
|
||||
tr.set_seed_config(TriblerConfigFile::from_env());
|
||||
// No env seed: the plugin reads config.json in configure() (sidecar URL,
|
||||
// api key, tmdb, meta-core URL) and builds its enrichment/self-publish
|
||||
// driver from config.meta_core_url there.
|
||||
let tr = TriblerPlugin::new();
|
||||
|
||||
let plugins: Vec<Box<dyn FeederPlugin>> = vec![Box::new(tr)];
|
||||
|
||||
|
||||
@@ -397,8 +397,8 @@ pub(crate) struct TmdbSearchItem {
|
||||
#[serde(default)]
|
||||
pub(crate) genre_ids: Vec<u32>,
|
||||
/// ISO 639-1 (2-letter) language the title was originally produced in.
|
||||
/// Mapped to `lang3` and used to file `original_title` under
|
||||
/// `titles/{lang3}` (METADATA_KEYS.md §3).
|
||||
/// Mapped to `lang3` and used to file `original_title` as a
|
||||
/// `titles/{lang3}/{name}` member (METADATA_KEYS.md §3).
|
||||
#[serde(default)]
|
||||
pub(crate) original_language: Option<String>,
|
||||
}
|
||||
@@ -438,8 +438,8 @@ impl TmdbSearchItem {
|
||||
/// Map an ISO 639-1 (2-letter) code to its ISO 639-3 (`lang3`) equivalent,
|
||||
/// matching the store's convention (`eng`, `jpn`, `fra`; the 639-2/T variant
|
||||
/// where B/T differ, e.g. `deu` not `ger`). Covers the languages TMDB
|
||||
/// commonly returns; unknown codes return `None` so the caller skips the
|
||||
/// `titles/{lang3}` write rather than persisting a non-`lang3` key.
|
||||
/// commonly returns; unknown codes return `None` so the caller files the name under `und`
|
||||
/// rather than persisting a non-`lang3` key.
|
||||
pub(crate) fn iso639_1_to_3(code: &str) -> Option<&'static str> {
|
||||
Some(match code.to_ascii_lowercase().as_str() {
|
||||
"en" => "eng",
|
||||
@@ -504,12 +504,53 @@ pub(crate) struct TmdbHit {
|
||||
impl TmdbHit {
|
||||
/// TMDB `original_language` (ISO 639-1) mapped to the store's `lang3`
|
||||
/// (ISO 639-3), or `None` when the language is outside the common set or
|
||||
/// absent. Used to file `original_title` under `titles/{lang3}` (§3).
|
||||
/// absent. Used to file `original_title` under `titles/{lang3}/{name}` (§3).
|
||||
pub(crate) fn original_lang3(&self) -> Option<&'static str> {
|
||||
iso639_1_to_3(self.original_language.as_deref()?.trim())
|
||||
}
|
||||
}
|
||||
|
||||
/// The language TMDB's `name`/`title` is written in. Every request goes out
|
||||
/// without a `language` param, so TMDB answers in its default, en-US.
|
||||
pub(crate) const METADATA_LANG3: &str = "eng";
|
||||
|
||||
/// The `titles/{lang3}/{name}` key-set member for `name` (METADATA_KEYS.md), or
|
||||
/// `None` when it normalises to empty: trimmed, whitespace runs collapsed,
|
||||
/// case/diacritics/`%` verbatim, and `/` (the key-set separator) written as
|
||||
/// U+2215 `∕`.
|
||||
///
|
||||
/// CROSS-BINARY CONTRACT: the indexer feeder's `tmdb.rs::title_member_key` and
|
||||
/// the card feeder's `tmdb_client.rs::title_member_key` write the same shape;
|
||||
/// two peers must produce the same key for one name.
|
||||
pub(crate) fn title_member_key(lang3: &str, name: &str) -> Option<String> {
|
||||
let name = name.split_whitespace().collect::<Vec<_>>().join(" ");
|
||||
if name.is_empty() {
|
||||
return None;
|
||||
}
|
||||
Some(format!("titles/{lang3}/{}", name.replace('/', "\u{2215}")))
|
||||
}
|
||||
|
||||
/// The `titles/{lang3}/{name}` members this feeder files from a TMDB hit: its
|
||||
/// title and original title (a search hit carries no AKAs). The original goes
|
||||
/// under its `original_language` (`und` when unmapped); the title under
|
||||
/// [`METADATA_LANG3`] **unless it equals the original** — TMDB falls back to
|
||||
/// the original name when it has no en-US translation, so `3%` is
|
||||
/// `titles/por/3%` only, not also an English name. Same rule as the indexer
|
||||
/// feeder's `tmdb_title_member_keys`.
|
||||
pub(crate) fn tmdb_title_member_keys(hit: &TmdbHit) -> Vec<String> {
|
||||
let norm = |s: &str| s.split_whitespace().collect::<Vec<_>>().join(" ");
|
||||
let mut keys = Vec::new();
|
||||
let original = hit.original_title.as_deref().map(norm).filter(|o| !o.is_empty());
|
||||
if let Some(o) = &original {
|
||||
keys.extend(title_member_key(hit.original_lang3().unwrap_or("und"), o));
|
||||
}
|
||||
let title = norm(&hit.title);
|
||||
if title != "(untitled)" && original.as_deref() != Some(title.as_str()) {
|
||||
keys.extend(title_member_key(METADATA_LANG3, &title));
|
||||
}
|
||||
keys
|
||||
}
|
||||
|
||||
/// Authoritative TV structure from `GET /3/tv/{id}`, used to bounds-check
|
||||
/// a title-parsed season/episode. Only the two structural fields are
|
||||
/// decoded; everything else in the (large) details payload is ignored.
|
||||
@@ -849,3 +890,38 @@ mod season_bounds_tests {
|
||||
assert_eq!(season_episode_bounds(&d, Some(-1), None), SeasonEpisodeBounds::Contradiction);
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod title_member_tests {
|
||||
use super::*;
|
||||
|
||||
fn hit(title: &str, orig: Option<&str>, lang: Option<&str>) -> TmdbHit {
|
||||
TmdbHit {
|
||||
tmdbid: 1,
|
||||
title: title.to_string(),
|
||||
original_title: orig.map(str::to_string),
|
||||
original_language: lang.map(str::to_string),
|
||||
overview: None,
|
||||
year: None,
|
||||
poster_path: None,
|
||||
genre_ids: Vec::new(),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn title_and_original_are_filed_as_language_nested_members() {
|
||||
assert_eq!(
|
||||
tmdb_title_member_keys(&hit("Naruto", Some("ナルト"), Some("ja"))),
|
||||
vec!["titles/jpn/ナルト", "titles/eng/Naruto"]
|
||||
);
|
||||
// Title == original (no en-US translation) is filed once.
|
||||
assert_eq!(tmdb_title_member_keys(&hit("3%", Some("3%"), Some("pt"))), vec!["titles/por/3%"]);
|
||||
// Unmapped language → und; `/` → U+2215; TMDB's placeholder is not a name.
|
||||
assert_eq!(
|
||||
tmdb_title_member_keys(&hit("(untitled)", Some("Fate/Zero"), Some("xx"))),
|
||||
vec!["titles/und/Fate\u{2215}Zero"]
|
||||
);
|
||||
assert_eq!(title_member_key("eng", " 3 Percent "), Some("titles/eng/3 Percent".to_string()));
|
||||
assert_eq!(title_member_key("eng", " "), None);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -92,39 +92,11 @@ pub struct TriblerConfigFile {
|
||||
}
|
||||
|
||||
impl TriblerConfigFile {
|
||||
/// Parse the config from the env (first-boot seed). Tribler-connection from
|
||||
/// `TRIBLER_*`; the enrichment TMDB key/language from `TMDB_TOKEN`
|
||||
/// (or `PROWLARR_TMDB_TOKEN`) / `TMDB_LANGUAGE` — so an operator can still seed
|
||||
/// them from env, while the dashboard form overrides at runtime.
|
||||
pub fn from_env() -> Self {
|
||||
let mut c = Self::default();
|
||||
if let Ok(u) = std::env::var("META_CORE_URL") {
|
||||
if !u.trim().is_empty() {
|
||||
c.meta_core_url = Some(u.trim().to_string());
|
||||
}
|
||||
}
|
||||
if let Ok(url) = std::env::var("TRIBLER_SIDECAR_URL") {
|
||||
if !url.trim().is_empty() {
|
||||
c.sidecar_url = Some(url);
|
||||
}
|
||||
}
|
||||
if let Ok(key) = std::env::var("TRIBLER_API_KEY") {
|
||||
if !key.trim().is_empty() {
|
||||
c.api_key = Some(key);
|
||||
}
|
||||
}
|
||||
if let Ok(t) = std::env::var("TMDB_TOKEN").or_else(|_| std::env::var("PROWLARR_TMDB_TOKEN")) {
|
||||
if !t.trim().is_empty() {
|
||||
c.tmdb_api_key = Some(t);
|
||||
}
|
||||
}
|
||||
if let Ok(l) = std::env::var("TMDB_LANGUAGE") {
|
||||
if !l.trim().is_empty() {
|
||||
c.tmdb_language = Some(l);
|
||||
}
|
||||
}
|
||||
c
|
||||
}
|
||||
// NOTE: `TriblerConfigFile::from_env()` was deliberately removed. Tribler
|
||||
// config (sidecar URL, api key, tmdb token/language, meta-core URL) is
|
||||
// dashboard config — it lives ONLY in the persisted config.json, never in
|
||||
// env. (Was: TRIBLER_SIDECAR_URL / TRIBLER_API_KEY / TMDB_TOKEN /
|
||||
// TMDB_LANGUAGE / META_CORE_URL — all now config-schema fields.)
|
||||
}
|
||||
|
||||
/// HTTP timeout for the *synchronous* calls (local search, remote-search
|
||||
@@ -223,9 +195,6 @@ pub struct TriblerPlugin {
|
||||
sidecar_url: String,
|
||||
api_key: Option<String>,
|
||||
cache: Option<MidhashCache>,
|
||||
/// Env-derived config seed (set by `main` via [`set_seed_config`]).
|
||||
/// `configure()` applies it when no persisted `config.json` exists.
|
||||
seed_config: TriblerConfigFile,
|
||||
/// Effective config snapshot, exposed (redacted) through the SDK config
|
||||
/// plane for the dashboard's per-plugin config form.
|
||||
cfg: TriblerConfigFile,
|
||||
@@ -261,7 +230,6 @@ impl TriblerPlugin {
|
||||
sidecar_url,
|
||||
api_key: None,
|
||||
cache: None,
|
||||
seed_config: TriblerConfigFile::default(),
|
||||
cfg: TriblerConfigFile::default(),
|
||||
enricher: None,
|
||||
inline_tmdb: crate::enrich::Enricher::new(),
|
||||
@@ -283,11 +251,8 @@ impl TriblerPlugin {
|
||||
self.api_key = Some(key);
|
||||
}
|
||||
|
||||
/// Install the env-derived seed config (applied by `configure()` when no
|
||||
/// persisted `config.json` exists). Mirrors the torznab feeder.
|
||||
pub fn set_seed_config(&mut self, cfg: TriblerConfigFile) {
|
||||
self.seed_config = cfg;
|
||||
}
|
||||
// set_seed_config was removed: there is no env seed. config.json is the only
|
||||
// config source (dashboard-written); apply_config takes it in configure().
|
||||
|
||||
/// Apply an effective config: override sidecar URL / API key when present,
|
||||
/// otherwise keep the constructor defaults. Records the effective snapshot
|
||||
@@ -565,6 +530,12 @@ impl TriblerPlugin {
|
||||
"movie"
|
||||
};
|
||||
rec.fields.insert("contentKind".to_string(), kind.to_string());
|
||||
if let Some(d) = meta_feeder_sdk::domain::domain_for_content_kind(kind) {
|
||||
rec.fields.insert("domain".to_string(), d.to_string());
|
||||
}
|
||||
if let Some(wf) = meta_feeder_sdk::domain::work_form_for_content_kind(kind) {
|
||||
rec.fields.insert("workForm".to_string(), wf.to_string());
|
||||
}
|
||||
}
|
||||
// For episode records, stamp the *parsed* season/episode so the consumer
|
||||
// renders a proper series with an ordered episode list (meta-watch keys
|
||||
@@ -636,10 +607,17 @@ impl TriblerPlugin {
|
||||
else {
|
||||
return vec![rec];
|
||||
};
|
||||
// Stamp the per-file CID in both shapes: the `cids/<cid>` key-set member
|
||||
// (what meta-share ingest and meta-watch's codec check actually read —
|
||||
// legacy `cid_*` names are no longer tolerated) plus the named field, kept
|
||||
// as an internal convergence key that the gateway's meta-core sink strips
|
||||
// before the record is stored.
|
||||
let with_cid = move |mut r: DiscoveryRecord, idx: u64| -> DiscoveryRecord {
|
||||
let cid = compute_bt_v1_file_cid(&ih20, idx);
|
||||
r.fields.entry(format!("cids/{cid}")).or_insert_with(|| "true".to_string());
|
||||
r.fields
|
||||
.entry("cid_btih_v1_file".to_string())
|
||||
.or_insert_with(|| compute_bt_v1_file_cid(&ih20, idx));
|
||||
.or_insert(cid);
|
||||
r
|
||||
};
|
||||
if is_obvious_single_file(&title) {
|
||||
@@ -665,15 +643,18 @@ impl TriblerPlugin {
|
||||
.collect();
|
||||
let per_file = |f: &TorrentFile, force_episode: bool| {
|
||||
let mut r = build_file_record(&rec.fields, &rec.record_id, f);
|
||||
r.fields.insert(
|
||||
"cid_btih_v1_file".to_string(),
|
||||
compute_bt_v1_file_cid(&ih20, f.index as u64),
|
||||
);
|
||||
let cid = compute_bt_v1_file_cid(&ih20, f.index as u64);
|
||||
// Key-set member first — it is the shape consumers read.
|
||||
r.fields.insert(format!("cids/{cid}"), "true".to_string());
|
||||
r.fields.insert("cid_btih_v1_file".to_string(), cid);
|
||||
if force_episode {
|
||||
// A multi-video pack is a TV season — force the kind so enrichment
|
||||
// resolves the show as TV (the show title drives the TMDB search).
|
||||
r.fields
|
||||
.insert("contentKind".to_string(), "episode".to_string());
|
||||
r.fields.insert("domain".to_string(), "screen".to_string());
|
||||
r.fields
|
||||
.insert("workForm".to_string(), "serial".to_string());
|
||||
}
|
||||
r
|
||||
};
|
||||
@@ -829,10 +810,12 @@ impl FeederPlugin for TriblerPlugin {
|
||||
// plane when the operator saves through the dashboard) wins over the env
|
||||
// seed. Both are optional — tribler has working defaults, so a fully
|
||||
// unconfigured feeder still serves (DEFAULT_SIDECAR_URL, no auth key).
|
||||
// config.json (dashboard-written) is the ONLY config source — no env
|
||||
// seed. Absent → tribler's built-in defaults (DEFAULT_SIDECAR_URL, no key).
|
||||
let effective = std::fs::read(cache_dir.join("config.json"))
|
||||
.ok()
|
||||
.and_then(|b| serde_json::from_slice::<TriblerConfigFile>(&b).ok())
|
||||
.unwrap_or_else(|| self.seed_config.clone());
|
||||
.unwrap_or_default();
|
||||
self.apply_config(effective);
|
||||
self.cache = Some(common::open_midhash_cache(cache_dir, "tribler")?);
|
||||
// Build the enrichment driver: infra (META_CORE_URL + plugin URLs + peer
|
||||
@@ -1448,7 +1431,6 @@ fn build_enricher(http: &reqwest::Client, cfg: &TriblerConfigFile) -> Option<Enr
|
||||
.as_ref()
|
||||
.map(|s| s.trim().to_string())
|
||||
.filter(|s| !s.is_empty())
|
||||
.or_else(|| EnrichmentConfig::from_env().map(|c| c.meta_core_url))
|
||||
.unwrap_or_else(|| meta_feeder_sdk::DEFAULT_META_CORE_URL.to_string());
|
||||
let mut ecfg = EnrichmentConfig::from_meta_core(meta_core_url);
|
||||
if let Some(k) = cfg.tmdb_api_key.as_ref().filter(|s| !s.trim().is_empty()) {
|
||||
|
||||
Reference in New Issue
Block a user