From d0389498edfaaeb2e2f7a03b5e5ede7a7166a3af Mon Sep 17 00:00:00 2001 From: Worph Date: Sun, 4 Oct 2026 12:30:13 +0000 Subject: [PATCH] MetaFeederUsenet: glue only (feeder 1.1.0 + enrichment), talks to the NNTmux app over its Newznab API; nntmux moved to worph/AppStore --- Apps/MetaFeederUsenet/docker-compose.yml | 767 ++--------------------- Apps/MetaFeederUsenet/rationale.md | 57 +- 2 files changed, 81 insertions(+), 743 deletions(-) diff --git a/Apps/MetaFeederUsenet/docker-compose.yml b/Apps/MetaFeederUsenet/docker-compose.yml index 1b8c045..8d7f118 100644 --- a/Apps/MetaFeederUsenet/docker-compose.yml +++ b/Apps/MetaFeederUsenet/docker-compose.yml @@ -1,313 +1,43 @@ name: metafeederusenet # ============================================================================= -# MetaFeeder · Usenet — a STANDALONE, self-hosted Usenet feeder for MetaMesh. +# MetaFeeder · Usenet — publishes YOUR OWN Usenet indexer to MetaMesh. # -# Unlike MetaFeeder · Prowlarr (which proxies an EXTERNAL Newznab/Torznab -# indexer and mints an `nzb-release` (0x1005) cid embedding that indexer's -# host), this app runs its OWN header scan (an `nntmux` — newznab-tmux fork — -# sidecar) so it can mint a PORTABLE `nzb-posting` (0x1003) cid: a digest over -# the article Message-ID set, embedding no indexer host, redeemable by ANY -# peer with a plain NNTP provider account. It ADDS a Usenet source; it does -# not replace the Prowlarr/Torznab one — a gateway query fans out to both. +# ⚠ NNTmux is a SEPARATE app (worph/AppStore) — install it first. This feeder +# reaches it on the shared `pcs` network at `http://nntmux` over its Newznab +# API (`/api/v1/api` + `/getnzb`), with the API key of NNTmux's `metamesh` +# user — the same contract MetaFeeder · Prowlarr uses with Prowlarr. It never +# touches NNTmux's database or storage. +# +# Unlike MetaFeeder · Prowlarr (which proxies EXTERNAL indexers and mints an +# `nzb-release` (0x1005) cid embedding that indexer's host), every release +# here comes from a scan you run, so the feeder downloads its .nzb once and +# mints a PORTABLE `nzb-posting` (0x1003) cid: a digest over the article +# Message-ID set, redeemable by ANY peer with a plain NNTP provider account. +# It also redeems external `nzb-release` locators with per-indexer keys set on +# its config page. +# +# A headless service: no web UI of its own. Configure it from MetaGateway → +# this feeder → *configure* (nntmux URL + API key, indexer keys). +# +# It needs, over `pcs`: **NNTmux** (`nntmux`), the shared **MetaCore** +# (`metacore-app:9000`, records + TMDB posters) and **MetaGateway**, which must +# list `http://metafeederusenet-feeder:8080` in its feeders config. # # Design record (meta-gateway repo): docs/others/self-hosted-usenet-indexer-study.md # Feeder repo: github.com/worph/meta-feeder-usenet -# -# It needs, over the shared `pcs` network: -# - the shared **MetaCore** app (`metacore-app:9000`) — where resolved -# records + TMDB posters are stored; the shared **MetaShare** app then -# announces them to the swarm. -# The **MetaGateway** app must list this feeder's URL -# (`http://metafeederusenet-feeder:8080`) in its feeders config so it -# discovers + queries it (MetaGateway, MetaCore and this feeder share `pcs`). -# -# Everything is scoped under the `metafeederusenet-*` prefix + the per-stack -# `metafeederusenet` network + /DATA/AppData/metafeederusenet/ data root, so it -# installs alongside the other Meta* apps with zero collisions. -# -# amd64 ONLY — `pyrowman/nntmux` ships no arm64 build. -# -# ⚠ Four silent-failure traps in the `pyrowman/nntmux` image, empirically -# verified (see the meta-gateway dev compose header for the full writeup): -# 1. `PATH_TO_NZBS` (and every other storage path var) must be ABSENT from -# the env, never present-but-blank — that reads as an explicit empty -# string, not "unset", and nntmux aborts NZB writing with 0 releases and -# no visible error. This compose bind-mounts a COMPLETE, hand-filled -# `.env` (below) rather than relying on individual `environment:` keys, -# for exactly this reason. -# 2. `DB_CONNECTION` must be `mariadb`, not `mysql` — the install migration -# loads its schema file by connection name; `mysql` finds no matching -# schema and fails partway through a from-scratch migration. -# 3. Manticore is a HARD first-boot dependency (unless -# `ELASTICSEARCH_ENABLED=true`, which just swaps it for a heavier one) — -# `metafeederusenet-manticore` below is not optional. -# 4. Seeded usenet groups default to `active=0, backfill=0` — scanning an -# inactive group silently does nothing. See the tips below for the SQL -# to activate one. -# One FIRST-BOOT-ONLY loud bug: the entrypoint runs `config:cache` before -# `key:generate`, so every session route 500s until you run -# `docker exec metafeederusenet-nntmux php artisan config:clear` once. See tips. -# -# ⚠ Four more, found the hard way on a live box (2026-07-26) after this stack -# came up with SEVEN healthy containers and returned ZERO results for every -# query — the feeder answering `"Done"` and nothing else. None of them logs -# anything; each one alone is enough to keep the catalog empty forever. -# -# 5. NEVER set a key in `nntmux.env` unless the value must DIFFER from the -# PHP default. `config/nntmux.php` does -# `env('NN_MULTIPROCESSING_MAX_CHILD_TIME', 1800)` and Laravel's `env()` -# returns STRINGS, so restating the default in the .env handed -# `Pool::timeout(float)` a `string("1800")` → TypeError → every Forking -# multiprocessing runner died on start. Those runners are exactly what a -# scan needs. This is trap #1's type-flavoured twin: a redundant -# restatement is not a no-op, it silently changes the value's TYPE. The -# key is therefore deliberately ABSENT from the seeded .env below (see the -# comment there); with it absent, `gettype()` reports `integer` and the -# runners execute. Same rule applies to any other key you are tempted to -# "document" by writing its default into the file. -# (Upstream record: meta-gateway `dev/docker-compose.feeder-usenet.yml`, -# the "CORRECTION (2026-07-26)" block.) -# 6. The image runs `frankenphp` and NOTHING ELSE — no cron, no supervisord, -# no queue worker. nntmux's own `settings.running=1` therefore starts -# nothing: there is no process to read it. Even with valid NNTP creds and -# active groups the catalog stays empty permanently. That is why -# `metafeederusenet-scanner` below exists — it is the missing loop, run as -# a SECOND container off the same image so the web container's -# install/populate entrypoint stays untouched. -# 7. The nntmux entrypoint RE-SEEDS `settings` and `usenet_groups` on EVERY -# container boot. Configuration applied before a restart silently reverts -# (observed: `running`→0, `delaytime`→2, every group→`active=0`). Apply -# group activation and scheduler settings AFTER the final restart, and -# re-check them after any `docker restart` / stack update. -# 8. `settings.delaytime` (HOURS, default 2) stalls a backfilled catalog: -# `ProcessReleases` refuses to advance a collection until it has aged past -# it. That is correct for live tailing (a posting may still be uploading) -# and pointless for backfill, where every article is already days old — -# and the symptom, collections climbing while releases flatline, is -# indistinguishable from a broken pipeline because nothing logs an error. -# Because trap #7 re-seeds it back to 2 on every restart, the scanner -# loop below ASSERTS `delaytime=0` on each pass **while `backfill=1`** — -# the one setting it writes, making the tier self-healing across -# restarts. It is scoped to the backfill phase on purpose: with -# `backfill=0` (pure forward tailing) the operator's own value stands. # ============================================================================= services: - # --------------------------------------------------------------------------- - # metafeederusenet-db — MariaDB backing nntmux's OWN release catalog. Not - # shared with the gateway's meta-core Redis or anything else in the mesh. - # --------------------------------------------------------------------------- - metafeederusenet-db: - image: mariadb:12.3 - cpu_shares: 70 - container_name: metafeederusenet-db - hostname: metafeederusenet-db - restart: unless-stopped - # 2g, not the 512m this shipped with: 512m is fine for an empty schema and - # far too small once the catalog is real. A header scan of a busy binary - # group builds `collections`/`binaries`/`parts` tables in the tens of - # millions of rows, and InnoDB starved of buffer pool turns collation into - # a disk-thrash that never catches up. Measured on a live box. - mem_limit: 2g - memswap_limit: 2g - environment: - MARIADB_DATABASE: nntmux - MARIADB_USER: nntmux - MARIADB_PASSWORD: "$APP_DEFAULT_PASSWORD" - MARIADB_RANDOM_ROOT_PASSWORD: "yes" - volumes: - - ${DATA_ROOT:-/DATA}/AppData/metafeederusenet/db:/var/lib/mysql - healthcheck: - test: ["CMD", "healthcheck.sh", "--connect", "--innodb_initialized"] - interval: 5s - timeout: 5s - retries: 20 - start_period: 20s - networks: - - metafeederusenet - - # --------------------------------------------------------------------------- - # metafeederusenet-redis — nntmux's OWN Redis. Deliberately separate from the - # gateway core's Redis (different failure domain; losing this one is free). - # --------------------------------------------------------------------------- - metafeederusenet-redis: - image: redis:7-alpine - cpu_shares: 70 - container_name: metafeederusenet-redis - hostname: metafeederusenet-redis - restart: unless-stopped - mem_limit: 128m - memswap_limit: 128m - networks: - - metafeederusenet - - # --------------------------------------------------------------------------- - # metafeederusenet-manticore — RT-index search backend. NOT optional (trap - # #3 above) — a hard first-boot dependency of nntmux's own entrypoint, even - # though this feeder reads `releases` via SQL directly and never touches it. - # Digest-pinned (upstream reuses low-looking version numbers across an old - # semver line and a newer CalVer-ish one; the digest is unambiguous). - # --------------------------------------------------------------------------- - metafeederusenet-manticore: - image: manticoresearch/manticore@sha256:032388dc77321e3ac898a14b9a0c3fc22ab8d08cd2582abd6c5b642d46b82181 - cpu_shares: 50 - container_name: metafeederusenet-manticore - hostname: metafeederusenet-manticore - restart: unless-stopped - mem_limit: 512m - memswap_limit: 512m - ulimits: - nofile: { soft: 65535, hard: 65535 } - networks: - - metafeederusenet - - # --------------------------------------------------------------------------- - # metafeederusenet-nntmux — the header scanner + web UI/API. Exposed DIRECTLY - # via Caddy (no AppShield gate): nntmux ships its own admin login - # (ADMIN_USER/ADMIN_PASS, seeded below), the same convention this store uses - # for every app that brings its own auth (Jellyfin, Nextcloud, Immich, ...). - # --------------------------------------------------------------------------- - metafeederusenet-nntmux: - # Digest-pinned, NOT :latest — a silent tag move could change the - # `misc/update/*.php` command surface the "collate via tinker" tip below - # depends on. Re-pin deliberately, re-verify the trap list above when you do. - image: pyrowman/nntmux@sha256:7fc221f7a806fee3e4a2145159e1a147962edc095282bdc692806968f791fabe - cpu_shares: 50 - container_name: metafeederusenet-nntmux - hostname: metafeederusenet-nntmux - restart: unless-stopped - # The working set scales with scan breadth (group set x backfill depth), an - # operator choice (see tips), so this is a ceiling rather than a size: 3g - # is what a real corpus needed on a live box — 1536m was reached and - # exceeded by a single busy group's header scan. Raise it if you scan wide. - mem_limit: 3g - memswap_limit: 3g - environment: - # Shell-script-consumed vars — the entrypoint's own `mkdir -p` step reads - # these from the PROCESS environment, not the mounted .env. Keep in - # lockstep with the matching keys in nntmux.env below. - PATH_TO_NZBS: /app/storage/nzb/ - COVERS_PATH: /app/storage/covers - TEMP_UNRAR_PATH: /app/storage/tmp/unrar/ - TEMP_UNZIP_PATH: /app/storage/tmp/unzip/ - ELASTICSEARCH_ENABLED: "false" - volumes: - # The complete, hand-filled .env (trap #1) — a full-file mount rather - # than individual `environment:` keys. Writable: `nntmux:install` - # persists a generated APP_KEY back into it on first boot. - - ${DATA_ROOT:-/DATA}/AppData/metafeederusenet/nntmux-env/nntmux.env:/app/.env:rw - # The .nzb store — shared with metafeederusenet-feeder below, - # read-write here (nntmux writes), read-only there (feeder only reads). - - ${DATA_ROOT:-/DATA}/AppData/metafeederusenet/nntmux-storage/nzb:/app/storage/nzb:rw - healthcheck: - test: ["CMD", "curl", "-sf", "-o", "/dev/null", "http://localhost/"] - interval: 10s - timeout: 5s - retries: 15 - start_period: 30s - depends_on: - metafeederusenet-db: - condition: service_healthy - metafeederusenet-redis: - condition: service_started - metafeederusenet-manticore: - condition: service_started - labels: - caddy_0: metafeederusenetnntmux-${APP_DOMAIN} - caddy_0.import: gateway_tls - caddy_0.reverse_proxy: "{{upstreams 80}}" - caddy_1: metafeederusenetnntmux-${APP_PUBLIC_IP_DASH}.nip.io - caddy_1.import: gateway_tls - caddy_1.reverse_proxy: "{{upstreams 80}}" - caddy_2: metafeederusenetnntmux-${APP_PUBLIC_IP_DASH}.sslip.io - caddy_2.reverse_proxy: "{{upstreams 80}}" - networks: - - metafeederusenet - - pcs - - # --------------------------------------------------------------------------- - # metafeederusenet-scanner — the SCAN LOOP (trap #6). The nntmux image runs - # frankenphp and nothing else, so without this container nothing ever calls - # the scan/backfill/collate entry points and the catalog stays empty forever, - # no matter how correct the credentials, the groups or `settings.running` are. - # - # It is a second container off the SAME digest-pinned image, sharing the same - # `/app/.env` and the same nzb store, rather than a second process inside the - # web container — the web container's entrypoint runs install/populate under - # `set -e` and must keep it; here it is REPLACED by `entrypoint:` below so - # that install/populate does NOT run twice against one database. - # - # Every knob (interval, binaries on/off, backfill on/off, backfill_qty, - # backfill_days, running) is read from nntmux's own `settings` table at - # /admin/tmux-edit — deliberately NOT from new env vars here, which would - # make two sources of truth for one number. - # --------------------------------------------------------------------------- - metafeederusenet-scanner: - # MUST stay byte-identical to metafeederusenet-nntmux's pin above: the two - # containers share one `/app/.env` and one database schema, and the script - # below calls `misc/update/*.php` + `Blacklight\` classes by name. - image: pyrowman/nntmux@sha256:7fc221f7a806fee3e4a2145159e1a147962edc095282bdc692806968f791fabe - cpu_shares: 20 - container_name: metafeederusenet-scanner - hostname: metafeederusenet-scanner - restart: unless-stopped - # Bypasses the image's own install/populate entrypoint on purpose — see the - # block comment above. Only the web container may run it. - entrypoint: ["/bin/sh", "/scanner.sh"] - # The image ships a HEALTHCHECK that curls its own web server. This - # container deliberately runs no web server — its entrypoint IS the scan - # loop — so the inherited probe can only ever fail. Observed live: a - # scanner actively backfilling and writing NZBs sitting at - # `Up 5 minutes (unhealthy)`. - # - # That is not cosmetic. A permanently-red container that is in fact working - # trains the operator to ignore health status, which is precisely how this - # stack's empty-catalog failure stayed invisible for so long — the feeder's - # own health() reporting `ok` over a zero-row catalog was the same bug from - # the other direction. Liveness for this container is its LOG - # (`docker logs -f metafeederusenet-scanner`), not a probe; a probe that - # cannot be honest is worse than none. - healthcheck: - disable: true - # A backfill pass holds a group's article range in memory; 2g is the - # headroom measured alongside nntmux's own 3g on a live box. - mem_limit: 2g - memswap_limit: 2g - environment: - # Same shell-script-consumed vars as metafeederusenet-nntmux: the wrapper - # scripts read these from the PROCESS environment, not from the mounted - # .env (trap #1). Keep the two blocks in lockstep. - PATH_TO_NZBS: /app/storage/nzb/ - COVERS_PATH: /app/storage/covers - TEMP_UNRAR_PATH: /app/storage/tmp/unrar/ - TEMP_UNZIP_PATH: /app/storage/tmp/unzip/ - ELASTICSEARCH_ENABLED: "false" - volumes: - # The same .env the web container filled in — one credential set, one - # APP_KEY. Read-write only because Laravel's .env handling expects it. - - ${DATA_ROOT:-/DATA}/AppData/metafeederusenet/nntmux-env/nntmux.env:/app/.env:rw - # nntmux writes .nzb.gz here; the feeder reads them read-only. - - ${DATA_ROOT:-/DATA}/AppData/metafeederusenet/nntmux-storage/nzb:/app/storage/nzb:rw - # The loop itself, provisioned by pre-install-cmd below. - - ${DATA_ROOT:-/DATA}/AppData/metafeederusenet/scanner/scanner.sh:/scanner.sh:ro - depends_on: - # Not just `started`: the loop's very first action is an artisan call, so - # it must not race the web container's install/migrate step. - metafeederusenet-nntmux: - condition: service_healthy - networks: - - metafeederusenet - # --------------------------------------------------------------------------- # metafeederusenet-feeder — the usenet feeder sidecar (meta-feeder-usenet). - # Reads nntmux's release catalog + shared .nzb store; self-publishes - # resolved records + seeds the `.nzb` manifest to the shared MetaCore. The + # Searches NNTmux over its Newznab API; self-publishes resolved records + + # seeds the `.nzb` manifest to the shared MetaCore. The # gateway reaches it directly on `pcs`; the human configures it from # the MetaGateway dashboard (configure panel). # --------------------------------------------------------------------------- metafeederusenet-feeder: - image: ghcr.io/worph/meta-feeder-usenet:1.0.6 + image: ghcr.io/worph/meta-feeder-usenet:1.1.0 cpu_shares: 30 container_name: metafeederusenet-feeder # Hard RAM cap, swap disabled (memswap_limit == mem_limit) → OOM instead @@ -321,7 +51,7 @@ services: - 8080 environment: # INFRA-ONLY env (the "config: no env if in webui" rule). Operator - # config beyond the two DB/path seeds below — TMDB token / language, + # config beyond the NNTmux URL seed below — TMDB token / language, # meta-core URL — is read from the persisted config.json (this # feeder's config page, opened from the MetaGateway dashboard (configure panel)). These env vars are # wiring only: the enrichment sidecars, the enrich callback, the peer id. @@ -335,14 +65,13 @@ services: META_FEEDER_CALLBACK_URL: "http://metafeederusenet-feeder:8080/enrich/callback" # First-boot SEED only (the persisted config.json wins on every later # restart — change these on the /config page after first boot, not here). - NNTMUX_DB_URL: "mysql://nntmux:${APP_DEFAULT_PASSWORD}@metafeederusenet-db:3306/nntmux" - NNTMUX_NZB_PATH: "/nntmux-nzb" + # The API key has no env seed: paste NNTmux's `metamesh` key on the page. + NNTMUX_URL: "http://nntmux" TZ: "$TZ" volumes: - # Persist feeder config (config.json saved via the config UI) + redb cache. + # Persist feeder config (config.json saved via the config UI), the redb + # cache, and the per-release .nzb cache (gateway/usenet/nntmux-nzb/). - ${DATA_ROOT:-/DATA}/AppData/metafeederusenet/feeder-state:/data/meta-feeder:rw - # Same volume nntmux writes .nzb.gz into — read-only here. - - ${DATA_ROOT:-/DATA}/AppData/metafeederusenet/nntmux-storage/nzb:/nntmux-nzb:ro healthcheck: test: ["CMD", "curl", "-sf", "http://localhost:8080/health"] interval: 5s @@ -350,8 +79,6 @@ services: retries: 10 start_period: 5s depends_on: - metafeederusenet-nntmux: - condition: service_healthy metafeederusenet-filename-parser: condition: service_started metafeederusenet-tmdb: @@ -420,68 +147,40 @@ x-casaos: description: en_us: | - **A standalone, self-hosted Usenet feeder for the MetaMesh network.** + **Publish your own Usenet indexer to the MetaMesh network.** - MetaFeeder · Usenet runs its own Usenet header scan (a bundled - **nntmux** indexer) instead of proxying a third-party Newznab site. - Because it finds article Message-IDs itself at scan time, it mints a - **portable** posting reference — no indexer host baked in, redeemable by - any peer with a plain NNTP provider account. It adds a Usenet source + MetaFeeder · Usenet searches your **NNTmux** app (worph/AppStore) over + its Newznab API and turns each release into a **portable** MetaMesh + posting — no indexer host baked in, redeemable by any peer with a plain + NNTP provider account — with a clean title, poster and plot from the + bundled filename-parser + TMDB enrichment. It adds a Usenet source alongside **MetaFeeder · Prowlarr**; it doesn't replace it. - **Self-contained:** ships its own nntmux scanner (MariaDB + Redis + - Manticore) + filename-parser + TMDB enrichment, so each release gets a - clean title, poster and plot. + It also **redeems** releases found through Prowlarr's Newznab indexers, + with the indexer keys you set on its config page. - **You still need a Usenet provider account** (NNTP login) — this app - scans headers, it does not supply an account, and nothing at all is - indexed until you supply one. Configure it after install (see the - install tips). Keep the connection count low: this scanner and your - playback share the same provider connection slots. - - **The catalog is not instant.** Once configured, a bundled scanner - container runs the header-scan / backfill / collation loop on the - interval you set. A first pass over a few busy binary groups pulls - tens of GB and runs for hours; a deep backfill is hundreds of GB. The - install tips show which knobs bound that. + **Install NNTmux first.** No web UI of its own: configure it from the + **MetaGateway** dashboard (nntmux URL + the `metamesh` API key). **Also requires:** **MetaGateway** — its meta-core stores the resolved - records and announces them to the swarm. The gateway must list this - feeder in its config. + records and announces them to the swarm. fr_fr: | - **Un feeder Usenet autonome et auto-hébergé pour le réseau MetaMesh.** + **Publiez votre propre indexeur Usenet sur le réseau MetaMesh.** - MetaFeeder · Usenet effectue lui-même le scan des en-têtes Usenet (un - indexeur **nntmux** embarqué) plutôt que de passer par un site Newznab - tiers. Comme il retrouve les Message-ID des articles dès le scan, il - génère une référence de posting **portable** — sans hôte d'indexeur - intégré, exploitable par n'importe quel pair disposant d'un simple - compte NNTP. Il ajoute une source Usenet en plus de **MetaFeeder - - Indexer**, sans le remplacer. + MetaFeeder · Usenet interroge votre app **NNTmux** (worph/AppStore) via + son API Newznab et transforme chaque release en posting MetaMesh + **portable**, exploitable par n'importe quel pair disposant d'un simple + compte NNTP. Il ajoute une source Usenet en plus de **MetaFeeder · + Prowlarr**. - **Autonome :** embarque son scanner nntmux (MariaDB + Redis + Manticore) - + ses plugins filename-parser + TMDB. - - **Un compte fournisseur Usenet (NNTP) reste nécessaire** — cette app - scanne les en-têtes, elle ne fournit pas d'accès, et rien n'est indexé - tant que vous n'en fournissez pas un. À configurer après installation - (voir les conseils d'installation). Gardez un nombre de connexions - faible : le scan et la lecture partagent les mêmes créneaux du - fournisseur. - - **Le catalogue ne se remplit pas instantanément.** Une fois configuré, - un conteneur de scan embarqué exécute la boucle scan / backfill / - collation à l'intervalle que vous choisissez. Un premier passage sur - quelques gros groupes binaires télécharge des dizaines de Go et dure - des heures ; un backfill profond, des centaines de Go. - - **Nécessite aussi :** l'app **MetaGateway** (son meta-core stocke et - annonce les enregistrements). + **Installez NNTmux d'abord.** Pas d'interface propre : configuration + depuis le tableau de bord **MetaGateway** (URL nntmux + clé API + `metamesh`). **Nécessite aussi** l'app **MetaGateway**. tagline: - en_us: Self-hosted Usenet header scanner (bundled nntmux) for the MetaMesh network - fr_fr: Scanner Usenet auto-hébergé (nntmux inclus) pour le réseau MetaMesh + en_us: Publishes your NNTmux Usenet indexer to MetaMesh as portable postings + fr_fr: Publie votre indexeur Usenet NNTmux sur MetaMesh title: en_us: MetaFeeder · Usenet @@ -499,342 +198,26 @@ x-casaos: tips: before_install: en_us: | - **Install order:** install **MetaGateway** first (this feeder - publishes to its shared meta-core). + **Install order:** **NNTmux** (worph/AppStore) and **MetaGateway** first. - **After installing this app, a Usenet provider account is required** - (this scans headers itself — it needs a plain NNTP login): - 1. Edit `/DATA/AppData/metafeederusenet/nntmux-env/nntmux.env` (via - SSH or a file-manager app) and fill in `NNTP_SERVER`, `NNTP_PORT`, - `NNTP_USERNAME`, `NNTP_PASSWORD` (use the SAME provider account - your MetaShare/meta-share peer is already credentialed against — - one account, two consumers). Keep `NNTP_CONNECTIONS` low (1-2): - the scanner and playback share the same provider connection slots, - so a greedy value here stalls playback mid-stream elsewhere. - 2. Restart the nntmux container: `docker restart metafeederusenet-nntmux`. - 3. **First boot only:** nntmux's install sequence caches its config - before generating an encryption key, so every page 500s until you - run `docker exec metafeederusenet-nntmux php artisan config:clear` - once. (`curl -sI https://metafeederusenetnntmux-/` — a - `500` means you need this; a `302` redirect to `/login` means - you're fine.) - 4. Log in to the nntmux dashboard at the **MetaFeeder Usenet nntmux** - link (`ADMIN_USER=admin`, password = this app's auto-generated - default password — check CasaOS's app credentials panel) and - change it. - - **⚠ Do steps 5-6 AFTER your LAST restart, and re-check them after - every later restart or stack update.** The nntmux entrypoint RE-SEEDS - the `settings` and `usenet_groups` tables on EVERY container boot, so - anything you configure before a restart silently reverts (observed: - `running` back to 0, `delaytime` back to 2, every group back to - `active=0`) — with no error anywhere. - - 5. **Activate at least one usenet group** — seeded groups start - inactive (`active=0`), and scanning an inactive group silently - touches nothing at all: + **After installing:** + 1. Read NNTmux's `metamesh` API key (the NNTmux app provisions this user, + with raised daily API/download caps): ``` - docker exec metafeederusenet-db mariadb -unntmux -p"" nntmux \ - -e "UPDATE usenet_groups SET active=1, backfill=1 WHERE name IN ('alt.binaries.teevee');" + docker exec nntmux php artisan tinker --execute="echo App\Models\User::where('username','metamesh')->value('api_token');" ``` - 6. **Start the scheduler** in nntmux's admin UI (**/admin/tmux-edit**) - or directly in the `settings` table: set `running=1`, pick your - `monitor_delay` (seconds between passes), and turn on `binaries` - and/or `backfill` with a `backfill_qty` / `backfill_days` you're - happy to pay for. The bundled `metafeederusenet-scanner` container - polls these settings and runs the scan/backfill/collate loop for - you — the nntmux image itself ships NO cron and NO queue worker, so - this container is what actually makes a scan happen. There are - deliberately no env vars for these knobs: the `settings` table is - the single source of truth. + 2. In **MetaGateway**, make sure this feeder is listed in its feeders + config (`http://metafeederusenet-feeder:8080`), then open its config + panel (*configure*): paste the key into **nntmux API key** (the URL + defaults to `http://nntmux`) and Save — the feeder restarts itself. + 3. Optionally add **indexer keys** there too, to redeem releases found + through Prowlarr's Newznab indexers, and a TMDB token for posters. - **`delaytime` (HOURS, default 2) will stall a backfilled catalog.** - nntmux won't turn a collection into a release until it has aged - past `delaytime`, which is right for tailing live posts and - pointless when backfilling history — and the symptom (collections - climbing, releases flat, zero query results) looks exactly like a - broken pipeline. **While `backfill=1`, the scanner forces - `delaytime=0` for you on every pass** and logs - `asserting delaytime=0 for the backfill phase (was N)` when it does - — it has to re-assert it because a container restart re-seeds the - value back to 2. This is the only setting the loop ever writes. + **Searches must be typed** (`fileType:video`, `contentKind:episode`…) to + reach this feeder — a plain search-box query never fans out to it. - So if you want a different steady-state `delaytime`, **set - `backfill=0` first**, then set your value — otherwise the loop will - keep pulling it back to 0. With `backfill=0` the scanner never - touches it and `1`-`2` is the sensible live-tailing value. - - **Budget the first scan.** A first header pass over a few busy - binary groups pulls tens of GB and runs for hours; a - full-retention backfill is hundreds of GB. `backfill_days` is the - knob that bounds it — start small, widen once you see releases. - - 7. Open its config panel (the **MetaGateway** dashboard → this feeder → *configure*) to optionally set a - TMDB v4 token for posters, then Save. - 8. In **MetaGateway**, make sure this feeder is listed in its feeders - config (`http://metafeederusenet-feeder:8080`) so it gets - discovered. - - **Note:** which groups to scan, backfill depth and scan interval are - your call as the operator — pick them for your own catalog. - - **Watch the loop:** `docker logs -f metafeederusenet-scanner` prints - one line per pass (`header scan` / `backfill direct` / `collate - ` / `sleeping`). `settings.running=0` shows up there as - `idle`. - - **Authentication:** the feeder's config page opens from the **MetaGateway** - dashboard (behind its login); nntmux uses its own login (seeded above). - - pre-install-cmd: | - # Seed nntmux's .env (trap #1 — must be a complete file, see header). - # NNTP_* are deliberately left blank; fill them in per the install tips. - if [ ! -f /DATA/AppData/metafeederusenet/nntmux-env/nntmux.env ]; then - cat > /DATA/AppData/metafeederusenet/nntmux-env/nntmux.env < /DATA/AppData/metafeederusenet/scanner/scanner.sh <<'SCANNER_SH' - #!/bin/sh - # nntmux scan launcher. - # - # The pyrowman/nntmux image runs frankenphp and NOTHING else: no cron, no - # supervisord, no queue worker. So `settings.running=1` changes nothing on - # its own, because nothing ever starts the loop. This is that loop. - # - # It runs as a SECOND CONTAINER off the same image + the same /app/.env and - # nzb volume, rather than as a second process inside the web container, so - # the web container's entrypoint (install/populate under `set -e`) is left - # untouched. - # - # INVOCATION IS NOT FREE CHOICE. Measured PER RUNNER on a live box, NOT a - # blanket "Forking works": - # * multiprocessing/binaries.php — Forking OK. - # * multiprocessing/backfill.php — NO-OP. Reports "...3 job(s) to do - # using a max of 0 child process(es)" and returns in 0s having done - # nothing. Backfill must be driven DIRECTLY, per group (see the - # backfill block below). - # * collation MUST go through tinker/ProcessReleases DIRECTLY. - # multiprocessing/releases.php aborts every batch on a benign PHP - # warning (ReleaseCleaning.php:165) that turns fatal inside the Spatie - # child -> 0 releases while collections pile into the tens of - # thousands. - # - # Every knob comes from nntmux's own `settings` table (edited at - # /admin/tmux-edit), never from env — otherwise there are two sources of - # truth for one number. - set -u - - cd /app || exit 1 - - setting() { php artisan tinker --execute="echo DB::table('settings')->where('name','$1')->value('value');" 2>/dev/null | tr -dc '0-9'; } - set_setting() { php artisan tinker --execute="DB::table('settings')->where('name','$1')->update(['value'=>'$2']);" >/dev/null 2>&1; } - groups() { php artisan tinker --execute="foreach (DB::table('usenet_groups')->where('active',1)->pluck('name') as \$g) echo \$g, PHP_EOL;" 2>/dev/null | grep -E '^[a-z0-9.]+$'; } - - log() { echo "[scanner $(date -u +%H:%M:%S)] $*"; } - - log "launcher started" - while true; do - if [ "$(setting running)" != "1" ]; then - log "settings.running=0 — idle" - sleep 30 - continue - fi - - if [ "$(setting binaries)" = "1" ]; then - qty="$(setting backfill_qty)"; [ -n "$qty" ] || qty=20000 - log "header scan (qty=$qty)" - php misc/update/multiprocessing/binaries.php "$qty" 2>&1 | tail -3 - fi - - # Backfill must run DIRECT, per group. multiprocessing/backfill.php - # reports "max of 0 child process(es)" and silently does nothing — the - # Forking wrapper leaves maxProcesses=0 for this runner specifically. - # Depth is capped by settings.backfill_days, so this reaches back that - # far and then stops advancing; it is NOT a full-retention backfill. - if [ "$(setting backfill)" = "1" ]; then - # The ONLY setting this loop writes. `delaytime` (hours, default 2) - # blocks ProcessReleases from advancing a collection until it ages - # past it. Correct for the leading edge — a posting may still be - # uploading — but pointless for backfill, where every article is - # already days old. Left at 2 during backfill it stalls the catalog - # for two hours with a failure mode indistinguishable from a broken - # pipeline (collections climb, releases flatline, nothing logs an - # error). - # - # Asserted every pass, not once at boot, because the nntmux - # entrypoint RE-SEEDS `settings` on every container start (trap #7) - # — a restart silently puts this back to 2 and the tier quietly - # stops producing. Scoped to `backfill=1` so a pure forward-tailing - # deployment (`backfill=0`) keeps the operator's own value: this is - # a per-phase mode, not a preference we get to override. - if [ "$(setting delaytime)" != "0" ]; then - log "asserting delaytime=0 for the backfill phase (was $(setting delaytime))" - set_setting delaytime 0 - fi - qty="$(setting backfill_qty)"; [ -n "$qty" ] || qty=20000 - log "backfill direct (backfill_days=$(setting backfill_days), qty=$qty)" - groups | while read -r g; do - [ -n "$g" ] || continue - php misc/update/backfill.php "$g" "$qty" 2>&1 | grep -iE "articles|backfill|Group|no new" | tail -2 - done - fi - - # Collation, per active group, DIRECT (never multiprocessing/releases.php). - # Never wrap this in a short `timeout`: killing it between the release - # INSERT and createNZBs() leaves a release row with no .nzb.gz, which - # the feeder then correctly skips forever. - groups | while read -r g; do - [ -n "$g" ] || continue - log "collate $g" - php artisan tinker --execute="(new Blacklight\\processing\\ProcessReleases)->processReleases(1, 1, \"$g\", new Blacklight\\NNTP);" 2>&1 | grep -iE "releases (ready|created)|Added [0-9]+ releases|NZB" | tail -3 - done - - delay="$(setting monitor_delay)"; [ -n "$delay" ] || delay=30 - log "pass complete; sleeping ${delay}s" - sleep "$delay" - done - SCANNER_SH - chmod 755 /DATA/AppData/metafeederusenet/scanner/scanner.sh + Each release's .nzb is downloaded from NNTmux once and cached here + (`feeder-state/gateway/usenet/nntmux-nzb/`). # Maison (Yundera dashboard) configuration. Read in preference to x-casaos. # schema_version 2 is required for `folders`: an older Maison refuses the app @@ -845,26 +228,10 @@ x-compose-app: # runs and before the containers start — on the first boot and every one after. # Compose would otherwise create a missing bind source as a root-owned dir. folders: - - path: /DATA/AppData/metafeederusenet/db - user: $PUID - group: $PGID - - path: /DATA/AppData/metafeederusenet/nntmux-env - user: $PUID - group: $PGID - - path: /DATA/AppData/metafeederusenet/nntmux-storage/nzb - user: $PUID - group: $PGID - path: /DATA/AppData/metafeederusenet/feeder-state user: $PUID group: $PGID - path: /DATA/AppData/metafeederusenet/tmdb-cache user: $PUID group: $PGID - - path: /DATA/AppData/metafeederusenet/scanner - user: $PUID - group: $PGID - # Same string as nntmux's caddy_0 label, so the tile link and the route agree. - # The tile opens nntmux's own admin UI (its own login, its own Caddy route). - # The feeder itself is configured from the MetaGateway dashboard. - webui-host: metafeederusenetnntmux-${domain} - webui-path: "/" + # No web UI: Maison files it under Services. Configured from MetaGateway. diff --git a/Apps/MetaFeederUsenet/rationale.md b/Apps/MetaFeederUsenet/rationale.md index a41e425..af477ff 100644 --- a/Apps/MetaFeederUsenet/rationale.md +++ b/Apps/MetaFeederUsenet/rationale.md @@ -3,62 +3,33 @@ ## What deviation / exception is being requested 1. `metafeederusenet-feeder` runs `user: 0:0`. -2. The **NNTmux web UI is published on its own hostname** - (`metafeederusenetnntmux-.`) with no AppShield sidecar in front of it — it - is protected by NNTmux's own login only. -3. The seeded `nntmux.env` contains TMDB and TVDB API keys as literals. + +(The NNTmux indexer this feeder used to bundle — and its own exceptions: a web UI on its own +login, seeded TMDB/TVDB keys — is now the separate **NNTmux** app in worph/AppStore, with its +own `rationale.md`.) ## Why it is necessary 1. The feeder publishes resolved records to the shared MetaCore and writes plugin artwork through its WebDAV endpoint; those targets are created and owned by MetaCore's root-running container. -2. NNTmux is a Laravel application with its own user table, session handling and admin - role. Its admin account is created at install from `$APP_DEFAULT_PASSWORD`, so - authentication is **enabled by default** — the "app's own built-in auth" alternative the - security checklist accepts. Fronting it with AppShield as well would double-prompt on - every page and breaks its own OTP and password-reset flows. -3. Those two keys are NNTmux's upstream project defaults, shipped in its own - `.env.example`. They are shared read-only application keys for public metadata catalogs, - not user credentials, and they are what makes the app work immediately after - installation with no manual configuration. An operator can replace them from the NNTmux - admin UI. ## Security mitigations in place -- The NNTmux admin password is `$APP_DEFAULT_PASSWORD` — the platform-generated secret — - and the database password comes from the same variable. Neither is a hardcoded literal, - and both are surfaced to the user in `tips.before_install`. -- The feeder's own configuration UI *is* behind AppShield + Authelia SSO - (`ghcr.io/yundera/appshield:2.0.9`). -- `metafeederusenet-db`, `-redis` and `-manticore` are on the app-private - `metafeederusenet` bridge network, carry no Caddy labels and publish no host port, so no - sibling app on the shared `pcs` network can resolve or dial them. -- NNTP provider credentials are deliberately left **blank** in the seed; the user fills - them in per `tips.before_install`. Nothing ships with a working news account. -- The `nntmux.env` seed is guarded by `[ ! -f … ]`, so a reinstall or version upgrade never - overwrites the operator's provider settings. `scanner.sh` is intentionally rewritten on - every run — it is shipped code, not operator config, so a store update must be able to - replace a stale copy; the write is idempotent. -- No user directories are mounted. All binds are under `/DATA/AppData/metafeederusenet/`, - declared in `x-compose-app.folders`. -- `cpu_shares` on all nine services; every image pinned (the NNTmux and Manticore images by - digest). +- No web UI and no Caddy route: the feeder's configuration page is reached only through the + MetaGateway dashboard's proxy, behind the gateway's login. +- It talks to NNTmux only over NNTmux's Newznab API, with the key of a dedicated + `metamesh` user — it holds no database credential and mounts no NNTmux storage. +- The API key and the indexer keys are secrets on the feeder's config page (write-only, + never echoed back); none has an env seed or a default. +- `cpu_shares` and memory caps on all three services; every image pinned. ## Alternatives considered and rejected -- **AppShield in front of NNTmux.** Two independent login gates on one UI; its OTP, - password-reset and API endpoints break behind the OIDC redirect. -- **Blank the TMDB/TVDB keys.** The app then installs into a state where covers and - metadata silently fail until the user finds two upstream registration flows — a direct - violation of "works immediately after installation, no manual configuration required for - basic functionality". - **Run the feeder as `$PUID:$PGID`.** Writes into MetaCore's root-owned tree fail. ## Data protection -All state — MariaDB, the NZB store, the feeder cursor, the TMDB cache — is under -`/DATA/AppData/metafeederusenet/`, the unit Maison archives on uninstall and restores from -backup, so an uninstall/reinstall with "keep user data" comes back with the release -catalogue intact rather than re-scanning Usenet from scratch. No user media directory is -mounted into any service. +All state — the feeder config, its redb cache, the per-release `.nzb` cache, the TMDB +cache — is under `/DATA/AppData/metafeederusenet/`. The `.nzb` cache is re-derivable from +NNTmux. No user media directory is mounted.