MetaFeederUsenet: glue only (feeder 1.1.0 + enrichment), talks to the NNTmux app over its Newznab API; nntmux moved to worph/AppStore

2026-10-04 12:30:13 +00:00
parent f2f206e5c8
commit d0389498ed
2 changed files with 81 additions and 743 deletions
@@ -1,313 +1,43 @@
 name: metafeederusenet
 # =============================================================================
-# MetaFeeder · Usenet — a STANDALONE, self-hosted Usenet feeder for MetaMesh.
+# MetaFeeder · Usenet — publishes YOUR OWN Usenet indexer to MetaMesh.
 #
-# Unlike MetaFeeder · Prowlarr (which proxies an EXTERNAL Newznab/Torznab
+# ⚠ NNTmux is a SEPARATE app (worph/AppStore) — install it first. This feeder
-# indexer and mints an `nzb-release` (0x1005) cid embedding that indexer's
+# reaches it on the shared `pcs` network at `http://nntmux` over its Newznab
-# host), this app runs its OWN header scan (an `nntmux` — newznab-tmux fork —
+# API (`/api/v1/api` + `/getnzb`), with the API key of NNTmux's `metamesh`
-# sidecar) so it can mint a PORTABLE `nzb-posting` (0x1003) cid: a digest over
+# user — the same contract MetaFeeder · Prowlarr uses with Prowlarr. It never
-# the article Message-ID set, embedding no indexer host, redeemable by ANY
+# touches NNTmux's database or storage.
-# peer with a plain NNTP provider account. It ADDS a Usenet source; it does
+#
-# not replace the Prowlarr/Torznab one — a gateway query fans out to both.
+# Unlike MetaFeeder · Prowlarr (which proxies EXTERNAL indexers and mints an
 # `nzb-release` (0x1005) cid embedding that indexer's host), every release
 # here comes from a scan you run, so the feeder downloads its .nzb once and
 # mints a PORTABLE `nzb-posting` (0x1003) cid: a digest over the article
 # Message-ID set, redeemable by ANY peer with a plain NNTP provider account.
 # It also redeems external `nzb-release` locators with per-indexer keys set on
 # its config page.
 #
 # A headless service: no web UI of its own. Configure it from MetaGateway →
 # this feeder → *configure* (nntmux URL + API key, indexer keys).
 #
 # It needs, over `pcs`: **NNTmux** (`nntmux`), the shared **MetaCore**
 # (`metacore-app:9000`, records + TMDB posters) and **MetaGateway**, which must
 # list `http://metafeederusenet-feeder:8080` in its feeders config.
 #
 # Design record (meta-gateway repo): docs/others/self-hosted-usenet-indexer-study.md
 # Feeder repo: github.com/worph/meta-feeder-usenet
 #
 # It needs, over the shared `pcs` network:
 #   - the shared **MetaCore** app (`metacore-app:9000`) — where resolved
 #     records + TMDB posters are stored; the shared **MetaShare** app then
 #     announces them to the swarm.
 # The **MetaGateway** app must list this feeder's URL
 # (`http://metafeederusenet-feeder:8080`) in its feeders config so it
 # discovers + queries it (MetaGateway, MetaCore and this feeder share `pcs`).
 #
 # Everything is scoped under the `metafeederusenet-*` prefix + the per-stack
 # `metafeederusenet` network + /DATA/AppData/metafeederusenet/ data root, so it
 # installs alongside the other Meta* apps with zero collisions.
 #
 # amd64 ONLY — `pyrowman/nntmux` ships no arm64 build.
 #
 # ⚠ Four silent-failure traps in the `pyrowman/nntmux` image, empirically
 # verified (see the meta-gateway dev compose header for the full writeup):
 #   1. `PATH_TO_NZBS` (and every other storage path var) must be ABSENT from
 #      the env, never present-but-blank — that reads as an explicit empty
 #      string, not "unset", and nntmux aborts NZB writing with 0 releases and
 #      no visible error. This compose bind-mounts a COMPLETE, hand-filled
 #      `.env` (below) rather than relying on individual `environment:` keys,
 #      for exactly this reason.
 #   2. `DB_CONNECTION` must be `mariadb`, not `mysql` — the install migration
 #      loads its schema file by connection name; `mysql` finds no matching
 #      schema and fails partway through a from-scratch migration.
 #   3. Manticore is a HARD first-boot dependency (unless
 #      `ELASTICSEARCH_ENABLED=true`, which just swaps it for a heavier one) —
 #      `metafeederusenet-manticore` below is not optional.
 #   4. Seeded usenet groups default to `active=0, backfill=0` — scanning an
 #      inactive group silently does nothing. See the tips below for the SQL
 #      to activate one.
 # One FIRST-BOOT-ONLY loud bug: the entrypoint runs `config:cache` before
 # `key:generate`, so every session route 500s until you run
 # `docker exec metafeederusenet-nntmux php artisan config:clear` once. See tips.
 #
 # ⚠ Four more, found the hard way on a live box (2026-07-26) after this stack
 # came up with SEVEN healthy containers and returned ZERO results for every
 # query — the feeder answering `"Done"` and nothing else. None of them logs
 # anything; each one alone is enough to keep the catalog empty forever.
 #
 #   5. NEVER set a key in `nntmux.env` unless the value must DIFFER from the
 #      PHP default. `config/nntmux.php` does
 #      `env('NN_MULTIPROCESSING_MAX_CHILD_TIME', 1800)` and Laravel's `env()`
 #      returns STRINGS, so restating the default in the .env handed
 #      `Pool::timeout(float)` a `string("1800")` → TypeError → every Forking
 #      multiprocessing runner died on start. Those runners are exactly what a
 #      scan needs. This is trap #1's type-flavoured twin: a redundant
 #      restatement is not a no-op, it silently changes the value's TYPE. The
 #      key is therefore deliberately ABSENT from the seeded .env below (see the
 #      comment there); with it absent, `gettype()` reports `integer` and the
 #      runners execute. Same rule applies to any other key you are tempted to
 #      "document" by writing its default into the file.
 #      (Upstream record: meta-gateway `dev/docker-compose.feeder-usenet.yml`,
 #      the "CORRECTION (2026-07-26)" block.)
 #   6. The image runs `frankenphp` and NOTHING ELSE — no cron, no supervisord,
 #      no queue worker. nntmux's own `settings.running=1` therefore starts
 #      nothing: there is no process to read it. Even with valid NNTP creds and
 #      active groups the catalog stays empty permanently. That is why
 #      `metafeederusenet-scanner` below exists — it is the missing loop, run as
 #      a SECOND container off the same image so the web container's
 #      install/populate entrypoint stays untouched.
 #   7. The nntmux entrypoint RE-SEEDS `settings` and `usenet_groups` on EVERY
 #      container boot. Configuration applied before a restart silently reverts
 #      (observed: `running`→0, `delaytime`→2, every group→`active=0`). Apply
 #      group activation and scheduler settings AFTER the final restart, and
 #      re-check them after any `docker restart` / stack update.
 #   8. `settings.delaytime` (HOURS, default 2) stalls a backfilled catalog:
 #      `ProcessReleases` refuses to advance a collection until it has aged past
 #      it. That is correct for live tailing (a posting may still be uploading)
 #      and pointless for backfill, where every article is already days old —
 #      and the symptom, collections climbing while releases flatline, is
 #      indistinguishable from a broken pipeline because nothing logs an error.
 #      Because trap #7 re-seeds it back to 2 on every restart, the scanner
 #      loop below ASSERTS `delaytime=0` on each pass **while `backfill=1`** —
 #      the one setting it writes, making the tier self-healing across
 #      restarts. It is scoped to the backfill phase on purpose: with
 #      `backfill=0` (pure forward tailing) the operator's own value stands.
 # =============================================================================
 services:
  # ---------------------------------------------------------------------------
  # metafeederusenet-db — MariaDB backing nntmux's OWN release catalog. Not
  # shared with the gateway's meta-core Redis or anything else in the mesh.
  # ---------------------------------------------------------------------------
  metafeederusenet-db:
    image: mariadb:12.3
    cpu_shares: 70
    container_name: metafeederusenet-db
    hostname: metafeederusenet-db
    restart: unless-stopped
    # 2g, not the 512m this shipped with: 512m is fine for an empty schema and
    # far too small once the catalog is real. A header scan of a busy binary
    # group builds `collections`/`binaries`/`parts` tables in the tens of
    # millions of rows, and InnoDB starved of buffer pool turns collation into
    # a disk-thrash that never catches up. Measured on a live box.
    mem_limit: 2g
    memswap_limit: 2g
    environment:
      MARIADB_DATABASE: nntmux
      MARIADB_USER: nntmux
      MARIADB_PASSWORD: "$APP_DEFAULT_PASSWORD"
      MARIADB_RANDOM_ROOT_PASSWORD: "yes"
    volumes:
      - ${DATA_ROOT:-/DATA}/AppData/metafeederusenet/db:/var/lib/mysql
    healthcheck:
      test: ["CMD", "healthcheck.sh", "--connect", "--innodb_initialized"]
      interval: 5s
      timeout: 5s
      retries: 20
      start_period: 20s
    networks:
      - metafeederusenet
  # ---------------------------------------------------------------------------
  # metafeederusenet-redis — nntmux's OWN Redis. Deliberately separate from the
  # gateway core's Redis (different failure domain; losing this one is free).
  # ---------------------------------------------------------------------------
  metafeederusenet-redis:
    image: redis:7-alpine
    cpu_shares: 70
    container_name: metafeederusenet-redis
    hostname: metafeederusenet-redis
    restart: unless-stopped
    mem_limit: 128m
    memswap_limit: 128m
    networks:
      - metafeederusenet
  # ---------------------------------------------------------------------------
  # metafeederusenet-manticore — RT-index search backend. NOT optional (trap
  # #3 above) — a hard first-boot dependency of nntmux's own entrypoint, even
  # though this feeder reads `releases` via SQL directly and never touches it.
  # Digest-pinned (upstream reuses low-looking version numbers across an old
  # semver line and a newer CalVer-ish one; the digest is unambiguous).
  # ---------------------------------------------------------------------------
  metafeederusenet-manticore:
    image: manticoresearch/manticore@sha256:032388dc77321e3ac898a14b9a0c3fc22ab8d08cd2582abd6c5b642d46b82181
    cpu_shares: 50
    container_name: metafeederusenet-manticore
    hostname: metafeederusenet-manticore
    restart: unless-stopped
    mem_limit: 512m
    memswap_limit: 512m
    ulimits:
      nofile: { soft: 65535, hard: 65535 }
    networks:
      - metafeederusenet
  # ---------------------------------------------------------------------------
  # metafeederusenet-nntmux — the header scanner + web UI/API. Exposed DIRECTLY
  # via Caddy (no AppShield gate): nntmux ships its own admin login
  # (ADMIN_USER/ADMIN_PASS, seeded below), the same convention this store uses
  # for every app that brings its own auth (Jellyfin, Nextcloud, Immich, ...).
  # ---------------------------------------------------------------------------
  metafeederusenet-nntmux:
    # Digest-pinned, NOT :latest — a silent tag move could change the
    # `misc/update/*.php` command surface the "collate via tinker" tip below
    # depends on. Re-pin deliberately, re-verify the trap list above when you do.
    image: pyrowman/nntmux@sha256:7fc221f7a806fee3e4a2145159e1a147962edc095282bdc692806968f791fabe
    cpu_shares: 50
    container_name: metafeederusenet-nntmux
    hostname: metafeederusenet-nntmux
    restart: unless-stopped
    # The working set scales with scan breadth (group set x backfill depth), an
    # operator choice (see tips), so this is a ceiling rather than a size: 3g
    # is what a real corpus needed on a live box — 1536m was reached and
    # exceeded by a single busy group's header scan. Raise it if you scan wide.
    mem_limit: 3g
    memswap_limit: 3g
    environment:
      # Shell-script-consumed vars — the entrypoint's own `mkdir -p` step reads
      # these from the PROCESS environment, not the mounted .env. Keep in
      # lockstep with the matching keys in nntmux.env below.
      PATH_TO_NZBS: /app/storage/nzb/
      COVERS_PATH: /app/storage/covers
      TEMP_UNRAR_PATH: /app/storage/tmp/unrar/
      TEMP_UNZIP_PATH: /app/storage/tmp/unzip/
      ELASTICSEARCH_ENABLED: "false"
    volumes:
      # The complete, hand-filled .env (trap #1) — a full-file mount rather
      # than individual `environment:` keys. Writable: `nntmux:install`
      # persists a generated APP_KEY back into it on first boot.
      - ${DATA_ROOT:-/DATA}/AppData/metafeederusenet/nntmux-env/nntmux.env:/app/.env:rw
      # The .nzb store — shared with metafeederusenet-feeder below,
      # read-write here (nntmux writes), read-only there (feeder only reads).
      - ${DATA_ROOT:-/DATA}/AppData/metafeederusenet/nntmux-storage/nzb:/app/storage/nzb:rw
    healthcheck:
      test: ["CMD", "curl", "-sf", "-o", "/dev/null", "http://localhost/"]
      interval: 10s
      timeout: 5s
      retries: 15
      start_period: 30s
    depends_on:
      metafeederusenet-db:
        condition: service_healthy
      metafeederusenet-redis:
        condition: service_started
      metafeederusenet-manticore:
        condition: service_started
    labels:
      caddy_0: metafeederusenetnntmux-${APP_DOMAIN}
      caddy_0.import: gateway_tls
      caddy_0.reverse_proxy: "{{upstreams 80}}"
      caddy_1: metafeederusenetnntmux-${APP_PUBLIC_IP_DASH}.nip.io
      caddy_1.import: gateway_tls
      caddy_1.reverse_proxy: "{{upstreams 80}}"
      caddy_2: metafeederusenetnntmux-${APP_PUBLIC_IP_DASH}.sslip.io
      caddy_2.reverse_proxy: "{{upstreams 80}}"
    networks:
      - metafeederusenet
      - pcs
  # ---------------------------------------------------------------------------
  # metafeederusenet-scanner — the SCAN LOOP (trap #6). The nntmux image runs
  # frankenphp and nothing else, so without this container nothing ever calls
  # the scan/backfill/collate entry points and the catalog stays empty forever,
  # no matter how correct the credentials, the groups or `settings.running` are.
  #
  # It is a second container off the SAME digest-pinned image, sharing the same
  # `/app/.env` and the same nzb store, rather than a second process inside the
  # web container — the web container's entrypoint runs install/populate under
  # `set -e` and must keep it; here it is REPLACED by `entrypoint:` below so
  # that install/populate does NOT run twice against one database.
  #
  # Every knob (interval, binaries on/off, backfill on/off, backfill_qty,
  # backfill_days, running) is read from nntmux's own `settings` table at
  # /admin/tmux-edit — deliberately NOT from new env vars here, which would
  # make two sources of truth for one number.
  # ---------------------------------------------------------------------------
  metafeederusenet-scanner:
    # MUST stay byte-identical to metafeederusenet-nntmux's pin above: the two
    # containers share one `/app/.env` and one database schema, and the script
    # below calls `misc/update/*.php` + `Blacklight\` classes by name.
    image: pyrowman/nntmux@sha256:7fc221f7a806fee3e4a2145159e1a147962edc095282bdc692806968f791fabe
    cpu_shares: 20
    container_name: metafeederusenet-scanner
    hostname: metafeederusenet-scanner
    restart: unless-stopped
    # Bypasses the image's own install/populate entrypoint on purpose — see the
    # block comment above. Only the web container may run it.
    entrypoint: ["/bin/sh", "/scanner.sh"]
    # The image ships a HEALTHCHECK that curls its own web server. This
    # container deliberately runs no web server — its entrypoint IS the scan
    # loop — so the inherited probe can only ever fail. Observed live: a
    # scanner actively backfilling and writing NZBs sitting at
    # `Up 5 minutes (unhealthy)`.
    #
    # That is not cosmetic. A permanently-red container that is in fact working
    # trains the operator to ignore health status, which is precisely how this
    # stack's empty-catalog failure stayed invisible for so long — the feeder's
    # own health() reporting `ok` over a zero-row catalog was the same bug from
    # the other direction. Liveness for this container is its LOG
    # (`docker logs -f metafeederusenet-scanner`), not a probe; a probe that
    # cannot be honest is worse than none.
    healthcheck:
      disable: true
    # A backfill pass holds a group's article range in memory; 2g is the
    # headroom measured alongside nntmux's own 3g on a live box.
    mem_limit: 2g
    memswap_limit: 2g
    environment:
      # Same shell-script-consumed vars as metafeederusenet-nntmux: the wrapper
      # scripts read these from the PROCESS environment, not from the mounted
      # .env (trap #1). Keep the two blocks in lockstep.
      PATH_TO_NZBS: /app/storage/nzb/
      COVERS_PATH: /app/storage/covers
      TEMP_UNRAR_PATH: /app/storage/tmp/unrar/
      TEMP_UNZIP_PATH: /app/storage/tmp/unzip/
      ELASTICSEARCH_ENABLED: "false"
    volumes:
      # The same .env the web container filled in — one credential set, one
      # APP_KEY. Read-write only because Laravel's .env handling expects it.
      - ${DATA_ROOT:-/DATA}/AppData/metafeederusenet/nntmux-env/nntmux.env:/app/.env:rw
      # nntmux writes .nzb.gz here; the feeder reads them read-only.
      - ${DATA_ROOT:-/DATA}/AppData/metafeederusenet/nntmux-storage/nzb:/app/storage/nzb:rw
      # The loop itself, provisioned by pre-install-cmd below.
      - ${DATA_ROOT:-/DATA}/AppData/metafeederusenet/scanner/scanner.sh:/scanner.sh:ro
    depends_on:
      # Not just `started`: the loop's very first action is an artisan call, so
      # it must not race the web container's install/migrate step.
      metafeederusenet-nntmux:
        condition: service_healthy
    networks:
      - metafeederusenet
  # ---------------------------------------------------------------------------
  # metafeederusenet-feeder — the usenet feeder sidecar (meta-feeder-usenet).
-  # Reads nntmux's release catalog + shared .nzb store; self-publishes
+  # Searches NNTmux over its Newznab API; self-publishes resolved records +
-  # resolved records + seeds the `.nzb` manifest to the shared MetaCore. The
+  # seeds the `.nzb` manifest to the shared MetaCore. The
  # gateway reaches it directly on `pcs`; the human configures it from
  # the MetaGateway dashboard (configure panel).
  # ---------------------------------------------------------------------------
  metafeederusenet-feeder:
-    image: ghcr.io/worph/meta-feeder-usenet:1.0.6
+    image: ghcr.io/worph/meta-feeder-usenet:1.1.0
    cpu_shares: 30
    container_name: metafeederusenet-feeder
    # Hard RAM cap, swap disabled (memswap_limit == mem_limit) → OOM instead
@@ -321,7 +51,7 @@ services:
      - 8080
    environment:
      # INFRA-ONLY env (the "config: no env if in webui" rule). Operator
-      # config beyond the two DB/path seeds below — TMDB token / language,
+      # config beyond the NNTmux URL seed below — TMDB token / language,
      # meta-core URL — is read from the persisted config.json (this
      # feeder's config page, opened from the MetaGateway dashboard (configure panel)). These env vars are
      # wiring only: the enrichment sidecars, the enrich callback, the peer id.
@@ -335,14 +65,13 @@ services:
      META_FEEDER_CALLBACK_URL: "http://metafeederusenet-feeder:8080/enrich/callback"
      # First-boot SEED only (the persisted config.json wins on every later
      # restart — change these on the /config page after first boot, not here).
-      NNTMUX_DB_URL: "mysql://nntmux:${APP_DEFAULT_PASSWORD}@metafeederusenet-db:3306/nntmux"
+      # The API key has no env seed: paste NNTmux's `metamesh` key on the page.
-      NNTMUX_NZB_PATH: "/nntmux-nzb"
+      NNTMUX_URL: "http://nntmux"
      TZ: "$TZ"
    volumes:
-      # Persist feeder config (config.json saved via the config UI) + redb cache.
+      # Persist feeder config (config.json saved via the config UI), the redb
      # cache, and the per-release .nzb cache (gateway/usenet/nntmux-nzb/).
      - ${DATA_ROOT:-/DATA}/AppData/metafeederusenet/feeder-state:/data/meta-feeder:rw
      # Same volume nntmux writes .nzb.gz into — read-only here.
      - ${DATA_ROOT:-/DATA}/AppData/metafeederusenet/nntmux-storage/nzb:/nntmux-nzb:ro
    healthcheck:
      test: ["CMD", "curl", "-sf", "http://localhost:8080/health"]
      interval: 5s
@@ -350,8 +79,6 @@ services:
      retries: 10
      start_period: 5s
    depends_on:
      metafeederusenet-nntmux:
        condition: service_healthy
      metafeederusenet-filename-parser:
        condition: service_started
      metafeederusenet-tmdb:
@@ -420,68 +147,40 @@ x-casaos:
  description:
    en_us: |
-      **A standalone, self-hosted Usenet feeder for the MetaMesh network.**
+      **Publish your own Usenet indexer to the MetaMesh network.**
-      MetaFeeder · Usenet runs its own Usenet header scan (a bundled
+      MetaFeeder · Usenet searches your **NNTmux** app (worph/AppStore) over
-      **nntmux** indexer) instead of proxying a third-party Newznab site.
+      its Newznab API and turns each release into a **portable** MetaMesh
-      Because it finds article Message-IDs itself at scan time, it mints a
+      posting — no indexer host baked in, redeemable by any peer with a plain
-      **portable** posting reference — no indexer host baked in, redeemable by
+      NNTP provider account — with a clean title, poster and plot from the
-      any peer with a plain NNTP provider account. It adds a Usenet source
+      bundled filename-parser + TMDB enrichment. It adds a Usenet source
      alongside **MetaFeeder · Prowlarr**; it doesn't replace it.
-      **Self-contained:** ships its own nntmux scanner (MariaDB + Redis +
+      It also **redeems** releases found through Prowlarr's Newznab indexers,
-      Manticore) + filename-parser + TMDB enrichment, so each release gets a
+      with the indexer keys you set on its config page.
      clean title, poster and plot.
-      **You still need a Usenet provider account** (NNTP login) — this app
+      **Install NNTmux first.** No web UI of its own: configure it from the
-      scans headers, it does not supply an account, and nothing at all is
+      **MetaGateway** dashboard (nntmux URL + the `metamesh` API key).
      indexed until you supply one. Configure it after install (see the
      install tips). Keep the connection count low: this scanner and your
      playback share the same provider connection slots.
      **The catalog is not instant.** Once configured, a bundled scanner
      container runs the header-scan / backfill / collation loop on the
      interval you set. A first pass over a few busy binary groups pulls
      tens of GB and runs for hours; a deep backfill is hundreds of GB. The
      install tips show which knobs bound that.
      **Also requires:** **MetaGateway** — its meta-core stores the resolved
-      records and announces them to the swarm. The gateway must list this
+      records and announces them to the swarm.
      feeder in its config.
    fr_fr: |
-      **Un feeder Usenet autonome et auto-hébergé pour le réseau MetaMesh.**
+      **Publiez votre propre indexeur Usenet sur le réseau MetaMesh.**
-      MetaFeeder · Usenet effectue lui-même le scan des en-têtes Usenet (un
+      MetaFeeder · Usenet interroge votre app **NNTmux** (worph/AppStore) via
-      indexeur **nntmux** embarqué) plutôt que de passer par un site Newznab
+      son API Newznab et transforme chaque release en posting MetaMesh
-      tiers. Comme il retrouve les Message-ID des articles dès le scan, il
+      **portable**, exploitable par n'importe quel pair disposant d'un simple
-      génère une référence de posting **portable** — sans hôte d'indexeur
+      compte NNTP. Il ajoute une source Usenet en plus de **MetaFeeder ·
-      intégré, exploitable par n'importe quel pair disposant d'un simple
+      Prowlarr**.
      compte NNTP. Il ajoute une source Usenet en plus de **MetaFeeder -
      Indexer**, sans le remplacer.
-      **Autonome :** embarque son scanner nntmux (MariaDB + Redis + Manticore)
+      **Installez NNTmux d'abord.** Pas d'interface propre : configuration
-      + ses plugins filename-parser + TMDB.
+      depuis le tableau de bord **MetaGateway** (URL nntmux + clé API
-
+      `metamesh`). **Nécessite aussi** l'app **MetaGateway**.
      **Un compte fournisseur Usenet (NNTP) reste nécessaire** — cette app
      scanne les en-têtes, elle ne fournit pas d'accès, et rien n'est indexé
      tant que vous n'en fournissez pas un. À configurer après installation
      (voir les conseils d'installation). Gardez un nombre de connexions
      faible : le scan et la lecture partagent les mêmes créneaux du
      fournisseur.
      **Le catalogue ne se remplit pas instantanément.** Une fois configuré,
      un conteneur de scan embarqué exécute la boucle scan / backfill /
      collation à l'intervalle que vous choisissez. Un premier passage sur
      quelques gros groupes binaires télécharge des dizaines de Go et dure
      des heures ; un backfill profond, des centaines de Go.
      **Nécessite aussi :** l'app **MetaGateway** (son meta-core stocke et
      annonce les enregistrements).
  tagline:
-    en_us: Self-hosted Usenet header scanner (bundled nntmux) for the MetaMesh network
+    en_us: Publishes your NNTmux Usenet indexer to MetaMesh as portable postings
-    fr_fr: Scanner Usenet auto-hébergé (nntmux inclus) pour le réseau MetaMesh
+    fr_fr: Publie votre indexeur Usenet NNTmux sur MetaMesh
  title:
    en_us: MetaFeeder · Usenet
@@ -499,342 +198,26 @@ x-casaos:
  tips:
    before_install:
      en_us: |
-        **Install order:** install **MetaGateway** first (this feeder
+        **Install order:** **NNTmux** (worph/AppStore) and **MetaGateway** first.
        publishes to its shared meta-core).
-        **After installing this app, a Usenet provider account is required**
+        **After installing:**
-        (this scans headers itself — it needs a plain NNTP login):
+        1. Read NNTmux's `metamesh` API key (the NNTmux app provisions this user,
-        1. Edit `/DATA/AppData/metafeederusenet/nntmux-env/nntmux.env` (via
+           with raised daily API/download caps):
           SSH or a file-manager app) and fill in `NNTP_SERVER`, `NNTP_PORT`,
           `NNTP_USERNAME`, `NNTP_PASSWORD` (use the SAME provider account
           your MetaShare/meta-share peer is already credentialed against —
           one account, two consumers). Keep `NNTP_CONNECTIONS` low (1-2):
           the scanner and playback share the same provider connection slots,
           so a greedy value here stalls playback mid-stream elsewhere.
        2. Restart the nntmux container: `docker restart metafeederusenet-nntmux`.
        3. **First boot only:** nntmux's install sequence caches its config
           before generating an encryption key, so every page 500s until you
           run `docker exec metafeederusenet-nntmux php artisan config:clear`
           once. (`curl -sI https://metafeederusenetnntmux-<your-domain>/` — a
           `500` means you need this; a `302` redirect to `/login` means
           you're fine.)
        4. Log in to the nntmux dashboard at the **MetaFeeder Usenet nntmux**
           link (`ADMIN_USER=admin`, password = this app's auto-generated
           default password — check CasaOS's app credentials panel) and
           change it.
        **⚠ Do steps 5-6 AFTER your LAST restart, and re-check them after
        every later restart or stack update.** The nntmux entrypoint RE-SEEDS
        the `settings` and `usenet_groups` tables on EVERY container boot, so
        anything you configure before a restart silently reverts (observed:
        `running` back to 0, `delaytime` back to 2, every group back to
        `active=0`) — with no error anywhere.
        5. **Activate at least one usenet group** — seeded groups start
           inactive (`active=0`), and scanning an inactive group silently
           touches nothing at all:
           ```
-           docker exec metafeederusenet-db mariadb -unntmux -p"<password>" nntmux \
+           docker exec nntmux php artisan tinker --execute="echo App\Models\User::where('username','metamesh')->value('api_token');"
             -e "UPDATE usenet_groups SET active=1, backfill=1 WHERE name IN ('alt.binaries.teevee');"
           ```
-        6. **Start the scheduler** in nntmux's admin UI (**/admin/tmux-edit**)
+        2. In **MetaGateway**, make sure this feeder is listed in its feeders
-           or directly in the `settings` table: set `running=1`, pick your
+           config (`http://metafeederusenet-feeder:8080`), then open its config
-           `monitor_delay` (seconds between passes), and turn on `binaries`
+           panel (*configure*): paste the key into **nntmux API key** (the URL
-           and/or `backfill` with a `backfill_qty` / `backfill_days` you're
+           defaults to `http://nntmux`) and Save — the feeder restarts itself.
-           happy to pay for. The bundled `metafeederusenet-scanner` container
+        3. Optionally add **indexer keys** there too, to redeem releases found
-           polls these settings and runs the scan/backfill/collate loop for
+           through Prowlarr's Newznab indexers, and a TMDB token for posters.
           you — the nntmux image itself ships NO cron and NO queue worker, so
           this container is what actually makes a scan happen. There are
           deliberately no env vars for these knobs: the `settings` table is
           the single source of truth.
-           **`delaytime` (HOURS, default 2) will stall a backfilled catalog.**
+        **Searches must be typed** (`fileType:video`, `contentKind:episode`…) to
-           nntmux won't turn a collection into a release until it has aged
+        reach this feeder — a plain search-box query never fans out to it.
           past `delaytime`, which is right for tailing live posts and
           pointless when backfilling history — and the symptom (collections
           climbing, releases flat, zero query results) looks exactly like a
           broken pipeline. **While `backfill=1`, the scanner forces
           `delaytime=0` for you on every pass** and logs
           `asserting delaytime=0 for the backfill phase (was N)` when it does
           — it has to re-assert it because a container restart re-seeds the
           value back to 2. This is the only setting the loop ever writes.
-           So if you want a different steady-state `delaytime`, **set
+        Each release's .nzb is downloaded from NNTmux once and cached here
-           `backfill=0` first**, then set your value — otherwise the loop will
+        (`feeder-state/gateway/usenet/nntmux-nzb/`).
           keep pulling it back to 0. With `backfill=0` the scanner never
           touches it and `1`-`2` is the sensible live-tailing value.
           **Budget the first scan.** A first header pass over a few busy
           binary groups pulls tens of GB and runs for hours; a
           full-retention backfill is hundreds of GB. `backfill_days` is the
           knob that bounds it — start small, widen once you see releases.
        7. Open its config panel (the **MetaGateway** dashboard → this feeder → *configure*) to optionally set a
           TMDB v4 token for posters, then Save.
        8. In **MetaGateway**, make sure this feeder is listed in its feeders
           config (`http://metafeederusenet-feeder:8080`) so it gets
           discovered.
        **Note:** which groups to scan, backfill depth and scan interval are
        your call as the operator — pick them for your own catalog.
        **Watch the loop:** `docker logs -f metafeederusenet-scanner` prints
        one line per pass (`header scan` / `backfill direct` / `collate
        <group>` / `sleeping`). `settings.running=0` shows up there as
        `idle`.
        **Authentication:** the feeder's config page opens from the **MetaGateway**
        dashboard (behind its login); nntmux uses its own login (seeded above).
  pre-install-cmd: |
    # Seed nntmux's .env (trap #1 — must be a complete file, see header).
    # NNTP_* are deliberately left blank; fill them in per the install tips.
    if [ ! -f /DATA/AppData/metafeederusenet/nntmux-env/nntmux.env ]; then
      cat > /DATA/AppData/metafeederusenet/nntmux-env/nntmux.env <<NNTMUX_ENV
    DB_CONNECTION=mariadb
    DB_HOST=metafeederusenet-db
    DB_PORT=3306
    DB_SOCKET=
    DB_USERNAME=nntmux
    DB_PASSWORD=$APP_DEFAULT_PASSWORD
    DB_DATABASE=nntmux
    MANTICORESEARCH_HOST=metafeederusenet-manticore
    MANTICORESEARCH_PORT=9308
    ELASTICSEARCH_HOST=localhost
    ELASTICSEARCH_PORT=9200
    ELASTICSEARCH_SCHEME=http
    ELASTICSEARCH_USER=
    ELASTICSEARCH_PASS=
    ELASTICSEARCH_LOGGING=false
    ELASTICSEARCH_ENABLED=false
    NNTP_COMPRESSED_HEADERS=false
    USE_ALTERNATE_NNTP_SERVER=false
    NNTP_USERNAME=
    NNTP_PASSWORD=
    NNTP_SERVER=
    NNTP_PORT=563
    NNTP_CONNECTIONS=1
    NNTP_SSLENABLED=true
    NNTP_SOCKET_TIMEOUT=120
    NNTP_USERNAME_A=
    NNTP_PASSWORD_A=
    NNTP_SERVER_A=
    NNTP_PORT_A=
    NNTP_CONNECTIONS_A=
    NNTP_SSLENABLED_A=false
    NNTP_SOCKET_TIMEOUT_A=120
    # NN_MULTIPROCESSING_MAX_CHILD_TIME is deliberately ABSENT (trap #5).
    # Writing it here, even as its own default of 1800, hands
    # Pool::timeout(float) a string via Laravel's env(), which returns STRINGS,
    # and every Forking multiprocessing runner dies with a TypeError.
    # RULE for this whole file: never set a key unless the value must DIFFER
    # from the PHP default in config/nntmux.php. Restating a default is not
    # documentation, it is a type change.
    ADMIN_USER=admin
    ADMIN_PASS=$APP_DEFAULT_PASSWORD
    ADMIN_EMAIL=admin@$APP_DOMAIN
    APP_NAME=NNTmux
    APP_ENV=production
    APP_DEBUG=false
    APP_TIMEZONE=UTC
    APP_URL=https://metafeederusenetnntmux-$APP_DOMAIN
    APP_LOCALE=en
    APP_FALLBACK_LOCALE=en
    APP_FAKER_LOCALE=en_US
    APP_MAINTENANCE_DRIVER=file
    APP_MAINTENANCE_STORE=database
    BCRYPT_ROUNDS=12
    APP_KEY=
    LOG_CHANNEL=stack
    LOG_STACK=single
    PASSWORD_HASH=argon2id
    BROADCAST_CONNECTION=log
    CACHE_STORE=file
    QUEUE_CONNECTION=sync
    REDIS_HOST=metafeederusenet-redis
    REDIS_PASSWORD=null
    REDIS_PORT=6379
    REDIS_CLIENT=phpredis
    SESSION_DRIVER=file
    SESSION_DOMAIN=null
    SESSION_SECURE_COOKIE=false
    SESSION_ENCRYPT=false
    SESSION_COOKIE=nntmux
    SESSION_PATH=/
    SCOUT_DRIVER=database
    SCOUT_QUEUE=true
    ITEMS_PER_PAGE=50
    ITEMS_PER_COVER_PAGE=25
    MAX_PAGER_RESULTS=125000
    ECHOCLI=true
    RENAME_PAR2=false
    ADD_PAR2=false
    RENAME_MUSIC_MEDIAINFO=true
    CACHE_EXPIRY_SHORT=5
    CACHE_EXPIRY_MEDIUM=10
    CACHE_EXPIRY_LONG=15
    SSL_CAFILE=
    SSL_CAPATH=
    SSL_VERIFY_PEER=false
    SSL_VERIFY_HOST=false
    SSL_ALLOW_SELF_SIGNED=true
    SMARTY_DEBUG=false
    SMARTY_CACHING=false
    SMARTY_CACHE_LIFE=120
    SMARTY_COMPILE_CHECK=false
    SMARTY_FORCE_COMPILE=false
    SMARTY_CACHE_DRIVER=file
    TMDB_APIKEY=9a4e16adddcd1e86da19bcaf5ff3c2a3
    TMDB_CACHE=false
    TMDB_LOG=false
    TVDB_APIKEY=ef6fb572-bcee-4d99-9b52-90549ad7553a
    TVDB_PIN=
    TEMP_UNRAR_PATH=/app/storage/tmp/unrar/
    TEMP_UNZIP_PATH=/app/storage/tmp/unzip/
    VIEW_COMPILED_PATH=/app/storage/framework/views
    PATH_TO_NZBS=/app/storage/nzb/
    COVERS_PATH=/app/storage/covers
    TELESCOPE_ENABLED=false
    PURGE_INACTIVE_USERS=false
    OTP_ENABLED=false
    CHECK_PASSWORDED_RARS=false
    DELETE_PASSWORDED_RELEASES=false
    DELETE_POSSIBLE_PASSWORDED_RELEASES=false
    EXTRACT_USING_RARINFO=false
    PRIVATE_PROFILES=true
    STORE_USER_IP=false
    FORUM_FRONTEND_ENABLED=false
    NNTMUX_ENV
    fi
    # Seed the scan loop run by metafeederusenet-scanner (trap #6). Unlike
    # nntmux.env above this is NOT guarded by `[ ! -f ]` and is NOT operator
    # config: it is shipped code, so a store update must be able to replace a
    # stale copy. Every tunable it reads lives in nntmux's `settings` table.
    # Quoted heredoc delimiter ('SCANNER_SH') — the script contains $, \ and
    # nested quotes that must reach the file verbatim, unexpanded.
    cat > /DATA/AppData/metafeederusenet/scanner/scanner.sh <<'SCANNER_SH'
    #!/bin/sh
    # nntmux scan launcher.
    #
    # The pyrowman/nntmux image runs frankenphp and NOTHING else: no cron, no
    # supervisord, no queue worker. So `settings.running=1` changes nothing on
    # its own, because nothing ever starts the loop. This is that loop.
    #
    # It runs as a SECOND CONTAINER off the same image + the same /app/.env and
    # nzb volume, rather than as a second process inside the web container, so
    # the web container's entrypoint (install/populate under `set -e`) is left
    # untouched.
    #
    # INVOCATION IS NOT FREE CHOICE. Measured PER RUNNER on a live box, NOT a
    # blanket "Forking works":
    #   * multiprocessing/binaries.php   — Forking OK.
    #   * multiprocessing/backfill.php   — NO-OP. Reports "...3 job(s) to do
    #     using a max of 0 child process(es)" and returns in 0s having done
    #     nothing. Backfill must be driven DIRECTLY, per group (see the
    #     backfill block below).
    #   * collation MUST go through tinker/ProcessReleases DIRECTLY.
    #     multiprocessing/releases.php aborts every batch on a benign PHP
    #     warning (ReleaseCleaning.php:165) that turns fatal inside the Spatie
    #     child -> 0 releases while collections pile into the tens of
    #     thousands.
    #
    # Every knob comes from nntmux's own `settings` table (edited at
    # /admin/tmux-edit), never from env — otherwise there are two sources of
    # truth for one number.
    set -u
    cd /app || exit 1
    setting()     { php artisan tinker --execute="echo DB::table('settings')->where('name','$1')->value('value');" 2>/dev/null | tr -dc '0-9'; }
    set_setting() { php artisan tinker --execute="DB::table('settings')->where('name','$1')->update(['value'=>'$2']);" >/dev/null 2>&1; }
    groups()  { php artisan tinker --execute="foreach (DB::table('usenet_groups')->where('active',1)->pluck('name') as \$g) echo \$g, PHP_EOL;" 2>/dev/null | grep -E '^[a-z0-9.]+$'; }
    log() { echo "[scanner $(date -u +%H:%M:%S)] $*"; }
    log "launcher started"
    while true; do
        if [ "$(setting running)" != "1" ]; then
            log "settings.running=0 — idle"
            sleep 30
            continue
        fi
        if [ "$(setting binaries)" = "1" ]; then
            qty="$(setting backfill_qty)"; [ -n "$qty" ] || qty=20000
            log "header scan (qty=$qty)"
            php misc/update/multiprocessing/binaries.php "$qty" 2>&1 | tail -3
        fi
        # Backfill must run DIRECT, per group. multiprocessing/backfill.php
        # reports "max of 0 child process(es)" and silently does nothing — the
        # Forking wrapper leaves maxProcesses=0 for this runner specifically.
        # Depth is capped by settings.backfill_days, so this reaches back that
        # far and then stops advancing; it is NOT a full-retention backfill.
        if [ "$(setting backfill)" = "1" ]; then
            # The ONLY setting this loop writes. `delaytime` (hours, default 2)
            # blocks ProcessReleases from advancing a collection until it ages
            # past it. Correct for the leading edge — a posting may still be
            # uploading — but pointless for backfill, where every article is
            # already days old. Left at 2 during backfill it stalls the catalog
            # for two hours with a failure mode indistinguishable from a broken
            # pipeline (collections climb, releases flatline, nothing logs an
            # error).
            #
            # Asserted every pass, not once at boot, because the nntmux
            # entrypoint RE-SEEDS `settings` on every container start (trap #7)
            # — a restart silently puts this back to 2 and the tier quietly
            # stops producing. Scoped to `backfill=1` so a pure forward-tailing
            # deployment (`backfill=0`) keeps the operator's own value: this is
            # a per-phase mode, not a preference we get to override.
            if [ "$(setting delaytime)" != "0" ]; then
                log "asserting delaytime=0 for the backfill phase (was $(setting delaytime))"
                set_setting delaytime 0
            fi
            qty="$(setting backfill_qty)"; [ -n "$qty" ] || qty=20000
            log "backfill direct (backfill_days=$(setting backfill_days), qty=$qty)"
            groups | while read -r g; do
                [ -n "$g" ] || continue
                php misc/update/backfill.php "$g" "$qty" 2>&1 | grep -iE "articles|backfill|Group|no new" | tail -2
            done
        fi
        # Collation, per active group, DIRECT (never multiprocessing/releases.php).
        # Never wrap this in a short `timeout`: killing it between the release
        # INSERT and createNZBs() leaves a release row with no .nzb.gz, which
        # the feeder then correctly skips forever.
        groups | while read -r g; do
            [ -n "$g" ] || continue
            log "collate $g"
            php artisan tinker --execute="(new Blacklight\\processing\\ProcessReleases)->processReleases(1, 1, \"$g\", new Blacklight\\NNTP);" 2>&1 | grep -iE "releases (ready|created)|Added [0-9]+ releases|NZB" | tail -3
        done
        delay="$(setting monitor_delay)"; [ -n "$delay" ] || delay=30
        log "pass complete; sleeping ${delay}s"
        sleep "$delay"
    done
    SCANNER_SH
    chmod 755 /DATA/AppData/metafeederusenet/scanner/scanner.sh
 # Maison (Yundera dashboard) configuration. Read in preference to x-casaos.
 # schema_version 2 is required for `folders`: an older Maison refuses the app
@@ -845,26 +228,10 @@ x-compose-app:
  # runs and before the containers start — on the first boot and every one after.
  # Compose would otherwise create a missing bind source as a root-owned dir.
  folders:
    - path: /DATA/AppData/metafeederusenet/db
      user: $PUID
      group: $PGID
    - path: /DATA/AppData/metafeederusenet/nntmux-env
      user: $PUID
      group: $PGID
    - path: /DATA/AppData/metafeederusenet/nntmux-storage/nzb
      user: $PUID
      group: $PGID
    - path: /DATA/AppData/metafeederusenet/feeder-state
      user: $PUID
      group: $PGID
    - path: /DATA/AppData/metafeederusenet/tmdb-cache
      user: $PUID
      group: $PGID
-    - path: /DATA/AppData/metafeederusenet/scanner
+  # No web UI: Maison files it under Services. Configured from MetaGateway.
      user: $PUID
      group: $PGID
  # Same string as nntmux's caddy_0 label, so the tile link and the route agree.
  # The tile opens nntmux's own admin UI (its own login, its own Caddy route).
  # The feeder itself is configured from the MetaGateway dashboard.
  webui-host: metafeederusenetnntmux-${domain}
  webui-path: "/"
@@ -3,62 +3,33 @@
 ## What deviation / exception is being requested
 1. `metafeederusenet-feeder` runs `user: 0:0`.
-2. The **NNTmux web UI is published on its own hostname**
+
-   (`metafeederusenetnntmux-<user>.<domain>`) with no AppShield sidecar in front of it — it
+(The NNTmux indexer this feeder used to bundle — and its own exceptions: a web UI on its own
-   is protected by NNTmux's own login only.
+login, seeded TMDB/TVDB keys — is now the separate **NNTmux** app in worph/AppStore, with its
-3. The seeded `nntmux.env` contains TMDB and TVDB API keys as literals.
+own `rationale.md`.)
 ## Why it is necessary
 1. The feeder publishes resolved records to the shared MetaCore and writes plugin artwork
   through its WebDAV endpoint; those targets are created and owned by MetaCore's
   root-running container.
 2. NNTmux is a Laravel application with its own user table, session handling and admin
   role. Its admin account is created at install from `$APP_DEFAULT_PASSWORD`, so
   authentication is **enabled by default** — the "app's own built-in auth" alternative the
   security checklist accepts. Fronting it with AppShield as well would double-prompt on
   every page and breaks its own OTP and password-reset flows.
 3. Those two keys are NNTmux's upstream project defaults, shipped in its own
   `.env.example`. They are shared read-only application keys for public metadata catalogs,
   not user credentials, and they are what makes the app work immediately after
   installation with no manual configuration. An operator can replace them from the NNTmux
   admin UI.
 ## Security mitigations in place
- The NNTmux admin password is `$APP_DEFAULT_PASSWORD` — the platform-generated secret —
+- No web UI and no Caddy route: the feeder's configuration page is reached only through the
-  and the database password comes from the same variable. Neither is a hardcoded literal,
+  MetaGateway dashboard's proxy, behind the gateway's login.
-  and both are surfaced to the user in `tips.before_install`.
+- It talks to NNTmux only over NNTmux's Newznab API, with the key of a dedicated
- The feeder's own configuration UI *is* behind AppShield + Authelia SSO
+  `metamesh` user — it holds no database credential and mounts no NNTmux storage.
-  (`ghcr.io/yundera/appshield:2.0.9`).
+- The API key and the indexer keys are secrets on the feeder's config page (write-only,
- `metafeederusenet-db`, `-redis` and `-manticore` are on the app-private
+  never echoed back); none has an env seed or a default.
-  `metafeederusenet` bridge network, carry no Caddy labels and publish no host port, so no
+- `cpu_shares` and memory caps on all three services; every image pinned.
  sibling app on the shared `pcs` network can resolve or dial them.
 - NNTP provider credentials are deliberately left **blank** in the seed; the user fills
  them in per `tips.before_install`. Nothing ships with a working news account.
 - The `nntmux.env` seed is guarded by `[ ! -f … ]`, so a reinstall or version upgrade never
  overwrites the operator's provider settings. `scanner.sh` is intentionally rewritten on
  every run — it is shipped code, not operator config, so a store update must be able to
  replace a stale copy; the write is idempotent.
 - No user directories are mounted. All binds are under `/DATA/AppData/metafeederusenet/`,
  declared in `x-compose-app.folders`.
 - `cpu_shares` on all nine services; every image pinned (the NNTmux and Manticore images by
  digest).
 ## Alternatives considered and rejected
 - **AppShield in front of NNTmux.** Two independent login gates on one UI; its OTP,
  password-reset and API endpoints break behind the OIDC redirect.
 - **Blank the TMDB/TVDB keys.** The app then installs into a state where covers and
  metadata silently fail until the user finds two upstream registration flows — a direct
  violation of "works immediately after installation, no manual configuration required for
  basic functionality".
 - **Run the feeder as `$PUID:$PGID`.** Writes into MetaCore's root-owned tree fail.
 ## Data protection
-All state — MariaDB, the NZB store, the feeder cursor, the TMDB cache — is under
+All state — the feeder config, its redb cache, the per-release `.nzb` cache, the TMDB
-`/DATA/AppData/metafeederusenet/`, the unit Maison archives on uninstall and restores from
+cache — is under `/DATA/AppData/metafeederusenet/`. The `.nzb` cache is re-derivable from
-backup, so an uninstall/reinstall with "keep user data" comes back with the release
+NNTmux. No user media directory is mounted.
 catalogue intact rather than re-scanning Usenet from scratch. No user media directory is
 mounted into any service.