name: metafeederusenet # ============================================================================= # MetaFeeder · Usenet — a STANDALONE, self-hosted Usenet feeder for MetaMesh. # # Unlike MetaFeeder · Prowlarr (which proxies an EXTERNAL Newznab/Torznab # indexer and mints an `nzb-release` (0x1005) cid embedding that indexer's # host), this app runs its OWN header scan (an `nntmux` — newznab-tmux fork — # sidecar) so it can mint a PORTABLE `nzb-posting` (0x1003) cid: a digest over # the article Message-ID set, embedding no indexer host, redeemable by ANY # peer with a plain NNTP provider account. It ADDS a Usenet source; it does # not replace the Prowlarr/Torznab one — a gateway query fans out to both. # # Design record (meta-gateway repo): docs/others/self-hosted-usenet-indexer-study.md # Feeder repo: github.com/worph/meta-feeder-usenet # # It needs, over the shared `pcs` network: # - the shared **MetaCore** app (`metacore-app:9000`) — where resolved # records + TMDB posters are stored; the shared **MetaShare** app then # announces them to the swarm. # The **MetaGateway** app must list this feeder's URL # (`http://metafeederusenet-feeder:8080`) in its feeders config so it # discovers + queries it (MetaGateway, MetaCore and this feeder share `pcs`). # # Everything is scoped under the `metafeederusenet-*` prefix + the per-stack # `metafeederusenet` network + /DATA/AppData/metafeederusenet/ data root, so it # installs alongside the other Meta* apps with zero collisions. # # amd64 ONLY — `pyrowman/nntmux` ships no arm64 build. # # ⚠ Four silent-failure traps in the `pyrowman/nntmux` image, empirically # verified (see the meta-gateway dev compose header for the full writeup): # 1. `PATH_TO_NZBS` (and every other storage path var) must be ABSENT from # the env, never present-but-blank — that reads as an explicit empty # string, not "unset", and nntmux aborts NZB writing with 0 releases and # no visible error. This compose bind-mounts a COMPLETE, hand-filled # `.env` (below) rather than relying on individual `environment:` keys, # for exactly this reason. # 2. `DB_CONNECTION` must be `mariadb`, not `mysql` — the install migration # loads its schema file by connection name; `mysql` finds no matching # schema and fails partway through a from-scratch migration. # 3. Manticore is a HARD first-boot dependency (unless # `ELASTICSEARCH_ENABLED=true`, which just swaps it for a heavier one) — # `metafeederusenet-manticore` below is not optional. # 4. Seeded usenet groups default to `active=0, backfill=0` — scanning an # inactive group silently does nothing. See the tips below for the SQL # to activate one. # One FIRST-BOOT-ONLY loud bug: the entrypoint runs `config:cache` before # `key:generate`, so every session route 500s until you run # `docker exec metafeederusenet-nntmux php artisan config:clear` once. See tips. # # ⚠ Four more, found the hard way on a live box (2026-07-26) after this stack # came up with SEVEN healthy containers and returned ZERO results for every # query — the feeder answering `"Done"` and nothing else. None of them logs # anything; each one alone is enough to keep the catalog empty forever. # # 5. NEVER set a key in `nntmux.env` unless the value must DIFFER from the # PHP default. `config/nntmux.php` does # `env('NN_MULTIPROCESSING_MAX_CHILD_TIME', 1800)` and Laravel's `env()` # returns STRINGS, so restating the default in the .env handed # `Pool::timeout(float)` a `string("1800")` → TypeError → every Forking # multiprocessing runner died on start. Those runners are exactly what a # scan needs. This is trap #1's type-flavoured twin: a redundant # restatement is not a no-op, it silently changes the value's TYPE. The # key is therefore deliberately ABSENT from the seeded .env below (see the # comment there); with it absent, `gettype()` reports `integer` and the # runners execute. Same rule applies to any other key you are tempted to # "document" by writing its default into the file. # (Upstream record: meta-gateway `dev/docker-compose.feeder-usenet.yml`, # the "CORRECTION (2026-07-26)" block.) # 6. The image runs `frankenphp` and NOTHING ELSE — no cron, no supervisord, # no queue worker. nntmux's own `settings.running=1` therefore starts # nothing: there is no process to read it. Even with valid NNTP creds and # active groups the catalog stays empty permanently. That is why # `metafeederusenet-scanner` below exists — it is the missing loop, run as # a SECOND container off the same image so the web container's # install/populate entrypoint stays untouched. # 7. The nntmux entrypoint RE-SEEDS `settings` and `usenet_groups` on EVERY # container boot. Configuration applied before a restart silently reverts # (observed: `running`→0, `delaytime`→2, every group→`active=0`). Apply # group activation and scheduler settings AFTER the final restart, and # re-check them after any `docker restart` / stack update. # 8. `settings.delaytime` (HOURS, default 2) stalls a backfilled catalog: # `ProcessReleases` refuses to advance a collection until it has aged past # it. That is correct for live tailing (a posting may still be uploading) # and pointless for backfill, where every article is already days old — # and the symptom, collections climbing while releases flatline, is # indistinguishable from a broken pipeline because nothing logs an error. # Because trap #7 re-seeds it back to 2 on every restart, the scanner # loop below ASSERTS `delaytime=0` on each pass **while `backfill=1`** — # the one setting it writes, making the tier self-healing across # restarts. It is scoped to the backfill phase on purpose: with # `backfill=0` (pure forward tailing) the operator's own value stands. # ============================================================================= services: # --------------------------------------------------------------------------- # metafeederusenet-db — MariaDB backing nntmux's OWN release catalog. Not # shared with the gateway's meta-core Redis or anything else in the mesh. # --------------------------------------------------------------------------- metafeederusenet-db: image: mariadb:12.3 cpu_shares: 70 container_name: metafeederusenet-db hostname: metafeederusenet-db restart: unless-stopped # 2g, not the 512m this shipped with: 512m is fine for an empty schema and # far too small once the catalog is real. A header scan of a busy binary # group builds `collections`/`binaries`/`parts` tables in the tens of # millions of rows, and InnoDB starved of buffer pool turns collation into # a disk-thrash that never catches up. Measured on a live box. mem_limit: 2g memswap_limit: 2g environment: MARIADB_DATABASE: nntmux MARIADB_USER: nntmux MARIADB_PASSWORD: "$APP_DEFAULT_PASSWORD" MARIADB_RANDOM_ROOT_PASSWORD: "yes" volumes: - ${DATA_ROOT:-/DATA}/AppData/metafeederusenet/db:/var/lib/mysql healthcheck: test: ["CMD", "healthcheck.sh", "--connect", "--innodb_initialized"] interval: 5s timeout: 5s retries: 20 start_period: 20s networks: - metafeederusenet # --------------------------------------------------------------------------- # metafeederusenet-redis — nntmux's OWN Redis. Deliberately separate from the # gateway core's Redis (different failure domain; losing this one is free). # --------------------------------------------------------------------------- metafeederusenet-redis: image: redis:7-alpine cpu_shares: 70 container_name: metafeederusenet-redis hostname: metafeederusenet-redis restart: unless-stopped mem_limit: 128m memswap_limit: 128m networks: - metafeederusenet # --------------------------------------------------------------------------- # metafeederusenet-manticore — RT-index search backend. NOT optional (trap # #3 above) — a hard first-boot dependency of nntmux's own entrypoint, even # though this feeder reads `releases` via SQL directly and never touches it. # Digest-pinned (upstream reuses low-looking version numbers across an old # semver line and a newer CalVer-ish one; the digest is unambiguous). # --------------------------------------------------------------------------- metafeederusenet-manticore: image: manticoresearch/manticore@sha256:032388dc77321e3ac898a14b9a0c3fc22ab8d08cd2582abd6c5b642d46b82181 cpu_shares: 50 container_name: metafeederusenet-manticore hostname: metafeederusenet-manticore restart: unless-stopped mem_limit: 512m memswap_limit: 512m ulimits: nofile: { soft: 65535, hard: 65535 } networks: - metafeederusenet # --------------------------------------------------------------------------- # metafeederusenet-nntmux — the header scanner + web UI/API. Exposed DIRECTLY # via Caddy (no AppShield gate): nntmux ships its own admin login # (ADMIN_USER/ADMIN_PASS, seeded below), the same convention this store uses # for every app that brings its own auth (Jellyfin, Nextcloud, Immich, ...). # --------------------------------------------------------------------------- metafeederusenet-nntmux: # Digest-pinned, NOT :latest — a silent tag move could change the # `misc/update/*.php` command surface the "collate via tinker" tip below # depends on. Re-pin deliberately, re-verify the trap list above when you do. image: pyrowman/nntmux@sha256:7fc221f7a806fee3e4a2145159e1a147962edc095282bdc692806968f791fabe cpu_shares: 50 container_name: metafeederusenet-nntmux hostname: metafeederusenet-nntmux restart: unless-stopped # The working set scales with scan breadth (group set x backfill depth), an # operator choice (see tips), so this is a ceiling rather than a size: 3g # is what a real corpus needed on a live box — 1536m was reached and # exceeded by a single busy group's header scan. Raise it if you scan wide. mem_limit: 3g memswap_limit: 3g environment: # Shell-script-consumed vars — the entrypoint's own `mkdir -p` step reads # these from the PROCESS environment, not the mounted .env. Keep in # lockstep with the matching keys in nntmux.env below. PATH_TO_NZBS: /app/storage/nzb/ COVERS_PATH: /app/storage/covers TEMP_UNRAR_PATH: /app/storage/tmp/unrar/ TEMP_UNZIP_PATH: /app/storage/tmp/unzip/ ELASTICSEARCH_ENABLED: "false" volumes: # The complete, hand-filled .env (trap #1) — a full-file mount rather # than individual `environment:` keys. Writable: `nntmux:install` # persists a generated APP_KEY back into it on first boot. - ${DATA_ROOT:-/DATA}/AppData/metafeederusenet/nntmux-env/nntmux.env:/app/.env:rw # The .nzb store — shared with metafeederusenet-feeder below, # read-write here (nntmux writes), read-only there (feeder only reads). - ${DATA_ROOT:-/DATA}/AppData/metafeederusenet/nntmux-storage/nzb:/app/storage/nzb:rw healthcheck: test: ["CMD", "curl", "-sf", "-o", "/dev/null", "http://localhost/"] interval: 10s timeout: 5s retries: 15 start_period: 30s depends_on: metafeederusenet-db: condition: service_healthy metafeederusenet-redis: condition: service_started metafeederusenet-manticore: condition: service_started labels: caddy_0: metafeederusenetnntmux-${APP_DOMAIN} caddy_0.import: gateway_tls caddy_0.reverse_proxy: "{{upstreams 80}}" caddy_1: metafeederusenetnntmux-${APP_PUBLIC_IP_DASH}.nip.io caddy_1.import: gateway_tls caddy_1.reverse_proxy: "{{upstreams 80}}" caddy_2: metafeederusenetnntmux-${APP_PUBLIC_IP_DASH}.sslip.io caddy_2.reverse_proxy: "{{upstreams 80}}" networks: - metafeederusenet - pcs # --------------------------------------------------------------------------- # metafeederusenet-scanner — the SCAN LOOP (trap #6). The nntmux image runs # frankenphp and nothing else, so without this container nothing ever calls # the scan/backfill/collate entry points and the catalog stays empty forever, # no matter how correct the credentials, the groups or `settings.running` are. # # It is a second container off the SAME digest-pinned image, sharing the same # `/app/.env` and the same nzb store, rather than a second process inside the # web container — the web container's entrypoint runs install/populate under # `set -e` and must keep it; here it is REPLACED by `entrypoint:` below so # that install/populate does NOT run twice against one database. # # Every knob (interval, binaries on/off, backfill on/off, backfill_qty, # backfill_days, running) is read from nntmux's own `settings` table at # /admin/tmux-edit — deliberately NOT from new env vars here, which would # make two sources of truth for one number. # --------------------------------------------------------------------------- metafeederusenet-scanner: # MUST stay byte-identical to metafeederusenet-nntmux's pin above: the two # containers share one `/app/.env` and one database schema, and the script # below calls `misc/update/*.php` + `Blacklight\` classes by name. image: pyrowman/nntmux@sha256:7fc221f7a806fee3e4a2145159e1a147962edc095282bdc692806968f791fabe cpu_shares: 20 container_name: metafeederusenet-scanner hostname: metafeederusenet-scanner restart: unless-stopped # Bypasses the image's own install/populate entrypoint on purpose — see the # block comment above. Only the web container may run it. entrypoint: ["/bin/sh", "/scanner.sh"] # The image ships a HEALTHCHECK that curls its own web server. This # container deliberately runs no web server — its entrypoint IS the scan # loop — so the inherited probe can only ever fail. Observed live: a # scanner actively backfilling and writing NZBs sitting at # `Up 5 minutes (unhealthy)`. # # That is not cosmetic. A permanently-red container that is in fact working # trains the operator to ignore health status, which is precisely how this # stack's empty-catalog failure stayed invisible for so long — the feeder's # own health() reporting `ok` over a zero-row catalog was the same bug from # the other direction. Liveness for this container is its LOG # (`docker logs -f metafeederusenet-scanner`), not a probe; a probe that # cannot be honest is worse than none. healthcheck: disable: true # A backfill pass holds a group's article range in memory; 2g is the # headroom measured alongside nntmux's own 3g on a live box. mem_limit: 2g memswap_limit: 2g environment: # Same shell-script-consumed vars as metafeederusenet-nntmux: the wrapper # scripts read these from the PROCESS environment, not from the mounted # .env (trap #1). Keep the two blocks in lockstep. PATH_TO_NZBS: /app/storage/nzb/ COVERS_PATH: /app/storage/covers TEMP_UNRAR_PATH: /app/storage/tmp/unrar/ TEMP_UNZIP_PATH: /app/storage/tmp/unzip/ ELASTICSEARCH_ENABLED: "false" volumes: # The same .env the web container filled in — one credential set, one # APP_KEY. Read-write only because Laravel's .env handling expects it. - ${DATA_ROOT:-/DATA}/AppData/metafeederusenet/nntmux-env/nntmux.env:/app/.env:rw # nntmux writes .nzb.gz here; the feeder reads them read-only. - ${DATA_ROOT:-/DATA}/AppData/metafeederusenet/nntmux-storage/nzb:/app/storage/nzb:rw # The loop itself, provisioned by pre-install-cmd below. - ${DATA_ROOT:-/DATA}/AppData/metafeederusenet/scanner/scanner.sh:/scanner.sh:ro depends_on: # Not just `started`: the loop's very first action is an artisan call, so # it must not race the web container's install/migrate step. metafeederusenet-nntmux: condition: service_healthy networks: - metafeederusenet # --------------------------------------------------------------------------- # metafeederusenet-feeder — the usenet feeder sidecar (meta-feeder-usenet). # Reads nntmux's release catalog + shared .nzb store; self-publishes # resolved records + seeds the `.nzb` manifest to the shared MetaCore. The # gateway reaches it directly on `pcs`; the human configures it through the # AppShield gate below. # --------------------------------------------------------------------------- metafeederusenet-feeder: image: ghcr.io/worph/meta-feeder-usenet:1.0.2 cpu_shares: 30 container_name: metafeederusenet-feeder # Hard RAM cap, swap disabled (memswap_limit == mem_limit) → OOM instead # of swap. Mirrors the dev sub-stack limits. mem_limit: 256m memswap_limit: 256m hostname: metafeederusenet-feeder restart: unless-stopped user: "0:0" expose: - 8080 environment: # INFRA-ONLY env (the "config: no env if in webui" rule). Operator # config beyond the two DB/path seeds below — TMDB token / language, # meta-core URL — is read from the persisted config.json (this # feeder's config page behind the AppShield gate). These env vars are # wiring only: the enrichment sidecars, the enrich callback, the peer id. RUST_LOG: "info,meta_feeder=info" # The peer id stamped into provenance.gateway_peer on every record this # feeder self-publishes — names the GATEWAY this feeder serves, not the # feeder itself (analytics group by it). Without it, falls back to HOSTNAME. META_GATEWAY_PEER_ID: "metagateway" FILENAME_PARSER_URL: "http://metafeederusenet-filename-parser:8080" TMDB_PLUGIN_URL: "http://metafeederusenet-tmdb:8080" META_FEEDER_CALLBACK_URL: "http://metafeederusenet-feeder:8080/enrich/callback" # First-boot SEED only (the persisted config.json wins on every later # restart — change these on the /config page after first boot, not here). NNTMUX_DB_URL: "mysql://nntmux:${APP_DEFAULT_PASSWORD}@metafeederusenet-db:3306/nntmux" NNTMUX_NZB_PATH: "/nntmux-nzb" TZ: "$TZ" volumes: # Persist feeder config (config.json saved via the config UI) + redb cache. - ${DATA_ROOT:-/DATA}/AppData/metafeederusenet/feeder-state:/data/meta-feeder:rw # Same volume nntmux writes .nzb.gz into — read-only here. - ${DATA_ROOT:-/DATA}/AppData/metafeederusenet/nntmux-storage/nzb:/nntmux-nzb:ro healthcheck: test: ["CMD", "curl", "-sf", "http://localhost:8080/health"] interval: 5s timeout: 3s retries: 10 start_period: 5s depends_on: metafeederusenet-nntmux: condition: service_healthy metafeederusenet-filename-parser: condition: service_started metafeederusenet-tmdb: condition: service_started networks: - metafeederusenet - pcs # --------------------------------------------------------------------------- # Bundled enrichment plugins (self-contained). filename-parser per file; tmdb # once per release — tmdb writes posters/backdrops to the shared meta-core's # WebDAV (over pcs) so the ingester seeds them. No healthcheck: these plugin # images don't ship one (curl absent). # --------------------------------------------------------------------------- metafeederusenet-filename-parser: image: ghcr.io/worph/metamesh-plugin-filename-parser:1.0.2 cpu_shares: 30 container_name: metafeederusenet-filename-parser mem_limit: 256m memswap_limit: 256m hostname: metafeederusenet-filename-parser restart: unless-stopped expose: - 8080 environment: EMIT_TITLE_TAGS: "true" networks: - metafeederusenet metafeederusenet-tmdb: image: ghcr.io/worph/metamesh-plugin-tmdb:1.0.2 cpu_shares: 30 container_name: metafeederusenet-tmdb mem_limit: 384m memswap_limit: 384m hostname: metafeederusenet-tmdb restart: unless-stopped expose: - 8080 environment: # Posters land in the shared meta-core (over pcs) so the ingester seeds # them. LOAD-BEARING at metamesh-plugin-tmdb:1.0.0 — do NOT remove (see # the sibling feeder apps' identical note: the published image needs # this, unlike the source build the dev stack runs). WEBDAV_URL: "http://metacore-app:9000/webdav" volumes: - ${DATA_ROOT:-/DATA}/AppData/metafeederusenet/tmdb-cache:/cache networks: - metafeederusenet - pcs # --------------------------------------------------------------------------- # AppShield gate — the feeder's config UI (nntmux DB URL / nzb path, TMDB # token) behind Yundera OIDC SSO. container_name == subdomain prefix. The # gateway still reaches the feeder directly on `pcs` (this gate is for the # human only); index → /config. # --------------------------------------------------------------------------- metafeederusenet: image: ghcr.io/yundera/appshield:2.0.9 cpu_shares: 80 container_name: metafeederusenet hostname: metafeederusenet restart: unless-stopped user: "0:0" expose: - 80 environment: OIDC_REGISTRAR_URL: "http://auth-registrar:9092" REDIRECT_HOST_SUFFIXES: "${APP_DOMAIN},${APP_PUBLIC_IP_DASH}.nip.io,${APP_PUBLIC_IP_DASH}.sslip.io" BACKEND_HOST: "metafeederusenet-feeder" BACKEND_PORT: "8080" LISTEN_PORT: "80" labels: caddy_0: metafeederusenet-${APP_DOMAIN} caddy_0.import: gateway_tls caddy_0.reverse_proxy: "{{upstreams 80}}" caddy_1: metafeederusenet-${APP_PUBLIC_IP_DASH}.nip.io caddy_1.import: gateway_tls caddy_1.reverse_proxy: "{{upstreams 80}}" caddy_2: metafeederusenet-${APP_PUBLIC_IP_DASH}.sslip.io caddy_2.reverse_proxy: "{{upstreams 80}}" depends_on: - metafeederusenet-feeder networks: - metafeederusenet - pcs networks: metafeederusenet: name: metafeederusenet driver: bridge pcs: name: ${APP_NET:-pcs} external: true x-casaos: architectures: - amd64 author: Worph Team category: Media developer: Aptero Team main: metafeederusenet webui_port: 80 index: /config description: en_us: | **A standalone, self-hosted Usenet feeder for the MetaMesh network.** MetaFeeder · Usenet runs its own Usenet header scan (a bundled **nntmux** indexer) instead of proxying a third-party Newznab site. Because it finds article Message-IDs itself at scan time, it mints a **portable** posting reference — no indexer host baked in, redeemable by any peer with a plain NNTP provider account. It adds a Usenet source alongside **MetaFeeder · Prowlarr**; it doesn't replace it. **Self-contained:** ships its own nntmux scanner (MariaDB + Redis + Manticore) + filename-parser + TMDB enrichment, so each release gets a clean title, poster and plot. **You still need a Usenet provider account** (NNTP login) — this app scans headers, it does not supply an account, and nothing at all is indexed until you supply one. Configure it after install (see the install tips). Keep the connection count low: this scanner and your playback share the same provider connection slots. **The catalog is not instant.** Once configured, a bundled scanner container runs the header-scan / backfill / collation loop on the interval you set. A first pass over a few busy binary groups pulls tens of GB and runs for hours; a deep backfill is hundreds of GB. The install tips show which knobs bound that. **Also requires:** **MetaGateway** — its meta-core stores the resolved records and announces them to the swarm. The gateway must list this feeder in its config. fr_fr: | **Un feeder Usenet autonome et auto-hébergé pour le réseau MetaMesh.** MetaFeeder · Usenet effectue lui-même le scan des en-têtes Usenet (un indexeur **nntmux** embarqué) plutôt que de passer par un site Newznab tiers. Comme il retrouve les Message-ID des articles dès le scan, il génère une référence de posting **portable** — sans hôte d'indexeur intégré, exploitable par n'importe quel pair disposant d'un simple compte NNTP. Il ajoute une source Usenet en plus de **MetaFeeder - Indexer**, sans le remplacer. **Autonome :** embarque son scanner nntmux (MariaDB + Redis + Manticore) + ses plugins filename-parser + TMDB. **Un compte fournisseur Usenet (NNTP) reste nécessaire** — cette app scanne les en-têtes, elle ne fournit pas d'accès, et rien n'est indexé tant que vous n'en fournissez pas un. À configurer après installation (voir les conseils d'installation). Gardez un nombre de connexions faible : le scan et la lecture partagent les mêmes créneaux du fournisseur. **Le catalogue ne se remplit pas instantanément.** Une fois configuré, un conteneur de scan embarqué exécute la boucle scan / backfill / collation à l'intervalle que vous choisissez. Un premier passage sur quelques gros groupes binaires télécharge des dizaines de Go et dure des heures ; un backfill profond, des centaines de Go. **Nécessite aussi :** l'app **MetaGateway** (son meta-core stocke et annonce les enregistrements). tagline: en_us: Self-hosted Usenet header scanner (bundled nntmux) for the MetaMesh network fr_fr: Scanner Usenet auto-hébergé (nntmux inclus) pour le réseau MetaMesh title: en_us: MetaFeeder · Usenet icon: https://cdn.jsdelivr.net/gh/Worph/MetaAppStore@main/Apps/MetaFeederUsenet/icon.png screenshot_link: - https://cdn.jsdelivr.net/gh/Worph/MetaAppStore@main/Apps/MetaFeederUsenet/screenshot-1.png - https://cdn.jsdelivr.net/gh/Worph/MetaAppStore@main/Apps/MetaFeederUsenet/screenshot-2.png - https://cdn.jsdelivr.net/gh/Worph/MetaAppStore@main/Apps/MetaFeederUsenet/screenshot-3.png thumbnail: https://cdn.jsdelivr.net/gh/Worph/MetaAppStore@main/Apps/MetaFeederUsenet/thumbnail.png store_app_id: metafeederusenet is_uncontrolled: false tips: before_install: en_us: | **Install order:** install **MetaGateway** first (this feeder publishes to its shared meta-core). **After installing this app, a Usenet provider account is required** (this scans headers itself — it needs a plain NNTP login): 1. Edit `/DATA/AppData/metafeederusenet/nntmux-env/nntmux.env` (via SSH or a file-manager app) and fill in `NNTP_SERVER`, `NNTP_PORT`, `NNTP_USERNAME`, `NNTP_PASSWORD` (use the SAME provider account your MetaShare/meta-share peer is already credentialed against — one account, two consumers). Keep `NNTP_CONNECTIONS` low (1-2): the scanner and playback share the same provider connection slots, so a greedy value here stalls playback mid-stream elsewhere. 2. Restart the nntmux container: `docker restart metafeederusenet-nntmux`. 3. **First boot only:** nntmux's install sequence caches its config before generating an encryption key, so every page 500s until you run `docker exec metafeederusenet-nntmux php artisan config:clear` once. (`curl -sI https://metafeederusenetnntmux-/` — a `500` means you need this; a `302` redirect to `/login` means you're fine.) 4. Log in to the nntmux dashboard at the **MetaFeeder Usenet nntmux** link (`ADMIN_USER=admin`, password = this app's auto-generated default password — check CasaOS's app credentials panel) and change it. **⚠ Do steps 5-6 AFTER your LAST restart, and re-check them after every later restart or stack update.** The nntmux entrypoint RE-SEEDS the `settings` and `usenet_groups` tables on EVERY container boot, so anything you configure before a restart silently reverts (observed: `running` back to 0, `delaytime` back to 2, every group back to `active=0`) — with no error anywhere. 5. **Activate at least one usenet group** — seeded groups start inactive (`active=0`), and scanning an inactive group silently touches nothing at all: ``` docker exec metafeederusenet-db mariadb -unntmux -p"" nntmux \ -e "UPDATE usenet_groups SET active=1, backfill=1 WHERE name IN ('alt.binaries.teevee');" ``` 6. **Start the scheduler** in nntmux's admin UI (**/admin/tmux-edit**) or directly in the `settings` table: set `running=1`, pick your `monitor_delay` (seconds between passes), and turn on `binaries` and/or `backfill` with a `backfill_qty` / `backfill_days` you're happy to pay for. The bundled `metafeederusenet-scanner` container polls these settings and runs the scan/backfill/collate loop for you — the nntmux image itself ships NO cron and NO queue worker, so this container is what actually makes a scan happen. There are deliberately no env vars for these knobs: the `settings` table is the single source of truth. **`delaytime` (HOURS, default 2) will stall a backfilled catalog.** nntmux won't turn a collection into a release until it has aged past `delaytime`, which is right for tailing live posts and pointless when backfilling history — and the symptom (collections climbing, releases flat, zero query results) looks exactly like a broken pipeline. **While `backfill=1`, the scanner forces `delaytime=0` for you on every pass** and logs `asserting delaytime=0 for the backfill phase (was N)` when it does — it has to re-assert it because a container restart re-seeds the value back to 2. This is the only setting the loop ever writes. So if you want a different steady-state `delaytime`, **set `backfill=0` first**, then set your value — otherwise the loop will keep pulling it back to 0. With `backfill=0` the scanner never touches it and `1`-`2` is the sensible live-tailing value. **Budget the first scan.** A first header pass over a few busy binary groups pulls tens of GB and runs for hours; a full-retention backfill is hundreds of GB. `backfill_days` is the knob that bounds it — start small, widen once you see releases. 7. Open this app's config UI (the dashboard link) to optionally set a TMDB v4 token for posters, then Save. 8. In **MetaGateway**, make sure this feeder is listed in its feeders config (`http://metafeederusenet-feeder:8080`) so it gets discovered. **Note:** which groups to scan, backfill depth and scan interval are your call as the operator — pick them for your own catalog. **Watch the loop:** `docker logs -f metafeederusenet-scanner` prints one line per pass (`header scan` / `backfill direct` / `collate ` / `sleeping`). `settings.running=0` shows up there as `idle`. **Authentication:** the feeder's config UI is protected by Yundera's Authelia (OIDC); nntmux uses its own login (seeded above). pre-install-cmd: | # Seed nntmux's .env (trap #1 — must be a complete file, see header). # NNTP_* are deliberately left blank; fill them in per the install tips. if [ ! -f /DATA/AppData/metafeederusenet/nntmux-env/nntmux.env ]; then cat > /DATA/AppData/metafeederusenet/nntmux-env/nntmux.env < /DATA/AppData/metafeederusenet/scanner/scanner.sh <<'SCANNER_SH' #!/bin/sh # nntmux scan launcher. # # The pyrowman/nntmux image runs frankenphp and NOTHING else: no cron, no # supervisord, no queue worker. So `settings.running=1` changes nothing on # its own, because nothing ever starts the loop. This is that loop. # # It runs as a SECOND CONTAINER off the same image + the same /app/.env and # nzb volume, rather than as a second process inside the web container, so # the web container's entrypoint (install/populate under `set -e`) is left # untouched. # # INVOCATION IS NOT FREE CHOICE. Measured PER RUNNER on a live box, NOT a # blanket "Forking works": # * multiprocessing/binaries.php — Forking OK. # * multiprocessing/backfill.php — NO-OP. Reports "...3 job(s) to do # using a max of 0 child process(es)" and returns in 0s having done # nothing. Backfill must be driven DIRECTLY, per group (see the # backfill block below). # * collation MUST go through tinker/ProcessReleases DIRECTLY. # multiprocessing/releases.php aborts every batch on a benign PHP # warning (ReleaseCleaning.php:165) that turns fatal inside the Spatie # child -> 0 releases while collections pile into the tens of # thousands. # # Every knob comes from nntmux's own `settings` table (edited at # /admin/tmux-edit), never from env — otherwise there are two sources of # truth for one number. set -u cd /app || exit 1 setting() { php artisan tinker --execute="echo DB::table('settings')->where('name','$1')->value('value');" 2>/dev/null | tr -dc '0-9'; } set_setting() { php artisan tinker --execute="DB::table('settings')->where('name','$1')->update(['value'=>'$2']);" >/dev/null 2>&1; } groups() { php artisan tinker --execute="foreach (DB::table('usenet_groups')->where('active',1)->pluck('name') as \$g) echo \$g, PHP_EOL;" 2>/dev/null | grep -E '^[a-z0-9.]+$'; } log() { echo "[scanner $(date -u +%H:%M:%S)] $*"; } log "launcher started" while true; do if [ "$(setting running)" != "1" ]; then log "settings.running=0 — idle" sleep 30 continue fi if [ "$(setting binaries)" = "1" ]; then qty="$(setting backfill_qty)"; [ -n "$qty" ] || qty=20000 log "header scan (qty=$qty)" php misc/update/multiprocessing/binaries.php "$qty" 2>&1 | tail -3 fi # Backfill must run DIRECT, per group. multiprocessing/backfill.php # reports "max of 0 child process(es)" and silently does nothing — the # Forking wrapper leaves maxProcesses=0 for this runner specifically. # Depth is capped by settings.backfill_days, so this reaches back that # far and then stops advancing; it is NOT a full-retention backfill. if [ "$(setting backfill)" = "1" ]; then # The ONLY setting this loop writes. `delaytime` (hours, default 2) # blocks ProcessReleases from advancing a collection until it ages # past it. Correct for the leading edge — a posting may still be # uploading — but pointless for backfill, where every article is # already days old. Left at 2 during backfill it stalls the catalog # for two hours with a failure mode indistinguishable from a broken # pipeline (collections climb, releases flatline, nothing logs an # error). # # Asserted every pass, not once at boot, because the nntmux # entrypoint RE-SEEDS `settings` on every container start (trap #7) # — a restart silently puts this back to 2 and the tier quietly # stops producing. Scoped to `backfill=1` so a pure forward-tailing # deployment (`backfill=0`) keeps the operator's own value: this is # a per-phase mode, not a preference we get to override. if [ "$(setting delaytime)" != "0" ]; then log "asserting delaytime=0 for the backfill phase (was $(setting delaytime))" set_setting delaytime 0 fi qty="$(setting backfill_qty)"; [ -n "$qty" ] || qty=20000 log "backfill direct (backfill_days=$(setting backfill_days), qty=$qty)" groups | while read -r g; do [ -n "$g" ] || continue php misc/update/backfill.php "$g" "$qty" 2>&1 | grep -iE "articles|backfill|Group|no new" | tail -2 done fi # Collation, per active group, DIRECT (never multiprocessing/releases.php). # Never wrap this in a short `timeout`: killing it between the release # INSERT and createNZBs() leaves a release row with no .nzb.gz, which # the feeder then correctly skips forever. groups | while read -r g; do [ -n "$g" ] || continue log "collate $g" php artisan tinker --execute="(new Blacklight\\processing\\ProcessReleases)->processReleases(1, 1, \"$g\", new Blacklight\\NNTP);" 2>&1 | grep -iE "releases (ready|created)|Added [0-9]+ releases|NZB" | tail -3 done delay="$(setting monitor_delay)"; [ -n "$delay" ] || delay=30 log "pass complete; sleeping ${delay}s" sleep "$delay" done SCANNER_SH chmod 755 /DATA/AppData/metafeederusenet/scanner/scanner.sh # Maison (Yundera dashboard) configuration. Read in preference to x-casaos. # schema_version 2 is required for `folders`: an older Maison refuses the app # outright rather than starting it without its directories. x-compose-app: schema_version: 2 # Created and chowned by Maison before any image is pulled, before any hook # runs and before the containers start — on the first boot and every one after. # Compose would otherwise create a missing bind source as a root-owned dir. folders: - path: /DATA/AppData/metafeederusenet/db user: $PUID group: $PGID - path: /DATA/AppData/metafeederusenet/nntmux-env user: $PUID group: $PGID - path: /DATA/AppData/metafeederusenet/nntmux-storage/nzb user: $PUID group: $PGID - path: /DATA/AppData/metafeederusenet/feeder-state user: $PUID group: $PGID - path: /DATA/AppData/metafeederusenet/tmdb-cache user: $PUID group: $PGID - path: /DATA/AppData/metafeederusenet/scanner user: $PUID group: $PGID # Same string as the caddy_0 label, so the tile link and the route agree. webui-host: metafeederusenet-${domain} webui-path: "/config"