metastar/MetaAppStore
4
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

MetaFeederUsenet: glue only (feeder 1.1.0 + enrichment), talks to the NNTmux app over its Newznab API; nntmux moved to worph/AppStore

Browse Source
This commit is contained in:
Worph
2026-10-04 12:30:13 +00:00
parent f2f206e5c8
commit d0389498ed
2 changed files with 81 additions and 743 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Apps/MetaFeederUsenet/docker-compose.yml
+67 -700
View File
@@ -1,313 +1,43 @@
name: metafeederusenet
# =============================================================================
# MetaFeeder · Usenet — a STANDALONE, self-hosted Usenet feeder for MetaMesh.
# MetaFeeder · Usenet — publishes YOUR OWN Usenet indexer to MetaMesh.
#
# Unlike MetaFeeder · Prowlarr (which proxies an EXTERNAL Newznab/Torznab
# indexer and mints an `nzb-release` (0x1005) cid embedding that indexer's
# host), this app runs its OWN header scan (an `nntmux` — newznab-tmux fork —
# sidecar) so it can mint a PORTABLE `nzb-posting` (0x1003) cid: a digest over
# the article Message-ID set, embedding no indexer host, redeemable by ANY
# peer with a plain NNTP provider account. It ADDS a Usenet source; it does
# not replace the Prowlarr/Torznab one — a gateway query fans out to both.
# ⚠ NNTmux is a SEPARATE app (worph/AppStore) — install it first. This feeder
# reaches it on the shared `pcs` network at `http://nntmux` over its Newznab
# API (`/api/v1/api` + `/getnzb`), with the API key of NNTmux's `metamesh`
# user — the same contract MetaFeeder · Prowlarr uses with Prowlarr. It never
# touches NNTmux's database or storage.
#
# Unlike MetaFeeder · Prowlarr (which proxies EXTERNAL indexers and mints an
# `nzb-release` (0x1005) cid embedding that indexer's host), every release
# here comes from a scan you run, so the feeder downloads its .nzb once and
# mints a PORTABLE `nzb-posting` (0x1003) cid: a digest over the article
# Message-ID set, redeemable by ANY peer with a plain NNTP provider account.
# It also redeems external `nzb-release` locators with per-indexer keys set on
# its config page.
#
# A headless service: no web UI of its own. Configure it from MetaGateway →
# this feeder → *configure* (nntmux URL + API key, indexer keys).
#
# It needs, over `pcs`: **NNTmux** (`nntmux`), the shared **MetaCore**
# (`metacore-app:9000`, records + TMDB posters) and **MetaGateway**, which must
# list `http://metafeederusenet-feeder:8080` in its feeders config.
#
# Design record (meta-gateway repo): docs/others/self-hosted-usenet-indexer-study.md
# Feeder repo: github.com/worph/meta-feeder-usenet
#
# It needs, over the shared `pcs` network:
# - the shared **MetaCore** app (`metacore-app:9000`) — where resolved
# records + TMDB posters are stored; the shared **MetaShare** app then
# announces them to the swarm.
# The **MetaGateway** app must list this feeder's URL
# (`http://metafeederusenet-feeder:8080`) in its feeders config so it
# discovers + queries it (MetaGateway, MetaCore and this feeder share `pcs`).
#
# Everything is scoped under the `metafeederusenet-*` prefix + the per-stack
# `metafeederusenet` network + /DATA/AppData/metafeederusenet/ data root, so it
# installs alongside the other Meta* apps with zero collisions.
#
# amd64 ONLY — `pyrowman/nntmux` ships no arm64 build.
#
# ⚠ Four silent-failure traps in the `pyrowman/nntmux` image, empirically
# verified (see the meta-gateway dev compose header for the full writeup):
# 1. `PATH_TO_NZBS` (and every other storage path var) must be ABSENT from
# the env, never present-but-blank — that reads as an explicit empty
# string, not "unset", and nntmux aborts NZB writing with 0 releases and
# no visible error. This compose bind-mounts a COMPLETE, hand-filled
# `.env` (below) rather than relying on individual `environment:` keys,
# for exactly this reason.
# 2. `DB_CONNECTION` must be `mariadb`, not `mysql` — the install migration
# loads its schema file by connection name; `mysql` finds no matching
# schema and fails partway through a from-scratch migration.
# 3. Manticore is a HARD first-boot dependency (unless
# `ELASTICSEARCH_ENABLED=true`, which just swaps it for a heavier one) —
# `metafeederusenet-manticore` below is not optional.
# 4. Seeded usenet groups default to `active=0, backfill=0` — scanning an
# inactive group silently does nothing. See the tips below for the SQL
# to activate one.
# One FIRST-BOOT-ONLY loud bug: the entrypoint runs `config:cache` before
# `key:generate`, so every session route 500s until you run
# `docker exec metafeederusenet-nntmux php artisan config:clear` once. See tips.
#
# ⚠ Four more, found the hard way on a live box (2026-07-26) after this stack
# came up with SEVEN healthy containers and returned ZERO results for every
# query — the feeder answering `"Done"` and nothing else. None of them logs
# anything; each one alone is enough to keep the catalog empty forever.
#
# 5. NEVER set a key in `nntmux.env` unless the value must DIFFER from the
# PHP default. `config/nntmux.php` does
# `env('NN_MULTIPROCESSING_MAX_CHILD_TIME', 1800)` and Laravel's `env()`
# returns STRINGS, so restating the default in the .env handed
# `Pool::timeout(float)` a `string("1800")` → TypeError → every Forking
# multiprocessing runner died on start. Those runners are exactly what a
# scan needs. This is trap #1's type-flavoured twin: a redundant
# restatement is not a no-op, it silently changes the value's TYPE. The
# key is therefore deliberately ABSENT from the seeded .env below (see the
# comment there); with it absent, `gettype()` reports `integer` and the
# runners execute. Same rule applies to any other key you are tempted to
# "document" by writing its default into the file.
# (Upstream record: meta-gateway `dev/docker-compose.feeder-usenet.yml`,
# the "CORRECTION (2026-07-26)" block.)
# 6. The image runs `frankenphp` and NOTHING ELSE — no cron, no supervisord,
# no queue worker. nntmux's own `settings.running=1` therefore starts
# nothing: there is no process to read it. Even with valid NNTP creds and
# active groups the catalog stays empty permanently. That is why
# `metafeederusenet-scanner` below exists — it is the missing loop, run as
# a SECOND container off the same image so the web container's
# install/populate entrypoint stays untouched.
# 7. The nntmux entrypoint RE-SEEDS `settings` and `usenet_groups` on EVERY
# container boot. Configuration applied before a restart silently reverts
# (observed: `running`→0, `delaytime`→2, every group→`active=0`). Apply
# group activation and scheduler settings AFTER the final restart, and
# re-check them after any `docker restart` / stack update.
# 8. `settings.delaytime` (HOURS, default 2) stalls a backfilled catalog:
# `ProcessReleases` refuses to advance a collection until it has aged past
# it. That is correct for live tailing (a posting may still be uploading)
# and pointless for backfill, where every article is already days old —
# and the symptom, collections climbing while releases flatline, is
# indistinguishable from a broken pipeline because nothing logs an error.
# Because trap #7 re-seeds it back to 2 on every restart, the scanner
# loop below ASSERTS `delaytime=0` on each pass **while `backfill=1`** —
# the one setting it writes, making the tier self-healing across
# restarts. It is scoped to the backfill phase on purpose: with
# `backfill=0` (pure forward tailing) the operator's own value stands.
# =============================================================================
services:
# ---------------------------------------------------------------------------
# metafeederusenet-db — MariaDB backing nntmux's OWN release catalog. Not
# shared with the gateway's meta-core Redis or anything else in the mesh.
# ---------------------------------------------------------------------------
metafeederusenet-db:
image: mariadb:12.3
cpu_shares: 70
container_name: metafeederusenet-db
hostname: metafeederusenet-db
restart: unless-stopped
# 2g, not the 512m this shipped with: 512m is fine for an empty schema and
# far too small once the catalog is real. A header scan of a busy binary
# group builds `collections`/`binaries`/`parts` tables in the tens of
# millions of rows, and InnoDB starved of buffer pool turns collation into
# a disk-thrash that never catches up. Measured on a live box.
mem_limit: 2g
memswap_limit: 2g
environment:
MARIADB_DATABASE: nntmux
MARIADB_USER: nntmux
MARIADB_PASSWORD: "$APP_DEFAULT_PASSWORD"
MARIADB_RANDOM_ROOT_PASSWORD: "yes"
volumes:
- ${DATA_ROOT:-/DATA}/AppData/metafeederusenet/db:/var/lib/mysql
healthcheck:
test: ["CMD", "healthcheck.sh", "--connect", "--innodb_initialized"]
interval: 5s
timeout: 5s
retries: 20
start_period: 20s
networks:
- metafeederusenet
# ---------------------------------------------------------------------------
# metafeederusenet-redis — nntmux's OWN Redis. Deliberately separate from the
# gateway core's Redis (different failure domain; losing this one is free).
# ---------------------------------------------------------------------------
metafeederusenet-redis:
image: redis:7-alpine
cpu_shares: 70
container_name: metafeederusenet-redis
hostname: metafeederusenet-redis
restart: unless-stopped
mem_limit: 128m
memswap_limit: 128m
networks:
- metafeederusenet
# ---------------------------------------------------------------------------
# metafeederusenet-manticore — RT-index search backend. NOT optional (trap
# #3 above) — a hard first-boot dependency of nntmux's own entrypoint, even
# though this feeder reads `releases` via SQL directly and never touches it.
# Digest-pinned (upstream reuses low-looking version numbers across an old
# semver line and a newer CalVer-ish one; the digest is unambiguous).
# ---------------------------------------------------------------------------
metafeederusenet-manticore:
image: manticoresearch/manticore@sha256:032388dc77321e3ac898a14b9a0c3fc22ab8d08cd2582abd6c5b642d46b82181
cpu_shares: 50
container_name: metafeederusenet-manticore
hostname: metafeederusenet-manticore
restart: unless-stopped
mem_limit: 512m
memswap_limit: 512m
ulimits:
nofile: { soft: 65535, hard: 65535 }
networks:
- metafeederusenet
# ---------------------------------------------------------------------------
# metafeederusenet-nntmux — the header scanner + web UI/API. Exposed DIRECTLY
# via Caddy (no AppShield gate): nntmux ships its own admin login
# (ADMIN_USER/ADMIN_PASS, seeded below), the same convention this store uses
# for every app that brings its own auth (Jellyfin, Nextcloud, Immich, ...).
# ---------------------------------------------------------------------------
metafeederusenet-nntmux:
# Digest-pinned, NOT :latest — a silent tag move could change the
# `misc/update/*.php` command surface the "collate via tinker" tip below
# depends on. Re-pin deliberately, re-verify the trap list above when you do.
image: pyrowman/nntmux@sha256:7fc221f7a806fee3e4a2145159e1a147962edc095282bdc692806968f791fabe
cpu_shares: 50
container_name: metafeederusenet-nntmux
hostname: metafeederusenet-nntmux
restart: unless-stopped
# The working set scales with scan breadth (group set x backfill depth), an
# operator choice (see tips), so this is a ceiling rather than a size: 3g
# is what a real corpus needed on a live box — 1536m was reached and
# exceeded by a single busy group's header scan. Raise it if you scan wide.
mem_limit: 3g
memswap_limit: 3g
environment:
# Shell-script-consumed vars — the entrypoint's own `mkdir -p` step reads
# these from the PROCESS environment, not the mounted .env. Keep in
# lockstep with the matching keys in nntmux.env below.
PATH_TO_NZBS: /app/storage/nzb/
COVERS_PATH: /app/storage/covers
TEMP_UNRAR_PATH: /app/storage/tmp/unrar/
TEMP_UNZIP_PATH: /app/storage/tmp/unzip/
ELASTICSEARCH_ENABLED: "false"
volumes:
# The complete, hand-filled .env (trap #1) — a full-file mount rather
# than individual `environment:` keys. Writable: `nntmux:install`
# persists a generated APP_KEY back into it on first boot.
- ${DATA_ROOT:-/DATA}/AppData/metafeederusenet/nntmux-env/nntmux.env:/app/.env:rw
# The .nzb store — shared with metafeederusenet-feeder below,
# read-write here (nntmux writes), read-only there (feeder only reads).
- ${DATA_ROOT:-/DATA}/AppData/metafeederusenet/nntmux-storage/nzb:/app/storage/nzb:rw
healthcheck:
test: ["CMD", "curl", "-sf", "-o", "/dev/null", "http://localhost/"]
interval: 10s
timeout: 5s
retries: 15
start_period: 30s
depends_on:
metafeederusenet-db:
condition: service_healthy
metafeederusenet-redis:
condition: service_started
metafeederusenet-manticore:
condition: service_started
labels:
caddy_0: metafeederusenetnntmux-${APP_DOMAIN}
caddy_0.import: gateway_tls
caddy_0.reverse_proxy: "{{upstreams 80}}"
caddy_1: metafeederusenetnntmux-${APP_PUBLIC_IP_DASH}.nip.io
caddy_1.import: gateway_tls
caddy_1.reverse_proxy: "{{upstreams 80}}"
caddy_2: metafeederusenetnntmux-${APP_PUBLIC_IP_DASH}.sslip.io
caddy_2.reverse_proxy: "{{upstreams 80}}"
networks:
- metafeederusenet
- pcs
# ---------------------------------------------------------------------------
# metafeederusenet-scanner — the SCAN LOOP (trap #6). The nntmux image runs
# frankenphp and nothing else, so without this container nothing ever calls
# the scan/backfill/collate entry points and the catalog stays empty forever,
# no matter how correct the credentials, the groups or `settings.running` are.
#
# It is a second container off the SAME digest-pinned image, sharing the same
# `/app/.env` and the same nzb store, rather than a second process inside the
# web container — the web container's entrypoint runs install/populate under
# `set -e` and must keep it; here it is REPLACED by `entrypoint:` below so
# that install/populate does NOT run twice against one database.
#
# Every knob (interval, binaries on/off, backfill on/off, backfill_qty,
# backfill_days, running) is read from nntmux's own `settings` table at
# /admin/tmux-edit — deliberately NOT from new env vars here, which would
# make two sources of truth for one number.
# ---------------------------------------------------------------------------
metafeederusenet-scanner:
# MUST stay byte-identical to metafeederusenet-nntmux's pin above: the two
# containers share one `/app/.env` and one database schema, and the script
# below calls `misc/update/*.php` + `Blacklight\` classes by name.
image: pyrowman/nntmux@sha256:7fc221f7a806fee3e4a2145159e1a147962edc095282bdc692806968f791fabe
cpu_shares: 20
container_name: metafeederusenet-scanner
hostname: metafeederusenet-scanner
restart: unless-stopped
# Bypasses the image's own install/populate entrypoint on purpose — see the
# block comment above. Only the web container may run it.
entrypoint: ["/bin/sh", "/scanner.sh"]
# The image ships a HEALTHCHECK that curls its own web server. This
# container deliberately runs no web server — its entrypoint IS the scan
# loop — so the inherited probe can only ever fail. Observed live: a
# scanner actively backfilling and writing NZBs sitting at
# `Up 5 minutes (unhealthy)`.
#
# That is not cosmetic. A permanently-red container that is in fact working
# trains the operator to ignore health status, which is precisely how this
# stack's empty-catalog failure stayed invisible for so long — the feeder's
# own health() reporting `ok` over a zero-row catalog was the same bug from
# the other direction. Liveness for this container is its LOG
# (`docker logs -f metafeederusenet-scanner`), not a probe; a probe that
# cannot be honest is worse than none.
healthcheck:
disable: true
# A backfill pass holds a group's article range in memory; 2g is the
# headroom measured alongside nntmux's own 3g on a live box.
mem_limit: 2g
memswap_limit: 2g
environment:
# Same shell-script-consumed vars as metafeederusenet-nntmux: the wrapper
# scripts read these from the PROCESS environment, not from the mounted
# .env (trap #1). Keep the two blocks in lockstep.
PATH_TO_NZBS: /app/storage/nzb/
COVERS_PATH: /app/storage/covers
TEMP_UNRAR_PATH: /app/storage/tmp/unrar/
TEMP_UNZIP_PATH: /app/storage/tmp/unzip/
ELASTICSEARCH_ENABLED: "false"
volumes:
# The same .env the web container filled in — one credential set, one
# APP_KEY. Read-write only because Laravel's .env handling expects it.
- ${DATA_ROOT:-/DATA}/AppData/metafeederusenet/nntmux-env/nntmux.env:/app/.env:rw
# nntmux writes .nzb.gz here; the feeder reads them read-only.
- ${DATA_ROOT:-/DATA}/AppData/metafeederusenet/nntmux-storage/nzb:/app/storage/nzb:rw
# The loop itself, provisioned by pre-install-cmd below.
- ${DATA_ROOT:-/DATA}/AppData/metafeederusenet/scanner/scanner.sh:/scanner.sh:ro
depends_on:
# Not just `started`: the loop's very first action is an artisan call, so
# it must not race the web container's install/migrate step.
metafeederusenet-nntmux:
condition: service_healthy
networks:
- metafeederusenet
# ---------------------------------------------------------------------------
# metafeederusenet-feeder — the usenet feeder sidecar (meta-feeder-usenet).
# Reads nntmux's release catalog + shared .nzb store; self-publishes
# resolved records + seeds the `.nzb` manifest to the shared MetaCore. The
# Searches NNTmux over its Newznab API; self-publishes resolved records +
# seeds the `.nzb` manifest to the shared MetaCore. The
# gateway reaches it directly on `pcs`; the human configures it from
# the MetaGateway dashboard (configure panel).
# ---------------------------------------------------------------------------
metafeederusenet-feeder:
image: ghcr.io/worph/meta-feeder-usenet:1.0.6
image: ghcr.io/worph/meta-feeder-usenet:1.1.0
cpu_shares: 30
container_name: metafeederusenet-feeder
# Hard RAM cap, swap disabled (memswap_limit == mem_limit) → OOM instead
@@ -321,7 +51,7 @@ services:
- 8080
environment:
# INFRA-ONLY env (the "config: no env if in webui" rule). Operator
# config beyond the two DB/path seeds below — TMDB token / language,
# config beyond the NNTmux URL seed below — TMDB token / language,
# meta-core URL — is read from the persisted config.json (this
# feeder's config page, opened from the MetaGateway dashboard (configure panel)). These env vars are
# wiring only: the enrichment sidecars, the enrich callback, the peer id.
@@ -335,14 +65,13 @@ services:
META_FEEDER_CALLBACK_URL: "http://metafeederusenet-feeder:8080/enrich/callback"
# First-boot SEED only (the persisted config.json wins on every later
# restart — change these on the /config page after first boot, not here).
NNTMUX_DB_URL: "mysql://nntmux:${APP_DEFAULT_PASSWORD}@metafeederusenet-db:3306/nntmux"
NNTMUX_NZB_PATH: "/nntmux-nzb"
# The API key has no env seed: paste NNTmux's `metamesh` key on the page.
NNTMUX_URL: "http://nntmux"
TZ: "$TZ"
volumes:
# Persist feeder config (config.json saved via the config UI) + redb cache.
# Persist feeder config (config.json saved via the config UI), the redb
# cache, and the per-release .nzb cache (gateway/usenet/nntmux-nzb/).
- ${DATA_ROOT:-/DATA}/AppData/metafeederusenet/feeder-state:/data/meta-feeder:rw
# Same volume nntmux writes .nzb.gz into — read-only here.
- ${DATA_ROOT:-/DATA}/AppData/metafeederusenet/nntmux-storage/nzb:/nntmux-nzb:ro
healthcheck:
test: ["CMD", "curl", "-sf", "http://localhost:8080/health"]
interval: 5s
@@ -350,8 +79,6 @@ services:
retries: 10
start_period: 5s
depends_on:
metafeederusenet-nntmux:
condition: service_healthy
metafeederusenet-filename-parser:
condition: service_started
metafeederusenet-tmdb:
@@ -420,68 +147,40 @@ x-casaos:
description:
en_us: |
**A standalone, self-hosted Usenet feeder for the MetaMesh network.**
**Publish your own Usenet indexer to the MetaMesh network.**
MetaFeeder · Usenet runs its own Usenet header scan (a bundled
**nntmux** indexer) instead of proxying a third-party Newznab site.
Because it finds article Message-IDs itself at scan time, it mints a
**portable** posting reference — no indexer host baked in, redeemable by
any peer with a plain NNTP provider account. It adds a Usenet source
MetaFeeder · Usenet searches your **NNTmux** app (worph/AppStore) over
its Newznab API and turns each release into a **portable** MetaMesh
posting — no indexer host baked in, redeemable by any peer with a plain
NNTP provider account — with a clean title, poster and plot from the
bundled filename-parser + TMDB enrichment. It adds a Usenet source
alongside **MetaFeeder · Prowlarr**; it doesn't replace it.
**Self-contained:** ships its own nntmux scanner (MariaDB + Redis +
Manticore) + filename-parser + TMDB enrichment, so each release gets a
clean title, poster and plot.
It also **redeems** releases found through Prowlarr's Newznab indexers,
with the indexer keys you set on its config page.
**You still need a Usenet provider account** (NNTP login) — this app
scans headers, it does not supply an account, and nothing at all is
indexed until you supply one. Configure it after install (see the
install tips). Keep the connection count low: this scanner and your
playback share the same provider connection slots.
**The catalog is not instant.** Once configured, a bundled scanner
container runs the header-scan / backfill / collation loop on the
interval you set. A first pass over a few busy binary groups pulls
tens of GB and runs for hours; a deep backfill is hundreds of GB. The
install tips show which knobs bound that.
**Install NNTmux first.** No web UI of its own: configure it from the
**MetaGateway** dashboard (nntmux URL + the `metamesh` API key).
**Also requires:** **MetaGateway** — its meta-core stores the resolved
records and announces them to the swarm. The gateway must list this
feeder in its config.
records and announces them to the swarm.
fr_fr: |
**Un feeder Usenet autonome et auto-hébergé pour le réseau MetaMesh.**
**Publiez votre propre indexeur Usenet sur le réseau MetaMesh.**
MetaFeeder · Usenet effectue lui-même le scan des en-têtes Usenet (un
indexeur **nntmux** embarqué) plutôt que de passer par un site Newznab
tiers. Comme il retrouve les Message-ID des articles dès le scan, il
génère une référence de posting **portable** — sans hôte d'indexeur
intégré, exploitable par n'importe quel pair disposant d'un simple
compte NNTP. Il ajoute une source Usenet en plus de **MetaFeeder -
Indexer**, sans le remplacer.
MetaFeeder · Usenet interroge votre app **NNTmux** (worph/AppStore) via
son API Newznab et transforme chaque release en posting MetaMesh
**portable**, exploitable par n'importe quel pair disposant d'un simple
compte NNTP. Il ajoute une source Usenet en plus de **MetaFeeder ·
Prowlarr**.
**Autonome :** embarque son scanner nntmux (MariaDB + Redis + Manticore)
+ ses plugins filename-parser + TMDB.
**Un compte fournisseur Usenet (NNTP) reste nécessaire** — cette app
scanne les en-têtes, elle ne fournit pas d'accès, et rien n'est indexé
tant que vous n'en fournissez pas un. À configurer après installation
(voir les conseils d'installation). Gardez un nombre de connexions
faible : le scan et la lecture partagent les mêmes créneaux du
fournisseur.
**Le catalogue ne se remplit pas instantanément.** Une fois configuré,
un conteneur de scan embarqué exécute la boucle scan / backfill /
collation à l'intervalle que vous choisissez. Un premier passage sur
quelques gros groupes binaires télécharge des dizaines de Go et dure
des heures ; un backfill profond, des centaines de Go.
**Nécessite aussi :** l'app **MetaGateway** (son meta-core stocke et
annonce les enregistrements).
**Installez NNTmux d'abord.** Pas d'interface propre : configuration
depuis le tableau de bord **MetaGateway** (URL nntmux + clé API
`metamesh`). **Nécessite aussi** l'app **MetaGateway**.
tagline:
en_us: Self-hosted Usenet header scanner (bundled nntmux) for the MetaMesh network
fr_fr: Scanner Usenet auto-hébergé (nntmux inclus) pour le réseau MetaMesh
en_us: Publishes your NNTmux Usenet indexer to MetaMesh as portable postings
fr_fr: Publie votre indexeur Usenet NNTmux sur MetaMesh
title:
en_us: MetaFeeder · Usenet
@@ -499,342 +198,26 @@ x-casaos:
tips:
before_install:
en_us: |
**Install order:** install **MetaGateway** first (this feeder
publishes to its shared meta-core).
**Install order:** **NNTmux** (worph/AppStore) and **MetaGateway** first.
**After installing this app, a Usenet provider account is required**
(this scans headers itself — it needs a plain NNTP login):
1. Edit `/DATA/AppData/metafeederusenet/nntmux-env/nntmux.env` (via
SSH or a file-manager app) and fill in `NNTP_SERVER`, `NNTP_PORT`,
`NNTP_USERNAME`, `NNTP_PASSWORD` (use the SAME provider account
your MetaShare/meta-share peer is already credentialed against —
one account, two consumers). Keep `NNTP_CONNECTIONS` low (1-2):
the scanner and playback share the same provider connection slots,
so a greedy value here stalls playback mid-stream elsewhere.
2. Restart the nntmux container: `docker restart metafeederusenet-nntmux`.
3. **First boot only:** nntmux's install sequence caches its config
before generating an encryption key, so every page 500s until you
run `docker exec metafeederusenet-nntmux php artisan config:clear`
once. (`curl -sI https://metafeederusenetnntmux-<your-domain>/` — a
`500` means you need this; a `302` redirect to `/login` means
you're fine.)
4. Log in to the nntmux dashboard at the **MetaFeeder Usenet nntmux**
link (`ADMIN_USER=admin`, password = this app's auto-generated
default password — check CasaOS's app credentials panel) and
change it.
**⚠ Do steps 5-6 AFTER your LAST restart, and re-check them after
every later restart or stack update.** The nntmux entrypoint RE-SEEDS
the `settings` and `usenet_groups` tables on EVERY container boot, so
anything you configure before a restart silently reverts (observed:
`running` back to 0, `delaytime` back to 2, every group back to
`active=0`) — with no error anywhere.
5. **Activate at least one usenet group** — seeded groups start
inactive (`active=0`), and scanning an inactive group silently
touches nothing at all:
**After installing:**
1. Read NNTmux's `metamesh` API key (the NNTmux app provisions this user,
with raised daily API/download caps):
```
docker exec metafeederusenet-db mariadb -unntmux -p"<password>" nntmux \
-e "UPDATE usenet_groups SET active=1, backfill=1 WHERE name IN ('alt.binaries.teevee');"
docker exec nntmux php artisan tinker --execute="echo App\Models\User::where('username','metamesh')->value('api_token');"
```
6. **Start the scheduler** in nntmux's admin UI (**/admin/tmux-edit**)
or directly in the `settings` table: set `running=1`, pick your
`monitor_delay` (seconds between passes), and turn on `binaries`
and/or `backfill` with a `backfill_qty` / `backfill_days` you're
happy to pay for. The bundled `metafeederusenet-scanner` container
polls these settings and runs the scan/backfill/collate loop for
you — the nntmux image itself ships NO cron and NO queue worker, so
this container is what actually makes a scan happen. There are
deliberately no env vars for these knobs: the `settings` table is
the single source of truth.
2. In **MetaGateway**, make sure this feeder is listed in its feeders
config (`http://metafeederusenet-feeder:8080`), then open its config
panel (*configure*): paste the key into **nntmux API key** (the URL
defaults to `http://nntmux`) and Save — the feeder restarts itself.
3. Optionally add **indexer keys** there too, to redeem releases found
through Prowlarr's Newznab indexers, and a TMDB token for posters.
**`delaytime` (HOURS, default 2) will stall a backfilled catalog.**
nntmux won't turn a collection into a release until it has aged
past `delaytime`, which is right for tailing live posts and
pointless when backfilling history — and the symptom (collections
climbing, releases flat, zero query results) looks exactly like a
broken pipeline. **While `backfill=1`, the scanner forces
`delaytime=0` for you on every pass** and logs
`asserting delaytime=0 for the backfill phase (was N)` when it does
— it has to re-assert it because a container restart re-seeds the
value back to 2. This is the only setting the loop ever writes.
**Searches must be typed** (`fileType:video`, `contentKind:episode`…) to
reach this feeder — a plain search-box query never fans out to it.
So if you want a different steady-state `delaytime`, **set
`backfill=0` first**, then set your value — otherwise the loop will
keep pulling it back to 0. With `backfill=0` the scanner never
touches it and `1`-`2` is the sensible live-tailing value.
**Budget the first scan.** A first header pass over a few busy
binary groups pulls tens of GB and runs for hours; a
full-retention backfill is hundreds of GB. `backfill_days` is the
knob that bounds it — start small, widen once you see releases.
7. Open its config panel (the **MetaGateway** dashboard → this feeder → *configure*) to optionally set a
TMDB v4 token for posters, then Save.
8. In **MetaGateway**, make sure this feeder is listed in its feeders
config (`http://metafeederusenet-feeder:8080`) so it gets
discovered.
**Note:** which groups to scan, backfill depth and scan interval are
your call as the operator — pick them for your own catalog.
**Watch the loop:** `docker logs -f metafeederusenet-scanner` prints
one line per pass (`header scan` / `backfill direct` / `collate
<group>` / `sleeping`). `settings.running=0` shows up there as
`idle`.
**Authentication:** the feeder's config page opens from the **MetaGateway**
dashboard (behind its login); nntmux uses its own login (seeded above).
pre-install-cmd: |
# Seed nntmux's .env (trap #1 — must be a complete file, see header).
# NNTP_* are deliberately left blank; fill them in per the install tips.
if [ ! -f /DATA/AppData/metafeederusenet/nntmux-env/nntmux.env ]; then
cat > /DATA/AppData/metafeederusenet/nntmux-env/nntmux.env <<NNTMUX_ENV
DB_CONNECTION=mariadb
DB_HOST=metafeederusenet-db
DB_PORT=3306
DB_SOCKET=
DB_USERNAME=nntmux
DB_PASSWORD=$APP_DEFAULT_PASSWORD
DB_DATABASE=nntmux
MANTICORESEARCH_HOST=metafeederusenet-manticore
MANTICORESEARCH_PORT=9308
ELASTICSEARCH_HOST=localhost
ELASTICSEARCH_PORT=9200
ELASTICSEARCH_SCHEME=http
ELASTICSEARCH_USER=
ELASTICSEARCH_PASS=
ELASTICSEARCH_LOGGING=false
ELASTICSEARCH_ENABLED=false
NNTP_COMPRESSED_HEADERS=false
USE_ALTERNATE_NNTP_SERVER=false
NNTP_USERNAME=
NNTP_PASSWORD=
NNTP_SERVER=
NNTP_PORT=563
NNTP_CONNECTIONS=1
NNTP_SSLENABLED=true
NNTP_SOCKET_TIMEOUT=120
NNTP_USERNAME_A=
NNTP_PASSWORD_A=
NNTP_SERVER_A=
NNTP_PORT_A=
NNTP_CONNECTIONS_A=
NNTP_SSLENABLED_A=false
NNTP_SOCKET_TIMEOUT_A=120
# NN_MULTIPROCESSING_MAX_CHILD_TIME is deliberately ABSENT (trap #5).
# Writing it here, even as its own default of 1800, hands
# Pool::timeout(float) a string via Laravel's env(), which returns STRINGS,
# and every Forking multiprocessing runner dies with a TypeError.
# RULE for this whole file: never set a key unless the value must DIFFER
# from the PHP default in config/nntmux.php. Restating a default is not
# documentation, it is a type change.
ADMIN_USER=admin
ADMIN_PASS=$APP_DEFAULT_PASSWORD
ADMIN_EMAIL=admin@$APP_DOMAIN
APP_NAME=NNTmux
APP_ENV=production
APP_DEBUG=false
APP_TIMEZONE=UTC
APP_URL=https://metafeederusenetnntmux-$APP_DOMAIN
APP_LOCALE=en
APP_FALLBACK_LOCALE=en
APP_FAKER_LOCALE=en_US
APP_MAINTENANCE_DRIVER=file
APP_MAINTENANCE_STORE=database
BCRYPT_ROUNDS=12
APP_KEY=
LOG_CHANNEL=stack
LOG_STACK=single
PASSWORD_HASH=argon2id
BROADCAST_CONNECTION=log
CACHE_STORE=file
QUEUE_CONNECTION=sync
REDIS_HOST=metafeederusenet-redis
REDIS_PASSWORD=null
REDIS_PORT=6379
REDIS_CLIENT=phpredis
SESSION_DRIVER=file
SESSION_DOMAIN=null
SESSION_SECURE_COOKIE=false
SESSION_ENCRYPT=false
SESSION_COOKIE=nntmux
SESSION_PATH=/
SCOUT_DRIVER=database
SCOUT_QUEUE=true
ITEMS_PER_PAGE=50
ITEMS_PER_COVER_PAGE=25
MAX_PAGER_RESULTS=125000
ECHOCLI=true
RENAME_PAR2=false
ADD_PAR2=false
RENAME_MUSIC_MEDIAINFO=true
CACHE_EXPIRY_SHORT=5
CACHE_EXPIRY_MEDIUM=10
CACHE_EXPIRY_LONG=15
SSL_CAFILE=
SSL_CAPATH=
SSL_VERIFY_PEER=false
SSL_VERIFY_HOST=false
SSL_ALLOW_SELF_SIGNED=true
SMARTY_DEBUG=false
SMARTY_CACHING=false
SMARTY_CACHE_LIFE=120
SMARTY_COMPILE_CHECK=false
SMARTY_FORCE_COMPILE=false
SMARTY_CACHE_DRIVER=file
TMDB_APIKEY=9a4e16adddcd1e86da19bcaf5ff3c2a3
TMDB_CACHE=false
TMDB_LOG=false
TVDB_APIKEY=ef6fb572-bcee-4d99-9b52-90549ad7553a
TVDB_PIN=
TEMP_UNRAR_PATH=/app/storage/tmp/unrar/
TEMP_UNZIP_PATH=/app/storage/tmp/unzip/
VIEW_COMPILED_PATH=/app/storage/framework/views
PATH_TO_NZBS=/app/storage/nzb/
COVERS_PATH=/app/storage/covers
TELESCOPE_ENABLED=false
PURGE_INACTIVE_USERS=false
OTP_ENABLED=false
CHECK_PASSWORDED_RARS=false
DELETE_PASSWORDED_RELEASES=false
DELETE_POSSIBLE_PASSWORDED_RELEASES=false
EXTRACT_USING_RARINFO=false
PRIVATE_PROFILES=true
STORE_USER_IP=false
FORUM_FRONTEND_ENABLED=false
NNTMUX_ENV
fi
# Seed the scan loop run by metafeederusenet-scanner (trap #6). Unlike
# nntmux.env above this is NOT guarded by `[ ! -f ]` and is NOT operator
# config: it is shipped code, so a store update must be able to replace a
# stale copy. Every tunable it reads lives in nntmux's `settings` table.
# Quoted heredoc delimiter ('SCANNER_SH') — the script contains $, \ and
# nested quotes that must reach the file verbatim, unexpanded.
cat > /DATA/AppData/metafeederusenet/scanner/scanner.sh <<'SCANNER_SH'
#!/bin/sh
# nntmux scan launcher.
#
# The pyrowman/nntmux image runs frankenphp and NOTHING else: no cron, no
# supervisord, no queue worker. So `settings.running=1` changes nothing on
# its own, because nothing ever starts the loop. This is that loop.
#
# It runs as a SECOND CONTAINER off the same image + the same /app/.env and
# nzb volume, rather than as a second process inside the web container, so
# the web container's entrypoint (install/populate under `set -e`) is left
# untouched.
#
# INVOCATION IS NOT FREE CHOICE. Measured PER RUNNER on a live box, NOT a
# blanket "Forking works":
# * multiprocessing/binaries.php — Forking OK.
# * multiprocessing/backfill.php — NO-OP. Reports "...3 job(s) to do
# using a max of 0 child process(es)" and returns in 0s having done
# nothing. Backfill must be driven DIRECTLY, per group (see the
# backfill block below).
# * collation MUST go through tinker/ProcessReleases DIRECTLY.
# multiprocessing/releases.php aborts every batch on a benign PHP
# warning (ReleaseCleaning.php:165) that turns fatal inside the Spatie
# child -> 0 releases while collections pile into the tens of
# thousands.
#
# Every knob comes from nntmux's own `settings` table (edited at
# /admin/tmux-edit), never from env — otherwise there are two sources of
# truth for one number.
set -u
cd /app || exit 1
setting() { php artisan tinker --execute="echo DB::table('settings')->where('name','$1')->value('value');" 2>/dev/null | tr -dc '0-9'; }
set_setting() { php artisan tinker --execute="DB::table('settings')->where('name','$1')->update(['value'=>'$2']);" >/dev/null 2>&1; }
groups() { php artisan tinker --execute="foreach (DB::table('usenet_groups')->where('active',1)->pluck('name') as \$g) echo \$g, PHP_EOL;" 2>/dev/null | grep -E '^[a-z0-9.]+$'; }
log() { echo "[scanner $(date -u +%H:%M:%S)] $*"; }
log "launcher started"
while true; do
if [ "$(setting running)" != "1" ]; then
log "settings.running=0 — idle"
sleep 30
continue
fi
if [ "$(setting binaries)" = "1" ]; then
qty="$(setting backfill_qty)"; [ -n "$qty" ] || qty=20000
log "header scan (qty=$qty)"
php misc/update/multiprocessing/binaries.php "$qty" 2>&1 | tail -3
fi
# Backfill must run DIRECT, per group. multiprocessing/backfill.php
# reports "max of 0 child process(es)" and silently does nothing — the
# Forking wrapper leaves maxProcesses=0 for this runner specifically.
# Depth is capped by settings.backfill_days, so this reaches back that
# far and then stops advancing; it is NOT a full-retention backfill.
if [ "$(setting backfill)" = "1" ]; then
# The ONLY setting this loop writes. `delaytime` (hours, default 2)
# blocks ProcessReleases from advancing a collection until it ages
# past it. Correct for the leading edge — a posting may still be
# uploading — but pointless for backfill, where every article is
# already days old. Left at 2 during backfill it stalls the catalog
# for two hours with a failure mode indistinguishable from a broken
# pipeline (collections climb, releases flatline, nothing logs an
# error).
#
# Asserted every pass, not once at boot, because the nntmux
# entrypoint RE-SEEDS `settings` on every container start (trap #7)
# — a restart silently puts this back to 2 and the tier quietly
# stops producing. Scoped to `backfill=1` so a pure forward-tailing
# deployment (`backfill=0`) keeps the operator's own value: this is
# a per-phase mode, not a preference we get to override.
if [ "$(setting delaytime)" != "0" ]; then
log "asserting delaytime=0 for the backfill phase (was $(setting delaytime))"
set_setting delaytime 0
fi
qty="$(setting backfill_qty)"; [ -n "$qty" ] || qty=20000
log "backfill direct (backfill_days=$(setting backfill_days), qty=$qty)"
groups | while read -r g; do
[ -n "$g" ] || continue
php misc/update/backfill.php "$g" "$qty" 2>&1 | grep -iE "articles|backfill|Group|no new" | tail -2
done
fi
# Collation, per active group, DIRECT (never multiprocessing/releases.php).
# Never wrap this in a short `timeout`: killing it between the release
# INSERT and createNZBs() leaves a release row with no .nzb.gz, which
# the feeder then correctly skips forever.
groups | while read -r g; do
[ -n "$g" ] || continue
log "collate $g"
php artisan tinker --execute="(new Blacklight\\processing\\ProcessReleases)->processReleases(1, 1, \"$g\", new Blacklight\\NNTP);" 2>&1 | grep -iE "releases (ready|created)|Added [0-9]+ releases|NZB" | tail -3
done
delay="$(setting monitor_delay)"; [ -n "$delay" ] || delay=30
log "pass complete; sleeping ${delay}s"
sleep "$delay"
done
SCANNER_SH
chmod 755 /DATA/AppData/metafeederusenet/scanner/scanner.sh
Each release's .nzb is downloaded from NNTmux once and cached here
(`feeder-state/gateway/usenet/nntmux-nzb/`).
# Maison (Yundera dashboard) configuration. Read in preference to x-casaos.
# schema_version 2 is required for `folders`: an older Maison refuses the app
@@ -845,26 +228,10 @@ x-compose-app:
# runs and before the containers start — on the first boot and every one after.
# Compose would otherwise create a missing bind source as a root-owned dir.
folders:
- path: /DATA/AppData/metafeederusenet/db
user: $PUID
group: $PGID
- path: /DATA/AppData/metafeederusenet/nntmux-env
user: $PUID
group: $PGID
- path: /DATA/AppData/metafeederusenet/nntmux-storage/nzb
user: $PUID
group: $PGID
- path: /DATA/AppData/metafeederusenet/feeder-state
user: $PUID
group: $PGID
- path: /DATA/AppData/metafeederusenet/tmdb-cache
user: $PUID
group: $PGID
- path: /DATA/AppData/metafeederusenet/scanner
user: $PUID
group: $PGID
# Same string as nntmux's caddy_0 label, so the tile link and the route agree.
# The tile opens nntmux's own admin UI (its own login, its own Caddy route).
# The feeder itself is configured from the MetaGateway dashboard.
webui-host: metafeederusenetnntmux-${domain}
webui-path: "/"
# No web UI: Maison files it under Services. Configured from MetaGateway.
Apps/MetaFeederUsenet/rationale.md
+14 -43
View File
@@ -3,62 +3,33 @@
## What deviation / exception is being requested
1. `metafeederusenet-feeder` runs `user: 0:0`.
2. The **NNTmux web UI is published on its own hostname**
(`metafeederusenetnntmux-<user>.<domain>`) with no AppShield sidecar in front of it — it
is protected by NNTmux's own login only.
3. The seeded `nntmux.env` contains TMDB and TVDB API keys as literals.
(The NNTmux indexer this feeder used to bundle — and its own exceptions: a web UI on its own
login, seeded TMDB/TVDB keys — is now the separate **NNTmux** app in worph/AppStore, with its
own `rationale.md`.)
## Why it is necessary
1. The feeder publishes resolved records to the shared MetaCore and writes plugin artwork
through its WebDAV endpoint; those targets are created and owned by MetaCore's
root-running container.
2. NNTmux is a Laravel application with its own user table, session handling and admin
role. Its admin account is created at install from `$APP_DEFAULT_PASSWORD`, so
authentication is **enabled by default** — the "app's own built-in auth" alternative the
security checklist accepts. Fronting it with AppShield as well would double-prompt on
every page and breaks its own OTP and password-reset flows.
3. Those two keys are NNTmux's upstream project defaults, shipped in its own
`.env.example`. They are shared read-only application keys for public metadata catalogs,
not user credentials, and they are what makes the app work immediately after
installation with no manual configuration. An operator can replace them from the NNTmux
admin UI.
## Security mitigations in place
- The NNTmux admin password is `$APP_DEFAULT_PASSWORD` — the platform-generated secret —
and the database password comes from the same variable. Neither is a hardcoded literal,
and both are surfaced to the user in `tips.before_install`.
- The feeder's own configuration UI *is* behind AppShield + Authelia SSO
(`ghcr.io/yundera/appshield:2.0.9`).
- `metafeederusenet-db`, `-redis` and `-manticore` are on the app-private
`metafeederusenet` bridge network, carry no Caddy labels and publish no host port, so no
sibling app on the shared `pcs` network can resolve or dial them.
- NNTP provider credentials are deliberately left **blank** in the seed; the user fills
them in per `tips.before_install`. Nothing ships with a working news account.
- The `nntmux.env` seed is guarded by `[ ! -f … ]`, so a reinstall or version upgrade never
overwrites the operator's provider settings. `scanner.sh` is intentionally rewritten on
every run — it is shipped code, not operator config, so a store update must be able to
replace a stale copy; the write is idempotent.
- No user directories are mounted. All binds are under `/DATA/AppData/metafeederusenet/`,
declared in `x-compose-app.folders`.
- `cpu_shares` on all nine services; every image pinned (the NNTmux and Manticore images by
digest).
- No web UI and no Caddy route: the feeder's configuration page is reached only through the
MetaGateway dashboard's proxy, behind the gateway's login.
- It talks to NNTmux only over NNTmux's Newznab API, with the key of a dedicated
`metamesh` user — it holds no database credential and mounts no NNTmux storage.
- The API key and the indexer keys are secrets on the feeder's config page (write-only,
never echoed back); none has an env seed or a default.
- `cpu_shares` and memory caps on all three services; every image pinned.
## Alternatives considered and rejected
- **AppShield in front of NNTmux.** Two independent login gates on one UI; its OTP,
password-reset and API endpoints break behind the OIDC redirect.
- **Blank the TMDB/TVDB keys.** The app then installs into a state where covers and
metadata silently fail until the user finds two upstream registration flows — a direct
violation of "works immediately after installation, no manual configuration required for
basic functionality".
- **Run the feeder as `$PUID:$PGID`.** Writes into MetaCore's root-owned tree fail.
## Data protection
All state — MariaDB, the NZB store, the feeder cursor, the TMDB cache — is under
`/DATA/AppData/metafeederusenet/`, the unit Maison archives on uninstall and restores from
backup, so an uninstall/reinstall with "keep user data" comes back with the release
catalogue intact rather than re-scanning Usenet from scratch. No user media directory is
mounted into any service.
All state — the feeder config, its redb cache, the per-release `.nzb` cache, the TMDB
cache — is under `/DATA/AppData/metafeederusenet/`. The `.nzb` cache is re-derivable from
NNTmux. No user media directory is mounted.