871 lines
40 KiB
YAML
871 lines
40 KiB
YAML
name: metafeederusenet
|
|
|
|
# =============================================================================
|
|
# MetaFeeder · Usenet — a STANDALONE, self-hosted Usenet feeder for MetaMesh.
|
|
#
|
|
# Unlike MetaFeeder · Prowlarr (which proxies an EXTERNAL Newznab/Torznab
|
|
# indexer and mints an `nzb-release` (0x1005) cid embedding that indexer's
|
|
# host), this app runs its OWN header scan (an `nntmux` — newznab-tmux fork —
|
|
# sidecar) so it can mint a PORTABLE `nzb-posting` (0x1003) cid: a digest over
|
|
# the article Message-ID set, embedding no indexer host, redeemable by ANY
|
|
# peer with a plain NNTP provider account. It ADDS a Usenet source; it does
|
|
# not replace the Prowlarr/Torznab one — a gateway query fans out to both.
|
|
#
|
|
# Design record (meta-gateway repo): docs/others/self-hosted-usenet-indexer-study.md
|
|
# Feeder repo: github.com/worph/meta-feeder-usenet
|
|
#
|
|
# It needs, over the shared `pcs` network:
|
|
# - the shared **MetaCore** app (`metacore-app:9000`) — where resolved
|
|
# records + TMDB posters are stored; the shared **MetaShare** app then
|
|
# announces them to the swarm.
|
|
# The **MetaGateway** app must list this feeder's URL
|
|
# (`http://metafeederusenet-feeder:8080`) in its feeders config so it
|
|
# discovers + queries it (MetaGateway, MetaCore and this feeder share `pcs`).
|
|
#
|
|
# Everything is scoped under the `metafeederusenet-*` prefix + the per-stack
|
|
# `metafeederusenet` network + /DATA/AppData/metafeederusenet/ data root, so it
|
|
# installs alongside the other Meta* apps with zero collisions.
|
|
#
|
|
# amd64 ONLY — `pyrowman/nntmux` ships no arm64 build.
|
|
#
|
|
# ⚠ Four silent-failure traps in the `pyrowman/nntmux` image, empirically
|
|
# verified (see the meta-gateway dev compose header for the full writeup):
|
|
# 1. `PATH_TO_NZBS` (and every other storage path var) must be ABSENT from
|
|
# the env, never present-but-blank — that reads as an explicit empty
|
|
# string, not "unset", and nntmux aborts NZB writing with 0 releases and
|
|
# no visible error. This compose bind-mounts a COMPLETE, hand-filled
|
|
# `.env` (below) rather than relying on individual `environment:` keys,
|
|
# for exactly this reason.
|
|
# 2. `DB_CONNECTION` must be `mariadb`, not `mysql` — the install migration
|
|
# loads its schema file by connection name; `mysql` finds no matching
|
|
# schema and fails partway through a from-scratch migration.
|
|
# 3. Manticore is a HARD first-boot dependency (unless
|
|
# `ELASTICSEARCH_ENABLED=true`, which just swaps it for a heavier one) —
|
|
# `metafeederusenet-manticore` below is not optional.
|
|
# 4. Seeded usenet groups default to `active=0, backfill=0` — scanning an
|
|
# inactive group silently does nothing. See the tips below for the SQL
|
|
# to activate one.
|
|
# One FIRST-BOOT-ONLY loud bug: the entrypoint runs `config:cache` before
|
|
# `key:generate`, so every session route 500s until you run
|
|
# `docker exec metafeederusenet-nntmux php artisan config:clear` once. See tips.
|
|
#
|
|
# ⚠ Four more, found the hard way on a live box (2026-07-26) after this stack
|
|
# came up with SEVEN healthy containers and returned ZERO results for every
|
|
# query — the feeder answering `"Done"` and nothing else. None of them logs
|
|
# anything; each one alone is enough to keep the catalog empty forever.
|
|
#
|
|
# 5. NEVER set a key in `nntmux.env` unless the value must DIFFER from the
|
|
# PHP default. `config/nntmux.php` does
|
|
# `env('NN_MULTIPROCESSING_MAX_CHILD_TIME', 1800)` and Laravel's `env()`
|
|
# returns STRINGS, so restating the default in the .env handed
|
|
# `Pool::timeout(float)` a `string("1800")` → TypeError → every Forking
|
|
# multiprocessing runner died on start. Those runners are exactly what a
|
|
# scan needs. This is trap #1's type-flavoured twin: a redundant
|
|
# restatement is not a no-op, it silently changes the value's TYPE. The
|
|
# key is therefore deliberately ABSENT from the seeded .env below (see the
|
|
# comment there); with it absent, `gettype()` reports `integer` and the
|
|
# runners execute. Same rule applies to any other key you are tempted to
|
|
# "document" by writing its default into the file.
|
|
# (Upstream record: meta-gateway `dev/docker-compose.feeder-usenet.yml`,
|
|
# the "CORRECTION (2026-07-26)" block.)
|
|
# 6. The image runs `frankenphp` and NOTHING ELSE — no cron, no supervisord,
|
|
# no queue worker. nntmux's own `settings.running=1` therefore starts
|
|
# nothing: there is no process to read it. Even with valid NNTP creds and
|
|
# active groups the catalog stays empty permanently. That is why
|
|
# `metafeederusenet-scanner` below exists — it is the missing loop, run as
|
|
# a SECOND container off the same image so the web container's
|
|
# install/populate entrypoint stays untouched.
|
|
# 7. The nntmux entrypoint RE-SEEDS `settings` and `usenet_groups` on EVERY
|
|
# container boot. Configuration applied before a restart silently reverts
|
|
# (observed: `running`→0, `delaytime`→2, every group→`active=0`). Apply
|
|
# group activation and scheduler settings AFTER the final restart, and
|
|
# re-check them after any `docker restart` / stack update.
|
|
# 8. `settings.delaytime` (HOURS, default 2) stalls a backfilled catalog:
|
|
# `ProcessReleases` refuses to advance a collection until it has aged past
|
|
# it. That is correct for live tailing (a posting may still be uploading)
|
|
# and pointless for backfill, where every article is already days old —
|
|
# and the symptom, collections climbing while releases flatline, is
|
|
# indistinguishable from a broken pipeline because nothing logs an error.
|
|
# Because trap #7 re-seeds it back to 2 on every restart, the scanner
|
|
# loop below ASSERTS `delaytime=0` on each pass **while `backfill=1`** —
|
|
# the one setting it writes, making the tier self-healing across
|
|
# restarts. It is scoped to the backfill phase on purpose: with
|
|
# `backfill=0` (pure forward tailing) the operator's own value stands.
|
|
# =============================================================================
|
|
|
|
services:
|
|
# ---------------------------------------------------------------------------
|
|
# metafeederusenet-db — MariaDB backing nntmux's OWN release catalog. Not
|
|
# shared with the gateway's meta-core Redis or anything else in the mesh.
|
|
# ---------------------------------------------------------------------------
|
|
metafeederusenet-db:
|
|
image: mariadb:12.3
|
|
cpu_shares: 70
|
|
container_name: metafeederusenet-db
|
|
hostname: metafeederusenet-db
|
|
restart: unless-stopped
|
|
# 2g, not the 512m this shipped with: 512m is fine for an empty schema and
|
|
# far too small once the catalog is real. A header scan of a busy binary
|
|
# group builds `collections`/`binaries`/`parts` tables in the tens of
|
|
# millions of rows, and InnoDB starved of buffer pool turns collation into
|
|
# a disk-thrash that never catches up. Measured on a live box.
|
|
mem_limit: 2g
|
|
memswap_limit: 2g
|
|
environment:
|
|
MARIADB_DATABASE: nntmux
|
|
MARIADB_USER: nntmux
|
|
MARIADB_PASSWORD: "$APP_DEFAULT_PASSWORD"
|
|
MARIADB_RANDOM_ROOT_PASSWORD: "yes"
|
|
volumes:
|
|
- ${DATA_ROOT:-/DATA}/AppData/metafeederusenet/db:/var/lib/mysql
|
|
healthcheck:
|
|
test: ["CMD", "healthcheck.sh", "--connect", "--innodb_initialized"]
|
|
interval: 5s
|
|
timeout: 5s
|
|
retries: 20
|
|
start_period: 20s
|
|
networks:
|
|
- metafeederusenet
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# metafeederusenet-redis — nntmux's OWN Redis. Deliberately separate from the
|
|
# gateway core's Redis (different failure domain; losing this one is free).
|
|
# ---------------------------------------------------------------------------
|
|
metafeederusenet-redis:
|
|
image: redis:7-alpine
|
|
cpu_shares: 70
|
|
container_name: metafeederusenet-redis
|
|
hostname: metafeederusenet-redis
|
|
restart: unless-stopped
|
|
mem_limit: 128m
|
|
memswap_limit: 128m
|
|
networks:
|
|
- metafeederusenet
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# metafeederusenet-manticore — RT-index search backend. NOT optional (trap
|
|
# #3 above) — a hard first-boot dependency of nntmux's own entrypoint, even
|
|
# though this feeder reads `releases` via SQL directly and never touches it.
|
|
# Digest-pinned (upstream reuses low-looking version numbers across an old
|
|
# semver line and a newer CalVer-ish one; the digest is unambiguous).
|
|
# ---------------------------------------------------------------------------
|
|
metafeederusenet-manticore:
|
|
image: manticoresearch/manticore@sha256:032388dc77321e3ac898a14b9a0c3fc22ab8d08cd2582abd6c5b642d46b82181
|
|
cpu_shares: 50
|
|
container_name: metafeederusenet-manticore
|
|
hostname: metafeederusenet-manticore
|
|
restart: unless-stopped
|
|
mem_limit: 512m
|
|
memswap_limit: 512m
|
|
ulimits:
|
|
nofile: { soft: 65535, hard: 65535 }
|
|
networks:
|
|
- metafeederusenet
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# metafeederusenet-nntmux — the header scanner + web UI/API. Exposed DIRECTLY
|
|
# via Caddy (no AppShield gate): nntmux ships its own admin login
|
|
# (ADMIN_USER/ADMIN_PASS, seeded below), the same convention this store uses
|
|
# for every app that brings its own auth (Jellyfin, Nextcloud, Immich, ...).
|
|
# ---------------------------------------------------------------------------
|
|
metafeederusenet-nntmux:
|
|
# Digest-pinned, NOT :latest — a silent tag move could change the
|
|
# `misc/update/*.php` command surface the "collate via tinker" tip below
|
|
# depends on. Re-pin deliberately, re-verify the trap list above when you do.
|
|
image: pyrowman/nntmux@sha256:7fc221f7a806fee3e4a2145159e1a147962edc095282bdc692806968f791fabe
|
|
cpu_shares: 50
|
|
container_name: metafeederusenet-nntmux
|
|
hostname: metafeederusenet-nntmux
|
|
restart: unless-stopped
|
|
# The working set scales with scan breadth (group set x backfill depth), an
|
|
# operator choice (see tips), so this is a ceiling rather than a size: 3g
|
|
# is what a real corpus needed on a live box — 1536m was reached and
|
|
# exceeded by a single busy group's header scan. Raise it if you scan wide.
|
|
mem_limit: 3g
|
|
memswap_limit: 3g
|
|
environment:
|
|
# Shell-script-consumed vars — the entrypoint's own `mkdir -p` step reads
|
|
# these from the PROCESS environment, not the mounted .env. Keep in
|
|
# lockstep with the matching keys in nntmux.env below.
|
|
PATH_TO_NZBS: /app/storage/nzb/
|
|
COVERS_PATH: /app/storage/covers
|
|
TEMP_UNRAR_PATH: /app/storage/tmp/unrar/
|
|
TEMP_UNZIP_PATH: /app/storage/tmp/unzip/
|
|
ELASTICSEARCH_ENABLED: "false"
|
|
volumes:
|
|
# The complete, hand-filled .env (trap #1) — a full-file mount rather
|
|
# than individual `environment:` keys. Writable: `nntmux:install`
|
|
# persists a generated APP_KEY back into it on first boot.
|
|
- ${DATA_ROOT:-/DATA}/AppData/metafeederusenet/nntmux-env/nntmux.env:/app/.env:rw
|
|
# The .nzb store — shared with metafeederusenet-feeder below,
|
|
# read-write here (nntmux writes), read-only there (feeder only reads).
|
|
- ${DATA_ROOT:-/DATA}/AppData/metafeederusenet/nntmux-storage/nzb:/app/storage/nzb:rw
|
|
healthcheck:
|
|
test: ["CMD", "curl", "-sf", "-o", "/dev/null", "http://localhost/"]
|
|
interval: 10s
|
|
timeout: 5s
|
|
retries: 15
|
|
start_period: 30s
|
|
depends_on:
|
|
metafeederusenet-db:
|
|
condition: service_healthy
|
|
metafeederusenet-redis:
|
|
condition: service_started
|
|
metafeederusenet-manticore:
|
|
condition: service_started
|
|
labels:
|
|
caddy_0: metafeederusenetnntmux-${APP_DOMAIN}
|
|
caddy_0.import: gateway_tls
|
|
caddy_0.reverse_proxy: "{{upstreams 80}}"
|
|
caddy_1: metafeederusenetnntmux-${APP_PUBLIC_IP_DASH}.nip.io
|
|
caddy_1.import: gateway_tls
|
|
caddy_1.reverse_proxy: "{{upstreams 80}}"
|
|
caddy_2: metafeederusenetnntmux-${APP_PUBLIC_IP_DASH}.sslip.io
|
|
caddy_2.reverse_proxy: "{{upstreams 80}}"
|
|
networks:
|
|
- metafeederusenet
|
|
- pcs
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# metafeederusenet-scanner — the SCAN LOOP (trap #6). The nntmux image runs
|
|
# frankenphp and nothing else, so without this container nothing ever calls
|
|
# the scan/backfill/collate entry points and the catalog stays empty forever,
|
|
# no matter how correct the credentials, the groups or `settings.running` are.
|
|
#
|
|
# It is a second container off the SAME digest-pinned image, sharing the same
|
|
# `/app/.env` and the same nzb store, rather than a second process inside the
|
|
# web container — the web container's entrypoint runs install/populate under
|
|
# `set -e` and must keep it; here it is REPLACED by `entrypoint:` below so
|
|
# that install/populate does NOT run twice against one database.
|
|
#
|
|
# Every knob (interval, binaries on/off, backfill on/off, backfill_qty,
|
|
# backfill_days, running) is read from nntmux's own `settings` table at
|
|
# /admin/tmux-edit — deliberately NOT from new env vars here, which would
|
|
# make two sources of truth for one number.
|
|
# ---------------------------------------------------------------------------
|
|
metafeederusenet-scanner:
|
|
# MUST stay byte-identical to metafeederusenet-nntmux's pin above: the two
|
|
# containers share one `/app/.env` and one database schema, and the script
|
|
# below calls `misc/update/*.php` + `Blacklight\` classes by name.
|
|
image: pyrowman/nntmux@sha256:7fc221f7a806fee3e4a2145159e1a147962edc095282bdc692806968f791fabe
|
|
cpu_shares: 20
|
|
container_name: metafeederusenet-scanner
|
|
hostname: metafeederusenet-scanner
|
|
restart: unless-stopped
|
|
# Bypasses the image's own install/populate entrypoint on purpose — see the
|
|
# block comment above. Only the web container may run it.
|
|
entrypoint: ["/bin/sh", "/scanner.sh"]
|
|
# The image ships a HEALTHCHECK that curls its own web server. This
|
|
# container deliberately runs no web server — its entrypoint IS the scan
|
|
# loop — so the inherited probe can only ever fail. Observed live: a
|
|
# scanner actively backfilling and writing NZBs sitting at
|
|
# `Up 5 minutes (unhealthy)`.
|
|
#
|
|
# That is not cosmetic. A permanently-red container that is in fact working
|
|
# trains the operator to ignore health status, which is precisely how this
|
|
# stack's empty-catalog failure stayed invisible for so long — the feeder's
|
|
# own health() reporting `ok` over a zero-row catalog was the same bug from
|
|
# the other direction. Liveness for this container is its LOG
|
|
# (`docker logs -f metafeederusenet-scanner`), not a probe; a probe that
|
|
# cannot be honest is worse than none.
|
|
healthcheck:
|
|
disable: true
|
|
# A backfill pass holds a group's article range in memory; 2g is the
|
|
# headroom measured alongside nntmux's own 3g on a live box.
|
|
mem_limit: 2g
|
|
memswap_limit: 2g
|
|
environment:
|
|
# Same shell-script-consumed vars as metafeederusenet-nntmux: the wrapper
|
|
# scripts read these from the PROCESS environment, not from the mounted
|
|
# .env (trap #1). Keep the two blocks in lockstep.
|
|
PATH_TO_NZBS: /app/storage/nzb/
|
|
COVERS_PATH: /app/storage/covers
|
|
TEMP_UNRAR_PATH: /app/storage/tmp/unrar/
|
|
TEMP_UNZIP_PATH: /app/storage/tmp/unzip/
|
|
ELASTICSEARCH_ENABLED: "false"
|
|
volumes:
|
|
# The same .env the web container filled in — one credential set, one
|
|
# APP_KEY. Read-write only because Laravel's .env handling expects it.
|
|
- ${DATA_ROOT:-/DATA}/AppData/metafeederusenet/nntmux-env/nntmux.env:/app/.env:rw
|
|
# nntmux writes .nzb.gz here; the feeder reads them read-only.
|
|
- ${DATA_ROOT:-/DATA}/AppData/metafeederusenet/nntmux-storage/nzb:/app/storage/nzb:rw
|
|
# The loop itself, provisioned by pre-install-cmd below.
|
|
- ${DATA_ROOT:-/DATA}/AppData/metafeederusenet/scanner/scanner.sh:/scanner.sh:ro
|
|
depends_on:
|
|
# Not just `started`: the loop's very first action is an artisan call, so
|
|
# it must not race the web container's install/migrate step.
|
|
metafeederusenet-nntmux:
|
|
condition: service_healthy
|
|
networks:
|
|
- metafeederusenet
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# metafeederusenet-feeder — the usenet feeder sidecar (meta-feeder-usenet).
|
|
# Reads nntmux's release catalog + shared .nzb store; self-publishes
|
|
# resolved records + seeds the `.nzb` manifest to the shared MetaCore. The
|
|
# gateway reaches it directly on `pcs`; the human configures it from
|
|
# the MetaGateway dashboard (configure panel).
|
|
# ---------------------------------------------------------------------------
|
|
metafeederusenet-feeder:
|
|
image: ghcr.io/worph/meta-feeder-usenet:1.0.6
|
|
cpu_shares: 30
|
|
container_name: metafeederusenet-feeder
|
|
# Hard RAM cap, swap disabled (memswap_limit == mem_limit) → OOM instead
|
|
# of swap. Mirrors the dev sub-stack limits.
|
|
mem_limit: 256m
|
|
memswap_limit: 256m
|
|
hostname: metafeederusenet-feeder
|
|
restart: unless-stopped
|
|
user: "0:0"
|
|
expose:
|
|
- 8080
|
|
environment:
|
|
# INFRA-ONLY env (the "config: no env if in webui" rule). Operator
|
|
# config beyond the two DB/path seeds below — TMDB token / language,
|
|
# meta-core URL — is read from the persisted config.json (this
|
|
# feeder's config page, opened from the MetaGateway dashboard (configure panel)). These env vars are
|
|
# wiring only: the enrichment sidecars, the enrich callback, the peer id.
|
|
RUST_LOG: "info,meta_feeder=info"
|
|
# The peer id stamped into provenance.gateway_peer on every record this
|
|
# feeder self-publishes — names the GATEWAY this feeder serves, not the
|
|
# feeder itself (analytics group by it). Without it, falls back to HOSTNAME.
|
|
META_GATEWAY_PEER_ID: "metagateway"
|
|
FILENAME_PARSER_URL: "http://metafeederusenet-filename-parser:8080"
|
|
TMDB_PLUGIN_URL: "http://metafeederusenet-tmdb:8080"
|
|
META_FEEDER_CALLBACK_URL: "http://metafeederusenet-feeder:8080/enrich/callback"
|
|
# First-boot SEED only (the persisted config.json wins on every later
|
|
# restart — change these on the /config page after first boot, not here).
|
|
NNTMUX_DB_URL: "mysql://nntmux:${APP_DEFAULT_PASSWORD}@metafeederusenet-db:3306/nntmux"
|
|
NNTMUX_NZB_PATH: "/nntmux-nzb"
|
|
TZ: "$TZ"
|
|
volumes:
|
|
# Persist feeder config (config.json saved via the config UI) + redb cache.
|
|
- ${DATA_ROOT:-/DATA}/AppData/metafeederusenet/feeder-state:/data/meta-feeder:rw
|
|
# Same volume nntmux writes .nzb.gz into — read-only here.
|
|
- ${DATA_ROOT:-/DATA}/AppData/metafeederusenet/nntmux-storage/nzb:/nntmux-nzb:ro
|
|
healthcheck:
|
|
test: ["CMD", "curl", "-sf", "http://localhost:8080/health"]
|
|
interval: 5s
|
|
timeout: 3s
|
|
retries: 10
|
|
start_period: 5s
|
|
depends_on:
|
|
metafeederusenet-nntmux:
|
|
condition: service_healthy
|
|
metafeederusenet-filename-parser:
|
|
condition: service_started
|
|
metafeederusenet-tmdb:
|
|
condition: service_started
|
|
networks:
|
|
- metafeederusenet
|
|
- pcs
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Bundled enrichment plugins (self-contained). filename-parser per file; tmdb
|
|
# once per release — tmdb writes posters/backdrops to the shared meta-core's
|
|
# WebDAV (over pcs) so the ingester seeds them. No healthcheck: these plugin
|
|
# images don't ship one (curl absent).
|
|
# ---------------------------------------------------------------------------
|
|
metafeederusenet-filename-parser:
|
|
image: ghcr.io/worph/metamesh-plugin-filename-parser:1.0.3
|
|
cpu_shares: 30
|
|
container_name: metafeederusenet-filename-parser
|
|
mem_limit: 256m
|
|
memswap_limit: 256m
|
|
hostname: metafeederusenet-filename-parser
|
|
restart: unless-stopped
|
|
expose:
|
|
- 8080
|
|
environment:
|
|
EMIT_TITLE_TAGS: "true"
|
|
networks:
|
|
- metafeederusenet
|
|
|
|
metafeederusenet-tmdb:
|
|
image: ghcr.io/worph/metamesh-plugin-tmdb:1.0.3
|
|
cpu_shares: 30
|
|
container_name: metafeederusenet-tmdb
|
|
mem_limit: 384m
|
|
memswap_limit: 384m
|
|
hostname: metafeederusenet-tmdb
|
|
restart: unless-stopped
|
|
expose:
|
|
- 8080
|
|
environment:
|
|
# Posters land in the shared meta-core (over pcs) so the ingester seeds
|
|
# them. LOAD-BEARING at metamesh-plugin-tmdb:1.0.0 — do NOT remove (see
|
|
# the sibling feeder apps' identical note: the published image needs
|
|
# this, unlike the source build the dev stack runs).
|
|
WEBDAV_URL: "http://metacore-app:9000/webdav"
|
|
volumes:
|
|
- ${DATA_ROOT:-/DATA}/AppData/metafeederusenet/tmdb-cache:/cache
|
|
networks:
|
|
- metafeederusenet
|
|
- pcs
|
|
networks:
|
|
metafeederusenet:
|
|
name: metafeederusenet
|
|
driver: bridge
|
|
pcs:
|
|
name: ${APP_NET:-pcs}
|
|
external: true
|
|
|
|
x-casaos:
|
|
architectures:
|
|
- amd64
|
|
|
|
author: Worph Team
|
|
category: Media
|
|
developer: Aptero Team
|
|
|
|
description:
|
|
en_us: |
|
|
**A standalone, self-hosted Usenet feeder for the MetaMesh network.**
|
|
|
|
MetaFeeder · Usenet runs its own Usenet header scan (a bundled
|
|
**nntmux** indexer) instead of proxying a third-party Newznab site.
|
|
Because it finds article Message-IDs itself at scan time, it mints a
|
|
**portable** posting reference — no indexer host baked in, redeemable by
|
|
any peer with a plain NNTP provider account. It adds a Usenet source
|
|
alongside **MetaFeeder · Prowlarr**; it doesn't replace it.
|
|
|
|
**Self-contained:** ships its own nntmux scanner (MariaDB + Redis +
|
|
Manticore) + filename-parser + TMDB enrichment, so each release gets a
|
|
clean title, poster and plot.
|
|
|
|
**You still need a Usenet provider account** (NNTP login) — this app
|
|
scans headers, it does not supply an account, and nothing at all is
|
|
indexed until you supply one. Configure it after install (see the
|
|
install tips). Keep the connection count low: this scanner and your
|
|
playback share the same provider connection slots.
|
|
|
|
**The catalog is not instant.** Once configured, a bundled scanner
|
|
container runs the header-scan / backfill / collation loop on the
|
|
interval you set. A first pass over a few busy binary groups pulls
|
|
tens of GB and runs for hours; a deep backfill is hundreds of GB. The
|
|
install tips show which knobs bound that.
|
|
|
|
**Also requires:** **MetaGateway** — its meta-core stores the resolved
|
|
records and announces them to the swarm. The gateway must list this
|
|
feeder in its config.
|
|
|
|
fr_fr: |
|
|
**Un feeder Usenet autonome et auto-hébergé pour le réseau MetaMesh.**
|
|
|
|
MetaFeeder · Usenet effectue lui-même le scan des en-têtes Usenet (un
|
|
indexeur **nntmux** embarqué) plutôt que de passer par un site Newznab
|
|
tiers. Comme il retrouve les Message-ID des articles dès le scan, il
|
|
génère une référence de posting **portable** — sans hôte d'indexeur
|
|
intégré, exploitable par n'importe quel pair disposant d'un simple
|
|
compte NNTP. Il ajoute une source Usenet en plus de **MetaFeeder -
|
|
Indexer**, sans le remplacer.
|
|
|
|
**Autonome :** embarque son scanner nntmux (MariaDB + Redis + Manticore)
|
|
+ ses plugins filename-parser + TMDB.
|
|
|
|
**Un compte fournisseur Usenet (NNTP) reste nécessaire** — cette app
|
|
scanne les en-têtes, elle ne fournit pas d'accès, et rien n'est indexé
|
|
tant que vous n'en fournissez pas un. À configurer après installation
|
|
(voir les conseils d'installation). Gardez un nombre de connexions
|
|
faible : le scan et la lecture partagent les mêmes créneaux du
|
|
fournisseur.
|
|
|
|
**Le catalogue ne se remplit pas instantanément.** Une fois configuré,
|
|
un conteneur de scan embarqué exécute la boucle scan / backfill /
|
|
collation à l'intervalle que vous choisissez. Un premier passage sur
|
|
quelques gros groupes binaires télécharge des dizaines de Go et dure
|
|
des heures ; un backfill profond, des centaines de Go.
|
|
|
|
**Nécessite aussi :** l'app **MetaGateway** (son meta-core stocke et
|
|
annonce les enregistrements).
|
|
|
|
tagline:
|
|
en_us: Self-hosted Usenet header scanner (bundled nntmux) for the MetaMesh network
|
|
fr_fr: Scanner Usenet auto-hébergé (nntmux inclus) pour le réseau MetaMesh
|
|
|
|
title:
|
|
en_us: MetaFeeder · Usenet
|
|
|
|
icon: https://cdn.jsdelivr.net/gh/Worph/MetaAppStore@main/Apps/MetaFeederUsenet/icon.png
|
|
screenshot_link:
|
|
- https://cdn.jsdelivr.net/gh/Worph/MetaAppStore@main/Apps/MetaFeederUsenet/screenshot-1.png
|
|
- https://cdn.jsdelivr.net/gh/Worph/MetaAppStore@main/Apps/MetaFeederUsenet/screenshot-2.png
|
|
- https://cdn.jsdelivr.net/gh/Worph/MetaAppStore@main/Apps/MetaFeederUsenet/screenshot-3.png
|
|
thumbnail: https://cdn.jsdelivr.net/gh/Worph/MetaAppStore@main/Apps/MetaFeederUsenet/thumbnail.png
|
|
|
|
store_app_id: metafeederusenet
|
|
is_uncontrolled: false
|
|
|
|
tips:
|
|
before_install:
|
|
en_us: |
|
|
**Install order:** install **MetaGateway** first (this feeder
|
|
publishes to its shared meta-core).
|
|
|
|
**After installing this app, a Usenet provider account is required**
|
|
(this scans headers itself — it needs a plain NNTP login):
|
|
1. Edit `/DATA/AppData/metafeederusenet/nntmux-env/nntmux.env` (via
|
|
SSH or a file-manager app) and fill in `NNTP_SERVER`, `NNTP_PORT`,
|
|
`NNTP_USERNAME`, `NNTP_PASSWORD` (use the SAME provider account
|
|
your MetaShare/meta-share peer is already credentialed against —
|
|
one account, two consumers). Keep `NNTP_CONNECTIONS` low (1-2):
|
|
the scanner and playback share the same provider connection slots,
|
|
so a greedy value here stalls playback mid-stream elsewhere.
|
|
2. Restart the nntmux container: `docker restart metafeederusenet-nntmux`.
|
|
3. **First boot only:** nntmux's install sequence caches its config
|
|
before generating an encryption key, so every page 500s until you
|
|
run `docker exec metafeederusenet-nntmux php artisan config:clear`
|
|
once. (`curl -sI https://metafeederusenetnntmux-<your-domain>/` — a
|
|
`500` means you need this; a `302` redirect to `/login` means
|
|
you're fine.)
|
|
4. Log in to the nntmux dashboard at the **MetaFeeder Usenet nntmux**
|
|
link (`ADMIN_USER=admin`, password = this app's auto-generated
|
|
default password — check CasaOS's app credentials panel) and
|
|
change it.
|
|
|
|
**⚠ Do steps 5-6 AFTER your LAST restart, and re-check them after
|
|
every later restart or stack update.** The nntmux entrypoint RE-SEEDS
|
|
the `settings` and `usenet_groups` tables on EVERY container boot, so
|
|
anything you configure before a restart silently reverts (observed:
|
|
`running` back to 0, `delaytime` back to 2, every group back to
|
|
`active=0`) — with no error anywhere.
|
|
|
|
5. **Activate at least one usenet group** — seeded groups start
|
|
inactive (`active=0`), and scanning an inactive group silently
|
|
touches nothing at all:
|
|
```
|
|
docker exec metafeederusenet-db mariadb -unntmux -p"<password>" nntmux \
|
|
-e "UPDATE usenet_groups SET active=1, backfill=1 WHERE name IN ('alt.binaries.teevee');"
|
|
```
|
|
6. **Start the scheduler** in nntmux's admin UI (**/admin/tmux-edit**)
|
|
or directly in the `settings` table: set `running=1`, pick your
|
|
`monitor_delay` (seconds between passes), and turn on `binaries`
|
|
and/or `backfill` with a `backfill_qty` / `backfill_days` you're
|
|
happy to pay for. The bundled `metafeederusenet-scanner` container
|
|
polls these settings and runs the scan/backfill/collate loop for
|
|
you — the nntmux image itself ships NO cron and NO queue worker, so
|
|
this container is what actually makes a scan happen. There are
|
|
deliberately no env vars for these knobs: the `settings` table is
|
|
the single source of truth.
|
|
|
|
**`delaytime` (HOURS, default 2) will stall a backfilled catalog.**
|
|
nntmux won't turn a collection into a release until it has aged
|
|
past `delaytime`, which is right for tailing live posts and
|
|
pointless when backfilling history — and the symptom (collections
|
|
climbing, releases flat, zero query results) looks exactly like a
|
|
broken pipeline. **While `backfill=1`, the scanner forces
|
|
`delaytime=0` for you on every pass** and logs
|
|
`asserting delaytime=0 for the backfill phase (was N)` when it does
|
|
— it has to re-assert it because a container restart re-seeds the
|
|
value back to 2. This is the only setting the loop ever writes.
|
|
|
|
So if you want a different steady-state `delaytime`, **set
|
|
`backfill=0` first**, then set your value — otherwise the loop will
|
|
keep pulling it back to 0. With `backfill=0` the scanner never
|
|
touches it and `1`-`2` is the sensible live-tailing value.
|
|
|
|
**Budget the first scan.** A first header pass over a few busy
|
|
binary groups pulls tens of GB and runs for hours; a
|
|
full-retention backfill is hundreds of GB. `backfill_days` is the
|
|
knob that bounds it — start small, widen once you see releases.
|
|
|
|
7. Open its config panel (the **MetaGateway** dashboard → this feeder → *configure*) to optionally set a
|
|
TMDB v4 token for posters, then Save.
|
|
8. In **MetaGateway**, make sure this feeder is listed in its feeders
|
|
config (`http://metafeederusenet-feeder:8080`) so it gets
|
|
discovered.
|
|
|
|
**Note:** which groups to scan, backfill depth and scan interval are
|
|
your call as the operator — pick them for your own catalog.
|
|
|
|
**Watch the loop:** `docker logs -f metafeederusenet-scanner` prints
|
|
one line per pass (`header scan` / `backfill direct` / `collate
|
|
<group>` / `sleeping`). `settings.running=0` shows up there as
|
|
`idle`.
|
|
|
|
**Authentication:** the feeder's config page opens from the **MetaGateway**
|
|
dashboard (behind its login); nntmux uses its own login (seeded above).
|
|
|
|
pre-install-cmd: |
|
|
# Seed nntmux's .env (trap #1 — must be a complete file, see header).
|
|
# NNTP_* are deliberately left blank; fill them in per the install tips.
|
|
if [ ! -f /DATA/AppData/metafeederusenet/nntmux-env/nntmux.env ]; then
|
|
cat > /DATA/AppData/metafeederusenet/nntmux-env/nntmux.env <<NNTMUX_ENV
|
|
DB_CONNECTION=mariadb
|
|
DB_HOST=metafeederusenet-db
|
|
DB_PORT=3306
|
|
DB_SOCKET=
|
|
DB_USERNAME=nntmux
|
|
DB_PASSWORD=$APP_DEFAULT_PASSWORD
|
|
DB_DATABASE=nntmux
|
|
|
|
MANTICORESEARCH_HOST=metafeederusenet-manticore
|
|
MANTICORESEARCH_PORT=9308
|
|
|
|
ELASTICSEARCH_HOST=localhost
|
|
ELASTICSEARCH_PORT=9200
|
|
ELASTICSEARCH_SCHEME=http
|
|
ELASTICSEARCH_USER=
|
|
ELASTICSEARCH_PASS=
|
|
ELASTICSEARCH_LOGGING=false
|
|
ELASTICSEARCH_ENABLED=false
|
|
|
|
NNTP_COMPRESSED_HEADERS=false
|
|
USE_ALTERNATE_NNTP_SERVER=false
|
|
|
|
NNTP_USERNAME=
|
|
NNTP_PASSWORD=
|
|
NNTP_SERVER=
|
|
NNTP_PORT=563
|
|
NNTP_CONNECTIONS=1
|
|
NNTP_SSLENABLED=true
|
|
NNTP_SOCKET_TIMEOUT=120
|
|
|
|
NNTP_USERNAME_A=
|
|
NNTP_PASSWORD_A=
|
|
NNTP_SERVER_A=
|
|
NNTP_PORT_A=
|
|
NNTP_CONNECTIONS_A=
|
|
NNTP_SSLENABLED_A=false
|
|
NNTP_SOCKET_TIMEOUT_A=120
|
|
|
|
# NN_MULTIPROCESSING_MAX_CHILD_TIME is deliberately ABSENT (trap #5).
|
|
# Writing it here, even as its own default of 1800, hands
|
|
# Pool::timeout(float) a string via Laravel's env(), which returns STRINGS,
|
|
# and every Forking multiprocessing runner dies with a TypeError.
|
|
# RULE for this whole file: never set a key unless the value must DIFFER
|
|
# from the PHP default in config/nntmux.php. Restating a default is not
|
|
# documentation, it is a type change.
|
|
|
|
ADMIN_USER=admin
|
|
ADMIN_PASS=$APP_DEFAULT_PASSWORD
|
|
ADMIN_EMAIL=admin@$APP_DOMAIN
|
|
|
|
APP_NAME=NNTmux
|
|
APP_ENV=production
|
|
APP_DEBUG=false
|
|
APP_TIMEZONE=UTC
|
|
APP_URL=https://metafeederusenetnntmux-$APP_DOMAIN
|
|
|
|
APP_LOCALE=en
|
|
APP_FALLBACK_LOCALE=en
|
|
APP_FAKER_LOCALE=en_US
|
|
|
|
APP_MAINTENANCE_DRIVER=file
|
|
APP_MAINTENANCE_STORE=database
|
|
|
|
BCRYPT_ROUNDS=12
|
|
APP_KEY=
|
|
LOG_CHANNEL=stack
|
|
LOG_STACK=single
|
|
PASSWORD_HASH=argon2id
|
|
|
|
BROADCAST_CONNECTION=log
|
|
CACHE_STORE=file
|
|
QUEUE_CONNECTION=sync
|
|
|
|
REDIS_HOST=metafeederusenet-redis
|
|
REDIS_PASSWORD=null
|
|
REDIS_PORT=6379
|
|
REDIS_CLIENT=phpredis
|
|
|
|
SESSION_DRIVER=file
|
|
SESSION_DOMAIN=null
|
|
SESSION_SECURE_COOKIE=false
|
|
SESSION_ENCRYPT=false
|
|
SESSION_COOKIE=nntmux
|
|
SESSION_PATH=/
|
|
|
|
SCOUT_DRIVER=database
|
|
SCOUT_QUEUE=true
|
|
|
|
ITEMS_PER_PAGE=50
|
|
ITEMS_PER_COVER_PAGE=25
|
|
MAX_PAGER_RESULTS=125000
|
|
ECHOCLI=true
|
|
RENAME_PAR2=false
|
|
ADD_PAR2=false
|
|
RENAME_MUSIC_MEDIAINFO=true
|
|
CACHE_EXPIRY_SHORT=5
|
|
CACHE_EXPIRY_MEDIUM=10
|
|
CACHE_EXPIRY_LONG=15
|
|
|
|
SSL_CAFILE=
|
|
SSL_CAPATH=
|
|
SSL_VERIFY_PEER=false
|
|
SSL_VERIFY_HOST=false
|
|
SSL_ALLOW_SELF_SIGNED=true
|
|
|
|
SMARTY_DEBUG=false
|
|
SMARTY_CACHING=false
|
|
SMARTY_CACHE_LIFE=120
|
|
SMARTY_COMPILE_CHECK=false
|
|
SMARTY_FORCE_COMPILE=false
|
|
SMARTY_CACHE_DRIVER=file
|
|
|
|
TMDB_APIKEY=9a4e16adddcd1e86da19bcaf5ff3c2a3
|
|
TMDB_CACHE=false
|
|
TMDB_LOG=false
|
|
TVDB_APIKEY=ef6fb572-bcee-4d99-9b52-90549ad7553a
|
|
TVDB_PIN=
|
|
|
|
TEMP_UNRAR_PATH=/app/storage/tmp/unrar/
|
|
TEMP_UNZIP_PATH=/app/storage/tmp/unzip/
|
|
VIEW_COMPILED_PATH=/app/storage/framework/views
|
|
PATH_TO_NZBS=/app/storage/nzb/
|
|
COVERS_PATH=/app/storage/covers
|
|
|
|
TELESCOPE_ENABLED=false
|
|
PURGE_INACTIVE_USERS=false
|
|
OTP_ENABLED=false
|
|
CHECK_PASSWORDED_RARS=false
|
|
DELETE_PASSWORDED_RELEASES=false
|
|
DELETE_POSSIBLE_PASSWORDED_RELEASES=false
|
|
EXTRACT_USING_RARINFO=false
|
|
PRIVATE_PROFILES=true
|
|
STORE_USER_IP=false
|
|
FORUM_FRONTEND_ENABLED=false
|
|
NNTMUX_ENV
|
|
fi
|
|
# Seed the scan loop run by metafeederusenet-scanner (trap #6). Unlike
|
|
# nntmux.env above this is NOT guarded by `[ ! -f ]` and is NOT operator
|
|
# config: it is shipped code, so a store update must be able to replace a
|
|
# stale copy. Every tunable it reads lives in nntmux's `settings` table.
|
|
# Quoted heredoc delimiter ('SCANNER_SH') — the script contains $, \ and
|
|
# nested quotes that must reach the file verbatim, unexpanded.
|
|
cat > /DATA/AppData/metafeederusenet/scanner/scanner.sh <<'SCANNER_SH'
|
|
#!/bin/sh
|
|
# nntmux scan launcher.
|
|
#
|
|
# The pyrowman/nntmux image runs frankenphp and NOTHING else: no cron, no
|
|
# supervisord, no queue worker. So `settings.running=1` changes nothing on
|
|
# its own, because nothing ever starts the loop. This is that loop.
|
|
#
|
|
# It runs as a SECOND CONTAINER off the same image + the same /app/.env and
|
|
# nzb volume, rather than as a second process inside the web container, so
|
|
# the web container's entrypoint (install/populate under `set -e`) is left
|
|
# untouched.
|
|
#
|
|
# INVOCATION IS NOT FREE CHOICE. Measured PER RUNNER on a live box, NOT a
|
|
# blanket "Forking works":
|
|
# * multiprocessing/binaries.php — Forking OK.
|
|
# * multiprocessing/backfill.php — NO-OP. Reports "...3 job(s) to do
|
|
# using a max of 0 child process(es)" and returns in 0s having done
|
|
# nothing. Backfill must be driven DIRECTLY, per group (see the
|
|
# backfill block below).
|
|
# * collation MUST go through tinker/ProcessReleases DIRECTLY.
|
|
# multiprocessing/releases.php aborts every batch on a benign PHP
|
|
# warning (ReleaseCleaning.php:165) that turns fatal inside the Spatie
|
|
# child -> 0 releases while collections pile into the tens of
|
|
# thousands.
|
|
#
|
|
# Every knob comes from nntmux's own `settings` table (edited at
|
|
# /admin/tmux-edit), never from env — otherwise there are two sources of
|
|
# truth for one number.
|
|
set -u
|
|
|
|
cd /app || exit 1
|
|
|
|
setting() { php artisan tinker --execute="echo DB::table('settings')->where('name','$1')->value('value');" 2>/dev/null | tr -dc '0-9'; }
|
|
set_setting() { php artisan tinker --execute="DB::table('settings')->where('name','$1')->update(['value'=>'$2']);" >/dev/null 2>&1; }
|
|
groups() { php artisan tinker --execute="foreach (DB::table('usenet_groups')->where('active',1)->pluck('name') as \$g) echo \$g, PHP_EOL;" 2>/dev/null | grep -E '^[a-z0-9.]+$'; }
|
|
|
|
log() { echo "[scanner $(date -u +%H:%M:%S)] $*"; }
|
|
|
|
log "launcher started"
|
|
while true; do
|
|
if [ "$(setting running)" != "1" ]; then
|
|
log "settings.running=0 — idle"
|
|
sleep 30
|
|
continue
|
|
fi
|
|
|
|
if [ "$(setting binaries)" = "1" ]; then
|
|
qty="$(setting backfill_qty)"; [ -n "$qty" ] || qty=20000
|
|
log "header scan (qty=$qty)"
|
|
php misc/update/multiprocessing/binaries.php "$qty" 2>&1 | tail -3
|
|
fi
|
|
|
|
# Backfill must run DIRECT, per group. multiprocessing/backfill.php
|
|
# reports "max of 0 child process(es)" and silently does nothing — the
|
|
# Forking wrapper leaves maxProcesses=0 for this runner specifically.
|
|
# Depth is capped by settings.backfill_days, so this reaches back that
|
|
# far and then stops advancing; it is NOT a full-retention backfill.
|
|
if [ "$(setting backfill)" = "1" ]; then
|
|
# The ONLY setting this loop writes. `delaytime` (hours, default 2)
|
|
# blocks ProcessReleases from advancing a collection until it ages
|
|
# past it. Correct for the leading edge — a posting may still be
|
|
# uploading — but pointless for backfill, where every article is
|
|
# already days old. Left at 2 during backfill it stalls the catalog
|
|
# for two hours with a failure mode indistinguishable from a broken
|
|
# pipeline (collections climb, releases flatline, nothing logs an
|
|
# error).
|
|
#
|
|
# Asserted every pass, not once at boot, because the nntmux
|
|
# entrypoint RE-SEEDS `settings` on every container start (trap #7)
|
|
# — a restart silently puts this back to 2 and the tier quietly
|
|
# stops producing. Scoped to `backfill=1` so a pure forward-tailing
|
|
# deployment (`backfill=0`) keeps the operator's own value: this is
|
|
# a per-phase mode, not a preference we get to override.
|
|
if [ "$(setting delaytime)" != "0" ]; then
|
|
log "asserting delaytime=0 for the backfill phase (was $(setting delaytime))"
|
|
set_setting delaytime 0
|
|
fi
|
|
qty="$(setting backfill_qty)"; [ -n "$qty" ] || qty=20000
|
|
log "backfill direct (backfill_days=$(setting backfill_days), qty=$qty)"
|
|
groups | while read -r g; do
|
|
[ -n "$g" ] || continue
|
|
php misc/update/backfill.php "$g" "$qty" 2>&1 | grep -iE "articles|backfill|Group|no new" | tail -2
|
|
done
|
|
fi
|
|
|
|
# Collation, per active group, DIRECT (never multiprocessing/releases.php).
|
|
# Never wrap this in a short `timeout`: killing it between the release
|
|
# INSERT and createNZBs() leaves a release row with no .nzb.gz, which
|
|
# the feeder then correctly skips forever.
|
|
groups | while read -r g; do
|
|
[ -n "$g" ] || continue
|
|
log "collate $g"
|
|
php artisan tinker --execute="(new Blacklight\\processing\\ProcessReleases)->processReleases(1, 1, \"$g\", new Blacklight\\NNTP);" 2>&1 | grep -iE "releases (ready|created)|Added [0-9]+ releases|NZB" | tail -3
|
|
done
|
|
|
|
delay="$(setting monitor_delay)"; [ -n "$delay" ] || delay=30
|
|
log "pass complete; sleeping ${delay}s"
|
|
sleep "$delay"
|
|
done
|
|
SCANNER_SH
|
|
chmod 755 /DATA/AppData/metafeederusenet/scanner/scanner.sh
|
|
|
|
# Maison (Yundera dashboard) configuration. Read in preference to x-casaos.
|
|
# schema_version 2 is required for `folders`: an older Maison refuses the app
|
|
# outright rather than starting it without its directories.
|
|
x-compose-app:
|
|
schema_version: 2
|
|
# Created and chowned by Maison before any image is pulled, before any hook
|
|
# runs and before the containers start — on the first boot and every one after.
|
|
# Compose would otherwise create a missing bind source as a root-owned dir.
|
|
folders:
|
|
- path: /DATA/AppData/metafeederusenet/db
|
|
user: $PUID
|
|
group: $PGID
|
|
- path: /DATA/AppData/metafeederusenet/nntmux-env
|
|
user: $PUID
|
|
group: $PGID
|
|
- path: /DATA/AppData/metafeederusenet/nntmux-storage/nzb
|
|
user: $PUID
|
|
group: $PGID
|
|
- path: /DATA/AppData/metafeederusenet/feeder-state
|
|
user: $PUID
|
|
group: $PGID
|
|
- path: /DATA/AppData/metafeederusenet/tmdb-cache
|
|
user: $PUID
|
|
group: $PGID
|
|
- path: /DATA/AppData/metafeederusenet/scanner
|
|
user: $PUID
|
|
group: $PGID
|
|
# Same string as nntmux's caddy_0 label, so the tile link and the route agree.
|
|
# The tile opens nntmux's own admin UI (its own login, its own Caddy route).
|
|
# The feeder itself is configured from the MetaGateway dashboard.
|
|
webui-host: metafeederusenetnntmux-${domain}
|
|
webui-path: "/"
|