Import the MetaMesh app family from Worph/AppStore
23 apps (MetaCore, the library/network/client tiers and 13 MetaFeeder*) move out of the general-purpose Worph/AppStore into their own store so the MetaMesh family releases on its own cadence. - asset URLs repointed to cdn.jsdelivr.net/gh/Worph/MetaAppStore@main - store scaffolding added (category/recommend/featured lists, licence, contributing, gitattributes) - psd-source/meta-icons-v2 icon generator follows the apps MetaMCP stays in Worph/AppStore: it is third-party metatool-ai software, not part of MetaMesh.
This commit is contained in:
@@ -0,0 +1,908 @@
|
||||
name: metafeederusenet
|
||||
|
||||
# =============================================================================
|
||||
# MetaFeeder · Usenet — a STANDALONE, self-hosted Usenet feeder for MetaMesh.
|
||||
#
|
||||
# Unlike MetaFeeder · Prowlarr (which proxies an EXTERNAL Newznab/Torznab
|
||||
# indexer and mints an `nzb-release` (0x1005) cid embedding that indexer's
|
||||
# host), this app runs its OWN header scan (an `nntmux` — newznab-tmux fork —
|
||||
# sidecar) so it can mint a PORTABLE `nzb-posting` (0x1003) cid: a digest over
|
||||
# the article Message-ID set, embedding no indexer host, redeemable by ANY
|
||||
# peer with a plain NNTP provider account. It ADDS a Usenet source; it does
|
||||
# not replace the Prowlarr/Torznab one — a gateway query fans out to both.
|
||||
#
|
||||
# Design record (meta-gateway repo): docs/others/self-hosted-usenet-indexer-study.md
|
||||
# Feeder repo: github.com/worph/meta-feeder-usenet
|
||||
#
|
||||
# It needs, over the shared `pcs` network:
|
||||
# - the shared **MetaCore** app (`metacore-app:9000`) — where resolved
|
||||
# records + TMDB posters are stored; the shared **MetaShare** app then
|
||||
# announces them to the swarm.
|
||||
# The **MetaGateway** app must list this feeder's URL
|
||||
# (`http://metafeederusenet-feeder:8080`) in its feeders config so it
|
||||
# discovers + queries it (MetaGateway, MetaCore and this feeder share `pcs`).
|
||||
#
|
||||
# Everything is scoped under the `metafeederusenet-*` prefix + the per-stack
|
||||
# `metafeederusenet` network + /DATA/AppData/metafeederusenet/ data root, so it
|
||||
# installs alongside the other Meta* apps with zero collisions.
|
||||
#
|
||||
# amd64 ONLY — `pyrowman/nntmux` ships no arm64 build.
|
||||
#
|
||||
# ⚠ Four silent-failure traps in the `pyrowman/nntmux` image, empirically
|
||||
# verified (see the meta-gateway dev compose header for the full writeup):
|
||||
# 1. `PATH_TO_NZBS` (and every other storage path var) must be ABSENT from
|
||||
# the env, never present-but-blank — that reads as an explicit empty
|
||||
# string, not "unset", and nntmux aborts NZB writing with 0 releases and
|
||||
# no visible error. This compose bind-mounts a COMPLETE, hand-filled
|
||||
# `.env` (below) rather than relying on individual `environment:` keys,
|
||||
# for exactly this reason.
|
||||
# 2. `DB_CONNECTION` must be `mariadb`, not `mysql` — the install migration
|
||||
# loads its schema file by connection name; `mysql` finds no matching
|
||||
# schema and fails partway through a from-scratch migration.
|
||||
# 3. Manticore is a HARD first-boot dependency (unless
|
||||
# `ELASTICSEARCH_ENABLED=true`, which just swaps it for a heavier one) —
|
||||
# `metafeederusenet-manticore` below is not optional.
|
||||
# 4. Seeded usenet groups default to `active=0, backfill=0` — scanning an
|
||||
# inactive group silently does nothing. See the tips below for the SQL
|
||||
# to activate one.
|
||||
# One FIRST-BOOT-ONLY loud bug: the entrypoint runs `config:cache` before
|
||||
# `key:generate`, so every session route 500s until you run
|
||||
# `docker exec metafeederusenet-nntmux php artisan config:clear` once. See tips.
|
||||
#
|
||||
# ⚠ Four more, found the hard way on a live box (2026-07-26) after this stack
|
||||
# came up with SEVEN healthy containers and returned ZERO results for every
|
||||
# query — the feeder answering `"Done"` and nothing else. None of them logs
|
||||
# anything; each one alone is enough to keep the catalog empty forever.
|
||||
#
|
||||
# 5. NEVER set a key in `nntmux.env` unless the value must DIFFER from the
|
||||
# PHP default. `config/nntmux.php` does
|
||||
# `env('NN_MULTIPROCESSING_MAX_CHILD_TIME', 1800)` and Laravel's `env()`
|
||||
# returns STRINGS, so restating the default in the .env handed
|
||||
# `Pool::timeout(float)` a `string("1800")` → TypeError → every Forking
|
||||
# multiprocessing runner died on start. Those runners are exactly what a
|
||||
# scan needs. This is trap #1's type-flavoured twin: a redundant
|
||||
# restatement is not a no-op, it silently changes the value's TYPE. The
|
||||
# key is therefore deliberately ABSENT from the seeded .env below (see the
|
||||
# comment there); with it absent, `gettype()` reports `integer` and the
|
||||
# runners execute. Same rule applies to any other key you are tempted to
|
||||
# "document" by writing its default into the file.
|
||||
# (Upstream record: meta-gateway `dev/docker-compose.feeder-usenet.yml`,
|
||||
# the "CORRECTION (2026-07-26)" block.)
|
||||
# 6. The image runs `frankenphp` and NOTHING ELSE — no cron, no supervisord,
|
||||
# no queue worker. nntmux's own `settings.running=1` therefore starts
|
||||
# nothing: there is no process to read it. Even with valid NNTP creds and
|
||||
# active groups the catalog stays empty permanently. That is why
|
||||
# `metafeederusenet-scanner` below exists — it is the missing loop, run as
|
||||
# a SECOND container off the same image so the web container's
|
||||
# install/populate entrypoint stays untouched.
|
||||
# 7. The nntmux entrypoint RE-SEEDS `settings` and `usenet_groups` on EVERY
|
||||
# container boot. Configuration applied before a restart silently reverts
|
||||
# (observed: `running`→0, `delaytime`→2, every group→`active=0`). Apply
|
||||
# group activation and scheduler settings AFTER the final restart, and
|
||||
# re-check them after any `docker restart` / stack update.
|
||||
# 8. `settings.delaytime` (HOURS, default 2) stalls a backfilled catalog:
|
||||
# `ProcessReleases` refuses to advance a collection until it has aged past
|
||||
# it. That is correct for live tailing (a posting may still be uploading)
|
||||
# and pointless for backfill, where every article is already days old —
|
||||
# and the symptom, collections climbing while releases flatline, is
|
||||
# indistinguishable from a broken pipeline because nothing logs an error.
|
||||
# Because trap #7 re-seeds it back to 2 on every restart, the scanner
|
||||
# loop below ASSERTS `delaytime=0` on each pass **while `backfill=1`** —
|
||||
# the one setting it writes, making the tier self-healing across
|
||||
# restarts. It is scoped to the backfill phase on purpose: with
|
||||
# `backfill=0` (pure forward tailing) the operator's own value stands.
|
||||
# =============================================================================
|
||||
|
||||
services:
|
||||
# ---------------------------------------------------------------------------
|
||||
# metafeederusenet-db — MariaDB backing nntmux's OWN release catalog. Not
|
||||
# shared with the gateway's meta-core Redis or anything else in the mesh.
|
||||
# ---------------------------------------------------------------------------
|
||||
metafeederusenet-db:
|
||||
image: mariadb:12.3
|
||||
cpu_shares: 70
|
||||
container_name: metafeederusenet-db
|
||||
hostname: metafeederusenet-db
|
||||
restart: unless-stopped
|
||||
# 2g, not the 512m this shipped with: 512m is fine for an empty schema and
|
||||
# far too small once the catalog is real. A header scan of a busy binary
|
||||
# group builds `collections`/`binaries`/`parts` tables in the tens of
|
||||
# millions of rows, and InnoDB starved of buffer pool turns collation into
|
||||
# a disk-thrash that never catches up. Measured on a live box.
|
||||
mem_limit: 2g
|
||||
memswap_limit: 2g
|
||||
environment:
|
||||
MARIADB_DATABASE: nntmux
|
||||
MARIADB_USER: nntmux
|
||||
MARIADB_PASSWORD: "$APP_DEFAULT_PASSWORD"
|
||||
MARIADB_RANDOM_ROOT_PASSWORD: "yes"
|
||||
volumes:
|
||||
- ${DATA_ROOT:-/DATA}/AppData/metafeederusenet/db:/var/lib/mysql
|
||||
healthcheck:
|
||||
test: ["CMD", "healthcheck.sh", "--connect", "--innodb_initialized"]
|
||||
interval: 5s
|
||||
timeout: 5s
|
||||
retries: 20
|
||||
start_period: 20s
|
||||
networks:
|
||||
- metafeederusenet
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# metafeederusenet-redis — nntmux's OWN Redis. Deliberately separate from the
|
||||
# gateway core's Redis (different failure domain; losing this one is free).
|
||||
# ---------------------------------------------------------------------------
|
||||
metafeederusenet-redis:
|
||||
image: redis:7-alpine
|
||||
cpu_shares: 70
|
||||
container_name: metafeederusenet-redis
|
||||
hostname: metafeederusenet-redis
|
||||
restart: unless-stopped
|
||||
mem_limit: 128m
|
||||
memswap_limit: 128m
|
||||
networks:
|
||||
- metafeederusenet
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# metafeederusenet-manticore — RT-index search backend. NOT optional (trap
|
||||
# #3 above) — a hard first-boot dependency of nntmux's own entrypoint, even
|
||||
# though this feeder reads `releases` via SQL directly and never touches it.
|
||||
# Digest-pinned (upstream reuses low-looking version numbers across an old
|
||||
# semver line and a newer CalVer-ish one; the digest is unambiguous).
|
||||
# ---------------------------------------------------------------------------
|
||||
metafeederusenet-manticore:
|
||||
image: manticoresearch/manticore@sha256:032388dc77321e3ac898a14b9a0c3fc22ab8d08cd2582abd6c5b642d46b82181
|
||||
cpu_shares: 50
|
||||
container_name: metafeederusenet-manticore
|
||||
hostname: metafeederusenet-manticore
|
||||
restart: unless-stopped
|
||||
mem_limit: 512m
|
||||
memswap_limit: 512m
|
||||
ulimits:
|
||||
nofile: { soft: 65535, hard: 65535 }
|
||||
networks:
|
||||
- metafeederusenet
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# metafeederusenet-nntmux — the header scanner + web UI/API. Exposed DIRECTLY
|
||||
# via Caddy (no AppShield gate): nntmux ships its own admin login
|
||||
# (ADMIN_USER/ADMIN_PASS, seeded below), the same convention this store uses
|
||||
# for every app that brings its own auth (Jellyfin, Nextcloud, Immich, ...).
|
||||
# ---------------------------------------------------------------------------
|
||||
metafeederusenet-nntmux:
|
||||
# Digest-pinned, NOT :latest — a silent tag move could change the
|
||||
# `misc/update/*.php` command surface the "collate via tinker" tip below
|
||||
# depends on. Re-pin deliberately, re-verify the trap list above when you do.
|
||||
image: pyrowman/nntmux@sha256:7fc221f7a806fee3e4a2145159e1a147962edc095282bdc692806968f791fabe
|
||||
cpu_shares: 50
|
||||
container_name: metafeederusenet-nntmux
|
||||
hostname: metafeederusenet-nntmux
|
||||
restart: unless-stopped
|
||||
# The working set scales with scan breadth (group set x backfill depth), an
|
||||
# operator choice (see tips), so this is a ceiling rather than a size: 3g
|
||||
# is what a real corpus needed on a live box — 1536m was reached and
|
||||
# exceeded by a single busy group's header scan. Raise it if you scan wide.
|
||||
mem_limit: 3g
|
||||
memswap_limit: 3g
|
||||
environment:
|
||||
# Shell-script-consumed vars — the entrypoint's own `mkdir -p` step reads
|
||||
# these from the PROCESS environment, not the mounted .env. Keep in
|
||||
# lockstep with the matching keys in nntmux.env below.
|
||||
PATH_TO_NZBS: /app/storage/nzb/
|
||||
COVERS_PATH: /app/storage/covers
|
||||
TEMP_UNRAR_PATH: /app/storage/tmp/unrar/
|
||||
TEMP_UNZIP_PATH: /app/storage/tmp/unzip/
|
||||
ELASTICSEARCH_ENABLED: "false"
|
||||
volumes:
|
||||
# The complete, hand-filled .env (trap #1) — a full-file mount rather
|
||||
# than individual `environment:` keys. Writable: `nntmux:install`
|
||||
# persists a generated APP_KEY back into it on first boot.
|
||||
- ${DATA_ROOT:-/DATA}/AppData/metafeederusenet/nntmux-env/nntmux.env:/app/.env:rw
|
||||
# The .nzb store — shared with metafeederusenet-feeder below,
|
||||
# read-write here (nntmux writes), read-only there (feeder only reads).
|
||||
- ${DATA_ROOT:-/DATA}/AppData/metafeederusenet/nntmux-storage/nzb:/app/storage/nzb:rw
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-sf", "-o", "/dev/null", "http://localhost/"]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 15
|
||||
start_period: 30s
|
||||
depends_on:
|
||||
metafeederusenet-db:
|
||||
condition: service_healthy
|
||||
metafeederusenet-redis:
|
||||
condition: service_started
|
||||
metafeederusenet-manticore:
|
||||
condition: service_started
|
||||
labels:
|
||||
caddy_0: metafeederusenetnntmux-${APP_DOMAIN}
|
||||
caddy_0.import: gateway_tls
|
||||
caddy_0.reverse_proxy: "{{upstreams 80}}"
|
||||
caddy_1: metafeederusenetnntmux-${APP_PUBLIC_IP_DASH}.nip.io
|
||||
caddy_1.import: gateway_tls
|
||||
caddy_1.reverse_proxy: "{{upstreams 80}}"
|
||||
caddy_2: metafeederusenetnntmux-${APP_PUBLIC_IP_DASH}.sslip.io
|
||||
caddy_2.reverse_proxy: "{{upstreams 80}}"
|
||||
networks:
|
||||
- metafeederusenet
|
||||
- pcs
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# metafeederusenet-scanner — the SCAN LOOP (trap #6). The nntmux image runs
|
||||
# frankenphp and nothing else, so without this container nothing ever calls
|
||||
# the scan/backfill/collate entry points and the catalog stays empty forever,
|
||||
# no matter how correct the credentials, the groups or `settings.running` are.
|
||||
#
|
||||
# It is a second container off the SAME digest-pinned image, sharing the same
|
||||
# `/app/.env` and the same nzb store, rather than a second process inside the
|
||||
# web container — the web container's entrypoint runs install/populate under
|
||||
# `set -e` and must keep it; here it is REPLACED by `entrypoint:` below so
|
||||
# that install/populate does NOT run twice against one database.
|
||||
#
|
||||
# Every knob (interval, binaries on/off, backfill on/off, backfill_qty,
|
||||
# backfill_days, running) is read from nntmux's own `settings` table at
|
||||
# /admin/tmux-edit — deliberately NOT from new env vars here, which would
|
||||
# make two sources of truth for one number.
|
||||
# ---------------------------------------------------------------------------
|
||||
metafeederusenet-scanner:
|
||||
# MUST stay byte-identical to metafeederusenet-nntmux's pin above: the two
|
||||
# containers share one `/app/.env` and one database schema, and the script
|
||||
# below calls `misc/update/*.php` + `Blacklight\` classes by name.
|
||||
image: pyrowman/nntmux@sha256:7fc221f7a806fee3e4a2145159e1a147962edc095282bdc692806968f791fabe
|
||||
cpu_shares: 20
|
||||
container_name: metafeederusenet-scanner
|
||||
hostname: metafeederusenet-scanner
|
||||
restart: unless-stopped
|
||||
# Bypasses the image's own install/populate entrypoint on purpose — see the
|
||||
# block comment above. Only the web container may run it.
|
||||
entrypoint: ["/bin/sh", "/scanner.sh"]
|
||||
# The image ships a HEALTHCHECK that curls its own web server. This
|
||||
# container deliberately runs no web server — its entrypoint IS the scan
|
||||
# loop — so the inherited probe can only ever fail. Observed live: a
|
||||
# scanner actively backfilling and writing NZBs sitting at
|
||||
# `Up 5 minutes (unhealthy)`.
|
||||
#
|
||||
# That is not cosmetic. A permanently-red container that is in fact working
|
||||
# trains the operator to ignore health status, which is precisely how this
|
||||
# stack's empty-catalog failure stayed invisible for so long — the feeder's
|
||||
# own health() reporting `ok` over a zero-row catalog was the same bug from
|
||||
# the other direction. Liveness for this container is its LOG
|
||||
# (`docker logs -f metafeederusenet-scanner`), not a probe; a probe that
|
||||
# cannot be honest is worse than none.
|
||||
healthcheck:
|
||||
disable: true
|
||||
# A backfill pass holds a group's article range in memory; 2g is the
|
||||
# headroom measured alongside nntmux's own 3g on a live box.
|
||||
mem_limit: 2g
|
||||
memswap_limit: 2g
|
||||
environment:
|
||||
# Same shell-script-consumed vars as metafeederusenet-nntmux: the wrapper
|
||||
# scripts read these from the PROCESS environment, not from the mounted
|
||||
# .env (trap #1). Keep the two blocks in lockstep.
|
||||
PATH_TO_NZBS: /app/storage/nzb/
|
||||
COVERS_PATH: /app/storage/covers
|
||||
TEMP_UNRAR_PATH: /app/storage/tmp/unrar/
|
||||
TEMP_UNZIP_PATH: /app/storage/tmp/unzip/
|
||||
ELASTICSEARCH_ENABLED: "false"
|
||||
volumes:
|
||||
# The same .env the web container filled in — one credential set, one
|
||||
# APP_KEY. Read-write only because Laravel's .env handling expects it.
|
||||
- ${DATA_ROOT:-/DATA}/AppData/metafeederusenet/nntmux-env/nntmux.env:/app/.env:rw
|
||||
# nntmux writes .nzb.gz here; the feeder reads them read-only.
|
||||
- ${DATA_ROOT:-/DATA}/AppData/metafeederusenet/nntmux-storage/nzb:/app/storage/nzb:rw
|
||||
# The loop itself, provisioned by pre-install-cmd below.
|
||||
- ${DATA_ROOT:-/DATA}/AppData/metafeederusenet/scanner/scanner.sh:/scanner.sh:ro
|
||||
depends_on:
|
||||
# Not just `started`: the loop's very first action is an artisan call, so
|
||||
# it must not race the web container's install/migrate step.
|
||||
metafeederusenet-nntmux:
|
||||
condition: service_healthy
|
||||
networks:
|
||||
- metafeederusenet
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# metafeederusenet-feeder — the usenet feeder sidecar (meta-feeder-usenet).
|
||||
# Reads nntmux's release catalog + shared .nzb store; self-publishes
|
||||
# resolved records + seeds the `.nzb` manifest to the shared MetaCore. The
|
||||
# gateway reaches it directly on `pcs`; the human configures it through the
|
||||
# AppShield gate below.
|
||||
# ---------------------------------------------------------------------------
|
||||
metafeederusenet-feeder:
|
||||
image: ghcr.io/worph/meta-feeder-usenet:1.0.2
|
||||
cpu_shares: 30
|
||||
container_name: metafeederusenet-feeder
|
||||
# Hard RAM cap, swap disabled (memswap_limit == mem_limit) → OOM instead
|
||||
# of swap. Mirrors the dev sub-stack limits.
|
||||
mem_limit: 256m
|
||||
memswap_limit: 256m
|
||||
hostname: metafeederusenet-feeder
|
||||
restart: unless-stopped
|
||||
user: "0:0"
|
||||
expose:
|
||||
- 8080
|
||||
environment:
|
||||
# INFRA-ONLY env (the "config: no env if in webui" rule). Operator
|
||||
# config beyond the two DB/path seeds below — TMDB token / language,
|
||||
# meta-core URL — is read from the persisted config.json (this
|
||||
# feeder's config page behind the AppShield gate). These env vars are
|
||||
# wiring only: the enrichment sidecars, the enrich callback, the peer id.
|
||||
RUST_LOG: "info,meta_feeder=info"
|
||||
# The peer id stamped into provenance.gateway_peer on every record this
|
||||
# feeder self-publishes — names the GATEWAY this feeder serves, not the
|
||||
# feeder itself (analytics group by it). Without it, falls back to HOSTNAME.
|
||||
META_GATEWAY_PEER_ID: "metagateway"
|
||||
FILENAME_PARSER_URL: "http://metafeederusenet-filename-parser:8080"
|
||||
TMDB_PLUGIN_URL: "http://metafeederusenet-tmdb:8080"
|
||||
META_FEEDER_CALLBACK_URL: "http://metafeederusenet-feeder:8080/enrich/callback"
|
||||
# First-boot SEED only (the persisted config.json wins on every later
|
||||
# restart — change these on the /config page after first boot, not here).
|
||||
NNTMUX_DB_URL: "mysql://nntmux:${APP_DEFAULT_PASSWORD}@metafeederusenet-db:3306/nntmux"
|
||||
NNTMUX_NZB_PATH: "/nntmux-nzb"
|
||||
TZ: "$TZ"
|
||||
volumes:
|
||||
# Persist feeder config (config.json saved via the config UI) + redb cache.
|
||||
- ${DATA_ROOT:-/DATA}/AppData/metafeederusenet/feeder-state:/data/meta-feeder:rw
|
||||
# Same volume nntmux writes .nzb.gz into — read-only here.
|
||||
- ${DATA_ROOT:-/DATA}/AppData/metafeederusenet/nntmux-storage/nzb:/nntmux-nzb:ro
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-sf", "http://localhost:8080/health"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 10
|
||||
start_period: 5s
|
||||
depends_on:
|
||||
metafeederusenet-nntmux:
|
||||
condition: service_healthy
|
||||
metafeederusenet-filename-parser:
|
||||
condition: service_started
|
||||
metafeederusenet-tmdb:
|
||||
condition: service_started
|
||||
networks:
|
||||
- metafeederusenet
|
||||
- pcs
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Bundled enrichment plugins (self-contained). filename-parser per file; tmdb
|
||||
# once per release — tmdb writes posters/backdrops to the shared meta-core's
|
||||
# WebDAV (over pcs) so the ingester seeds them. No healthcheck: these plugin
|
||||
# images don't ship one (curl absent).
|
||||
# ---------------------------------------------------------------------------
|
||||
metafeederusenet-filename-parser:
|
||||
image: ghcr.io/worph/metamesh-plugin-filename-parser:1.0.2
|
||||
cpu_shares: 30
|
||||
container_name: metafeederusenet-filename-parser
|
||||
mem_limit: 256m
|
||||
memswap_limit: 256m
|
||||
hostname: metafeederusenet-filename-parser
|
||||
restart: unless-stopped
|
||||
expose:
|
||||
- 8080
|
||||
environment:
|
||||
EMIT_TITLE_TAGS: "true"
|
||||
networks:
|
||||
- metafeederusenet
|
||||
|
||||
metafeederusenet-tmdb:
|
||||
image: ghcr.io/worph/metamesh-plugin-tmdb:1.0.2
|
||||
cpu_shares: 30
|
||||
container_name: metafeederusenet-tmdb
|
||||
mem_limit: 384m
|
||||
memswap_limit: 384m
|
||||
hostname: metafeederusenet-tmdb
|
||||
restart: unless-stopped
|
||||
expose:
|
||||
- 8080
|
||||
environment:
|
||||
# Posters land in the shared meta-core (over pcs) so the ingester seeds
|
||||
# them. LOAD-BEARING at metamesh-plugin-tmdb:1.0.0 — do NOT remove (see
|
||||
# the sibling feeder apps' identical note: the published image needs
|
||||
# this, unlike the source build the dev stack runs).
|
||||
WEBDAV_URL: "http://metacore-app:9000/webdav"
|
||||
volumes:
|
||||
- ${DATA_ROOT:-/DATA}/AppData/metafeederusenet/tmdb-cache:/cache
|
||||
networks:
|
||||
- metafeederusenet
|
||||
- pcs
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# AppShield gate — the feeder's config UI (nntmux DB URL / nzb path, TMDB
|
||||
# token) behind Yundera OIDC SSO. container_name == subdomain prefix. The
|
||||
# gateway still reaches the feeder directly on `pcs` (this gate is for the
|
||||
# human only); index → /config.
|
||||
# ---------------------------------------------------------------------------
|
||||
metafeederusenet:
|
||||
image: ghcr.io/yundera/appshield:2.0.9
|
||||
cpu_shares: 80
|
||||
container_name: metafeederusenet
|
||||
hostname: metafeederusenet
|
||||
restart: unless-stopped
|
||||
user: "0:0"
|
||||
expose:
|
||||
- 80
|
||||
environment:
|
||||
OIDC_REGISTRAR_URL: "http://auth-registrar:9092"
|
||||
REDIRECT_HOST_SUFFIXES: "${APP_DOMAIN},${APP_PUBLIC_IP_DASH}.nip.io,${APP_PUBLIC_IP_DASH}.sslip.io"
|
||||
BACKEND_HOST: "metafeederusenet-feeder"
|
||||
BACKEND_PORT: "8080"
|
||||
LISTEN_PORT: "80"
|
||||
labels:
|
||||
caddy_0: metafeederusenet-${APP_DOMAIN}
|
||||
caddy_0.import: gateway_tls
|
||||
caddy_0.reverse_proxy: "{{upstreams 80}}"
|
||||
caddy_1: metafeederusenet-${APP_PUBLIC_IP_DASH}.nip.io
|
||||
caddy_1.import: gateway_tls
|
||||
caddy_1.reverse_proxy: "{{upstreams 80}}"
|
||||
caddy_2: metafeederusenet-${APP_PUBLIC_IP_DASH}.sslip.io
|
||||
caddy_2.reverse_proxy: "{{upstreams 80}}"
|
||||
depends_on:
|
||||
- metafeederusenet-feeder
|
||||
networks:
|
||||
- metafeederusenet
|
||||
- pcs
|
||||
|
||||
networks:
|
||||
metafeederusenet:
|
||||
name: metafeederusenet
|
||||
driver: bridge
|
||||
pcs:
|
||||
name: ${APP_NET:-pcs}
|
||||
external: true
|
||||
|
||||
x-casaos:
|
||||
architectures:
|
||||
- amd64
|
||||
|
||||
author: Worph Team
|
||||
category: Media
|
||||
developer: Aptero Team
|
||||
main: metafeederusenet
|
||||
webui_port: 80
|
||||
index: /config
|
||||
|
||||
description:
|
||||
en_us: |
|
||||
**A standalone, self-hosted Usenet feeder for the MetaMesh network.**
|
||||
|
||||
MetaFeeder · Usenet runs its own Usenet header scan (a bundled
|
||||
**nntmux** indexer) instead of proxying a third-party Newznab site.
|
||||
Because it finds article Message-IDs itself at scan time, it mints a
|
||||
**portable** posting reference — no indexer host baked in, redeemable by
|
||||
any peer with a plain NNTP provider account. It adds a Usenet source
|
||||
alongside **MetaFeeder · Prowlarr**; it doesn't replace it.
|
||||
|
||||
**Self-contained:** ships its own nntmux scanner (MariaDB + Redis +
|
||||
Manticore) + filename-parser + TMDB enrichment, so each release gets a
|
||||
clean title, poster and plot.
|
||||
|
||||
**You still need a Usenet provider account** (NNTP login) — this app
|
||||
scans headers, it does not supply an account, and nothing at all is
|
||||
indexed until you supply one. Configure it after install (see the
|
||||
install tips). Keep the connection count low: this scanner and your
|
||||
playback share the same provider connection slots.
|
||||
|
||||
**The catalog is not instant.** Once configured, a bundled scanner
|
||||
container runs the header-scan / backfill / collation loop on the
|
||||
interval you set. A first pass over a few busy binary groups pulls
|
||||
tens of GB and runs for hours; a deep backfill is hundreds of GB. The
|
||||
install tips show which knobs bound that.
|
||||
|
||||
**Also requires:** **MetaGateway** — its meta-core stores the resolved
|
||||
records and announces them to the swarm. The gateway must list this
|
||||
feeder in its config.
|
||||
|
||||
fr_fr: |
|
||||
**Un feeder Usenet autonome et auto-hébergé pour le réseau MetaMesh.**
|
||||
|
||||
MetaFeeder · Usenet effectue lui-même le scan des en-têtes Usenet (un
|
||||
indexeur **nntmux** embarqué) plutôt que de passer par un site Newznab
|
||||
tiers. Comme il retrouve les Message-ID des articles dès le scan, il
|
||||
génère une référence de posting **portable** — sans hôte d'indexeur
|
||||
intégré, exploitable par n'importe quel pair disposant d'un simple
|
||||
compte NNTP. Il ajoute une source Usenet en plus de **MetaFeeder -
|
||||
Indexer**, sans le remplacer.
|
||||
|
||||
**Autonome :** embarque son scanner nntmux (MariaDB + Redis + Manticore)
|
||||
+ ses plugins filename-parser + TMDB.
|
||||
|
||||
**Un compte fournisseur Usenet (NNTP) reste nécessaire** — cette app
|
||||
scanne les en-têtes, elle ne fournit pas d'accès, et rien n'est indexé
|
||||
tant que vous n'en fournissez pas un. À configurer après installation
|
||||
(voir les conseils d'installation). Gardez un nombre de connexions
|
||||
faible : le scan et la lecture partagent les mêmes créneaux du
|
||||
fournisseur.
|
||||
|
||||
**Le catalogue ne se remplit pas instantanément.** Une fois configuré,
|
||||
un conteneur de scan embarqué exécute la boucle scan / backfill /
|
||||
collation à l'intervalle que vous choisissez. Un premier passage sur
|
||||
quelques gros groupes binaires télécharge des dizaines de Go et dure
|
||||
des heures ; un backfill profond, des centaines de Go.
|
||||
|
||||
**Nécessite aussi :** l'app **MetaGateway** (son meta-core stocke et
|
||||
annonce les enregistrements).
|
||||
|
||||
tagline:
|
||||
en_us: Self-hosted Usenet header scanner (bundled nntmux) for the MetaMesh network
|
||||
fr_fr: Scanner Usenet auto-hébergé (nntmux inclus) pour le réseau MetaMesh
|
||||
|
||||
title:
|
||||
en_us: MetaFeeder · Usenet
|
||||
|
||||
icon: https://cdn.jsdelivr.net/gh/Worph/MetaAppStore@main/Apps/MetaFeederUsenet/icon.png
|
||||
screenshot_link:
|
||||
- https://cdn.jsdelivr.net/gh/Worph/MetaAppStore@main/Apps/MetaFeederUsenet/screenshot-1.png
|
||||
- https://cdn.jsdelivr.net/gh/Worph/MetaAppStore@main/Apps/MetaFeederUsenet/screenshot-2.png
|
||||
- https://cdn.jsdelivr.net/gh/Worph/MetaAppStore@main/Apps/MetaFeederUsenet/screenshot-3.png
|
||||
thumbnail: https://cdn.jsdelivr.net/gh/Worph/MetaAppStore@main/Apps/MetaFeederUsenet/thumbnail.png
|
||||
|
||||
store_app_id: metafeederusenet
|
||||
is_uncontrolled: false
|
||||
|
||||
tips:
|
||||
before_install:
|
||||
en_us: |
|
||||
**Install order:** install **MetaGateway** first (this feeder
|
||||
publishes to its shared meta-core).
|
||||
|
||||
**After installing this app, a Usenet provider account is required**
|
||||
(this scans headers itself — it needs a plain NNTP login):
|
||||
1. Edit `/DATA/AppData/metafeederusenet/nntmux-env/nntmux.env` (via
|
||||
SSH or a file-manager app) and fill in `NNTP_SERVER`, `NNTP_PORT`,
|
||||
`NNTP_USERNAME`, `NNTP_PASSWORD` (use the SAME provider account
|
||||
your MetaShare/meta-share peer is already credentialed against —
|
||||
one account, two consumers). Keep `NNTP_CONNECTIONS` low (1-2):
|
||||
the scanner and playback share the same provider connection slots,
|
||||
so a greedy value here stalls playback mid-stream elsewhere.
|
||||
2. Restart the nntmux container: `docker restart metafeederusenet-nntmux`.
|
||||
3. **First boot only:** nntmux's install sequence caches its config
|
||||
before generating an encryption key, so every page 500s until you
|
||||
run `docker exec metafeederusenet-nntmux php artisan config:clear`
|
||||
once. (`curl -sI https://metafeederusenetnntmux-<your-domain>/` — a
|
||||
`500` means you need this; a `302` redirect to `/login` means
|
||||
you're fine.)
|
||||
4. Log in to the nntmux dashboard at the **MetaFeeder Usenet nntmux**
|
||||
link (`ADMIN_USER=admin`, password = this app's auto-generated
|
||||
default password — check CasaOS's app credentials panel) and
|
||||
change it.
|
||||
|
||||
**⚠ Do steps 5-6 AFTER your LAST restart, and re-check them after
|
||||
every later restart or stack update.** The nntmux entrypoint RE-SEEDS
|
||||
the `settings` and `usenet_groups` tables on EVERY container boot, so
|
||||
anything you configure before a restart silently reverts (observed:
|
||||
`running` back to 0, `delaytime` back to 2, every group back to
|
||||
`active=0`) — with no error anywhere.
|
||||
|
||||
5. **Activate at least one usenet group** — seeded groups start
|
||||
inactive (`active=0`), and scanning an inactive group silently
|
||||
touches nothing at all:
|
||||
```
|
||||
docker exec metafeederusenet-db mariadb -unntmux -p"<password>" nntmux \
|
||||
-e "UPDATE usenet_groups SET active=1, backfill=1 WHERE name IN ('alt.binaries.teevee');"
|
||||
```
|
||||
6. **Start the scheduler** in nntmux's admin UI (**/admin/tmux-edit**)
|
||||
or directly in the `settings` table: set `running=1`, pick your
|
||||
`monitor_delay` (seconds between passes), and turn on `binaries`
|
||||
and/or `backfill` with a `backfill_qty` / `backfill_days` you're
|
||||
happy to pay for. The bundled `metafeederusenet-scanner` container
|
||||
polls these settings and runs the scan/backfill/collate loop for
|
||||
you — the nntmux image itself ships NO cron and NO queue worker, so
|
||||
this container is what actually makes a scan happen. There are
|
||||
deliberately no env vars for these knobs: the `settings` table is
|
||||
the single source of truth.
|
||||
|
||||
**`delaytime` (HOURS, default 2) will stall a backfilled catalog.**
|
||||
nntmux won't turn a collection into a release until it has aged
|
||||
past `delaytime`, which is right for tailing live posts and
|
||||
pointless when backfilling history — and the symptom (collections
|
||||
climbing, releases flat, zero query results) looks exactly like a
|
||||
broken pipeline. **While `backfill=1`, the scanner forces
|
||||
`delaytime=0` for you on every pass** and logs
|
||||
`asserting delaytime=0 for the backfill phase (was N)` when it does
|
||||
— it has to re-assert it because a container restart re-seeds the
|
||||
value back to 2. This is the only setting the loop ever writes.
|
||||
|
||||
So if you want a different steady-state `delaytime`, **set
|
||||
`backfill=0` first**, then set your value — otherwise the loop will
|
||||
keep pulling it back to 0. With `backfill=0` the scanner never
|
||||
touches it and `1`-`2` is the sensible live-tailing value.
|
||||
|
||||
**Budget the first scan.** A first header pass over a few busy
|
||||
binary groups pulls tens of GB and runs for hours; a
|
||||
full-retention backfill is hundreds of GB. `backfill_days` is the
|
||||
knob that bounds it — start small, widen once you see releases.
|
||||
|
||||
7. Open this app's config UI (the dashboard link) to optionally set a
|
||||
TMDB v4 token for posters, then Save.
|
||||
8. In **MetaGateway**, make sure this feeder is listed in its feeders
|
||||
config (`http://metafeederusenet-feeder:8080`) so it gets
|
||||
discovered.
|
||||
|
||||
**Note:** which groups to scan, backfill depth and scan interval are
|
||||
your call as the operator — pick them for your own catalog.
|
||||
|
||||
**Watch the loop:** `docker logs -f metafeederusenet-scanner` prints
|
||||
one line per pass (`header scan` / `backfill direct` / `collate
|
||||
<group>` / `sleeping`). `settings.running=0` shows up there as
|
||||
`idle`.
|
||||
|
||||
**Authentication:** the feeder's config UI is protected by Yundera's
|
||||
Authelia (OIDC); nntmux uses its own login (seeded above).
|
||||
|
||||
pre-install-cmd: |
|
||||
# Seed nntmux's .env (trap #1 — must be a complete file, see header).
|
||||
# NNTP_* are deliberately left blank; fill them in per the install tips.
|
||||
if [ ! -f /DATA/AppData/metafeederusenet/nntmux-env/nntmux.env ]; then
|
||||
cat > /DATA/AppData/metafeederusenet/nntmux-env/nntmux.env <<NNTMUX_ENV
|
||||
DB_CONNECTION=mariadb
|
||||
DB_HOST=metafeederusenet-db
|
||||
DB_PORT=3306
|
||||
DB_SOCKET=
|
||||
DB_USERNAME=nntmux
|
||||
DB_PASSWORD=$APP_DEFAULT_PASSWORD
|
||||
DB_DATABASE=nntmux
|
||||
|
||||
MANTICORESEARCH_HOST=metafeederusenet-manticore
|
||||
MANTICORESEARCH_PORT=9308
|
||||
|
||||
ELASTICSEARCH_HOST=localhost
|
||||
ELASTICSEARCH_PORT=9200
|
||||
ELASTICSEARCH_SCHEME=http
|
||||
ELASTICSEARCH_USER=
|
||||
ELASTICSEARCH_PASS=
|
||||
ELASTICSEARCH_LOGGING=false
|
||||
ELASTICSEARCH_ENABLED=false
|
||||
|
||||
NNTP_COMPRESSED_HEADERS=false
|
||||
USE_ALTERNATE_NNTP_SERVER=false
|
||||
|
||||
NNTP_USERNAME=
|
||||
NNTP_PASSWORD=
|
||||
NNTP_SERVER=
|
||||
NNTP_PORT=563
|
||||
NNTP_CONNECTIONS=1
|
||||
NNTP_SSLENABLED=true
|
||||
NNTP_SOCKET_TIMEOUT=120
|
||||
|
||||
NNTP_USERNAME_A=
|
||||
NNTP_PASSWORD_A=
|
||||
NNTP_SERVER_A=
|
||||
NNTP_PORT_A=
|
||||
NNTP_CONNECTIONS_A=
|
||||
NNTP_SSLENABLED_A=false
|
||||
NNTP_SOCKET_TIMEOUT_A=120
|
||||
|
||||
# NN_MULTIPROCESSING_MAX_CHILD_TIME is deliberately ABSENT (trap #5).
|
||||
# Writing it here, even as its own default of 1800, hands
|
||||
# Pool::timeout(float) a string via Laravel's env(), which returns STRINGS,
|
||||
# and every Forking multiprocessing runner dies with a TypeError.
|
||||
# RULE for this whole file: never set a key unless the value must DIFFER
|
||||
# from the PHP default in config/nntmux.php. Restating a default is not
|
||||
# documentation, it is a type change.
|
||||
|
||||
ADMIN_USER=admin
|
||||
ADMIN_PASS=$APP_DEFAULT_PASSWORD
|
||||
ADMIN_EMAIL=admin@$APP_DOMAIN
|
||||
|
||||
APP_NAME=NNTmux
|
||||
APP_ENV=production
|
||||
APP_DEBUG=false
|
||||
APP_TIMEZONE=UTC
|
||||
APP_URL=https://metafeederusenetnntmux-$APP_DOMAIN
|
||||
|
||||
APP_LOCALE=en
|
||||
APP_FALLBACK_LOCALE=en
|
||||
APP_FAKER_LOCALE=en_US
|
||||
|
||||
APP_MAINTENANCE_DRIVER=file
|
||||
APP_MAINTENANCE_STORE=database
|
||||
|
||||
BCRYPT_ROUNDS=12
|
||||
APP_KEY=
|
||||
LOG_CHANNEL=stack
|
||||
LOG_STACK=single
|
||||
PASSWORD_HASH=argon2id
|
||||
|
||||
BROADCAST_CONNECTION=log
|
||||
CACHE_STORE=file
|
||||
QUEUE_CONNECTION=sync
|
||||
|
||||
REDIS_HOST=metafeederusenet-redis
|
||||
REDIS_PASSWORD=null
|
||||
REDIS_PORT=6379
|
||||
REDIS_CLIENT=phpredis
|
||||
|
||||
SESSION_DRIVER=file
|
||||
SESSION_DOMAIN=null
|
||||
SESSION_SECURE_COOKIE=false
|
||||
SESSION_ENCRYPT=false
|
||||
SESSION_COOKIE=nntmux
|
||||
SESSION_PATH=/
|
||||
|
||||
SCOUT_DRIVER=database
|
||||
SCOUT_QUEUE=true
|
||||
|
||||
ITEMS_PER_PAGE=50
|
||||
ITEMS_PER_COVER_PAGE=25
|
||||
MAX_PAGER_RESULTS=125000
|
||||
ECHOCLI=true
|
||||
RENAME_PAR2=false
|
||||
ADD_PAR2=false
|
||||
RENAME_MUSIC_MEDIAINFO=true
|
||||
CACHE_EXPIRY_SHORT=5
|
||||
CACHE_EXPIRY_MEDIUM=10
|
||||
CACHE_EXPIRY_LONG=15
|
||||
|
||||
SSL_CAFILE=
|
||||
SSL_CAPATH=
|
||||
SSL_VERIFY_PEER=false
|
||||
SSL_VERIFY_HOST=false
|
||||
SSL_ALLOW_SELF_SIGNED=true
|
||||
|
||||
SMARTY_DEBUG=false
|
||||
SMARTY_CACHING=false
|
||||
SMARTY_CACHE_LIFE=120
|
||||
SMARTY_COMPILE_CHECK=false
|
||||
SMARTY_FORCE_COMPILE=false
|
||||
SMARTY_CACHE_DRIVER=file
|
||||
|
||||
TMDB_APIKEY=9a4e16adddcd1e86da19bcaf5ff3c2a3
|
||||
TMDB_CACHE=false
|
||||
TMDB_LOG=false
|
||||
TVDB_APIKEY=ef6fb572-bcee-4d99-9b52-90549ad7553a
|
||||
TVDB_PIN=
|
||||
|
||||
TEMP_UNRAR_PATH=/app/storage/tmp/unrar/
|
||||
TEMP_UNZIP_PATH=/app/storage/tmp/unzip/
|
||||
VIEW_COMPILED_PATH=/app/storage/framework/views
|
||||
PATH_TO_NZBS=/app/storage/nzb/
|
||||
COVERS_PATH=/app/storage/covers
|
||||
|
||||
TELESCOPE_ENABLED=false
|
||||
PURGE_INACTIVE_USERS=false
|
||||
OTP_ENABLED=false
|
||||
CHECK_PASSWORDED_RARS=false
|
||||
DELETE_PASSWORDED_RELEASES=false
|
||||
DELETE_POSSIBLE_PASSWORDED_RELEASES=false
|
||||
EXTRACT_USING_RARINFO=false
|
||||
PRIVATE_PROFILES=true
|
||||
STORE_USER_IP=false
|
||||
FORUM_FRONTEND_ENABLED=false
|
||||
NNTMUX_ENV
|
||||
fi
|
||||
# Seed the scan loop run by metafeederusenet-scanner (trap #6). Unlike
|
||||
# nntmux.env above this is NOT guarded by `[ ! -f ]` and is NOT operator
|
||||
# config: it is shipped code, so a store update must be able to replace a
|
||||
# stale copy. Every tunable it reads lives in nntmux's `settings` table.
|
||||
# Quoted heredoc delimiter ('SCANNER_SH') — the script contains $, \ and
|
||||
# nested quotes that must reach the file verbatim, unexpanded.
|
||||
cat > /DATA/AppData/metafeederusenet/scanner/scanner.sh <<'SCANNER_SH'
|
||||
#!/bin/sh
|
||||
# nntmux scan launcher.
|
||||
#
|
||||
# The pyrowman/nntmux image runs frankenphp and NOTHING else: no cron, no
|
||||
# supervisord, no queue worker. So `settings.running=1` changes nothing on
|
||||
# its own, because nothing ever starts the loop. This is that loop.
|
||||
#
|
||||
# It runs as a SECOND CONTAINER off the same image + the same /app/.env and
|
||||
# nzb volume, rather than as a second process inside the web container, so
|
||||
# the web container's entrypoint (install/populate under `set -e`) is left
|
||||
# untouched.
|
||||
#
|
||||
# INVOCATION IS NOT FREE CHOICE. Measured PER RUNNER on a live box, NOT a
|
||||
# blanket "Forking works":
|
||||
# * multiprocessing/binaries.php — Forking OK.
|
||||
# * multiprocessing/backfill.php — NO-OP. Reports "...3 job(s) to do
|
||||
# using a max of 0 child process(es)" and returns in 0s having done
|
||||
# nothing. Backfill must be driven DIRECTLY, per group (see the
|
||||
# backfill block below).
|
||||
# * collation MUST go through tinker/ProcessReleases DIRECTLY.
|
||||
# multiprocessing/releases.php aborts every batch on a benign PHP
|
||||
# warning (ReleaseCleaning.php:165) that turns fatal inside the Spatie
|
||||
# child -> 0 releases while collections pile into the tens of
|
||||
# thousands.
|
||||
#
|
||||
# Every knob comes from nntmux's own `settings` table (edited at
|
||||
# /admin/tmux-edit), never from env — otherwise there are two sources of
|
||||
# truth for one number.
|
||||
set -u
|
||||
|
||||
cd /app || exit 1
|
||||
|
||||
setting() { php artisan tinker --execute="echo DB::table('settings')->where('name','$1')->value('value');" 2>/dev/null | tr -dc '0-9'; }
|
||||
set_setting() { php artisan tinker --execute="DB::table('settings')->where('name','$1')->update(['value'=>'$2']);" >/dev/null 2>&1; }
|
||||
groups() { php artisan tinker --execute="foreach (DB::table('usenet_groups')->where('active',1)->pluck('name') as \$g) echo \$g, PHP_EOL;" 2>/dev/null | grep -E '^[a-z0-9.]+$'; }
|
||||
|
||||
log() { echo "[scanner $(date -u +%H:%M:%S)] $*"; }
|
||||
|
||||
log "launcher started"
|
||||
while true; do
|
||||
if [ "$(setting running)" != "1" ]; then
|
||||
log "settings.running=0 — idle"
|
||||
sleep 30
|
||||
continue
|
||||
fi
|
||||
|
||||
if [ "$(setting binaries)" = "1" ]; then
|
||||
qty="$(setting backfill_qty)"; [ -n "$qty" ] || qty=20000
|
||||
log "header scan (qty=$qty)"
|
||||
php misc/update/multiprocessing/binaries.php "$qty" 2>&1 | tail -3
|
||||
fi
|
||||
|
||||
# Backfill must run DIRECT, per group. multiprocessing/backfill.php
|
||||
# reports "max of 0 child process(es)" and silently does nothing — the
|
||||
# Forking wrapper leaves maxProcesses=0 for this runner specifically.
|
||||
# Depth is capped by settings.backfill_days, so this reaches back that
|
||||
# far and then stops advancing; it is NOT a full-retention backfill.
|
||||
if [ "$(setting backfill)" = "1" ]; then
|
||||
# The ONLY setting this loop writes. `delaytime` (hours, default 2)
|
||||
# blocks ProcessReleases from advancing a collection until it ages
|
||||
# past it. Correct for the leading edge — a posting may still be
|
||||
# uploading — but pointless for backfill, where every article is
|
||||
# already days old. Left at 2 during backfill it stalls the catalog
|
||||
# for two hours with a failure mode indistinguishable from a broken
|
||||
# pipeline (collections climb, releases flatline, nothing logs an
|
||||
# error).
|
||||
#
|
||||
# Asserted every pass, not once at boot, because the nntmux
|
||||
# entrypoint RE-SEEDS `settings` on every container start (trap #7)
|
||||
# — a restart silently puts this back to 2 and the tier quietly
|
||||
# stops producing. Scoped to `backfill=1` so a pure forward-tailing
|
||||
# deployment (`backfill=0`) keeps the operator's own value: this is
|
||||
# a per-phase mode, not a preference we get to override.
|
||||
if [ "$(setting delaytime)" != "0" ]; then
|
||||
log "asserting delaytime=0 for the backfill phase (was $(setting delaytime))"
|
||||
set_setting delaytime 0
|
||||
fi
|
||||
qty="$(setting backfill_qty)"; [ -n "$qty" ] || qty=20000
|
||||
log "backfill direct (backfill_days=$(setting backfill_days), qty=$qty)"
|
||||
groups | while read -r g; do
|
||||
[ -n "$g" ] || continue
|
||||
php misc/update/backfill.php "$g" "$qty" 2>&1 | grep -iE "articles|backfill|Group|no new" | tail -2
|
||||
done
|
||||
fi
|
||||
|
||||
# Collation, per active group, DIRECT (never multiprocessing/releases.php).
|
||||
# Never wrap this in a short `timeout`: killing it between the release
|
||||
# INSERT and createNZBs() leaves a release row with no .nzb.gz, which
|
||||
# the feeder then correctly skips forever.
|
||||
groups | while read -r g; do
|
||||
[ -n "$g" ] || continue
|
||||
log "collate $g"
|
||||
php artisan tinker --execute="(new Blacklight\\processing\\ProcessReleases)->processReleases(1, 1, \"$g\", new Blacklight\\NNTP);" 2>&1 | grep -iE "releases (ready|created)|Added [0-9]+ releases|NZB" | tail -3
|
||||
done
|
||||
|
||||
delay="$(setting monitor_delay)"; [ -n "$delay" ] || delay=30
|
||||
log "pass complete; sleeping ${delay}s"
|
||||
sleep "$delay"
|
||||
done
|
||||
SCANNER_SH
|
||||
chmod 755 /DATA/AppData/metafeederusenet/scanner/scanner.sh
|
||||
|
||||
# Maison (Yundera dashboard) configuration. Read in preference to x-casaos.
|
||||
# schema_version 2 is required for `folders`: an older Maison refuses the app
|
||||
# outright rather than starting it without its directories.
|
||||
x-compose-app:
|
||||
schema_version: 2
|
||||
# Created and chowned by Maison before any image is pulled, before any hook
|
||||
# runs and before the containers start — on the first boot and every one after.
|
||||
# Compose would otherwise create a missing bind source as a root-owned dir.
|
||||
folders:
|
||||
- path: /DATA/AppData/metafeederusenet/db
|
||||
user: $PUID
|
||||
group: $PGID
|
||||
- path: /DATA/AppData/metafeederusenet/nntmux-env
|
||||
user: $PUID
|
||||
group: $PGID
|
||||
- path: /DATA/AppData/metafeederusenet/nntmux-storage/nzb
|
||||
user: $PUID
|
||||
group: $PGID
|
||||
- path: /DATA/AppData/metafeederusenet/feeder-state
|
||||
user: $PUID
|
||||
group: $PGID
|
||||
- path: /DATA/AppData/metafeederusenet/tmdb-cache
|
||||
user: $PUID
|
||||
group: $PGID
|
||||
- path: /DATA/AppData/metafeederusenet/scanner
|
||||
user: $PUID
|
||||
group: $PGID
|
||||
# Same string as the caddy_0 label, so the tile link and the route agree.
|
||||
webui-host: metafeederusenet-${domain}
|
||||
webui-path: "/config"
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 130 KiB |
@@ -0,0 +1,64 @@
|
||||
# MetaFeeder · Usenet — Rationale
|
||||
|
||||
## What deviation / exception is being requested
|
||||
|
||||
1. `metafeederusenet-feeder` runs `user: 0:0`.
|
||||
2. The **NNTmux web UI is published on its own hostname**
|
||||
(`metafeederusenetnntmux-<user>.<domain>`) with no AppShield sidecar in front of it — it
|
||||
is protected by NNTmux's own login only.
|
||||
3. The seeded `nntmux.env` contains TMDB and TVDB API keys as literals.
|
||||
|
||||
## Why it is necessary
|
||||
|
||||
1. The feeder publishes resolved records to the shared MetaCore and writes plugin artwork
|
||||
through its WebDAV endpoint; those targets are created and owned by MetaCore's
|
||||
root-running container.
|
||||
2. NNTmux is a Laravel application with its own user table, session handling and admin
|
||||
role. Its admin account is created at install from `$APP_DEFAULT_PASSWORD`, so
|
||||
authentication is **enabled by default** — the "app's own built-in auth" alternative the
|
||||
security checklist accepts. Fronting it with AppShield as well would double-prompt on
|
||||
every page and breaks its own OTP and password-reset flows.
|
||||
3. Those two keys are NNTmux's upstream project defaults, shipped in its own
|
||||
`.env.example`. They are shared read-only application keys for public metadata catalogs,
|
||||
not user credentials, and they are what makes the app work immediately after
|
||||
installation with no manual configuration. An operator can replace them from the NNTmux
|
||||
admin UI.
|
||||
|
||||
## Security mitigations in place
|
||||
|
||||
- The NNTmux admin password is `$APP_DEFAULT_PASSWORD` — the platform-generated secret —
|
||||
and the database password comes from the same variable. Neither is a hardcoded literal,
|
||||
and both are surfaced to the user in `tips.before_install`.
|
||||
- The feeder's own configuration UI *is* behind AppShield + Authelia SSO
|
||||
(`ghcr.io/yundera/appshield:2.0.9`).
|
||||
- `metafeederusenet-db`, `-redis` and `-manticore` are on the app-private
|
||||
`metafeederusenet` bridge network, carry no Caddy labels and publish no host port, so no
|
||||
sibling app on the shared `pcs` network can resolve or dial them.
|
||||
- NNTP provider credentials are deliberately left **blank** in the seed; the user fills
|
||||
them in per `tips.before_install`. Nothing ships with a working news account.
|
||||
- The `nntmux.env` seed is guarded by `[ ! -f … ]`, so a reinstall or version upgrade never
|
||||
overwrites the operator's provider settings. `scanner.sh` is intentionally rewritten on
|
||||
every run — it is shipped code, not operator config, so a store update must be able to
|
||||
replace a stale copy; the write is idempotent.
|
||||
- No user directories are mounted. All binds are under `/DATA/AppData/metafeederusenet/`,
|
||||
declared in `x-compose-app.folders`.
|
||||
- `cpu_shares` on all nine services; every image pinned (the NNTmux and Manticore images by
|
||||
digest).
|
||||
|
||||
## Alternatives considered and rejected
|
||||
|
||||
- **AppShield in front of NNTmux.** Two independent login gates on one UI; its OTP,
|
||||
password-reset and API endpoints break behind the OIDC redirect.
|
||||
- **Blank the TMDB/TVDB keys.** The app then installs into a state where covers and
|
||||
metadata silently fail until the user finds two upstream registration flows — a direct
|
||||
violation of "works immediately after installation, no manual configuration required for
|
||||
basic functionality".
|
||||
- **Run the feeder as `$PUID:$PGID`.** Writes into MetaCore's root-owned tree fail.
|
||||
|
||||
## Data protection
|
||||
|
||||
All state — MariaDB, the NZB store, the feeder cursor, the TMDB cache — is under
|
||||
`/DATA/AppData/metafeederusenet/`, the unit Maison archives on uninstall and restores from
|
||||
backup, so an uninstall/reinstall with "keep user data" comes back with the release
|
||||
catalogue intact rather than re-scanning Usenet from scratch. No user media directory is
|
||||
mounted into any service.
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 165 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 45 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 50 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 130 KiB |
Reference in New Issue
Block a user