Import the MetaMesh app family from Worph/AppStore

23 apps (MetaCore, the library/network/client tiers and 13 MetaFeeder*) move out of the general-purpose Worph/AppStore into their own store so the MetaMesh family releases on its own cadence. - asset URLs repointed to cdn.jsdelivr.net/gh/Worph/MetaAppStore@main - store scaffolding added (category/recommend/featured lists, licence, contributing, gitattributes) - psd-source/meta-icons-v2 icon generator follows the apps MetaMCP stays in Worph/AppStore: it is third-party metatool-ai software, not part of MetaMesh.
2026-09-13 16:18:29 +00:00
parent 998eba19f4
commit 39f82e1f96
145 changed files with 12964 additions and 1 deletions
@@ -0,0 +1,134 @@
+# MetaSort - MetaMesh Ecosystem
+
+MetaSort is part of the **MetaMesh ecosystem**, a suite of 4 interconnected apps that work together to organize and serve your media library.
+
+## The MetaMesh Ecosystem
+
+| App | Role | Port | Dependency |
+|-----|------|------|------------|
+| **MetaCore** | Leader election, Redis, WebDAV, Authelia SSO | 80, 9091 | None (install first) |
+| **MetaSort** | Metadata extraction, file indexing, plugin system | 80 | Requires MetaCore |
+| **MetaFuse** | Virtual filesystem (FUSE + WebDAV) | 80 | Requires MetaCore |
+| **MetaStremio** | Stremio addon with HLS transcoding | 7000 | Requires MetaCore |
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│                     User's Media Files                       │
+│              /DATA/Downloads, /DATA/Media                    │
+└──────────────────────────┬──────────────────────────────────┘
+                           │ (read-only bind mounts at /files/watch/*)
+                           ▼
+┌─────────────────────────────────────────────────────────────┐
+│                       MetaCore                               │
+│  • Leader election (flock)                                   │
+│  • Redis (metadata storage)                                  │
+│  • Authelia OIDC (single sign-on for all dashboards)         │
+│  • WebDAV file serving + dynamic mount management            │
+└──────────────────────────┬──────────────────────────────────┘
+                           │
+           ┌───────────────┼───────────────┐
+           ▼               ▼               ▼
+      /DATA/AppData/metacore/meta-core    pcs (Yundera platform network)
+      (bind mount)              (inter-container comms)
+           │                           │
+     ┌─────┴─────┐  ┌─────────────┐  ┌─────────────┐
+     ▼           ▼  ▼             ▼  ▼             ▼
+┌─────────┐ ┌─────────┐  ┌─────────────┐
+│MetaSort │ │MetaFuse │  │ MetaStremio │
+└─────────┘ └─────────┘  └─────────────┘
+```
+
+## Shared Infrastructure
+
+### MetaCore Directory (`/DATA/AppData/metacore/meta-core`)
+
+A **bind mount** shared by all MetaMesh apps. Contains:
+
+```
+/DATA/AppData/metacore/meta-core/
+├── db/redis/          # Redis database (metadata storage)
+├── locks/             # Leader election lock files (kv-leader.info)
+└── services/          # Service discovery JSON files
+```
+
+**Why a bind mount?**
+- Persists independently of Docker volume lifecycle
+- Predictable location for debugging/backup
+- Survives app reinstalls
+
+### User media directories — only mounted into MetaCore
+
+User media is bind-mounted **only into the MetaCore container**:
+
+```
+/files/                       (inside MetaCore container)
+└── watch/
+    ├── downloads/            # ← /DATA/Downloads (host bind, read-only)
+    └── media/                # ← /DATA/Media     (host bind, read-only)
+```
+
+**MetaSort, MetaFuse, and MetaStremio do not mount `/files` at all.** They reach files through MetaCore's WebDAV endpoint (`http://metacore/webdav/*`), discovering the URL at runtime from the leader-info file in `/meta-core/locks/kv-leader.info`.
+
+This matches the dev stack (Architecture V3) — MetaCore is the single source of truth for media files; siblings are stateless w.r.t. file storage.
+
+Earlier versions had two problems here:
+1. A `metamesh-files` named volume mounted at `/files` with the user's media bind-mounted as subpaths. `docker volume rm metamesh-files` then recursively traversed the bind-mount targets (which shared inodes with the host paths) and **deleted everything in `/DATA/Downloads` and `/DATA/Media`**.
+2. Even after dropping the named volume, every sibling app duplicated the bind mounts of `/DATA/Downloads` and `/DATA/Media` — multiplying the blast radius of any future bind-mount footgun and breaking the "MetaCore owns files" architectural promise.
+
+Mounting media only into MetaCore eliminates both issues.
+
+### Network (`pcs`)
+
+All MetaMesh containers attach to the Yundera platform network `pcs` (declared `external: true` everywhere). No app-private docker network — services discover each other by container hostname over `pcs`.
+
+## Installation Order
+
+**MetaCore must be installed first** — it provisions the Authelia secrets and OIDC clients that the other apps depend on.
+
+1. **Install MetaCore** → Creates Authelia secrets and OIDC clients (`metasort`, `metafuse`, `metacore`)
+2. **Install MetaSort, MetaFuse, MetaStremio** in any order
+
+Each app's `pre-install-cmd` checks for `/DATA/AppData/metacore/authelia/secrets/client-<app>.plain` and aborts with an error if MetaCore hasn't been installed.
+
+## Volume Permissions
+
+| Resource | MetaCore | MetaSort | MetaFuse | MetaStremio |
+|----------|----------|----------|----------|-------------|
+| `/DATA/AppData/metacore/meta-core` | rw | rw | rw | rw |
+| `/DATA/Downloads`, `/DATA/Media` | ro (mounts at `/files/watch/*`) | — (WebDAV) | — (WebDAV) | — (WebDAV) |
+| `/DATA/AppData/<app>/cache` | rw (own) | rw (own) | — | rw (own) |
+| `/DATA/AppData/metafuse/config` | — | — | rw | — |
+| `/DATA/MetaFuse/Library` | — | — | rw (FUSE mount) | — |
+| `/DATA/AppData/metacore/authelia/secrets` | rw (creates) | ro (oauth sidecar) | ro (oauth sidecar) | — |
+| `pcs` network (Yundera platform) | external | external | external | external |
+
+## Developer Notes
+
+### Adding a New MetaMesh App
+
+1. Add `pre-install-cmd` that checks for `/DATA/AppData/metacore/authelia/secrets/client-<app>.plain`
+2. Bind-mount `/DATA/AppData/metacore/meta-core:/meta-core:rw` for leader info / service registration
+3. **Do NOT bind-mount user media** — read files via MetaCore's WebDAV (`http://metacore/webdav/*`), URL discovered from `/meta-core/locks/kv-leader.info`
+4. Join the `pcs` network as `external: true` (Yundera platform-managed; do not declare an app-private network)
+5. Add an `oauth2-proxy` sidecar that reads its client secret from `/DATA/AppData/metacore/authelia/secrets/client-<app>.plain`
+6. Discover Redis at runtime by reading `/meta-core/locks/kv-leader.info`
+
+### Debugging
+
+```bash
+# Check Redis data (Redis runs in MetaCore)
+docker exec metacore redis-cli keys '*'
+
+# Check leader election
+cat /DATA/AppData/metacore/meta-core/locks/kv-leader.info
+
+# Check service discovery
+ls /DATA/AppData/metacore/meta-core/services/
+
+# Check what's mounted into a service
+docker exec metasort ls -la /files/watch/
+```
+
+### Testing Locally
+
+The MetaMesh dev environment (`meta-root-v2/dev/`) mirrors this architecture: direct bind mounts of `${DATA_WATCH_PATH}`, `./cache/test/files`, and `./cache/meta-sort/plugin-output` at `/files/watch`, `/files/test`, `/files/plugin` — no parent named volume.
@@ -0,0 +1,201 @@
+name: metasort
+
+services:
+  # Backend service. Reachable on the pcs network as `metasort-app`.
+  # Public ingress goes through the `metasort` hash-lock proxy below.
+  metasort-app:
+    image: ghcr.io/worph/meta-sort:1.0.3
+    container_name: metasort-app
+    hostname: metasort-app
+    restart: unless-stopped
+    cpu_shares: 20
+
+    expose:
+      - 80
+
+    volumes:
+      # Shared infrastructure state (leader election, Redis dump, service discovery)
+      # Owned by MetaCore; mounted rw here for service registration + leader reads.
+      - ${DATA_ROOT:-/DATA}/AppData/metacore/meta-core:/meta-core:rw
+
+      # Application cache storage
+      - ${DATA_ROOT:-/DATA}/AppData/metasort/cache:/data/cache
+
+      # Docker socket for container plugin management
+      - /var/run/docker.sock:/var/run/docker.sock:ro
+
+    # Note: no /files mounts — files are accessed via WebDAV from MetaCore
+    # (Architecture V3). MetaCore is the single source of truth for media files.
+
+    environment:
+      - NODE_ENV=production
+      - META_CORE_PATH=/meta-core
+      - CACHE_FOLDER_PATH=/data/cache
+      - SERVICE_NAME=meta-sort
+      - SERVICE_VERSION=1.0.0
+      - BASE_URL=https://metasort-$APP_DOMAIN
+      - TMDB_API_KEY=
+      - TMDB_LANGUAGE=en-US
+      - TZ=$TZ
+      # Container plugin configuration
+      - CONTAINER_CALLBACK_URL=http://metasort-app/api/plugins/callback
+      - CONTAINER_NETWORK=pcs
+
+    # Note: Runs as root for nginx, Redis, and file management
+    user: 0:0
+
+    networks:
+      - pcs
+
+  # OIDC perimeter for the MetaSort dashboard. Self-registers as an OIDC client
+  # with the platform's auth-registrar on first login (no secrets to manage).
+  #
+  # Naming: container_name MUST equal the URL-prefix of the public hostname so
+  # the registrar's redirect-URI allowlist accepts `metasort-<user>.<domain>`.
+  metasort:
+    image: ghcr.io/yundera/appshield:2.0.9
+    cpu_shares: 80
+    container_name: metasort
+    hostname: metasort
+    restart: unless-stopped
+    user: "0:0"
+
+    expose:
+      - 80
+
+    environment:
+      OIDC_REGISTRAR_URL: "http://auth-registrar:9092"
+      BACKEND_HOST: "metasort-app"
+      BACKEND_PORT: "80"
+      LISTEN_PORT: "80"
+      REDIRECT_HOST_SUFFIXES: "${APP_DOMAIN},${APP_PUBLIC_IP_DASH}.nip.io,${APP_PUBLIC_IP_DASH}.sslip.io"
+
+    depends_on:
+      - metasort-app
+
+    labels:
+      # Only /health bypasses hash-lock — internal cluster traffic (plugin
+      # callbacks, /meta/*, /webdav/*, /file/cid) reaches metasort-app via
+      # the pcs network and never traverses the public proxy, so no public
+      # bypass is required for those paths.
+
+      # ---- metasort-${APP_DOMAIN} (gateway TLS) ----
+      caddy_0: metasort-${APP_DOMAIN}
+      caddy_0.import: gateway_tls
+      caddy_0.0_handle: "/health"
+      caddy_0.0_handle.reverse_proxy: "metasort-app:80"
+      caddy_0.1_handle: ""
+      caddy_0.1_handle.reverse_proxy: "{{upstreams 80}}"
+
+      # ---- metasort-IP.nip.io (gateway TLS) ----
+      caddy_1: metasort-${APP_PUBLIC_IP_DASH}.nip.io
+      caddy_1.import: gateway_tls
+      caddy_1.0_handle: "/health"
+      caddy_1.0_handle.reverse_proxy: "metasort-app:80"
+      caddy_1.1_handle: ""
+      caddy_1.1_handle.reverse_proxy: "{{upstreams 80}}"
+
+      # ---- metasort-IP.sslip.io (Let's Encrypt) ----
+      caddy_2: metasort-${APP_PUBLIC_IP_DASH}.sslip.io
+      caddy_2.0_handle: "/health"
+      caddy_2.0_handle.reverse_proxy: "metasort-app:80"
+      caddy_2.1_handle: ""
+      caddy_2.1_handle.reverse_proxy: "{{upstreams 80}}"
+
+    networks:
+      - pcs
+
+networks:
+  pcs:
+    name: ${APP_NET:-pcs}
+    external: true
+
+x-casaos:
+  architectures:
+    - amd64
+    - arm64
+  author: Worph Team
+  category: Media
+  developer: Aptero Team
+
+  description:
+    en_us: |
+      **Smart metadata extraction and file indexing for massive media libraries.**
+
+      MetaSort is the brain of the MetaMesh ecosystem. It watches your media folders, extracts metadata using intelligent plugins, and stores everything in a distributed database for other services to use.
+
+      **Key Features:**
+      - **Plugin-Based Processing** - Containerized plugins for FFmpeg analysis, TMDB enrichment, filename parsing, and more
+      - **Smart Metadata Extraction** - Automatically extracts titles, seasons, episodes, quality, language, codecs
+      - **Duplicate Detection** - Finds exact duplicates and different quality versions of the same content
+      - **Real-time Monitoring** - Watches folders and instantly processes new files
+      - **Built for Scale** - Handles collections up to 900k+ files and 100TB+
+      - **Leader Election** - Distributed architecture with automatic failover
+
+      **Requires:** MetaCore (install first)
+
+      **Works with:** MetaFuse (virtual filesystem), MetaStremio (streaming)
+
+      **Dashboard Features:** File monitoring, plugin status, duplicate detection, metadata browser, processing queue
+
+    fr_fr: |
+      **Extraction intelligente de metadonnees et indexation de fichiers pour d'immenses bibliotheques multimedia.**
+
+      MetaSort est le cerveau de l'ecosysteme MetaMesh. Il surveille vos dossiers multimedia, extrait les metadonnees a l'aide de plugins intelligents et stocke tout dans une base de donnees distribuee.
+
+      **Fonctionnalites Cles:**
+      - **Traitement par Plugins** - Plugins conteneurises pour analyse FFmpeg, enrichissement TMDB, parsing de noms de fichiers
+      - **Extraction Intelligente** - Extrait automatiquement titres, saisons, episodes, qualite, langue, codecs
+      - **Detection de Doublons** - Trouve les doublons exacts et differentes versions de qualite
+      - **Surveillance Temps Reel** - Surveille les dossiers et traite instantanement les nouveaux fichiers
+      - **Concu pour l'Echelle** - Gere des collections jusqu'a 900k+ fichiers et 100TB+
+
+      **Necessite:** MetaCore (installer en premier)
+
+      **Fonctionne avec:** MetaFuse (systeme de fichiers virtuel), MetaStremio (streaming)
+
+  tagline:
+    en_us: Smart metadata extraction engine - Index and organize massive media libraries with intelligent plugins
+    fr_fr: Moteur d'extraction de metadonnees - Indexez et organisez d'immenses bibliotheques avec des plugins intelligents
+
+  title:
+    en_us: MetaSort
+
+  icon: https://cdn.jsdelivr.net/gh/Worph/MetaAppStore@main/Apps/MetaSort/icon.png
+  screenshot_link:
+    - https://cdn.jsdelivr.net/gh/Worph/MetaAppStore@main/Apps/MetaSort/screenshot-1.png
+    - https://cdn.jsdelivr.net/gh/Worph/MetaAppStore@main/Apps/MetaSort/screenshot-2.png
+    - https://cdn.jsdelivr.net/gh/Worph/MetaAppStore@main/Apps/MetaSort/screenshot-3.png
+  thumbnail: https://cdn.jsdelivr.net/gh/Worph/MetaAppStore@main/Apps/MetaSort/thumbnail.png
+
+  index: /
+  store_app_id: metasort
+  main: metasort
+  is_uncontrolled: false
+
+  tips:
+    before_install:
+      en_us: |
+        **Requires MetaCore** - Install MetaCore first.
+
+        MetaSort connects to MetaCore for metadata storage. The dashboard is protected by
+        Yundera's built-in Authelia via OIDC — sign in with your existing Yundera account.
+
+# Maison (Yundera dashboard) configuration. Read in preference to x-casaos.
+# schema_version 2 is required for `folders`: an older Maison refuses the app
+# outright rather than starting it without its directories.
+x-compose-app:
+  schema_version: 2
+  # Created and chowned by Maison before any image is pulled, before any hook
+  # runs and before the containers start — on the first boot and every one after.
+  # Compose would otherwise create a missing bind source as a root-owned dir.
+  folders:
+    - path: /DATA/AppData/metacore/meta-core
+      user: $PUID
+      group: $PGID
+    - path: /DATA/AppData/metasort/cache
+      user: $PUID
+      group: $PGID
+  # Same string as the caddy_0 label, so the tile link and the route agree.
+  webui-host: metasort-${domain}
+  webui-path: "/"
@@ -0,0 +1,48 @@
+# MetaSort — Rationale
+
+## What deviation / exception is being requested
+
+1. `metasort-app` runs `user: 0:0`.
+2. It mounts the Docker socket, `/var/run/docker.sock`, read-only.
+3. It mounts `/DATA/AppData/metacore/meta-core` — another app's AppData directory.
+
+## Why it is necessary
+
+1. MetaSort writes its service registration into `/meta-core/services/`, a directory
+   created and owned by MetaCore's root-running container, and runs nginx internally.
+2. MetaSort's enrichment plugins are **containers**. It lists, starts and stops them on the
+   `pcs` network (`CONTAINER_NETWORK=pcs`), and receives their results on
+   `CONTAINER_CALLBACK_URL`. Driving container lifecycle requires the Docker API.
+3. `/meta-core` is the shared leader-election and service-discovery directory. MetaCore is
+   a hard prerequisite of this app.
+
+## Security mitigations in place
+
+- **The socket is mounted `:ro`.** That limits the *bind*, not the API — a read-only socket
+  bind still permits write calls — so the meaningful control is the perimeter: the
+  container carries no Caddy labels, publishes no host port, and is reachable only from
+  the `pcs` network.
+- Public ingress terminates at the AppShield sidecar (`ghcr.io/yundera/appshield:2.0.9`)
+  behind Authelia SSO; only `/health` is exempt.
+- MetaSort's own state is confined to `/DATA/AppData/metasort/cache/`, declared in
+  `x-compose-app.folders`.
+- `cpu_shares` set on both services; image tags pinned.
+
+## Alternatives considered and rejected
+
+- **A Docker-socket proxy (e.g. tecnativa/docker-socket-proxy) allowing only the endpoints
+  MetaSort uses.** This is the right long-term shape and is the intended follow-up. It is
+  not shipped here because plugin lifecycle needs `containers`, `images`, `exec` and
+  `networks` write access, which is most of the API surface anyway; the proxy would narrow
+  the blast radius without eliminating it.
+- **Run plugins as sibling compose services instead of on-demand containers.** Plugins are
+  installed and removed by the user at runtime; a static compose file cannot express that.
+- **Run as `$PUID:$PGID`.** Fails writing into MetaCore's root-owned
+  `/meta-core/services/`.
+
+## Data protection
+
+MetaSort holds no user media — it reads metadata through MetaCore and caches derived data
+under `/DATA/AppData/metasort/cache/`, which Maison archives and restores. Its access to the
+user's files is indirect, through MetaCore's WebDAV endpoint, and is bounded by what
+MetaCore itself exposes (read-only mounts of `/DATA/Downloads` and `/DATA/Media`).