README: sync with current code

README.md

@@ -3,17 +3,18 @@
 File-watcher and metadata-extraction service for the MetaMesh ecosystem.
 meta-sort consumes file events from meta-core (over SSE), drives metadata
 through a streaming pipeline + container plugins, and writes results back
-to Redis through meta-core's API. It exposes a Fastify REST API and serves
+to Redis through meta-core's HTTP API. It exposes a Fastify REST API and
-the `/files` volume over WebDAV so that other services and plugin
+reads media exclusively over **meta-core's WebDAV** — it mounts neither
-containers can read media without mounting the same volume.
+`/meta-core` nor `/files`.
 
 > **Service role:** meta-sort is **Redis-consuming**, not Redis-managing.
-> It does not run leader election, it does not spawn Redis, and it does
+> It does not run leader election, it does not spawn Redis, and it has
-> not own `kv-leader.lock`. Those responsibilities belong to **meta-core**
+> no direct Redis connection: every read and write goes through
-> (a separate Go service). meta-sort reads
+> **meta-core** (a separate Go service) over HTTP. meta-sort locates
-> `/meta-core/locks/kv-leader.info` via `LeaderClient` to discover the
+> meta-core over UDP multicast (**meta-discovery v1**,
-> Redis URL, API URL, and WebDAV URLs, and registers its own URLs into
+> `239.255.77.1:9399`) — meta-core's announce carries its API and WebDAV
-> `/meta-core/services/<hostname>.json`.
+> URLs — and announces itself on the same group so sibling dashboards can
+> list it. `META_CORE_URL` pins meta-core and disables wire selection.
 
 ---
 
@@ -29,14 +30,18 @@ What this service does:
 3. **Manages container plugins**: spawns plugin containers via the Docker
    socket, exposes a callback endpoint, lets plugins write metadata back
    into Redis via the meta-core-compatible `/meta/*` API on this service.
-4. **Serves files over WebDAV** at `/webdav/*` so container plugins and
+4. **Computes file identity**: `midhash256` in the light phase, full-file
-   sibling services (meta-fuse, meta-stremio) can read media without
+   digests (sha1, sha256, md5, sha3, crc32, btih v2) in the background
-   needing the underlying bind/SMB/rclone mounts.
+   phase. Digests are persisted only as the bare-CID key-set
+   (`cids/<cid>`), never as per-algorithm `cid_*` fields
+   (`src/config/TargetHash.ts`, `src/kv/MetaCoreApiWriter.ts`).
 
 What this service does **not** do (anymore):
 
 - It does not run leader election.
-- It does not spawn or supervise Redis.
+- It does not spawn or supervise Redis, nor talk to it directly.
+- It does not serve WebDAV — files are read from meta-core's WebDAV, and
+  plugin containers get that same URL (`WEBDAV_URL`).
 - It does not watch the filesystem directly (chokidar lives in meta-core).
 - It does not manage remote mounts (mounts are now in meta-core / rclone-smb).
 - It does not have an in-process plugin runtime — all plugins are
@@ -48,43 +53,46 @@ What this service does **not** do (anymore):
 
 ## Architecture
 
-### Two-volume layout
+### Volumes
 
-meta-sort consumes two host-side volumes via standard bind mounts:
+meta-sort needs **no shared volume**. Media lives on meta-core's
+`FILES_VOLUME` and is read over meta-core's WebDAV; file paths stored in
+Redis are **relative to `/files`**, which keeps storage portable across
+hosts. The only mounts are its own cache (`/data/cache`), the Docker
+socket (for plugin containers) and `plugins.yml`.
 
-| Volume | Container path | Owner | Purpose |
+### Locating meta-core (meta-discovery v1)
-|--------|----------------|-------|---------|
-| `META_CORE_VOLUME` | `/meta-core` | meta-core writes, all services read | Leader-info file, Redis data, service registry, SSE cursor |
-| `FILES_VOLUME`     | `/files`     | meta-sort serves over WebDAV       | Watched media, plugin output, dynamic mounts |
-
-File paths stored in Redis are **relative to `/files`**, which makes the
-storage portable across hosts. See the root `CLAUDE.md` for the
-authoritative volume layout description.
-
-### Leader discovery (read-only)
 
 ```
 meta-core (separate container)
-   ├── runs Redis
+   ├── runs Redis, the file watcher, and the WebDAV server for /files
-   ├── runs file watcher (chokidar) and publishes file:events
+   ├── HTTP API: /urls, /api/events/files (SSE), /meta/*, ...
-   ├── runs HTTP API on /api/urls, /api/events/files, /meta/*, /api/services, ...
+   └── announces role=core on 239.255.77.1:9399 (UDP multicast),
-   └── writes /meta-core/locks/kv-leader.info
+       carrying its /urls payload (apiUrl, webdavUrl, webdavUrlInternal)
                        │
                        ▼
               meta-sort (this service)
-              ├── LeaderClient reads kv-leader.info → Redis URL, API URL,
+              ├── LeaderClient → MetaCoreLocator (src/discovery/meshdisco.ts)
-              │   WebDAV URLs, etc.
+              │   picks up the core announce (or GET {META_CORE_URL}/urls
-              ├── KVManager creates RedisKVClient (writes are routed
+              │   when pinned) and announces meta-sort on the same group
-              │   through meta-core's HTTP API for the api-mediated-access
+              ├── KVManager builds an HTTP-only RedisKVClient against
-              │   model)
+              │   meta-core's API and configures the WebDAV client
-              ├── ServiceRegistration writes
+              ├── GET /api/neighbors serves the UDP neighbour map (nav menu)
-              │   /meta-core/services/meta-sort-<hostname>.json
               └── FileEventConsumer subscribes to
                   ${apiUrl}/api/events/files (SSE)
 ```
 
-Source: `packages/meta-sort-core/src/kv/{KVManager,LeaderClient,RedisClient,ServiceRegistration}.ts`,
+Source: `packages/meta-sort-core/src/discovery/meshdisco.ts`,
+`packages/meta-sort-core/src/kv/{KVManager,LeaderClient,RedisClient,MetaCoreApiWriter}.ts`,
 `packages/meta-sort-core/src/events/{FileEventConsumer,SSEEventClient}.ts`.
+The protocol spec is `docs/project-architecture/service-discovery.md` in
+the meta-root. The `meshdisco.ts` port is mirrored across services and
+guarded by the meta-root's `scripts/check-mirrors.sh` — change it there in
+lockstep.
+
+The SSE cursor is persisted to `/meta-core/cursors/meta-sort-files.cursor`
+(`FileEventConsumer.ts` default); since `/meta-core` is no longer mounted,
+that path is container-local and does not survive a container recreate.
 
 ### Processing pipeline
 
@@ -119,7 +127,9 @@ them via the Docker socket and a callback endpoint:
 - **Built-in plugins** (each is a separate git submodule under
   `packages/plugins/metamesh-plugin-*` at the repo root): `file-info`,
   `ffmpeg`, `filename-parser`, `jellyfin-nfo`, `tmdb`, `anime-detector`,
-  `language`, `subtitle`, `subtitle-extractor`, `torrent`, `fullhash`.
+  `language`, `subtitle`, `subtitle-extractor`, `still-extractor`,
+  `opensubtitle`, `torrent`, `fullhash`. Which ones actually run is
+  decided by `plugins.yml` — the image ships none.
 - **Dispatch:** `ContainerPluginScheduler.dispatchAllPlugins(hashId,
   filePath, metaFlat)` — fire-and-forget. The scheduler keeps its own
   fast / background queues based on each plugin manifest's
@@ -128,8 +138,10 @@ them via the Docker socket and a callback endpoint:
   `/api/plugins/callback`, and they can read/write Redis metadata
   through the meta-core-compatible `/meta/:hash` routes that meta-sort
   also exposes (so plugins only need one HTTP target).
-- **Plugins access files** by hitting meta-sort's WebDAV
+- **Plugins access files** over meta-core's WebDAV: each plugin
-  (`http://metasort-app/webdav/...` inside the docker network).
+  container is started with `WEBDAV_URL` (meta-core's
+  `webdavUrlInternal`), `META_CORE_URL` (= `CONTAINER_META_CORE_URL`),
+  `CALLBACK_URL` and `PLUGIN_ID` (`ContainerManager.ts`).
 
 For the plugin HTTP contract, see
 [`docs/containerized-plugin-architecture.md`](docs/containerized-plugin-architecture.md)
@@ -143,15 +155,16 @@ and [`docs/plugin-task-queue-architecture.md`](docs/plugin-task-queue-architectu
 
 | Variable | Default | Description |
 |----------|---------|-------------|
-| `META_CORE_PATH` | `/meta-core` | Mount of META_CORE_VOLUME. Required — startup fails fast if absent. |
+| `META_CORE_URL` | — | Pins meta-core's API URL; UDP discovery then never overrides it. |
-| `FILES_PATH` | `/files` | Mount of FILES_VOLUME. |
+| `ENABLE_UDP_DISCOVERY` | on | `false`/`0` disables the meta-discovery v1 announce/listen. |
+| `ALLOW_LEGACY_REDIS_URL` | — | `1` downgrades the "meta-core still publishes `redisUrl`" startup error to a warning. |
+| `FILES_PATH` | `/files` | Virtual root that relative paths resolve under (on meta-core's WebDAV). |
 | `CACHE_FOLDER_PATH` | `/data/cache` | Local cache root. |
-| `SERVICE_NAME` | `meta-sort` | Identifier used for service registration. |
+| `SERVICE_NAME` | `meta-sort` | Name announced on the discovery group. |
-| `SERVICE_VERSION` | `1.0.0` | Recorded in the service registry. |
 | `FUSE_API_PORT` | `3000` | HTTP API port (legacy name; this is the Fastify API). |
 | `FUSE_API_HOST` | `0.0.0.0` | API bind host. |
-| `BASE_URL` | — | External base URL recorded in service registration. |
+| `PUBLIC_URL` | — | Browser-facing URL announced for neighbours' nav menus. Wins over `BASE_URL` (use it for a debug-direct port when no Caddy is in front). |
-| `ADVERTISE_HOST` | — | Hostname to advertise to other services. Falls back to auto-detected IP. |
+| `BASE_URL` | — | Perimeter URL announced when `PUBLIC_URL` is unset; falls back to `http://<hostname>:<FUSE_API_PORT>`. |
 | `MAX_WORKER_THREADS` | `os.cpus().length` | If set, overrides all pipeline concurrencies to `(M*2, M, M)`. |
 | `FAST_QUEUE_CONCURRENCY` | `32` | Container-plugin fast queue concurrency (see plugin-task-queue-architecture.md). |
 | `FILE_LIGHT_SLOTS` | `16` | Max files concurrently in the light phase. |
@@ -160,16 +173,21 @@ and [`docs/plugin-task-queue-architecture.md`](docs/plugin-task-queue-architectu
 | `CONTAINER_PLUGINS_CONFIG` | `/app/plugins.yml` | Path to plugin manifest YAML. If missing, container plugins are skipped. |
 | `DOCKER_SOCKET_PATH` | `/var/run/docker.sock` | Docker socket for container management. |
 | `CONTAINER_CALLBACK_URL` | `http://meta-sort:8180` | URL plugins POST callbacks to. |
-| `CONTAINER_META_CORE_URL` | `http://meta-sort` | URL plugins use for meta-core-compatible `/meta/*` routes (served by meta-sort itself for now). |
+| `CONTAINER_META_CORE_URL` | `http://meta-sort` | URL plugins use for meta-core-compatible `/meta/*` routes (served by meta-sort itself). The default host doesn't resolve on current stacks — dev sets `http://metasort-app` (port 80 dev nginx proxies `/meta/`), the store app `http://metasort-app:3000` (the stock nginx has no `/meta/` location). |
 | `CONTAINER_NETWORK` | `meta-network` | Docker network to attach plugin containers to. |
 | `PLUGIN_STACK_NAME` | — | Docker Compose project label so plugin containers group together in Docker Desktop. |
 | `PLUGIN_CACHE_HOST_PATH` | — | Host path bind-mounted into each plugin for persistent caches. |
 
-Source of truth: `packages/meta-sort-core/src/config/EnvConfig.ts`.
+Source of truth: `packages/meta-sort-core/src/config/EnvConfig.ts` (plus
+`PUBLIC_URL` in `kv/KVManager.ts` and `ENABLE_UDP_DISCOVERY` /
+`ALLOW_LEGACY_REDIS_URL` in `kv/LeaderClient.ts`). `META_CORE_PATH`,
+`REDIS_URL`, `ADVERTISE_HOST` and `SERVICE_VERSION` are still parsed by
+`EnvConfig.ts` but nothing acts on them any more. The discovery group is
+fixed at the protocol default (`239.255.77.1:9399`, 10 s interval).
 
-The Redis URL and WebDAV URLs are **not** configured via environment
+The WebDAV URLs are **not** configured via environment variables —
-variables — meta-sort discovers them from `kv-leader.info` (or its
+meta-sort takes them from meta-core's announce (or `GET /urls` when
-in-memory equivalent fetched through meta-core's `/api/urls`).
+pinned).
 
 ### Concurrency defaults
 
@@ -239,7 +257,9 @@ Defined in `packages/meta-sort-core/src/api/UnifiedAPIServer.ts`.
 | `POST /api/plugins/callback` | Container plugin task-complete callback (used by plugins, not humans). |
 | `GET /api/plugins/containers` | Container-plugin status (health, instances, queue depths). |
 | `GET /api/plugins/containers/:pluginId/manifest` | Plugin manifest. |
+| `GET /api/plugins/containers/:pluginId/logs` | Container logs (`?instance=`, `?tail=`). |
 | `POST /api/plugins/containers/:pluginId/restart` | Restart instances. |
+| `POST /api/plugins/containers/:pluginId/stop` / `start` | Stop / start instances. |
 | `POST /api/plugins/containers/restart-all` | Restart all. |
 | `POST /api/plugins/containers` | Add a new plugin (image, instances, resources, config, defaultQueue). |
 | `PUT /api/plugins/containers/:pluginId` | Update plugin spec. |
@@ -249,9 +269,8 @@ Defined in `packages/meta-sort-core/src/api/UnifiedAPIServer.ts`.
 
 | Endpoint | Description |
 |----------|-------------|
-| `GET /api/file/download?path=...` | Download a file via the configured WebDAV backend. |
+| `GET /api/file/download?path=...` | Download a file via meta-core's WebDAV. |
 | `POST /file/cid` | Compute the CID (midhash256) for a path (`{ path }`). |
-| `GET /webdav/*` (and `PUT`, `DELETE`, `MKCOL`, `PROPFIND`) | WebDAV view of `/files`. Unauthenticated. |
 
 ### meta-core-compatible KV (for container plugins)
 
@@ -272,8 +291,8 @@ single host. They are only registered when the KV client is up.
 | Endpoint | Description |
 |----------|-------------|
 | `GET /api/metrics` | Performance metrics. |
-| `GET /api/stats` | Redis stats summary (only when KV is up). |
+| `GET /api/stats` | File count + total size from meta-core's `/api/files/tuples` summary, cached stale-while-revalidate. |
-| `GET /api/services` | Proxy to meta-core's `/api/services` via the LeaderClient. |
+| `GET /api/neighbors` | meta-discovery v1 neighbour map (nav menu). Served locally — works while meta-core is down. Replaces the removed `/api/services`. |
 
 ---
 
@@ -286,6 +305,18 @@ single host. They are only registered when the KV client is up.
 - Debug-direct backend (no auth, no TLS): `http://localhost:18180`
 - Container name (backend): `metasort-app`
 - Container name (hash-lock proxy): `metasort`
+- CasaOS store app: `MetaSort` in the meta-root's `packages/MetaAppStore`
+  (`ghcr.io/worph/meta-sort`, fronted by an `appshield` perimeter).
+
+### Docker image
+
+The repo `Dockerfile` (published to `ghcr.io/worph/meta-sort` by
+`.github/workflows/docker-publish.yml`) is an all-in-one image: nginx on
+`:80` → Fastify on `:3000`, rclone, and a bundled `meta-core` binary
+copied from `ghcr.io/worph/meta-core` and started by
+`docker/supervisord.conf`. Deployments run meta-core as its own app, so
+both the dev stack and the store app mount their own supervisord/nginx
+configs that drop the bundled meta-core.
 
 ### Logs
 
@@ -310,7 +341,8 @@ cd dev
 ### Running tests
 
 ```bash
-docker exec meta-test-runner /app/test/test.sh sort
+docker exec meta-test-runner /app/test/test.sh sort   # E2E (meta-root dev stack)
+pnpm test                                             # unit tests (@meta-sort/core, mocha)
 ```
 
 See the root `CLAUDE.md` "Running Tests" section for the full matrix.
@@ -336,9 +368,10 @@ packages/meta-sort/
 │   │   │   ├── api/              # UnifiedAPIServer.ts (Fastify)
 │   │   │   ├── config/           # EnvConfig.ts, SupportedFileTypes.ts
 │   │   │   ├── container-plugins/# ContainerManager, ContainerPluginScheduler, ...
+│   │   │   ├── discovery/        # meshdisco.ts (meta-discovery v1, mirrored port)
 │   │   │   ├── events/           # FileEventConsumer, SSEEventClient
 │   │   │   ├── jellyfin/         # Jellyfin / NFO output helpers
-│   │   │   ├── kv/               # KVManager, LeaderClient, RedisClient
+│   │   │   ├── kv/               # KVManager, LeaderClient, RedisClient, MetaCoreApiWriter
 │   │   │   ├── logic/            # StreamingPipeline, state managers, fileProcessor
 │   │   │   ├── metrics/          # PerformanceMetrics
 │   │   │   ├── plugin-engine/    # Vestigial in-process types + TaskScheduler glue
@@ -350,27 +383,28 @@ packages/meta-sort/
 │   ├── meta-sort-ui/             # @meta-sort/ui - React dashboard (Vite)
 │   ├── meta-sort-editor/         # Metadata editor UI (Vite)
 │   ├── async-utils/              # @worph/async-utils - MultiQueue, etc.
-│   └── shared/                   # Symlinks to filename-tool, meta-hash, meta-interface
+│   └── shared/                   # Vendored copies of filename-tool, meta-hash, meta-interface
 ├── pnpm-workspace.yaml
 └── pnpm-lock.yaml
 ```
 
 > Built-in container plugins **do not live here.** They live at the
 > repo root as separate submodules:
-> `packages/plugins/metamesh-plugin-{file-info,ffmpeg,filename-parser,jellyfin-nfo,tmdb,anime-detector,language,subtitle,subtitle-extractor,torrent,fullhash}`.
+> `packages/plugins/metamesh-plugin-{file-info,ffmpeg,filename-parser,jellyfin-nfo,tmdb,anime-detector,language,subtitle,subtitle-extractor,still-extractor,opensubtitle,torrent,fullhash}`.
 
 ### Key components
 
 | Component | File | Purpose |
 |-----------|------|---------|
 | Entry point | `src/index.ts` | Wires KVManager → API server → plugin manager → ContainerManager → StreamingPipeline → FileEventConsumer. |
+| Discovery | `src/discovery/meshdisco.ts` | meta-discovery v1 node + `MetaCoreLocator`. |
 | Streaming pipeline | `src/logic/pipeline/StreamingPipeline.ts` | 3-queue (validation / fast / background) orchestrator. |
 | State manager | `src/logic/UnifiedProcessingStateManager.ts` | Tracks `discovered → lightProcessing → hashProcessing → done`. |
 | File processor | `src/logic/WatchedFileProcessor.ts` | Light + hash phase implementation. |
 | SSE consumer | `src/events/FileEventConsumer.ts` | Translates meta-core events into pipeline calls. |
 | SSE transport | `src/events/SSEEventClient.ts` | Long-lived HTTP SSE with cursor persistence. |
-| KV manager | `src/kv/KVManager.ts` | Bootstraps RedisKVClient + ServiceRegistration from LeaderClient. |
+| KV manager | `src/kv/KVManager.ts` | Bootstraps the HTTP-only RedisKVClient + WebDAV client from LeaderClient. |
-| Leader client | `src/kv/LeaderClient.ts` | Reads `/meta-core/locks/kv-leader.info`, fetches `/api/urls`. |
+| Leader client | `src/kv/LeaderClient.ts` | Legacy name: thin adapter over `MetaCoreLocator` (UDP), or `GET /urls` when `META_CORE_URL` is pinned. |
 | Container manager | `src/container-plugins/ContainerManager.ts` | Docker socket orchestration. |
 | Container scheduler | `src/container-plugins/ContainerPluginScheduler.ts` | Plugin fast/background queues + callbacks. |
 | API server | `src/api/UnifiedAPIServer.ts` | Fastify REST + WebDAV-facing meta routes. |
@@ -393,8 +427,9 @@ Repo-level docs of interest:
 - Root `CLAUDE.md` — authoritative ground truth on the dev stack, ports,
   containers, and operational commands.
 - `/METADATA_KEYS.md` — schema for the `/file/{cid}` Redis hash.
-- `packages/meta-core/docs/` — leader election and Redis storage owner.
+- `packages/meta-core/docs/` — Redis storage owner, api-mediated access.
+- `docs/project-architecture/service-discovery.md` — meta-discovery v1 spec.
 
 ## License
 
-MIT — see LICENSE.
+MIT (see `package.json`).