File-watcher and metadata-extraction service for the MetaMesh ecosystem. meta-sort consumes file events from meta-core (over SSE), drives metadata through a streaming pipeline + container plugins, and writes results back to Redis through meta-core's API. It exposes a Fastify REST API and serves the /files volume over WebDAV so that other services and plugin containers can read media without mounting the same volume.

Service role: meta-sort is Redis-consuming, not Redis-managing. It does not run leader election, it does not spawn Redis, and it does not own kv-leader.lock. Those responsibilities belong to meta-core (a separate Go service). meta-sort reads /meta-core/locks/kv-leader.info via LeaderClient to discover the Redis URL, API URL, and WebDAV URLs, and registers its own URLs into /meta-core/services/<hostname>.json.

Overview

What this service does:

Subscribes to meta-core's /api/events/files SSE stream (file discovery events: add, change, delete, rename, reset).
Drives the streaming pipeline (StreamingPipeline): extension filter → light phase (filename parse, basic stats, midhash256) → write to Redis → dispatch container plugins → background phase (SHA-256).
Manages container plugins: spawns plugin containers via the Docker socket, exposes a callback endpoint, lets plugins write metadata back into Redis via the meta-core-compatible /meta/* API on this service.
Serves files over WebDAV at /webdav/* so container plugins and sibling services (meta-fuse, meta-stremio) can read media without needing the underlying bind/SMB/rclone mounts.

What this service does not do (anymore):

It does not run leader election.
It does not spawn or supervise Redis.
It does not watch the filesystem directly (chokidar lives in meta-core).
It does not manage remote mounts (mounts are now in meta-core / rclone-smb).
It does not have an in-process plugin runtime — all plugins are containerized.
/api/scan/trigger and /api/metadata/clear no longer live here; use meta-core's endpoints instead.

Architecture

Two-volume layout

meta-sort consumes two host-side volumes via standard bind mounts:

Volume	Container path	Owner	Purpose
`META_CORE_VOLUME`	`/meta-core`	meta-core writes, all services read	Leader-info file, Redis data, service registry, SSE cursor
`FILES_VOLUME`	`/files`	meta-sort serves over WebDAV	Watched media, plugin output, dynamic mounts

File paths stored in Redis are relative to /files, which makes the storage portable across hosts. See the root CLAUDE.md for the authoritative volume layout description.

Leader discovery (read-only)

meta-core (separate container)
   ├── runs Redis
   ├── runs file watcher (chokidar) and publishes file:events
   ├── runs HTTP API on /api/urls, /api/events/files, /meta/*, /api/services, ...
   └── writes /meta-core/locks/kv-leader.info
                       │
                       ▼
              meta-sort (this service)
              ├── LeaderClient reads kv-leader.info → Redis URL, API URL,
              │   WebDAV URLs, etc.
              ├── KVManager creates RedisKVClient (writes are routed
              │   through meta-core's HTTP API for the api-mediated-access
              │   model)
              ├── ServiceRegistration writes
              │   /meta-core/services/meta-sort-<hostname>.json
              └── FileEventConsumer subscribes to
                  ${apiUrl}/api/events/files (SSE)

Source: packages/meta-sort-core/src/kv/{KVManager,LeaderClient,RedisClient,ServiceRegistration}.ts, packages/meta-sort-core/src/events/{FileEventConsumer,SSEEventClient}.ts.

Processing pipeline

meta-core SSE  ─►  FileEventConsumer  ─►  StreamingPipeline
                                                │
                                                ├─ validationQueue (extension filter)
                                                │
                                                ├─ fastQueue
                                                │     processLightPhase:
                                                │       midhash256 (permanent ID)
                                                │       basic metadata
                                                │       write to Redis  ───►  file visible in VFS
                                                │       dispatchAllPlugins (containers)
                                                │
                                                └─ backgroundQueue
                                                      processHashPhase:
                                                        SHA-256, heavier hashes
                                                      await ContainerPluginScheduler
                                                      file:complete  ───►  DONE

midhash256 is the permanent file identifier — there is no tempId and no rename step when SHA-256 finishes. Full details in docs/streaming-pipeline-architecture.md.

Container plugins

All plugins run as separate Docker containers. meta-sort orchestrates them via the Docker socket and a callback endpoint:

Built-in plugins (each is a separate git submodule under packages/plugins/metamesh-plugin-* at the repo root): file-info, ffmpeg, filename-parser, jellyfin-nfo, tmdb, anime-detector, language, subtitle, subtitle-extractor, torrent, fullhash.
Dispatch: ContainerPluginScheduler.dispatchAllPlugins(hashId, filePath, metaFlat) — fire-and-forget. The scheduler keeps its own fast / background queues based on each plugin manifest's defaultQueue.
Plugin → meta-sort: plugins POST results to /api/plugins/callback, and they can read/write Redis metadata through the meta-core-compatible /meta/:hash routes that meta-sort also exposes (so plugins only need one HTTP target).
Plugins access files by hitting meta-sort's WebDAV (http://metasort-app/webdav/... inside the docker network).

For the plugin HTTP contract, see docs/containerized-plugin-architecture.md and docs/plugin-task-queue-architecture.md.

Configuration

Environment variables

Variable	Default	Description
`META_CORE_PATH`	`/meta-core`	Mount of META_CORE_VOLUME. Required — startup fails fast if absent.
`FILES_PATH`	`/files`	Mount of FILES_VOLUME.
`CACHE_FOLDER_PATH`	`/data/cache`	Local cache root.
`SERVICE_NAME`	`meta-sort`	Identifier used for service registration.
`SERVICE_VERSION`	`1.0.0`	Recorded in the service registry.
`FUSE_API_PORT`	`3000`	HTTP API port (legacy name; this is the Fastify API).
`FUSE_API_HOST`	`0.0.0.0`	API bind host.
`BASE_URL`	—	External base URL recorded in service registration.
`ADVERTISE_HOST`	—	Hostname to advertise to other services. Falls back to auto-detected IP.
`MAX_WORKER_THREADS`	`os.cpus().length`	If set, overrides all pipeline concurrencies to `(M*2, M, M)`.
`FAST_QUEUE_CONCURRENCY`	`32`	Container-plugin fast queue concurrency (see plugin-task-queue-architecture.md).
`FILE_LIGHT_SLOTS`	`16`	Max files concurrently in the light phase.
`FILE_BG_SLOTS`	`16`	Max files concurrently in the background phase.
`METADATA_FORMATS`	`meta`	Comma-separated list. `meta` = `.meta` YAML files; `jellyfin` = `.nfo` XML; empty disables sidecar generation.
`CONTAINER_PLUGINS_CONFIG`	`/app/plugins.yml`	Path to plugin manifest YAML. If missing, container plugins are skipped.
`DOCKER_SOCKET_PATH`	`/var/run/docker.sock`	Docker socket for container management.
`CONTAINER_CALLBACK_URL`	`http://meta-sort:8180`	URL plugins POST callbacks to.
`CONTAINER_META_CORE_URL`	`http://meta-sort`	URL plugins use for meta-core-compatible `/meta/*` routes (served by meta-sort itself for now).
`CONTAINER_NETWORK`	`meta-network`	Docker network to attach plugin containers to.
`PLUGIN_STACK_NAME`	—	Docker Compose project label so plugin containers group together in Docker Desktop.
`PLUGIN_CACHE_HOST_PATH`	—	Host path bind-mounted into each plugin for persistent caches.

Source of truth: packages/meta-sort-core/src/config/EnvConfig.ts.

The Redis URL and WebDAV URLs are not configured via environment variables — meta-sort discovers them from kv-leader.info (or its in-memory equivalent fetched through meta-core's /api/urls).

Concurrency defaults

Computed in src/index.ts at startup:

cpuCount = os.cpus().length
defaultBackgroundWorkers = max(1, floor(cpuCount / 2))

validationConcurrency = MAX_WORKER_THREADS ? M*2 : cpuCount * 2
fastQueueConcurrency  = MAX_WORKER_THREADS ?? cpuCount
backgroundQueueConcurrency = MAX_WORKER_THREADS ?? defaultBackgroundWorkers

The pipeline logs the chosen values at startup — check container logs if you need to know what was selected.

Plugin configuration

Container plugins are described in dev/config/plugins.yml (copy from plugins.yml.example). Plugin runtime config (API keys, language, etc.) is set through meta-sort's REST API, not in YAML. See the root CLAUDE.md "Plugin Development" section for examples.

REST API

Defined in packages/meta-sort-core/src/api/UnifiedAPIServer.ts.

Health

Endpoint	Description
`GET /health`	Liveness check.
`GET /api/health`	Liveness + KV connectivity.
`GET /meta-health`	meta-core-compatible health probe.

Processing

Endpoint	Description
`GET /api/processing/status`	Counters + state-manager snapshot.
`GET /api/processing/queue`	Per-queue depths (validation / fast / background).
`GET /api/processing/failed`	Failed-file roster.
`POST /api/processing/retry`	Retry a single failed file (`{ filePath }`).
`POST /api/processing/retry-all`	Retry every failed file.
`POST /api/processing/wait-empty`	Block until queues drain (with optional `timeout` query).

Note: /api/scan/trigger and /api/metadata/clear were removed — they now live on meta-core. The setupScanRoutes() method in the source is intentionally a no-op left as a marker.

Plugins

Endpoint	Description
`GET /api/plugins`	Discovered plugins (manifest + status + execution order).
`GET /api/plugins/timings`	Plugin processing-time stats.
`POST /api/plugins/:pluginId/activate`	Activate a plugin.
`POST /api/plugins/:pluginId/deactivate`	Deactivate a plugin.
`PUT /api/plugins/:pluginId/config`	Set runtime config (e.g. `{ apiKey: "..." }`).
`POST /api/plugins/:pluginId/clear-cache`	Clear a single plugin's cache.
`POST /api/plugins/clear-cache`	Clear every plugin's cache.
`POST /api/plugins/rescan`	Re-scan plugin manifests.
`POST /api/plugins/:pluginId/recompute`	Recompute this plugin for all files.
`POST /api/plugins/callback`	Container plugin task-complete callback (used by plugins, not humans).
`GET /api/plugins/containers`	Container-plugin status (health, instances, queue depths).
`GET /api/plugins/containers/:pluginId/manifest`	Plugin manifest.
`POST /api/plugins/containers/:pluginId/restart`	Restart instances.
`POST /api/plugins/containers/restart-all`	Restart all.
`POST /api/plugins/containers`	Add a new plugin (image, instances, resources, config, defaultQueue).
`PUT /api/plugins/containers/:pluginId`	Update plugin spec.
`DELETE /api/plugins/containers/:pluginId`	Remove plugin.

Files and metadata

Endpoint	Description
`GET /api/file/download?path=...`	Download a file via the configured WebDAV backend.
`POST /file/cid`	Compute the CID (midhash256) for a path (`{ path }`).
`GET /webdav/*` (and `PUT`, `DELETE`, `MKCOL`, `PROPFIND`)	WebDAV view of `/files`. Unauthenticated.

meta-core-compatible KV (for container plugins)

These mirror meta-core's /meta/* so plugin containers can talk to a single host. They are only registered when the KV client is up.

Endpoint	Description
`GET /meta/:hash`	All metadata for a file.
`GET /meta/:hash/*`	Single property.
`PUT /meta/:hash/*`	Set single property (`{ value }`).
`DELETE /meta/:hash/*`	Delete single property.
`PATCH /meta/:hash`	Merge partial metadata.
`POST /meta/:hash/_add/:key`	Append to a comma-separated set field.

Other

Endpoint	Description
`GET /api/metrics`	Performance metrics.
`GET /api/stats`	Redis stats summary (only when KV is up).
`GET /api/services`	Proxy to meta-core's `/api/services` via the LeaderClient.

Operations

Dev URLs

Behind Caddy (auth enforced, self-signed cert): https://metasort-dev.localhost:8180
Debug-direct backend (no auth, no TLS): http://localhost:18180
Container name (backend): metasort-app
Container name (hash-lock proxy): metasort

Logs

docker compose -f dev/docker-compose.yml logs -f metasort-app
docker exec metasort-app supervisorctl status

Reload after a code change

Use the reload script (in-container build via supervisord). Do not docker restart metasort-app.

cd dev
./scripts/reload-meta-sort.sh            # full rebuild
./scripts/reload-meta-sort.sh --backend  # backend only
./scripts/reload-meta-sort.sh --ui       # UI only
./scripts/reload-meta-sort.sh --no-deps  # skip dependency rebuild

Running tests

docker exec meta-test-runner /app/test/test.sh sort

See the root CLAUDE.md "Running Tests" section for the full matrix.

Package layout

This service is a nested pnpm workspace. Top-level structure:

packages/meta-sort/
├── Dockerfile
├── docker/                       # nginx, redis (legacy), supervisord configs
│   ├── nginx.conf
│   ├── redis.conf
│   └── supervisord.conf
├── docs/                         # See "Further reading" below
├── package.json                  # Nested workspace root
├── packages/
│   ├── meta-sort-core/           # @meta-sort/core - main service
│   │   ├── src/
│   │   │   ├── api/              # UnifiedAPIServer.ts (Fastify)
│   │   │   ├── config/           # EnvConfig.ts, SupportedFileTypes.ts
│   │   │   ├── container-plugins/# ContainerManager, ContainerPluginScheduler, ...
│   │   │   ├── events/           # FileEventConsumer, SSEEventClient
│   │   │   ├── jellyfin/         # Jellyfin / NFO output helpers
│   │   │   ├── kv/               # KVManager, LeaderClient, RedisClient
│   │   │   ├── logic/            # StreamingPipeline, state managers, fileProcessor
│   │   │   ├── metrics/          # PerformanceMetrics
│   │   │   ├── plugin-engine/    # Vestigial in-process types + TaskScheduler glue
│   │   │   ├── types/            # Shared TypeScript types
│   │   │   ├── utils/            # Utilities
│   │   │   ├── webdav/           # WebDAV client + endpoint configuration
│   │   │   └── index.ts          # Entry point
│   │   └── package.json
│   ├── meta-sort-ui/             # @meta-sort/ui - React dashboard (Vite)
│   ├── meta-sort-editor/         # Metadata editor UI (Vite)
│   ├── async-utils/              # @worph/async-utils - MultiQueue, etc.
│   └── shared/                   # Symlinks to filename-tool, meta-hash, meta-interface
├── pnpm-workspace.yaml
└── pnpm-lock.yaml

Built-in container plugins do not live here. They live at the repo root as separate submodules: packages/plugins/metamesh-plugin-{file-info,ffmpeg,filename-parser,jellyfin-nfo,tmdb,anime-detector,language,subtitle,subtitle-extractor,torrent,fullhash}.

Key components

Component	File	Purpose
Entry point	`src/index.ts`	Wires KVManager → API server → plugin manager → ContainerManager → StreamingPipeline → FileEventConsumer.
Streaming pipeline	`src/logic/pipeline/StreamingPipeline.ts`	3-queue (validation / fast / background) orchestrator.
State manager	`src/logic/UnifiedProcessingStateManager.ts`	Tracks `discovered → lightProcessing → hashProcessing → done`.
File processor	`src/logic/WatchedFileProcessor.ts`	Light + hash phase implementation.
SSE consumer	`src/events/FileEventConsumer.ts`	Translates meta-core events into pipeline calls.
SSE transport	`src/events/SSEEventClient.ts`	Long-lived HTTP SSE with cursor persistence.
KV manager	`src/kv/KVManager.ts`	Bootstraps RedisKVClient + ServiceRegistration from LeaderClient.
Leader client	`src/kv/LeaderClient.ts`	Reads `/meta-core/locks/kv-leader.info`, fetches `/api/urls`.
Container manager	`src/container-plugins/ContainerManager.ts`	Docker socket orchestration.
Container scheduler	`src/container-plugins/ContainerPluginScheduler.ts`	Plugin fast/background queues + callbacks.
API server	`src/api/UnifiedAPIServer.ts`	Fastify REST + WebDAV-facing meta routes.
Config	`src/config/EnvConfig.ts`	Environment variable parsing.

License

MIT — see LICENSE.