diff --git a/README.md b/README.md index 6536ef5..547ab5e 100644 --- a/README.md +++ b/README.md @@ -1,31 +1,38 @@ # meta-dup Standalone duplicate detection service for MetaMesh v2. Detects duplicate files by: -- **Hash (SHA-256)**: Files with identical content -- **Title**: Files with the same parsed title (from filename patterns) +- **Hash**: Files with identical content — same `midhash256` (record address) seen at several paths, or same full-file SHA-256 +- **Title**: Files with the same parsed title (+ year / season / episode when known) + +> **Status: not wired into the current stack.** meta-dup was removed from the main dev compose (`dev/docker-compose.yml`) in March 2026 and has no MetaAppStore app. Its storage layer still talks to Redis directly (ioredis + Redis Streams), while meta-core has since stopped publishing a `redisUrl` (api-mediated-access lockdown) — so against a current meta-core it has no Redis to connect to. meta-fuse shows the migrated pattern (HTTP reads via `/meta/{hash}*` + SSE `/api/events/meta`). Treat the sections below as a description of the code, not of a running service. ## Architecture meta-dup is a read-only service that: -1. Subscribes to Redis Streams (`meta-sort:events`) for real-time updates -2. Maintains an in-memory index of duplicates -3. Provides REST API and web dashboard +1. Locates meta-core over UDP multicast (meta-discovery v1, `239.255.77.1:9399`) — or uses `META_CORE_URL` when pinned +2. Builds an in-memory duplicate index from every record in Redis on startup +3. Consumes Redis Streams for real-time updates (`file:events` for file add/change/delete/rename, `meta:events` for title-field changes) +4. Provides a REST API and web dashboard (nginx on port 80 → Fastify on 3000) ``` -meta-sort ──► Redis Streams ──► meta-dup ──► Dashboard (port 8183) - (events) (consumer) +meta-core ──► Redis Streams ──► meta-dup ──► Dashboard / REST API + (file:events, (consumer) + meta:events) ``` +See [service-discovery.md](../../docs/project-architecture/service-discovery.md) for the discovery protocol. + ## API Endpoints | Endpoint | Method | Description | |----------|--------|-------------| -| `/health` | GET | Health check with stats | +| `/health` | GET | Health check with stats (`status: ok\|degraded`) | | `/api/duplicates` | GET | All duplicates (hash + title) | | `/api/duplicates/hash` | GET | Hash duplicates only | | `/api/duplicates/title` | GET | Title duplicates only | | `/api/duplicates/stats` | GET | Duplicate statistics | | `/api/duplicates/rebuild` | POST | Force rebuild from Redis | +| `/api/neighbors` | GET | Services heard over meta-discovery v1 (dashboard nav menu) | ## Response Format @@ -33,7 +40,7 @@ meta-sort ──► Redis Streams ──► meta-dup ──► Dashboard (port 8 { "hashDuplicates": [ { - "key": "sha256-hash", + "key": "", "files": [ { "hashId": "...", "filePath": "/files/...", "sizeByte": 123456 } ] @@ -42,10 +49,13 @@ meta-sort ──► Redis Streams ──► meta-dup ──► Dashboard (port 8 "titleDuplicates": [...], "stats": { "hashGroupCount": 5, + "hashFileCount": 12, "titleGroupCount": 3, - "totalFilesTracked": 100 + "titleFileCount": 7, + "totalFilesTracked": 100, + "lastUpdated": "2026-01-01T00:00:00.000Z" }, - "computedAt": 1234567890 + "computedAt": "2026-01-01T00:00:00.000Z" } ``` @@ -58,34 +68,35 @@ pnpm install pnpm build ``` -### Run (in Docker dev environment) +### Docker image ```bash -cd dev -./scripts/start.sh # Starts all services including meta-dup +docker build -t meta-dup . +# The image also copies the meta-core binary from META_CORE_IMAGE +# (default ghcr.io/worph/meta-core:latest) and runs it under supervisord. ``` -### Rebuild -```bash -cd dev -./scripts/reload-meta-dup.sh -``` +CI publishes `ghcr.io/worph/meta-dup` on `v*` tags (`.github/workflows/docker-publish.yml`). -### Tests -```bash -docker exec meta-test-runner /app/test/test.sh dup -``` +### Dev stack / tests +The main dev stack no longer defines a meta-dup container, so `dev/scripts/reload-meta-dup.sh` (targets container `meta-dup`) and the `dup` suite of `dev/test/test.sh` have nothing to run against until it is re-added. ## Configuration -meta-dup uses service discovery via `/meta-core/locks/kv-leader.info` to find the Redis instance. No additional configuration required. +meta-core is located over UDP (meta-discovery v1); no `/meta-core` volume mount or `REDIS_URL` is needed. ### Environment Variables | Variable | Default | Description | |----------|---------|-------------| -| `BASE_URL` | `http://localhost:8183` | External URL for service | -| `SERVICE_NAME` | `meta-dup` | Service identifier | +| `META_CORE_URL` | — | Pin meta-core's API URL; UDP discovery never overrides it | +| `ENABLE_UDP_DISCOVERY` | on | Set `false`/`0` to disable meta-discovery | +| `BASE_URL` | — | External URL (announced for the nav menu) | +| `PUBLIC_URL` | — | Announced URL; wins over `BASE_URL` (debug-direct, no Caddy) | | `API_PORT` | `3000` | Internal API port (nginx proxies to 80) | +| `API_HOST` | `0.0.0.0` | API bind address | +| `REDIS_PREFIX` | `''` | Key prefix in Redis | +| `FILES_VOLUME` | `/files` | Prepended to the relative `filePath` of each record | +| `ALLOW_LEGACY_REDIS_URL` | — | `1` downgrades the "meta-core still publishes redisUrl" startup error to a warning | ## Package Structure @@ -93,14 +104,19 @@ meta-dup uses service discovery via `/meta-core/locks/kv-leader.info` to find th packages/meta-dup/ ├── package.json # Workspace root ├── pnpm-workspace.yaml +├── Dockerfile +├── docker/ # nginx.conf, supervisord.conf └── packages/ ├── meta-dup-core/ # Backend service │ ├── src/ - │ │ ├── index.ts # Entry point - │ │ ├── DuplicateIndex.ts # Core duplicate detection + │ │ ├── index.ts # Entry point, file:events consumer + │ │ ├── DuplicateIndex.ts # Core duplicate detection + │ │ ├── MetaEventConsumer.ts # meta:events consumer (title updates) │ │ ├── api/ - │ │ │ └── APIServer.ts # Fastify REST API - │ │ └── kv/ # Redis client & leader discovery + │ │ │ └── APIServer.ts # Fastify REST API + │ │ ├── discovery/ + │ │ │ └── meshdisco.ts # meta-discovery v1 (mirrored, see scripts/check-mirrors.sh) + │ │ └── kv/ # Redis client + meta-core locator (LeaderClient) │ └── package.json └── meta-dup-ui/ # React dashboard ├── src/ @@ -112,23 +128,21 @@ packages/meta-dup/ ## How Duplicate Detection Works ### Hash Duplicates -Files with the same `cid_sha2-256` (full file hash) are exact content duplicates. +- **Path-based**: `file:events` add/change/rename messages carry `midhash256` + `path`; one midhash seen at several paths is a duplicate. +- **Full hash**: the full-file SHA-256 is read from the record's bare-CID key-set (`cids/`, picking the sha2-256 multicodec). Records with the same SHA-256 are exact content duplicates. See [METADATA_KEYS.md](../../METADATA_KEYS.md). ### Title Duplicates -Files parsed to the same title (e.g., "Movie Name (2020)") from their filename patterns. Useful for finding different quality versions of the same content. +Files resolving to the same normalized key `title[:year][:sN][:eN]`. Useful for finding different quality versions of the same content. Title fields (`title`, `originalTitle`, `fileName`, `year`, `movieYear`, `season`, `episode`, `tmdb`) are re-read on `meta:events` changes, debounced 500 ms. ## Redis Streams Integration -meta-dup subscribes to `meta-sort:events` stream with consumer group `meta-dup-consumer`: - -``` -Event types: -- batch: File batch processed (add/update duplicates) -- reset: Full index rebuild required -``` +| Stream | Consumer group | Handles | +|--------|----------------|---------| +| `file:events` | `meta-dup-consumer` | `add`, `change`, `delete`, `rename`; legacy `batch`, `reset`, `plugin:complete` | +| `meta:events` | `meta-dup-meta-consumer` | `set`/`del` on `file:{hashId}/{field}` for title fields | On startup, meta-dup: -1. Creates/joins consumer group -2. Processes any pending entries -3. Rebuilds full index from Redis `file:*` keys -4. Starts listening for new events +1. Rebuilds the full index from Redis (via the `file:__index__` set) +2. Creates/joins the `file:events` consumer group +3. Processes pending entries idle > 30 s +4. Starts listening for new events on both streams