README: sync with current code
This commit is contained in:
Worph
@@ -1,31 +1,38 @@
|
||||
# meta-dup
|
||||
|
||||
Standalone duplicate detection service for MetaMesh v2. Detects duplicate files by:
|
||||
- **Hash (SHA-256)**: Files with identical content
|
||||
- **Title**: Files with the same parsed title (from filename patterns)
|
||||
- **Hash**: Files with identical content — same `midhash256` (record address) seen at several paths, or same full-file SHA-256
|
||||
- **Title**: Files with the same parsed title (+ year / season / episode when known)
|
||||
|
||||
> **Status: not wired into the current stack.** meta-dup was removed from the main dev compose (`dev/docker-compose.yml`) in March 2026 and has no MetaAppStore app. Its storage layer still talks to Redis directly (ioredis + Redis Streams), while meta-core has since stopped publishing a `redisUrl` (api-mediated-access lockdown) — so against a current meta-core it has no Redis to connect to. meta-fuse shows the migrated pattern (HTTP reads via `/meta/{hash}*` + SSE `/api/events/meta`). Treat the sections below as a description of the code, not of a running service.
|
||||
|
||||
## Architecture
|
||||
|
||||
meta-dup is a read-only service that:
|
||||
1. Subscribes to Redis Streams (`meta-sort:events`) for real-time updates
|
||||
2. Maintains an in-memory index of duplicates
|
||||
3. Provides REST API and web dashboard
|
||||
1. Locates meta-core over UDP multicast (meta-discovery v1, `239.255.77.1:9399`) — or uses `META_CORE_URL` when pinned
|
||||
2. Builds an in-memory duplicate index from every record in Redis on startup
|
||||
3. Consumes Redis Streams for real-time updates (`file:events` for file add/change/delete/rename, `meta:events` for title-field changes)
|
||||
4. Provides a REST API and web dashboard (nginx on port 80 → Fastify on 3000)
|
||||
|
||||
```
|
||||
meta-sort ──► Redis Streams ──► meta-dup ──► Dashboard (port 8183)
|
||||
(events) (consumer)
|
||||
meta-core ──► Redis Streams ──► meta-dup ──► Dashboard / REST API
|
||||
(file:events, (consumer)
|
||||
meta:events)
|
||||
```
|
||||
|
||||
See [service-discovery.md](../../docs/project-architecture/service-discovery.md) for the discovery protocol.
|
||||
|
||||
## API Endpoints
|
||||
|
||||
| Endpoint | Method | Description |
|
||||
|----------|--------|-------------|
|
||||
| `/health` | GET | Health check with stats |
|
||||
| `/health` | GET | Health check with stats (`status: ok\|degraded`) |
|
||||
| `/api/duplicates` | GET | All duplicates (hash + title) |
|
||||
| `/api/duplicates/hash` | GET | Hash duplicates only |
|
||||
| `/api/duplicates/title` | GET | Title duplicates only |
|
||||
| `/api/duplicates/stats` | GET | Duplicate statistics |
|
||||
| `/api/duplicates/rebuild` | POST | Force rebuild from Redis |
|
||||
| `/api/neighbors` | GET | Services heard over meta-discovery v1 (dashboard nav menu) |
|
||||
|
||||
## Response Format
|
||||
|
||||
@@ -33,7 +40,7 @@ meta-sort ──► Redis Streams ──► meta-dup ──► Dashboard (port 8
|
||||
{
|
||||
"hashDuplicates": [
|
||||
{
|
||||
"key": "sha256-hash",
|
||||
"key": "<midhash256 or sha256>",
|
||||
"files": [
|
||||
{ "hashId": "...", "filePath": "/files/...", "sizeByte": 123456 }
|
||||
]
|
||||
@@ -42,10 +49,13 @@ meta-sort ──► Redis Streams ──► meta-dup ──► Dashboard (port 8
|
||||
"titleDuplicates": [...],
|
||||
"stats": {
|
||||
"hashGroupCount": 5,
|
||||
"hashFileCount": 12,
|
||||
"titleGroupCount": 3,
|
||||
"totalFilesTracked": 100
|
||||
"titleFileCount": 7,
|
||||
"totalFilesTracked": 100,
|
||||
"lastUpdated": "2026-01-01T00:00:00.000Z"
|
||||
},
|
||||
"computedAt": 1234567890
|
||||
"computedAt": "2026-01-01T00:00:00.000Z"
|
||||
}
|
||||
```
|
||||
|
||||
@@ -58,34 +68,35 @@ pnpm install
|
||||
pnpm build
|
||||
```
|
||||
|
||||
### Run (in Docker dev environment)
|
||||
### Docker image
|
||||
```bash
|
||||
cd dev
|
||||
./scripts/start.sh # Starts all services including meta-dup
|
||||
docker build -t meta-dup .
|
||||
# The image also copies the meta-core binary from META_CORE_IMAGE
|
||||
# (default ghcr.io/worph/meta-core:latest) and runs it under supervisord.
|
||||
```
|
||||
|
||||
### Rebuild
|
||||
```bash
|
||||
cd dev
|
||||
./scripts/reload-meta-dup.sh
|
||||
```
|
||||
CI publishes `ghcr.io/worph/meta-dup` on `v*` tags (`.github/workflows/docker-publish.yml`).
|
||||
|
||||
### Tests
|
||||
```bash
|
||||
docker exec meta-test-runner /app/test/test.sh dup
|
||||
```
|
||||
### Dev stack / tests
|
||||
The main dev stack no longer defines a meta-dup container, so `dev/scripts/reload-meta-dup.sh` (targets container `meta-dup`) and the `dup` suite of `dev/test/test.sh` have nothing to run against until it is re-added.
|
||||
|
||||
## Configuration
|
||||
|
||||
meta-dup uses service discovery via `/meta-core/locks/kv-leader.info` to find the Redis instance. No additional configuration required.
|
||||
meta-core is located over UDP (meta-discovery v1); no `/meta-core` volume mount or `REDIS_URL` is needed.
|
||||
|
||||
### Environment Variables
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `BASE_URL` | `http://localhost:8183` | External URL for service |
|
||||
| `SERVICE_NAME` | `meta-dup` | Service identifier |
|
||||
| `META_CORE_URL` | — | Pin meta-core's API URL; UDP discovery never overrides it |
|
||||
| `ENABLE_UDP_DISCOVERY` | on | Set `false`/`0` to disable meta-discovery |
|
||||
| `BASE_URL` | — | External URL (announced for the nav menu) |
|
||||
| `PUBLIC_URL` | — | Announced URL; wins over `BASE_URL` (debug-direct, no Caddy) |
|
||||
| `API_PORT` | `3000` | Internal API port (nginx proxies to 80) |
|
||||
| `API_HOST` | `0.0.0.0` | API bind address |
|
||||
| `REDIS_PREFIX` | `''` | Key prefix in Redis |
|
||||
| `FILES_VOLUME` | `/files` | Prepended to the relative `filePath` of each record |
|
||||
| `ALLOW_LEGACY_REDIS_URL` | — | `1` downgrades the "meta-core still publishes redisUrl" startup error to a warning |
|
||||
|
||||
## Package Structure
|
||||
|
||||
@@ -93,14 +104,19 @@ meta-dup uses service discovery via `/meta-core/locks/kv-leader.info` to find th
|
||||
packages/meta-dup/
|
||||
├── package.json # Workspace root
|
||||
├── pnpm-workspace.yaml
|
||||
├── Dockerfile
|
||||
├── docker/ # nginx.conf, supervisord.conf
|
||||
└── packages/
|
||||
├── meta-dup-core/ # Backend service
|
||||
│ ├── src/
|
||||
│ │ ├── index.ts # Entry point
|
||||
│ │ ├── DuplicateIndex.ts # Core duplicate detection
|
||||
│ │ ├── index.ts # Entry point, file:events consumer
|
||||
│ │ ├── DuplicateIndex.ts # Core duplicate detection
|
||||
│ │ ├── MetaEventConsumer.ts # meta:events consumer (title updates)
|
||||
│ │ ├── api/
|
||||
│ │ │ └── APIServer.ts # Fastify REST API
|
||||
│ │ └── kv/ # Redis client & leader discovery
|
||||
│ │ │ └── APIServer.ts # Fastify REST API
|
||||
│ │ ├── discovery/
|
||||
│ │ │ └── meshdisco.ts # meta-discovery v1 (mirrored, see scripts/check-mirrors.sh)
|
||||
│ │ └── kv/ # Redis client + meta-core locator (LeaderClient)
|
||||
│ └── package.json
|
||||
└── meta-dup-ui/ # React dashboard
|
||||
├── src/
|
||||
@@ -112,23 +128,21 @@ packages/meta-dup/
|
||||
## How Duplicate Detection Works
|
||||
|
||||
### Hash Duplicates
|
||||
Files with the same `cid_sha2-256` (full file hash) are exact content duplicates.
|
||||
- **Path-based**: `file:events` add/change/rename messages carry `midhash256` + `path`; one midhash seen at several paths is a duplicate.
|
||||
- **Full hash**: the full-file SHA-256 is read from the record's bare-CID key-set (`cids/<cid>`, picking the sha2-256 multicodec). Records with the same SHA-256 are exact content duplicates. See [METADATA_KEYS.md](../../METADATA_KEYS.md).
|
||||
|
||||
### Title Duplicates
|
||||
Files parsed to the same title (e.g., "Movie Name (2020)") from their filename patterns. Useful for finding different quality versions of the same content.
|
||||
Files resolving to the same normalized key `title[:year][:sN][:eN]`. Useful for finding different quality versions of the same content. Title fields (`title`, `originalTitle`, `fileName`, `year`, `movieYear`, `season`, `episode`, `tmdb`) are re-read on `meta:events` changes, debounced 500 ms.
|
||||
|
||||
## Redis Streams Integration
|
||||
|
||||
meta-dup subscribes to `meta-sort:events` stream with consumer group `meta-dup-consumer`:
|
||||
|
||||
```
|
||||
Event types:
|
||||
- batch: File batch processed (add/update duplicates)
|
||||
- reset: Full index rebuild required
|
||||
```
|
||||
| Stream | Consumer group | Handles |
|
||||
|--------|----------------|---------|
|
||||
| `file:events` | `meta-dup-consumer` | `add`, `change`, `delete`, `rename`; legacy `batch`, `reset`, `plugin:complete` |
|
||||
| `meta:events` | `meta-dup-meta-consumer` | `set`/`del` on `file:{hashId}/{field}` for title fields |
|
||||
|
||||
On startup, meta-dup:
|
||||
1. Creates/joins consumer group
|
||||
2. Processes any pending entries
|
||||
3. Rebuilds full index from Redis `file:*` keys
|
||||
4. Starts listening for new events
|
||||
1. Rebuilds the full index from Redis (via the `file:__index__` set)
|
||||
2. Creates/joins the `file:events` consumer group
|
||||
3. Processes pending entries idle > 30 s
|
||||
4. Starts listening for new events on both streams
|
||||
|
||||
Reference in New Issue
Block a user