bugfix and new internal API
This commit is contained in:
Pierre
@@ -64,10 +64,24 @@ clean:
|
|||||||
rm -f coverage.out coverage.html
|
rm -f coverage.out coverage.html
|
||||||
|
|
||||||
# Build Docker image
|
# Build Docker image
|
||||||
|
#
|
||||||
|
# We tag THREE names:
|
||||||
|
# - meta-core:$(VERSION) — the canonical versioned tag
|
||||||
|
# - meta-core:latest — convenience for `docker run meta-core`
|
||||||
|
# - meta-core:local — the tag every sub-stack docker-compose.yml uses
|
||||||
|
# (meta-share federation, meta-gateway dev stacks).
|
||||||
|
# Without this, `make docker` produces an image
|
||||||
|
# the dev stacks can't see by name, and previously
|
||||||
|
# -tagged stale `:local` images linger, silently
|
||||||
|
# breaking downstream API fields (e.g. /urls
|
||||||
|
# missing webdavUrlInternal → meta-gateway's
|
||||||
|
# poster auto-store fails). See README.md →
|
||||||
|
# Quick start → Build for the audit recipe.
|
||||||
docker:
|
docker:
|
||||||
@echo "Building Docker image..."
|
@echo "Building Docker image..."
|
||||||
docker build -t $(DOCKER_IMAGE):$(VERSION) .
|
docker build -t $(DOCKER_IMAGE):$(VERSION) .
|
||||||
docker tag $(DOCKER_IMAGE):$(VERSION) $(DOCKER_IMAGE):latest
|
docker tag $(DOCKER_IMAGE):$(VERSION) $(DOCKER_IMAGE):latest
|
||||||
|
docker tag $(DOCKER_IMAGE):$(VERSION) $(DOCKER_IMAGE):local
|
||||||
|
|
||||||
# Run locally (for development)
|
# Run locally (for development)
|
||||||
run: build
|
run: build
|
||||||
|
|||||||
@@ -1,520 +1,374 @@
|
|||||||
# meta-core
|
# meta-core
|
||||||
|
|
||||||
A Go-based sidecar binary that provides unified leader election, Redis management, and HTTP API for MetaMesh services.
|
Standalone Go service that owns Redis, the leader-info file, the HTTP+SSE
|
||||||
|
metadata surface, and a WebDAV file server for the MetaMesh dev/prod stack.
|
||||||
|
|
||||||
## Overview
|
## Overview
|
||||||
|
|
||||||
meta-core runs alongside each MetaMesh service (meta-sort, meta-fuse, meta-stremio) as a sidecar process managed by supervisord. It provides the foundational infrastructure that enables MetaMesh's distributed architecture without requiring external coordination services like etcd or Consul.
|
meta-core runs as its own container (`metacore-app`). Other MetaMesh services
|
||||||
|
(meta-sort, meta-fuse, meta-stremio, meta-dup, meta-share) discover it by
|
||||||
|
reading `/meta-core/locks/kv-leader.info` and then call its HTTP API. Redis
|
||||||
|
is not exposed outside the container — all metadata I/O is mediated by the
|
||||||
|
HTTP API.
|
||||||
|
|
||||||
### Key Capabilities
|
### Key capabilities
|
||||||
|
|
||||||
| Feature | Description |
|
| Feature | Description |
|
||||||
|---------|-------------|
|
|---------|-------------|
|
||||||
| **Leader Election** | flock-based leader election (Go binary only runs as leader) |
|
| **Leader gate** | Bash `flock` loop in `docker/leader-election.sh` only `exec`s supervisord on the winner. The Go binary therefore only ever runs as leader. |
|
||||||
| **Redis Management** | Leader spawns Redis with AOF+RDB persistence, auto-restart on crash |
|
| **Redis owner** | Supervisord starts Redis with AOF + RDB persistence inside the container; meta-core connects to it on localhost. |
|
||||||
| **HTTP API** | Language-agnostic REST interface for metadata and service discovery |
|
| **HTTP API** | Typed REST surface (gorilla/mux) for metadata, KV browsing, snapshots, schema inference, mounts, watchers, services, and file access by CID. |
|
||||||
| **Service Discovery** | File-based registry with heartbeats and stale detection |
|
| **SSE event streams** | `/api/events/files` and `/api/events/meta` mirror the underlying Redis Streams so external services never speak Redis. |
|
||||||
| **Metadata Storage** | Flat key-value schema with connection pooling and batch operations |
|
| **WebDAV server** | `/webdav/*` exposes the `/files` volume (read+write) for cross-service file access. nginx in front of meta-core handles caching. |
|
||||||
| **File Watching** | Directory scanning, MidHash256 computation, event dispatch |
|
| **Service discovery** | File-based registry under `/meta-core/services/*.json` with heartbeat + dead-service cleanup. |
|
||||||
| **Mount Management** | SMB/rclone configuration, health monitoring |
|
| **File watcher / scanner** | Recursive scan + MidHash256 computation; emits events on `file:events`. |
|
||||||
| **WebDAV Server** | File access with nginx proxy_cache |
|
| **Mount management** | rclone-only mount manager (SMB rendered into `:smb:`, plus pre-defined remotes). Read-only by construction. |
|
||||||
| **Event Publishing** | Redis keyspace notifications to meta:events stream |
|
| **UUID-rooted storage** | Roots are UUIDv7 (Crockford Base32, ULID layout); CIDs are reverse-index aliases. Schema-version sentinel refuses to boot against stale data. |
|
||||||
|
|
||||||
### Design Characteristics
|
### Design characteristics
|
||||||
|
|
||||||
- **No external dependencies** - Uses filesystem locks (works on NFS/CIFS shared volumes)
|
- **Single-leader** — flock on a shared volume; no external consensus.
|
||||||
- **Automatic failover** - Lock releases on process death, followers detect and re-elect
|
- **HTTP-only externally** — Redis is bound to the container, never published.
|
||||||
- **Thread-safe** - RWMutex protection on all shared state
|
- **Schema-gated boot** — `EnsureSchemaVersion` aborts startup when the
|
||||||
- **Graceful shutdown** - Signal handling with reverse startup order
|
on-disk Redis layout predates the current build (alpha clean-wipe policy;
|
||||||
- **Static binary** - CGO disabled, single binary deployment
|
see `docs/uuid-rooted-metadata.md`).
|
||||||
|
- **Static binary** — CGO disabled, single Go binary plus a thin Alpine
|
||||||
|
image with `redis-server`, `rclone`, nginx, and supervisord.
|
||||||
|
|
||||||
## Architecture
|
## Architecture
|
||||||
|
|
||||||
meta-core runs as a **standalone container** that other services connect to via HTTP API:
|
|
||||||
|
|
||||||
```
|
```
|
||||||
┌───────────────────────────────────────────────────────────────────────┐
|
┌──────────────────────────────────────────────────────────────────────────┐
|
||||||
│ meta-core container (standalone) │
|
│ metacore-app (standalone container) │
|
||||||
│ │
|
│ │
|
||||||
│ ┌──────────────┐ ┌─────────┐ ┌──────────┐ ┌────────────────────┐ │
|
│ supervisord │
|
||||||
│ │ Role │ │ Redis │ │ WebDAV │ │ HTTP API (:9000) │ │
|
│ ├─ leader-election.sh (flock /meta-core/locks/kv-leader.lock, │
|
||||||
│ │ Provider │ │ Server │ │ Server │ │ /meta, /urls, │ │
|
│ │ writes kv-leader.info, exec supervisord) │
|
||||||
│ │ │ │ │ │ │ │ /health, /services │ │
|
│ ├─ redis-server (AOF + RDB on /meta-core/db/redis) │
|
||||||
│ └──────────────┘ └─────────┘ └──────────┘ └────────────────────┘ │
|
│ ├─ nginx (TLS termination, proxy_cache for /webdav) │
|
||||||
│ │
|
│ ├─ rclone rcd (mount daemon, RC API) │
|
||||||
│ Writes: /meta-core/locks/kv-leader.info (URLs for all services) │
|
│ ├─ mount-watcher.sh (rclone mount lifecycle) │
|
||||||
└──────────────────────────────────────────────┬────────────────────────┘
|
│ └─ meta-core (Go) ─► HTTP API + SSE + WebDAV on :9000 │
|
||||||
|
└──────────────────────────────────────────────────────────────────────────┘
|
||||||
│
|
│
|
||||||
┌──────────────────────────────────────┴──────────────────────────┐
|
┌──────────────────────┴──────────────────────┐
|
||||||
│ /meta-core (shared volume) │
|
│ /meta-core (shared volume) │
|
||||||
│ ├── locks/kv-leader.info (leader URLs) │
|
│ locks/kv-leader.{lock,info} │
|
||||||
│ ├── db/redis/ (Redis data) │
|
│ db/redis/ │
|
||||||
│ ├── services/ (hostname-based .json) │
|
│ services/*.json │
|
||||||
│ └── mounts/ (mount configurations) │
|
│ mounts/{mounts.json, errors/} │
|
||||||
└─────────────────────────────────────────────────────────────────┘
|
│ watchers.json │
|
||||||
▲ ▲ ▲
|
│ cache/ (nginx proxy_cache) │
|
||||||
┌────────────┘ │ └────────────┐
|
└─────────────────────────────────────────────┘
|
||||||
|
▲
|
||||||
|
┌──────────────────────────────┼────────────────────────────────┐
|
||||||
│ │ │
|
│ │ │
|
||||||
┌───────────────┐ ┌───────────────┐ ┌───────────────┐
|
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
|
||||||
│ meta-sort │ │ meta-fuse │ │ meta-stremio│
|
│ meta-sort │ │ meta-fuse │ │ meta-stremio│
|
||||||
│ (reads info, │ │ (reads info, │ │ (reads info, │
|
│ │ │ │ │ │
|
||||||
│ calls HTTP) │ │ calls HTTP) │ │ calls HTTP) │
|
│ reads │ HTTP+SSE │ reads │ HTTP+SSE │ reads │
|
||||||
└───────────────┘ └───────────────┘ └───────────────┘
|
│ leader.info├──────────────►│ leader.info├───────────────►│ leader.info│
|
||||||
|
└─────────────┘ └─────────────┘ └─────────────┘
|
||||||
```
|
```
|
||||||
|
|
||||||
## Quick Start
|
## Quick start
|
||||||
|
|
||||||
### Build
|
### Build
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Build binary
|
make build # build static binary to bin/meta-core
|
||||||
make build
|
make docker # build container image as meta-core:1.0.0 + :latest + :local
|
||||||
|
make test # run Go unit tests
|
||||||
# Build Docker image
|
|
||||||
make docker
|
|
||||||
|
|
||||||
# Run tests
|
|
||||||
make test
|
|
||||||
```
|
```
|
||||||
|
|
||||||
### Run
|
`make docker` tags the image as `:1.0.0`, `:latest`, **and** `:local`. The
|
||||||
|
`:local` tag is the one every sub-stack docker-compose.yml references
|
||||||
|
(meta-share 10-peer federation, meta-gateway dev stacks); without it the
|
||||||
|
sub-stacks can't see a freshly-built image by name.
|
||||||
|
|
||||||
|
> **Source-change → recreate.** `make docker` rebuilds and re-tags the
|
||||||
|
> image, but **already-running containers stay on the old image SHA** —
|
||||||
|
> `docker compose up -d` won't recreate them unless the compose file
|
||||||
|
> itself changed. After a meta-core source change, force-recreate the
|
||||||
|
> downstream containers so they pick up the new binary:
|
||||||
|
>
|
||||||
|
> ```bash
|
||||||
|
> make docker
|
||||||
|
> docker compose -f packages/meta-gateway/dev/docker-compose.yml \
|
||||||
|
> up -d --force-recreate metagateway-generic-core
|
||||||
|
> docker compose -f packages/meta-gateway/dev/docker-compose.torznab.yml \
|
||||||
|
> up -d --force-recreate metagateway-torznab-core
|
||||||
|
> docker compose -f packages/meta-share/dev/docker-compose.yml \
|
||||||
|
> --profile share-10node up -d --force-recreate metashare-core metashare-core-test
|
||||||
|
> ```
|
||||||
|
>
|
||||||
|
> The symptom of a missed recreate is missing API fields on `/urls`
|
||||||
|
> (e.g. `webdavUrlInternal` absent → meta-gateway's
|
||||||
|
> `store_poster_for_record` silently fails and torznab posters never
|
||||||
|
> land). Audit recipe:
|
||||||
|
>
|
||||||
|
> ```bash
|
||||||
|
> docker inspect meta-core:local --format '{{.Id}}' # fresh SHA
|
||||||
|
> docker ps --filter ancestor=meta-core:local --format '{{.Names}}'
|
||||||
|
> # any container whose Image SHA doesn't match the fresh SHA needs --force-recreate.
|
||||||
|
> ```
|
||||||
|
|
||||||
|
### Dev container
|
||||||
|
|
||||||
|
In the MetaMesh dev stack, the container is brought up by `dev/docker-compose.yml`:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# With environment variables
|
cd dev && ./scripts/start.sh
|
||||||
META_CORE_PATH=/meta-core \
|
|
||||||
FILES_PATH=/files \
|
|
||||||
SERVICE_NAME=meta-sort \
|
|
||||||
META_CORE_HTTP_PORT=9000 \
|
|
||||||
./bin/meta-core
|
|
||||||
```
|
```
|
||||||
|
|
||||||
### Docker
|
The user-facing URL is `https://metacore-dev.localhost:8083`; the direct-backend
|
||||||
|
HTTP port (no auth, bypasses Caddy) is `http://localhost:18083`.
|
||||||
```bash
|
|
||||||
docker build -t meta-core .
|
|
||||||
docker run -v meta-core:/meta-core -v files:/files meta-core
|
|
||||||
```
|
|
||||||
|
|
||||||
## Configuration
|
## Configuration
|
||||||
|
|
||||||
| Environment Variable | Default | Description |
|
Environment variables consumed by `internal/config/config.go`:
|
||||||
|---------------------|---------|-------------|
|
|
||||||
| `META_CORE_PATH` | `/meta-core` | Shared volume for locks/db |
|
|
||||||
| `FILES_PATH` | `/files` | Shared volume for media files |
|
|
||||||
| `SERVICE_NAME` | `meta-core` | Service identifier |
|
|
||||||
| `SERVICE_VERSION` | `1.0.0` | Service version |
|
|
||||||
| `API_PORT` | `8180` | Main service HTTP port |
|
|
||||||
| `BASE_URL` | - | Stable service URL |
|
|
||||||
| `REDIS_PORT` | `6379` | Redis port (leader only) |
|
|
||||||
| `META_CORE_HTTP_PORT` | `9000` | HTTP API port |
|
|
||||||
| `META_CORE_HTTP_HOST` | `127.0.0.1` | HTTP API bind address |
|
|
||||||
| `HEALTH_CHECK_INTERVAL_MS` | `5000` | Health check interval |
|
|
||||||
| `HEARTBEAT_INTERVAL_MS` | `30000` | Service heartbeat interval |
|
|
||||||
| `STALE_THRESHOLD_MS` | `60000` | Stale service threshold |
|
|
||||||
| `ENABLE_FILE_WATCHER` | `true` | Enable file scanning |
|
|
||||||
|
|
||||||
## API Reference
|
| Variable | Default | Description |
|
||||||
|
|---|---|---|
|
||||||
|
| `META_CORE_PATH` | `/meta-core` | Shared volume root (locks, db, services, mounts, cache). |
|
||||||
|
| `FILES_PATH` | `/files` | Files volume root. |
|
||||||
|
| `SERVICE_NAME` | `meta-core` | Identity in the service registry. |
|
||||||
|
| `SERVICE_VERSION` | `1.0.0` | Reported via `/status` and registry. |
|
||||||
|
| `API_PORT` | `8180` | External port baked into the leader-info `baseUrl`. |
|
||||||
|
| `BASE_URL` | _empty_ | Overrides the constructed `baseUrl` if set (used for HTTPS perimeter). |
|
||||||
|
| `REDIS_PORT` | `6379` | Local Redis port. |
|
||||||
|
| `META_CORE_HTTP_PORT` | `9000` | Go HTTP+SSE+WebDAV port. |
|
||||||
|
| `META_CORE_HTTP_HOST` | `127.0.0.1` | HTTP bind. |
|
||||||
|
| `HEALTH_CHECK_INTERVAL_MS` | `5000` | Internal health-loop cadence. |
|
||||||
|
| `HEARTBEAT_INTERVAL_MS` | `30000` | Service registry heartbeat. |
|
||||||
|
| `STALE_THRESHOLD_MS` | `60000` | Age past which a registry entry is marked stale. |
|
||||||
|
| `CLEANUP_INTERVAL_MS` | `600000` | Dead-service cleaner cadence. |
|
||||||
|
| `DEAD_SERVICE_THRESHOLD_MS` | `600000` | Age past which a stale entry is deleted. |
|
||||||
|
| `WATCH_INTERVAL_MS` | `1000` | Watcher poll cadence. |
|
||||||
|
| `DEBOUNCE_MS` | `30000` | File-event debounce window. |
|
||||||
|
| `ENABLE_FILE_WATCHER` | `true` | Disable to suppress the in-process watcher entirely. |
|
||||||
|
|
||||||
### Health & Status
|
Note: there is no `META_CORE_REDIS_PORT`, no `META_CORE_SERVICE_NAME`, and
|
||||||
|
no `META_CORE_LEADER_TIMEOUT`. The Redis URL is discovered locally; service
|
||||||
|
identity uses `SERVICE_NAME`; leader timeout is governed by the bash flock
|
||||||
|
retry interval (`ELECTION_RETRY_SECS`, default `5`).
|
||||||
|
|
||||||
|
## API reference
|
||||||
|
|
||||||
|
All endpoints below are served by `internal/api/server.go`. Examples assume
|
||||||
|
the in-container Go port `:9000`; from the host use
|
||||||
|
`https://metacore-dev.localhost:8083` (Caddy) or `http://localhost:18083`
|
||||||
|
(debug-direct).
|
||||||
|
|
||||||
|
### Health, leader, URLs
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Health check
|
GET /health # storage + role
|
||||||
curl http://localhost:9000/health
|
GET /status # version, uptime, etc.
|
||||||
# {"status":"ok","redis":true,"timestamp":"..."}
|
GET /leader # current leader info (hostname, PID, timestamps)
|
||||||
|
GET /urls # baseUrl, apiUrl, webdavUrl, webdavUrlInternal (redisUrl is intentionally empty)
|
||||||
# Detailed status
|
GET /services # registered services + heartbeats
|
||||||
curl http://localhost:9000/status
|
GET /api/services # alias for /services
|
||||||
# {"status":"ok","serviceName":"meta-core","version":"1.0.0",...}
|
GET /services/{name} # one service
|
||||||
|
GET /services/cleanup/stats
|
||||||
# Current leader
|
|
||||||
curl http://localhost:9000/leader
|
|
||||||
# {"host":"meta-sort-dev","api":"redis://10.0.1.50:6379","http":"http://10.0.1.50:8180",...}
|
|
||||||
```
|
```
|
||||||
|
|
||||||
### Metadata Operations
|
### Metadata — primary surface
|
||||||
|
|
||||||
|
These are the routes other services use. `{hash}` may be a UUID root or
|
||||||
|
any registered CID alias; CID aliases resolve to the underlying UUID
|
||||||
|
before reads/writes (`internal/storage/cid_resolution.go`).
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# List all file hashes
|
GET /meta # list of root IDs
|
||||||
curl http://localhost:9000/meta
|
GET /meta/{hash} # full document (flat keys → nested JSON)
|
||||||
# {"hashIds":["midhash256:abc123",...],"count":42}
|
PUT /meta/{hash} # replace document
|
||||||
|
PATCH /meta/{hash} # merge into document
|
||||||
# Get metadata for a file
|
DELETE /meta/{hash} # delete root + all CID aliases
|
||||||
curl http://localhost:9000/meta/{hash}
|
GET /meta/{hash}/{key...} # single property (key may contain "/")
|
||||||
# {"hashId":"midhash256:abc123","metadata":{"title":"Movie","year":"2024",...}}
|
PUT /meta/{hash}/{key...} # set single property
|
||||||
|
DELETE /meta/{hash}/{key...} # delete single property
|
||||||
# Update metadata
|
POST /meta/{hash}/_add/{key...} # add value to set-valued property
|
||||||
curl -X PUT http://localhost:9000/meta/{hash} \
|
|
||||||
-H "Content-Type: application/json" \
|
|
||||||
-d '{"title":"New Title"}'
|
|
||||||
|
|
||||||
# Delete metadata
|
|
||||||
curl -X DELETE http://localhost:9000/meta/{hash}
|
|
||||||
```
|
```
|
||||||
|
|
||||||
### Data Operations
|
CID-addressed access (public; auth-bypassed in the Caddy perimeter):
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Get file path
|
GET /api/meta/{cid} # resolve CID → metadata document
|
||||||
curl http://localhost:9000/data/{hash}/path
|
GET /api/file/{cid}/info # CID resolution metadata
|
||||||
# {"hashId":"midhash256:abc123","path":"/files/movies/Movie.mkv","exists":true}
|
GET /file/{cid} # serve file bytes (range-aware)
|
||||||
|
POST /file/cid # compute CID for an uploaded file
|
||||||
# Check if file exists
|
HEAD /data/{hash} # existence + size
|
||||||
curl -I http://localhost:9000/data/{hash}
|
GET /data/{hash}/path # absolute path on /files
|
||||||
# HTTP/1.1 200 OK (or 404)
|
|
||||||
```
|
```
|
||||||
|
|
||||||
### File Operations (by CID)
|
(There is no `/data/{hash}/stream`; bulk reads go through `/webdav/*` or
|
||||||
|
the `/file/{cid}` reverse-lookup path.)
|
||||||
|
|
||||||
|
### Editor / KV browser surface
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Serve file by CID (looks up poster/backdrop CIDs in metadata)
|
GET /api/metadata/hash-ids
|
||||||
curl http://localhost:9000/file/{cid} --output poster.jpg
|
GET /api/metadata/list
|
||||||
# Returns raw file bytes with appropriate Content-Type header
|
POST /api/metadata/search
|
||||||
|
POST /api/metadata/batch
|
||||||
|
POST /api/metadata/clear
|
||||||
|
GET /api/metadata/{hashId}
|
||||||
|
PUT /api/metadata/{hashId}
|
||||||
|
DELETE /api/metadata/{hashId}
|
||||||
|
GET /api/metadata/{hashId}/property
|
||||||
|
PUT /api/metadata/{hashId}/property
|
||||||
|
|
||||||
# Example: fetch a poster image
|
GET /api/kv/info
|
||||||
curl -v http://localhost:9000/file/bafkreih5aznjvttude6c3wbvqeebb6rlx5wkbzyppv7garber7ndsuxku4
|
GET /api/kv/keys
|
||||||
# Content-Type: image/jpeg
|
GET /api/kv/tree
|
||||||
# [binary image data]
|
GET /api/kv/search
|
||||||
|
GET /api/kv/find
|
||||||
|
GET /api/kv/value
|
||||||
|
PUT /api/kv/value
|
||||||
|
DELETE /api/kv/value
|
||||||
|
GET /api/kv/key/{key...}
|
||||||
|
|
||||||
|
GET /api/schema
|
||||||
|
POST /api/schema/rescan
|
||||||
|
|
||||||
|
GET /api/snapshot/export
|
||||||
|
POST /api/snapshot/import
|
||||||
|
POST /api/snapshot/wipe
|
||||||
```
|
```
|
||||||
|
|
||||||
The `/file/{cid}` endpoint searches all file metadata for matching `poster` or `backdrop` CIDs and serves the corresponding file from disk. Supports range requests for partial content retrieval.
|
### SSE event streams
|
||||||
|
|
||||||
### Service Discovery
|
The HTTP-mirror of the Redis Streams. External services never read Redis
|
||||||
|
directly — they subscribe here with `Last-Event-ID` for resume. See
|
||||||
|
[`docs/api-mediated-access.md`](docs/api-mediated-access.md) for the wire
|
||||||
|
contract (one event per stream entry, opaque IDs, heartbeats, gap
|
||||||
|
signalling).
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# List all services
|
GET /api/events/files # file:events stream (watcher events)
|
||||||
curl http://localhost:9000/services
|
GET /api/events/meta # meta:events stream (metadata mutations)
|
||||||
# {"services":[{"name":"meta-sort",...}],"count":3}
|
|
||||||
|
|
||||||
# Get specific service
|
|
||||||
curl http://localhost:9000/services/meta-sort
|
|
||||||
# {"name":"meta-sort","version":"2.0.0","api":"http://...","status":"running",...}
|
|
||||||
```
|
```
|
||||||
|
|
||||||
### Mount Management
|
### Mounts
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# List all mounts
|
GET /api/mounts
|
||||||
curl http://localhost:9000/api/mounts/list
|
POST /api/mounts
|
||||||
# {"mounts":[{"id":"smb-nas","type":"smb","path":"/files/corn/nas",...}]}
|
GET /api/mounts/{id}
|
||||||
|
PUT /api/mounts/{id} # also PATCH
|
||||||
# Add a new mount
|
DELETE /api/mounts/{id}
|
||||||
curl -X POST http://localhost:9000/api/mounts/add \
|
POST /api/mounts/{id}/mount
|
||||||
-H "Content-Type: application/json" \
|
POST /api/mounts/{id}/unmount
|
||||||
-d '{"type":"smb","path":"/files/corn/nas","config":{...}}'
|
POST /api/mounts/{id}/safe-unmount
|
||||||
|
POST /api/mounts/{id}/scan
|
||||||
# Remove a mount
|
GET /api/mounts/rclone/remotes
|
||||||
curl -X DELETE http://localhost:9000/api/mounts/smb-nas
|
|
||||||
```
|
```
|
||||||
|
|
||||||
### File Watching
|
### Watchers
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Get scan status
|
GET /api/watchers
|
||||||
curl http://localhost:9000/api/scan/status
|
POST /api/watchers
|
||||||
# {"scanning":false,"lastScan":"2024-01-15T10:30:00Z","filesFound":12847}
|
GET /api/watchers/{id}
|
||||||
|
PUT /api/watchers/{id}
|
||||||
# Trigger rescan
|
DELETE /api/watchers/{id}
|
||||||
curl -X POST http://localhost:9000/api/scan/trigger
|
POST /api/watchers/{id}/scan
|
||||||
|
POST /api/watchers/{id}/reset
|
||||||
# Stream file events (SSE)
|
POST /api/watchers/scan-all
|
||||||
curl http://localhost:9000/api/events/stream
|
POST /api/watchers/reset-all
|
||||||
# data: {"type":"add","path":"movies/Movie.mkv","midhash256":"bafkr..."}
|
|
||||||
```
|
```
|
||||||
|
|
||||||
### KV Browser
|
The legacy `/api/scan/trigger` and `/api/scan/status` still respond but
|
||||||
|
return a deprecation pointer at the `/api/watchers/*` routes
|
||||||
|
(`internal/watcher/handlers.go:32-33`).
|
||||||
|
|
||||||
|
### Admin / WebDAV
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Storage statistics
|
POST /api/admin/migrate-dual-roots # reunify stranded midhash-rooted entries
|
||||||
curl http://localhost:9000/api/kv/info
|
GET/PUT/DELETE /webdav/... # mounted on /files
|
||||||
# {"totalKeys":50000,"memoryUsed":"256MB",...}
|
|
||||||
|
|
||||||
# List keys with cursor pagination
|
|
||||||
curl "http://localhost:9000/api/kv/keys?cursor=0&count=100"
|
|
||||||
|
|
||||||
# Get specific key value
|
|
||||||
curl http://localhost:9000/api/kv/key/file:bafkr.../title
|
|
||||||
```
|
```
|
||||||
|
|
||||||
### Metadata Editor
|
No `/metrics` endpoint is exposed; observability is via container logs and
|
||||||
|
the `/health` / `/status` JSON.
|
||||||
|
|
||||||
```bash
|
## Leader gate
|
||||||
# Get all hash IDs
|
|
||||||
curl http://localhost:9000/api/metadata/hash-ids
|
|
||||||
# {"hashIds":["bafkr...","bafks..."]}
|
|
||||||
|
|
||||||
# Paginated metadata listing
|
Leader election is a bash flock loop in `docker/leader-election.sh`, not a
|
||||||
curl "http://localhost:9000/api/metadata/list?page=0&pageSize=50"
|
Go-level state machine. When `flock` succeeds on `kv-leader.lock`, the
|
||||||
|
script writes the leader API URL to `kv-leader.info` and `exec`s
|
||||||
|
supervisord. When the process tree dies, the kernel releases the lock and
|
||||||
|
any standby instance can take over. There is no follower path inside the
|
||||||
|
Go binary — there is no `OnBecomeLeader` / `OnBecomeFollower` callback API.
|
||||||
|
|
||||||
# Search metadata
|
`kv-leader.info` is a single plaintext line containing the API base URL:
|
||||||
curl -X POST http://localhost:9000/api/metadata/search \
|
|
||||||
-H "Content-Type: application/json" \
|
|
||||||
-d '{"query":"inception","fields":["title"]}'
|
|
||||||
|
|
||||||
# Batch update
|
|
||||||
curl -X POST http://localhost:9000/api/metadata/batch \
|
|
||||||
-H "Content-Type: application/json" \
|
|
||||||
-d '{"operations":[{"hashId":"bafkr...","updates":{"title":"New Title"}}]}'
|
|
||||||
|
|
||||||
# Clear all metadata
|
|
||||||
curl -X POST http://localhost:9000/api/metadata/clear
|
|
||||||
```
|
|
||||||
|
|
||||||
### WebDAV Access
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# List directory (returns JSON)
|
|
||||||
curl http://localhost:9000/webdav/
|
|
||||||
# [{"name":"watch","isDir":true},{"name":"plugin","isDir":true}]
|
|
||||||
|
|
||||||
# Get file
|
|
||||||
curl http://localhost:9000/webdav/watch/movie.mkv --output movie.mkv
|
|
||||||
|
|
||||||
# Upload file
|
|
||||||
curl -X PUT http://localhost:9000/webdav/plugin/output.txt \
|
|
||||||
-H "Content-Type: application/octet-stream" \
|
|
||||||
--data-binary @output.txt
|
|
||||||
|
|
||||||
# Create directory
|
|
||||||
curl -X MKCOL http://localhost:9000/webdav/newdir/
|
|
||||||
|
|
||||||
# Delete file/directory
|
|
||||||
curl -X DELETE http://localhost:9000/webdav/oldfile.txt
|
|
||||||
```
|
|
||||||
|
|
||||||
## Leader Election
|
|
||||||
|
|
||||||
meta-core uses POSIX file locking (flock) for distributed leader election:
|
|
||||||
|
|
||||||
1. Each instance tries to acquire exclusive lock on `/meta-core/locks/kv-leader.lock`
|
|
||||||
2. Winner becomes **leader**, spawns Redis, writes info to `/meta-core/locks/kv-leader.info`
|
|
||||||
3. Losers become **followers**, read leader info, connect to leader's Redis
|
|
||||||
4. Lock automatically releases when process dies (no stale locks)
|
|
||||||
|
|
||||||
### Election Flow
|
|
||||||
|
|
||||||
```
|
|
||||||
Instance 1 Instance 2 Instance 3
|
|
||||||
│ │ │
|
|
||||||
├── Try flock ──────────┼───────────────────────┤
|
|
||||||
│ │ │
|
|
||||||
[WIN] [LOSE] [LOSE]
|
|
||||||
LEADER FOLLOWER FOLLOWER
|
|
||||||
│ │ │
|
|
||||||
Start Redis Read leader.info Read leader.info
|
|
||||||
│ │ │
|
|
||||||
Write leader.info Connect to Redis Connect to Redis
|
|
||||||
│ │ │
|
|
||||||
[Health loop 5s] [Health loop 5s] [Health loop 5s]
|
|
||||||
- Update timestamp - Re-read leader - Re-read leader
|
|
||||||
- Check Redis alive - Detect changes - Detect changes
|
|
||||||
```
|
|
||||||
|
|
||||||
### Role Callbacks
|
|
||||||
|
|
||||||
The election system provides callbacks for role transitions:
|
|
||||||
|
|
||||||
```go
|
|
||||||
election.OnBecomeLeader(func(info LeaderLockInfo) {
|
|
||||||
// Start Redis, initialize resources
|
|
||||||
})
|
|
||||||
election.OnBecomeFollower(func(info LeaderLockInfo) {
|
|
||||||
// Connect to leader's Redis
|
|
||||||
})
|
|
||||||
election.OnLeaderLost(func() {
|
|
||||||
// Handle leader failure, attempt re-election
|
|
||||||
})
|
|
||||||
```
|
|
||||||
|
|
||||||
### Lock Info Format
|
|
||||||
|
|
||||||
The `kv-leader.info` file contains a plain text API URL:
|
|
||||||
|
|
||||||
```
|
```
|
||||||
http://10.0.1.50:9000
|
http://10.0.1.50:9000
|
||||||
```
|
```
|
||||||
|
|
||||||
Services read this file to discover the leader, then call the `/urls` API endpoint to get full connection details (Redis URL, WebDAV URL, base URL, etc.).
|
Other services read this file, then call `GET /urls` for full discovery.
|
||||||
|
|
||||||
## Redis Management
|
The dev compose runs a single `metacore-app`; the flock primitive is
|
||||||
|
retained as a correctness anchor (one process owns Redis and the
|
||||||
|
writable state under `/meta-core`), not as live failover.
|
||||||
|
|
||||||
The leader is responsible for spawning and managing the Redis server.
|
## Metadata storage shape (one-line summary)
|
||||||
|
|
||||||
### Redis Configuration
|
Roots are opaque UUIDv7 strings; every property is a separate Redis
|
||||||
|
STRING at `file:<uuid>/<property>`; every known CID for a file is a
|
||||||
|
reverse-index entry `cid:<algorithm>:<value> → <uuid>`. Full details in
|
||||||
|
[`docs/metadata-storage-structure.md`](docs/metadata-storage-structure.md)
|
||||||
|
and design rationale in [`docs/uuid-rooted-metadata.md`](docs/uuid-rooted-metadata.md).
|
||||||
|
The authoritative registry of field semantics and value formats lives in
|
||||||
|
the repo-root [`METADATA_KEYS.md`](../../METADATA_KEYS.md).
|
||||||
|
|
||||||
```bash
|
## Internal packages
|
||||||
redis-server --port 6379 --bind 0.0.0.0 \
|
|
||||||
--dir /meta-core/db/redis \
|
|
||||||
--appendonly yes --appendfilename appendonly.aof \
|
|
||||||
--dbfilename dump.rdb --save 60 1 \
|
|
||||||
--loglevel warning
|
|
||||||
```
|
|
||||||
|
|
||||||
### Persistence Strategy
|
|
||||||
|
|
||||||
| Method | Configuration | Purpose |
|
|
||||||
|--------|--------------|---------|
|
|
||||||
| **AOF** | `appendonly yes` | Write-ahead log for durability |
|
|
||||||
| **RDB** | `save 60 1` | Snapshot after 60s if ≥1 key changed |
|
|
||||||
|
|
||||||
### Failure Handling
|
|
||||||
|
|
||||||
- **Redis crash**: Auto-detected via health check, automatically restarted
|
|
||||||
- **Leader crash**: Lock released, followers detect via stale timestamp, re-election occurs
|
|
||||||
- **Graceful shutdown**: SIGTERM sent to Redis (10s timeout), then SIGKILL if needed
|
|
||||||
|
|
||||||
## Metadata Storage
|
|
||||||
|
|
||||||
meta-core uses a flat key-value schema in Redis for storing file metadata.
|
|
||||||
|
|
||||||
### Storage Format
|
|
||||||
|
|
||||||
```
|
|
||||||
Key: /file/{hashId}/{property}
|
|
||||||
Value: string
|
|
||||||
|
|
||||||
Examples:
|
|
||||||
/file/midhash256:abc123/title → "Movie Title"
|
|
||||||
/file/midhash256:abc123/year → "2024"
|
|
||||||
/file/midhash256:abc123/genres → "action,drama"
|
|
||||||
/file/midhash256:abc123/filePath → "movies/Movie.mkv"
|
|
||||||
```
|
|
||||||
|
|
||||||
### Storage Client Features
|
|
||||||
|
|
||||||
| Feature | Details |
|
|
||||||
|---------|---------|
|
|
||||||
| Connection Pool | 10 connections |
|
|
||||||
| Timeouts | 5s dial, 30s read/write |
|
|
||||||
| Batch Operations | Pipelined writes for performance |
|
|
||||||
| Key Scanning | SCAN-based iteration (non-blocking) |
|
|
||||||
| CID Lookup | Find file paths by poster/backdrop CID |
|
|
||||||
|
|
||||||
## Service Discovery
|
|
||||||
|
|
||||||
Services register by writing JSON to `/meta-core/services/{name}.json`.
|
|
||||||
|
|
||||||
### Registration Format
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"name": "meta-sort",
|
|
||||||
"version": "2.0.0",
|
|
||||||
"api": "http://10.0.1.50:8180",
|
|
||||||
"status": "running",
|
|
||||||
"pid": 12345,
|
|
||||||
"hostname": "meta-sort-dev",
|
|
||||||
"startedAt": "2024-01-01T00:00:00Z",
|
|
||||||
"lastHeartbeat": "2024-01-01T00:01:00Z",
|
|
||||||
"capabilities": ["meta-core"],
|
|
||||||
"endpoints": {
|
|
||||||
"health": "http://10.0.1.50:9000/health",
|
|
||||||
"meta": "http://10.0.1.50:9000/meta"
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### Heartbeat & Stale Detection
|
|
||||||
|
|
||||||
| Parameter | Default | Purpose |
|
|
||||||
|-----------|---------|---------|
|
|
||||||
| `HEARTBEAT_INTERVAL_MS` | 30000 | How often services update `lastHeartbeat` |
|
|
||||||
| `STALE_THRESHOLD_MS` | 60000 | Mark service "stale" if heartbeat older than this |
|
|
||||||
|
|
||||||
Services are automatically marked as `stale` when discovered if their `lastHeartbeat` exceeds the threshold. On graceful shutdown, services remove their registration file.
|
|
||||||
|
|
||||||
## Development
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Format code
|
|
||||||
make fmt
|
|
||||||
|
|
||||||
# Run linter
|
|
||||||
make lint
|
|
||||||
|
|
||||||
# Run tests with coverage
|
|
||||||
make test-cover
|
|
||||||
|
|
||||||
# Clean build artifacts
|
|
||||||
make clean
|
|
||||||
```
|
|
||||||
|
|
||||||
## Internal Packages
|
|
||||||
|
|
||||||
| Package | Purpose |
|
| Package | Purpose |
|
||||||
|---------|---------|
|
|---------|---------|
|
||||||
| `cmd/meta-core` | Main entry point, startup/shutdown orchestration |
|
| `cmd/meta-core` | Entry point; Redis connect + schema sentinel + service wiring. |
|
||||||
| `internal/config` | Environment-based configuration with defaults |
|
| `internal/config` | Env-driven configuration, path helpers. |
|
||||||
| `internal/leader` | Election system (`election.go`) and Redis manager (`redis.go`) |
|
| `internal/leader` | `LeaderInfoProvider` (builds the leader info from local hostname/IP/config; no election logic). |
|
||||||
| `internal/storage` | Redis client wrapper with flat key-value operations |
|
| `internal/storage` | Redis wrapper (`client.go`), UUIDv7 minting (`uuid.go`), CID reverse index (`cid_resolution.go`), schema sentinel (`schema_version.go`), dual-root migration (`dual_root_migration.go`). |
|
||||||
| `internal/discovery` | Service registration, heartbeat loop, discovery |
|
| `internal/cid` | CID parsing + algorithm ranking for canonical-CID selection. |
|
||||||
| `internal/api` | HTTP server, router, and all endpoint handlers |
|
| `internal/discovery` | Service registry + dead-service cleaner. |
|
||||||
| `internal/events` | Redis keyspace notification publisher to meta:events |
|
| `internal/events` | Keyspace-notification → `meta:events` stream publisher (`meta_publisher.go`). |
|
||||||
| `internal/watcher` | File system scanning, MidHash256 computation, event dispatch |
|
| `internal/api` | HTTP server + all handlers + SSE event endpoints. |
|
||||||
| `internal/watchers` | Polling-based watcher configuration management |
|
| `internal/schema` | Live per-field schema indexer (consumes `meta:events`). |
|
||||||
| `internal/mounts` | SMB/rclone mount configuration and health monitoring |
|
| `internal/snapshot` | Snapshot export / import / wipe. |
|
||||||
| `internal/webdav` | WebDAV protocol handler with caching integration |
|
| `internal/watcher` | File system scanner, MidHash256 computation, dispatcher, state registry. |
|
||||||
|
| `internal/watchers` | Polling-based watcher configurations (manager + poller + handlers). |
|
||||||
|
| `internal/mounts` | rclone mount manager, lifecycle handlers, stats poller. |
|
||||||
|
| `internal/webdav` | WebDAV handler (mounted at `/webdav/*`; nginx handles caching upstream). |
|
||||||
|
|
||||||
### Startup Sequence
|
### Startup sequence
|
||||||
|
|
||||||
```
|
```
|
||||||
1. Load configuration from environment
|
1. Load configuration from environment
|
||||||
2. Create storage client (Redis wrapper)
|
2. Build LeaderInfoProvider (no election — the bash gate already won)
|
||||||
3. Initialize leader election
|
3. Connect to local Redis (retry up to 30× / 1s)
|
||||||
4. Register role transition callbacks
|
4. EnsureSchemaVersion — abort if the existing Redis layout is stale
|
||||||
5. Start election (acquire lock or become follower)
|
5. Start service discovery + dead-service cleaner
|
||||||
6. Start service discovery (register + heartbeat loop)
|
6. Construct API server (initialises watcher, mounts, WebDAV)
|
||||||
7. Initialize dead service cleaner
|
7. Start API server; on first connect, start MetaPublisher + schema Indexer
|
||||||
8. Initialize file watcher (if enabled)
|
and republish existing metadata onto meta:events
|
||||||
9. Initialize mount manager
|
8. Wait for SIGINT/SIGTERM, shutdown in reverse order
|
||||||
10. Start HTTP API server (WebDAV caching handled by nginx)
|
|
||||||
11. Wait for SIGINT/SIGTERM
|
|
||||||
12. Shutdown in reverse order
|
|
||||||
```
|
|
||||||
|
|
||||||
## Integration
|
|
||||||
|
|
||||||
### As Sidecar in Docker
|
|
||||||
|
|
||||||
Add to existing service's supervisord config:
|
|
||||||
|
|
||||||
```ini
|
|
||||||
[program:meta-core]
|
|
||||||
command=/usr/local/bin/meta-core
|
|
||||||
priority=10
|
|
||||||
autostart=true
|
|
||||||
autorestart=true
|
|
||||||
```
|
|
||||||
|
|
||||||
### Client Usage (TypeScript)
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
const response = await fetch('http://localhost:9000/meta/' + hashId);
|
|
||||||
const { metadata } = await response.json();
|
|
||||||
```
|
|
||||||
|
|
||||||
### Client Usage (Python)
|
|
||||||
|
|
||||||
```python
|
|
||||||
import requests
|
|
||||||
response = requests.get(f'http://localhost:9000/meta/{hash_id}')
|
|
||||||
metadata = response.json()['metadata']
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## Dependencies
|
## Dependencies
|
||||||
|
|
||||||
| Dependency | Version | Purpose |
|
| Dependency | Purpose |
|
||||||
|------------|---------|---------|
|
|------------|---------|
|
||||||
| `github.com/gorilla/mux` | v1.8.1 | HTTP router |
|
| `github.com/gorilla/mux` | HTTP router |
|
||||||
| `github.com/redis/go-redis/v9` | v9.4.0 | Redis client |
|
| `github.com/redis/go-redis/v9` | Redis client |
|
||||||
| `golang.org/x/net` | v0.20.0 | WebDAV support |
|
| `github.com/google/uuid` | UUIDv7 minting |
|
||||||
| `github.com/google/uuid` | v1.6.0 | UUID generation |
|
| `golang.org/x/net` | WebDAV support |
|
||||||
| Go stdlib | - | context, encoding/json, net, os, sync, syscall |
|
| Go stdlib | context, net/http, os, sync, syscall |
|
||||||
|
|
||||||
### Runtime Requirements
|
Runtime requirements: Go 1.21+ at build time; the container ships Redis,
|
||||||
|
rclone, nginx, and supervisord alongside the binary.
|
||||||
- **Go 1.21+** for building
|
|
||||||
- **Redis** binary available in PATH (leader spawns it)
|
|
||||||
- **Shared filesystem** accessible by all instances (for flock)
|
|
||||||
|
|
||||||
## License
|
## License
|
||||||
|
|
||||||
|
|||||||
+172
-26
@@ -2,11 +2,17 @@
|
|||||||
|
|
||||||
## Status
|
## Status
|
||||||
|
|
||||||
**Proposed** — design accepted; implementation pending. Builds on
|
**Landed** — all four PRs (A/B/C/D) deployed and verified end-to-end on the
|
||||||
[uuid-rooted-metadata.md](uuid-rooted-metadata.md), which assumes meta-core
|
dev stack. Redis is unreachable from outside `metacore-app`; every metadata
|
||||||
is the single writer to the metadata keyspace. That assumption is currently
|
read, write, and event flows through meta-core's HTTP+SSE surface. Builds
|
||||||
violated by meta-sort, and this doc is the plan to put it back on solid
|
on [uuid-rooted-metadata.md](uuid-rooted-metadata.md), which assumes
|
||||||
ground.
|
meta-core is the single writer to the metadata keyspace — an assumption
|
||||||
|
that's now actually enforced rather than aspirational.
|
||||||
|
|
||||||
|
See "Implementation outcome" at the bottom for the delta between this doc's
|
||||||
|
plan and what shipped (notably the write-path CID resolution and the
|
||||||
|
dual-root migration sweep, which weren't in the original plan but turned
|
||||||
|
out to be load-bearing).
|
||||||
|
|
||||||
## Motivation
|
## Motivation
|
||||||
|
|
||||||
@@ -441,6 +447,10 @@ Closing the network access (PR D) is the moment the lock becomes a
|
|||||||
lock; everything before it is opt-in migration that consumers can adopt
|
lock; everything before it is opt-in migration that consumers can adopt
|
||||||
at their own pace.
|
at their own pace.
|
||||||
|
|
||||||
|
**All four PRs have landed.** What follows is the original plan as
|
||||||
|
written; see "Implementation outcome" at the bottom for the delta
|
||||||
|
between plan and ship.
|
||||||
|
|
||||||
### PR A — SSE endpoints in meta-core
|
### PR A — SSE endpoints in meta-core
|
||||||
|
|
||||||
Adds `GET /api/events/files` and `GET /api/events/meta`. Implements the
|
Adds `GET /api/events/files` and `GET /api/events/meta`. Implements the
|
||||||
@@ -584,28 +594,34 @@ Stable `error` slugs the client should switch on:
|
|||||||
| `storage_unavailable` | 503 | yes | Redis transient. Caller should retry with backoff. |
|
| `storage_unavailable` | 503 | yes | Redis transient. Caller should retry with backoff. |
|
||||||
| `internal` | 500 | yes (cautiously) | Unhandled server-side error. Retry once; escalate if persistent. |
|
| `internal` | 500 | yes (cautiously) | Unhandled server-side error. Retry once; escalate if persistent. |
|
||||||
|
|
||||||
Today's handlers return `{error, message}` strings; PR A is the natural
|
**Implemented.** Every 4xx/5xx response across `internal/api`,
|
||||||
place to introduce the typed shape (along with the new SSE endpoints
|
`internal/mounts`, `internal/watcher`, and `internal/watchers` now
|
||||||
which will need their own error responses anyway). Going through a
|
emits the typed envelope. The slug-mapping helper is duplicated across
|
||||||
typed envelope from the start avoids the retrofit cost.
|
those packages (one ~15-line `slugForStatus` per package) to avoid an
|
||||||
|
import cycle on the api package; consolidating it into a shared
|
||||||
|
`internal/httperr` package is a worthwhile follow-up but not load-bearing.
|
||||||
|
|
||||||
## Remaining open questions
|
## Remaining open questions
|
||||||
|
|
||||||
The audit closed everything that was answerable. What's left:
|
Mostly answered post-implementation:
|
||||||
|
|
||||||
- **Deprecation timeline for `/api/metadata/{hashId}`.** It stays during
|
- **Deprecation timeline for `/api/metadata/{hashId}`.** Still open. The
|
||||||
the migration (editor uses it), but once we've consolidated on
|
editor's React UI continues to call it; the surface itself was untouched
|
||||||
`/meta/{hash}` for service-to-service writes the editor could migrate
|
by this work. Worth a follow-up to consolidate the editor onto
|
||||||
too and we'd delete the duplicate surface. Not urgent; revisit
|
`/meta/{hash}` and delete the parallel routes.
|
||||||
post-PR D when everything else is stable.
|
|
||||||
|
|
||||||
- **What does `/api/events/files` emit during the watcher's initial
|
- **Initial bootstrap flood on `/api/events/files`.** Resolved in
|
||||||
bootstrap scan?** When meta-core starts on a populated library, the
|
practice: meta-fuse uses `initialCursor: '0-0'` to deliberately
|
||||||
watcher discovers N files in quick succession and publishes N `add`
|
replay every entry on each restart and the StreamingStateBuilder
|
||||||
events. A consumer subscribing for the first time will receive that
|
handles the resulting burst (~2.5k property events on a 50-file
|
||||||
whole flood. Probably fine — consumers are designed to handle bursts
|
library) without issue. meta-sort persists its cursor so only sees
|
||||||
— but worth a smoke test once PR A lands to confirm meta-share's
|
the flood once per fresh deploy. meta-stremio filters to "interesting
|
||||||
ingest doesn't trip on the cardinality.
|
fields" before triggering cache invalidation, so the burst is mostly
|
||||||
|
no-ops.
|
||||||
|
|
||||||
|
- **meta-dup SSE port.** Still pending — meta-dup is in no active
|
||||||
|
compose file. When revived, copy meta-fuse's pattern wholesale
|
||||||
|
(SSEEventClient + MetaCoreApiClient). Tracked separately.
|
||||||
|
|
||||||
## Non-goals
|
## Non-goals
|
||||||
|
|
||||||
@@ -619,7 +635,137 @@ The audit closed everything that was answerable. What's left:
|
|||||||
- **Introducing new transport, broker, proxy, or sidecar.** Everything
|
- **Introducing new transport, broker, proxy, or sidecar.** Everything
|
||||||
builds on what's deployed today.
|
builds on what's deployed today.
|
||||||
- **Migrating meta-fuse / meta-stremio / meta-dup direct Redis reads
|
- **Migrating meta-fuse / meta-stremio / meta-dup direct Redis reads
|
||||||
via XSCAN/GET** (separate from event-stream reads). These are
|
via XSCAN/GET** (separate from event-stream reads). ~~Was a non-goal~~
|
||||||
read-only and don't break schema invariants, but they will fail once
|
ended up being required for PR D's network closure to be safe.
|
||||||
PR D closes Redis access. PR C is the migration window for them —
|
meta-fuse and meta-stremio migrated; meta-dup deferred (not deployed).
|
||||||
the same SSE adoption work covers it.
|
See PR C / "Implementation outcome" for details.
|
||||||
|
|
||||||
|
## Implementation outcome
|
||||||
|
|
||||||
|
What actually shipped vs. what this doc planned.
|
||||||
|
|
||||||
|
### Delta against the original plan
|
||||||
|
|
||||||
|
- **Read-path migration for meta-fuse and meta-stremio is now in scope
|
||||||
|
and done.** The original "out of scope" carve-out was wrong: closing
|
||||||
|
the Redis network port doesn't just break stream consumers, it also
|
||||||
|
breaks every `kvClient.get` / `redis_storage.get_all_videos` call.
|
||||||
|
Both services grew their own `MetaCoreApiClient` (TS for meta-fuse,
|
||||||
|
Python for meta-stremio) and now route reads through `/meta/{hash}`
|
||||||
|
+ `/meta` + `/api/file/{cid}/info`.
|
||||||
|
- **meta-stremio's HTTP mode auto-recovers from transient meta-core
|
||||||
|
outages.** `is_connected` re-probes meta-core's `/health` on every
|
||||||
|
call (cheap; a stateless GET), so a metacore-app restart no longer
|
||||||
|
latches the addon into a permanent disconnected state. The
|
||||||
|
background reconnect thread is only needed for legacy Redis mode
|
||||||
|
where the socket itself is sticky.
|
||||||
|
- **Hard `redisUrl` guard is gated by `ALLOW_LEGACY_REDIS_URL=1`.** All
|
||||||
|
three consumer LeaderClients (TS x2 + Python) throw on seeing the
|
||||||
|
field by default. Setting the env var downgrades to a warn — escape
|
||||||
|
hatch for a deliberate temporary rollback to a meta-core that still
|
||||||
|
publishes the field.
|
||||||
|
- **Write-path CID resolution** turned out to be a hard prerequisite
|
||||||
|
for the lockdown to work in practice. Documented separately below.
|
||||||
|
- **PR D ordering matched the doc** (consumer migration first, then
|
||||||
|
publisher stops emitting, then network closure). The `meta-core`
|
||||||
|
docker-network alias and the `6380:6379` host port mapping are both
|
||||||
|
removed; sibling services reach the HTTP/SSE/WebDAV surface via the
|
||||||
|
canonical `metacore-app` hostname.
|
||||||
|
- **meta-dup is the one service that didn't migrate.** It's in no
|
||||||
|
active compose file. The TS port is straightforward when needed: copy
|
||||||
|
meta-fuse's pattern (SSEEventClient for streams, MetaCoreApiClient
|
||||||
|
for reads).
|
||||||
|
|
||||||
|
### Write-path CID resolution (`storage.ResolveRoot`)
|
||||||
|
|
||||||
|
Not in the original plan but load-bearing.
|
||||||
|
|
||||||
|
**The problem.** meta-sort's PATCH-via-HTTP migration (PR B) routed
|
||||||
|
every write through `/meta/{hash}`. But `{hash}` is the midhash256 CID
|
||||||
|
that meta-sort received via file:events, not the watcher's minted UUID.
|
||||||
|
With no resolution step in the handler, every write went to
|
||||||
|
`file:<midhash>/*` — a parallel root the watcher's UUID never saw. The
|
||||||
|
auto-alias hook on `cid_midhash256=<value>` then OVERWROTE the
|
||||||
|
watcher's `cid:midhash256:<value>` → UUID alias to point at the
|
||||||
|
midhash itself, leaving the UUID stranded with only the watcher's
|
||||||
|
sparse fields (filePath, sizeByte, mtimeNano, duplicates) and the
|
||||||
|
midhash root carrying the rich plugin output. This is the exact
|
||||||
|
"dual roots per file" failure the doc's Motivation section listed as
|
||||||
|
the prior bug — it just moved from "convention violation" to "API
|
||||||
|
contract violation."
|
||||||
|
|
||||||
|
**The fix.** Three layers in `internal/storage/cid_resolution.go`:
|
||||||
|
|
||||||
|
1. **`ResolveRoot(hash)`** — at the entry of every `/meta/{hash}`
|
||||||
|
read/write handler, look up `cid:midhash256:<hash>` in the reverse
|
||||||
|
index. If found, use the UUID as the storage root. Falls through to
|
||||||
|
the bare hash for direct-UUID writes / legacy entries.
|
||||||
|
2. **Self-pointing alias guard in `addAliasLocked`** — refuse to
|
||||||
|
register `cid:<algo>:<v>` → `<v>`. The dual-root bug's chicken-and-
|
||||||
|
egg is: meta-sort writes first, registers self-pointing alias,
|
||||||
|
watcher then can't recover. Refusing the self-write keeps the
|
||||||
|
watcher's legitimate alias intact.
|
||||||
|
3. **`MigrateDualRoots` sweep** (`POST /api/admin/migrate-dual-roots`).
|
||||||
|
One-shot fixer for already-stranded entries: walks every UUID's
|
||||||
|
`midhash256` field to build the orphan index, merges each stranded
|
||||||
|
`file:<midhash>/*` into the matching UUID via `MergeMetadataFlat`,
|
||||||
|
deletes the stranded root, and re-points the alias.
|
||||||
|
|
||||||
|
Verified on the dev stack: 48 stranded entries were reunited; the
|
||||||
|
editor now shows every file with its full plugin metadata under one
|
||||||
|
UUID; future writes stay unified.
|
||||||
|
|
||||||
|
### Typed error envelope
|
||||||
|
|
||||||
|
Done across all four handler packages (`api`, `mounts`, `watcher`,
|
||||||
|
`watchers`). Each has its own `writeError` returning
|
||||||
|
`{error: <slug>, message, retryable}`. The slug-mapping helper is
|
||||||
|
duplicated in each to avoid an import cycle on the `api` package.
|
||||||
|
|
||||||
|
### What's NOT done
|
||||||
|
|
||||||
|
- **meta-dup SSE + read migration.** Not deployed, no test harness.
|
||||||
|
When revived, copy meta-fuse's pattern.
|
||||||
|
- **Editor `/api/metadata/{hashId}` consolidation.** The duplicate
|
||||||
|
editor-internal API surface is untouched; could be retired in a
|
||||||
|
follow-up by porting the editor UI to `/meta/{hash}`.
|
||||||
|
- **Consolidated `internal/httperr` package** for the typed envelope
|
||||||
|
helpers (currently duplicated 4× to avoid the import cycle). Code
|
||||||
|
hygiene; functionally a no-op.
|
||||||
|
|
||||||
|
### Verification done
|
||||||
|
|
||||||
|
- All four service dashboards load cleanly with no console errors.
|
||||||
|
- meta-sort SSE consumer + HTTP write path verified end-to-end by
|
||||||
|
delete + inject + reprocess of Sintel.
|
||||||
|
- meta-fuse SSE consumer rebuilds VFS to 48 files on each boot.
|
||||||
|
- meta-stremio catalog API now surfaces the full library (47 videos,
|
||||||
|
1 movie + 5 series + 46 episodes — was 0/0/0 before the Python
|
||||||
|
read-path migration because LeaderStorage broke when it tried Redis).
|
||||||
|
- `redis-cli` from any container other than `metacore-app` produces a
|
||||||
|
connection refused; `cid:midhash256:<H>` aliases all point to UUIDs
|
||||||
|
(no self-pointing); `file:__index__` has only UUID-style hashIds.
|
||||||
|
- ALLOW_LEGACY_REDIS_URL=1 escape hatch present in all three consumer
|
||||||
|
LeaderClients (TS x2 + Python).
|
||||||
|
|
||||||
|
### Known test-suite gaps (post-implementation bats run)
|
||||||
|
|
||||||
|
Running `docker exec meta-test-runner /app/test/test.sh` after the
|
||||||
|
lockdown shipped reveals a number of failing suites, almost all of
|
||||||
|
which are pre-existing test/environment drift unrelated to this work:
|
||||||
|
|
||||||
|
| Failure | Root cause | Relation to lockdown |
|
||||||
|
|---|---|---|
|
||||||
|
| meta-stremio (~35 tests), stremio-tmdb (6 tests) | Tests call `/manifest.json` directly; dev `.env` sets `STREMIO_HASH_API_SEED=dev-stremio-secret` which puts the addon behind a path-token prefix | Unrelated — pre-existing env mismatch |
|
||||||
|
| meta-fuse FUSE-mount (3 tests) | Tests use `docker exec meta-fuse` but the container is named `metafuse-app` (Yundera prod-parity rename) | Unrelated — predates this work |
|
||||||
|
| meta-fuse WebDAV PROPFIND (4 tests) | Tests use Caddy URL with WebDAV basic-auth credentials that no longer match the hash-lock perimeter | Unrelated — predates this work |
|
||||||
|
| meta-sort "API integration tests" (1 test) | Test runs `docker exec meta-sort pnpm test`; container is `metasort-app` | Unrelated — predates this work |
|
||||||
|
| meta-core "create and delete mount config" (1 test) | Test creates an NFS mount; production code dropped NFS support ("NFS mounts are no longer supported") | Unrelated — predates this work |
|
||||||
|
| service-discovery (9 tests) | Cascading from meta-stremio breakage (test expects unauth `/api/services` on stremio) | Unrelated to lockdown directly |
|
||||||
|
| meta-share / share-1c (multiple) | Tests assert `"phase":"1f"` in `/health`; meta-share has moved past that phase | Unrelated — meta-share evolution |
|
||||||
|
| flat-keys: "filePath field exists" (1 test) | Test SRANDMEMBERs file:__index__; if it lands on a UUID with no filePath (e.g. a manual test write) it fails | Was hit transiently after my interactive testing left stub entries; resolved by cleanup |
|
||||||
|
|
||||||
|
No failure traceable to a real regression from this work was found.
|
||||||
|
The dev-environment test harness needs an unrelated refresh to catch
|
||||||
|
up with the prod-parity container renames, the NFS removal, the
|
||||||
|
STREMIO_HASH_API_SEED default, and meta-share's current phase.
|
||||||
|
|||||||
+391
-898
File diff suppressed because it is too large
Load Diff
+211
-534
@@ -1,595 +1,272 @@
|
|||||||
# Redis Metadata Storage Architecture
|
# Redis Metadata Storage Structure
|
||||||
|
|
||||||
This document describes how MetaMesh stores file metadata in Redis, the key structure, plugin metadata contributions, and special cases like FFmpeg stream handling.
|
How meta-core lays out file metadata in Redis. This document covers the
|
||||||
|
on-disk shape (UUID-rooted flat string keys, CID reverse index,
|
||||||
|
per-root sets) and the schema-version gate. It does **not** enumerate
|
||||||
|
field semantics — that's the job of the repo-root
|
||||||
|
[`METADATA_KEYS.md`](../../../METADATA_KEYS.md), the authoritative key
|
||||||
|
registry for everyone who writes to or reads from metadata.
|
||||||
|
|
||||||
## Overview
|
For the design rationale (why UUIDs instead of content hashes, why a
|
||||||
|
reverse index), see [`uuid-rooted-metadata.md`](uuid-rooted-metadata.md).
|
||||||
|
For the external-access contract (HTTP-only, SSE for events), see
|
||||||
|
[`api-mediated-access.md`](api-mediated-access.md).
|
||||||
|
|
||||||
MetaMesh uses Redis as its primary metadata storage, storing file metadata as **flat string keys**. Each property is stored as a separate Redis key rather than using Redis Hashes. This design enables:
|
## Quick summary
|
||||||
|
|
||||||
- **Redis keyspace notifications** for field-level change detection
|
- **Roots are UUIDv7** encoded as 26-char Crockford Base32 (ULID layout),
|
||||||
- **Pattern-based subscriptions** (e.g., `file:*/tmdb/*` for TMDB updates)
|
not content hashes. Examples in this doc use the placeholder
|
||||||
- **meta:events stream** integration for real-time updates across services
|
`01JKR8XW5T4QMV7AHNJ2DEFGHK`.
|
||||||
- Individual field updates without affecting other properties
|
- **Properties are flat Redis STRING keys**, one per leaf:
|
||||||
- Fast VFS lookups using content-based hash IDs
|
`file:<uuid>/<property/path>` → string value. *Not* Redis Hashes.
|
||||||
|
- **CIDs are reverse-index aliases**: `cid:<algorithm>:<value>` → `<uuid>`
|
||||||
|
(a Redis STRING per known CID).
|
||||||
|
- **Per-root sets**: `file:<uuid>/cids` (every CID), `file:<uuid>/duplicates`
|
||||||
|
(alternate paths with the same content), plus `file:__index__` (every UUID).
|
||||||
|
- **A schema-version sentinel** (`meta-core:schema-version`) refuses to
|
||||||
|
boot against legacy data.
|
||||||
|
|
||||||
## Key Structure
|
Implementation lives in `internal/storage/client.go`,
|
||||||
|
`internal/storage/cid_resolution.go`, `internal/storage/uuid.go`, and
|
||||||
|
`internal/storage/schema_version.go`.
|
||||||
|
|
||||||
### Flat Key Storage
|
## Key structure
|
||||||
|
|
||||||
Each file property is stored as a separate Redis STRING key:
|
### Roots
|
||||||
|
|
||||||
```
|
```
|
||||||
Key Pattern: file:{hashId}/{property/path}
|
file:<uuid>/<property/path> → Redis STRING (always)
|
||||||
Key Type: STRING
|
|
||||||
Value: String value
|
|
||||||
```
|
```
|
||||||
|
|
||||||
**Example Redis commands:**
|
UUIDs are produced by `storage.NewUUID()` (`internal/storage/uuid.go`):
|
||||||
```bash
|
UUIDv7 → 26-char Crockford Base32 (skips I/L/O/U; ULID-compatible). The
|
||||||
# Store file metadata (each property is a separate key)
|
time-sortable prefix lets you `KEYS file:01JKR8*` to find roots minted
|
||||||
SET "file:bafkr4ih5kapbjzq.../cid_midhash256" "bafkr4ih5kapbjzq..."
|
in a given window, which restored some of the debug-ability lost when
|
||||||
SET "file:bafkr4ih5kapbjzq.../filePath" "/files/watch/movie.mkv"
|
midhash256 stopped being the root.
|
||||||
SET "file:bafkr4ih5kapbjzq.../fileinfo/duration" "7200.5"
|
|
||||||
SET "file:bafkr4ih5kapbjzq.../fileinfo/formatName" "matroska,webm"
|
|
||||||
|
|
||||||
# Retrieve single field
|
Properties use `/` as the path separator. Nested objects in source code
|
||||||
GET "file:bafkr4ih5kapbjzq.../fileinfo/duration"
|
are flattened to flat keys on write and reconstructed on read. Examples:
|
||||||
|
|
||||||
# Retrieve all metadata for a file (scan by prefix)
|
|
||||||
SCAN 0 MATCH "file:bafkr4ih5kapbjzq.../*" COUNT 1000
|
|
||||||
```
|
|
||||||
|
|
||||||
### Why Flat Keys?
|
|
||||||
|
|
||||||
The flat key structure (vs. Redis Hashes) enables:
|
|
||||||
|
|
||||||
1. **Keyspace Notifications**: Redis publishes events when individual keys change
|
|
||||||
2. **Pattern Subscriptions**: Services can subscribe to `__keyspace@0__:file:*/tmdb/*`
|
|
||||||
3. **Selective Updates**: Update one field without touching others
|
|
||||||
4. **Stream Integration**: The `meta:events` stream captures field-level changes
|
|
||||||
|
|
||||||
### File Index
|
|
||||||
|
|
||||||
A Redis Set tracks all known file hash IDs:
|
|
||||||
|
|
||||||
```
|
```
|
||||||
Key: file:__index__
|
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/filePath = "/files/watch/movies/Inception.2010.mkv"
|
||||||
Type: Set
|
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/sizeByte = "15032385536"
|
||||||
Value: {hashId1, hashId2, hashId3, ...}
|
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/mtimeNano = "1729872000000000000"
|
||||||
|
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/title = "Inception"
|
||||||
|
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/fileinfo/duration = "8878.5"
|
||||||
|
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/fileinfo/streamdetails/video/0/codec = "h264"
|
||||||
|
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/tmdb/poster = "bafkreih..."
|
||||||
```
|
```
|
||||||
|
|
||||||
This enables fast bulk queries like "get all files" without scanning keys.
|
### Reverse-index aliases (CIDs)
|
||||||
|
|
||||||
|
Every CID known for a file is registered as its own Redis STRING:
|
||||||
|
|
||||||
|
```
|
||||||
|
cid:<algorithm>:<value> → <uuid>
|
||||||
|
```
|
||||||
|
|
||||||
|
Examples:
|
||||||
|
|
||||||
|
```
|
||||||
|
cid:midhash256:bafkreih5kapbjzq... → 01JKR8XW5T4QMV7AHNJ2DEFGHK
|
||||||
|
cid:sha256:abc... → 01JKR8XW5T4QMV7AHNJ2DEFGHK
|
||||||
|
cid:ipfs:bafy... → 01JKR8XW5T4QMV7AHNJ2DEFGHK
|
||||||
|
```
|
||||||
|
|
||||||
|
The reverse index makes `GET /api/meta/<cid>` an O(1) lookup for any
|
||||||
|
registered CID, independent of which hash algorithm "primary" — there is
|
||||||
|
no primary. The auto-alias hooks in `SetProperty` and `SetMetadataFlat`
|
||||||
|
(see `internal/storage/cid_resolution.go`) catch any write of a `cid_*`
|
||||||
|
or `midhash256` field and register the alias automatically, so plugins
|
||||||
|
don't need to know about the reverse index.
|
||||||
|
|
||||||
|
### Per-root sets
|
||||||
|
|
||||||
|
```
|
||||||
|
file:<uuid>/cids → Redis SET of "<algorithm>:<value>" tokens
|
||||||
|
file:<uuid>/canonical_cid → STRING — the highest-ranked CID for advertising externally
|
||||||
|
file:<uuid>/duplicates → Redis SET of alternate paths with same content
|
||||||
|
```
|
||||||
|
|
||||||
|
`canonical_cid` is reconciled by `reconcileCanonicalCIDLocked` whenever
|
||||||
|
the `cids` set changes; ranking is done by `internal/cid/rank.go`.
|
||||||
|
|
||||||
|
### Global index
|
||||||
|
|
||||||
|
```
|
||||||
|
file:__index__ → Redis SET of all UUIDs
|
||||||
|
```
|
||||||
|
|
||||||
|
`internal/storage/client.go::GetAllHashIDs` reads this set to enumerate
|
||||||
|
every file without scanning the keyspace.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Get all hash IDs
|
SMEMBERS file:__index__ # all UUIDs
|
||||||
SMEMBERS "file:__index__"
|
SCARD file:__index__ # total file count
|
||||||
|
SISMEMBER file:__index__ <uuid>
|
||||||
# Count total files
|
|
||||||
SCARD "file:__index__"
|
|
||||||
|
|
||||||
# Check if a file exists
|
|
||||||
SISMEMBER "file:__index__" "bafkr4ih5kapbjzq..."
|
|
||||||
```
|
```
|
||||||
|
|
||||||
### Keyspace Notifications
|
### Schema-version sentinel
|
||||||
|
|
||||||
Redis keyspace notifications are enabled to detect metadata changes. This requires the `notify-keyspace-events` configuration:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Required Redis config (set by meta-core)
|
|
||||||
CONFIG SET notify-keyspace-events "K$"
|
|
||||||
|
|
||||||
# K = Keyspace events (published on __keyspace@<db>__:<key>)
|
|
||||||
# $ = String commands (SET, etc.)
|
|
||||||
```
|
|
||||||
|
|
||||||
Services can subscribe to keyspace events for real-time updates:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Subscribe to all file key changes
|
|
||||||
PSUBSCRIBE '__keyspace@0__:file:*'
|
|
||||||
|
|
||||||
# Subscribe to specific field pattern (e.g., TMDB updates)
|
|
||||||
PSUBSCRIBE '__keyspace@0__:file:*/tmdb/*'
|
|
||||||
```
|
|
||||||
|
|
||||||
### meta:events Stream
|
|
||||||
|
|
||||||
The `meta:events` stream provides reliable event delivery for file system events from meta-core:
|
|
||||||
|
|
||||||
```
|
```
|
||||||
Stream Key: meta:events
|
meta-core:schema-version → STRING (currently "2")
|
||||||
Entry Fields:
|
|
||||||
- type: add | change | delete | rename | reset
|
|
||||||
- path: File path relative to /files
|
|
||||||
- midhash256: Content hash (for add/change events)
|
|
||||||
- oldPath: Previous path (for rename events)
|
|
||||||
- timestamp: Event timestamp
|
|
||||||
```
|
```
|
||||||
|
|
||||||
Consumer groups enable reliable processing with acknowledgment:
|
Set at first boot, checked on every subsequent boot
|
||||||
|
(`internal/storage/schema_version.go`). If the sentinel is missing but
|
||||||
|
`file:__index__` is non-empty, the gate refuses to boot (legacy data).
|
||||||
|
If the sentinel is present but doesn't match `SchemaVersion`, the gate
|
||||||
|
also refuses to boot. Alpha-stage policy: operators wipe and restart;
|
||||||
|
there is no automated migration.
|
||||||
|
|
||||||
```bash
|
## Why flat STRING keys?
|
||||||
# Create consumer group
|
|
||||||
XGROUP CREATE meta:events mygroup 0 MKSTREAM
|
|
||||||
|
|
||||||
# Read new events (blocking)
|
The store does not use Redis Hashes. The reasons are:
|
||||||
XREADGROUP GROUP mygroup myconsumer BLOCK 5000 STREAMS meta:events >
|
|
||||||
|
|
||||||
# Acknowledge processed events
|
1. **Field-level keyspace notifications.** Redis publishes
|
||||||
XACK meta:events mygroup <entry-id>
|
`__keyspace@0__:<key>` on every SET/DEL of a STRING key. With Hashes
|
||||||
```
|
you'd only get one notification on the hash itself. meta-core's
|
||||||
|
`MetaPublisher` (`internal/events/meta_publisher.go`) consumes these
|
||||||
|
notifications and publishes structured events to the `meta:events`
|
||||||
|
Redis Stream.
|
||||||
|
2. **Selective updates without read-modify-write.** A plugin writing
|
||||||
|
`tmdb/poster` doesn't have to read the rest of the document.
|
||||||
|
3. **Compatibility with the CID alias hooks.** Field writes can trigger
|
||||||
|
alias-maintenance work in `SetProperty` without the plugin caring.
|
||||||
|
|
||||||
## Primary Hash ID
|
## Reads and writes
|
||||||
|
|
||||||
The primary identifier for each file is `cid_midhash256` - an IPFS-compatible CID generated from:
|
The Go API in `internal/storage/client.go`:
|
||||||
|
|
||||||
1. SHA-256 hash of the middle 1MB of the file
|
| Method | Behaviour |
|
||||||
2. File size in bytes
|
|---|---|
|
||||||
3. IPFS CID v1 encoding (multicodec 0x1000)
|
| `GetMetadataFlat(uuid)` | `SCAN` for `file:<uuid>/*`, `MGET` the values, return `map[string]string`. |
|
||||||
|
| `SetMetadataFlat(uuid, m)` | Pipeline `SET` per field; `SADD file:__index__`; auto-register any `cid_*` aliases. |
|
||||||
|
| `MergeMetadataFlat(uuid, m)` | Like `SetMetadataFlat` but doesn't pre-clear; used by `PATCH /meta/{hash}`. |
|
||||||
|
| `GetProperty(uuid, field)` | `GET file:<uuid>/<field>`. |
|
||||||
|
| `SetProperty(uuid, field, value)` | `SET file:<uuid>/<field>`; auto-register alias if the field is a CID. |
|
||||||
|
| `DeleteRoot(uuid)` | `SCAN` + `DEL` for `file:<uuid>/*`, unmap every reverse-index entry. |
|
||||||
|
| `Mint(filePath, size, mtimeNano)` | `NewUUID()` + write `filePath/sizeByte/mtimeNano` + add to index. |
|
||||||
|
| `AddAlias(uuid, "<algo>:<value>")` | Set `cid:<algo>:<value>` → `<uuid>`, add to `file:<uuid>/cids`, reconcile `canonical_cid`. |
|
||||||
|
| `ResolveCID("<algo>:<value>")` | `GET cid:<algo>:<value>` → uuid. |
|
||||||
|
|
||||||
**Example:** `bafkr4ih5kapbjzqvylwj7kx7zmkpxn6qj5xw...`
|
## Reconstruction (flat → nested)
|
||||||
|
|
||||||
This provides:
|
When the HTTP API returns a document, it reconstructs the flattened
|
||||||
- Content-based addressing (same file = same ID)
|
storage back into a nested JSON object:
|
||||||
- Fast computation (only reads 1MB)
|
|
||||||
- IPFS compatibility for future distributed storage
|
|
||||||
|
|
||||||
### Collision Handling
|
|
||||||
|
|
||||||
If two different files produce the same midhash (rare), the system falls back to full SHA-256:
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
// Normal case
|
|
||||||
hashId = midHash256; // "bafkr4ih5kapbjzq..."
|
|
||||||
|
|
||||||
// Collision detected (different filePath with same midhash)
|
|
||||||
hashId = metadata['cid_sha2-256']; // Full hash fallback
|
|
||||||
```
|
|
||||||
|
|
||||||
## Metadata Flattening
|
|
||||||
|
|
||||||
Nested objects are flattened into path-based keys for efficient Redis storage.
|
|
||||||
|
|
||||||
### Flattening Rules
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
// Input (nested object)
|
|
||||||
{
|
|
||||||
title: "Inception",
|
|
||||||
video: {
|
|
||||||
codec: "h265",
|
|
||||||
width: "1920"
|
|
||||||
},
|
|
||||||
streams: [
|
|
||||||
{ language: "eng" },
|
|
||||||
{ language: "jpn" }
|
|
||||||
]
|
|
||||||
}
|
|
||||||
|
|
||||||
// Output (flattened for Redis)
|
|
||||||
title = "Inception"
|
|
||||||
video/codec = "h265"
|
|
||||||
video/width = "1920"
|
|
||||||
streams/0/language = "eng"
|
|
||||||
streams/1/language = "jpn"
|
|
||||||
```
|
|
||||||
|
|
||||||
**Key rules:**
|
|
||||||
- Nested objects use `/` as path separator
|
|
||||||
- Arrays use numeric indices (0, 1, 2...)
|
|
||||||
- All values stored as strings
|
|
||||||
- `null`/`undefined` stored as empty strings
|
|
||||||
- Numbers and booleans converted via `String()`
|
|
||||||
|
|
||||||
### Reconstruction
|
|
||||||
|
|
||||||
When reading, the flattened structure is reconstructed into nested objects:
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
const metadata = await kvClient.getMetadataFlat(hashId);
|
|
||||||
// Returns: { title: "Inception", video: { codec: "h265", width: "1920" }, ... }
|
|
||||||
```
|
|
||||||
|
|
||||||
## Plugin Metadata Structure
|
|
||||||
|
|
||||||
Each plugin writes metadata using a prefixed key pattern:
|
|
||||||
|
|
||||||
```
|
```
|
||||||
{plugin-id}/{field} # Simple field
|
flat nested
|
||||||
{plugin-id}/{group}/{field} # Grouped field
|
───────────────────────────────────── ───────────────────────────
|
||||||
{plugin-id}/{group}/{index}/{field} # Indexed field (for arrays)
|
title = "Inception" title: "Inception"
|
||||||
|
fileinfo/duration = "8878.5" fileinfo: { duration: "8878.5",
|
||||||
|
fileinfo/streamdetails/ streamdetails: { video: { 0: {
|
||||||
|
video/0/codec = "h264" codec: "h264", ... } }, ... } }
|
||||||
|
streams/0/language = "eng" streams: [ { language: "eng" }, ... ]
|
||||||
```
|
```
|
||||||
|
|
||||||
### Plugin Contribution Flow
|
Rules:
|
||||||
|
|
||||||
|
- `/` is the path separator.
|
||||||
|
- Numeric path segments (`0`, `1`, ...) become array indices on read.
|
||||||
|
- All values are strings on the wire; type interpretation is the
|
||||||
|
caller's job (and is documented per-field in `METADATA_KEYS.md`).
|
||||||
|
- `null`/`undefined` round-trip as empty strings.
|
||||||
|
|
||||||
|
## Plugin contribution flow
|
||||||
|
|
||||||
|
Plugins don't talk to Redis. They write metadata via meta-core's HTTP
|
||||||
|
API (`PUT /meta/{hash}/{key...}` or `PATCH /meta/{hash}`). Each plugin
|
||||||
|
writes under a prefix:
|
||||||
|
|
||||||
```
|
```
|
||||||
File Discovered
|
{plugin-id}/<field> single value
|
||||||
↓
|
{plugin-id}/<group>/<field> grouped
|
||||||
[FileProcessorPiscina]
|
{plugin-id}/<group>/<n>/<field> indexed (array)
|
||||||
├─ Compute midhash256
|
|
||||||
├─ Store preliminary metadata
|
|
||||||
└─ Dispatch plugin tasks
|
|
||||||
↓
|
|
||||||
[Plugin Processing]
|
|
||||||
├─ file-info → fileType, mimeType, sizeByte
|
|
||||||
├─ ffmpeg → fileinfo/duration, fileinfo/streamdetails/...
|
|
||||||
├─ filename-parser → titles/eng, season, episode, videoType
|
|
||||||
├─ anime-detector → anime, titles/jpn, titles/jpl
|
|
||||||
├─ tmdb → tmdbid, plot/eng, genres
|
|
||||||
└─ full-hash → cid_sha2-256, cid_sha1
|
|
||||||
↓
|
|
||||||
[Plugin Callbacks]
|
|
||||||
└─ Store metadata via flat keys (SET per property)
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## Core Metadata Types
|
See [`METADATA_KEYS.md`](../../../METADATA_KEYS.md) for the full,
|
||||||
|
authoritative list of fields each plugin writes (content identification,
|
||||||
|
file stats, video classification, anime detection, subtitle references,
|
||||||
|
Jellyfin metadata, FFmpeg stream details, etc.) and the value formats
|
||||||
|
(`int-string`, `lang3`, `path-relfiles`, `csv-set`, ...). This doc
|
||||||
|
deliberately does not duplicate that registry — it would drift.
|
||||||
|
|
||||||
### HashMeta - Content Identification
|
## Event streams
|
||||||
|
|
||||||
```
|
Two Redis Streams are produced inside meta-core:
|
||||||
cid_crc32 = "bafkr..." # CRC32 (multicodec 0x0132)
|
|
||||||
cid_md5 = "bafkr..." # MD5 (multicodec 0xd5)
|
|
||||||
cid_sha1 = "bafkr..." # SHA-1 (multicodec 0x11)
|
|
||||||
cid_sha2-256 = "bafkr..." # SHA-256 (multicodec 0x12)
|
|
||||||
cid_midhash256 = "bafkr..." # Middle-hash (PRIMARY ID, multicodec 0x1000)
|
|
||||||
```
|
|
||||||
|
|
||||||
### FileStatMeta - File Properties
|
| Stream | Producer | Contents |
|
||||||
|
|---|---|---|
|
||||||
|
| `file:events` | `internal/watcher` dispatcher | `add` / `change` / `delete` / `rename` / `reset` events for files on the `/files` volume. |
|
||||||
|
| `meta:events` | `internal/events.MetaPublisher` | Field-level metadata mutations derived from Redis keyspace notifications. |
|
||||||
|
|
||||||
```
|
External consumers do **not** read these streams directly. They subscribe
|
||||||
fileType = "video" # video, audio, image, text
|
via SSE:
|
||||||
mimeType = "video/x-matroska" # MIME type
|
|
||||||
sizeByte = "4294967296" # File size in bytes
|
|
||||||
```
|
|
||||||
|
|
||||||
### VideoMeta - Classification
|
- `GET /api/events/files` — mirror of `file:events`
|
||||||
|
- `GET /api/events/meta` — mirror of `meta:events`
|
||||||
|
|
||||||
```
|
The SSE wire contract uses `Last-Event-ID` for resume (the opaque
|
||||||
videoType = "movie" # movie, episode, unknown
|
`<ms>-<seq>` Redis Stream entry ID). Heartbeats are SSE comments
|
||||||
originalTitle = "Inception"
|
(`:keep-alive\n\n`) every 30s of silence. If retention has trimmed past
|
||||||
titles/eng = "Inception" # Localized titles
|
a client's cursor, the handler emits one `event: gap` payload before
|
||||||
titles/jpn = "インセプション"
|
resuming. See `internal/api/sse_events.go` and
|
||||||
season = "1"
|
[`api-mediated-access.md`](api-mediated-access.md) for the full
|
||||||
episode = "5"
|
contract.
|
||||||
movieYear = "2010"
|
|
||||||
```
|
|
||||||
|
|
||||||
### AnimeMeta - Anime Detection
|
Internally, meta-core uses a Redis consumer group (XREADGROUP / XACK)
|
||||||
|
inside `MetaPublisher` to drain keyspace notifications, but this is not
|
||||||
```
|
the external surface — external services use SSE.
|
||||||
anime = "true" # "true" or "false"
|
|
||||||
titles/jpn = "進撃の巨人" # Japanese title
|
|
||||||
titles/jpl = "Shingeki no Kyojin" # Romaji
|
|
||||||
```
|
|
||||||
|
|
||||||
### SubtitleMeta - Subtitle References
|
|
||||||
|
|
||||||
```
|
|
||||||
subtitles/eng/0 = "bafkr..." # Reference to subtitle file CID
|
|
||||||
subtitles/eng/1 = "bafks..."
|
|
||||||
subtitles/jpn/0 = "bafkt..."
|
|
||||||
```
|
|
||||||
|
|
||||||
### JellyfinMeta - Comprehensive Metadata
|
|
||||||
|
|
||||||
```
|
|
||||||
imdbid = "tt1375666"
|
|
||||||
tmdbid = "27205"
|
|
||||||
anidbid = "12345"
|
|
||||||
plot/eng = "A movie about..."
|
|
||||||
genre/0 = "Action"
|
|
||||||
genre/1 = "Sci-Fi"
|
|
||||||
rating = "8.8"
|
|
||||||
mpaa = "PG-13"
|
|
||||||
releasedate = "2010-07-16"
|
|
||||||
art/poster = "bafkr..." # Reference to poster image CID
|
|
||||||
art/fanart = "bafks..." # Reference to backdrop image CID
|
|
||||||
```
|
|
||||||
|
|
||||||
## FFmpeg Plugin - Special Case
|
|
||||||
|
|
||||||
The FFmpeg plugin extracts detailed stream information with a specific nested structure.
|
|
||||||
|
|
||||||
### Stream Details Structure
|
|
||||||
|
|
||||||
```
|
|
||||||
fileinfo/duration = "7200.5"
|
|
||||||
fileinfo/formatName = "matroska,webm"
|
|
||||||
fileinfo/streamdetails/video/0/codec = "h264"
|
|
||||||
fileinfo/streamdetails/video/0/width = "1920"
|
|
||||||
fileinfo/streamdetails/video/0/height = "1080"
|
|
||||||
fileinfo/streamdetails/video/0/aspect = "16:9"
|
|
||||||
fileinfo/streamdetails/video/0/bitrate = "8000000"
|
|
||||||
fileinfo/streamdetails/audio/0/codec = "aac"
|
|
||||||
fileinfo/streamdetails/audio/0/language = "eng"
|
|
||||||
fileinfo/streamdetails/audio/0/channels = "6"
|
|
||||||
fileinfo/streamdetails/audio/1/codec = "dts"
|
|
||||||
fileinfo/streamdetails/audio/1/language = "fra"
|
|
||||||
fileinfo/streamdetails/audio/1/channels = "6"
|
|
||||||
fileinfo/streamdetails/subtitle/0/language = "eng"
|
|
||||||
fileinfo/streamdetails/subtitle/0/format = "subrip"
|
|
||||||
fileinfo/streamdetails/subtitle/1/language = "jpn"
|
|
||||||
fileinfo/streamdetails/subtitle/1/format = "ass"
|
|
||||||
fileinfo/streamdetails/embeddedimage/0/type = "cover"
|
|
||||||
```
|
|
||||||
|
|
||||||
### Stream Type Keys
|
|
||||||
|
|
||||||
| Stream Type | Key Pattern | Description |
|
|
||||||
|------------|-------------|-------------|
|
|
||||||
| Video | `fileinfo/streamdetails/video/{n}/...` | Video streams (0, 1, 2...) |
|
|
||||||
| Audio | `fileinfo/streamdetails/audio/{n}/...` | Audio tracks |
|
|
||||||
| Subtitle | `fileinfo/streamdetails/subtitle/{n}/...` | Embedded subtitles |
|
|
||||||
| Embedded Image | `fileinfo/streamdetails/embeddedimage/{n}/...` | Cover art, thumbnails |
|
|
||||||
|
|
||||||
### Common Stream Fields
|
|
||||||
|
|
||||||
**Video streams:**
|
|
||||||
- `codec` - h264, hevc, av1, vp9, etc.
|
|
||||||
- `width`, `height` - Resolution
|
|
||||||
- `aspect` - Aspect ratio
|
|
||||||
- `bitrate` - Video bitrate
|
|
||||||
- `profile` - Codec profile (main, high, etc.)
|
|
||||||
- `level` - Codec level
|
|
||||||
|
|
||||||
**Audio streams:**
|
|
||||||
- `codec` - aac, dts, ac3, flac, etc.
|
|
||||||
- `language` - ISO 639-2 code (eng, jpn, fra)
|
|
||||||
- `channels` - Channel count (2, 6, 8)
|
|
||||||
- `bitrate` - Audio bitrate
|
|
||||||
- `samplerate` - Sample rate
|
|
||||||
|
|
||||||
**Subtitle streams:**
|
|
||||||
- `language` - ISO 639-2 code
|
|
||||||
- `format` - subrip, ass, pgs, vobsub
|
|
||||||
|
|
||||||
## Real-Time Notifications
|
|
||||||
|
|
||||||
MetaMesh provides multiple mechanisms for real-time metadata change notifications:
|
|
||||||
|
|
||||||
### 1. meta:events Stream (Recommended)
|
|
||||||
|
|
||||||
The `meta:events` Redis stream provides reliable, ordered event delivery with consumer groups:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# File system events from meta-core watcher
|
|
||||||
XREAD STREAMS meta:events 0
|
|
||||||
|
|
||||||
# Event types: add, change, delete, rename, reset
|
|
||||||
```
|
|
||||||
|
|
||||||
Benefits:
|
|
||||||
- **Reliable delivery** via consumer groups and acknowledgment
|
|
||||||
- **Replay capability** for catching up after restarts
|
|
||||||
- **Ordered processing** with stream IDs
|
|
||||||
|
|
||||||
### 2. Keyspace Notifications
|
|
||||||
|
|
||||||
For field-level change detection, subscribe to Redis keyspace notifications:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# All file metadata changes
|
|
||||||
PSUBSCRIBE '__keyspace@0__:file:*'
|
|
||||||
|
|
||||||
# Specific plugin updates (e.g., TMDB enrichment)
|
|
||||||
PSUBSCRIBE '__keyspace@0__:file:*/tmdb/*'
|
|
||||||
```
|
|
||||||
|
|
||||||
### Cache Invalidation
|
|
||||||
|
|
||||||
The WebDAV cache in meta-core integrates with keyspace notifications for automatic invalidation:
|
|
||||||
|
|
||||||
```
|
|
||||||
┌─────────────────────────────────────────────────────────────────┐
|
|
||||||
│ Cache Invalidation Flow │
|
|
||||||
│ │
|
|
||||||
│ Redis Keyspace Cache │
|
|
||||||
│ ┌──────────────┐ Invalidator │
|
|
||||||
│ │ SET file:*/ │ ──────────► ┌──────────────┐ │
|
|
||||||
│ │ DEL file:*/ │ subscribe │ Listens to │ │
|
|
||||||
│ └──────────────┘ │ __keyspace@ │ │
|
|
||||||
│ │ 0__:file:* │ │
|
|
||||||
│ └──────┬───────┘ │
|
|
||||||
│ │ │
|
|
||||||
│ ▼ │
|
|
||||||
│ ┌──────────────┐ │
|
|
||||||
│ │ Remove from │ │
|
|
||||||
│ │ LRU cache │ │
|
|
||||||
│ └──────────────┘ │
|
|
||||||
└─────────────────────────────────────────────────────────────────┘
|
|
||||||
```
|
|
||||||
|
|
||||||
Flow:
|
|
||||||
1. Cache invalidator subscribes to `__keyspace@0__:file:*`
|
|
||||||
2. When metadata changes (SET/DEL), invalidator receives notification
|
|
||||||
3. Invalidator removes the corresponding file from the WebDAV cache
|
|
||||||
4. Next access fetches fresh copy from source filesystem
|
|
||||||
|
|
||||||
This ensures cached files stay synchronized with metadata changes.
|
|
||||||
|
|
||||||
### 3. Pub/Sub Channels (Legacy)
|
|
||||||
|
|
||||||
Batch notification channels are still supported:
|
|
||||||
|
|
||||||
```
|
|
||||||
meta-sort:file:batch # Batch file updates (every 5 seconds)
|
|
||||||
meta-sort:scan:reset # Full rescan notification
|
|
||||||
```
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"timestamp": 1704067200000,
|
|
||||||
"changes": [
|
|
||||||
{ "action": "add", "hashId": "bafkr4ih5..." },
|
|
||||||
{ "action": "update", "hashId": "bafkr4ih5..." },
|
|
||||||
{ "action": "remove", "hashId": "bafkr4ih5..." }
|
|
||||||
]
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
## Complete Example
|
|
||||||
|
|
||||||
Here's a complete metadata example for a movie file:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Query all keys for a file
|
|
||||||
redis-cli --scan --pattern 'file:bafkr4ih5kapbjzq.../*'
|
|
||||||
|
|
||||||
# Example flat keys stored in Redis:
|
|
||||||
# file:bafkr4ih5kapbjzq.../cid_midhash256 = "bafkr4ih5kapbjzq..."
|
|
||||||
# file:bafkr4ih5kapbjzq.../filePath = "/files/watch/movies/Inception.2010.1080p.BluRay.x264.mkv"
|
|
||||||
# file:bafkr4ih5kapbjzq.../fileType = "video"
|
|
||||||
# file:bafkr4ih5kapbjzq.../fileinfo/duration = "8878.5"
|
|
||||||
# file:bafkr4ih5kapbjzq.../titles/eng = "Inception"
|
|
||||||
# ...
|
|
||||||
|
|
||||||
# When reconstructed by the API (nested object representation)
|
|
||||||
{
|
|
||||||
# Content identification
|
|
||||||
"cid_midhash256": "bafkr4ih5kapbjzq...",
|
|
||||||
"cid_sha2-256": "bafks7ij2kbpck...",
|
|
||||||
"filePath": "/files/watch/movies/Inception.2010.1080p.BluRay.x264.mkv",
|
|
||||||
|
|
||||||
# File properties
|
|
||||||
"fileType": "video",
|
|
||||||
"mimeType": "video/x-matroska",
|
|
||||||
"sizeByte": "4294967296",
|
|
||||||
|
|
||||||
# Classification
|
|
||||||
"videoType": "movie",
|
|
||||||
"originalTitle": "Inception",
|
|
||||||
"movieYear": "2010",
|
|
||||||
|
|
||||||
# Localized titles
|
|
||||||
"titles": {
|
|
||||||
"eng": "Inception",
|
|
||||||
"fra": "Inception",
|
|
||||||
"jpn": "インセプション"
|
|
||||||
},
|
|
||||||
|
|
||||||
# FFmpeg stream details
|
|
||||||
"fileinfo": {
|
|
||||||
"duration": "8878.5",
|
|
||||||
"formatName": "matroska,webm",
|
|
||||||
"streamdetails": {
|
|
||||||
"video": {
|
|
||||||
"0": {
|
|
||||||
"codec": "h264",
|
|
||||||
"width": "1920",
|
|
||||||
"height": "1080",
|
|
||||||
"aspect": "16:9",
|
|
||||||
"bitrate": "8000000"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"audio": {
|
|
||||||
"0": { "codec": "dts", "language": "eng", "channels": "6" },
|
|
||||||
"1": { "codec": "ac3", "language": "fra", "channels": "6" }
|
|
||||||
},
|
|
||||||
"subtitle": {
|
|
||||||
"0": { "language": "eng", "format": "subrip" },
|
|
||||||
"1": { "language": "fra", "format": "subrip" }
|
|
||||||
}
|
|
||||||
}
|
|
||||||
},
|
|
||||||
|
|
||||||
# External metadata (TMDB)
|
|
||||||
"tmdbid": "27205",
|
|
||||||
"imdbid": "tt1375666",
|
|
||||||
"plot": {
|
|
||||||
"eng": "A thief who steals corporate secrets through dream-sharing technology..."
|
|
||||||
},
|
|
||||||
"genre": ["Action", "Science Fiction", "Adventure"],
|
|
||||||
"rating": "8.4",
|
|
||||||
"releasedate": "2010-07-16",
|
|
||||||
|
|
||||||
# Media assets
|
|
||||||
"art": {
|
|
||||||
"poster": "bafkr4poster...",
|
|
||||||
"fanart": "bafkr4fanart..."
|
|
||||||
},
|
|
||||||
|
|
||||||
# Detection flags
|
|
||||||
"anime": "false"
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
## Key Implementation Files
|
|
||||||
|
|
||||||
| Component | File Path |
|
|
||||||
|-----------|-----------|
|
|
||||||
| Redis Client | `packages/meta-sort/packages/meta-sort-core/src/kv/RedisClient.ts` |
|
|
||||||
| Metadata Utils | `packages/meta-sort/packages/meta-sort-core/src/kv/MetadataUtils.ts` |
|
|
||||||
| KV Interface | `packages/meta-sort/packages/meta-sort-core/src/kv/IKVClient.ts` |
|
|
||||||
| File Processor | `packages/meta-sort/packages/meta-sort-core/src/logic/fileProcessor/FileProcessorPiscina.ts` |
|
|
||||||
| Pub/Sub Pipeline | `packages/meta-sort/packages/meta-sort-core/src/logic/pipeline/StreamingPipeline.ts` |
|
|
||||||
| Type Definitions | `packages/meta-interface/src/lib/metadata-type/*.ts` |
|
|
||||||
| meta-core Storage | `packages/meta-core/internal/storage/client.go` |
|
|
||||||
| meta-core Events | `packages/meta-core/internal/events/publisher.go` |
|
|
||||||
| meta-core Cache | `packages/meta-core/internal/cache/` |
|
|
||||||
|
|
||||||
## Debugging
|
## Debugging
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# List all file hash IDs
|
# Roots
|
||||||
docker exec meta-core-dev redis-cli SMEMBERS 'file:__index__'
|
docker exec metacore-app redis-cli SMEMBERS 'file:__index__'
|
||||||
|
docker exec metacore-app redis-cli SCARD 'file:__index__'
|
||||||
|
docker exec metacore-app redis-cli SRANDMEMBER 'file:__index__'
|
||||||
|
|
||||||
# Count total files
|
# Full document for one root (flat scan)
|
||||||
docker exec meta-core-dev redis-cli SCARD 'file:__index__'
|
docker exec metacore-app redis-cli --scan --pattern 'file:01JKR8XW5T4QMV7AHNJ2DEFGHK/*'
|
||||||
|
|
||||||
# Get a sample hash ID
|
# One property
|
||||||
docker exec meta-core-dev redis-cli SRANDMEMBER 'file:__index__'
|
docker exec metacore-app redis-cli GET 'file:01JKR8XW5T4QMV7AHNJ2DEFGHK/fileinfo/duration'
|
||||||
|
|
||||||
# List all keys for a specific file (flat key scan)
|
# Confirm keys are STRING (not HASH)
|
||||||
docker exec meta-core-dev redis-cli --scan --pattern 'file:bafkr4ih5.../*'
|
docker exec metacore-app redis-cli TYPE 'file:01JKR8XW5T4QMV7AHNJ2DEFGHK/filePath'
|
||||||
|
# string
|
||||||
|
|
||||||
# Get specific field (flat key)
|
# Reverse-index lookups
|
||||||
docker exec meta-core-dev redis-cli GET 'file:bafkr4ih5.../fileinfo/duration'
|
docker exec metacore-app redis-cli GET 'cid:midhash256:bafkreih...' # → uuid
|
||||||
|
docker exec metacore-app redis-cli SMEMBERS 'file:01JKR8XW5T4QMV7AHNJ2DEFGHK/cids'
|
||||||
|
docker exec metacore-app redis-cli GET 'file:01JKR8XW5T4QMV7AHNJ2DEFGHK/canonical_cid'
|
||||||
|
|
||||||
# Get filePath for a file
|
# Schema-version sentinel
|
||||||
docker exec meta-core-dev redis-cli GET 'file:bafkr4ih5.../filePath'
|
docker exec metacore-app redis-cli GET 'meta-core:schema-version'
|
||||||
|
|
||||||
# Verify keys are STRING type (not HASH)
|
# Streams (server-side; external consumers should use SSE)
|
||||||
docker exec meta-core-dev redis-cli TYPE 'file:bafkr4ih5.../filePath'
|
docker exec metacore-app redis-cli XRANGE 'file:events' - + COUNT 10
|
||||||
# Should return: string
|
docker exec metacore-app redis-cli XRANGE 'meta:events' - + COUNT 10
|
||||||
|
|
||||||
# Monitor metadata changes via stream
|
# HTTP equivalents — preferred
|
||||||
docker exec meta-core-dev redis-cli XREAD STREAMS 'meta:events' 0
|
curl -k 'https://metacore-dev.localhost:8083/api/meta/<cid>' # CID → document
|
||||||
|
curl -k 'https://metacore-dev.localhost:8083/meta' # list root IDs
|
||||||
# Monitor keyspace notifications (requires notify-keyspace-events enabled)
|
curl -k 'https://metacore-dev.localhost:8083/meta/<uuid>' # document by UUID
|
||||||
docker exec meta-core-dev redis-cli PSUBSCRIBE '__keyspace@0__:file:*'
|
curl -k 'https://metacore-dev.localhost:8083/api/events/meta' # live SSE
|
||||||
|
curl -k 'https://metacore-dev.localhost:8083/api/kv/info' # KV stats
|
||||||
# meta-core API debugging
|
|
||||||
# Cache status
|
|
||||||
curl http://localhost:9000/api/cache/status
|
|
||||||
|
|
||||||
# Clear cache
|
|
||||||
curl -X POST http://localhost:9000/api/cache/clear
|
|
||||||
|
|
||||||
# Watch events stream (SSE)
|
|
||||||
curl http://localhost:9000/api/events/stream
|
|
||||||
|
|
||||||
# KV browser - storage info
|
|
||||||
curl http://localhost:9000/api/kv/info
|
|
||||||
|
|
||||||
# Scan status
|
|
||||||
curl http://localhost:9000/api/scan/status
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## meta:events Stream
|
## Key implementation files
|
||||||
|
|
||||||
The `meta:events` stream captures metadata change events for real-time notifications:
|
| Component | Path |
|
||||||
|
|---|---|
|
||||||
```bash
|
| Redis client + flat-key ops | `internal/storage/client.go` |
|
||||||
# View recent events
|
| UUIDv7 minting | `internal/storage/uuid.go` |
|
||||||
docker exec meta-core-dev redis-cli XRANGE 'meta:events' - + COUNT 10
|
| CID reverse index + aliases + canonical-CID reconciliation | `internal/storage/cid_resolution.go` |
|
||||||
|
| Schema-version gate | `internal/storage/schema_version.go` |
|
||||||
# Read events from a consumer group
|
| Dual-root migration (legacy clean-up) | `internal/storage/dual_root_migration.go` |
|
||||||
docker exec meta-core-dev redis-cli XREADGROUP GROUP mygroup myconsumer STREAMS 'meta:events' >
|
| Keyspace-notification → `meta:events` publisher | `internal/events/meta_publisher.go` |
|
||||||
```
|
| Watcher → `file:events` dispatcher | `internal/watcher/dispatcher.go` |
|
||||||
|
| SSE event mirror | `internal/api/sse_events.go` |
|
||||||
Stream events contain:
|
| Field semantics (authoritative registry) | [`/METADATA_KEYS.md`](../../../METADATA_KEYS.md) |
|
||||||
- `type`: Event type (add, change, delete, rename, reset)
|
|
||||||
- `path`: File path
|
|
||||||
- `midhash256`: Content hash (for add/change)
|
|
||||||
- `timestamp`: Event timestamp
|
|
||||||
|
|||||||
@@ -2,9 +2,7 @@
|
|||||||
|
|
||||||
## Status
|
## Status
|
||||||
|
|
||||||
**Proposed** — design under discussion. Not yet implemented. Supersedes the
|
**Status:** Implemented (2026 alpha clean-wipe gate landed). Kept for design rationale.
|
||||||
"midhash256 as primary key" model currently described in
|
|
||||||
[metadata-storage-structure.md](metadata-storage-structure.md).
|
|
||||||
|
|
||||||
## Motivation
|
## Motivation
|
||||||
|
|
||||||
|
|||||||
+57
-121
@@ -1,140 +1,76 @@
|
|||||||
# @metazla/meta-editor
|
# @meta-core/editor
|
||||||
|
|
||||||
User-friendly metadata editor for MetaMesh. Search and edit file metadata directly through a web interface.
|
React + Vite metadata browser/editor that ships inside the meta-core
|
||||||
|
container. Walks the UUID-rooted Redis keyspace, edits leaf values,
|
||||||
|
imports/exports snapshots, and consults the live schema indexer.
|
||||||
|
|
||||||
## Features
|
## What it talks to
|
||||||
|
|
||||||
- **Search**: Find files by title, filename, or hash ID
|
The editor is a pure SPA — it has no server of its own. It calls the
|
||||||
- **Edit**: Modify all metadata fields including title, season, episode, year, Jellyfin metadata, and more
|
meta-core HTTP API directly under three prefixes (see `src/api/`):
|
||||||
- **Batch Operations**: Update multiple files simultaneously
|
|
||||||
- **Real-time**: Direct integration with etcd via REST API
|
|
||||||
- **User-friendly**: Clean, intuitive interface with form validation
|
|
||||||
|
|
||||||
## Development
|
- `/api/kv/*` — slash-tree KV browser (`info`, `tree`, `search`, `find`,
|
||||||
|
`value`, `key/{key...}`). This is the UUID-exposing surface — keys look
|
||||||
|
like `file:<uuid>/<property>` and `cid:<algo>:<value>`.
|
||||||
|
- `/api/schema` — live per-field schema (type hints, value breakdowns),
|
||||||
|
produced by meta-core's schema indexer that consumes `meta:events`.
|
||||||
|
- `/api/snapshot/*` — snapshot export / import / wipe.
|
||||||
|
|
||||||
### Prerequisites
|
In production the editor is served by meta-core's nginx (the Vite build
|
||||||
|
output is copied into the image at the same path) under
|
||||||
|
`https://metacore-dev.localhost:8083/editor/`. The Vite `base` is set to
|
||||||
|
`/editor/` accordingly.
|
||||||
|
|
||||||
- Node.js 21.6.2+
|
## Project layout
|
||||||
- pnpm 10.19.0+
|
|
||||||
- Meta-mesh API server running (provides the `/api/metadata/*` endpoints)
|
|
||||||
|
|
||||||
### Running Locally
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# From the root directory
|
|
||||||
pnpm run start:editor
|
|
||||||
|
|
||||||
# Or from the package directory
|
|
||||||
cd packages/meta-editor
|
|
||||||
pnpm run dev
|
|
||||||
```
|
|
||||||
|
|
||||||
The editor will be available at `http://localhost:5173` with API proxying to `http://localhost:3000`.
|
|
||||||
|
|
||||||
### Building
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# From the root directory
|
|
||||||
pnpm run build:editor
|
|
||||||
|
|
||||||
# Or from the package directory
|
|
||||||
cd packages/meta-editor
|
|
||||||
pnpm run build
|
|
||||||
```
|
|
||||||
|
|
||||||
The built files will be in `dist/` directory.
|
|
||||||
|
|
||||||
## Docker Deployment
|
|
||||||
|
|
||||||
In the Docker development environment, the editor is automatically available at:
|
|
||||||
|
|
||||||
```
|
```
|
||||||
http://localhost/editor
|
packages/meta-core/editor/
|
||||||
```
|
|
||||||
|
|
||||||
The nginx configuration serves the built files from `/app/packages/meta-editor/dist/`.
|
|
||||||
|
|
||||||
## API Integration
|
|
||||||
|
|
||||||
The editor communicates with the UnifiedAPIServer (meta-mesh) via REST API:
|
|
||||||
|
|
||||||
- `POST /api/metadata/search` - Search for files
|
|
||||||
- `GET /api/metadata/:hashId` - Get file metadata
|
|
||||||
- `PUT /api/metadata/:hashId` - Update complete metadata
|
|
||||||
- `PUT /api/metadata/:hashId/property` - Update specific property
|
|
||||||
- `POST /api/metadata/batch` - Batch update multiple files
|
|
||||||
|
|
||||||
See `src/api/metadataApi.ts` for the API client implementation.
|
|
||||||
|
|
||||||
## Usage
|
|
||||||
|
|
||||||
### Searching for Files
|
|
||||||
|
|
||||||
1. Select search type: "Search by Title/Filename" or "Search by Hash ID"
|
|
||||||
2. Enter your search query
|
|
||||||
3. Click "Search"
|
|
||||||
4. Results will appear in the left sidebar
|
|
||||||
|
|
||||||
### Editing Metadata
|
|
||||||
|
|
||||||
1. Select a file from the search results
|
|
||||||
2. Modify any fields in the editor form
|
|
||||||
3. Click "Save Changes" to persist updates
|
|
||||||
4. Use "Reset" to discard changes
|
|
||||||
|
|
||||||
### Supported Metadata Fields
|
|
||||||
|
|
||||||
**Basic Information:**
|
|
||||||
- Title, Original Title, Show Title
|
|
||||||
- Video Type (Movie, TV, Anime, Other)
|
|
||||||
- Year, Season, Episode
|
|
||||||
|
|
||||||
**Jellyfin Metadata:**
|
|
||||||
- Plot, IMDb ID, TMDb ID, AniDB ID
|
|
||||||
- Director, Studio, Rating, Runtime
|
|
||||||
- Genres, Premiered Date, Release Date
|
|
||||||
|
|
||||||
**File Information:**
|
|
||||||
- Filename, File Size, MIME Type
|
|
||||||
- Hash values (SHA-256, MD5, etc.) - read-only
|
|
||||||
|
|
||||||
## Project Structure
|
|
||||||
|
|
||||||
```
|
|
||||||
packages/meta-editor/
|
|
||||||
├── src/
|
|
||||||
│ ├── api/ # API client for metadata operations
|
|
||||||
│ ├── components/ # React components
|
|
||||||
│ │ ├── SearchBar.tsx # Search interface
|
|
||||||
│ │ ├── FileList.tsx # Search results list
|
|
||||||
│ │ └── MetadataEditor.tsx # Main editing form
|
|
||||||
│ ├── types/ # TypeScript type definitions
|
|
||||||
│ ├── utils/ # Utility functions
|
|
||||||
│ ├── App.tsx # Main application component
|
|
||||||
│ ├── main.tsx # Entry point
|
|
||||||
│ └── index.css # Global styles
|
|
||||||
├── index.html
|
├── index.html
|
||||||
├── vite.config.ts
|
├── vite.config.ts
|
||||||
├── tsconfig.json
|
├── tsconfig.json
|
||||||
└── package.json
|
├── package.json @meta-core/editor
|
||||||
|
└── src/
|
||||||
|
├── main.tsx entry
|
||||||
|
├── api/
|
||||||
|
│ ├── kvApi.ts /api/kv client
|
||||||
|
│ ├── schemaApi.ts /api/schema client + cache
|
||||||
|
│ └── snapshotApi.ts /api/snapshot client
|
||||||
|
├── kv/ KV browser + leaf/branch editors
|
||||||
|
├── components/ shared widgets (language combobox, snapshot panel, ...)
|
||||||
|
└── index.css
|
||||||
```
|
```
|
||||||
|
|
||||||
## Technology Stack
|
## Local development
|
||||||
|
|
||||||
- **React 18** - UI framework
|
Prerequisites: Node 21.6.2+, pnpm pinned in the root `package.json`.
|
||||||
- **TypeScript 5.7** - Type safety
|
|
||||||
- **Vite 6** - Build tool and dev server
|
|
||||||
- **Native CSS** - Styling (no CSS framework)
|
|
||||||
|
|
||||||
## Contributing
|
```bash
|
||||||
|
cd packages/meta-core/editor
|
||||||
|
pnpm install
|
||||||
|
pnpm dev # vite dev server on http://localhost:5173
|
||||||
|
```
|
||||||
|
|
||||||
When adding new features:
|
The Vite dev server proxies `/api/*` to `http://localhost:3000` (see
|
||||||
|
`vite.config.ts`). For local development against a running meta-core
|
||||||
|
container, change that target to wherever meta-core is reachable on your
|
||||||
|
host — e.g. `http://localhost:18083` for the dev compose's direct-backend
|
||||||
|
port.
|
||||||
|
|
||||||
1. Add type definitions to `src/types/index.ts`
|
No root-level `start:editor` or `build:editor` scripts exist; run pnpm
|
||||||
2. Update API client in `src/api/metadataApi.ts`
|
commands from this directory.
|
||||||
3. Create/update components as needed
|
|
||||||
4. Update this README with new features
|
|
||||||
|
|
||||||
## License
|
## Build
|
||||||
|
|
||||||
Part of the MetaMesh project.
|
```bash
|
||||||
|
pnpm build # tsc -b && vite build → dist/
|
||||||
|
```
|
||||||
|
|
||||||
|
The container build copies `dist/` into the nginx web root so the editor
|
||||||
|
is served at `/editor/`.
|
||||||
|
|
||||||
|
## Tech
|
||||||
|
|
||||||
|
- React 18, TypeScript 5
|
||||||
|
- Vite 5 (`base: '/editor/'`)
|
||||||
|
- Native CSS — no UI framework
|
||||||
|
- `iso-639-3` for language code rendering
|
||||||
|
|||||||
@@ -79,12 +79,24 @@ function labelOf(path: string, parentPath: string): string {
|
|||||||
return rest.replace(/[/:]+$/, '');
|
return rest.replace(/[/:]+$/, '');
|
||||||
}
|
}
|
||||||
|
|
||||||
// Hide internal Redis structures from the tree:
|
// Hide internal Redis structures from the tree. The editor is for managing
|
||||||
// - the `meta:` namespace (event streams, internal state)
|
// file metadata; everything else is meta-core's own bookkeeping and noise
|
||||||
// - bare top-level leaves under a colon-namespace like file:__index__ and
|
// for the user. Hidden:
|
||||||
// file:events (Redis sets / streams that aren't user data).
|
// - `meta:` — event streams (file:events, meta:events, etc.)
|
||||||
|
// - `cid:` — reverse index (cid:<algo>:<value> → uuid), derived
|
||||||
|
// from each file's cid_* fields; the editor surfaces
|
||||||
|
// these per-file under their parent root
|
||||||
|
// - `meta-core:` — internal service state (mounts, watcher config,
|
||||||
|
// leader info), edited via dedicated UI pages, not
|
||||||
|
// the KV tree
|
||||||
|
// - bare top-level leaves under a colon-namespace like file:__index__
|
||||||
|
// and file:events (Redis sets / streams that aren't user data).
|
||||||
function isHiddenNode(node: TreeNode): boolean {
|
function isHiddenNode(node: TreeNode): boolean {
|
||||||
if (node.kind === 'branch' && node.path === 'meta:') return true;
|
if (node.kind === 'branch') {
|
||||||
|
if (node.path === 'meta:' || node.path === 'cid:' || node.path === 'meta-core:') {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
}
|
||||||
if (node.kind === 'leaf' && /^[^/]+:[^/]+$/.test(node.path)) return true;
|
if (node.kind === 'leaf' && /^[^/]+:[^/]+$/.test(node.path)) return true;
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -132,3 +132,20 @@ func getEnvBool(key string, defaultValue bool) bool {
|
|||||||
func (c *Config) DefaultWatchPath() string {
|
func (c *Config) DefaultWatchPath() string {
|
||||||
return c.FilesPath + "/watch"
|
return c.FilesPath + "/watch"
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// DefaultWatcherPaths returns every directory that should be watched out of
|
||||||
|
// the box on a fresh install. The watcher seeds these on first start.
|
||||||
|
//
|
||||||
|
// - /files/watch — user-curated media. The main library.
|
||||||
|
// - /files/plugin — plugin output (TMDB posters/backdrops, extracted
|
||||||
|
// subtitles, etc.). Indexed so each artefact gets a
|
||||||
|
// midhash256 alias and CID-based references from
|
||||||
|
// other files (e.g. `backdrop=<cid>`) resolve through
|
||||||
|
// the standard reverse index instead of needing a
|
||||||
|
// path companion field.
|
||||||
|
func (c *Config) DefaultWatcherPaths() []string {
|
||||||
|
return []string{
|
||||||
|
c.FilesPath + "/watch",
|
||||||
|
c.FilesPath + "/plugin",
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -292,13 +292,35 @@ func writeJSON(w http.ResponseWriter, status int, data interface{}) {
|
|||||||
json.NewEncoder(w).Encode(data)
|
json.NewEncoder(w).Encode(data)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// writeError writes the typed error envelope (see api/handlers.go for the
|
||||||
|
// canonical definition + slug vocabulary). Kept local to avoid an import
|
||||||
|
// cycle with the api package.
|
||||||
func writeError(w http.ResponseWriter, status int, message string) {
|
func writeError(w http.ResponseWriter, status int, message string) {
|
||||||
writeJSON(w, status, map[string]string{
|
slug, retryable := slugForStatus(status)
|
||||||
"error": http.StatusText(status),
|
writeJSON(w, status, map[string]interface{}{
|
||||||
|
"error": slug,
|
||||||
"message": message,
|
"message": message,
|
||||||
|
"retryable": retryable,
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func slugForStatus(status int) (string, bool) {
|
||||||
|
switch status {
|
||||||
|
case http.StatusConflict:
|
||||||
|
return "alias_collision", false
|
||||||
|
case http.StatusNotFound:
|
||||||
|
return "unknown_root", false
|
||||||
|
case http.StatusBadRequest:
|
||||||
|
return "schema_violation", false
|
||||||
|
case http.StatusServiceUnavailable:
|
||||||
|
return "storage_unavailable", true
|
||||||
|
case http.StatusInternalServerError:
|
||||||
|
return "internal", true
|
||||||
|
default:
|
||||||
|
return "internal", status >= 500
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func parseIntOrDefault(s string, defaultVal int) (int, error) {
|
func parseIntOrDefault(s string, defaultVal int) (int, error) {
|
||||||
var result int
|
var result int
|
||||||
if s == "" {
|
if s == "" {
|
||||||
|
|||||||
@@ -462,39 +462,15 @@ func (c *Client) GetMetadataDocument(uuid string) (*MetadataDocument, error) {
|
|||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// lookupSidecarPathByCID is the fallback that LookupPathByCID falls into
|
// lookupSidecarPathByCID — REMOVED.
|
||||||
// when the token isn't a known root alias. Used to resolve poster/backdrop
|
|
||||||
// CIDs, which are stored as values pointing at sidecar files (posterPath /
|
|
||||||
// backdropPath) rather than registered as aliases of the file itself.
|
|
||||||
//
|
//
|
||||||
// O(n) — walks every root. Acceptable because (a) sidecar lookups are rare
|
// This function used to walk every metadata entry looking for poster/backdrop
|
||||||
// vs. file CID lookups and (b) the alternative is registering sidecars in
|
// matches and return their `posterPath`/`backdropPath` companion fields. It
|
||||||
// the reverse index, which conflates "this CID is the file" with "this CID
|
// was a workaround for plugin output (TMDB images, etc.) not being indexed
|
||||||
// is an image associated with the file."
|
// by the watcher.
|
||||||
func (c *Client) lookupSidecarPathByCID(token string) (string, error) {
|
//
|
||||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
// Now that the plugin output directory is a first-class watcher root
|
||||||
defer cancel()
|
// (see config.DefaultWatcherPaths), every plugin artefact has its own
|
||||||
|
// midhash256 alias and resolves through the standard reverse index. Storing
|
||||||
hashIDs, err := c.getAllHashIDsInternal(ctx)
|
// paths in metadata broke the CID-as-identity contract; the watcher
|
||||||
if err != nil {
|
// indexing fixes the root cause.
|
||||||
return "", err
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, hashID := range hashIDs {
|
|
||||||
meta, err := c.getMetadataFlatInternal(ctx, hashID)
|
|
||||||
if err != nil || len(meta) == 0 {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
if meta["poster"] == token {
|
|
||||||
if p := meta["posterPath"]; p != "" {
|
|
||||||
return normalizeFilesRelativePath(p), nil
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if meta["backdrop"] == token {
|
|
||||||
if p := meta["backdropPath"]; p != "" {
|
|
||||||
return normalizeFilesRelativePath(p), nil
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return "", nil
|
|
||||||
}
|
|
||||||
|
|||||||
+22
-15
@@ -347,34 +347,41 @@ func (c *Client) CountFiles() (int, error) {
|
|||||||
return len(hashIDs), nil
|
return len(hashIDs), nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// LookupPathByCID resolves a CID to its file path on disk.
|
// LookupPathByCID resolves a CID to its file path on disk via the reverse
|
||||||
|
// index. O(1) lookup followed by a single GET of file:<uuid>/filePath.
|
||||||
//
|
//
|
||||||
// Primary path: the reverse index (cid:<token> → uuid). O(1) lookup followed
|
// Accepts both prefixed tokens ("midhash256:bafk…", "sha256:…",
|
||||||
// by a single GET of file:<uuid>/filePath.
|
// "ipfs:bafy…") and bare midhash256 CIDs — the bare form is what the
|
||||||
|
// editor passes to /api/file/{cid}/info because that's how plugins
|
||||||
|
// write the value (e.g. `backdrop=<bare-cid>`). For bare tokens we
|
||||||
|
// try the midhash256 prefix as a fallback.
|
||||||
//
|
//
|
||||||
// Fallback path: poster/backdrop CIDs aren't registered as aliases (they
|
// Plugin output (TMDB posters/backdrops, extracted subtitles, etc.) is
|
||||||
// refer to sidecar images, not the main file content), so unknown tokens
|
// resolved through this same path: the plugin output directory is a
|
||||||
// fall through to a scan of poster/backdrop fields. O(n) but rare in
|
// watcher root (see config.DefaultWatcherPaths), so every plugin
|
||||||
// practice — sidecar lookups are infrequent compared to file CID lookups.
|
// artefact gets a midhash256 alias and resolves through the reverse
|
||||||
//
|
// index like any other file. There is no path-companion sidecar field.
|
||||||
// Accepts CID tokens in prefixed form ("midhash256:bafk…", "sha256:…",
|
|
||||||
// "ipfs:bafy…"). Bare CIDs without an algorithm prefix won't resolve via
|
|
||||||
// the reverse index — callers should pass the prefixed form.
|
|
||||||
func (c *Client) LookupPathByCID(token string) (string, error) {
|
func (c *Client) LookupPathByCID(token string) (string, error) {
|
||||||
uuid, err := c.GetByCID(token)
|
uuid, err := c.GetByCID(token)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return "", err
|
return "", err
|
||||||
}
|
}
|
||||||
if uuid != "" {
|
if uuid == "" && !strings.Contains(token, ":") {
|
||||||
|
// Bare CID — most likely a midhash256 written by a plugin.
|
||||||
|
uuid, err = c.GetByCID("midhash256:" + token)
|
||||||
|
if err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if uuid == "" {
|
||||||
|
return "", nil
|
||||||
|
}
|
||||||
fp, err := c.GetProperty(uuid, "filePath")
|
fp, err := c.GetProperty(uuid, "filePath")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return "", err
|
return "", err
|
||||||
}
|
}
|
||||||
return normalizeFilesRelativePath(fp), nil
|
return normalizeFilesRelativePath(fp), nil
|
||||||
}
|
}
|
||||||
// Not a file alias — try poster/backdrop sidecars.
|
|
||||||
return c.lookupSidecarPathByCID(token)
|
|
||||||
}
|
|
||||||
|
|
||||||
// normalizeFilesRelativePath strips a leading "/files/" prefix so handlers can
|
// normalizeFilesRelativePath strips a leading "/files/" prefix so handlers can
|
||||||
// safely filepath.Join(FilesPath, …) without double-prefixing. Different
|
// safely filepath.Join(FilesPath, …) without double-prefixing. Different
|
||||||
|
|||||||
@@ -100,9 +100,31 @@ func writeJSON(w http.ResponseWriter, status int, data interface{}) {
|
|||||||
json.NewEncoder(w).Encode(data)
|
json.NewEncoder(w).Encode(data)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// writeError writes the typed error envelope (see api/handlers.go for the
|
||||||
|
// canonical definition + slug vocabulary). Kept local to avoid an import
|
||||||
|
// cycle with the api package.
|
||||||
func writeError(w http.ResponseWriter, status int, message string) {
|
func writeError(w http.ResponseWriter, status int, message string) {
|
||||||
writeJSON(w, status, map[string]string{
|
slug, retryable := slugForStatus(status)
|
||||||
"error": http.StatusText(status),
|
writeJSON(w, status, map[string]interface{}{
|
||||||
|
"error": slug,
|
||||||
"message": message,
|
"message": message,
|
||||||
|
"retryable": retryable,
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func slugForStatus(status int) (string, bool) {
|
||||||
|
switch status {
|
||||||
|
case http.StatusConflict:
|
||||||
|
return "alias_collision", false
|
||||||
|
case http.StatusNotFound:
|
||||||
|
return "unknown_root", false
|
||||||
|
case http.StatusBadRequest:
|
||||||
|
return "schema_violation", false
|
||||||
|
case http.StatusServiceUnavailable:
|
||||||
|
return "storage_unavailable", true
|
||||||
|
case http.StatusInternalServerError:
|
||||||
|
return "internal", true
|
||||||
|
default:
|
||||||
|
return "internal", status >= 500
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -242,9 +242,31 @@ func writeJSON(w http.ResponseWriter, status int, data interface{}) {
|
|||||||
json.NewEncoder(w).Encode(data)
|
json.NewEncoder(w).Encode(data)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// writeError writes the typed error envelope (see api/handlers.go for the
|
||||||
|
// canonical definition + slug vocabulary). Kept local to avoid an import
|
||||||
|
// cycle with the api package.
|
||||||
func writeError(w http.ResponseWriter, status int, message string) {
|
func writeError(w http.ResponseWriter, status int, message string) {
|
||||||
writeJSON(w, status, map[string]string{
|
slug, retryable := slugForStatus(status)
|
||||||
"error": http.StatusText(status),
|
writeJSON(w, status, map[string]interface{}{
|
||||||
|
"error": slug,
|
||||||
"message": message,
|
"message": message,
|
||||||
|
"retryable": retryable,
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func slugForStatus(status int) (string, bool) {
|
||||||
|
switch status {
|
||||||
|
case http.StatusConflict:
|
||||||
|
return "alias_collision", false
|
||||||
|
case http.StatusNotFound:
|
||||||
|
return "unknown_root", false
|
||||||
|
case http.StatusBadRequest:
|
||||||
|
return "schema_violation", false
|
||||||
|
case http.StatusServiceUnavailable:
|
||||||
|
return "storage_unavailable", true
|
||||||
|
case http.StatusInternalServerError:
|
||||||
|
return "internal", true
|
||||||
|
default:
|
||||||
|
return "internal", status >= 500
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -55,47 +55,86 @@ func (m *Manager) SetOnChanged(cb func()) {
|
|||||||
m.onChanged = cb
|
m.onChanged = cb
|
||||||
}
|
}
|
||||||
|
|
||||||
// loadOrCreateDefault loads existing config or creates default watcher
|
// loadOrCreateDefault loads existing config or creates default watchers.
|
||||||
|
// Also reconciles after load: any path in DefaultWatcherPaths that isn't
|
||||||
|
// yet present gets added. This catches deployments whose watchers.json
|
||||||
|
// pre-dates the addition of a new default (e.g. /files/plugin), without
|
||||||
|
// requiring operators to manually edit JSON.
|
||||||
func (m *Manager) loadOrCreateDefault() error {
|
func (m *Manager) loadOrCreateDefault() error {
|
||||||
// Try to load existing file
|
|
||||||
if _, err := os.Stat(m.watchersFile); err == nil {
|
if _, err := os.Stat(m.watchersFile); err == nil {
|
||||||
return m.load()
|
if err := m.load(); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
return m.ensureDefaultWatchers()
|
||||||
}
|
}
|
||||||
|
|
||||||
// File doesn't exist, create default watcher for /files/watch
|
log.Printf("[Watchers] No existing config, creating defaults")
|
||||||
log.Printf("[Watchers] No existing config, creating default watcher for %s", m.config.DefaultWatchPath())
|
return m.createDefaultWatchers()
|
||||||
return m.createDefaultWatcher()
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// createDefaultWatcher creates the default /files/watch watcher and folder
|
// ensureDefaultWatchers makes sure every path in DefaultWatcherPaths has a
|
||||||
func (m *Manager) createDefaultWatcher() error {
|
// corresponding watcher entry. Existing entries (matched by path) are left
|
||||||
|
// untouched — only missing ones are added.
|
||||||
|
func (m *Manager) ensureDefaultWatchers() error {
|
||||||
m.mu.Lock()
|
m.mu.Lock()
|
||||||
defer m.mu.Unlock()
|
defer m.mu.Unlock()
|
||||||
|
|
||||||
defaultPath := m.config.DefaultWatchPath()
|
existing := make(map[string]bool, len(m.watchers))
|
||||||
|
for _, w := range m.watchers {
|
||||||
|
existing[w.Path] = true
|
||||||
|
}
|
||||||
|
|
||||||
// Create the watch folder if it doesn't exist
|
added := false
|
||||||
if err := os.MkdirAll(defaultPath, 0755); err != nil {
|
for _, p := range m.config.DefaultWatcherPaths() {
|
||||||
log.Printf("[Watchers] Warning: failed to create default watch folder %s: %v", defaultPath, err)
|
if existing[p] {
|
||||||
// Continue anyway - folder might be created later or mounted
|
continue
|
||||||
|
}
|
||||||
|
if err := m.addDefaultWatcherUnsafe(p); err != nil {
|
||||||
|
log.Printf("[Watchers] Warning: failed to add default watcher %s: %v", p, err)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
added = true
|
||||||
|
}
|
||||||
|
if !added {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return m.saveUnsafe()
|
||||||
|
}
|
||||||
|
|
||||||
|
// createDefaultWatchers seeds the watcher config with every entry in
|
||||||
|
// DefaultWatcherPaths. Used on first start when no config file exists.
|
||||||
|
func (m *Manager) createDefaultWatchers() error {
|
||||||
|
m.mu.Lock()
|
||||||
|
defer m.mu.Unlock()
|
||||||
|
|
||||||
|
for _, p := range m.config.DefaultWatcherPaths() {
|
||||||
|
if err := m.addDefaultWatcherUnsafe(p); err != nil {
|
||||||
|
log.Printf("[Watchers] Warning: failed to seed default watcher %s: %v", p, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return m.saveUnsafe()
|
||||||
|
}
|
||||||
|
|
||||||
|
// addDefaultWatcherUnsafe creates the on-disk folder (if missing) and adds
|
||||||
|
// a watcher entry. Caller must hold m.mu.
|
||||||
|
func (m *Manager) addDefaultWatcherUnsafe(path string) error {
|
||||||
|
if err := os.MkdirAll(path, 0755); err != nil {
|
||||||
|
log.Printf("[Watchers] Warning: failed to create folder %s: %v", path, err)
|
||||||
|
// Continue — folder might be mounted later
|
||||||
} else {
|
} else {
|
||||||
log.Printf("[Watchers] Created default watch folder: %s", defaultPath)
|
log.Printf("[Watchers] Created folder: %s", path)
|
||||||
}
|
}
|
||||||
|
|
||||||
// Create default watcher
|
|
||||||
id := uuid.New().String()[:8]
|
id := uuid.New().String()[:8]
|
||||||
watcher := &WatcherConfig{
|
m.watchers[id] = &WatcherConfig{
|
||||||
ID: id,
|
ID: id,
|
||||||
Path: defaultPath,
|
Path: path,
|
||||||
IntervalMs: DefaultIntervalMs,
|
IntervalMs: DefaultIntervalMs,
|
||||||
Enabled: true,
|
Enabled: true,
|
||||||
CreatedAt: NowMS(),
|
CreatedAt: NowMS(),
|
||||||
}
|
}
|
||||||
m.watchers[id] = watcher
|
log.Printf("[Watchers] Created default watcher: %s -> %s", id, path)
|
||||||
log.Printf("[Watchers] Created default watcher: %s -> %s", id, defaultPath)
|
return nil
|
||||||
|
|
||||||
// Save config
|
|
||||||
return m.saveUnsafe()
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// load loads watchers from file
|
// load loads watchers from file
|
||||||
|
|||||||
@@ -68,14 +68,20 @@ func (p *Poller) Start() error {
|
|||||||
|
|
||||||
log.Println("[WatcherPoller] Starting file watcher polling service")
|
log.Println("[WatcherPoller] Starting file watcher polling service")
|
||||||
|
|
||||||
// Clear file:events stream on startup to prevent duplicate events from accumulating
|
// No startup reset. The file:events stream is bounded by MAXLEN
|
||||||
// across container restarts. Since in-memory state is empty, all files will be
|
// (api-mediated-access PR A) and SSE consumers resume from their
|
||||||
// re-emitted as "add" events - we need a fresh stream to match.
|
// persisted Last-Event-ID, so a startup wipe would only orphan
|
||||||
if p.dispatcher != nil {
|
// consumer state without buying anything. The historical reason for
|
||||||
if err := p.dispatcher.EmitReset("startup"); err != nil {
|
// wiping — "in-memory state is empty, all files re-emit as add" —
|
||||||
log.Printf("[WatcherPoller] Warning: failed to clear stream on startup: %v", err)
|
// stopped being true when HydrateStateFromStorage landed: state is
|
||||||
}
|
// hydrated from Redis before this runs, so a startup EmitReset would
|
||||||
}
|
// tell consumers to throw out their world without sending the
|
||||||
|
// re-emission they need to rebuild it.
|
||||||
|
//
|
||||||
|
// Operator-triggered resets (POST /api/watchers/{id}/reset) still
|
||||||
|
// work — that path goes through TriggerReset, which clears both the
|
||||||
|
// stream AND the watcher's in-memory state, so the next scan tick
|
||||||
|
// emits add events for every file.
|
||||||
|
|
||||||
// Initial sync with watcher configurations
|
// Initial sync with watcher configurations
|
||||||
go p.syncWithWatchers()
|
go p.syncWithWatchers()
|
||||||
|
|||||||
Reference in New Issue
Block a user