From 9254f043a44926b90b0c79eefb9f237da2fedb47 Mon Sep 17 00:00:00 2001 From: Worph Date: Sat, 3 Oct 2026 11:51:55 +0000 Subject: [PATCH] README: sync with current code --- README.md | 226 ++++++++++++++++++++++++++++++++++-------------------- 1 file changed, 141 insertions(+), 85 deletions(-) diff --git a/README.md b/README.md index 2090966..b57f89c 100644 --- a/README.md +++ b/README.md @@ -1,34 +1,42 @@ # meta-core -Standalone Go service that owns Redis, the leader-info file, the HTTP+SSE -metadata surface, and a WebDAV file server for the MetaMesh dev/prod stack. +Standalone Go service that owns Redis, the HTTP+SSE metadata surface, the +User Data Layer store and signing keystore, and a WebDAV file server for the +MetaMesh dev/prod stack. ## Overview meta-core runs as its own container (`metacore-app`). Other MetaMesh services -(meta-sort, meta-fuse, meta-stremio, meta-dup, meta-share) discover it by -reading `/meta-core/locks/kv-leader.info` and then call its HTTP API. Redis -is not exposed outside the container — all metadata I/O is mediated by the -HTTP API. +(meta-sort, meta-fuse, meta-stremio, meta-dup, meta-search, meta-share, +meta-gateway, meta-watch, …) locate it over UDP multicast (meta-discovery v1 — +its `role=core` announce carries the full `/urls` payload) and then call its +HTTP API. `META_CORE_URL` on a client pins it instead. Redis is not published +outside the container — all metadata I/O is mediated by the HTTP/SSE API +(see [`docs/api-mediated-access.md`](docs/api-mediated-access.md)). ### Key capabilities | Feature | Description | |---------|-------------| -| **Leader gate** | Bash `flock` loop in `docker/leader-election.sh` only `exec`s supervisord on the winner. The Go binary therefore only ever runs as leader. | +| **Redis-volume mutex** | Bash `flock` loop in `docker/leader-election.sh` only `exec`s supervisord on the winner, so two containers sharing one `/meta-core` volume can never run two Redis writers. It is a mutex, not discovery — nothing reads the lock. | | **Redis owner** | Supervisord starts Redis with AOF + RDB persistence inside the container; meta-core connects to it on localhost. | -| **HTTP API** | Typed REST surface (gorilla/mux) for metadata, KV browsing, snapshots, schema inference, mounts, watchers, neighbours, and file access by CID. | +| **HTTP API** | Typed REST surface (gorilla/mux) for metadata, KV browsing, snapshots, schema inference, mounts, watchers, neighbours, identity, UDL, and file access by CID. | +| **In-memory search index** | `POST /api/metadata/search` matches against an in-memory snapshot of every record, kept current by incremental refreshes (full reconcile every 30 min) — no Redis round-trip per record on the request path (`internal/api/search_index.go`). | | **SSE event streams** | `/api/events/files` and `/api/events/meta` mirror the underlying Redis Streams so external services never speak Redis. | | **WebDAV server** | `/webdav/*` exposes the `/files` volume (read+write) for cross-service file access. nginx in front of meta-core handles caching. | | **Service discovery** | meta-discovery v1: UDP multicast announce/listen on `239.255.77.1:9399`, neighbours served at `/neighbors`. The file-based registry under `/meta-core/services/*.json` it replaced is gone. | -| **File watcher / scanner** | Recursive scan + MidHash256 computation; emits events on `file:events`. | +| **File watcher / scanner** | Recursive scan + MidHash256 computation; emits events on `file:events`. Watched roots are POSTed watcher configs (`/meta-core/watchers.json`), not env vars. | | **Mount management** | rclone-only mount manager (SMB rendered into `:smb:`, plus pre-defined remotes). Read-only by construction. | | **UUID-rooted storage** | Roots are UUIDv7 (Crockford Base32, ULID layout); CIDs are reverse-index aliases. Schema-version sentinel refuses to boot against stale data. | +| **Identity keystore** | Multi-account secp256k1 signing keys under `/meta-core/identity/accounts/`, created from the dashboard's Identity menu (never auto-generated). A private key is returned only by generate, or by reveal against a proof-of-possession signature (delete needs the same proof); `/api/identity/sign` refuses challenge-domain payloads. | +| **User Data Layer** | Version-gated store for per-user signed (and, for private-tier keys, encrypted) records — My List, Continue Watching, ratings — indexed by user and CID. meta-core never inspects the crypto. See [User Data Layer](../../docs/project-architecture/user-data-layer.md). | +| **Dashboard + editor** | nginx serves a React dashboard at `/` and the standalone metadata editor at `/editor/`. | ### Design characteristics -- **Single-leader** — flock on a shared volume; no external consensus. -- **HTTP-only externally** — Redis is bound to the container, never published. +- **Single writer** — flock on the `/meta-core` volume; no external consensus. +- **HTTP-only externally** — Redis is never published by the compose stacks; + debug with `docker exec metacore-app redis-cli …`. - **Schema-gated boot** — `EnsureSchemaVersion` aborts startup when the on-disk Redis layout predates the current build (alpha clean-wipe policy; see `docs/uuid-rooted-metadata.md`). @@ -41,34 +49,31 @@ HTTP API. ┌──────────────────────────────────────────────────────────────────────────┐ │ metacore-app (standalone container) │ │ │ +│ leader-election.sh (flock /meta-core/locks/kv-leader.lock, │ +│ then exec supervisord) │ │ supervisord │ -│ ├─ leader-election.sh (flock /meta-core/locks/kv-leader.lock, │ -│ │ writes kv-leader.info, exec supervisord) │ │ ├─ redis-server (AOF + RDB on /meta-core/db/redis) │ -│ ├─ nginx (TLS termination, proxy_cache for /webdav) │ -│ ├─ rclone rcd (mount daemon, RC API) │ +│ ├─ nginx :80 (dashboard, /editor, API proxy, │ +│ │ proxy_cache for /webdav, rclone GUI /rclone) │ +│ ├─ rclone rcd (mount daemon, RC API on :5572) │ │ ├─ mount-watcher.sh (rclone mount lifecycle) │ │ └─ meta-core (Go) ─► HTTP API + SSE + WebDAV on :9000 │ +│ ─► UDP announce/listen 239.255.77.1:9399 │ └──────────────────────────────────────────────────────────────────────────┘ │ ┌──────────────────────┴──────────────────────┐ - │ /meta-core (shared volume) │ - │ locks/kv-leader.{lock,info} │ + │ /meta-core (meta-core's own volume) │ + │ locks/kv-leader.lock │ │ db/redis/ │ - │ services/*.json │ + │ identity/accounts/*.json │ │ mounts/{mounts.json, errors/} │ │ watchers.json │ │ cache/ (nginx proxy_cache) │ └─────────────────────────────────────────────┘ - ▲ - ┌──────────────────────────────┼────────────────────────────────┐ - │ │ │ -┌─────────────┐ ┌─────────────┐ ┌─────────────┐ -│ meta-sort │ │ meta-fuse │ │ meta-stremio│ -│ │ │ │ │ │ -│ reads │ HTTP+SSE │ reads │ HTTP+SSE │ reads │ -│ leader.info├──────────────►│ leader.info├───────────────►│ leader.info│ -└─────────────┘ └─────────────┘ └─────────────┘ + + meta-sort / meta-fuse / meta-stremio / meta-search / meta-share / … + │ 1. hear role=core announce on UDP multicast (carries /urls) + └─ 2. HTTP + SSE (+ WebDAV) to metacore-app — never Redis, never the volume ``` ## Quick start @@ -82,8 +87,9 @@ make test # run Go unit tests ``` `make docker` tags the image as `:1.0.0`, `:latest`, **and** `:local`. The -`:local` tag is the one every sub-stack docker-compose.yml references -(meta-share 10-peer federation, meta-gateway dev stacks); without it the +`:local` tag is the one the sub-stack docker-compose files reference +(meta-share federation, meta-read and meta-listen dev stacks); the meta-gateway +dev stack (`docker-compose.gateway.yml`) uses `:latest`. Without these tags the sub-stacks can't see a freshly-built image by name. > **Source-change → recreate.** `make docker` rebuilds and re-tags the @@ -94,12 +100,14 @@ sub-stacks can't see a freshly-built image by name. > > ```bash > make docker -> docker compose -f packages/meta-gateway/dev/docker-compose.yml \ -> up -d --force-recreate metagateway-generic-core -> docker compose -f packages/meta-gateway/dev/docker-compose.torznab.yml \ -> up -d --force-recreate metagateway-torznab-core +> docker compose -f packages/meta-gateway/dev/docker-compose.gateway.yml \ +> up -d --force-recreate metagateway-hub-metacore > docker compose -f packages/meta-share/dev/docker-compose.yml \ > --profile share-10node up -d --force-recreate metashare-core metashare-core-test +> docker compose -f packages/meta-read/dev/docker-compose.yml \ +> up -d --force-recreate metaread-dev-core +> docker compose -f packages/meta-listen/dev/docker-compose.yml \ +> up -d --force-recreate metalisten-dev-core > ``` > > The symptom of a missed recreate is missing API fields on `/urls` @@ -109,7 +117,7 @@ sub-stacks can't see a freshly-built image by name. > > ```bash > docker inspect meta-core:local --format '{{.Id}}' # fresh SHA -> docker ps --filter ancestor=meta-core:local --format '{{.Names}}' +> docker ps --filter ancestor=meta-core:local --format '{{.Names}}' # repeat for :latest > # any container whose Image SHA doesn't match the fresh SHA needs --force-recreate. > ``` @@ -126,36 +134,39 @@ HTTP port (no auth, bypasses Caddy) is `http://localhost:18083`. ## Configuration -Environment variables consumed by `internal/config/config.go`: +Environment variables (from `internal/config/config.go` unless noted): | Variable | Default | Description | |---|---|---| -| `META_CORE_PATH` | `/meta-core` | Shared volume root (locks, db, services, mounts, cache). | +| `META_CORE_PATH` | `/meta-core` | meta-core's own volume root (lock, Redis data, identity, mounts, watchers, cache). No other service mounts it. | | `FILES_PATH` | `/files` | Files volume root. | -| `SERVICE_NAME` | `meta-core` | Identity in the service registry. | -| `SERVICE_VERSION` | `1.0.0` | Reported via `/status` and registry. | -| `API_PORT` | `8180` | External port baked into the leader-info `baseUrl`. | -| `BASE_URL` | _empty_ | Overrides the constructed `baseUrl` if set (used for HTTPS perimeter). | +| `SERVICE_NAME` | `meta-core` | Name in the meta-discovery announce. | +| `SERVICE_VERSION` | `1.0.0` | Reported via `/status`. | +| `API_PORT` | `8180` | External port baked into the constructed `baseUrl` when `BASE_URL` is unset. | +| `BASE_URL` | _empty_ | Overrides the constructed `baseUrl` (the Caddy/HTTPS perimeter URL). | +| `META_CORE_PUBLIC_URL` | _empty_ | Browser-facing nav URL in the announce; wins over `BASE_URL` (e.g. a debug-direct port with no Caddy). | | `REDIS_PORT` | `6379` | Local Redis port. | | `META_CORE_HTTP_PORT` | `9000` | Go HTTP+SSE+WebDAV port. | | `META_CORE_HTTP_HOST` | `0.0.0.0` | HTTP bind. | -| `HEALTH_CHECK_INTERVAL_MS` | `5000` | Internal health-loop cadence. | -| `HEARTBEAT_INTERVAL_MS` | `30000` | Service registry heartbeat. | -| `STALE_THRESHOLD_MS` | `60000` | Age past which a registry entry is marked stale. | -| `CLEANUP_INTERVAL_MS` | `600000` | Dead-service cleaner cadence. | -| `DEAD_SERVICE_THRESHOLD_MS` | `600000` | Age past which a stale entry is deleted. | -| `WATCH_INTERVAL_MS` | `1000` | Watcher poll cadence. | -| `DEBOUNCE_MS` | `30000` | File-event debounce window. | -| `ENABLE_FILE_WATCHER` | `true` | Disable to suppress the in-process watcher entirely. | +| `ENABLE_FILE_WATCHER` | `true` | Disable to suppress the in-process watcher entirely (`POST /api/files/register` stays available). | +| `ENABLE_UDP_DISCOVERY` | `true` | meta-discovery v1 announce + listen. | +| `DISCOVERY_GROUP` / `DISCOVERY_PORT` | `239.255.77.1` / `9399` | Multicast group and UDP port. | +| `DISCOVERY_INTERVAL_MS` | `10000` | Unsolicited announce interval. | +| `META_CORE_INDEX_EXCLUDE_PREFIXES` | `categories/newznab/` | Comma-separated field prefixes left out of the in-memory search index; set empty to index everything (`internal/storage/client.go`). | +| `ELECTION_RETRY_SECS` | `5` | flock retry interval (read by `docker/leader-election.sh`, not the Go binary). | -Note: there is no `META_CORE_REDIS_PORT`, no `META_CORE_SERVICE_NAME`, and -no `META_CORE_LEADER_TIMEOUT`. The Redis URL is discovered locally; service -identity uses `SERVICE_NAME`; leader timeout is governed by the bash flock -retry interval (`ELECTION_RETRY_SECS`, default `5`). +`HEALTH_CHECK_INTERVAL_MS`, `HEARTBEAT_INTERVAL_MS`, `STALE_THRESHOLD_MS`, +`CLEANUP_INTERVAL_MS`, `DEAD_SERVICE_THRESHOLD_MS`, `WATCH_INTERVAL_MS` and +`DEBOUNCE_MS` are still parsed by `internal/config/config.go` but nothing reads +them (leftovers of the file-based registry); setting them has no effect. There +is no `META_CORE_REDIS_PORT` / `META_CORE_SERVICE_NAME` / +`META_CORE_LEADER_TIMEOUT`, and no `WATCH_PATHS` — watched folders are watcher +configs created via `/api/watchers`. ## API reference -All endpoints below are served by `internal/api/server.go`. Examples assume +All endpoints below are registered in `internal/api/server.go` (mounts, +watcher and watchers routes in their own packages' `handlers.go`). Examples assume the in-container Go port `:9000`; from the host use `https://metacore-dev.localhost:8083` (Caddy) or `http://localhost:18083` (debug-direct). @@ -229,6 +240,8 @@ GET /api/meta/{cid} # resolve CID → metadata document GET /api/file/{cid}/info # CID resolution metadata GET /file/{cid} # serve file bytes (range-aware) POST /file/cid # compute CID for an uploaded file +POST /api/files/register # mint + alias one file under /files/plugin by midhash + # (the watcher's path for one file; works with the watcher OFF) HEAD /data/{hash} # existence + size GET /data/{hash}/path # absolute path on /files ``` @@ -280,6 +293,44 @@ signalling). ```bash GET /api/events/files # file:events stream (watcher events) GET /api/events/meta # meta:events stream (metadata mutations) +GET /api/events/poll # DEPRECATED long-poll over file:events — use SSE +``` + +### Identity (signing keystore) + +Reached through meta-search's identity proxy (and directly by meta-gateway, +meta-read, meta-listen) for record signing, UDL writes and keypair sign-in. + +```bash +GET /api/identity # status (never the private key) +GET /api/identity/accounts +POST /api/identity/generate # the only call that returns a new private key +POST /api/identity/import +POST /api/identity/challenge # one-shot challenge for proof-of-possession +POST /api/identity/reveal # needs proof-of-possession +DELETE /api/identity # needs proof-of-possession; reports purged UDL counts +POST /api/identity/sign +POST /api/identity/sign-batch +GET /api/identity/aead-key +``` + +### User Data Layer + +Records arrive signed (private-tier ones encrypted) as opaque base64 CBOR; +writes are version-gated (stale → `409 stale_version`, or `accepted:false` per +cell in a batch). Redis model in `internal/storage/udl.go`. + +```bash +GET /api/udl/record?uid&cid&key +PUT /api/udl/record +PUT /api/udl/records # batch +GET /api/udl/user/{uid}/key/{key} +GET /api/udl/user/{uid}/cid/{cid} +GET /api/udl/user/{uid}/all +GET /api/udl/user/{uid}/count +GET /api/udl/cid/{cid}/users +GET /api/udl/cid/{cid}/aggregate # meta-watch Aggregate wire shape +GET /api/udl/users/stats ``` ### Mounts @@ -319,6 +370,9 @@ return a deprecation pointer at the `/api/watchers/*` routes ```bash POST /api/admin/migrate-dual-roots # reunify stranded midhash-rooted entries +POST /api/admin/migrate-domain-screen # domain film|tv → screen, workForm backfill (idempotent) +POST /api/admin/migrate-user-cids # backfill udl:idx:user::cids; until run, pre-index + # profiles export as EMPTY. Returns {fixed} POST /api/admin/sweep-literature-echoes[?apply=true] # strip query echoes off literature cards: anilistid on site cards, # workcid/chapterNumber/volumeNumber/chapterStart/chapterEnd on any card. @@ -329,32 +383,29 @@ GET/PUT/DELETE /webdav/... # mounted on /files No `/metrics` endpoint is exposed; observability is via container logs and the `/health` / `/status` JSON. -## Leader gate +## Redis-volume mutex -Leader election is a bash flock loop in `docker/leader-election.sh`, not a -Go-level state machine. When `flock` succeeds on `kv-leader.lock`, the -script writes the leader API URL to `kv-leader.info` and `exec`s -supervisord. When the process tree dies, the kernel releases the lock and -any standby instance can take over. There is no follower path inside the -Go binary — there is no `OnBecomeLeader` / `OnBecomeFollower` callback API. +`docker/leader-election.sh` is a bash flock loop, not a Go-level state +machine. When `flock` succeeds on `/meta-core/locks/kv-leader.lock`, the +script `exec`s supervisord; a second container mounting the same volume +retries every `ELECTION_RETRY_SECS` and stays dormant. When the process tree +dies, the kernel releases the lock. There is no follower path inside the Go +binary. -`kv-leader.info` is a single plaintext line containing the API base URL: - -``` -http://10.0.1.50:9000 -``` - -Other services read this file, then call `GET /urls` for full discovery. - -The dev compose runs a single `metacore-app`; the flock primitive is -retained as a correctness anchor (one process owns Redis and the -writable state under `/meta-core`), not as live failover. +The lock is a **mutex, not service discovery** — it only stops two Redis +writers landing on one RDB/AOF. Nothing reads it, and the `kv-leader.info` +file it used to publish is gone: siblings locate meta-core over UDP +(see [service-discovery.md](../../docs/project-architecture/service-discovery.md)). +`internal/leader` keeps its name for history, but only builds the `/urls` +payload (`baseUrl`, `apiUrl`, `webdavUrl`, `webdavUrlInternal`) that `/urls` +and the UDP announce share. ## Metadata storage shape (one-line summary) Roots are opaque UUIDv7 strings; every property is a separate Redis -STRING at `file:/`; every known CID for a file is a -reverse-index entry `cid:: → `. Full details in +STRING at `file:/`; every known CID is a key-set member +`file:/cids/` plus a reverse-index entry `cid: → ` +(the canonical CID is derived by rank on read, never stored). Full details in [`docs/metadata-storage-structure.md`](docs/metadata-storage-structure.md) and design rationale in [`docs/uuid-rooted-metadata.md`](docs/uuid-rooted-metadata.md). The authoritative registry of field semantics and value formats lives in @@ -364,14 +415,15 @@ the repo-root [`METADATA_KEYS.md`](../../METADATA_KEYS.md). | Package | Purpose | |---------|---------| -| `cmd/meta-core` | Entry point; Redis connect + schema sentinel + service wiring. | +| `cmd/meta-core` | Entry point; Redis connect + schema sentinel + UDP discovery + service wiring. | | `internal/config` | Env-driven configuration, path helpers. | -| `internal/leader` | `LeaderInfoProvider` (builds the leader info from local hostname/IP/config; no election logic). | -| `internal/storage` | Redis wrapper (`client.go`), UUIDv7 minting (`uuid.go`), CID reverse index (`cid_resolution.go`), schema sentinel (`schema_version.go`), dual-root migration (`dual_root_migration.go`). | -| `internal/cid` | CID parsing + algorithm ranking for canonical-CID selection. | -| `internal/discovery` | Service registry + dead-service cleaner. | +| `internal/leader` | `LeaderInfoProvider` — builds the `/urls` payload from hostname/IP/config; no election logic. | +| `internal/meshdisco` | meta-discovery v1 announcer + listener (the Go reference implementation). | +| `internal/storage` | Redis wrapper (`client.go`), UUIDv7 minting, CID reverse index (`cid_resolution.go`), schema sentinel, field indexes + search-index bulk reads, UDL store (`udl.go`, `udl_admin.go`), stats, one-shot migrations/sweeps. | +| `internal/cid` | CID parsing + rank ladder for canonical-CID selection (`cid-rank-vectors.json`, guarded by the meta-root `scripts/check-cid-vectors.sh`). | +| `internal/identity` | Multi-account secp256k1 keystore, challenge/proof-of-possession, signature verification. | | `internal/events` | Keyspace-notification → `meta:events` stream publisher (`meta_publisher.go`). | -| `internal/api` | HTTP server + all handlers + SSE event endpoints. | +| `internal/api` | HTTP server + all handlers + SSE event endpoints + in-memory search index + `/api/stats`. | | `internal/schema` | Live per-field schema indexer (consumes `meta:events`). | | `internal/snapshot` | Snapshot export / import / wipe. | | `internal/watcher` | File system scanner, MidHash256 computation, dispatcher, state registry. | @@ -386,10 +438,12 @@ the repo-root [`METADATA_KEYS.md`](../../METADATA_KEYS.md). 2. Build LeaderInfoProvider (no election — the bash gate already won) 3. Connect to local Redis (retry up to 30× / 1s) 4. EnsureSchemaVersion — abort if the existing Redis layout is stale -5. Start service discovery + dead-service cleaner -6. Construct API server (initialises watcher, mounts, WebDAV) -7. Start API server; on first connect, start MetaPublisher + schema Indexer - and republish existing metadata onto meta:events +5. Start meta-discovery v1 node (unless ENABLE_UDP_DISCOVERY=false) +6. Construct API server (identity keystore migration, watcher, watchers, + mounts, WebDAV) +7. Start API server + pollers + search-index refresh loop; start + MetaPublisher + schema Indexer, then warm field indexes and republish + existing metadata onto meta:events in the background 8. Wait for SIGINT/SIGTERM, shutdown in reverse order ``` @@ -400,11 +454,13 @@ the repo-root [`METADATA_KEYS.md`](../../METADATA_KEYS.md). | `github.com/gorilla/mux` | HTTP router | | `github.com/redis/go-redis/v9` | Redis client | | `github.com/google/uuid` | UUIDv7 minting | +| `github.com/decred/dcrd/dcrec/secp256k1/v4`, `golang.org/x/crypto`, `github.com/mr-tron/base58` | Identity keys, signing, AEAD key derivation | | `golang.org/x/net` | WebDAV support | -| Go stdlib | context, net/http, os, sync, syscall | +| `github.com/alicebob/miniredis/v2` | In-memory Redis for tests | Runtime requirements: Go 1.21+ at build time; the container ships Redis, -rclone, nginx, and supervisord alongside the binary. +rclone, nginx, and supervisord alongside the binary, plus the React dashboard +and metadata editor (built from `dashboard/` and `editor/` in the Dockerfile). ## License