beacon v2 discovery: advertise metamesh.core + service resources on 239.255.99.1:9099, /api/neighbors ?cap= filter

README.md

@@ -1,34 +1,42 @@
 # meta-core
 
-Standalone Go service that owns Redis, the leader-info file, the HTTP+SSE
+Standalone Go service that owns Redis, the HTTP+SSE metadata surface, the
-metadata surface, and a WebDAV file server for the MetaMesh dev/prod stack.
+User Data Layer store and signing keystore, and a WebDAV file server for the
+MetaMesh dev/prod stack.
 
 ## Overview
 
 meta-core runs as its own container (`metacore-app`). Other MetaMesh services
-(meta-sort, meta-fuse, meta-stremio, meta-dup, meta-share) discover it by
+(meta-sort, meta-fuse, meta-stremio, meta-dup, meta-search, meta-share,
-reading `/meta-core/locks/kv-leader.info` and then call its HTTP API. Redis
+meta-gateway, meta-watch, …) locate it over UDP multicast (beacon v2 —
-is not exposed outside the container — all metadata I/O is mediated by the
+its `metamesh.core` advertise carries the full `/urls` payload) and then call its
-HTTP API.
+HTTP API. `META_CORE_URL` on a client pins it instead. Redis is not published
+outside the container — all metadata I/O is mediated by the HTTP/SSE API
+(see [`docs/api-mediated-access.md`](docs/api-mediated-access.md)).
 
 ### Key capabilities
 
 | Feature | Description |
 |---------|-------------|
-| **Leader gate** | Bash `flock` loop in `docker/leader-election.sh` only `exec`s supervisord on the winner. The Go binary therefore only ever runs as leader. |
+| **Redis-volume mutex** | Bash `flock` loop in `docker/leader-election.sh` only `exec`s supervisord on the winner, so two containers sharing one `/meta-core` volume can never run two Redis writers. It is a mutex, not discovery — nothing reads the lock. |
 | **Redis owner** | Supervisord starts Redis with AOF + RDB persistence inside the container; meta-core connects to it on localhost. |
-| **HTTP API** | Typed REST surface (gorilla/mux) for metadata, KV browsing, snapshots, schema inference, mounts, watchers, services, and file access by CID. |
+| **HTTP API** | Typed REST surface (gorilla/mux) for metadata, KV browsing, snapshots, schema inference, mounts, watchers, neighbours, identity, UDL, and file access by CID. |
+| **In-memory search index** | `POST /api/metadata/search` matches against an in-memory snapshot of every record, kept current by incremental refreshes (full reconcile every 30 min) — no Redis round-trip per record on the request path (`internal/api/search_index.go`). |
 | **SSE event streams** | `/api/events/files` and `/api/events/meta` mirror the underlying Redis Streams so external services never speak Redis. |
 | **WebDAV server** | `/webdav/*` exposes the `/files` volume (read+write) for cross-service file access. nginx in front of meta-core handles caching. |
-| **Service discovery** | File-based registry under `/meta-core/services/*.json` with heartbeat + dead-service cleanup. |
+| **Service discovery** | beacon v2: UDP multicast advertise/listen on `239.255.99.1:9099`, neighbours served at `/neighbors`. The file-based registry under `/meta-core/services/*.json` it replaced is gone. |
-| **File watcher / scanner** | Recursive scan + MidHash256 computation; emits events on `file:events`. |
+| **File watcher / scanner** | Recursive scan + MidHash256 computation; emits events on `file:events`. Watched roots are POSTed watcher configs (`/meta-core/watchers.json`), not env vars. |
 | **Mount management** | rclone-only mount manager (SMB rendered into `:smb:`, plus pre-defined remotes). Read-only by construction. |
 | **UUID-rooted storage** | Roots are UUIDv7 (Crockford Base32, ULID layout); CIDs are reverse-index aliases. Schema-version sentinel refuses to boot against stale data. |
+| **Identity keystore** | Multi-account secp256k1 signing keys under `/meta-core/identity/accounts/`, created from the dashboard's Identity menu (never auto-generated). A private key is returned only by generate, or by reveal against a proof-of-possession signature (delete needs the same proof); `/api/identity/sign` refuses challenge-domain payloads. |
+| **User Data Layer** | Version-gated store for per-user signed (and, for private-tier keys, encrypted) records — My List, Continue Watching, ratings — indexed by user and CID. meta-core never inspects the crypto. See [User Data Layer](../../docs/project-architecture/user-data-layer.md). |
+| **Dashboard + editor** | nginx serves a React dashboard at `/` and the standalone metadata editor at `/editor/`. |
 
 ### Design characteristics
 
-- **Single-leader** — flock on a shared volume; no external consensus.
+- **Single writer** — flock on the `/meta-core` volume; no external consensus.
-- **HTTP-only externally** — Redis is bound to the container, never published.
+- **HTTP-only externally** — Redis is never published by the compose stacks;
+  debug with `docker exec metacore-app redis-cli …`.
 - **Schema-gated boot** — `EnsureSchemaVersion` aborts startup when the
   on-disk Redis layout predates the current build (alpha clean-wipe policy;
   see `docs/uuid-rooted-metadata.md`).
@@ -41,34 +49,31 @@ HTTP API.
 ┌──────────────────────────────────────────────────────────────────────────┐
 │                    metacore-app  (standalone container)                  │
 │                                                                          │
+│  leader-election.sh  (flock /meta-core/locks/kv-leader.lock,             │
+│                       then exec supervisord)                             │
 │  supervisord                                                             │
-│   ├─ leader-election.sh  (flock /meta-core/locks/kv-leader.lock,         │
-│   │                       writes kv-leader.info, exec supervisord)      │
 │   ├─ redis-server        (AOF + RDB on /meta-core/db/redis)              │
-│   ├─ nginx               (TLS termination, proxy_cache for /webdav)      │
+│   ├─ nginx  :80          (dashboard, /editor, API proxy,                 │
-│   ├─ rclone rcd          (mount daemon, RC API)                          │
+│   │                       proxy_cache for /webdav, rclone GUI /rclone)   │
+│   ├─ rclone rcd          (mount daemon, RC API on :5572)                 │
 │   ├─ mount-watcher.sh    (rclone mount lifecycle)                        │
 │   └─ meta-core (Go)      ─►  HTTP API + SSE + WebDAV on :9000            │
+│                          ─►  UDP advertise/listen 239.255.99.1:9099      │
 └──────────────────────────────────────────────────────────────────────────┘
                                        │
                 ┌──────────────────────┴──────────────────────┐
-                │  /meta-core (shared volume)                 │
+                │  /meta-core (meta-core's own volume)        │
-                │   locks/kv-leader.{lock,info}               │
+                │   locks/kv-leader.lock                      │
                 │   db/redis/                                 │
-                │   services/*.json                           │
+                │   identity/accounts/*.json                  │
                 │   mounts/{mounts.json, errors/}             │
                 │   watchers.json                             │
                 │   cache/ (nginx proxy_cache)                │
                 └─────────────────────────────────────────────┘
-                                       ▲
+
-        ┌──────────────────────────────┼────────────────────────────────┐
+   meta-sort / meta-fuse / meta-stremio / meta-search / meta-share / …
-        │                              │                                │
+      │  1. hear the metamesh.core advertise on UDP multicast (carries /urls)
-┌─────────────┐               ┌─────────────┐                ┌─────────────┐
+      └─ 2. HTTP + SSE (+ WebDAV) to metacore-app — never Redis, never the volume
-│  meta-sort  │               │  meta-fuse  │                │ meta-stremio│
-│             │               │             │                │             │
-│  reads      │  HTTP+SSE     │  reads      │  HTTP+SSE      │  reads      │
-│  leader.info├──────────────►│  leader.info├───────────────►│  leader.info│
-└─────────────┘               └─────────────┘                └─────────────┘
 ```
 
 ## Quick start
@@ -82,8 +87,9 @@ make test            # run Go unit tests
 ```
 
 `make docker` tags the image as `:1.0.0`, `:latest`, **and** `:local`. The
-`:local` tag is the one every sub-stack docker-compose.yml references
+`:local` tag is the one the sub-stack docker-compose files reference
-(meta-share 10-peer federation, meta-gateway dev stacks); without it the
+(meta-share federation, meta-read and meta-listen dev stacks); the meta-gateway
+dev stack (`docker-compose.gateway.yml`) uses `:latest`. Without these tags the
 sub-stacks can't see a freshly-built image by name.
 
 > **Source-change → recreate.** `make docker` rebuilds and re-tags the
@@ -94,12 +100,14 @@ sub-stacks can't see a freshly-built image by name.
 >
 > ```bash
 > make docker
-> docker compose -f packages/meta-gateway/dev/docker-compose.yml \
+> docker compose -f packages/meta-gateway/dev/docker-compose.gateway.yml \
->   up -d --force-recreate metagateway-generic-core
+>   up -d --force-recreate metagateway-hub-metacore
-> docker compose -f packages/meta-gateway/dev/docker-compose.torznab.yml \
->   up -d --force-recreate metagateway-torznab-core
 > docker compose -f packages/meta-share/dev/docker-compose.yml \
 >   --profile share-10node up -d --force-recreate metashare-core metashare-core-test
+> docker compose -f packages/meta-read/dev/docker-compose.yml \
+>   up -d --force-recreate metaread-dev-core
+> docker compose -f packages/meta-listen/dev/docker-compose.yml \
+>   up -d --force-recreate metalisten-dev-core
 > ```
 >
 > The symptom of a missed recreate is missing API fields on `/urls`
@@ -109,7 +117,7 @@ sub-stacks can't see a freshly-built image by name.
 >
 > ```bash
 > docker inspect meta-core:local --format '{{.Id}}'                # fresh SHA
-> docker ps --filter ancestor=meta-core:local --format '{{.Names}}'
+> docker ps --filter ancestor=meta-core:local --format '{{.Names}}'   # repeat for :latest
 > # any container whose Image SHA doesn't match the fresh SHA needs --force-recreate.
 > ```
 
@@ -126,36 +134,39 @@ HTTP port (no auth, bypasses Caddy) is `http://localhost:18083`.
 
 ## Configuration
 
-Environment variables consumed by `internal/config/config.go`:
+Environment variables (from `internal/config/config.go` unless noted):
 
 | Variable | Default | Description |
 |---|---|---|
-| `META_CORE_PATH` | `/meta-core` | Shared volume root (locks, db, services, mounts, cache). |
+| `META_CORE_PATH` | `/meta-core` | meta-core's own volume root (lock, Redis data, identity, mounts, watchers, cache). No other service mounts it. |
 | `FILES_PATH` | `/files` | Files volume root. |
-| `SERVICE_NAME` | `meta-core` | Identity in the service registry. |
+| `SERVICE_NAME` | `meta-core` | Name in the beacon v2 advertise. |
-| `SERVICE_VERSION` | `1.0.0` | Reported via `/status` and registry. |
+| `SERVICE_VERSION` | `1.0.0` | Reported via `/status`. |
-| `API_PORT` | `8180` | External port baked into the leader-info `baseUrl`. |
+| `API_PORT` | `8180` | External port baked into the constructed `baseUrl` when `BASE_URL` is unset. |
-| `BASE_URL` | _empty_ | Overrides the constructed `baseUrl` if set (used for HTTPS perimeter). |
+| `BASE_URL` | _empty_ | Overrides the constructed `baseUrl` (the Caddy/HTTPS perimeter URL). |
+| `META_CORE_PUBLIC_URL` | _empty_ | Browser-facing nav URL in the announce; wins over `BASE_URL` (e.g. a debug-direct port with no Caddy). |
 | `REDIS_PORT` | `6379` | Local Redis port. |
 | `META_CORE_HTTP_PORT` | `9000` | Go HTTP+SSE+WebDAV port. |
 | `META_CORE_HTTP_HOST` | `0.0.0.0` | HTTP bind. |
-| `HEALTH_CHECK_INTERVAL_MS` | `5000` | Internal health-loop cadence. |
+| `ENABLE_FILE_WATCHER` | `true` | Disable to suppress the in-process watcher entirely (`POST /api/files/register` stays available). |
-| `HEARTBEAT_INTERVAL_MS` | `30000` | Service registry heartbeat. |
+| `ENABLE_UDP_DISCOVERY` | `true` | beacon v2 advertise + listen. |
-| `STALE_THRESHOLD_MS` | `60000` | Age past which a registry entry is marked stale. |
+| `BEACON_GROUP` / `BEACON_PORT` | `239.255.99.1` / `9099` | Multicast group and UDP port. |
-| `CLEANUP_INTERVAL_MS` | `600000` | Dead-service cleaner cadence. |
+| `BEACON_INTERVAL_MS` | `10000` | Unsolicited advertise interval. |
-| `DEAD_SERVICE_THRESHOLD_MS` | `600000` | Age past which a stale entry is deleted. |
+| `META_CORE_INDEX_EXCLUDE_PREFIXES` | `categories/newznab/` | Comma-separated field prefixes left out of the in-memory search index; set empty to index everything (`internal/storage/client.go`). |
-| `WATCH_INTERVAL_MS` | `1000` | Watcher poll cadence. |
+| `ELECTION_RETRY_SECS` | `5` | flock retry interval (read by `docker/leader-election.sh`, not the Go binary). |
-| `DEBOUNCE_MS` | `30000` | File-event debounce window. |
-| `ENABLE_FILE_WATCHER` | `true` | Disable to suppress the in-process watcher entirely. |
 
-Note: there is no `META_CORE_REDIS_PORT`, no `META_CORE_SERVICE_NAME`, and
+`HEALTH_CHECK_INTERVAL_MS`, `HEARTBEAT_INTERVAL_MS`, `STALE_THRESHOLD_MS`,
-no `META_CORE_LEADER_TIMEOUT`. The Redis URL is discovered locally; service
+`CLEANUP_INTERVAL_MS`, `DEAD_SERVICE_THRESHOLD_MS`, `WATCH_INTERVAL_MS` and
-identity uses `SERVICE_NAME`; leader timeout is governed by the bash flock
+`DEBOUNCE_MS` are still parsed by `internal/config/config.go` but nothing reads
-retry interval (`ELECTION_RETRY_SECS`, default `5`).
+them (leftovers of the file-based registry); setting them has no effect. There
+is no `META_CORE_REDIS_PORT` / `META_CORE_SERVICE_NAME` /
+`META_CORE_LEADER_TIMEOUT`, and no `WATCH_PATHS` — watched folders are watcher
+configs created via `/api/watchers`.
 
 ## API reference
 
-All endpoints below are served by `internal/api/server.go`. Examples assume
+All endpoints below are registered in `internal/api/server.go` (mounts,
+watcher and watchers routes in their own packages' `handlers.go`). Examples assume
 the in-container Go port `:9000`; from the host use
 `https://metacore-dev.localhost:8083` (Caddy) or `http://localhost:18083`
 (debug-direct).
@@ -167,12 +178,43 @@ GET  /health           # storage + role
 GET  /status           # version, uptime, etc.
 GET  /leader           # current leader info (hostname, PID, timestamps)
 GET  /urls             # baseUrl, apiUrl, webdavUrl, webdavUrlInternal (redisUrl is intentionally empty)
-GET  /services         # registered services + heartbeats
+GET  /neighbors        # neighbours heard over UDP (beacon v2)
-GET  /api/services     # alias for /services
+GET  /api/neighbors    # same, under /api
-GET  /services/{name}  # one service
+GET  /api/stats        # dashboard counters (see below)
-GET  /services/cleanup/stats
 ```
 
+`/services`, `/api/services`, `/services/{name}` and `/services/cleanup/stats`
+are **gone** — the registration-and-heartbeat registry they served was replaced
+by beacon v2 (`/neighbors`). nginx answers an unknown non-`/api` path
+with the dashboard's `index.html`, so a stale caller gets `200 text/html` rather
+than a 404; that is what made the dashboard fail with
+`SyntaxError: Unexpected token '<'` until 1.0.21.
+
+#### `GET /api/stats`
+
+Every headline counter in one request, split by cost:
+
+```jsonc
+{
+  "records": 88709,        // SCARD file:__index__ — metadata roots, file or not
+  "redisKeys": 3115447,    // DBSIZE — the flat per-field keys dominate
+  "redisMemory": "588.74M",
+  "identities": 3,         // signing accounts on disk
+  "files": { "count": 4864, "totalSize": 64587914424 },  // null until first sweep
+  "udlUsers": 5,           // accounts holding User Data Layer records
+  "sweptAt": 1789402401970,
+  "sweeping": false
+}
+```
+
+The first four are O(1) commands and are live on every call. `files` and
+`udlUsers` need a keyspace walk (~0.5s and ~3s respectively on an 88k-record
+box), so they are served from a single-flight cache refreshed in the background
+with a 60s TTL — the walk is **never** on the request path, because the
+dashboard polls and ten open tabs must not cost ten Redis walks. Prefer this
+over `/api/kv/info`, which does one `GET` per root (6s+) and sums `sizeByte`
+across records that share a file.
+
 ### Metadata — primary surface
 
 These are the routes other services use. `{hash}` may be a UUID root or
@@ -198,6 +240,8 @@ GET    /api/meta/{cid}                # resolve CID → metadata document
 GET    /api/file/{cid}/info           # CID resolution metadata
 GET    /file/{cid}                    # serve file bytes (range-aware)
 POST   /file/cid                      # compute CID for an uploaded file
+POST   /api/files/register        # mint + alias one file under /files/plugin by midhash
+                                  # (the watcher's path for one file; works with the watcher OFF)
 HEAD   /data/{hash}                   # existence + size
 GET    /data/{hash}/path              # absolute path on /files
 ```
@@ -249,6 +293,44 @@ signalling).
 ```bash
 GET /api/events/files     # file:events stream (watcher events)
 GET /api/events/meta      # meta:events stream (metadata mutations)
+GET /api/events/poll      # DEPRECATED long-poll over file:events — use SSE
+```
+
+### Identity (signing keystore)
+
+Reached through meta-search's identity proxy (and directly by meta-gateway,
+meta-read, meta-listen) for record signing, UDL writes and keypair sign-in.
+
+```bash
+GET    /api/identity               # status (never the private key)
+GET    /api/identity/accounts
+POST   /api/identity/generate      # the only call that returns a new private key
+POST   /api/identity/import
+POST   /api/identity/challenge     # one-shot challenge for proof-of-possession
+POST   /api/identity/reveal        # needs proof-of-possession
+DELETE /api/identity               # needs proof-of-possession; reports purged UDL counts
+POST   /api/identity/sign
+POST   /api/identity/sign-batch
+GET    /api/identity/aead-key
+```
+
+### User Data Layer
+
+Records arrive signed (private-tier ones encrypted) as opaque base64 CBOR;
+writes are version-gated (stale → `409 stale_version`, or `accepted:false` per
+cell in a batch). Redis model in `internal/storage/udl.go`.
+
+```bash
+GET  /api/udl/record?uid&cid&key
+PUT  /api/udl/record
+PUT  /api/udl/records                      # batch
+GET  /api/udl/user/{uid}/key/{key}
+GET  /api/udl/user/{uid}/cid/{cid}
+GET  /api/udl/user/{uid}/all
+GET  /api/udl/user/{uid}/count
+GET  /api/udl/cid/{cid}/users
+GET  /api/udl/cid/{cid}/aggregate          # meta-watch Aggregate wire shape
+GET  /api/udl/users/stats
 ```
 
 ### Mounts
@@ -288,6 +370,9 @@ return a deprecation pointer at the `/api/watchers/*` routes
 
 ```bash
 POST   /api/admin/migrate-dual-roots  # reunify stranded midhash-rooted entries
+POST   /api/admin/migrate-domain-screen  # domain film|tv → screen, workForm backfill (idempotent)
+POST   /api/admin/migrate-user-cids   # backfill udl:idx:user:<uid>:cids; until run, pre-index
+                                      # profiles export as EMPTY. Returns {fixed}
 POST   /api/admin/sweep-literature-echoes[?apply=true]
        # strip query echoes off literature cards: anilistid on site cards,
        # workcid/chapterNumber/volumeNumber/chapterStart/chapterEnd on any card.
@@ -298,32 +383,29 @@ GET/PUT/DELETE /webdav/...            # mounted on /files
 No `/metrics` endpoint is exposed; observability is via container logs and
 the `/health` / `/status` JSON.
 
-## Leader gate
+## Redis-volume mutex
 
-Leader election is a bash flock loop in `docker/leader-election.sh`, not a
+`docker/leader-election.sh` is a bash flock loop, not a Go-level state
-Go-level state machine. When `flock` succeeds on `kv-leader.lock`, the
+machine. When `flock` succeeds on `/meta-core/locks/kv-leader.lock`, the
-script writes the leader API URL to `kv-leader.info` and `exec`s
+script `exec`s supervisord; a second container mounting the same volume
-supervisord. When the process tree dies, the kernel releases the lock and
+retries every `ELECTION_RETRY_SECS` and stays dormant. When the process tree
-any standby instance can take over. There is no follower path inside the
+dies, the kernel releases the lock. There is no follower path inside the Go
-Go binary — there is no `OnBecomeLeader` / `OnBecomeFollower` callback API.
+binary.
 
-`kv-leader.info` is a single plaintext line containing the API base URL:
+The lock is a **mutex, not service discovery** — it only stops two Redis
-
+writers landing on one RDB/AOF. Nothing reads it, and the `kv-leader.info`
-```
+file it used to publish is gone: siblings locate meta-core over UDP
-http://10.0.1.50:9000
+(see [beacon-v2.md](../../docs/project-architecture/beacon-v2.md)).
-```
+`internal/leader` keeps its name for history, but only builds the `/urls`
-
+payload (`baseUrl`, `apiUrl`, `webdavUrl`, `webdavUrlInternal`) that `/urls`
-Other services read this file, then call `GET /urls` for full discovery.
+and the UDP announce share.
-
-The dev compose runs a single `metacore-app`; the flock primitive is
-retained as a correctness anchor (one process owns Redis and the
-writable state under `/meta-core`), not as live failover.
 
 ## Metadata storage shape (one-line summary)
 
 Roots are opaque UUIDv7 strings; every property is a separate Redis
-STRING at `file:<uuid>/<property>`; every known CID for a file is a
+STRING at `file:<uuid>/<property>`; every known CID is a key-set member
-reverse-index entry `cid:<algorithm>:<value> → <uuid>`. Full details in
+`file:<uuid>/cids/<bareCid>` plus a reverse-index entry `cid:<bareCid> → <uuid>`
+(the canonical CID is derived by rank on read, never stored). Full details in
 [`docs/metadata-storage-structure.md`](docs/metadata-storage-structure.md)
 and design rationale in [`docs/uuid-rooted-metadata.md`](docs/uuid-rooted-metadata.md).
 The authoritative registry of field semantics and value formats lives in
@@ -333,14 +415,15 @@ the repo-root [`METADATA_KEYS.md`](../../METADATA_KEYS.md).
 
 | Package | Purpose |
 |---------|---------|
-| `cmd/meta-core` | Entry point; Redis connect + schema sentinel + service wiring. |
+| `cmd/meta-core` | Entry point; Redis connect + schema sentinel + UDP discovery + service wiring. |
 | `internal/config` | Env-driven configuration, path helpers. |
-| `internal/leader` | `LeaderInfoProvider` (builds the leader info from local hostname/IP/config; no election logic). |
+| `internal/leader` | `LeaderInfoProvider` — builds the `/urls` payload from hostname/IP/config; no election logic. |
-| `internal/storage` | Redis wrapper (`client.go`), UUIDv7 minting (`uuid.go`), CID reverse index (`cid_resolution.go`), schema sentinel (`schema_version.go`), dual-root migration (`dual_root_migration.go`). |
+| `internal/meshdisco` | beacon v2 node (advertise, probe, neighbour map) — the Go port of the spec. |
-| `internal/cid` | CID parsing + algorithm ranking for canonical-CID selection. |
+| `internal/storage` | Redis wrapper (`client.go`), UUIDv7 minting, CID reverse index (`cid_resolution.go`), schema sentinel, field indexes + search-index bulk reads, UDL store (`udl.go`, `udl_admin.go`), stats, one-shot migrations/sweeps. |
-| `internal/discovery` | Service registry + dead-service cleaner. |
+| `internal/cid` | CID parsing + rank ladder for canonical-CID selection (`cid-rank-vectors.json`, guarded by the meta-root `scripts/check-cid-vectors.sh`). |
+| `internal/identity` | Multi-account secp256k1 keystore, challenge/proof-of-possession, signature verification. |
 | `internal/events` | Keyspace-notification → `meta:events` stream publisher (`meta_publisher.go`). |
-| `internal/api` | HTTP server + all handlers + SSE event endpoints. |
+| `internal/api` | HTTP server + all handlers + SSE event endpoints + in-memory search index + `/api/stats`. |
 | `internal/schema` | Live per-field schema indexer (consumes `meta:events`). |
 | `internal/snapshot` | Snapshot export / import / wipe. |
 | `internal/watcher` | File system scanner, MidHash256 computation, dispatcher, state registry. |
@@ -355,10 +438,12 @@ the repo-root [`METADATA_KEYS.md`](../../METADATA_KEYS.md).
 2. Build LeaderInfoProvider (no election — the bash gate already won)
 3. Connect to local Redis (retry up to 30× / 1s)
 4. EnsureSchemaVersion — abort if the existing Redis layout is stale
-5. Start service discovery + dead-service cleaner
+5. Start beacon v2 node (unless ENABLE_UDP_DISCOVERY=false)
-6. Construct API server (initialises watcher, mounts, WebDAV)
+6. Construct API server (identity keystore migration, watcher, watchers,
-7. Start API server; on first connect, start MetaPublisher + schema Indexer
+   mounts, WebDAV)
-   and republish existing metadata onto meta:events
+7. Start API server + pollers + search-index refresh loop; start
+   MetaPublisher + schema Indexer, then warm field indexes and republish
+   existing metadata onto meta:events in the background
 8. Wait for SIGINT/SIGTERM, shutdown in reverse order
 ```
 
@@ -369,11 +454,13 @@ the repo-root [`METADATA_KEYS.md`](../../METADATA_KEYS.md).
 | `github.com/gorilla/mux` | HTTP router |
 | `github.com/redis/go-redis/v9` | Redis client |
 | `github.com/google/uuid` | UUIDv7 minting |
+| `github.com/decred/dcrd/dcrec/secp256k1/v4`, `golang.org/x/crypto`, `github.com/mr-tron/base58` | Identity keys, signing, AEAD key derivation |
 | `golang.org/x/net` | WebDAV support |
-| Go stdlib | context, net/http, os, sync, syscall |
+| `github.com/alicebob/miniredis/v2` | In-memory Redis for tests |
 
 Runtime requirements: Go 1.21+ at build time; the container ships Redis,
-rclone, nginx, and supervisord alongside the binary.
+rclone, nginx, and supervisord alongside the binary, plus the React dashboard
+and metadata editor (built from `dashboard/` and `editor/` in the Dockerfile).
 
 ## License
 

cmd/meta-core/main.go

@@ -1,6 +1,7 @@
 package main
 
 import (
+	"encoding/json"
 	"fmt"
 	"log"
 	"os"
@@ -10,8 +11,8 @@ import (
 
 	"github.com/metazla/meta-core/internal/api"
 	"github.com/metazla/meta-core/internal/config"
-	"github.com/metazla/meta-core/internal/discovery"
 	"github.com/metazla/meta-core/internal/leader"
+	"github.com/metazla/meta-core/internal/meshdisco"
 	"github.com/metazla/meta-core/internal/storage"
 )
 
@@ -60,20 +61,57 @@ func main() {
 	}
 	log.Printf("[meta-core] Redis schema version OK (v%d)", storage.SchemaVersion)
 
-	// Create and start service discovery
+	// Beacon v2 (docs/project-architecture/beacon-v2.md): advertise meta-core
-	disc := discovery.NewService(cfg)
+	// as metamesh.core — carrying the /urls block services locate us by — plus
-	if err := disc.Start(); err != nil {
+	// its own metamesh.service/<name> nav entry. The payload is rebuilt from
-		log.Fatalf("[meta-core] Failed to start service discovery: %v", err)
+	// leaderProvider on every tick, which keeps it and GET /urls from drifting.
+	var mesh *meshdisco.Node
+	if cfg.EnableUDPDiscovery {
+		hostname, _ := os.Hostname()
+		mesh = meshdisco.New(meshdisco.Config{
+			Name:     cfg.ServiceName,
+			Instance: hostname,
+			Version:  Version,
+			Group:    cfg.DiscoveryGroup,
+			Port:     cfg.DiscoveryPort,
+			Interval: time.Duration(cfg.DiscoveryIntervalMS) * time.Millisecond,
+			Payload: func() (string, []meshdisco.Resource) {
+				service := meshdisco.Resource{
+					ID:        "service",
+					Caps:      []string{meshdisco.CapServicePrefix + cfg.ServiceName},
+					Endpoints: map[string]string{"ui": coreBaseURL(cfg)},
 				}
-
+				info := leaderProvider.LeaderInfo()
-	// Create and start dead service cleaner
+				if info == nil {
-	cleaner := discovery.NewCleaner(cfg)
+					return "starting", []meshdisco.Resource{service}
-	if err := cleaner.Start(); err != nil {
+				}
-		log.Printf("[meta-core] Warning: failed to start service cleaner: %v", err)
+				data, _ := json.Marshal(map[string]meshdisco.URLs{"urls": {
+					Hostname:          info.Hostname,
+					BaseUrl:           info.BaseUrl,
+					ApiUrl:            info.ApiUrl,
+					WebdavUrl:         info.WebdavUrl,
+					WebdavUrlInternal: info.WebdavUrlInternal,
+				}})
+				core := meshdisco.Resource{
+					ID:        "core",
+					Caps:      []string{meshdisco.CapCore},
+					Endpoints: map[string]string{"http": info.ApiUrl},
+					Data:      data,
+				}
+				return "running", []meshdisco.Resource{core, service}
+			},
+		})
+	if err := mesh.Start(); err != nil {
+			log.Printf("[meta-core] Warning: UDP discovery disabled: %v", err)
+			mesh = nil
+		}
+	} else {
+		log.Println("[meta-core] UDP discovery disabled by ENABLE_UDP_DISCOVERY")
 	}
 
 	// Create and start API server
-	apiServer := api.NewServer(cfg, leaderProvider, disc, cleaner, storageClient)
+	apiServer := api.NewServer(cfg, leaderProvider, storageClient)
+	apiServer.SetMeshNode(mesh)
 	if err := apiServer.Start(); err != nil {
 		log.Fatalf("[meta-core] Failed to start API server: %v", err)
 	}
@@ -90,12 +128,10 @@ func main() {
 		log.Printf("[meta-core] Error stopping API server: %v", err)
 	}
 
-	if err := cleaner.Stop(); err != nil {
+	if mesh != nil {
-		log.Printf("[meta-core] Error stopping service cleaner: %v", err)
+		if err := mesh.Stop(); err != nil {
+			log.Printf("[meta-core] Error stopping UDP discovery: %v", err)
 		}
-
-	if err := disc.Stop(); err != nil {
-		log.Printf("[meta-core] Error stopping service discovery: %v", err)
 	}
 
 	if err := storageClient.Close(); err != nil {
@@ -112,3 +148,17 @@ func waitForShutdown() {
 	<-sigChan
 }
 
+// coreBaseURL is the browser-facing URL announced for the nav menu. It keeps
+// the three-way fallback the file registry used (discovery/service.go):
+// META_CORE_PUBLIC_URL wins (the reachable URL when there is no Caddy
+// perimeter, e.g. a debug-direct port), then BASE_URL (the Caddy URL), then
+// the container IP.
+func coreBaseURL(cfg *config.Config) string {
+	if cfg.PublicURL != "" {
+		return cfg.PublicURL
+	}
+	if cfg.BaseURL != "" {
+		return cfg.BaseURL
+	}
+	return fmt.Sprintf("http://%s:%d", meshdisco.LocalIPv4(), cfg.APIPort)
+}

dashboard/dist/index.html

@@ -21,7 +21,7 @@
         min-height: 100vh;
       }
     </style>
-    <script type="module" crossorigin src="/assets/index-BW9EQGCP.js"></script>
+    <script type="module" crossorigin src="/assets/index-Bgo6moAc.js"></script>
     <link rel="stylesheet" crossorigin href="/assets/index-DtR_2V4s.css">
   </head>
   <body>

dashboard/src/components/ServiceNav.tsx

@@ -1,155 +1,53 @@
-import { useState, useEffect } from 'react';
+import { useEffect } from 'react';
+// eslint-disable-next-line @typescript-eslint/ban-ts-comment
+// @ts-ignore -- plain ES module, no .d.ts; imported for its side effect
+// of defining the custom element.
+import '../meta-service-menu.js';
 
-interface ServiceInfo {
+/**
-  name: string;
+ * ServiceNav — the neighbouring meta-* services, as a burger menu.
-  hostname: string;
+ *
-  baseUrl: string;
+ * Since meta-discovery v1 this is a thin wrapper around <meta-service-menu>,
-  status: string;
+ * a framework-free custom element shared by every meta-* UI (see
-  lastHeartbeat: string;
+ * scripts/check-mirrors.sh). The old implementation was ~165 lines of React
+ * that existed in four near-identical copies and polled meta-core's
+ * /api/services; the element polls this service's own /api/neighbors instead,
+ * so the nav still renders when meta-core is down.
+ *
+ * The element lives in shadow DOM and cannot see this app's CSS variables, so
+ * the four --mm-nav-* tokens below map our palette onto it.
+ */
+
+declare global {
+    // eslint-disable-next-line @typescript-eslint/no-namespace
+    namespace JSX {
+        interface IntrinsicElements {
+            'meta-service-menu': React.DetailedHTMLProps<
+                React.HTMLAttributes<HTMLElement>,
+                HTMLElement
+            > & { current?: string; endpoint?: string; label?: string };
         }
-
-interface ServicesResponse {
-  services: ServiceInfo[];
-  current: string;
     }
-
-const serviceIcons: Record<string, string> = {
-  'meta-core': '🗄️',
-  'meta-sort': '📁',
-  'meta-fuse': '🗂️',
-  'meta-stremio': '🎬',
-  'meta-orbit': '🌐',
-  'default': '📦'
-};
-
-function formatServiceName(name: string): string {
-  return name.split('-').map(word =>
-    word.charAt(0).toUpperCase() + word.slice(1)
-  ).join(' ');
 }
 
 function ServiceNav() {
-  const [services, setServices] = useState<ServiceInfo[]>([]);
-  const [loading, setLoading] = useState(true);
-  const currentService = 'meta-core';
-
     useEffect(() => {
-    const fetchServices = async () => {
+        // Import above is for the side effect of defining the element; this
-      try {
+        // keeps bundlers from tree-shaking it away in production builds.
-        const response = await fetch('/services');
-        if (response.ok) {
-          const data: ServicesResponse = await response.json();
-          setServices(data.services || []);
-        }
-      } catch (error) {
-        console.error('Failed to fetch services:', error);
-      } finally {
-        setLoading(false);
-      }
-    };
-
-    fetchServices();
-    const interval = setInterval(fetchServices, 30000);
-    return () => clearInterval(interval);
     }, []);
 
-  if (loading) return null;
-
-  const sortedServices = [...services].sort((a, b) => a.name.localeCompare(b.name));
-
-  if (sortedServices.length === 0) return null;
-
     return (
-    <nav className="services-nav">
+        <>
-      <span className="services-nav-label">Services:</span>
+            <meta-service-menu current="meta-core" />
-      <div className="services-nav-items">
-        {sortedServices.map(service => {
-          const icon = serviceIcons[service.name] || serviceIcons.default;
-          const isActive = service.name === currentService;
-          return (
-            <a
-              key={service.name}
-              href={isActive ? '#' : service.baseUrl}
-              className={`service-link${isActive ? ' active' : ''}`}
-              onClick={(e) => {
-                e.preventDefault();
-                if (!isActive) {
-                  window.location.href = service.baseUrl;
-                }
-              }}
-            >
-              <span className="service-icon">{icon}</span>
-              <span>{formatServiceName(service.name)}</span>
-              <span className="service-status"></span>
-            </a>
-          );
-        })}
-      </div>
-
             <style>{`
-        .services-nav {
+                meta-service-menu {
-          display: flex;
+                    --mm-nav-fg: var(--text-primary, #e0e0e0);
-          align-items: center;
+                    --mm-nav-bg: var(--bg-tertiary, var(--bg-secondary, #1a1a2e));
-          gap: 0.5rem;
+                    --mm-nav-border: var(--border-color, rgba(255,255,255,0.14));
-        }
+                    --mm-nav-accent: var(--accent-primary, #4ecdc4);
-
-        .services-nav-label {
-          color: #888;
-          font-size: 0.8rem;
-          white-space: nowrap;
-        }
-
-        .services-nav-items {
-          display: flex;
-          gap: 0.35rem;
-        }
-
-        .service-link {
-          display: inline-flex;
-          align-items: center;
-          gap: 0.3rem;
-          padding: 0.35rem 0.6rem;
-          border-radius: 6px;
-          text-decoration: none;
-          font-size: 0.8rem;
-          font-weight: 500;
-          transition: all 0.2s;
-          color: #ccc;
-          background: #0f3460;
-          border: 1px solid #1a4a7a;
-        }
-
-        .service-link:hover {
-          background: #1a4a7a;
-          border-color: rgba(78, 205, 196, 0.5);
-          text-decoration: none;
-          color: #fff;
-        }
-
-        .service-link.active {
-          background: linear-gradient(135deg, rgba(78, 205, 196, 0.2), rgba(68, 160, 141, 0.2));
-          border-color: rgba(78, 205, 196, 0.5);
-          color: #fff;
-        }
-
-        .service-link .service-icon {
-          font-size: 0.9rem;
-        }
-
-        .service-link .service-status {
-          width: 5px;
-          height: 5px;
-          border-radius: 50%;
-          background: #4ecdc4;
-        }
-
-        @media (max-width: 800px) {
-          .services-nav-label {
-            display: none;
-          }
                 }
             `}</style>
-    </nav>
+        </>
     );
 }
 

dashboard/src/lib/api.ts

@@ -0,0 +1,28 @@
+// JSON fetch that fails with a sentence instead of a parser error.
+//
+// nginx serves the dashboard with `try_files $uri $uri/ /index.html`, so ANY
+// path it does not have a proxy rule for answers 200 text/html with the SPA
+// shell. A caller that does `res.ok && res.json()` on such a path gets
+// `SyntaxError: Unexpected token '<', "<!doctype "... is not valid JSON` and
+// no clue which request produced it — which is exactly how a route deleted
+// from the Go router (`/services`, removed in favour of /api/neighbors) sat
+// broken on the dashboard.
+//
+// So: check the content type, and name the URL in every error.
+
+export async function getJSON<T>(url: string, init?: RequestInit): Promise<T> {
+  const res = await fetch(url, init);
+  const ct = res.headers.get('content-type') || '';
+
+  if (!res.ok) {
+    throw new Error(`${url} → HTTP ${res.status}`);
+  }
+  if (!ct.includes('application/json')) {
+    // The SPA fallback is the overwhelmingly likely cause; say so.
+    throw new Error(
+      `${url} → ${res.status} ${ct || 'no content-type'} (not JSON — the route is ` +
+        `missing and nginx served the dashboard shell instead)`
+    );
+  }
+  return (await res.json()) as T;
+}

dashboard/src/meta-service-menu.js

@@ -0,0 +1,308 @@
+/**
+ * <meta-service-menu> — the neighbouring meta-* services, as a burger menu.
+ *
+ * Replaces the eight divergent "services bar" implementations (four copies of
+ * a React `ServiceNav.tsx`, two mirrored `services.js`, two bespoke inline
+ * ones) with a single framework-free custom element. It drops into the React
+ * dashboards as a plain tag — no React dependency — and into the vanilla UIs
+ * as an ES module.
+ *
+ * Data comes from the host service's own `GET /api/neighbors`, same-origin.
+ * Never cross-origin: the dev stack's self-signed certs and the OIDC redirects
+ * make a fetch to a neighbour a dead end. Links are plain navigations.
+ *
+ * ⚠ MIRRORED FILE. Byte-identical copies live in every meta-* UI tree; see
+ * `scripts/check-mirrors.sh`. Edit one → edit all.
+ *
+ * Shadow DOM is not optional here. The eight UIs use four incompatible token
+ * vocabularies, and two of them define *different colours under the same
+ * names* (`--bg-secondary` is #12121a in meta-sort, #16213e in meta-core,
+ * #161b22 in meta-dup). meta-search and meta-share are also the only
+ * light-mode-capable UIs, so a hardcoded dark palette would break them.
+ * The element therefore styles itself from four tokens with dark defaults:
+ *
+ *     --mm-nav-fg, --mm-nav-bg, --mm-nav-border, --mm-nav-accent
+ *
+ * Each host maps its own variables onto those, e.g.
+ *     meta-service-menu { --mm-nav-bg: var(--bg-secondary); }
+ *
+ * Attributes:
+ *   current   name of the host service, so it can be marked and not linked
+ *   endpoint  override the default "/api/neighbors"
+ *   label     button tooltip (default "Services")
+ *
+ * Property:
+ *   rewriteUrl  optional (name, url, neighbour) => url, for hosts that must
+ *               translate internal hostnames into browser-reachable ones
+ *               (meta-gateway does this for its dev/prod split).
+ */
+
+const ICONS = {
+  "meta-core": "⚙️",
+  "meta-sort": "📁",
+  "meta-fuse": "🗂️",
+  "meta-stremio": "🎬",
+  "meta-dup": "🔍",
+  "meta-search": "🔎",
+  "meta-share": "🔗",
+  "meta-gateway": "🌉",
+  "meta-watch": "📺",
+  "meta-listen": "🎧",
+  "meta-read": "📖",
+};
+const DEFAULT_ICON = "📦";
+const POLL_MS = 30000;
+
+function titleCase(name) {
+  return String(name || "")
+    .split("-")
+    .map((w) => w.charAt(0).toUpperCase() + w.slice(1))
+    .join(" ");
+}
+
+const TEMPLATE = `
+<style>
+  :host {
+    --_fg: var(--mm-nav-fg, #e8eaed);
+    --_bg: var(--mm-nav-bg, #1a1a2e);
+    --_border: var(--mm-nav-border, rgba(255, 255, 255, 0.14));
+    --_accent: var(--mm-nav-accent, #4ecdc4);
+    position: relative;
+    display: inline-block;
+    font-family: inherit;
+    font-size: 14px;
+    line-height: 1.4;
+  }
+  button.trigger {
+    display: inline-flex;
+    align-items: center;
+    justify-content: center;
+    gap: 0.4rem;
+    width: 2.1rem;
+    height: 2.1rem;
+    padding: 0;
+    cursor: pointer;
+    color: var(--_fg);
+    background: transparent;
+    border: 1px solid var(--_border);
+    border-radius: 8px;
+    font: inherit;
+    transition: border-color 0.15s, background 0.15s;
+  }
+  button.trigger:hover { border-color: var(--_accent); }
+  button.trigger:focus-visible {
+    outline: 2px solid var(--_accent);
+    outline-offset: 2px;
+  }
+  button.trigger[aria-expanded="true"] { border-color: var(--_accent); }
+  .bars { display: block; width: 15px; }
+  .bars span {
+    display: block;
+    height: 2px;
+    margin: 3px 0;
+    border-radius: 2px;
+    background: currentColor;
+  }
+  .panel {
+    position: absolute;
+    top: calc(100% + 6px);
+    right: 0;
+    z-index: 1000;
+    min-width: 13rem;
+    max-height: 70vh;
+    overflow-y: auto;
+    padding: 0.3rem;
+    background: var(--_bg);
+    border: 1px solid var(--_border);
+    border-radius: 10px;
+    box-shadow: 0 10px 30px rgba(0, 0, 0, 0.35);
+  }
+  .panel[hidden] { display: none; }
+  a.item, .item {
+    display: flex;
+    align-items: center;
+    gap: 0.55rem;
+    padding: 0.45rem 0.6rem;
+    border-radius: 6px;
+    color: var(--_fg);
+    text-decoration: none;
+    white-space: nowrap;
+  }
+  a.item:hover { background: color-mix(in srgb, var(--_accent) 18%, transparent); }
+  .item.active { color: var(--_accent); font-weight: 600; cursor: default; }
+  .item.dead { opacity: 0.55; cursor: not-allowed; }
+  .icon { font-size: 1rem; line-height: 1; }
+  .dot {
+    width: 6px;
+    height: 6px;
+    margin-left: auto;
+    border-radius: 50%;
+    background: var(--_accent);
+  }
+  .empty {
+    padding: 0.6rem;
+    opacity: 0.7;
+    font-size: 0.85em;
+    white-space: nowrap;
+  }
+  /* Respect a host that opts out of motion. */
+  @media (prefers-reduced-motion: reduce) {
+    button.trigger { transition: none; }
+  }
+</style>
+<button class="trigger" type="button" aria-haspopup="menu" aria-expanded="false">
+  <span class="bars" aria-hidden="true"><span></span><span></span><span></span></span>
+</button>
+<div class="panel" role="menu" hidden></div>
+`;
+
+class MetaServiceMenu extends HTMLElement {
+  constructor() {
+    super();
+    this.attachShadow({ mode: "open" });
+    this.shadowRoot.innerHTML = TEMPLATE;
+    this._btn = this.shadowRoot.querySelector("button.trigger");
+    this._panel = this.shadowRoot.querySelector(".panel");
+    this._neighbors = [];
+    this._timer = null;
+    this._open = false;
+
+    this._onDocClick = (e) => {
+      if (!this._open) return;
+      if (e.composedPath().includes(this)) return;
+      this._setOpen(false);
+    };
+    this._onKey = (e) => {
+      if (e.key === "Escape" && this._open) {
+        this._setOpen(false);
+        this._btn.focus();
+      }
+    };
+  }
+
+  connectedCallback() {
+    this._btn.title = this.getAttribute("label") || "Services";
+    this._btn.setAttribute("aria-label", this._btn.title);
+    this._btn.addEventListener("click", () => this._setOpen(!this._open));
+    document.addEventListener("click", this._onDocClick);
+    document.addEventListener("keydown", this._onKey);
+
+    this.refresh();
+    this._timer = setInterval(() => this.refresh(), POLL_MS);
+  }
+
+  disconnectedCallback() {
+    if (this._timer) clearInterval(this._timer);
+    this._timer = null;
+    document.removeEventListener("click", this._onDocClick);
+    document.removeEventListener("keydown", this._onKey);
+  }
+
+  get endpoint() {
+    return this.getAttribute("endpoint") || "/api/neighbors";
+  }
+
+  get current() {
+    return this.getAttribute("current") || "";
+  }
+
+  async refresh() {
+    try {
+      const res = await fetch(this.endpoint, {
+        headers: { Accept: "application/json" },
+      });
+      if (!res.ok) throw new Error(String(res.status));
+      const data = await res.json();
+      // `services` is the legacy alias some backends still emit.
+      this._neighbors = data.neighbors || data.services || [];
+    } catch {
+      // Silent: the menu is navigation, not a health indicator. Keep whatever
+      // we last had rather than blanking the list on one failed poll.
+    }
+    this._render();
+  }
+
+  _setOpen(open) {
+    this._open = open;
+    this._panel.hidden = !open;
+    this._btn.setAttribute("aria-expanded", String(open));
+    if (open) this.refresh();
+  }
+
+  _render() {
+    const panel = this._panel;
+    panel.textContent = "";
+
+    const seen = new Set();
+    const rows = [];
+    for (const n of this._neighbors) {
+      const name = n.name || "";
+      if (!name || seen.has(name)) continue;
+      // Beacon v2 neighbours include plugins; the nav lists services only.
+      // (A row without `caps` comes from a pre-v2 backend — keep it.)
+      const isService = (c) => String(c).startsWith("metamesh.service/");
+      if (Array.isArray(n.caps) && !n.caps.some(isService)) continue;
+      seen.add(name);
+      rows.push(n);
+    }
+    // Always show the host itself, even before its own echo comes back.
+    if (this.current && !seen.has(this.current)) {
+      rows.push({ name: this.current, baseUrl: "" });
+    }
+    rows.sort((a, b) => String(a.name).localeCompare(String(b.name)));
+
+    if (rows.length === 0) {
+      const empty = document.createElement("div");
+      empty.className = "empty";
+      empty.textContent = "No services found";
+      panel.appendChild(empty);
+      return;
+    }
+
+    for (const n of rows) {
+      const isActive = n.name === this.current;
+      let url = n.baseUrl || "";
+      if (typeof this.rewriteUrl === "function") {
+        try {
+          url = this.rewriteUrl(n.name, url, n) || "";
+        } catch {
+          /* a bad hook must not blank the menu */
+        }
+      }
+
+      // A neighbour with no reachable URL is shown but not linked — better
+      // than a link that goes nowhere.
+      const el = document.createElement(isActive || !url ? "span" : "a");
+      el.className =
+        "item" + (isActive ? " active" : "") + (!isActive && !url ? " dead" : "");
+      if (el.tagName === "A") {
+        el.href = url;
+        el.setAttribute("role", "menuitem");
+      }
+      if (!isActive && !url) el.title = "No reachable URL advertised";
+
+      const icon = document.createElement("span");
+      icon.className = "icon";
+      icon.textContent = ICONS[n.name] || DEFAULT_ICON;
+
+      const label = document.createElement("span");
+      label.textContent = titleCase(n.name);
+
+      el.append(icon, label);
+      // Beacon v2 omits `status` when running.
+      if ((n.status || "running") === "running" || isActive) {
+        const dot = document.createElement("span");
+        dot.className = "dot";
+        dot.title = n.status || "running";
+        el.appendChild(dot);
+      }
+      panel.appendChild(el);
+    }
+  }
+}
+
+if (!customElements.get("meta-service-menu")) {
+  customElements.define("meta-service-menu", MetaServiceMenu);
+}
+
+export default MetaServiceMenu;

dashboard/src/pages/Dashboard.tsx

@@ -1,4 +1,5 @@
 import { useEffect, useState } from 'react';
+import { getJSON } from '../lib/api';
 
 interface HealthStatus {
   status: string;
@@ -6,6 +7,22 @@ interface HealthStatus {
   timestamp: string;
 }
 
+interface FileStats {
+  count: number;
+  totalSize: number;
+}
+
+interface Stats {
+  records: number;
+  redisKeys: number;
+  redisMemory: string;
+  identities: number;
+  files: FileStats | null;
+  udlUsers: number | null;
+  sweptAt: number;
+  sweeping: boolean;
+}
+
 interface WatcherStatus {
   id: string;
   path: string;
@@ -22,13 +39,6 @@ interface WatchersResponse {
   count: number;
 }
 
-interface ServiceInfo {
-  name: string;
-  api: string;
-  capabilities: string[];
-  timestamp: number;
-}
-
 const cardStyle: React.CSSProperties = {
   background: '#16213e',
   borderRadius: '8px',
@@ -38,53 +48,110 @@ const cardStyle: React.CSSProperties = {
 
 const gridStyle: React.CSSProperties = {
   display: 'grid',
-  gridTemplateColumns: 'repeat(auto-fit, minmax(300px, 1fr))',
+  // 200px, not 220: five tiles must fit one row inside the 1200px container
+  // (1136 content - 4x16 gap = 1072 / 5 = 214 each) before wrapping.
+  gridTemplateColumns: 'repeat(auto-fit, minmax(200px, 1fr))',
   gap: '1rem',
+  marginBottom: '1rem',
 };
 
 const statStyle: React.CSSProperties = {
   fontSize: '2rem',
   fontWeight: 'bold',
   color: '#4ade80',
+  lineHeight: 1.2,
 };
 
+const labelStyle: React.CSSProperties = {
+  marginBottom: '0.75rem',
+  color: '#888',
+  fontSize: '0.85rem',
+  textTransform: 'uppercase',
+  letterSpacing: '0.04em',
+};
+
+const noteStyle: React.CSSProperties = {
+  color: '#6b7a99',
+  fontSize: '0.8rem',
+  marginTop: '0.4rem',
+};
+
+const numberFmt = new Intl.NumberFormat();
+
+function formatCount(n: number | null | undefined): string {
+  return n === null || n === undefined ? '—' : numberFmt.format(n);
+}
+
+// Binary units: this counts bytes on disk, and every other tool the operator
+// compares against (du, docker, the mount page) reports GiB.
+function formatBytes(bytes: number | null | undefined): string {
+  if (bytes === null || bytes === undefined) return '—';
+  const units = ['B', 'KiB', 'MiB', 'GiB', 'TiB', 'PiB'];
+  let value = bytes;
+  let unit = 0;
+  while (value >= 1024 && unit < units.length - 1) {
+    value /= 1024;
+    unit++;
+  }
+  return `${value.toFixed(value >= 100 || unit === 0 ? 0 : 1)} ${units[unit]}`;
+}
+
+function Tile({
+  label,
+  value,
+  note,
+  pending,
+}: {
+  label: string;
+  value: string;
+  note?: string;
+  pending?: boolean;
+}) {
+  return (
+    <div style={cardStyle}>
+      <div style={labelStyle}>{label}</div>
+      <div style={{ ...statStyle, color: pending ? '#6b7a99' : statStyle.color }}>
+        {value}
+      </div>
+      {note && <div style={noteStyle}>{note}</div>}
+    </div>
+  );
+}
+
 export default function Dashboard() {
   const [health, setHealth] = useState<HealthStatus | null>(null);
+  const [stats, setStats] = useState<Stats | null>(null);
   const [watchersData, setWatchersData] = useState<WatchersResponse | null>(null);
-  const [services, setServices] = useState<ServiceInfo[]>([]);
   const [error, setError] = useState<string | null>(null);
 
   useEffect(() => {
+    let cancelled = false;
+
     const fetchData = async () => {
-      try {
+      // Settled, not all: one dead endpoint must not blank every tile.
-        // Fetch health
+      const [healthRes, statsRes, watchersRes] = await Promise.allSettled([
-        const healthRes = await fetch('/health');
+        getJSON<HealthStatus>('/health'),
-        if (healthRes.ok) {
+        getJSON<Stats>('/api/stats'),
-          setHealth(await healthRes.json());
+        getJSON<WatchersResponse>('/api/watchers'),
-        }
+      ]);
+      if (cancelled) return;
 
-        // Fetch watchers status
+      if (healthRes.status === 'fulfilled') setHealth(healthRes.value);
-        const watchersRes = await fetch('/api/watchers');
+      if (statsRes.status === 'fulfilled') setStats(statsRes.value);
-        if (watchersRes.ok) {
+      if (watchersRes.status === 'fulfilled') setWatchersData(watchersRes.value);
-          setWatchersData(await watchersRes.json());
-        }
 
-        // Fetch services
+      const failures = [healthRes, statsRes, watchersRes]
-        const servicesRes = await fetch('/services');
+        .filter((r): r is PromiseRejectedResult => r.status === 'rejected')
-        if (servicesRes.ok) {
+        .map((r) => String(r.reason?.message ?? r.reason));
-          const data = await servicesRes.json();
+      setError(failures.length ? failures.join(' · ') : null);
-          setServices(data.services || []);
-        }
-
-        setError(null);
-      } catch (err) {
-        setError(String(err));
-      }
     };
 
     fetchData();
     const interval = setInterval(fetchData, 5000);
-    return () => clearInterval(interval);
+    return () => {
+      cancelled = true;
+      clearInterval(interval);
+    };
   }, []);
 
   const handleTriggerScan = async () => {
@@ -95,49 +162,102 @@ export default function Dashboard() {
     }
   };
 
-  // Aggregate watcher stats
+  const isScanning = watchersData?.watchers.some((w) => w.isScanning) || false;
-  const totalFiles = watchersData?.watchers.reduce((sum, w) => sum + w.fileCount, 0) || 0;
+  const activeWatchers = watchersData?.watchers.filter((w) => w.active).length || 0;
-  const isScanning = watchersData?.watchers.some(w => w.isScanning) || false;
+  const lastScan =
-  const activeWatchers = watchersData?.watchers.filter(w => w.active).length || 0;
+    watchersData?.watchers.reduce((max, w) => Math.max(max, w.lastScan || 0), 0) || 0;
-  const lastScan = watchersData?.watchers.reduce((max, w) => Math.max(max, w.lastScan || 0), 0) || 0;
+
+  // Records we hold metadata for but have no local file: gateway/cid-rooted
+  // entries, plus anything whose file went away. The interesting number on a
+  // client box, where it should be most of them.
+  const fileless =
+    stats && stats.files ? Math.max(0, stats.records - stats.files.count) : null;
+
+  const sweepPending = !!stats && !stats.files;
+  const sweptNote = (() => {
+    if (!stats) return undefined;
+    if (sweepPending) return stats.sweeping ? 'scanning…' : 'not scanned yet';
+    if (stats.sweeping) return 'refreshing…';
+    if (stats.sweptAt) return `as of ${new Date(stats.sweptAt).toLocaleTimeString()}`;
+    return undefined;
+  })();
 
   return (
-    <div style={{ padding: '2rem', maxWidth: '1200px', margin: '0 auto' }}>
+    <div style={{ padding: '2rem', maxWidth: '1200px', margin: '0 auto', overflowY: 'auto', height: '100%' }}>
       <h1 style={{ marginBottom: '2rem' }}>meta-core Dashboard</h1>
 
       {error && (
         <div style={{ ...cardStyle, background: '#4a1a1a', color: '#f87171' }}>
-          Error: {error}
+          {error}
         </div>
       )}
 
       <div style={gridStyle}>
-        {/* Health Card */}
+        <Tile
+          label="Files discovered"
+          value={formatCount(stats?.files?.count)}
+          note={sweptNote}
+          pending={sweepPending}
+        />
+        <Tile
+          label="Library size"
+          value={formatBytes(stats?.files?.totalSize)}
+          note={sweptNote}
+          pending={sweepPending}
+        />
+        <Tile
+          label="Metadata entries"
+          value={formatCount(stats?.records)}
+          note={
+            fileless === null
+              ? 'records in the index'
+              : `${formatCount(fileless)} with no local file`
+          }
+        />
+        <Tile
+          label="Redis keys"
+          value={formatCount(stats?.redisKeys)}
+          note={stats?.redisMemory ? `${stats.redisMemory} in use` : undefined}
+        />
+        <Tile
+          label="Identities"
+          value={formatCount(stats?.identities)}
+          note={
+            stats?.udlUsers !== null && stats?.udlUsers !== undefined
+              ? `${formatCount(stats.udlUsers)} with user data`
+              : 'signing accounts'
+          }
+        />
+      </div>
+
+      <div style={{ ...gridStyle, gridTemplateColumns: 'repeat(auto-fit, minmax(300px, 1fr))' }}>
         <div style={cardStyle}>
-          <h3 style={{ marginBottom: '1rem', color: '#888' }}>Service Health</h3>
+          <div style={labelStyle}>Service health</div>
           {health ? (
             <>
               <div style={statStyle}>
                 {health.status === 'ok' ? 'Healthy' : health.status}
               </div>
-              <p>Redis: {health.redis ? 'Connected' : 'Disconnected'}</p>
+              <p style={noteStyle}>
+                Redis: {health.redis ? 'Connected' : 'Disconnected'}
+              </p>
             </>
           ) : (
-            <p>Loading...</p>
+            <p style={noteStyle}>Loading…</p>
           )}
         </div>
 
-        {/* Watchers Status Card */}
         <div style={cardStyle}>
-          <h3 style={{ marginBottom: '1rem', color: '#888' }}>File Watchers</h3>
+          <div style={labelStyle}>File watchers</div>
           {watchersData ? (
             <>
-              <div style={statStyle}>{totalFiles}</div>
+              <div style={statStyle}>
-              <p>Files discovered</p>
+                {activeWatchers} / {watchersData.count}
-              <p>Watchers: {activeWatchers} active / {watchersData.count} total</p>
+              </div>
-              <p>Status: {isScanning ? 'Scanning...' : 'Idle'}</p>
+              <p style={noteStyle}>active watchers</p>
+              <p style={noteStyle}>Status: {isScanning ? 'Scanning…' : 'Idle'}</p>
               {lastScan > 0 && (
-                <p>Last scan: {new Date(lastScan).toLocaleString()}</p>
+                <p style={noteStyle}>Last scan: {new Date(lastScan).toLocaleString()}</p>
               )}
               <button
                 onClick={handleTriggerScan}
@@ -155,27 +275,7 @@ export default function Dashboard() {
               </button>
             </>
           ) : (
-            <p>Loading...</p>
+            <p style={noteStyle}>Loading…</p>
-          )}
-        </div>
-
-        {/* Services Card */}
-        <div style={cardStyle}>
-          <h3 style={{ marginBottom: '1rem', color: '#888' }}>Connected Services</h3>
-          <div style={statStyle}>{services.length}</div>
-          <p>Services registered</p>
-          {services.length > 0 && (
-            <ul style={{ marginTop: '1rem', paddingLeft: '1.5rem' }}>
-              {services.map((svc) => (
-                <li key={svc.name} style={{ marginBottom: '0.5rem' }}>
-                  <strong>{svc.name}</strong>
-                  <br />
-                  <span style={{ color: '#888', fontSize: '0.9rem' }}>
-                    {svc.capabilities?.join(', ') || 'No capabilities'}
-                  </span>
-                </li>
-              ))}
-            </ul>
           )}
         </div>
       </div>

docker/entrypoint.sh

@@ -10,7 +10,6 @@ set -e
 # Create required directories
 mkdir -p /meta-core/locks
 mkdir -p /meta-core/db/redis
-mkdir -p /meta-core/services
 mkdir -p /meta-core/mounts/errors
 mkdir -p /files
 mkdir -p /var/log/supervisor

docker/leader-election.sh

@@ -2,9 +2,13 @@
 # =============================================================================
 # Leader Election Script for meta-core
 #
-# Uses flock for leader election. Only the leader runs the full application
+# The flock here is a MUTEX, not service discovery. It is the only thing
-# stack (supervisord with Redis, meta-core, nginx, rclone, mount-watcher).
+# stopping two containers that mount the same /meta-core volume from putting
-# Followers remain dormant, periodically retrying to acquire the lock.
+# two Redis writers on one RDB/AOF. Sibling services no longer read anything
+# from this directory — they locate meta-core over UDP (meta-discovery v1, see
+# docs/project-architecture/service-discovery.md), so the kv-leader.info file
+# this script used to publish is gone. Keep the lock; it costs nothing and the
+# corruption it prevents is unrecoverable.
 #
 # Environment variables:
 #   META_CORE_PATH         - Path to meta-core data directory (default: /meta-core)
@@ -20,7 +24,6 @@ META_CORE_HTTP_PORT="${META_CORE_HTTP_PORT:-9000}"
 ELECTION_RETRY_SECS="${ELECTION_RETRY_SECS:-5}"
 
 LOCK_FILE="${META_CORE_PATH}/locks/kv-leader.lock"
-INFO_FILE="${META_CORE_PATH}/locks/kv-leader.info"
 SUPERVISORD_CONF="/etc/supervisor/conf.d/supervisord.conf"
 
 # Get the container's IP address
@@ -40,35 +43,13 @@ get_local_ip() {
     echo "$ip"
 }
 
-# Write leader info file (plain text API URL)
-write_leader_info() {
-    local ip=$(get_local_ip)
-    local api_url="http://${ip}:${META_CORE_HTTP_PORT}"
-
-    # Write atomically using temp file + rename
-    local temp_file="${INFO_FILE}.tmp"
-    echo "$api_url" > "$temp_file"
-    mv "$temp_file" "$INFO_FILE"
-
-    echo "[election] Wrote leader info: $api_url"
-}
-
-# Clean up leader info file on shutdown
-cleanup() {
-    echo "[election] Cleaning up..."
-    rm -f "$INFO_FILE"
-    exit 0
-}
-
 # Set up signal handlers for graceful shutdown
-trap cleanup SIGTERM SIGINT
 
 # Ensure lock directory exists
 mkdir -p "$(dirname "$LOCK_FILE")"
 
 echo "[election] Starting leader election..."
 echo "[election] Lock file: $LOCK_FILE"
-echo "[election] Info file: $INFO_FILE"
 
 # Open lock file for flock (file descriptor 200)
 exec 200>"$LOCK_FILE"
@@ -79,9 +60,6 @@ while true; do
     if flock -n 200; then
         echo "[election] Acquired flock - transitioning to LEADER"
 
-        # Write leader info before starting services
-        write_leader_info
-
         # exec supervisord - replaces this process
         # When supervisord exits, the flock is released automatically
         echo "[election] Starting supervisord..."

docker/nginx.conf

@@ -123,14 +123,10 @@ http {
         # =========================================================================
         # Service discovery
         # =========================================================================
-        location /services {
+        # meta-discovery v1 neighbours (supersedes /services, which stays
-            proxy_pass http://meta-core-api/services;
+        # while consumers migrate).
-            proxy_http_version 1.1;
+        location /neighbors {
-            proxy_set_header Host $host;
+            proxy_pass http://meta-core-api/neighbors;
-        }
-
-        location /services/ {
-            proxy_pass http://meta-core-api/services/;
             proxy_http_version 1.1;
             proxy_set_header Host $host;
         }

docs/meta-core-architecture.md

@@ -276,9 +276,8 @@ Bootstrap / discovery
   GET  /status
   GET  /leader
   GET  /urls
-  GET  /services         (alias: /api/services)
+  GET  /neighbors        (alias: /api/neighbors)
-  GET  /services/{name}
+  GET  /api/stats
-  GET  /services/cleanup/stats
 
 Metadata (primary surface for other services)
   GET    /meta

go.mod

@@ -18,4 +18,5 @@ require (
 	github.com/cespare/xxhash/v2 v2.2.0 // indirect
 	github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
 	github.com/yuin/gopher-lua v1.1.0 // indirect
+	golang.org/x/sys v0.28.0
 )

go.sum

@@ -34,3 +34,5 @@ golang.org/x/crypto v0.31.0/go.mod h1:kDsLvtWBEx7MV9tJOj9bnXsPbxwJQ6csT/x4KIN4Ss
 golang.org/x/net v0.21.0 h1:AQyQV4dYCvJ7vGmJyKki9+PBdyvhkSd8EIx/qb0AYv4=
 golang.org/x/net v0.21.0/go.mod h1:bIjVDfnllIU7BJ2DNgfnXvpSvtn8VRwhlsaeUTyUS44=
 golang.org/x/sys v0.0.0-20190204203706-41f3e6584952/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
+golang.org/x/sys v0.28.0 h1:Fksou7UEQUWlKvIdsqzJmUmCX3cZuD2+P3XyyzwMhlA=
+golang.org/x/sys v0.28.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=

internal/api/admin_migrate.go

@@ -99,3 +99,33 @@ func (s *Server) handleSweepLiteratureEchoes(w http.ResponseWriter, r *http.Requ
 		"byKey":   res.ByKey,
 	})
 }
+
+// handleMigrateUserCids triggers the one-shot sweep that builds
+// `udl:idx:user:<uid>:cids` for User Data Layer records written before that
+// index existed. See storage.UDLBackfillUserCids.
+//
+// The index is what makes whole-profile export a set read instead of a
+// keyspace scan, and it is maintained by the upsert script from now on — so
+// this only ever matters once, for data already on disk. Until it has run, a
+// profile that predates the index exports as **empty**, which means a
+// cross-instance recovery reports success having restored nothing.
+//
+// Returns {fixed: N} — cids added, not cids examined. Safe to call repeatedly:
+// a second pass adds nothing.
+func (s *Server) handleMigrateUserCids(w http.ResponseWriter, r *http.Request) {
+	if !s.storage.IsConnected() {
+		writeError(w, http.StatusServiceUnavailable, "storage not connected")
+		return
+	}
+
+	fixed, err := s.storage.UDLBackfillUserCids()
+	if err != nil {
+		writeError(w, http.StatusInternalServerError, err.Error())
+		return
+	}
+
+	writeJSON(w, http.StatusOK, map[string]interface{}{
+		"success": true,
+		"fixed":   fixed,
+	})
+}

internal/api/handlers.go

@@ -14,6 +14,7 @@ import (
 
 	"github.com/gorilla/mux"
 	"github.com/metazla/meta-core/internal/leader"
+	"github.com/metazla/meta-core/internal/meshdisco"
 )
 
 // contentTypeByExt maps file extensions to MIME types
@@ -547,62 +548,49 @@ func (s *Server) handleGetFileByCID(w http.ResponseWriter, r *http.Request) {
 	http.ServeContent(w, r, fileInfo.Name(), fileInfo.ModTime(), file)
 }
 
-// handleListServices handles GET /services and GET /api/services
+// handleListNeighbors handles GET /neighbors and GET /api/neighbors.
-// Supports optional ?current=<service-name> query parameter
+//
-func (s *Server) handleListServices(w http.ResponseWriter, r *http.Request) {
+// Beacon v2 (docs/project-architecture/beacon-v2.md). The list is this node's
-	services, err := s.discovery.DiscoverAll()
+// in-memory view of the UDP group, so it works across stacks that share a
-	if err != nil {
+// network but no volume. Instances of one name collapse to the most recent
-		writeError(w, http.StatusInternalServerError, err.Error())
+// (one row per service, the nav menu's shape); `?all=1` returns every
-		return
+// instance, `?cap=<pattern>` keeps only nodes with a matching capability.
-	}
+func (s *Server) handleListNeighbors(w http.ResponseWriter, r *http.Request) {
-
 	response := map[string]interface{}{
-		"services": services,
+		"current":   s.config.ServiceName,
-		"count":    len(services),
+		"neighbors": []interface{}{},
-	}
+		"count":     0,
-
-	// Add 'current' field if query parameter is provided
-	current := r.URL.Query().Get("current")
-	if current != "" {
-		response["current"] = current
 	}
 
+	if s.mesh == nil {
+		// Discovery off or failed to bind. Report empty rather than 503 —
+		// the nav is decoration and callers already treat [] as "hide".
+		response["enabled"] = false
 		writeJSON(w, http.StatusOK, response)
-}
-
-// handleGetService handles GET /services/{name}
-func (s *Server) handleGetService(w http.ResponseWriter, r *http.Request) {
-	vars := mux.Vars(r)
-	name := vars["name"]
-
-	if name == "" {
-		writeError(w, http.StatusBadRequest, "service name is required")
 		return
 	}
 
-	service, err := s.discovery.Discover(name)
+	var list []meshdisco.Neighbor
-	if err != nil {
+	if r.URL.Query().Get("all") != "" {
-		writeError(w, http.StatusInternalServerError, err.Error())
+		list = s.mesh.Neighbors()
-		return
+	} else {
+		list = s.mesh.NeighborsByName()
+	}
+	if pattern := r.URL.Query().Get("cap"); pattern != "" {
+		kept := list[:0]
+		for _, nb := range list {
+			if nb.Matches(pattern) {
+				kept = append(kept, nb)
+			}
+		}
+		list = kept
 	}
 
-	if service == nil {
+	response["enabled"] = true
-		writeError(w, http.StatusNotFound, "service not found")
+	response["neighbors"] = list
-		return
+	response["count"] = len(list)
-	}
+	response["self"] = s.mesh.Self()
-
+	writeJSON(w, http.StatusOK, response)
-	writeJSON(w, http.StatusOK, service)
-}
-
-// handleCleanupStats handles GET /services/cleanup/stats
-func (s *Server) handleCleanupStats(w http.ResponseWriter, r *http.Request) {
-	if s.cleaner == nil {
-		writeError(w, http.StatusServiceUnavailable, "cleaner not initialized")
-		return
-	}
-
-	stats := s.cleaner.Stats()
-	writeJSON(w, http.StatusOK, stats)
 }
 
 // writeJSON writes a JSON response

internal/api/server.go

@@ -9,10 +9,10 @@ import (
 
 	"github.com/gorilla/mux"
 	"github.com/metazla/meta-core/internal/config"
-	"github.com/metazla/meta-core/internal/discovery"
 	"github.com/metazla/meta-core/internal/events"
 	"github.com/metazla/meta-core/internal/identity"
 	"github.com/metazla/meta-core/internal/leader"
+	"github.com/metazla/meta-core/internal/meshdisco"
 	"github.com/metazla/meta-core/internal/mounts"
 	"github.com/metazla/meta-core/internal/schema"
 	"github.com/metazla/meta-core/internal/storage"
@@ -25,8 +25,6 @@ import (
 type Server struct {
 	config            *config.Config
 	leaderProvider    *leader.LeaderInfoProvider
-	discovery         *discovery.Service
-	cleaner           *discovery.Cleaner
 	storage           *storage.Client
 	mountsManager     *mounts.Manager
 	mountsHandlers    *mounts.Handlers
@@ -45,26 +43,37 @@ type Server struct {
 	// challenges backs proof-of-possession on reveal/delete. In-process, so a
 	// restart invalidates the ones in flight — see identity/challenge.go.
 	challenges *identity.ChallengeStore
+	// mesh is the UDP discovery node (beacon v2). nil when discovery
+	// is disabled or failed to bind, in which case /api/neighbors reports an
+	// empty list rather than erroring — a dashboard nav is decoration and
+	// must not take the API down with it.
+	mesh   *meshdisco.Node
 	router *mux.Router
 	server *http.Server
+	// stats caches the keyspace-walk half of GET /api/stats so a polling
+	// dashboard never puts a multi-second Redis walk on the request path.
+	stats *statsCache
+}
+
+// SetMeshNode attaches the UDP discovery node. Separate from NewServer so the
+// server's constructor signature (and its tests) stay untouched.
+func (s *Server) SetMeshNode(n *meshdisco.Node) {
+	s.mesh = n
 }
 
 // NewServer creates a new API server
 func NewServer(
 	cfg *config.Config,
 	leaderProvider *leader.LeaderInfoProvider,
-	disc *discovery.Service,
-	cleaner *discovery.Cleaner,
 	stor *storage.Client,
 ) *Server {
 	s := &Server{
 		config:         cfg,
 		leaderProvider: leaderProvider,
-		discovery:      disc,
-		cleaner:        cleaner,
 		storage:        stor,
 		search:         &searchIndex{},
 		router:         mux.NewRouter(),
+		stats:          &statsCache{},
 	}
 
 	// Initialize file watcher/scanner and dispatcher
@@ -164,6 +173,10 @@ func (s *Server) setupRoutes() {
 	s.router.HandleFunc("/leader", s.handleLeader).Methods("GET")
 	s.router.HandleFunc("/urls", s.handleURLs).Methods("GET")
 
+	// Dashboard headline counters in one request. Cheap half live, keyspace
+	// walks served from a background-refreshed cache — see stats.go.
+	s.router.HandleFunc("/api/stats", s.handleStats).Methods("GET")
+
 	// Metadata Editor API routes (must be before /meta/{hash} routes)
 	s.router.HandleFunc("/api/metadata/hash-ids", s.handleGetHashIds).Methods("GET")
 	s.router.HandleFunc("/api/metadata/list", s.handleListMetadata).Methods("GET")
@@ -216,6 +229,8 @@ func (s *Server) setupRoutes() {
 	s.router.HandleFunc("/api/udl/records", s.handleUDLRecordsPut).Methods("PUT")
 	s.router.HandleFunc("/api/udl/user/{uid}/key/{key}", s.handleUDLUserKey).Methods("GET")
 	s.router.HandleFunc("/api/udl/user/{uid}/cid/{cid}", s.handleUDLUserCid).Methods("GET")
+	s.router.HandleFunc("/api/udl/user/{uid}/all", s.handleUDLUserAll).Methods("GET")
+	s.router.HandleFunc("/api/udl/user/{uid}/count", s.handleUDLUserCount).Methods("GET")
 	s.router.HandleFunc("/api/udl/cid/{cid}/users", s.handleUDLCidUsers).Methods("GET")
 	s.router.HandleFunc("/api/udl/cid/{cid}/aggregate", s.handleUDLAggregate).Methods("GET")
 	s.router.HandleFunc("/api/udl/users/stats", s.handleUDLUserStats).Methods("GET")
@@ -264,6 +279,7 @@ func (s *Server) setupRoutes() {
 	// One-shot vocabulary sweep: domain film|tv → screen, workForm backfill
 	// (METADATA_KEYS.md §14.17). Idempotent.
 	s.router.HandleFunc("/api/admin/migrate-domain-screen", s.handleMigrateDomainScreen).Methods("POST")
+	s.router.HandleFunc("/api/admin/migrate-user-cids", s.handleMigrateUserCids).Methods("POST")
 	// One-shot sweep of query echoes on literature cards (meta-read
 	// reading-model.md §7). Dry-run unless ?apply=true. Idempotent.
 	s.router.HandleFunc("/api/admin/sweep-literature-echoes", s.handleSweepLiteratureEchoes).Methods("POST")
@@ -275,12 +291,10 @@ func (s *Server) setupRoutes() {
 	s.router.HandleFunc("/api/events/files", s.handleEventsFiles).Methods("GET")
 	s.router.HandleFunc("/api/events/meta", s.handleEventsMeta).Methods("GET")
 
-	// Service discovery
+	// beacon v2: neighbours heard over UDP. Supersedes /api/services,
-	// Both /services and /api/services are supported for consistency with other services
+	// which is still served above while consumers migrate.
-	s.router.HandleFunc("/services", s.handleListServices).Methods("GET")
+	s.router.HandleFunc("/neighbors", s.handleListNeighbors).Methods("GET")
-	s.router.HandleFunc("/api/services", s.handleListServices).Methods("GET")
+	s.router.HandleFunc("/api/neighbors", s.handleListNeighbors).Methods("GET")
-	s.router.HandleFunc("/services/cleanup/stats", s.handleCleanupStats).Methods("GET")
-	s.router.HandleFunc("/services/{name}", s.handleGetService).Methods("GET")
 
 	// Mount management routes (if manager initialized)
 	if s.mountsHandlers != nil {

internal/api/stats.go

@@ -0,0 +1,164 @@
+package api
+
+import (
+	"log"
+	"net/http"
+	"sync"
+	"time"
+
+	"github.com/metazla/meta-core/internal/identity"
+)
+
+// statsSweepTTL is how long a keyspace-walk result stays good. The walk is
+// never on the request path (see statsCache), so this only controls how stale
+// the library tiles may look, not how slow the endpoint is.
+const statsSweepTTL = 60 * time.Second
+
+// fileStatsJSON is the file-backed half of the library: the roots that carry
+// filePath+sizeByte+mtimeNano. Gateway / cid-rooted records have no file on
+// disk and are counted in Records, not here — the gap between the two numbers
+// is exactly "metadata we hold for files we do not have".
+type fileStatsJSON struct {
+	Count     int   `json:"count"`
+	TotalSize int64 `json:"totalSize"`
+}
+
+// StatsResponse is the body of GET /api/stats — every headline number the
+// dashboard shows, in one request.
+//
+// Deliberately split by cost. Records/RedisKeys/RedisMemory/Identities are
+// O(1) commands (SCARD, DBSIZE, INFO) plus a directory listing, recomputed on
+// every call. Files and UDLUsers need a keyspace walk — ~0.5s for the file
+// tuples and ~3s for the UDL SCAN on an 88k-record box — so they are served
+// from a background-refreshed cache and are null until the first sweep lands.
+type StatsResponse struct {
+	Records     int64          `json:"records"`
+	RedisKeys   int64          `json:"redisKeys"`
+	RedisMemory string         `json:"redisMemory"`
+	Identities  int            `json:"identities"`
+	Files       *fileStatsJSON `json:"files"`
+	UDLUsers    *int           `json:"udlUsers"`
+	// SweptAt is when the cached half was computed (unix ms), 0 if never.
+	SweptAt int64 `json:"sweptAt"`
+	// Sweeping reports that a refresh is in flight, so the UI can show the
+	// cached numbers as provisional instead of pretending they are live.
+	Sweeping bool `json:"sweeping"`
+}
+
+// statsCache holds the expensive half of /api/stats.
+//
+// Why a cache and not just "compute it": the dashboard polls. A 3s Redis walk
+// per tick, per open tab, is a self-inflicted outage on a box that is also
+// serving playback — the same shape as meta-sort's /api/stats full sweep. So
+// the handler NEVER waits on a walk: it returns whatever the last sweep
+// produced and kicks off a new one if the old one is stale. `running` makes
+// that single-flight, so ten tabs cost one walk.
+type statsCache struct {
+	mu       sync.Mutex
+	files    *fileStatsJSON
+	udlUsers *int
+	sweptAt  time.Time
+	running  bool
+}
+
+// snapshot returns the cached values and whether a refresh is in flight.
+func (c *statsCache) snapshot() (*fileStatsJSON, *int, time.Time, bool) {
+	c.mu.Lock()
+	defer c.mu.Unlock()
+	return c.files, c.udlUsers, c.sweptAt, c.running
+}
+
+// refreshIfStale starts a background sweep unless one is already running or
+// the last result is still fresh. Returns true if it started one.
+func (c *statsCache) refreshIfStale(sweep func() (*fileStatsJSON, *int)) bool {
+	c.mu.Lock()
+	if c.running || time.Since(c.sweptAt) < statsSweepTTL {
+		c.mu.Unlock()
+		return false
+	}
+	c.running = true
+	c.mu.Unlock()
+
+	go func() {
+		files, udl := sweep()
+		c.mu.Lock()
+		defer c.mu.Unlock()
+		c.running = false
+		// A failed sweep leaves the previous numbers in place rather than
+		// blanking the tiles; sweptAt still advances so a hard-failing walk
+		// (Redis down) does not spin once per poll.
+		c.sweptAt = time.Now()
+		if files != nil {
+			c.files = files
+		}
+		if udl != nil {
+			c.udlUsers = udl
+		}
+	}()
+	return true
+}
+
+// handleStats handles GET /api/stats.
+func (s *Server) handleStats(w http.ResponseWriter, r *http.Request) {
+	if !s.storage.IsConnected() {
+		writeError(w, http.StatusServiceUnavailable, "storage not connected")
+		return
+	}
+
+	resp := StatsResponse{}
+
+	if n, err := s.storage.CountRoots(); err != nil {
+		writeError(w, http.StatusInternalServerError, err.Error())
+		return
+	} else {
+		resp.Records = n
+	}
+	if n, err := s.storage.CountKeys(); err == nil {
+		resp.RedisKeys = n
+	}
+	if mem, err := s.storage.GetMemoryInfo(); err == nil {
+		resp.RedisMemory = mem
+	} else {
+		resp.RedisMemory = "N/A"
+	}
+	if list, err := identity.List(s.config.IdentityAccountsDir()); err == nil {
+		resp.Identities = len(list)
+	}
+
+	s.stats.refreshIfStale(s.sweepStats)
+	files, udl, sweptAt, sweeping := s.stats.snapshot()
+	resp.Files = files
+	resp.UDLUsers = udl
+	resp.Sweeping = sweeping
+	if !sweptAt.IsZero() {
+		resp.SweptAt = sweptAt.UnixMilli()
+	}
+
+	writeJSON(w, http.StatusOK, resp)
+}
+
+// sweepStats runs the two keyspace walks behind the cached tiles. Called off
+// the request path only. A failing half returns nil so the other half still
+// lands.
+func (s *Server) sweepStats() (*fileStatsJSON, *int) {
+	var files *fileStatsJSON
+	if tuples, err := s.storage.GetTuplesForAllFiles(); err != nil {
+		log.Printf("[API] stats: file tuple sweep failed: %v", err)
+	} else {
+		fs := fileStatsJSON{Count: len(tuples)}
+		for _, t := range tuples {
+			fs.TotalSize += t.Size
+		}
+		files = &fs
+	}
+
+	var udl *int
+	if stats, err := s.storage.UDLAllUserStats(); err != nil {
+		log.Printf("[API] stats: UDL user sweep failed: %v", err)
+	} else {
+		n := len(stats)
+		udl = &n
+	}
+
+	return files, udl
+}

internal/api/udl.go

@@ -13,6 +13,7 @@ import (
 	"encoding/json"
 	"math"
 	"net/http"
+	"strconv"
 	"strings"
 
 	"github.com/gorilla/mux"
@@ -271,6 +272,75 @@ func (s *Server) handleUDLUserCid(w http.ResponseWriter, r *http.Request) {
 	writeJSON(w, http.StatusOK, map[string]interface{}{"entries": entries})
 }
 
+// handleUDLUserAll handles GET /api/udl/user/{uid}/all?after=<cid>&limit=<n>
+// — every record a uid owns, for cross-instance profile recovery.
+//
+// The other user-scoped reads answer a question with one coordinate already
+// known. Recovery knows only the uid: a box that has just imported a secret key
+// has no idea which cids or keys the profile ever touched, so it needs the
+// whole set or it gets nothing.
+//
+// Paged on a cid boundary — `nextCid` empty means the export is complete. The
+// response is opaque signed records, exactly as stored; whoever consumes it is
+// responsible for verifying each one before trusting it (meta-core has never
+// held the account key and cannot verify on anyone's behalf).
+func (s *Server) handleUDLUserAll(w http.ResponseWriter, r *http.Request) {
+	if !s.storage.IsConnected() {
+		writeError(w, http.StatusServiceUnavailable, "storage not connected")
+		return
+	}
+	uid := mux.Vars(r)["uid"]
+	if uid == "" {
+		writeError(w, http.StatusBadRequest, "uid is required")
+		return
+	}
+	limit := maxUDLBatch
+	if v := r.URL.Query().Get("limit"); v != "" {
+		n, err := strconv.Atoi(v)
+		if err != nil || n <= 0 {
+			writeError(w, http.StatusBadRequest, "limit must be a positive integer")
+			return
+		}
+		if n < limit {
+			limit = n
+		}
+	}
+	entries, nextCid, err := s.storage.UDLAllForUser(uid, r.URL.Query().Get("after"), limit)
+	if err != nil {
+		writeError(w, http.StatusInternalServerError, err.Error())
+		return
+	}
+	writeJSON(w, http.StatusOK, map[string]interface{}{
+		"entries": entries,
+		"nextCid": nextCid,
+	})
+}
+
+// handleUDLUserCount handles GET /api/udl/user/{uid}/count — how many cids a
+// uid holds records for.
+//
+// Exists so a gateway mirror can enforce a per-profile quota without walking
+// anything: 32 random bytes are a valid uid, so an unbounded mirror is a
+// disk-fill DoS that needs no account. A quota check that itself cost a scan
+// would just move the denial of service one layer up.
+func (s *Server) handleUDLUserCount(w http.ResponseWriter, r *http.Request) {
+	if !s.storage.IsConnected() {
+		writeError(w, http.StatusServiceUnavailable, "storage not connected")
+		return
+	}
+	uid := mux.Vars(r)["uid"]
+	if uid == "" {
+		writeError(w, http.StatusBadRequest, "uid is required")
+		return
+	}
+	n, err := s.storage.UDLCountUserCids(uid)
+	if err != nil {
+		writeError(w, http.StatusInternalServerError, err.Error())
+		return
+	}
+	writeJSON(w, http.StatusOK, map[string]interface{}{"cids": n})
+}
+
 // handleUDLCidUsers handles GET /api/udl/cid/{cid}/users (uids_for_cid).
 func (s *Server) handleUDLCidUsers(w http.ResponseWriter, r *http.Request) {
 	if !s.storage.IsConnected() {

internal/cid/cid-rank-vectors.json

@@ -128,6 +128,12 @@
       "rank": 5,
       "note": "codec 0x1009 + IDENTITY multihash 0x00, digest = the verbatim UTF-8 URL. WIRE-IDENTICAL to url (0x1006) apart from the codec, and that difference is the entire contract: 0x1006 means 'fetch these bytes once and seed them', 0x1009 means 'open this in a browser, there are no bytes for us'. A decoder that matched on SHAPE rather than on the codec slot would see an identity multihash wrapping an http(s) URL, take the 0x1006 path, fetch a watch/Bandcamp page, chunk it, seed the HTML and link that content CID onto the record AS THE FILE — the MSR1 poisoning shape, except permanent and replicated. Same tier/reasoning as the rest of the locator family."
     },
+    {
+      "name": "provider-file-locator",
+      "cid": "bagfcaaa2bvxxazloon2we5djorwgk43gnfwgkorxga3dcobtgq",
+      "rank": 5,
+      "note": "codec 0x100A + IDENTITY multihash 0x00, digest = varint(len(source)) || source || id - here (\"opensubtitles\",\"file:7061834\"). The GENERIC fetchable locator: one codec for every provider whose bytes are fetched the same way; the provider is inside the digest. Framing byte-identical to card (0x1007), but the byte path RESOLVES it (via source's fetch tier) instead of 404ing. Same codec-slot rule as the rest of the family: match on the CODEC, never the mh (identity 0x00 would fall through and score 0). Tier 5: once the fetched bytes' sha2-256 is linked onto the record it wins the canonical election with no addressing change."
+    },
     {
       "name": "sha1",
       "cid": "baeircff3eqvxxaqqot2lt242stkdtk3gq4c2vra",

internal/cid/rank.go

@@ -75,6 +75,14 @@ const (
 	CodeYtVideo = 0x1008 // custom — identity-multihash CID wrapping a YouTube (kind, id)
 	CodeExtPlay = 0x1009 // custom — identity-multihash CID wrapping a URL to OPEN, never fetch
 
+	// CodeProviderFile is the FETCHABLE twin of CodeExtPlay: a provider-held
+	// file addressed as varint(len(source))‖source‖id — ("opensubtitles",
+	// "file:7061834"). Framing is byte-identical to CodeCard; only the byte
+	// path differs (it redeems the id through that source's feeder instead of
+	// 404ing). Same codec-slot rule and same tier: once the fetched bytes'
+	// sha2-256 is linked onto the record it wins with no addressing change.
+	CodeProviderFile = 0x100A // custom — identity-multihash CID wrapping a (source, id) provider file
+
 	// CodeNzbPosting is the INVERSE trap of the two above: its multihash is a
 	// REAL sha2-256 digest (over the normalised, sorted article Message-ID
 	// set of a Usenet posting we scanned ourselves — never a locator, no
@@ -183,7 +191,7 @@ func Rank(cidStr string) int {
 	// Opaque locators. Codec-only: their multihash is identity (0x00), so
 	// there is nothing to match on the mh side.
 	if codec == CodeNzbRelease || codec == CodeURL || codec == CodeCard ||
-		codec == CodeYtVideo || codec == CodeExtPlay {
+		codec == CodeYtVideo || codec == CodeExtPlay || codec == CodeProviderFile {
 		return RankLocator
 	}
 	// nzb-posting: codec-only, checked BEFORE the mh switch below — its

internal/config/config.go

@@ -40,6 +40,14 @@ type Config struct {
 
 	// Mount configuration
 	MountsDir string // Path to mounts configuration (default: /meta-core/mounts)
+
+	// UDP discovery (beacon v2). Replaces the file-based registry that used
+	// to live under /meta-core/services; see
+	// docs/project-architecture/beacon-v2.md.
+	EnableUDPDiscovery  bool   // Advertise + listen on the multicast group (default: true)
+	DiscoveryGroup      string // BEACON_GROUP: IPv4 multicast group (default: 239.255.99.1)
+	DiscoveryPort       int    // BEACON_PORT: UDP port (default: 9099)
+	DiscoveryIntervalMS int    // BEACON_INTERVAL_MS: unsolicited advertise interval (default: 10000)
 }
 
 // Load creates a Config from environment variables
@@ -63,6 +71,10 @@ func Load() *Config {
 		WatchIntervalMS:        getEnvInt("WATCH_INTERVAL_MS", 1000),
 		DebounceMS:             getEnvInt("DEBOUNCE_MS", 30000),
 		EnableFileWatcher:      getEnvBool("ENABLE_FILE_WATCHER", true),
+		EnableUDPDiscovery:     getEnvBool("ENABLE_UDP_DISCOVERY", true),
+		DiscoveryGroup:         getEnv("BEACON_GROUP", "239.255.99.1"),
+		DiscoveryPort:          getEnvInt("BEACON_PORT", 9099),
+		DiscoveryIntervalMS:    getEnvInt("BEACON_INTERVAL_MS", 10000),
 	}
 
 	// Set mounts directory
@@ -71,26 +83,18 @@ func Load() *Config {
 	return cfg
 }
 
-// LockFilePath returns the path to the leader lock file
+// LockFilePath returns the path to the Redis-volume mutex. Held by
+// docker/leader-election.sh; nothing reads it, and it is NOT service
+// discovery — see docs/project-architecture/beacon-v2.md.
 func (c *Config) LockFilePath() string {
 	return c.MetaCorePath + "/locks/kv-leader.lock"
 }
 
-// InfoFilePath returns the path to the leader info file
-func (c *Config) InfoFilePath() string {
-	return c.MetaCorePath + "/locks/kv-leader.info"
-}
-
 // RedisDataDir returns the path to Redis data directory
 func (c *Config) RedisDataDir() string {
 	return c.MetaCorePath + "/db/redis"
 }
 
-// ServicesDir returns the path to services directory
-func (c *Config) ServicesDir() string {
-	return c.MetaCorePath + "/services"
-}
-
 // MountsFilePath returns the path to the mounts configuration file
 func (c *Config) MountsFilePath() string {
 	return c.MountsDir + "/mounts.json"

internal/discovery/cleaner.go

@@ -1,196 +0,0 @@
-package discovery
-
-import (
-	"encoding/json"
-	"log"
-	"os"
-	"path/filepath"
-	"sync"
-	"time"
-
-	"github.com/metazla/meta-core/internal/config"
-)
-
-// CleanupStats holds statistics about the cleanup process
-type CleanupStats struct {
-	LastRunTime     string `json:"lastRunTime,omitempty"`
-	ServicesRemoved int    `json:"servicesRemoved"`
-	TotalRuns       int    `json:"totalRuns"`
-	Errors          int    `json:"errors"`
-}
-
-// Cleaner handles periodic cleanup of dead service registration files
-type Cleaner struct {
-	config      *config.Config
-	servicesDir string
-	ownFile     string // Our own service file (never delete)
-
-	stats    CleanupStats
-	stopChan chan struct{}
-	wg       sync.WaitGroup
-	mu       sync.RWMutex
-}
-
-// NewCleaner creates a new service cleaner
-func NewCleaner(cfg *config.Config) *Cleaner {
-	hostname, _ := os.Hostname()
-	return &Cleaner{
-		config:      cfg,
-		servicesDir: cfg.ServicesDir(),
-		ownFile:     filepath.Join(cfg.ServicesDir(), cfg.ServiceName+"-"+hostname+".json"),
-		stopChan:    make(chan struct{}),
-	}
-}
-
-// Start begins the cleanup loop
-func (c *Cleaner) Start() error {
-	log.Printf("[Cleaner] Starting dead service cleanup (interval: %dms, threshold: %dms)",
-		c.config.CleanupIntervalMS, c.config.DeadServiceThresholdMS)
-
-	c.wg.Add(1)
-	go c.cleanupLoop()
-
-	return nil
-}
-
-// Stop stops the cleanup loop
-func (c *Cleaner) Stop() error {
-	log.Println("[Cleaner] Stopping dead service cleanup...")
-	close(c.stopChan)
-	c.wg.Wait()
-	return nil
-}
-
-// cleanupLoop runs the periodic cleanup
-func (c *Cleaner) cleanupLoop() {
-	defer c.wg.Done()
-
-	// Run an initial cleanup shortly after starting
-	initialDelay := time.NewTimer(30 * time.Second)
-	select {
-	case <-c.stopChan:
-		initialDelay.Stop()
-		return
-	case <-initialDelay.C:
-		c.runCleanup()
-	}
-
-	ticker := time.NewTicker(time.Duration(c.config.CleanupIntervalMS) * time.Millisecond)
-	defer ticker.Stop()
-
-	for {
-		select {
-		case <-c.stopChan:
-			return
-		case <-ticker.C:
-			c.runCleanup()
-		}
-	}
-}
-
-// runCleanup scans the services directory and removes dead services
-func (c *Cleaner) runCleanup() {
-	c.mu.Lock()
-	c.stats.TotalRuns++
-	c.stats.LastRunTime = time.Now().UTC().Format(time.RFC3339)
-	c.mu.Unlock()
-
-	entries, err := os.ReadDir(c.servicesDir)
-	if err != nil {
-		if !os.IsNotExist(err) {
-			log.Printf("[Cleaner] Failed to read services directory: %v", err)
-			c.mu.Lock()
-			c.stats.Errors++
-			c.mu.Unlock()
-		}
-		return
-	}
-
-	threshold := time.Duration(c.config.DeadServiceThresholdMS) * time.Millisecond
-	removedCount := 0
-
-	for _, entry := range entries {
-		if entry.IsDir() || filepath.Ext(entry.Name()) != ".json" {
-			continue
-		}
-
-		filePath := filepath.Join(c.servicesDir, entry.Name())
-
-		// Never delete our own service file
-		if filePath == c.ownFile {
-			continue
-		}
-
-		// Check if service is dead
-		isDead, serviceName, age := c.isServiceDead(filePath, threshold)
-		if isDead {
-			if err := os.Remove(filePath); err != nil {
-				log.Printf("[Cleaner] Failed to remove dead service file %s: %v", entry.Name(), err)
-				c.mu.Lock()
-				c.stats.Errors++
-				c.mu.Unlock()
-			} else {
-				ageMinutes := age.Minutes()
-				log.Printf("[Cleaner] Removed dead service: %s (last heartbeat: %.1f minutes ago)", serviceName, ageMinutes)
-				removedCount++
-			}
-		}
-	}
-
-	if removedCount > 0 {
-		c.mu.Lock()
-		c.stats.ServicesRemoved += removedCount
-		c.mu.Unlock()
-		log.Printf("[Cleaner] Cleanup complete: removed %d dead service(s)", removedCount)
-	}
-}
-
-// isServiceDead checks if a service file represents a dead service
-// Returns: isDead, serviceName, age since last heartbeat
-func (c *Cleaner) isServiceDead(filePath string, threshold time.Duration) (bool, string, time.Duration) {
-	data, err := os.ReadFile(filePath)
-	if err != nil {
-		// Can't read file - treat as dead to clean up corrupted files
-		log.Printf("[Cleaner] Cannot read service file %s: %v (treating as dead)", filepath.Base(filePath), err)
-		return true, filepath.Base(filePath), threshold
-	}
-
-	var info ServiceInfo
-	if err := json.Unmarshal(data, &info); err != nil {
-		// Corrupted JSON - skip rather than delete (might be recoverable)
-		log.Printf("[Cleaner] Skipping corrupted service file %s: %v", filepath.Base(filePath), err)
-		return false, "", 0
-	}
-
-	// Parse last heartbeat timestamp
-	if info.LastHeartbeat == "" {
-		// No heartbeat timestamp - treat as dead
-		return true, info.Name, threshold
-	}
-
-	lastHeartbeat, err := time.Parse(time.RFC3339, info.LastHeartbeat)
-	if err != nil {
-		// Invalid timestamp - treat as dead
-		log.Printf("[Cleaner] Invalid timestamp in service file %s: %v (treating as dead)", filepath.Base(filePath), err)
-		return true, info.Name, threshold
-	}
-
-	age := time.Since(lastHeartbeat)
-	if age > threshold {
-		return true, info.Name, age
-	}
-
-	return false, info.Name, age
-}
-
-// Stats returns a copy of the current cleanup statistics
-func (c *Cleaner) Stats() CleanupStats {
-	c.mu.RLock()
-	defer c.mu.RUnlock()
-	return CleanupStats{
-		LastRunTime:     c.stats.LastRunTime,
-		ServicesRemoved: c.stats.ServicesRemoved,
-		TotalRuns:       c.stats.TotalRuns,
-		Errors:          c.stats.Errors,
-	}
-}

internal/discovery/service.go

@@ -1,286 +0,0 @@
-package discovery
-
-import (
-	"encoding/json"
-	"fmt"
-	"log"
-	"net"
-	"os"
-	"path/filepath"
-	"sync"
-	"time"
-
-	"github.com/metazla/meta-core/internal/config"
-)
-
-// ServiceInfo contains simplified service registration data
-// URLs are obtained via the /urls API endpoint
-type ServiceInfo struct {
-	Name          string `json:"name"`
-	Hostname      string `json:"hostname"`
-	BaseUrl       string `json:"baseUrl"`
-	Status        string `json:"status"`
-	LastHeartbeat string `json:"lastHeartbeat"`
-}
-
-// Service handles service registration and discovery
-type Service struct {
-	config      *config.Config
-	servicesDir string
-	serviceFile string
-	info        *ServiceInfo
-	mu          sync.RWMutex
-
-	stopChan chan struct{}
-	wg       sync.WaitGroup
-}
-
-// NewService creates a new service discovery instance
-func NewService(cfg *config.Config) *Service {
-	hostname, _ := os.Hostname()
-	return &Service{
-		config:      cfg,
-		servicesDir: cfg.ServicesDir(),
-		serviceFile: filepath.Join(cfg.ServicesDir(), cfg.ServiceName+"-"+hostname+".json"),
-		stopChan:    make(chan struct{}),
-	}
-}
-
-// Start begins service registration and heartbeat
-func (s *Service) Start() error {
-	log.Printf("[Discovery] Starting service discovery for %s", s.config.ServiceName)
-
-	// Ensure services directory exists
-	if err := os.MkdirAll(s.servicesDir, 0755); err != nil {
-		return fmt.Errorf("failed to create services directory: %w", err)
-	}
-
-	// Build and register service info
-	s.info = s.buildServiceInfo()
-	if err := s.register(); err != nil {
-		return fmt.Errorf("failed to register service: %w", err)
-	}
-
-	// Start heartbeat loop
-	s.wg.Add(1)
-	go s.heartbeatLoop()
-
-	return nil
-}
-
-// Stop stops service discovery and unregisters
-func (s *Service) Stop() error {
-	log.Println("[Discovery] Stopping service discovery...")
-	close(s.stopChan)
-	s.wg.Wait()
-
-	// Unregister by removing service file
-	if err := os.Remove(s.serviceFile); err != nil && !os.IsNotExist(err) {
-		log.Printf("[Discovery] Failed to remove service file: %v", err)
-	}
-
-	return nil
-}
-
-// buildServiceInfo creates the service info for this instance
-func (s *Service) buildServiceInfo() *ServiceInfo {
-	hostname, _ := os.Hostname()
-	ip := getLocalIP()
-
-	// Browser-facing nav URL: prefer META_CORE_PUBLIC_URL (the reachable URL
-	// when there's no Caddy perimeter in front — e.g. a debug-direct port),
-	// then BASE_URL (the Caddy perimeter URL), then the internal IP.
-	var baseUrl string
-	if s.config.PublicURL != "" {
-		baseUrl = s.config.PublicURL
-	} else if s.config.BaseURL != "" {
-		baseUrl = s.config.BaseURL
-	} else {
-		baseUrl = fmt.Sprintf("http://%s:%d", ip, s.config.APIPort)
-	}
-
-	info := &ServiceInfo{
-		Name:          s.config.ServiceName,
-		Hostname:      hostname,
-		BaseUrl:       baseUrl,
-		Status:        "running",
-		LastHeartbeat: time.Now().UTC().Format(time.RFC3339),
-	}
-
-	return info
-}
-
-// register writes service info to file
-func (s *Service) register() error {
-	s.mu.Lock()
-	defer s.mu.Unlock()
-
-	return s.writeServiceInfo(s.info)
-}
-
-// writeServiceInfo atomically writes service info to file
-func (s *Service) writeServiceInfo(info *ServiceInfo) error {
-	tempPath := s.serviceFile + ".tmp"
-
-	data, err := json.MarshalIndent(info, "", "  ")
-	if err != nil {
-		return err
-	}
-
-	if err := os.WriteFile(tempPath, data, 0644); err != nil {
-		return err
-	}
-
-	return os.Rename(tempPath, s.serviceFile)
-}
-
-// heartbeatLoop periodically updates the last heartbeat
-func (s *Service) heartbeatLoop() {
-	defer s.wg.Done()
-
-	ticker := time.NewTicker(time.Duration(s.config.HeartbeatIntervalMS) * time.Millisecond)
-	defer ticker.Stop()
-
-	for {
-		select {
-		case <-s.stopChan:
-			return
-		case <-ticker.C:
-			s.heartbeat()
-		}
-	}
-}
-
-// heartbeat updates the last heartbeat timestamp
-func (s *Service) heartbeat() {
-	s.mu.Lock()
-	defer s.mu.Unlock()
-
-	if s.info == nil {
-		return
-	}
-
-	s.info.LastHeartbeat = time.Now().UTC().Format(time.RFC3339)
-
-	if err := s.writeServiceInfo(s.info); err != nil {
-		log.Printf("[Discovery] Failed to update heartbeat: %v", err)
-	}
-}
-
-// UpdateStatus updates the service status
-func (s *Service) UpdateStatus(status string) error {
-	s.mu.Lock()
-	defer s.mu.Unlock()
-
-	if s.info == nil {
-		return fmt.Errorf("service not registered")
-	}
-
-	s.info.Status = status
-	s.info.LastHeartbeat = time.Now().UTC().Format(time.RFC3339)
-	return s.writeServiceInfo(s.info)
-}
-
-// Discover finds a service by name
-// Looks for files matching pattern: name-*.json (hostname-based naming)
-func (s *Service) Discover(name string) (*ServiceInfo, error) {
-	// First try exact match for backward compatibility
-	exactPath := filepath.Join(s.servicesDir, name+".json")
-	if data, err := os.ReadFile(exactPath); err == nil {
-		var info ServiceInfo
-		if err := json.Unmarshal(data, &info); err == nil {
-			return s.checkStale(&info), nil
-		}
-	}
-
-	// Search for hostname-based files: name-*.json
-	pattern := filepath.Join(s.servicesDir, name+"-*.json")
-	matches, err := filepath.Glob(pattern)
-	if err != nil {
-		return nil, err
-	}
-
-	// Return the first valid service found
-	for _, match := range matches {
-		data, err := os.ReadFile(match)
-		if err != nil {
-			continue
-		}
-
-		var info ServiceInfo
-		if err := json.Unmarshal(data, &info); err != nil {
-			continue
-		}
-
-		return s.checkStale(&info), nil
-	}
-
-	return nil, nil
-}
-
-// checkStale marks a service as stale if heartbeat is too old
-func (s *Service) checkStale(info *ServiceInfo) *ServiceInfo {
-	lastHeartbeat, err := time.Parse(time.RFC3339, info.LastHeartbeat)
-	if err == nil {
-		staleThreshold := time.Duration(s.config.StaleThresholdMS) * time.Millisecond
-		if time.Since(lastHeartbeat) > staleThreshold {
-			info.Status = "stale"
-		}
-	}
-	return info
-}
-
-// DiscoverAll finds all registered services
-func (s *Service) DiscoverAll() ([]*ServiceInfo, error) {
-	entries, err := os.ReadDir(s.servicesDir)
-	if err != nil {
-		if os.IsNotExist(err) {
-			return []*ServiceInfo{}, nil
-		}
-		return nil, err
-	}
-
-	var services []*ServiceInfo
-	for _, entry := range entries {
-		if entry.IsDir() || filepath.Ext(entry.Name()) != ".json" {
-			continue
-		}
-
-		filePath := filepath.Join(s.servicesDir, entry.Name())
-		data, err := os.ReadFile(filePath)
-		if err != nil {
-			log.Printf("[Discovery] Failed to read service file %s: %v", entry.Name(), err)
-			continue
-		}
-
-		var info ServiceInfo
-		if err := json.Unmarshal(data, &info); err != nil {
-			log.Printf("[Discovery] Failed to parse service file %s: %v", entry.Name(), err)
-			continue
-		}
-
-		services = append(services, s.checkStale(&info))
-	}
-
-	return services, nil
-}
-
-// getLocalIP returns the local IP address
-func getLocalIP() string {
-	addrs, err := net.InterfaceAddrs()
-	if err != nil {
-		hostname, _ := os.Hostname()
-		return hostname
-	}
-
-	for _, addr := range addrs {
-		if ipnet, ok := addr.(*net.IPNet); ok && !ipnet.IP.IsLoopback() {
-			if ipnet.IP.To4() != nil {
-				return ipnet.IP.String()
-			}
-		}
-	}
-
-	hostname, _ := os.Hostname()
-	return hostname
-}

internal/meshdisco/node.go

@@ -0,0 +1,439 @@
+package meshdisco
+
+import (
+	"context"
+	"encoding/json"
+	"fmt"
+	"log"
+	"net"
+	"os"
+	"sort"
+	"sync"
+	"syscall"
+	"time"
+
+	"golang.org/x/net/ipv4"
+	"golang.org/x/sys/unix"
+)
+
+// Config configures a Node. Payload is a function rather than a value so the
+// advertise is rebuilt from live state on every tick — that is what keeps the
+// advertised core URLs and GET /urls from drifting apart.
+type Config struct {
+	Name     string
+	Instance string
+	Version  string
+
+	Group    string
+	Port     int
+	Interval time.Duration
+
+	// Payload returns this node's status ("starting" / "running") and its
+	// resources. The envelope and NodeInfo are filled in by the Node.
+	Payload func() (status string, resources []Resource)
+}
+
+func (c *Config) applyDefaults() {
+	if c.Group == "" {
+		c.Group = DefaultGroup
+	}
+	if c.Port == 0 {
+		c.Port = DefaultPort
+	}
+	if c.Interval <= 0 {
+		c.Interval = DefaultInterval
+	}
+}
+
+type seen struct {
+	node      NodeInfo
+	resources []Resource
+	addr      string
+	lastSeen  time.Time
+}
+
+// Node is one beacon v2 participant: it advertises its resources, answers
+// probes, and keeps a TTL map of every other node it has heard from.
+type Node struct {
+	cfg   Config
+	group net.IP
+
+	pc     *ipv4.PacketConn
+	closer interface{ Close() error }
+
+	// ifaces are the multicast-capable interfaces we joined and send on.
+	// A container attached to two docker networks (metashare-app on `pcs`
+	// and `metamesh-mesh`, metawatch-* on `metawatch` and `metamesh-mesh`)
+	// has two entries here, and MUST be advertised on both — sending via the
+	// default route alone silently covers only one network.
+	ifaces []net.Interface
+
+	// sendMu serialises SetMulticastInterface + WriteTo, which are a
+	// two-step operation on one shared socket.
+	sendMu sync.Mutex
+
+	mu    sync.RWMutex
+	nodes map[string]*seen
+
+	stopChan chan struct{}
+	stopOnce sync.Once
+	wg       sync.WaitGroup
+}
+
+// New creates a Node. It does not touch the network until Start.
+func New(cfg Config) *Node {
+	cfg.applyDefaults()
+	return &Node{
+		cfg:      cfg,
+		nodes:    make(map[string]*seen),
+		stopChan: make(chan struct{}),
+	}
+}
+
+// Start opens the multicast socket, joins the group on every eligible
+// interface, and begins advertising. A join failure on one interface is logged
+// and skipped rather than fatal — a host that blocks multicast degrades
+// instead of failing to boot.
+func (n *Node) Start() error {
+	n.group = net.ParseIP(n.cfg.Group)
+	if n.group == nil || n.group.To4() == nil {
+		return fmt.Errorf("beacon: invalid multicast group %q", n.cfg.Group)
+	}
+
+	lc := net.ListenConfig{Control: reusePort}
+	conn, err := lc.ListenPacket(context.Background(), "udp4", fmt.Sprintf(":%d", n.cfg.Port))
+	if err != nil {
+		return fmt.Errorf("beacon: listen on :%d: %w", n.cfg.Port, err)
+	}
+	n.closer = conn
+
+	pc := ipv4.NewPacketConn(conn)
+	// TTL 1: link-local only. Discovery never routes off the local segment;
+	// the two-host topology keeps two separate meshes by design.
+	if err := pc.SetMulticastTTL(1); err != nil {
+		log.Printf("[beacon] Warning: could not set multicast TTL: %v", err)
+	}
+	n.pc = pc
+
+	n.ifaces = eligibleInterfaces()
+	if len(n.ifaces) == 0 {
+		log.Println("[beacon] Warning: no multicast-capable interfaces found; discovery will be inert")
+	}
+
+	group := &net.UDPAddr{IP: n.group}
+	joined := 0
+	for i := range n.ifaces {
+		if err := pc.JoinGroup(&n.ifaces[i], group); err != nil {
+			log.Printf("[beacon] Warning: join %s on %s failed: %v", n.cfg.Group, n.ifaces[i].Name, err)
+			continue
+		}
+		joined++
+	}
+	log.Printf("[beacon] Listening on %s:%d as %s/%s (joined %d of %d interfaces)",
+		n.cfg.Group, n.cfg.Port, n.cfg.Name, n.cfg.Instance, joined, len(n.ifaces))
+
+	n.wg.Add(2)
+	go n.readLoop()
+	go n.advertiseLoop()
+
+	return nil
+}
+
+// Stop multicasts a bye — so neighbours drop us immediately instead of
+// waiting out the staleness window — then closes the socket and waits for the
+// loops to exit.
+func (n *Node) Stop() error {
+	var err error
+	n.stopOnce.Do(func() {
+		n.broadcast(Message{Proto: Proto, V: Version, Type: TypeBye, Node: n.nodeInfo("")})
+		close(n.stopChan)
+		if n.closer != nil {
+			err = n.closer.Close()
+		}
+		n.wg.Wait()
+	})
+	return err
+}
+
+// Probe multicasts a probe on every interface. Every node owning a resource
+// matching one of want (every node, when want is empty) replies immediately,
+// so a caller does not have to wait out an interval.
+func (n *Node) Probe(want ...string) {
+	n.broadcast(Message{Proto: Proto, V: Version, Type: TypeProbe, From: n.cfg.Instance, Want: want})
+}
+
+// Neighbors returns every node heard from within the staleness window,
+// sorted by (name, instance). Expired entries are dropped as a side effect —
+// expiry is a read-time concern, so there is no reaper goroutine.
+func (n *Node) Neighbors() []Neighbor {
+	cutoff := time.Now().Add(-n.cfg.Interval * LivenessFactor)
+
+	n.mu.Lock()
+	out := make([]Neighbor, 0, len(n.nodes))
+	for k, s := range n.nodes {
+		if s.lastSeen.Before(cutoff) {
+			delete(n.nodes, k)
+			continue
+		}
+		out = append(out, newNeighbor(s.node, s.resources, s.addr, s.lastSeen))
+	}
+	n.mu.Unlock()
+
+	sort.Slice(out, func(i, j int) bool {
+		if out[i].Name != out[j].Name {
+			return out[i].Name < out[j].Name
+		}
+		return out[i].Instance < out[j].Instance
+	})
+	return out
+}
+
+// NeighborsByName collapses multiple instances of one name down to the most
+// recently seen — the shape the nav menu wants.
+func (n *Node) NeighborsByName() []Neighbor {
+	best := make(map[string]Neighbor)
+	for _, nb := range n.Neighbors() {
+		if cur, ok := best[nb.Name]; !ok || nb.LastSeen > cur.LastSeen {
+			best[nb.Name] = nb
+		}
+	}
+	out := make([]Neighbor, 0, len(best))
+	for _, nb := range best {
+		out = append(out, nb)
+	}
+	sort.Slice(out, func(i, j int) bool { return out[i].Name < out[j].Name })
+	return out
+}
+
+// Self returns this node as a row, so callers can render themselves without
+// waiting to hear their own multicast echo back.
+func (n *Node) Self() Neighbor {
+	status, resources := n.payload()
+	return newNeighbor(*n.nodeInfo(status), resources, "", time.Now())
+}
+
+func (n *Node) payload() (string, []Resource) {
+	if n.cfg.Payload == nil {
+		return "running", nil
+	}
+	return n.cfg.Payload()
+}
+
+func (n *Node) nodeInfo(status string) *NodeInfo {
+	return &NodeInfo{Name: n.cfg.Name, Instance: n.cfg.Instance, Version: n.cfg.Version, Status: status}
+}
+
+func (n *Node) advertiseMsg() Message {
+	status, resources := n.payload()
+	return Message{Proto: Proto, V: Version, Type: TypeAdvertise, Node: n.nodeInfo(status), Resources: resources}
+}
+
+func (n *Node) readLoop() {
+	defer n.wg.Done()
+	buf := make([]byte, ReadBufferSize)
+
+	for {
+		select {
+		case <-n.stopChan:
+			return
+		default:
+		}
+
+		nRead, _, src, err := n.pc.ReadFrom(buf)
+		if err != nil {
+			select {
+			case <-n.stopChan:
+				return
+			default:
+			}
+			log.Printf("[beacon] Read error: %v", err)
+			continue
+		}
+
+		msg, ok := Parse(buf[:nRead])
+		if !ok {
+			continue // v1 or foreign traffic on the shared group
+		}
+
+		switch msg.Type {
+		case TypeProbe:
+			if msg.From == n.cfg.Instance {
+				continue
+			}
+			ad := n.advertiseMsg()
+			wanted := len(msg.Want) == 0
+			for _, r := range ad.Resources {
+				if r.MatchesAny(msg.Want) {
+					wanted = true
+					break
+				}
+			}
+			if wanted {
+				// Unicast straight back to the prober's source address; its
+				// ephemeral source port is where it is listening.
+				n.send(ad, src)
+			}
+		case TypeAdvertise, TypeBye:
+			// Drop our own multicast echo. Loopback stays enabled so a
+			// single-container test still sees traffic; we filter by identity.
+			if msg.Node.Instance == n.cfg.Instance {
+				continue
+			}
+			n.record(msg, src)
+		}
+	}
+}
+
+func (n *Node) record(msg Message, src net.Addr) {
+	addr := ""
+	if src != nil {
+		if udp, ok := src.(*net.UDPAddr); ok {
+			addr = udp.IP.String()
+		} else {
+			addr = src.String()
+		}
+	}
+
+	n.mu.Lock()
+	defer n.mu.Unlock()
+	if msg.Type == TypeBye {
+		delete(n.nodes, msg.Node.Instance)
+		return
+	}
+	n.nodes[msg.Node.Instance] = &seen{node: *msg.Node, resources: msg.Resources, addr: addr, lastSeen: time.Now()}
+}
+
+func (n *Node) advertiseLoop() {
+	defer n.wg.Done()
+
+	// Advertise immediately, then probe once so the map fills without
+	// waiting a full interval for everyone else's ticker.
+	n.broadcast(n.advertiseMsg())
+	n.Probe()
+
+	ticker := time.NewTicker(n.cfg.Interval)
+	defer ticker.Stop()
+
+	for {
+		select {
+		case <-n.stopChan:
+			return
+		case <-ticker.C:
+			n.broadcast(n.advertiseMsg())
+		}
+	}
+}
+
+func encode(msg Message) ([]byte, bool) {
+	body, err := json.Marshal(msg)
+	if err != nil {
+		return nil, false
+	}
+	if len(body) > MaxDatagram {
+		log.Printf("[beacon] Warning: %d-byte datagram exceeds the fragmentation-safe %d", len(body), MaxDatagram)
+	}
+	return body, true
+}
+
+// send unicasts msg to dst (a probe reply).
+func (n *Node) send(msg Message, dst net.Addr) {
+	body, ok := encode(msg)
+	if !ok {
+		return
+	}
+	n.sendMu.Lock()
+	defer n.sendMu.Unlock()
+	if _, err := n.pc.WriteTo(body, nil, dst); err != nil {
+		log.Printf("[beacon] Reply to %s failed: %v", dst, err)
+	}
+}
+
+// broadcast writes one message to the group once per eligible interface.
+// Writing once and letting the routing table choose would reach exactly one
+// network, which is the failure mode this protocol exists to avoid.
+func (n *Node) broadcast(msg Message) {
+	if n.pc == nil {
+		return
+	}
+	body, ok := encode(msg)
+	if !ok {
+		return
+	}
+	dst := &net.UDPAddr{IP: n.group, Port: n.cfg.Port}
+
+	n.sendMu.Lock()
+	defer n.sendMu.Unlock()
+	for i := range n.ifaces {
+		if err := n.pc.SetMulticastInterface(&n.ifaces[i]); err != nil {
+			log.Printf("[beacon] Warning: select interface %s failed: %v", n.ifaces[i].Name, err)
+			continue
+		}
+		if _, err := n.pc.WriteTo(body, nil, dst); err != nil {
+			log.Printf("[beacon] Warning: send on %s failed: %v", n.ifaces[i].Name, err)
+		}
+	}
+}
+
+// eligibleInterfaces returns every up, non-loopback, multicast-capable
+// interface that has an IPv4 address.
+func eligibleInterfaces() []net.Interface {
+	all, err := net.Interfaces()
+	if err != nil {
+		log.Printf("[beacon] Warning: could not enumerate interfaces: %v", err)
+		return nil
+	}
+
+	var out []net.Interface
+	for _, ifi := range all {
+		if ifi.Flags&net.FlagUp == 0 ||
+			ifi.Flags&net.FlagLoopback != 0 ||
+			ifi.Flags&net.FlagMulticast == 0 {
+			continue
+		}
+		addrs, err := ifi.Addrs()
+		if err != nil {
+			continue
+		}
+		for _, a := range addrs {
+			if ipnet, ok := a.(*net.IPNet); ok && ipnet.IP.To4() != nil {
+				out = append(out, ifi)
+				break
+			}
+		}
+	}
+	return out
+}
+
+// reusePort sets SO_REUSEADDR + SO_REUSEPORT so several listeners (a test, a
+// sidecar, a beacon v1 responder in the same netns) can bind 9099 on one host.
+func reusePort(_, _ string, c syscall.RawConn) error {
+	var serr error
+	err := c.Control(func(fd uintptr) {
+		serr = unix.SetsockoptInt(int(fd), unix.SOL_SOCKET, unix.SO_REUSEADDR, 1)
+		if serr == nil {
+			serr = unix.SetsockoptInt(int(fd), unix.SOL_SOCKET, unix.SO_REUSEPORT, 1)
+		}
+	})
+	if err != nil {
+		return err
+	}
+	return serr
+}
+
+// LocalIPv4 returns this host's first non-loopback IPv4 address, or the
+// hostname when none can be determined. Same fallback the file-based registry
+// used (discovery/service.go), kept identical so announced URLs don't shift.
+func LocalIPv4() string {
+	addrs, err := net.InterfaceAddrs()
+	if err == nil {
+		for _, addr := range addrs {
+			if ipnet, ok := addr.(*net.IPNet); ok && !ipnet.IP.IsLoopback() {
+				if ipnet.IP.To4() != nil {
+					return ipnet.IP.String()
+				}
+			}
+		}
+	}
+	host, _ := os.Hostname()
+	return host
+}

internal/meshdisco/proto.go

@@ -0,0 +1,237 @@
+// Package meshdisco implements beacon v2: local resource advertise / discover
+// over UDP multicast (239.255.99.1:9099), the protocol every meta-* service
+// and SDK-hosted plugin uses to find its neighbours.
+//
+// A node advertises Resources, each tagged with capability strings
+// ("metamesh.core", "metamesh.service/meta-sort", "metamesh.transport/nzb@1");
+// consumers keep a live view of everyone else's and filter by capability.
+// meta-core advertises "metamesh.core" carrying its /urls block, which is how
+// services locate it without a shared volume.
+//
+// The wire protocol is documented in docs/project-architecture/beacon-v2.md.
+// Keep this package and that doc in step — the Rust (meta-feeder-sdk),
+// TypeScript and Python implementations are written against the doc.
+package meshdisco
+
+import (
+	"encoding/json"
+	"strings"
+	"time"
+)
+
+const (
+	// Proto and Version form the envelope every v2 message carries. Anything
+	// else on the group (beacon v1, junk) is dropped silently.
+	Proto   = "beacon"
+	Version = 2
+
+	// DefaultGroup / DefaultPort are beacon's endpoint, on purpose: one
+	// well-known place for every kind of local resource. v1 and v2 coexist
+	// because v2 never sends v1's "discovery" / "announce" types.
+	DefaultGroup = "239.255.99.1"
+	DefaultPort  = 9099
+
+	// DefaultInterval is how often an unsolicited advertise is emitted.
+	DefaultInterval = 10 * time.Second
+
+	// LivenessFactor multiplies the interval to get the staleness window.
+	// Expiry is evaluated at read time; there is no reaper.
+	LivenessFactor = 3
+
+	// ReadBufferSize is the receive buffer; MaxDatagram is what a sender
+	// stays under so nothing fragments.
+	ReadBufferSize = 8192
+	MaxDatagram    = 1400
+
+	TypeProbe     = "probe"
+	TypeAdvertise = "advertise"
+	TypeBye       = "bye"
+
+	// CapCore is meta-core's capability; its resource carries Data.urls.
+	CapCore = "metamesh.core"
+	// CapServicePrefix + name is every service's capability.
+	CapServicePrefix = "metamesh.service/"
+	CapAnyService    = "metamesh.service/*"
+)
+
+// URLs mirrors meta-core's GET /urls response (api.URLsResponse) minus
+// redisUrl. It travels as the metamesh.core resource's Data.urls, so a service
+// locates meta-core from the advertise alone.
+type URLs struct {
+	Hostname          string `json:"hostname"`
+	BaseUrl           string `json:"baseUrl"`
+	ApiUrl            string `json:"apiUrl"`
+	WebdavUrl         string `json:"webdavUrl"`
+	WebdavUrlInternal string `json:"webdavUrlInternal"`
+}
+
+// NodeInfo says who is advertising.
+type NodeInfo struct {
+	Name     string `json:"name"`
+	Instance string `json:"instance"`
+	Version  string `json:"version,omitempty"`
+	// Status is "starting" | "running"; absent reads as running.
+	Status string `json:"status,omitempty"`
+}
+
+// Resource is one advertised thing.
+type Resource struct {
+	ID   string   `json:"id"`
+	Caps []string `json:"caps"`
+	// Endpoints: well-known names http, ui, manifest, mcp. A value starting
+	// with "/" is relative to endpoints.http.
+	Endpoints map[string]string `json:"endpoints,omitempty"`
+	Rev       string            `json:"rev,omitempty"`
+	// Binds names the one consumer instance this resource belongs to.
+	Binds string          `json:"binds,omitempty"`
+	Data  json.RawMessage `json:"data,omitempty"`
+}
+
+// Matches reports whether any cap matches pattern.
+func (r Resource) Matches(pattern string) bool {
+	for _, c := range r.Caps {
+		if CapMatches(pattern, c) {
+			return true
+		}
+	}
+	return false
+}
+
+// MatchesAny reports whether any cap matches any pattern; an empty list matches.
+func (r Resource) MatchesAny(patterns []string) bool {
+	if len(patterns) == 0 {
+		return true
+	}
+	for _, p := range patterns {
+		if r.Matches(p) {
+			return true
+		}
+	}
+	return false
+}
+
+// Endpoint returns the named endpoint as an absolute URL.
+func (r Resource) Endpoint(name string) string {
+	v := r.Endpoints[name]
+	if strings.HasPrefix(v, "/") {
+		base := r.Endpoints["http"]
+		if base == "" {
+			return ""
+		}
+		return strings.TrimRight(base, "/") + v
+	}
+	return v
+}
+
+// Message is every v2 datagram.
+type Message struct {
+	Proto     string     `json:"proto"`
+	V         int        `json:"v"`
+	Type      string     `json:"type"`
+	Node      *NodeInfo  `json:"node,omitempty"`
+	Resources []Resource `json:"resources,omitempty"`
+	// From and Want are probe-only.
+	From string   `json:"from,omitempty"`
+	Want []string `json:"want,omitempty"`
+}
+
+// Parse decodes a datagram, returning ok=false for anything that is not a
+// well-formed beacon v2 message (v1 traffic, other versions, unknown types,
+// an advertise/bye without a node).
+func Parse(b []byte) (Message, bool) {
+	var m Message
+	if err := json.Unmarshal(b, &m); err != nil {
+		return m, false
+	}
+	if m.Proto != Proto || m.V != Version {
+		return m, false
+	}
+	switch m.Type {
+	case TypeProbe:
+		return m, true
+	case TypeAdvertise, TypeBye:
+		if m.Node == nil || m.Node.Name == "" || m.Node.Instance == "" {
+			return m, false
+		}
+		return m, true
+	}
+	return m, false
+}
+
+func splitContract(s string) (string, string, bool) {
+	if i := strings.LastIndex(s, "@"); i >= 0 {
+		return s[:i], s[i+1:], true
+	}
+	return s, "", false
+}
+
+// CapMatches reports whether capability cap satisfies pattern:
+// "*" matches everything; an "@N" on the pattern must equal the cap's
+// contract (none on the pattern ignores it); "x/*" matches any variant of x;
+// otherwise the bases must be equal.
+func CapMatches(pattern, cap string) bool {
+	if pattern == "*" {
+		return true
+	}
+	pbase, pc, phas := splitContract(pattern)
+	cbase, cc, chas := splitContract(cap)
+	if phas && (!chas || pc != cc) {
+		return false
+	}
+	if prefix, ok := strings.CutSuffix(pbase, "/*"); ok {
+		rest, ok := strings.CutPrefix(cbase, prefix)
+		return ok && strings.HasPrefix(rest, "/") && len(rest) > 1
+	}
+	return pbase == cbase
+}
+
+// Neighbor is a node heard on the wire, as served by /api/neighbors.
+type Neighbor struct {
+	NodeInfo
+	Resources []Resource `json:"resources"`
+	// Caps flattens every resource's caps, for consumers that just filter.
+	Caps []string `json:"caps"`
+	// BaseUrl is the metamesh.service/* resource's endpoints.ui — kept so
+	// v1-era readers (the nav menu) keep working.
+	BaseUrl string `json:"baseUrl,omitempty"`
+	// Addr is the packet's source address, never a payload claim.
+	Addr string `json:"addr"`
+	// LastSeen is unix seconds.
+	LastSeen float64 `json:"lastSeen"`
+}
+
+// Matches reports whether any resource matches pattern.
+func (n Neighbor) Matches(pattern string) bool {
+	for _, r := range n.Resources {
+		if r.Matches(pattern) {
+			return true
+		}
+	}
+	return false
+}
+
+func newNeighbor(node NodeInfo, resources []Resource, addr string, seen time.Time) Neighbor {
+	nb := Neighbor{
+		NodeInfo:  node,
+		Resources: resources,
+		Caps:      []string{},
+		Addr:      addr,
+		LastSeen:  float64(seen.UnixNano()) / 1e9,
+	}
+	if nb.Resources == nil {
+		nb.Resources = []Resource{}
+	}
+	seenCap := map[string]bool{}
+	for _, r := range resources {
+		for _, c := range r.Caps {
+			if !seenCap[c] {
+				seenCap[c] = true
+				nb.Caps = append(nb.Caps, c)
+			}
+		}
+		if nb.BaseUrl == "" && r.Matches(CapAnyService) {
+			nb.BaseUrl = r.Endpoint("ui")
+		}
+	}
+	return nb
+}

internal/meshdisco/proto_test.go

@@ -0,0 +1,140 @@
+package meshdisco
+
+import (
+	"encoding/json"
+	"net"
+	"testing"
+	"time"
+)
+
+func TestCapMatches(t *testing.T) {
+	cases := []struct {
+		pattern, cap string
+		want         bool
+	}{
+		{"metamesh.transport/*", "metamesh.transport/nzb@1", true},
+		{"metamesh.transport/*", "metamesh.transport/ipfs", true},
+		{"metamesh.transport/*", "metamesh.transport", false},
+		{"metamesh.transport/*", "metamesh.transportx/a", false},
+		{"metamesh.transport/*", "metamesh.transport/", false},
+		{"metamesh.transport/nzb", "metamesh.transport/nzb@2", true},
+		{"metamesh.transport/nzb@2", "metamesh.transport/nzb@2", true},
+		{"metamesh.transport/nzb@2", "metamesh.transport/nzb@1", false},
+		{"metamesh.transport/nzb@2", "metamesh.transport/nzb", false},
+		{"metamesh.transport/*@1", "metamesh.transport/nzb@1", true},
+		{"metamesh.core", "metamesh.core", true},
+		{"metamesh.core", "metamesh.core2", false},
+		{"mcp", "mcp/x", false},
+		{"*", "anything@3", true},
+	}
+	for _, c := range cases {
+		if got := CapMatches(c.pattern, c.cap); got != c.want {
+			t.Errorf("CapMatches(%q, %q) = %v, want %v", c.pattern, c.cap, got, c.want)
+		}
+	}
+}
+
+func TestParseRejectsV1AndForeign(t *testing.T) {
+	for _, raw := range []string{
+		`{"type":"discovery"}`,
+		`{"type":"announce","name":"x","tools":[]}`,
+		`{"v":1,"type":"announce","name":"meta-sort"}`,
+		`{"proto":"beacon","v":3,"type":"probe"}`,
+		`{"proto":"beacon","v":2,"type":"announce"}`,
+		`{"proto":"beacon","v":2,"type":"advertise"}`,
+		"\x00garbage",
+	} {
+		if _, ok := Parse([]byte(raw)); ok {
+			t.Errorf("Parse(%q) accepted", raw)
+		}
+	}
+	if _, ok := Parse([]byte(`{"proto":"beacon","v":2,"type":"probe","want":["metamesh.core"]}`)); !ok {
+		t.Error("valid probe rejected")
+	}
+}
+
+func TestNeighborRowCarriesCapsAndBaseUrl(t *testing.T) {
+	nb := newNeighbor(
+		NodeInfo{Name: "meta-sort", Instance: "metasort-app"},
+		[]Resource{{ID: "service", Caps: []string{"metamesh.service/meta-sort"}, Endpoints: map[string]string{"ui": "https://x"}}},
+		"10.0.0.1", time.Now(),
+	)
+	if nb.BaseUrl != "https://x" || len(nb.Caps) != 1 || !nb.Matches(CapAnyService) {
+		t.Fatalf("unexpected row: %+v", nb)
+	}
+}
+
+// End to end over loopback: a probe with a non-matching want gets no reply;
+// a matching one gets the advertise; advertise then bye updates the view.
+func TestProbeAndRecord(t *testing.T) {
+	port := freePort(t)
+	n := New(Config{
+		Name: "meta-core", Instance: "core-1", Port: port, Interval: time.Hour,
+		Payload: func() (string, []Resource) {
+			return "running", []Resource{{ID: "core", Caps: []string{CapCore}}}
+		},
+	})
+	if err := n.Start(); err != nil {
+		t.Fatal(err)
+	}
+	defer n.Stop()
+
+	c, err := net.ListenUDP("udp4", &net.UDPAddr{IP: net.IPv4(127, 0, 0, 1)})
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer c.Close()
+	dst := &net.UDPAddr{IP: net.IPv4(127, 0, 0, 1), Port: port}
+	buf := make([]byte, ReadBufferSize)
+
+	send := func(m Message) {
+		b, _ := json.Marshal(m)
+		if _, err := c.WriteToUDP(b, dst); err != nil {
+			t.Fatal(err)
+		}
+	}
+
+	send(Message{Proto: Proto, V: Version, Type: TypeProbe, From: "cli", Want: []string{"metamesh.service/*"}})
+	c.SetReadDeadline(time.Now().Add(300 * time.Millisecond))
+	if _, _, err := c.ReadFromUDP(buf); err == nil {
+		t.Fatal("non-matching probe got a reply")
+	}
+
+	send(Message{Proto: Proto, V: Version, Type: TypeProbe, From: "cli", Want: []string{CapCore}})
+	c.SetReadDeadline(time.Now().Add(2 * time.Second))
+	nr, _, err := c.ReadFromUDP(buf)
+	if err != nil {
+		t.Fatalf("matching probe got no reply: %v", err)
+	}
+	if m, ok := Parse(buf[:nr]); !ok || m.Type != TypeAdvertise || m.Node.Instance != "core-1" {
+		t.Fatalf("bad reply: %s", buf[:nr])
+	}
+
+	peer := &NodeInfo{Name: "meta-sort", Instance: "sort-1"}
+	send(Message{Proto: Proto, V: Version, Type: TypeAdvertise, Node: peer,
+		Resources: []Resource{{ID: "service", Caps: []string{"metamesh.service/meta-sort"}}}})
+	waitFor(t, func() bool { return len(n.Neighbors()) == 1 })
+	send(Message{Proto: Proto, V: Version, Type: TypeBye, Node: peer})
+	waitFor(t, func() bool { return len(n.Neighbors()) == 0 })
+}
+
+func freePort(t *testing.T) int {
+	c, err := net.ListenUDP("udp4", &net.UDPAddr{IP: net.IPv4(127, 0, 0, 1)})
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer c.Close()
+	return c.LocalAddr().(*net.UDPAddr).Port
+}
+
+func waitFor(t *testing.T, cond func() bool) {
+	t.Helper()
+	deadline := time.Now().Add(2 * time.Second)
+	for time.Now().Before(deadline) {
+		if cond() {
+			return
+		}
+		time.Sleep(20 * time.Millisecond)
+	}
+	t.Fatal("condition not reached")
+}

internal/storage/cid_resolution.go

@@ -157,6 +157,42 @@ func (c *Client) addAliasLocked(ctx context.Context, uuid, cidStr string) error
 	return nil
 }
 
+// removeAliasLocked is the inverse of addAliasLocked: it drops the reverse-index
+// entry cid:<cid> → <uuid>. The caller must already hold c.mu, and must delete
+// the cids/<cid> key-set member itself (DeleteProperty does both).
+//
+// ⚠ Why this exists. A cids/ member write touches THREE keys — the flat
+// file:<uuid>/cids/<cid>, the __fields__ index entry, and this reverse alias —
+// while the delete path only ever removed the first two. A "deleted" CID
+// therefore still resolved: GET /meta/<cid>, ResolveRoot and GetByCID all kept
+// answering with the record. The asymmetry is not theoretical. meta-share
+// published dag-pb roots computed over RAR volumes and par2 files onto six
+// records on watch.nsl.sh; removing the member was expressible, removing the
+// claim was not, so the false identity survived its own retraction.
+//
+// Guarded, not blind: the alias is removed only when it currently points at
+// THIS uuid. Two records can legitimately race to claim a cid (see the
+// dual-root guard in addAliasLocked, which leaves the first writer's alias in
+// place); deleting a member from the loser must not unpick the winner's index.
+func (c *Client) removeAliasLocked(ctx context.Context, uuid, cidStr string) error {
+	if uuid == "" || cidStr == "" {
+		return nil
+	}
+	indexKey := c.buildCIDIndexKey(cidStr)
+	existing, err := c.client.Get(ctx, indexKey).Result()
+	if err == redis.Nil {
+		return nil
+	}
+	if err != nil {
+		return fmt.Errorf("read existing alias: %w", err)
+	}
+	if existing != uuid {
+		// Somebody else owns this alias. Leave it.
+		return nil
+	}
+	return c.client.Del(ctx, indexKey).Err()
+}
+
 // ResolveRoot maps a hash supplied by a caller (a meta-sort write, an editor
 // lookup, an SSE consumer) to the actual storage root key.
 //

internal/storage/cid_retraction_test.go

@@ -0,0 +1,158 @@
+package storage
+
+import (
+	"context"
+	"testing"
+)
+
+// Tests for retracting a `cids/<cid>` key-set member (cid_resolution.go's
+// removeAliasLocked, reached through DeleteProperty).
+//
+// Why this is worth its own file. Adding a member writes THREE keys — the flat
+// `file:<uuid>/cids/<cid>`, the `__fields__` index entry, and the reverse alias
+// `cid:<cid>` → `<uuid>`. Until removeAliasLocked existed, deleting removed only
+// the first two, so a retracted CID kept resolving to the record: the member was
+// gone from the document and the claim was not.
+//
+// That asymmetry had a live consequence. meta-share minted dag-pb roots and
+// midhash256s over whatever `archive::pick_media` elected, which on an
+// obfuscated posting was an arbitrary RAR volume or a par2 file. Six records on
+// watch.nsl.sh carry such members. `METADATA_KEYS.md` §2 defines a `cids/`
+// member as a digest of *this record's file* and notes that every member is a
+// reverse index back to the record — so those entries assert that an archive
+// volume is the episode. Retracting one has to remove the assertion, not just
+// the row.
+
+// aliasTarget returns the uuid that cid:<cidStr> currently resolves to, or "".
+func aliasTarget(t *testing.T, c *Client, cidStr string) string {
+	t.Helper()
+	got, err := c.client.Get(context.Background(), c.buildCIDIndexKey(cidStr)).Result()
+	if err != nil {
+		return ""
+	}
+	return got
+}
+
+func hasField(t *testing.T, c *Client, hashID, field string) bool {
+	t.Helper()
+	ok, err := c.client.SIsMember(context.Background(), c.buildFieldsKey(hashID), field).Result()
+	if err != nil {
+		t.Fatalf("sismember: %v", err)
+	}
+	return ok
+}
+
+// The real shape: `…glnrqztt…` is the nzb-release locator whose record published
+// a dag-pb root computed over volume 3 of a 35-volume RAR5 set.
+const (
+	poisonedRoot = "bafybeicicuzdvuztca2hp2g2k7scgr4nnibilzgoppljeprphybqot7v6i"
+	goodMidhash  = "bagacbabaed6uiulnv7tscpxne22eztkjvnxg5almn43dpfecppixarfe7xhlu"
+)
+
+func TestRetractingACidMemberRemovesTheClaimNotJustTheRow(t *testing.T) {
+	forEachPrefix(t, func(t *testing.T, c *Client) {
+		const uuid = "01KSBHMTESTRECORD"
+
+		if err := c.SetMetadataFlat(uuid, map[string]string{
+			"cids/" + poisonedRoot: "true",
+			"cids/" + goodMidhash:  "true",
+			"fileType":             "video",
+		}); err != nil {
+			t.Fatalf("seed: %v", err)
+		}
+
+		// All three writes landed, and the CID resolves to the record.
+		if got := aliasTarget(t, c, poisonedRoot); got != uuid {
+			t.Fatalf("alias should point at the record before retraction, got %q", got)
+		}
+		if !hasField(t, c, uuid, "cids/"+poisonedRoot) {
+			t.Fatal("field index should list the member before retraction")
+		}
+		if got := c.ResolveRoot(poisonedRoot); got != uuid {
+			t.Fatalf("ResolveRoot should find the record before retraction, got %q", got)
+		}
+
+		if err := c.DeleteProperty(uuid, "cids/"+poisonedRoot); err != nil {
+			t.Fatalf("DeleteProperty: %v", err)
+		}
+
+		// 1. the flat key
+		if v, err := c.client.Get(context.Background(),
+			c.buildKeyPrefix(uuid)+"cids/"+poisonedRoot).Result(); err == nil {
+			t.Fatalf("flat key should be gone, still %q", v)
+		}
+		// 2. the field index
+		if hasField(t, c, uuid, "cids/"+poisonedRoot) {
+			t.Fatal("field index still lists the retracted member")
+		}
+		// 3. the reverse alias — the one that used to survive
+		if got := aliasTarget(t, c, poisonedRoot); got != "" {
+			t.Fatalf("reverse alias survived retraction, still points at %q", got)
+		}
+		if got := c.ResolveRoot(poisonedRoot); got == uuid {
+			t.Fatal("a retracted cid must not resolve to the record any more")
+		}
+
+		// The siblings are untouched: retraction is surgical, not a purge.
+		if got := aliasTarget(t, c, goodMidhash); got != uuid {
+			t.Fatalf("an unrelated sibling lost its alias, got %q", got)
+		}
+		if !hasField(t, c, uuid, "cids/"+goodMidhash) {
+			t.Fatal("an unrelated sibling lost its field-index entry")
+		}
+		if v, err := c.GetProperty(uuid, "fileType"); err != nil || v != "video" {
+			t.Fatalf("an unrelated field changed: %q %v", v, err)
+		}
+	})
+}
+
+// Two records can race to claim one cid; addAliasLocked leaves the first
+// writer's alias in place. Retracting the loser's member must not unpick the
+// winner's index — which is why removeAliasLocked compares before deleting.
+func TestRetractingDoesNotStealAnotherRecordsAlias(t *testing.T) {
+	forEachPrefix(t, func(t *testing.T, c *Client) {
+		const winner = "01KSBHMWINNER"
+		const loser = "01KSBHMLOSER"
+		const shared = "bafybeiaeb2lqi4z3np3q4tarldtyjp2t5lvnoodmtpvpi2tz73azn5td2m"
+
+		if err := c.SetMetadataFlat(winner, map[string]string{"cids/" + shared: "true"}); err != nil {
+			t.Fatalf("seed winner: %v", err)
+		}
+		if err := c.SetMetadataFlat(loser, map[string]string{"cids/" + shared: "true"}); err != nil {
+			t.Fatalf("seed loser: %v", err)
+		}
+		if got := aliasTarget(t, c, shared); got != winner {
+			t.Fatalf("the first writer should own the alias, got %q", got)
+		}
+
+		if err := c.DeleteProperty(loser, "cids/"+shared); err != nil {
+			t.Fatalf("DeleteProperty: %v", err)
+		}
+
+		if got := aliasTarget(t, c, shared); got != winner {
+			t.Fatalf("retracting the loser's member stole the winner's alias, now %q", got)
+		}
+		if hasField(t, c, loser, "cids/"+shared) {
+			t.Fatal("the loser's own member should still be gone")
+		}
+	})
+}
+
+// A non-cids field has no alias to clean up, and the delete must not invent one.
+func TestRetractingAnOrdinaryFieldTouchesNoAlias(t *testing.T) {
+	forEachPrefix(t, func(t *testing.T, c *Client) {
+		const uuid = "01KSBHMORDINARY"
+		if err := c.SetMetadataFlat(uuid, map[string]string{
+			"cids/" + goodMidhash: "true",
+			"quality":             "1080p",
+		}); err != nil {
+			t.Fatalf("seed: %v", err)
+		}
+		if err := c.DeleteProperty(uuid, "quality"); err != nil {
+			t.Fatalf("DeleteProperty: %v", err)
+		}
+		if got := aliasTarget(t, c, goodMidhash); got != uuid {
+			t.Fatalf("deleting an ordinary field disturbed a cid alias, got %q", got)
+		}
+	})
+}

internal/storage/client.go

@@ -460,6 +460,10 @@ func (c *Client) MergeMetadataFlat(hashID string, metadata map[string]string) (i
 
 // DeleteProperty deletes a single property
 // Uses flat key: DEL file:{hashId}/{property}
+//
+// For a cids/<cid> member this is the full inverse of the write: the flat key,
+// the __fields__ entry AND the cid:<cid> reverse alias all go. Anything less
+// leaves the CID still resolving to this record — see removeAliasLocked.
 func (c *Client) DeleteProperty(hashID, property string) error {
 	c.mu.RLock()
 	defer c.mu.RUnlock()
@@ -475,6 +479,13 @@ func (c *Client) DeleteProperty(hashID, property string) error {
 	if err := c.client.Del(ctx, key).Err(); err != nil {
 		return err
 	}
+	// Before the field index forgets the name, use it: a cids/ member owns a
+	// reverse-index entry that nothing else will ever clean up.
+	if cidStr := cidFromKeysetField(property); cidStr != "" {
+		if err := c.removeAliasLocked(ctx, hashID, cidStr); err != nil {
+			return fmt.Errorf("remove alias for %s: %w", cidStr, err)
+		}
+	}
 	c.removeFieldLocked(ctx, hashID, property)
 	c.markDirtyLocked(ctx, hashID)
 	return nil

internal/storage/field_index_test.go

@@ -59,7 +59,7 @@ func TestFieldIndex_SetMetadataFlatIndexesFields(t *testing.T) {
 			"filePath":                      "/files/watch/a.mkv",
 			"sizeByte":                      "123",
 			"cids/bafkreiaaa":               "true",
-			"titles/jpn":                    "ナルト",
+			"titles/jpn/ナルト":                "true",
 			"categories/TV/UHD":             "true",
 			"info/files/0/path":             "a.mkv",
 			"provenance":                    `{"source":"gateway"}`,
@@ -380,7 +380,7 @@ func TestFieldIndex_WarmBuildsEveryMissingIndex(t *testing.T) {
 		// written through the normal path (already indexed).
 		legacy := map[string]map[string]string{
 			"old1": {"filePath": "/a.mkv", "cids/bafkreiaaa": "true"},
-			"old2": {"filePath": "/b.mkv", "titles/jpn": "ナルト", "deep/a/b": "x"},
+			"old2": {"filePath": "/b.mkv", "titles/jpn/ナルト": "true", "deep/a/b": "x"},
 		}
 		for root, fields := range legacy {
 			for f, v := range fields {
@@ -407,7 +407,7 @@ func TestFieldIndex_WarmBuildsEveryMissingIndex(t *testing.T) {
 		if got := fieldNames(t, c, "old1"); !reflect.DeepEqual(got, []string{"cids/bafkreiaaa", "filePath"}) {
 			t.Errorf("old1 index = %v", got)
 		}
-		if got := fieldNames(t, c, "old2"); !reflect.DeepEqual(got, []string{"deep/a/b", "filePath", "titles/jpn"}) {
+		if got := fieldNames(t, c, "old2"); !reflect.DeepEqual(got, []string{"deep/a/b", "filePath", "titles/jpn/ナルト"}) {
 			t.Errorf("old2 index = %v", got)
 		}
 		if got := fieldNames(t, c, "fresh"); !reflect.DeepEqual(got, []string{"title"}) {
@@ -425,7 +425,7 @@ func TestFieldIndex_WarmBuildsEveryMissingIndex(t *testing.T) {
 		if _, err := c.WarmFieldIndexes(); err != nil {
 			t.Fatalf("second WarmFieldIndexes: %v", err)
 		}
-		if got := fieldNames(t, c, "old2"); !reflect.DeepEqual(got, []string{"deep/a/b", "filePath", "titles/jpn"}) {
+		if got := fieldNames(t, c, "old2"); !reflect.DeepEqual(got, []string{"deep/a/b", "filePath", "titles/jpn/ナルト"}) {
 			t.Errorf("old2 index after second warm = %v", got)
 		}
 	})

internal/storage/search_dirty.go

@@ -4,6 +4,7 @@ import (
 	"context"
 	"fmt"
 	"log"
+	"sort"
 	"strings"
 	"time"
 )
@@ -202,6 +203,11 @@ func (c *Client) GetSearchIndexTuplesFor(hashIDs []string) (tuples []SearchTuple
 // blob. Shared by the full and incremental index reads so the two can never
 // disagree about what is searchable — a drift there would make a record's
 // matchability depend on which path happened to index it.
+//
+// Beyond the flat fields it folds in every `titles/<lang3>/<name>` member's
+// name (METADATA_KEYS.md §3): the name lives in the key and the value is the
+// sentinel "true", so a search for an alternative title (`3 percent`) finds the
+// record whose `title` is `3%`.
 func buildHaystack(fields map[string]string) string {
 	var sb strings.Builder
 	for _, f := range searchIndexFields {
@@ -210,5 +216,30 @@ func buildHaystack(fields map[string]string) string {
 			sb.WriteByte('\n')
 		}
 	}
+	for _, name := range titleMemberNames(fields) {
+		sb.WriteString(strings.ToLower(name))
+		sb.WriteByte('\n')
+	}
 	return sb.String()
 }
+
+// titleMemberNames returns the names of a record's `titles/<lang3>/<name>`
+// key-set members, with the `∕` (U+2215) a writer substitutes for `/` mapped
+// back. Sorted: the fields map has no order, and the full and incremental
+// builds must produce byte-identical haystacks.
+func titleMemberNames(fields map[string]string) []string {
+	var names []string
+	for k, v := range fields {
+		rest, ok := strings.CutPrefix(k, "titles/")
+		if !ok || v == "" || v == "false" {
+			continue
+		}
+		_, name, ok := strings.Cut(rest, "/")
+		if !ok || name == "" {
+			continue
+		}
+		names = append(names, strings.ReplaceAll(name, "\u2215", "/"))
+	}
+	sort.Strings(names)
+	return names
+}

internal/storage/search_dirty_test.go

@@ -174,6 +174,8 @@ func TestDirty_IncrementalTuplesMatchTheFullRebuild(t *testing.T) {
 			"title":                   "Parity Release",
 			"fileName":                "parity.s01e01.mkv",
 			"fileType":                "video",
+			"titles/eng/3 Percent":    "true",
+			"titles/fra/3 Pourcent":   "true",
 			"categories/XXX":          "true",
 			"categories/newznab/5070": "true",
 		}); err != nil {
@@ -209,3 +211,20 @@ func TestDirty_IncrementalTuplesMatchTheFullRebuild(t *testing.T) {
 		}
 	})
 }
+
+// A `titles/<lang3>/<name>` member contributes its NAME (the key), never its
+// "true" value, and the `∕` a writer substituted for `/` reads back as `/`.
+func TestBuildHaystack_FoldsTitleMemberNames(t *testing.T) {
+	got := buildHaystack(map[string]string{
+		"title":                     "3%",
+		"titles/fra/3 Pourcent":     "true",
+		"titles/eng/3 Percent":      "true",
+		"titles/und/Fate\u2215Zero": "true",
+		"titles/eng":                "Retired Scalar",
+		"titles/eng/":               "true",
+	})
+	want := "3%\n3 percent\n3 pourcent\nfate/zero\n"
+	if got != want {
+		t.Errorf("haystack = %q, want %q", got, want)
+	}
+}

internal/storage/stats.go

@@ -0,0 +1,52 @@
+package storage
+
+import (
+	"context"
+	"fmt"
+	"time"
+)
+
+// CountRoots returns the number of metadata roots in the file index.
+//
+// SCARD, not SMEMBERS: GetAllHashIDs ships every member over the wire (~3MB
+// and rising on an 88k-record box) just so the caller can take len(). A
+// dashboard tile that polls needs the number, never the list.
+func (c *Client) CountRoots() (int64, error) {
+	c.mu.RLock()
+	defer c.mu.RUnlock()
+
+	if c.client == nil {
+		return 0, fmt.Errorf("not connected")
+	}
+
+	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
+	defer cancel()
+
+	n, err := c.client.SCard(ctx, c.buildIndexKey()).Result()
+	if err != nil {
+		return 0, fmt.Errorf("scard failed: %w", err)
+	}
+	return n, nil
+}
+
+// CountKeys returns DBSIZE — every key in the Redis database, not just the
+// ones meta-core owns. It is the denominator for "how much of Redis is
+// records": ~3.1M keys against ~89k roots means the flat per-field keys
+// (file:{root}/{field}) dominate, which is expected.
+func (c *Client) CountKeys() (int64, error) {
+	c.mu.RLock()
+	defer c.mu.RUnlock()
+
+	if c.client == nil {
+		return 0, fmt.Errorf("not connected")
+	}
+
+	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
+	defer cancel()
+
+	n, err := c.client.DBSize(ctx).Result()
+	if err != nil {
+		return 0, fmt.Errorf("dbsize failed: %w", err)
+	}
+	return n, nil
+}

internal/storage/udl.go

@@ -17,6 +17,7 @@ package storage
 //	udl:rec:<uid>/<cid>/<key>       HASH { record, version, ts, value? }
 //	udl:idx:user:<uid>:key:<key>    SET of <cid>   -> active_for_user_key (My List, Continue Watching)
 //	udl:idx:user:<uid>:cid:<cid>    SET of <key>   -> list_for_user_cid
+//	udl:idx:user:<uid>:cids         SET of <cid>   -> whole-profile export (recovery)
 //	udl:idx:cid:<cid>:key:<key>     SET of <uid>   -> aggregate
 //	udl:idx:cid:<cid>:users         SET of <uid>   -> uids_for_cid
 //
@@ -27,7 +28,9 @@ package storage
 import (
 	"context"
 	"fmt"
+	"sort"
 	"strconv"
+	"strings"
 	"time"
 
 	"github.com/redis/go-redis/v9"
@@ -66,11 +69,24 @@ func udlIdxUserCid(uid, cid string) string { return fmt.Sprintf("udl:idx:user:%s
 func udlIdxCidKey(cid, key string) string  { return fmt.Sprintf("udl:idx:cid:%s:key:%s", cid, key) }
 func udlIdxCidUsers(cid string) string      { return fmt.Sprintf("udl:idx:cid:%s:users", cid) }
 
+// udlIdxUserCids is the set of every cid a uid has ANY record for — the one
+// index that makes "export this whole profile" a set read instead of a
+// keyspace scan.
+//
+// The other four indexes each answer a question with one coordinate already
+// known (a key, a cid). Cross-instance recovery knows neither: it has a uid and
+// nothing else. Without this set the only ways to enumerate a profile are to
+// iterate the hardcoded key list — which silently misses any key added later —
+// or to SCAN `udl:rec:<uid>/*`, which is the full-keyspace-read pathology that
+// already made `/api/udl/users/stats` admin-only.
+func udlIdxUserCids(uid string) string { return fmt.Sprintf("udl:idx:user:%s:cids", uid) }
+
 // udlUpsertScript performs the version-gated upsert + index maintenance
 // atomically. Accepts (returns 1) only when the incoming version strictly
 // exceeds the stored one; otherwise returns 0 (stale).
 //
 //	KEYS: 1=cell 2=idx:user:key 3=idx:user:cid 4=idx:cid:key 5=idx:cid:users
+//	      6=idx:user:cids
 //	ARGV: 1=version 2=ts 3=record 4=value 5=hasValue("1"/"0")
 //	      6=cid 7=key 8=uid 9=tombstone("1"/"0")
 //
@@ -78,9 +94,16 @@ func udlIdxCidUsers(cid string) string      { return fmt.Sprintf("udl:idx:cid:%s
 // would let an older write resurrect the value — but it SREMs the cid from the
 // user+key index. That index is what `active_for_user_key` returns, so leaving
 // deleted cids in it means every My List / Continue Watching read carries every
-// title ever un-liked or finished, forever. The other three indexes are left
+// title ever un-liked or finished, forever. The other four indexes are left
 // alone: they answer "who has an opinion about this cid", where a tombstone is
 // still an answer.
+//
+// ⚠ `idx:user:cids` (KEYS[6]) in particular must NEVER be pruned on a
+// tombstone, even though it looks symmetric with KEYS[2]. It is the export
+// index: a cid dropped from it is a cid whose tombstone never travels, and a
+// tombstone that does not travel is exactly how an un-liked title comes back
+// from the dead on the other box — the peer still holds the older `like` at a
+// lower version, and with no tombstone to beat it, nothing ever supersedes it.
 var udlUpsertScript = redis.NewScript(`
 local stored = redis.call('HGET', KEYS[1], 'version')
 if stored and tonumber(stored) >= tonumber(ARGV[1]) then
@@ -100,6 +123,7 @@ end
 redis.call('SADD', KEYS[3], ARGV[7])
 redis.call('SADD', KEYS[4], ARGV[8])
 redis.call('SADD', KEYS[5], ARGV[8])
+redis.call('SADD', KEYS[6], ARGV[6])
 return 1
 `)
 
@@ -126,6 +150,7 @@ func (c *Client) UDLUpsertIfNewer(uid, cid, key string, version, ts int64, recor
 		c.buildKey(udlIdxUserCid(uid, cid)),
 		c.buildKey(udlIdxCidKey(cid, key)),
 		c.buildKey(udlIdxCidUsers(cid)),
+		c.buildKey(udlIdxUserCids(uid)),
 	}
 	hv := "0"
 	if hasValue {
@@ -250,6 +275,160 @@ func (c *Client) UDLListUserCid(uid, cid string) ([]UDLUserCidEntry, error) {
 	return out, nil
 }
 
+// UDLUserAllEntry is one row of a whole-profile export.
+type UDLUserAllEntry struct {
+	Cid     string `json:"cid"`
+	Key     string `json:"key"`
+	Record  string `json:"record"`
+	Version int64  `json:"version"`
+	Ts      int64  `json:"ts"`
+}
+
+// UDLAllForUser returns every cell a uid owns, for cross-instance recovery.
+//
+// Ordered by cid (sorted, so paging is stable across calls even though the
+// underlying index is a Redis SET), starting strictly after `afterCid`. A cid's
+// keys are never split across a page: the page ends on a cid boundary once
+// `limit` rows have been emitted, and `nextCid` is the last cid included — pass
+// it back as `afterCid` to continue. `nextCid` is empty when the export is
+// complete.
+//
+// Tombstones are included. They are the CRDT state that stops a deleted value
+// resurrecting on the box that receives this export, so omitting them would
+// make recovery actively wrong rather than merely incomplete.
+func (c *Client) UDLAllForUser(uid, afterCid string, limit int) ([]UDLUserAllEntry, string, error) {
+	c.mu.RLock()
+	defer c.mu.RUnlock()
+
+	if c.client == nil {
+		return nil, "", fmt.Errorf("not connected")
+	}
+	if limit <= 0 {
+		limit = 1000
+	}
+
+	ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
+	defer cancel()
+
+	cids, err := c.client.SMembers(ctx, c.buildKey(udlIdxUserCids(uid))).Result()
+	if err != nil {
+		return nil, "", fmt.Errorf("smembers failed: %w", err)
+	}
+	sort.Strings(cids)
+
+	out := make([]UDLUserAllEntry, 0, limit)
+	nextCid := ""
+	for _, cid := range cids {
+		if afterCid != "" && cid <= afterCid {
+			continue
+		}
+		if len(out) >= limit {
+			nextCid = out[len(out)-1].Cid
+			break
+		}
+		keys, err := c.client.SMembers(ctx, c.buildKey(udlIdxUserCid(uid, cid))).Result()
+		if err != nil {
+			return nil, "", fmt.Errorf("smembers failed: %w", err)
+		}
+		sort.Strings(keys)
+		for _, key := range keys {
+			cell, ok, err := c.udlReadCell(ctx, udlRecKey(uid, cid, key))
+			if err != nil {
+				return nil, "", err
+			}
+			if !ok {
+				continue
+			}
+			out = append(out, UDLUserAllEntry{
+				Cid: cid, Key: key, Record: cell.Record, Version: cell.Version, Ts: cell.Ts,
+			})
+		}
+	}
+	return out, nextCid, nil
+}
+
+// UDLCountUserCids is the number of distinct cids a uid holds records for.
+//
+// O(1) (a Redis SCARD), which is what makes it usable as a per-push quota
+// check on the gateway mirror — a quota that cost a keyspace walk would be a
+// denial-of-service in its own right. Counts cids rather than cells because
+// that is what the index holds; a profile is bounded by the titles it has
+// touched, not by how many keys it has per title.
+func (c *Client) UDLCountUserCids(uid string) (int64, error) {
+	c.mu.RLock()
+	defer c.mu.RUnlock()
+
+	if c.client == nil {
+		return 0, fmt.Errorf("not connected")
+	}
+
+	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
+	defer cancel()
+
+	n, err := c.client.SCard(ctx, c.buildKey(udlIdxUserCids(uid))).Result()
+	if err != nil {
+		return 0, fmt.Errorf("scard failed: %w", err)
+	}
+	return n, nil
+}
+
+// UDLBackfillUserCids rebuilds the `idx:user:<uid>:cids` sets for data written
+// before that index existed, and reports how many cids it added.
+//
+// One-shot: the upsert script maintains the set from now on. It is a SCAN, but
+// a targeted one (`udl:idx:user:*:cid:*`, which is one key per (uid,cid), not
+// per record) run once at boot rather than per read — the distinction that
+// makes the difference between a migration and the pathology this index exists
+// to avoid.
+func (c *Client) UDLBackfillUserCids() (int, error) {
+	c.mu.Lock()
+	defer c.mu.Unlock()
+
+	if c.client == nil {
+		return 0, fmt.Errorf("not connected")
+	}
+
+	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)
+	defer cancel()
+
+	pattern := c.buildKey("udl:idx:user:*:cid:*")
+	prefix := c.buildKey("udl:idx:user:")
+	added := 0
+	var cursor uint64
+	for {
+		keys, next, err := c.client.Scan(ctx, cursor, pattern, 500).Result()
+		if err != nil {
+			return added, fmt.Errorf("scan failed: %w", err)
+		}
+		for _, k := range keys {
+			// <prefix><uid>:cid:<cid> — split on the FIRST separator, not the
+			// last. A uid is multibase base58btc, whose alphabet contains no
+			// colon at all, so the first ":cid:" always ends the uid. A cid
+			// that happens to contain ":cid:" in its own name would make the
+			// last occurrence split in the wrong place.
+			rest := strings.TrimPrefix(k, prefix)
+			i := strings.Index(rest, ":cid:")
+			if i < 0 {
+				continue
+			}
+			uid, cid := rest[:i], rest[i+len(":cid:"):]
+			if uid == "" || cid == "" {
+				continue
+			}
+			n, err := c.client.SAdd(ctx, c.buildKey(udlIdxUserCids(uid)), cid).Result()
+			if err != nil {
+				return added, fmt.Errorf("sadd failed: %w", err)
+			}
+			added += int(n)
+		}
+		cursor = next
+		if cursor == 0 {
+			break
+		}
+	}
+	return added, nil
+}
+
 // UDLCidUsers returns every uid that has any record for cid — uids_for_cid.
 func (c *Client) UDLCidUsers(cid string) ([]string, error) {
 	c.mu.RLock()

internal/storage/udl_admin.go

@@ -179,7 +179,7 @@ func (c *Client) UDLAllUserStats() (map[string]UDLUserStats, error) {
 }
 
 // UDLPurgeUser removes every trace of one account from the User Data Layer:
-// its cells, its two user-scoped indexes, and its membership in the shared
+// its cells, its three user-scoped indexes, and its membership in the shared
 // cid-scoped indexes. Returns what was removed.
 //
 // The shared indexes are the part that is easy to miss and expensive to leave.
@@ -225,8 +225,10 @@ func (c *Client) UDLPurgeUser(uid string) (UDLUserStats, error) {
 	stats.Cids = len(distinctCids)
 	stats.Keys = len(distinctKeys)
 
-	// The two user-scoped indexes are enumerable directly, and are dropped
+	// The three user-scoped indexes (`:key:*`, `:cid:*`, `:cids`) are enumerable
-	// whole rather than member-by-member. Scanning them separately also catches
+	// directly, and are dropped whole rather than member-by-member. The glob is
+	// what keeps this correct as indexes are added — `:cids` needed no change
+	// here when it landed. Scanning them separately also catches
 	// index entries whose cell is already gone — exactly the state a previously
 	// half-finished purge leaves behind.
 	idxPrefix := c.buildKey("udl:idx:user:") + uid + ":"

internal/storage/udl_export_test.go

@@ -0,0 +1,234 @@
+package storage
+
+import (
+	"context"
+	"testing"
+)
+
+// The whole-profile export is what cross-instance recovery reads: a box that
+// has just imported a secret key knows the uid and nothing else — not which
+// cids the profile touched, not which keys. Every one of these tests is about
+// that one caller.
+
+func TestUDLAllForUser_ReturnsEveryCellAcrossCidsAndKeys(t *testing.T) {
+	c, _ := newTestClient(t, "")
+
+	const uid = "zAlice"
+	writes := []struct{ cid, key, rec string }{
+		{"bagcsaaaaone", "like", "rec-one-like"},
+		{"bagcsaaaaone", "seek", "rec-one-seek"},
+		{"bagcsaaaatwo", "rating", "rec-two-rating"},
+		{"bagcsaaaatwo", "profile:name", "rec-two-name"},
+		{"bagcsaaathree", "watched", "rec-three-watched"},
+	}
+	for _, w := range writes {
+		if ok, err := c.UDLUpsertIfNewer(uid, w.cid, w.key, 1, 100, w.rec, "", false, false); err != nil || !ok {
+			t.Fatalf("upsert %s/%s: ok=%v err=%v", w.cid, w.key, ok, err)
+		}
+	}
+
+	entries, next, err := c.UDLAllForUser(uid, "", 0)
+	if err != nil {
+		t.Fatalf("export: %v", err)
+	}
+	if next != "" {
+		t.Fatalf("export of 5 rows should be complete, got nextCid=%q", next)
+	}
+	if len(entries) != len(writes) {
+		t.Fatalf("want %d rows, got %d: %+v", len(writes), len(entries), entries)
+	}
+	seen := map[string]string{}
+	for _, e := range entries {
+		seen[e.Cid+"/"+e.Key] = e.Record
+	}
+	for _, w := range writes {
+		if got := seen[w.cid+"/"+w.key]; got != w.rec {
+			t.Errorf("%s/%s: want record %q, got %q", w.cid, w.key, w.rec, got)
+		}
+	}
+}
+
+// The failure this guards is silent and total: a uid whose cids were never
+// indexed exports as empty, and recovery reports success having restored
+// nothing.
+func TestUDLAllForUser_IsEmptyForAnUnknownUid(t *testing.T) {
+	c, _ := newTestClient(t, "")
+
+	if ok, err := c.UDLUpsertIfNewer("zAlice", "bagcsaaaaone", "like", 1, 100, "rec", "", false, false); err != nil || !ok {
+		t.Fatalf("upsert: ok=%v err=%v", ok, err)
+	}
+	entries, _, err := c.UDLAllForUser("zBob", "", 0)
+	if err != nil {
+		t.Fatalf("export: %v", err)
+	}
+	if len(entries) != 0 {
+		t.Fatalf("Bob owns nothing; got %+v", entries)
+	}
+}
+
+// ⚠ The tombstone rule, and the reason the cids index is never pruned.
+//
+// A tombstone must EXPORT. It is the only thing that can beat the older live
+// value the other box still holds at a lower version — drop it from the export
+// and an un-liked title comes back from the dead on every peer that already
+// saw the `like`. This is the one place the cids index deliberately diverges
+// from the user+key index, which does prune.
+func TestUDLAllForUser_ExportsTombstones(t *testing.T) {
+	c, _ := newTestClient(t, "")
+
+	const (
+		uid = "zAlice"
+		cid = "bagcsaaaaone"
+		key = "like"
+	)
+	if ok, err := c.UDLUpsertIfNewer(uid, cid, key, 1, 100, "rec-liked", "", false, false); err != nil || !ok {
+		t.Fatalf("live upsert: ok=%v err=%v", ok, err)
+	}
+	if ok, err := c.UDLUpsertIfNewer(uid, cid, key, 2, 200, "rec-unliked", "", false, true); err != nil || !ok {
+		t.Fatalf("tombstone upsert: ok=%v err=%v", ok, err)
+	}
+
+	// The user+key index prunes it — that is what keeps My List short.
+	live, err := c.UDLListUserKey(uid, key)
+	if err != nil {
+		t.Fatalf("active_for_user_key: %v", err)
+	}
+	if len(live) != 0 {
+		t.Fatalf("tombstoned cid should leave the user+key index, got %+v", live)
+	}
+
+	// The export must still carry it, at the tombstone's version.
+	entries, _, err := c.UDLAllForUser(uid, "", 0)
+	if err != nil {
+		t.Fatalf("export: %v", err)
+	}
+	if len(entries) != 1 {
+		t.Fatalf("tombstone must export, got %+v", entries)
+	}
+	if entries[0].Record != "rec-unliked" || entries[0].Version != 2 {
+		t.Fatalf("export must carry the tombstone, not the superseded value: %+v", entries[0])
+	}
+}
+
+func TestUDLAllForUser_PagesOnACidBoundaryAndResumes(t *testing.T) {
+	c, _ := newTestClient(t, "")
+
+	const uid = "zAlice"
+	cids := []string{"bagcsaaaaa1", "bagcsaaaaa2", "bagcsaaaaa3"}
+	for _, cid := range cids {
+		for _, key := range []string{"like", "seek"} {
+			if ok, err := c.UDLUpsertIfNewer(uid, cid, key, 1, 100, "rec-"+cid+"-"+key, "", false, false); err != nil || !ok {
+				t.Fatalf("upsert: ok=%v err=%v", ok, err)
+			}
+		}
+	}
+
+	// A cid's keys are never split across a page, so a limit of 1 still yields
+	// both of the first cid's rows.
+	page1, next1, err := c.UDLAllForUser(uid, "", 1)
+	if err != nil {
+		t.Fatalf("page 1: %v", err)
+	}
+	if len(page1) != 2 || page1[0].Cid != cids[0] {
+		t.Fatalf("page 1 should hold both rows of %s, got %+v", cids[0], page1)
+	}
+	if next1 != cids[0] {
+		t.Fatalf("cursor should name the last cid emitted, want %q got %q", cids[0], next1)
+	}
+
+	// Walking the cursor must visit every row exactly once.
+	all := append([]UDLUserAllEntry{}, page1...)
+	cursor := next1
+	for cursor != "" {
+		page, next, err := c.UDLAllForUser(uid, cursor, 1)
+		if err != nil {
+			t.Fatalf("page after %q: %v", cursor, err)
+		}
+		all = append(all, page...)
+		cursor = next
+	}
+	if len(all) != 6 {
+		t.Fatalf("want 6 rows across all pages, got %d: %+v", len(all), all)
+	}
+	seen := map[string]bool{}
+	for _, e := range all {
+		k := e.Cid + "/" + e.Key
+		if seen[k] {
+			t.Fatalf("%s emitted twice across pages", k)
+		}
+		seen[k] = true
+	}
+}
+
+// Data written before the cids index existed must still export, or every
+// profile that predates this feature silently recovers as empty.
+func TestUDLBackfillUserCids_MakesLegacyDataExportable(t *testing.T) {
+	c, _ := newTestClient(t, "")
+
+	const (
+		uid = "zAlice"
+		cid = "bagcsaaaaone"
+		key = "like"
+	)
+	if ok, err := c.UDLUpsertIfNewer(uid, cid, key, 1, 100, "rec", "", false, false); err != nil || !ok {
+		t.Fatalf("upsert: ok=%v err=%v", ok, err)
+	}
+
+	// Simulate the pre-index state: the cell and the other indexes exist, the
+	// cids set does not.
+	ctx := context.Background()
+	if err := c.client.Del(ctx, c.buildKey(udlIdxUserCids(uid))).Err(); err != nil {
+		t.Fatalf("del cids index: %v", err)
+	}
+	if entries, _, err := c.UDLAllForUser(uid, "", 0); err != nil || len(entries) != 0 {
+		t.Fatalf("precondition: legacy data should export as empty, got %+v err=%v", entries, err)
+	}
+
+	added, err := c.UDLBackfillUserCids()
+	if err != nil {
+		t.Fatalf("backfill: %v", err)
+	}
+	if added != 1 {
+		t.Fatalf("want 1 cid backfilled, got %d", added)
+	}
+	entries, _, err := c.UDLAllForUser(uid, "", 0)
+	if err != nil {
+		t.Fatalf("export after backfill: %v", err)
+	}
+	if len(entries) != 1 || entries[0].Cid != cid || entries[0].Key != key {
+		t.Fatalf("backfilled data should export, got %+v", entries)
+	}
+
+	// Idempotent: a second run adds nothing.
+	if added, err := c.UDLBackfillUserCids(); err != nil || added != 0 {
+		t.Fatalf("second backfill should be a no-op, added=%d err=%v", added, err)
+	}
+}
+
+// The backfill splits `<uid>:cid:<cid>` on the FIRST separator, because a uid
+// is multibase base58btc and its alphabet has no colon. Splitting on the last
+// occurrence — the obvious-looking choice — mis-parses a cid that contains
+// ":cid:" in its own name into uid "zAlice:cid:weird", cid "name".
+func TestUDLBackfillUserCids_SplitsOnTheFirstSeparator(t *testing.T) {
+	c, _ := newTestClient(t, "")
+
+	const uid = "zAlice"
+	cid := "weird:cid:name"
+	if ok, err := c.UDLUpsertIfNewer(uid, cid, "like", 1, 100, "rec", "", false, false); err != nil || !ok {
+		t.Fatalf("upsert: ok=%v err=%v", ok, err)
+	}
+	ctx := context.Background()
+	if err := c.client.Del(ctx, c.buildKey(udlIdxUserCids(uid))).Err(); err != nil {
+		t.Fatalf("del cids index: %v", err)
+	}
+	if _, err := c.UDLBackfillUserCids(); err != nil {
+		t.Fatalf("backfill: %v", err)
+	}
+	entries, _, err := c.UDLAllForUser(uid, "", 0)
+	if err != nil {
+		t.Fatalf("export: %v", err)
+	}
+	if len(entries) != 1 || entries[0].Cid != cid {
+		t.Fatalf("want the cid parsed whole, got %+v", entries)
+	}
+}

test/config_test.go

@@ -77,15 +77,9 @@ func TestConfigPaths(t *testing.T) {
 		t.Errorf("Unexpected LockFilePath: %s", cfg.LockFilePath())
 	}
 
-	if cfg.InfoFilePath() != "/test/locks/kv-leader.info" {
-		t.Errorf("Unexpected InfoFilePath: %s", cfg.InfoFilePath())
-	}
 
 	if cfg.RedisDataDir() != "/test/db/redis" {
 		t.Errorf("Unexpected RedisDataDir: %s", cfg.RedisDataDir())
 	}
 
-	if cfg.ServicesDir() != "/test/services" {
-		t.Errorf("Unexpected ServicesDir: %s", cfg.ServicesDir())
-	}
 }

test/election_test.go

@@ -1,144 +0,0 @@
-package test
-
-import (
-	"encoding/json"
-	"os"
-	"path/filepath"
-	"testing"
-
-	"github.com/metazla/meta-core/internal/leader"
-)
-
-func TestLeaderLockInfoJSON(t *testing.T) {
-	info := &leader.LeaderLockInfo{
-		Host:      "test-host",
-		API:       "redis://10.0.0.1:6379",
-		HTTP:      "http://10.0.0.1:8180",
-		BaseURL:   "http://localhost:8180",
-		Timestamp: 1704067200000,
-		PID:       12345,
-	}
-
-	// Marshal to JSON
-	data, err := json.MarshalIndent(info, "", "  ")
-	if err != nil {
-		t.Fatalf("Failed to marshal: %v", err)
-	}
-
-	// Verify JSON structure matches TypeScript format
-	var parsed map[string]interface{}
-	if err := json.Unmarshal(data, &parsed); err != nil {
-		t.Fatalf("Failed to unmarshal: %v", err)
-	}
-
-	if parsed["host"] != "test-host" {
-		t.Errorf("Expected host 'test-host', got '%v'", parsed["host"])
-	}
-
-	if parsed["api"] != "redis://10.0.0.1:6379" {
-		t.Errorf("Expected api 'redis://10.0.0.1:6379', got '%v'", parsed["api"])
-	}
-
-	if parsed["http"] != "http://10.0.0.1:8180" {
-		t.Errorf("Expected http 'http://10.0.0.1:8180', got '%v'", parsed["http"])
-	}
-
-	if parsed["baseUrl"] != "http://localhost:8180" {
-		t.Errorf("Expected baseUrl 'http://localhost:8180', got '%v'", parsed["baseUrl"])
-	}
-
-	// Timestamp should be a number
-	if _, ok := parsed["timestamp"].(float64); !ok {
-		t.Errorf("Expected timestamp to be a number, got %T", parsed["timestamp"])
-	}
-
-	// PID should be a number
-	if _, ok := parsed["pid"].(float64); !ok {
-		t.Errorf("Expected pid to be a number, got %T", parsed["pid"])
-	}
-}
-
-func TestLeaderLockInfoJSONOmitEmpty(t *testing.T) {
-	info := &leader.LeaderLockInfo{
-		Host:      "test-host",
-		API:       "redis://10.0.0.1:6379",
-		HTTP:      "http://10.0.0.1:8180",
-		BaseURL:   "", // Empty - should be omitted
-		Timestamp: 1704067200000,
-		PID:       12345,
-	}
-
-	data, err := json.Marshal(info)
-	if err != nil {
-		t.Fatalf("Failed to marshal: %v", err)
-	}
-
-	// baseUrl should not be present when empty
-	var parsed map[string]interface{}
-	if err := json.Unmarshal(data, &parsed); err != nil {
-		t.Fatalf("Failed to unmarshal: %v", err)
-	}
-
-	if _, exists := parsed["baseUrl"]; exists {
-		t.Error("Expected baseUrl to be omitted when empty")
-	}
-}
-
-func TestLeaderInfoFileFormat(t *testing.T) {
-	// Create a temp directory
-	tmpDir, err := os.MkdirTemp("", "meta-core-test")
-	if err != nil {
-		t.Fatalf("Failed to create temp dir: %v", err)
-	}
-	defer os.RemoveAll(tmpDir)
-
-	info := &leader.LeaderLockInfo{
-		Host:      "test-host",
-		API:       "redis://10.0.0.1:6379",
-		HTTP:      "http://10.0.0.1:8180",
-		Timestamp: 1704067200000,
-		PID:       12345,
-	}
-
-	// Write to file
-	infoPath := filepath.Join(tmpDir, "kv-leader.info")
-	data, err := json.MarshalIndent(info, "", "  ")
-	if err != nil {
-		t.Fatalf("Failed to marshal: %v", err)
-	}
-
-	if err := os.WriteFile(infoPath, data, 0644); err != nil {
-		t.Fatalf("Failed to write file: %v", err)
-	}
-
-	// Read and parse
-	readData, err := os.ReadFile(infoPath)
-	if err != nil {
-		t.Fatalf("Failed to read file: %v", err)
-	}
-
-	var parsed leader.LeaderLockInfo
-	if err := json.Unmarshal(readData, &parsed); err != nil {
-		t.Fatalf("Failed to unmarshal: %v", err)
-	}
-
-	if parsed.Host != info.Host {
-		t.Errorf("Host mismatch: expected '%s', got '%s'", info.Host, parsed.Host)
-	}
-
-	if parsed.API != info.API {
-		t.Errorf("API mismatch: expected '%s', got '%s'", info.API, parsed.API)
-	}
-
-	if parsed.HTTP != info.HTTP {
-		t.Errorf("HTTP mismatch: expected '%s', got '%s'", info.HTTP, parsed.HTTP)
-	}
-
-	if parsed.Timestamp != info.Timestamp {
-		t.Errorf("Timestamp mismatch: expected %d, got %d", info.Timestamp, parsed.Timestamp)
-	}
-
-	if parsed.PID != info.PID {
-		t.Errorf("PID mismatch: expected %d, got %d", info.PID, parsed.PID)
-	}
-}