# UUID-Rooted Metadata with CID Aliasing ## Status **Status:** Design record. Implemented in 2026 alpha (clean-wipe gate); **reframed by meta-core v2.0.0 (claims model)** — see [`docs/implementation/metacore-claims-model.md`](../../../docs/implementation/metacore-claims-model.md) §4. This doc keeps the *why* behind UUID rids and CID symmetry; it is not the schema reference. For what is stored, see [metadata-storage-structure.md](./metadata-storage-structure.md) and `METADATA_KEYS.md` §2/§14.13. What still holds, unchanged since the original proposal: - **No CID is privileged.** Any known CID resolves to its record; all hash types are symmetric. - **Rids are UUIDv7**, 26-char Crockford Base32, **minted locally** with no central allocator. - **`canonical_cid` is derived on read** by rank (`internal/cid/rank.go` → `cid.Better`), never stored. - **`cids` is a key-set of bare CIDs** (`cids/` = `"true"`), no `:` prefix — a CIDv1's multicodec already names the hash function. What changed in v2.0.0: > 1. **The rid is a hidden local name for a group of cids.** Metadata is no > longer stored under `file:/`. Every value is a signed claim > about a **cid** (`claim(uid, cid, key) → value`); `cid_group(cid → rid)` > says which cids form one record. The rid is **derived from cid groups, > never signed and never on the wire** (swarm, gateway); each box mints its > own. > 2. **The `cid:` reverse index is gone**, replaced by the `cid_group` > table (still a primary-key lookup, so still O(1)-class). > 3. **meta-core no longer writes `cids/*` members.** They are signed votes > from the writers that learn a cid (e.g. the full-hash plugin); a non-null > `cids/` vote links two groups. Only the watcher's midhash enters > `cid_group` directly. > 4. **`duplicates` / `dup:/` are gone.** Box-local paths live in > the cache's `local_file(path → midhash, size, mtime)`; several paths with > the same midhash are the duplicates. > > Earlier superseded points, kept for the record: the proposed > `reconcileCanonicalCIDLocked` never shipped; `cids` never was a Redis SET > of `:` tokens; Redis itself went in v1.1.0 (2026-10). Text > below marked *(proposal)* or *(v1.x)* describes those shapes, not v2.0.0. ## Motivation (history) Before UUID rids, every metadata entry in Redis was keyed by a content hash, specifically midhash256: ``` file:midhash256:abc.../filePath file:midhash256:abc.../title file:midhash256:abc.../tmdb/poster ``` Other digests (sha256, IPFS CIDs, BitTorrent infohashes) lived as fields *inside* that entry (`cid_sha256`, `cid_ipfs`). Two problems: 1. **midhash256 was privileged by accident.** It *was* the key, while every other CID was a secondary field. Hash-algorithm changes, or new digest types added by plugins, forced rewrites of every root. The privilege itself was the issue, not which hash held it. 2. **CID lookup was O(n).** "Given any CID, return the metadata" scanned every entry (`storage/client.go::LookupPathByCID`). Unacceptable for a public API, especially when peers query by any CID variant. The requirement: **any CID a client knows must resolve to that content's metadata in O(1)**, and the storage layout must not bias toward any digest type. The claims model adds a second one: **what is signed must name something every peer agrees on** — a rid minted at random on one box does not qualify, a cid does. ## Goal ``` GET /api/meta/ → full (merged) metadata document for that content ``` `` may be any supported type (midhash, sha256, IPFS, btih, sha3, future additions), all treated symmetrically. ## Design (v2.0.0) ### Votes name cids; a rid names a group ``` votes (signed, travel) derived (local, persistent: truth cid_group) (A, M, title) = "The Matrix" M → R1 (A, M, cids/Q) = "true" ← alias vote Q → R1 (B, Q, title) = "Matrix, The" (B, Q, sizeByte)= "734003200" record R1 = every vote on M or Q, merged per key ``` - **First-seen cid, no link** → new group, fresh UUIDv7 rid. - **`cids/` vote on X** → X and `` join one group. - **Link between two existing groups** → the group with the **older rid survives**; the other's cids move to it; the losing rid gets a `del` event for each of its keys, the survivor `set` events. No `rid_alias`: consumers already handle deletes, and cids re-resolve. - **Reading record R** = every vote on every cid of R's group, merged per key (one signer counts once per key), precomputed into the cache's `file:/` rows (the v1 key layout, now derived). - **Retracting a `cids/` link does not split the group** (phase 1): the tombstone drops out of the merge, `cid_group` keeps both cids under one rid. In the record view, membership is synthesised from `cid_group` (`cids/ = "true"` for every cid of the group). Why keep a rid at all, if votes never mention it: local consumers (meta-sort, meta-fuse, meta-stremio) and the merged view need one stable name per record, and a group's cid set grows over time. Why `cid_group` lives in **truth** although it could be recomputed from `cids/*` votes: the rid names are minted at random and consumers hold them, so a recomputation would rename every record. ### Why UUIDv7 (still valid) - **128 bits**: collision-free under distributed minting; no allocator. Each box minting its own rids for the same content is now the normal case, not a corner case — which is exactly why rids never travel. - **Time-sortable prefix**: rids minted in a window sort together (`k >= 'file:01JKR8' AND k < 'file:01JKR9'` on the cache's merged rows), and "older rid survives" on a group merge is a plain string comparison. - **Monotonic-ish**: adjacent rids cluster in the cache's `file:/…` key range. Encoded as 26-char Crockford Base32 (ULID-compatible) for compactness and case-insensitive matching. Migrated records kept their v1.x UUIDs as rids. ### The `cids` key-set `cids/` = `"true"`, one claim per member, by whichever signer knows the link. A key-set merges conflict-free: a member is present while at least one vote supports it, and removal is a signed retraction. That is the reason the proposal's Redis SET was reversed before shipping, and it is what lets the claims model treat `cids` with no special merge code. Box-local file facts (`filePath`, size, `mtimeNano`, midhash) are **not** claims — meta-core cannot sign and the disk can always be rescanned. The watcher records them in `local_file` and puts the midhash into `cid_group`; the record view joins them in unsigned. When several local paths hold the same content, `filePath` is the lexicographically first one. ### Canonical CID selection (unchanged) The canonical CID is a label, not a key, so nothing external breaks when it changes. It is computed on every read from the group's cids: 1. IPFS CID (any variant) 2. SHA-256 / SHA-3 family 3. BitTorrent infohash (v2 preferred over v1) 4. midhash256 Ranked by **utility for external retrieval**, not cryptographic strength: midhash256 and sha256 are both 256-bit, but only sha256/IPFS CIDs are externally meaningful; btih ranks above midhash because it enables swarm interop. The full ladder (including the custom locator codecs) lives in `internal/cid/rank.go`, mirrored in meta-search / meta-share (`scripts/check-cid-vectors.sh`). ## Lifecycles (v2.0.0) ### File arrival ``` 1. Watcher sees /files/watch/foo.mkv, computes its midhash M. 2. local_file: /files/watch/foo.mkv → (M, size, mtime). 3. cid_group: M already grouped → nothing; otherwise M → fresh rid. 4. Plugins process the file by cid M (the rid is never disclosed to them) and sign claims about M with their own keys. 5. The full-hash plugin computes sha256 S and signs (plugin, M, cids/S) = "true". → S joins M's group (or the two groups merge, older rid surviving). → The next read derives canonical_cid = S, because it outranks the midhash. ``` A second path with the same midhash is just another `local_file` row: same cid, same group, same record; the record's `filePath` is the lexicographically first of the paths. ### File deletion The watcher removes the path's `local_file` row; the record view loses that `filePath`. Claims are signed truth from their writers and are not deleted by meta-core (the store is permanent and additive, `docs/project-architecture/metacore-as-permanent-store.md`). *(v1.x)* Deletion promoted a duplicate to `filePath`, or called `DeleteRoot`, which removed every `cid:*` alias and every `file:/*` key. ## Public API ``` GET /api/meta/ → 200 merged document { "filePath", "canonical_cid", "cids": [...], ... } (optional ?as=: the viewer's own votes win) → 404 if no such cid GET /api/meta//claims → {rid, cids, claims}: every live signed claim of the cid's group, all signers (?key=, ?uid=); 404 if no such cid GET /api/metadata//claims → the same, addressed by rid (local consumers) ``` The rid is not part of the cid-addressed contract and never appears in a claim. It is visible only to local consumers that list or address records by rid (`/api/metadata/{rid}`, `/api/metadata/{rid}/claims`, `hash-ids`, SSE event keys; the claims routes also echo it as `rid`), and may change on a group merge (delete event, then re-resolve by cid). Auth: `/api/meta/` bypasses the perimeter like `/api/file/` — it is the metadata twin of the content endpoint. ## Cross-service impact (v2.0.0) - **Writers** (plugins, self-publishing feeders, gateway core, meta-share, meta-search): address claims by cid only and sign them; they never resolve a rid. - **meta-fuse, meta-stremio**: unchanged — merged reads and SSE keep their shape; a group merge reaches them as ordinary `del` + `set` events. - **meta-search / meta-share**: announce the derived canonical cid (each mirrors the Go rank) and relay raw claims; every cid of the group travels as `cids/*` claims, so a peer querying by any digest gets a hit. ## Migration (history) > **The v1.0 sweep below was never run.** The alpha shipped behind a > clean-wipe gate. The v1.x → v2.0.0 conversion of watch.nsl.sh is a > one-off tool outside the service image, `packages/meta-core/tools/claims-migrate` > (canonical cid by rank per record; `cid:` aliases → `cid_group`, rids keep > their UUIDs, and the `cid:` index wins when a record's `cids/*` member > disagrees; fields re-signed by a throwaway legacy key); every other box > starts empty. Implementation basis §11. *(proposal)* One-time sweep over `file:__index__`: mint a uuid, `AddAlias` the old hashID and every `cid_*` field, `RENAME` every property under the new root, reconcile canonical, swap the index entry; crash-recoverable by detecting which step completed. ## Decisions recorded | Question | Decision | Notes | |---|---|---| | When is the rid minted? | When a cid is first seen with no link to a known group | v1.x: at first hash of a watched file. Now any first-seen cid (watcher midhash or a claim about a new cid) mints one | | Rid format | UUIDv7, 26-char Crockford Base32 | Time-sortable, 128-bit, distributed-safe | | Rid visible externally? | No — never signed, never on the swarm/gateway wire | Local HTTP consumers may hold it | | What do signed votes name? | A cid | Every peer agrees on a cid; rids differ per box | | Record identity | A group of cids (`cid_group`, in truth) | Linked by `cids/` votes | | Group merge | Older rid survives; losing rid → `del` events | No `rid_alias` (out of scope for phase 1) | | Duplicate paths | `local_file` rows sharing a midhash (cache) | v1.x: `duplicates` SET, then `dup:/` | | Canonical CID rank | IPFS > sha-family > btih > midhash, derived on read | Utility-weighted; never stored | | `cids` shape | Key-set of bare CIDs, one claim per member | Merges with the generic rule; a SET would not | | Reverse index | `cid_group(cid PK → rid)` in truth, copied to cache `cid:` rows for reads | v1.x: flat `cid:` keys were the only copy | | Unlinking | A retracted `cids/*` link does not split a group | Phase 1 | ## Open questions - **Discovery-time rid minting.** Minting at discovery (before hashing) would give two same-content paths two rids until hashing links them. The claims model already handles that case — it is an ordinary group merge — so this is now a policy choice, not a missing mechanism. - **IPFS gateway endpoint.** Should meta-core expose `/ipfs/` for external IPFS clients? Adjacent, not part of this design. - ~~**meta-share announce strategy.**~~ Settled: the canonical cid addresses the record, every group cid travels as a `cids/*` claim. - ~~**Keyspace-notification consumers.**~~ Obsolete since Redis left. ## Non-goals - **Not** a content-replication or pinning system. A `cids/` vote pins nothing; it states "hashing these bytes with IPFS's chunker gives this CID." - **Not** a history. `ts` versions each `(uid, cid, key)` and the newest vote replaces the older one; only tombstones are kept, to block replay. - **Not** strongly consistent across the two databases: the cache is applied after the truth commit. `POST /api/claims` waits for the apply before answering, so the writer reads its own write; another reader may briefly miss a just-accepted claim. It is behind, never wrong.