meta-core/docs/uuid-rooted-metadata.md

UUID-Rooted Metadata with CID Aliasing

Status

Status: Design record. Implemented in 2026 alpha (clean-wipe gate); reframed by meta-core v2.0.0 (claims model) — see docs/implementation/metacore-claims-model.md §4. This doc keeps the why behind UUID rids and CID symmetry; it is not the schema reference. For what is stored, see metadata-storage-structure.md and METADATA_KEYS.md §2/§14.13.

What still holds, unchanged since the original proposal:

• No CID is privileged. Any known CID resolves to its record; all hash types are symmetric.
• Rids are UUIDv7, 26-char Crockford Base32, minted locally with no central allocator.
• canonical_cid is derived on read by rank (internal/cid/rank.go → cid.Better), never stored.
• cids is a key-set of bare CIDs (cids/<bareCid> = "true"), no <algo>: prefix — a CIDv1's multicodec already names the hash function.

What changed in v2.0.0:

1. The rid is a hidden local name for a group of cids. Metadata is no longer stored under file:<uuid>/<field>. Every value is a signed claim about a cid (claim(uid, cid, key) → value); cid_group(cid → rid) says which cids form one record. The rid is derived from cid groups, never signed and never on the wire (swarm, gateway); each box mints its own.
2. The cid:<cid> reverse index is gone, replaced by the cid_group table (still a primary-key lookup, so still O(1)-class).
3. meta-core no longer writes cids/* members. They are signed votes from the writers that learn a cid (e.g. the full-hash plugin); a non-null cids/<other> vote links two groups. Only the watcher's midhash enters cid_group directly.
4. duplicates / dup:<uuid>/<path> are gone. Box-local paths live in the cache's local_file(path → midhash, size, mtime); several paths with the same midhash are the duplicates.

Earlier superseded points, kept for the record: the proposed reconcileCanonicalCIDLocked never shipped; cids never was a Redis SET of <algo>:<value> tokens; Redis itself went in v1.1.0 (2026-10). Text below marked (proposal) or (v1.x) describes those shapes, not v2.0.0.

Motivation (history)

Before UUID rids, every metadata entry in Redis was keyed by a content hash, specifically midhash256:

file:midhash256:abc.../filePath
file:midhash256:abc.../title
file:midhash256:abc.../tmdb/poster

Other digests (sha256, IPFS CIDs, BitTorrent infohashes) lived as fields inside that entry (cid_sha256, cid_ipfs). Two problems:

1. midhash256 was privileged by accident. It was the key, while every other CID was a secondary field. Hash-algorithm changes, or new digest types added by plugins, forced rewrites of every root. The privilege itself was the issue, not which hash held it.
2. CID lookup was O(n). "Given any CID, return the metadata" scanned every entry (storage/client.go::LookupPathByCID). Unacceptable for a public API, especially when peers query by any CID variant.

The requirement: any CID a client knows must resolve to that content's metadata in O(1), and the storage layout must not bias toward any digest type. The claims model adds a second one: what is signed must name something every peer agrees on — a rid minted at random on one box does not qualify, a cid does.

Goal

GET /api/meta/<cid>  →  full (merged) metadata document for that content

<cid> may be any supported type (midhash, sha256, IPFS, btih, sha3, future additions), all treated symmetrically.

Design (v2.0.0)

Votes name cids; a rid names a group

votes (signed, travel)                 derived (local, persistent: truth cid_group)
(A, M, title)   = "The Matrix"         M → R1
(A, M, cids/Q)  = "true"  ← alias vote  Q → R1
(B, Q, title)   = "Matrix, The"
(B, Q, sizeByte)= "734003200"          record R1 = every vote on M or Q, merged per key

• First-seen cid, no link → new group, fresh UUIDv7 rid.
• cids/<other> vote on X → X and <other> join one group.
• Link between two existing groups → the group with the older rid survives; the other's cids move to it; the losing rid gets a del event for each of its keys, the survivor set events. No rid_alias: consumers already handle deletes, and cids re-resolve.
• Reading record R = every vote on every cid of R's group, merged per key (one signer counts once per key), precomputed into the cache's file:<rid>/<field> rows (the v1 key layout, now derived).
• Retracting a cids/<other> link does not split the group (phase 1): the tombstone drops out of the merge, cid_group keeps both cids under one rid. In the record view, membership is synthesised from cid_group (cids/<cid> = "true" for every cid of the group).

Why keep a rid at all, if votes never mention it: local consumers (meta-sort, meta-fuse, meta-stremio) and the merged view need one stable name per record, and a group's cid set grows over time. Why cid_group lives in truth although it could be recomputed from cids/* votes: the rid names are minted at random and consumers hold them, so a recomputation would rename every record.

Why UUIDv7 (still valid)

• 128 bits: collision-free under distributed minting; no allocator. Each box minting its own rids for the same content is now the normal case, not a corner case — which is exactly why rids never travel.
• Time-sortable prefix: rids minted in a window sort together (k >= 'file:01JKR8' AND k < 'file:01JKR9' on the cache's merged rows), and "older rid survives" on a group merge is a plain string comparison.
• Monotonic-ish: adjacent rids cluster in the cache's file:<rid>/… key range.

Encoded as 26-char Crockford Base32 (ULID-compatible) for compactness and case-insensitive matching. Migrated records kept their v1.x UUIDs as rids.

The cids key-set

cids/<bareCid> = "true", one claim per member, by whichever signer knows the link. A key-set merges conflict-free: a member is present while at least one vote supports it, and removal is a signed retraction. That is the reason the proposal's Redis SET was reversed before shipping, and it is what lets the claims model treat cids with no special merge code.

Box-local file facts (filePath, size, mtimeNano, midhash) are not claims — meta-core cannot sign and the disk can always be rescanned. The watcher records them in local_file and puts the midhash into cid_group; the record view joins them in unsigned. When several local paths hold the same content, filePath is the lexicographically first one.

Canonical CID selection (unchanged)

The canonical CID is a label, not a key, so nothing external breaks when it changes. It is computed on every read from the group's cids:

1. IPFS CID (any variant)
2. SHA-256 / SHA-3 family
3. BitTorrent infohash (v2 preferred over v1)
4. midhash256

Ranked by utility for external retrieval, not cryptographic strength: midhash256 and sha256 are both 256-bit, but only sha256/IPFS CIDs are externally meaningful; btih ranks above midhash because it enables swarm interop. The full ladder (including the custom locator codecs) lives in internal/cid/rank.go, mirrored in meta-search / meta-share (scripts/check-cid-vectors.sh).

Lifecycles (v2.0.0)

File arrival

1. Watcher sees /files/watch/foo.mkv, computes its midhash M.
2. local_file: /files/watch/foo.mkv → (M, size, mtime).
3. cid_group: M already grouped → nothing; otherwise M → fresh rid.
4. Plugins process the file by cid M (the rid is never disclosed to them)
   and sign claims about M with their own keys.
5. The full-hash plugin computes sha256 S and signs (plugin, M, cids/S) = "true".
   → S joins M's group (or the two groups merge, older rid surviving).
   → The next read derives canonical_cid = S, because it outranks the midhash.

A second path with the same midhash is just another local_file row: same cid, same group, same record; the record's filePath is the lexicographically first of the paths.

File deletion

The watcher removes the path's local_file row; the record view loses that filePath. Claims are signed truth from their writers and are not deleted by meta-core (the store is permanent and additive, docs/project-architecture/metacore-as-permanent-store.md).

(v1.x) Deletion promoted a duplicate to filePath, or called DeleteRoot, which removed every cid:* alias and every file:<uuid>/* key.

Public API

GET /api/meta/<cid>          → 200 merged document { "filePath", "canonical_cid", "cids": [...], ... }
                               (optional ?as=<uid>: the viewer's own votes win)
                             → 404 if no such cid
GET /api/meta/<cid>/claims   → {rid, cids, claims}: every live signed claim of the cid's
                               group, all signers (?key=, ?uid=); 404 if no such cid
GET /api/metadata/<rid>/claims → the same, addressed by rid (local consumers)

The rid is not part of the cid-addressed contract and never appears in a claim. It is visible only to local consumers that list or address records by rid (/api/metadata/{rid}, /api/metadata/{rid}/claims, hash-ids, SSE event keys; the claims routes also echo it as rid), and may change on a group merge (delete event, then re-resolve by cid).

Auth: /api/meta/<cid> bypasses the perimeter like /api/file/<cid> — it is the metadata twin of the content endpoint.

Cross-service impact (v2.0.0)

• Writers (plugins, self-publishing feeders, gateway core, meta-share, meta-search): address claims by cid only and sign them; they never resolve a rid.
• meta-fuse, meta-stremio: unchanged — merged reads and SSE keep their shape; a group merge reaches them as ordinary del + set events.
• meta-search / meta-share: announce the derived canonical cid (each mirrors the Go rank) and relay raw claims; every cid of the group travels as cids/* claims, so a peer querying by any digest gets a hit.

Migration (history)

The v1.0 sweep below was never run. The alpha shipped behind a clean-wipe gate. The v1.x → v2.0.0 conversion of watch.nsl.sh is a one-off tool outside the service image, packages/meta-core/tools/claims-migrate (canonical cid by rank per record; cid: aliases → cid_group, rids keep their UUIDs, and the cid: index wins when a record's cids/* member disagrees; fields re-signed by a throwaway legacy key); every other box starts empty. Implementation basis §11.

(proposal) One-time sweep over file:__index__: mint a uuid, AddAlias the old hashID and every cid_* field, RENAME every property under the new root, reconcile canonical, swap the index entry; crash-recoverable by detecting which step completed.

Decisions recorded

Question	Decision	Notes
When is the rid minted?	When a cid is first seen with no link to a known group	v1.x: at first hash of a watched file. Now any first-seen cid (watcher midhash or a claim about a new cid) mints one
Rid format	UUIDv7, 26-char Crockford Base32	Time-sortable, 128-bit, distributed-safe
Rid visible externally?	No — never signed, never on the swarm/gateway wire	Local HTTP consumers may hold it
What do signed votes name?	A cid	Every peer agrees on a cid; rids differ per box
Record identity	A group of cids (cid_group, in truth)	Linked by cids/<other> votes
Group merge	Older rid survives; losing rid → del events	No rid_alias (out of scope for phase 1)
Duplicate paths	local_file rows sharing a midhash (cache)	v1.x: duplicates SET, then dup:<uuid>/<path>
Canonical CID rank	IPFS > sha-family > btih > midhash, derived on read	Utility-weighted; never stored
cids shape	Key-set of bare CIDs, one claim per member	Merges with the generic rule; a SET would not
Reverse index	cid_group(cid PK → rid) in truth, copied to cache cid:<cid> rows for reads	v1.x: flat cid:<cid> keys were the only copy
Unlinking	A retracted cids/* link does not split a group	Phase 1

Open questions

• Discovery-time rid minting. Minting at discovery (before hashing) would give two same-content paths two rids until hashing links them. The claims model already handles that case — it is an ordinary group merge — so this is now a policy choice, not a missing mechanism.
• IPFS gateway endpoint. Should meta-core expose /ipfs/<cid> for external IPFS clients? Adjacent, not part of this design.
• meta-share announce strategy. Settled: the canonical cid addresses the record, every group cid travels as a cids/* claim.
• Keyspace-notification consumers. Obsolete since Redis left.

Non-goals

• Not a content-replication or pinning system. A cids/<ipfs-cid> vote pins nothing; it states "hashing these bytes with IPFS's chunker gives this CID."
• Not a history. ts versions each (uid, cid, key) and the newest vote replaces the older one; only tombstones are kept, to block replay.
• Not strongly consistent across the two databases: the cache is applied after the truth commit. POST /api/claims waits for the apply before answering, so the writer reads its own write; another reader may briefly miss a just-accepted claim. It is behind, never wrong.