meta-core/docs/uuid-rooted-metadata.md

# UUID-Rooted Metadata with CID Aliasing

## Status

**Status:** Design record. Implemented in 2026 alpha (clean-wipe gate);
**reframed by meta-core v2.0.0 (claims model)** — see
[`docs/implementation/metacore-claims-model.md`](../../../docs/implementation/metacore-claims-model.md)
§4. This doc keeps the *why* behind UUID rids and CID symmetry; it is not
the schema reference. For what is stored, see
[metadata-storage-structure.md](./metadata-storage-structure.md) and
`METADATA_KEYS.md` §2/§14.13.

What still holds, unchanged since the original proposal:

- **No CID is privileged.** Any known CID resolves to its record; all hash
  types are symmetric.
- **Rids are UUIDv7**, 26-char Crockford Base32, **minted locally** with no
  central allocator.
- **`canonical_cid` is derived on read** by rank (`internal/cid/rank.go` →
  `cid.Better`), never stored.
- **`cids` is a key-set of bare CIDs** (`cids/<bareCid>` = `"true"`), no
  `<algo>:` prefix — a CIDv1's multicodec already names the hash function.

What changed in v2.0.0:

> 1. **The rid is a hidden local name for a group of cids.** Metadata is no
>    longer stored under `file:<uuid>/<field>`. Every value is a signed claim
>    about a **cid** (`claim(uid, cid, key) → value`); `cid_group(cid → rid)`
>    says which cids form one record. The rid is **derived from cid groups,
>    never signed and never on the wire** (swarm, gateway); each box mints its
>    own.
> 2. **The `cid:<cid>` reverse index is gone**, replaced by the `cid_group`
>    table (still a primary-key lookup, so still O(1)-class).
> 3. **meta-core no longer writes `cids/*` members.** They are signed votes
>    from the writers that learn a cid (e.g. the full-hash plugin); a non-null
>    `cids/<other>` vote links two groups. Only the watcher's midhash enters
>    `cid_group` directly.
> 4. **`duplicates` / `dup:<uuid>/<path>` are gone.** Box-local paths live in
>    the cache's `local_file(path → midhash, size, mtime)`; several paths with
>    the same midhash are the duplicates.
>
> Earlier superseded points, kept for the record: the proposed
> `reconcileCanonicalCIDLocked` never shipped; `cids` never was a Redis SET
> of `<algo>:<value>` tokens; Redis itself went in v1.1.0 (2026-10). Text
> below marked *(proposal)* or *(v1.x)* describes those shapes, not v2.0.0.

## Motivation (history)

Before UUID rids, every metadata entry in Redis was keyed by a content hash,
specifically midhash256:

```
file:midhash256:abc.../filePath
file:midhash256:abc.../title
file:midhash256:abc.../tmdb/poster
```

Other digests (sha256, IPFS CIDs, BitTorrent infohashes) lived as fields
*inside* that entry (`cid_sha256`, `cid_ipfs`). Two problems:

1. **midhash256 was privileged by accident.** It *was* the key, while every
   other CID was a secondary field. Hash-algorithm changes, or new digest
   types added by plugins, forced rewrites of every root. The privilege
   itself was the issue, not which hash held it.
2. **CID lookup was O(n).** "Given any CID, return the metadata" scanned
   every entry (`storage/client.go::LookupPathByCID`). Unacceptable for a
   public API, especially when peers query by any CID variant.

The requirement: **any CID a client knows must resolve to that content's
metadata in O(1)**, and the storage layout must not bias toward any digest
type. The claims model adds a second one: **what is signed must name
something every peer agrees on** — a rid minted at random on one box does
not qualify, a cid does.

## Goal

```
GET /api/meta/<cid>  →  full (merged) metadata document for that content
```

`<cid>` may be any supported type (midhash, sha256, IPFS, btih, sha3, future
additions), all treated symmetrically.

## Design (v2.0.0)

### Votes name cids; a rid names a group

```
votes (signed, travel)                 derived (local, persistent: truth cid_group)
(A, M, title)   = "The Matrix"         M → R1
(A, M, cids/Q)  = "true"  ← alias vote  Q → R1
(B, Q, title)   = "Matrix, The"
(B, Q, sizeByte)= "734003200"          record R1 = every vote on M or Q, merged per key
```

- **First-seen cid, no link** → new group, fresh UUIDv7 rid.
- **`cids/<other>` vote on X** → X and `<other>` join one group.
- **Link between two existing groups** → the group with the **older rid
  survives**; the other's cids move to it; the losing rid gets a `del` event
  for each of its keys, the survivor `set` events. No `rid_alias`: consumers
  already handle deletes, and cids re-resolve.
- **Reading record R** = every vote on every cid of R's group, merged per key
  (one signer counts once per key), precomputed into the cache's
  `file:<rid>/<field>` rows (the v1 key layout, now derived).
- **Retracting a `cids/<other>` link does not split the group** (phase 1):
  the tombstone drops out of the merge, `cid_group` keeps both cids under one
  rid. In the record view, membership is synthesised from `cid_group`
  (`cids/<cid> = "true"` for every cid of the group).

Why keep a rid at all, if votes never mention it: local consumers (meta-sort,
meta-fuse, meta-stremio) and the merged view need one stable name per record,
and a group's cid set grows over time. Why `cid_group` lives in **truth**
although it could be recomputed from `cids/*` votes: the rid names are minted
at random and consumers hold them, so a recomputation would rename every
record.

### Why UUIDv7 (still valid)

- **128 bits**: collision-free under distributed minting; no allocator. Each
  box minting its own rids for the same content is now the normal case, not a
  corner case — which is exactly why rids never travel.
- **Time-sortable prefix**: rids minted in a window sort together
  (`k >= 'file:01JKR8' AND k < 'file:01JKR9'` on the cache's merged rows),
  and "older rid
  survives" on a group merge is a plain string comparison.
- **Monotonic-ish**: adjacent rids cluster in the cache's `file:<rid>/…`
  key range.

Encoded as 26-char Crockford Base32 (ULID-compatible) for compactness and
case-insensitive matching. Migrated records kept their v1.x UUIDs as rids.

### The `cids` key-set

`cids/<bareCid>` = `"true"`, one claim per member, by whichever signer knows
the link. A key-set merges conflict-free: a member is present while at least
one vote supports it, and removal is a signed retraction. That is the reason
the proposal's Redis SET was reversed before shipping, and it is what lets
the claims model treat `cids` with no special merge code.

Box-local file facts (`filePath`, size, `mtimeNano`, midhash) are **not**
claims — meta-core cannot sign and the disk can always be rescanned. The
watcher records them in `local_file` and puts the midhash into `cid_group`;
the record view joins them in unsigned. When several local paths hold the
same content, `filePath` is the lexicographically first one.

### Canonical CID selection (unchanged)

The canonical CID is a label, not a key, so nothing external breaks when it
changes. It is computed on every read from the group's cids:

1. IPFS CID (any variant)
2. SHA-256 / SHA-3 family
3. BitTorrent infohash (v2 preferred over v1)
4. midhash256

Ranked by **utility for external retrieval**, not cryptographic strength:
midhash256 and sha256 are both 256-bit, but only sha256/IPFS CIDs are
externally meaningful; btih ranks above midhash because it enables swarm
interop. The full ladder (including the custom locator codecs) lives in
`internal/cid/rank.go`, mirrored in meta-search / meta-share
(`scripts/check-cid-vectors.sh`).

## Lifecycles (v2.0.0)

### File arrival

```
1. Watcher sees /files/watch/foo.mkv, computes its midhash M.
2. local_file: /files/watch/foo.mkv → (M, size, mtime).
3. cid_group: M already grouped → nothing; otherwise M → fresh rid.
4. Plugins process the file by cid M (the rid is never disclosed to them)
   and sign claims about M with their own keys.
5. The full-hash plugin computes sha256 S and signs (plugin, M, cids/S) = "true".
   → S joins M's group (or the two groups merge, older rid surviving).
   → The next read derives canonical_cid = S, because it outranks the midhash.
```

A second path with the same midhash is just another `local_file` row: same
cid, same group, same record; the record's `filePath` is the
lexicographically first of the paths.

### File deletion

The watcher removes the path's `local_file` row; the record view loses that
`filePath`. Claims are signed truth from their writers and are not deleted
by meta-core (the store is permanent and additive,
`docs/project-architecture/metacore-as-permanent-store.md`).

*(v1.x)* Deletion promoted a duplicate to `filePath`, or called `DeleteRoot`,
which removed every `cid:*` alias and every `file:<uuid>/*` key.

## Public API

```
GET /api/meta/<cid>          → 200 merged document { "filePath", "canonical_cid", "cids": [...], ... }
                               (optional ?as=<uid>: the viewer's own votes win)
                             → 404 if no such cid
GET /api/meta/<cid>/claims   → {rid, cids, claims}: every live signed claim of the cid's
                               group, all signers (?key=, ?uid=); 404 if no such cid
GET /api/metadata/<rid>/claims → the same, addressed by rid (local consumers)
```

The rid is not part of the cid-addressed contract and never appears in a
claim. It is visible only to local consumers that list or address records by
rid (`/api/metadata/{rid}`, `/api/metadata/{rid}/claims`, `hash-ids`, SSE event
keys; the claims routes also echo it as `rid`), and may change on a
group merge (delete event, then re-resolve by cid).

Auth: `/api/meta/<cid>` bypasses the perimeter like `/api/file/<cid>` — it is
the metadata twin of the content endpoint.

## Cross-service impact (v2.0.0)

- **Writers** (plugins, self-publishing feeders, gateway core, meta-share,
  meta-search): address claims by cid only and sign them; they never resolve
  a rid.
- **meta-fuse, meta-stremio**: unchanged — merged reads and SSE keep their
  shape; a group merge reaches them as ordinary `del` + `set` events.
- **meta-search / meta-share**: announce the derived canonical cid (each
  mirrors the Go rank) and relay raw claims; every cid of the group travels as
  `cids/*` claims, so a peer querying by any digest gets a hit.

## Migration (history)

> **The v1.0 sweep below was never run.** The alpha shipped behind a
> clean-wipe gate. The v1.x → v2.0.0 conversion of watch.nsl.sh is a
> one-off tool outside the service image, `packages/meta-core/tools/claims-migrate`
> (canonical cid by rank per record; `cid:` aliases → `cid_group`, rids keep
> their UUIDs, and the `cid:` index wins when a record's `cids/*` member
> disagrees; fields re-signed by a throwaway legacy key); every other box
> starts empty. Implementation basis §11.

*(proposal)* One-time sweep over `file:__index__`: mint a uuid, `AddAlias`
the old hashID and every `cid_*` field, `RENAME` every property under the new
root, reconcile canonical, swap the index entry; crash-recoverable by
detecting which step completed.

## Decisions recorded

| Question | Decision | Notes |
|---|---|---|
| When is the rid minted? | When a cid is first seen with no link to a known group | v1.x: at first hash of a watched file. Now any first-seen cid (watcher midhash or a claim about a new cid) mints one |
| Rid format | UUIDv7, 26-char Crockford Base32 | Time-sortable, 128-bit, distributed-safe |
| Rid visible externally? | No — never signed, never on the swarm/gateway wire | Local HTTP consumers may hold it |
| What do signed votes name? | A cid | Every peer agrees on a cid; rids differ per box |
| Record identity | A group of cids (`cid_group`, in truth) | Linked by `cids/<other>` votes |
| Group merge | Older rid survives; losing rid → `del` events | No `rid_alias` (out of scope for phase 1) |
| Duplicate paths | `local_file` rows sharing a midhash (cache) | v1.x: `duplicates` SET, then `dup:<uuid>/<path>` |
| Canonical CID rank | IPFS > sha-family > btih > midhash, derived on read | Utility-weighted; never stored |
| `cids` shape | Key-set of bare CIDs, one claim per member | Merges with the generic rule; a SET would not |
| Reverse index | `cid_group(cid PK → rid)` in truth, copied to cache `cid:<cid>` rows for reads | v1.x: flat `cid:<cid>` keys were the only copy |
| Unlinking | A retracted `cids/*` link does not split a group | Phase 1 |

## Open questions

- **Discovery-time rid minting.** Minting at discovery (before hashing)
  would give two same-content paths two rids until hashing links them. The
  claims model already handles that case — it is an ordinary group merge —
  so this is now a policy choice, not a missing mechanism.
- **IPFS gateway endpoint.** Should meta-core expose `/ipfs/<cid>` for
  external IPFS clients? Adjacent, not part of this design.
- ~~**meta-share announce strategy.**~~ Settled: the canonical cid addresses
  the record, every group cid travels as a `cids/*` claim.
- ~~**Keyspace-notification consumers.**~~ Obsolete since Redis left.

## Non-goals

- **Not** a content-replication or pinning system. A `cids/<ipfs-cid>` vote
  pins nothing; it states "hashing these bytes with IPFS's chunker gives this
  CID."
- **Not** a history. `ts` versions each `(uid, cid, key)` and the newest vote
  replaces the older one; only tombstones are kept, to block replay.
- **Not** strongly consistent across the two databases: the cache is applied
  after the truth commit. `POST /api/claims` waits for the apply before
  answering, so the writer reads its own write; another reader may briefly
  miss a just-accepted claim. It is behind, never wrong.