v2.0.0: claims model - signed MMUR2 votes in a truth DB, merged view in a rebuildable cache DB; POST /api/claims, claims and merge routes, unsigned writes answer 410
Build and Push Docker Image / build (push) Successful in 5m1s
Build and Push Docker Image / build (push) Successful in 5m1s
This commit is contained in:
Worph
+202
-295
@@ -2,368 +2,275 @@
|
||||
|
||||
## Status
|
||||
|
||||
**Status:** Implemented (2026 alpha clean-wipe gate landed). Kept for design rationale.
|
||||
**Status:** Design record. Implemented in 2026 alpha (clean-wipe gate);
|
||||
**reframed by meta-core v2.0.0 (claims model)** — see
|
||||
[`docs/implementation/metacore-claims-model.md`](../../../docs/implementation/metacore-claims-model.md)
|
||||
§4. This doc keeps the *why* behind UUID rids and CID symmetry; it is not
|
||||
the schema reference. For what is stored, see
|
||||
[metadata-storage-structure.md](./metadata-storage-structure.md) and
|
||||
`METADATA_KEYS.md` §2/§14.13.
|
||||
|
||||
> ⚠ **Two parts of this proposal were superseded before shipping.** This doc
|
||||
> is the design record; it is *not* the schema reference. For what is actually
|
||||
> stored, see `METADATA_KEYS.md` §2/§14.13 and
|
||||
> [metadata-storage-structure.md](./metadata-storage-structure.md).
|
||||
>
|
||||
> 1. **`canonical_cid` is not a stored key.** The proposed reconciler
|
||||
> (`reconcileCanonicalCIDLocked`) was never kept — a stored scalar isn't
|
||||
> conflict-free to merge across peers. The canonical CID is **derived on
|
||||
> read** by ranking the `cids` members (`internal/cid/rank.go` →
|
||||
> `cid.Better`), and appears only in the `GET /api/meta/{cid}` response.
|
||||
> 2. **`cids` is a key-set, not a Redis SET, and holds bare CIDs, not
|
||||
> `<algo>:<value>` tokens.** One flat STRING per member:
|
||||
> `file:<uuid>/cids/<bareCid>` = `"true"`. The reverse index is likewise
|
||||
> `cid:<bareCid>` → uuid. A CIDv1's multicodec already names its hash
|
||||
> function, so the `<algo>:` prefix was redundant.
|
||||
>
|
||||
> 3. **The store is no longer Redis (2026-10).** The same keys now live in
|
||||
> one string→string table of an embedded SQLite file; Redis SETs became
|
||||
> one key per member (`file:__index__` → `idx:roots/<uuid>`,
|
||||
> `file:<uuid>/duplicates` → `dup:<uuid>/<path>`). Redis types named
|
||||
> below are the shape at the time.
|
||||
>
|
||||
> Text below that references either shape describes the *proposal*, not the
|
||||
> code. The rank ordering and everything else did ship as written.
|
||||
What still holds, unchanged since the original proposal:
|
||||
|
||||
## Motivation
|
||||
- **No CID is privileged.** Any known CID resolves to its record; all hash
|
||||
types are symmetric.
|
||||
- **Rids are UUIDv7**, 26-char Crockford Base32, **minted locally** with no
|
||||
central allocator.
|
||||
- **`canonical_cid` is derived on read** by rank (`internal/cid/rank.go` →
|
||||
`cid.Better`), never stored.
|
||||
- **`cids` is a key-set of bare CIDs** (`cids/<bareCid>` = `"true"`), no
|
||||
`<algo>:` prefix — a CIDv1's multicodec already names the hash function.
|
||||
|
||||
Today, every metadata entry in Redis is keyed by a content hash, specifically
|
||||
midhash256:
|
||||
What changed in v2.0.0:
|
||||
|
||||
> 1. **The rid is a hidden local name for a group of cids.** Metadata is no
|
||||
> longer stored under `file:<uuid>/<field>`. Every value is a signed claim
|
||||
> about a **cid** (`claim(uid, cid, key) → value`); `cid_group(cid → rid)`
|
||||
> says which cids form one record. The rid is **derived from cid groups,
|
||||
> never signed and never on the wire** (swarm, gateway); each box mints its
|
||||
> own.
|
||||
> 2. **The `cid:<cid>` reverse index is gone**, replaced by the `cid_group`
|
||||
> table (still a primary-key lookup, so still O(1)-class).
|
||||
> 3. **meta-core no longer writes `cids/*` members.** They are signed votes
|
||||
> from the writers that learn a cid (e.g. the full-hash plugin); a non-null
|
||||
> `cids/<other>` vote links two groups. Only the watcher's midhash enters
|
||||
> `cid_group` directly.
|
||||
> 4. **`duplicates` / `dup:<uuid>/<path>` are gone.** Box-local paths live in
|
||||
> the cache's `local_file(path → midhash, size, mtime)`; several paths with
|
||||
> the same midhash are the duplicates.
|
||||
>
|
||||
> Earlier superseded points, kept for the record: the proposed
|
||||
> `reconcileCanonicalCIDLocked` never shipped; `cids` never was a Redis SET
|
||||
> of `<algo>:<value>` tokens; Redis itself went in v1.1.0 (2026-10). Text
|
||||
> below marked *(proposal)* or *(v1.x)* describes those shapes, not v2.0.0.
|
||||
|
||||
## Motivation (history)
|
||||
|
||||
Before UUID rids, every metadata entry in Redis was keyed by a content hash,
|
||||
specifically midhash256:
|
||||
|
||||
```
|
||||
file:midhash256:abc.../filePath
|
||||
file:midhash256:abc.../title
|
||||
file:midhash256:abc.../fileinfo/duration
|
||||
file:midhash256:abc.../tmdb/poster
|
||||
...
|
||||
```
|
||||
|
||||
Other digests (sha256, IPFS CIDs, BitTorrent infohashes) live as fields
|
||||
*inside* that entry:
|
||||
Other digests (sha256, IPFS CIDs, BitTorrent infohashes) lived as fields
|
||||
*inside* that entry (`cid_sha256`, `cid_ipfs`). Two problems:
|
||||
|
||||
```
|
||||
file:midhash256:abc.../cid_sha256 = sha256:def...
|
||||
file:midhash256:abc.../cid_ipfs = bafy...
|
||||
```
|
||||
1. **midhash256 was privileged by accident.** It *was* the key, while every
|
||||
other CID was a secondary field. Hash-algorithm changes, or new digest
|
||||
types added by plugins, forced rewrites of every root. The privilege
|
||||
itself was the issue, not which hash held it.
|
||||
2. **CID lookup was O(n).** "Given any CID, return the metadata" scanned
|
||||
every entry (`storage/client.go::LookupPathByCID`). Unacceptable for a
|
||||
public API, especially when peers query by any CID variant.
|
||||
|
||||
Two problems with this:
|
||||
|
||||
1. **midhash256 is privileged by accident.** It IS the key, while every other
|
||||
CID is a secondary field. There's nothing intrinsically special about
|
||||
midhash256 — it's just the first hash computed. Hash-algorithm changes, or
|
||||
new digest types added by plugins, force rewrites of every root. The
|
||||
"promotion" workarounds people reach for (re-keying entries when a stronger
|
||||
hash arrives) solve the wrong problem; the privilege itself is the issue.
|
||||
|
||||
2. **CID lookup is O(n).** Resolving "given any CID, return that file's
|
||||
metadata" currently scans every indexed entry (see
|
||||
`storage/client.go::LookupPathByCID`, which acknowledges this in a
|
||||
comment). Acceptable for admin tooling, unacceptable for a public API
|
||||
surface — especially when peers on meta-share may query by any CID variant.
|
||||
|
||||
The product requirement is clear: **any CID a client knows about must resolve
|
||||
to that file's metadata in O(1).** IPFS-compatible CIDs are the externally
|
||||
meaningful identity for content retrieval; the internal storage layout should
|
||||
not bias toward any one digest type.
|
||||
The requirement: **any CID a client knows must resolve to that content's
|
||||
metadata in O(1)**, and the storage layout must not bias toward any digest
|
||||
type. The claims model adds a second one: **what is signed must name
|
||||
something every peer agrees on** — a rid minted at random on one box does
|
||||
not qualify, a cid does.
|
||||
|
||||
## Goal
|
||||
|
||||
A single externally-meaningful contract:
|
||||
|
||||
```
|
||||
GET /api/meta/<cid> → full metadata document for that file
|
||||
GET /api/meta/<cid> → full (merged) metadata document for that content
|
||||
```
|
||||
|
||||
Where `<cid>` may be any supported type (midhash256, sha256, IPFS, btih,
|
||||
sha3, future additions). All CID types are treated symmetrically. No CID type
|
||||
is "the key."
|
||||
`<cid>` may be any supported type (midhash, sha256, IPFS, btih, sha3, future
|
||||
additions), all treated symmetrically.
|
||||
|
||||
## Design
|
||||
## Design (v2.0.0)
|
||||
|
||||
### Two-tier key structure
|
||||
|
||||
The root key becomes an opaque internal UUID; CIDs are aliases that point to
|
||||
it.
|
||||
### Votes name cids; a rid names a group
|
||||
|
||||
```
|
||||
file:<uuid>/<property...> ← canonical storage (UUIDv7, never disclosed externally)
|
||||
cid:<cid> → <uuid> ← reverse index (one Redis key per known CID)
|
||||
votes (signed, travel) derived (local, persistent: truth cid_group)
|
||||
(A, M, title) = "The Matrix" M → R1
|
||||
(A, M, cids/Q) = "true" ← alias vote Q → R1
|
||||
(B, Q, title) = "Matrix, The"
|
||||
(B, Q, sizeByte)= "734003200" record R1 = every vote on M or Q, merged per key
|
||||
```
|
||||
|
||||
The metadata keyspace under `file:<uuid>/` remains open and flat, exactly as
|
||||
today — anything that wrote `file:<midhash>/fileinfo/duration` now writes
|
||||
`file:<uuid>/fileinfo/duration`. Only the root identifier changes.
|
||||
- **First-seen cid, no link** → new group, fresh UUIDv7 rid.
|
||||
- **`cids/<other>` vote on X** → X and `<other>` join one group.
|
||||
- **Link between two existing groups** → the group with the **older rid
|
||||
survives**; the other's cids move to it; the losing rid gets a `del` event
|
||||
for each of its keys, the survivor `set` events. No `rid_alias`: consumers
|
||||
already handle deletes, and cids re-resolve.
|
||||
- **Reading record R** = every vote on every cid of R's group, merged per key
|
||||
(one signer counts once per key), precomputed into the cache's
|
||||
`file:<rid>/<field>` rows (the v1 key layout, now derived).
|
||||
- **Retracting a `cids/<other>` link does not split the group** (phase 1):
|
||||
the tombstone drops out of the merge, `cid_group` keeps both cids under one
|
||||
rid. In the record view, membership is synthesised from `cid_group`
|
||||
(`cids/<cid> = "true"` for every cid of the group).
|
||||
|
||||
**Example layout:**
|
||||
Why keep a rid at all, if votes never mention it: local consumers (meta-sort,
|
||||
meta-fuse, meta-stremio) and the merged view need one stable name per record,
|
||||
and a group's cid set grows over time. Why `cid_group` lives in **truth**
|
||||
although it could be recomputed from `cids/*` votes: the rid names are minted
|
||||
at random and consumers hold them, so a recomputation would rename every
|
||||
record.
|
||||
|
||||
```
|
||||
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/filePath = /watch/Inception.mkv
|
||||
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/size = 4500000000
|
||||
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/mtime = 1729872000000000000
|
||||
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/cids/bagacbabaec… = "true" ← midhash (record address)
|
||||
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/cids/bafybeigd… = "true" ← sha2-256
|
||||
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/duplicates = SET{/watch/dup1.mkv, /watch/dup2.mkv}
|
||||
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/title = "Inception"
|
||||
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/fileinfo/duration = 7200.5
|
||||
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/tmdb/poster = bafy...
|
||||
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/subtitle/eng = ...
|
||||
### Why UUIDv7 (still valid)
|
||||
|
||||
cid:bagacbabaec… → 01JKR8XW5T4QMV7AHNJ2DEFGHK
|
||||
cid:bafybeigd… → 01JKR8XW5T4QMV7AHNJ2DEFGHK
|
||||
```
|
||||
- **128 bits**: collision-free under distributed minting; no allocator. Each
|
||||
box minting its own rids for the same content is now the normal case, not a
|
||||
corner case — which is exactly why rids never travel.
|
||||
- **Time-sortable prefix**: rids minted in a window sort together
|
||||
(`k >= 'file:01JKR8' AND k < 'file:01JKR9'` on the cache's merged rows),
|
||||
and "older rid
|
||||
survives" on a group merge is a plain string comparison.
|
||||
- **Monotonic-ish**: adjacent rids cluster in the cache's `file:<rid>/…`
|
||||
key range.
|
||||
|
||||
*(As shipped — the proposal originally wrote `cids` as a Redis SET of
|
||||
`<algo>:<value>` tokens plus a `canonical_cid` scalar. See the Status note.)*
|
||||
Encoded as 26-char Crockford Base32 (ULID-compatible) for compactness and
|
||||
case-insensitive matching. Migrated records kept their v1.x UUIDs as rids.
|
||||
|
||||
### UUID format: UUIDv7
|
||||
### The `cids` key-set
|
||||
|
||||
UUIDv7 chosen over UUIDv4 or 64-bit random:
|
||||
`cids/<bareCid>` = `"true"`, one claim per member, by whichever signer knows
|
||||
the link. A key-set merges conflict-free: a member is present while at least
|
||||
one vote supports it, and removal is a signed retraction. That is the reason
|
||||
the proposal's Redis SET was reversed before shipping, and it is what lets
|
||||
the claims model treat `cids` with no special merge code.
|
||||
|
||||
- **128 bits**: collision-free under distributed minting; no central allocator.
|
||||
- **Time-sortable prefix**: `KEYS file:01JKR8*` retrieves entries minted in a
|
||||
given window. Restores some of the debuggability lost by going opaque.
|
||||
- **Monotonic-ish**: adjacent UUIDs cluster on disk; modestly better SCAN
|
||||
locality.
|
||||
Box-local file facts (`filePath`, size, `mtimeNano`, midhash) are **not**
|
||||
claims — meta-core cannot sign and the disk can always be rescanned. The
|
||||
watcher records them in `local_file` and puts the midhash into `cid_group`;
|
||||
the record view joins them in unsigned. When several local paths hold the
|
||||
same content, `filePath` is the lexicographically first one.
|
||||
|
||||
Encoded as a 26-character Crockford Base32 string (ULID-compatible) for
|
||||
compactness and case-insensitive matching.
|
||||
### Canonical CID selection (unchanged)
|
||||
|
||||
### New root fields owned by meta-core
|
||||
|
||||
The metadata schema is open — any writer can put any key under
|
||||
`file:<uuid>/`. This proposal does not change that. It adds three keys whose
|
||||
sole writer is meta-core itself:
|
||||
|
||||
| Key | Type (as shipped) | Purpose |
|
||||
|-----|------|---------|
|
||||
| `cids/<bareCid>` | STRING `"true"` — a *key-set*, one key per member | All CIDs known for this file, midhash included. Mirror of the `cid:*` reverse index. Source of truth for delete-time GC. **Proposed as a Redis SET of `<algo>:<value>` tokens; shipped as a key-set of bare CIDs** — a key-set merges conflict-free across peers and the multicodec already names the algorithm. |
|
||||
| `duplicates` | Redis SET at the time; since 2026-10 one `dup:<uuid>/<path>` key per path | Additional file paths whose content matches this entry. One UUID = one set of metadata; multiple physical files may share that content. Consumed by meta-dup. |
|
||||
| ~~`canonical_cid`~~ | *(not stored)* | **Dropped.** Was to hold the externally-presented CID. Now derived on read by ranking the `cids` members; surfaced only on `GET /api/meta/{cid}`. |
|
||||
|
||||
Existing meta-core-owned fields (`filePath`, `size`, `mtime`, `midhash256`)
|
||||
keep their names and writers. Plugin-written fields (`tmdb/*`, `fileinfo/*`,
|
||||
`subtitle/<lang>`, …) are unaffected.
|
||||
|
||||
~~Redis SET is a departure from the all-strings convention in the current
|
||||
storage doc, but `file:__index__` is already a SET — so the precedent exists
|
||||
for using sets where they're the right structure.~~ **Reversed for `cids`.**
|
||||
The all-strings convention won: `cids` ships as flat `cids/<cid>` STRING keys
|
||||
so that two peers merging the same record can never disagree about set
|
||||
membership. `duplicates` remained a Redis SET (it is leader-local and never
|
||||
merged across peers) until the 2026-10 SQLite switch, which flattened it to
|
||||
`dup:<uuid>/<path>` keys.
|
||||
|
||||
### Canonical CID selection
|
||||
|
||||
The canonical CID is a label, not a key — so nothing external breaks when it
|
||||
changes. **As shipped it is not written at all**: it is computed on every read
|
||||
from the `cids` members, which makes the "update" cost zero and removes the
|
||||
reconciler entirely.
|
||||
|
||||
Selection rule, highest-wins (this part shipped as designed —
|
||||
`internal/cid/rank.go`):
|
||||
The canonical CID is a label, not a key, so nothing external breaks when it
|
||||
changes. It is computed on every read from the group's cids:
|
||||
|
||||
1. IPFS CID (any variant)
|
||||
2. SHA-256 / SHA-3 family
|
||||
3. BitTorrent infohash (v2 preferred over v1)
|
||||
4. midhash256
|
||||
|
||||
Ranked by **utility for external retrieval**, not by cryptographic strength.
|
||||
Ranked by **utility for external retrieval**, not cryptographic strength:
|
||||
midhash256 and sha256 are both 256-bit, but only sha256/IPFS CIDs are
|
||||
externally meaningful for content addressing. btih ranks above midhash because
|
||||
it enables swarm interop even if it's weaker as a digest.
|
||||
externally meaningful; btih ranks above midhash because it enables swarm
|
||||
interop. The full ladder (including the custom locator codecs) lives in
|
||||
`internal/cid/rank.go`, mirrored in meta-search / meta-share
|
||||
(`scripts/check-cid-vectors.sh`).
|
||||
|
||||
~~Reconciler runs on the leader, recomputes `canonical_cid` whenever
|
||||
`AddAlias` registers a new CID for an entry.~~ **Not shipped** — there is no
|
||||
reconciler. `GetMetadataDocument` folds the members through `cid.Better` on
|
||||
each read.
|
||||
## Lifecycles (v2.0.0)
|
||||
|
||||
## Internal operations
|
||||
|
||||
The meta-core storage interface adds five operations. None are exposed
|
||||
externally — they're internal to meta-core. External callers see only the
|
||||
CID-based API.
|
||||
|
||||
```go
|
||||
// Mint creates a new root with no aliases. Called when the watcher
|
||||
// computes a midhash for a previously-unseen file. Returns the new UUID.
|
||||
Mint(filePath string, size int64, mtime int64) (uuid string, err error)
|
||||
|
||||
// AddAlias registers a CID as resolving to this UUID. Updates both the
|
||||
// cid:<cid> reverse index and the file:<uuid>/cids/<cid> key-set member.
|
||||
// (As shipped: no reconciliation step — canonical is derived on read.)
|
||||
AddAlias(uuid string, cid string) error
|
||||
|
||||
// GetByCID resolves any known CID to its UUID. Returns ("", nil) if unknown.
|
||||
GetByCID(cid string) (uuid string, err error)
|
||||
|
||||
// MergeIfAliasExists is called when the watcher mints a fresh UUID for a
|
||||
// file path but, on hashing, discovers the resulting CID already belongs
|
||||
// to a different UUID. Merges the new (mostly-empty) UUID into the
|
||||
// existing one, transferring filePath into the duplicates set, and
|
||||
// returns the canonical UUID.
|
||||
MergeIfAliasExists(newUUID, cid string) (canonicalUUID string, merged bool, err error)
|
||||
|
||||
// DeleteRoot removes the entire entry, including every cid:* alias and
|
||||
// every file:<uuid>/* key. Used for file deletion events.
|
||||
DeleteRoot(uuid string) error
|
||||
```
|
||||
|
||||
### Lifecycle: file arrival
|
||||
### File arrival
|
||||
|
||||
```
|
||||
1. Watcher sees /watch/foo.mkv created.
|
||||
2. Watcher computes midhash256 → "midhash256:abc...".
|
||||
3. Watcher calls GetByCID("midhash256:abc...").
|
||||
→ uuid_X (existing entry) or "" (new file).
|
||||
|
||||
4a. If existing (uuid_X):
|
||||
- Add "/watch/foo.mkv" to file:<uuid_X>/duplicates.
|
||||
- No new root created.
|
||||
- meta-dup is notified via the existing keyspace event channel.
|
||||
|
||||
4b. If new:
|
||||
- Mint("/watch/foo.mkv", size, mtime) → uuid_Y.
|
||||
- AddAlias(uuid_Y, "midhash256:abc...").
|
||||
- Plugins now process the file, referenced by midhash256:abc...
|
||||
(UUID is never disclosed to them.)
|
||||
|
||||
5. Later, the full-hash plugin computes sha256.
|
||||
- Plugin reports back: "for CID midhash256:abc..., sha256 is sha256:def...".
|
||||
- meta-core resolves midhash256:abc → uuid_Y via GetByCID.
|
||||
- meta-core calls AddAlias(uuid_Y, "sha256:def...").
|
||||
- Nothing else to do: the next read of this entry derives a canonical CID
|
||||
of sha2-256, because it outranks the midhash.
|
||||
1. Watcher sees /files/watch/foo.mkv, computes its midhash M.
|
||||
2. local_file: /files/watch/foo.mkv → (M, size, mtime).
|
||||
3. cid_group: M already grouped → nothing; otherwise M → fresh rid.
|
||||
4. Plugins process the file by cid M (the rid is never disclosed to them)
|
||||
and sign claims about M with their own keys.
|
||||
5. The full-hash plugin computes sha256 S and signs (plugin, M, cids/S) = "true".
|
||||
→ S joins M's group (or the two groups merge, older rid surviving).
|
||||
→ The next read derives canonical_cid = S, because it outranks the midhash.
|
||||
```
|
||||
|
||||
### Lifecycle: file deletion
|
||||
A second path with the same midhash is just another `local_file` row: same
|
||||
cid, same group, same record; the record's `filePath` is the
|
||||
lexicographically first of the paths.
|
||||
|
||||
```
|
||||
1. Watcher sees /watch/foo.mkv deleted.
|
||||
2. Resolve which entry owns this path:
|
||||
a. If /watch/foo.mkv is in some entry's duplicates set:
|
||||
- Remove from set. Done.
|
||||
b. If /watch/foo.mkv is the filePath of some entry:
|
||||
- If duplicates is non-empty: promote one duplicate to filePath.
|
||||
- If duplicates is empty: DeleteRoot(uuid).
|
||||
- Iterates the cids set, deletes each cid:<cid> reverse-index key.
|
||||
- Deletes every file:<uuid>/* key.
|
||||
- Removes uuid from file:__index__.
|
||||
```
|
||||
### File deletion
|
||||
|
||||
The watcher removes the path's `local_file` row; the record view loses that
|
||||
`filePath`. Claims are signed truth from their writers and are not deleted
|
||||
by meta-core (the store is permanent and additive,
|
||||
`docs/project-architecture/metacore-as-permanent-store.md`).
|
||||
|
||||
*(v1.x)* Deletion promoted a duplicate to `filePath`, or called `DeleteRoot`,
|
||||
which removed every `cid:*` alias and every `file:<uuid>/*` key.
|
||||
|
||||
## Public API
|
||||
|
||||
```
|
||||
GET /api/meta/<cid>
|
||||
→ 200 { "filePath": "...", "canonical_cid": "...",
|
||||
"cids": [...], "title": "...", "tmdb": {...}, ... }
|
||||
→ 404 if no such CID
|
||||
|
||||
GET /api/file/<cid> (already exists — implementation now O(1))
|
||||
→ streams file content
|
||||
GET /api/meta/<cid> → 200 merged document { "filePath", "canonical_cid", "cids": [...], ... }
|
||||
(optional ?as=<uid>: the viewer's own votes win)
|
||||
→ 404 if no such cid
|
||||
GET /api/meta/<cid>/claims → {rid, cids, claims}: every live signed claim of the cid's
|
||||
group, all signers (?key=, ?uid=); 404 if no such cid
|
||||
GET /api/metadata/<rid>/claims → the same, addressed by rid (local consumers)
|
||||
```
|
||||
|
||||
The UUID is **not** included in the response. Clients address everything by
|
||||
CID; meta-core's choice of internal identifier remains a private
|
||||
implementation detail and may change.
|
||||
The rid is not part of the cid-addressed contract and never appears in a
|
||||
claim. It is visible only to local consumers that list or address records by
|
||||
rid (`/api/metadata/{rid}`, `/api/metadata/{rid}/claims`, `hash-ids`, SSE event
|
||||
keys; the claims routes also echo it as `rid`), and may change on a
|
||||
group merge (delete event, then re-resolve by cid).
|
||||
|
||||
Auth: `/api/meta/<cid>` follows the same pattern as `/api/file/<cid>` and
|
||||
bypasses the auth perimeter. It's the metadata twin of the content endpoint,
|
||||
and meta-share calls it from federated search responses.
|
||||
Auth: `/api/meta/<cid>` bypasses the perimeter like `/api/file/<cid>` — it is
|
||||
the metadata twin of the content endpoint.
|
||||
|
||||
## Cross-service impact
|
||||
## Cross-service impact (v2.0.0)
|
||||
|
||||
This is a meta-core schema change. Other services interact only through CIDs,
|
||||
so the impact is contained but real.
|
||||
- **Writers** (plugins, self-publishing feeders, gateway core, meta-share,
|
||||
meta-search): address claims by cid only and sign them; they never resolve
|
||||
a rid.
|
||||
- **meta-fuse, meta-stremio**: unchanged — merged reads and SSE keep their
|
||||
shape; a group merge reaches them as ordinary `del` + `set` events.
|
||||
- **meta-search / meta-share**: announce the derived canonical cid (each
|
||||
mirrors the Go rank) and relay raw claims; every cid of the group travels as
|
||||
`cids/*` claims, so a peer querying by any digest gets a hit.
|
||||
|
||||
- **meta-sort**: writers must resolve plugin callbacks to a UUID before
|
||||
writing. The existing `LeaderClient` gains a `getByCID` cache.
|
||||
- **meta-fuse**: VFS rebuild already iterates `file:__index__`; only the
|
||||
ID format changes.
|
||||
- **meta-stremio**: queries by CID; needs no logic change beyond its storage
|
||||
client.
|
||||
- **meta-dup**: gains a cleaner data source — duplicates are now explicit in
|
||||
the `duplicates` set, not implicit in colliding writes.
|
||||
- **meta-search / meta-share**: announce the *derived* canonical CID rather
|
||||
than the midhash — each mirrors the Go rank in `ingest::canonical_cid_from_metadata`
|
||||
rather than reading a stored field. Federated search responses include all
|
||||
CIDs from the `cids` key-set so any peer querying by any digest gets a hit.
|
||||
(Written before the 2026-06 discovery/transport split; discovery is now
|
||||
meta-search's job.)
|
||||
- **Plugins**: no change. They speak CIDs only.
|
||||
## Migration (history)
|
||||
|
||||
## Migration
|
||||
> **The v1.0 sweep below was never run.** The alpha shipped behind a
|
||||
> clean-wipe gate. The v1.x → v2.0.0 conversion of watch.nsl.sh is a
|
||||
> one-off tool outside the service image, `packages/meta-core/tools/claims-migrate`
|
||||
> (canonical cid by rank per record; `cid:` aliases → `cid_group`, rids keep
|
||||
> their UUIDs, and the `cid:` index wins when a record's `cids/*` member
|
||||
> disagrees; fields re-signed by a throwaway legacy key); every other box
|
||||
> starts empty. Implementation basis §11.
|
||||
|
||||
> **Historical — this sweep was never run.** The alpha shipped behind a
|
||||
> clean-wipe gate (Redis was flushed and the library re-ingested), so no
|
||||
> in-place migration code exists. Kept for the record.
|
||||
|
||||
One-time sweep, idempotent, runs on first boot after upgrade:
|
||||
|
||||
```
|
||||
For each hashID in file:__index__ (current shape: midhash256:abc, sha256:def, …):
|
||||
1. Mint a fresh uuid_Y.
|
||||
2. AddAlias(uuid_Y, hashID). The existing primary becomes an alias.
|
||||
3. For each cid_* field on the old entry, AddAlias(uuid_Y, that value).
|
||||
4. RENAME file:<hashID>/<property> → file:<uuid_Y>/<property> for every
|
||||
property (Lua script, one entry at a time; non-atomic across entries,
|
||||
but each entry stays internally consistent).
|
||||
5. Reconcile canonical_cid.
|
||||
6. Remove hashID from file:__index__, add uuid_Y.
|
||||
```
|
||||
|
||||
Crash-recoverable: re-running the sweep finds entries with both old and new
|
||||
roots and resumes by detecting which step completed (presence of
|
||||
`file:<uuid>/cids` indicates step 3 finished; absence of any
|
||||
`file:<hashID>/*` keys indicates step 4 finished).
|
||||
*(proposal)* One-time sweep over `file:__index__`: mint a uuid, `AddAlias`
|
||||
the old hashID and every `cid_*` field, `RENAME` every property under the new
|
||||
root, reconcile canonical, swap the index entry; crash-recoverable by
|
||||
detecting which step completed.
|
||||
|
||||
## Decisions recorded
|
||||
|
||||
| Question | Decision | Notes |
|
||||
|---|---|---|
|
||||
| When is the UUID minted? | At first-hash | Pragmatic for the initial migration. Long-term target: at discovery, fully decoupled from hashing. Tracked as an open question below. |
|
||||
| UUID format | UUIDv7, 26-char Crockford Base32 | Time-sortable, 128-bit, distributed-safe. |
|
||||
| UUID visible externally? | No | Pure meta-core internal. Clients address by CID only. |
|
||||
| Merge semantics | One UUID per content; additional paths into `duplicates` | One file (logical) = one metadata set. Multiple physical paths supported via `duplicates`. |
|
||||
| Canonical CID rank | IPFS > sha-family > btih > midhash | Utility-weighted, not strength-weighted. |
|
||||
| ~~`cids` and `duplicates` field type~~ | ~~Redis SET~~ → **`cids` is a key-set of bare CIDs; `duplicates` stayed a SET** | Reversed before shipping. Flat `cids/<cid>` STRING keys merge conflict-free across peers; a SET does not. See §14.13 of `METADATA_KEYS.md`. |
|
||||
| ~~Stored `canonical_cid`~~ | **Dropped — derived on read** | A stored scalar is not conflict-free to merge. `GetMetadataDocument` ranks the `cids` members per request. |
|
||||
| Reverse-index format | Flat `cid:<cid>` STRING keys | Simple, scan-friendly, no hot-key contention. |
|
||||
| When is the rid minted? | When a cid is first seen with no link to a known group | v1.x: at first hash of a watched file. Now any first-seen cid (watcher midhash or a claim about a new cid) mints one |
|
||||
| Rid format | UUIDv7, 26-char Crockford Base32 | Time-sortable, 128-bit, distributed-safe |
|
||||
| Rid visible externally? | No — never signed, never on the swarm/gateway wire | Local HTTP consumers may hold it |
|
||||
| What do signed votes name? | A cid | Every peer agrees on a cid; rids differ per box |
|
||||
| Record identity | A group of cids (`cid_group`, in truth) | Linked by `cids/<other>` votes |
|
||||
| Group merge | Older rid survives; losing rid → `del` events | No `rid_alias` (out of scope for phase 1) |
|
||||
| Duplicate paths | `local_file` rows sharing a midhash (cache) | v1.x: `duplicates` SET, then `dup:<uuid>/<path>` |
|
||||
| Canonical CID rank | IPFS > sha-family > btih > midhash, derived on read | Utility-weighted; never stored |
|
||||
| `cids` shape | Key-set of bare CIDs, one claim per member | Merges with the generic rule; a SET would not |
|
||||
| Reverse index | `cid_group(cid PK → rid)` in truth, copied to cache `cid:<cid>` rows for reads | v1.x: flat `cid:<cid>` keys were the only copy |
|
||||
| Unlinking | A retracted `cids/*` link does not split a group | Phase 1 |
|
||||
|
||||
## Open questions
|
||||
|
||||
- **Discovery-time UUID minting.** The cleaner long-term model: watcher mints
|
||||
a UUID at discovery (before hashing), then `AddAlias` runs when the midhash
|
||||
arrives. Requires an explicit "reconcile if alias already exists" step
|
||||
because two paths with the same content would each get their own UUID until
|
||||
hashing reveals the collision. Worth doing eventually; not required for
|
||||
this iteration.
|
||||
|
||||
- ~~**meta-share announce strategy.**~~ **Settled.** The record carries *every*
|
||||
member of `cids` on the wire (`Record.cids`) and addresses itself by the
|
||||
single derived canonical (`Record.cid`), so a peer querying by any digest
|
||||
gets a hit without the DHT footprint of announcing each one separately.
|
||||
|
||||
- **IPFS gateway endpoint.** Should meta-core expose `/ipfs/<cid>` as a
|
||||
content-streaming endpoint for external IPFS clients? Adjacent to this
|
||||
design but not part of it.
|
||||
|
||||
- **Keyspace-notification consumers.** Today's parsers (`schema/indexer.go`)
|
||||
handle `file:<type>:<hash>/...` with a special split rule for the colon
|
||||
inside `midhash256:abc`. With UUIDs (no colons), `parseFieldPath` simplifies
|
||||
— but any external consumer subscribed by hashID pattern needs updating.
|
||||
Inventory those before flipping the migration.
|
||||
- **Discovery-time rid minting.** Minting at discovery (before hashing)
|
||||
would give two same-content paths two rids until hashing links them. The
|
||||
claims model already handles that case — it is an ordinary group merge —
|
||||
so this is now a policy choice, not a missing mechanism.
|
||||
- **IPFS gateway endpoint.** Should meta-core expose `/ipfs/<cid>` for
|
||||
external IPFS clients? Adjacent, not part of this design.
|
||||
- ~~**meta-share announce strategy.**~~ Settled: the canonical cid addresses
|
||||
the record, every group cid travels as a `cids/*` claim.
|
||||
- ~~**Keyspace-notification consumers.**~~ Obsolete since Redis left.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- **Not** a content-replication or IPFS-pinning system. Registering an IPFS
|
||||
CID alias does not pin the file anywhere — it only states "if somebody
|
||||
hashes this file with IPFS's chunker, this is the CID they'd get."
|
||||
- **Not** a versioning scheme. Metadata is a snapshot of current state;
|
||||
historical versions are out of scope.
|
||||
- **Not** a strong-consistency system. Reverse-index writes happen under the
|
||||
leader lock; reads from followers may briefly miss a freshly-added alias.
|
||||
- **Not** a content-replication or pinning system. A `cids/<ipfs-cid>` vote
|
||||
pins nothing; it states "hashing these bytes with IPFS's chunker gives this
|
||||
CID."
|
||||
- **Not** a history. `ts` versions each `(uid, cid, key)` and the newest vote
|
||||
replaces the older one; only tombstones are kept, to block replay.
|
||||
- **Not** strongly consistent across the two databases: the cache is applied
|
||||
after the truth commit. `POST /api/claims` waits for the apply before
|
||||
answering, so the writer reads its own write; another reader may briefly
|
||||
miss a just-accepted claim. It is behind, never wrong.
|
||||
|
||||
Reference in New Issue
Block a user