v2.0.0: claims model - signed MMUR2 votes in a truth DB, merged view in a rebuildable cache DB; POST /api/claims, claims and merge routes, unsigned writes answer 410
Build and Push Docker Image / build (push) Successful in 5m1s
Build and Push Docker Image / build (push) Successful in 5m1s
This commit is contained in:
Worph
+220
-31
@@ -1,4 +1,14 @@
|
||||
# API-Mediated Access (Redis Lockdown)
|
||||
# API-Mediated Access (Redis Lockdown → Signed Claims)
|
||||
|
||||
> **meta-core v2.0.0 — the claims model.** The implementation basis is
|
||||
> [`docs/implementation/metacore-claims-model.md`](../../../docs/implementation/metacore-claims-model.md)
|
||||
> (meta-root). Where this file and that one disagree, that one wins. The
|
||||
> *access* contract below still holds unchanged — HTTP only, events over
|
||||
> SSE, no database reachable from outside `metacore-app`. The *write*
|
||||
> contract changed: every metadata write is a **signed claim** posted to
|
||||
> `POST /api/claims`, and the legacy unsigned write endpoints answer
|
||||
> `410 Gone`. See ["Write contract — signed claims"](#write-contract--signed-claims-v200)
|
||||
> below.
|
||||
|
||||
## Status
|
||||
|
||||
@@ -69,20 +79,23 @@ we don't fully control.
|
||||
## The architecture in one picture
|
||||
|
||||
```
|
||||
meta-core (only Redis writer + reader)
|
||||
meta-core v2.0.0 (only store writer + reader)
|
||||
┌──────────────────────────────────────┐
|
||||
│ HTTP API surface: │
|
||||
│ POST/PUT/PATCH /api/metadata/* │ ◄── writes
|
||||
│ GET /api/meta/{cid} │ ◄── reads (by CID)
|
||||
│ GET /api/metadata/* │ ◄── reads (by id)
|
||||
│ GET /api/file/{cid} │ ◄── content
|
||||
│ GET /api/events/files │ ◄── SSE stream
|
||||
│ GET /api/events/meta │ ◄── SSE stream
|
||||
│ POST /api/claims │ ◄── writes (signed claims only)
|
||||
│ POST /api/merge │ ◄── stateless merge, no write
|
||||
│ GET /api/meta/{cid} │ ◄── reads (by CID, merged)
|
||||
│ GET /api/meta/{cid}/claims │ ◄── reads (raw signed claims)
|
||||
│ GET /api/metadata/* │ ◄── reads (by rid, merged)
|
||||
│ GET /api/file/{cid} │ ◄── content
|
||||
│ GET /api/events/files │ ◄── SSE stream
|
||||
│ GET /api/events/meta │ ◄── SSE stream (merged changes)
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ Redis (file:*, cid:*, streams) │
|
||||
│ ◄── container-local, no exposed │
|
||||
│ port, no docker-network alias │
|
||||
│ ▼ verify sig → truth → apply │
|
||||
│ truth /meta-core/db/meta-core.db │
|
||||
│ cache /meta-core/db-cache/cache.db │
|
||||
│ ◄── embedded SQLite, no port, │
|
||||
│ no docker-network alias │
|
||||
└──────────────────────────────────────┘
|
||||
▲ ▲
|
||||
│ HTTP │ HTTP SSE
|
||||
@@ -95,9 +108,18 @@ we don't fully control.
|
||||
```
|
||||
|
||||
**The single-writer rule.** meta-core has exclusive write authority over
|
||||
the `file:*`, `cid:*`, `file:__index__` (now `idx:*`), and `meta-core:*` keyspaces. No
|
||||
other service can write — not because of an honoured convention, but
|
||||
because they can't reach Redis at all.
|
||||
its store — in v2.0.0 the two SQLite databases (truth and cache); in the
|
||||
Redis era the `file:*`, `cid:*`, `file:__index__` (later `idx:*`), and
|
||||
`meta-core:*` keyspaces. No other service can write the store — not because
|
||||
of an honoured convention, but because they can't reach it at all.
|
||||
|
||||
**What changed in v2.0.0 is who may *author* metadata.** meta-core is still
|
||||
the only process that touches the database, but it no longer accepts
|
||||
unsigned values: a writer submits signed votes (claims), meta-core verifies
|
||||
each signature and stores the vote. meta-core **never signs a claim and
|
||||
holds no writer key** — a store that could sign for everyone would prove
|
||||
nothing. The signature is the write authentication; there are no
|
||||
per-writer tokens.
|
||||
|
||||
**Stream reads also mediated.** Events go out through SSE endpoints on
|
||||
meta-core's HTTP surface. Consumers don't see the underlying Redis
|
||||
@@ -108,6 +130,147 @@ without breaking every consumer.
|
||||
|
||||
**Redis is private.** No host port mapping, no docker-network alias, no
|
||||
URL in `kv-leader.info`. Discovery returns only meta-core's HTTP base.
|
||||
(Since 2026-10 there is no Redis at all; the SQLite files are reachable only
|
||||
from inside `metacore-app`.)
|
||||
|
||||
## Write contract — signed claims (v2.0.0)
|
||||
|
||||
Every metadata value is a **signed vote**: *signer `uid` says `key = value`
|
||||
about content `cid`* — `claim(uid, cid, key) → value + ts + sig`. The
|
||||
record a reader sees is the **merged view** that meta-core precomputes from
|
||||
the votes; a reader's own votes always win in their view. Full model,
|
||||
schema and merge rules: the
|
||||
[implementation basis](../../../docs/implementation/metacore-claims-model.md)
|
||||
§1–§7.
|
||||
|
||||
### Endpoint table
|
||||
|
||||
| | Endpoint | Behaviour |
|
||||
|---|---|---|
|
||||
| **Write** | `POST /api/claims` | A batch of signed claims (a JSON array, or `{"claims": [...]}`). **The only metadata write path.** |
|
||||
| Removed | `PUT/PATCH/DELETE /meta/{hash}`, `PUT/DELETE /meta/{hash}/{key...}`, `POST /meta/{hash}/_add/{key...}`, `PUT/DELETE /api/metadata/{hashId}`, `PUT /api/metadata/{hashId}/property`, `POST /api/metadata/batch` | `410 Gone`, with a pointer to `POST /api/claims` |
|
||||
| Kept | `PUT/GET/DELETE /api/kv/value` | Service scratch KV (`btmagnet:*`, `searchcov:*`), plain rows of the cache database's `kv` table (any key outside the reserved prefixes `file:`, `cid:`, `idx:`, `dup:`, `ev:`, `evhwm:`, `meta:`, `meta-core:`) — losing it only costs a refetch |
|
||||
| Kept | `POST /api/files/register`, the file watcher | Fill `local_file` (box-local `filePath`, size, mtime, midhash — unsigned) and `cid_group` |
|
||||
| UDL facade | `/api/udl/*` (same paths) | Writes go through the claims write path; reads come from the claims (per-signer index `claim_by_pid`). Records are MMUR2 claims. Shapes: see "UDL facade shapes" below |
|
||||
| Read | `GET /api/meta/{cid}`, `GET /api/metadata/{rid}`, `GET /file/{cid}`, `GET /meta/{hash}`, `GET /api/metadata/hash-ids`, `GET /api/metadata/list`, `POST /api/metadata/search`, `GET /api/files/tuples`, `GET /api/schema`, `GET /api/stats` | Unchanged paths and response shapes, served from the merged view (+ `local_file`); optional `?as=<uid>`. They carry no `nsrc` |
|
||||
| **New** | `GET /api/meta/{cid}/claims` | `{rid, cids, claims}`: the record's rid, the cids of its group and every live signed claim on them (all signers, tombstones excluded). `?key=` keeps one exact key, `?uid=` one signer. `404 unknown_cid` when no record holds the cid |
|
||||
| **New** | `GET /api/metadata/{rid}/claims` | Same body and filters, addressed by rid (a cid is also accepted); `404` for an unknown record. What meta-search ingest and the gateway's corpus search use |
|
||||
| **New** | `POST /api/merge` | Stateless: body `{claims, as?}`; verifies each claim (invalid ones dropped and counted, never a batch error), merges with the store's merge rule, overlays `as`'s own votes (in the request and held by this core for the same cids). Returns `{fields, nsrc, rejected}` — merged `key → value`, `key → agreeing signers`, rejected count. `413` above 10 000 claims. Writes nothing |
|
||||
| Events | `GET /api/events/meta` | Unchanged `{type: set\|del, key, ts}` shape, now emitted for changes of the **merged** view, `key` still the literal `file:<rid>/<field>`; `GET /api/events/files` unchanged. Both streams are rows of the cache `kv` table (`ev:meta:events/…`, `ev:file:events/…`), ~100k entries each |
|
||||
| Admin | `POST /api/metadata/clear` | Wipes both databases (truth and cache) |
|
||||
| Removed | `/api/snapshot/export`, `/api/snapshot/import`, `/api/snapshot/wipe` | No snapshot format in v2.0.0 |
|
||||
|
||||
Endpoints the claims model does not touch keep their behaviour: discovery
|
||||
(`/health`, `/status`, `/leader`, `/urls`, `/neighbors`), the account-key
|
||||
identity service (`/api/identity/*` — its removal is a separate ticket;
|
||||
v2.0.0 adds the `format` field to `sign` / `sign-batch` — `"compact"` or
|
||||
`"der"`, default compact for `"MMUR2\n"` payloads and DER otherwise — and
|
||||
makes `DELETE /api/identity` sign and store a retraction of every live
|
||||
claim of the account before removing it),
|
||||
`/api/file/{cid}/info`, `/file/cid`, `/data/{hash}*`, mounts, watchers and
|
||||
WebDAV.
|
||||
|
||||
### The claim on the wire (JSON form)
|
||||
|
||||
```json
|
||||
{ "cid": "bafy…", "uid": "zQ3s…", "key": "title",
|
||||
"value": "The Matrix", // or {"b64": "…"} for bytes, or null
|
||||
"ts": 1793001234567,
|
||||
"sig": "base64(64 bytes)" }
|
||||
```
|
||||
|
||||
| Field | Meaning |
|
||||
|---|---|
|
||||
| `cid` | The subject: a content CID as the writer knows it (bare CID string, as in `cids/*`). Never a rid — rids are local and never signed |
|
||||
| `uid` | The signer: `"z" + base58btc(33-byte compressed secp256k1 pubkey)` |
|
||||
| `key` | The field (`METADATA_KEYS.md`); a key-set member is inside the key (`genres/Drama`) |
|
||||
| `value` | Text = public plaintext; `{"b64": …}` = bytes, a private (ciphertext) envelope; `null` = retraction |
|
||||
| `ts` | Milliseconds since epoch. It **is** the version: only increases per `(uid, cid, key)` |
|
||||
| `sig` | MMUR2: secp256k1 ECDSA, compact 64 bytes, over SHA-256 of `"MMUR2\n" ‖ canonical CBOR [cid, uid, key, value, ts]` |
|
||||
|
||||
A batch (the `POST /api/claims` body) is a JSON array of these (or
|
||||
`{"claims": [...]}`). Kamilata and
|
||||
the gateway wire carry the same five fields plus `sig`; the signed bytes
|
||||
never change on any hop, and relays (meta-search, a client persisting a
|
||||
gateway hit) forward claims verbatim — they never re-sign.
|
||||
|
||||
### Per-claim statuses
|
||||
|
||||
`POST /api/claims` answers with one status per claim:
|
||||
|
||||
| Status | Meaning |
|
||||
|---|---|
|
||||
| `accepted` | Verified and stored (or a retraction stored as a tombstone) |
|
||||
| `unchanged` | The stored row for `(uid, cid, key)` already has this `ts` and `sig` — **or the same value under any `ts`** (a signer re-asserting what it holds): no verification, no write, and the stored row keeps its older `ts` |
|
||||
| `stale` | The stored `ts` is ≥ the incoming `ts`; ignored (blocks replay of old votes) |
|
||||
| `bad_signature` | The MMUR2 signature does not verify against the `uid`'s public key |
|
||||
| `bad_request` | The claim is malformed |
|
||||
|
||||
The response is `200` with a JSON array of `{status, reason?}`, one entry
|
||||
per claim, in input order, even when some are refused. A batch above 10 000
|
||||
claims answers `413 too_many_claims`; a body that is not claims JSON
|
||||
`400 invalid_json`. The
|
||||
removed write routes answer `410 {"error":"gone","use":"POST /api/claims"}`.
|
||||
`POST /api/claims`, `POST /api/merge` and the two claims read routes are
|
||||
container-to-container routes, not routed through Caddy.
|
||||
|
||||
Rules: every claim is signed — ciphertext and retractions included; an
|
||||
unsigned or badly signed claim is refused. A retraction (`value: null`) is
|
||||
kept as a tombstone, so a peer cannot replay the vote it retracted. Values
|
||||
are stored exactly as signed; normalisation (trim + case-fold, no NFC)
|
||||
happens only inside the merge.
|
||||
|
||||
Accepted claims commit to the truth database in one group-committed
|
||||
transaction; an apply loop then folds them, in commit order, into the
|
||||
cache (the merged `file:<rid>/<field>` rows of its v1-layout `kv` table,
|
||||
FTS, counters, events). `POST /api/claims` answers only once the cache has
|
||||
applied the batch, so the writer reads its own write; another reader may
|
||||
briefly see the previous merged value — the cache is behind, never wrong.
|
||||
|
||||
### `?as=<uid>` — the viewer's overlay
|
||||
|
||||
Every merged read accepts an optional `?as=<uid>`. The merge then puts that
|
||||
signer's own live text vote first for each key (read-time overlay found
|
||||
through the per-signer index, never stored in the shared merged view). Without
|
||||
`?as=` the read returns the shared merge: most distinct signers → oldest
|
||||
`ts` → byte-wise smallest value; ciphertext never merges.
|
||||
|
||||
### Raw claims and client-side merging
|
||||
|
||||
The claims read routes (`GET /api/metadata/{rid}/claims` for meta-search
|
||||
ingest and the gateway's corpus hits, `GET /api/meta/{cid}/claims` by cid)
|
||||
return the raw signed votes of every signer, not a merged map. A client that receives
|
||||
claims from the swarm or a gateway displays them by calling its **local**
|
||||
meta-core's `POST /api/merge` (one merge implementation, in Go), and
|
||||
persists a hit by posting the claims **as received** to `POST /api/claims`.
|
||||
|
||||
### UDL facade shapes
|
||||
|
||||
Claims in the JSON form above (`internal/api/udl.go`):
|
||||
|
||||
| Route | Request | Response |
|
||||
|---|---|---|
|
||||
| `PUT /api/udl/record` | one claim | `200 {accepted: true, status}` for `accepted` / `unchanged`; `409 stale_version` for `stale`; `400` (status as slug) for `bad_signature` / `bad_request` |
|
||||
| `PUT /api/udl/records` | `{records: [claim, …]}` | `200 {results: [{key, accepted, status, error?}, …]}`, input order; `accepted` = `accepted` or `unchanged` |
|
||||
| `GET /api/udl/record?uid=&cid=&key=` | — | `{exists, claim}` — the stored claim, tombstone included; `exists` false when absent or a tombstone |
|
||||
| `GET /api/udl/user/{uid}/key/{key}`, `/user/{uid}/cid/{cid}` | — | `{entries: [claim, …]}` |
|
||||
| `GET /api/udl/user/{uid}/all?after=&limit=` | — | `{entries: [claim, …], nextCid}` |
|
||||
| `GET /api/udl/user/{uid}/count` | — | `{cids}` |
|
||||
| `GET /api/udl/cid/{cid}/users` | — | `{users: [uid, …]}` |
|
||||
| `GET /api/udl/cid/{cid}/aggregate?keys=` | — | `{aggregates: [{key, count, avg, trues, sample_size}]}` |
|
||||
| `GET /api/udl/users/stats` | — | `{users: {uid: {records, cids, keys}}}` for **every signer** with live claims — plugin, feeder, gateway and service keys included. `/api/stats` `udlUsers` is the size of this map |
|
||||
|
||||
### What a writer needs
|
||||
|
||||
Each writing container (every meta-sort plugin, every self-publishing
|
||||
feeder, gateway core, meta-sort, meta-share, meta-search) signs with its
|
||||
**own** key; users sign in the browser. meta-core holds none of these keys.
|
||||
Key configuration per signer: implementation basis §9.
|
||||
|
||||
Box-local file facts (`filePath`, size, `mtimeNano`, midhash) are not
|
||||
claims: the watcher and `POST /api/files/register` write them into the
|
||||
cache's `local_file` table, they are joined into the record view unsigned,
|
||||
and they never appear in `/api/meta/{cid}/claims` or on the swarm.
|
||||
|
||||
## Eventing — HTTP Server-Sent Events
|
||||
|
||||
@@ -263,12 +426,19 @@ seen). The contract above holds unchanged:
|
||||
- Retention is ~100k entries per stream, trimmed every 10k appends.
|
||||
- IDs keep the Redis `<ms>-<seq>` form, so cursors persisted by meta-sort /
|
||||
meta-share / meta-stremio before the switch still compare.
|
||||
- `meta:events` entries (`{type: set|del, key, ts}`) are appended by the
|
||||
storage engine **in the same transaction** as the `file:` key change —
|
||||
`del` only when the key existed, `set` only when the value actually
|
||||
changed (since v1.2.0; an unchanged write is a no-op). The Redis
|
||||
keyspace-notification hop (PSUBSCRIBE in `MetaPublisher`), which could
|
||||
drop events when it lagged, is gone.
|
||||
- `meta:events` entries (`{type: set|del, key, ts}`) describe changes of
|
||||
the **merged** view (v2.0.0): the apply loop emits `set`/`del` (keyed by
|
||||
rid + key) only when a claim actually changed a merged value, in the same
|
||||
cache transaction as that change. A group merge (two cid groups linked by
|
||||
a `cids/*` vote) emits `del` for every key of the losing rid and `set` for
|
||||
the survivor. *History:* v1.x appended them in the same transaction as the
|
||||
`file:` key change; before 2026-10 a Redis keyspace-notification hop
|
||||
(PSUBSCRIBE in `MetaPublisher`) could drop events when it lagged.
|
||||
- **Cache rebuild (v2.0.0).** Events live in the cache database. A cache
|
||||
rebuild (missing cache folder, or catch-up after an unclean shutdown)
|
||||
emits no per-key events; SSE consumers get `event: gap` and take a
|
||||
snapshot, as they do for a trimmed cursor. Event IDs stay time-based, so
|
||||
they keep rising across a rebuild.
|
||||
- **`Last-Event-ID: 0` / `0-0` on `/api/events/meta` returns a snapshot**
|
||||
(since v1.2.0): one `set` per existing record key, then the live tail from
|
||||
the head captured before the snapshot. Only the last snapshot event carries
|
||||
@@ -522,6 +692,10 @@ durability across a meta-core restart).
|
||||
|
||||
### PR B — meta-sort writes via HTTP
|
||||
|
||||
> *History.* PR B moved writes onto the unsigned `/meta/{hash}` family.
|
||||
> In v2.0.0 those routes answer `410 Gone`; writers sign claims and post
|
||||
> them to `POST /api/claims` (see "Write contract — signed claims").
|
||||
|
||||
Swap `RedisClient` for an `HttpKVClient` implementation of the existing
|
||||
`IKVClient` interface in `packages/meta-sort-core`. All writes
|
||||
(`setMetadataFlat`, `setMetadataProperty`, batch updates) become HTTP
|
||||
@@ -603,7 +777,7 @@ metacore-app produces a connection refused. The lock is real.
|
||||
| Backing store for streams? | Redis Streams at the time; SQLite kv rows since 2026-10 | SSE wraps it; the 2026-10 swap proved it — no consumer changed |
|
||||
| Consumer groups / horizontal scale? | Not supported via SSE | Not needed today; revisit if scale ever requires it |
|
||||
| Storage portability? | Not a goal but a free side-effect | Will not be exercised in alpha |
|
||||
| **Canonical write API path?** | **`/meta/{hash}` family** | Has GET/PUT/**PATCH**/DELETE plus `/_add/{key}` and per-property `{key:.*}` operations. `PATCH` calls `MergeMetadataFlat` — matches the partial-update semantics PR B needs. `/api/metadata/{hashId}` stays as the editor-internal surface (no PATCH, no per-property routes) but is not the migration target. |
|
||||
| **Canonical write API path?** | **`POST /api/claims` (v2.0.0)** — signed claims only; every unsigned write route answers `410 Gone` | *History (v1.x):* the **`/meta/{hash}` family** was the canonical write surface. It had GET/PUT/**PATCH**/DELETE plus `/_add/{key}` and per-property `{key:.*}` operations. `PATCH` calls `MergeMetadataFlat` — matches the partial-update semantics PR B needs. `/api/metadata/{hashId}` stays as the editor-internal surface (no PATCH, no per-property routes) but is not the migration target. |
|
||||
| **SSE event-id contract?** | **Opaque to consumers** | It is `<ms>-<seq>` (the Redis Stream entry form, kept after the 2026-10 SQLite switch); consumers MUST treat as opaque so the backing store can be swapped without breaking the wire contract. Documented in the SSE endpoint comment and in this doc. |
|
||||
| **Typed error envelope?** | **`{error, message, retryable}` with stable `error` slug** | See "Error envelope" subsection below. |
|
||||
| **Plugin direct-Redis risk?** | **Confirmed: no plugin uses Redis directly** | Grep across `packages/plugins/` for `ioredis` / `redis.createClient` / `new Redis()` returned empty. Plugins POST results back to meta-sort via `/api/plugins/callback`; meta-sort then writes. PR B picks up the plugin write path transparently. |
|
||||
@@ -611,6 +785,11 @@ metacore-app produces a connection refused. The lock is real.
|
||||
|
||||
## Error envelope
|
||||
|
||||
(`410 Gone` from the removed v1.x write routes carries this envelope with a
|
||||
message pointing to `POST /api/claims`. Per-claim outcomes of
|
||||
`POST /api/claims` are statuses inside the response, not envelopes — see
|
||||
"Per-claim statuses" above.)
|
||||
|
||||
Standard shape for all 4xx / 5xx responses from the metadata API:
|
||||
|
||||
```json
|
||||
@@ -629,6 +808,10 @@ Stable `error` slugs the client should switch on:
|
||||
| `unknown_root` | 404 | no | The hashId / UUID doesn't exist. Caller probably has a stale reference. |
|
||||
| `unknown_cid` | 404 | no | The CID isn't in the reverse index. No retry helps. |
|
||||
| `schema_violation` | 400 | no | Body shape rejected (missing required field, invalid value). Caller bug. |
|
||||
| `invalid_json` | 400 | no | `POST /api/claims` / `POST /api/merge` body is not claims JSON. Caller bug. |
|
||||
| `too_many_claims` | 413 | no | More than 10 000 claims in one `POST /api/claims` or `POST /api/merge`. Split the batch. |
|
||||
| `stale_version` | 409 | yes | `PUT /api/udl/record`: an equal or newer `ts` is stored for this signer, cid and key. Re-read the record, sign with a newer `ts`, retry. |
|
||||
| `gone` | 410 | no | A removed unsigned write route. Body is `{"error":"gone","use":"POST /api/claims"}` (no `message` / `retryable`). |
|
||||
| `storage_unavailable` | 503 | yes | Store transient (Redis at the time; the embedded SQLite store since 2026-10). Caller should retry with backoff. |
|
||||
| `internal` | 500 | yes (cautiously) | Unhandled server-side error. Retry once; escalate if persistent. |
|
||||
|
||||
@@ -643,10 +826,9 @@ import cycle on the api package; consolidating it into a shared
|
||||
|
||||
Mostly answered post-implementation:
|
||||
|
||||
- **Deprecation timeline for `/api/metadata/{hashId}`.** Still open. The
|
||||
editor's React UI continues to call it; the surface itself was untouched
|
||||
by this work. Worth a follow-up to consolidate the editor onto
|
||||
`/meta/{hash}` and delete the parallel routes.
|
||||
- **Deprecation timeline for `/api/metadata/{hashId}`.** Answered by
|
||||
v2.0.0: its `PUT`/`DELETE` (and `PUT …/property`) answer `410 Gone`
|
||||
like every unsigned write; `GET` stays as a merged read.
|
||||
|
||||
- **Initial bootstrap flood on `/api/events/files`.** Resolved in
|
||||
practice: meta-fuse uses `initialCursor: '0-0'` to deliberately
|
||||
@@ -666,8 +848,9 @@ Mostly answered post-implementation:
|
||||
- **Authentication between internal services.** The hash-lock OIDC
|
||||
perimeter handles external clients; internal service-to-service
|
||||
trust is by network isolation (docker network for dev, equivalent
|
||||
isolation in prod deployments). Adding service-to-service auth is a
|
||||
separate concern.
|
||||
isolation in prod deployments). Since v2.0.0 metadata *writes* carry
|
||||
their own authentication — the claim signature — so meta-core needs no
|
||||
per-writer tokens; transport-level service auth remains out of scope.
|
||||
- **Replacing the public HTTP API surface.** This proposal exercises
|
||||
what already exists; the only new endpoints are the two SSE streams.
|
||||
- **Introducing new transport, broker, proxy, or sidecar.** Everything
|
||||
@@ -716,6 +899,11 @@ What actually shipped vs. what this doc planned.
|
||||
|
||||
### Write-path CID resolution (`storage.ResolveRoot`)
|
||||
|
||||
> *History (v1.x).* In v2.0.0 a claim names a cid, never a rid, and the
|
||||
> hidden rid comes from `cid_group` (a `cids/<other>` vote links two
|
||||
> groups; the older rid survives). The dual-root failure below cannot
|
||||
> recur: there is no write that addresses a root.
|
||||
|
||||
Not in the original plan but load-bearing.
|
||||
|
||||
**The problem.** meta-sort's PATCH-via-HTTP migration (PR B) routed
|
||||
@@ -766,9 +954,10 @@ duplicated in each to avoid an import cycle on the `api` package.
|
||||
|
||||
- **meta-dup SSE + read migration.** Not deployed, no test harness.
|
||||
When revived, copy meta-fuse's pattern.
|
||||
- **Editor `/api/metadata/{hashId}` consolidation.** The duplicate
|
||||
editor-internal API surface is untouched; could be retired in a
|
||||
follow-up by porting the editor UI to `/meta/{hash}`.
|
||||
- **Editor writes (resolved in v2.0.0).** `PUT /api/metadata/{hashId}`
|
||||
answers `410`. meta-sort-editor's manual edits go through meta-sort's
|
||||
backend, which signs them as meta-sort (its own key) and posts them to
|
||||
`POST /api/claims`. The editor's reads keep using `/api/metadata/{hashId}`.
|
||||
- **Consolidated `internal/httperr` package** for the typed envelope
|
||||
helpers (currently duplicated 4× to avoid the import cycle). Code
|
||||
hygiene; functionally a no-op.
|
||||
|
||||
+315
-103
@@ -1,15 +1,22 @@
|
||||
# Meta-Core Architecture
|
||||
|
||||
> **meta-core v2.0.0 — the claims model.** Implementation basis:
|
||||
> [`docs/implementation/metacore-claims-model.md`](../../../docs/implementation/metacore-claims-model.md)
|
||||
> (meta-root). Where this file and that one disagree, that one wins.
|
||||
|
||||
## Document Purpose
|
||||
|
||||
This document describes the architectural design of **meta-core**, the
|
||||
standalone container that owns the metadata store (an embedded SQLite
|
||||
file; Redis until 2026-10), the leader-info file, the HTTP+SSE
|
||||
standalone container that owns the metadata store (two embedded SQLite
|
||||
databases since v2.0.0 — *truth* and *cache*; a single SQLite file in
|
||||
v1.x; Redis until 2026-10), the leader-info file, the HTTP+SSE
|
||||
metadata surface, and a WebDAV file server. It covers the leader gate,
|
||||
storage data model, API design, the SSE event surface, and the integration
|
||||
contract with other MetaMesh services.
|
||||
the claims data model and its write/apply pipeline, API design, the SSE
|
||||
event surface, and the integration contract with other MetaMesh services.
|
||||
|
||||
For the on-disk key layout, see
|
||||
For the engine design (SQLite, group commit, measurements), see
|
||||
[metacore-storage-engine.md](../../../docs/project-architecture/metacore-storage-engine.md);
|
||||
for the on-disk layout (tables of both databases), see
|
||||
[metadata-storage-structure.md](metadata-storage-structure.md). For the
|
||||
external-access design (store lockdown, SSE wire contract, auth rules),
|
||||
see [api-mediated-access.md](api-mediated-access.md). For the UUID-rooted
|
||||
@@ -24,7 +31,7 @@ view of file metadata:
|
||||
|
||||
| Service | Language | Role |
|
||||
|---|---|---|
|
||||
| **meta-sort** | TypeScript | Watches folders, runs plugins, writes metadata via meta-core's HTTP API. |
|
||||
| **meta-sort** | TypeScript | Watches folders, runs plugins; each plugin signs its own claims and posts them to meta-core. |
|
||||
| **meta-fuse** | TypeScript / Rust | Serves the virtual filesystem from metadata. |
|
||||
| **meta-stremio** | Python | Stremio addon for HLS streaming. |
|
||||
| **meta-dup** | TypeScript | Duplicate detection. |
|
||||
@@ -32,8 +39,14 @@ view of file metadata:
|
||||
|
||||
meta-core sits below all of these as the single owner of:
|
||||
|
||||
1. **The metadata store** — an embedded SQLite file
|
||||
(`/meta-core/db/meta-core.db`) holding metadata and the event streams.
|
||||
1. **The metadata store** — two embedded SQLite databases: **truth**
|
||||
(`/meta-core/db/meta-core.db`, backed up) holding the signed claims, and
|
||||
**cache** (`/meta-core/db-cache/cache.db`, never backed up) holding every
|
||||
index and derived row — a v1-layout `kv` table (the merged view as
|
||||
`file:<rid>/<field>` rows, lookup, counters, event streams, service
|
||||
scratch KV) with its field index and FTS, plus the `claim_by_pid` and
|
||||
`local_file` tables. (v2.0.0 is implemented, uncommitted and not yet
|
||||
deployed as of 2026-10-25.)
|
||||
2. **The leader-info file** — the one place services look to discover meta-core's URLs.
|
||||
3. **The HTTP+SSE surface** — the only way external services touch the data plane.
|
||||
4. **A WebDAV server** — file access for plugins and other services.
|
||||
@@ -44,7 +57,12 @@ meta-core sits below all of these as the single owner of:
|
||||
> through meta-core's HTTP+SSE surface.
|
||||
|
||||
This is the contract that PR D of [api-mediated-access.md](api-mediated-access.md)
|
||||
landed and that the rest of this document assumes.
|
||||
landed and that the rest of this document assumes. v2.0.0 adds a second:
|
||||
|
||||
> Every metadata value is a **signed vote** — `claim(uid, cid, key) → value`
|
||||
> + `ts` + `sig`. The only write is `POST /api/claims`. meta-core verifies
|
||||
> every claim and **never signs one**: it holds no writer key, and the
|
||||
> signature is the write authentication.
|
||||
|
||||
---
|
||||
|
||||
@@ -68,13 +86,15 @@ database file and the writable state under `/meta-core`.
|
||||
│ ├─ rclone rcd (mount daemon) │
|
||||
│ ├─ mount-watcher.sh (rclone mount lifecycle) │
|
||||
│ └─ meta-core (Go) ──► HTTP + SSE + WebDAV on :9000 │
|
||||
│ + embedded SQLite (db/meta-core.db) │
|
||||
│ + embedded SQLite: truth (db/meta-core.db) │
|
||||
│ and cache (cache/cache.db) │
|
||||
└────────────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
┌────────────────────────────┴────────────────────────────┐
|
||||
│ /meta-core (shared volume) │
|
||||
│ locks/kv-leader.{lock,info} │
|
||||
│ db/meta-core.db (+ -wal, -shm) │
|
||||
│ db/meta-core.db (+ -wal, -shm) truth, backed up │
|
||||
│ cache/cache.db (+ -wal, -shm) cache, never backed │
|
||||
│ services/*.json │
|
||||
│ mounts/{mounts.json, errors/} │
|
||||
│ watchers.json │
|
||||
@@ -203,67 +223,166 @@ Other services read this file, then call `GET /urls` for full discovery
|
||||
|
||||
## Schema-Version Gate
|
||||
|
||||
Before serving any traffic, `cmd/meta-core/main.go:54-61` calls
|
||||
`storage.EnsureSchemaVersion`. The sentinel lives in the store at
|
||||
`meta-core:schema-version` (`internal/storage/schema_version.go`). The
|
||||
gate refuses to boot when the existing store layout predates the
|
||||
UUID-rooted schema, so legacy `file:midhash256:*` data can't leak into
|
||||
the current code paths. In alpha there is no automated migration —
|
||||
operators wipe with `pnpm run clean:all` and restart.
|
||||
Before serving any traffic, `cmd/meta-core/main.go` opens both databases
|
||||
and calls `storage.EnsureSchemaVersion` (before the cache catch-up). In
|
||||
v2.0.0 the sentinel is the `schema-version = "3"` row of the truth
|
||||
database's `meta` table. The gate:
|
||||
|
||||
- **explicitly refuses a v1.x store** — a truth file holding the v1 `kv`
|
||||
table (`meta-core:schema-version = "2"`), which the default path still
|
||||
points at — with a message naming `tools/claims-migrate`;
|
||||
- refuses a store with claims but no sentinel, or any other version;
|
||||
- stamps a fresh store with `"3"`.
|
||||
|
||||
So no pre-claims data leaks into the current code paths. The meta-core
|
||||
**service binary contains no migration code**: watch.nsl.sh is converted
|
||||
once by the separate one-off tool `packages/meta-core/tools/claims-migrate`
|
||||
(not built into the image; implementation basis §11); every other box and
|
||||
dev stack starts empty (operators wipe with `pnpm run clean:all` and
|
||||
restart).
|
||||
|
||||
---
|
||||
|
||||
## Data Model
|
||||
|
||||
### Two concepts: metadata vs. file bytes
|
||||
### The one concept: signed votes
|
||||
|
||||
```
|
||||
┌────────────────────────────────────────────────────────────────────┐
|
||||
│ METADATA (embedded SQLite, kv table) │
|
||||
│ │
|
||||
│ Root: opaque UUIDv7 (Crockford Base32; ULID layout, 26 chars) │
|
||||
│ Storage: flat string keys │
|
||||
│ file:<uuid>/<property> → string value │
|
||||
│ file:<uuid>/cids/<bareCid> → "true" (key-set) │
|
||||
│ dup:<uuid>/<path> → "" (one per path) │
|
||||
│ │
|
||||
│ No stored canonical_cid — derived by rank on read. │
|
||||
│ │
|
||||
│ Aliases (reverse index, one key each): │
|
||||
│ cid:<bareCid> → <uuid> │
|
||||
│ │
|
||||
│ Index: idx:roots/<uuid> → "" (one per root) │
|
||||
│ │
|
||||
│ Mintable without a local file (remote / deleted entries OK). │
|
||||
│ Resolvable by any registered CID in O(1). │
|
||||
└────────────────────────────────────────────────────────────────────┘
|
||||
|
||||
┌────────────────────────────────────────────────────────────────────┐
|
||||
│ FILE BYTES │
|
||||
│ │
|
||||
│ Location: /files volume (shared across containers). │
|
||||
│ Access: path resolution via metadata, served via /webdav/*, │
|
||||
│ /file/{cid} (reverse-lookup), or /data/{hash}. │
|
||||
│ Path stored as filePath in the metadata, relative to /files. │
|
||||
└────────────────────────────────────────────────────────────────────┘
|
||||
claim(uid, cid, key) → value + ts + sig
|
||||
```
|
||||
|
||||
### Why UUID-rooted?
|
||||
- `uid` — the signer (a secp256k1 public key, `z…` base58btc). One key per
|
||||
writing container (each meta-sort plugin, each self-publishing feeder,
|
||||
gateway core, meta-share, meta-search) and one per user.
|
||||
- `cid` — the subject, a content CID; never a rid.
|
||||
- `key` — the field (`METADATA_KEYS.md`); key-set members live inside the
|
||||
key (`genres/Drama`, `cids/<cid>`).
|
||||
- `value` — text (public), bytes (signed + encrypted envelope) or null
|
||||
(retraction, kept as a tombstone).
|
||||
- `ts` — milliseconds; it **is** the version and only increases per
|
||||
`(uid, cid, key)`.
|
||||
- `sig` — MMUR2: secp256k1 ECDSA (compact 64 bytes) over SHA-256 of
|
||||
`"MMUR2\n" ‖ canonical CBOR [cid, uid, key, value, ts]`.
|
||||
|
||||
Roots were previously content-hash tokens (`file:midhash256:abc/...`),
|
||||
which privileged one hash algorithm. UUID-rooted storage treats every CID
|
||||
symmetrically as an alias, makes `GET /api/meta/<cid>` O(1) for any
|
||||
registered CID, and lets new digest types be added without rewriting root
|
||||
keys. See [uuid-rooted-metadata.md](uuid-rooted-metadata.md) for the
|
||||
design rationale and [metadata-storage-structure.md](metadata-storage-structure.md)
|
||||
for the on-disk layout.
|
||||
The old shared record (`file:<rid>/<field>`) and the old User Data Layer
|
||||
store (`udl:rec:*`) both collapse into this shape. The store distinguishes
|
||||
only plaintext (signed) from ciphertext (signed + encrypted); "personal vs
|
||||
shared" is app logic.
|
||||
|
||||
### Two databases
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────────────────────────┐
|
||||
│ TRUTH /meta-core/db/meta-core.db (backed up; PKs only) │
|
||||
│ │
|
||||
│ principal(pid, uid) uid ↔ local surrogate pid │
|
||||
│ claim(cid, key, pid → value, ts, sig, seq) │
|
||||
│ the votes; PK (cid, key, pid) │
|
||||
│ cid_group(cid → rid) hidden local record identity │
|
||||
│ meta(name → value) schema-version, last_seq │
|
||||
└──────────────────────────────────────────────────────────────────────┘
|
||||
│ apply loop (seq order)
|
||||
▼
|
||||
┌──────────────────────────────────────────────────────────────────────┐
|
||||
│ CACHE /meta-core/db-cache/cache.db (never backed up; rebuildable)│
|
||||
│ │
|
||||
│ kv(k → v) the v1 layout, every row derived: │
|
||||
│ file:<rid>/<field> the merged view every read uses │
|
||||
│ cid:<cid> cid → rid lookup │
|
||||
│ idx:roots/ idx:fts/ idx:count/ idx:schema/ roots, FTS rowid, │
|
||||
│ counters (search totals, stats) │
|
||||
│ ev:<stream>/…, evhwm:… /api/events/meta + /files backlog │
|
||||
│ meta:applied_seq, meta:rebuilt_at apply-loop bookkeeping │
|
||||
│ anything else /api/kv/value scratch KV │
|
||||
│ kv_field_v (index), fts filter + free-text search │
|
||||
│ claim_by_pid(pid, cid, key) "everything uid X wrote" (?as=, │
|
||||
│ UDL facade, identity retraction) │
|
||||
│ local_file(path → midhash, size, mtime) watcher facts, unsigned │
|
||||
└──────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
Disaster recovery is "delete the cache, replay truth". `META_CORE_DB_PATH`
|
||||
names the truth file, `META_CORE_CACHE_PATH` the cache file. The cache keeps
|
||||
the v1 `kv` layout so every v1 reader (record reads, field index, FTS,
|
||||
counters, SSE + snapshot) runs unchanged on derived rows; `nsrc` is computed
|
||||
by the merge (`POST /api/merge`), not stored. Full schema: implementation
|
||||
basis §5.
|
||||
|
||||
### Records: cids, groups and the hidden rid
|
||||
|
||||
A vote never mentions a rid. A **rid** (UUIDv7, 26-char Crockford, as in
|
||||
v1.x) is a local name for a group of cids known to be the same content,
|
||||
held in `cid_group`:
|
||||
|
||||
- A new cid with no link gets its own group and a fresh rid.
|
||||
- A `cids/<other>` vote on cid X links X's group and `<other>`'s group; the
|
||||
group with the older rid survives, the losing rid gets a `del` event per
|
||||
key and the survivor `set` events (no alias table). Retracting the link
|
||||
does **not** split the group (phase 1); the record view synthesises
|
||||
`cids/<cid> = "true"` for every cid of the group from `cid_group`.
|
||||
- Reading record R = every vote on every cid of R's group, merged per key;
|
||||
within a group one signer counts once per key.
|
||||
- rids are never signed and never travel; consumers that hold them
|
||||
(meta-sort, meta-fuse, meta-stremio) keep working because `cid_group`
|
||||
lives in truth.
|
||||
|
||||
### Merge
|
||||
|
||||
Per `(rid, key)`, over the group's votes, each signer's newest vote counted
|
||||
once (`internal/storage/merge.go`, shared by the apply loop, `?as=` and
|
||||
`POST /api/merge`):
|
||||
|
||||
1. the viewer's own live text vote (`?as=<uid>`, applied at read time,
|
||||
never stored);
|
||||
2. retractions and ciphertext drop out — only text merges;
|
||||
3. most distinct signers for the normalised value (**trim + case-fold
|
||||
only**, no NFC) — `nsrc` is that count (computed, not stored);
|
||||
4. tie → oldest `ts`;
|
||||
5. tie → byte-wise smallest value.
|
||||
|
||||
The stored value is the winning vote's exact text. Key-sets need no special
|
||||
code (each member is a key, present while one vote supports it). A
|
||||
different kind of fact gets a different key, so a guesser (e.g.
|
||||
filename-parser) never competes with a lookup on the same key.
|
||||
|
||||
### Box-local file facts
|
||||
|
||||
`filePath`, size, `mtimeNano` and midhash are **not claims** — meta-core
|
||||
cannot sign, and the disk can always be rescanned. The file watcher and
|
||||
`POST /api/files/register` write them into the cache's `local_file` table
|
||||
(and the midhash into `cid_group`); they are joined into the record view
|
||||
unsigned and never reach the claims API or the swarm. When several local
|
||||
paths hold the same content, `filePath` is the lexicographically first one
|
||||
(size and mtime from the same row); a signed `sizeByte` claim wins over the
|
||||
on-disk size.
|
||||
|
||||
### File bytes
|
||||
|
||||
```
|
||||
Location: /files volume (shared across containers).
|
||||
Access: path resolution via local_file, served via /webdav/*,
|
||||
/file/{cid} (reverse-lookup), or /data/{hash}.
|
||||
Path: filePath (from local_file) in the record view, relative to /files.
|
||||
```
|
||||
|
||||
### History: UUID-rooted flat keys (v1.x)
|
||||
|
||||
Until v2.0.0 a record was a set of flat keys under an opaque UUID root —
|
||||
`file:<uuid>/<property>`, `file:<uuid>/cids/<bareCid>` key-sets,
|
||||
`dup:<uuid>/<path>`, a reverse index `cid:<bareCid> → <uuid>` and a root
|
||||
index `idx:roots/<uuid>`, all in one kv table. Roots had previously been
|
||||
content-hash tokens (`file:midhash256:abc/...`); UUID roots treated every
|
||||
CID symmetrically as an alias. The rid of v2.0.0 is that UUID, kept hidden
|
||||
and local; migrated records kept their UUIDs. See
|
||||
[uuid-rooted-metadata.md](uuid-rooted-metadata.md) (the *why*) and the
|
||||
History note of [metadata-storage-structure.md](metadata-storage-structure.md).
|
||||
|
||||
### Path resolution
|
||||
|
||||
`filePath` values stored in metadata are absolute paths under the `/files`
|
||||
volume (e.g. `/files/watch/movies/Movie.2024.mkv`). The `/files` prefix is
|
||||
itself a per-host detail — moving the volume only requires re-mounting it
|
||||
at the same path inside the container.
|
||||
`filePath` values are absolute paths under the `/files` volume (e.g.
|
||||
`/files/watch/movies/Movie.2024.mkv`). The `/files` prefix is itself a
|
||||
per-host detail — moving the volume only requires re-mounting it at the
|
||||
same path inside the container.
|
||||
|
||||
---
|
||||
|
||||
@@ -282,16 +401,16 @@ Bootstrap / discovery
|
||||
GET /neighbors (alias: /api/neighbors)
|
||||
GET /api/stats
|
||||
|
||||
Metadata (primary surface for other services)
|
||||
Claims (the only metadata write path)
|
||||
POST /api/claims signed claim batch → per-claim status
|
||||
POST /api/merge {claims, as?} → {fields, nsrc, rejected}; no write
|
||||
GET /api/meta/{cid}/claims {rid, cids, claims} of the cid's group (?key=, ?uid=; 404 unknown)
|
||||
GET /api/metadata/{rid}/claims same, addressed by rid
|
||||
|
||||
Metadata reads (merged view + local_file; optional ?as=<uid>)
|
||||
GET /meta
|
||||
GET /meta/{hash}
|
||||
PUT /meta/{hash}
|
||||
PATCH /meta/{hash}
|
||||
DELETE /meta/{hash}
|
||||
GET /meta/{hash}/{key...}
|
||||
PUT /meta/{hash}/{key...}
|
||||
DELETE /meta/{hash}/{key...}
|
||||
POST /meta/{hash}/_add/{key...}
|
||||
|
||||
CID-addressed (public, auth-bypassed at the perimeter)
|
||||
GET /api/meta/{cid}
|
||||
@@ -301,24 +420,45 @@ CID-addressed (public, auth-bypassed at the perimeter)
|
||||
HEAD /data/{hash}
|
||||
GET /data/{hash}/path
|
||||
|
||||
Editor / KV / schema / snapshot
|
||||
Editor / KV / schema
|
||||
GET /api/metadata/hash-ids
|
||||
GET /api/files/tuples
|
||||
POST /api/files/register (fills local_file + cid_group)
|
||||
GET /api/metadata/list
|
||||
POST /api/metadata/search
|
||||
POST /api/metadata/batch
|
||||
POST /api/metadata/clear
|
||||
{GET,PUT,DELETE} /api/metadata/{hashId}
|
||||
{GET,PUT} /api/metadata/{hashId}/property
|
||||
GET /api/kv/{info,keys,tree,search,find,value}
|
||||
PUT /api/kv/value
|
||||
DELETE /api/kv/value
|
||||
GET /api/kv/key/{key...}
|
||||
POST /api/metadata/clear (wipes both databases)
|
||||
GET /api/metadata/{hashId} (hashId = rid)
|
||||
GET /api/metadata/{hashId}/property
|
||||
GET /api/kv/{info,keys,tree,search,find} (read-only browser over the cache kv table)
|
||||
{GET,PUT,DELETE} /api/kv/value (service scratch keys only; reserved prefixes refused)
|
||||
GET /api/kv/key/{key...} (read-only)
|
||||
GET /api/schema
|
||||
POST /api/schema/rescan
|
||||
GET /api/snapshot/export
|
||||
POST /api/snapshot/import
|
||||
POST /api/snapshot/wipe
|
||||
|
||||
User Data Layer (facade over claims; records are MMUR2)
|
||||
{GET,PUT} /api/udl/record
|
||||
PUT /api/udl/records
|
||||
GET /api/udl/user/{uid}/{key/{key},cid/{cid},all,count}
|
||||
GET /api/udl/cid/{cid}/{users,aggregate}
|
||||
GET /api/udl/users/stats (every signer with live claims, plugin keys included)
|
||||
|
||||
Account-key identity service (kept in v2.0.0; removal is a separate ticket)
|
||||
{GET,DELETE} /api/identity (DELETE signs + stores a retraction of every live claim first)
|
||||
GET /api/identity/{accounts,aead-key}
|
||||
POST /api/identity/{challenge,generate,import,reveal,sign,sign-batch}
|
||||
(sign/sign-batch `format`: compact | der; default
|
||||
compact for "MMUR2\n" payloads, DER otherwise)
|
||||
|
||||
Removed in v2.0.0 — answer 410 Gone, pointing to POST /api/claims
|
||||
PUT/PATCH/DELETE /meta/{hash}
|
||||
PUT/DELETE /meta/{hash}/{key...}
|
||||
POST /meta/{hash}/_add/{key...}
|
||||
PUT/DELETE /api/metadata/{hashId}
|
||||
PUT /api/metadata/{hashId}/property
|
||||
POST /api/metadata/batch
|
||||
|
||||
Removed in v2.0.0 — no snapshot format
|
||||
GET /api/snapshot/export, POST /api/snapshot/{import,wipe}
|
||||
|
||||
SSE event streams (HTTP-mirror of the store's event streams)
|
||||
GET /api/events/files
|
||||
@@ -366,6 +506,13 @@ peers and unauthenticated tooling can reach them):
|
||||
- (`/api/events/files`, `/api/events/meta` are also auth-bypassed but
|
||||
**should not** be exposed publicly via Caddy; they're inside-only.)
|
||||
|
||||
Since v2.0.0 `/meta/*` serves reads only — its write verbs answer `410
|
||||
Gone` — so the bypass no longer exposes an unsigned write. Writes go to
|
||||
`POST /api/claims`, where the claim signature (not the perimeter) is the
|
||||
write authentication. `POST /api/claims`, `POST /api/merge` and
|
||||
`GET /api/meta/{cid}/claims` are container-to-container routes, not routed
|
||||
through Caddy, like the v1.x write routes were.
|
||||
|
||||
See `docs/api-mediated-access.md` "Auth" for the rationale.
|
||||
|
||||
### SSE event streams — wire contract
|
||||
@@ -389,9 +536,46 @@ the kv table (`internal/kv/stream.go`). The contract
|
||||
`event: gap` payload before resuming from the oldest available entry.
|
||||
|
||||
External services use SSE with `Last-Event-ID` as the resume primitive.
|
||||
`meta:events` entries are appended by the storage engine in the same
|
||||
transaction as the change they describe; the Redis-era keyspace-notification
|
||||
hop through `MetaPublisher` is gone.
|
||||
Since v2.0.0 `meta:events` describes changes of the **merged** view, same
|
||||
`{type: set|del, key, ts}` shape: the cache engine appends an
|
||||
`ev:meta:events/…` row for every change of a merged `file:` row, in the same
|
||||
cache transaction as the change (both streams are v1 stream rows of the
|
||||
cache `kv` table). A cache rebuild
|
||||
emits no per-key events — consumers get `event: gap` and take a snapshot;
|
||||
event IDs stay time-based and keep rising. *History:* v1.x appended them in
|
||||
the same transaction as the `file:` key change; the Redis-era
|
||||
keyspace-notification hop through `MetaPublisher` is long gone.
|
||||
|
||||
### Write path and apply loop
|
||||
|
||||
```
|
||||
POST /api/claims ──► per claim, before the writer:
|
||||
0. validate (required fields, reserved keys, ts range) → else bad_request
|
||||
1. same ts + sig as stored? → unchanged (no verify)
|
||||
2. same value as stored (any ts)? → unchanged; stored row + older ts kept
|
||||
3. stored ts ≥ incoming ts? → stale
|
||||
4. verify MMUR2 signature → else bad_signature
|
||||
──► accepted claims, one group-committed truth transaction:
|
||||
5. re-check 1–3 under the writer
|
||||
6. uid → pid (principal; insert on first sight)
|
||||
7. upsert claim (tombstone if null), seq = ++last_seq
|
||||
8. cid_group: mint rid for unknown cid; text cids/* links groups
|
||||
(a retraction never splits)
|
||||
9. commit (last_seq in truth meta); answer once the cache applied it
|
||||
──► apply loop, in seq order, into the cache:
|
||||
sync cid:/cids/<cid>/idx:roots rows → claim_by_pid
|
||||
→ re-merge (rid, key) from truth PK range reads
|
||||
→ set/delete file:<rid>/<key> (+ local_file for filePath/sizeByte/mtimeNano)
|
||||
⇒ meta:events row + counter delta in the same tx
|
||||
→ FTS row on a haystack change → meta:applied_seq
|
||||
```
|
||||
|
||||
The verifier lives in the write path (secp256k1 from `internal/identity`);
|
||||
meta-core holds **no** writer key and never signs a claim. Response: one
|
||||
status per claim — `accepted` · `unchanged` · `stale` · `bad_signature` ·
|
||||
`bad_request`. Request/response shapes and the 410 contract for removed
|
||||
routes: [api-mediated-access.md](api-mediated-access.md) "Write contract —
|
||||
signed claims".
|
||||
|
||||
### WebDAV URLs
|
||||
|
||||
@@ -421,7 +605,8 @@ WebDAV caching is handled by nginx's `proxy_cache` (see
|
||||
| Single binary | Static compilation |
|
||||
| Low resource | ~10-20 MB resident |
|
||||
| HTTP server | stdlib `net/http` + gorilla/mux |
|
||||
| Metadata store | embedded SQLite via `modernc.org/sqlite` (pure Go, CGO off) |
|
||||
| Metadata store | embedded SQLite via `modernc.org/sqlite` (pure Go, CGO off) — truth + cache databases |
|
||||
| Claim verification | secp256k1 ECDSA (`internal/identity`) |
|
||||
| WebDAV | `golang.org/x/net/webdav` |
|
||||
|
||||
### Repo layout
|
||||
@@ -434,12 +619,12 @@ packages/meta-core/
|
||||
│ ├── cid/ CID parsing + algorithm ranking
|
||||
│ ├── config/ env-driven configuration
|
||||
│ ├── discovery/ service registry + dead-service cleaner
|
||||
│ ├── kv/ SQLite kv engine + event streams
|
||||
│ ├── identity/ secp256k1 (claim verification; account-key service)
|
||||
│ ├── kv/ SQLite engine (single writer, group commit) + event streams
|
||||
│ ├── leader/ LeaderInfoProvider (no election logic)
|
||||
│ ├── mounts/ rclone mount manager + handlers + stats
|
||||
│ ├── schema/ per-field schema from transactional counters
|
||||
│ ├── snapshot/ snapshot export / import / wipe
|
||||
│ ├── storage/ flat-key records + search + UUID + CID reverse index + schema gate
|
||||
│ ├── storage/ claims write path, apply loop, merge, cid_group, search, schema gate
|
||||
│ ├── watcher/ scanner + dispatcher + midhash + state
|
||||
│ ├── watchers/ polling-based watcher configs
|
||||
│ └── webdav/ /webdav/* handler
|
||||
@@ -460,15 +645,22 @@ packages/meta-core/
|
||||
```
|
||||
1. Load configuration from environment
|
||||
2. Build LeaderInfoProvider (no election; bash gate already won)
|
||||
3. Open the SQLite store (META_CORE_DB_PATH or /meta-core/db/meta-core.db)
|
||||
4. EnsureSchemaVersion (abort if the store layout is stale)
|
||||
5. Start service discovery + dead-service cleaner
|
||||
6. NewServer + Server.Start (boots watcher, mounts, WebDAV, API)
|
||||
7. EnsureCounters (one-time build of the derived counters
|
||||
on a store that predates them)
|
||||
8. Rebuild FTS rows in the background if the FTS table is empty
|
||||
3. Create the cache folder; open the truth database (META_CORE_DB_PATH or
|
||||
/meta-core/db/meta-core.db) and the cache (META_CORE_CACHE_PATH or
|
||||
/meta-core/db-cache/cache.db) (storage.Connect)
|
||||
4. EnsureSchemaVersion (abort on a v1 kv store or an unknown layout)
|
||||
5. storage.Start — cache catch-up (meta:applied_seq ≥ last_seq → nothing; behind →
|
||||
replay claims with seq > applied_seq, or a full
|
||||
rebuild if a cids/* link is among them; no
|
||||
cache → full rebuild), then the apply loop
|
||||
6. Derived counters ready
|
||||
7. Start service discovery + dead-service cleaner
|
||||
8. NewServer + Server.Start (boots watcher, mounts, WebDAV, API)
|
||||
9. Wait for SIGINT/SIGTERM
|
||||
10. Shutdown in reverse order
|
||||
|
||||
(v1.x ran `EnsureCounters` and a background FTS rebuild here; in v2.0.0
|
||||
counters and FTS are cache rows maintained with every merged-row change.)
|
||||
```
|
||||
|
||||
### Process management
|
||||
@@ -489,10 +681,25 @@ internal crashes don't trigger re-election.
|
||||
|
||||
### Store durability
|
||||
|
||||
The SQLite file runs in WAL mode with `synchronous=NORMAL`: every
|
||||
committed transaction survives a meta-core process crash. A record write — fields, root index, CID aliases,
|
||||
FTS row and its events — is one transaction, so a crash never leaves a
|
||||
half-written record.
|
||||
Both SQLite files run in WAL mode; truth with `synchronous=NORMAL` (every
|
||||
committed transaction survives a meta-core process crash), the cache with
|
||||
`synchronous=OFF` (it is rebuildable). A claim batch
|
||||
commits to truth in one transaction; the apply loop then commits the
|
||||
derived cache changes. **A crash between the two commits leaves the cache
|
||||
behind, never wrong**: at boot, if the cache's `applied_seq` is below
|
||||
truth's `last_seq`, meta-core scans truth's `claim` table for
|
||||
`seq > applied_seq` (a full scan — there is no seq index; it only happens
|
||||
after an unclean shutdown) and applies them — or rebuilds the cache in full
|
||||
if one of them is a `cids/*` link.
|
||||
|
||||
### Cache loss
|
||||
|
||||
The cache is never backed up and is fully derived (except `local_file`,
|
||||
which a rescan of `/files` refills, and the service scratch keys, whose
|
||||
loss costs a refetch). A missing cache folder (no `meta:applied_seq`) means
|
||||
meta-core rebuilds every derived row from truth and stamps
|
||||
`meta:rebuilt_at`. A rebuild emits no per-key events;
|
||||
SSE consumers get `event: gap` and re-snapshot.
|
||||
|
||||
### Container crash
|
||||
|
||||
@@ -503,8 +710,9 @@ the container."
|
||||
### Schema mismatch
|
||||
|
||||
`EnsureSchemaVersion` refuses to boot. Operators wipe with
|
||||
`pnpm run clean:all` (or equivalent) and restart. This is an alpha-stage
|
||||
hard-fail; there is no automated migration path.
|
||||
`pnpm run clean:all` (or equivalent) and restart, or — once, on
|
||||
watch.nsl.sh — convert a v1.x store with `tools/claims-migrate`. This is an
|
||||
alpha-stage hard-fail; the service never migrates automatically.
|
||||
|
||||
### Shared volume unavailable
|
||||
|
||||
@@ -525,7 +733,8 @@ via `/health`; metadata I/O and SSE event streams stop.
|
||||
| `SERVICE_VERSION` | `1.0.0` | Reported in `/status`. |
|
||||
| `API_PORT` | `8180` | External port baked into `baseUrl`. |
|
||||
| `BASE_URL` | _empty_ | Overrides constructed `baseUrl` (HTTPS perimeter). |
|
||||
| `META_CORE_DB_PATH` | `<META_CORE_PATH>/db/meta-core.db` | SQLite database file. Dev stacks set `/meta-core-db/meta-core.db` (named volume — SQLite WAL is unsafe on the 9p `/meta-core` bind of Windows dev boxes). |
|
||||
| `META_CORE_DB_PATH` | `<META_CORE_PATH>/db/meta-core.db` | Truth database file (backed up). Dev stacks set `/meta-core-db/meta-core.db` (named volume — SQLite WAL is unsafe on the 9p `/meta-core` bind of Windows dev boxes). |
|
||||
| `META_CORE_CACHE_PATH` | `/meta-core/db-cache/cache.db` | Cache database file (never backed up; rebuilt from truth when missing). Dev stacks keep it on a named volume too; store apps put the cache folder in Maison's `backup.exclude`. |
|
||||
| `META_CORE_DB_CACHE_MB` | `64` | Page cache of the writer connection. |
|
||||
| `META_CORE_DB_READER_CACHE_MB` | `16` | Page cache per reader connection. |
|
||||
| `META_CORE_DB_READERS` | `8` | Reader connection pool size. |
|
||||
@@ -547,7 +756,8 @@ via `/health`; metadata I/O and SSE event streams stop.
|
||||
|---|---|
|
||||
| `locks/kv-leader.lock` | Flock target. |
|
||||
| `locks/kv-leader.info` | Leader API URL (plaintext). |
|
||||
| `db/meta-core.db` (+ `-wal`, `-shm`) | Metadata store (SQLite). Replaced `db/redis/` in 2026-10. |
|
||||
| `db/meta-core.db` (+ `-wal`, `-shm`) | Truth database (claims, principals, `cid_group`). Backed up. Replaced `db/redis/` in 2026-10; holds the claims schema since v2.0.0. |
|
||||
| `db-cache/cache.db` (+ `-wal`, `-shm`) | Cache database (v1-layout `kv` table: merged view, lookup, counters, events, service scratch keys; field index, FTS, `claim_by_pid`, `local_file`). Never backed up. |
|
||||
| `services/*.json` | Service registry. |
|
||||
| `mounts/mounts.json` | rclone mount configuration. |
|
||||
| `mounts/errors/{id}.error` | Mount error logs. |
|
||||
@@ -561,7 +771,9 @@ via `/health`; metadata I/O and SSE event streams stop.
|
||||
meta-core consolidates leader gating, metadata storage, event mirroring,
|
||||
and file access for MetaMesh into one container with a typed HTTP+SSE
|
||||
surface. The bash flock gate guarantees single-leader semantics without
|
||||
an external consensus system. UUID-rooted storage with CID aliasing
|
||||
treats all content hashes symmetrically and makes any-CID resolution
|
||||
a single lookup. API-mediated access ensures no other service ever touches
|
||||
an external consensus system. Metadata is a table of signed votes in a
|
||||
backed-up truth database, merged by an apply loop into a rebuildable cache
|
||||
that serves every read; meta-core verifies every claim and signs none.
|
||||
CID groups behind a hidden local rid treat all content hashes
|
||||
symmetrically. API-mediated access ensures no other service ever touches
|
||||
the store.
|
||||
|
||||
+254
-323
@@ -1,256 +1,199 @@
|
||||
# Metadata Storage Structure
|
||||
|
||||
How meta-core lays out file metadata in its store. This document covers the
|
||||
on-disk shape (UUID-rooted flat string keys, CID reverse index,
|
||||
per-root sets) and the schema-version gate. It does **not** enumerate
|
||||
field semantics — that's the job of the repo-root
|
||||
[`METADATA_KEYS.md`](../../../METADATA_KEYS.md), the authoritative key
|
||||
registry for everyone who writes to or reads from metadata.
|
||||
How meta-core lays out metadata on disk since **v2.0.0 (claims model)**: the
|
||||
two SQLite databases, their tables, how a record is assembled from signed
|
||||
votes, and the schema-version gate. Implementation basis:
|
||||
[`docs/implementation/metacore-claims-model.md`](../../../docs/implementation/metacore-claims-model.md).
|
||||
This document does **not** enumerate field semantics — that is the job of the
|
||||
repo-root [`METADATA_KEYS.md`](../../../METADATA_KEYS.md), the authoritative
|
||||
key registry for everyone who writes or reads metadata.
|
||||
|
||||
For the design rationale (why UUIDs instead of content hashes, why a
|
||||
reverse index), see [`uuid-rooted-metadata.md`](uuid-rooted-metadata.md).
|
||||
For the external-access contract (HTTP-only, SSE for events), see
|
||||
Related: why rids are UUIDs and hidden —
|
||||
[`uuid-rooted-metadata.md`](uuid-rooted-metadata.md); why the engine is
|
||||
shaped this way, its guarantees and measurements —
|
||||
[`metacore-storage-engine.md`](../../../docs/project-architecture/metacore-storage-engine.md);
|
||||
the external-access contract (HTTP-only, SSE for events) —
|
||||
[`api-mediated-access.md`](api-mediated-access.md).
|
||||
|
||||
> **Storage engine (since 2026-10): an embedded SQLite file, not Redis.**
|
||||
> Until 2026-10 these keys lived in a Redis server spawned inside the
|
||||
> container. They now live in one string→string table of an embedded SQLite
|
||||
> database (pure-Go driver `modernc.org/sqlite`, CGO off). The **key names
|
||||
> did not change** — only the Redis *types* that some bookkeeping used (SETs,
|
||||
> HASHes, Streams) were flattened into plain keys, listed below. Holding
|
||||
> everything in RAM had cost ~6.6 GB on watch.nsl.sh's 169k records (Redis
|
||||
> plus the API's in-memory search copy) and caused repeated OOM kills.
|
||||
> **History.** Until 2026-10 metadata lived in Redis; v1.1.0–v1.2.x kept the
|
||||
> same flat keys (`file:<rid>/<field>`, `cid:<cid>`, `idx:*`, `dup:*`,
|
||||
> `udl:*`, `ev:*`) in one SQLite string→string `kv` table. v2.0.0 replaced
|
||||
> that keyspace entirely; the old layout is summarised at the end
|
||||
> ([Previous layout](#previous-layout-v11v12-kv-table)).
|
||||
|
||||
## Quick summary
|
||||
|
||||
- **One table**: `kv(k TEXT PRIMARY KEY, v TEXT NOT NULL) WITHOUT ROWID`,
|
||||
schema-free — a new metadata field is just a new key, never a migration.
|
||||
File: `/meta-core/db/meta-core.db` (override `META_CORE_DB_PATH`), WAL
|
||||
mode, `synchronous=NORMAL`.
|
||||
- **Roots are UUIDv7** encoded as 26-char Crockford Base32 (ULID layout),
|
||||
not content hashes. Examples in this doc use the placeholder
|
||||
`01JKR8XW5T4QMV7AHNJ2DEFGHK`.
|
||||
- **Properties are flat string keys**, one per leaf:
|
||||
`file:<uuid>/<property/path>` → string value. A record is exactly the
|
||||
key range `[file:<uuid>/, file:<uuid>0)` — there is no per-record field
|
||||
list.
|
||||
- **CIDs are reverse-index aliases**: `cid:<bareCid>` → `<uuid>`
|
||||
(one key per known CID; no algorithm prefix — the multicodec
|
||||
already carries it).
|
||||
- **Per-root sets**: `file:<uuid>/cids/<bareCid>` (a *key-set*: one key
|
||||
`"true"` per known CID, midhash included), `dup:<uuid>/<path>` (alternate
|
||||
paths with the same content), plus `idx:roots/<uuid>` (every UUID).
|
||||
There is **no stored `canonical_cid`** — it is derived by rank on read.
|
||||
- **A schema-version sentinel** (`meta-core:schema-version`) refuses to
|
||||
boot against legacy data.
|
||||
- **Every metadata value is a signed claim**: `claim(uid, cid, key) → value`
|
||||
+ `ts` + `sig` (MMUR2, secp256k1). One row per signer per field. Writers
|
||||
sign; meta-core never signs.
|
||||
- **Two databases.** *Truth* `/meta-core/db/meta-core.db`
|
||||
(`META_CORE_DB_PATH`, backed up, primary keys only): `principal`, `claim`,
|
||||
`cid_group`, `meta`. *Cache* `/meta-core/db-cache/cache.db`
|
||||
(`META_CORE_CACHE_PATH`, never backed up, rebuildable): the v1 `kv` table
|
||||
(merged view as `file:<rid>/<field>` rows, lookup, counters, event streams,
|
||||
bookkeeping, service scratch keys) with its `kv_field_v` index and `fts`,
|
||||
plus two real tables, `claim_by_pid` and `local_file`.
|
||||
- **Schema-free.** A field is a `key` string in a row; a new field never
|
||||
needs DDL.
|
||||
- **Records are cid groups.** A vote names a cid, never a rid. `cid_group`
|
||||
maps each cid to a **rid** — a hidden, locally minted UUIDv7 (26-char
|
||||
Crockford) naming the group. Examples use `01JKR8XW5T4QMV7AHNJ2DEFGHK`.
|
||||
- **What readers get is the merged view**: one precomputed value per
|
||||
`(rid, key)`, stored as the cache row `file:<rid>/<key>`. `nsrc` (how many
|
||||
signers support it) is computed by the merge and returned by
|
||||
`POST /api/merge`, not stored. `canonical_cid` is still **derived on read**
|
||||
by rank, never stored.
|
||||
- **Watcher facts are not claims**: `filePath`, size, `mtimeNano`, midhash
|
||||
live in `local_file` and are joined into the record view unsigned.
|
||||
- **A schema-version sentinel** (`schema-version = "3"` in truth `meta`).
|
||||
meta-core explicitly refuses a truth file holding the v1 `kv` table, and any
|
||||
store with claims but no sentinel or another version.
|
||||
|
||||
Implementation lives in `internal/kv/kv.go` (engine),
|
||||
`internal/kv/stream.go` (event streams), `internal/storage/client.go`,
|
||||
`internal/storage/search.go`, `internal/storage/cid_resolution.go`,
|
||||
`internal/storage/uuid.go`, and `internal/storage/schema_version.go`.
|
||||
## Truth database — `/meta-core/db/meta-core.db`
|
||||
|
||||
## Key structure
|
||||
Backed up; only primary keys (plus `principal.uid`'s implicit UNIQUE index).
|
||||
|
||||
Everything is one string→string table. The full layout:
|
||||
|
||||
| Key | Value | Notes |
|
||||
| Table | Columns (PK first) | Notes |
|
||||
|---|---|---|
|
||||
| `file:<rid>/<field>` | field value | a record property — the **only** thing under `file:` |
|
||||
| `cid:<cid>` | `<rid>` | reverse index |
|
||||
| `meta-core:schema-version` | `"2"` | schema gate |
|
||||
| `idx:roots/<rid>` | `""` | set of every root (was the Redis SET `file:__index__`) |
|
||||
| `idx:fts/<rid>` | FTS rowid | the record's row in the FTS table |
|
||||
| `idx:count/roots` | integer | number of roots — search's `total`, `/api/stats` records (v1.2.0) |
|
||||
| `idx:schema/<field>\x1f<keyHint>\x1f<prim>\x1f<hint>` | integer | per-field value-type counts behind `/api/schema` (v1.2.0) |
|
||||
| `idx:count/ready` | `"1"` | marks the counters as built (absent → one-time build at boot) |
|
||||
| `dup:<rid>/<path>` | `""` | a duplicate path (was the SET `file:<rid>/duplicates`) |
|
||||
| `udl:rec:<uid>/<cid>/<key>` | JSON `{record,version,ts,value?}` | User Data Layer entry (was a HASH) |
|
||||
| `udl:idx:…<set>\x1f<member>` | `""` | UDL index sets, one key per member |
|
||||
| `ev:<stream>/<ms13>-<seq10>` | JSON of the entry's fields | `meta:events` / `file:events` entries (were Redis Streams) |
|
||||
| `evhwm:<stream>` | last issued ID | stream high-water mark |
|
||||
| `btmagnet:*`, `searchcov:*`, `nzb:*` | — | written by other services via `/api/kv/value`; unchanged |
|
||||
| `principal` | `pid` INTEGER PK, `uid` TEXT UNIQUE | `uid` = `"z" + base58btc(33-byte compressed pubkey)`. `pid` is a local surrogate: never signed, never on the wire |
|
||||
| `claim` | `(cid, key, pid)` PK, `value` ANY, `ts`, `sig`, `seq` | `STRICT, WITHOUT ROWID`. `value`: TEXT = plaintext, BLOB = ciphertext envelope, NULL = retraction (tombstone, kept). `ts` (ms) is the version. `seq` = global change counter, not indexed |
|
||||
| `cid_group` | `cid` PK, `rid` | the cid → rid map: the record identity layer |
|
||||
| `meta` | `name` PK, `value` | `schema-version = "3"`, `last_seq` |
|
||||
|
||||
Keeping bookkeeping **out of `file:`** is load-bearing: the `file:` range is
|
||||
exactly the records, so the field index covers only real fields and every
|
||||
change under `file:` is a real field change worth publishing on
|
||||
`meta:events`. The per-record `file:<rid>/__fields__` SET and the
|
||||
`file:__dirty__` set of the Redis era no longer exist. Roots are split from
|
||||
the field at the **first** `/`, so CID-rooted and UUID-rooted records both
|
||||
parse.
|
||||
`claim` is written **only** by `POST /api/claims`. A claim replaces the
|
||||
stored row for its `(cid, key, pid)` only if its `ts` is strictly greater;
|
||||
identical `(ts, sig)` is `unchanged`; the **same value** from the same signer
|
||||
(any `ts`) is also `unchanged` and keeps the stored row with its older `ts`;
|
||||
otherwise older or equal is `stale`, and a bad signature is refused. Why `STRICT` with an `ANY` column: a non-STRICT `ANY`
|
||||
column gets NUMERIC affinity and turns the text `"007"` into the integer 7;
|
||||
STRICT keeps the written type, which is the cipher flag.
|
||||
|
||||
### Roots
|
||||
## Cache database — `/meta-core/db-cache/cache.db`
|
||||
|
||||
Never backed up (the store app lists the folder in `backup.exclude`).
|
||||
Deleting it is the recovery procedure: meta-core rebuilds it from truth at
|
||||
boot (no `meta:applied_seq` → full rebuild), and `local_file` refills from a
|
||||
rescan.
|
||||
|
||||
It keeps the **v1 key/value layout** (`internal/storage/client.go`): one
|
||||
`kv(k TEXT PRIMARY KEY, v TEXT NOT NULL) WITHOUT ROWID` table, whose rows are
|
||||
all derived — no client writes `file:` rows any more.
|
||||
|
||||
| `kv` rows | Holds | Written by | Read by |
|
||||
|---|---|---|---|
|
||||
| `file:<rid>/<field>` | the **merged view**, one row per `(rid, key)` — the only rows under `file:`. Includes the synthesised `cids/<cid> = "true"` members (from `cid_group`) and the box-local `filePath` / `sizeByte` / `mtimeNano` (from `local_file`) | apply loop | every record read; search |
|
||||
| `cid:<cid>` | cid → rid (copy of truth `cid_group`) | apply loop, watcher | every cid-addressed request |
|
||||
| `idx:roots/<rid>` | the set of every record | apply loop | listing, counts |
|
||||
| `idx:fts/<rid>` | rowid of the record's `fts` row | apply loop | FTS maintenance |
|
||||
| `idx:count/*`, `idx:schema/*` | record count, per-field schema counters | the cache engine's change hook, from `file:` deltas | search `total`, `/api/stats`, `/api/schema` |
|
||||
| `ev:<stream>/<ms13>-<seq10>`, `evhwm:<stream>` | `meta:events` (every `file:` change, same transaction) and `file:events` (watcher) | cache engine; watcher | `/api/events/meta`, `/api/events/files` (SSE) |
|
||||
| `meta:applied_seq`, `meta:rebuilt_at` | how far the cache has applied truth; last rebuild (older SSE cursors get `gap`) | apply loop, rebuild | boot catch-up; SSE |
|
||||
| anything else | service scratch KV (`btmagnet:*`, `searchcov:*`, `nzb:*`) | other services via `/api/kv/value` (reserved prefixes refused) | same |
|
||||
|
||||
| Index / table | Shape | Written by | Read by |
|
||||
|---|---|---|---|
|
||||
| `kv_field_v` (index) | `kv(lower(field parsed from k), lower(v), v)` over `file:` rows only | — | filter search |
|
||||
| `fts` | FTS5 trigram `(rid UNINDEXED, hay)`, one row per record | apply loop, when a haystack field's merged value changes | free-text search |
|
||||
| `claim_by_pid` | `(pid, cid, key)` PK | apply loop | `?as=<uid>` overlay, `/api/udl/*` facade, identity retraction, `/api/udl/users/stats` |
|
||||
| `local_file` | `path` PK → `midhash`, `size`, `mtime`; index `(midhash, path)` | file watcher, `POST /api/files/register` | record view (`filePath`, `sizeByte`, `mtimeNano`) |
|
||||
|
||||
The cache is **behind, never wrong**: after an unclean shutdown, boot finds
|
||||
`meta:applied_seq < last_seq` and re-applies every truth claim with `seq >
|
||||
applied_seq` (or rebuilds in full when one of them is a `cids/*` link). Write
|
||||
path, apply loop and durability are detailed in
|
||||
[`metacore-storage-engine.md`](../../../docs/project-architecture/metacore-storage-engine.md) §4.
|
||||
|
||||
## Keys
|
||||
|
||||
The `key` column carries the same field paths METADATA_KEYS.md defines —
|
||||
what used to follow `file:<rid>/`:
|
||||
|
||||
```
|
||||
file:<uuid>/<property/path> → string (always)
|
||||
claim(cid=bagacbabaec…, key=title, uid=<tmdb plugin>) = "Inception"
|
||||
claim(cid=bagacbabaec…, key=fileinfo/duration, uid=<ffmpeg plugin>) = "8878.5"
|
||||
claim(cid=bagacbabaec…, key=fileinfo/streamdetails/video/0/codec, uid=<ffmpeg plugin>) = "h264"
|
||||
claim(cid=bagacbabaec…, key=genres/Drama, uid=<tmdb plugin>) = "true"
|
||||
claim(cid=bagacbabaec…, key=cids/bafybeigd…, uid=<full-hash plugin>) = "true"
|
||||
```
|
||||
|
||||
UUIDs are produced by `storage.NewUUID()` (`internal/storage/uuid.go`):
|
||||
UUIDv7 → 26-char Crockford Base32 (skips I/L/O/U; ULID-compatible). The
|
||||
time-sortable prefix means roots minted in a given window sort together
|
||||
(`k >= 'file:01JKR8' AND k < 'file:01JKR9'`), which restored some of the
|
||||
debug-ability lost when midhash256 stopped being the root.
|
||||
- `/` is the path separator; nested objects are flattened on write and
|
||||
reconstructed on read.
|
||||
- **Key-sets** put the member inside the key (`genres/<g>`, `cids/<cid>`,
|
||||
`titles/eng/<name>`, `anchoredBy/tmdb:123`): one claim per member, union by
|
||||
the ordinary merge, removal by a signed retraction.
|
||||
- Plugins keep writing under their prefixes (`{plugin-id}/<field>`,
|
||||
`{plugin-id}/<group>/<n>/<field>`), each signing with its own key. Two
|
||||
sources compete on one key only when they make the same kind of statement;
|
||||
a different fact (e.g. a filename guess vs a TMDB lookup) gets a different
|
||||
key.
|
||||
|
||||
Properties use `/` as the path separator. Nested objects in source code
|
||||
are flattened to flat keys on write and reconstructed on read. Examples:
|
||||
## Records: `cid_group` and the hidden rid
|
||||
|
||||
```
|
||||
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/filePath = "/files/watch/movies/Inception.2010.mkv"
|
||||
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/sizeByte = "15032385536"
|
||||
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/mtimeNano = "1729872000000000000"
|
||||
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/title = "Inception"
|
||||
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/fileinfo/duration = "8878.5"
|
||||
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/fileinfo/streamdetails/video/0/codec = "h264"
|
||||
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/tmdb/poster = "bafkreih..."
|
||||
votes (truth, signed) cid_group (truth) cache kv rows (merged)
|
||||
(A, M, title) = "The Matrix" M → R1 file:R1/title = "The Matrix"
|
||||
(A, M, cids/Q) = "true" Q → R1 file:R1/cids/M = "true" (synthesised)
|
||||
(B, Q, title) = "Matrix, The" file:R1/cids/Q = "true" (synthesised)
|
||||
file:R1/sizeByte = "734003200"
|
||||
(B, Q, sizeByte) = "734003200"
|
||||
```
|
||||
|
||||
### Reverse-index aliases (CIDs)
|
||||
- A text `cids/<other>` vote on cid X links X and `<other>` into one
|
||||
group. Retracting it does **not** split the group (phase 1). In the merged
|
||||
view, membership is synthesised from `cid_group` — `cids/<cid> = "true"` for
|
||||
every cid of the group — and `cids/*` claims are never merged. Record R = every vote on every cid of R's group, merged per key,
|
||||
each signer counted once per key.
|
||||
- First-seen cid with no link → new group with a fresh rid
|
||||
(`storage.NewUUID()`, UUIDv7 → 26-char Crockford Base32, ULID layout).
|
||||
- Linking two groups → the **older rid survives**; the other group's cids
|
||||
move to it; `/api/events/meta` emits `del` for each key of the losing rid
|
||||
and `set` for the survivor. There is no rid alias table.
|
||||
- `GET /api/meta/<cid>` is a `cid:<cid>` lookup in the cache, then a range
|
||||
read `[file:<rid>/, file:<rid>0)` — any known cid, any hash algorithm, no
|
||||
primary.
|
||||
- Rids are local names: never signed, never on the swarm or gateway wire.
|
||||
Local consumers (meta-sort, meta-fuse, meta-stremio) do hold them, which is
|
||||
why `cid_group` lives in truth.
|
||||
|
||||
Every CID known for a file is registered as its own key, keyed by
|
||||
the **bare CID** — no algorithm prefix. A CIDv1 already encodes its hash
|
||||
function in the multicodec, so naming it again in the key would be
|
||||
redundant (and would let the same digest be indexed under two spellings):
|
||||
There is **no stored `canonical_cid`**. `GetMetadataDocument` ranks the
|
||||
group's cids through `internal/cid/rank.go` (`cid.Better`) and returns the
|
||||
winner as `canonical_cid` in the JSON response only.
|
||||
|
||||
```
|
||||
cid:<bareCid> → <uuid>
|
||||
```
|
||||
## Merge
|
||||
|
||||
Examples:
|
||||
|
||||
```
|
||||
cid:bafkreih5kapbjzq... → 01JKR8XW5T4QMV7AHNJ2DEFGHK
|
||||
cid:bagacbabaec7v3... → 01JKR8XW5T4QMV7AHNJ2DEFGHK
|
||||
```
|
||||
|
||||
The reverse index makes `GET /api/meta/<cid>` a primary-key lookup for any
|
||||
registered CID, independent of which hash algorithm — there is no primary.
|
||||
The record write path (`writeFieldsTx` in `internal/storage/client.go`, used
|
||||
by `SetProperty`, `SetMetadataFlat` and `MergeMetadataFlat`) registers the
|
||||
alias for any field whose name starts with `cids/`, in the same transaction,
|
||||
so writers emit key-set members without knowing the reverse index exists.
|
||||
|
||||
> ⚠ The hook matches **only** the `cids/` prefix. A legacy `cid_*` or
|
||||
> `midhash256` *named* field written to a root is stored verbatim and is
|
||||
> **never indexed** — the record becomes unresolvable by that CID. Writers
|
||||
> must collapse such fields to `cids/<cid>` members before the write. See
|
||||
> `METADATA_KEYS.md` §14.13.
|
||||
|
||||
### Per-root sets
|
||||
|
||||
```
|
||||
file:<uuid>/cids/<bareCid> → "true" — one key per known CID (a key-set)
|
||||
dup:<uuid>/<path> → "" — one key per alternate path with same content
|
||||
```
|
||||
|
||||
`cids` is a **key-set**: one flat `file:<uuid>/cids/<cid>` key per member,
|
||||
which keeps it conflict-free to merge across peers (a single set-valued
|
||||
field would not be).
|
||||
The record's own address (the midhash) is just the `0x1000` member — it has
|
||||
no separate field.
|
||||
|
||||
There is **no stored `canonical_cid`**. The canonical CID is *derived on
|
||||
read*: `GetMetadataDocument` (`internal/storage/cid_resolution.go`) ranks the
|
||||
key-set's members through `internal/cid/rank.go` (`cid.Better`) and returns
|
||||
the winner as the `canonical_cid` field of the JSON response only. Nothing
|
||||
reconciles or persists it — an earlier `reconcileCanonicalCIDLocked` did, and
|
||||
was removed with the flat `cid_*` keys.
|
||||
|
||||
### Global index
|
||||
|
||||
```
|
||||
idx:roots/<uuid> → "" (one key per root; was the Redis SET file:__index__)
|
||||
```
|
||||
|
||||
`internal/storage/client.go::GetAllHashIDs` range-reads `idx:roots/` to
|
||||
enumerate every file without scanning the records.
|
||||
|
||||
### Search structures (derived)
|
||||
|
||||
Two derived structures are maintained on top of `kv`, in the same
|
||||
transaction as every record write. Both are rebuildable from the kv rows and
|
||||
never authoritative (`internal/storage/search.go`):
|
||||
|
||||
- **`kv_field_v`** — a partial expression index
|
||||
`ON kv(lower(substr(k,instr(k,'/')+1)), lower(v), v) WHERE k>='file:' AND k<'file;' AND instr(k,'/')>0`.
|
||||
It answers "which records have field F with a value like V" for **any**
|
||||
field, including ones invented tomorrow — the index is over the key itself.
|
||||
It carries `v` so the substring test is answered from the index alone
|
||||
(covering; 13 ms vs 1.48 s without it, for tmdbid on watch).
|
||||
- **`fts`** — an FTS5 trigram table `fts(rid UNINDEXED, hay)` holding one row
|
||||
per record: its search haystack plus its rid (`idx:fts/<rid>` points at the
|
||||
row). Trigrams make every 3+ character substring indexable.
|
||||
|
||||
`POST /api/metadata/search` uses SQL to build a superset of candidates, then
|
||||
verifies each with the unchanged Go matchers (`recordMatchesFilter`,
|
||||
`matchAllTokens`), so semantics are identical to the old in-memory index.
|
||||
The smallest part (by a capped count) is collected once and sorted; every
|
||||
other filter is probed per page as an EXISTS over the candidate's own key
|
||||
range. The plan, its measurements and the rejected shapes are in
|
||||
[`docs/project-architecture/metacore-storage-engine.md`](../../../docs/project-architecture/metacore-storage-engine.md).
|
||||
Results still come in hashID order and are fresh immediately (no refresh
|
||||
lag). This replaced the API's in-memory `searchIndex` (60 s refresh, ~30 KB
|
||||
RAM per record).
|
||||
|
||||
### Schema-version sentinel
|
||||
|
||||
```
|
||||
meta-core:schema-version → string (currently "2")
|
||||
```
|
||||
|
||||
Set at first boot, checked on every subsequent boot
|
||||
(`internal/storage/schema_version.go`). If the sentinel is missing but
|
||||
the root index is non-empty, the gate refuses to boot (legacy data).
|
||||
If the sentinel is present but doesn't match `SchemaVersion`, the gate
|
||||
also refuses to boot. Alpha-stage policy: operators wipe and restart;
|
||||
there is no automated migration.
|
||||
|
||||
## Why flat string keys?
|
||||
|
||||
The store keeps one flat key per leaf rather than a document or a
|
||||
per-record row. The reasons are:
|
||||
|
||||
1. **Schema-free.** A plugin inventing a field writes a new key; nothing
|
||||
declares it, and the table never needs a migration. The field index
|
||||
covers it automatically.
|
||||
2. **Conflict-free merging across peers.** Key-sets (`cids/<cid>`, and the
|
||||
flat fields in general) merge by union of keys — no read-modify-write of
|
||||
a shared value, nothing to reconcile.
|
||||
3. **Selective updates without read-modify-write.** A plugin writing
|
||||
`tmdb/poster` doesn't have to read the rest of the document.
|
||||
4. **Compatibility with the CID alias hooks.** Field writes can trigger
|
||||
alias-maintenance work in the write path without the plugin caring.
|
||||
|
||||
(Under Redis there was a fourth reason — field-level keyspace notifications
|
||||
fed `meta:events`. That is obsolete: events are now written by the storage
|
||||
engine itself, see [Event streams](#event-streams).)
|
||||
The cache row `file:<rid>/<key>` holds the winner over the group's votes
|
||||
(each signer's newest vote counts once; retractions and ciphertext drop out):
|
||||
most distinct signers (values compared after **trim + case-fold only**, no
|
||||
NFC) → oldest `ts` → byte-wise smallest. The stored value is the winner's
|
||||
exact text; `nsrc` is not stored (`POST /api/merge` returns it). With
|
||||
`?as=<uid>`, the viewer's own live text votes (found via `claim_by_pid`)
|
||||
override at read time; they are never written to the cache. Box-local
|
||||
fields: when several local paths hold the same content, `filePath` is the
|
||||
lexicographically first (size and mtime from the same `local_file` row); a
|
||||
signed `sizeByte` claim wins over the on-disk size. Full rules:
|
||||
implementation basis §7 (`internal/storage/merge.go`).
|
||||
|
||||
## Reads and writes
|
||||
|
||||
All writes go through **one writer goroutine** with group commit: concurrent
|
||||
writes share one SQLite transaction, each under its own SAVEPOINT so a
|
||||
failure rolls back alone. A record write — fields + root index + CID aliases
|
||||
+ FTS row + events — is atomic. (Measured ~835 records/s with 16 concurrent
|
||||
writers of 40-field records.) Reads use a separate connection pool and see
|
||||
the last committed state.
|
||||
| Operation | Endpoint | Store access |
|
||||
|---|---|---|
|
||||
| write metadata | `POST /api/claims` (batch of signed claims) | truth upsert, then the apply loop updates the cache |
|
||||
| removed writes | `PUT/PATCH/DELETE /meta/*`, `POST /meta/{h}/_add/{k}`, `PUT/DELETE /api/metadata/{id}`, `PUT /api/metadata/{id}/property`, `POST /api/metadata/batch` | `410 Gone`, pointing at `POST /api/claims` |
|
||||
| read a record | `GET /api/meta/{cid}`, `GET /api/metadata/{rid}`, `GET /file/{cid}`, `GET /meta/{h}` | `cid:` lookup → `file:<rid>/*` rows, optional `?as=` |
|
||||
| list / search | `GET /api/metadata/hash-ids`, `/list`, `POST /api/metadata/search`, `GET /api/files/tuples`, `/api/schema`, `/api/stats` | `file:` rows, `kv_field_v`, `fts`, `idx:count/*` / `idx:schema/*` |
|
||||
| raw votes | `GET /api/meta/{cid}/claims`, `GET /api/metadata/{rid}/claims` (`?key=`, `?uid=`) | `{rid, cids, claims}`: every live signed claim of the group, all signers, tombstones excluded; 404 for an unknown cid / rid |
|
||||
| merge without storing | `POST /api/merge` `{claims, as?}` → `{fields, nsrc, rejected}` | none — verify + merge in memory (plus `as`'s votes this core holds) |
|
||||
| watcher facts | the watcher, `POST /api/files/register` | `local_file` + `cid_group` |
|
||||
| service scratch | `PUT/GET/DELETE /api/kv/value` | non-reserved cache `kv` rows |
|
||||
| wipe | `POST /api/metadata/clear` | both databases |
|
||||
|
||||
The Go API in `internal/storage/client.go`:
|
||||
|
||||
| Method | Behaviour |
|
||||
|---|---|
|
||||
| `GetMetadataFlat(uuid)` | Range-read `[file:<uuid>/, file:<uuid>0)`, return `map[string]string`. |
|
||||
| `SetMetadataFlat(uuid, m)` | Put each field; add `idx:roots/<uuid>`; auto-register an alias for every `cids/<cid>` member; refresh the FTS row. One transaction. |
|
||||
| `MergeMetadataFlat(uuid, m)` | Like `SetMetadataFlat` but doesn't pre-clear; used by `PATCH /meta/{hash}`. |
|
||||
| `GetProperty(uuid, field)` | Get `file:<uuid>/<field>`. |
|
||||
| `SetProperty(uuid, field, value)` | Put `file:<uuid>/<field>`; auto-register alias if the field is a CID. |
|
||||
| `DeleteRoot(uuid)` | Delete the `file:<uuid>/` range, its duplicates, search row and index membership; unmap every reverse-index entry. |
|
||||
| `Mint(filePath, size, mtimeNano)` | `NewUUID()` + write `filePath/sizeByte/mtimeNano` + add to index. |
|
||||
| `AddAlias(uuid, "<bareCid>")` | Set `cid:<bareCid>` → `<uuid>` and the key-set member `file:<uuid>/cids/<bareCid>` = `"true"`. Nothing to reconcile — canonical is derived on read. |
|
||||
| `ResolveRoot("<bareCid>")` | Get `cid:<bareCid>` → uuid (returns the input unchanged if it is already a UUID root). |
|
||||
| `GetMetadataDocument(uuid)` | Flat fields + `cids` hoisted to an array + `canonical_cid` **derived** by `cid.Better` rank + `duplicates`. Backs `GET /api/meta/{cid}`. |
|
||||
Truth writes go through **one writer goroutine** with group commit; reads use
|
||||
a separate pool and see committed state (WAL). Read paths and shapes are
|
||||
unchanged from v1.x — they are now served from the derived `file:` rows.
|
||||
|
||||
## Reconstruction (flat → nested)
|
||||
|
||||
When the HTTP API returns a document, it reconstructs the flattened
|
||||
storage back into a nested JSON object:
|
||||
The HTTP API reconstructs the merged flat keys into a nested JSON object:
|
||||
|
||||
```
|
||||
flat nested
|
||||
@@ -262,134 +205,122 @@ fileinfo/streamdetails/ streamdetails: { video:
|
||||
streams/0/language = "eng" streams: [ { language: "eng" }, ... ]
|
||||
```
|
||||
|
||||
Rules:
|
||||
- Numeric path segments become array indices on read.
|
||||
- All values are strings on the wire; type interpretation is the caller's
|
||||
job (documented per field in `METADATA_KEYS.md`).
|
||||
- Box-local fields from `local_file` are joined in unsigned; they never
|
||||
appear in `GET /api/meta/{cid}/claims` or on the swarm.
|
||||
|
||||
- `/` is the path separator.
|
||||
- Numeric path segments (`0`, `1`, ...) become array indices on read.
|
||||
- All values are strings on the wire; type interpretation is the
|
||||
caller's job (and is documented per-field in `METADATA_KEYS.md`).
|
||||
- `null`/`undefined` round-trip as empty strings.
|
||||
## Search structures
|
||||
|
||||
## Plugin contribution flow
|
||||
Both live in the cache and are maintained in the same transaction as the
|
||||
merged-row change, so results are fresh as soon as a change is applied:
|
||||
|
||||
Plugins don't talk to the store directly. They write metadata via meta-core's HTTP
|
||||
API (`PUT /meta/{hash}/{key...}` or `PATCH /meta/{hash}`). Each plugin
|
||||
writes under a prefix:
|
||||
- **`kv_field_v`** — an expression index over the field name parsed out of
|
||||
the key, `lower(v)` and `v`, restricted to `file:` rows. Answers
|
||||
"which records have field F with a value like V" for any field, including
|
||||
ones invented tomorrow; covering for the substring test.
|
||||
- **`fts`** — FTS5 trigram `(rid UNINDEXED, hay)`, one row per record: the
|
||||
haystack of its searchable merged fields plus the rid. Every 3+ character substring is
|
||||
indexable.
|
||||
|
||||
```
|
||||
{plugin-id}/<field> single value
|
||||
{plugin-id}/<group>/<field> grouped
|
||||
{plugin-id}/<group>/<n>/<field> indexed (array)
|
||||
```
|
||||
`POST /api/metadata/search` builds a candidate superset in SQL and verifies
|
||||
each hydrated merged record with the Go matchers (`recordMatchesFilter`,
|
||||
`matchAllTokens`); results come in rid order. Plan, rejected shapes and the
|
||||
measurement status are in
|
||||
[`metacore-storage-engine.md`](../../../docs/project-architecture/metacore-storage-engine.md) §8.
|
||||
|
||||
See [`METADATA_KEYS.md`](../../../METADATA_KEYS.md) for the full,
|
||||
authoritative list of fields each plugin writes (content identification,
|
||||
file stats, video classification, anime detection, subtitle references,
|
||||
Jellyfin metadata, FFmpeg stream details, etc.) and the value formats
|
||||
(`int-string`, `lang3`, `path-relfiles`, `csv-set`, ...). This doc
|
||||
deliberately does not duplicate that registry — it would drift.
|
||||
## Schema-version sentinel
|
||||
|
||||
Truth `meta` row `schema-version` = `"3"`, set at first boot and checked on
|
||||
every boot (`internal/storage/schema_version.go`), after both files are
|
||||
opened and before the cache catch-up. A truth file holding the v1 `kv` table
|
||||
(v1.x, `meta-core:schema-version = "2"`) is **explicitly refused**, naming the
|
||||
migration tool; claims without a sentinel, or another version, are refused
|
||||
too; a fresh store is stamped. Alpha policy: no migration code in the
|
||||
service binary; operators wipe and restart. watch.nsl.sh is converted once by
|
||||
the separate one-off tool `packages/meta-core/tools/claims-migrate` (not in
|
||||
the image; rids keep their UUIDs; legacy fields re-signed by a throwaway key;
|
||||
UDL cells re-signed by their account keys — implementation basis §11).
|
||||
|
||||
## Event streams
|
||||
|
||||
Two event streams are kept inside the store, as rows
|
||||
`ev:<stream>/<ms13>-<seq10>` (zero-padded so key order is ID order) plus a
|
||||
high-water mark `evhwm:<stream>` (`internal/kv/stream.go`):
|
||||
|
||||
| Stream | Producer | Contents |
|
||||
|---|---|---|
|
||||
| `file:events` | `internal/watcher` dispatcher | `add` / `change` / `delete` / `rename` / `reset` events for files on the `/files` volume. |
|
||||
| `meta:events` | the storage engine (`internal/kv`) | `{type: set\|del, key, ts}` for every Put/Delete of a `file:` key, appended **in the same transaction** as the change. |
|
||||
| `/api/events/meta` | the cache engine, as `ev:meta:events/…` rows of the cache `kv` table | `{type: set\|del, key, ts}` for every change of a **merged** value, in the same cache transaction as the change |
|
||||
| `/api/events/files` | `internal/watcher` dispatcher (`ev:file:events/…` rows) | `add` / `change` / `delete` / `rename` / `reset` for files on `/files` (unchanged) |
|
||||
|
||||
`set` is emitted only for a **real** change: a Put of the value a key
|
||||
already holds is a no-op — no row write, no index churn, no event (v1.2.0;
|
||||
Redis published `set` even then, and with gateway hits re-persisted on
|
||||
every repeated query most writes were exactly that). `del` is emitted only
|
||||
when the key existed. Retention is ~100k entries, trimmed every 10k appends.
|
||||
|
||||
There is **no boot-time republish** (v1.2.0). A consumer that wants the whole
|
||||
current state asks `/api/events/meta` for `Last-Event-ID: 0` / `0-0` and gets
|
||||
a **snapshot**: one `set` per existing record key, streamed from the kv
|
||||
range, then the live tail from the stream head captured before the snapshot
|
||||
(only the last snapshot event carries an `id:`, so a consumer that drops
|
||||
mid-snapshot gets a fresh one). The old republish wrote every field into a
|
||||
stream that retains only ~100k entries, so a consumer replaying it later
|
||||
never saw most records — and it cost ~6 minutes on every boot.
|
||||
|
||||
The Redis-era path — keyspace notifications drained by `MetaPublisher`
|
||||
through a PSUBSCRIBE / consumer-group hop — is gone (2026-10).
|
||||
|
||||
External consumers do **not** read these streams directly. They subscribe
|
||||
via SSE:
|
||||
|
||||
- `GET /api/events/files` — mirror of `file:events`
|
||||
- `GET /api/events/meta` — mirror of `meta:events`
|
||||
|
||||
The SSE wire contract uses `Last-Event-ID` for resume. IDs keep the Redis
|
||||
`<ms>-<seq>` form so cursors persisted by meta-sort / meta-share /
|
||||
meta-stremio across the switch still compare. Heartbeats are SSE comments
|
||||
(`:keep-alive\n\n`) every 30s of silence. If retention has trimmed past
|
||||
a client's cursor, the handler emits one `event: gap` payload before
|
||||
resuming. See `internal/api/sse_events.go` and
|
||||
[`api-mediated-access.md`](api-mediated-access.md) for the full
|
||||
contract.
|
||||
- A vote that does not change the merged value emits nothing.
|
||||
- A group merge emits `del` for every key of the losing rid and `set` for the
|
||||
survivor's keys.
|
||||
- IDs keep the `<ms>-<seq>` form and stay time-based, so persisted cursors
|
||||
keep comparing, including across a cache rebuild.
|
||||
- `Last-Event-ID: 0` / `0-0` returns a snapshot (one `set` per merged key)
|
||||
then the live tail; only the last snapshot event carries an `id:`.
|
||||
- A trimmed cursor gets one `event: gap`. A cache rebuild emits no per-key
|
||||
events; consumers get `gap` and take a snapshot.
|
||||
- Heartbeats are SSE comments (`:keep-alive\n\n`) every 30 s of silence.
|
||||
Contract: `internal/api/sse_events.go`, [`api-mediated-access.md`](api-mediated-access.md).
|
||||
|
||||
## Debugging
|
||||
|
||||
The database path is `/meta-core/db/meta-core.db` in the image default and
|
||||
on the store app; dev stacks use `/meta-core-db/meta-core.db` (a named
|
||||
volume — the `/meta-core` bind is 9p on Windows dev boxes, where SQLite WAL
|
||||
is unsafe). Always open it **read-only**; writes must go through the API.
|
||||
Paths: truth `/meta-core/db/meta-core.db`, cache `/meta-core/db-cache/cache.db`
|
||||
(image default and store app); dev stacks keep both on named volumes (the
|
||||
`/meta-core` bind is 9p on Windows dev boxes, where SQLite WAL is unsafe).
|
||||
Always open **read-only**; writes go through `POST /api/claims`.
|
||||
|
||||
```bash
|
||||
DB=/meta-core/db/meta-core.db # dev stack: /meta-core-db/meta-core.db
|
||||
Q() { docker exec metacore-app sqlite3 -readonly "$DB" "$1"; }
|
||||
T=/meta-core/db/meta-core.db; C=/meta-core/db-cache/cache.db # dev: the named-volume paths
|
||||
QT() { docker exec metacore-app sqlite3 -readonly "$T" "$1"; }
|
||||
QC() { docker exec metacore-app sqlite3 -readonly "$C" "$1"; }
|
||||
|
||||
# Roots
|
||||
Q "SELECT count(*) FROM kv WHERE k >= 'idx:roots/' AND k < 'idx:roots0'"
|
||||
Q "SELECT substr(k,11) FROM kv WHERE k >= 'idx:roots/' AND k < 'idx:roots0' LIMIT 10"
|
||||
|
||||
# Full document for one root
|
||||
Q "SELECT k,v FROM kv WHERE k >= 'file:01JKR8XW5T4QMV7AHNJ2DEFGHK/' AND k < 'file:01JKR8XW5T4QMV7AHNJ2DEFGHK0'"
|
||||
|
||||
# One property
|
||||
Q "SELECT v FROM kv WHERE k = 'file:01JKR8XW5T4QMV7AHNJ2DEFGHK/fileinfo/duration'"
|
||||
|
||||
# Reverse-index lookup (bare CID — no algorithm prefix)
|
||||
Q "SELECT v FROM kv WHERE k = 'cid:bafkreih...'" # → uuid
|
||||
|
||||
# The cids key-set
|
||||
Q "SELECT k FROM kv WHERE k >= 'file:01JKR8XW5T4QMV7AHNJ2DEFGHK/cids/' AND k < 'file:01JKR8XW5T4QMV7AHNJ2DEFGHK/cids0'"
|
||||
|
||||
# canonical_cid is NOT stored. Derive it via the API instead:
|
||||
curl -k https://metacore-dev.localhost:8083/api/meta/<cid> | jq '.canonical_cid, .cids'
|
||||
|
||||
# Schema-version sentinel
|
||||
Q "SELECT v FROM kv WHERE k = 'meta-core:schema-version'"
|
||||
|
||||
# Streams (server-side; external consumers should use SSE)
|
||||
Q "SELECT k,v FROM kv WHERE k >= 'ev:meta:events/' AND k < 'ev:meta:events0' ORDER BY k DESC LIMIT 10"
|
||||
QT "SELECT rid FROM cid_group WHERE cid = 'bafkreih...'" # cid → rid
|
||||
QT "SELECT cid FROM cid_group WHERE rid = '01JKR8XW5T4QMV7AHNJ2DEFGHK'" # the group (full scan: no rid index)
|
||||
QC "SELECT k, v FROM kv WHERE k >= 'file:01JKR8XW5T4QMV7AHNJ2DEFGHK/' AND k < 'file:01JKR8XW5T4QMV7AHNJ2DEFGHK0'" # merged record
|
||||
QT "SELECT c.key, typeof(c.value), c.value, p.uid, c.ts, c.seq
|
||||
FROM claim c JOIN principal p USING(pid) WHERE c.cid = 'bafkreih...'" # every vote on one cid
|
||||
QT "SELECT name, value FROM meta"; QC "SELECT k, v FROM kv WHERE k LIKE 'meta:%'" # last_seq vs applied_seq
|
||||
QC "SELECT path, midhash FROM local_file WHERE path LIKE '/files/watch/%' LIMIT 10"
|
||||
QC "SELECT k, v FROM kv WHERE k >= 'ev:meta:events/' AND k < 'ev:meta:events0' ORDER BY k DESC LIMIT 10"
|
||||
|
||||
# HTTP equivalents — preferred
|
||||
curl -k 'https://metacore-dev.localhost:8083/api/meta/<cid>' # CID → document
|
||||
curl -k 'https://metacore-dev.localhost:8083/meta' # list root IDs
|
||||
curl -k 'https://metacore-dev.localhost:8083/meta/<uuid>' # document by UUID
|
||||
curl -k 'https://metacore-dev.localhost:8083/api/events/meta' # live SSE
|
||||
curl -k 'https://metacore-dev.localhost:8083/api/kv/info' # KV stats
|
||||
curl -k 'https://metacore-dev.localhost:8083/api/meta/<cid>' # merged document (+ canonical_cid)
|
||||
curl -k 'https://metacore-dev.localhost:8083/api/meta/<cid>/claims' # signed votes
|
||||
curl -k 'https://metacore-dev.localhost:8083/api/events/meta' # live SSE
|
||||
```
|
||||
|
||||
## Key implementation files
|
||||
|
||||
| Component | Path |
|
||||
|---|---|
|
||||
| Storage engine (SQLite kv table, writer, group commit) | `internal/kv/kv.go` |
|
||||
| Event streams as kv rows | `internal/kv/stream.go` |
|
||||
| Flat-key record ops | `internal/storage/client.go` |
|
||||
| Search (field index + FTS) | `internal/storage/search.go` |
|
||||
| Single-writer, group-commit SQLite engine | `internal/kv/` |
|
||||
| Truth, apply loop, merge, local files, counters | `internal/storage/{truth,apply,merge,localfile,client}.go` |
|
||||
| Search (`kv_field_v` + FTS) | `internal/storage/search.go` |
|
||||
| MMUR2 claim encoding + verification | `internal/claim/` (secp256k1 in `internal/identity/`) |
|
||||
| UUIDv7 minting | `internal/storage/uuid.go` |
|
||||
| CID reverse index + aliases + canonical-CID derivation | `internal/storage/cid_resolution.go` |
|
||||
| Canonical-CID rank | `internal/cid/rank.go` |
|
||||
| Schema-version gate | `internal/storage/schema_version.go` |
|
||||
| `meta:events` snapshot for `0`/`0-0` | `internal/api/sse_events.go` (`streamSnapshot`) |
|
||||
| Derived counters (root count, per-field schema) | `internal/storage/client.go` (`onChange`), `internal/schema/indexer.go` |
|
||||
| Watcher → `file:events` dispatcher | `internal/watcher/dispatcher.go` |
|
||||
| SSE event mirror | `internal/api/sse_events.go` |
|
||||
| SSE event mirror + snapshot | `internal/api/sse_events.go` |
|
||||
| Watcher → `local_file`, `/api/events/files` | `internal/watcher/` |
|
||||
| Field semantics (authoritative registry) | [`/METADATA_KEYS.md`](../../../METADATA_KEYS.md) |
|
||||
|
||||
## Previous layout (v1.1–v1.2 `kv` table)
|
||||
|
||||
Kept so old notes and scripts can be read. In v2.0.0 the **truth** database
|
||||
has no `kv` table at all; the **cache** keeps the same key layout for derived
|
||||
rows (see above), but nothing in it is written by clients.
|
||||
|
||||
| Key | Value | Replaced by |
|
||||
|---|---|---|
|
||||
| `file:<rid>/<field>` | one value per field, last writer wins | `claim` (one row per signer) → derived cache `file:` rows |
|
||||
| `cid:<cid>` | `<rid>` | `cid_group` (truth), copied to cache `cid:` rows |
|
||||
| `idx:roots/<rid>`, `idx:count/*`, `idx:schema/*`, `idx:fts/<rid>` | root set, counters, FTS rowid | same keys, derived, in the cache |
|
||||
| `dup:<rid>/<path>` | duplicate path | `local_file` rows |
|
||||
| `udl:rec:<uid>/<cid>/<key>`, `udl:idx:*` | User Data Layer cells (MMUR1, `version`) | `claim` (MMUR2, `ts`) + `claim_by_pid` |
|
||||
| `ev:<stream>/<ms13>-<seq10>`, `evhwm:<stream>` | stream rows | same keys, in the cache |
|
||||
| `btmagnet:*`, `searchcov:*`, `nzb:*` | service scratch | same keys, in the cache |
|
||||
| `meta-core:schema-version` = `"2"` | gate | truth `meta` `schema-version` = `"3"` |
|
||||
|
||||
The flat-key idea survived — a field is still one string key, a key-set
|
||||
member is still inside the key — but the record moved from "one value per
|
||||
field" to "one value per signer per field".
|
||||
|
||||
+202
-295
@@ -2,368 +2,275 @@
|
||||
|
||||
## Status
|
||||
|
||||
**Status:** Implemented (2026 alpha clean-wipe gate landed). Kept for design rationale.
|
||||
**Status:** Design record. Implemented in 2026 alpha (clean-wipe gate);
|
||||
**reframed by meta-core v2.0.0 (claims model)** — see
|
||||
[`docs/implementation/metacore-claims-model.md`](../../../docs/implementation/metacore-claims-model.md)
|
||||
§4. This doc keeps the *why* behind UUID rids and CID symmetry; it is not
|
||||
the schema reference. For what is stored, see
|
||||
[metadata-storage-structure.md](./metadata-storage-structure.md) and
|
||||
`METADATA_KEYS.md` §2/§14.13.
|
||||
|
||||
> ⚠ **Two parts of this proposal were superseded before shipping.** This doc
|
||||
> is the design record; it is *not* the schema reference. For what is actually
|
||||
> stored, see `METADATA_KEYS.md` §2/§14.13 and
|
||||
> [metadata-storage-structure.md](./metadata-storage-structure.md).
|
||||
>
|
||||
> 1. **`canonical_cid` is not a stored key.** The proposed reconciler
|
||||
> (`reconcileCanonicalCIDLocked`) was never kept — a stored scalar isn't
|
||||
> conflict-free to merge across peers. The canonical CID is **derived on
|
||||
> read** by ranking the `cids` members (`internal/cid/rank.go` →
|
||||
> `cid.Better`), and appears only in the `GET /api/meta/{cid}` response.
|
||||
> 2. **`cids` is a key-set, not a Redis SET, and holds bare CIDs, not
|
||||
> `<algo>:<value>` tokens.** One flat STRING per member:
|
||||
> `file:<uuid>/cids/<bareCid>` = `"true"`. The reverse index is likewise
|
||||
> `cid:<bareCid>` → uuid. A CIDv1's multicodec already names its hash
|
||||
> function, so the `<algo>:` prefix was redundant.
|
||||
>
|
||||
> 3. **The store is no longer Redis (2026-10).** The same keys now live in
|
||||
> one string→string table of an embedded SQLite file; Redis SETs became
|
||||
> one key per member (`file:__index__` → `idx:roots/<uuid>`,
|
||||
> `file:<uuid>/duplicates` → `dup:<uuid>/<path>`). Redis types named
|
||||
> below are the shape at the time.
|
||||
>
|
||||
> Text below that references either shape describes the *proposal*, not the
|
||||
> code. The rank ordering and everything else did ship as written.
|
||||
What still holds, unchanged since the original proposal:
|
||||
|
||||
## Motivation
|
||||
- **No CID is privileged.** Any known CID resolves to its record; all hash
|
||||
types are symmetric.
|
||||
- **Rids are UUIDv7**, 26-char Crockford Base32, **minted locally** with no
|
||||
central allocator.
|
||||
- **`canonical_cid` is derived on read** by rank (`internal/cid/rank.go` →
|
||||
`cid.Better`), never stored.
|
||||
- **`cids` is a key-set of bare CIDs** (`cids/<bareCid>` = `"true"`), no
|
||||
`<algo>:` prefix — a CIDv1's multicodec already names the hash function.
|
||||
|
||||
Today, every metadata entry in Redis is keyed by a content hash, specifically
|
||||
midhash256:
|
||||
What changed in v2.0.0:
|
||||
|
||||
> 1. **The rid is a hidden local name for a group of cids.** Metadata is no
|
||||
> longer stored under `file:<uuid>/<field>`. Every value is a signed claim
|
||||
> about a **cid** (`claim(uid, cid, key) → value`); `cid_group(cid → rid)`
|
||||
> says which cids form one record. The rid is **derived from cid groups,
|
||||
> never signed and never on the wire** (swarm, gateway); each box mints its
|
||||
> own.
|
||||
> 2. **The `cid:<cid>` reverse index is gone**, replaced by the `cid_group`
|
||||
> table (still a primary-key lookup, so still O(1)-class).
|
||||
> 3. **meta-core no longer writes `cids/*` members.** They are signed votes
|
||||
> from the writers that learn a cid (e.g. the full-hash plugin); a non-null
|
||||
> `cids/<other>` vote links two groups. Only the watcher's midhash enters
|
||||
> `cid_group` directly.
|
||||
> 4. **`duplicates` / `dup:<uuid>/<path>` are gone.** Box-local paths live in
|
||||
> the cache's `local_file(path → midhash, size, mtime)`; several paths with
|
||||
> the same midhash are the duplicates.
|
||||
>
|
||||
> Earlier superseded points, kept for the record: the proposed
|
||||
> `reconcileCanonicalCIDLocked` never shipped; `cids` never was a Redis SET
|
||||
> of `<algo>:<value>` tokens; Redis itself went in v1.1.0 (2026-10). Text
|
||||
> below marked *(proposal)* or *(v1.x)* describes those shapes, not v2.0.0.
|
||||
|
||||
## Motivation (history)
|
||||
|
||||
Before UUID rids, every metadata entry in Redis was keyed by a content hash,
|
||||
specifically midhash256:
|
||||
|
||||
```
|
||||
file:midhash256:abc.../filePath
|
||||
file:midhash256:abc.../title
|
||||
file:midhash256:abc.../fileinfo/duration
|
||||
file:midhash256:abc.../tmdb/poster
|
||||
...
|
||||
```
|
||||
|
||||
Other digests (sha256, IPFS CIDs, BitTorrent infohashes) live as fields
|
||||
*inside* that entry:
|
||||
Other digests (sha256, IPFS CIDs, BitTorrent infohashes) lived as fields
|
||||
*inside* that entry (`cid_sha256`, `cid_ipfs`). Two problems:
|
||||
|
||||
```
|
||||
file:midhash256:abc.../cid_sha256 = sha256:def...
|
||||
file:midhash256:abc.../cid_ipfs = bafy...
|
||||
```
|
||||
1. **midhash256 was privileged by accident.** It *was* the key, while every
|
||||
other CID was a secondary field. Hash-algorithm changes, or new digest
|
||||
types added by plugins, forced rewrites of every root. The privilege
|
||||
itself was the issue, not which hash held it.
|
||||
2. **CID lookup was O(n).** "Given any CID, return the metadata" scanned
|
||||
every entry (`storage/client.go::LookupPathByCID`). Unacceptable for a
|
||||
public API, especially when peers query by any CID variant.
|
||||
|
||||
Two problems with this:
|
||||
|
||||
1. **midhash256 is privileged by accident.** It IS the key, while every other
|
||||
CID is a secondary field. There's nothing intrinsically special about
|
||||
midhash256 — it's just the first hash computed. Hash-algorithm changes, or
|
||||
new digest types added by plugins, force rewrites of every root. The
|
||||
"promotion" workarounds people reach for (re-keying entries when a stronger
|
||||
hash arrives) solve the wrong problem; the privilege itself is the issue.
|
||||
|
||||
2. **CID lookup is O(n).** Resolving "given any CID, return that file's
|
||||
metadata" currently scans every indexed entry (see
|
||||
`storage/client.go::LookupPathByCID`, which acknowledges this in a
|
||||
comment). Acceptable for admin tooling, unacceptable for a public API
|
||||
surface — especially when peers on meta-share may query by any CID variant.
|
||||
|
||||
The product requirement is clear: **any CID a client knows about must resolve
|
||||
to that file's metadata in O(1).** IPFS-compatible CIDs are the externally
|
||||
meaningful identity for content retrieval; the internal storage layout should
|
||||
not bias toward any one digest type.
|
||||
The requirement: **any CID a client knows must resolve to that content's
|
||||
metadata in O(1)**, and the storage layout must not bias toward any digest
|
||||
type. The claims model adds a second one: **what is signed must name
|
||||
something every peer agrees on** — a rid minted at random on one box does
|
||||
not qualify, a cid does.
|
||||
|
||||
## Goal
|
||||
|
||||
A single externally-meaningful contract:
|
||||
|
||||
```
|
||||
GET /api/meta/<cid> → full metadata document for that file
|
||||
GET /api/meta/<cid> → full (merged) metadata document for that content
|
||||
```
|
||||
|
||||
Where `<cid>` may be any supported type (midhash256, sha256, IPFS, btih,
|
||||
sha3, future additions). All CID types are treated symmetrically. No CID type
|
||||
is "the key."
|
||||
`<cid>` may be any supported type (midhash, sha256, IPFS, btih, sha3, future
|
||||
additions), all treated symmetrically.
|
||||
|
||||
## Design
|
||||
## Design (v2.0.0)
|
||||
|
||||
### Two-tier key structure
|
||||
|
||||
The root key becomes an opaque internal UUID; CIDs are aliases that point to
|
||||
it.
|
||||
### Votes name cids; a rid names a group
|
||||
|
||||
```
|
||||
file:<uuid>/<property...> ← canonical storage (UUIDv7, never disclosed externally)
|
||||
cid:<cid> → <uuid> ← reverse index (one Redis key per known CID)
|
||||
votes (signed, travel) derived (local, persistent: truth cid_group)
|
||||
(A, M, title) = "The Matrix" M → R1
|
||||
(A, M, cids/Q) = "true" ← alias vote Q → R1
|
||||
(B, Q, title) = "Matrix, The"
|
||||
(B, Q, sizeByte)= "734003200" record R1 = every vote on M or Q, merged per key
|
||||
```
|
||||
|
||||
The metadata keyspace under `file:<uuid>/` remains open and flat, exactly as
|
||||
today — anything that wrote `file:<midhash>/fileinfo/duration` now writes
|
||||
`file:<uuid>/fileinfo/duration`. Only the root identifier changes.
|
||||
- **First-seen cid, no link** → new group, fresh UUIDv7 rid.
|
||||
- **`cids/<other>` vote on X** → X and `<other>` join one group.
|
||||
- **Link between two existing groups** → the group with the **older rid
|
||||
survives**; the other's cids move to it; the losing rid gets a `del` event
|
||||
for each of its keys, the survivor `set` events. No `rid_alias`: consumers
|
||||
already handle deletes, and cids re-resolve.
|
||||
- **Reading record R** = every vote on every cid of R's group, merged per key
|
||||
(one signer counts once per key), precomputed into the cache's
|
||||
`file:<rid>/<field>` rows (the v1 key layout, now derived).
|
||||
- **Retracting a `cids/<other>` link does not split the group** (phase 1):
|
||||
the tombstone drops out of the merge, `cid_group` keeps both cids under one
|
||||
rid. In the record view, membership is synthesised from `cid_group`
|
||||
(`cids/<cid> = "true"` for every cid of the group).
|
||||
|
||||
**Example layout:**
|
||||
Why keep a rid at all, if votes never mention it: local consumers (meta-sort,
|
||||
meta-fuse, meta-stremio) and the merged view need one stable name per record,
|
||||
and a group's cid set grows over time. Why `cid_group` lives in **truth**
|
||||
although it could be recomputed from `cids/*` votes: the rid names are minted
|
||||
at random and consumers hold them, so a recomputation would rename every
|
||||
record.
|
||||
|
||||
```
|
||||
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/filePath = /watch/Inception.mkv
|
||||
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/size = 4500000000
|
||||
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/mtime = 1729872000000000000
|
||||
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/cids/bagacbabaec… = "true" ← midhash (record address)
|
||||
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/cids/bafybeigd… = "true" ← sha2-256
|
||||
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/duplicates = SET{/watch/dup1.mkv, /watch/dup2.mkv}
|
||||
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/title = "Inception"
|
||||
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/fileinfo/duration = 7200.5
|
||||
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/tmdb/poster = bafy...
|
||||
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/subtitle/eng = ...
|
||||
### Why UUIDv7 (still valid)
|
||||
|
||||
cid:bagacbabaec… → 01JKR8XW5T4QMV7AHNJ2DEFGHK
|
||||
cid:bafybeigd… → 01JKR8XW5T4QMV7AHNJ2DEFGHK
|
||||
```
|
||||
- **128 bits**: collision-free under distributed minting; no allocator. Each
|
||||
box minting its own rids for the same content is now the normal case, not a
|
||||
corner case — which is exactly why rids never travel.
|
||||
- **Time-sortable prefix**: rids minted in a window sort together
|
||||
(`k >= 'file:01JKR8' AND k < 'file:01JKR9'` on the cache's merged rows),
|
||||
and "older rid
|
||||
survives" on a group merge is a plain string comparison.
|
||||
- **Monotonic-ish**: adjacent rids cluster in the cache's `file:<rid>/…`
|
||||
key range.
|
||||
|
||||
*(As shipped — the proposal originally wrote `cids` as a Redis SET of
|
||||
`<algo>:<value>` tokens plus a `canonical_cid` scalar. See the Status note.)*
|
||||
Encoded as 26-char Crockford Base32 (ULID-compatible) for compactness and
|
||||
case-insensitive matching. Migrated records kept their v1.x UUIDs as rids.
|
||||
|
||||
### UUID format: UUIDv7
|
||||
### The `cids` key-set
|
||||
|
||||
UUIDv7 chosen over UUIDv4 or 64-bit random:
|
||||
`cids/<bareCid>` = `"true"`, one claim per member, by whichever signer knows
|
||||
the link. A key-set merges conflict-free: a member is present while at least
|
||||
one vote supports it, and removal is a signed retraction. That is the reason
|
||||
the proposal's Redis SET was reversed before shipping, and it is what lets
|
||||
the claims model treat `cids` with no special merge code.
|
||||
|
||||
- **128 bits**: collision-free under distributed minting; no central allocator.
|
||||
- **Time-sortable prefix**: `KEYS file:01JKR8*` retrieves entries minted in a
|
||||
given window. Restores some of the debuggability lost by going opaque.
|
||||
- **Monotonic-ish**: adjacent UUIDs cluster on disk; modestly better SCAN
|
||||
locality.
|
||||
Box-local file facts (`filePath`, size, `mtimeNano`, midhash) are **not**
|
||||
claims — meta-core cannot sign and the disk can always be rescanned. The
|
||||
watcher records them in `local_file` and puts the midhash into `cid_group`;
|
||||
the record view joins them in unsigned. When several local paths hold the
|
||||
same content, `filePath` is the lexicographically first one.
|
||||
|
||||
Encoded as a 26-character Crockford Base32 string (ULID-compatible) for
|
||||
compactness and case-insensitive matching.
|
||||
### Canonical CID selection (unchanged)
|
||||
|
||||
### New root fields owned by meta-core
|
||||
|
||||
The metadata schema is open — any writer can put any key under
|
||||
`file:<uuid>/`. This proposal does not change that. It adds three keys whose
|
||||
sole writer is meta-core itself:
|
||||
|
||||
| Key | Type (as shipped) | Purpose |
|
||||
|-----|------|---------|
|
||||
| `cids/<bareCid>` | STRING `"true"` — a *key-set*, one key per member | All CIDs known for this file, midhash included. Mirror of the `cid:*` reverse index. Source of truth for delete-time GC. **Proposed as a Redis SET of `<algo>:<value>` tokens; shipped as a key-set of bare CIDs** — a key-set merges conflict-free across peers and the multicodec already names the algorithm. |
|
||||
| `duplicates` | Redis SET at the time; since 2026-10 one `dup:<uuid>/<path>` key per path | Additional file paths whose content matches this entry. One UUID = one set of metadata; multiple physical files may share that content. Consumed by meta-dup. |
|
||||
| ~~`canonical_cid`~~ | *(not stored)* | **Dropped.** Was to hold the externally-presented CID. Now derived on read by ranking the `cids` members; surfaced only on `GET /api/meta/{cid}`. |
|
||||
|
||||
Existing meta-core-owned fields (`filePath`, `size`, `mtime`, `midhash256`)
|
||||
keep their names and writers. Plugin-written fields (`tmdb/*`, `fileinfo/*`,
|
||||
`subtitle/<lang>`, …) are unaffected.
|
||||
|
||||
~~Redis SET is a departure from the all-strings convention in the current
|
||||
storage doc, but `file:__index__` is already a SET — so the precedent exists
|
||||
for using sets where they're the right structure.~~ **Reversed for `cids`.**
|
||||
The all-strings convention won: `cids` ships as flat `cids/<cid>` STRING keys
|
||||
so that two peers merging the same record can never disagree about set
|
||||
membership. `duplicates` remained a Redis SET (it is leader-local and never
|
||||
merged across peers) until the 2026-10 SQLite switch, which flattened it to
|
||||
`dup:<uuid>/<path>` keys.
|
||||
|
||||
### Canonical CID selection
|
||||
|
||||
The canonical CID is a label, not a key — so nothing external breaks when it
|
||||
changes. **As shipped it is not written at all**: it is computed on every read
|
||||
from the `cids` members, which makes the "update" cost zero and removes the
|
||||
reconciler entirely.
|
||||
|
||||
Selection rule, highest-wins (this part shipped as designed —
|
||||
`internal/cid/rank.go`):
|
||||
The canonical CID is a label, not a key, so nothing external breaks when it
|
||||
changes. It is computed on every read from the group's cids:
|
||||
|
||||
1. IPFS CID (any variant)
|
||||
2. SHA-256 / SHA-3 family
|
||||
3. BitTorrent infohash (v2 preferred over v1)
|
||||
4. midhash256
|
||||
|
||||
Ranked by **utility for external retrieval**, not by cryptographic strength.
|
||||
Ranked by **utility for external retrieval**, not cryptographic strength:
|
||||
midhash256 and sha256 are both 256-bit, but only sha256/IPFS CIDs are
|
||||
externally meaningful for content addressing. btih ranks above midhash because
|
||||
it enables swarm interop even if it's weaker as a digest.
|
||||
externally meaningful; btih ranks above midhash because it enables swarm
|
||||
interop. The full ladder (including the custom locator codecs) lives in
|
||||
`internal/cid/rank.go`, mirrored in meta-search / meta-share
|
||||
(`scripts/check-cid-vectors.sh`).
|
||||
|
||||
~~Reconciler runs on the leader, recomputes `canonical_cid` whenever
|
||||
`AddAlias` registers a new CID for an entry.~~ **Not shipped** — there is no
|
||||
reconciler. `GetMetadataDocument` folds the members through `cid.Better` on
|
||||
each read.
|
||||
## Lifecycles (v2.0.0)
|
||||
|
||||
## Internal operations
|
||||
|
||||
The meta-core storage interface adds five operations. None are exposed
|
||||
externally — they're internal to meta-core. External callers see only the
|
||||
CID-based API.
|
||||
|
||||
```go
|
||||
// Mint creates a new root with no aliases. Called when the watcher
|
||||
// computes a midhash for a previously-unseen file. Returns the new UUID.
|
||||
Mint(filePath string, size int64, mtime int64) (uuid string, err error)
|
||||
|
||||
// AddAlias registers a CID as resolving to this UUID. Updates both the
|
||||
// cid:<cid> reverse index and the file:<uuid>/cids/<cid> key-set member.
|
||||
// (As shipped: no reconciliation step — canonical is derived on read.)
|
||||
AddAlias(uuid string, cid string) error
|
||||
|
||||
// GetByCID resolves any known CID to its UUID. Returns ("", nil) if unknown.
|
||||
GetByCID(cid string) (uuid string, err error)
|
||||
|
||||
// MergeIfAliasExists is called when the watcher mints a fresh UUID for a
|
||||
// file path but, on hashing, discovers the resulting CID already belongs
|
||||
// to a different UUID. Merges the new (mostly-empty) UUID into the
|
||||
// existing one, transferring filePath into the duplicates set, and
|
||||
// returns the canonical UUID.
|
||||
MergeIfAliasExists(newUUID, cid string) (canonicalUUID string, merged bool, err error)
|
||||
|
||||
// DeleteRoot removes the entire entry, including every cid:* alias and
|
||||
// every file:<uuid>/* key. Used for file deletion events.
|
||||
DeleteRoot(uuid string) error
|
||||
```
|
||||
|
||||
### Lifecycle: file arrival
|
||||
### File arrival
|
||||
|
||||
```
|
||||
1. Watcher sees /watch/foo.mkv created.
|
||||
2. Watcher computes midhash256 → "midhash256:abc...".
|
||||
3. Watcher calls GetByCID("midhash256:abc...").
|
||||
→ uuid_X (existing entry) or "" (new file).
|
||||
|
||||
4a. If existing (uuid_X):
|
||||
- Add "/watch/foo.mkv" to file:<uuid_X>/duplicates.
|
||||
- No new root created.
|
||||
- meta-dup is notified via the existing keyspace event channel.
|
||||
|
||||
4b. If new:
|
||||
- Mint("/watch/foo.mkv", size, mtime) → uuid_Y.
|
||||
- AddAlias(uuid_Y, "midhash256:abc...").
|
||||
- Plugins now process the file, referenced by midhash256:abc...
|
||||
(UUID is never disclosed to them.)
|
||||
|
||||
5. Later, the full-hash plugin computes sha256.
|
||||
- Plugin reports back: "for CID midhash256:abc..., sha256 is sha256:def...".
|
||||
- meta-core resolves midhash256:abc → uuid_Y via GetByCID.
|
||||
- meta-core calls AddAlias(uuid_Y, "sha256:def...").
|
||||
- Nothing else to do: the next read of this entry derives a canonical CID
|
||||
of sha2-256, because it outranks the midhash.
|
||||
1. Watcher sees /files/watch/foo.mkv, computes its midhash M.
|
||||
2. local_file: /files/watch/foo.mkv → (M, size, mtime).
|
||||
3. cid_group: M already grouped → nothing; otherwise M → fresh rid.
|
||||
4. Plugins process the file by cid M (the rid is never disclosed to them)
|
||||
and sign claims about M with their own keys.
|
||||
5. The full-hash plugin computes sha256 S and signs (plugin, M, cids/S) = "true".
|
||||
→ S joins M's group (or the two groups merge, older rid surviving).
|
||||
→ The next read derives canonical_cid = S, because it outranks the midhash.
|
||||
```
|
||||
|
||||
### Lifecycle: file deletion
|
||||
A second path with the same midhash is just another `local_file` row: same
|
||||
cid, same group, same record; the record's `filePath` is the
|
||||
lexicographically first of the paths.
|
||||
|
||||
```
|
||||
1. Watcher sees /watch/foo.mkv deleted.
|
||||
2. Resolve which entry owns this path:
|
||||
a. If /watch/foo.mkv is in some entry's duplicates set:
|
||||
- Remove from set. Done.
|
||||
b. If /watch/foo.mkv is the filePath of some entry:
|
||||
- If duplicates is non-empty: promote one duplicate to filePath.
|
||||
- If duplicates is empty: DeleteRoot(uuid).
|
||||
- Iterates the cids set, deletes each cid:<cid> reverse-index key.
|
||||
- Deletes every file:<uuid>/* key.
|
||||
- Removes uuid from file:__index__.
|
||||
```
|
||||
### File deletion
|
||||
|
||||
The watcher removes the path's `local_file` row; the record view loses that
|
||||
`filePath`. Claims are signed truth from their writers and are not deleted
|
||||
by meta-core (the store is permanent and additive,
|
||||
`docs/project-architecture/metacore-as-permanent-store.md`).
|
||||
|
||||
*(v1.x)* Deletion promoted a duplicate to `filePath`, or called `DeleteRoot`,
|
||||
which removed every `cid:*` alias and every `file:<uuid>/*` key.
|
||||
|
||||
## Public API
|
||||
|
||||
```
|
||||
GET /api/meta/<cid>
|
||||
→ 200 { "filePath": "...", "canonical_cid": "...",
|
||||
"cids": [...], "title": "...", "tmdb": {...}, ... }
|
||||
→ 404 if no such CID
|
||||
|
||||
GET /api/file/<cid> (already exists — implementation now O(1))
|
||||
→ streams file content
|
||||
GET /api/meta/<cid> → 200 merged document { "filePath", "canonical_cid", "cids": [...], ... }
|
||||
(optional ?as=<uid>: the viewer's own votes win)
|
||||
→ 404 if no such cid
|
||||
GET /api/meta/<cid>/claims → {rid, cids, claims}: every live signed claim of the cid's
|
||||
group, all signers (?key=, ?uid=); 404 if no such cid
|
||||
GET /api/metadata/<rid>/claims → the same, addressed by rid (local consumers)
|
||||
```
|
||||
|
||||
The UUID is **not** included in the response. Clients address everything by
|
||||
CID; meta-core's choice of internal identifier remains a private
|
||||
implementation detail and may change.
|
||||
The rid is not part of the cid-addressed contract and never appears in a
|
||||
claim. It is visible only to local consumers that list or address records by
|
||||
rid (`/api/metadata/{rid}`, `/api/metadata/{rid}/claims`, `hash-ids`, SSE event
|
||||
keys; the claims routes also echo it as `rid`), and may change on a
|
||||
group merge (delete event, then re-resolve by cid).
|
||||
|
||||
Auth: `/api/meta/<cid>` follows the same pattern as `/api/file/<cid>` and
|
||||
bypasses the auth perimeter. It's the metadata twin of the content endpoint,
|
||||
and meta-share calls it from federated search responses.
|
||||
Auth: `/api/meta/<cid>` bypasses the perimeter like `/api/file/<cid>` — it is
|
||||
the metadata twin of the content endpoint.
|
||||
|
||||
## Cross-service impact
|
||||
## Cross-service impact (v2.0.0)
|
||||
|
||||
This is a meta-core schema change. Other services interact only through CIDs,
|
||||
so the impact is contained but real.
|
||||
- **Writers** (plugins, self-publishing feeders, gateway core, meta-share,
|
||||
meta-search): address claims by cid only and sign them; they never resolve
|
||||
a rid.
|
||||
- **meta-fuse, meta-stremio**: unchanged — merged reads and SSE keep their
|
||||
shape; a group merge reaches them as ordinary `del` + `set` events.
|
||||
- **meta-search / meta-share**: announce the derived canonical cid (each
|
||||
mirrors the Go rank) and relay raw claims; every cid of the group travels as
|
||||
`cids/*` claims, so a peer querying by any digest gets a hit.
|
||||
|
||||
- **meta-sort**: writers must resolve plugin callbacks to a UUID before
|
||||
writing. The existing `LeaderClient` gains a `getByCID` cache.
|
||||
- **meta-fuse**: VFS rebuild already iterates `file:__index__`; only the
|
||||
ID format changes.
|
||||
- **meta-stremio**: queries by CID; needs no logic change beyond its storage
|
||||
client.
|
||||
- **meta-dup**: gains a cleaner data source — duplicates are now explicit in
|
||||
the `duplicates` set, not implicit in colliding writes.
|
||||
- **meta-search / meta-share**: announce the *derived* canonical CID rather
|
||||
than the midhash — each mirrors the Go rank in `ingest::canonical_cid_from_metadata`
|
||||
rather than reading a stored field. Federated search responses include all
|
||||
CIDs from the `cids` key-set so any peer querying by any digest gets a hit.
|
||||
(Written before the 2026-06 discovery/transport split; discovery is now
|
||||
meta-search's job.)
|
||||
- **Plugins**: no change. They speak CIDs only.
|
||||
## Migration (history)
|
||||
|
||||
## Migration
|
||||
> **The v1.0 sweep below was never run.** The alpha shipped behind a
|
||||
> clean-wipe gate. The v1.x → v2.0.0 conversion of watch.nsl.sh is a
|
||||
> one-off tool outside the service image, `packages/meta-core/tools/claims-migrate`
|
||||
> (canonical cid by rank per record; `cid:` aliases → `cid_group`, rids keep
|
||||
> their UUIDs, and the `cid:` index wins when a record's `cids/*` member
|
||||
> disagrees; fields re-signed by a throwaway legacy key); every other box
|
||||
> starts empty. Implementation basis §11.
|
||||
|
||||
> **Historical — this sweep was never run.** The alpha shipped behind a
|
||||
> clean-wipe gate (Redis was flushed and the library re-ingested), so no
|
||||
> in-place migration code exists. Kept for the record.
|
||||
|
||||
One-time sweep, idempotent, runs on first boot after upgrade:
|
||||
|
||||
```
|
||||
For each hashID in file:__index__ (current shape: midhash256:abc, sha256:def, …):
|
||||
1. Mint a fresh uuid_Y.
|
||||
2. AddAlias(uuid_Y, hashID). The existing primary becomes an alias.
|
||||
3. For each cid_* field on the old entry, AddAlias(uuid_Y, that value).
|
||||
4. RENAME file:<hashID>/<property> → file:<uuid_Y>/<property> for every
|
||||
property (Lua script, one entry at a time; non-atomic across entries,
|
||||
but each entry stays internally consistent).
|
||||
5. Reconcile canonical_cid.
|
||||
6. Remove hashID from file:__index__, add uuid_Y.
|
||||
```
|
||||
|
||||
Crash-recoverable: re-running the sweep finds entries with both old and new
|
||||
roots and resumes by detecting which step completed (presence of
|
||||
`file:<uuid>/cids` indicates step 3 finished; absence of any
|
||||
`file:<hashID>/*` keys indicates step 4 finished).
|
||||
*(proposal)* One-time sweep over `file:__index__`: mint a uuid, `AddAlias`
|
||||
the old hashID and every `cid_*` field, `RENAME` every property under the new
|
||||
root, reconcile canonical, swap the index entry; crash-recoverable by
|
||||
detecting which step completed.
|
||||
|
||||
## Decisions recorded
|
||||
|
||||
| Question | Decision | Notes |
|
||||
|---|---|---|
|
||||
| When is the UUID minted? | At first-hash | Pragmatic for the initial migration. Long-term target: at discovery, fully decoupled from hashing. Tracked as an open question below. |
|
||||
| UUID format | UUIDv7, 26-char Crockford Base32 | Time-sortable, 128-bit, distributed-safe. |
|
||||
| UUID visible externally? | No | Pure meta-core internal. Clients address by CID only. |
|
||||
| Merge semantics | One UUID per content; additional paths into `duplicates` | One file (logical) = one metadata set. Multiple physical paths supported via `duplicates`. |
|
||||
| Canonical CID rank | IPFS > sha-family > btih > midhash | Utility-weighted, not strength-weighted. |
|
||||
| ~~`cids` and `duplicates` field type~~ | ~~Redis SET~~ → **`cids` is a key-set of bare CIDs; `duplicates` stayed a SET** | Reversed before shipping. Flat `cids/<cid>` STRING keys merge conflict-free across peers; a SET does not. See §14.13 of `METADATA_KEYS.md`. |
|
||||
| ~~Stored `canonical_cid`~~ | **Dropped — derived on read** | A stored scalar is not conflict-free to merge. `GetMetadataDocument` ranks the `cids` members per request. |
|
||||
| Reverse-index format | Flat `cid:<cid>` STRING keys | Simple, scan-friendly, no hot-key contention. |
|
||||
| When is the rid minted? | When a cid is first seen with no link to a known group | v1.x: at first hash of a watched file. Now any first-seen cid (watcher midhash or a claim about a new cid) mints one |
|
||||
| Rid format | UUIDv7, 26-char Crockford Base32 | Time-sortable, 128-bit, distributed-safe |
|
||||
| Rid visible externally? | No — never signed, never on the swarm/gateway wire | Local HTTP consumers may hold it |
|
||||
| What do signed votes name? | A cid | Every peer agrees on a cid; rids differ per box |
|
||||
| Record identity | A group of cids (`cid_group`, in truth) | Linked by `cids/<other>` votes |
|
||||
| Group merge | Older rid survives; losing rid → `del` events | No `rid_alias` (out of scope for phase 1) |
|
||||
| Duplicate paths | `local_file` rows sharing a midhash (cache) | v1.x: `duplicates` SET, then `dup:<uuid>/<path>` |
|
||||
| Canonical CID rank | IPFS > sha-family > btih > midhash, derived on read | Utility-weighted; never stored |
|
||||
| `cids` shape | Key-set of bare CIDs, one claim per member | Merges with the generic rule; a SET would not |
|
||||
| Reverse index | `cid_group(cid PK → rid)` in truth, copied to cache `cid:<cid>` rows for reads | v1.x: flat `cid:<cid>` keys were the only copy |
|
||||
| Unlinking | A retracted `cids/*` link does not split a group | Phase 1 |
|
||||
|
||||
## Open questions
|
||||
|
||||
- **Discovery-time UUID minting.** The cleaner long-term model: watcher mints
|
||||
a UUID at discovery (before hashing), then `AddAlias` runs when the midhash
|
||||
arrives. Requires an explicit "reconcile if alias already exists" step
|
||||
because two paths with the same content would each get their own UUID until
|
||||
hashing reveals the collision. Worth doing eventually; not required for
|
||||
this iteration.
|
||||
|
||||
- ~~**meta-share announce strategy.**~~ **Settled.** The record carries *every*
|
||||
member of `cids` on the wire (`Record.cids`) and addresses itself by the
|
||||
single derived canonical (`Record.cid`), so a peer querying by any digest
|
||||
gets a hit without the DHT footprint of announcing each one separately.
|
||||
|
||||
- **IPFS gateway endpoint.** Should meta-core expose `/ipfs/<cid>` as a
|
||||
content-streaming endpoint for external IPFS clients? Adjacent to this
|
||||
design but not part of it.
|
||||
|
||||
- **Keyspace-notification consumers.** Today's parsers (`schema/indexer.go`)
|
||||
handle `file:<type>:<hash>/...` with a special split rule for the colon
|
||||
inside `midhash256:abc`. With UUIDs (no colons), `parseFieldPath` simplifies
|
||||
— but any external consumer subscribed by hashID pattern needs updating.
|
||||
Inventory those before flipping the migration.
|
||||
- **Discovery-time rid minting.** Minting at discovery (before hashing)
|
||||
would give two same-content paths two rids until hashing links them. The
|
||||
claims model already handles that case — it is an ordinary group merge —
|
||||
so this is now a policy choice, not a missing mechanism.
|
||||
- **IPFS gateway endpoint.** Should meta-core expose `/ipfs/<cid>` for
|
||||
external IPFS clients? Adjacent, not part of this design.
|
||||
- ~~**meta-share announce strategy.**~~ Settled: the canonical cid addresses
|
||||
the record, every group cid travels as a `cids/*` claim.
|
||||
- ~~**Keyspace-notification consumers.**~~ Obsolete since Redis left.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- **Not** a content-replication or IPFS-pinning system. Registering an IPFS
|
||||
CID alias does not pin the file anywhere — it only states "if somebody
|
||||
hashes this file with IPFS's chunker, this is the CID they'd get."
|
||||
- **Not** a versioning scheme. Metadata is a snapshot of current state;
|
||||
historical versions are out of scope.
|
||||
- **Not** a strong-consistency system. Reverse-index writes happen under the
|
||||
leader lock; reads from followers may briefly miss a freshly-added alias.
|
||||
- **Not** a content-replication or pinning system. A `cids/<ipfs-cid>` vote
|
||||
pins nothing; it states "hashing these bytes with IPFS's chunker gives this
|
||||
CID."
|
||||
- **Not** a history. `ts` versions each `(uid, cid, key)` and the newest vote
|
||||
replaces the older one; only tombstones are kept, to block replay.
|
||||
- **Not** strongly consistent across the two databases: the cache is applied
|
||||
after the truth commit. `POST /api/claims` waits for the apply before
|
||||
answering, so the writer reads its own write; another reader may briefly
|
||||
miss a just-accepted claim. It is behind, never wrong.
|
||||
|
||||
Reference in New Issue
Block a user