v2.0.0: claims model - signed MMUR2 votes in a truth DB, merged view in a rebuildable cache DB; POST /api/claims, claims and merge routes, unsigned writes answer 410

docs/api-mediated-access.md

@@ -1,4 +1,14 @@
-# API-Mediated Access (Redis Lockdown)
+# API-Mediated Access (Redis Lockdown → Signed Claims)
+
+> **meta-core v2.0.0 — the claims model.** The implementation basis is
+> [`docs/implementation/metacore-claims-model.md`](../../../docs/implementation/metacore-claims-model.md)
+> (meta-root). Where this file and that one disagree, that one wins. The
+> *access* contract below still holds unchanged — HTTP only, events over
+> SSE, no database reachable from outside `metacore-app`. The *write*
+> contract changed: every metadata write is a **signed claim** posted to
+> `POST /api/claims`, and the legacy unsigned write endpoints answer
+> `410 Gone`. See ["Write contract — signed claims"](#write-contract--signed-claims-v200)
+> below.
 
 ## Status
 
@@ -69,20 +79,23 @@ we don't fully control.
 ## The architecture in one picture
 
 ```
-                meta-core (only Redis writer + reader)
+                meta-core v2.0.0 (only store writer + reader)
                 ┌──────────────────────────────────────┐
                 │  HTTP API surface:                   │
-                │    POST/PUT/PATCH /api/metadata/*    │  ◄── writes
-                │    GET            /api/meta/{cid}    │  ◄── reads (by CID)
-                │    GET            /api/metadata/*    │  ◄── reads (by id)
-                │    GET            /api/file/{cid}    │  ◄── content
-                │    GET            /api/events/files  │  ◄── SSE stream
-                │    GET            /api/events/meta   │  ◄── SSE stream
+                │    POST /api/claims                  │  ◄── writes (signed claims only)
+                │    POST /api/merge                   │  ◄── stateless merge, no write
+                │    GET  /api/meta/{cid}              │  ◄── reads (by CID, merged)
+                │    GET  /api/meta/{cid}/claims       │  ◄── reads (raw signed claims)
+                │    GET  /api/metadata/*              │  ◄── reads (by rid, merged)
+                │    GET  /api/file/{cid}              │  ◄── content
+                │    GET  /api/events/files            │  ◄── SSE stream
+                │    GET  /api/events/meta             │  ◄── SSE stream (merged changes)
                 │     │                                │
-                │     ▼                                │
-                │  Redis  (file:*, cid:*, streams)     │
-                │  ◄── container-local, no exposed     │
-                │      port, no docker-network alias   │
+                │     ▼  verify sig → truth → apply    │
+                │  truth  /meta-core/db/meta-core.db   │
+                │  cache  /meta-core/db-cache/cache.db    │
+                │  ◄── embedded SQLite, no port,       │
+                │      no docker-network alias         │
                 └──────────────────────────────────────┘
                           ▲             ▲
                           │ HTTP        │ HTTP SSE
@@ -95,9 +108,18 @@ we don't fully control.
 ```
 
 **The single-writer rule.** meta-core has exclusive write authority over
-the `file:*`, `cid:*`, `file:__index__` (now `idx:*`), and `meta-core:*` keyspaces. No
-other service can write — not because of an honoured convention, but
-because they can't reach Redis at all.
+its store — in v2.0.0 the two SQLite databases (truth and cache); in the
+Redis era the `file:*`, `cid:*`, `file:__index__` (later `idx:*`), and
+`meta-core:*` keyspaces. No other service can write the store — not because
+of an honoured convention, but because they can't reach it at all.
+
+**What changed in v2.0.0 is who may *author* metadata.** meta-core is still
+the only process that touches the database, but it no longer accepts
+unsigned values: a writer submits signed votes (claims), meta-core verifies
+each signature and stores the vote. meta-core **never signs a claim and
+holds no writer key** — a store that could sign for everyone would prove
+nothing. The signature is the write authentication; there are no
+per-writer tokens.
 
 **Stream reads also mediated.** Events go out through SSE endpoints on
 meta-core's HTTP surface. Consumers don't see the underlying Redis
@@ -108,6 +130,147 @@ without breaking every consumer.
 
 **Redis is private.** No host port mapping, no docker-network alias, no
 URL in `kv-leader.info`. Discovery returns only meta-core's HTTP base.
+(Since 2026-10 there is no Redis at all; the SQLite files are reachable only
+from inside `metacore-app`.)
+
+## Write contract — signed claims (v2.0.0)
+
+Every metadata value is a **signed vote**: *signer `uid` says `key = value`
+about content `cid`* — `claim(uid, cid, key) → value + ts + sig`. The
+record a reader sees is the **merged view** that meta-core precomputes from
+the votes; a reader's own votes always win in their view. Full model,
+schema and merge rules: the
+[implementation basis](../../../docs/implementation/metacore-claims-model.md)
+§1–§7.
+
+### Endpoint table
+
+| | Endpoint | Behaviour |
+|---|---|---|
+| **Write** | `POST /api/claims` | A batch of signed claims (a JSON array, or `{"claims": [...]}`). **The only metadata write path.** |
+| Removed | `PUT/PATCH/DELETE /meta/{hash}`, `PUT/DELETE /meta/{hash}/{key...}`, `POST /meta/{hash}/_add/{key...}`, `PUT/DELETE /api/metadata/{hashId}`, `PUT /api/metadata/{hashId}/property`, `POST /api/metadata/batch` | `410 Gone`, with a pointer to `POST /api/claims` |
+| Kept | `PUT/GET/DELETE /api/kv/value` | Service scratch KV (`btmagnet:*`, `searchcov:*`), plain rows of the cache database's `kv` table (any key outside the reserved prefixes `file:`, `cid:`, `idx:`, `dup:`, `ev:`, `evhwm:`, `meta:`, `meta-core:`) — losing it only costs a refetch |
+| Kept | `POST /api/files/register`, the file watcher | Fill `local_file` (box-local `filePath`, size, mtime, midhash — unsigned) and `cid_group` |
+| UDL facade | `/api/udl/*` (same paths) | Writes go through the claims write path; reads come from the claims (per-signer index `claim_by_pid`). Records are MMUR2 claims. Shapes: see "UDL facade shapes" below |
+| Read | `GET /api/meta/{cid}`, `GET /api/metadata/{rid}`, `GET /file/{cid}`, `GET /meta/{hash}`, `GET /api/metadata/hash-ids`, `GET /api/metadata/list`, `POST /api/metadata/search`, `GET /api/files/tuples`, `GET /api/schema`, `GET /api/stats` | Unchanged paths and response shapes, served from the merged view (+ `local_file`); optional `?as=<uid>`. They carry no `nsrc` |
+| **New** | `GET /api/meta/{cid}/claims` | `{rid, cids, claims}`: the record's rid, the cids of its group and every live signed claim on them (all signers, tombstones excluded). `?key=` keeps one exact key, `?uid=` one signer. `404 unknown_cid` when no record holds the cid |
+| **New** | `GET /api/metadata/{rid}/claims` | Same body and filters, addressed by rid (a cid is also accepted); `404` for an unknown record. What meta-search ingest and the gateway's corpus search use |
+| **New** | `POST /api/merge` | Stateless: body `{claims, as?}`; verifies each claim (invalid ones dropped and counted, never a batch error), merges with the store's merge rule, overlays `as`'s own votes (in the request and held by this core for the same cids). Returns `{fields, nsrc, rejected}` — merged `key → value`, `key → agreeing signers`, rejected count. `413` above 10 000 claims. Writes nothing |
+| Events | `GET /api/events/meta` | Unchanged `{type: set\|del, key, ts}` shape, now emitted for changes of the **merged** view, `key` still the literal `file:<rid>/<field>`; `GET /api/events/files` unchanged. Both streams are rows of the cache `kv` table (`ev:meta:events/…`, `ev:file:events/…`), ~100k entries each |
+| Admin | `POST /api/metadata/clear` | Wipes both databases (truth and cache) |
+| Removed | `/api/snapshot/export`, `/api/snapshot/import`, `/api/snapshot/wipe` | No snapshot format in v2.0.0 |
+
+Endpoints the claims model does not touch keep their behaviour: discovery
+(`/health`, `/status`, `/leader`, `/urls`, `/neighbors`), the account-key
+identity service (`/api/identity/*` — its removal is a separate ticket;
+v2.0.0 adds the `format` field to `sign` / `sign-batch` — `"compact"` or
+`"der"`, default compact for `"MMUR2\n"` payloads and DER otherwise — and
+makes `DELETE /api/identity` sign and store a retraction of every live
+claim of the account before removing it),
+`/api/file/{cid}/info`, `/file/cid`, `/data/{hash}*`, mounts, watchers and
+WebDAV.
+
+### The claim on the wire (JSON form)
+
+```json
+{ "cid": "bafy…", "uid": "zQ3s…", "key": "title",
+  "value": "The Matrix",            // or {"b64": "…"} for bytes, or null
+  "ts": 1793001234567,
+  "sig": "base64(64 bytes)" }
+```
+
+| Field | Meaning |
+|---|---|
+| `cid` | The subject: a content CID as the writer knows it (bare CID string, as in `cids/*`). Never a rid — rids are local and never signed |
+| `uid` | The signer: `"z" + base58btc(33-byte compressed secp256k1 pubkey)` |
+| `key` | The field (`METADATA_KEYS.md`); a key-set member is inside the key (`genres/Drama`) |
+| `value` | Text = public plaintext; `{"b64": …}` = bytes, a private (ciphertext) envelope; `null` = retraction |
+| `ts` | Milliseconds since epoch. It **is** the version: only increases per `(uid, cid, key)` |
+| `sig` | MMUR2: secp256k1 ECDSA, compact 64 bytes, over SHA-256 of `"MMUR2\n" ‖ canonical CBOR [cid, uid, key, value, ts]` |
+
+A batch (the `POST /api/claims` body) is a JSON array of these (or
+`{"claims": [...]}`). Kamilata and
+the gateway wire carry the same five fields plus `sig`; the signed bytes
+never change on any hop, and relays (meta-search, a client persisting a
+gateway hit) forward claims verbatim — they never re-sign.
+
+### Per-claim statuses
+
+`POST /api/claims` answers with one status per claim:
+
+| Status | Meaning |
+|---|---|
+| `accepted` | Verified and stored (or a retraction stored as a tombstone) |
+| `unchanged` | The stored row for `(uid, cid, key)` already has this `ts` and `sig` — **or the same value under any `ts`** (a signer re-asserting what it holds): no verification, no write, and the stored row keeps its older `ts` |
+| `stale` | The stored `ts` is ≥ the incoming `ts`; ignored (blocks replay of old votes) |
+| `bad_signature` | The MMUR2 signature does not verify against the `uid`'s public key |
+| `bad_request` | The claim is malformed |
+
+The response is `200` with a JSON array of `{status, reason?}`, one entry
+per claim, in input order, even when some are refused. A batch above 10 000
+claims answers `413 too_many_claims`; a body that is not claims JSON
+`400 invalid_json`. The
+removed write routes answer `410 {"error":"gone","use":"POST /api/claims"}`.
+`POST /api/claims`, `POST /api/merge` and the two claims read routes are
+container-to-container routes, not routed through Caddy.
+
+Rules: every claim is signed — ciphertext and retractions included; an
+unsigned or badly signed claim is refused. A retraction (`value: null`) is
+kept as a tombstone, so a peer cannot replay the vote it retracted. Values
+are stored exactly as signed; normalisation (trim + case-fold, no NFC)
+happens only inside the merge.
+
+Accepted claims commit to the truth database in one group-committed
+transaction; an apply loop then folds them, in commit order, into the
+cache (the merged `file:<rid>/<field>` rows of its v1-layout `kv` table,
+FTS, counters, events). `POST /api/claims` answers only once the cache has
+applied the batch, so the writer reads its own write; another reader may
+briefly see the previous merged value — the cache is behind, never wrong.
+
+### `?as=<uid>` — the viewer's overlay
+
+Every merged read accepts an optional `?as=<uid>`. The merge then puts that
+signer's own live text vote first for each key (read-time overlay found
+through the per-signer index, never stored in the shared merged view). Without
+`?as=` the read returns the shared merge: most distinct signers → oldest
+`ts` → byte-wise smallest value; ciphertext never merges.
+
+### Raw claims and client-side merging
+
+The claims read routes (`GET /api/metadata/{rid}/claims` for meta-search
+ingest and the gateway's corpus hits, `GET /api/meta/{cid}/claims` by cid)
+return the raw signed votes of every signer, not a merged map. A client that receives
+claims from the swarm or a gateway displays them by calling its **local**
+meta-core's `POST /api/merge` (one merge implementation, in Go), and
+persists a hit by posting the claims **as received** to `POST /api/claims`.
+
+### UDL facade shapes
+
+Claims in the JSON form above (`internal/api/udl.go`):
+
+| Route | Request | Response |
+|---|---|---|
+| `PUT /api/udl/record` | one claim | `200 {accepted: true, status}` for `accepted` / `unchanged`; `409 stale_version` for `stale`; `400` (status as slug) for `bad_signature` / `bad_request` |
+| `PUT /api/udl/records` | `{records: [claim, …]}` | `200 {results: [{key, accepted, status, error?}, …]}`, input order; `accepted` = `accepted` or `unchanged` |
+| `GET /api/udl/record?uid=&cid=&key=` | — | `{exists, claim}` — the stored claim, tombstone included; `exists` false when absent or a tombstone |
+| `GET /api/udl/user/{uid}/key/{key}`, `/user/{uid}/cid/{cid}` | — | `{entries: [claim, …]}` |
+| `GET /api/udl/user/{uid}/all?after=&limit=` | — | `{entries: [claim, …], nextCid}` |
+| `GET /api/udl/user/{uid}/count` | — | `{cids}` |
+| `GET /api/udl/cid/{cid}/users` | — | `{users: [uid, …]}` |
+| `GET /api/udl/cid/{cid}/aggregate?keys=` | — | `{aggregates: [{key, count, avg, trues, sample_size}]}` |
+| `GET /api/udl/users/stats` | — | `{users: {uid: {records, cids, keys}}}` for **every signer** with live claims — plugin, feeder, gateway and service keys included. `/api/stats` `udlUsers` is the size of this map |
+
+### What a writer needs
+
+Each writing container (every meta-sort plugin, every self-publishing
+feeder, gateway core, meta-sort, meta-share, meta-search) signs with its
+**own** key; users sign in the browser. meta-core holds none of these keys.
+Key configuration per signer: implementation basis §9.
+
+Box-local file facts (`filePath`, size, `mtimeNano`, midhash) are not
+claims: the watcher and `POST /api/files/register` write them into the
+cache's `local_file` table, they are joined into the record view unsigned,
+and they never appear in `/api/meta/{cid}/claims` or on the swarm.
 
 ## Eventing — HTTP Server-Sent Events
 
@@ -263,12 +426,19 @@ seen). The contract above holds unchanged:
 - Retention is ~100k entries per stream, trimmed every 10k appends.
 - IDs keep the Redis `<ms>-<seq>` form, so cursors persisted by meta-sort /
   meta-share / meta-stremio before the switch still compare.
-- `meta:events` entries (`{type: set|del, key, ts}`) are appended by the
-  storage engine **in the same transaction** as the `file:` key change —
-  `del` only when the key existed, `set` only when the value actually
-  changed (since v1.2.0; an unchanged write is a no-op). The Redis
-  keyspace-notification hop (PSUBSCRIBE in `MetaPublisher`), which could
-  drop events when it lagged, is gone.
+- `meta:events` entries (`{type: set|del, key, ts}`) describe changes of
+  the **merged** view (v2.0.0): the apply loop emits `set`/`del` (keyed by
+  rid + key) only when a claim actually changed a merged value, in the same
+  cache transaction as that change. A group merge (two cid groups linked by
+  a `cids/*` vote) emits `del` for every key of the losing rid and `set` for
+  the survivor. *History:* v1.x appended them in the same transaction as the
+  `file:` key change; before 2026-10 a Redis keyspace-notification hop
+  (PSUBSCRIBE in `MetaPublisher`) could drop events when it lagged.
+- **Cache rebuild (v2.0.0).** Events live in the cache database. A cache
+  rebuild (missing cache folder, or catch-up after an unclean shutdown)
+  emits no per-key events; SSE consumers get `event: gap` and take a
+  snapshot, as they do for a trimmed cursor. Event IDs stay time-based, so
+  they keep rising across a rebuild.
 - **`Last-Event-ID: 0` / `0-0` on `/api/events/meta` returns a snapshot**
   (since v1.2.0): one `set` per existing record key, then the live tail from
   the head captured before the snapshot. Only the last snapshot event carries
@@ -522,6 +692,10 @@ durability across a meta-core restart).
 
 ### PR B — meta-sort writes via HTTP
 
+> *History.* PR B moved writes onto the unsigned `/meta/{hash}` family.
+> In v2.0.0 those routes answer `410 Gone`; writers sign claims and post
+> them to `POST /api/claims` (see "Write contract — signed claims").
+
 Swap `RedisClient` for an `HttpKVClient` implementation of the existing
 `IKVClient` interface in `packages/meta-sort-core`. All writes
 (`setMetadataFlat`, `setMetadataProperty`, batch updates) become HTTP
@@ -603,7 +777,7 @@ metacore-app produces a connection refused. The lock is real.
 | Backing store for streams? | Redis Streams at the time; SQLite kv rows since 2026-10 | SSE wraps it; the 2026-10 swap proved it — no consumer changed |
 | Consumer groups / horizontal scale? | Not supported via SSE | Not needed today; revisit if scale ever requires it |
 | Storage portability? | Not a goal but a free side-effect | Will not be exercised in alpha |
-| **Canonical write API path?** | **`/meta/{hash}` family** | Has GET/PUT/**PATCH**/DELETE plus `/_add/{key}` and per-property `{key:.*}` operations. `PATCH` calls `MergeMetadataFlat` — matches the partial-update semantics PR B needs. `/api/metadata/{hashId}` stays as the editor-internal surface (no PATCH, no per-property routes) but is not the migration target. |
+| **Canonical write API path?** | **`POST /api/claims` (v2.0.0)** — signed claims only; every unsigned write route answers `410 Gone` | *History (v1.x):* the **`/meta/{hash}` family** was the canonical write surface. It had GET/PUT/**PATCH**/DELETE plus `/_add/{key}` and per-property `{key:.*}` operations. `PATCH` calls `MergeMetadataFlat` — matches the partial-update semantics PR B needs. `/api/metadata/{hashId}` stays as the editor-internal surface (no PATCH, no per-property routes) but is not the migration target. |
 | **SSE event-id contract?** | **Opaque to consumers** | It is `<ms>-<seq>` (the Redis Stream entry form, kept after the 2026-10 SQLite switch); consumers MUST treat as opaque so the backing store can be swapped without breaking the wire contract. Documented in the SSE endpoint comment and in this doc. |
 | **Typed error envelope?** | **`{error, message, retryable}` with stable `error` slug** | See "Error envelope" subsection below. |
 | **Plugin direct-Redis risk?** | **Confirmed: no plugin uses Redis directly** | Grep across `packages/plugins/` for `ioredis` / `redis.createClient` / `new Redis()` returned empty. Plugins POST results back to meta-sort via `/api/plugins/callback`; meta-sort then writes. PR B picks up the plugin write path transparently. |
@@ -611,6 +785,11 @@ metacore-app produces a connection refused. The lock is real.
 
 ## Error envelope
 
+(`410 Gone` from the removed v1.x write routes carries this envelope with a
+message pointing to `POST /api/claims`. Per-claim outcomes of
+`POST /api/claims` are statuses inside the response, not envelopes — see
+"Per-claim statuses" above.)
+
 Standard shape for all 4xx / 5xx responses from the metadata API:
 
 ```json
@@ -629,6 +808,10 @@ Stable `error` slugs the client should switch on:
 | `unknown_root` | 404 | no | The hashId / UUID doesn't exist. Caller probably has a stale reference. |
 | `unknown_cid` | 404 | no | The CID isn't in the reverse index. No retry helps. |
 | `schema_violation` | 400 | no | Body shape rejected (missing required field, invalid value). Caller bug. |
+| `invalid_json` | 400 | no | `POST /api/claims` / `POST /api/merge` body is not claims JSON. Caller bug. |
+| `too_many_claims` | 413 | no | More than 10 000 claims in one `POST /api/claims` or `POST /api/merge`. Split the batch. |
+| `stale_version` | 409 | yes | `PUT /api/udl/record`: an equal or newer `ts` is stored for this signer, cid and key. Re-read the record, sign with a newer `ts`, retry. |
+| `gone` | 410 | no | A removed unsigned write route. Body is `{"error":"gone","use":"POST /api/claims"}` (no `message` / `retryable`). |
 | `storage_unavailable` | 503 | yes | Store transient (Redis at the time; the embedded SQLite store since 2026-10). Caller should retry with backoff. |
 | `internal` | 500 | yes (cautiously) | Unhandled server-side error. Retry once; escalate if persistent. |
 
@@ -643,10 +826,9 @@ import cycle on the api package; consolidating it into a shared
 
 Mostly answered post-implementation:
 
-- **Deprecation timeline for `/api/metadata/{hashId}`.** Still open. The
-  editor's React UI continues to call it; the surface itself was untouched
-  by this work. Worth a follow-up to consolidate the editor onto
-  `/meta/{hash}` and delete the parallel routes.
+- **Deprecation timeline for `/api/metadata/{hashId}`.** Answered by
+  v2.0.0: its `PUT`/`DELETE` (and `PUT …/property`) answer `410 Gone`
+  like every unsigned write; `GET` stays as a merged read.
 
 - **Initial bootstrap flood on `/api/events/files`.** Resolved in
   practice: meta-fuse uses `initialCursor: '0-0'` to deliberately
@@ -666,8 +848,9 @@ Mostly answered post-implementation:
 - **Authentication between internal services.** The hash-lock OIDC
   perimeter handles external clients; internal service-to-service
   trust is by network isolation (docker network for dev, equivalent
-  isolation in prod deployments). Adding service-to-service auth is a
-  separate concern.
+  isolation in prod deployments). Since v2.0.0 metadata *writes* carry
+  their own authentication — the claim signature — so meta-core needs no
+  per-writer tokens; transport-level service auth remains out of scope.
 - **Replacing the public HTTP API surface.** This proposal exercises
   what already exists; the only new endpoints are the two SSE streams.
 - **Introducing new transport, broker, proxy, or sidecar.** Everything
@@ -716,6 +899,11 @@ What actually shipped vs. what this doc planned.
 
 ### Write-path CID resolution (`storage.ResolveRoot`)
 
+> *History (v1.x).* In v2.0.0 a claim names a cid, never a rid, and the
+> hidden rid comes from `cid_group` (a `cids/<other>` vote links two
+> groups; the older rid survives). The dual-root failure below cannot
+> recur: there is no write that addresses a root.
+
 Not in the original plan but load-bearing.
 
 **The problem.** meta-sort's PATCH-via-HTTP migration (PR B) routed
@@ -766,9 +954,10 @@ duplicated in each to avoid an import cycle on the `api` package.
 
 - **meta-dup SSE + read migration.** Not deployed, no test harness.
   When revived, copy meta-fuse's pattern.
-- **Editor `/api/metadata/{hashId}` consolidation.** The duplicate
-  editor-internal API surface is untouched; could be retired in a
-  follow-up by porting the editor UI to `/meta/{hash}`.
+- **Editor writes (resolved in v2.0.0).** `PUT /api/metadata/{hashId}`
+  answers `410`. meta-sort-editor's manual edits go through meta-sort's
+  backend, which signs them as meta-sort (its own key) and posts them to
+  `POST /api/claims`. The editor's reads keep using `/api/metadata/{hashId}`.
 - **Consolidated `internal/httperr` package** for the typed envelope
   helpers (currently duplicated 4× to avoid the import cycle). Code
   hygiene; functionally a no-op.