v2.0.0: claims model - signed MMUR2 votes in a truth DB, merged view in a rebuildable cache DB; POST /api/claims, claims and merge routes, unsigned writes answer 410
Build and Push Docker Image / build (push) Successful in 5m1s
Build and Push Docker Image / build (push) Successful in 5m1s
This commit is contained in:
Worph
+254
-323
@@ -1,256 +1,199 @@
|
||||
# Metadata Storage Structure
|
||||
|
||||
How meta-core lays out file metadata in its store. This document covers the
|
||||
on-disk shape (UUID-rooted flat string keys, CID reverse index,
|
||||
per-root sets) and the schema-version gate. It does **not** enumerate
|
||||
field semantics — that's the job of the repo-root
|
||||
[`METADATA_KEYS.md`](../../../METADATA_KEYS.md), the authoritative key
|
||||
registry for everyone who writes to or reads from metadata.
|
||||
How meta-core lays out metadata on disk since **v2.0.0 (claims model)**: the
|
||||
two SQLite databases, their tables, how a record is assembled from signed
|
||||
votes, and the schema-version gate. Implementation basis:
|
||||
[`docs/implementation/metacore-claims-model.md`](../../../docs/implementation/metacore-claims-model.md).
|
||||
This document does **not** enumerate field semantics — that is the job of the
|
||||
repo-root [`METADATA_KEYS.md`](../../../METADATA_KEYS.md), the authoritative
|
||||
key registry for everyone who writes or reads metadata.
|
||||
|
||||
For the design rationale (why UUIDs instead of content hashes, why a
|
||||
reverse index), see [`uuid-rooted-metadata.md`](uuid-rooted-metadata.md).
|
||||
For the external-access contract (HTTP-only, SSE for events), see
|
||||
Related: why rids are UUIDs and hidden —
|
||||
[`uuid-rooted-metadata.md`](uuid-rooted-metadata.md); why the engine is
|
||||
shaped this way, its guarantees and measurements —
|
||||
[`metacore-storage-engine.md`](../../../docs/project-architecture/metacore-storage-engine.md);
|
||||
the external-access contract (HTTP-only, SSE for events) —
|
||||
[`api-mediated-access.md`](api-mediated-access.md).
|
||||
|
||||
> **Storage engine (since 2026-10): an embedded SQLite file, not Redis.**
|
||||
> Until 2026-10 these keys lived in a Redis server spawned inside the
|
||||
> container. They now live in one string→string table of an embedded SQLite
|
||||
> database (pure-Go driver `modernc.org/sqlite`, CGO off). The **key names
|
||||
> did not change** — only the Redis *types* that some bookkeeping used (SETs,
|
||||
> HASHes, Streams) were flattened into plain keys, listed below. Holding
|
||||
> everything in RAM had cost ~6.6 GB on watch.nsl.sh's 169k records (Redis
|
||||
> plus the API's in-memory search copy) and caused repeated OOM kills.
|
||||
> **History.** Until 2026-10 metadata lived in Redis; v1.1.0–v1.2.x kept the
|
||||
> same flat keys (`file:<rid>/<field>`, `cid:<cid>`, `idx:*`, `dup:*`,
|
||||
> `udl:*`, `ev:*`) in one SQLite string→string `kv` table. v2.0.0 replaced
|
||||
> that keyspace entirely; the old layout is summarised at the end
|
||||
> ([Previous layout](#previous-layout-v11v12-kv-table)).
|
||||
|
||||
## Quick summary
|
||||
|
||||
- **One table**: `kv(k TEXT PRIMARY KEY, v TEXT NOT NULL) WITHOUT ROWID`,
|
||||
schema-free — a new metadata field is just a new key, never a migration.
|
||||
File: `/meta-core/db/meta-core.db` (override `META_CORE_DB_PATH`), WAL
|
||||
mode, `synchronous=NORMAL`.
|
||||
- **Roots are UUIDv7** encoded as 26-char Crockford Base32 (ULID layout),
|
||||
not content hashes. Examples in this doc use the placeholder
|
||||
`01JKR8XW5T4QMV7AHNJ2DEFGHK`.
|
||||
- **Properties are flat string keys**, one per leaf:
|
||||
`file:<uuid>/<property/path>` → string value. A record is exactly the
|
||||
key range `[file:<uuid>/, file:<uuid>0)` — there is no per-record field
|
||||
list.
|
||||
- **CIDs are reverse-index aliases**: `cid:<bareCid>` → `<uuid>`
|
||||
(one key per known CID; no algorithm prefix — the multicodec
|
||||
already carries it).
|
||||
- **Per-root sets**: `file:<uuid>/cids/<bareCid>` (a *key-set*: one key
|
||||
`"true"` per known CID, midhash included), `dup:<uuid>/<path>` (alternate
|
||||
paths with the same content), plus `idx:roots/<uuid>` (every UUID).
|
||||
There is **no stored `canonical_cid`** — it is derived by rank on read.
|
||||
- **A schema-version sentinel** (`meta-core:schema-version`) refuses to
|
||||
boot against legacy data.
|
||||
- **Every metadata value is a signed claim**: `claim(uid, cid, key) → value`
|
||||
+ `ts` + `sig` (MMUR2, secp256k1). One row per signer per field. Writers
|
||||
sign; meta-core never signs.
|
||||
- **Two databases.** *Truth* `/meta-core/db/meta-core.db`
|
||||
(`META_CORE_DB_PATH`, backed up, primary keys only): `principal`, `claim`,
|
||||
`cid_group`, `meta`. *Cache* `/meta-core/db-cache/cache.db`
|
||||
(`META_CORE_CACHE_PATH`, never backed up, rebuildable): the v1 `kv` table
|
||||
(merged view as `file:<rid>/<field>` rows, lookup, counters, event streams,
|
||||
bookkeeping, service scratch keys) with its `kv_field_v` index and `fts`,
|
||||
plus two real tables, `claim_by_pid` and `local_file`.
|
||||
- **Schema-free.** A field is a `key` string in a row; a new field never
|
||||
needs DDL.
|
||||
- **Records are cid groups.** A vote names a cid, never a rid. `cid_group`
|
||||
maps each cid to a **rid** — a hidden, locally minted UUIDv7 (26-char
|
||||
Crockford) naming the group. Examples use `01JKR8XW5T4QMV7AHNJ2DEFGHK`.
|
||||
- **What readers get is the merged view**: one precomputed value per
|
||||
`(rid, key)`, stored as the cache row `file:<rid>/<key>`. `nsrc` (how many
|
||||
signers support it) is computed by the merge and returned by
|
||||
`POST /api/merge`, not stored. `canonical_cid` is still **derived on read**
|
||||
by rank, never stored.
|
||||
- **Watcher facts are not claims**: `filePath`, size, `mtimeNano`, midhash
|
||||
live in `local_file` and are joined into the record view unsigned.
|
||||
- **A schema-version sentinel** (`schema-version = "3"` in truth `meta`).
|
||||
meta-core explicitly refuses a truth file holding the v1 `kv` table, and any
|
||||
store with claims but no sentinel or another version.
|
||||
|
||||
Implementation lives in `internal/kv/kv.go` (engine),
|
||||
`internal/kv/stream.go` (event streams), `internal/storage/client.go`,
|
||||
`internal/storage/search.go`, `internal/storage/cid_resolution.go`,
|
||||
`internal/storage/uuid.go`, and `internal/storage/schema_version.go`.
|
||||
## Truth database — `/meta-core/db/meta-core.db`
|
||||
|
||||
## Key structure
|
||||
Backed up; only primary keys (plus `principal.uid`'s implicit UNIQUE index).
|
||||
|
||||
Everything is one string→string table. The full layout:
|
||||
|
||||
| Key | Value | Notes |
|
||||
| Table | Columns (PK first) | Notes |
|
||||
|---|---|---|
|
||||
| `file:<rid>/<field>` | field value | a record property — the **only** thing under `file:` |
|
||||
| `cid:<cid>` | `<rid>` | reverse index |
|
||||
| `meta-core:schema-version` | `"2"` | schema gate |
|
||||
| `idx:roots/<rid>` | `""` | set of every root (was the Redis SET `file:__index__`) |
|
||||
| `idx:fts/<rid>` | FTS rowid | the record's row in the FTS table |
|
||||
| `idx:count/roots` | integer | number of roots — search's `total`, `/api/stats` records (v1.2.0) |
|
||||
| `idx:schema/<field>\x1f<keyHint>\x1f<prim>\x1f<hint>` | integer | per-field value-type counts behind `/api/schema` (v1.2.0) |
|
||||
| `idx:count/ready` | `"1"` | marks the counters as built (absent → one-time build at boot) |
|
||||
| `dup:<rid>/<path>` | `""` | a duplicate path (was the SET `file:<rid>/duplicates`) |
|
||||
| `udl:rec:<uid>/<cid>/<key>` | JSON `{record,version,ts,value?}` | User Data Layer entry (was a HASH) |
|
||||
| `udl:idx:…<set>\x1f<member>` | `""` | UDL index sets, one key per member |
|
||||
| `ev:<stream>/<ms13>-<seq10>` | JSON of the entry's fields | `meta:events` / `file:events` entries (were Redis Streams) |
|
||||
| `evhwm:<stream>` | last issued ID | stream high-water mark |
|
||||
| `btmagnet:*`, `searchcov:*`, `nzb:*` | — | written by other services via `/api/kv/value`; unchanged |
|
||||
| `principal` | `pid` INTEGER PK, `uid` TEXT UNIQUE | `uid` = `"z" + base58btc(33-byte compressed pubkey)`. `pid` is a local surrogate: never signed, never on the wire |
|
||||
| `claim` | `(cid, key, pid)` PK, `value` ANY, `ts`, `sig`, `seq` | `STRICT, WITHOUT ROWID`. `value`: TEXT = plaintext, BLOB = ciphertext envelope, NULL = retraction (tombstone, kept). `ts` (ms) is the version. `seq` = global change counter, not indexed |
|
||||
| `cid_group` | `cid` PK, `rid` | the cid → rid map: the record identity layer |
|
||||
| `meta` | `name` PK, `value` | `schema-version = "3"`, `last_seq` |
|
||||
|
||||
Keeping bookkeeping **out of `file:`** is load-bearing: the `file:` range is
|
||||
exactly the records, so the field index covers only real fields and every
|
||||
change under `file:` is a real field change worth publishing on
|
||||
`meta:events`. The per-record `file:<rid>/__fields__` SET and the
|
||||
`file:__dirty__` set of the Redis era no longer exist. Roots are split from
|
||||
the field at the **first** `/`, so CID-rooted and UUID-rooted records both
|
||||
parse.
|
||||
`claim` is written **only** by `POST /api/claims`. A claim replaces the
|
||||
stored row for its `(cid, key, pid)` only if its `ts` is strictly greater;
|
||||
identical `(ts, sig)` is `unchanged`; the **same value** from the same signer
|
||||
(any `ts`) is also `unchanged` and keeps the stored row with its older `ts`;
|
||||
otherwise older or equal is `stale`, and a bad signature is refused. Why `STRICT` with an `ANY` column: a non-STRICT `ANY`
|
||||
column gets NUMERIC affinity and turns the text `"007"` into the integer 7;
|
||||
STRICT keeps the written type, which is the cipher flag.
|
||||
|
||||
### Roots
|
||||
## Cache database — `/meta-core/db-cache/cache.db`
|
||||
|
||||
Never backed up (the store app lists the folder in `backup.exclude`).
|
||||
Deleting it is the recovery procedure: meta-core rebuilds it from truth at
|
||||
boot (no `meta:applied_seq` → full rebuild), and `local_file` refills from a
|
||||
rescan.
|
||||
|
||||
It keeps the **v1 key/value layout** (`internal/storage/client.go`): one
|
||||
`kv(k TEXT PRIMARY KEY, v TEXT NOT NULL) WITHOUT ROWID` table, whose rows are
|
||||
all derived — no client writes `file:` rows any more.
|
||||
|
||||
| `kv` rows | Holds | Written by | Read by |
|
||||
|---|---|---|---|
|
||||
| `file:<rid>/<field>` | the **merged view**, one row per `(rid, key)` — the only rows under `file:`. Includes the synthesised `cids/<cid> = "true"` members (from `cid_group`) and the box-local `filePath` / `sizeByte` / `mtimeNano` (from `local_file`) | apply loop | every record read; search |
|
||||
| `cid:<cid>` | cid → rid (copy of truth `cid_group`) | apply loop, watcher | every cid-addressed request |
|
||||
| `idx:roots/<rid>` | the set of every record | apply loop | listing, counts |
|
||||
| `idx:fts/<rid>` | rowid of the record's `fts` row | apply loop | FTS maintenance |
|
||||
| `idx:count/*`, `idx:schema/*` | record count, per-field schema counters | the cache engine's change hook, from `file:` deltas | search `total`, `/api/stats`, `/api/schema` |
|
||||
| `ev:<stream>/<ms13>-<seq10>`, `evhwm:<stream>` | `meta:events` (every `file:` change, same transaction) and `file:events` (watcher) | cache engine; watcher | `/api/events/meta`, `/api/events/files` (SSE) |
|
||||
| `meta:applied_seq`, `meta:rebuilt_at` | how far the cache has applied truth; last rebuild (older SSE cursors get `gap`) | apply loop, rebuild | boot catch-up; SSE |
|
||||
| anything else | service scratch KV (`btmagnet:*`, `searchcov:*`, `nzb:*`) | other services via `/api/kv/value` (reserved prefixes refused) | same |
|
||||
|
||||
| Index / table | Shape | Written by | Read by |
|
||||
|---|---|---|---|
|
||||
| `kv_field_v` (index) | `kv(lower(field parsed from k), lower(v), v)` over `file:` rows only | — | filter search |
|
||||
| `fts` | FTS5 trigram `(rid UNINDEXED, hay)`, one row per record | apply loop, when a haystack field's merged value changes | free-text search |
|
||||
| `claim_by_pid` | `(pid, cid, key)` PK | apply loop | `?as=<uid>` overlay, `/api/udl/*` facade, identity retraction, `/api/udl/users/stats` |
|
||||
| `local_file` | `path` PK → `midhash`, `size`, `mtime`; index `(midhash, path)` | file watcher, `POST /api/files/register` | record view (`filePath`, `sizeByte`, `mtimeNano`) |
|
||||
|
||||
The cache is **behind, never wrong**: after an unclean shutdown, boot finds
|
||||
`meta:applied_seq < last_seq` and re-applies every truth claim with `seq >
|
||||
applied_seq` (or rebuilds in full when one of them is a `cids/*` link). Write
|
||||
path, apply loop and durability are detailed in
|
||||
[`metacore-storage-engine.md`](../../../docs/project-architecture/metacore-storage-engine.md) §4.
|
||||
|
||||
## Keys
|
||||
|
||||
The `key` column carries the same field paths METADATA_KEYS.md defines —
|
||||
what used to follow `file:<rid>/`:
|
||||
|
||||
```
|
||||
file:<uuid>/<property/path> → string (always)
|
||||
claim(cid=bagacbabaec…, key=title, uid=<tmdb plugin>) = "Inception"
|
||||
claim(cid=bagacbabaec…, key=fileinfo/duration, uid=<ffmpeg plugin>) = "8878.5"
|
||||
claim(cid=bagacbabaec…, key=fileinfo/streamdetails/video/0/codec, uid=<ffmpeg plugin>) = "h264"
|
||||
claim(cid=bagacbabaec…, key=genres/Drama, uid=<tmdb plugin>) = "true"
|
||||
claim(cid=bagacbabaec…, key=cids/bafybeigd…, uid=<full-hash plugin>) = "true"
|
||||
```
|
||||
|
||||
UUIDs are produced by `storage.NewUUID()` (`internal/storage/uuid.go`):
|
||||
UUIDv7 → 26-char Crockford Base32 (skips I/L/O/U; ULID-compatible). The
|
||||
time-sortable prefix means roots minted in a given window sort together
|
||||
(`k >= 'file:01JKR8' AND k < 'file:01JKR9'`), which restored some of the
|
||||
debug-ability lost when midhash256 stopped being the root.
|
||||
- `/` is the path separator; nested objects are flattened on write and
|
||||
reconstructed on read.
|
||||
- **Key-sets** put the member inside the key (`genres/<g>`, `cids/<cid>`,
|
||||
`titles/eng/<name>`, `anchoredBy/tmdb:123`): one claim per member, union by
|
||||
the ordinary merge, removal by a signed retraction.
|
||||
- Plugins keep writing under their prefixes (`{plugin-id}/<field>`,
|
||||
`{plugin-id}/<group>/<n>/<field>`), each signing with its own key. Two
|
||||
sources compete on one key only when they make the same kind of statement;
|
||||
a different fact (e.g. a filename guess vs a TMDB lookup) gets a different
|
||||
key.
|
||||
|
||||
Properties use `/` as the path separator. Nested objects in source code
|
||||
are flattened to flat keys on write and reconstructed on read. Examples:
|
||||
## Records: `cid_group` and the hidden rid
|
||||
|
||||
```
|
||||
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/filePath = "/files/watch/movies/Inception.2010.mkv"
|
||||
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/sizeByte = "15032385536"
|
||||
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/mtimeNano = "1729872000000000000"
|
||||
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/title = "Inception"
|
||||
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/fileinfo/duration = "8878.5"
|
||||
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/fileinfo/streamdetails/video/0/codec = "h264"
|
||||
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/tmdb/poster = "bafkreih..."
|
||||
votes (truth, signed) cid_group (truth) cache kv rows (merged)
|
||||
(A, M, title) = "The Matrix" M → R1 file:R1/title = "The Matrix"
|
||||
(A, M, cids/Q) = "true" Q → R1 file:R1/cids/M = "true" (synthesised)
|
||||
(B, Q, title) = "Matrix, The" file:R1/cids/Q = "true" (synthesised)
|
||||
file:R1/sizeByte = "734003200"
|
||||
(B, Q, sizeByte) = "734003200"
|
||||
```
|
||||
|
||||
### Reverse-index aliases (CIDs)
|
||||
- A text `cids/<other>` vote on cid X links X and `<other>` into one
|
||||
group. Retracting it does **not** split the group (phase 1). In the merged
|
||||
view, membership is synthesised from `cid_group` — `cids/<cid> = "true"` for
|
||||
every cid of the group — and `cids/*` claims are never merged. Record R = every vote on every cid of R's group, merged per key,
|
||||
each signer counted once per key.
|
||||
- First-seen cid with no link → new group with a fresh rid
|
||||
(`storage.NewUUID()`, UUIDv7 → 26-char Crockford Base32, ULID layout).
|
||||
- Linking two groups → the **older rid survives**; the other group's cids
|
||||
move to it; `/api/events/meta` emits `del` for each key of the losing rid
|
||||
and `set` for the survivor. There is no rid alias table.
|
||||
- `GET /api/meta/<cid>` is a `cid:<cid>` lookup in the cache, then a range
|
||||
read `[file:<rid>/, file:<rid>0)` — any known cid, any hash algorithm, no
|
||||
primary.
|
||||
- Rids are local names: never signed, never on the swarm or gateway wire.
|
||||
Local consumers (meta-sort, meta-fuse, meta-stremio) do hold them, which is
|
||||
why `cid_group` lives in truth.
|
||||
|
||||
Every CID known for a file is registered as its own key, keyed by
|
||||
the **bare CID** — no algorithm prefix. A CIDv1 already encodes its hash
|
||||
function in the multicodec, so naming it again in the key would be
|
||||
redundant (and would let the same digest be indexed under two spellings):
|
||||
There is **no stored `canonical_cid`**. `GetMetadataDocument` ranks the
|
||||
group's cids through `internal/cid/rank.go` (`cid.Better`) and returns the
|
||||
winner as `canonical_cid` in the JSON response only.
|
||||
|
||||
```
|
||||
cid:<bareCid> → <uuid>
|
||||
```
|
||||
## Merge
|
||||
|
||||
Examples:
|
||||
|
||||
```
|
||||
cid:bafkreih5kapbjzq... → 01JKR8XW5T4QMV7AHNJ2DEFGHK
|
||||
cid:bagacbabaec7v3... → 01JKR8XW5T4QMV7AHNJ2DEFGHK
|
||||
```
|
||||
|
||||
The reverse index makes `GET /api/meta/<cid>` a primary-key lookup for any
|
||||
registered CID, independent of which hash algorithm — there is no primary.
|
||||
The record write path (`writeFieldsTx` in `internal/storage/client.go`, used
|
||||
by `SetProperty`, `SetMetadataFlat` and `MergeMetadataFlat`) registers the
|
||||
alias for any field whose name starts with `cids/`, in the same transaction,
|
||||
so writers emit key-set members without knowing the reverse index exists.
|
||||
|
||||
> ⚠ The hook matches **only** the `cids/` prefix. A legacy `cid_*` or
|
||||
> `midhash256` *named* field written to a root is stored verbatim and is
|
||||
> **never indexed** — the record becomes unresolvable by that CID. Writers
|
||||
> must collapse such fields to `cids/<cid>` members before the write. See
|
||||
> `METADATA_KEYS.md` §14.13.
|
||||
|
||||
### Per-root sets
|
||||
|
||||
```
|
||||
file:<uuid>/cids/<bareCid> → "true" — one key per known CID (a key-set)
|
||||
dup:<uuid>/<path> → "" — one key per alternate path with same content
|
||||
```
|
||||
|
||||
`cids` is a **key-set**: one flat `file:<uuid>/cids/<cid>` key per member,
|
||||
which keeps it conflict-free to merge across peers (a single set-valued
|
||||
field would not be).
|
||||
The record's own address (the midhash) is just the `0x1000` member — it has
|
||||
no separate field.
|
||||
|
||||
There is **no stored `canonical_cid`**. The canonical CID is *derived on
|
||||
read*: `GetMetadataDocument` (`internal/storage/cid_resolution.go`) ranks the
|
||||
key-set's members through `internal/cid/rank.go` (`cid.Better`) and returns
|
||||
the winner as the `canonical_cid` field of the JSON response only. Nothing
|
||||
reconciles or persists it — an earlier `reconcileCanonicalCIDLocked` did, and
|
||||
was removed with the flat `cid_*` keys.
|
||||
|
||||
### Global index
|
||||
|
||||
```
|
||||
idx:roots/<uuid> → "" (one key per root; was the Redis SET file:__index__)
|
||||
```
|
||||
|
||||
`internal/storage/client.go::GetAllHashIDs` range-reads `idx:roots/` to
|
||||
enumerate every file without scanning the records.
|
||||
|
||||
### Search structures (derived)
|
||||
|
||||
Two derived structures are maintained on top of `kv`, in the same
|
||||
transaction as every record write. Both are rebuildable from the kv rows and
|
||||
never authoritative (`internal/storage/search.go`):
|
||||
|
||||
- **`kv_field_v`** — a partial expression index
|
||||
`ON kv(lower(substr(k,instr(k,'/')+1)), lower(v), v) WHERE k>='file:' AND k<'file;' AND instr(k,'/')>0`.
|
||||
It answers "which records have field F with a value like V" for **any**
|
||||
field, including ones invented tomorrow — the index is over the key itself.
|
||||
It carries `v` so the substring test is answered from the index alone
|
||||
(covering; 13 ms vs 1.48 s without it, for tmdbid on watch).
|
||||
- **`fts`** — an FTS5 trigram table `fts(rid UNINDEXED, hay)` holding one row
|
||||
per record: its search haystack plus its rid (`idx:fts/<rid>` points at the
|
||||
row). Trigrams make every 3+ character substring indexable.
|
||||
|
||||
`POST /api/metadata/search` uses SQL to build a superset of candidates, then
|
||||
verifies each with the unchanged Go matchers (`recordMatchesFilter`,
|
||||
`matchAllTokens`), so semantics are identical to the old in-memory index.
|
||||
The smallest part (by a capped count) is collected once and sorted; every
|
||||
other filter is probed per page as an EXISTS over the candidate's own key
|
||||
range. The plan, its measurements and the rejected shapes are in
|
||||
[`docs/project-architecture/metacore-storage-engine.md`](../../../docs/project-architecture/metacore-storage-engine.md).
|
||||
Results still come in hashID order and are fresh immediately (no refresh
|
||||
lag). This replaced the API's in-memory `searchIndex` (60 s refresh, ~30 KB
|
||||
RAM per record).
|
||||
|
||||
### Schema-version sentinel
|
||||
|
||||
```
|
||||
meta-core:schema-version → string (currently "2")
|
||||
```
|
||||
|
||||
Set at first boot, checked on every subsequent boot
|
||||
(`internal/storage/schema_version.go`). If the sentinel is missing but
|
||||
the root index is non-empty, the gate refuses to boot (legacy data).
|
||||
If the sentinel is present but doesn't match `SchemaVersion`, the gate
|
||||
also refuses to boot. Alpha-stage policy: operators wipe and restart;
|
||||
there is no automated migration.
|
||||
|
||||
## Why flat string keys?
|
||||
|
||||
The store keeps one flat key per leaf rather than a document or a
|
||||
per-record row. The reasons are:
|
||||
|
||||
1. **Schema-free.** A plugin inventing a field writes a new key; nothing
|
||||
declares it, and the table never needs a migration. The field index
|
||||
covers it automatically.
|
||||
2. **Conflict-free merging across peers.** Key-sets (`cids/<cid>`, and the
|
||||
flat fields in general) merge by union of keys — no read-modify-write of
|
||||
a shared value, nothing to reconcile.
|
||||
3. **Selective updates without read-modify-write.** A plugin writing
|
||||
`tmdb/poster` doesn't have to read the rest of the document.
|
||||
4. **Compatibility with the CID alias hooks.** Field writes can trigger
|
||||
alias-maintenance work in the write path without the plugin caring.
|
||||
|
||||
(Under Redis there was a fourth reason — field-level keyspace notifications
|
||||
fed `meta:events`. That is obsolete: events are now written by the storage
|
||||
engine itself, see [Event streams](#event-streams).)
|
||||
The cache row `file:<rid>/<key>` holds the winner over the group's votes
|
||||
(each signer's newest vote counts once; retractions and ciphertext drop out):
|
||||
most distinct signers (values compared after **trim + case-fold only**, no
|
||||
NFC) → oldest `ts` → byte-wise smallest. The stored value is the winner's
|
||||
exact text; `nsrc` is not stored (`POST /api/merge` returns it). With
|
||||
`?as=<uid>`, the viewer's own live text votes (found via `claim_by_pid`)
|
||||
override at read time; they are never written to the cache. Box-local
|
||||
fields: when several local paths hold the same content, `filePath` is the
|
||||
lexicographically first (size and mtime from the same `local_file` row); a
|
||||
signed `sizeByte` claim wins over the on-disk size. Full rules:
|
||||
implementation basis §7 (`internal/storage/merge.go`).
|
||||
|
||||
## Reads and writes
|
||||
|
||||
All writes go through **one writer goroutine** with group commit: concurrent
|
||||
writes share one SQLite transaction, each under its own SAVEPOINT so a
|
||||
failure rolls back alone. A record write — fields + root index + CID aliases
|
||||
+ FTS row + events — is atomic. (Measured ~835 records/s with 16 concurrent
|
||||
writers of 40-field records.) Reads use a separate connection pool and see
|
||||
the last committed state.
|
||||
| Operation | Endpoint | Store access |
|
||||
|---|---|---|
|
||||
| write metadata | `POST /api/claims` (batch of signed claims) | truth upsert, then the apply loop updates the cache |
|
||||
| removed writes | `PUT/PATCH/DELETE /meta/*`, `POST /meta/{h}/_add/{k}`, `PUT/DELETE /api/metadata/{id}`, `PUT /api/metadata/{id}/property`, `POST /api/metadata/batch` | `410 Gone`, pointing at `POST /api/claims` |
|
||||
| read a record | `GET /api/meta/{cid}`, `GET /api/metadata/{rid}`, `GET /file/{cid}`, `GET /meta/{h}` | `cid:` lookup → `file:<rid>/*` rows, optional `?as=` |
|
||||
| list / search | `GET /api/metadata/hash-ids`, `/list`, `POST /api/metadata/search`, `GET /api/files/tuples`, `/api/schema`, `/api/stats` | `file:` rows, `kv_field_v`, `fts`, `idx:count/*` / `idx:schema/*` |
|
||||
| raw votes | `GET /api/meta/{cid}/claims`, `GET /api/metadata/{rid}/claims` (`?key=`, `?uid=`) | `{rid, cids, claims}`: every live signed claim of the group, all signers, tombstones excluded; 404 for an unknown cid / rid |
|
||||
| merge without storing | `POST /api/merge` `{claims, as?}` → `{fields, nsrc, rejected}` | none — verify + merge in memory (plus `as`'s votes this core holds) |
|
||||
| watcher facts | the watcher, `POST /api/files/register` | `local_file` + `cid_group` |
|
||||
| service scratch | `PUT/GET/DELETE /api/kv/value` | non-reserved cache `kv` rows |
|
||||
| wipe | `POST /api/metadata/clear` | both databases |
|
||||
|
||||
The Go API in `internal/storage/client.go`:
|
||||
|
||||
| Method | Behaviour |
|
||||
|---|---|
|
||||
| `GetMetadataFlat(uuid)` | Range-read `[file:<uuid>/, file:<uuid>0)`, return `map[string]string`. |
|
||||
| `SetMetadataFlat(uuid, m)` | Put each field; add `idx:roots/<uuid>`; auto-register an alias for every `cids/<cid>` member; refresh the FTS row. One transaction. |
|
||||
| `MergeMetadataFlat(uuid, m)` | Like `SetMetadataFlat` but doesn't pre-clear; used by `PATCH /meta/{hash}`. |
|
||||
| `GetProperty(uuid, field)` | Get `file:<uuid>/<field>`. |
|
||||
| `SetProperty(uuid, field, value)` | Put `file:<uuid>/<field>`; auto-register alias if the field is a CID. |
|
||||
| `DeleteRoot(uuid)` | Delete the `file:<uuid>/` range, its duplicates, search row and index membership; unmap every reverse-index entry. |
|
||||
| `Mint(filePath, size, mtimeNano)` | `NewUUID()` + write `filePath/sizeByte/mtimeNano` + add to index. |
|
||||
| `AddAlias(uuid, "<bareCid>")` | Set `cid:<bareCid>` → `<uuid>` and the key-set member `file:<uuid>/cids/<bareCid>` = `"true"`. Nothing to reconcile — canonical is derived on read. |
|
||||
| `ResolveRoot("<bareCid>")` | Get `cid:<bareCid>` → uuid (returns the input unchanged if it is already a UUID root). |
|
||||
| `GetMetadataDocument(uuid)` | Flat fields + `cids` hoisted to an array + `canonical_cid` **derived** by `cid.Better` rank + `duplicates`. Backs `GET /api/meta/{cid}`. |
|
||||
Truth writes go through **one writer goroutine** with group commit; reads use
|
||||
a separate pool and see committed state (WAL). Read paths and shapes are
|
||||
unchanged from v1.x — they are now served from the derived `file:` rows.
|
||||
|
||||
## Reconstruction (flat → nested)
|
||||
|
||||
When the HTTP API returns a document, it reconstructs the flattened
|
||||
storage back into a nested JSON object:
|
||||
The HTTP API reconstructs the merged flat keys into a nested JSON object:
|
||||
|
||||
```
|
||||
flat nested
|
||||
@@ -262,134 +205,122 @@ fileinfo/streamdetails/ streamdetails: { video:
|
||||
streams/0/language = "eng" streams: [ { language: "eng" }, ... ]
|
||||
```
|
||||
|
||||
Rules:
|
||||
- Numeric path segments become array indices on read.
|
||||
- All values are strings on the wire; type interpretation is the caller's
|
||||
job (documented per field in `METADATA_KEYS.md`).
|
||||
- Box-local fields from `local_file` are joined in unsigned; they never
|
||||
appear in `GET /api/meta/{cid}/claims` or on the swarm.
|
||||
|
||||
- `/` is the path separator.
|
||||
- Numeric path segments (`0`, `1`, ...) become array indices on read.
|
||||
- All values are strings on the wire; type interpretation is the
|
||||
caller's job (and is documented per-field in `METADATA_KEYS.md`).
|
||||
- `null`/`undefined` round-trip as empty strings.
|
||||
## Search structures
|
||||
|
||||
## Plugin contribution flow
|
||||
Both live in the cache and are maintained in the same transaction as the
|
||||
merged-row change, so results are fresh as soon as a change is applied:
|
||||
|
||||
Plugins don't talk to the store directly. They write metadata via meta-core's HTTP
|
||||
API (`PUT /meta/{hash}/{key...}` or `PATCH /meta/{hash}`). Each plugin
|
||||
writes under a prefix:
|
||||
- **`kv_field_v`** — an expression index over the field name parsed out of
|
||||
the key, `lower(v)` and `v`, restricted to `file:` rows. Answers
|
||||
"which records have field F with a value like V" for any field, including
|
||||
ones invented tomorrow; covering for the substring test.
|
||||
- **`fts`** — FTS5 trigram `(rid UNINDEXED, hay)`, one row per record: the
|
||||
haystack of its searchable merged fields plus the rid. Every 3+ character substring is
|
||||
indexable.
|
||||
|
||||
```
|
||||
{plugin-id}/<field> single value
|
||||
{plugin-id}/<group>/<field> grouped
|
||||
{plugin-id}/<group>/<n>/<field> indexed (array)
|
||||
```
|
||||
`POST /api/metadata/search` builds a candidate superset in SQL and verifies
|
||||
each hydrated merged record with the Go matchers (`recordMatchesFilter`,
|
||||
`matchAllTokens`); results come in rid order. Plan, rejected shapes and the
|
||||
measurement status are in
|
||||
[`metacore-storage-engine.md`](../../../docs/project-architecture/metacore-storage-engine.md) §8.
|
||||
|
||||
See [`METADATA_KEYS.md`](../../../METADATA_KEYS.md) for the full,
|
||||
authoritative list of fields each plugin writes (content identification,
|
||||
file stats, video classification, anime detection, subtitle references,
|
||||
Jellyfin metadata, FFmpeg stream details, etc.) and the value formats
|
||||
(`int-string`, `lang3`, `path-relfiles`, `csv-set`, ...). This doc
|
||||
deliberately does not duplicate that registry — it would drift.
|
||||
## Schema-version sentinel
|
||||
|
||||
Truth `meta` row `schema-version` = `"3"`, set at first boot and checked on
|
||||
every boot (`internal/storage/schema_version.go`), after both files are
|
||||
opened and before the cache catch-up. A truth file holding the v1 `kv` table
|
||||
(v1.x, `meta-core:schema-version = "2"`) is **explicitly refused**, naming the
|
||||
migration tool; claims without a sentinel, or another version, are refused
|
||||
too; a fresh store is stamped. Alpha policy: no migration code in the
|
||||
service binary; operators wipe and restart. watch.nsl.sh is converted once by
|
||||
the separate one-off tool `packages/meta-core/tools/claims-migrate` (not in
|
||||
the image; rids keep their UUIDs; legacy fields re-signed by a throwaway key;
|
||||
UDL cells re-signed by their account keys — implementation basis §11).
|
||||
|
||||
## Event streams
|
||||
|
||||
Two event streams are kept inside the store, as rows
|
||||
`ev:<stream>/<ms13>-<seq10>` (zero-padded so key order is ID order) plus a
|
||||
high-water mark `evhwm:<stream>` (`internal/kv/stream.go`):
|
||||
|
||||
| Stream | Producer | Contents |
|
||||
|---|---|---|
|
||||
| `file:events` | `internal/watcher` dispatcher | `add` / `change` / `delete` / `rename` / `reset` events for files on the `/files` volume. |
|
||||
| `meta:events` | the storage engine (`internal/kv`) | `{type: set\|del, key, ts}` for every Put/Delete of a `file:` key, appended **in the same transaction** as the change. |
|
||||
| `/api/events/meta` | the cache engine, as `ev:meta:events/…` rows of the cache `kv` table | `{type: set\|del, key, ts}` for every change of a **merged** value, in the same cache transaction as the change |
|
||||
| `/api/events/files` | `internal/watcher` dispatcher (`ev:file:events/…` rows) | `add` / `change` / `delete` / `rename` / `reset` for files on `/files` (unchanged) |
|
||||
|
||||
`set` is emitted only for a **real** change: a Put of the value a key
|
||||
already holds is a no-op — no row write, no index churn, no event (v1.2.0;
|
||||
Redis published `set` even then, and with gateway hits re-persisted on
|
||||
every repeated query most writes were exactly that). `del` is emitted only
|
||||
when the key existed. Retention is ~100k entries, trimmed every 10k appends.
|
||||
|
||||
There is **no boot-time republish** (v1.2.0). A consumer that wants the whole
|
||||
current state asks `/api/events/meta` for `Last-Event-ID: 0` / `0-0` and gets
|
||||
a **snapshot**: one `set` per existing record key, streamed from the kv
|
||||
range, then the live tail from the stream head captured before the snapshot
|
||||
(only the last snapshot event carries an `id:`, so a consumer that drops
|
||||
mid-snapshot gets a fresh one). The old republish wrote every field into a
|
||||
stream that retains only ~100k entries, so a consumer replaying it later
|
||||
never saw most records — and it cost ~6 minutes on every boot.
|
||||
|
||||
The Redis-era path — keyspace notifications drained by `MetaPublisher`
|
||||
through a PSUBSCRIBE / consumer-group hop — is gone (2026-10).
|
||||
|
||||
External consumers do **not** read these streams directly. They subscribe
|
||||
via SSE:
|
||||
|
||||
- `GET /api/events/files` — mirror of `file:events`
|
||||
- `GET /api/events/meta` — mirror of `meta:events`
|
||||
|
||||
The SSE wire contract uses `Last-Event-ID` for resume. IDs keep the Redis
|
||||
`<ms>-<seq>` form so cursors persisted by meta-sort / meta-share /
|
||||
meta-stremio across the switch still compare. Heartbeats are SSE comments
|
||||
(`:keep-alive\n\n`) every 30s of silence. If retention has trimmed past
|
||||
a client's cursor, the handler emits one `event: gap` payload before
|
||||
resuming. See `internal/api/sse_events.go` and
|
||||
[`api-mediated-access.md`](api-mediated-access.md) for the full
|
||||
contract.
|
||||
- A vote that does not change the merged value emits nothing.
|
||||
- A group merge emits `del` for every key of the losing rid and `set` for the
|
||||
survivor's keys.
|
||||
- IDs keep the `<ms>-<seq>` form and stay time-based, so persisted cursors
|
||||
keep comparing, including across a cache rebuild.
|
||||
- `Last-Event-ID: 0` / `0-0` returns a snapshot (one `set` per merged key)
|
||||
then the live tail; only the last snapshot event carries an `id:`.
|
||||
- A trimmed cursor gets one `event: gap`. A cache rebuild emits no per-key
|
||||
events; consumers get `gap` and take a snapshot.
|
||||
- Heartbeats are SSE comments (`:keep-alive\n\n`) every 30 s of silence.
|
||||
Contract: `internal/api/sse_events.go`, [`api-mediated-access.md`](api-mediated-access.md).
|
||||
|
||||
## Debugging
|
||||
|
||||
The database path is `/meta-core/db/meta-core.db` in the image default and
|
||||
on the store app; dev stacks use `/meta-core-db/meta-core.db` (a named
|
||||
volume — the `/meta-core` bind is 9p on Windows dev boxes, where SQLite WAL
|
||||
is unsafe). Always open it **read-only**; writes must go through the API.
|
||||
Paths: truth `/meta-core/db/meta-core.db`, cache `/meta-core/db-cache/cache.db`
|
||||
(image default and store app); dev stacks keep both on named volumes (the
|
||||
`/meta-core` bind is 9p on Windows dev boxes, where SQLite WAL is unsafe).
|
||||
Always open **read-only**; writes go through `POST /api/claims`.
|
||||
|
||||
```bash
|
||||
DB=/meta-core/db/meta-core.db # dev stack: /meta-core-db/meta-core.db
|
||||
Q() { docker exec metacore-app sqlite3 -readonly "$DB" "$1"; }
|
||||
T=/meta-core/db/meta-core.db; C=/meta-core/db-cache/cache.db # dev: the named-volume paths
|
||||
QT() { docker exec metacore-app sqlite3 -readonly "$T" "$1"; }
|
||||
QC() { docker exec metacore-app sqlite3 -readonly "$C" "$1"; }
|
||||
|
||||
# Roots
|
||||
Q "SELECT count(*) FROM kv WHERE k >= 'idx:roots/' AND k < 'idx:roots0'"
|
||||
Q "SELECT substr(k,11) FROM kv WHERE k >= 'idx:roots/' AND k < 'idx:roots0' LIMIT 10"
|
||||
|
||||
# Full document for one root
|
||||
Q "SELECT k,v FROM kv WHERE k >= 'file:01JKR8XW5T4QMV7AHNJ2DEFGHK/' AND k < 'file:01JKR8XW5T4QMV7AHNJ2DEFGHK0'"
|
||||
|
||||
# One property
|
||||
Q "SELECT v FROM kv WHERE k = 'file:01JKR8XW5T4QMV7AHNJ2DEFGHK/fileinfo/duration'"
|
||||
|
||||
# Reverse-index lookup (bare CID — no algorithm prefix)
|
||||
Q "SELECT v FROM kv WHERE k = 'cid:bafkreih...'" # → uuid
|
||||
|
||||
# The cids key-set
|
||||
Q "SELECT k FROM kv WHERE k >= 'file:01JKR8XW5T4QMV7AHNJ2DEFGHK/cids/' AND k < 'file:01JKR8XW5T4QMV7AHNJ2DEFGHK/cids0'"
|
||||
|
||||
# canonical_cid is NOT stored. Derive it via the API instead:
|
||||
curl -k https://metacore-dev.localhost:8083/api/meta/<cid> | jq '.canonical_cid, .cids'
|
||||
|
||||
# Schema-version sentinel
|
||||
Q "SELECT v FROM kv WHERE k = 'meta-core:schema-version'"
|
||||
|
||||
# Streams (server-side; external consumers should use SSE)
|
||||
Q "SELECT k,v FROM kv WHERE k >= 'ev:meta:events/' AND k < 'ev:meta:events0' ORDER BY k DESC LIMIT 10"
|
||||
QT "SELECT rid FROM cid_group WHERE cid = 'bafkreih...'" # cid → rid
|
||||
QT "SELECT cid FROM cid_group WHERE rid = '01JKR8XW5T4QMV7AHNJ2DEFGHK'" # the group (full scan: no rid index)
|
||||
QC "SELECT k, v FROM kv WHERE k >= 'file:01JKR8XW5T4QMV7AHNJ2DEFGHK/' AND k < 'file:01JKR8XW5T4QMV7AHNJ2DEFGHK0'" # merged record
|
||||
QT "SELECT c.key, typeof(c.value), c.value, p.uid, c.ts, c.seq
|
||||
FROM claim c JOIN principal p USING(pid) WHERE c.cid = 'bafkreih...'" # every vote on one cid
|
||||
QT "SELECT name, value FROM meta"; QC "SELECT k, v FROM kv WHERE k LIKE 'meta:%'" # last_seq vs applied_seq
|
||||
QC "SELECT path, midhash FROM local_file WHERE path LIKE '/files/watch/%' LIMIT 10"
|
||||
QC "SELECT k, v FROM kv WHERE k >= 'ev:meta:events/' AND k < 'ev:meta:events0' ORDER BY k DESC LIMIT 10"
|
||||
|
||||
# HTTP equivalents — preferred
|
||||
curl -k 'https://metacore-dev.localhost:8083/api/meta/<cid>' # CID → document
|
||||
curl -k 'https://metacore-dev.localhost:8083/meta' # list root IDs
|
||||
curl -k 'https://metacore-dev.localhost:8083/meta/<uuid>' # document by UUID
|
||||
curl -k 'https://metacore-dev.localhost:8083/api/events/meta' # live SSE
|
||||
curl -k 'https://metacore-dev.localhost:8083/api/kv/info' # KV stats
|
||||
curl -k 'https://metacore-dev.localhost:8083/api/meta/<cid>' # merged document (+ canonical_cid)
|
||||
curl -k 'https://metacore-dev.localhost:8083/api/meta/<cid>/claims' # signed votes
|
||||
curl -k 'https://metacore-dev.localhost:8083/api/events/meta' # live SSE
|
||||
```
|
||||
|
||||
## Key implementation files
|
||||
|
||||
| Component | Path |
|
||||
|---|---|
|
||||
| Storage engine (SQLite kv table, writer, group commit) | `internal/kv/kv.go` |
|
||||
| Event streams as kv rows | `internal/kv/stream.go` |
|
||||
| Flat-key record ops | `internal/storage/client.go` |
|
||||
| Search (field index + FTS) | `internal/storage/search.go` |
|
||||
| Single-writer, group-commit SQLite engine | `internal/kv/` |
|
||||
| Truth, apply loop, merge, local files, counters | `internal/storage/{truth,apply,merge,localfile,client}.go` |
|
||||
| Search (`kv_field_v` + FTS) | `internal/storage/search.go` |
|
||||
| MMUR2 claim encoding + verification | `internal/claim/` (secp256k1 in `internal/identity/`) |
|
||||
| UUIDv7 minting | `internal/storage/uuid.go` |
|
||||
| CID reverse index + aliases + canonical-CID derivation | `internal/storage/cid_resolution.go` |
|
||||
| Canonical-CID rank | `internal/cid/rank.go` |
|
||||
| Schema-version gate | `internal/storage/schema_version.go` |
|
||||
| `meta:events` snapshot for `0`/`0-0` | `internal/api/sse_events.go` (`streamSnapshot`) |
|
||||
| Derived counters (root count, per-field schema) | `internal/storage/client.go` (`onChange`), `internal/schema/indexer.go` |
|
||||
| Watcher → `file:events` dispatcher | `internal/watcher/dispatcher.go` |
|
||||
| SSE event mirror | `internal/api/sse_events.go` |
|
||||
| SSE event mirror + snapshot | `internal/api/sse_events.go` |
|
||||
| Watcher → `local_file`, `/api/events/files` | `internal/watcher/` |
|
||||
| Field semantics (authoritative registry) | [`/METADATA_KEYS.md`](../../../METADATA_KEYS.md) |
|
||||
|
||||
## Previous layout (v1.1–v1.2 `kv` table)
|
||||
|
||||
Kept so old notes and scripts can be read. In v2.0.0 the **truth** database
|
||||
has no `kv` table at all; the **cache** keeps the same key layout for derived
|
||||
rows (see above), but nothing in it is written by clients.
|
||||
|
||||
| Key | Value | Replaced by |
|
||||
|---|---|---|
|
||||
| `file:<rid>/<field>` | one value per field, last writer wins | `claim` (one row per signer) → derived cache `file:` rows |
|
||||
| `cid:<cid>` | `<rid>` | `cid_group` (truth), copied to cache `cid:` rows |
|
||||
| `idx:roots/<rid>`, `idx:count/*`, `idx:schema/*`, `idx:fts/<rid>` | root set, counters, FTS rowid | same keys, derived, in the cache |
|
||||
| `dup:<rid>/<path>` | duplicate path | `local_file` rows |
|
||||
| `udl:rec:<uid>/<cid>/<key>`, `udl:idx:*` | User Data Layer cells (MMUR1, `version`) | `claim` (MMUR2, `ts`) + `claim_by_pid` |
|
||||
| `ev:<stream>/<ms13>-<seq10>`, `evhwm:<stream>` | stream rows | same keys, in the cache |
|
||||
| `btmagnet:*`, `searchcov:*`, `nzb:*` | service scratch | same keys, in the cache |
|
||||
| `meta-core:schema-version` = `"2"` | gate | truth `meta` `schema-version` = `"3"` |
|
||||
|
||||
The flat-key idea survived — a field is still one string key, a key-set
|
||||
member is still inside the key — but the record moved from "one value per
|
||||
field" to "one value per signer per field".
|
||||
|
||||
Reference in New Issue
Block a user