meta-core/docs/metadata-storage-structure.md

# Metadata Storage Structure

How meta-core lays out file metadata in its store. This document covers the
on-disk shape (UUID-rooted flat string keys, CID reverse index,
per-root sets) and the schema-version gate. It does **not** enumerate
field semantics — that's the job of the repo-root
[`METADATA_KEYS.md`](../../../METADATA_KEYS.md), the authoritative key
registry for everyone who writes to or reads from metadata.

For the design rationale (why UUIDs instead of content hashes, why a
reverse index), see [`uuid-rooted-metadata.md`](uuid-rooted-metadata.md).
For the external-access contract (HTTP-only, SSE for events), see
[`api-mediated-access.md`](api-mediated-access.md).

> **Storage engine (since 2026-10): an embedded SQLite file, not Redis.**
> Until 2026-10 these keys lived in a Redis server spawned inside the
> container. They now live in one string→string table of an embedded SQLite
> database (pure-Go driver `modernc.org/sqlite`, CGO off). The **key names
> did not change** — only the Redis *types* that some bookkeeping used (SETs,
> HASHes, Streams) were flattened into plain keys, listed below. Holding
> everything in RAM had cost ~6.6 GB on watch.nsl.sh's 169k records (Redis
> plus the API's in-memory search copy) and caused repeated OOM kills.

## Quick summary

- **One table**: `kv(k TEXT PRIMARY KEY, v TEXT NOT NULL) WITHOUT ROWID`,
  schema-free — a new metadata field is just a new key, never a migration.
  File: `/meta-core/db/meta-core.db` (override `META_CORE_DB_PATH`), WAL
  mode, `synchronous=NORMAL`.
- **Roots are UUIDv7** encoded as 26-char Crockford Base32 (ULID layout),
  not content hashes. Examples in this doc use the placeholder
  `01JKR8XW5T4QMV7AHNJ2DEFGHK`.
- **Properties are flat string keys**, one per leaf:
  `file:<uuid>/<property/path>` → string value. A record is exactly the
  key range `[file:<uuid>/, file:<uuid>0)` — there is no per-record field
  list.
- **CIDs are reverse-index aliases**: `cid:<bareCid>` → `<uuid>`
  (one key per known CID; no algorithm prefix — the multicodec
  already carries it).
- **Per-root sets**: `file:<uuid>/cids/<bareCid>` (a *key-set*: one key
  `"true"` per known CID, midhash included), `dup:<uuid>/<path>` (alternate
  paths with the same content), plus `idx:roots/<uuid>` (every UUID).
  There is **no stored `canonical_cid`** — it is derived by rank on read.
- **A schema-version sentinel** (`meta-core:schema-version`) refuses to
  boot against legacy data.

Implementation lives in `internal/kv/kv.go` (engine),
`internal/kv/stream.go` (event streams), `internal/storage/client.go`,
`internal/storage/search.go`, `internal/storage/cid_resolution.go`,
`internal/storage/uuid.go`, and `internal/storage/schema_version.go`.

## Key structure

Everything is one string→string table. The full layout:

| Key | Value | Notes |
|---|---|---|
| `file:<rid>/<field>` | field value | a record property — the **only** thing under `file:` |
| `cid:<cid>` | `<rid>` | reverse index |
| `meta-core:schema-version` | `"2"` | schema gate |
| `idx:roots/<rid>` | `""` | set of every root (was the Redis SET `file:__index__`) |
| `idx:fts/<rid>` | FTS rowid | the record's row in the FTS table |
| `dup:<rid>/<path>` | `""` | a duplicate path (was the SET `file:<rid>/duplicates`) |
| `udl:rec:<uid>/<cid>/<key>` | JSON `{record,version,ts,value?}` | User Data Layer entry (was a HASH) |
| `udl:idx:…<set>\x1f<member>` | `""` | UDL index sets, one key per member |
| `ev:<stream>/<ms13>-<seq10>` | JSON of the entry's fields | `meta:events` / `file:events` entries (were Redis Streams) |
| `evhwm:<stream>` | last issued ID | stream high-water mark |
| `btmagnet:*`, `searchcov:*`, `nzb:*` | — | written by other services via `/api/kv/value`; unchanged |

Keeping bookkeeping **out of `file:`** is load-bearing: the `file:` range is
exactly the records, so the field index covers only real fields and every
change under `file:` is a real field change worth publishing on
`meta:events`. The per-record `file:<rid>/__fields__` SET and the
`file:__dirty__` set of the Redis era no longer exist. Roots are split from
the field at the **first** `/`, so CID-rooted and UUID-rooted records both
parse.

### Roots

```
file:<uuid>/<property/path>   →   string (always)
```

UUIDs are produced by `storage.NewUUID()` (`internal/storage/uuid.go`):
UUIDv7 → 26-char Crockford Base32 (skips I/L/O/U; ULID-compatible). The
time-sortable prefix means roots minted in a given window sort together
(`k >= 'file:01JKR8' AND k < 'file:01JKR9'`), which restored some of the
debug-ability lost when midhash256 stopped being the root.

Properties use `/` as the path separator. Nested objects in source code
are flattened to flat keys on write and reconstructed on read. Examples:

```
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/filePath                          = "/files/watch/movies/Inception.2010.mkv"
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/sizeByte                          = "15032385536"
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/mtimeNano                         = "1729872000000000000"
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/title                             = "Inception"
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/fileinfo/duration                 = "8878.5"
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/fileinfo/streamdetails/video/0/codec = "h264"
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/tmdb/poster                       = "bafkreih..."
```

### Reverse-index aliases (CIDs)

Every CID known for a file is registered as its own key, keyed by
the **bare CID** — no algorithm prefix. A CIDv1 already encodes its hash
function in the multicodec, so naming it again in the key would be
redundant (and would let the same digest be indexed under two spellings):

```
cid:<bareCid>   →   <uuid>
```

Examples:

```
cid:bafkreih5kapbjzq...   →   01JKR8XW5T4QMV7AHNJ2DEFGHK
cid:bagacbabaec7v3...     →   01JKR8XW5T4QMV7AHNJ2DEFGHK
```

The reverse index makes `GET /api/meta/<cid>` a primary-key lookup for any
registered CID, independent of which hash algorithm — there is no primary.
The record write path (`writeFieldsTx` in `internal/storage/client.go`, used
by `SetProperty`, `SetMetadataFlat` and `MergeMetadataFlat`) registers the
alias for any field whose name starts with `cids/`, in the same transaction,
so writers emit key-set members without knowing the reverse index exists.

> ⚠ The hook matches **only** the `cids/` prefix. A legacy `cid_*` or
> `midhash256` *named* field written to a root is stored verbatim and is
> **never indexed** — the record becomes unresolvable by that CID. Writers
> must collapse such fields to `cids/<cid>` members before the write. See
> `METADATA_KEYS.md` §14.13.

### Per-root sets

```
file:<uuid>/cids/<bareCid>    →   "true" — one key per known CID (a key-set)
dup:<uuid>/<path>             →   ""     — one key per alternate path with same content
```

`cids` is a **key-set**: one flat `file:<uuid>/cids/<cid>` key per member,
which keeps it conflict-free to merge across peers (a single set-valued
field would not be).
The record's own address (the midhash) is just the `0x1000` member — it has
no separate field.

There is **no stored `canonical_cid`**. The canonical CID is *derived on
read*: `GetMetadataDocument` (`internal/storage/cid_resolution.go`) ranks the
key-set's members through `internal/cid/rank.go` (`cid.Better`) and returns
the winner as the `canonical_cid` field of the JSON response only. Nothing
reconciles or persists it — an earlier `reconcileCanonicalCIDLocked` did, and
was removed with the flat `cid_*` keys.

### Global index

```
idx:roots/<uuid>   →   ""   (one key per root; was the Redis SET file:__index__)
```

`internal/storage/client.go::GetAllHashIDs` range-reads `idx:roots/` to
enumerate every file without scanning the records.

### Search structures (derived)

Two derived structures are maintained on top of `kv`, in the same
transaction as every record write. Both are rebuildable from the kv rows and
never authoritative (`internal/storage/search.go`):

- **`kv_field`** — a partial expression index
  `ON kv(lower(substr(k,instr(k,'/')+1)), lower(v)) WHERE k>='file:' AND k<'file;' AND instr(k,'/')>0`.
  It answers "which records have field F with a value like V" for **any**
  field, including ones invented tomorrow — the index is over the key itself.
- **`fts`** — an FTS5 trigram table `fts(rid UNINDEXED, hay)` holding one row
  per record: its search haystack plus its rid (`idx:fts/<rid>` points at the
  row). Trigrams make every 3+ character substring indexable.

`POST /api/metadata/search` uses SQL to build a superset of candidates, then
verifies each with the unchanged Go matchers (`recordMatchesFilter`,
`matchAllTokens`), so semantics are identical to the old in-memory index.
Results still come in hashID order and are fresh immediately (no refresh
lag). This replaced the API's in-memory `searchIndex` (60 s refresh, ~30 KB
RAM per record).

### Schema-version sentinel

```
meta-core:schema-version   →   string (currently "2")
```

Set at first boot, checked on every subsequent boot
(`internal/storage/schema_version.go`). If the sentinel is missing but
the root index is non-empty, the gate refuses to boot (legacy data).
If the sentinel is present but doesn't match `SchemaVersion`, the gate
also refuses to boot. Alpha-stage policy: operators wipe and restart;
there is no automated migration.

## Why flat string keys?

The store keeps one flat key per leaf rather than a document or a
per-record row. The reasons are:

1. **Schema-free.** A plugin inventing a field writes a new key; nothing
   declares it, and the table never needs a migration. The field index
   covers it automatically.
2. **Conflict-free merging across peers.** Key-sets (`cids/<cid>`, and the
   flat fields in general) merge by union of keys — no read-modify-write of
   a shared value, nothing to reconcile.
3. **Selective updates without read-modify-write.** A plugin writing
   `tmdb/poster` doesn't have to read the rest of the document.
4. **Compatibility with the CID alias hooks.** Field writes can trigger
   alias-maintenance work in the write path without the plugin caring.

(Under Redis there was a fourth reason — field-level keyspace notifications
fed `meta:events`. That is obsolete: events are now written by the storage
engine itself, see [Event streams](#event-streams).)

## Reads and writes

All writes go through **one writer goroutine** with group commit: concurrent
writes share one SQLite transaction, each under its own SAVEPOINT so a
failure rolls back alone. A record write — fields + root index + CID aliases
+ FTS row + events — is atomic. (Measured ~835 records/s with 16 concurrent
writers of 40-field records.) Reads use a separate connection pool and see
the last committed state.

The Go API in `internal/storage/client.go`:

| Method | Behaviour |
|---|---|
| `GetMetadataFlat(uuid)` | Range-read `[file:<uuid>/, file:<uuid>0)`, return `map[string]string`. |
| `SetMetadataFlat(uuid, m)` | Put each field; add `idx:roots/<uuid>`; auto-register an alias for every `cids/<cid>` member; refresh the FTS row. One transaction. |
| `MergeMetadataFlat(uuid, m)` | Like `SetMetadataFlat` but doesn't pre-clear; used by `PATCH /meta/{hash}`. |
| `GetProperty(uuid, field)` | Get `file:<uuid>/<field>`. |
| `SetProperty(uuid, field, value)` | Put `file:<uuid>/<field>`; auto-register alias if the field is a CID. |
| `DeleteRoot(uuid)` | Delete the `file:<uuid>/` range, its duplicates, search row and index membership; unmap every reverse-index entry. |
| `Mint(filePath, size, mtimeNano)` | `NewUUID()` + write `filePath/sizeByte/mtimeNano` + add to index. |
| `AddAlias(uuid, "<bareCid>")` | Set `cid:<bareCid>` → `<uuid>` and the key-set member `file:<uuid>/cids/<bareCid>` = `"true"`. Nothing to reconcile — canonical is derived on read. |
| `ResolveRoot("<bareCid>")` | Get `cid:<bareCid>` → uuid (returns the input unchanged if it is already a UUID root). |
| `GetMetadataDocument(uuid)` | Flat fields + `cids` hoisted to an array + `canonical_cid` **derived** by `cid.Better` rank + `duplicates`. Backs `GET /api/meta/{cid}`. |

## Reconstruction (flat → nested)

When the HTTP API returns a document, it reconstructs the flattened
storage back into a nested JSON object:

```
flat                                          nested
─────────────────────────────────────         ───────────────────────────
title                       = "Inception"     title: "Inception"
fileinfo/duration           = "8878.5"        fileinfo: { duration: "8878.5",
fileinfo/streamdetails/                                  streamdetails: { video: { 0: {
  video/0/codec             = "h264"                       codec: "h264", ... } }, ... } }
streams/0/language          = "eng"           streams: [ { language: "eng" }, ... ]
```

Rules:

- `/` is the path separator.
- Numeric path segments (`0`, `1`, ...) become array indices on read.
- All values are strings on the wire; type interpretation is the
  caller's job (and is documented per-field in `METADATA_KEYS.md`).
- `null`/`undefined` round-trip as empty strings.

## Plugin contribution flow

Plugins don't talk to the store directly. They write metadata via meta-core's HTTP
API (`PUT /meta/{hash}/{key...}` or `PATCH /meta/{hash}`). Each plugin
writes under a prefix:

```
{plugin-id}/<field>                 single value
{plugin-id}/<group>/<field>         grouped
{plugin-id}/<group>/<n>/<field>     indexed (array)
```

See [`METADATA_KEYS.md`](../../../METADATA_KEYS.md) for the full,
authoritative list of fields each plugin writes (content identification,
file stats, video classification, anime detection, subtitle references,
Jellyfin metadata, FFmpeg stream details, etc.) and the value formats
(`int-string`, `lang3`, `path-relfiles`, `csv-set`, ...). This doc
deliberately does not duplicate that registry — it would drift.

## Event streams

Two event streams are kept inside the store, as rows
`ev:<stream>/<ms13>-<seq10>` (zero-padded so key order is ID order) plus a
high-water mark `evhwm:<stream>` (`internal/kv/stream.go`):

| Stream | Producer | Contents |
|---|---|---|
| `file:events` | `internal/watcher` dispatcher | `add` / `change` / `delete` / `rename` / `reset` events for files on the `/files` volume. |
| `meta:events` | the storage engine (`internal/kv`) | `{type: set\|del, key, ts}` for every Put/Delete of a `file:` key, appended **in the same transaction** as the change. |

`del` is emitted only when the key existed; `set` is emitted even for an
unchanged value (Redis semantics, kept). Retention is ~100k entries,
trimmed every 10k appends. At boot `MetaPublisher`
(`internal/events/meta_publisher.go`) still republishes every field into
`meta:events` (meta-fuse rebuilds its VFS only by replaying from `0-0`),
batched 5k events per transaction.

The Redis-era path — keyspace notifications drained by `MetaPublisher`
through a PSUBSCRIBE / consumer-group hop — is gone (2026-10).

External consumers do **not** read these streams directly. They subscribe
via SSE:

- `GET /api/events/files` — mirror of `file:events`
- `GET /api/events/meta` — mirror of `meta:events`

The SSE wire contract uses `Last-Event-ID` for resume. IDs keep the Redis
`<ms>-<seq>` form so cursors persisted by meta-sort / meta-share /
meta-stremio across the switch still compare. Heartbeats are SSE comments
(`:keep-alive\n\n`) every 30s of silence. If retention has trimmed past
a client's cursor, the handler emits one `event: gap` payload before
resuming. See `internal/api/sse_events.go` and
[`api-mediated-access.md`](api-mediated-access.md) for the full
contract.

## Debugging

The database path is `/meta-core/db/meta-core.db` in the image default and
on the store app; dev stacks use `/meta-core-db/meta-core.db` (a named
volume — the `/meta-core` bind is 9p on Windows dev boxes, where SQLite WAL
is unsafe). Always open it **read-only**; writes must go through the API.

```bash
DB=/meta-core/db/meta-core.db      # dev stack: /meta-core-db/meta-core.db
Q() { docker exec metacore-app sqlite3 -readonly "$DB" "$1"; }

# Roots
Q "SELECT count(*) FROM kv WHERE k >= 'idx:roots/' AND k < 'idx:roots0'"
Q "SELECT substr(k,11) FROM kv WHERE k >= 'idx:roots/' AND k < 'idx:roots0' LIMIT 10"

# Full document for one root
Q "SELECT k,v FROM kv WHERE k >= 'file:01JKR8XW5T4QMV7AHNJ2DEFGHK/' AND k < 'file:01JKR8XW5T4QMV7AHNJ2DEFGHK0'"

# One property
Q "SELECT v FROM kv WHERE k = 'file:01JKR8XW5T4QMV7AHNJ2DEFGHK/fileinfo/duration'"

# Reverse-index lookup (bare CID — no algorithm prefix)
Q "SELECT v FROM kv WHERE k = 'cid:bafkreih...'"                   # → uuid

# The cids key-set
Q "SELECT k FROM kv WHERE k >= 'file:01JKR8XW5T4QMV7AHNJ2DEFGHK/cids/' AND k < 'file:01JKR8XW5T4QMV7AHNJ2DEFGHK/cids0'"

# canonical_cid is NOT stored. Derive it via the API instead:
curl -k https://metacore-dev.localhost:8083/api/meta/<cid> | jq '.canonical_cid, .cids'

# Schema-version sentinel
Q "SELECT v FROM kv WHERE k = 'meta-core:schema-version'"

# Streams (server-side; external consumers should use SSE)
Q "SELECT k,v FROM kv WHERE k >= 'ev:meta:events/' AND k < 'ev:meta:events0' ORDER BY k DESC LIMIT 10"

# HTTP equivalents — preferred
curl -k 'https://metacore-dev.localhost:8083/api/meta/<cid>'      # CID → document
curl -k 'https://metacore-dev.localhost:8083/meta'                # list root IDs
curl -k 'https://metacore-dev.localhost:8083/meta/<uuid>'         # document by UUID
curl -k 'https://metacore-dev.localhost:8083/api/events/meta'     # live SSE
curl -k 'https://metacore-dev.localhost:8083/api/kv/info'         # KV stats
```

## Key implementation files

| Component | Path |
|---|---|
| Storage engine (SQLite kv table, writer, group commit) | `internal/kv/kv.go` |
| Event streams as kv rows | `internal/kv/stream.go` |
| Flat-key record ops | `internal/storage/client.go` |
| Search (field index + FTS) | `internal/storage/search.go` |
| UUIDv7 minting | `internal/storage/uuid.go` |
| CID reverse index + aliases + canonical-CID derivation | `internal/storage/cid_resolution.go` |
| Schema-version gate | `internal/storage/schema_version.go` |
| Boot-time `meta:events` republish | `internal/events/meta_publisher.go` |
| Watcher → `file:events` dispatcher | `internal/watcher/dispatcher.go` |
| SSE event mirror | `internal/api/sse_events.go` |
| Field semantics (authoritative registry) | [`/METADATA_KEYS.md`](../../../METADATA_KEYS.md) |