meta-core/docs/metadata-storage-structure.md

Metadata Storage Structure

How meta-core lays out file metadata in its store. This document covers the on-disk shape (UUID-rooted flat string keys, CID reverse index, per-root sets) and the schema-version gate. It does not enumerate field semantics — that's the job of the repo-root METADATA_KEYS.md, the authoritative key registry for everyone who writes to or reads from metadata.

For the design rationale (why UUIDs instead of content hashes, why a reverse index), see uuid-rooted-metadata.md. For the external-access contract (HTTP-only, SSE for events), see api-mediated-access.md.

Storage engine (since 2026-10): an embedded SQLite file, not Redis. Until 2026-10 these keys lived in a Redis server spawned inside the container. They now live in one string→string table of an embedded SQLite database (pure-Go driver modernc.org/sqlite, CGO off). The key names did not change — only the Redis types that some bookkeeping used (SETs, HASHes, Streams) were flattened into plain keys, listed below. Holding everything in RAM had cost ~6.6 GB on watch.nsl.sh's 169k records (Redis plus the API's in-memory search copy) and caused repeated OOM kills.

Quick summary

• One table: kv(k TEXT PRIMARY KEY, v TEXT NOT NULL) WITHOUT ROWID, schema-free — a new metadata field is just a new key, never a migration. File: /meta-core/db/meta-core.db (override META_CORE_DB_PATH), WAL mode, synchronous=NORMAL.
• Roots are UUIDv7 encoded as 26-char Crockford Base32 (ULID layout), not content hashes. Examples in this doc use the placeholder 01JKR8XW5T4QMV7AHNJ2DEFGHK.
• Properties are flat string keys, one per leaf: file:<uuid>/<property/path> → string value. A record is exactly the key range [file:<uuid>/, file:<uuid>0) — there is no per-record field list.
• CIDs are reverse-index aliases: cid:<bareCid> → <uuid> (one key per known CID; no algorithm prefix — the multicodec already carries it).
• Per-root sets: file:<uuid>/cids/<bareCid> (a key-set: one key "true" per known CID, midhash included), dup:<uuid>/<path> (alternate paths with the same content), plus idx:roots/<uuid> (every UUID). There is no stored canonical_cid — it is derived by rank on read.
• A schema-version sentinel (meta-core:schema-version) refuses to boot against legacy data.

Implementation lives in internal/kv/kv.go (engine), internal/kv/stream.go (event streams), internal/storage/client.go, internal/storage/search.go, internal/storage/cid_resolution.go, internal/storage/uuid.go, and internal/storage/schema_version.go.

Key structure

Everything is one string→string table. The full layout:

Key	Value	Notes
file:<rid>/<field>	field value	a record property — the only thing under file:
cid:<cid>	<rid>	reverse index
meta-core:schema-version	"2"	schema gate
idx:roots/<rid>	""	set of every root (was the Redis SET file:__index__)
idx:fts/<rid>	FTS rowid	the record's row in the FTS table
idx:count/roots	integer	number of roots — search's total, /api/stats records (v1.2.0)
idx:schema/<field>\x1f<keyHint>\x1f<prim>\x1f<hint>	integer	per-field value-type counts behind /api/schema (v1.2.0)
idx:count/ready	"1"	marks the counters as built (absent → one-time build at boot)
dup:<rid>/<path>	""	a duplicate path (was the SET file:<rid>/duplicates)
udl:rec:<uid>/<cid>/<key>	JSON {record,version,ts,value?}	User Data Layer entry (was a HASH)
udl:idx:…<set>\x1f<member>	""	UDL index sets, one key per member
ev:<stream>/<ms13>-<seq10>	JSON of the entry's fields	meta:events / file:events entries (were Redis Streams)
evhwm:<stream>	last issued ID	stream high-water mark
btmagnet:*, searchcov:*, nzb:*	—	written by other services via /api/kv/value; unchanged

Keeping bookkeeping out of file: is load-bearing: the file: range is exactly the records, so the field index covers only real fields and every change under file: is a real field change worth publishing on meta:events. The per-record file:<rid>/__fields__ SET and the file:__dirty__ set of the Redis era no longer exist. Roots are split from the field at the first /, so CID-rooted and UUID-rooted records both parse.

Roots

file:<uuid>/<property/path>   →   string (always)

UUIDs are produced by storage.NewUUID() (internal/storage/uuid.go): UUIDv7 → 26-char Crockford Base32 (skips I/L/O/U; ULID-compatible). The time-sortable prefix means roots minted in a given window sort together (k >= 'file:01JKR8' AND k < 'file:01JKR9'), which restored some of the debug-ability lost when midhash256 stopped being the root.

Properties use / as the path separator. Nested objects in source code are flattened to flat keys on write and reconstructed on read. Examples:

file:01JKR8XW5T4QMV7AHNJ2DEFGHK/filePath                          = "/files/watch/movies/Inception.2010.mkv"
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/sizeByte                          = "15032385536"
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/mtimeNano                         = "1729872000000000000"
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/title                             = "Inception"
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/fileinfo/duration                 = "8878.5"
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/fileinfo/streamdetails/video/0/codec = "h264"
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/tmdb/poster                       = "bafkreih..."

Reverse-index aliases (CIDs)

Every CID known for a file is registered as its own key, keyed by the bare CID — no algorithm prefix. A CIDv1 already encodes its hash function in the multicodec, so naming it again in the key would be redundant (and would let the same digest be indexed under two spellings):

cid:<bareCid>   →   <uuid>

Examples:

cid:bafkreih5kapbjzq...   →   01JKR8XW5T4QMV7AHNJ2DEFGHK
cid:bagacbabaec7v3...     →   01JKR8XW5T4QMV7AHNJ2DEFGHK

The reverse index makes GET /api/meta/<cid> a primary-key lookup for any registered CID, independent of which hash algorithm — there is no primary. The record write path (writeFieldsTx in internal/storage/client.go, used by SetProperty, SetMetadataFlat and MergeMetadataFlat) registers the alias for any field whose name starts with cids/, in the same transaction, so writers emit key-set members without knowing the reverse index exists.

⚠ The hook matches only the cids/ prefix. A legacy cid_* or midhash256 named field written to a root is stored verbatim and is never indexed — the record becomes unresolvable by that CID. Writers must collapse such fields to cids/<cid> members before the write. See METADATA_KEYS.md §14.13.

Per-root sets

file:<uuid>/cids/<bareCid>    →   "true" — one key per known CID (a key-set)
dup:<uuid>/<path>             →   ""     — one key per alternate path with same content

cids is a key-set: one flat file:<uuid>/cids/<cid> key per member, which keeps it conflict-free to merge across peers (a single set-valued field would not be). The record's own address (the midhash) is just the 0x1000 member — it has no separate field.

There is no stored canonical_cid. The canonical CID is derived on read: GetMetadataDocument (internal/storage/cid_resolution.go) ranks the key-set's members through internal/cid/rank.go (cid.Better) and returns the winner as the canonical_cid field of the JSON response only. Nothing reconciles or persists it — an earlier reconcileCanonicalCIDLocked did, and was removed with the flat cid_* keys.

Global index

idx:roots/<uuid>   →   ""   (one key per root; was the Redis SET file:__index__)

internal/storage/client.go::GetAllHashIDs range-reads idx:roots/ to enumerate every file without scanning the records.

Search structures (derived)

Two derived structures are maintained on top of kv, in the same transaction as every record write. Both are rebuildable from the kv rows and never authoritative (internal/storage/search.go):

• kv_field_v — a partial expression index ON kv(lower(substr(k,instr(k,'/')+1)), lower(v), v) WHERE k>='file:' AND k<'file;' AND instr(k,'/')>0. It answers "which records have field F with a value like V" for any field, including ones invented tomorrow — the index is over the key itself. It carries v so the substring test is answered from the index alone (covering; 13 ms vs 1.48 s without it, for tmdbid on watch).
• fts — an FTS5 trigram table fts(rid UNINDEXED, hay) holding one row per record: its search haystack plus its rid (idx:fts/<rid> points at the row). Trigrams make every 3+ character substring indexable.

POST /api/metadata/search uses SQL to build a superset of candidates, then verifies each with the unchanged Go matchers (recordMatchesFilter, matchAllTokens), so semantics are identical to the old in-memory index. The smallest part (by a capped count) is collected once and sorted; every other filter is probed per page as an EXISTS over the candidate's own key range. The plan, its measurements and the rejected shapes are in docs/project-architecture/metacore-storage-engine.md. Results still come in hashID order and are fresh immediately (no refresh lag). This replaced the API's in-memory searchIndex (60 s refresh, ~30 KB RAM per record).

Schema-version sentinel

meta-core:schema-version   →   string (currently "2")

Set at first boot, checked on every subsequent boot (internal/storage/schema_version.go). If the sentinel is missing but the root index is non-empty, the gate refuses to boot (legacy data). If the sentinel is present but doesn't match SchemaVersion, the gate also refuses to boot. Alpha-stage policy: operators wipe and restart; there is no automated migration.

Why flat string keys?

The store keeps one flat key per leaf rather than a document or a per-record row. The reasons are:

1. Schema-free. A plugin inventing a field writes a new key; nothing declares it, and the table never needs a migration. The field index covers it automatically.
2. Conflict-free merging across peers. Key-sets (cids/<cid>, and the flat fields in general) merge by union of keys — no read-modify-write of a shared value, nothing to reconcile.
3. Selective updates without read-modify-write. A plugin writing tmdb/poster doesn't have to read the rest of the document.
4. Compatibility with the CID alias hooks. Field writes can trigger alias-maintenance work in the write path without the plugin caring.

(Under Redis there was a fourth reason — field-level keyspace notifications fed meta:events. That is obsolete: events are now written by the storage engine itself, see Event streams.)

Reads and writes

All writes go through one writer goroutine with group commit: concurrent writes share one SQLite transaction, each under its own SAVEPOINT so a failure rolls back alone. A record write — fields + root index + CID aliases

• FTS row + events — is atomic. (Measured ~835 records/s with 16 concurrent writers of 40-field records.) Reads use a separate connection pool and see the last committed state.

The Go API in internal/storage/client.go:

Method	Behaviour
GetMetadataFlat(uuid)	Range-read [file:<uuid>/, file:<uuid>0), return map[string]string.
SetMetadataFlat(uuid, m)	Put each field; add idx:roots/<uuid>; auto-register an alias for every cids/<cid> member; refresh the FTS row. One transaction.
MergeMetadataFlat(uuid, m)	Like SetMetadataFlat but doesn't pre-clear; used by PATCH /meta/{hash}.
GetProperty(uuid, field)	Get file:<uuid>/<field>.
SetProperty(uuid, field, value)	Put file:<uuid>/<field>; auto-register alias if the field is a CID.
DeleteRoot(uuid)	Delete the file:<uuid>/ range, its duplicates, search row and index membership; unmap every reverse-index entry.
Mint(filePath, size, mtimeNano)	NewUUID() + write filePath/sizeByte/mtimeNano + add to index.
AddAlias(uuid, "<bareCid>")	Set cid:<bareCid> → <uuid> and the key-set member file:<uuid>/cids/<bareCid> = "true". Nothing to reconcile — canonical is derived on read.
ResolveRoot("<bareCid>")	Get cid:<bareCid> → uuid (returns the input unchanged if it is already a UUID root).
GetMetadataDocument(uuid)	Flat fields + cids hoisted to an array + canonical_cid derived by cid.Better rank + duplicates. Backs GET /api/meta/{cid}.

Reconstruction (flat → nested)

When the HTTP API returns a document, it reconstructs the flattened storage back into a nested JSON object:

flat                                          nested
─────────────────────────────────────         ───────────────────────────
title                       = "Inception"     title: "Inception"
fileinfo/duration           = "8878.5"        fileinfo: { duration: "8878.5",
fileinfo/streamdetails/                                  streamdetails: { video: { 0: {
  video/0/codec             = "h264"                       codec: "h264", ... } }, ... } }
streams/0/language          = "eng"           streams: [ { language: "eng" }, ... ]

Rules:

• / is the path separator.
• Numeric path segments (0, 1, ...) become array indices on read.
• All values are strings on the wire; type interpretation is the caller's job (and is documented per-field in METADATA_KEYS.md).
• null/undefined round-trip as empty strings.

Plugin contribution flow

Plugins don't talk to the store directly. They write metadata via meta-core's HTTP API (PUT /meta/{hash}/{key...} or PATCH /meta/{hash}). Each plugin writes under a prefix:

{plugin-id}/<field>                 single value
{plugin-id}/<group>/<field>         grouped
{plugin-id}/<group>/<n>/<field>     indexed (array)

See METADATA_KEYS.md for the full, authoritative list of fields each plugin writes (content identification, file stats, video classification, anime detection, subtitle references, Jellyfin metadata, FFmpeg stream details, etc.) and the value formats (int-string, lang3, path-relfiles, csv-set, ...). This doc deliberately does not duplicate that registry — it would drift.

Event streams

Two event streams are kept inside the store, as rows ev:<stream>/<ms13>-<seq10> (zero-padded so key order is ID order) plus a high-water mark evhwm:<stream> (internal/kv/stream.go):

Stream	Producer	Contents
file:events	internal/watcher dispatcher	add / change / delete / rename / reset events for files on the /files volume.
meta:events	the storage engine (internal/kv)	{type: set|del, key, ts} for every Put/Delete of a file: key, appended in the same transaction as the change.

set is emitted only for a real change: a Put of the value a key already holds is a no-op — no row write, no index churn, no event (v1.2.0; Redis published set even then, and with gateway hits re-persisted on every repeated query most writes were exactly that). del is emitted only when the key existed. Retention is ~100k entries, trimmed every 10k appends.

There is no boot-time republish (v1.2.0). A consumer that wants the whole current state asks /api/events/meta for Last-Event-ID: 0 / 0-0 and gets a snapshot: one set per existing record key, streamed from the kv range, then the live tail from the stream head captured before the snapshot (only the last snapshot event carries an id:, so a consumer that drops mid-snapshot gets a fresh one). The old republish wrote every field into a stream that retains only ~100k entries, so a consumer replaying it later never saw most records — and it cost ~6 minutes on every boot.

The Redis-era path — keyspace notifications drained by MetaPublisher through a PSUBSCRIBE / consumer-group hop — is gone (2026-10).

External consumers do not read these streams directly. They subscribe via SSE:

• GET /api/events/files — mirror of file:events
• GET /api/events/meta — mirror of meta:events

The SSE wire contract uses Last-Event-ID for resume. IDs keep the Redis <ms>-<seq> form so cursors persisted by meta-sort / meta-share / meta-stremio across the switch still compare. Heartbeats are SSE comments (:keep-alive\n\n) every 30s of silence. If retention has trimmed past a client's cursor, the handler emits one event: gap payload before resuming. See internal/api/sse_events.go and api-mediated-access.md for the full contract.

Debugging

The database path is /meta-core/db/meta-core.db in the image default and on the store app; dev stacks use /meta-core-db/meta-core.db (a named volume — the /meta-core bind is 9p on Windows dev boxes, where SQLite WAL is unsafe). Always open it read-only; writes must go through the API.

DB=/meta-core/db/meta-core.db      # dev stack: /meta-core-db/meta-core.db
Q() { docker exec metacore-app sqlite3 -readonly "$DB" "$1"; }

# Roots
Q "SELECT count(*) FROM kv WHERE k >= 'idx:roots/' AND k < 'idx:roots0'"
Q "SELECT substr(k,11) FROM kv WHERE k >= 'idx:roots/' AND k < 'idx:roots0' LIMIT 10"

# Full document for one root
Q "SELECT k,v FROM kv WHERE k >= 'file:01JKR8XW5T4QMV7AHNJ2DEFGHK/' AND k < 'file:01JKR8XW5T4QMV7AHNJ2DEFGHK0'"

# One property
Q "SELECT v FROM kv WHERE k = 'file:01JKR8XW5T4QMV7AHNJ2DEFGHK/fileinfo/duration'"

# Reverse-index lookup (bare CID — no algorithm prefix)
Q "SELECT v FROM kv WHERE k = 'cid:bafkreih...'"                   # → uuid

# The cids key-set
Q "SELECT k FROM kv WHERE k >= 'file:01JKR8XW5T4QMV7AHNJ2DEFGHK/cids/' AND k < 'file:01JKR8XW5T4QMV7AHNJ2DEFGHK/cids0'"

# canonical_cid is NOT stored. Derive it via the API instead:
curl -k https://metacore-dev.localhost:8083/api/meta/<cid> | jq '.canonical_cid, .cids'

# Schema-version sentinel
Q "SELECT v FROM kv WHERE k = 'meta-core:schema-version'"

# Streams (server-side; external consumers should use SSE)
Q "SELECT k,v FROM kv WHERE k >= 'ev:meta:events/' AND k < 'ev:meta:events0' ORDER BY k DESC LIMIT 10"

# HTTP equivalents — preferred
curl -k 'https://metacore-dev.localhost:8083/api/meta/<cid>'      # CID → document
curl -k 'https://metacore-dev.localhost:8083/meta'                # list root IDs
curl -k 'https://metacore-dev.localhost:8083/meta/<uuid>'         # document by UUID
curl -k 'https://metacore-dev.localhost:8083/api/events/meta'     # live SSE
curl -k 'https://metacore-dev.localhost:8083/api/kv/info'         # KV stats

Key implementation files

Component	Path
Storage engine (SQLite kv table, writer, group commit)	internal/kv/kv.go
Event streams as kv rows	internal/kv/stream.go
Flat-key record ops	internal/storage/client.go
Search (field index + FTS)	internal/storage/search.go
UUIDv7 minting	internal/storage/uuid.go
CID reverse index + aliases + canonical-CID derivation	internal/storage/cid_resolution.go
Schema-version gate	internal/storage/schema_version.go
meta:events snapshot for 0/0-0	internal/api/sse_events.go (streamSnapshot)
Derived counters (root count, per-field schema)	internal/storage/client.go (onChange), internal/schema/indexer.go
Watcher → file:events dispatcher	internal/watcher/dispatcher.go
SSE event mirror	internal/api/sse_events.go
Field semantics (authoritative registry)	/METADATA_KEYS.md