meta-core/docs/metadata-storage-structure.md

Redis Metadata Storage Structure

How meta-core lays out file metadata in Redis. This document covers the on-disk shape (UUID-rooted flat string keys, CID reverse index, per-root sets) and the schema-version gate. It does not enumerate field semantics — that's the job of the repo-root METADATA_KEYS.md, the authoritative key registry for everyone who writes to or reads from metadata.

For the design rationale (why UUIDs instead of content hashes, why a reverse index), see uuid-rooted-metadata.md. For the external-access contract (HTTP-only, SSE for events), see api-mediated-access.md.

Quick summary

• Roots are UUIDv7 encoded as 26-char Crockford Base32 (ULID layout), not content hashes. Examples in this doc use the placeholder 01JKR8XW5T4QMV7AHNJ2DEFGHK.
• Properties are flat Redis STRING keys, one per leaf: file:<uuid>/<property/path> → string value. Not Redis Hashes.
• CIDs are reverse-index aliases: cid:<bareCid> → <uuid> (a Redis STRING per known CID; no algorithm prefix — the multicodec already carries it).
• Per-root sets: file:<uuid>/cids/<bareCid> (a key-set: one STRING "true" per known CID, midhash included), file:<uuid>/duplicates (alternate paths with the same content), plus file:__index__ (every UUID). There is no stored canonical_cid — it is derived by rank on read.
• A schema-version sentinel (meta-core:schema-version) refuses to boot against legacy data.

Implementation lives in internal/storage/client.go, internal/storage/cid_resolution.go, internal/storage/uuid.go, and internal/storage/schema_version.go.

Key structure

Roots

file:<uuid>/<property/path>   →   Redis STRING (always)

UUIDs are produced by storage.NewUUID() (internal/storage/uuid.go): UUIDv7 → 26-char Crockford Base32 (skips I/L/O/U; ULID-compatible). The time-sortable prefix lets you KEYS file:01JKR8* to find roots minted in a given window, which restored some of the debug-ability lost when midhash256 stopped being the root.

Properties use / as the path separator. Nested objects in source code are flattened to flat keys on write and reconstructed on read. Examples:

file:01JKR8XW5T4QMV7AHNJ2DEFGHK/filePath                          = "/files/watch/movies/Inception.2010.mkv"
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/sizeByte                          = "15032385536"
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/mtimeNano                         = "1729872000000000000"
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/title                             = "Inception"
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/fileinfo/duration                 = "8878.5"
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/fileinfo/streamdetails/video/0/codec = "h264"
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/tmdb/poster                       = "bafkreih..."

Reverse-index aliases (CIDs)

Every CID known for a file is registered as its own Redis STRING, keyed by the bare CID — no algorithm prefix. A CIDv1 already encodes its hash function in the multicodec, so naming it again in the key would be redundant (and would let the same digest be indexed under two spellings):

cid:<bareCid>   →   <uuid>

Examples:

cid:bafkreih5kapbjzq...   →   01JKR8XW5T4QMV7AHNJ2DEFGHK
cid:bagacbabaec7v3...     →   01JKR8XW5T4QMV7AHNJ2DEFGHK

The reverse index makes GET /api/meta/<cid> an O(1) lookup for any registered CID, independent of which hash algorithm — there is no primary. The auto-alias hook (maybeAddAliasFromFieldLocked in internal/storage/cid_resolution.go, called from SetProperty, SetMetadataFlat and MergeMetadataFlat) registers the alias for any field whose name starts with cids/, so writers emit key-set members without knowing the reverse index exists.

⚠ The hook matches only the cids/ prefix. A legacy cid_* or midhash256 named field written to a root is stored verbatim and is never indexed — the record becomes unresolvable by that CID. Writers must collapse such fields to cids/<cid> members before the write. See METADATA_KEYS.md §14.13.

Per-root sets

file:<uuid>/cids/<bareCid>    →   STRING "true" — one key per known CID (a key-set, not a Redis SET)
file:<uuid>/duplicates        →   Redis SET of alternate paths with same content

cids is a key-set: one flat file:<uuid>/cids/<cid> STRING per member, which keeps it conflict-free to merge across peers (a SET would not be). The record's own address (the midhash) is just the 0x1000 member — it has no separate field.

There is no stored canonical_cid. The canonical CID is derived on read: GetMetadataDocument (internal/storage/cid_resolution.go) ranks the key-set's members through internal/cid/rank.go (cid.Better) and returns the winner as the canonical_cid field of the JSON response only. Nothing reconciles or persists it — an earlier reconcileCanonicalCIDLocked did, and was removed with the flat cid_* keys.

Global index

file:__index__   →   Redis SET of all UUIDs

internal/storage/client.go::GetAllHashIDs reads this set to enumerate every file without scanning the keyspace.

SMEMBERS file:__index__         # all UUIDs
SCARD    file:__index__         # total file count
SISMEMBER file:__index__ <uuid>

Schema-version sentinel

meta-core:schema-version   →   STRING (currently "2")

Set at first boot, checked on every subsequent boot (internal/storage/schema_version.go). If the sentinel is missing but file:__index__ is non-empty, the gate refuses to boot (legacy data). If the sentinel is present but doesn't match SchemaVersion, the gate also refuses to boot. Alpha-stage policy: operators wipe and restart; there is no automated migration.

Why flat STRING keys?

The store does not use Redis Hashes. The reasons are:

1. Field-level keyspace notifications. Redis publishes __keyspace@0__:<key> on every SET/DEL of a STRING key. With Hashes you'd only get one notification on the hash itself. meta-core's MetaPublisher (internal/events/meta_publisher.go) consumes these notifications and publishes structured events to the meta:events Redis Stream.
2. Selective updates without read-modify-write. A plugin writing tmdb/poster doesn't have to read the rest of the document.
3. Compatibility with the CID alias hooks. Field writes can trigger alias-maintenance work in SetProperty without the plugin caring.

Reads and writes

The Go API in internal/storage/client.go:

Method	Behaviour
GetMetadataFlat(uuid)	SCAN for file:<uuid>/*, MGET the values, return map[string]string.
SetMetadataFlat(uuid, m)	Pipeline SET per field; SADD file:__index__; auto-register an alias for every cids/<cid> member.
MergeMetadataFlat(uuid, m)	Like SetMetadataFlat but doesn't pre-clear; used by PATCH /meta/{hash}.
GetProperty(uuid, field)	GET file:<uuid>/<field>.
SetProperty(uuid, field, value)	SET file:<uuid>/<field>; auto-register alias if the field is a CID.
DeleteRoot(uuid)	SCAN + DEL for file:<uuid>/*, unmap every reverse-index entry.
Mint(filePath, size, mtimeNano)	NewUUID() + write filePath/sizeByte/mtimeNano + add to index.
AddAlias(uuid, "<bareCid>")	Set cid:<bareCid> → <uuid> and the key-set member file:<uuid>/cids/<bareCid> = "true". Nothing to reconcile — canonical is derived on read.
ResolveRoot("<bareCid>")	GET cid:<bareCid> → uuid (returns the input unchanged if it is already a UUID root).
GetMetadataDocument(uuid)	Flat fields + cids hoisted to an array + canonical_cid derived by cid.Better rank + duplicates. Backs GET /api/meta/{cid}.

Reconstruction (flat → nested)

When the HTTP API returns a document, it reconstructs the flattened storage back into a nested JSON object:

flat                                          nested
─────────────────────────────────────         ───────────────────────────
title                       = "Inception"     title: "Inception"
fileinfo/duration           = "8878.5"        fileinfo: { duration: "8878.5",
fileinfo/streamdetails/                                  streamdetails: { video: { 0: {
  video/0/codec             = "h264"                       codec: "h264", ... } }, ... } }
streams/0/language          = "eng"           streams: [ { language: "eng" }, ... ]

Rules:

• / is the path separator.
• Numeric path segments (0, 1, ...) become array indices on read.
• All values are strings on the wire; type interpretation is the caller's job (and is documented per-field in METADATA_KEYS.md).
• null/undefined round-trip as empty strings.

Plugin contribution flow

Plugins don't talk to Redis. They write metadata via meta-core's HTTP API (PUT /meta/{hash}/{key...} or PATCH /meta/{hash}). Each plugin writes under a prefix:

{plugin-id}/<field>                 single value
{plugin-id}/<group>/<field>         grouped
{plugin-id}/<group>/<n>/<field>     indexed (array)

See METADATA_KEYS.md for the full, authoritative list of fields each plugin writes (content identification, file stats, video classification, anime detection, subtitle references, Jellyfin metadata, FFmpeg stream details, etc.) and the value formats (int-string, lang3, path-relfiles, csv-set, ...). This doc deliberately does not duplicate that registry — it would drift.

Event streams

Two Redis Streams are produced inside meta-core:

Stream	Producer	Contents
file:events	internal/watcher dispatcher	add / change / delete / rename / reset events for files on the /files volume.
meta:events	internal/events.MetaPublisher	Field-level metadata mutations derived from Redis keyspace notifications.

External consumers do not read these streams directly. They subscribe via SSE:

• GET /api/events/files — mirror of file:events
• GET /api/events/meta — mirror of meta:events

The SSE wire contract uses Last-Event-ID for resume (the opaque <ms>-<seq> Redis Stream entry ID). Heartbeats are SSE comments (:keep-alive\n\n) every 30s of silence. If retention has trimmed past a client's cursor, the handler emits one event: gap payload before resuming. See internal/api/sse_events.go and api-mediated-access.md for the full contract.

Internally, meta-core uses a Redis consumer group (XREADGROUP / XACK) inside MetaPublisher to drain keyspace notifications, but this is not the external surface — external services use SSE.

Debugging

# Roots
docker exec metacore-app redis-cli SMEMBERS 'file:__index__'
docker exec metacore-app redis-cli SCARD    'file:__index__'
docker exec metacore-app redis-cli SRANDMEMBER 'file:__index__'

# Full document for one root (flat scan)
docker exec metacore-app redis-cli --scan --pattern 'file:01JKR8XW5T4QMV7AHNJ2DEFGHK/*'

# One property
docker exec metacore-app redis-cli GET 'file:01JKR8XW5T4QMV7AHNJ2DEFGHK/fileinfo/duration'

# Confirm keys are STRING (not HASH)
docker exec metacore-app redis-cli TYPE 'file:01JKR8XW5T4QMV7AHNJ2DEFGHK/filePath'
# string

# Reverse-index lookup (bare CID — no algorithm prefix)
docker exec metacore-app redis-cli GET 'cid:bafkreih...'      # → uuid

# The cids key-set (one STRING per member, NOT a Redis SET — SMEMBERS won't work)
docker exec metacore-app redis-cli --scan --pattern 'file:01JKR8XW5T4QMV7AHNJ2DEFGHK/cids/*'

# canonical_cid is NOT stored. Derive it via the API instead:
curl -k https://metacore-dev.localhost:8083/api/meta/<cid> | jq '.canonical_cid, .cids'

# Schema-version sentinel
docker exec metacore-app redis-cli GET 'meta-core:schema-version'

# Streams (server-side; external consumers should use SSE)
docker exec metacore-app redis-cli XRANGE 'file:events' - + COUNT 10
docker exec metacore-app redis-cli XRANGE 'meta:events' - + COUNT 10

# HTTP equivalents — preferred
curl -k 'https://metacore-dev.localhost:8083/api/meta/<cid>'      # CID → document
curl -k 'https://metacore-dev.localhost:8083/meta'                # list root IDs
curl -k 'https://metacore-dev.localhost:8083/meta/<uuid>'         # document by UUID
curl -k 'https://metacore-dev.localhost:8083/api/events/meta'     # live SSE
curl -k 'https://metacore-dev.localhost:8083/api/kv/info'         # KV stats

Key implementation files

Component	Path
Redis client + flat-key ops	internal/storage/client.go
UUIDv7 minting	internal/storage/uuid.go
CID reverse index + aliases + canonical-CID reconciliation	internal/storage/cid_resolution.go
Schema-version gate	internal/storage/schema_version.go
Dual-root migration (legacy clean-up)	internal/storage/dual_root_migration.go
Keyspace-notification → meta:events publisher	internal/events/meta_publisher.go
Watcher → file:events dispatcher	internal/watcher/dispatcher.go
SSE event mirror	internal/api/sse_events.go
Field semantics (authoritative registry)	/METADATA_KEYS.md