298 lines
13 KiB
Markdown
298 lines
13 KiB
Markdown
# Redis Metadata Storage Structure
|
|
|
|
How meta-core lays out file metadata in Redis. This document covers the
|
|
on-disk shape (UUID-rooted flat string keys, CID reverse index,
|
|
per-root sets) and the schema-version gate. It does **not** enumerate
|
|
field semantics — that's the job of the repo-root
|
|
[`METADATA_KEYS.md`](../../../METADATA_KEYS.md), the authoritative key
|
|
registry for everyone who writes to or reads from metadata.
|
|
|
|
For the design rationale (why UUIDs instead of content hashes, why a
|
|
reverse index), see [`uuid-rooted-metadata.md`](uuid-rooted-metadata.md).
|
|
For the external-access contract (HTTP-only, SSE for events), see
|
|
[`api-mediated-access.md`](api-mediated-access.md).
|
|
|
|
## Quick summary
|
|
|
|
- **Roots are UUIDv7** encoded as 26-char Crockford Base32 (ULID layout),
|
|
not content hashes. Examples in this doc use the placeholder
|
|
`01JKR8XW5T4QMV7AHNJ2DEFGHK`.
|
|
- **Properties are flat Redis STRING keys**, one per leaf:
|
|
`file:<uuid>/<property/path>` → string value. *Not* Redis Hashes.
|
|
- **CIDs are reverse-index aliases**: `cid:<bareCid>` → `<uuid>`
|
|
(a Redis STRING per known CID; no algorithm prefix — the multicodec
|
|
already carries it).
|
|
- **Per-root sets**: `file:<uuid>/cids/<bareCid>` (a *key-set*: one STRING
|
|
`"true"` per known CID, midhash included), `file:<uuid>/duplicates`
|
|
(alternate paths with the same content), plus `file:__index__` (every UUID).
|
|
There is **no stored `canonical_cid`** — it is derived by rank on read.
|
|
- **A schema-version sentinel** (`meta-core:schema-version`) refuses to
|
|
boot against legacy data.
|
|
|
|
Implementation lives in `internal/storage/client.go`,
|
|
`internal/storage/cid_resolution.go`, `internal/storage/uuid.go`, and
|
|
`internal/storage/schema_version.go`.
|
|
|
|
## Key structure
|
|
|
|
### Roots
|
|
|
|
```
|
|
file:<uuid>/<property/path> → Redis STRING (always)
|
|
```
|
|
|
|
UUIDs are produced by `storage.NewUUID()` (`internal/storage/uuid.go`):
|
|
UUIDv7 → 26-char Crockford Base32 (skips I/L/O/U; ULID-compatible). The
|
|
time-sortable prefix lets you `KEYS file:01JKR8*` to find roots minted
|
|
in a given window, which restored some of the debug-ability lost when
|
|
midhash256 stopped being the root.
|
|
|
|
Properties use `/` as the path separator. Nested objects in source code
|
|
are flattened to flat keys on write and reconstructed on read. Examples:
|
|
|
|
```
|
|
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/filePath = "/files/watch/movies/Inception.2010.mkv"
|
|
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/sizeByte = "15032385536"
|
|
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/mtimeNano = "1729872000000000000"
|
|
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/title = "Inception"
|
|
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/fileinfo/duration = "8878.5"
|
|
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/fileinfo/streamdetails/video/0/codec = "h264"
|
|
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/tmdb/poster = "bafkreih..."
|
|
```
|
|
|
|
### Reverse-index aliases (CIDs)
|
|
|
|
Every CID known for a file is registered as its own Redis STRING, keyed by
|
|
the **bare CID** — no algorithm prefix. A CIDv1 already encodes its hash
|
|
function in the multicodec, so naming it again in the key would be
|
|
redundant (and would let the same digest be indexed under two spellings):
|
|
|
|
```
|
|
cid:<bareCid> → <uuid>
|
|
```
|
|
|
|
Examples:
|
|
|
|
```
|
|
cid:bafkreih5kapbjzq... → 01JKR8XW5T4QMV7AHNJ2DEFGHK
|
|
cid:bagacbabaec7v3... → 01JKR8XW5T4QMV7AHNJ2DEFGHK
|
|
```
|
|
|
|
The reverse index makes `GET /api/meta/<cid>` an O(1) lookup for any
|
|
registered CID, independent of which hash algorithm — there is no primary.
|
|
The auto-alias hook (`maybeAddAliasFromFieldLocked` in
|
|
`internal/storage/cid_resolution.go`, called from `SetProperty`,
|
|
`SetMetadataFlat` and `MergeMetadataFlat`) registers the alias for any
|
|
field whose name starts with `cids/`, so writers emit key-set members
|
|
without knowing the reverse index exists.
|
|
|
|
> ⚠ The hook matches **only** the `cids/` prefix. A legacy `cid_*` or
|
|
> `midhash256` *named* field written to a root is stored verbatim and is
|
|
> **never indexed** — the record becomes unresolvable by that CID. Writers
|
|
> must collapse such fields to `cids/<cid>` members before the write. See
|
|
> `METADATA_KEYS.md` §14.13.
|
|
|
|
### Per-root sets
|
|
|
|
```
|
|
file:<uuid>/cids/<bareCid> → STRING "true" — one key per known CID (a key-set, not a Redis SET)
|
|
file:<uuid>/duplicates → Redis SET of alternate paths with same content
|
|
```
|
|
|
|
`cids` is a **key-set**: one flat `file:<uuid>/cids/<cid>` STRING per member,
|
|
which keeps it conflict-free to merge across peers (a SET would not be).
|
|
The record's own address (the midhash) is just the `0x1000` member — it has
|
|
no separate field.
|
|
|
|
There is **no stored `canonical_cid`**. The canonical CID is *derived on
|
|
read*: `GetMetadataDocument` (`internal/storage/cid_resolution.go`) ranks the
|
|
key-set's members through `internal/cid/rank.go` (`cid.Better`) and returns
|
|
the winner as the `canonical_cid` field of the JSON response only. Nothing
|
|
reconciles or persists it — an earlier `reconcileCanonicalCIDLocked` did, and
|
|
was removed with the flat `cid_*` keys.
|
|
|
|
### Global index
|
|
|
|
```
|
|
file:__index__ → Redis SET of all UUIDs
|
|
```
|
|
|
|
`internal/storage/client.go::GetAllHashIDs` reads this set to enumerate
|
|
every file without scanning the keyspace.
|
|
|
|
```bash
|
|
SMEMBERS file:__index__ # all UUIDs
|
|
SCARD file:__index__ # total file count
|
|
SISMEMBER file:__index__ <uuid>
|
|
```
|
|
|
|
### Schema-version sentinel
|
|
|
|
```
|
|
meta-core:schema-version → STRING (currently "2")
|
|
```
|
|
|
|
Set at first boot, checked on every subsequent boot
|
|
(`internal/storage/schema_version.go`). If the sentinel is missing but
|
|
`file:__index__` is non-empty, the gate refuses to boot (legacy data).
|
|
If the sentinel is present but doesn't match `SchemaVersion`, the gate
|
|
also refuses to boot. Alpha-stage policy: operators wipe and restart;
|
|
there is no automated migration.
|
|
|
|
## Why flat STRING keys?
|
|
|
|
The store does not use Redis Hashes. The reasons are:
|
|
|
|
1. **Field-level keyspace notifications.** Redis publishes
|
|
`__keyspace@0__:<key>` on every SET/DEL of a STRING key. With Hashes
|
|
you'd only get one notification on the hash itself. meta-core's
|
|
`MetaPublisher` (`internal/events/meta_publisher.go`) consumes these
|
|
notifications and publishes structured events to the `meta:events`
|
|
Redis Stream.
|
|
2. **Selective updates without read-modify-write.** A plugin writing
|
|
`tmdb/poster` doesn't have to read the rest of the document.
|
|
3. **Compatibility with the CID alias hooks.** Field writes can trigger
|
|
alias-maintenance work in `SetProperty` without the plugin caring.
|
|
|
|
## Reads and writes
|
|
|
|
The Go API in `internal/storage/client.go`:
|
|
|
|
| Method | Behaviour |
|
|
|---|---|
|
|
| `GetMetadataFlat(uuid)` | `SCAN` for `file:<uuid>/*`, `MGET` the values, return `map[string]string`. |
|
|
| `SetMetadataFlat(uuid, m)` | Pipeline `SET` per field; `SADD file:__index__`; auto-register an alias for every `cids/<cid>` member. |
|
|
| `MergeMetadataFlat(uuid, m)` | Like `SetMetadataFlat` but doesn't pre-clear; used by `PATCH /meta/{hash}`. |
|
|
| `GetProperty(uuid, field)` | `GET file:<uuid>/<field>`. |
|
|
| `SetProperty(uuid, field, value)` | `SET file:<uuid>/<field>`; auto-register alias if the field is a CID. |
|
|
| `DeleteRoot(uuid)` | `SCAN` + `DEL` for `file:<uuid>/*`, unmap every reverse-index entry. |
|
|
| `Mint(filePath, size, mtimeNano)` | `NewUUID()` + write `filePath/sizeByte/mtimeNano` + add to index. |
|
|
| `AddAlias(uuid, "<bareCid>")` | Set `cid:<bareCid>` → `<uuid>` and the key-set member `file:<uuid>/cids/<bareCid>` = `"true"`. Nothing to reconcile — canonical is derived on read. |
|
|
| `ResolveRoot("<bareCid>")` | `GET cid:<bareCid>` → uuid (returns the input unchanged if it is already a UUID root). |
|
|
| `GetMetadataDocument(uuid)` | Flat fields + `cids` hoisted to an array + `canonical_cid` **derived** by `cid.Better` rank + `duplicates`. Backs `GET /api/meta/{cid}`. |
|
|
|
|
## Reconstruction (flat → nested)
|
|
|
|
When the HTTP API returns a document, it reconstructs the flattened
|
|
storage back into a nested JSON object:
|
|
|
|
```
|
|
flat nested
|
|
───────────────────────────────────── ───────────────────────────
|
|
title = "Inception" title: "Inception"
|
|
fileinfo/duration = "8878.5" fileinfo: { duration: "8878.5",
|
|
fileinfo/streamdetails/ streamdetails: { video: { 0: {
|
|
video/0/codec = "h264" codec: "h264", ... } }, ... } }
|
|
streams/0/language = "eng" streams: [ { language: "eng" }, ... ]
|
|
```
|
|
|
|
Rules:
|
|
|
|
- `/` is the path separator.
|
|
- Numeric path segments (`0`, `1`, ...) become array indices on read.
|
|
- All values are strings on the wire; type interpretation is the
|
|
caller's job (and is documented per-field in `METADATA_KEYS.md`).
|
|
- `null`/`undefined` round-trip as empty strings.
|
|
|
|
## Plugin contribution flow
|
|
|
|
Plugins don't talk to Redis. They write metadata via meta-core's HTTP
|
|
API (`PUT /meta/{hash}/{key...}` or `PATCH /meta/{hash}`). Each plugin
|
|
writes under a prefix:
|
|
|
|
```
|
|
{plugin-id}/<field> single value
|
|
{plugin-id}/<group>/<field> grouped
|
|
{plugin-id}/<group>/<n>/<field> indexed (array)
|
|
```
|
|
|
|
See [`METADATA_KEYS.md`](../../../METADATA_KEYS.md) for the full,
|
|
authoritative list of fields each plugin writes (content identification,
|
|
file stats, video classification, anime detection, subtitle references,
|
|
Jellyfin metadata, FFmpeg stream details, etc.) and the value formats
|
|
(`int-string`, `lang3`, `path-relfiles`, `csv-set`, ...). This doc
|
|
deliberately does not duplicate that registry — it would drift.
|
|
|
|
## Event streams
|
|
|
|
Two Redis Streams are produced inside meta-core:
|
|
|
|
| Stream | Producer | Contents |
|
|
|---|---|---|
|
|
| `file:events` | `internal/watcher` dispatcher | `add` / `change` / `delete` / `rename` / `reset` events for files on the `/files` volume. |
|
|
| `meta:events` | `internal/events.MetaPublisher` | Field-level metadata mutations derived from Redis keyspace notifications. |
|
|
|
|
External consumers do **not** read these streams directly. They subscribe
|
|
via SSE:
|
|
|
|
- `GET /api/events/files` — mirror of `file:events`
|
|
- `GET /api/events/meta` — mirror of `meta:events`
|
|
|
|
The SSE wire contract uses `Last-Event-ID` for resume (the opaque
|
|
`<ms>-<seq>` Redis Stream entry ID). Heartbeats are SSE comments
|
|
(`:keep-alive\n\n`) every 30s of silence. If retention has trimmed past
|
|
a client's cursor, the handler emits one `event: gap` payload before
|
|
resuming. See `internal/api/sse_events.go` and
|
|
[`api-mediated-access.md`](api-mediated-access.md) for the full
|
|
contract.
|
|
|
|
Internally, meta-core uses a Redis consumer group (XREADGROUP / XACK)
|
|
inside `MetaPublisher` to drain keyspace notifications, but this is not
|
|
the external surface — external services use SSE.
|
|
|
|
## Debugging
|
|
|
|
```bash
|
|
# Roots
|
|
docker exec metacore-app redis-cli SMEMBERS 'file:__index__'
|
|
docker exec metacore-app redis-cli SCARD 'file:__index__'
|
|
docker exec metacore-app redis-cli SRANDMEMBER 'file:__index__'
|
|
|
|
# Full document for one root (flat scan)
|
|
docker exec metacore-app redis-cli --scan --pattern 'file:01JKR8XW5T4QMV7AHNJ2DEFGHK/*'
|
|
|
|
# One property
|
|
docker exec metacore-app redis-cli GET 'file:01JKR8XW5T4QMV7AHNJ2DEFGHK/fileinfo/duration'
|
|
|
|
# Confirm keys are STRING (not HASH)
|
|
docker exec metacore-app redis-cli TYPE 'file:01JKR8XW5T4QMV7AHNJ2DEFGHK/filePath'
|
|
# string
|
|
|
|
# Reverse-index lookup (bare CID — no algorithm prefix)
|
|
docker exec metacore-app redis-cli GET 'cid:bafkreih...' # → uuid
|
|
|
|
# The cids key-set (one STRING per member, NOT a Redis SET — SMEMBERS won't work)
|
|
docker exec metacore-app redis-cli --scan --pattern 'file:01JKR8XW5T4QMV7AHNJ2DEFGHK/cids/*'
|
|
|
|
# canonical_cid is NOT stored. Derive it via the API instead:
|
|
curl -k https://metacore-dev.localhost:8083/api/meta/<cid> | jq '.canonical_cid, .cids'
|
|
|
|
# Schema-version sentinel
|
|
docker exec metacore-app redis-cli GET 'meta-core:schema-version'
|
|
|
|
# Streams (server-side; external consumers should use SSE)
|
|
docker exec metacore-app redis-cli XRANGE 'file:events' - + COUNT 10
|
|
docker exec metacore-app redis-cli XRANGE 'meta:events' - + COUNT 10
|
|
|
|
# HTTP equivalents — preferred
|
|
curl -k 'https://metacore-dev.localhost:8083/api/meta/<cid>' # CID → document
|
|
curl -k 'https://metacore-dev.localhost:8083/meta' # list root IDs
|
|
curl -k 'https://metacore-dev.localhost:8083/meta/<uuid>' # document by UUID
|
|
curl -k 'https://metacore-dev.localhost:8083/api/events/meta' # live SSE
|
|
curl -k 'https://metacore-dev.localhost:8083/api/kv/info' # KV stats
|
|
```
|
|
|
|
## Key implementation files
|
|
|
|
| Component | Path |
|
|
|---|---|
|
|
| Redis client + flat-key ops | `internal/storage/client.go` |
|
|
| UUIDv7 minting | `internal/storage/uuid.go` |
|
|
| CID reverse index + aliases + canonical-CID reconciliation | `internal/storage/cid_resolution.go` |
|
|
| Schema-version gate | `internal/storage/schema_version.go` |
|
|
| Dual-root migration (legacy clean-up) | `internal/storage/dual_root_migration.go` |
|
|
| Keyspace-notification → `meta:events` publisher | `internal/events/meta_publisher.go` |
|
|
| Watcher → `file:events` dispatcher | `internal/watcher/dispatcher.go` |
|
|
| SSE event mirror | `internal/api/sse_events.go` |
|
|
| Field semantics (authoritative registry) | [`/METADATA_KEYS.md`](../../../METADATA_KEYS.md) |
|