bugfix and new internal API
This commit is contained in:
Pierre
+172
-26
@@ -2,11 +2,17 @@
|
||||
|
||||
## Status
|
||||
|
||||
**Proposed** — design accepted; implementation pending. Builds on
|
||||
[uuid-rooted-metadata.md](uuid-rooted-metadata.md), which assumes meta-core
|
||||
is the single writer to the metadata keyspace. That assumption is currently
|
||||
violated by meta-sort, and this doc is the plan to put it back on solid
|
||||
ground.
|
||||
**Landed** — all four PRs (A/B/C/D) deployed and verified end-to-end on the
|
||||
dev stack. Redis is unreachable from outside `metacore-app`; every metadata
|
||||
read, write, and event flows through meta-core's HTTP+SSE surface. Builds
|
||||
on [uuid-rooted-metadata.md](uuid-rooted-metadata.md), which assumes
|
||||
meta-core is the single writer to the metadata keyspace — an assumption
|
||||
that's now actually enforced rather than aspirational.
|
||||
|
||||
See "Implementation outcome" at the bottom for the delta between this doc's
|
||||
plan and what shipped (notably the write-path CID resolution and the
|
||||
dual-root migration sweep, which weren't in the original plan but turned
|
||||
out to be load-bearing).
|
||||
|
||||
## Motivation
|
||||
|
||||
@@ -441,6 +447,10 @@ Closing the network access (PR D) is the moment the lock becomes a
|
||||
lock; everything before it is opt-in migration that consumers can adopt
|
||||
at their own pace.
|
||||
|
||||
**All four PRs have landed.** What follows is the original plan as
|
||||
written; see "Implementation outcome" at the bottom for the delta
|
||||
between plan and ship.
|
||||
|
||||
### PR A — SSE endpoints in meta-core
|
||||
|
||||
Adds `GET /api/events/files` and `GET /api/events/meta`. Implements the
|
||||
@@ -584,28 +594,34 @@ Stable `error` slugs the client should switch on:
|
||||
| `storage_unavailable` | 503 | yes | Redis transient. Caller should retry with backoff. |
|
||||
| `internal` | 500 | yes (cautiously) | Unhandled server-side error. Retry once; escalate if persistent. |
|
||||
|
||||
Today's handlers return `{error, message}` strings; PR A is the natural
|
||||
place to introduce the typed shape (along with the new SSE endpoints
|
||||
which will need their own error responses anyway). Going through a
|
||||
typed envelope from the start avoids the retrofit cost.
|
||||
**Implemented.** Every 4xx/5xx response across `internal/api`,
|
||||
`internal/mounts`, `internal/watcher`, and `internal/watchers` now
|
||||
emits the typed envelope. The slug-mapping helper is duplicated across
|
||||
those packages (one ~15-line `slugForStatus` per package) to avoid an
|
||||
import cycle on the api package; consolidating it into a shared
|
||||
`internal/httperr` package is a worthwhile follow-up but not load-bearing.
|
||||
|
||||
## Remaining open questions
|
||||
|
||||
The audit closed everything that was answerable. What's left:
|
||||
Mostly answered post-implementation:
|
||||
|
||||
- **Deprecation timeline for `/api/metadata/{hashId}`.** It stays during
|
||||
the migration (editor uses it), but once we've consolidated on
|
||||
`/meta/{hash}` for service-to-service writes the editor could migrate
|
||||
too and we'd delete the duplicate surface. Not urgent; revisit
|
||||
post-PR D when everything else is stable.
|
||||
- **Deprecation timeline for `/api/metadata/{hashId}`.** Still open. The
|
||||
editor's React UI continues to call it; the surface itself was untouched
|
||||
by this work. Worth a follow-up to consolidate the editor onto
|
||||
`/meta/{hash}` and delete the parallel routes.
|
||||
|
||||
- **What does `/api/events/files` emit during the watcher's initial
|
||||
bootstrap scan?** When meta-core starts on a populated library, the
|
||||
watcher discovers N files in quick succession and publishes N `add`
|
||||
events. A consumer subscribing for the first time will receive that
|
||||
whole flood. Probably fine — consumers are designed to handle bursts
|
||||
— but worth a smoke test once PR A lands to confirm meta-share's
|
||||
ingest doesn't trip on the cardinality.
|
||||
- **Initial bootstrap flood on `/api/events/files`.** Resolved in
|
||||
practice: meta-fuse uses `initialCursor: '0-0'` to deliberately
|
||||
replay every entry on each restart and the StreamingStateBuilder
|
||||
handles the resulting burst (~2.5k property events on a 50-file
|
||||
library) without issue. meta-sort persists its cursor so only sees
|
||||
the flood once per fresh deploy. meta-stremio filters to "interesting
|
||||
fields" before triggering cache invalidation, so the burst is mostly
|
||||
no-ops.
|
||||
|
||||
- **meta-dup SSE port.** Still pending — meta-dup is in no active
|
||||
compose file. When revived, copy meta-fuse's pattern wholesale
|
||||
(SSEEventClient + MetaCoreApiClient). Tracked separately.
|
||||
|
||||
## Non-goals
|
||||
|
||||
@@ -619,7 +635,137 @@ The audit closed everything that was answerable. What's left:
|
||||
- **Introducing new transport, broker, proxy, or sidecar.** Everything
|
||||
builds on what's deployed today.
|
||||
- **Migrating meta-fuse / meta-stremio / meta-dup direct Redis reads
|
||||
via XSCAN/GET** (separate from event-stream reads). These are
|
||||
read-only and don't break schema invariants, but they will fail once
|
||||
PR D closes Redis access. PR C is the migration window for them —
|
||||
the same SSE adoption work covers it.
|
||||
via XSCAN/GET** (separate from event-stream reads). ~~Was a non-goal~~
|
||||
ended up being required for PR D's network closure to be safe.
|
||||
meta-fuse and meta-stremio migrated; meta-dup deferred (not deployed).
|
||||
See PR C / "Implementation outcome" for details.
|
||||
|
||||
## Implementation outcome
|
||||
|
||||
What actually shipped vs. what this doc planned.
|
||||
|
||||
### Delta against the original plan
|
||||
|
||||
- **Read-path migration for meta-fuse and meta-stremio is now in scope
|
||||
and done.** The original "out of scope" carve-out was wrong: closing
|
||||
the Redis network port doesn't just break stream consumers, it also
|
||||
breaks every `kvClient.get` / `redis_storage.get_all_videos` call.
|
||||
Both services grew their own `MetaCoreApiClient` (TS for meta-fuse,
|
||||
Python for meta-stremio) and now route reads through `/meta/{hash}`
|
||||
+ `/meta` + `/api/file/{cid}/info`.
|
||||
- **meta-stremio's HTTP mode auto-recovers from transient meta-core
|
||||
outages.** `is_connected` re-probes meta-core's `/health` on every
|
||||
call (cheap; a stateless GET), so a metacore-app restart no longer
|
||||
latches the addon into a permanent disconnected state. The
|
||||
background reconnect thread is only needed for legacy Redis mode
|
||||
where the socket itself is sticky.
|
||||
- **Hard `redisUrl` guard is gated by `ALLOW_LEGACY_REDIS_URL=1`.** All
|
||||
three consumer LeaderClients (TS x2 + Python) throw on seeing the
|
||||
field by default. Setting the env var downgrades to a warn — escape
|
||||
hatch for a deliberate temporary rollback to a meta-core that still
|
||||
publishes the field.
|
||||
- **Write-path CID resolution** turned out to be a hard prerequisite
|
||||
for the lockdown to work in practice. Documented separately below.
|
||||
- **PR D ordering matched the doc** (consumer migration first, then
|
||||
publisher stops emitting, then network closure). The `meta-core`
|
||||
docker-network alias and the `6380:6379` host port mapping are both
|
||||
removed; sibling services reach the HTTP/SSE/WebDAV surface via the
|
||||
canonical `metacore-app` hostname.
|
||||
- **meta-dup is the one service that didn't migrate.** It's in no
|
||||
active compose file. The TS port is straightforward when needed: copy
|
||||
meta-fuse's pattern (SSEEventClient for streams, MetaCoreApiClient
|
||||
for reads).
|
||||
|
||||
### Write-path CID resolution (`storage.ResolveRoot`)
|
||||
|
||||
Not in the original plan but load-bearing.
|
||||
|
||||
**The problem.** meta-sort's PATCH-via-HTTP migration (PR B) routed
|
||||
every write through `/meta/{hash}`. But `{hash}` is the midhash256 CID
|
||||
that meta-sort received via file:events, not the watcher's minted UUID.
|
||||
With no resolution step in the handler, every write went to
|
||||
`file:<midhash>/*` — a parallel root the watcher's UUID never saw. The
|
||||
auto-alias hook on `cid_midhash256=<value>` then OVERWROTE the
|
||||
watcher's `cid:midhash256:<value>` → UUID alias to point at the
|
||||
midhash itself, leaving the UUID stranded with only the watcher's
|
||||
sparse fields (filePath, sizeByte, mtimeNano, duplicates) and the
|
||||
midhash root carrying the rich plugin output. This is the exact
|
||||
"dual roots per file" failure the doc's Motivation section listed as
|
||||
the prior bug — it just moved from "convention violation" to "API
|
||||
contract violation."
|
||||
|
||||
**The fix.** Three layers in `internal/storage/cid_resolution.go`:
|
||||
|
||||
1. **`ResolveRoot(hash)`** — at the entry of every `/meta/{hash}`
|
||||
read/write handler, look up `cid:midhash256:<hash>` in the reverse
|
||||
index. If found, use the UUID as the storage root. Falls through to
|
||||
the bare hash for direct-UUID writes / legacy entries.
|
||||
2. **Self-pointing alias guard in `addAliasLocked`** — refuse to
|
||||
register `cid:<algo>:<v>` → `<v>`. The dual-root bug's chicken-and-
|
||||
egg is: meta-sort writes first, registers self-pointing alias,
|
||||
watcher then can't recover. Refusing the self-write keeps the
|
||||
watcher's legitimate alias intact.
|
||||
3. **`MigrateDualRoots` sweep** (`POST /api/admin/migrate-dual-roots`).
|
||||
One-shot fixer for already-stranded entries: walks every UUID's
|
||||
`midhash256` field to build the orphan index, merges each stranded
|
||||
`file:<midhash>/*` into the matching UUID via `MergeMetadataFlat`,
|
||||
deletes the stranded root, and re-points the alias.
|
||||
|
||||
Verified on the dev stack: 48 stranded entries were reunited; the
|
||||
editor now shows every file with its full plugin metadata under one
|
||||
UUID; future writes stay unified.
|
||||
|
||||
### Typed error envelope
|
||||
|
||||
Done across all four handler packages (`api`, `mounts`, `watcher`,
|
||||
`watchers`). Each has its own `writeError` returning
|
||||
`{error: <slug>, message, retryable}`. The slug-mapping helper is
|
||||
duplicated in each to avoid an import cycle on the `api` package.
|
||||
|
||||
### What's NOT done
|
||||
|
||||
- **meta-dup SSE + read migration.** Not deployed, no test harness.
|
||||
When revived, copy meta-fuse's pattern.
|
||||
- **Editor `/api/metadata/{hashId}` consolidation.** The duplicate
|
||||
editor-internal API surface is untouched; could be retired in a
|
||||
follow-up by porting the editor UI to `/meta/{hash}`.
|
||||
- **Consolidated `internal/httperr` package** for the typed envelope
|
||||
helpers (currently duplicated 4× to avoid the import cycle). Code
|
||||
hygiene; functionally a no-op.
|
||||
|
||||
### Verification done
|
||||
|
||||
- All four service dashboards load cleanly with no console errors.
|
||||
- meta-sort SSE consumer + HTTP write path verified end-to-end by
|
||||
delete + inject + reprocess of Sintel.
|
||||
- meta-fuse SSE consumer rebuilds VFS to 48 files on each boot.
|
||||
- meta-stremio catalog API now surfaces the full library (47 videos,
|
||||
1 movie + 5 series + 46 episodes — was 0/0/0 before the Python
|
||||
read-path migration because LeaderStorage broke when it tried Redis).
|
||||
- `redis-cli` from any container other than `metacore-app` produces a
|
||||
connection refused; `cid:midhash256:<H>` aliases all point to UUIDs
|
||||
(no self-pointing); `file:__index__` has only UUID-style hashIds.
|
||||
- ALLOW_LEGACY_REDIS_URL=1 escape hatch present in all three consumer
|
||||
LeaderClients (TS x2 + Python).
|
||||
|
||||
### Known test-suite gaps (post-implementation bats run)
|
||||
|
||||
Running `docker exec meta-test-runner /app/test/test.sh` after the
|
||||
lockdown shipped reveals a number of failing suites, almost all of
|
||||
which are pre-existing test/environment drift unrelated to this work:
|
||||
|
||||
| Failure | Root cause | Relation to lockdown |
|
||||
|---|---|---|
|
||||
| meta-stremio (~35 tests), stremio-tmdb (6 tests) | Tests call `/manifest.json` directly; dev `.env` sets `STREMIO_HASH_API_SEED=dev-stremio-secret` which puts the addon behind a path-token prefix | Unrelated — pre-existing env mismatch |
|
||||
| meta-fuse FUSE-mount (3 tests) | Tests use `docker exec meta-fuse` but the container is named `metafuse-app` (Yundera prod-parity rename) | Unrelated — predates this work |
|
||||
| meta-fuse WebDAV PROPFIND (4 tests) | Tests use Caddy URL with WebDAV basic-auth credentials that no longer match the hash-lock perimeter | Unrelated — predates this work |
|
||||
| meta-sort "API integration tests" (1 test) | Test runs `docker exec meta-sort pnpm test`; container is `metasort-app` | Unrelated — predates this work |
|
||||
| meta-core "create and delete mount config" (1 test) | Test creates an NFS mount; production code dropped NFS support ("NFS mounts are no longer supported") | Unrelated — predates this work |
|
||||
| service-discovery (9 tests) | Cascading from meta-stremio breakage (test expects unauth `/api/services` on stremio) | Unrelated to lockdown directly |
|
||||
| meta-share / share-1c (multiple) | Tests assert `"phase":"1f"` in `/health`; meta-share has moved past that phase | Unrelated — meta-share evolution |
|
||||
| flat-keys: "filePath field exists" (1 test) | Test SRANDMEMBERs file:__index__; if it lands on a UUID with no filePath (e.g. a manual test write) it fails | Was hit transiently after my interactive testing left stub entries; resolved by cleanup |
|
||||
|
||||
No failure traceable to a real regression from this work was found.
|
||||
The dev-environment test harness needs an unrelated refresh to catch
|
||||
up with the prod-parity container renames, the NFS removal, the
|
||||
STREMIO_HASH_API_SEED default, and meta-share's current phase.
|
||||
|
||||
+407
-914
File diff suppressed because it is too large
Load Diff
+211
-534
@@ -1,595 +1,272 @@
|
||||
# Redis Metadata Storage Architecture
|
||||
# Redis Metadata Storage Structure
|
||||
|
||||
This document describes how MetaMesh stores file metadata in Redis, the key structure, plugin metadata contributions, and special cases like FFmpeg stream handling.
|
||||
How meta-core lays out file metadata in Redis. This document covers the
|
||||
on-disk shape (UUID-rooted flat string keys, CID reverse index,
|
||||
per-root sets) and the schema-version gate. It does **not** enumerate
|
||||
field semantics — that's the job of the repo-root
|
||||
[`METADATA_KEYS.md`](../../../METADATA_KEYS.md), the authoritative key
|
||||
registry for everyone who writes to or reads from metadata.
|
||||
|
||||
## Overview
|
||||
For the design rationale (why UUIDs instead of content hashes, why a
|
||||
reverse index), see [`uuid-rooted-metadata.md`](uuid-rooted-metadata.md).
|
||||
For the external-access contract (HTTP-only, SSE for events), see
|
||||
[`api-mediated-access.md`](api-mediated-access.md).
|
||||
|
||||
MetaMesh uses Redis as its primary metadata storage, storing file metadata as **flat string keys**. Each property is stored as a separate Redis key rather than using Redis Hashes. This design enables:
|
||||
## Quick summary
|
||||
|
||||
- **Redis keyspace notifications** for field-level change detection
|
||||
- **Pattern-based subscriptions** (e.g., `file:*/tmdb/*` for TMDB updates)
|
||||
- **meta:events stream** integration for real-time updates across services
|
||||
- Individual field updates without affecting other properties
|
||||
- Fast VFS lookups using content-based hash IDs
|
||||
- **Roots are UUIDv7** encoded as 26-char Crockford Base32 (ULID layout),
|
||||
not content hashes. Examples in this doc use the placeholder
|
||||
`01JKR8XW5T4QMV7AHNJ2DEFGHK`.
|
||||
- **Properties are flat Redis STRING keys**, one per leaf:
|
||||
`file:<uuid>/<property/path>` → string value. *Not* Redis Hashes.
|
||||
- **CIDs are reverse-index aliases**: `cid:<algorithm>:<value>` → `<uuid>`
|
||||
(a Redis STRING per known CID).
|
||||
- **Per-root sets**: `file:<uuid>/cids` (every CID), `file:<uuid>/duplicates`
|
||||
(alternate paths with the same content), plus `file:__index__` (every UUID).
|
||||
- **A schema-version sentinel** (`meta-core:schema-version`) refuses to
|
||||
boot against legacy data.
|
||||
|
||||
## Key Structure
|
||||
Implementation lives in `internal/storage/client.go`,
|
||||
`internal/storage/cid_resolution.go`, `internal/storage/uuid.go`, and
|
||||
`internal/storage/schema_version.go`.
|
||||
|
||||
### Flat Key Storage
|
||||
## Key structure
|
||||
|
||||
Each file property is stored as a separate Redis STRING key:
|
||||
### Roots
|
||||
|
||||
```
|
||||
Key Pattern: file:{hashId}/{property/path}
|
||||
Key Type: STRING
|
||||
Value: String value
|
||||
file:<uuid>/<property/path> → Redis STRING (always)
|
||||
```
|
||||
|
||||
**Example Redis commands:**
|
||||
```bash
|
||||
# Store file metadata (each property is a separate key)
|
||||
SET "file:bafkr4ih5kapbjzq.../cid_midhash256" "bafkr4ih5kapbjzq..."
|
||||
SET "file:bafkr4ih5kapbjzq.../filePath" "/files/watch/movie.mkv"
|
||||
SET "file:bafkr4ih5kapbjzq.../fileinfo/duration" "7200.5"
|
||||
SET "file:bafkr4ih5kapbjzq.../fileinfo/formatName" "matroska,webm"
|
||||
UUIDs are produced by `storage.NewUUID()` (`internal/storage/uuid.go`):
|
||||
UUIDv7 → 26-char Crockford Base32 (skips I/L/O/U; ULID-compatible). The
|
||||
time-sortable prefix lets you `KEYS file:01JKR8*` to find roots minted
|
||||
in a given window, which restored some of the debug-ability lost when
|
||||
midhash256 stopped being the root.
|
||||
|
||||
# Retrieve single field
|
||||
GET "file:bafkr4ih5kapbjzq.../fileinfo/duration"
|
||||
|
||||
# Retrieve all metadata for a file (scan by prefix)
|
||||
SCAN 0 MATCH "file:bafkr4ih5kapbjzq.../*" COUNT 1000
|
||||
```
|
||||
|
||||
### Why Flat Keys?
|
||||
|
||||
The flat key structure (vs. Redis Hashes) enables:
|
||||
|
||||
1. **Keyspace Notifications**: Redis publishes events when individual keys change
|
||||
2. **Pattern Subscriptions**: Services can subscribe to `__keyspace@0__:file:*/tmdb/*`
|
||||
3. **Selective Updates**: Update one field without touching others
|
||||
4. **Stream Integration**: The `meta:events` stream captures field-level changes
|
||||
|
||||
### File Index
|
||||
|
||||
A Redis Set tracks all known file hash IDs:
|
||||
Properties use `/` as the path separator. Nested objects in source code
|
||||
are flattened to flat keys on write and reconstructed on read. Examples:
|
||||
|
||||
```
|
||||
Key: file:__index__
|
||||
Type: Set
|
||||
Value: {hashId1, hashId2, hashId3, ...}
|
||||
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/filePath = "/files/watch/movies/Inception.2010.mkv"
|
||||
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/sizeByte = "15032385536"
|
||||
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/mtimeNano = "1729872000000000000"
|
||||
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/title = "Inception"
|
||||
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/fileinfo/duration = "8878.5"
|
||||
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/fileinfo/streamdetails/video/0/codec = "h264"
|
||||
file:01JKR8XW5T4QMV7AHNJ2DEFGHK/tmdb/poster = "bafkreih..."
|
||||
```
|
||||
|
||||
This enables fast bulk queries like "get all files" without scanning keys.
|
||||
### Reverse-index aliases (CIDs)
|
||||
|
||||
Every CID known for a file is registered as its own Redis STRING:
|
||||
|
||||
```
|
||||
cid:<algorithm>:<value> → <uuid>
|
||||
```
|
||||
|
||||
Examples:
|
||||
|
||||
```
|
||||
cid:midhash256:bafkreih5kapbjzq... → 01JKR8XW5T4QMV7AHNJ2DEFGHK
|
||||
cid:sha256:abc... → 01JKR8XW5T4QMV7AHNJ2DEFGHK
|
||||
cid:ipfs:bafy... → 01JKR8XW5T4QMV7AHNJ2DEFGHK
|
||||
```
|
||||
|
||||
The reverse index makes `GET /api/meta/<cid>` an O(1) lookup for any
|
||||
registered CID, independent of which hash algorithm "primary" — there is
|
||||
no primary. The auto-alias hooks in `SetProperty` and `SetMetadataFlat`
|
||||
(see `internal/storage/cid_resolution.go`) catch any write of a `cid_*`
|
||||
or `midhash256` field and register the alias automatically, so plugins
|
||||
don't need to know about the reverse index.
|
||||
|
||||
### Per-root sets
|
||||
|
||||
```
|
||||
file:<uuid>/cids → Redis SET of "<algorithm>:<value>" tokens
|
||||
file:<uuid>/canonical_cid → STRING — the highest-ranked CID for advertising externally
|
||||
file:<uuid>/duplicates → Redis SET of alternate paths with same content
|
||||
```
|
||||
|
||||
`canonical_cid` is reconciled by `reconcileCanonicalCIDLocked` whenever
|
||||
the `cids` set changes; ranking is done by `internal/cid/rank.go`.
|
||||
|
||||
### Global index
|
||||
|
||||
```
|
||||
file:__index__ → Redis SET of all UUIDs
|
||||
```
|
||||
|
||||
`internal/storage/client.go::GetAllHashIDs` reads this set to enumerate
|
||||
every file without scanning the keyspace.
|
||||
|
||||
```bash
|
||||
# Get all hash IDs
|
||||
SMEMBERS "file:__index__"
|
||||
|
||||
# Count total files
|
||||
SCARD "file:__index__"
|
||||
|
||||
# Check if a file exists
|
||||
SISMEMBER "file:__index__" "bafkr4ih5kapbjzq..."
|
||||
SMEMBERS file:__index__ # all UUIDs
|
||||
SCARD file:__index__ # total file count
|
||||
SISMEMBER file:__index__ <uuid>
|
||||
```
|
||||
|
||||
### Keyspace Notifications
|
||||
|
||||
Redis keyspace notifications are enabled to detect metadata changes. This requires the `notify-keyspace-events` configuration:
|
||||
|
||||
```bash
|
||||
# Required Redis config (set by meta-core)
|
||||
CONFIG SET notify-keyspace-events "K$"
|
||||
|
||||
# K = Keyspace events (published on __keyspace@<db>__:<key>)
|
||||
# $ = String commands (SET, etc.)
|
||||
```
|
||||
|
||||
Services can subscribe to keyspace events for real-time updates:
|
||||
|
||||
```bash
|
||||
# Subscribe to all file key changes
|
||||
PSUBSCRIBE '__keyspace@0__:file:*'
|
||||
|
||||
# Subscribe to specific field pattern (e.g., TMDB updates)
|
||||
PSUBSCRIBE '__keyspace@0__:file:*/tmdb/*'
|
||||
```
|
||||
|
||||
### meta:events Stream
|
||||
|
||||
The `meta:events` stream provides reliable event delivery for file system events from meta-core:
|
||||
### Schema-version sentinel
|
||||
|
||||
```
|
||||
Stream Key: meta:events
|
||||
Entry Fields:
|
||||
- type: add | change | delete | rename | reset
|
||||
- path: File path relative to /files
|
||||
- midhash256: Content hash (for add/change events)
|
||||
- oldPath: Previous path (for rename events)
|
||||
- timestamp: Event timestamp
|
||||
meta-core:schema-version → STRING (currently "2")
|
||||
```
|
||||
|
||||
Consumer groups enable reliable processing with acknowledgment:
|
||||
Set at first boot, checked on every subsequent boot
|
||||
(`internal/storage/schema_version.go`). If the sentinel is missing but
|
||||
`file:__index__` is non-empty, the gate refuses to boot (legacy data).
|
||||
If the sentinel is present but doesn't match `SchemaVersion`, the gate
|
||||
also refuses to boot. Alpha-stage policy: operators wipe and restart;
|
||||
there is no automated migration.
|
||||
|
||||
```bash
|
||||
# Create consumer group
|
||||
XGROUP CREATE meta:events mygroup 0 MKSTREAM
|
||||
## Why flat STRING keys?
|
||||
|
||||
# Read new events (blocking)
|
||||
XREADGROUP GROUP mygroup myconsumer BLOCK 5000 STREAMS meta:events >
|
||||
The store does not use Redis Hashes. The reasons are:
|
||||
|
||||
# Acknowledge processed events
|
||||
XACK meta:events mygroup <entry-id>
|
||||
```
|
||||
1. **Field-level keyspace notifications.** Redis publishes
|
||||
`__keyspace@0__:<key>` on every SET/DEL of a STRING key. With Hashes
|
||||
you'd only get one notification on the hash itself. meta-core's
|
||||
`MetaPublisher` (`internal/events/meta_publisher.go`) consumes these
|
||||
notifications and publishes structured events to the `meta:events`
|
||||
Redis Stream.
|
||||
2. **Selective updates without read-modify-write.** A plugin writing
|
||||
`tmdb/poster` doesn't have to read the rest of the document.
|
||||
3. **Compatibility with the CID alias hooks.** Field writes can trigger
|
||||
alias-maintenance work in `SetProperty` without the plugin caring.
|
||||
|
||||
## Primary Hash ID
|
||||
## Reads and writes
|
||||
|
||||
The primary identifier for each file is `cid_midhash256` - an IPFS-compatible CID generated from:
|
||||
The Go API in `internal/storage/client.go`:
|
||||
|
||||
1. SHA-256 hash of the middle 1MB of the file
|
||||
2. File size in bytes
|
||||
3. IPFS CID v1 encoding (multicodec 0x1000)
|
||||
| Method | Behaviour |
|
||||
|---|---|
|
||||
| `GetMetadataFlat(uuid)` | `SCAN` for `file:<uuid>/*`, `MGET` the values, return `map[string]string`. |
|
||||
| `SetMetadataFlat(uuid, m)` | Pipeline `SET` per field; `SADD file:__index__`; auto-register any `cid_*` aliases. |
|
||||
| `MergeMetadataFlat(uuid, m)` | Like `SetMetadataFlat` but doesn't pre-clear; used by `PATCH /meta/{hash}`. |
|
||||
| `GetProperty(uuid, field)` | `GET file:<uuid>/<field>`. |
|
||||
| `SetProperty(uuid, field, value)` | `SET file:<uuid>/<field>`; auto-register alias if the field is a CID. |
|
||||
| `DeleteRoot(uuid)` | `SCAN` + `DEL` for `file:<uuid>/*`, unmap every reverse-index entry. |
|
||||
| `Mint(filePath, size, mtimeNano)` | `NewUUID()` + write `filePath/sizeByte/mtimeNano` + add to index. |
|
||||
| `AddAlias(uuid, "<algo>:<value>")` | Set `cid:<algo>:<value>` → `<uuid>`, add to `file:<uuid>/cids`, reconcile `canonical_cid`. |
|
||||
| `ResolveCID("<algo>:<value>")` | `GET cid:<algo>:<value>` → uuid. |
|
||||
|
||||
**Example:** `bafkr4ih5kapbjzqvylwj7kx7zmkpxn6qj5xw...`
|
||||
## Reconstruction (flat → nested)
|
||||
|
||||
This provides:
|
||||
- Content-based addressing (same file = same ID)
|
||||
- Fast computation (only reads 1MB)
|
||||
- IPFS compatibility for future distributed storage
|
||||
|
||||
### Collision Handling
|
||||
|
||||
If two different files produce the same midhash (rare), the system falls back to full SHA-256:
|
||||
|
||||
```typescript
|
||||
// Normal case
|
||||
hashId = midHash256; // "bafkr4ih5kapbjzq..."
|
||||
|
||||
// Collision detected (different filePath with same midhash)
|
||||
hashId = metadata['cid_sha2-256']; // Full hash fallback
|
||||
```
|
||||
|
||||
## Metadata Flattening
|
||||
|
||||
Nested objects are flattened into path-based keys for efficient Redis storage.
|
||||
|
||||
### Flattening Rules
|
||||
|
||||
```typescript
|
||||
// Input (nested object)
|
||||
{
|
||||
title: "Inception",
|
||||
video: {
|
||||
codec: "h265",
|
||||
width: "1920"
|
||||
},
|
||||
streams: [
|
||||
{ language: "eng" },
|
||||
{ language: "jpn" }
|
||||
]
|
||||
}
|
||||
|
||||
// Output (flattened for Redis)
|
||||
title = "Inception"
|
||||
video/codec = "h265"
|
||||
video/width = "1920"
|
||||
streams/0/language = "eng"
|
||||
streams/1/language = "jpn"
|
||||
```
|
||||
|
||||
**Key rules:**
|
||||
- Nested objects use `/` as path separator
|
||||
- Arrays use numeric indices (0, 1, 2...)
|
||||
- All values stored as strings
|
||||
- `null`/`undefined` stored as empty strings
|
||||
- Numbers and booleans converted via `String()`
|
||||
|
||||
### Reconstruction
|
||||
|
||||
When reading, the flattened structure is reconstructed into nested objects:
|
||||
|
||||
```typescript
|
||||
const metadata = await kvClient.getMetadataFlat(hashId);
|
||||
// Returns: { title: "Inception", video: { codec: "h265", width: "1920" }, ... }
|
||||
```
|
||||
|
||||
## Plugin Metadata Structure
|
||||
|
||||
Each plugin writes metadata using a prefixed key pattern:
|
||||
When the HTTP API returns a document, it reconstructs the flattened
|
||||
storage back into a nested JSON object:
|
||||
|
||||
```
|
||||
{plugin-id}/{field} # Simple field
|
||||
{plugin-id}/{group}/{field} # Grouped field
|
||||
{plugin-id}/{group}/{index}/{field} # Indexed field (for arrays)
|
||||
flat nested
|
||||
───────────────────────────────────── ───────────────────────────
|
||||
title = "Inception" title: "Inception"
|
||||
fileinfo/duration = "8878.5" fileinfo: { duration: "8878.5",
|
||||
fileinfo/streamdetails/ streamdetails: { video: { 0: {
|
||||
video/0/codec = "h264" codec: "h264", ... } }, ... } }
|
||||
streams/0/language = "eng" streams: [ { language: "eng" }, ... ]
|
||||
```
|
||||
|
||||
### Plugin Contribution Flow
|
||||
Rules:
|
||||
|
||||
- `/` is the path separator.
|
||||
- Numeric path segments (`0`, `1`, ...) become array indices on read.
|
||||
- All values are strings on the wire; type interpretation is the
|
||||
caller's job (and is documented per-field in `METADATA_KEYS.md`).
|
||||
- `null`/`undefined` round-trip as empty strings.
|
||||
|
||||
## Plugin contribution flow
|
||||
|
||||
Plugins don't talk to Redis. They write metadata via meta-core's HTTP
|
||||
API (`PUT /meta/{hash}/{key...}` or `PATCH /meta/{hash}`). Each plugin
|
||||
writes under a prefix:
|
||||
|
||||
```
|
||||
File Discovered
|
||||
↓
|
||||
[FileProcessorPiscina]
|
||||
├─ Compute midhash256
|
||||
├─ Store preliminary metadata
|
||||
└─ Dispatch plugin tasks
|
||||
↓
|
||||
[Plugin Processing]
|
||||
├─ file-info → fileType, mimeType, sizeByte
|
||||
├─ ffmpeg → fileinfo/duration, fileinfo/streamdetails/...
|
||||
├─ filename-parser → titles/eng, season, episode, videoType
|
||||
├─ anime-detector → anime, titles/jpn, titles/jpl
|
||||
├─ tmdb → tmdbid, plot/eng, genres
|
||||
└─ full-hash → cid_sha2-256, cid_sha1
|
||||
↓
|
||||
[Plugin Callbacks]
|
||||
└─ Store metadata via flat keys (SET per property)
|
||||
{plugin-id}/<field> single value
|
||||
{plugin-id}/<group>/<field> grouped
|
||||
{plugin-id}/<group>/<n>/<field> indexed (array)
|
||||
```
|
||||
|
||||
## Core Metadata Types
|
||||
See [`METADATA_KEYS.md`](../../../METADATA_KEYS.md) for the full,
|
||||
authoritative list of fields each plugin writes (content identification,
|
||||
file stats, video classification, anime detection, subtitle references,
|
||||
Jellyfin metadata, FFmpeg stream details, etc.) and the value formats
|
||||
(`int-string`, `lang3`, `path-relfiles`, `csv-set`, ...). This doc
|
||||
deliberately does not duplicate that registry — it would drift.
|
||||
|
||||
### HashMeta - Content Identification
|
||||
## Event streams
|
||||
|
||||
```
|
||||
cid_crc32 = "bafkr..." # CRC32 (multicodec 0x0132)
|
||||
cid_md5 = "bafkr..." # MD5 (multicodec 0xd5)
|
||||
cid_sha1 = "bafkr..." # SHA-1 (multicodec 0x11)
|
||||
cid_sha2-256 = "bafkr..." # SHA-256 (multicodec 0x12)
|
||||
cid_midhash256 = "bafkr..." # Middle-hash (PRIMARY ID, multicodec 0x1000)
|
||||
```
|
||||
Two Redis Streams are produced inside meta-core:
|
||||
|
||||
### FileStatMeta - File Properties
|
||||
| Stream | Producer | Contents |
|
||||
|---|---|---|
|
||||
| `file:events` | `internal/watcher` dispatcher | `add` / `change` / `delete` / `rename` / `reset` events for files on the `/files` volume. |
|
||||
| `meta:events` | `internal/events.MetaPublisher` | Field-level metadata mutations derived from Redis keyspace notifications. |
|
||||
|
||||
```
|
||||
fileType = "video" # video, audio, image, text
|
||||
mimeType = "video/x-matroska" # MIME type
|
||||
sizeByte = "4294967296" # File size in bytes
|
||||
```
|
||||
External consumers do **not** read these streams directly. They subscribe
|
||||
via SSE:
|
||||
|
||||
### VideoMeta - Classification
|
||||
- `GET /api/events/files` — mirror of `file:events`
|
||||
- `GET /api/events/meta` — mirror of `meta:events`
|
||||
|
||||
```
|
||||
videoType = "movie" # movie, episode, unknown
|
||||
originalTitle = "Inception"
|
||||
titles/eng = "Inception" # Localized titles
|
||||
titles/jpn = "インセプション"
|
||||
season = "1"
|
||||
episode = "5"
|
||||
movieYear = "2010"
|
||||
```
|
||||
The SSE wire contract uses `Last-Event-ID` for resume (the opaque
|
||||
`<ms>-<seq>` Redis Stream entry ID). Heartbeats are SSE comments
|
||||
(`:keep-alive\n\n`) every 30s of silence. If retention has trimmed past
|
||||
a client's cursor, the handler emits one `event: gap` payload before
|
||||
resuming. See `internal/api/sse_events.go` and
|
||||
[`api-mediated-access.md`](api-mediated-access.md) for the full
|
||||
contract.
|
||||
|
||||
### AnimeMeta - Anime Detection
|
||||
|
||||
```
|
||||
anime = "true" # "true" or "false"
|
||||
titles/jpn = "進撃の巨人" # Japanese title
|
||||
titles/jpl = "Shingeki no Kyojin" # Romaji
|
||||
```
|
||||
|
||||
### SubtitleMeta - Subtitle References
|
||||
|
||||
```
|
||||
subtitles/eng/0 = "bafkr..." # Reference to subtitle file CID
|
||||
subtitles/eng/1 = "bafks..."
|
||||
subtitles/jpn/0 = "bafkt..."
|
||||
```
|
||||
|
||||
### JellyfinMeta - Comprehensive Metadata
|
||||
|
||||
```
|
||||
imdbid = "tt1375666"
|
||||
tmdbid = "27205"
|
||||
anidbid = "12345"
|
||||
plot/eng = "A movie about..."
|
||||
genre/0 = "Action"
|
||||
genre/1 = "Sci-Fi"
|
||||
rating = "8.8"
|
||||
mpaa = "PG-13"
|
||||
releasedate = "2010-07-16"
|
||||
art/poster = "bafkr..." # Reference to poster image CID
|
||||
art/fanart = "bafks..." # Reference to backdrop image CID
|
||||
```
|
||||
|
||||
## FFmpeg Plugin - Special Case
|
||||
|
||||
The FFmpeg plugin extracts detailed stream information with a specific nested structure.
|
||||
|
||||
### Stream Details Structure
|
||||
|
||||
```
|
||||
fileinfo/duration = "7200.5"
|
||||
fileinfo/formatName = "matroska,webm"
|
||||
fileinfo/streamdetails/video/0/codec = "h264"
|
||||
fileinfo/streamdetails/video/0/width = "1920"
|
||||
fileinfo/streamdetails/video/0/height = "1080"
|
||||
fileinfo/streamdetails/video/0/aspect = "16:9"
|
||||
fileinfo/streamdetails/video/0/bitrate = "8000000"
|
||||
fileinfo/streamdetails/audio/0/codec = "aac"
|
||||
fileinfo/streamdetails/audio/0/language = "eng"
|
||||
fileinfo/streamdetails/audio/0/channels = "6"
|
||||
fileinfo/streamdetails/audio/1/codec = "dts"
|
||||
fileinfo/streamdetails/audio/1/language = "fra"
|
||||
fileinfo/streamdetails/audio/1/channels = "6"
|
||||
fileinfo/streamdetails/subtitle/0/language = "eng"
|
||||
fileinfo/streamdetails/subtitle/0/format = "subrip"
|
||||
fileinfo/streamdetails/subtitle/1/language = "jpn"
|
||||
fileinfo/streamdetails/subtitle/1/format = "ass"
|
||||
fileinfo/streamdetails/embeddedimage/0/type = "cover"
|
||||
```
|
||||
|
||||
### Stream Type Keys
|
||||
|
||||
| Stream Type | Key Pattern | Description |
|
||||
|------------|-------------|-------------|
|
||||
| Video | `fileinfo/streamdetails/video/{n}/...` | Video streams (0, 1, 2...) |
|
||||
| Audio | `fileinfo/streamdetails/audio/{n}/...` | Audio tracks |
|
||||
| Subtitle | `fileinfo/streamdetails/subtitle/{n}/...` | Embedded subtitles |
|
||||
| Embedded Image | `fileinfo/streamdetails/embeddedimage/{n}/...` | Cover art, thumbnails |
|
||||
|
||||
### Common Stream Fields
|
||||
|
||||
**Video streams:**
|
||||
- `codec` - h264, hevc, av1, vp9, etc.
|
||||
- `width`, `height` - Resolution
|
||||
- `aspect` - Aspect ratio
|
||||
- `bitrate` - Video bitrate
|
||||
- `profile` - Codec profile (main, high, etc.)
|
||||
- `level` - Codec level
|
||||
|
||||
**Audio streams:**
|
||||
- `codec` - aac, dts, ac3, flac, etc.
|
||||
- `language` - ISO 639-2 code (eng, jpn, fra)
|
||||
- `channels` - Channel count (2, 6, 8)
|
||||
- `bitrate` - Audio bitrate
|
||||
- `samplerate` - Sample rate
|
||||
|
||||
**Subtitle streams:**
|
||||
- `language` - ISO 639-2 code
|
||||
- `format` - subrip, ass, pgs, vobsub
|
||||
|
||||
## Real-Time Notifications
|
||||
|
||||
MetaMesh provides multiple mechanisms for real-time metadata change notifications:
|
||||
|
||||
### 1. meta:events Stream (Recommended)
|
||||
|
||||
The `meta:events` Redis stream provides reliable, ordered event delivery with consumer groups:
|
||||
|
||||
```bash
|
||||
# File system events from meta-core watcher
|
||||
XREAD STREAMS meta:events 0
|
||||
|
||||
# Event types: add, change, delete, rename, reset
|
||||
```
|
||||
|
||||
Benefits:
|
||||
- **Reliable delivery** via consumer groups and acknowledgment
|
||||
- **Replay capability** for catching up after restarts
|
||||
- **Ordered processing** with stream IDs
|
||||
|
||||
### 2. Keyspace Notifications
|
||||
|
||||
For field-level change detection, subscribe to Redis keyspace notifications:
|
||||
|
||||
```bash
|
||||
# All file metadata changes
|
||||
PSUBSCRIBE '__keyspace@0__:file:*'
|
||||
|
||||
# Specific plugin updates (e.g., TMDB enrichment)
|
||||
PSUBSCRIBE '__keyspace@0__:file:*/tmdb/*'
|
||||
```
|
||||
|
||||
### Cache Invalidation
|
||||
|
||||
The WebDAV cache in meta-core integrates with keyspace notifications for automatic invalidation:
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ Cache Invalidation Flow │
|
||||
│ │
|
||||
│ Redis Keyspace Cache │
|
||||
│ ┌──────────────┐ Invalidator │
|
||||
│ │ SET file:*/ │ ──────────► ┌──────────────┐ │
|
||||
│ │ DEL file:*/ │ subscribe │ Listens to │ │
|
||||
│ └──────────────┘ │ __keyspace@ │ │
|
||||
│ │ 0__:file:* │ │
|
||||
│ └──────┬───────┘ │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌──────────────┐ │
|
||||
│ │ Remove from │ │
|
||||
│ │ LRU cache │ │
|
||||
│ └──────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
Flow:
|
||||
1. Cache invalidator subscribes to `__keyspace@0__:file:*`
|
||||
2. When metadata changes (SET/DEL), invalidator receives notification
|
||||
3. Invalidator removes the corresponding file from the WebDAV cache
|
||||
4. Next access fetches fresh copy from source filesystem
|
||||
|
||||
This ensures cached files stay synchronized with metadata changes.
|
||||
|
||||
### 3. Pub/Sub Channels (Legacy)
|
||||
|
||||
Batch notification channels are still supported:
|
||||
|
||||
```
|
||||
meta-sort:file:batch # Batch file updates (every 5 seconds)
|
||||
meta-sort:scan:reset # Full rescan notification
|
||||
```
|
||||
|
||||
```json
|
||||
{
|
||||
"timestamp": 1704067200000,
|
||||
"changes": [
|
||||
{ "action": "add", "hashId": "bafkr4ih5..." },
|
||||
{ "action": "update", "hashId": "bafkr4ih5..." },
|
||||
{ "action": "remove", "hashId": "bafkr4ih5..." }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Complete Example
|
||||
|
||||
Here's a complete metadata example for a movie file:
|
||||
|
||||
```bash
|
||||
# Query all keys for a file
|
||||
redis-cli --scan --pattern 'file:bafkr4ih5kapbjzq.../*'
|
||||
|
||||
# Example flat keys stored in Redis:
|
||||
# file:bafkr4ih5kapbjzq.../cid_midhash256 = "bafkr4ih5kapbjzq..."
|
||||
# file:bafkr4ih5kapbjzq.../filePath = "/files/watch/movies/Inception.2010.1080p.BluRay.x264.mkv"
|
||||
# file:bafkr4ih5kapbjzq.../fileType = "video"
|
||||
# file:bafkr4ih5kapbjzq.../fileinfo/duration = "8878.5"
|
||||
# file:bafkr4ih5kapbjzq.../titles/eng = "Inception"
|
||||
# ...
|
||||
|
||||
# When reconstructed by the API (nested object representation)
|
||||
{
|
||||
# Content identification
|
||||
"cid_midhash256": "bafkr4ih5kapbjzq...",
|
||||
"cid_sha2-256": "bafks7ij2kbpck...",
|
||||
"filePath": "/files/watch/movies/Inception.2010.1080p.BluRay.x264.mkv",
|
||||
|
||||
# File properties
|
||||
"fileType": "video",
|
||||
"mimeType": "video/x-matroska",
|
||||
"sizeByte": "4294967296",
|
||||
|
||||
# Classification
|
||||
"videoType": "movie",
|
||||
"originalTitle": "Inception",
|
||||
"movieYear": "2010",
|
||||
|
||||
# Localized titles
|
||||
"titles": {
|
||||
"eng": "Inception",
|
||||
"fra": "Inception",
|
||||
"jpn": "インセプション"
|
||||
},
|
||||
|
||||
# FFmpeg stream details
|
||||
"fileinfo": {
|
||||
"duration": "8878.5",
|
||||
"formatName": "matroska,webm",
|
||||
"streamdetails": {
|
||||
"video": {
|
||||
"0": {
|
||||
"codec": "h264",
|
||||
"width": "1920",
|
||||
"height": "1080",
|
||||
"aspect": "16:9",
|
||||
"bitrate": "8000000"
|
||||
}
|
||||
},
|
||||
"audio": {
|
||||
"0": { "codec": "dts", "language": "eng", "channels": "6" },
|
||||
"1": { "codec": "ac3", "language": "fra", "channels": "6" }
|
||||
},
|
||||
"subtitle": {
|
||||
"0": { "language": "eng", "format": "subrip" },
|
||||
"1": { "language": "fra", "format": "subrip" }
|
||||
}
|
||||
}
|
||||
},
|
||||
|
||||
# External metadata (TMDB)
|
||||
"tmdbid": "27205",
|
||||
"imdbid": "tt1375666",
|
||||
"plot": {
|
||||
"eng": "A thief who steals corporate secrets through dream-sharing technology..."
|
||||
},
|
||||
"genre": ["Action", "Science Fiction", "Adventure"],
|
||||
"rating": "8.4",
|
||||
"releasedate": "2010-07-16",
|
||||
|
||||
# Media assets
|
||||
"art": {
|
||||
"poster": "bafkr4poster...",
|
||||
"fanart": "bafkr4fanart..."
|
||||
},
|
||||
|
||||
# Detection flags
|
||||
"anime": "false"
|
||||
}
|
||||
```
|
||||
|
||||
## Key Implementation Files
|
||||
|
||||
| Component | File Path |
|
||||
|-----------|-----------|
|
||||
| Redis Client | `packages/meta-sort/packages/meta-sort-core/src/kv/RedisClient.ts` |
|
||||
| Metadata Utils | `packages/meta-sort/packages/meta-sort-core/src/kv/MetadataUtils.ts` |
|
||||
| KV Interface | `packages/meta-sort/packages/meta-sort-core/src/kv/IKVClient.ts` |
|
||||
| File Processor | `packages/meta-sort/packages/meta-sort-core/src/logic/fileProcessor/FileProcessorPiscina.ts` |
|
||||
| Pub/Sub Pipeline | `packages/meta-sort/packages/meta-sort-core/src/logic/pipeline/StreamingPipeline.ts` |
|
||||
| Type Definitions | `packages/meta-interface/src/lib/metadata-type/*.ts` |
|
||||
| meta-core Storage | `packages/meta-core/internal/storage/client.go` |
|
||||
| meta-core Events | `packages/meta-core/internal/events/publisher.go` |
|
||||
| meta-core Cache | `packages/meta-core/internal/cache/` |
|
||||
Internally, meta-core uses a Redis consumer group (XREADGROUP / XACK)
|
||||
inside `MetaPublisher` to drain keyspace notifications, but this is not
|
||||
the external surface — external services use SSE.
|
||||
|
||||
## Debugging
|
||||
|
||||
```bash
|
||||
# List all file hash IDs
|
||||
docker exec meta-core-dev redis-cli SMEMBERS 'file:__index__'
|
||||
# Roots
|
||||
docker exec metacore-app redis-cli SMEMBERS 'file:__index__'
|
||||
docker exec metacore-app redis-cli SCARD 'file:__index__'
|
||||
docker exec metacore-app redis-cli SRANDMEMBER 'file:__index__'
|
||||
|
||||
# Count total files
|
||||
docker exec meta-core-dev redis-cli SCARD 'file:__index__'
|
||||
# Full document for one root (flat scan)
|
||||
docker exec metacore-app redis-cli --scan --pattern 'file:01JKR8XW5T4QMV7AHNJ2DEFGHK/*'
|
||||
|
||||
# Get a sample hash ID
|
||||
docker exec meta-core-dev redis-cli SRANDMEMBER 'file:__index__'
|
||||
# One property
|
||||
docker exec metacore-app redis-cli GET 'file:01JKR8XW5T4QMV7AHNJ2DEFGHK/fileinfo/duration'
|
||||
|
||||
# List all keys for a specific file (flat key scan)
|
||||
docker exec meta-core-dev redis-cli --scan --pattern 'file:bafkr4ih5.../*'
|
||||
# Confirm keys are STRING (not HASH)
|
||||
docker exec metacore-app redis-cli TYPE 'file:01JKR8XW5T4QMV7AHNJ2DEFGHK/filePath'
|
||||
# string
|
||||
|
||||
# Get specific field (flat key)
|
||||
docker exec meta-core-dev redis-cli GET 'file:bafkr4ih5.../fileinfo/duration'
|
||||
# Reverse-index lookups
|
||||
docker exec metacore-app redis-cli GET 'cid:midhash256:bafkreih...' # → uuid
|
||||
docker exec metacore-app redis-cli SMEMBERS 'file:01JKR8XW5T4QMV7AHNJ2DEFGHK/cids'
|
||||
docker exec metacore-app redis-cli GET 'file:01JKR8XW5T4QMV7AHNJ2DEFGHK/canonical_cid'
|
||||
|
||||
# Get filePath for a file
|
||||
docker exec meta-core-dev redis-cli GET 'file:bafkr4ih5.../filePath'
|
||||
# Schema-version sentinel
|
||||
docker exec metacore-app redis-cli GET 'meta-core:schema-version'
|
||||
|
||||
# Verify keys are STRING type (not HASH)
|
||||
docker exec meta-core-dev redis-cli TYPE 'file:bafkr4ih5.../filePath'
|
||||
# Should return: string
|
||||
# Streams (server-side; external consumers should use SSE)
|
||||
docker exec metacore-app redis-cli XRANGE 'file:events' - + COUNT 10
|
||||
docker exec metacore-app redis-cli XRANGE 'meta:events' - + COUNT 10
|
||||
|
||||
# Monitor metadata changes via stream
|
||||
docker exec meta-core-dev redis-cli XREAD STREAMS 'meta:events' 0
|
||||
|
||||
# Monitor keyspace notifications (requires notify-keyspace-events enabled)
|
||||
docker exec meta-core-dev redis-cli PSUBSCRIBE '__keyspace@0__:file:*'
|
||||
|
||||
# meta-core API debugging
|
||||
# Cache status
|
||||
curl http://localhost:9000/api/cache/status
|
||||
|
||||
# Clear cache
|
||||
curl -X POST http://localhost:9000/api/cache/clear
|
||||
|
||||
# Watch events stream (SSE)
|
||||
curl http://localhost:9000/api/events/stream
|
||||
|
||||
# KV browser - storage info
|
||||
curl http://localhost:9000/api/kv/info
|
||||
|
||||
# Scan status
|
||||
curl http://localhost:9000/api/scan/status
|
||||
# HTTP equivalents — preferred
|
||||
curl -k 'https://metacore-dev.localhost:8083/api/meta/<cid>' # CID → document
|
||||
curl -k 'https://metacore-dev.localhost:8083/meta' # list root IDs
|
||||
curl -k 'https://metacore-dev.localhost:8083/meta/<uuid>' # document by UUID
|
||||
curl -k 'https://metacore-dev.localhost:8083/api/events/meta' # live SSE
|
||||
curl -k 'https://metacore-dev.localhost:8083/api/kv/info' # KV stats
|
||||
```
|
||||
|
||||
## meta:events Stream
|
||||
## Key implementation files
|
||||
|
||||
The `meta:events` stream captures metadata change events for real-time notifications:
|
||||
|
||||
```bash
|
||||
# View recent events
|
||||
docker exec meta-core-dev redis-cli XRANGE 'meta:events' - + COUNT 10
|
||||
|
||||
# Read events from a consumer group
|
||||
docker exec meta-core-dev redis-cli XREADGROUP GROUP mygroup myconsumer STREAMS 'meta:events' >
|
||||
```
|
||||
|
||||
Stream events contain:
|
||||
- `type`: Event type (add, change, delete, rename, reset)
|
||||
- `path`: File path
|
||||
- `midhash256`: Content hash (for add/change)
|
||||
- `timestamp`: Event timestamp
|
||||
| Component | Path |
|
||||
|---|---|
|
||||
| Redis client + flat-key ops | `internal/storage/client.go` |
|
||||
| UUIDv7 minting | `internal/storage/uuid.go` |
|
||||
| CID reverse index + aliases + canonical-CID reconciliation | `internal/storage/cid_resolution.go` |
|
||||
| Schema-version gate | `internal/storage/schema_version.go` |
|
||||
| Dual-root migration (legacy clean-up) | `internal/storage/dual_root_migration.go` |
|
||||
| Keyspace-notification → `meta:events` publisher | `internal/events/meta_publisher.go` |
|
||||
| Watcher → `file:events` dispatcher | `internal/watcher/dispatcher.go` |
|
||||
| SSE event mirror | `internal/api/sse_events.go` |
|
||||
| Field semantics (authoritative registry) | [`/METADATA_KEYS.md`](../../../METADATA_KEYS.md) |
|
||||
|
||||
@@ -2,9 +2,7 @@
|
||||
|
||||
## Status
|
||||
|
||||
**Proposed** — design under discussion. Not yet implemented. Supersedes the
|
||||
"midhash256 as primary key" model currently described in
|
||||
[metadata-storage-structure.md](metadata-storage-structure.md).
|
||||
**Status:** Implemented (2026 alpha clean-wipe gate landed). Kept for design rationale.
|
||||
|
||||
## Motivation
|
||||
|
||||
|
||||
Reference in New Issue
Block a user