replace Redis with an embedded SQLite string->string KV store (group commit, in-tx events, SQL search)

docs/metadata-storage-structure.md

@@ -1,6 +1,6 @@
-# Redis Metadata Storage Structure
+# Metadata Storage Structure
 
-How meta-core lays out file metadata in Redis. This document covers the
+How meta-core lays out file metadata in its store. This document covers the
 on-disk shape (UUID-rooted flat string keys, CID reverse index,
 per-root sets) and the schema-version gate. It does **not** enumerate
 field semantics — that's the job of the repo-root
@@ -12,40 +12,80 @@ reverse index), see [`uuid-rooted-metadata.md`](uuid-rooted-metadata.md).
 For the external-access contract (HTTP-only, SSE for events), see
 [`api-mediated-access.md`](api-mediated-access.md).
 
+> **Storage engine (since 2026-10): an embedded SQLite file, not Redis.**
+> Until 2026-10 these keys lived in a Redis server spawned inside the
+> container. They now live in one string→string table of an embedded SQLite
+> database (pure-Go driver `modernc.org/sqlite`, CGO off). The **key names
+> did not change** — only the Redis *types* that some bookkeeping used (SETs,
+> HASHes, Streams) were flattened into plain keys, listed below. Holding
+> everything in RAM had cost ~6.6 GB on watch.nsl.sh's 169k records (Redis
+> plus the API's in-memory search copy) and caused repeated OOM kills.
+
 ## Quick summary
 
+- **One table**: `kv(k TEXT PRIMARY KEY, v TEXT NOT NULL) WITHOUT ROWID`,
+  schema-free — a new metadata field is just a new key, never a migration.
+  File: `/meta-core/db/meta-core.db` (override `META_CORE_DB_PATH`), WAL
+  mode, `synchronous=NORMAL`.
 - **Roots are UUIDv7** encoded as 26-char Crockford Base32 (ULID layout),
   not content hashes. Examples in this doc use the placeholder
   `01JKR8XW5T4QMV7AHNJ2DEFGHK`.
-- **Properties are flat Redis STRING keys**, one per leaf:
-  `file:<uuid>/<property/path>` → string value. *Not* Redis Hashes.
+- **Properties are flat string keys**, one per leaf:
+  `file:<uuid>/<property/path>` → string value. A record is exactly the
+  key range `[file:<uuid>/, file:<uuid>0)` — there is no per-record field
+  list.
 - **CIDs are reverse-index aliases**: `cid:<bareCid>` → `<uuid>`
-  (a Redis STRING per known CID; no algorithm prefix — the multicodec
+  (one key per known CID; no algorithm prefix — the multicodec
   already carries it).
-- **Per-root sets**: `file:<uuid>/cids/<bareCid>` (a *key-set*: one STRING
-  `"true"` per known CID, midhash included), `file:<uuid>/duplicates`
-  (alternate paths with the same content), plus `file:__index__` (every UUID).
+- **Per-root sets**: `file:<uuid>/cids/<bareCid>` (a *key-set*: one key
+  `"true"` per known CID, midhash included), `dup:<uuid>/<path>` (alternate
+  paths with the same content), plus `idx:roots/<uuid>` (every UUID).
   There is **no stored `canonical_cid`** — it is derived by rank on read.
 - **A schema-version sentinel** (`meta-core:schema-version`) refuses to
   boot against legacy data.
 
-Implementation lives in `internal/storage/client.go`,
-`internal/storage/cid_resolution.go`, `internal/storage/uuid.go`, and
-`internal/storage/schema_version.go`.
+Implementation lives in `internal/kv/kv.go` (engine),
+`internal/kv/stream.go` (event streams), `internal/storage/client.go`,
+`internal/storage/search.go`, `internal/storage/cid_resolution.go`,
+`internal/storage/uuid.go`, and `internal/storage/schema_version.go`.
 
 ## Key structure
 
+Everything is one string→string table. The full layout:
+
+| Key | Value | Notes |
+|---|---|---|
+| `file:<rid>/<field>` | field value | a record property — the **only** thing under `file:` |
+| `cid:<cid>` | `<rid>` | reverse index |
+| `meta-core:schema-version` | `"2"` | schema gate |
+| `idx:roots/<rid>` | `""` | set of every root (was the Redis SET `file:__index__`) |
+| `idx:fts/<rid>` | FTS rowid | the record's row in the FTS table |
+| `dup:<rid>/<path>` | `""` | a duplicate path (was the SET `file:<rid>/duplicates`) |
+| `udl:rec:<uid>/<cid>/<key>` | JSON `{record,version,ts,value?}` | User Data Layer entry (was a HASH) |
+| `udl:idx:…<set>\x1f<member>` | `""` | UDL index sets, one key per member |
+| `ev:<stream>/<ms13>-<seq10>` | JSON of the entry's fields | `meta:events` / `file:events` entries (were Redis Streams) |
+| `evhwm:<stream>` | last issued ID | stream high-water mark |
+| `btmagnet:*`, `searchcov:*`, `nzb:*` | — | written by other services via `/api/kv/value`; unchanged |
+
+Keeping bookkeeping **out of `file:`** is load-bearing: the `file:` range is
+exactly the records, so the field index covers only real fields and every
+change under `file:` is a real field change worth publishing on
+`meta:events`. The per-record `file:<rid>/__fields__` SET and the
+`file:__dirty__` set of the Redis era no longer exist. Roots are split from
+the field at the **first** `/`, so CID-rooted and UUID-rooted records both
+parse.
+
 ### Roots
 
 ```
-file:<uuid>/<property/path>   →   Redis STRING (always)
+file:<uuid>/<property/path>   →   string (always)
 ```
 
 UUIDs are produced by `storage.NewUUID()` (`internal/storage/uuid.go`):
 UUIDv7 → 26-char Crockford Base32 (skips I/L/O/U; ULID-compatible). The
-time-sortable prefix lets you `KEYS file:01JKR8*` to find roots minted
-in a given window, which restored some of the debug-ability lost when
-midhash256 stopped being the root.
+time-sortable prefix means roots minted in a given window sort together
+(`k >= 'file:01JKR8' AND k < 'file:01JKR9'`), which restored some of the
+debug-ability lost when midhash256 stopped being the root.
 
 Properties use `/` as the path separator. Nested objects in source code
 are flattened to flat keys on write and reconstructed on read. Examples:
@@ -62,7 +102,7 @@ file:01JKR8XW5T4QMV7AHNJ2DEFGHK/tmdb/poster                       = "bafkreih...
 
 ### Reverse-index aliases (CIDs)
 
-Every CID known for a file is registered as its own Redis STRING, keyed by
+Every CID known for a file is registered as its own key, keyed by
 the **bare CID** — no algorithm prefix. A CIDv1 already encodes its hash
 function in the multicodec, so naming it again in the key would be
 redundant (and would let the same digest be indexed under two spellings):
@@ -78,13 +118,12 @@ cid:bafkreih5kapbjzq...   →   01JKR8XW5T4QMV7AHNJ2DEFGHK
 cid:bagacbabaec7v3...     →   01JKR8XW5T4QMV7AHNJ2DEFGHK
 ```
 
-The reverse index makes `GET /api/meta/<cid>` an O(1) lookup for any
+The reverse index makes `GET /api/meta/<cid>` a primary-key lookup for any
 registered CID, independent of which hash algorithm — there is no primary.
-The auto-alias hook (`maybeAddAliasFromFieldLocked` in
-`internal/storage/cid_resolution.go`, called from `SetProperty`,
-`SetMetadataFlat` and `MergeMetadataFlat`) registers the alias for any
-field whose name starts with `cids/`, so writers emit key-set members
-without knowing the reverse index exists.
+The record write path (`writeFieldsTx` in `internal/storage/client.go`, used
+by `SetProperty`, `SetMetadataFlat` and `MergeMetadataFlat`) registers the
+alias for any field whose name starts with `cids/`, in the same transaction,
+so writers emit key-set members without knowing the reverse index exists.
 
 > ⚠ The hook matches **only** the `cids/` prefix. A legacy `cid_*` or
 > `midhash256` *named* field written to a root is stored verbatim and is
@@ -95,12 +134,13 @@ without knowing the reverse index exists.
 ### Per-root sets
 
 ```
-file:<uuid>/cids/<bareCid>    →   STRING "true" — one key per known CID (a key-set, not a Redis SET)
-file:<uuid>/duplicates        →   Redis SET of alternate paths with same content
+file:<uuid>/cids/<bareCid>    →   "true" — one key per known CID (a key-set)
+dup:<uuid>/<path>             →   ""     — one key per alternate path with same content
 ```
 
-`cids` is a **key-set**: one flat `file:<uuid>/cids/<cid>` STRING per member,
-which keeps it conflict-free to merge across peers (a SET would not be).
+`cids` is a **key-set**: one flat `file:<uuid>/cids/<cid>` key per member,
+which keeps it conflict-free to merge across peers (a single set-valued
+field would not be).
 The record's own address (the midhash) is just the `0x1000` member — it has
 no separate field.
 
@@ -114,61 +154,88 @@ was removed with the flat `cid_*` keys.
 ### Global index
 
 ```
-file:__index__   →   Redis SET of all UUIDs
+idx:roots/<uuid>   →   ""   (one key per root; was the Redis SET file:__index__)
 ```
 
-`internal/storage/client.go::GetAllHashIDs` reads this set to enumerate
-every file without scanning the keyspace.
+`internal/storage/client.go::GetAllHashIDs` range-reads `idx:roots/` to
+enumerate every file without scanning the records.
 
-```bash
-SMEMBERS file:__index__         # all UUIDs
-SCARD    file:__index__         # total file count
-SISMEMBER file:__index__ <uuid>
-```
+### Search structures (derived)
+
+Two derived structures are maintained on top of `kv`, in the same
+transaction as every record write. Both are rebuildable from the kv rows and
+never authoritative (`internal/storage/search.go`):
+
+- **`kv_field`** — a partial expression index
+  `ON kv(lower(substr(k,instr(k,'/')+1)), lower(v)) WHERE k>='file:' AND k<'file;' AND instr(k,'/')>0`.
+  It answers "which records have field F with a value like V" for **any**
+  field, including ones invented tomorrow — the index is over the key itself.
+- **`fts`** — an FTS5 trigram table `fts(rid UNINDEXED, hay)` holding one row
+  per record: its search haystack plus its rid (`idx:fts/<rid>` points at the
+  row). Trigrams make every 3+ character substring indexable.
+
+`POST /api/metadata/search` uses SQL to build a superset of candidates, then
+verifies each with the unchanged Go matchers (`recordMatchesFilter`,
+`matchAllTokens`), so semantics are identical to the old in-memory index.
+Results still come in hashID order and are fresh immediately (no refresh
+lag). This replaced the API's in-memory `searchIndex` (60 s refresh, ~30 KB
+RAM per record).
 
 ### Schema-version sentinel
 
 ```
-meta-core:schema-version   →   STRING (currently "2")
+meta-core:schema-version   →   string (currently "2")
 ```
 
 Set at first boot, checked on every subsequent boot
 (`internal/storage/schema_version.go`). If the sentinel is missing but
-`file:__index__` is non-empty, the gate refuses to boot (legacy data).
+the root index is non-empty, the gate refuses to boot (legacy data).
 If the sentinel is present but doesn't match `SchemaVersion`, the gate
 also refuses to boot. Alpha-stage policy: operators wipe and restart;
 there is no automated migration.
 
-## Why flat STRING keys?
+## Why flat string keys?
 
-The store does not use Redis Hashes. The reasons are:
+The store keeps one flat key per leaf rather than a document or a
+per-record row. The reasons are:
 
-1. **Field-level keyspace notifications.** Redis publishes
-   `__keyspace@0__:<key>` on every SET/DEL of a STRING key. With Hashes
-   you'd only get one notification on the hash itself. meta-core's
-   `MetaPublisher` (`internal/events/meta_publisher.go`) consumes these
-   notifications and publishes structured events to the `meta:events`
-   Redis Stream.
-2. **Selective updates without read-modify-write.** A plugin writing
+1. **Schema-free.** A plugin inventing a field writes a new key; nothing
+   declares it, and the table never needs a migration. The field index
+   covers it automatically.
+2. **Conflict-free merging across peers.** Key-sets (`cids/<cid>`, and the
+   flat fields in general) merge by union of keys — no read-modify-write of
+   a shared value, nothing to reconcile.
+3. **Selective updates without read-modify-write.** A plugin writing
    `tmdb/poster` doesn't have to read the rest of the document.
-3. **Compatibility with the CID alias hooks.** Field writes can trigger
-   alias-maintenance work in `SetProperty` without the plugin caring.
+4. **Compatibility with the CID alias hooks.** Field writes can trigger
+   alias-maintenance work in the write path without the plugin caring.
+
+(Under Redis there was a fourth reason — field-level keyspace notifications
+fed `meta:events`. That is obsolete: events are now written by the storage
+engine itself, see [Event streams](#event-streams).)
 
 ## Reads and writes
 
+All writes go through **one writer goroutine** with group commit: concurrent
+writes share one SQLite transaction, each under its own SAVEPOINT so a
+failure rolls back alone. A record write — fields + root index + CID aliases
++ FTS row + events — is atomic. (Measured ~835 records/s with 16 concurrent
+writers of 40-field records.) Reads use a separate connection pool and see
+the last committed state.
+
 The Go API in `internal/storage/client.go`:
 
 | Method | Behaviour |
 |---|---|
-| `GetMetadataFlat(uuid)` | `SCAN` for `file:<uuid>/*`, `MGET` the values, return `map[string]string`. |
-| `SetMetadataFlat(uuid, m)` | Pipeline `SET` per field; `SADD file:__index__`; auto-register an alias for every `cids/<cid>` member. |
+| `GetMetadataFlat(uuid)` | Range-read `[file:<uuid>/, file:<uuid>0)`, return `map[string]string`. |
+| `SetMetadataFlat(uuid, m)` | Put each field; add `idx:roots/<uuid>`; auto-register an alias for every `cids/<cid>` member; refresh the FTS row. One transaction. |
 | `MergeMetadataFlat(uuid, m)` | Like `SetMetadataFlat` but doesn't pre-clear; used by `PATCH /meta/{hash}`. |
-| `GetProperty(uuid, field)` | `GET file:<uuid>/<field>`. |
-| `SetProperty(uuid, field, value)` | `SET file:<uuid>/<field>`; auto-register alias if the field is a CID. |
-| `DeleteRoot(uuid)` | `SCAN` + `DEL` for `file:<uuid>/*`, unmap every reverse-index entry. |
+| `GetProperty(uuid, field)` | Get `file:<uuid>/<field>`. |
+| `SetProperty(uuid, field, value)` | Put `file:<uuid>/<field>`; auto-register alias if the field is a CID. |
+| `DeleteRoot(uuid)` | Delete the `file:<uuid>/` range, its duplicates, search row and index membership; unmap every reverse-index entry. |
 | `Mint(filePath, size, mtimeNano)` | `NewUUID()` + write `filePath/sizeByte/mtimeNano` + add to index. |
 | `AddAlias(uuid, "<bareCid>")` | Set `cid:<bareCid>` → `<uuid>` and the key-set member `file:<uuid>/cids/<bareCid>` = `"true"`. Nothing to reconcile — canonical is derived on read. |
-| `ResolveRoot("<bareCid>")` | `GET cid:<bareCid>` → uuid (returns the input unchanged if it is already a UUID root). |
+| `ResolveRoot("<bareCid>")` | Get `cid:<bareCid>` → uuid (returns the input unchanged if it is already a UUID root). |
 | `GetMetadataDocument(uuid)` | Flat fields + `cids` hoisted to an array + `canonical_cid` **derived** by `cid.Better` rank + `duplicates`. Backs `GET /api/meta/{cid}`. |
 
 ## Reconstruction (flat → nested)
@@ -196,7 +263,7 @@ Rules:
 
 ## Plugin contribution flow
 
-Plugins don't talk to Redis. They write metadata via meta-core's HTTP
+Plugins don't talk to the store directly. They write metadata via meta-core's HTTP
 API (`PUT /meta/{hash}/{key...}` or `PATCH /meta/{hash}`). Each plugin
 writes under a prefix:
 
@@ -215,12 +282,24 @@ deliberately does not duplicate that registry — it would drift.
 
 ## Event streams
 
-Two Redis Streams are produced inside meta-core:
+Two event streams are kept inside the store, as rows
+`ev:<stream>/<ms13>-<seq10>` (zero-padded so key order is ID order) plus a
+high-water mark `evhwm:<stream>` (`internal/kv/stream.go`):
 
 | Stream | Producer | Contents |
 |---|---|---|
 | `file:events` | `internal/watcher` dispatcher | `add` / `change` / `delete` / `rename` / `reset` events for files on the `/files` volume. |
-| `meta:events` | `internal/events.MetaPublisher` | Field-level metadata mutations derived from Redis keyspace notifications. |
+| `meta:events` | the storage engine (`internal/kv`) | `{type: set\|del, key, ts}` for every Put/Delete of a `file:` key, appended **in the same transaction** as the change. |
+
+`del` is emitted only when the key existed; `set` is emitted even for an
+unchanged value (Redis semantics, kept). Retention is ~100k entries,
+trimmed every 10k appends. At boot `MetaPublisher`
+(`internal/events/meta_publisher.go`) still republishes every field into
+`meta:events` (meta-fuse rebuilds its VFS only by replaying from `0-0`),
+batched 5k events per transaction.
+
+The Redis-era path — keyspace notifications drained by `MetaPublisher`
+through a PSUBSCRIBE / consumer-group hop — is gone (2026-10).
 
 External consumers do **not** read these streams directly. They subscribe
 via SSE:
@@ -228,51 +307,50 @@ via SSE:
 - `GET /api/events/files` — mirror of `file:events`
 - `GET /api/events/meta` — mirror of `meta:events`
 
-The SSE wire contract uses `Last-Event-ID` for resume (the opaque
-`<ms>-<seq>` Redis Stream entry ID). Heartbeats are SSE comments
+The SSE wire contract uses `Last-Event-ID` for resume. IDs keep the Redis
+`<ms>-<seq>` form so cursors persisted by meta-sort / meta-share /
+meta-stremio across the switch still compare. Heartbeats are SSE comments
 (`:keep-alive\n\n`) every 30s of silence. If retention has trimmed past
 a client's cursor, the handler emits one `event: gap` payload before
 resuming. See `internal/api/sse_events.go` and
 [`api-mediated-access.md`](api-mediated-access.md) for the full
 contract.
 
-Internally, meta-core uses a Redis consumer group (XREADGROUP / XACK)
-inside `MetaPublisher` to drain keyspace notifications, but this is not
-the external surface — external services use SSE.
-
 ## Debugging
 
-```bash
-# Roots
-docker exec metacore-app redis-cli SMEMBERS 'file:__index__'
-docker exec metacore-app redis-cli SCARD    'file:__index__'
-docker exec metacore-app redis-cli SRANDMEMBER 'file:__index__'
+The database path is `/meta-core/db/meta-core.db` in the image default and
+on the store app; dev stacks use `/meta-core-db/meta-core.db` (a named
+volume — the `/meta-core` bind is 9p on Windows dev boxes, where SQLite WAL
+is unsafe). Always open it **read-only**; writes must go through the API.
 
-# Full document for one root (flat scan)
-docker exec metacore-app redis-cli --scan --pattern 'file:01JKR8XW5T4QMV7AHNJ2DEFGHK/*'
+```bash
+DB=/meta-core/db/meta-core.db      # dev stack: /meta-core-db/meta-core.db
+Q() { docker exec metacore-app sqlite3 -readonly "$DB" "$1"; }
+
+# Roots
+Q "SELECT count(*) FROM kv WHERE k >= 'idx:roots/' AND k < 'idx:roots0'"
+Q "SELECT substr(k,11) FROM kv WHERE k >= 'idx:roots/' AND k < 'idx:roots0' LIMIT 10"
+
+# Full document for one root
+Q "SELECT k,v FROM kv WHERE k >= 'file:01JKR8XW5T4QMV7AHNJ2DEFGHK/' AND k < 'file:01JKR8XW5T4QMV7AHNJ2DEFGHK0'"
 
 # One property
-docker exec metacore-app redis-cli GET 'file:01JKR8XW5T4QMV7AHNJ2DEFGHK/fileinfo/duration'
-
-# Confirm keys are STRING (not HASH)
-docker exec metacore-app redis-cli TYPE 'file:01JKR8XW5T4QMV7AHNJ2DEFGHK/filePath'
-# string
+Q "SELECT v FROM kv WHERE k = 'file:01JKR8XW5T4QMV7AHNJ2DEFGHK/fileinfo/duration'"
 
 # Reverse-index lookup (bare CID — no algorithm prefix)
-docker exec metacore-app redis-cli GET 'cid:bafkreih...'      # → uuid
+Q "SELECT v FROM kv WHERE k = 'cid:bafkreih...'"                   # → uuid
 
-# The cids key-set (one STRING per member, NOT a Redis SET — SMEMBERS won't work)
-docker exec metacore-app redis-cli --scan --pattern 'file:01JKR8XW5T4QMV7AHNJ2DEFGHK/cids/*'
+# The cids key-set
+Q "SELECT k FROM kv WHERE k >= 'file:01JKR8XW5T4QMV7AHNJ2DEFGHK/cids/' AND k < 'file:01JKR8XW5T4QMV7AHNJ2DEFGHK/cids0'"
 
 # canonical_cid is NOT stored. Derive it via the API instead:
 curl -k https://metacore-dev.localhost:8083/api/meta/<cid> | jq '.canonical_cid, .cids'
 
 # Schema-version sentinel
-docker exec metacore-app redis-cli GET 'meta-core:schema-version'
+Q "SELECT v FROM kv WHERE k = 'meta-core:schema-version'"
 
 # Streams (server-side; external consumers should use SSE)
-docker exec metacore-app redis-cli XRANGE 'file:events' - + COUNT 10
-docker exec metacore-app redis-cli XRANGE 'meta:events' - + COUNT 10
+Q "SELECT k,v FROM kv WHERE k >= 'ev:meta:events/' AND k < 'ev:meta:events0' ORDER BY k DESC LIMIT 10"
 
 # HTTP equivalents — preferred
 curl -k 'https://metacore-dev.localhost:8083/api/meta/<cid>'      # CID → document
@@ -286,12 +364,14 @@ curl -k 'https://metacore-dev.localhost:8083/api/kv/info'         # KV stats
 
 | Component | Path |
 |---|---|
-| Redis client + flat-key ops | `internal/storage/client.go` |
+| Storage engine (SQLite kv table, writer, group commit) | `internal/kv/kv.go` |
+| Event streams as kv rows | `internal/kv/stream.go` |
+| Flat-key record ops | `internal/storage/client.go` |
+| Search (field index + FTS) | `internal/storage/search.go` |
 | UUIDv7 minting | `internal/storage/uuid.go` |
-| CID reverse index + aliases + canonical-CID reconciliation | `internal/storage/cid_resolution.go` |
+| CID reverse index + aliases + canonical-CID derivation | `internal/storage/cid_resolution.go` |
 | Schema-version gate | `internal/storage/schema_version.go` |
-| Dual-root migration (legacy clean-up) | `internal/storage/dual_root_migration.go` |
-| Keyspace-notification → `meta:events` publisher | `internal/events/meta_publisher.go` |
+| Boot-time `meta:events` republish | `internal/events/meta_publisher.go` |
 | Watcher → `file:events` dispatcher | `internal/watcher/dispatcher.go` |
 | SSE event mirror | `internal/api/sse_events.go` |
 | Field semantics (authoritative registry) | [`/METADATA_KEYS.md`](../../../METADATA_KEYS.md) |