replace Redis with an embedded SQLite string->string KV store (group commit, in-tx events, SQL search)

docs/api-mediated-access.md

@@ -9,6 +9,16 @@ on [uuid-rooted-metadata.md](uuid-rooted-metadata.md), which assumes
 meta-core is the single writer to the metadata keyspace — an assumption
 that's now actually enforced rather than aspirational.
 
+> **2026-10: meta-core no longer uses Redis at all.** Its store is now an
+> embedded SQLite file (`/meta-core/db/meta-core.db`, see
+> [metadata-storage-structure.md](metadata-storage-structure.md)) and the
+> `file:events` / `meta:events` streams are rows of that store. The lockdown
+> below is now structural — there is no database port anywhere — and the
+> SSE wire contract was kept byte-for-byte, so no consumer changed. The
+> rest of this doc is the design record of the Redis-era lockdown; the
+> sections that describe *current* behaviour (SSE retention, reconnect,
+> error handling) have been updated in place.
+
 See "Implementation outcome" at the bottom for the delta between this doc's
 plan and what shipped (notably the write-path CID resolution and the
 dual-root migration sweep, which weren't in the original plan but turned
@@ -85,7 +95,7 @@ we don't fully control.
 ```
 
 **The single-writer rule.** meta-core has exclusive write authority over
-the `file:*`, `cid:*`, `file:__index__`, and `meta-core:*` keyspaces. No
+the `file:*`, `cid:*`, `file:__index__` (now `idx:*`), and `meta-core:*` keyspaces. No
 other service can write — not because of an honoured convention, but
 because they can't reach Redis at all.
 
@@ -243,6 +253,25 @@ After PR A, the cursor contract is **"durable across meta-core restart
 and bounded by `MAXLEN ~ N`"** — matching what consumers already assume
 when they persist a Last-Event-ID.
 
+**Current backing store (since 2026-10).** The streams are rows of
+meta-core's SQLite kv table — `ev:<stream>/<ms13>-<seq10>` → JSON of the
+entry's fields, plus a high-water mark `evhwm:<stream>` that seeds the next
+ID (so a `ClearStream` reset never reissues an ID an old cursor treats as
+seen). The contract above holds unchanged:
+
+- Entries persist across restart (they are in the database file).
+- Retention is ~100k entries per stream, trimmed every 10k appends.
+- IDs keep the Redis `<ms>-<seq>` form, so cursors persisted by meta-sort /
+  meta-share / meta-stremio before the switch still compare.
+- `meta:events` entries (`{type: set|del, key, ts}`) are appended by the
+  storage engine **in the same transaction** as the `file:` key change —
+  `del` only when the key existed, `set` even for an unchanged value. The
+  Redis keyspace-notification hop (PSUBSCRIBE in `MetaPublisher`), which
+  could drop events when it lagged, is gone.
+- The boot-time republish of every field into `meta:events` is kept
+  (meta-fuse rebuilds its VFS only by replaying from `0-0`), batched 5k
+  events per transaction.
+
 ### Heartbeats
 
 Server writes an SSE comment line (`:keep-alive\n\n`) every 30 seconds
@@ -254,10 +283,13 @@ per the SSE spec.
 
 Per the SSE spec the client (or `EventSource`) auto-reconnects after a
 network error, sending `Last-Event-ID`. The server resumes from the next
-entry after that ID by translating to Redis Stream `XREAD … STREAMS file:events <id>`.
+entry after that ID (a range read of the stream's rows; under Redis this
+was `XREAD … STREAMS file:events <id>`). With no `Last-Event-ID` the stream
+starts at the head, resolved once at connect; `0` / `0-0` go through the
+gap path and replay from the beginning.
 
-**Gap handling.** If the requested ID has been trimmed out of the Redis
-Stream by retention, the server emits one synthetic event before resuming
+**Gap handling.** If the requested ID is older than the oldest retained
+entry, the server emits one synthetic event before resuming
 from the oldest available entry:
 
 ```
@@ -271,9 +303,10 @@ react. Consumers that don't care ignore it.
 
 ### Error handling
 
-- **Redis unreachable on connect**: 503 with `Retry-After: 5`.
-- **Redis fails mid-stream**: close the SSE connection. Client reconnects
-  with `Last-Event-ID`; the cycle resumes when Redis recovers.
+- **Store unavailable on connect**: 503 with `Retry-After: 5`.
+- **Store read fails mid-stream**: close the SSE connection. Client
+  reconnects with `Last-Event-ID`; the cycle resumes when meta-core is back
+  (a meta-core restart is the common case — SSE clients just reconnect).
 - **Client disconnects**: handler goroutine exits cleanly; no per-client
   state on meta-core.
 
@@ -307,7 +340,8 @@ inside-only until a concrete external consumer appears.
 ### Implementation footprint
 
 The SSE handler itself is ~80 lines of Go in meta-core: one handler per
-stream, each running an `XREAD … BLOCK 30s` loop and writing
+stream, each running a read-after-cursor loop (an `XREAD … BLOCK 30s` loop
+in the Redis era) and writing
 SSE-formatted lines to the response. Go's `net/http` natively supports
 streaming responses with `http.Flusher.Flush()`. Standard SSE clients
 exist for every language we use: Node `eventsource`, Rust
@@ -562,11 +596,11 @@ metacore-app produces a connection refused. The lock is real.
 | Reads from `file:*` direct? | Forbidden by network topology | Same lock, same reason |
 | Event streams direct? | No — mediated through SSE | Kubernetes-style. Costs one HTTP hop; buys schema evolution + threat-model fit |
 | Enforcement mechanism? | Network isolation (primary) + docs + code review (secondary) | No ACLs, no proxy, no lint |
-| Backing store for streams? | Redis Streams (unchanged) | SSE wraps it; could be swapped later without consumer impact |
+| Backing store for streams? | Redis Streams at the time; SQLite kv rows since 2026-10 | SSE wraps it; the 2026-10 swap proved it — no consumer changed |
 | Consumer groups / horizontal scale? | Not supported via SSE | Not needed today; revisit if scale ever requires it |
 | Storage portability? | Not a goal but a free side-effect | Will not be exercised in alpha |
 | **Canonical write API path?** | **`/meta/{hash}` family** | Has GET/PUT/**PATCH**/DELETE plus `/_add/{key}` and per-property `{key:.*}` operations. `PATCH` calls `MergeMetadataFlat` — matches the partial-update semantics PR B needs. `/api/metadata/{hashId}` stays as the editor-internal surface (no PATCH, no per-property routes) but is not the migration target. |
-| **SSE event-id contract?** | **Opaque to consumers** | Today it's the Redis Stream entry ID (`<ms>-<seq>`); consumers MUST treat as opaque so the backing store can be swapped without breaking the wire contract. Documented in the SSE endpoint comment and in this doc. |
+| **SSE event-id contract?** | **Opaque to consumers** | It is `<ms>-<seq>` (the Redis Stream entry form, kept after the 2026-10 SQLite switch); consumers MUST treat as opaque so the backing store can be swapped without breaking the wire contract. Documented in the SSE endpoint comment and in this doc. |
 | **Typed error envelope?** | **`{error, message, retryable}` with stable `error` slug** | See "Error envelope" subsection below. |
 | **Plugin direct-Redis risk?** | **Confirmed: no plugin uses Redis directly** | Grep across `packages/plugins/` for `ioredis` / `redis.createClient` / `new Redis()` returned empty. Plugins POST results back to meta-sort via `/api/plugins/callback`; meta-sort then writes. PR B picks up the plugin write path transparently. |
 | **meta-sort non-`file:*` writes?** | **Confirmed: none in production** | Full audit of `kvClient.*` callers in `packages/meta-sort/packages/meta-sort-core/src/` shows only: `setMetadataFlat`, `deleteMetadataFlat`, `getMetadataFlat`, `getMetadata`, `getAllHashIds`, `health`, plus the `file:events` stream consumer (`initStreamConsumer` / `startStreamConsumer` / `processPendingEntries`). The generic `kv.set(key, value)` / `kv.setProperty(key, value)` methods on the interface exist for plugin sandboxes and tests but are not invoked by production code with non-`file:*` keys. PR B + PR C cover the full meta-sort surface. |
@@ -591,7 +625,7 @@ Stable `error` slugs the client should switch on:
 | `unknown_root` | 404 | no | The hashId / UUID doesn't exist. Caller probably has a stale reference. |
 | `unknown_cid` | 404 | no | The CID isn't in the reverse index. No retry helps. |
 | `schema_violation` | 400 | no | Body shape rejected (missing required field, invalid value). Caller bug. |
-| `storage_unavailable` | 503 | yes | Redis transient. Caller should retry with backoff. |
+| `storage_unavailable` | 503 | yes | Store transient (Redis at the time; the embedded SQLite store since 2026-10). Caller should retry with backoff. |
 | `internal` | 500 | yes (cautiously) | Unhandled server-side error. Retry once; escalate if persistent. |
 
 **Implemented.** Every 4xx/5xx response across `internal/api`,
@@ -709,7 +743,9 @@ contract violation."
    One-shot fixer for already-stranded entries: walks every UUID's
    `midhash256` field to build the orphan index, merges each stranded
    `file:<midhash>/*` into the matching UUID via `MergeMetadataFlat`,
-   deletes the stranded root, and re-points the alias.
+   deletes the stranded root, and re-points the alias. *Removed in
+   2026-10* with the SQLite switch, together with the other one-shot Redis
+   migrations; it had been run on watch.nsl.sh before the migration.
 
 Verified on the dev stack: 48 stranded entries were reunited; the
 editor now shows every file with its full plugin metadata under one

docs/meta-core-architecture.md

@@ -3,14 +3,15 @@
 ## Document Purpose
 
 This document describes the architectural design of **meta-core**, the
-standalone container that owns Redis, the leader-info file, the HTTP+SSE
+standalone container that owns the metadata store (an embedded SQLite
+file; Redis until 2026-10), the leader-info file, the HTTP+SSE
 metadata surface, and a WebDAV file server. It covers the leader gate,
 storage data model, API design, the SSE event surface, and the integration
 contract with other MetaMesh services.
 
-For the on-disk Redis layout, see
+For the on-disk key layout, see
 [metadata-storage-structure.md](metadata-storage-structure.md). For the
-external-access design (Redis lockdown, SSE wire contract, auth rules),
+external-access design (store lockdown, SSE wire contract, auth rules),
 see [api-mediated-access.md](api-mediated-access.md). For the UUID-rooted
 storage rationale, see [uuid-rooted-metadata.md](uuid-rooted-metadata.md).
 
@@ -31,14 +32,15 @@ view of file metadata:
 
 meta-core sits below all of these as the single owner of:
 
-1. **Redis** — the metadata store and event source.
+1. **The metadata store** — an embedded SQLite file
+   (`/meta-core/db/meta-core.db`) holding metadata and the event streams.
 2. **The leader-info file** — the one place services look to discover meta-core's URLs.
 3. **The HTTP+SSE surface** — the only way external services touch the data plane.
 4. **A WebDAV server** — file access for plugins and other services.
 
 ### Core design principle
 
-> External services never speak Redis. Every read, write, and event flows
+> External services never touch the store. Every read, write, and event flows
 > through meta-core's HTTP+SSE surface.
 
 This is the contract that PR D of [api-mediated-access.md](api-mediated-access.md)
@@ -53,8 +55,8 @@ landed and that the rest of this document assumes.
 meta-core runs as a dedicated container (`metacore-app`). The dev compose
 runs a single instance; the legacy multi-instance topology (sibling
 followers) was deprecated. The flock primitive is retained as the
-correctness anchor — it guarantees that at most one process owns Redis and
-the writable state under `/meta-core`.
+correctness anchor — it guarantees that at most one process writes the
+database file and the writable state under `/meta-core`.
 
 ```
 ┌────────────────────────────────────────────────────────────────────────┐
@@ -62,17 +64,17 @@ the writable state under `/meta-core`.
 │                                                                        │
 │  supervisord                                                           │
 │   ├─ leader-election.sh   (flock → write kv-leader.info → exec self)   │
-│   ├─ redis-server         (AOF + RDB on /meta-core/db/redis)           │
 │   ├─ nginx                (TLS termination, proxy_cache for /webdav)   │
 │   ├─ rclone rcd           (mount daemon)                               │
 │   ├─ mount-watcher.sh     (rclone mount lifecycle)                     │
 │   └─ meta-core (Go) ──►   HTTP + SSE + WebDAV on :9000                 │
+│                           + embedded SQLite (db/meta-core.db)          │
 └────────────────────────────────────────────────────────────────────────┘
                                     │
        ┌────────────────────────────┴────────────────────────────┐
        │  /meta-core (shared volume)                             │
        │    locks/kv-leader.{lock,info}                          │
-       │    db/redis/                                            │
+       │    db/meta-core.db (+ -wal, -shm)                       │
        │    services/*.json                                      │
        │    mounts/{mounts.json, errors/}                        │
        │    watchers.json                                        │
@@ -152,7 +154,7 @@ Container startup
        ▼
 ┌──────────────────────────────┐
 │ supervisord starts:          │
-│   redis → nginx → rclone →   │
+│   nginx → rclone →           │
 │   meta-core Go binary        │
 └──────────────────────────────┘
 ```
@@ -167,10 +169,11 @@ gate wins. Therefore:
   local hostname + IP + config and returns it on demand. There are no
   `OnBecomeLeader` / `OnBecomeFollower` / `OnLeaderLost` callbacks; those
   states do not exist inside the Go process.
-- `LeaderLockInfo.RedisUrl` is intentionally empty in the response from
-  `/urls` — see `internal/leader/election.go:82`. PR D of
-  api-mediated-access.md removed direct Redis exposure; consumers route
-  metadata I/O through meta-core's HTTP API and SSE event streams.
+- `LeaderLockInfo.RedisUrl` is always empty in the response from
+  `/urls` (kept for wire compatibility). PR D of
+  api-mediated-access.md removed direct Redis exposure, and since 2026-10
+  there is no Redis at all; consumers route metadata I/O through
+  meta-core's HTTP API and SSE event streams.
 
 ### Lock-info file
 
@@ -192,18 +195,18 @@ Other services read this file, then call `GET /urls` for full discovery
   `metacore-app` is provisioned, its bash loop will acquire on next tick.
   The dev compose runs a single instance; failover is "restart the
   container."
-- **Redis crash inside the container** → supervisord restarts redis. The
-  Go binary's connect loop retries up to 30 times (1s delay) at startup;
-  at steady state, individual operations fail and clients retry.
+- **The store is in-process** — there is no separate database process to
+  crash. A meta-core crash is a process restart; SQLite's WAL recovers the
+  last committed transaction on reopen.
 
 ---
 
 ## Schema-Version Gate
 
 Before serving any traffic, `cmd/meta-core/main.go:54-61` calls
-`storage.EnsureSchemaVersion`. The sentinel lives in Redis at
+`storage.EnsureSchemaVersion`. The sentinel lives in the store at
 `meta-core:schema-version` (`internal/storage/schema_version.go`). The
-gate refuses to boot when the existing Redis layout predates the
+gate refuses to boot when the existing store layout predates the
 UUID-rooted schema, so legacy `file:midhash256:*` data can't leak into
 the current code paths. In alpha there is no automated migration —
 operators wipe with `pnpm run clean:all` and restart.
@@ -216,20 +219,20 @@ operators wipe with `pnpm run clean:all` and restart.
 
 ```
 ┌────────────────────────────────────────────────────────────────────┐
-│                       METADATA  (Redis)                            │
+│                 METADATA  (embedded SQLite, kv table)              │
 │                                                                    │
 │  Root: opaque UUIDv7 (Crockford Base32; ULID layout, 26 chars)     │
 │  Storage: flat string keys                                         │
 │           file:<uuid>/<property>           → string value          │
 │           file:<uuid>/cids/<bareCid>       → "true" (key-set)      │
-│           file:<uuid>/duplicates           → SET of paths          │
+│           dup:<uuid>/<path>                → "" (one per path)     │
 │                                                                    │
 │  No stored canonical_cid — derived by rank on read.                │
 │                                                                    │
-│  Aliases (reverse index, one Redis STRING each):                   │
+│  Aliases (reverse index, one key each):                            │
 │           cid:<bareCid>                    → <uuid>                │
 │                                                                    │
-│  Index:   file:__index__                   → SET of all UUIDs      │
+│  Index:   idx:roots/<uuid>                 → "" (one per root)     │
 │                                                                    │
 │  Mintable without a local file (remote / deleted entries OK).      │
 │  Resolvable by any registered CID in O(1).                         │
@@ -317,7 +320,7 @@ Editor / KV / schema / snapshot
   POST /api/snapshot/import
   POST /api/snapshot/wipe
 
-SSE event streams (HTTP-mirror of Redis Streams)
+SSE event streams (HTTP-mirror of the store's event streams)
   GET /api/events/files
   GET /api/events/meta
 
@@ -337,11 +340,10 @@ Watchers
   POST /api/watchers/{scan-all,reset-all}
   (deprecated: /api/scan/trigger, /api/scan/status — see watcher/handlers.go:32)
 
-Admin
-  POST /api/admin/migrate-dual-roots
-  POST /api/admin/sweep-literature-echoes   (dry-run unless ?apply=true;
-       strips query echoes off domain=literature fileType=card records —
-       see storage/literature_echo_sweep.go)
+(The one-shot Redis-era admin migrations — `/api/admin/migrate-dual-roots`,
+`migrate-domain-screen`, `migrate-user-cids`, `sweep-literature-echoes` —
+were removed with the 2026-10 SQLite switch; they had been run once on
+watch.nsl.sh beforehand.)
 
 WebDAV
   /webdav/...   (GET, PUT, DELETE, MKCOL, COPY, MOVE, PROPFIND)
@@ -369,12 +371,16 @@ See `docs/api-mediated-access.md` "Auth" for the rationale.
 ### SSE event streams — wire contract
 
 `/api/events/files` and `/api/events/meta` are SSE mirrors of the
-`file:events` and `meta:events` Redis Streams. The contract
+`file:events` and `meta:events` streams, stored as `ev:<stream>/…` rows of
+the kv table (`internal/kv/stream.go`). The contract
 (`internal/api/sse_events.go`):
 
-- One SSE event per Redis Stream entry.
-- `id:` is the opaque `<ms>-<seq>` Redis Stream entry ID — clients echo
-  it back via `Last-Event-ID` on reconnect.
+- One SSE event per stream entry.
+- `id:` is the opaque `<ms>-<seq>` entry ID (the Redis Stream form, kept so
+  cursors persisted before the 2026-10 switch still compare) — clients echo
+  it back via `Last-Event-ID` on reconnect. Without `Last-Event-ID` the
+  stream starts at the head (resolved once at connect); `0` / `0-0` replay
+  from the beginning.
 - `event:` is the `type` field of the underlying entry.
 - `data:` is the JSON of the rest of the entry.
 - Heartbeats: SSE comment `:keep-alive\n\n` every 30s of silence.
@@ -383,9 +389,9 @@ See `docs/api-mediated-access.md` "Auth" for the rationale.
   `event: gap` payload before resuming from the oldest available entry.
 
 External services use SSE with `Last-Event-ID` as the resume primitive.
-A Redis consumer-group (XREADGROUP / XACK) exists *internally* — the
-`MetaPublisher` uses one to drain keyspace notifications — but it is not
-the external contract.
+`meta:events` entries are appended by the storage engine in the same
+transaction as the change they describe; the Redis-era keyspace-notification
+hop through `MetaPublisher` is gone.
 
 ### WebDAV URLs
 
@@ -415,7 +421,7 @@ WebDAV caching is handled by nginx's `proxy_cache` (see
 | Single binary | Static compilation |
 | Low resource | ~10-20 MB resident |
 | HTTP server | stdlib `net/http` + gorilla/mux |
-| Redis client | go-redis |
+| Metadata store | embedded SQLite via `modernc.org/sqlite` (pure Go, CGO off) |
 | WebDAV | `golang.org/x/net/webdav` |
 
 ### Repo layout
@@ -428,12 +434,13 @@ packages/meta-core/
 │   ├── cid/                      CID parsing + algorithm ranking
 │   ├── config/                   env-driven configuration
 │   ├── discovery/                service registry + dead-service cleaner
-│   ├── events/                   meta:events stream publisher
+│   ├── events/                   boot-time meta:events republish
+│   ├── kv/                       SQLite kv engine + event streams
 │   ├── leader/                   LeaderInfoProvider (no election logic)
 │   ├── mounts/                   rclone mount manager + handlers + stats
 │   ├── schema/                   live per-field schema indexer
 │   ├── snapshot/                 snapshot export / import / wipe
-│   ├── storage/                  Redis wrapper + UUID + CID reverse index + schema gate
+│   ├── storage/                  flat-key records + search + UUID + CID reverse index + schema gate
 │   ├── watcher/                  scanner + dispatcher + midhash + state
 │   ├── watchers/                 polling-based watcher configs
 │   └── webdav/                   /webdav/* handler
@@ -454,8 +461,8 @@ packages/meta-core/
 ```
 1. Load configuration from environment
 2. Build LeaderInfoProvider          (no election; bash gate already won)
-3. Connect to local Redis            (retry up to 30× / 1s)
-4. EnsureSchemaVersion               (abort if Redis layout is stale)
+3. Open the SQLite store             (META_CORE_DB_PATH or /meta-core/db/meta-core.db)
+4. EnsureSchemaVersion               (abort if the store layout is stale)
 5. Start service discovery + dead-service cleaner
 6. NewServer + Server.Start          (boots watcher, mounts, WebDAV, API)
 7. On first connection: start MetaPublisher + schema Indexer,
@@ -466,9 +473,9 @@ packages/meta-core/
 
 ### Process management
 
-Redis is not spawned by the Go binary; supervisord starts it before
-meta-core (`docker/supervisord.conf`). The Go binary connects to
-`localhost:6379` and retries until Redis is reachable.
+There is no database process: the store is embedded in the Go binary
+(`docker/supervisord.conf` runs meta-core, nginx and rclone). Until
+2026-10 supervisord also ran `redis-server` on `localhost:6379`.
 
 ---
 
@@ -476,15 +483,16 @@ meta-core (`docker/supervisord.conf`). The Go binary connects to
 
 ### Container restart
 
-Supervisord restarts crashed processes (meta-core, redis, nginx, rclone)
+Supervisord restarts crashed processes (meta-core, nginx, rclone)
 individually. The bash flock is held by supervisord's process tree, so
 internal crashes don't trigger re-election.
 
-### Redis crash
+### Store durability
 
-Supervisord restarts redis. The Go binary's connect retry handles the
-window. Persistence is RDB + AOF on the `/meta-core` volume, so no data is
-lost across restarts.
+The SQLite file runs in WAL mode with `synchronous=NORMAL`: every
+committed transaction survives a meta-core process crash. A record write — fields, root index, CID aliases,
+FTS row and its events — is one transaction, so a crash never leaves a
+half-written record.
 
 ### Container crash
 
@@ -517,7 +525,10 @@ via `/health`; metadata I/O and SSE event streams stop.
 | `SERVICE_VERSION` | `1.0.0` | Reported in `/status`. |
 | `API_PORT` | `8180` | External port baked into `baseUrl`. |
 | `BASE_URL` | _empty_ | Overrides constructed `baseUrl` (HTTPS perimeter). |
-| `REDIS_PORT` | `6379` | Local Redis port. |
+| `META_CORE_DB_PATH` | `<META_CORE_PATH>/db/meta-core.db` | SQLite database file. Dev stacks set `/meta-core-db/meta-core.db` (named volume — SQLite WAL is unsafe on the 9p `/meta-core` bind of Windows dev boxes). |
+| `META_CORE_DB_CACHE_MB` | `64` | Page cache of the writer connection. |
+| `META_CORE_DB_READER_CACHE_MB` | `16` | Page cache per reader connection. |
+| `META_CORE_DB_READERS` | `8` | Reader connection pool size. |
 | `META_CORE_HTTP_PORT` | `9000` | Go HTTP+SSE+WebDAV port. |
 | `META_CORE_HTTP_HOST` | `0.0.0.0` | HTTP bind address. |
 | `HEALTH_CHECK_INTERVAL_MS` | `5000` | Internal health-loop cadence. |
@@ -536,7 +547,7 @@ via `/health`; metadata I/O and SSE event streams stop.
 |---|---|
 | `locks/kv-leader.lock` | Flock target. |
 | `locks/kv-leader.info` | Leader API URL (plaintext). |
-| `db/redis/{dump.rdb, appendonly.aof}` | Redis persistence. |
+| `db/meta-core.db` (+ `-wal`, `-shm`) | Metadata store (SQLite). Replaced `db/redis/` in 2026-10. |
 | `services/*.json` | Service registry. |
 | `mounts/mounts.json` | rclone mount configuration. |
 | `mounts/errors/{id}.error` | Mount error logs. |
@@ -552,5 +563,5 @@ and file access for MetaMesh into one container with a typed HTTP+SSE
 surface. The bash flock gate guarantees single-leader semantics without
 an external consensus system. UUID-rooted storage with CID aliasing
 treats all content hashes symmetrically and makes any-CID resolution
-O(1). API-mediated access ensures no other service ever needs to speak
-Redis.
+a single lookup. API-mediated access ensures no other service ever touches
+the store.

docs/metadata-storage-structure.md

@@ -1,6 +1,6 @@
-# Redis Metadata Storage Structure
+# Metadata Storage Structure
 
-How meta-core lays out file metadata in Redis. This document covers the
+How meta-core lays out file metadata in its store. This document covers the
 on-disk shape (UUID-rooted flat string keys, CID reverse index,
 per-root sets) and the schema-version gate. It does **not** enumerate
 field semantics — that's the job of the repo-root
@@ -12,40 +12,80 @@ reverse index), see [`uuid-rooted-metadata.md`](uuid-rooted-metadata.md).
 For the external-access contract (HTTP-only, SSE for events), see
 [`api-mediated-access.md`](api-mediated-access.md).
 
+> **Storage engine (since 2026-10): an embedded SQLite file, not Redis.**
+> Until 2026-10 these keys lived in a Redis server spawned inside the
+> container. They now live in one string→string table of an embedded SQLite
+> database (pure-Go driver `modernc.org/sqlite`, CGO off). The **key names
+> did not change** — only the Redis *types* that some bookkeeping used (SETs,
+> HASHes, Streams) were flattened into plain keys, listed below. Holding
+> everything in RAM had cost ~6.6 GB on watch.nsl.sh's 169k records (Redis
+> plus the API's in-memory search copy) and caused repeated OOM kills.
+
 ## Quick summary
 
+- **One table**: `kv(k TEXT PRIMARY KEY, v TEXT NOT NULL) WITHOUT ROWID`,
+  schema-free — a new metadata field is just a new key, never a migration.
+  File: `/meta-core/db/meta-core.db` (override `META_CORE_DB_PATH`), WAL
+  mode, `synchronous=NORMAL`.
 - **Roots are UUIDv7** encoded as 26-char Crockford Base32 (ULID layout),
   not content hashes. Examples in this doc use the placeholder
   `01JKR8XW5T4QMV7AHNJ2DEFGHK`.
-- **Properties are flat Redis STRING keys**, one per leaf:
-  `file:<uuid>/<property/path>` → string value. *Not* Redis Hashes.
+- **Properties are flat string keys**, one per leaf:
+  `file:<uuid>/<property/path>` → string value. A record is exactly the
+  key range `[file:<uuid>/, file:<uuid>0)` — there is no per-record field
+  list.
 - **CIDs are reverse-index aliases**: `cid:<bareCid>` → `<uuid>`
-  (a Redis STRING per known CID; no algorithm prefix — the multicodec
+  (one key per known CID; no algorithm prefix — the multicodec
   already carries it).
-- **Per-root sets**: `file:<uuid>/cids/<bareCid>` (a *key-set*: one STRING
-  `"true"` per known CID, midhash included), `file:<uuid>/duplicates`
-  (alternate paths with the same content), plus `file:__index__` (every UUID).
+- **Per-root sets**: `file:<uuid>/cids/<bareCid>` (a *key-set*: one key
+  `"true"` per known CID, midhash included), `dup:<uuid>/<path>` (alternate
+  paths with the same content), plus `idx:roots/<uuid>` (every UUID).
   There is **no stored `canonical_cid`** — it is derived by rank on read.
 - **A schema-version sentinel** (`meta-core:schema-version`) refuses to
   boot against legacy data.
 
-Implementation lives in `internal/storage/client.go`,
-`internal/storage/cid_resolution.go`, `internal/storage/uuid.go`, and
-`internal/storage/schema_version.go`.
+Implementation lives in `internal/kv/kv.go` (engine),
+`internal/kv/stream.go` (event streams), `internal/storage/client.go`,
+`internal/storage/search.go`, `internal/storage/cid_resolution.go`,
+`internal/storage/uuid.go`, and `internal/storage/schema_version.go`.
 
 ## Key structure
 
+Everything is one string→string table. The full layout:
+
+| Key | Value | Notes |
+|---|---|---|
+| `file:<rid>/<field>` | field value | a record property — the **only** thing under `file:` |
+| `cid:<cid>` | `<rid>` | reverse index |
+| `meta-core:schema-version` | `"2"` | schema gate |
+| `idx:roots/<rid>` | `""` | set of every root (was the Redis SET `file:__index__`) |
+| `idx:fts/<rid>` | FTS rowid | the record's row in the FTS table |
+| `dup:<rid>/<path>` | `""` | a duplicate path (was the SET `file:<rid>/duplicates`) |
+| `udl:rec:<uid>/<cid>/<key>` | JSON `{record,version,ts,value?}` | User Data Layer entry (was a HASH) |
+| `udl:idx:…<set>\x1f<member>` | `""` | UDL index sets, one key per member |
+| `ev:<stream>/<ms13>-<seq10>` | JSON of the entry's fields | `meta:events` / `file:events` entries (were Redis Streams) |
+| `evhwm:<stream>` | last issued ID | stream high-water mark |
+| `btmagnet:*`, `searchcov:*`, `nzb:*` | — | written by other services via `/api/kv/value`; unchanged |
+
+Keeping bookkeeping **out of `file:`** is load-bearing: the `file:` range is
+exactly the records, so the field index covers only real fields and every
+change under `file:` is a real field change worth publishing on
+`meta:events`. The per-record `file:<rid>/__fields__` SET and the
+`file:__dirty__` set of the Redis era no longer exist. Roots are split from
+the field at the **first** `/`, so CID-rooted and UUID-rooted records both
+parse.
+
 ### Roots
 
 ```
-file:<uuid>/<property/path>   →   Redis STRING (always)
+file:<uuid>/<property/path>   →   string (always)
 ```
 
 UUIDs are produced by `storage.NewUUID()` (`internal/storage/uuid.go`):
 UUIDv7 → 26-char Crockford Base32 (skips I/L/O/U; ULID-compatible). The
-time-sortable prefix lets you `KEYS file:01JKR8*` to find roots minted
-in a given window, which restored some of the debug-ability lost when
-midhash256 stopped being the root.
+time-sortable prefix means roots minted in a given window sort together
+(`k >= 'file:01JKR8' AND k < 'file:01JKR9'`), which restored some of the
+debug-ability lost when midhash256 stopped being the root.
 
 Properties use `/` as the path separator. Nested objects in source code
 are flattened to flat keys on write and reconstructed on read. Examples:
@@ -62,7 +102,7 @@ file:01JKR8XW5T4QMV7AHNJ2DEFGHK/tmdb/poster                       = "bafkreih...
 
 ### Reverse-index aliases (CIDs)
 
-Every CID known for a file is registered as its own Redis STRING, keyed by
+Every CID known for a file is registered as its own key, keyed by
 the **bare CID** — no algorithm prefix. A CIDv1 already encodes its hash
 function in the multicodec, so naming it again in the key would be
 redundant (and would let the same digest be indexed under two spellings):
@@ -78,13 +118,12 @@ cid:bafkreih5kapbjzq...   →   01JKR8XW5T4QMV7AHNJ2DEFGHK
 cid:bagacbabaec7v3...     →   01JKR8XW5T4QMV7AHNJ2DEFGHK
 ```
 
-The reverse index makes `GET /api/meta/<cid>` an O(1) lookup for any
+The reverse index makes `GET /api/meta/<cid>` a primary-key lookup for any
 registered CID, independent of which hash algorithm — there is no primary.
-The auto-alias hook (`maybeAddAliasFromFieldLocked` in
-`internal/storage/cid_resolution.go`, called from `SetProperty`,
-`SetMetadataFlat` and `MergeMetadataFlat`) registers the alias for any
-field whose name starts with `cids/`, so writers emit key-set members
-without knowing the reverse index exists.
+The record write path (`writeFieldsTx` in `internal/storage/client.go`, used
+by `SetProperty`, `SetMetadataFlat` and `MergeMetadataFlat`) registers the
+alias for any field whose name starts with `cids/`, in the same transaction,
+so writers emit key-set members without knowing the reverse index exists.
 
 > ⚠ The hook matches **only** the `cids/` prefix. A legacy `cid_*` or
 > `midhash256` *named* field written to a root is stored verbatim and is
@@ -95,12 +134,13 @@ without knowing the reverse index exists.
 ### Per-root sets
 
 ```
-file:<uuid>/cids/<bareCid>    →   STRING "true" — one key per known CID (a key-set, not a Redis SET)
-file:<uuid>/duplicates        →   Redis SET of alternate paths with same content
+file:<uuid>/cids/<bareCid>    →   "true" — one key per known CID (a key-set)
+dup:<uuid>/<path>             →   ""     — one key per alternate path with same content
 ```
 
-`cids` is a **key-set**: one flat `file:<uuid>/cids/<cid>` STRING per member,
-which keeps it conflict-free to merge across peers (a SET would not be).
+`cids` is a **key-set**: one flat `file:<uuid>/cids/<cid>` key per member,
+which keeps it conflict-free to merge across peers (a single set-valued
+field would not be).
 The record's own address (the midhash) is just the `0x1000` member — it has
 no separate field.
 
@@ -114,61 +154,88 @@ was removed with the flat `cid_*` keys.
 ### Global index
 
 ```
-file:__index__   →   Redis SET of all UUIDs
+idx:roots/<uuid>   →   ""   (one key per root; was the Redis SET file:__index__)
 ```
 
-`internal/storage/client.go::GetAllHashIDs` reads this set to enumerate
-every file without scanning the keyspace.
+`internal/storage/client.go::GetAllHashIDs` range-reads `idx:roots/` to
+enumerate every file without scanning the records.
 
-```bash
-SMEMBERS file:__index__         # all UUIDs
-SCARD    file:__index__         # total file count
-SISMEMBER file:__index__ <uuid>
-```
+### Search structures (derived)
+
+Two derived structures are maintained on top of `kv`, in the same
+transaction as every record write. Both are rebuildable from the kv rows and
+never authoritative (`internal/storage/search.go`):
+
+- **`kv_field`** — a partial expression index
+  `ON kv(lower(substr(k,instr(k,'/')+1)), lower(v)) WHERE k>='file:' AND k<'file;' AND instr(k,'/')>0`.
+  It answers "which records have field F with a value like V" for **any**
+  field, including ones invented tomorrow — the index is over the key itself.
+- **`fts`** — an FTS5 trigram table `fts(rid UNINDEXED, hay)` holding one row
+  per record: its search haystack plus its rid (`idx:fts/<rid>` points at the
+  row). Trigrams make every 3+ character substring indexable.
+
+`POST /api/metadata/search` uses SQL to build a superset of candidates, then
+verifies each with the unchanged Go matchers (`recordMatchesFilter`,
+`matchAllTokens`), so semantics are identical to the old in-memory index.
+Results still come in hashID order and are fresh immediately (no refresh
+lag). This replaced the API's in-memory `searchIndex` (60 s refresh, ~30 KB
+RAM per record).
 
 ### Schema-version sentinel
 
 ```
-meta-core:schema-version   →   STRING (currently "2")
+meta-core:schema-version   →   string (currently "2")
 ```
 
 Set at first boot, checked on every subsequent boot
 (`internal/storage/schema_version.go`). If the sentinel is missing but
-`file:__index__` is non-empty, the gate refuses to boot (legacy data).
+the root index is non-empty, the gate refuses to boot (legacy data).
 If the sentinel is present but doesn't match `SchemaVersion`, the gate
 also refuses to boot. Alpha-stage policy: operators wipe and restart;
 there is no automated migration.
 
-## Why flat STRING keys?
+## Why flat string keys?
 
-The store does not use Redis Hashes. The reasons are:
+The store keeps one flat key per leaf rather than a document or a
+per-record row. The reasons are:
 
-1. **Field-level keyspace notifications.** Redis publishes
-   `__keyspace@0__:<key>` on every SET/DEL of a STRING key. With Hashes
-   you'd only get one notification on the hash itself. meta-core's
-   `MetaPublisher` (`internal/events/meta_publisher.go`) consumes these
-   notifications and publishes structured events to the `meta:events`
-   Redis Stream.
-2. **Selective updates without read-modify-write.** A plugin writing
+1. **Schema-free.** A plugin inventing a field writes a new key; nothing
+   declares it, and the table never needs a migration. The field index
+   covers it automatically.
+2. **Conflict-free merging across peers.** Key-sets (`cids/<cid>`, and the
+   flat fields in general) merge by union of keys — no read-modify-write of
+   a shared value, nothing to reconcile.
+3. **Selective updates without read-modify-write.** A plugin writing
    `tmdb/poster` doesn't have to read the rest of the document.
-3. **Compatibility with the CID alias hooks.** Field writes can trigger
-   alias-maintenance work in `SetProperty` without the plugin caring.
+4. **Compatibility with the CID alias hooks.** Field writes can trigger
+   alias-maintenance work in the write path without the plugin caring.
+
+(Under Redis there was a fourth reason — field-level keyspace notifications
+fed `meta:events`. That is obsolete: events are now written by the storage
+engine itself, see [Event streams](#event-streams).)
 
 ## Reads and writes
 
+All writes go through **one writer goroutine** with group commit: concurrent
+writes share one SQLite transaction, each under its own SAVEPOINT so a
+failure rolls back alone. A record write — fields + root index + CID aliases
++ FTS row + events — is atomic. (Measured ~835 records/s with 16 concurrent
+writers of 40-field records.) Reads use a separate connection pool and see
+the last committed state.
+
 The Go API in `internal/storage/client.go`:
 
 | Method | Behaviour |
 |---|---|
-| `GetMetadataFlat(uuid)` | `SCAN` for `file:<uuid>/*`, `MGET` the values, return `map[string]string`. |
-| `SetMetadataFlat(uuid, m)` | Pipeline `SET` per field; `SADD file:__index__`; auto-register an alias for every `cids/<cid>` member. |
+| `GetMetadataFlat(uuid)` | Range-read `[file:<uuid>/, file:<uuid>0)`, return `map[string]string`. |
+| `SetMetadataFlat(uuid, m)` | Put each field; add `idx:roots/<uuid>`; auto-register an alias for every `cids/<cid>` member; refresh the FTS row. One transaction. |
 | `MergeMetadataFlat(uuid, m)` | Like `SetMetadataFlat` but doesn't pre-clear; used by `PATCH /meta/{hash}`. |
-| `GetProperty(uuid, field)` | `GET file:<uuid>/<field>`. |
-| `SetProperty(uuid, field, value)` | `SET file:<uuid>/<field>`; auto-register alias if the field is a CID. |
-| `DeleteRoot(uuid)` | `SCAN` + `DEL` for `file:<uuid>/*`, unmap every reverse-index entry. |
+| `GetProperty(uuid, field)` | Get `file:<uuid>/<field>`. |
+| `SetProperty(uuid, field, value)` | Put `file:<uuid>/<field>`; auto-register alias if the field is a CID. |
+| `DeleteRoot(uuid)` | Delete the `file:<uuid>/` range, its duplicates, search row and index membership; unmap every reverse-index entry. |
 | `Mint(filePath, size, mtimeNano)` | `NewUUID()` + write `filePath/sizeByte/mtimeNano` + add to index. |
 | `AddAlias(uuid, "<bareCid>")` | Set `cid:<bareCid>` → `<uuid>` and the key-set member `file:<uuid>/cids/<bareCid>` = `"true"`. Nothing to reconcile — canonical is derived on read. |
-| `ResolveRoot("<bareCid>")` | `GET cid:<bareCid>` → uuid (returns the input unchanged if it is already a UUID root). |
+| `ResolveRoot("<bareCid>")` | Get `cid:<bareCid>` → uuid (returns the input unchanged if it is already a UUID root). |
 | `GetMetadataDocument(uuid)` | Flat fields + `cids` hoisted to an array + `canonical_cid` **derived** by `cid.Better` rank + `duplicates`. Backs `GET /api/meta/{cid}`. |
 
 ## Reconstruction (flat → nested)
@@ -196,7 +263,7 @@ Rules:
 
 ## Plugin contribution flow
 
-Plugins don't talk to Redis. They write metadata via meta-core's HTTP
+Plugins don't talk to the store directly. They write metadata via meta-core's HTTP
 API (`PUT /meta/{hash}/{key...}` or `PATCH /meta/{hash}`). Each plugin
 writes under a prefix:
 
@@ -215,12 +282,24 @@ deliberately does not duplicate that registry — it would drift.
 
 ## Event streams
 
-Two Redis Streams are produced inside meta-core:
+Two event streams are kept inside the store, as rows
+`ev:<stream>/<ms13>-<seq10>` (zero-padded so key order is ID order) plus a
+high-water mark `evhwm:<stream>` (`internal/kv/stream.go`):
 
 | Stream | Producer | Contents |
 |---|---|---|
 | `file:events` | `internal/watcher` dispatcher | `add` / `change` / `delete` / `rename` / `reset` events for files on the `/files` volume. |
-| `meta:events` | `internal/events.MetaPublisher` | Field-level metadata mutations derived from Redis keyspace notifications. |
+| `meta:events` | the storage engine (`internal/kv`) | `{type: set\|del, key, ts}` for every Put/Delete of a `file:` key, appended **in the same transaction** as the change. |
+
+`del` is emitted only when the key existed; `set` is emitted even for an
+unchanged value (Redis semantics, kept). Retention is ~100k entries,
+trimmed every 10k appends. At boot `MetaPublisher`
+(`internal/events/meta_publisher.go`) still republishes every field into
+`meta:events` (meta-fuse rebuilds its VFS only by replaying from `0-0`),
+batched 5k events per transaction.
+
+The Redis-era path — keyspace notifications drained by `MetaPublisher`
+through a PSUBSCRIBE / consumer-group hop — is gone (2026-10).
 
 External consumers do **not** read these streams directly. They subscribe
 via SSE:
@@ -228,51 +307,50 @@ via SSE:
 - `GET /api/events/files` — mirror of `file:events`
 - `GET /api/events/meta` — mirror of `meta:events`
 
-The SSE wire contract uses `Last-Event-ID` for resume (the opaque
-`<ms>-<seq>` Redis Stream entry ID). Heartbeats are SSE comments
+The SSE wire contract uses `Last-Event-ID` for resume. IDs keep the Redis
+`<ms>-<seq>` form so cursors persisted by meta-sort / meta-share /
+meta-stremio across the switch still compare. Heartbeats are SSE comments
 (`:keep-alive\n\n`) every 30s of silence. If retention has trimmed past
 a client's cursor, the handler emits one `event: gap` payload before
 resuming. See `internal/api/sse_events.go` and
 [`api-mediated-access.md`](api-mediated-access.md) for the full
 contract.
 
-Internally, meta-core uses a Redis consumer group (XREADGROUP / XACK)
-inside `MetaPublisher` to drain keyspace notifications, but this is not
-the external surface — external services use SSE.
-
 ## Debugging
 
-```bash
-# Roots
-docker exec metacore-app redis-cli SMEMBERS 'file:__index__'
-docker exec metacore-app redis-cli SCARD    'file:__index__'
-docker exec metacore-app redis-cli SRANDMEMBER 'file:__index__'
+The database path is `/meta-core/db/meta-core.db` in the image default and
+on the store app; dev stacks use `/meta-core-db/meta-core.db` (a named
+volume — the `/meta-core` bind is 9p on Windows dev boxes, where SQLite WAL
+is unsafe). Always open it **read-only**; writes must go through the API.
 
-# Full document for one root (flat scan)
-docker exec metacore-app redis-cli --scan --pattern 'file:01JKR8XW5T4QMV7AHNJ2DEFGHK/*'
+```bash
+DB=/meta-core/db/meta-core.db      # dev stack: /meta-core-db/meta-core.db
+Q() { docker exec metacore-app sqlite3 -readonly "$DB" "$1"; }
+
+# Roots
+Q "SELECT count(*) FROM kv WHERE k >= 'idx:roots/' AND k < 'idx:roots0'"
+Q "SELECT substr(k,11) FROM kv WHERE k >= 'idx:roots/' AND k < 'idx:roots0' LIMIT 10"
+
+# Full document for one root
+Q "SELECT k,v FROM kv WHERE k >= 'file:01JKR8XW5T4QMV7AHNJ2DEFGHK/' AND k < 'file:01JKR8XW5T4QMV7AHNJ2DEFGHK0'"
 
 # One property
-docker exec metacore-app redis-cli GET 'file:01JKR8XW5T4QMV7AHNJ2DEFGHK/fileinfo/duration'
-
-# Confirm keys are STRING (not HASH)
-docker exec metacore-app redis-cli TYPE 'file:01JKR8XW5T4QMV7AHNJ2DEFGHK/filePath'
-# string
+Q "SELECT v FROM kv WHERE k = 'file:01JKR8XW5T4QMV7AHNJ2DEFGHK/fileinfo/duration'"
 
 # Reverse-index lookup (bare CID — no algorithm prefix)
-docker exec metacore-app redis-cli GET 'cid:bafkreih...'      # → uuid
+Q "SELECT v FROM kv WHERE k = 'cid:bafkreih...'"                   # → uuid
 
-# The cids key-set (one STRING per member, NOT a Redis SET — SMEMBERS won't work)
-docker exec metacore-app redis-cli --scan --pattern 'file:01JKR8XW5T4QMV7AHNJ2DEFGHK/cids/*'
+# The cids key-set
+Q "SELECT k FROM kv WHERE k >= 'file:01JKR8XW5T4QMV7AHNJ2DEFGHK/cids/' AND k < 'file:01JKR8XW5T4QMV7AHNJ2DEFGHK/cids0'"
 
 # canonical_cid is NOT stored. Derive it via the API instead:
 curl -k https://metacore-dev.localhost:8083/api/meta/<cid> | jq '.canonical_cid, .cids'
 
 # Schema-version sentinel
-docker exec metacore-app redis-cli GET 'meta-core:schema-version'
+Q "SELECT v FROM kv WHERE k = 'meta-core:schema-version'"
 
 # Streams (server-side; external consumers should use SSE)
-docker exec metacore-app redis-cli XRANGE 'file:events' - + COUNT 10
-docker exec metacore-app redis-cli XRANGE 'meta:events' - + COUNT 10
+Q "SELECT k,v FROM kv WHERE k >= 'ev:meta:events/' AND k < 'ev:meta:events0' ORDER BY k DESC LIMIT 10"
 
 # HTTP equivalents — preferred
 curl -k 'https://metacore-dev.localhost:8083/api/meta/<cid>'      # CID → document
@@ -286,12 +364,14 @@ curl -k 'https://metacore-dev.localhost:8083/api/kv/info'         # KV stats
 
 | Component | Path |
 |---|---|
-| Redis client + flat-key ops | `internal/storage/client.go` |
+| Storage engine (SQLite kv table, writer, group commit) | `internal/kv/kv.go` |
+| Event streams as kv rows | `internal/kv/stream.go` |
+| Flat-key record ops | `internal/storage/client.go` |
+| Search (field index + FTS) | `internal/storage/search.go` |
 | UUIDv7 minting | `internal/storage/uuid.go` |
-| CID reverse index + aliases + canonical-CID reconciliation | `internal/storage/cid_resolution.go` |
+| CID reverse index + aliases + canonical-CID derivation | `internal/storage/cid_resolution.go` |
 | Schema-version gate | `internal/storage/schema_version.go` |
-| Dual-root migration (legacy clean-up) | `internal/storage/dual_root_migration.go` |
-| Keyspace-notification → `meta:events` publisher | `internal/events/meta_publisher.go` |
+| Boot-time `meta:events` republish | `internal/events/meta_publisher.go` |
 | Watcher → `file:events` dispatcher | `internal/watcher/dispatcher.go` |
 | SSE event mirror | `internal/api/sse_events.go` |
 | Field semantics (authoritative registry) | [`/METADATA_KEYS.md`](../../../METADATA_KEYS.md) |

docs/uuid-rooted-metadata.md

@@ -20,6 +20,12 @@
 >    `cid:<bareCid>` → uuid. A CIDv1's multicodec already names its hash
 >    function, so the `<algo>:` prefix was redundant.
 >
+> 3. **The store is no longer Redis (2026-10).** The same keys now live in
+>    one string→string table of an embedded SQLite file; Redis SETs became
+>    one key per member (`file:__index__` → `idx:roots/<uuid>`,
+>    `file:<uuid>/duplicates` → `dup:<uuid>/<path>`). Redis types named
+>    below are the shape at the time.
+>
 > Text below that references either shape describes the *proposal*, not the
 > code. The rank ordering and everything else did ship as written.
 
@@ -135,7 +141,7 @@ sole writer is meta-core itself:
 | Key | Type (as shipped) | Purpose |
 |-----|------|---------|
 | `cids/<bareCid>` | STRING `"true"` — a *key-set*, one key per member | All CIDs known for this file, midhash included. Mirror of the `cid:*` reverse index. Source of truth for delete-time GC. **Proposed as a Redis SET of `<algo>:<value>` tokens; shipped as a key-set of bare CIDs** — a key-set merges conflict-free across peers and the multicodec already names the algorithm. |
-| `duplicates` | Redis SET | Additional file paths whose content matches this entry. One UUID = one set of metadata; multiple physical files may share that content. Consumed by meta-dup. |
+| `duplicates` | Redis SET at the time; since 2026-10 one `dup:<uuid>/<path>` key per path | Additional file paths whose content matches this entry. One UUID = one set of metadata; multiple physical files may share that content. Consumed by meta-dup. |
 | ~~`canonical_cid`~~ | *(not stored)* | **Dropped.** Was to hold the externally-presented CID. Now derived on read by ranking the `cids` members; surfaced only on `GET /api/meta/{cid}`. |
 
 Existing meta-core-owned fields (`filePath`, `size`, `mtime`, `midhash256`)
@@ -148,7 +154,8 @@ for using sets where they're the right structure.~~ **Reversed for `cids`.**
 The all-strings convention won: `cids` ships as flat `cids/<cid>` STRING keys
 so that two peers merging the same record can never disagree about set
 membership. `duplicates` remained a Redis SET (it is leader-local and never
-merged across peers).
+merged across peers) until the 2026-10 SQLite switch, which flattened it to
+`dup:<uuid>/<path>` keys.
 
 ### Canonical CID selection