replace Redis with an embedded SQLite string->string KV store (group commit, in-tx events, SQL search)

docs/api-mediated-access.md

@@ -9,6 +9,16 @@ on [uuid-rooted-metadata.md](uuid-rooted-metadata.md), which assumes
 meta-core is the single writer to the metadata keyspace — an assumption
 that's now actually enforced rather than aspirational.
 
+> **2026-10: meta-core no longer uses Redis at all.** Its store is now an
+> embedded SQLite file (`/meta-core/db/meta-core.db`, see
+> [metadata-storage-structure.md](metadata-storage-structure.md)) and the
+> `file:events` / `meta:events` streams are rows of that store. The lockdown
+> below is now structural — there is no database port anywhere — and the
+> SSE wire contract was kept byte-for-byte, so no consumer changed. The
+> rest of this doc is the design record of the Redis-era lockdown; the
+> sections that describe *current* behaviour (SSE retention, reconnect,
+> error handling) have been updated in place.
+
 See "Implementation outcome" at the bottom for the delta between this doc's
 plan and what shipped (notably the write-path CID resolution and the
 dual-root migration sweep, which weren't in the original plan but turned
@@ -85,7 +95,7 @@ we don't fully control.
 ```
 
 **The single-writer rule.** meta-core has exclusive write authority over
-the `file:*`, `cid:*`, `file:__index__`, and `meta-core:*` keyspaces. No
+the `file:*`, `cid:*`, `file:__index__` (now `idx:*`), and `meta-core:*` keyspaces. No
 other service can write — not because of an honoured convention, but
 because they can't reach Redis at all.
 
@@ -243,6 +253,25 @@ After PR A, the cursor contract is **"durable across meta-core restart
 and bounded by `MAXLEN ~ N`"** — matching what consumers already assume
 when they persist a Last-Event-ID.
 
+**Current backing store (since 2026-10).** The streams are rows of
+meta-core's SQLite kv table — `ev:<stream>/<ms13>-<seq10>` → JSON of the
+entry's fields, plus a high-water mark `evhwm:<stream>` that seeds the next
+ID (so a `ClearStream` reset never reissues an ID an old cursor treats as
+seen). The contract above holds unchanged:
+
+- Entries persist across restart (they are in the database file).
+- Retention is ~100k entries per stream, trimmed every 10k appends.
+- IDs keep the Redis `<ms>-<seq>` form, so cursors persisted by meta-sort /
+  meta-share / meta-stremio before the switch still compare.
+- `meta:events` entries (`{type: set|del, key, ts}`) are appended by the
+  storage engine **in the same transaction** as the `file:` key change —
+  `del` only when the key existed, `set` even for an unchanged value. The
+  Redis keyspace-notification hop (PSUBSCRIBE in `MetaPublisher`), which
+  could drop events when it lagged, is gone.
+- The boot-time republish of every field into `meta:events` is kept
+  (meta-fuse rebuilds its VFS only by replaying from `0-0`), batched 5k
+  events per transaction.
+
 ### Heartbeats
 
 Server writes an SSE comment line (`:keep-alive\n\n`) every 30 seconds
@@ -254,10 +283,13 @@ per the SSE spec.
 
 Per the SSE spec the client (or `EventSource`) auto-reconnects after a
 network error, sending `Last-Event-ID`. The server resumes from the next
-entry after that ID by translating to Redis Stream `XREAD … STREAMS file:events <id>`.
+entry after that ID (a range read of the stream's rows; under Redis this
+was `XREAD … STREAMS file:events <id>`). With no `Last-Event-ID` the stream
+starts at the head, resolved once at connect; `0` / `0-0` go through the
+gap path and replay from the beginning.
 
-**Gap handling.** If the requested ID has been trimmed out of the Redis
-Stream by retention, the server emits one synthetic event before resuming
+**Gap handling.** If the requested ID is older than the oldest retained
+entry, the server emits one synthetic event before resuming
 from the oldest available entry:
 
 ```
@@ -271,9 +303,10 @@ react. Consumers that don't care ignore it.
 
 ### Error handling
 
-- **Redis unreachable on connect**: 503 with `Retry-After: 5`.
-- **Redis fails mid-stream**: close the SSE connection. Client reconnects
-  with `Last-Event-ID`; the cycle resumes when Redis recovers.
+- **Store unavailable on connect**: 503 with `Retry-After: 5`.
+- **Store read fails mid-stream**: close the SSE connection. Client
+  reconnects with `Last-Event-ID`; the cycle resumes when meta-core is back
+  (a meta-core restart is the common case — SSE clients just reconnect).
 - **Client disconnects**: handler goroutine exits cleanly; no per-client
   state on meta-core.
 
@@ -307,7 +340,8 @@ inside-only until a concrete external consumer appears.
 ### Implementation footprint
 
 The SSE handler itself is ~80 lines of Go in meta-core: one handler per
-stream, each running an `XREAD … BLOCK 30s` loop and writing
+stream, each running a read-after-cursor loop (an `XREAD … BLOCK 30s` loop
+in the Redis era) and writing
 SSE-formatted lines to the response. Go's `net/http` natively supports
 streaming responses with `http.Flusher.Flush()`. Standard SSE clients
 exist for every language we use: Node `eventsource`, Rust
@@ -562,11 +596,11 @@ metacore-app produces a connection refused. The lock is real.
 | Reads from `file:*` direct? | Forbidden by network topology | Same lock, same reason |
 | Event streams direct? | No — mediated through SSE | Kubernetes-style. Costs one HTTP hop; buys schema evolution + threat-model fit |
 | Enforcement mechanism? | Network isolation (primary) + docs + code review (secondary) | No ACLs, no proxy, no lint |
-| Backing store for streams? | Redis Streams (unchanged) | SSE wraps it; could be swapped later without consumer impact |
+| Backing store for streams? | Redis Streams at the time; SQLite kv rows since 2026-10 | SSE wraps it; the 2026-10 swap proved it — no consumer changed |
 | Consumer groups / horizontal scale? | Not supported via SSE | Not needed today; revisit if scale ever requires it |
 | Storage portability? | Not a goal but a free side-effect | Will not be exercised in alpha |
 | **Canonical write API path?** | **`/meta/{hash}` family** | Has GET/PUT/**PATCH**/DELETE plus `/_add/{key}` and per-property `{key:.*}` operations. `PATCH` calls `MergeMetadataFlat` — matches the partial-update semantics PR B needs. `/api/metadata/{hashId}` stays as the editor-internal surface (no PATCH, no per-property routes) but is not the migration target. |
-| **SSE event-id contract?** | **Opaque to consumers** | Today it's the Redis Stream entry ID (`<ms>-<seq>`); consumers MUST treat as opaque so the backing store can be swapped without breaking the wire contract. Documented in the SSE endpoint comment and in this doc. |
+| **SSE event-id contract?** | **Opaque to consumers** | It is `<ms>-<seq>` (the Redis Stream entry form, kept after the 2026-10 SQLite switch); consumers MUST treat as opaque so the backing store can be swapped without breaking the wire contract. Documented in the SSE endpoint comment and in this doc. |
 | **Typed error envelope?** | **`{error, message, retryable}` with stable `error` slug** | See "Error envelope" subsection below. |
 | **Plugin direct-Redis risk?** | **Confirmed: no plugin uses Redis directly** | Grep across `packages/plugins/` for `ioredis` / `redis.createClient` / `new Redis()` returned empty. Plugins POST results back to meta-sort via `/api/plugins/callback`; meta-sort then writes. PR B picks up the plugin write path transparently. |
 | **meta-sort non-`file:*` writes?** | **Confirmed: none in production** | Full audit of `kvClient.*` callers in `packages/meta-sort/packages/meta-sort-core/src/` shows only: `setMetadataFlat`, `deleteMetadataFlat`, `getMetadataFlat`, `getMetadata`, `getAllHashIds`, `health`, plus the `file:events` stream consumer (`initStreamConsumer` / `startStreamConsumer` / `processPendingEntries`). The generic `kv.set(key, value)` / `kv.setProperty(key, value)` methods on the interface exist for plugin sandboxes and tests but are not invoked by production code with non-`file:*` keys. PR B + PR C cover the full meta-sort surface. |
@@ -591,7 +625,7 @@ Stable `error` slugs the client should switch on:
 | `unknown_root` | 404 | no | The hashId / UUID doesn't exist. Caller probably has a stale reference. |
 | `unknown_cid` | 404 | no | The CID isn't in the reverse index. No retry helps. |
 | `schema_violation` | 400 | no | Body shape rejected (missing required field, invalid value). Caller bug. |
-| `storage_unavailable` | 503 | yes | Redis transient. Caller should retry with backoff. |
+| `storage_unavailable` | 503 | yes | Store transient (Redis at the time; the embedded SQLite store since 2026-10). Caller should retry with backoff. |
 | `internal` | 500 | yes (cautiously) | Unhandled server-side error. Retry once; escalate if persistent. |
 
 **Implemented.** Every 4xx/5xx response across `internal/api`,
@@ -709,7 +743,9 @@ contract violation."
    One-shot fixer for already-stranded entries: walks every UUID's
    `midhash256` field to build the orphan index, merges each stranded
    `file:<midhash>/*` into the matching UUID via `MergeMetadataFlat`,
-   deletes the stranded root, and re-points the alias.
+   deletes the stranded root, and re-points the alias. *Removed in
+   2026-10* with the SQLite switch, together with the other one-shot Redis
+   migrations; it had been run on watch.nsl.sh before the migration.
 
 Verified on the dev stack: 48 stranded entries were reunited; the
 editor now shows every file with its full plugin metadata under one