metastar/meta-core
4
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 2 Wiki Activity

replace Redis with an embedded SQLite string->string KV store (group commit, in-tx events, SQL search)
Build and Push Docker Image / build (push) Successful in 2m44s
Details

Browse Source
This commit is contained in:
Worph
2026-10-19 20:51:00 +00:00
parent c53146e514
commit dc5cbdfb9b
80 changed files with 4601 additions and 6077 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/meta-core-architecture.md
+64 -53
View File
@@ -3,14 +3,15 @@
## Document Purpose
This document describes the architectural design of **meta-core**, the
standalone container that owns Redis, the leader-info file, the HTTP+SSE
standalone container that owns the metadata store (an embedded SQLite
file; Redis until 2026-10), the leader-info file, the HTTP+SSE
metadata surface, and a WebDAV file server. It covers the leader gate,
storage data model, API design, the SSE event surface, and the integration
contract with other MetaMesh services.
For the on-disk Redis layout, see
For the on-disk key layout, see
[metadata-storage-structure.md](metadata-storage-structure.md). For the
external-access design (Redis lockdown, SSE wire contract, auth rules),
external-access design (store lockdown, SSE wire contract, auth rules),
see [api-mediated-access.md](api-mediated-access.md). For the UUID-rooted
storage rationale, see [uuid-rooted-metadata.md](uuid-rooted-metadata.md).
@@ -31,14 +32,15 @@ view of file metadata:
meta-core sits below all of these as the single owner of:
1. **Redis** — the metadata store and event source.
1. **The metadata store** — an embedded SQLite file
(`/meta-core/db/meta-core.db`) holding metadata and the event streams.
2. **The leader-info file** — the one place services look to discover meta-core's URLs.
3. **The HTTP+SSE surface** — the only way external services touch the data plane.
4. **A WebDAV server** — file access for plugins and other services.
### Core design principle
> External services never speak Redis. Every read, write, and event flows
> External services never touch the store. Every read, write, and event flows
> through meta-core's HTTP+SSE surface.
This is the contract that PR D of [api-mediated-access.md](api-mediated-access.md)
@@ -53,8 +55,8 @@ landed and that the rest of this document assumes.
meta-core runs as a dedicated container (`metacore-app`). The dev compose
runs a single instance; the legacy multi-instance topology (sibling
followers) was deprecated. The flock primitive is retained as the
correctness anchor — it guarantees that at most one process owns Redis and
the writable state under `/meta-core`.
correctness anchor — it guarantees that at most one process writes the
database file and the writable state under `/meta-core`.
```
┌────────────────────────────────────────────────────────────────────────┐
@@ -62,17 +64,17 @@ the writable state under `/meta-core`.
│ │
│ supervisord │
│ ├─ leader-election.sh (flock → write kv-leader.info → exec self) │
│ ├─ redis-server (AOF + RDB on /meta-core/db/redis) │
│ ├─ nginx (TLS termination, proxy_cache for /webdav) │
│ ├─ rclone rcd (mount daemon) │
│ ├─ mount-watcher.sh (rclone mount lifecycle) │
│ └─ meta-core (Go) ──► HTTP + SSE + WebDAV on :9000 │
│ + embedded SQLite (db/meta-core.db) │
└────────────────────────────────────────────────────────────────────────┘
│
┌────────────────────────────┴────────────────────────────┐
│ /meta-core (shared volume) │
│ locks/kv-leader.{lock,info} │
│ db/redis/ │
│ db/meta-core.db (+ -wal, -shm) │
│ services/*.json │
│ mounts/{mounts.json, errors/} │
│ watchers.json │
@@ -152,7 +154,7 @@ Container startup
▼
┌──────────────────────────────┐
│ supervisord starts: │
│ redis → nginx → rclone → │
│ nginx → rclone → │
│ meta-core Go binary │
└──────────────────────────────┘
```
@@ -167,10 +169,11 @@ gate wins. Therefore:
local hostname + IP + config and returns it on demand. There are no
`OnBecomeLeader` / `OnBecomeFollower` / `OnLeaderLost` callbacks; those
states do not exist inside the Go process.
- `LeaderLockInfo.RedisUrl` is intentionally empty in the response from
`/urls` — see `internal/leader/election.go:82`. PR D of
api-mediated-access.md removed direct Redis exposure; consumers route
metadata I/O through meta-core's HTTP API and SSE event streams.
- `LeaderLockInfo.RedisUrl` is always empty in the response from
`/urls` (kept for wire compatibility). PR D of
api-mediated-access.md removed direct Redis exposure, and since 2026-10
there is no Redis at all; consumers route metadata I/O through
meta-core's HTTP API and SSE event streams.
### Lock-info file
@@ -192,18 +195,18 @@ Other services read this file, then call `GET /urls` for full discovery
`metacore-app` is provisioned, its bash loop will acquire on next tick.
The dev compose runs a single instance; failover is "restart the
container."
- **Redis crash inside the container** → supervisord restarts redis. The
Go binary's connect loop retries up to 30 times (1s delay) at startup;
at steady state, individual operations fail and clients retry.
- **The store is in-process** — there is no separate database process to
crash. A meta-core crash is a process restart; SQLite's WAL recovers the
last committed transaction on reopen.
---
## Schema-Version Gate
Before serving any traffic, `cmd/meta-core/main.go:54-61` calls
`storage.EnsureSchemaVersion`. The sentinel lives in Redis at
`storage.EnsureSchemaVersion`. The sentinel lives in the store at
`meta-core:schema-version` (`internal/storage/schema_version.go`). The
gate refuses to boot when the existing Redis layout predates the
gate refuses to boot when the existing store layout predates the
UUID-rooted schema, so legacy `file:midhash256:*` data can't leak into
the current code paths. In alpha there is no automated migration —
operators wipe with `pnpm run clean:all` and restart.
@@ -216,20 +219,20 @@ operators wipe with `pnpm run clean:all` and restart.
```
┌────────────────────────────────────────────────────────────────────┐
│ METADATA (Redis) │
│ METADATA (embedded SQLite, kv table) │
│ │
│ Root: opaque UUIDv7 (Crockford Base32; ULID layout, 26 chars) │
│ Storage: flat string keys │
│ file:<uuid>/<property> → string value │
│ file:<uuid>/cids/<bareCid> → "true" (key-set) │
│ file:<uuid>/duplicates → SET of paths │
│ dup:<uuid>/<path> → "" (one per path) │
│ │
│ No stored canonical_cid — derived by rank on read. │
│ │
│ Aliases (reverse index, one Redis STRING each): │
│ Aliases (reverse index, one key each): │
│ cid:<bareCid> → <uuid> │
│ │
│ Index: file:__index__ → SET of all UUIDs │
│ Index: idx:roots/<uuid> → "" (one per root) │
│ │
│ Mintable without a local file (remote / deleted entries OK). │
│ Resolvable by any registered CID in O(1). │
@@ -317,7 +320,7 @@ Editor / KV / schema / snapshot
POST /api/snapshot/import
POST /api/snapshot/wipe
SSE event streams (HTTP-mirror of Redis Streams)
SSE event streams (HTTP-mirror of the store's event streams)
GET /api/events/files
GET /api/events/meta
@@ -337,11 +340,10 @@ Watchers
POST /api/watchers/{scan-all,reset-all}
(deprecated: /api/scan/trigger, /api/scan/status — see watcher/handlers.go:32)
Admin
POST /api/admin/migrate-dual-roots
POST /api/admin/sweep-literature-echoes (dry-run unless ?apply=true;
strips query echoes off domain=literature fileType=card records —
see storage/literature_echo_sweep.go)
(The one-shot Redis-era admin migrations — `/api/admin/migrate-dual-roots`,
`migrate-domain-screen`, `migrate-user-cids`, `sweep-literature-echoes` —
were removed with the 2026-10 SQLite switch; they had been run once on
watch.nsl.sh beforehand.)
WebDAV
/webdav/... (GET, PUT, DELETE, MKCOL, COPY, MOVE, PROPFIND)
@@ -369,12 +371,16 @@ See `docs/api-mediated-access.md` "Auth" for the rationale.
### SSE event streams — wire contract
`/api/events/files` and `/api/events/meta` are SSE mirrors of the
`file:events` and `meta:events` Redis Streams. The contract
`file:events` and `meta:events` streams, stored as `ev:<stream>/…` rows of
the kv table (`internal/kv/stream.go`). The contract
(`internal/api/sse_events.go`):
- One SSE event per Redis Stream entry.
- `id:` is the opaque `<ms>-<seq>` Redis Stream entry ID — clients echo
it back via `Last-Event-ID` on reconnect.
- One SSE event per stream entry.
- `id:` is the opaque `<ms>-<seq>` entry ID (the Redis Stream form, kept so
cursors persisted before the 2026-10 switch still compare) — clients echo
it back via `Last-Event-ID` on reconnect. Without `Last-Event-ID` the
stream starts at the head (resolved once at connect); `0` / `0-0` replay
from the beginning.
- `event:` is the `type` field of the underlying entry.
- `data:` is the JSON of the rest of the entry.
- Heartbeats: SSE comment `:keep-alive\n\n` every 30s of silence.
@@ -383,9 +389,9 @@ See `docs/api-mediated-access.md` "Auth" for the rationale.
`event: gap` payload before resuming from the oldest available entry.
External services use SSE with `Last-Event-ID` as the resume primitive.
A Redis consumer-group (XREADGROUP / XACK) exists *internally* — the
`MetaPublisher` uses one to drain keyspace notifications — but it is not
the external contract.
`meta:events` entries are appended by the storage engine in the same
transaction as the change they describe; the Redis-era keyspace-notification
hop through `MetaPublisher` is gone.
### WebDAV URLs
@@ -415,7 +421,7 @@ WebDAV caching is handled by nginx's `proxy_cache` (see
| Single binary | Static compilation |
| Low resource | ~10-20 MB resident |
| HTTP server | stdlib `net/http` + gorilla/mux |
| Redis client | go-redis |
| Metadata store | embedded SQLite via `modernc.org/sqlite` (pure Go, CGO off) |
| WebDAV | `golang.org/x/net/webdav` |
### Repo layout
@@ -428,12 +434,13 @@ packages/meta-core/
│ ├── cid/ CID parsing + algorithm ranking
│ ├── config/ env-driven configuration
│ ├── discovery/ service registry + dead-service cleaner
│ ├── events/ meta:events stream publisher
│ ├── events/ boot-time meta:events republish
│ ├── kv/ SQLite kv engine + event streams
│ ├── leader/ LeaderInfoProvider (no election logic)
│ ├── mounts/ rclone mount manager + handlers + stats
│ ├── schema/ live per-field schema indexer
│ ├── snapshot/ snapshot export / import / wipe
│ ├── storage/ Redis wrapper + UUID + CID reverse index + schema gate
│ ├── storage/ flat-key records + search + UUID + CID reverse index + schema gate
│ ├── watcher/ scanner + dispatcher + midhash + state
│ ├── watchers/ polling-based watcher configs
│ └── webdav/ /webdav/* handler
@@ -454,8 +461,8 @@ packages/meta-core/
```
1. Load configuration from environment
2. Build LeaderInfoProvider (no election; bash gate already won)
3. Connect to local Redis (retry up to 30× / 1s)
4. EnsureSchemaVersion (abort if Redis layout is stale)
3. Open the SQLite store (META_CORE_DB_PATH or /meta-core/db/meta-core.db)
4. EnsureSchemaVersion (abort if the store layout is stale)
5. Start service discovery + dead-service cleaner
6. NewServer + Server.Start (boots watcher, mounts, WebDAV, API)
7. On first connection: start MetaPublisher + schema Indexer,
@@ -466,9 +473,9 @@ packages/meta-core/
### Process management
Redis is not spawned by the Go binary; supervisord starts it before
meta-core (`docker/supervisord.conf`). The Go binary connects to
`localhost:6379` and retries until Redis is reachable.
There is no database process: the store is embedded in the Go binary
(`docker/supervisord.conf` runs meta-core, nginx and rclone). Until
2026-10 supervisord also ran `redis-server` on `localhost:6379`.
---
@@ -476,15 +483,16 @@ meta-core (`docker/supervisord.conf`). The Go binary connects to
### Container restart
Supervisord restarts crashed processes (meta-core, redis, nginx, rclone)
Supervisord restarts crashed processes (meta-core, nginx, rclone)
individually. The bash flock is held by supervisord's process tree, so
internal crashes don't trigger re-election.
### Redis crash
### Store durability
Supervisord restarts redis. The Go binary's connect retry handles the
window. Persistence is RDB + AOF on the `/meta-core` volume, so no data is
lost across restarts.
The SQLite file runs in WAL mode with `synchronous=NORMAL`: every
committed transaction survives a meta-core process crash. A record write — fields, root index, CID aliases,
FTS row and its events — is one transaction, so a crash never leaves a
half-written record.
### Container crash
@@ -517,7 +525,10 @@ via `/health`; metadata I/O and SSE event streams stop.
| `SERVICE_VERSION` | `1.0.0` | Reported in `/status`. |
| `API_PORT` | `8180` | External port baked into `baseUrl`. |
| `BASE_URL` | _empty_ | Overrides constructed `baseUrl` (HTTPS perimeter). |
| `REDIS_PORT` | `6379` | Local Redis port. |
| `META_CORE_DB_PATH` | `<META_CORE_PATH>/db/meta-core.db` | SQLite database file. Dev stacks set `/meta-core-db/meta-core.db` (named volume — SQLite WAL is unsafe on the 9p `/meta-core` bind of Windows dev boxes). |
| `META_CORE_DB_CACHE_MB` | `64` | Page cache of the writer connection. |
| `META_CORE_DB_READER_CACHE_MB` | `16` | Page cache per reader connection. |
| `META_CORE_DB_READERS` | `8` | Reader connection pool size. |
| `META_CORE_HTTP_PORT` | `9000` | Go HTTP+SSE+WebDAV port. |
| `META_CORE_HTTP_HOST` | `0.0.0.0` | HTTP bind address. |
| `HEALTH_CHECK_INTERVAL_MS` | `5000` | Internal health-loop cadence. |
@@ -536,7 +547,7 @@ via `/health`; metadata I/O and SSE event streams stop.
|---|---|
| `locks/kv-leader.lock` | Flock target. |
| `locks/kv-leader.info` | Leader API URL (plaintext). |
| `db/redis/{dump.rdb, appendonly.aof}` | Redis persistence. |
| `db/meta-core.db` (+ `-wal`, `-shm`) | Metadata store (SQLite). Replaced `db/redis/` in 2026-10. |
| `services/*.json` | Service registry. |
| `mounts/mounts.json` | rclone mount configuration. |
| `mounts/errors/{id}.error` | Mount error logs. |
@@ -552,5 +563,5 @@ and file access for MetaMesh into one container with a typed HTTP+SSE
surface. The bash flock gate guarantees single-leader semantics without
an external consensus system. UUID-rooted storage with CID aliasing
treats all content hashes symmetrically and makes any-CID resolution
O(1). API-mediated access ensures no other service ever needs to speak
Redis.
a single lookup. API-mediated access ensures no other service ever touches
the store.