metastar/meta-core
4
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

replace Redis with an embedded SQLite string->string KV store (group commit, in-tx events, SQL search)
Build and Push Docker Image / build (push) Successful in 2m44s
Details

Browse Source
This commit is contained in:
Worph
2026-10-19 20:51:00 +00:00
parent c53146e514
commit dc5cbdfb9b
80 changed files with 4601 additions and 6077 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+61 -56
View File
@@ -1,6 +1,7 @@
# meta-core
Standalone Go service that owns Redis, the HTTP+SSE metadata surface, the
Standalone Go service that owns the metadata store (an embedded SQLite file),
the HTTP+SSE metadata surface, the
User Data Layer store and signing keystore, and a WebDAV file server for the
MetaMesh dev/prod stack.
@@ -10,19 +11,19 @@ meta-core runs as its own container (`metacore-app`). Other MetaMesh services
(meta-sort, meta-fuse, meta-stremio, meta-dup, meta-search, meta-share,
meta-gateway, meta-watch, …) locate it over UDP multicast (beacon v2 —
its `metamesh.core` advertise carries the full `/urls` payload) and then call its
HTTP API. `META_CORE_URL` on a client pins it instead. Redis is not published
outside the container — all metadata I/O is mediated by the HTTP/SSE API
HTTP API. `META_CORE_URL` on a client pins it instead. The store is embedded
in the Go binary (no database port at all) — all metadata I/O is mediated by the HTTP/SSE API
(see [`docs/api-mediated-access.md`](docs/api-mediated-access.md)).
### Key capabilities
| Feature | Description |
|---------|-------------|
| **Redis-volume mutex** | Bash `flock` loop in `docker/leader-election.sh` only `exec`s supervisord on the winner, so two containers sharing one `/meta-core` volume can never run two Redis writers. It is a mutex, not discovery — nothing reads the lock. |
| **Redis owner** | Supervisord starts Redis with AOF + RDB persistence inside the container; meta-core connects to it on localhost. |
| **DB-file mutex** | Bash `flock` loop in `docker/leader-election.sh` only `exec`s supervisord on the winner, so two containers sharing one `/meta-core` volume can never run two writers on the database file. It is a mutex, not discovery — nothing reads the lock. |
| **Embedded SQLite store** | One schema-free `kv(k, v)` table in `/meta-core/db/meta-core.db` (pure-Go `modernc.org/sqlite`, WAL, `synchronous=NORMAL`). One writer goroutine with group commit; a record write (fields + root index + CID aliases + FTS row + events) is atomic. Replaced Redis in 2026-10 (Redis + the in-memory search copy needed ~6.6 GB on watch.nsl.sh's 169k records; the SQLite file is ~2.5 GB on disk with a bounded page cache). |
| **HTTP API** | Typed REST surface (gorilla/mux) for metadata, KV browsing, snapshots, schema inference, mounts, watchers, neighbours, identity, UDL, and file access by CID. |
| **In-memory search index** | `POST /api/metadata/search` matches against an in-memory snapshot of every record, kept current by incremental refreshes (full reconcile every 30 min) — no Redis round-trip per record on the request path (`internal/api/search_index.go`). |
| **SSE event streams** | `/api/events/files` and `/api/events/meta` mirror the underlying Redis Streams so external services never speak Redis. |
| **Indexed search** | `POST /api/metadata/search` builds a superset of candidates in SQL — a partial expression index `kv_field` over (field name, value) of every record key, plus an FTS5 trigram table — then verifies each with the unchanged Go matchers. Fresh immediately, no in-memory copy (`internal/storage/search.go`, `internal/api/search_index.go`). |
| **SSE event streams** | `/api/events/files` and `/api/events/meta` mirror the store's `file:events` / `meta:events` streams so external services never touch the store. `meta:events` entries are written in the same transaction as the change. |
| **WebDAV server** | `/webdav/*` exposes the `/files` volume (read+write) for cross-service file access. nginx in front of meta-core handles caching. |
| **Service discovery** | beacon v2: UDP multicast advertise/listen on `239.255.99.1:9099`, neighbours served at `/neighbors`. The file-based registry under `/meta-core/services/*.json` it replaced is gone. |
| **File watcher / scanner** | Recursive scan + MidHash256 computation; emits events on `file:events`. Watched roots are POSTed watcher configs (`/meta-core/watchers.json`), not env vars. |
@@ -35,13 +36,15 @@ outside the container — all metadata I/O is mediated by the HTTP/SSE API
### Design characteristics
- **Single writer** — flock on the `/meta-core` volume; no external consensus.
- **HTTP-only externally** — Redis is never published by the compose stacks;
debug with `docker exec metacore-app redis-cli …`.
- **HTTP-only externally** — there is no database port; debug read-only with
`docker exec metacore-app sqlite3 -readonly /meta-core/db/meta-core.db …`
(dev stacks: `/meta-core-db/meta-core.db`).
- **Schema-gated boot** — `EnsureSchemaVersion` aborts startup when the
on-disk Redis layout predates the current build (alpha clean-wipe policy;
on-disk store layout predates the current build (alpha clean-wipe policy;
see `docs/uuid-rooted-metadata.md`).
- **Static binary** — CGO disabled, single Go binary plus a thin Alpine
image with `redis-server`, `rclone`, nginx, and supervisord.
- **Static binary** — CGO disabled (pure-Go SQLite), single Go binary plus a
thin Alpine image with `rclone`, nginx, supervisord and the `sqlite` CLI
(for debugging).
## Architecture
@@ -52,19 +55,19 @@ outside the container — all metadata I/O is mediated by the HTTP/SSE API
│ leader-election.sh (flock /meta-core/locks/kv-leader.lock, │
│ then exec supervisord) │
│ supervisord │
│ ├─ redis-server (AOF + RDB on /meta-core/db/redis) │
│ ├─ nginx :80 (dashboard, /editor, API proxy, │
│ │ proxy_cache for /webdav, rclone GUI /rclone) │
│ ├─ rclone rcd (mount daemon, RC API on :5572) │
│ ├─ mount-watcher.sh (rclone mount lifecycle) │
│ └─ meta-core (Go) ─► HTTP API + SSE + WebDAV on :9000 │
│ ─► embedded SQLite (db/meta-core.db) │
│ ─► UDP advertise/listen 239.255.99.1:9099 │
└──────────────────────────────────────────────────────────────────────────┘
│
┌──────────────────────┴──────────────────────┐
│ /meta-core (meta-core's own volume) │
│ locks/kv-leader.lock │
│ db/redis/ │
│ db/meta-core.db (+ -wal, -shm) │
│ identity/accounts/*.json │
│ mounts/{mounts.json, errors/} │
│ watchers.json │
@@ -73,7 +76,7 @@ outside the container — all metadata I/O is mediated by the HTTP/SSE API
meta-sort / meta-fuse / meta-stremio / meta-search / meta-share / …
│ 1. hear the metamesh.core advertise on UDP multicast (carries /urls)
└─ 2. HTTP + SSE (+ WebDAV) to metacore-app — never Redis, never the volume
└─ 2. HTTP + SSE (+ WebDAV) to metacore-app — never the store, never the volume
```
## Quick start
@@ -138,28 +141,32 @@ Environment variables (from `internal/config/config.go` unless noted):
| Variable | Default | Description |
|---|---|---|
| `META_CORE_PATH` | `/meta-core` | meta-core's own volume root (lock, Redis data, identity, mounts, watchers, cache). No other service mounts it. |
| `META_CORE_PATH` | `/meta-core` | meta-core's own volume root (lock, SQLite DB, identity, mounts, watchers, cache). No other service mounts it. |
| `FILES_PATH` | `/files` | Files volume root. |
| `SERVICE_NAME` | `meta-core` | Name in the beacon v2 advertise. |
| `SERVICE_VERSION` | `1.0.0` | Reported via `/status`. |
| `API_PORT` | `8180` | External port baked into the constructed `baseUrl` when `BASE_URL` is unset. |
| `BASE_URL` | _empty_ | Overrides the constructed `baseUrl` (the Caddy/HTTPS perimeter URL). |
| `META_CORE_PUBLIC_URL` | _empty_ | Browser-facing nav URL in the announce; wins over `BASE_URL` (e.g. a debug-direct port with no Caddy). |
| `REDIS_PORT` | `6379` | Local Redis port. |
| `META_CORE_DB_PATH` | `<META_CORE_PATH>/db/meta-core.db` | SQLite database file. Dev stacks set `/meta-core-db/meta-core.db` on a named volume (the `/meta-core` bind is 9p on Windows dev boxes and SQLite WAL is unsafe there). |
| `META_CORE_DB_CACHE_MB` | `64` | Page cache of the writer connection (`internal/storage/client.go`). |
| `META_CORE_DB_READER_CACHE_MB` | `16` | Page cache per reader connection. |
| `META_CORE_DB_READERS` | `8` | Reader connection pool size. |
| `META_CORE_HTTP_PORT` | `9000` | Go HTTP+SSE+WebDAV port. |
| `META_CORE_HTTP_HOST` | `0.0.0.0` | HTTP bind. |
| `ENABLE_FILE_WATCHER` | `true` | Disable to suppress the in-process watcher entirely (`POST /api/files/register` stays available). |
| `ENABLE_UDP_DISCOVERY` | `true` | beacon v2 advertise + listen. |
| `BEACON_GROUP` / `BEACON_PORT` | `239.255.99.1` / `9099` | Multicast group and UDP port. |
| `BEACON_INTERVAL_MS` | `10000` | Unsolicited advertise interval. |
| `META_CORE_INDEX_EXCLUDE_PREFIXES` | `categories/newznab/` | Comma-separated field prefixes left out of the in-memory search index; set empty to index everything (`internal/storage/client.go`). |
| `ELECTION_RETRY_SECS` | `5` | flock retry interval (read by `docker/leader-election.sh`, not the Go binary). |
`HEALTH_CHECK_INTERVAL_MS`, `HEARTBEAT_INTERVAL_MS`, `STALE_THRESHOLD_MS`,
`CLEANUP_INTERVAL_MS`, `DEAD_SERVICE_THRESHOLD_MS`, `WATCH_INTERVAL_MS` and
`DEBOUNCE_MS` are still parsed by `internal/config/config.go` but nothing reads
them (leftovers of the file-based registry); setting them has no effect. There
is no `META_CORE_REDIS_PORT` / `META_CORE_SERVICE_NAME` /
is no `REDIS_PORT` / `META_CORE_REDIS_PORT` / `META_CORE_INDEX_EXCLUDE_PREFIXES`
(gone with Redis and the in-memory search index; search results now include
`categories/newznab/*` like every record read) / `META_CORE_SERVICE_NAME` /
`META_CORE_LEADER_TIMEOUT`, and no `WATCH_PATHS` — watched folders are watcher
configs created via `/api/watchers`.
@@ -174,10 +181,10 @@ the in-container Go port `:9000`; from the host use
### Health, leader, URLs
```bash
GET /health # storage + role
GET /health # storage + role (`redis: bool` kept for compatibility = store open)
GET /status # version, uptime, etc.
GET /leader # current leader info (hostname, PID, timestamps)
GET /urls # baseUrl, apiUrl, webdavUrl, webdavUrlInternal (redisUrl is intentionally empty)
GET /urls # baseUrl, apiUrl, webdavUrl, webdavUrlInternal (redisUrl is always "", kept for compatibility)
GET /neighbors # neighbours heard over UDP (beacon v2)
GET /api/neighbors # same, under /api
GET /api/stats # dashboard counters (see below)
@@ -196,9 +203,9 @@ Every headline counter in one request, split by cost:
```jsonc
{
"records": 88709, // SCARD file:__index__ — metadata roots, file or not
"redisKeys": 3115447, // DBSIZE — the flat per-field keys dominate
"redisMemory": "588.74M",
"records": 88709, // count of idx:roots/* — metadata roots, file or not
"redisKeys": 3115447, // kv row count (name kept for compatibility; cached 60 s)
"redisMemory": "588.74M",// DB + WAL file size (name kept for compatibility)
"identities": 3, // signing accounts on disk
"files": { "count": 4864, "totalSize": 64587914424 }, // null until first sweep
"udlUsers": 5, // accounts holding User Data Layer records
@@ -207,11 +214,11 @@ Every headline counter in one request, split by cost:
}
```
The first four are O(1) commands and are live on every call. `files` and
`records` and `identities` are live on every call; `redisKeys` is cached 60 s. `files` and
`udlUsers` need a keyspace walk (~0.5s and ~3s respectively on an 88k-record
box), so they are served from a single-flight cache refreshed in the background
with a 60s TTL — the walk is **never** on the request path, because the
dashboard polls and ten open tabs must not cost ten Redis walks. Prefer this
dashboard polls and ten open tabs must not cost ten keyspace walks. Prefer this
over `/api/kv/info`, which does one `GET` per root (6s+) and sums `sizeByte`
across records that share a file.
@@ -266,10 +273,10 @@ PUT /api/metadata/{hashId}/property
GET /api/kv/info
GET /api/kv/keys
GET /api/kv/tree
GET /api/kv/tree # cursor is an opaque next-key ("0" when exhausted)
GET /api/kv/search
GET /api/kv/find
GET /api/kv/value
GET /api/kv/value # type is always "string" for an existing key
PUT /api/kv/value
DELETE /api/kv/value
GET /api/kv/key/{key...}
@@ -284,7 +291,8 @@ POST /api/snapshot/wipe
### SSE event streams
The HTTP-mirror of the Redis Streams. External services never read Redis
The HTTP-mirror of the store's event streams (Redis Streams until 2026-10;
IDs keep the `<ms>-<seq>` form). External services never read the store
directly — they subscribe here with `Last-Event-ID` for resume. See
[`docs/api-mediated-access.md`](docs/api-mediated-access.md) for the wire
contract (one event per stream entry, opaque IDs, heartbeats, gap
@@ -318,7 +326,7 @@ GET /api/identity/aead-key
Records arrive signed (private-tier ones encrypted) as opaque base64 CBOR;
writes are version-gated (stale → `409 stale_version`, or `accepted:false` per
cell in a batch). Redis model in `internal/storage/udl.go`.
cell in a batch). Storage model in `internal/storage/udl.go` (`udl:rec:<uid>/<cid>/<key>` → JSON `{record,version,ts,value?}`; index sets as one key per member).
```bash
GET /api/udl/record?uid&cid&key
@@ -366,24 +374,21 @@ The legacy `/api/scan/trigger` and `/api/scan/status` still respond but
return a deprecation pointer at the `/api/watchers/*` routes
(`internal/watcher/handlers.go:32-33`).
### Admin / WebDAV
### WebDAV
```bash
POST /api/admin/migrate-dual-roots # reunify stranded midhash-rooted entries
POST /api/admin/migrate-domain-screen # domain film|tv → screen, workForm backfill (idempotent)
POST /api/admin/migrate-user-cids # backfill udl:idx:user:<uid>:cids; until run, pre-index
# profiles export as EMPTY. Returns {fixed}
POST /api/admin/sweep-literature-echoes[?apply=true]
# strip query echoes off literature cards: anilistid on site cards,
# workcid/chapterNumber/volumeNumber/chapterStart/chapterEnd on any card.
# Dry-run unless apply=true; returns {dryRun, planned, fixed, byKey}
GET/PUT/DELETE /webdav/... # mounted on /files
```
The one-shot Redis-era admin migrations (`/api/admin/migrate-dual-roots`,
`migrate-domain-screen`, `migrate-user-cids`, `sweep-literature-echoes`) were
removed with the 2026-10 SQLite switch; they had been run once on
watch.nsl.sh before the migration.
No `/metrics` endpoint is exposed; observability is via container logs and
the `/health` / `/status` JSON.
## Redis-volume mutex
## DB-file mutex
`docker/leader-election.sh` is a bash flock loop, not a Go-level state
machine. When `flock` succeeds on `/meta-core/locks/kv-leader.lock`, the
@@ -392,8 +397,8 @@ retries every `ELECTION_RETRY_SECS` and stays dormant. When the process tree
dies, the kernel releases the lock. There is no follower path inside the Go
binary.
The lock is a **mutex, not service discovery** — it only stops two Redis
writers landing on one RDB/AOF. Nothing reads it, and the `kv-leader.info`
The lock is a **mutex, not service discovery** — it only stops two
writers landing on one database file (`db/meta-core.db`). Nothing reads it, and the `kv-leader.info`
file it used to publish is gone: siblings locate meta-core over UDP
(see [beacon-v2.md](../../docs/project-architecture/beacon-v2.md)).
`internal/leader` keeps its name for history, but only builds the `/urls`
@@ -402,8 +407,8 @@ and the UDP announce share.
## Metadata storage shape (one-line summary)
Roots are opaque UUIDv7 strings; every property is a separate Redis
STRING at `file:<uuid>/<property>`; every known CID is a key-set member
Roots are opaque UUIDv7 strings; every property is a separate row
of the `kv` table at `file:<uuid>/<property>`; every known CID is a key-set member
`file:<uuid>/cids/<bareCid>` plus a reverse-index entry `cid:<bareCid> → <uuid>`
(the canonical CID is derived by rank on read, never stored). Full details in
[`docs/metadata-storage-structure.md`](docs/metadata-storage-structure.md)
@@ -415,15 +420,16 @@ the repo-root [`METADATA_KEYS.md`](../../METADATA_KEYS.md).
| Package | Purpose |
|---------|---------|
| `cmd/meta-core` | Entry point; Redis connect + schema sentinel + UDP discovery + service wiring. |
| `cmd/meta-core` | Entry point; store open + schema sentinel + UDP discovery + service wiring. |
| `internal/config` | Env-driven configuration, path helpers. |
| `internal/leader` | `LeaderInfoProvider` — builds the `/urls` payload from hostname/IP/config; no election logic. |
| `internal/meshdisco` | beacon v2 node (advertise, probe, neighbour map) — the Go port of the spec. |
| `internal/storage` | Redis wrapper (`client.go`), UUIDv7 minting, CID reverse index (`cid_resolution.go`), schema sentinel, field indexes + search-index bulk reads, UDL store (`udl.go`, `udl_admin.go`), stats, one-shot migrations/sweeps. |
| `internal/kv` | Storage engine: SQLite `kv` table, single writer + group commit, reader pool, in-transaction change events (`kv.go`), event streams as rows (`stream.go`). |
| `internal/storage` | Flat-key records (`client.go`), UUIDv7 minting, CID reverse index (`cid_resolution.go`), schema sentinel, search over `kv_field` + FTS (`search.go`), UDL store (`udl.go`, `udl_admin.go`), stats. |
| `internal/cid` | CID parsing + rank ladder for canonical-CID selection (`cid-rank-vectors.json`, guarded by the meta-root `scripts/check-cid-vectors.sh`). |
| `internal/identity` | Multi-account secp256k1 keystore, challenge/proof-of-possession, signature verification. |
| `internal/events` | Keyspace-notification → `meta:events` stream publisher (`meta_publisher.go`). |
| `internal/api` | HTTP server + all handlers + SSE event endpoints + in-memory search index + `/api/stats`. |
| `internal/events` | Boot-time republish of every field into `meta:events` (`meta_publisher.go`); live events are written by `internal/kv`. |
| `internal/api` | HTTP server + all handlers + SSE event endpoints + search matchers + `/api/stats`. |
| `internal/schema` | Live per-field schema indexer (consumes `meta:events`). |
| `internal/snapshot` | Snapshot export / import / wipe. |
| `internal/watcher` | File system scanner, MidHash256 computation, dispatcher, state registry. |
@@ -436,14 +442,14 @@ the repo-root [`METADATA_KEYS.md`](../../METADATA_KEYS.md).
```
1. Load configuration from environment
2. Build LeaderInfoProvider (no election — the bash gate already won)
3. Connect to local Redis (retry up to 30× / 1s)
4. EnsureSchemaVersion — abort if the existing Redis layout is stale
3. Open the SQLite store (META_CORE_DB_PATH or /meta-core/db/meta-core.db)
4. EnsureSchemaVersion — abort if the existing store layout is stale
5. Start beacon v2 node (unless ENABLE_UDP_DISCOVERY=false)
6. Construct API server (identity keystore migration, watcher, watchers,
mounts, WebDAV)
7. Start API server + pollers + search-index refresh loop; start
MetaPublisher + schema Indexer, then warm field indexes and republish
existing metadata onto meta:events in the background
7. Start API server + pollers; start MetaPublisher + schema Indexer, then
republish existing metadata onto meta:events in the background
(5k events per transaction)
8. Wait for SIGINT/SIGTERM, shutdown in reverse order
```
@@ -452,13 +458,12 @@ the repo-root [`METADATA_KEYS.md`](../../METADATA_KEYS.md).
| Dependency | Purpose |
|------------|---------|
| `github.com/gorilla/mux` | HTTP router |
| `github.com/redis/go-redis/v9` | Redis client |
| `modernc.org/sqlite` | Embedded SQLite (pure Go, CGO off) |
| `github.com/google/uuid` | UUIDv7 minting |
| `github.com/decred/dcrd/dcrec/secp256k1/v4`, `golang.org/x/crypto`, `github.com/mr-tron/base58` | Identity keys, signing, AEAD key derivation |
| `golang.org/x/net` | WebDAV support |
| `github.com/alicebob/miniredis/v2` | In-memory Redis for tests |
Runtime requirements: Go 1.21+ at build time; the container ships Redis,
Runtime requirements: Go 1.21+ at build time; the container ships
rclone, nginx, and supervisord alongside the binary, plus the React dashboard
and metadata editor (built from `dashboard/` and `editor/` in the Dockerfile).