# Meta-Core Architecture ## Document Purpose This document describes the architectural design of **meta-core**, the standalone container that owns the metadata store (an embedded SQLite file; Redis until 2026-10), the leader-info file, the HTTP+SSE metadata surface, and a WebDAV file server. It covers the leader gate, storage data model, API design, the SSE event surface, and the integration contract with other MetaMesh services. For the on-disk key layout, see [metadata-storage-structure.md](metadata-storage-structure.md). For the external-access design (store lockdown, SSE wire contract, auth rules), see [api-mediated-access.md](api-mediated-access.md). For the UUID-rooted storage rationale, see [uuid-rooted-metadata.md](uuid-rooted-metadata.md). --- ## System Overview MetaMesh is a collection of services that need a shared, ordered, durable view of file metadata: | Service | Language | Role | |---|---|---| | **meta-sort** | TypeScript | Watches folders, runs plugins, writes metadata via meta-core's HTTP API. | | **meta-fuse** | TypeScript / Rust | Serves the virtual filesystem from metadata. | | **meta-stremio** | Python | Stremio addon for HLS streaming. | | **meta-dup** | TypeScript | Duplicate detection. | | **meta-share** | Rust | Decentralised metadata sharing (libp2p / Kamilata). | meta-core sits below all of these as the single owner of: 1. **The metadata store** — an embedded SQLite file (`/meta-core/db/meta-core.db`) holding metadata and the event streams. 2. **The leader-info file** — the one place services look to discover meta-core's URLs. 3. **The HTTP+SSE surface** — the only way external services touch the data plane. 4. **A WebDAV server** — file access for plugins and other services. ### Core design principle > External services never touch the store. Every read, write, and event flows > through meta-core's HTTP+SSE surface. This is the contract that PR D of [api-mediated-access.md](api-mediated-access.md) landed and that the rest of this document assumes. --- ## Architecture ### Standalone container topology meta-core runs as a dedicated container (`metacore-app`). The dev compose runs a single instance; the legacy multi-instance topology (sibling followers) was deprecated. The flock primitive is retained as the correctness anchor — it guarantees that at most one process writes the database file and the writable state under `/meta-core`. ``` ┌────────────────────────────────────────────────────────────────────────┐ │ metacore-app (standalone container) │ │ │ │ supervisord │ │ ├─ leader-election.sh (flock → write kv-leader.info → exec self) │ │ ├─ nginx (TLS termination, proxy_cache for /webdav) │ │ ├─ rclone rcd (mount daemon) │ │ ├─ mount-watcher.sh (rclone mount lifecycle) │ │ └─ meta-core (Go) ──► HTTP + SSE + WebDAV on :9000 │ │ + embedded SQLite (db/meta-core.db) │ └────────────────────────────────────────────────────────────────────────┘ │ ┌────────────────────────────┴────────────────────────────┐ │ /meta-core (shared volume) │ │ locks/kv-leader.{lock,info} │ │ db/meta-core.db (+ -wal, -shm) │ │ services/*.json │ │ mounts/{mounts.json, errors/} │ │ watchers.json │ │ cache/ (nginx proxy_cache) │ └─────────────────────────────────────────────────────────┘ ▲ ┌────────────────────────────┼────────────────────────────┐ │ │ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ meta-sort │ HTTP / SSE │ meta-fuse │ HTTP / SSE │ meta-stremio│ │ ├────────────►│ ├─────────────►│ │ │ │ │ │ │ │ └─────────────┘ └─────────────┘ └─────────────┘ ``` ### `/files` shared volume Independently of `/meta-core`, every service mounts the `/files` volume read-only (write access scoped to specific subtrees: `/files/plugin`, `/files/corn` mounts). meta-core exposes this volume over WebDAV at `/webdav/*` so plugins running in their own containers don't have to mount it themselves. ``` /files ├── watch/ (host media, read-only) ├── test/ (test fixtures) ├── plugin/ (plugin output, read+write) └── corn/ (rclone-managed mounts) ``` --- ## Leader Gate ### Mechanism: bash + flock Leader election is implemented in `docker/leader-election.sh`, not in Go. The script: 1. Opens `${META_CORE_PATH}/locks/kv-leader.lock` on file descriptor 200. 2. Tries `flock -n 200`. On success it transitions to LEADER, writes the API URL to `${META_CORE_PATH}/locks/kv-leader.info`, and `exec`s supervisord — replacing itself in the process tree. When supervisord exits, the kernel releases the flock automatically. 3. On failure (someone else holds the lock) it sleeps `ELECTION_RETRY_SECS` (default `5s`) and retries. Followers loop in bash and never start the Go binary. ``` Container startup │ ▼ ┌──────────────────┐ │ leader-election │ │ .sh │ │ flock(kv-leader │ │ .lock) │ └────────┬─────────┘ │ ┌────┴────┐ │ │ acquired blocked │ │ ▼ ▼ ┌─────────┐ ┌──────────────┐ │ Write │ │ sleep N, │ │ kv- │ │ retry │ │ leader │ └──────────────┘ │ .info │ │ │ │ exec │ │ super- │ │ visord │ └─────────┘ │ ▼ ┌──────────────────────────────┐ │ supervisord starts: │ │ nginx → rclone → │ │ meta-core Go binary │ └──────────────────────────────┘ ``` ### Implications for the Go binary The Go binary at `cmd/meta-core/main.go` is invoked only after the bash gate wins. Therefore: - `internal/leader/election.go` does **not** run an election loop. It contains `LeaderInfoProvider`, which builds the leader info struct from local hostname + IP + config and returns it on demand. There are no `OnBecomeLeader` / `OnBecomeFollower` / `OnLeaderLost` callbacks; those states do not exist inside the Go process. - `LeaderLockInfo.RedisUrl` is always empty in the response from `/urls` (kept for wire compatibility). PR D of api-mediated-access.md removed direct Redis exposure, and since 2026-10 there is no Redis at all; consumers route metadata I/O through meta-core's HTTP API and SSE event streams. ### Lock-info file `/meta-core/locks/kv-leader.info` is a single plaintext line: ``` http://10.0.1.50:9000 ``` Other services read this file, then call `GET /urls` for full discovery (hostname, baseUrl, apiUrl, webdavUrl, webdavUrlInternal). ### Failure handling - **meta-core process dies** → supervisord restarts it. The bash flock loop does not need to participate — supervisord owns the lock through its lifetime. - **Container dies** → kernel releases the flock. If a standby `metacore-app` is provisioned, its bash loop will acquire on next tick. The dev compose runs a single instance; failover is "restart the container." - **The store is in-process** — there is no separate database process to crash. A meta-core crash is a process restart; SQLite's WAL recovers the last committed transaction on reopen. --- ## Schema-Version Gate Before serving any traffic, `cmd/meta-core/main.go:54-61` calls `storage.EnsureSchemaVersion`. The sentinel lives in the store at `meta-core:schema-version` (`internal/storage/schema_version.go`). The gate refuses to boot when the existing store layout predates the UUID-rooted schema, so legacy `file:midhash256:*` data can't leak into the current code paths. In alpha there is no automated migration — operators wipe with `pnpm run clean:all` and restart. --- ## Data Model ### Two concepts: metadata vs. file bytes ``` ┌────────────────────────────────────────────────────────────────────┐ │ METADATA (embedded SQLite, kv table) │ │ │ │ Root: opaque UUIDv7 (Crockford Base32; ULID layout, 26 chars) │ │ Storage: flat string keys │ │ file:/ → string value │ │ file:/cids/ → "true" (key-set) │ │ dup:/ → "" (one per path) │ │ │ │ No stored canonical_cid — derived by rank on read. │ │ │ │ Aliases (reverse index, one key each): │ │ cid: → │ │ │ │ Index: idx:roots/ → "" (one per root) │ │ │ │ Mintable without a local file (remote / deleted entries OK). │ │ Resolvable by any registered CID in O(1). │ └────────────────────────────────────────────────────────────────────┘ ┌────────────────────────────────────────────────────────────────────┐ │ FILE BYTES │ │ │ │ Location: /files volume (shared across containers). │ │ Access: path resolution via metadata, served via /webdav/*, │ │ /file/{cid} (reverse-lookup), or /data/{hash}. │ │ Path stored as filePath in the metadata, relative to /files. │ └────────────────────────────────────────────────────────────────────┘ ``` ### Why UUID-rooted? Roots were previously content-hash tokens (`file:midhash256:abc/...`), which privileged one hash algorithm. UUID-rooted storage treats every CID symmetrically as an alias, makes `GET /api/meta/` O(1) for any registered CID, and lets new digest types be added without rewriting root keys. See [uuid-rooted-metadata.md](uuid-rooted-metadata.md) for the design rationale and [metadata-storage-structure.md](metadata-storage-structure.md) for the on-disk layout. ### Path resolution `filePath` values stored in metadata are absolute paths under the `/files` volume (e.g. `/files/watch/movies/Movie.2024.mkv`). The `/files` prefix is itself a per-host detail — moving the volume only requires re-mounting it at the same path inside the container. --- ## API Design ### Endpoint catalogue The router is in `internal/api/server.go`. Endpoint set, by surface: ``` Bootstrap / discovery GET /health GET /status GET /leader GET /urls GET /neighbors (alias: /api/neighbors) GET /api/stats Metadata (primary surface for other services) GET /meta GET /meta/{hash} PUT /meta/{hash} PATCH /meta/{hash} DELETE /meta/{hash} GET /meta/{hash}/{key...} PUT /meta/{hash}/{key...} DELETE /meta/{hash}/{key...} POST /meta/{hash}/_add/{key...} CID-addressed (public, auth-bypassed at the perimeter) GET /api/meta/{cid} GET /api/file/{cid}/info GET /file/{cid} POST /file/cid HEAD /data/{hash} GET /data/{hash}/path Editor / KV / schema / snapshot GET /api/metadata/hash-ids GET /api/files/tuples GET /api/metadata/list POST /api/metadata/search POST /api/metadata/batch POST /api/metadata/clear {GET,PUT,DELETE} /api/metadata/{hashId} {GET,PUT} /api/metadata/{hashId}/property GET /api/kv/{info,keys,tree,search,find,value} PUT /api/kv/value DELETE /api/kv/value GET /api/kv/key/{key...} GET /api/schema POST /api/schema/rescan GET /api/snapshot/export POST /api/snapshot/import POST /api/snapshot/wipe SSE event streams (HTTP-mirror of the store's event streams) GET /api/events/files GET /api/events/meta Mounts (rclone-only) GET /api/mounts POST /api/mounts GET /api/mounts/{id} PUT /api/mounts/{id} (also PATCH) DELETE /api/mounts/{id} POST /api/mounts/{id}/{mount,unmount,safe-unmount,scan} GET /api/mounts/rclone/remotes Watchers {GET,POST} /api/watchers {GET,PUT,DELETE} /api/watchers/{id} POST /api/watchers/{id}/{scan,reset} POST /api/watchers/{scan-all,reset-all} (deprecated: /api/scan/trigger, /api/scan/status — see watcher/handlers.go:32) (The one-shot Redis-era admin migrations — `/api/admin/migrate-dual-roots`, `migrate-domain-screen`, `migrate-user-cids`, `sweep-literature-echoes` — were removed with the 2026-10 SQLite switch; they had been run once on watch.nsl.sh beforehand.) WebDAV /webdav/... (GET, PUT, DELETE, MKCOL, COPY, MOVE, PROPFIND) ``` No `/metrics` endpoint is exposed. Observability is via container logs + the `/health` and `/status` JSON. ### Auth perimeter In the dev and prod stacks Caddy + an nginx-hash-lock sidecar enforce OIDC auth in front of meta-core. The following paths bypass auth (so peers and unauthenticated tooling can reach them): - `/health` - `/webdav/*` - `/meta/*` - `/file/cid` - `/api/file/{cid}` and `/api/meta/{cid}` - (`/api/events/files`, `/api/events/meta` are also auth-bypassed but **should not** be exposed publicly via Caddy; they're inside-only.) See `docs/api-mediated-access.md` "Auth" for the rationale. ### SSE event streams — wire contract `/api/events/files` and `/api/events/meta` are SSE mirrors of the `file:events` and `meta:events` streams, stored as `ev:/…` rows of the kv table (`internal/kv/stream.go`). The contract (`internal/api/sse_events.go`): - One SSE event per stream entry. - `id:` is the opaque `-` entry ID (the Redis Stream form, kept so cursors persisted before the 2026-10 switch still compare) — clients echo it back via `Last-Event-ID` on reconnect. Without `Last-Event-ID` the stream starts at the head (resolved once at connect); `0` / `0-0` replay from the beginning. - `event:` is the `type` field of the underlying entry. - `data:` is the JSON of the rest of the entry. - Heartbeats: SSE comment `:keep-alive\n\n` every 30s of silence. - On reconnect with `Last-Event-ID`, the handler resumes from the next entry. If retention has trimmed past the cursor, the handler emits one `event: gap` payload before resuming from the oldest available entry. External services use SSE with `Last-Event-ID` as the resume primitive. `meta:events` entries are appended by the storage engine in the same transaction as the change they describe; the Redis-era keyspace-notification hop through `MetaPublisher` is gone. ### WebDAV URLs `/urls` exposes two WebDAV URLs (`internal/leader/election.go`): | Field | Purpose | Construction | |---|---|---| | `webdavUrl` | External access | `{BASE_URL}/webdav` (or `http://{ip}:{API_PORT}/webdav` if `BASE_URL` is unset) | | `webdavUrlInternal` | Container-to-container | `http://{hostname}:{META_CORE_HTTP_PORT}/webdav` | Container plugins should use `webdavUrlInternal`. Browsers / external clients use `webdavUrl` (which goes through nginx + Caddy). WebDAV caching is handled by nginx's `proxy_cache` (see `docker/nginx.conf`), not by a Go cache layer. There is no `internal/cache` package. --- ## Implementation ### Why Go | Requirement | Go advantage | |---|---| | Language-agnostic | HTTP API, no runtime dependency on callers | | Single binary | Static compilation | | Low resource | ~10-20 MB resident | | HTTP server | stdlib `net/http` + gorilla/mux | | Metadata store | embedded SQLite via `modernc.org/sqlite` (pure Go, CGO off) | | WebDAV | `golang.org/x/net/webdav` | ### Repo layout ``` packages/meta-core/ ├── cmd/meta-core/main.go entry point ├── internal/ │ ├── api/ HTTP server + handlers + SSE │ ├── cid/ CID parsing + algorithm ranking │ ├── config/ env-driven configuration │ ├── discovery/ service registry + dead-service cleaner │ ├── kv/ SQLite kv engine + event streams │ ├── leader/ LeaderInfoProvider (no election logic) │ ├── mounts/ rclone mount manager + handlers + stats │ ├── schema/ per-field schema from transactional counters │ ├── snapshot/ snapshot export / import / wipe │ ├── storage/ flat-key records + search + UUID + CID reverse index + schema gate │ ├── watcher/ scanner + dispatcher + midhash + state │ ├── watchers/ polling-based watcher configs │ └── webdav/ /webdav/* handler ├── docker/ │ ├── entrypoint.sh │ ├── leader-election.sh the actual leader election │ ├── mount-watcher.sh │ ├── nginx.conf │ └── supervisord.conf ├── editor/ React metadata browser (Vite) ├── dashboard/ React status dashboard (Vite) ├── docs/ └── go.mod ``` ### Startup sequence (`cmd/meta-core/main.go`) ``` 1. Load configuration from environment 2. Build LeaderInfoProvider (no election; bash gate already won) 3. Open the SQLite store (META_CORE_DB_PATH or /meta-core/db/meta-core.db) 4. EnsureSchemaVersion (abort if the store layout is stale) 5. Start service discovery + dead-service cleaner 6. NewServer + Server.Start (boots watcher, mounts, WebDAV, API) 7. EnsureCounters (one-time build of the derived counters on a store that predates them) 8. Rebuild FTS rows in the background if the FTS table is empty 9. Wait for SIGINT/SIGTERM 10. Shutdown in reverse order ``` ### Process management There is no database process: the store is embedded in the Go binary (`docker/supervisord.conf` runs meta-core, nginx and rclone). Until 2026-10 supervisord also ran `redis-server` on `localhost:6379`. --- ## Failure Modes ### Container restart Supervisord restarts crashed processes (meta-core, nginx, rclone) individually. The bash flock is held by supervisord's process tree, so internal crashes don't trigger re-election. ### Store durability The SQLite file runs in WAL mode with `synchronous=NORMAL`: every committed transaction survives a meta-core process crash. A record write — fields, root index, CID aliases, FTS row and its events — is one transaction, so a crash never leaves a half-written record. ### Container crash Kernel releases the flock; a standby container (if any) can take over. The dev compose runs a single instance, so the recovery path is "restart the container." ### Schema mismatch `EnsureSchemaVersion` refuses to boot. Operators wipe with `pnpm run clean:all` (or equivalent) and restart. This is an alpha-stage hard-fail; there is no automated migration path. ### Shared volume unavailable All meta-core operations fail. Other services see meta-core as unhealthy via `/health`; metadata I/O and SSE event streams stop. --- ## Configuration Reference ### Environment variables | Variable | Default | Description | |---|---|---| | `META_CORE_PATH` | `/meta-core` | Shared volume root. | | `FILES_PATH` | `/files` | Files volume root. | | `SERVICE_NAME` | `meta-core` | Identity in the service registry. | | `SERVICE_VERSION` | `1.0.0` | Reported in `/status`. | | `API_PORT` | `8180` | External port baked into `baseUrl`. | | `BASE_URL` | _empty_ | Overrides constructed `baseUrl` (HTTPS perimeter). | | `META_CORE_DB_PATH` | `/db/meta-core.db` | SQLite database file. Dev stacks set `/meta-core-db/meta-core.db` (named volume — SQLite WAL is unsafe on the 9p `/meta-core` bind of Windows dev boxes). | | `META_CORE_DB_CACHE_MB` | `64` | Page cache of the writer connection. | | `META_CORE_DB_READER_CACHE_MB` | `16` | Page cache per reader connection. | | `META_CORE_DB_READERS` | `8` | Reader connection pool size. | | `META_CORE_HTTP_PORT` | `9000` | Go HTTP+SSE+WebDAV port. | | `META_CORE_HTTP_HOST` | `0.0.0.0` | HTTP bind address. | | `HEALTH_CHECK_INTERVAL_MS` | `5000` | Internal health-loop cadence. | | `HEARTBEAT_INTERVAL_MS` | `30000` | Service registry heartbeat. | | `STALE_THRESHOLD_MS` | `60000` | Age past which a registry entry is stale. | | `CLEANUP_INTERVAL_MS` | `600000` | Dead-service cleaner cadence. | | `DEAD_SERVICE_THRESHOLD_MS` | `600000` | Age past which a stale entry is deleted. | | `WATCH_INTERVAL_MS` | `1000` | Watcher poll cadence. | | `DEBOUNCE_MS` | `30000` | File-event debounce window. | | `ENABLE_FILE_WATCHER` | `true` | Disable to suppress the in-process watcher. | | `ELECTION_RETRY_SECS` | `5` | Bash flock retry interval (consumed by `docker/leader-election.sh`, not the Go binary). | ### File paths under `/meta-core` | Path | Purpose | |---|---| | `locks/kv-leader.lock` | Flock target. | | `locks/kv-leader.info` | Leader API URL (plaintext). | | `db/meta-core.db` (+ `-wal`, `-shm`) | Metadata store (SQLite). Replaced `db/redis/` in 2026-10. | | `services/*.json` | Service registry. | | `mounts/mounts.json` | rclone mount configuration. | | `mounts/errors/{id}.error` | Mount error logs. | | `watchers.json` | Polling watcher configuration. | | `cache/` | nginx `proxy_cache` directory. | --- ## Summary meta-core consolidates leader gating, metadata storage, event mirroring, and file access for MetaMesh into one container with a typed HTTP+SSE surface. The bash flock gate guarantees single-leader semantics without an external consensus system. UUID-rooted storage with CID aliasing treats all content hashes symmetrically and makes any-CID resolution a single lookup. API-mediated access ensures no other service ever touches the store.