780 lines
36 KiB
Markdown
780 lines
36 KiB
Markdown
# Meta-Core Architecture
|
||
|
||
> **meta-core v2.0.0 — the claims model.** Implementation basis:
|
||
> [`docs/implementation/metacore-claims-model.md`](../../../docs/implementation/metacore-claims-model.md)
|
||
> (meta-root). Where this file and that one disagree, that one wins.
|
||
|
||
## Document Purpose
|
||
|
||
This document describes the architectural design of **meta-core**, the
|
||
standalone container that owns the metadata store (two embedded SQLite
|
||
databases since v2.0.0 — *truth* and *cache*; a single SQLite file in
|
||
v1.x; Redis until 2026-10), the leader-info file, the HTTP+SSE
|
||
metadata surface, and a WebDAV file server. It covers the leader gate,
|
||
the claims data model and its write/apply pipeline, API design, the SSE
|
||
event surface, and the integration contract with other MetaMesh services.
|
||
|
||
For the engine design (SQLite, group commit, measurements), see
|
||
[metacore-storage-engine.md](../../../docs/project-architecture/metacore-storage-engine.md);
|
||
for the on-disk layout (tables of both databases), see
|
||
[metadata-storage-structure.md](metadata-storage-structure.md). For the
|
||
external-access design (store lockdown, SSE wire contract, auth rules),
|
||
see [api-mediated-access.md](api-mediated-access.md). For the UUID-rooted
|
||
storage rationale, see [uuid-rooted-metadata.md](uuid-rooted-metadata.md).
|
||
|
||
---
|
||
|
||
## System Overview
|
||
|
||
MetaMesh is a collection of services that need a shared, ordered, durable
|
||
view of file metadata:
|
||
|
||
| Service | Language | Role |
|
||
|---|---|---|
|
||
| **meta-sort** | TypeScript | Watches folders, runs plugins; each plugin signs its own claims and posts them to meta-core. |
|
||
| **meta-fuse** | TypeScript / Rust | Serves the virtual filesystem from metadata. |
|
||
| **meta-stremio** | Python | Stremio addon for HLS streaming. |
|
||
| **meta-dup** | TypeScript | Duplicate detection. |
|
||
| **meta-share** | Rust | Decentralised metadata sharing (libp2p / Kamilata). |
|
||
|
||
meta-core sits below all of these as the single owner of:
|
||
|
||
1. **The metadata store** — two embedded SQLite databases: **truth**
|
||
(`/meta-core/db/meta-core.db`, backed up) holding the signed claims, and
|
||
**cache** (`/meta-core/db-cache/cache.db`, never backed up) holding every
|
||
index and derived row — a v1-layout `kv` table (the merged view as
|
||
`file:<rid>/<field>` rows, lookup, counters, event streams, service
|
||
scratch KV) with its field index and FTS, plus the `claim_by_pid` and
|
||
`local_file` tables. (v2.0.0 is implemented, uncommitted and not yet
|
||
deployed as of 2026-10-25.)
|
||
2. **The leader-info file** — the one place services look to discover meta-core's URLs.
|
||
3. **The HTTP+SSE surface** — the only way external services touch the data plane.
|
||
4. **A WebDAV server** — file access for plugins and other services.
|
||
|
||
### Core design principle
|
||
|
||
> External services never touch the store. Every read, write, and event flows
|
||
> through meta-core's HTTP+SSE surface.
|
||
|
||
This is the contract that PR D of [api-mediated-access.md](api-mediated-access.md)
|
||
landed and that the rest of this document assumes. v2.0.0 adds a second:
|
||
|
||
> Every metadata value is a **signed vote** — `claim(uid, cid, key) → value`
|
||
> + `ts` + `sig`. The only write is `POST /api/claims`. meta-core verifies
|
||
> every claim and **never signs one**: it holds no writer key, and the
|
||
> signature is the write authentication.
|
||
|
||
---
|
||
|
||
## Architecture
|
||
|
||
### Standalone container topology
|
||
|
||
meta-core runs as a dedicated container (`metacore-app`). The dev compose
|
||
runs a single instance; the legacy multi-instance topology (sibling
|
||
followers) was deprecated. The flock primitive is retained as the
|
||
correctness anchor — it guarantees that at most one process writes the
|
||
database file and the writable state under `/meta-core`.
|
||
|
||
```
|
||
┌────────────────────────────────────────────────────────────────────────┐
|
||
│ metacore-app (standalone container) │
|
||
│ │
|
||
│ supervisord │
|
||
│ ├─ leader-election.sh (flock → write kv-leader.info → exec self) │
|
||
│ ├─ nginx (TLS termination, proxy_cache for /webdav) │
|
||
│ ├─ rclone rcd (mount daemon) │
|
||
│ ├─ mount-watcher.sh (rclone mount lifecycle) │
|
||
│ └─ meta-core (Go) ──► HTTP + SSE + WebDAV on :9000 │
|
||
│ + embedded SQLite: truth (db/meta-core.db) │
|
||
│ and cache (cache/cache.db) │
|
||
└────────────────────────────────────────────────────────────────────────┘
|
||
│
|
||
┌────────────────────────────┴────────────────────────────┐
|
||
│ /meta-core (shared volume) │
|
||
│ locks/kv-leader.{lock,info} │
|
||
│ db/meta-core.db (+ -wal, -shm) truth, backed up │
|
||
│ cache/cache.db (+ -wal, -shm) cache, never backed │
|
||
│ services/*.json │
|
||
│ mounts/{mounts.json, errors/} │
|
||
│ watchers.json │
|
||
│ cache/ (nginx proxy_cache) │
|
||
└─────────────────────────────────────────────────────────┘
|
||
▲
|
||
┌────────────────────────────┼────────────────────────────┐
|
||
│ │ │
|
||
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
|
||
│ meta-sort │ HTTP / SSE │ meta-fuse │ HTTP / SSE │ meta-stremio│
|
||
│ ├────────────►│ ├─────────────►│ │
|
||
│ │ │ │ │ │
|
||
└─────────────┘ └─────────────┘ └─────────────┘
|
||
```
|
||
|
||
### `/files` shared volume
|
||
|
||
Independently of `/meta-core`, every service mounts the `/files` volume
|
||
read-only (write access scoped to specific subtrees: `/files/plugin`,
|
||
`/files/corn` mounts). meta-core exposes this volume over WebDAV at
|
||
`/webdav/*` so plugins running in their own containers don't have to mount
|
||
it themselves.
|
||
|
||
```
|
||
/files
|
||
├── watch/ (host media, read-only)
|
||
├── test/ (test fixtures)
|
||
├── plugin/ (plugin output, read+write)
|
||
└── corn/ (rclone-managed mounts)
|
||
```
|
||
|
||
---
|
||
|
||
## Leader Gate
|
||
|
||
### Mechanism: bash + flock
|
||
|
||
Leader election is implemented in `docker/leader-election.sh`, not in Go.
|
||
The script:
|
||
|
||
1. Opens `${META_CORE_PATH}/locks/kv-leader.lock` on file descriptor 200.
|
||
2. Tries `flock -n 200`. On success it transitions to LEADER, writes the
|
||
API URL to `${META_CORE_PATH}/locks/kv-leader.info`, and `exec`s
|
||
supervisord — replacing itself in the process tree. When supervisord
|
||
exits, the kernel releases the flock automatically.
|
||
3. On failure (someone else holds the lock) it sleeps `ELECTION_RETRY_SECS`
|
||
(default `5s`) and retries. Followers loop in bash and never start the
|
||
Go binary.
|
||
|
||
```
|
||
Container startup
|
||
│
|
||
▼
|
||
┌──────────────────┐
|
||
│ leader-election │
|
||
│ .sh │
|
||
│ flock(kv-leader │
|
||
│ .lock) │
|
||
└────────┬─────────┘
|
||
│
|
||
┌────┴────┐
|
||
│ │
|
||
acquired blocked
|
||
│ │
|
||
▼ ▼
|
||
┌─────────┐ ┌──────────────┐
|
||
│ Write │ │ sleep N, │
|
||
│ kv- │ │ retry │
|
||
│ leader │ └──────────────┘
|
||
│ .info │
|
||
│ │
|
||
│ exec │
|
||
│ super- │
|
||
│ visord │
|
||
└─────────┘
|
||
│
|
||
▼
|
||
┌──────────────────────────────┐
|
||
│ supervisord starts: │
|
||
│ nginx → rclone → │
|
||
│ meta-core Go binary │
|
||
└──────────────────────────────┘
|
||
```
|
||
|
||
### Implications for the Go binary
|
||
|
||
The Go binary at `cmd/meta-core/main.go` is invoked only after the bash
|
||
gate wins. Therefore:
|
||
|
||
- `internal/leader/election.go` does **not** run an election loop. It
|
||
contains `LeaderInfoProvider`, which builds the leader info struct from
|
||
local hostname + IP + config and returns it on demand. There are no
|
||
`OnBecomeLeader` / `OnBecomeFollower` / `OnLeaderLost` callbacks; those
|
||
states do not exist inside the Go process.
|
||
- `LeaderLockInfo.RedisUrl` is always empty in the response from
|
||
`/urls` (kept for wire compatibility). PR D of
|
||
api-mediated-access.md removed direct Redis exposure, and since 2026-10
|
||
there is no Redis at all; consumers route metadata I/O through
|
||
meta-core's HTTP API and SSE event streams.
|
||
|
||
### Lock-info file
|
||
|
||
`/meta-core/locks/kv-leader.info` is a single plaintext line:
|
||
|
||
```
|
||
http://10.0.1.50:9000
|
||
```
|
||
|
||
Other services read this file, then call `GET /urls` for full discovery
|
||
(hostname, baseUrl, apiUrl, webdavUrl, webdavUrlInternal).
|
||
|
||
### Failure handling
|
||
|
||
- **meta-core process dies** → supervisord restarts it. The bash flock
|
||
loop does not need to participate — supervisord owns the lock through
|
||
its lifetime.
|
||
- **Container dies** → kernel releases the flock. If a standby
|
||
`metacore-app` is provisioned, its bash loop will acquire on next tick.
|
||
The dev compose runs a single instance; failover is "restart the
|
||
container."
|
||
- **The store is in-process** — there is no separate database process to
|
||
crash. A meta-core crash is a process restart; SQLite's WAL recovers the
|
||
last committed transaction on reopen.
|
||
|
||
---
|
||
|
||
## Schema-Version Gate
|
||
|
||
Before serving any traffic, `cmd/meta-core/main.go` opens both databases
|
||
and calls `storage.EnsureSchemaVersion` (before the cache catch-up). In
|
||
v2.0.0 the sentinel is the `schema-version = "3"` row of the truth
|
||
database's `meta` table. The gate:
|
||
|
||
- **explicitly refuses a v1.x store** — a truth file holding the v1 `kv`
|
||
table (`meta-core:schema-version = "2"`), which the default path still
|
||
points at — with a message naming `tools/claims-migrate`;
|
||
- refuses a store with claims but no sentinel, or any other version;
|
||
- stamps a fresh store with `"3"`.
|
||
|
||
So no pre-claims data leaks into the current code paths. The meta-core
|
||
**service binary contains no migration code**: watch.nsl.sh is converted
|
||
once by the separate one-off tool `packages/meta-core/tools/claims-migrate`
|
||
(not built into the image; implementation basis §11); every other box and
|
||
dev stack starts empty (operators wipe with `pnpm run clean:all` and
|
||
restart).
|
||
|
||
---
|
||
|
||
## Data Model
|
||
|
||
### The one concept: signed votes
|
||
|
||
```
|
||
claim(uid, cid, key) → value + ts + sig
|
||
```
|
||
|
||
- `uid` — the signer (a secp256k1 public key, `z…` base58btc). One key per
|
||
writing container (each meta-sort plugin, each self-publishing feeder,
|
||
gateway core, meta-share, meta-search) and one per user.
|
||
- `cid` — the subject, a content CID; never a rid.
|
||
- `key` — the field (`METADATA_KEYS.md`); key-set members live inside the
|
||
key (`genres/Drama`, `cids/<cid>`).
|
||
- `value` — text (public), bytes (signed + encrypted envelope) or null
|
||
(retraction, kept as a tombstone).
|
||
- `ts` — milliseconds; it **is** the version and only increases per
|
||
`(uid, cid, key)`.
|
||
- `sig` — MMUR2: secp256k1 ECDSA (compact 64 bytes) over SHA-256 of
|
||
`"MMUR2\n" ‖ canonical CBOR [cid, uid, key, value, ts]`.
|
||
|
||
The old shared record (`file:<rid>/<field>`) and the old User Data Layer
|
||
store (`udl:rec:*`) both collapse into this shape. The store distinguishes
|
||
only plaintext (signed) from ciphertext (signed + encrypted); "personal vs
|
||
shared" is app logic.
|
||
|
||
### Two databases
|
||
|
||
```
|
||
┌──────────────────────────────────────────────────────────────────────┐
|
||
│ TRUTH /meta-core/db/meta-core.db (backed up; PKs only) │
|
||
│ │
|
||
│ principal(pid, uid) uid ↔ local surrogate pid │
|
||
│ claim(cid, key, pid → value, ts, sig, seq) │
|
||
│ the votes; PK (cid, key, pid) │
|
||
│ cid_group(cid → rid) hidden local record identity │
|
||
│ meta(name → value) schema-version, last_seq │
|
||
└──────────────────────────────────────────────────────────────────────┘
|
||
│ apply loop (seq order)
|
||
▼
|
||
┌──────────────────────────────────────────────────────────────────────┐
|
||
│ CACHE /meta-core/db-cache/cache.db (never backed up; rebuildable)│
|
||
│ │
|
||
│ kv(k → v) the v1 layout, every row derived: │
|
||
│ file:<rid>/<field> the merged view every read uses │
|
||
│ cid:<cid> cid → rid lookup │
|
||
│ idx:roots/ idx:fts/ idx:count/ idx:schema/ roots, FTS rowid, │
|
||
│ counters (search totals, stats) │
|
||
│ ev:<stream>/…, evhwm:… /api/events/meta + /files backlog │
|
||
│ meta:applied_seq, meta:rebuilt_at apply-loop bookkeeping │
|
||
│ anything else /api/kv/value scratch KV │
|
||
│ kv_field_v (index), fts filter + free-text search │
|
||
│ claim_by_pid(pid, cid, key) "everything uid X wrote" (?as=, │
|
||
│ UDL facade, identity retraction) │
|
||
│ local_file(path → midhash, size, mtime) watcher facts, unsigned │
|
||
└──────────────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
Disaster recovery is "delete the cache, replay truth". `META_CORE_DB_PATH`
|
||
names the truth file, `META_CORE_CACHE_PATH` the cache file. The cache keeps
|
||
the v1 `kv` layout so every v1 reader (record reads, field index, FTS,
|
||
counters, SSE + snapshot) runs unchanged on derived rows; `nsrc` is computed
|
||
by the merge (`POST /api/merge`), not stored. Full schema: implementation
|
||
basis §5.
|
||
|
||
### Records: cids, groups and the hidden rid
|
||
|
||
A vote never mentions a rid. A **rid** (UUIDv7, 26-char Crockford, as in
|
||
v1.x) is a local name for a group of cids known to be the same content,
|
||
held in `cid_group`:
|
||
|
||
- A new cid with no link gets its own group and a fresh rid.
|
||
- A `cids/<other>` vote on cid X links X's group and `<other>`'s group; the
|
||
group with the older rid survives, the losing rid gets a `del` event per
|
||
key and the survivor `set` events (no alias table). Retracting the link
|
||
does **not** split the group (phase 1); the record view synthesises
|
||
`cids/<cid> = "true"` for every cid of the group from `cid_group`.
|
||
- Reading record R = every vote on every cid of R's group, merged per key;
|
||
within a group one signer counts once per key.
|
||
- rids are never signed and never travel; consumers that hold them
|
||
(meta-sort, meta-fuse, meta-stremio) keep working because `cid_group`
|
||
lives in truth.
|
||
|
||
### Merge
|
||
|
||
Per `(rid, key)`, over the group's votes, each signer's newest vote counted
|
||
once (`internal/storage/merge.go`, shared by the apply loop, `?as=` and
|
||
`POST /api/merge`):
|
||
|
||
1. the viewer's own live text vote (`?as=<uid>`, applied at read time,
|
||
never stored);
|
||
2. retractions and ciphertext drop out — only text merges;
|
||
3. most distinct signers for the normalised value (**trim + case-fold
|
||
only**, no NFC) — `nsrc` is that count (computed, not stored);
|
||
4. tie → oldest `ts`;
|
||
5. tie → byte-wise smallest value.
|
||
|
||
The stored value is the winning vote's exact text. Key-sets need no special
|
||
code (each member is a key, present while one vote supports it). A
|
||
different kind of fact gets a different key, so a guesser (e.g.
|
||
filename-parser) never competes with a lookup on the same key.
|
||
|
||
### Box-local file facts
|
||
|
||
`filePath`, size, `mtimeNano` and midhash are **not claims** — meta-core
|
||
cannot sign, and the disk can always be rescanned. The file watcher and
|
||
`POST /api/files/register` write them into the cache's `local_file` table
|
||
(and the midhash into `cid_group`); they are joined into the record view
|
||
unsigned and never reach the claims API or the swarm. When several local
|
||
paths hold the same content, `filePath` is the lexicographically first one
|
||
(size and mtime from the same row); a signed `sizeByte` claim wins over the
|
||
on-disk size.
|
||
|
||
### File bytes
|
||
|
||
```
|
||
Location: /files volume (shared across containers).
|
||
Access: path resolution via local_file, served via /webdav/*,
|
||
/file/{cid} (reverse-lookup), or /data/{hash}.
|
||
Path: filePath (from local_file) in the record view, relative to /files.
|
||
```
|
||
|
||
### History: UUID-rooted flat keys (v1.x)
|
||
|
||
Until v2.0.0 a record was a set of flat keys under an opaque UUID root —
|
||
`file:<uuid>/<property>`, `file:<uuid>/cids/<bareCid>` key-sets,
|
||
`dup:<uuid>/<path>`, a reverse index `cid:<bareCid> → <uuid>` and a root
|
||
index `idx:roots/<uuid>`, all in one kv table. Roots had previously been
|
||
content-hash tokens (`file:midhash256:abc/...`); UUID roots treated every
|
||
CID symmetrically as an alias. The rid of v2.0.0 is that UUID, kept hidden
|
||
and local; migrated records kept their UUIDs. See
|
||
[uuid-rooted-metadata.md](uuid-rooted-metadata.md) (the *why*) and the
|
||
History note of [metadata-storage-structure.md](metadata-storage-structure.md).
|
||
|
||
### Path resolution
|
||
|
||
`filePath` values are absolute paths under the `/files` volume (e.g.
|
||
`/files/watch/movies/Movie.2024.mkv`). The `/files` prefix is itself a
|
||
per-host detail — moving the volume only requires re-mounting it at the
|
||
same path inside the container.
|
||
|
||
---
|
||
|
||
## API Design
|
||
|
||
### Endpoint catalogue
|
||
|
||
The router is in `internal/api/server.go`. Endpoint set, by surface:
|
||
|
||
```
|
||
Bootstrap / discovery
|
||
GET /health
|
||
GET /status
|
||
GET /leader
|
||
GET /urls
|
||
GET /neighbors (alias: /api/neighbors)
|
||
GET /api/stats
|
||
|
||
Claims (the only metadata write path)
|
||
POST /api/claims signed claim batch → per-claim status
|
||
POST /api/merge {claims, as?} → {fields, nsrc, rejected}; no write
|
||
GET /api/meta/{cid}/claims {rid, cids, claims} of the cid's group (?key=, ?uid=; 404 unknown)
|
||
GET /api/metadata/{rid}/claims same, addressed by rid
|
||
|
||
Metadata reads (merged view + local_file; optional ?as=<uid>)
|
||
GET /meta
|
||
GET /meta/{hash}
|
||
GET /meta/{hash}/{key...}
|
||
|
||
CID-addressed (public, auth-bypassed at the perimeter)
|
||
GET /api/meta/{cid}
|
||
GET /api/file/{cid}/info
|
||
GET /file/{cid}
|
||
POST /file/cid
|
||
HEAD /data/{hash}
|
||
GET /data/{hash}/path
|
||
|
||
Editor / KV / schema
|
||
GET /api/metadata/hash-ids
|
||
GET /api/files/tuples
|
||
POST /api/files/register (fills local_file + cid_group)
|
||
GET /api/metadata/list
|
||
POST /api/metadata/search
|
||
POST /api/metadata/clear (wipes both databases)
|
||
GET /api/metadata/{hashId} (hashId = rid)
|
||
GET /api/metadata/{hashId}/property
|
||
GET /api/kv/{info,keys,tree,search,find} (read-only browser over the cache kv table)
|
||
{GET,PUT,DELETE} /api/kv/value (service scratch keys only; reserved prefixes refused)
|
||
GET /api/kv/key/{key...} (read-only)
|
||
GET /api/schema
|
||
POST /api/schema/rescan
|
||
|
||
User Data Layer (facade over claims; records are MMUR2)
|
||
{GET,PUT} /api/udl/record
|
||
PUT /api/udl/records
|
||
GET /api/udl/user/{uid}/{key/{key},cid/{cid},all,count}
|
||
GET /api/udl/cid/{cid}/{users,aggregate}
|
||
GET /api/udl/users/stats (every signer with live claims, plugin keys included)
|
||
|
||
Account-key identity service (kept in v2.0.0; removal is a separate ticket)
|
||
{GET,DELETE} /api/identity (DELETE signs + stores a retraction of every live claim first)
|
||
GET /api/identity/{accounts,aead-key}
|
||
POST /api/identity/{challenge,generate,import,reveal,sign,sign-batch}
|
||
(sign/sign-batch `format`: compact | der; default
|
||
compact for "MMUR2\n" payloads, DER otherwise)
|
||
|
||
Removed in v2.0.0 — answer 410 Gone, pointing to POST /api/claims
|
||
PUT/PATCH/DELETE /meta/{hash}
|
||
PUT/DELETE /meta/{hash}/{key...}
|
||
POST /meta/{hash}/_add/{key...}
|
||
PUT/DELETE /api/metadata/{hashId}
|
||
PUT /api/metadata/{hashId}/property
|
||
POST /api/metadata/batch
|
||
|
||
Removed in v2.0.0 — no snapshot format
|
||
GET /api/snapshot/export, POST /api/snapshot/{import,wipe}
|
||
|
||
SSE event streams (HTTP-mirror of the store's event streams)
|
||
GET /api/events/files
|
||
GET /api/events/meta
|
||
|
||
Mounts (rclone-only)
|
||
GET /api/mounts
|
||
POST /api/mounts
|
||
GET /api/mounts/{id}
|
||
PUT /api/mounts/{id} (also PATCH)
|
||
DELETE /api/mounts/{id}
|
||
POST /api/mounts/{id}/{mount,unmount,safe-unmount,scan}
|
||
GET /api/mounts/rclone/remotes
|
||
|
||
Watchers
|
||
{GET,POST} /api/watchers
|
||
{GET,PUT,DELETE} /api/watchers/{id}
|
||
POST /api/watchers/{id}/{scan,reset}
|
||
POST /api/watchers/{scan-all,reset-all}
|
||
(deprecated: /api/scan/trigger, /api/scan/status — see watcher/handlers.go:32)
|
||
|
||
(The one-shot Redis-era admin migrations — `/api/admin/migrate-dual-roots`,
|
||
`migrate-domain-screen`, `migrate-user-cids`, `sweep-literature-echoes` —
|
||
were removed with the 2026-10 SQLite switch; they had been run once on
|
||
watch.nsl.sh beforehand.)
|
||
|
||
WebDAV
|
||
/webdav/... (GET, PUT, DELETE, MKCOL, COPY, MOVE, PROPFIND)
|
||
```
|
||
|
||
No `/metrics` endpoint is exposed. Observability is via container logs +
|
||
the `/health` and `/status` JSON.
|
||
|
||
### Auth perimeter
|
||
|
||
In the dev and prod stacks Caddy + an nginx-hash-lock sidecar enforce
|
||
OIDC auth in front of meta-core. The following paths bypass auth (so
|
||
peers and unauthenticated tooling can reach them):
|
||
|
||
- `/health`
|
||
- `/webdav/*`
|
||
- `/meta/*`
|
||
- `/file/cid`
|
||
- `/api/file/{cid}` and `/api/meta/{cid}`
|
||
- (`/api/events/files`, `/api/events/meta` are also auth-bypassed but
|
||
**should not** be exposed publicly via Caddy; they're inside-only.)
|
||
|
||
Since v2.0.0 `/meta/*` serves reads only — its write verbs answer `410
|
||
Gone` — so the bypass no longer exposes an unsigned write. Writes go to
|
||
`POST /api/claims`, where the claim signature (not the perimeter) is the
|
||
write authentication. `POST /api/claims`, `POST /api/merge` and
|
||
`GET /api/meta/{cid}/claims` are container-to-container routes, not routed
|
||
through Caddy, like the v1.x write routes were.
|
||
|
||
See `docs/api-mediated-access.md` "Auth" for the rationale.
|
||
|
||
### SSE event streams — wire contract
|
||
|
||
`/api/events/files` and `/api/events/meta` are SSE mirrors of the
|
||
`file:events` and `meta:events` streams, stored as `ev:<stream>/…` rows of
|
||
the kv table (`internal/kv/stream.go`). The contract
|
||
(`internal/api/sse_events.go`):
|
||
|
||
- One SSE event per stream entry.
|
||
- `id:` is the opaque `<ms>-<seq>` entry ID (the Redis Stream form, kept so
|
||
cursors persisted before the 2026-10 switch still compare) — clients echo
|
||
it back via `Last-Event-ID` on reconnect. Without `Last-Event-ID` the
|
||
stream starts at the head (resolved once at connect); `0` / `0-0` replay
|
||
from the beginning.
|
||
- `event:` is the `type` field of the underlying entry.
|
||
- `data:` is the JSON of the rest of the entry.
|
||
- Heartbeats: SSE comment `:keep-alive\n\n` every 30s of silence.
|
||
- On reconnect with `Last-Event-ID`, the handler resumes from the next
|
||
entry. If retention has trimmed past the cursor, the handler emits one
|
||
`event: gap` payload before resuming from the oldest available entry.
|
||
|
||
External services use SSE with `Last-Event-ID` as the resume primitive.
|
||
Since v2.0.0 `meta:events` describes changes of the **merged** view, same
|
||
`{type: set|del, key, ts}` shape: the cache engine appends an
|
||
`ev:meta:events/…` row for every change of a merged `file:` row, in the same
|
||
cache transaction as the change (both streams are v1 stream rows of the
|
||
cache `kv` table). A cache rebuild
|
||
emits no per-key events — consumers get `event: gap` and take a snapshot;
|
||
event IDs stay time-based and keep rising. *History:* v1.x appended them in
|
||
the same transaction as the `file:` key change; the Redis-era
|
||
keyspace-notification hop through `MetaPublisher` is long gone.
|
||
|
||
### Write path and apply loop
|
||
|
||
```
|
||
POST /api/claims ──► per claim, before the writer:
|
||
0. validate (required fields, reserved keys, ts range) → else bad_request
|
||
1. same ts + sig as stored? → unchanged (no verify)
|
||
2. same value as stored (any ts)? → unchanged; stored row + older ts kept
|
||
3. stored ts ≥ incoming ts? → stale
|
||
4. verify MMUR2 signature → else bad_signature
|
||
──► accepted claims, one group-committed truth transaction:
|
||
5. re-check 1–3 under the writer
|
||
6. uid → pid (principal; insert on first sight)
|
||
7. upsert claim (tombstone if null), seq = ++last_seq
|
||
8. cid_group: mint rid for unknown cid; text cids/* links groups
|
||
(a retraction never splits)
|
||
9. commit (last_seq in truth meta); answer once the cache applied it
|
||
──► apply loop, in seq order, into the cache:
|
||
sync cid:/cids/<cid>/idx:roots rows → claim_by_pid
|
||
→ re-merge (rid, key) from truth PK range reads
|
||
→ set/delete file:<rid>/<key> (+ local_file for filePath/sizeByte/mtimeNano)
|
||
⇒ meta:events row + counter delta in the same tx
|
||
→ FTS row on a haystack change → meta:applied_seq
|
||
```
|
||
|
||
The verifier lives in the write path (secp256k1 from `internal/identity`);
|
||
meta-core holds **no** writer key and never signs a claim. Response: one
|
||
status per claim — `accepted` · `unchanged` · `stale` · `bad_signature` ·
|
||
`bad_request`. Request/response shapes and the 410 contract for removed
|
||
routes: [api-mediated-access.md](api-mediated-access.md) "Write contract —
|
||
signed claims".
|
||
|
||
### WebDAV URLs
|
||
|
||
`/urls` exposes two WebDAV URLs (`internal/leader/election.go`):
|
||
|
||
| Field | Purpose | Construction |
|
||
|---|---|---|
|
||
| `webdavUrl` | External access | `{BASE_URL}/webdav` (or `http://{ip}:{API_PORT}/webdav` if `BASE_URL` is unset) |
|
||
| `webdavUrlInternal` | Container-to-container | `http://{hostname}:{META_CORE_HTTP_PORT}/webdav` |
|
||
|
||
Container plugins should use `webdavUrlInternal`. Browsers / external
|
||
clients use `webdavUrl` (which goes through nginx + Caddy).
|
||
|
||
WebDAV caching is handled by nginx's `proxy_cache` (see
|
||
`docker/nginx.conf`), not by a Go cache layer. There is no
|
||
`internal/cache` package.
|
||
|
||
---
|
||
|
||
## Implementation
|
||
|
||
### Why Go
|
||
|
||
| Requirement | Go advantage |
|
||
|---|---|
|
||
| Language-agnostic | HTTP API, no runtime dependency on callers |
|
||
| Single binary | Static compilation |
|
||
| Low resource | ~10-20 MB resident |
|
||
| HTTP server | stdlib `net/http` + gorilla/mux |
|
||
| Metadata store | embedded SQLite via `modernc.org/sqlite` (pure Go, CGO off) — truth + cache databases |
|
||
| Claim verification | secp256k1 ECDSA (`internal/identity`) |
|
||
| WebDAV | `golang.org/x/net/webdav` |
|
||
|
||
### Repo layout
|
||
|
||
```
|
||
packages/meta-core/
|
||
├── cmd/meta-core/main.go entry point
|
||
├── internal/
|
||
│ ├── api/ HTTP server + handlers + SSE
|
||
│ ├── cid/ CID parsing + algorithm ranking
|
||
│ ├── config/ env-driven configuration
|
||
│ ├── discovery/ service registry + dead-service cleaner
|
||
│ ├── identity/ secp256k1 (claim verification; account-key service)
|
||
│ ├── kv/ SQLite engine (single writer, group commit) + event streams
|
||
│ ├── leader/ LeaderInfoProvider (no election logic)
|
||
│ ├── mounts/ rclone mount manager + handlers + stats
|
||
│ ├── schema/ per-field schema from transactional counters
|
||
│ ├── storage/ claims write path, apply loop, merge, cid_group, search, schema gate
|
||
│ ├── watcher/ scanner + dispatcher + midhash + state
|
||
│ ├── watchers/ polling-based watcher configs
|
||
│ └── webdav/ /webdav/* handler
|
||
├── docker/
|
||
│ ├── entrypoint.sh
|
||
│ ├── leader-election.sh the actual leader election
|
||
│ ├── mount-watcher.sh
|
||
│ ├── nginx.conf
|
||
│ └── supervisord.conf
|
||
├── editor/ React metadata browser (Vite)
|
||
├── dashboard/ React status dashboard (Vite)
|
||
├── docs/
|
||
└── go.mod
|
||
```
|
||
|
||
### Startup sequence (`cmd/meta-core/main.go`)
|
||
|
||
```
|
||
1. Load configuration from environment
|
||
2. Build LeaderInfoProvider (no election; bash gate already won)
|
||
3. Create the cache folder; open the truth database (META_CORE_DB_PATH or
|
||
/meta-core/db/meta-core.db) and the cache (META_CORE_CACHE_PATH or
|
||
/meta-core/db-cache/cache.db) (storage.Connect)
|
||
4. EnsureSchemaVersion (abort on a v1 kv store or an unknown layout)
|
||
5. storage.Start — cache catch-up (meta:applied_seq ≥ last_seq → nothing; behind →
|
||
replay claims with seq > applied_seq, or a full
|
||
rebuild if a cids/* link is among them; no
|
||
cache → full rebuild), then the apply loop
|
||
6. Derived counters ready
|
||
7. Start service discovery + dead-service cleaner
|
||
8. NewServer + Server.Start (boots watcher, mounts, WebDAV, API)
|
||
9. Wait for SIGINT/SIGTERM
|
||
10. Shutdown in reverse order
|
||
|
||
(v1.x ran `EnsureCounters` and a background FTS rebuild here; in v2.0.0
|
||
counters and FTS are cache rows maintained with every merged-row change.)
|
||
```
|
||
|
||
### Process management
|
||
|
||
There is no database process: the store is embedded in the Go binary
|
||
(`docker/supervisord.conf` runs meta-core, nginx and rclone). Until
|
||
2026-10 supervisord also ran `redis-server` on `localhost:6379`.
|
||
|
||
---
|
||
|
||
## Failure Modes
|
||
|
||
### Container restart
|
||
|
||
Supervisord restarts crashed processes (meta-core, nginx, rclone)
|
||
individually. The bash flock is held by supervisord's process tree, so
|
||
internal crashes don't trigger re-election.
|
||
|
||
### Store durability
|
||
|
||
Both SQLite files run in WAL mode; truth with `synchronous=NORMAL` (every
|
||
committed transaction survives a meta-core process crash), the cache with
|
||
`synchronous=OFF` (it is rebuildable). A claim batch
|
||
commits to truth in one transaction; the apply loop then commits the
|
||
derived cache changes. **A crash between the two commits leaves the cache
|
||
behind, never wrong**: at boot, if the cache's `applied_seq` is below
|
||
truth's `last_seq`, meta-core scans truth's `claim` table for
|
||
`seq > applied_seq` (a full scan — there is no seq index; it only happens
|
||
after an unclean shutdown) and applies them — or rebuilds the cache in full
|
||
if one of them is a `cids/*` link.
|
||
|
||
### Cache loss
|
||
|
||
The cache is never backed up and is fully derived (except `local_file`,
|
||
which a rescan of `/files` refills, and the service scratch keys, whose
|
||
loss costs a refetch). A missing cache folder (no `meta:applied_seq`) means
|
||
meta-core rebuilds every derived row from truth and stamps
|
||
`meta:rebuilt_at`. A rebuild emits no per-key events;
|
||
SSE consumers get `event: gap` and re-snapshot.
|
||
|
||
### Container crash
|
||
|
||
Kernel releases the flock; a standby container (if any) can take over.
|
||
The dev compose runs a single instance, so the recovery path is "restart
|
||
the container."
|
||
|
||
### Schema mismatch
|
||
|
||
`EnsureSchemaVersion` refuses to boot. Operators wipe with
|
||
`pnpm run clean:all` (or equivalent) and restart, or — once, on
|
||
watch.nsl.sh — convert a v1.x store with `tools/claims-migrate`. This is an
|
||
alpha-stage hard-fail; the service never migrates automatically.
|
||
|
||
### Shared volume unavailable
|
||
|
||
All meta-core operations fail. Other services see meta-core as unhealthy
|
||
via `/health`; metadata I/O and SSE event streams stop.
|
||
|
||
---
|
||
|
||
## Configuration Reference
|
||
|
||
### Environment variables
|
||
|
||
| Variable | Default | Description |
|
||
|---|---|---|
|
||
| `META_CORE_PATH` | `/meta-core` | Shared volume root. |
|
||
| `FILES_PATH` | `/files` | Files volume root. |
|
||
| `SERVICE_NAME` | `meta-core` | Identity in the service registry. |
|
||
| `SERVICE_VERSION` | `1.0.0` | Reported in `/status`. |
|
||
| `API_PORT` | `8180` | External port baked into `baseUrl`. |
|
||
| `BASE_URL` | _empty_ | Overrides constructed `baseUrl` (HTTPS perimeter). |
|
||
| `META_CORE_DB_PATH` | `<META_CORE_PATH>/db/meta-core.db` | Truth database file (backed up). Dev stacks set `/meta-core-db/meta-core.db` (named volume — SQLite WAL is unsafe on the 9p `/meta-core` bind of Windows dev boxes). |
|
||
| `META_CORE_CACHE_PATH` | `/meta-core/db-cache/cache.db` | Cache database file (never backed up; rebuilt from truth when missing). Dev stacks keep it on a named volume too; store apps put the cache folder in Maison's `backup.exclude`. |
|
||
| `META_CORE_DB_CACHE_MB` | `64` | Page cache of the writer connection. |
|
||
| `META_CORE_DB_READER_CACHE_MB` | `16` | Page cache per reader connection. |
|
||
| `META_CORE_DB_READERS` | `8` | Reader connection pool size. |
|
||
| `META_CORE_HTTP_PORT` | `9000` | Go HTTP+SSE+WebDAV port. |
|
||
| `META_CORE_HTTP_HOST` | `0.0.0.0` | HTTP bind address. |
|
||
| `HEALTH_CHECK_INTERVAL_MS` | `5000` | Internal health-loop cadence. |
|
||
| `HEARTBEAT_INTERVAL_MS` | `30000` | Service registry heartbeat. |
|
||
| `STALE_THRESHOLD_MS` | `60000` | Age past which a registry entry is stale. |
|
||
| `CLEANUP_INTERVAL_MS` | `600000` | Dead-service cleaner cadence. |
|
||
| `DEAD_SERVICE_THRESHOLD_MS` | `600000` | Age past which a stale entry is deleted. |
|
||
| `WATCH_INTERVAL_MS` | `1000` | Watcher poll cadence. |
|
||
| `DEBOUNCE_MS` | `30000` | File-event debounce window. |
|
||
| `ENABLE_FILE_WATCHER` | `true` | Disable to suppress the in-process watcher. |
|
||
| `ELECTION_RETRY_SECS` | `5` | Bash flock retry interval (consumed by `docker/leader-election.sh`, not the Go binary). |
|
||
|
||
### File paths under `/meta-core`
|
||
|
||
| Path | Purpose |
|
||
|---|---|
|
||
| `locks/kv-leader.lock` | Flock target. |
|
||
| `locks/kv-leader.info` | Leader API URL (plaintext). |
|
||
| `db/meta-core.db` (+ `-wal`, `-shm`) | Truth database (claims, principals, `cid_group`). Backed up. Replaced `db/redis/` in 2026-10; holds the claims schema since v2.0.0. |
|
||
| `db-cache/cache.db` (+ `-wal`, `-shm`) | Cache database (v1-layout `kv` table: merged view, lookup, counters, events, service scratch keys; field index, FTS, `claim_by_pid`, `local_file`). Never backed up. |
|
||
| `services/*.json` | Service registry. |
|
||
| `mounts/mounts.json` | rclone mount configuration. |
|
||
| `mounts/errors/{id}.error` | Mount error logs. |
|
||
| `watchers.json` | Polling watcher configuration. |
|
||
| `cache/` | nginx `proxy_cache` directory. |
|
||
|
||
---
|
||
|
||
## Summary
|
||
|
||
meta-core consolidates leader gating, metadata storage, event mirroring,
|
||
and file access for MetaMesh into one container with a typed HTTP+SSE
|
||
surface. The bash flock gate guarantees single-leader semantics without
|
||
an external consensus system. Metadata is a table of signed votes in a
|
||
backed-up truth database, merged by an apply loop into a rebuildable cache
|
||
that serves every read; meta-core verifies every claim and signs none.
|
||
CID groups behind a hidden local rid treat all content hashes
|
||
symmetrically. API-mediated access ensures no other service ever touches
|
||
the store.
|