Files

T

Build and Push Docker Image / build (push) Successful in 5m1s

Details

v2.0.0: claims model - signed MMUR2 votes in a truth DB, merged view in a rebuildable cache DB; POST /api/claims, claims and merge routes, unsigned writes answer 410

2026-10-25 17:36:00 +00:00

36 KiB

Raw Permalink Blame History

Meta-Core Architecture

meta-core v2.0.0 — the claims model. Implementation basis: docs/implementation/metacore-claims-model.md (meta-root). Where this file and that one disagree, that one wins.

Document Purpose

This document describes the architectural design of meta-core, the standalone container that owns the metadata store (two embedded SQLite databases since v2.0.0 — truth and cache; a single SQLite file in v1.x; Redis until 2026-10), the leader-info file, the HTTP+SSE metadata surface, and a WebDAV file server. It covers the leader gate, the claims data model and its write/apply pipeline, API design, the SSE event surface, and the integration contract with other MetaMesh services.

For the engine design (SQLite, group commit, measurements), see metacore-storage-engine.md; for the on-disk layout (tables of both databases), see metadata-storage-structure.md. For the external-access design (store lockdown, SSE wire contract, auth rules), see api-mediated-access.md. For the UUID-rooted storage rationale, see uuid-rooted-metadata.md.

System Overview

MetaMesh is a collection of services that need a shared, ordered, durable view of file metadata:

Service	Language	Role
meta-sort	TypeScript	Watches folders, runs plugins; each plugin signs its own claims and posts them to meta-core.
meta-fuse	TypeScript / Rust	Serves the virtual filesystem from metadata.
meta-stremio	Python	Stremio addon for HLS streaming.
meta-dup	TypeScript	Duplicate detection.
meta-share	Rust	Decentralised metadata sharing (libp2p / Kamilata).

meta-core sits below all of these as the single owner of:

The metadata store — two embedded SQLite databases: truth (/meta-core/db/meta-core.db, backed up) holding the signed claims, and cache (/meta-core/db-cache/cache.db, never backed up) holding every index and derived row — a v1-layout kv table (the merged view as file:<rid>/<field> rows, lookup, counters, event streams, service scratch KV) with its field index and FTS, plus the claim_by_pid and local_file tables. (v2.0.0 is implemented, uncommitted and not yet deployed as of 2026-10-25.)
The leader-info file — the one place services look to discover meta-core's URLs.
The HTTP+SSE surface — the only way external services touch the data plane.
A WebDAV server — file access for plugins and other services.

Core design principle

External services never touch the store. Every read, write, and event flows through meta-core's HTTP+SSE surface.

This is the contract that PR D of api-mediated-access.md landed and that the rest of this document assumes. v2.0.0 adds a second:

Every metadata value is a signed vote — claim(uid, cid, key) → value

ts + sig. The only write is POST /api/claims. meta-core verifies every claim and never signs one: it holds no writer key, and the signature is the write authentication.

Architecture

Standalone container topology

meta-core runs as a dedicated container (metacore-app). The dev compose runs a single instance; the legacy multi-instance topology (sibling followers) was deprecated. The flock primitive is retained as the correctness anchor — it guarantees that at most one process writes the database file and the writable state under /meta-core.

┌────────────────────────────────────────────────────────────────────────┐
│                  metacore-app  (standalone container)                  │
│                                                                        │
│  supervisord                                                           │
│   ├─ leader-election.sh   (flock → write kv-leader.info → exec self)   │
│   ├─ nginx                (TLS termination, proxy_cache for /webdav)   │
│   ├─ rclone rcd           (mount daemon)                               │
│   ├─ mount-watcher.sh     (rclone mount lifecycle)                     │
│   └─ meta-core (Go) ──►   HTTP + SSE + WebDAV on :9000                 │
│                           + embedded SQLite: truth (db/meta-core.db)   │
│                             and cache (cache/cache.db)                 │
└────────────────────────────────────────────────────────────────────────┘
                                    │
       ┌────────────────────────────┴────────────────────────────┐
       │  /meta-core (shared volume)                             │
       │    locks/kv-leader.{lock,info}                          │
       │    db/meta-core.db (+ -wal, -shm)   truth, backed up    │
       │    cache/cache.db  (+ -wal, -shm)   cache, never backed │
       │    services/*.json                                      │
       │    mounts/{mounts.json, errors/}                        │
       │    watchers.json                                        │
       │    cache/  (nginx proxy_cache)                          │
       └─────────────────────────────────────────────────────────┘
                                    ▲
       ┌────────────────────────────┼────────────────────────────┐
       │                            │                            │
┌─────────────┐             ┌─────────────┐              ┌─────────────┐
│  meta-sort  │ HTTP / SSE  │  meta-fuse  │ HTTP / SSE   │ meta-stremio│
│             ├────────────►│             ├─────────────►│             │
│             │             │             │              │             │
└─────────────┘             └─────────────┘              └─────────────┘

`/files` shared volume

Independently of /meta-core, every service mounts the /files volume read-only (write access scoped to specific subtrees: /files/plugin, /files/corn mounts). meta-core exposes this volume over WebDAV at /webdav/* so plugins running in their own containers don't have to mount it themselves.

/files
  ├── watch/    (host media, read-only)
  ├── test/     (test fixtures)
  ├── plugin/   (plugin output, read+write)
  └── corn/     (rclone-managed mounts)

Leader Gate

Mechanism: bash + flock

Leader election is implemented in docker/leader-election.sh, not in Go. The script:

Opens ${META_CORE_PATH}/locks/kv-leader.lock on file descriptor 200.
Tries flock -n 200. On success it transitions to LEADER, writes the API URL to ${META_CORE_PATH}/locks/kv-leader.info, and execs supervisord — replacing itself in the process tree. When supervisord exits, the kernel releases the flock automatically.
On failure (someone else holds the lock) it sleeps ELECTION_RETRY_SECS (default 5s) and retries. Followers loop in bash and never start the Go binary.

Container startup
       │
       ▼
┌──────────────────┐
│ leader-election  │
│ .sh              │
│ flock(kv-leader  │
│  .lock)          │
└────────┬─────────┘
         │
    ┌────┴────┐
    │         │
 acquired  blocked
    │         │
    ▼         ▼
┌─────────┐  ┌──────────────┐
│ Write   │  │ sleep N,     │
│ kv-     │  │ retry        │
│ leader  │  └──────────────┘
│ .info   │
│         │
│ exec    │
│ super-  │
│ visord  │
└─────────┘
       │
       ▼
┌──────────────────────────────┐
│ supervisord starts:          │
│   nginx → rclone →           │
│   meta-core Go binary        │
└──────────────────────────────┘

Implications for the Go binary

The Go binary at cmd/meta-core/main.go is invoked only after the bash gate wins. Therefore:

internal/leader/election.go does not run an election loop. It contains LeaderInfoProvider, which builds the leader info struct from local hostname + IP + config and returns it on demand. There are no OnBecomeLeader / OnBecomeFollower / OnLeaderLost callbacks; those states do not exist inside the Go process.
LeaderLockInfo.RedisUrl is always empty in the response from /urls (kept for wire compatibility). PR D of api-mediated-access.md removed direct Redis exposure, and since 2026-10 there is no Redis at all; consumers route metadata I/O through meta-core's HTTP API and SSE event streams.

Lock-info file

/meta-core/locks/kv-leader.info is a single plaintext line:

http://10.0.1.50:9000

Other services read this file, then call GET /urls for full discovery (hostname, baseUrl, apiUrl, webdavUrl, webdavUrlInternal).

Failure handling

meta-core process dies → supervisord restarts it. The bash flock loop does not need to participate — supervisord owns the lock through its lifetime.
Container dies → kernel releases the flock. If a standby metacore-app is provisioned, its bash loop will acquire on next tick. The dev compose runs a single instance; failover is "restart the container."
The store is in-process — there is no separate database process to crash. A meta-core crash is a process restart; SQLite's WAL recovers the last committed transaction on reopen.

Schema-Version Gate

Before serving any traffic, cmd/meta-core/main.go opens both databases and calls storage.EnsureSchemaVersion (before the cache catch-up). In v2.0.0 the sentinel is the schema-version = "3" row of the truth database's meta table. The gate:

explicitly refuses a v1.x store — a truth file holding the v1 kv table (meta-core:schema-version = "2"), which the default path still points at — with a message naming tools/claims-migrate;
refuses a store with claims but no sentinel, or any other version;
stamps a fresh store with "3".

So no pre-claims data leaks into the current code paths. The meta-core service binary contains no migration code: watch.nsl.sh is converted once by the separate one-off tool packages/meta-core/tools/claims-migrate (not built into the image; implementation basis §11); every other box and dev stack starts empty (operators wipe with pnpm run clean:all and restart).

Data Model

The one concept: signed votes

claim(uid, cid, key) → value      + ts + sig

uid — the signer (a secp256k1 public key, z… base58btc). One key per writing container (each meta-sort plugin, each self-publishing feeder, gateway core, meta-share, meta-search) and one per user.
cid — the subject, a content CID; never a rid.
key — the field (METADATA_KEYS.md); key-set members live inside the key (genres/Drama, cids/<cid>).
value — text (public), bytes (signed + encrypted envelope) or null (retraction, kept as a tombstone).
ts — milliseconds; it is the version and only increases per (uid, cid, key).
sig — MMUR2: secp256k1 ECDSA (compact 64 bytes) over SHA-256 of "MMUR2\n" ‖ canonical CBOR [cid, uid, key, value, ts].

The old shared record (file:<rid>/<field>) and the old User Data Layer store (udl:rec:*) both collapse into this shape. The store distinguishes only plaintext (signed) from ciphertext (signed + encrypted); "personal vs shared" is app logic.

Two databases

┌──────────────────────────────────────────────────────────────────────┐
│     TRUTH  /meta-core/db/meta-core.db   (backed up; PKs only)        │
│                                                                      │
│  principal(pid, uid)              uid ↔ local surrogate pid          │
│  claim(cid, key, pid → value, ts, sig, seq)                          │
│                                   the votes; PK (cid, key, pid)      │
│  cid_group(cid → rid)             hidden local record identity       │
│  meta(name → value)               schema-version, last_seq           │
└──────────────────────────────────────────────────────────────────────┘
                     │  apply loop (seq order)
                     ▼
┌──────────────────────────────────────────────────────────────────────┐
│     CACHE  /meta-core/db-cache/cache.db   (never backed up; rebuildable)│
│                                                                      │
│  kv(k → v)   the v1 layout, every row derived:                       │
│    file:<rid>/<field>             the merged view every read uses    │
│    cid:<cid>                      cid → rid lookup                   │
│    idx:roots/ idx:fts/ idx:count/ idx:schema/   roots, FTS rowid,    │
│                                    counters (search totals, stats)   │
│    ev:<stream>/…, evhwm:…         /api/events/meta + /files backlog  │
│    meta:applied_seq, meta:rebuilt_at   apply-loop bookkeeping        │
│    anything else                  /api/kv/value scratch KV           │
│  kv_field_v (index), fts          filter + free-text search          │
│  claim_by_pid(pid, cid, key)      "everything uid X wrote" (?as=,    │
│                                    UDL facade, identity retraction)  │
│  local_file(path → midhash, size, mtime)   watcher facts, unsigned   │
└──────────────────────────────────────────────────────────────────────┘

Disaster recovery is "delete the cache, replay truth". META_CORE_DB_PATH names the truth file, META_CORE_CACHE_PATH the cache file. The cache keeps the v1 kv layout so every v1 reader (record reads, field index, FTS, counters, SSE + snapshot) runs unchanged on derived rows; nsrc is computed by the merge (POST /api/merge), not stored. Full schema: implementation basis §5.

Records: cids, groups and the hidden rid

A vote never mentions a rid. A rid (UUIDv7, 26-char Crockford, as in v1.x) is a local name for a group of cids known to be the same content, held in cid_group:

A new cid with no link gets its own group and a fresh rid.
A cids/<other> vote on cid X links X's group and <other>'s group; the group with the older rid survives, the losing rid gets a del event per key and the survivor set events (no alias table). Retracting the link does not split the group (phase 1); the record view synthesises cids/<cid> = "true" for every cid of the group from cid_group.
Reading record R = every vote on every cid of R's group, merged per key; within a group one signer counts once per key.
rids are never signed and never travel; consumers that hold them (meta-sort, meta-fuse, meta-stremio) keep working because cid_group lives in truth.

Merge

Per (rid, key), over the group's votes, each signer's newest vote counted once (internal/storage/merge.go, shared by the apply loop, ?as= and POST /api/merge):

the viewer's own live text vote (?as=<uid>, applied at read time, never stored);
retractions and ciphertext drop out — only text merges;
most distinct signers for the normalised value (trim + case-fold only, no NFC) — nsrc is that count (computed, not stored);
tie → oldest ts;
tie → byte-wise smallest value.

The stored value is the winning vote's exact text. Key-sets need no special code (each member is a key, present while one vote supports it). A different kind of fact gets a different key, so a guesser (e.g. filename-parser) never competes with a lookup on the same key.

Box-local file facts

filePath, size, mtimeNano and midhash are not claims — meta-core cannot sign, and the disk can always be rescanned. The file watcher and POST /api/files/register write them into the cache's local_file table (and the midhash into cid_group); they are joined into the record view unsigned and never reach the claims API or the swarm. When several local paths hold the same content, filePath is the lexicographically first one (size and mtime from the same row); a signed sizeByte claim wins over the on-disk size.

File bytes

Location: /files volume (shared across containers).
Access:   path resolution via local_file, served via /webdav/*,
          /file/{cid} (reverse-lookup), or /data/{hash}.
Path:     filePath (from local_file) in the record view, relative to /files.

History: UUID-rooted flat keys (v1.x)

Until v2.0.0 a record was a set of flat keys under an opaque UUID root — file:<uuid>/<property>, file:<uuid>/cids/<bareCid> key-sets, dup:<uuid>/<path>, a reverse index cid:<bareCid> → <uuid> and a root index idx:roots/<uuid>, all in one kv table. Roots had previously been content-hash tokens (file:midhash256:abc/...); UUID roots treated every CID symmetrically as an alias. The rid of v2.0.0 is that UUID, kept hidden and local; migrated records kept their UUIDs. See uuid-rooted-metadata.md (the why) and the History note of metadata-storage-structure.md.

Path resolution

filePath values are absolute paths under the /files volume (e.g. /files/watch/movies/Movie.2024.mkv). The /files prefix is itself a per-host detail — moving the volume only requires re-mounting it at the same path inside the container.

API Design

Endpoint catalogue

The router is in internal/api/server.go. Endpoint set, by surface:

Bootstrap / discovery
  GET  /health
  GET  /status
  GET  /leader
  GET  /urls
  GET  /neighbors        (alias: /api/neighbors)
  GET  /api/stats

Claims (the only metadata write path)
  POST   /api/claims                 signed claim batch → per-claim status
  POST   /api/merge                  {claims, as?} → {fields, nsrc, rejected}; no write
  GET    /api/meta/{cid}/claims      {rid, cids, claims} of the cid's group (?key=, ?uid=; 404 unknown)
  GET    /api/metadata/{rid}/claims  same, addressed by rid

Metadata reads (merged view + local_file; optional ?as=<uid>)
  GET    /meta
  GET    /meta/{hash}
  GET    /meta/{hash}/{key...}

CID-addressed (public, auth-bypassed at the perimeter)
  GET    /api/meta/{cid}
  GET    /api/file/{cid}/info
  GET    /file/{cid}
  POST   /file/cid
  HEAD   /data/{hash}
  GET    /data/{hash}/path

Editor / KV / schema
  GET /api/metadata/hash-ids
  GET /api/files/tuples
  POST /api/files/register        (fills local_file + cid_group)
  GET /api/metadata/list
  POST /api/metadata/search
  POST /api/metadata/clear        (wipes both databases)
  GET /api/metadata/{hashId}      (hashId = rid)
  GET /api/metadata/{hashId}/property
  GET /api/kv/{info,keys,tree,search,find}   (read-only browser over the cache kv table)
  {GET,PUT,DELETE} /api/kv/value  (service scratch keys only; reserved prefixes refused)
  GET /api/kv/key/{key...}                    (read-only)
  GET /api/schema
  POST /api/schema/rescan

User Data Layer (facade over claims; records are MMUR2)
  {GET,PUT} /api/udl/record
  PUT /api/udl/records
  GET /api/udl/user/{uid}/{key/{key},cid/{cid},all,count}
  GET /api/udl/cid/{cid}/{users,aggregate}
  GET /api/udl/users/stats        (every signer with live claims, plugin keys included)

Account-key identity service (kept in v2.0.0; removal is a separate ticket)
  {GET,DELETE} /api/identity      (DELETE signs + stores a retraction of every live claim first)
  GET  /api/identity/{accounts,aead-key}
  POST /api/identity/{challenge,generate,import,reveal,sign,sign-batch}
                                  (sign/sign-batch `format`: compact | der; default
                                   compact for "MMUR2\n" payloads, DER otherwise)

Removed in v2.0.0 — answer 410 Gone, pointing to POST /api/claims
  PUT/PATCH/DELETE /meta/{hash}
  PUT/DELETE /meta/{hash}/{key...}
  POST /meta/{hash}/_add/{key...}
  PUT/DELETE /api/metadata/{hashId}
  PUT /api/metadata/{hashId}/property
  POST /api/metadata/batch

Removed in v2.0.0 — no snapshot format
  GET /api/snapshot/export, POST /api/snapshot/{import,wipe}

SSE event streams (HTTP-mirror of the store's event streams)
  GET /api/events/files
  GET /api/events/meta

Mounts (rclone-only)
  GET    /api/mounts
  POST   /api/mounts
  GET    /api/mounts/{id}
  PUT    /api/mounts/{id}        (also PATCH)
  DELETE /api/mounts/{id}
  POST   /api/mounts/{id}/{mount,unmount,safe-unmount,scan}
  GET    /api/mounts/rclone/remotes

Watchers
  {GET,POST} /api/watchers
  {GET,PUT,DELETE} /api/watchers/{id}
  POST /api/watchers/{id}/{scan,reset}
  POST /api/watchers/{scan-all,reset-all}
  (deprecated: /api/scan/trigger, /api/scan/status — see watcher/handlers.go:32)

(The one-shot Redis-era admin migrations — `/api/admin/migrate-dual-roots`,
`migrate-domain-screen`, `migrate-user-cids`, `sweep-literature-echoes` —
were removed with the 2026-10 SQLite switch; they had been run once on
watch.nsl.sh beforehand.)

WebDAV
  /webdav/...   (GET, PUT, DELETE, MKCOL, COPY, MOVE, PROPFIND)

No /metrics endpoint is exposed. Observability is via container logs + the /health and /status JSON.

Auth perimeter

In the dev and prod stacks Caddy + an nginx-hash-lock sidecar enforce OIDC auth in front of meta-core. The following paths bypass auth (so peers and unauthenticated tooling can reach them):

/health
/webdav/*
/meta/*
/file/cid
/api/file/{cid} and /api/meta/{cid}
(/api/events/files, /api/events/meta are also auth-bypassed but should not be exposed publicly via Caddy; they're inside-only.)

Since v2.0.0 /meta/* serves reads only — its write verbs answer 410 Gone — so the bypass no longer exposes an unsigned write. Writes go to POST /api/claims, where the claim signature (not the perimeter) is the write authentication. POST /api/claims, POST /api/merge and GET /api/meta/{cid}/claims are container-to-container routes, not routed through Caddy, like the v1.x write routes were.

See docs/api-mediated-access.md "Auth" for the rationale.

SSE event streams — wire contract

/api/events/files and /api/events/meta are SSE mirrors of the file:events and meta:events streams, stored as ev:<stream>/… rows of the kv table (internal/kv/stream.go). The contract (internal/api/sse_events.go):

One SSE event per stream entry.
id: is the opaque <ms>-<seq> entry ID (the Redis Stream form, kept so cursors persisted before the 2026-10 switch still compare) — clients echo it back via Last-Event-ID on reconnect. Without Last-Event-ID the stream starts at the head (resolved once at connect); 0 / 0-0 replay from the beginning.
event: is the type field of the underlying entry.
data: is the JSON of the rest of the entry.
Heartbeats: SSE comment :keep-alive\n\n every 30s of silence.
On reconnect with Last-Event-ID, the handler resumes from the next entry. If retention has trimmed past the cursor, the handler emits one event: gap payload before resuming from the oldest available entry.

External services use SSE with Last-Event-ID as the resume primitive. Since v2.0.0 meta:events describes changes of the merged view, same {type: set|del, key, ts} shape: the cache engine appends an ev:meta:events/… row for every change of a merged file: row, in the same cache transaction as the change (both streams are v1 stream rows of the cache kv table). A cache rebuild emits no per-key events — consumers get event: gap and take a snapshot; event IDs stay time-based and keep rising. History: v1.x appended them in the same transaction as the file: key change; the Redis-era keyspace-notification hop through MetaPublisher is long gone.

Write path and apply loop

POST /api/claims ──► per claim, before the writer:
                       0. validate (required fields, reserved keys, ts range) → else bad_request
                       1. same ts + sig as stored?       → unchanged (no verify)
                       2. same value as stored (any ts)? → unchanged; stored row + older ts kept
                       3. stored ts ≥ incoming ts?       → stale
                       4. verify MMUR2 signature         → else bad_signature
                 ──► accepted claims, one group-committed truth transaction:
                       5. re-check 1–3 under the writer
                       6. uid → pid (principal; insert on first sight)
                       7. upsert claim (tombstone if null), seq = ++last_seq
                       8. cid_group: mint rid for unknown cid; text cids/* links groups
                          (a retraction never splits)
                       9. commit (last_seq in truth meta); answer once the cache applied it
                 ──► apply loop, in seq order, into the cache:
                       sync cid:/cids/<cid>/idx:roots rows → claim_by_pid
                       → re-merge (rid, key) from truth PK range reads
                       → set/delete file:<rid>/<key> (+ local_file for filePath/sizeByte/mtimeNano)
                         ⇒ meta:events row + counter delta in the same tx
                       → FTS row on a haystack change → meta:applied_seq

The verifier lives in the write path (secp256k1 from internal/identity); meta-core holds no writer key and never signs a claim. Response: one status per claim — accepted · unchanged · stale · bad_signature · bad_request. Request/response shapes and the 410 contract for removed routes: api-mediated-access.md "Write contract — signed claims".

WebDAV URLs

/urls exposes two WebDAV URLs (internal/leader/election.go):

Field	Purpose	Construction
`webdavUrl`	External access	`{BASE_URL}/webdav` (or `http://{ip}:{API_PORT}/webdav` if `BASE_URL` is unset)
`webdavUrlInternal`	Container-to-container	`http://{hostname}:{META_CORE_HTTP_PORT}/webdav`

Container plugins should use webdavUrlInternal. Browsers / external clients use webdavUrl (which goes through nginx + Caddy).

WebDAV caching is handled by nginx's proxy_cache (see docker/nginx.conf), not by a Go cache layer. There is no internal/cache package.

Implementation

Why Go

Requirement	Go advantage
Language-agnostic	HTTP API, no runtime dependency on callers
Single binary	Static compilation
Low resource	~10-20 MB resident
HTTP server	stdlib `net/http` + gorilla/mux
Metadata store	embedded SQLite via `modernc.org/sqlite` (pure Go, CGO off) — truth + cache databases
Claim verification	secp256k1 ECDSA (`internal/identity`)
WebDAV	`golang.org/x/net/webdav`

Repo layout

packages/meta-core/
├── cmd/meta-core/main.go         entry point
├── internal/
│   ├── api/                      HTTP server + handlers + SSE
│   ├── cid/                      CID parsing + algorithm ranking
│   ├── config/                   env-driven configuration
│   ├── discovery/                service registry + dead-service cleaner
│   ├── identity/                 secp256k1 (claim verification; account-key service)
│   ├── kv/                       SQLite engine (single writer, group commit) + event streams
│   ├── leader/                   LeaderInfoProvider (no election logic)
│   ├── mounts/                   rclone mount manager + handlers + stats
│   ├── schema/                   per-field schema from transactional counters
│   ├── storage/                  claims write path, apply loop, merge, cid_group, search, schema gate
│   ├── watcher/                  scanner + dispatcher + midhash + state
│   ├── watchers/                 polling-based watcher configs
│   └── webdav/                   /webdav/* handler
├── docker/
│   ├── entrypoint.sh
│   ├── leader-election.sh        the actual leader election
│   ├── mount-watcher.sh
│   ├── nginx.conf
│   └── supervisord.conf
├── editor/                       React metadata browser (Vite)
├── dashboard/                    React status dashboard (Vite)
├── docs/
└── go.mod

Startup sequence (`cmd/meta-core/main.go`)

1. Load configuration from environment
2. Build LeaderInfoProvider          (no election; bash gate already won)
3. Create the cache folder; open the truth database (META_CORE_DB_PATH or
   /meta-core/db/meta-core.db) and the cache (META_CORE_CACHE_PATH or
   /meta-core/db-cache/cache.db)     (storage.Connect)
4. EnsureSchemaVersion               (abort on a v1 kv store or an unknown layout)
5. storage.Start — cache catch-up    (meta:applied_seq ≥ last_seq → nothing; behind →
                                       replay claims with seq > applied_seq, or a full
                                       rebuild if a cids/* link is among them; no
                                       cache → full rebuild), then the apply loop
6. Derived counters ready
7. Start service discovery + dead-service cleaner
8. NewServer + Server.Start          (boots watcher, mounts, WebDAV, API)
9. Wait for SIGINT/SIGTERM
10. Shutdown in reverse order

(v1.x ran `EnsureCounters` and a background FTS rebuild here; in v2.0.0
counters and FTS are cache rows maintained with every merged-row change.)

Process management

There is no database process: the store is embedded in the Go binary (docker/supervisord.conf runs meta-core, nginx and rclone). Until 2026-10 supervisord also ran redis-server on localhost:6379.

Failure Modes

Container restart

Supervisord restarts crashed processes (meta-core, nginx, rclone) individually. The bash flock is held by supervisord's process tree, so internal crashes don't trigger re-election.

Store durability

Both SQLite files run in WAL mode; truth with synchronous=NORMAL (every committed transaction survives a meta-core process crash), the cache with synchronous=OFF (it is rebuildable). A claim batch commits to truth in one transaction; the apply loop then commits the derived cache changes. A crash between the two commits leaves the cache behind, never wrong: at boot, if the cache's applied_seq is below truth's last_seq, meta-core scans truth's claim table for seq > applied_seq (a full scan — there is no seq index; it only happens after an unclean shutdown) and applies them — or rebuilds the cache in full if one of them is a cids/* link.

Cache loss

The cache is never backed up and is fully derived (except local_file, which a rescan of /files refills, and the service scratch keys, whose loss costs a refetch). A missing cache folder (no meta:applied_seq) means meta-core rebuilds every derived row from truth and stamps meta:rebuilt_at. A rebuild emits no per-key events; SSE consumers get event: gap and re-snapshot.

Container crash

Kernel releases the flock; a standby container (if any) can take over. The dev compose runs a single instance, so the recovery path is "restart the container."

Schema mismatch

EnsureSchemaVersion refuses to boot. Operators wipe with pnpm run clean:all (or equivalent) and restart, or — once, on watch.nsl.sh — convert a v1.x store with tools/claims-migrate. This is an alpha-stage hard-fail; the service never migrates automatically.

Shared volume unavailable

All meta-core operations fail. Other services see meta-core as unhealthy via /health; metadata I/O and SSE event streams stop.

Configuration Reference

Environment variables

Variable	Default	Description
`META_CORE_PATH`	`/meta-core`	Shared volume root.
`FILES_PATH`	`/files`	Files volume root.
`SERVICE_NAME`	`meta-core`	Identity in the service registry.
`SERVICE_VERSION`	`1.0.0`	Reported in `/status`.
`API_PORT`	`8180`	External port baked into `baseUrl`.
`BASE_URL`	empty	Overrides constructed `baseUrl` (HTTPS perimeter).
`META_CORE_DB_PATH`	`<META_CORE_PATH>/db/meta-core.db`	Truth database file (backed up). Dev stacks set `/meta-core-db/meta-core.db` (named volume — SQLite WAL is unsafe on the 9p `/meta-core` bind of Windows dev boxes).
`META_CORE_CACHE_PATH`	`/meta-core/db-cache/cache.db`	Cache database file (never backed up; rebuilt from truth when missing). Dev stacks keep it on a named volume too; store apps put the cache folder in Maison's `backup.exclude`.
`META_CORE_DB_CACHE_MB`	`64`	Page cache of the writer connection.
`META_CORE_DB_READER_CACHE_MB`	`16`	Page cache per reader connection.
`META_CORE_DB_READERS`	`8`	Reader connection pool size.
`META_CORE_HTTP_PORT`	`9000`	Go HTTP+SSE+WebDAV port.
`META_CORE_HTTP_HOST`	`0.0.0.0`	HTTP bind address.
`HEALTH_CHECK_INTERVAL_MS`	`5000`	Internal health-loop cadence.
`HEARTBEAT_INTERVAL_MS`	`30000`	Service registry heartbeat.
`STALE_THRESHOLD_MS`	`60000`	Age past which a registry entry is stale.
`CLEANUP_INTERVAL_MS`	`600000`	Dead-service cleaner cadence.
`DEAD_SERVICE_THRESHOLD_MS`	`600000`	Age past which a stale entry is deleted.
`WATCH_INTERVAL_MS`	`1000`	Watcher poll cadence.
`DEBOUNCE_MS`	`30000`	File-event debounce window.
`ENABLE_FILE_WATCHER`	`true`	Disable to suppress the in-process watcher.
`ELECTION_RETRY_SECS`	`5`	Bash flock retry interval (consumed by `docker/leader-election.sh`, not the Go binary).

File paths under `/meta-core`

Path	Purpose
`locks/kv-leader.lock`	Flock target.
`locks/kv-leader.info`	Leader API URL (plaintext).
`db/meta-core.db` (+ `-wal`, `-shm`)	Truth database (claims, principals, `cid_group`). Backed up. Replaced `db/redis/` in 2026-10; holds the claims schema since v2.0.0.
`db-cache/cache.db` (+ `-wal`, `-shm`)	Cache database (v1-layout `kv` table: merged view, lookup, counters, events, service scratch keys; field index, FTS, `claim_by_pid`, `local_file`). Never backed up.
`services/*.json`	Service registry.
`mounts/mounts.json`	rclone mount configuration.
`mounts/errors/{id}.error`	Mount error logs.
`watchers.json`	Polling watcher configuration.
`cache/`	nginx `proxy_cache` directory.

Summary

meta-core consolidates leader gating, metadata storage, event mirroring, and file access for MetaMesh into one container with a typed HTTP+SSE surface. The bash flock gate guarantees single-leader semantics without an external consensus system. Metadata is a table of signed votes in a backed-up truth database, merged by an apply loop into a rebuildable cache that serves every read; meta-core verifies every claim and signs none. CID groups behind a hidden local rid treat all content hashes symmetrically. API-mediated access ensures no other service ever touches the store.

36 KiB Raw Permalink Blame History Unescape Escape