# meta-core Standalone Go service that owns the metadata store (an embedded SQLite file), the HTTP+SSE metadata surface, the User Data Layer store and signing keystore, and a WebDAV file server for the MetaMesh dev/prod stack. ## Overview meta-core runs as its own container (`metacore-app`). Other MetaMesh services (meta-sort, meta-fuse, meta-stremio, meta-dup, meta-search, meta-share, meta-gateway, meta-watch, …) locate it over UDP multicast (beacon v2 — its `metamesh.core` advertise carries the full `/urls` payload) and then call its HTTP API. `META_CORE_URL` on a client pins it instead. The store is embedded in the Go binary (no database port at all) — all metadata I/O is mediated by the HTTP/SSE API (see [`docs/api-mediated-access.md`](docs/api-mediated-access.md)). ### Key capabilities | Feature | Description | |---------|-------------| | **DB-file mutex** | Bash `flock` loop in `docker/leader-election.sh` only `exec`s supervisord on the winner, so two containers sharing one `/meta-core` volume can never run two writers on the database file. It is a mutex, not discovery — nothing reads the lock. | | **Embedded SQLite store** | One schema-free `kv(k, v)` table in `/meta-core/db/meta-core.db` (pure-Go `modernc.org/sqlite`, WAL, `synchronous=NORMAL`). One writer goroutine with group commit; a record write (fields + root index + CID aliases + FTS row + events) is atomic. Replaced Redis in 2026-10 (Redis + the in-memory search copy needed ~6.6 GB on watch.nsl.sh's 169k records; the SQLite file is ~2.5 GB on disk with a bounded page cache). | | **HTTP API** | Typed REST surface (gorilla/mux) for metadata, KV browsing, snapshots, schema inference, mounts, watchers, neighbours, identity, UDL, and file access by CID. | | **Indexed search** | `POST /api/metadata/search` builds a superset of candidates in SQL — a partial expression index `kv_field_v` over (field name, value) of every record key, plus an FTS5 trigram table; the smallest part drives, the other filters are probed per candidate — then verifies each with the unchanged Go matchers. Fresh immediately, no in-memory copy (`internal/storage/search.go`, `internal/api/search_index.go`). | | **SSE event streams** | `/api/events/files` and `/api/events/meta` mirror the store's `file:events` / `meta:events` streams so external services never touch the store. `meta:events` entries are written in the same transaction as the change. | | **WebDAV server** | `/webdav/*` exposes the `/files` volume (read+write) for cross-service file access. nginx in front of meta-core handles caching. | | **Service discovery** | beacon v2: UDP multicast advertise/listen on `239.255.99.1:9099`, neighbours served at `/neighbors`. The file-based registry under `/meta-core/services/*.json` it replaced is gone. | | **File watcher / scanner** | Recursive scan + MidHash256 computation; emits events on `file:events`. Watched roots are POSTed watcher configs (`/meta-core/watchers.json`), not env vars. | | **Mount management** | rclone-only mount manager (SMB rendered into `:smb:`, plus pre-defined remotes). Read-only by construction. | | **UUID-rooted storage** | Roots are UUIDv7 (Crockford Base32, ULID layout); CIDs are reverse-index aliases. Schema-version sentinel refuses to boot against stale data. | | **Identity keystore** | Multi-account secp256k1 signing keys under `/meta-core/identity/accounts/`, created from the dashboard's Identity menu (never auto-generated). A private key is returned only by generate, or by reveal against a proof-of-possession signature (delete needs the same proof); `/api/identity/sign` refuses challenge-domain payloads. | | **User Data Layer** | Version-gated store for per-user signed (and, for private-tier keys, encrypted) records — My List, Continue Watching, ratings — indexed by user and CID. meta-core never inspects the crypto. See [User Data Layer](../../docs/project-architecture/user-data-layer.md). | | **Report inbox** | `/api/reports` stores the bug reports / feedback the client apps' 🐛 buttons file (forwarded server-to-server by meta-watch and meta-read), in its own bounded `db/reports.db`; the dashboard's **Reports** page lists, reads and triages them. See [architecture: Report inbox](docs/meta-core-architecture.md#report-inbox). | | **Dashboard + editor** | nginx serves a React dashboard at `/` and the standalone metadata editor at `/editor/`. | ### Design characteristics - **Single writer** — flock on the `/meta-core` volume; no external consensus. - **HTTP-only externally** — there is no database port; debug read-only with `docker exec metacore-app sqlite3 -readonly /meta-core/db/meta-core.db …` (dev stacks: `/meta-core-db/meta-core.db`). - **Schema-gated boot** — `EnsureSchemaVersion` aborts startup when the on-disk store layout predates the current build (alpha clean-wipe policy; see `docs/uuid-rooted-metadata.md`). - **Static binary** — CGO disabled (pure-Go SQLite), single Go binary plus a thin Alpine image with `rclone`, nginx, supervisord and the `sqlite` CLI (for debugging). ## Architecture ``` ┌──────────────────────────────────────────────────────────────────────────┐ │ metacore-app (standalone container) │ │ │ │ leader-election.sh (flock /meta-core/locks/kv-leader.lock, │ │ then exec supervisord) │ │ supervisord │ │ ├─ nginx :80 (dashboard, /editor, API proxy, │ │ │ proxy_cache for /webdav, rclone GUI /rclone) │ │ ├─ rclone rcd (mount daemon, RC API on :5572) │ │ ├─ mount-watcher.sh (rclone mount lifecycle) │ │ └─ meta-core (Go) ─► HTTP API + SSE + WebDAV on :9000 │ │ ─► embedded SQLite (db/meta-core.db) │ │ ─► UDP advertise/listen 239.255.99.1:9099 │ └──────────────────────────────────────────────────────────────────────────┘ │ ┌──────────────────────┴──────────────────────┐ │ /meta-core (meta-core's own volume) │ │ locks/kv-leader.lock │ │ db/meta-core.db (+ -wal, -shm) │ │ identity/accounts/*.json │ │ mounts/{mounts.json, errors/} │ │ watchers.json │ │ cache/ (nginx proxy_cache) │ └─────────────────────────────────────────────┘ meta-sort / meta-fuse / meta-stremio / meta-search / meta-share / … │ 1. hear the metamesh.core advertise on UDP multicast (carries /urls) └─ 2. HTTP + SSE (+ WebDAV) to metacore-app — never the store, never the volume ``` ## Quick start ### Build ```bash make build # build static binary to bin/meta-core make docker # build container image as meta-core:1.0.0 + :latest + :local make test # run Go unit tests ``` `make docker` tags the image as `:1.0.0`, `:latest`, **and** `:local`. The `:local` tag is the one the sub-stack docker-compose files reference (meta-share federation, meta-read and meta-listen dev stacks); the meta-gateway dev stack (`docker-compose.gateway.yml`) uses `:latest`. Without these tags the sub-stacks can't see a freshly-built image by name. > **Source-change → recreate.** `make docker` rebuilds and re-tags the > image, but **already-running containers stay on the old image SHA** — > `docker compose up -d` won't recreate them unless the compose file > itself changed. After a meta-core source change, force-recreate the > downstream containers so they pick up the new binary: > > ```bash > make docker > docker compose -f packages/meta-gateway/dev/docker-compose.gateway.yml \ > up -d --force-recreate metagateway-hub-metacore > docker compose -f packages/meta-share/dev/docker-compose.yml \ > --profile share-10node up -d --force-recreate metashare-core metashare-core-test > docker compose -f packages/meta-read/dev/docker-compose.yml \ > up -d --force-recreate metaread-dev-core > docker compose -f packages/meta-listen/dev/docker-compose.yml \ > up -d --force-recreate metalisten-dev-core > ``` > > The symptom of a missed recreate is missing API fields on `/urls` > (e.g. `webdavUrlInternal` absent → meta-gateway's > `store_poster_for_record` silently fails and torznab posters never > land). Audit recipe: > > ```bash > docker inspect meta-core:local --format '{{.Id}}' # fresh SHA > docker ps --filter ancestor=meta-core:local --format '{{.Names}}' # repeat for :latest > # any container whose Image SHA doesn't match the fresh SHA needs --force-recreate. > ``` ### Dev container In the MetaMesh dev stack, the container is brought up by `dev/docker-compose.yml`: ```bash cd dev && ./scripts/start.sh ``` The user-facing URL is `https://metacore-dev.localhost:8083`; the direct-backend HTTP port (no auth, bypasses Caddy) is `http://localhost:18083`. ## Configuration Environment variables (from `internal/config/config.go` unless noted): | Variable | Default | Description | |---|---|---| | `META_CORE_PATH` | `/meta-core` | meta-core's own volume root (lock, SQLite DB, identity, mounts, watchers, cache). No other service mounts it. | | `FILES_PATH` | `/files` | Files volume root. | | `SERVICE_NAME` | `meta-core` | Name in the beacon v2 advertise. | | `SERVICE_VERSION` | `1.0.0` | Reported via `/status`. | | `API_PORT` | `8180` | External port baked into the constructed `baseUrl` when `BASE_URL` is unset. | | `BASE_URL` | _empty_ | Overrides the constructed `baseUrl` (the Caddy/HTTPS perimeter URL). | | `META_CORE_PUBLIC_URL` | _empty_ | Browser-facing nav URL in the announce; wins over `BASE_URL` (e.g. a debug-direct port with no Caddy). | | `META_CORE_DB_PATH` | `/db/meta-core.db` | SQLite database file. Dev stacks set `/meta-core-db/meta-core.db` on a named volume (the `/meta-core` bind is 9p on Windows dev boxes and SQLite WAL is unsafe there). | | `META_CORE_DB_CACHE_MB` | `64` | Page cache of the writer connection (`internal/storage/client.go`). | | `META_CORE_DB_READER_CACHE_MB` | `16` | Page cache per reader connection. | | `META_CORE_DB_READERS` | `8` | Reader connection pool size. | | `META_CORE_REPORTS_DB_PATH` | `/db/reports.db` | Report inbox database. | | `META_CORE_REPORTS_MAX` / `META_CORE_REPORTS_MAX_MB` | `5000` / `256` | Inbox bounds; closed reports are evicted first, then the oldest. | | `META_CORE_HTTP_PORT` | `9000` | Go HTTP+SSE+WebDAV port. | | `META_CORE_HTTP_HOST` | `0.0.0.0` | HTTP bind. | | `ENABLE_FILE_WATCHER` | `true` | Disable to suppress the in-process watcher entirely (`POST /api/files/register` stays available). | | `ENABLE_UDP_DISCOVERY` | `true` | beacon v2 advertise + listen. | | `BEACON_GROUP` / `BEACON_PORT` | `239.255.99.1` / `9099` | Multicast group and UDP port. | | `BEACON_INTERVAL_MS` | `10000` | Unsolicited advertise interval. | | `ELECTION_RETRY_SECS` | `5` | flock retry interval (read by `docker/leader-election.sh`, not the Go binary). | `HEALTH_CHECK_INTERVAL_MS`, `HEARTBEAT_INTERVAL_MS`, `STALE_THRESHOLD_MS`, `CLEANUP_INTERVAL_MS`, `DEAD_SERVICE_THRESHOLD_MS`, `WATCH_INTERVAL_MS` and `DEBOUNCE_MS` are still parsed by `internal/config/config.go` but nothing reads them (leftovers of the file-based registry); setting them has no effect. There is no `REDIS_PORT` / `META_CORE_REDIS_PORT` / `META_CORE_INDEX_EXCLUDE_PREFIXES` (gone with Redis and the in-memory search index; search results now include `categories/newznab/*` like every record read) / `META_CORE_SERVICE_NAME` / `META_CORE_LEADER_TIMEOUT`, and no `WATCH_PATHS` — watched folders are watcher configs created via `/api/watchers`. ## API reference All endpoints below are registered in `internal/api/server.go` (mounts, watcher and watchers routes in their own packages' `handlers.go`). Examples assume the in-container Go port `:9000`; from the host use `https://metacore-dev.localhost:8083` (Caddy) or `http://localhost:18083` (debug-direct). ### Health, leader, URLs ```bash GET /health # storage + role (`redis: bool` kept for compatibility = store open) GET /status # version, uptime, etc. GET /leader # current leader info (hostname, PID, timestamps) GET /urls # baseUrl, apiUrl, webdavUrl, webdavUrlInternal (redisUrl is always "", kept for compatibility) GET /neighbors # neighbours heard over UDP (beacon v2) GET /api/neighbors # same, under /api GET /api/stats # dashboard counters (see below) ``` `/services`, `/api/services`, `/services/{name}` and `/services/cleanup/stats` are **gone** — the registration-and-heartbeat registry they served was replaced by beacon v2 (`/neighbors`). nginx answers an unknown non-`/api` path with the dashboard's `index.html`, so a stale caller gets `200 text/html` rather than a 404; that is what made the dashboard fail with `SyntaxError: Unexpected token '<'` until 1.0.21. #### `GET /api/stats` Every headline counter in one request, split by cost: ```jsonc { "records": 88709, // count of idx:roots/* — metadata roots, file or not "redisKeys": 3115447, // kv row count (name kept for compatibility; cached 60 s) "redisMemory": "588.74M",// DB + WAL file size (name kept for compatibility) "identities": 3, // signing accounts on disk "files": { "count": 4864, "totalSize": 64587914424 }, // null until first sweep "udlUsers": 5, // accounts holding User Data Layer records "sweptAt": 1789402401970, "sweeping": false } ``` `records` and `identities` are live on every call; `redisKeys` is cached 60 s. `files` and `udlUsers` need a keyspace walk (~0.5s and ~3s respectively on an 88k-record box), so they are served from a single-flight cache refreshed in the background with a 60s TTL — the walk is **never** on the request path, because the dashboard polls and ten open tabs must not cost ten keyspace walks. Prefer this over `/api/kv/info`, which does one `GET` per root (6s+) and sums `sizeByte` across records that share a file. ### Metadata — primary surface These are the routes other services use. `{hash}` may be a UUID root or any registered CID alias; CID aliases resolve to the underlying UUID before reads/writes (`internal/storage/cid_resolution.go`). ```bash GET /meta # list of root IDs GET /meta/{hash} # full document (flat keys → nested JSON) PUT /meta/{hash} # replace document PATCH /meta/{hash} # merge into document DELETE /meta/{hash} # delete root + all CID aliases GET /meta/{hash}/{key...} # single property (key may contain "/") PUT /meta/{hash}/{key...} # set single property DELETE /meta/{hash}/{key...} # delete single property POST /meta/{hash}/_add/{key...} # add value to set-valued property ``` CID-addressed access (public; auth-bypassed in the Caddy perimeter): ```bash GET /api/meta/{cid} # resolve CID → metadata document GET /api/file/{cid}/info # CID resolution metadata GET /file/{cid} # serve file bytes (range-aware) POST /file/cid # compute CID for an uploaded file POST /api/files/register # mint + alias one file under /files/plugin by midhash # (the watcher's path for one file; works with the watcher OFF) HEAD /data/{hash} # existence + size GET /data/{hash}/path # absolute path on /files ``` (There is no `/data/{hash}/stream`; bulk reads go through `/webdav/*` or the `/file/{cid}` reverse-lookup path.) ### Editor / KV browser surface ```bash GET /api/metadata/hash-ids GET /api/files/tuples GET /api/metadata/list POST /api/metadata/search POST /api/metadata/batch POST /api/metadata/clear GET /api/metadata/{hashId} PUT /api/metadata/{hashId} DELETE /api/metadata/{hashId} GET /api/metadata/{hashId}/property PUT /api/metadata/{hashId}/property GET /api/kv/info GET /api/kv/keys GET /api/kv/tree # cursor is an opaque next-key ("0" when exhausted) GET /api/kv/search GET /api/kv/find GET /api/kv/value # type is always "string" for an existing key PUT /api/kv/value DELETE /api/kv/value GET /api/kv/key/{key...} GET /api/schema POST /api/schema/rescan GET /api/snapshot/export POST /api/snapshot/import POST /api/snapshot/wipe ``` ### SSE event streams The HTTP-mirror of the store's event streams (Redis Streams until 2026-10; IDs keep the `-` form). External services never read the store directly — they subscribe here with `Last-Event-ID` for resume. See [`docs/api-mediated-access.md`](docs/api-mediated-access.md) for the wire contract (one event per stream entry, opaque IDs, heartbeats, gap signalling). ```bash GET /api/events/files # file:events stream (watcher events) GET /api/events/meta # meta:events stream (metadata mutations) GET /api/events/poll # DEPRECATED long-poll over file:events — use SSE ``` ### Identity (signing keystore) Reached through meta-search's identity proxy (and directly by meta-gateway, meta-read, meta-listen) for record signing, UDL writes and keypair sign-in. ```bash GET /api/identity # status (never the private key) GET /api/identity/accounts POST /api/identity/generate # the only call that returns a new private key POST /api/identity/import POST /api/identity/challenge # one-shot challenge for proof-of-possession POST /api/identity/reveal # needs proof-of-possession DELETE /api/identity # needs proof-of-possession; reports purged UDL counts POST /api/identity/sign POST /api/identity/sign-batch GET /api/identity/aead-key ``` ### User Data Layer Records arrive signed (private-tier ones encrypted) as opaque base64 CBOR; writes are version-gated (stale → `409 stale_version`, or `accepted:false` per cell in a batch). Storage model in `internal/storage/udl.go` (`udl:rec://` → JSON `{record,version,ts,value?}`; index sets as one key per member). ```bash GET /api/udl/record?uid&cid&key PUT /api/udl/record PUT /api/udl/records # batch GET /api/udl/user/{uid}/key/{key} GET /api/udl/user/{uid}/cid/{cid} GET /api/udl/user/{uid}/all GET /api/udl/user/{uid}/count GET /api/udl/cid/{cid}/users GET /api/udl/cid/{cid}/aggregate # meta-watch Aggregate wire shape GET /api/udl/users/stats ``` ### Reports (bug-report / feedback inbox) ``` POST /api/reports # {app, note?, page?, route?, cid?, title?, uid?, build?, filedAt?, log?, …blocks} → 201 {id} GET /api/reports # ?app=&status=open|triaged|closed&q=&before=&limit= → {reports, apps, counts, more} GET /api/reports/{id} # full report incl. body PATCH /api/reports/{id} # {status?, triage_note?} DELETE /api/reports/{id} ``` ### Mounts ```bash GET /api/mounts POST /api/mounts GET /api/mounts/{id} PUT /api/mounts/{id} # also PATCH DELETE /api/mounts/{id} POST /api/mounts/{id}/mount POST /api/mounts/{id}/unmount POST /api/mounts/{id}/safe-unmount POST /api/mounts/{id}/scan GET /api/mounts/rclone/remotes ``` ### Watchers ```bash GET /api/watchers POST /api/watchers GET /api/watchers/{id} PUT /api/watchers/{id} DELETE /api/watchers/{id} POST /api/watchers/{id}/scan POST /api/watchers/{id}/reset POST /api/watchers/scan-all POST /api/watchers/reset-all ``` The legacy `/api/scan/trigger` and `/api/scan/status` still respond but return a deprecation pointer at the `/api/watchers/*` routes (`internal/watcher/handlers.go:32-33`). ### WebDAV ```bash GET/PUT/DELETE /webdav/... # mounted on /files ``` The one-shot Redis-era admin migrations (`/api/admin/migrate-dual-roots`, `migrate-domain-screen`, `migrate-user-cids`, `sweep-literature-echoes`) were removed with the 2026-10 SQLite switch; they had been run once on watch.nsl.sh before the migration. No `/metrics` endpoint is exposed; observability is via container logs and the `/health` / `/status` JSON. ## DB-file mutex `docker/leader-election.sh` is a bash flock loop, not a Go-level state machine. When `flock` succeeds on `/meta-core/locks/kv-leader.lock`, the script `exec`s supervisord; a second container mounting the same volume retries every `ELECTION_RETRY_SECS` and stays dormant. When the process tree dies, the kernel releases the lock. There is no follower path inside the Go binary. The lock is a **mutex, not service discovery** — it only stops two writers landing on one database file (`db/meta-core.db`). Nothing reads it, and the `kv-leader.info` file it used to publish is gone: siblings locate meta-core over UDP (see [beacon-v2.md](../../docs/project-architecture/beacon-v2.md)). `internal/leader` keeps its name for history, but only builds the `/urls` payload (`baseUrl`, `apiUrl`, `webdavUrl`, `webdavUrlInternal`) that `/urls` and the UDP announce share. ## Metadata storage shape (one-line summary) Roots are opaque UUIDv7 strings; every property is a separate row of the `kv` table at `file:/`; every known CID is a key-set member `file:/cids/` plus a reverse-index entry `cid: → ` (the canonical CID is derived by rank on read, never stored). Full details in [`docs/metadata-storage-structure.md`](docs/metadata-storage-structure.md) and design rationale in [`docs/uuid-rooted-metadata.md`](docs/uuid-rooted-metadata.md). The authoritative registry of field semantics and value formats lives in the repo-root [`METADATA_KEYS.md`](../../METADATA_KEYS.md). ## Internal packages | Package | Purpose | |---------|---------| | `cmd/meta-core` | Entry point; store open + schema sentinel + UDP discovery + service wiring. | | `internal/config` | Env-driven configuration, path helpers. | | `internal/leader` | `LeaderInfoProvider` — builds the `/urls` payload from hostname/IP/config; no election logic. | | `internal/meshdisco` | beacon v2 node (advertise, probe, neighbour map) — the Go port of the spec. | | `internal/kv` | Storage engine: SQLite `kv` table, single writer + group commit, reader pool, in-transaction change events (`kv.go`), event streams as rows (`stream.go`). | | `internal/storage` | Flat-key records (`client.go`), UUIDv7 minting, CID reverse index (`cid_resolution.go`), schema sentinel, search over `kv_field_v` + FTS (`search.go`), derived counters (`onChange`), UDL store (`udl.go`, `udl_admin.go`), stats. | | `internal/cid` | CID parsing + rank ladder for canonical-CID selection (`cid-rank-vectors.json`, guarded by the meta-root `scripts/check-cid-vectors.sh`). | | `internal/reports` | Report inbox: its own SQLite file, normalisation/bounds, list/filter/paging, prune (closed first). | | `internal/identity` | Multi-account secp256k1 keystore, challenge/proof-of-possession, signature verification. | | `internal/api` | HTTP server + all handlers + SSE event endpoints + search matchers + `/api/stats`. | | `internal/schema` | Per-field schema, folded from counters the write path keeps exact (`idx:schema/…`). | | `internal/snapshot` | Snapshot export / import / wipe. | | `internal/watcher` | File system scanner, MidHash256 computation, dispatcher, state registry. | | `internal/watchers` | Polling-based watcher configurations (manager + poller + handlers). | | `internal/mounts` | rclone mount manager, lifecycle handlers, stats poller. | | `internal/webdav` | WebDAV handler (mounted at `/webdav/*`; nginx handles caching upstream). | ### Startup sequence ``` 1. Load configuration from environment 2. Build LeaderInfoProvider (no election — the bash gate already won) 3. Open the SQLite store (META_CORE_DB_PATH or /meta-core/db/meta-core.db) 4. EnsureSchemaVersion — abort if the existing store layout is stale 5. Start beacon v2 node (unless ENABLE_UDP_DISCOVERY=false) 6. Construct API server (identity keystore migration, watcher, watchers, mounts, WebDAV) 7. EnsureCounters — one-time build of the derived counters (record count, per-field schema) on a store that predates them; blocking, ~1 min at 169k records, then never again 8. Start API server + pollers (no republish: consumers that want the whole state ask /api/events/meta for `0-0` and get a snapshot) 9. Wait for SIGINT/SIGTERM, shutdown in reverse order ``` ## Dependencies | Dependency | Purpose | |------------|---------| | `github.com/gorilla/mux` | HTTP router | | `modernc.org/sqlite` | Embedded SQLite (pure Go, CGO off) | | `github.com/google/uuid` | UUIDv7 minting | | `github.com/decred/dcrd/dcrec/secp256k1/v4`, `golang.org/x/crypto`, `github.com/mr-tron/base58` | Identity keys, signing, AEAD key derivation | | `golang.org/x/net` | WebDAV support | Runtime requirements: Go 1.21+ at build time; the container ships rclone, nginx, and supervisord alongside the binary, plus the React dashboard and metadata editor (built from `dashboard/` and `editor/` in the Dockerfile). ## License Part of the MetaMesh project.