meta-core

Standalone Go service that owns the metadata store (an embedded SQLite file), the HTTP+SSE metadata surface, the User Data Layer store and signing keystore, and a WebDAV file server for the MetaMesh dev/prod stack.

Overview

meta-core runs as its own container (metacore-app). Other MetaMesh services (meta-sort, meta-fuse, meta-stremio, meta-dup, meta-search, meta-share, meta-gateway, meta-watch, …) locate it over UDP multicast (beacon v2 — its metamesh.core advertise carries the full /urls payload) and then call its HTTP API. META_CORE_URL on a client pins it instead. The store is embedded in the Go binary (no database port at all) — all metadata I/O is mediated by the HTTP/SSE API (see docs/api-mediated-access.md).

Key capabilities

Feature	Description
DB-file mutex	Bash flock loop in docker/leader-election.sh only execs supervisord on the winner, so two containers sharing one /meta-core volume can never run two writers on the database file. It is a mutex, not discovery — nothing reads the lock.
Embedded SQLite store	One schema-free kv(k, v) table in /meta-core/db/meta-core.db (pure-Go modernc.org/sqlite, WAL, synchronous=NORMAL). One writer goroutine with group commit; a record write (fields + root index + CID aliases + FTS row + events) is atomic. Replaced Redis in 2026-10 (Redis + the in-memory search copy needed ~6.6 GB on watch.nsl.sh's 169k records; the SQLite file is ~2.5 GB on disk with a bounded page cache).
HTTP API	Typed REST surface (gorilla/mux) for metadata, KV browsing, snapshots, schema inference, mounts, watchers, neighbours, identity, UDL, and file access by CID.
Indexed search	POST /api/metadata/search builds a superset of candidates in SQL — a partial expression index kv_field over (field name, value) of every record key, plus an FTS5 trigram table — then verifies each with the unchanged Go matchers. Fresh immediately, no in-memory copy (internal/storage/search.go, internal/api/search_index.go).
SSE event streams	/api/events/files and /api/events/meta mirror the store's file:events / meta:events streams so external services never touch the store. meta:events entries are written in the same transaction as the change.
WebDAV server	/webdav/* exposes the /files volume (read+write) for cross-service file access. nginx in front of meta-core handles caching.
Service discovery	beacon v2: UDP multicast advertise/listen on 239.255.99.1:9099, neighbours served at /neighbors. The file-based registry under /meta-core/services/*.json it replaced is gone.
File watcher / scanner	Recursive scan + MidHash256 computation; emits events on file:events. Watched roots are POSTed watcher configs (/meta-core/watchers.json), not env vars.
Mount management	rclone-only mount manager (SMB rendered into :smb:, plus pre-defined remotes). Read-only by construction.
UUID-rooted storage	Roots are UUIDv7 (Crockford Base32, ULID layout); CIDs are reverse-index aliases. Schema-version sentinel refuses to boot against stale data.
Identity keystore	Multi-account secp256k1 signing keys under /meta-core/identity/accounts/, created from the dashboard's Identity menu (never auto-generated). A private key is returned only by generate, or by reveal against a proof-of-possession signature (delete needs the same proof); /api/identity/sign refuses challenge-domain payloads.
User Data Layer	Version-gated store for per-user signed (and, for private-tier keys, encrypted) records — My List, Continue Watching, ratings — indexed by user and CID. meta-core never inspects the crypto. See User Data Layer.
Dashboard + editor	nginx serves a React dashboard at / and the standalone metadata editor at /editor/.

Design characteristics

• Single writer — flock on the /meta-core volume; no external consensus.
• HTTP-only externally — there is no database port; debug read-only with docker exec metacore-app sqlite3 -readonly /meta-core/db/meta-core.db … (dev stacks: /meta-core-db/meta-core.db).
• Schema-gated boot — EnsureSchemaVersion aborts startup when the on-disk store layout predates the current build (alpha clean-wipe policy; see docs/uuid-rooted-metadata.md).
• Static binary — CGO disabled (pure-Go SQLite), single Go binary plus a thin Alpine image with rclone, nginx, supervisord and the sqlite CLI (for debugging).

Architecture

┌──────────────────────────────────────────────────────────────────────────┐
│                    metacore-app  (standalone container)                  │
│                                                                          │
│  leader-election.sh  (flock /meta-core/locks/kv-leader.lock,             │
│                       then exec supervisord)                             │
│  supervisord                                                             │
│   ├─ nginx  :80          (dashboard, /editor, API proxy,                 │
│   │                       proxy_cache for /webdav, rclone GUI /rclone)   │
│   ├─ rclone rcd          (mount daemon, RC API on :5572)                 │
│   ├─ mount-watcher.sh    (rclone mount lifecycle)                        │
│   └─ meta-core (Go)      ─►  HTTP API + SSE + WebDAV on :9000            │
│                          ─►  embedded SQLite (db/meta-core.db)           │
│                          ─►  UDP advertise/listen 239.255.99.1:9099      │
└──────────────────────────────────────────────────────────────────────────┘
                                       │
                ┌──────────────────────┴──────────────────────┐
                │  /meta-core (meta-core's own volume)        │
                │   locks/kv-leader.lock                      │
                │   db/meta-core.db (+ -wal, -shm)            │
                │   identity/accounts/*.json                  │
                │   mounts/{mounts.json, errors/}             │
                │   watchers.json                             │
                │   cache/ (nginx proxy_cache)                │
                └─────────────────────────────────────────────┘

   meta-sort / meta-fuse / meta-stremio / meta-search / meta-share / …
      │  1. hear the metamesh.core advertise on UDP multicast (carries /urls)
      └─ 2. HTTP + SSE (+ WebDAV) to metacore-app — never the store, never the volume

Quick start

Build

make build           # build static binary to bin/meta-core
make docker          # build container image as meta-core:1.0.0 + :latest + :local
make test            # run Go unit tests

make docker tags the image as :1.0.0, :latest, and :local. The :local tag is the one the sub-stack docker-compose files reference (meta-share federation, meta-read and meta-listen dev stacks); the meta-gateway dev stack (docker-compose.gateway.yml) uses :latest. Without these tags the sub-stacks can't see a freshly-built image by name.

Source-change → recreate. make docker rebuilds and re-tags the image, but already-running containers stay on the old image SHA — docker compose up -d won't recreate them unless the compose file itself changed. After a meta-core source change, force-recreate the downstream containers so they pick up the new binary:

make docker
docker compose -f packages/meta-gateway/dev/docker-compose.gateway.yml \
  up -d --force-recreate metagateway-hub-metacore
docker compose -f packages/meta-share/dev/docker-compose.yml \
  --profile share-10node up -d --force-recreate metashare-core metashare-core-test
docker compose -f packages/meta-read/dev/docker-compose.yml \
  up -d --force-recreate metaread-dev-core
docker compose -f packages/meta-listen/dev/docker-compose.yml \
  up -d --force-recreate metalisten-dev-core

The symptom of a missed recreate is missing API fields on /urls (e.g. webdavUrlInternal absent → meta-gateway's store_poster_for_record silently fails and torznab posters never land). Audit recipe:

docker inspect meta-core:local --format '{{.Id}}'                # fresh SHA
docker ps --filter ancestor=meta-core:local --format '{{.Names}}'   # repeat for :latest
# any container whose Image SHA doesn't match the fresh SHA needs --force-recreate.

Dev container

In the MetaMesh dev stack, the container is brought up by dev/docker-compose.yml:

cd dev && ./scripts/start.sh

The user-facing URL is https://metacore-dev.localhost:8083; the direct-backend HTTP port (no auth, bypasses Caddy) is http://localhost:18083.

Configuration

Environment variables (from internal/config/config.go unless noted):

Variable	Default	Description
META_CORE_PATH	/meta-core	meta-core's own volume root (lock, SQLite DB, identity, mounts, watchers, cache). No other service mounts it.
FILES_PATH	/files	Files volume root.
SERVICE_NAME	meta-core	Name in the beacon v2 advertise.
SERVICE_VERSION	1.0.0	Reported via /status.
API_PORT	8180	External port baked into the constructed baseUrl when BASE_URL is unset.
BASE_URL	empty	Overrides the constructed baseUrl (the Caddy/HTTPS perimeter URL).
META_CORE_PUBLIC_URL	empty	Browser-facing nav URL in the announce; wins over BASE_URL (e.g. a debug-direct port with no Caddy).
META_CORE_DB_PATH	<META_CORE_PATH>/db/meta-core.db	SQLite database file. Dev stacks set /meta-core-db/meta-core.db on a named volume (the /meta-core bind is 9p on Windows dev boxes and SQLite WAL is unsafe there).
META_CORE_DB_CACHE_MB	64	Page cache of the writer connection (internal/storage/client.go).
META_CORE_DB_READER_CACHE_MB	16	Page cache per reader connection.
META_CORE_DB_READERS	8	Reader connection pool size.
META_CORE_HTTP_PORT	9000	Go HTTP+SSE+WebDAV port.
META_CORE_HTTP_HOST	0.0.0.0	HTTP bind.
ENABLE_FILE_WATCHER	true	Disable to suppress the in-process watcher entirely (POST /api/files/register stays available).
ENABLE_UDP_DISCOVERY	true	beacon v2 advertise + listen.
BEACON_GROUP / BEACON_PORT	239.255.99.1 / 9099	Multicast group and UDP port.
BEACON_INTERVAL_MS	10000	Unsolicited advertise interval.
ELECTION_RETRY_SECS	5	flock retry interval (read by docker/leader-election.sh, not the Go binary).

HEALTH_CHECK_INTERVAL_MS, HEARTBEAT_INTERVAL_MS, STALE_THRESHOLD_MS, CLEANUP_INTERVAL_MS, DEAD_SERVICE_THRESHOLD_MS, WATCH_INTERVAL_MS and DEBOUNCE_MS are still parsed by internal/config/config.go but nothing reads them (leftovers of the file-based registry); setting them has no effect. There is no REDIS_PORT / META_CORE_REDIS_PORT / META_CORE_INDEX_EXCLUDE_PREFIXES (gone with Redis and the in-memory search index; search results now include categories/newznab/* like every record read) / META_CORE_SERVICE_NAME / META_CORE_LEADER_TIMEOUT, and no WATCH_PATHS — watched folders are watcher configs created via /api/watchers.

API reference

All endpoints below are registered in internal/api/server.go (mounts, watcher and watchers routes in their own packages' handlers.go). Examples assume the in-container Go port :9000; from the host use https://metacore-dev.localhost:8083 (Caddy) or http://localhost:18083 (debug-direct).

Health, leader, URLs

GET  /health           # storage + role (`redis: bool` kept for compatibility = store open)
GET  /status           # version, uptime, etc.
GET  /leader           # current leader info (hostname, PID, timestamps)
GET  /urls             # baseUrl, apiUrl, webdavUrl, webdavUrlInternal (redisUrl is always "", kept for compatibility)
GET  /neighbors        # neighbours heard over UDP (beacon v2)
GET  /api/neighbors    # same, under /api
GET  /api/stats        # dashboard counters (see below)

/services, /api/services, /services/{name} and /services/cleanup/stats are gone — the registration-and-heartbeat registry they served was replaced by beacon v2 (/neighbors). nginx answers an unknown non-/api path with the dashboard's index.html, so a stale caller gets 200 text/html rather than a 404; that is what made the dashboard fail with SyntaxError: Unexpected token '<' until 1.0.21.

GET /api/stats

Every headline counter in one request, split by cost:

{
  "records": 88709,        // count of idx:roots/* — metadata roots, file or not
  "redisKeys": 3115447,    // kv row count (name kept for compatibility; cached 60 s)
  "redisMemory": "588.74M",// DB + WAL file size (name kept for compatibility)
  "identities": 3,         // signing accounts on disk
  "files": { "count": 4864, "totalSize": 64587914424 },  // null until first sweep
  "udlUsers": 5,           // accounts holding User Data Layer records
  "sweptAt": 1789402401970,
  "sweeping": false
}

records and identities are live on every call; redisKeys is cached 60 s. files and udlUsers need a keyspace walk (~0.5s and ~3s respectively on an 88k-record box), so they are served from a single-flight cache refreshed in the background with a 60s TTL — the walk is never on the request path, because the dashboard polls and ten open tabs must not cost ten keyspace walks. Prefer this over /api/kv/info, which does one GET per root (6s+) and sums sizeByte across records that share a file.

Metadata — primary surface

These are the routes other services use. {hash} may be a UUID root or any registered CID alias; CID aliases resolve to the underlying UUID before reads/writes (internal/storage/cid_resolution.go).

GET    /meta                          # list of root IDs
GET    /meta/{hash}                   # full document (flat keys → nested JSON)
PUT    /meta/{hash}                   # replace document
PATCH  /meta/{hash}                   # merge into document
DELETE /meta/{hash}                   # delete root + all CID aliases
GET    /meta/{hash}/{key...}          # single property (key may contain "/")
PUT    /meta/{hash}/{key...}          # set single property
DELETE /meta/{hash}/{key...}          # delete single property
POST   /meta/{hash}/_add/{key...}     # add value to set-valued property

CID-addressed access (public; auth-bypassed in the Caddy perimeter):

GET    /api/meta/{cid}                # resolve CID → metadata document
GET    /api/file/{cid}/info           # CID resolution metadata
GET    /file/{cid}                    # serve file bytes (range-aware)
POST   /file/cid                      # compute CID for an uploaded file
POST   /api/files/register        # mint + alias one file under /files/plugin by midhash
                                  # (the watcher's path for one file; works with the watcher OFF)
HEAD   /data/{hash}                   # existence + size
GET    /data/{hash}/path              # absolute path on /files

(There is no /data/{hash}/stream; bulk reads go through /webdav/* or the /file/{cid} reverse-lookup path.)

Editor / KV browser surface

GET    /api/metadata/hash-ids
GET    /api/files/tuples
GET    /api/metadata/list
POST   /api/metadata/search
POST   /api/metadata/batch
POST   /api/metadata/clear
GET    /api/metadata/{hashId}
PUT    /api/metadata/{hashId}
DELETE /api/metadata/{hashId}
GET    /api/metadata/{hashId}/property
PUT    /api/metadata/{hashId}/property

GET    /api/kv/info
GET    /api/kv/keys
GET    /api/kv/tree            # cursor is an opaque next-key ("0" when exhausted)
GET    /api/kv/search
GET    /api/kv/find
GET    /api/kv/value           # type is always "string" for an existing key
PUT    /api/kv/value
DELETE /api/kv/value
GET    /api/kv/key/{key...}

GET    /api/schema
POST   /api/schema/rescan

GET    /api/snapshot/export
POST   /api/snapshot/import
POST   /api/snapshot/wipe

SSE event streams

The HTTP-mirror of the store's event streams (Redis Streams until 2026-10; IDs keep the <ms>-<seq> form). External services never read the store directly — they subscribe here with Last-Event-ID for resume. See docs/api-mediated-access.md for the wire contract (one event per stream entry, opaque IDs, heartbeats, gap signalling).

GET /api/events/files     # file:events stream (watcher events)
GET /api/events/meta      # meta:events stream (metadata mutations)
GET /api/events/poll      # DEPRECATED long-poll over file:events — use SSE

Identity (signing keystore)

Reached through meta-search's identity proxy (and directly by meta-gateway, meta-read, meta-listen) for record signing, UDL writes and keypair sign-in.

GET    /api/identity               # status (never the private key)
GET    /api/identity/accounts
POST   /api/identity/generate      # the only call that returns a new private key
POST   /api/identity/import
POST   /api/identity/challenge     # one-shot challenge for proof-of-possession
POST   /api/identity/reveal        # needs proof-of-possession
DELETE /api/identity               # needs proof-of-possession; reports purged UDL counts
POST   /api/identity/sign
POST   /api/identity/sign-batch
GET    /api/identity/aead-key

User Data Layer

Records arrive signed (private-tier ones encrypted) as opaque base64 CBOR; writes are version-gated (stale → 409 stale_version, or accepted:false per cell in a batch). Storage model in internal/storage/udl.go (udl:rec:<uid>/<cid>/<key> → JSON {record,version,ts,value?}; index sets as one key per member).

GET  /api/udl/record?uid&cid&key
PUT  /api/udl/record
PUT  /api/udl/records                      # batch
GET  /api/udl/user/{uid}/key/{key}
GET  /api/udl/user/{uid}/cid/{cid}
GET  /api/udl/user/{uid}/all
GET  /api/udl/user/{uid}/count
GET  /api/udl/cid/{cid}/users
GET  /api/udl/cid/{cid}/aggregate          # meta-watch Aggregate wire shape
GET  /api/udl/users/stats

Mounts

GET    /api/mounts
POST   /api/mounts
GET    /api/mounts/{id}
PUT    /api/mounts/{id}       # also PATCH
DELETE /api/mounts/{id}
POST   /api/mounts/{id}/mount
POST   /api/mounts/{id}/unmount
POST   /api/mounts/{id}/safe-unmount
POST   /api/mounts/{id}/scan
GET    /api/mounts/rclone/remotes

Watchers

GET    /api/watchers
POST   /api/watchers
GET    /api/watchers/{id}
PUT    /api/watchers/{id}
DELETE /api/watchers/{id}
POST   /api/watchers/{id}/scan
POST   /api/watchers/{id}/reset
POST   /api/watchers/scan-all
POST   /api/watchers/reset-all

The legacy /api/scan/trigger and /api/scan/status still respond but return a deprecation pointer at the /api/watchers/* routes (internal/watcher/handlers.go:32-33).

WebDAV

GET/PUT/DELETE /webdav/...            # mounted on /files

The one-shot Redis-era admin migrations (/api/admin/migrate-dual-roots, migrate-domain-screen, migrate-user-cids, sweep-literature-echoes) were removed with the 2026-10 SQLite switch; they had been run once on watch.nsl.sh before the migration.

No /metrics endpoint is exposed; observability is via container logs and the /health / /status JSON.

DB-file mutex

docker/leader-election.sh is a bash flock loop, not a Go-level state machine. When flock succeeds on /meta-core/locks/kv-leader.lock, the script execs supervisord; a second container mounting the same volume retries every ELECTION_RETRY_SECS and stays dormant. When the process tree dies, the kernel releases the lock. There is no follower path inside the Go binary.

The lock is a mutex, not service discovery — it only stops two writers landing on one database file (db/meta-core.db). Nothing reads it, and the kv-leader.info file it used to publish is gone: siblings locate meta-core over UDP (see beacon-v2.md). internal/leader keeps its name for history, but only builds the /urls payload (baseUrl, apiUrl, webdavUrl, webdavUrlInternal) that /urls and the UDP announce share.

Metadata storage shape (one-line summary)

Roots are opaque UUIDv7 strings; every property is a separate row of the kv table at file:<uuid>/<property>; every known CID is a key-set member file:<uuid>/cids/<bareCid> plus a reverse-index entry cid:<bareCid> → <uuid> (the canonical CID is derived by rank on read, never stored). Full details in docs/metadata-storage-structure.md and design rationale in docs/uuid-rooted-metadata.md. The authoritative registry of field semantics and value formats lives in the repo-root METADATA_KEYS.md.

Internal packages

Package	Purpose
cmd/meta-core	Entry point; store open + schema sentinel + UDP discovery + service wiring.
internal/config	Env-driven configuration, path helpers.
internal/leader	LeaderInfoProvider — builds the /urls payload from hostname/IP/config; no election logic.
internal/meshdisco	beacon v2 node (advertise, probe, neighbour map) — the Go port of the spec.
internal/kv	Storage engine: SQLite kv table, single writer + group commit, reader pool, in-transaction change events (kv.go), event streams as rows (stream.go).
internal/storage	Flat-key records (client.go), UUIDv7 minting, CID reverse index (cid_resolution.go), schema sentinel, search over kv_field + FTS (search.go), UDL store (udl.go, udl_admin.go), stats.
internal/cid	CID parsing + rank ladder for canonical-CID selection (cid-rank-vectors.json, guarded by the meta-root scripts/check-cid-vectors.sh).
internal/identity	Multi-account secp256k1 keystore, challenge/proof-of-possession, signature verification.
internal/events	Boot-time republish of every field into meta:events (meta_publisher.go); live events are written by internal/kv.
internal/api	HTTP server + all handlers + SSE event endpoints + search matchers + /api/stats.
internal/schema	Live per-field schema indexer (consumes meta:events).
internal/snapshot	Snapshot export / import / wipe.
internal/watcher	File system scanner, MidHash256 computation, dispatcher, state registry.
internal/watchers	Polling-based watcher configurations (manager + poller + handlers).
internal/mounts	rclone mount manager, lifecycle handlers, stats poller.
internal/webdav	WebDAV handler (mounted at /webdav/*; nginx handles caching upstream).

Startup sequence

1. Load configuration from environment
2. Build LeaderInfoProvider (no election — the bash gate already won)
3. Open the SQLite store (META_CORE_DB_PATH or /meta-core/db/meta-core.db)
4. EnsureSchemaVersion — abort if the existing store layout is stale
5. Start beacon v2 node (unless ENABLE_UDP_DISCOVERY=false)
6. Construct API server (identity keystore migration, watcher, watchers,
   mounts, WebDAV)
7. Start API server + pollers; start MetaPublisher + schema Indexer, then
   republish existing metadata onto meta:events in the background
   (5k events per transaction)
8. Wait for SIGINT/SIGTERM, shutdown in reverse order

Dependencies

Dependency	Purpose
github.com/gorilla/mux	HTTP router
modernc.org/sqlite	Embedded SQLite (pure Go, CGO off)
github.com/google/uuid	UUIDv7 minting
github.com/decred/dcrd/dcrec/secp256k1/v4, golang.org/x/crypto, github.com/mr-tron/base58	Identity keys, signing, AEAD key derivation
golang.org/x/net	WebDAV support

Runtime requirements: Go 1.21+ at build time; the container ships rclone, nginx, and supervisord alongside the binary, plus the React dashboard and metadata editor (built from dashboard/ and editor/ in the Dockerfile).

License

Part of the MetaMesh project.