beacon v2 discovery: advertise metamesh.core + service resources on 239.255.99.1:9099, /api/neighbors ?cap= filter

README.md

@@ -1,34 +1,42 @@
 # meta-core
 
-Standalone Go service that owns Redis, the leader-info file, the HTTP+SSE
+Standalone Go service that owns Redis, the HTTP+SSE metadata surface, the
-metadata surface, and a WebDAV file server for the MetaMesh dev/prod stack.
+User Data Layer store and signing keystore, and a WebDAV file server for the
+MetaMesh dev/prod stack.
 
 ## Overview
 
 meta-core runs as its own container (`metacore-app`). Other MetaMesh services
-(meta-sort, meta-fuse, meta-stremio, meta-dup, meta-share) discover it by
+(meta-sort, meta-fuse, meta-stremio, meta-dup, meta-search, meta-share,
-reading `/meta-core/locks/kv-leader.info` and then call its HTTP API. Redis
+meta-gateway, meta-watch, …) locate it over UDP multicast (beacon v2 —
-is not exposed outside the container — all metadata I/O is mediated by the
+its `metamesh.core` advertise carries the full `/urls` payload) and then call its
-HTTP API.
+HTTP API. `META_CORE_URL` on a client pins it instead. Redis is not published
+outside the container — all metadata I/O is mediated by the HTTP/SSE API
+(see [`docs/api-mediated-access.md`](docs/api-mediated-access.md)).
 
 ### Key capabilities
 
 | Feature | Description |
 |---------|-------------|
-| **Leader gate** | Bash `flock` loop in `docker/leader-election.sh` only `exec`s supervisord on the winner. The Go binary therefore only ever runs as leader. |
+| **Redis-volume mutex** | Bash `flock` loop in `docker/leader-election.sh` only `exec`s supervisord on the winner, so two containers sharing one `/meta-core` volume can never run two Redis writers. It is a mutex, not discovery — nothing reads the lock. |
 | **Redis owner** | Supervisord starts Redis with AOF + RDB persistence inside the container; meta-core connects to it on localhost. |
-| **HTTP API** | Typed REST surface (gorilla/mux) for metadata, KV browsing, snapshots, schema inference, mounts, watchers, services, and file access by CID. |
+| **HTTP API** | Typed REST surface (gorilla/mux) for metadata, KV browsing, snapshots, schema inference, mounts, watchers, neighbours, identity, UDL, and file access by CID. |
+| **In-memory search index** | `POST /api/metadata/search` matches against an in-memory snapshot of every record, kept current by incremental refreshes (full reconcile every 30 min) — no Redis round-trip per record on the request path (`internal/api/search_index.go`). |
 | **SSE event streams** | `/api/events/files` and `/api/events/meta` mirror the underlying Redis Streams so external services never speak Redis. |
 | **WebDAV server** | `/webdav/*` exposes the `/files` volume (read+write) for cross-service file access. nginx in front of meta-core handles caching. |
-| **Service discovery** | File-based registry under `/meta-core/services/*.json` with heartbeat + dead-service cleanup. |
+| **Service discovery** | beacon v2: UDP multicast advertise/listen on `239.255.99.1:9099`, neighbours served at `/neighbors`. The file-based registry under `/meta-core/services/*.json` it replaced is gone. |
-| **File watcher / scanner** | Recursive scan + MidHash256 computation; emits events on `file:events`. |
+| **File watcher / scanner** | Recursive scan + MidHash256 computation; emits events on `file:events`. Watched roots are POSTed watcher configs (`/meta-core/watchers.json`), not env vars. |
 | **Mount management** | rclone-only mount manager (SMB rendered into `:smb:`, plus pre-defined remotes). Read-only by construction. |
 | **UUID-rooted storage** | Roots are UUIDv7 (Crockford Base32, ULID layout); CIDs are reverse-index aliases. Schema-version sentinel refuses to boot against stale data. |
+| **Identity keystore** | Multi-account secp256k1 signing keys under `/meta-core/identity/accounts/`, created from the dashboard's Identity menu (never auto-generated). A private key is returned only by generate, or by reveal against a proof-of-possession signature (delete needs the same proof); `/api/identity/sign` refuses challenge-domain payloads. |
+| **User Data Layer** | Version-gated store for per-user signed (and, for private-tier keys, encrypted) records — My List, Continue Watching, ratings — indexed by user and CID. meta-core never inspects the crypto. See [User Data Layer](../../docs/project-architecture/user-data-layer.md). |
+| **Dashboard + editor** | nginx serves a React dashboard at `/` and the standalone metadata editor at `/editor/`. |
 
 ### Design characteristics
 
-- **Single-leader** — flock on a shared volume; no external consensus.
+- **Single writer** — flock on the `/meta-core` volume; no external consensus.
-- **HTTP-only externally** — Redis is bound to the container, never published.
+- **HTTP-only externally** — Redis is never published by the compose stacks;
+  debug with `docker exec metacore-app redis-cli …`.
 - **Schema-gated boot** — `EnsureSchemaVersion` aborts startup when the
   on-disk Redis layout predates the current build (alpha clean-wipe policy;
   see `docs/uuid-rooted-metadata.md`).
@@ -41,34 +49,31 @@ HTTP API.
 ┌──────────────────────────────────────────────────────────────────────────┐
 │                    metacore-app  (standalone container)                  │
 │                                                                          │
+│  leader-election.sh  (flock /meta-core/locks/kv-leader.lock,             │
+│                       then exec supervisord)                             │
 │  supervisord                                                             │
-│   ├─ leader-election.sh  (flock /meta-core/locks/kv-leader.lock,         │
-│   │                       writes kv-leader.info, exec supervisord)      │
 │   ├─ redis-server        (AOF + RDB on /meta-core/db/redis)              │
-│   ├─ nginx               (TLS termination, proxy_cache for /webdav)      │
+│   ├─ nginx  :80          (dashboard, /editor, API proxy,                 │
-│   ├─ rclone rcd          (mount daemon, RC API)                          │
+│   │                       proxy_cache for /webdav, rclone GUI /rclone)   │
+│   ├─ rclone rcd          (mount daemon, RC API on :5572)                 │
 │   ├─ mount-watcher.sh    (rclone mount lifecycle)                        │
 │   └─ meta-core (Go)      ─►  HTTP API + SSE + WebDAV on :9000            │
+│                          ─►  UDP advertise/listen 239.255.99.1:9099      │
 └──────────────────────────────────────────────────────────────────────────┘
                                        │
                 ┌──────────────────────┴──────────────────────┐
-                │  /meta-core (shared volume)                 │
+                │  /meta-core (meta-core's own volume)        │
-                │   locks/kv-leader.{lock,info}               │
+                │   locks/kv-leader.lock                      │
                 │   db/redis/                                 │
-                │   services/*.json                           │
+                │   identity/accounts/*.json                  │
                 │   mounts/{mounts.json, errors/}             │
                 │   watchers.json                             │
                 │   cache/ (nginx proxy_cache)                │
                 └─────────────────────────────────────────────┘
-                                       ▲
+
-        ┌──────────────────────────────┼────────────────────────────────┐
+   meta-sort / meta-fuse / meta-stremio / meta-search / meta-share / …
-        │                              │                                │
+      │  1. hear the metamesh.core advertise on UDP multicast (carries /urls)
-┌─────────────┐               ┌─────────────┐                ┌─────────────┐
+      └─ 2. HTTP + SSE (+ WebDAV) to metacore-app — never Redis, never the volume
-│  meta-sort  │               │  meta-fuse  │                │ meta-stremio│
-│             │               │             │                │             │
-│  reads      │  HTTP+SSE     │  reads      │  HTTP+SSE      │  reads      │
-│  leader.info├──────────────►│  leader.info├───────────────►│  leader.info│
-└─────────────┘               └─────────────┘                └─────────────┘
 ```
 
 ## Quick start
@@ -82,8 +87,9 @@ make test            # run Go unit tests
 ```
 
 `make docker` tags the image as `:1.0.0`, `:latest`, **and** `:local`. The
-`:local` tag is the one every sub-stack docker-compose.yml references
+`:local` tag is the one the sub-stack docker-compose files reference
-(meta-share 10-peer federation, meta-gateway dev stacks); without it the
+(meta-share federation, meta-read and meta-listen dev stacks); the meta-gateway
+dev stack (`docker-compose.gateway.yml`) uses `:latest`. Without these tags the
 sub-stacks can't see a freshly-built image by name.
 
 > **Source-change → recreate.** `make docker` rebuilds and re-tags the
@@ -94,12 +100,14 @@ sub-stacks can't see a freshly-built image by name.
 >
 > ```bash
 > make docker
-> docker compose -f packages/meta-gateway/dev/docker-compose.yml \
+> docker compose -f packages/meta-gateway/dev/docker-compose.gateway.yml \
->   up -d --force-recreate metagateway-generic-core
+>   up -d --force-recreate metagateway-hub-metacore
-> docker compose -f packages/meta-gateway/dev/docker-compose.torznab.yml \
->   up -d --force-recreate metagateway-torznab-core
 > docker compose -f packages/meta-share/dev/docker-compose.yml \
 >   --profile share-10node up -d --force-recreate metashare-core metashare-core-test
+> docker compose -f packages/meta-read/dev/docker-compose.yml \
+>   up -d --force-recreate metaread-dev-core
+> docker compose -f packages/meta-listen/dev/docker-compose.yml \
+>   up -d --force-recreate metalisten-dev-core
 > ```
 >
 > The symptom of a missed recreate is missing API fields on `/urls`
@@ -109,7 +117,7 @@ sub-stacks can't see a freshly-built image by name.
 >
 > ```bash
 > docker inspect meta-core:local --format '{{.Id}}'                # fresh SHA
-> docker ps --filter ancestor=meta-core:local --format '{{.Names}}'
+> docker ps --filter ancestor=meta-core:local --format '{{.Names}}'   # repeat for :latest
 > # any container whose Image SHA doesn't match the fresh SHA needs --force-recreate.
 > ```
 
@@ -126,36 +134,39 @@ HTTP port (no auth, bypasses Caddy) is `http://localhost:18083`.
 
 ## Configuration
 
-Environment variables consumed by `internal/config/config.go`:
+Environment variables (from `internal/config/config.go` unless noted):
 
 | Variable | Default | Description |
 |---|---|---|
-| `META_CORE_PATH` | `/meta-core` | Shared volume root (locks, db, services, mounts, cache). |
+| `META_CORE_PATH` | `/meta-core` | meta-core's own volume root (lock, Redis data, identity, mounts, watchers, cache). No other service mounts it. |
 | `FILES_PATH` | `/files` | Files volume root. |
-| `SERVICE_NAME` | `meta-core` | Identity in the service registry. |
+| `SERVICE_NAME` | `meta-core` | Name in the beacon v2 advertise. |
-| `SERVICE_VERSION` | `1.0.0` | Reported via `/status` and registry. |
+| `SERVICE_VERSION` | `1.0.0` | Reported via `/status`. |
-| `API_PORT` | `8180` | External port baked into the leader-info `baseUrl`. |
+| `API_PORT` | `8180` | External port baked into the constructed `baseUrl` when `BASE_URL` is unset. |
-| `BASE_URL` | _empty_ | Overrides the constructed `baseUrl` if set (used for HTTPS perimeter). |
+| `BASE_URL` | _empty_ | Overrides the constructed `baseUrl` (the Caddy/HTTPS perimeter URL). |
+| `META_CORE_PUBLIC_URL` | _empty_ | Browser-facing nav URL in the announce; wins over `BASE_URL` (e.g. a debug-direct port with no Caddy). |
 | `REDIS_PORT` | `6379` | Local Redis port. |
 | `META_CORE_HTTP_PORT` | `9000` | Go HTTP+SSE+WebDAV port. |
 | `META_CORE_HTTP_HOST` | `0.0.0.0` | HTTP bind. |
-| `HEALTH_CHECK_INTERVAL_MS` | `5000` | Internal health-loop cadence. |
+| `ENABLE_FILE_WATCHER` | `true` | Disable to suppress the in-process watcher entirely (