Compare commits
23
Commits
v1.0.5
...
27ed45999b
| Author | SHA1 | Date | |
|---|---|---|---|
 Worph
|
27ed45999b | ||
|
Worph
|
9254f043a4 | ||
|
Worph
|
ebc894fa42 | ||
|
Worph
|
515ec04369 | ||
|
Worph
|
0403a6f3bc | ||
|
Worph
|
355c666183 | ||
|
Worph
|
0cccceccd9 | ||
|
Worph
|
40b553016c | ||
|
Worph
|
4894810fd4 | ||
|
Worph
|
ac21315255 | ||
|
Worph
|
ec1b227631 | ||
|
Worph
|
8f4e08b6f1 | ||
|
Worph
|
0c4e3a2227 | ||
|
Worph
|
1f41ea0166 | ||
|
Worph
|
5efdd2c07d | ||
|
Worph
|
6ae87a2cdb | ||
|
Worph
|
604e718cdc | ||
|
Worph
|
0bbc656263 | ||
|
Worph
|
a98734171c | ||
|
Worph
|
a61057cd81 | ||
|
Worph
|
55d08e77ed | ||
|
Worph
|
1f084b817c | ||
|
Worph
|
80050cd2a8 |
@@ -1,34 +1,42 @@
|
||||
# meta-core
|
||||
|
||||
Standalone Go service that owns Redis, the leader-info file, the HTTP+SSE
|
||||
metadata surface, and a WebDAV file server for the MetaMesh dev/prod stack.
|
||||
Standalone Go service that owns Redis, the HTTP+SSE metadata surface, the
|
||||
User Data Layer store and signing keystore, and a WebDAV file server for the
|
||||
MetaMesh dev/prod stack.
|
||||
|
||||
## Overview
|
||||
|
||||
meta-core runs as its own container (`metacore-app`). Other MetaMesh services
|
||||
(meta-sort, meta-fuse, meta-stremio, meta-dup, meta-share) discover it by
|
||||
reading `/meta-core/locks/kv-leader.info` and then call its HTTP API. Redis
|
||||
is not exposed outside the container — all metadata I/O is mediated by the
|
||||
HTTP API.
|
||||
(meta-sort, meta-fuse, meta-stremio, meta-dup, meta-search, meta-share,
|
||||
meta-gateway, meta-watch, …) locate it over UDP multicast (beacon v2 —
|
||||
its `metamesh.core` advertise carries the full `/urls` payload) and then call its
|
||||
HTTP API. `META_CORE_URL` on a client pins it instead. Redis is not published
|
||||
outside the container — all metadata I/O is mediated by the HTTP/SSE API
|
||||
(see [`docs/api-mediated-access.md`](docs/api-mediated-access.md)).
|
||||
|
||||
### Key capabilities
|
||||
|
||||
| Feature | Description |
|
||||
|---------|-------------|
|
||||
| **Leader gate** | Bash `flock` loop in `docker/leader-election.sh` only `exec`s supervisord on the winner. The Go binary therefore only ever runs as leader. |
|
||||
| **Redis-volume mutex** | Bash `flock` loop in `docker/leader-election.sh` only `exec`s supervisord on the winner, so two containers sharing one `/meta-core` volume can never run two Redis writers. It is a mutex, not discovery — nothing reads the lock. |
|
||||
| **Redis owner** | Supervisord starts Redis with AOF + RDB persistence inside the container; meta-core connects to it on localhost. |
|
||||
| **HTTP API** | Typed REST surface (gorilla/mux) for metadata, KV browsing, snapshots, schema inference, mounts, watchers, services, and file access by CID. |
|
||||
| **HTTP API** | Typed REST surface (gorilla/mux) for metadata, KV browsing, snapshots, schema inference, mounts, watchers, neighbours, identity, UDL, and file access by CID. |
|
||||
| **In-memory search index** | `POST /api/metadata/search` matches against an in-memory snapshot of every record, kept current by incremental refreshes (full reconcile every 30 min) — no Redis round-trip per record on the request path (`internal/api/search_index.go`). |
|
||||
| **SSE event streams** | `/api/events/files` and `/api/events/meta` mirror the underlying Redis Streams so external services never speak Redis. |
|
||||
| **WebDAV server** | `/webdav/*` exposes the `/files` volume (read+write) for cross-service file access. nginx in front of meta-core handles caching. |
|
||||
| **Service discovery** | File-based registry under `/meta-core/services/*.json` with heartbeat + dead-service cleanup. |
|
||||
| **File watcher / scanner** | Recursive scan + MidHash256 computation; emits events on `file:events`. |
|
||||
| **Service discovery** | beacon v2: UDP multicast advertise/listen on `239.255.99.1:9099`, neighbours served at `/neighbors`. The file-based registry under `/meta-core/services/*.json` it replaced is gone. |
|
||||
| **File watcher / scanner** | Recursive scan + MidHash256 computation; emits events on `file:events`. Watched roots are POSTed watcher configs (`/meta-core/watchers.json`), not env vars. |
|
||||
| **Mount management** | rclone-only mount manager (SMB rendered into `:smb:`, plus pre-defined remotes). Read-only by construction. |
|
||||
| **UUID-rooted storage** | Roots are UUIDv7 (Crockford Base32, ULID layout); CIDs are reverse-index aliases. Schema-version sentinel refuses to boot against stale data. |
|
||||
| **Identity keystore** | Multi-account secp256k1 signing keys under `/meta-core/identity/accounts/`, created from the dashboard's Identity menu (never auto-generated). A private key is returned only by generate, or by reveal against a proof-of-possession signature (delete needs the same proof); `/api/identity/sign` refuses challenge-domain payloads. |
|
||||
| **User Data Layer** | Version-gated store for per-user signed (and, for private-tier keys, encrypted) records — My List, Continue Watching, ratings — indexed by user and CID. meta-core never inspects the crypto. See [User Data Layer](../../docs/project-architecture/user-data-layer.md). |
|
||||
| **Dashboard + editor** | nginx serves a React dashboard at `/` and the standalone metadata editor at `/editor/`. |
|
||||
|
||||
### Design characteristics
|
||||
|
||||
- **Single-leader** — flock on a shared volume; no external consensus.
|
||||
- **HTTP-only externally** — Redis is bound to the container, never published.
|
||||
- **Single writer** — flock on the `/meta-core` volume; no external consensus.
|
||||
- **HTTP-only externally** — Redis is never published by the compose stacks;
|
||||
debug with `docker exec metacore-app redis-cli …`.
|
||||
- **Schema-gated boot** — `EnsureSchemaVersion` aborts startup when the
|
||||
on-disk Redis layout predates the current build (alpha clean-wipe policy;
|
||||
see `docs/uuid-rooted-metadata.md`).
|
||||
@@ -41,34 +49,31 @@ HTTP API.
|
||||
┌──────────────────────────────────────────────────────────────────────────┐
|
||||
│ metacore-app (standalone container) │
|
||||
│ │
|
||||
│ leader-election.sh (flock /meta-core/locks/kv-leader.lock, │
|
||||
│ then exec supervisord) │
|
||||
│ supervisord │
|
||||
│ ├─ leader-election.sh (flock /meta-core/locks/kv-leader.lock, │
|
||||
│ │ writes kv-leader.info, exec supervisord) │
|
||||
│ ├─ redis-server (AOF + RDB on /meta-core/db/redis) │
|
||||
│ ├─ nginx (TLS termination, proxy_cache for /webdav) │
|
||||
│ ├─ rclone rcd (mount daemon, RC API) │
|
||||
│ ├─ nginx :80 (dashboard, /editor, API proxy, │
|
||||
│ │ proxy_cache for /webdav, rclone GUI /rclone) │
|
||||
│ ├─ rclone rcd (mount daemon, RC API on :5572) │
|
||||
│ ├─ mount-watcher.sh (rclone mount lifecycle) │
|
||||
│ └─ meta-core (Go) ─► HTTP API + SSE + WebDAV on :9000 │
|
||||
│ ─► UDP advertise/listen 239.255.99.1:9099 │
|
||||
└──────────────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
┌──────────────────────┴──────────────────────┐
|
||||
│ /meta-core (shared volume) │
|
||||
│ locks/kv-leader.{lock,info} │
|
||||
│ /meta-core (meta-core's own volume) │
|
||||
│ locks/kv-leader.lock │
|
||||
│ db/redis/ │
|
||||
│ services/*.json │
|
||||
│ identity/accounts/*.json │
|
||||
│ mounts/{mounts.json, errors/} │
|
||||
│ watchers.json │
|
||||
│ cache/ (nginx proxy_cache) │
|
||||
└─────────────────────────────────────────────┘
|
||||
▲
|
||||
┌──────────────────────────────┼────────────────────────────────┐
|
||||
│ │ │
|
||||
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
|
||||
│ meta-sort │ │ meta-fuse │ │ meta-stremio│
|
||||
│ │ │ │ │ │
|
||||
│ reads │ HTTP+SSE │ reads │ HTTP+SSE │ reads │
|
||||
│ leader.info├──────────────►│ leader.info├───────────────►│ leader.info│
|
||||
└─────────────┘ └─────────────┘ └─────────────┘
|
||||
|
||||
meta-sort / meta-fuse / meta-stremio / meta-search / meta-share / …
|
||||
│ 1. hear the metamesh.core advertise on UDP multicast (carries /urls)
|
||||
└─ 2. HTTP + SSE (+ WebDAV) to metacore-app — never Redis, never the volume
|
||||
```
|
||||
|
||||
## Quick start
|
||||
@@ -82,8 +87,9 @@ make test # run Go unit tests
|
||||
```
|
||||
|
||||
`make docker` tags the image as `:1.0.0`, `:latest`, **and** `:local`. The
|
||||
`:local` tag is the one every sub-stack docker-compose.yml references
|
||||
(meta-share 10-peer federation, meta-gateway dev stacks); without it the
|
||||
`:local` tag is the one the sub-stack docker-compose files reference
|
||||
(meta-share federation, meta-read and meta-listen dev stacks); the meta-gateway
|
||||
dev stack (`docker-compose.gateway.yml`) uses `:latest`. Without these tags the
|
||||
sub-stacks can't see a freshly-built image by name.
|
||||
|
||||
> **Source-change → recreate.** `make docker` rebuilds and re-tags the
|
||||
@@ -94,12 +100,14 @@ sub-stacks can't see a freshly-built image by name.
|
||||
>
|
||||
> ```bash
|
||||
> make docker
|
||||
> docker compose -f packages/meta-gateway/dev/docker-compose.yml \
|
||||
> up -d --force-recreate metagateway-generic-core
|
||||
> docker compose -f packages/meta-gateway/dev/docker-compose.torznab.yml \
|
||||
> up -d --force-recreate metagateway-torznab-core
|
||||
> docker compose -f packages/meta-gateway/dev/docker-compose.gateway.yml \
|
||||
> up -d --force-recreate metagateway-hub-metacore
|
||||
> docker compose -f packages/meta-share/dev/docker-compose.yml \
|
||||
> --profile share-10node up -d --force-recreate metashare-core metashare-core-test
|
||||
> docker compose -f packages/meta-read/dev/docker-compose.yml \
|
||||
> up -d --force-recreate metaread-dev-core
|
||||
> docker compose -f packages/meta-listen/dev/docker-compose.yml \
|
||||
> up -d --force-recreate metalisten-dev-core
|
||||
> ```
|
||||
>
|
||||
> The symptom of a missed recreate is missing API fields on `/urls`
|
||||
@@ -109,7 +117,7 @@ sub-stacks can't see a freshly-built image by name.
|
||||
>
|
||||
> ```bash
|
||||
> docker inspect meta-core:local --format '{{.Id}}' # fresh SHA
|
||||
> docker ps --filter ancestor=meta-core:local --format '{{.Names}}'
|
||||
> docker ps --filter ancestor=meta-core:local --format '{{.Names}}' # repeat for :latest
|
||||
> # any container whose Image SHA doesn't match the fresh SHA needs --force-recreate.
|
||||
> ```
|
||||
|
||||
@@ -126,36 +134,39 @@ HTTP port (no auth, bypasses Caddy) is `http://localhost:18083`.
|
||||
|
||||
## Configuration
|
||||
|
||||
Environment variables consumed by `internal/config/config.go`:
|
||||
Environment variables (from `internal/config/config.go` unless noted):
|
||||
|
||||
| Variable | Default | Description |
|
||||
|---|---|---|
|
||||
| `META_CORE_PATH` | `/meta-core` | Shared volume root (locks, db, services, mounts, cache). |
|
||||
| `META_CORE_PATH` | `/meta-core` | meta-core's own volume root (lock, Redis data, identity, mounts, watchers, cache). No other service mounts it. |
|
||||
| `FILES_PATH` | `/files` | Files volume root. |
|
||||
| `SERVICE_NAME` | `meta-core` | Identity in the service registry. |
|
||||
| `SERVICE_VERSION` | `1.0.0` | Reported via `/status` and registry. |
|
||||
| `API_PORT` | `8180` | External port baked into the leader-info `baseUrl`. |
|
||||
| `BASE_URL` | _empty_ | Overrides the constructed `baseUrl` if set (used for HTTPS perimeter). |
|
||||
| `SERVICE_NAME` | `meta-core` | Name in the beacon v2 advertise. |
|
||||
| `SERVICE_VERSION` | `1.0.0` | Reported via `/status`. |
|
||||
| `API_PORT` | `8180` | External port baked into the constructed `baseUrl` when `BASE_URL` is unset. |
|
||||
| `BASE_URL` | _empty_ | Overrides the constructed `baseUrl` (the Caddy/HTTPS perimeter URL). |
|
||||
| `META_CORE_PUBLIC_URL` | _empty_ | Browser-facing nav URL in the announce; wins over `BASE_URL` (e.g. a debug-direct port with no Caddy). |
|
||||
| `REDIS_PORT` | `6379` | Local Redis port. |
|
||||
| `META_CORE_HTTP_PORT` | `9000` | Go HTTP+SSE+WebDAV port. |
|
||||
| `META_CORE_HTTP_HOST` | `0.0.0.0` | HTTP bind. |
|
||||
| `HEALTH_CHECK_INTERVAL_MS` | `5000` | Internal health-loop cadence. |
|
||||
| `HEARTBEAT_INTERVAL_MS` | `30000` | Service registry heartbeat. |
|
||||
| `STALE_THRESHOLD_MS` | `60000` | Age past which a registry entry is marked stale. |
|
||||
| `CLEANUP_INTERVAL_MS` | `600000` | Dead-service cleaner cadence. |
|
||||
| `DEAD_SERVICE_THRESHOLD_MS` | `600000` | Age past which a stale entry is deleted. |
|
||||
| `WATCH_INTERVAL_MS` | `1000` | Watcher poll cadence. |
|
||||
| `DEBOUNCE_MS` | `30000` | File-event debounce window. |
|
||||
| `ENABLE_FILE_WATCHER` | `true` | Disable to suppress the in-process watcher entirely. |
|
||||
| `ENABLE_FILE_WATCHER` | `true` | Disable to suppress the in-process watcher entirely (`POST /api/files/register` stays available). |
|
||||
| `ENABLE_UDP_DISCOVERY` | `true` | beacon v2 advertise + listen. |
|
||||
| `BEACON_GROUP` / `BEACON_PORT` | `239.255.99.1` / `9099` | Multicast group and UDP port. |
|
||||
| `BEACON_INTERVAL_MS` | `10000` | Unsolicited advertise interval. |
|
||||
| `META_CORE_INDEX_EXCLUDE_PREFIXES` | `categories/newznab/` | Comma-separated field prefixes left out of the in-memory search index; set empty to index everything (`internal/storage/client.go`). |
|
||||
| `ELECTION_RETRY_SECS` | `5` | flock retry interval (read by `docker/leader-election.sh`, not the Go binary). |
|
||||
|
||||
Note: there is no `META_CORE_REDIS_PORT`, no `META_CORE_SERVICE_NAME`, and
|
||||
no `META_CORE_LEADER_TIMEOUT`. The Redis URL is discovered locally; service
|
||||
identity uses `SERVICE_NAME`; leader timeout is governed by the bash flock
|
||||
retry interval (`ELECTION_RETRY_SECS`, default `5`).
|
||||
`HEALTH_CHECK_INTERVAL_MS`, `HEARTBEAT_INTERVAL_MS`, `STALE_THRESHOLD_MS`,
|
||||
`CLEANUP_INTERVAL_MS`, `DEAD_SERVICE_THRESHOLD_MS`, `WATCH_INTERVAL_MS` and
|
||||
`DEBOUNCE_MS` are still parsed by `internal/config/config.go` but nothing reads
|
||||
them (leftovers of the file-based registry); setting them has no effect. There
|
||||
is no `META_CORE_REDIS_PORT` / `META_CORE_SERVICE_NAME` /
|
||||
`META_CORE_LEADER_TIMEOUT`, and no `WATCH_PATHS` — watched folders are watcher
|
||||
configs created via `/api/watchers`.
|
||||
|
||||
## API reference
|
||||
|
||||
All endpoints below are served by `internal/api/server.go`. Examples assume
|
||||
All endpoints below are registered in `internal/api/server.go` (mounts,
|
||||
watcher and watchers routes in their own packages' `handlers.go`). Examples assume
|
||||
the in-container Go port `:9000`; from the host use
|
||||
`https://metacore-dev.localhost:8083` (Caddy) or `http://localhost:18083`
|
||||
(debug-direct).
|
||||
@@ -167,12 +178,43 @@ GET /health # storage + role
|
||||
GET /status # version, uptime, etc.
|
||||
GET /leader # current leader info (hostname, PID, timestamps)
|
||||
GET /urls # baseUrl, apiUrl, webdavUrl, webdavUrlInternal (redisUrl is intentionally empty)
|
||||
GET /services # registered services + heartbeats
|
||||
GET /api/services # alias for /services
|
||||
GET /services/{name} # one service
|
||||
GET /services/cleanup/stats
|
||||
GET /neighbors # neighbours heard over UDP (beacon v2)
|
||||
GET /api/neighbors # same, under /api
|
||||
GET /api/stats # dashboard counters (see below)
|
||||
```
|
||||
|
||||
`/services`, `/api/services`, `/services/{name}` and `/services/cleanup/stats`
|
||||
are **gone** — the registration-and-heartbeat registry they served was replaced
|
||||
by beacon v2 (`/neighbors`). nginx answers an unknown non-`/api` path
|
||||
with the dashboard's `index.html`, so a stale caller gets `200 text/html` rather
|
||||
than a 404; that is what made the dashboard fail with
|
||||
`SyntaxError: Unexpected token '<'` until 1.0.21.
|
||||
|
||||
#### `GET /api/stats`
|
||||
|
||||
Every headline counter in one request, split by cost:
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"records": 88709, // SCARD file:__index__ — metadata roots, file or not
|
||||
"redisKeys": 3115447, // DBSIZE — the flat per-field keys dominate
|
||||
"redisMemory": "588.74M",
|
||||
"identities": 3, // signing accounts on disk
|
||||
"files": { "count": 4864, "totalSize": 64587914424 }, // null until first sweep
|
||||
"udlUsers": 5, // accounts holding User Data Layer records
|
||||
"sweptAt": 1789402401970,
|
||||
"sweeping": false
|
||||
}
|
||||
```
|
||||
|
||||
The first four are O(1) commands and are live on every call. `files` and
|
||||
`udlUsers` need a keyspace walk (~0.5s and ~3s respectively on an 88k-record
|
||||
box), so they are served from a single-flight cache refreshed in the background
|
||||
with a 60s TTL — the walk is **never** on the request path, because the
|
||||
dashboard polls and ten open tabs must not cost ten Redis walks. Prefer this
|
||||
over `/api/kv/info`, which does one `GET` per root (6s+) and sums `sizeByte`
|
||||
across records that share a file.
|
||||
|
||||
### Metadata — primary surface
|
||||
|
||||
These are the routes other services use. `{hash}` may be a UUID root or
|
||||
@@ -198,6 +240,8 @@ GET /api/meta/{cid} # resolve CID → metadata document
|
||||
GET /api/file/{cid}/info # CID resolution metadata
|
||||
GET /file/{cid} # serve file bytes (range-aware)
|
||||
POST /file/cid # compute CID for an uploaded file
|
||||
POST /api/files/register # mint + alias one file under /files/plugin by midhash
|
||||
# (the watcher's path for one file; works with the watcher OFF)
|
||||
HEAD /data/{hash} # existence + size
|
||||
GET /data/{hash}/path # absolute path on /files
|
||||
```
|
||||
@@ -209,6 +253,7 @@ the `/file/{cid}` reverse-lookup path.)
|
||||
|
||||
```bash
|
||||
GET /api/metadata/hash-ids
|
||||
GET /api/files/tuples
|
||||
GET /api/metadata/list
|
||||
POST /api/metadata/search
|
||||
POST /api/metadata/batch
|
||||
@@ -248,6 +293,44 @@ signalling).
|
||||
```bash
|
||||
GET /api/events/files # file:events stream (watcher events)
|
||||
GET /api/events/meta # meta:events stream (metadata mutations)
|
||||
GET /api/events/poll # DEPRECATED long-poll over file:events — use SSE
|
||||
```
|
||||
|
||||
### Identity (signing keystore)
|
||||
|
||||
Reached through meta-search's identity proxy (and directly by meta-gateway,
|
||||
meta-read, meta-listen) for record signing, UDL writes and keypair sign-in.
|
||||
|
||||
```bash
|
||||
GET /api/identity # status (never the private key)
|
||||
GET /api/identity/accounts
|
||||
POST /api/identity/generate # the only call that returns a new private key
|
||||
POST /api/identity/import
|
||||
POST /api/identity/challenge # one-shot challenge for proof-of-possession
|
||||
POST /api/identity/reveal # needs proof-of-possession
|
||||
DELETE /api/identity # needs proof-of-possession; reports purged UDL counts
|
||||
POST /api/identity/sign
|
||||
POST /api/identity/sign-batch
|
||||
GET /api/identity/aead-key
|
||||
```
|
||||
|
||||
### User Data Layer
|
||||
|
||||
Records arrive signed (private-tier ones encrypted) as opaque base64 CBOR;
|
||||
writes are version-gated (stale → `409 stale_version`, or `accepted:false` per
|
||||
cell in a batch). Redis model in `internal/storage/udl.go`.
|
||||
|
||||
```bash
|
||||
GET /api/udl/record?uid&cid&key
|
||||
PUT /api/udl/record
|
||||
PUT /api/udl/records # batch
|
||||
GET /api/udl/user/{uid}/key/{key}
|
||||
GET /api/udl/user/{uid}/cid/{cid}
|
||||
GET /api/udl/user/{uid}/all
|
||||
GET /api/udl/user/{uid}/count
|
||||
GET /api/udl/cid/{cid}/users
|
||||
GET /api/udl/cid/{cid}/aggregate # meta-watch Aggregate wire shape
|
||||
GET /api/udl/users/stats
|
||||
```
|
||||
|
||||
### Mounts
|
||||
@@ -287,38 +370,42 @@ return a deprecation pointer at the `/api/watchers/*` routes
|
||||
|
||||
```bash
|
||||
POST /api/admin/migrate-dual-roots # reunify stranded midhash-rooted entries
|
||||
POST /api/admin/migrate-domain-screen # domain film|tv → screen, workForm backfill (idempotent)
|
||||
POST /api/admin/migrate-user-cids # backfill udl:idx:user:<uid>:cids; until run, pre-index
|
||||
# profiles export as EMPTY. Returns {fixed}
|
||||
POST /api/admin/sweep-literature-echoes[?apply=true]
|
||||
# strip query echoes off literature cards: anilistid on site cards,
|
||||
# workcid/chapterNumber/volumeNumber/chapterStart/chapterEnd on any card.
|
||||
# Dry-run unless apply=true; returns {dryRun, planned, fixed, byKey}
|
||||
GET/PUT/DELETE /webdav/... # mounted on /files
|
||||
```
|
||||
|
||||
No `/metrics` endpoint is exposed; observability is via container logs and
|
||||
the `/health` / `/status` JSON.
|
||||
|
||||
## Leader gate
|
||||
## Redis-volume mutex
|
||||
|
||||
Leader election is a bash flock loop in `docker/leader-election.sh`, not a
|
||||
Go-level state machine. When `flock` succeeds on `kv-leader.lock`, the
|
||||
script writes the leader API URL to `kv-leader.info` and `exec`s
|
||||
supervisord. When the process tree dies, the kernel releases the lock and
|
||||
any standby instance can take over. There is no follower path inside the
|
||||
Go binary — there is no `OnBecomeLeader` / `OnBecomeFollower` callback API.
|
||||
`docker/leader-election.sh` is a bash flock loop, not a Go-level state
|
||||
machine. When `flock` succeeds on `/meta-core/locks/kv-leader.lock`, the
|
||||
script `exec`s supervisord; a second container mounting the same volume
|
||||
retries every `ELECTION_RETRY_SECS` and stays dormant. When the process tree
|
||||
dies, the kernel releases the lock. There is no follower path inside the Go
|
||||
binary.
|
||||
|
||||
`kv-leader.info` is a single plaintext line containing the API base URL:
|
||||
|
||||
```
|
||||
http://10.0.1.50:9000
|
||||
```
|
||||
|
||||
Other services read this file, then call `GET /urls` for full discovery.
|
||||
|
||||
The dev compose runs a single `metacore-app`; the flock primitive is
|
||||
retained as a correctness anchor (one process owns Redis and the
|
||||
writable state under `/meta-core`), not as live failover.
|
||||
The lock is a **mutex, not service discovery** — it only stops two Redis
|
||||
writers landing on one RDB/AOF. Nothing reads it, and the `kv-leader.info`
|
||||
file it used to publish is gone: siblings locate meta-core over UDP
|
||||
(see [beacon-v2.md](../../docs/project-architecture/beacon-v2.md)).
|
||||
`internal/leader` keeps its name for history, but only builds the `/urls`
|
||||
payload (`baseUrl`, `apiUrl`, `webdavUrl`, `webdavUrlInternal`) that `/urls`
|
||||
and the UDP announce share.
|
||||
|
||||
## Metadata storage shape (one-line summary)
|
||||
|
||||
Roots are opaque UUIDv7 strings; every property is a separate Redis
|
||||
STRING at `file:<uuid>/<property>`; every known CID for a file is a
|
||||
reverse-index entry `cid:<algorithm>:<value> → <uuid>`. Full details in
|
||||
STRING at `file:<uuid>/<property>`; every known CID is a key-set member
|
||||
`file:<uuid>/cids/<bareCid>` plus a reverse-index entry `cid:<bareCid> → <uuid>`
|
||||
(the canonical CID is derived by rank on read, never stored). Full details in
|
||||
[`docs/metadata-storage-structure.md`](docs/metadata-storage-structure.md)
|
||||
and design rationale in [`docs/uuid-rooted-metadata.md`](docs/uuid-rooted-metadata.md).
|
||||
The authoritative registry of field semantics and value formats lives in
|
||||
@@ -328,14 +415,15 @@ the repo-root [`METADATA_KEYS.md`](../../METADATA_KEYS.md).
|
||||
|
||||
| Package | Purpose |
|
||||
|---------|---------|
|
||||
| `cmd/meta-core` | Entry point; Redis connect + schema sentinel + service wiring. |
|
||||
| `cmd/meta-core` | Entry point; Redis connect + schema sentinel + UDP discovery + service wiring. |
|
||||
| `internal/config` | Env-driven configuration, path helpers. |
|
||||
| `internal/leader` | `LeaderInfoProvider` (builds the leader info from local hostname/IP/config; no election logic). |
|
||||
| `internal/storage` | Redis wrapper (`client.go`), UUIDv7 minting (`uuid.go`), CID reverse index (`cid_resolution.go`), schema sentinel (`schema_version.go`), dual-root migration (`dual_root_migration.go`). |
|
||||
| `internal/cid` | CID parsing + algorithm ranking for canonical-CID selection. |
|
||||
| `internal/discovery` | Service registry + dead-service cleaner. |
|
||||
| `internal/leader` | `LeaderInfoProvider` — builds the `/urls` payload from hostname/IP/config; no election logic. |
|
||||
| `internal/meshdisco` | beacon v2 node (advertise, probe, neighbour map) — the Go port of the spec. |
|
||||
| `internal/storage` | Redis wrapper (`client.go`), UUIDv7 minting, CID reverse index (`cid_resolution.go`), schema sentinel, field indexes + search-index bulk reads, UDL store (`udl.go`, `udl_admin.go`), stats, one-shot migrations/sweeps. |
|
||||
| `internal/cid` | CID parsing + rank ladder for canonical-CID selection (`cid-rank-vectors.json`, guarded by the meta-root `scripts/check-cid-vectors.sh`). |
|
||||
| `internal/identity` | Multi-account secp256k1 keystore, challenge/proof-of-possession, signature verification. |
|
||||
| `internal/events` | Keyspace-notification → `meta:events` stream publisher (`meta_publisher.go`). |
|
||||
| `internal/api` | HTTP server + all handlers + SSE event endpoints. |
|
||||
| `internal/api` | HTTP server + all handlers + SSE event endpoints + in-memory search index + `/api/stats`. |
|
||||
| `internal/schema` | Live per-field schema indexer (consumes `meta:events`). |
|
||||
| `internal/snapshot` | Snapshot export / import / wipe. |
|
||||
| `internal/watcher` | File system scanner, MidHash256 computation, dispatcher, state registry. |
|
||||
@@ -350,10 +438,12 @@ the repo-root [`METADATA_KEYS.md`](../../METADATA_KEYS.md).
|
||||
2. Build LeaderInfoProvider (no election — the bash gate already won)
|
||||
3. Connect to local Redis (retry up to 30× / 1s)
|
||||
4. EnsureSchemaVersion — abort if the existing Redis layout is stale
|
||||
5. Start service discovery + dead-service cleaner
|
||||
6. Construct API server (initialises watcher, mounts, WebDAV)
|
||||
7. Start API server; on first connect, start MetaPublisher + schema Indexer
|
||||
and republish existing metadata onto meta:events
|
||||
5. Start beacon v2 node (unless ENABLE_UDP_DISCOVERY=false)
|
||||
6. Construct API server (identity keystore migration, watcher, watchers,
|
||||
mounts, WebDAV)
|
||||
7. Start API server + pollers + search-index refresh loop; start
|
||||
MetaPublisher + schema Indexer, then warm field indexes and republish
|
||||
existing metadata onto meta:events in the background
|
||||
8. Wait for SIGINT/SIGTERM, shutdown in reverse order
|
||||
```
|
||||
|
||||
@@ -364,11 +454,13 @@ the repo-root [`METADATA_KEYS.md`](../../METADATA_KEYS.md).
|
||||
| `github.com/gorilla/mux` | HTTP router |
|
||||
| `github.com/redis/go-redis/v9` | Redis client |
|
||||
| `github.com/google/uuid` | UUIDv7 minting |
|
||||
| `github.com/decred/dcrd/dcrec/secp256k1/v4`, `golang.org/x/crypto`, `github.com/mr-tron/base58` | Identity keys, signing, AEAD key derivation |
|
||||
| `golang.org/x/net` | WebDAV support |
|
||||
| Go stdlib | context, net/http, os, sync, syscall |
|
||||
| `github.com/alicebob/miniredis/v2` | In-memory Redis for tests |
|
||||
|
||||
Runtime requirements: Go 1.21+ at build time; the container ships Redis,
|
||||
rclone, nginx, and supervisord alongside the binary.
|
||||
rclone, nginx, and supervisord alongside the binary, plus the React dashboard
|
||||
and metadata editor (built from `dashboard/` and `editor/` in the Dockerfile).
|
||||
|
||||
## License
|
||||
|
||||
|
||||
+66
-16
@@ -1,6 +1,7 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"log"
|
||||
"os"
|
||||
@@ -10,8 +11,8 @@ import (
|
||||
|
||||
"github.com/metazla/meta-core/internal/api"
|
||||
"github.com/metazla/meta-core/internal/config"
|
||||
"github.com/metazla/meta-core/internal/discovery"
|
||||
"github.com/metazla/meta-core/internal/leader"
|
||||
"github.com/metazla/meta-core/internal/meshdisco"
|
||||
"github.com/metazla/meta-core/internal/storage"
|
||||
)
|
||||
|
||||
@@ -60,20 +61,57 @@ func main() {
|
||||
}
|
||||
log.Printf("[meta-core] Redis schema version OK (v%d)", storage.SchemaVersion)
|
||||
|
||||
// Create and start service discovery
|
||||
disc := discovery.NewService(cfg)
|
||||
if err := disc.Start(); err != nil {
|
||||
log.Fatalf("[meta-core] Failed to start service discovery: %v", err)
|
||||
// Beacon v2 (docs/project-architecture/beacon-v2.md): advertise meta-core
|
||||
// as metamesh.core — carrying the /urls block services locate us by — plus
|
||||
// its own metamesh.service/<name> nav entry. The payload is rebuilt from
|
||||
// leaderProvider on every tick, which keeps it and GET /urls from drifting.
|
||||
var mesh *meshdisco.Node
|
||||
if cfg.EnableUDPDiscovery {
|
||||
hostname, _ := os.Hostname()
|
||||
mesh = meshdisco.New(meshdisco.Config{
|
||||
Name: cfg.ServiceName,
|
||||
Instance: hostname,
|
||||
Version: Version,
|
||||
Group: cfg.DiscoveryGroup,
|
||||
Port: cfg.DiscoveryPort,
|
||||
Interval: time.Duration(cfg.DiscoveryIntervalMS) * time.Millisecond,
|
||||
Payload: func() (string, []meshdisco.Resource) {
|
||||
service := meshdisco.Resource{
|
||||
ID: "service",
|
||||
Caps: []string{meshdisco.CapServicePrefix + cfg.ServiceName},
|
||||
Endpoints: map[string]string{"ui": coreBaseURL(cfg)},
|
||||
}
|
||||
|
||||
// Create and start dead service cleaner
|
||||
cleaner := discovery.NewCleaner(cfg)
|
||||
if err := cleaner.Start(); err != nil {
|
||||
log.Printf("[meta-core] Warning: failed to start service cleaner: %v", err)
|
||||
info := leaderProvider.LeaderInfo()
|
||||
if info == nil {
|
||||
return "starting", []meshdisco.Resource{service}
|
||||
}
|
||||
data, _ := json.Marshal(map[string]meshdisco.URLs{"urls": {
|
||||
Hostname: info.Hostname,
|
||||
BaseUrl: info.BaseUrl,
|
||||
ApiUrl: info.ApiUrl,
|
||||
WebdavUrl: info.WebdavUrl,
|
||||
WebdavUrlInternal: info.WebdavUrlInternal,
|
||||
}})
|
||||
core := meshdisco.Resource{
|
||||
ID: "core",
|
||||
Caps: []string{meshdisco.CapCore},
|
||||
Endpoints: map[string]string{"http": info.ApiUrl},
|
||||
Data: data,
|
||||
}
|
||||
return "running", []meshdisco.Resource{core, service}
|
||||
},
|
||||
})
|
||||
if err := mesh.Start(); err != nil {
|
||||
log.Printf("[meta-core] Warning: UDP discovery disabled: %v", err)
|
||||
mesh = nil
|
||||
}
|
||||
} else {
|
||||
log.Println("[meta-core] UDP discovery disabled by ENABLE_UDP_DISCOVERY")
|
||||
}
|
||||
|
||||
// Create and start API server
|
||||
apiServer := api.NewServer(cfg, leaderProvider, disc, cleaner, storageClient)
|
||||
apiServer := api.NewServer(cfg, leaderProvider, storageClient)
|
||||
apiServer.SetMeshNode(mesh)
|
||||
if err := apiServer.Start(); err != nil {
|
||||
log.Fatalf("[meta-core] Failed to start API server: %v", err)
|
||||
}
|
||||
@@ -90,12 +128,10 @@ func main() {
|
||||
log.Printf("[meta-core] Error stopping API server: %v", err)
|
||||
}
|
||||
|
||||
if err := cleaner.Stop(); err != nil {
|
||||
log.Printf("[meta-core] Error stopping service cleaner: %v", err)
|
||||
if mesh != nil {
|
||||
if err := mesh.Stop(); err != nil {
|
||||
log.Printf("[meta-core] Error stopping UDP discovery: %v", err)
|
||||
}
|
||||
|
||||
if err := disc.Stop(); err != nil {
|
||||
log.Printf("[meta-core] Error stopping service discovery: %v", err)
|
||||
}
|
||||
|
||||
if err := storageClient.Close(); err != nil {
|
||||
@@ -112,3 +148,17 @@ func waitForShutdown() {
|
||||
<-sigChan
|
||||
}
|
||||
|
||||
// coreBaseURL is the browser-facing URL announced for the nav menu. It keeps
|
||||
// the three-way fallback the file registry used (discovery/service.go):
|
||||
// META_CORE_PUBLIC_URL wins (the reachable URL when there is no Caddy
|
||||
// perimeter, e.g. a debug-direct port), then BASE_URL (the Caddy URL), then
|
||||
// the container IP.
|
||||
func coreBaseURL(cfg *config.Config) string {
|
||||
if cfg.PublicURL != "" {
|
||||
return cfg.PublicURL
|
||||
}
|
||||
if cfg.BaseURL != "" {
|
||||
return cfg.BaseURL
|
||||
}
|
||||
return fmt.Sprintf("http://%s:%d", meshdisco.LocalIPv4(), cfg.APIPort)
|
||||
}
|
||||
|
||||
Vendored
+1
-1
@@ -21,7 +21,7 @@
|
||||
min-height: 100vh;
|
||||
}
|
||||
</style>
|
||||
<script type="module" crossorigin src="/assets/index-Cbd1uH5A.js"></script>
|
||||
<script type="module" crossorigin src="/assets/index-Bgo6moAc.js"></script>
|
||||
<link rel="stylesheet" crossorigin href="/assets/index-DtR_2V4s.css">
|
||||
</head>
|
||||
<body>
|
||||
|
||||
Generated
+10
@@ -8,6 +8,7 @@
|
||||
"name": "@meta-core/dashboard",
|
||||
"version": "1.0.0",
|
||||
"dependencies": {
|
||||
"@noble/secp256k1": "^3.1.0",
|
||||
"iso-639-3": "^3.0.1",
|
||||
"react": "^18.2.0",
|
||||
"react-dom": "^18.2.0",
|
||||
@@ -744,6 +745,15 @@
|
||||
"@jridgewell/sourcemap-codec": "^1.4.14"
|
||||
}
|
||||
},
|
||||
"node_modules/@noble/secp256k1": {
|
||||
"version": "3.1.0",
|
||||
"resolved": "https://registry.npmjs.org/@noble/secp256k1/-/secp256k1-3.1.0.tgz",
|
||||
"integrity": "sha512-+F7iS7tUMaNGXcc9X3PjmjvuQnXEuSjCRNzVVA2xAcKXgCaP0dHYz4SFyt4FKNHef7sOP//xihowcySSS7PK9g==",
|
||||
"license": "MIT",
|
||||
"funding": {
|
||||
"url": "https://paulmillr.com/funding/"
|
||||
}
|
||||
},
|
||||
"node_modules/@remix-run/router": {
|
||||
"version": "1.23.2",
|
||||
"resolved": "https://registry.npmjs.org/@remix-run/router/-/router-1.23.2.tgz",
|
||||
|
||||
@@ -9,10 +9,11 @@
|
||||
"preview": "vite preview"
|
||||
},
|
||||
"dependencies": {
|
||||
"@noble/secp256k1": "^3.1.0",
|
||||
"iso-639-3": "^3.0.1",
|
||||
"react": "^18.2.0",
|
||||
"react-dom": "^18.2.0",
|
||||
"react-router-dom": "^6.20.0",
|
||||
"iso-639-3": "^3.0.1"
|
||||
"react-router-dom": "^6.20.0"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/react": "^18.2.38",
|
||||
|
||||
@@ -1,155 +1,53 @@
|
||||
import { useState, useEffect } from 'react';
|
||||
import { useEffect } from 'react';
|
||||
// eslint-disable-next-line @typescript-eslint/ban-ts-comment
|
||||
// @ts-ignore -- plain ES module, no .d.ts; imported for its side effect
|
||||
// of defining the custom element.
|
||||
import '../meta-service-menu.js';
|
||||
|
||||
interface ServiceInfo {
|
||||
name: string;
|
||||
hostname: string;
|
||||
baseUrl: string;
|
||||
status: string;
|
||||
lastHeartbeat: string;
|
||||
}
|
||||
/**
|
||||
* ServiceNav — the neighbouring meta-* services, as a burger menu.
|
||||
*
|
||||
* Since meta-discovery v1 this is a thin wrapper around <meta-service-menu>,
|
||||
* a framework-free custom element shared by every meta-* UI (see
|
||||
* scripts/check-mirrors.sh). The old implementation was ~165 lines of React
|
||||
* that existed in four near-identical copies and polled meta-core's
|
||||
* /api/services; the element polls this service's own /api/neighbors instead,
|
||||
* so the nav still renders when meta-core is down.
|
||||
*
|
||||
* The element lives in shadow DOM and cannot see this app's CSS variables, so
|
||||
* the four --mm-nav-* tokens below map our palette onto it.
|
||||
*/
|
||||
|
||||
interface ServicesResponse {
|
||||
services: ServiceInfo[];
|
||||
current: string;
|
||||
}
|
||||
|
||||
const serviceIcons: Record<string, string> = {
|
||||
'meta-core': '🗄️',
|
||||
'meta-sort': '📁',
|
||||
'meta-fuse': '🗂️',
|
||||
'meta-stremio': '🎬',
|
||||
'meta-orbit': '🌐',
|
||||
'default': '📦'
|
||||
};
|
||||
|
||||
function formatServiceName(name: string): string {
|
||||
return name.split('-').map(word =>
|
||||
word.charAt(0).toUpperCase() + word.slice(1)
|
||||
).join(' ');
|
||||
declare global {
|
||||
// eslint-disable-next-line @typescript-eslint/no-namespace
|
||||
namespace JSX {
|
||||
interface IntrinsicElements {
|
||||
'meta-service-menu': React.DetailedHTMLProps<
|
||||
React.HTMLAttributes<HTMLElement>,
|
||||
HTMLElement
|
||||
> & { current?: string; endpoint?: string; label?: string };
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function ServiceNav() {
|
||||
const [services, setServices] = useState<ServiceInfo[]>([]);
|
||||
const [loading, setLoading] = useState(true);
|
||||
const currentService = 'meta-core';
|
||||
|
||||
useEffect(() => {
|
||||
const fetchServices = async () => {
|
||||
try {
|
||||
const response = await fetch('/services');
|
||||
if (response.ok) {
|
||||
const data: ServicesResponse = await response.json();
|
||||
setServices(data.services || []);
|
||||
}
|
||||
} catch (error) {
|
||||
console.error('Failed to fetch services:', error);
|
||||
} finally {
|
||||
setLoading(false);
|
||||
}
|
||||
};
|
||||
|
||||
fetchServices();
|
||||
const interval = setInterval(fetchServices, 30000);
|
||||
return () => clearInterval(interval);
|
||||
// Import above is for the side effect of defining the element; this
|
||||
// keeps bundlers from tree-shaking it away in production builds.
|
||||
}, []);
|
||||
|
||||
if (loading) return null;
|
||||
|
||||
const sortedServices = [...services].sort((a, b) => a.name.localeCompare(b.name));
|
||||
|
||||
if (sortedServices.length === 0) return null;
|
||||
|
||||
return (
|
||||
<nav className="services-nav">
|
||||
<span className="services-nav-label">Services:</span>
|
||||
<div className="services-nav-items">
|
||||
{sortedServices.map(service => {
|
||||
const icon = serviceIcons[service.name] || serviceIcons.default;
|
||||
const isActive = service.name === currentService;
|
||||
return (
|
||||
<a
|
||||
key={service.name}
|
||||
href={isActive ? '#' : service.baseUrl}
|
||||
className={`service-link${isActive ? ' active' : ''}`}
|
||||
onClick={(e) => {
|
||||
e.preventDefault();
|
||||
if (!isActive) {
|
||||
window.location.href = service.baseUrl;
|
||||
}
|
||||
}}
|
||||
>
|
||||
<span className="service-icon">{icon}</span>
|
||||
<span>{formatServiceName(service.name)}</span>
|
||||
<span className="service-status"></span>
|
||||
</a>
|
||||
);
|
||||
})}
|
||||
</div>
|
||||
|
||||
<>
|
||||
<meta-service-menu current="meta-core" />
|
||||
<style>{`
|
||||
.services-nav {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 0.5rem;
|
||||
}
|
||||
|
||||
.services-nav-label {
|
||||
color: #888;
|
||||
font-size: 0.8rem;
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
.services-nav-items {
|
||||
display: flex;
|
||||
gap: 0.35rem;
|
||||
}
|
||||
|
||||
.service-link {
|
||||
display: inline-flex;
|
||||
align-items: center;
|
||||
gap: 0.3rem;
|
||||
padding: 0.35rem 0.6rem;
|
||||
border-radius: 6px;
|
||||
text-decoration: none;
|
||||
font-size: 0.8rem;
|
||||
font-weight: 500;
|
||||
transition: all 0.2s;
|
||||
color: #ccc;
|
||||
background: #0f3460;
|
||||
border: 1px solid #1a4a7a;
|
||||
}
|
||||
|
||||
.service-link:hover {
|
||||
background: #1a4a7a;
|
||||
border-color: rgba(78, 205, 196, 0.5);
|
||||
text-decoration: none;
|
||||
color: #fff;
|
||||
}
|
||||
|
||||
.service-link.active {
|
||||
background: linear-gradient(135deg, rgba(78, 205, 196, 0.2), rgba(68, 160, 141, 0.2));
|
||||
border-color: rgba(78, 205, 196, 0.5);
|
||||
color: #fff;
|
||||
}
|
||||
|
||||
.service-link .service-icon {
|
||||
font-size: 0.9rem;
|
||||
}
|
||||
|
||||
.service-link .service-status {
|
||||
width: 5px;
|
||||
height: 5px;
|
||||
border-radius: 50%;
|
||||
background: #4ecdc4;
|
||||
}
|
||||
|
||||
@media (max-width: 800px) {
|
||||
.services-nav-label {
|
||||
display: none;
|
||||
}
|
||||
meta-service-menu {
|
||||
--mm-nav-fg: var(--text-primary, #e0e0e0);
|
||||
--mm-nav-bg: var(--bg-tertiary, var(--bg-secondary, #1a1a2e));
|
||||
--mm-nav-border: var(--border-color, rgba(255,255,255,0.14));
|
||||
--mm-nav-accent: var(--accent-primary, #4ecdc4);
|
||||
}
|
||||
`}</style>
|
||||
</nav>
|
||||
</>
|
||||
);
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,28 @@
|
||||
// JSON fetch that fails with a sentence instead of a parser error.
|
||||
//
|
||||
// nginx serves the dashboard with `try_files $uri $uri/ /index.html`, so ANY
|
||||
// path it does not have a proxy rule for answers 200 text/html with the SPA
|
||||
// shell. A caller that does `res.ok && res.json()` on such a path gets
|
||||
// `SyntaxError: Unexpected token '<', "<!doctype "... is not valid JSON` and
|
||||
// no clue which request produced it — which is exactly how a route deleted
|
||||
// from the Go router (`/services`, removed in favour of /api/neighbors) sat
|
||||
// broken on the dashboard.
|
||||
//
|
||||
// So: check the content type, and name the URL in every error.
|
||||
|
||||
export async function getJSON<T>(url: string, init?: RequestInit): Promise<T> {
|
||||
const res = await fetch(url, init);
|
||||
const ct = res.headers.get('content-type') || '';
|
||||
|
||||
if (!res.ok) {
|
||||
throw new Error(`${url} → HTTP ${res.status}`);
|
||||
}
|
||||
if (!ct.includes('application/json')) {
|
||||
// The SPA fallback is the overwhelmingly likely cause; say so.
|
||||
throw new Error(
|
||||
`${url} → ${res.status} ${ct || 'no content-type'} (not JSON — the route is ` +
|
||||
`missing and nginx served the dashboard shell instead)`
|
||||
);
|
||||
}
|
||||
return (await res.json()) as T;
|
||||
}
|
||||
@@ -0,0 +1,127 @@
|
||||
// Client-side proof of possession for account operations.
|
||||
//
|
||||
// meta-core stores every private key in plaintext, so it can sign as anybody —
|
||||
// which means a signature it produced proves nothing about who asked. The only
|
||||
// signature that carries authority is one made where the key actually is, and
|
||||
// for a browser-driven admin console that is here. meta-core enforces the other
|
||||
// half by refusing to sign anything in the challenge domain
|
||||
// (identity.ChallengeDomain), so this path cannot be short-circuited.
|
||||
//
|
||||
// The secret key entered on this page is used for exactly one signature and is
|
||||
// never sent anywhere: `/api/identity/challenge` receives only the uid, and the
|
||||
// request that follows carries only the challenge and its signature.
|
||||
//
|
||||
// Same scheme, same curve and same encodings as meta-watch's sign-in
|
||||
// (packages/meta-watch/ui/js/auth.js) — deliberately, so one implementation on
|
||||
// the server verifies both.
|
||||
|
||||
import * as secp from '@noble/secp256k1';
|
||||
|
||||
export type IdentityAction = 'reveal' | 'delete';
|
||||
|
||||
function hexToBytes(hex: string): Uint8Array | null {
|
||||
let h = hex.trim();
|
||||
if (h.startsWith('0x') || h.startsWith('0X')) h = h.slice(2);
|
||||
if (h.length !== 64 || /[^0-9a-fA-F]/.test(h)) return null;
|
||||
const out = new Uint8Array(32);
|
||||
for (let i = 0; i < 32; i++) out[i] = parseInt(h.slice(i * 2, i * 2 + 2), 16);
|
||||
return out;
|
||||
}
|
||||
|
||||
function bytesToBase64(bytes: Uint8Array): string {
|
||||
let s = '';
|
||||
for (const b of bytes) s += String.fromCharCode(b);
|
||||
return btoa(s);
|
||||
}
|
||||
|
||||
const B58 = '123456789ABCDEFGHJKLMNPQRSTUVWXYZabcdefghijkmnopqrstuvwxyz';
|
||||
|
||||
/// base58btc, the multibase "z" alphabet. Spelled out rather than pulled in:
|
||||
/// the alternative is a second dependency for thirty lines of arithmetic.
|
||||
function base58Encode(bytes: Uint8Array): string {
|
||||
const digits: number[] = [0];
|
||||
for (const byte of bytes) {
|
||||
let carry = byte;
|
||||
for (let i = 0; i < digits.length; i++) {
|
||||
carry += digits[i] << 8;
|
||||
digits[i] = carry % 58;
|
||||
carry = (carry / 58) | 0;
|
||||
}
|
||||
while (carry) {
|
||||
digits.push(carry % 58);
|
||||
carry = (carry / 58) | 0;
|
||||
}
|
||||
}
|
||||
// Leading zero bytes encode as leading '1's rather than being dropped.
|
||||
let out = '';
|
||||
for (let i = 0; i < bytes.length && bytes[i] === 0; i++) out += B58[0];
|
||||
for (let i = digits.length - 1; i >= 0; i--) out += B58[digits[i]];
|
||||
return out;
|
||||
}
|
||||
|
||||
/// The uid a secret key belongs to: "z" + base58btc(33-byte compressed pubkey).
|
||||
/// Derived locally so a mistyped or wrong-account key is caught before any
|
||||
/// request goes out. Mirrors `identityFromPrivKey` in meta-core's identity.go.
|
||||
export function uidFromSecretKey(secretKeyHex: string): string | null {
|
||||
const raw = hexToBytes(secretKeyHex);
|
||||
if (!raw) return null;
|
||||
try {
|
||||
return 'z' + base58Encode(secp.getPublicKey(raw, true));
|
||||
} catch {
|
||||
return null; // not a valid scalar
|
||||
}
|
||||
}
|
||||
|
||||
/// Whether this page can sign at all.
|
||||
///
|
||||
/// noble's signAsync takes its sha256 from crypto.subtle, which browsers expose
|
||||
/// only in a secure context. The dashboard behind Caddy is https and fine; the
|
||||
/// debug-direct port (18083) is plain http, where every signature would fail
|
||||
/// with a cryptic error unless we say so first.
|
||||
export function canSign(): boolean {
|
||||
return Boolean(globalThis.crypto?.subtle);
|
||||
}
|
||||
|
||||
export const INSECURE_CONTEXT_MESSAGE =
|
||||
'This page cannot sign over a plain-http connection — browsers only expose the ' +
|
||||
'needed crypto on https (or localhost). Reach the dashboard through Caddy ' +
|
||||
'(https://metacore-dev.localhost:8083) rather than the debug-direct port.';
|
||||
|
||||
/// Ask meta-core for a challenge, sign it locally, and hand back the proof.
|
||||
///
|
||||
/// Throws with a message fit to show the user. The caller passes the proof
|
||||
/// straight into the reveal/delete request body.
|
||||
export async function proveOwnership(
|
||||
uid: string,
|
||||
action: IdentityAction,
|
||||
secretKeyHex: string
|
||||
): Promise<{ challenge: string; signature: string }> {
|
||||
if (!canSign()) throw new Error(INSECURE_CONTEXT_MESSAGE);
|
||||
|
||||
const raw = hexToBytes(secretKeyHex);
|
||||
if (!raw) {
|
||||
throw new Error('That is not a valid secret key: expected 64 hex characters.');
|
||||
}
|
||||
const derived = uidFromSecretKey(secretKeyHex);
|
||||
if (!derived) throw new Error('That is not a valid secp256k1 secret key.');
|
||||
if (derived !== uid) {
|
||||
// Caught here rather than by the server, so the message can say which
|
||||
// account the key actually is — the useful half of the answer.
|
||||
throw new Error(`That key belongs to a different account (${derived}), not this one.`);
|
||||
}
|
||||
|
||||
const cRes = await fetch('/api/identity/challenge', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ uid, action }),
|
||||
});
|
||||
const cBody = await cRes.json().catch(() => ({}));
|
||||
if (!cRes.ok || !cBody.challenge) {
|
||||
throw new Error(cBody.message ?? `Could not start verification (HTTP ${cRes.status}).`);
|
||||
}
|
||||
|
||||
// signAsync prehashes with sha256 and returns a 64-byte compact (r‖s)
|
||||
// signature — noble v3 dropped DER. meta-core's identity.Verify accepts both.
|
||||
const sig = await secp.signAsync(new TextEncoder().encode(cBody.challenge), raw);
|
||||
return { challenge: cBody.challenge, signature: bytesToBase64(sig) };
|
||||
}
|
||||
@@ -0,0 +1,308 @@
|
||||
/**
|
||||
* <meta-service-menu> — the neighbouring meta-* services, as a burger menu.
|
||||
*
|
||||
* Replaces the eight divergent "services bar" implementations (four copies of
|
||||
* a React `ServiceNav.tsx`, two mirrored `services.js`, two bespoke inline
|
||||
* ones) with a single framework-free custom element. It drops into the React
|
||||
* dashboards as a plain tag — no React dependency — and into the vanilla UIs
|
||||
* as an ES module.
|
||||
*
|
||||
* Data comes from the host service's own `GET /api/neighbors`, same-origin.
|
||||
* Never cross-origin: the dev stack's self-signed certs and the OIDC redirects
|
||||
* make a fetch to a neighbour a dead end. Links are plain navigations.
|
||||
*
|
||||
* ⚠ MIRRORED FILE. Byte-identical copies live in every meta-* UI tree; see
|
||||
* `scripts/check-mirrors.sh`. Edit one → edit all.
|
||||
*
|
||||
* Shadow DOM is not optional here. The eight UIs use four incompatible token
|
||||
* vocabularies, and two of them define *different colours under the same
|
||||
* names* (`--bg-secondary` is #12121a in meta-sort, #16213e in meta-core,
|
||||
* #161b22 in meta-dup). meta-search and meta-share are also the only
|
||||
* light-mode-capable UIs, so a hardcoded dark palette would break them.
|
||||
* The element therefore styles itself from four tokens with dark defaults:
|
||||
*
|
||||
* --mm-nav-fg, --mm-nav-bg, --mm-nav-border, --mm-nav-accent
|
||||
*
|
||||
* Each host maps its own variables onto those, e.g.
|
||||
* meta-service-menu { --mm-nav-bg: var(--bg-secondary); }
|
||||
*
|
||||
* Attributes:
|
||||
* current name of the host service, so it can be marked and not linked
|
||||
* endpoint override the default "/api/neighbors"
|
||||
* label button tooltip (default "Services")
|
||||
*
|
||||
* Property:
|
||||
* rewriteUrl optional (name, url, neighbour) => url, for hosts that must
|
||||
* translate internal hostnames into browser-reachable ones
|
||||
* (meta-gateway does this for its dev/prod split).
|
||||
*/
|
||||
|
||||
const ICONS = {
|
||||
"meta-core": "⚙️",
|
||||
"meta-sort": "📁",
|
||||
"meta-fuse": "🗂️",
|
||||
"meta-stremio": "🎬",
|
||||
"meta-dup": "🔍",
|
||||
"meta-search": "🔎",
|
||||
"meta-share": "🔗",
|
||||
"meta-gateway": "🌉",
|
||||
"meta-watch": "📺",
|
||||
"meta-listen": "🎧",
|
||||
"meta-read": "📖",
|
||||
};
|
||||
const DEFAULT_ICON = "📦";
|
||||
const POLL_MS = 30000;
|
||||
|
||||
function titleCase(name) {
|
||||
return String(name || "")
|
||||
.split("-")
|
||||
.map((w) => w.charAt(0).toUpperCase() + w.slice(1))
|
||||
.join(" ");
|
||||
}
|
||||
|
||||
const TEMPLATE = `
|
||||
<style>
|
||||
:host {
|
||||
--_fg: var(--mm-nav-fg, #e8eaed);
|
||||
--_bg: var(--mm-nav-bg, #1a1a2e);
|
||||
--_border: var(--mm-nav-border, rgba(255, 255, 255, 0.14));
|
||||
--_accent: var(--mm-nav-accent, #4ecdc4);
|
||||
position: relative;
|
||||
display: inline-block;
|
||||
font-family: inherit;
|
||||
font-size: 14px;
|
||||
line-height: 1.4;
|
||||
}
|
||||
button.trigger {
|
||||
display: inline-flex;
|
||||
align-items: center;
|
||||
justify-content: center;
|
||||
gap: 0.4rem;
|
||||
width: 2.1rem;
|
||||
height: 2.1rem;
|
||||
padding: 0;
|
||||
cursor: pointer;
|
||||
color: var(--_fg);
|
||||
background: transparent;
|
||||
border: 1px solid var(--_border);
|
||||
border-radius: 8px;
|
||||
font: inherit;
|
||||
transition: border-color 0.15s, background 0.15s;
|
||||
}
|
||||
button.trigger:hover { border-color: var(--_accent); }
|
||||
button.trigger:focus-visible {
|
||||
outline: 2px solid var(--_accent);
|
||||
outline-offset: 2px;
|
||||
}
|
||||
button.trigger[aria-expanded="true"] { border-color: var(--_accent); }
|
||||
.bars { display: block; width: 15px; }
|
||||
.bars span {
|
||||
display: block;
|
||||
height: 2px;
|
||||
margin: 3px 0;
|
||||
border-radius: 2px;
|
||||
background: currentColor;
|
||||
}
|
||||
.panel {
|
||||
position: absolute;
|
||||
top: calc(100% + 6px);
|
||||
right: 0;
|
||||
z-index: 1000;
|
||||
min-width: 13rem;
|
||||
max-height: 70vh;
|
||||
overflow-y: auto;
|
||||
padding: 0.3rem;
|
||||
background: var(--_bg);
|
||||
border: 1px solid var(--_border);
|
||||
border-radius: 10px;
|
||||
box-shadow: 0 10px 30px rgba(0, 0, 0, 0.35);
|
||||
}
|
||||
.panel[hidden] { display: none; }
|
||||
a.item, .item {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 0.55rem;
|
||||
padding: 0.45rem 0.6rem;
|
||||
border-radius: 6px;
|
||||
color: var(--_fg);
|
||||
text-decoration: none;
|
||||
white-space: nowrap;
|
||||
}
|
||||
a.item:hover { background: color-mix(in srgb, var(--_accent) 18%, transparent); }
|
||||
.item.active { color: var(--_accent); font-weight: 600; cursor: default; }
|
||||
.item.dead { opacity: 0.55; cursor: not-allowed; }
|
||||
.icon { font-size: 1rem; line-height: 1; }
|
||||
.dot {
|
||||
width: 6px;
|
||||
height: 6px;
|
||||
margin-left: auto;
|
||||
border-radius: 50%;
|
||||
background: var(--_accent);
|
||||
}
|
||||
.empty {
|
||||
padding: 0.6rem;
|
||||
opacity: 0.7;
|
||||
font-size: 0.85em;
|
||||
white-space: nowrap;
|
||||
}
|
||||
/* Respect a host that opts out of motion. */
|
||||
@media (prefers-reduced-motion: reduce) {
|
||||
button.trigger { transition: none; }
|
||||
}
|
||||
</style>
|
||||
<button class="trigger" type="button" aria-haspopup="menu" aria-expanded="false">
|
||||
<span class="bars" aria-hidden="true"><span></span><span></span><span></span></span>
|
||||
</button>
|
||||
<div class="panel" role="menu" hidden></div>
|
||||
`;
|
||||
|
||||
class MetaServiceMenu extends HTMLElement {
|
||||
constructor() {
|
||||
super();
|
||||
this.attachShadow({ mode: "open" });
|
||||
this.shadowRoot.innerHTML = TEMPLATE;
|
||||
this._btn = this.shadowRoot.querySelector("button.trigger");
|
||||
this._panel = this.shadowRoot.querySelector(".panel");
|
||||
this._neighbors = [];
|
||||
this._timer = null;
|
||||
this._open = false;
|
||||
|
||||
this._onDocClick = (e) => {
|
||||
if (!this._open) return;
|
||||
if (e.composedPath().includes(this)) return;
|
||||
this._setOpen(false);
|
||||
};
|
||||
this._onKey = (e) => {
|
||||
if (e.key === "Escape" && this._open) {
|
||||
this._setOpen(false);
|
||||
this._btn.focus();
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
connectedCallback() {
|
||||
this._btn.title = this.getAttribute("label") || "Services";
|
||||
this._btn.setAttribute("aria-label", this._btn.title);
|
||||
this._btn.addEventListener("click", () => this._setOpen(!this._open));
|
||||
document.addEventListener("click", this._onDocClick);
|
||||
document.addEventListener("keydown", this._onKey);
|
||||
|
||||
this.refresh();
|
||||
this._timer = setInterval(() => this.refresh(), POLL_MS);
|
||||
}
|
||||
|
||||
disconnectedCallback() {
|
||||
if (this._timer) clearInterval(this._timer);
|
||||
this._timer = null;
|
||||
document.removeEventListener("click", this._onDocClick);
|
||||
document.removeEventListener("keydown", this._onKey);
|
||||
}
|
||||
|
||||
get endpoint() {
|
||||
return this.getAttribute("endpoint") || "/api/neighbors";
|
||||
}
|
||||
|
||||
get current() {
|
||||
return this.getAttribute("current") || "";
|
||||
}
|
||||
|
||||
async refresh() {
|
||||
try {
|
||||
const res = await fetch(this.endpoint, {
|
||||
headers: { Accept: "application/json" },
|
||||
});
|
||||
if (!res.ok) throw new Error(String(res.status));
|
||||
const data = await res.json();
|
||||
// `services` is the legacy alias some backends still emit.
|
||||
this._neighbors = data.neighbors || data.services || [];
|
||||
} catch {
|
||||
// Silent: the menu is navigation, not a health indicator. Keep whatever
|
||||
// we last had rather than blanking the list on one failed poll.
|
||||
}
|
||||
this._render();
|
||||
}
|
||||
|
||||
_setOpen(open) {
|
||||
this._open = open;
|
||||
this._panel.hidden = !open;
|
||||
this._btn.setAttribute("aria-expanded", String(open));
|
||||
if (open) this.refresh();
|
||||
}
|
||||
|
||||
_render() {
|
||||
const panel = this._panel;
|
||||
panel.textContent = "";
|
||||
|
||||
const seen = new Set();
|
||||
const rows = [];
|
||||
for (const n of this._neighbors) {
|
||||
const name = n.name || "";
|
||||
if (!name || seen.has(name)) continue;
|
||||
// Beacon v2 neighbours include plugins; the nav lists services only.
|
||||
// (A row without `caps` comes from a pre-v2 backend — keep it.)
|
||||
const isService = (c) => String(c).startsWith("metamesh.service/");
|
||||
if (Array.isArray(n.caps) && !n.caps.some(isService)) continue;
|
||||
seen.add(name);
|
||||
rows.push(n);
|
||||
}
|
||||
// Always show the host itself, even before its own echo comes back.
|
||||
if (this.current && !seen.has(this.current)) {
|
||||
rows.push({ name: this.current, baseUrl: "" });
|
||||
}
|
||||
rows.sort((a, b) => String(a.name).localeCompare(String(b.name)));
|
||||
|
||||
if (rows.length === 0) {
|
||||
const empty = document.createElement("div");
|
||||
empty.className = "empty";
|
||||
empty.textContent = "No services found";
|
||||
panel.appendChild(empty);
|
||||
return;
|
||||
}
|
||||
|
||||
for (const n of rows) {
|
||||
const isActive = n.name === this.current;
|
||||
let url = n.baseUrl || "";
|
||||
if (typeof this.rewriteUrl === "function") {
|
||||
try {
|
||||
url = this.rewriteUrl(n.name, url, n) || "";
|
||||
} catch {
|
||||
/* a bad hook must not blank the menu */
|
||||
}
|
||||
}
|
||||
|
||||
// A neighbour with no reachable URL is shown but not linked — better
|
||||
// than a link that goes nowhere.
|
||||
const el = document.createElement(isActive || !url ? "span" : "a");
|
||||
el.className =
|
||||
"item" + (isActive ? " active" : "") + (!isActive && !url ? " dead" : "");
|
||||
if (el.tagName === "A") {
|
||||
el.href = url;
|
||||
el.setAttribute("role", "menuitem");
|
||||
}
|
||||
if (!isActive && !url) el.title = "No reachable URL advertised";
|
||||
|
||||
const icon = document.createElement("span");
|
||||
icon.className = "icon";
|
||||
icon.textContent = ICONS[n.name] || DEFAULT_ICON;
|
||||
|
||||
const label = document.createElement("span");
|
||||
label.textContent = titleCase(n.name);
|
||||
|
||||
el.append(icon, label);
|
||||
// Beacon v2 omits `status` when running.
|
||||
if ((n.status || "running") === "running" || isActive) {
|
||||
const dot = document.createElement("span");
|
||||
dot.className = "dot";
|
||||
dot.title = n.status || "running";
|
||||
el.appendChild(dot);
|
||||
}
|
||||
panel.appendChild(el);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (!customElements.get("meta-service-menu")) {
|
||||
customElements.define("meta-service-menu", MetaServiceMenu);
|
||||
}
|
||||
|
||||
export default MetaServiceMenu;
|
||||
@@ -1,4 +1,5 @@
|
||||
import { useEffect, useState } from 'react';
|
||||
import { getJSON } from '../lib/api';
|
||||
|
||||
interface HealthStatus {
|
||||
status: string;
|
||||
@@ -6,6 +7,22 @@ interface HealthStatus {
|
||||
timestamp: string;
|
||||
}
|
||||
|
||||
interface FileStats {
|
||||
count: number;
|
||||
totalSize: number;
|
||||
}
|
||||
|
||||
interface Stats {
|
||||
records: number;
|
||||
redisKeys: number;
|
||||
redisMemory: string;
|
||||
identities: number;
|
||||
files: FileStats | null;
|
||||
udlUsers: number | null;
|
||||
sweptAt: number;
|
||||
sweeping: boolean;
|
||||
}
|
||||
|
||||
interface WatcherStatus {
|
||||
id: string;
|
||||
path: string;
|
||||
@@ -22,13 +39,6 @@ interface WatchersResponse {
|
||||
count: number;
|
||||
}
|
||||
|
||||
interface ServiceInfo {
|
||||
name: string;
|
||||
api: string;
|
||||
capabilities: string[];
|
||||
timestamp: number;
|
||||
}
|
||||
|
||||
const cardStyle: React.CSSProperties = {
|
||||
background: '#16213e',
|
||||
borderRadius: '8px',
|
||||
@@ -38,53 +48,110 @@ const cardStyle: React.CSSProperties = {
|
||||
|
||||
const gridStyle: React.CSSProperties = {
|
||||
display: 'grid',
|
||||
gridTemplateColumns: 'repeat(auto-fit, minmax(300px, 1fr))',
|
||||
// 200px, not 220: five tiles must fit one row inside the 1200px container
|
||||
// (1136 content - 4x16 gap = 1072 / 5 = 214 each) before wrapping.
|
||||
gridTemplateColumns: 'repeat(auto-fit, minmax(200px, 1fr))',
|
||||
gap: '1rem',
|
||||
marginBottom: '1rem',
|
||||
};
|
||||
|
||||
const statStyle: React.CSSProperties = {
|
||||
fontSize: '2rem',
|
||||
fontWeight: 'bold',
|
||||
color: '#4ade80',
|
||||
lineHeight: 1.2,
|
||||
};
|
||||
|
||||
const labelStyle: React.CSSProperties = {
|
||||
marginBottom: '0.75rem',
|
||||
color: '#888',
|
||||
fontSize: '0.85rem',
|
||||
textTransform: 'uppercase',
|
||||
letterSpacing: '0.04em',
|
||||
};
|
||||
|
||||
const noteStyle: React.CSSProperties = {
|
||||
color: '#6b7a99',
|
||||
fontSize: '0.8rem',
|
||||
marginTop: '0.4rem',
|
||||
};
|
||||
|
||||
const numberFmt = new Intl.NumberFormat();
|
||||
|
||||
function formatCount(n: number | null | undefined): string {
|
||||
return n === null || n === undefined ? '—' : numberFmt.format(n);
|
||||
}
|
||||
|
||||
// Binary units: this counts bytes on disk, and every other tool the operator
|
||||
// compares against (du, docker, the mount page) reports GiB.
|
||||
function formatBytes(bytes: number | null | undefined): string {
|
||||
if (bytes === null || bytes === undefined) return '—';
|
||||
const units = ['B', 'KiB', 'MiB', 'GiB', 'TiB', 'PiB'];
|
||||
let value = bytes;
|
||||
let unit = 0;
|
||||
while (value >= 1024 && unit < units.length - 1) {
|
||||
value /= 1024;
|
||||
unit++;
|
||||
}
|
||||
return `${value.toFixed(value >= 100 || unit === 0 ? 0 : 1)} ${units[unit]}`;
|
||||
}
|
||||
|
||||
function Tile({
|
||||
label,
|
||||
value,
|
||||
note,
|
||||
pending,
|
||||
}: {
|
||||
label: string;
|
||||
value: string;
|
||||
note?: string;
|
||||
pending?: boolean;
|
||||
}) {
|
||||
return (
|
||||
<div style={cardStyle}>
|
||||
<div style={labelStyle}>{label}</div>
|
||||
<div style={{ ...statStyle, color: pending ? '#6b7a99' : statStyle.color }}>
|
||||
{value}
|
||||
</div>
|
||||
{note && <div style={noteStyle}>{note}</div>}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
export default function Dashboard() {
|
||||
const [health, setHealth] = useState<HealthStatus | null>(null);
|
||||
const [stats, setStats] = useState<Stats | null>(null);
|
||||
const [watchersData, setWatchersData] = useState<WatchersResponse | null>(null);
|
||||
const [services, setServices] = useState<ServiceInfo[]>([]);
|
||||
const [error, setError] = useState<string | null>(null);
|
||||
|
||||
useEffect(() => {
|
||||
let cancelled = false;
|
||||
|
||||
const fetchData = async () => {
|
||||
try {
|
||||
// Fetch health
|
||||
const healthRes = await fetch('/health');
|
||||
if (healthRes.ok) {
|
||||
setHealth(await healthRes.json());
|
||||
}
|
||||
// Settled, not all: one dead endpoint must not blank every tile.
|
||||
const [healthRes, statsRes, watchersRes] = await Promise.allSettled([
|
||||
getJSON<HealthStatus>('/health'),
|
||||
getJSON<Stats>('/api/stats'),
|
||||
getJSON<WatchersResponse>('/api/watchers'),
|
||||
]);
|
||||
if (cancelled) return;
|
||||
|
||||
// Fetch watchers status
|
||||
const watchersRes = await fetch('/api/watchers');
|
||||
if (watchersRes.ok) {
|
||||
setWatchersData(await watchersRes.json());
|
||||
}
|
||||
if (healthRes.status === 'fulfilled') setHealth(healthRes.value);
|
||||
if (statsRes.status === 'fulfilled') setStats(statsRes.value);
|
||||
if (watchersRes.status === 'fulfilled') setWatchersData(watchersRes.value);
|
||||
|
||||
// Fetch services
|
||||
const servicesRes = await fetch('/services');
|
||||
if (servicesRes.ok) {
|
||||
const data = await servicesRes.json();
|
||||
setServices(data.services || []);
|
||||
}
|
||||
|
||||
setError(null);
|
||||
} catch (err) {
|
||||
setError(String(err));
|
||||
}
|
||||
const failures = [healthRes, statsRes, watchersRes]
|
||||
.filter((r): r is PromiseRejectedResult => r.status === 'rejected')
|
||||
.map((r) => String(r.reason?.message ?? r.reason));
|
||||
setError(failures.length ? failures.join(' · ') : null);
|
||||
};
|
||||
|
||||
fetchData();
|
||||
const interval = setInterval(fetchData, 5000);
|
||||
return () => clearInterval(interval);
|
||||
return () => {
|
||||
cancelled = true;
|
||||
clearInterval(interval);
|
||||
};
|
||||
}, []);
|
||||
|
||||
const handleTriggerScan = async () => {
|
||||
@@ -95,49 +162,102 @@ export default function Dashboard() {
|
||||
}
|
||||
};
|
||||
|
||||
// Aggregate watcher stats
|
||||
const totalFiles = watchersData?.watchers.reduce((sum, w) => sum + w.fileCount, 0) || 0;
|
||||
const isScanning = watchersData?.watchers.some(w => w.isScanning) || false;
|
||||
const activeWatchers = watchersData?.watchers.filter(w => w.active).length || 0;
|
||||
const lastScan = watchersData?.watchers.reduce((max, w) => Math.max(max, w.lastScan || 0), 0) || 0;
|
||||
const isScanning = watchersData?.watchers.some((w) => w.isScanning) || false;
|
||||
const activeWatchers = watchersData?.watchers.filter((w) => w.active).length || 0;
|
||||
const lastScan =
|
||||
watchersData?.watchers.reduce((max, w) => Math.max(max, w.lastScan || 0), 0) || 0;
|
||||
|
||||
// Records we hold metadata for but have no local file: gateway/cid-rooted
|
||||
// entries, plus anything whose file went away. The interesting number on a
|
||||
// client box, where it should be most of them.
|
||||
const fileless =
|
||||
stats && stats.files ? Math.max(0, stats.records - stats.files.count) : null;
|
||||
|
||||
const sweepPending = !!stats && !stats.files;
|
||||
const sweptNote = (() => {
|
||||
if (!stats) return undefined;
|
||||
if (sweepPending) return stats.sweeping ? 'scanning…' : 'not scanned yet';
|
||||
if (stats.sweeping) return 'refreshing…';
|
||||
if (stats.sweptAt) return `as of ${new Date(stats.sweptAt).toLocaleTimeString()}`;
|
||||
return undefined;
|
||||
})();
|
||||
|
||||
return (
|
||||
<div style={{ padding: '2rem', maxWidth: '1200px', margin: '0 auto' }}>
|
||||
<div style={{ padding: '2rem', maxWidth: '1200px', margin: '0 auto', overflowY: 'auto', height: '100%' }}>
|
||||
<h1 style={{ marginBottom: '2rem' }}>meta-core Dashboard</h1>
|
||||
|
||||
{error && (
|
||||
<div style={{ ...cardStyle, background: '#4a1a1a', color: '#f87171' }}>
|
||||
Error: {error}
|
||||
{error}
|
||||
</div>
|
||||
)}
|
||||
|
||||
<div style={gridStyle}>
|
||||
{/* Health Card */}
|
||||
<Tile
|
||||
label="Files discovered"
|
||||
value={formatCount(stats?.files?.count)}
|
||||
note={sweptNote}
|
||||
pending={sweepPending}
|
||||
/>
|
||||
<Tile
|
||||
label="Library size"
|
||||
value={formatBytes(stats?.files?.totalSize)}
|
||||
note={sweptNote}
|
||||
pending={sweepPending}
|
||||
/>
|
||||
<Tile
|
||||
label="Metadata entries"
|
||||
value={formatCount(stats?.records)}
|
||||
note={
|
||||
fileless === null
|
||||
? 'records in the index'
|
||||
: `${formatCount(fileless)} with no local file`
|
||||
}
|
||||
/>
|
||||
<Tile
|
||||
label="Redis keys"
|
||||
value={formatCount(stats?.redisKeys)}
|
||||
note={stats?.redisMemory ? `${stats.redisMemory} in use` : undefined}
|
||||
/>
|
||||
<Tile
|
||||
label="Identities"
|
||||
value={formatCount(stats?.identities)}
|
||||
note={
|
||||
stats?.udlUsers !== null && stats?.udlUsers !== undefined
|
||||
? `${formatCount(stats.udlUsers)} with user data`
|
||||
: 'signing accounts'
|
||||
}
|
||||
/>
|
||||
</div>
|
||||
|
||||
<div style={{ ...gridStyle, gridTemplateColumns: 'repeat(auto-fit, minmax(300px, 1fr))' }}>
|
||||
<div style={cardStyle}>
|
||||
<h3 style={{ marginBottom: '1rem', color: '#888' }}>Service Health</h3>
|
||||
<div style={labelStyle}>Service health</div>
|
||||
{health ? (
|
||||
<>
|
||||
<div style={statStyle}>
|
||||
{health.status === 'ok' ? 'Healthy' : health.status}
|
||||
</div>
|
||||
<p>Redis: {health.redis ? 'Connected' : 'Disconnected'}</p>
|
||||
<p style={noteStyle}>
|
||||
Redis: {health.redis ? 'Connected' : 'Disconnected'}
|
||||
</p>
|
||||
</>
|
||||
) : (
|
||||
<p>Loading...</p>
|
||||
<p style={noteStyle}>Loading…</p>
|
||||
)}
|
||||
</div>
|
||||
|
||||
{/* Watchers Status Card */}
|
||||
<div style={cardStyle}>
|
||||
<h3 style={{ marginBottom: '1rem', color: '#888' }}>File Watchers</h3>
|
||||
<div style={labelStyle}>File watchers</div>
|
||||
{watchersData ? (
|
||||
<>
|
||||
<div style={statStyle}>{totalFiles}</div>
|
||||
<p>Files discovered</p>
|
||||
<p>Watchers: {activeWatchers} active / {watchersData.count} total</p>
|
||||
<p>Status: {isScanning ? 'Scanning...' : 'Idle'}</p>
|
||||
<div style={statStyle}>
|
||||
{activeWatchers} / {watchersData.count}
|
||||
</div>
|
||||
<p style={noteStyle}>active watchers</p>
|
||||
<p style={noteStyle}>Status: {isScanning ? 'Scanning…' : 'Idle'}</p>
|
||||
{lastScan > 0 && (
|
||||
<p>Last scan: {new Date(lastScan).toLocaleString()}</p>
|
||||
<p style={noteStyle}>Last scan: {new Date(lastScan).toLocaleString()}</p>
|
||||
)}
|
||||
<button
|
||||
onClick={handleTriggerScan}
|
||||
@@ -155,27 +275,7 @@ export default function Dashboard() {
|
||||
</button>
|
||||
</>
|
||||
) : (
|
||||
<p>Loading...</p>
|
||||
)}
|
||||
</div>
|
||||
|
||||
{/* Services Card */}
|
||||
<div style={cardStyle}>
|
||||
<h3 style={{ marginBottom: '1rem', color: '#888' }}>Connected Services</h3>
|
||||
<div style={statStyle}>{services.length}</div>
|
||||
<p>Services registered</p>
|
||||
{services.length > 0 && (
|
||||
<ul style={{ marginTop: '1rem', paddingLeft: '1.5rem' }}>
|
||||
{services.map((svc) => (
|
||||
<li key={svc.name} style={{ marginBottom: '0.5rem' }}>
|
||||
<strong>{svc.name}</strong>
|
||||
<br />
|
||||
<span style={{ color: '#888', fontSize: '0.9rem' }}>
|
||||
{svc.capabilities?.join(', ') || 'No capabilities'}
|
||||
</span>
|
||||
</li>
|
||||
))}
|
||||
</ul>
|
||||
<p style={noteStyle}>Loading…</p>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
+604
-120
@@ -1,21 +1,74 @@
|
||||
import { useEffect, useState } from 'react';
|
||||
import { useEffect, useRef, useState } from 'react';
|
||||
|
||||
// Status returned by GET /api/identity. Private key is never present here —
|
||||
// it is shown exactly once at generation time (see GeneratedKeyPanel).
|
||||
interface IdentityStatus {
|
||||
hasIdentity: boolean;
|
||||
uid?: string;
|
||||
curve?: string;
|
||||
createdAt?: number;
|
||||
}
|
||||
import {
|
||||
INSECURE_CONTEXT_MESSAGE,
|
||||
canSign,
|
||||
proveOwnership,
|
||||
uidFromSecretKey,
|
||||
type IdentityAction,
|
||||
} from '../lib/sign';
|
||||
|
||||
interface GeneratedKey {
|
||||
// Admin console for meta-core's account keystore.
|
||||
//
|
||||
// meta-core stores one secp256k1 keypair per file under
|
||||
// {META_CORE_PATH}/identity/accounts/<uid>.json and has NO notion of a current
|
||||
// user — which account is "you" is decided by whichever client signs in
|
||||
// (meta-watch). So this page lists accounts; it never speaks of an active one.
|
||||
//
|
||||
// The private key is returned by exactly two calls: POST /api/identity/generate
|
||||
// (once, at creation) and POST /api/identity/reveal (on demand, for an account
|
||||
// that already exists). Both land in the same exclusive SecretKeyPanel.
|
||||
//
|
||||
// Reveal and remove both require the account's own secret key, entered here and
|
||||
// used to sign a one-shot challenge locally — the key itself is never sent. So
|
||||
// this page can enumerate and create, but it holds no authority over an
|
||||
// existing account that its operator does not already have the key for. There
|
||||
// is deliberately no admin override: on this node the private key *is* the
|
||||
// account.
|
||||
|
||||
interface Account {
|
||||
uid: string;
|
||||
curve: string;
|
||||
createdAt: number;
|
||||
privateKeyHex: string;
|
||||
}
|
||||
|
||||
// What one account holds in the User Data Layer. Absent from
|
||||
// /api/udl/users/stats means zero, not unknown.
|
||||
interface Stats {
|
||||
records: number;
|
||||
cids: number;
|
||||
keys: number;
|
||||
}
|
||||
|
||||
// A secret on screen. Carries no createdAt: /api/identity/reveal answers with
|
||||
// {uid, curve, privateKeyHex} only.
|
||||
interface SecretKeyView {
|
||||
uid: string;
|
||||
privateKeyHex: string;
|
||||
kind: 'generated' | 'revealed';
|
||||
}
|
||||
|
||||
// Which row has an open confirmation, and for what. One discriminated value
|
||||
// rather than a pair of ids, so at most one confirm panel can ever be open.
|
||||
interface Pending {
|
||||
uid: string;
|
||||
kind: 'remove' | 'reveal';
|
||||
}
|
||||
|
||||
// Both confirmable operations map to a server-side action name, and the
|
||||
// challenge is bound to it: a challenge minted to reveal a key cannot delete
|
||||
// the account. Keeping the mapping in one place stops the two drifting.
|
||||
const ACTION_FOR: Record<Pending['kind'], IdentityAction> = {
|
||||
reveal: 'reveal',
|
||||
remove: 'delete',
|
||||
};
|
||||
|
||||
// meta-watch's sentinel cell for profile records: display name and avatar are
|
||||
// about the *user*, not a content file, so they live at (uid, "self", ...).
|
||||
// meta-core stores them opaquely and knows nothing of the convention.
|
||||
const PROFILE_CID = 'self';
|
||||
const PROFILE_NAME_KEY = 'profile:name';
|
||||
|
||||
const cardStyle: React.CSSProperties = {
|
||||
background: '#16213e',
|
||||
borderRadius: '8px',
|
||||
@@ -59,6 +112,21 @@ const monoStyle: React.CSSProperties = {
|
||||
borderRadius: '4px',
|
||||
};
|
||||
|
||||
const dangerPanelStyle: React.CSSProperties = {
|
||||
background: '#1f0a0a',
|
||||
border: '1px solid #7f1d1d',
|
||||
padding: '0.75rem 1rem',
|
||||
borderRadius: '4px',
|
||||
marginTop: '0.75rem',
|
||||
};
|
||||
|
||||
const errorTextStyle: React.CSSProperties = {
|
||||
color: '#f87171',
|
||||
background: '#1f0a0a',
|
||||
padding: '0.5rem 0.75rem',
|
||||
borderRadius: '4px',
|
||||
};
|
||||
|
||||
function formatCreatedAt(unixSec?: number): string {
|
||||
if (!unixSec) return '';
|
||||
try {
|
||||
@@ -79,25 +147,165 @@ async function errorMessage(res: Response, fallback: string): Promise<string> {
|
||||
return `${fallback} (HTTP ${res.status})`;
|
||||
}
|
||||
|
||||
function GeneratedKeyPanel({ data, onDismiss }: { data: GeneratedKey; onDismiss: () => void }) {
|
||||
const [copied, setCopied] = useState(false);
|
||||
const copy = async () => {
|
||||
// Copy text, falling back to selecting the element that displays it.
|
||||
//
|
||||
// The fallback is load-bearing, not politeness: the dashboard is served over
|
||||
// plain HTTP, where navigator.clipboard is undefined — without it the copy
|
||||
// buttons do nothing at all on a real deployment.
|
||||
async function copyToClipboard(text: string, fallbackElementId: string): Promise<void> {
|
||||
try {
|
||||
await navigator.clipboard.writeText(data.privateKeyHex);
|
||||
setCopied(true);
|
||||
setTimeout(() => setCopied(false), 2000);
|
||||
await navigator.clipboard.writeText(text);
|
||||
} catch {
|
||||
// Clipboard API blocked (non-HTTPS context, etc.) — fall back to selection.
|
||||
const sel = window.getSelection();
|
||||
const range = document.createRange();
|
||||
const el = document.getElementById('mm-privkey-box');
|
||||
const el = document.getElementById(fallbackElementId);
|
||||
if (sel && el) {
|
||||
const range = document.createRange();
|
||||
range.selectNodeContents(el);
|
||||
sel.removeAllRanges();
|
||||
sel.addRange(range);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async function fetchAccounts(): Promise<Account[]> {
|
||||
const res = await fetch('/api/identity/accounts');
|
||||
if (!res.ok) throw new Error(await errorMessage(res, 'Failed to load accounts'));
|
||||
const data = (await res.json()) as { accounts?: Account[] };
|
||||
// Backend order is CreatedAt ascending then uid (identity.List) — stable, so
|
||||
// render it as given rather than re-sorting here.
|
||||
return data.accounts ?? [];
|
||||
}
|
||||
|
||||
// Resolve an account's display name from the User Data Layer, or null.
|
||||
//
|
||||
// Never throws and never reports an error upward: a name is decoration on top
|
||||
// of the uid, and the account list must render identically whether Redis is up
|
||||
// (identity itself is file-backed, the UDL is not). The name is also *unsigned*
|
||||
// — meta-core stores the plaintext beside the signed record and verifies
|
||||
// nothing — so the uid stays on screen as the real identity, and the length cap
|
||||
// bounds what a hostile writer can do to the layout.
|
||||
async function fetchDisplayName(uid: string): Promise<string | null> {
|
||||
const url =
|
||||
`/api/udl/record?uid=${encodeURIComponent(uid)}` +
|
||||
`&cid=${encodeURIComponent(PROFILE_CID)}` +
|
||||
`&key=${encodeURIComponent(PROFILE_NAME_KEY)}`;
|
||||
try {
|
||||
const res = await fetch(url);
|
||||
if (!res.ok) return null; // 503 when storage is down, 500, …
|
||||
const data = (await res.json()) as { exists?: boolean; value?: unknown };
|
||||
// Key off `value`, not `exists`: clearing a name writes a tombstone, which
|
||||
// leaves the cell existing with no plaintext. Private-tier cells look the
|
||||
// same, which is exactly right — there is nothing to show for either.
|
||||
if (typeof data.value !== 'string') return null;
|
||||
const name = data.value.trim();
|
||||
return name ? name.slice(0, 64) : null;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
// How much each account holds. One call for the whole page, not one per row:
|
||||
// meta-core answers it from a keyspace SCAN (there is no per-user cid index),
|
||||
// so per-row calls would walk the keyspace once per account.
|
||||
//
|
||||
// Never throws, for the same reason fetchDisplayName does not: the counts are
|
||||
// context on top of the uid, and the list must render identically when Redis is
|
||||
// down — accounts are file-backed, their data is not.
|
||||
//
|
||||
// Returns null on failure rather than an empty map. The difference is
|
||||
// load-bearing: an empty map renders as "no stored data" against every account,
|
||||
// which inside a delete confirmation is a false statement about what is at
|
||||
// stake. Unknown has to look like unknown.
|
||||
async function fetchStats(): Promise<Record<string, Stats> | null> {
|
||||
try {
|
||||
const res = await fetch('/api/udl/users/stats');
|
||||
if (!res.ok) return null;
|
||||
const data = (await res.json()) as { users?: Record<string, Stats> };
|
||||
return data.users ?? {};
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
// The secret-key box shared by the reveal and remove confirmations.
|
||||
//
|
||||
// Separate from the surrounding panel so both operations ask the same way, and
|
||||
// so the key lives in this component's state rather than the page's — it is
|
||||
// discarded with the panel instead of lingering in a parent that outlives it.
|
||||
function SecretKeyConfirm({
|
||||
label,
|
||||
actionLabel,
|
||||
busy,
|
||||
onConfirm,
|
||||
onCancel,
|
||||
}: {
|
||||
label: string;
|
||||
actionLabel: string;
|
||||
busy: boolean;
|
||||
onConfirm: (secretKeyHex: string) => void;
|
||||
onCancel: () => void;
|
||||
}) {
|
||||
const [secret, setSecret] = useState('');
|
||||
const signable = canSign();
|
||||
// Derived locally, purely to tell the operator they have the wrong key before
|
||||
// they hit the button. The server would refuse either way.
|
||||
const derived = secret.trim() ? uidFromSecretKey(secret) : null;
|
||||
const mismatch = Boolean(secret.trim() && derived && derived !== label);
|
||||
|
||||
return (
|
||||
<form
|
||||
onSubmit={(e) => {
|
||||
e.preventDefault();
|
||||
onConfirm(secret.trim());
|
||||
}}
|
||||
>
|
||||
<p style={{ color: '#fecaca', margin: '0 0 0.5rem' }}>
|
||||
Paste this account's secret key to prove it is yours. It is used to sign a one-time
|
||||
challenge <em>in this browser</em> and is never sent to the server.
|
||||
</p>
|
||||
{!signable && <p style={errorTextStyle}>{INSECURE_CONTEXT_MESSAGE}</p>}
|
||||
<input
|
||||
style={inputStyle}
|
||||
type="password"
|
||||
autoComplete="off"
|
||||
placeholder="64 hex characters"
|
||||
value={secret}
|
||||
onChange={(e) => setSecret(e.target.value)}
|
||||
disabled={busy || !signable}
|
||||
autoFocus
|
||||
/>
|
||||
{mismatch && (
|
||||
<p style={{ ...errorTextStyle, marginTop: 0 }}>
|
||||
That key belongs to a different account ({derived}).
|
||||
</p>
|
||||
)}
|
||||
<button
|
||||
type="submit"
|
||||
style={dangerButtonStyle}
|
||||
disabled={busy || !signable || !secret.trim() || mismatch}
|
||||
>
|
||||
{busy ? 'Verifying…' : actionLabel}
|
||||
</button>
|
||||
<button type="button" style={buttonStyle} onClick={onCancel} disabled={busy}>
|
||||
Cancel
|
||||
</button>
|
||||
</form>
|
||||
);
|
||||
}
|
||||
|
||||
// The one place a private key is ever displayed. Rendered exclusively (the
|
||||
// account list is hidden while it is open) so the "save it now" contract of a
|
||||
// freshly generated key is not competing with anything, and so the hardcoded
|
||||
// element id below stays unique.
|
||||
function SecretKeyPanel({ data, onDismiss }: { data: SecretKeyView; onDismiss: () => void }) {
|
||||
const [copied, setCopied] = useState(false);
|
||||
|
||||
const copy = async () => {
|
||||
await copyToClipboard(data.privateKeyHex, 'mm-privkey-box');
|
||||
setCopied(true);
|
||||
setTimeout(() => setCopied(false), 2000);
|
||||
};
|
||||
|
||||
return (
|
||||
<div style={{ ...cardStyle, border: '2px solid #dc2626' }}>
|
||||
<div
|
||||
@@ -109,9 +317,19 @@ function GeneratedKeyPanel({ data, onDismiss }: { data: GeneratedKey; onDismiss:
|
||||
marginBottom: '1rem',
|
||||
}}
|
||||
>
|
||||
{data.kind === 'generated' ? (
|
||||
<>
|
||||
<strong>This private key is shown ONCE.</strong> Save it now in a password manager.
|
||||
It will never be displayed again — there is no recovery path. Losing it means losing access
|
||||
to every record signed under this identity.
|
||||
You can ask for it again later with <em>Reveal private key</em>, but only for as long
|
||||
as this account exists on this node — delete it and the key is gone for good.
|
||||
</>
|
||||
) : (
|
||||
<>
|
||||
<strong>This is the live private key for an existing account.</strong> Anyone holding
|
||||
it can sign as this uid forever, on any machine. There is no revocation and no
|
||||
rotation. Do not paste it into chat, tickets or screenshots.
|
||||
</>
|
||||
)}
|
||||
</div>
|
||||
<p style={{ color: '#9ca3af', margin: '0 0 0.5rem' }}>uid (public):</p>
|
||||
<div style={monoStyle}>{data.uid}</div>
|
||||
@@ -124,36 +342,40 @@ function GeneratedKeyPanel({ data, onDismiss }: { data: GeneratedKey; onDismiss:
|
||||
{copied ? 'Copied' : 'Copy private key'}
|
||||
</button>
|
||||
<button style={dangerButtonStyle} onClick={onDismiss}>
|
||||
I have saved it
|
||||
{data.kind === 'generated' ? 'I have saved it' : 'Done — hide it'}
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
function NoIdentityPanel({
|
||||
// Create / import. Always available, not just on an empty keystore: this is an
|
||||
// admin console, so adding an account to a populated node has to be possible.
|
||||
function NewAccountPanel({
|
||||
onGenerated,
|
||||
onImported,
|
||||
}: {
|
||||
onGenerated: (g: GeneratedKey) => void;
|
||||
onGenerated: (s: SecretKeyView) => void;
|
||||
onImported: () => void;
|
||||
}) {
|
||||
const [busy, setBusy] = useState(false);
|
||||
const [importing, setImporting] = useState(false);
|
||||
const [importHex, setImportHex] = useState('');
|
||||
const [error, setError] = useState<string | null>(null);
|
||||
const [notice, setNotice] = useState<string | null>(null);
|
||||
|
||||
const generate = async () => {
|
||||
setBusy(true);
|
||||
setError(null);
|
||||
setNotice(null);
|
||||
try {
|
||||
const res = await fetch('/api/identity/generate', { method: 'POST' });
|
||||
if (!res.ok) {
|
||||
setError(await errorMessage(res, 'Generate failed'));
|
||||
return;
|
||||
}
|
||||
const data = (await res.json()) as GeneratedKey;
|
||||
onGenerated(data);
|
||||
const data = (await res.json()) as { uid: string; privateKeyHex: string };
|
||||
onGenerated({ uid: data.uid, privateKeyHex: data.privateKeyHex, kind: 'generated' });
|
||||
} catch (e) {
|
||||
setError(`Generate failed: ${e instanceof Error ? e.message : String(e)}`);
|
||||
} finally {
|
||||
@@ -165,6 +387,7 @@ function NoIdentityPanel({
|
||||
e.preventDefault();
|
||||
setBusy(true);
|
||||
setError(null);
|
||||
setNotice(null);
|
||||
try {
|
||||
const res = await fetch('/api/identity/import', {
|
||||
method: 'POST',
|
||||
@@ -175,6 +398,15 @@ function NoIdentityPanel({
|
||||
setError(await errorMessage(res, 'Import failed'));
|
||||
return;
|
||||
}
|
||||
// Import is idempotent — presenting the key IS the proof of ownership, so
|
||||
// a key this node already holds is a success, not a conflict. `created`
|
||||
// only decides the wording.
|
||||
const data = (await res.json()) as { created?: boolean };
|
||||
setNotice(
|
||||
data.created
|
||||
? 'Imported — a new account was added to this keystore.'
|
||||
: 'That key was already in this keystore; nothing changed.'
|
||||
);
|
||||
setImportHex('');
|
||||
setImporting(false);
|
||||
onImported();
|
||||
@@ -187,15 +419,16 @@ function NoIdentityPanel({
|
||||
|
||||
return (
|
||||
<div style={cardStyle}>
|
||||
<h3 style={{ marginTop: 0 }}>No identity yet</h3>
|
||||
<p style={{ color: '#9ca3af' }}>
|
||||
The User Data Layer needs a signing key for your ratings, watch progress, and "liked" flags
|
||||
to be attributed to you across devices. Nothing happens automatically — generate one below,
|
||||
or import a key you already have.
|
||||
<h3 style={{ marginTop: 0 }}>Add an account</h3>
|
||||
<p style={{ color: '#9ca3af', marginTop: 0 }}>
|
||||
Generating mints a fresh secp256k1 keypair here. Importing adds a key created elsewhere —
|
||||
it is also how an existing account is moved onto this node, so importing a key the
|
||||
keystore already holds succeeds and changes nothing.
|
||||
</p>
|
||||
{error && (
|
||||
<p style={{ color: '#f87171', background: '#1f0a0a', padding: '0.5rem 0.75rem', borderRadius: '4px' }}>
|
||||
{error}
|
||||
{error && <p style={errorTextStyle}>{error}</p>}
|
||||
{notice && (
|
||||
<p style={{ color: '#9ca3af', background: '#0a0a18', padding: '0.5rem 0.75rem', borderRadius: '4px' }}>
|
||||
{notice}
|
||||
</p>
|
||||
)}
|
||||
<div>
|
||||
@@ -235,73 +468,156 @@ function NoIdentityPanel({
|
||||
);
|
||||
}
|
||||
|
||||
function HasIdentityPanel({ status, onReplace }: { status: IdentityStatus; onReplace: () => void }) {
|
||||
const [confirming, setConfirming] = useState(false);
|
||||
const [busy, setBusy] = useState(false);
|
||||
const [error, setError] = useState<string | null>(null);
|
||||
function AccountCard({
|
||||
account,
|
||||
name,
|
||||
stats,
|
||||
busy,
|
||||
disabled,
|
||||
pending,
|
||||
error,
|
||||
onAsk,
|
||||
onCancel,
|
||||
onReveal,
|
||||
onRemove,
|
||||
}: {
|
||||
account: Account;
|
||||
name?: string;
|
||||
// undefined means "counted, and it holds nothing"; null means "could not be
|
||||
// counted". The two must not read the same on a confirmation screen.
|
||||
stats?: Stats | null;
|
||||
busy: boolean;
|
||||
disabled: boolean;
|
||||
pending: Pending | null;
|
||||
error: string | null;
|
||||
onAsk: (p: Pending) => void;
|
||||
onCancel: () => void;
|
||||
onReveal: (uid: string, secretKeyHex: string) => void;
|
||||
onRemove: (uid: string, secretKeyHex: string) => void;
|
||||
}) {
|
||||
const [copied, setCopied] = useState(false);
|
||||
const uidBoxId = `mm-uid-${account.uid}`;
|
||||
const label = name ?? account.uid;
|
||||
const counted = stats !== null;
|
||||
const records = stats?.records ?? 0;
|
||||
const titles = stats?.cids ?? 0;
|
||||
|
||||
const remove = async () => {
|
||||
setBusy(true);
|
||||
setError(null);
|
||||
try {
|
||||
const res = await fetch('/api/identity?confirm=true', { method: 'DELETE' });
|
||||
if (!res.ok && res.status !== 204) {
|
||||
setError(await errorMessage(res, 'Delete failed'));
|
||||
return;
|
||||
}
|
||||
onReplace();
|
||||
} catch (e) {
|
||||
setError(`Delete failed: ${e instanceof Error ? e.message : String(e)}`);
|
||||
} finally {
|
||||
setBusy(false);
|
||||
setConfirming(false);
|
||||
}
|
||||
const copyUid = async () => {
|
||||
await copyToClipboard(account.uid, uidBoxId);
|
||||
setCopied(true);
|
||||
setTimeout(() => setCopied(false), 2000);
|
||||
};
|
||||
|
||||
return (
|
||||
<div style={cardStyle}>
|
||||
<h3 style={{ marginTop: 0 }}>Active identity</h3>
|
||||
<p style={{ color: '#9ca3af', margin: '0 0 0.5rem' }}>uid:</p>
|
||||
<div style={monoStyle}>{status.uid}</div>
|
||||
<p style={{ color: '#888', marginTop: '1rem' }}>
|
||||
Curve: {status.curve} · Created: {formatCreatedAt(status.createdAt)}
|
||||
<div style={{ display: 'flex', justifyContent: 'space-between', alignItems: 'flex-start', gap: '1rem' }}>
|
||||
<div style={{ minWidth: 0, flex: 1 }}>
|
||||
<h3 style={{ margin: '0 0 0.5rem', color: name ? '#fff' : '#888' }}>
|
||||
{name ?? 'Unnamed account'}
|
||||
</h3>
|
||||
<div id={uidBoxId} style={monoStyle}>
|
||||
{account.uid}
|
||||
</div>
|
||||
<p style={{ color: '#888', margin: '0.75rem 0 0', fontSize: '0.85rem' }}>
|
||||
Curve: {account.curve} · Created: {formatCreatedAt(account.createdAt)}
|
||||
</p>
|
||||
<p style={{ color: '#9ca3af', fontSize: '0.85rem' }}>
|
||||
The private key lives at <code>/meta-core/identity/identity.json</code> on this volume and
|
||||
is never returned through the API after generation. Other services on this stack (e.g.
|
||||
meta-watch) sign records by calling <code>POST /api/identity/sign</code>.
|
||||
<p style={{ color: '#888', margin: '0.25rem 0 0', fontSize: '0.85rem' }}>
|
||||
{/* An account that has never been used legitimately holds nothing, so
|
||||
zero is a real answer — but only once the count actually ran. */}
|
||||
{!counted
|
||||
? 'Stored data: could not be counted (storage unavailable)'
|
||||
: records === 0
|
||||
? 'No stored data'
|
||||
: `${records.toLocaleString()} stored ${records === 1 ? 'record' : 'records'}` +
|
||||
` across ${titles.toLocaleString()} ${titles === 1 ? 'title' : 'titles'}`}
|
||||
</p>
|
||||
{error && (
|
||||
<p style={{ color: '#f87171', background: '#1f0a0a', padding: '0.5rem 0.75rem', borderRadius: '4px' }}>
|
||||
{error}
|
||||
</p>
|
||||
)}
|
||||
{!confirming ? (
|
||||
<button style={dangerButtonStyle} onClick={() => setConfirming(true)} disabled={busy}>
|
||||
Replace identity
|
||||
</div>
|
||||
<div style={{ display: 'flex', flexDirection: 'column', gap: '0.5rem', flexShrink: 0 }}>
|
||||
<button style={{ ...buttonStyle, marginRight: 0 }} onClick={copyUid} disabled={disabled}>
|
||||
{copied ? 'Copied' : 'Copy uid'}
|
||||
</button>
|
||||
) : (
|
||||
<div
|
||||
style={{
|
||||
background: '#1f0a0a',
|
||||
border: '1px solid #7f1d1d',
|
||||
padding: '0.75rem 1rem',
|
||||
borderRadius: '4px',
|
||||
marginTop: '0.5rem',
|
||||
}}
|
||||
<button
|
||||
style={{ ...dangerButtonStyle, marginRight: 0 }}
|
||||
onClick={() => onAsk({ uid: account.uid, kind: 'reveal' })}
|
||||
disabled={disabled}
|
||||
>
|
||||
Reveal private key
|
||||
</button>
|
||||
<button
|
||||
style={{ ...dangerButtonStyle, marginRight: 0 }}
|
||||
onClick={() => onAsk({ uid: account.uid, kind: 'remove' })}
|
||||
disabled={disabled}
|
||||
>
|
||||
Remove
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{error && <p style={{ ...errorTextStyle, marginBottom: 0 }}>{error}</p>}
|
||||
|
||||
{pending?.uid === account.uid && pending.kind === 'reveal' && (
|
||||
<div style={dangerPanelStyle}>
|
||||
<p style={{ color: '#fecaca', marginTop: 0 }}>
|
||||
Replacing the identity deletes the current keypair from disk. Records signed by the
|
||||
current uid will keep verifying for anyone who has them cached, but this node will no
|
||||
longer be able to write under that uid. <strong>Make sure you have backed up the key
|
||||
first.</strong>
|
||||
Show the private key for <strong>{label}</strong>?
|
||||
</p>
|
||||
<button style={dangerButtonStyle} onClick={remove} disabled={busy}>
|
||||
{busy ? 'Deleting…' : 'Yes, delete identity'}
|
||||
</button>
|
||||
<button style={buttonStyle} onClick={() => setConfirming(false)} disabled={busy}>
|
||||
Cancel
|
||||
</button>
|
||||
<p style={{ color: '#fecaca', marginTop: 0 }}>
|
||||
The key <em>is</em> the account. Anyone who copies it can sign records as this uid on
|
||||
any machine, forever — there is no way to revoke or rotate it afterwards. Because
|
||||
proving ownership already requires the key, this only confirms that the copy this node
|
||||
holds matches yours.
|
||||
</p>
|
||||
<SecretKeyConfirm
|
||||
label={account.uid}
|
||||
actionLabel="Show the private key"
|
||||
busy={busy}
|
||||
onConfirm={(secret) => onReveal(account.uid, secret)}
|
||||
onCancel={onCancel}
|
||||
/>
|
||||
</div>
|
||||
)}
|
||||
|
||||
{pending?.uid === account.uid && pending.kind === 'remove' && (
|
||||
<div style={dangerPanelStyle}>
|
||||
<p style={{ color: '#fecaca', marginTop: 0 }}>
|
||||
Remove <strong>{label}</strong> from this node?
|
||||
</p>
|
||||
<ul style={{ color: '#fecaca', paddingLeft: '1.2rem', margin: '0 0 0.75rem' }}>
|
||||
<li>
|
||||
<strong>The key is gone for good.</strong> The uid <em>is</em> the public key, and
|
||||
the private half exists only in this node's keystore. If nobody saved it, nothing —
|
||||
not you, not meta-core — can ever sign as this uid again. Reveal and copy it first
|
||||
if there is any doubt.
|
||||
</li>
|
||||
<li>
|
||||
<strong>Nothing is retracted.</strong> Records already signed under this uid keep
|
||||
verifying on every peer that replicated them. Deleting a key is not a recall.
|
||||
</li>
|
||||
<li>
|
||||
<strong>Its data is destroyed, not orphaned.</strong>{' '}
|
||||
{!counted
|
||||
? 'This account\u2019s records could not be counted because storage is unavailable — ' +
|
||||
'so meta-core will refuse this delete rather than remove the key and strand them.'
|
||||
: records === 0
|
||||
? 'This account holds no User Data Layer records, so there is nothing to purge.'
|
||||
: `All ${records.toLocaleString()} User Data Layer ${
|
||||
records === 1 ? 'record' : 'records'
|
||||
} — likes, ratings, watch progress across ${titles.toLocaleString()} ${
|
||||
titles === 1 ? 'title' : 'titles'
|
||||
} — are permanently deleted from Redis, along with every index that names this uid.`}{' '}
|
||||
This cannot be undone and there is no backup.
|
||||
</li>
|
||||
<li>
|
||||
<strong>This signs nobody out elsewhere.</strong> Any device that holds this key
|
||||
still holds it, and can import it back here at any time.
|
||||
</li>
|
||||
</ul>
|
||||
<SecretKeyConfirm
|
||||
label={account.uid}
|
||||
actionLabel="Remove this account and all its data"
|
||||
busy={busy}
|
||||
onConfirm={(secret) => onRemove(account.uid, secret)}
|
||||
onCancel={onCancel}
|
||||
/>
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
@@ -309,67 +625,235 @@ function HasIdentityPanel({ status, onReplace }: { status: IdentityStatus; onRep
|
||||
}
|
||||
|
||||
export default function Identity() {
|
||||
const [status, setStatus] = useState<IdentityStatus | null>(null);
|
||||
const [generated, setGenerated] = useState<GeneratedKey | null>(null);
|
||||
const [accounts, setAccounts] = useState<Account[] | null>(null);
|
||||
const [names, setNames] = useState<Record<string, string>>({});
|
||||
const [stats, setStats] = useState<Record<string, Stats> | null>({});
|
||||
const [notice, setNotice] = useState<string | null>(null);
|
||||
const [loading, setLoading] = useState(true);
|
||||
const [fetchError, setFetchError] = useState<string | null>(null);
|
||||
const [secret, setSecret] = useState<SecretKeyView | null>(null);
|
||||
const [pending, setPending] = useState<Pending | null>(null);
|
||||
const [busyUid, setBusyUid] = useState<string | null>(null);
|
||||
const [rowError, setRowError] = useState<{ uid: string; message: string } | null>(null);
|
||||
|
||||
// Bumped per refresh so a slow response from a superseded load cannot
|
||||
// overwrite a newer one. The page does not poll (unlike the rest of the
|
||||
// dashboard) — a key list has no business changing under the cursor.
|
||||
const reqGen = useRef(0);
|
||||
|
||||
const refresh = async () => {
|
||||
const gen = ++reqGen.current;
|
||||
setLoading(true);
|
||||
setFetchError(null);
|
||||
let list: Account[];
|
||||
try {
|
||||
const res = await fetch('/api/identity');
|
||||
if (!res.ok) {
|
||||
setFetchError(await errorMessage(res, 'Failed to load identity status'));
|
||||
list = await fetchAccounts();
|
||||
} catch (e) {
|
||||
if (gen !== reqGen.current) return;
|
||||
setFetchError(e instanceof Error ? e.message : String(e));
|
||||
setLoading(false);
|
||||
return;
|
||||
}
|
||||
setStatus((await res.json()) as IdentityStatus);
|
||||
} catch (e) {
|
||||
setFetchError(`Failed to load identity status: ${e instanceof Error ? e.message : String(e)}`);
|
||||
} finally {
|
||||
if (gen !== reqGen.current) return;
|
||||
// Render the list before resolving names, so a down Redis delays nothing.
|
||||
setAccounts(list);
|
||||
setLoading(false);
|
||||
}
|
||||
|
||||
// Names and counts both come after the list is on screen, and neither can
|
||||
// fail the page: a down Redis costs decoration, not the account list.
|
||||
const [settled, counts] = await Promise.all([
|
||||
Promise.allSettled(list.map((a) => fetchDisplayName(a.uid))),
|
||||
fetchStats(),
|
||||
]);
|
||||
if (gen !== reqGen.current) return;
|
||||
const next: Record<string, string> = {};
|
||||
settled.forEach((r, i) => {
|
||||
if (r.status === 'fulfilled' && r.value) next[list[i].uid] = r.value;
|
||||
});
|
||||
// Replace rather than merge, so a removed account's name and counts go with
|
||||
// it instead of lingering against a uid that no longer exists.
|
||||
setNames(next);
|
||||
setStats(counts);
|
||||
};
|
||||
|
||||
useEffect(() => {
|
||||
refresh();
|
||||
}, []);
|
||||
|
||||
const reveal = async (uid: string, secretKeyHex: string) => {
|
||||
setBusyUid(uid);
|
||||
setRowError(null);
|
||||
try {
|
||||
const proof = await proveOwnership(uid, ACTION_FOR.reveal, secretKeyHex);
|
||||
// POST with a body rather than ?uid=, so neither the target nor the
|
||||
// signature reaches the proxy's access log. The handler accepts the uid
|
||||
// either way; the proof must be in the body.
|
||||
const res = await fetch('/api/identity/reveal', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ confirm: true, uid, ...proof }),
|
||||
});
|
||||
if (!res.ok) {
|
||||
setRowError({ uid, message: await errorMessage(res, 'Reveal failed') });
|
||||
return;
|
||||
}
|
||||
const data = (await res.json()) as { uid: string; privateKeyHex: string };
|
||||
setPending(null);
|
||||
setSecret({ uid: data.uid, privateKeyHex: data.privateKeyHex, kind: 'revealed' });
|
||||
} catch (e) {
|
||||
setRowError({ uid, message: `Reveal failed: ${e instanceof Error ? e.message : String(e)}` });
|
||||
} finally {
|
||||
setBusyUid(null);
|
||||
}
|
||||
};
|
||||
|
||||
const remove = async (uid: string, secretKeyHex: string) => {
|
||||
setBusyUid(uid);
|
||||
setRowError(null);
|
||||
setNotice(null);
|
||||
try {
|
||||
const proof = await proveOwnership(uid, ACTION_FOR.remove, secretKeyHex);
|
||||
// The uid is mandatory: the signature authorises one specific account, so
|
||||
// the backend refuses an unqualified delete rather than guess a target.
|
||||
const res = await fetch('/api/identity', {
|
||||
method: 'DELETE',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ confirm: true, uid, ...proof }),
|
||||
});
|
||||
if (!res.ok) {
|
||||
setRowError({ uid, message: await errorMessage(res, 'Remove failed') });
|
||||
return;
|
||||
}
|
||||
// Report what the purge actually removed. "Account removed" on its own
|
||||
// would not distinguish a clean delete from one that stranded the data.
|
||||
const data = (await res.json().catch(() => ({}))) as { purged?: Stats };
|
||||
const purged = data.purged;
|
||||
setNotice(
|
||||
purged && purged.records > 0
|
||||
? `Account removed. Purged ${purged.records.toLocaleString()} User Data Layer ` +
|
||||
`${purged.records === 1 ? 'record' : 'records'} across ` +
|
||||
`${purged.cids.toLocaleString()} ${purged.cids === 1 ? 'title' : 'titles'}.`
|
||||
: 'Account removed. It held no User Data Layer records.'
|
||||
);
|
||||
setPending(null);
|
||||
await refresh();
|
||||
} catch (e) {
|
||||
setRowError({ uid, message: `Remove failed: ${e instanceof Error ? e.message : String(e)}` });
|
||||
} finally {
|
||||
setBusyUid(null);
|
||||
}
|
||||
};
|
||||
|
||||
const dismissSecret = () => {
|
||||
setSecret(null);
|
||||
refresh();
|
||||
};
|
||||
|
||||
const ask = (p: Pending) => {
|
||||
setRowError(null);
|
||||
setNotice(null);
|
||||
setPending(p);
|
||||
};
|
||||
|
||||
const showList = !loading && !fetchError && !secret;
|
||||
|
||||
return (
|
||||
// App.tsx's content area is `overflow: hidden`, so every page owns its own
|
||||
// scrolling (Files.tsx does the same). Without this the account list is
|
||||
// simply clipped at the viewport with no scrollbar — invisible on a node
|
||||
// with two accounts, and a dead end on one with fifteen.
|
||||
//
|
||||
// The scroller is the outer element and the padding lives inside it, so the
|
||||
// scrollbar sits at the edge of the content area rather than floating in
|
||||
// 2rem of whitespace, and `margin: 0 auto` still centres the column.
|
||||
<div style={{ height: '100%', overflowY: 'auto', minHeight: 0 }}>
|
||||
<div style={{ padding: '2rem', maxWidth: '900px', margin: '0 auto' }}>
|
||||
<h1>Identity</h1>
|
||||
<div style={{ display: 'flex', justifyContent: 'space-between', alignItems: 'center' }}>
|
||||
<h1 style={{ marginBottom: 0 }}>Identity</h1>
|
||||
<button style={{ ...buttonStyle, marginRight: 0 }} onClick={refresh} disabled={loading}>
|
||||
{loading ? 'Loading…' : 'Refresh'}
|
||||
</button>
|
||||
</div>
|
||||
<p style={{ color: '#9ca3af' }}>
|
||||
Signing key for the User Data Layer (per-user ratings, progress, "liked" flags). One
|
||||
identity per node. The same key on another device makes that device count as "you".
|
||||
Every signing key in this node's keystore. meta-core has no "current user" — it stores
|
||||
accounts, and which one is <em>you</em> is decided by whichever client signs in (e.g.
|
||||
meta-watch). Keys live one JSON file per account under{' '}
|
||||
<code>{'{META_CORE_PATH}'}/identity/accounts/<uid>.json</code>, and are used to sign
|
||||
User Data Layer records (ratings, watch progress, "liked" flags).
|
||||
</p>
|
||||
|
||||
<div style={{ ...dangerPanelStyle, marginTop: 0, marginBottom: '1rem' }}>
|
||||
<p style={{ color: '#fecaca', margin: 0 }}>
|
||||
<strong>The key is the only authority.</strong> Revealing or removing an account requires
|
||||
that account's secret key, entered here and used to sign a one-time challenge in your
|
||||
browser — there is no admin override, and holding this page is not one. Listing and
|
||||
creating accounts are still open to anything that can reach this API, so the perimeter in
|
||||
front of this port is what keeps the roster private.
|
||||
</p>
|
||||
<p style={{ color: '#fecaca', margin: '0.5rem 0 0' }}>
|
||||
What this does <em>not</em> defend against: anyone with filesystem access to{' '}
|
||||
<code>{'{META_CORE_PATH}'}/identity/accounts/</code> reads every private key directly.
|
||||
The keys are stored in plaintext, matching the trust model of the rest of{' '}
|
||||
<code>/meta-core</code>.
|
||||
</p>
|
||||
</div>
|
||||
|
||||
{notice && (
|
||||
<div style={{ ...cardStyle, border: '1px solid #0f3460' }}>
|
||||
<p style={{ color: '#9ca3af', margin: 0 }}>{notice}</p>
|
||||
</div>
|
||||
)}
|
||||
|
||||
{loading && <div style={cardStyle}>Loading…</div>}
|
||||
|
||||
{fetchError && (
|
||||
<div style={{ ...cardStyle, border: '1px solid #7f1d1d' }}>
|
||||
<p style={{ color: '#f87171', margin: 0 }}>{fetchError}</p>
|
||||
<p style={{ color: '#f87171', margin: '0 0 1rem' }}>{fetchError}</p>
|
||||
<button style={buttonStyle} onClick={refresh}>
|
||||
Retry
|
||||
</button>
|
||||
</div>
|
||||
)}
|
||||
|
||||
{!loading && !fetchError && generated && (
|
||||
<GeneratedKeyPanel
|
||||
data={generated}
|
||||
onDismiss={() => {
|
||||
setGenerated(null);
|
||||
refresh();
|
||||
}}
|
||||
{!loading && !fetchError && secret && <SecretKeyPanel data={secret} onDismiss={dismissSecret} />}
|
||||
|
||||
{showList && (
|
||||
<>
|
||||
<NewAccountPanel onGenerated={(s) => setSecret(s)} onImported={refresh} />
|
||||
|
||||
<h2 style={{ fontSize: '1.1rem', color: '#9ca3af', margin: '1.5rem 0 1rem' }}>
|
||||
Accounts ({accounts?.length ?? 0})
|
||||
</h2>
|
||||
|
||||
{accounts && accounts.length === 0 && (
|
||||
<div style={cardStyle}>
|
||||
<p style={{ color: '#9ca3af', margin: 0 }}>
|
||||
No accounts in this keystore. Accounts are normally created by a client when
|
||||
someone signs up; use <em>Add an account</em> above if you are bootstrapping this
|
||||
node.
|
||||
</p>
|
||||
</div>
|
||||
)}
|
||||
|
||||
{accounts?.map((a) => (
|
||||
<AccountCard
|
||||
key={a.uid}
|
||||
account={a}
|
||||
name={names[a.uid]}
|
||||
stats={stats === null ? null : stats[a.uid]}
|
||||
busy={busyUid === a.uid}
|
||||
disabled={busyUid !== null}
|
||||
pending={pending}
|
||||
error={rowError?.uid === a.uid ? rowError.message : null}
|
||||
onAsk={ask}
|
||||
onCancel={() => setPending(null)}
|
||||
onReveal={reveal}
|
||||
onRemove={remove}
|
||||
/>
|
||||
))}
|
||||
</>
|
||||
)}
|
||||
|
||||
{!loading && !fetchError && !generated && status && !status.hasIdentity && (
|
||||
<NoIdentityPanel onGenerated={(g) => setGenerated(g)} onImported={refresh} />
|
||||
)}
|
||||
|
||||
{!loading && !fetchError && !generated && status && status.hasIdentity && (
|
||||
<HasIdentityPanel status={status} onReplace={refresh} />
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
@@ -10,7 +10,6 @@ set -e
|
||||
# Create required directories
|
||||
mkdir -p /meta-core/locks
|
||||
mkdir -p /meta-core/db/redis
|
||||
mkdir -p /meta-core/services
|
||||
mkdir -p /meta-core/mounts/errors
|
||||
mkdir -p /files
|
||||
mkdir -p /var/log/supervisor
|
||||
|
||||
@@ -2,9 +2,13 @@
|
||||
# =============================================================================
|
||||
# Leader Election Script for meta-core
|
||||
#
|
||||
# Uses flock for leader election. Only the leader runs the full application
|
||||
# stack (supervisord with Redis, meta-core, nginx, rclone, mount-watcher).
|
||||
# Followers remain dormant, periodically retrying to acquire the lock.
|
||||
# The flock here is a MUTEX, not service discovery. It is the only thing
|
||||
# stopping two containers that mount the same /meta-core volume from putting
|
||||
# two Redis writers on one RDB/AOF. Sibling services no longer read anything
|
||||
# from this directory — they locate meta-core over UDP (meta-discovery v1, see
|
||||
# docs/project-architecture/service-discovery.md), so the kv-leader.info file
|
||||
# this script used to publish is gone. Keep the lock; it costs nothing and the
|
||||
# corruption it prevents is unrecoverable.
|
||||
#
|
||||
# Environment variables:
|
||||
# META_CORE_PATH - Path to meta-core data directory (default: /meta-core)
|
||||
@@ -20,7 +24,6 @@ META_CORE_HTTP_PORT="${META_CORE_HTTP_PORT:-9000}"
|
||||
ELECTION_RETRY_SECS="${ELECTION_RETRY_SECS:-5}"
|
||||
|
||||
LOCK_FILE="${META_CORE_PATH}/locks/kv-leader.lock"
|
||||
INFO_FILE="${META_CORE_PATH}/locks/kv-leader.info"
|
||||
SUPERVISORD_CONF="/etc/supervisor/conf.d/supervisord.conf"
|
||||
|
||||
# Get the container's IP address
|
||||
@@ -40,35 +43,13 @@ get_local_ip() {
|
||||
echo "$ip"
|
||||
}
|
||||
|
||||
# Write leader info file (plain text API URL)
|
||||
write_leader_info() {
|
||||
local ip=$(get_local_ip)
|
||||
local api_url="http://${ip}:${META_CORE_HTTP_PORT}"
|
||||
|
||||
# Write atomically using temp file + rename
|
||||
local temp_file="${INFO_FILE}.tmp"
|
||||
echo "$api_url" > "$temp_file"
|
||||
mv "$temp_file" "$INFO_FILE"
|
||||
|
||||
echo "[election] Wrote leader info: $api_url"
|
||||
}
|
||||
|
||||
# Clean up leader info file on shutdown
|
||||
cleanup() {
|
||||
echo "[election] Cleaning up..."
|
||||
rm -f "$INFO_FILE"
|
||||
exit 0
|
||||
}
|
||||
|
||||
# Set up signal handlers for graceful shutdown
|
||||
trap cleanup SIGTERM SIGINT
|
||||
|
||||
# Ensure lock directory exists
|
||||
mkdir -p "$(dirname "$LOCK_FILE")"
|
||||
|
||||
echo "[election] Starting leader election..."
|
||||
echo "[election] Lock file: $LOCK_FILE"
|
||||
echo "[election] Info file: $INFO_FILE"
|
||||
|
||||
# Open lock file for flock (file descriptor 200)
|
||||
exec 200>"$LOCK_FILE"
|
||||
@@ -79,9 +60,6 @@ while true; do
|
||||
if flock -n 200; then
|
||||
echo "[election] Acquired flock - transitioning to LEADER"
|
||||
|
||||
# Write leader info before starting services
|
||||
write_leader_info
|
||||
|
||||
# exec supervisord - replaces this process
|
||||
# When supervisord exits, the flock is released automatically
|
||||
echo "[election] Starting supervisord..."
|
||||
|
||||
+4
-8
@@ -123,14 +123,10 @@ http {
|
||||
# =========================================================================
|
||||
# Service discovery
|
||||
# =========================================================================
|
||||
location /services {
|
||||
proxy_pass http://meta-core-api/services;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
}
|
||||
|
||||
location /services/ {
|
||||
proxy_pass http://meta-core-api/services/;
|
||||
# meta-discovery v1 neighbours (supersedes /services, which stays
|
||||
# while consumers migrate).
|
||||
location /neighbors {
|
||||
proxy_pass http://meta-core-api/neighbors;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
}
|
||||
|
||||
@@ -276,9 +276,8 @@ Bootstrap / discovery
|
||||
GET /status
|
||||
GET /leader
|
||||
GET /urls
|
||||
GET /services (alias: /api/services)
|
||||
GET /services/{name}
|
||||
GET /services/cleanup/stats
|
||||
GET /neighbors (alias: /api/neighbors)
|
||||
GET /api/stats
|
||||
|
||||
Metadata (primary surface for other services)
|
||||
GET /meta
|
||||
@@ -301,6 +300,7 @@ CID-addressed (public, auth-bypassed at the perimeter)
|
||||
|
||||
Editor / KV / schema / snapshot
|
||||
GET /api/metadata/hash-ids
|
||||
GET /api/files/tuples
|
||||
GET /api/metadata/list
|
||||
POST /api/metadata/search
|
||||
POST /api/metadata/batch
|
||||
@@ -339,6 +339,9 @@ Watchers
|
||||
|
||||
Admin
|
||||
POST /api/admin/migrate-dual-roots
|
||||
POST /api/admin/sweep-literature-echoes (dry-run unless ?apply=true;
|
||||
strips query echoes off domain=literature fileType=card records —
|
||||
see storage/literature_echo_sweep.go)
|
||||
|
||||
WebDAV
|
||||
/webdav/... (GET, PUT, DELETE, MKCOL, COPY, MOVE, PROPFIND)
|
||||
|
||||
@@ -18,4 +18,5 @@ require (
|
||||
github.com/cespare/xxhash/v2 v2.2.0 // indirect
|
||||
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
|
||||
github.com/yuin/gopher-lua v1.1.0 // indirect
|
||||
golang.org/x/sys v0.28.0
|
||||
)
|
||||
|
||||
@@ -34,3 +34,5 @@ golang.org/x/crypto v0.31.0/go.mod h1:kDsLvtWBEx7MV9tJOj9bnXsPbxwJQ6csT/x4KIN4Ss
|
||||
golang.org/x/net v0.21.0 h1:AQyQV4dYCvJ7vGmJyKki9+PBdyvhkSd8EIx/qb0AYv4=
|
||||
golang.org/x/net v0.21.0/go.mod h1:bIjVDfnllIU7BJ2DNgfnXvpSvtn8VRwhlsaeUTyUS44=
|
||||
golang.org/x/sys v0.0.0-20190204203706-41f3e6584952/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
|
||||
golang.org/x/sys v0.28.0 h1:Fksou7UEQUWlKvIdsqzJmUmCX3cZuD2+P3XyyzwMhlA=
|
||||
golang.org/x/sys v0.28.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
|
||||
|
||||
@@ -2,6 +2,7 @@ package api
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"strconv"
|
||||
)
|
||||
|
||||
// handleMigrateDualRoots triggers the one-shot sweep that re-unifies
|
||||
@@ -28,3 +29,103 @@ func (s *Server) handleMigrateDualRoots(w http.ResponseWriter, r *http.Request)
|
||||
"fixed": fixed,
|
||||
})
|
||||
}
|
||||
|
||||
// handleMigrateDomainScreen triggers the one-shot sweep that rewrites the
|
||||
// retired `domain=film|tv` values to `screen` and backfills `workForm` from
|
||||
// `contentKind` (METADATA_KEYS.md §14.17). See
|
||||
// storage.MigrateDomainScreen for why the vocabulary change is paid in the
|
||||
// data rather than with a read-side alias.
|
||||
//
|
||||
// Returns {fixed: N} — records changed, not records examined. Safe to call
|
||||
// repeatedly: a record already on the current vocabulary is skipped.
|
||||
func (s *Server) handleMigrateDomainScreen(w http.ResponseWriter, r *http.Request) {
|
||||
if !s.storage.IsConnected() {
|
||||
writeError(w, http.StatusServiceUnavailable, "storage not connected")
|
||||
return
|
||||
}
|
||||
|
||||
fixed, err := s.storage.MigrateDomainScreenWithTimeout()
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
writeJSON(w, http.StatusOK, map[string]interface{}{
|
||||
"success": true,
|
||||
"fixed": fixed,
|
||||
})
|
||||
}
|
||||
|
||||
// handleSweepLiteratureEchoes removes query echoes stored on literature cards:
|
||||
// `anilistid` on a site card, and `workcid` / position keys on any card
|
||||
// (meta-read docs/reading-model.md §7). See storage.SweepLiteratureEchoes for
|
||||
// why chapters and anchor cards' ids are deliberately left alone.
|
||||
//
|
||||
// Dry-run unless `?apply=true`: unlike the migrations above this one deletes
|
||||
// keys, so a bare call only reports what it would delete. An unparseable
|
||||
// `apply` is a 400 rather than a silent dry run, so a typo cannot pass for a
|
||||
// sweep that ran.
|
||||
//
|
||||
// Returns {success, dryRun, planned, fixed, byKey} — planned/fixed count
|
||||
// records, byKey counts fields per key. Safe to call repeatedly: a swept
|
||||
// record plans nothing on the next pass.
|
||||
func (s *Server) handleSweepLiteratureEchoes(w http.ResponseWriter, r *http.Request) {
|
||||
if !s.storage.IsConnected() {
|
||||
writeError(w, http.StatusServiceUnavailable, "storage not connected")
|
||||
return
|
||||
}
|
||||
|
||||
apply := false
|
||||
if v := r.URL.Query().Get("apply"); v != "" {
|
||||
parsed, err := strconv.ParseBool(v)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadRequest, "apply must be true or false")
|
||||
return
|
||||
}
|
||||
apply = parsed
|
||||
}
|
||||
|
||||
res, err := s.storage.SweepLiteratureEchoesWithTimeout(apply)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
writeJSON(w, http.StatusOK, map[string]interface{}{
|
||||
"success": true,
|
||||
"dryRun": !apply,
|
||||
"planned": res.Planned,
|
||||
"fixed": res.Fixed,
|
||||
"byKey": res.ByKey,
|
||||
})
|
||||
}
|
||||
|
||||
// handleMigrateUserCids triggers the one-shot sweep that builds
|
||||
// `udl:idx:user:<uid>:cids` for User Data Layer records written before that
|
||||
// index existed. See storage.UDLBackfillUserCids.
|
||||
//
|
||||
// The index is what makes whole-profile export a set read instead of a
|
||||
// keyspace scan, and it is maintained by the upsert script from now on — so
|
||||
// this only ever matters once, for data already on disk. Until it has run, a
|
||||
// profile that predates the index exports as **empty**, which means a
|
||||
// cross-instance recovery reports success having restored nothing.
|
||||
//
|
||||
// Returns {fixed: N} — cids added, not cids examined. Safe to call repeatedly:
|
||||
// a second pass adds nothing.
|
||||
func (s *Server) handleMigrateUserCids(w http.ResponseWriter, r *http.Request) {
|
||||
if !s.storage.IsConnected() {
|
||||
writeError(w, http.StatusServiceUnavailable, "storage not connected")
|
||||
return
|
||||
}
|
||||
|
||||
fixed, err := s.storage.UDLBackfillUserCids()
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
writeJSON(w, http.StatusOK, map[string]interface{}{
|
||||
"success": true,
|
||||
"fixed": fixed,
|
||||
})
|
||||
}
|
||||
|
||||
@@ -0,0 +1,93 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
|
||||
"github.com/alicebob/miniredis/v2"
|
||||
|
||||
"github.com/metazla/meta-core/internal/storage"
|
||||
)
|
||||
|
||||
// newSweepTestServer is a Server over miniredis with one literature card
|
||||
// carrying a copied `workcid`.
|
||||
func newSweepTestServer(t *testing.T) *Server {
|
||||
t.Helper()
|
||||
mr := miniredis.RunT(t)
|
||||
c := storage.NewClient("")
|
||||
if err := c.Connect("redis://" + mr.Addr()); err != nil {
|
||||
t.Fatalf("connect: %v", err)
|
||||
}
|
||||
t.Cleanup(func() { _ = c.Close() })
|
||||
for k, v := range map[string]string{
|
||||
"domain": "literature",
|
||||
"fileType": "card",
|
||||
"workcid": "bagdsaaaua5qw42lmnfzxi3lbnztwcorrga2tonzy",
|
||||
} {
|
||||
if err := c.SetProperty("card-1", k, v); err != nil {
|
||||
t.Fatalf("seed %s: %v", k, err)
|
||||
}
|
||||
}
|
||||
return &Server{storage: c}
|
||||
}
|
||||
|
||||
type sweepResponse struct {
|
||||
Success bool `json:"success"`
|
||||
DryRun bool `json:"dryRun"`
|
||||
Planned int `json:"planned"`
|
||||
Fixed int `json:"fixed"`
|
||||
ByKey map[string]int `json:"byKey"`
|
||||
}
|
||||
|
||||
func postSweep(t *testing.T, s *Server, query string) (*httptest.ResponseRecorder, sweepResponse) {
|
||||
t.Helper()
|
||||
rr := httptest.NewRecorder()
|
||||
req := httptest.NewRequest(http.MethodPost, "/api/admin/sweep-literature-echoes"+query, nil)
|
||||
s.handleSweepLiteratureEchoes(rr, req)
|
||||
var body sweepResponse
|
||||
if rr.Code == http.StatusOK {
|
||||
if err := json.Unmarshal(rr.Body.Bytes(), &body); err != nil {
|
||||
t.Fatalf("decode %s: %v", rr.Body.String(), err)
|
||||
}
|
||||
}
|
||||
return rr, body
|
||||
}
|
||||
|
||||
// A bare call must never delete: the default is the dry run.
|
||||
func TestSweepLiteratureEchoes_DefaultsToDryRun(t *testing.T) {
|
||||
s := newSweepTestServer(t)
|
||||
|
||||
rr, body := postSweep(t, s, "")
|
||||
if rr.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, body %s", rr.Code, rr.Body.String())
|
||||
}
|
||||
if !body.Success || !body.DryRun || body.Planned != 1 || body.Fixed != 0 || body.ByKey["workcid"] != 1 {
|
||||
t.Errorf("dry run = %+v", body)
|
||||
}
|
||||
if v, _ := s.storage.GetProperty("card-1", "workcid"); v == "" {
|
||||
t.Errorf("dry run deleted workcid")
|
||||
}
|
||||
|
||||
rr, body = postSweep(t, s, "?apply=true")
|
||||
if rr.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, body %s", rr.Code, rr.Body.String())
|
||||
}
|
||||
if body.DryRun || body.Planned != 1 || body.Fixed != 1 || body.ByKey["workcid"] != 1 {
|
||||
t.Errorf("apply = %+v", body)
|
||||
}
|
||||
if v, _ := s.storage.GetProperty("card-1", "workcid"); v != "" {
|
||||
t.Errorf("apply left workcid = %q", v)
|
||||
}
|
||||
}
|
||||
|
||||
// A typo must not pass for a sweep that ran.
|
||||
func TestSweepLiteratureEchoes_RejectsUnparseableApply(t *testing.T) {
|
||||
s := newSweepTestServer(t)
|
||||
|
||||
rr, _ := postSweep(t, s, "?apply=yes-please")
|
||||
if rr.Code != http.StatusBadRequest {
|
||||
t.Errorf("status = %d, want 400", rr.Code)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,132 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"os"
|
||||
"path"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
"github.com/metazla/meta-core/internal/watcher"
|
||||
)
|
||||
|
||||
// registerFileRequest is the body of POST /api/files/register.
|
||||
type registerFileRequest struct {
|
||||
// Path relative to FILES_PATH, e.g. "plugin/subtitles/ab12….eng.srt".
|
||||
Path string `json:"path"`
|
||||
}
|
||||
|
||||
type registerFileResponse struct {
|
||||
Root string `json:"root"`
|
||||
CID string `json:"cid"`
|
||||
Created bool `json:"created"`
|
||||
Path string `json:"path"`
|
||||
Size int64 `json:"size"`
|
||||
}
|
||||
|
||||
// pluginRelPath normalises a FILES_PATH-relative path and refuses anything
|
||||
// outside plugin/. Plugin output is the only tree a service may register this
|
||||
// way: /files/watch is the user's library and belongs to the watcher.
|
||||
func pluginRelPath(raw string) (string, bool) {
|
||||
p := strings.TrimPrefix(filepath.ToSlash(strings.TrimSpace(raw)), "/")
|
||||
if p == "" {
|
||||
return "", false
|
||||
}
|
||||
for _, seg := range strings.Split(p, "/") {
|
||||
if seg == ".." {
|
||||
return "", false
|
||||
}
|
||||
}
|
||||
clean := path.Clean(p)
|
||||
if !strings.HasPrefix(clean, "plugin/") || clean == "plugin/" {
|
||||
return "", false
|
||||
}
|
||||
return clean, true
|
||||
}
|
||||
|
||||
// handleRegisterFile handles POST /api/files/register.
|
||||
//
|
||||
// Does for ONE file what the watcher does for every file it scans: midhash256
|
||||
// it, mint a root carrying filePath/sizeByte/mtimeNano, register the
|
||||
// cid:<midhash> alias, and append a file:events entry so meta-share's
|
||||
// ipfs_seed chunks, seeds and provides it.
|
||||
//
|
||||
// It exists for cores that run with ENABLE_FILE_WATCHER=false (a meta-watch
|
||||
// client box). Without it, bytes written under /files/plugin over WebDAV are
|
||||
// never resolvable by CID and never reach the swarm. Idempotent: the same
|
||||
// content at the same path returns the existing root with created=false.
|
||||
func (s *Server) handleRegisterFile(w http.ResponseWriter, r *http.Request) {
|
||||
if s.storage == nil || !s.storage.IsConnected() {
|
||||
writeError(w, http.StatusServiceUnavailable, "storage not connected")
|
||||
return
|
||||
}
|
||||
|
||||
var req registerFileRequest
|
||||
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
|
||||
writeError(w, http.StatusBadRequest, "invalid JSON body")
|
||||
return
|
||||
}
|
||||
rel, ok := pluginRelPath(req.Path)
|
||||
if !ok {
|
||||
writeError(w, http.StatusBadRequest, "path must be a file under plugin/")
|
||||
return
|
||||
}
|
||||
|
||||
fullPath := filepath.Join(s.config.FilesPath, filepath.FromSlash(rel))
|
||||
absFiles, _ := filepath.Abs(filepath.Join(s.config.FilesPath, "plugin"))
|
||||
absFull, _ := filepath.Abs(fullPath)
|
||||
if !strings.HasPrefix(absFull, absFiles+string(filepath.Separator)) {
|
||||
writeError(w, http.StatusBadRequest, "path must be a file under plugin/")
|
||||
return
|
||||
}
|
||||
|
||||
info, err := os.Stat(fullPath)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusNotFound, "file not found")
|
||||
return
|
||||
}
|
||||
if info.IsDir() {
|
||||
writeError(w, http.StatusBadRequest, "path is a directory, not a file")
|
||||
return
|
||||
}
|
||||
|
||||
midhash, err := watcher.ComputeMidHash256(fullPath, info.Size())
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "failed to hash file")
|
||||
return
|
||||
}
|
||||
|
||||
root, created, err := s.storage.RegisterFileTuple(fullPath, info.Size(), info.ModTime().UnixNano(), midhash)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
if created {
|
||||
ev := watcher.FileEvent{
|
||||
Type: watcher.EventTypeAdd,
|
||||
Path: rel,
|
||||
Size: info.Size(),
|
||||
MidHash256: midhash,
|
||||
Timestamp: watcher.NowMS(),
|
||||
}
|
||||
if err := watcher.PublishEvent(s.storage, ev); err != nil {
|
||||
// The record is written and resolvable; only seeding is late.
|
||||
writeError(w, http.StatusInternalServerError, "registered, but file event not published: "+err.Error())
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
status := http.StatusOK
|
||||
if created {
|
||||
status = http.StatusCreated
|
||||
}
|
||||
writeJSON(w, status, registerFileResponse{
|
||||
Root: root,
|
||||
CID: midhash,
|
||||
Created: created,
|
||||
Path: rel,
|
||||
Size: info.Size(),
|
||||
})
|
||||
}
|
||||
@@ -0,0 +1,144 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/alicebob/miniredis/v2"
|
||||
"github.com/metazla/meta-core/internal/config"
|
||||
"github.com/metazla/meta-core/internal/storage"
|
||||
)
|
||||
|
||||
// newRegisterTestServer wires handleRegisterFile to an in-memory Redis and a
|
||||
// temp FILES_PATH. Built directly, not through NewServer, so no watcher runs —
|
||||
// which is exactly the configuration this route exists for.
|
||||
func newRegisterTestServer(t *testing.T) (*Server, string, *miniredis.Miniredis) {
|
||||
t.Helper()
|
||||
mr := miniredis.RunT(t)
|
||||
c := storage.NewClient("")
|
||||
if err := c.Connect("redis://" + mr.Addr()); err != nil {
|
||||
t.Fatalf("connect to miniredis: %v", err)
|
||||
}
|
||||
t.Cleanup(func() { _ = c.Close() })
|
||||
files := t.TempDir()
|
||||
return &Server{storage: c, config: &config.Config{FilesPath: files}}, files, mr
|
||||
}
|
||||
|
||||
func writeFixture(t *testing.T, files, rel, body string) string {
|
||||
t.Helper()
|
||||
full := filepath.Join(files, filepath.FromSlash(rel))
|
||||
if err := os.MkdirAll(filepath.Dir(full), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(full, []byte(body), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return full
|
||||
}
|
||||
|
||||
func postRegister(t *testing.T, s *Server, path string) (int, registerFileResponse) {
|
||||
t.Helper()
|
||||
body, _ := json.Marshal(registerFileRequest{Path: path})
|
||||
req := httptest.NewRequest(http.MethodPost, "/api/files/register", strings.NewReader(string(body)))
|
||||
rr := httptest.NewRecorder()
|
||||
s.handleRegisterFile(rr, req)
|
||||
var resp registerFileResponse
|
||||
_ = json.Unmarshal(rr.Body.Bytes(), &resp)
|
||||
return rr.Code, resp
|
||||
}
|
||||
|
||||
func streamLen(t *testing.T, mr *miniredis.Miniredis) int {
|
||||
t.Helper()
|
||||
entries, err := mr.Stream("file:events")
|
||||
if err != nil {
|
||||
return 0
|
||||
}
|
||||
return len(entries)
|
||||
}
|
||||
|
||||
// The whole point of the route: with the watcher off, a file written under
|
||||
// plugin/ becomes a record resolvable by its midhash, and meta-share hears
|
||||
// about it on file:events.
|
||||
func TestRegisterFile_MintsAliasAndPublishesEvent(t *testing.T) {
|
||||
s, files, mr := newRegisterTestServer(t)
|
||||
rel := "plugin/subtitles/ab12cd34.eng.srt"
|
||||
full := writeFixture(t, files, rel, "1\n00:00:01,000 --> 00:00:02,000\nHello\n")
|
||||
|
||||
code, resp := postRegister(t, s, rel)
|
||||
if code != http.StatusCreated || !resp.Created {
|
||||
t.Fatalf("want 201 created, got %d %+v", code, resp)
|
||||
}
|
||||
if !strings.HasPrefix(resp.CID, "b") || resp.Root == "" {
|
||||
t.Fatalf("want a multibase cid and a root, got %+v", resp)
|
||||
}
|
||||
|
||||
root, err := s.storage.GetByCID(resp.CID)
|
||||
if err != nil || root != resp.Root {
|
||||
t.Fatalf("cid alias must resolve to the root: got %q err=%v, want %q", root, err, resp.Root)
|
||||
}
|
||||
if fp, _ := s.storage.GetProperty(resp.Root, "filePath"); fp != full {
|
||||
t.Fatalf("filePath = %q, want %q", fp, full)
|
||||
}
|
||||
|
||||
entries, err := mr.Stream("file:events")
|
||||
if err != nil || len(entries) != 1 {
|
||||
t.Fatalf("want exactly one file:events entry, got %d err=%v", len(entries), err)
|
||||
}
|
||||
vals := strings.Join(entries[0].Values, "|")
|
||||
if !strings.Contains(vals, "midhash256|"+resp.CID) || !strings.Contains(vals, "path|"+rel) {
|
||||
t.Fatalf("event must carry the midhash and the relative path, got %q", vals)
|
||||
}
|
||||
}
|
||||
|
||||
// Re-uploading the same subtitle is a no-op: same root, no second event (which
|
||||
// would make meta-share re-chunk and re-provide the file).
|
||||
func TestRegisterFile_IsIdempotent(t *testing.T) {
|
||||
s, files, mr := newRegisterTestServer(t)
|
||||
rel := "plugin/subtitles/same.eng.srt"
|
||||
writeFixture(t, files, rel, "same bytes")
|
||||
|
||||
_, first := postRegister(t, s, rel)
|
||||
code, second := postRegister(t, s, rel)
|
||||
if code != http.StatusOK || second.Created {
|
||||
t.Fatalf("second register must be 200 created=false, got %d %+v", code, second)
|
||||
}
|
||||
if second.Root != first.Root || second.CID != first.CID {
|
||||
t.Fatalf("second register must return the same record: %+v vs %+v", second, first)
|
||||
}
|
||||
if n := streamLen(t, mr); n != 1 {
|
||||
t.Fatalf("an idempotent re-register must not publish again, stream has %d", n)
|
||||
}
|
||||
}
|
||||
|
||||
// Only plugin output may be registered here. /files/watch is the user's
|
||||
// library and belongs to the watcher; traversal must not escape either.
|
||||
func TestRegisterFile_RejectsPathsOutsidePlugin(t *testing.T) {
|
||||
s, files, _ := newRegisterTestServer(t)
|
||||
writeFixture(t, files, "watch/movie.mkv", "x")
|
||||
for _, p := range []string{
|
||||
"", "plugin", "plugin/", "watch/movie.mkv", "/watch/movie.mkv",
|
||||
"plugin/../watch/movie.mkv", "../etc/passwd", "plugin/../../etc/passwd",
|
||||
} {
|
||||
if code, _ := postRegister(t, s, p); code != http.StatusBadRequest {
|
||||
t.Errorf("path %q: want 400, got %d", p, code)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestRegisterFile_MissingFileAndDirectory(t *testing.T) {
|
||||
s, files, _ := newRegisterTestServer(t)
|
||||
if code, _ := postRegister(t, s, "plugin/subtitles/nope.srt"); code != http.StatusNotFound {
|
||||
t.Errorf("missing file: want 404, got %d", code)
|
||||
}
|
||||
if err := os.MkdirAll(filepath.Join(files, "plugin", "subtitles", "dir"), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if code, _ := postRegister(t, s, "plugin/subtitles/dir"); code != http.StatusBadRequest {
|
||||
t.Errorf("directory: want 400, got %d", code)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,74 @@
|
||||
package api
|
||||
|
||||
import "net/http"
|
||||
|
||||
// fileTupleJSON is one file-backed root on the wire.
|
||||
type fileTupleJSON struct {
|
||||
HashID string `json:"hashId"`
|
||||
FilePath string `json:"filePath"`
|
||||
SizeByte int64 `json:"sizeByte"`
|
||||
MtimeNano int64 `json:"mtimeNano"`
|
||||
}
|
||||
|
||||
// FileTuplesResponse is the body of GET /api/files/tuples.
|
||||
//
|
||||
// Files is a pointer so the summary form can omit it while the full form
|
||||
// always carries it — an empty library is `"files": []`, never an absent key,
|
||||
// which a client could mistake for a meta-core that lacks the rows.
|
||||
type FileTuplesResponse struct {
|
||||
Files *[]fileTupleJSON `json:"files,omitempty"`
|
||||
Count int `json:"count"`
|
||||
TotalSize int64 `json:"totalSize"`
|
||||
}
|
||||
|
||||
// handleGetFileTuples handles GET /api/files/tuples.
|
||||
//
|
||||
// Lists the FILE-BACKED roots — records carrying filePath, sizeByte and
|
||||
// mtimeNano, i.e. what the watcher or /api/files/register minted — with their
|
||||
// sizes, plus the count and total. Gateway / cid-rooted records carry no file
|
||||
// and are not included; neither is a locator record that only merged a cache
|
||||
// file's metadata (it has no mtimeNano).
|
||||
//
|
||||
// It exists so a client never walks every hashId to answer "how many files,
|
||||
// how big": GetTuplesForAllFiles reads the three fields for every root in
|
||||
// chunked MGETs server-side (~250 round-trips for 82k roots, no SCAN), where
|
||||
// the client-side equivalent was one HTTP request per record.
|
||||
//
|
||||
// ?summary=1 omits the rows and returns only {count, totalSize}.
|
||||
func (s *Server) handleGetFileTuples(w http.ResponseWriter, r *http.Request) {
|
||||
if !s.storage.IsConnected() {
|
||||
writeError(w, http.StatusServiceUnavailable, "storage not connected")
|
||||
return
|
||||
}
|
||||
|
||||
tuples, err := s.storage.GetTuplesForAllFiles()
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
summary := r.URL.Query().Get("summary")
|
||||
withRows := summary == "" || summary == "0" || summary == "false"
|
||||
|
||||
resp := FileTuplesResponse{Count: len(tuples)}
|
||||
var rows []fileTupleJSON
|
||||
if withRows {
|
||||
rows = make([]fileTupleJSON, 0, len(tuples))
|
||||
}
|
||||
for _, t := range tuples {
|
||||
resp.TotalSize += t.Size
|
||||
if withRows {
|
||||
rows = append(rows, fileTupleJSON{
|
||||
HashID: t.HashID,
|
||||
FilePath: t.FilePath,
|
||||
SizeByte: t.Size,
|
||||
MtimeNano: t.MtimeNano,
|
||||
})
|
||||
}
|
||||
}
|
||||
if withRows {
|
||||
resp.Files = &rows
|
||||
}
|
||||
|
||||
writeJSON(w, http.StatusOK, resp)
|
||||
}
|
||||
@@ -0,0 +1,114 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"sort"
|
||||
"testing"
|
||||
|
||||
"github.com/alicebob/miniredis/v2"
|
||||
"github.com/metazla/meta-core/internal/storage"
|
||||
)
|
||||
|
||||
func newFileTuplesTestServer(t *testing.T) (*Server, *storage.Client) {
|
||||
t.Helper()
|
||||
mr := miniredis.RunT(t)
|
||||
c := storage.NewClient("")
|
||||
if err := c.Connect("redis://" + mr.Addr()); err != nil {
|
||||
t.Fatalf("connect to miniredis: %v", err)
|
||||
}
|
||||
t.Cleanup(func() { _ = c.Close() })
|
||||
return &Server{storage: c}, c
|
||||
}
|
||||
|
||||
func getFileTuples(t *testing.T, s *Server, query string) (int, FileTuplesResponse) {
|
||||
t.Helper()
|
||||
req := httptest.NewRequest(http.MethodGet, "/api/files/tuples"+query, nil)
|
||||
rr := httptest.NewRecorder()
|
||||
s.handleGetFileTuples(rr, req)
|
||||
var resp FileTuplesResponse
|
||||
_ = json.Unmarshal(rr.Body.Bytes(), &resp)
|
||||
return rr.Code, resp
|
||||
}
|
||||
|
||||
func mustMint(t *testing.T, c *storage.Client, path string, size, mtime int64) string {
|
||||
t.Helper()
|
||||
id, err := c.Mint(path, size, mtime)
|
||||
if err != nil {
|
||||
t.Fatalf("mint %s: %v", path, err)
|
||||
}
|
||||
return id
|
||||
}
|
||||
|
||||
// Only records that are actually files count: a gateway record and a record
|
||||
// that merely carries a path + size (no mtimeNano) are both left out.
|
||||
func TestFileTuples_OnlyFileBackedRoots(t *testing.T) {
|
||||
s, c := newFileTuplesTestServer(t)
|
||||
a := mustMint(t, c, "/files/watch/a.mkv", 1000, 1)
|
||||
b := mustMint(t, c, "/files/watch/b.mkv", 2500, 2)
|
||||
if err := c.SetMetadataFlat("bagcsgatewayrecord", map[string]string{"title": "Some Release"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := c.SetMetadataFlat("01partialroot", map[string]string{"filePath": "/files/cache/x.mkv", "sizeByte": "999"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
code, resp := getFileTuples(t, s, "")
|
||||
if code != http.StatusOK {
|
||||
t.Fatalf("want 200, got %d", code)
|
||||
}
|
||||
if resp.Count != 2 || resp.TotalSize != 3500 {
|
||||
t.Fatalf("want count 2 / totalSize 3500, got %d / %d", resp.Count, resp.TotalSize)
|
||||
}
|
||||
if resp.Files == nil || len(*resp.Files) != 2 {
|
||||
t.Fatalf("want 2 rows, got %+v", resp.Files)
|
||||
}
|
||||
got := []string{(*resp.Files)[0].HashID, (*resp.Files)[1].HashID}
|
||||
want := []string{a, b}
|
||||
sort.Strings(got)
|
||||
sort.Strings(want)
|
||||
if got[0] != want[0] || got[1] != want[1] {
|
||||
t.Fatalf("want roots %v, got %v", want, got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFileTuples_SummaryOmitsRows(t *testing.T) {
|
||||
s, c := newFileTuplesTestServer(t)
|
||||
mustMint(t, c, "/files/watch/a.mkv", 1000, 1)
|
||||
mustMint(t, c, "/files/watch/b.mkv", 2500, 2)
|
||||
|
||||
code, resp := getFileTuples(t, s, "?summary=1")
|
||||
if code != http.StatusOK {
|
||||
t.Fatalf("want 200, got %d", code)
|
||||
}
|
||||
if resp.Files != nil {
|
||||
t.Fatalf("summary must omit rows, got %d", len(*resp.Files))
|
||||
}
|
||||
if resp.Count != 2 || resp.TotalSize != 3500 {
|
||||
t.Fatalf("want count 2 / totalSize 3500, got %d / %d", resp.Count, resp.TotalSize)
|
||||
}
|
||||
}
|
||||
|
||||
// An empty library still carries "files": [] — clients treat an absent key as
|
||||
// "this meta-core cannot list files", so it must not disappear when empty.
|
||||
func TestFileTuples_EmptyLibraryKeepsFilesKey(t *testing.T) {
|
||||
s, _ := newFileTuplesTestServer(t)
|
||||
|
||||
code, resp := getFileTuples(t, s, "")
|
||||
if code != http.StatusOK {
|
||||
t.Fatalf("want 200, got %d", code)
|
||||
}
|
||||
if resp.Files == nil || len(*resp.Files) != 0 || resp.Count != 0 {
|
||||
t.Fatalf("want an empty files array and count 0, got %+v", resp)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFileTuples_StorageNotConnected(t *testing.T) {
|
||||
s := &Server{storage: storage.NewClient("")}
|
||||
|
||||
code, _ := getFileTuples(t, s, "")
|
||||
if code != http.StatusServiceUnavailable {
|
||||
t.Fatalf("want 503, got %d", code)
|
||||
}
|
||||
}
|
||||
+35
-47
@@ -14,6 +14,7 @@ import (
|
||||
|
||||
"github.com/gorilla/mux"
|
||||
"github.com/metazla/meta-core/internal/leader"
|
||||
"github.com/metazla/meta-core/internal/meshdisco"
|
||||
)
|
||||
|
||||
// contentTypeByExt maps file extensions to MIME types
|
||||
@@ -547,62 +548,49 @@ func (s *Server) handleGetFileByCID(w http.ResponseWriter, r *http.Request) {
|
||||
http.ServeContent(w, r, fileInfo.Name(), fileInfo.ModTime(), file)
|
||||
}
|
||||
|
||||
// handleListServices handles GET /services and GET /api/services
|
||||
// Supports optional ?current=<service-name> query parameter
|
||||
func (s *Server) handleListServices(w http.ResponseWriter, r *http.Request) {
|
||||
services, err := s.discovery.DiscoverAll()
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
// handleListNeighbors handles GET /neighbors and GET /api/neighbors.
|
||||
//
|
||||
// Beacon v2 (docs/project-architecture/beacon-v2.md). The list is this node's
|
||||
// in-memory view of the UDP group, so it works across stacks that share a
|
||||
// network but no volume. Instances of one name collapse to the most recent
|
||||
// (one row per service, the nav menu's shape); `?all=1` returns every
|
||||
// instance, `?cap=<pattern>` keeps only nodes with a matching capability.
|
||||
func (s *Server) handleListNeighbors(w http.ResponseWriter, r *http.Request) {
|
||||
response := map[string]interface{}{
|
||||
"services": services,
|
||||
"count": len(services),
|
||||
}
|
||||
|
||||
// Add 'current' field if query parameter is provided
|
||||
current := r.URL.Query().Get("current")
|
||||
if current != "" {
|
||||
response["current"] = current
|
||||
"current": s.config.ServiceName,
|
||||
"neighbors": []interface{}{},
|
||||
"count": 0,
|
||||
}
|
||||
|
||||
if s.mesh == nil {
|
||||
// Discovery off or failed to bind. Report empty rather than 503 —
|
||||
// the nav is decoration and callers already treat [] as "hide".
|
||||
response["enabled"] = false
|
||||
writeJSON(w, http.StatusOK, response)
|
||||
}
|
||||
|
||||
// handleGetService handles GET /services/{name}
|
||||
func (s *Server) handleGetService(w http.ResponseWriter, r *http.Request) {
|
||||
vars := mux.Vars(r)
|
||||
name := vars["name"]
|
||||
|
||||
if name == "" {
|
||||
writeError(w, http.StatusBadRequest, "service name is required")
|
||||
return
|
||||
}
|
||||
|
||||
service, err := s.discovery.Discover(name)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
var list []meshdisco.Neighbor
|
||||
if r.URL.Query().Get("all") != "" {
|
||||
list = s.mesh.Neighbors()
|
||||
} else {
|
||||
list = s.mesh.NeighborsByName()
|
||||
}
|
||||
if pattern := r.URL.Query().Get("cap"); pattern != "" {
|
||||
kept := list[:0]
|
||||
for _, nb := range list {
|
||||
if nb.Matches(pattern) {
|
||||
kept = append(kept, nb)
|
||||
}
|
||||
}
|
||||
list = kept
|
||||
}
|
||||
|
||||
if service == nil {
|
||||
writeError(w, http.StatusNotFound, "service not found")
|
||||
return
|
||||
}
|
||||
|
||||
writeJSON(w, http.StatusOK, service)
|
||||
}
|
||||
|
||||
// handleCleanupStats handles GET /services/cleanup/stats
|
||||
func (s *Server) handleCleanupStats(w http.ResponseWriter, r *http.Request) {
|
||||
if s.cleaner == nil {
|
||||
writeError(w, http.StatusServiceUnavailable, "cleaner not initialized")
|
||||
return
|
||||
}
|
||||
|
||||
stats := s.cleaner.Stats()
|
||||
writeJSON(w, http.StatusOK, stats)
|
||||
response["enabled"] = true
|
||||
response["neighbors"] = list
|
||||
response["count"] = len(list)
|
||||
response["self"] = s.mesh.Self()
|
||||
writeJSON(w, http.StatusOK, response)
|
||||
}
|
||||
|
||||
// writeJSON writes a JSON response
|
||||
|
||||
@@ -3,10 +3,14 @@ package api
|
||||
import (
|
||||
"encoding/base64"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"log"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"github.com/metazla/meta-core/internal/identity"
|
||||
"github.com/metazla/meta-core/internal/storage"
|
||||
)
|
||||
|
||||
// identityStatusResponse is the shape of GET /api/identity. The private key
|
||||
@@ -50,11 +54,68 @@ type identityImportResponse struct {
|
||||
UID string `json:"uid"`
|
||||
Curve string `json:"curve"`
|
||||
CreatedAt int64 `json:"createdAt"`
|
||||
// Created distinguishes "this key was new here" (201) from "this key was
|
||||
// already in the keystore" (200). Import is the sign-in primitive for
|
||||
// clients: presenting the secret key is the proof of ownership, so a
|
||||
// re-import must succeed rather than conflict. Clients use the flag only
|
||||
// to word the confirmation ("new profile" vs "welcome back").
|
||||
Created bool `json:"created"`
|
||||
}
|
||||
|
||||
// identityChallengeRequest asks for a one-shot challenge to sign.
|
||||
type identityChallengeRequest struct {
|
||||
UID string `json:"uid"`
|
||||
Action string `json:"action"`
|
||||
}
|
||||
|
||||
type identityChallengeResponse struct {
|
||||
Challenge string `json:"challenge"`
|
||||
ExpiresAt int64 `json:"expiresAt"`
|
||||
}
|
||||
|
||||
// identityProof is the proof-of-possession carried by reveal and delete: the
|
||||
// challenge text as issued, and a signature over it by the account's key.
|
||||
//
|
||||
// The signature is produced by whoever holds the key — the browser, locally.
|
||||
// It is NOT obtainable from POST /api/identity/sign, which refuses any payload
|
||||
// in the challenge domain; that refusal is what makes this proof mean
|
||||
// something on a node that stores every private key in plaintext.
|
||||
type identityProof struct {
|
||||
Challenge string `json:"challenge"`
|
||||
Signature string `json:"signature"`
|
||||
}
|
||||
|
||||
// identityRevealRequest asks for an existing account's private key back.
|
||||
//
|
||||
// Gated on proof-of-possession, which makes it a "does this node hold the same
|
||||
// key I do" check rather than the recovery path it once was. That is the
|
||||
// deliberate consequence of the rule this file enforces: the private key, and
|
||||
// nothing else, is authority over the account. An operator with root on the box
|
||||
// can still read the file; an operator with only the API cannot.
|
||||
type identityRevealRequest struct {
|
||||
Confirm bool `json:"confirm"`
|
||||
UID string `json:"uid"`
|
||||
identityProof
|
||||
}
|
||||
|
||||
type identityRevealResponse struct {
|
||||
UID string `json:"uid"`
|
||||
Curve string `json:"curve"`
|
||||
PrivateKeyHex string `json:"privateKeyHex"`
|
||||
}
|
||||
|
||||
type identityDeleteRequest struct {
|
||||
Confirm bool `json:"confirm"`
|
||||
UID string `json:"uid"`
|
||||
identityProof
|
||||
}
|
||||
|
||||
// identityDeleteResponse reports what the delete actually removed. The purge
|
||||
// counts are the point: "account removed" alone leaves the caller unable to
|
||||
// tell a clean removal from one that silently stranded a decade of ratings.
|
||||
type identityDeleteResponse struct {
|
||||
UID string `json:"uid"`
|
||||
Purged storage.UDLUserStats `json:"purged"`
|
||||
}
|
||||
|
||||
type identitySignRequest struct {
|
||||
@@ -68,6 +129,41 @@ type identitySignResponse struct {
|
||||
SigB64 string `json:"sigB64"`
|
||||
}
|
||||
|
||||
// identitySignBatchRequest signs many payloads under one account in one call.
|
||||
//
|
||||
// Exists because a User Data Layer write signs exactly one record, and the bulk
|
||||
// verbs above it ("mark this whole series seen") produce one record per episode:
|
||||
// a 300-episode show is 300 sequential sign round trips through
|
||||
// meta-watch -> meta-search -> here, which dominates the operation. The private
|
||||
// key still never leaves this process; only the transport is batched.
|
||||
type identitySignBatchRequest struct {
|
||||
// PayloadsB64 are the canonical byte strings to sign, in order. The
|
||||
// response's signatures line up index-for-index.
|
||||
PayloadsB64 []string `json:"payloadsB64"`
|
||||
// UID selects which account signs — same optional single-identity
|
||||
// back-compat as identitySignRequest.
|
||||
UID string `json:"uid,omitempty"`
|
||||
}
|
||||
|
||||
type identitySignBatchResponse struct {
|
||||
SigsB64 []string `json:"sigsB64"`
|
||||
}
|
||||
|
||||
// maxSignBatch bounds one batch. A signature is cheap but not free, and the
|
||||
// request holds the account lock for its duration; a caller with more than this
|
||||
// should page. Sized well above the longest real series (Supernatural, 327).
|
||||
const maxSignBatch = 1000
|
||||
|
||||
// reservedPayloadMessage explains the one refusal the signing endpoints make.
|
||||
//
|
||||
// This is the load-bearing half of proof-of-possession on a node that holds
|
||||
// every private key: without it, a caller wanting to delete someone else's
|
||||
// account just asks meta-core to sign the challenge for them, and the gate on
|
||||
// reveal/delete becomes a formality. See identity.ChallengeDomain.
|
||||
const reservedPayloadMessage = "payload is in the reserved account-authorisation domain (" +
|
||||
identity.ChallengeDomain + "…); meta-core will not sign one. " +
|
||||
"Signing a challenge is what proves you hold the key, so it must be done where the key is."
|
||||
|
||||
type identityAEADKeyResponse struct {
|
||||
AEADKeyB64 string `json:"aeadKeyB64"`
|
||||
}
|
||||
@@ -172,9 +268,17 @@ func (s *Server) handleIdentityImport(w http.ResponseWriter, r *http.Request) {
|
||||
return
|
||||
}
|
||||
dir := s.config.IdentityAccountsDir()
|
||||
if identity.ExistsByUID(dir, id.UID) {
|
||||
writeErrorSlug(w, http.StatusConflict, "identity_exists",
|
||||
"an account with this key already exists", false)
|
||||
// Idempotent: importing a key that is already in the keystore is how a
|
||||
// second device signs in to an existing account. Answer 200 with the
|
||||
// stored account (preserving its original createdAt) instead of 409 —
|
||||
// the caller proved ownership by presenting the key.
|
||||
if existing, err := identity.LoadByUID(dir, id.UID); err == nil && existing != nil {
|
||||
writeJSON(w, http.StatusOK, identityImportResponse{
|
||||
UID: existing.UID,
|
||||
Curve: existing.Curve,
|
||||
CreatedAt: existing.CreatedAt,
|
||||
Created: false,
|
||||
})
|
||||
return
|
||||
}
|
||||
if err := identity.SaveAccount(dir, id); err != nil {
|
||||
@@ -185,49 +289,176 @@ func (s *Server) handleIdentityImport(w http.ResponseWriter, r *http.Request) {
|
||||
UID: id.UID,
|
||||
Curve: id.Curve,
|
||||
CreatedAt: id.CreatedAt,
|
||||
Created: true,
|
||||
})
|
||||
}
|
||||
|
||||
func (s *Server) handleIdentityDelete(w http.ResponseWriter, r *http.Request) {
|
||||
var body identityDeleteRequest
|
||||
// Tolerate empty body — confirm flag in a query param is also acceptable
|
||||
// for shells where sending a JSON body with DELETE is awkward.
|
||||
_ = json.NewDecoder(r.Body).Decode(&body)
|
||||
if !body.Confirm && r.URL.Query().Get("confirm") != "true" {
|
||||
writeErrorSlug(w, http.StatusBadRequest, "confirm_required",
|
||||
"identity deletion requires {\"confirm\": true} or ?confirm=true", false)
|
||||
// handleIdentityChallenge issues the one-shot text a client signs to prove it
|
||||
// holds an account's private key.
|
||||
//
|
||||
// Unauthenticated by design: a challenge is useless without the key, and
|
||||
// GET /api/identity/accounts already publishes every uid, so refusing to mint
|
||||
// one for an unknown caller would protect nothing. Requiring the account to
|
||||
// exist is for a clear error, not secrecy.
|
||||
func (s *Server) handleIdentityChallenge(w http.ResponseWriter, r *http.Request) {
|
||||
var body identityChallengeRequest
|
||||
if err := json.NewDecoder(r.Body).Decode(&body); err != nil {
|
||||
writeError(w, http.StatusBadRequest, "invalid JSON body")
|
||||
return
|
||||
}
|
||||
dir := s.config.IdentityAccountsDir()
|
||||
uid := strings.TrimSpace(body.UID)
|
||||
if uid == "" {
|
||||
writeErrorSlug(w, http.StatusBadRequest, "uid_required", "uid is required", false)
|
||||
return
|
||||
}
|
||||
action := identity.Action(strings.TrimSpace(body.Action))
|
||||
if !identity.ValidAction(action) {
|
||||
writeErrorSlug(w, http.StatusBadRequest, "bad_action",
|
||||
`action must be "reveal" or "delete"`, false)
|
||||
return
|
||||
}
|
||||
if !identity.ExistsByUID(s.config.IdentityAccountsDir(), uid) {
|
||||
writeErrorSlug(w, http.StatusNotFound, "no_identity", "no identity for the given uid", false)
|
||||
return
|
||||
}
|
||||
text, expires, err := s.challenges.Issue(uid, action)
|
||||
if err != nil {
|
||||
if errors.Is(err, identity.ErrChallengeFull) {
|
||||
writeErrorSlug(w, http.StatusServiceUnavailable, "challenges_full", err.Error(), true)
|
||||
return
|
||||
}
|
||||
writeErrorSlug(w, http.StatusBadRequest, "bad_uid", err.Error(), false)
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, identityChallengeResponse{
|
||||
Challenge: text,
|
||||
ExpiresAt: expires.Unix(),
|
||||
})
|
||||
}
|
||||
|
||||
// requireProof resolves the target account and verifies the caller's
|
||||
// proof-of-possession for `action`. status == 0 means the proof held.
|
||||
//
|
||||
// The uid is mandatory here, unlike elsewhere in this file: the old "sole
|
||||
// account" fallback would have this pick a target the signature does not name,
|
||||
// and a proof that authorises an unnamed account authorises the wrong one as
|
||||
// soon as a second profile exists.
|
||||
func (s *Server) requireProof(uid string, proof identityProof, action identity.Action) (
|
||||
id *identity.Identity, status int, slug, msg string,
|
||||
) {
|
||||
uid = strings.TrimSpace(uid)
|
||||
if uid == "" {
|
||||
return nil, http.StatusBadRequest, "uid_required",
|
||||
"uid is required: the signature authorises one specific account"
|
||||
}
|
||||
if proof.Challenge == "" || proof.Signature == "" {
|
||||
return nil, http.StatusUnauthorized, "signature_required",
|
||||
"this operation requires proof you hold the account's private key: " +
|
||||
"POST /api/identity/challenge, sign the returned text, and resend it as {challenge, signature}"
|
||||
}
|
||||
sig, err := base64.StdEncoding.DecodeString(proof.Signature)
|
||||
if err != nil {
|
||||
return nil, http.StatusBadRequest, "bad_signature", "signature is not valid base64"
|
||||
}
|
||||
// Resolve before redeeming so a request naming a missing account does not
|
||||
// burn a challenge it could not have used anyway.
|
||||
id, status, slug, msg = s.resolveAccount(uid)
|
||||
if status != 0 {
|
||||
return nil, status, slug, msg
|
||||
}
|
||||
// Redeem first, verify second: the challenge is single-use either way, so a
|
||||
// wrong signature costs the attacker a fresh round trip per guess rather
|
||||
// than letting them grind one challenge offline against the same nonce.
|
||||
if err := s.challenges.Redeem(proof.Challenge, uid, action); err != nil {
|
||||
return nil, http.StatusUnauthorized, "bad_challenge", err.Error()
|
||||
}
|
||||
if err := identity.Verify(uid, []byte(proof.Challenge), sig); err != nil {
|
||||
return nil, http.StatusUnauthorized, "bad_signature", err.Error()
|
||||
}
|
||||
return id, 0, "", ""
|
||||
}
|
||||
|
||||
// handleIdentityReveal returns an existing account's private key to a caller
|
||||
// that proves it already holds that key.
|
||||
//
|
||||
// Deliberately POST (never a bare GET) so it cannot be triggered by a link or
|
||||
// an image tag. The proof is what actually guards it: before this gate existed,
|
||||
// anything that could reach the port could read every private key on the node.
|
||||
func (s *Server) handleIdentityReveal(w http.ResponseWriter, r *http.Request) {
|
||||
var body identityRevealRequest
|
||||
_ = json.NewDecoder(r.Body).Decode(&body)
|
||||
uid := strings.TrimSpace(body.UID)
|
||||
if uid == "" {
|
||||
uid = strings.TrimSpace(r.URL.Query().Get("uid"))
|
||||
}
|
||||
id, status, slug, msg := s.requireProof(uid, body.identityProof, identity.ActionReveal)
|
||||
if status != 0 {
|
||||
writeErrorSlug(w, status, slug, msg, false)
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, identityRevealResponse{
|
||||
UID: id.UID,
|
||||
Curve: id.Curve,
|
||||
PrivateKeyHex: id.PrivateKeyHex,
|
||||
})
|
||||
}
|
||||
|
||||
// handleIdentityDelete removes one account and everything it wrote.
|
||||
//
|
||||
// Two things happen, in this order and no other:
|
||||
//
|
||||
// 1. the account's User Data Layer rows are purged (cells plus every index
|
||||
// that names the uid, including the cid-scoped ones shared with other
|
||||
// accounts);
|
||||
// 2. the keypair file is removed.
|
||||
//
|
||||
// Purge-then-delete so a failure never strands data: if the purge errors the
|
||||
// key is still on disk, the account still exists, and the caller can retry.
|
||||
// The reverse order would leave rows nobody can ever attribute, reach or
|
||||
// rewrite — the orphaning this endpoint exists to prevent.
|
||||
//
|
||||
// What deletion still does NOT do, and cannot: records already signed under
|
||||
// this uid and replicated to other peers keep verifying there. Removing a key
|
||||
// is not a recall.
|
||||
func (s *Server) handleIdentityDelete(w http.ResponseWriter, r *http.Request) {
|
||||
var body identityDeleteRequest
|
||||
// Tolerate an empty body so the uid may ride in the query — meta-watch's
|
||||
// proxy puts it there. The proof itself must be in the body: a signature in
|
||||
// a URL ends up in every access log between here and the browser.
|
||||
_ = json.NewDecoder(r.Body).Decode(&body)
|
||||
uid := strings.TrimSpace(body.UID)
|
||||
if uid == "" {
|
||||
// Back-compat: with no uid, delete the sole account. Ambiguous (and so
|
||||
// refused) once more than one account exists.
|
||||
list, err := identity.List(dir)
|
||||
uid = strings.TrimSpace(r.URL.Query().Get("uid"))
|
||||
}
|
||||
id, status, slug, msg := s.requireProof(uid, body.identityProof, identity.ActionDelete)
|
||||
if status != 0 {
|
||||
writeErrorSlug(w, status, slug, msg, false)
|
||||
return
|
||||
}
|
||||
|
||||
// A purge needs storage. Refusing here — rather than deleting the key and
|
||||
// reporting the purge as "0 records" — is the difference between a retry
|
||||
// and permanent orphaned data: once the key is gone the account cannot be
|
||||
// named for a second attempt through this API.
|
||||
if !s.storage.IsConnected() {
|
||||
writeErrorSlug(w, http.StatusServiceUnavailable, "storage_unavailable",
|
||||
"storage is not connected; refusing to delete the key while its data cannot be purged", true)
|
||||
return
|
||||
}
|
||||
purged, err := s.storage.UDLPurgeUser(id.UID)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
writeErrorSlug(w, http.StatusInternalServerError, "purge_failed",
|
||||
"could not purge this account's data; the account was NOT deleted: "+err.Error(), true)
|
||||
return
|
||||
}
|
||||
switch len(list) {
|
||||
case 0:
|
||||
w.WriteHeader(http.StatusNoContent) // idempotent
|
||||
return
|
||||
case 1:
|
||||
uid = list[0].UID
|
||||
default:
|
||||
writeErrorSlug(w, http.StatusBadRequest, "uid_required",
|
||||
"multiple accounts exist; specify uid to delete", false)
|
||||
|
||||
if err := identity.DeleteByUID(s.config.IdentityAccountsDir(), id.UID); err != nil {
|
||||
writeErrorSlug(w, http.StatusInternalServerError, "delete_failed", err.Error(), false)
|
||||
return
|
||||
}
|
||||
}
|
||||
if err := identity.DeleteByUID(dir, uid); err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
log.Printf("[API] Deleted account %s and purged %d User Data Layer records across %d cids",
|
||||
id.UID, purged.Records, purged.Cids)
|
||||
writeJSON(w, http.StatusOK, identityDeleteResponse{UID: id.UID, Purged: purged})
|
||||
}
|
||||
|
||||
func (s *Server) handleIdentitySign(w http.ResponseWriter, r *http.Request) {
|
||||
@@ -250,6 +481,10 @@ func (s *Server) handleIdentitySign(w http.ResponseWriter, r *http.Request) {
|
||||
writeError(w, http.StatusBadRequest, "bytesB64 is not valid base64")
|
||||
return
|
||||
}
|
||||
if identity.IsChallengePayload(raw) {
|
||||
writeErrorSlug(w, http.StatusForbidden, "reserved_payload", reservedPayloadMessage, false)
|
||||
return
|
||||
}
|
||||
sig, err := identity.Sign(id, raw)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
@@ -260,6 +495,58 @@ func (s *Server) handleIdentitySign(w http.ResponseWriter, r *http.Request) {
|
||||
})
|
||||
}
|
||||
|
||||
// handleIdentitySignBatch signs every payload in the request under one account.
|
||||
//
|
||||
// All-or-nothing: a single bad base64 or signing failure fails the whole
|
||||
// request rather than returning a short array. The caller pairs signatures with
|
||||
// records by index, so a partial response would silently misalign them — and a
|
||||
// misaligned signature verifies against the wrong record, which is worse than
|
||||
// an error.
|
||||
func (s *Server) handleIdentitySignBatch(w http.ResponseWriter, r *http.Request) {
|
||||
var body identitySignBatchRequest
|
||||
if err := json.NewDecoder(r.Body).Decode(&body); err != nil {
|
||||
writeError(w, http.StatusBadRequest, "invalid JSON body")
|
||||
return
|
||||
}
|
||||
uid := body.UID
|
||||
if strings.TrimSpace(uid) == "" {
|
||||
uid = r.URL.Query().Get("uid")
|
||||
}
|
||||
if len(body.PayloadsB64) == 0 {
|
||||
writeError(w, http.StatusBadRequest, "payloadsB64 is required")
|
||||
return
|
||||
}
|
||||
if len(body.PayloadsB64) > maxSignBatch {
|
||||
writeError(w, http.StatusBadRequest, "too many payloads in one batch")
|
||||
return
|
||||
}
|
||||
id, status, slug, msg := s.resolveAccount(uid)
|
||||
if status != 0 {
|
||||
writeErrorSlug(w, status, slug, msg, false)
|
||||
return
|
||||
}
|
||||
sigs := make([]string, 0, len(body.PayloadsB64))
|
||||
for i, p := range body.PayloadsB64 {
|
||||
raw, err := base64.StdEncoding.DecodeString(p)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadRequest, "payloadsB64["+strconv.Itoa(i)+"] is not valid base64")
|
||||
return
|
||||
}
|
||||
if identity.IsChallengePayload(raw) {
|
||||
writeErrorSlug(w, http.StatusForbidden, "reserved_payload",
|
||||
"payloadsB64["+strconv.Itoa(i)+"]: "+reservedPayloadMessage, false)
|
||||
return
|
||||
}
|
||||
sig, err := identity.Sign(id, raw)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
sigs = append(sigs, base64.StdEncoding.EncodeToString(sig))
|
||||
}
|
||||
writeJSON(w, http.StatusOK, identitySignBatchResponse{SigsB64: sigs})
|
||||
}
|
||||
|
||||
func (s *Server) handleIdentityAEADKey(w http.ResponseWriter, r *http.Request) {
|
||||
id, status, slug, msg := s.resolveAccount(r.URL.Query().Get("uid"))
|
||||
if status != 0 {
|
||||
|
||||
@@ -0,0 +1,257 @@
|
||||
package api
|
||||
|
||||
// End-to-end exercise of the account gate against a live router + miniredis:
|
||||
// generate -> write UDL rows -> stats -> challenge -> sign -> delete -> verify
|
||||
// both the key and every row are gone. Deliberately drives the real HTTP
|
||||
// handlers rather than the packages beneath them, because the bugs that matter
|
||||
// here (a challenge accepted for the wrong action, a purge that runs after the
|
||||
// key is gone) live in the wiring.
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/base64"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
|
||||
"github.com/alicebob/miniredis/v2"
|
||||
"github.com/gorilla/mux"
|
||||
"github.com/metazla/meta-core/internal/config"
|
||||
"github.com/metazla/meta-core/internal/identity"
|
||||
"github.com/metazla/meta-core/internal/storage"
|
||||
)
|
||||
|
||||
func newTestServer(t *testing.T) *Server {
|
||||
t.Helper()
|
||||
mr := miniredis.RunT(t)
|
||||
stor := storage.NewClient("")
|
||||
if err := stor.Connect("redis://" + mr.Addr()); err != nil {
|
||||
t.Fatalf("connect: %v", err)
|
||||
}
|
||||
t.Cleanup(func() { _ = stor.Close() })
|
||||
|
||||
s := &Server{
|
||||
config: &config.Config{MetaCorePath: t.TempDir()},
|
||||
storage: stor,
|
||||
challenges: identity.NewChallengeStore(),
|
||||
router: mux.NewRouter(),
|
||||
}
|
||||
s.setupIdentityTestRoutes()
|
||||
return s
|
||||
}
|
||||
|
||||
// Only the routes under test, so this does not depend on every other subsystem
|
||||
// booting.
|
||||
func (s *Server) setupIdentityTestRoutes() {
|
||||
s.router.HandleFunc("/api/identity/accounts", s.handleIdentityAccounts).Methods("GET")
|
||||
s.router.HandleFunc("/api/identity/challenge", s.handleIdentityChallenge).Methods("POST")
|
||||
s.router.HandleFunc("/api/identity/generate", s.handleIdentityGenerate).Methods("POST")
|
||||
s.router.HandleFunc("/api/identity/reveal", s.handleIdentityReveal).Methods("POST")
|
||||
s.router.HandleFunc("/api/identity", s.handleIdentityDelete).Methods("DELETE")
|
||||
s.router.HandleFunc("/api/identity/sign", s.handleIdentitySign).Methods("POST")
|
||||
s.router.HandleFunc("/api/udl/users/stats", s.handleUDLUserStats).Methods("GET")
|
||||
}
|
||||
|
||||
func do(t *testing.T, s *Server, method, path string, body any) (*httptest.ResponseRecorder, map[string]any) {
|
||||
t.Helper()
|
||||
var rdr *bytes.Reader
|
||||
if body != nil {
|
||||
b, _ := json.Marshal(body)
|
||||
rdr = bytes.NewReader(b)
|
||||
} else {
|
||||
rdr = bytes.NewReader(nil)
|
||||
}
|
||||
req := httptest.NewRequest(method, path, rdr)
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
rec := httptest.NewRecorder()
|
||||
s.router.ServeHTTP(rec, req)
|
||||
out := map[string]any{}
|
||||
_ = json.Unmarshal(rec.Body.Bytes(), &out)
|
||||
return rec, out
|
||||
}
|
||||
|
||||
// signAs produces the proof a browser would, from the raw key.
|
||||
func signAs(t *testing.T, privHex, challenge string) map[string]any {
|
||||
t.Helper()
|
||||
raw, err := hex.DecodeString(privHex)
|
||||
if err != nil {
|
||||
t.Fatalf("decode key: %v", err)
|
||||
}
|
||||
id, err := identity.Import(hex.EncodeToString(raw))
|
||||
if err != nil {
|
||||
t.Fatalf("import: %v", err)
|
||||
}
|
||||
sig, err := identity.Sign(id, []byte(challenge))
|
||||
if err != nil {
|
||||
t.Fatalf("sign: %v", err)
|
||||
}
|
||||
return map[string]any{
|
||||
"challenge": challenge,
|
||||
"signature": base64.StdEncoding.EncodeToString(sig),
|
||||
}
|
||||
}
|
||||
|
||||
func TestE2E_DeleteRequiresProofAndPurgesEverything(t *testing.T) {
|
||||
s := newTestServer(t)
|
||||
|
||||
rec, gen := do(t, s, "POST", "/api/identity/generate", nil)
|
||||
if rec.Code != http.StatusCreated {
|
||||
t.Fatalf("generate: %d %s", rec.Code, rec.Body)
|
||||
}
|
||||
uid := gen["uid"].(string)
|
||||
priv := gen["privateKeyHex"].(string)
|
||||
|
||||
// Some data to lose.
|
||||
for _, cell := range [][2]string{{"cidA", "like"}, {"cidA", "rating"}, {"cidB", "seek"}} {
|
||||
if ok, err := s.storage.UDLUpsertIfNewer(uid, cell[0], cell[1], 1, 100, "rec", "", false, false); err != nil || !ok {
|
||||
t.Fatalf("seed: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// The count the UI shows in the confirmation.
|
||||
rec, stats := do(t, s, "GET", "/api/udl/users/stats", nil)
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("stats: %d %s", rec.Code, rec.Body)
|
||||
}
|
||||
users := stats["users"].(map[string]any)
|
||||
mine := users[uid].(map[string]any)
|
||||
if mine["records"].(float64) != 3 || mine["cids"].(float64) != 2 {
|
||||
t.Fatalf("stats = %v, want 3 records / 2 cids", mine)
|
||||
}
|
||||
|
||||
// Unsigned delete must be refused, and must not have touched anything.
|
||||
rec, body := do(t, s, "DELETE", "/api/identity", map[string]any{"uid": uid, "confirm": true})
|
||||
if rec.Code != http.StatusUnauthorized || body["error"] != "signature_required" {
|
||||
t.Fatalf("unsigned delete = %d %v, want 401 signature_required", rec.Code, body)
|
||||
}
|
||||
if !identity.ExistsByUID(s.config.IdentityAccountsDir(), uid) {
|
||||
t.Fatal("a refused delete removed the key anyway")
|
||||
}
|
||||
|
||||
// meta-core must refuse to sign the caller's authorisation for them —
|
||||
// without this the gate is decoration on a node holding every key.
|
||||
ch, _, err := s.challenges.Issue(uid, identity.ActionDelete)
|
||||
if err != nil {
|
||||
t.Fatalf("issue: %v", err)
|
||||
}
|
||||
rec, body = do(t, s, "POST", "/api/identity/sign", map[string]any{
|
||||
"uid": uid,
|
||||
"bytesB64": base64.StdEncoding.EncodeToString([]byte(ch)),
|
||||
})
|
||||
if rec.Code != http.StatusForbidden || body["error"] != "reserved_payload" {
|
||||
t.Fatalf("sign of a challenge = %d %v, want 403 reserved_payload", rec.Code, body)
|
||||
}
|
||||
|
||||
// A challenge minted for reveal must not authorise a delete.
|
||||
rec, chBody := do(t, s, "POST", "/api/identity/challenge", map[string]any{"uid": uid, "action": "reveal"})
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("challenge: %d %s", rec.Code, rec.Body)
|
||||
}
|
||||
proof := signAs(t, priv, chBody["challenge"].(string))
|
||||
req := map[string]any{"uid": uid, "confirm": true}
|
||||
for k, v := range proof {
|
||||
req[k] = v
|
||||
}
|
||||
rec, body = do(t, s, "DELETE", "/api/identity", req)
|
||||
if rec.Code != http.StatusUnauthorized {
|
||||
t.Fatalf("reveal challenge used for delete = %d %v, want 401", rec.Code, body)
|
||||
}
|
||||
|
||||
// The real thing.
|
||||
rec, chBody = do(t, s, "POST", "/api/identity/challenge", map[string]any{"uid": uid, "action": "delete"})
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("challenge: %d %s", rec.Code, rec.Body)
|
||||
}
|
||||
proof = signAs(t, priv, chBody["challenge"].(string))
|
||||
req = map[string]any{"uid": uid, "confirm": true}
|
||||
for k, v := range proof {
|
||||
req[k] = v
|
||||
}
|
||||
rec, body = do(t, s, "DELETE", "/api/identity", req)
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("signed delete = %d %s", rec.Code, rec.Body)
|
||||
}
|
||||
purged := body["purged"].(map[string]any)
|
||||
if purged["records"].(float64) != 3 || purged["cids"].(float64) != 2 {
|
||||
t.Fatalf("purged = %v, want 3 records / 2 cids", purged)
|
||||
}
|
||||
|
||||
// Key gone.
|
||||
if identity.ExistsByUID(s.config.IdentityAccountsDir(), uid) {
|
||||
t.Fatal("the account file survived the delete")
|
||||
}
|
||||
// Data gone — the orphaning this whole change exists to prevent.
|
||||
after, err := s.storage.UDLAllUserStats()
|
||||
if err != nil {
|
||||
t.Fatalf("stats after: %v", err)
|
||||
}
|
||||
if got, present := after[uid]; present {
|
||||
t.Fatalf("deleted account still holds %+v", got)
|
||||
}
|
||||
for _, cid := range []string{"cidA", "cidB"} {
|
||||
uids, err := s.storage.UDLCidUsers(cid)
|
||||
if err != nil {
|
||||
t.Fatalf("cid users: %v", err)
|
||||
}
|
||||
if len(uids) != 0 {
|
||||
t.Fatalf("%s still names %v after the purge", cid, uids)
|
||||
}
|
||||
}
|
||||
|
||||
// The challenge is single-use, so replaying the same proof does nothing.
|
||||
rec, _ = do(t, s, "DELETE", "/api/identity", req)
|
||||
if rec.Code == http.StatusOK {
|
||||
t.Fatal("a replayed proof must not succeed")
|
||||
}
|
||||
}
|
||||
|
||||
func TestE2E_RevealRequiresProof(t *testing.T) {
|
||||
s := newTestServer(t)
|
||||
_, gen := do(t, s, "POST", "/api/identity/generate", nil)
|
||||
uid := gen["uid"].(string)
|
||||
priv := gen["privateKeyHex"].(string)
|
||||
|
||||
rec, body := do(t, s, "POST", "/api/identity/reveal", map[string]any{"uid": uid, "confirm": true})
|
||||
if rec.Code != http.StatusUnauthorized || body["error"] != "signature_required" {
|
||||
t.Fatalf("unsigned reveal = %d %v, want 401 signature_required", rec.Code, body)
|
||||
}
|
||||
|
||||
_, chBody := do(t, s, "POST", "/api/identity/challenge", map[string]any{"uid": uid, "action": "reveal"})
|
||||
proof := signAs(t, priv, chBody["challenge"].(string))
|
||||
req := map[string]any{"uid": uid, "confirm": true}
|
||||
for k, v := range proof {
|
||||
req[k] = v
|
||||
}
|
||||
rec, body = do(t, s, "POST", "/api/identity/reveal", req)
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("signed reveal = %d %s", rec.Code, rec.Body)
|
||||
}
|
||||
if body["privateKeyHex"] != priv {
|
||||
t.Fatal("reveal returned a different key than generate did")
|
||||
}
|
||||
}
|
||||
|
||||
// Another account's key must not authorise this one, even with a challenge
|
||||
// minted for the right uid and action.
|
||||
func TestE2E_AnotherAccountsKeyIsRefused(t *testing.T) {
|
||||
s := newTestServer(t)
|
||||
_, alice := do(t, s, "POST", "/api/identity/generate", nil)
|
||||
_, mallory := do(t, s, "POST", "/api/identity/generate", nil)
|
||||
|
||||
uid := alice["uid"].(string)
|
||||
_, chBody := do(t, s, "POST", "/api/identity/challenge", map[string]any{"uid": uid, "action": "delete"})
|
||||
proof := signAs(t, mallory["privateKeyHex"].(string), chBody["challenge"].(string))
|
||||
req := map[string]any{"uid": uid, "confirm": true}
|
||||
for k, v := range proof {
|
||||
req[k] = v
|
||||
}
|
||||
rec, body := do(t, s, "DELETE", "/api/identity", req)
|
||||
if rec.Code != http.StatusUnauthorized || body["error"] != "bad_signature" {
|
||||
t.Fatalf("mallory deleting alice = %d %v, want 401 bad_signature", rec.Code, body)
|
||||
}
|
||||
if !identity.ExistsByUID(s.config.IdentityAccountsDir(), uid) {
|
||||
t.Fatal("alice's account was deleted by mallory's signature")
|
||||
}
|
||||
}
|
||||
+41
-59
@@ -4,7 +4,6 @@ import (
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"github.com/gorilla/mux"
|
||||
)
|
||||
@@ -39,6 +38,36 @@ type MetadataSearchRequest struct {
|
||||
Property string `json:"property,omitempty"`
|
||||
PropertyValue string `json:"propertyValue,omitempty"`
|
||||
Limit int `json:"limit,omitempty"`
|
||||
|
||||
// Filters is the structured form: AND across entries, OR within an
|
||||
// entry's Values. Evaluated in memory against the search index alongside
|
||||
// the free-text Query, so an id-anchored lookup costs a map probe per
|
||||
// record rather than a Redis round-trip per record.
|
||||
//
|
||||
// Property/PropertyValue is the single-filter legacy shape and folds into
|
||||
// this list (see normalizedFilters) with Exact=false, preserving its
|
||||
// substring semantics byte-for-byte.
|
||||
Filters []MetadataFilter `json:"filters,omitempty"`
|
||||
}
|
||||
|
||||
// MetadataFilter is one structured constraint on a record.
|
||||
//
|
||||
// A record satisfies it when ANY of Values matches, and a search result must
|
||||
// satisfy EVERY filter (OR within, AND across). An entry with no usable value
|
||||
// constrains nothing — same rule the gateway's `genres_filter_matches` uses, so
|
||||
// a caller that lowers an empty filter doesn't accidentally empty the result.
|
||||
type MetadataFilter struct {
|
||||
// Property is the field name, WITHOUT a key-set suffix: `genres`, not
|
||||
// `genres/Action`. Both storage shapes are probed — see recordMatchesFilter.
|
||||
Property string `json:"property"`
|
||||
// Values are OR'd. Compared case-insensitively.
|
||||
Values []string `json:"values,omitempty"`
|
||||
// Exact selects CSV-exact comparison (the semantics meta-gateway's
|
||||
// `any_csv_value_matches` applies at its own Layer-B re-filter) over
|
||||
// substring containment. Callers doing identity lookups want this: with a
|
||||
// result limit, a substring match on `tmdbid=1405` also admits `21405` and
|
||||
// can crowd the real matches out of the page.
|
||||
Exact bool `json:"exact,omitempty"`
|
||||
}
|
||||
|
||||
// MetadataSearchResult is a single search result
|
||||
@@ -415,65 +444,18 @@ func (s *Server) handleSearchMetadata(w http.ResponseWriter, r *http.Request) {
|
||||
return
|
||||
}
|
||||
|
||||
// Free-text query: served from the in-memory search index (search_index.go)
|
||||
// so we match in memory instead of doing a SCAN+MGET Redis round-trip per
|
||||
// record (that was 40s+ over ~72k records). Full metadata is fetched only
|
||||
// for the <=limit matches.
|
||||
if req.Query != "" && req.Property == "" {
|
||||
// Everything else — free text, structured filters, both, or neither
|
||||
// (which returns the first `limit`) — is served from the in-memory search
|
||||
// index (search_index.go). Matching and the returned metadata both come
|
||||
// from memory, so a query costs zero Redis round-trips.
|
||||
//
|
||||
// There is deliberately no second path. The property branch used to fall
|
||||
// through to a `GetAllHashIDs` + per-record `GetMetadataFlat` scan, which
|
||||
// is what made an id-anchored lookup 3.63s over ~20k records (vs 10ms for
|
||||
// the free-text branch on the same corpus) and kept meta-core out of the
|
||||
// gateway's anchored search path entirely. The index already holds every
|
||||
// field of every record, so that scan was re-reading data it had in hand.
|
||||
s.serveIndexedSearch(w, req)
|
||||
return
|
||||
}
|
||||
|
||||
// Property filter (rare) or empty query (returns the first `limit`): the
|
||||
// legacy scan. Empty query only fetches `limit` records; property search is
|
||||
// low-volume and not on the hot search path.
|
||||
allHashIDs, err := s.storage.GetAllHashIDs()
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
var results []MetadataSearchResult
|
||||
total := len(allHashIDs)
|
||||
|
||||
for _, hashID := range allHashIDs {
|
||||
if len(results) >= req.Limit {
|
||||
break
|
||||
}
|
||||
|
||||
metadata, err := s.storage.GetMetadataFlat(hashID)
|
||||
if err != nil || metadata == nil {
|
||||
continue
|
||||
}
|
||||
|
||||
matches := false
|
||||
|
||||
// Property search
|
||||
if req.Property != "" && req.PropertyValue != "" {
|
||||
value, ok := metadata[req.Property]
|
||||
if ok && strings.Contains(strings.ToLower(value), strings.ToLower(req.PropertyValue)) {
|
||||
matches = true
|
||||
}
|
||||
}
|
||||
|
||||
// If no query or property filter, return all
|
||||
if req.Query == "" && req.Property == "" {
|
||||
matches = true
|
||||
}
|
||||
|
||||
if matches {
|
||||
results = append(results, MetadataSearchResult{
|
||||
HashID: hashID,
|
||||
Metadata: metadata,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
writeJSON(w, http.StatusOK, MetadataSearchResponse{
|
||||
Results: results,
|
||||
Count: len(results),
|
||||
Total: total,
|
||||
})
|
||||
}
|
||||
|
||||
// handleBatchUpdate handles POST /api/metadata/batch
|
||||
|
||||
@@ -0,0 +1,211 @@
|
||||
package api
|
||||
|
||||
import "testing"
|
||||
|
||||
// Tests for the structured-filter matcher that lets meta-core answer an
|
||||
// id-anchored query from the in-memory search index (search_index.go).
|
||||
//
|
||||
// Why this matters beyond "filters work": meta-core is a *coarse pre-filter*
|
||||
// for meta-gateway, which re-evaluates every record it emits through its own
|
||||
// `query_eval::record_matches`. So the property that has to hold is that this
|
||||
// matcher stays a SUPERSET of that one — it may admit records the gateway will
|
||||
// drop, but it must never drop a record the gateway would have kept. Every
|
||||
// case below is written from that angle:
|
||||
//
|
||||
// - key-set fields (`genres/Action`) must match a `genres:action` filter, or
|
||||
// card-tier records vanish from a genre-scoped query;
|
||||
// - a missing field fails CLOSED, matching record_matches;
|
||||
// - an empty/unusable filter constrains NOTHING, so a caller that lowers a
|
||||
// blank filter gets its results rather than an empty page.
|
||||
|
||||
func f(property string, exact bool, values ...string) MetadataFilter {
|
||||
return MetadataFilter{Property: property, Values: values, Exact: exact}
|
||||
}
|
||||
|
||||
func TestRecordMatchesFilter(t *testing.T) {
|
||||
fields := map[string]string{
|
||||
"fileType": "video",
|
||||
"videoType": "tv,anime",
|
||||
"tmdbid": "1405",
|
||||
"genres/Action": "true",
|
||||
"genres/Science Fiction": "true",
|
||||
"source/gateway:nyaa.si": "true",
|
||||
"categories/XXX": "true",
|
||||
"retracted": "false",
|
||||
}
|
||||
|
||||
cases := []struct {
|
||||
name string
|
||||
filter MetadataFilter
|
||||
want bool
|
||||
}{
|
||||
// Flat scalar.
|
||||
{"flat exact hit", f("fileType", true, "video"), true},
|
||||
{"flat exact miss", f("fileType", true, "audio"), false},
|
||||
{"flat exact is case-insensitive", f("fileType", true, "VIDEO"), true},
|
||||
|
||||
// csv-set field: each comma piece is compared independently.
|
||||
{"csv member hit", f("videoType", true, "anime"), true},
|
||||
{"csv member miss", f("videoType", true, "movie"), false},
|
||||
|
||||
// Exact vs contains — the reason Exact exists. A substring match on an
|
||||
// identity field admits neighbours and, under a result limit, crowds
|
||||
// the real matches off the page.
|
||||
{"contains admits a superstring", f("tmdbid", false, "140"), true},
|
||||
{"exact rejects a superstring", f("tmdbid", true, "140"), false},
|
||||
{"exact accepts the whole value", f("tmdbid", true, "1405"), true},
|
||||
|
||||
// Key-set shape. The stored member is verbatim (`genres/Action`) while
|
||||
// query values arrive lowercased, so the case-insensitive sweep is what
|
||||
// makes a genre-scoped row work at all.
|
||||
{"key-set exact casing", f("genres", true, "Action"), true},
|
||||
{"key-set lowercased query", f("genres", true, "action"), true},
|
||||
{"key-set with a space", f("genres", true, "science fiction"), true},
|
||||
{"key-set miss", f("genres", true, "horror"), false},
|
||||
{"key-set label containing a colon", f("source", true, "gateway:nyaa.si"), true},
|
||||
|
||||
// OR within one filter's values.
|
||||
{"or hits on the second value", f("fileType", true, "audio", "video"), true},
|
||||
{"or misses on all values", f("fileType", true, "audio", "image"), false},
|
||||
|
||||
// Fail-closed on an absent field — mirrors record_matches, and is what
|
||||
// keeps a filtered query from returning unrelated records.
|
||||
{"absent field fails closed", f("imdbid", true, "tt0903747"), false},
|
||||
|
||||
// A filter that constrains nothing must not empty the result.
|
||||
{"no values constrains nothing", f("fileType", true), true},
|
||||
{"blank value constrains nothing", f("fileType", true, " "), true},
|
||||
{"no property constrains nothing", f("", true, "video"), true},
|
||||
|
||||
// A FLAT field whose value happens to be "false" is still a value, and
|
||||
// an exact filter for it matches. Only the key-set arm treats "false"
|
||||
// as non-membership (pinned in the test below).
|
||||
{"flat field valued false still matches", f("retracted", true, "false"), true},
|
||||
}
|
||||
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
if got := recordMatchesFilter(fields, &tc.filter); got != tc.want {
|
||||
t.Fatalf("recordMatchesFilter(%+v) = %v, want %v", tc.filter, got, tc.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// `retracted = "false"` is a flat field, not a key-set member, so the flat arm
|
||||
// legitimately matches the literal string "false". The key-set arm is what must
|
||||
// treat it as non-membership — pinned separately so the two arms don't get
|
||||
// conflated if either is refactored.
|
||||
func TestFalseyKeySetMemberIsNotMembership(t *testing.T) {
|
||||
fields := map[string]string{"genres/Action": "false", "genres/Drama": "0"}
|
||||
for _, v := range []string{"Action", "Drama"} {
|
||||
filter := f("genres", true, v)
|
||||
if recordMatchesFilter(fields, &filter) {
|
||||
t.Fatalf("genres/%s with a falsey value must not assert membership", v)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestRecordMatchesFiltersIsAndAcrossEntries(t *testing.T) {
|
||||
fields := map[string]string{"fileType": "video", "season": "2"}
|
||||
|
||||
all := []MetadataFilter{f("fileType", true, "video"), f("season", true, "2")}
|
||||
if !recordMatchesFilters(fields, all) {
|
||||
t.Fatal("a record satisfying every filter must match")
|
||||
}
|
||||
|
||||
one := []MetadataFilter{f("fileType", true, "video"), f("season", true, "3")}
|
||||
if recordMatchesFilters(fields, one) {
|
||||
t.Fatal("filters are AND across entries — one miss must reject the record")
|
||||
}
|
||||
|
||||
if !recordMatchesFilters(fields, nil) {
|
||||
t.Fatal("no filters must match everything")
|
||||
}
|
||||
}
|
||||
|
||||
// The legacy Property/PropertyValue pair keeps its substring semantics: it is
|
||||
// the shape existing callers already use, and tightening it silently would
|
||||
// change results for them. New callers pass Filters and opt into Exact.
|
||||
func TestNormalizedFiltersFoldsTheLegacyPair(t *testing.T) {
|
||||
got := normalizedFilters(MetadataSearchRequest{Property: "tmdbid", PropertyValue: "1405"})
|
||||
if len(got) != 1 {
|
||||
t.Fatalf("legacy pair must fold into exactly one filter, got %d", len(got))
|
||||
}
|
||||
if got[0].Property != "tmdbid" || len(got[0].Values) != 1 || got[0].Values[0] != "1405" {
|
||||
t.Fatalf("legacy pair folded incorrectly: %+v", got[0])
|
||||
}
|
||||
if got[0].Exact {
|
||||
t.Fatal("the legacy pair must stay substring (Exact=false) for compatibility")
|
||||
}
|
||||
|
||||
// A half-specified pair is not a filter — it must not become one that
|
||||
// matches nothing and empties the response.
|
||||
if n := len(normalizedFilters(MetadataSearchRequest{Property: "tmdbid"})); n != 0 {
|
||||
t.Fatalf("a property with no value must not fold into a filter, got %d", n)
|
||||
}
|
||||
if n := len(normalizedFilters(MetadataSearchRequest{PropertyValue: "1405"})); n != 0 {
|
||||
t.Fatalf("a value with no property must not fold into a filter, got %d", n)
|
||||
}
|
||||
|
||||
both := normalizedFilters(MetadataSearchRequest{
|
||||
Property: "tmdbid",
|
||||
PropertyValue: "1405",
|
||||
Filters: []MetadataFilter{f("fileType", true, "video")},
|
||||
})
|
||||
if len(both) != 2 {
|
||||
t.Fatalf("the legacy pair must be appended to explicit filters, got %d", len(both))
|
||||
}
|
||||
}
|
||||
|
||||
// The acceptance bar for routing the anchored path through the index: a
|
||||
// full-miss filtered sweep over a realistic corpus must stay in the same class
|
||||
// as the free-text branch (measured 10 ms over 19 622 live records), not the
|
||||
// legacy per-record Redis scan it replaces (measured 3.63 s over the same
|
||||
// corpus). Full-miss is the worst case — the loop cannot short-circuit on the
|
||||
// result limit, so every record is evaluated.
|
||||
//
|
||||
// Run: go test ./internal/api/ -bench BenchmarkFilterSweep -benchtime 10x
|
||||
func BenchmarkFilterSweep(b *testing.B) {
|
||||