v1.2.0: skip unchanged writes, transactional schema/root counters, snapshot replay for meta:events 0-0 (drops boot republish), collect-once search driver
Build and Push Docker Image / build (push) Successful in 7m38s

This commit is contained in:
Worph
2026-10-19 21:41:55 +00:00
parent dc5cbdfb9b
commit fb4bdad61b
21 changed files with 897 additions and 540 deletions
+19 -7
View File
@@ -60,6 +60,9 @@ Everything is one string→string table. The full layout:
| `meta-core:schema-version` | `"2"` | schema gate |
| `idx:roots/<rid>` | `""` | set of every root (was the Redis SET `file:__index__`) |
| `idx:fts/<rid>` | FTS rowid | the record's row in the FTS table |
| `idx:count/roots` | integer | number of roots — search's `total`, `/api/stats` records (v1.2.0) |
| `idx:schema/<field>\x1f<keyHint>\x1f<prim>\x1f<hint>` | integer | per-field value-type counts behind `/api/schema` (v1.2.0) |
| `idx:count/ready` | `"1"` | marks the counters as built (absent → one-time build at boot) |
| `dup:<rid>/<path>` | `""` | a duplicate path (was the SET `file:<rid>/duplicates`) |
| `udl:rec:<uid>/<cid>/<key>` | JSON `{record,version,ts,value?}` | User Data Layer entry (was a HASH) |
| `udl:idx:…<set>\x1f<member>` | `""` | UDL index sets, one key per member |
@@ -291,12 +294,20 @@ high-water mark `evhwm:<stream>` (`internal/kv/stream.go`):
| `file:events` | `internal/watcher` dispatcher | `add` / `change` / `delete` / `rename` / `reset` events for files on the `/files` volume. |
| `meta:events` | the storage engine (`internal/kv`) | `{type: set\|del, key, ts}` for every Put/Delete of a `file:` key, appended **in the same transaction** as the change. |
`del` is emitted only when the key existed; `set` is emitted even for an
unchanged value (Redis semantics, kept). Retention is ~100k entries,
trimmed every 10k appends. At boot `MetaPublisher`
(`internal/events/meta_publisher.go`) still republishes every field into
`meta:events` (meta-fuse rebuilds its VFS only by replaying from `0-0`),
batched 5k events per transaction.
`set` is emitted only for a **real** change: a Put of the value a key
already holds is a no-op — no row write, no index churn, no event (v1.2.0;
Redis published `set` even then, and with gateway hits re-persisted on
every repeated query most writes were exactly that). `del` is emitted only
when the key existed. Retention is ~100k entries, trimmed every 10k appends.
There is **no boot-time republish** (v1.2.0). A consumer that wants the whole
current state asks `/api/events/meta` for `Last-Event-ID: 0` / `0-0` and gets
a **snapshot**: one `set` per existing record key, streamed from the kv
range, then the live tail from the stream head captured before the snapshot
(only the last snapshot event carries an `id:`, so a consumer that drops
mid-snapshot gets a fresh one). The old republish wrote every field into a
stream that retains only ~100k entries, so a consumer replaying it later
never saw most records — and it cost ~6 minutes on every boot.
The Redis-era path — keyspace notifications drained by `MetaPublisher`
through a PSUBSCRIBE / consumer-group hop — is gone (2026-10).
@@ -371,7 +382,8 @@ curl -k 'https://metacore-dev.localhost:8083/api/kv/info' # KV stats
| UUIDv7 minting | `internal/storage/uuid.go` |
| CID reverse index + aliases + canonical-CID derivation | `internal/storage/cid_resolution.go` |
| Schema-version gate | `internal/storage/schema_version.go` |
| Boot-time `meta:events` republish | `internal/events/meta_publisher.go` |
| `meta:events` snapshot for `0`/`0-0` | `internal/api/sse_events.go` (`streamSnapshot`) |
| Derived counters (root count, per-field schema) | `internal/storage/client.go` (`onChange`), `internal/schema/indexer.go` |
| Watcher → `file:events` dispatcher | `internal/watcher/dispatcher.go` |
| SSE event mirror | `internal/api/sse_events.go` |
| Field semantics (authoritative registry) | [`/METADATA_KEYS.md`](../../../METADATA_KEYS.md) |