v1.2.0: skip unchanged writes, transactional schema/root counters, snapshot replay for meta:events 0-0 (drops boot republish), collect-once search driver
Build and Push Docker Image / build (push) Successful in 7m38s
Build and Push Docker Image / build (push) Successful in 7m38s
This commit is contained in:
@@ -60,6 +60,9 @@ Everything is one string→string table. The full layout:
|
||||
| `meta-core:schema-version` | `"2"` | schema gate |
|
||||
| `idx:roots/<rid>` | `""` | set of every root (was the Redis SET `file:__index__`) |
|
||||
| `idx:fts/<rid>` | FTS rowid | the record's row in the FTS table |
|
||||
| `idx:count/roots` | integer | number of roots — search's `total`, `/api/stats` records (v1.2.0) |
|
||||
| `idx:schema/<field>\x1f<keyHint>\x1f<prim>\x1f<hint>` | integer | per-field value-type counts behind `/api/schema` (v1.2.0) |
|
||||
| `idx:count/ready` | `"1"` | marks the counters as built (absent → one-time build at boot) |
|
||||
| `dup:<rid>/<path>` | `""` | a duplicate path (was the SET `file:<rid>/duplicates`) |
|
||||
| `udl:rec:<uid>/<cid>/<key>` | JSON `{record,version,ts,value?}` | User Data Layer entry (was a HASH) |
|
||||
| `udl:idx:…<set>\x1f<member>` | `""` | UDL index sets, one key per member |
|
||||
@@ -291,12 +294,20 @@ high-water mark `evhwm:<stream>` (`internal/kv/stream.go`):
|
||||
| `file:events` | `internal/watcher` dispatcher | `add` / `change` / `delete` / `rename` / `reset` events for files on the `/files` volume. |
|
||||
| `meta:events` | the storage engine (`internal/kv`) | `{type: set\|del, key, ts}` for every Put/Delete of a `file:` key, appended **in the same transaction** as the change. |
|
||||
|
||||
`del` is emitted only when the key existed; `set` is emitted even for an
|
||||
unchanged value (Redis semantics, kept). Retention is ~100k entries,
|
||||
trimmed every 10k appends. At boot `MetaPublisher`
|
||||
(`internal/events/meta_publisher.go`) still republishes every field into
|
||||
`meta:events` (meta-fuse rebuilds its VFS only by replaying from `0-0`),
|
||||
batched 5k events per transaction.
|
||||
`set` is emitted only for a **real** change: a Put of the value a key
|
||||
already holds is a no-op — no row write, no index churn, no event (v1.2.0;
|
||||
Redis published `set` even then, and with gateway hits re-persisted on
|
||||
every repeated query most writes were exactly that). `del` is emitted only
|
||||
when the key existed. Retention is ~100k entries, trimmed every 10k appends.
|
||||
|
||||
There is **no boot-time republish** (v1.2.0). A consumer that wants the whole
|
||||
current state asks `/api/events/meta` for `Last-Event-ID: 0` / `0-0` and gets
|
||||
a **snapshot**: one `set` per existing record key, streamed from the kv
|
||||
range, then the live tail from the stream head captured before the snapshot
|
||||
(only the last snapshot event carries an `id:`, so a consumer that drops
|
||||
mid-snapshot gets a fresh one). The old republish wrote every field into a
|
||||
stream that retains only ~100k entries, so a consumer replaying it later
|
||||
never saw most records — and it cost ~6 minutes on every boot.
|
||||
|
||||
The Redis-era path — keyspace notifications drained by `MetaPublisher`
|
||||
through a PSUBSCRIBE / consumer-group hop — is gone (2026-10).
|
||||
@@ -371,7 +382,8 @@ curl -k 'https://metacore-dev.localhost:8083/api/kv/info' # KV stats
|
||||
| UUIDv7 minting | `internal/storage/uuid.go` |
|
||||
| CID reverse index + aliases + canonical-CID derivation | `internal/storage/cid_resolution.go` |
|
||||
| Schema-version gate | `internal/storage/schema_version.go` |
|
||||
| Boot-time `meta:events` republish | `internal/events/meta_publisher.go` |
|
||||
| `meta:events` snapshot for `0`/`0-0` | `internal/api/sse_events.go` (`streamSnapshot`) |
|
||||
| Derived counters (root count, per-field schema) | `internal/storage/client.go` (`onChange`), `internal/schema/indexer.go` |
|
||||
| Watcher → `file:events` dispatcher | `internal/watcher/dispatcher.go` |
|
||||
| SSE event mirror | `internal/api/sse_events.go` |
|
||||
| Field semantics (authoritative registry) | [`/METADATA_KEYS.md`](../../../METADATA_KEYS.md) |
|
||||
|
||||
Reference in New Issue
Block a user