docs: search plan + counters for v1.2.0; tools/search-parity (parity + latency vs a brute-force reference)
Build and Push Docker Image / build (push) Successful in 2m50s

This commit is contained in:
Worph
2026-10-19 21:59:36 +00:00
parent fb4bdad61b
commit 5369dc4b9c
5 changed files with 241 additions and 4 deletions
+8 -2
View File
@@ -169,10 +169,12 @@ Two derived structures are maintained on top of `kv`, in the same
transaction as every record write. Both are rebuildable from the kv rows and
never authoritative (`internal/storage/search.go`):
- **`kv_field`** — a partial expression index
`ON kv(lower(substr(k,instr(k,'/')+1)), lower(v)) WHERE k>='file:' AND k<'file;' AND instr(k,'/')>0`.
- **`kv_field_v`** — a partial expression index
`ON kv(lower(substr(k,instr(k,'/')+1)), lower(v), v) WHERE k>='file:' AND k<'file;' AND instr(k,'/')>0`.
It answers "which records have field F with a value like V" for **any**
field, including ones invented tomorrow — the index is over the key itself.
It carries `v` so the substring test is answered from the index alone
(covering; 13 ms vs 1.48 s without it, for tmdbid on watch).
- **`fts`** — an FTS5 trigram table `fts(rid UNINDEXED, hay)` holding one row
per record: its search haystack plus its rid (`idx:fts/<rid>` points at the
row). Trigrams make every 3+ character substring indexable.
@@ -180,6 +182,10 @@ never authoritative (`internal/storage/search.go`):
`POST /api/metadata/search` uses SQL to build a superset of candidates, then
verifies each with the unchanged Go matchers (`recordMatchesFilter`,
`matchAllTokens`), so semantics are identical to the old in-memory index.
The smallest part (by a capped count) is collected once and sorted; every
other filter is probed per page as an EXISTS over the candidate's own key
range. The plan, its measurements and the rejected shapes are in
[`docs/project-architecture/metacore-storage-engine.md`](../../../docs/project-architecture/metacore-storage-engine.md).
Results still come in hashID order and are fresh immediately (no refresh
lag). This replaced the API's in-memory `searchIndex` (60 s refresh, ~30 KB
RAM per record).