docs: search plan + counters for v1.2.0; tools/search-parity (parity + latency vs a brute-force reference)
Build and Push Docker Image / build (push) Successful in 2m50s
Build and Push Docker Image / build (push) Successful in 2m50s
This commit is contained in:
@@ -169,10 +169,12 @@ Two derived structures are maintained on top of `kv`, in the same
|
||||
transaction as every record write. Both are rebuildable from the kv rows and
|
||||
never authoritative (`internal/storage/search.go`):
|
||||
|
||||
- **`kv_field`** — a partial expression index
|
||||
`ON kv(lower(substr(k,instr(k,'/')+1)), lower(v)) WHERE k>='file:' AND k<'file;' AND instr(k,'/')>0`.
|
||||
- **`kv_field_v`** — a partial expression index
|
||||
`ON kv(lower(substr(k,instr(k,'/')+1)), lower(v), v) WHERE k>='file:' AND k<'file;' AND instr(k,'/')>0`.
|
||||
It answers "which records have field F with a value like V" for **any**
|
||||
field, including ones invented tomorrow — the index is over the key itself.
|
||||
It carries `v` so the substring test is answered from the index alone
|
||||
(covering; 13 ms vs 1.48 s without it, for tmdbid on watch).
|
||||
- **`fts`** — an FTS5 trigram table `fts(rid UNINDEXED, hay)` holding one row
|
||||
per record: its search haystack plus its rid (`idx:fts/<rid>` points at the
|
||||
row). Trigrams make every 3+ character substring indexable.
|
||||
@@ -180,6 +182,10 @@ never authoritative (`internal/storage/search.go`):
|
||||
`POST /api/metadata/search` uses SQL to build a superset of candidates, then
|
||||
verifies each with the unchanged Go matchers (`recordMatchesFilter`,
|
||||
`matchAllTokens`), so semantics are identical to the old in-memory index.
|
||||
The smallest part (by a capped count) is collected once and sorted; every
|
||||
other filter is probed per page as an EXISTS over the candidate's own key
|
||||
range. The plan, its measurements and the rejected shapes are in
|
||||
[`docs/project-architecture/metacore-storage-engine.md`](../../../docs/project-architecture/metacore-storage-engine.md).
|
||||
Results still come in hashID order and are fresh immediately (no refresh
|
||||
lag). This replaced the API's in-memory `searchIndex` (60 s refresh, ~30 KB
|
||||
RAM per record).
|
||||
|
||||
Reference in New Issue
Block a user