bug fix vector

docs/api-mediated-access.md

@@ -18,7 +18,7 @@ out to be load-bearing).
 
 meta-core was designed as the sidecar that mediates every metadata read and
 write — a single point that owns the Redis schema, enforces invariants
-(reverse index, canonical_cid reconciliation, schema-version sentinel),
+(reverse index, `cids/` key-set integrity, schema-version sentinel),
 and presents a stable HTTP contract. Other services were meant to talk to
 meta-core, not to Redis.
 
@@ -392,7 +392,7 @@ specifically with the Kubernetes flavour where streams are mediated too.
   coordinating releases across every other service. The UUID-root
   migration is the cautionary tale of what happens without this.
 - **Validation choke point.** Every write passes through one code path
-  that can enforce invariants (reverse index, canonical_cid, type
+  that can enforce invariants (reverse index, `cids/` key-set, type
   constraints, schema version).
 - **Observability.** Audit logs, request metrics, slow-query detection
   all live at one layer.
@@ -697,11 +697,11 @@ contract violation."
 **The fix.** Three layers in `internal/storage/cid_resolution.go`:
 
 1. **`ResolveRoot(hash)`** — at the entry of every `/meta/{hash}`
-   read/write handler, look up `cid:midhash256:<hash>` in the reverse
-   index. If found, use the UUID as the storage root. Falls through to
-   the bare hash for direct-UUID writes / legacy entries.
+   read/write handler, look up `cid:<hash>` in the reverse index (bare
+   CID — no algorithm prefix). If found, use the UUID as the storage root.
+   Falls through to the bare hash for direct-UUID writes / legacy entries.
 2. **Self-pointing alias guard in `addAliasLocked`** — refuse to
-   register `cid:<algo>:<v>` → `<v>`. The dual-root bug's chicken-and-
+   register `cid:<v>` → `<v>`. The dual-root bug's chicken-and-
    egg is: meta-sort writes first, registers self-pointing alias,
    watcher then can't recover. Refusing the self-write keeps the
    watcher's legitimate alias intact.

docs/meta-core-architecture.md

@@ -221,12 +221,13 @@ operators wipe with `pnpm run clean:all` and restart.
 │  Root: opaque UUIDv7 (Crockford Base32; ULID layout, 26 chars)     │
 │  Storage: flat string keys                                         │
 │           file:<uuid>/<property>           → string value          │
-│           file:<uuid>/cids                 → SET of CID tokens     │
-│           file:<uuid>/canonical_cid        → preferred CID         │
+│           file:<uuid>/cids/<bareCid>       → "true" (key-set)      │
 │           file:<uuid>/duplicates           → SET of paths          │
 │                                                                    │
+│  No stored canonical_cid — derived by rank on read.                │
+│                                                                    │
 │  Aliases (reverse index, one Redis STRING each):                   │
-│           cid:<algorithm>:<value>          → <uuid>                │
+│           cid:<bareCid>                    → <uuid>                │
 │                                                                    │
 │  Index:   file:__index__                   → SET of all UUIDs      │
 │                                                                    │

docs/metadata-storage-structure.md

@@ -19,10 +19,13 @@ For the external-access contract (HTTP-only, SSE for events), see
   `01JKR8XW5T4QMV7AHNJ2DEFGHK`.
 - **Properties are flat Redis STRING keys**, one per leaf:
   `file:<uuid>/<property/path>` → string value. *Not* Redis Hashes.
-- **CIDs are reverse-index aliases**: `cid:<algorithm>:<value>` → `<uuid>`
-  (a Redis STRING per known CID).
-- **Per-root sets**: `file:<uuid>/cids` (every CID), `file:<uuid>/duplicates`
+- **CIDs are reverse-index aliases**: `cid:<bareCid>` → `<uuid>`
+  (a Redis STRING per known CID; no algorithm prefix — the multicodec
+  already carries it).
+- **Per-root sets**: `file:<uuid>/cids/<bareCid>` (a *key-set*: one STRING
+  `"true"` per known CID, midhash included), `file:<uuid>/duplicates`
   (alternate paths with the same content), plus `file:__index__` (every UUID).
+  There is **no stored `canonical_cid`** — it is derived by rank on read.
 - **A schema-version sentinel** (`meta-core:schema-version`) refuses to
   boot against legacy data.
 
@@ -59,37 +62,54 @@ file:01JKR8XW5T4QMV7AHNJ2DEFGHK/tmdb/poster                       = "bafkreih...
 
 ### Reverse-index aliases (CIDs)
 
-Every CID known for a file is registered as its own Redis STRING:
+Every CID known for a file is registered as its own Redis STRING, keyed by
+the **bare CID** — no algorithm prefix. A CIDv1 already encodes its hash
+function in the multicodec, so naming it again in the key would be
+redundant (and would let the same digest be indexed under two spellings):
 
 ```
-cid:<algorithm>:<value>   →   <uuid>
+cid:<bareCid>   →   <uuid>
 ```
 
 Examples:
 
 ```
-cid:midhash256:bafkreih5kapbjzq...   →   01JKR8XW5T4QMV7AHNJ2DEFGHK
-cid:sha256:abc...                    →   01JKR8XW5T4QMV7AHNJ2DEFGHK
-cid:ipfs:bafy...                     →   01JKR8XW5T4QMV7AHNJ2DEFGHK
+cid:bafkreih5kapbjzq...   →   01JKR8XW5T4QMV7AHNJ2DEFGHK
+cid:bagacbabaec7v3...     →   01JKR8XW5T4QMV7AHNJ2DEFGHK
 ```
 
 The reverse index makes `GET /api/meta/<cid>` an O(1) lookup for any
-registered CID, independent of which hash algorithm "primary" — there is
-no primary. The auto-alias hooks in `SetProperty` and `SetMetadataFlat`
-(see `internal/storage/cid_resolution.go`) catch any write of a `cid_*`
-or `midhash256` field and register the alias automatically, so plugins
-don't need to know about the reverse index.
+registered CID, independent of which hash algorithm — there is no primary.
+The auto-alias hook (`maybeAddAliasFromFieldLocked` in
+`internal/storage/cid_resolution.go`, called from `SetProperty`,
+`SetMetadataFlat` and `MergeMetadataFlat`) registers the alias for any
+field whose name starts with `cids/`, so writers emit key-set members
+without knowing the reverse index exists.
+
+> ⚠ The hook matches **only** the `cids/` prefix. A legacy `cid_*` or
+> `midhash256` *named* field written to a root is stored verbatim and is
+> **never indexed** — the record becomes unresolvable by that CID. Writers
+> must collapse such fields to `cids/<cid>` members before the write. See
+> `METADATA_KEYS.md` §14.13.
 
 ### Per-root sets
 
 ```
-file:<uuid>/cids              →   Redis SET of "<algorithm>:<value>" tokens
-file:<uuid>/canonical_cid     →   STRING — the highest-ranked CID for advertising externally
+file:<uuid>/cids/<bareCid>    →   STRING "true" — one key per known CID (a key-set, not a Redis SET)
 file:<uuid>/duplicates        →   Redis SET of alternate paths with same content
 ```
 
-`canonical_cid` is reconciled by `reconcileCanonicalCIDLocked` whenever
-the `cids` set changes; ranking is done by `internal/cid/rank.go`.
+`cids` is a **key-set**: one flat `file:<uuid>/cids/<cid>` STRING per member,
+which keeps it conflict-free to merge across peers (a SET would not be).
+The record's own address (the midhash) is just the `0x1000` member — it has
+no separate field.
+
+There is **no stored `canonical_cid`**. The canonical CID is *derived on
+read*: `GetMetadataDocument` (`internal/storage/cid_resolution.go`) ranks the
+key-set's members through `internal/cid/rank.go` (`cid.Better`) and returns
+the winner as the `canonical_cid` field of the JSON response only. Nothing
+reconciles or persists it — an earlier `reconcileCanonicalCIDLocked` did, and
+was removed with the flat `cid_*` keys.
 
 ### Global index
 
@@ -141,14 +161,15 @@ The Go API in `internal/storage/client.go`:
 | Method | Behaviour |
 |---|---|
 | `GetMetadataFlat(uuid)` | `SCAN` for `file:<uuid>/*`, `MGET` the values, return `map[string]string`. |
-| `SetMetadataFlat(uuid, m)` | Pipeline `SET` per field; `SADD file:__index__`; auto-register any `cid_*` aliases. |
+| `SetMetadataFlat(uuid, m)` | Pipeline `SET` per field; `SADD file:__index__`; auto-register an alias for every `cids/<cid>` member. |
 | `MergeMetadataFlat(uuid, m)` | Like `SetMetadataFlat` but doesn't pre-clear; used by `PATCH /meta/{hash}`. |
 | `GetProperty(uuid, field)` | `GET file:<uuid>/<field>`. |
 | `SetProperty(uuid, field, value)` | `SET file:<uuid>/<field>`; auto-register alias if the field is a CID. |
 | `DeleteRoot(uuid)` | `SCAN` + `DEL` for `file:<uuid>/*`, unmap every reverse-index entry. |
 | `Mint(filePath, size, mtimeNano)` | `NewUUID()` + write `filePath/sizeByte/mtimeNano` + add to index. |
-| `AddAlias(uuid, "<algo>:<value>")` | Set `cid:<algo>:<value>` → `<uuid>`, add to `file:<uuid>/cids`, reconcile `canonical_cid`. |
-| `ResolveCID("<algo>:<value>")` | `GET cid:<algo>:<value>` → uuid. |
+| `AddAlias(uuid, "<bareCid>")` | Set `cid:<bareCid>` → `<uuid>` and the key-set member `file:<uuid>/cids/<bareCid>` = `"true"`. Nothing to reconcile — canonical is derived on read. |
+| `ResolveRoot("<bareCid>")` | `GET cid:<bareCid>` → uuid (returns the input unchanged if it is already a UUID root). |
+| `GetMetadataDocument(uuid)` | Flat fields + `cids` hoisted to an array + `canonical_cid` **derived** by `cid.Better` rank + `duplicates`. Backs `GET /api/meta/{cid}`. |
 
 ## Reconstruction (flat → nested)
 
@@ -237,10 +258,14 @@ docker exec metacore-app redis-cli GET 'file:01JKR8XW5T4QMV7AHNJ2DEFGHK/fileinfo
 docker exec metacore-app redis-cli TYPE 'file:01JKR8XW5T4QMV7AHNJ2DEFGHK/filePath'
 # string
 
-# Reverse-index lookups
-docker exec metacore-app redis-cli GET 'cid:midhash256:bafkreih...'      # → uuid
-docker exec metacore-app redis-cli SMEMBERS 'file:01JKR8XW5T4QMV7AHNJ2DEFGHK/cids'
-docker exec metacore-app redis-cli GET 'file:01JKR8XW5T4QMV7AHNJ2DEFGHK/canonical_cid'
+# Reverse-index lookup (bare CID — no algorithm prefix)
+docker exec metacore-app redis-cli GET 'cid:bafkreih...'      # → uuid
+
+# The cids key-set (one STRING per member, NOT a Redis SET — SMEMBERS won't work)
+docker exec metacore-app redis-cli --scan --pattern 'file:01JKR8XW5T4QMV7AHNJ2DEFGHK/cids/*'
+
+# canonical_cid is NOT stored. Derive it via the API instead:
+curl -k https://metacore-dev.localhost:8083/api/meta/<cid> | jq '.canonical_cid, .cids'
 
 # Schema-version sentinel
 docker exec metacore-app redis-cli GET 'meta-core:schema-version'

docs/uuid-rooted-metadata.md

@@ -4,6 +4,25 @@
 
 **Status:** Implemented (2026 alpha clean-wipe gate landed). Kept for design rationale.
 
+> ⚠ **Two parts of this proposal were superseded before shipping.** This doc
+> is the design record; it is *not* the schema reference. For what is actually
+> stored, see `METADATA_KEYS.md` §2/§14.13 and
+> [metadata-storage-structure.md](./metadata-storage-structure.md).
+>
+> 1. **`canonical_cid` is not a stored key.** The proposed reconciler
+>    (`reconcileCanonicalCIDLocked`) was never kept — a stored scalar isn't
+>    conflict-free to merge across peers. The canonical CID is **derived on
+>    read** by ranking the `cids` members (`internal/cid/rank.go` →
+>    `cid.Better`), and appears only in the `GET /api/meta/{cid}` response.
+> 2. **`cids` is a key-set, not a Redis SET, and holds bare CIDs, not
+>    `<algo>:<value>` tokens.** One flat STRING per member:
+>    `file:<uuid>/cids/<bareCid>` = `"true"`. The reverse index is likewise
+>    `cid:<bareCid>` → uuid. A CIDv1's multicodec already names its hash
+>    function, so the `<algo>:` prefix was redundant.
+>
+> Text below that references either shape describes the *proposal*, not the
+> code. The rank ordering and everything else did ship as written.
+
 ## Motivation
 
 Today, every metadata entry in Redis is keyed by a content hash, specifically
@@ -79,19 +98,21 @@ today — anything that wrote `file:<midhash>/fileinfo/duration` now writes
 file:01JKR8XW5T4QMV7AHNJ2DEFGHK/filePath          = /watch/Inception.mkv
 file:01JKR8XW5T4QMV7AHNJ2DEFGHK/size              = 4500000000
 file:01JKR8XW5T4QMV7AHNJ2DEFGHK/mtime             = 1729872000000000000
-file:01JKR8XW5T4QMV7AHNJ2DEFGHK/cids              = SET{midhash256:abc, sha256:def, ipfs:bafy...}
+file:01JKR8XW5T4QMV7AHNJ2DEFGHK/cids/bagacbabaec…  = "true"    ← midhash (record address)
+file:01JKR8XW5T4QMV7AHNJ2DEFGHK/cids/bafybeigd…    = "true"    ← sha2-256
 file:01JKR8XW5T4QMV7AHNJ2DEFGHK/duplicates        = SET{/watch/dup1.mkv, /watch/dup2.mkv}
-file:01JKR8XW5T4QMV7AHNJ2DEFGHK/canonical_cid     = ipfs:bafy...
 file:01JKR8XW5T4QMV7AHNJ2DEFGHK/title             = "Inception"
 file:01JKR8XW5T4QMV7AHNJ2DEFGHK/fileinfo/duration = 7200.5
 file:01JKR8XW5T4QMV7AHNJ2DEFGHK/tmdb/poster       = bafy...
 file:01JKR8XW5T4QMV7AHNJ2DEFGHK/subtitle/eng      = ...
 
-cid:midhash256:abc  → 01JKR8XW5T4QMV7AHNJ2DEFGHK
-cid:sha256:def      → 01JKR8XW5T4QMV7AHNJ2DEFGHK
-cid:ipfs:bafy...    → 01JKR8XW5T4QMV7AHNJ2DEFGHK
+cid:bagacbabaec…  → 01JKR8XW5T4QMV7AHNJ2DEFGHK
+cid:bafybeigd…    → 01JKR8XW5T4QMV7AHNJ2DEFGHK
 ```
 
+*(As shipped — the proposal originally wrote `cids` as a Redis SET of
+`<algo>:<value>` tokens plus a `canonical_cid` scalar. See the Status note.)*
+
 ### UUID format: UUIDv7
 
 UUIDv7 chosen over UUIDv4 or 64-bit random:
@@ -111,26 +132,33 @@ The metadata schema is open — any writer can put any key under
 `file:<uuid>/`. This proposal does not change that. It adds three keys whose
 sole writer is meta-core itself:
 
-| Key | Type | Purpose |
+| Key | Type (as shipped) | Purpose |
 |-----|------|---------|
-| `cids` | Redis SET | All CIDs known for this file. Mirror of the `cid:*` reverse index. Source of truth for delete-time GC. |
+| `cids/<bareCid>` | STRING `"true"` — a *key-set*, one key per member | All CIDs known for this file, midhash included. Mirror of the `cid:*` reverse index. Source of truth for delete-time GC. **Proposed as a Redis SET of `<algo>:<value>` tokens; shipped as a key-set of bare CIDs** — a key-set merges conflict-free across peers and the multicodec already names the algorithm. |
 | `duplicates` | Redis SET | Additional file paths whose content matches this entry. One UUID = one set of metadata; multiple physical files may share that content. Consumed by meta-dup. |
-| `canonical_cid` | STRING | The CID this entry should be presented as externally (meta-share announce, UI display). Selected by the rank function (below). |
+| ~~`canonical_cid`~~ | *(not stored)* | **Dropped.** Was to hold the externally-presented CID. Now derived on read by ranking the `cids` members; surfaced only on `GET /api/meta/{cid}`. |
 
 Existing meta-core-owned fields (`filePath`, `size`, `mtime`, `midhash256`)
 keep their names and writers. Plugin-written fields (`tmdb/*`, `fileinfo/*`,
 `subtitle/<lang>`, …) are unaffected.
 
-Redis SET is a departure from the all-strings convention in the current
+~~Redis SET is a departure from the all-strings convention in the current
 storage doc, but `file:__index__` is already a SET — so the precedent exists
-for using sets where they're the right structure.
+for using sets where they're the right structure.~~ **Reversed for `cids`.**
+The all-strings convention won: `cids` ships as flat `cids/<cid>` STRING keys
+so that two peers merging the same record can never disagree about set
+membership. `duplicates` remained a Redis SET (it is leader-local and never
+merged across peers).
 
 ### Canonical CID selection
 
-`canonical_cid` is a label, not a key. Updating it costs one write — no
-rename, no race, no broken external references.
+The canonical CID is a label, not a key — so nothing external breaks when it
+changes. **As shipped it is not written at all**: it is computed on every read
+from the `cids` members, which makes the "update" cost zero and removes the
+reconciler entirely.
 
-Selection rule, highest-wins:
+Selection rule, highest-wins (this part shipped as designed —
+`internal/cid/rank.go`):
 
 1. IPFS CID (any variant)
 2. SHA-256 / SHA-3 family
@@ -142,8 +170,10 @@ midhash256 and sha256 are both 256-bit, but only sha256/IPFS CIDs are
 externally meaningful for content addressing. btih ranks above midhash because
 it enables swarm interop even if it's weaker as a digest.
 
-Reconciler runs on the leader, recomputes `canonical_cid` whenever
-`AddAlias` registers a new CID for an entry.
+~~Reconciler runs on the leader, recomputes `canonical_cid` whenever
+`AddAlias` registers a new CID for an entry.~~ **Not shipped** — there is no
+reconciler. `GetMetadataDocument` folds the members through `cid.Better` on
+each read.
 
 ## Internal operations
 
@@ -157,8 +187,8 @@ CID-based API.
 Mint(filePath string, size int64, mtime int64) (uuid string, err error)
 
 // AddAlias registers a CID as resolving to this UUID. Updates both the
-// cid:<cid> reverse index and the file:<uuid>/cids set atomically.
-// Triggers canonical_cid reconciliation.
+// cid:<cid> reverse index and the file:<uuid>/cids/<cid> key-set member.
+// (As shipped: no reconciliation step — canonical is derived on read.)
 AddAlias(uuid string, cid string) error
 
 // GetByCID resolves any known CID to its UUID. Returns ("", nil) if unknown.
@@ -199,7 +229,8 @@ DeleteRoot(uuid string) error
    - Plugin reports back: "for CID midhash256:abc..., sha256 is sha256:def...".
    - meta-core resolves midhash256:abc → uuid_Y via GetByCID.
    - meta-core calls AddAlias(uuid_Y, "sha256:def...").
-   - Reconciler updates canonical_cid if sha256 outranks midhash256.
+   - Nothing else to do: the next read of this entry derives a canonical CID
+     of sha2-256, because it outranks the midhash.
 ```
 
 ### Lifecycle: file deletion
@@ -250,13 +281,20 @@ so the impact is contained but real.
   client.
 - **meta-dup**: gains a cleaner data source — duplicates are now explicit in
   the `duplicates` set, not implicit in colliding writes.
-- **meta-share**: announces `canonical_cid` rather than midhash. Federated
-  search responses include all CIDs from the `cids` set so any peer querying
-  by any digest gets a hit.
+- **meta-search / meta-share**: announce the *derived* canonical CID rather
+  than the midhash — each mirrors the Go rank in `ingest::canonical_cid_from_metadata`
+  rather than reading a stored field. Federated search responses include all
+  CIDs from the `cids` key-set so any peer querying by any digest gets a hit.
+  (Written before the 2026-06 discovery/transport split; discovery is now
+  meta-search's job.)
 - **Plugins**: no change. They speak CIDs only.
 
 ## Migration
 
+> **Historical — this sweep was never run.** The alpha shipped behind a
+> clean-wipe gate (Redis was flushed and the library re-ingested), so no
+> in-place migration code exists. Kept for the record.
+
 One-time sweep, idempotent, runs on first boot after upgrade:
 
 ```
@@ -285,7 +323,8 @@ roots and resumes by detecting which step completed (presence of
 | UUID visible externally? | No | Pure meta-core internal. Clients address by CID only. |
 | Merge semantics | One UUID per content; additional paths into `duplicates` | One file (logical) = one metadata set. Multiple physical paths supported via `duplicates`. |
 | Canonical CID rank | IPFS > sha-family > btih > midhash | Utility-weighted, not strength-weighted. |
-| `cids` and `duplicates` field type | Redis SET | Departure from "all-strings" convention but consistent with the existing `file:__index__` precedent. |
+| ~~`cids` and `duplicates` field type~~ | ~~Redis SET~~ → **`cids` is a key-set of bare CIDs; `duplicates` stayed a SET** | Reversed before shipping. Flat `cids/<cid>` STRING keys merge conflict-free across peers; a SET does not. See §14.13 of `METADATA_KEYS.md`. |
+| ~~Stored `canonical_cid`~~ | **Dropped — derived on read** | A stored scalar is not conflict-free to merge. `GetMetadataDocument` ranks the `cids` members per request. |
 | Reverse-index format | Flat `cid:<cid>` STRING keys | Simple, scan-friendly, no hot-key contention. |
 
 ## Open questions
@@ -297,9 +336,10 @@ roots and resumes by detecting which step completed (presence of
   hashing reveals the collision. Worth doing eventually; not required for
   this iteration.
 
-- **meta-share announce strategy.** Announce only `canonical_cid`, or every
-  entry in `cids`? Maximum discoverability vs. DHT footprint. Decide as part
-  of the meta-share work, not here.
+- ~~**meta-share announce strategy.**~~ **Settled.** The record carries *every*
+  member of `cids` on the wire (`Record.cids`) and addresses itself by the
+  single derived canonical (`Record.cid`), so a peer querying by any digest
+  gets a hit without the DHT footprint of announcing each one separately.
 
 - **IPFS gateway endpoint.** Should meta-core expose `/ipfs/<cid>` as a
   content-streaming endpoint for external IPFS clients? Adjacent to this