research: benchmark hybrid (lexical+dense+graph) recall vs current FTS

Viktor asked to enhance the memory system with 'semantics' — remember concepts
(not just tokens) linked in a graph — and to prove, by benchmarking against the
current system, that it actually improves recall. A multi-phase research workflow
(18 agents) did landscape research, an adversarially-reviewed integration design,
a stratified eval set over the real 5,452-memory corpus, and a head-to-head
prototype-vs-current benchmark.

Result: hybrid (lexical FTS + dense embeddings, RRF-fused) beats FTS on every
overall metric, driven by a robust paraphrase win (recall@10 +0.350). Recommend
adopting lexical+dense; the concept graph is DEFERRED.

Post-run adversarial review correction (applied to all docs before commit): the
prototype's fusion config structurally barred the graph leg from the ranked top-k,
so the 'graph contributes nothing' ablation was a math artifact, NOT an empirical
result — the graph is UNEVALUATED, not disproven (deferred on cost+uncertainty).
Multi-hop deltas are not statistically significant. Glossary in CONTEXT.md; framing
in ADR-0001-0003; findings in ADR-0004-0006 + docs/research/.

Privacy: the corpus/queries/qrels/results are the user's real memories and stay
gitignored (data/, cache/, results/, build_eval_set.py); only harness code,
aggregate numbers, and synthetic examples are committed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

docs/adr/0004-phase-the-hybrid-lexical-dense-first-graph-gated.md

@@ -0,0 +1,45 @@
+# Phase the hybrid: lexical + dense first, concept graph gated
+
+The [benchmark](../research/benchmark-report.md) shows the hybrid read path beats lexical FTS on
+every overall metric (recall@10 **+0.139**, driven by the statistically-robust paraphrase win recall@10
+**+0.350**), with no recall regression on exact — so [ADR-0001](0001-pursue-hybrid-retrieval-embeddings-and-concept-graph.md)'s
+quality gate **is met by the lexical + dense fusion**. The ablation also showed full-hybrid **identical
+to three decimals** to FTS+dense alone — **but a post-run adversarial review proved this was a structural
+artifact, not an empirical result**: the prototype's fusion barred the graph leg from the fused top-k by
+construction (graph candidates excluded from the FTS∪dense base set; `w_graph=0.35` → max graph RRF
+`0.0057` < base-leg min `0.0091`). So the concept graph was **never validly tested — it is *unevaluated*,
+not disproven.**
+
+We therefore **phase** the hybrid:
+
+- **Phase 1 (adopt now):** lexical FTS ⊕ dense pgvector embeddings, fused with weighted RRF, the
+  importance prior preserved as a post-fusion multiplier. This *is* the measured uplift.
+- **Phase 2 (gated, NOT shipped):** the typed-relation concept graph. It stays designed
+  ([integration design §A.5](../research/integration-design.md)) but disabled — its value is **unproven**
+  and it carries real operational cost (LLM extraction + two extra Postgres tables + traversal).
+
+The graph's prototype was a **zero-LLM keyword-co-occurrence** graph (concepts = tags ∪
+expanded_keywords ∪ regex noun-phrases, edges from co-occurrence), **not** the LLM-extracted
+typed-relation graph the production design specifies. So the null result kills *that cheap
+construction* on *this eval set* — it does not prove an LLM-extracted graph is also null. The graph
+is **gated, not killed**: re-open it only with evidence it helps — e.g. a multi-hop slice whose hops
+are *not* semantically adjacent (where the dense leg can't shortcut), built from real typed-relation
+extraction.
+
+## Why the graph result is inconclusive
+
+The ablation `A≡B` was **guaranteed by the fusion config** (graph candidates excluded from the FTS∪dense
+base set; `w_graph=0.35` caps a graph-only id's RRF below any base-leg id), so it tested nothing about the
+graph — the review even found a relevant memory the graph surfaced that both base legs missed, which
+fusion then discarded. Separately, the multi-hop deltas (recall@10 +0.064) are **not statistically
+significant** (3 of 4 CIs cross zero), so there is no distinguishable multi-hop win to attribute to
+*either* leg. The graph is deferred on **cost + uncertainty**, not on evidence it fails.
+
+## Consequences
+
+- Production ships embeddings + RRF fusion; the graph schema is documented but not migrated.
+- The concept-graph research (Zep/Graphiti split, HippoRAG PPR, EDC canonicalization) is preserved as
+  the phase-2 blueprint, behind the gate.
+- Phasing avoids paying the graph's cost while its value is unproven; the robust **lexical+dense**
+  paraphrase win is what ADR-0001's gate actually surfaced. The graph stays a designed, gated follow-up
+  pending a valid retest (graph candidates in the fused pool / swept weight, real typed-relation extraction).

docs/adr/0005-rrf-default-cc-challenger.md

@@ -0,0 +1,40 @@
+# Weighted RRF as the fusion default; convex combination as a benchmark challenger
+
+The hybrid read path must fuse three retrieval signals on **incomparable scales** — unbounded
+`ts_rank` (BM25), bounded cosine, and (phase 2) an arbitrary graph-proximity score — where the graph
+list is **sparse and often empty**. We adopt **weighted Reciprocal Rank Fusion** as the default
+fusion function: `score(d) = Σ_s w_s/(60 + rank_s(d))`, default `w_lex = w_dense = 1.0`,
+`w_graph = 0.35`, with the existing **importance** value applied as a *post-fusion prior multiplier*
+(`final = fused × (0.7 + 0.3·importance)` for `sort_by="relevance"`) — importance is a prior, **not**
+a fourth fused list.
+
+RRF is the right default because it is **score-scale-free** (no BM25-vs-cosine calibration to
+maintain), treats a missing leg as a clean **0** contribution (no missing-modality bias), is
+near-parameter-free (`k=60` is demonstrably uncritical across [10,100]), and **collapses to today's
+exact lexical ordering** when the dense/graph legs are empty — which is the SQLite-only
+graceful-degrade path ([ADR-0002](0002-api-postgres-first-sqlite-stays-lexical.md)) running the *same*
+code.
+
+## Considered options
+
+- **Convex combination / TM2C2** (Bruch et al., min-max normalization) — the literature consistently
+  shows it **edges RRF on nDCG/recall** when scores are calibratable (Weaviate switched its default to it).
+  It is the **standing challenger**. ⚠️ **Correction:** an earlier draft claimed "the benchmark ran CC
+  against RRF and RRF was chosen on our eval set" — **no CC results were actually produced or persisted in
+  this run.** RRF was adopted on *principled* grounds (scale-free, treats a missing/empty leg as a clean 0,
+  collapses to today's exact lexical ordering for the SQLite-only degrade path), **not** a measured
+  head-to-head. Benchmarking CC vs RRF on our eval set is an open follow-up — do it before locking fusion,
+  and especially if the graph is ever adopted or score distributions shift.
+- **Cross-encoder stage-2 re-rank** (e.g. `bge-reranker-v2-m3` over the fused top ~20–30) — a
+  *separate*, independently-gated stage, not a fusion function. Deferred; ship only if it clears both
+  the quality bar and the hot-path p95 budget on the GPU node.
+
+## Consequences
+
+- Fusion is ~30 lines over three top-N queries; the lexical leg reuses the existing
+  `plainto_tsquery`/`ts_rank` query + OR-broaden fallback verbatim.
+- The exact-stratum nDCG/MRR dip (~0.018/0.025, recall unaffected) is the known RRF cost of blending
+  one perfect hit with near-ties; a small **exact-match rank bonus** is the tunable recovery and a
+  cheap follow-up.
+- `k=60` is borrowed from TREC ad-hoc IR; a quick re-sweep on the eval set is worthwhile but the
+  literature says it is insensitive.

docs/adr/0006-pgvector-hnsw-halfvec-1024d-embeddings.md

@@ -0,0 +1,49 @@
+# Production vector storage: pgvector HNSW + halfvec(1024); 1024-d embeddings (Voyage-3.5 / bge-large)
+
+Phase 1 of the hybrid ([ADR-0004](0004-phase-the-hybrid-lexical-dense-first-graph-gated.md)) needs a
+production home for the dense embeddings. Per [ADR-0002](0002-api-postgres-first-sqlite-stays-lexical.md)
+that is **pgvector on the shared CNPG Postgres**, where claude-memory is already a database tenant — no new
+datastore.
+
+> ⚠️ **Correction (verified against live infra by a design challenger):** an earlier draft justified this
+> with "Immich already runs pgvector on the same cluster." That is **false** — Immich runs its **own**
+> Postgres, not the shared CNPG — so it is NOT evidence the shared cluster has the extension. **pgvector
+> must be explicitly enabled on CNPG** (extension install, and possibly a CNPG operand-image change) via
+> Terraform **before this can land**; do not assume it is already available.
+
+Decisions:
+
+- **Index: HNSW** (`USING hnsw (embedding halfvec_cosine_ops) WITH (m=16, ef_construction=64)`,
+  query knob `hnsw.ef_search` set via `SET LOCAL` inside the recall txn under PgBouncer). Best
+  speed-recall tradeoff, buildable on an empty table. **IVFFlat rejected** — it must be built *after*
+  data exists (empty-table footgun) and has a lower recall ceiling.
+- **Type: `halfvec(1024)`** (fp16) — halves index size at ~no recall loss; 1024-d halfvec = 2048
+  bytes/row → single-digit MB for the whole corpus.
+- **Dimension fixed at 1024**, chosen **once** (changing it later forces a full re-embed + HNSW
+  rebuild). 1024 matches both the production model (Voyage-3.5) and the prototype model
+  (bge-large-en-v1.5), so the column and all fusion code are identical regardless of model.
+- **Model: Voyage-3.5** (1024-d, hosted) for **non-sensitive** memories (highest measured retrieval
+  quality of the hosted options, free tier covers the corpus); **bge-large-en-v1.5** (1024-d, local,
+  MIT) for **sensitive memories and the no-API-key fallback** ([ADR-0003](0003-external-embedding-apis-allowed-for-non-sensitive-memories.md)).
+  `is_sensitive=1` rows are never embedded externally — `embedding=NULL`, lexical-only.
+- **pgvectorscale / StreamingDiskANN deferred** — Rust/pgrx must be compiled into the CNPG operand
+  image, and it only earns its keep above ~1–5M vectors; our corpus is orders of magnitude below that.
+
+## Migration shape
+
+A single **additive** Alembic migration: `ALTER TABLE memories ADD COLUMN embedding halfvec(1024)`
+(NULL for sensitive) + `CREATE INDEX CONCURRENTLY … USING hnsw …`. The existing generated
+`search_vector tsvector` + GIN index (`migrations/001`) are **untouched**, so lexical behaviour and
+the SQLite-only degrade path are unchanged. pgvector enablement on CNPG and any extension/operand
+change land as **Terraform/Terragrunt** in `infra/stacks/…` (GitOps, never kubectl) and trigger a
+rolling restart of the shared cluster — coordinate accordingly.
+
+## Consequences
+
+- The prototype's in-process numpy matrix maps directly to this column; only the substrate changes,
+  not the retrieval math.
+- The prototype measured **bge-large** quality; a cheap follow-up should re-run the dense leg with
+  **Voyage-3.5** on the non-sensitive corpus to confirm the hosted ceiling holds on our content
+  before locking the production default.
+- Production latency/ANN-approximation/filtered-top-k behaviour are unmeasured in the prototype and
+  must be validated post-migration (a stated benchmark limitation).