1cc8a2b378
Viktor asked to enhance the memory system with 'semantics' — remember concepts (not just tokens) linked in a graph — and to prove, by benchmarking against the current system, that it actually improves recall. A multi-phase research workflow (18 agents) did landscape research, an adversarially-reviewed integration design, a stratified eval set over the real 5,452-memory corpus, and a head-to-head prototype-vs-current benchmark. Result: hybrid (lexical FTS + dense embeddings, RRF-fused) beats FTS on every overall metric, driven by a robust paraphrase win (recall@10 +0.350). Recommend adopting lexical+dense; the concept graph is DEFERRED. Post-run adversarial review correction (applied to all docs before commit): the prototype's fusion config structurally barred the graph leg from the ranked top-k, so the 'graph contributes nothing' ablation was a math artifact, NOT an empirical result — the graph is UNEVALUATED, not disproven (deferred on cost+uncertainty). Multi-hop deltas are not statistically significant. Glossary in CONTEXT.md; framing in ADR-0001-0003; findings in ADR-0004-0006 + docs/research/. Privacy: the corpus/queries/qrels/results are the user's real memories and stay gitignored (data/, cache/, results/, build_eval_set.py); only harness code, aggregate numbers, and synthetic examples are committed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2.7 KiB
Weighted RRF as the fusion default; convex combination as a benchmark challenger
The hybrid read path must fuse three retrieval signals on incomparable scales — unbounded
ts_rank (BM25), bounded cosine, and (phase 2) an arbitrary graph-proximity score — where the graph
list is sparse and often empty. We adopt weighted Reciprocal Rank Fusion as the default
fusion function: score(d) = Σ_s w_s/(60 + rank_s(d)), default w_lex = w_dense = 1.0,
w_graph = 0.35, with the existing importance value applied as a post-fusion prior multiplier
(final = fused × (0.7 + 0.3·importance) for sort_by="relevance") — importance is a prior, not
a fourth fused list.
RRF is the right default because it is score-scale-free (no BM25-vs-cosine calibration to
maintain), treats a missing leg as a clean 0 contribution (no missing-modality bias), is
near-parameter-free (k=60 is demonstrably uncritical across [10,100]), and collapses to today's
exact lexical ordering when the dense/graph legs are empty — which is the SQLite-only
graceful-degrade path (ADR-0002) running the same
code.
Considered options
- Convex combination / TM2C2 (Bruch et al., min-max normalization) — the literature consistently shows it edges RRF on nDCG/recall when scores are calibratable (Weaviate switched its default to it). It is the standing challenger. ⚠️ Correction: an earlier draft claimed "the benchmark ran CC against RRF and RRF was chosen on our eval set" — no CC results were actually produced or persisted in this run. RRF was adopted on principled grounds (scale-free, treats a missing/empty leg as a clean 0, collapses to today's exact lexical ordering for the SQLite-only degrade path), not a measured head-to-head. Benchmarking CC vs RRF on our eval set is an open follow-up — do it before locking fusion, and especially if the graph is ever adopted or score distributions shift.
- Cross-encoder stage-2 re-rank (e.g.
bge-reranker-v2-m3over the fused top ~20–30) — a separate, independently-gated stage, not a fusion function. Deferred; ship only if it clears both the quality bar and the hot-path p95 budget on the GPU node.
Consequences
- Fusion is ~30 lines over three top-N queries; the lexical leg reuses the existing
plainto_tsquery/ts_rankquery + OR-broaden fallback verbatim. - The exact-stratum nDCG/MRR dip (~0.018/0.025, recall unaffected) is the known RRF cost of blending one perfect hit with near-ties; a small exact-match rank bonus is the tunable recovery and a cheap follow-up.
k=60is borrowed from TREC ad-hoc IR; a quick re-sweep on the eval set is worthwhile but the literature says it is insensitive.