research: benchmark hybrid (lexical+dense+graph) recall vs current FTS
Viktor asked to enhance the memory system with 'semantics' — remember concepts (not just tokens) linked in a graph — and to prove, by benchmarking against the current system, that it actually improves recall. A multi-phase research workflow (18 agents) did landscape research, an adversarially-reviewed integration design, a stratified eval set over the real 5,452-memory corpus, and a head-to-head prototype-vs-current benchmark. Result: hybrid (lexical FTS + dense embeddings, RRF-fused) beats FTS on every overall metric, driven by a robust paraphrase win (recall@10 +0.350). Recommend adopting lexical+dense; the concept graph is DEFERRED. Post-run adversarial review correction (applied to all docs before commit): the prototype's fusion config structurally barred the graph leg from the ranked top-k, so the 'graph contributes nothing' ablation was a math artifact, NOT an empirical result — the graph is UNEVALUATED, not disproven (deferred on cost+uncertainty). Multi-hop deltas are not statistically significant. Glossary in CONTEXT.md; framing in ADR-0001-0003; findings in ADR-0004-0006 + docs/research/. Privacy: the corpus/queries/qrels/results are the user's real memories and stay gitignored (data/, cache/, results/, build_eval_set.py); only harness code, aggregate numbers, and synthetic examples are committed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
parent
7439540f8f
commit
1cc8a2b378
23 changed files with 3428 additions and 0 deletions
10
benchmarks/retrievers/__init__.py
Normal file
10
benchmarks/retrievers/__init__.py
Normal file
|
|
@ -0,0 +1,10 @@
|
|||
"""Pluggable retrievers for the claude-memory recall benchmark.
|
||||
|
||||
Each retriever implements the harness `retrieve(query, k) -> list[int]` contract
|
||||
(see ``harness/types.py`` :: ``Retriever``) and, optionally, the ``build_index`` /
|
||||
``index_size_bytes`` lifecycle hooks the runner duck-types.
|
||||
|
||||
``fts.FtsRetriever`` is the LEXICAL BASELINE — the product's current local-store
|
||||
recall (SQLite FTS5/BM25). It is the "current system" any hybrid retriever must
|
||||
beat on recall@k / nDCG@10 / MRR (ADR-0001).
|
||||
"""
|
||||
Loading…
Add table
Add a link
Reference in a new issue