viktor/claude-memory-mcp
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

research: benchmark hybrid (lexical+dense+graph) recall vs current FTS
Some checks failed
Build and Push / lint-and-test (push) Has been cancelled
Details
Build and Push / build (push) Has been cancelled
Details
Build and Push / deploy (push) Has been cancelled
Details
Build and Push / notify-failure (push) Has been cancelled
Details

Browse source 
Viktor asked to enhance the memory system with 'semantics' — remember concepts
(not just tokens) linked in a graph — and to prove, by benchmarking against the
current system, that it actually improves recall. A multi-phase research workflow
(18 agents) did landscape research, an adversarially-reviewed integration design,
a stratified eval set over the real 5,452-memory corpus, and a head-to-head
prototype-vs-current benchmark.

Result: hybrid (lexical FTS + dense embeddings, RRF-fused) beats FTS on every
overall metric, driven by a robust paraphrase win (recall@10 +0.350). Recommend
adopting lexical+dense; the concept graph is DEFERRED.

Post-run adversarial review correction (applied to all docs before commit): the
prototype's fusion config structurally barred the graph leg from the ranked top-k,
so the 'graph contributes nothing' ablation was a math artifact, NOT an empirical
result — the graph is UNEVALUATED, not disproven (deferred on cost+uncertainty).
Multi-hop deltas are not statistically significant. Glossary in CONTEXT.md; framing
in ADR-0001-0003; findings in ADR-0004-0006 + docs/research/.

Privacy: the corpus/queries/qrels/results are the user's real memories and stay
gitignored (data/, cache/, results/, build_eval_set.py); only harness code,
aggregate numbers, and synthetic examples are committed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Viktor Barzin 2026-06-25 17:51:53 +00:00
parent 7439540f8f
commit 1cc8a2b378
23 changed files with 3428 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

28
benchmarks/harness/__init__.py Normal file
View file

@ -0,0 +1,28 @@
"""Benchmark harness for claude-memory recall evaluation.
Public API:
from harness import Retriever, load_dataset, run_benchmark, BenchmarkResult
from harness import metrics
A retriever is any object (or callable) implementing:
retrieve(query: str, k: int) -> list[memory_id] # ranked, best first
memory_id matches the `id` field in corpus.jsonl / qrels.jsonl (int).
"""
from .types import Retriever, Query, Memory, Qrels
from .dataset import load_dataset, Dataset
from .runner import run_benchmark, BenchmarkResult, StratumResult
from . import metrics
__all__ = [
"Retriever",
"Query",
"Memory",
"Qrels",
"load_dataset",
"Dataset",
"run_benchmark",
"BenchmarkResult",
"StratumResult",
"metrics",
]