viktor/claude-memory-mcp
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

research: benchmark hybrid (lexical+dense+graph) recall vs current FTS
Some checks failed
Build and Push / lint-and-test (push) Has been cancelled
Details
Build and Push / build (push) Has been cancelled
Details
Build and Push / deploy (push) Has been cancelled
Details
Build and Push / notify-failure (push) Has been cancelled
Details

Browse source 
Viktor asked to enhance the memory system with 'semantics' — remember concepts
(not just tokens) linked in a graph — and to prove, by benchmarking against the
current system, that it actually improves recall. A multi-phase research workflow
(18 agents) did landscape research, an adversarially-reviewed integration design,
a stratified eval set over the real 5,452-memory corpus, and a head-to-head
prototype-vs-current benchmark.

Result: hybrid (lexical FTS + dense embeddings, RRF-fused) beats FTS on every
overall metric, driven by a robust paraphrase win (recall@10 +0.350). Recommend
adopting lexical+dense; the concept graph is DEFERRED.

Post-run adversarial review correction (applied to all docs before commit): the
prototype's fusion config structurally barred the graph leg from the ranked top-k,
so the 'graph contributes nothing' ablation was a math artifact, NOT an empirical
result — the graph is UNEVALUATED, not disproven (deferred on cost+uncertainty).
Multi-hop deltas are not statistically significant. Glossary in CONTEXT.md; framing
in ADR-0001-0003; findings in ADR-0004-0006 + docs/research/.

Privacy: the corpus/queries/qrels/results are the user's real memories and stay
gitignored (data/, cache/, results/, build_eval_set.py); only harness code,
aggregate numbers, and synthetic examples are committed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Viktor Barzin 2026-06-25 17:51:53 +00:00
parent 7439540f8f
commit 1cc8a2b378
23 changed files with 3428 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

25
benchmarks/.gitignore vendored Normal file
View file

@ -0,0 +1,25 @@
# Benchmark dataset is the user's REAL personal memories — NEVER commit.
# Privacy hard-rule (research task brief): corpus/queries/qrels stay LOCAL.
data/
.venv/
cache/
*.npy
*.faiss
*.db
# The eval-set GENERATOR embeds real memory-derived query text + author notes
# (paraphrases of real memories, real ids/notes). Treat it as a data artifact:
# LOCAL-ONLY, never commit. Regenerates data/ from corpus.jsonl. The HARNESS
# itself (harness/*.py, the other scripts) contains NO real content and is safe.
scripts/build_eval_set.py
# Python noise
__pycache__/
*.pyc
.pytest_cache/
*.egg-info/
.ipynb_checkpoints/
# Results from runs may quote real content — keep local by default.
results/
*.results.json