1cc8a2b378
Viktor asked to enhance the memory system with 'semantics' — remember concepts (not just tokens) linked in a graph — and to prove, by benchmarking against the current system, that it actually improves recall. A multi-phase research workflow (18 agents) did landscape research, an adversarially-reviewed integration design, a stratified eval set over the real 5,452-memory corpus, and a head-to-head prototype-vs-current benchmark. Result: hybrid (lexical FTS + dense embeddings, RRF-fused) beats FTS on every overall metric, driven by a robust paraphrase win (recall@10 +0.350). Recommend adopting lexical+dense; the concept graph is DEFERRED. Post-run adversarial review correction (applied to all docs before commit): the prototype's fusion config structurally barred the graph leg from the ranked top-k, so the 'graph contributes nothing' ablation was a math artifact, NOT an empirical result — the graph is UNEVALUATED, not disproven (deferred on cost+uncertainty). Multi-hop deltas are not statistically significant. Glossary in CONTEXT.md; framing in ADR-0001-0003; findings in ADR-0004-0006 + docs/research/. Privacy: the corpus/queries/qrels/results are the user's real memories and stay gitignored (data/, cache/, results/, build_eval_set.py); only harness code, aggregate numbers, and synthetic examples are committed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
25 lines
709 B
Text
25 lines
709 B
Text
# Benchmark dataset is the user's REAL personal memories — NEVER commit.
|
|
# Privacy hard-rule (research task brief): corpus/queries/qrels stay LOCAL.
|
|
data/
|
|
.venv/
|
|
cache/
|
|
*.npy
|
|
*.faiss
|
|
*.db
|
|
|
|
# The eval-set GENERATOR embeds real memory-derived query text + author notes
|
|
# (paraphrases of real memories, real ids/notes). Treat it as a data artifact:
|
|
# LOCAL-ONLY, never commit. Regenerates data/ from corpus.jsonl. The HARNESS
|
|
# itself (harness/*.py, the other scripts) contains NO real content and is safe.
|
|
scripts/build_eval_set.py
|
|
|
|
# Python noise
|
|
__pycache__/
|
|
*.pyc
|
|
.pytest_cache/
|
|
*.egg-info/
|
|
.ipynb_checkpoints/
|
|
|
|
# Results from runs may quote real content — keep local by default.
|
|
results/
|
|
*.results.json
|