1cc8a2b378
Viktor asked to enhance the memory system with 'semantics' — remember concepts (not just tokens) linked in a graph — and to prove, by benchmarking against the current system, that it actually improves recall. A multi-phase research workflow (18 agents) did landscape research, an adversarially-reviewed integration design, a stratified eval set over the real 5,452-memory corpus, and a head-to-head prototype-vs-current benchmark. Result: hybrid (lexical FTS + dense embeddings, RRF-fused) beats FTS on every overall metric, driven by a robust paraphrase win (recall@10 +0.350). Recommend adopting lexical+dense; the concept graph is DEFERRED. Post-run adversarial review correction (applied to all docs before commit): the prototype's fusion config structurally barred the graph leg from the ranked top-k, so the 'graph contributes nothing' ablation was a math artifact, NOT an empirical result — the graph is UNEVALUATED, not disproven (deferred on cost+uncertainty). Multi-hop deltas are not statistically significant. Glossary in CONTEXT.md; framing in ADR-0001-0003; findings in ADR-0004-0006 + docs/research/. Privacy: the corpus/queries/qrels/results are the user's real memories and stay gitignored (data/, cache/, results/, build_eval_set.py); only harness code, aggregate numbers, and synthetic examples are committed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
28 lines
757 B
Python
28 lines
757 B
Python
"""Benchmark harness for claude-memory recall evaluation.
|
|
|
|
Public API:
|
|
from harness import Retriever, load_dataset, run_benchmark, BenchmarkResult
|
|
from harness import metrics
|
|
|
|
A retriever is any object (or callable) implementing:
|
|
retrieve(query: str, k: int) -> list[memory_id] # ranked, best first
|
|
|
|
memory_id matches the `id` field in corpus.jsonl / qrels.jsonl (int).
|
|
"""
|
|
from .types import Retriever, Query, Memory, Qrels
|
|
from .dataset import load_dataset, Dataset
|
|
from .runner import run_benchmark, BenchmarkResult, StratumResult
|
|
from . import metrics
|
|
|
|
__all__ = [
|
|
"Retriever",
|
|
"Query",
|
|
"Memory",
|
|
"Qrels",
|
|
"load_dataset",
|
|
"Dataset",
|
|
"run_benchmark",
|
|
"BenchmarkResult",
|
|
"StratumResult",
|
|
"metrics",
|
|
]
|