research: benchmark hybrid (lexical+dense+graph) recall vs current FTS

Viktor asked to enhance the memory system with 'semantics' — remember concepts
(not just tokens) linked in a graph — and to prove, by benchmarking against the
current system, that it actually improves recall. A multi-phase research workflow
(18 agents) did landscape research, an adversarially-reviewed integration design,
a stratified eval set over the real 5,452-memory corpus, and a head-to-head
prototype-vs-current benchmark.

Result: hybrid (lexical FTS + dense embeddings, RRF-fused) beats FTS on every
overall metric, driven by a robust paraphrase win (recall@10 +0.350). Recommend
adopting lexical+dense; the concept graph is DEFERRED.

Post-run adversarial review correction (applied to all docs before commit): the
prototype's fusion config structurally barred the graph leg from the ranked top-k,
so the 'graph contributes nothing' ablation was a math artifact, NOT an empirical
result — the graph is UNEVALUATED, not disproven (deferred on cost+uncertainty).
Multi-hop deltas are not statistically significant. Glossary in CONTEXT.md; framing
in ADR-0001-0003; findings in ADR-0004-0006 + docs/research/.

Privacy: the corpus/queries/qrels/results are the user's real memories and stay
gitignored (data/, cache/, results/, build_eval_set.py); only harness code,
aggregate numbers, and synthetic examples are committed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

benchmarks/harness/metrics.py

@@ -0,0 +1,100 @@
+"""Retrieval metrics with BINARY relevance.
+
+Conventions
+-----------
+- `ranked`: list of memory ids, best-first, as returned by a retriever.
+- `relevant`: set of relevant memory ids for the query (from qrels).
+- All functions are pure and operate on a single query; the runner aggregates
+  (macro-average over queries).
+
+Definitions
+-----------
+recall@k   = |relevant ∩ ranked[:k]| / |relevant|
+             (fraction of all relevant items retrieved within the top k)
+MRR        = 1 / rank_of_first_relevant  (0 if none retrieved at all)
+nDCG@k     = DCG@k / IDCG@k  with binary gains (gain=1 for relevant)
+             DCG@k = sum over i in [1..k] of rel_i / log2(i + 1)
+             IDCG@k is the DCG of the ideal ranking (all relevant first),
+             capped at min(|relevant|, k) ones.
+
+Notes
+-----
+- nDCG uses the standard log2(rank+1) discount (Järvelin & Kekäläinen 2002);
+  with binary gains this is the common IR convention also used by BEIR/pytrec_eval.
+- MRR is reported as the reciprocal rank of the FIRST relevant hit, which for a
+  single query equals the per-query reciprocal-rank that the runner averages.
+- Duplicate ids in `ranked` are de-duplicated keeping first occurrence, so a
+  retriever cannot inflate recall by repeating an id.
+"""
+from __future__ import annotations
+
+import math
+from collections.abc import Iterable, Sequence
+
+MemoryId = int
+
+
+def _dedup_keep_order(ranked: Sequence[MemoryId]) -> list[MemoryId]:
+    seen: set[MemoryId] = set()
+    out: list[MemoryId] = []
+    for x in ranked:
+        if x not in seen:
+            seen.add(x)
+            out.append(x)
+    return out
+
+
+def recall_at_k(ranked: Sequence[MemoryId], relevant: Iterable[MemoryId], k: int) -> float:
+    rel = set(relevant)
+    if not rel:
+        # Undefined; treat as 0 contribution. Runner should never pass empty.
+        return 0.0
+    top = _dedup_keep_order(ranked)[:k]
+    hits = sum(1 for x in top if x in rel)
+    return hits / len(rel)
+
+
+def reciprocal_rank(ranked: Sequence[MemoryId], relevant: Iterable[MemoryId]) -> float:
+    rel = set(relevant)
+    if not rel:
+        return 0.0
+    for i, x in enumerate(_dedup_keep_order(ranked), start=1):
+        if x in rel:
+            return 1.0 / i
+    return 0.0
+
+
+def dcg_at_k(ranked: Sequence[MemoryId], relevant: Iterable[MemoryId], k: int) -> float:
+    rel = set(relevant)
+    top = _dedup_keep_order(ranked)[:k]
+    dcg = 0.0
+    for i, x in enumerate(top, start=1):
+        if x in rel:
+            dcg += 1.0 / math.log2(i + 1)
+    return dcg
+
+
+def ndcg_at_k(ranked: Sequence[MemoryId], relevant: Iterable[MemoryId], k: int) -> float:
+    rel = set(relevant)
+    if not rel:
+        return 0.0
+    dcg = dcg_at_k(ranked, rel, k)
+    ideal_hits = min(len(rel), k)
+    idcg = sum(1.0 / math.log2(i + 1) for i in range(1, ideal_hits + 1))
+    if idcg == 0.0:
+        return 0.0
+    return dcg / idcg
+
+
+def per_query_metrics(ranked: Sequence[MemoryId], relevant: Iterable[MemoryId]) -> dict[str, float]:
+    """All headline metrics for one query."""
+    rel = set(relevant)
+    return {
+        "recall@5": recall_at_k(ranked, rel, 5),
+        "recall@10": recall_at_k(ranked, rel, 10),
+        "ndcg@10": ndcg_at_k(ranked, rel, 10),
+        "mrr": reciprocal_rank(ranked, rel),
+    }
+
+
+METRIC_NAMES = ("recall@5", "recall@10", "ndcg@10", "mrr")