research: benchmark hybrid (lexical+dense+graph) recall vs current FTS

Viktor asked to enhance the memory system with 'semantics' — remember concepts
(not just tokens) linked in a graph — and to prove, by benchmarking against the
current system, that it actually improves recall. A multi-phase research workflow
(18 agents) did landscape research, an adversarially-reviewed integration design,
a stratified eval set over the real 5,452-memory corpus, and a head-to-head
prototype-vs-current benchmark.

Result: hybrid (lexical FTS + dense embeddings, RRF-fused) beats FTS on every
overall metric, driven by a robust paraphrase win (recall@10 +0.350). Recommend
adopting lexical+dense; the concept graph is DEFERRED.

Post-run adversarial review correction (applied to all docs before commit): the
prototype's fusion config structurally barred the graph leg from the ranked top-k,
so the 'graph contributes nothing' ablation was a math artifact, NOT an empirical
result — the graph is UNEVALUATED, not disproven (deferred on cost+uncertainty).
Multi-hop deltas are not statistically significant. Glossary in CONTEXT.md; framing
in ADR-0001-0003; findings in ADR-0004-0006 + docs/research/.

Privacy: the corpus/queries/qrels/results are the user's real memories and stay
gitignored (data/, cache/, results/, build_eval_set.py); only harness code,
aggregate numbers, and synthetic examples are committed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

benchmarks/harness/example_retriever.py

@@ -0,0 +1,59 @@
+"""Worked example: how a later agent plugs a retriever into the harness.
+
+A retriever needs only one method:
+
+    retrieve(self, query: str, k: int) -> list[int]   # ranked memory ids
+
+Optionally it may implement lifecycle hooks the runner will use if present:
+
+    build_index(self, corpus: list[Memory]) -> None    # timed separately
+    index_size_bytes(self) -> int                      # reported
+
+Run this file directly for a smoke test against the local eval set:
+    .venv/bin/python -m harness.example_retriever
+"""
+from __future__ import annotations
+
+from collections.abc import Sequence
+
+from .types import Memory, MemoryId
+
+
+class SubstringRetriever:
+    """Trivial baseline: rank by count of query-word occurrences in content.
+
+    Deliberately weak — exists only to demonstrate the interface. The real
+    lexical baseline is harness.baselines.SqliteFtsRetriever.
+    """
+
+    name = "substring_demo"
+
+    def __init__(self) -> None:
+        self._corpus: list[Memory] = []
+
+    def build_index(self, corpus: Sequence[Memory]) -> None:
+        self._corpus = list(corpus)
+
+    def retrieve(self, query: str, k: int) -> list[MemoryId]:
+        words = [w for w in query.lower().split() if len(w) > 2]
+        scored: list[tuple[int, float]] = []
+        for m in self._corpus:
+            hay = (m.content + " " + m.expanded_keywords + " " + m.tags).lower()
+            score = sum(hay.count(w) for w in words)
+            if score:
+                scored.append((m.id, score + m.importance))  # importance tiebreak
+        scored.sort(key=lambda t: t[1], reverse=True)
+        return [mid for mid, _ in scored[:k]]
+
+
+def _smoke() -> None:
+    from .dataset import load_dataset
+    from .runner import run_benchmark
+
+    ds = load_dataset()
+    res = run_benchmark(SubstringRetriever(), ds)
+    print(res.summary())
+
+
+if __name__ == "__main__":
+    _smoke()