1cc8a2b378
Some checks are pending
Viktor asked to enhance the memory system with 'semantics' — remember concepts (not just tokens) linked in a graph — and to prove, by benchmarking against the current system, that it actually improves recall. A multi-phase research workflow (18 agents) did landscape research, an adversarially-reviewed integration design, a stratified eval set over the real 5,452-memory corpus, and a head-to-head prototype-vs-current benchmark. Result: hybrid (lexical FTS + dense embeddings, RRF-fused) beats FTS on every overall metric, driven by a robust paraphrase win (recall@10 +0.350). Recommend adopting lexical+dense; the concept graph is DEFERRED. Post-run adversarial review correction (applied to all docs before commit): the prototype's fusion config structurally barred the graph leg from the ranked top-k, so the 'graph contributes nothing' ablation was a math artifact, NOT an empirical result — the graph is UNEVALUATED, not disproven (deferred on cost+uncertainty). Multi-hop deltas are not statistically significant. Glossary in CONTEXT.md; framing in ADR-0001-0003; findings in ADR-0004-0006 + docs/research/. Privacy: the corpus/queries/qrels/results are the user's real memories and stay gitignored (data/, cache/, results/, build_eval_set.py); only harness code, aggregate numbers, and synthetic examples are committed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
59 lines
1.8 KiB
Python
59 lines
1.8 KiB
Python
"""Worked example: how a later agent plugs a retriever into the harness.
|
|
|
|
A retriever needs only one method:
|
|
|
|
retrieve(self, query: str, k: int) -> list[int] # ranked memory ids
|
|
|
|
Optionally it may implement lifecycle hooks the runner will use if present:
|
|
|
|
build_index(self, corpus: list[Memory]) -> None # timed separately
|
|
index_size_bytes(self) -> int # reported
|
|
|
|
Run this file directly for a smoke test against the local eval set:
|
|
.venv/bin/python -m harness.example_retriever
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
from collections.abc import Sequence
|
|
|
|
from .types import Memory, MemoryId
|
|
|
|
|
|
class SubstringRetriever:
|
|
"""Trivial baseline: rank by count of query-word occurrences in content.
|
|
|
|
Deliberately weak — exists only to demonstrate the interface. The real
|
|
lexical baseline is harness.baselines.SqliteFtsRetriever.
|
|
"""
|
|
|
|
name = "substring_demo"
|
|
|
|
def __init__(self) -> None:
|
|
self._corpus: list[Memory] = []
|
|
|
|
def build_index(self, corpus: Sequence[Memory]) -> None:
|
|
self._corpus = list(corpus)
|
|
|
|
def retrieve(self, query: str, k: int) -> list[MemoryId]:
|
|
words = [w for w in query.lower().split() if len(w) > 2]
|
|
scored: list[tuple[int, float]] = []
|
|
for m in self._corpus:
|
|
hay = (m.content + " " + m.expanded_keywords + " " + m.tags).lower()
|
|
score = sum(hay.count(w) for w in words)
|
|
if score:
|
|
scored.append((m.id, score + m.importance)) # importance tiebreak
|
|
scored.sort(key=lambda t: t[1], reverse=True)
|
|
return [mid for mid, _ in scored[:k]]
|
|
|
|
|
|
def _smoke() -> None:
|
|
from .dataset import load_dataset
|
|
from .runner import run_benchmark
|
|
|
|
ds = load_dataset()
|
|
res = run_benchmark(SubstringRetriever(), ds)
|
|
print(res.summary())
|
|
|
|
|
|
if __name__ == "__main__":
|
|
_smoke()
|