Cognitive Benchmarks

HEBBS’s four-strategy recall system provides significant precision improvements over similarity-only vector search. This page documents the cognitive benchmark results that validate the multi-path approach.

Benchmark Design

The cognitive benchmark suite evaluates recall quality using curated datasets where the “correct” memories for each query are known (ground truth). Each benchmark measures precision@k — the fraction of the top-k results that are genuinely relevant.

Datasets

Dataset	Memories	Queries	Domain
Sales Conversations	5,000	200	CRM, deal management
Support Tickets	10,000	300	Customer support, issue tracking
Research Notes	3,000	150	Multi-domain academic research
Incident Reports	2,000	100	Infrastructure, post-mortems

Results: Precision Improvement Over Similarity-Only

Temporal Queries

Queries where time context is critical (e.g., “What happened before the outage?”):

Method	Precision@5	Improvement
Similarity only	0.42	baseline
HEBBS temporal	0.71	+68%

Similarity search retrieves semantically related memories but ignores temporal context. Temporal recall surfaces memories that are chronologically proximate, which is what these queries actually need.

Causal Queries

Queries that require understanding cause-and-effect (e.g., “What caused the deal to stall?”):

Method	Precision@5	Improvement
Similarity only	0.38	baseline
HEBBS causal	0.62	+63%

Similarity search finds memories that mention similar topics but cannot follow causal chains. Causal recall traces CausedBy edges to surface the actual chain of events.

Analogical Queries

Queries that require cross-domain pattern matching (e.g., “Have we seen similar dynamics in other deals?”):

Method	Precision@5	Improvement
Similarity only	0.35	baseline
HEBBS analogical	0.50	+43%

Similarity search is limited to surface-level semantic similarity. Analogical recall identifies structural parallels across different entities and domains.

Similarity Queries

For standard semantic queries, HEBBS matches vector search baselines:

Method	Precision@5
Similarity only (baseline)	0.78
HEBBS similarity	0.79

No degradation — HEBBS’s HNSW implementation matches or slightly exceeds baseline vector search quality.

Combined Strategy Performance

When the agent can choose the optimal strategy per query (simulating a strategy-selection model):

Method	Average Precision@5
Similarity only (all queries)	0.48
HEBBS with optimal strategy selection	0.66
Overall improvement	+37%

Reflection Impact

After running the reflection pipeline on the test datasets:

Metric	Before Reflection	After Reflection	Change
Precision@5 (similarity)	0.78	0.82	+5%
Precision@5 (analogical)	0.50	0.58	+16%
Unique insights generated	0	142	—

Insights generated by reflection improve recall quality by providing higher-level summarizations that match a broader range of queries.

Methodology

• All benchmarks run on standardized hardware (m6i.2xlarge, 8 vCPU, 32 GB RAM).
• Each query is run 10 times; reported numbers are the mean.
• Ground truth labels are created by three independent annotators with majority vote.
• The benchmark suite is available in benches/cognitive/ and can be reproduced with hebbs-bench cognitive.