kai’s promise: store what matters and recall it on demand — surface the right memory for the ask, ranked, with no relevant memory missed.
The claim SQA tests: The jobs the metaintro-chat search engine returned, in the chat thread, are relevant to what the user asked for. (claim C0).
This run tested the system against its contract, clause by clause. A single run can only witness some clauses; the rest stay UNKNOWN — never a faked pass.
1 pass · 0 fail · 7 unknown
C0 MUST
Headline promise
relevancy score = 76/100 (pass-band ≥ 60)
PASS
C1 MUST
User can sign in
no 'login' step in this run
UNKNOWN
C2 MUST
User can open a new thread
no 'open-thread' step in this run
UNKNOWN
C3 SHOULD
Onboarding gate completes
no 'onboarding' step in this run
UNKNOWN
C4 SHOULD
Filters from onboarding don't bias the query
no 'clear-filters' step in this run
UNKNOWN
C5 MUST
User-typed query is what the engine sees
no 'submit-query' step in this run
UNKNOWN
C10 SHOULD
Score holds across reruns
needs a sweep — a single run cannot witness this clause — needs a sweep
UNKNOWN
C12 MAY
Run completes within budget
needs a sweep — a single run cannot witness this clause — needs a sweep
UNKNOWN
TL;DR · 30-second primer
·KAI (SUT) ran 1 run on profile longmem-phase-a.
·Result: Memory Recall Index 76/100. Strong recall— see “Why this verdict” (each gap maps to a claim in the Contract).
1 ·THE VERDICT
the answer in one number
30-day MRI history
KAI · MEMORY RECALL · RUN #20
Strong recall.
Run #20 of kai on profile longmem-phase-a for the query "MRI sweep — memory-recall". Memory Recall Index 76/100.
Verdict WARN: C1 Multi-hop recovering after reranker revert.
AI synthesis · openai/gpt-4o-mini
The system successfully completed its job but with a degraded performance, achieving a Memory Recall Index (MRI) score of 76 out of 100. The primary factor in this outcome was the multi-hop recovering after a reranker revert, which contributed to the warning status. The entire process took 45.2 seconds, indicating a need for improvement in the memory-recall functionality.
2 ·WHY THIS VERDICT
ranked by severity
SOFT
Multi-hop recovering after reranker revert
Expected
Multi-hop recall@10 ≥ 0.78
Observed
Multi-hop recall@10 back to 0.71; PRF re-enabled for weak (<0.5) initial hits
Why it matters
Trending back toward SLA but bridge-document recall not yet restored.
Recommended action· <1 sprint
Let prefetch-K tuning bake; re-run Phase-A benchmark next sweep.