kai’s promise: store what matters and recall it on demand — surface the right memory for the ask, ranked, with no relevant memory missed.
The claim SQA tests: The jobs the metaintro-chat search engine returned, in the chat thread, are relevant to what the user asked for. (claim C0).
This run tested the system against its contract, clause by clause. A single run can only witness some clauses; the rest stay UNKNOWN — never a faked pass.
1 pass · 0 fail · 7 unknown
C0 MUST
Headline promise
relevancy score = 82/100 (pass-band ≥ 60)
PASS
C1 MUST
User can sign in
no 'login' step in this run
UNKNOWN
C2 MUST
User can open a new thread
no 'open-thread' step in this run
UNKNOWN
C3 SHOULD
Onboarding gate completes
no 'onboarding' step in this run
UNKNOWN
C4 SHOULD
Filters from onboarding don't bias the query
no 'clear-filters' step in this run
UNKNOWN
C5 MUST
User-typed query is what the engine sees
no 'submit-query' step in this run
UNKNOWN
C10 SHOULD
Score holds across reruns
needs a sweep — a single run cannot witness this clause — needs a sweep
UNKNOWN
C12 MAY
Run completes within budget
needs a sweep — a single run cannot witness this clause — needs a sweep
UNKNOWN
TL;DR · 30-second primer
·KAI (SUT) ran 1 run on profile longmem-phase-a.
·Result: Memory Recall Index 82/100. Strong recall— see “Why this verdict” (each gap maps to a claim in the Contract).
1 ·THE VERDICT
the answer in one number
30-day MRI history
KAI · MEMORY RECALL · RUN #18
Strong recall.
Run #18 of kai on profile longmem-phase-a for the query "MRI sweep — memory-recall". Memory Recall Index 82/100.
Verdict WARN: C4 Hybrid relevance dipped after a reranker weight change.
AI synthesis · openai/gpt-4o-mini
The system successfully completed its job but with a degraded performance, achieving a Memory Recall Index (MRI) score of 82 out of 100. This decline was attributed to a dip in hybrid relevance following a change in the reranker weight. The run lasted 43.5 seconds, but no jobs were returned.
2 ·WHY THIS VERDICT
ranked by severity
SOFT
Hybrid relevance dipped after a reranker weight change
Expected
Reranked nDCG@10 ≥ 0.85
Observed
nDCG@10 = 0.81 after the rescore-weight shift toward dense
Why it matters
Over-weighting dense buries exact-term (BM25) hits on proper nouns.
Recommended action· 1 sprint
Re-balance RRF dense/sparse split; verify on proper-noun query slice.