Snappy’s promise: crawl and structure the web into a reliable, high-quality corpus — complete, fresh, and on-target for what was asked.
The claim SQA tests: The jobs the metaintro-chat search engine returned, in the chat thread, are relevant to what the user asked for. (claim C0).
This run tested the system against its contract, clause by clause. A single run can only witness some clauses; the rest stay UNKNOWN — never a faked pass.
1 pass · 0 fail · 7 unknown
C0 MUST
Headline promise
relevancy score = 80/100 (pass-band ≥ 60)
PASS
C1 MUST
User can sign in
no 'login' step in this run
UNKNOWN
C2 MUST
User can open a new thread
no 'open-thread' step in this run
UNKNOWN
C3 SHOULD
Onboarding gate completes
no 'onboarding' step in this run
UNKNOWN
C4 SHOULD
Filters from onboarding don't bias the query
no 'clear-filters' step in this run
UNKNOWN
C5 MUST
User-typed query is what the engine sees
no 'submit-query' step in this run
UNKNOWN
C10 SHOULD
Score holds across reruns
needs a sweep — a single run cannot witness this clause — needs a sweep
UNKNOWN
C12 MAY
Run completes within budget
needs a sweep — a single run cannot witness this clause — needs a sweep
UNKNOWN
TL;DR · 30-second primer
·Snappy (SUT) ran 1 run on profile corpus-f500.
·Result: Corpus Quality Index 80/100. Strong— see “Why this verdict” (each gap maps to a claim in the Contract).
1 ·THE VERDICT
the answer in one number
30-day CQI history
SNAPPY · DOMAIN ACTIVATION · RUN #7
Strong.
Run #7 of snappy on profile corpus-f500 for the query "CQI sweep — domain-activation". Corpus Quality Index 80/100.
Verdict WARN: C5 Snapshot freshness slipping on the long-tail shard.
AI synthesis · openai/gpt-4o-mini
The system successfully activated the domain but showed some degradation, achieving a Corpus Quality Index (CQI) score of 80 out of 100. The primary concern was that snapshot freshness was slipping on the long-tail shard, which contributed to the warning outcome. Overall, the run took 192.3 seconds and returned no jobs.
2 ·WHY THIS VERDICT
ranked by severity
SOFT
Snapshot freshness slipping on the long-tail shard
Expected
p90 snapshot age ≤ 14 days across all shards
Observed
Shard 0x7f p90 age climbed to 19 days; freshness-checker cron lag of ~6h observed
Why it matters
Stale snapshots feed stale extraction; the change-detector can't fire snapshot:changed if the crawl never re-runs.
Recommended action· 1 sprint
Investigate freshness-checker queue backpressure on the long-tail shard.