Safe first run #

Diátaxis form: how-to. Get a feel for SQA's output without touching real infrastructure or credentials.

When to use this guide #

You've just cloned the repo. Before pointing SQA at production systems (which is what make run does - see the warning below), you want to see what a finished run actually looks like.

The mock system #

src/systems/mock/ is a self-contained system that returns one of every outcome - pass, warn, fail, error, skip - without any external services. It exists for exactly this moment.

make install     # bun install --frozen-lockfile (runs make doctor first)
make run-mock    # SQA_SYSTEM=mock LOG_FORMAT=pretty bun run src/index.ts

You should see a summary block like:

══════════════════════════════════════════════════════════════════════════════
 SQA RUN SUMMARY
 trace: <id>   trigger: unknown (<user>)   duration: ~1ms
──────────────────────────────────────────────────────────────────────────────
  ✔ pass   1.1   mock.always-pass   ~200µs
  ⚠ warn   1.2   mock.always-warn   ~200µs   synthetic warn for demo purposes
  ✘ fail   1.3   mock.always-fail   ~200µs   synthetic fail for demo purposes
  ✘ error  1.4   mock.always-error  ~200µs   synthetic error for demo purposes
  ⊘ skip   1.5   mock.always-skip   ~200µs   synthetic skip for demo purposes
──────────────────────────────────────────────────────────────────────────────
 RESULT: ✘ ERROR - 4 of 5 probes did not pass (pass=1, warn=1, fail=1, error=1, skip=1)
 HEADLINE: <first non-pass probe> - <its reason>
 EXIT:     1
══════════════════════════════════════════════════════════════════════════════

The run exits 1 because error (worse than fail per the severity rule in ADR-0012) maps to a non-zero exit code via exitCodeFor(). That's the correct behaviour for the demo - a real cron/CI run with this outcome would also exit 1, and you want to see it once.

Why the mock exists #

Two LLM cold-start reviews flagged the same friction: a newcomer's first make run either touches real systems (S3 PUT/DELETE, Redis SET/GET, OpenRouter calls - credentials needed, probes mutate state) or returns a wall of skip (uninstructive).

The mock system gives the newcomer a complete summary - every outcome rendered, every colour code visible, every column shape explained - in a sub-millisecond run with no env file required.

The probes #

Five probes, one per outcome. Each is a few lines:

• mock.always-pass - returns pass.
• mock.always-warn - returns warn ("system answered, degraded but not blocking").
• mock.always-fail - returns fail ("system answered, answer was wrong").
• mock.always-error - returns error ("could not run to completion").
• mock.always-skip - returns skip ("deliberately not run; env not configured").

Defined in src/systems/mock/probes.ts (co-located with the segment, since they're not reusable against any real external system - see the file's header comment). The segment that runs all five is src/systems/mock/demo.ts. Both follow the same contract as a real probe / segment, so reading them is a useful introduction to add-a-component ↗ and add-a-segment ↗.

What make run does (and why you don't run it first) #

make run invokes the default system, currently snappy. Its probes hit production: live HTTP endpoints, S3 (signed PUT/DELETE of a probe key), Redis (SET/GET), MongoDB, ClickHouse, Hatchet, Loki, OpenRouter. Without .env populated, the probes return skip for missing targets - which is safe but uninformative. With .env populated, the probes mutate real systems briefly (probe keys are deleted at the end). It's the right behaviour for operational verification, but it's not what you want for a first run.

When you're ready, copy .env.example to .env, fill in the targets you have credentials for, and make run. Probes for unconfigured targets return skip; the run still completes and the summary tells you what was and wasn't verified.