Testing guide

Check what can run

Inspect the exact project version, test definition, required capabilities, and runner_available. A declared harness alone does not establish runner support. Documentation-only projects remain useful without an executable test.

The runner supports bounded declarative text tests and fixed workflows with Ollama or LM Studio. Runtime, engine, model, and structured-output support must pass preflight. Camera, audio, actuation, arbitrary tools, and dynamic graphs are outside this workflow.

pnpm build:contracts
pnpm --filter localfirstdotsh build
node packages/runner/dist/cli/bin.js help

Commands below run from that repository root. Install your runtime and a suitable model separately; model downloads are not part of planning or execution.

Discover and plan a component test

node packages/runner/dist/cli/bin.js runtime discover --runtime ollama --endpoint http://127.0.0.1:11434
node packages/runner/dist/cli/bin.js test plan <project-id> --version <version-id> --test <test-key> --runtime ollama --endpoint http://127.0.0.1:11434 --model <installed-model> --registry https://staging.localfirst.sh

Replace angle-bracket placeholders with IDs from the selected immutable project version and your installed model. For LM Studio, use --runtime lmstudio and its configured loopback endpoint. Approve runtime discovery when prompted.

Planning reads the bundle, checks compatibility, and saves an exact local plan without inference. Review the selected model, settings, fixtures, file identities, network effects, and request/token/time limits.

Approve execution and inspect the outcome

node packages/runner/dist/cli/bin.js test run <saved-plan-path>
node packages/runner/dist/cli/bin.js report show <saved-report-path>

Use the paths printed by the runner. A person approves the exact execution plan locally. Changed content or settings require a new plan and approval. A cancelled, failed, or partial run remains visible with its actual outcome.

For multi-agent environments, use environment plan and environment run instead. The bakery walkthrough shows the full sequence and expected fixture outcomes.

--offline allows a saved plan to run without asking the registry about withdrawal or changes. It does not prove that inference was offline. Read the report's actual locality and network observations.

Read workflow reports and share deliberately

node packages/runner/dist/cli/bin.js report trace <saved-report-path> --workspace <prepared-workspace> --out <new-trail.html>
node packages/runner/dist/cli/bin.js report export <saved-report-path> --review
node packages/runner/dist/cli/bin.js report submit <export-file> --registry https://staging.localfirst.sh

The local trace shows role prompts, inputs, outputs, steps, and assertion mismatches within its privacy rules. Choose a new output filename. Private-content inclusion in a trace is a separate explicit choice.

Export and submission are separate from execution. Inspect the sanitized file before uploading. Runs with private input or business-data overrides cannot be exported as public fixture evidence. Submission requires an authorized registry connection and remains private and pending until operator review.

Component, workflow, and live integration coverage are different. A passing fictional fixture does not prove an inbox, calendar, payment, or business-system integration. See integration readiness before using a workflow operationally.

When a test is blocked

• Runtime unreachable: start the local server and confirm its loopback endpoint.
• Model or profile unsupported: choose an installed compatible model and supported engine. Unknown capability is not a pass.
• Digest or workspace changed: prepare the intended bundle in a new directory and plan again.
• Handoff or upload unavailable: check deployment support. Retain the local report; local execution does not require a hosted upload.
• Assertion mismatch: inspect expected and actual values and the exact prompts/model. Report the failure without replacing it with an assumed success.