transcription/.github/instructions/tests.instructions.md

description, applyTo

description	applyTo
Authoring rules for the test suite, including markers, async discipline, and guard-test design.	tests/**/*.py

Tests

Primary references:

• AGENTS.md (Change Protocol — failing test first)
• docs/index.md and docs/invariant/*
• .github/skills/test-effectiveness-auditor/skill.md (periodic audit of this suite)

The suite is not only regression protection here — it is where several architectural rules are defined. tests/test_service_boundaries.py, tests/test_ui_boundaries.py, tests/test_provider_boundaries.py, tests/test_model_contract_guards.py, and tests/test_meta_contract_guards.py are the enforcement layer named in the AGENTS.md authority order. A weak test in this repository does not merely fail to catch a bug; it can silently repeal a documented invariant.

The baseline is green. uv run pytest -q -m "not external" must report zero failures and zero errors, and there is no tolerated set of known-failing tests.

Write the Failing Test First

For any behavioral fix, write the test before the fix and confirm it fails for the reason you expect. A test that passes against the broken code proves nothing, and several defects in this repository were subtle enough that a test written afterward would have done exactly that. If the new test passes immediately, you have not reproduced the defect yet.

Runner Configuration

Configured in pyproject.toml; do not work around these:

• --strict-markers — an unregistered marker is an error. Register new markers in [tool.pytest.ini_options] markers with a description rather than inventing one at the call site.
• asyncio_mode = "strict" — every async test needs an explicit @pytest.mark.asyncio, and async fixtures use @pytest_asyncio.fixture. There is no implicit promotion.
• filterwarnings = ["error:coroutine .* was never awaited:RuntimeWarning"] — an un-awaited coroutine is an error, not a warning. This usually means a mock replaced an async callable with a sync one, or an await was dropped. Fix the call; never silence the warning.

Markers and Layout

• unit — pure logic, no framework or database.
• integration — touches framework, database, or multi-component contracts.
• external — calls live services; slow and credential-dependent.

external tests must also carry their own skipif so the suite stays green without credentials (see tests/services/test_transcription_external.py). Local and documented runs use -m "not external"; CI intentionally runs unfiltered, which is equivalent because those tests skip themselves. Never let an unmarked test reach the network.

Place tests by the layer under test: tests/services/, tests/ui/, tests/api/, tests/providers/, tests/integration/, with cross-cutting guards at the top level.

Fixtures and Isolation

• Prefer the shared fixtures in tests/conftest.py (default_settings, async_session, default_session_factory, and the per-aggregate service fixtures) over building settings or engines by hand.
• Settings is isolated suite-wide by the session-scoped autouse fixture in conftest.py, because env_file resolves against the working directory. Tests that need env-file loading pass _env_file= explicitly; tests asserting declared defaults need nothing. Do not reintroduce reliance on a developer's local env file. Guarded by tests/test_config_isolation.py.
• Database fixtures refuse to run against anything but the per-test path, and that refusal is deliberate. Never relax it to point a destructive fixture at a real database.
• Tests must not leave artifacts outside tmp_path.

Assertion Strength

Assert on the domain effect, not on the fact that code ran.

• Prefer persisted state, status transitions, error categories, and evidence records over "no exception raised", "not None", or a bare status code.
• Read committed state through a separate session. Asserting against the same session that performed the write can pass on unflushed in-memory state and prove nothing about durability. This is how the atomicity guarantees in tests/services/test_workflows_reliability.py and tests/integration/test_pipeline_atomicity.py are made real.
• Critical paths need negative-path coverage — timeouts, provider failures, validation errors, cancellation. Happy-path-only coverage of a critical module is a gap, not a suite.
• Avoid count-threshold assertions as a proxy for correctness. A test asserting "at least N items were discovered" passes indefinitely while the thing it was meant to protect rots; assert on a specific known member instead.

Guard Tests

Structural guards carry extra obligations, because they are cited as proof that a rule holds.

• Guard the guard. Every scanning guard needs a companion assertion that the scan actually found something, following the existing test_*_are_discovered pattern. A guard that silently scans an empty set passes forever.
• Scope must match the claim. A guard's name and docstring must describe only what it actually verifies. A test covering one function while appearing to enforce a repo-wide rule is worse than no test, because it stops anyone from writing the real one.
• Prove non-vacuity by injected fault. Temporarily introduce the violation, confirm the guard fails with a comprehensible message, then revert. Do this whenever you add or materially change a guard. Revert with an explicit edit if the file has uncommitted changes — git checkout -- will discard them.
• Prefer structural analysis to substring matching. AST inspection of imports and definitions is resistant to false negatives; a bare-name search across the repository is not, since an unrelated mention anywhere makes dead code look reachable.
• Failure messages should name the offending file, symbol, and the remedy. These fire for people who did not write the guard.
• Any new file under .github/** must be added to ACTIVE_CONTRACT_FILES in tests/test_meta_contract_guards.py, or the completeness guard fails by design.

Redundancy

Duplicate coverage across layers costs runtime and dilutes signal. Pick the canonical layer for a behavior — unit for logic, integration for wiring — and let the other layer assert only what is unique to it. Retire tests superseded by a stronger guard instead of accumulating both, and record deliberate retentions with a rationale rather than leaving them unexplained.

Contract Sync Rule

When a test encodes or relaxes a documented rule, update the corresponding instruction file or docs/* page in the same change. When a guard test is the enforcement for a rule stated in AGENTS.md or an instruction file, cite the test by name there so the link survives refactoring.