transcription/docs/test-effectiveness-audit-2026-08-20.md

Test Effectiveness Audit Report

1. Executive Verdict

• Effective with Conditions
• The suite has strong contract coverage for architecture governance, worker reliability, error taxonomy, evidence append-only semantics, and media safety.
• The largest confidence risk is a placeholder module with eight empty tests that always pass and contribute no regression signal.
• A smaller risk is several exception-path tests that assert only exception type and do not validate envelope/category/detail semantics.

2. Contract Coverage Matrix

Contract	Guarding Tests	Signal Quality	Gap	Action
Service boundary isolation	tests/test_service_boundaries.py	Strong	None	Keep as-is
UI boundary isolation	tests/test_ui_boundaries.py	Strong	None	Keep as-is
Canonical docs/instruction authority + settings parity	tests/test_meta_contract_guards.py	Strong	None	Keep as-is
Error envelope taxonomy mapping	tests/api/test_error_responses.py	Strong	None	Keep as-is
Worker non-retriable stop + resilience behavior	tests/test_worker.py, tests/services/test_workflows_reliability.py	Strong	None	Keep as-is
Evidence append-only + candidate promotion invariants	tests/services/test_v45_candidates.py	Strong	None	Keep as-is
UI/API media path safety	tests/test_media_path_safety.py, tests/ui/test_media_urls.py	Strong	None	Keep as-is
Service base behavior contract	tests/services/test_service_base.py	None (current tests empty)	High	Replace placeholders with real assertions or remove file

3. Weak/Redundant Test Findings

Finding ID	Location	Why Low-Signal	Risk	Recommendation
TE-01	tests/services/test_service_base.py:6-35	Contains eight test_* functions with docstrings only and no executable assertions.	High: false confidence and inflated pass count.	Replace with real behavior checks against ServiceBase session-scope semantics, or delete file until concrete tests exist.
TE-02	tests/test_engine_registry.py:36-37	test_disposing_an_unregistered_url_is_a_noop asserts only “no exception.” It does not verify registry state invariants before/after call.	Medium: regression may survive if behavior changes silently without raising.	Assert that previously created engine/session-factory instances for other URLs remain unchanged after noop disposal path.
TE-03	tests/services/test_v45_candidates.py:90-94	pytest.raises(CandidatePromotionError) validates type only; no assertions on message/category/suggestion for user-safe failure semantics.	Low-Med: weaker diagnostics contract protection.	Capture exception and assert critical error metadata fields to strengthen failure-path guarantees.

4. Prune/Strengthen Backlog

Task ID	Goal	Files	Acceptance Criteria	Validation
TE-T1	Eliminate zero-signal placeholder tests	tests/services/test_service_base.py, src/transcription/services/base.py	No empty test_* functions remain; each test has behavior assertions that fail on meaningful ServiceBase regressions.	uv run pytest tests/services/test_service_base.py
TE-T2	Strengthen noop disposal invariant test	tests/test_engine_registry.py	Noop disposal test verifies unaffected URL registries remain intact and disposed URL behavior is unchanged.	uv run pytest tests/test_engine_registry.py
TE-T3	Strengthen exception-path semantics checks	tests/services/test_v45_candidates.py (and similar raise-only tests where high-value)	Exception tests assert key semantic fields (message/category/suggestion or equivalent domain signal), not only type.	uv run pytest tests/services/test_v45_candidates.py

5. Confidence Recommendation

• Go with Conditions for test-confidence governance.
• Exit criteria:
  1. Complete TE-T1 (highest priority).
  2. Complete TE-T2.
  3. Apply TE-T3 at least on high-risk service error paths.