gpt-5.3 codex review: Phase 7 and the addition of the new test-effectiveness-auditor skill.

docs/production-runbook.md

@@ -0,0 +1,84 @@
+# Production Runbook
+
+This runbook is the operational checklist for releasing and monitoring the transcription system.
+
+## 1. Pre-release gate checklist
+
+1. Run the full suite: `uv run pytest`
+2. Confirm contract guardrails are green:
+   - `uv run pytest tests/test_meta_contract_guards.py`
+3. Confirm health endpoint includes worker liveness payload (`/healthz` returns `worker.state`).
+4. Confirm required runtime settings are present in deployment environment:
+   - `OPENROUTER_API_KEY`
+   - `DATABASE__*`
+   - filesystem paths for data/logs/backups.
+5. Confirm schema contract alignment is current:
+   - `src/transcription/db/models.py`
+   - `docs/ver4/schema_v4.md`
+
+## 2. Release execution steps
+
+1. Deploy artifact/config to target environment.
+2. Validate service startup:
+   - `/healthz` responds `200`
+   - `worker.state` is `running`
+3. Execute one smoke workflow:
+   - create a document/job with at least one source
+   - verify terminal job outcome updates
+   - verify execution evidence row appended
+4. Verify log flow:
+   - stdout aggregation receives events
+   - file logs are written under `./data/logs`
+
+## 3. Rollback triggers and actions
+
+### Trigger conditions
+
+1. `/healthz` reports `worker.state=failed`
+2. Repeated provider timeout/error spikes beyond normal baseline
+3. Evidence write failures or DB persistence failures
+
+### Actions
+
+1. Roll back app artifact and config to previous release.
+2. Restart service and re-check `/healthz`.
+3. Re-run smoke workflow and confirm worker returns to `running`.
+4. Preserve incident evidence:
+   - `./data/logs`
+   - relevant DB rows (`job`, `job_source`, `execution_attempt`)
+
+## 4. Post-release monitoring checklist
+
+## First 24 hours
+
+1. Monitor `/healthz` periodically for `worker.state`.
+2. Track job terminal distribution (`transcribed`, `partial_success`, `failed`).
+3. Sample timeout/error categories for abnormal increase.
+4. Spot-check new `execution_attempt` records for append-only growth and timing metadata.
+
+## First 72 hours
+
+1. Re-check error/timeout trend versus 24h baseline.
+2. Verify no recurring worker-failed states.
+3. Verify storage growth and rotation behavior under `./data/logs`.
+4. Confirm incident response notes are captured for any production anomalies.
+
+## 5. Operator playbook for common incidents
+
+### Worker failed
+
+1. Check `/healthz` payload (`error_id`, `error_category`).
+2. Locate matching error in logs.
+3. If non-transient defect persists, roll back.
+
+### Provider timeout spike
+
+1. Confirm provider reachability and rate limits.
+2. Review timeout frequency and impacted job volume.
+3. If sustained, execute rollback criteria and notify stakeholders.
+
+### Partial-success increase
+
+1. Inspect affected `job_source` and `execution_attempt` records.
+2. Confirm failures are category-aligned (`external`/`timeout`/`internal`).
+3. Triage whether issue is source quality, provider, or runtime regression.

docs/test-effectiveness-audit-2026-08-20.md

@@ -0,0 +1,45 @@
+# Test Effectiveness Audit Report
+
+## 1. Executive Verdict
+
+- **Effective with Conditions**
+- The suite has strong contract coverage for architecture governance, worker reliability, error taxonomy, evidence append-only semantics, and media safety.
+- The largest confidence risk is a placeholder module with eight empty tests that always pass and contribute no regression signal.
+- A smaller risk is several exception-path tests that assert only exception type and do not validate envelope/category/detail semantics.
+
+## 2. Contract Coverage Matrix
+
+| Contract | Guarding Tests | Signal Quality | Gap | Action |
+| :--- | :--- | :--- | :--- | :--- |
+| Service boundary isolation | `tests/test_service_boundaries.py` | Strong | None | Keep as-is |
+| UI boundary isolation | `tests/test_ui_boundaries.py` | Strong | None | Keep as-is |
+| Canonical docs/instruction authority + settings parity | `tests/test_meta_contract_guards.py` | Strong | None | Keep as-is |
+| Error envelope taxonomy mapping | `tests/api/test_error_responses.py` | Strong | None | Keep as-is |
+| Worker non-retriable stop + resilience behavior | `tests/test_worker.py`, `tests/services/test_workflows_reliability.py` | Strong | None | Keep as-is |
+| Evidence append-only + candidate promotion invariants | `tests/services/test_v45_candidates.py` | Strong | None | Keep as-is |
+| UI/API media path safety | `tests/test_media_path_safety.py`, `tests/ui/test_media_urls.py` | Strong | None | Keep as-is |
+| Service base behavior contract | `tests/services/test_service_base.py` | **None (current tests empty)** | **High** | Replace placeholders with real assertions or remove file |
+
+## 3. Weak/Redundant Test Findings
+
+| Finding ID | Location | Why Low-Signal | Risk | Recommendation |
+| :--- | :--- | :--- | :--- | :--- |
+| TE-01 | `tests/services/test_service_base.py:6-35` | Contains eight `test_*` functions with docstrings only and no executable assertions. | High: false confidence and inflated pass count. | Replace with real behavior checks against `ServiceBase` session-scope semantics, or delete file until concrete tests exist. |
+| TE-02 | `tests/test_engine_registry.py:36-37` | `test_disposing_an_unregistered_url_is_a_noop` asserts only “no exception.” It does not verify registry state invariants before/after call. | Medium: regression may survive if behavior changes silently without raising. | Assert that previously created engine/session-factory instances for other URLs remain unchanged after noop disposal path. |
+| TE-03 | `tests/services/test_v45_candidates.py:90-94` | `pytest.raises(CandidatePromotionError)` validates type only; no assertions on message/category/suggestion for user-safe failure semantics. | Low-Med: weaker diagnostics contract protection. | Capture exception and assert critical error metadata fields to strengthen failure-path guarantees. |
+
+## 4. Prune/Strengthen Backlog
+
+| Task ID | Goal | Files | Acceptance Criteria | Validation |
+| :--- | :--- | :--- | :--- | :--- |
+| TE-T1 | Eliminate zero-signal placeholder tests | `tests/services/test_service_base.py`, `src/transcription/services/base.py` | No empty `test_*` functions remain; each test has behavior assertions that fail on meaningful `ServiceBase` regressions. | `uv run pytest tests/services/test_service_base.py` |
+| TE-T2 | Strengthen noop disposal invariant test | `tests/test_engine_registry.py` | Noop disposal test verifies unaffected URL registries remain intact and disposed URL behavior is unchanged. | `uv run pytest tests/test_engine_registry.py` |
+| TE-T3 | Strengthen exception-path semantics checks | `tests/services/test_v45_candidates.py` (and similar raise-only tests where high-value) | Exception tests assert key semantic fields (message/category/suggestion or equivalent domain signal), not only type. | `uv run pytest tests/services/test_v45_candidates.py` |
+
+## 5. Confidence Recommendation
+
+- **Go with Conditions** for test-confidence governance.
+- Exit criteria:
+  1. Complete TE-T1 (highest priority).
+  2. Complete TE-T2.
+  3. Apply TE-T3 at least on high-risk service error paths.