transcription/docs/production-runbook.md

Production Runbook

This runbook is the operational checklist for releasing and monitoring the transcription system.

1. Pre-release gate checklist

1. Run the full suite: uv run pytest
2. Confirm contract guardrails are green:
   • uv run pytest tests/test_meta_contract_guards.py
3. Confirm health endpoint includes worker liveness payload (/healthz returns worker.state).
4. Confirm required runtime settings are present in deployment environment:
   • OPENROUTER_API_KEY
   • DATABASE__*
   • filesystem paths for data/logs/backups.
5. Confirm schema contract alignment is current:
   • src/transcription/db/models.py
   • docs/schema.md

2. Release execution steps

1. Deploy artifact/config to target environment.
2. Validate service startup:
   • /healthz responds 200
   • worker.state is running
3. Execute one smoke workflow:
   • create a document/job with at least one source
   • verify terminal job outcome updates
   • verify execution evidence row appended
4. Verify log flow:
   • stdout aggregation receives events
   • file logs are written under ./data/logs

3. Rollback triggers and actions

Trigger conditions

1. /healthz reports worker.state=failed
2. Repeated provider timeout/error spikes beyond normal baseline
3. Evidence write failures or DB persistence failures

Actions

1. Roll back app artifact and config to previous release.
2. Restart service and re-check /healthz.
3. Re-run smoke workflow and confirm worker returns to running.
4. Preserve incident evidence:
   • ./data/logs
   • relevant DB rows (job, job_source, execution_attempt)

4. Post-release monitoring checklist

First 24 hours

1. Monitor /healthz periodically for worker.state.
2. Track job terminal distribution (transcribed, partial_success, failed).
3. Sample timeout/error categories for abnormal increase.
4. Spot-check new execution_attempt records for append-only growth and timing metadata.

First 72 hours

1. Re-check error/timeout trend versus 24h baseline.
2. Verify no recurring worker-failed states.
3. Verify storage growth and rotation behavior under ./data/logs.
4. Confirm incident response notes are captured for any production anomalies.

5. Operator playbook for common incidents

Worker failed

1. Check /healthz payload (error_id, error_category).
2. Locate matching error in logs.
3. If non-transient defect persists, roll back.

Provider timeout spike

1. Confirm provider reachability and rate limits.
2. Review timeout frequency and impacted job volume.
3. If sustained, execute rollback criteria and notify stakeholders.

Partial-success increase

1. Inspect affected job_source and execution_attempt records.
2. Confirm failures are category-aligned (external/timeout/internal).
3. Triage whether issue is source quality, provider, or runtime regression.

6. Dependency upgrade policy

Dependencies are declared in pyproject.toml and resolved through the committed uv.lock. The lockfile guarantees reproducible installs; the version specifiers control what a deliberate uv lock --upgrade is allowed to move.

NiceGUI is pinned exactly (nicegui==3.13.0)

1. Rationale. NiceGUI bundles Quasar and Vue. Minor releases change component props, slots, and styling, which surfaces as visual and interaction regressions rather than import or type errors. The UI suite under tests/ui/ asserts structure and behavior, not rendered appearance, so a NiceGUI bump can pass the full test suite and still degrade the interface.
2. Scope of risk. All NiceGUI usage is confined to src/transcription/ui/ and uses only the public nicegui.ui and nicegui.events surfaces. The coupling is shallow, so the pin is about release stability, not about unpicking deep framework entanglement.
3. Current stance. Hold the exact pin through release stabilization. Do not widen it as incidental cleanup, and do not let automated dependency updates move it. This includes forgoing patch releases, which is the accepted cost.
4. Revisiting. Treat a NiceGUI upgrade as scheduled work with its own change window: bump the pin deliberately, run uv run pytest -m "not external", then manually verify each page contract in docs/ui/pages/ before accepting.

All other dependencies

Declared with >= floors and moved by explicit uv lock --upgrade. Verify with uv run ruff check ., uv run ty check, and uv run pytest -q -m "not external" before committing a changed lockfile.

7. Type-check suppression policy

uv run ty check is a blocking pre-commit gate. Suppressions are allowed only for proven SQLAlchemy descriptor false positives where runtime behavior is correct and the checker cannot represent the descriptor protocol at that call site.

Every suppression must be:

1. Targeted to a single rule (for example # ty: ignore[unresolved-attribute]).
2. Inline on the expression it suppresses (not file-wide).
3. Followed by a one-line rationale stating it is a SQLAlchemy descriptor false positive.

Do not use broad or rationale-free suppressions. If a diagnostic is not a known false positive, fix the code instead of suppressing it.

8. Worker shutdown budget

Worker shutdown waits for at most:

WORKER_PROVIDER_TIMEOUT_SECONDS + WORKER_SHUTDOWN_GRACE_SECONDS

WORKER_PROVIDER_TIMEOUT_SECONDS covers an in-flight provider call, and WORKER_SHUTDOWN_GRACE_SECONDS is extra time for the loop to persist outcomes and exit cleanly after the call returns.

Set the container or service termination grace period above this total budget. If termination grace is shorter, the process may be killed before terminal status and evidence writes are finalized.