transcription/docs/production-runbook.md

Production Runbook

This runbook is the operational checklist for releasing and monitoring the transcription system.

1. Pre-release gate checklist

1. Run the full suite: uv run pytest
2. Confirm contract guardrails are green:
   • uv run pytest tests/test_meta_contract_guards.py
3. Confirm health endpoint includes worker liveness payload (/healthz returns worker.state).
4. Confirm required runtime settings are present in deployment environment:
   • OPENROUTER_API_KEY
   • DATABASE__*
   • filesystem paths for data/logs/backups.
   • CLOUDFLARE_TUNNEL_TOKEN
5. Confirm schema contract alignment is current:
   • src/transcription/db/models.py
   • docs/schema.md

2. Release execution steps

1. Deploy artifact/config to target environment.
   • V6.0 Phase 1 production stack: docker compose -f docker-compose.production.yml up -d --build
   • For Runtime Settings writes in production, mount .env.production into the app container and set RUNTIME_SETTINGS_ENV_FILE=/app/.env.production.
   • For SQLite -> PostgreSQL cutover, run uv run python tools/export_import_migration.py verify --source-db <sqlite-path-or-url> --target-db <postgres-url> before switching runtime.
2. Validate service startup:
   • /healthz responds 200
   • if RUN_EMBEDDED_WORKER=true, worker.state is running
   • if RUN_EMBEDDED_WORKER=false, validate worker container is running in Compose (worker healthcheck is intentionally disabled because it does not expose /healthz)
   • validate cloudflared logs show active tunnel routes and no ingress errors
3. Execute one smoke workflow:
   • create a document/job with at least one source
   • verify terminal job outcome updates
   • verify execution evidence row appended
4. Verify log flow:
   • stdout aggregation receives events
   • file logs are written under ./data/logs
5. Create a fresh PostgreSQL backup after successful deployment:
   • sh deploy/backup/create_postgres_backup.sh (creates local DB dump + local incremental uploads mirror under DATABASE_BACKUP_DIR)

3. Rollback triggers and actions

Trigger conditions

1. /healthz reports worker.state=failed
2. Repeated provider timeout/error spikes beyond normal baseline
3. Evidence write failures or DB persistence failures

Actions

1. Roll back app artifact and config to previous release.
2. Restart service and re-check /healthz.
3. Re-run smoke workflow and confirm worker returns to running.
4. Preserve incident evidence:
   • ./data/logs
   • relevant DB rows (job, job_source, execution_attempt)
5. If persistence regression is confirmed, restore the latest valid DB dump:
   • sh deploy/backup/restore_postgres_backup.sh <dump-file>
   • Synology media mirror and paired config snapshot (same timestamp) are restored automatically when present.

4. Post-release monitoring checklist

First 24 hours

1. Monitor /healthz periodically for worker.state.
2. Track job terminal distribution (transcribed, partial_success, failed).
3. Sample timeout/error categories for abnormal increase.
4. Spot-check new execution_attempt records for append-only growth and timing metadata.

First 72 hours

1. Re-check error/timeout trend versus 24h baseline.
2. Verify no recurring worker-failed states.
3. Verify storage growth and rotation behavior under ./data/logs.
4. Confirm incident response notes are captured for any production anomalies.

5. Operator playbook for common incidents

Worker failed

1. Check /healthz payload (error_id, error_category).
2. Locate matching error in logs.
3. If non-transient defect persists, roll back.

Provider timeout spike

1. Confirm provider reachability and rate limits.
2. Review timeout frequency and impacted job volume.
3. If sustained, execute rollback criteria and notify stakeholders.

Partial-success increase

1. Inspect affected job_source and execution_attempt records.
2. Confirm failures are category-aligned (external/timeout/internal).
3. Triage whether issue is source quality, provider, or runtime regression.

Cloudflare ingress/access failure

1. Check cloudflared container logs for ingress parse, DNS, or auth failures.
2. Confirm deploy/cloudflared/config.yml hostname mappings are correct.
3. Confirm CLOUDFLARE_TUNNEL_TOKEN in .env.production matches the tunnel configured in Cloudflare.
4. Confirm Cloudflare Access app policy includes the intended identity/group for that hostname.
5. If logs show SRV/DNS failures via 127.0.0.11, use the compose-defined resolver override (dns: 1.1.1.1, 1.0.0.1) and ensure outbound TCP 443 is allowed.

Backup or restore failure

1. Verify postgres container is healthy and accepting connections.
2. Confirm dump file exists and is non-zero size.
3. Re-run backup/restore scripts with explicit ENV_FILE and COMPOSE_FILE if using non-default paths.
4. If direct Synology copy fails, keep local backup and resolve mount/network before next backup cycle.
   • For LXC setups, use deploy/backup/mount_synology_cifs.example.sh as the persistent mount template.

6. Dependency upgrade policy

Dependencies are declared in pyproject.toml and resolved through the committed uv.lock. The lockfile guarantees reproducible installs; the version specifiers control what a deliberate uv lock --upgrade is allowed to move.

NiceGUI is pinned exactly (nicegui==3.13.0)

1. Rationale. NiceGUI bundles Quasar and Vue. Minor releases change component props, slots, and styling, which surfaces as visual and interaction regressions rather than import or type errors. The UI suite under tests/ui/ asserts structure and behavior, not rendered appearance, so a NiceGUI bump can pass the full test suite and still degrade the interface.
2. Scope of risk. All NiceGUI usage is confined to src/transcription/ui/ and uses only the public nicegui.ui and nicegui.events surfaces. The coupling is shallow, so the pin is about release stability, not about unpicking deep framework entanglement.
3. Current stance. Hold the exact pin through release stabilization. Do not widen it as incidental cleanup, and do not let automated dependency updates move it. This includes forgoing patch releases, which is the accepted cost.
4. Revisiting. Treat a NiceGUI upgrade as scheduled work with its own change window: bump the pin deliberately, run uv run pytest -m "not external", then manually verify each page contract in docs/ui/pages/ before accepting.

All other dependencies

Declared with >= floors and moved by explicit uv lock --upgrade. Verify with uv run ruff check ., uv run ty check, and uv run pytest -q -m "not external" before committing a changed lockfile.

7. Type-check suppression policy

uv run ty check is a blocking pre-commit gate. Suppressions are allowed only for proven SQLAlchemy descriptor false positives where runtime behavior is correct and the checker cannot represent the descriptor protocol at that call site.

Every suppression must be:

1. Targeted to a single rule (for example # ty: ignore[unresolved-attribute]).
2. Inline on the expression it suppresses (not file-wide).
3. Followed by a one-line rationale stating it is a SQLAlchemy descriptor false positive.

Do not use broad or rationale-free suppressions. If a diagnostic is not a known false positive, fix the code instead of suppressing it.

8. Worker shutdown budget

Worker shutdown waits for at most:

WORKER_PROVIDER_TIMEOUT_SECONDS + WORKER_SHUTDOWN_GRACE_SECONDS

WORKER_PROVIDER_TIMEOUT_SECONDS covers an in-flight provider call, and WORKER_SHUTDOWN_GRACE_SECONDS is extra time for the loop to persist outcomes and exit cleanly after the call returns.

Set the container or service termination grace period above this total budget. If termination grace is shorter, the process may be killed before terminal status and evidence writes are finalized.

9. Horizontal scaling precondition

Multiple worker replicas can race on execution-attempt numbering for the same (job_id, source_id) pair. The runtime now retries boundedly on unique-key conflicts (uq_execution_attempt_number) and surfaces a conflict-domain error if retries are exhausted.

Do not deploy additional worker replicas unless this conflict-retry path and its tests are present and green in the target build.