transcription/docs/production-runbook.md

# Production Runbook

This runbook is the operational checklist for releasing and monitoring the transcription system.

## 1. Pre-release gate checklist

1. Run the full suite: `uv run pytest`
2. Confirm contract guardrails are green:
   - `uv run pytest tests/test_meta_contract_guards.py`
3. Confirm health endpoint includes worker liveness payload (`/healthz` returns `worker.state`).
4. Confirm required runtime settings are present in deployment environment:
   - `OPENROUTER_API_KEY`
   - `DATABASE__*`
   - filesystem paths for data/logs/backups.
   - `CLOUDFLARE_TUNNEL_TOKEN`
5. Confirm schema contract alignment is current:
   - `src/transcription/db/models.py`
   - `docs/schema.md`

## 2. Release execution steps

1. Deploy artifact/config to target environment.
   - Production stack: `docker compose -f docker-compose.production.yml up -d --build`
   - `Settings` loads from explicit `_env_file`, then `ENV_FILE`, then the repository-root `.env.production`; it does not resolve relative to the process working directory.
   - For Runtime Settings writes in production, mount `.env.production` into the app container and set `RUNTIME_SETTINGS_ENV_FILE=/app/.env.production`.
   - If deployment uses a non-default env-file location, set both `ENV_FILE` and `RUNTIME_SETTINGS_ENV_FILE` to that absolute path so startup reads and Settings-page writes stay aligned.
   - For SQLite -> PostgreSQL cutover, run `uv run python tools/export_import_migration.py verify --source-db <sqlite-path-or-url> --target-db <postgres-url>` before switching runtime.
2. Validate service startup:
   - `/healthz` responds `200`
   - if `RUN_EMBEDDED_WORKER=true`, `worker.state` is `running`
   - if `RUN_EMBEDDED_WORKER=false`, validate `worker` container is running in Compose
     (worker healthcheck is intentionally disabled because it does not expose `/healthz`)
   - validate `cloudflared` logs show active tunnel routes and no ingress errors
3. Execute one smoke workflow:
   - create a document/job with at least one source
   - verify terminal job outcome updates
   - verify execution evidence row appended
4. Verify log flow:
   - stdout aggregation receives events
   - file logs are written under `./data/logs`
5. Create a fresh PostgreSQL backup after successful deployment:
   - `sh deploy/backup/create_postgres_backup.sh` (creates DB dump plus uploads/prompts backups under `BACKUP_DIR`)

## 3. Rollback triggers and actions

### Trigger conditions

1. `/healthz` reports `worker.state=failed`
2. Repeated provider timeout/error spikes beyond normal baseline
3. Evidence write failures or DB persistence failures

### Actions

1. Roll back app artifact and config to previous release.
2. Restart service and re-check `/healthz`.
3. Re-run smoke workflow and confirm worker returns to `running`.
4. Preserve incident evidence:
   - `./data/logs`
   - relevant DB rows (`job`, `job_source`, `execution_attempt`)
5. If persistence regression is confirmed, restore the latest valid DB dump:
   - `sh deploy/backup/restore_postgres_backup.sh <dump-file>`
   - Synology media mirror and paired config snapshot (same timestamp) are restored automatically when present.

## 4. Post-release monitoring checklist

## First 24 hours

1. Monitor `/healthz` periodically for `worker.state`.
2. Track job terminal distribution (`transcribed`, `partial_success`, `failed`).
3. Sample timeout/error categories for abnormal increase.
4. Spot-check new `execution_attempt` records for append-only growth and timing metadata.

## First 72 hours

1. Re-check error/timeout trend versus 24h baseline.
2. Verify no recurring worker-failed states.
3. Verify storage growth and rotation behavior under `./data/logs`.
4. Confirm incident response notes are captured for any production anomalies.

## 5. Operator playbook for common incidents

### Worker failed

1. Check `/healthz` payload (`error_id`, `error_category`).
2. Locate matching error in logs.
3. If non-transient defect persists, roll back.

### Provider timeout spike

1. Confirm provider reachability and rate limits.
2. Review timeout frequency and impacted job volume.
3. If sustained, execute rollback criteria and notify stakeholders.

### Partial-success increase

1. Inspect affected `job_source` and `execution_attempt` records.
2. Confirm failures are category-aligned (`external`/`timeout`/`internal`).
3. Triage whether issue is source quality, provider, or runtime regression.

### Cloudflare ingress/access failure

1. Check `cloudflared` container logs for ingress parse, DNS, or auth failures.
2. Confirm `deploy/cloudflared/config.yml` hostname mappings are correct.
3. Confirm `CLOUDFLARE_TUNNEL_TOKEN` in `.env.production` matches the tunnel configured in Cloudflare.
4. Confirm Cloudflare Access app policy includes the intended identity/group for that hostname.
5. If logs show SRV/DNS failures via `127.0.0.11`, use the compose-defined resolver override
   (`dns: 1.1.1.1, 1.0.0.1`) and ensure outbound TCP 443 is allowed.

### Backup or restore failure

1. Verify `postgres` container is healthy and accepting connections.
2. Confirm dump file exists and is non-zero size.
3. Re-run backup/restore scripts with explicit `ENV_FILE` and `COMPOSE_FILE` if using non-default paths.
4. If direct Synology copy fails, keep local backup and resolve mount/network before next backup cycle.
   - For LXC setups, use `deploy/backup/mount_synology_cifs.example.sh` as the persistent mount template.

## 6. Dependency upgrade policy

Dependencies are declared in `pyproject.toml` and resolved through the committed
`uv.lock`. The lockfile guarantees reproducible installs; the version specifiers
control what a deliberate `uv lock --upgrade` is allowed to move.

### NiceGUI is pinned exactly (`nicegui==3.13.0`)

1. **Rationale.** NiceGUI bundles Quasar and Vue. Minor releases change component
   props, slots, and styling, which surfaces as visual and interaction regressions
   rather than import or type errors. The UI suite under `tests/ui/` asserts
   structure and behavior, not rendered appearance, so a NiceGUI bump can pass the
   full test suite and still degrade the interface.
2. **Scope of risk.** All NiceGUI usage is confined to `src/transcription/ui/` and
   uses only the public `nicegui.ui` and `nicegui.events` surfaces. The coupling is
   shallow, so the pin is about release stability, not about unpicking deep
   framework entanglement.
3. **Current stance.** Hold the exact pin through release stabilization. Do not
   widen it as incidental cleanup, and do not let automated dependency updates move
   it. This includes forgoing patch releases, which is the accepted cost.
4. **Revisiting.** Treat a NiceGUI upgrade as scheduled work with its own change
   window: bump the pin deliberately, run `uv run pytest -m "not external"`, then
   manually verify each page contract in `docs/ui/pages/` before accepting.

### All other dependencies

Declared with `>=` floors and moved by explicit `uv lock --upgrade`. Verify with
`uv run ruff check .`, `uv run ty check`, and `uv run pytest -q -m "not external"`
before committing a changed lockfile.

## 7. Type-check suppression policy

`uv run ty check` is a blocking pre-commit gate. Suppressions are allowed only for
proven SQLAlchemy descriptor false positives where runtime behavior is correct and
the checker cannot represent the descriptor protocol at that call site.

Every suppression must be:

1. **Targeted** to a single rule (for example `# ty: ignore[unresolved-attribute]`).
2. **Inline** on the expression it suppresses (not file-wide).
3. Followed by a **one-line rationale** stating it is a SQLAlchemy descriptor false positive.

Do not use broad or rationale-free suppressions. If a diagnostic is not a known
false positive, fix the code instead of suppressing it.

## 8. Worker shutdown budget

Worker shutdown waits for at most:

`WORKER_PROVIDER_TIMEOUT_SECONDS + WORKER_SHUTDOWN_GRACE_SECONDS`

`WORKER_PROVIDER_TIMEOUT_SECONDS` covers an in-flight provider call, and
`WORKER_SHUTDOWN_GRACE_SECONDS` is extra time for the loop to persist outcomes
and exit cleanly after the call returns.

Set the container or service termination grace period **above this total**
budget. If termination grace is shorter, the process may be killed before
terminal status and evidence writes are finalized.

## 9. Horizontal scaling precondition

Multiple worker replicas can race on execution-attempt numbering for the same
`(job_id, source_id)` pair. The runtime now retries boundedly on unique-key
conflicts (`uq_execution_attempt_number`) and surfaces a conflict-domain error
if retries are exhausted.

Do not deploy additional worker replicas unless this conflict-retry path and its
tests are present and green in the target build.