generated from john/python-template
Jim LancasterandCopilot App
5566f48fc0
Quality Gate / gate (push) Failing after 11s
Update the reviewer skill so its procedure matches how this repo actually works: - Route review reports to docs/reviews/ and mark them non-canonical, resolving the conflict where reports landed in the same docs/ tree they resolve findings against. - Pin verification commands to uv (uv run ruff check / ty check / pytest -m 'not external'). - Record the pytest contract: strict markers, strict asyncio mode, and the never-awaited-coroutine warning promoted to an error. - Convert the deterministic checks to a table with an Enforced by column; three checks are unenforced and one only partial, which are now findings by construction. - Add a consequence-based severity rubric and a Direction column for bidirectional drift. - Escalate test-suite concerns to test-effectiveness-auditor. Also fix tests/test_db.py, which was missing 'from sqlalchemy import text' while using it in 14 places. Three tests were failing with NameError. Wrapped the pre-existing long lines in the same file so it lints clean. Document the deliberate nicegui==3.13.0 pin in pyproject.toml, a new runbook dependency upgrade policy, and the reviewer skill, so the pin is not flagged as a defect or widened as incidental cleanup. Co-authored-by: Copilot App <[email protected]>
116 lines
4.5 KiB
Markdown
116 lines
4.5 KiB
Markdown
# Production Runbook
|
|
|
|
This runbook is the operational checklist for releasing and monitoring the transcription system.
|
|
|
|
## 1. Pre-release gate checklist
|
|
|
|
1. Run the full suite: `uv run pytest`
|
|
2. Confirm contract guardrails are green:
|
|
- `uv run pytest tests/test_meta_contract_guards.py`
|
|
3. Confirm health endpoint includes worker liveness payload (`/healthz` returns `worker.state`).
|
|
4. Confirm required runtime settings are present in deployment environment:
|
|
- `OPENROUTER_API_KEY`
|
|
- `DATABASE__*`
|
|
- filesystem paths for data/logs/backups.
|
|
5. Confirm schema contract alignment is current:
|
|
- `src/transcription/db/models.py`
|
|
- `docs/schema.md`
|
|
|
|
## 2. Release execution steps
|
|
|
|
1. Deploy artifact/config to target environment.
|
|
2. Validate service startup:
|
|
- `/healthz` responds `200`
|
|
- `worker.state` is `running`
|
|
3. Execute one smoke workflow:
|
|
- create a document/job with at least one source
|
|
- verify terminal job outcome updates
|
|
- verify execution evidence row appended
|
|
4. Verify log flow:
|
|
- stdout aggregation receives events
|
|
- file logs are written under `./data/logs`
|
|
|
|
## 3. Rollback triggers and actions
|
|
|
|
### Trigger conditions
|
|
|
|
1. `/healthz` reports `worker.state=failed`
|
|
2. Repeated provider timeout/error spikes beyond normal baseline
|
|
3. Evidence write failures or DB persistence failures
|
|
|
|
### Actions
|
|
|
|
1. Roll back app artifact and config to previous release.
|
|
2. Restart service and re-check `/healthz`.
|
|
3. Re-run smoke workflow and confirm worker returns to `running`.
|
|
4. Preserve incident evidence:
|
|
- `./data/logs`
|
|
- relevant DB rows (`job`, `job_source`, `execution_attempt`)
|
|
|
|
## 4. Post-release monitoring checklist
|
|
|
|
## First 24 hours
|
|
|
|
1. Monitor `/healthz` periodically for `worker.state`.
|
|
2. Track job terminal distribution (`transcribed`, `partial_success`, `failed`).
|
|
3. Sample timeout/error categories for abnormal increase.
|
|
4. Spot-check new `execution_attempt` records for append-only growth and timing metadata.
|
|
|
|
## First 72 hours
|
|
|
|
1. Re-check error/timeout trend versus 24h baseline.
|
|
2. Verify no recurring worker-failed states.
|
|
3. Verify storage growth and rotation behavior under `./data/logs`.
|
|
4. Confirm incident response notes are captured for any production anomalies.
|
|
|
|
## 5. Operator playbook for common incidents
|
|
|
|
### Worker failed
|
|
|
|
1. Check `/healthz` payload (`error_id`, `error_category`).
|
|
2. Locate matching error in logs.
|
|
3. If non-transient defect persists, roll back.
|
|
|
|
### Provider timeout spike
|
|
|
|
1. Confirm provider reachability and rate limits.
|
|
2. Review timeout frequency and impacted job volume.
|
|
3. If sustained, execute rollback criteria and notify stakeholders.
|
|
|
|
### Partial-success increase
|
|
|
|
1. Inspect affected `job_source` and `execution_attempt` records.
|
|
2. Confirm failures are category-aligned (`external`/`timeout`/`internal`).
|
|
3. Triage whether issue is source quality, provider, or runtime regression.
|
|
|
|
## 6. Dependency upgrade policy
|
|
|
|
Dependencies are declared in `pyproject.toml` and resolved through the committed
|
|
`uv.lock`. The lockfile guarantees reproducible installs; the version specifiers
|
|
control what a deliberate `uv lock --upgrade` is allowed to move.
|
|
|
|
### NiceGUI is pinned exactly (`nicegui==3.13.0`)
|
|
|
|
1. **Rationale.** NiceGUI bundles Quasar and Vue. Minor releases change component
|
|
props, slots, and styling, which surfaces as visual and interaction regressions
|
|
rather than import or type errors. The UI suite under `tests/ui/` asserts
|
|
structure and behavior, not rendered appearance, so a NiceGUI bump can pass the
|
|
full test suite and still degrade the interface.
|
|
2. **Scope of risk.** All NiceGUI usage is confined to `src/transcription/ui/` and
|
|
uses only the public `nicegui.ui` and `nicegui.events` surfaces. The coupling is
|
|
shallow, so the pin is about release stability, not about unpicking deep
|
|
framework entanglement.
|
|
3. **Current stance.** Hold the exact pin through release stabilization. Do not
|
|
widen it as incidental cleanup, and do not let automated dependency updates move
|
|
it. This includes forgoing patch releases, which is the accepted cost.
|
|
4. **Revisiting.** Treat a NiceGUI upgrade as scheduled work with its own change
|
|
window: bump the pin deliberately, run `uv run pytest -m "not external"`, then
|
|
manually verify each page contract in `docs/ui/pages/` before accepting.
|
|
|
|
### All other dependencies
|
|
|
|
Declared with `>=` floors and moved by explicit `uv lock --upgrade`. Verify with
|
|
`uv run ruff check .`, `uv run ty check`, and `uv run pytest -q -m "not external"`
|
|
before committing a changed lockfile.
|
|
|