generated from john/python-template
130 lines
3.3 KiB
Markdown
130 lines
3.3 KiB
Markdown
# V1 Operations Runbook
|
|
|
|
This runbook provides day-2 operational procedures for the V1 baseline.
|
|
|
|
## Scope
|
|
|
|
Applies to:
|
|
|
|
- local/hosted V1 runtime
|
|
- SQLite-backed persistence
|
|
- in-process worker lifecycle
|
|
- OpenRouter provider integration
|
|
|
|
## Preconditions
|
|
|
|
- `.env` contains `OPENROUTER_API_KEY`
|
|
- app starts successfully
|
|
- `uploads/` and `prompts/` are writable
|
|
- health endpoint responds at `/healthz`
|
|
|
|
## Standard Startup Procedure
|
|
|
|
1. Start the app using the project-standard command.
|
|
2. Open `/healthz` and verify `{"status":"ok"}`.
|
|
3. Open `/ui/upload` and submit a small valid file.
|
|
4. Confirm job transitions from `queued` -> `processing` -> `transcribed` (or `failed` with detail).
|
|
|
|
## Standard Shutdown Procedure
|
|
|
|
1. Stop the application process.
|
|
2. Ensure no active process still holds the SQLite file.
|
|
3. If maintenance is planned, copy the DB file before edits:
|
|
- `transcription.db` (or configured `DATABASE_URL` file path)
|
|
|
|
## Incident: Jobs Stuck In `processing`
|
|
|
|
### Symptoms
|
|
|
|
- Jobs remain `processing` for longer than provider timeout
|
|
- New uploads queue but do not complete
|
|
- provider usage increases but no terminal job state is visible
|
|
|
|
### Checks
|
|
|
|
1. Confirm app process is still running.
|
|
2. Confirm worker loop is active (startup logs include worker lifespan start).
|
|
3. Inspect recent app logs for:
|
|
- `worker.process_job`
|
|
- `error_id`
|
|
- `category`
|
|
- `job_id` / `document_id` / `source_id`
|
|
4. Verify provider credentials and provider status.
|
|
|
|
### Recovery
|
|
|
|
1. Restart the app to trigger stale-processing recovery.
|
|
2. On startup, app re-queues stale processing jobs based on timeout policy.
|
|
3. Re-check jobs page and confirm terminal state progression.
|
|
4. If persistent, capture logs + error IDs and move to deep investigation.
|
|
|
|
## Incident: Provider Authentication Failures
|
|
|
|
### Symptoms
|
|
|
|
- failures categorized as provider/auth
|
|
- jobs fail quickly with authentication guidance
|
|
|
|
### Recovery
|
|
|
|
1. Validate `OPENROUTER_API_KEY` value.
|
|
2. Restart app after updating env.
|
|
3. Re-run a small transcription to confirm recovery.
|
|
|
|
## Incident: Upload Failures
|
|
|
|
### Symptoms
|
|
|
|
- UI reports upload errors
|
|
- unsupported extension or empty payload
|
|
|
|
### Recovery
|
|
|
|
1. Validate file extension (`.jpg`, `.jpeg`, `.png`, `.tif`, `.tiff`, `.pdf`).
|
|
2. Validate file is not empty.
|
|
3. Validate upload directory permissions.
|
|
4. Retry upload.
|
|
|
|
## Incident: Database File/Permission Issues
|
|
|
|
### Symptoms
|
|
|
|
- persistence errors during upload/job update
|
|
- startup failures around schema/runtime
|
|
|
|
### Recovery
|
|
|
|
1. Confirm the configured DB file path exists and is writable.
|
|
2. Confirm parent directory permissions.
|
|
3. Restore from last known backup copy if corruption is suspected.
|
|
4. Restart app and run smoke test.
|
|
|
|
## Logging Requirements (Operational)
|
|
|
|
Operational triage should always capture:
|
|
|
|
- `error_id`
|
|
- category
|
|
- operation name
|
|
- `job_id`, `document_id`, `source_id` when applicable
|
|
- UTC timestamp
|
|
|
|
## Escalation Packet (When opening an issue)
|
|
|
|
Include:
|
|
|
|
- exact timestamp window
|
|
- one failing `job_id`
|
|
- relevant `error_id` values
|
|
- latest 100 lines of app logs
|
|
- environment summary (`DATABASE_URL` type, app version/commit)
|
|
|
|
## Post-Incident Validation
|
|
|
|
After mitigation, verify:
|
|
|
|
1. Upload works.
|
|
2. One job reaches `transcribed`.
|
|
3. One induced failure reaches `failed` with error detail.
|
|
4. Jobs page and detail page render correctly.
|