V1 mostly complete except for some testing. Linting in the last step changed nearly every file which is why this commit is so larger.

2026-07-29 17:27:21 -05:00
parent bc21a97019
commit a3b3bab571
37 changed files with 1383 additions and 410 deletions
@@ -0,0 +1,129 @@
+# V1 Operations Runbook
+
+This runbook provides day-2 operational procedures for the V1 baseline.
+
+## Scope
+
+Applies to:
+
+- local/hosted V1 runtime
+- SQLite-backed persistence
+- in-process worker lifecycle
+- OpenRouter provider integration
+
+## Preconditions
+
+- `.env` contains `OPENROUTER_API_KEY`
+- app starts successfully
+- `uploads/` and `prompts/` are writable
+- health endpoint responds at `/healthz`
+
+## Standard Startup Procedure
+
+1. Start the app using the project-standard command.
+2. Open `/healthz` and verify `{"status":"ok"}`.
+3. Open `/ui/upload` and submit a small valid file.
+4. Confirm job transitions from `queued` -> `processing` -> `transcribed` (or `failed` with detail).
+
+## Standard Shutdown Procedure
+
+1. Stop the application process.
+2. Ensure no active process still holds the SQLite file.
+3. If maintenance is planned, copy the DB file before edits:
+   - `transcription.db` (or configured `DATABASE_URL` file path)
+
+## Incident: Jobs Stuck In `processing`
+
+### Symptoms
+
+- Jobs remain `processing` for longer than provider timeout
+- New uploads queue but do not complete
+- provider usage increases but no terminal job state is visible
+
+### Checks
+
+1. Confirm app process is still running.
+2. Confirm worker loop is active (startup logs include worker lifespan start).
+3. Inspect recent app logs for:
+   - `worker.process_job`
+   - `error_id`
+   - `category`
+   - `job_id` / `document_id` / `source_id`
+4. Verify provider credentials and provider status.
+
+### Recovery
+
+1. Restart the app to trigger stale-processing recovery.
+2. On startup, app re-queues stale processing jobs based on timeout policy.
+3. Re-check jobs page and confirm terminal state progression.
+4. If persistent, capture logs + error IDs and move to deep investigation.
+
+## Incident: Provider Authentication Failures
+
+### Symptoms
+
+- failures categorized as provider/auth
+- jobs fail quickly with authentication guidance
+
+### Recovery
+
+1. Validate `OPENROUTER_API_KEY` value.
+2. Restart app after updating env.
+3. Re-run a small transcription to confirm recovery.
+
+## Incident: Upload Failures
+
+### Symptoms
+
+- UI reports upload errors
+- unsupported extension or empty payload
+
+### Recovery
+
+1. Validate file extension (`.jpg`, `.jpeg`, `.png`, `.tif`, `.tiff`, `.pdf`).
+2. Validate file is not empty.
+3. Validate upload directory permissions.
+4. Retry upload.
+
+## Incident: Database File/Permission Issues
+
+### Symptoms
+
+- persistence errors during upload/job update
+- startup failures around schema/runtime
+
+### Recovery
+
+1. Confirm the configured DB file path exists and is writable.
+2. Confirm parent directory permissions.
+3. Restore from last known backup copy if corruption is suspected.
+4. Restart app and run smoke test.
+
+## Logging Requirements (Operational)
+
+Operational triage should always capture:
+
+- `error_id`
+- category
+- operation name
+- `job_id`, `document_id`, `source_id` when applicable
+- UTC timestamp
+
+## Escalation Packet (When opening an issue)
+
+Include:
+
+- exact timestamp window
+- one failing `job_id`
+- relevant `error_id` values
+- latest 100 lines of app logs
+- environment summary (`DATABASE_URL` type, app version/commit)
+
+## Post-Incident Validation
+
+After mitigation, verify:
+
+1. Upload works.
+2. One job reaches `transcribed`.
+3. One induced failure reaches `failed` with error detail.
+4. Jobs page and detail page render correctly.