transcription/docs/ver1/runbook_v1.md

V1 Operations Runbook

This runbook provides day-2 operational procedures for the V1 baseline.

Scope

Applies to:

• local/hosted V1 runtime
• SQLite-backed persistence
• in-process worker lifecycle
• OpenRouter provider integration

Preconditions

• .env contains OPENROUTER_API_KEY
• app starts successfully
• uploads/ and prompts/ are writable
• health endpoint responds at /healthz

Standard Startup Procedure

1. Start the app using the project-standard command.
2. Open /healthz and verify {"status":"ok"}.
3. Open /ui/upload and submit a small valid file.
4. Confirm job transitions from queued -> processing -> transcribed (or failed with detail).

Standard Shutdown Procedure

1. Stop the application process.
2. Ensure no active process still holds the SQLite file.
3. If maintenance is planned, copy the DB file before edits:
   • transcription.db (or configured DATABASE_URL file path)

Incident: Jobs Stuck In processing

Symptoms

• Jobs remain processing for longer than provider timeout
• New uploads queue but do not complete
• provider usage increases but no terminal job state is visible

Checks

1. Confirm app process is still running.
2. Confirm worker loop is active (startup logs include worker lifespan start).
3. Inspect recent app logs for:
   • worker.process_job
   • error_id
   • category
   • job_id / document_id / source_id
4. Verify provider credentials and provider status.

Recovery

1. Restart the app to trigger stale-processing recovery.
2. On startup, app re-queues stale processing jobs based on timeout policy.
3. Re-check jobs page and confirm terminal state progression.
4. If persistent, capture logs + error IDs and move to deep investigation.

Incident: Provider Authentication Failures

Symptoms

• failures categorized as provider/auth
• jobs fail quickly with authentication guidance

Recovery

1. Validate OPENROUTER_API_KEY value.
2. Restart app after updating env.
3. Re-run a small transcription to confirm recovery.

Incident: Upload Failures

Symptoms

• UI reports upload errors
• unsupported extension or empty payload

Recovery

1. Validate file extension (.jpg, .jpeg, .png, .tif, .tiff, .pdf).
2. Validate file is not empty.
3. Validate upload directory permissions.
4. Retry upload.

Incident: Database File/Permission Issues

Symptoms

• persistence errors during upload/job update
• startup failures around schema/runtime

Recovery

1. Confirm the configured DB file path exists and is writable.
2. Confirm parent directory permissions.
3. Restore from last known backup copy if corruption is suspected.
4. Restart app and run smoke test.

Logging Requirements (Operational)

Operational triage should always capture:

• error_id
• category
• operation name
• job_id, document_id, source_id when applicable
• UTC timestamp

Escalation Packet (When opening an issue)

Include:

• exact timestamp window
• one failing job_id
• relevant error_id values
• latest 100 lines of app logs
• environment summary (DATABASE_URL type, app version/commit)

Post-Incident Validation

After mitigation, verify:

1. Upload works.
2. One job reaches transcribed.
3. One induced failure reaches failed with error detail.
4. Jobs page and detail page render correctly.