# V1 Operations Runbook

This runbook provides day-2 operational procedures for the V1 baseline.

## Scope

Applies to:

- local/hosted V1 runtime
- SQLite-backed persistence
- in-process worker lifecycle
- OpenRouter provider integration

## Preconditions

- `.env` contains `OPENROUTER_API_KEY`
- app starts successfully
- `uploads/` and `prompts/` are writable
- health endpoint responds at `/healthz`

## Standard Startup Procedure

1. Start the app using the project-standard command.
2. Open `/healthz` and verify `{"status":"ok"}`.
3. Open `/ui/upload` and submit a small valid file.
4. Confirm job transitions from `queued` -> `processing` -> `transcribed` (or `failed` with detail).

## Standard Shutdown Procedure

1. Stop the application process.
2. Ensure no active process still holds the SQLite file.
3. If maintenance is planned, copy the DB file before edits:
   - `transcription.db` (or configured `DATABASE_URL` file path)

## Incident: Jobs Stuck In `processing`

### Symptoms

- Jobs remain `processing` for longer than provider timeout
- New uploads queue but do not complete
- provider usage increases but no terminal job state is visible

### Checks

1. Confirm app process is still running.
2. Confirm worker loop is active (startup logs include worker lifespan start).
3. Inspect recent app logs for:
   - `worker.process_job`
   - `error_id`
   - `category`
   - `job_id` / `document_id` / `source_id`
4. Verify provider credentials and provider status.

### Recovery

1. Restart the app to trigger stale-processing recovery.
2. On startup, app re-queues stale processing jobs based on timeout policy.
3. Re-check jobs page and confirm terminal state progression.
4. If persistent, capture logs + error IDs and move to deep investigation.

## Incident: Provider Authentication Failures

### Symptoms

- failures categorized as provider/auth
- jobs fail quickly with authentication guidance

### Recovery

1. Validate `OPENROUTER_API_KEY` value.
2. Restart app after updating env.
3. Re-run a small transcription to confirm recovery.

## Incident: Upload Failures

### Symptoms

- UI reports upload errors
- unsupported extension or empty payload

### Recovery

1. Validate file extension (`.jpg`, `.jpeg`, `.png`, `.tif`, `.tiff`, `.pdf`).
2. Validate file is not empty.
3. Validate upload directory permissions.
4. Retry upload.

## Incident: Database File/Permission Issues

### Symptoms

- persistence errors during upload/job update
- startup failures around schema/runtime

### Recovery

1. Confirm the configured DB file path exists and is writable.
2. Confirm parent directory permissions.
3. Restore from last known backup copy if corruption is suspected.
4. Restart app and run smoke test.

## Logging Requirements (Operational)

Operational triage should always capture:

- `error_id`
- category
- operation name
- `job_id`, `document_id`, `source_id` when applicable
- UTC timestamp

## Escalation Packet (When opening an issue)

Include:

- exact timestamp window
- one failing `job_id`
- relevant `error_id` values
- latest 100 lines of app logs
- environment summary (`DATABASE_URL` type, app version/commit)

## Post-Incident Validation

After mitigation, verify:

1. Upload works.
2. One job reaches `transcribed`.
3. One induced failure reaches `failed` with error detail.
4. Jobs page and detail page render correctly.
