diff --git a/docs/step4.md b/docs/step4.md new file mode 100644 index 0000000..20810d2 --- /dev/null +++ b/docs/step4.md @@ -0,0 +1,190 @@ +## Step 4: `services/upload.py` + `worker.py` + +## Objective + +Implement the MVP upload and background processing pipeline so the system can: + +1. Save uploaded files into `UPLOAD_DIR` +2. Create `Document` + `Job(status="queued")` +3. Process queued jobs in a worker loop: + - `queued -> processing` + - invoke Step 3 transcription + - persist `Transcript` + - finalize as `transcribed` or `failed` + +This covers MVP Feature 1 + 2 and supports REQ-1, REQ-2, REQ-3, REQ-4, REQ-6. + +--- + +## Scope + +### In scope +- `src/transcription/services/upload.py` +- `src/transcription/worker.py` +- Upload persistence logic + job creation +- Worker polling and single-job processing lifecycle +- Tests for upload + worker behavior (deterministic, no live API) + +### Out of scope +- UI wiring (Step 5) +- Job/transcript pages (Step 5) +- External queue infrastructure +- Async DB/session architecture changes (post-MVP) + +--- + +## Planned Deliverables + +### Source files +- `src/transcription/services/upload.py` +- `src/transcription/worker.py` +- `src/transcription/services/__init__.py` (export updates as needed) + +### Tests +- `tests/services/test_upload.py` +- `tests/services/test_worker.py` *(recommended for mirror consistency)* + +--- + +## Design Decisions + +1. **Upload service owns file persistence + initial DB records** + - Save file first, then create `Document` + `Job(queued)`. + +2. **Worker owns job lifecycle transitions** + - Transition only in worker: + - `queued -> processing` + - `processing -> transcribed|failed` + +3. **Worker calls Step 3 service, not provider SDK directly** + - Use `transcribe_document_image(...)` from `services/transcription.py`. + +4. **Failure details are always persisted** + - On failure, persist transcript row with `text=None`, `error_detail=...`. + +5. **Worker loop is stoppable** + - Simple in-process loop with stop signal/event and poll interval. + +--- + +## Task-by-Task Execution Checklist + +## Phase A — Upload service (`services/upload.py`) + +- [ ] Create `src/transcription/services/upload.py` +- [ ] Add `UploadError` exception type +- [ ] Add result dataclass (e.g., `UploadJobResult`) with: + - [ ] `document_id` + - [ ] `job_id` + - [ ] `stored_path` + - [ ] `original_filename` +- [ ] Implement safe filename handling: + - [ ] basename normalization + - [ ] collision-safe naming (e.g., UUID prefix) + - [ ] reject path traversal patterns +- [ ] Implement extension validation (`.jpg/.jpeg/.png/.tif/.tiff/.pdf`) +- [ ] Implement empty-bytes validation +- [ ] Ensure `UPLOAD_DIR` exists (`mkdir(parents=True, exist_ok=True)`) +- [ ] Persist file bytes to disk +- [ ] Insert `Document` and `Job(status=queued)` in one transaction +- [ ] Return structured `UploadJobResult` +- [ ] Add logging for upload success/failure (no sensitive payload logging) + +--- + +## Phase B — Worker core (`worker.py`) + +- [ ] Create `src/transcription/worker.py` +- [ ] Implement `process_next_queued_job(...) -> bool` + - [ ] fetch oldest queued job + - [ ] return `False` if none found + - [ ] set job to `processing` + update timestamp + - [ ] call `transcribe_document_image(image_path=...)` + - [ ] success path: persist transcript text, mark `transcribed` + - [ ] failure path: persist error detail, mark `failed` + - [ ] return `True` when a job is processed +- [ ] Implement transcript upsert behavior (avoid duplicate unique `job_id` insert issues) +- [ ] Add structured logs for lifecycle transitions and outcomes + +--- + +## Phase C — Worker loop (`worker.py`) + +- [ ] Implement `run_worker_loop(...)` + - [ ] accepts stop signal/event + - [ ] configurable `poll_interval_seconds` + - [ ] repeatedly calls `process_next_queued_job` + - [ ] sleeps when no work + - [ ] exits cleanly on stop request +- [ ] Keep implementation single-process/simple for MVP assumptions + +--- + +## Phase D — Service exports + +- [ ] Update `src/transcription/services/__init__.py` exports + - [ ] include upload service symbols + - [ ] keep transcription exports intact + +--- + +## Phase E — Tests (scaffold -> fill) + +### E1 Scaffold +- [ ] Create `tests/services/test_upload.py` with class/method skeletons + one-line docstrings +- [ ] Create `tests/services/test_worker.py` with class/method skeletons + one-line docstrings +- [ ] Validate scaffold collection: + - [ ] `uv run pytest --collect-only -q` + +### E2 Fill + +#### Upload tests +- [ ] `test_create_upload_job_writes_file_and_creates_records` (`integration`) +- [ ] `test_create_upload_job_rejects_empty_bytes` (`unit`) +- [ ] `test_create_upload_job_rejects_unsupported_extension` (`unit`) +- [ ] `test_create_upload_job_uses_unique_stored_filename` (`integration`) +- [ ] `test_create_upload_job_sets_job_status_queued` (`integration`) + +#### Worker tests +- [ ] `test_process_next_queued_job_returns_false_when_queue_empty` (`integration`) +- [ ] `test_process_next_queued_job_transitions_processing_to_transcribed` (`integration`) +- [ ] `test_process_next_queued_job_persists_transcript_text_on_success` (`integration`) +- [ ] `test_process_next_queued_job_sets_failed_and_error_detail_on_failure` (`integration`) +- [ ] `test_process_next_queued_job_updates_existing_transcript_if_present` (`integration`) +- [ ] `test_run_worker_loop_stops_when_stop_event_is_set` (`unit`) + +### Marker usage +- [ ] Mark pure logic tests as `@pytest.mark.unit` +- [ ] Mark DB/service orchestration tests as `@pytest.mark.integration` +- [ ] Reuse existing `external` only for future live endpoint checks (not required in Step 4) + +--- + +## Phase F — Verification + +- [ ] `uv run pytest --collect-only -q` +- [ ] `uv run pytest tests/services/test_upload.py -q` +- [ ] `uv run pytest tests/services/test_worker.py -q` +- [ ] `uv run pytest -m "unit or integration" -q` +- [ ] `uv run pytest -q` + +--- + +## Implementation Guardrails + +- [ ] Do not couple worker directly to provider SDK internals +- [ ] Do not swallow exceptions silently +- [ ] Always persist terminal failure details +- [ ] Keep status transitions explicit and timestamped +- [ ] Keep default test suite deterministic and fast (no live network) + +--- + +## Definition of Done (Step 4) + +- [ ] Upload service persists file + creates `Document` and queued `Job` +- [ ] Worker processes queued jobs end-to-end via Step 3 service +- [ ] Success writes transcript text and final `transcribed` status +- [ ] Failure writes `error_detail` and final `failed` status +- [ ] Queue-empty worker call returns cleanly without error +- [ ] Full test suite passes with `uv run pytest -q`