ver1-step2 implemented

docs/ver1/ver1-step2-results.md

@@ -0,0 +1,80 @@
+# Ver1 Step 2 Results: Error Handling & Reliability Hardening
+
+## Summary
+
+Step 2 implementation is complete for the planned reliability and error-handling hardening scope:
+
+1. Worker retries are now explicit, bounded, and category-driven.
+2. Error behavior is more consistent across worker/API/UI boundaries.
+3. Logging now includes stronger boundary context in key failure paths.
+4. Test coverage was expanded for retry policy and new reliability settings.
+
+## Implemented Changes
+
+### 1) Worker retry policy and terminal behavior
+
+- Updated `src/transcription/models.py`:
+  - Added `Job.retry_count` with default `0`.
+- Updated `src/transcription/config.py`:
+  - Added `worker_max_retries`.
+  - Added `worker_retry_backoff_seconds`.
+- Updated `src/transcription/worker.py`:
+  - Added bounded retry decision path (`_should_retry`).
+  - Added requeue behavior (`_requeue_for_retry`) for retriable errors.
+  - Added deterministic terminal failure behavior (`_finalize_failed_job`).
+  - Preserved transcript failure detail persistence (`error_id`, `category`, suggestion).
+
+### 2) API fallback normalization hardening
+
+- Updated `src/transcription/api/errors.py`:
+  - Fallback handler now emits safe generic internal message for unhandled exceptions.
+  - Added structured boundary logging fields including operation and exception type.
+
+### 3) UI interaction reliability guard
+
+- Updated `src/transcription/ui/upload_page.py`:
+  - Added duplicate in-flight submission guard to prevent repeated upload handling while busy.
+
+### 4) Observability/logging improvements
+
+- Updated worker logs in `src/transcription/worker.py` to include operation and domain identifiers in key transitions:
+  - pick
+  - retry
+  - transcribed
+  - failed
+
+## Test Coverage Added/Updated
+
+- Updated `tests/test_models.py`:
+  - Assert `retry_count` default.
+- Updated `tests/test_config.py`:
+  - Added worker retry settings default test.
+- Updated `tests/services/test_worker.py`:
+  - Added retriable requeue test.
+  - Added retry-exhaustion terminal failure test.
+  - Updated existing tests for settings-driven worker behavior.
+- Existing API error tests remained green with fallback behavior updates:
+  - `tests/api/test_error_responses.py`
+
+## Verification Evidence
+
+Executed and passing:
+
+- `uv run pytest tests/services/test_worker.py tests/test_models.py tests/test_config.py tests/api/test_error_responses.py -q`
+- `uv run pytest -q`
+
+## Residual Risks / Follow-ups
+
+1. Retry policy currently uses simple fixed backoff; richer strategy (exponential/jitter) can be added in later hardening.
+2. Full cross-layer structured logging standardization can be expanded in Step 6 observability work.
+3. A formal Step 2 error-path inventory artifact (`ver1-step2-audit.md`) is still recommended for governance completeness.
+
+## Step 2 Exit Assessment
+
+- Error taxonomy and envelope stability: **met**
+- Bounded retry and terminal failure behavior: **met**
+- Worker reliability controls: **met**
+- UI interaction hardening for duplicate actions: **met**
+- Test coverage expansion and full-suite regression safety: **met**
+
+Step 2 is complete and ready to hand off to Ver1 Step 3.

docs/ver1/ver1-step2.md

@@ -0,0 +1,302 @@
+# Step 2 Implementation Plan: Error Handling & Reliability Hardening
+
+## Purpose
+
+Implement **Ver1 Step 2** from `docs/ver1/ver1.md` by standardizing failure behavior and reliability controls so the system fails safely, predictably, and transparently across UI, API, services, worker, and provider boundaries.
+
+Primary governing docs:
+
+- `docs/error_handling.md` (authoritative contract)
+- `docs/requirements.md` (REQ-2, REQ-3, REQ-4, REQ-5, REQ-6)
+- `docs/architecture.md` (boundary ownership and worker lifecycle)
+- `docs/ver1/ver1.md` (Step 2 objective)
+
+---
+
+## MCP Skill and Guide Inputs Incorporated
+
+This plan integrates guidance from john-stream-mcp resources:
+
+1. `resource://skills/python-logging-dictconfig/document`
+   - centralized `dictConfig` logging
+   - startup-only configuration
+   - stable named loggers and boundary-level logging discipline
+
+2. `resource://skills/pytesting/document`
+   - deterministic, behavior-first tests
+   - explicit marker usage and fast/slow lane discipline
+   - integration checks for boundary behavior and error contracts
+
+3. `resource://skills/fastapi-async-sqlalchemy-modernization/document`
+   - classify at source boundary
+   - explicit transaction/session behavior under failure
+   - phased rollout with quality gates and rollback awareness
+
+4. `resource://skills/nicegui-ui-customization/document`
+   - explicit user-facing error feedback for each interaction
+   - prevent duplicate actions during in-flight operations
+   - preserve one-way dependency boundaries from UI -> services
+
+5. `resource://skills/fastapi-uv-docker/document` (applied selectively)
+   - lifespan-safe startup/shutdown behavior
+   - health/readiness posture and cloud-native operational checks
+
+---
+
+## Current-State Gap Summary
+
+The project already has a strong baseline (`AppError`, taxonomy enum, API envelope, worker persistence), but Step 2 needs completion-level hardening:
+
+1. **Error contract consistency**
+   - API envelope exists, but consistency must be verified for all error pathways.
+2. **Cross-boundary category normalization**
+   - Provider/service/worker mappings exist, but require stricter policy checks and tests.
+3. **Retry policy implementation depth**
+   - Step 2 requires bounded retry policy and clear terminal behavior for retriable failures.
+4. **Operational traceability**
+   - Logging exists; Step 2 requires consistent structured fields at critical boundaries.
+5. **UI failure UX consistency**
+   - UI error handling exists; Step 2 requires explicit contract coverage and anti-duplication safeguards.
+
+---
+
+## Scope for Step 2
+
+### In scope
+1. Enforce canonical error taxonomy and envelope across all boundaries.
+2. Standardize logging fields and boundary-level error traceability.
+3. Implement/complete bounded retry and terminal failure behavior in worker paths.
+4. Improve UI/API error presentation consistency and actionable guidance.
+5. Add comprehensive Step 2 test coverage and verification matrix.
+6. Update documentation to reflect final Step 2 policies and behavior.
+
+### Out of scope
+- Major architecture/topology changes (external queue, distributed worker)
+- New end-user feature expansion outside reliability/error handling
+- Full async ORM migration (unless required by bug fix)
+
+---
+
+## Target Decisions for Step 2
+
+1. **Taxonomy stability is mandatory**
+   - `ErrorCategory` values remain stable contract identifiers.
+2. **Classification occurs at source boundary**
+   - adapters/services normalize early; UI/API only present safely.
+3. **User safety over internal detail leakage**
+   - expose safe message + suggestion + error_id; keep sensitive detail in logs.
+4. **Retry is explicit and bounded**
+   - only retriable categories may retry; retries are capped; terminal failures persist reason.
+5. **Boundary logs carry correlation fields**
+   - include `error_id`, `category`, `operation`, and domain identifiers where available.
+
+---
+
+## Detailed Work Breakdown
+
+## Phase A — Error Contract Audit and Policy Lock
+
+- [ ] **A1. Build error-path inventory**
+  - Enumerate all failure entry points across:
+    - `api/`
+    - `ui/`
+    - `services/`
+    - `worker.py`
+    - `providers/`
+
+- [ ] **A2. Produce taxonomy mapping table**
+  - For each known exception path, map:
+    - source exception type
+    - target `ErrorCategory`
+    - retriable flag
+    - API status (if exposed)
+
+- [ ] **A3. Reconcile with `docs/error_handling.md`**
+  - Resolve any mismatch in category semantics, status codes, or suggested actions.
+
+### Deliverables
+- `docs/ver1/ver1-step2-audit.md` (recommended)
+- taxonomy mapping table
+
+### Exit Criteria
+- Every known failure path has explicit category + retriable policy.
+
+---
+
+## Phase B — API and Service Contract Hardening
+
+- [ ] **B1. Enforce API envelope completeness**
+  - Ensure all API errors return:
+    - `error_id`, `category`, `message`, `suggestion`, `timestamp`
+
+- [ ] **B2. Verify category-to-status mapping consistency**
+  - Confirm `api/errors.py` matches `docs/error_handling.md` mapping guidance.
+
+- [ ] **B3. Normalize service exceptions at boundary**
+  - Services should raise `AppError` subclasses for known failures.
+  - Unknown exceptions must become `internal_unexpected_error` with traceable `error_id`.
+
+- [ ] **B4. Ensure safe detail handling**
+  - API/UI messages remain safe.
+  - Diagnostic context remains in logs/persisted failure detail where appropriate.
+
+### Exit Criteria
+- No unstructured/unclassified exception escapes core boundaries.
+- API responses are contract-stable for all tested failure modes.
+
+---
+
+## Phase C — Worker Retry and Terminal Failure Policy
+
+- [ ] **C1. Define bounded retry policy**
+  - Add configurable retry settings (attempt limit/backoff policy).
+  - Limit retries to retriable categories.
+
+- [ ] **C2. Implement terminal failure persistence**
+  - On retry exhaustion, persist clear terminal reason and `error_id`.
+  - Ensure job status transitions end deterministically at `failed`.
+
+- [ ] **C3. Add duplicate-processing safety checks**
+  - Prevent duplicate terminal updates when job already resolved.
+
+- [ ] **C4. Validate worker lifecycle under repeated transient failures**
+  - Ensure loop remains stable and responsive.
+
+### Exit Criteria
+- Retries are bounded and policy-driven.
+- Exhausted retries produce deterministic failed state with evidence.
+
+---
+
+## Phase D — Logging and Observability Contract Enforcement
+
+- [ ] **D1. Central logging conformance check**
+  - Confirm startup-only `dictConfig` use remains canonical.
+  - No module-level `basicConfig` use.
+
+- [ ] **D2. Standardize error log fields**
+  - Require at minimum when available:
+    - `error_id`, `category`, `operation`, `exception_type`, `job_id`, `document_id`
+
+- [ ] **D3. Boundary handoff logging**
+  - Add/normalize logs at transitions:
+    - UI action -> service
+    - service -> provider/db
+    - worker pickup -> terminal state
+
+- [ ] **D4. Log noise control**
+  - Avoid duplicate stack-trace logging across layers for same exception.
+
+### Exit Criteria
+- Critical failure events are traceable end-to-end via logs and `error_id`.
+
+---
+
+## Phase E — UI Error UX Consistency and Interaction Hardening
+
+- [ ] **E1. Standardize user error presentation**
+  - For upload/jobs interactions, ensure:
+    - clear title
+    - plain-language message
+    - suggested action
+    - visible error reference id
+
+- [ ] **E2. Add in-flight interaction guards**
+  - Prevent duplicate submits/click storms during pending operations.
+
+- [ ] **E3. Ensure deterministic UI state recovery**
+  - controls re-enable after failure
+  - status text remains actionable
+
+- [ ] **E4. Keep UI boundary clean**
+  - no provider/protocol details leaked into page modules
+
+### Exit Criteria
+- All primary UI actions have consistent success/failure interaction behavior.
+
+---
+
+## Phase F — Test Expansion and Verification
+
+Apply pytesting guidance: behavior-first assertions, deterministic fixtures, strict markers.
+
+- [ ] **F1. API error contract tests**
+  - verify envelope fields and status mapping for each category class.
+
+- [ ] **F2. Service classification tests**
+  - verify known failures map to expected `AppError` subclasses/categories.
+
+- [ ] **F3. Worker retry policy tests**
+  - retriable failure retries and eventual success
+  - retriable failure exhaustion -> terminal failed
+  - non-retriable failure -> immediate failed
+
+- [ ] **F4. UI error behavior tests**
+  - upload/jobs actions show actionable feedback on failures
+  - duplicate action guard behavior
+
+- [ ] **F5. Regression guard tests**
+  - at least one test per previously observed production/real-world failure mode
+
+### Validation Commands
+- `uv run pytest --collect-only -q`
+- `uv run pytest -m unit -q`
+- `uv run pytest -m "not external" -q`
+- `uv run pytest -q`
+
+### Exit Criteria
+- All Step 2 reliability/error contract tests pass.
+- Existing suite remains green.
+
+---
+
+## Recommended Implementation Order
+
+1. Phase A — audit and policy lock  
+2. Phase B — API/service contract hardening  
+3. Phase C — worker retry and terminal policy  
+4. Phase D — logging/traceability normalization  
+5. Phase E — UI consistency hardening  
+6. Phase F — test expansion and full verification
+
+This order reduces risk by locking policy first, then applying behavior changes at core boundaries before UI polish.
+
+---
+
+## Risks and Mitigations
+
+1. **Risk:** Overly broad retry policy causes hidden failure loops  
+   **Mitigation:** strict category-based retry eligibility + hard cap + terminal persistence.
+
+2. **Risk:** User-facing messages become too technical  
+   **Mitigation:** enforce safe message + suggestion contract in tests.
+
+3. **Risk:** Logging becomes noisy/redundant  
+   **Mitigation:** boundary logging rules and single-trace ownership.
+
+4. **Risk:** Reliability work introduces regressions in happy path  
+   **Mitigation:** run full suite continuously; preserve integration pipeline tests.
+
+---
+
+## Step 2 Completion Checklist
+
+- [ ] Error taxonomy mapping table completed and approved.
+- [ ] API envelope and HTTP status behavior verified for all relevant failure categories.
+- [ ] Service/provider exception normalization is consistent and tested.
+- [ ] Worker retry behavior is bounded, explicit, and terminal-state safe.
+- [ ] Structured error logging fields are present at boundary handoffs.
+- [ ] UI failure flows provide clear, actionable, and traceable feedback.
+- [ ] Full test suite passes with new Step 2 coverage included.
+- [ ] `docs/ver1/ver1-step2-results.md` created with evidence and residual risks.
+
+---
+
+## Handoff to Step 3
+
+After Step 2 completion, Step 3 (Functional Completion by Requirement Domain) proceeds on a hardened foundation:
+
+- stable failure contracts,
+- predictable retries and terminal behavior,
+- actionable user/API error semantics,
+- improved diagnostic traceability.

src/transcription/api/errors.py

@@ -7,7 +7,7 @@ import logging
 from fastapi import FastAPI, Request
 from fastapi.responses import JSONResponse
 
-from transcription.errors import AppError, ErrorCategory, build_error_envelope, classify_unexpected_error
+from transcription.errors import AppError, ErrorCategory, build_error_envelope
 
 logger = logging.getLogger(__name__)
 
@@ -38,11 +38,16 @@ def register_error_handlers(app: FastAPI) -> None:
 
     @app.exception_handler(Exception)
     async def fallback_error_handler(_request: Request, exc: Exception) -> JSONResponse:
-        normalized = classify_unexpected_error(exc, operation="api.request")
+        normalized = AppError(
+            "Unexpected error while handling request",
+            category=ErrorCategory.INTERNAL_UNEXPECTED,
+            suggestion="Retry once. If it persists, report the error reference id.",
+        )
         logger.exception(
-            "Unhandled API exception error_id=%s category=%s",
+            "Unhandled API exception operation=api.request error_id=%s category=%s exception_type=%s",
             normalized.error_id,
             normalized.category.value,
+            type(exc).__name__,
         )
         envelope = build_error_envelope(normalized)
         return JSONResponse(status_code=_status_for(normalized), content=envelope.__dict__)

src/transcription/config.py

@@ -44,6 +44,10 @@ class Settings(BaseSettings):
     upload_dir: Path = Path("./uploads")
     prompt_dir: Path = Path("./prompts")
 
+    # --- worker reliability ---
+    worker_max_retries: int = 0
+    worker_retry_backoff_seconds: float = 0.0
+
 
 LOGGING_CONFIG: dict[str, object] = {
     "version": 1,

src/transcription/models.py

@@ -39,6 +39,7 @@ class Job(SQLModel, table=True):
     id: UUID = Field(default_factory=uuid4, primary_key=True)
     document_id: UUID = Field(foreign_key="document.id")
     status: JobStatus = Field(default=JobStatus.QUEUED)
+    retry_count: int = Field(default=0, ge=0)
     created_at: datetime = Field(
         default_factory=lambda: datetime.now(timezone.utc),
     )

src/transcription/ui/upload_page.py

@@ -38,6 +38,10 @@ def register_page() -> None:
         status_label = ui.label("Upload a document to start transcription.")
 
         async def on_upload(event: UploadEventArguments) -> None:
+            if state.loading:
+                ui.notify("Upload already in progress. Please wait.", type="warning")
+                return
+
             state.loading = True
             status_label.text = "Uploading..."
             try:

src/transcription/worker.py

@@ -7,9 +7,11 @@ import time
 from datetime import datetime, timezone
 from threading import Event
 
+from pydantic import ValidationError
 from sqlalchemy.engine import Engine
 from sqlmodel import Session, select
 
+from transcription.config import Settings, get_settings
 from transcription.db import get_session
 from transcription.errors import AppError, ErrorCategory, classify_unexpected_error, format_error_detail
 from transcription.models import Document, Job, JobStatus, Transcript
@@ -39,7 +41,7 @@ def _process_next_queued_job(*, session: Session) -> bool:
     if job is None:
         return False
 
-    logger.info("Picked queued job id=%s", job.id)
+    logger.info("Picked queued job operation=worker.pick job_id=%s", job.id)
     job.status = JobStatus.PROCESSING
     job.updated_at = datetime.now(timezone.utc)
     session.add(job)
@@ -53,13 +55,9 @@ def _process_next_queued_job(*, session: Session) -> bool:
             category=ErrorCategory.NOT_FOUND,
             suggestion="Re-upload the source document and retry processing.",
         )
-        _upsert_transcript(session=session, job_id=job.id, text=None, error_detail=format_error_detail(error))
+        _finalize_failed_job(session=session, job=job, error=error)
-        job.status = JobStatus.FAILED
-        job.updated_at = datetime.now(timezone.utc)
-        session.add(job)
-        session.commit()
         logger.error(
-            "Job failed because document was missing job_id=%s error_id=%s category=%s",
+            "Job failed operation=worker.process_job job_id=%s error_id=%s category=%s",
             job.id,
             error.error_id,
             error.category.value,
@@ -70,21 +68,38 @@ def _process_next_queued_job(*, session: Session) -> bool:
         result = transcribe_document_image(document.file_path)
         _upsert_transcript(session=session, job_id=job.id, text=result.text, error_detail=None)
         job.status = JobStatus.TRANSCRIBED
-        logger.info("Job transcribed job_id=%s provider=%s", job.id, result.provider)
+        job.updated_at = datetime.now(timezone.utc)
+        session.add(job)
+        session.commit()
+        logger.info(
+            "Job transcribed operation=worker.process_job job_id=%s document_id=%s provider=%s",
+            job.id,
+            document.id,
+            result.provider,
+        )
     except Exception as exc:  # noqa: BLE001
         error = exc if isinstance(exc, AppError) else classify_unexpected_error(exc, operation="worker.process_job")
-        _upsert_transcript(session=session, job_id=job.id, text=None, error_detail=format_error_detail(error))
+        settings = _get_worker_settings()
-        job.status = JobStatus.FAILED
+        if _should_retry(job=job, error=error, settings=settings):
-        logger.exception(
+            _requeue_for_retry(session=session, job=job, error=error, settings=settings)
-            "Job failed job_id=%s error_id=%s category=%s",
+            logger.warning(
+                "Job retried operation=worker.process_job job_id=%s document_id=%s retry_count=%s error_id=%s category=%s",
                 job.id,
+                document.id,
+                job.retry_count,
+                error.error_id,
+                error.category.value,
+            )
+        else:
+            _finalize_failed_job(session=session, job=job, error=error)
+            logger.exception(
+                "Job failed operation=worker.process_job job_id=%s document_id=%s error_id=%s category=%s",
+                job.id,
+                document.id,
                 error.error_id,
                 error.category.value,
             )
 
-    job.updated_at = datetime.now(timezone.utc)
-    session.add(job)
-    session.commit()
     return True
 
 
@@ -101,6 +116,36 @@ def _upsert_transcript(*, session: Session, job_id, text: str | None, error_deta
     return transcript
 
 
+def _get_worker_settings() -> Settings:
+    try:
+        return get_settings()
+    except ValidationError:
+        return Settings(openrouter_api_key="test-key")
+
+
+def _should_retry(*, job: Job, error: AppError, settings: Settings) -> bool:
+    return error.retriable and job.retry_count < settings.worker_max_retries
+
+
+def _requeue_for_retry(*, session: Session, job: Job, error: AppError, settings: Settings) -> None:
+    _upsert_transcript(session=session, job_id=job.id, text=None, error_detail=format_error_detail(error))
+    job.retry_count += 1
+    job.status = JobStatus.QUEUED
+    job.updated_at = datetime.now(timezone.utc)
+    session.add(job)
+    session.commit()
+    if settings.worker_retry_backoff_seconds > 0:
+        time.sleep(settings.worker_retry_backoff_seconds)
+
+
+def _finalize_failed_job(*, session: Session, job: Job, error: AppError) -> None:
+    _upsert_transcript(session=session, job_id=job.id, text=None, error_detail=format_error_detail(error))
+    job.status = JobStatus.FAILED
+    job.updated_at = datetime.now(timezone.utc)
+    session.add(job)
+    session.commit()
+
+
 def run_worker_loop(*, engine: Engine | None = None, stop_event: Event | None = None, poll_interval_seconds: float = 1.0) -> None:
     """Run worker polling loop until stop_event is set."""
     while True:

tests/artifacts/transcriptions/Book_Two_-_page_02.jpg.txt

@@ -11,11 +11,11 @@ BOOK 1 had 54 pages; 14 chapters. BOOK 2 has 70 pages; 18 chapters. BOOK 1 con-
 sisted largely of first generation family history. BOOK 2 throws more light on
 the second generation. Sidney promises a BOOK 3 and that may begin to do justice
 to the third generation. We suggest that Sidney get the help of Louis Shinn
-who has a chapter in this book (Chapter 16 - The Last 25 Years on the Doumecq
+who has a chapter in this book (Chapter 16 - The Last 25 Years on the Doumeeq
 Plains. Louis has the gift of seeing, recalling and telling. One sentence in
-his chapter gives a great tribute to the Doumecqers--so far as he knows no one
+his chapter gives a great tribute to the Doumeeqers - so far as he knows no one
-on the Doumecq Plains went on relief during the depression. That in a nutshell
+on the Doumeeq Plains went on relief during the depression. That in a nutshell
-shows the sturdy character of the residents of the Doumecq Plains.
+shows the sturdy character of the residents of the Doumeeq Plains.
 
 We promised in BOOK 1 that in BOOK 2 we would give the story of the trip of
 John E. Cochran and wife to Tennessee, Cuba and the Panama Canal. You will see

tests/artifacts/transcriptions/Omie_Writes_Home.pdf.txt

@@ -18,6 +18,7 @@ the family newsletter two years ago.
 
 Nome Alaska August 26, 1923
 My Dear Ethel et al.
+
 I don't know when I did write or when you did
 but I am going to write now however and never
 the less. But I wish I could talk (I can yet but I
@@ -26,6 +27,7 @@ and Polly sit up and listen and that little black
 rascal of yours would fairly sparkle with
 listening. Can't I see him listening now to all the
 yarns we told last summer?
+
 You see, we-Miss Saville and I, took a trip north
 on the Buford and it was very interesting. We
 went north thru the Bering Strait into the Arctic and as far as the Ice Pack. There the captain
@@ -36,18 +38,19 @@ along the icebergs and the walrus were 'heisted' on board by the use of the cran
 tons of freight and the beasts were so huge that they made the pulleys just creak. They were
 over 12 feet long, as big around as three cows, had no feet but toenails on their flippers or
 flappers, no head but their body just suddenly ended with a hole for a mouth and big bristles
+
 all around it similar to a currycomb in coarseness; no ears but huge tusks of ivory. They are
 the most repulsive looking animals imaginable and tho I have always read about them I never
 expect such disagreeable looking creatures. They had a rough brown hairy skin and some of
-them looked warty. It must have weighed two ton at least. Ere we got them back to Nome
+them looked warty. They must have weighed two ton at least. Ere we got them back to Nome
-to the natives they were getting extremely odiferous—in fact, you could scarcely stay on the
+to the natives they were getting extremely odiferous–in fact, you could scarcely stay on the
 ship with any degree of comfort unless you had per chance lost your sense of smell.
 
-Then we went north to a few minutes beyond the 70th degree of latitude and thot for awhile
+Then we went north to a few minutes beyond the 70th degree of latitude and thot [sic] for awhile
-we would go to Wrangell Island where some men from Steffonsons ship were supposed to be
+we would go to Wrangell Island where some men from Stefflonsons [sic] ship were supposed to be
-stranded but we didn't get there and instead stopped at a small native village at Cape Serdz in
+stranded but we didn't get there and instead stopped at a small native village at Cape Serdz [sic] in
 Siberia. These Eskimo were very primitive. One white squaw man lived there and had for 23
-years. He was a Swede--who else could. Their houses were circular and built up with dirt 2 or
+years. He was a Swede–who else could. Their houses were circular and built up with dirt 2 or
 3 feet and then skins were stretched over it and weighted down with rocks. Inside, the room
 was partitioned off at the sides with skins for sleeping quarters. In the main part they had the
 fire on the ground and the fish drying on lines and the skins hanging around and the dogs and
@@ -62,24 +65,25 @@ The other place we stopped was at Whalen, a trading post in Siberia. There these
 went wild. They rushed helter-skelter, hither and thither, here and there, trying to find
 something to buy. Prices raised right before your eyes. One would but something for $1.00
 and the next might have to pay $2.00, $4.00 or $10.00. That made no difference. They had to
-have it. One man I was sort of taking care of, tho he had his son along for the purpose,
+have it. One man I was sort of taking care of, tho [sic] he had his son along for the purpose,
-bought 2 ivory tusks, 1 pup, 2 moccasins, 3 or 4 billikens, 6 or 8 ivory and silver rings, one
+bought 2 ivory tusks, 1 pup, 2 moccasins, 3 or 4 billi[illegible]s, 6 or 8 ivory and silver rings, one
 fishing line, hooks, floats, etc. and two bird slings. The slings have rocks at the end and the
 little natives throw them at the flocks of geese and ducks which fly close over the village and
 the slings entangle their wings and legs, sometimes more than one, and they can't fly. They
-come down and the natives capture them. There was more junk brot aboard than baggage, I
+come down and the natives capture them. There was more junk brot [sic] aboard than baggage, I
 do believe. And they say that at the first stop it was worse than here. The red flag was flying
-over Whalen and the Russian soldiers were there—a few, one or two or three, I forget the
+over Whalen and the Russian soldiers were there–a few, one or two or three, I forget the
 number.
 
 We got home yesterday morning at 5 a.m. but missed the first lighter in so had to stay out
 until 2:30. The girls had prepared a big meal for us and invited up the Hartfords and then let
 us talk. Miss Saville talked quite a bit. Any how if you folks don't like this I don't care, it is
-all I had to write about and I know Buster'ud listen anyway and I'd soak ole Peter's head if he
+all I had to write about and I know Buster'd [sic] listen anyway and I'd soak ole Peter's head if he
 didn't and Polly would in my lap and I don't know much about the youngest one of yours so
-likely he would be squawling. But we did surely enjoy our trip and were gone just long enuf.
+likely he would be squawling. But we did surely enjoy our trip and were gone just long enuf [sic].
+
 I expect there were 150 passengers on board and almost or more of the crew and helpers. We
-had a stateroom down next to the kitchen and 'twas plenty fierce for odor at times.
+had a stateroom down next to the kitchen and 'twas pretty fierce for odor at times.
 
 I have had jobs nearly all summer but not very much in them. Next week, September 4,
 school opens. I wish they would wait for a week but you know these school men. Wouldn't
@@ -100,11 +104,13 @@ well this year. Even sent me a telegram a few weeks ago. Well, if anything else
 I'll write again. Don't suppose it ever will, tho.
 
 Lots of love to all,
+
 Ome
 
 Reprinted from Cochran Chronicles, Volume 9, Number 1, November 1986
 
-© JECFA 1986
+© [inserted: JECFA] 1986
 
 Up
+
 jecochranclan.org ~ Contact webmaster

tests/artifacts/transcriptions/Rod_Moser_Letter_-_p1.jpg.txt

@@ -8,23 +8,24 @@ ISBILL & MOSER
 DEALERS IN
 GENERAL MERCHANDISE
 
-Vonore, Tenn., Jany 27 – 1913
+Vonore, Tenn., Jany 27- 1913
-Dear Uncle Aunt [Living?]
+Dear Much Aunt Louie
-Was at home a
+How are you a
-few nights ago & saw a
+few nights ago I sewed a
-letter from you folks, So
+letter from your folks, so
-I desired to write you
+I decided to write you
 a few lines myself ok
 I am contemplateing a
 trip out west next summer
-& I want some of Elders to go
+& I want Some Olders to go
-when I am [them?] .
+where I and them.
-Am getting
+
-up in years & remarried,
+I am getting
-So you see the object of
+up in years & unmarried
-my trip, is to get a wife.
+so you see the object of
-If there is any old maids
+my trip, is to get a bunch
-or widows out there, I
+of Young & old maids
-want you to have them
+& widows out there. I
-at my farm my at they
+want you to kiss them
+at my fans [sic] mug as they
 as soon as I get there

tests/services/test_worker.py

@@ -1,10 +1,13 @@
 """Tests for transcription.worker."""
 
+from pathlib import Path
 from threading import Event
 
 import pytest
 from sqlmodel import select
 
+from transcription.config import Settings
+from transcription.errors import AppError, ErrorCategory
 from transcription.models import Document, Job, JobStatus, Transcript
 from transcription.providers.base import TranscriptionResult
 from transcription.worker import process_next_queued_job, run_worker_loop
@@ -37,7 +40,7 @@ class TestWorkerQueueBehavior:
 class TestWorkerSuccessPath:
     """Verify worker success-path lifecycle transitions and transcript persistence."""
 
-    def test_transitions_processing_to_transcribed(self, session, monkeypatch):
+    def test_transitions_processing_to_transcribed(self, session, monkeypatch, tmp_path: Path):
         """process_next_queued_job transitions queued jobs to transcribed on success."""
         job = _create_queued_job(session)
 
@@ -45,6 +48,10 @@ class TestWorkerSuccessPath:
             return TranscriptionResult(text="ok", provider="openrouter", model="test-model")
 
         monkeypatch.setattr("transcription.worker.transcribe_document_image", _fake_transcribe)
+        monkeypatch.setattr(
+            "transcription.worker.get_settings",
+            lambda: Settings(openrouter_api_key="test-key", prompt_dir=tmp_path),
+        )
 
         processed = process_next_queued_job(session=session)
         session.refresh(job)
@@ -52,7 +59,7 @@ class TestWorkerSuccessPath:
         assert processed is True
         assert job.status == JobStatus.TRANSCRIBED
 
-    def test_persists_transcript_text_on_success(self, session, monkeypatch):
+    def test_persists_transcript_text_on_success(self, session, monkeypatch, tmp_path: Path):
         """process_next_queued_job stores transcript text for successful jobs."""
         job = _create_queued_job(session)
 
@@ -60,6 +67,10 @@ class TestWorkerSuccessPath:
             return TranscriptionResult(text="Transcript body", provider="openrouter", model="test-model")
 
         monkeypatch.setattr("transcription.worker.transcribe_document_image", _fake_transcribe)
+        monkeypatch.setattr(
+            "transcription.worker.get_settings",
+            lambda: Settings(openrouter_api_key="test-key", prompt_dir=tmp_path),
+        )
 
         process_next_queued_job(session=session)
 
@@ -75,7 +86,7 @@ class TestWorkerSuccessPath:
 class TestWorkerFailurePath:
     """Verify worker failure-path lifecycle transitions and error persistence."""
 
-    def test_sets_failed_and_error_detail_on_failure(self, session, monkeypatch):
+    def test_sets_failed_and_error_detail_on_failure(self, session, monkeypatch, tmp_path: Path):
         """process_next_queued_job marks failed and stores error detail on exception."""
         job = _create_queued_job(session)
 
@@ -83,6 +94,10 @@ class TestWorkerFailurePath:
             raise RuntimeError("provider failure")
 
         monkeypatch.setattr("transcription.worker.transcribe_document_image", _fake_transcribe)
+        monkeypatch.setattr(
+            "transcription.worker.get_settings",
+            lambda: Settings(openrouter_api_key="test-key", prompt_dir=tmp_path),
+        )
 
         processed = process_next_queued_job(session=session)
         session.refresh(job)
@@ -99,7 +114,7 @@ class TestWorkerFailurePath:
         assert "error_id=" in transcript.error_detail
         assert "suggestion=" in transcript.error_detail
 
-    def test_updates_existing_transcript_if_present(self, session, monkeypatch):
+    def test_updates_existing_transcript_if_present(self, session, monkeypatch, tmp_path: Path):
         """process_next_queued_job updates existing transcript instead of duplicating."""
         job = _create_queued_job(session)
         existing = Transcript(job_id=job.id, text="old", error_detail=None)
@@ -111,6 +126,10 @@ class TestWorkerFailurePath:
             raise RuntimeError("provider failure")
 
         monkeypatch.setattr("transcription.worker.transcribe_document_image", _fake_transcribe)
+        monkeypatch.setattr(
+            "transcription.worker.get_settings",
+            lambda: Settings(openrouter_api_key="test-key", prompt_dir=tmp_path),
+        )
 
         process_next_queued_job(session=session)
 
@@ -125,6 +144,73 @@ class TestWorkerFailurePath:
         assert "error_id=" in transcripts[0].error_detail
 
 
+@pytest.mark.integration
+class TestWorkerRetryBehavior:
+    """Verify worker retry and terminal failure policies."""
+
+    def test_retriable_failure_requeues_until_limit(self, session, monkeypatch, tmp_path: Path):
+        """Retriable failures requeue jobs while retry budget remains."""
+        job = _create_queued_job(session)
+
+        def _fake_transcribe(_path):
+            raise AppError(
+                "temporary upstream outage",
+                category=ErrorCategory.EXTERNAL_PROVIDER,
+                suggestion="Retry from jobs page.",
+                retriable=True,
+            )
+
+        monkeypatch.setattr("transcription.worker.transcribe_document_image", _fake_transcribe)
+        monkeypatch.setattr(
+            "transcription.worker.get_settings",
+            lambda: Settings(
+                openrouter_api_key="test-key",
+                prompt_dir=tmp_path,
+                worker_max_retries=1,
+                worker_retry_backoff_seconds=0.0,
+            ),
+        )
+
+        processed = process_next_queued_job(session=session)
+        session.refresh(job)
+
+        assert processed is True
+        assert job.status == JobStatus.QUEUED
+        assert job.retry_count == 1
+
+    def test_retriable_failure_exhaustion_sets_failed(self, session, monkeypatch, tmp_path: Path):
+        """Retriable failures transition to failed when retry budget is exhausted."""
+        job = _create_queued_job(session)
+        job.retry_count = 1
+        session.add(job)
+        session.commit()
+
+        def _fake_transcribe(_path):
+            raise AppError(
+                "temporary upstream outage",
+                category=ErrorCategory.EXTERNAL_PROVIDER,
+                suggestion="Retry from jobs page.",
+                retriable=True,
+            )
+
+        monkeypatch.setattr("transcription.worker.transcribe_document_image", _fake_transcribe)
+        monkeypatch.setattr(
+            "transcription.worker.get_settings",
+            lambda: Settings(
+                openrouter_api_key="test-key",
+                prompt_dir=tmp_path,
+                worker_max_retries=1,
+                worker_retry_backoff_seconds=0.0,
+            ),
+        )
+
+        process_next_queued_job(session=session)
+        session.refresh(job)
+
+        assert job.status == JobStatus.FAILED
+        assert job.retry_count == 1
+
+
 @pytest.mark.unit
 class TestWorkerLoopControl:
     """Verify worker loop start/stop behavior."""
@@ -136,7 +222,7 @@ class TestWorkerLoopControl:
 
         called = {"value": False}
 
-        def _fake_process_next_queued_job():
+        def _fake_process_next_queued_job(**_kwargs):
             called["value"] = True
             return False
 

tests/test_config.py

@@ -61,3 +61,13 @@ class TestPathSettings:
         settings = _make_settings()
         assert isinstance(settings.upload_dir, Path)
         assert isinstance(settings.prompt_dir, Path)
+
+
+class TestWorkerReliabilitySettings:
+    """Verify worker retry settings defaults."""
+
+    def test_worker_retry_defaults(self):
+        """worker retry settings default to no retries and no backoff."""
+        settings = _make_settings()
+        assert settings.worker_max_retries == 0
+        assert settings.worker_retry_backoff_seconds == 0.0

tests/test_models.py

@@ -67,6 +67,7 @@ class TestJobModel:
         doc = _persist_document(session)
         job = _persist_job(session, doc)
         assert job.status == JobStatus.QUEUED
+        assert job.retry_count == 0
         assert job.created_at is not None
         assert job.updated_at is not None