error_handling.md added

2026-06-25 12:10:49 -05:00
parent 31ef94d4f5
commit 0cc6b0e1eb
5 changed files with 294 additions and 11 deletions
@@ -0,0 +1,282 @@
+# Error Handling
+
+This document defines the canonical error-handling policy for the document transcription system. It is the single source of truth for how errors are classified, surfaced to users, logged for diagnosis, and handled across UI, API, service, worker, and provider boundaries.
+
+## Error Handling Objectives
+
+The production error-handling model is designed to:
+
+- make failures visible to the user in clear, actionable language
+- preserve enough diagnostic detail for fast troubleshooting
+- keep module behavior consistent across all boundaries
+- distinguish expected domain failures from unexpected defects
+- support safe retries for transient failures without hiding persistent faults
+
+## Scope And Authority
+
+This page governs error-handling behavior for:
+
+- UI interactions (NiceGUI pages)
+- API endpoints (FastAPI routes)
+- application services and orchestration logic
+- in-process background worker execution
+- external provider adapters and persistence adapters
+
+If implementation behavior conflicts with this document, this document is authoritative and implementation should be updated.
+
+## Core Principles
+
+- **Clarity first:** user-facing messages should explain what failed in plain language.
+- **Actionability required:** each surfaced error should include a suggested next step.
+- **Safety by default:** internal details are logged; sensitive details are not exposed by default in UI/API.
+- **Consistency across boundaries:** category and structure should remain stable from source to surface.
+- **Fail explicitly:** silent failure is prohibited.
+- **Traceability:** every non-trivial error should be traceable with an error reference ID.
+
+## Error Taxonomy
+
+The system uses stable, implementation-independent categories:
+
+| Category | Definition | Typical Source | Retriable |
+| --- | --- | --- | --- |
+| `validation_error` | Payload or parameter shape/content is invalid | UI/API input validation, service guards | no |
+| `user_input_error` | User-provided artifact is unacceptable though structurally valid | unsupported file type, empty file, oversized upload | sometimes |
+| `not_found_error` | Requested resource does not exist | missing job/document/transcript | no |
+| `conflict_error` | Requested operation violates current state constraints | invalid state transition | no |
+| `external_provider_error` | External AI/provider call fails | upstream HTTP/API/provider failures | sometimes |
+| `infrastructure_transient_error` | Temporary environment issue | network timeout, DB connection reset | yes |
+| `infrastructure_persistent_error` | Non-transient environment issue | missing permissions, misconfiguration | no |
+| `internal_unexpected_error` | Unhandled defect or unknown failure | uncaught exceptions, logic errors | unknown (default no) |
+
+### Classification Rules
+
+- Classification occurs as close as possible to the origin boundary.
+- Provider-specific exceptions must be normalized into taxonomy categories before crossing service boundaries.
+- Unknown exceptions are classified as `internal_unexpected_error` and logged with traceback.
+- Category names are stable contracts and must not be changed casually.
+
+## User-Facing Error Experience Contract
+
+When an error is shown in the GUI, it must include:
+
+1. **Title** (short context, e.g., “Upload failed”)
+2. **Message** (plain-language explanation)
+3. **Suggested action** (explicit next step)
+4. **Error reference ID** (for support/debug traceability)
+5. **Technical details** (optional/collapsible for advanced users)
+
+### UI Message Rules
+
+- Do not expose raw stack traces by default.
+- Do not expose secrets, credentials, connection strings, or filesystem internals unless explicitly in debug tooling.
+- Prefer domain language over implementation language.
+- Use persistent visibility for important failures (dialog/card), not only transient toasts.
+
+### Suggested Action Requirements
+
+Every user-visible error must include a suggested course of action, such as:
+
+- retry the operation
+- check file type/size constraints
+- refresh the jobs page
+- verify environment configuration
+- contact operator with error ID and timestamp
+
+## API Error Response Contract
+
+API errors should return a structured envelope with stable fields:
+
+- `error_id`: short unique reference ID
+- `category`: taxonomy category
+- `message`: safe human-readable summary
+- `suggestion`: recommended next step
+- `details`: optional, only when safe and appropriate
+- `timestamp`: UTC ISO-8601
+
+HTTP status mapping guidance:
+
+- `validation_error`, `user_input_error` -> `400`
+- `not_found_error` -> `404`
+- `conflict_error` -> `409`
+- `external_provider_error` -> `502` or `503` (depending on failure mode)
+- `infrastructure_transient_error` -> `503`
+- `infrastructure_persistent_error` -> `500`
+- `internal_unexpected_error` -> `500`
+
+## Logging And Observability Contract
+
+All logged errors must include, where available:
+
+- `error_id`
+- `category`
+- `operation` (e.g., `upload.submit`, `worker.process_job`, `jobs.refresh`)
+- `exception_type`
+- `job_id`, `document_id` (when relevant)
+- UTC timestamp
+
+Rules:
+
+- Use structured logging fields where practical.
+- Use full traceback for unexpected errors (`internal_unexpected_error`).
+- Log at boundary handoff points to preserve causal trail.
+- Avoid duplicate noisy logging for the same exception at every layer.
+
+## Recovery And Retry Policy
+
+### Retriable Conditions
+
+Retriable failures include:
+
+- transient network/provider timeouts
+- intermittent provider unavailability
+- temporary DB/network interruptions
+
+### Non-Retriable Conditions
+
+Non-retriable failures include:
+
+- invalid file formats
+- missing required data
+- permission/configuration failures
+- deterministic domain conflicts
+
+### Worker Behavior
+
+- The worker must classify and persist failure details consistently.
+- Retries should be bounded by configured limits.
+- Exhausted retries must end in explicit failed status with recorded reason.
+- No infinite retry loops are allowed.
+
+## Boundary-Specific Responsibilities
+
+### UI Layer
+
+Responsibility:
+
+- display user-safe error summaries and suggested actions
+- show persistent error visibility for critical failures
+- include error reference IDs in visible output
+
+Out of scope:
+
+- low-level exception parsing
+- provider-specific protocol interpretation
+
+### API Layer
+
+Responsibility:
+
+- map application exceptions into stable error envelopes and HTTP statuses
+- preserve category and error_id continuity
+
+Out of scope:
+
+- domain-specific remediation logic
+
+### Service Layer
+
+Responsibility:
+
+- classify domain and infrastructure exceptions
+- convert adapter-specific failures into taxonomy categories
+- return deterministic error types to callers
+
+Out of scope:
+
+- presentation formatting for UI
+
+### Worker Layer
+
+Responsibility:
+
+- execute retry policy for retriable failures
+- persist terminal failure details for jobs
+- emit operational logs with category and identifiers
+
+Out of scope:
+
+- direct UI messaging
+
+### Provider Adapter Layer
+
+Responsibility:
+
+- normalize provider SDK/HTTP failures into domain-neutral exceptions
+- preserve raw provider context for logs (safely)
+
+Out of scope:
+
+- choosing user-facing wording
+
+## Error Lifecycle Workflow
+
+Standard lifecycle:
+
+1. Failure occurs at a boundary or operation.
+2. Exception is classified into taxonomy category.
+3. `error_id` is created (or propagated).
+4. Error is logged with required structured fields.
+5. User/API receives safe message + suggested action.
+6. Persistent job/resource state is updated when applicable.
+7. Tests verify contract behavior for the pathway.
+
+## Test Strategy For Error Handling
+
+### Unit Tests
+
+- category classification behavior
+- retry eligibility decisions
+- exception-to-message mapping safety
+
+### Integration Tests
+
+- UI pathways show clear message + suggested action for known failures
+- API returns structured error envelope with expected status/category
+- worker persists failed status and failure detail as required
+
+### Regression Tests
+
+- each previously observed production issue should have a guarding test
+- contract tests must cover adapter error normalization behavior
+
+## Known Failure Patterns And Prescribed Responses
+
+| Pattern | Category | User Message | Suggested Action |
+| --- | --- | --- | --- |
+| Upload payload cannot be parsed by UI handler | `internal_unexpected_error` (until narrowed) | Upload failed due to unexpected processing error | Retry once; if repeated, report error ID and check runtime version compatibility |
+| Unsupported extension | `user_input_error` | File type is not supported | Upload JPG, PNG, TIFF, or PDF |
+| Empty file upload | `validation_error` | Uploaded file is empty | Choose a valid non-empty file and retry |
+| Provider timeout | `external_provider_error` or `infrastructure_transient_error` | Transcription provider timed out | Retry from jobs page; if repeated, check provider status |
+| Job lookup missing | `not_found_error` | Requested job was not found | Refresh jobs list and open a valid job |
+
+## Governance And Update Process
+
+This document is a living policy artifact.
+
+Update this document when:
+
+- new error categories are introduced
+- handling behavior changes at any boundary
+- a production incident reveals missing guidance
+- API/UI error contracts change
+
+Change requirements:
+
+- update this document and associated tests in the same change set
+- preserve taxonomy stability; if changed, document migration impact
+- record noteworthy policy changes in project release notes or changelog
+
+## Related Pages
+
+- [System overview](index.md)
+- [Architecture](architecture.md)
+- [Requirements](requirements.md)
+- [Intent](intent.md)
+
+## Glossary
+
+- Error category: Stable classification used to drive handling, messaging, and status mapping.
+- Error envelope: Structured API payload describing a failure.
+- Error reference ID: Short identifier used to correlate user-visible failure with logs.
+- Retriable error: Failure likely to succeed on a later attempt without code changes.
+- Terminal failure: Failure state after retries are exhausted or retry is not allowed.
@@ -42,6 +42,7 @@ This operating model keeps deployment and maintenance simple while preserving cl
 - Architecture and technical design: [architecture.md](architecture.md)
 - Testing strategy and guidance: [tests.md](tests.md)
 - Runtime and deployment requirements: [requirements.md](requirements.md)
+- Error handling policy and operational guidance: [error_handling.md](error_handling.md)
 - Domain context and transcription policy: [intent.md](intent.md)

 ## Glossary
@@ -24,7 +24,7 @@ The MVP deliberately defers full-text search, export, revision history, MongoDB,
 | UI views for status and transcript reading | REQ-5 | The user needs to see what happened. |
 | Background processing to keep UI responsive | REQ-6 | Essential for usability during long AI calls. |
 | Centralized config and logging at startup | REQ-8 | Small effort, high payoff for debugging. |
-| Store transcription prompts as Markdown files | REQ-12 | Core to the Prompt Curation Policy in Intent.md. Start with a single prompt file. |
+| Store transcription prompts as Markdown files | REQ-12 | Core to the Prompt Curation Policy in intent.md. Start with a single prompt file. |


 ### Deferred to Post-MVP
@@ -52,7 +52,7 @@ The MVP deliberately defers full-text search, export, revision history, MongoDB,
    5. On failure: saves the error detail, transitions to failed.

 #### Feature 3: Transcription Prompt (Markdown Asset)
-* A single Markdown file (prompts/transcribe_document.md) encoding the verbatim transcription rules from Intent.md (the Document Issues table, scholarly guidelines, etc.).
+* A single Markdown file (prompts/transcribe_document.md) encoding the verbatim transcription rules from intent.md (the Document Issues table, scholarly guidelines, etc.).
 * The worker reads this file at invocation time and injects it as the system/user prompt.

 #### Feature 4: Job Status & Transcript Viewer (UI)
@@ -163,7 +163,7 @@ The MVP is considered validated when:
 1. ✅ A user can upload an image of a document through the browser.
 2. ✅ The system asynchronously sends the image to the configured AI vision model with the curated prompt.
 3. ✅ The transcript (or failure reason) is persisted and visible in the UI.
-4. ✅ The transcription follows verbatim scholarly rules defined in Intent.md (spot-checked by the user on real family documents).
+4. ✅ The transcription follows verbatim scholarly rules defined in intent.md (spot-checked by the user on real family documents).
 5. ✅ The transcription prompt is stored as a standalone Markdown file and can be edited without code changes.
 6. ✅ Job status transitions are visible: queued → processing → transcribed/failed.

@@ -175,7 +175,7 @@ These are the real unknowns this MVP exists to resolve:
 | #	| Question | How We Learn |
 | --- | --- | --- |
 | 1	| Is AI transcription quality good enough for this document corpus?	| User reviews 20–50 real transcriptions against originals. |
-| 2	| Does the verbatim prompt produce scholarly-quality output, or does it need major rework?	| Compare output to the Document Issues table rules in Intent.md. |
+| 2	| Does the verbatim prompt produce scholarly-quality output, or does it need major rework?	| Compare output to the Document Issues table rules in intent.md. |
 | 3	| What document types are hardest (old cursive, faded ink, pencil, postcards)?	| Track which uploads produce failed or low-quality results. |
 | 4	| Is single-image upload sufficient, or is batch upload needed early?	| User friction during real scanning sessions. |
 | 5	| What metadata is missing that the user wishes they could capture at upload time?	| User feedback after processing real batches. |
@@ -200,7 +200,7 @@ Recommended build order for the MVP (each step produces a testable increment):
 | Step | Deliverable | Validates |
 | --- | --- | --- |
 | 1 | config.py + models.py + db.py — data layer with SQLite | Schema and config foundation |
-| 2 | prompts/transcribe_document.md — curated prompt from Intent.md | Prompt asset pattern |
+| 2 | prompts/transcribe_document.md — curated prompt from intent.md | Prompt asset pattern |
 | 3 | services/transcription.py + providers/ — call AI vision provider with prompt + image | Core AI integration |
 | 4 | services/upload.py + worker.py — upload handling + background job loop | End-to-end pipeline (CLI-testable) |
 | 5 | ui/upload_page.py + ui/jobs_page.py — NiceGUI pages | User-facing interface |
@@ -9,7 +9,7 @@ Implement the MVP prompt artifact system by creating a curated transcription pro
 This step primarily satisfies:

 - **REQ-12**: prompts stored as individual Markdown artifacts
- MVP Feature 3: prompt-driven verbatim transcription behavior grounded in `docs/Intent.md`
+- MVP Feature 3: prompt-driven verbatim transcription behavior grounded in `docs/intent.md`

 ---

@@ -17,7 +17,7 @@ This step primarily satisfies:

 ### In scope
 1. Create prompt artifact directory and first prompt file.
-2. Encode transcription rules from `docs/Intent.md` into a model-facing prompt.
+2. Encode transcription rules from `docs/intent.md` into a model-facing prompt.
 3. Define stable prompt structure so future revisions are easy to diff/review.
 4. Add lightweight tests that validate artifact presence and baseline quality constraints.
 5. Update docs/README references so Step 3 can consume prompt file directly.
@@ -150,7 +150,7 @@ In Step 2, ensure docs reflect this and that Step 3 will resolve:
  - Preserve meaningful structure and reading order
  - No fabricated text

- [ ] **B3. Add Rule Set from `docs/Intent.md`**
+- [ ] **B3. Add Rule Set from `docs/intent.md`**
  - Misspellings/errors: `[sic]`
  - Missing words: `[word]`
  - Uncertain readings: `[guess?]`
@@ -237,7 +237,7 @@ In Step 2, ensure docs reflect this and that Step 3 will resolve:
 ## Done Criteria (quick gate)

 - [ ] Canonical prompt exists and is curated for verbatim transcription.
- [ ] Prompt encodes all high-value handling rules from `docs/Intent.md`.
+- [ ] Prompt encodes all high-value handling rules from `docs/intent.md`.
 - [ ] Prompt tests pass.
 - [ ] Full project tests pass with `uv`.
 - [ ] Ready for Step 3 provider integration.
@@ -249,7 +249,7 @@ In Step 2, ensure docs reflect this and that Step 3 will resolve:
 Step 2 is complete when all are true:

 1. `prompts/transcribe_document.md` exists and is committed.
-2. Prompt includes all critical handling rules from `docs/Intent.md`.
+2. Prompt includes all critical handling rules from `docs/intent.md`.
 3. Prompt is structured with stable section headings for future curation.
 4. Prompt tests pass under `uv run pytest -q`.
 5. Existing tests remain green (total suite still passes).
@@ -13,7 +13,7 @@ readme = "README.md"
 requires-python = ">=3.12"
 dependencies = [
    "fastapi>=0.138.0",
-    "nicegui>=3.13.0",
+    "nicegui==3.13.0",
    "openrouter>=0.7.0",
    "pydantic>=2.13.4",
    "pydantic-settings>=2.9.1",