# Step 2 Implementation Plan: Error Handling & Reliability Hardening

## Purpose

Implement **Ver1 Step 2** from `docs/ver1/ver1.md` by standardizing failure behavior and reliability controls so the system fails safely, predictably, and transparently across UI, API, services, worker, and provider boundaries.

Primary governing docs:

- `docs/error_handling.md` (authoritative contract)
- `docs/requirements.md` (REQ-2, REQ-3, REQ-4, REQ-5, REQ-6)
- `docs/architecture.md` (boundary ownership and worker lifecycle)
- `docs/ver1/ver1.md` (Step 2 objective)

---

## MCP Skill and Guide Inputs Incorporated

This plan integrates guidance from john-stream-mcp resources:

1. `resource://skills/python-logging-dictconfig/document`
   - centralized `dictConfig` logging
   - startup-only configuration
   - stable named loggers and boundary-level logging discipline

2. `resource://skills/pytesting/document`
   - deterministic, behavior-first tests
   - explicit marker usage and fast/slow lane discipline
   - integration checks for boundary behavior and error contracts

3. `resource://skills/fastapi-async-sqlalchemy-modernization/document`
   - classify at source boundary
   - explicit transaction/session behavior under failure
   - phased rollout with quality gates and rollback awareness

4. `resource://skills/nicegui-ui-customization/document`
   - explicit user-facing error feedback for each interaction
   - prevent duplicate actions during in-flight operations
   - preserve one-way dependency boundaries from UI -> services

5. `resource://skills/fastapi-uv-docker/document` (applied selectively)
   - lifespan-safe startup/shutdown behavior
   - health/readiness posture and cloud-native operational checks

---

## Current-State Gap Summary

The project already has a strong baseline (`AppError`, taxonomy enum, API envelope, worker persistence), but Step 2 needs completion-level hardening:

1. **Error contract consistency**
   - API envelope exists, but consistency must be verified for all error pathways.
2. **Cross-boundary category normalization**
   - Provider/service/worker mappings exist, but require stricter policy checks and tests.
3. **Retry policy implementation depth**
   - Step 2 requires bounded retry policy and clear terminal behavior for retriable failures.
4. **Operational traceability**
   - Logging exists; Step 2 requires consistent structured fields at critical boundaries.
5. **UI failure UX consistency**
   - UI error handling exists; Step 2 requires explicit contract coverage and anti-duplication safeguards.

---

## Scope for Step 2

### In scope
1. Enforce canonical error taxonomy and envelope across all boundaries.
2. Standardize logging fields and boundary-level error traceability.
3. Implement/complete bounded retry and terminal failure behavior in worker paths.
4. Improve UI/API error presentation consistency and actionable guidance.
5. Add comprehensive Step 2 test coverage and verification matrix.
6. Update documentation to reflect final Step 2 policies and behavior.

### Out of scope
- Major architecture/topology changes (external queue, distributed worker)
- New end-user feature expansion outside reliability/error handling
- Full async ORM migration (unless required by bug fix)

---

## Target Decisions for Step 2

1. **Taxonomy stability is mandatory**
   - `ErrorCategory` values remain stable contract identifiers.
2. **Classification occurs at source boundary**
   - adapters/services normalize early; UI/API only present safely.
3. **User safety over internal detail leakage**
   - expose safe message + suggestion + error_id; keep sensitive detail in logs.
4. **Retry is explicit and bounded**
   - only retriable categories may retry; retries are capped; terminal failures persist reason.
5. **Boundary logs carry correlation fields**
   - include `error_id`, `category`, `operation`, and domain identifiers where available.

---

## Detailed Work Breakdown

## Phase A — Error Contract Audit and Policy Lock

- [ ] **A1. Build error-path inventory**
  - Enumerate all failure entry points across:
    - `api/`
    - `ui/`
    - `services/`
    - `worker.py`
    - `providers/`

- [ ] **A2. Produce taxonomy mapping table**
  - For each known exception path, map:
    - source exception type
    - target `ErrorCategory`
    - retriable flag
    - API status (if exposed)

- [ ] **A3. Reconcile with `docs/error_handling.md`**
  - Resolve any mismatch in category semantics, status codes, or suggested actions.

### Deliverables
- `docs/ver1/ver1-step2-audit.md` (recommended)
- taxonomy mapping table

### Exit Criteria
- Every known failure path has explicit category + retriable policy.

---

## Phase B — API and Service Contract Hardening

- [ ] **B1. Enforce API envelope completeness**
  - Ensure all API errors return:
    - `error_id`, `category`, `message`, `suggestion`, `timestamp`

- [ ] **B2. Verify category-to-status mapping consistency**
  - Confirm `api/errors.py` matches `docs/error_handling.md` mapping guidance.

- [ ] **B3. Normalize service exceptions at boundary**
  - Services should raise `AppError` subclasses for known failures.
  - Unknown exceptions must become `internal_unexpected_error` with traceable `error_id`.

- [ ] **B4. Ensure safe detail handling**
  - API/UI messages remain safe.
  - Diagnostic context remains in logs/persisted failure detail where appropriate.

### Exit Criteria
- No unstructured/unclassified exception escapes core boundaries.
- API responses are contract-stable for all tested failure modes.

---

## Phase C — Worker Retry and Terminal Failure Policy

- [ ] **C1. Define bounded retry policy**
  - Add configurable retry settings (attempt limit/backoff policy).
  - Limit retries to retriable categories.

- [ ] **C2. Implement terminal failure persistence**
  - On retry exhaustion, persist clear terminal reason and `error_id`.
  - Ensure job status transitions end deterministically at `failed`.

- [ ] **C3. Add duplicate-processing safety checks**
  - Prevent duplicate terminal updates when job already resolved.

- [ ] **C4. Validate worker lifecycle under repeated transient failures**
  - Ensure loop remains stable and responsive.

### Exit Criteria
- Retries are bounded and policy-driven.
- Exhausted retries produce deterministic failed state with evidence.

---

## Phase D — Logging and Observability Contract Enforcement

- [ ] **D1. Central logging conformance check**
  - Confirm startup-only `dictConfig` use remains canonical.
  - No module-level `basicConfig` use.

- [ ] **D2. Standardize error log fields**
  - Require at minimum when available:
    - `error_id`, `category`, `operation`, `exception_type`, `job_id`, `document_id`

- [ ] **D3. Boundary handoff logging**
  - Add/normalize logs at transitions:
    - UI action -> service
    - service -> provider/db
    - worker pickup -> terminal state

- [ ] **D4. Log noise control**
  - Avoid duplicate stack-trace logging across layers for same exception.

### Exit Criteria
- Critical failure events are traceable end-to-end via logs and `error_id`.

---

## Phase E — UI Error UX Consistency and Interaction Hardening

- [ ] **E1. Standardize user error presentation**
  - For upload/jobs interactions, ensure:
    - clear title
    - plain-language message
    - suggested action
    - visible error reference id

- [ ] **E2. Add in-flight interaction guards**
  - Prevent duplicate submits/click storms during pending operations.

- [ ] **E3. Ensure deterministic UI state recovery**
  - controls re-enable after failure
  - status text remains actionable

- [ ] **E4. Keep UI boundary clean**
  - no provider/protocol details leaked into page modules

### Exit Criteria
- All primary UI actions have consistent success/failure interaction behavior.

---

## Phase F — Test Expansion and Verification

Apply pytesting guidance: behavior-first assertions, deterministic fixtures, strict markers.

- [ ] **F1. API error contract tests**
  - verify envelope fields and status mapping for each category class.

- [ ] **F2. Service classification tests**
  - verify known failures map to expected `AppError` subclasses/categories.

- [ ] **F3. Worker retry policy tests**
  - retriable failure retries and eventual success
  - retriable failure exhaustion -> terminal failed
  - non-retriable failure -> immediate failed

- [ ] **F4. UI error behavior tests**
  - upload/jobs actions show actionable feedback on failures
  - duplicate action guard behavior

- [ ] **F5. Regression guard tests**
  - at least one test per previously observed production/real-world failure mode

### Validation Commands
- `uv run pytest --collect-only -q`
- `uv run pytest -m unit -q`
- `uv run pytest -m "not external" -q`
- `uv run pytest -q`

### Exit Criteria
- All Step 2 reliability/error contract tests pass.
- Existing suite remains green.

---

## Recommended Implementation Order

1. Phase A — audit and policy lock  
2. Phase B — API/service contract hardening  
3. Phase C — worker retry and terminal policy  
4. Phase D — logging/traceability normalization  
5. Phase E — UI consistency hardening  
6. Phase F — test expansion and full verification

This order reduces risk by locking policy first, then applying behavior changes at core boundaries before UI polish.

---

## Risks and Mitigations

1. **Risk:** Overly broad retry policy causes hidden failure loops  
   **Mitigation:** strict category-based retry eligibility + hard cap + terminal persistence.

2. **Risk:** User-facing messages become too technical  
   **Mitigation:** enforce safe message + suggestion contract in tests.

3. **Risk:** Logging becomes noisy/redundant  
   **Mitigation:** boundary logging rules and single-trace ownership.

4. **Risk:** Reliability work introduces regressions in happy path  
   **Mitigation:** run full suite continuously; preserve integration pipeline tests.

---

## Step 2 Completion Checklist

- [ ] Error taxonomy mapping table completed and approved.
- [ ] API envelope and HTTP status behavior verified for all relevant failure categories.
- [ ] Service/provider exception normalization is consistent and tested.
- [ ] Worker retry behavior is bounded, explicit, and terminal-state safe.
- [ ] Structured error logging fields are present at boundary handoffs.
- [ ] UI failure flows provide clear, actionable, and traceable feedback.
- [ ] Full test suite passes with new Step 2 coverage included.
- [ ] `docs/ver1/ver1-step2-results.md` created with evidence and residual risks.

---

## Handoff to Step 3

After Step 2 completion, Step 3 (Functional Completion by Requirement Domain) proceeds on a hardened foundation:

- stable failure contracts,
- predictable retries and terminal behavior,
- actionable user/API error semantics,
- improved diagnostic traceability.