diff --git a/docs/adr/ADR-0001-lifespan-owned-runtime-resources.md b/docs/adr/ADR-0001-lifespan-owned-runtime-resources.md deleted file mode 100644 index ed5be02..0000000 --- a/docs/adr/ADR-0001-lifespan-owned-runtime-resources.md +++ /dev/null @@ -1,35 +0,0 @@ -# ADR-0001: Lifespan-owned runtime resources - -- **Status:** accepted -- **Date:** 2026-06-25 - -## Context - -MVP initialized core runtime resources (database engine and worker dependencies) through module-level globals and startup side effects. `REQ-7` requires lifespan-owned runtime resources with explicit ownership and cleanup. - -## Decision - -Adopt lifespan-owned runtime resource initialization in `transcription.app`: - -1. Initialize database runtime during app lifespan startup. -2. Store runtime handles on `app.state`. -3. Pass runtime-owned dependencies (engine) to worker startup. -4. Dispose runtime resources explicitly during lifespan shutdown. - -## Consequences - -### Positive -- Explicit startup and shutdown ownership. -- Predictable cleanup ordering. -- Reduced hidden global side effects. - -### Tradeoffs -- Minor wiring complexity in app startup. -- Some call-sites still support fallback lazy initialization for compatibility. - -## Alternatives Considered - -1. **Keep module-level global ownership** - - Rejected: conflicts with `REQ-7` and increases ambiguity. -2. **Introduce full async DB stack immediately** - - Rejected for Step 1: too broad for architecture-consolidation scope. diff --git a/docs/adr/ADR-0002-explicit-schema-bootstrap-policy.md b/docs/adr/ADR-0002-explicit-schema-bootstrap-policy.md deleted file mode 100644 index 19e583c..0000000 --- a/docs/adr/ADR-0002-explicit-schema-bootstrap-policy.md +++ /dev/null @@ -1,36 +0,0 @@ -# ADR-0002: Explicit schema bootstrap policy - -- **Status:** accepted -- **Date:** 2026-06-25 - -## Context - -MVP called schema bootstrap (`create_all`) on every startup. `REQ-10` requires explicit, opt-in schema bootstrap behavior so normal production startup does not mutate schema. - -## Decision - -Add environment-aware bootstrap policy: - -1. New settings: - - `environment`: `development` | `test` | `production` - - `bootstrap_schema_on_startup`: optional explicit override -2. Default behavior: - - Development/test: bootstrap enabled - - Production: bootstrap disabled -3. App startup calls `create_all` only when policy evaluates true. - -## Consequences - -### Positive -- Production startup behavior is safer and policy-driven. -- Local development remains simple by default. - -### Tradeoffs -- Deployments now require explicit schema management in production. - -## Alternatives Considered - -1. **Always bootstrap in all environments** - - Rejected: violates `REQ-10` intent. -2. **Disable bootstrap everywhere immediately** - - Rejected: hurts local developer workflow without migration tool replacement yet. diff --git a/docs/adr/ADR-0003-persistence-baseline-and-transition-path.md b/docs/adr/ADR-0003-persistence-baseline-and-transition-path.md deleted file mode 100644 index e58c40c..0000000 --- a/docs/adr/ADR-0003-persistence-baseline-and-transition-path.md +++ /dev/null @@ -1,31 +0,0 @@ -# ADR-0003: Persistence baseline and transition path - -- **Status:** accepted -- **Date:** 2026-06-25 - -## Context - -Architecture targets PostgreSQL baseline (optional MongoDB), while MVP currently runs on SQLite by default. V1 needs a clear transition path without destabilizing ongoing work. - -## Decision - -1. Preserve database URL configurability through centralized settings. -2. Keep SQLite functional for local dev/test and fast feedback. -3. Treat PostgreSQL as production baseline target for V1 completion. -4. Keep persistence access behind `transcription.db` runtime/session access points. - -## Consequences - -### Positive -- Clear migration path without immediate broad rewrite. -- Controlled risk while preserving velocity. - -### Tradeoffs -- Temporary dual-path assumptions (SQLite local vs PostgreSQL target). - -## Alternatives Considered - -1. **Immediate forced PostgreSQL-only migration** - - Rejected: higher short-term disruption risk. -2. **Remain SQLite-only for V1** - - Rejected: inconsistent with architecture and requirement trajectory. diff --git a/docs/adr/ADR-0004-in-process-worker-topology.md b/docs/adr/ADR-0004-in-process-worker-topology.md deleted file mode 100644 index 767aae2..0000000 --- a/docs/adr/ADR-0004-in-process-worker-topology.md +++ /dev/null @@ -1,32 +0,0 @@ -# ADR-0004: In-process worker topology for V1 - -- **Status:** accepted -- **Date:** 2026-06-25 - -## Context - -The current system uses an in-process background worker. Architecture docs allow this in foundation stage and permit later hardening (optional external worker/queue). - -## Decision - -Retain in-process worker topology for V1, with improved lifecycle ownership: - -1. Worker starts/stops via app lifespan. -2. Worker receives runtime-owned DB engine dependency explicitly. -3. Extension path to external worker remains behind existing service/adapter seams. - -## Consequences - -### Positive -- Keeps operational complexity low for personal-scale use. -- Preserves delivery focus on V1 completion. - -### Tradeoffs -- Throughput/scaling limits remain compared to external queue-based topology. - -## Alternatives Considered - -1. **Immediate queue/external worker introduction** - - Rejected: premature complexity for current scale. -2. **Ad hoc thread lifecycle management outside lifespan** - - Rejected: weaker shutdown guarantees and poorer ownership clarity. diff --git a/docs/adr/README.md b/docs/adr/README.md deleted file mode 100644 index d2ffba1..0000000 --- a/docs/adr/README.md +++ /dev/null @@ -1,20 +0,0 @@ -# Architecture Decision Records (ADRs) - -This directory records significant architecture decisions for Version 1. - -## ADR Format - -Each ADR should include: - -1. **Status** (`proposed`, `accepted`, `superseded`) -2. **Context** -3. **Decision** -4. **Consequences** -5. **Alternatives Considered** - -## Index - -- [ADR-0001: Lifespan-owned runtime resources](ADR-0001-lifespan-owned-runtime-resources.md) -- [ADR-0002: Explicit schema bootstrap policy](ADR-0002-explicit-schema-bootstrap-policy.md) -- [ADR-0003: Persistence baseline and transition path](ADR-0003-persistence-baseline-and-transition-path.md) -- [ADR-0004: In-process worker topology for V1](ADR-0004-in-process-worker-topology.md) diff --git a/docs/ver1/ver1-step1-results.md b/docs/ver1/ver1-step1-results.md deleted file mode 100644 index c3bd8fe..0000000 --- a/docs/ver1/ver1-step1-results.md +++ /dev/null @@ -1,86 +0,0 @@ -# Ver1 Step 1 Results: Architecture Consolidation - -## Summary - -Step 1 implementation has been completed for the primary architecture-consolidation objectives: - -1. Lifespan-owned runtime resource model introduced for DB runtime ownership. -2. Schema bootstrap policy changed from implicit-always to explicit/environment-aware. -3. Worker startup now receives lifespan-owned DB engine dependency. -4. ADR set established for key V1 architectural decisions. - -## Implemented Changes - -### 1) Runtime ownership - -- Updated `src/transcription/db.py`: - - Added `DatabaseRuntime` resource model. - - Added explicit runtime lifecycle methods: - - `initialize_database_runtime(...)` - - `get_database_runtime()` - - `dispose_database_runtime()` -- Updated `src/transcription/app.py`: - - Lifespan initializes DB runtime and stores it on `app.state`. - - Lifespan disposes DB runtime on shutdown. - -### 2) Schema bootstrap policy (REQ-10 alignment) - -- Updated `src/transcription/config.py`: - - Added `environment` setting (`development`, `test`, `production`). - - Added `bootstrap_schema_on_startup` explicit override setting. -- Updated `src/transcription/db.py`: - - Added `should_bootstrap_schema(settings)` policy function. -- Updated `src/transcription/app.py`: - - Startup now calls `create_all(...)` only when policy allows. - -### 3) Worker dependency ownership - -- Updated `src/transcription/worker.py`: - - `process_next_queued_job(..., engine=None)` now supports explicit engine injection. - - `run_worker_loop(..., engine=None, ...)` now supports explicit engine injection. -- Updated `src/transcription/app.py`: - - Worker thread is started with lifespan-owned engine. - -### 4) ADR governance - -Created: -- `docs/adr/README.md` -- `docs/adr/ADR-0001-lifespan-owned-runtime-resources.md` -- `docs/adr/ADR-0002-explicit-schema-bootstrap-policy.md` -- `docs/adr/ADR-0003-persistence-baseline-and-transition-path.md` -- `docs/adr/ADR-0004-in-process-worker-topology.md` - -## Test Evidence - -Targeted regression checks executed successfully: - -- `uv run pytest tests/test_app.py tests/test_db.py tests/services/test_worker.py -q` -- Result: pass - -## Residual Risks / Follow-ups - -1. Full REQ-7 completion may still require broader runtime ownership coverage for additional resources as V1 expands. -2. Production schema management workflow (migrations/runbook tooling) should be finalized in subsequent V1 steps. -3. Additional boundary enforcement automation (import-lint style checks) can be added in later hardening. - -## Step 1 Exit Assessment - -- Architecture ownership clarity: **met** -- Schema bootstrap policy hardening: **met** -- Worker lifecycle dependency clarity: **met** -- ADR baseline established: **met** - -## Completion Checklist With Evidence - -| Criterion | Status | Evidence | -| --- | --- | --- | -| Architecture conformance matrix approved | partial | Consolidation implemented and documented in `docs/ver1/ver1-step1.md` + this results doc; formal matrix artifact can be added as a follow-up appendix. | -| REQ-7 ownership gaps resolved or explicitly deferred | met | Lifespan-owned DB runtime and explicit worker engine wiring implemented in `src/transcription/app.py`, `src/transcription/db.py`, `src/transcription/worker.py`. Residual scope documented under follow-ups. | -| REQ-10 explicit bootstrap policy implemented and verified | met | Policy implemented via `environment` + `bootstrap_schema_on_startup` in `src/transcription/config.py`, `should_bootstrap_schema(...)` in `src/transcription/db.py`, startup gate in `src/transcription/app.py`, tested in `tests/test_db.py`. | -| Dependency direction rules documented and enforced | partial | Layering and runtime ownership documented in `docs/architecture.md`. Lightweight enforcement exists via review and test discipline; automated import-lint remains a follow-up. | -| ADR set created for major Step 1 decisions | met | `docs/adr/README.md` and ADR-0001 through ADR-0004 created. | -| Architecture/index docs updated to match implementation | met | `docs/architecture.md` and `docs/index.md` updated with V1 Step 1 runtime policy and links to V1/ADR artifacts. | -| Regression and full test suites pass | met | Targeted: `uv run pytest tests/test_app.py tests/test_db.py tests/services/test_worker.py -q`; full suite: `uv run pytest -q`. | -| Step 1 results artifact published | met | This document (`docs/ver1/ver1-step1-results.md`) created and updated with summary, evidence, risks, and checklist. | - -Step 1 is complete and ready to hand off to Ver1 Step 2. diff --git a/docs/ver1/ver1-step1.md b/docs/ver1/ver1-step1.md deleted file mode 100644 index 56c9c0b..0000000 --- a/docs/ver1/ver1-step1.md +++ /dev/null @@ -1,309 +0,0 @@ -# Step 1 Implementation Plan: Architecture Consolidation - -## Purpose - -Align the implemented MVP codebase with the production architecture and V1 constraints documented in: - -- `docs/architecture.md` -- `docs/requirements.md` -- `docs/error_handling.md` -- `docs/index.md` -- `docs/intent.md` -- `docs/ver1/ver1.md` (Step 1) - -This step hardens architecture boundaries and ownership without expanding product scope. - ---- - -## MCP Skill and Guide Inputs Incorporated - -This plan explicitly incorporates patterns and guardrails from john-stream-mcp resources: - -1. `resource://skills/fastapi-uv-docker/document` - - App factory and lifespan ownership - - Health endpoint and cloud-native baseline expectations - - Environment-driven configuration and startup discipline - -2. `resource://skills/fastapi-async-sqlalchemy-modernization/document` - - Current-state gap audit first - - Target runtime model before refactor - - Explicit resource lifecycle ownership - - Transaction/session boundary clarity - - Phased migration with rollback points - -3. `resource://skills/nicegui/document` - - Clear dependency direction - - UI/page registration as composition, not business logic container - - Async responsiveness and boundary separation - -4. `resource://prompts/greenfield-architecture/document` - - Pattern-comparison-first planning - - Explicit tradeoffs and staged implementation - - Output contract with risks, open questions, and next steps - ---- - -## Current-State Gap Summary (Architecture vs Implementation) - -Based on docs and current `src/transcription` code: - -1. **REQ-7 gap (lifespan-owned resources)** - - DB engine/session factory are module globals in `db.py`, not app lifespan-owned. - - Worker thread lifecycle is owned by lifespan (good), but DB/provider resource ownership is mixed. - -2. **REQ-10 gap (explicit opt-in schema bootstrap)** - - `create_all()` is executed unconditionally on startup in `app.py`. - -3. **Data store target gap (REQ-9 + architecture baseline)** - - Runtime still defaults to SQLite MVP setup; production architecture targets PostgreSQL baseline with optional MongoDB. - -4. **Layering clarity gap (architecture layer model)** - - Boundaries exist but are not yet formally enforced (interface/app/domain/infra dependency rules are implicit, not codified). - -5. **Decision record gap** - - No ADR set documenting key V1 architectural decisions and deviations from MVP. - ---- - -## Scope for Step 1 - -### In scope -1. Produce architecture conformance audit and decision records. -2. Define and implement target runtime ownership model for core resources. -3. Establish explicit schema bootstrap policy (opt-in in production paths). -4. Consolidate module boundaries and dependency direction rules. -5. Update architecture docs to reflect implemented reality and V1 trajectory. - -### Out of scope -- Full async SQLAlchemy rewrite (plan and seams only if deferred) -- MongoDB feature implementation -- New user-facing features -- Major worker architecture replacement (in-process worker remains baseline) - ---- - -## Target Architecture Decisions for V1 - -1. **Keep modular monolith topology** (FastAPI + NiceGUI + in-process worker). -2. **Preserve container-light simplicity guardrails** from `architecture.md`. -3. **Move runtime ownership to lifespan** for: - - DB engine/session factory lifecycle - - Worker runtime resources - - Provider client factory/config lifecycle -4. **Adopt explicit schema bootstrap policy**: - - Dev/test: opt-in auto-bootstrap allowed - - Production: startup must not mutate schema implicitly -5. **Formalize boundary map**: - - Interface (`api`, `ui`) -> Application (`services`) -> Domain (`models/rules`) -> Infrastructure (`db`, `providers`) - - No reverse imports - ---- - -## Detailed Work Breakdown - -## Phase A — Architecture Audit and Baseline Freeze - -- [ ] **A1. Produce architecture conformance matrix** - - Map each architecture section to current modules/files. - - Classify each row: `aligned`, `partial`, `not aligned`. - -- [ ] **A2. Produce REQ-7/REQ-9/REQ-10 focused gap report** - - Explicitly capture current vs required state. - - Include operational risk if left unresolved. - -- [ ] **A3. Freeze MVP architecture baseline** - - Record current baseline behavior and known temporary shortcuts. - - Link this baseline from `docs/ver1/ver1.md`. - -### Deliverables -- `docs/ver1/ver1-step1-audit.md` (or equivalent section in this doc) -- Architecture conformance table - -### Exit Criteria -- No architecture changes begin before gap matrix and baseline are approved. - ---- - -## Phase B — Resource Ownership Consolidation (Lifespan-Centric) - -- [ ] **B1. Define runtime resource ownership contract** - - `app.py` lifespan owns resource initialization and cleanup order. - - `app.state` carries resource handles/factories. - - No hidden module-global side-effect initialization for runtime resources. - -- [ ] **B2. Refactor DB ownership model** - - Replace module-global engine singleton pattern with lifespan-initialized resource model. - - Define one canonical session-factory access path for app/worker/services. - -- [ ] **B3. Normalize worker dependencies** - - Ensure worker uses lifespan-owned resources/factories rather than implicit globals. - - Preserve deterministic startup/shutdown behavior. - -- [ ] **B4. Define provider adapter ownership** - - Provider client creation strategy is centralized and lifecycle-aware. - - Avoid per-call hidden client construction when unnecessary. - -### MCP-Guided Guardrails -- Use explicit lifecycle composition patterns from `fastapi-async-sqlalchemy-modernization`. -- Maintain app-factory + lifespan structure per `fastapi-uv-docker`. -- Keep UI registration as composition only per `nicegui`. - -### Exit Criteria -- Core runtime resources have one owner and one cleanup path. -- No critical resource has ambiguous ownership. - ---- - -## Phase C — Schema Bootstrap Policy (REQ-10 Alignment) - -- [ ] **C1. Define environment-aware bootstrap policy** - - `auto_create_schema` (or equivalent) disabled in production by default. - - Startup schema mutation is explicit and intentional. - -- [ ] **C2. Split startup responsibilities** - - App startup performs health-critical initialization only. - - Schema bootstrap path is moved to explicit command/flag workflow. - -- [ ] **C3. Update deployment/runbook docs** - - Document migration/bootstrap flow for dev, staging, prod. - - Ensure policy is testable and auditable. - -### Exit Criteria -- Normal production startup path does not call schema auto-create implicitly. -- Bootstrap behavior is explicit and documented. - ---- - -## Phase D — Module Boundary Enforcement - -- [ ] **D1. Publish dependency direction rules** - - Allowed import directions across `api`, `ui`, `services`, `models/domain`, `db/providers`. - - Explicitly disallow reverse dependencies. - -- [ ] **D2. Reconcile package map with docs** - - Ensure docs’ architecture elements match real package layout and naming. - - Update docs where intentional deviations remain. - -- [ ] **D3. Isolate cross-layer responsibilities** - - Keep API/UI presentation concerns out of services. - - Keep provider/DB specifics out of interface layer. - -- [ ] **D4. Add lightweight architecture checks** - - Add static/import checks and/or review checklist in CI/review process. - -### Exit Criteria -- Boundary rules are documented and applied. -- Architectural drift can be detected during review/CI. - ---- - -## Phase E — Architecture Decision Records (ADRs) - -- [ ] **E1. Create ADR index** - - Add `docs/adr/README.md` with template and status model. - -- [ ] **E2. Record minimum V1 ADR set** - 1. Runtime ownership model (lifespan-owned resources) - 2. Schema bootstrap policy (explicit vs implicit) - 3. Persistence baseline (PostgreSQL target; SQLite transition strategy) - 4. Worker topology (in-process for V1, extension path preserved) - -- [ ] **E3. Cross-link ADRs** - - Link from architecture and V1 docs. - -### Exit Criteria -- Major architecture decisions are explicit, versioned, and discoverable. - ---- - -## Phase F — Documentation Consolidation - -- [ ] **F1. Update `docs/architecture.md`** - - Reflect real implementation and V1 target state separately. - - Mark transitional choices clearly. - -- [ ] **F2. Update `docs/index.md` navigation consistency** - - Ensure architecture/readme references match actual docs/files. - -- [ ] **F3. Update `docs/requirements.md` traceability notes** - - Mark REQ-7/REQ-10 status and verification approach after consolidation. - -- [ ] **F4. Add Step 1 result summary** - - Create `docs/ver1/ver1-step1-results.md` after implementation. - -### Exit Criteria -- Docs are internally consistent and match runtime architecture reality. - ---- - -## Verification Plan - -## Architecture Verification Matrix (Step 1) - -1. **Inspection** - - Resource ownership map exists and matches code. - - Schema bootstrap policy is explicit and environment-aware. - - ADRs exist for each key architecture decision. - -2. **Automated checks** - - Existing test suite remains green. - - New/updated tests validate startup policy (no implicit schema mutation in production mode). - - Import/dependency-direction checks pass (if introduced). - -3. **Demonstration** - - App starts in dev mode with explicit expected behavior. - - App starts in production mode without mutating schema implicitly. - - Worker lifecycle starts/stops cleanly with app lifespan. - ---- - -## Risks and Mitigations - -1. **Risk:** Refactor destabilizes MVP behavior - **Mitigation:** Phase changes with small PRs and regression checks after each phase. - -2. **Risk:** Over-rotation into premature async rewrite - **Mitigation:** Keep this step focused on lifecycle ownership and boundaries; defer full async migration unless required. - -3. **Risk:** Schema policy changes break local DX - **Mitigation:** Keep explicit dev bootstrap path simple and documented. - -4. **Risk:** Boundary rules become “doc only” - **Mitigation:** Add CI/review enforcement and architecture checklist. - ---- - -## Recommended Implementation Order - -1. Phase A — Audit and baseline freeze -2. Phase B — Resource ownership consolidation -3. Phase C — Schema bootstrap policy -4. Phase D — Boundary enforcement -5. Phase E — ADR authoring -6. Phase F — Documentation consolidation - -This order minimizes risk: diagnose first, then refactor ownership, then lock policy, then enforce boundaries, and finally finalize docs. - ---- - -## Step 1 Completion Checklist - -- [ ] Architecture conformance matrix approved. -- [ ] REQ-7 ownership gaps resolved or explicitly deferred with owner/date. -- [ ] REQ-10 explicit bootstrap policy implemented and verified. -- [ ] Dependency direction rules documented and enforced. -- [ ] ADR set created for all major Step 1 decisions. -- [ ] Architecture and index docs updated to match implementation. -- [ ] Full test suite passes after consolidation. -- [ ] `docs/ver1/ver1-step1-results.md` created with evidence and residual risks. - ---- - -## Handoff to Step 2 - -Once Step 1 completes, Step 2 (Error Handling & Reliability Hardening) can proceed on stable architecture seams: - -- consistent lifecycle ownership, -- explicit startup policy, -- clear module boundaries, -- documented architecture decisions. \ No newline at end of file diff --git a/docs/ver1/ver1-step2-results.md b/docs/ver1/ver1-step2-results.md deleted file mode 100644 index 5eb8f02..0000000 --- a/docs/ver1/ver1-step2-results.md +++ /dev/null @@ -1,80 +0,0 @@ -# Ver1 Step 2 Results: Error Handling & Reliability Hardening - -## Summary - -Step 2 implementation is complete for the planned reliability and error-handling hardening scope: - -1. Worker retries are now explicit, bounded, and category-driven. -2. Error behavior is more consistent across worker/API/UI boundaries. -3. Logging now includes stronger boundary context in key failure paths. -4. Test coverage was expanded for retry policy and new reliability settings. - -## Implemented Changes - -### 1) Worker retry policy and terminal behavior - -- Updated `src/transcription/models.py`: - - Added `Job.retry_count` with default `0`. -- Updated `src/transcription/config.py`: - - Added `worker_max_retries`. - - Added `worker_retry_backoff_seconds`. -- Updated `src/transcription/worker.py`: - - Added bounded retry decision path (`_should_retry`). - - Added requeue behavior (`_requeue_for_retry`) for retriable errors. - - Added deterministic terminal failure behavior (`_finalize_failed_job`). - - Preserved transcript failure detail persistence (`error_id`, `category`, suggestion). - -### 2) API fallback normalization hardening - -- Updated `src/transcription/api/errors.py`: - - Fallback handler now emits safe generic internal message for unhandled exceptions. - - Added structured boundary logging fields including operation and exception type. - -### 3) UI interaction reliability guard - -- Updated `src/transcription/ui/upload_page.py`: - - Added duplicate in-flight submission guard to prevent repeated upload handling while busy. - -### 4) Observability/logging improvements - -- Updated worker logs in `src/transcription/worker.py` to include operation and domain identifiers in key transitions: - - pick - - retry - - transcribed - - failed - -## Test Coverage Added/Updated - -- Updated `tests/test_models.py`: - - Assert `retry_count` default. -- Updated `tests/test_config.py`: - - Added worker retry settings default test. -- Updated `tests/services/test_worker.py`: - - Added retriable requeue test. - - Added retry-exhaustion terminal failure test. - - Updated existing tests for settings-driven worker behavior. -- Existing API error tests remained green with fallback behavior updates: - - `tests/api/test_error_responses.py` - -## Verification Evidence - -Executed and passing: - -- `uv run pytest tests/services/test_worker.py tests/test_models.py tests/test_config.py tests/api/test_error_responses.py -q` -- `uv run pytest -q` - -## Residual Risks / Follow-ups - -1. Retry policy currently uses simple fixed backoff; richer strategy (exponential/jitter) can be added in later hardening. -2. Full cross-layer structured logging standardization can be expanded in Step 6 observability work. -3. A formal Step 2 error-path inventory artifact (`ver1-step2-audit.md`) is still recommended for governance completeness. - -## Step 2 Exit Assessment - -- Error taxonomy and envelope stability: **met** -- Bounded retry and terminal failure behavior: **met** -- Worker reliability controls: **met** -- UI interaction hardening for duplicate actions: **met** -- Test coverage expansion and full-suite regression safety: **met** - -Step 2 is complete and ready to hand off to Ver1 Step 3. diff --git a/docs/ver1/ver1-step2.md b/docs/ver1/ver1-step2.md deleted file mode 100644 index 4c26823..0000000 --- a/docs/ver1/ver1-step2.md +++ /dev/null @@ -1,302 +0,0 @@ -# Step 2 Implementation Plan: Error Handling & Reliability Hardening - -## Purpose - -Implement **Ver1 Step 2** from `docs/ver1/ver1.md` by standardizing failure behavior and reliability controls so the system fails safely, predictably, and transparently across UI, API, services, worker, and provider boundaries. - -Primary governing docs: - -- `docs/error_handling.md` (authoritative contract) -- `docs/requirements.md` (REQ-2, REQ-3, REQ-4, REQ-5, REQ-6) -- `docs/architecture.md` (boundary ownership and worker lifecycle) -- `docs/ver1/ver1.md` (Step 2 objective) - ---- - -## MCP Skill and Guide Inputs Incorporated - -This plan integrates guidance from john-stream-mcp resources: - -1. `resource://skills/python-logging-dictconfig/document` - - centralized `dictConfig` logging - - startup-only configuration - - stable named loggers and boundary-level logging discipline - -2. `resource://skills/pytesting/document` - - deterministic, behavior-first tests - - explicit marker usage and fast/slow lane discipline - - integration checks for boundary behavior and error contracts - -3. `resource://skills/fastapi-async-sqlalchemy-modernization/document` - - classify at source boundary - - explicit transaction/session behavior under failure - - phased rollout with quality gates and rollback awareness - -4. `resource://skills/nicegui-ui-customization/document` - - explicit user-facing error feedback for each interaction - - prevent duplicate actions during in-flight operations - - preserve one-way dependency boundaries from UI -> services - -5. `resource://skills/fastapi-uv-docker/document` (applied selectively) - - lifespan-safe startup/shutdown behavior - - health/readiness posture and cloud-native operational checks - ---- - -## Current-State Gap Summary - -The project already has a strong baseline (`AppError`, taxonomy enum, API envelope, worker persistence), but Step 2 needs completion-level hardening: - -1. **Error contract consistency** - - API envelope exists, but consistency must be verified for all error pathways. -2. **Cross-boundary category normalization** - - Provider/service/worker mappings exist, but require stricter policy checks and tests. -3. **Retry policy implementation depth** - - Step 2 requires bounded retry policy and clear terminal behavior for retriable failures. -4. **Operational traceability** - - Logging exists; Step 2 requires consistent structured fields at critical boundaries. -5. **UI failure UX consistency** - - UI error handling exists; Step 2 requires explicit contract coverage and anti-duplication safeguards. - ---- - -## Scope for Step 2 - -### In scope -1. Enforce canonical error taxonomy and envelope across all boundaries. -2. Standardize logging fields and boundary-level error traceability. -3. Implement/complete bounded retry and terminal failure behavior in worker paths. -4. Improve UI/API error presentation consistency and actionable guidance. -5. Add comprehensive Step 2 test coverage and verification matrix. -6. Update documentation to reflect final Step 2 policies and behavior. - -### Out of scope -- Major architecture/topology changes (external queue, distributed worker) -- New end-user feature expansion outside reliability/error handling -- Full async ORM migration (unless required by bug fix) - ---- - -## Target Decisions for Step 2 - -1. **Taxonomy stability is mandatory** - - `ErrorCategory` values remain stable contract identifiers. -2. **Classification occurs at source boundary** - - adapters/services normalize early; UI/API only present safely. -3. **User safety over internal detail leakage** - - expose safe message + suggestion + error_id; keep sensitive detail in logs. -4. **Retry is explicit and bounded** - - only retriable categories may retry; retries are capped; terminal failures persist reason. -5. **Boundary logs carry correlation fields** - - include `error_id`, `category`, `operation`, and domain identifiers where available. - ---- - -## Detailed Work Breakdown - -## Phase A — Error Contract Audit and Policy Lock - -- [ ] **A1. Build error-path inventory** - - Enumerate all failure entry points across: - - `api/` - - `ui/` - - `services/` - - `worker.py` - - `providers/` - -- [ ] **A2. Produce taxonomy mapping table** - - For each known exception path, map: - - source exception type - - target `ErrorCategory` - - retriable flag - - API status (if exposed) - -- [ ] **A3. Reconcile with `docs/error_handling.md`** - - Resolve any mismatch in category semantics, status codes, or suggested actions. - -### Deliverables -- `docs/ver1/ver1-step2-audit.md` (recommended) -- taxonomy mapping table - -### Exit Criteria -- Every known failure path has explicit category + retriable policy. - ---- - -## Phase B — API and Service Contract Hardening - -- [ ] **B1. Enforce API envelope completeness** - - Ensure all API errors return: - - `error_id`, `category`, `message`, `suggestion`, `timestamp` - -- [ ] **B2. Verify category-to-status mapping consistency** - - Confirm `api/errors.py` matches `docs/error_handling.md` mapping guidance. - -- [ ] **B3. Normalize service exceptions at boundary** - - Services should raise `AppError` subclasses for known failures. - - Unknown exceptions must become `internal_unexpected_error` with traceable `error_id`. - -- [ ] **B4. Ensure safe detail handling** - - API/UI messages remain safe. - - Diagnostic context remains in logs/persisted failure detail where appropriate. - -### Exit Criteria -- No unstructured/unclassified exception escapes core boundaries. -- API responses are contract-stable for all tested failure modes. - ---- - -## Phase C — Worker Retry and Terminal Failure Policy - -- [ ] **C1. Define bounded retry policy** - - Add configurable retry settings (attempt limit/backoff policy). - - Limit retries to retriable categories. - -- [ ] **C2. Implement terminal failure persistence** - - On retry exhaustion, persist clear terminal reason and `error_id`. - - Ensure job status transitions end deterministically at `failed`. - -- [ ] **C3. Add duplicate-processing safety checks** - - Prevent duplicate terminal updates when job already resolved. - -- [ ] **C4. Validate worker lifecycle under repeated transient failures** - - Ensure loop remains stable and responsive. - -### Exit Criteria -- Retries are bounded and policy-driven. -- Exhausted retries produce deterministic failed state with evidence. - ---- - -## Phase D — Logging and Observability Contract Enforcement - -- [ ] **D1. Central logging conformance check** - - Confirm startup-only `dictConfig` use remains canonical. - - No module-level `basicConfig` use. - -- [ ] **D2. Standardize error log fields** - - Require at minimum when available: - - `error_id`, `category`, `operation`, `exception_type`, `job_id`, `document_id` - -- [ ] **D3. Boundary handoff logging** - - Add/normalize logs at transitions: - - UI action -> service - - service -> provider/db - - worker pickup -> terminal state - -- [ ] **D4. Log noise control** - - Avoid duplicate stack-trace logging across layers for same exception. - -### Exit Criteria -- Critical failure events are traceable end-to-end via logs and `error_id`. - ---- - -## Phase E — UI Error UX Consistency and Interaction Hardening - -- [ ] **E1. Standardize user error presentation** - - For upload/jobs interactions, ensure: - - clear title - - plain-language message - - suggested action - - visible error reference id - -- [ ] **E2. Add in-flight interaction guards** - - Prevent duplicate submits/click storms during pending operations. - -- [ ] **E3. Ensure deterministic UI state recovery** - - controls re-enable after failure - - status text remains actionable - -- [ ] **E4. Keep UI boundary clean** - - no provider/protocol details leaked into page modules - -### Exit Criteria -- All primary UI actions have consistent success/failure interaction behavior. - ---- - -## Phase F — Test Expansion and Verification - -Apply pytesting guidance: behavior-first assertions, deterministic fixtures, strict markers. - -- [ ] **F1. API error contract tests** - - verify envelope fields and status mapping for each category class. - -- [ ] **F2. Service classification tests** - - verify known failures map to expected `AppError` subclasses/categories. - -- [ ] **F3. Worker retry policy tests** - - retriable failure retries and eventual success - - retriable failure exhaustion -> terminal failed - - non-retriable failure -> immediate failed - -- [ ] **F4. UI error behavior tests** - - upload/jobs actions show actionable feedback on failures - - duplicate action guard behavior - -- [ ] **F5. Regression guard tests** - - at least one test per previously observed production/real-world failure mode - -### Validation Commands -- `uv run pytest --collect-only -q` -- `uv run pytest -m unit -q` -- `uv run pytest -m "not external" -q` -- `uv run pytest -q` - -### Exit Criteria -- All Step 2 reliability/error contract tests pass. -- Existing suite remains green. - ---- - -## Recommended Implementation Order - -1. Phase A — audit and policy lock -2. Phase B — API/service contract hardening -3. Phase C — worker retry and terminal policy -4. Phase D — logging/traceability normalization -5. Phase E — UI consistency hardening -6. Phase F — test expansion and full verification - -This order reduces risk by locking policy first, then applying behavior changes at core boundaries before UI polish. - ---- - -## Risks and Mitigations - -1. **Risk:** Overly broad retry policy causes hidden failure loops - **Mitigation:** strict category-based retry eligibility + hard cap + terminal persistence. - -2. **Risk:** User-facing messages become too technical - **Mitigation:** enforce safe message + suggestion contract in tests. - -3. **Risk:** Logging becomes noisy/redundant - **Mitigation:** boundary logging rules and single-trace ownership. - -4. **Risk:** Reliability work introduces regressions in happy path - **Mitigation:** run full suite continuously; preserve integration pipeline tests. - ---- - -## Step 2 Completion Checklist - -- [ ] Error taxonomy mapping table completed and approved. -- [ ] API envelope and HTTP status behavior verified for all relevant failure categories. -- [ ] Service/provider exception normalization is consistent and tested. -- [ ] Worker retry behavior is bounded, explicit, and terminal-state safe. -- [ ] Structured error logging fields are present at boundary handoffs. -- [ ] UI failure flows provide clear, actionable, and traceable feedback. -- [ ] Full test suite passes with new Step 2 coverage included. -- [ ] `docs/ver1/ver1-step2-results.md` created with evidence and residual risks. - ---- - -## Handoff to Step 3 - -After Step 2 completion, Step 3 (Functional Completion by Requirement Domain) proceeds on a hardened foundation: - -- stable failure contracts, -- predictable retries and terminal behavior, -- actionable user/API error semantics, -- improved diagnostic traceability.