generated from john/python-template
pruning
This commit is contained in:
John Lancaster
@@ -1,35 +0,0 @@
|
||||
# ADR-0001: Lifespan-owned runtime resources
|
||||
|
||||
- **Status:** accepted
|
||||
- **Date:** 2026-06-25
|
||||
|
||||
## Context
|
||||
|
||||
MVP initialized core runtime resources (database engine and worker dependencies) through module-level globals and startup side effects. `REQ-7` requires lifespan-owned runtime resources with explicit ownership and cleanup.
|
||||
|
||||
## Decision
|
||||
|
||||
Adopt lifespan-owned runtime resource initialization in `transcription.app`:
|
||||
|
||||
1. Initialize database runtime during app lifespan startup.
|
||||
2. Store runtime handles on `app.state`.
|
||||
3. Pass runtime-owned dependencies (engine) to worker startup.
|
||||
4. Dispose runtime resources explicitly during lifespan shutdown.
|
||||
|
||||
## Consequences
|
||||
|
||||
### Positive
|
||||
- Explicit startup and shutdown ownership.
|
||||
- Predictable cleanup ordering.
|
||||
- Reduced hidden global side effects.
|
||||
|
||||
### Tradeoffs
|
||||
- Minor wiring complexity in app startup.
|
||||
- Some call-sites still support fallback lazy initialization for compatibility.
|
||||
|
||||
## Alternatives Considered
|
||||
|
||||
1. **Keep module-level global ownership**
|
||||
- Rejected: conflicts with `REQ-7` and increases ambiguity.
|
||||
2. **Introduce full async DB stack immediately**
|
||||
- Rejected for Step 1: too broad for architecture-consolidation scope.
|
||||
@@ -1,36 +0,0 @@
|
||||
# ADR-0002: Explicit schema bootstrap policy
|
||||
|
||||
- **Status:** accepted
|
||||
- **Date:** 2026-06-25
|
||||
|
||||
## Context
|
||||
|
||||
MVP called schema bootstrap (`create_all`) on every startup. `REQ-10` requires explicit, opt-in schema bootstrap behavior so normal production startup does not mutate schema.
|
||||
|
||||
## Decision
|
||||
|
||||
Add environment-aware bootstrap policy:
|
||||
|
||||
1. New settings:
|
||||
- `environment`: `development` | `test` | `production`
|
||||
- `bootstrap_schema_on_startup`: optional explicit override
|
||||
2. Default behavior:
|
||||
- Development/test: bootstrap enabled
|
||||
- Production: bootstrap disabled
|
||||
3. App startup calls `create_all` only when policy evaluates true.
|
||||
|
||||
## Consequences
|
||||
|
||||
### Positive
|
||||
- Production startup behavior is safer and policy-driven.
|
||||
- Local development remains simple by default.
|
||||
|
||||
### Tradeoffs
|
||||
- Deployments now require explicit schema management in production.
|
||||
|
||||
## Alternatives Considered
|
||||
|
||||
1. **Always bootstrap in all environments**
|
||||
- Rejected: violates `REQ-10` intent.
|
||||
2. **Disable bootstrap everywhere immediately**
|
||||
- Rejected: hurts local developer workflow without migration tool replacement yet.
|
||||
@@ -1,31 +0,0 @@
|
||||
# ADR-0003: Persistence baseline and transition path
|
||||
|
||||
- **Status:** accepted
|
||||
- **Date:** 2026-06-25
|
||||
|
||||
## Context
|
||||
|
||||
Architecture targets PostgreSQL baseline (optional MongoDB), while MVP currently runs on SQLite by default. V1 needs a clear transition path without destabilizing ongoing work.
|
||||
|
||||
## Decision
|
||||
|
||||
1. Preserve database URL configurability through centralized settings.
|
||||
2. Keep SQLite functional for local dev/test and fast feedback.
|
||||
3. Treat PostgreSQL as production baseline target for V1 completion.
|
||||
4. Keep persistence access behind `transcription.db` runtime/session access points.
|
||||
|
||||
## Consequences
|
||||
|
||||
### Positive
|
||||
- Clear migration path without immediate broad rewrite.
|
||||
- Controlled risk while preserving velocity.
|
||||
|
||||
### Tradeoffs
|
||||
- Temporary dual-path assumptions (SQLite local vs PostgreSQL target).
|
||||
|
||||
## Alternatives Considered
|
||||
|
||||
1. **Immediate forced PostgreSQL-only migration**
|
||||
- Rejected: higher short-term disruption risk.
|
||||
2. **Remain SQLite-only for V1**
|
||||
- Rejected: inconsistent with architecture and requirement trajectory.
|
||||
@@ -1,32 +0,0 @@
|
||||
# ADR-0004: In-process worker topology for V1
|
||||
|
||||
- **Status:** accepted
|
||||
- **Date:** 2026-06-25
|
||||
|
||||
## Context
|
||||
|
||||
The current system uses an in-process background worker. Architecture docs allow this in foundation stage and permit later hardening (optional external worker/queue).
|
||||
|
||||
## Decision
|
||||
|
||||
Retain in-process worker topology for V1, with improved lifecycle ownership:
|
||||
|
||||
1. Worker starts/stops via app lifespan.
|
||||
2. Worker receives runtime-owned DB engine dependency explicitly.
|
||||
3. Extension path to external worker remains behind existing service/adapter seams.
|
||||
|
||||
## Consequences
|
||||
|
||||
### Positive
|
||||
- Keeps operational complexity low for personal-scale use.
|
||||
- Preserves delivery focus on V1 completion.
|
||||
|
||||
### Tradeoffs
|
||||
- Throughput/scaling limits remain compared to external queue-based topology.
|
||||
|
||||
## Alternatives Considered
|
||||
|
||||
1. **Immediate queue/external worker introduction**
|
||||
- Rejected: premature complexity for current scale.
|
||||
2. **Ad hoc thread lifecycle management outside lifespan**
|
||||
- Rejected: weaker shutdown guarantees and poorer ownership clarity.
|
||||
@@ -1,20 +0,0 @@
|
||||
# Architecture Decision Records (ADRs)
|
||||
|
||||
This directory records significant architecture decisions for Version 1.
|
||||
|
||||
## ADR Format
|
||||
|
||||
Each ADR should include:
|
||||
|
||||
1. **Status** (`proposed`, `accepted`, `superseded`)
|
||||
2. **Context**
|
||||
3. **Decision**
|
||||
4. **Consequences**
|
||||
5. **Alternatives Considered**
|
||||
|
||||
## Index
|
||||
|
||||
- [ADR-0001: Lifespan-owned runtime resources](ADR-0001-lifespan-owned-runtime-resources.md)
|
||||
- [ADR-0002: Explicit schema bootstrap policy](ADR-0002-explicit-schema-bootstrap-policy.md)
|
||||
- [ADR-0003: Persistence baseline and transition path](ADR-0003-persistence-baseline-and-transition-path.md)
|
||||
- [ADR-0004: In-process worker topology for V1](ADR-0004-in-process-worker-topology.md)
|
||||
@@ -1,86 +0,0 @@
|
||||
# Ver1 Step 1 Results: Architecture Consolidation
|
||||
|
||||
## Summary
|
||||
|
||||
Step 1 implementation has been completed for the primary architecture-consolidation objectives:
|
||||
|
||||
1. Lifespan-owned runtime resource model introduced for DB runtime ownership.
|
||||
2. Schema bootstrap policy changed from implicit-always to explicit/environment-aware.
|
||||
3. Worker startup now receives lifespan-owned DB engine dependency.
|
||||
4. ADR set established for key V1 architectural decisions.
|
||||
|
||||
## Implemented Changes
|
||||
|
||||
### 1) Runtime ownership
|
||||
|
||||
- Updated `src/transcription/db.py`:
|
||||
- Added `DatabaseRuntime` resource model.
|
||||
- Added explicit runtime lifecycle methods:
|
||||
- `initialize_database_runtime(...)`
|
||||
- `get_database_runtime()`
|
||||
- `dispose_database_runtime()`
|
||||
- Updated `src/transcription/app.py`:
|
||||
- Lifespan initializes DB runtime and stores it on `app.state`.
|
||||
- Lifespan disposes DB runtime on shutdown.
|
||||
|
||||
### 2) Schema bootstrap policy (REQ-10 alignment)
|
||||
|
||||
- Updated `src/transcription/config.py`:
|
||||
- Added `environment` setting (`development`, `test`, `production`).
|
||||
- Added `bootstrap_schema_on_startup` explicit override setting.
|
||||
- Updated `src/transcription/db.py`:
|
||||
- Added `should_bootstrap_schema(settings)` policy function.
|
||||
- Updated `src/transcription/app.py`:
|
||||
- Startup now calls `create_all(...)` only when policy allows.
|
||||
|
||||
### 3) Worker dependency ownership
|
||||
|
||||
- Updated `src/transcription/worker.py`:
|
||||
- `process_next_queued_job(..., engine=None)` now supports explicit engine injection.
|
||||
- `run_worker_loop(..., engine=None, ...)` now supports explicit engine injection.
|
||||
- Updated `src/transcription/app.py`:
|
||||
- Worker thread is started with lifespan-owned engine.
|
||||
|
||||
### 4) ADR governance
|
||||
|
||||
Created:
|
||||
- `docs/adr/README.md`
|
||||
- `docs/adr/ADR-0001-lifespan-owned-runtime-resources.md`
|
||||
- `docs/adr/ADR-0002-explicit-schema-bootstrap-policy.md`
|
||||
- `docs/adr/ADR-0003-persistence-baseline-and-transition-path.md`
|
||||
- `docs/adr/ADR-0004-in-process-worker-topology.md`
|
||||
|
||||
## Test Evidence
|
||||
|
||||
Targeted regression checks executed successfully:
|
||||
|
||||
- `uv run pytest tests/test_app.py tests/test_db.py tests/services/test_worker.py -q`
|
||||
- Result: pass
|
||||
|
||||
## Residual Risks / Follow-ups
|
||||
|
||||
1. Full REQ-7 completion may still require broader runtime ownership coverage for additional resources as V1 expands.
|
||||
2. Production schema management workflow (migrations/runbook tooling) should be finalized in subsequent V1 steps.
|
||||
3. Additional boundary enforcement automation (import-lint style checks) can be added in later hardening.
|
||||
|
||||
## Step 1 Exit Assessment
|
||||
|
||||
- Architecture ownership clarity: **met**
|
||||
- Schema bootstrap policy hardening: **met**
|
||||
- Worker lifecycle dependency clarity: **met**
|
||||
- ADR baseline established: **met**
|
||||
|
||||
## Completion Checklist With Evidence
|
||||
|
||||
| Criterion | Status | Evidence |
|
||||
| --- | --- | --- |
|
||||
| Architecture conformance matrix approved | partial | Consolidation implemented and documented in `docs/ver1/ver1-step1.md` + this results doc; formal matrix artifact can be added as a follow-up appendix. |
|
||||
| REQ-7 ownership gaps resolved or explicitly deferred | met | Lifespan-owned DB runtime and explicit worker engine wiring implemented in `src/transcription/app.py`, `src/transcription/db.py`, `src/transcription/worker.py`. Residual scope documented under follow-ups. |
|
||||
| REQ-10 explicit bootstrap policy implemented and verified | met | Policy implemented via `environment` + `bootstrap_schema_on_startup` in `src/transcription/config.py`, `should_bootstrap_schema(...)` in `src/transcription/db.py`, startup gate in `src/transcription/app.py`, tested in `tests/test_db.py`. |
|
||||
| Dependency direction rules documented and enforced | partial | Layering and runtime ownership documented in `docs/architecture.md`. Lightweight enforcement exists via review and test discipline; automated import-lint remains a follow-up. |
|
||||
| ADR set created for major Step 1 decisions | met | `docs/adr/README.md` and ADR-0001 through ADR-0004 created. |
|
||||
| Architecture/index docs updated to match implementation | met | `docs/architecture.md` and `docs/index.md` updated with V1 Step 1 runtime policy and links to V1/ADR artifacts. |
|
||||
| Regression and full test suites pass | met | Targeted: `uv run pytest tests/test_app.py tests/test_db.py tests/services/test_worker.py -q`; full suite: `uv run pytest -q`. |
|
||||
| Step 1 results artifact published | met | This document (`docs/ver1/ver1-step1-results.md`) created and updated with summary, evidence, risks, and checklist. |
|
||||
|
||||
Step 1 is complete and ready to hand off to Ver1 Step 2.
|
||||
@@ -1,309 +0,0 @@
|
||||
# Step 1 Implementation Plan: Architecture Consolidation
|
||||
|
||||
## Purpose
|
||||
|
||||
Align the implemented MVP codebase with the production architecture and V1 constraints documented in:
|
||||
|
||||
- `docs/architecture.md`
|
||||
- `docs/requirements.md`
|
||||
- `docs/error_handling.md`
|
||||
- `docs/index.md`
|
||||
- `docs/intent.md`
|
||||
- `docs/ver1/ver1.md` (Step 1)
|
||||
|
||||
This step hardens architecture boundaries and ownership without expanding product scope.
|
||||
|
||||
---
|
||||
|
||||
## MCP Skill and Guide Inputs Incorporated
|
||||
|
||||
This plan explicitly incorporates patterns and guardrails from john-stream-mcp resources:
|
||||
|
||||
1. `resource://skills/fastapi-uv-docker/document`
|
||||
- App factory and lifespan ownership
|
||||
- Health endpoint and cloud-native baseline expectations
|
||||
- Environment-driven configuration and startup discipline
|
||||
|
||||
2. `resource://skills/fastapi-async-sqlalchemy-modernization/document`
|
||||
- Current-state gap audit first
|
||||
- Target runtime model before refactor
|
||||
- Explicit resource lifecycle ownership
|
||||
- Transaction/session boundary clarity
|
||||
- Phased migration with rollback points
|
||||
|
||||
3. `resource://skills/nicegui/document`
|
||||
- Clear dependency direction
|
||||
- UI/page registration as composition, not business logic container
|
||||
- Async responsiveness and boundary separation
|
||||
|
||||
4. `resource://prompts/greenfield-architecture/document`
|
||||
- Pattern-comparison-first planning
|
||||
- Explicit tradeoffs and staged implementation
|
||||
- Output contract with risks, open questions, and next steps
|
||||
|
||||
---
|
||||
|
||||
## Current-State Gap Summary (Architecture vs Implementation)
|
||||
|
||||
Based on docs and current `src/transcription` code:
|
||||
|
||||
1. **REQ-7 gap (lifespan-owned resources)**
|
||||
- DB engine/session factory are module globals in `db.py`, not app lifespan-owned.
|
||||
- Worker thread lifecycle is owned by lifespan (good), but DB/provider resource ownership is mixed.
|
||||
|
||||
2. **REQ-10 gap (explicit opt-in schema bootstrap)**
|
||||
- `create_all()` is executed unconditionally on startup in `app.py`.
|
||||
|
||||
3. **Data store target gap (REQ-9 + architecture baseline)**
|
||||
- Runtime still defaults to SQLite MVP setup; production architecture targets PostgreSQL baseline with optional MongoDB.
|
||||
|
||||
4. **Layering clarity gap (architecture layer model)**
|
||||
- Boundaries exist but are not yet formally enforced (interface/app/domain/infra dependency rules are implicit, not codified).
|
||||
|
||||
5. **Decision record gap**
|
||||
- No ADR set documenting key V1 architectural decisions and deviations from MVP.
|
||||
|
||||
---
|
||||
|
||||
## Scope for Step 1
|
||||
|
||||
### In scope
|
||||
1. Produce architecture conformance audit and decision records.
|
||||
2. Define and implement target runtime ownership model for core resources.
|
||||
3. Establish explicit schema bootstrap policy (opt-in in production paths).
|
||||
4. Consolidate module boundaries and dependency direction rules.
|
||||
5. Update architecture docs to reflect implemented reality and V1 trajectory.
|
||||
|
||||
### Out of scope
|
||||
- Full async SQLAlchemy rewrite (plan and seams only if deferred)
|
||||
- MongoDB feature implementation
|
||||
- New user-facing features
|
||||
- Major worker architecture replacement (in-process worker remains baseline)
|
||||
|
||||
---
|
||||
|
||||
## Target Architecture Decisions for V1
|
||||
|
||||
1. **Keep modular monolith topology** (FastAPI + NiceGUI + in-process worker).
|
||||
2. **Preserve container-light simplicity guardrails** from `architecture.md`.
|
||||
3. **Move runtime ownership to lifespan** for:
|
||||
- DB engine/session factory lifecycle
|
||||
- Worker runtime resources
|
||||
- Provider client factory/config lifecycle
|
||||
4. **Adopt explicit schema bootstrap policy**:
|
||||
- Dev/test: opt-in auto-bootstrap allowed
|
||||
- Production: startup must not mutate schema implicitly
|
||||
5. **Formalize boundary map**:
|
||||
- Interface (`api`, `ui`) -> Application (`services`) -> Domain (`models/rules`) -> Infrastructure (`db`, `providers`)
|
||||
- No reverse imports
|
||||
|
||||
---
|
||||
|
||||
## Detailed Work Breakdown
|
||||
|
||||
## Phase A — Architecture Audit and Baseline Freeze
|
||||
|
||||
- [ ] **A1. Produce architecture conformance matrix**
|
||||
- Map each architecture section to current modules/files.
|
||||
- Classify each row: `aligned`, `partial`, `not aligned`.
|
||||
|
||||
- [ ] **A2. Produce REQ-7/REQ-9/REQ-10 focused gap report**
|
||||
- Explicitly capture current vs required state.
|
||||
- Include operational risk if left unresolved.
|
||||
|
||||
- [ ] **A3. Freeze MVP architecture baseline**
|
||||
- Record current baseline behavior and known temporary shortcuts.
|
||||
- Link this baseline from `docs/ver1/ver1.md`.
|
||||
|
||||
### Deliverables
|
||||
- `docs/ver1/ver1-step1-audit.md` (or equivalent section in this doc)
|
||||
- Architecture conformance table
|
||||
|
||||
### Exit Criteria
|
||||
- No architecture changes begin before gap matrix and baseline are approved.
|
||||
|
||||
---
|
||||
|
||||
## Phase B — Resource Ownership Consolidation (Lifespan-Centric)
|
||||
|
||||
- [ ] **B1. Define runtime resource ownership contract**
|
||||
- `app.py` lifespan owns resource initialization and cleanup order.
|
||||
- `app.state` carries resource handles/factories.
|
||||
- No hidden module-global side-effect initialization for runtime resources.
|
||||
|
||||
- [ ] **B2. Refactor DB ownership model**
|
||||
- Replace module-global engine singleton pattern with lifespan-initialized resource model.
|
||||
- Define one canonical session-factory access path for app/worker/services.
|
||||
|
||||
- [ ] **B3. Normalize worker dependencies**
|
||||
- Ensure worker uses lifespan-owned resources/factories rather than implicit globals.
|
||||
- Preserve deterministic startup/shutdown behavior.
|
||||
|
||||
- [ ] **B4. Define provider adapter ownership**
|
||||
- Provider client creation strategy is centralized and lifecycle-aware.
|
||||
- Avoid per-call hidden client construction when unnecessary.
|
||||
|
||||
### MCP-Guided Guardrails
|
||||
- Use explicit lifecycle composition patterns from `fastapi-async-sqlalchemy-modernization`.
|
||||
- Maintain app-factory + lifespan structure per `fastapi-uv-docker`.
|
||||
- Keep UI registration as composition only per `nicegui`.
|
||||
|
||||
### Exit Criteria
|
||||
- Core runtime resources have one owner and one cleanup path.
|
||||
- No critical resource has ambiguous ownership.
|
||||
|
||||
---
|
||||
|
||||
## Phase C — Schema Bootstrap Policy (REQ-10 Alignment)
|
||||
|
||||
- [ ] **C1. Define environment-aware bootstrap policy**
|
||||
- `auto_create_schema` (or equivalent) disabled in production by default.
|
||||
- Startup schema mutation is explicit and intentional.
|
||||
|
||||
- [ ] **C2. Split startup responsibilities**
|
||||
- App startup performs health-critical initialization only.
|
||||
- Schema bootstrap path is moved to explicit command/flag workflow.
|
||||
|
||||
- [ ] **C3. Update deployment/runbook docs**
|
||||
- Document migration/bootstrap flow for dev, staging, prod.
|
||||
- Ensure policy is testable and auditable.
|
||||
|
||||
### Exit Criteria
|
||||
- Normal production startup path does not call schema auto-create implicitly.
|
||||
- Bootstrap behavior is explicit and documented.
|
||||
|
||||
---
|
||||
|
||||
## Phase D — Module Boundary Enforcement
|
||||
|
||||
- [ ] **D1. Publish dependency direction rules**
|
||||
- Allowed import directions across `api`, `ui`, `services`, `models/domain`, `db/providers`.
|
||||
- Explicitly disallow reverse dependencies.
|
||||
|
||||
- [ ] **D2. Reconcile package map with docs**
|
||||
- Ensure docs’ architecture elements match real package layout and naming.
|
||||
- Update docs where intentional deviations remain.
|
||||
|
||||
- [ ] **D3. Isolate cross-layer responsibilities**
|
||||
- Keep API/UI presentation concerns out of services.
|
||||
- Keep provider/DB specifics out of interface layer.
|
||||
|
||||
- [ ] **D4. Add lightweight architecture checks**
|
||||
- Add static/import checks and/or review checklist in CI/review process.
|
||||
|
||||
### Exit Criteria
|
||||
- Boundary rules are documented and applied.
|
||||
- Architectural drift can be detected during review/CI.
|
||||
|
||||
---
|
||||
|
||||
## Phase E — Architecture Decision Records (ADRs)
|
||||
|
||||
- [ ] **E1. Create ADR index**
|
||||
- Add `docs/adr/README.md` with template and status model.
|
||||
|
||||
- [ ] **E2. Record minimum V1 ADR set**
|
||||
1. Runtime ownership model (lifespan-owned resources)
|
||||
2. Schema bootstrap policy (explicit vs implicit)
|
||||
3. Persistence baseline (PostgreSQL target; SQLite transition strategy)
|
||||
4. Worker topology (in-process for V1, extension path preserved)
|
||||
|
||||
- [ ] **E3. Cross-link ADRs**
|
||||
- Link from architecture and V1 docs.
|
||||
|
||||
### Exit Criteria
|
||||
- Major architecture decisions are explicit, versioned, and discoverable.
|
||||
|
||||
---
|
||||
|
||||
## Phase F — Documentation Consolidation
|
||||
|
||||
- [ ] **F1. Update `docs/architecture.md`**
|
||||
- Reflect real implementation and V1 target state separately.
|
||||
- Mark transitional choices clearly.
|
||||
|
||||
- [ ] **F2. Update `docs/index.md` navigation consistency**
|
||||
- Ensure architecture/readme references match actual docs/files.
|
||||
|
||||
- [ ] **F3. Update `docs/requirements.md` traceability notes**
|
||||
- Mark REQ-7/REQ-10 status and verification approach after consolidation.
|
||||
|
||||
- [ ] **F4. Add Step 1 result summary**
|
||||
- Create `docs/ver1/ver1-step1-results.md` after implementation.
|
||||
|
||||
### Exit Criteria
|
||||
- Docs are internally consistent and match runtime architecture reality.
|
||||
|
||||
---
|
||||
|
||||
## Verification Plan
|
||||
|
||||
## Architecture Verification Matrix (Step 1)
|
||||
|
||||
1. **Inspection**
|
||||
- Resource ownership map exists and matches code.
|
||||
- Schema bootstrap policy is explicit and environment-aware.
|
||||
- ADRs exist for each key architecture decision.
|
||||
|
||||
2. **Automated checks**
|
||||
- Existing test suite remains green.
|
||||
- New/updated tests validate startup policy (no implicit schema mutation in production mode).
|
||||
- Import/dependency-direction checks pass (if introduced).
|
||||
|
||||
3. **Demonstration**
|
||||
- App starts in dev mode with explicit expected behavior.
|
||||
- App starts in production mode without mutating schema implicitly.
|
||||
- Worker lifecycle starts/stops cleanly with app lifespan.
|
||||
|
||||
---
|
||||
|
||||
## Risks and Mitigations
|
||||
|
||||
1. **Risk:** Refactor destabilizes MVP behavior
|
||||
**Mitigation:** Phase changes with small PRs and regression checks after each phase.
|
||||
|
||||
2. **Risk:** Over-rotation into premature async rewrite
|
||||
**Mitigation:** Keep this step focused on lifecycle ownership and boundaries; defer full async migration unless required.
|
||||
|
||||
3. **Risk:** Schema policy changes break local DX
|
||||
**Mitigation:** Keep explicit dev bootstrap path simple and documented.
|
||||
|
||||
4. **Risk:** Boundary rules become “doc only”
|
||||
**Mitigation:** Add CI/review enforcement and architecture checklist.
|
||||
|
||||
---
|
||||
|
||||
## Recommended Implementation Order
|
||||
|
||||
1. Phase A — Audit and baseline freeze
|
||||
2. Phase B — Resource ownership consolidation
|
||||
3. Phase C — Schema bootstrap policy
|
||||
4. Phase D — Boundary enforcement
|
||||
5. Phase E — ADR authoring
|
||||
6. Phase F — Documentation consolidation
|
||||
|
||||
This order minimizes risk: diagnose first, then refactor ownership, then lock policy, then enforce boundaries, and finally finalize docs.
|
||||
|
||||
---
|
||||
|
||||
## Step 1 Completion Checklist
|
||||
|
||||
- [ ] Architecture conformance matrix approved.
|
||||
- [ ] REQ-7 ownership gaps resolved or explicitly deferred with owner/date.
|
||||
- [ ] REQ-10 explicit bootstrap policy implemented and verified.
|
||||
- [ ] Dependency direction rules documented and enforced.
|
||||
- [ ] ADR set created for all major Step 1 decisions.
|
||||
- [ ] Architecture and index docs updated to match implementation.
|
||||
- [ ] Full test suite passes after consolidation.
|
||||
- [ ] `docs/ver1/ver1-step1-results.md` created with evidence and residual risks.
|
||||
|
||||
---
|
||||
|
||||
## Handoff to Step 2
|
||||
|
||||
Once Step 1 completes, Step 2 (Error Handling & Reliability Hardening) can proceed on stable architecture seams:
|
||||
|
||||
- consistent lifecycle ownership,
|
||||
- explicit startup policy,
|
||||
- clear module boundaries,
|
||||
- documented architecture decisions.
|
||||
@@ -1,80 +0,0 @@
|
||||
# Ver1 Step 2 Results: Error Handling & Reliability Hardening
|
||||
|
||||
## Summary
|
||||
|
||||
Step 2 implementation is complete for the planned reliability and error-handling hardening scope:
|
||||
|
||||
1. Worker retries are now explicit, bounded, and category-driven.
|
||||
2. Error behavior is more consistent across worker/API/UI boundaries.
|
||||
3. Logging now includes stronger boundary context in key failure paths.
|
||||
4. Test coverage was expanded for retry policy and new reliability settings.
|
||||
|
||||
## Implemented Changes
|
||||
|
||||
### 1) Worker retry policy and terminal behavior
|
||||
|
||||
- Updated `src/transcription/models.py`:
|
||||
- Added `Job.retry_count` with default `0`.
|
||||
- Updated `src/transcription/config.py`:
|
||||
- Added `worker_max_retries`.
|
||||
- Added `worker_retry_backoff_seconds`.
|
||||
- Updated `src/transcription/worker.py`:
|
||||
- Added bounded retry decision path (`_should_retry`).
|
||||
- Added requeue behavior (`_requeue_for_retry`) for retriable errors.
|
||||
- Added deterministic terminal failure behavior (`_finalize_failed_job`).
|
||||
- Preserved transcript failure detail persistence (`error_id`, `category`, suggestion).
|
||||
|
||||
### 2) API fallback normalization hardening
|
||||
|
||||
- Updated `src/transcription/api/errors.py`:
|
||||
- Fallback handler now emits safe generic internal message for unhandled exceptions.
|
||||
- Added structured boundary logging fields including operation and exception type.
|
||||
|
||||
### 3) UI interaction reliability guard
|
||||
|
||||
- Updated `src/transcription/ui/upload_page.py`:
|
||||
- Added duplicate in-flight submission guard to prevent repeated upload handling while busy.
|
||||
|
||||
### 4) Observability/logging improvements
|
||||
|
||||
- Updated worker logs in `src/transcription/worker.py` to include operation and domain identifiers in key transitions:
|
||||
- pick
|
||||
- retry
|
||||
- transcribed
|
||||
- failed
|
||||
|
||||
## Test Coverage Added/Updated
|
||||
|
||||
- Updated `tests/test_models.py`:
|
||||
- Assert `retry_count` default.
|
||||
- Updated `tests/test_config.py`:
|
||||
- Added worker retry settings default test.
|
||||
- Updated `tests/services/test_worker.py`:
|
||||
- Added retriable requeue test.
|
||||
- Added retry-exhaustion terminal failure test.
|
||||
- Updated existing tests for settings-driven worker behavior.
|
||||
- Existing API error tests remained green with fallback behavior updates:
|
||||
- `tests/api/test_error_responses.py`
|
||||
|
||||
## Verification Evidence
|
||||
|
||||
Executed and passing:
|
||||
|
||||
- `uv run pytest tests/services/test_worker.py tests/test_models.py tests/test_config.py tests/api/test_error_responses.py -q`
|
||||
- `uv run pytest -q`
|
||||
|
||||
## Residual Risks / Follow-ups
|
||||
|
||||
1. Retry policy currently uses simple fixed backoff; richer strategy (exponential/jitter) can be added in later hardening.
|
||||
2. Full cross-layer structured logging standardization can be expanded in Step 6 observability work.
|
||||
3. A formal Step 2 error-path inventory artifact (`ver1-step2-audit.md`) is still recommended for governance completeness.
|
||||
|
||||
## Step 2 Exit Assessment
|
||||
|
||||
- Error taxonomy and envelope stability: **met**
|
||||
- Bounded retry and terminal failure behavior: **met**
|
||||
- Worker reliability controls: **met**
|
||||
- UI interaction hardening for duplicate actions: **met**
|
||||
- Test coverage expansion and full-suite regression safety: **met**
|
||||
|
||||
Step 2 is complete and ready to hand off to Ver1 Step 3.
|
||||
@@ -1,302 +0,0 @@
|
||||
# Step 2 Implementation Plan: Error Handling & Reliability Hardening
|
||||
|
||||
## Purpose
|
||||
|
||||
Implement **Ver1 Step 2** from `docs/ver1/ver1.md` by standardizing failure behavior and reliability controls so the system fails safely, predictably, and transparently across UI, API, services, worker, and provider boundaries.
|
||||
|
||||
Primary governing docs:
|
||||
|
||||
- `docs/error_handling.md` (authoritative contract)
|
||||
- `docs/requirements.md` (REQ-2, REQ-3, REQ-4, REQ-5, REQ-6)
|
||||
- `docs/architecture.md` (boundary ownership and worker lifecycle)
|
||||
- `docs/ver1/ver1.md` (Step 2 objective)
|
||||
|
||||
---
|
||||
|
||||
## MCP Skill and Guide Inputs Incorporated
|
||||
|
||||
This plan integrates guidance from john-stream-mcp resources:
|
||||
|
||||
1. `resource://skills/python-logging-dictconfig/document`
|
||||
- centralized `dictConfig` logging
|
||||
- startup-only configuration
|
||||
- stable named loggers and boundary-level logging discipline
|
||||
|
||||
2. `resource://skills/pytesting/document`
|
||||
- deterministic, behavior-first tests
|
||||
- explicit marker usage and fast/slow lane discipline
|
||||
- integration checks for boundary behavior and error contracts
|
||||
|
||||
3. `resource://skills/fastapi-async-sqlalchemy-modernization/document`
|
||||
- classify at source boundary
|
||||
- explicit transaction/session behavior under failure
|
||||
- phased rollout with quality gates and rollback awareness
|
||||
|
||||
4. `resource://skills/nicegui-ui-customization/document`
|
||||
- explicit user-facing error feedback for each interaction
|
||||
- prevent duplicate actions during in-flight operations
|
||||
- preserve one-way dependency boundaries from UI -> services
|
||||
|
||||
5. `resource://skills/fastapi-uv-docker/document` (applied selectively)
|
||||
- lifespan-safe startup/shutdown behavior
|
||||
- health/readiness posture and cloud-native operational checks
|
||||
|
||||
---
|
||||
|
||||
## Current-State Gap Summary
|
||||
|
||||
The project already has a strong baseline (`AppError`, taxonomy enum, API envelope, worker persistence), but Step 2 needs completion-level hardening:
|
||||
|
||||
1. **Error contract consistency**
|
||||
- API envelope exists, but consistency must be verified for all error pathways.
|
||||
2. **Cross-boundary category normalization**
|
||||
- Provider/service/worker mappings exist, but require stricter policy checks and tests.
|
||||
3. **Retry policy implementation depth**
|
||||
- Step 2 requires bounded retry policy and clear terminal behavior for retriable failures.
|
||||
4. **Operational traceability**
|
||||
- Logging exists; Step 2 requires consistent structured fields at critical boundaries.
|
||||
5. **UI failure UX consistency**
|
||||
- UI error handling exists; Step 2 requires explicit contract coverage and anti-duplication safeguards.
|
||||
|
||||
---
|
||||
|
||||
## Scope for Step 2
|
||||
|
||||
### In scope
|
||||
1. Enforce canonical error taxonomy and envelope across all boundaries.
|
||||
2. Standardize logging fields and boundary-level error traceability.
|
||||
3. Implement/complete bounded retry and terminal failure behavior in worker paths.
|
||||
4. Improve UI/API error presentation consistency and actionable guidance.
|
||||
5. Add comprehensive Step 2 test coverage and verification matrix.
|
||||
6. Update documentation to reflect final Step 2 policies and behavior.
|
||||
|
||||
### Out of scope
|
||||
- Major architecture/topology changes (external queue, distributed worker)
|
||||
- New end-user feature expansion outside reliability/error handling
|
||||
- Full async ORM migration (unless required by bug fix)
|
||||
|
||||
---
|
||||
|
||||
## Target Decisions for Step 2
|
||||
|
||||
1. **Taxonomy stability is mandatory**
|
||||
- `ErrorCategory` values remain stable contract identifiers.
|
||||
2. **Classification occurs at source boundary**
|
||||
- adapters/services normalize early; UI/API only present safely.
|
||||
3. **User safety over internal detail leakage**
|
||||
- expose safe message + suggestion + error_id; keep sensitive detail in logs.
|
||||
4. **Retry is explicit and bounded**
|
||||
- only retriable categories may retry; retries are capped; terminal failures persist reason.
|
||||
5. **Boundary logs carry correlation fields**
|
||||
- include `error_id`, `category`, `operation`, and domain identifiers where available.
|
||||
|
||||
---
|
||||
|
||||
## Detailed Work Breakdown
|
||||
|
||||
## Phase A — Error Contract Audit and Policy Lock
|
||||
|
||||
- [ ] **A1. Build error-path inventory**
|
||||
- Enumerate all failure entry points across:
|
||||
- `api/`
|
||||
- `ui/`
|
||||
- `services/`
|
||||
- `worker.py`
|
||||
- `providers/`
|
||||
|
||||
- [ ] **A2. Produce taxonomy mapping table**
|
||||
- For each known exception path, map:
|
||||
- source exception type
|
||||
- target `ErrorCategory`
|
||||
- retriable flag
|
||||
- API status (if exposed)
|
||||
|
||||
- [ ] **A3. Reconcile with `docs/error_handling.md`**
|
||||
- Resolve any mismatch in category semantics, status codes, or suggested actions.
|
||||
|
||||
### Deliverables
|
||||
- `docs/ver1/ver1-step2-audit.md` (recommended)
|
||||
- taxonomy mapping table
|
||||
|
||||
### Exit Criteria
|
||||
- Every known failure path has explicit category + retriable policy.
|
||||
|
||||
---
|
||||
|
||||
## Phase B — API and Service Contract Hardening
|
||||
|
||||
- [ ] **B1. Enforce API envelope completeness**
|
||||
- Ensure all API errors return:
|
||||
- `error_id`, `category`, `message`, `suggestion`, `timestamp`
|
||||
|
||||
- [ ] **B2. Verify category-to-status mapping consistency**
|
||||
- Confirm `api/errors.py` matches `docs/error_handling.md` mapping guidance.
|
||||
|
||||
- [ ] **B3. Normalize service exceptions at boundary**
|
||||
- Services should raise `AppError` subclasses for known failures.
|
||||
- Unknown exceptions must become `internal_unexpected_error` with traceable `error_id`.
|
||||
|
||||
- [ ] **B4. Ensure safe detail handling**
|
||||
- API/UI messages remain safe.
|
||||
- Diagnostic context remains in logs/persisted failure detail where appropriate.
|
||||
|
||||
### Exit Criteria
|
||||
- No unstructured/unclassified exception escapes core boundaries.
|
||||
- API responses are contract-stable for all tested failure modes.
|
||||
|
||||
---
|
||||
|
||||
## Phase C — Worker Retry and Terminal Failure Policy
|
||||
|
||||
- [ ] **C1. Define bounded retry policy**
|
||||
- Add configurable retry settings (attempt limit/backoff policy).
|
||||
- Limit retries to retriable categories.
|
||||
|
||||
- [ ] **C2. Implement terminal failure persistence**
|
||||
- On retry exhaustion, persist clear terminal reason and `error_id`.
|
||||
- Ensure job status transitions end deterministically at `failed`.
|
||||
|
||||
- [ ] **C3. Add duplicate-processing safety checks**
|
||||
- Prevent duplicate terminal updates when job already resolved.
|
||||
|
||||
- [ ] **C4. Validate worker lifecycle under repeated transient failures**
|
||||
- Ensure loop remains stable and responsive.
|
||||
|
||||
### Exit Criteria
|
||||
- Retries are bounded and policy-driven.
|
||||
- Exhausted retries produce deterministic failed state with evidence.
|
||||
|
||||
---
|
||||
|
||||
## Phase D — Logging and Observability Contract Enforcement
|
||||
|
||||
- [ ] **D1. Central logging conformance check**
|
||||
- Confirm startup-only `dictConfig` use remains canonical.
|
||||
- No module-level `basicConfig` use.
|
||||
|
||||
- [ ] **D2. Standardize error log fields**
|
||||
- Require at minimum when available:
|
||||
- `error_id`, `category`, `operation`, `exception_type`, `job_id`, `document_id`
|
||||
|
||||
- [ ] **D3. Boundary handoff logging**
|
||||
- Add/normalize logs at transitions:
|
||||
- UI action -> service
|
||||
- service -> provider/db
|
||||
- worker pickup -> terminal state
|
||||
|
||||
- [ ] **D4. Log noise control**
|
||||
- Avoid duplicate stack-trace logging across layers for same exception.
|
||||
|
||||
### Exit Criteria
|
||||
- Critical failure events are traceable end-to-end via logs and `error_id`.
|
||||
|
||||
---
|
||||
|
||||
## Phase E — UI Error UX Consistency and Interaction Hardening
|
||||
|
||||
- [ ] **E1. Standardize user error presentation**
|
||||
- For upload/jobs interactions, ensure:
|
||||
- clear title
|
||||
- plain-language message
|
||||
- suggested action
|
||||
- visible error reference id
|
||||
|
||||
- [ ] **E2. Add in-flight interaction guards**
|
||||
- Prevent duplicate submits/click storms during pending operations.
|
||||
|
||||
- [ ] **E3. Ensure deterministic UI state recovery**
|
||||
- controls re-enable after failure
|
||||
- status text remains actionable
|
||||
|
||||
- [ ] **E4. Keep UI boundary clean**
|
||||
- no provider/protocol details leaked into page modules
|
||||
|
||||
### Exit Criteria
|
||||
- All primary UI actions have consistent success/failure interaction behavior.
|
||||
|
||||
---
|
||||
|
||||
## Phase F — Test Expansion and Verification
|
||||
|
||||
Apply pytesting guidance: behavior-first assertions, deterministic fixtures, strict markers.
|
||||
|
||||
- [ ] **F1. API error contract tests**
|
||||
- verify envelope fields and status mapping for each category class.
|
||||
|
||||
- [ ] **F2. Service classification tests**
|
||||
- verify known failures map to expected `AppError` subclasses/categories.
|
||||
|
||||
- [ ] **F3. Worker retry policy tests**
|
||||
- retriable failure retries and eventual success
|
||||
- retriable failure exhaustion -> terminal failed
|
||||
- non-retriable failure -> immediate failed
|
||||
|
||||
- [ ] **F4. UI error behavior tests**
|
||||
- upload/jobs actions show actionable feedback on failures
|
||||
- duplicate action guard behavior
|
||||
|
||||
- [ ] **F5. Regression guard tests**
|
||||
- at least one test per previously observed production/real-world failure mode
|
||||
|
||||
### Validation Commands
|
||||
- `uv run pytest --collect-only -q`
|
||||
- `uv run pytest -m unit -q`
|
||||
- `uv run pytest -m "not external" -q`
|
||||
- `uv run pytest -q`
|
||||
|
||||
### Exit Criteria
|
||||
- All Step 2 reliability/error contract tests pass.
|
||||
- Existing suite remains green.
|
||||
|
||||
---
|
||||
|
||||
## Recommended Implementation Order
|
||||
|
||||
1. Phase A — audit and policy lock
|
||||
2. Phase B — API/service contract hardening
|
||||
3. Phase C — worker retry and terminal policy
|
||||
4. Phase D — logging/traceability normalization
|
||||
5. Phase E — UI consistency hardening
|
||||
6. Phase F — test expansion and full verification
|
||||
|
||||
This order reduces risk by locking policy first, then applying behavior changes at core boundaries before UI polish.
|
||||
|
||||
---
|
||||
|
||||
## Risks and Mitigations
|
||||
|
||||
1. **Risk:** Overly broad retry policy causes hidden failure loops
|
||||
**Mitigation:** strict category-based retry eligibility + hard cap + terminal persistence.
|
||||
|
||||
2. **Risk:** User-facing messages become too technical
|
||||
**Mitigation:** enforce safe message + suggestion contract in tests.
|
||||
|
||||
3. **Risk:** Logging becomes noisy/redundant
|
||||
**Mitigation:** boundary logging rules and single-trace ownership.
|
||||
|
||||
4. **Risk:** Reliability work introduces regressions in happy path
|
||||
**Mitigation:** run full suite continuously; preserve integration pipeline tests.
|
||||
|
||||
---
|
||||
|
||||
## Step 2 Completion Checklist
|
||||
|
||||
- [ ] Error taxonomy mapping table completed and approved.
|
||||
- [ ] API envelope and HTTP status behavior verified for all relevant failure categories.
|
||||
- [ ] Service/provider exception normalization is consistent and tested.
|
||||
- [ ] Worker retry behavior is bounded, explicit, and terminal-state safe.
|
||||
- [ ] Structured error logging fields are present at boundary handoffs.
|
||||
- [ ] UI failure flows provide clear, actionable, and traceable feedback.
|
||||
- [ ] Full test suite passes with new Step 2 coverage included.
|
||||
- [ ] `docs/ver1/ver1-step2-results.md` created with evidence and residual risks.
|
||||
|
||||
---
|
||||
|
||||
## Handoff to Step 3
|
||||
|
||||
After Step 2 completion, Step 3 (Functional Completion by Requirement Domain) proceeds on a hardened foundation:
|
||||
|
||||
- stable failure contracts,
|
||||
- predictable retries and terminal behavior,
|
||||
- actionable user/API error semantics,
|
||||
- improved diagnostic traceability.
|
||||
Reference in New Issue
Block a user