Ver1 Implementation Plan, and a detailed impl plan for step 1.

docs/ver1/ver1-step1.md

@@ -0,0 +1,309 @@
+# Step 1 Implementation Plan: Architecture Consolidation
+
+## Purpose
+
+Align the implemented MVP codebase with the production architecture and V1 constraints documented in:
+
+- `docs/architecture.md`
+- `docs/requirements.md`
+- `docs/error_handling.md`
+- `docs/index.md`
+- `docs/intent.md`
+- `docs/ver1/ver1.md` (Step 1)
+
+This step hardens architecture boundaries and ownership without expanding product scope.
+
+---
+
+## MCP Skill and Guide Inputs Incorporated
+
+This plan explicitly incorporates patterns and guardrails from john-stream-mcp resources:
+
+1. `resource://skills/fastapi-uv-docker/document`
+   - App factory and lifespan ownership
+   - Health endpoint and cloud-native baseline expectations
+   - Environment-driven configuration and startup discipline
+
+2. `resource://skills/fastapi-async-sqlalchemy-modernization/document`
+   - Current-state gap audit first
+   - Target runtime model before refactor
+   - Explicit resource lifecycle ownership
+   - Transaction/session boundary clarity
+   - Phased migration with rollback points
+
+3. `resource://skills/nicegui/document`
+   - Clear dependency direction
+   - UI/page registration as composition, not business logic container
+   - Async responsiveness and boundary separation
+
+4. `resource://prompts/greenfield-architecture/document`
+   - Pattern-comparison-first planning
+   - Explicit tradeoffs and staged implementation
+   - Output contract with risks, open questions, and next steps
+
+---
+
+## Current-State Gap Summary (Architecture vs Implementation)
+
+Based on docs and current `src/transcription` code:
+
+1. **REQ-7 gap (lifespan-owned resources)**  
+   - DB engine/session factory are module globals in `db.py`, not app lifespan-owned.
+   - Worker thread lifecycle is owned by lifespan (good), but DB/provider resource ownership is mixed.
+
+2. **REQ-10 gap (explicit opt-in schema bootstrap)**  
+   - `create_all()` is executed unconditionally on startup in `app.py`.
+
+3. **Data store target gap (REQ-9 + architecture baseline)**  
+   - Runtime still defaults to SQLite MVP setup; production architecture targets PostgreSQL baseline with optional MongoDB.
+
+4. **Layering clarity gap (architecture layer model)**  
+   - Boundaries exist but are not yet formally enforced (interface/app/domain/infra dependency rules are implicit, not codified).
+
+5. **Decision record gap**  
+   - No ADR set documenting key V1 architectural decisions and deviations from MVP.
+
+---
+
+## Scope for Step 1
+
+### In scope
+1. Produce architecture conformance audit and decision records.
+2. Define and implement target runtime ownership model for core resources.
+3. Establish explicit schema bootstrap policy (opt-in in production paths).
+4. Consolidate module boundaries and dependency direction rules.
+5. Update architecture docs to reflect implemented reality and V1 trajectory.
+
+### Out of scope
+- Full async SQLAlchemy rewrite (plan and seams only if deferred)
+- MongoDB feature implementation
+- New user-facing features
+- Major worker architecture replacement (in-process worker remains baseline)
+
+---
+
+## Target Architecture Decisions for V1
+
+1. **Keep modular monolith topology** (FastAPI + NiceGUI + in-process worker).
+2. **Preserve container-light simplicity guardrails** from `architecture.md`.
+3. **Move runtime ownership to lifespan** for:
+   - DB engine/session factory lifecycle
+   - Worker runtime resources
+   - Provider client factory/config lifecycle
+4. **Adopt explicit schema bootstrap policy**:
+   - Dev/test: opt-in auto-bootstrap allowed
+   - Production: startup must not mutate schema implicitly
+5. **Formalize boundary map**:
+   - Interface (`api`, `ui`) -> Application (`services`) -> Domain (`models/rules`) -> Infrastructure (`db`, `providers`)
+   - No reverse imports
+
+---
+
+## Detailed Work Breakdown
+
+## Phase A — Architecture Audit and Baseline Freeze
+
+- [ ] **A1. Produce architecture conformance matrix**
+  - Map each architecture section to current modules/files.
+  - Classify each row: `aligned`, `partial`, `not aligned`.
+
+- [ ] **A2. Produce REQ-7/REQ-9/REQ-10 focused gap report**
+  - Explicitly capture current vs required state.
+  - Include operational risk if left unresolved.
+
+- [ ] **A3. Freeze MVP architecture baseline**
+  - Record current baseline behavior and known temporary shortcuts.
+  - Link this baseline from `docs/ver1/ver1.md`.
+
+### Deliverables
+- `docs/ver1/ver1-step1-audit.md` (or equivalent section in this doc)
+- Architecture conformance table
+
+### Exit Criteria
+- No architecture changes begin before gap matrix and baseline are approved.
+
+---
+
+## Phase B — Resource Ownership Consolidation (Lifespan-Centric)
+
+- [ ] **B1. Define runtime resource ownership contract**
+  - `app.py` lifespan owns resource initialization and cleanup order.
+  - `app.state` carries resource handles/factories.
+  - No hidden module-global side-effect initialization for runtime resources.
+
+- [ ] **B2. Refactor DB ownership model**
+  - Replace module-global engine singleton pattern with lifespan-initialized resource model.
+  - Define one canonical session-factory access path for app/worker/services.
+
+- [ ] **B3. Normalize worker dependencies**
+  - Ensure worker uses lifespan-owned resources/factories rather than implicit globals.
+  - Preserve deterministic startup/shutdown behavior.
+
+- [ ] **B4. Define provider adapter ownership**
+  - Provider client creation strategy is centralized and lifecycle-aware.
+  - Avoid per-call hidden client construction when unnecessary.
+
+### MCP-Guided Guardrails
+- Use explicit lifecycle composition patterns from `fastapi-async-sqlalchemy-modernization`.
+- Maintain app-factory + lifespan structure per `fastapi-uv-docker`.
+- Keep UI registration as composition only per `nicegui`.
+
+### Exit Criteria
+- Core runtime resources have one owner and one cleanup path.
+- No critical resource has ambiguous ownership.
+
+---
+
+## Phase C — Schema Bootstrap Policy (REQ-10 Alignment)
+
+- [ ] **C1. Define environment-aware bootstrap policy**
+  - `auto_create_schema` (or equivalent) disabled in production by default.
+  - Startup schema mutation is explicit and intentional.
+
+- [ ] **C2. Split startup responsibilities**
+  - App startup performs health-critical initialization only.
+  - Schema bootstrap path is moved to explicit command/flag workflow.
+
+- [ ] **C3. Update deployment/runbook docs**
+  - Document migration/bootstrap flow for dev, staging, prod.
+  - Ensure policy is testable and auditable.
+
+### Exit Criteria
+- Normal production startup path does not call schema auto-create implicitly.
+- Bootstrap behavior is explicit and documented.
+
+---
+
+## Phase D — Module Boundary Enforcement
+
+- [ ] **D1. Publish dependency direction rules**
+  - Allowed import directions across `api`, `ui`, `services`, `models/domain`, `db/providers`.
+  - Explicitly disallow reverse dependencies.
+
+- [ ] **D2. Reconcile package map with docs**
+  - Ensure docs’ architecture elements match real package layout and naming.
+  - Update docs where intentional deviations remain.
+
+- [ ] **D3. Isolate cross-layer responsibilities**
+  - Keep API/UI presentation concerns out of services.
+  - Keep provider/DB specifics out of interface layer.
+
+- [ ] **D4. Add lightweight architecture checks**
+  - Add static/import checks and/or review checklist in CI/review process.
+
+### Exit Criteria
+- Boundary rules are documented and applied.
+- Architectural drift can be detected during review/CI.
+
+---
+
+## Phase E — Architecture Decision Records (ADRs)
+
+- [ ] **E1. Create ADR index**
+  - Add `docs/adr/README.md` with template and status model.
+
+- [ ] **E2. Record minimum V1 ADR set**
+  1. Runtime ownership model (lifespan-owned resources)
+  2. Schema bootstrap policy (explicit vs implicit)
+  3. Persistence baseline (PostgreSQL target; SQLite transition strategy)
+  4. Worker topology (in-process for V1, extension path preserved)
+
+- [ ] **E3. Cross-link ADRs**
+  - Link from architecture and V1 docs.
+
+### Exit Criteria
+- Major architecture decisions are explicit, versioned, and discoverable.
+
+---
+
+## Phase F — Documentation Consolidation
+
+- [ ] **F1. Update `docs/architecture.md`**
+  - Reflect real implementation and V1 target state separately.
+  - Mark transitional choices clearly.
+
+- [ ] **F2. Update `docs/index.md` navigation consistency**
+  - Ensure architecture/readme references match actual docs/files.
+
+- [ ] **F3. Update `docs/requirements.md` traceability notes**
+  - Mark REQ-7/REQ-10 status and verification approach after consolidation.
+
+- [ ] **F4. Add Step 1 result summary**
+  - Create `docs/ver1/ver1-step1-results.md` after implementation.
+
+### Exit Criteria
+- Docs are internally consistent and match runtime architecture reality.
+
+---
+
+## Verification Plan
+
+## Architecture Verification Matrix (Step 1)
+
+1. **Inspection**
+   - Resource ownership map exists and matches code.
+   - Schema bootstrap policy is explicit and environment-aware.
+   - ADRs exist for each key architecture decision.
+
+2. **Automated checks**
+   - Existing test suite remains green.
+   - New/updated tests validate startup policy (no implicit schema mutation in production mode).
+   - Import/dependency-direction checks pass (if introduced).
+
+3. **Demonstration**
+   - App starts in dev mode with explicit expected behavior.
+   - App starts in production mode without mutating schema implicitly.
+   - Worker lifecycle starts/stops cleanly with app lifespan.
+
+---
+
+## Risks and Mitigations
+
+1. **Risk:** Refactor destabilizes MVP behavior  
+   **Mitigation:** Phase changes with small PRs and regression checks after each phase.
+
+2. **Risk:** Over-rotation into premature async rewrite  
+   **Mitigation:** Keep this step focused on lifecycle ownership and boundaries; defer full async migration unless required.
+
+3. **Risk:** Schema policy changes break local DX  
+   **Mitigation:** Keep explicit dev bootstrap path simple and documented.
+
+4. **Risk:** Boundary rules become “doc only”  
+   **Mitigation:** Add CI/review enforcement and architecture checklist.
+
+---
+
+## Recommended Implementation Order
+
+1. Phase A — Audit and baseline freeze  
+2. Phase B — Resource ownership consolidation  
+3. Phase C — Schema bootstrap policy  
+4. Phase D — Boundary enforcement  
+5. Phase E — ADR authoring  
+6. Phase F — Documentation consolidation
+
+This order minimizes risk: diagnose first, then refactor ownership, then lock policy, then enforce boundaries, and finally finalize docs.
+
+---
+
+## Step 1 Completion Checklist
+
+- [ ] Architecture conformance matrix approved.
+- [ ] REQ-7 ownership gaps resolved or explicitly deferred with owner/date.
+- [ ] REQ-10 explicit bootstrap policy implemented and verified.
+- [ ] Dependency direction rules documented and enforced.
+- [ ] ADR set created for all major Step 1 decisions.
+- [ ] Architecture and index docs updated to match implementation.
+- [ ] Full test suite passes after consolidation.
+- [ ] `docs/ver1/ver1-step1-results.md` created with evidence and residual risks.
+
+---
+
+## Handoff to Step 2
+
+Once Step 1 completes, Step 2 (Error Handling & Reliability Hardening) can proceed on stable architecture seams:
+
+- consistent lifecycle ownership,
+- explicit startup policy,
+- clear module boundaries,
+- documented architecture decisions.

docs/ver1/ver1.md

@@ -0,0 +1,320 @@
+# Version 1 Implementation Plan
+
+This plan defines the path from MVP to **Version 1 complete**.  
+The objective is to deliver the full scoped product with production readiness, while explicitly separating refinements/enhancements into a future document.
+
+---
+
+## 0) Plan Governance & Scope Control (Foundation)
+
+**Goal:** Keep execution focused on V1 completion, not optimization/perfection.
+
+### Implementation Steps
+1. Create and maintain a **V1 Traceability Matrix**:
+   - Requirement ID
+   - Current status (`done`, `partial`, `not started`)
+   - Owner
+   - Validation method
+2. Define V1 completion gates:
+   - Functional complete
+   - Operationally complete
+   - Production-ready complete
+3. Snapshot the MVP baseline (tag/changelog reference).
+4. Create a standing rule: any non-V1 idea is logged to a separate enhancements backlog document (to be named later), not added to active V1 scope unless explicitly approved.
+
+### Deliverables
+- `docs/ver1/ver1.md` (this plan)
+- V1 traceability artifact (linked from here when created)
+
+### Exit Criteria
+- Every in-scope requirement has explicit ownership and status.
+- Scope-change process is agreed and followed.
+
+---
+
+## 1) Architecture Consolidation
+
+**Goal:** Align implementation with the intended architecture and reduce MVP shortcuts.
+
+### Implementation Steps
+1. Compare implemented modules/components with architecture documentation.
+2. Identify and classify architectural debt:
+   - Temporary coupling
+   - Missing interfaces
+   - Placeholder services/components
+3. Resolve high-risk architectural gaps first.
+4. Record key decisions and tradeoffs in ADRs.
+
+### Deliverables
+- Updated architecture diagrams and boundaries
+- ADR entries for major decisions
+
+### Exit Criteria
+- Architecture documentation reflects system reality.
+- Critical architecture risks are addressed or scheduled with owners/dates.
+
+---
+
+## 2) Error Handling & Reliability Hardening
+
+**Goal:** Ensure predictable, safe behavior under failure conditions.
+
+### Implementation Steps
+1. Standardize error taxonomy and envelope format across all layers.
+2. Ensure clear distinction between:
+   - User-facing errors
+   - Internal/system errors
+   - Retryable vs non-retryable failures
+3. Add resilience controls where needed:
+   - Timeouts
+   - Retries with backoff
+   - Circuit breaking / fallback logic
+4. Add failure-path tests for critical workflows.
+
+### Deliverables
+- Error code catalog/reference
+- Failure mode test coverage for critical paths
+
+### Exit Criteria
+- Error behavior is consistent across major flows.
+- Known failure scenarios are tested and pass.
+
+---
+
+## 3) Functional Completion by Requirement Domain
+
+**Goal:** Complete all V1 functional requirements in a risk-aware order.
+
+### Recommended Order
+1. Business-critical end-user flows
+2. Data integrity and consistency capabilities
+3. Admin/operational controls
+4. Lower-priority UX and quality-of-life items that are in V1 scope
+
+### Implementation Steps
+For each requirement slice:
+1. Finalize contract/schema
+2. Implement domain logic
+3. Implement persistence/state changes
+4. Integrate API/UI
+5. Add automated tests
+6. Update docs
+
+### Deliverables
+- Requirement completion report with validation evidence
+
+### Exit Criteria
+- All V1 “must-have” requirements are complete and validated.
+
+---
+
+## 4) Data Model, Migration, and Backfill Safety
+
+**Goal:** Ensure data model and migrations are production-safe.
+
+### Implementation Steps
+1. Validate schema against final V1 domain needs.
+2. Implement forward-safe migrations.
+3. Define rollback/mitigation plans for migration failures.
+4. Build and verify backfill scripts (if needed).
+5. Add migration rehearsal in staging with representative data.
+
+### Deliverables
+- Migration runbook
+- Backfill verification checklist
+
+### Exit Criteria
+- Migration plan validated in staging.
+- No unresolved data-loss risk for V1 rollout.
+
+---
+
+## 5) Security, Access Control, and Compliance Baseline
+
+**Goal:** Close MVP security gaps and establish V1 baseline controls.
+
+### Implementation Steps
+1. Complete authn/authz coverage for all routes/actions.
+2. Enforce input validation and output sanitization.
+3. Verify secret management and credential rotation process.
+4. Add audit logging for sensitive operations.
+5. Run dependency/security scanning in CI and remediate findings.
+
+### Deliverables
+- Security checklist with status
+- Threat/risk update for V1 scope
+
+### Exit Criteria
+- No unresolved critical/high vulnerabilities for V1 launch.
+- Access control behavior verified by tests.
+
+---
+
+## 6) Observability & Operability
+
+**Goal:** Make system behavior observable and supportable in production.
+
+### Implementation Steps
+1. Standardize structured logging and correlation IDs.
+2. Add core metrics:
+   - Latency
+   - Throughput
+   - Error rates
+   - Resource saturation
+3. Add tracing for critical request/workflow paths.
+4. Define SLOs/SLIs and alert thresholds.
+5. Prepare incident response and rollback runbooks.
+
+### Deliverables
+- Dashboards and alerts
+- Operations runbooks
+
+### Exit Criteria
+- Team can detect, triage, and remediate incidents quickly.
+- Core production signals are available and reliable.
+
+---
+
+## 7) Test Strategy Expansion & Quality Gates
+
+**Goal:** Raise confidence for repeatable, low-risk releases.
+
+### Implementation Steps
+1. Expand unit and integration tests across V1 features.
+2. Add contract tests between key components/services.
+3. Add end-to-end tests for critical user journeys.
+4. Add non-functional tests where relevant:
+   - Performance/load
+   - Soak
+   - Failure-injection scenarios
+5. Enforce CI quality gates (tests, lint, type checks, security scans).
+
+### Deliverables
+- Test matrix with ownership
+- CI gate definition and thresholds
+
+### Exit Criteria
+- Critical-path regressions are blocked automatically.
+- Test coverage and reliability thresholds meet V1 targets.
+
+---
+
+## 8) Performance & Scalability Validation
+
+**Goal:** Meet expected V1 performance at projected load.
+
+### Implementation Steps
+1. Define performance budgets per key flow.
+2. Benchmark current behavior in staging.
+3. Optimize bottlenecks (queries, caching, concurrency, etc.).
+4. Re-test after each optimization and compare against budget.
+5. Document known limits and safe operating bounds.
+
+### Deliverables
+- Performance benchmark report
+- Optimization log
+
+### Exit Criteria
+- V1 performance targets met for expected usage profile.
+
+---
+
+## 9) Release Engineering & Environment Readiness
+
+**Goal:** Make deployment repeatable, controlled, and reversible.
+
+### Implementation Steps
+1. Harden CI/CD pipeline with clear promotion gates.
+2. Ensure config parity and consistency across environments.
+3. Define rollout strategy (phased/canary/limited release as applicable).
+4. Validate rollback procedures in staging.
+5. Produce release checklist and ownership model.
+
+### Deliverables
+- Release playbook
+- Environment readiness checklist
+
+### Exit Criteria
+- Deployment and rollback are rehearsed and reliable.
+- Release process is executable without tribal knowledge.
+
+---
+
+## 10) Documentation Completion
+
+**Goal:** Ensure V1 can be built, operated, and supported from documentation.
+
+### Implementation Steps
+1. Update core project docs to match final V1 behavior:
+   - Architecture
+   - Error handling
+   - Requirements status
+   - Index/navigation
+   - Intent alignment summary
+2. Add operator troubleshooting guides.
+3. Add integration/API examples for consumers.
+4. Publish changelog/version notes for V1.
+
+### Deliverables
+- Updated documentation set for V1
+- V1 release notes
+
+### Exit Criteria
+- A new team member can run/support the system using docs alone.
+
+---
+
+## 11) Final Validation, UAT, and Launch
+
+**Goal:** Confirm readiness and launch V1 safely.
+
+### Implementation Steps
+1. Run full-system acceptance validation against the V1 traceability matrix.
+2. Conduct stakeholder UAT and capture sign-off.
+3. Execute production readiness review.
+4. Launch in controlled phases and monitor key signals.
+
+### Deliverables
+- UAT/PRR sign-off records
+- Launch checklist and monitoring plan
+
+### Exit Criteria
+- Stakeholder approval achieved.
+- Launch metrics are stable within defined thresholds.
+
+---
+
+## 12) Post-Launch Stabilization (30–60 Days)
+
+**Goal:** Consolidate V1 in production before major expansion.
+
+### Implementation Steps
+1. Track incidents, defects, and user feedback.
+2. Prioritize stabilization fixes with short cycle times.
+3. Remove temporary flags/mitigations introduced during launch.
+4. Produce post-launch retrospective and handoff to standard roadmap cadence.
+
+### Deliverables
+- Stabilization report
+- Prioritized backlog update
+
+### Exit Criteria
+- Incident/error rates converge to steady-state targets.
+- V1 transitions from launch mode to normal operations.
+
+---
+
+## Recommended Execution Rhythm
+
+- **Weekly:** Requirement closure + risk review
+- **Biweekly:** Release train with quality gates
+- **Milestone reviews:** After phases 2, 6, 9, and 11
+
+---
+
+## Scope Discipline Rule (V1 Focus)
+
+To preserve delivery focus:
+- V1 execution prioritizes completion of scoped requirements.
+- Refinements/enhancements are captured in a separate future document and backlog.
+- Only explicitly approved scope changes may enter this plan.