transcription/docs/ver1/ver1-step1.md

Step 1 Implementation Plan: Architecture Consolidation

Purpose

Align the implemented MVP codebase with the production architecture and V1 constraints documented in:

• docs/architecture.md
• docs/requirements.md
• docs/error_handling.md
• docs/index.md
• docs/intent.md
• docs/ver1/ver1.md (Step 1)

This step hardens architecture boundaries and ownership without expanding product scope.

---

MCP Skill and Guide Inputs Incorporated

This plan explicitly incorporates patterns and guardrails from john-stream-mcp resources:

1. resource://skills/fastapi-uv-docker/document

   • App factory and lifespan ownership
   • Health endpoint and cloud-native baseline expectations
   • Environment-driven configuration and startup discipline

2. resource://skills/fastapi-async-sqlalchemy-modernization/document

   • Current-state gap audit first
   • Target runtime model before refactor
   • Explicit resource lifecycle ownership
   • Transaction/session boundary clarity
   • Phased migration with rollback points

3. resource://skills/nicegui/document

   • Clear dependency direction
   • UI/page registration as composition, not business logic container
   • Async responsiveness and boundary separation

4. resource://prompts/greenfield-architecture/document

   • Pattern-comparison-first planning
   • Explicit tradeoffs and staged implementation
   • Output contract with risks, open questions, and next steps

---

Current-State Gap Summary (Architecture vs Implementation)

Based on docs and current src/transcription code:

1. REQ-7 gap (lifespan-owned resources)

   • DB engine/session factory are module globals in db.py, not app lifespan-owned.
   • Worker thread lifecycle is owned by lifespan (good), but DB/provider resource ownership is mixed.

2. REQ-10 gap (explicit opt-in schema bootstrap)

   • create_all() is executed unconditionally on startup in app.py.

3. Data store target gap (REQ-9 + architecture baseline)

   • Runtime still defaults to SQLite MVP setup; production architecture targets PostgreSQL baseline with optional MongoDB.

4. Layering clarity gap (architecture layer model)

   • Boundaries exist but are not yet formally enforced (interface/app/domain/infra dependency rules are implicit, not codified).

5. Decision record gap

   • No ADR set documenting key V1 architectural decisions and deviations from MVP.

---

Scope for Step 1

In scope

1. Produce architecture conformance audit and decision records.
2. Define and implement target runtime ownership model for core resources.
3. Establish explicit schema bootstrap policy (opt-in in production paths).
4. Consolidate module boundaries and dependency direction rules.
5. Update architecture docs to reflect implemented reality and V1 trajectory.

Out of scope

• Full async SQLAlchemy rewrite (plan and seams only if deferred)
• MongoDB feature implementation
• New user-facing features
• Major worker architecture replacement (in-process worker remains baseline)

---

Target Architecture Decisions for V1

1. Keep modular monolith topology (FastAPI + NiceGUI + in-process worker).
2. Preserve container-light simplicity guardrails from architecture.md.
3. Move runtime ownership to lifespan for:
   • DB engine/session factory lifecycle
   • Worker runtime resources
   • Provider client factory/config lifecycle
4. Adopt explicit schema bootstrap policy:
   • Dev/test: opt-in auto-bootstrap allowed
   • Production: startup must not mutate schema implicitly
5. Formalize boundary map:
   • Interface (api, ui) -> Application (services) -> Domain (models/rules) -> Infrastructure (db, providers)
   • No reverse imports

---

Detailed Work Breakdown

Phase A — Architecture Audit and Baseline Freeze

• A1. Produce architecture conformance matrix

  • Map each architecture section to current modules/files.
  • Classify each row: aligned, partial, not aligned.

• A2. Produce REQ-7/REQ-9/REQ-10 focused gap report

  • Explicitly capture current vs required state.
  • Include operational risk if left unresolved.

• A3. Freeze MVP architecture baseline

  • Record current baseline behavior and known temporary shortcuts.
  • Link this baseline from docs/ver1/ver1.md.

Deliverables

• docs/ver1/ver1-step1-audit.md (or equivalent section in this doc)
• Architecture conformance table

Exit Criteria

• No architecture changes begin before gap matrix and baseline are approved.

---

Phase B — Resource Ownership Consolidation (Lifespan-Centric)

• B1. Define runtime resource ownership contract

  • app.py lifespan owns resource initialization and cleanup order.
  • app.state carries resource handles/factories.
  • No hidden module-global side-effect initialization for runtime resources.

• B2. Refactor DB ownership model

  • Replace module-global engine singleton pattern with lifespan-initialized resource model.
  • Define one canonical session-factory access path for app/worker/services.

• B3. Normalize worker dependencies

  • Ensure worker uses lifespan-owned resources/factories rather than implicit globals.
  • Preserve deterministic startup/shutdown behavior.

• B4. Define provider adapter ownership

  • Provider client creation strategy is centralized and lifecycle-aware.
  • Avoid per-call hidden client construction when unnecessary.

MCP-Guided Guardrails

• Use explicit lifecycle composition patterns from fastapi-async-sqlalchemy-modernization.
• Maintain app-factory + lifespan structure per fastapi-uv-docker.
• Keep UI registration as composition only per nicegui.

Exit Criteria

• Core runtime resources have one owner and one cleanup path.
• No critical resource has ambiguous ownership.

---

Phase C — Schema Bootstrap Policy (REQ-10 Alignment)

• C1. Define environment-aware bootstrap policy

  • auto_create_schema (or equivalent) disabled in production by default.
  • Startup schema mutation is explicit and intentional.

• C2. Split startup responsibilities

  • App startup performs health-critical initialization only.
  • Schema bootstrap path is moved to explicit command/flag workflow.

• C3. Update deployment/runbook docs

  • Document migration/bootstrap flow for dev, staging, prod.
  • Ensure policy is testable and auditable.

Exit Criteria

• Normal production startup path does not call schema auto-create implicitly.
• Bootstrap behavior is explicit and documented.

---

Phase D — Module Boundary Enforcement

• D1. Publish dependency direction rules

  • Allowed import directions across api, ui, services, models/domain, db/providers.
  • Explicitly disallow reverse dependencies.

• D2. Reconcile package map with docs

  • Ensure docs’ architecture elements match real package layout and naming.
  • Update docs where intentional deviations remain.

• D3. Isolate cross-layer responsibilities

  • Keep API/UI presentation concerns out of services.
  • Keep provider/DB specifics out of interface layer.

• D4. Add lightweight architecture checks

  • Add static/import checks and/or review checklist in CI/review process.

Exit Criteria

• Boundary rules are documented and applied.
• Architectural drift can be detected during review/CI.

---

Phase E — Architecture Decision Records (ADRs)

• E1. Create ADR index

  • Add docs/adr/README.md with template and status model.

• E2. Record minimum V1 ADR set

  1. Runtime ownership model (lifespan-owned resources)
  2. Schema bootstrap policy (explicit vs implicit)
  3. Persistence baseline (PostgreSQL target; SQLite transition strategy)
  4. Worker topology (in-process for V1, extension path preserved)

• E3. Cross-link ADRs

  • Link from architecture and V1 docs.

Exit Criteria

• Major architecture decisions are explicit, versioned, and discoverable.

---

Phase F — Documentation Consolidation

• F1. Update docs/architecture.md

  • Reflect real implementation and V1 target state separately.
  • Mark transitional choices clearly.

• F2. Update docs/index.md navigation consistency

  • Ensure architecture/readme references match actual docs/files.

• F3. Update docs/requirements.md traceability notes

  • Mark REQ-7/REQ-10 status and verification approach after consolidation.

• F4. Add Step 1 result summary

  • Create docs/ver1/ver1-step1-results.md after implementation.

Exit Criteria

• Docs are internally consistent and match runtime architecture reality.

---

Verification Plan

Architecture Verification Matrix (Step 1)

1. Inspection

   • Resource ownership map exists and matches code.
   • Schema bootstrap policy is explicit and environment-aware.
   • ADRs exist for each key architecture decision.

2. Automated checks

   • Existing test suite remains green.
   • New/updated tests validate startup policy (no implicit schema mutation in production mode).
   • Import/dependency-direction checks pass (if introduced).

3. Demonstration

   • App starts in dev mode with explicit expected behavior.
   • App starts in production mode without mutating schema implicitly.
   • Worker lifecycle starts/stops cleanly with app lifespan.

---

Risks and Mitigations

1. Risk: Refactor destabilizes MVP behavior
   Mitigation: Phase changes with small PRs and regression checks after each phase.

2. Risk: Over-rotation into premature async rewrite
   Mitigation: Keep this step focused on lifecycle ownership and boundaries; defer full async migration unless required.

3. Risk: Schema policy changes break local DX
   Mitigation: Keep explicit dev bootstrap path simple and documented.

4. Risk: Boundary rules become “doc only”
   Mitigation: Add CI/review enforcement and architecture checklist.

---

Recommended Implementation Order

1. Phase A — Audit and baseline freeze
2. Phase B — Resource ownership consolidation
3. Phase C — Schema bootstrap policy
4. Phase D — Boundary enforcement
5. Phase E — ADR authoring
6. Phase F — Documentation consolidation

This order minimizes risk: diagnose first, then refactor ownership, then lock policy, then enforce boundaries, and finally finalize docs.

---

Step 1 Completion Checklist

• Architecture conformance matrix approved.
• REQ-7 ownership gaps resolved or explicitly deferred with owner/date.
• REQ-10 explicit bootstrap policy implemented and verified.
• Dependency direction rules documented and enforced.
• ADR set created for all major Step 1 decisions.
• Architecture and index docs updated to match implementation.
• Full test suite passes after consolidation.
• docs/ver1/ver1-step1-results.md created with evidence and residual risks.

---

Handoff to Step 2

Once Step 1 completes, Step 2 (Error Handling & Reliability Hardening) can proceed on stable architecture seams:

• consistent lifecycle ownership,
• explicit startup policy,
• clear module boundaries,
• documented architecture decisions.