Ver1 Step4 implementation plan

2026-06-26 14:37:30 -05:00
parent 06bb4290be
commit 754a273ead
3 changed files with 496 additions and 0 deletions
@@ -0,0 +1,118 @@
+# Ver1 Step 4 Results: Data Model and Migration Safety
+
+## Summary
+
+Step 4 implementation status: **in progress**.
+
+This document records completed migration-safety work, validation evidence, and remaining follow-ups for Ver1 Step 4.
+
+Implemented in this step:
+
+1. _TBD_
+2. _TBD_
+3. _TBD_
+
+---
+
+## Implemented Changes
+
+### 1) Schema audit and invariant lock
+
+_TBD_
+
+### 2) Migration policy/tooling lock
+
+_TBD_
+
+### 3) Forward migration implementation
+
+_TBD_
+
+### 4) Rollback and mitigation runbook
+
+_TBD_
+
+### 5) Backfill implementation or explicit no-backfill decision
+
+_TBD_
+
+---
+
+## Test and Verification Evidence
+
+### Added/Updated Tests
+
+1. _TBD_
+2. _TBD_
+3. _TBD_
+
+### Validation Runs
+
+Run and record outcomes:
+
+- `uv run pytest --collect-only -q` -> _TBD_
+- `uv run pytest -m unit -q` -> _TBD_
+- `uv run pytest -m "not external" -q` -> _TBD_
+- `uv run pytest -q` -> _TBD_
+
+### Migration Rehearsal Evidence
+
+Record migration rehearsal details:
+
+- baseline data set used: _TBD_
+- forward migration result: _TBD_
+- post-migration verification result: _TBD_
+- rollback/mitigation rehearsal result: _TBD_
+
+---
+
+## Requirement Traceability (Step 4)
+
+| Step 4 Area | REQ Coverage | Status | Evidence |
+| --- | --- | --- | --- |
+| Schema lifecycle and state persistence safety | REQ-3, REQ-4, REQ-11 | _TBD_ | _TBD_ |
+| Lifespan/runtime ownership continuity | REQ-7 | _TBD_ | _TBD_ |
+| Explicit non-mutating production startup policy | REQ-10 | _TBD_ | _TBD_ |
+| Prompt/data continuity constraints | REQ-12 | _TBD_ | _TBD_ |
+
+---
+
+## Operational Artifacts Produced
+
+- `docs/ver1/ver1-step4.md`
+- `docs/ver1/ver1-step4-migration-runbook.md` (_if created_)
+- _TBD additional artifacts_
+
+---
+
+## Risks, Exceptions, and Follow-Ups
+
+1. _TBD_
+2. _TBD_
+3. _TBD_
+
+Open follow-ups to carry forward:
+
+- _TBD_
+
+---
+
+## Step 4 Exit Assessment
+
+- Schema validation against finalized V1 domain: **_TBD_**
+- Forward migration path safety and repeatability: **_TBD_**
+- Rollback/mitigation readiness: **_TBD_**
+- Backfill risk closure: **_TBD_**
+- Test and regression safety: **_TBD_**
+
+Step 4 completion status: **_TBD_**
+
+---
+
+## Handoff to Step 5
+
+Once Step 4 is marked complete, Step 5 can proceed with:
+
+- verified migration safety baseline
+- explicit rollback and recovery procedures
+- reduced data-integrity risk entering private-network safety hardening
@@ -0,0 +1,378 @@
+# Step 4 Implementation Plan: Data Model and Migration Safety
+
+## Purpose
+
+Implement **Ver1 Step 4** from `docs/ver1/ver1.md` by making data-model evolution safe, explicit, and repeatable for personal-scale deployment.
+
+Step 4 ensures schema changes are handled through deterministic migration workflows rather than implicit startup mutation, while preserving:
+
+- personal-scale operational simplicity
+- single-operator deployment model
+- lifecycle-owned runtime resource boundaries
+- stable requirement traceability and low rollback risk
+
+Primary governing docs:
+
+- `docs/ver1/ver1.md` (Step 4 objective and sequencing)
+- `docs/architecture.md` (runtime ownership, persistence boundaries, simplicity guardrails)
+- `docs/requirements.md` (REQ-3, REQ-4, REQ-7, REQ-10, REQ-11, REQ-12 emphasis)
+- `docs/error_handling.md` (failure classification and safe error surfacing)
+- `docs/intent.md` (verbatim/transcription/revision domain behavior)
+
+---
+
+## MCP Resources Reviewed and Applied
+
+All currently available resources on `john-stream-mcp` were reviewed. Step 4 applies the following guidance directly:
+
+1. `resource://skills/fastapi-async-sqlalchemy-modernization/document`
+   - explicit engine/session lifecycle ownership
+   - transaction boundary clarity for schema transitions and backfills
+   - phased rollout with rollback-aware checkpoints
+
+2. `resource://skills/pydantic-settings/document`
+   - typed migration/runtime safety settings
+   - explicit source-precedence behavior for operational toggles
+   - fail-fast config semantics for unsafe startup paths
+
+3. `resource://skills/pytesting/document`
+   - deterministic migration verification lanes
+   - strict marker discipline
+   - behavior-first test coverage for migration outcomes
+
+4. `resource://skills/python-logging-dictconfig/document`
+   - startup-centralized logging configuration
+   - structured migration and rollback event traceability
+
+5. `resource://skills/fastapi-uv-docker/document`
+   - deployment and rehearsal discipline
+   - startup/health posture validation during migration windows
+
+6. `resource://skills/python-typing/document`
+   - modern typing hygiene for touched migration/persistence modules
+
+7. `resource://skills/ruff-linting-formating/document`
+   - lint/format consistency for migration scripts and database modules
+
+Planning methodology inputs also applied:
+
+8. `resource://prompts/greenfield-architecture/document`
+   - staged execution with explicit risk and extension handling
+
+9. `resource://prompts/pytest-scaffold/document`
+10. `resource://prompts/pytest-fill-scaffold/document`
+   - test-structure-first and deterministic fill-in sequencing
+
+Reviewed but not directly Step 4 execution-critical:
+
+- skills: `copilot-customization`, `mcp-details`, `nicegui`, `nicegui-ui-customization`, `vscode-configuration`, `zensical-docs`
+- prompts: `authoring`, `mcp-consumer-repo-shim`
+
+---
+
+## Current-State Gap Summary (Step 4 Scope)
+
+Based on Step 1–3 outcomes and current docs/tests:
+
+1. **Bootstrap policy baseline is present**
+   - Environment-aware schema bootstrap policy exists and aligns with REQ-10 intent.
+2. **Functional model expanded in Step 3**
+   - Revision/acceptance features introduce schema evolution requirements that need formal migration safety rehearsal.
+3. **Runbook maturity required**
+   - Step 4 requires explicit migration + rollback procedures and evidence.
+4. **Backfill risk must be evaluated**
+   - New/changed fields and semantics must be checked for historical data reconciliation needs.
+5. **Release-path integration needed**
+   - Step 4 artifacts must feed Step 9 release readiness and Step 10 docs completion.
+
+---
+
+## Scope for Step 4
+
+### In scope
+
+1. Validate final V1 schema against implemented domain behavior (post-Step 3 reality).
+2. Define and implement forward-safe migration path for expected upgrades.
+3. Define and document rollback/mitigation strategy for migration failures.
+4. Implement backfill scripts only if required, with idempotent behavior.
+5. Rehearse migration + rollback locally using representative sample data.
+6. Add Step 4-specific verification tests and operational checks.
+7. Produce operator-facing migration/rollback runbook and Step 4 results evidence.
+
+### Out of scope
+
+- Distributed/externally orchestrated migration systems
+- Major persistence-architecture rewrites beyond V1 scope
+- Non-V1 enhancement migrations unrelated to implemented requirement slices
+
+---
+
+## Target Decisions for Step 4
+
+1. **Production startup remains non-mutating by default**
+   - Preserve REQ-10 posture and avoid implicit schema mutation at normal startup.
+
+2. **Schema changes are explicit operator workflows**
+   - Migrations run as deliberate operational actions, not hidden side effects.
+
+3. **Migration safety beats migration speed**
+   - Additive and reversible-first patterns are preferred where possible.
+
+4. **Rollback policy is explicit per change**
+   - Each migration must declare rollback class:
+     - direct rollback supported
+     - forward-fix required
+     - backup restore required
+
+5. **Backfills are optional and minimal**
+   - Introduce only when required by correctness/invariants, never by convenience.
+
+6. **Migration observability is mandatory**
+   - Structured logs include operation, migration identifier, status, and failure classification.
+
+---
+
+## Detailed Work Breakdown
+
+## Phase A — Schema and Domain Invariant Audit
+
+- [ ] **A1. Build canonical V1 schema inventory**
+  - Enumerate all persisted entities and key fields:
+    - document records
+    - jobs and statuses
+    - transcripts
+    - transcript revisions
+    - failure/provenance fields
+- [ ] **A2. Validate invariants against implemented behavior**
+  - Cross-check Step 3 functionality and current domain expectations:
+    - append-only revision history
+    - accepted revision semantics
+    - canonical transcript synchronization behavior
+- [ ] **A3. Classify required schema deltas**
+  - Categorize deltas:
+    - additive and safe
+    - compatibility-sensitive
+    - potentially destructive (must be staged or deferred)
+
+### Deliverables
+
+- `docs/ver1/ver1-step4-schema-audit.md` (recommended)
+- schema-delta matrix with risk class and owning module
+
+### Exit Criteria
+
+- all required schema changes have explicit rationale and risk classification
+- no ambiguous domain invariant remains
+
+---
+
+## Phase B — Migration Policy and Tooling Lock
+
+- [ ] **B1. Lock migration workflow policy**
+  - Define canonical migration execution path and artifact conventions.
+- [ ] **B2. Define migration authoring checklist**
+  - Include:
+    - preconditions
+    - forward steps
+    - rollback class
+    - post-verification checks
+- [ ] **B3. Align policy with runtime startup safeguards**
+  - Ensure production startup remains explicit/non-mutating by default.
+- [ ] **B4. Define operator invocation standard**
+  - One documented command path for local and production-like workflows.
+
+### Deliverables
+
+- migration policy section (this doc + runbook)
+- migration authoring/review checklist
+
+### Exit Criteria
+
+- one unambiguous migration process exists and is documented
+- startup policy and migration policy are consistent and non-conflicting
+
+---
+
+## Phase C — Forward Migration Implementation
+
+- [ ] **C1. Implement required migration set**
+  - Build migration artifacts for all approved Step 4 deltas.
+- [ ] **C2. Preserve compatibility where needed**
+  - Use staged expand/contract strategy when direct cutover is unsafe.
+- [ ] **C3. Add migration logging checkpoints**
+  - Log start, phase boundaries, completion, and failure details.
+- [ ] **C4. Verify post-migration schema state**
+  - Confirm expected tables/columns/constraints/indexes are present.
+
+### Deliverables
+
+- migration artifacts/scripts for V1 target schema
+- schema verification checklist outputs
+
+### Exit Criteria
+
+- baseline-to-target forward migration executes successfully
+- post-migration checks pass deterministically
+
+---
+
+## Phase D — Rollback and Mitigation Strategy
+
+- [ ] **D1. Define rollback classes per migration**
+  - direct downgrade vs forward-fix vs backup-restore.
+- [ ] **D2. Create rollback decision tree**
+  - trigger conditions, safe stop points, and recovery path.
+- [ ] **D3. Align failure classification with `error_handling.md`**
+  - normalize migration failures into canonical categories:
+    - `infrastructure_transient_error`
+    - `infrastructure_persistent_error`
+    - `internal_unexpected_error` (as needed)
+- [ ] **D4. Rehearse rollback flow**
+  - run at least one migration failure simulation and execute chosen recovery path.
+
+### Deliverables
+
+- rollback/mitigation decision tree
+- rehearsal evidence notes
+
+### Exit Criteria
+
+- operator can execute rollback/mitigation without undocumented steps
+- migration failure paths are diagnosable and classified
+
+---
+
+## Phase E — Backfill Decision and Execution (Conditional)
+
+- [ ] **E1. Determine backfill necessity**
+  - inspect whether existing records violate new invariants.
+- [ ] **E2. If required, implement idempotent backfill**
+  - resumable, batch-safe, and deterministic update semantics.
+- [ ] **E3. Add post-backfill verification**
+  - validate:
+    - revision sequencing integrity
+    - accepted/current transcript consistency
+    - job lifecycle consistency
+- [ ] **E4. If not required, record explicit “no backfill needed” evidence**
+
+### Deliverables
+
+- backfill script(s) and checklist (if applicable)
+- no-backfill rationale artifact (if not applicable)
+
+### Exit Criteria
+
+- required backfills completed and verified OR formally ruled out with evidence
+
+---
+
+## Phase F — Verification and Test Expansion
+
+Apply `pytesting` guidance (deterministic, behavior-first, strict markers).
+
+- [ ] **F1. Migration application tests**
+  - verify forward migration from representative baseline.
+- [ ] **F2. Post-migration schema contract tests**
+  - verify expected schema shape and key constraints.
+- [ ] **F3. Rollback/mitigation tests**
+  - verify chosen rollback class behavior where practical.
+- [ ] **F4. Startup policy regression tests**
+  - confirm production-mode startup does not mutate schema implicitly.
+- [ ] **F5. Backfill behavior tests (if applicable)**
+  - idempotency and invariants after repeated execution.
+
+### Validation Commands
+
+- `uv run pytest --collect-only -q`
+- `uv run pytest -m unit -q`
+- `uv run pytest -m "not external" -q`
+- `uv run pytest -q`
+
+### Exit Criteria
+
+- all Step 4 migration-safety checks pass
+- no REQ-10 regression introduced
+
+---
+
+## Phase G — Runbook and Documentation Closure
+
+- [ ] **G1. Create migration and rollback runbook**
+  - include:
+    - prerequisites
+    - backup step
+    - migration execution
+    - verification
+    - rollback/mitigation
+- [ ] **G2. Update traceability artifacts**
+  - map Step 4 outcomes to REQ IDs and evidence.
+- [ ] **G3. Prepare Step 4 handoff artifacts**
+  - ensure outputs feed Step 9 release readiness and Step 10 docs completion.
+
+### Deliverables
+
+- `docs/ver1/ver1-step4-migration-runbook.md` (recommended)
+- `docs/ver1/ver1-step4-results.md`
+- updated traceability references where needed
+
+### Exit Criteria
+
+- migration operations are executable using docs alone
+- Step 4 evidence is complete and auditable
+
+---
+
+## Recommended Implementation Order
+
+1. Phase A — schema/invariant audit
+2. Phase B — migration policy and tooling lock
+3. Phase C — forward migration implementation
+4. Phase D — rollback/mitigation strategy + rehearsal
+5. Phase E — backfill decision and execution (conditional)
+6. Phase F — test and verification expansion
+7. Phase G — runbook + traceability closure
+
+This sequence minimizes risk by locking policy and scope before irreversible data changes.
+
+---
+
+## Risks and Mitigations
+
+1. **Risk:** Data loss from unsafe schema transitions
+   - **Mitigation:** backup-first gate, staged migration strategies, post-check verification.
+
+2. **Risk:** Startup policy drift reintroduces implicit schema mutation
+   - **Mitigation:** explicit regression tests for production startup behavior (REQ-10 guard).
+
+3. **Risk:** Rollback path is incomplete or untested
+   - **Mitigation:** mandatory rollback class declaration + rehearsal evidence.
+
+4. **Risk:** Backfill scripts cause partial/inconsistent state
+   - **Mitigation:** idempotent design, batching, and invariant-focused verification.
+
+5. **Risk:** Migration failure diagnostics are unclear
+   - **Mitigation:** structured logging + error category mapping per `error_handling.md`.
+
+---
+
+## Step 4 Completion Checklist
+
+- [ ] V1 schema audit completed and approved.
+- [ ] Migration workflow policy is locked and documented.
+- [ ] Required forward migrations are implemented and validated.
+- [ ] Rollback/mitigation decision tree is documented and rehearsed.
+- [ ] Backfill required/not-required decision is evidenced.
+- [ ] Migration-safety test coverage is added and passing.
+- [ ] Startup non-mutation policy remains verified in production mode.
+- [ ] Step 4 runbook and results artifacts are completed.
+
+---
+
+## Handoff to Step 5
+
+Step 4 completion enables Step 5 (Private-Network Safety Baseline) with:
+
+- stable, explicit schema evolution mechanics
+- reduced upgrade risk for single-operator deployments
+- migration/rollback procedures suitable for personal-scale production
+- traceable evidence for release-readiness gates