diff --git a/docs/ver1/ver1-step4-results.md b/docs/ver1/ver1-step4-results.md new file mode 100644 index 0000000..839ef63 --- /dev/null +++ b/docs/ver1/ver1-step4-results.md @@ -0,0 +1,118 @@ +# Ver1 Step 4 Results: Data Model and Migration Safety + +## Summary + +Step 4 implementation status: **in progress**. + +This document records completed migration-safety work, validation evidence, and remaining follow-ups for Ver1 Step 4. + +Implemented in this step: + +1. _TBD_ +2. _TBD_ +3. _TBD_ + +--- + +## Implemented Changes + +### 1) Schema audit and invariant lock + +_TBD_ + +### 2) Migration policy/tooling lock + +_TBD_ + +### 3) Forward migration implementation + +_TBD_ + +### 4) Rollback and mitigation runbook + +_TBD_ + +### 5) Backfill implementation or explicit no-backfill decision + +_TBD_ + +--- + +## Test and Verification Evidence + +### Added/Updated Tests + +1. _TBD_ +2. _TBD_ +3. _TBD_ + +### Validation Runs + +Run and record outcomes: + +- `uv run pytest --collect-only -q` -> _TBD_ +- `uv run pytest -m unit -q` -> _TBD_ +- `uv run pytest -m "not external" -q` -> _TBD_ +- `uv run pytest -q` -> _TBD_ + +### Migration Rehearsal Evidence + +Record migration rehearsal details: + +- baseline data set used: _TBD_ +- forward migration result: _TBD_ +- post-migration verification result: _TBD_ +- rollback/mitigation rehearsal result: _TBD_ + +--- + +## Requirement Traceability (Step 4) + +| Step 4 Area | REQ Coverage | Status | Evidence | +| --- | --- | --- | --- | +| Schema lifecycle and state persistence safety | REQ-3, REQ-4, REQ-11 | _TBD_ | _TBD_ | +| Lifespan/runtime ownership continuity | REQ-7 | _TBD_ | _TBD_ | +| Explicit non-mutating production startup policy | REQ-10 | _TBD_ | _TBD_ | +| Prompt/data continuity constraints | REQ-12 | _TBD_ | _TBD_ | + +--- + +## Operational Artifacts Produced + +- `docs/ver1/ver1-step4.md` +- `docs/ver1/ver1-step4-migration-runbook.md` (_if created_) +- _TBD additional artifacts_ + +--- + +## Risks, Exceptions, and Follow-Ups + +1. _TBD_ +2. _TBD_ +3. _TBD_ + +Open follow-ups to carry forward: + +- _TBD_ + +--- + +## Step 4 Exit Assessment + +- Schema validation against finalized V1 domain: **_TBD_** +- Forward migration path safety and repeatability: **_TBD_** +- Rollback/mitigation readiness: **_TBD_** +- Backfill risk closure: **_TBD_** +- Test and regression safety: **_TBD_** + +Step 4 completion status: **_TBD_** + +--- + +## Handoff to Step 5 + +Once Step 4 is marked complete, Step 5 can proceed with: + +- verified migration safety baseline +- explicit rollback and recovery procedures +- reduced data-integrity risk entering private-network safety hardening \ No newline at end of file diff --git a/docs/ver1/ver1-step4.md b/docs/ver1/ver1-step4.md new file mode 100644 index 0000000..a2662c3 --- /dev/null +++ b/docs/ver1/ver1-step4.md @@ -0,0 +1,378 @@ +# Step 4 Implementation Plan: Data Model and Migration Safety + +## Purpose + +Implement **Ver1 Step 4** from `docs/ver1/ver1.md` by making data-model evolution safe, explicit, and repeatable for personal-scale deployment. + +Step 4 ensures schema changes are handled through deterministic migration workflows rather than implicit startup mutation, while preserving: + +- personal-scale operational simplicity +- single-operator deployment model +- lifecycle-owned runtime resource boundaries +- stable requirement traceability and low rollback risk + +Primary governing docs: + +- `docs/ver1/ver1.md` (Step 4 objective and sequencing) +- `docs/architecture.md` (runtime ownership, persistence boundaries, simplicity guardrails) +- `docs/requirements.md` (REQ-3, REQ-4, REQ-7, REQ-10, REQ-11, REQ-12 emphasis) +- `docs/error_handling.md` (failure classification and safe error surfacing) +- `docs/intent.md` (verbatim/transcription/revision domain behavior) + +--- + +## MCP Resources Reviewed and Applied + +All currently available resources on `john-stream-mcp` were reviewed. Step 4 applies the following guidance directly: + +1. `resource://skills/fastapi-async-sqlalchemy-modernization/document` + - explicit engine/session lifecycle ownership + - transaction boundary clarity for schema transitions and backfills + - phased rollout with rollback-aware checkpoints + +2. `resource://skills/pydantic-settings/document` + - typed migration/runtime safety settings + - explicit source-precedence behavior for operational toggles + - fail-fast config semantics for unsafe startup paths + +3. `resource://skills/pytesting/document` + - deterministic migration verification lanes + - strict marker discipline + - behavior-first test coverage for migration outcomes + +4. `resource://skills/python-logging-dictconfig/document` + - startup-centralized logging configuration + - structured migration and rollback event traceability + +5. `resource://skills/fastapi-uv-docker/document` + - deployment and rehearsal discipline + - startup/health posture validation during migration windows + +6. `resource://skills/python-typing/document` + - modern typing hygiene for touched migration/persistence modules + +7. `resource://skills/ruff-linting-formating/document` + - lint/format consistency for migration scripts and database modules + +Planning methodology inputs also applied: + +8. `resource://prompts/greenfield-architecture/document` + - staged execution with explicit risk and extension handling + +9. `resource://prompts/pytest-scaffold/document` +10. `resource://prompts/pytest-fill-scaffold/document` + - test-structure-first and deterministic fill-in sequencing + +Reviewed but not directly Step 4 execution-critical: + +- skills: `copilot-customization`, `mcp-details`, `nicegui`, `nicegui-ui-customization`, `vscode-configuration`, `zensical-docs` +- prompts: `authoring`, `mcp-consumer-repo-shim` + +--- + +## Current-State Gap Summary (Step 4 Scope) + +Based on Step 1–3 outcomes and current docs/tests: + +1. **Bootstrap policy baseline is present** + - Environment-aware schema bootstrap policy exists and aligns with REQ-10 intent. +2. **Functional model expanded in Step 3** + - Revision/acceptance features introduce schema evolution requirements that need formal migration safety rehearsal. +3. **Runbook maturity required** + - Step 4 requires explicit migration + rollback procedures and evidence. +4. **Backfill risk must be evaluated** + - New/changed fields and semantics must be checked for historical data reconciliation needs. +5. **Release-path integration needed** + - Step 4 artifacts must feed Step 9 release readiness and Step 10 docs completion. + +--- + +## Scope for Step 4 + +### In scope + +1. Validate final V1 schema against implemented domain behavior (post-Step 3 reality). +2. Define and implement forward-safe migration path for expected upgrades. +3. Define and document rollback/mitigation strategy for migration failures. +4. Implement backfill scripts only if required, with idempotent behavior. +5. Rehearse migration + rollback locally using representative sample data. +6. Add Step 4-specific verification tests and operational checks. +7. Produce operator-facing migration/rollback runbook and Step 4 results evidence. + +### Out of scope + +- Distributed/externally orchestrated migration systems +- Major persistence-architecture rewrites beyond V1 scope +- Non-V1 enhancement migrations unrelated to implemented requirement slices + +--- + +## Target Decisions for Step 4 + +1. **Production startup remains non-mutating by default** + - Preserve REQ-10 posture and avoid implicit schema mutation at normal startup. + +2. **Schema changes are explicit operator workflows** + - Migrations run as deliberate operational actions, not hidden side effects. + +3. **Migration safety beats migration speed** + - Additive and reversible-first patterns are preferred where possible. + +4. **Rollback policy is explicit per change** + - Each migration must declare rollback class: + - direct rollback supported + - forward-fix required + - backup restore required + +5. **Backfills are optional and minimal** + - Introduce only when required by correctness/invariants, never by convenience. + +6. **Migration observability is mandatory** + - Structured logs include operation, migration identifier, status, and failure classification. + +--- + +## Detailed Work Breakdown + +## Phase A — Schema and Domain Invariant Audit + +- [ ] **A1. Build canonical V1 schema inventory** + - Enumerate all persisted entities and key fields: + - document records + - jobs and statuses + - transcripts + - transcript revisions + - failure/provenance fields +- [ ] **A2. Validate invariants against implemented behavior** + - Cross-check Step 3 functionality and current domain expectations: + - append-only revision history + - accepted revision semantics + - canonical transcript synchronization behavior +- [ ] **A3. Classify required schema deltas** + - Categorize deltas: + - additive and safe + - compatibility-sensitive + - potentially destructive (must be staged or deferred) + +### Deliverables + +- `docs/ver1/ver1-step4-schema-audit.md` (recommended) +- schema-delta matrix with risk class and owning module + +### Exit Criteria + +- all required schema changes have explicit rationale and risk classification +- no ambiguous domain invariant remains + +--- + +## Phase B — Migration Policy and Tooling Lock + +- [ ] **B1. Lock migration workflow policy** + - Define canonical migration execution path and artifact conventions. +- [ ] **B2. Define migration authoring checklist** + - Include: + - preconditions + - forward steps + - rollback class + - post-verification checks +- [ ] **B3. Align policy with runtime startup safeguards** + - Ensure production startup remains explicit/non-mutating by default. +- [ ] **B4. Define operator invocation standard** + - One documented command path for local and production-like workflows. + +### Deliverables + +- migration policy section (this doc + runbook) +- migration authoring/review checklist + +### Exit Criteria + +- one unambiguous migration process exists and is documented +- startup policy and migration policy are consistent and non-conflicting + +--- + +## Phase C — Forward Migration Implementation + +- [ ] **C1. Implement required migration set** + - Build migration artifacts for all approved Step 4 deltas. +- [ ] **C2. Preserve compatibility where needed** + - Use staged expand/contract strategy when direct cutover is unsafe. +- [ ] **C3. Add migration logging checkpoints** + - Log start, phase boundaries, completion, and failure details. +- [ ] **C4. Verify post-migration schema state** + - Confirm expected tables/columns/constraints/indexes are present. + +### Deliverables + +- migration artifacts/scripts for V1 target schema +- schema verification checklist outputs + +### Exit Criteria + +- baseline-to-target forward migration executes successfully +- post-migration checks pass deterministically + +--- + +## Phase D — Rollback and Mitigation Strategy + +- [ ] **D1. Define rollback classes per migration** + - direct downgrade vs forward-fix vs backup-restore. +- [ ] **D2. Create rollback decision tree** + - trigger conditions, safe stop points, and recovery path. +- [ ] **D3. Align failure classification with `error_handling.md`** + - normalize migration failures into canonical categories: + - `infrastructure_transient_error` + - `infrastructure_persistent_error` + - `internal_unexpected_error` (as needed) +- [ ] **D4. Rehearse rollback flow** + - run at least one migration failure simulation and execute chosen recovery path. + +### Deliverables + +- rollback/mitigation decision tree +- rehearsal evidence notes + +### Exit Criteria + +- operator can execute rollback/mitigation without undocumented steps +- migration failure paths are diagnosable and classified + +--- + +## Phase E — Backfill Decision and Execution (Conditional) + +- [ ] **E1. Determine backfill necessity** + - inspect whether existing records violate new invariants. +- [ ] **E2. If required, implement idempotent backfill** + - resumable, batch-safe, and deterministic update semantics. +- [ ] **E3. Add post-backfill verification** + - validate: + - revision sequencing integrity + - accepted/current transcript consistency + - job lifecycle consistency +- [ ] **E4. If not required, record explicit “no backfill needed” evidence** + +### Deliverables + +- backfill script(s) and checklist (if applicable) +- no-backfill rationale artifact (if not applicable) + +### Exit Criteria + +- required backfills completed and verified OR formally ruled out with evidence + +--- + +## Phase F — Verification and Test Expansion + +Apply `pytesting` guidance (deterministic, behavior-first, strict markers). + +- [ ] **F1. Migration application tests** + - verify forward migration from representative baseline. +- [ ] **F2. Post-migration schema contract tests** + - verify expected schema shape and key constraints. +- [ ] **F3. Rollback/mitigation tests** + - verify chosen rollback class behavior where practical. +- [ ] **F4. Startup policy regression tests** + - confirm production-mode startup does not mutate schema implicitly. +- [ ] **F5. Backfill behavior tests (if applicable)** + - idempotency and invariants after repeated execution. + +### Validation Commands + +- `uv run pytest --collect-only -q` +- `uv run pytest -m unit -q` +- `uv run pytest -m "not external" -q` +- `uv run pytest -q` + +### Exit Criteria + +- all Step 4 migration-safety checks pass +- no REQ-10 regression introduced + +--- + +## Phase G — Runbook and Documentation Closure + +- [ ] **G1. Create migration and rollback runbook** + - include: + - prerequisites + - backup step + - migration execution + - verification + - rollback/mitigation +- [ ] **G2. Update traceability artifacts** + - map Step 4 outcomes to REQ IDs and evidence. +- [ ] **G3. Prepare Step 4 handoff artifacts** + - ensure outputs feed Step 9 release readiness and Step 10 docs completion. + +### Deliverables + +- `docs/ver1/ver1-step4-migration-runbook.md` (recommended) +- `docs/ver1/ver1-step4-results.md` +- updated traceability references where needed + +### Exit Criteria + +- migration operations are executable using docs alone +- Step 4 evidence is complete and auditable + +--- + +## Recommended Implementation Order + +1. Phase A — schema/invariant audit +2. Phase B — migration policy and tooling lock +3. Phase C — forward migration implementation +4. Phase D — rollback/mitigation strategy + rehearsal +5. Phase E — backfill decision and execution (conditional) +6. Phase F — test and verification expansion +7. Phase G — runbook + traceability closure + +This sequence minimizes risk by locking policy and scope before irreversible data changes. + +--- + +## Risks and Mitigations + +1. **Risk:** Data loss from unsafe schema transitions + - **Mitigation:** backup-first gate, staged migration strategies, post-check verification. + +2. **Risk:** Startup policy drift reintroduces implicit schema mutation + - **Mitigation:** explicit regression tests for production startup behavior (REQ-10 guard). + +3. **Risk:** Rollback path is incomplete or untested + - **Mitigation:** mandatory rollback class declaration + rehearsal evidence. + +4. **Risk:** Backfill scripts cause partial/inconsistent state + - **Mitigation:** idempotent design, batching, and invariant-focused verification. + +5. **Risk:** Migration failure diagnostics are unclear + - **Mitigation:** structured logging + error category mapping per `error_handling.md`. + +--- + +## Step 4 Completion Checklist + +- [ ] V1 schema audit completed and approved. +- [ ] Migration workflow policy is locked and documented. +- [ ] Required forward migrations are implemented and validated. +- [ ] Rollback/mitigation decision tree is documented and rehearsed. +- [ ] Backfill required/not-required decision is evidenced. +- [ ] Migration-safety test coverage is added and passing. +- [ ] Startup non-mutation policy remains verified in production mode. +- [ ] Step 4 runbook and results artifacts are completed. + +--- + +## Handoff to Step 5 + +Step 4 completion enables Step 5 (Private-Network Safety Baseline) with: + +- stable, explicit schema evolution mechanics +- reduced upgrade risk for single-operator deployments +- migration/rollback procedures suitable for personal-scale production +- traceable evidence for release-readiness gates \ No newline at end of file diff --git a/uploads/0a0444f5-7f58-40c2-ac4e-84be0e9af9cf_Time Rolls On - page 023.jpg b/uploads/0a0444f5-7f58-40c2-ac4e-84be0e9af9cf_Time Rolls On - page 023.jpg new file mode 100644 index 0000000..8a245a4 Binary files /dev/null and b/uploads/0a0444f5-7f58-40c2-ac4e-84be0e9af9cf_Time Rolls On - page 023.jpg differ