transcription/docs/ver1/ver1-step4.md

Step 4 Implementation Plan: Data Model and Migration Safety

Purpose

Implement Ver1 Step 4 from docs/ver1/ver1.md by making data-model evolution safe, explicit, and repeatable for personal-scale deployment.

Step 4 ensures schema changes are handled through deterministic migration workflows rather than implicit startup mutation, while preserving:

• personal-scale operational simplicity
• single-operator deployment model
• lifecycle-owned runtime resource boundaries
• stable requirement traceability and low rollback risk

Primary governing docs:

• docs/ver1/ver1.md (Step 4 objective and sequencing)
• docs/architecture.md (runtime ownership, persistence boundaries, simplicity guardrails)
• docs/requirements.md (REQ-3, REQ-4, REQ-7, REQ-10, REQ-11, REQ-12 emphasis)
• docs/error_handling.md (failure classification and safe error surfacing)
• docs/intent.md (verbatim/transcription/revision domain behavior)

---

MCP Resources Reviewed and Applied

All currently available resources on john-stream-mcp were reviewed. Step 4 applies the following guidance directly:

1. resource://skills/fastapi-async-sqlalchemy-modernization/document

   • explicit engine/session lifecycle ownership
   • transaction boundary clarity for schema transitions and backfills
   • phased rollout with rollback-aware checkpoints

2. resource://skills/pydantic-settings/document

   • typed migration/runtime safety settings
   • explicit source-precedence behavior for operational toggles
   • fail-fast config semantics for unsafe startup paths

3. resource://skills/pytesting/document

   • deterministic migration verification lanes
   • strict marker discipline
   • behavior-first test coverage for migration outcomes

4. resource://skills/python-logging-dictconfig/document

   • startup-centralized logging configuration
   • structured migration and rollback event traceability

5. resource://skills/fastapi-uv-docker/document

   • deployment and rehearsal discipline
   • startup/health posture validation during migration windows

6. resource://skills/python-typing/document

   • modern typing hygiene for touched migration/persistence modules

7. resource://skills/ruff-linting-formating/document

   • lint/format consistency for migration scripts and database modules

Planning methodology inputs also applied:

8. resource://prompts/greenfield-architecture/document

   • staged execution with explicit risk and extension handling

9. resource://prompts/pytest-scaffold/document

10. resource://prompts/pytest-fill-scaffold/document

• test-structure-first and deterministic fill-in sequencing

Reviewed but not directly Step 4 execution-critical:

• skills: copilot-customization, mcp-details, nicegui, nicegui-ui-customization, vscode-configuration, zensical-docs
• prompts: authoring, mcp-consumer-repo-shim

---

Current-State Gap Summary (Step 4 Scope)

Based on Step 1–3 outcomes and current docs/tests:

1. Bootstrap policy baseline is present
   • Environment-aware schema bootstrap policy exists and aligns with REQ-10 intent.
2. Functional model expanded in Step 3
   • Revision/acceptance features introduce schema evolution requirements that need formal migration safety rehearsal.
3. Runbook maturity required
   • Step 4 requires explicit migration + rollback procedures and evidence.
4. Backfill risk must be evaluated
   • New/changed fields and semantics must be checked for historical data reconciliation needs.
5. Release-path integration needed
   • Step 4 artifacts must feed Step 9 release readiness and Step 10 docs completion.

---

Scope for Step 4

In scope

1. Validate final V1 schema against implemented domain behavior (post-Step 3 reality).
2. Define and implement forward-safe migration path for expected upgrades.
3. Define and document rollback/mitigation strategy for migration failures.
4. Implement backfill scripts only if required, with idempotent behavior.
5. Rehearse migration + rollback locally using representative sample data.
6. Add Step 4-specific verification tests and operational checks.
7. Produce operator-facing migration/rollback runbook and Step 4 results evidence.

Out of scope

• Distributed/externally orchestrated migration systems
• Major persistence-architecture rewrites beyond V1 scope
• Non-V1 enhancement migrations unrelated to implemented requirement slices

---

Target Decisions for Step 4

1. Production startup remains non-mutating by default

   • Preserve REQ-10 posture and avoid implicit schema mutation at normal startup.

2. Schema changes are explicit operator workflows

   • Migrations run as deliberate operational actions, not hidden side effects.

3. Migration safety beats migration speed

   • Additive and reversible-first patterns are preferred where possible.

4. Rollback policy is explicit per change

   • Each migration must declare rollback class:
     • direct rollback supported
     • forward-fix required
     • backup restore required

5. Backfills are optional and minimal

   • Introduce only when required by correctness/invariants, never by convenience.

6. Migration observability is mandatory

   • Structured logs include operation, migration identifier, status, and failure classification.

---

Detailed Work Breakdown

Phase A — Schema and Domain Invariant Audit

• A1. Build canonical V1 schema inventory
  • Enumerate all persisted entities and key fields:
    • document records
    • jobs and statuses
    • transcripts
    • transcript revisions
    • failure/provenance fields
• A2. Validate invariants against implemented behavior
  • Cross-check Step 3 functionality and current domain expectations:
    • append-only revision history
    • accepted revision semantics
    • canonical transcript synchronization behavior
• A3. Classify required schema deltas
  • Categorize deltas:
    • additive and safe
    • compatibility-sensitive
    • potentially destructive (must be staged or deferred)

Deliverables

• docs/ver1/ver1-step4-schema-audit.md (recommended)
• schema-delta matrix with risk class and owning module

Exit Criteria

• all required schema changes have explicit rationale and risk classification
• no ambiguous domain invariant remains

---

Phase B — Migration Policy and Tooling Lock

• B1. Lock migration workflow policy
  • Define canonical migration execution path and artifact conventions.
• B2. Define migration authoring checklist
  • Include:
    • preconditions
    • forward steps
    • rollback class
    • post-verification checks
• B3. Align policy with runtime startup safeguards
  • Ensure production startup remains explicit/non-mutating by default.
• B4. Define operator invocation standard
  • One documented command path for local and production-like workflows.

Deliverables

• migration policy section (this doc + runbook)
• migration authoring/review checklist

Exit Criteria

• one unambiguous migration process exists and is documented
• startup policy and migration policy are consistent and non-conflicting

---

Phase C — Forward Migration Implementation

• C1. Implement required migration set
  • Build migration artifacts for all approved Step 4 deltas.
• C2. Preserve compatibility where needed
  • Use staged expand/contract strategy when direct cutover is unsafe.
• C3. Add migration logging checkpoints
  • Log start, phase boundaries, completion, and failure details.
• C4. Verify post-migration schema state
  • Confirm expected tables/columns/constraints/indexes are present.

Deliverables

• migration artifacts/scripts for V1 target schema
• schema verification checklist outputs

Exit Criteria

• baseline-to-target forward migration executes successfully
• post-migration checks pass deterministically

---

Phase D — Rollback and Mitigation Strategy

• D1. Define rollback classes per migration
  • direct downgrade vs forward-fix vs backup-restore.
• D2. Create rollback decision tree
  • trigger conditions, safe stop points, and recovery path.
• D3. Align failure classification with error_handling.md
  • normalize migration failures into canonical categories:
    • infrastructure_transient_error
    • infrastructure_persistent_error
    • internal_unexpected_error (as needed)
• D4. Rehearse rollback flow
  • run at least one migration failure simulation and execute chosen recovery path.

Deliverables

• rollback/mitigation decision tree
• rehearsal evidence notes

Exit Criteria

• operator can execute rollback/mitigation without undocumented steps
• migration failure paths are diagnosable and classified

---

Phase E — Backfill Decision and Execution (Conditional)

• E1. Determine backfill necessity
  • inspect whether existing records violate new invariants.
• E2. If required, implement idempotent backfill
  • resumable, batch-safe, and deterministic update semantics.
• E3. Add post-backfill verification
  • validate:
    • revision sequencing integrity
    • accepted/current transcript consistency
    • job lifecycle consistency
• E4. If not required, record explicit “no backfill needed” evidence

Deliverables

• backfill script(s) and checklist (if applicable)
• no-backfill rationale artifact (if not applicable)

Exit Criteria

• required backfills completed and verified OR formally ruled out with evidence

---

Phase F — Verification and Test Expansion

Apply pytesting guidance (deterministic, behavior-first, strict markers).

• F1. Migration application tests
  • verify forward migration from representative baseline.
• F2. Post-migration schema contract tests
  • verify expected schema shape and key constraints.
• F3. Rollback/mitigation tests
  • verify chosen rollback class behavior where practical.
• F4. Startup policy regression tests
  • confirm production-mode startup does not mutate schema implicitly.
• F5. Backfill behavior tests (if applicable)
  • idempotency and invariants after repeated execution.

Validation Commands

• uv run pytest --collect-only -q
• uv run pytest -m unit -q
• uv run pytest -m "not external" -q
• uv run pytest -q

Exit Criteria

• all Step 4 migration-safety checks pass
• no REQ-10 regression introduced

---

Phase G — Runbook and Documentation Closure

• G1. Create migration and rollback runbook
  • include:
    • prerequisites
    • backup step
    • migration execution
    • verification
    • rollback/mitigation
• G2. Update traceability artifacts
  • map Step 4 outcomes to REQ IDs and evidence.
• G3. Prepare Step 4 handoff artifacts
  • ensure outputs feed Step 9 release readiness and Step 10 docs completion.

Deliverables

• docs/ver1/ver1-step4-migration-runbook.md (recommended)
• docs/ver1/ver1-step4-results.md
• updated traceability references where needed

Exit Criteria

• migration operations are executable using docs alone
• Step 4 evidence is complete and auditable

---

Recommended Implementation Order

1. Phase A — schema/invariant audit
2. Phase B — migration policy and tooling lock
3. Phase C — forward migration implementation
4. Phase D — rollback/mitigation strategy + rehearsal
5. Phase E — backfill decision and execution (conditional)
6. Phase F — test and verification expansion
7. Phase G — runbook + traceability closure

This sequence minimizes risk by locking policy and scope before irreversible data changes.

---

Risks and Mitigations

1. Risk: Data loss from unsafe schema transitions

   • Mitigation: backup-first gate, staged migration strategies, post-check verification.

2. Risk: Startup policy drift reintroduces implicit schema mutation

   • Mitigation: explicit regression tests for production startup behavior (REQ-10 guard).

3. Risk: Rollback path is incomplete or untested

   • Mitigation: mandatory rollback class declaration + rehearsal evidence.

4. Risk: Backfill scripts cause partial/inconsistent state

   • Mitigation: idempotent design, batching, and invariant-focused verification.

5. Risk: Migration failure diagnostics are unclear

   • Mitigation: structured logging + error category mapping per error_handling.md.

---

Step 4 Completion Checklist

• V1 schema audit completed and approved.
• Migration workflow policy is locked and documented.
• Required forward migrations are implemented and validated.
• Rollback/mitigation decision tree is documented and rehearsed.
• Backfill required/not-required decision is evidenced.
• Migration-safety test coverage is added and passing.
• Startup non-mutation policy remains verified in production mode.
• Step 4 runbook and results artifacts are completed.

---

Handoff to Step 5

Step 4 completion enables Step 5 (Private-Network Safety Baseline) with:

• stable, explicit schema evolution mechanics
• reduced upgrade risk for single-operator deployments
• migration/rollback procedures suitable for personal-scale production
• traceable evidence for release-readiness gates