# Step 4 Implementation Plan: Data Model and Migration Safety

## Purpose

Implement **Ver1 Step 4** from `docs/ver1/ver1.md` by making data-model evolution safe, explicit, and repeatable for personal-scale deployment.

Step 4 ensures schema changes are handled through deterministic migration workflows rather than implicit startup mutation, while preserving:

- personal-scale operational simplicity
- single-operator deployment model
- lifecycle-owned runtime resource boundaries
- stable requirement traceability and low rollback risk

Primary governing docs:

- `docs/ver1/ver1.md` (Step 4 objective and sequencing)
- `docs/architecture.md` (runtime ownership, persistence boundaries, simplicity guardrails)
- `docs/requirements.md` (REQ-3, REQ-4, REQ-7, REQ-10, REQ-11, REQ-12 emphasis)
- `docs/error_handling.md` (failure classification and safe error surfacing)
- `docs/intent.md` (verbatim/transcription/revision domain behavior)

---

## MCP Resources Reviewed and Applied

All currently available resources on `john-stream-mcp` were reviewed. Step 4 applies the following guidance directly:

1. `resource://skills/fastapi-async-sqlalchemy-modernization/document`
   - explicit engine/session lifecycle ownership
   - transaction boundary clarity for schema transitions and backfills
   - phased rollout with rollback-aware checkpoints

2. `resource://skills/pydantic-settings/document`
   - typed migration/runtime safety settings
   - explicit source-precedence behavior for operational toggles
   - fail-fast config semantics for unsafe startup paths

3. `resource://skills/pytesting/document`
   - deterministic migration verification lanes
   - strict marker discipline
   - behavior-first test coverage for migration outcomes

4. `resource://skills/python-logging-dictconfig/document`
   - startup-centralized logging configuration
   - structured migration and rollback event traceability

5. `resource://skills/fastapi-uv-docker/document`
   - deployment and rehearsal discipline
   - startup/health posture validation during migration windows

6. `resource://skills/python-typing/document`
   - modern typing hygiene for touched migration/persistence modules

7. `resource://skills/ruff-linting-formating/document`
   - lint/format consistency for migration scripts and database modules

Planning methodology inputs also applied:

8. `resource://prompts/greenfield-architecture/document`
   - staged execution with explicit risk and extension handling

9. `resource://prompts/pytest-scaffold/document`
10. `resource://prompts/pytest-fill-scaffold/document`
   - test-structure-first and deterministic fill-in sequencing

Reviewed but not directly Step 4 execution-critical:

- skills: `copilot-customization`, `mcp-details`, `nicegui`, `nicegui-ui-customization`, `vscode-configuration`, `zensical-docs`
- prompts: `authoring`, `mcp-consumer-repo-shim`

---

## Current-State Gap Summary (Step 4 Scope)

Based on Step 1–3 outcomes and current docs/tests:

1. **Bootstrap policy baseline is present**
   - Environment-aware schema bootstrap policy exists and aligns with REQ-10 intent.
2. **Functional model expanded in Step 3**
   - Revision/acceptance features introduce schema evolution requirements that need formal migration safety rehearsal.
3. **Runbook maturity required**
   - Step 4 requires explicit migration + rollback procedures and evidence.
4. **Backfill risk must be evaluated**
   - New/changed fields and semantics must be checked for historical data reconciliation needs.
5. **Release-path integration needed**
   - Step 4 artifacts must feed Step 9 release readiness and Step 10 docs completion.

---

## Scope for Step 4

### In scope

1. Validate final V1 schema against implemented domain behavior (post-Step 3 reality).
2. Define and implement forward-safe migration path for expected upgrades.
3. Define and document rollback/mitigation strategy for migration failures.
4. Implement backfill scripts only if required, with idempotent behavior.
5. Rehearse migration + rollback locally using representative sample data.
6. Add Step 4-specific verification tests and operational checks.
7. Produce operator-facing migration/rollback runbook and Step 4 results evidence.

### Out of scope

- Distributed/externally orchestrated migration systems
- Major persistence-architecture rewrites beyond V1 scope
- Non-V1 enhancement migrations unrelated to implemented requirement slices

---

## Target Decisions for Step 4

1. **Production startup remains non-mutating by default**
   - Preserve REQ-10 posture and avoid implicit schema mutation at normal startup.

2. **Schema changes are explicit operator workflows**
   - Migrations run as deliberate operational actions, not hidden side effects.

3. **Migration safety beats migration speed**
   - Additive and reversible-first patterns are preferred where possible.

4. **Rollback policy is explicit per change**
   - Each migration must declare rollback class:
     - direct rollback supported
     - forward-fix required
     - backup restore required

5. **Backfills are optional and minimal**
   - Introduce only when required by correctness/invariants, never by convenience.

6. **Migration observability is mandatory**
   - Structured logs include operation, migration identifier, status, and failure classification.

---

## Detailed Work Breakdown

## Phase A — Schema and Domain Invariant Audit

- [ ] **A1. Build canonical V1 schema inventory**
  - Enumerate all persisted entities and key fields:
    - document records
    - jobs and statuses
    - transcripts
    - transcript revisions
    - failure/provenance fields
- [ ] **A2. Validate invariants against implemented behavior**
  - Cross-check Step 3 functionality and current domain expectations:
    - append-only revision history
    - accepted revision semantics
    - canonical transcript synchronization behavior
- [ ] **A3. Classify required schema deltas**
  - Categorize deltas:
    - additive and safe
    - compatibility-sensitive
    - potentially destructive (must be staged or deferred)

### Deliverables

- `docs/ver1/ver1-step4-schema-audit.md` (recommended)
- schema-delta matrix with risk class and owning module

### Exit Criteria

- all required schema changes have explicit rationale and risk classification
- no ambiguous domain invariant remains

---

## Phase B — Migration Policy and Tooling Lock

- [ ] **B1. Lock migration workflow policy**
  - Define canonical migration execution path and artifact conventions.
- [ ] **B2. Define migration authoring checklist**
  - Include:
    - preconditions
    - forward steps
    - rollback class
    - post-verification checks
- [ ] **B3. Align policy with runtime startup safeguards**
  - Ensure production startup remains explicit/non-mutating by default.
- [ ] **B4. Define operator invocation standard**
  - One documented command path for local and production-like workflows.

### Deliverables

- migration policy section (this doc + runbook)
- migration authoring/review checklist

### Exit Criteria

- one unambiguous migration process exists and is documented
- startup policy and migration policy are consistent and non-conflicting

---

## Phase C — Forward Migration Implementation

- [ ] **C1. Implement required migration set**
  - Build migration artifacts for all approved Step 4 deltas.
- [ ] **C2. Preserve compatibility where needed**
  - Use staged expand/contract strategy when direct cutover is unsafe.
- [ ] **C3. Add migration logging checkpoints**
  - Log start, phase boundaries, completion, and failure details.
- [ ] **C4. Verify post-migration schema state**
  - Confirm expected tables/columns/constraints/indexes are present.

### Deliverables

- migration artifacts/scripts for V1 target schema
- schema verification checklist outputs

### Exit Criteria

- baseline-to-target forward migration executes successfully
- post-migration checks pass deterministically

---

## Phase D — Rollback and Mitigation Strategy

- [ ] **D1. Define rollback classes per migration**
  - direct downgrade vs forward-fix vs backup-restore.
- [ ] **D2. Create rollback decision tree**
  - trigger conditions, safe stop points, and recovery path.
- [ ] **D3. Align failure classification with `error_handling.md`**
  - normalize migration failures into canonical categories:
    - `infrastructure_transient_error`
    - `infrastructure_persistent_error`
    - `internal_unexpected_error` (as needed)
- [ ] **D4. Rehearse rollback flow**
  - run at least one migration failure simulation and execute chosen recovery path.

### Deliverables

- rollback/mitigation decision tree
- rehearsal evidence notes

### Exit Criteria

- operator can execute rollback/mitigation without undocumented steps
- migration failure paths are diagnosable and classified

---

## Phase E — Backfill Decision and Execution (Conditional)

- [ ] **E1. Determine backfill necessity**
  - inspect whether existing records violate new invariants.
- [ ] **E2. If required, implement idempotent backfill**
  - resumable, batch-safe, and deterministic update semantics.
- [ ] **E3. Add post-backfill verification**
  - validate:
    - revision sequencing integrity
    - accepted/current transcript consistency
    - job lifecycle consistency
- [ ] **E4. If not required, record explicit “no backfill needed” evidence**

### Deliverables

- backfill script(s) and checklist (if applicable)
- no-backfill rationale artifact (if not applicable)

### Exit Criteria

- required backfills completed and verified OR formally ruled out with evidence

---

## Phase F — Verification and Test Expansion

Apply `pytesting` guidance (deterministic, behavior-first, strict markers).

- [ ] **F1. Migration application tests**
  - verify forward migration from representative baseline.
- [ ] **F2. Post-migration schema contract tests**
  - verify expected schema shape and key constraints.
- [ ] **F3. Rollback/mitigation tests**
  - verify chosen rollback class behavior where practical.
- [ ] **F4. Startup policy regression tests**
  - confirm production-mode startup does not mutate schema implicitly.
- [ ] **F5. Backfill behavior tests (if applicable)**
  - idempotency and invariants after repeated execution.

### Validation Commands

- `uv run pytest --collect-only -q`
- `uv run pytest -m unit -q`
- `uv run pytest -m "not external" -q`
- `uv run pytest -q`

### Exit Criteria

- all Step 4 migration-safety checks pass
- no REQ-10 regression introduced

---

## Phase G — Runbook and Documentation Closure

- [ ] **G1. Create migration and rollback runbook**
  - include:
    - prerequisites
    - backup step
    - migration execution
    - verification
    - rollback/mitigation
- [ ] **G2. Update traceability artifacts**
  - map Step 4 outcomes to REQ IDs and evidence.
- [ ] **G3. Prepare Step 4 handoff artifacts**
  - ensure outputs feed Step 9 release readiness and Step 10 docs completion.

### Deliverables

- `docs/ver1/ver1-step4-migration-runbook.md` (recommended)
- `docs/ver1/ver1-step4-results.md`
- updated traceability references where needed

### Exit Criteria

- migration operations are executable using docs alone
- Step 4 evidence is complete and auditable

---

## Recommended Implementation Order

1. Phase A — schema/invariant audit
2. Phase B — migration policy and tooling lock
3. Phase C — forward migration implementation
4. Phase D — rollback/mitigation strategy + rehearsal
5. Phase E — backfill decision and execution (conditional)
6. Phase F — test and verification expansion
7. Phase G — runbook + traceability closure

This sequence minimizes risk by locking policy and scope before irreversible data changes.

---

## Risks and Mitigations

1. **Risk:** Data loss from unsafe schema transitions
   - **Mitigation:** backup-first gate, staged migration strategies, post-check verification.

2. **Risk:** Startup policy drift reintroduces implicit schema mutation
   - **Mitigation:** explicit regression tests for production startup behavior (REQ-10 guard).

3. **Risk:** Rollback path is incomplete or untested
   - **Mitigation:** mandatory rollback class declaration + rehearsal evidence.

4. **Risk:** Backfill scripts cause partial/inconsistent state
   - **Mitigation:** idempotent design, batching, and invariant-focused verification.

5. **Risk:** Migration failure diagnostics are unclear
   - **Mitigation:** structured logging + error category mapping per `error_handling.md`.

---

## Step 4 Completion Checklist

- [ ] V1 schema audit completed and approved.
- [ ] Migration workflow policy is locked and documented.
- [ ] Required forward migrations are implemented and validated.
- [ ] Rollback/mitigation decision tree is documented and rehearsed.
- [ ] Backfill required/not-required decision is evidenced.
- [ ] Migration-safety test coverage is added and passing.
- [ ] Startup non-mutation policy remains verified in production mode.
- [ ] Step 4 runbook and results artifacts are completed.

---

## Handoff to Step 5

Step 4 completion enables Step 5 (Private-Network Safety Baseline) with:

- stable, explicit schema evolution mechanics
- reduced upgrade risk for single-operator deployments
- migration/rollback procedures suitable for personal-scale production
- traceable evidence for release-readiness gates