generated from john/python-template
ver1 - Step 4 implemented - Revision functionality added
This commit is contained in:
Jim Lancaster
@@ -0,0 +1,113 @@
|
||||
# Ver1 Step 4 Migration and Rollback Runbook
|
||||
|
||||
## Purpose
|
||||
|
||||
Provide a concise, operator-safe procedure for schema migration execution,
|
||||
compatibility validation, and rollback/mitigation for personal-scale deployments.
|
||||
|
||||
This runbook supports `docs/ver1/ver1-step4.md` and REQ-10 by keeping normal
|
||||
production startup non-mutating unless explicitly configured otherwise.
|
||||
|
||||
---
|
||||
|
||||
## Preconditions
|
||||
|
||||
1. Application version to deploy is known and checked out.
|
||||
2. `.env` values are configured for target environment.
|
||||
3. Database backup path is prepared.
|
||||
4. Application process is stopped before migration on production-like systems.
|
||||
|
||||
---
|
||||
|
||||
## Commands
|
||||
|
||||
Use explicit migration runner operations:
|
||||
|
||||
1. List pending migrations:
|
||||
- `uv run python -m transcription.migration_runner --list`
|
||||
2. Apply pending migrations:
|
||||
- `uv run python -m transcription.migration_runner --apply`
|
||||
3. Validate schema compatibility:
|
||||
- `uv run python -m transcription.migration_runner --check`
|
||||
|
||||
Recommended execution order:
|
||||
|
||||
1. `--list`
|
||||
2. backup database
|
||||
3. `--apply`
|
||||
4. `--check`
|
||||
5. start application
|
||||
|
||||
---
|
||||
|
||||
## Backup Procedure (SQLite Baseline)
|
||||
|
||||
For SQLite deployments, copy the DB file before migration:
|
||||
|
||||
- Example DB path default: `./transcription.db`
|
||||
- Keep timestamped backup copy in a safe location.
|
||||
|
||||
If the file is in active use, stop the app first.
|
||||
|
||||
---
|
||||
|
||||
## Verification Checklist
|
||||
|
||||
After migration apply:
|
||||
|
||||
1. `--check` exits successfully.
|
||||
2. `schema_migration_history` includes applied revisions.
|
||||
3. Application starts successfully.
|
||||
4. Health endpoint responds: `/healthz`.
|
||||
5. Critical flows smoke-check:
|
||||
- upload
|
||||
- job processing
|
||||
- revision listing/acceptance
|
||||
|
||||
---
|
||||
|
||||
## Rollback and Mitigation Decision Tree
|
||||
|
||||
1. If migration fails before changes commit:
|
||||
- fix issue
|
||||
- re-run apply
|
||||
2. If migration partially applied or compatibility check fails:
|
||||
- stop app
|
||||
- restore from backup
|
||||
- investigate and produce forward-fix migration if needed
|
||||
3. If app starts but functional invariants fail:
|
||||
- stop app
|
||||
- restore backup
|
||||
- add corrective migration/backfill and rehearse before retry
|
||||
|
||||
For this Step 4 baseline, backup restore is the primary rollback mechanism.
|
||||
|
||||
---
|
||||
|
||||
## Failure Classification Guidance
|
||||
|
||||
Classify migration failures using `docs/error_handling.md` categories:
|
||||
|
||||
- transient connection issues -> `infrastructure_transient_error`
|
||||
- permissions/misconfiguration -> `infrastructure_persistent_error`
|
||||
- unexpected migration logic defects -> `internal_unexpected_error`
|
||||
|
||||
Record failure details with operation context and timestamp.
|
||||
|
||||
---
|
||||
|
||||
## Operational Notes
|
||||
|
||||
- `migration_auto_apply_on_startup` defaults to `False`.
|
||||
- `validate_schema_on_startup` defaults to `True`.
|
||||
- Startup schema validation fails fast on incompatibility.
|
||||
|
||||
This protects production from accidental schema drift.
|
||||
|
||||
---
|
||||
|
||||
## Post-Step-4 Follow-Up
|
||||
|
||||
If migration complexity grows beyond lightweight revision scripts,
|
||||
introduce a dedicated migration framework in a future step while preserving
|
||||
this runbook structure and operator-first workflow.
|
||||
@@ -2,111 +2,146 @@
|
||||
|
||||
## Summary
|
||||
|
||||
Step 4 implementation status: **in progress**.
|
||||
Step 4 implementation status: **complete (baseline scope)**.
|
||||
|
||||
This document records completed migration-safety work, validation evidence, and remaining follow-ups for Ver1 Step 4.
|
||||
|
||||
Implemented in this step:
|
||||
|
||||
1. _TBD_
|
||||
2. _TBD_
|
||||
3. _TBD_
|
||||
|
||||
1. Added explicit migration framework module with revision history tracking.
|
||||
2. Added schema compatibility validation and startup guardrails.
|
||||
3. Added migration runner CLI for list/apply/check operations.
|
||||
4. Added migration tests and Step 4 validation evidence.
|
||||
5. Added Step 4 migration/rollback runbook.
|
||||
---
|
||||
|
||||
## Implemented Changes
|
||||
|
||||
### 1) Schema audit and invariant lock
|
||||
|
||||
_TBD_
|
||||
Implemented read-only compatibility checks in `src/transcription/db.py`:
|
||||
|
||||
- `validate_schema_compatibility(...)` verifies required V1 tables:
|
||||
- `document`
|
||||
- `job`
|
||||
- `transcript`
|
||||
- `transcriptrevision`
|
||||
- verifies required `job.retry_count` column
|
||||
- returns explicit issue identifiers (non-mutating check)
|
||||
|
||||
### 2) Migration policy/tooling lock
|
||||
|
||||
_TBD_
|
||||
Added explicit migration revision model in `src/transcription/migrations.py`:
|
||||
|
||||
- `MigrationRevision` dataclass
|
||||
- ordered `MIGRATIONS` registry
|
||||
- migration history table: `schema_migration_history`
|
||||
- explicit pending-list and apply operations
|
||||
|
||||
### 3) Forward migration implementation
|
||||
|
||||
_TBD_
|
||||
Implemented two baseline forward migrations:
|
||||
|
||||
1. `0001_add_retry_count_to_job`
|
||||
2. `0002_create_transcriptrevision_table`
|
||||
|
||||
Each migration is idempotent and recorded in migration history.
|
||||
|
||||
### 4) Rollback and mitigation runbook
|
||||
|
||||
_TBD_
|
||||
Created `docs/ver1/ver1-step4-migration-runbook.md` with:
|
||||
|
||||
- preconditions
|
||||
- list/apply/check command sequence
|
||||
- backup-first procedure
|
||||
- verification checklist
|
||||
- rollback/mitigation decision tree
|
||||
- error classification guidance aligned to `docs/error_handling.md`
|
||||
|
||||
### 5) Backfill implementation or explicit no-backfill decision
|
||||
|
||||
_TBD_
|
||||
No backfill required for this baseline Step 4 scope.
|
||||
|
||||
Rationale:
|
||||
|
||||
- additive migration operations only
|
||||
- default values and new-table creation do not require historical row rewrites for current V1 invariants
|
||||
- residual advanced backfill scenarios deferred unless future schema evolution introduces incompatible transforms
|
||||
---
|
||||
|
||||
## Test and Verification Evidence
|
||||
|
||||
### Added/Updated Tests
|
||||
|
||||
1. _TBD_
|
||||
2. _TBD_
|
||||
3. _TBD_
|
||||
|
||||
1. `tests/test_migrations.py`
|
||||
- pending migration discovery
|
||||
- migration apply + history recording
|
||||
- idempotent re-apply behavior
|
||||
2. `tests/test_db.py`
|
||||
- compatibility-check behavior on fresh schema
|
||||
- table expectation updates for `transcriptrevision`
|
||||
3. `tests/test_config.py`
|
||||
- migration safety setting defaults
|
||||
4. `tests/test_app.py`
|
||||
- lifespan test compatibility with migration/validation startup hooks
|
||||
### Validation Runs
|
||||
|
||||
Run and record outcomes:
|
||||
|
||||
- `uv run pytest --collect-only -q` -> _TBD_
|
||||
- `uv run pytest -m unit -q` -> _TBD_
|
||||
- `uv run pytest -m "not external" -q` -> _TBD_
|
||||
- `uv run pytest -q` -> _TBD_
|
||||
|
||||
- `uv run pytest --collect-only -q` -> passed
|
||||
- `uv run pytest -m unit -q` -> passed
|
||||
- `uv run pytest -m "not external" -q` -> passed
|
||||
- `uv run pytest -q` -> passed
|
||||
### Migration Rehearsal Evidence
|
||||
|
||||
Record migration rehearsal details:
|
||||
|
||||
- baseline data set used: _TBD_
|
||||
- forward migration result: _TBD_
|
||||
- post-migration verification result: _TBD_
|
||||
- rollback/mitigation rehearsal result: _TBD_
|
||||
Migration rehearsal details (test-based):
|
||||
|
||||
- baseline data set used: in-memory SQLite legacy-shaped schema fixture (`job` table missing Step 4 additions)
|
||||
- forward migration result: pending revisions applied successfully (`0001`, `0002`)
|
||||
- post-migration verification result: schema checks pass and migration history recorded
|
||||
- rollback/mitigation rehearsal result: runbook defined backup-restore primary rollback class for personal-scale SQLite deployment
|
||||
---
|
||||
|
||||
## Requirement Traceability (Step 4)
|
||||
|
||||
| Step 4 Area | REQ Coverage | Status | Evidence |
|
||||
| --- | --- | --- | --- |
|
||||
| Schema lifecycle and state persistence safety | REQ-3, REQ-4, REQ-11 | _TBD_ | _TBD_ |
|
||||
| Lifespan/runtime ownership continuity | REQ-7 | _TBD_ | _TBD_ |
|
||||
| Explicit non-mutating production startup policy | REQ-10 | _TBD_ | _TBD_ |
|
||||
| Prompt/data continuity constraints | REQ-12 | _TBD_ | _TBD_ |
|
||||
|
||||
| Schema lifecycle and state persistence safety | REQ-3, REQ-4, REQ-11 | met | `src/transcription/migrations.py`, `tests/test_migrations.py`, `tests/test_db.py` |
|
||||
| Lifespan/runtime ownership continuity | REQ-7 | met | `src/transcription/app.py` startup checks + existing lifespan ownership model |
|
||||
| Explicit non-mutating production startup policy | REQ-10 | met | `migration_auto_apply_on_startup=False` default + explicit runner workflow + startup validation gate |
|
||||
| Prompt/data continuity constraints | REQ-12 | met (continued) | no prompt-contract mutation in Step 4 changes |
|
||||
---
|
||||
|
||||
## Operational Artifacts Produced
|
||||
|
||||
- `docs/ver1/ver1-step4.md`
|
||||
- `docs/ver1/ver1-step4-migration-runbook.md` (_if created_)
|
||||
- _TBD additional artifacts_
|
||||
|
||||
- `docs/ver1/ver1-step4-migration-runbook.md`
|
||||
- `src/transcription/migrations.py`
|
||||
- `src/transcription/migration_runner.py`
|
||||
- README migration workflow updates
|
||||
---
|
||||
|
||||
## Risks, Exceptions, and Follow-Ups
|
||||
|
||||
1. _TBD_
|
||||
2. _TBD_
|
||||
3. _TBD_
|
||||
1. This lightweight migration system is appropriate for current personal-scale scope but may require a dedicated framework as schema complexity grows.
|
||||
2. Rollback remains backup-restore primary; reversible down-migration coverage is intentionally limited in this baseline.
|
||||
3. Startup compatibility checks currently fail fast with generic runtime error text and can be further normalized under API/operator error envelopes in later hardening.
|
||||
|
||||
Open follow-ups to carry forward:
|
||||
|
||||
- _TBD_
|
||||
|
||||
- Evaluate migration framework escalation criteria in Step 9/10 readiness updates.
|
||||
- Add optional richer structured migration logging fields if observability scope expands.
|
||||
---
|
||||
|
||||
## Step 4 Exit Assessment
|
||||
|
||||
- Schema validation against finalized V1 domain: **_TBD_**
|
||||
- Forward migration path safety and repeatability: **_TBD_**
|
||||
- Rollback/mitigation readiness: **_TBD_**
|
||||
- Backfill risk closure: **_TBD_**
|
||||
- Test and regression safety: **_TBD_**
|
||||
|
||||
Step 4 completion status: **_TBD_**
|
||||
- Schema validation against finalized V1 domain: **met**
|
||||
- Forward migration path safety and repeatability: **met (baseline scope)**
|
||||
- Rollback/mitigation readiness: **met (backup-restore primary path)**
|
||||
- Backfill risk closure: **met (no backfill required for current deltas)**
|
||||
- Test and regression safety: **met**
|
||||
|
||||
Step 4 completion status: **complete (baseline scope)**
|
||||
---
|
||||
|
||||
## Handoff to Step 5
|
||||
|
||||
Reference in New Issue
Block a user