diff --git a/README.md b/README.md index e152d6c..d11c094 100644 --- a/README.md +++ b/README.md @@ -42,11 +42,32 @@ PROMPT_DIR=./prompts uv run uvicorn transcription.app:create_app --factory --reload ``` -### 4) Open in browser +### 4) (Optional) Run explicit migrations/checks + +Use the migration runner for Step 4 schema safety workflows: + +```bash +uv run python -m transcription.migration_runner --list +uv run python -m transcription.migration_runner --apply +uv run python -m transcription.migration_runner --check +``` + +### 5) Open in browser + - GUI: [http://[IP_ADDRESS]:8000/ui](http://[IP_ADDRESS]:8000/ui) - Health check: [http://[IP_ADDRESS]:8000/healthz](http://[IP_ADDRESS]:8000/healthz) +### Schema safety settings + +Optional environment settings (defaults shown): + +```env +MIGRATION_AUTO_APPLY_ON_STARTUP=false +VALIDATE_SCHEMA_ON_STARTUP=true +``` + + ## How to navigate the GUI - **Upload page** (`/ui`) diff --git a/docs/ver1/ver1-step4-migration-runbook.md b/docs/ver1/ver1-step4-migration-runbook.md new file mode 100644 index 0000000..a8c5881 --- /dev/null +++ b/docs/ver1/ver1-step4-migration-runbook.md @@ -0,0 +1,113 @@ +# Ver1 Step 4 Migration and Rollback Runbook + +## Purpose + +Provide a concise, operator-safe procedure for schema migration execution, +compatibility validation, and rollback/mitigation for personal-scale deployments. + +This runbook supports `docs/ver1/ver1-step4.md` and REQ-10 by keeping normal +production startup non-mutating unless explicitly configured otherwise. + +--- + +## Preconditions + +1. Application version to deploy is known and checked out. +2. `.env` values are configured for target environment. +3. Database backup path is prepared. +4. Application process is stopped before migration on production-like systems. + +--- + +## Commands + +Use explicit migration runner operations: + +1. List pending migrations: + - `uv run python -m transcription.migration_runner --list` +2. Apply pending migrations: + - `uv run python -m transcription.migration_runner --apply` +3. Validate schema compatibility: + - `uv run python -m transcription.migration_runner --check` + +Recommended execution order: + +1. `--list` +2. backup database +3. `--apply` +4. `--check` +5. start application + +--- + +## Backup Procedure (SQLite Baseline) + +For SQLite deployments, copy the DB file before migration: + +- Example DB path default: `./transcription.db` +- Keep timestamped backup copy in a safe location. + +If the file is in active use, stop the app first. + +--- + +## Verification Checklist + +After migration apply: + +1. `--check` exits successfully. +2. `schema_migration_history` includes applied revisions. +3. Application starts successfully. +4. Health endpoint responds: `/healthz`. +5. Critical flows smoke-check: + - upload + - job processing + - revision listing/acceptance + +--- + +## Rollback and Mitigation Decision Tree + +1. If migration fails before changes commit: + - fix issue + - re-run apply +2. If migration partially applied or compatibility check fails: + - stop app + - restore from backup + - investigate and produce forward-fix migration if needed +3. If app starts but functional invariants fail: + - stop app + - restore backup + - add corrective migration/backfill and rehearse before retry + +For this Step 4 baseline, backup restore is the primary rollback mechanism. + +--- + +## Failure Classification Guidance + +Classify migration failures using `docs/error_handling.md` categories: + +- transient connection issues -> `infrastructure_transient_error` +- permissions/misconfiguration -> `infrastructure_persistent_error` +- unexpected migration logic defects -> `internal_unexpected_error` + +Record failure details with operation context and timestamp. + +--- + +## Operational Notes + +- `migration_auto_apply_on_startup` defaults to `False`. +- `validate_schema_on_startup` defaults to `True`. +- Startup schema validation fails fast on incompatibility. + +This protects production from accidental schema drift. + +--- + +## Post-Step-4 Follow-Up + +If migration complexity grows beyond lightweight revision scripts, +introduce a dedicated migration framework in a future step while preserving +this runbook structure and operator-first workflow. \ No newline at end of file diff --git a/docs/ver1/ver1-step4-results.md b/docs/ver1/ver1-step4-results.md index 839ef63..b12fc64 100644 --- a/docs/ver1/ver1-step4-results.md +++ b/docs/ver1/ver1-step4-results.md @@ -2,111 +2,146 @@ ## Summary -Step 4 implementation status: **in progress**. +Step 4 implementation status: **complete (baseline scope)**. This document records completed migration-safety work, validation evidence, and remaining follow-ups for Ver1 Step 4. Implemented in this step: -1. _TBD_ -2. _TBD_ -3. _TBD_ - +1. Added explicit migration framework module with revision history tracking. +2. Added schema compatibility validation and startup guardrails. +3. Added migration runner CLI for list/apply/check operations. +4. Added migration tests and Step 4 validation evidence. +5. Added Step 4 migration/rollback runbook. --- ## Implemented Changes ### 1) Schema audit and invariant lock -_TBD_ +Implemented read-only compatibility checks in `src/transcription/db.py`: + +- `validate_schema_compatibility(...)` verifies required V1 tables: + - `document` + - `job` + - `transcript` + - `transcriptrevision` +- verifies required `job.retry_count` column +- returns explicit issue identifiers (non-mutating check) ### 2) Migration policy/tooling lock -_TBD_ +Added explicit migration revision model in `src/transcription/migrations.py`: + +- `MigrationRevision` dataclass +- ordered `MIGRATIONS` registry +- migration history table: `schema_migration_history` +- explicit pending-list and apply operations ### 3) Forward migration implementation -_TBD_ +Implemented two baseline forward migrations: + +1. `0001_add_retry_count_to_job` +2. `0002_create_transcriptrevision_table` + +Each migration is idempotent and recorded in migration history. ### 4) Rollback and mitigation runbook -_TBD_ +Created `docs/ver1/ver1-step4-migration-runbook.md` with: + +- preconditions +- list/apply/check command sequence +- backup-first procedure +- verification checklist +- rollback/mitigation decision tree +- error classification guidance aligned to `docs/error_handling.md` ### 5) Backfill implementation or explicit no-backfill decision -_TBD_ +No backfill required for this baseline Step 4 scope. +Rationale: + +- additive migration operations only +- default values and new-table creation do not require historical row rewrites for current V1 invariants +- residual advanced backfill scenarios deferred unless future schema evolution introduces incompatible transforms --- ## Test and Verification Evidence ### Added/Updated Tests -1. _TBD_ -2. _TBD_ -3. _TBD_ - +1. `tests/test_migrations.py` + - pending migration discovery + - migration apply + history recording + - idempotent re-apply behavior +2. `tests/test_db.py` + - compatibility-check behavior on fresh schema + - table expectation updates for `transcriptrevision` +3. `tests/test_config.py` + - migration safety setting defaults +4. `tests/test_app.py` + - lifespan test compatibility with migration/validation startup hooks ### Validation Runs Run and record outcomes: -- `uv run pytest --collect-only -q` -> _TBD_ -- `uv run pytest -m unit -q` -> _TBD_ -- `uv run pytest -m "not external" -q` -> _TBD_ -- `uv run pytest -q` -> _TBD_ - +- `uv run pytest --collect-only -q` -> passed +- `uv run pytest -m unit -q` -> passed +- `uv run pytest -m "not external" -q` -> passed +- `uv run pytest -q` -> passed ### Migration Rehearsal Evidence -Record migration rehearsal details: - -- baseline data set used: _TBD_ -- forward migration result: _TBD_ -- post-migration verification result: _TBD_ -- rollback/mitigation rehearsal result: _TBD_ +Migration rehearsal details (test-based): +- baseline data set used: in-memory SQLite legacy-shaped schema fixture (`job` table missing Step 4 additions) +- forward migration result: pending revisions applied successfully (`0001`, `0002`) +- post-migration verification result: schema checks pass and migration history recorded +- rollback/mitigation rehearsal result: runbook defined backup-restore primary rollback class for personal-scale SQLite deployment --- ## Requirement Traceability (Step 4) | Step 4 Area | REQ Coverage | Status | Evidence | | --- | --- | --- | --- | -| Schema lifecycle and state persistence safety | REQ-3, REQ-4, REQ-11 | _TBD_ | _TBD_ | -| Lifespan/runtime ownership continuity | REQ-7 | _TBD_ | _TBD_ | -| Explicit non-mutating production startup policy | REQ-10 | _TBD_ | _TBD_ | -| Prompt/data continuity constraints | REQ-12 | _TBD_ | _TBD_ | - +| Schema lifecycle and state persistence safety | REQ-3, REQ-4, REQ-11 | met | `src/transcription/migrations.py`, `tests/test_migrations.py`, `tests/test_db.py` | +| Lifespan/runtime ownership continuity | REQ-7 | met | `src/transcription/app.py` startup checks + existing lifespan ownership model | +| Explicit non-mutating production startup policy | REQ-10 | met | `migration_auto_apply_on_startup=False` default + explicit runner workflow + startup validation gate | +| Prompt/data continuity constraints | REQ-12 | met (continued) | no prompt-contract mutation in Step 4 changes | --- ## Operational Artifacts Produced - `docs/ver1/ver1-step4.md` -- `docs/ver1/ver1-step4-migration-runbook.md` (_if created_) -- _TBD additional artifacts_ - +- `docs/ver1/ver1-step4-migration-runbook.md` +- `src/transcription/migrations.py` +- `src/transcription/migration_runner.py` +- README migration workflow updates --- ## Risks, Exceptions, and Follow-Ups -1. _TBD_ -2. _TBD_ -3. _TBD_ +1. This lightweight migration system is appropriate for current personal-scale scope but may require a dedicated framework as schema complexity grows. +2. Rollback remains backup-restore primary; reversible down-migration coverage is intentionally limited in this baseline. +3. Startup compatibility checks currently fail fast with generic runtime error text and can be further normalized under API/operator error envelopes in later hardening. Open follow-ups to carry forward: -- _TBD_ - +- Evaluate migration framework escalation criteria in Step 9/10 readiness updates. +- Add optional richer structured migration logging fields if observability scope expands. --- ## Step 4 Exit Assessment -- Schema validation against finalized V1 domain: **_TBD_** -- Forward migration path safety and repeatability: **_TBD_** -- Rollback/mitigation readiness: **_TBD_** -- Backfill risk closure: **_TBD_** -- Test and regression safety: **_TBD_** - -Step 4 completion status: **_TBD_** +- Schema validation against finalized V1 domain: **met** +- Forward migration path safety and repeatability: **met (baseline scope)** +- Rollback/mitigation readiness: **met (backup-restore primary path)** +- Backfill risk closure: **met (no backfill required for current deltas)** +- Test and regression safety: **met** +Step 4 completion status: **complete (baseline scope)** --- ## Handoff to Step 5 diff --git a/src/transcription/app.py b/src/transcription/app.py index a65b572..cfd8c6a 100644 --- a/src/transcription/app.py +++ b/src/transcription/app.py @@ -16,7 +16,9 @@ from transcription.db import ( dispose_database_runtime, initialize_database_runtime, should_bootstrap_schema, + validate_schema_compatibility, ) +from transcription.migrations import apply_pending_migrations from transcription.ui import register_pages from transcription.worker import run_worker_loop @@ -58,6 +60,15 @@ async def _lifespan(app: FastAPI): if should_bootstrap_schema(settings): create_all(engine=app.state.db_runtime.engine) + if settings.migration_auto_apply_on_startup: + apply_pending_migrations(engine=app.state.db_runtime.engine) + + if settings.validate_schema_on_startup: + compatibility_issues = validate_schema_compatibility(engine=app.state.db_runtime.engine) + if compatibility_issues: + issues_text = ", ".join(compatibility_issues) + raise RuntimeError(f"Schema compatibility check failed: {issues_text}") + settings.upload_dir.mkdir(parents=True, exist_ok=True) settings.prompt_dir.mkdir(parents=True, exist_ok=True) diff --git a/src/transcription/config.py b/src/transcription/config.py index 05b3ed1..50be5e5 100644 --- a/src/transcription/config.py +++ b/src/transcription/config.py @@ -39,6 +39,8 @@ class Settings(BaseSettings): # --- persistence --- database_url: str = "sqlite:///./transcription.db" bootstrap_schema_on_startup: bool | None = None + migration_auto_apply_on_startup: bool = False + validate_schema_on_startup: bool = True # --- filesystem paths --- upload_dir: Path = Path("./uploads") diff --git a/src/transcription/db.py b/src/transcription/db.py index 67f75fc..27fcd25 100644 --- a/src/transcription/db.py +++ b/src/transcription/db.py @@ -5,17 +5,15 @@ startup/shutdown behavior is predictable and lifespan-managed. """ import contextlib -import logging from collections.abc import Generator from dataclasses import dataclass -from sqlalchemy import inspect, text +from sqlalchemy import inspect from sqlalchemy.engine import Engine from sqlmodel import Session, SQLModel, create_engine from transcription.config import Settings, get_settings -logger = logging.getLogger(__name__) @dataclass(frozen=True) @@ -80,32 +78,30 @@ def create_all(*, engine: Engine | None = None) -> None: active_engine = engine or get_database_runtime().engine SQLModel.metadata.create_all(active_engine) - _ensure_sqlite_compat_columns(active_engine) -def _ensure_sqlite_compat_columns(engine: Engine) -> None: - """Apply lightweight dev/test SQLite compatibility column patches. +def validate_schema_compatibility(*, engine: Engine | None = None) -> list[str]: + """Return schema compatibility issues for known V1 requirements. - This keeps local bootstrap resilient when models evolve but no full - migration tooling is in place yet. + This performs read-only validation and never mutates schema. """ - if engine.url.get_backend_name() != "sqlite": - return + active_engine = engine or get_database_runtime().engine + inspector = inspect(active_engine) - inspector = inspect(engine) + issues: list[str] = [] table_names = set(inspector.get_table_names()) - if "job" not in table_names: - return - columns = {column["name"] for column in inspector.get_columns("job")} - if "retry_count" not in columns: - with engine.begin() as connection: - connection.execute( - text("ALTER TABLE job ADD COLUMN retry_count INTEGER NOT NULL DEFAULT 0") - ) - logger.warning( - "Applied SQLite compatibility schema patch table=job column=retry_count default=0" - ) + required_tables = {"document", "job", "transcript", "transcriptrevision"} + missing_tables = sorted(required_tables - table_names) + for table_name in missing_tables: + issues.append(f"missing_table:{table_name}") + + if "job" in table_names: + job_columns = {column["name"] for column in inspector.get_columns("job")} + if "retry_count" not in job_columns: + issues.append("missing_column:job.retry_count") + + return issues @contextlib.contextmanager diff --git a/src/transcription/migration_runner.py b/src/transcription/migration_runner.py new file mode 100644 index 0000000..97adf2e --- /dev/null +++ b/src/transcription/migration_runner.py @@ -0,0 +1,74 @@ +"""CLI entrypoint for explicit schema migration and compatibility checks.""" + +from __future__ import annotations + +import argparse +import sys + +from transcription.config import get_settings +from transcription.db import initialize_database_runtime, validate_schema_compatibility +from transcription.migrations import apply_pending_migrations, list_pending_migrations + + +def _build_parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser(description="Transcription schema migration runner") + parser.add_argument( + "--apply", + action="store_true", + help="Apply all pending migrations.", + ) + parser.add_argument( + "--list", + action="store_true", + help="List pending migrations.", + ) + parser.add_argument( + "--check", + action="store_true", + help="Run schema compatibility check.", + ) + return parser + + +def main() -> int: + parser = _build_parser() + args = parser.parse_args() + + if not (args.apply or args.list or args.check): + parser.error("Specify at least one action: --list, --apply, or --check") + + runtime = initialize_database_runtime(settings=get_settings()) + engine = runtime.engine + + if args.list: + pending = list_pending_migrations(engine=engine) + if not pending: + print("No pending migrations.") + else: + print("Pending migrations:") + for migration in pending: + print(f"- {migration.revision_id}: {migration.description}") + + if args.apply: + applied = apply_pending_migrations(engine=engine) + if not applied: + print("No migrations applied.") + else: + print("Applied migrations:") + for revision_id in applied: + print(f"- {revision_id}") + + if args.check: + issues = validate_schema_compatibility(engine=engine) + if issues: + print("Schema compatibility check failed:") + for issue in issues: + print(f"- {issue}") + return 1 + print("Schema compatibility check passed.") + + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) \ No newline at end of file diff --git a/src/transcription/migrations.py b/src/transcription/migrations.py new file mode 100644 index 0000000..9d06d17 --- /dev/null +++ b/src/transcription/migrations.py @@ -0,0 +1,128 @@ +"""Lightweight schema migration helpers for V1 Step 4. + +This module provides explicit, operator-invoked migration execution for +personal-scale deployments without introducing heavyweight migration tooling. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from datetime import datetime, timezone +import logging +from typing import Callable + +from sqlalchemy import inspect, text +from sqlalchemy.engine import Connection, Engine + +logger = logging.getLogger(__name__) + + +@dataclass(frozen=True) +class MigrationRevision: + """Represents one ordered schema migration revision.""" + + revision_id: str + description: str + apply: Callable[[Connection], None] + + +def _ensure_history_table(connection: Connection) -> None: + """Create migration history table when missing.""" + connection.execute( + text( + """ + CREATE TABLE IF NOT EXISTS schema_migration_history ( + revision_id VARCHAR(64) PRIMARY KEY, + description VARCHAR(255) NOT NULL, + applied_at VARCHAR(64) NOT NULL + ) + """ + ) + ) + + +def _get_applied_revisions(connection: Connection) -> set[str]: + """Return applied migration revision IDs.""" + _ensure_history_table(connection) + rows = connection.execute(text("SELECT revision_id FROM schema_migration_history")).fetchall() + return {row[0] for row in rows} + + +def _record_revision(connection: Connection, revision: MigrationRevision) -> None: + """Persist one applied migration revision record.""" + connection.execute( + text( + """ + INSERT INTO schema_migration_history (revision_id, description, applied_at) + VALUES (:revision_id, :description, :applied_at) + """ + ), + { + "revision_id": revision.revision_id, + "description": revision.description, + "applied_at": datetime.now(timezone.utc).isoformat(), + }, + ) + + +def _apply_0001_add_retry_count(connection: Connection) -> None: + """Ensure job.retry_count exists for legacy databases.""" + inspector = inspect(connection) + table_names = set(inspector.get_table_names()) + if "job" not in table_names: + return + + columns = {column["name"] for column in inspector.get_columns("job")} + if "retry_count" in columns: + return + + # Compatible with SQLite and PostgreSQL for this additive integer column. + connection.execute(text("ALTER TABLE job ADD COLUMN retry_count INTEGER NOT NULL DEFAULT 0")) + + +def _apply_0002_create_transcriptrevision(connection: Connection) -> None: + """Ensure transcriptrevision table exists.""" + # Import models lazily so metadata is fully populated. + from transcription.models import TranscriptRevision # noqa: F401 + from sqlmodel import SQLModel + + table = SQLModel.metadata.tables["transcriptrevision"] + table.create(bind=connection, checkfirst=True) + + +MIGRATIONS: tuple[MigrationRevision, ...] = ( + MigrationRevision( + revision_id="0001_add_retry_count_to_job", + description="Add retry_count column to job table with default 0", + apply=_apply_0001_add_retry_count, + ), + MigrationRevision( + revision_id="0002_create_transcriptrevision_table", + description="Create transcriptrevision table for immutable transcript history", + apply=_apply_0002_create_transcriptrevision, + ), +) + + +def list_pending_migrations(*, engine: Engine) -> list[MigrationRevision]: + """Return pending migrations ordered by revision.""" + with engine.begin() as connection: + applied = _get_applied_revisions(connection) + return [revision for revision in MIGRATIONS if revision.revision_id not in applied] + + +def apply_pending_migrations(*, engine: Engine) -> list[str]: + """Apply all pending migrations and return applied revision IDs.""" + pending = list_pending_migrations(engine=engine) + applied_ids: list[str] = [] + + for revision in pending: + logger.info("Applying migration revision=%s", revision.revision_id) + with engine.begin() as connection: + _ensure_history_table(connection) + revision.apply(connection) + _record_revision(connection, revision) + applied_ids.append(revision.revision_id) + logger.info("Applied migration revision=%s", revision.revision_id) + + return applied_ids \ No newline at end of file diff --git a/tests/artifacts/transcriptions/Book_Two_-_page_02.jpg.txt b/tests/artifacts/transcriptions/Book_Two_-_page_02.jpg.txt index 29d7898..3667f7e 100644 --- a/tests/artifacts/transcriptions/Book_Two_-_page_02.jpg.txt +++ b/tests/artifacts/transcriptions/Book_Two_-_page_02.jpg.txt @@ -32,7 +32,7 @@ enough pictures but we had to take only part of them. We think there are great possibilities in reproducing old pictures. We wish we had a Pickard group. Some Pickard descendant may wish to make a collection. -We are much impressed with the future possibilities of getting a complete geneal- +We are much impressed with the future possibilities of getting a complete geneol-[sic] ogy of the Pickard family. Mr. Cochran has a fine chapter on the Pickards but to date we have not had the pleasure of finding all of the family dates. We had intended to give more family data in this book but it takes time to get the diff --git a/tests/artifacts/transcriptions/Omie_Writes_Home.pdf.txt b/tests/artifacts/transcriptions/Omie_Writes_Home.pdf.txt index 161a800..824fb14 100644 --- a/tests/artifacts/transcriptions/Omie_Writes_Home.pdf.txt +++ b/tests/artifacts/transcriptions/Omie_Writes_Home.pdf.txt @@ -6,9 +6,7 @@ JOHN E. COCHRAN FAMILY ASSOCIATION Family Only Home | Sibling's Stories | 1st Cousins | Ancestor's Stories | Reunion History | Next Reunion | JECMEF - OMIE WRITES HOME - Ed. Note: The following letter was written by Omie Cochran in Nome, Alaska and sent to her sister, Ethel Shinn, in Canfield, Idaho in 1923. It has been stored away these 63 years in the original envelope with its 2 cent stamp. The letter has a number of references to the Shinn @@ -17,7 +15,9 @@ Miss Saville was the nurse at the Nome Hospital that was mentioned in the articl the family newsletter two years ago. Nome Alaska August 26, 1923 + My Dear Ethel et al. + I don't know when I did write or when you did but I am going to write now however and never the less. But I wish I could talk (I can yet but I @@ -26,7 +26,7 @@ and Polly sit up and listen and that little black rascal of yours would fairly sparkle with listening. Can't I see him listening now to all the yarns we told last summer? -[photo of people on ice with ship in background] +[photo of people on ice with kayak and dog sled] You see, we-Miss Saville and I, took a trip north on the Buford and it was very interesting. We went north thru the Bering Strait into the Arctic and as far as the Ice Pack. There the captain @@ -37,7 +37,6 @@ along the icebergs and the walrus were 'heisted' on board by the use of the cran tons of freight and the beasts were so huge that they made the pulleys just creak. They were over 12 feet long, as big around as three cows, had no feet but toenails on their flippers or flappers, no head but their body just suddenly ended with a hole for a mouth and big bristles - all around it similar to a currycomb in coarseness; no ears but huge tusks of ivory. They are the most repulsive looking animals imaginable and tho I have always read about them I never expect such disagreeable looking creatures. They had a rough brown hairy skin and some of @@ -80,7 +79,6 @@ us talk. Miss Saville talked quite a bit. Any how if you folks don't like this I all I had to write about and I know Buster'ud listen anyway and I'd soak ole Peter's head if he didn't and Polly would in my lap and I don't know much about the youngest one of yours so likely he would be squawling. But we did surely enjoy our trip and were gone just long enuf. - I expect there were 150 passengers on board and almost or more of the crew and helpers. We had a stateroom down next to the kitchen and 'twas pretty fierce for odor at times. diff --git a/tests/artifacts/transcriptions/Rod_Moser_Letter_-_p1.jpg.txt b/tests/artifacts/transcriptions/Rod_Moser_Letter_-_p1.jpg.txt index 357bf9d..7454d1a 100644 --- a/tests/artifacts/transcriptions/Rod_Moser_Letter_-_p1.jpg.txt +++ b/tests/artifacts/transcriptions/Rod_Moser_Letter_-_p1.jpg.txt @@ -2,30 +2,28 @@ source: Rod Moser Letter - p1.jpg provider: openrouter model: google/gemini-2.5-flash --- -JOHN ISBILL -R. T. MOSER - +JOHN ISBILL R. T. MOSER ISBILL & MOSER DEALERS IN GENERAL MERCHANDISE -Vonore, Tenn. January 27 – 1913 -Dear Uncle Aunt Adeline +Vonore, Tenn. January 27 - 1913 +Dear Uncle [sic] Aun[t Adeline?] Was at home a few nights ago & saw a -letter from your folks, So +letter from you folks, so I decided to write you -a few lines myself ok -I am contemplate a -trip out west next summer -& [inserted: I] would like of adders [sic] to go -where I [inserted: am] them. +a few lines myself & +I am contemplating a +trip out west next summ[er] +& I want both of fillers [sic] to go +when I am [to] them. Am getting -up in years & unmarried +up in years & unmarried, so you see the object of -my trip is to get a wife -& if there is any old maid -or widows out there, I -want you to kiss them -at my [inserted: mind] for me at there [sic] -as soon as I get there. +my trip, is to get a wife +& I hear is a lot old maids +& widows out there. I +want you to see them +at my land my [sic] at there [sic] +as soon as I get there diff --git a/tests/test_app.py b/tests/test_app.py index 302a6f8..cdc0132 100644 --- a/tests/test_app.py +++ b/tests/test_app.py @@ -35,6 +35,8 @@ class TestAppLifespan: monkeypatch.setattr("transcription.app.should_bootstrap_schema", lambda _settings: True) monkeypatch.setattr("transcription.app._start_worker", lambda _app: calls.append("start_worker")) monkeypatch.setattr("transcription.app._stop_worker", lambda _app: calls.append("stop_worker")) + monkeypatch.setattr("transcription.app.apply_pending_migrations", lambda **_kwargs: calls.append("migrate")) + monkeypatch.setattr("transcription.app.validate_schema_compatibility", lambda **_kwargs: []) class _Dir: def mkdir(self, parents: bool, exist_ok: bool): @@ -43,6 +45,8 @@ class TestAppLifespan: class _Settings: upload_dir = _Dir() prompt_dir = _Dir() + migration_auto_apply_on_startup = False + validate_schema_on_startup = True monkeypatch.setattr("transcription.app.get_settings", lambda: _Settings()) @@ -70,6 +74,8 @@ class TestAppLifespan: monkeypatch.setattr("transcription.app.should_bootstrap_schema", lambda _settings: True) monkeypatch.setattr("transcription.app._start_worker", lambda _app: calls.append("start_worker")) monkeypatch.setattr("transcription.app._stop_worker", lambda _app: calls.append("stop_worker")) + monkeypatch.setattr("transcription.app.apply_pending_migrations", lambda **_kwargs: calls.append("migrate")) + monkeypatch.setattr("transcription.app.validate_schema_compatibility", lambda **_kwargs: []) class _Dir: def mkdir(self, parents: bool, exist_ok: bool): @@ -78,6 +84,8 @@ class TestAppLifespan: class _Settings: upload_dir = _Dir() prompt_dir = _Dir() + migration_auto_apply_on_startup = False + validate_schema_on_startup = True monkeypatch.setattr("transcription.app.get_settings", lambda: _Settings()) diff --git a/tests/test_config.py b/tests/test_config.py index d8ba603..44e2fcc 100644 --- a/tests/test_config.py +++ b/tests/test_config.py @@ -63,6 +63,16 @@ class TestPathSettings: assert isinstance(settings.prompt_dir, Path) +class TestMigrationSafetySettings: + """Verify migration safety settings defaults.""" + + def test_migration_safety_defaults(self): + """Migration auto-apply is off and startup schema validation is on by default.""" + settings = _make_settings() + assert settings.migration_auto_apply_on_startup is False + assert settings.validate_schema_on_startup is True + + class TestWorkerReliabilitySettings: """Verify worker retry settings defaults.""" diff --git a/tests/test_db.py b/tests/test_db.py index d6d615f..dbea161 100644 --- a/tests/test_db.py +++ b/tests/test_db.py @@ -18,10 +18,10 @@ class TestSchemaBootstrap: """Verify create_all produces the expected table set.""" def test_create_all_creates_expected_tables(self): - """After create_all(), document, job, and transcript tables exist.""" + """After create_all(), core V1 tables exist.""" engine = _in_memory_engine() # Ensure models are imported so metadata is populated - from transcription.models import Document, Job, Transcript # noqa: F401 + from transcription.models import Document, Job, Transcript, TranscriptRevision # noqa: F401 import transcription.db as db_module @@ -32,6 +32,19 @@ class TestSchemaBootstrap: assert "document" in table_names assert "job" in table_names assert "transcript" in table_names + assert "transcriptrevision" in table_names + + + def test_validate_schema_compatibility_returns_no_issues_for_fresh_schema(self): + """validate_schema_compatibility reports no issues on fresh schema.""" + engine = _in_memory_engine() + + import transcription.db as db_module + + db_module.create_all(engine=engine) + issues = db_module.validate_schema_compatibility(engine=engine) + + assert issues == [] class TestSessionFactory: diff --git a/tests/test_migrations.py b/tests/test_migrations.py new file mode 100644 index 0000000..e12d2c6 --- /dev/null +++ b/tests/test_migrations.py @@ -0,0 +1,106 @@ +"""Tests for transcription.migrations — explicit Step 4 migration safety behavior.""" + +from sqlalchemy import inspect, text +from sqlmodel import create_engine +from sqlmodel.pool import StaticPool + +from transcription.migrations import apply_pending_migrations, list_pending_migrations + + +def _in_memory_engine(): + """Create isolated in-memory SQLite engine.""" + return create_engine( + "sqlite://", + connect_args={"check_same_thread": False}, + poolclass=StaticPool, + ) + + +class TestMigrations: + """Verify migration listing and application behavior.""" + + def test_list_pending_returns_all_before_apply(self): + """All known migrations are pending on a fresh legacy-shaped database.""" + engine = _in_memory_engine() + + with engine.begin() as connection: + connection.execute( + text( + """ + CREATE TABLE job ( + id TEXT PRIMARY KEY, + document_id TEXT NOT NULL, + status TEXT NOT NULL, + created_at TEXT NOT NULL, + updated_at TEXT NOT NULL + ) + """ + ) + ) + + pending = list_pending_migrations(engine=engine) + assert [migration.revision_id for migration in pending] == [ + "0001_add_retry_count_to_job", + "0002_create_transcriptrevision_table", + ] + + def test_apply_pending_migrations_records_history_and_schema(self): + """Applying pending migrations mutates schema and records revision history.""" + engine = _in_memory_engine() + + with engine.begin() as connection: + connection.execute( + text( + """ + CREATE TABLE job ( + id TEXT PRIMARY KEY, + document_id TEXT NOT NULL, + status TEXT NOT NULL, + created_at TEXT NOT NULL, + updated_at TEXT NOT NULL + ) + """ + ) + ) + + applied = apply_pending_migrations(engine=engine) + assert applied == [ + "0001_add_retry_count_to_job", + "0002_create_transcriptrevision_table", + ] + + inspector = inspect(engine) + job_columns = {column["name"] for column in inspector.get_columns("job")} + assert "retry_count" in job_columns + assert "transcriptrevision" in set(inspector.get_table_names()) + + with engine.begin() as connection: + rows = connection.execute( + text("SELECT revision_id FROM schema_migration_history ORDER BY revision_id") + ).fetchall() + assert [row[0] for row in rows] == applied + + def test_apply_pending_migrations_is_idempotent(self): + """Re-running apply_pending_migrations with no pending revisions is a no-op.""" + engine = _in_memory_engine() + + with engine.begin() as connection: + connection.execute( + text( + """ + CREATE TABLE job ( + id TEXT PRIMARY KEY, + document_id TEXT NOT NULL, + status TEXT NOT NULL, + created_at TEXT NOT NULL, + updated_at TEXT NOT NULL + ) + """ + ) + ) + + first_apply = apply_pending_migrations(engine=engine) + second_apply = apply_pending_migrations(engine=engine) + + assert len(first_apply) == 2 + assert second_apply == [] \ No newline at end of file diff --git a/uploads/f4322c0e-a1df-434e-88b2-acb88a09a021_Hig's postcards to Zenna - January 7, 1925.jpg b/uploads/f4322c0e-a1df-434e-88b2-acb88a09a021_Hig's postcards to Zenna - January 7, 1925.jpg new file mode 100644 index 0000000..a7e18dd Binary files /dev/null and b/uploads/f4322c0e-a1df-434e-88b2-acb88a09a021_Hig's postcards to Zenna - January 7, 1925.jpg differ