transcription/docs/data_migration.md

Database Rebuild Migration Workflow

This project uses an explicit export/import rebuild workflow for schema migration.

Policy:

• Do not add runtime legacy-compatibility write paths.
• Rebuild a fresh target database from current models.
• Export current data/media, then import into the fresh target.

Commands

1) Export current DB + uploads into a bundle

uv run python tools/export_import_migration.py export --bundle-dir .migration-bundle

Optional source overrides:

• --source-db <path-or-sqlalchemy-url>
• --source-upload-dir <path>

2) Import bundle into a fresh target (SQLite or PostgreSQL)

uv run python tools/export_import_migration.py import --bundle-dir .migration-bundle --target-db .\data\transcription-new.db --target-upload-dir .\data-new

PostgreSQL target example:

uv run python tools/export_import_migration.py import --bundle-dir .migration-bundle --target-db postgresql://transcription:change-me@localhost:5432/transcription --target-upload-dir .\data-new

3) Verify migration parity and integrity

uv run python tools/export_import_migration.py verify --source-db .\data\transcription.db --target-db postgresql://transcription:change-me@localhost:5432/transcription

The verify command checks:

• row-count parity across migration tables
• orphan-reference checks for source, job, job_source, and execution_attempt
• duplicate (job_id, source_id, attempt_number) in execution_attempt

Exit code:

• 0 when counts and integrity checks pass
• 1 when mismatches or integrity violations are detected

4) One-shot export+import

uv run python tools/export_import_migration.py migrate --bundle-dir .migration-bundle --target-db .\data\transcription-new.db --target-upload-dir .\data-new

What gets migrated

• Tables (in dependency order): document_type, person_role, tag, document, person, photo, document_person, document_tag, person_tag, job, source, job_source, execution_attempt.
• Media tree under UPLOAD_DIR.

The bundle contains:

• database.json (row export)
• uploads/ (copied media files)

Path normalization during export/import:

• source.file_path is normalized to documents/... (upload-root-relative POSIX).
• photo.path is normalized to photos/... (upload-root-relative POSIX).

Legacy V4.x portrait/homepage backfill in the export step:

• If the source DB has no photo table, the exporter synthesizes photo rows from legacy person.portrait_path values and from legacy homepage image files under UPLOAD_DIR/homepage.
• Legacy portrait and homepage image files are copied into the unified UPLOAD_DIR/photos/{photo_id}{suffix} layout in the migration bundle.
• Legacy homepage markdown is relocated from UPLOAD_DIR/homepage/homepage.md to UPLOAD_DIR/homepage.md.
• Legacy person.full_name values are split into given_names + last_name for V5.1 schema compatibility.

Cutover (SQLite -> PostgreSQL)

After importing to a fresh target:

1. Stop app and worker services to freeze writes.
2. Export a migration bundle from the last SQLite state.
3. Import bundle to PostgreSQL target.
4. Run verify against source and target before switching runtime.
5. Switch runtime config to PostgreSQL (DATABASE__DRIVER=postgres and related DATABASE__* values).
6. Start app and worker services.
7. Run smoke checks (/healthz, create/upload/process one job).

Rollback

If verify or smoke checks fail:

1. Stop app and worker services.
2. Revert runtime config to SQLite.
3. Start app and worker against pre-cutover SQLite database.
4. Preserve failed migration bundle and logs for analysis.