transcription/docs/data_migration.md

Database Rebuild Migration Workflow

This project uses an explicit export/import rebuild workflow for schema migration.

Policy:

• Do not add runtime legacy-compatibility write paths.
• Rebuild a fresh target database from current models.
• Export current data/media, then import into the fresh target.

Commands

1) Export current DB + uploads into a bundle

uv run python tools/export_import_migration.py export --bundle-dir .migration-bundle

Optional source overrides:

• --source-db <path-or-sqlalchemy-url>
• --source-upload-dir <path>

2) Import bundle into a fresh DB + uploads root

uv run python tools/export_import_migration.py import --bundle-dir .migration-bundle --target-db .\data\transcription-new.db --target-upload-dir .\data-new

3) One-shot export+import

uv run python tools/export_import_migration.py migrate --bundle-dir .migration-bundle --target-db .\data\transcription-new.db --target-upload-dir .\data-new

What gets migrated

• Tables (in dependency order): document_type, person_role, tag, document, person, photo, document_person, document_tag, job, source, job_source, execution_attempt.
• Media tree under UPLOAD_DIR.

The bundle contains:

• database.json (row export)
• uploads/ (copied media files)

Path normalization during export/import:

• source.file_path is normalized to documents/... (upload-root-relative POSIX).
• photo.path is normalized to photos/... (upload-root-relative POSIX).

Legacy V4.x portrait/homepage backfill in the export step:

• If the source DB has no photo table, the exporter synthesizes photo rows from legacy person.portrait_path values and from legacy homepage image files under UPLOAD_DIR/homepage.
• Legacy portrait and homepage image files are copied into the unified UPLOAD_DIR/photos/{photo_id}{suffix} layout in the migration bundle.
• Legacy homepage markdown is relocated from UPLOAD_DIR/homepage/homepage.md to UPLOAD_DIR/homepage.md.

Cutover

After importing to a fresh target:

1. Stop the app.
2. Point DATABASE__* and UPLOAD_DIR to the new targets.
3. Start the app and run smoke checks (/healthz, create/upload/process one job).