generated from john/python-template
95 lines
3.6 KiB
Markdown
95 lines
3.6 KiB
Markdown
# Database Rebuild Migration Workflow
|
|
|
|
This project uses an explicit **export/import rebuild workflow** for schema migration.
|
|
|
|
Policy:
|
|
- Do not add runtime legacy-compatibility write paths.
|
|
- Rebuild a fresh target database from current models.
|
|
- Export current data/media, then import into the fresh target.
|
|
|
|
## Commands
|
|
|
|
### 1) Export current DB + uploads into a bundle
|
|
|
|
```bash
|
|
uv run python tools/export_import_migration.py export --bundle-dir .migration-bundle
|
|
```
|
|
|
|
Optional source overrides:
|
|
- `--source-db <path-or-sqlalchemy-url>`
|
|
- `--source-upload-dir <path>`
|
|
|
|
### 2) Import bundle into a fresh target (SQLite or PostgreSQL)
|
|
|
|
```bash
|
|
uv run python tools/export_import_migration.py import --bundle-dir .migration-bundle --target-db .\data\transcription-new.db --target-upload-dir .\data-new
|
|
```
|
|
|
|
PostgreSQL target example:
|
|
|
|
```bash
|
|
uv run python tools/export_import_migration.py import --bundle-dir .migration-bundle --target-db postgresql://transcription:change-me@localhost:5432/transcription --target-upload-dir .\data-new
|
|
```
|
|
|
|
### 3) Verify migration parity and integrity
|
|
|
|
```bash
|
|
uv run python tools/export_import_migration.py verify --source-db .\data\transcription.db --target-db postgresql://transcription:change-me@localhost:5432/transcription
|
|
```
|
|
|
|
The verify command checks:
|
|
|
|
- row-count parity across migration tables
|
|
- orphan-reference checks for `source`, `job`, `job_source`, and `execution_attempt`
|
|
- duplicate `(job_id, source_id, attempt_number)` in `execution_attempt`
|
|
|
|
Exit code:
|
|
|
|
- `0` when counts and integrity checks pass
|
|
- `1` when mismatches or integrity violations are detected
|
|
|
|
### 4) One-shot export+import
|
|
|
|
```bash
|
|
uv run python tools/export_import_migration.py migrate --bundle-dir .migration-bundle --target-db .\data\transcription-new.db --target-upload-dir .\data-new
|
|
```
|
|
|
|
## What gets migrated
|
|
|
|
- Tables (in dependency order): `document_type`, `person_role`, `tag`, `document`, `person`, `photo`, `document_person`, `document_tag`, `person_tag`, `job`, `source`, `job_source`, `execution_attempt`.
|
|
- Media tree under `UPLOAD_DIR`.
|
|
|
|
The bundle contains:
|
|
- `database.json` (row export)
|
|
- `uploads/` (copied media files)
|
|
|
|
Path normalization during export/import:
|
|
- `source.file_path` is normalized to `documents/...` (upload-root-relative POSIX).
|
|
- `photo.path` is normalized to `photos/...` (upload-root-relative POSIX).
|
|
|
|
Legacy V4.x portrait/homepage backfill in the export step:
|
|
- If the source DB has no `photo` table, the exporter synthesizes `photo` rows from legacy `person.portrait_path` values and from legacy homepage image files under `UPLOAD_DIR/homepage`.
|
|
- Legacy portrait and homepage image files are copied into the unified `UPLOAD_DIR/photos/{photo_id}{suffix}` layout in the migration bundle.
|
|
- Legacy homepage markdown is relocated from `UPLOAD_DIR/homepage/homepage.md` to `UPLOAD_DIR/homepage.md`.
|
|
- Legacy `person.full_name` values are split into `given_names` + `last_name` for V5.1 schema compatibility.
|
|
|
|
## Cutover (SQLite -> PostgreSQL)
|
|
|
|
After importing to a fresh target:
|
|
1. Stop app and worker services to freeze writes.
|
|
2. Export a migration bundle from the last SQLite state.
|
|
3. Import bundle to PostgreSQL target.
|
|
4. Run `verify` against source and target before switching runtime.
|
|
5. Switch runtime config to PostgreSQL (`DATABASE__DRIVER=postgres` and related `DATABASE__*` values).
|
|
6. Start app and worker services.
|
|
7. Run smoke checks (`/healthz`, create/upload/process one job).
|
|
|
|
## Rollback
|
|
|
|
If verify or smoke checks fail:
|
|
|
|
1. Stop app and worker services.
|
|
2. Revert runtime config to SQLite.
|
|
3. Start app and worker against pre-cutover SQLite database.
|
|
4. Preserve failed migration bundle and logs for analysis.
|