Implementation Plan (Version 3)

Goal

Replace the current v2 SQLModel schema with the approved v3 schema and make sure every database operation works through the existing async SQLAlchemy/SQLModel session layer.

Use a fresh database. There will be no migrations, data conversion, legacy compatibility shims, or parallel v2/v3 code paths.

Current Project Impact

src/transcription/db/models.py defines the SQLModel tables. It must be updated to match the approved v3 schema (Document, Person, DocumentPerson, Source, Job, JobSource).
The v3 target adds frozen submission-time prompt snapshot fields (prompt_name, prompt_hash, system_prompt, user_prompt, temperature, top_p) to Job and full output payloads (raw_api_response, ai_metadata) to JobSource.
The v3 target adds image asset verification fields (file_hash, file_size_bytes) to Source.
Database operations must utilize JSONBCompat and the existing SQLModel/SQLAlchemy abstractions to preserve the same logical schema and JSON behavior across the supported backends, while keeping PostgreSQL as the intended production database.
Async CRUD lives in DocumentService, JobService, TranscriptionService, and upload helpers. Their queries and relationship loading must be updated for v3 fields.
Canonical operator tooling must remain OS-independent; safety workflows such as destructive-test backup and restore should run through Python or other cross-platform entry points rather than platform-specific shells.

Implementation

1. Update the Schema and Domain Models

Replace the models in src/transcription/db/models.py with the approved v3 tables, enums, relationships, foreign keys, constraints, and indexes.
Ensure all JSON fields use JSONBCompat for dialect portability across SQLite and PostgreSQL.
Keep SQLModel.metadata.create_all() as the schema bootstrap for fresh databases.
Delete _ensure_sqlite_compat_columns() and all legacy schema patching from src/transcription/db/operations.py.
Keep the Python models and docs/schema_v3.md perfectly synchronized.

2. Update Data Services and Async Worker Layer

Update job creation and worker orchestration so prompt configuration is resolved at submission and frozen onto Job (prompt_name, prompt_hash, system_prompt, user_prompt, temperature, top_p) before execution starts.
Update TranscriptionService and provider adapters to store the complete unedited API REST response dictionary into job_source.raw_api_response alongside operational metrics in job_source.ai_metadata.
Update upload handlers to calculate and store file metadata (file_hash via SHA-256, file_size_bytes) on Source records during file ingestion.
Remove legacy single-source compatibility flows so worker paths persist per-page outcomes only through JobSource updates.

3. Update Integration Tests and Mock AI Providers

Update mock provider fixtures in test suites to return realistic complete API response envelopes.
Verify test coverage for JSONBCompat field writes and reads under SQLite in-memory test databases.
Add assertions in async workflow tests to verify frozen prompt snapshot fields on Job, plus per-page failure isolation and output evidence on JobSource.

4. Update the UI for the v3 Schema

Review the UI components and views displaying document, job, person, and source data so they reference v3 schema properties instead of v2 relationships.
Ensure the UI correctly renders COALESCE(revised_text, raw_transcription) for page viewing and inline editing.
Ensure resubmit actions only queue failed pages and preserve frozen prompt snapshot behavior on the existing Job.
Consider the guidance in docs/ui_style_guide.md when making UI changes so updated views remain consistent with the project’s visual conventions.

5. Keep Operational Tooling Portable

Implement destructive-test backup and restore workflows in Python so the canonical path runs on Windows, Linux, and macOS.
Avoid making core developer or recovery procedures depend on PowerShell-only or shell-specific semantics.
Keep operational documentation aligned with the cross-platform command path used by the repository.

Done When

A fresh database is created directly from the v3 SQLModel metadata.
Frozen prompt input provenance is captured on Job for each submission, and full per-page output evidence is captured on JobSource for every AI execution task.
The focused tests and full test suite pass on both SQLite and PostgreSQL backends.
Canonical operator workflows required for development and destructive-test recovery run without a Windows-only shell dependency.

Out of Scope

Database migrations or preservation of v2 data
Legacy compatibility code
UI redesign, batch orchestration, worker concurrency, deployment, and operational runbooks

5.2 KiB Raw Blame History Unescape Escape