Implementation Plan (Version 3)

Goal

Replace the current v2 SQLModel schema with the approved v3 schema and make sure every database operation works through the existing async SQLAlchemy/SQLModel session layer.

Use a fresh database. There will be no migrations, data conversion, legacy compatibility shims, or parallel v2/v3 code paths.

Current Project Impact

src/transcription/db/models.py defines the SQLModel tables. It must be updated to match the approved v3 schema (Document, Person, DocumentPerson, Source, Job, JobSource).
The v3 target adds input provenance fields (prompt_name, prompt_hash, system_prompt, user_prompt, temperature, top_p) and full output payloads (raw_api_response, ai_metadata) to JobSource.
The v3 target adds image asset verification fields (file_hash, file_size_bytes) to Source.
Database operations must utilize the JSONBCompat decorator to remain database-agnostic (SQLite for local development/testing and PostgreSQL for production).
Async CRUD lives in DocumentService, JobService, TranscriptionService, and upload helpers. Their queries and relationship loading must be updated for v3 fields.

Implementation

1. Update the Schema and Domain Models

Replace the models in src/transcription/db/models.py with the approved v3 tables, enums, relationships, foreign keys, constraints, and indexes.
Ensure all JSON fields use JSONBCompat for dialect portability across SQLite and PostgreSQL.
Keep SQLModel.metadata.create_all() as the schema bootstrap for fresh databases.
Delete _ensure_sqlite_compat_columns() and all legacy schema patching from src/transcription/db/operations.py.
Keep the Python models and docs/schema_v3.md perfectly synchronized.

2. Update Data Services and Async Worker Layer

Update JobService and worker tasks (worker.py) to construct and save page-level input prompt fields (prompt_name, prompt_hash, system_prompt, user_prompt, temperature, top_p) directly onto JobSource records upon execution.
Update TranscriptionService and provider adapters to store the complete unedited API REST response dictionary into job_source.raw_api_response alongside operational metrics in job_source.ai_metadata.
Update upload handlers to calculate and store file metadata (file_hash via SHA-256, file_size_bytes) on Source records during file ingestion.

3. Update Integration Tests and Mock AI Providers

Update mock provider fixtures in test suites to return realistic complete API response envelopes.
Verify test coverage for JSONBCompat field writes and reads under SQLite in-memory test databases.
Add assertions in async workflow tests to verify page-level prompt provenance and failure isolation on JobSource.

4. Update the UI for the v3 Schema

Review the UI components and views displaying document, job, person, and source data so they reference v3 schema properties instead of v2 relationships.
Ensure the UI correctly renders COALESCE(revised_text, raw_transcription) for page viewing and inline editing.
Ensure the "Retry Failed Pages" UI action spawns targeted jobs correctly using page-level JobSource failure states.
Consider the guidance in docs/ui_style_guide.md when making UI changes so updated views remain consistent with the project’s visual conventions.

Done When

A fresh database is created directly from the v3 SQLModel metadata.
Full input/output provenance is captured on JobSource for every AI execution task.
The focused tests and full test suite pass on both SQLite and PostgreSQL backends.

Out of Scope

Database migrations or preservation of v2 data
Legacy compatibility code
UI redesign, batch orchestration, worker concurrency, deployment, and operational runbooks

4.1 KiB Raw Blame History Unescape Escape