generated from john/python-template
73 lines
4.1 KiB
Markdown
73 lines
4.1 KiB
Markdown
# Implementation Plan (Version 3)
|
||
|
||
## Goal
|
||
|
||
Replace the current v2 SQLModel schema with the approved v3 schema and make sure every database operation works through the existing async SQLAlchemy/SQLModel session layer.
|
||
|
||
Use a fresh database. There will be no migrations, data conversion, legacy compatibility shims, or parallel v2/v3 code paths.
|
||
|
||
## Current Project Impact
|
||
|
||
* `src/transcription/db/models.py` defines the SQLModel tables. It must be updated to match the approved v3 schema (`Document`, `Person`, `DocumentPerson`, `Source`, `Job`, `JobSource`).
|
||
* The v3 target adds input provenance fields (`prompt_name`, `prompt_hash`, `system_prompt`, `user_prompt`, `temperature`, `top_p`) and full output payloads (`raw_api_response`, `ai_metadata`) to `JobSource`.
|
||
* The v3 target adds image asset verification fields (`file_hash`, `file_size_bytes`) to `Source`.
|
||
* Database operations must utilize the `JSONBCompat` decorator to remain database-agnostic (SQLite for local development/testing and PostgreSQL for production).
|
||
* Async CRUD lives in `DocumentService`, `JobService`, `TranscriptionService`, and upload helpers. Their queries and relationship loading must be updated for v3 fields.
|
||
|
||
## Implementation
|
||
|
||
### 1. Update the Schema and Domain Models
|
||
|
||
* Replace the models in `src/transcription/db/models.py` with the approved v3 tables, enums, relationships, foreign keys, constraints, and indexes.
|
||
* Ensure all JSON fields use `JSONBCompat` for dialect portability across SQLite and PostgreSQL.
|
||
* Keep `SQLModel.metadata.create_all()` as the schema bootstrap for fresh databases.
|
||
* Delete `_ensure_sqlite_compat_columns()` and all legacy schema patching from `src/transcription/db/operations.py`.
|
||
* Keep the Python models and `docs/schema_v3.md` perfectly synchronized.
|
||
|
||
### 2. Update Data Services and Async Worker Layer
|
||
|
||
* Update `JobService` and worker tasks (`worker.py`) to construct and save page-level input prompt fields (`prompt_name`, `prompt_hash`, `system_prompt`, `user_prompt`, `temperature`, `top_p`) directly onto `JobSource` records upon execution.
|
||
* Update `TranscriptionService` and provider adapters to store the complete unedited API REST response dictionary into `job_source.raw_api_response` alongside operational metrics in `job_source.ai_metadata`.
|
||
* Update upload handlers to calculate and store file metadata (`file_hash` via SHA-256, `file_size_bytes`) on `Source` records during file ingestion.
|
||
|
||
### 3. Update Integration Tests and Mock AI Providers
|
||
|
||
* Update mock provider fixtures in test suites to return realistic complete API response envelopes.
|
||
* Verify test coverage for `JSONBCompat` field writes and reads under SQLite in-memory test databases.
|
||
* Add assertions in async workflow tests to verify page-level prompt provenance and failure isolation on `JobSource`.
|
||
|
||
### 4. Update the UI for the v3 Schema
|
||
|
||
* Review the UI components and views displaying document, job, person, and source data so they reference v3 schema properties instead of v2 relationships.
|
||
* Ensure the UI correctly renders `COALESCE(revised_text, raw_transcription)` for page viewing and inline editing.
|
||
* Ensure the "Retry Failed Pages" UI action spawns targeted jobs correctly using page-level `JobSource` failure states.
|
||
* Consider the guidance in `docs/ui_style_guide.md` when making UI changes so updated views remain consistent with the project’s visual conventions.
|
||
|
||
## Done When
|
||
|
||
* A fresh database is created directly from the v3 SQLModel metadata.
|
||
* Full input/output provenance is captured on `JobSource` for every AI execution task.
|
||
* The focused tests and full test suite pass on both SQLite and PostgreSQL backends.
|
||
|
||
## Out of Scope
|
||
|
||
* Database migrations or preservation of v2 data
|
||
* Legacy compatibility code
|
||
* UI redesign, batch orchestration, worker concurrency, deployment, and operational runbooks
|
||
|
||
---
|
||
|
||
## Related Local References
|
||
|
||
- [System Overview](index_v3.md)
|
||
- [System Design Intent](invariant/intent.md)
|
||
- [Transcription Methodology](invariant/transcription_methodology.md)
|
||
- [System Architecture](architecture_v3.md)
|
||
- [System Requirements](requirements_v3.md)
|
||
- [Data model](schema_v3.md)
|
||
- [Error Handling Policy](error_handling_v3.md)
|
||
- Implementation Plan (this document)
|
||
|
||
|
||
|