V4 plan created: Adding many-to-many links between Documents & People in the UI. Also adding some new tables for Document Type, Person role.

2026-08-09 13:29:50 -05:00
parent e6549277c6
commit 5753eb0135
13 changed files with 1356 additions and 0 deletions
@@ -0,0 +1,81 @@
+# Implementation Plan (Version 3)
+
+## Goal
+
+Replace the current v2 SQLModel schema with the approved v3 schema and make sure every database operation works through the existing async SQLAlchemy/SQLModel session layer.
+
+Use a fresh database. There will be no migrations, data conversion, legacy compatibility shims, or parallel v2/v3 code paths.
+
+## Current Project Impact
+
+* `src/transcription/db/models.py` defines the SQLModel tables. It must be updated to match the approved v3 schema (`Document`, `Person`, `DocumentPerson`, `Source`, `Job`, `JobSource`).
+* The v3 target adds frozen submission-time prompt snapshot fields (`prompt_name`, `prompt_hash`, `system_prompt`, `user_prompt`, `temperature`, `top_p`) to `Job` and full output payloads (`raw_api_response`, `ai_metadata`) to `JobSource`.
+* The v3 target adds image asset verification fields (`file_hash`, `file_size_bytes`) to `Source`.
+* Database operations must utilize `JSONBCompat` and the existing SQLModel/SQLAlchemy abstractions to preserve the same logical schema and JSON behavior across the supported backends, while keeping PostgreSQL as the intended production database.
+* Async CRUD lives in `DocumentService`, `JobService`, `TranscriptionService`, and upload helpers. Their queries and relationship loading must be updated for v3 fields.
+* Canonical operator tooling must remain OS-independent; safety workflows such as destructive-test backup and restore should run through Python or other cross-platform entry points rather than platform-specific shells.
+
+## Implementation
+
+### 1. Update the Schema and Domain Models
+
+* Replace the models in `src/transcription/db/models.py` with the approved v3 tables, enums, relationships, foreign keys, constraints, and indexes.
+* Ensure all JSON fields use `JSONBCompat` for dialect portability across SQLite and PostgreSQL.
+* Keep `SQLModel.metadata.create_all()` as the schema bootstrap for fresh databases.
+* Delete `_ensure_sqlite_compat_columns()` and all legacy schema patching from `src/transcription/db/operations.py`.
+* Keep the Python models and `docs/schema_v3.md` perfectly synchronized.
+
+### 2. Update Data Services and Async Worker Layer
+
+* Update job creation and worker orchestration so prompt configuration is resolved at submission and frozen onto `Job` (`prompt_name`, `prompt_hash`, `system_prompt`, `user_prompt`, `temperature`, `top_p`) before execution starts.
+* Update `TranscriptionService` and provider adapters to store the complete unedited API REST response dictionary into `job_source.raw_api_response` alongside operational metrics in `job_source.ai_metadata`.
+* Update upload handlers to calculate and store file metadata (`file_hash` via SHA-256, `file_size_bytes`) on `Source` records during file ingestion.
+* Remove legacy single-source compatibility flows so worker paths persist per-page outcomes only through `JobSource` updates.
+
+### 3. Update Integration Tests and Mock AI Providers
+
+* Update mock provider fixtures in test suites to return realistic complete API response envelopes.
+* Verify test coverage for `JSONBCompat` field writes and reads under SQLite in-memory test databases.
+* Add assertions in async workflow tests to verify frozen prompt snapshot fields on `Job`, plus per-page failure isolation and output evidence on `JobSource`.
+
+### 4. Update the UI for the v3 Schema
+
+* Review the UI components and views displaying document, job, person, and source data so they reference v3 schema properties instead of v2 relationships.
+* Ensure the UI correctly renders `COALESCE(revised_text, raw_transcription)` for page viewing and inline editing.
+* Ensure resubmit actions only queue failed pages and preserve frozen prompt snapshot behavior on the existing `Job`.
+* Consider the guidance in `docs/ui_style_guide.md` when making UI changes so updated views remain consistent with the project’s visual conventions.
+
+### 5. Keep Operational Tooling Portable
+
+* Implement destructive-test backup and restore workflows in Python so the canonical path runs on Windows, Linux, and macOS.
+* Avoid making core developer or recovery procedures depend on PowerShell-only or shell-specific semantics.
+* Keep operational documentation aligned with the cross-platform command path used by the repository.
+
+## Done When
+
+* A fresh database is created directly from the v3 SQLModel metadata.
+* Frozen prompt input provenance is captured on `Job` for each submission, and full per-page output evidence is captured on `JobSource` for every AI execution task.
+* The focused tests and full test suite pass on both SQLite and PostgreSQL backends.
+* Canonical operator workflows required for development and destructive-test recovery run without a Windows-only shell dependency.
+
+## Out of Scope
+
+* Database migrations or preservation of v2 data
+* Legacy compatibility code
+* UI redesign, batch orchestration, worker concurrency, deployment, and operational runbooks
+ 
+---
+
+## Related Local References
+
+- [System Overview](index_v3.md)
+- [System Design Intent](invariant/intent.md)
+- [Transcription Methodology](invariant/transcription_methodology.md)
+- [System Architecture](architecture_v3.md)
+- [System Requirements](requirements_v3.md)
+- [Data model](schema_v3.md)
+- [Error Handling Policy](error_handling_v3.md)
+- Implementation Plan (this document)
+
+
+