bbchops/transcription
1
0
Fork 0
generated from john/python-template
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

V3 Updated V3 core documents. Added data folder backup/restore before/after running destructive tests.

Browse Source
This commit is contained in:
Jim Lancaster
2026-08-09 10:51:30 -05:00
parent b59d3da23e
commit e6549277c6
14 changed files with 417 additions and 271 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/implementation_plan_v3.md
+15 -6
View File
@@ -9,10 +9,11 @@ Use a fresh database. There will be no migrations, data conversion, legacy compa
## Current Project Impact
* `src/transcription/db/models.py` defines the SQLModel tables. It must be updated to match the approved v3 schema (`Document`, `Person`, `DocumentPerson`, `Source`, `Job`, `JobSource`).
* The v3 target adds input provenance fields (`prompt_name`, `prompt_hash`, `system_prompt`, `user_prompt`, `temperature`, `top_p`) and full output payloads (`raw_api_response`, `ai_metadata`) to `JobSource`.
* The v3 target adds frozen submission-time prompt snapshot fields (`prompt_name`, `prompt_hash`, `system_prompt`, `user_prompt`, `temperature`, `top_p`) to `Job` and full output payloads (`raw_api_response`, `ai_metadata`) to `JobSource`.
* The v3 target adds image asset verification fields (`file_hash`, `file_size_bytes`) to `Source`.
* Database operations must utilize the `JSONBCompat` decorator to remain database-agnostic (SQLite for local development/testing and PostgreSQL for production).
* Database operations must utilize `JSONBCompat` and the existing SQLModel/SQLAlchemy abstractions to preserve the same logical schema and JSON behavior across the supported backends, while keeping PostgreSQL as the intended production database.
* Async CRUD lives in `DocumentService`, `JobService`, `TranscriptionService`, and upload helpers. Their queries and relationship loading must be updated for v3 fields.
* Canonical operator tooling must remain OS-independent; safety workflows such as destructive-test backup and restore should run through Python or other cross-platform entry points rather than platform-specific shells.
## Implementation
@@ -26,28 +27,36 @@ Use a fresh database. There will be no migrations, data conversion, legacy compa
### 2. Update Data Services and Async Worker Layer
* Update `JobService` and worker tasks (`worker.py`) to construct and save page-level input prompt fields (`prompt_name`, `prompt_hash`, `system_prompt`, `user_prompt`, `temperature`, `top_p`) directly onto `JobSource` records upon execution.
* Update job creation and worker orchestration so prompt configuration is resolved at submission and frozen onto `Job` (`prompt_name`, `prompt_hash`, `system_prompt`, `user_prompt`, `temperature`, `top_p`) before execution starts.
* Update `TranscriptionService` and provider adapters to store the complete unedited API REST response dictionary into `job_source.raw_api_response` alongside operational metrics in `job_source.ai_metadata`.
* Update upload handlers to calculate and store file metadata (`file_hash` via SHA-256, `file_size_bytes`) on `Source` records during file ingestion.
* Remove legacy single-source compatibility flows so worker paths persist per-page outcomes only through `JobSource` updates.
### 3. Update Integration Tests and Mock AI Providers
* Update mock provider fixtures in test suites to return realistic complete API response envelopes.
* Verify test coverage for `JSONBCompat` field writes and reads under SQLite in-memory test databases.
* Add assertions in async workflow tests to verify page-level prompt provenance and failure isolation on `JobSource`.
* Add assertions in async workflow tests to verify frozen prompt snapshot fields on `Job`, plus per-page failure isolation and output evidence on `JobSource`.
### 4. Update the UI for the v3 Schema
* Review the UI components and views displaying document, job, person, and source data so they reference v3 schema properties instead of v2 relationships.
* Ensure the UI correctly renders `COALESCE(revised_text, raw_transcription)` for page viewing and inline editing.
* Ensure the "Retry Failed Pages" UI action spawns targeted jobs correctly using page-level `JobSource` failure states.
* Ensure resubmit actions only queue failed pages and preserve frozen prompt snapshot behavior on the existing `Job`.
* Consider the guidance in `docs/ui_style_guide.md` when making UI changes so updated views remain consistent with the project’s visual conventions.
### 5. Keep Operational Tooling Portable
* Implement destructive-test backup and restore workflows in Python so the canonical path runs on Windows, Linux, and macOS.
* Avoid making core developer or recovery procedures depend on PowerShell-only or shell-specific semantics.
* Keep operational documentation aligned with the cross-platform command path used by the repository.
## Done When
* A fresh database is created directly from the v3 SQLModel metadata.
* Full input/output provenance is captured on `JobSource` for every AI execution task.
* Frozen prompt input provenance is captured on `Job` for each submission, and full per-page output evidence is captured on `JobSource` for every AI execution task.
* The focused tests and full test suite pass on both SQLite and PostgreSQL backends.
* Canonical operator workflows required for development and destructive-test recovery run without a Windows-only shell dependency.
## Out of Scope