bbchops/transcription
1
0
Fork 0
generated from john/python-template
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

V3 Updated V3 core documents. Added data folder backup/restore before/after running destructive tests.

Browse Source
This commit is contained in:
Jim Lancaster
2026-08-09 10:51:30 -05:00
parent b59d3da23e
commit e6549277c6
14 changed files with 417 additions and 271 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/architecture_v3.md
+18 -16
View File
@@ -6,9 +6,10 @@ This document describes the V3 production architecture of the personal historica
* Preserve source material as immutable transcribed text alongside page-level spatial AI metadata and complete provider API envelopes.
* Support batching multi-image and folder uploads cleanly into sequential pages (`page_number`).
* Capture complete input prompt provenance (`system_prompt`, `user_prompt`, `prompt_hash`) and execution parameters (`temperature`, `top_p`) at the page execution level (`JobSource`).
* Capture complete input prompt provenance (`system_prompt`, `user_prompt`, `prompt_hash`) and execution parameters (`temperature`, `top_p`) at submission time on `Job`.
* Leverage asynchronous worker pools (`asyncio`) for parallel single-image API execution bounded by rate limiters (`asyncio.Semaphore`).
* Maintain relational database portability across engines (SQLite for development/testing, PostgreSQL for production) using SQLModel and generic JSON abstraction layers.
* Maintain relational data-model portability across the supported backends by using SQLModel/SQLAlchemy and compatibility types so the same domain schema works in SQLite for local development/testing and PostgreSQL in production.
* Keep operator tooling and local maintenance workflows OS-independent by using Python or other cross-platform interfaces for canonical project automation.
* Verify image asset integrity via SHA-256 file hashing (`file_hash`) while storing binary assets on the local filesystem.
* Standardize all data validation, API parsing, and database models on **Pydantic V2** and **SQLModel**.
* Support rich historical attribution (multi-author and multi-recipient relationships via `DocumentPerson`).
@@ -19,8 +20,9 @@ The V3 runtime operates as an asynchronous Python application:
* FastAPI + NiceGUI web application process.
* In-process `asyncio` background task orchestrator for parallel API execution.
* Relational persistence via SQLModel / SQLAlchemy (SQLite engine in local development/testing, PostgreSQL engine in production).
* Relational persistence via SQLModel / SQLAlchemy, using SQLite for local development/testing and PostgreSQL as the production persistence target.
* Pydantic V2 validation layer wrapping API payloads, prompt configurations, and JSON metadata schemas.
* Cross-platform operator workflows implemented in Python so core local operations run consistently on Windows, Linux, and macOS.
^^^mermaid
flowchart LR
@@ -55,9 +57,9 @@ Application lifespan owns runtime setup/teardown:
Responsibilities:
* Batch orchestration and status transitions (`queued` -> `processing` -> `completed` | `partial_success` | `failed`).
* Batch orchestration and status transitions (`queued` -> `processing` -> `transcribed` | `partial_success` | `failed`).
* Parallel single-image API execution using `asyncio.gather` bounded by `asyncio.Semaphore`.
* Page-level prompt construction, logging full `system_prompt` and `user_prompt` to `JobSource`.
* Resolve prompt configuration at submission time and persist frozen snapshot fields on `Job`.
* Pydantic schema parsing and validation prior to database storage.
### Domain & Service Layer
@@ -75,25 +77,25 @@ Responsibilities:
1. User uploads a folder or batch of images for a `Document`.
2. System hashes each image file (SHA-256), writes image files to filesystem storage, and creates `Document`, `Job(status='queued')`, and ordered `Source` pages (`page_number = 1..N`).
3. Worker claims job, sets `Job.status = 'processing'`, and spawns parallel `asyncio` tasks bounded by semaphore.
4. Each task builds page-specific system/user prompts and calls Vision API for a **single** `Source` image.
4. Each task reads the frozen prompt snapshot from `Job` and calls Vision API for a **single** `Source` image.
5. On task completion:
* Writes a `JobSource` record containing `status='transcribed'`, `raw_transcription`, complete input details (`prompt_name`, `prompt_hash`, `system_prompt`, `user_prompt`, `temperature`, `top_p`), operational `ai_metadata`, and complete unedited `raw_api_response`.
* Writes a `JobSource` record containing `status='transcribed'`, `raw_transcription`, operational `ai_metadata`, and complete unedited `raw_api_response`.
* Caches active output text to `Source.raw_transcription`.
6. On page failure:
* Writes `JobSource` record with `status='failed'`, recorded prompt inputs, and `error_detail`.
* Writes `JobSource` record with `status='failed'` and `error_detail`.
7. Once all page tasks resolve:
* Marks `Job.status` as `completed` (100% success), `partial_success` (at least 1 success, 1 failure), or `failed` (all failed).
* Marks `Job.status` as `transcribed` (100% success), `partial_success` (at least 1 success, 1 failure), or `failed` (all failed).
## Domain Ownership & Invariants
* **Immutable AI Outputs:** `source.raw_transcription` and `job_source.raw_transcription` store original, point-in-time machine output and are immutable.
* **Complete Input & Output Provenance:** Every `job_source` record contains both the exact input configuration sent to the model and the complete REST response envelope returned.
* **Complete Input & Output Provenance:** Every `job` stores the exact frozen input configuration sent to the model, and every `job_source` stores per-page output evidence including the complete REST response envelope returned.
* **Inlined Revisions:** Human corrections occur on `source.revised_text`. UI renders `COALESCE(revised_text, raw_transcription)`.
* **Sequential Integrity:** Multi-page documents are strictly ordered by `source.page_number ASC`.
* **Page Execution Isolation:** A failure on one page image does not invalidate successful transcriptions on sister pages in the same batch job.
@@ -103,7 +105,7 @@ Responsibilities:
* `Document` has many `Source` pages, many `Job` runs, and many `Person` records via `DocumentPerson` junction (`author` or `recipient`).
* `Source` belongs to one `Document` and can be processed across many `JobSource` executions.
* `Job` has many `JobSource` execution records.
* `JobSource` holds page-level prompts, parameters, execution status, and raw response JSON.
* `JobSource` holds page-level execution status, output text, and raw response JSON.
## Test Strategy
@@ -125,14 +127,14 @@ Responsibilities:
## Related Local References
- [System Overview](index_v2.md)
- [System Overview](index_v3.md)
- [System Design Intent](invariant/intent.md)
- [Transcription Methodology](invariant/transcription_methodology.md)
- System Architecture (this document)
- [System Requirements](requirements_v2.md)
- [Data model](schema_v2.md)
- [Error Handling Policy](error_handling_v2.md)
- [Implementation Plan](implementation_plan_v2.md)
- [System Requirements](requirements_v3.md)
- [Data model](schema_v3.md)
- [Error Handling Policy](error_handling_v3.md)
- [Implementation Plan](implementation_plan_v3.md)