generated from john/python-template
V3 Updated V3 core documents. Added data folder backup/restore before/after running destructive tests.
This commit is contained in:
Jim Lancaster
+18
-16
@@ -6,9 +6,10 @@ This document describes the V3 production architecture of the personal historica
|
||||
|
||||
* Preserve source material as immutable transcribed text alongside page-level spatial AI metadata and complete provider API envelopes.
|
||||
* Support batching multi-image and folder uploads cleanly into sequential pages (`page_number`).
|
||||
* Capture complete input prompt provenance (`system_prompt`, `user_prompt`, `prompt_hash`) and execution parameters (`temperature`, `top_p`) at the page execution level (`JobSource`).
|
||||
* Capture complete input prompt provenance (`system_prompt`, `user_prompt`, `prompt_hash`) and execution parameters (`temperature`, `top_p`) at submission time on `Job`.
|
||||
* Leverage asynchronous worker pools (`asyncio`) for parallel single-image API execution bounded by rate limiters (`asyncio.Semaphore`).
|
||||
* Maintain relational database portability across engines (SQLite for development/testing, PostgreSQL for production) using SQLModel and generic JSON abstraction layers.
|
||||
* Maintain relational data-model portability across the supported backends by using SQLModel/SQLAlchemy and compatibility types so the same domain schema works in SQLite for local development/testing and PostgreSQL in production.
|
||||
* Keep operator tooling and local maintenance workflows OS-independent by using Python or other cross-platform interfaces for canonical project automation.
|
||||
* Verify image asset integrity via SHA-256 file hashing (`file_hash`) while storing binary assets on the local filesystem.
|
||||
* Standardize all data validation, API parsing, and database models on **Pydantic V2** and **SQLModel**.
|
||||
* Support rich historical attribution (multi-author and multi-recipient relationships via `DocumentPerson`).
|
||||
@@ -19,8 +20,9 @@ The V3 runtime operates as an asynchronous Python application:
|
||||
|
||||
* FastAPI + NiceGUI web application process.
|
||||
* In-process `asyncio` background task orchestrator for parallel API execution.
|
||||
* Relational persistence via SQLModel / SQLAlchemy (SQLite engine in local development/testing, PostgreSQL engine in production).
|
||||
* Relational persistence via SQLModel / SQLAlchemy, using SQLite for local development/testing and PostgreSQL as the production persistence target.
|
||||
* Pydantic V2 validation layer wrapping API payloads, prompt configurations, and JSON metadata schemas.
|
||||
* Cross-platform operator workflows implemented in Python so core local operations run consistently on Windows, Linux, and macOS.
|
||||
|
||||
^^^mermaid
|
||||
flowchart LR
|
||||
@@ -55,9 +57,9 @@ Application lifespan owns runtime setup/teardown:
|
||||
|
||||
Responsibilities:
|
||||
|
||||
* Batch orchestration and status transitions (`queued` -> `processing` -> `completed` | `partial_success` | `failed`).
|
||||
* Batch orchestration and status transitions (`queued` -> `processing` -> `transcribed` | `partial_success` | `failed`).
|
||||
* Parallel single-image API execution using `asyncio.gather` bounded by `asyncio.Semaphore`.
|
||||
* Page-level prompt construction, logging full `system_prompt` and `user_prompt` to `JobSource`.
|
||||
* Resolve prompt configuration at submission time and persist frozen snapshot fields on `Job`.
|
||||
* Pydantic schema parsing and validation prior to database storage.
|
||||
|
||||
### Domain & Service Layer
|
||||
@@ -75,25 +77,25 @@ Responsibilities:
|
||||
1. User uploads a folder or batch of images for a `Document`.
|
||||
2. System hashes each image file (SHA-256), writes image files to filesystem storage, and creates `Document`, `Job(status='queued')`, and ordered `Source` pages (`page_number = 1..N`).
|
||||
3. Worker claims job, sets `Job.status = 'processing'`, and spawns parallel `asyncio` tasks bounded by semaphore.
|
||||
4. Each task builds page-specific system/user prompts and calls Vision API for a **single** `Source` image.
|
||||
4. Each task reads the frozen prompt snapshot from `Job` and calls Vision API for a **single** `Source` image.
|
||||
5. On task completion:
|
||||
* Writes a `JobSource` record containing `status='transcribed'`, `raw_transcription`, complete input details (`prompt_name`, `prompt_hash`, `system_prompt`, `user_prompt`, `temperature`, `top_p`), operational `ai_metadata`, and complete unedited `raw_api_response`.
|
||||
* Writes a `JobSource` record containing `status='transcribed'`, `raw_transcription`, operational `ai_metadata`, and complete unedited `raw_api_response`.
|
||||
* Caches active output text to `Source.raw_transcription`.
|
||||
|
||||
|
||||
6. On page failure:
|
||||
* Writes `JobSource` record with `status='failed'`, recorded prompt inputs, and `error_detail`.
|
||||
* Writes `JobSource` record with `status='failed'` and `error_detail`.
|
||||
|
||||
|
||||
7. Once all page tasks resolve:
|
||||
* Marks `Job.status` as `completed` (100% success), `partial_success` (at least 1 success, 1 failure), or `failed` (all failed).
|
||||
* Marks `Job.status` as `transcribed` (100% success), `partial_success` (at least 1 success, 1 failure), or `failed` (all failed).
|
||||
|
||||
|
||||
|
||||
## Domain Ownership & Invariants
|
||||
|
||||
* **Immutable AI Outputs:** `source.raw_transcription` and `job_source.raw_transcription` store original, point-in-time machine output and are immutable.
|
||||
* **Complete Input & Output Provenance:** Every `job_source` record contains both the exact input configuration sent to the model and the complete REST response envelope returned.
|
||||
* **Complete Input & Output Provenance:** Every `job` stores the exact frozen input configuration sent to the model, and every `job_source` stores per-page output evidence including the complete REST response envelope returned.
|
||||
* **Inlined Revisions:** Human corrections occur on `source.revised_text`. UI renders `COALESCE(revised_text, raw_transcription)`.
|
||||
* **Sequential Integrity:** Multi-page documents are strictly ordered by `source.page_number ASC`.
|
||||
* **Page Execution Isolation:** A failure on one page image does not invalidate successful transcriptions on sister pages in the same batch job.
|
||||
@@ -103,7 +105,7 @@ Responsibilities:
|
||||
* `Document` has many `Source` pages, many `Job` runs, and many `Person` records via `DocumentPerson` junction (`author` or `recipient`).
|
||||
* `Source` belongs to one `Document` and can be processed across many `JobSource` executions.
|
||||
* `Job` has many `JobSource` execution records.
|
||||
* `JobSource` holds page-level prompts, parameters, execution status, and raw response JSON.
|
||||
* `JobSource` holds page-level execution status, output text, and raw response JSON.
|
||||
|
||||
## Test Strategy
|
||||
|
||||
@@ -125,14 +127,14 @@ Responsibilities:
|
||||
|
||||
## Related Local References
|
||||
|
||||
- [System Overview](index_v2.md)
|
||||
- [System Overview](index_v3.md)
|
||||
- [System Design Intent](invariant/intent.md)
|
||||
- [Transcription Methodology](invariant/transcription_methodology.md)
|
||||
- System Architecture (this document)
|
||||
- [System Requirements](requirements_v2.md)
|
||||
- [Data model](schema_v2.md)
|
||||
- [Error Handling Policy](error_handling_v2.md)
|
||||
- [Implementation Plan](implementation_plan_v2.md)
|
||||
- [System Requirements](requirements_v3.md)
|
||||
- [Data model](schema_v3.md)
|
||||
- [Error Handling Policy](error_handling_v3.md)
|
||||
- [Implementation Plan](implementation_plan_v3.md)
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -9,10 +9,11 @@ Use a fresh database. There will be no migrations, data conversion, legacy compa
|
||||
## Current Project Impact
|
||||
|
||||
* `src/transcription/db/models.py` defines the SQLModel tables. It must be updated to match the approved v3 schema (`Document`, `Person`, `DocumentPerson`, `Source`, `Job`, `JobSource`).
|
||||
* The v3 target adds input provenance fields (`prompt_name`, `prompt_hash`, `system_prompt`, `user_prompt`, `temperature`, `top_p`) and full output payloads (`raw_api_response`, `ai_metadata`) to `JobSource`.
|
||||
* The v3 target adds frozen submission-time prompt snapshot fields (`prompt_name`, `prompt_hash`, `system_prompt`, `user_prompt`, `temperature`, `top_p`) to `Job` and full output payloads (`raw_api_response`, `ai_metadata`) to `JobSource`.
|
||||
* The v3 target adds image asset verification fields (`file_hash`, `file_size_bytes`) to `Source`.
|
||||
* Database operations must utilize the `JSONBCompat` decorator to remain database-agnostic (SQLite for local development/testing and PostgreSQL for production).
|
||||
* Database operations must utilize `JSONBCompat` and the existing SQLModel/SQLAlchemy abstractions to preserve the same logical schema and JSON behavior across the supported backends, while keeping PostgreSQL as the intended production database.
|
||||
* Async CRUD lives in `DocumentService`, `JobService`, `TranscriptionService`, and upload helpers. Their queries and relationship loading must be updated for v3 fields.
|
||||
* Canonical operator tooling must remain OS-independent; safety workflows such as destructive-test backup and restore should run through Python or other cross-platform entry points rather than platform-specific shells.
|
||||
|
||||
## Implementation
|
||||
|
||||
@@ -26,28 +27,36 @@ Use a fresh database. There will be no migrations, data conversion, legacy compa
|
||||
|
||||
### 2. Update Data Services and Async Worker Layer
|
||||
|
||||
* Update `JobService` and worker tasks (`worker.py`) to construct and save page-level input prompt fields (`prompt_name`, `prompt_hash`, `system_prompt`, `user_prompt`, `temperature`, `top_p`) directly onto `JobSource` records upon execution.
|
||||
* Update job creation and worker orchestration so prompt configuration is resolved at submission and frozen onto `Job` (`prompt_name`, `prompt_hash`, `system_prompt`, `user_prompt`, `temperature`, `top_p`) before execution starts.
|
||||
* Update `TranscriptionService` and provider adapters to store the complete unedited API REST response dictionary into `job_source.raw_api_response` alongside operational metrics in `job_source.ai_metadata`.
|
||||
* Update upload handlers to calculate and store file metadata (`file_hash` via SHA-256, `file_size_bytes`) on `Source` records during file ingestion.
|
||||
* Remove legacy single-source compatibility flows so worker paths persist per-page outcomes only through `JobSource` updates.
|
||||
|
||||
### 3. Update Integration Tests and Mock AI Providers
|
||||
|
||||
* Update mock provider fixtures in test suites to return realistic complete API response envelopes.
|
||||
* Verify test coverage for `JSONBCompat` field writes and reads under SQLite in-memory test databases.
|
||||
* Add assertions in async workflow tests to verify page-level prompt provenance and failure isolation on `JobSource`.
|
||||
* Add assertions in async workflow tests to verify frozen prompt snapshot fields on `Job`, plus per-page failure isolation and output evidence on `JobSource`.
|
||||
|
||||
### 4. Update the UI for the v3 Schema
|
||||
|
||||
* Review the UI components and views displaying document, job, person, and source data so they reference v3 schema properties instead of v2 relationships.
|
||||
* Ensure the UI correctly renders `COALESCE(revised_text, raw_transcription)` for page viewing and inline editing.
|
||||
* Ensure the "Retry Failed Pages" UI action spawns targeted jobs correctly using page-level `JobSource` failure states.
|
||||
* Ensure resubmit actions only queue failed pages and preserve frozen prompt snapshot behavior on the existing `Job`.
|
||||
* Consider the guidance in `docs/ui_style_guide.md` when making UI changes so updated views remain consistent with the project’s visual conventions.
|
||||
|
||||
### 5. Keep Operational Tooling Portable
|
||||
|
||||
* Implement destructive-test backup and restore workflows in Python so the canonical path runs on Windows, Linux, and macOS.
|
||||
* Avoid making core developer or recovery procedures depend on PowerShell-only or shell-specific semantics.
|
||||
* Keep operational documentation aligned with the cross-platform command path used by the repository.
|
||||
|
||||
## Done When
|
||||
|
||||
* A fresh database is created directly from the v3 SQLModel metadata.
|
||||
* Full input/output provenance is captured on `JobSource` for every AI execution task.
|
||||
* Frozen prompt input provenance is captured on `Job` for each submission, and full per-page output evidence is captured on `JobSource` for every AI execution task.
|
||||
* The focused tests and full test suite pass on both SQLite and PostgreSQL backends.
|
||||
* Canonical operator workflows required for development and destructive-test recovery run without a Windows-only shell dependency.
|
||||
|
||||
## Out of Scope
|
||||
|
||||
|
||||
+3
-2
@@ -10,8 +10,9 @@ Read [architecture_v3.md](https://www.google.com/search?q=architecture_v3.md) fi
|
||||
|
||||
* **Folder & Multi-Image Ingestion:** Upload one or more images that map sequentially (`page_number`) under a single `Document`.
|
||||
* **Parallel Async AI Vision Engine:** Concurrently process single-page image transcriptions using Python `asyncio` bounded by rate limiters.
|
||||
* **Portable Relational Storage:** Flexible relational persistence using SQLModel and SQLAlchemy supporting SQLite during local development/testing and PostgreSQL in production.
|
||||
* **Complete Auditability & Provenance:** Capture full input prompts (`system_prompt`, `user_prompt`), hyperparameters (`temperature`, `top_p`), operational metrics (`ai_metadata`), and full provider response envelopes (`raw_api_response`) on every page execution (`JobSource`).
|
||||
* **Portable Relational Storage:** SQLModel and SQLAlchemy preserve a portable relational model across the supported backends, with SQLite for local development/testing and PostgreSQL as the production database target.
|
||||
* **Cross-Platform Operations:** Canonical developer and recovery workflows run through Python-based, OS-independent tooling rather than platform-specific shell scripts.
|
||||
* **Complete Auditability & Provenance:** Capture frozen submission-time input prompts (`system_prompt`, `user_prompt`) and hyperparameters (`temperature`, `top_p`) on `Job`, plus per-page operational metrics (`ai_metadata`) and full provider response envelopes (`raw_api_response`) on `JobSource`.
|
||||
* **Asset Integrity Tracking:** Calculate and store cryptographic hashes (SHA-256) and file sizes on `Source` image records while preserving clean filesystem storage.
|
||||
* **Pydantic V2 Validation:** End-to-end type safety, DB row mapping, and JSON payload validation.
|
||||
* **Historical Person Management:** Track authors and recipients across documents with rich biographical entities (`Person`).
|
||||
|
||||
@@ -9,15 +9,16 @@ This document captures the **Version 3 baseline requirements** for the productio
|
||||
| REQ-0 | System | Provide end-to-end multi-page document transcription with persistent, inspectable async job states. | demonstration |
|
||||
| REQ-1 | Functional | Allow users to upload multi-image batches as sequential `Source` pages under a `Document`. | test |
|
||||
| REQ-2 | Functional | Process multi-page jobs asynchronously using an `asyncio` worker pool bounded by rate limits. | test |
|
||||
| REQ-3 | Functional | Persist page-level execution parameters, full input prompts, and output responses (`system_prompt`, `user_prompt`, `raw_transcription`, `ai_metadata`, `raw_api_response`) on `JobSource`. | test |
|
||||
| REQ-3 | Functional | Persist frozen submission-time execution parameters and full input prompts (`system_prompt`, `user_prompt`, `prompt_name`, `prompt_hash`, `temperature`, `top_p`) on `Job`, and persist page-level output responses (`raw_transcription`, `ai_metadata`, `raw_api_response`) on `JobSource`. | test |
|
||||
| REQ-4 | Functional | Support job states (`queued`, `processing`, `completed`, `partial_success`, `failed`) and page states (`pending`, `transcribed`, `failed`). | inspection |
|
||||
| REQ-5 | Functional | Allow users to manage historical `Person` records and link multiple authors/recipients to a `Document` via `DocumentPerson`. | test |
|
||||
| REQ-6 | Functional | Maintain immutable original machine output on `Source.raw_transcription` while permitting inline human edits on `Source.revised_text`. | test |
|
||||
| REQ-7 | Data Constraint | Support relational persistence via SQLModel/SQLAlchemy across database backends (SQLite for local testing/development and PostgreSQL for production). | inspection |
|
||||
| REQ-7 | Data Constraint | Use SQLModel/SQLAlchemy to preserve a portable relational domain model and compatible data shape across the supported backends, with SQLite for local development/testing and PostgreSQL as the production system of record. | inspection |
|
||||
| REQ-8 | Data Constraint | Validate all API requests, database rows, and JSON structures using Pydantic V2 schemas and SQLModel. | test |
|
||||
| REQ-9 | Interface | Render multi-page transcriptions sequentially by `page_number` in the web UI with author/recipient metadata. | demonstration |
|
||||
| REQ-10 | Operations | Allow operators to retry only failed pages for jobs in a `partial_success` state. | test |
|
||||
| REQ-10 | Operations | Allow operators to resubmit only failed pages for queued reprocessing while preserving the frozen prompt snapshot on the existing `Job`. | test |
|
||||
| REQ-11 | Data Constraint | Calculate and store cryptographic file hashes (SHA-256) and file sizes for uploaded source images to track asset integrity. | test |
|
||||
| REQ-12 | Operations Constraint | Keep core development, testing, restore, and recovery workflows OS-independent across Windows, Linux, and macOS; do not require a platform-specific shell for canonical project processes. | inspection |
|
||||
|
||||
## Element Satisfaction Mapping
|
||||
|
||||
@@ -26,6 +27,7 @@ This document captures the **Version 3 baseline requirements** for the productio
|
||||
* **WORKER (asyncio):** Satisfies REQ-2, REQ-3, REQ-4, REQ-10.
|
||||
* **PERSISTENCE (SQLModel/SQLAlchemy):** Satisfies REQ-3, REQ-6, REQ-7, REQ-11.
|
||||
* **MODELS (Pydantic V2 / SQLModel):** Satisfies REQ-8.
|
||||
* **OPERATIONS TOOLING (Python / OS-neutral automation):** Satisfies REQ-12.
|
||||
|
||||
---
|
||||
|
||||
|
||||
+8
-8
@@ -1,6 +1,6 @@
|
||||
# Database Schema (Version 3)
|
||||
|
||||
This document describes the relational schema for the transcription platform. It incorporates multi-image batch orchestration, page-level execution tracking, many-to-many author/recipient attribution, input prompt provenance capture, and raw API payload evidence for archival auditing.
|
||||
This document describes the relational schema for the transcription platform. It incorporates multi-image batch orchestration, page-level execution tracking, many-to-many author/recipient attribution, submission-time prompt snapshot capture, and raw API payload evidence for archival auditing.
|
||||
|
||||
The schema uses generic JSON columns compatible with SQLite in local development and PostgreSQL native JSONB/UUID types in production.
|
||||
|
||||
@@ -54,6 +54,12 @@ JOB {
|
||||
INTEGER retry_count
|
||||
TEXT provider
|
||||
TEXT model
|
||||
TEXT prompt_name
|
||||
TEXT prompt_hash
|
||||
TEXT system_prompt
|
||||
TEXT user_prompt
|
||||
FLOAT temperature
|
||||
FLOAT top_p
|
||||
TIMESTAMPTZ date_created
|
||||
TIMESTAMPTZ date_updated
|
||||
}
|
||||
@@ -79,12 +85,6 @@ JOB_SOURCE {
|
||||
UUID source_id FK
|
||||
VARCHAR status "pending | transcribed | failed"
|
||||
TEXT raw_transcription
|
||||
TEXT prompt_name
|
||||
TEXT prompt_hash
|
||||
TEXT system_prompt
|
||||
TEXT user_prompt
|
||||
FLOAT temperature
|
||||
FLOAT top_p
|
||||
JSONB ai_metadata
|
||||
JSONB raw_api_response
|
||||
TEXT error_detail
|
||||
@@ -104,7 +104,7 @@ SOURCE ||--o{ JOB_SOURCE : "processed_in"
|
||||
### Page-Level Execution & AI Outputs
|
||||
|
||||
* **Execution Granularity:** Every single image execution attempt by an AI model produces a dedicated record in `job_source`.
|
||||
* **Page-Level Input Provenance:** Every `job_source` execution captures its exact hyperparameters (`temperature`, `top_p`), prompt identifier details (`prompt_name`, `prompt_hash`), and full prompt text strings (`system_prompt`, `user_prompt`) used for that specific page call.
|
||||
* **Submission Snapshot Provenance:** Every `job` captures the frozen prompt identifier details (`prompt_name`, `prompt_hash`), full prompt text strings (`system_prompt`, `user_prompt`), and hyperparameters (`temperature`, `top_p`) at submission time.
|
||||
* **Point-in-Time Output Auditability:** `job_source.raw_api_response` stores the complete, unedited provider REST response envelope for that specific image page call. `job_source.ai_metadata` stores spatial bounding boxes, normalized token usage, latency, and cost details for fast querying.
|
||||
* **Active Output Caching:** Upon successful completion of an image call, `source.raw_transcription` is updated with the latest output string from `job_source.raw_transcription` for fast UI rendering.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user