generated from john/python-template
141 lines
7.0 KiB
Markdown
141 lines
7.0 KiB
Markdown
# System Architecture (Version 3)
|
|
|
|
This document describes the V3 production architecture of the personal historical-document transcription system.
|
|
|
|
## Architecture Objectives
|
|
|
|
* Preserve source material as immutable transcribed text alongside page-level spatial AI metadata and complete provider API envelopes.
|
|
* Support batching multi-image and folder uploads cleanly into sequential pages (`page_number`).
|
|
* Capture complete input prompt provenance (`system_prompt`, `user_prompt`, `prompt_hash`) and execution parameters (`temperature`, `top_p`) at the page execution level (`JobSource`).
|
|
* Leverage asynchronous worker pools (`asyncio`) for parallel single-image API execution bounded by rate limiters (`asyncio.Semaphore`).
|
|
* Maintain relational database portability across engines (SQLite for development/testing, PostgreSQL for production) using SQLModel and generic JSON abstraction layers.
|
|
* Verify image asset integrity via SHA-256 file hashing (`file_hash`) while storing binary assets on the local filesystem.
|
|
* Standardize all data validation, API parsing, and database models on **Pydantic V2** and **SQLModel**.
|
|
* Support rich historical attribution (multi-author and multi-recipient relationships via `DocumentPerson`).
|
|
|
|
## Runtime Topology
|
|
|
|
The V3 runtime operates as an asynchronous Python application:
|
|
|
|
* FastAPI + NiceGUI web application process.
|
|
* In-process `asyncio` background task orchestrator for parallel API execution.
|
|
* Relational persistence via SQLModel / SQLAlchemy (SQLite engine in local development/testing, PostgreSQL engine in production).
|
|
* Pydantic V2 validation layer wrapping API payloads, prompt configurations, and JSON metadata schemas.
|
|
|
|
^^^mermaid
|
|
flowchart LR
|
|
U[Browser User] --> A[FastAPI + NiceGUI App]
|
|
A --> W[Asyncio Worker Engine]
|
|
A --> DB[(Relational DB\nSQLite / PostgreSQL)]
|
|
W --> P[Vision Provider APIs\nOpenAI / Claude / OpenRouter]
|
|
W --> DB
|
|
^^^
|
|
|
|
## Lifecycle Ownership
|
|
|
|
Application lifespan owns runtime setup/teardown:
|
|
|
|
* Initialize environment logging, directory paths, and Pydantic configuration.
|
|
* Manage asynchronous database engine connection pools (`aiosqlite` or `asyncpg`).
|
|
* Execute database bootstrap (`SQLModel.metadata.create_all()`) or migrations.
|
|
* Recover stale or interrupted processing jobs on startup.
|
|
* Manage graceful shutdown of active `asyncio` worker pools.
|
|
|
|
## Layered Module Structure
|
|
|
|
### Interface Layer
|
|
|
|
* `src/transcription/ui/**` (NiceGUI pages, multi-page renderers, person cards)
|
|
* `src/transcription/api/**` (FastAPI routes and JSON error handlers)
|
|
|
|
### Application & Async Worker Layer
|
|
|
|
* `src/transcription/services/workflows.py`
|
|
* `src/transcription/worker.py`
|
|
|
|
Responsibilities:
|
|
|
|
* Batch orchestration and status transitions (`queued` -> `processing` -> `completed` | `partial_success` | `failed`).
|
|
* Parallel single-image API execution using `asyncio.gather` bounded by `asyncio.Semaphore`.
|
|
* Page-level prompt construction, logging full `system_prompt` and `user_prompt` to `JobSource`.
|
|
* Pydantic schema parsing and validation prior to database storage.
|
|
|
|
### Domain & Service Layer
|
|
|
|
* `src/transcription/db/models.py` (SQLModel schema definitions for Document, Source, Job, JobSource, Person, DocumentPerson)
|
|
* `src/transcription/services/*.py` (Transactional operations for `Document`, `Person`, `Source`, `Job`, and `JobSource`)
|
|
|
|
### Infrastructure Layer
|
|
|
|
* `src/transcription/db/**` (Async database session factory, engine creation, and JSON dialect abstractions)
|
|
* `src/transcription/providers/**` (OpenAI, Anthropic, and OpenRouter Vision SDK adapters)
|
|
|
|
## Processing Workflow
|
|
|
|
1. User uploads a folder or batch of images for a `Document`.
|
|
2. System hashes each image file (SHA-256), writes image files to filesystem storage, and creates `Document`, `Job(status='queued')`, and ordered `Source` pages (`page_number = 1..N`).
|
|
3. Worker claims job, sets `Job.status = 'processing'`, and spawns parallel `asyncio` tasks bounded by semaphore.
|
|
4. Each task builds page-specific system/user prompts and calls Vision API for a **single** `Source` image.
|
|
5. On task completion:
|
|
* Writes a `JobSource` record containing `status='transcribed'`, `raw_transcription`, complete input details (`prompt_name`, `prompt_hash`, `system_prompt`, `user_prompt`, `temperature`, `top_p`), operational `ai_metadata`, and complete unedited `raw_api_response`.
|
|
* Caches active output text to `Source.raw_transcription`.
|
|
|
|
|
|
6. On page failure:
|
|
* Writes `JobSource` record with `status='failed'`, recorded prompt inputs, and `error_detail`.
|
|
|
|
|
|
7. Once all page tasks resolve:
|
|
* Marks `Job.status` as `completed` (100% success), `partial_success` (at least 1 success, 1 failure), or `failed` (all failed).
|
|
|
|
|
|
|
|
## Domain Ownership & Invariants
|
|
|
|
* **Immutable AI Outputs:** `source.raw_transcription` and `job_source.raw_transcription` store original, point-in-time machine output and are immutable.
|
|
* **Complete Input & Output Provenance:** Every `job_source` record contains both the exact input configuration sent to the model and the complete REST response envelope returned.
|
|
* **Inlined Revisions:** Human corrections occur on `source.revised_text`. UI renders `COALESCE(revised_text, raw_transcription)`.
|
|
* **Sequential Integrity:** Multi-page documents are strictly ordered by `source.page_number ASC`.
|
|
* **Page Execution Isolation:** A failure on one page image does not invalidate successful transcriptions on sister pages in the same batch job.
|
|
|
|
## Data Model Summary
|
|
|
|
* `Document` has many `Source` pages, many `Job` runs, and many `Person` records via `DocumentPerson` junction (`author` or `recipient`).
|
|
* `Source` belongs to one `Document` and can be processed across many `JobSource` executions.
|
|
* `Job` has many `JobSource` execution records.
|
|
* `JobSource` holds page-level prompts, parameters, execution status, and raw response JSON.
|
|
|
|
## Test Strategy
|
|
|
|
* Unit tests for SQLModel/Pydantic V2 models, JSON cross-dialect serialization, and file hashing functions.
|
|
* Integration tests for async database connection handling, session management, and queries.
|
|
* Async workflow tests using mock AI providers to verify `partial_success`, page-level failure isolation, and retry logic.
|
|
* UI integration tests for multi-page rendering and person attribution management.
|
|
|
|
---
|
|
|
|
## Technology References
|
|
|
|
* [FastAPI documentation](https://fastapi.tiangolo.com/)
|
|
* [NiceGUI documentation](https://nicegui.io/documentation)
|
|
* [SQLModel documentation](https://sqlmodel.tiangolo.com/)
|
|
* [SQLAlchemy Async I/O documentation](https://docs.sqlalchemy.org/en/20/orm/extensions/asyncio.html)
|
|
* [Python asyncio](https://www.google.com/search?q=https://docs.python.org/3/library/asyncio.html%23module-asyncio)
|
|
* [Pydantic Validation](https://pydantic.dev/docs/validation/latest/get-started/)
|
|
|
|
## Related Local References
|
|
|
|
- [System Overview](index_v2.md)
|
|
- [System Design Intent](invariant/intent.md)
|
|
- [Transcription Methodology](invariant/transcription_methodology.md)
|
|
- System Architecture (this document)
|
|
- [System Requirements](requirements_v2.md)
|
|
- [Data model](schema_v2.md)
|
|
- [Error Handling Policy](error_handling_v2.md)
|
|
- [Implementation Plan](implementation_plan_v2.md)
|
|
|
|
|
|
|
|
|
|
|