System Architecture (Version 3)

This document describes the V3 production architecture of the personal historical-document transcription system.

Architecture Objectives

Preserve source material as immutable transcribed text alongside page-level spatial AI metadata and complete provider API envelopes.
Support batching multi-image and folder uploads cleanly into sequential pages (page_number).
Capture complete input prompt provenance (system_prompt, user_prompt, prompt_hash) and execution parameters (temperature, top_p) at the page execution level (JobSource).
Leverage asynchronous worker pools (asyncio) for parallel single-image API execution bounded by rate limiters (asyncio.Semaphore).
Maintain relational database portability across engines (SQLite for development/testing, PostgreSQL for production) using SQLModel and generic JSON abstraction layers.
Verify image asset integrity via SHA-256 file hashing (file_hash) while storing binary assets on the local filesystem.
Standardize all data validation, API parsing, and database models on Pydantic V2 and SQLModel.
Support rich historical attribution (multi-author and multi-recipient relationships via DocumentPerson).

Runtime Topology

The V3 runtime operates as an asynchronous Python application:

FastAPI + NiceGUI web application process.
In-process asyncio background task orchestrator for parallel API execution.
Relational persistence via SQLModel / SQLAlchemy (SQLite engine in local development/testing, PostgreSQL engine in production).
Pydantic V2 validation layer wrapping API payloads, prompt configurations, and JSON metadata schemas.

^^^mermaid flowchart LR U[Browser User] --> A[FastAPI + NiceGUI App] A --> W[Asyncio Worker Engine] A --> DB[(Relational DB\nSQLite / PostgreSQL)] W --> P[Vision Provider APIs\nOpenAI / Claude / OpenRouter] W --> DB ^^^

Lifecycle Ownership

Application lifespan owns runtime setup/teardown:

Initialize environment logging, directory paths, and Pydantic configuration.
Manage asynchronous database engine connection pools (aiosqlite or asyncpg).
Execute database bootstrap (SQLModel.metadata.create_all()) or migrations.
Recover stale or interrupted processing jobs on startup.
Manage graceful shutdown of active asyncio worker pools.

Layered Module Structure

Interface Layer

src/transcription/ui/** (NiceGUI pages, multi-page renderers, person cards)
src/transcription/api/** (FastAPI routes and JSON error handlers)

Application & Async Worker Layer

src/transcription/services/workflows.py
src/transcription/worker.py

Responsibilities:

Batch orchestration and status transitions (queued -> processing -> completed | partial_success | failed).
Parallel single-image API execution using asyncio.gather bounded by asyncio.Semaphore.
Page-level prompt construction, logging full system_prompt and user_prompt to JobSource.
Pydantic schema parsing and validation prior to database storage.

Domain & Service Layer

src/transcription/db/models.py (SQLModel schema definitions for Document, Source, Job, JobSource, Person, DocumentPerson)
src/transcription/services/*.py (Transactional operations for Document, Person, Source, Job, and JobSource)

Infrastructure Layer

src/transcription/db/** (Async database session factory, engine creation, and JSON dialect abstractions)
src/transcription/providers/** (OpenAI, Anthropic, and OpenRouter Vision SDK adapters)

Processing Workflow

User uploads a folder or batch of images for a Document.
System hashes each image file (SHA-256), writes image files to filesystem storage, and creates Document, Job(status='queued'), and ordered Source pages (page_number = 1..N).
Worker claims job, sets Job.status = 'processing', and spawns parallel asyncio tasks bounded by semaphore.
Each task builds page-specific system/user prompts and calls Vision API for a single Source image.
On task completion:

Writes a JobSource record containing status='transcribed', raw_transcription, complete input details (prompt_name, prompt_hash, system_prompt, user_prompt, temperature, top_p), operational ai_metadata, and complete unedited raw_api_response.
Caches active output text to Source.raw_transcription.

On page failure:

Writes JobSource record with status='failed', recorded prompt inputs, and error_detail.

Once all page tasks resolve:

Marks Job.status as completed (100% success), partial_success (at least 1 success, 1 failure), or failed (all failed).

Domain Ownership & Invariants

Immutable AI Outputs: source.raw_transcription and job_source.raw_transcription store original, point-in-time machine output and are immutable.
Complete Input & Output Provenance: Every job_source record contains both the exact input configuration sent to the model and the complete REST response envelope returned.
Inlined Revisions: Human corrections occur on source.revised_text. UI renders COALESCE(revised_text, raw_transcription).
Sequential Integrity: Multi-page documents are strictly ordered by source.page_number ASC.
Page Execution Isolation: A failure on one page image does not invalidate successful transcriptions on sister pages in the same batch job.

Data Model Summary

Document has many Source pages, many Job runs, and many Person records via DocumentPerson junction (author or recipient).
Source belongs to one Document and can be processed across many JobSource executions.
Job has many JobSource execution records.
JobSource holds page-level prompts, parameters, execution status, and raw response JSON.

Test Strategy

Unit tests for SQLModel/Pydantic V2 models, JSON cross-dialect serialization, and file hashing functions.
Integration tests for async database connection handling, session management, and queries.
Async workflow tests using mock AI providers to verify partial_success, page-level failure isolation, and retry logic.
UI integration tests for multi-page rendering and person attribution management.

7.0 KiB Raw Blame History