System Architecture (Version 3)

This document describes the V3 production architecture of the personal historical-document transcription system.

Architecture Objectives

Preserve source material as immutable transcribed text alongside page-level spatial AI metadata and complete provider API envelopes.
Support batching multi-image and folder uploads cleanly into sequential pages (page_number).
Capture complete input prompt provenance (system_prompt, user_prompt, prompt_hash) and execution parameters (temperature, top_p) at submission time on Job.
Leverage asynchronous worker pools (asyncio) for parallel single-image API execution bounded by rate limiters (asyncio.Semaphore).
Maintain relational data-model portability across the supported backends by using SQLModel/SQLAlchemy and compatibility types so the same domain schema works in SQLite for local development/testing and PostgreSQL in production.
Keep operator tooling and local maintenance workflows OS-independent by using Python or other cross-platform interfaces for canonical project automation.
Verify image asset integrity via SHA-256 file hashing (file_hash) while storing binary assets on the local filesystem.
Standardize all data validation, API parsing, and database models on Pydantic V2 and SQLModel.
Support rich historical attribution (multi-author and multi-recipient relationships via DocumentPerson).

Runtime Topology

The V3 runtime operates as an asynchronous Python application:

FastAPI + NiceGUI web application process.
In-process asyncio background task orchestrator for parallel API execution.
Relational persistence via SQLModel / SQLAlchemy, using SQLite for local development/testing and PostgreSQL as the production persistence target.
Pydantic V2 validation layer wrapping API payloads, prompt configurations, and JSON metadata schemas.
Cross-platform operator workflows implemented in Python so core local operations run consistently on Windows, Linux, and macOS.

^^^mermaid flowchart LR U[Browser User] --> A[FastAPI + NiceGUI App] A --> W[Asyncio Worker Engine] A --> DB[(Relational DB\nSQLite / PostgreSQL)] W --> P[Vision Provider APIs\nOpenAI / Claude / OpenRouter] W --> DB ^^^

Lifecycle Ownership

Application lifespan owns runtime setup/teardown:

Initialize environment logging, directory paths, and Pydantic configuration.
Manage asynchronous database engine connection pools (aiosqlite or asyncpg).
Execute database bootstrap (SQLModel.metadata.create_all()) or migrations.
Recover stale or interrupted processing jobs on startup.
Manage graceful shutdown of active asyncio worker pools.

Layered Module Structure

Interface Layer

src/transcription/ui/** (NiceGUI pages, multi-page renderers, person cards)
src/transcription/api/** (FastAPI routes and JSON error handlers)

Application & Async Worker Layer

src/transcription/services/workflows.py
src/transcription/worker.py

Responsibilities:

Batch orchestration and status transitions (queued -> processing -> transcribed | partial_success | failed).
Parallel single-image API execution using asyncio.gather bounded by asyncio.Semaphore.
Resolve prompt configuration at submission time and persist frozen snapshot fields on Job.
Pydantic schema parsing and validation prior to database storage.

Domain & Service Layer

src/transcription/db/models.py (SQLModel schema definitions for Document, Source, Job, JobSource, Person, DocumentPerson)
src/transcription/services/*.py (Transactional operations for Document, Person, Source, Job, and JobSource)

Infrastructure Layer

src/transcription/db/** (Async database session factory, engine creation, and JSON dialect abstractions)
src/transcription/providers/** (OpenAI, Anthropic, and OpenRouter Vision SDK adapters)

Processing Workflow

User uploads a folder or batch of images for a Document.
System hashes each image file (SHA-256), writes image files to filesystem storage, and creates Document, Job(status='queued'), and ordered Source pages (page_number = 1..N).
Worker claims job, sets Job.status = 'processing', and spawns parallel asyncio tasks bounded by semaphore.
Each task reads the frozen prompt snapshot from Job and calls Vision API for a single Source image.
On task completion:

Writes a JobSource record containing status='transcribed', raw_transcription, operational ai_metadata, and complete unedited raw_api_response.
Caches active output text to Source.raw_transcription.

On page failure:

Writes JobSource record with status='failed' and error_detail.

Once all page tasks resolve:

Marks Job.status as transcribed (100% success), partial_success (at least 1 success, 1 failure), or failed (all failed).

Domain Ownership & Invariants

Immutable AI Outputs: source.raw_transcription and job_source.raw_transcription store original, point-in-time machine output and are immutable.
Complete Input & Output Provenance: Every job stores the exact frozen input configuration sent to the model, and every job_source stores per-page output evidence including the complete REST response envelope returned.
Inlined Revisions: Human corrections occur on source.revised_text. UI renders COALESCE(revised_text, raw_transcription).
Sequential Integrity: Multi-page documents are strictly ordered by source.page_number ASC.
Page Execution Isolation: A failure on one page image does not invalidate successful transcriptions on sister pages in the same batch job.

Data Model Summary

Document has many Source pages, many Job runs, and many Person records via DocumentPerson junction (author or recipient).
Source belongs to one Document and can be processed across many JobSource executions.
Job has many JobSource execution records.
JobSource holds page-level execution status, output text, and raw response JSON.

Test Strategy

Unit tests for SQLModel/Pydantic V2 models, JSON cross-dialect serialization, and file hashing functions.
Integration tests for async database connection handling, session management, and queries.
Async workflow tests using mock AI providers to verify partial_success, page-level failure isolation, and retry logic.
UI integration tests for multi-page rendering and person attribution management.

7.2 KiB Raw Blame History