transcription/docs/architecture_v2.md

System Architecture (Version 2)

This document describes the V2 production architecture of the personal historical-document transcription system.

Architecture Objectives

• Preserve source material as immutable transcribed text alongside page-level spatial AI metadata.
• Support batching multi-image and folder uploads cleanly into sequential pages (page_number).
• Leverage asynchronous worker pools (asyncio) for parallel single-image API execution bounded by rate limiters (asyncio.Semaphore).
• Migrate persistence to PostgreSQL using native UUID, TIMESTAMPTZ, and JSONB document storage.
• Standardize all data validation, API parsing, and database models on Pydantic V2.
• Support rich historical attribution (multi-author and multi-recipient relationships).

Runtime Topology

The V2 runtime operates as an asynchronous Python application:

• FastAPI + NiceGUI web application process.
• In-process asyncio background task orchestrator for parallel API execution.
• Relational persistence via PostgreSQL (using asyncpg or psycopg3).
• Pydantic V2 validation layer wrapping API payloads and PostgreSQL JSONB schemas.

flowchart LR
U[Browser User] --> A[FastAPI + NiceGUI App]
A --> W[Asyncio Worker Engine]
A --> DB[(PostgreSQL Database)]
W --> P[Vision Provider APIs\nOpenAI / Claude]
W --> DB

Lifecycle Ownership

Application lifespan owns runtime setup/teardown:

• Initialize environment logging and Pydantic configuration.
• Manage asynchronous PostgreSQL connection pools (asyncpg / psycopg3).
• Execute database migrations and index initialization.
• Recover stale processing jobs on startup.
• Manage graceful shutdown of active asyncio worker pools.

Layered Module Structure

Interface Layer

• src/transcription/ui/** (NiceGUI pages, multi-page renderers, person cards)
• src/transcription/api/** (FastAPI routes and JSON error handlers)

Application & Async Worker Layer

• src/transcription/services/workflows.py
• src/transcription/worker.py

Responsibilities:

• Batch orchestration and status transitions (queued -> processing -> completed | partial_success | failed).
• Parallel single-image API execution using asyncio.gather bounded by asyncio.Semaphore.
• Pydantic schema parsing (PageAIMetadata) and validation prior to database storage.

Domain & Service Layer

• src/transcription/models/*.py (Pydantic V2 schemas and entity definitions)
• src/transcription/services/*.py (Transactional operations for Document, Person, Source, Job, and JobSource)

Infrastructure Layer

• src/transcription/db/** (PostgreSQL connection pooling and raw parameterized SQL execution)
• src/transcription/providers/** (OpenAI & Anthropic Vision SDK adapters)

Processing Workflow

1. User uploads a folder or batch of images for a Document.
2. System creates Document, Job(status='queued'), and ordered Source pages (page_number = 1..N).
3. Worker claims job, sets Job.status = 'processing', and spawns parallel asyncio tasks bounded by semaphore.
4. Each task calls Vision API for a single Source image.
5. On task completion:

• Writes a JobSource record containing status='transcribed', raw_transcription, ai_metadata (bounding boxes/confidence), and raw_api_response.
• Caches active text to Source.raw_transcription.

6. On page failure:

• Writes JobSource record with status='failed' and error_detail.

7. Once all page tasks resolve:

• Marks Job.status as completed (100% success), partial_success (at least 1 success, 1 failure), or failed (all failed).

Domain Ownership & Invariants

• Immutable AI Outputs: source.raw_transcription and job_source.raw_transcription store original, point-in-time machine output and are immutable.
• Inlined Revisions: Human corrections occur on source.revised_text. UI renders COALESCE(revised_text, raw_transcription).
• Sequential Integrity: Multi-page documents are strictly ordered by source.page_number ASC.
• Page Execution Isolation: A failure on one page image does not invalidate successful transcriptions on sister pages in the same batch job.

Data Model Summary

• Document has many Source pages, many Job runs, and many Person records via DocumentPerson junction (author or recipient).
• Source belongs to one Document and can be processed across many JobSource executions.
• Job has many JobSource execution records.

Test Strategy

• Unit tests for Pydantic V2 schemas, custom validators, and JSONB serialization.
• Integration tests for async PostgreSQL connection handling and parameterized queries.
• Async workflow tests using mock AI providers to verify partial_success and retry logic.
• UI integration tests for multi-page rendering and person management.

---

Technology References

• FastAPI documentation
• NiceGUI documentation
• PostgreSQL documentation
• Python asyncio
• Pydantic Validation
• Pydantic AI

Related Local References

• System Overview
• System Design Intent
• Transcription Methodology
• System Architecture (this document)
• System Requirements
• Data model
• Error Handling Policy
• Implementation Plan