transcription/docs/index_v2.md

Document Transcription System Overview (Version 2)

This project is a personal-scale application for transcribing, indexing, and preserving historical family documents, letters, postcards, and journals.

Start Here

Read architecture_v2.md first for technical overview and system design.

Core V2 Capabilities

• Folder & Multi-Image Ingestion: Upload whole folders or image batches that map sequentially (page_number) under a single Document.
• Parallel Async AI Vision Engine: Concurrently process single-page image transcriptions using Python asyncio bounded by rate limiters.
• Robust PostgreSQL Storage: Relational storage for entities with native UUID, TIMESTAMPTZ, and JSONB for deep AI spatial metadata and raw envelopes.
• Pydantic V2 Validation: End-to-end type safety, DB row mapping, and JSONB payload validation.
• Historical Person Management: Track authors and recipients across documents with rich biographical entities (Person).
• Page-Level Execution Auditing & Revisions: Store immutable point-in-time machine output per run while enabling inline human corrections (revised_text).
• Partial Failure Recovery: Bounded batch execution that isolates single-page API errors (partial_success) for simple retries.

Technical Stack

• Application Web Framework: FastAPI + NiceGUI
• Persistence Engine: PostgreSQL 18+
• Data Validation & Schemas: Pydantic V2
• Concurrency & Workers: Python asyncio worker pool with asyncio.Semaphore
• Vision Providers: OpenAI (GPT-4o) and Anthropic (Claude 3.5 Sonnet) via native SDKs

---

Technology References

• FastAPI documentation
• NiceGUI documentation
• PostgreSQL documentation
• Python asyncio
• Pydantic Validation
• Pydantic AI

Documentation Index

• System Overview (this document)
• System Design Intent
• Transcription Methodology
• System Architecture
• System Requirements
• Data model
• Error Handling Policy
• Implementation Plan