transcription/docs/index_v3.md

Document Transcription System Overview (Version 3)

This project is a personal-scale application for transcribing, indexing, and preserving historical family documents, letters, postcards, and journals.

Start Here

Read architecture_v3.md first for technical overview and system design.

Core V3 Capabilities

• Folder & Multi-Image Ingestion: Upload one or more images that map sequentially (page_number) under a single Document.
• Parallel Async AI Vision Engine: Concurrently process single-page image transcriptions using Python asyncio bounded by rate limiters.
• Portable Relational Storage: Flexible relational persistence using SQLModel and SQLAlchemy supporting SQLite during local development/testing and PostgreSQL in production.
• Complete Auditability & Provenance: Capture full input prompts (system_prompt, user_prompt), hyperparameters (temperature, top_p), operational metrics (ai_metadata), and full provider response envelopes (raw_api_response) on every page execution (JobSource).
• Asset Integrity Tracking: Calculate and store cryptographic hashes (SHA-256) and file sizes on Source image records while preserving clean filesystem storage.
• Pydantic V2 Validation: End-to-end type safety, DB row mapping, and JSON payload validation.
• Historical Person Management: Track authors and recipients across documents with rich biographical entities (Person).
• Page-Level Execution Auditing & Revisions: Store immutable point-in-time machine output per run while enabling inline human corrections (revised_text).
• Partial Failure Recovery: Bounded batch execution that isolates single-page API errors (partial_success) for simple retries.

Technical Stack

• Application Web Framework: FastAPI + NiceGUI
• Persistence Engine: SQLModel / SQLAlchemy (SQLite for development/testing, PostgreSQL for production)
• Data Validation & Schemas: Pydantic V2
• Concurrency & Workers: Python asyncio worker pool with asyncio.Semaphore
• Vision Providers: OpenAI, Anthropic, and OpenRouter Vision models via native SDK adapters

---

Technology References

• FastAPI documentation
• NiceGUI documentation
• SQLModel documentation
• Python asyncio
• Pydantic Validation

Documentation Index

• System Overview (this document)
• System Design Intent
• Transcription Methodology
• System Architecture
• System Requirements
• Data model
• Error Handling Policy
• Implementation Plan