generated from john/python-template
48 lines
2.4 KiB
Markdown
48 lines
2.4 KiB
Markdown
# Document Transcription System Overview (Version 2)
|
|
|
|
This project is a personal-scale application for transcribing, indexing, and preserving historical family documents, letters, postcards, and journals.
|
|
|
|
## Start Here
|
|
|
|
Read [architecture_v2.md](architecture_v2.md) first for technical overview and system design.
|
|
|
|
## Core V2 Capabilities
|
|
|
|
* **Folder & Multi-Image Ingestion:** Upload whole folders or image batches that map sequentially (`page_number`) under a single `Document`.
|
|
* **Parallel Async AI Vision Engine:** Concurrently process single-page image transcriptions using Python `asyncio` bounded by rate limiters.
|
|
* **Robust PostgreSQL Storage:** Relational storage for entities with native `UUID`, `TIMESTAMPTZ`, and `JSONB` for deep AI spatial metadata and raw envelopes.
|
|
* **Pydantic V2 Validation:** End-to-end type safety, DB row mapping, and JSONB payload validation.
|
|
* **Historical Person Management:** Track authors and recipients across documents with rich biographical entities (`Person`).
|
|
* **Page-Level Execution Auditing & Revisions:** Store immutable point-in-time machine output per run while enabling inline human corrections (`revised_text`).
|
|
* **Partial Failure Recovery:** Bounded batch execution that isolates single-page API errors (`partial_success`) for simple retries.
|
|
|
|
## Technical Stack
|
|
|
|
* **Application Web Framework:** FastAPI + NiceGUI
|
|
* **Persistence Engine:** PostgreSQL 18+
|
|
* **Data Validation & Schemas:** Pydantic V2
|
|
* **Concurrency & Workers:** Python `asyncio` worker pool with `asyncio.Semaphore`
|
|
* **Vision Providers:** OpenAI (GPT-4o) and Anthropic (Claude 3.5 Sonnet) via native SDKs
|
|
|
|
---
|
|
|
|
## Technology References
|
|
|
|
- [FastAPI documentation](https://fastapi.tiangolo.com/)
|
|
- [NiceGUI documentation](https://nicegui.io/documentation)
|
|
- [PostgreSQL documentation](https://www.postgresql.org/docs/)
|
|
- [Python asyncio](https://docs.python.org/3/library/asyncio.html#module-asyncio)
|
|
- [Pydantic Validation](https://pydantic.dev/docs/validation/latest/get-started/)
|
|
- [Pydantic AI](https://pydantic.dev/docs/ai/overview/)
|
|
|
|
## Documentation Index
|
|
|
|
- System Overview (this document)
|
|
- [System Design Intent](invariant/intent.md)
|
|
- [Transcription Methodology](invariant/transcription_methodology.md)
|
|
- [System Architecture](architecture_v2.md)
|
|
- [System Requirements](requirements_v2.md)
|
|
- [Data model](schema_v2.md)
|
|
- [Error Handling Policy](error_handling_v2.md)
|
|
- [Implementation Plan](implementation_plan_v2.md)
|