generated from john/python-template
50 lines
3.1 KiB
Markdown
50 lines
3.1 KiB
Markdown
# Document Transcription System Overview (Version 3)
|
|
|
|
This project is a personal-scale application for transcribing, indexing, and preserving historical family documents, letters, postcards, and journals.
|
|
|
|
## Start Here
|
|
|
|
Read [architecture_v3.md](https://www.google.com/search?q=architecture_v3.md) first for technical overview and system design.
|
|
|
|
## Core V3 Capabilities
|
|
|
|
* **Folder & Multi-Image Ingestion:** Upload one or more images that map sequentially (`page_number`) under a single `Document`.
|
|
* **Parallel Async AI Vision Engine:** Concurrently process single-page image transcriptions using Python `asyncio` bounded by rate limiters.
|
|
* **Portable Relational Storage:** SQLModel and SQLAlchemy preserve a portable relational model across the supported backends, with SQLite for local development/testing and PostgreSQL as the production database target.
|
|
* **Cross-Platform Operations:** Canonical developer and recovery workflows run through Python-based, OS-independent tooling rather than platform-specific shell scripts.
|
|
* **Complete Auditability & Provenance:** Capture frozen submission-time input prompts (`system_prompt`, `user_prompt`) and hyperparameters (`temperature`, `top_p`) on `Job`, plus per-page operational metrics (`ai_metadata`) and full provider response envelopes (`raw_api_response`) on `JobSource`.
|
|
* **Asset Integrity Tracking:** Calculate and store cryptographic hashes (SHA-256) and file sizes on `Source` image records while preserving clean filesystem storage.
|
|
* **Pydantic V2 Validation:** End-to-end type safety, DB row mapping, and JSON payload validation.
|
|
* **Historical Person Management:** Track authors and recipients across documents with rich biographical entities (`Person`).
|
|
* **Page-Level Execution Auditing & Revisions:** Store immutable point-in-time machine output per run while enabling inline human corrections (`revised_text`).
|
|
* **Partial Failure Recovery:** Bounded batch execution that isolates single-page API errors (`partial_success`) for simple retries.
|
|
|
|
## Technical Stack
|
|
|
|
* **Application Web Framework:** FastAPI + NiceGUI
|
|
* **Persistence Engine:** SQLModel / SQLAlchemy (SQLite for development/testing, PostgreSQL for production)
|
|
* **Data Validation & Schemas:** Pydantic V2
|
|
* **Concurrency & Workers:** Python `asyncio` worker pool with `asyncio.Semaphore`
|
|
* **Vision Providers:** OpenAI, Anthropic, and OpenRouter Vision models via native SDK adapters
|
|
|
|
---
|
|
|
|
## Technology References
|
|
|
|
* [FastAPI documentation](https://fastapi.tiangolo.com/)
|
|
* [NiceGUI documentation](https://nicegui.io/documentation)
|
|
* [SQLModel documentation](https://sqlmodel.tiangolo.com/)
|
|
* [Python asyncio](https://www.google.com/search?q=https://docs.python.org/3/library/asyncio.html%23module-asyncio)
|
|
* [Pydantic Validation](https://pydantic.dev/docs/validation/latest/get-started/)
|
|
|
|
## Documentation Index
|
|
|
|
- System Overview (this document)
|
|
- [System Design Intent](invariant/intent.md)
|
|
- [Transcription Methodology](invariant/transcription_methodology.md)
|
|
- [System Architecture](architecture_v3.md)
|
|
- [System Requirements](requirements_v3.md)
|
|
- [Data model](schema_v3.md)
|
|
- [Error Handling Policy](error_handling_v3.md)
|
|
- [Implementation Plan](implementation_plan_v3.md)
|