Add starting docs

2026-06-21 18:54:42 -05:00
parent 974e328150
commit d0c5e3cb7a
2 changed files with 128 additions and 0 deletions
@@ -0,0 +1,33 @@
 # Historical Document Transcription
 I have several thousand pages of family history told through letters, postcards, books, and other documents that I want to transcribe to text. 
 ## Goals
 1. Preserve our family history
 2. Unburden my family (and descendants) from having to store and care for the physical media. Once the documents have been transcribed and organized, they can be donated (or kept by a family member that wants to retain them).
 3. Make the text easily available and searchable by family members, as well as AI (which may have different requirements).
 4. Ability create timelines or assemble the historical record of the family or specific individuals from across the complete document archive. Perhaps use AI to create the timelines in a more narrative form.
 ## Source material
 1. **letters, cards, diaries** - handwritten; mostly stored in tubs, with little organization
 2. **books** - typed or typeset; mostly self-published books 50-100 pages in length. This may be expanded to include selected pages from other publications.
 3. **newspaper clippings, event programs, invitations, and other ephemera**
 ## Methodology
 ### Verbatim vs. Clean Copy
 Transcriptions should be Verbatim and follow scholarly research guidelines, with no modifications to the original text.
 ### Potential Document Issues
 | Document Issue | How to Handle It | Example |
 | :--- | :--- | :--- |
 | **Misspellings & Errors** | Retain original spelling and insert italicized `[sic]` directly after the error. | `The weather was very cold and publick [sic] business delayed.` |
 | **Missing Words / Slips** | Insert the missing word inside square brackets to restore basic readability. | `We went [to] the store to buy supplies.` |
 | **Uncertain / Guesswork** | Place your best hypothesis followed by a question mark inside square brackets. | `He went to [Boston?] yesterday to meet the governor.` |
 | **Completely Illegible** | Use a clear descriptive term like `[illegible]` or specify the reason (e.g., `[torn]`, `[ink blot]`). | `The total cost was [illegible] dollars.` or `The letter ends here [remainder of page torn].` |
 | **Crossed-out Text** | Wrap the removed word or phrase in a deleted tag to preserve the author's edits. | `We left at [deleted: noon] one o'clock instead.` |
 | **Squeezed-in Text** | Wrap text that was added above the line or in a tight space in an inserted tag. | `The [inserted: red] house on the hill was abandoned.` |
 | **Superscripts & Abbreviations** | Bring raised letters down to the main line, or optionally expand them in brackets. | `Change Gen^l to Genl` OR `Change to Gen[era]l depending on project preference.` |
 | **Images / Seals / Signs** | Describe the non-textual element using italicized text inside square brackets. | `[wax notary seal attached here]` or `[sketch of a fort layout]` |
 | **Marginalia / Notes** | Note the spatial transition clearly before transcribing the note itself. | `[written in left margin:] Do not share this with anyone.` |
 | **Line Breaks / Hyphens** | Rejoin words split across a page margin silently, dropping the line-break hyphen. | `Original: "estab- / lishment" becomes "establishment"` |
 | **Ambiguous Capitalization** | Default to modern capitalization rules unless an archaic uppercase letter is clearly intentional. | `If a standard noun like 'Farm' looks randomly capitalized, type 'farm'.` |
 **Hierarchical Outlines** | Preserve exact numbering characters (including lowercase Roman numerals or terminal 'j'). Replicate indentation levels using spaces/tabs. Do not correct math or sequence errors silently. | `I. Main Topic`<br>`&nbsp;&nbsp;a. Sub-point`<br>`&nbsp;&nbsp;b. Next point`<br>`III. [sic] Third Topic` |
@@ -0,0 +1,95 @@
 ## Handwriting Transcription System
 This project is a starter for transcribing historical documents with an LLM-powered, graph-based backend. It combines a NiceGUI web interface, a FastAPI + LangGraph application, and PostgreSQL persistence behind an Nginx entrypoint.
 The system is designed to be easy to run locally with Docker Compose and easy to extend for more advanced orchestration, scaling, and model strategies.
 ## What The System Does
 At a high level, users upload one or more document images from the web UI. Each upload is tracked as a job in the database, processed through a LangGraph workflow, transcribed by an LLM, and stored with status history and events.
 Core outcomes:
 - Upload handwritten images through a minimal web interface.
 - Persist image data, job state, events, errors, and transcription text in PostgreSQL.
 - Track lifecycle states from upload through completion or failure.
 - Inspect job status and results through API endpoints and UI pages.
 ## Architecture Overview
 The project uses a single Python backend that serves both API endpoints and NiceGUI pages.
 ### Front End
 - NiceGUI pages mounted on FastAPI with periodic refresh for live job status updates.
 - Supports uploading images, viewing current job states, and reading completed transcriptions.
 - Displays failure states and error messages for troubleshooting.
 ### Backend
 - FastAPI app for HTTP endpoints and page rendering.
 - LangGraph workflow for multi-step transcription execution.
 - Background task execution for asynchronous processing after upload.
 - Centralized app configuration through pydantic-settings.
 - Centralized logging initialization via one logging.config setup call at startup.
 - Lifespan-owned runtime resources for the SQLAlchemy engine, async session factory, PostgreSQL checkpoint connection, and compiled graph.
 ### Database
 - PostgreSQL is the only persistent store.
 - SQLModel defines schema and data access.
 - SQLAlchemy async access uses one engine per process and one async session per request or concurrent background task.
 - Persists:
 	- image records
 	- transcription jobs
 	- processing events/history
 	- transcription output
 	- error details
 - LangGraph checkpointing is stored in PostgreSQL for resumable workflow state.
 - Schema bootstrap is explicit and opt-in; normal startup does not mutate production schema automatically.
 ### Infrastructure
 - Docker Compose runs exactly three containers:
 	- backend (FastAPI + LangGraph)
 	- frontend (Nginx reverse proxy)
 	- db (PostgreSQL)
 - Nginx acts as the public entrypoint and proxies requests to the backend.
 ## Processing Lifecycle
 Each uploaded image moves through explicit statuses:
 - upload
 - queued
 - processing
 - transcribed
 - failed
 - completed
 Typical flow:
 1. Image is uploaded and validated.
 2. Image and job metadata are stored in PostgreSQL.
 3. Job is queued and processed through LangGraph nodes.
 4. LLM transcription is generated.
 5. Result and processing events are saved.
 6. Job ends as completed or failed with error details.
 This state-driven model enables reliable inspection, retries, and recovery.
 ## Configuration And Observability
 Configuration is loaded once at startup using a pydantic-settings class and can be propagated through request/workflow execution via context variables where scoped access is needed.
 Logging is initialized once through a centralized logging.config call, and modules use named loggers for consistent observability across API, workflow, and persistence layers.
 Readiness checks validate both SQLAlchemy connectivity and graph runtime initialization so operational status reflects the actual owned runtime resources.
 ## Why This Starter Exists
 This project intentionally balances practicality and extensibility:
 - Minimal UI and straightforward APIs for fast iteration.
 - Durable workflow state and clear job history for operational visibility.
 - Clean separation of concerns across API, graph nodes, data models, and infrastructure.
 - Local-first developer experience with uv and Docker Compose.
 It is suitable as a baseline for production systems that need better queueing, multi-worker scaling, richer auth, or additional document processing features.
 ## Related Documentation
 - Project description: [docs/Historical_Document_Transcription.md](docs/Historical_Document_Transcription.md)
 - Project build prompt: