diff --git a/docs/Historical_Document_Transcription.md b/docs/Historical_Document_Transcription.md new file mode 100644 index 0000000..38b1ace --- /dev/null +++ b/docs/Historical_Document_Transcription.md @@ -0,0 +1,33 @@ +# Historical Document Transcription +I have several thousand pages of family history told through letters, postcards, books, and other documents that I want to transcribe to text. + +## Goals +1. Preserve our family history +2. Unburden my family (and descendants) from having to store and care for the physical media. Once the documents have been transcribed and organized, they can be donated (or kept by a family member that wants to retain them). +3. Make the text easily available and searchable by family members, as well as AI (which may have different requirements). +4. Ability create timelines or assemble the historical record of the family or specific individuals from across the complete document archive. Perhaps use AI to create the timelines in a more narrative form. + +## Source material +1. **letters, cards, diaries** - handwritten; mostly stored in tubs, with little organization +2. **books** - typed or typeset; mostly self-published books 50-100 pages in length. This may be expanded to include selected pages from other publications. +3. **newspaper clippings, event programs, invitations, and other ephemera** + + ## Methodology +### Verbatim vs. Clean Copy +Transcriptions should be Verbatim and follow scholarly research guidelines, with no modifications to the original text. + +### Potential Document Issues +| Document Issue | How to Handle It | Example | +| :--- | :--- | :--- | +| **Misspellings & Errors** | Retain original spelling and insert italicized `[sic]` directly after the error. | `The weather was very cold and publick [sic] business delayed.` | +| **Missing Words / Slips** | Insert the missing word inside square brackets to restore basic readability. | `We went [to] the store to buy supplies.` | +| **Uncertain / Guesswork** | Place your best hypothesis followed by a question mark inside square brackets. | `He went to [Boston?] yesterday to meet the governor.` | +| **Completely Illegible** | Use a clear descriptive term like `[illegible]` or specify the reason (e.g., `[torn]`, `[ink blot]`). | `The total cost was [illegible] dollars.` or `The letter ends here [remainder of page torn].` | +| **Crossed-out Text** | Wrap the removed word or phrase in a deleted tag to preserve the author's edits. | `We left at [deleted: noon] one o'clock instead.` | +| **Squeezed-in Text** | Wrap text that was added above the line or in a tight space in an inserted tag. | `The [inserted: red] house on the hill was abandoned.` | +| **Superscripts & Abbreviations** | Bring raised letters down to the main line, or optionally expand them in brackets. | `Change Gen^l to Genl` OR `Change to Gen[era]l depending on project preference.` | +| **Images / Seals / Signs** | Describe the non-textual element using italicized text inside square brackets. | `[wax notary seal attached here]` or `[sketch of a fort layout]` | +| **Marginalia / Notes** | Note the spatial transition clearly before transcribing the note itself. | `[written in left margin:] Do not share this with anyone.` | +| **Line Breaks / Hyphens** | Rejoin words split across a page margin silently, dropping the line-break hyphen. | `Original: "estab- / lishment" becomes "establishment"` | +| **Ambiguous Capitalization** | Default to modern capitalization rules unless an archaic uppercase letter is clearly intentional. | `If a standard noun like 'Farm' looks randomly capitalized, type 'farm'.` | +**Hierarchical Outlines** | Preserve exact numbering characters (including lowercase Roman numerals or terminal 'j'). Replicate indentation levels using spaces/tabs. Do not correct math or sequence errors silently. | `I. Main Topic`
`  a. Sub-point`
`  b. Next point`
`III. [sic] Third Topic` | \ No newline at end of file diff --git a/docs/index.md b/docs/index.md new file mode 100644 index 0000000..28df09c --- /dev/null +++ b/docs/index.md @@ -0,0 +1,95 @@ +## Handwriting Transcription System + +This project is a starter for transcribing historical documents with an LLM-powered, graph-based backend. It combines a NiceGUI web interface, a FastAPI + LangGraph application, and PostgreSQL persistence behind an Nginx entrypoint. + +The system is designed to be easy to run locally with Docker Compose and easy to extend for more advanced orchestration, scaling, and model strategies. + +## What The System Does + +At a high level, users upload one or more document images from the web UI. Each upload is tracked as a job in the database, processed through a LangGraph workflow, transcribed by an LLM, and stored with status history and events. + +Core outcomes: +- Upload handwritten images through a minimal web interface. +- Persist image data, job state, events, errors, and transcription text in PostgreSQL. +- Track lifecycle states from upload through completion or failure. +- Inspect job status and results through API endpoints and UI pages. + +## Architecture Overview + +The project uses a single Python backend that serves both API endpoints and NiceGUI pages. + +### Front End +- NiceGUI pages mounted on FastAPI with periodic refresh for live job status updates. +- Supports uploading images, viewing current job states, and reading completed transcriptions. +- Displays failure states and error messages for troubleshooting. + +### Backend +- FastAPI app for HTTP endpoints and page rendering. +- LangGraph workflow for multi-step transcription execution. +- Background task execution for asynchronous processing after upload. +- Centralized app configuration through pydantic-settings. +- Centralized logging initialization via one logging.config setup call at startup. +- Lifespan-owned runtime resources for the SQLAlchemy engine, async session factory, PostgreSQL checkpoint connection, and compiled graph. + +### Database +- PostgreSQL is the only persistent store. +- SQLModel defines schema and data access. +- SQLAlchemy async access uses one engine per process and one async session per request or concurrent background task. +- Persists: + - image records + - transcription jobs + - processing events/history + - transcription output + - error details +- LangGraph checkpointing is stored in PostgreSQL for resumable workflow state. +- Schema bootstrap is explicit and opt-in; normal startup does not mutate production schema automatically. + +### Infrastructure +- Docker Compose runs exactly three containers: + - backend (FastAPI + LangGraph) + - frontend (Nginx reverse proxy) + - db (PostgreSQL) +- Nginx acts as the public entrypoint and proxies requests to the backend. + +## Processing Lifecycle + +Each uploaded image moves through explicit statuses: +- upload +- queued +- processing +- transcribed +- failed +- completed + +Typical flow: +1. Image is uploaded and validated. +2. Image and job metadata are stored in PostgreSQL. +3. Job is queued and processed through LangGraph nodes. +4. LLM transcription is generated. +5. Result and processing events are saved. +6. Job ends as completed or failed with error details. + +This state-driven model enables reliable inspection, retries, and recovery. + +## Configuration And Observability + +Configuration is loaded once at startup using a pydantic-settings class and can be propagated through request/workflow execution via context variables where scoped access is needed. + +Logging is initialized once through a centralized logging.config call, and modules use named loggers for consistent observability across API, workflow, and persistence layers. + +Readiness checks validate both SQLAlchemy connectivity and graph runtime initialization so operational status reflects the actual owned runtime resources. + +## Why This Starter Exists + +This project intentionally balances practicality and extensibility: +- Minimal UI and straightforward APIs for fast iteration. +- Durable workflow state and clear job history for operational visibility. +- Clean separation of concerns across API, graph nodes, data models, and infrastructure. +- Local-first developer experience with uv and Docker Compose. + +It is suitable as a baseline for production systems that need better queueing, multi-worker scaling, richer auth, or additional document processing features. + +## Related Documentation + +- Project description: [docs/Historical_Document_Transcription.md](docs/Historical_Document_Transcription.md) +- Project build prompt: