generated from john/python-template
96 lines
4.3 KiB
Markdown
96 lines
4.3 KiB
Markdown
## Handwriting Transcription System
|
|
|
|
This project is a starter for transcribing historical documents with an LLM-powered, graph-based backend. It combines a NiceGUI web interface, a FastAPI + LangGraph application, and PostgreSQL persistence behind an Nginx entrypoint.
|
|
|
|
The system is designed to be easy to run locally with Docker Compose and easy to extend for more advanced orchestration, scaling, and model strategies.
|
|
|
|
## What The System Does
|
|
|
|
At a high level, users upload one or more document images from the web UI. Each upload is tracked as a job in the database, processed through a LangGraph workflow, transcribed by an LLM, and stored with status history and events.
|
|
|
|
Core outcomes:
|
|
- Upload handwritten images through a minimal web interface.
|
|
- Persist image data, job state, events, errors, and transcription text in PostgreSQL.
|
|
- Track lifecycle states from upload through completion or failure.
|
|
- Inspect job status and results through API endpoints and UI pages.
|
|
|
|
## Architecture Overview
|
|
|
|
The project uses a single Python backend that serves both API endpoints and NiceGUI pages.
|
|
|
|
### Front End
|
|
- NiceGUI pages mounted on FastAPI with periodic refresh for live job status updates.
|
|
- Supports uploading images, viewing current job states, and reading completed transcriptions.
|
|
- Displays failure states and error messages for troubleshooting.
|
|
|
|
### Backend
|
|
- FastAPI app for HTTP endpoints and page rendering.
|
|
- LangGraph workflow for multi-step transcription execution.
|
|
- Background task execution for asynchronous processing after upload.
|
|
- Centralized app configuration through pydantic-settings.
|
|
- Centralized logging initialization via one logging.config setup call at startup.
|
|
- Lifespan-owned runtime resources for the SQLAlchemy engine, async session factory, PostgreSQL checkpoint connection, and compiled graph.
|
|
|
|
### Database
|
|
- PostgreSQL is the only persistent store.
|
|
- SQLModel defines schema and data access.
|
|
- SQLAlchemy async access uses one engine per process and one async session per request or concurrent background task.
|
|
- Persists:
|
|
- image records
|
|
- transcription jobs
|
|
- processing events/history
|
|
- transcription output
|
|
- error details
|
|
- LangGraph checkpointing is stored in PostgreSQL for resumable workflow state.
|
|
- Schema bootstrap is explicit and opt-in; normal startup does not mutate production schema automatically.
|
|
|
|
### Infrastructure
|
|
- Docker Compose runs exactly three containers:
|
|
- backend (FastAPI + LangGraph)
|
|
- frontend (Nginx reverse proxy)
|
|
- db (PostgreSQL)
|
|
- Nginx acts as the public entrypoint and proxies requests to the backend.
|
|
|
|
## Processing Lifecycle
|
|
|
|
Each uploaded image moves through explicit statuses:
|
|
- upload
|
|
- queued
|
|
- processing
|
|
- transcribed
|
|
- failed
|
|
- completed
|
|
|
|
Typical flow:
|
|
1. Image is uploaded and validated.
|
|
2. Image and job metadata are stored in PostgreSQL.
|
|
3. Job is queued and processed through LangGraph nodes.
|
|
4. LLM transcription is generated.
|
|
5. Result and processing events are saved.
|
|
6. Job ends as completed or failed with error details.
|
|
|
|
This state-driven model enables reliable inspection, retries, and recovery.
|
|
|
|
## Configuration And Observability
|
|
|
|
Configuration is loaded once at startup using a pydantic-settings class and can be propagated through request/workflow execution via context variables where scoped access is needed.
|
|
|
|
Logging is initialized once through a centralized logging.config call, and modules use named loggers for consistent observability across API, workflow, and persistence layers.
|
|
|
|
Readiness checks validate both SQLAlchemy connectivity and graph runtime initialization so operational status reflects the actual owned runtime resources.
|
|
|
|
## Why This Starter Exists
|
|
|
|
This project intentionally balances practicality and extensibility:
|
|
- Minimal UI and straightforward APIs for fast iteration.
|
|
- Durable workflow state and clear job history for operational visibility.
|
|
- Clean separation of concerns across API, graph nodes, data models, and infrastructure.
|
|
- Local-first developer experience with uv and Docker Compose.
|
|
|
|
It is suitable as a baseline for production systems that need better queueing, multi-worker scaling, richer auth, or additional document processing features.
|
|
|
|
## Related Documentation
|
|
|
|
- Project description: [docs/Historical_Document_Transcription.md](docs/Historical_Document_Transcription.md)
|
|
- Project build prompt:
|