generated from john/python-template
Update docs
This commit is contained in:
+33
-77
@@ -1,95 +1,51 @@
|
||||
## Handwriting Transcription System
|
||||
|
||||
This project is a starter for transcribing historical documents with an LLM-powered, graph-based backend. It combines a NiceGUI web interface, a FastAPI + LangGraph application, and PostgreSQL persistence behind an Nginx entrypoint.
|
||||
This project is a production application for transcribing and preserving historical family documents. It is intentionally designed for personal-scale use, with a simplicity-first architecture that is easy to operate and easy to extend.
|
||||
|
||||
The system is designed to be easy to run locally with Docker Compose and easy to extend for more advanced orchestration, scaling, and model strategies.
|
||||
## Start Here
|
||||
|
||||
## What The System Does
|
||||
Read [architecture.md](architecture.md) first.
|
||||
|
||||
At a high level, users upload one or more document images from the web UI. Each upload is tracked as a job in the database, processed through a LangGraph workflow, transcribed by an LLM, and stored with status history and events.
|
||||
The architecture page is the primary technical reference and defines:
|
||||
|
||||
Core outcomes:
|
||||
- Upload handwritten images through a minimal web interface.
|
||||
- Persist image data, job state, events, errors, and transcription text in PostgreSQL.
|
||||
- Track lifecycle states from upload through completion or failure.
|
||||
- Inspect job status and results through API endpoints and UI pages.
|
||||
- deployed topology and infrastructure limits
|
||||
- module boundaries and dependency flow
|
||||
- processing life cycle and data ownership
|
||||
- test strategy, risk controls, and extension path
|
||||
|
||||
## Architecture Overview
|
||||
## What The Application Does
|
||||
|
||||
The project uses a single Python backend that serves both API endpoints and NiceGUI pages.
|
||||
At a high level, users upload images of handwritten, typed, or typeset documents, run asynchronous transcription jobs, review and edit transcript revisions, and search across accepted text.
|
||||
|
||||
### Front End
|
||||
- NiceGUI pages mounted on FastAPI with periodic refresh for live job status updates.
|
||||
- Supports uploading images, viewing current job states, and reading completed transcriptions.
|
||||
- Displays failure states and error messages for troubleshooting.
|
||||
Core capabilities:
|
||||
|
||||
### Backend
|
||||
- FastAPI app for HTTP endpoints and page rendering.
|
||||
- LangGraph workflow for multi-step transcription execution.
|
||||
- Background task execution for asynchronous processing after upload.
|
||||
- Centralized app configuration through pydantic-settings.
|
||||
- Centralized logging initialization via one logging.config setup call at startup.
|
||||
- Lifespan-owned runtime resources for the SQLAlchemy engine, async session factory, PostgreSQL checkpoint connection, and compiled graph.
|
||||
- document upload and metadata capture
|
||||
- asynchronous transcription with visible job status
|
||||
- transcription prompt management with one Markdown file per prompt for human refinement over time
|
||||
- revision history for transcript edits
|
||||
- full-text search over accepted transcripts
|
||||
- export of transcript data
|
||||
|
||||
### Database
|
||||
- PostgreSQL is the only persistent store.
|
||||
- SQLModel defines schema and data access.
|
||||
- SQLAlchemy async access uses one engine per process and one async session per request or concurrent background task.
|
||||
- Persists:
|
||||
- image records
|
||||
- transcription jobs
|
||||
- processing events/history
|
||||
- transcription output
|
||||
- error details
|
||||
- LangGraph checkpointing is stored in PostgreSQL for resumable workflow state.
|
||||
- Schema bootstrap is explicit and opt-in; normal startup does not mutate production schema automatically.
|
||||
## Production Operating Model
|
||||
|
||||
### Infrastructure
|
||||
- Docker Compose runs exactly three containers:
|
||||
- backend (FastAPI + LangGraph)
|
||||
- frontend (Nginx reverse proxy)
|
||||
- db (PostgreSQL)
|
||||
- Nginx acts as the public entrypoint and proxies requests to the backend.
|
||||
The system runs with minimal operational overhead:
|
||||
|
||||
## Processing Lifecycle
|
||||
- PostgreSQL in a dedicated Docker container is considered extremely lightweight and simple for this system
|
||||
- MongoDB in a dedicated Docker container is also considered extremely lightweight and simple for document-centric persistence
|
||||
- a three-container deployment (app, PostgreSQL, MongoDB) is a simple and acceptable baseline
|
||||
- no required queue or search-engine containers in the baseline setup
|
||||
|
||||
Each uploaded image moves through explicit statuses:
|
||||
- upload
|
||||
- queued
|
||||
- processing
|
||||
- transcribed
|
||||
- failed
|
||||
- completed
|
||||
This operating model keeps deployment and maintenance simple while preserving clean boundaries for future scale.
|
||||
|
||||
Typical flow:
|
||||
1. Image is uploaded and validated.
|
||||
2. Image and job metadata are stored in PostgreSQL.
|
||||
3. Job is queued and processed through LangGraph nodes.
|
||||
4. LLM transcription is generated.
|
||||
5. Result and processing events are saved.
|
||||
6. Job ends as completed or failed with error details.
|
||||
## Documentation Map
|
||||
|
||||
This state-driven model enables reliable inspection, retries, and recovery.
|
||||
- Architecture and technical design: [architecture.md](architecture.md)
|
||||
- Testing strategy and guidance: [tests.md](tests.md)
|
||||
- Runtime and deployment requirements: [requirements.md](requirements.md)
|
||||
- Domain context and transcription policy: [intent.md](intent.md)
|
||||
|
||||
## Configuration And Observability
|
||||
## Glossary
|
||||
|
||||
Configuration is loaded once at startup using a pydantic-settings class and can be propagated through request/workflow execution via context variables where scoped access is needed.
|
||||
|
||||
Logging is initialized once through a centralized logging.config call, and modules use named loggers for consistent observability across API, workflow, and persistence layers.
|
||||
|
||||
Readiness checks validate both SQLAlchemy connectivity and graph runtime initialization so operational status reflects the actual owned runtime resources.
|
||||
|
||||
## Why This Starter Exists
|
||||
|
||||
This project intentionally balances practicality and extensibility:
|
||||
- Minimal UI and straightforward APIs for fast iteration.
|
||||
- Durable workflow state and clear job history for operational visibility.
|
||||
- Clean separation of concerns across API, graph nodes, data models, and infrastructure.
|
||||
- Local-first developer experience with uv and Docker Compose.
|
||||
|
||||
It is suitable as a baseline for production systems that need better queueing, multi-worker scaling, richer auth, or additional document processing features.
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- Project description: [docs/Historical_Document_Transcription.md](docs/Historical_Document_Transcription.md)
|
||||
- Project build prompt:
|
||||
- Document-oriented persistence: Storing data as flexible records instead of fixed relational rows.
|
||||
- Prompt artifact: A single Markdown file that defines one transcription prompt and is edited independently.
|
||||
- System of record: The authoritative persistent store for canonical data.
|
||||
|
||||
Reference in New Issue
Block a user