Update docs

2026-06-23 10:49:11 -05:00
parent 2bb74cd1cc
commit 187324d903
5 changed files with 522 additions and 77 deletions
@@ -16,6 +16,13 @@ I have several thousand pages of family history told through letters, postcards,
 ### Verbatim vs. Clean Copy
 Transcriptions should be Verbatim and follow scholarly research guidelines, with no modifications to the original text.
 ### Prompt Curation Policy
 Transcription behavior should be implemented with prompt assets that are human-maintainable over time.
 1. Each transcription prompt is stored as an individual Markdown file.
 2. Prompt files are refined iteratively as document quality and edge cases are discovered.
 3. Prompt changes should be scoped to one prompt file at a time whenever possible to keep review history clear.
 ### Potential Document Issues
 | Document Issue | How to Handle It | Example |
 | :--- | :--- | :--- |
@@ -0,0 +1,277 @@
 # Architecture
 This document describes the production architecture of the personal historical-document transcription system. The system is intentionally optimized for single-user operation, low operational overhead, and clean internal boundaries that support future growth without rewrites.
 ## Architecture Objectives
 The production architecture is designed to:
 - preserve verbatim family-history source material as searchable text
 - keep operational complexity low for a personal deployment
 - support asynchronous transcription without requiring distributed infrastructure
 - maintain clear module boundaries so extensions can be added incrementally
 ## Production Scope And Scale
 The deployed system targets personal use and a corpus of several thousand documents processed over time. The architecture favors simple, composable building blocks over distributed orchestration.
 Current scope includes:
 - document upload and metadata capture
 - asynchronous transcription jobs
 - prompt-library driven transcription behavior, with one Markdown file per prompt
 - transcript review and revision history
 - full-text search over accepted transcripts
 - export of transcript data
 ## Deployment Topology
 The production deployment uses [Docker Compose](https://docs.docker.com/compose/) and treats containerized databases as extremely lightweight operational dependencies.
 Running [PostgreSQL](https://www.postgresql.org/docs/) in its own container is considered simple by default for this system.
 Running [MongoDB](https://www.mongodb.com/docs/) in its own container is also considered simple when document-centric storage is enabled.
 Container count is not a hard architectural limit; a three-container deployment (app, PostgreSQL, MongoDB) is an acceptable baseline.
 ### Baseline Topology (Two Containers)
 - one application container
 - one PostgreSQL container
 - embedded background worker execution inside the app process
 ### Expanded Topology (Three Containers)
 - application container
 - PostgreSQL container
 - MongoDB container
 No additional queue, scheduler, or search-engine containers are required in the baseline production setup.
 ## Runtime Architecture
 ```mermaid
 flowchart LR
    User[Browser User] --> App[FastAPI + NiceGUI Service]
    App --> Worker[In-process Background Worker]
    App --> PG[(PostgreSQL)]
    App --> MG[(MongoDB Document Store)]
    Worker --> AI[Transcription Provider]
    Worker --> PG
    Worker --> MG
 ```
 ## Layered Module Structure
 ### Interface Layer
 Responsibility:
 - HTTP API and UI routes
 - request/response validation
 - status and result presentation
 Out of scope:
 - business-rule enforcement
 - data-access implementation
 ### Application Layer
 Responsibility:
 - upload and job orchestration
 - state transitions and retry policy
 - coordination across domain and infrastructure ports
 Out of scope:
 - provider-specific protocol details
 - ORM or storage-specific logic
 ### Domain Layer
 Responsibility:
 - verbatim transcription policy
 - revision and provenance invariants
 - confidence and annotation semantics
 Out of scope:
 - web framework concerns
 - database and network I/O
 ### Infrastructure Layer
 Responsibility:
 - persistence adapters (PostgreSQL and MongoDB)
 - transcription-provider adapter
 Out of scope:
 - business policy decisions
 ## Processing Workflow
 Production transcription flow:
 1. A user uploads an image or PDF through the UI or API.
 2. The application validates payloads and creates document and job records.
 3. The in-process worker dequeues the job and calls the transcription provider.
 4. The application persists transcript output, confidence metadata, and provenance events.
 5. Job status transitions from queued to processing to completed or failed.
 6. The UI and API expose status, revision history, and searchable transcript text.
 ## Data Model Ownership
 System-of-record entities:
 - documents and pages
 - transcription jobs and status events
 - transcript revisions
 - provenance metadata
 Storage strategy:
 - PostgreSQL for relational system-of-record entities
 - MongoDB for document-oriented payloads and large transcription artifacts
 - versioned prompt artifacts stored as individual Markdown files for human editing and refinement
 - in-memory execution state treated as ephemeral
 ## Transcription Prompt Asset Policy
 The production system treats transcription prompts as maintainable content assets.
 - each transcription prompt is stored in its own Markdown file
 - prompt files are designed for direct human editing and iterative refinement
 - prompt updates are independent and do not require bundling unrelated prompt changes
 - prompt file identity and revision history are tracked through normal repository version control
 ## Simplicity Guardrails
 The production system enforces these constraints to prevent accidental over-engineering:
 - PostgreSQL in a container is treated as a lightweight default dependency
 - MongoDB in a container is treated as a lightweight optional dependency
 - three containers (app, PostgreSQL, MongoDB) is an acceptable simple deployment
 - no dedicated queue or search cluster is introduced without measured need
 - external infrastructure is added only behind existing ports/adapters
 ## Extension Path
 The architecture supports additive growth without changing domain contracts.
 ### Stage 1: Foundation (Current)
 - upload, transcription, review, search, export
 - in-process worker execution
 - single provider adapter
 - app plus PostgreSQL deployment
 ### Stage 2: Throughput Hardening
 - optional MongoDB document-store enablement
 - optional external worker/queue process
 - stronger retry and dead-letter handling
 ### Stage 3: Intelligence Features
 - entity extraction and cross-document linking
 - timeline and narrative assembly
 - optional multi-provider routing
 Each stage preserves existing module boundaries and keeps migration risk low.
 ## Test Strategy
 The test strategy is aligned to personal-scale operation with fast, deterministic feedback.
 ### Unit Tests
 - domain transcription rules and annotation behavior
 - revision-history invariants
 - job state-transition logic
 ### Integration Tests
 - repository behavior and transaction boundaries
 - persistence-adapter and provider adapter contract mapping
 - upload-to-persistence roundtrip
 ### End-to-End Tests
 - happy path: upload, transcribe, review, search, export
 - failure path: provider error, retry, surfaced failed status
 ### CI Execution Model
 - fast suite on each push
 - optional slower provider-sandbox checks on scheduled runs
 ## Risks And Controls
 ### Runtime Responsiveness
 Risk:
 - long jobs can reduce responsiveness in a single-process deployment
 Control:
 - bounded concurrency and visible job status in the UI
 ### Database Concurrency Limits
 Risk:
 - contention can appear under sustained concurrent writes in personal-scale infrastructure
 Control:
 - tuned connection pooling and phased use of MongoDB for document-heavy workloads
 ### Provider Output Variance
 Risk:
 - transcription quality varies by handwriting and image quality
 Control:
 - first-class human review and immutable revision history
 ## Technology References
 - [FastAPI documentation](https://fastapi.tiangolo.com/)
 - [NiceGUI documentation](https://nicegui.io/documentation)
 - [Docker Compose documentation](https://docs.docker.com/compose/)
 - [PostgreSQL documentation](https://www.postgresql.org/docs/)
 - [MongoDB documentation](https://www.mongodb.com/docs/)
 ## Related Pages
 - [System overview](index.md)
 - [Testing guide](tests.md)
 ## Glossary
 - Adapter: A component that translates between internal interfaces and external systems such as databases or AI services.
 - Background job: Work executed outside the request/response path so the UI remains responsive.
 - Boundary: A strict separation between modules with different responsibilities.
 - CI (Continuous Integration): Automated test execution for code changes.
 - Contract test: A test that verifies an adapter follows expected input/output behavior at a boundary.
 - Domain layer: The module that contains core business rules and invariants.
 - End-to-end test: A test that validates a full user flow across the running system.
 - Full-text search: Text indexing and querying optimized for natural-language search.
 - In-process worker: A background executor that runs within the same application process.
 - Integration test: A test that verifies interactions between real modules and infrastructure components.
 - MongoDB: A document-oriented database used for flexible, high-variance data structures.
 - Modular monolith: A single deployable application with strongly separated internal modules.
 - Port/Interface: A stable contract used by application/domain code to call infrastructure implementations.
 - Prompt artifact: A single Markdown file that defines one transcription prompt and can be revised independently.
 - Provenance: Metadata that records where generated data came from and how it was produced.
 - Revision history: Versioned record of transcript edits over time.
 - System of record: The authoritative persistent store for canonical data.
 - Vertical slice: A minimal end-to-end feature path spanning UI/API, application logic, and persistence.
@@ -1,95 +1,51 @@
 ## Handwriting Transcription System
-This project is a starter for transcribing historical documents with an LLM-powered, graph-based backend. It combines a NiceGUI web interface, a FastAPI + LangGraph application, and PostgreSQL persistence behind an Nginx entrypoint.
+This project is a production application for transcribing and preserving historical family documents. It is intentionally designed for personal-scale use, with a simplicity-first architecture that is easy to operate and easy to extend.
-The system is designed to be easy to run locally with Docker Compose and easy to extend for more advanced orchestration, scaling, and model strategies.
+## Start Here
-## What The System Does
+Read [architecture.md](architecture.md) first.
-At a high level, users upload one or more document images from the web UI. Each upload is tracked as a job in the database, processed through a LangGraph workflow, transcribed by an LLM, and stored with status history and events.
+The architecture page is the primary technical reference and defines:
-Core outcomes:
+- deployed topology and infrastructure limits
- Upload handwritten images through a minimal web interface.
+- module boundaries and dependency flow
- Persist image data, job state, events, errors, and transcription text in PostgreSQL.
+- processing life cycle and data ownership
- Track lifecycle states from upload through completion or failure.
+- test strategy, risk controls, and extension path
 - Inspect job status and results through API endpoints and UI pages.
-## Architecture Overview
+## What The Application Does
-The project uses a single Python backend that serves both API endpoints and NiceGUI pages.
+At a high level, users upload images of handwritten, typed, or typeset documents, run asynchronous transcription jobs, review and edit transcript revisions, and search across accepted text.
-### Front End
+Core capabilities:
 - NiceGUI pages mounted on FastAPI with periodic refresh for live job status updates.
 - Supports uploading images, viewing current job states, and reading completed transcriptions.
 - Displays failure states and error messages for troubleshooting.
-### Backend
+- document upload and metadata capture
- FastAPI app for HTTP endpoints and page rendering.
+- asynchronous transcription with visible job status
- LangGraph workflow for multi-step transcription execution.
+- transcription prompt management with one Markdown file per prompt for human refinement over time
- Background task execution for asynchronous processing after upload.
+- revision history for transcript edits
- Centralized app configuration through pydantic-settings.
+- full-text search over accepted transcripts
- Centralized logging initialization via one logging.config setup call at startup.
+- export of transcript data
 - Lifespan-owned runtime resources for the SQLAlchemy engine, async session factory, PostgreSQL checkpoint connection, and compiled graph.
-### Database
+## Production Operating Model
 - PostgreSQL is the only persistent store.
 - SQLModel defines schema and data access.
 - SQLAlchemy async access uses one engine per process and one async session per request or concurrent background task.
 - Persists:
 	- image records
 	- transcription jobs
 	- processing events/history
 	- transcription output
 	- error details
 - LangGraph checkpointing is stored in PostgreSQL for resumable workflow state.
 - Schema bootstrap is explicit and opt-in; normal startup does not mutate production schema automatically.
-### Infrastructure
+The system runs with minimal operational overhead:
 - Docker Compose runs exactly three containers:
 	- backend (FastAPI + LangGraph)
 	- frontend (Nginx reverse proxy)
 	- db (PostgreSQL)
 - Nginx acts as the public entrypoint and proxies requests to the backend.
-## Processing Lifecycle
+- PostgreSQL in a dedicated Docker container is considered extremely lightweight and simple for this system
 - MongoDB in a dedicated Docker container is also considered extremely lightweight and simple for document-centric persistence
 - a three-container deployment (app, PostgreSQL, MongoDB) is a simple and acceptable baseline
 - no required queue or search-engine containers in the baseline setup
-Each uploaded image moves through explicit statuses:
+This operating model keeps deployment and maintenance simple while preserving clean boundaries for future scale.
 - upload
 - queued
 - processing
 - transcribed
 - failed
 - completed
-Typical flow:
+## Documentation Map
 1. Image is uploaded and validated.
 2. Image and job metadata are stored in PostgreSQL.
 3. Job is queued and processed through LangGraph nodes.
 4. LLM transcription is generated.
 5. Result and processing events are saved.
 6. Job ends as completed or failed with error details.
-This state-driven model enables reliable inspection, retries, and recovery.
+- Architecture and technical design: [architecture.md](architecture.md)
 - Testing strategy and guidance: [tests.md](tests.md)
 - Runtime and deployment requirements: [requirements.md](requirements.md)
 - Domain context and transcription policy: [intent.md](intent.md)
-## Configuration And Observability
+## Glossary
-Configuration is loaded once at startup using a pydantic-settings class and can be propagated through request/workflow execution via context variables where scoped access is needed.
+- Document-oriented persistence: Storing data as flexible records instead of fixed relational rows.
-
+- Prompt artifact: A single Markdown file that defines one transcription prompt and is edited independently.
-Logging is initialized once through a centralized logging.config call, and modules use named loggers for consistent observability across API, workflow, and persistence layers.
+- System of record: The authoritative persistent store for canonical data.
 Readiness checks validate both SQLAlchemy connectivity and graph runtime initialization so operational status reflects the actual owned runtime resources.
 ## Why This Starter Exists
 This project intentionally balances practicality and extensibility:
 - Minimal UI and straightforward APIs for fast iteration.
 - Durable workflow state and clear job history for operational visibility.
 - Clean separation of concerns across API, graph nodes, data models, and infrastructure.
 - Local-first developer experience with uv and Docker Compose.
 It is suitable as a baseline for production systems that need better queueing, multi-worker scaling, richer auth, or additional document processing features.
 ## Related Documentation
 - Project description: [docs/Historical_Document_Transcription.md](docs/Historical_Document_Transcription.md)
 - Project build prompt: 
@@ -0,0 +1,84 @@
 ## Handwriting Transcription System Requirements
 This page captures a SysML v1.6-style requirements baseline for the production system described in [index.md](index.md). The model is represented as concise tables and traceability lists that preserve SysML-style IDs and relationship semantics.
 ## Scope
 - System of interest: the single Python application service (NiceGUI + FastAPI) with PostgreSQL as the relational system of record and optional MongoDB for document-oriented persistence.
 - Operational context: local-first execution with Docker Compose and an intentionally lightweight production trajectory.
 - Primary concern: end-to-end transcription job lifecycle from upload through completion or failure.
 ## Requirements Model (Concise Text Form)
 ### Requirements
 | ID | Category | Requirement | Risk | Verify Method |
 | --- | --- | --- | --- | --- |
 | REQ-0 | System | Provide end-to-end handwriting transcription with persistent, inspectable lifecycle state. | medium | demonstration |
 | REQ-1 | Functional | Allow users to upload one or more handwriting images from the web UI. | low | test |
 | REQ-2 | Functional | Run each upload through asynchronous processing that returns a transcription or explicit failure. | high | test |
 | REQ-3 | Functional | Persist and expose job states: upload, queued, processing, transcribed, failed, completed. | high | inspection |
 | REQ-4 | Functional | Persist transcription output, processing history, and failure details. | medium | test |
 | REQ-5 | Interface | Expose API and UI views for status inspection and completed transcription reading. | medium | demonstration |
 | REQ-6 | Performance | Trigger background processing on upload to preserve UI responsiveness. | medium | analysis |
 | REQ-7 | Design Constraint | Keep lifespan-owned runtime resources: SQLAlchemy engine, async session factory, worker resources, provider clients. | medium | inspection |
 | REQ-8 | Design Constraint | Initialize configuration and logging once at startup through centralized mechanisms. | low | inspection |
 | REQ-9 | Design Constraint | Use Docker Compose baseline of app plus PostgreSQL; allow optional MongoDB container when enabled. | medium | demonstration |
 | REQ-10 | Design Constraint | Keep schema bootstrap explicit and opt-in; normal startup does not mutate production schema. | high | inspection |
 | REQ-11 | Design Constraint | Use service-backed persistence for core document and job data. | medium | inspection |
 | REQ-12 | Design Constraint | Store transcription prompts as individual Markdown artifacts for iterative refinement. | medium | inspection |
 ### Requirement Relationships
 - Contains: REQ-0 contains REQ-1 through REQ-12.
 - Derives: REQ-2 -> REQ-3, REQ-3 -> REQ-4.
 - Traces: REQ-5 -> REQ-3.
 - Refines: REQ-6 -> REQ-2.
 ### Architecture Elements
 | Element | Type | Doc Reference |
 | --- | --- | --- |
 | UI | NiceGUI pages | src/handwriting/ui/pages |
 | API | FastAPI routes | src/handwriting/api/routes.py |
 | GRAPH | Async processing workflow | src/handwriting/services, src/handwriting/ai |
 | DBREL | PostgreSQL + SQLModel relational persistence | src/handwriting/db |
 | DBDOC | MongoDB document persistence | src/handwriting/db, src/handwriting/services |
 | OPS | Docker Compose runtime | docker-compose.yml |
 | PROMPTS | Transcription prompt artifact library (Markdown files) | .github/prompts, docs |
 | TESTS | Pytest verification suite | tests |
 ### Satisfaction Mapping
 - UI satisfies REQ-1, REQ-5.
 - API satisfies REQ-5.
 - GRAPH satisfies REQ-2, REQ-6.
 - DBREL satisfies REQ-3, REQ-10.
 - DBDOC satisfies REQ-4, REQ-11.
 - OPS satisfies REQ-9.
 - PROMPTS satisfies REQ-12.
 ### Verification Mapping
 - TESTS verifies REQ-1, REQ-2, REQ-3, REQ-4, REQ-5, REQ-10, REQ-11, REQ-12.
 ## Requirement Notes
 - Requirement IDs (`REQ-*`) are stable references for planning, implementation, and test traceability.
 - The model uses compact tables and traceability lists for renderer compatibility while preserving SysML-style requirement IDs and relationship semantics.
 - Requirement categories (functional, interface, performance, and design constraints) are preserved as explicit REQ entries and relationship labels to keep change impact visible.
 - PostgreSQL containerization and optional MongoDB containerization are both treated as extremely lightweight and simple operational choices in this architecture.
 ## Verification Intent
 - Demonstration: validate end-to-end behavior via running system flows and operator-visible outcomes.
 - Inspection: verify architecture and startup/runtime policies in code and configuration.
 - Analysis: evaluate asynchronous execution behavior and design sufficiency.
 - Test: automate behavioral checks through pytest suites and service-level tests.
 ## Glossary
 - Document-oriented persistence: A storage approach that uses flexible document structures for variable data shapes.
 - Prompt artifact: A single Markdown file that defines one transcription prompt and is revised independently.
 - SysML: Systems Modeling Language used to express structured requirements and traceability.
 - System of record: The authoritative persistent store for canonical business data.
@@ -0,0 +1,121 @@
 # Testing
 The greenfield test structure mirrors the source tree at a high level and keeps the first pass shallow and easy to extend.
 ## Source To Test Map
 - `src/handwriting/bootstrap.py`, `src/handwriting/config.py`, `src/handwriting/logging_buffer.py`, `src/handwriting/main.py` -> `tests/handwriting/test_core.py`
 - `src/handwriting/ai/**` -> `tests/handwriting/ai/`
 - `src/handwriting/api/**` -> `tests/handwriting/api/`
 - `src/handwriting/db/**` -> `tests/handwriting/db/`
 - `src/handwriting/services/**` -> `tests/handwriting/services/`
 - `src/handwriting/ui/components/**` and `src/handwriting/ui/pages/**` -> `tests/handwriting/ui/`
 - Shared test helpers and fixtures -> `tests/conftest.py` plus subtree `conftest.py` files where needed
 ## Major Sections
 - `tests/handwriting/test_core.py`: bootstrap, config, logging, and entrypoint coverage.
 - `tests/handwriting/ai/`: AI contracts, runtime orchestration, graph wiring, and node behavior.
 - `tests/handwriting/api/`: HTTP routes and request/response contract checks.
 - `tests/handwriting/db/`: session setup, models, and repositories.
 - `tests/handwriting/services/`: service-layer orchestration and domain logic.
 - `tests/handwriting/ui/`: NiceGUI components and pages.
 - `tests/conftest.py`: shared lightweight fixtures and deterministic defaults.
 - `tests/handwriting/**/conftest.py`: subtree-specific fixtures only where a package needs its own setup.
 ## Implemented In This Pass (Core + DB)
 ### Created Files
 - `tests/handwriting/test_core.py`
 - `tests/handwriting/conftest.py`
 - `tests/handwriting/db/conftest.py`
 - `tests/handwriting/db/test_session.py`
 - `tests/handwriting/db/test_models.py`
 - `tests/handwriting/db/test_job_repo.py`
 - `tests/handwriting/db/test_service_job.py`
 ### Updated Files
 - `tests/conftest.py`: added shared `settings_factory` fixture.
 - `pyproject.toml`: registered strict markers and async pytest mode.
 ### Marker Taxonomy
 - `unit`: fast deterministic unit tests.
 - `db`: database-backed tests against PostgreSQL-backed fixtures.
 - `document`: document-store tests for MongoDB-backed persistence behavior.
 - `integration`: cross-layer integration tests (reserved for expanded next pass).
 - `smoke`: high-value end-to-end surface checks.
 - `slow`: long-running tests.
 - `external`: tests that require external services/credentials.
 ### Fixture Ownership
 - `tests/conftest.py`: global lightweight fixtures used across all sections.
 - `tests/handwriting/conftest.py`: core package-level environment fixtures.
 - `tests/handwriting/db/conftest.py`:
 	- PostgreSQL fixtures (`postgres_engine`, `db_session`) for default data-layer tests.
 	- optional MongoDB fixture (`mongo_client`) gated by `PYTEST_MONGO_URL` for document-store integration checks.
 ### Initial Coverage Added
 - `tests/handwriting/test_core.py`
 	- settings identity and URL validation behavior.
 	- logging UI buffer wiring.
 	- buffer incremental read behavior.
 	- minimal main module export smoke check.
 - `tests/handwriting/db/test_session.py`
 	- session scope, factory, and engine helper behavior.
 - `tests/handwriting/db/test_models.py`
 	- model defaults and enum/value shape checks.
 - `tests/handwriting/db/test_job_repo.py`
 	- repository list/detail happy path and not-found/missing-image edges.
 - `tests/handwriting/db/test_service_job.py`
 	- job creation happy path and validation errors for unsupported type / oversized payload.
 ## Run Commands
 - Collect only: `uv run pytest --collect-only -q`
 - Fast local path: `uv run pytest -m unit -q`
 - DB path: `uv run pytest -m "db or integration" -q`
 - Full path: `uv run pytest -q`
 ## Current Verification Snapshot
 - `uv run pytest --collect-only -q` -> 19 tests collected.
 - `uv run pytest -m unit -q` -> 11 passed, 8 deselected.
 - `uv run pytest -m "db or integration" -q` -> 7 passed, 12 deselected.
 ## Jobs Page Diagnostics
 To investigate cases where `/jobs` renders without a table, the scaffold now includes focused service and UI coverage:
 - `tests/handwriting/services/test_ui_jobs.py`
 	- verifies the `list_jobs_for_ui` contract used by the page.
 	- includes the join edge case where a job with a missing image relation does not appear in list output.
 - `tests/handwriting/ui/test_jobs_page_runtime.py`
 	- verifies runtime guard behavior for missing app state (`settings`, `session_factory`).
 - `tests/handwriting/ui/test_jobs_page_rendering.py`
 	- includes a detector test for initial render behavior on `/jobs`.
 	- verifies empty, table, and error rendering branches after refresh.
 - `tests/handwriting/ui/test_jobs_page_smoke.py`
 	- confirms route registration and refresh button wiring.
 Targeted commands:
 - `uv run pytest tests/handwriting/ui/test_jobs_page_rendering.py -q`
 - `uv run pytest -m "unit or integration" -q`
 - `uv run pytest -m smoke -k jobs -q`
 - `HANDWRITING_E2E_BASE_URL=http://localhost uv run pytest tests/handwriting/ui/test_jobs_page_browser_smoke.py -q`
 ## Next Pass
 The next pass can expand each major section into markers, fixtures, and test case placeholders.
 ## Glossary
 - Contract mapping: Verifying that adapter input/output shapes match expected boundaries.
 - Document-store tests: Tests that validate behavior against document-oriented persistence components.
 - Marker taxonomy: The test marker classification scheme used to select test slices.