Update docs

docs/architecture.md

@@ -0,0 +1,277 @@
+# Architecture
+
+This document describes the production architecture of the personal historical-document transcription system. The system is intentionally optimized for single-user operation, low operational overhead, and clean internal boundaries that support future growth without rewrites.
+
+## Architecture Objectives
+
+The production architecture is designed to:
+
+- preserve verbatim family-history source material as searchable text
+- keep operational complexity low for a personal deployment
+- support asynchronous transcription without requiring distributed infrastructure
+- maintain clear module boundaries so extensions can be added incrementally
+
+## Production Scope And Scale
+
+The deployed system targets personal use and a corpus of several thousand documents processed over time. The architecture favors simple, composable building blocks over distributed orchestration.
+
+Current scope includes:
+
+- document upload and metadata capture
+- asynchronous transcription jobs
+- prompt-library driven transcription behavior, with one Markdown file per prompt
+- transcript review and revision history
+- full-text search over accepted transcripts
+- export of transcript data
+
+## Deployment Topology
+
+The production deployment uses [Docker Compose](https://docs.docker.com/compose/) and treats containerized databases as extremely lightweight operational dependencies.
+
+Running [PostgreSQL](https://www.postgresql.org/docs/) in its own container is considered simple by default for this system.
+
+Running [MongoDB](https://www.mongodb.com/docs/) in its own container is also considered simple when document-centric storage is enabled.
+
+Container count is not a hard architectural limit; a three-container deployment (app, PostgreSQL, MongoDB) is an acceptable baseline.
+
+### Baseline Topology (Two Containers)
+
+- one application container
+- one PostgreSQL container
+- embedded background worker execution inside the app process
+
+### Expanded Topology (Three Containers)
+
+- application container
+- PostgreSQL container
+- MongoDB container
+
+No additional queue, scheduler, or search-engine containers are required in the baseline production setup.
+
+## Runtime Architecture
+
+```mermaid
+flowchart LR
+    User[Browser User] --> App[FastAPI + NiceGUI Service]
+    App --> Worker[In-process Background Worker]
+    App --> PG[(PostgreSQL)]
+    App --> MG[(MongoDB Document Store)]
+    Worker --> AI[Transcription Provider]
+    Worker --> PG
+    Worker --> MG
+```
+
+## Layered Module Structure
+
+### Interface Layer
+
+Responsibility:
+
+- HTTP API and UI routes
+- request/response validation
+- status and result presentation
+
+Out of scope:
+
+- business-rule enforcement
+- data-access implementation
+
+### Application Layer
+
+Responsibility:
+
+- upload and job orchestration
+- state transitions and retry policy
+- coordination across domain and infrastructure ports
+
+Out of scope:
+
+- provider-specific protocol details
+- ORM or storage-specific logic
+
+### Domain Layer
+
+Responsibility:
+
+- verbatim transcription policy
+- revision and provenance invariants
+- confidence and annotation semantics
+
+Out of scope:
+
+- web framework concerns
+- database and network I/O
+
+### Infrastructure Layer
+
+Responsibility:
+
+- persistence adapters (PostgreSQL and MongoDB)
+- transcription-provider adapter
+
+Out of scope:
+
+- business policy decisions
+
+## Processing Workflow
+
+Production transcription flow:
+
+1. A user uploads an image or PDF through the UI or API.
+2. The application validates payloads and creates document and job records.
+3. The in-process worker dequeues the job and calls the transcription provider.
+4. The application persists transcript output, confidence metadata, and provenance events.
+5. Job status transitions from queued to processing to completed or failed.
+6. The UI and API expose status, revision history, and searchable transcript text.
+
+## Data Model Ownership
+
+System-of-record entities:
+
+- documents and pages
+- transcription jobs and status events
+- transcript revisions
+- provenance metadata
+
+Storage strategy:
+
+- PostgreSQL for relational system-of-record entities
+- MongoDB for document-oriented payloads and large transcription artifacts
+- versioned prompt artifacts stored as individual Markdown files for human editing and refinement
+- in-memory execution state treated as ephemeral
+
+## Transcription Prompt Asset Policy
+
+The production system treats transcription prompts as maintainable content assets.
+
+- each transcription prompt is stored in its own Markdown file
+- prompt files are designed for direct human editing and iterative refinement
+- prompt updates are independent and do not require bundling unrelated prompt changes
+- prompt file identity and revision history are tracked through normal repository version control
+
+## Simplicity Guardrails
+
+The production system enforces these constraints to prevent accidental over-engineering:
+
+- PostgreSQL in a container is treated as a lightweight default dependency
+- MongoDB in a container is treated as a lightweight optional dependency
+- three containers (app, PostgreSQL, MongoDB) is an acceptable simple deployment
+- no dedicated queue or search cluster is introduced without measured need
+- external infrastructure is added only behind existing ports/adapters
+
+## Extension Path
+
+The architecture supports additive growth without changing domain contracts.
+
+### Stage 1: Foundation (Current)
+
+- upload, transcription, review, search, export
+- in-process worker execution
+- single provider adapter
+- app plus PostgreSQL deployment
+
+### Stage 2: Throughput Hardening
+
+- optional MongoDB document-store enablement
+- optional external worker/queue process
+- stronger retry and dead-letter handling
+
+### Stage 3: Intelligence Features
+
+- entity extraction and cross-document linking
+- timeline and narrative assembly
+- optional multi-provider routing
+
+Each stage preserves existing module boundaries and keeps migration risk low.
+
+## Test Strategy
+
+The test strategy is aligned to personal-scale operation with fast, deterministic feedback.
+
+### Unit Tests
+
+- domain transcription rules and annotation behavior
+- revision-history invariants
+- job state-transition logic
+
+### Integration Tests
+
+- repository behavior and transaction boundaries
+- persistence-adapter and provider adapter contract mapping
+- upload-to-persistence roundtrip
+
+### End-to-End Tests
+
+- happy path: upload, transcribe, review, search, export
+- failure path: provider error, retry, surfaced failed status
+
+### CI Execution Model
+
+- fast suite on each push
+- optional slower provider-sandbox checks on scheduled runs
+
+## Risks And Controls
+
+### Runtime Responsiveness
+
+Risk:
+
+- long jobs can reduce responsiveness in a single-process deployment
+
+Control:
+
+- bounded concurrency and visible job status in the UI
+
+### Database Concurrency Limits
+
+Risk:
+
+- contention can appear under sustained concurrent writes in personal-scale infrastructure
+
+Control:
+
+- tuned connection pooling and phased use of MongoDB for document-heavy workloads
+
+### Provider Output Variance
+
+Risk:
+
+- transcription quality varies by handwriting and image quality
+
+Control:
+
+- first-class human review and immutable revision history
+
+## Technology References
+
+- [FastAPI documentation](https://fastapi.tiangolo.com/)
+- [NiceGUI documentation](https://nicegui.io/documentation)
+- [Docker Compose documentation](https://docs.docker.com/compose/)
+- [PostgreSQL documentation](https://www.postgresql.org/docs/)
+- [MongoDB documentation](https://www.mongodb.com/docs/)
+
+## Related Pages
+
+- [System overview](index.md)
+- [Testing guide](tests.md)
+
+## Glossary
+
+- Adapter: A component that translates between internal interfaces and external systems such as databases or AI services.
+- Background job: Work executed outside the request/response path so the UI remains responsive.
+- Boundary: A strict separation between modules with different responsibilities.
+- CI (Continuous Integration): Automated test execution for code changes.
+- Contract test: A test that verifies an adapter follows expected input/output behavior at a boundary.
+- Domain layer: The module that contains core business rules and invariants.
+- End-to-end test: A test that validates a full user flow across the running system.
+- Full-text search: Text indexing and querying optimized for natural-language search.
+- In-process worker: A background executor that runs within the same application process.
+- Integration test: A test that verifies interactions between real modules and infrastructure components.
+- MongoDB: A document-oriented database used for flexible, high-variance data structures.
+- Modular monolith: A single deployable application with strongly separated internal modules.
+- Port/Interface: A stable contract used by application/domain code to call infrastructure implementations.
+- Prompt artifact: A single Markdown file that defines one transcription prompt and can be revised independently.
+- Provenance: Metadata that records where generated data came from and how it was produced.
+- Revision history: Versioned record of transcript edits over time.
+- System of record: The authoritative persistent store for canonical data.
+- Vertical slice: A minimal end-to-end feature path spanning UI/API, application logic, and persistence.