generated from john/python-template
300 lines
10 KiB
Markdown
300 lines
10 KiB
Markdown
# Architecture
|
|
|
|
This document describes the production architecture of the personal historical-document transcription system. The system is intentionally optimized for single-user operation, low operational overhead, and clean internal boundaries that support future growth without rewrites.
|
|
|
|
## Architecture Objectives
|
|
|
|
The production architecture is designed to:
|
|
|
|
- preserve verbatim family-history source material as searchable text
|
|
- keep operational complexity low for a personal deployment
|
|
- support asynchronous transcription without requiring distributed infrastructure
|
|
- maintain clear module boundaries so extensions can be added incrementally
|
|
|
|
## Production Scope And Scale
|
|
|
|
The deployed system targets personal use and a corpus of several thousand documents processed over time. The architecture favors simple, composable building blocks over distributed orchestration.
|
|
|
|
Current scope includes:
|
|
|
|
- content source upload and metadata capture
|
|
- asynchronous transcription jobs
|
|
- prompt-library driven transcription behavior, with one Markdown file per prompt
|
|
- original transcription review and optional revision review
|
|
- full-text search over accepted transcripts
|
|
- export of transcript data
|
|
|
|
## Deployment Topology
|
|
|
|
The production deployment uses [Docker Compose](https://docs.docker.com/compose/) and treats containerized databases as extremely lightweight operational dependencies.
|
|
|
|
Running [PostgreSQL](https://www.postgresql.org/docs/) in its own container is considered simple by default for this system.
|
|
|
|
Running [MongoDB](https://www.mongodb.com/docs/) in its own container is also considered simple when document-centric storage is enabled.
|
|
|
|
Container count is not a hard architectural limit; a three-container deployment (app, PostgreSQL, MongoDB) is an acceptable baseline.
|
|
|
|
### Baseline Topology (Two Containers)
|
|
|
|
- one application container
|
|
- one PostgreSQL container
|
|
- embedded background worker execution inside the app process
|
|
|
|
### Expanded Topology (Three Containers)
|
|
|
|
- application container
|
|
- PostgreSQL container
|
|
- MongoDB container
|
|
|
|
No additional queue, scheduler, or search-engine containers are required in the baseline production setup.
|
|
|
|
## Runtime Architecture
|
|
|
|
```mermaid
|
|
flowchart LR
|
|
User[Browser User] --> App[FastAPI + NiceGUI Service]
|
|
App --> Worker[In-process Background Worker]
|
|
App --> PG[(PostgreSQL)]
|
|
App --> MG[(MongoDB Document Store)]
|
|
Worker --> AI[Transcription Provider]
|
|
Worker --> PG
|
|
Worker --> MG
|
|
```
|
|
|
|
## Runtime Ownership And Startup Policy
|
|
|
|
The current implementation now uses explicit lifespan-owned runtime resources.
|
|
|
|
- application lifespan initializes and disposes database runtime resources
|
|
- worker lifecycle is owned by application lifespan startup/shutdown
|
|
- worker receives lifespan-owned database engine dependency explicitly
|
|
- schema bootstrap policy is environment-aware and explicit:
|
|
- development/test default to bootstrap enabled
|
|
- production defaults to bootstrap disabled
|
|
- explicit override is available via configuration
|
|
|
|
This aligns implementation toward REQ-7 and REQ-10 while preserving personal-scale operational simplicity.
|
|
|
|
## Layered Module Structure
|
|
|
|
### Interface Layer
|
|
|
|
Responsibility:
|
|
|
|
- HTTP API and UI routes
|
|
- request/response validation
|
|
- status and result presentation
|
|
|
|
Out of scope:
|
|
|
|
- business-rule enforcement
|
|
- data-access implementation
|
|
|
|
### Application Layer
|
|
|
|
Responsibility:
|
|
|
|
- upload and job orchestration
|
|
- state transitions and retry policy
|
|
- coordination across domain and infrastructure ports
|
|
|
|
Out of scope:
|
|
|
|
- provider-specific protocol details
|
|
- ORM or storage-specific logic
|
|
|
|
### Domain Layer
|
|
|
|
Responsibility:
|
|
|
|
- verbatim transcription policy
|
|
- revision and provenance invariants
|
|
- confidence and annotation semantics
|
|
|
|
Out of scope:
|
|
|
|
- web framework concerns
|
|
- database and network I/O
|
|
|
|
### Infrastructure Layer
|
|
|
|
Responsibility:
|
|
|
|
- persistence adapters (PostgreSQL and MongoDB)
|
|
- transcription-provider adapter
|
|
|
|
Out of scope:
|
|
|
|
- business policy decisions
|
|
|
|
## Processing Workflow
|
|
|
|
Production transcription flow:
|
|
|
|
1. A user uploads one or more content sources through the UI or API.
|
|
2. The application validates payloads and creates document, source, and job records.
|
|
3. The in-process worker de-queues the job and calls the transcription provider.
|
|
4. The application persists original transcription output on the job, plus confidence metadata and provenance events.
|
|
5. Job status transitions from queued to processing to transcribed or failed.
|
|
6. The UI and API expose status, optional revision to original transcription, and searchable transcription text.
|
|
|
|
## Data Model Ownership
|
|
|
|
System-of-record entities:
|
|
|
|
- documents and content sources
|
|
- transcription jobs, original transcription, and status events
|
|
- transcript revisions
|
|
- provenance metadata
|
|
|
|
### Original Transcription And Revision Ownership
|
|
|
|
- each processing job stores the original immutable provider output (`text`)
|
|
- provider metadata (`provider`, `model`, `prompt_name`) and failure detail (`error_detail`) are job-owned processing artifacts
|
|
- revisions are optional user-authored edits linked to a content source
|
|
- a revision can be created from original `job.text`
|
|
- many jobs will have zero revisions; revisions are additive and never overwrite original provider output
|
|
- a document groups one or more content sources (images, PDFs, and future source types)
|
|
|
|
Storage strategy:
|
|
|
|
- PostgreSQL for relational system-of-record entities
|
|
- MongoDB for document-oriented payloads and large transcription artifacts
|
|
- versioned prompt artifacts stored as individual Markdown files for human editing and refinement
|
|
- in-memory execution state treated as ephemeral
|
|
|
|
## Transcription Prompt Asset Policy
|
|
|
|
The production system treats transcription prompts as maintainable content assets.
|
|
|
|
- each transcription prompt is stored in its own Markdown file
|
|
- prompt files are designed for direct human editing and iterative refinement
|
|
- prompt updates are independent and do not require bundling unrelated prompt changes
|
|
- prompt file identity and revision history are tracked through normal repository version control
|
|
|
|
## Simplicity Guardrails
|
|
|
|
The production system enforces these constraints to prevent accidental over-engineering:
|
|
|
|
- PostgreSQL in a container is treated as a lightweight default dependency
|
|
- MongoDB in a container is treated as a lightweight optional dependency
|
|
- three containers (app, PostgreSQL, MongoDB) is an acceptable simple deployment
|
|
- no dedicated queue or search cluster is introduced without measured need
|
|
- external infrastructure is added only behind existing ports/adapters
|
|
|
|
## Extension Path
|
|
|
|
The architecture supports additive growth without changing domain contracts.
|
|
|
|
### Stage 1: Foundation (Current)
|
|
|
|
- upload, transcription, review, search, export
|
|
- in-process worker execution
|
|
- single provider adapter
|
|
- app plus PostgreSQL deployment
|
|
|
|
### Stage 2: Throughput Hardening
|
|
|
|
- optional MongoDB document-store enablement
|
|
- optional external worker/queue process
|
|
- stronger retry and dead-letter handling
|
|
|
|
### Stage 3: Intelligence Features
|
|
|
|
- entity extraction and cross-document linking
|
|
- timeline and narrative assembly
|
|
- optional multi-provider routing
|
|
|
|
Each stage preserves existing module boundaries and keeps migration risk low.
|
|
|
|
## Test Strategy
|
|
|
|
The test strategy is aligned to personal-scale operation with fast, deterministic feedback.
|
|
|
|
### Unit Tests
|
|
|
|
- domain transcription rules and annotation behavior
|
|
- revision-history invariants
|
|
- job state-transition logic
|
|
|
|
### Integration Tests
|
|
|
|
- repository behavior and transaction boundaries
|
|
- persistence-adapter and provider adapter contract mapping
|
|
- upload-to-persistence roundtrip
|
|
|
|
### End-to-End Tests
|
|
|
|
- happy path: upload, transcribe, review, search, export
|
|
- failure path: provider error, retry, surfaced failed status
|
|
|
|
### CI Execution Model
|
|
|
|
- fast suite on each push
|
|
- optional slower provider-sandbox checks on scheduled runs
|
|
|
|
## Risks And Controls
|
|
|
|
### Runtime Responsiveness
|
|
|
|
Risk:
|
|
|
|
- long jobs can reduce responsiveness in a single-process deployment
|
|
|
|
Control:
|
|
|
|
- bounded concurrency and visible job status in the UI
|
|
|
|
### Database Concurrency Limits
|
|
|
|
Risk:
|
|
|
|
- contention can appear under sustained concurrent writes in personal-scale infrastructure
|
|
|
|
Control:
|
|
|
|
- tuned connection pooling and phased use of MongoDB for document-heavy workloads
|
|
|
|
### Provider Output Variance
|
|
|
|
Risk:
|
|
|
|
- transcription quality varies by content source type, handwriting legibility, and source quality
|
|
|
|
Control:
|
|
|
|
- first-class human review and immutable revision history
|
|
|
|
## Technology References
|
|
|
|
- [FastAPI documentation](https://fastapi.tiangolo.com/)
|
|
- [NiceGUI documentation](https://nicegui.io/documentation)
|
|
- [Docker Compose documentation](https://docs.docker.com/compose/)
|
|
- [PostgreSQL documentation](https://www.postgresql.org/docs/)
|
|
- [MongoDB documentation](https://www.mongodb.com/docs/)
|
|
|
|
## Related Local References
|
|
|
|
- [System overview](index.md)
|
|
|
|
## Glossary
|
|
|
|
- Adapter: A component that translates between internal interfaces and external systems such as databases or AI services.
|
|
- Background job: Work executed outside the request/response path so the UI remains responsive.
|
|
- Boundary: A strict separation between modules with different responsibilities.
|
|
- CI (Continuous Integration): Automated test execution for code changes.
|
|
- Contract test: A test that verifies an adapter follows expected input/output behavior at a boundary.
|
|
- Domain layer: The module that contains core business rules and invariants.
|
|
- End-to-end test: A test that validates a full user flow across the running system.
|
|
- Full-text search: Text indexing and querying optimized for natural-language search.
|
|
- In-process worker: A background executor that runs within the same application process.
|
|
- Integration test: A test that verifies interactions between real modules and infrastructure components.
|
|
- MongoDB: A document-oriented database used for flexible, high-variance data structures.
|
|
- Modular monolith: A single deployable application with strongly separated internal modules.
|
|
- Port/Interface: A stable contract used by application/domain code to call infrastructure implementations.
|
|
- Prompt artifact: A single Markdown file that defines one transcription prompt and can be revised independently.
|
|
- Provenance: Metadata that records where generated data came from and how it was produced.
|
|
- Revision history: Optional versioned record of user-authored transcription edits over time.
|
|
- System of record: The authoritative persistent store for canonical data.
|
|
- Vertical slice: A minimal end-to-end feature path spanning UI/API, application logic, and persistence.
|