generated from john/python-template
V1 mostly complete except for some testing. Linting in the last step changed nearly every file which is why this commit is so larger.
This commit is contained in:
Jim Lancaster
+79
-248
@@ -1,299 +1,130 @@
|
||||
# Architecture
|
||||
# Architecture (V1 Baseline)
|
||||
|
||||
This document describes the production architecture of the personal historical-document transcription system. The system is intentionally optimized for single-user operation, low operational overhead, and clean internal boundaries that support future growth without rewrites.
|
||||
This document describes the current architecture of the personal historical-document transcription system and serves as the V1 technical baseline.
|
||||
|
||||
## Architecture Objectives
|
||||
|
||||
The production architecture is designed to:
|
||||
- preserve source material as transcribed text
|
||||
- keep operational complexity low for personal-scale deployment
|
||||
- support asynchronous processing without external queue infrastructure
|
||||
- maintain clear module boundaries for incremental extension
|
||||
|
||||
- preserve verbatim family-history source material as searchable text
|
||||
- keep operational complexity low for a personal deployment
|
||||
- support asynchronous transcription without requiring distributed infrastructure
|
||||
- maintain clear module boundaries so extensions can be added incrementally
|
||||
## Runtime Topology
|
||||
|
||||
## Production Scope And Scale
|
||||
V1 runtime is a modular monolith:
|
||||
|
||||
The deployed system targets personal use and a corpus of several thousand documents processed over time. The architecture favors simple, composable building blocks over distributed orchestration.
|
||||
|
||||
Current scope includes:
|
||||
|
||||
- content source upload and metadata capture
|
||||
- asynchronous transcription jobs
|
||||
- prompt-library driven transcription behavior, with one Markdown file per prompt
|
||||
- original transcription review and optional revision review
|
||||
- full-text search over accepted transcripts
|
||||
- export of transcript data
|
||||
|
||||
## Deployment Topology
|
||||
|
||||
The production deployment uses [Docker Compose](https://docs.docker.com/compose/) and treats containerized databases as extremely lightweight operational dependencies.
|
||||
|
||||
Running [PostgreSQL](https://www.postgresql.org/docs/) in its own container is considered simple by default for this system.
|
||||
|
||||
Running [MongoDB](https://www.mongodb.com/docs/) in its own container is also considered simple when document-centric storage is enabled.
|
||||
|
||||
Container count is not a hard architectural limit; a three-container deployment (app, PostgreSQL, MongoDB) is an acceptable baseline.
|
||||
|
||||
### Baseline Topology (Two Containers)
|
||||
|
||||
- one application container
|
||||
- one PostgreSQL container
|
||||
- embedded background worker execution inside the app process
|
||||
|
||||
### Expanded Topology (Three Containers)
|
||||
|
||||
- application container
|
||||
- PostgreSQL container
|
||||
- MongoDB container
|
||||
|
||||
No additional queue, scheduler, or search-engine containers are required in the baseline production setup.
|
||||
|
||||
## Runtime Architecture
|
||||
- one FastAPI + NiceGUI application process
|
||||
- one in-process async worker loop
|
||||
- relational persistence via SQLModel (SQLite baseline)
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
User[Browser User] --> App[FastAPI + NiceGUI Service]
|
||||
App --> Worker[In-process Background Worker]
|
||||
App --> PG[(PostgreSQL)]
|
||||
App --> MG[(MongoDB Document Store)]
|
||||
Worker --> AI[Transcription Provider]
|
||||
Worker --> PG
|
||||
Worker --> MG
|
||||
U[Browser User] --> A[FastAPI + NiceGUI App]
|
||||
A --> W[In-process Worker]
|
||||
A --> DB[(SQLite via SQLModel)]
|
||||
W --> P[OpenRouter Provider]
|
||||
W --> DB
|
||||
```
|
||||
|
||||
## Runtime Ownership And Startup Policy
|
||||
## Lifecycle Ownership
|
||||
|
||||
The current implementation now uses explicit lifespan-owned runtime resources.
|
||||
Application lifespan owns runtime setup/teardown:
|
||||
|
||||
- application lifespan initializes and disposes database runtime resources
|
||||
- worker lifecycle is owned by application lifespan startup/shutdown
|
||||
- worker receives lifespan-owned database engine dependency explicitly
|
||||
- schema bootstrap policy is environment-aware and explicit:
|
||||
- development/test default to bootstrap enabled
|
||||
- production defaults to bootstrap disabled
|
||||
- explicit override is available via configuration
|
||||
|
||||
This aligns implementation toward REQ-7 and REQ-10 while preserving personal-scale operational simplicity.
|
||||
- configure logging
|
||||
- initialize and dispose DB runtime resources
|
||||
- optional schema bootstrap by environment policy
|
||||
- recover stale processing jobs
|
||||
- start/stop worker consumer lifespan
|
||||
|
||||
## Layered Module Structure
|
||||
|
||||
### Interface Layer
|
||||
|
||||
Responsibility:
|
||||
- `src/transcription/ui/**` (NiceGUI pages/components)
|
||||
- `src/transcription/api/**` (FastAPI routes and error handlers)
|
||||
|
||||
- HTTP API and UI routes
|
||||
- request/response validation
|
||||
- status and result presentation
|
||||
### Application/Workflow Layer
|
||||
|
||||
Out of scope:
|
||||
- `src/transcription/services/workflows.py`
|
||||
- `src/transcription/worker.py`
|
||||
|
||||
- business-rule enforcement
|
||||
- data-access implementation
|
||||
Responsibilities:
|
||||
|
||||
### Application Layer
|
||||
- orchestration and status transitions
|
||||
- retry/timeout behavior
|
||||
- provider call coordination
|
||||
|
||||
Responsibility:
|
||||
### Service Layer
|
||||
|
||||
- upload and job orchestration
|
||||
- state transitions and retry policy
|
||||
- coordination across domain and infrastructure ports
|
||||
- `src/transcription/services/*.py`
|
||||
|
||||
Out of scope:
|
||||
Responsibilities:
|
||||
|
||||
- provider-specific protocol details
|
||||
- ORM or storage-specific logic
|
||||
|
||||
### Domain Layer
|
||||
|
||||
Responsibility:
|
||||
|
||||
- verbatim transcription policy
|
||||
- revision and provenance invariants
|
||||
- confidence and annotation semantics
|
||||
|
||||
Out of scope:
|
||||
|
||||
- web framework concerns
|
||||
- database and network I/O
|
||||
- CRUD and transactional boundaries
|
||||
- domain-aligned persistence operations
|
||||
|
||||
### Infrastructure Layer
|
||||
|
||||
Responsibility:
|
||||
|
||||
- persistence adapters (PostgreSQL and MongoDB)
|
||||
- transcription-provider adapter
|
||||
|
||||
Out of scope:
|
||||
|
||||
- business policy decisions
|
||||
- `src/transcription/db/**` (runtime/session/bootstrap)
|
||||
- `src/transcription/providers/**` (OpenRouter adapter)
|
||||
|
||||
## Processing Workflow
|
||||
|
||||
Production transcription flow:
|
||||
1. User uploads a source file from the UI.
|
||||
2. App persists `Document`, `Job(queued)`, and `Source`.
|
||||
3. Worker claims next queued job and marks `processing`.
|
||||
4. Worker calls provider with prompt + source bytes.
|
||||
5. On success, app writes immutable `Job.text` and marks `transcribed`.
|
||||
6. On failure, app writes `Job.error_detail` and marks `failed`.
|
||||
7. UI exposes job detail, original transcription, and optional revision.
|
||||
|
||||
1. A user uploads one or more content sources through the UI or API.
|
||||
2. The application validates payloads and creates document, source, and job records.
|
||||
3. The in-process worker de-queues the job and calls the transcription provider.
|
||||
4. The application persists original transcription output on the job, plus confidence metadata and provenance events.
|
||||
5. Job status transitions from queued to processing to transcribed or failed.
|
||||
6. The UI and API expose status, optional revision to original transcription, and searchable transcription text.
|
||||
## Domain Ownership Invariants
|
||||
|
||||
## Data Model Ownership
|
||||
- `Job.text` is immutable original provider output.
|
||||
- `Revision` is optional, user-authored, and linked to `Source`.
|
||||
- `Revision` does not overwrite original job transcription.
|
||||
- Status lifecycle is fixed to: `queued -> processing -> transcribed|failed`.
|
||||
|
||||
System-of-record entities:
|
||||
## Data Model Summary
|
||||
|
||||
- documents and content sources
|
||||
- transcription jobs, original transcription, and status events
|
||||
- transcript revisions
|
||||
- provenance metadata
|
||||
- `Document` has many `Source` and many `Job`.
|
||||
- `Source` belongs to one `Document` and one `Job`.
|
||||
- `Source` has optional `Revision` (`0..1`) enforced by unique `revision.source_id`.
|
||||
|
||||
### Original Transcription And Revision Ownership
|
||||
## Simplicity Guardrails (V1)
|
||||
|
||||
- each processing job stores the original immutable provider output (`text`)
|
||||
- provider metadata (`provider`, `model`, `prompt_name`) and failure detail (`error_detail`) are job-owned processing artifacts
|
||||
- revisions are optional user-authored edits linked to a content source
|
||||
- a revision can be created from original `job.text`
|
||||
- many jobs will have zero revisions; revisions are additive and never overwrite original provider output
|
||||
- a document groups one or more content sources (images, PDFs, and future source types)
|
||||
|
||||
Storage strategy:
|
||||
|
||||
- PostgreSQL for relational system-of-record entities
|
||||
- MongoDB for document-oriented payloads and large transcription artifacts
|
||||
- versioned prompt artifacts stored as individual Markdown files for human editing and refinement
|
||||
- in-memory execution state treated as ephemeral
|
||||
|
||||
## Transcription Prompt Asset Policy
|
||||
|
||||
The production system treats transcription prompts as maintainable content assets.
|
||||
|
||||
- each transcription prompt is stored in its own Markdown file
|
||||
- prompt files are designed for direct human editing and iterative refinement
|
||||
- prompt updates are independent and do not require bundling unrelated prompt changes
|
||||
- prompt file identity and revision history are tracked through normal repository version control
|
||||
|
||||
## Simplicity Guardrails
|
||||
|
||||
The production system enforces these constraints to prevent accidental over-engineering:
|
||||
|
||||
- PostgreSQL in a container is treated as a lightweight default dependency
|
||||
- MongoDB in a container is treated as a lightweight optional dependency
|
||||
- three containers (app, PostgreSQL, MongoDB) is an acceptable simple deployment
|
||||
- no dedicated queue or search cluster is introduced without measured need
|
||||
- external infrastructure is added only behind existing ports/adapters
|
||||
- no external queue/broker required
|
||||
- no search engine required
|
||||
- no distributed worker fleet required
|
||||
- keep provider integration behind adapter boundary
|
||||
|
||||
## Extension Path
|
||||
|
||||
The architecture supports additive growth without changing domain contracts.
|
||||
### V1 (current)
|
||||
|
||||
### Stage 1: Foundation (Current)
|
||||
- SQLite baseline
|
||||
- OpenRouter provider
|
||||
- in-process worker
|
||||
- optional single revision workflow
|
||||
|
||||
- upload, transcription, review, search, export
|
||||
- in-process worker execution
|
||||
- single provider adapter
|
||||
- app plus PostgreSQL deployment
|
||||
### V2 (planned)
|
||||
|
||||
### Stage 2: Throughput Hardening
|
||||
- PostgreSQL as relational baseline
|
||||
- optional MongoDB adjunct store for scoped use cases
|
||||
- migration-first schema evolution
|
||||
|
||||
- optional MongoDB document-store enablement
|
||||
- optional external worker/queue process
|
||||
- stronger retry and dead-letter handling
|
||||
|
||||
### Stage 3: Intelligence Features
|
||||
|
||||
- entity extraction and cross-document linking
|
||||
- timeline and narrative assembly
|
||||
- optional multi-provider routing
|
||||
|
||||
Each stage preserves existing module boundaries and keeps migration risk low.
|
||||
See [ver2/ver2.md](ver2/ver2.md) for roadmap details.
|
||||
|
||||
## Test Strategy
|
||||
|
||||
The test strategy is aligned to personal-scale operation with fast, deterministic feedback.
|
||||
- unit tests for model/service behaviors
|
||||
- integration tests for upload/workflow reliability
|
||||
- UI integration tests for page/render contracts
|
||||
- external provider tests opt-in via marker/config
|
||||
|
||||
### Unit Tests
|
||||
## Related References
|
||||
|
||||
- domain transcription rules and annotation behavior
|
||||
- revision-history invariants
|
||||
- job state-transition logic
|
||||
|
||||
### Integration Tests
|
||||
|
||||
- repository behavior and transaction boundaries
|
||||
- persistence-adapter and provider adapter contract mapping
|
||||
- upload-to-persistence roundtrip
|
||||
|
||||
### End-to-End Tests
|
||||
|
||||
- happy path: upload, transcribe, review, search, export
|
||||
- failure path: provider error, retry, surfaced failed status
|
||||
|
||||
### CI Execution Model
|
||||
|
||||
- fast suite on each push
|
||||
- optional slower provider-sandbox checks on scheduled runs
|
||||
|
||||
## Risks And Controls
|
||||
|
||||
### Runtime Responsiveness
|
||||
|
||||
Risk:
|
||||
|
||||
- long jobs can reduce responsiveness in a single-process deployment
|
||||
|
||||
Control:
|
||||
|
||||
- bounded concurrency and visible job status in the UI
|
||||
|
||||
### Database Concurrency Limits
|
||||
|
||||
Risk:
|
||||
|
||||
- contention can appear under sustained concurrent writes in personal-scale infrastructure
|
||||
|
||||
Control:
|
||||
|
||||
- tuned connection pooling and phased use of MongoDB for document-heavy workloads
|
||||
|
||||
### Provider Output Variance
|
||||
|
||||
Risk:
|
||||
|
||||
- transcription quality varies by content source type, handwriting legibility, and source quality
|
||||
|
||||
Control:
|
||||
|
||||
- first-class human review and immutable revision history
|
||||
|
||||
## Technology References
|
||||
|
||||
- [FastAPI documentation](https://fastapi.tiangolo.com/)
|
||||
- [NiceGUI documentation](https://nicegui.io/documentation)
|
||||
- [Docker Compose documentation](https://docs.docker.com/compose/)
|
||||
- [PostgreSQL documentation](https://www.postgresql.org/docs/)
|
||||
- [MongoDB documentation](https://www.mongodb.com/docs/)
|
||||
|
||||
## Related Local References
|
||||
|
||||
- [System overview](index.md)
|
||||
|
||||
## Glossary
|
||||
|
||||
- Adapter: A component that translates between internal interfaces and external systems such as databases or AI services.
|
||||
- Background job: Work executed outside the request/response path so the UI remains responsive.
|
||||
- Boundary: A strict separation between modules with different responsibilities.
|
||||
- CI (Continuous Integration): Automated test execution for code changes.
|
||||
- Contract test: A test that verifies an adapter follows expected input/output behavior at a boundary.
|
||||
- Domain layer: The module that contains core business rules and invariants.
|
||||
- End-to-end test: A test that validates a full user flow across the running system.
|
||||
- Full-text search: Text indexing and querying optimized for natural-language search.
|
||||
- In-process worker: A background executor that runs within the same application process.
|
||||
- Integration test: A test that verifies interactions between real modules and infrastructure components.
|
||||
- MongoDB: A document-oriented database used for flexible, high-variance data structures.
|
||||
- Modular monolith: A single deployable application with strongly separated internal modules.
|
||||
- Port/Interface: A stable contract used by application/domain code to call infrastructure implementations.
|
||||
- Prompt artifact: A single Markdown file that defines one transcription prompt and can be revised independently.
|
||||
- Provenance: Metadata that records where generated data came from and how it was produced.
|
||||
- Revision history: Optional versioned record of user-authored transcription edits over time.
|
||||
- System of record: The authoritative persistent store for canonical data.
|
||||
- Vertical slice: A minimal end-to-end feature path spanning UI/API, application logic, and persistence.
|
||||
- [index.md](index.md)
|
||||
- [requirements.md](requirements.md)
|
||||
- [schema.md](schema.md)
|
||||
- [error_handling.md](error_handling.md)
|
||||
|
||||
Reference in New Issue
Block a user