transcription/docs/architecture.md

Architecture

This document describes the production architecture of the personal historical-document transcription system. The system is intentionally optimized for single-user operation, low operational overhead, and clean internal boundaries that support future growth without rewrites.

Architecture Objectives

The production architecture is designed to:

• preserve verbatim family-history source material as searchable text
• keep operational complexity low for a personal deployment
• support asynchronous transcription without requiring distributed infrastructure
• maintain clear module boundaries so extensions can be added incrementally

Production Scope And Scale

The deployed system targets personal use and a corpus of several thousand documents processed over time. The architecture favors simple, composable building blocks over distributed orchestration.

Current scope includes:

• document upload and metadata capture
• asynchronous transcription jobs
• prompt-library driven transcription behavior, with one Markdown file per prompt
• transcript review and revision history
• full-text search over accepted transcripts
• export of transcript data

Deployment Topology

The production deployment uses Docker Compose and treats containerized databases as extremely lightweight operational dependencies.

Running PostgreSQL in its own container is considered simple by default for this system.

Running MongoDB in its own container is also considered simple when document-centric storage is enabled.

Container count is not a hard architectural limit; a three-container deployment (app, PostgreSQL, MongoDB) is an acceptable baseline.

Baseline Topology (Two Containers)

• one application container
• one PostgreSQL container
• embedded background worker execution inside the app process

Expanded Topology (Three Containers)

• application container
• PostgreSQL container
• MongoDB container

No additional queue, scheduler, or search-engine containers are required in the baseline production setup.

Runtime Architecture

flowchart LR
    User[Browser User] --> App[FastAPI + NiceGUI Service]
    App --> Worker[In-process Background Worker]
    App --> PG[(PostgreSQL)]
    App --> MG[(MongoDB Document Store)]
    Worker --> AI[Transcription Provider]
    Worker --> PG
    Worker --> MG

Runtime Ownership And Startup Policy (V1 Step 1)

The current implementation now uses explicit lifespan-owned runtime resources.

• application lifespan initializes and disposes database runtime resources
• worker lifecycle is owned by application lifespan startup/shutdown
• worker receives lifespan-owned database engine dependency explicitly
• schema bootstrap policy is environment-aware and explicit:
  • development/test default to bootstrap enabled
  • production defaults to bootstrap disabled
  • explicit override is available via configuration

This aligns implementation toward REQ-7 and REQ-10 while preserving personal-scale operational simplicity.

Layered Module Structure

Interface Layer

Responsibility:

• HTTP API and UI routes
• request/response validation
• status and result presentation

Out of scope:

• business-rule enforcement
• data-access implementation

Application Layer

Responsibility:

• upload and job orchestration
• state transitions and retry policy
• coordination across domain and infrastructure ports

Out of scope:

• provider-specific protocol details
• ORM or storage-specific logic

Domain Layer

Responsibility:

• verbatim transcription policy
• revision and provenance invariants
• confidence and annotation semantics

Out of scope:

• web framework concerns
• database and network I/O

Infrastructure Layer

Responsibility:

• persistence adapters (PostgreSQL and MongoDB)
• transcription-provider adapter

Out of scope:

• business policy decisions

Processing Workflow

Production transcription flow:

1. A user uploads an image or PDF through the UI or API.
2. The application validates payloads and creates document and job records.
3. The in-process worker dequeues the job and calls the transcription provider.
4. The application persists transcript output, confidence metadata, and provenance events.
5. Job status transitions from queued to processing to transcribed or failed.
6. The UI and API expose status, revision history, and searchable transcript text.

Data Model Ownership

System-of-record entities:

• documents and pages
• transcription jobs and status events
• transcript revisions
• provenance metadata

Storage strategy:

• PostgreSQL for relational system-of-record entities
• MongoDB for document-oriented payloads and large transcription artifacts
• versioned prompt artifacts stored as individual Markdown files for human editing and refinement
• in-memory execution state treated as ephemeral

Transcription Prompt Asset Policy

The production system treats transcription prompts as maintainable content assets.

• each transcription prompt is stored in its own Markdown file
• prompt files are designed for direct human editing and iterative refinement
• prompt updates are independent and do not require bundling unrelated prompt changes
• prompt file identity and revision history are tracked through normal repository version control

Simplicity Guardrails

The production system enforces these constraints to prevent accidental over-engineering:

• PostgreSQL in a container is treated as a lightweight default dependency
• MongoDB in a container is treated as a lightweight optional dependency
• three containers (app, PostgreSQL, MongoDB) is an acceptable simple deployment
• no dedicated queue or search cluster is introduced without measured need
• external infrastructure is added only behind existing ports/adapters

Extension Path

The architecture supports additive growth without changing domain contracts.

Stage 1: Foundation (Current)

• upload, transcription, review, search, export
• in-process worker execution
• single provider adapter
• app plus PostgreSQL deployment

Stage 2: Throughput Hardening

• optional MongoDB document-store enablement
• optional external worker/queue process
• stronger retry and dead-letter handling

Stage 3: Intelligence Features

• entity extraction and cross-document linking
• timeline and narrative assembly
• optional multi-provider routing

Each stage preserves existing module boundaries and keeps migration risk low.

Test Strategy

The test strategy is aligned to personal-scale operation with fast, deterministic feedback.

Unit Tests

• domain transcription rules and annotation behavior
• revision-history invariants
• job state-transition logic

Integration Tests

• repository behavior and transaction boundaries
• persistence-adapter and provider adapter contract mapping
• upload-to-persistence roundtrip

End-to-End Tests

• happy path: upload, transcribe, review, search, export
• failure path: provider error, retry, surfaced failed status

CI Execution Model

• fast suite on each push
• optional slower provider-sandbox checks on scheduled runs

Risks And Controls

Runtime Responsiveness

Risk:

• long jobs can reduce responsiveness in a single-process deployment

Control:

• bounded concurrency and visible job status in the UI

Database Concurrency Limits

Risk:

• contention can appear under sustained concurrent writes in personal-scale infrastructure

Control:

• tuned connection pooling and phased use of MongoDB for document-heavy workloads

Provider Output Variance

Risk:

• transcription quality varies by document type, handwriting legibility, and image quality

Control:

• first-class human review and immutable revision history

Technology References

• FastAPI documentation
• NiceGUI documentation
• Docker Compose documentation
• PostgreSQL documentation
• MongoDB documentation

Related Pages

• System overview
• Version 1 plan
• Version 1 Step 1 plan
• Version 1 Step 1 results
• Architecture decision records index

Glossary

• Adapter: A component that translates between internal interfaces and external systems such as databases or AI services.
• Background job: Work executed outside the request/response path so the UI remains responsive.
• Boundary: A strict separation between modules with different responsibilities.
• CI (Continuous Integration): Automated test execution for code changes.
• Contract test: A test that verifies an adapter follows expected input/output behavior at a boundary.
• Domain layer: The module that contains core business rules and invariants.
• End-to-end test: A test that validates a full user flow across the running system.
• Full-text search: Text indexing and querying optimized for natural-language search.
• In-process worker: A background executor that runs within the same application process.
• Integration test: A test that verifies interactions between real modules and infrastructure components.
• MongoDB: A document-oriented database used for flexible, high-variance data structures.
• Modular monolith: A single deployable application with strongly separated internal modules.
• Port/Interface: A stable contract used by application/domain code to call infrastructure implementations.
• Prompt artifact: A single Markdown file that defines one transcription prompt and can be revised independently.
• Provenance: Metadata that records where generated data came from and how it was produced.
• Revision history: Versioned record of transcript edits over time.
• System of record: The authoritative persistent store for canonical data.
• Vertical slice: A minimal end-to-end feature path spanning UI/API, application logic, and persistence.