transcription/docs/architecture.md

System Architecture (Version 4)

This document defines the current Version 4 architecture baseline.

Architecture Objectives

• Preserve durable archival records for Documents, Sources, People, and processing runs.
• Execute page transcription asynchronously with bounded worker behavior.
• Preserve append-only machine-attempt evidence with request/response provenance.
• Keep UI, API, service, persistence, and provider boundaries explicit and testable.

Technical Stack

• Runtime: Python 3.12+
• Web application: FastAPI + NiceGUI
• Persistence: SQLModel / SQLAlchemy (SQLite-first, PostgreSQL-compatible model design)
• Validation and settings: Pydantic V2 + pydantic-settings
• Concurrency: asyncio worker loop
• Provider integration: OpenRouter adapter behind provider interface
• Quality and tests: Ruff, ty, pytest, pytest-asyncio

Runtime Topology

flowchart LR
U[Browser User] --> A[FastAPI + NiceGUI App]
A --> W[Asyncio Worker]
A --> DB[(SQLite/PostgreSQL Model)]
W --> P[Provider Adapter]
W --> DB

Layered Boundaries

Interface Layer

• src/transcription/ui/**
• src/transcription/api/**

Responsibilities:

• Route registration, page orchestration, presentation adapters.
• Structured user messaging through shared error presenter.
• No direct persistence access from pages/components.

Service and Orchestration Layer

• src/transcription/services/documents.py
• src/transcription/services/people.py
• src/transcription/services/jobs.py
• src/transcription/services/sources.py
• src/transcription/services/evidence.py
• src/transcription/services/store.py
• src/transcription/services/workflows.py

Responsibilities:

• Aggregate ownership and invariants.
• Transaction-aware write helpers.
• Cross-service workflows in orchestration modules (store.py, workflows.py).

Persistence Layer

• src/transcription/db/**

Responsibilities:

• SQLModel definitions, async session/engine runtime, registry bootstrap.
• Loader helpers that enforce explicit eager loading with lazy="raise" relationships.

Provider Layer

• src/transcription/providers/**

Responsibilities:

• Provider API encapsulation.
• Request manifest and transport evidence capture.
• Normalized transcription result contract.

Core Domain Model

• Document owns archival metadata and links to Source, Job, and DocumentPerson.
• Source is a document page/file record with selected machine projection and human revision.
• Job is an aggregate processing run with status and frozen prompt/runtime settings.
• JobSource is queue/membership state for one (job, source) pair.
• ExecutionAttempt is append-only evidence for each provider call.
• DocumentType and PersonRole are UUID-backed registries with optional protected semantic_key.

Processing and Evidence Workflow

1. User creates/updates Document metadata and linked People atomically through workflow orchestration.
2. User creates a Job by uploading one or more Source files or by retranscribing an existing Source.
3. Source files are validated and stored; orientation normalization may be applied at ingest, and stored bytes become the canonical processing bytes.
4. Worker claims queued Job, transitions to processing, and processes pending pages in deterministic order.
5. Each provider call writes one immutable ExecutionAttempt with:
   • request manifest + hash
   • transport evidence (when response exists)
   • SDK snapshot and normalized metadata
   • outcome, timing, and error details when applicable
6. JobSource status is updated as queue/projection state; Source.raw_transcription is set on first successful attempt and can be explicitly re-pointed by candidate promotion.
7. Job terminal status resolves to transcribed, partial_success, or failed.

Status Semantics

• Job statuses: queued, processing, transcribed, partial_success, failed
  • Operational success path resolves to transcribed.
• JobSource statuses: pending, transcribed, failed, cancelled

Security and Path Handling Boundaries

• Print media delivery uses record-validated API route:
  • src/transcription/api/print_api.py
• General UI media links resolve through:
  • src/transcription/ui/components/media_urls.py
• Local filesystem paths must never be accepted from user input as trusted media routes.

Concurrency and Reliability Principles

• Worker loop reuses service bundle/provider resources for pooled calls.
• Provider-call timeout is explicit and bounded.
• Non-retriable worker-loop faults are surfaced and stop loop spin.
• Per-page outcomes are durably persisted before processing next page.

Design Decisions and Rationale

Why transcribed is the success terminal state

• The worker and job orchestration resolve successful completion to JobStatus.TRANSCRIBED, with mixed and failure outcomes represented by partial_success and failed.
• This keeps terminal status vocabulary aligned with what the pipeline actually produces: transcribed page content and evidence, not a generic completion marker.

Why evidence history is append-only while page text is a projection

• ExecutionAttempt stores immutable per-call evidence and preserves full attempt history across retries.
• Source.raw_transcription is intentionally a mutable projection so UI and exports can show a selected current machine text without mutating historical evidence.
• This split keeps auditability and UX both first-class: history is durable, presentation is editable.

Why orchestration modules own cross-service workflows

• Service modules do not import each other; aggregate ownership remains local to each service.
• Multi-aggregate writes are coordinated in orchestration modules (store.py, workflows.py) so transaction boundaries are explicit and testable.
• This avoids circular dependencies and keeps cross-cutting workflow logic centralized.

Why explicit eager loading is required

• ORM relationships are configured with lazy="raise" in key paths, so code must request needed relationships up front.
• This prevents hidden query behavior in UI/service code and makes read shape deterministic and reviewable.

Why canonical source bytes may be ingest-normalized

• Ingest normalization can correct orientation before persistence so provider calls, evidence hashes, and rendered processing source are consistent.
• The canonical stored bytes, digest, and size become the durable processing identity for that source.

Why media access uses controlled routes/helpers

• Print/export media uses record-validated API endpoints to avoid direct filesystem path exposure.
• General UI media URLs are generated through shared resolver helpers to keep path handling consistent and centralized.

Scope Boundary

Current architecture rules live in docs/*.

Related References

• System Requirements
• Data Model
• Error Handling Policy
• Error Handling invariant
• AI evidence invariant