transcription/docs/architecture.md

# System Architecture (Current Baseline: V6.1)

This document defines the current V6.1 architecture baseline.

## Architecture Objectives

- Preserve durable archival records for Documents, Sources, People, and processing runs.
- Execute page transcription asynchronously with bounded worker behavior.
- Preserve append-only machine-attempt evidence with request/response provenance.
- Keep UI, API, service, persistence, and provider boundaries explicit and testable.

## Technical Stack

- **Runtime:** Python 3.12+
- **Web application:** FastAPI + NiceGUI
- **Persistence:** SQLModel / SQLAlchemy — PostgreSQL in production, SQLite for local development and tests
- **Validation and settings:** Pydantic V2 + pydantic-settings
- **Concurrency:** asyncio worker loop
- **Provider integration:** OpenRouter adapter behind provider interface
- **Deployment:** Docker Compose (app, worker, PostgreSQL, Cloudflare Tunnel)
- **Quality and tests:** Ruff, ty, pytest, pytest-asyncio

## Runtime Topology

```mermaid
flowchart LR
U[Browser User] --> A[FastAPI + NiceGUI App]
A --> W[Asyncio Worker]
A --> DB[(PostgreSQL / SQLite)]
W --> P[Provider Adapter]
W --> DB
```

The worker loop drains two queues in the same pass: queued transcription Jobs and queued
`MaintenanceRun` records. When neither has work, it idles.

### Production deployment

Production runs as a Docker Compose stack with the app and worker as separate services, so the app
process runs with `RUN_EMBEDDED_WORKER=false` and the worker process owns queue draining. Local
development runs a single process with the worker embedded.

```mermaid
flowchart LR
I[Internet] --> CF[cloudflared tunnel + Access]
CF --> APP[app service]
APP --> PG[(postgres service)]
WK[worker service] --> PG
WK --> PROV[OpenRouter]
```

Deployment, rollback, and recovery procedures are in [Production Runbook](production-runbook.md);
backup configuration and restore are in [Backup and Restore](backup_restore.md).

## Layered Boundaries

### Interface Layer

- `src/transcription/ui/**`
- `src/transcription/api/**`

Responsibilities:

- Route registration, page orchestration, presentation adapters.
- Structured user messaging through shared error presenter.
- No direct persistence access from pages/components.

### Service and Orchestration Layer

Aggregate services:

- `src/transcription/services/documents.py`
- `src/transcription/services/people.py`
- `src/transcription/services/jobs.py`
- `src/transcription/services/sources.py`
- `src/transcription/services/photos.py`
- `src/transcription/services/maintenance.py`
- `src/transcription/services/evidence.py` (read/projection only)

Orchestration modules:

- `src/transcription/services/store.py`
- `src/transcription/services/workflows.py`

Responsibilities:

- Aggregate ownership and invariants.
- Transaction-aware write helpers.
- Cross-service workflows in orchestration modules (`store.py`, `workflows.py`).
- Lookup-table CRUD through the generic `RegistryService` base (`registry.py`), which is not an
  aggregate owner itself.

Module classification and per-model ownership are defined in
[services instructions](../.github/instructions/services.instructions.md).

### Persistence Layer

- `src/transcription/db/**`

Responsibilities:

- SQLModel definitions, async session/engine runtime, registry bootstrap.
- Loader helpers that enforce explicit eager loading with `lazy="raise"` relationships.

### Provider Layer

- `src/transcription/providers/**`

Responsibilities:

- Provider API encapsulation.
- Request manifest and transport evidence capture.
- Normalized transcription result contract.

## Core Domain Model

- `Document` owns archival metadata and links to `Source`, `Job`, and `DocumentPerson`.
- `Source` is a document page/file record with selected machine projection and human revision.
- `Job` is an aggregate processing run with status and frozen prompt/runtime settings.
- `JobSource` is queue/membership state for one `(job, source)` pair.
- `ExecutionAttempt` is append-only evidence for each provider call.
- `Photo` is person imagery owned by `PhotosService`.
- `MaintenanceRun` is one queued or executed operational maintenance run.
- `GenealogyPerson`, `GenealogyFamily`, `GenealogyFamilyChild`, and `GenealogyCitation` store
  imported GEDCOM genealogy data and citation provenance.
- `DocumentType` and `PersonRole` are UUID-backed registries with optional protected `semantic_key`.
- `Tag` is a shared registry reached through both document and person tagging, linked by
  `DocumentTag` and `PersonTag`.

## Processing and Evidence Workflow

1. User creates/updates Document metadata and linked People atomically through workflow orchestration.
2. User creates a Job by uploading one or more Source files or by retranscribing an existing Source.
3. Source files are validated and stored; orientation normalization may be applied at ingest, and stored bytes become the canonical processing bytes.
4. Worker claims queued Job, transitions to `processing`, and processes pending pages in deterministic order.
5. Each provider call writes one immutable `ExecutionAttempt` with:
   - request manifest + hash
   - transport evidence (when response exists)
   - SDK snapshot and normalized metadata
   - outcome, timing, and error details when applicable
6. `JobSource` status is updated as queue/projection state; `Source.raw_transcription` is set on first successful attempt and can be explicitly re-pointed by candidate promotion.
7. Job terminal status resolves to `transcribed`, `partial_success`, or `failed`.

## Status Semantics

- **Job statuses:** `queued`, `processing`, `transcribed`, `partial_success`, `failed`
  - Operational success path resolves to `transcribed`.
- **JobSource statuses:** `pending`, `transcribed`, `failed`, `cancelled`
- **MaintenanceRun statuses:** `queued`, `processing`, `succeeded`, `failed`
  - Maintenance uses `succeeded` rather than `transcribed`; the transcription vocabulary does not
    apply to operational runs.
- **Maintenance job types:** `backup`, `storage_reconciliation`, `gedcom_import`

## Maintenance Execution

Operational maintenance is queue-backed rather than run inline from the UI, so it survives request
lifetime and is recorded:

1. Settings enqueues a `MaintenanceRun` with `status=queued` and a `triggered_by` marker.
2. The worker claims the oldest queued run with a conditional update, moving it to `processing`.
3. `backup` runs the deploy backup script; `storage_reconciliation` compares stored media against
   `Document`/`Source` records; `gedcom_import` parses the latest uploaded `.ged` file and upserts
   genealogy records.
4. The run finalizes to `succeeded` or `failed` with summary, timing, log path, and `error_detail`.

`MaintenanceRun` records operational history and is not evidence in the `ExecutionAttempt` sense;
append-only guarantees apply to transcription attempts.

## Security and Path Handling Boundaries

- Print media delivery uses record-validated API route:
  - `src/transcription/api/print_api.py`
- General UI media links resolve through:
  - `src/transcription/ui/components/media_urls.py`
- Local filesystem paths must never be accepted from user input as trusted media routes.

## Concurrency and Reliability Principles

- Worker loop reuses service bundle/provider resources for pooled calls.
- Provider-call timeout is explicit and bounded.
- Non-retriable worker-loop faults are surfaced and stop loop spin.
- Per-page outcomes are durably persisted before processing next page.

## Design Decisions and Rationale

### Why `transcribed` is the success terminal state

- The worker and job orchestration resolve successful completion to `JobStatus.TRANSCRIBED`, with mixed and failure outcomes represented by `partial_success` and `failed`.
- This keeps terminal status vocabulary aligned with what the pipeline actually produces: transcribed page content and evidence, not a generic completion marker.

### Why evidence history is append-only while page text is a projection

- `ExecutionAttempt` stores immutable per-call evidence and preserves full attempt history across retries.
- `Source.raw_transcription` is intentionally a mutable projection so UI and exports can show a selected current machine text without mutating historical evidence.
- This split keeps auditability and UX both first-class: history is durable, presentation is editable.

### Why orchestration modules own cross-service workflows

- Service modules do not import each other; aggregate ownership remains local to each service.
- Multi-aggregate writes are coordinated in orchestration modules (`store.py`, `workflows.py`) so transaction boundaries are explicit and testable.
- This avoids circular dependencies and keeps cross-cutting workflow logic centralized.

### Why explicit eager loading is required

- ORM relationships are configured with `lazy="raise"` in key paths, so code must request needed relationships up front.
- This prevents hidden query behavior in UI/service code and makes read shape deterministic and reviewable.

### Why canonical source bytes may be ingest-normalized

- Ingest normalization can correct orientation before persistence so provider calls, evidence hashes, and rendered processing source are consistent.
- The canonical stored bytes, digest, and size become the durable processing identity for that source.

### Why media access uses controlled routes/helpers

- Print/export media uses record-validated API endpoints to avoid direct filesystem path exposure.
- General UI media URLs are generated through shared resolver helpers to keep path handling consistent and centralized.

## Scope Boundary

Current architecture rules live in `docs/*`.

## Related References

- [System Requirements](requirements.md)
- [Data Model](schema.md)
- [Error Handling Policy](error_handling.md)
- [Production Runbook](production-runbook.md)
- [Backup and Restore](backup_restore.md)
- [Error Handling invariant](./invariant/error_handling.md)
- [AI evidence invariant](./invariant/ai_evidence_and_provenance.md)