generated from john/python-template
172 lines
7.2 KiB
Markdown
172 lines
7.2 KiB
Markdown
# System Architecture (Version 4)
|
|
|
|
This document defines the current V4 architecture baseline.
|
|
|
|
## Architecture Objectives
|
|
|
|
- Preserve durable archival records for Documents, Sources, People, and processing runs.
|
|
- Execute page transcription asynchronously with bounded worker behavior.
|
|
- Preserve append-only machine-attempt evidence with request/response provenance.
|
|
- Keep UI, API, service, persistence, and provider boundaries explicit and testable.
|
|
|
|
## Technical Stack
|
|
|
|
- **Runtime:** Python 3.12+
|
|
- **Web application:** FastAPI + NiceGUI
|
|
- **Persistence:** SQLModel / SQLAlchemy (SQLite-first, PostgreSQL-compatible model design)
|
|
- **Validation and settings:** Pydantic V2 + pydantic-settings
|
|
- **Concurrency:** asyncio worker loop
|
|
- **Provider integration:** OpenRouter adapter behind provider interface
|
|
- **Quality and tests:** Ruff, ty, pytest, pytest-asyncio
|
|
|
|
## Runtime Topology
|
|
|
|
```mermaid
|
|
flowchart LR
|
|
U[Browser User] --> A[FastAPI + NiceGUI App]
|
|
A --> W[Asyncio Worker]
|
|
A --> DB[(SQLite/PostgreSQL Model)]
|
|
W --> P[Provider Adapter]
|
|
W --> DB
|
|
```
|
|
|
|
## Layered Boundaries
|
|
|
|
### Interface Layer
|
|
|
|
- `src/transcription/ui/**`
|
|
- `src/transcription/api/**`
|
|
|
|
Responsibilities:
|
|
|
|
- Route registration, page orchestration, presentation adapters.
|
|
- Structured user messaging through shared error presenter.
|
|
- No direct persistence access from pages/components.
|
|
|
|
### Service and Orchestration Layer
|
|
|
|
- `src/transcription/services/documents.py`
|
|
- `src/transcription/services/people.py`
|
|
- `src/transcription/services/jobs.py`
|
|
- `src/transcription/services/sources.py`
|
|
- `src/transcription/services/evidence.py`
|
|
- `src/transcription/services/store.py`
|
|
- `src/transcription/services/workflows.py`
|
|
|
|
Responsibilities:
|
|
|
|
- Aggregate ownership and invariants.
|
|
- Transaction-aware write helpers.
|
|
- Cross-service workflows in orchestration modules (`store.py`, `workflows.py`).
|
|
|
|
### Persistence Layer
|
|
|
|
- `src/transcription/db/**`
|
|
|
|
Responsibilities:
|
|
|
|
- SQLModel definitions, async session/engine runtime, registry bootstrap.
|
|
- Loader helpers that enforce explicit eager loading with `lazy="raise"` relationships.
|
|
|
|
### Provider Layer
|
|
|
|
- `src/transcription/providers/**`
|
|
|
|
Responsibilities:
|
|
|
|
- Provider API encapsulation.
|
|
- Request manifest and transport evidence capture.
|
|
- Normalized transcription result contract.
|
|
|
|
## Core Domain Model
|
|
|
|
- `Document` owns archival metadata and links to `Source`, `Job`, and `DocumentPerson`.
|
|
- `Source` is a document page/file record with selected machine projection and human revision.
|
|
- `Job` is an aggregate processing run with status and frozen prompt/runtime settings.
|
|
- `JobSource` is queue/membership state for one `(job, source)` pair.
|
|
- `ExecutionAttempt` is append-only evidence for each provider call.
|
|
- `DocumentType` and `PersonRole` are UUID-backed registries with optional protected `semantic_key`.
|
|
|
|
## Processing and Evidence Workflow
|
|
|
|
1. User creates/updates Document metadata and linked People atomically through workflow orchestration.
|
|
2. User creates a Job by uploading one or more Source files or by retranscribing an existing Source.
|
|
3. Source files are validated and stored; orientation normalization may be applied at ingest, and stored bytes become the canonical processing bytes.
|
|
4. Worker claims queued Job, transitions to `processing`, and processes pending pages in deterministic order.
|
|
5. Each provider call writes one immutable `ExecutionAttempt` with:
|
|
- request manifest + hash
|
|
- transport evidence (when response exists)
|
|
- SDK snapshot and normalized metadata
|
|
- outcome, timing, and error details when applicable
|
|
6. `JobSource` status is updated as queue/projection state; `Source.raw_transcription` is set on first successful attempt and can be explicitly re-pointed by candidate promotion.
|
|
7. Job terminal status resolves to `transcribed`, `partial_success`, or `failed`.
|
|
|
|
## Status Semantics
|
|
|
|
- **Job statuses:** `queued`, `processing`, `transcribed`, `partial_success`, `failed`
|
|
- Operational success path resolves to `transcribed`.
|
|
- **JobSource statuses:** `pending`, `transcribed`, `failed`, `cancelled`
|
|
|
|
## Security and Path Handling Boundaries
|
|
|
|
- Print media delivery uses record-validated API route:
|
|
- `src/transcription/api/v4_print.py`
|
|
- General UI media links resolve through:
|
|
- `src/transcription/ui/components/media_urls.py`
|
|
- Local filesystem paths must never be accepted from user input as trusted media routes.
|
|
|
|
## Concurrency and Reliability Principles
|
|
|
|
- Worker loop reuses service bundle/provider resources for pooled calls.
|
|
- Provider-call timeout is explicit and bounded.
|
|
- Non-retriable worker-loop faults are surfaced and stop loop spin.
|
|
- Per-page outcomes are durably persisted before processing next page.
|
|
|
|
## Design Decisions and Rationale
|
|
|
|
### Why `transcribed` is the success terminal state
|
|
|
|
- The worker and job orchestration resolve successful completion to `JobStatus.TRANSCRIBED`, with mixed and failure outcomes represented by `partial_success` and `failed`.
|
|
- This keeps terminal status vocabulary aligned with what the pipeline actually produces: transcribed page content and evidence, not a generic completion marker.
|
|
|
|
### Why evidence history is append-only while page text is a projection
|
|
|
|
- `ExecutionAttempt` stores immutable per-call evidence and preserves full attempt history across retries.
|
|
- `Source.raw_transcription` is intentionally a mutable projection so UI and exports can show a selected current machine text without mutating historical evidence.
|
|
- This split keeps auditability and UX both first-class: history is durable, presentation is editable.
|
|
|
|
### Why orchestration modules own cross-service workflows
|
|
|
|
- Service modules do not import each other; aggregate ownership remains local to each service.
|
|
- Multi-aggregate writes are coordinated in orchestration modules (`store.py`, `workflows.py`) so transaction boundaries are explicit and testable.
|
|
- This avoids circular dependencies and keeps cross-cutting workflow logic centralized.
|
|
|
|
### Why explicit eager loading is required
|
|
|
|
- ORM relationships are configured with `lazy="raise"` in key paths, so code must request needed relationships up front.
|
|
- This prevents hidden query behavior in UI/service code and makes read shape deterministic and reviewable.
|
|
|
|
### Why canonical source bytes may be ingest-normalized
|
|
|
|
- Ingest normalization can correct orientation before persistence so provider calls, evidence hashes, and rendered processing source are consistent.
|
|
- The canonical stored bytes, digest, and size become the durable processing identity for that source.
|
|
|
|
### Why media access uses controlled routes/helpers
|
|
|
|
- Print/export media uses record-validated API endpoints to avoid direct filesystem path exposure.
|
|
- General UI media URLs are generated through shared resolver helpers to keep path handling consistent and centralized.
|
|
|
|
## Historical Context Boundary
|
|
|
|
Superseded V4.x scope/plan/review documents were intentionally removed from the active tree and archived at git tag `docs-v4x-archive`.
|
|
Current architecture rules live only in `docs/ver4/*`; historical files are reference material only.
|
|
|
|
## Related References
|
|
|
|
- [System Requirements](requirements_v4.md)
|
|
- [Data Model](schema_v4.md)
|
|
- [Error Handling Policy](error_handling_v4.md)
|
|
- [V4 Revision History](history.md)
|
|
- [Error Handling invariant](../invariant/error_handling.md)
|
|
- [AI evidence invariant](../invariant/ai_evidence_and_provenance.md)
|