gpt-5.3-codex review phase 1 - Flatten the documentation

docs/ver4/architecture_v4.md

@@ -1,75 +1,36 @@
 # System Architecture (Version 4)
 
-This document describes the production architecture of the document transcription system.
+This document defines the current V4 architecture baseline.
 
 ## Architecture Objectives
 
-- Preserve original source material, per-execution machine output, and separate human revision.
-- Support batching one or more images into ordered multi-page documents.
-- Capture submission-time prompt provenance and a per-page OpenRouter SDK response snapshot.
-- Execute page transcription concurrently with bounded `asyncio` workers.
-- Maintain relational portability across SQLite and PostgreSQL.
-- Keep operator workflows cross-platform and Python-driven.
-- Support one role-bearing link per Person and Document through an extensible role registry.
-- Support registry-driven document classification with protected semantic built-ins.
-
-## Core Capabilities
-
-- Ingest one or more images into sequential `Source` pages under a `Document`.
-- Execute asynchronous vision transcription with bounded worker concurrency.
-- Preserve original source files with SHA-256 digests and byte sizes.
-- Freeze prompt text, prompt hash, model, and explicitly configured sampling parameters on each `Job`.
-- Preserve page-level machine output, normalized metadata, and an SDK-serialized OpenRouter response snapshot on `JobSource`.
-- Organize historical `Person` records through UUID-identified Document links and extensible roles.
-- Classify Documents through a UUID-identified registry with hidden semantic built-ins and unique labels.
-- Maintain human revision separately from machine-generated text.
-- Isolate page failures so multi-page jobs can complete with partial success.
-- Operate across supported platforms through Python-based application and maintenance tooling.
-
-V4.2 extends this baseline with immutable execution attempts, exact OpenRouter transport evidence, safe
-versioned exports, and provider-neutral derived-artifact provenance. `JobSource` remains the mutable queue and
-compatibility projection; `ExecutionAttempt` is the authoritative append-only processing history. See the
-[V4.2 Scope Boundary](../ver4.2/scope_boundary_v4_2.md).
+- Preserve durable archival records for Documents, Sources, People, and processing runs.
+- Execute page transcription asynchronously with bounded worker behavior.
+- Preserve append-only machine-attempt evidence with request/response provenance.
+- Keep UI, API, service, persistence, and provider boundaries explicit and testable.
 
 ## Technical Stack
 
-- **Runtime:** Python 3.12 or later.
-- **Web application:** FastAPI and NiceGUI.
-- **Persistence:** SQLModel and SQLAlchemy, with SQLite and PostgreSQL support.
-- **Validation and settings:** Pydantic V2 and pydantic-settings.
-- **Concurrency:** Python `asyncio` workers.
-- **Vision integration:** OpenRouter through the application's provider adapter.
-- **Testing and quality:** pytest, pytest-asyncio, Ruff, and ty.
+- **Runtime:** Python 3.12+
+- **Web application:** FastAPI + NiceGUI
+- **Persistence:** SQLModel / SQLAlchemy (SQLite-first, PostgreSQL-compatible model design)
+- **Validation and settings:** Pydantic V2 + pydantic-settings
+- **Concurrency:** asyncio worker loop
+- **Provider integration:** OpenRouter adapter behind provider interface
+- **Quality and tests:** Ruff, ty, pytest, pytest-asyncio
 
 ## Runtime Topology
 
-The runtime operates as an asynchronous Python application:
-
-- FastAPI + NiceGUI web application process.
-- In-process `asyncio` worker engine for transcription execution.
-- Relational persistence via SQLModel / SQLAlchemy.
-- Pydantic V2 validation across API payloads, prompt configuration, and structured metadata.
-
-^^^mermaid
+```mermaid
 flowchart LR
 U[Browser User] --> A[FastAPI + NiceGUI App]
-A --> W[Asyncio Worker Engine]
-A --> DB[(Relational DB)]
-W --> P[Vision Provider APIs]
+A --> W[Asyncio Worker]
+A --> DB[(SQLite/PostgreSQL Model)]
+W --> P[Provider Adapter]
 W --> DB
-^^^
+```
 
-## Lifecycle Ownership
-
-Application lifespan owns runtime setup and teardown:
-
-- Initialize logging, settings, directories, and prompt configuration.
-- Manage asynchronous database engine connection pools.
-- Execute database bootstrap or migrations.
-- Recover stale or interrupted jobs on startup.
-- Manage graceful shutdown of active background tasks.
-
-## Layered Module Structure
+## Layered Boundaries
 
 ### Interface Layer
 
@@ -78,136 +39,95 @@ Application lifespan owns runtime setup and teardown:
 
 Responsibilities:
 
-- Render document, source, person, job, and classification views.
-- Accept user input for uploads, editing, linking, and revisions.
-- Present structured validation and conflict feedback.
+- Route registration, page orchestration, presentation adapters.
+- Structured user messaging through shared error presenter.
+- No direct persistence access from pages/components.
 
-### Application and Async Worker Layer
+### Service and Orchestration Layer
 
-- `src/transcription/services/workflows.py`
-- `src/transcription/worker.py`
-
-Responsibilities:
-
-- Orchestrate uploads, job creation, and status transitions.
-- Execute per-page provider calls through bounded concurrency.
-- Persist page-level outcomes and update aggregate job state.
-
-### Domain and Service Layer
-
-- `src/transcription/db/models.py`
 - `src/transcription/services/documents.py`
-- `src/transcription/services/sources.py`
-- `src/transcription/services/jobs.py`
 - `src/transcription/services/people.py`
+- `src/transcription/services/jobs.py`
+- `src/transcription/services/sources.py`
+- `src/transcription/services/evidence.py`
+- `src/transcription/services/store.py`
 - `src/transcription/services/workflows.py`
 
 Responsibilities:
 
-- Keep one primary service boundary per aggregate: Documents, Sources, Jobs, and People.
-- Documents own document records and the document-type registry.
-- Sources own source records, revisions, source media formats, MIME resolution, and page execution evidence.
-- Jobs own job lifecycle state and transitions.
-- People own person records, relationship roles, document-person links, and portrait media.
-- Apply deterministic conflict handling for relationship-role writes.
-- Synchronize each Document's complete Person link set in the same transaction as Document fields.
-- Resolve and validate registry records by UUID; use hidden semantic keys only for application-owned built-in behavior.
+- Aggregate ownership and invariants.
+- Transaction-aware write helpers.
+- Cross-service workflows in orchestration modules (`store.py`, `workflows.py`).
 
-### Source Media Policy
-
-- `services/sources.py` is the single authority for accepted Source extensions and canonical MIME types.
-- Storage and provider payload loading must call the same Source validation functions.
-- Supported Source formats are JPEG, PNG, TIFF, and PDF.
-- Upload is an interface action, not a domain aggregate. Service names, errors, and workflow variables use
-  `Source` terminology; compatibility aliases may remain temporarily at old import boundaries.
-
-### Infrastructure Layer
+### Persistence Layer
 
 - `src/transcription/db/**`
+
+Responsibilities:
+
+- SQLModel definitions, async session/engine runtime, registry bootstrap.
+- Loader helpers that enforce explicit eager loading with `lazy="raise"` relationships.
+
+### Provider Layer
+
 - `src/transcription/providers/**`
 
 Responsibilities:
 
-- Provide async database sessions and engine configuration.
-- Provide provider adapters for vision model execution.
+- Provider API encapsulation.
+- Request manifest and transport evidence capture.
+- Normalized transcription result contract.
 
-## Core Workflows
+## Core Domain Model
 
-### 1. Multi-Page Transcription
+- `Document` owns archival metadata and links to `Source`, `Job`, and `DocumentPerson`.
+- `Source` is a document page/file record with selected machine projection and human revision.
+- `Job` is an aggregate processing run with status and frozen prompt/runtime settings.
+- `JobSource` is queue/membership state for one `(job, source)` pair.
+- `ExecutionAttempt` is append-only evidence for each provider call.
+- `DocumentType` and `PersonRole` are UUID-backed registries with optional protected `semantic_key`.
 
-1. User uploads one or more images for a `Document`.
-2. System stores files, hashes them, creates ordered `Source` rows, and creates a `Job`.
-3. Worker claims the job, marks it `processing`, resolves metadata-directed orientation, and sends either the
-   immutable original or an exact normalized derivative to the provider.
-4. Each provider call appends an `ExecutionAttempt` with its request manifest, transport evidence, SDK snapshot,
-   normalized metadata, timing, and outcome.
-5. The linked `JobSource` is updated as a compatibility projection. The first successful attempt establishes
-   `Source.preferred_execution_attempt_id` and `Source.raw_transcription`; later successes remain candidates.
-6. Aggregate status becomes `completed`, `partial_success`, or `failed`.
+## Processing and Evidence Workflow
 
-### 2. Document-Person Relationship Management
+1. User creates/updates Document metadata and linked People atomically through workflow orchestration.
+2. User creates a Job by uploading one or more Source files or by retranscribing an existing Source.
+3. Source files are validated and stored; orientation normalization may be applied at ingest, and stored bytes become the canonical processing bytes.
+4. Worker claims queued Job, transitions to `processing`, and processes pending pages in deterministic order.
+5. Each provider call writes one immutable `ExecutionAttempt` with:
+   - request manifest + hash
+   - transport evidence (when response exists)
+   - SDK snapshot and normalized metadata
+   - outcome, timing, and error details when applicable
+6. `JobSource` status is updated as queue/projection state; `Source.raw_transcription` is set on first successful attempt and can be explicitly re-pointed by candidate promotion.
+7. Job terminal status resolves to `transcribed`, `partial_success`, or `failed`.
 
-1. User opens Document Create or Edit.
-2. UI loads one Linked People table containing Person and Role.
-3. Add, Edit, and Delete operations change staged UI state only.
-4. Service validates the complete desired set and computes deterministic add, update, and remove deltas.
-5. Document fields and links commit once in one transaction; any failure leaves both unchanged.
+## Status Semantics
 
-### 3. Document Type Management
+- **Job statuses:** `queued`, `processing`, `transcribed`, `completed`, `partial_success`, `failed`
+  - Operational success path currently resolves to `transcribed`.
+  - `completed` remains a recognized legacy-compatible status value.
+- **JobSource statuses:** `pending`, `transcribed`, `failed`, `cancelled`
 
-1. User selects a registry-backed document type for a document.
-2. Service resolves the Document Type UUID.
-3. Persistence stores the `document_type_id` reference.
-4. Inactive types remain valid for historical rows but are excluded from default selectors.
+## Security and Path Handling Boundaries
 
-### 4. Document Printing
+- Print media delivery uses record-validated API route:
+  - `src/transcription/api/v4_print.py`
+- General UI media links resolve through:
+  - `src/transcription/ui/components/media_urls.py`
+- Local filesystem paths must never be accepted from user input as trusted media routes.
 
-1. User opens Print from persisted Document Detail.
-2. Service builds a safe projection containing archival metadata, semantic Author links, ordered Sources, current text,
-   and oldest-to-newest Job metadata.
-3. The preview renders Facsimile or Text-only HTML without exposing local file paths.
-4. An explicit action opens the browser print dialog; browser Save as PDF remains available.
+## Concurrency and Reliability Principles
 
-## V4 Domain Rules
+- Worker loop reuses service bundle/provider resources for pooled calls.
+- Provider-call timeout is explicit and bounded.
+- Non-retriable worker-loop faults are surfaced and stop loop spin.
+- Per-page outcomes are durably persisted before processing next page.
 
-- `JobSource.raw_transcription` preserves page output for its Job execution.
-- `Source.raw_transcription` is the selected preferred-machine-output projection for a page.
-- `Source.preferred_execution_attempt_id` identifies its exact immutable provenance; candidate promotion updates
-  both fields atomically.
-- Human corrections occur only in `Source.revised_text`.
-- Prompt and parameter provenance is frozen on `Job` at submission time.
-- The SDK-serialized OpenRouter response snapshot is stored on `JobSource` for each successful page execution.
-- Every V4.2 provider call appends a distinct `ExecutionAttempt`; retries never rewrite earlier attempts.
-- Exact response bytes identify the OpenRouter HTTP boundary and are not labeled as native upstream-provider JSON.
-- Generic `ProcessingArtifact` records use versioned schemas, digests, and one inline or external content location.
-- Orientation-normalized model inputs and deterministic quality warnings are versioned `ProcessingArtifact` evidence
-  attached to the consuming `ExecutionAttempt`.
-- A `retranscription` Job contains one locked existing Source and freezes one configured allowlisted model.
-- `DocumentPerson` links are unique for `(document_id, person_id)` and require one `role_id`.
-- Relationship mutations are deterministic, set-based, and atomic with Document writes.
-- `DocumentType.id` and `PersonRole.id` are canonical relationship identities; unique labels may evolve.
-- Nullable immutable `semantic_key` values identify protected application-defined built-ins and are never public selectors.
-- Current printable text uses non-null `Source.revised_text`; otherwise it uses `Source.raw_transcription`.
+## Related References
 
-## Data Model Summary
-
-- `Document` has one `DocumentType`, many `Source` pages, many `Job` runs, and many `Person` records through `DocumentPerson`.
-- `Source` belongs to one `Document` and may participate in many `JobSource` executions.
-- `Job` has many `JobSource` rows.
-- `PersonRole` defines available relationship roles; `DocumentType` and `PersonRole` may carry hidden semantic identity.
-
-## Test Strategy
-
-- Unit tests for models, validation, hashing, and registry resolution.
-- Service tests for registry protection, atomic link synchronization, uniqueness conflicts, and print projections.
-- Async workflow tests for page isolation, partial failure handling, and stored evidence.
-- UI integration tests for Linked People staging, registry selection, and safe print rendering.
-
-## Related Local References
-
-- [System Overview](index_v4.md)
 - [System Requirements](requirements_v4.md)
 - [Data Model](schema_v4.md)
 - [Error Handling Policy](error_handling_v4.md)
-- [Error Handling Invariant](../invariant/error_handling.md)
-- [Digital Evidence and AI Processing Provenance](../invariant/ai_evidence_and_provenance.md)
+- [V4 Revision History](history.md)
+- [Error Handling invariant](../invariant/error_handling.md)
+- [AI evidence invariant](../invariant/ai_evidence_and_provenance.md)