gpt-5.3-codex review phase 1 - Flatten the documentation

docs/ver4/architecture_v4.md

@@ -1,75 +1,36 @@
 # System Architecture (Version 4)
 
-This document describes the production architecture of the document transcription system.
+This document defines the current V4 architecture baseline.
 
 ## Architecture Objectives
 
-- Preserve original source material, per-execution machine output, and separate human revision.
-- Support batching one or more images into ordered multi-page documents.
-- Capture submission-time prompt provenance and a per-page OpenRouter SDK response snapshot.
-- Execute page transcription concurrently with bounded `asyncio` workers.
-- Maintain relational portability across SQLite and PostgreSQL.
-- Keep operator workflows cross-platform and Python-driven.
-- Support one role-bearing link per Person and Document through an extensible role registry.
-- Support registry-driven document classification with protected semantic built-ins.
-
-## Core Capabilities
-
-- Ingest one or more images into sequential `Source` pages under a `Document`.
-- Execute asynchronous vision transcription with bounded worker concurrency.
-- Preserve original source files with SHA-256 digests and byte sizes.
-- Freeze prompt text, prompt hash, model, and explicitly configured sampling parameters on each `Job`.
-- Preserve page-level machine output, normalized metadata, and an SDK-serialized OpenRouter response snapshot on `JobSource`.
-- Organize historical `Person` records through UUID-identified Document links and extensible roles.
-- Classify Documents through a UUID-identified registry with hidden semantic built-ins and unique labels.
-- Maintain human revision separately from machine-generated text.
-- Isolate page failures so multi-page jobs can complete with partial success.
-- Operate across supported platforms through Python-based application and maintenance tooling.
-
-V4.2 extends this baseline with immutable execution attempts, exact OpenRouter transport evidence, safe
-versioned exports, and provider-neutral derived-artifact provenance. `JobSource` remains the mutable queue and
-compatibility projection; `ExecutionAttempt` is the authoritative append-only processing history. See the
-[V4.2 Scope Boundary](../ver4.2/scope_boundary_v4_2.md).
+- Preserve durable archival records for Documents, Sources, People, and processing runs.
+- Execute page transcription asynchronously with bounded worker behavior.
+- Preserve append-only machine-attempt evidence with request/response provenance.
+- Keep UI, API, service, persistence, and provider boundaries explicit and testable.
 
 ## Technical Stack
 
-- **Runtime:** Python 3.12 or later.
-- **Web application:** FastAPI and NiceGUI.
-- **Persistence:** SQLModel and SQLAlchemy, with SQLite and PostgreSQL support.
-- **Validation and settings:** Pydantic V2 and pydantic-settings.
-- **Concurrency:** Python `asyncio` workers.
-- **Vision integration:** OpenRouter through the application's provider adapter.
-- **Testing and quality:** pytest, pytest-asyncio, Ruff, and ty.
+- **Runtime:** Python 3.12+
+- **Web application:** FastAPI + NiceGUI
+- **Persistence:** SQLModel / SQLAlchemy (SQLite-first, PostgreSQL-compatible model design)
+- **Validation and settings:** Pydantic V2 + pydantic-settings
+- **Concurrency:** asyncio worker loop
+- **Provider integration:** OpenRouter adapter behind provider interface
+- **Quality and tests:** Ruff, ty, pytest, pytest-asyncio
 
 ## Runtime Topology
 
-The runtime operates as an asynchronous Python application:
-
-- FastAPI + NiceGUI web application process.
-- In-process `asyncio` worker engine for transcription execution.
-- Relational persistence via SQLModel / SQLAlchemy.
-- Pydantic V2 validation across API payloads, prompt configuration, and structured metadata.
-
-^^^mermaid
+```mermaid
 flowchart LR
 U[Browser User] --> A[FastAPI + NiceGUI App]
-A --> W[Asyncio Worker Engine]
-A --> DB[(Relational DB)]
-W --> P[Vision Provider APIs]
+A --> W[Asyncio Worker]
+A --> DB[(SQLite/PostgreSQL Model)]
+W --> P[Provider Adapter]
 W --> DB
-^^^
+```
 
-## Lifecycle Ownership
-
-Application lifespan owns runtime setup and teardown:
-
-- Initialize logging, settings, directories, and prompt configuration.
-- Manage asynchronous database engine connection pools.
-- Execute database bootstrap or migrations.
-- Recover stale or interrupted jobs on startup.
-- Manage graceful shutdown of active background tasks.
-
-## Layered Module Structure
+## Layered Boundaries
 
 ### Interface Layer
 
@@ -78,136 +39,95 @@ Application lifespan owns runtime setup and teardown:
 
 Responsibilities:
 
-- Render document, source, person, job, and classification views.
-- Accept user input for uploads, editing, linking, and revisions.
-- Present structured validation and conflict feedback.
+- Route registration, page orchestration, presentation adapters.
+- Structured user messaging through shared error presenter.
+- No direct persistence access from pages/components.
 
-### Application and Async Worker Layer
+### Service and Orchestration Layer
 
-- `src/transcription/services/workflows.py`
-- `src/transcription/worker.py`
-
-Responsibilities:
-
-- Orchestrate uploads, job creation, and status transitions.
-- Execute per-page provider calls through bounded concurrency.
-- Persist page-level outcomes and update aggregate job state.
-
-### Domain and Service Layer
-
-- `src/transcription/db/models.py`
 - `src/transcription/services/documents.py`
-- `src/transcription/services/sources.py`
-- `src/transcription/services/jobs.py`
 - `src/transcription/services/people.py`
+- `src/transcription/services/jobs.py`
+- `src/transcription/services/sources.py`
+- `src/transcription/services/evidence.py`
+- `src/transcription/services/store.py`
 - `src/transcription/services/workflows.py`
 
 Responsibilities:
 
-- Keep one primary service boundary per aggregate: Documents, Sources, Jobs, and People.
-- Documents own document records and the document-type registry.
-- Sources own source records, revisions, source media formats, MIME resolution, and page execution evidence.
-- Jobs own job lifecycle state and transitions.
-- People own person records, relationship roles, document-person links, and portrait media.
-- Apply deterministic conflict handling for relationship-role writes.
-- Synchronize each Document's complete Person link set in the same transaction as Document fields.
-- Resolve and validate registry records by UUID; use hidden semantic keys only for application-owned built-in behavior.
+- Aggregate ownership and invariants.
+- Transaction-aware write helpers.
+- Cross-service workflows in orchestration modules (`store.py`, `workflows.py`).
 
-### Source Media Policy
-
-- `services/sources.py` is the single authority for accepted Source extensions and canonical MIME types.
-- Storage and provider payload loading must call the same Source validation functions.
-- Supported Source formats are JPEG, PNG, TIFF, and PDF.
-- Upload is an interface action, not a domain aggregate. Service names, errors, and workflow variables use
-  `Source` terminology; compatibility aliases may remain temporarily at old import boundaries.
-
-### Infrastructure Layer
+### Persistence Layer
 
 - `src/transcription/db/**`
+
+Responsibilities:
+
+- SQLModel definitions, async session/engine runtime, registry bootstrap.
+- Loader helpers that enforce explicit eager loading with `lazy="raise"` relationships.
+
+### Provider Layer
+
 - `src/transcription/providers/**`
 
 Responsibilities:
 
-- Provide async database sessions and engine configuration.
-- Provide provider adapters for vision model execution.
+- Provider API encapsulation.
+- Request manifest and transport evidence capture.
+- Normalized transcription result contract.
 
-## Core Workflows
+## Core Domain Model
 
-### 1. Multi-Page Transcription
+- `Document` owns archival metadata and links to `Source`, `Job`, and `DocumentPerson`.
+- `Source` is a document page/file record with selected machine projection and human revision.
+- `Job` is an aggregate processing run with status and frozen prompt/runtime settings.
+- `JobSource` is queue/membership state for one `(job, source)` pair.
+- `ExecutionAttempt` is append-only evidence for each provider call.
+- `DocumentType` and `PersonRole` are UUID-backed registries with optional protected `semantic_key`.
 
-1. User uploads one or more images for a `Document`.
-2. System stores files, hashes them, creates ordered `Source` rows, and creates a `Job`.
-3. Worker claims the job, marks it `processing`, resolves metadata-directed orientation, and sends either the
-   immutable original or an exact normalized derivative to the provider.
-4. Each provider call appends an `ExecutionAttempt` with its request manifest, transport evidence, SDK snapshot,
-   normalized metadata, timing, and outcome.
-5. The linked `JobSource` is updated as a compatibility projection. The first successful attempt establishes
-   `Source.preferred_execution_attempt_id` and `Source.raw_transcription`; later successes remain candidates.
-6. Aggregate status becomes `completed`, `partial_success`, or `failed`.
+## Processing and Evidence Workflow
 
-### 2. Document-Person Relationship Management
+1. User creates/updates Document metadata and linked People atomically through workflow orchestration.
+2. User creates a Job by uploading one or more Source files or by retranscribing an existing Source.
+3. Source files are validated and stored; orientation normalization may be applied at ingest, and stored bytes become the canonical processing bytes.
+4. Worker claims queued Job, transitions to `processing`, and processes pending pages in deterministic order.
+5. Each provider call writes one immutable `ExecutionAttempt` with:
+   - request manifest + hash
+   - transport evidence (when response exists)
+   - SDK snapshot and normalized metadata
+   - outcome, timing, and error details when applicable
+6. `JobSource` status is updated as queue/projection state; `Source.raw_transcription` is set on first successful attempt and can be explicitly re-pointed by candidate promotion.
+7. Job terminal status resolves to `transcribed`, `partial_success`, or `failed`.
 
-1. User opens Document Create or Edit.
-2. UI loads one Linked People table containing Person and Role.
-3. Add, Edit, and Delete operations change staged UI state only.
-4. Service validates the complete desired set and computes deterministic add, update, and remove deltas.
-5. Document fields and links commit once in one transaction; any failure leaves both unchanged.
+## Status Semantics
 
-### 3. Document Type Management
+- **Job statuses:** `queued`, `processing`, `transcribed`, `completed`, `partial_success`, `failed`
+  - Operational success path currently resolves to `transcribed`.
+  - `completed` remains a recognized legacy-compatible status value.
+- **JobSource statuses:** `pending`, `transcribed`, `failed`, `cancelled`
 
-1. User selects a registry-backed document type for a document.
-2. Service resolves the Document Type UUID.
-3. Persistence stores the `document_type_id` reference.
-4. Inactive types remain valid for historical rows but are excluded from default selectors.
+## Security and Path Handling Boundaries
 
-### 4. Document Printing
+- Print media delivery uses record-validated API route:
+  - `src/transcription/api/v4_print.py`
+- General UI media links resolve through:
+  - `src/transcription/ui/components/media_urls.py`
+- Local filesystem paths must never be accepted from user input as trusted media routes.
 
-1. User opens Print from persisted Document Detail.
-2. Service builds a safe projection containing archival metadata, semantic Author links, ordered Sources, current text,
-   and oldest-to-newest Job metadata.
-3. The preview renders Facsimile or Text-only HTML without exposing local file paths.
-4. An explicit action opens the browser print dialog; browser Save as PDF remains available.
+## Concurrency and Reliability Principles
 
-## V4 Domain Rules
+- Worker loop reuses service bundle/provider resources for pooled calls.
+- Provider-call timeout is explicit and bounded.
+- Non-retriable worker-loop faults are surfaced and stop loop spin.
+- Per-page outcomes are durably persisted before processing next page.
 
-- `JobSource.raw_transcription` preserves page output for its Job execution.
-- `Source.raw_transcription` is the selected preferred-machine-output projection for a page.
-- `Source.preferred_execution_attempt_id` identifies its exact immutable provenance; candidate promotion updates
-  both fields atomically.
-- Human corrections occur only in `Source.revised_text`.
-- Prompt and parameter provenance is frozen on `Job` at submission time.
-- The SDK-serialized OpenRouter response snapshot is stored on `JobSource` for each successful page execution.
-- Every V4.2 provider call appends a distinct `ExecutionAttempt`; retries never rewrite earlier attempts.
-- Exact response bytes identify the OpenRouter HTTP boundary and are not labeled as native upstream-provider JSON.
-- Generic `ProcessingArtifact` records use versioned schemas, digests, and one inline or external content location.
-- Orientation-normalized model inputs and deterministic quality warnings are versioned `ProcessingArtifact` evidence
-  attached to the consuming `ExecutionAttempt`.
-- A `retranscription` Job contains one locked existing Source and freezes one configured allowlisted model.
-- `DocumentPerson` links are unique for `(document_id, person_id)` and require one `role_id`.
-- Relationship mutations are deterministic, set-based, and atomic with Document writes.
-- `DocumentType.id` and `PersonRole.id` are canonical relationship identities; unique labels may evolve.
-- Nullable immutable `semantic_key` values identify protected application-defined built-ins and are never public selectors.
-- Current printable text uses non-null `Source.revised_text`; otherwise it uses `Source.raw_transcription`.
+## Related References
 
-## Data Model Summary
-
-- `Document` has one `DocumentType`, many `Source` pages, many `Job` runs, and many `Person` records through `DocumentPerson`.
-- `Source` belongs to one `Document` and may participate in many `JobSource` executions.
-- `Job` has many `JobSource` rows.
-- `PersonRole` defines available relationship roles; `DocumentType` and `PersonRole` may carry hidden semantic identity.
-
-## Test Strategy
-
-- Unit tests for models, validation, hashing, and registry resolution.
-- Service tests for registry protection, atomic link synchronization, uniqueness conflicts, and print projections.
-- Async workflow tests for page isolation, partial failure handling, and stored evidence.
-- UI integration tests for Linked People staging, registry selection, and safe print rendering.
-
-## Related Local References
-
-- [System Overview](index_v4.md)
 - [System Requirements](requirements_v4.md)
 - [Data Model](schema_v4.md)
 - [Error Handling Policy](error_handling_v4.md)
-- [Error Handling Invariant](../invariant/error_handling.md)
-- [Digital Evidence and AI Processing Provenance](../invariant/ai_evidence_and_provenance.md)
+- [V4 Revision History](history.md)
+- [Error Handling invariant](../invariant/error_handling.md)
+- [AI evidence invariant](../invariant/ai_evidence_and_provenance.md)

docs/ver4/error_handling_v4.md

@@ -1,115 +1,58 @@
 # Error Handling Policy (Version 4)
 
-This document defines the Version 4 taxonomy, contracts, and framework behavior used to satisfy the cross-version [Error Handling invariant](../invariant/error_handling.md).
+This policy defines active V4 error taxonomy, translation boundaries, and retry semantics.
 
-## Invariant Alignment
+## Error Categories
 
-Version 4 implements the invariant through:
+| Category | Meaning | Typical Origin | User Treatment |
+| :--- | :--- | :--- | :--- |
+| `validation` | Input payload/selection is invalid | UI form parsing, service validators | Inline correction guidance |
+| `not_found` | Target record is missing | ID lookup in service layer | Non-blocking warning or redirect |
+| `conflict` | State prevents requested action | lifecycle transitions, duplicate semantic keys | Explain required precondition |
+| `external` | Provider/network dependency failure | OpenRouter/provider adapter | Retry path and evidence retained |
+| `timeout` | Provider call exceeded configured bound | worker/provider client timeout | Retry path and bounded messaging |
+| `internal` | Unexpected local failure | unhandled service/runtime faults | Safe generic message + diagnostics capture |
 
-- The shared error taxonomy below.
-- Structured error envelopes with correlation IDs.
-- Page-level failure isolation and explicit aggregate job status.
-- Atomic relationship and classification writes.
-- Consistent translation across API, UI, service, worker, persistence, and provider boundaries.
-- Bounded retry guidance based on category and idempotency.
+## Translation Boundaries
 
-## Scope and Authority
+- **Provider layer:** raise provider-scoped exceptions with provider context; do not emit UI text.
+- **Service layer:** map raw exceptions into domain-aware categories and preserve causal chain.
+- **UI/API layer:** convert category to user-safe message with contextual action guidance.
 
-This policy governs error behavior across:
+## Job and Page Failure Semantics
 
-- NiceGUI pages
-- FastAPI routes
-- Service-layer orchestration
-- `asyncio` worker tasks
-- Database interactions
-- Provider adapters
+### Page-Level (`JobSource`)
 
-## Error Taxonomy
+- `pending` -> `transcribed` when attempt succeeds.
+- `pending` -> `failed` when attempt fails terminally.
+- `pending` -> `cancelled` on job cancellation before processing.
 
-| Category | Definition | Retriable |
-| --- | --- | --- |
-| `validation_error` | Payload, parameter, or schema validation failure | no |
-| `user_input_error` | Unacceptable file, invalid selection, or malformed request from the operator | no |
-| `not_found_error` | Requested `Document`, `Source`, `Person`, `Job`, role, or type does not exist | no |
-| `conflict_error` | Operation violates uniqueness or relationship-write policy | no |
-| `external_provider_error` | Provider API failure, rate limit, or execution problem | yes |
-| `infrastructure_transient_error` | Temporary DB, file-system, or network instability | yes |
-| `infrastructure_persistent_error` | Persistent configuration, credential, or database availability failure | no |
-| `internal_unexpected_error` | Uncaught exception or logic defect | no |
+### Job-Level (`Job`)
 
-## Async Batch and Page-Level Error Behavior
+- `transcribed` when all pages transcribe successfully.
+- `partial_success` when mixed success/failure outcomes exist.
+- `failed` when no page transcribes successfully.
 
-In multi-page `asyncio` processing:
+## Retry and Retranscription Rules
 
-1. Exceptions from individual page calls are trapped within the page task wrapper.
-2. Failed page detail is written to `JobSource.error_detail` and the page state becomes `failed`.
-3. Aggregate job status is derived from page outcomes:
-   - all pages succeed -> `completed`
-   - some succeed and some fail -> `partial_success`
-   - all fail -> `failed`
-4. Successful pages remain valid even when sister pages fail.
+1. Failed/cancelled pages may be re-queued through retranscription workflows.
+2. Retry attempts must append new `ExecutionAttempt` rows; prior evidence remains immutable.
+3. Selecting a better candidate must update projection pointers, not mutate historical attempt rows.
 
-## Relationship and Classification Conflict Behavior
+## Logging and Diagnostics Rules
 
-When relationship or document-type writes fail policy checks:
+1. Persist sufficient attempt error metadata (`error_category`, `error_message`, transport evidence) for post-hoc analysis.
+2. Avoid leaking stack traces or local paths into user-facing message envelopes.
+3. Preserve causal exception chains for internal diagnostics.
 
-1. Reject the full write operation.
-2. Return structured conflict detail including target identifiers and the violated rule.
-3. Preserve existing persisted relationships unchanged.
+## UI Messaging Contract
 
-## API Error Response Contract
+- User-visible errors must be actionable, bounded, and category-consistent.
+- Multi-page jobs must show partial outcomes instead of collapsing into a single opaque failure.
+- Recovery actions (`retry`, `retranscribe`, `edit input`) must be offered where available.
 
-API error responses return a structured envelope:
+## Cross-Reference
 
-^^^json
-{
-  "error_id": "err_uuid_12345",
-  "category": "conflict_error",
-  "message": "Relationship write conflicts with existing links.",
-  "suggestion": "Adjust the requested relationship links and retry.",
-  "details": {
-    "document_id": "...",
-    "person_id": "...",
-    "attempted_role": "recipient",
-    "operation": "add_link",
-    "conflict_reason": "duplicate document-person-role link"
-  },
-  "timestamp": "2026-08-10T15:00:00Z"
-}
-^^^
-
-HTTP status mappings:
-
-- `validation_error`, `user_input_error` -> `400`
-- `not_found_error` -> `404`
-- `conflict_error` -> `409`
-- `external_provider_error` -> `502` or `503`
-- `infrastructure_transient_error` -> `503`
-- `infrastructure_persistent_error`, `internal_unexpected_error` -> `500`
-
-## UI Error Presentation Rules
-
-- Display concise failure summaries with the next action the operator can take.
-- Keep form state in context when feasible.
-- Distinguish validation issues, conflict issues, provider failures, and infrastructure failures.
-- For bulk relationship updates, identify the specific role or person that caused a conflict.
-
-## Logging and Audit Expectations
-
-- Log worker failures with correlation IDs and provider context.
-- Log relationship and classification conflicts with machine-readable detail.
-- Log persisted provider errors and page-level execution failures.
-
-## Retry Guidance
-
-- Do not auto-retry validation or conflict failures.
-- Permit user-driven retry after the input or selection changes.
-- Allow bounded retry for transient provider or infrastructure failures when the operation is idempotent.
-
-## Related Local References
-
-- [Error Handling Invariant](../invariant/error_handling.md)
-- [System Overview](index_v4.md)
+- [Error Handling invariant](../invariant/error_handling.md)
 - [System Requirements](requirements_v4.md)
 - [Data Model](schema_v4.md)
-- [System Architecture](architecture_v4.md)

docs/ver4/history.md

@@ -0,0 +1,33 @@
+# V4 Revision History (Archived)
+
+This index tracks superseded V4.x documents as historical context.
+These revisions were intentionally removed from the active working tree to prevent accidental reuse by tools and AI agents.
+They are preserved at immutable git tag `docs-v4x-archive`.
+
+## Archive Map
+
+| Revision | Archived Paths at `docs-v4x-archive` |
+| :--- | :--- |
+| V4.0 | `docs/ver4.0/scope_boundary_v4.md`, `docs/ver4.0/implementation_plan_v4.md` |
+| V4.1 | `docs/ver4.1/scope_boundary_v4_1.md`, `docs/ver4.1/implementation_plan_v4_1.md` |
+| V4.2 | `docs/ver4.2/scope_boundary_v4_2.md`, `docs/ver4.2/implementation_plan_v4_2.md` |
+| V4.3 | `docs/ver4.3/scope_boundary_v4_3.md`, `docs/ver4.3/implementation_plan_v4_3.md` |
+| V4.4 | `docs/ver4.4/scope_boundary_v4_4.md`, `docs/ver4.4/implementation_plan_v4_4.md` |
+| V4.5 | `docs/ver4.5/scope_boundary_v4_5.md`, `docs/ver4.5/implementation_plan_v4_5.md` |
+| V4.6 | `docs/ver4.6/scope_boundary_v4_6.md`, `docs/ver4.6/implementation_plan_v4_6.md`, `docs/ver4.6/review_log_v4_6.md` |
+| V4.7 | `docs/ver4.7/scope_boundary_v4_7.md`, `docs/ver4.7/implementation_plan_v4_7.md`, `docs/ver4.7/review_log_v4_7.md` |
+| V4.8 | `../ver4.8/feature_backlog_v4_8.md` |
+
+To inspect archived content locally:
+
+`git show docs-v4x-archive:docs/ver4.6/scope_boundary_v4_6.md`
+
+## Canonical Contract Reminder
+
+Use `docs/ver4/` for current-state requirements, architecture, schema, and error policy:
+
+- `index_v4.md`
+- `architecture_v4.md`
+- `requirements_v4.md`
+- `schema_v4.md`
+- `error_handling_v4.md`

docs/ver4/index_v4.md

@@ -1,13 +1,14 @@
 # Document Transcription System Overview (Version 4)
 
-Version 4 is the architecture baseline for the personal-scale application used to transcribe, organize, and preserve historical documents, source images, and related people records.
+This directory is the single source of truth for current V4 behavior and architecture.
 
-## Recommended Reading Order
+## Canonical Reading Order
 
-1. [System Architecture](architecture_v4.md) for capabilities, technical stack, runtime structure, workflows, and component ownership.
-2. [System Requirements](requirements_v4.md) for the verifiable V4 contract.
-3. [Data Model](schema_v4.md) for entities, relationships, constraints, and persistence rules.
-4. [Error Handling Policy](error_handling_v4.md) for the V4 taxonomy and boundary contracts.
+1. [System Architecture](architecture_v4.md) for runtime topology, boundaries, and lifecycle ownership.
+2. [System Requirements](requirements_v4.md) for verifiable current-state requirements.
+3. [Data Model](schema_v4.md) for entities, constraints, and evidence persistence rules.
+4. [Error Handling Policy](error_handling_v4.md) for category, translation, and retry behavior.
+5. [V4 Revision History](history.md) for superseded scope and implementation documents.
 
 ## Cross-Version Invariants
 
@@ -17,16 +18,7 @@ Version 4 is the architecture baseline for the personal-scale application used t
 - [Digital Evidence and AI Processing Provenance](../invariant/ai_evidence_and_provenance.md)
 - [UI Style Guide](../invariant/ui_style_guide.md)
 
-## V4 Transition Documents
+## Baseline Statement
 
-- [Scope Boundary](scope_boundary_v4.md)
-- [Implementation Plan](implementation_plan_v4.md)
-
-## Incremental Revisions
-
-- [V4.1 Scope](../ver4.1/scope_boundary_v4_1.md) and [Implementation Plan](../ver4.1/implementation_plan_v4_1.md)
-- [V4.2 Evidence and Provenance Scope](../ver4.2/scope_boundary_v4_2.md) and [Implementation Plan](../ver4.2/implementation_plan_v4_2.md)
-- [V4.3 Settings Scope](../ver4.3/scope_boundary_v4_3.md) and [Implementation Plan](../ver4.3/implementation_plan_v4_3.md)
-- [V4.4 Semantic Registries, Linked People, and Printing Scope](../ver4.4/scope_boundary_v4_4.md) and [Implementation Plan](../ver4.4/implementation_plan_v4_4.md)
-- [V4.5 Transcription Input Normalization and Quality Scope](../ver4.5/scope_boundary_v4_5.md) and [Implementation Plan](../ver4.5/implementation_plan_v4_5.md)
-- [V4.6 Architecture Conformance and Reliability Scope](../ver4.6/scope_boundary_v4_6.md) and [Implementation Plan](../ver4.6/implementation_plan_v4_6.md)
+The current V4 baseline includes behavior delivered through V4.7 architectural cleanup.
+Versioned V4.x scope and implementation documents are retained as historical records only at git tag `docs-v4x-archive` and do not define active contracts.

docs/ver4/requirements_v4.md

@@ -1,60 +1,61 @@
-# Document Transcription System Requirements (Version 4)
+# System Requirements (Version 4)
 
-This document defines the baseline requirements for the document transcription system.
+These requirements define the active V4 contract and align to current implementation.
 
-## Requirements Model
+## Functional Requirements
 
-| ID | Category | Requirement | Verify Method |
-| --- | --- | --- | --- |
-| REQ-0 | System | Provide end-to-end multi-page document transcription with persistent, inspectable async job states. | demonstration |
-| REQ-1 | Functional | Allow users to upload one or more images as ordered `Source` pages under a `Document`. | test |
-| REQ-2 | Functional | Process page transcription asynchronously using an `asyncio` worker pool bounded by rate limits. | test |
-| REQ-3 | Functional | Persist submission-time request provenance and accurately labeled page-level SDK evidence; V4.2 adds exact OpenRouter-boundary transport evidence for new attempts. | test |
-| REQ-4 | Functional | Support job states `queued`, `processing`, `completed`, `partial_success`, and `failed`, plus page states `pending`, `transcribed`, and `failed`. | inspection |
-| REQ-5 | Functional | Allow users to manage historical `Person` records and link each Person to a Document once with exactly one role. | test |
-| REQ-6 | Functional | Support an extensible role taxonomy for document-person relationships. | inspection |
-| REQ-7 | Policy Constraint | Enforce deterministic relationship-role writes with uniqueness on `(document_id, person_id)` and explicit conflict responses for duplicate Person links. | test |
-| REQ-8 | Functional | Use set-based synchronization for document-person mutations so updates add and remove only the intended links. | test |
-| REQ-9 | Functional | Maintain selected machine output and exact attempt provenance on `Source` while permitting independent human edits on `Source.revised_text`. | test |
-| REQ-10 | Functional | Support a UUID-identified `DocumentType` taxonomy with unique user-facing labels and active/inactive lifecycle control. | test |
-| REQ-11 | Data Constraint | Store `Document` type as a controlled reference to `DocumentType`. | test |
-| REQ-12 | Interface | Render multi-page transcriptions sequentially by `page_number` with document, people, and document-type metadata. | demonstration |
-| REQ-13 | Interface | Document create/edit UI must provide one staged Linked People table and select active registry entries by UUID and label. | demonstration |
-| REQ-14 | API Constraint | Expose additive, role-aware retrieval and write behavior for document-person links and UUID-based selection for document types. | test |
-| REQ-15 | Data Constraint | Calculate and store cryptographic file hashes (SHA-256) and file sizes for uploaded source images. | test |
-| REQ-16 | Data Constraint | Preserve a portable relational model across supported backends using SQLModel, SQLAlchemy, SQLite, and PostgreSQL. | inspection |
-| REQ-17 | Reliability | Ensure delete and update flows for documents, people, and relationship links remain deterministic and safe. | test |
-| REQ-18 | Operations Constraint | Keep canonical development, testing, restore, and recovery workflows OS-independent; for AI-run unit tests, require a pre-test backup of `./data` and an always-shown post-success confirmation prompt before any restore action. | inspection |
-| REQ-19 | Quality | Provide automated coverage for async transcription workflows, relationship-role enforcement, document-type selection, and regression behavior. | test |
-| REQ-20 | Data Constraint | Permit hidden immutable semantic keys only on protected built-in Document Types and Person Roles while retaining UUID as relationship identity. | test |
-| REQ-21 | Reliability | Persist Document fields and their complete Linked People set atomically. | test |
-| REQ-22 | Interface | Provide safe browser-native Facsimile and Text-only print views from persisted Document Detail. | demonstration |
-| REQ-23 | Security | Escape stored print text and serve Source images through record-validated application routes without disclosing local paths. | test |
-| REQ-24 | Functional | Print current human-preferred Source text, semantic Author metadata, deterministic Source order, and oldest-to-newest Job metadata. | test |
-| REQ-25 | Quality | Physically apply recognized raster orientation metadata to provider-input derivatives without changing original Source bytes. | test |
-| REQ-26 | Quality | Persist deterministic, non-mutating output warnings without automatic paid retries. | test |
-| REQ-27 | Functional | Create one-Source retranscription Jobs from a configured model allowlist and preserve later successes as candidates until explicit promotion. | test |
+### Domain and Record Management
 
-## Clarifying Constraints
+- **REQ-4-001 Document Registry:** The system must create and update `Document` records with title, type, language, comments, date metadata, and optional location.
+- **REQ-4-002 Source Registry:** The system must create and update `Source` records linked to exactly one `Document`.
+- **REQ-4-003 People Registry:** The system must create and update `Person` records and support many-to-many links to `Document` with role and confidence.
+- **REQ-4-004 Registry Semantics:** Document types and person roles must support optional immutable semantic keys and hard-delete only when unreferenced.
 
-1. `DocumentType.id` and `PersonRole.id` are their public and relationship identities; labels are unique ignoring case and surrounding whitespace.
-2. Nullable `semantic_key` values identify protected application built-ins, remain internal, and never change.
-3. Relationship-write policy and conflict handling must be consistent across UI, API, services, and persistence.
-4. One Person may appear only once per Document and every link has exactly one role.
-5. Relationship conflicts must fail deterministically without partial Document or link mutation.
-6. Source page reordering and server-generated PDF files remain outside this revision.
+### Job and Workflow Behavior
 
-## Element Satisfaction Mapping
+- **REQ-4-010 Job Creation:** The system must create `Job` records from uploaded sources and from retranscription of existing sources.
+- **REQ-4-011 Prompt Snapshotting:** Job creation must persist effective prompt and runtime settings as immutable per-job snapshots.
+- **REQ-4-012 Queue Membership:** Each `(job, source)` pair must be represented by one `JobSource` row.
+- **REQ-4-013 Job Status Lifecycle:** `Job.status` must use one of `queued`, `processing`, `transcribed`, `completed`, `partial_success`, `failed`.
+- **REQ-4-014 JobSource Status Lifecycle:** `JobSource.status` must use one of `pending`, `transcribed`, `failed`, `cancelled`.
+- **REQ-4-015 Terminal Job Resolution:** Job terminal status must derive from page outcomes as `transcribed`, `partial_success`, or `failed`.
+- **REQ-4-016 Cancellation Semantics:** Job cancellation must set remaining `pending` page entries to `cancelled`.
 
-- UI (NiceGUI): Satisfies REQ-0, REQ-1, REQ-5, REQ-9, REQ-12, REQ-13, REQ-22, REQ-24.
-- API (FastAPI): Satisfies REQ-1, REQ-4, REQ-5, REQ-7, REQ-8, REQ-14, REQ-23.
-- Worker (`asyncio`): Satisfies REQ-2, REQ-3, REQ-4.
-- Persistence (SQLModel / SQLAlchemy): Satisfies REQ-3, REQ-7, REQ-9, REQ-10, REQ-11, REQ-15, REQ-16, REQ-17, REQ-20, REQ-21.
-- Test Suite: Verifies all test-marked requirements and satisfies REQ-19.
+### Transcription and Evidence
 
-## Related Local References
+- **REQ-4-020 Attempt Evidence:** Each provider call must emit one append-only `ExecutionAttempt` record.
+- **REQ-4-021 Attempt Payload:** `ExecutionAttempt` must retain request manifest/hash, outcome, timing, model/provider fields, and error details when present.
+- **REQ-4-022 Transport Evidence:** Provider response evidence must be attached to the attempt when a response is available.
+- **REQ-4-023 Source Projection Rule:** `Source.raw_transcription` is a projection chosen from attempt outcomes and can be repointed by explicit promotion.
+- **REQ-4-024 Candidate Visibility:** UI must expose candidate attempts with metadata needed for comparative review and selection.
 
-- [System Overview](index_v4.md)
-- [System Architecture](architecture_v4.md)
-- [Data Model](schema_v4.md)
-- [Error Handling Policy](error_handling_v4.md)
+### Media and Access
+
+- **REQ-4-030 Ingest Canonicalization:** Stored source bytes may be normalized at ingest (for example orientation correction); stored bytes are the canonical processing source.
+- **REQ-4-031 Path Safety:** Client-facing media URLs must be generated from controlled application paths only.
+- **REQ-4-032 Print Media Validation:** Print/export source media must be served through record-validated API routes.
+
+### Error and UX Contracts
+
+- **REQ-4-040 Error Envelope:** Service/API errors must map to structured, user-safe error categories and messages.
+- **REQ-4-041 Partial Failure Visibility:** Mixed page outcomes must be visible at job and page level.
+- **REQ-4-042 Retry Support:** Failed and cancelled pages must support targeted retranscription without requiring full document recreation.
+
+## Non-Functional Requirements
+
+- **REQ-4-100 Boundary Integrity:** UI pages/components must not access persistence directly and must call service APIs.
+- **REQ-4-101 Service Ownership:** Aggregate writes must occur in owning service/workflow modules, not in UI handlers.
+- **REQ-4-102 Deterministic Loading:** ORM relationship reads in service/UI code must use explicit eager loading compatible with `lazy="raise"`.
+- **REQ-4-103 Async Safety:** Long-running provider calls must not block UI event handlers directly.
+- **REQ-4-104 Evidence Durability:** Attempt evidence must survive process restart once the transaction commits.
+- **REQ-4-105 Test Guardrails:** Architecture boundary tests must remain in place for services and UI boundaries.
+
+## Traceability Notes
+
+- Source of truth for status enums:
+  - `src/transcription/db/models.py`
+- Source of truth for workflow transitions:
+  - `src/transcription/services/workflows.py`
+  - `src/transcription/services/jobs.py`
+- Source of truth for attempt evidence writes:
+  - `src/transcription/services/sources.py`

docs/ver4/schema_v4.md

@@ -1,258 +1,130 @@
-# Database Schema (Version 4)
+# Data Model and Persistence Schema (Version 4)
 
-This document defines the relational schema for the document transcription system.
+This schema reflects the current V4 persistence contract.
 
-## Entity Relationship Diagram
+## Entity Relationship Overview
 
 ```mermaid
 erDiagram
-DOCUMENT_TYPE {
-UUID id PK
-TEXT semantic_key UK
-TEXT label
-TEXT normalized_label
-BOOLEAN is_active
-TIMESTAMPTZ created_at
-TIMESTAMPTZ updated_at
-}
+    Document ||--o{ Source : has
+    Document ||--o{ Job : has
+    Document ||--o{ DocumentPerson : links
+    Person ||--o{ DocumentPerson : links
+    Job ||--o{ JobSource : includes
+    Source ||--o{ JobSource : participates
+    Source ||--o{ ExecutionAttempt : records
 
-PERSON_ROLE {
-UUID id PK
-TEXT semantic_key UK
-TEXT label
-TEXT normalized_label
-BOOLEAN is_active
-TIMESTAMPTZ created_at
-TIMESTAMPTZ updated_at
-}
+    Document {
+      uuid id PK
+      string title
+      uuid type_id FK
+      string language
+      datetime doc_date
+      string date_note
+      string comments
+      string location
+      datetime created_at
+      datetime updated_at
+    }
 
-PERSON {
-UUID id PK
-TEXT full_name
-TEXT display_name
-TEXT maiden_name
-DATE birth_date
-TEXT birth_date_raw
-TEXT birth_place
-DATE death_date
-TEXT death_date_raw
-TEXT death_place
-TEXT biography
-TEXT portrait_path
-JSONB metadata
-TIMESTAMPTZ created_at
-TIMESTAMPTZ updated_at
-}
+    Source {
+      uuid id PK
+      uuid document_id FK
+      string original_name
+      string media_type
+      string storage_path
+      int file_size
+      string file_hash
+      string raw_transcription
+      datetime created_at
+      datetime updated_at
+    }
 
-DOCUMENT {
-UUID id PK
-UUID document_type_id FK
-TEXT name
-DATE document_date
-TEXT document_date_raw
-TEXT location_created
-TEXT notes
-TEXT archive_identifier
-TIMESTAMPTZ created_at
-TIMESTAMPTZ updated_at
-}
+    Job {
+      uuid id PK
+      uuid document_id FK
+      enum status
+      string prompt
+      json model_settings_json
+      datetime created_at
+      datetime updated_at
+    }
 
-DOCUMENT_PERSON {
-UUID id PK
-UUID document_id FK
-UUID person_id FK
-UUID role_id FK
-TIMESTAMPTZ created_at
-TIMESTAMPTZ updated_at
-}
+    JobSource {
+      uuid job_id FK
+      uuid source_id FK
+      enum status
+      uuid selected_attempt_id FK
+      string error_message
+      datetime created_at
+      datetime updated_at
+    }
 
-JOB {
-UUID id PK
-UUID document_id FK
-VARCHAR status
-INTEGER retry_count
-VARCHAR purpose
-TEXT provider
-TEXT model
-TEXT prompt_name
-TEXT prompt_hash
-TEXT system_prompt
-TEXT user_prompt
-FLOAT temperature
-FLOAT top_p
-TIMESTAMPTZ date_created
-TIMESTAMPTZ date_updated
-}
-
-SOURCE {
-UUID id PK
-UUID document_id FK
-INTEGER page_number
-TEXT upload_name
-TEXT filename
-TEXT file_path
-TEXT file_hash
-BIGINT file_size_bytes
-TEXT raw_transcription
-UUID preferred_execution_attempt_id FK
-TEXT revised_text
-TIMESTAMPTZ date_uploaded
-TIMESTAMPTZ date_revised
-}
-
-JOB_SOURCE {
-UUID id PK
-UUID job_id FK
-UUID source_id FK
-VARCHAR status
-TEXT raw_transcription
-JSONB ai_metadata
-JSONB raw_api_response
-TEXT error_detail
-TIMESTAMPTZ executed_at
-}
-
-EXECUTION_ATTEMPT {
-UUID id PK
-UUID job_source_id FK
-UUID job_id FK
-UUID source_id FK
-INTEGER attempt_number
-VARCHAR status
-JSONB request_manifest
-TEXT request_manifest_sha256
-INTEGER transport_status_code
-BINARY transport_body
-JSONB transport_safe_headers
-JSONB sdk_response_snapshot
-JSONB normalized_metadata
-JSONB software_context
-TEXT raw_transcription
-TEXT failure_phase
-TIMESTAMPTZ started_at
-TIMESTAMPTZ finished_at
-INTEGER duration_ms
-}
-
-PROCESSING_ARTIFACT {
-UUID id PK
-UUID source_id FK
-UUID execution_attempt_id FK
-TEXT artifact_type
-TEXT media_type
-TEXT schema_name
-TEXT schema_version
-TEXT producer
-TEXT producer_version
-JSONB inline_payload
-TEXT external_reference
-TEXT payload_sha256
-BIGINT byte_size
-JSONB coordinate_metadata
-TIMESTAMPTZ created_at
-}
-
-DOCUMENT_TYPE ||--o{ DOCUMENT : classifies
-DOCUMENT ||--o{ DOCUMENT_PERSON : has_people
-PERSON ||--o{ DOCUMENT_PERSON : appears_in
-PERSON_ROLE ||--o{ DOCUMENT_PERSON : labels
-DOCUMENT ||--o{ JOB : has_jobs
-DOCUMENT ||--o{ SOURCE : contains_pages
-JOB ||--o{ JOB_SOURCE : executes
-SOURCE ||--o{ JOB_SOURCE : processed_in
-JOB_SOURCE ||--o{ EXECUTION_ATTEMPT : projects
-SOURCE ||--o{ PROCESSING_ARTIFACT : derives
-EXECUTION_ATTEMPT ||--o{ PROCESSING_ARTIFACT : produces
+    ExecutionAttempt {
+      uuid id PK
+      uuid source_id FK
+      uuid job_id FK
+      enum outcome
+      string provider_name
+      string provider_model
+      string request_manifest_hash
+      json request_manifest_json
+      json transport_evidence_json
+      string transcript_text
+      string error_category
+      string error_message
+      float duration_seconds
+      datetime started_at
+      datetime completed_at
+      datetime created_at
+    }
 ```
 
-## Domain Invariants and Provenance Rules
+## Authoritative Enumerations
 
-### Page-Level Execution and AI Outputs
+### JobStatus
 
-- Every single page execution by an AI model produces a dedicated `JOB_SOURCE` record.
-- Every `JOB` stores the frozen prompt identifier, prompt text, and hyperparameters used at submission time.
-- `JOB_SOURCE.raw_api_response` is a compatibility projection containing an SDK-serialized OpenRouter response
-  snapshot. It is neither the exact HTTP body nor the native upstream-provider response.
-- Every new provider call creates an immutable `EXECUTION_ATTEMPT` containing the frozen request manifest,
-  exact OpenRouter-boundary response bytes when received, safe transport metadata, SDK snapshot, normalized
-  metadata, timing, and outcome.
-- `EXECUTION_ATTEMPT(job_id, source_id, attempt_number)` is unique; retries increment the persisted attempt number.
-- Historical `JOB_SOURCE` rows without an `EXECUTION_ATTEMPT` remain SDK snapshots and are explicitly labeled as
-  lacking transport evidence.
-- `SOURCE.raw_transcription` caches the explicitly selected preferred machine output for that page.
-- `SOURCE.preferred_execution_attempt_id` records exact successful-attempt provenance. Legacy projections may remain
-  null until a new successful result is selected.
+- `queued`
+- `processing`
+- `transcribed`
+- `completed` (legacy-compatible)
+- `partial_success`
+- `failed`
 
-### Generic Processing Artifacts
+### JobSourceStatus
 
-- `PROCESSING_ARTIFACT` stores provider-neutral versioned derived outputs.
-- Exactly one of `inline_payload` and `external_reference` is populated.
-- Externally stored artifacts use application-managed relative references and are verified by SHA-256 and byte size.
-- Coordinate metadata declares units, origin, dimensions, and transformations when geometry is present.
-- Orientation-normalized binary model inputs and JSON quality-warning results use distinct versioned artifact types
-  and are attached to the exact consuming `EXECUTION_ATTEMPT`.
+- `pending`
+- `transcribed`
+- `failed`
+- `cancelled`
 
-### Image Storage and Integrity
+## Aggregate Ownership
 
-- Binary images are stored on disk; `SOURCE.file_path` stores the persisted path.
-- `SOURCE.file_hash` stores a SHA-256 digest.
-- `SOURCE.file_size_bytes` stores the original file size.
+- `Document` aggregate: `Document`, linked `Source`, linked `DocumentPerson`.
+- `Job` aggregate: `Job`, `JobSource` rows, selected-attempt pointers.
+- Evidence aggregate: append-only `ExecutionAttempt` rows keyed by `source_id` + `job_id`.
 
-### Page Ordering and Revisions
+## Persistence Invariants
 
-- `SOURCE.page_number` dictates page ordering within a document.
-- `SOURCE.raw_transcription` changes only through first-success selection or explicit candidate promotion.
-- `SOURCE.revised_text` stores human edits and is the preferred display value when present.
+1. `ExecutionAttempt` rows are immutable after creation, except explicit support fields reserved for compatibility migrations.
+2. `JobSource.status` is queue/projection state; it does not duplicate full attempt payload.
+3. `Source.raw_transcription` is a projection, not the complete evidence record.
+4. `Job` terminal status is derived from `JobSource` outcomes.
+5. Registry semantic keys, when present, are immutable once created.
 
-### Semantic Registry Governance
+## Media Storage Semantics
 
-- `DOCUMENT_TYPE.id` and `PERSON_ROLE.id` are the only relationship and public API identities.
-- Nullable unique `semantic_key` values identify application-defined built-ins and are immutable after creation.
-- Semantic keys are internal and are never accepted from Settings or public relationship APIs.
-- A non-null semantic key marks a protected built-in; built-ins may be relabeled or disabled but not deleted.
-- Custom entries have null semantic keys and may be deleted only when unreferenced.
-- Labels are mutable display text and are unique after trimming and case normalization.
-- Inactive entries remain valid for historical rows but are excluded from new-assignment selectors.
+1. `Source.storage_path` references canonical stored bytes used by processing.
+2. Canonical stored bytes may reflect ingest-time normalization.
+3. File hash and size fields describe canonical stored bytes.
 
-### Document-Person Role Governance
+## Query and Loading Requirements
 
-- Documents support zero or one relationship for each Person.
-- Relationship roles are defined by `PERSON_ROLE` rather than hardcoded columns.
-- `DOCUMENT_PERSON.role_id` is required.
-- `DOCUMENT_PERSON` must be unique for `(document_id, person_id)`.
-- Complete link sets and Document fields are validated and persisted in one atomic transaction.
-- Existing inactive roles may remain unchanged; new or changed assignments require active roles.
+- Relationship access from service/UI layers must use explicit eager loading patterns compatible with `lazy="raise"`.
+- Candidate-attempt views should select latest/selected attempts explicitly; do not rely on implicit lazy traversal.
 
-### Document Type Governance
+## Cross-Reference
 
-- Every document type is defined by `DOCUMENT_TYPE`.
-- `DOCUMENT_TYPE.id` is the relationship identity; hidden semantic keys identify protected built-in meaning.
-- `DOCUMENT_TYPE.label` is mutable display text and is unique after trimming and case normalization.
-- `DOCUMENT_TYPE.normalized_label` stores the normalized uniqueness key.
-- Inactive types remain valid for historical rows but should be excluded from default selection UIs.
-
-## Constraint Summary
-
-- `DOCUMENT_TYPE.normalized_label` is unique.
-- `DOCUMENT_TYPE.semantic_key` is nullable and unique.
-- `PERSON_ROLE.normalized_label` is unique.
-- `PERSON_ROLE.semantic_key` is nullable and unique.
-- `DOCUMENT_PERSON(document_id, person_id)` is unique.
-
-## Indexing Guidance
-
-- `document(document_type_id)`
-- `document_person(document_id)`
-- `document_person(person_id)`
-- `document_person(role_id)`
-- `source(document_id, page_number)`
-- `job(document_id, status)`
-- `job_source(job_id)`
-- `job_source(source_id)`
-
-## Related Local References
-
-- [System Overview](index_v4.md)
 - [System Architecture](architecture_v4.md)
 - [System Requirements](requirements_v4.md)
 - [Error Handling Policy](error_handling_v4.md)