V4.4 Scope defined (part 2)

V4.4 Scope defined
V4.3 revision to Document Types
2026-08-15 14:04:58 -05:00 · 2026-08-15 14:04:42 -05:00 · 2026-08-15 13:29:53 -05:00 · 2026-08-14 16:17:23 -05:00 · 2026-08-14 16:08:07 -05:00 · 2026-08-14 15:59:38 -05:00
157 changed files with 18751 additions and 6006 deletions
@@ -1,8 +1,54 @@
-PROVIDER=openrouter
+# --- NiceGUI Server ---
-OPENROUTER_API_KEY=sk-or-...
+# HOST=`0.0.0.0` (default)
-# PROVIDER_MODEL=          # optional: OpenRouter adapter supplies default
+# PORT=8000 (default)
 # LOG_LEVEL: [`critical`, `error`, `warning`, `info` (default), `debug`, `trace`]
 # RELOAD=false (default)
 # --- AI provider ---
 # PROVIDER=[`openrouter`(default), `google_genai`] 
 PROVIDER=openrouter
 # OPENROUTER_API_KEY - Required when `PROVIDER=openrouter`
 OPENROUTER_API_KEY=your-api-key-goes-here
 # GEMINI_API_KEY - Required when `PROVIDER=google_genai`
 # PROVIDER_MODEL= specify model. If left blank OpenRouter will supply default.
 PROVIDER_MODEL=google/gemini-2.5-flash
 # OPENROUTER_HTTP_REFERER=https://example.com
-# OPENROUTER_APP_TITLE=Historical Transcription MVP
+# OPENROUTER_APP_TITLE="Google: Gemini 2.5 Flash (openrouter)"
-# DATABASE_URL=sqlite:///./transcription.db
+
-# UPLOAD_DIR=./uploads
+# --- runtime environment ---
-# PROMPT_DIR=./prompts
+# ENVIRONMENT: [`development`(default), `test`, `production`]
 # --- persistence ---
 # Use nested settings with double underscore because env_nested_delimiter="__".
 # SQLite example:
 # DATABASE__DRIVER=sqlite
 # DATABASE__PATH=app.db
 #
 # SQLite with custom relative path:
 # DATABASE__DRIVER=sqlite
 DATABASE__PATH=./data/transcription.db
 #
 # Postgres example:
 # DATABASE__DRIVER=postgres
 # DATABASE__HOST=localhost
 # DATABASE__PORT=5432
 # DATABASE__DATABASE=transcription
 # DATABASE__USER=postgres
 # DATABASE__PASSWORD=change-me
 #
 # Optional persistence flags:
 # BOOTSTRAP_SCHEMA_ON_STARTUP=false
 # SQLITE_CHECK_SAME_THREAD=false
 # --- filesystem paths ---
 UPLOAD_DIR="./data"
 PROMPT_DIR="./prompts"
 # --- worker reliability ---
 WORKER_MAX_RETRIES=0
 WORKER_RETRY_BACKOFF_SECONDS=0
 # WORKER_PROVIDER_TIMEOUT_SECONDS=[0-20]
 WORKER_PROVIDER_TIMEOUT_SECONDS=20
 WORKER_MIN_TRANSCRIPTION_CHARS=0
 WORKER_MIN_TRANSCRIPTION_LINES=0
 WORKER_FAIL_ON_FINISH_REASON_LENGTH=false
@@ -1,6 +1,53 @@
 ---
-description: Copilot rules for modifying the UI
+description: "Use when modifying the NiceGUI application under src/transcription/ui. Defines ownership and dependency boundaries for UI registration, pages, components, services, persistence, state, and static assets."
 applyTo: 'src/transcription/ui/**/*.py'
 ---
-The UI is forbidden from directly touching the database. All interactions should be done using the [services](../../src/transcription/services/)
+# UI Conceptual Boundaries
 Keep dependencies flowing in this direction:
 `ui/__init__.py` -> `pages` -> `components`
 Pages may depend on application services and framework-provided dependencies. Components may depend on smaller components and shared presentation helpers. Services and domain modules must never depend on the UI.
 ## Package Root
 - Keep `ui/__init__.py` as the UI composition root: register global assets, register pages, and mount NiceGUI on FastAPI.
 - Do not put feature rendering, service calls, persistence, or route-specific state in the package root.
 ## Pages
 - Pages own route registration and route-level orchestration.
 - Resolve request or application dependencies, call [services](../../src/transcription/services/), adapt returned data for presentation when needed, and coordinate refresh, navigation, and notifications here.
 - Do not query, mutate, commit, or roll back the database from a page. Do not import database engines, sessions, operations, or query-building APIs. Persistence belongs to services or workflow functions.
 - Framework dependency types may cross into page handlers only to construct or invoke services; do not pass sessions or session factories into components.
 - Keep business rules, lifecycle transitions, transaction boundaries, and cross-service workflows out of page callbacks.
 ## Components
 - Components own reusable rendering, widget-local state, input normalization, and presentation-only formatting.
 - Expose user actions through typed callback parameters. The calling page decides which service or workflow runs and what refresh or navigation follows.
 - Do not register routes, resolve request/app state, instantiate services, or access persistence from components.
 - Components may accept ORM models returned by services as read-only snapshots. Only use fields and relationships that the service loaded eagerly; never mutate models, trigger lazy loading, or expose session behavior.
 - A component may compose lower-level components, but it must not import from `pages`.
 ## Shared UI Infrastructure
 - Keep app-wide navigation and layout primitives in `components/app_shell.py`.
 - Keep generic table/event adaptation in `components/table/common.py`; feature-specific columns, row read models, and formatting belong in the feature table module.
 - Keep exception normalization and user-facing error display in `components/error_presenter.py`; preserve `AppError` details and operation identifiers at page/component boundaries.
 ## CSS Assets
 - Keep all application CSS in `ui/static/theme.css`; do not add page- or component-specific stylesheets or embed style blocks in Python components.
 - Load `theme.css` once from the composition root with `ui.add_css(..., shared=True)`.
 - Read stylesheet text through `importlib.resources.files(...)` so loading works from installed packages and is independent of the working directory.
 - Centralize CSS reading in one typed helper cached by resource path with `functools.cache` or an equivalent unbounded `lru_cache`. Cache the immutable stylesheet text to avoid repeated resource I/O; keep NiceGUI registration at the composition root.
 - Do not encode application behavior in CSS or other static assets.
 ## State and Side Effects
 - Limit component state to ephemeral interaction state such as loading flags, form values, dialogs, and expansion state.
 - Application and worker state must be resolved at the page or application boundary and passed through narrow interfaces such as callbacks or notifier protocols.
 - Keep filesystem, network, provider, and worker orchestration behind application services or dedicated adapters. UI code may trigger those operations but must not implement them.
@@ -15,7 +15,9 @@ wheels/
 # SQLite database
 *.db
-upload/
+# Document images
-*.jpg
+uploads/*
-*.jpeg
+data/*
-*.png
+
 # Local destructive-test backups
 .test-backups/
@@ -8,14 +8,10 @@
            "module": "debugpy",
            "args": [
                "-m",
-                "uvicorn",
+                "transcription",
-                "transcription.app:create_app",
+                "--host", "127.0.0.1",
-                "--factory",
+                "--port", "9999",
-                "--host",
+                "--database.driver", "sqlite"
                // "127.0.0.1",
                "0.0.0.0",
                "--port",
                "8080"
            ],
            "justMyCode": true,
            "console": "integratedTerminal",
@@ -0,0 +1,3 @@
 {
    "chat.sessionSync.enabled": true
 }
@@ -22,30 +22,105 @@ uv sync
 ### 2) Configure environment
-Create a `.env` file in the project root (minimum required setting shown):
+Create a `.env` file in the project root with the required OpenRouter API key:
 ```env
 OPENROUTER_API_KEY=your_openrouter_api_key
 ```
-Optional settings (defaults shown):
+Settings are read from CLI arguments first, then environment variables, then `.env`, then the defaults below.
 ### Configuration Source Precedence
 When the same setting is provided in multiple places, the value is chosen in this order (highest priority first):
 1. CLI arguments (for example `--port 9999`)
 2. Settings constructor arguments (used mainly in tests)
 3. Environment variables
 4. `.env` file values
 5. Model defaults in code
 Practical examples:
 - `--port 9999` overrides both `PORT=8000` in the shell and `PORT=7000` in `.env`.
 - `DATABASE__PATH=prod.db` in the shell overrides `DATABASE__PATH=dev.db` in `.env`.
 #### Server and runtime
 | Environment variable | Default | Description |
 | --- | --- | --- |
 | `HOST` | `0.0.0.0` | Address on which the server listens. |
 | `PORT` | `8000` | Server port. |
 | `LOG_LEVEL` | `info` | Uvicorn and application log level. |
 | `RELOAD` | `false` | Restart the development server when source files change. |
 | `ENVIRONMENT` | `development` | Runtime environment: `development`, `test`, or `production`. |
 #### Provider
 | Environment variable | Default | Description |
 | --- | --- | --- |
 | `PROVIDER` | `openrouter` | Transcription provider. |
 | `OPENROUTER_API_KEY` | Required | OpenRouter API key. |
 | `PROVIDER_MODEL` | Provider default | Optional model override. |
 | `OPENROUTER_HTTP_REFERER` | Unset | Optional OpenRouter attribution URL. |
 | `OPENROUTER_APP_TITLE` | Unset | Optional OpenRouter attribution title. |
 #### Database and files
 Use nested env vars for database settings (recommended):
 ```env
-DATABASE_URL=sqlite:///./transcription.db
+DATABASE__DRIVER=sqlite
 DATABASE__PATH=app.db
 # BOOTSTRAP_SCHEMA_ON_STARTUP=true
 SQLITE_CHECK_SAME_THREAD=false
 UPLOAD_DIR=./uploads
 PROMPT_DIR=./prompts
 DEFAULT_PROMPT_NAME=transcribe_document.md
 # TRANSCRIPTION_TEMPERATURE=0.2  # range: 0.0-2.0
 # TRANSCRIPTION_TOP_P=0.9       # range: 0.0-1.0
 ```
 For PostgreSQL:
 ```env
 DATABASE__DRIVER=postgres
 DATABASE__HOST=localhost
 DATABASE__PORT=5432
 DATABASE__DATABASE=transcription
 DATABASE__USER=postgres
 DATABASE__PASSWORD=change-me
 ```
 This uses Pydantic nested settings (`env_nested_delimiter='__'`) and avoids JSON blobs in `.env`. A top-level `DATABASE={...}` JSON value is still supported as a fallback, and nested keys such as `DATABASE__PATH` take precedence over conflicting JSON keys.
 `BOOTSTRAP_SCHEMA_ON_STARTUP` creates missing tables when the app starts. When unset, it is enabled in `development` and `test`, and disabled in `production`; set it explicitly to override that policy. `SQLITE_CHECK_SAME_THREAD` defaults to `false`.
 #### Worker
 ```env
 WORKER_MAX_RETRIES=0
 WORKER_RETRY_BACKOFF_SECONDS=0
 WORKER_PROVIDER_TIMEOUT_SECONDS=20
 WORKER_MIN_TRANSCRIPTION_CHARS=0
 WORKER_MIN_TRANSCRIPTION_LINES=0
 WORKER_FAIL_ON_FINISH_REASON_LENGTH=false
 ```
 ### 3) Run the app
 ```bash
-uv run uvicorn transcription.app:create_app --factory --reload
+uv run python -m transcription --port 9999 --reload --database.driver sqlite --bootstrap-schema-on-startup
 ```
 This starts the development server with SQLite, creates missing tables, and enables automatic reload. Run `uv run python -m transcription --help` for all CLI options; CLI names use kebab case and nested database options use dot notation, such as `--database.path ./data/transcription.db`.
 ### 4) Open in browser
- GUI: [http://[IP_ADDRESS]:8000/ui](http://[IP_ADDRESS]:8000/ui)
+- GUI: [http://localhost:9999/ui](http://localhost:9999/ui)
- Health check: [http://[IP_ADDRESS]:8000/healthz](http://[IP_ADDRESS]:8000/healthz)
+- Health check: [http://localhost:9999/healthz](http://localhost:9999/healthz)
 Replace `localhost` with the server's hostname or IP address when connecting from another machine.
 ## How to navigate the GUI
@@ -66,7 +141,65 @@ uv run uvicorn transcription.app:create_app --factory --reload
 ## Prompt artifacts
-Prompt files are stored in `prompts/` and loaded from `PROMPT_DIR` (default: `./prompts`).
+Prompt files are stored directly in `PROMPT_DIR` (default: `./prompts`). `DEFAULT_PROMPT_NAME` must be a filename,
 not a path. Each job snapshots the validated prompt text, SHA-256 hash, and sampling values for reproducibility.
 The canonical MVP prompt is:
 - `prompts/transcribe_document.md`
 ## Destructive test procedure (with data backup)
 AI execution policy: before the first unit-test run in a test/fix cycle, create one backup of `./data`. Reuse that same backup for every subsequent test run in the cycle. After tests succeed, always pause and ask whether to restore now.
 Use the cross-platform Python wrapper below whenever an AI agent runs tests against this repository.
 1. Create one backup of `./data` and mark it as the active test-cycle backup.
 2. Run your test command.
 3. On failure, fix the errors and run the wrapper again; it reuses the active backup and never backs up post-test data.
 4. On success, always prompt whether to restore now (do not auto-restore unless explicitly approved).
 5. Close the cycle only by restoring the active backup or explicitly accepting the current data.
 Preflight behavior:
 - Backup preflight is warning-only when `data/transcription.db` appears in use.
 - Restore preflight is blocking: the script prompts you to close conflicting applications, then type `retry` to re-check or `cancel` to skip restore.
 ### Run with confirmation-gated restore (default)
 ```bash
 uv run python tools/run_destructive_tests.py -- pytest tests/services/test_job_service.py tests/ui/test_jobs_page.py
 ```
 After tests pass, the script asks whether to restore backup immediately.
 This is the required default mode for AI-assisted test runs because it gives time to verify and accept code changes before any restoration happens.
 ### Run with automatic restore (non-interactive)
 ```bash
 uv run python tools/run_destructive_tests.py --auto-restore -- pytest
 ```
 ### Run without terminal prompt (decide restore later)
 ```bash
 uv run python tools/run_destructive_tests.py --skip-restore-prompt -- pytest
 ```
 This keeps both the current post-test state and the backup, so restore can be decided explicitly later.
 Repeated wrapper invocations reuse the backup recorded in `.test-backups/.active-backup`. If that backup is missing, the wrapper stops rather than creating a replacement from potentially destructive post-test data.
 ### Restore later from a saved backup
 ```bash
 uv run python tools/run_destructive_tests.py --restore-from data-backup-YYYYMMDD-HHMMSS
 ```
 To keep the current data and close the active cycle without restoring:
 ```bash
 uv run python tools/run_destructive_tests.py --accept-current-data
 ```
 Backups are stored in `.test-backups/` and ignored by git.
@@ -1,40 +0,0 @@
 # Historical Document Transcription
 I have several thousand pages of family history told through letters, postcards, books, and other documents that I want to transcribe to text. 
 ## Goals
 1. Preserve our family history
 2. Unburden my family (and descendants) from having to store and care for the physical media. Once the documents have been transcribed and organized, they can be donated (or kept by a family member that wants to retain them).
 3. Make the text easily available and searchable by family members, as well as AI (which may have different requirements).
 4. Ability create timelines or assemble the historical record of the family or specific individuals from across the complete document archive. Perhaps use AI to create the timelines in a more narrative form.
 ## Source material
 1. **letters, cards, diaries** - handwritten; mostly stored in tubs, with little organization
 2. **books** - typed or typeset; mostly self-published books 50-100 pages in length. This may be expanded to include selected pages from other publications.
 3. **newspaper clippings, event programs, invitations, and other ephemera**
 ## Methodology
 ### Verbatim vs. Clean Copy
 Transcriptions should be Verbatim and follow scholarly research guidelines, with no modifications to the original text.
 ### Prompt Curation Policy
 Transcription behavior should be implemented with prompt assets that are human-maintainable over time.
 1. Each transcription prompt is stored as an individual Markdown file.
 2. Prompt files are refined iteratively as document quality and edge cases are discovered.
 3. Prompt changes should be scoped to one prompt file at a time whenever possible to keep review history clear.
 ### Potential Document Issues
 | Document Issue | How to Handle It | Example |
 | :--- | :--- | :--- |
 | **Misspellings & Errors** | Retain original spelling and insert italicized `[sic]` directly after the error. | `The weather was very cold and publick [sic] business delayed.` |
 | **Missing Words / Slips** | Insert the missing word inside square brackets to restore basic readability. | `We went [to] the store to buy supplies.` |
 | **Uncertain / Guesswork** | Place your best hypothesis followed by a question mark inside square brackets. | `He went to [Boston?] yesterday to meet the governor.` |
 | **Completely Illegible** | Use a clear descriptive term like `[illegible]` or specify the reason (e.g., `[torn]`, `[ink blot]`). | `The total cost was [illegible] dollars.` or `The letter ends here [remainder of page torn].` |
 | **Crossed-out Text** | Wrap the removed word or phrase in a deleted tag to preserve the author's edits. | `We left at [deleted: noon] one o'clock instead.` |
 | **Squeezed-in Text** | Wrap text that was added above the line or in a tight space in an inserted tag. | `The [inserted: red] house on the hill was abandoned.` |
 | **Superscripts & Abbreviations** | Bring raised letters down to the main line, or optionally expand them in brackets. | `Change Gen^l to Genl` OR `Change to Gen[era]l depending on project preference.` |
 | **Images / Seals / Signs** | Describe the non-textual element using italicized text inside square brackets. | `[wax notary seal attached here]` or `[sketch of a fort layout]` |
 | **Marginalia / Notes** | Note the spatial transition clearly before transcribing the note itself. | `[written in left margin:] Do not share this with anyone.` |
 | **Line Breaks / Hyphens** | Rejoin words split across a page margin silently, dropping the line-break hyphen. | `Original: "estab- / lishment" becomes "establishment"` |
 | **Ambiguous Capitalization** | Default to modern capitalization rules unless an archaic uppercase letter is clearly intentional. | `If a standard noun like 'Farm' looks randomly capitalized, type 'farm'.` |
 **Hierarchical Outlines** | Preserve exact numbering characters (including lowercase Roman numerals or terminal 'j'). Replicate indentation levels using spaces/tabs. Do not correct math or sequence errors silently. | `I. Main Topic`<br>`&nbsp;&nbsp;a. Sub-point`<br>`&nbsp;&nbsp;b. Next point`<br>`III. [sic] Third Topic` |
@@ -1,294 +0,0 @@
 # Architecture
 This document describes the production architecture of the personal historical-document transcription system. The system is intentionally optimized for single-user operation, low operational overhead, and clean internal boundaries that support future growth without rewrites.
 ## Architecture Objectives
 The production architecture is designed to:
 - preserve verbatim family-history source material as searchable text
 - keep operational complexity low for a personal deployment
 - support asynchronous transcription without requiring distributed infrastructure
 - maintain clear module boundaries so extensions can be added incrementally
 ## Production Scope And Scale
 The deployed system targets personal use and a corpus of several thousand documents processed over time. The architecture favors simple, composable building blocks over distributed orchestration.
 Current scope includes:
 - document upload and metadata capture
 - asynchronous transcription jobs
 - prompt-library driven transcription behavior, with one Markdown file per prompt
 - transcript review and revision history
 - full-text search over accepted transcripts
 - export of transcript data
 ## Deployment Topology
 The production deployment uses [Docker Compose](https://docs.docker.com/compose/) and treats containerized databases as extremely lightweight operational dependencies.
 Running [PostgreSQL](https://www.postgresql.org/docs/) in its own container is considered simple by default for this system.
 Running [MongoDB](https://www.mongodb.com/docs/) in its own container is also considered simple when document-centric storage is enabled.
 Container count is not a hard architectural limit; a three-container deployment (app, PostgreSQL, MongoDB) is an acceptable baseline.
 ### Baseline Topology (Two Containers)
 - one application container
 - one PostgreSQL container
 - embedded background worker execution inside the app process
 ### Expanded Topology (Three Containers)
 - application container
 - PostgreSQL container
 - MongoDB container
 No additional queue, scheduler, or search-engine containers are required in the baseline production setup.
 ## Runtime Architecture
 ```mermaid
 flowchart LR
    User[Browser User] --> App[FastAPI + NiceGUI Service]
    App --> Worker[In-process Background Worker]
    App --> PG[(PostgreSQL)]
    App --> MG[(MongoDB Document Store)]
    Worker --> AI[Transcription Provider]
    Worker --> PG
    Worker --> MG
 ```
 ## Runtime Ownership And Startup Policy (V1 Step 1)
 The current implementation now uses explicit lifespan-owned runtime resources.
 - application lifespan initializes and disposes database runtime resources
 - worker lifecycle is owned by application lifespan startup/shutdown
 - worker receives lifespan-owned database engine dependency explicitly
 - schema bootstrap policy is environment-aware and explicit:
  - development/test default to bootstrap enabled
  - production defaults to bootstrap disabled
  - explicit override is available via configuration
 This aligns implementation toward REQ-7 and REQ-10 while preserving personal-scale operational simplicity.
 ## Layered Module Structure
 ### Interface Layer
 Responsibility:
 - HTTP API and UI routes
 - request/response validation
 - status and result presentation
 Out of scope:
 - business-rule enforcement
 - data-access implementation
 ### Application Layer
 Responsibility:
 - upload and job orchestration
 - state transitions and retry policy
 - coordination across domain and infrastructure ports
 Out of scope:
 - provider-specific protocol details
 - ORM or storage-specific logic
 ### Domain Layer
 Responsibility:
 - verbatim transcription policy
 - revision and provenance invariants
 - confidence and annotation semantics
 Out of scope:
 - web framework concerns
 - database and network I/O
 ### Infrastructure Layer
 Responsibility:
 - persistence adapters (PostgreSQL and MongoDB)
 - transcription-provider adapter
 Out of scope:
 - business policy decisions
 ## Processing Workflow
 Production transcription flow:
 1. A user uploads an image or PDF through the UI or API.
 2. The application validates payloads and creates document and job records.
 3. The in-process worker dequeues the job and calls the transcription provider.
 4. The application persists transcript output, confidence metadata, and provenance events.
 5. Job status transitions from queued to processing to transcribed or failed.
 6. The UI and API expose status, revision history, and searchable transcript text.
 ## Data Model Ownership
 System-of-record entities:
 - documents and pages
 - transcription jobs and status events
 - transcript revisions
 - provenance metadata
 Storage strategy:
 - PostgreSQL for relational system-of-record entities
 - MongoDB for document-oriented payloads and large transcription artifacts
 - versioned prompt artifacts stored as individual Markdown files for human editing and refinement
 - in-memory execution state treated as ephemeral
 ## Transcription Prompt Asset Policy
 The production system treats transcription prompts as maintainable content assets.
 - each transcription prompt is stored in its own Markdown file
 - prompt files are designed for direct human editing and iterative refinement
 - prompt updates are independent and do not require bundling unrelated prompt changes
 - prompt file identity and revision history are tracked through normal repository version control
 ## Simplicity Guardrails
 The production system enforces these constraints to prevent accidental over-engineering:
 - PostgreSQL in a container is treated as a lightweight default dependency
 - MongoDB in a container is treated as a lightweight optional dependency
 - three containers (app, PostgreSQL, MongoDB) is an acceptable simple deployment
 - no dedicated queue or search cluster is introduced without measured need
 - external infrastructure is added only behind existing ports/adapters
 ## Extension Path
 The architecture supports additive growth without changing domain contracts.
 ### Stage 1: Foundation (Current)
 - upload, transcription, review, search, export
 - in-process worker execution
 - single provider adapter
 - app plus PostgreSQL deployment
 ### Stage 2: Throughput Hardening
 - optional MongoDB document-store enablement
 - optional external worker/queue process
 - stronger retry and dead-letter handling
 ### Stage 3: Intelligence Features
 - entity extraction and cross-document linking
 - timeline and narrative assembly
 - optional multi-provider routing
 Each stage preserves existing module boundaries and keeps migration risk low.
 ## Test Strategy
 The test strategy is aligned to personal-scale operation with fast, deterministic feedback.
 ### Unit Tests
 - domain transcription rules and annotation behavior
 - revision-history invariants
 - job state-transition logic
 ### Integration Tests
 - repository behavior and transaction boundaries
 - persistence-adapter and provider adapter contract mapping
 - upload-to-persistence roundtrip
 ### End-to-End Tests
 - happy path: upload, transcribe, review, search, export
 - failure path: provider error, retry, surfaced failed status
 ### CI Execution Model
 - fast suite on each push
 - optional slower provider-sandbox checks on scheduled runs
 ## Risks And Controls
 ### Runtime Responsiveness
 Risk:
 - long jobs can reduce responsiveness in a single-process deployment
 Control:
 - bounded concurrency and visible job status in the UI
 ### Database Concurrency Limits
 Risk:
 - contention can appear under sustained concurrent writes in personal-scale infrastructure
 Control:
 - tuned connection pooling and phased use of MongoDB for document-heavy workloads
 ### Provider Output Variance
 Risk:
 - transcription quality varies by document type, handwriting legibility, and image quality
 Control:
 - first-class human review and immutable revision history
 ## Technology References
 - [FastAPI documentation](https://fastapi.tiangolo.com/)
 - [NiceGUI documentation](https://nicegui.io/documentation)
 - [Docker Compose documentation](https://docs.docker.com/compose/)
 - [PostgreSQL documentation](https://www.postgresql.org/docs/)
 - [MongoDB documentation](https://www.mongodb.com/docs/)
 ## Related Pages
 - [System overview](index.md)
 - [Version 1 plan](ver1/ver1.md)
 - [Version 1 Step 1 plan](ver1/ver1-step1.md)
 - [Version 1 Step 1 results](ver1/ver1-step1-results.md)
 - [Architecture decision records index](adr/README.md)
 ## Glossary
 - Adapter: A component that translates between internal interfaces and external systems such as databases or AI services.
 - Background job: Work executed outside the request/response path so the UI remains responsive.
 - Boundary: A strict separation between modules with different responsibilities.
 - CI (Continuous Integration): Automated test execution for code changes.
 - Contract test: A test that verifies an adapter follows expected input/output behavior at a boundary.
 - Domain layer: The module that contains core business rules and invariants.
 - End-to-end test: A test that validates a full user flow across the running system.
 - Full-text search: Text indexing and querying optimized for natural-language search.
 - In-process worker: A background executor that runs within the same application process.
 - Integration test: A test that verifies interactions between real modules and infrastructure components.
 - MongoDB: A document-oriented database used for flexible, high-variance data structures.
 - Modular monolith: A single deployable application with strongly separated internal modules.
 - Port/Interface: A stable contract used by application/domain code to call infrastructure implementations.
 - Prompt artifact: A single Markdown file that defines one transcription prompt and can be revised independently.
 - Provenance: Metadata that records where generated data came from and how it was produced.
 - Revision history: Versioned record of transcript edits over time.
 - System of record: The authoritative persistent store for canonical data.
 - Vertical slice: A minimal end-to-end feature path spanning UI/API, application logic, and persistence.
@@ -1,282 +0,0 @@
 # Error Handling
 This document defines the canonical error-handling policy for the document transcription system. It is the single source of truth for how errors are classified, surfaced to users, logged for diagnosis, and handled across UI, API, service, worker, and provider boundaries.
 ## Error Handling Objectives
 The production error-handling model is designed to:
 - make failures visible to the user in clear, actionable language
 - preserve enough diagnostic detail for fast troubleshooting
 - keep module behavior consistent across all boundaries
 - distinguish expected domain failures from unexpected defects
 - support safe retries for transient failures without hiding persistent faults
 ## Scope And Authority
 This page governs error-handling behavior for:
 - UI interactions (NiceGUI pages)
 - API endpoints (FastAPI routes)
 - application services and orchestration logic
 - in-process background worker execution
 - external provider adapters and persistence adapters
 If implementation behavior conflicts with this document, this document is authoritative and implementation should be updated.
 ## Core Principles
 - **Clarity first:** user-facing messages should explain what failed in plain language.
 - **Actionability required:** each surfaced error should include a suggested next step.
 - **Safety by default:** internal details are logged; sensitive details are not exposed by default in UI/API.
 - **Consistency across boundaries:** category and structure should remain stable from source to surface.
 - **Fail explicitly:** silent failure is prohibited.
 - **Traceability:** every non-trivial error should be traceable with an error reference ID.
 ## Error Taxonomy
 The system uses stable, implementation-independent categories:
 | Category | Definition | Typical Source | Retriable |
 | --- | --- | --- | --- |
 | `validation_error` | Payload or parameter shape/content is invalid | UI/API input validation, service guards | no |
 | `user_input_error` | User-provided artifact is unacceptable though structurally valid | unsupported file type, empty file, oversized upload | sometimes |
 | `not_found_error` | Requested resource does not exist | missing job/document/transcript | no |
 | `conflict_error` | Requested operation violates current state constraints | invalid state transition | no |
 | `external_provider_error` | External AI/provider call fails | upstream HTTP/API/provider failures | sometimes |
 | `infrastructure_transient_error` | Temporary environment issue | network timeout, DB connection reset | yes |
 | `infrastructure_persistent_error` | Non-transient environment issue | missing permissions, misconfiguration | no |
 | `internal_unexpected_error` | Unhandled defect or unknown failure | uncaught exceptions, logic errors | unknown (default no) |
 ### Classification Rules
 - Classification occurs as close as possible to the origin boundary.
 - Provider-specific exceptions must be normalized into taxonomy categories before crossing service boundaries.
 - Unknown exceptions are classified as `internal_unexpected_error` and logged with traceback.
 - Category names are stable contracts and must not be changed casually.
 ## User-Facing Error Experience Contract
 When an error is shown in the GUI, it must include:
 1. **Title** (short context, e.g., “Upload failed”)
 2. **Message** (plain-language explanation)
 3. **Suggested action** (explicit next step)
 4. **Error reference ID** (for support/debug traceability)
 5. **Technical details** (optional/collapsible for advanced users)
 ### UI Message Rules
 - Do not expose raw stack traces by default.
 - Do not expose secrets, credentials, connection strings, or filesystem internals unless explicitly in debug tooling.
 - Prefer domain language over implementation language.
 - Use persistent visibility for important failures (dialog/card), not only transient toasts.
 ### Suggested Action Requirements
 Every user-visible error must include a suggested course of action, such as:
 - retry the operation
 - check file type/size constraints
 - refresh the jobs page
 - verify environment configuration
 - contact operator with error ID and timestamp
 ## API Error Response Contract
 API errors should return a structured envelope with stable fields:
 - `error_id`: short unique reference ID
 - `category`: taxonomy category
 - `message`: safe human-readable summary
 - `suggestion`: recommended next step
 - `details`: optional, only when safe and appropriate
 - `timestamp`: UTC ISO-8601
 HTTP status mapping guidance:
 - `validation_error`, `user_input_error` -> `400`
 - `not_found_error` -> `404`
 - `conflict_error` -> `409`
 - `external_provider_error` -> `502` or `503` (depending on failure mode)
 - `infrastructure_transient_error` -> `503`
 - `infrastructure_persistent_error` -> `500`
 - `internal_unexpected_error` -> `500`
 ## Logging And Observability Contract
 All logged errors must include, where available:
 - `error_id`
 - `category`
 - `operation` (e.g., `upload.submit`, `worker.process_job`, `jobs.refresh`)
 - `exception_type`
 - `job_id`, `document_id` (when relevant)
 - UTC timestamp
 Rules:
 - Use structured logging fields where practical.
 - Use full traceback for unexpected errors (`internal_unexpected_error`).
 - Log at boundary handoff points to preserve causal trail.
 - Avoid duplicate noisy logging for the same exception at every layer.
 ## Recovery And Retry Policy
 ### Retriable Conditions
 Retriable failures include:
 - transient network/provider timeouts
 - intermittent provider unavailability
 - temporary DB/network interruptions
 ### Non-Retriable Conditions
 Non-retriable failures include:
 - invalid file formats
 - missing required data
 - permission/configuration failures
 - deterministic domain conflicts
 ### Worker Behavior
 - The worker must classify and persist failure details consistently.
 - Retries should be bounded by configured limits.
 - Exhausted retries must end in explicit failed status with recorded reason.
 - No infinite retry loops are allowed.
 ## Boundary-Specific Responsibilities
 ### UI Layer
 Responsibility:
 - display user-safe error summaries and suggested actions
 - show persistent error visibility for critical failures
 - include error reference IDs in visible output
 Out of scope:
 - low-level exception parsing
 - provider-specific protocol interpretation
 ### API Layer
 Responsibility:
 - map application exceptions into stable error envelopes and HTTP statuses
 - preserve category and error_id continuity
 Out of scope:
 - domain-specific remediation logic
 ### Service Layer
 Responsibility:
 - classify domain and infrastructure exceptions
 - convert adapter-specific failures into taxonomy categories
 - return deterministic error types to callers
 Out of scope:
 - presentation formatting for UI
 ### Worker Layer
 Responsibility:
 - execute retry policy for retriable failures
 - persist terminal failure details for jobs
 - emit operational logs with category and identifiers
 Out of scope:
 - direct UI messaging
 ### Provider Adapter Layer
 Responsibility:
 - normalize provider SDK/HTTP failures into domain-neutral exceptions
 - preserve raw provider context for logs (safely)
 Out of scope:
 - choosing user-facing wording
 ## Error Lifecycle Workflow
 Standard lifecycle:
 1. Failure occurs at a boundary or operation.
 2. Exception is classified into taxonomy category.
 3. `error_id` is created (or propagated).
 4. Error is logged with required structured fields.
 5. User/API receives safe message + suggested action.
 6. Persistent job/resource state is updated when applicable.
 7. Tests verify contract behavior for the pathway.
 ## Test Strategy For Error Handling
 ### Unit Tests
 - category classification behavior
 - retry eligibility decisions
 - exception-to-message mapping safety
 ### Integration Tests
 - UI pathways show clear message + suggested action for known failures
 - API returns structured error envelope with expected status/category
 - worker persists failed status and failure detail as required
 ### Regression Tests
 - each previously observed production issue should have a guarding test
 - contract tests must cover adapter error normalization behavior
 ## Known Failure Patterns And Prescribed Responses
 | Pattern | Category | User Message | Suggested Action |
 | --- | --- | --- | --- |
 | Upload payload cannot be parsed by UI handler | `internal_unexpected_error` (until narrowed) | Upload failed due to unexpected processing error | Retry once; if repeated, report error ID and check runtime version compatibility |
 | Unsupported extension | `user_input_error` | File type is not supported | Upload JPG, PNG, TIFF, or PDF |
 | Empty file upload | `validation_error` | Uploaded file is empty | Choose a valid non-empty file and retry |
 | Provider timeout | `external_provider_error` or `infrastructure_transient_error` | Transcription provider timed out | Retry from jobs page; if repeated, check provider status |
 | Job lookup missing | `not_found_error` | Requested job was not found | Refresh jobs list and open a valid job |
 ## Governance And Update Process
 This document is a living policy artifact.
 Update this document when:
 - new error categories are introduced
 - handling behavior changes at any boundary
 - a production incident reveals missing guidance
 - API/UI error contracts change
 Change requirements:
 - update this document and associated tests in the same change set
 - preserve taxonomy stability; if changed, document migration impact
 - record noteworthy policy changes in project release notes or changelog
 ## Related Pages
 - [System overview](index.md)
 - [Architecture](architecture.md)
 - [Requirements](requirements.md)
 - [Intent](intent.md)
 ## Glossary
 - Error category: Stable classification used to drive handling, messaging, and status mapping.
 - Error envelope: Structured API payload describing a failure.
 - Error reference ID: Short identifier used to correlate user-visible failure with logs.
 - Retriable error: Failure likely to succeed on a later attempt without code changes.
 - Terminal failure: Failure state after retries are exhausted or retry is not allowed.
@@ -1,57 +0,0 @@
 ## Document Transcription System
 This project is a production application for transcribing and preserving historical family documents. It is intentionally designed for personal-scale use, with a simplicity-first architecture that is easy to operate and easy to extend.
 ## Start Here
 Read [architecture.md](architecture.md) first.
 Then review [ver1/ver1.md](ver1/ver1.md) for completion scope and [ver1/ver1-step1-results.md](ver1/ver1-step1-results.md) for current architecture-consolidation status.
 The architecture page is the primary technical reference and defines:
 - deployed topology and infrastructure limits
 - module boundaries and dependency flow
 - processing life cycle and data ownership
 - test strategy, risk controls, and extension path
 ## What The Application Does
 At a high level, users upload images of handwritten, typed, or typeset documents, run asynchronous transcription jobs, review and edit transcript revisions, and search across accepted text.
 Core capabilities:
 - document upload and metadata capture
 - asynchronous transcription with visible job status
 - transcription prompt management with one Markdown file per prompt for human refinement over time
 - revision history for transcript edits
 - full-text search over accepted transcripts
 - export of transcript data
 ## Production Operating Model
 The system runs with minimal operational overhead:
 - PostgreSQL in a dedicated Docker container is considered extremely lightweight and simple for this system
 - MongoDB in a dedicated Docker container is also considered extremely lightweight and simple for document-centric persistence
 - a three-container deployment (app, PostgreSQL, MongoDB) is a simple and acceptable baseline
 - no required queue or search-engine containers in the baseline setup
 This operating model keeps deployment and maintenance simple while preserving clean boundaries for future scale.
 ## Documentation Map
 - Architecture and technical design: [architecture.md](architecture.md)
 - Version 1 implementation plan: [ver1/ver1.md](ver1/ver1.md)
 - Version 1 Step 1 plan: [ver1/ver1-step1.md](ver1/ver1-step1.md)
 - Version 1 Step 1 results: [ver1/ver1-step1-results.md](ver1/ver1-step1-results.md)
 - Architecture decision records (ADR index): [adr/README.md](adr/README.md)
 - Runtime and deployment requirements: [requirements.md](requirements.md)
 - Error handling policy and operational guidance: [error_handling.md](error_handling.md)
 - Domain context and transcription policy: [intent.md](intent.md)
 ## Glossary
 - Document-oriented persistence: Storing data as flexible records instead of fixed relational rows.
 - Prompt artifact: A single Markdown file that defines one transcription prompt and is edited independently.
 - System of record: The authoritative persistent store for canonical data.
@@ -0,0 +1,141 @@
 # Digital Evidence and AI Processing Provenance (Invariant)
 ## 1. Purpose
 This document defines non-negotiable evidence and provenance rules for the transcription application.
 The application exists to preserve historical source material and produce useful transcriptions without losing the ability to inspect, reinterpret, or reprocess the evidence later. Provider integrations, model names, schemas, and user interfaces may change; the principles below must remain true.
 ## 2. Evidence Model
 The application distinguishes five kinds of information:
 1. **Source evidence**: the original uploaded media and the facts needed to identify and verify it.
 2. **Execution specification**: the frozen instructions, parameters, source identity, and software context for one processing attempt.
 3. **Transport evidence**: the response received at the application/provider boundary, including safe protocol metadata.
 4. **Normalized data**: selected fields extracted for search, display, accounting, and workflow behavior.
 5. **Derived artifacts**: outputs produced from source evidence, such as transcription text, OCR geometry, confidence data, layout analysis, or entity extraction.
 Normalized data and derived artifacts never replace source or transport evidence.
 ## 3. Core Invariants
 ### 3.1 Original Source Preservation
 1. The original uploaded bytes are the primary evidence and must be preserved without transformation.
 2. Each source must have a cryptographic content digest, byte size, and stable identity.
 3. Processing may use transformed derivatives, but those derivatives must not overwrite the original.
 4. A derivative used for processing must record its relationship to the original, its transformation, and its own digest.
 5. Moving or renaming a stored file must not change its evidence identity.
 ### 3.2 Append-Only Processing History
 1. Every processing attempt must have a distinct execution record, whether it succeeds, partially succeeds, times out, or fails.
 2. A later attempt must not overwrite the evidence from an earlier attempt.
 3. A convenient “latest transcription” value may be maintained as a cache or projection, but it is not the authoritative execution history.
 4. Human revisions must remain distinguishable from all machine-generated outputs.
 5. Reprocessing a source must create new evidence rather than rewriting historical evidence.
 ### 3.3 Frozen Execution Specification
 Each execution must preserve enough information to understand what the application asked the processor to do:
 1. Requested provider, model, and provider-routing constraints.
 2. Full effective system and user instructions.
 3. Prompt asset name and content digest when a prompt asset is used.
 4. Every explicitly supplied generation or processing parameter.
 5. Whether an optional parameter was explicitly set or omitted.
 6. Source and derivative digests, media type, dimensions or page geometry when known, and page identity.
 7. A secret-safe representation of the request structure.
 8. Application, provider-adapter, and client-library versions sufficient to interpret the execution.
 The execution specification must not contain credentials, authorization headers, secret query values, or unnecessary duplicate source binaries.
 ### 3.4 Evidence-Layer Terminology
 The following terms are not interchangeable:
 - **Transport response**: the status, safe headers, and exact response body received by the application at its HTTP boundary.
 - **Router-normalized response**: a response transformed by an intermediary into its common schema.
 - **SDK-parsed response**: an object created when a client library validates or filters a response.
 - **Normalized metadata**: application-selected fields derived from a response.
 - **Native provider response**: the upstream provider's own response before any intermediary transformation.
 The application and its documentation must identify which layer is stored. A response must not be described as “raw,” “complete,” or “native” without naming the boundary at which that claim is true.
 ### 3.5 Transport Evidence
 1. Preserve the exact successful response body received at the application's transport boundary before SDK model parsing can discard unknown fields.
 2. Preserve the response status and an allowlisted set of non-secret headers needed for correlation, content interpretation, rate-limit diagnosis, or audit.
 3. Preserve provider/router request and generation identifiers when available.
 4. Preserve safe response evidence for unsuccessful calls when a response was received.
 5. Record explicitly when no response was received, such as a local timeout or connection failure.
 6. Retain parsed and normalized forms only as additional representations of the preserved response.
 Wire-level packet capture, TLS session data, credentials, and unrestricted headers are neither required nor permitted.
 These requirements apply to executions performed after transport capture is implemented. For earlier executions, the absence of transport evidence must be represented explicitly. An SDK snapshot or normalized record must never be relabeled or backfilled as transport evidence.
 ### 3.6 Derived Artifact Provenance
 1. Every derived artifact must identify its source evidence and producing execution.
 2. Each artifact must declare its semantic type, media/serialization format, schema name and version, producer, producer version, and creation time.
 3. Artifact content must be stored directly or referenced by a stable path or object identifier and protected by a cryptographic digest.
 4. Coordinates must declare their coordinate system, units, origin, page/image dimensions, and transformation history.
 5. Confidence values must identify the producer and scope to which they apply; values from different producers must not be treated as directly comparable without validation.
 6. Provider-specific payloads may be retained, but durable application behavior must not depend on undocumented provider fields.
 This model must accommodate future OCR text, word or line polygons, layout regions, confidence data, alternate transcriptions, and structured extraction without adding a dedicated column for every possible feature.
 ### 3.7 Integrity and Auditability
 1. Stored evidence must be exportable with enough identifiers and metadata to verify relationships and digests outside the application.
 2. Evidence mutation, deletion, and retention behavior must be explicit and testable.
 3. Schema upgrades must preserve existing evidence and its original meaning.
 4. Backfills must be identified as backfills; they must not imply that previously uncaptured evidence existed.
 5. Integrity verification must distinguish a missing file, digest mismatch, unavailable external artifact, and malformed metadata.
 ### 3.8 Security and Privacy
 1. API keys, authorization headers, cookies, and credentials must never be persisted as provenance.
 2. Persist only headers and metadata fields that appear on an explicit allowlist of known-safe fields. Discard all other fields before storage; never persist an unrestricted capture and attempt to redact it afterward.
 3. Request manifests should reference source content by identity instead of duplicating base64 source data.
 4. Diagnostic displays and exports must avoid exposing secrets or machine-local details that are not necessary for evidence interpretation.
 ## 4. Reproducibility Limits
 Provenance supports explanation, comparison, and best-effort reproduction; it does not guarantee identical output.
 Identical requests may produce different results because of model updates, provider routing, nondeterministic computation, undocumented defaults, safety systems, or retired endpoints. The application must preserve whether a parameter was omitted rather than pretending to know the provider default used at that time.
 Likewise, preserving a general vision-model response does not create OCR coordinates that were never returned. Future coordinate extraction remains possible because the original source evidence is preserved and can be processed again by a suitable system.
 ## 5. Model Evaluation Policy
 Model selection must be based on a representative sample of the actual archive rather than vendor claims alone.
 Evaluation should:
 1. Use manually reviewed reference transcriptions following the project's [Transcription Methodology](transcription_methodology.md).
 2. Represent printed, typed, handwritten, degraded, tabular, multilingual, and spatially complex material present in the archive.
 3. Measure character and word error rates where appropriate.
 4. Separately record silent corrections, invented text, omitted text, uncertainty handling, layout fidelity, cost, and latency.
 5. Preserve the exact model, endpoint or route, parameters, prompt, source digest, and scoring method for every comparison.
 6. Treat model rankings as corpus- and version-specific, not permanent declarations of a universal “best” model.
 Benchmark material containing family records remains private application data unless explicitly approved for publication.
 ## 6. Ownership and Change Policy
 1. Versioned architecture, schema, scope, and implementation documents define how a release satisfies this invariant.
 2. Provider adapters own the capture of provider-boundary evidence.
 3. Services own validation, persistence, retention, and export behavior.
 4. UI pages may inspect evidence through service contracts but do not define evidence semantics.
 5. If implementation conflicts with this invariant, either correct the implementation or explicitly revise this document before accepting the behavior.
 6. Revisions to this document require deliberate review because they change the long-term preservation contract.
 ## 7. Related Invariants
 - [Historical Document Transcription Design Intent](intent.md)
 - [Transcription Methodology & Style Guide](transcription_methodology.md)
 - [UI Style Guide](ui_style_guide.md)
@@ -0,0 +1,101 @@
 # Error Handling (Invariant)
 ## 1. Purpose
 This document defines the non-negotiable failure-handling principles for the transcription application.
 Error categories, API envelopes, status codes, framework integrations, and persistence fields may change between versions. Failures must nevertheless remain visible, safe, diagnosable, and consistent across every application boundary.
 ## 2. Core Invariants
 ### 2.1 Failures Are Visible
 1. An operation must not report success when all or part of the requested work failed.
 2. Invalid input, unavailable dependencies, persistence failures, provider failures, and unexpected defects must be surfaced through the application's established error path.
 3. Code must not silently discard an exception, provider response, invalid value, or failed state transition.
 4. When work can partially succeed, the successful and failed portions must be identified separately.
 ### 2.2 Messages Are Actionable
 1. Operator-facing errors must explain what failed in concise language.
 2. When a safe corrective action is known, the error must state it.
 3. Expected validation or conflict failures must not be presented as unexplained internal defects.
 4. Internal diagnostics must not replace a usable operator-facing message.
 ### 2.3 Errors Have Stable Identity and Classification
 1. Every surfaced failure must have a stable correlation identifier or equivalent trace identity.
 2. Failures must be classified into a documented, machine-readable category.
 3. Boundary-specific representations must preserve the original category and correlation identity.
 4. Unknown exceptions must be converted at an explicit boundary, retain their causal chain for diagnostics, and be classified as unexpected rather than disguised as an expected failure.
 ### 2.4 Boundary Translation Is Consistent
 1. UI, API, service, worker, persistence, and provider boundaries must use one shared error model or deterministic translations between documented models.
 2. A boundary may simplify presentation, but it must not change the meaning, retryability, or identity of a failure.
 3. Domain and service code must not depend on UI notifications or HTTP response types.
 4. UI and API layers must not infer error categories by parsing message text.
 ### 2.5 State Changes Are Safe
 1. A failed atomic operation must leave persisted state unchanged.
 2. Batch operations may preserve successful independent items only when partial success is an explicit part of the workflow contract.
 3. A failed item must retain enough state to identify what was attempted and whether retry is safe.
 4. Error handling must not overwrite earlier successful results or historical execution evidence.
 ### 2.6 Retry Is Explicit and Bounded
 1. Validation, authorization, policy, conflict, and other deterministic failures must not be retried automatically without a relevant input or state change.
 2. Automatic retry is permitted only for failures classified as transient and only when the operation is idempotent or otherwise protected from duplicate effects.
 3. Retry count, delay, and terminal behavior must be bounded and observable.
 4. Exhausted retries must end in a visible terminal failure rather than an indefinitely pending state.
 ### 2.7 Diagnostics Are Preserved Safely
 1. Logs and persisted diagnostic evidence must retain enough context to correlate the failure with the affected operation and record.
 2. Provider and infrastructure failures must preserve safe diagnostic evidence at the boundary where it is available.
 3. Credentials, authorization headers, cookies, secret values, and unnecessary personal data must not appear in errors, logs, notifications, or exports.
 4. Diagnostic metadata capture must use explicit safe-field allowlists where unrestricted content could contain secrets.
 5. User-facing messages must not expose stack traces, local filesystem details, database credentials, or raw internal exceptions.
 AI execution failures also follow the evidence rules in [Digital Evidence and AI Processing Provenance](ai_evidence_and_provenance.md).
 ### 2.8 Cancellation and Timeout Are Distinct Outcomes
 1. User cancellation, application shutdown, local timeout, remote timeout, and provider rejection must remain distinguishable.
 2. Cancellation must not be converted into success or a generic unexpected error.
 3. Timeout handling must identify whether a provider response was received when that fact is known.
 4. Cleanup after cancellation or timeout must preserve consistency and must not conceal a completed side effect.
 ### 2.9 Logging Must Support Audit Without Becoming the Record
 1. Structured logs must include correlation identity, operation, category, and relevant non-secret record identifiers.
 2. Expected operator errors may be logged less severely than unexpected defects, but they must remain observable.
 3. Logs are operational diagnostics and do not replace required database state or archival evidence.
 4. Duplicate logging of the same failure at every layer should be avoided; ownership of the authoritative log event must be clear.
 ## 3. Verification Policy
 Each version must verify:
 1. Every documented error category reaches the intended UI and API representation.
 2. Failed atomic writes roll back completely.
 3. Partial-success workflows preserve successful independent results and identify failed items.
 4. Retry behavior is bounded and restricted to eligible failures.
 5. Unexpected exceptions retain correlation and causal information without exposing sensitive details.
 6. Logs, persisted evidence, UI messages, and exports contain no credentials.
 7. Cancellation, timeout, provider response failure, and no-response failure remain distinguishable.
 ## 4. Versioned Ownership
 1. Version-specific error taxonomies, envelopes, HTTP mappings, model fields, and framework behavior belong in the applicable version documentation.
 2. Each versioned error-handling document must state how it satisfies this invariant.
 3. A version may add stricter safeguards but must not weaken these principles without first revising this invariant deliberately.
 4. Implementation and tests must be updated together when a versioned error contract changes.
 ## 5. Related Invariants
 - [Historical Document Transcription Design Intent](intent.md)
 - [Digital Evidence and AI Processing Provenance](ai_evidence_and_provenance.md)
 - [UI Style Guide](ui_style_guide.md)
@@ -0,0 +1,25 @@
 # Historical Document Transcription Design Intent
 I have several thousand pages of family history told through letters, postcards, books, and other documents that I want to transcribe to text. 
 ---
 ## Goals
 1. Preserve our family history
 2. Unburden my family (and descendants) from having to store and care for the physical media. Once the documents have been transcribed and organized, they can be donated (or kept by a family member that wants to retain and preserve them).
 3. Make the document text easily available and easily searchable.
 4. Ability create timelines for individuals and/or families through document dates or the data contained in them. Perhaps even use AI to generate biographies or family histories.
 ---
 ## Source material
 1. **letters, cards, diaries** - handwritten; mostly stored in boxes and tubs with little organization
 2. **books** - typed or typeset; mostly self-published books 50-100 pages in length. This may be expanded to include selected pages from other publications.
 3. **photos** - notes written on the backs of photos and the pages of photo albums
 4. **other ephemera** - newspaper clippings, event programs, invitations, military records, immigration records, etc
 ---
 ## Methodology
 1. Follow current best practices per **A Guide to Documentary Editing** by Mary-Jo Kline. (See [Transcription Methodology](transcription_methodology.md))
@@ -0,0 +1,54 @@
 # Transcription Methodology & Style Guide
 ## 1. Overview & Core Philosophy
 This document defines the formal transcription standard for processing historical manuscripts, letters, diaries, and printed ephemera.
 Following the principles established by Mary-Jo Kline in A Guide to Documentary Editing, this project adheres to a Strict Literal Transcription (Verbatim) model as its foundational layer. The primary goal is total textual fidelity—capturing what the author wrote, not what they intended to write—while ensuring the output remains machine-readable and indexable for downstream digital query and search systems.
 ## 2. Textual Policy
 Transcribers (human or AI) must record the exact text of the source document without silent corrections, modernizations, or stylistic smoothing except where explicitly instructed in this guide.
 * **Substantives:** Words, letter forms, structural layout, and semantic content must be recorded strictly as presented in the original document.
 * **Accidentals:** Punctuation, capitalization, misspellings, and archaic character representations must be preserved unless an explicit rule below allows for standardization.
 ## 3. Standard Transcription Rules & Markup
 The following rules map directly to editorial conventions for handling common manuscript anomalies and physical document features.
 ### 3.1 Textual Anomalies & Corrections
 | Document Feature | Rule | Standard Markup Format | Output Example |
 | --- | --- | --- | --- |
 | **Misspellings & Errors** | Retain original spelling verbatim. Insert an italicized [sic] immediately following the error. Do not correct spelling silently. | [sic] | The weather was very cold and publick [sic] business delayed. |
 | **Missing Words / Omissions** | Insert necessary words required to restore basic grammatical sense inside square brackets. | [word] | We went [to] the store to buy supplies. |
 | **Uncertain / Conjectural** | Place best hypothesis followed by a question mark inside square brackets when handwriting is doubtful. | [word?] | He went to [Boston?] yesterday to meet the governor. |
 | **Completely Illegible** | Use [illegible] for unreadable script. Use explicit damage descriptors when physical impairment prevents reading. | [illegible] or [reason] | The total cost was [illegible] dollars. or The letter ends here [remainder of page torn]. |
 | **Canceled / Struck-through** | Wrap text removed by the author inside a [deleted: ...] tag to preserve authorial revisions. | [deleted: text] | We left at [deleted: noon] one o'clock instead. |
 | **Interlineations / Additions** | Wrap text inserted above, below, or in margins into the narrative flow inside an [inserted: ...] tag. | [inserted: text] | The [inserted: red] house on the hill was abandoned. |
 ### 3.2 Typography, Characters & Layout
 | Document Feature | Rule | Standard Markup Format | Output Example |
 | --- | --- | --- | --- |
 | **Superscripts & Abbreviations** | Bring raised letters down to the main line. Optionally expand abbreviations within square brackets based on project configuration. | [expanded] | Gen^l becomes Genl or Gen[era]l. |
 | **Line-End Hyphenation** | Rejoin words split across a page or line boundary silently, dropping the soft hyphen. | Silently rejoin | Original: "estab- / lishment" becomes establishment |
 | **Capitalization** | Preserve explicit capitalization. Default to modern capitalization rules only when authorial intent is ambiguous or archaic forms confuse sentence structure. | Literal / Contextual | If a standard noun like 'Farm' is clearly capitalized, record 'Farm'. If ambiguous, default to 'farm'. |
 | **Hierarchical Outlines** | Preserve exact numbering characters (including lowercase Roman numerals or terminal 'j'). Replicate indentation levels using standard spacing. Do not correct sequence or mathematical errors. | Preserve syntax | I. Main Topic  a. Sub-point  b. Next pointIII. [sic] Third Topic |
 ### 3.3 Visual & Spatial Elements
 | Document Feature | Rule | Standard Markup Format | Output Example |
 | --- | --- | --- | --- |
 | **Non-Textual Artifacts** | Record non-textual elements (seals, stamps, sketches, physical damage) using brief descriptive text inside square brackets. | [description] | [wax notary seal attached here] or [sketch of a fort layout] |
 | **Marginalia & Addenda** | Explicitly indicate spatial transitions before transcribing content located in margins or non-standard orientations. | [location:] | [written in left margin:] Do not share this with anyone. |
 ## 4. Prompt Asset Integration
 When executing programmatic transcriptions via LLM APIs or local models, processing instructions must be packaged into single-purpose system prompts aligned with these rules.
 1. **Isolation:** Each transcription prompt file exists as an independent Markdown asset in the repository.
 2. **Deterministic Output:** Prompts must explicitly instruct models to follow the markup standards in Section 3 without introducing conversational wrappers, extra prose, or structural markdown outside the source document's native layout.
 3. **Iterative Scoping:** Rule modifications or edge-case additions must be submitted as isolated delta commits to individual prompt files to maintain clean revision tracking.
@@ -0,0 +1,110 @@
 # UI Style Guide (Invariant)
 ## 1. Purpose
 This guide defines non-negotiable UI styling rules for the transcription application.
 The design system is token-first and class-driven:
 1. Theme tokens are defined in [src/transcription/ui/static/theme.css](src/transcription/ui/static/theme.css).
 2. Python UI code composes semantic classes instead of inline color values.
 3. Pages and components should share a single visual language across Documents, Jobs, People, and Sources flows.
 ## 2. Source of Truth
 Use these files as the style authority:
 1. [src/transcription/ui/static/theme.css](src/transcription/ui/static/theme.css) for color tokens, semantic utility classes, table styles, and viewer surfaces.
 2. [src/transcription/ui/theme.py](src/transcription/ui/theme.py) for runtime NiceGUI theme bridge and shared UI helpers.
 If this document conflicts with implementation, update this document to match the code immediately after intentional style changes.
 ## 3. Core Design Invariants
 1. Flat, high-density surfaces over decorative depth.
 2. Strong content hierarchy with subdued backgrounds and border-based separation.
 3. Viewer area remains the highest contrast region in image/transcription workflows.
 4. Primary actions are consistent and visually recognizable.
 5. Accessible focus rings are always visible for keyboard users.
 ## 4. Token System
 ### 4.1 Palette Tokens
 Base palette variables live under :root in [src/transcription/ui/static/theme.css](src/transcription/ui/static/theme.css):
 1. --palette-carbon-black: #1c2321
 2. --palette-cool-steel: #7d98a1
 3. --palette-blue-slate: #5e6572
 4. --palette-powder-blue: #a9b4c2
 5. --palette-platinum: #eef1ef
 ### 4.2 Semantic Theme Tokens
 Do not style components directly with palette tokens when a semantic token exists.
 Semantic tokens currently include:
 1. --theme-text and --theme-text-muted
 2. --theme-page, --theme-surface, --theme-surface-raised, --theme-surface-muted
 3. --theme-border
 4. --theme-primary and --theme-primary-hover
 5. --theme-secondary and --theme-focus
 6. --theme-inverse-text
 7. --theme-viewer, --theme-viewer-border, --theme-viewer-muted
 ## 5. Approved Semantic Classes
 ### 5.1 Text and Background
 1. ui-text-primary
 2. ui-text-muted
 3. ui-text-inverse
 4. ui-bg-page
 5. ui-bg-surface
 6. ui-bg-surface-raised
 7. ui-bg-surface-muted
 8. ui-bg-viewer
 9. ui-bg-viewer-overlay
 10. ui-bg-viewer-overlay-soft
 ### 5.2 Borders and Surfaces
 1. ui-border-subtle
 2. ui-border-viewer
 3. ui-header-divider
 4. ui-card-surface
 5. ui-row-surface
 6. ui-note-box
 7. ui-card-error
 ### 5.3 Interactive Elements
 1. ui-btn-primary
 2. ui-btn-secondary
 3. ui-link-primary
 4. ui-text-accent
 5. ui-chip-primary
 6. ui-badge-secondary
 7. ui-status and ui-status--<status>
 ### 5.4 Table Patterns
 1. ui-table
 2. ui-table-header
 3. ui-table-body
 Use existing class combinations from [src/transcription/ui/components](src/transcription/ui/components) and [src/transcription/ui/pages](src/transcription/ui/pages) as reference implementations.
 ## 6. Legacy Class Policy
 Legacy `vibe-` presentation classes are prohibited. Use `ui-` semantic classes from `theme.css`.
 ## 7. Prohibited Patterns
 1. Inline hex colors in Python UI class strings or style blocks, except in isolated bridge code explicitly marked for migration.
 2. Ad-hoc one-off class names that duplicate existing semantic class intent.
 3. Page-specific palette forks that bypass theme tokens.
 4. Hidden or low-contrast focus states on interactive controls.
 5. Embedded `<style>` blocks or NiceGUI `.style(...)` calls in Python UI code.
 6. Additional page- or component-specific stylesheets; `theme.css` is the single CSS source.
 ## 8. Implementation Rules For Contributors
 1. Prefer composing existing semantic classes before creating new ones.
 2. If a new class is required, add it to [src/transcription/ui/static/theme.css](src/transcription/ui/static/theme.css) with a semantic name, then reuse it.
 3. Keep behavior ownership in Python and appearance ownership in CSS.
 4. Update UI tests that assert exact text or labels when intentional copy changes are made.
 5. Avoid introducing class churn unrelated to the feature being changed.
 ## 9. Verification Checklist
 Before merging UI changes, verify:
 1. No new inline hex colors were introduced in UI pages/components.
 2. New styles are token-backed and added to [src/transcription/ui/static/theme.css](src/transcription/ui/static/theme.css).
 3. Primary buttons, links, cards, and tables still render with consistent semantics.
 4. Keyboard focus ring visibility is preserved.
 5. Relevant UI and integration tests pass.
@@ -1,572 +0,0 @@
 # Step 1 Implementation Plan: `config.py` + `models.py` + `db.py`
 ## Purpose
 Establish the foundational data layer and configuration system that every subsequent MVP step builds on. At the end of this step, the project has a runnable Python package with a validated schema, typed configuration, and a test suite proving the data layer works — before any UI, worker, or AI provider code exists.
 ---
 ## 1. Prerequisite: Project Structure Scaffolding
 Before writing any logic, create the package skeleton so imports work correctly.
 ### Files to create (empty `__init__.py` stubs)
 ```
 src/
 └── transcription/
    ├── __init__.py
    ├── providers/
    │   └── __init__.py
    ├── services/
    │   └── __init__.py
    └── ui/
        └── __init__.py
 ```
 ### Files to create (with logic — the Step 1 deliverables)
 ```
 src/transcription/config.py
 src/transcription/models.py
 src/transcription/db.py
 ```
 ### Test files to create
 ```
 tests/
 ├── __init__.py
 ├── conftest.py
 ├── test_config.py
 ├── test_models.py
 └── test_db.py
 ```
 ### Update `pyproject.toml`
 Add the dependencies that Step 1 requires and won't change later:
 ```toml pyproject.toml
 [project]
 name = "transcription"
 version = "0.1.0"
 description = "Historical document transcription system"
 readme = "README.md"
 requires-python = ">=3.12"
 dependencies = [
    "openrouter>=0.7.0",
    "pydantic>=2.13.4",
    "pydantic-settings>=2.9.1",
    "sqlmodel>=0.0.25",
 ]
 [project.optional-dependencies]
 dev = [
    "pytest>=8.0",
    "pytest-asyncio>=0.25",
 ]
 [tool.pytest.ini_options]
 addopts = "--strict-markers -q"
 markers = [
    "unit: pure logic tests with no external dependencies",
    "integration: tests that touch framework or database contracts",
    "external: tests that call external services (slow, requires credentials)",
 ]
 ```
 Key additions:
 - **`openrouter`** — official OpenRouter Python SDK used for model calls
 - **`pydantic-settings`** — for `BaseSettings` with env-var loading (this was split out of `pydantic` core in v2)
 - **`sqlmodel`** — provides SQLModel (which bundles SQLAlchemy + Pydantic model integration) and the SQLite driver
 - **`pytest` + `pytest-asyncio`** — in `dev` extras for test execution
 - **`[tool.pytest.ini_options]`** — strict marker checking enabled from the start; markers registered upfront per pytesting skill conventions
 ### Delete `hello.py`
 The placeholder file is no longer needed.
 ---
 ## 2. `config.py` — Centralized Configuration
 **Satisfies:** REQ-8 (centralized config and logging at startup)
 ### Design Decisions
 | Decision | Rationale |
 |----------|-----------|
 | Use `pydantic-settings` `BaseSettings` | Type-safe, validates on construction, loads from env vars and `.env` files automatically |
 | `PROVIDER` constrained to `openrouter` for MVP | Keeps configuration explicit while avoiding premature multi-provider complexity |
 | `OPENROUTER_API_KEY` required | Matches official SDK docs and avoids ambiguous provider-agnostic naming |
 | `PROVIDER_MODEL` defaults to `None` | OpenRouter adapter (Step 3) supplies a sensible default when `None` |
 | `OPENROUTER_HTTP_REFERER` and `OPENROUTER_APP_TITLE` optional | Matches SDK optional app-attribution fields |
 | `DATABASE_URL` defaults to SQLite | Zero-setup local development; PostgreSQL swap is a single env-var change post-MVP |
 | `UPLOAD_DIR` and `PROMPT_DIR` as `Path` objects | Enables `.mkdir(parents=True, exist_ok=True)` and path validation at startup |
 | Logging configured via `logging.config.dictConfig` in `setup_logging()` | Centralized, explicit formatter/handler/root logger topology; called once at startup with `disable_existing_loggers=False` |
 ### Proposed Implementation
 ```python src/transcription/config.py
 """Centralized application configuration.
 All settings are loaded from environment variables (or a .env file)
 once at startup. Provider-specific defaults (model names, base URLs)
 are resolved by the provider adapters, not here.
 """
 from enum import StrEnum
 from functools import lru_cache
 from pathlib import Path
 import logging
 import logging.config
 from pydantic_settings import BaseSettings, SettingsConfigDict
 class Provider(StrEnum):
    OPENROUTER = "openrouter"
 class Settings(BaseSettings):
    model_config = SettingsConfigDict(
        env_file=".env",
        env_file_encoding="utf-8",
        extra="ignore",
    )
    # --- AI provider ---
    provider: Provider = Provider.OPENROUTER
    openrouter_api_key: str
    provider_model: str | None = None
    openrouter_http_referer: str | None = None
    openrouter_app_title: str | None = None
    # --- persistence ---
    database_url: str = "sqlite:///./transcription.db"
    # --- filesystem paths ---
    upload_dir: Path = Path("./uploads")
    prompt_dir: Path = Path("./prompts")
 LOGGING_CONFIG: dict[str, object] = {
    "version": 1,
    "disable_existing_loggers": False,
    "formatters": {
        "standard": {
            "format": "%(asctime)s | %(levelname)-8s | %(name)s | %(message)s",
            "datefmt": "%Y-%m-%d %H:%M:%S",
        }
    },
    "handlers": {
        "console": {
            "class": "logging.StreamHandler",
            "formatter": "standard",
            "stream": "ext://sys.stdout",
        }
    },
    "root": {
        "level": "INFO",
        "handlers": ["console"],
    },
 }
@lru_cache(maxsize=1)
 def get_settings() -> Settings:
    """Return the singleton Settings instance.
    Cached so the entire application shares one validated config.
    """
    return Settings()
 def setup_logging() -> None:
    """Configure root logging once at startup."""
    logging.config.dictConfig(LOGGING_CONFIG)
 ```
 ### Key Behaviors
 - **Startup validation**: If `OPENROUTER_API_KEY` is missing from the environment, `Settings()` raises a `ValidationError` immediately — the app won't start with a missing key.
 - **`.env` support**: Developers can create a `.env` file in the project root for local keys; it's never committed (already covered by the existing `.gitignore` pattern or a new entry).
 - **`extra="ignore"`**: Unknown env vars don't cause errors, keeping the config resilient to unrelated environment variables.
 - **`lru_cache`**: `get_settings()` is the single access point. All modules import and call this function rather than constructing `Settings` directly.
 - **Centralized logging**: `setup_logging()` calls `dictConfig` exactly once at startup; all modules should use `logging.getLogger(__name__)` and avoid `basicConfig`.
 ### `.env` template (not committed — add to `.gitignore`)
 ```bash .env.example
 PROVIDER=openrouter
 OPENROUTER_API_KEY=sk-or-...
 # PROVIDER_MODEL=          # optional: OpenRouter adapter supplies default
 # OPENROUTER_HTTP_REFERER=https://example.com
 # OPENROUTER_APP_TITLE=Historical Transcription MVP
 # DATABASE_URL=sqlite:///./transcription.db
 # UPLOAD_DIR=./uploads
 # PROMPT_DIR=./prompts
 ```
 ### `.gitignore` addition
 ```gitignore .gitignore
 # ... existing entries ...
 # Environment secrets
 .env
 ```
 ---
 ## 3. `models.py` — SQLModel Domain Models
 **Satisfies:** REQ-3 (persist and expose job states), REQ-4 (persist transcription output and failure details)
 ### Design Decisions
 | Decision | Rationale |
 |----------|-----------|
 | Three models: `Document`, `Job`, `Transcript` | Minimal set from MVP Feature 5. One-to-many from Document→Job and one-to-one from Job→Transcript |
 | `JobStatus` as a `StrEnum` | Readable in the database (`"queued"` not `1`), type-safe in Python, trivially serializable to JSON for the UI |
 | Status values: `queued`, `processing`, `transcribed`, `failed` | Matches MVP Feature 2 lifecycle. REQ-3 also lists `upload` and `completed` — these are deferred to post-MVP when revision/review workflows exist |
 | UUIDs for primary keys | Avoids auto-increment collision concerns if we later move to PostgreSQL; safe for distributed ID generation; `uuid4` is simple |
 | `uploaded_at`, `created_at`, `updated_at` as UTC `datetime` | Timezone-naive UTC by convention for MVP. Sufficient for single-user, single-timezone operation |
 | `Transcript.text` is nullable | A failed job creates a Transcript with `text=None` and `error_detail` populated, keeping the query model uniform |
 | Relationships via SQLModel `Relationship` | Enables `document.jobs` and `job.transcript` navigation in service code without manual joins |
 ### Proposed Implementation
 - `resource://skills/fastapi-async-sqlalchemy-modernization/document`
 ```python src/transcription/models.py
 """SQLModel domain models for the transcription system.
 Three models capture the MVP lifecycle:
  Document  →  one-to-many  →  Job  →  one-to-one  →  Transcript
 """
 from datetime import datetime, timezone
 from enum import StrEnum
 from uuid import UUID, uuid4
 from sqlmodel import Field, Relationship, SQLModel
 class JobStatus(StrEnum):
    QUEUED = "queued"
    PROCESSING = "processing"
    TRANSCRIBED = "transcribed"
    FAILED = "failed"
 class Document(SQLModel, table=True):
    """An uploaded document image."""
    id: UUID = Field(default_factory=uuid4, primary_key=True)
    filename: str
    file_path: str
    uploaded_at: datetime = Field(
        default_factory=lambda: datetime.now(timezone.utc),
    )
    # --- relationships ---
    jobs: list["Job"] = Relationship(back_populates="document")
 class Job(SQLModel, table=True):
    """A transcription job tied to a single document."""
    id: UUID = Field(default_factory=uuid4, primary_key=True)
    document_id: UUID = Field(foreign_key="document.id")
    status: JobStatus = Field(default=JobStatus.QUEUED)
    created_at: datetime = Field(
        default_factory=lambda: datetime.now(timezone.utc),
    )
    updated_at: datetime = Field(
        default_factory=lambda: datetime.now(timezone.utc),
    )
    # --- relationships ---
    document: Document = Relationship(back_populates="jobs")
    transcript: "Transcript | None" = Relationship(back_populates="job")
 class Transcript(SQLModel, table=True):
    """The output of a transcription job."""
    id: UUID = Field(default_factory=uuid4, primary_key=True)
    job_id: UUID = Field(foreign_key="job.id", unique=True)
    text: str | None = None
    error_detail: str | None = None
    created_at: datetime = Field(
        default_factory=lambda: datetime.now(timezone.utc),
    )
    # --- relationships ---
    job: Job = Relationship(back_populates="transcript")
 ```
 ### Entity-Relationship Summary
 ```
 ┌──────────┐       ┌──────────┐       ┌─────────────┐
 │ Document │ 1───* │   Job    │ 1───1 │ Transcript  │
 ├──────────┤       ├──────────┤       ├─────────────┤
 │ id (PK)  │       │ id (PK)  │       │ id (PK)     │
 │ filename │       │ doc_id   │──FK──▶│ job_id (FK) │
 │ file_path│       │ status   │       │ text        │
 │ uploaded │       │ created  │       │ error_detail│
 │          │       │ updated  │       │ created     │
 └──────────┘       └──────────┘       └─────────────┘
 ```
 ### Why Only Four Status Values
 REQ-3 lists six states: `upload`, `queued`, `processing`, `transcribed`, `failed`, `completed`. The MVP simplifies this:
 | REQ-3 State | MVP Treatment |
 |-------------|---------------|
 | `upload` | Implicit — the Document record exists before a Job is created. No separate job state needed. |
 | `queued` | ✅ Included — job created, waiting for worker pickup |
 | `processing` | ✅ Included — worker is actively transcribing |
 | `transcribed` | ✅ Included — AI output received and stored |
 | `failed` | ✅ Included — error captured |
 | `completed` | Deferred — implies human review/acceptance. In MVP, `transcribed` is the terminal success state. |
 ---
 ## 4. `db.py` — Database Engine and Session Management
 **Satisfies:** MVP Feature 5 (SQLite auto-created on first startup)
 ### Design Decisions
 | Decision | Rationale |
 |----------|-----------|
 | Module-level `create_engine` + `Session` factory | REQ-7 (lifespan-owned resources) is deferred. A module-level engine is adequate for MVP's single-process, single-user operation |
 | `create_all()` as an explicit function | Called at app startup. MVP auto-creates tables (REQ-10 deferred), but the function is isolated so it's easy to gate behind a flag later |
 | `get_session()` as a generator | Standard FastAPI/SQLModel pattern — yields a session, ensures cleanup. Compatible with `Depends()` when the API layer arrives in Step 5 |
 | `echo=False` default | Keeps logs clean. Can be toggled for debugging |
 ### Proposed Implementation
 ```python src/transcription/db.py
 """Database engine, session factory, and schema bootstrap.
 MVP uses SQLite with auto-create-tables at startup.
 PostgreSQL migration is a post-MVP configuration change.
 """
 import contextlib
 from collections.abc import Generator
 from sqlmodel import Session, SQLModel, create_engine
 from transcription.config import get_settings
 def _build_engine():
    settings = get_settings()
    connect_args = {}
    if settings.database_url.startswith("sqlite"):
        connect_args["check_same_thread"] = False
    return create_engine(
        settings.database_url,
        echo=False,
        connect_args=connect_args,
    )
 engine = _build_engine()
 def create_all() -> None:
    """Create all tables. Called once at application startup."""
    SQLModel.metadata.create_all(engine)
@contextlib.contextmanager
 def get_session() -> Generator[Session]:
    """Yield a database session and ensure cleanup."""
    with Session(engine) as session:
        yield session
 ```
 ### SQLite-Specific Note
 `check_same_thread=False` is required for SQLite when the session may be accessed from different threads (e.g., a background worker on a different thread than the request handler). This setting is harmless and ignored for PostgreSQL connection strings.
 ---
 ## 5. Test Plan
 Refer to these resources for rules and guidelines about structure:
 - `resource://skills/pytesting/document`
 - `resource://catalog/prompts/pytest-scaffold`
 - `resource://catalog/prompts/pytest-fill-scaffold`
 Hierarchy pattern used in this step:
 ```text
 tests/
    conftest.py
    test_config.py
        TestSettingsLoading
            test_loads_from_env
            test_requires_api_key
        TestProviderSettings
            test_defaults_to_openrouter
            test_rejects_invalid_value
            test_optional_fields_default_to_none
        TestPathSettings
            test_path_fields_are_path_objects
    test_models.py
        TestDocumentModel
            test_can_be_persisted
            test_defaults_are_populated
        TestJobModel
            test_can_be_created_for_document
            test_defaults_are_populated
            test_transitions_to_transcribed
            test_transitions_to_failed
        TestTranscriptModel
            test_success_record_persists
            test_failure_record_persists
            test_job_id_is_unique
        TestRelationships
            test_document_exposes_jobs
            test_job_exposes_transcript
    test_db.py
        TestSchemaBootstrap
            test_create_all_creates_expected_tables
        TestSessionFactory
            test_get_session_yields_session
            test_session_is_closed_after_generator_exit
 ```
 ### `tests/conftest.py` — Shared Fixtures
 ```python tests/conftest.py
 """Shared test fixtures.
 Every test gets a fresh in-memory SQLite database so tests are
 isolated, fast, and leave no artifacts on disk.
 """
 import pytest
 from sqlmodel import Session, SQLModel, create_engine
 from sqlmodel.pool import StaticPool
@pytest.fixture
 def session():
    """Provide a clean database session for each test."""
    engine = create_engine(
        "sqlite://",
        connect_args={"check_same_thread": False},
        poolclass=StaticPool,
    )
    SQLModel.metadata.create_all(engine)
    with Session(engine) as session:
        yield session
 ```
 `StaticPool` ensures a single in-memory SQLite connection is shared across threads, which is required when `TestClient` (Step 5) spawns threads that would otherwise get separate in-memory databases. Establishing it now keeps the fixture stable across all future steps.
 ### `tests/test_config.py` — Configuration Hierarchy
 | Class | Method | What It Verifies |
 |------|--------|------------------|
 | `TestSettingsLoading` | `test_loads_from_env` | `Settings` constructs successfully when `OPENROUTER_API_KEY` is set via env var |
 | `TestSettingsLoading` | `test_requires_api_key` | `Settings()` raises `ValidationError` when `OPENROUTER_API_KEY` is missing |
 | `TestProviderSettings` | `test_defaults_to_openrouter` | Default provider is `openrouter` when not explicitly set |
 | `TestProviderSettings` | `test_rejects_invalid_value` | Setting `PROVIDER=invalid` raises `ValidationError` |
 | `TestProviderSettings` | `test_optional_fields_default_to_none` | `provider_model`, `openrouter_http_referer`, and `openrouter_app_title` are `None` when unset |
 | `TestPathSettings` | `test_path_fields_are_path_objects` | `upload_dir` and `prompt_dir` are `Path` instances |
 ### `tests/test_models.py` — Model & Relationship Hierarchy
 | Class | Method | What It Verifies |
 |------|--------|------------------|
 | `TestDocumentModel` | `test_can_be_persisted` | A `Document` can be persisted and read back with correct fields |
 | `TestDocumentModel` | `test_defaults_are_populated` | `id` is auto-generated UUID, `uploaded_at` is populated |
 | `TestJobModel` | `test_can_be_created_for_document` | A `Job` linked to a `Document` via FK persists correctly |
 | `TestJobModel` | `test_defaults_are_populated` | Default status is `queued`, `created_at` and `updated_at` are populated |
 | `TestJobModel` | `test_transitions_to_transcribed` | Status can be updated from `queued` → `processing` → `transcribed` |
 | `TestJobModel` | `test_transitions_to_failed` | Status can be updated from `processing` → `failed` |
 | `TestTranscriptModel` | `test_success_record_persists` | A `Transcript` with `text` set and `error_detail=None` persists correctly |
 | `TestTranscriptModel` | `test_failure_record_persists` | A `Transcript` with `text=None` and `error_detail` set persists correctly |
 | `TestRelationships` | `test_document_exposes_jobs` | `document.jobs` returns the linked `Job` list |
 | `TestRelationships` | `test_job_exposes_transcript` | `job.transcript` returns the linked `Transcript` |
 | `TestTranscriptModel` | `test_job_id_is_unique` | Inserting two transcripts with the same `job_id` raises an integrity error |
 ### `tests/test_db.py` — Database Bootstrap Hierarchy
 | Class | Method | What It Verifies |
 |------|--------|------------------|
 | `TestSchemaBootstrap` | `test_create_all_creates_expected_tables` | After `create_all()`, the expected tables (`document`, `job`, `transcript`) exist in the database |
 | `TestSessionFactory` | `test_get_session_yields_session` | `get_session()` yields a usable `Session` object |
 | `TestSessionFactory` | `test_session_is_closed_after_generator_exit` | After the generator is exhausted, the session is closed |
 ### Marker Strategy (Step 1)
 - Markers (`unit`, `integration`, `external`) are registered upfront in `pyproject.toml` with `--strict-markers` enabled, per pytesting skill conventions.
 - All Step 1 tests are unmarked — they run in the default lane since they are fast, deterministic, and have no external dependencies.
 - When slower integration or external tests are introduced in later steps, apply explicit markers and keep test names unchanged.
 ### Test Workflow
 Follow the two-phase approach from `resource://catalog/prompts/pytest-scaffold` and `resource://catalog/prompts/pytest-fill-scaffold`:
 1. **Scaffold phase**: Create test files with class hierarchy, method names, and one-line docstrings only. Validate collection:
   - `uv run pytest --collect-only -q`
 2. **Fill phase**: Implement assertions, fixtures, and minimal test data. Treat scaffolded names and docstrings as locked. Validate execution:
   - `uv run pytest -q`
 Scaffolded structure is treated as a stable baseline — do not rename, move, merge, split, or re-nest tests once the scaffold is reviewed.
 ---
 ## 6. Step 1 Completion Checklist
 When all of the following are true, Step 1 is done and Step 2 can begin:
 | # | Criterion | How to Verify |
 |---|-----------|---------------|
 | 1 | `src/transcription/` package exists with `config.py`, `models.py`, `db.py` | `ls` / file inspection |
 | 2 | Empty `__init__.py` stubs exist for `providers/`, `services/`, `ui/` | `ls` / file inspection |
 | 3 | `Settings` loads from environment and validates `OPENROUTER_API_KEY` is present | `test_config.py` passes |
 | 4 | `Document`, `Job`, `Transcript` models create tables in SQLite | `test_models.py` passes |
 | 5 | `JobStatus` enum has exactly four values: `queued`, `processing`, `transcribed`, `failed` | `test_models.py` passes |
 | 6 | Foreign key relationships work: Document→Job→Transcript | `test_models.py` passes |
 | 7 | `create_all()` bootstraps the schema; `get_session()` yields a working session | `test_db.py` passes |
 | 8 | All tests pass: `uv run pytest -q` | CI / local run |
 | 9 | `hello.py` is deleted | File inspection |
 | 10 | `pyproject.toml` includes `openrouter`, `sqlmodel`, `pydantic-settings`, `pytest`, `pytest-asyncio` | File inspection |
 | 10a | `pyproject.toml` has `[tool.pytest.ini_options]` with `--strict-markers` and registered markers | File inspection |
 | 11 | `.env.example` documents all config vars; `.env` is in `.gitignore` | File inspection |
 | 12 | `setup_logging()` uses `logging.config.dictConfig` with centralized formatter/handler/root config | File inspection |
 | 13 | `uv run pytest --collect-only -q` shows expected test hierarchy | Local run |
 | 14 | `uv run pytest -q` passes all tests | Local run |
 ---
 ## 7. What This Step Does NOT Include
 Explicitly out of scope to prevent scope creep:
 | Excluded | Reason |
 |----------|--------|
 | FastAPI / NiceGUI app entrypoint | Step 5 |
 | Additional provider adapters beyond OpenRouter | Post-MVP |
 | Upload service logic | Step 4 |
 | Worker / background processing | Step 4 |
 | Transcription prompt files | Step 2 |
 | Alembic or migration tooling | Post-MVP (REQ-10 deferred) |
 | Async session factory | Post-MVP (REQ-7 deferred) |
 ---
 This plan produces a fully tested, importable data foundation. Every subsequent step imports from `transcription.config`, `transcription.models`, and `transcription.db` without modification.
@@ -1,278 +0,0 @@
 ## Step 2: prompts/transcribe_document.md
 ### Goal
 Implement the MVP prompt artifact system by creating a curated transcription prompt file:
 - `prompts/transcribe_document.md`
 This step primarily satisfies:
 - **REQ-12**: prompts stored as individual Markdown artifacts
 - MVP Feature 3: prompt-driven verbatim transcription behavior grounded in `docs/intent.md`
 ---
 ## Scope for Step 2
 ### In scope
 1. Create prompt artifact directory and first prompt file.
 2. Encode transcription rules from `docs/intent.md` into a model-facing prompt.
 3. Define stable prompt structure so future revisions are easy to diff/review.
 4. Add lightweight tests that validate artifact presence and baseline quality constraints.
 5. Update docs/README references so Step 3 can consume prompt file directly.
 ### Out of scope
 - Provider integration logic (Step 3)
 - Worker/job orchestration (Step 4)
 - UI behavior (Step 5)
 ---
 ## Proposed Deliverables
 1. **`prompts/transcribe_document.md`**
   - production prompt text for historical document transcription
 2. **`prompts/README.md`** (recommended)
   - conventions for prompt files, revision policy, naming
 3. **`tests/test_prompts.py`** (recommended)
   - artifact existence + structure checks
 4. **Small docs update** (README or docs reference)
   - indicate that prompts are file-based and loaded from `PROMPT_DIR`
 ---
 ## Detailed Work Breakdown
 ### 1) Create prompt artifact folder and canonical file
 - Add `prompts/` at repo root.
 - Add `transcribe_document.md` as the first curated artifact.
 - Keep filename stable; this becomes the default in Step 3 unless overridden.
 ### 2) Author prompt content using a strict, sectioned format
 Use section headers so future diffs are clean and policy changes are isolated.
 Suggested sections:
 1. **Purpose**
   - verbatim scholarly transcription of historical documents
 2. **Output requirements**
   - plain text only
   - no summaries, no paraphrasing
   - preserve reading order and meaningful structure
 3. **Core fidelity rules**
   - preserve original wording and punctuation
   - don’t silently normalize grammar/spelling
   - no invented content
 4. **Issue-handling rules (mapped from Intent table)**
   - misspellings with `[sic]`
   - missing words with `[word]`
   - uncertainty with `[guess?]`
   - illegible with `[illegible]` / reason tags
   - crossed-out text as `[deleted: ...]`
   - inserted text as `[inserted: ...]`
   - superscripts handling guidance
   - non-text elements as `[description]`
   - marginalia format `[written in left margin: ...]`
   - line-break hyphen rejoin behavior
   - capitalization policy
   - hierarchical outline preservation (including unusual numbering)
 5. **Confidence/ambiguity policy**
   - prefer explicit uncertainty markers over hallucination
 6. **Final self-checklist for model**
   - did I preserve structure?
   - did I mark uncertain text?
   - did I avoid silent corrections?
 ### 3) Add prompt-library conventions (`prompts/README.md`)
 Recommended conventions:
 - one prompt per file
 - snake_case names
 - each file starts with purpose + behavior contract
 - iterative edits, one prompt per PR where possible
 - no secrets in prompt files
 ### 4) Add tests for prompt assets (`tests/test_prompts.py`)
 Keep tests robust but not brittle.
 Recommended tests:
 1. `test_prompt_file_exists`
 2. `test_prompt_file_is_not_empty`
 3. `test_prompt_mentions_verbatim_behavior`
 4. `test_prompt_includes_uncertainty_and_illegible_markers`
 5. `test_prompt_includes_deleted_and_inserted_conventions`
 Avoid exact full-text matching; verify key semantic anchors only.
 ### 5) Optional config alignment check
 Current config already has:
 - `prompt_dir: Path = Path("./prompts")`
 In Step 2, ensure docs reflect this and that Step 3 will resolve:
 - `PROMPT_DIR / "transcribe_document.md"`
 ---
 ## Task-by-Task Execution Checklist
 ## Phase A — Scaffold files
 - [ ] **A1. Create prompt directory**
  - Path: `prompts/`
  - Verify: directory exists at repo root
 - [ ] **A2. Create canonical prompt file**
  - Path: `prompts/transcribe_document.md`
  - Verify: file exists and is non-empty
 - [ ] **A3. (Recommended) Create prompt library README**
  - Path: `prompts/README.md`
  - Verify: includes naming + revision conventions
 ---
 ## Phase B — Author prompt content (core work)
 - [ ] **B1. Add Purpose section**
  - States verbatim historical transcription objective
  - Explicitly disallows summarization/paraphrase
 - [ ] **B2. Add Output Contract section**
  - Plain text output expectation
  - Preserve meaningful structure and reading order
  - No fabricated text
 - [ ] **B3. Add Rule Set from `docs/intent.md`**
  - Misspellings/errors: `[sic]`
  - Missing words: `[word]`
  - Uncertain readings: `[guess?]`
  - Illegible regions: `[illegible]` / reason labels
  - Crossed-out text: `[deleted: ...]`
  - Squeezed-in text: `[inserted: ...]`
  - Superscripts/abbrev handling guidance
  - Non-text visuals: bracketed descriptive labels
  - Marginalia formatting cue
  - Rejoin line-break hyphenated words silently
  - Ambiguous capitalization policy
  - Hierarchical outline numbering preservation
 - [ ] **B4. Add Ambiguity and Confidence policy**
  - “Mark uncertainty instead of guessing”
  - “Never silently normalize uncertain passages”
 - [ ] **B5. Add Final Self-Check section**
  - Checklist for fidelity, uncertainty labeling, and format compliance
 ---
 ## Phase C — Add validations (tests)
 - [ ] **C1. Create prompt tests file**
  - Path: `tests/test_prompts.py`
 - [ ] **C2. Add existence/health checks**
  - Prompt file exists
  - Prompt file has content (non-whitespace)
 - [ ] **C3. Add semantic anchor checks**
  - Mentions verbatim behavior
  - Mentions uncertainty marker pattern (`?` in brackets conceptually)
  - Mentions illegible handling
  - Mentions deleted/inserted conventions
 - [ ] **C4. Keep tests resilient**
  - Avoid exact full-file snapshot assertions
  - Assert required concepts, not precise phrasing
 ---
 ## Phase D — Documentation alignment
 - [ ] **D1. Update top-level docs/README reference**
  - Mention that prompts live in `prompts/`
  - Mention Step 3 loads from `PROMPT_DIR`
 - [ ] **D2. Confirm config compatibility**
  - `src/transcription/config.py` already uses `prompt_dir = Path("./prompts")`
  - No code change needed unless naming/path mismatch appears
 ---
 ## Phase E — Verification
 - [ ] **E1. Run targeted test file**
  - `uv run pytest tests/test_prompts.py -q`
 - [ ] **E2. Run full suite**
  - `uv run pytest -q`
 - [ ] **E3. Confirm no regressions**
  - All existing tests still green (expected: previous 20 + new prompt tests)
 ---
 ## Phase F — Commit plan (recommended granularity)
 - [ ] **F1. Commit 1: scaffold**
  - `prompts/transcribe_document.md` (initial structure)
  - `prompts/README.md` (if included)
 - [ ] **F2. Commit 2: finalized prompt content**
  - full rule-complete prompt text
 - [ ] **F3. Commit 3: tests + docs alignment**
  - `tests/test_prompts.py`
  - README/docs mention of prompt artifact pattern
 ---
 ## Done Criteria (quick gate)
 - [ ] Canonical prompt exists and is curated for verbatim transcription.
 - [ ] Prompt encodes all high-value handling rules from `docs/intent.md`.
 - [ ] Prompt tests pass.
 - [ ] Full project tests pass with `uv`.
 - [ ] Ready for Step 3 provider integration.
 ---
 ## Acceptance Criteria (Definition of Done)
 Step 2 is complete when all are true:
 1. `prompts/transcribe_document.md` exists and is committed.
 2. Prompt includes all critical handling rules from `docs/intent.md`.
 3. Prompt is structured with stable section headings for future curation.
 4. Prompt tests pass under `uv run pytest -q`.
 5. Existing tests remain green (total suite still passes).
 6. Docs indicate prompt artifact location and curation policy.
 ---
 ## Risks and Mitigations
 1. **Risk: prompt too vague → hallucinated reconstructions**
   - Mitigation: explicit uncertainty/illegible conventions and “no invention” rule.
 2. **Risk: prompt too rigid for mixed document types**
   - Mitigation: include neutral defaults + clear annotation formats.
 3. **Risk: brittle tests block iterative prompt tuning**
   - Mitigation: test semantic anchors, not exact wording.
 ---
 ## Handoff to Step 3
 After Step 2, Step 3 can immediately:
 1. Load `transcribe_document.md` from `PROMPT_DIR`
 2. Inject prompt into OpenRouter request
 3. Start validating real transcription behavior with minimal glue code
@@ -1,236 +0,0 @@
 ## Step 3: services/transcription.py + providers/ 
 ### Objective
 Implement the **AI transcription integration layer** so the app can:
 1. Read the curated prompt from `PROMPT_DIR`
 2. Send prompt + image to the configured provider (OpenRouter)
 3. Return normalized transcription output (or structured failure)
 This corresponds to MVP Step 3 from `docs/mvp.md`:
 - `services/transcription.py`
 - `providers/` adapter(s)
 ---
 ## Scope for Step 3
 ### In scope
 - Provider abstraction and OpenRouter adapter
 - Prompt file loading utility in service layer
 - Image payload preparation
 - One high-level transcription service function usable by Step 4 worker
 - Unit tests (mocked provider SDK, no external calls)
 ### Out of scope
 - Job polling/background loop (Step 4)
 - DB status transition orchestration in worker loop (Step 4)
 - UI invocation/wiring (Step 5)
 ---
 ## Planned Deliverables
 ### Source files
 - `src/transcription/providers/base.py`
 - `src/transcription/providers/openrouter.py`
 - `src/transcription/providers/__init__.py` (exports + factory)
 - `src/transcription/services/transcription.py`
 - `src/transcription/services/__init__.py` (optional export)
 ### Tests
 - `tests/providers/test_openrouter.py`
 - `tests/services/test_transcription.py`
 ### Test directory convention
 - Mirror source domains under `tests/`.
 - Provider adapter tests live under `tests/providers/`.
 - Service-layer tests live under `tests/services/`.
 - Prefer one focused test module per production module (for Step 3: `test_openrouter.py`, `test_transcription.py`).
 ---
 ## Design Decisions (before coding)
 1. **Provider interface first**
   - Define a stable contract independent of SDK specifics.
   - Prevent Step 4 from depending on raw SDK response shapes.
 2. **Service returns normalized result object**
   - Include: `text`, `provider`, `model`, `raw_error`/exception metadata.
   - Worker can map this cleanly to `Transcript` and `JobStatus`.
 3. **Prompt loaded from file at call time**
   - Uses `get_settings().prompt_dir / "transcribe_document.md"`.
   - Keeps prompt edits hot-swappable without code changes.
 4. **Clear exception boundary**
   - SDK/network/model failures become predictable domain exceptions:
     - `ProviderError`
     - `PromptLoadError`
     - `TranscriptionError` (optional top-level wrapper)
 5. **Model resolution policy**
   - Use `settings.provider_model` if set
   - Otherwise use adapter default constant (e.g., vision-capable model slug)
 ---
 ## Task-by-Task Execution Checklist
 ## Phase A — Provider contract
 - [ ] Create `src/transcription/providers/base.py`
 - [ ] Define protocol/ABC for transcription providers:
  - [ ] method signature accepts prompt text + image bytes (or data URL) + mime type
  - [ ] returns normalized text result (and optional metadata)
 - [ ] Define shared provider exceptions:
  - [ ] `ProviderError`
  - [ ] optional subclasses (`ProviderAuthError`, `ProviderResponseError`)
 ---
 ## Phase B — OpenRouter adapter
 - [ ] Create `src/transcription/providers/openrouter.py`
 - [ ] Implement `OpenRouterTranscriptionProvider` with:
  - [ ] config-driven API key usage
  - [ ] optional referer/title attribution headers
  - [ ] model resolution fallback when `provider_model` is unset
 - [ ] Implement request building:
  - [ ] prompt included as instruction content
  - [ ] image included in supported format for vision call
 - [ ] Implement response parsing:
  - [ ] extract final transcript text from SDK response
  - [ ] validate non-empty text
 - [ ] Wrap SDK failures into `ProviderError` with clean message
 ---
 ## Phase C — Provider factory
 - [ ] Update `src/transcription/providers/__init__.py`
 - [ ] Add `get_transcription_provider()` factory:
  - [ ] reads `settings.provider`
  - [ ] returns OpenRouter adapter for `openrouter`
  - [ ] raises explicit error for unsupported provider values
 ---
 ## Phase D — Transcription service (Step 3 core)
 - [ ] Create `src/transcription/services/transcription.py`
 - [ ] Add prompt loader function:
  - [ ] default file: `transcribe_document.md`
  - [ ] raises `PromptLoadError` on missing/empty file
 - [ ] Add image loader/validator:
  - [ ] path existence check
  - [ ] allowed mime detection (`.jpg/.jpeg/.png/.tiff/.pdf` policy aligned to MVP)
 - [ ] Add high-level function (name example):
  - [ ] `transcribe_document_image(image_path, prompt_name="transcribe_document.md")`
  - [ ] loads prompt + image
  - [ ] calls provider from factory
  - [ ] returns normalized transcription result object
 - [ ] Add structured logging at key boundaries:
  - [ ] prompt loaded
  - [ ] provider invoked
  - [ ] success/failure outcome (no sensitive data in logs)
 ---
 ## Phase E — Tests (two-phase scaffold -> fill)
 ### Required execution resources
 Load and reference these directly during test planning/implementation so the two-phase flow is enforced:
 - [ ] `resource://catalog/prompts/pytest-scaffold`
 - [ ] `resource://prompts/pytest-scaffold/document`
 - [ ] `resource://catalog/prompts/pytest-fill-scaffold`
 - [ ] `resource://prompts/pytest-fill-scaffold/document`
 ### Phase E1 — Scaffold test structure first
 Prompt: `resource://catalog/prompts/pytest-scaffold`
 Suggested arguments:
 - [ ] `target_modules` = `src/transcription/providers/openrouter.py`, `src/transcription/services/transcription.py`
 - [ ] `mode` = `scaffold`
 - [ ] `path_strategy` = `src-to-tests-mirror`
 - [ ] `naming_style` = `concise-behavior`
 Expected scaffold outcomes:
 - [ ] `tests/providers/test_openrouter.py` exists with class/method skeletons and one-line docstrings
 - [ ] `tests/services/test_transcription.py` exists with class/method skeletons and one-line docstrings
 - [ ] collection succeeds on scaffold-only tests
 Scaffold coverage targets:
 - [ ] adapter initializes from settings
 - [ ] model fallback when `provider_model is None`
 - [ ] referer/title options included when set
 - [ ] successful SDK response parses transcript text
 - [ ] SDK exception maps to `ProviderError`
 - [ ] empty/invalid response maps to `ProviderError`
 - [ ] prompt loader reads canonical prompt file
 - [ ] missing prompt raises `PromptLoadError`
 - [ ] transcription function loads file and calls provider once
 - [ ] image path missing raises clear error
 - [ ] provider error is propagated/wrapped predictably
 - [ ] returned result includes transcript text and metadata
 ### Phase E2 — Fill scaffolded tests with assertions
 Prompt: `resource://catalog/prompts/pytest-fill-scaffold`
 Suggested arguments:
 - [ ] `target_files` = `tests/providers/test_openrouter.py`, `tests/services/test_transcription.py`
 - [ ] `stack` = `pure-python`
 - [ ] `strategy` = `minimal`
 - [ ] `marker_lane` = `unit`
 Fill constraints:
 - [ ] preserve scaffold class/method names and one-line docstrings
 - [ ] keep mocks to an absolute minimum; mock only network boundaries and non-deterministic failures
 - [ ] keep one behavior target per test method
 > Default suite should remain deterministic and fast, but mocking should be minimal and intentional.
 ### Optional real-endpoint validation lane
 - [ ] Add an opt-in integration lane for real provider calls (for example `@pytest.mark.integration` and `@pytest.mark.live_api`).
 - [ ] Gate live tests behind explicit env vars (for example `OPENROUTER_API_KEY`, optional `RUN_LIVE_API_TESTS=1`).
 - [ ] Exclude live tests from default CI/local runs unless explicitly requested.
 - [ ] Keep at least one thin smoke path that can validate request/response compatibility against the real endpoint.
 ---
 ## Phase F — Verification commands
 - [ ] E1 scaffold validation: `uv run pytest --collect-only -q`
 - [ ] E2 fill validation (unit lane): `uv run pytest -m unit -q`
 - [ ] E2 targeted provider file: `uv run pytest tests/providers/test_openrouter.py -q`
 - [ ] E2 targeted service file: `uv run pytest tests/services/test_transcription.py -q`
 - [ ] E2 final full-suite check: `uv run pytest -q`
 ---
 ## Implementation Notes / Guardrails
 - Avoid coupling Step 3 service to DB models directly (that belongs in Step 4 orchestration).
 - Do not silently swallow provider errors.
 - Keep prompt filename stable (`transcribe_document.md`) unless explicitly parameterized.
 - Keep request/response normalization inside provider adapter, not worker/UI layers.
 ---
 ## Definition of Done (Step 3)
 Step 3 is done when:
 1. Provider abstraction exists and OpenRouter adapter is implemented.
 2. Service can transcribe a local image using prompt file content.
 3. Failures are returned as structured exceptions, not raw SDK traceback noise.
 4. Unit tests for provider and service pass.
 5. Full suite remains green under `uv run pytest -q`.
 6. Step 4 can call a single service function to process queued jobs.
@@ -1,262 +0,0 @@
 ## Step 4: `services/upload.py` + `worker.py`
 ### Objective
 Implement the MVP upload and background-processing pipeline so the system can:
 1. Save uploaded files into `UPLOAD_DIR`
 2. Create `Document` + `Job(status="queued")`
 3. Process queued jobs in a worker loop:
   - `queued -> processing`
   - call Step 3 transcription service
   - persist `Transcript`
   - finalize as `transcribed` or `failed`
 This step advances MVP Feature 1 + Feature 2 and supports REQ-1, REQ-2, REQ-3, REQ-4, REQ-6.
 ---
 ## Scope
 ### In scope
 - `src/transcription/services/upload.py`
 - `src/transcription/worker.py`
 - Upload persistence logic and initial job creation
 - Worker polling and single-job lifecycle execution
 - Deterministic test coverage for upload + worker (default suite)
 ### Out of scope
 - UI integration and pages (Step 5)
 - Queue infrastructure beyond in-process loop
 - Async DB/session architecture refactor
 - Broad production hardening beyond MVP needs
 ---
 ## Planned Deliverables
 ### Source files
 - `src/transcription/services/upload.py`
 - `src/transcription/worker.py`
 - `src/transcription/services/__init__.py` (export updates as needed)
 ### Test files
 - `tests/services/test_upload.py`
 - `tests/services/test_worker.py`
 ### Optional external lane (already present pattern)
 - reuse `external` marker for live-provider checks where appropriate
 - keep external out of default lane
 ---
 ## Required MCP Prompt References (for test workflow)
 Apply these resources directly during Step 4 test creation:
 1. `resource://catalog/prompts/pytest-scaffold`
 2. `resource://prompts/pytest-scaffold/document`
 3. `resource://catalog/prompts/pytest-fill-scaffold`
 4. `resource://prompts/pytest-fill-scaffold/document`
 And (as referenced by those prompts) apply relevant pytest skill references for:
 - naming/hierarchy
 - marker defaults
 - SQLAlchemy sync testing behavior where applicable
 ---
 ## Design Decisions
 1. **Upload service owns initial file + record creation**
   - Writes file, creates `Document`, creates queued `Job`, returns IDs/path.
 2. **Worker owns lifecycle transitions**
   - Worker is the single owner of `queued -> processing -> terminal` job state changes.
 3. **Worker uses Step 3 service boundary**
   - Worker calls `transcribe_document_image(...)`; no provider-specific SDK logic in worker.
 4. **Failure information is always persisted**
   - On failure: store `Transcript(text=None, error_detail=...)` and set `Job.status=failed`.
 5. **Loop remains simple and stoppable**
   - In-process polling loop with stop event and poll interval for MVP simplicity and testability.
 ---
 ## Task-by-Task Execution Checklist
 ## Phase A — Implement upload service (`src/transcription/services/upload.py`)
 - [ ] Create `UploadError` exception
 - [ ] Create `UploadJobResult` dataclass with:
  - [ ] `document_id`
  - [ ] `job_id`
  - [ ] `stored_path`
  - [ ] `original_filename`
 - [ ] Add filename safety handling:
  - [ ] normalize to basename
  - [ ] avoid path traversal
  - [ ] collision-safe stored name (e.g., UUID prefix/suffix)
 - [ ] Validate upload payload:
  - [ ] non-empty bytes required
  - [ ] extension in supported set (`.jpg/.jpeg/.png/.tif/.tiff/.pdf`)
 - [ ] Ensure upload directory exists (`mkdir(parents=True, exist_ok=True)`)
 - [ ] Write file bytes to `UPLOAD_DIR`
 - [ ] Persist DB records in one transaction:
  - [ ] `Document(filename, file_path)`
  - [ ] `Job(document_id=..., status=queued)`
 - [ ] Return `UploadJobResult`
 - [ ] Add logging for success/failure boundaries
 ---
 ## Phase B — Implement worker core (`src/transcription/worker.py`)
 - [ ] Add `process_next_queued_job(...) -> bool`
 - [ ] Fetch oldest queued job
 - [ ] Return `False` when no queued jobs exist
 - [ ] Transition picked job to `processing` and update timestamp
 - [ ] Resolve associated `Document.file_path`
 - [ ] Call `transcribe_document_image(image_path=...)`
 - [ ] On success:
  - [ ] insert/update transcript text
  - [ ] clear error detail
  - [ ] mark job `transcribed`
  - [ ] update timestamp
 - [ ] On failure:
  - [ ] insert/update transcript with `text=None`, `error_detail=...`
  - [ ] mark job `failed`
  - [ ] update timestamp
 - [ ] Commit terminal state and return `True`
 - [ ] Add logs around job pickup, transition, and terminal outcome
 ---
 ## Phase C — Implement worker loop (`src/transcription/worker.py`)
 - [ ] Add `run_worker_loop(...)`
 - [ ] Accept configurable stop event/signal
 - [ ] Accept configurable poll interval
 - [ ] Repeatedly call `process_next_queued_job`
 - [ ] Sleep only when queue is empty
 - [ ] Exit cleanly when stop event is set
 ---
 ## Phase D — Exports
 - [ ] Update `src/transcription/services/__init__.py` to expose upload APIs
 - [ ] Keep existing transcription exports intact
 ---
 ## Phase E — Tests via MCP scaffold -> fill flow
 ## E1 Scaffold (structure only)
 Use scaffold prompt workflow first for:
 - `src/transcription/services/upload.py`
 - `src/transcription/worker.py`
 Expected scaffold targets:
 - `tests/services/test_upload.py`
 - `tests/services/test_worker.py`
 Scaffold rules:
 - [ ] Class hierarchy + method names + one-line docstrings only
 - [ ] No assertions or implementation details in scaffold phase
 - [ ] Keep method names concise and behavior-focused
 Validation:
 - [ ] `uv run pytest --collect-only -q`
 ## E2 Fill scaffold (implementation)
 Use fill prompt workflow for:
 - `tests/services/test_upload.py`
 - `tests/services/test_worker.py`
 - stack: `sqlalchemy-sync` (or `mixed` if combining pure + DB behaviors)
 - marker lane preference: `unit` and `integration` as appropriate
 - strategy: minimal deterministic implementation
 Fill rules (invariants):
 - [ ] Preserve scaffold class names, method names, and one-line docstrings
 - [ ] Do not rename/re-nest scaffolded tests unless explicitly approved
 - [ ] One behavior target per test
 - [ ] Minimal mocking; mock only network/nondeterministic boundaries
 Suggested test coverage:
 ### `tests/services/test_upload.py`
 - [ ] creates file + document + queued job (`integration`)
 - [ ] rejects empty bytes (`unit`)
 - [ ] rejects unsupported extension (`unit`)
 - [ ] writes collision-safe unique filename (`integration`)
 - [ ] persisted job status is `queued` (`integration`)
 ### `tests/services/test_worker.py`
 - [ ] returns `False` when queue empty (`integration`)
 - [ ] transitions `queued -> processing -> transcribed` on success (`integration`)
 - [ ] stores transcript text on success (`integration`)
 - [ ] transitions to `failed` and stores `error_detail` on failure (`integration`)
 - [ ] updates existing transcript instead of duplicate create (`integration`)
 - [ ] worker loop exits when stop event set (`unit`)
 ---
 ## Marker Strategy
 - `unit`: pure logic tests (filename handling, loop stop behavior, validation logic)
 - `integration`: DB + service orchestration tests (SQLite/session/contracts)
 - `external`: opt-in live provider tests only (not part of default Step 4 lane)
 No new marker needed; reuse existing marker registration.
 ---
 ## Validation Sequence (strict order)
 - [ ] `uv run pytest --collect-only -q`
 - [ ] `uv run pytest -m unit -q` *(if unit tests touched)*
 - [ ] `uv run pytest tests/services/test_upload.py -q`
 - [ ] `uv run pytest tests/services/test_worker.py -q`
 - [ ] `uv run pytest -q`
 ---
 ## Reporting Requirements (after implementation)
 Implementation report must include:
 1. Files created/updated
 2. Fixture and marker decisions
 3. MCP references used and why
 4. Validation command results
 5. Remaining risks/open questions (only blockers)
 ---
 ## Guardrails
 - Keep Step 4 independent from UI concerns.
 - Do not call provider SDK directly from worker.
 - Do not silently swallow exceptions.
 - Always persist terminal job outcome.
 - Keep default suite deterministic and fast.
 - Preserve scaffold invariants during fill phase.
 ---
 ## Definition of Done (Step 4)
 Step 4 is complete when:
 - [ ] Upload service writes file and creates `Document` + queued `Job`
 - [ ] Worker processes queued jobs end-to-end using Step 3 transcription service
 - [ ] Success path persists transcript text and sets `transcribed`
 - [ ] Failure path persists error detail and sets `failed`
 - [ ] Queue-empty path returns cleanly
 - [ ] New tests pass and full suite is green (`uv run pytest -q`)
 - [ ] Output report includes MCP reference usage + validation evidence
@@ -1,70 +0,0 @@
 Step 5 is implemented and passing ✅
 ### What I implemented
 #### App/bootstrap
 - Added `src/transcription/app.py`:
  - `create_app()` factory
  - FastAPI lifespan startup/shutdown:
    - `setup_logging()`
    - `create_all()`
    - ensure `upload_dir` and `prompt_dir`
    - start/stop worker thread (`run_worker_loop`)
  - includes health router
  - registers NiceGUI pages
 #### API
 - Added `src/transcription/api/__init__.py`
 - Added `src/transcription/api/health.py`:
  - `GET /healthz` returns `{"status": "ok"}`
 #### UI
 - Added `src/transcription/ui/upload_page.py`:
  - upload page route
  - accepted extensions helper
  - upload submit helper to Step 4 service (`create_upload_job`)
  - success/error state messaging
 - Added `src/transcription/ui/jobs_page.py`:
  - jobs list page
  - job detail page
  - `fetch_jobs()` and `fetch_job_detail()` helpers
 - Updated `src/transcription/ui/__init__.py`:
  - explicit page registration
  - NiceGUI mounted via `ui.run_with(app, mount_path="/ui", ...)`
 #### Dependencies
 - Added runtime dependencies:
  - `fastapi`
  - `nicegui`
 ---
 ### Tests created and filled (scaffold → fill)
 - `tests/test_app.py`
 - `tests/api/test_health.py`
 - `tests/ui/test_pages_registration.py`
 - `tests/ui/test_upload_page.py`
 - `tests/ui/test_jobs_page.py`
 All scaffolded method names/docstrings were preserved and filled with executable assertions.
 ---
 ### Validation results (strict sequence)
 - `uv run pytest --collect-only -q` ✅
 - `uv run pytest -m unit -q` ✅
 - `uv run pytest tests/api/test_health.py -q` ✅
 - `uv run pytest tests/ui/test_pages_registration.py -q` ✅
 - `uv run pytest tests/test_app.py -q` ✅
 - `uv run pytest tests/ui/test_upload_page.py -q` ✅
 - `uv run pytest tests/ui/test_jobs_page.py -q` ✅
 - `uv run pytest -q` ✅
 Full suite is green.
 ---
 ### Note
 You’ll see one warning from FastAPI/Starlette test client about `httpx` deprecation; it does not affect correctness and all tests pass.
@@ -1,310 +0,0 @@
 ## Step 5: `app.py` + UI Pages (NiceGUI + FastAPI composition)
 ## Objective
 Implement the MVP user-facing application layer so users can:
 1. Upload a document from the UI
 2. Trigger Step 4 upload/job creation flow
 3. See live job lifecycle status (`queued`, `processing`, `transcribed`, `failed`)
 4. Open a job detail view to read transcript text or failure details
 This step composes Steps 1–4 into a usable UI.
 ---
 ## Architecture Summary (NiceGUI-aligned)
 Step 5 uses a **FastAPI app factory + lifespan orchestration** and mounts/registers NiceGUI pages via explicit page modules.
 Reference baseline: `resource://skills/nicegui/document`
 ### Core architecture decisions
 - **App factory:** `create_app()`
 - **Lifespan-managed resources:** worker start/stop managed in startup/shutdown
 - **Modular pages:** upload and jobs pages in separate modules (no monolithic UI file)
 - **Health endpoint:** FastAPI-side `/healthz`
 - **UI composition:** route pages stay modular and reusable shared shell/components live under `ui/components` as needed
 - **Styling architecture:** shared CSS loaded once at startup; avoid ad-hoc per-page styling drift
 - **Dependency direction (one-way):**
  - `app` -> `config/logging/db/worker/ui/api`
  - `ui/pages` -> `ui/components` + `services`
  - `services` -> `db/models/providers`
  - no reverse imports from services into UI/API
 ### DB and AI stance (explicit)
 - **DB:** already enabled (SQLModel + SQLite), session lifecycle remains request/service-scoped as built in prior steps.
 - **AI workflow:** already in place via Step 3 transcription service + Step 4 worker; UI does not call provider SDK directly.
 - **Mounted docs:** not in Step 5 scope; docs mounting remains disabled for MVP.
 ### Async and responsiveness stance
 - Prefer `async def` for page handlers and service boundaries when I/O is involved.
 - Keep UI handlers non-blocking (no blocking sleeps or synchronous long I/O calls).
 - For long-running user actions, always provide explicit loading/progress/error states.
 - Keep cancellation/timeout behavior explicit for refresh/poll operations where applicable.
 ---
 ## Scope
 ### In scope
 - `src/transcription/app.py`
 - `src/transcription/ui/upload_page.py`
 - `src/transcription/ui/jobs_page.py`
 - `src/transcription/ui/__init__.py`
 - `src/transcription/api/health.py` (or equivalent FastAPI health route module)
 - UI/app tests with MCP scaffold->fill flow
 ### Out of scope
 - Auth
 - advanced filtering/search UX
 - batch upload UX beyond MVP
 - deployment/container hardening
 ---
 ## Planned Deliverables
 ### Source files
 - `src/transcription/app.py` (app factory + lifespan wiring)
 - `src/transcription/api/health.py` (GET `/healthz`)
 - `src/transcription/ui/upload_page.py` (upload flow)
 - `src/transcription/ui/jobs_page.py` (status list + detail)
 - `src/transcription/ui/__init__.py` (explicit `register_pages(...)` export)
 - `src/transcription/ui/components/*` (shared shell/navigation/status components if introduced)
 - `src/transcription/ui/static/*.css` (optional shared CSS loaded once at startup)
 ### Test files
 - `tests/test_app.py`
 - `tests/api/test_health.py`
 - `tests/ui/test_pages_registration.py`
 - `tests/ui/test_upload_page.py`
 - `tests/ui/test_jobs_page.py`
 ---
 ## Implementation Plan + Checklist
 Plan baseline and guardrails source: `resource://skills/nicegui/document`
 ## Phase A — App factory and lifespan orchestration
 - [ ] Create `create_app()` in `src/transcription/app.py`
 - [ ] Add FastAPI lifespan startup/shutdown handlers
 - [ ] Startup responsibilities:
  - [ ] `setup_logging()`
  - [ ] `create_all()`
  - [ ] ensure directories exist (`upload_dir`, `prompt_dir`)
  - [ ] create worker stop event
  - [ ] start worker background thread/task
 - [ ] Shutdown responsibilities:
  - [ ] signal stop event
  - [ ] join/cleanup worker thread/task cleanly
 - [ ] Register API router(s), including health route
 - [ ] Register NiceGUI pages via explicit page registration function
 - [ ] Load shared CSS once at startup (if present)
 ## Phase B — FastAPI health endpoint
 - [ ] Create `src/transcription/api/health.py`
 - [ ] Add `GET /healthz` returning simple healthy payload
 - [ ] Wire route into app factory
 ## Phase C — Upload page (`ui/upload_page.py`)
 - [ ] Add upload route/page registration function
 - [ ] Render file input accepting supported extensions
 - [ ] On submit:
  - [ ] show loading/progress state
  - [ ] call `create_upload_job(filename, file_bytes, ...)`
  - [ ] show success state with job reference/link
 - [ ] On error:
  - [ ] show user-safe error message
  - [ ] restore ready UI state
 - [ ] Ensure non-blocking I/O in UI event handlers; offload CPU-heavy work to worker path
 - [ ] Make timeout/cancellation behavior explicit for any long-running action
 ## Phase D — Jobs page (`ui/jobs_page.py`)
 - [ ] Add jobs list route/page registration function
 - [ ] Display jobs with status + timestamps
 - [ ] Add job detail route/view
 - [ ] Show transcript on success, error detail on failure
 - [ ] Include explicit refresh action and loading state
 - [ ] Ensure error states are surfaced to user and logged
 - [ ] Keep refresh path async and bounded to avoid UI freeze
 ## Phase E — UI registration module
 - [ ] Update `src/transcription/ui/__init__.py`
 - [ ] Export `register_pages(...)`
 - [ ] Ensure each page module exports `register_page(...)`
 - [ ] Keep page registration explicit and modular
 ## Phase F — Shared components and style consistency
 - [ ] Add `ui/components` module only for reusable shell elements (header/nav/status chips), not page-local logic
 - [ ] Keep structural layout in Python; keep visual polish in shared CSS
 - [ ] Avoid one-off styling duplication across upload/jobs pages
 ---
 ## MCP Testing Workflow (Required)
 Use these resources directly:
 - `resource://catalog/prompts/pytest-scaffold`
 - `resource://prompts/pytest-scaffold/document`
 - `resource://catalog/prompts/pytest-fill-scaffold`
 - `resource://prompts/pytest-fill-scaffold/document`
 ## E1 — Scaffold tests first (structure only)
 Target modules:
 - `src/transcription/app.py`
 - `src/transcription/api/health.py`
 - `src/transcription/ui/upload_page.py`
 - `src/transcription/ui/jobs_page.py`
 Scaffold test files:
 - `tests/test_app.py`
 - `tests/api/test_health.py`
 - `tests/ui/test_pages_registration.py`
 - `tests/ui/test_upload_page.py`
 - `tests/ui/test_jobs_page.py`
 Scaffold constraints:
 - [ ] class/method skeletons only
 - [ ] one-line docstrings
 - [ ] concise behavior-focused names
 - [ ] no implementation assertions yet
 Validation:
 - [ ] `uv run pytest --collect-only -q`
 ## E2 — Fill scaffold tests
 Fill constraints from MCP guidance:
 - [ ] preserve scaffold class/method names and docstrings (locked baseline)
 - [ ] one behavior target per method
 - [ ] deterministic tests preferred
 - [ ] minimal mocking; only nondeterministic boundaries
 Stack:
 - [ ] `fastapi` (or `mixed` if needed for UI+DB fixture combination)
 Suggested coverage:
 ### `tests/api/test_health.py`
 - [ ] `/healthz` returns success status and expected payload shape
 ### `tests/ui/test_pages_registration.py`
 - [ ] page registration wiring succeeds
 - [ ] expected routes are present
 ### `tests/test_app.py`
 - [ ] startup path initializes runtime dependencies
 - [ ] worker start is invoked on startup
 - [ ] worker shutdown signal/cleanup is invoked on shutdown
 ### `tests/ui/test_upload_page.py`
 - [ ] upload action calls upload service
 - [ ] success feedback displayed
 - [ ] error feedback displayed for `UploadError`
 - [ ] loading/progress state behavior covered
 - [ ] timeout/cancellation behavior covered (if implemented)
 ### `tests/ui/test_jobs_page.py`
 - [ ] list renders job statuses
 - [ ] detail shows transcript text for successful job
 - [ ] detail shows error detail for failed job
 - [ ] refresh/loading state behavior covered
 Marker strategy:
 - [ ] `unit` for pure helpers/state formatting
 - [ ] `integration` for app/page/service+DB contracts
 - [ ] `external` not required for default Step 5 lane
 Async behavior assertions:
 - [ ] long-running actions keep button/inputs in expected disabled state
 - [ ] completion/failure returns controls to ready state
 ---
 ## Validation Sequence (strict)
 - [ ] `uv run pytest --collect-only -q`
 - [ ] `uv run pytest -m unit -q` *(if unit tests touched)*
 - [ ] `uv run pytest tests/api/test_health.py -q`
 - [ ] `uv run pytest tests/ui/test_pages_registration.py -q`
 - [ ] `uv run pytest tests/test_app.py -q`
 - [ ] `uv run pytest tests/ui/test_upload_page.py -q`
 - [ ] `uv run pytest tests/ui/test_jobs_page.py -q`
 - [ ] `uv run pytest -q`
 ---
 ## Guardrails (NiceGUI + MVP)
 - [ ] Do not collapse pages into one file.
 - [ ] Do not use implicit global side effects for runtime wiring.
 - [ ] Keep UI responsive with explicit loading/progress/error states.
 - [ ] Do not block UI handlers with synchronous long I/O.
 - [ ] Do not place provider SDK calls in UI handlers.
 - [ ] Keep dependency direction one-way and maintainable.
 - [ ] Keep shared UI in `ui/components`; keep service logic out of page modules.
 ---
 ## Definition of Done
 - [ ] App factory + lifespan are in place
 - [ ] Health endpoint exists and is tested
 - [ ] Upload page creates queued jobs through service boundary
 - [ ] Jobs list/detail pages render status/transcript/failure data
 - [ ] Worker lifecycle is started/stopped by app lifespan
 - [ ] Async UI states (loading/success/error) are deterministic and tested
 - [ ] Scaffold->fill testing flow completed and validated
 - [ ] Full suite passes: `uv run pytest -q`
 ## Completion Checks (NiceGUI skill aligned)
 - [ ] Uses app factory and FastAPI lifespan
 - [ ] Pages are modularized (not single-file UI)
 - [ ] Health endpoint exists on FastAPI side
 - [ ] Dependency direction is clean and one-way
 - [ ] Async-first guidance is applied where I/O exists, with explicit non-blocking UX states
 - [ ] DB/AI/docs decisions are explicit and reflected in structure
 - [ ] Plan references baseline URI: `resource://skills/nicegui/document`
 ---
 ## PR Checklist (Integrated)
 ### Implementation
 - [ ] `app.py` app factory + lifespan implemented
 - [ ] FastAPI health route (`/healthz`) implemented
 - [ ] `ui/upload_page.py` implemented
 - [ ] `ui/jobs_page.py` implemented
 - [ ] `ui/__init__.py` explicit page registration implemented
 - [ ] Worker startup/shutdown managed by lifespan
 ### Testing (MCP-compliant)
 - [ ] Scaffold phase completed first for all Step 5 tests
 - [ ] `--collect-only` passed on scaffolds
 - [ ] Fill phase completed without renaming/re-nesting scaffolded tests
 - [ ] Marker decisions documented (`unit` vs `integration`)
 - [ ] Targeted tests passed
 - [ ] Full suite passed
 ### Evidence
 - [ ] Validation command outputs captured
 - [ ] Files created/updated listed
 - [ ] MCP prompt resources referenced in implementation notes
 - [ ] Any residual risks/questions documented
 ---
@@ -1,50 +0,0 @@
 Step 6 implemented ✅
 ### Files added
 - `tests/integration/test_pipeline_flow.py`
 - `tests/test_traceability.py`
 ### What was implemented
 #### 1) End-to-end pipeline integration coverage
 `tests/integration/test_pipeline_flow.py` now verifies:
 - **Success path**
  - upload creates queued job
  - worker processes job
  - job becomes `transcribed`
  - transcript text is persisted
  - error detail remains `None`
 - **Failure path**
  - upload creates queued job
  - worker processing failure is handled
  - job becomes `failed`
  - transcript has `text=None`
  - `error_detail` contains provider failure reason
 #### 2) MVP requirement traceability check
 `tests/test_traceability.py` adds a concrete map:
 - REQ-0, REQ-1, REQ-2, REQ-3, REQ-4, REQ-5, REQ-6, REQ-8, REQ-12
 - each REQ maps to one or more existing test files
 - test asserts each mapped path exists (guards against drift)
 ---
 ### Validation run results
 Executed and passed:
 1. `uv run pytest --collect-only -q` ✅  
 2. `uv run pytest -m unit -q` ✅  
 3. `uv run pytest -m integration -q` ✅  
 4. `uv run pytest -m "not external" -q` ✅  
 5. `uv run pytest tests/integration/test_pipeline_flow.py -q` ✅  
 6. `uv run pytest tests/ui/test_upload_page.py -q` ✅  
 7. `uv run pytest tests/ui/test_jobs_page.py -q` ✅  
 8. `uv run pytest -q` ✅  
 All green.  
 (Only existing non-blocking FastAPI TestClient deprecation warning remains.)
@@ -1,229 +0,0 @@
 ## Step 6: Test and Verification Hardening (MVP closeout)
 ## Objective
 Complete MVP verification by building a **requirements-traceable, deterministic test strategy** across unit/integration/external lanes, then enforcing stable validation commands and reporting.
 This step finalizes the MVP implementation sequence from `docs/mvp.md` (Step 6 in the build order: tests and automated verification).
 ---
 ## MCP Resource Integration (what was applied)
 I reviewed all top-level skills/prompts from `john-stream-mcp` and integrated the relevant guidance into this plan:
 ### Directly applied
 - `resource://skills/pytesting/document`
 - `resource://catalog/prompts/pytest-scaffold`
 - `resource://prompts/pytest-scaffold/document`
 - `resource://catalog/prompts/pytest-fill-scaffold`
 - `resource://prompts/pytest-fill-scaffold/document`
 - `resource://skills/nicegui/document`
 - `resource://skills/nicegui-ui-customization/document`
 - `resource://skills/fastapi-uv-docker/document`
 - `resource://skills/python-logging-dictconfig/document`
 - `resource://skills/python-typing/document`
 - `resource://skills/ruff-linting-formating/document`
 ### Reviewed but informational/non-blocking for Step 6
 - `copilot-customization`, `mcp-details`, `vscode-configuration`, `zensical-docs`, and authoring/shim prompts.
  - These are primarily customization/documentation tooling resources, not core MVP test-lane blockers.
  - Step 6 includes optional workflow follow-ups where relevant (e.g., VS Code task conveniences).
 ---
 ## Scope
 ### In scope
 - Strengthen and complete test coverage for the shipped MVP slice (Steps 1–5)
 - Add requirement-to-test traceability for REQ-0..REQ-12 (MVP subset emphasized)
 - Enforce deterministic default lanes (`unit`, `integration`)
 - Keep `external` lane opt-in and isolated
 - Validate app/UI/service/worker contracts end-to-end at test level
 ### Out of scope
 - Major architecture rewrites (async SQLAlchemy migration, queue system, etc.)
 - Full production deployment rollout
 - Post-MVP feature expansion (revision history, search, export)
 ---
 ## Planned Deliverables
 ### Test files (new/updated)
 - `tests/test_traceability.py` *(or docs-based traceability matrix if preferred)*
 - `tests/integration/test_pipeline_flow.py` *(upload -> queued -> worker -> transcript/failed)*
 - `tests/ui/test_upload_page.py` (augment loading/error/ready-state checks as practical)
 - `tests/ui/test_jobs_page.py` (augment refresh/error behavior checks as practical)
 - Existing tests touched only when needed; preserve naming/hierarchy unless explicitly approved.
 ### Optional docs output
 - `docs/tests.md` or `docs/verification.md` with lane definitions and command matrix
 - REQ-to-test mapping table
 ---
 ## Design and Policy Decisions (MCP-aligned)
 1. **Scaffold-first, fill-second workflow is mandatory**
   - First create/adjust skeletons and collect.
   - Then fill test bodies.
   - Preserve scaffold names/docstrings during fill.
 2. **Deterministic-first default lanes**
   - `unit` and `integration` run by default.
   - `external` remains explicit opt-in.
 3. **One behavior target per test**
   - Short, behavior-focused names.
   - Precise assertions on observable outcomes.
 4. **Test double discipline (from pytesting skill)**
   - Prefer real-input/real-object paths first.
   - If monkeypatch/mocks/fakes are needed for a boundary, keep narrowly scoped.
   - Avoid call-only assertions.
 5. **NiceGUI responsiveness expectations**
   - Verify loading/success/error state transitions where testable.
   - Ensure user-facing feedback behavior is covered.
 6. **FastAPI/ops baseline checks**
   - Keep `/healthz` route validation in default lanes.
   - Keep startup/shutdown lifecycle assertions present.
 ---
 ## Implementation Plan + Checklist
 ## Phase A — Coverage and traceability audit
 - [ ] Build a REQ-to-test matrix for MVP requirements:
  - [ ] REQ-0, REQ-1, REQ-2, REQ-3, REQ-4, REQ-5, REQ-6, REQ-8, REQ-12
 - [ ] Identify weak spots:
  - [ ] full pipeline integration (service + worker + persistence)
  - [ ] UI state transition assertions (loading/error/ready)
  - [ ] failure-path persistence verification robustness
 - [ ] Record current baseline command results before edits
 ## Phase B — Scaffold phase (pytest-scaffold resources)
 Target modules/areas:
 - pipeline integration flow
 - UI behavior augmentations
 - traceability checks/document validators (if test-backed)
 - [ ] Scaffold new/adjusted test files/classes/methods only
 - [ ] Keep one-line intent docstrings
 - [ ] Keep behavior-focused names
 - [ ] Run: `uv run pytest --collect-only -q`
 ## Phase C — Fill phase (pytest-fill-scaffold resources)
 - [ ] Fill scaffolded methods with deterministic setup/assertions
 - [ ] Preserve scaffold names/hierarchy/docstrings
 - [ ] Add/adjust fixtures at nearest useful scope
 - [ ] Keep DB tests in `integration`; pure helper tests in `unit`
 ### Required coverage additions
 #### Pipeline integration
 - [ ] Upload service creates document/job and file path persists
 - [ ] Worker success path creates transcript and terminal status
 - [ ] Worker failure path persists error detail and terminal failed status
 - [ ] Queue-empty behavior remains stable (`False` return / no side effects)
 #### UI behavior (practical, testable boundaries)
 - [ ] Upload helper flow success and UploadError surfacing
 - [ ] Jobs data helpers return stable normalized view models
 - [ ] Refresh/detail fallback behavior for missing/invalid job IDs
 #### Traceability
 - [ ] Every in-scope MVP REQ has at least one mapped test/assertion point
 - [ ] Document and/or enforce mapping consistency
 ## Phase D — External lane stability
 - [ ] Keep real-image external tests isolated under `@pytest.mark.external`
 - [ ] Ensure no external test leaks into default runs
 - [ ] Confirm artifact capture behavior remains stable
 ## Phase E — Quality gates and workflow
 - [ ] Confirm logging/lifecycle startup tests still pass after changes
 - [ ] (If enabled) add/update lint/type check commands in docs:
  - [ ] Ruff lane (if configured)
  - [ ] typing lane (if configured)
 - [ ] Optionally add VS Code task aliases for test lanes (non-blocking)
 ---
 ## Marker and Fixture Strategy
 - `unit`: pure logic, helper behavior, formatting/normalization
 - `integration`: DB + service + app lifecycle contracts
 - `external`: live provider/real image checks only
 Fixture policy:
 - Prefer reusable fixtures in `tests/conftest.py` only when broadly shared
 - Use subtree/local fixtures for domain-specific setup
 - Keep setup explicit and readable
 ---
 ## Validation Sequence (strict)
 - [ ] `uv run pytest --collect-only -q`
 - [ ] `uv run pytest -m unit -q`
 - [ ] `uv run pytest -m integration -q`
 - [ ] `uv run pytest -m "not external" -q`
 - [ ] `uv run pytest tests/integration/test_pipeline_flow.py -q` *(if added)*
 - [ ] `uv run pytest tests/ui/test_upload_page.py -q`
 - [ ] `uv run pytest tests/ui/test_jobs_page.py -q`
 - [ ] `uv run pytest -q`
 Optional external verification:
 - [ ] `uv run pytest -m external -q`
 ---
 ## Guardrails
 - Do not rename/re-nest scaffolded tests during fill unless explicitly requested.
 - Do not broaden external dependencies in default lane.
 - Do not add flaky timing-based assertions; keep deterministic boundaries.
 - Keep business logic out of UI tests; test through service/helper boundaries.
 - Preserve one-way dependency direction in test setup patterns.
 ---
 ## Definition of Done (Step 6)
 - [ ] MVP requirement coverage is explicitly traceable
 - [ ] Deterministic lanes (`unit` + `integration`) are stable and green
 - [ ] External lane remains opt-in and green when enabled
 - [ ] Pipeline success/failure lifecycle paths are verified end-to-end
 - [ ] UI helper/state behavior has explicit success/error assertions
 - [ ] Full suite passes with `uv run pytest -q`
 - [ ] Verification evidence is captured in implementation report
 ---
 ## PR Checklist (Step 6)
 ### Implementation
 - [ ] Added/updated test files per scoped gaps
 - [ ] Added REQ traceability mapping
 - [ ] Kept default lanes deterministic
 - [ ] Preserved scaffold invariants during fill
 ### Testing (MCP-compliant)
 - [ ] Used scaffold prompt flow first
 - [ ] Used fill prompt flow second
 - [ ] Preserved naming/docstrings/hierarchy
 - [ ] Marker usage documented (`unit`, `integration`, `external`)
 ### Evidence
 - [ ] Collected command outputs in strict order
 - [ ] Listed files changed
 - [ ] Listed MCP resources used and why
 - [ ] Noted residual risks/open questions (if any)
@@ -1,134 +0,0 @@
 ## Step 7 Results: Error Handling Standardization and Operational Visibility
 ## Summary
 Step 7 was implemented across the MVP runtime boundaries with a shared error taxonomy, actionable UI error surfacing, worker failure normalization, and API error envelope handling.
 All required validation gates in `docs/step7.md` were executed and passed.
 ---
 ## Scope Delivered
 ### Implemented
 - Shared application error contract and taxonomy
 - Service-layer error normalization (upload + transcription)
 - UI error presentation helpers with suggested actions and error references
 - Worker failure persistence format with category/suggestion/error_id markers
 - API exception handlers for structured error responses
 - Targeted tests for new error contract behavior
 ### Not implemented in this step
 - External lane execution (`-m external`) was not required for Step 7 completion and was not run in this pass.
 ---
 ## Files Added
 - `src/transcription/errors.py`
 - `src/transcription/api/errors.py`
 - `src/transcription/ui/error_presenter.py`
 - `tests/test_errors.py`
 - `tests/api/test_error_responses.py`
 - `docs/step7.md`
 ## Files Updated
 - `src/transcription/app.py`
 - `src/transcription/services/upload.py`
 - `src/transcription/services/transcription.py`
 - `src/transcription/ui/upload_page.py`
 - `src/transcription/ui/jobs_page.py`
 - `src/transcription/worker.py`
 - `tests/services/test_upload.py`
 - `tests/services/test_transcription.py`
 - `tests/services/test_worker.py`
 - `tests/integration/test_pipeline_flow.py`
 - `uv.lock`
 ---
 ## Implementation Notes by Phase
 ### Phase A/B (Foundation)
 - Added `ErrorCategory` enum and `AppError` base type in `src/transcription/errors.py`.
 - Added helper utilities:
  - `new_error_id()`
  - `build_error_envelope(...)`
  - `classify_unexpected_error(...)`
  - `format_error_detail(...)`
 ### Phase C (Service/Provider normalization)
 - `UploadError` now extends `AppError` and includes category/suggestion/retriable metadata.
 - `PromptLoadError` and `TranscriptionError` now extend `AppError`.
 - Provider failures are mapped with deterministic category semantics (auth/payload/provider-failure cases).
 ### Phase D (UI visibility)
 - Added `src/transcription/ui/error_presenter.py`.
 - Upload and jobs pages now use centralized UI error rendering and summary helpers.
 - UI error paths now include more visible/actionable guidance and reference IDs.
 ### Phase E (Worker failure handling)
 - Worker now normalizes exception handling into structured persisted `error_detail` strings with:
  - category marker
  - suggestion marker
  - error_id marker
 - Logging now includes category/error_id context in failure paths.
 ### Phase F (API envelope)
 - Added `src/transcription/api/errors.py` and registered handlers in app factory.
 - AppError and unexpected exceptions now serialize to stable API envelopes with mapped status codes.
 ---
 ## Validation Commands and Outcomes
 All commands were executed with `uv run python -m pytest ...` and completed successfully.
 1. `uv run python -m pytest tests/test_errors.py -q` ✅
 2. `uv run python -m pytest tests/services/test_upload.py -q` ✅
 3. `uv run python -m pytest tests/services/test_transcription.py -q` ✅
 4. `uv run python -m pytest tests/providers/test_openrouter.py -q` ✅
 5. `uv run python -m pytest tests/services/test_worker.py -q` ✅
 6. `uv run python -m pytest tests/integration/test_pipeline_flow.py -q` ✅
 7. `uv run python -m pytest tests/api/test_error_responses.py -q` ✅
 8. `uv run python -m pytest tests/ui/test_upload_page.py -q` ✅
 9. `uv run python -m pytest tests/ui/test_jobs_page.py -q` ✅
 10. `uv run python -m pytest -m "not external" -q` ✅
 11. `uv run python -m pytest --collect-only -q` ✅
 12. `uv run python -m pytest -m unit -q` ✅
 13. `uv run python -m pytest -m integration -q` ✅
 14. `uv run python -m pytest tests/integration/test_pipeline_flow.py -q` ✅
 15. `uv run python -m pytest tests/ui/test_upload_page.py -q` ✅
 16. `uv run python -m pytest tests/ui/test_jobs_page.py -q` ✅
 17. `uv run python -m pytest -q` ✅
 Observed warning (non-blocking): Starlette/FastAPI TestClient deprecation warning related to `httpx` package naming.
 ---
 ## Policy Alignment Check (`docs/error_handling.md`)
 Aligned items:
 - Stable taxonomy categories are implemented.
 - Unexpected errors are normalized.
 - User-facing UI paths include actionable guidance and references.
 - Worker persistence includes trace-friendly failure detail.
 - API error responses are structured and category-aware.
 Follow-up candidates:
 - Add richer UI tests that validate rendered suggested-action content end-to-end (current tests focus helper/service contracts).
 - Consider typed storage fields for error metadata instead of packed `error_detail` strings in a future schema revision.
 ---
 ## Step 7 Definition of Done Status
 - [x] Shared error taxonomy implemented across MVP layers
 - [x] GUI error paths upgraded for visibility/actionability
 - [x] Worker failure persistence and log context standardized
 - [x] API error envelope handling added and tested
 - [x] Phase-level and full-suite validation gates passed
 - [x] Results documented in this report
 Step 7 is complete.
@@ -1,267 +0,0 @@
 ## Step 7: Error Handling Standardization and Operational Visibility
 ## Objective
 Apply the canonical error policy from `docs/error_handling.md` to the MVP implementation so failures are:
 - consistently classified
 - visibly surfaced in the GUI
 - paired with suggested corrective actions
 - traceable through logs via error reference IDs
 - validated through deterministic tests after each phase
 This step extends MVP hardening by converting current ad hoc exception behavior into a stable cross-layer contract.
 ---
 ## Scope
 ### In scope
 - Introduce a shared application error contract and taxonomy implementation
 - Normalize service/provider exceptions into taxonomy categories
 - Improve GUI error visibility and suggested-action UX
 - Standardize worker failure persistence and logging context
 - Add API error-envelope policy hooks for current/future endpoints
 - Add targeted tests and phase-level/full-suite validation gates
 ### Out of scope
 - Major architecture rewrites (distributed queue, multi-service decomposition)
 - Post-MVP feature expansion unrelated to error handling
 - Full observability platform rollout (tracing backends, APM)
 ---
 ## Policy Source of Truth
 - Canonical policy document: `docs/error_handling.md`
 - If implementation and policy diverge, policy is authoritative and code/tests must be updated.
 ---
 ## Planned Deliverables
 ### Runtime code
 - `src/transcription/errors.py` *(new shared contract module)*
 - `src/transcription/ui/error_presenter.py` *(new UI error rendering helper)*
 - Updates to:
  - `src/transcription/services/upload.py`
  - `src/transcription/services/transcription.py`
  - `src/transcription/providers/openrouter.py`
  - `src/transcription/worker.py`
  - `src/transcription/ui/upload_page.py`
  - `src/transcription/ui/jobs_page.py`
  - `src/transcription/api/*` *(as needed for envelope/handlers)*
 ### Tests
 - `tests/test_errors.py` *(new shared error contract tests)*
 - updates/additions in:
  - `tests/services/test_upload.py`
  - `tests/services/test_transcription.py` *(add if missing)*
  - `tests/providers/test_openrouter.py`
  - `tests/services/test_worker.py`
  - `tests/ui/test_upload_page.py`
  - `tests/ui/test_jobs_page.py`
  - `tests/api/test_error_responses.py` *(new, if API handlers added)*
 ### Documentation
 - Update `docs/error_handling.md` only if implementation reveals policy gaps
 - Capture validation evidence in a Step 7 results artifact (`docs/step7-results.md`)
 ---
 ## Design and Policy Decisions
 1. **Stable taxonomy contract**
   - Use policy categories as stable identifiers (`validation_error`, `user_input_error`, etc.).
 2. **Actionable UX is mandatory**
   - User-visible errors must include a suggested course of action.
 3. **Traceability by default**
   - Non-trivial errors include an `error_id` in both logs and user-facing output.
 4. **Safe surface / rich logs**
   - UI/API show safe summaries; logs retain diagnostic detail and traceback.
 5. **Deterministic verification cadence**
   - Targeted tests after each change batch, then phase-level regression gates.
 ---
 ## Implementation Plan + Checklist
 ## Phase A — Baseline Validation and Gap Confirmation
 - [ ] Run baseline tests before changes
 - [ ] Record baseline outputs and any known flaky behavior
 - [ ] Confirm current behavior against `docs/error_handling.md` requirements
 ### Validation gate
 - [ ] `uv run pytest -m "not external" -q`
 - [ ] `uv run pytest -q`
 ## Phase B — Shared Error Contract Foundation
 - [ ] Add `src/transcription/errors.py` with:
  - [ ] stable category enum
  - [ ] base `AppError` (category/message/suggestion/error_id/retriable)
  - [ ] helpers for error-id generation and fallback classification
 - [ ] Keep category names aligned with `docs/error_handling.md`
 ### Tests
 - [ ] Add `tests/test_errors.py`
  - [ ] category stability assertions
  - [ ] error_id creation behavior
  - [ ] fallback classification for unexpected exceptions
 ### Validation gate
 - [ ] `uv run pytest tests/test_errors.py -q`
 - [ ] `uv run pytest -m "not external" -q`
 ## Phase C — Service and Provider Normalization
 - [ ] Refactor upload service exceptions to shared taxonomy
 - [ ] Refactor transcription service exceptions to shared taxonomy
 - [ ] Normalize provider adapter failures into deterministic categories
 - [ ] Preserve causal chaining (`raise ... from exc`)
 ### Tests
 - [ ] Extend `tests/services/test_upload.py`:
  - [ ] empty payload category/suggestion
  - [ ] unsupported extension category/suggestion
  - [ ] persistence failure category mapping
 - [ ] Add/extend `tests/services/test_transcription.py`:
  - [ ] missing/empty prompt behavior
  - [ ] unsupported file type behavior
  - [ ] provider failure mapping behavior
 - [ ] Extend `tests/providers/test_openrouter.py`:
  - [ ] auth error mapping
  - [ ] malformed response mapping
 ### Validation gate
 - [ ] `uv run pytest tests/services/test_upload.py -q`
 - [ ] `uv run pytest tests/services/test_transcription.py -q`
 - [ ] `uv run pytest tests/providers/test_openrouter.py -q`
 - [ ] `uv run pytest -m "not external" -q`
 ## Phase D — GUI Visibility and Suggested Actions
 - [ ] Add `src/transcription/ui/error_presenter.py`
 - [ ] Update upload/jobs pages to use centralized error presentation
 - [ ] Ensure GUI surfaces:
  - [ ] user-safe message
  - [ ] suggested action
  - [ ] error reference ID
  - [ ] optional technical details panel
 - [ ] Replace raw `str(exc)` UX where policy requires safer messaging
 ### Tests
 - [ ] Extend `tests/ui/test_upload_page.py` for actionable error UX paths
 - [ ] Extend `tests/ui/test_jobs_page.py` for refresh/detail error guidance
 - [ ] Add `tests/ui/test_error_presenter.py` *(optional but recommended)*
 ### Validation gate
 - [ ] `uv run pytest tests/ui/test_upload_page.py -q`
 - [ ] `uv run pytest tests/ui/test_jobs_page.py -q`
 - [ ] `uv run pytest -m "not external" -q`
 ## Phase E — Worker Failure Persistence and Logging Context
 - [ ] Update worker failure handling to classify errors before persistence
 - [ ] Ensure failed jobs persist actionable, structured error detail
 - [ ] Add log context fields where available (`error_id`, `category`, `operation`, `job_id`)
 - [ ] Ensure retry semantics are explicit and bounded (or clearly documented as deferred)
 ### Tests
 - [ ] Extend `tests/services/test_worker.py`:
  - [ ] missing document failure contract
  - [ ] provider/transcription failure contract
  - [ ] persisted error detail includes category/suggestion/error_id markers
 - [ ] Validate integration failure flow in `tests/integration/test_pipeline_flow.py`
 ### Validation gate
 - [ ] `uv run pytest tests/services/test_worker.py -q`
 - [ ] `uv run pytest tests/integration/test_pipeline_flow.py -q`
 - [ ] `uv run pytest -m "not external" -q`
 ## Phase F — API Error Envelope Alignment (Current + Future Routes)
 - [ ] Add shared API error serialization utilities/handlers (as needed)
 - [ ] Ensure API responses can include:
  - [ ] `error_id`
  - [ ] `category`
  - [ ] `message`
  - [ ] `suggestion`
  - [ ] `timestamp`
 - [ ] Map categories to HTTP status guidance from `docs/error_handling.md`
 ### Tests
 - [ ] Add `tests/api/test_error_responses.py` *(if handlers added)*
 - [ ] Keep `tests/api/test_health.py` passing
 ### Validation gate
 - [ ] `uv run pytest tests/api/test_error_responses.py -q` *(if added)*
 - [ ] `uv run pytest tests/api/test_health.py -q`
 - [ ] `uv run pytest -m "not external" -q`
 ## Phase G — Final Regression and Documentation Closure
 - [ ] Reconcile implementation details with `docs/error_handling.md`
 - [ ] Update policy doc only where required by confirmed implementation learning
 - [ ] Capture execution evidence in `docs/step7-results.md`
 ### Final validation sequence (strict)
 - [ ] `uv run pytest --collect-only -q`
 - [ ] `uv run pytest -m unit -q`
 - [ ] `uv run pytest -m integration -q`
 - [ ] `uv run pytest -m "not external" -q`
 - [ ] `uv run pytest tests/integration/test_pipeline_flow.py -q`
 - [ ] `uv run pytest tests/ui/test_upload_page.py -q`
 - [ ] `uv run pytest tests/ui/test_jobs_page.py -q`
 - [ ] `uv run pytest -q`
 Optional:
 - [ ] `uv run pytest -m external -q`
 ---
 ## Guardrails
 - Do not weaken user-facing clarity to expose raw internals.
 - Do not introduce silent exception swallowing.
 - Do not break category-name stability without policy update.
 - Do not merge phase changes without passing that phase validation gate.
 - Keep targeted tests fast and deterministic; isolate external-provider tests under `external`.
 ---
 ## Definition of Done (Step 7)
 - [ ] Shared error taxonomy is implemented and used across MVP layers
 - [ ] GUI error experiences are visible, actionable, and traceable
 - [ ] Worker persists and logs failure context consistently
 - [ ] API error contract path is aligned for current/future endpoints
 - [ ] Phase-by-phase test gates pass
 - [ ] Full suite remains green (`uv run pytest -q`)
 - [ ] Step 7 results are documented with evidence
 ---
 ## PR Checklist (Step 7)
 ### Implementation
 - [ ] Added shared error contract module
 - [ ] Updated service/provider/worker/UI error handling paths
 - [ ] Added actionable GUI guidance for user-visible failures
 - [ ] Added error reference IDs for traceability
 ### Testing
 - [ ] Added/updated tests per phase scope
 - [ ] Ran targeted phase tests after each change batch
 - [ ] Ran `not external` regression at each phase boundary
 - [ ] Ran full suite before closeout
 ### Documentation and Evidence
 - [ ] `docs/error_handling.md` reviewed for alignment
 - [ ] `docs/step7-results.md` includes executed command outputs
 - [ ] Residual risks and deferred items explicitly recorded
@@ -1,209 +0,0 @@
 ## MVP Definition: Historical Document Transcription System
 ### 1. MVP Objective
 Deliver the thinnest possible end-to-end vertical slice — a user uploads an image of a document, the system transcribes it via the OpenRouter Python SDK, and the user reads the resulting transcript — with just enough persistence and structure to validate the core value proposition: *can AI-driven transcription, guided by curated prompts, produce useful verbatim transcripts of historical family documents?*  
 The MVP deliberately defers full-text search, export, revision history, MongoDB, and timeline assembly. These are additive features that don't need validation before the core transcription loop is proven.
 ---
 ### 2. Core User Story
 *As a family historian, I can upload a photo of a historical document, wait for it to be transcribed, and read the verbatim transcript — so I can evaluate whether this system will work for my thousands of documents.*
 ---
 ### 3. In-Scope Requirements (from ```requirements.md```)  
 | Requirement | ID | MVP Rationale |
 | --- |  --- |  --- |
 | End-to-end transcription with lifecycle state | REQ-0 | This is the MVP. |
 | Upload one or more images from the web UI | REQ-1 | Core entry point. MVP supports single-image upload (multi-image is a stretch goal). |
 | Asynchronous processing → transcription or failure | REQ-2 | Validates the AI transcription pipeline. |
 | Persist and expose job states (queued → processing → transcribed/failed) | 	REQ-3 | Minimum feedback loop for the user. |
 | Persist transcription output and failure details | REQ-4 | User must be able to read the result. |
 | UI views for status and transcript reading | REQ-5 | The user needs to see what happened. |
 | Background processing to keep UI responsive | REQ-6 | Essential for usability during long AI calls. |
 | Centralized config and logging at startup | REQ-8 | Small effort, high payoff for debugging. |
 | Store transcription prompts as Markdown files | REQ-12 | Core to the Prompt Curation Policy in intent.md. Start with a single prompt file. |
 ### Deferred to Post-MVP
 | Requirement | ID | Why Deferred |
 | ---  | --- | --- |
 | Lifespan-owned runtime resources (engine, session factory, etc.) | REQ-7 | Important for production robustness, but a simple global or module-level setup is adequate for MVP validation. |
 | Docker Compose (app + PostgreSQL + optional MongoDB) | REQ-9 | MVP runs locally with SQLite to eliminate container overhead during rapid iteration. PostgreSQL migration is Stage 1 hardening. |
 | Explicit, opt-in schema bootstrap | REQ-10 | MVP uses auto-create-tables at startup (SQLModel create_all). Production schema discipline comes after the model stabilizes. |
 | Service-backed persistence for core data | REQ-11 | MVP uses a thin repository layer over SQLite. Full service abstraction follows once the domain model is proven. |
 ---
 ### 4. MVP Feature Set
 #### Feature 1: Document Upload (UI)
 * A single NiceGUI page with a file-upload widget (accepts .jpg, .png, .tiff, .pdf).
 * On upload: save the file to a local uploads/ directory, create a Document record, create a Job record with status queued.
 * Minimal metadata capture: original filename, upload timestamp.
 #### Feature 2: Asynchronous Transcription Worker
 * An in-process background worker (Python asyncio task or BackgroundTasks) that:
    1. Picks up queued jobs.
    2. Transitions status to processing.
    3. Sends the image + the curated Markdown prompt to an AI vision model via OpenRouter.
    4. On success: saves the transcript text, transitions to transcribed.
    5. On failure: saves the error detail, transitions to failed.
 #### Feature 3: Transcription Prompt (Markdown Asset)
 * A single Markdown file (prompts/transcribe_document.md) encoding the verbatim transcription rules from intent.md (the Document Issues table, scholarly guidelines, etc.).
 * The worker reads this file at invocation time and injects it as the system/user prompt.
 #### Feature 4: Job Status & Transcript Viewer (UI)
 * A job list page showing all jobs with their current status (queued / processing / transcribed / failed).
 * A transcript detail page showing:
    * The original uploaded image (rendered inline).
    * The transcription text (or the failure reason).
    * Timestamp metadata.
 #### Feature 5: Minimal Persistence (SQLite + SQLModel)
 * Three tables/models:
    * Document: id, filename, file_path, uploaded_at.
    * Job: id, document_id (FK), status, created_at, updated_at.
    * Transcript: id, job_id (FK), text, error_detail, created_at.
 * SQLite database file stored locally. Auto-created on first startup.
 #### Feature 6: Centralized Configuration
 * A single config.py (or Pydantic BaseSettings) loading:
    * PROVIDER (fixed to openrouter for MVP)
    * OPENROUTER_API_KEY (required)
    * PROVIDER_MODEL (default: OpenRouter model slug for vision transcription)
    * OPENROUTER_HTTP_REFERER (optional; app attribution)
    * OPENROUTER_APP_TITLE (optional; app attribution)
    * DATABASE_URL (default: sqlite:///./transcription.db)
    * UPLOAD_DIR (default: ./uploads)
    * PROMPT_DIR (default: ./prompts)
 #### Feature 7: MVP Dependency Baseline (OpenRouter-Centric)
 * Runtime dependencies:
    * openrouter (official OpenRouter Python SDK)
    * pydantic
    * pydantic-settings
    * sqlmodel
 * Explicitly out of MVP runtime dependencies:
    * google-genai (deferred until/if Gemini is introduced post-MVP)
 ---
 ### 5. MVP Architecture (Simplified)
 ```Apply
 ┌─────────────────────────────────────────────┐
 │              NiceGUI Web UI                 │
 │  ┌──────────────┐   ┌───────────────────┐   │
 │  │ Upload Page  │   │ Jobs / Transcript │   │
 │  └──────┬───────┘   └───────┬───────────┘   │
 │         │                   │               │
 │         ▼                   ▼               │
 │     ┌───────────────────────────┐           │
 │     │   Application Service     │           │
 │     │  (upload, job lifecycle)  │           │
 │     └─────┬─────────────┬───────┘           │
 │           │             │                   │
 │     ┌─────▼─────┐ ┌─────▼───────────────┐   │
 │     │ SQLite DB │ │ Background Worker   │   │
 │     │ (SQLModel)│ │ → AI Vision Provider│   │
 │     └───────────┘ └─────────────────────┘   │
 │                          │                  │
 │                    ┌─────▼──────┐           │
 │                    │ prompts/   │           │
 │                    │ *.md files │           │
 │                    └────────────┘           │
 └─────────────────────────────────────────────┘
 ```
 ---
 #### 6. Proposed File Structure
 ```Apply
 project-root/
 ├── docs/                          # (existing)
 ├── prompts/
 │   └── transcribe_document.md     # curated transcription prompt
 ├── src/
 │   └── transcription/
 │       ├── __init__.py
 │       ├── app.py                 # FastAPI + NiceGUI app entrypoint
 │       ├── config.py              # Pydantic BaseSettings
 │       ├── models.py              # SQLModel: Document, Job, Transcript
 │       ├── db.py                  # engine, session, create_all
 │       ├── providers/
 │       │   ├── __init__.py
 │       │   ├── base.py            # provider interface (transcribe contract)
 │       │   ├── openrouter.py      # OpenRouter via official Python SDK
 │       ├── services/
 │       │   ├── __init__.py
 │       │   ├── upload.py          # save file + create records
 │       │   └── transcription.py   # call provider, update job
 │       ├── worker.py              # background job loop
 │       └── ui/
 │           ├── __init__.py
 │           ├── upload_page.py     # NiceGUI upload page
 │           └── jobs_page.py       # NiceGUI job list + detail
 ├── tests/
 │   ├── test_models.py
 │   ├── test_upload.py
 │   └── test_transcription.py
 ├── pyproject.toml
 └── README.md
 ```
 ---
 #### 7. MVP Validation Criteria
 The MVP is considered validated when:
 1. ✅ A user can upload an image of a document through the browser.
 2. ✅ The system asynchronously sends the image to the configured AI vision model with the curated prompt.
 3. ✅ The transcript (or failure reason) is persisted and visible in the UI.
 4. ✅ The transcription follows verbatim scholarly rules defined in intent.md (spot-checked by the user on real family documents).
 5. ✅ The transcription prompt is stored as a standalone Markdown file and can be edited without code changes.
 6. ✅ Job status transitions are visible: queued → processing → transcribed/failed.
 ---
 ### 8. Key Feedback Questions the MVP Should Answer
 These are the real unknowns this MVP exists to resolve:
 | #	| Question | How We Learn |
 | --- | --- | --- |
 | 1	| Is AI transcription quality good enough for this document corpus?	| User reviews 20–50 real transcriptions against originals. |
 | 2	| Does the verbatim prompt produce scholarly-quality output, or does it need major rework?	| Compare output to the Document Issues table rules in intent.md. |
 | 3	| What document types are hardest (old cursive, faded ink, pencil, postcards)?	| Track which uploads produce failed or low-quality results. |
 | 4	| Is single-image upload sufficient, or is batch upload needed early?	| User friction during real scanning sessions. |
 | 5	| What metadata is missing that the user wishes they could capture at upload time?	| User feedback after processing real batches. |
 ---
 #### 9. What Comes After MVP (Immediate Post-MVP)
 Once the core transcription loop is validated, the next priorities (aligned to Architecture Stage 1) are:
 1. **Multi-image upload** — process a batch from a scanning session.
 2. **PostgreSQL migration** — swap SQLite for containerized PostgreSQL (REQ-9, REQ-10).
 3. **Revision history** — allow the user to edit/correct transcripts with immutable version tracking.
 4. **Full-text search** — search across all accepted transcripts.
 5. **Repository/service layer formalization** — proper ports/adapters as the domain model stabilizes.
 6. **Docker Compose deployment** — containerize the app for reproducible operation.
 ---
 #### 10. Implementation Approach
 Recommended build order for the MVP (each step produces a testable increment):
 | Step | Deliverable | Validates |
 | --- | --- | --- |
 | 1 | config.py + models.py + db.py — data layer with SQLite | Schema and config foundation |
 | 2 | prompts/transcribe_document.md — curated prompt from intent.md | Prompt asset pattern |
 | 3 | services/transcription.py + providers/ — call AI vision provider with prompt + image | Core AI integration |
 | 4 | services/upload.py + worker.py — upload handling + background job loop | End-to-end pipeline (CLI-testable) |
 | 5 | ui/upload_page.py + ui/jobs_page.py — NiceGUI pages | User-facing interface |
 | 6 | tests/ — unit + integration tests	Automated verification |
 This MVP is deliberately narrow: **one prompt, one provider (OpenRouter), one user, one image at a time, SQLite, no containers**. Every omission is intentional — the goal is to get real family documents through the transcription pipeline as fast as possible and let the quality of the output guide every subsequent decision.
@@ -1,84 +0,0 @@
 ## Document Transcription System Requirements
 This page captures a SysML v1.6-style requirements baseline for the production system described in [index.md](index.md). The model is represented as concise tables and traceability lists that preserve SysML-style IDs and relationship semantics.
 ## Scope
 - System of interest: the single Python application service (NiceGUI + FastAPI) with PostgreSQL as the relational system of record and optional MongoDB for document-oriented persistence.
 - Operational context: local-first execution with Docker Compose and an intentionally lightweight production trajectory.
 - Primary concern: end-to-end transcription job lifecycle from upload through completion or failure.
 ## Requirements Model (Concise Text Form)
 ### Requirements
 | ID | Category | Requirement | Risk | Verify Method |
 | --- | --- | --- | --- | --- |
 | REQ-0 | System | Provide end-to-end document transcription with persistent, inspectable lifecycle state. | medium | demonstration |
 | REQ-1 | Functional | Allow users to upload one or more document images from the web UI. | low | test |
 | REQ-2 | Functional | Run each upload through asynchronous processing that returns a transcription or explicit failure. | high | test |
 | REQ-3 | Functional | Persist and expose job states: upload, queued, processing, transcribed, failed, completed. | high | inspection |
 | REQ-4 | Functional | Persist transcription output, processing history, and failure details. | medium | test |
 | REQ-5 | Interface | Expose API and UI views for status inspection and completed transcription reading. | medium | demonstration |
 | REQ-6 | Performance | Trigger background processing on upload to preserve UI responsiveness. | medium | analysis |
 | REQ-7 | Design Constraint | Keep lifespan-owned runtime resources: SQLAlchemy engine, async session factory, worker resources, provider clients. | medium | inspection |
 | REQ-8 | Design Constraint | Initialize configuration and logging once at startup through centralized mechanisms. | low | inspection |
 | REQ-9 | Design Constraint | Use Docker Compose baseline of app plus PostgreSQL; allow optional MongoDB container when enabled. | medium | demonstration |
 | REQ-10 | Design Constraint | Keep schema bootstrap explicit and opt-in; normal startup does not mutate production schema. | high | inspection |
 | REQ-11 | Design Constraint | Use service-backed persistence for core document and job data. | medium | inspection |
 | REQ-12 | Design Constraint | Store transcription prompts as individual Markdown artifacts for iterative refinement. | medium | inspection |
 ### Requirement Relationships
 - Contains: REQ-0 contains REQ-1 through REQ-12.
 - Derives: REQ-2 -> REQ-3, REQ-3 -> REQ-4.
 - Traces: REQ-5 -> REQ-3.
 - Refines: REQ-6 -> REQ-2.
 ### Architecture Elements
 | Element | Type | Doc Reference |
 | --- | --- | --- |
 | UI | NiceGUI pages | src/transcription/ui/pages |
 | API | FastAPI routes | src/transcription/api/routes.py |
 | GRAPH | Async processing workflow | src/transcription/services, src/transcription/ai |
 | DBREL | PostgreSQL + SQLModel relational persistence | src/transcription/db |
 | DBDOC | MongoDB document persistence | src/transcription/db, src/transcription/services |
 | OPS | Docker Compose runtime | docker-compose.yml |
 | PROMPTS | Transcription prompt artifact library (Markdown files) | .github/prompts, docs |
 | TESTS | Pytest verification suite | tests |
 ### Satisfaction Mapping
 - UI satisfies REQ-1, REQ-5.
 - API satisfies REQ-5.
 - GRAPH satisfies REQ-2, REQ-6.
 - DBREL satisfies REQ-3, REQ-10.
 - DBDOC satisfies REQ-4, REQ-11.
 - OPS satisfies REQ-9.
 - PROMPTS satisfies REQ-12.
 ### Verification Mapping
 - TESTS verifies REQ-1, REQ-2, REQ-3, REQ-4, REQ-5, REQ-10, REQ-11, REQ-12.
 ## Requirement Notes
 - Requirement IDs (`REQ-*`) are stable references for planning, implementation, and test traceability.
 - The model uses compact tables and traceability lists for renderer compatibility while preserving SysML-style requirement IDs and relationship semantics.
 - Requirement categories (functional, interface, performance, and design constraints) are preserved as explicit REQ entries and relationship labels to keep change impact visible.
 - PostgreSQL containerization and optional MongoDB containerization are both treated as extremely lightweight and simple operational choices in this architecture.
 ## Verification Intent
 - Demonstration: validate end-to-end behavior via running system flows and operator-visible outcomes.
 - Inspection: verify architecture and startup/runtime policies in code and configuration.
 - Analysis: evaluate asynchronous execution behavior and design sufficiency.
 - Test: automate behavioral checks through pytest suites and service-level tests.
 ## Glossary
 - Document-oriented persistence: A storage approach that uses flexible document structures for variable data shapes.
 - Prompt artifact: A single Markdown file that defines one transcription prompt and is revised independently.
 - SysML: Systems Modeling Language used to express structured requirements and traceability.
 - System of record: The authoritative persistent store for canonical business data.
@@ -0,0 +1,61 @@
 # UI Behavioral Contracts
 ## Purpose
 This directory defines the current user-facing behavior of the NiceGUI application. It records what each page is for, which routes and actions it exposes, what information it presents, and how success, empty, validation, and failure states behave.
 These documents are written for maintainers and AI contributors. They are behavioral contracts, not historical implementation notes and not substitutes for the database schema.
 ## Current Page Contracts
 - [Home](pages/home.md)
 - [Documents](pages/documents.md)
 - [People](pages/people.md)
 - [Jobs](pages/jobs.md)
 - [Sources](pages/sources.md)
 NiceGUI registers the routes shown in each contract without the `/ui` prefix. The application mounts NiceGUI under `/ui`, so `/documents` in page code is served to a browser as `/ui/documents`.
 ## Authority Hierarchy
 When documents disagree, use this order:
 1. User-facing page intent and accepted behavior: the page contracts in this directory.
 2. Visual and interaction styling: [UI Style Guide](../invariant/ui_style_guide.md).
 3. UI dependency and ownership boundaries: [UI contributor instructions](../../.github/instructions/ui.instructions.md).
 4. Durable failure behavior: [Error Handling invariant](../invariant/error_handling.md).
 5. Durable AI evidence behavior: [Digital Evidence and AI Processing Provenance](../invariant/ai_evidence_and_provenance.md).
 6. Data definitions and relationships: current models plus the [V4 schema](../ver4/schema_v4.md).
 7. Planned behavior changes: the applicable V4.x scope and implementation documents.
 8. Implementation truth: current code and tests.
 If code intentionally changes accepted page behavior, update the corresponding page contract in the same change. If code accidentally differs, correct the implementation rather than rewriting intent to match a defect.
 ## Contract Contents
 Each page contract contains:
 1. Purpose and user goals.
 2. Registered routes and navigation context.
 3. List, detail, and form behavior.
 4. Editable and system-managed information.
 5. Validation, empty, loading, and failure states.
 6. A concise acceptance checklist.
 7. Current implementation and test anchors.
 8. Known limitations and deferred work.
 ## Maintenance Rules
 - Describe current accepted behavior in present tense.
 - Do not mix an obsolete “first release” design with current behavior.
 - Keep future changes in versioned scope documents and link to them from a Deferred Work section.
 - Do not reproduce the complete database field inventory here; include only fields that affect page behavior.
 - Keep service, file, and test anchors current.
 - Do not create separate current-state, target-state, and traceability copies of the same contract.
 - Keep cross-page visual rules in the UI Style Guide instead of repeating them on each page.
 - Keep database joins such as `DocumentPerson` and `JobSource` in schema/architecture documentation unless they directly affect a page interaction.
 ## Current Baseline
 These contracts describe the V4 baseline with completed V4.1 behavior and V4.2 evidence/provenance behavior.
 Draft V4.3 Settings changes are not described as current behavior.
@@ -0,0 +1,105 @@
 # Documents Page Contract
 ## Purpose
 Documents manages the archival record for each historical artifact independently of its source files and transcription jobs. A Document can be created first, linked to people in one or more roles, and used later as the parent for Sources and Jobs.
 ## Routes
 | Route | Purpose |
 | --- | --- |
 | `/documents` | Searchable archival Document list. |
 | `/documents/new` | Create a Document. |
 | `/documents/{document_id}` | View one Document and its related records. |
 | `/documents/{document_id}/edit` | Edit metadata and people-by-role links. |
 | `/documents/{document_id}/delete` | Confirm or block deletion. |
 | `/documents/{document_id}/jobs` | Show Jobs belonging to the Document. |
 | `/documents/{document_id}/sources` | Redirect to the Document-filtered Sources list. |
 ## List Behavior
 - The title is **Archival Documents**.
 - **Create new document** opens the create route.
 - The table defaults to Document Title order and supports search and column sorting.
 - Columns are Document Title, Type, Author, Document Date, and Archive Ref.
 - Document Title is left-aligned; the remaining columns are centered.
 - Author lists all linked people in the `author` role.
 - Date display prefers exact date, then approximate date, then `Unknown`.
 - Selecting a row opens Document Detail.
 - No records displays `No documents found in repository.`
 ## Create and Edit Behavior
 Required:
 - Document name.
 - Document type selected from the Document Type registry.
 Optional:
 - Exact date.
 - Approximate date.
 - Document location.
 - Archive identifier.
 - Notes.
 - Multiple people for every configured Person Role.
 Rules:
 - Exact date must parse as `YYYY-MM-DD`; browser presentation may follow locale.
 - Existing people appear with disambiguating labels.
 - **Create new person** opens Person creation.
 - `person_id` may preselect that Person in the author role on Document creation.
 - An invalid requested Person produces a warning rather than a broken form.
 - `return_to=jobs_new` returns a successful create to Job creation with the new Document selected.
 - Edit includes active and inactive Document Types so historical values remain maintainable.
 - Save success returns to Document Detail.
 ## Detail Behavior
 - The heading shows name, type, and internal ID.
 - The first Source, when present, appears in the dark-room viewer.
 - Archival Metadata shows authors, compact Document date, location, and archive identifier. Notes appear in a separate archival-notes block within the same card.
 - System Logistics shows created and updated timestamps.
 - Related People are grouped by role and link to Person Detail.
 - **Sources & Pipeline Jobs** shows counts and actions for filtered Sources, Document Jobs, and adding a Job.
 - **Edit Document** and **Delete** are available from the header.
 - Invalid IDs and missing Documents produce explicit states without rendering a partial page.
 ## Document Jobs Behavior
 - The page lists the Document's Jobs newest first with status and Job ID.
 - **Open Job** navigates to Job Detail.
 - **Create Job** opens Job creation with the Document selected.
 - No jobs displays an explicit empty state.
 ## Delete Behavior
 - Deletion is blocked while any Source or Job belongs to the Document.
 - The blocked state names the dependency categories and provides navigation back and to Jobs.
 - An unlinked Document requires an explicit permanent-delete action.
 - Success returns to the Documents list.
 ## Acceptance Checklist
 - List columns, alignment, search, sorting, date fallback, and row navigation match this contract.
 - Create/edit enforce name, registered type, and valid exact-date input.
 - Multiple people can be selected independently for each configured role.
 - Person-first Document creation preselects the requested Person as author.
 - Detail links people, Sources, and Jobs to the correct records.
 - Delete never removes a Document with Source or Job dependencies.
 - Service failures use the shared error presenter and never report false success.
 ## Implementation Anchors
 - `src/transcription/ui/pages/documents_page.py`
 - `src/transcription/ui/components/table/documents.py`
 - `src/transcription/services/documents.py`
 - `src/transcription/services/people.py`
 - `tests/ui/test_documents_page.py`
 - `tests/services/test_document_service.py`
 ## Known Limitations and Deferred Work
 - Document creation persists the Document before adding relationship links; a later link failure is surfaced but is not currently one atomic write.
 - Source ordering controls are deferred to the [draft V4.3 scope](../../ver4.3/scope_boundary_v4_3.md).
@@ -0,0 +1,63 @@
 # Home Page Contract
 ## Purpose
 Home provides a user-maintained landing page for the local archive. It combines one current image with Markdown text and lets the operator edit both without changing application source or prompt assets.
 ## Routes
 | Route | Browser path | Purpose |
 | --- | --- | --- |
 | `/homepage` | `/ui/homepage` | View current homepage image and Markdown. |
 | `/homepage/edit` | `/ui/homepage/edit` | Upload an image and edit Markdown. |
 The application root and `/ui` redirect to `/ui/homepage`.
 ## View Behavior
 - The visible page heading is **Home**; the browser tab title is **VibeScribe Home**.
 - The latest homepage image appears in the shared dark-room viewer.
 - Saved Markdown is rendered in the **Home Text** card.
 - Missing text displays `No homepage text saved yet.`
 - Missing image displays the viewer's empty state.
 - **Edit Home Page** opens the edit route.
 ## Edit Behavior
 - The image upload accepts JPEG, PNG, GIF, WebP, BMP, and TIFF files.
 - A successful upload immediately stores the file, updates the preview to that image, and displays a positive notification.
 - The Markdown textarea is initialized from the currently stored homepage text.
 - **Save** writes the textarea content, displays `Homepage saved`, and returns to Home.
 - **Cancel** returns to Home without saving textarea changes. An image already uploaded during the edit session remains stored.
 ## Storage Contract
 - Homepage content is mutable application data under `data/homepage`.
 - Markdown is stored in `homepage.md`.
 - Uploaded images keep a sanitized basename.
 - The view selects the supported image with the most recent modification time.
 - Homepage files are not transcription prompts and are not database records.
 ## Acceptance Checklist
 - `/`, `/ui`, and the application brand reach Home.
 - Home renders with or without stored Markdown and image content.
 - Edit loads existing Markdown.
 - A supported image upload updates the preview and becomes the latest homepage image.
 - Save persists Markdown and returns to Home.
 - Cancel does not save changed Markdown.
 ## Implementation Anchors
 - `src/transcription/ui/pages/home_page.py`
 - `src/transcription/ui/homepage_store.py`
 - `src/transcription/ui/components/app_shell.py`
 - `tests/ui/test_upload_page.py`
 - `tests/ui/test_navigation_and_mounts.py`
 - `tests/ui/test_pages_registration.py`
 ## Known Limitations
 - Homepage storage is fixed under the repository/application `data` directory rather than a configured application-data root.
 - Uploading an image is immediate and is not rolled back by Cancel.
 - The editor does not currently delete or select among previously uploaded images.
@@ -0,0 +1,94 @@
 # Jobs Page Contract
 ## Purpose
 Jobs manages transcription processing runs. A Job belongs to one Document, links one or more Source pages, records processing provenance, and exposes lifecycle actions without making lifecycle fields directly editable.
 ## Routes
 | Route | Purpose |
 | --- | --- |
 | `/jobs` | Searchable processing Job list. |
 | `/jobs/new` | Create and queue a Job. |
 | `/jobs/{job_id}` | View status, execution logistics, and related records. |
 | `/jobs/{job_id}/cancel` | Confirm cancellation. |
 | `/jobs/{job_id}/resubmit` | Confirm resubmission of failed Sources. |
 | `/jobs/{job_id}/delete` | Confirm or block deletion. |
 ## List Behavior
 - The title is **Transcription Pipeline Jobs**.
 - **Create job** opens Job creation and **Refresh** reloads the table.
 - Columns are Job ID, Status, Source Filename, Retries, Created, and Updated.
 - Search covers Job ID, filename, and status.
 - Status is displayed as a semantic status chip.
 - Selecting a row opens Job Detail.
 - No records displays `No job records found in repository.`
 ## Create Behavior
 - A Target Document and at least one source file are required.
 - `document_id` may preselect a Target Document.
 - If no Documents exist, the page explains the prerequisite and links to Document creation with a return path.
 - Provider and Model are optional request overrides.
 - Upload accepts JPEG, PNG, TIFF, and PDF files and supports multiple/folder selection.
 - The visible upload queue is sorted alphabetically by original filename.
 - Files can be removed individually or cleared before submission.
 - Helper text explains numeric filename prefixes for page ordering.
 - Submission creates the Job, Source records, and JobSource links, notifies the worker, and opens Job Detail.
 ## Detail and Lifecycle Behavior
 - The heading shows Job ID and a status badge.
 - Execution Logistics shows provider, model, prompt, retry count, and last update.
 - Document Links open the parent Document and Job-filtered Sources.
 - Queued and processing Jobs show an auto-refresh notice and reload every four seconds.
 - Polling stops when the Job becomes terminal or a refresh fails.
 - Queued and processing Jobs expose **Cancel**.
 - Jobs other than `transcribed` expose **Resubmit** under the current UI rule. The service blocks resubmission while processing is active or when no failed Sources exist.
 - All Jobs expose **Delete Job**, subject to explicit evidence-deletion guardrails.
 - Invalid and missing IDs produce explicit states.
 ## Cancel Behavior
 - The confirmation explains that processing stops and remaining non-transcribed Sources become failed.
 - The service decides whether the current state permits cancellation.
 - Success updates the Job, notifies the worker, and returns to Job Detail.
 ## Resubmit Behavior
 - The page shows current status and failed Source count.
 - The page explains that resubmission queues failed linked Sources while preserving immutable prior attempt evidence.
 - The service blocks submission while processing is active or when no failed Sources exist.
 - `JobSource` remains the latest compatibility projection, while every provider call appends an `ExecutionAttempt`.
 - The latest successful `Source.raw_transcription` projection remains available while a retry is pending or fails.
 - Success reports the number of resubmitted Sources and returns to Job Detail.
 ## Delete Behavior
 - Deletion is blocked while status is `processing`.
 - Allowed deletion explicitly warns that related `JobSource` projections,
  immutable execution attempts, captured transport responses, and attempt-owned
  artifacts are permanently removed.
 - Source records and source files remain available for separate deletion.
 - Success returns to the Jobs list.
 ## Acceptance Checklist
 - Job creation cannot proceed without a valid Document and at least one Source.
 - Upload ordering and removal controls match the displayed queue.
 - Detail shows current status and provenance summary with correct related links.
 - Active Jobs refresh without overlapping permanent polling after terminal state.
 - Cancel, resubmit, and delete honor service guardrails and show actionable failures.
 - Lifecycle fields cannot be edited directly.
 ## Implementation Anchors
 - `src/transcription/ui/pages/jobs_page.py`
 - `src/transcription/ui/components/table/jobs.py`
 - `src/transcription/services/jobs.py`
 - `src/transcription/services/store.py`
 - `src/transcription/services/workflows.py`
 - `tests/ui/test_jobs_page.py`
 - `tests/services/test_job_service.py`
 - `tests/services/test_store.py`
@@ -0,0 +1,90 @@
 # People Page Contract
 ## Purpose
 People manages reusable historical-person records. A Person may appear in many Documents under different relationship roles and may optionally carry a portrait and FamilySearch identifier.
 ## Routes
 | Route | Purpose |
 | --- | --- |
 | `/people` | Searchable People list. |
 | `/people/new` | Create a Person. |
 | `/people/{person_id}` | View one Person and linked Documents. |
 | `/people/{person_id}/edit` | Edit the Person. |
 | `/people/{person_id}/delete` | Confirm permanent deletion. |
 ## List Behavior
 - The title is **Archival Entities: People**.
 - **Create new person** opens the create route.
 - The table defaults to Full Name order and supports search and column sorting.
 - Columns are Full Name, Display Name, Maiden Name, Birth Date, and Death Date.
 - Full Name is left-aligned; Display Name, Maiden Name, and date columns are centered.
 - Birth and death values independently prefer exact date, then approximate date, then `Unknown`.
 - Selecting a row opens Person Detail.
 - No records displays `No person records found in repository.`
 ## Create and Edit Behavior
 Required:
 - Full name.
 Optional:
 - Display name and maiden name.
 - Exact and approximate birth/death dates.
 - Birth/death places.
 - Biography.
 - Portrait path or uploaded portrait.
 - FamilySearch ID.
 Rules:
 - Missing Full name blocks save with a warning.
 - Exact date inputs are native browser date inputs.
 - FamilySearch IDs are normalized and validated by `PeopleService`.
 - Portrait uploads are stored under the configured upload root in a Person-specific directory and update Portrait path.
 - Metadata JSON remains hidden.
 - Save success returns to Person Detail.
 ## Detail Behavior
 - The header provides **New Document**, **Edit Person**, and **Delete**.
 - **New Document** opens Document creation with this Person requested for author preselection.
 - The portrait viewer resolves supported relative upload paths and absolute HTTP/data URLs.
 - Biographical Record shows names, compact birth/death dates, places, and an **Open in FamilySearch** link when an ID exists.
 - Biography has an explicit empty value.
 - Linked Documents show Document name, relationship role, and an action to open Document Detail.
 - No links shows both an empty state and guidance to link from a Document workflow.
 - System Logistics shows created and updated timestamps.
 ## Delete Behavior
 - The page warns when linked Document relationships exist.
 - Confirmed deletion removes the Person and its relationship links; it does not delete Documents.
 - Success returns to the People list.
 - Missing or already-deleted records return to a safe list state.
 ## Acceptance Checklist
 - List fields, alignment, date fallback, search, sorting, and navigation match this contract.
 - Full name is enforced on create and edit.
 - FamilySearch ID validation and link generation use the fixed supported identifier format.
 - Portrait upload and rendering remain constrained to supported media paths.
 - New Document carries the Person context.
 - Linked Documents show the correct role and target.
 - Delete wording distinguishes removal of relationship links from deletion of Documents.
 ## Implementation Anchors
 - `src/transcription/ui/pages/people_page.py`
 - `src/transcription/ui/components/table/people.py`
 - `src/transcription/services/people.py`
 - `tests/ui/test_people_page.py`
 - `tests/services/test_v2_crud.py`
 ## Deferred Work
 - Structured name fields, merge/deduplication, advanced metadata editing, and Person-side relationship editing are not current behavior.
@@ -0,0 +1,89 @@
 # Sources Page Contract
 ## Purpose
 Sources manages individual archived page/file records. It provides source-media viewing, current processing context, provider evidence inspection, previous/next page navigation, and human revision without allowing machine output to be edited.
 ## Routes
 | Route | Purpose |
 | --- | --- |
 | `/sources` | Global or filtered Source list. |
 | `/sources/{source_id}` | View media, transcription, revision, metadata, and evidence. |
 | `/sources/{source_id}/delete` | Confirm or block deletion. |
 The list accepts optional `document_id` and `job_id` query parameters. Document context takes precedence if both parse successfully.
 ## List Behavior
 - The title is **Source Asset Records**, **Sources for Document**, or **Sources for Job** according to context.
 - Global context provides **Create Job**.
 - Filtered context provides **Back to Document** or **Back to Job**.
 - Rows are ordered by page number and then upload name.
 - Columns are Document Name, Page Number, Upload Title, Status, and Error Detail.
 - Document Name, Upload Title, and Error Detail are left-aligned; Status is centered.
 - Stored Filename is intentionally absent from the list.
 - Selecting a row opens Source Detail.
 - No records displays `No source asset records found in repository.`
 ## Detail Behavior
 - The heading shows page number, upload name, and Source ID.
 - **Back to Sources** returns to the global list.
 - **Delete Source** opens the guarded delete route.
 - Previous and Next navigate only among Sources belonging to the same Document in page order; unavailable boundary actions are disabled.
 - The media viewer resolves the stored Source path through the configured upload root.
 - Transcription Text is read-only and prefers the latest JobSource transcription, then the Source projection.
 - Editable Revision is seeded from an existing revision or the machine transcription.
 - Source Metadata shows upload name, stored filename, page number, Document Name, Document ID, and stored path. Source ID appears in the page-header subtitle.
 - SourceJob Metadata shows latest status, Job ID, execution time, provider, model, prompt, and failure detail.
 - Revision Logistics shows revised state, last-revised time, and upload time.
 ## Provider Evidence
 - Provider Evidence is associated with the latest JobSource execution.
 - New attempts display separate expandable Request Manifest, Transport Response, OpenRouter SDK Response Snapshot,
  Normalized Metadata, Software Context, and Derived Artifacts sections.
 - Historical `raw_api_response` values are labeled as OpenRouter SDK response snapshots.
 - Missing evidence has an explicit empty state.
 - Historical executions explicitly state that exact transport evidence was not captured.
 - **Export Evidence** downloads a versioned package containing source identity, attempts, artifacts, relationships,
  schema versions, and integrity digests without source binaries, credentials, or machine-local source paths.
 ## Revision Behavior
 - Machine transcription is never edited directly.
 - A revision must contain non-whitespace text.
 - Save persists revised text and updates the saved timestamp without leaving the page.
 - Reset restores the in-memory revision from page load or the most recent successful save. When no revision exists, it restores the machine transcription; it does not re-read the database.
 - A failed latest execution displays guidance that a human revision can preserve corrected text.
 ## Delete Behavior
 - Deletion is allowed only when the Source has no JobSource links.
 - A linked Source shows cleanup guidance and navigation to Jobs.
 - An unlinked Source requires explicit permanent deletion.
 - Success returns to the Sources list.
 ## Acceptance Checklist
 - Global, Document-filtered, and Job-filtered lists show the correct context and return action.
 - List columns and alignments match this contract and omit Stored Filename.
 - Previous/next navigation never crosses Document boundaries.
 - Detail keeps machine output read-only and human revision separately editable.
 - Empty, failed, and missing-evidence states remain explicit.
 - JSON evidence is readable without being mislabeled as native transport evidence.
 - Delete cannot remove a Source with processing-history links.
 ## Implementation Anchors
 - `src/transcription/ui/pages/sources_page.py`
 - `src/transcription/ui/components/table/sources.py`
 - `src/transcription/services/sources.py`
 - `tests/ui/test_sources_page.py`
 - `tests/services/test_transcription_service.py`
 - `tests/services/test_v2_crud.py`
 ## Planned Changes
 - Source page reordering is deferred beyond V4.3 and may be reconsidered if a demonstrated workflow need emerges.
@@ -1,320 +0,0 @@
 # Version 1 Implementation Plan
 This plan defines the path from MVP to **Version 1 complete**.  
 The objective is to deliver the full scoped product with production readiness, while explicitly separating refinements/enhancements into a future document.
 ---
 ## 0) Plan Governance & Scope Control (Foundation)
 **Goal:** Keep execution focused on V1 completion, not optimization/perfection.
 ### Implementation Steps
 1. Create and maintain a **V1 Traceability Matrix**:
   - Requirement ID
   - Current status (`done`, `partial`, `not started`)
   - Owner
   - Validation method
 2. Define V1 completion gates:
   - Functional complete
   - Operationally complete
   - Production-ready complete
 3. Snapshot the MVP baseline (tag/changelog reference).
 4. Create a standing rule: any non-V1 idea is logged to a separate enhancements backlog document (to be named later), not added to active V1 scope unless explicitly approved.
 ### Deliverables
 - `docs/ver1/ver1.md` (this plan)
 - V1 traceability artifact (linked from here when created)
 ### Exit Criteria
 - Every in-scope requirement has explicit ownership and status.
 - Scope-change process is agreed and followed.
 ---
 ## 1) Architecture Consolidation
 **Goal:** Align implementation with the intended architecture and reduce MVP shortcuts.
 ### Implementation Steps
 1. Compare implemented modules/components with architecture documentation.
 2. Identify and classify architectural debt:
   - Temporary coupling
   - Missing interfaces
   - Placeholder services/components
 3. Resolve high-risk architectural gaps first.
 4. Record key decisions and tradeoffs in ADRs.
 ### Deliverables
 - Updated architecture diagrams and boundaries
 - ADR entries for major decisions
 ### Exit Criteria
 - Architecture documentation reflects system reality.
 - Critical architecture risks are addressed or scheduled with owners/dates.
 ---
 ## 2) Error Handling & Reliability Hardening
 **Goal:** Ensure predictable, safe behavior under failure conditions.
 ### Implementation Steps
 1. Standardize error taxonomy and envelope format across all layers.
 2. Ensure clear distinction between:
   - User-facing errors
   - Internal/system errors
   - Retryable vs non-retryable failures
 3. Add resilience controls where needed:
   - Timeouts
   - Retries with backoff
   - Circuit breaking / fallback logic
 4. Add failure-path tests for critical workflows.
 ### Deliverables
 - Error code catalog/reference
 - Failure mode test coverage for critical paths
 ### Exit Criteria
 - Error behavior is consistent across major flows.
 - Known failure scenarios are tested and pass.
 ---
 ## 3) Functional Completion by Requirement Domain
 **Goal:** Complete all V1 functional requirements in a risk-aware order.
 ### Recommended Order
 1. Business-critical end-user flows
 2. Data integrity and consistency capabilities
 3. Admin/operational controls
 4. Lower-priority UX and quality-of-life items that are in V1 scope
 ### Implementation Steps
 For each requirement slice:
 1. Finalize contract/schema
 2. Implement domain logic
 3. Implement persistence/state changes
 4. Integrate API/UI
 5. Add automated tests
 6. Update docs
 ### Deliverables
 - Requirement completion report with validation evidence
 ### Exit Criteria
 - All V1 “must-have” requirements are complete and validated.
 ---
 ## 4) Data Model, Migration, and Backfill Safety
 **Goal:** Ensure data model and migrations are production-safe.
 ### Implementation Steps
 1. Validate schema against final V1 domain needs.
 2. Implement forward-safe migrations.
 3. Define rollback/mitigation plans for migration failures.
 4. Build and verify backfill scripts (if needed).
 5. Add migration rehearsal in staging with representative data.
 ### Deliverables
 - Migration runbook
 - Backfill verification checklist
 ### Exit Criteria
 - Migration plan validated in staging.
 - No unresolved data-loss risk for V1 rollout.
 ---
 ## 5) Security, Access Control, and Compliance Baseline
 **Goal:** Close MVP security gaps and establish V1 baseline controls.
 ### Implementation Steps
 1. Complete authn/authz coverage for all routes/actions.
 2. Enforce input validation and output sanitization.
 3. Verify secret management and credential rotation process.
 4. Add audit logging for sensitive operations.
 5. Run dependency/security scanning in CI and remediate findings.
 ### Deliverables
 - Security checklist with status
 - Threat/risk update for V1 scope
 ### Exit Criteria
 - No unresolved critical/high vulnerabilities for V1 launch.
 - Access control behavior verified by tests.
 ---
 ## 6) Observability & Operability
 **Goal:** Make system behavior observable and supportable in production.
 ### Implementation Steps
 1. Standardize structured logging and correlation IDs.
 2. Add core metrics:
   - Latency
   - Throughput
   - Error rates
   - Resource saturation
 3. Add tracing for critical request/workflow paths.
 4. Define SLOs/SLIs and alert thresholds.
 5. Prepare incident response and rollback runbooks.
 ### Deliverables
 - Dashboards and alerts
 - Operations runbooks
 ### Exit Criteria
 - Team can detect, triage, and remediate incidents quickly.
 - Core production signals are available and reliable.
 ---
 ## 7) Test Strategy Expansion & Quality Gates
 **Goal:** Raise confidence for repeatable, low-risk releases.
 ### Implementation Steps
 1. Expand unit and integration tests across V1 features.
 2. Add contract tests between key components/services.
 3. Add end-to-end tests for critical user journeys.
 4. Add non-functional tests where relevant:
   - Performance/load
   - Soak
   - Failure-injection scenarios
 5. Enforce CI quality gates (tests, lint, type checks, security scans).
 ### Deliverables
 - Test matrix with ownership
 - CI gate definition and thresholds
 ### Exit Criteria
 - Critical-path regressions are blocked automatically.
 - Test coverage and reliability thresholds meet V1 targets.
 ---
 ## 8) Performance & Scalability Validation
 **Goal:** Meet expected V1 performance at projected load.
 ### Implementation Steps
 1. Define performance budgets per key flow.
 2. Benchmark current behavior in staging.
 3. Optimize bottlenecks (queries, caching, concurrency, etc.).
 4. Re-test after each optimization and compare against budget.
 5. Document known limits and safe operating bounds.
 ### Deliverables
 - Performance benchmark report
 - Optimization log
 ### Exit Criteria
 - V1 performance targets met for expected usage profile.
 ---
 ## 9) Release Engineering & Environment Readiness
 **Goal:** Make deployment repeatable, controlled, and reversible.
 ### Implementation Steps
 1. Harden CI/CD pipeline with clear promotion gates.
 2. Ensure config parity and consistency across environments.
 3. Define rollout strategy (phased/canary/limited release as applicable).
 4. Validate rollback procedures in staging.
 5. Produce release checklist and ownership model.
 ### Deliverables
 - Release playbook
 - Environment readiness checklist
 ### Exit Criteria
 - Deployment and rollback are rehearsed and reliable.
 - Release process is executable without tribal knowledge.
 ---
 ## 10) Documentation Completion
 **Goal:** Ensure V1 can be built, operated, and supported from documentation.
 ### Implementation Steps
 1. Update core project docs to match final V1 behavior:
   - Architecture
   - Error handling
   - Requirements status
   - Index/navigation
   - Intent alignment summary
 2. Add operator troubleshooting guides.
 3. Add integration/API examples for consumers.
 4. Publish changelog/version notes for V1.
 ### Deliverables
 - Updated documentation set for V1
 - V1 release notes
 ### Exit Criteria
 - A new team member can run/support the system using docs alone.
 ---
 ## 11) Final Validation, UAT, and Launch
 **Goal:** Confirm readiness and launch V1 safely.
 ### Implementation Steps
 1. Run full-system acceptance validation against the V1 traceability matrix.
 2. Conduct stakeholder UAT and capture sign-off.
 3. Execute production readiness review.
 4. Launch in controlled phases and monitor key signals.
 ### Deliverables
 - UAT/PRR sign-off records
 - Launch checklist and monitoring plan
 ### Exit Criteria
 - Stakeholder approval achieved.
 - Launch metrics are stable within defined thresholds.
 ---
 ## 12) Post-Launch Stabilization (30–60 Days)
 **Goal:** Consolidate V1 in production before major expansion.
 ### Implementation Steps
 1. Track incidents, defects, and user feedback.
 2. Prioritize stabilization fixes with short cycle times.
 3. Remove temporary flags/mitigations introduced during launch.
 4. Produce post-launch retrospective and handoff to standard roadmap cadence.
 ### Deliverables
 - Stabilization report
 - Prioritized backlog update
 ### Exit Criteria
 - Incident/error rates converge to steady-state targets.
 - V1 transitions from launch mode to normal operations.
 ---
 ## Recommended Execution Rhythm
 - **Weekly:** Requirement closure + risk review
 - **Biweekly:** Release train with quality gates
 - **Milestone reviews:** After phases 2, 6, 9, and 11
 ---
 ## Scope Discipline Rule (V1 Focus)
 To preserve delivery focus:
 - V1 execution prioritizes completion of scoped requirements.
 - Refinements/enhancements are captured in a separate future document and backlog.
 - Only explicitly approved scope changes may enter this plan.
@@ -0,0 +1,146 @@
 # Implementation Plan (Version 4.1)
 ## Goal
 Deliver the V4.1 usability revision as a small, behavior-safe increment over the V4 baseline.
 ## Implementation Principles
 - Keep presentation formatting in UI components and route orchestration in pages.
 - Keep persistence and cross-record queries behind service boundaries.
 - Reuse shared table and date-label helpers instead of duplicating fallback logic.
 - Make the FamilySearch schema change additive and nullable.
 - Add focused tests for changed behavior before broad regression verification.
 ## Current Project Impact
 | Area | Expected impact |
 | --- | --- |
 | Persistence | Add nullable `Person.family_search_id`; provide the repository's supported schema-upgrade path for existing databases. |
 | People service | Normalize and validate FamilySearch IDs at the domain/service boundary if model validation does not fully cover writes. |
 | Documents UI | Add table data, improve relationship labels/links, compact date display, and combine processing navigation. |
 | People UI | Add table date fields, Person-first Document creation, compact date display, and FamilySearch controls. |
 | Sources service/UI | Query adjacent document Sources and add bounded navigation; revise list columns and wrapping. |
 | Jobs UI | Refresh the active detail read model on a timer until terminal status. |
 | Shared UI | Add reusable constrained/wrapped table presentation and compact date formatting where appropriate. |
 | Tests | Update model/service and UI coverage for all affected workflows. |
 ## Implementation Phases
 ### 1. Add Shared Presentation Rules
 - Review `ui/components/table/common.py` and packaged theme CSS for the narrowest reusable table-width solution.
 - Add reusable styles or column slots for constrained, wrapping, left-aligned text.
 - Add a shared formatter for exact/approximate/unknown dates if it can be reused without coupling components to persistence.
 - Preserve sorting and search behavior for rendered display values.
 ### 2. Update Archival List Tables
 - Extend the Document table read model with author names and the compact document date.
 - Build author display from eagerly loaded document-person links using the `author` role.
 - Apply title/type alignment and constrained title wrapping.
 - Extend the Person table read model with compact birth and death date values.
 - Apply Display Name and Maiden Name alignment.
 - Remove Stored Filename from the Source table read model only if no other list behavior consumes it; always remove its rendered column.
 - Constrain and left-align the requested Source columns.
 - Add or update UI component tests for serialized rows, columns, and fallback formatting.
 ### 3. Improve Document Relationship Workflows
 - Introduce one person-label formatter that combines preferred Display Name, Full Name context, and known birth year without implying uniqueness.
 - Use Person UUIDs as selector values.
 - Apply the formatter to every relationship role selector.
 - Change Related People rows into actions that navigate to `/people/{person_id}`.
 - Replace separate exact/approximate rows in view mode with one conditional Document Date row.
 - Combine Pipeline Jobs and Sources into one related-processing card beneath Related People.
 - Preserve existing job/source counts and navigation actions.
 ### 4. Add the Person-First Document Workflow
 - Add a New Document action on Person Detail.
 - Pass the Person UUID through a narrowly defined query parameter to `/documents/new`.
 - Validate the requested UUID against the loaded people list.
 - Preselect that person in the intended default relationship role. Use `author` unless a different role is explicitly encoded later.
 - Ignore invalid or unavailable preselection values with the application's normal visible error/notification behavior.
 - Confirm ordinary `/documents/new` behavior remains unchanged.
 ### 5. Add FamilySearch Person References
 - Add nullable, unique `family_search_id` to the `Person` model and schema.
 - Implement a non-destructive upgrade for existing SQLite and PostgreSQL databases using the repository's established schema-management approach.
 - Normalize values by trimming and uppercasing.
 - Validate the `XXXX-XXX` alphanumeric identifier shape and return a clear validation error for malformed input.
 - Report duplicate identifiers as a deterministic conflict rather than a generic persistence failure.
 - Add the field to Person create/edit forms and preserve it during updates.
 - Add a URL builder that safely inserts only a validated identifier into the fixed FamilySearch details URL.
 - Render a FamilySearch action on Person Detail only when an identifier is present.
 - Add persistence, normalization, validation, form, and link-generation tests.
 ### 6. Add Source Page Navigation
 - Add a Sources service query that returns previous/current/next context for a Source within its Document.
 - Define ordering by `page_number`, with a stable secondary key such as Source UUID for defensive determinism.
 - Keep navigation bounded to the current `document_id`.
 - Render previous and next actions adjacent to the source viewer or detail header.
 - Disable or omit unavailable boundary actions.
 - Test first, middle, last, single-page, and cross-document cases.
 ### 7. Add Job Detail Auto-Refresh
 - Make Job Detail content refreshable without rebuilding unrelated global navigation.
 - Start a NiceGUI timer only for queued or processing jobs.
 - On each tick, re-read the Job through `JobService` and refresh the detail content.
 - Use a 4-second default interval.
 - Stop or deactivate the timer when status becomes completed, partial success, failed, or cancelled, according to the model's actual terminal states.
 - Prevent overlapping refresh callbacks.
 - Retain existing error presentation if a refresh read fails.
 - Add UI tests for timer creation, refresh, and terminal-state stopping.
 ### 8. Simplify View-Mode Date Rows
 - On Document Detail, show exact date, else approximate date, else one not-set value.
 - On Person Detail, apply the same independent rule to birth and death.
 - Do not hide either input in create/edit mode.
 - Test each exact, approximate, and absent state.
 ### 9. Verification and Documentation Alignment
 - Run the focused model/service/UI tests covering changed surfaces.
 - Run the existing regression suite appropriate to persistence and UI changes.
 - Confirm SQLite and PostgreSQL model compatibility at the schema-definition level.
 - Update V4.1 documentation if implementation reveals a necessary boundary change; do not silently expand scope.
 ## Recommended Delivery Order
 1. Shared formatters and table presentation.
 2. Additive Person schema change and FamilySearch validation.
 3. Document and Person list/detail changes.
 4. Person-first Document workflow.
 5. Source navigation.
 6. Job polling.
 7. Focused and regression verification.
 ## Done When
 - Every V4.1 acceptance criterion is demonstrated or covered by a focused test.
 - Existing Person rows remain valid after the nullable schema addition.
 - Duplicate FamilySearch references cannot be assigned to multiple local Person records.
 - FamilySearch links are generated only from normalized, validated IDs.
 - Auto-refresh performs no polling after a terminal job state.
 - Adjacent Source navigation never crosses Document boundaries.
 - The existing V4 workflows remain operational.
 ## Out of Scope
 - Page reordering.
 - Settings management.
 - External genealogy API integration.
 - Raw `.env` editing.
 - Theme editing.
 ## Related Local References
 - [V4.1 Scope Boundary](scope_boundary_v4_1.md)
 - [V4 Implementation Plan](../ver4/implementation_plan_v4.md)
 - [V4 Requirements](../ver4/requirements_v4.md)
 - [V4 Error Handling Policy](../ver4/error_handling_v4.md)
@@ -0,0 +1,142 @@
 # V4.1 Scope Boundary
 This document defines the scope of the first incremental revision to Version 4. V4 remains the product and architecture baseline; V4.1 adds focused usability improvements and one additive Person field.
 ## Purpose
 - Improve common archival record workflows without redesigning the application.
 - Resolve table overflow, ambiguous person selection, and unnecessary navigation.
 - Add a manually maintained FamilySearch person reference without introducing external API integration.
 ## In Scope
 ### 1. Archival Documents List
 - Keep every table column within the available page width.
 - Limit and wrap long Document Title values.
 - Left-align Document Title and Type.
 - Add Author and Document Date columns.
 - Display all people linked through the `author` role in the Author column.
 - Display exact document date when present, otherwise approximate date when present, otherwise `Unknown`.
 ### 2. Document Detail and Editing
 - Use an unambiguous label in person selectors. The label should prefer Display Name, retain Full Name for context, and include the birth year when known.
 - Do not require Display Name to be unique.
 - Link each Related People entry to its Person Detail page.
 - In view mode, display only the populated exact or approximate document date row. Display a single unknown/not-set state when neither exists.
 - Keep both exact and approximate inputs available in create/edit mode.
 - Move source navigation out from beneath the media viewer.
 - Present Pipeline Jobs and Sources together in one related-processing card with counts and actions.
 ### 3. People List
 - Left-align Display Name and Maiden Name.
 - Display exact birth date when present, otherwise approximate birth date when present, otherwise `Unknown`.
 - Add a Death Date column with the same fallback rule.
 ### 4. Person Detail and Editing
 - Add a New Document action that opens Document creation with the current person preselected.
 - Preserve the existing Document-first workflow.
 - In view mode, display only the populated exact or approximate row for each of birth and death date. Display a single unknown/not-set state when neither value exists.
 - Keep both exact and approximate inputs available in create/edit mode.
 ### 5. FamilySearch Reference
 - Add a nullable, unique `family_search_id` field to `Person`.
 - Allow the field to be entered and changed in Person create/edit flows.
 - Trim whitespace, normalize the identifier to uppercase, and validate it against the supported
  `XXXX-XXX` alphanumeric shape before persistence.
 - When an identifier exists, show a FamilySearch action on Person Detail linking to:
  `https://www.familysearch.org/tree/person/details/{family_search_id}`
 - Construct the URL in application code; do not persist the full URL.
 ### 6. Source List and Detail
 - Keep every Source Asset Records table column within the available page width.
 - Limit and wrap long Document Name, Upload Title, and Error Detail values.
 - Left-align Document Name, Upload Title, and Error Detail.
 - Remove Stored Filename only from the Source Asset Records table. Continue storing it and showing it on Source Detail.
 - On Source Detail, add previous and next navigation for Sources belonging to the same Document, ordered by `page_number`.
 - Disable or omit the previous/next action at the first/last page.
 ### 7. Job Detail
 - Automatically refresh Job Detail while the job is in a non-terminal state.
 - Use a modest interval in the 3-5 second range.
 - Stop polling when the job reaches a terminal state or the page is no longer active.
 - Preserve manual navigation and existing job actions.
 ### 8. Homepage Storage Decision
 - Continue treating homepage markdown and images as mutable application data, not prompt artifacts or packaged source assets.
 - Keep homepage content separate from `prompts`.
 - Defer relocation to a configurable application-data root unless the existing location prevents normal installed or deployed operation.
 ## Out of Scope
 - Source page renumbering or reordering.
 - A Settings page.
 - Editing `.env` or secrets through the UI.
 - Runtime theme editing.
 - FamilySearch authentication, API calls, search, import, synchronization, or conflict resolution.
 - Ancestry references or other genealogy providers.
 - Google Maps links from place fields.
 - Enforcing unique Display Name values.
 - Changes to transcription execution or provider behavior.
 - Changes to the V4 API solely to expose the V4.1 presentation enhancements.
 ## Locked Design Decisions
 ### A. Person Selector Identity
 - Selection values remain internal Person UUIDs.
 - Display labels provide disambiguating context but are not identity keys.
 - Duplicate Full Name and Display Name values remain valid.
 ### B. Date Presentation
 - Exact dates take precedence over approximate/raw dates for compact list and view presentation.
 - Create/edit forms retain both fields so either representation can be maintained.
 - V4.1 does not introduce a new mutual-exclusion database constraint.
 ### C. FamilySearch Storage
 - Store only the FamilySearch person identifier.
 - Treat a FamilySearch person identifier as unique across local Person records.
 - Use one dedicated nullable Person field while FamilySearch is the only supported external genealogy reference.
 - Reconsider a generic external-reference model only when a second provider or multiple references per person are required.
 ### D. Stored Filename
 - Stored Filename remains part of the Source model and Source Detail diagnostics.
 - Only the list-table column is removed.
 ## Data and Compatibility Policy
 - The `family_search_id` addition must be nullable and non-destructive for existing Person rows.
 - Existing records, routes, relationships, jobs, Sources, prompt provenance, and uploaded media remain valid.
 - UI changes must preserve both Document-first and Person-first workflows.
 - V4.1 must remain portable across SQLite and PostgreSQL.
 ## Acceptance Criteria
 1. Document, Person, and Source tables fit their page containers at supported desktop widths without losing requested columns.
 2. Long table text wraps or is constrained without forcing important columns outside the table container.
 3. Duplicate-named people can be distinguished in every document relationship selector.
 4. Related People entries navigate to the correct Person Detail page.
 5. Compact date displays consistently use exact, then approximate, then unknown fallback behavior.
 6. Starting from Person Detail can create a Document with that person preselected without breaking normal Document creation.
 7. A valid FamilySearch ID is persisted and produces the correct Person Detail hyperlink; absent IDs produce no action.
 8. Source previous/next actions remain within the same Document and follow `page_number`.
 9. Active Job Detail pages update without manual refresh and stop polling after terminal status.
 10. Stored Filename is absent from the Source list table but remains available on Source Detail.
 11. Focused automated tests pass and unaffected V4 behavior remains intact.
 ## Related Local References
 - [V4.1 Implementation Plan](implementation_plan_v4_1.md)
 - [V4 Scope Boundary](../ver4/scope_boundary_v4.md)
 - [V4 Architecture](../ver4/architecture_v4.md)
 - [V4 Schema](../ver4/schema_v4.md)
@@ -0,0 +1,258 @@
 # Implementation Plan (Version 4.2)
 ## Goal
 Make processing evidence precise, append-only, secret-safe, and exportable while preserving every existing record and creating a provider-neutral home for future OCR/layout artifacts.
 ## Implementation Principles
 - Implement the [Digital Evidence and AI Processing Provenance invariant](../invariant/ai_evidence_and_provenance.md), not a provider-specific approximation of it.
 - Capture transport evidence before SDK parsing.
 - Keep exact evidence separate from parsed and normalized representations.
 - Prefer additive schema evolution and explicit compatibility behavior.
 - Reference source content by digest rather than duplicating it in request JSON.
 - Use allowlists for safe metadata capture.
 - Keep persistence and evidence semantics behind service boundaries.
 - Do not change the default model until a representative benchmark supports that decision.
 ## Current-State Gaps
 | Current behavior | Gap to close |
 | --- | --- |
 | `Source` stores original file path, digest, and size. | Media type and image/page geometry used for an execution are not frozen with that execution. |
 | `Job` stores prompt text/hash, requested model after resolution, temperature, and `top_p`. | The complete effective request structure, omitted-versus-explicit parameter state, routing constraints, and software versions are not frozen. |
 | `JobSource.raw_api_response` stores `model_dump()` output from the OpenRouter SDK. | The exact HTTP body can be normalized by OpenRouter and filtered again by the SDK before persistence. |
 | `JobSource.ai_metadata` stores finish reason and basic token counts. | Detailed accounting remains only in the SDK snapshot and is not a substitute for exact evidence. |
 | Provider exceptions become application errors. | Safe HTTP error bodies, statuses, headers, and no-response distinctions are not persisted. |
 | Worker logs elapsed time. | Execution duration is not stored on `JobSource`. |
 | Source Detail displays AI metadata and the SDK snapshot. | The UI does not identify evidence layers or expose request/transport/software provenance. |
 | No generic processing-artifact model exists. | Future OCR geometry would require ad hoc provider fields or an unrelated schema. |
 ## Expected Project Impact
 | Area | Expected impact |
 | --- | --- |
 | Database models and upgrades | Add execution-specification, transport-evidence, timing, software-context, and generic artifact storage without removing existing columns. |
 | OpenRouter adapter | Introduce a transport boundary that can capture exact body/status/safe headers before typed SDK parsing, or use supported SDK hooks that expose the unparsed response reliably. |
 | Provider contract | Return structured evidence for success and failure without leaking provider-specific transport concerns into workflow orchestration. |
 | Source and workflow services | Persist one append-only execution outcome and its artifacts transactionally; retain compatibility projections. |
 | UI | Label and inspect evidence layers; export safe evidence packages through service operations. |
 | Benchmarking | Add a private manifest and repeatable evaluator using the literal-transcription methodology. |
 | Tests and documentation | Add compatibility, capture, security, integrity, export, and benchmark-scoring coverage; correct overstated V4 evidence language. |
 ## Proposed Data Design
 Exact names should be confirmed against existing conventions before migration code is written. The design should provide the following logical records.
 ### 1. Execution Evidence
 Extend `JobSource` or associate it one-to-one with a new execution-evidence record containing:
 - Request manifest JSON and manifest schema version.
 - Transport status, body bytes or exact decoded body plus encoding/content type, and safe headers.
 - Parsed SDK snapshot retained separately from transport content.
 - Application, adapter, SDK, and runtime version metadata.
 - Start, finish, and duration values.
 - Router/provider request and generation identifiers when available.
 - Failure phase and whether an HTTP response was received.
 The implementation should evaluate a companion table rather than continuing to widen `JobSource`. A companion record better isolates large/optional evidence and permits clear one-to-one compatibility semantics.
 ### 2. Generic Processing Artifact
 Add a one-to-many artifact model associated with a source and, when applicable, a producing execution:
 - Stable artifact UUID.
 - `source_id` and optional execution/`job_source_id`.
 - Semantic artifact type.
 - Media/serialization format.
 - Schema name and version.
 - Producer and producer version.
 - Inline JSON payload or external location.
 - Payload digest and byte size.
 - Coordinate-system metadata when relevant.
 - Creation timestamp.
 Enforce exactly one content location: inline payload or external reference. An external artifact must be written durably and hashed before its database record commits.
 ### 3. Compatibility Projections
 - Keep `JobSource.raw_api_response` unchanged for existing and new compatibility reads until a later deprecation decision.
 - Keep `JobSource.ai_metadata` for indexed/display-ready normalized values.
 - Keep `Source.raw_transcription` as the latest successful machine-output projection while treating per-execution `JobSource.raw_transcription` as history.
 - Document that older rows have an SDK snapshot but no exact transport capture.
 ## Implementation Phases
 ### 1. Correct Terminology and Define Typed Contracts
 - Add typed domain models for request manifests, software context, transport metadata, failure phase, and artifact descriptors.
 - Version every persisted JSON contract from its first release.
 - Define the safe response-header allowlist. Begin with correlation, content type/encoding, date, retry/rate-limit, and router-specific generation identifiers only when documented and non-secret.
 - Define size limits and external-storage thresholds for exact bodies and artifacts.
 - Correct `docs/ver4/schema_v4.md` under “Page-Level Execution and AI Outputs” so the existing column is described as an SDK-serialized OpenRouter response snapshot, not a complete provider envelope, exact HTTP body, or native upstream-provider response. Apply the same terminology to architecture and UI schema references.
 - Add serialization and secret-rejection unit tests before provider changes.
 ### 2. Add Additive Persistence and Upgrade Behavior
 - Add the selected execution-evidence and artifact models.
 - Add foreign keys, uniqueness constraints, and indexes for source/execution lookup.
 - Implement idempotent upgrades following the repository's existing schema-upgrade policy.
 - Do not populate exact response fields for historical rows.
 - Do not write a capture-time classification onto historical rows during migration. Compatibility reads may describe a populated legacy `raw_api_response` as an SDK snapshot, but exports must identify that description as a later compatibility interpretation rather than execution-time metadata.
 - Verify JSON portability and large-payload behavior for SQLite and PostgreSQL.
 - Add upgrade tests starting from a representative pre-V4.2 schema.
 ### 3. Build Secret-Safe Request Manifests
 - Build the manifest from the concrete outgoing request body immediately before transport, not from a narrower typed projection that may discard unrecognized request fields.
 - Replace each image payload in that concrete representation with a source reference containing source UUID, digest, byte size, media type, dimensions, and transformation identity.
 - Store exact prompt content and preserve omitted-versus-explicit parameter state.
 - Include requested model, routing preferences, response-format requirements, and timeout/retry policy.
 - Record application version/commit when available, adapter contract version, SDK package/version, and request-manifest schema version.
 - Hash the canonical manifest representation for integrity checks.
 - Test that credentials and embedded image data cannot enter the persisted manifest.
 - Test that every field actually sent to the provider, including routing and future provider options, is represented or explicitly excluded by the manifest transform.
 ### 4. Capture OpenRouter Transport Evidence
 - Evaluate the installed OpenRouter SDK hooks/client injection first.
 - If hooks cannot expose an exact stable response before typed parsing, implement the non-streaming OpenRouter call through the existing async HTTP client boundary while retaining typed validation in the adapter.
 - Read the response body once, preserve it exactly, then parse and normalize it.
 - Store status, content type/encoding, allowlisted headers, request/generation ID, and timing.
 - Maintain current authentication, referer/title headers, timeout behavior, and error classification.
 - Explicitly document that the captured body is the OpenRouter-normalized transport response, not Gemini/Anthropic/OpenAI native upstream JSON.
 - Add fixture-based tests proving unknown response fields survive transport capture even if a typed parser ignores them.
 ### 5. Preserve Failure Evidence
 - Return or raise a typed provider failure that carries safe evidence separately from its user-facing error.
 - Persist non-success status/body/allowlisted headers before marking an execution failed.
 - Represent DNS/connect/TLS/local timeout failures as no-response outcomes with a failure phase and safe diagnostic category.
 - Preserve response-validation failures with both the exact body and validation details.
 - Keep transcription-quality rejection distinct from provider failure because a valid provider response was received.
 - Ensure error strings and logs do not contain authorization data or embedded image payloads.
 - Add tests for 4xx, 5xx, malformed JSON, schema mismatch, timeout, connection failure, and quality rejection.
 ### 6. Make Execution History Reliably Append-Only
 - Confirm retry behavior creates a distinct execution attempt rather than reusing and overwriting a completed evidence record.
 - Separate queue linkage from execution-attempt identity; the current update-in-place behavior cannot serve as append-only execution history.
 - Assign each attempt a deterministic, monotonically increasing attempt number scoped to its Job and Source, enforced by a database uniqueness constraint.
 - Update the latest-transcription projection only after a successful attempt.
 - Never update prior response bodies, manifests, timings, or artifacts during a retry.
 - Select the latest attempt and latest successful attempt by the persisted attempt number with a stable identifier as a defensive secondary key, never by timestamp alone.
 - Add service/workflow tests covering retries, partial success, interrupted jobs, and historical projection behavior.
 ### 7. Add Generic Artifact Persistence
 - Implement service operations to create, read, list, verify, export, and, only under explicit retention policy, delete artifacts.
 - Validate semantic type, schema/version, digest, media type, and coordinate metadata.
 - Support JSON artifacts inline initially when within the agreed size threshold.
 - Support external artifacts through a constrained application-data root with atomic write, digest verification, and explicit missing-file errors.
 - Add a provider-neutral example fixture representing OCR words/lines with polygons and confidence values.
 - Do not integrate a live OCR vendor in this phase.
 ### 8. Add Evidence Inspection and Export
 - Rename the current Source Detail label to identify historical values as an OpenRouter SDK Response Snapshot.
 - Add separate sections for Request Manifest, Transport Response, Normalized Metadata, Software Context, and Derived Artifacts.
 - Show an explicit “not captured for this historical execution” state instead of an empty object.
 - Keep large bodies collapsed by default and avoid rendering embedded source data.
 - Add a service-owned export that packages a versioned manifest, evidence JSON/body files, artifact content or references, and digest inventory.
 - Exclude secrets and machine-local paths that are not required to interpret the evidence.
 - Add UI and export tests for new, historical, failed, and large-evidence records.
 ### 9. Establish the Private Benchmark
 - Select a small initial corpus, then expand only when it exposes meaningful differences.
 - Stratify examples by printed/typed text, handwriting style, degradation, layout complexity, language, and editorial anomaly.
 - Reference existing Source UUIDs and digests in a private manifest; do not copy family documents into public test fixtures.
 - Create manually reviewed reference transcriptions following the invariant methodology.
 - Implement or adopt existing project-compatible CER/WER calculations without changing dependencies unless justified.
 - Score omissions, inventions, silent modernization, uncertainty markup, and layout fidelity separately from CER/WER.
 - Record cost and latency from preserved execution evidence.
 - Run the current `google/gemini-2.5-flash` configuration as the baseline before testing alternatives.
 - Treat results as model-version/route/corpus specific and preserve each comparison run.
 ### 10. Verify, Migrate, and Align Documentation
 - Run the smallest focused model, provider, service, workflow, UI, upgrade, and export test groups first.
 - Run broader regression tests only after focused validation passes.
 - Execute all destructive tests through `tools/run_destructive_tests.py`.
 - Verify backup creation and required restoration behavior before any test touching real application data.
 - Confirm existing Source Detail records remain readable after upgrade.
 - Update V4 architecture, schema, requirements, and UI schema mappings to point to V4.2 semantics.
 - Record any deliberate deviation from this plan in the V4.2 scope before release.
 ## Recommended Delivery Order
 1. Typed/versioned evidence contracts and terminology.
 2. Additive execution-evidence persistence.
 3. Secret-safe request manifests.
 4. Exact OpenRouter transport capture.
 5. Failure evidence and append-only retry semantics.
 6. Generic artifact persistence.
 7. Inspection and export.
 8. Private benchmark tooling and baseline run.
 9. Migration, regression verification, and documentation alignment.
 ## Key Implementation Decisions to Resolve
 1. Whether execution evidence is a one-to-one companion to `JobSource` or part of a new execution-attempt model required for append-only retries.
 2. Whether exact response bodies remain database values at expected sizes or move to hashed external files above a threshold.
 3. The canonical JSON algorithm used to hash request manifests.
 4. The safe-header allowlist supported by OpenRouter and future adapters.
 5. The application version identity available in local, packaged, and uncommitted development builds.
 6. The initial inline/external artifact size threshold and application-data root.
 7. Whether evidence exports include original source binaries by default, optionally, or only by reference.
 8. The minimum private benchmark corpus size and review process before model comparisons influence defaults.
 These decisions must be settled before their corresponding implementation phase; they do not weaken the invariant or expand V4.2 into live OCR integration.
 ## Resolved Implementation Decisions
 1. `JobSource` remains queue linkage and a compatibility projection; immutable retries use a one-to-many
   `ExecutionAttempt` model with a unique `(job_id, source_id, attempt_number)` constraint.
 2. Exact OpenRouter response bytes remain database values for V4.2. Generic artifacts use inline canonical JSON up
   to 1 MiB by default and constrained, atomically written external files above that threshold.
 3. Request manifests use `transcription-canonical-json-v1`: UTF-8 JSON with sorted keys, compact separators,
   preserved Unicode, and non-finite numbers rejected.
 4. Safe response headers are explicitly allowlisted in the evidence contract; all others are discarded before
   persistence.
 5. Software identity records the package version, optional `TRANSCRIPTION_COMMIT`, adapter contract version,
   OpenRouter SDK version, and Python version.
 6. The artifact root defaults to `data/artifacts` and stores source-scoped relative references.
 7. Evidence exports include source identity and digest by reference, not original source binaries.
 8. The benchmark manifest is private and digest-referenced. Corpus size remains archive-dependent, but every run
   uses preserved execution-attempt identity and the fixed literal scoring contract.
 ## Done When
 - Every V4.2 acceptance criterion is satisfied by focused tests or an explicit demonstration.
 - Existing SDK snapshots retain their content and are labeled accurately.
 - New successful and failed calls preserve secret-safe provider-boundary evidence.
 - Unknown transport fields survive even when the typed SDK/parser does not recognize them.
 - Retries cannot overwrite prior execution evidence.
 - A generic versioned artifact can represent OCR geometry and pass integrity verification.
 - Evidence can be safely inspected and exported with schema identities and digests.
 - The current model has a reproducible private benchmark baseline.
 - No credential or embedded source payload appears in persisted manifests, safe headers, logs, or exports.
 - Existing V4.1 behavior remains compatible.
 ## Out of Scope
 - Live OCR/document-AI provider integration.
 - Automatic model switching.
 - Archive-wide reprocessing.
 - Native upstream-provider response capture through OpenRouter when OpenRouter does not expose it.
 - Guarantees of deterministic hosted-model output.
 ## Related Local References
 - [V4.2 Scope Boundary](scope_boundary_v4_2.md)
 - [Digital Evidence and AI Processing Provenance](../invariant/ai_evidence_and_provenance.md)
 - [V4 Architecture](../ver4/architecture_v4.md)
 - [V4 Schema](../ver4/schema_v4.md)
 - [V4 Requirements](../ver4/requirements_v4.md)
 - [Draft V4.3 Implementation Plan](../ver4.3/implementation_plan_v4_3.md)
@@ -0,0 +1,163 @@
 # V4.2 Scope Boundary
 This document defines the boundary for the digital-evidence and AI-provenance revision that follows V4.1 and precedes the planned V4.3 settings work. V4 remains the architecture baseline; V4.2 makes the existing evidence claims precise and adds a provider-neutral foundation for future processing artifacts.
 ## Purpose
 - Align the application with the [Digital Evidence and AI Processing Provenance invariant](../invariant/ai_evidence_and_provenance.md).
 - Preserve provider-boundary evidence before SDK parsing can remove unknown fields.
 - Make successful and failed processing attempts inspectable without storing secrets.
 - Support future OCR and layout outputs without coupling the database to one vendor.
 - Establish a repeatable method for comparing transcription models against this archive.
 ## In Scope
 ### 1. Evidence Terminology and Existing-Data Compatibility
 - Define transport response, router-normalized response, SDK response, normalized metadata, and derived artifact consistently in code, schema documentation, and UI labels.
 - Treat existing `JobSource.raw_api_response` values as historical SDK response snapshots.
 - Preserve every existing `Job`, `Source`, and `JobSource` row.
 - Use additive migrations and compatibility reads; do not reinterpret previously stored values as exact transport captures.
 - Correct the “Page-Level Execution and AI Outputs” rule in `docs/ver4/schema_v4.md` that currently describes `JOB_SOURCE` as storing a complete provider response envelope. The corrected rule must identify `raw_api_response` as an SDK-serialized OpenRouter response snapshot and state that it is neither the exact HTTP body nor the native upstream-provider response.
 ### 2. Secret-Safe Request Manifests
 - Persist the effective request specification for each page execution without storing credentials or duplicate base64 media.
 - Include requested provider/model, routing constraints, prompt content and hash, explicitly supplied parameters, source digest, media type, dimensions when known, and page identity.
 - Distinguish an omitted optional parameter from an explicitly supplied null or value.
 - Record application, provider-adapter, Python client, and relevant schema versions.
 - Use source or derivative references in place of embedded media bytes.
 ### 3. Provider-Boundary Response Capture
 - Capture the exact HTTP response body before OpenRouter SDK parsing for non-streaming transcription calls.
 - Store HTTP status and an explicit allowlist of safe response headers.
 - Store router request/generation identifiers and resolved model/provider-routing metadata when exposed.
 - Preserve the current parsed SDK snapshot and normalized metadata where useful.
 - Keep exact body, parsed representation, and normalized fields distinguishable.
 ### 4. Failure Evidence and Timing
 - Create or update a page execution record for every attempted provider call.
 - Persist safe response evidence for non-success HTTP responses.
 - Distinguish HTTP response failures, connection failures, local timeouts, response-validation failures, and transcription-quality failures.
 - Store execution start/end times or duration using a clearly defined clock policy.
 - Do not collapse a provider error body into only a generic user-facing message.
 ### 5. Generic Processing Artifacts
 - Add a provider-neutral representation for versioned derived artifacts.
 - Support inline JSON and externally stored payloads with a digest and stable reference.
 - Record artifact type, format, schema/version, producer/version, source, producing execution, and creation time.
 - Define coordinate-system metadata sufficient for word, line, block, or page geometry.
 - Permit future OCR/layout/confidence results without implementing a vendor-specific table for each provider.
 ### 6. Evidence Inspection and Export
 - Expand Source Detail and/or Job Detail to identify the evidence layer being displayed.
 - Provide readable JSON inspection for request manifests, transport metadata, parsed responses, normalized metadata, and derived artifacts.
 - Provide a safe export containing evidence content or references, relationships, schema versions, and digests.
 - Clearly label evidence that was not captured for historical records.
 - Do not display or export credentials, unrestricted headers, or embedded base64 source media.
 ### 7. Representative-Corpus Benchmark Protocol
 - Define a private benchmark manifest referencing source digests rather than duplicating archival media.
 - Include representative printed, typed, handwritten, degraded, tabular, and spatially complex pages.
 - Pair each benchmark item with a manually reviewed literal transcription.
 - Score character error rate, word error rate, omissions, inventions, silent normalization, uncertainty handling, layout fidelity, cost, and latency.
 - Preserve the complete execution provenance for every benchmark run.
 - Keep the current model as a baseline; do not change the application default solely from vendor benchmarks.
 ### 8. Migration, Integrity, and Verification
 - Provide non-destructive upgrade behavior for supported SQLite and PostgreSQL deployments.
 - Backfill only facts that can be derived reliably from existing records.
 - Mark unavailable historical evidence as unavailable rather than fabricating it.
 - Add digest, serialization, header-allowlist, failure-path, compatibility, artifact, export, and UI inspection tests.
 - Run destructive tests only through the repository's required backup-and-restore wrapper.
 ## Out of Scope
 - Selecting or declaring a permanent best transcription model.
 - Changing the default transcription model without benchmark evidence and a separate decision.
 - Integrating Azure Document Intelligence, Google Document AI, Transkribus, Mistral OCR, or another OCR provider in V4.2.
 - Generating bounding boxes retroactively for existing transcriptions.
 - Bulk reprocessing the archive.
 - Packet capture, TLS evidence, full unrestricted request/response headers, or credential retention.
 - Storing duplicate base64 source images in request manifests.
 - Guaranteeing byte-identical reproduction from nondeterministic or updated hosted models.
 - Automatic entity extraction, biography generation, or genealogical inference.
 - Replacing the relational database with an event store or content-addressed object store.
 - Destructive renaming or removal of `raw_api_response`.
 ## Locked Design Decisions
 ### A. The Original Source Is Primary Evidence
 - Original uploaded bytes and their digest remain authoritative.
 - Processing derivatives and outputs are independently identified derived evidence.
 - Future OCR/layout work reuses the original or a documented derivative.
 ### B. Evidence Is Layered
 - Exact transport evidence, SDK-parsed objects, normalized metadata, and transcription text serve different purposes.
 - One representation must not silently stand in for another.
 - UI and export labels name the stored evidence layer.
 ### C. History Is Append-Only
 - A retry or reprocessing attempt creates new execution evidence.
 - Convenience caches may change, but historical execution output does not.
 - Human revisions remain separate from machine output.
 ### D. Capture Is Secret-Safe by Construction
 - Safe headers are allowlisted.
 - Authorization, cookies, API keys, and unrestricted headers are never persisted.
 - Request manifests reference source digests instead of embedding source bytes.
 ### E. Derived Artifacts Are Generic and Versioned
 - Artifact storage is not limited to bounding boxes.
 - Coordinate metadata declares units, origin, dimensions, and transformations.
 - Provider-specific payloads may be retained without making provider-specific fields the durable application contract.
 ### F. Existing Evidence Keeps Its Original Meaning
 - Existing `raw_api_response` data remains an SDK response snapshot.
 - A migration may label or classify it but may not claim that missing transport data was captured.
 - Historical nulls and absent fields remain distinguishable from new explicitly captured values.
 ## Data and Compatibility Policy
 - All schema changes are additive in V4.2.
 - Existing source files, hashes, transcriptions, revisions, prompts, jobs, and relationships remain valid.
 - Compatibility reads continue to display historical SDK snapshots.
 - Large derived artifacts may be stored outside the database when the database retains a stable reference, digest, media type, and schema identity.
 - JSON evidence must remain portable across SQLite and PostgreSQL.
 - Exports use explicit schema versions so later releases can interpret older packages.
 ## Acceptance Criteria
 1. A new execution can be traced from its source digest through its frozen request manifest, transport response, parsed/normalized data, and derived outputs.
 2. Exact response content is captured before SDK parsing and is clearly distinguished from the existing SDK snapshot.
 3. Failed HTTP calls retain safe provider evidence; calls with no response record that fact explicitly.
 4. Omitted parameters remain distinguishable from explicit values.
 5. No persisted request, header set, UI display, log, or export contains API credentials.
 6. Retrying or reprocessing does not overwrite prior execution evidence.
 7. Historical records remain readable and are not mislabeled as exact transport captures.
 8. A versioned generic artifact can represent OCR/layout JSON and its coordinate system without a provider-specific schema change.
 9. Evidence exports include relationships, schema identities, and digests sufficient for independent integrity checks.
 10. The benchmark protocol can compare the current baseline with another model on the same private corpus and scoring rules.
 11. Additive migrations and focused tests work across the supported persistence model.
 12. All destructive-test runs comply with the backup-and-restore protocol.
 ## Related Local References
 - [V4.2 Implementation Plan](implementation_plan_v4_2.md)
 - [Digital Evidence and AI Processing Provenance](../invariant/ai_evidence_and_provenance.md)
 - [V4 Architecture](../ver4/architecture_v4.md)
 - [V4 Schema](../ver4/schema_v4.md)
 - [V4 Requirements](../ver4/requirements_v4.md)
 - [Transcription Methodology](../invariant/transcription_methodology.md)
@@ -0,0 +1,121 @@
 # Implementation Plan (Version 4.3)
 ## Goal
 Deliver constrained, installation-local application settings while preserving the completed V4.2 behavioral baseline and historical provenance.
 ## Planning Constraints
 - V4, V4.1, and V4.2 remain the behavioral baseline.
 - Settings must use explicit domain operations rather than direct database, environment-file, or arbitrary filesystem access from UI pages.
 - Prompt changes must preserve historical Job provenance and use a defined safe-write policy.
 - Source Page Reordering is excluded.
 - Database, integration, and UI tests must use confirmed isolated test data and must never modify `data/transcription.db`.
 - Potentially destructive tests must run only through `tools/run_destructive_tests.py`.
 ## Expected Project Impact
 | Area | Expected impact |
 | --- | --- |
 | Documents service | Expand controlled Document Type maintenance operations. |
 | People service | Expand controlled Person Role maintenance operations. |
 | Prompt adapter/service | Add constrained listing, reading, validation, atomic writing, backup, and explicit recovery of existing prompt artifacts. |
 | UI composition/navigation | Register Settings routes and navigation without moving persistence into UI code. |
 | Tests | Add isolated registry lifecycle, prompt safety, and UI workflow coverage. |
 ## Implementation Phases
 ### 1. Define Service Contracts
 - Define Document Type maintenance commands for create, relabel, activate, deactivate, and delete-if-unreferenced.
 - Define Person Role maintenance commands for create, relabel, activate, deactivate, and delete-if-unreferenced.
 - Define a Prompt Store interface for constrained list, read, write, backup-status, and explicit recovery behavior.
 - Map validation, conflict, not-found, dependency, and filesystem failures to existing `AppError` categories.
 ### 2. Expand Registry Maintenance Services
 - Reuse existing Document and People service ownership.
 - Add explicit write methods rather than passing UI-mutated ORM objects directly where practical.
 - Normalize Document Type labels and reject case-insensitive duplicates deterministically.
 - Keep Person Role stable-code validation and duplicate rejection.
 - Permit deletion only after a service-owned reference check proves the entry is unreferenced.
 - Reject deletion of referenced entries deterministically without partial mutation.
 - Permit label changes whether or not an entry is referenced.
 - Preserve inactive entries for historical reads.
 - Order Document Types alphabetically by normalized label.
 - Order Person Roles deterministically by label and then code without adding a schema field.
 - Add service tests for create, relabel, activation, deactivation, duplicates, immutable codes, ordering, allowed deletion, and blocked referenced deletion.
 ### 3. Add Constrained Prompt Storage
 - Place filesystem access behind a dedicated Prompt Store/service boundary.
 - Resolve all filenames directly beneath the configured prompt root and reject traversal.
 - Permit only existing files with the agreed Markdown extension and reject empty content.
 - Exclude prompt creation and deletion.
 - Write new content to a sibling temporary file, flush and sync it, preserve the active file as the sole previous-version backup, and atomically replace the active file.
 - Expose explicit backup recovery through the same filename validation and safe-write path; never perform automatic rollback.
 - Clean up temporary files after failed writes while preserving the active prompt and any valid backup.
 - Preserve file encoding and provide explicit failures for read-only or unavailable storage.
 - Do not modify any Job row when prompt defaults change.
 - Add unit tests for valid reads/writes, traversal, invalid names, nonexistent-file creation attempts, empty content, atomic replacement failures, single-backup rotation, explicit recovery, filesystem failures, and unchanged Job provenance.
 ### 4. Build the Settings UI
 - Register a Settings landing page and navigation entry.
 - Add separate pages or panels for Document Types, Person Roles, and Prompts.
 - Keep pages responsible for orchestration and notifications only.
 - Use service callbacks for all mutations.
 - Explain inactive historical entries and future-only prompt effects in the UI.
 - Present deletion only for unreferenced registry entries and preserve clear conflict feedback if references appear before submission.
 - Present prompt backup availability and recovery as an explicit operator action.
 - Do not render raw environment values or secrets.
 - Add no settings API routes.
 ### 5. Verification and Rollout
 - Confirm every database, integration, and UI test is configured for an isolated test database before execution.
 - Never run those tests against live data and never modify or replace `data/transcription.db`.
 - Invoke potentially destructive tests only through `tools/run_destructive_tests.py`.
 - Run focused service tests before UI integration tests.
 - Verify inactive registry behavior in both historical display and create/edit selectors.
 - Verify referenced entries can be relabeled or deactivated but not deleted.
 - Verify unreferenced entries can be deleted.
 - Verify prompt changes are picked up by newly created Jobs while historical Jobs retain frozen content/hash.
 - Run the relevant regression suite.
 ## Migration and Compatibility Notes
 - Existing registry records remain valid.
 - Prompt editing changes mutable application files, not database provenance already captured on Jobs.
 - V4.3 must not require users to recreate existing Sources, Documents, People, roles, or types.
 - Person Role ordering requires no schema migration.
 - Registry deletion introduces no cascade behavior; references always block deletion.
 ## Delivery Order
 1. Implement registry maintenance service operations.
 2. Implement the Prompt Store and safety policy.
 3. Build Settings pages.
 4. Run isolated integration and regression verification.
 ## Done Criteria
 - All V4.3 acceptance criteria are testable and satisfied.
 - Settings mutations cross explicit service or adapter boundaries.
 - Document Types use UUID-only identity and unique labels; Person Role codes cannot be accidentally changed.
 - Referenced registry entries can be relabeled or deactivated but cannot be deleted.
 - Unreferenced registry entries can be deleted without cascade behavior.
 - Prompt writes cannot escape the configured directory or rewrite historical provenance.
 - Prompt writes are atomic, retain one backup, and support explicit recovery.
 - No secret or raw environment editor exists.
 - No settings API surface exists.
 - V4.1 and V4.2 workflows remain intact.
 - Verification does not touch live data or `data/transcription.db`.
 ## Related Local References
 - [V4.3 Scope Boundary](scope_boundary_v4_3.md)
 - [V4.2 Implementation Plan](../ver4.2/implementation_plan_v4_2.md)
 - [V4.1 Implementation Plan](../ver4.1/implementation_plan_v4_1.md)
 - [V4 Implementation Plan](../ver4/implementation_plan_v4.md)
 - [V4 Error Handling Policy](../ver4/error_handling_v4.md)
@@ -0,0 +1,154 @@
 # V4.3 Scope Boundary
 This document defines the frozen boundary for the constrained-settings revision that follows the completed V4.2 evidence-and-provenance work. V4, V4.1, and V4.2 remain the behavioral and architecture baseline.
 ## Purpose
 - Provide a constrained Settings area for safe maintenance of selected application-managed configuration.
 - Avoid exposing secrets, restart-sensitive settings, or unrestricted filesystem editing through the UI.
 ## In Scope
 ### 1. Settings Navigation
 - Add a Settings entry to application navigation.
 - Provide separate, clearly described settings areas rather than a raw configuration editor.
 - Restrict V4.3 settings to application-managed values that can be validated and safely changed at runtime.
 ### 2. Document Type Maintenance
 - List active and inactive Document Types.
 - Add new types with a unique user-facing label.
 - Edit labels and active state.
 - Activate or deactivate types without invalidating historical Documents.
 - Allow deletion only when no Document references the type.
 - Allow label changes regardless of whether the type is referenced.
 - Display types alphabetically by label.
 ### 3. Person Role Maintenance
 - List active and inactive Person Roles.
 - Add new roles with a stable unique code and user-facing label.
 - Edit mutable labels.
 - Activate or deactivate roles without invalidating historical links.
 - Do not allow changing a stable code after creation.
 - Order roles deterministically by label and then code; do not add persisted role sort order.
 - Allow deletion only when no document-person link references the role.
 - Allow label changes regardless of whether the role is referenced.
 ### 4. Prompt Maintenance
 - List prompt markdown files from the configured prompt directory.
 - View a prompt with a concise explanation of its purpose and use.
 - Edit an existing prompt as plain markdown text.
 - Validate the filename boundary and reject empty prompt content.
 - Save changes explicitly and report filesystem failures.
 - Preserve submission-time prompt text and hash already frozen on existing Jobs.
 - Edit existing prompt files only; prompt creation and deletion are excluded.
 - Save through a sibling temporary file, flush and sync file content, retain one previous-version backup, and atomically replace the active file.
 - Provide an explicit recovery operation that restores the retained backup through the same safe-write path; do not silently roll back a failed or unwanted edit.
 ### 5. Deployment Boundary
 - Settings changes apply only to the current installation.
 - V4.3 adds no settings API endpoints.
 - Service contracts must remain independent of the UI so a separately authorized API can be considered later.
 ## Out of Scope
 - Viewing or editing raw `.env` files.
 - Displaying or changing provider API keys and other secrets.
 - Editing host, port, database connection, upload paths, or other restart-sensitive runtime settings.
 - Arbitrary file browsing or arbitrary prompt paths.
 - Runtime theme/CSS editing.
 - Installing themes or plugins.
 - Source page renumbering or reordering.
 - Source movement between Documents.
 - Automatic ordering based on filenames, OCR, or image content.
 - Prompt creation, deletion, and multi-version history.
 - Persisted sort-order maintenance for Person Roles.
 - Settings read or write API endpoints.
 - FamilySearch API synchronization.
 - A generic external-reference registry.
 - Ancestry references and Google Maps links.
 ## Locked Design Decisions
 ### A. Registry Identity and Lifecycle
 - Document Types use UUID identity and case-insensitively unique labels; no separate code is exposed or stored.
 - Person Role codes remain stable identifiers.
 - Labels and active state remain mutable.
 - Historical references remain valid when a registry entry is inactive.
 - Labels may be updated for referenced and unreferenced entries.
 - Unreferenced entries may be deleted; referenced entries may only be deactivated.
 ### B. No Raw Environment Editor
 - `.env` may contain secrets and values that are not safely reloadable.
 - V4.3 exposes only purpose-built forms backed by explicit validation and service methods.
 ### C. Prompt Editing Is Constrained
 - Prompt maintenance is limited to direct children of the configured prompt directory.
 - Existing Job provenance is never rewritten when a prompt file changes.
 - The UI must distinguish editing the default for future submissions from inspecting historical Job prompts.
 ### D. Prompt Writes Are Atomic and Recoverable
 - Writes use a sibling temporary file and atomic replacement so readers observe either the old or new complete prompt.
 - The immediately previous prompt version is retained as the sole backup.
 - Recovery is an explicit operator action and uses the same validated safe-write path.
 - Prompt creation and deletion are not available in V4.3.
 ### E. Person Role Ordering Is Deterministic, Not Persisted
 - Person Roles are ordered by label and then stable code.
 - V4.3 does not add a `sort_order` field to Person Roles.
 - Document Types use alphabetical label ordering and have no persisted sort order.
 ### F. Settings Are Installation-Local
 - V4.3 provides Settings through the local application UI and domain services only.
 - No settings API surface is introduced.
 ## Data and Compatibility Policy
 - V4.3 does not rewrite existing Documents, document-person links, Jobs, Sources, execution evidence, or prompt provenance.
 - Deactivation preserves referenced registry entries for historical display while excluding them from default create selectors.
 - Deletion checks are performed at the service boundary and must fail deterministically when references exist.
 - Prompt files are constrained to existing Markdown files that are direct children of the configured prompt root.
 - Settings UI code performs no direct database, environment-file, or arbitrary filesystem mutations.
 - Source page numbering and ordering behavior is unchanged.
 ## Acceptance Criteria
 1. Document Type UUID identity and Person Role stable codes preserve historical references.
 2. Inactive registry entries remain visible on historical records but are excluded from default create selectors.
 3. Labels can be changed for referenced or unreferenced registry entries.
 4. An unreferenced Document Type or Person Role can be deleted, while deletion of a referenced entry fails without partial mutation.
 5. Document Types use alphabetical label ordering; Person Roles use deterministic label/code ordering.
 6. Prompt edits are restricted to existing Markdown files directly beneath the configured prompt directory.
 7. Prompt saves use atomic replacement, retain exactly one previous-version backup, and support explicit recovery.
 8. A prompt edit affects future Jobs only and leaves stored Job provenance unchanged.
 9. No Settings page exposes secrets, unrestricted filesystem access, or a settings API.
 10. Focused service and UI tests pass without regressing V4.1 or V4.2 workflows.
 11. Database, integration, and UI tests use confirmed isolated test data and never modify `data/transcription.db`; potentially destructive tests run only through `tools/run_destructive_tests.py`.
 ## Scope Freeze Gate
 V4.3 is sufficiently frozen to begin implementation:
 - V4.2 is the completed behavioral baseline.
 - Registry lifecycle and ordering behavior are resolved.
 - Prompt lifecycle, atomic-write, backup, and recovery behavior are resolved.
 - The installation-local deployment boundary is resolved.
 - The implementation plan is a committed delivery plan.
 ## Related Local References
 - [V4.3 Implementation Plan](implementation_plan_v4_3.md)
 - [V4.2 Scope Boundary](../ver4.2/scope_boundary_v4_2.md)
 - [V4.1 Scope Boundary](../ver4.1/scope_boundary_v4_1.md)
 - [V4 Architecture](../ver4/architecture_v4.md)
 - [V4 Schema](../ver4/schema_v4.md)
@@ -0,0 +1,188 @@
 # Implementation Plan (Version 4.4)
 ## Goal
 Deliver hidden semantic identity for built-in registries, a single atomic Linked People workflow, and safe browser-native printing of archival Documents and their current transcriptions.
 ## Planning Constraints
 - V4.3 is the completed implementation baseline.
 - V4.4 may replace V4/V4.3 registry and document-person contracts only as specified by the V4.4 scope.
 - Semantic keys are internal and immutable; UI and public API contracts use UUIDs and labels.
 - Document and link edits must not partially commit.
 - Print output must not execute stored text or expose machine-local source paths.
 - Source page reordering remains excluded.
 - Database, integration, and UI tests must use confirmed isolated data and never modify `data/transcription.db`.
 - Potentially destructive tests must run only through `tools/run_destructive_tests.py`.
 ## Expected Project Impact
 | Area | Expected impact |
 | --- | --- |
 | Models and schema bootstrap | Add nullable unique semantic keys, simplify document-person identity, and seed frozen built-ins. |
 | Document service | Maintain built-in Document Types, usage summaries, UUID assignment, and atomic Document/link writes. |
 | People service | Maintain built-in Person Roles, link summaries, UUID-only role assignment, and one-person-per-document enforcement. |
 | V4 document API | Remove role-code selectors and compatibility role fields; enforce UUID-only relationship writes. |
 | Settings UI | Use matching table workflows for Document Types and Person Roles. |
 | Document Create/Edit | Replace role-specific multiselects with one staged Linked People table and inline editor. |
 | Document Detail/printing | Add format selection, print preview, safe Source media rendering, print CSS, and job metadata. |
 | Tests and documentation | Replace superseded cardinality/identity assertions and add isolated registry, editor, transaction, and print coverage. |
 ## Implementation Phases
 ### 1. Align Durable Registry Contracts
 - Add nullable, unique `semantic_key` fields to `DocumentType` and `PersonRole`.
 - Keep UUIDs as primary and foreign-key identity.
 - Add normalized-label storage and uniqueness to Person Roles using the same trim and case-normalization policy as Document Types.
 - Remove the user-created Person Role code contract.
 - Define built-in detection as `semantic_key is not null`.
 - Centralize the frozen built-in definitions in one domain-owned location.
 - Seed six Document Types and three Person Roles idempotently.
 - Ensure label edits never change semantic keys.
 - Reject deletion of every built-in before checking references.
 - Continue blocking deletion of referenced custom entries.
 - Return deterministic validation, conflict, dependency, and not-found errors through existing error categories.
 ### 2. Establish the Clean Schema
 - Remove legacy `DocumentPerson.role` compatibility storage and the fixed `DocumentPersonRole` enum.
 - Make `DocumentPerson.role_id` required.
 - Replace role-specific uniqueness with a unique `(document_id, person_id)` constraint.
 - Remove obsolete Document Type and Person Role migration paths that exist only for disposable development data.
 - Keep fresh schema creation and built-in seeding portable across SQLite and PostgreSQL.
 - Make configured development-database recreation a separate operator-confirmed step that displays the resolved target path rather than assuming `app.db` or `data/transcription.db`.
 - Never invoke recreation from application startup or test setup.
 - Add isolated schema tests for fresh creation, seed idempotence, semantic-key uniqueness, normalized-label uniqueness, required roles, and link uniqueness.
 ### 3. Refine Registry Services and API Contracts
 - Add summary queries for Document counts and Person Role link counts without per-row queries.
 - Order both registries by normalized label with UUID as a deterministic tie-breaker.
 - Expose built-in status as a derived read value where the Settings UI needs it.
 - Keep semantic-key lookup behind service methods for application-owned behavior such as resolving authors.
 - Ensure create operations always produce custom entries with null semantic keys.
 - Ensure update operations accept only label and active state.
 - Remove `role_code` request alternatives and compatibility role responses from the V4 document API.
 - Require `role_id` for document-person creation and updates.
 - Add service and API tests for hidden semantic identity, relabeling, activation, built-in protection, custom deletion, counts, ordering, UUID-only writes, and conflicts.
 ### 4. Build Matching Settings Tables
 - Retain the existing Document Types table workflow and add the Built-in column.
 - Replace the current per-row Person Role controls with the same selection-based table pattern.
 - Render the agreed columns and usage counts.
 - Keep labels as the only registry text shown in selectors.
 - Add creates custom entries only.
 - Edit dialogs expose label and active state only.
 - Delete reports protected-built-in and referenced-custom conflicts clearly.
 - Avoid direct persistence queries from the Settings page.
 - Add component-level UI assertions for columns, actions, label-only selectors, and immutable built-in presentation.
 ### 5. Add a Staged Linked People Editor
 - Introduce a small typed UI-state model for staged `(person_id, role_id)` rows rather than storing raw widget values.
 - Share the editor component between Create Document and Edit Document.
 - Render a multi-selection table with Person and Role labels.
 - Add an inline editor whose mode is explicitly Add or Edit.
 - Disable already-linked People when adding; retain the edited Person as an option during Edit.
 - Require exactly one row for Edit and allow one or more rows for Delete.
 - Save and Delete mutate only staged UI state.
 - Cancel discards only the active inline edit.
 - Preserve inactive-role historical rows in Edit while restricting new assignments and changes to active roles.
 - Preserve `person_id` preselection by staging that Person with the active built-in `author` role, with warning behavior for invalid or unavailable selections.
 - Preserve the `return_to=jobs_new` success path.
 - Keep navigation to Person creation separate; V4.4 does not add an embedded Person editor.
 - Add UI tests for staging, duplicate prevention, selection rules, inactive roles, cancel behavior, and both Document forms.
 ### 6. Persist Document and Links Atomically
 - Add service commands for Create Document with complete links and Update Document with complete links.
 - Validate Document Type, every Person, every Person Role, active assignment rules, and duplicate People before mutation.
 - Compute deterministic add, update, and remove deltas for Edit.
 - Apply Document and link mutations in one database transaction and commit once.
 - Roll back the complete operation on any validation, conflict, or persistence failure.
 - Return the persisted Document detail required by the UI after success.
 - Reuse these commands from UI orchestration rather than sequencing independent service commits.
 - Add failure-injection tests proving no partial Document or link mutation survives.
 ### 7. Define a Print Projection
 - Add a read-only service projection containing:
  - Document title and selected archival metadata.
  - Authors resolved by the `author` semantic key.
  - Notes.
  - Ordered Sources with application media URLs and current transcription text.
  - Ordered Job metadata.
 - Load the projection with bounded queries and deterministic ordering.
 - Use non-null `revised_text`, including an intentionally empty revision; otherwise fall back to `raw_transcription`.
 - Map empty or whitespace-only current text to the explicit unavailable state without falling back past an intentional revision.
 - Represent unavailable text and optional metadata explicitly.
 - Do not expose semantic keys, direct file paths, full prompts, provider evidence, or raw API responses.
 - Keep the projection independent of NiceGUI rendering so formatting tests can use plain typed values.
 ### 8. Build Print Preview and Styles
 - Add a Print action to Document Detail.
 - Open a dedicated persisted-Document print route with a Facsimile/Text-only format choice.
 - Render the exact content order frozen in the scope.
 - Render stored Notes and transcription as escaped text.
 - For Text-only mode, normalize whitespace by joining single line breaks inside paragraphs while preserving blank-line paragraph boundaries.
 - For Facsimile mode, preserve line breaks and use a two-column Source layout.
 - Start each Facsimile Source on a new printed sheet with CSS page breaks.
 - Allow long transcription content to continue rather than clipping it.
 - Fetch images through an application-controlled Source media route.
 - Add print-only CSS that hides navigation, controls, and non-document chrome.
 - Invoke the browser print dialog only from an explicit user action.
 - Add rendering tests for both modes, missing data, long text, special characters, image URLs, and page ordering.
 ### 9. Align Documentation and Verification
 - Update V4 architecture, requirements, schema, and Document UI contracts to reflect:
  - UUID plus hidden semantic-key registries.
  - Built-in protection.
  - One Person per Document.
  - UUID-only role API writes.
  - Atomic Document/link synchronization.
  - Browser-native print projection and formats.
 - Confirm every database, integration, and UI test target is isolated before execution.
 - Run focused registry and service tests first.
 - Run schema tests only through the destructive-test wrapper when they are potentially destructive.
 - Run Linked People UI and print rendering tests against isolated fixtures.
 - Run the broader non-external regression suite after focused coverage passes.
 - Verify that `data/transcription.db` was not changed by test execution.
 ## Delivery Order
 1. Registry and clean-schema contracts.
 2. Registry services, API changes, and Settings tables.
 3. Atomic Document/link service commands.
 4. Shared staged Linked People editor.
 5. Print projection.
 6. Print preview and styles.
 7. Documentation alignment and regression verification.
 ## Done Criteria
 - All V4.4 acceptance criteria are implemented and testable.
 - UI and API contracts use UUID identity and never expose semantic keys.
 - Built-in registries retain meaning after relabeling and cannot be deleted.
 - Custom registries retain reference-aware deletion.
 - Both Document forms use one Linked People table.
 - One Person cannot be linked twice to the same Document.
 - Main Document saves are atomic across fields and relationships.
 - Print preview provides both frozen formats and content sections.
 - Print output uses current human-preferred text, deterministic ordering, escaped content, and application media URLs.
 - Job metadata lists every Job oldest-to-newest and ends with Status.
 - Source page reordering and server-generated PDFs are not introduced.
 - Verification uses isolated data and does not modify `data/transcription.db`.
 ## Related Local References
 - [V4.4 Scope Boundary](scope_boundary_v4_4.md)
 - [V4.3 Scope Boundary](../ver4.3/scope_boundary_v4_3.md)
 - [V4.3 Implementation Plan](../ver4.3/implementation_plan_v4_3.md)
 - [V4 Architecture](../ver4/architecture_v4.md)
 - [V4 Schema](../ver4/schema_v4.md)
 - [V4 Requirements](../ver4/requirements_v4.md)
 - [V4 Error Handling Policy](../ver4/error_handling_v4.md)
@@ -0,0 +1,232 @@
 # V4.4 Scope Boundary
 This document defines the frozen boundary for the semantic-registry, linked-people, and document-printing revision that follows the completed V4.3 Settings work. V4 through V4.3 remain the architecture and behavioral baseline except where this document explicitly replaces a registry or document-person contract.
 ## Purpose
 - Keep registry identifiers stable without exposing duplicate machine codes in Settings tables or selectors.
 - Replace role-specific person selectors with one coherent Linked People editor.
 - Provide an archival print view containing document metadata, source pages, current transcription text, and transcription-job metadata.
 ## In Scope
 ### 1. Semantic Registry Identity
 - `DocumentType` and `PersonRole` use UUIDs as their canonical record and relationship identity.
 - Both registries may carry a nullable, unique, immutable `semantic_key` used only for application-defined built-ins.
 - Semantic keys are internal implementation details. Settings tables, selectors, and public API payloads do not display or accept them.
 - Labels are trimmed, case-insensitively unique, editable, and used for all user-facing display.
 - Active state remains editable. Inactive entries remain valid for historical records and are excluded from default assignment selectors.
 - Built-in status is derived from the presence of a semantic key and is displayed as a read-only Yes/No value.
 - Built-in entries cannot be deleted or converted to custom entries.
 - Custom entries have no semantic key and may be deleted only when unreferenced.
 - Users may create custom entries but cannot create, change, or assign semantic keys through the UI or API.
 ### 2. Built-In Document Types
 - Seed these built-in semantic keys and initial labels:
 | Semantic key | Initial label |
 | --- | --- |
 | `book` | Book |
 | `letter` | Letter |
 | `postcard` | Postcard |
 | `photo` | Photo |
 | `journal` | Journal |
 | `form` | Form |
 - The Document Types Settings table contains Select, Label, Documents, Active, and Built-in columns.
 - Document Types are ordered alphabetically by normalized label.
 - Add, Edit, and Delete actions operate on table selection.
 - Add creates a custom type. Edit changes only label and active state.
 - The Documents count is the number of Documents referencing the type.
 - Document Type selectors display labels only and submit UUIDs.
 ### 3. Built-In Person Roles
 - Seed these built-in semantic keys and initial labels:
 | Semantic key | Initial label |
 | --- | --- |
 | `author` | Author |
 | `recipient` | Recipient |
 | `mentioned` | Mentioned |
 - Application behavior that requires authorship resolves the built-in `author` semantic key rather than matching a mutable label.
 - The Person Roles Settings table contains Select, Label, Links, Active, and Built-in columns.
 - Person Roles are ordered alphabetically by normalized label.
 - Add, Edit, and Delete actions operate on table selection.
 - Add creates a custom role. Edit changes only label and active state.
 - The Links count is the number of document-person relationships referencing the role.
 - Person Role selectors display labels only and submit UUIDs.
 ### 4. Linked People Editor
 - Replace the separate role-specific person selectors on both Create Document and Edit Document with one Linked People table.
 - The table contains Select, Person, and Role columns.
 - Add opens an inline editor beneath the table with Person and Person Role selectors.
 - Edit requires exactly one selected row and loads it into the inline editor.
 - Save stages the inline addition or edit in the table.
 - Cancel exits the inline editor without changing the staged link set.
 - Delete stages removal of one or more selected rows.
 - A Person may be linked to a Document only once, regardless of role.
 - Every link has exactly one Person Role.
 - Already-linked People are unavailable when adding another row.
 - Existing links using inactive roles remain visible and unchanged unless explicitly edited.
 - Only active roles are available for new links or role changes.
 - Create Document preserves the existing `person_id` preselection workflow by staging that Person with the active built-in `author` role. An invalid Person or unavailable Author role produces a warning rather than an invalid link.
 - Create Document preserves the existing `return_to=jobs_new` success path.
 - Linked People changes remain staged until the main Create Document or Save Changes action.
 - The Document and its complete staged link set are persisted atomically. A conflict or validation failure leaves both unchanged.
 - The API and service contracts identify roles by `role_id`; role-code selectors and compatibility role strings are removed.
 - Persistence enforces uniqueness on `(document_id, person_id)`.
 ### 5. Document Print View
 - Add a Print action to Document Detail.
 - The action opens a dedicated print-preview page for the persisted Document.
 - The preview offers two formats:
  - **Facsimile:** source image on the left and current transcription on the right. Original transcription line breaks are preserved, and each Source begins on a new printed sheet.
  - **Text only:** no source images. Single line breaks inside a paragraph are reflowed as spaces, while blank-line paragraph boundaries remain.
 - Both formats use browser printing through a dedicated print stylesheet and the browser print dialog.
 - Server-generated PDF files are not part of V4.4; users may select the browser's Save as PDF destination.
 - Sources are ordered by existing `page_number`, with UUID as a deterministic tie-breaker.
 - The current transcription for each Source is the non-null `revised_text`, including an intentionally empty revision, otherwise the latest successful machine-output projection in `raw_transcription`.
 - Empty or whitespace-only current text displays the explicit unavailable message rather than falling back past an intentional revision.
 - A Source with no current transcription displays an explicit unavailable message.
 - Transcription and Notes content is treated as text and escaped; model output is not executed as arbitrary HTML.
 - Facsimile images use an application-controlled Source media route. Generated markup does not expose direct machine-local file paths.
 ### 6. Printed Content Contract
 The print view contains, in this order:
 1. Document title using the Document name.
 2. Archival Metadata table:
   - Author, containing People linked through the built-in `author` role.
   - Date.
   - Location Created.
   - Archival Identifier.
 3. Notes.
 4. Document section containing one numbered section per Source.
 5. Transcription Job Metadata table.
 Empty metadata values remain visible as `Not set`. Empty Notes display `No notes recorded`.
 The job metadata table:
 - Lists field names in the first column and adds one column for every Job associated with the Document.
 - Orders Job columns from oldest to newest by creation date, then UUID.
 - Includes every Job status: `queued`, `processing`, `transcribed`, `completed`, `partial_success`, and `failed`.
 - Contains these rows in order:
  - Job ID.
  - Date, using the Job creation/submission timestamp with timezone.
  - Provider.
  - Model.
  - Prompt, using the frozen prompt filename/name rather than full prompt content.
  - Retry Count.
  - Status as the final row.
 - Displays `Not set` for unavailable optional metadata.
 ### 7. Clean Development Schema
 - V4.4 does not require preservation or migration of rows in the operator-configured development database.
 - Implementation may recreate the configured development database, including `data/transcription.db` when it is the explicitly selected target, only through a separate operator-confirmed action that identifies the resolved path. Startup and test execution never delete it automatically.
 - Fresh schema creation seeds the agreed built-in Document Types and Person Roles idempotently.
 - No test may use, modify, replace, or restore `data/transcription.db`.
 - Database, integration, and UI tests use confirmed isolated databases.
 - Potentially destructive tests run only through `tools/run_destructive_tests.py`.
 ## Out of Scope
 - User creation, editing, deletion, or direct display of semantic keys.
 - Treating custom registry entries as built-ins.
 - Additional built-in Document Types or Person Roles beyond the frozen lists.
 - Assigning more than one role to the same Person on the same Document.
 - Preserving multiple historical links that violate the new one-person-per-document constraint.
 - Source page renumbering or reordering.
 - Printing unsaved Create/Edit Document state.
 - Print actions on Job Detail or other pages.
 - Batch printing multiple Documents.
 - Server-side PDF generation or PDF file storage.
 - Markdown, DOCX, or evidence-package export through the print feature.
 - User-editable print templates, fonts, margins, headers, or footers.
 - Full frozen prompt content, prompt hashes, transport evidence, API responses, or execution-attempt details in the print footer.
 - Rendering transcription text as unrestricted Markdown or HTML.
 - Pixel-identical pagination across browsers and printer drivers.
 ## Locked Design Decisions
 ### A. UUID Identifies the Row; Semantic Key Identifies Built-In Meaning
 - UUIDs remain the only relationship and API identity.
 - A hidden semantic key permits reliable built-in behavior after a label is renamed.
 - Mutable labels are never used to infer built-in meaning.
 ### B. Built-Ins Are Protected but Mutable in Presentation
 - Built-in labels and active state may change.
 - Built-in semantic identity cannot change, and built-ins cannot be deleted.
 - Custom entries remain reference-aware and deletable when unreferenced.
 ### C. Linked People Is a Single-Role Relationship
 - One `(document_id, person_id)` row represents the complete relationship.
 - Changing a role updates that row rather than adding another relationship.
 - The main Document save owns one atomic Document-and-links transaction.
 ### D. Printing Uses the Current Human-Preferred Text
 - Human-revised text takes precedence over the latest successful machine-output projection.
 - Job metadata provides processing context but does not claim that a later human revision is raw output from a listed Job.
 ### E. Printing Is Browser-Native
 - A print-specific HTML view and CSS support physical printing and browser Save as PDF.
 - Source media is served through application-controlled routes, and all textual content is escaped.
 ### F. Source Order Is Read-Only in V4.4
 - Print order follows existing page numbers.
 - Source page reordering remains explicitly excluded.
 ## Acceptance Criteria
 1. Registry selectors and Settings forms never display a machine code or semantic key.
 2. Document Types and Person Roles use UUID relationship identity and case-insensitively unique labels.
 3. The six Document Type and three Person Role built-ins are seeded with immutable internal semantic keys.
 4. Built-ins may be relabeled or disabled but cannot be deleted.
 5. Unreferenced custom entries may be deleted; referenced custom entries may only be relabeled or disabled.
 6. Settings tables show the agreed columns, alphabetical label order, usage counts, and selection-based actions.
 7. Create and Edit Document use one Linked People table with inline staged Add/Edit/Save/Cancel and multi-row Delete.
 8. The same Person cannot be staged or persisted twice for one Document, even under different roles.
 9. Document fields and Linked People changes commit atomically.
 10. Historical inactive roles remain displayable, while only active roles are assignable.
 11. Document Detail opens a print preview with Facsimile and Text-only formats.
 12. Print pages use current revised text when available and deterministic Source ordering.
 13. Printed archival metadata resolves authors through the hidden `author` semantic key after any label change.
 14. The job table contains one oldest-to-newest column per Job and ends with the Status row.
 15. Print output escapes stored text and does not disclose direct local source paths.
 16. Source reordering, server PDF generation, and print-template editing are absent.
 17. Database, integration, and UI verification uses isolated test data and never modifies `data/transcription.db`.
 ## Scope Freeze Gate
 V4.4 is sufficiently frozen to begin implementation:
 - Built-in registry identity, membership, lifecycle, display, and selector behavior are resolved.
 - Linked People selection, editing, uniqueness, inactive-role, staging, and transaction behavior are resolved.
 - Print entry point, formats, content order, transcription precedence, page order, job metadata, and output mechanism are resolved.
 - Clean development-schema and destructive-test boundaries are resolved.
 - Source page reordering remains excluded.
 Any expansion of the built-in catalogs, relationship cardinality, print formats, export formats, or print customization requires an explicit V4.4 scope amendment or a later revision.
 ## Related Local References
 - [V4.4 Implementation Plan](implementation_plan_v4_4.md)
 - [V4.3 Scope Boundary](../ver4.3/scope_boundary_v4_3.md)
 - [V4.3 Implementation Plan](../ver4.3/implementation_plan_v4_3.md)
 - [V4 Architecture](../ver4/architecture_v4.md)
 - [V4 Schema](../ver4/schema_v4.md)
 - [V4 Requirements](../ver4/requirements_v4.md)
@@ -0,0 +1,197 @@
 # System Architecture (Version 4)
 This document describes the production architecture of the document transcription system.
 ## Architecture Objectives
 - Preserve original source material, per-execution machine output, and separate human revision.
 - Support batching one or more images into ordered multi-page documents.
 - Capture submission-time prompt provenance and a per-page OpenRouter SDK response snapshot.
 - Execute page transcription concurrently with bounded `asyncio` workers.
 - Maintain relational portability across SQLite and PostgreSQL.
 - Keep operator workflows cross-platform and Python-driven.
 - Support many-to-many document-person relationships with extensible roles.
 - Support registry-driven document type classification.
 ## Core Capabilities
 - Ingest one or more images into sequential `Source` pages under a `Document`.
 - Execute asynchronous vision transcription with bounded worker concurrency.
 - Preserve original source files with SHA-256 digests and byte sizes.
 - Freeze prompt text, prompt hash, model, and explicitly configured sampling parameters on each `Job`.
 - Preserve page-level machine output, normalized metadata, and an SDK-serialized OpenRouter response snapshot on `JobSource`.
 - Organize historical `Person` records through many-to-many Document relationships and extensible roles.
 - Classify Documents through a UUID-identified registry with unique labels.
 - Maintain human revision separately from machine-generated text.
 - Isolate page failures so multi-page jobs can complete with partial success.
 - Operate across supported platforms through Python-based application and maintenance tooling.
 V4.2 extends this baseline with immutable execution attempts, exact OpenRouter transport evidence, safe
 versioned exports, and provider-neutral derived-artifact provenance. `JobSource` remains the mutable queue and
 compatibility projection; `ExecutionAttempt` is the authoritative append-only processing history. See the
 [V4.2 Scope Boundary](../ver4.2/scope_boundary_v4_2.md).
 ## Technical Stack
 - **Runtime:** Python 3.12 or later.
 - **Web application:** FastAPI and NiceGUI.
 - **Persistence:** SQLModel and SQLAlchemy, with SQLite and PostgreSQL support.
 - **Validation and settings:** Pydantic V2 and pydantic-settings.
 - **Concurrency:** Python `asyncio` workers.
 - **Vision integration:** OpenRouter through the application's provider adapter.
 - **Testing and quality:** pytest, pytest-asyncio, Ruff, and ty.
 ## Runtime Topology
 The runtime operates as an asynchronous Python application:
 - FastAPI + NiceGUI web application process.
 - In-process `asyncio` worker engine for transcription execution.
 - Relational persistence via SQLModel / SQLAlchemy.
 - Pydantic V2 validation across API payloads, prompt configuration, and structured metadata.
 ^^^mermaid
 flowchart LR
 U[Browser User] --> A[FastAPI + NiceGUI App]
 A --> W[Asyncio Worker Engine]
 A --> DB[(Relational DB)]
 W --> P[Vision Provider APIs]
 W --> DB
 ^^^
 ## Lifecycle Ownership
 Application lifespan owns runtime setup and teardown:
 - Initialize logging, settings, directories, and prompt configuration.
 - Manage asynchronous database engine connection pools.
 - Execute database bootstrap or migrations.
 - Recover stale or interrupted jobs on startup.
 - Manage graceful shutdown of active background tasks.
 ## Layered Module Structure
 ### Interface Layer
 - `src/transcription/ui/**`
 - `src/transcription/api/**`
 Responsibilities:
 - Render document, source, person, job, and classification views.
 - Accept user input for uploads, editing, linking, and revisions.
 - Present structured validation and conflict feedback.
 ### Application and Async Worker Layer
 - `src/transcription/services/workflows.py`
 - `src/transcription/worker.py`
 Responsibilities:
 - Orchestrate uploads, job creation, and status transitions.
 - Execute per-page provider calls through bounded concurrency.
 - Persist page-level outcomes and update aggregate job state.
 ### Domain and Service Layer
 - `src/transcription/db/models.py`
 - `src/transcription/services/documents.py`
 - `src/transcription/services/sources.py`
 - `src/transcription/services/jobs.py`
 - `src/transcription/services/people.py`
 - `src/transcription/services/workflows.py`
 Responsibilities:
 - Keep one primary service boundary per aggregate: Documents, Sources, Jobs, and People.
 - Documents own document records and the document-type registry.
 - Sources own source records, revisions, source media formats, MIME resolution, and page execution evidence.
 - Jobs own job lifecycle state and transitions.
 - People own person records, relationship roles, document-person links, and portrait media.
 - Apply deterministic conflict handling for relationship-role writes.
 - Use set-based synchronization for many-to-many relationship updates.
 - Resolve and validate registry-backed document types by UUID.
 ### Source Media Policy
 - `services/sources.py` is the single authority for accepted Source extensions and canonical MIME types.
 - Storage and provider payload loading must call the same Source validation functions.
 - Supported Source formats are JPEG, PNG, TIFF, and PDF.
 - Upload is an interface action, not a domain aggregate. Service names, errors, and workflow variables use
  `Source` terminology; compatibility aliases may remain temporarily at old import boundaries.
 ### Infrastructure Layer
 - `src/transcription/db/**`
 - `src/transcription/providers/**`
 Responsibilities:
 - Provide async database sessions and engine configuration.
 - Provide provider adapters for vision model execution.
 ## Core Workflows
 ### 1. Multi-Page Transcription
 1. User uploads one or more images for a `Document`.
 2. System stores files, hashes them, creates ordered `Source` rows, and creates a `Job`.
 3. Worker claims the job, marks it `processing`, and executes page calls concurrently.
 4. Each provider call appends an `ExecutionAttempt` with its request manifest, transport evidence, SDK snapshot,
   normalized metadata, timing, and outcome.
 5. The linked `JobSource` is updated as a compatibility projection, and a successful attempt updates the
   `Source.raw_transcription` latest-success projection.
 6. Aggregate status becomes `completed`, `partial_success`, or `failed`.
 ### 2. Document-Person Relationship Management
 1. User opens a document or person edit flow.
 2. UI loads existing links grouped by role.
 3. User adds or removes people within one or more roles.
 4. Service computes add/remove deltas rather than replacing all links blindly.
 5. Conflict checks enforce uniqueness and deterministic write semantics before persistence commits.
 ### 3. Document Type Management
 1. User selects a registry-backed document type for a document.
 2. Service resolves the Document Type UUID.
 3. Persistence stores the `document_type_id` reference.
 4. Inactive types remain valid for historical rows but are excluded from default selectors.
 ## V4 Domain Rules
 - `JobSource.raw_transcription` preserves page output for its Job execution.
 - `Source.raw_transcription` is the latest-success machine-output projection for a page.
 - Human corrections occur only in `Source.revised_text`.
 - Prompt and parameter provenance is frozen on `Job` at submission time.
 - The SDK-serialized OpenRouter response snapshot is stored on `JobSource` for each successful page execution.
 - Every V4.2 provider call appends a distinct `ExecutionAttempt`; retries never rewrite earlier attempts.
 - Exact response bytes identify the OpenRouter HTTP boundary and are not labeled as native upstream-provider JSON.
 - Generic `ProcessingArtifact` records use versioned schemas, digests, and one inline or external content location.
 - `DocumentPerson` links are unique for `(document_id, person_id, role_id)`.
 - Relationship mutations are deterministic and set-based.
 - `DocumentType.id` is canonical identity; its unique label may evolve.
 ## Data Model Summary
 - `Document` has one `DocumentType`, many `Source` pages, many `Job` runs, and many `Person` records through `DocumentPerson`.
 - `Source` belongs to one `Document` and may participate in many `JobSource` executions.
 - `Job` has many `JobSource` rows.
 - `PersonRole` defines available relationship roles.
 ## Test Strategy
 - Unit tests for models, validation, hashing, and registry resolution.
 - Service tests for CRUD, set-based sync, uniqueness conflicts, and deterministic relationship writes.
 - Async workflow tests for page isolation, partial failure handling, and stored evidence.
 - UI integration tests for multi-page rendering, role grouping, and document type selection.
 ## Related Local References
 - [System Overview](index_v4.md)
 - [System Requirements](requirements_v4.md)
 - [Data Model](schema_v4.md)
 - [Error Handling Policy](error_handling_v4.md)
 - [Error Handling Invariant](../invariant/error_handling.md)
 - [Digital Evidence and AI Processing Provenance](../invariant/ai_evidence_and_provenance.md)
@@ -0,0 +1,115 @@
 # Error Handling Policy (Version 4)
 This document defines the Version 4 taxonomy, contracts, and framework behavior used to satisfy the cross-version [Error Handling invariant](../invariant/error_handling.md).
 ## Invariant Alignment
 Version 4 implements the invariant through:
 - The shared error taxonomy below.
 - Structured error envelopes with correlation IDs.
 - Page-level failure isolation and explicit aggregate job status.
 - Atomic relationship and classification writes.
 - Consistent translation across API, UI, service, worker, persistence, and provider boundaries.
 - Bounded retry guidance based on category and idempotency.
 ## Scope and Authority
 This policy governs error behavior across:
 - NiceGUI pages
 - FastAPI routes
 - Service-layer orchestration
 - `asyncio` worker tasks
 - Database interactions
 - Provider adapters
 ## Error Taxonomy
 | Category | Definition | Retriable |
 | --- | --- | --- |
 | `validation_error` | Payload, parameter, or schema validation failure | no |
 | `user_input_error` | Unacceptable file, invalid selection, or malformed request from the operator | no |
 | `not_found_error` | Requested `Document`, `Source`, `Person`, `Job`, role, or type does not exist | no |
 | `conflict_error` | Operation violates uniqueness or relationship-write policy | no |
 | `external_provider_error` | Provider API failure, rate limit, or execution problem | yes |
 | `infrastructure_transient_error` | Temporary DB, file-system, or network instability | yes |
 | `infrastructure_persistent_error` | Persistent configuration, credential, or database availability failure | no |
 | `internal_unexpected_error` | Uncaught exception or logic defect | no |
 ## Async Batch and Page-Level Error Behavior
 In multi-page `asyncio` processing:
 1. Exceptions from individual page calls are trapped within the page task wrapper.
 2. Failed page detail is written to `JobSource.error_detail` and the page state becomes `failed`.
 3. Aggregate job status is derived from page outcomes:
   - all pages succeed -> `completed`
   - some succeed and some fail -> `partial_success`
   - all fail -> `failed`
 4. Successful pages remain valid even when sister pages fail.
 ## Relationship and Classification Conflict Behavior
 When relationship or document-type writes fail policy checks:
 1. Reject the full write operation.
 2. Return structured conflict detail including target identifiers and the violated rule.
 3. Preserve existing persisted relationships unchanged.
 ## API Error Response Contract
 API error responses return a structured envelope:
 ^^^json
 {
  "error_id": "err_uuid_12345",
  "category": "conflict_error",
  "message": "Relationship write conflicts with existing links.",
  "suggestion": "Adjust the requested relationship links and retry.",
  "details": {
    "document_id": "...",
    "person_id": "...",
    "attempted_role": "recipient",
    "operation": "add_link",
    "conflict_reason": "duplicate document-person-role link"
  },
  "timestamp": "2026-08-10T15:00:00Z"
 }
 ^^^
 HTTP status mappings:
 - `validation_error`, `user_input_error` -> `400`
 - `not_found_error` -> `404`
 - `conflict_error` -> `409`
 - `external_provider_error` -> `502` or `503`
 - `infrastructure_transient_error` -> `503`
 - `infrastructure_persistent_error`, `internal_unexpected_error` -> `500`
 ## UI Error Presentation Rules
 - Display concise failure summaries with the next action the operator can take.
 - Keep form state in context when feasible.
 - Distinguish validation issues, conflict issues, provider failures, and infrastructure failures.
 - For bulk relationship updates, identify the specific role or person that caused a conflict.
 ## Logging and Audit Expectations
 - Log worker failures with correlation IDs and provider context.
 - Log relationship and classification conflicts with machine-readable detail.
 - Log persisted provider errors and page-level execution failures.
 ## Retry Guidance
 - Do not auto-retry validation or conflict failures.
 - Permit user-driven retry after the input or selection changes.
 - Allow bounded retry for transient provider or infrastructure failures when the operation is idempotent.
 ## Related Local References
 - [Error Handling Invariant](../invariant/error_handling.md)
 - [System Overview](index_v4.md)
 - [System Requirements](requirements_v4.md)
 - [Data Model](schema_v4.md)
 - [System Architecture](architecture_v4.md)
@@ -0,0 +1,102 @@
 # Implementation Plan (Version 4)
 ## Goal
 Implement the Version 4 project definition from the current repository state while preserving existing data by default.
 ## Migration Policy
 - Database changes are non-destructive by default.
 - Exception: the legacy `document_type` text field may be replaced by a `document_type_id` reference without migrating existing text values.
 - Exception: `document_person` links may be recreated manually.
 ## Current Project Impact
 - `src/transcription/db/models.py` requires full schema alignment with the V4 core documents.
 - `src/transcription/services/documents.py` requires set-based document-person sync and document-type resolution.
 - API modules require additive role-aware relationship behavior and document-type selection behavior.
 - UI pages require grouped role displays, multi-role editing, and registry-backed document-type selection.
 - Existing tests require updates for role enforcement, document-type selection, and regression safety.
 ## Implementation Phases
 ### 1. Finalize the Transition Documents
 - Confirm the reset scope.
 - Confirm the database exception policy.
 - Keep core V4 documents as the only authoritative product definition.
 ### 2. Align the Persistence Layer
 - Update SQLModel definitions to match the final V4 schema.
 - Add `person_role` and `document_type` support.
 - Replace legacy document-type storage with `document_type_id`.
 - Apply the accepted manual exception strategy for `document_type` and `document_person` data.
 - Preserve all other data structures non-destructively.
 ### 3. Update Services and Write Semantics
 - Organize service ownership around Documents, Sources, Jobs, and People.
 - Centralize Source extension and MIME policy in the Sources service.
 - Treat upload as an interface action and remove it from domain service naming where compatibility permits.
 - Implement set-based synchronization for document-person updates.
 - Implement deterministic uniqueness and relationship-write conflict checks.
 - Remove suggestion-related service behavior.
 - Add document-type resolution and validation by UUID.
 ### 4. Update API Contracts
 - Keep API evolution additive.
 - Add role-aware relationship retrieval and write behavior.
 - Add document-type catalog retrieval and UUID-based selection for document writes.
 - Remove suggestion-related API surfaces from the V4 target state.
 ### 5. Update UI Workflows
 - Replace single-person link editing with grouped multi-role editing.
 - Render grouped role links on document and person detail views.
 - Replace free-text document type entry with registry-backed selection.
 - Preserve clear validation and conflict messaging.
 ### 6. Verification and Hardening
 - Add or update service tests for many-per-role behavior, uniqueness conflict handling, and set-based sync correctness.
 - Add API tests for relationship behavior and document-type selection.
 - Add UI tests or walkthrough coverage for grouped roles and type selection.
 - Add regression coverage for delete and cleanup semantics.
 - Enforce backup-first test execution for AI-run unit tests: backup `./data` before tests, then always prompt for restore after successful tests.
 - Keep restore confirmation-gated by default so code and test outcomes can be reviewed before data is reverted.
 ## Done When
 - Core V4 documents and code paths agree on the final project definition.
 - Relationship-role writes are deterministic and non-destructive.
 - Relationship-write conflict rules are enforced consistently.
 - Document type selection is registry-backed.
 - The accepted manual exceptions for `document_type` and `document_person` are completed.
 - The focused test coverage passes.
 ## Out of Scope
 - Suggested/asserted relationship state.
 - Suggestion review or extraction workflows.
 - Global person entity-resolution engine.
 - Automated semantic document-type classification.
 ## Delivery Order Recommendation
 1. Freeze scope boundary and implementation plan.
 2. Freeze core V4 documents.
 3. Align persistence models.
 4. Align services and API behavior.
 5. Align UI behavior.
 6. Run focused verification and regression checks.
 ## Related Local References
 - [V4 Scope Boundary](scope_boundary_v4.md)
 - [System Overview](index_v4.md)
 - [System Requirements](requirements_v4.md)
 - [Data Model](schema_v4.md)
 - [System Architecture](architecture_v4.md)
 - [Error Handling Policy](error_handling_v4.md)
@@ -0,0 +1,30 @@
 # Document Transcription System Overview (Version 4)
 Version 4 is the architecture baseline for the personal-scale application used to transcribe, organize, and preserve historical documents, source images, and related people records.
 ## Recommended Reading Order
 1. [System Architecture](architecture_v4.md) for capabilities, technical stack, runtime structure, workflows, and component ownership.
 2. [System Requirements](requirements_v4.md) for the verifiable V4 contract.
 3. [Data Model](schema_v4.md) for entities, relationships, constraints, and persistence rules.
 4. [Error Handling Policy](error_handling_v4.md) for the V4 taxonomy and boundary contracts.
 ## Cross-Version Invariants
 - [Historical Document Transcription Design Intent](../invariant/intent.md)
 - [Transcription Methodology](../invariant/transcription_methodology.md)
 - [Error Handling](../invariant/error_handling.md)
 - [Digital Evidence and AI Processing Provenance](../invariant/ai_evidence_and_provenance.md)
 - [UI Style Guide](../invariant/ui_style_guide.md)
 ## V4 Transition Documents
 - [Scope Boundary](scope_boundary_v4.md)
 - [Implementation Plan](implementation_plan_v4.md)
 ## Incremental Revisions
 - [V4.1 Scope](../ver4.1/scope_boundary_v4_1.md) and [Implementation Plan](../ver4.1/implementation_plan_v4_1.md)
 - [V4.2 Evidence and Provenance Scope](../ver4.2/scope_boundary_v4_2.md) and [Implementation Plan](../ver4.2/implementation_plan_v4_2.md)
 - [V4.3 Settings Scope](../ver4.3/scope_boundary_v4_3.md) and [Implementation Plan](../ver4.3/implementation_plan_v4_3.md)
 - [V4.4 Semantic Registries, Linked People, and Printing Scope](../ver4.4/scope_boundary_v4_4.md) and [Implementation Plan](../ver4.4/implementation_plan_v4_4.md)
@@ -0,0 +1,51 @@
 # Document Transcription System Requirements (Version 4)
 This document defines the baseline requirements for the document transcription system.
 ## Requirements Model
 | ID | Category | Requirement | Verify Method |
 | --- | --- | --- | --- |
 | REQ-0 | System | Provide end-to-end multi-page document transcription with persistent, inspectable async job states. | demonstration |
 | REQ-1 | Functional | Allow users to upload one or more images as ordered `Source` pages under a `Document`. | test |
 | REQ-2 | Functional | Process page transcription asynchronously using an `asyncio` worker pool bounded by rate limits. | test |
 | REQ-3 | Functional | Persist submission-time request provenance and accurately labeled page-level SDK evidence; V4.2 adds exact OpenRouter-boundary transport evidence for new attempts. | test |
 | REQ-4 | Functional | Support job states `queued`, `processing`, `completed`, `partial_success`, and `failed`, plus page states `pending`, `transcribed`, and `failed`. | inspection |
 | REQ-5 | Functional | Allow users to manage historical `Person` records and link multiple people per role to a `Document`. | test |
 | REQ-6 | Functional | Support an extensible role taxonomy for document-person relationships. | inspection |
 | REQ-7 | Policy Constraint | Enforce deterministic relationship-role writes with uniqueness on `(document_id, person_id, role_id)` and explicit conflict responses for invalid duplicate link attempts. | test |
 | REQ-8 | Functional | Use set-based synchronization for document-person mutations so updates add and remove only the intended links. | test |
 | REQ-9 | Functional | Maintain immutable machine output on `Source.raw_transcription` while permitting inline human edits on `Source.revised_text`. | test |
 | REQ-10 | Functional | Support a UUID-identified `DocumentType` taxonomy with unique user-facing labels and active/inactive lifecycle control. | test |
 | REQ-11 | Data Constraint | Store `Document` type as a controlled reference to `DocumentType`. | test |
 | REQ-12 | Interface | Render multi-page transcriptions sequentially by `page_number` with document, people, and document-type metadata. | demonstration |
 | REQ-13 | Interface | Document create/edit UI must support selecting multiple people per role and selecting an active document type from the registry. | demonstration |
 | REQ-14 | API Constraint | Expose additive, role-aware retrieval and write behavior for document-person links and UUID-based selection for document types. | test |
 | REQ-15 | Data Constraint | Calculate and store cryptographic file hashes (SHA-256) and file sizes for uploaded source images. | test |
 | REQ-16 | Data Constraint | Preserve a portable relational model across supported backends using SQLModel, SQLAlchemy, SQLite, and PostgreSQL. | inspection |
 | REQ-17 | Reliability | Ensure delete and update flows for documents, people, and relationship links remain deterministic and safe. | test |
 | REQ-18 | Operations Constraint | Keep canonical development, testing, restore, and recovery workflows OS-independent; for AI-run unit tests, require a pre-test backup of `./data` and an always-shown post-success confirmation prompt before any restore action. | inspection |
 | REQ-19 | Quality | Provide automated coverage for async transcription workflows, relationship-role enforcement, document-type selection, and regression behavior. | test |
 ## Clarifying Constraints
 1. `DocumentType.id` is its sole identity; labels are unique ignoring case and surrounding whitespace.
 2. `PersonRole.code` is a stable machine identifier; `PersonRole.label` may evolve.
 3. Relationship-write policy and conflict handling must be consistent across UI, API, services, and persistence.
 4. Many-per-role behavior is required for document-person links.
 5. Relationship conflicts must fail deterministically without partial mutation.
 ## Element Satisfaction Mapping
 - UI (NiceGUI): Satisfies REQ-0, REQ-1, REQ-5, REQ-9, REQ-12, REQ-13.
 - API (FastAPI): Satisfies REQ-1, REQ-4, REQ-5, REQ-7, REQ-8, REQ-14.
 - Worker (`asyncio`): Satisfies REQ-2, REQ-3, REQ-4.
 - Persistence (SQLModel / SQLAlchemy): Satisfies REQ-3, REQ-9, REQ-10, REQ-11, REQ-15, REQ-16, REQ-17.
 - Test Suite: Verifies all test-marked requirements and satisfies REQ-19.
 ## Related Local References
 - [System Overview](index_v4.md)
 - [System Architecture](architecture_v4.md)
 - [Data Model](schema_v4.md)
 - [Error Handling Policy](error_handling_v4.md)
@@ -0,0 +1,236 @@
 # Database Schema (Version 4)
 This document defines the relational schema for the document transcription system.
 ## Entity Relationship Diagram
 ```mermaid
 erDiagram
 DOCUMENT_TYPE {
 UUID id PK
 TEXT label
 TEXT normalized_label
 BOOLEAN is_active
 TIMESTAMPTZ created_at
 TIMESTAMPTZ updated_at
 }
 PERSON_ROLE {
 UUID id PK
 TEXT code
 TEXT label
 BOOLEAN is_active
 TIMESTAMPTZ created_at
 TIMESTAMPTZ updated_at
 }
 PERSON {
 UUID id PK
 TEXT full_name
 TEXT display_name
 TEXT maiden_name
 DATE birth_date
 TEXT birth_date_raw
 TEXT birth_place
 DATE death_date
 TEXT death_date_raw
 TEXT death_place
 TEXT biography
 TEXT portrait_path
 JSONB metadata
 TIMESTAMPTZ created_at
 TIMESTAMPTZ updated_at
 }
 DOCUMENT {
 UUID id PK
 UUID document_type_id FK
 TEXT name
 DATE document_date
 TEXT document_date_raw
 TEXT location_created
 TEXT notes
 TEXT archive_identifier
 TIMESTAMPTZ created_at
 TIMESTAMPTZ updated_at
 }
 DOCUMENT_PERSON {
 UUID id PK
 UUID document_id FK
 UUID person_id FK
 UUID role_id FK
 TIMESTAMPTZ created_at
 TIMESTAMPTZ updated_at
 }
 JOB {
 UUID id PK
 UUID document_id FK
 VARCHAR status
 INTEGER retry_count
 TEXT provider
 TEXT model
 TEXT prompt_name
 TEXT prompt_hash
 TEXT system_prompt
 TEXT user_prompt
 FLOAT temperature
 FLOAT top_p
 TIMESTAMPTZ date_created
 TIMESTAMPTZ date_updated
 }
 SOURCE {
 UUID id PK
 UUID document_id FK
 INTEGER page_number
 TEXT upload_name
 TEXT filename
 TEXT file_path
 TEXT file_hash
 BIGINT file_size_bytes
 TEXT raw_transcription
 TEXT revised_text
 TIMESTAMPTZ date_uploaded
 TIMESTAMPTZ date_revised
 }
 JOB_SOURCE {
 UUID id PK
 UUID job_id FK
 UUID source_id FK
 VARCHAR status
 TEXT raw_transcription
 JSONB ai_metadata
 JSONB raw_api_response
 TEXT error_detail
 TIMESTAMPTZ executed_at
 }
 EXECUTION_ATTEMPT {
 UUID id PK
 UUID job_source_id FK
 UUID job_id FK
 UUID source_id FK
 INTEGER attempt_number
 VARCHAR status
 JSONB request_manifest
 TEXT request_manifest_sha256
 INTEGER transport_status_code
 BINARY transport_body
 JSONB transport_safe_headers
 JSONB sdk_response_snapshot
 JSONB normalized_metadata
 JSONB software_context
 TEXT raw_transcription
 TEXT failure_phase
 TIMESTAMPTZ started_at
 TIMESTAMPTZ finished_at
 INTEGER duration_ms
 }
 PROCESSING_ARTIFACT {
 UUID id PK
 UUID source_id FK
 UUID execution_attempt_id FK
 TEXT artifact_type
 TEXT media_type
 TEXT schema_name
 TEXT schema_version
 TEXT producer
 TEXT producer_version
 JSONB inline_payload
 TEXT external_reference
 TEXT payload_sha256
 BIGINT byte_size
 JSONB coordinate_metadata
 TIMESTAMPTZ created_at
 }
 DOCUMENT_TYPE ||--o{ DOCUMENT : classifies
 DOCUMENT ||--o{ DOCUMENT_PERSON : has_people
 PERSON ||--o{ DOCUMENT_PERSON : appears_in
 PERSON_ROLE ||--o{ DOCUMENT_PERSON : labels
 DOCUMENT ||--o{ JOB : has_jobs
 DOCUMENT ||--o{ SOURCE : contains_pages
 JOB ||--o{ JOB_SOURCE : executes
 SOURCE ||--o{ JOB_SOURCE : processed_in
 JOB_SOURCE ||--o{ EXECUTION_ATTEMPT : projects
 SOURCE ||--o{ PROCESSING_ARTIFACT : derives
 EXECUTION_ATTEMPT ||--o{ PROCESSING_ARTIFACT : produces
 ```
 ## Domain Invariants and Provenance Rules
 ### Page-Level Execution and AI Outputs
 - Every single page execution by an AI model produces a dedicated `JOB_SOURCE` record.
 - Every `JOB` stores the frozen prompt identifier, prompt text, and hyperparameters used at submission time.
 - `JOB_SOURCE.raw_api_response` is a compatibility projection containing an SDK-serialized OpenRouter response
  snapshot. It is neither the exact HTTP body nor the native upstream-provider response.
 - Every new provider call creates an immutable `EXECUTION_ATTEMPT` containing the frozen request manifest,
  exact OpenRouter-boundary response bytes when received, safe transport metadata, SDK snapshot, normalized
  metadata, timing, and outcome.
 - `EXECUTION_ATTEMPT(job_id, source_id, attempt_number)` is unique; retries increment the persisted attempt number.
 - Historical `JOB_SOURCE` rows without an `EXECUTION_ATTEMPT` remain SDK snapshots and are explicitly labeled as
  lacking transport evidence.
 - `SOURCE.raw_transcription` caches the latest successful machine output for that page.
 ### Generic Processing Artifacts
 - `PROCESSING_ARTIFACT` stores provider-neutral versioned derived outputs.
 - Exactly one of `inline_payload` and `external_reference` is populated.
 - Externally stored artifacts use application-managed relative references and are verified by SHA-256 and byte size.
 - Coordinate metadata declares units, origin, dimensions, and transformations when geometry is present.
 ### Image Storage and Integrity
 - Binary images are stored on disk; `SOURCE.file_path` stores the persisted path.
 - `SOURCE.file_hash` stores a SHA-256 digest.
 - `SOURCE.file_size_bytes` stores the original file size.
 ### Page Ordering and Revisions
 - `SOURCE.page_number` dictates page ordering within a document.
 - `SOURCE.raw_transcription` remains immutable machine output.
 - `SOURCE.revised_text` stores human edits and is the preferred display value when present.
 ### Document-Person Role Governance
 - Documents support zero, one, or many people per relationship role.
 - Relationship roles are defined by `PERSON_ROLE` rather than hardcoded columns.
 - `DOCUMENT_PERSON` must be unique for `(document_id, person_id, role_id)`.
 - Relationship writes must be deterministic and use explicit add/remove link intent.
 ### Document Type Governance
 - Every document type is defined by `DOCUMENT_TYPE`.
 - `DOCUMENT_TYPE.id` is the sole machine identity.
 - `DOCUMENT_TYPE.label` is mutable display text and is unique after trimming and case normalization.
 - `DOCUMENT_TYPE.normalized_label` stores the normalized uniqueness key.
 - Inactive types remain valid for historical rows but should be excluded from default selection UIs.
 ## Constraint Summary
 - `DOCUMENT_TYPE.normalized_label` is unique.
 - `PERSON_ROLE.code` is unique.
 - `DOCUMENT_PERSON(document_id, person_id, role_id)` is unique.
 ## Indexing Guidance
 - `document(document_type_id)`
 - `document_person(document_id)`
 - `document_person(person_id)`
 - `document_person(role_id)`
 - `source(document_id, page_number)`
 - `job(document_id, status)`
 - `job_source(job_id)`
 - `job_source(source_id)`
 ## Related Local References
 - [System Overview](index_v4.md)
 - [System Architecture](architecture_v4.md)
 - [System Requirements](requirements_v4.md)
 - [Error Handling Policy](error_handling_v4.md)
@@ -0,0 +1,89 @@
 # V4 Scope Boundary
 This document defines the scope for the transition from the current repository state to the Version 4 project definition.
 ## Purpose
 Define what this revision includes, what it intentionally excludes, and what migration rules govern the transition work.
 ## In Scope
 ### 1. Relationship Model
 - Extensible role taxonomy for document-person relationships.
 - Many-to-many document-person links with many people per role.
 - Set-based add/remove synchronization for document-person updates.
 ### 2. Document Type Governance
 - Registry-driven `DocumentType` model with UUID identity, unique labels, and controlled selection.
 - Minimal rollout for the current corpus with no alias helper table.
 ### 3. UI and API Behavior
 - Grouped role links on document and person views.
 - Multi-role relationship editing on document create/edit flows.
 - Role-aware API retrieval and write behavior.
 - Additive API evolution with explicit deprecations.
 ### 4. Verification
 - Tests for many-per-role behavior.
 - Tests for set-based relationship mutation behavior.
 - Tests for document and person delete/link cleanup regressions.
 ## Out of Scope
 - Suggested versus asserted relationship states.
 - Suggestion storage, review, acceptance, or rejection workflows.
 - Automatic relationship extraction or recommendation features.
 - Full entity resolution or identity merge across all people.
 - Automated semantic document type classification.
 - Redesign of the core transcription execution model.
 ## Locked Design Decisions
 ### A. Role Extensibility Mechanism
 - Use registry tables for relationship roles.
 ### B. API Compatibility Strategy
 - Use additive API evolution.
 - In development mode, the current revision is authoritative.
 - Deprecations should be explicit and short-lived.
 ### C. Document Type Rollout Strategy
 - Use a minimal registry rollout for the current corpus.
 - Do not introduce a `document_type_alias` helper table.
 ### D. Database Change Policy
 - Future schema changes are non-destructive by default.
 - Exception: `document_type` text may be replaced by `document_type_id` without migrating the legacy text values.
 - Exception: `document_person` links may be recreated manually.
 ## Compatibility and Rollout
 - Preserve existing repository behavior where unaffected by the V4 scope.
 - Treat scope boundary and implementation plan as the only transition documents.
 - Treat core V4 documents as the authoritative project definition once rewritten.
 ## Exit Criteria for Scope Freeze
 V4 scope is considered frozen when:
 - Relationship model and document-type governance are approved.
 - Relationship model and document-type governance are approved.
 - Additive API change list and deprecation schedule are approved.
 - Migration exceptions are explicitly acknowledged.
 ## Core V4 Documents
 1. `docs/ver4/index_v4.md`
 2. `docs/ver4/requirements_v4.md`
 3. `docs/ver4/schema_v4.md`
 4. `docs/ver4/architecture_v4.md`
 5. `docs/ver4/error_handling_v4.md`
 6. `docs/ver4/implementation_plan_v4.md`
@@ -5,9 +5,11 @@ This directory stores transcription prompts as individual Markdown artifacts.
 ## Conventions
 - Keep one prompt per file.
 - Use stable, descriptive snake_case file names.
 - Store prompt files directly in this directory; nested paths are rejected.
 - Prefer incremental edits to a single prompt per change for clean history.
 - Keep prompts human-readable and policy-focused.
 - Do not store secrets in prompt files.
 - Runtime jobs snapshot prompt text, SHA-256 provenance, and sampling configuration.
 ## Current Prompt
 - `transcribe_document.md`: baseline verbatim transcription policy for historical documents.
@@ -1,10 +0,0 @@
 You are an assistant that may call tools.
 Tool safety rules:
 1) Tool arguments MUST be strict JSON matching the schema exactly.
 2) Never place disallowed, sensitive, explicit, or policy-violating text directly into tool arguments.
 3) If user content may be unsafe, first produce a brief neutral summary and pass only that summary.
 4) Prefer IDs, enums, booleans, and short fields over raw free-form text.
 5) Keep all string arguments <= 300 chars unless schema says otherwise.
 6) If you cannot safely provide valid tool args, do not call the tool; respond with "NO_TOOL_CALL" and explain briefly.
 7) Never include markdown/code fences in tool arguments.
@@ -46,6 +46,16 @@ Do not summarize. Do not paraphrase. Do not modernize style.
 - Signal location before the note text.
 - Example form: `[written in left margin: ...]`
 ### Printed and handwritten text
 - Preserve printed and handwritten text together in their original reading context.
 - On mixed documents such as completed forms, leave printed labels and instructions unmarked.
 - Wrap handwritten entries in `[handwritten: ...]`.
 - Mark handwritten signatures as `[handwritten signature: ...]`.
 - If the main body is entirely handwritten, add `[document body handwritten]` once at the beginning rather than marking every line.
 - Mark later notes or uncertain additions as `[handwritten annotation: ...]`.
 - When authorship is unclear, use `[handwritten annotation, author uncertain: ...]`.
 - Do not infer authorship, writing date, or whether different handwriting belongs to different people unless explicitly evident.
 ### Line-break hyphenation
 - Rejoin words split across line breaks when they are clearly one word.
 - Remove only line-break hyphens used for wrapping.
@@ -70,3 +80,4 @@ Before finalizing, ensure:
 2. Uncertain/illegible areas are explicitly marked.
 3. Crossed-out and inserted text are preserved with required tags.
 4. Structure/ordering is preserved as faithfully as possible.
 5. Handwriting is identified using the mixed-text conventions without separating it from its printed context.
@@ -39,6 +39,10 @@ dev = [
 [tool.pytest.ini_options]
 addopts = "--strict-markers -q"
 asyncio_mode = "strict"
 filterwarnings = [
    "error:coroutine .* was never awaited:RuntimeWarning",
 ]
 markers = [
    "unit: pure logic tests with no external dependencies",
    "integration: tests that touch framework or database contracts",
@@ -0,0 +1,27 @@
 import uvicorn
 from fastapi import FastAPI
 from .app import create_app
 from .config import parse_cli_settings
 def create_cli_app() -> FastAPI:
    """Create an app from CLI settings for Uvicorn's reload process."""
    return create_app(settings=parse_cli_settings())
 def main() -> None:
    settings = parse_cli_settings()
    application = "transcription.__main__:create_cli_app" if settings.reload else create_app(settings=settings)
    uvicorn.run(
        application,
        factory=settings.reload,
        host=settings.host,
        port=settings.port,
        log_level=settings.log_level,
        reload=settings.reload,
    )
 if __name__ == "__main__":
    main()
@@ -0,0 +1,252 @@
 """Additive V4 API routes for relationship and classification registries."""
 from __future__ import annotations
 from typing import Annotated
 from uuid import UUID
 from fastapi import APIRouter
 from fastapi import Depends
 from fastapi import Request
 from fastapi import Response
 from pydantic import BaseModel
 from pydantic import ConfigDict
 from pydantic import Field
 from pydantic import model_validator
 from transcription.db.models import Document
 from transcription.db.models import DocumentPerson
 from transcription.db.models import DocumentType
 from transcription.db.models import PersonRole
 from transcription.services import DocumentService
 from transcription.services import PeopleService
 router = APIRouter(prefix="/api/v4", tags=["v4-documents"])
 class ApiModel(BaseModel):
    model_config = ConfigDict(extra="forbid", frozen=True, str_strip_whitespace=True)
 class SelectorRequest(ApiModel):
    @model_validator(mode="after")
    def require_exactly_one_selector(self):
        values = (self.selector_id, self.selector_code)
        if sum(value is not None for value in values) != 1:
            raise ValueError(f"Provide exactly one of {self.selector_names[0]} or {self.selector_names[1]}")
        return self
    @property
    def selector_id(self) -> UUID | None:
        raise NotImplementedError
    @property
    def selector_code(self) -> str | None:
        raise NotImplementedError
    @property
    def selector_names(self) -> tuple[str, str]:
        raise NotImplementedError
 class DocumentTypeRead(ApiModel):
    id: UUID
    label: str
    is_active: bool
 class PersonRoleRead(ApiModel):
    id: UUID
    code: str
    label: str
    is_active: bool
 class DocumentTypeWriteRequest(ApiModel):
    document_type_id: UUID
 class DocumentTypeWriteResponse(ApiModel):
    document_id: UUID
    document_type_id: UUID
 class DocumentPersonWriteRequest(ApiModel):
    person_id: UUID
    role_id: UUID | None = None
    role_code: str | None = Field(default=None, min_length=1, pattern=r"^[a-z0-9_]+$")
    @model_validator(mode="after")
    def reject_conflicting_role_selectors(self):
        if self.role_id is not None and self.role_code is not None:
            raise ValueError("Provide role_id or role_code, not both")
        return self
 class DocumentPersonRoleUpdateRequest(SelectorRequest):
    role_id: UUID | None = None
    role_code: str | None = Field(default=None, min_length=1, pattern=r"^[a-z0-9_]+$")
    @property
    def selector_id(self) -> UUID | None:
        return self.role_id
    @property
    def selector_code(self) -> str | None:
        return self.role_code
    @property
    def selector_names(self) -> tuple[str, str]:
        return "role_id", "role_code"
 class DocumentPersonRead(ApiModel):
    id: UUID
    document_id: UUID
    person_id: UUID
    role_id: UUID | None
    role_code: str
    person_name: str | None = None
 class DocumentPeopleResponse(ApiModel):
    document_id: UUID
    links: list[DocumentPersonRead] = Field(default_factory=list)
 def _document_type_to_read(item: DocumentType) -> DocumentTypeRead:
    return DocumentTypeRead(
        id=item.id,
        label=item.label,
        is_active=item.is_active,
    )
 def _person_role_to_read(item: PersonRole) -> PersonRoleRead:
    return PersonRoleRead(
        id=item.id,
        code=item.code,
        label=item.label,
        is_active=item.is_active,
    )
 def _document_person_to_read(item: DocumentPerson) -> DocumentPersonRead:
    role_code = item.role_ref.code if item.role_ref is not None else str(item.role)
    person_name = item.person.full_name if item.person is not None else None
    return DocumentPersonRead(
        id=item.id,
        document_id=item.document_id,
        person_id=item.person_id,
        role_id=item.role_id,
        role_code=role_code,
        person_name=person_name,
    )
 def _document_to_type_response(item: Document) -> DocumentTypeWriteResponse:
    if item.document_type_id is None:
        raise ValueError("Document Type assignment did not persist")
    return DocumentTypeWriteResponse(
        document_id=item.id,
        document_type_id=item.document_type_id,
    )
 def get_document_service(request: Request) -> DocumentService:
    """Resolve the document service from app lifespan state when available."""
    services = getattr(request.app.state, "services", None)
    if services is not None:
        return services.documents
    return DocumentService()
 def get_people_service(request: Request) -> PeopleService:
    """Resolve the People service from app lifespan state when available."""
    services = getattr(request.app.state, "services", None)
    if services is not None:
        return services.people
    return PeopleService()
 DocumentServiceDependency = Annotated[DocumentService, Depends(get_document_service)]
 PeopleServiceDependency = Annotated[PeopleService, Depends(get_people_service)]
@router.get("/document-types", response_model=list[DocumentTypeRead])
 async def list_document_types(
    service: DocumentServiceDependency,
    active_only: bool = True,
 ) -> list[DocumentTypeRead]:
    items = await service.list_document_types(active_only=active_only)
    return [_document_type_to_read(item) for item in items]
@router.get("/person-roles", response_model=list[PersonRoleRead])
 async def list_person_roles(
    service: PeopleServiceDependency,
    active_only: bool = True,
 ) -> list[PersonRoleRead]:
    items = await service.list_person_roles(active_only=active_only)
    return [_person_role_to_read(item) for item in items]
@router.put("/documents/{document_id}/type", response_model=DocumentTypeWriteResponse)
 async def set_document_type(
    document_id: UUID,
    payload: DocumentTypeWriteRequest,
    service: DocumentServiceDependency,
 ) -> DocumentTypeWriteResponse:
    document = await service.set_document_type(
        document_id=document_id,
        document_type_id=payload.document_type_id,
    )
    return _document_to_type_response(document)
@router.get("/documents/{document_id}/people", response_model=DocumentPeopleResponse)
 async def list_document_people(
    document_id: UUID,
    service: PeopleServiceDependency,
 ) -> DocumentPeopleResponse:
    links = await service.list_document_people(document_id=document_id)
    return DocumentPeopleResponse(document_id=document_id, links=[_document_person_to_read(item) for item in links])
@router.post("/documents/{document_id}/people", response_model=DocumentPersonRead)
 async def add_document_person_link(
    document_id: UUID,
    payload: DocumentPersonWriteRequest,
    service: PeopleServiceDependency,
 ) -> DocumentPersonRead:
    link = await service.add_document_person_link(
        document_id=document_id,
        person_id=payload.person_id,
        role_id=payload.role_id,
        role_code=payload.role_code,
    )
    return _document_person_to_read(link)
@router.patch("/document-people/{document_person_id}", response_model=DocumentPersonRead)
 async def set_document_person_role(
    document_person_id: UUID,
    payload: DocumentPersonRoleUpdateRequest,
    service: PeopleServiceDependency,
 ) -> DocumentPersonRead:
    link = await service.set_document_person_role(
        document_person_id=document_person_id,
        role_id=payload.role_id,
        role_code=payload.role_code,
    )
    return _document_person_to_read(link)
@router.delete("/document-people/{document_person_id}", status_code=204)
 async def delete_document_person_link(
    document_person_id: UUID,
    service: PeopleServiceDependency,
 ) -> Response:
    await service.remove_document_person_link(document_person_id=document_person_id)
    return Response(status_code=204)
@@ -2,8 +2,12 @@
 from __future__ import annotations
 import logging
 from contextlib import AsyncExitStack
 from contextlib import asynccontextmanager
 from datetime import UTC
 from datetime import datetime
 from datetime import timedelta
 from fastapi import FastAPI
 from fastapi import status
@@ -12,24 +16,37 @@ from fastapi.staticfiles import StaticFiles
 from .api.errors import register_error_handlers
 from .api.health import router as health_router
 from .api.v4_documents import router as v4_documents_router
 from .config import Settings
 from .config import configure_logging
 from .config import get_settings
 from .db import create_all
 from .db import dispose_database_runtime
 from .db import initialize_database_runtime
 from .services import ServiceBundle
 from .services.documents import DocumentService
 from .services.jobs import JobService
 from .services.people import PeopleService
 from .services.sources import SourceService
 from .ui import register_pages
 from .worker import worker_consumer_lifespan
 logger = logging.getLogger(__name__)
@asynccontextmanager
 async def _lifespan(app: FastAPI):
    configure_logging()
    settings = getattr(app.state, "settings", None) or get_settings()
    configure_logging(settings)
    app.state.settings = settings
    app.state.services = ServiceBundle()
    app.state.runtime = initialize_database_runtime(settings=settings)
    session_factory = app.state.runtime.session_factory
    app.state.services = ServiceBundle(
        documents=DocumentService(session_factory=session_factory, settings=settings),
        sources=SourceService(session_factory=session_factory, settings=settings),
        jobs=JobService(session_factory=session_factory, settings=settings),
        people=PeopleService(session_factory=session_factory, settings=settings),
    )
    if settings.should_bootstrap_schema:
        await create_all(engine=app.state.runtime.engine)
@@ -37,6 +54,8 @@ async def _lifespan(app: FastAPI):
    settings.upload_dir.mkdir(parents=True, exist_ok=True)
    settings.prompt_dir.mkdir(parents=True, exist_ok=True)
    await _recover_stale_processing_jobs(app)
    async with AsyncExitStack() as stack:
        stack.push_async_callback(dispose_database_runtime)
        stop_event, worker_notifier = await stack.enter_async_context(
@@ -50,26 +69,41 @@ async def _lifespan(app: FastAPI):
        yield
-def create_app() -> FastAPI:
+async def _recover_stale_processing_jobs(app: FastAPI) -> None:
    """Re-queue stale processing jobs at startup.
    Any job left in PROCESSING longer than the configured provider timeout is
    assumed orphaned and moved back to QUEUED before the worker starts.
    """
    settings = app.state.settings
    stale_before = datetime.now(UTC) - timedelta(seconds=settings.worker_provider_timeout_seconds)
    job_service = JobService(session_factory=app.state.runtime.session_factory)
    recovered = await job_service.requeue_stale_processing_jobs(stale_before=stale_before)
    if recovered > 0:
        logger.warning("Recovered %s stale processing job(s) at startup", recovered)
 def create_app(settings: Settings | None = None) -> FastAPI:
    """Create and configure the FastAPI application."""
    app = FastAPI(title="Transcription", lifespan=_lifespan)
-    settings = get_settings()
+    active_settings = settings or get_settings()
-    app.state.settings = settings
+    app.state.settings = active_settings
    app.mount(
        "/uploads",
-        StaticFiles(directory=settings.upload_dir, check_dir=False),
+        StaticFiles(directory=active_settings.upload_dir, check_dir=False),
        name="uploads",
    )
    @app.get("/", include_in_schema=False)
    async def root_redirect() -> RedirectResponse:
-        return RedirectResponse(url="/ui", status_code=status.HTTP_307_TEMPORARY_REDIRECT)
+        return RedirectResponse(url="/ui/homepage", status_code=status.HTTP_307_TEMPORARY_REDIRECT)
    @app.get("/ui", include_in_schema=False)
    async def ui_redirect() -> RedirectResponse:
-        return RedirectResponse(url="/ui/upload", status_code=status.HTTP_307_TEMPORARY_REDIRECT)
+        return RedirectResponse(url="/ui/homepage", status_code=status.HTTP_307_TEMPORARY_REDIRECT)
    register_error_handlers(app)
    register_pages(app)
    app.include_router(health_router)
    app.include_router(v4_documents_router)
    register_pages(app)
    return app
@@ -7,7 +7,7 @@ from sqlalchemy.ext.asyncio import async_sessionmaker
 from sqlmodel.ext.asyncio.session import AsyncSession
 from transcription.db.runtime import DatabaseRuntime
-from transcription.db.runtime import get_session_factory
+from transcription.db.session import get_session_factory
 from transcription.worker import WorkerNotifier
 from transcription.worker import resolve_worker_notifier
@@ -0,0 +1,102 @@
 """Private-corpus benchmark contracts and deterministic text scoring."""
 from __future__ import annotations
 from uuid import UUID
 from pydantic import BaseModel
 from pydantic import ConfigDict
 from pydantic import Field
 class BenchmarkModel(BaseModel):
    model_config = ConfigDict(extra="forbid", frozen=True)
 class BenchmarkItem(BenchmarkModel):
    """One private benchmark item referenced by archival identity."""
    source_id: UUID
    source_digest_sha256: str = Field(pattern=r"^[0-9a-f]{64}$")
    categories: frozenset[str] = Field(min_length=1)
    reference_transcription: str = Field(min_length=1)
 class BenchmarkManifest(BenchmarkModel):
    """Versioned private benchmark definition without copied source media."""
    schema_name: str = "transcription.private-benchmark"
    schema_version: str = "1"
    name: str = Field(min_length=1)
    items: tuple[BenchmarkItem, ...] = Field(min_length=1)
 class EditorialAssessment(BenchmarkModel):
    """Manually reviewed errors not represented adequately by CER or WER."""
    omissions: int = Field(default=0, ge=0)
    inventions: int = Field(default=0, ge=0)
    silent_normalizations: int = Field(default=0, ge=0)
    uncertainty_errors: int = Field(default=0, ge=0)
    layout_errors: int = Field(default=0, ge=0)
 class BenchmarkScore(BenchmarkModel):
    """Measured score for one preserved execution attempt."""
    execution_attempt_id: UUID
    character_error_rate: float = Field(ge=0)
    word_error_rate: float = Field(ge=0)
    character_edits: int = Field(ge=0)
    word_edits: int = Field(ge=0)
    reference_characters: int = Field(ge=0)
    reference_words: int = Field(ge=0)
    assessment: EditorialAssessment
    latency_ms: int = Field(ge=0)
    cost_usd: float | None = Field(default=None, ge=0)
 def score_transcription(
    *,
    execution_attempt_id: UUID,
    reference: str,
    candidate: str,
    assessment: EditorialAssessment,
    latency_ms: int,
    cost_usd: float | None = None,
 ) -> BenchmarkScore:
    """Score literal text without case-folding or silent normalization."""
    reference_words = reference.split()
    candidate_words = candidate.split()
    character_edits = _levenshtein(list(reference), list(candidate))
    word_edits = _levenshtein(reference_words, candidate_words)
    return BenchmarkScore(
        execution_attempt_id=execution_attempt_id,
        character_error_rate=character_edits / max(1, len(reference)),
        word_error_rate=word_edits / max(1, len(reference_words)),
        character_edits=character_edits,
        word_edits=word_edits,
        reference_characters=len(reference),
        reference_words=len(reference_words),
        assessment=assessment,
        latency_ms=latency_ms,
        cost_usd=cost_usd,
    )
 def _levenshtein(reference: list[str], candidate: list[str]) -> int:
    if len(reference) < len(candidate):
        reference, candidate = candidate, reference
    previous = list(range(len(candidate) + 1))
    for reference_index, reference_value in enumerate(reference, start=1):
        current = [reference_index]
        for candidate_index, candidate_value in enumerate(candidate, start=1):
            current.append(
                min(
                    current[-1] + 1,
                    previous[candidate_index] + 1,
                    previous[candidate_index - 1] + (reference_value != candidate_value),
                )
            )
        previous = current
    return previous[-1]
@@ -6,11 +6,19 @@ are resolved by the provider adapters, not here.
 """
 import logging.config
-from contextvars import ContextVar
+from collections.abc import Sequence
 from enum import StrEnum
 from functools import cache
 from pathlib import Path
 from typing import Annotated
 from typing import Any
 from typing import Literal
 from pydantic import BaseModel
 from pydantic import ConfigDict
 from pydantic import Field
 from pydantic import SecretStr
 from pydantic import StringConstraints
 from pydantic_settings import BaseSettings
 from pydantic_settings import SettingsConfigDict
@@ -21,56 +29,106 @@ class Provider(StrEnum):
    OPENROUTER = "openrouter"
 NonEmptyStr = Annotated[str, StringConstraints(strip_whitespace=True, min_length=1)]
 PromptFilename = Annotated[str, StringConstraints(strip_whitespace=True, min_length=1, pattern=r"^[^/\\]+$")]
 Probability = Annotated[float, Field(ge=0.0, le=1.0)]
 Temperature = Annotated[float, Field(ge=0.0, le=2.0)]
 class SqliteSettings(BaseModel):
    model_config = ConfigDict(extra="forbid", frozen=True)
    driver: Literal["sqlite"] = "sqlite"
    path: NonEmptyStr = "app.db"
 class PostgresSettings(BaseModel):
    model_config = ConfigDict(extra="forbid", frozen=True)
    driver: Literal["postgres"] = "postgres"
    host: NonEmptyStr
    port: int = Field(default=5432, ge=1, le=65535)
    database: NonEmptyStr
    user: NonEmptyStr
    password: SecretStr
 DatabaseSettings = Annotated[
    SqliteSettings | PostgresSettings,
    Field(discriminator="driver"),
 ]
 class Settings(BaseSettings):
    model_config = SettingsConfigDict(
        env_file=".env",
        env_file_encoding="utf-8",
        extra="ignore",
        env_nested_delimiter="__",
        cli_implicit_flags=True,
        cli_kebab_case=True,
        frozen=True,
    )
    # --- NiceGUI Server ---
    host: str = "0.0.0.0"
    port: int = 8000
    log_level: Literal["critical", "error", "warning", "info", "debug", "trace"] = "info"
    reload: bool = False
    # --- AI provider ---
    provider: Provider = Provider.OPENROUTER
-    openrouter_api_key: str
+    openrouter_api_key: SecretStr
-    provider_model: str | None = None
+    provider_model: NonEmptyStr | None = None
-    openrouter_http_referer: str | None = None
+    openrouter_http_referer: NonEmptyStr | None = None
-    openrouter_app_title: str | None = None
+    openrouter_app_title: NonEmptyStr | None = None
    default_prompt_name: PromptFilename = "transcribe_document.md"
    transcription_temperature: Temperature | None = None
    transcription_top_p: Probability | None = None
    # --- runtime environment ---
    environment: Literal["development", "test", "production"] = "development"
    # --- persistence ---
-    database_url: str = "sqlite:///./transcription.db"
+    database: DatabaseSettings = Field(default_factory=SqliteSettings)
-    bootstrap_schema_on_startup: bool | None = None
+    bootstrap_schema_on_startup: bool = False
    sqlite_check_same_thread: bool = False
    # --- filesystem paths ---
    upload_dir: Path = Path("./uploads")
    prompt_dir: Path = Path("./prompts")
    artifact_dir: Path = Path("./data/artifacts")
    artifact_inline_threshold_bytes: int = Field(default=1_048_576, ge=1)
    # --- worker reliability ---
-    worker_max_retries: int = 0
+    worker_max_retries: int = Field(default=0, ge=0)
-    worker_retry_backoff_seconds: float = 0.0
+    worker_retry_backoff_seconds: float = Field(default=0.0, ge=0.0)
    worker_provider_timeout_seconds: float = Field(default=20.0, gt=0.0, le=20.0)
    worker_min_transcription_chars: int = Field(default=0, ge=0)
    worker_min_transcription_lines: int = Field(default=0, ge=0)
    worker_fail_on_finish_reason_length: bool = False
    @property
    def should_bootstrap_schema(self) -> bool:
        """Return whether startup should auto-create schema for this environment."""
-        if self.bootstrap_schema_on_startup is not None:
+        if "bootstrap_schema_on_startup" in self.model_fields_set:
            return self.bootstrap_schema_on_startup
        return self.environment in {"development", "test"}
-_settings: ContextVar[Settings | None] = ContextVar("settings", default=None)
+@cache
 def get_settings(**kwargs: Any) -> Settings:
    """Load cached settings without reading process CLI arguments."""
    return Settings(_cli_parse_args=False, **kwargs)  # pyright: ignore[reportCallIssue]
-def get_settings(**kwargs) -> Settings:
+def parse_cli_settings(args: Sequence[str] | None = None) -> Settings:
-    settings = _settings.get()
+    """Load settings with CLI arguments at the executable boundary."""
-    if settings is None:
+    cli_args = True if args is None else list(args)
-        settings = Settings(**kwargs)  # pyright: ignore[reportCallIssue]
+    return Settings(_cli_parse_args=cli_args)  # pyright: ignore[reportCallIssue]
        _settings.set(settings)
    return settings
-LOGGING_CONFIG: dict[str, object] = {
+LOGGING_CONFIG: dict[str, Any] = {
    "version": 1,
    "disable_existing_loggers": False,
    "formatters": {
@@ -100,7 +158,10 @@ LOGGING_CONFIG: dict[str, object] = {
 }
-def configure_logging() -> None:
+def configure_logging(settings: Settings | None = None) -> None:
    """Configure root logging once at startup."""
-    logging.config.dictConfig(LOGGING_CONFIG)
+    cfg = LOGGING_CONFIG.copy()
    active_settings = settings or get_settings()
    cfg["loggers"]["transcription"]["level"] = active_settings.log_level.upper()
    logging.config.dictConfig(cfg)
    logger.debug("Logging configured")
@@ -1,6 +1,15 @@
 from .operations import create_all
 from .operations import upgrade_schema
 from .runtime import dispose_database_runtime
 from .runtime import get_session
 from .runtime import initialize_database_runtime
 from .session import session_scope
 from .session import transaction_scope
-__all__ = ["create_all", "dispose_database_runtime", "get_session", "initialize_database_runtime"]
+__all__ = [
    "create_all",
    "dispose_database_runtime",
    "initialize_database_runtime",
    "session_scope",
    "transaction_scope",
    "upgrade_schema",
 ]
@@ -0,0 +1,60 @@
 from functools import cache
 from typing import Any
 from sqlalchemy import URL
 from sqlalchemy import StaticPool
 from sqlalchemy.ext.asyncio import AsyncEngine
 from sqlalchemy.ext.asyncio import create_async_engine
 from ..config import PostgresSettings
 from ..config import Settings
 from ..config import SqliteSettings
 from ..config import get_settings
 def get_database_url(settings: Settings) -> str:
    match settings.database:
        case SqliteSettings(path=path):
            url = URL.create(
                drivername="sqlite+aiosqlite",
                database=path,
            )
        case PostgresSettings() as database:
            url = URL.create(
                drivername="postgresql+asyncpg",
                host=database.host,
                port=database.port,
                database=database.database,
                username=database.user,
                password=database.password.get_secret_value(),
            )
    return url.render_as_string(hide_password=False)
 def resolve_engine(settings: Settings | None = None) -> AsyncEngine:
    active_settings = settings or get_settings()
    return get_engine(get_database_url(active_settings))
@cache
 def get_engine(database_url: str) -> AsyncEngine:
    kwargs: dict[str, Any] = {"echo": False, "pool_pre_ping": True}
    if database_url.startswith("sqlite"):
        kwargs["connect_args"] = {"check_same_thread": False}
        if ":memory:" in database_url:
            kwargs["poolclass"] = StaticPool
    return create_async_engine(database_url, **kwargs)
 async def dispose_engine(database_url: str) -> None:
    engine = get_engine(database_url)
    try:
        await engine.dispose()
    finally:
        get_engine.cache_clear()
 async def refresh_engine(database_url: str) -> AsyncEngine:
    await dispose_engine(database_url)
    return get_engine(database_url)
@@ -0,0 +1,406 @@
 """SQLModel domain models for the V3 transcription system."""
 from datetime import UTC
 from datetime import date
 from datetime import datetime
 from enum import StrEnum
 from typing import Optional
 from uuid import UUID
 from uuid import uuid4
 from pydantic import JsonValue
 from sqlalchemy import JSON
 from sqlalchemy import BigInteger
 from sqlalchemy import CheckConstraint
 from sqlalchemy import Column
 from sqlalchemy import Enum as SAEnum
 from sqlalchemy import LargeBinary
 from sqlalchemy import UniqueConstraint
 from sqlalchemy.dialects.postgresql import JSONB
 from sqlalchemy.orm.exc import DetachedInstanceError
 from sqlalchemy.types import TypeDecorator
 from sqlmodel import Field
 from sqlmodel import Relationship
 from sqlmodel import SQLModel
 class JSONBCompat(TypeDecorator):
    """JSONB for PostgreSQL and JSON for SQLite/testing backends."""
    impl = JSON(none_as_null=True)
    def load_dialect_impl(self, dialect):
        if dialect.name == "postgresql":
            return dialect.type_descriptor(JSONB(none_as_null=True))
        return dialect.type_descriptor(JSON(none_as_null=True))
 class JobStatus(StrEnum):
    QUEUED = "queued"
    PROCESSING = "processing"
    TRANSCRIBED = "transcribed"
    COMPLETED = "completed"
    PARTIAL_SUCCESS = "partial_success"
    FAILED = "failed"
 class DocumentPersonRole(StrEnum):
    AUTHOR = "author"
    RECIPIENT = "recipient"
    MENTIONED = "mentioned"
 class JobSourceStatus(StrEnum):
    PENDING = "pending"
    TRANSCRIBED = "transcribed"
    FAILED = "failed"
 class DocumentType(SQLModel, table=True):
    """Registry of allowed document types."""
    __tablename__ = "document_type"
    id: UUID = Field(default_factory=uuid4, primary_key=True)
    label: str
    normalized_label: str = Field(index=True, unique=True)
    is_active: bool = True
    created_at: datetime = Field(default_factory=lambda: datetime.now(UTC))
    updated_at: datetime = Field(default_factory=lambda: datetime.now(UTC))
    documents: list["Document"] = Relationship(
        back_populates="document_type_ref", sa_relationship_kwargs={"lazy": "selectin"}
    )
 class PersonRole(SQLModel, table=True):
    """Registry of allowed document-person relationship roles."""
    __tablename__ = "person_role"
    id: UUID = Field(default_factory=uuid4, primary_key=True)
    code: str = Field(index=True, unique=True)
    label: str
    is_active: bool = True
    created_at: datetime = Field(default_factory=lambda: datetime.now(UTC))
    updated_at: datetime = Field(default_factory=lambda: datetime.now(UTC))
    document_people: list["DocumentPerson"] = Relationship(
        back_populates="role_ref", sa_relationship_kwargs={"lazy": "selectin"}
    )
 class Document(SQLModel, table=True):
    """An historical document."""
    id: UUID = Field(default_factory=uuid4, primary_key=True)
    name: str
    document_type_id: UUID | None = Field(default=None, foreign_key="document_type.id")
    document_date: date | None = None
    document_date_raw: str | None = None
    location_created: str | None = None
    notes: str | None = None
    archive_identifier: str | None = None
    created_at: datetime = Field(default_factory=lambda: datetime.now(UTC))
    updated_at: datetime = Field(default_factory=lambda: datetime.now(UTC))
    jobs: list["Job"] = Relationship(back_populates="document", sa_relationship_kwargs={"lazy": "selectin"})
    sources: list["Source"] = Relationship(back_populates="document", sa_relationship_kwargs={"lazy": "selectin"})
    document_people: list["DocumentPerson"] = Relationship(
        back_populates="document", sa_relationship_kwargs={"lazy": "selectin"}
    )
    document_type_ref: Optional["DocumentType"] = Relationship(
        back_populates="documents", sa_relationship_kwargs={"lazy": "selectin"}
    )
 class Person(SQLModel, table=True):
    """A historical person linked to one or more documents."""
    id: UUID = Field(default_factory=uuid4, primary_key=True)
    full_name: str
    display_name: str | None = None
    maiden_name: str | None = None
    birth_date: date | None = None
    birth_date_raw: str | None = None
    birth_place: str | None = None
    death_date: date | None = None
    death_date_raw: str | None = None
    death_place: str | None = None
    biography: str | None = None
    portrait_path: str | None = None
    family_search_id: str | None = Field(default=None, unique=True)
    metadata_: dict[str, JsonValue] | None = Field(
        default=None,
        sa_column=Column("metadata", JSONBCompat(), nullable=True),
    )
    created_at: datetime = Field(default_factory=lambda: datetime.now(UTC))
    updated_at: datetime = Field(default_factory=lambda: datetime.now(UTC))
    document_people: list["DocumentPerson"] = Relationship(
        back_populates="person", sa_relationship_kwargs={"lazy": "selectin"}
    )
 class DocumentPerson(SQLModel, table=True):
    """Associates documents with people in a given role."""
    __tablename__ = "document_person"
    id: UUID = Field(default_factory=uuid4, primary_key=True)
    document_id: UUID = Field(foreign_key="document.id")
    person_id: UUID = Field(foreign_key="person.id")
    role_id: UUID | None = Field(default=None, foreign_key="person_role.id")
    role: str = Field(default=DocumentPersonRole.AUTHOR.value, nullable=False)
    created_at: datetime = Field(default_factory=lambda: datetime.now(UTC))
    updated_at: datetime = Field(default_factory=lambda: datetime.now(UTC))
    __table_args__ = (
        UniqueConstraint("document_id", "person_id", "role_id", name="uq_document_person_role_id"),
        UniqueConstraint("document_id", "person_id", "role", name="uq_document_person_role"),
    )
    document: Optional["Document"] = Relationship(
        back_populates="document_people", sa_relationship_kwargs={"lazy": "selectin"}
    )
    person: Optional["Person"] = Relationship(
        back_populates="document_people", sa_relationship_kwargs={"lazy": "selectin"}
    )
    role_ref: Optional["PersonRole"] = Relationship(
        back_populates="document_people", sa_relationship_kwargs={"lazy": "selectin"}
    )
 class Job(SQLModel, table=True):
    """A transcription job tied to a single document."""
    id: UUID = Field(default_factory=uuid4, primary_key=True)
    document_id: UUID = Field(foreign_key="document.id")
    status: JobStatus = Field(
        default=JobStatus.QUEUED,
        sa_column=Column(
            SAEnum(
                JobStatus,
                values_callable=lambda enum_cls: [item.value for item in enum_cls],
                native_enum=False,
            ),
            nullable=False,
        ),
    )
    retry_count: int = Field(default=0, ge=0)
    date_created: datetime = Field(default_factory=lambda: datetime.now(UTC))
    date_updated: datetime = Field(default_factory=lambda: datetime.now(UTC))
    provider: str | None = None
    model: str | None = None
    prompt_name: str | None = None
    prompt_hash: str | None = None
    system_prompt: str | None = None
    user_prompt: str | None = None
    temperature: float | None = None
    top_p: float | None = None
    document: Optional["Document"] = Relationship(back_populates="jobs", sa_relationship_kwargs={"lazy": "selectin"})
    job_sources: list["JobSource"] = Relationship(back_populates="job", sa_relationship_kwargs={"lazy": "selectin"})
    @property
    def filename(self) -> str:
        """Return the filename of the associated source, when available."""
        if not self.job_sources:
            return "unknown"
        for job_source in self.job_sources:
            source = job_source.__dict__.get("source")
            if source is None:
                try:
                    source = job_source.source
                except DetachedInstanceError:
                    source = None
                except Exception:  # noqa: BLE001
                    source = None
            if source is not None:
                return source.filename
        return "unknown"
    @property
    def error_detail(self) -> str | None:
        """Return the first available source-level error detail for the job."""
        if not self.job_sources:
            return None
        for job_source in self.job_sources:
            if job_source.error_detail:
                return job_source.error_detail
        return None
 class Source(SQLModel, table=True):
    """A document source image or PDF page."""
    id: UUID = Field(default_factory=uuid4, primary_key=True)
    document_id: UUID = Field(foreign_key="document.id")
    page_number: int = Field(default=1, ge=1)
    upload_name: str
    filename: str
    file_path: str
    file_hash: str
    file_size_bytes: int = Field(sa_column=Column(BigInteger(), nullable=False))
    raw_transcription: str | None = None
    revised_text: str | None = None
    date_uploaded: datetime = Field(default_factory=lambda: datetime.now(UTC))
    date_revised: datetime | None = None
    document: Optional["Document"] = Relationship(
        back_populates="sources",
        sa_relationship_kwargs={"lazy": "selectin"},
    )
    job_sources: list["JobSource"] = Relationship(
        back_populates="source",
        sa_relationship_kwargs={"lazy": "selectin"},
    )
    processing_artifacts: list["ProcessingArtifact"] = Relationship(
        back_populates="source",
        sa_relationship_kwargs={"lazy": "noload"},
    )
    @property
    def latest_job_source(self) -> Optional["JobSource"]:
        """Return the most recent job execution record for this source."""
        if not self.job_sources:
            return None
        return max(self.job_sources, key=lambda js: js.executed_at)
    @property
    def latest_status(self) -> JobSourceStatus | None:
        """Return the execution status of the latest job run."""
        latest = self.latest_job_source
        return latest.status if latest else None
    @property
    def latest_error_detail(self) -> str | None:
        """Return the error detail from the latest job run, if present."""
        latest = self.latest_job_source
        return latest.error_detail if latest else None
    @property
    def document_name(self) -> str | None:
        """Return the parent document name if loaded."""
        return self.document.name if self.document else None
 class JobSource(SQLModel, table=True):
    """A single AI execution record for one source page."""
    __tablename__ = "job_source"
    id: UUID = Field(default_factory=uuid4, primary_key=True)
    job_id: UUID = Field(foreign_key="job.id")
    source_id: UUID = Field(foreign_key="source.id")
    status: JobSourceStatus = Field(
        default=JobSourceStatus.PENDING,
        sa_column=Column(
            SAEnum(
                JobSourceStatus,
                values_callable=lambda enum_cls: [item.value for item in enum_cls],
                native_enum=False,
            ),
            nullable=False,
        ),
    )
    raw_transcription: str | None = None
    ai_metadata: dict[str, JsonValue] | None = Field(default=None, sa_column=Column(JSONBCompat(), nullable=True))
    raw_api_response: dict[str, JsonValue] | None = Field(default=None, sa_column=Column(JSONBCompat(), nullable=True))
    error_detail: str | None = None
    executed_at: datetime = Field(default_factory=lambda: datetime.now(UTC))
    job: Optional["Job"] = Relationship(back_populates="job_sources", sa_relationship_kwargs={"lazy": "selectin"})
    source: Optional["Source"] = Relationship(back_populates="job_sources", sa_relationship_kwargs={"lazy": "selectin"})
    execution_attempts: list["ExecutionAttempt"] = Relationship(
        back_populates="job_source",
        sa_relationship_kwargs={"lazy": "noload", "order_by": "ExecutionAttempt.attempt_number"},
    )
 class ExecutionAttempt(SQLModel, table=True):
    """Immutable evidence for one provider call attempt."""
    __tablename__ = "execution_attempt"
    __table_args__ = (UniqueConstraint("job_id", "source_id", "attempt_number", name="uq_execution_attempt_number"),)
    id: UUID = Field(default_factory=uuid4, primary_key=True)
    job_source_id: UUID = Field(foreign_key="job_source.id", index=True)
    job_id: UUID = Field(foreign_key="job.id", index=True)
    source_id: UUID = Field(foreign_key="source.id", index=True)
    attempt_number: int = Field(ge=1)
    status: JobSourceStatus
    provider: str
    model: str | None = None
    request_manifest: dict[str, JsonValue] | None = Field(default=None, sa_column=Column(JSONBCompat(), nullable=True))
    request_manifest_sha256: str | None = None
    request_manifest_schema_version: str | None = None
    response_received: bool = False
    transport_status_code: int | None = None
    transport_body: bytes | None = Field(default=None, sa_column=Column(LargeBinary(), nullable=True))
    transport_content_type: str | None = None
    transport_content_encoding: str | None = None
    transport_safe_headers: dict[str, JsonValue] | None = Field(
        default=None, sa_column=Column(JSONBCompat(), nullable=True)
    )
    router_request_id: str | None = None
    router_generation_id: str | None = None
    sdk_response_snapshot: dict[str, JsonValue] | None = Field(
        default=None, sa_column=Column(JSONBCompat(), nullable=True)
    )
    normalized_metadata: dict[str, JsonValue] | None = Field(
        default=None, sa_column=Column(JSONBCompat(), nullable=True)
    )
    software_context: dict[str, JsonValue] | None = Field(default=None, sa_column=Column(JSONBCompat(), nullable=True))
    raw_transcription: str | None = None
    error_category: str | None = None
    error_detail: str | None = None
    failure_phase: str | None = None
    started_at: datetime
    finished_at: datetime
    duration_ms: int = Field(ge=0)
    created_at: datetime = Field(default_factory=lambda: datetime.now(UTC))
    job_source: Optional["JobSource"] = Relationship(back_populates="execution_attempts")
    artifacts: list["ProcessingArtifact"] = Relationship(
        back_populates="execution_attempt", sa_relationship_kwargs={"lazy": "noload"}
    )
 class ProcessingArtifact(SQLModel, table=True):
    """Provider-neutral, versioned output derived from a Source."""
    __tablename__ = "processing_artifact"
    __table_args__ = (
        CheckConstraint(
            "(inline_payload IS NOT NULL AND external_reference IS NULL) OR "
            "(inline_payload IS NULL AND external_reference IS NOT NULL)",
            name="ck_processing_artifact_one_content_location",
        ),
    )
    id: UUID = Field(default_factory=uuid4, primary_key=True)
    source_id: UUID = Field(foreign_key="source.id", index=True)
    execution_attempt_id: UUID | None = Field(default=None, foreign_key="execution_attempt.id", index=True)
    artifact_type: str
    media_type: str
    schema_name: str
    schema_version: str
    producer: str
    producer_version: str
    inline_payload: dict[str, JsonValue] | None = Field(default=None, sa_column=Column(JSONBCompat(), nullable=True))
    external_reference: str | None = None
    payload_sha256: str = Field(index=True)
    byte_size: int = Field(sa_column=Column(BigInteger(), nullable=False))
    coordinate_metadata: dict[str, JsonValue] | None = Field(
        default=None, sa_column=Column(JSONBCompat(), nullable=True)
    )
    created_at: datetime = Field(default_factory=lambda: datetime.now(UTC))
    execution_attempt: Optional["ExecutionAttempt"] = Relationship(back_populates="artifacts")
    source: Optional["Source"] = Relationship(back_populates="processing_artifacts")
@@ -4,62 +4,191 @@ import logging
 from sqlalchemy import inspect
 from sqlalchemy import text
-from sqlalchemy.engine import Connection
+from sqlalchemy.ext.asyncio import AsyncConnection
 from sqlalchemy.ext.asyncio import AsyncEngine
 from sqlalchemy.ext.asyncio import async_sessionmaker
 from sqlmodel import SQLModel
 from sqlmodel import select
 from sqlmodel.ext.asyncio.session import AsyncSession
-from ..models import Job
+from .engine import resolve_engine
-from ..models import JobStatus
+from .models import DocumentType
-from .runtime import get_engine
+from .models import Job
 from .models import JobStatus
 from .models import PersonRole
 logger = logging.getLogger(__name__)
 DEFAULT_PERSON_ROLES: tuple[tuple[str, str], ...] = (
    ("author", "Author"),
    ("recipient", "Recipient"),
    ("mentioned", "Mentioned"),
 )
 DEFAULT_DOCUMENT_TYPES: tuple[str, ...] = (
    "Letter",
    "Record",
    "Memo",
    "Postcard",
    "Journal",
    "Note",
 )
 async def create_all(*, engine: AsyncEngine | None = None) -> None:
    """Create any missing tables on the selected engine."""
    # Import models so SQLModel metadata is fully registered before bootstrap.
    from transcription.db import models as _models  # noqa: F401
    active_engine = engine or resolve_engine()
    async with active_engine.begin() as connection:
        await connection.run_sync(SQLModel.metadata.create_all)
        await _upgrade_document_type_uuid_identity(connection)
        await _upgrade_person_family_search_id(connection)
        await _upgrade_v42_evidence_tables(connection)
    await seed_registry_defaults(engine=active_engine)
    logger.debug("Database schema bootstrap complete for database_url=%s", active_engine.url)
 async def upgrade_schema(*, engine: AsyncEngine | None = None) -> None:
    """Apply non-destructive additive upgrades to an existing schema."""
    active_engine = engine or resolve_engine()
    async with active_engine.begin() as connection:
        await _upgrade_document_type_uuid_identity(connection)
        await _upgrade_person_family_search_id(connection)
        await _upgrade_v42_evidence_tables(connection)
 async def _upgrade_v42_evidence_tables(connection: AsyncConnection) -> None:
    """Create the additive V4.2 evidence tables without rewriting historical rows."""
    def create_tables(sync_connection) -> None:
        SQLModel.metadata.tables["execution_attempt"].create(sync_connection, checkfirst=True)
        SQLModel.metadata.tables["processing_artifact"].create(sync_connection, checkfirst=True)
    await connection.run_sync(create_tables)
 async def _upgrade_document_type_uuid_identity(connection: AsyncConnection) -> None:
    """Backfill UUID references and retire legacy Document Type code/order columns."""
    def inspect_schema(sync_connection) -> tuple[set[str], set[str]]:
        database = inspect(sync_connection)
        tables = set(database.get_table_names())
        type_columns = (
            {column["name"] for column in database.get_columns("document_type")} if "document_type" in tables else set()
        )
        document_columns = (
            {column["name"] for column in database.get_columns("document")} if "document" in tables else set()
        )
        return type_columns, document_columns
    type_columns, document_columns = await connection.run_sync(inspect_schema)
    if not type_columns:
        return
    if "normalized_label" not in type_columns:
        await connection.execute(text("ALTER TABLE document_type ADD COLUMN normalized_label VARCHAR"))
    await connection.execute(
        text("UPDATE document_type SET normalized_label = lower(trim(label)) WHERE normalized_label IS NULL")
    )
    duplicates = (
        await connection.execute(
            text("SELECT normalized_label FROM document_type GROUP BY normalized_label HAVING count(*) > 1")
        )
    ).first()
    if duplicates is not None:
        raise RuntimeError(
            "Document Type migration requires unique labels ignoring case and whitespace; "
            f"duplicate normalized label: {duplicates[0]!r}"
        )
    if "code" in type_columns and {"document_type", "document_type_id"}.issubset(document_columns):
        await connection.execute(
            text(
                "UPDATE document SET document_type_id = ("
                "SELECT id FROM document_type WHERE "
                "lower(trim(document_type.code)) = lower(trim(document.document_type))"
                ") WHERE document_type_id IS NULL AND document_type IS NOT NULL"
            )
        )
    if connection.dialect.name == "postgresql":
        await connection.execute(text("ALTER TABLE document_type ALTER COLUMN normalized_label SET NOT NULL"))
        if "code" in type_columns:
            await connection.execute(text("ALTER TABLE document_type DROP COLUMN code CASCADE"))
        if "sort_order" in type_columns:
            await connection.execute(text("ALTER TABLE document_type DROP COLUMN sort_order"))
        if "document_type" in document_columns:
            await connection.execute(text("ALTER TABLE document DROP COLUMN document_type"))
    elif connection.dialect.name == "sqlite":
        await connection.execute(text("DROP INDEX IF EXISTS ix_document_type_code"))
        if "code" in type_columns:
            await connection.execute(text("ALTER TABLE document_type DROP COLUMN code"))
        if "sort_order" in type_columns:
            await connection.execute(text("ALTER TABLE document_type DROP COLUMN sort_order"))
        if "document_type" in document_columns:
            await connection.execute(text("ALTER TABLE document DROP COLUMN document_type"))
    await connection.execute(
        text("CREATE UNIQUE INDEX IF NOT EXISTS ix_document_type_normalized_label ON document_type (normalized_label)")
    )
 async def _upgrade_person_family_search_id(connection: AsyncConnection) -> None:
    """Add the nullable V4.1 FamilySearch field to an existing database."""
    def inspect_person(sync_connection) -> tuple[bool, bool]:
        database = inspect(sync_connection)
        if "person" not in database.get_table_names():
            return False, False
        columns = {column["name"] for column in database.get_columns("person")}
        indexes = database.get_indexes("person")
        constraints = database.get_unique_constraints("person")
        has_unique_id = any(entry.get("column_names") == ["family_search_id"] for entry in [*indexes, *constraints])
        return "family_search_id" in columns, has_unique_id
    has_column, has_unique_id = await connection.run_sync(inspect_person)
    if not has_column and not await connection.run_sync(
        lambda sync_connection: "person" in inspect(sync_connection).get_table_names()
    ):
        return
    if not has_column:
        await connection.execute(text("ALTER TABLE person ADD COLUMN family_search_id VARCHAR"))
    if not has_unique_id:
        await connection.execute(
            text("CREATE UNIQUE INDEX IF NOT EXISTS ix_person_family_search_id ON person (family_search_id)")
        )
 async def seed_registry_defaults(*, engine: AsyncEngine | None = None) -> None:
    """Seed default registry rows for role and document type taxonomies."""
    active_engine = engine or resolve_engine()
    session_factory = async_sessionmaker(active_engine, class_=AsyncSession, expire_on_commit=False)
    async with session_factory() as session:
        role_codes = set((await session.exec(select(PersonRole.code))).all())
        for code, label in DEFAULT_PERSON_ROLES:
            if code not in role_codes:
                session.add(PersonRole(code=code, label=label))
        type_labels = set((await session.exec(select(DocumentType.normalized_label))).all())
        for label in DEFAULT_DOCUMENT_TYPES:
            normalized_label = label.casefold()
            if normalized_label not in type_labels:
                session.add(DocumentType(label=label, normalized_label=normalized_label))
        await session.commit()
 async def get_next_queued_job(*, session: AsyncSession) -> Job | None:
    """Get the next queued job, if any."""
    result = await session.exec(
        select(Job)
        .where(Job.status == JobStatus.QUEUED)
-        .order_by(Job.created_at)  # pyright: ignore[reportArgumentType]
+        .order_by(Job.date_created)  # pyright: ignore[reportArgumentType]
        .limit(1)
    )  # fmt: skip
    return result.first()
 async def create_all(*, engine: AsyncEngine | None = None) -> None:
    """Create all tables on the selected engine."""
    # Import models so SQLModel metadata is fully registered before bootstrap.
    from transcription import models as _models  # noqa: F401
    active_engine = engine or get_engine()
    async with active_engine.begin() as connection:
        await connection.run_sync(SQLModel.metadata.create_all)
        await connection.run_sync(_ensure_sqlite_compat_columns)
    logger.debug("Database schema bootstrap complete for database_url=%s", active_engine.url)
 def _ensure_sqlite_compat_columns(connection: Connection) -> None:
    """Apply lightweight dev/test SQLite compatibility column patches.
    This keeps local bootstrap resilient when models evolve but no full
    migration tooling is in place yet.
    """
    if connection.engine.url.get_backend_name() != "sqlite":
        return
    inspector = inspect(connection)
    table_names = set(inspector.get_table_names())
    if "job" in table_names:
        job_columns = {column["name"] for column in inspector.get_columns("job")}
        if "retry_count" not in job_columns:
            connection.execute(text("ALTER TABLE job ADD COLUMN retry_count INTEGER NOT NULL DEFAULT 0"))
            logger.warning("Applied SQLite compatibility schema patch table=job column=retry_count default=0")
    if "transcript" in table_names:
        transcript_columns = {column["name"] for column in inspector.get_columns("transcript")}
        if "model" not in transcript_columns:
            connection.execute(text("ALTER TABLE transcript ADD COLUMN model VARCHAR NOT NULL DEFAULT 'unknown'"))
            logger.warning("Applied SQLite compatibility schema patch table=transcript column=model default=unknown")
@@ -1,18 +1,15 @@
 import logging
 from collections.abc import AsyncGenerator
 from contextlib import asynccontextmanager
 from contextvars import ContextVar
 from dataclasses import dataclass
 from functools import partial
 from sqlalchemy.ext.asyncio import AsyncEngine
 from sqlalchemy.ext.asyncio import async_sessionmaker
 from sqlalchemy.ext.asyncio import create_async_engine
 from sqlmodel.ext.asyncio.session import AsyncSession
 from sqlmodel.pool import StaticPool
 from ..config import Settings
 from ..config import get_settings
 from .engine import get_database_url
 from .engine import get_engine
 from .session import get_session_factory
 logger = logging.getLogger(__name__)
@@ -25,79 +22,42 @@ class DatabaseRuntime:
    session_factory: async_sessionmaker[AsyncSession]
-_runtime: ContextVar[DatabaseRuntime | None] = ContextVar("database_runtime", default=None)
+_runtime: DatabaseRuntime | None = None
 def get_database_runtime() -> DatabaseRuntime | None:
    """Return the process-owned database runtime."""
    return _runtime
 async def dispose_database_runtime() -> None:
    """Dispose lifespan-owned async database resources."""
-    runtime = _runtime.get()
+    global _runtime
    runtime = _runtime
    if runtime is None:
        return
    await runtime.engine.dispose()
-    _runtime.set(None)
+    _runtime = None
 def _to_async_database_url(database_url: str) -> str:
    """Normalize configured database URL to an async SQLAlchemy driver URL."""
    if database_url.startswith("sqlite://") and not database_url.startswith("sqlite+aiosqlite://"):
        return database_url.replace("sqlite://", "sqlite+aiosqlite://", 1)
    if database_url.startswith("postgresql://") and not database_url.startswith("postgresql+asyncpg://"):
        return database_url.replace("postgresql://", "postgresql+asyncpg://", 1)
    return database_url
 def _build_engine(settings: Settings) -> AsyncEngine:
    database_url = _to_async_database_url(settings.database_url)
    engine_factory = partial(
        create_async_engine,
        url=database_url,
        echo=False,
        pool_pre_ping=True,
    )
    if database_url.startswith("sqlite"):
        sqlite_connect_settings = {"check_same_thread": settings.sqlite_check_same_thread}
        engine_factory = partial(engine_factory, connect_args=sqlite_connect_settings)
        if ":memory:" in database_url:
            engine_factory = partial(engine_factory, poolclass=StaticPool)
    return engine_factory()
 def initialize_database_runtime(*, settings: Settings | None = None) -> DatabaseRuntime:
    """Initialize lifespan-owned async DB resources once per process."""
-    runtime = _runtime.get()
+    global _runtime
    active_settings = settings or get_settings()
    database_url = get_database_url(active_settings)
    runtime = _runtime
    if runtime is not None:
        runtime_url = runtime.engine.url.render_as_string(hide_password=False)
        if runtime_url != database_url:
            raise RuntimeError(
                "Database runtime is already initialized for a different database: "
                f"{runtime_url!r} != {database_url!r}"
            )
        return runtime
-    active_settings = settings or get_settings()
+    engine = get_engine(database_url)
-    engine = _build_engine(active_settings)
+    session_factory = get_session_factory(database_url)
    session_factory = async_sessionmaker(engine, class_=AsyncSession, expire_on_commit=False)
    runtime = DatabaseRuntime(engine=engine, session_factory=session_factory)
-    _runtime.set(runtime)
+    _runtime = runtime
    logger.debug("Initialized async database runtime for database_url=%s", engine.url)
    return runtime
 def get_engine(settings: Settings | None = None) -> AsyncEngine:
    """Return the current async SQLAlchemy engine."""
    runtime = _runtime.get() or initialize_database_runtime(settings=settings)
    return runtime.engine
 def get_session_factory(settings: Settings | None = None) -> async_sessionmaker[AsyncSession]:
    """Return the shared async session factory."""
    runtime = _runtime.get() or initialize_database_runtime(settings=settings)
    return runtime.session_factory
@asynccontextmanager
 async def get_session(
    *,
    settings: Settings | None = None,
    session_factory: async_sessionmaker[AsyncSession] | None = None,
 ) -> AsyncGenerator[AsyncSession]:
    """Yield a database session and ensure cleanup."""
    active_session_factory = session_factory or get_session_factory(settings)
    async with active_session_factory() as session:
        yield session
@@ -0,0 +1,105 @@
 from collections.abc import AsyncGenerator
 from contextlib import asynccontextmanager
 from functools import cache
 from typing import Annotated
 from fastapi import Depends
 from sqlalchemy.ext.asyncio import AsyncSessionTransaction
 from sqlalchemy.ext.asyncio import async_sessionmaker
 from sqlmodel.ext.asyncio.session import AsyncSession
 from ..config import Settings
 from ..config import get_settings
 from .engine import dispose_engine
 from .engine import get_database_url
 from .engine import get_engine
 type SessionFactory = async_sessionmaker[AsyncSession]
@cache
 def get_session_factory(database_url: str) -> SessionFactory:
    return async_sessionmaker(
        bind=get_engine(database_url),
        class_=AsyncSession,
        expire_on_commit=False,
    )
 def resolve_session_factory(
    database_url: str | None = None,
    *,
    settings: Settings | None = None,
 ) -> SessionFactory:
    if database_url is not None:
        return get_session_factory(database_url)
    if settings is None:
        from .runtime import get_database_runtime
        runtime = get_database_runtime()
        if runtime is not None:
            return runtime.session_factory
    return get_session_factory(get_database_url(settings or get_settings()))
 type SessionFactoryDep = Annotated[SessionFactory, Depends(resolve_session_factory)]
 async def dispose_session_factory(database_url: str) -> None:
    get_session_factory.cache_clear()
    await dispose_engine(database_url)
@asynccontextmanager
 async def session_scope(
    *,
    settings: Settings | None = None,
    database_url: str | None = None,
    session_factory: SessionFactory | None = None,
    session: AsyncSession | None = None,
 ) -> AsyncGenerator[AsyncSession]:
    if session is not None:
        yield session
        return
    active_session_factory = session_factory or resolve_session_factory(
        database_url,
        settings=settings,
    )
    async with active_session_factory() as owned_session:
        yield owned_session
 type SessionScopeDep = Annotated[AsyncSession, Depends(session_scope)]
@asynccontextmanager
 async def transaction_scope(
    *,
    settings: Settings | None = None,
    database_url: str | None = None,
    session_factory: SessionFactory | None = None,
    session: AsyncSession | AsyncSessionTransaction | None = None,
 ) -> AsyncGenerator[AsyncSession | AsyncSessionTransaction]:
    match session:
        case AsyncSession() as async_session:
            if not async_session.in_transaction():
                raise RuntimeError("A supplied session must have an active transaction")
            yield async_session
            return
        case AsyncSessionTransaction() as async_transaction:
            yield async_transaction
            return
    active_session_factory = session_factory or resolve_session_factory(
        database_url,
        settings=settings,
    )
    async with active_session_factory.begin() as owned_session:
        yield owned_session
 type TransactionScopeDep = Annotated[
    AsyncSession | AsyncSessionTransaction,
    Depends(transaction_scope),
 ]
@@ -1,81 +0,0 @@
 """SQLModel domain models for the transcription system.
 Three models capture the MVP lifecycle:
    Document  ->  one-to-many  ->  Job  ->  one-to-many  ->  Transcript
 """
 from datetime import UTC
 from datetime import datetime
 from enum import StrEnum
 from uuid import UUID
 from uuid import uuid4
 from sqlalchemy import UniqueConstraint
 from sqlmodel import Field
 from sqlmodel import Relationship
 from sqlmodel import SQLModel
 class JobStatus(StrEnum):
    QUEUED = "queued"
    PROCESSING = "processing"
    TRANSCRIBED = "transcribed"
    FAILED = "failed"
 class Document(SQLModel, table=True):
    """An uploaded document image."""
    id: UUID = Field(default_factory=uuid4, primary_key=True)
    filename: str
    file_path: str
    uploaded_at: datetime = Field(default_factory=lambda: datetime.now(UTC))
    # --- relationships ---
    jobs: list["Job"] = Relationship(back_populates="document")
 class Job(SQLModel, table=True):
    """A transcription job tied to a single document."""
    id: UUID = Field(default_factory=uuid4, primary_key=True)
    document_id: UUID = Field(foreign_key="document.id")
    status: JobStatus = Field(default=JobStatus.QUEUED)
    retry_count: int = Field(default=0, ge=0)
    created_at: datetime = Field(default_factory=lambda: datetime.now(UTC))
    updated_at: datetime = Field(default_factory=lambda: datetime.now(UTC))
    # --- relationships ---
    document: Document = Relationship(back_populates="jobs")
    transcripts: list["Transcript"] = Relationship(back_populates="job")
    @property
    def filename(self) -> str:
        """Return the filename of the associated document."""
        return self.document.filename if self.document else "unknown"
 class Transcript(SQLModel, table=True):
    """The output of a transcription job."""
    id: UUID = Field(default_factory=uuid4, primary_key=True)
    job_id: UUID = Field(foreign_key="job.id")
    """ID for the associated job."""
    revision: int = Field(default=0, ge=0)
    """Revision number for this job's transcript history, starting at 0."""
    provider: str
    """Name of the transcription provider used to generate this transcript."""
    model: str
    """Model identifier used to generate this transcript revision."""
    prompt_name: str
    """Name of the prompt used to generate this transcript."""
    text: str | None = None
    """The transcribed text. This may be None if the job failed or is still in progress."""
    error_detail: str | None = None
    """Details of any error that occurred during transcription."""
    created_at: datetime = Field(default_factory=lambda: datetime.now(UTC))
    __table_args__ = (UniqueConstraint("job_id", "revision", name="uq_transcript_job_revision"),)
    # --- relationships ---
    job: Job = Relationship(back_populates="transcripts")
@@ -6,8 +6,12 @@ from transcription.config import get_settings
 from transcription.providers.base import ProviderAuthError
 from transcription.providers.base import ProviderError
 from transcription.providers.base import ProviderResponseError
 from transcription.providers.base import TranscriptionMetadata
 from transcription.providers.base import TranscriptionProvider
 from transcription.providers.base import TranscriptionResult
 from transcription.providers.evidence import RequestManifest
 from transcription.providers.evidence import SourceEvidenceReference
 from transcription.providers.evidence import TransportEvidence
 from transcription.providers.openrouter import OpenRouterTranscriptionProvider
@@ -25,7 +29,11 @@ __all__ = [
    "ProviderAuthError",
    "ProviderError",
    "ProviderResponseError",
    "RequestManifest",
    "SourceEvidenceReference",
    "TranscriptionMetadata",
    "TranscriptionProvider",
    "TranscriptionResult",
    "TransportEvidence",
    "get_transcription_provider",
 ]
@@ -1,15 +1,33 @@
-"""Provider interfaces and shared types for transcription adapters."""
+"""Provider interfaces and validated shared contracts for transcription adapters."""
 from dataclasses import dataclass
 from typing import Protocol
 from uuid import UUID
-from ..models import Transcript
+from pydantic import BaseModel
 from pydantic import ConfigDict
 from pydantic import Field
 from pydantic import JsonValue
 from transcription.providers.evidence import RequestManifest
 from transcription.providers.evidence import SourceEvidenceReference
 from transcription.providers.evidence import TransportEvidence
 class ProviderError(RuntimeError):
    """Base error for provider failures."""
    def __init__(
        self,
        message: str,
        *,
        request_manifest: RequestManifest | None = None,
        transport_evidence: TransportEvidence | None = None,
        failure_phase: str = "provider_request",
    ) -> None:
        super().__init__(message)
        self.request_manifest = request_manifest
        self.transport_evidence = transport_evidence
        self.failure_phase = failure_phase
 class ProviderAuthError(ProviderError):
    """Raised when provider authentication fails."""
@@ -19,30 +37,80 @@ class ProviderResponseError(ProviderError):
    """Raised when provider responses are malformed or unusable."""
-@dataclass(frozen=True)
+class ProviderUsage(BaseModel):
-class TranscriptionResult:
+    """Normalized provider token accounting."""
    model_config = ConfigDict(extra="forbid", frozen=True)
    input_tokens: int | None = Field(default=None, ge=0)
    output_tokens: int | None = Field(default=None, ge=0)
    total_tokens: int | None = Field(default=None, ge=0)
 class TranscriptionMetadata(BaseModel):
    """Stable structured metadata persisted for one provider execution."""
    model_config = ConfigDict(extra="forbid", frozen=True)
    finish_reason: str | None = Field(default=None, min_length=1)
    usage: ProviderUsage | None = None
    def as_json_object(self) -> dict[str, JsonValue] | None:
        payload = self.model_dump(mode="json", exclude_none=True)
        return payload or None
 class TranscriptionResult(BaseModel):
    """Normalized output returned by any transcription provider."""
-    text: str
+    model_config = ConfigDict(extra="forbid", frozen=True, str_strip_whitespace=True)
    provider: str
    prompt_name: str
    model: str
-    def to_transcript(self, job_id: UUID, *, revision: int = 0) -> Transcript:
+    text: str = Field(min_length=1)
-        """Convert a TranscriptionResult to a Transcript model instance."""
+    provider: str = Field(min_length=1)
-        return Transcript(
+    model: str = Field(min_length=1)
-            job_id=job_id,
+    prompt_name: str | None = None
-            revision=revision,
+    prompt_hash: str | None = Field(default=None, pattern=r"^[0-9a-f]{64}$")
-            provider=self.provider,
+    system_prompt: str | None = None
-            prompt_name=self.prompt_name,
+    user_prompt: str | None = None
-            model=self.model,
+    temperature: float | None = Field(default=None, ge=0.0, le=2.0)
-            text=self.text,
+    top_p: float | None = Field(default=None, ge=0.0, le=1.0)
-        )
+    metadata: TranscriptionMetadata = Field(default_factory=TranscriptionMetadata)
    raw_api_response: dict[str, JsonValue] | None = None
    request_manifest: RequestManifest | None = None
    transport_evidence: TransportEvidence | None = None
    @property
    def finish_reason(self) -> str | None:
        return self.metadata.finish_reason
    @property
    def usage_input_tokens(self) -> int | None:
        return self.metadata.usage.input_tokens if self.metadata.usage else None
    @property
    def usage_output_tokens(self) -> int | None:
        return self.metadata.usage.output_tokens if self.metadata.usage else None
    @property
    def usage_total_tokens(self) -> int | None:
        return self.metadata.usage.total_tokens if self.metadata.usage else None
    def metadata_payload(self) -> dict[str, JsonValue] | None:
        return self.metadata.as_json_object()
 class TranscriptionProvider(Protocol):
    """Contract every transcription provider adapter must satisfy."""
-    async def transcribe(self, *, prompt_text: str, image_bytes: bytes, mime_type: str) -> TranscriptionResult:
+    async def transcribe(
        self,
        *,
        prompt_text: str,
        image_bytes: bytes,
        mime_type: str,
        temperature: float | None = None,
        top_p: float | None = None,
        source_reference: SourceEvidenceReference | None = None,
    ) -> TranscriptionResult:
        """Transcribe the provided image according to the prompt text."""
        ...
@@ -0,0 +1,154 @@
 """Versioned, provider-neutral contracts for processing evidence."""
 from __future__ import annotations
 import hashlib
 import json
 import os
 import platform
 from importlib.metadata import PackageNotFoundError
 from importlib.metadata import version
 from typing import Any
 from typing import Literal
 from uuid import UUID
 from pydantic import BaseModel
 from pydantic import ConfigDict
 from pydantic import Field
 from pydantic import JsonValue
 REQUEST_MANIFEST_SCHEMA = "transcription.request-manifest"
 REQUEST_MANIFEST_VERSION = "1"
 SOFTWARE_CONTEXT_SCHEMA = "transcription.software-context"
 SOFTWARE_CONTEXT_VERSION = "1"
 TRANSPORT_EVIDENCE_SCHEMA = "transcription.transport-evidence"
 TRANSPORT_EVIDENCE_VERSION = "1"
 CANONICAL_JSON_ALGORITHM = "transcription-canonical-json-v1"
 SAFE_RESPONSE_HEADERS = frozenset(
    {
        "content-type",
        "content-encoding",
        "date",
        "retry-after",
        "x-request-id",
        "x-openrouter-generation-id",
        "x-ratelimit-limit",
        "x-ratelimit-remaining",
        "x-ratelimit-reset",
    }
 )
 class EvidenceModel(BaseModel):
    """Strict immutable base for persisted evidence contracts."""
    model_config = ConfigDict(extra="forbid", frozen=True)
 class SourceEvidenceReference(EvidenceModel):
    """Secret-safe identity for source content used by one execution."""
    source_id: UUID
    digest_sha256: str = Field(pattern=r"^[0-9a-f]{64}$")
    byte_size: int = Field(ge=0)
    media_type: str = Field(min_length=1)
    page_number: int = Field(ge=1)
    width: int | None = Field(default=None, ge=1)
    height: int | None = Field(default=None, ge=1)
    derivative_id: UUID | None = None
    transformation: str | None = None
 class SoftwareContext(EvidenceModel):
    """Versions needed to interpret a provider execution."""
    schema_name: Literal["transcription.software-context"] = SOFTWARE_CONTEXT_SCHEMA
    schema_version: Literal["1"] = SOFTWARE_CONTEXT_VERSION
    application_version: str
    application_commit: str | None = None
    adapter_name: str
    adapter_version: str
    client_library: str
    client_library_version: str
    python_version: str
 class RequestManifest(EvidenceModel):
    """Frozen, secret-safe representation of one concrete provider request."""
    schema_name: Literal["transcription.request-manifest"] = REQUEST_MANIFEST_SCHEMA
    schema_version: Literal["1"] = REQUEST_MANIFEST_VERSION
    provider: str = Field(min_length=1)
    requested_model: str = Field(min_length=1)
    request: dict[str, JsonValue]
    source: SourceEvidenceReference
    explicitly_supplied_parameters: tuple[str, ...] = ()
    omitted_optional_parameters: tuple[str, ...] = ()
    optional_parameter_states: dict[str, Literal["omitted", "null", "value"]]
    prompt_content: str = Field(min_length=1)
    prompt_sha256: str = Field(pattern=r"^[0-9a-f]{64}$")
    timeout_seconds: float = Field(gt=0)
    retry_policy: str = Field(min_length=1)
    software: SoftwareContext
    canonicalization: Literal["transcription-canonical-json-v1"] = CANONICAL_JSON_ALGORITHM
    def canonical_bytes(self) -> bytes:
        return canonical_json_bytes(self.model_dump(mode="json"))
    def digest(self) -> str:
        return hashlib.sha256(self.canonical_bytes()).hexdigest()
 class TransportEvidence(EvidenceModel):
    """Exact response captured at the application/router HTTP boundary."""
    schema_name: Literal["transcription.transport-evidence"] = TRANSPORT_EVIDENCE_SCHEMA
    schema_version: Literal["1"] = TRANSPORT_EVIDENCE_VERSION
    response_received: bool
    status_code: int | None = Field(default=None, ge=100, le=599)
    body: bytes | None = None
    safe_headers: dict[str, str] = Field(default_factory=dict)
    content_type: str | None = None
    content_encoding: str | None = None
    request_id: str | None = None
    generation_id: str | None = None
 def canonical_json_bytes(value: Any) -> bytes:
    """Serialize JSON deterministically for evidence integrity hashes."""
    return json.dumps(
        value,
        ensure_ascii=False,
        allow_nan=False,
        sort_keys=True,
        separators=(",", ":"),
    ).encode("utf-8")
 def filter_safe_response_headers(headers: Any) -> dict[str, str]:
    """Return only explicitly allowlisted response headers."""
    return {
        str(name).lower(): str(value) for name, value in headers.items() if str(name).lower() in SAFE_RESPONSE_HEADERS
    }
 def package_version(package: str) -> str:
    """Return an installed package version without failing evidence capture."""
    try:
        return version(package)
    except PackageNotFoundError:
        return "unknown"
 def build_software_context(*, adapter_name: str, adapter_version: str, client_library: str) -> SoftwareContext:
    """Build the runtime software identity for an execution."""
    return SoftwareContext(
        application_version=package_version("transcription"),
        application_commit=os.environ.get("TRANSCRIPTION_COMMIT") or None,
        adapter_name=adapter_name,
        adapter_version=adapter_version,
        client_library=client_library,
        client_library_version=package_version(client_library),
        python_version=platform.python_version(),
    )
@@ -3,127 +3,515 @@
 from __future__ import annotations
 import base64
 import hashlib
 import json
 import logging
-from dataclasses import dataclass
+from collections.abc import AsyncIterator
 from collections.abc import Callable
 from typing import Annotated
 from typing import Any
-from typing import cast
+from typing import Literal
 import httpx
 from openrouter import OpenRouter
-from openrouter.components.chatmessages import ChatMessagesTypedDict
+from openrouter import errors as openrouter_errors
 from pydantic import BaseModel
 from pydantic import ConfigDict
 from pydantic import Field
 from pydantic import JsonValue
 from pydantic import TypeAdapter
 from pydantic import ValidationError
 from transcription.config import Settings
 from transcription.config import get_settings
 from transcription.providers.base import ProviderAuthError
 from transcription.providers.base import ProviderError
 from transcription.providers.base import ProviderResponseError
 from transcription.providers.base import ProviderUsage
 from transcription.providers.base import TranscriptionMetadata
 from transcription.providers.base import TranscriptionResult
 from transcription.providers.evidence import RequestManifest
 from transcription.providers.evidence import SourceEvidenceReference
 from transcription.providers.evidence import TransportEvidence
 from transcription.providers.evidence import build_software_context
 from transcription.providers.evidence import filter_safe_response_headers
 logger = logging.getLogger(__name__)
 DEFAULT_OPENROUTER_MODEL = "google/gemini-2.5-flash"
 OPENROUTER_ADAPTER_VERSION = "2"
-@dataclass(frozen=True)
+class _CapturingAsyncByteStream(httpx.AsyncByteStream):
-class OpenRouterRequest:
+    """Copy streamed response bytes without changing what the SDK consumes."""
    def __init__(self, stream: httpx.AsyncByteStream, on_complete: Callable[[bytes], None]):
        self._stream = stream
        self._on_complete = on_complete
    async def __aiter__(self) -> AsyncIterator[bytes]:
        content = bytearray()
        async for chunk in self._stream:
            content.extend(chunk)
            yield chunk
        self._on_complete(bytes(content))
    async def aclose(self) -> None:
        await self._stream.aclose()
 class _CapturingAsyncClient:
    """Delegate SDK HTTP calls while retaining the response before SDK parsing."""
    def __init__(self, client: httpx.AsyncClient):
        self._client = client
        self.last_response: httpx.Response | None = None
        self.last_body: bytes | None = None
    async def send(self, request: httpx.Request, **kwargs: Any) -> httpx.Response:
        response = await self._client.send(request, **kwargs)
        self.last_response = response
        try:
            self.last_body = response.content
        except httpx.ResponseNotRead:
            response.stream = _CapturingAsyncByteStream(response.stream, self._capture_body)
        return response
    def build_request(self, *args: Any, **kwargs: Any) -> httpx.Request:
        return self._client.build_request(*args, **kwargs)
    async def aclose(self) -> None:
        await self._client.aclose()
    def reset(self) -> None:
        self.last_response = None
        self.last_body = None
    def _capture_body(self, body: bytes) -> None:
        self.last_body = body
 class _ProviderModel(BaseModel):
    model_config = ConfigDict(extra="forbid", frozen=True)
 class TextContent(_ProviderModel):
    type: Literal["text"] = "text"
    text: str = Field(min_length=1)
 class ImageUrl(_ProviderModel):
    url: str = Field(min_length=1)
 class ImageContent(_ProviderModel):
    type: Literal["image_url"] = "image_url"
    image_url: ImageUrl
 class FileData(_ProviderModel):
    filename: str = Field(min_length=1)
    file_data: str = Field(min_length=1)
 class FileContent(_ProviderModel):
    type: Literal["file"] = "file"
    file: FileData
 MessageContent = Annotated[TextContent | ImageContent | FileContent, Field(discriminator="type")]
 class UserMessage(_ProviderModel):
    role: Literal["user"] = "user"
    content: tuple[MessageContent, ...] = Field(min_length=2)
 class OpenRouterRequest(_ProviderModel):
    """Normalized request payload fields for OpenRouter calls."""
-    model: str
+    model: str = Field(min_length=1)
-    messages: list[dict[str, Any]]
+    messages: tuple[UserMessage, ...] = Field(min_length=1)
    http_referer: str | None
    x_open_router_title: str | None
    temperature: float | None = Field(ge=0.0, le=2.0)
    top_p: float | None = Field(ge=0.0, le=1.0)
 class ResponseContentPart(BaseModel):
    model_config = ConfigDict(extra="allow", frozen=True)
    text: str | None = None
 class ResponseMessage(BaseModel):
    model_config = ConfigDict(extra="allow", frozen=True)
    content: str | tuple[ResponseContentPart, ...] | None = None
 class ResponseChoice(BaseModel):
    model_config = ConfigDict(extra="allow", frozen=True)
    message: ResponseMessage
    finish_reason: str | None = None
 class ResponseUsage(BaseModel):
    model_config = ConfigDict(extra="allow", frozen=True)
    prompt_tokens: int | None = Field(default=None, ge=0)
    completion_tokens: int | None = Field(default=None, ge=0)
    total_tokens: int | None = Field(default=None, ge=0)
    input_tokens: int | None = Field(default=None, ge=0)
    output_tokens: int | None = Field(default=None, ge=0)
    total: int | None = Field(default=None, ge=0)
 class OpenRouterResponse(BaseModel):
    model_config = ConfigDict(extra="allow", frozen=True)
    model: str | None = None
    choices: tuple[ResponseChoice, ...] = Field(min_length=1)
    usage: dict[str, JsonValue] | None = None
 JSON_OBJECT_ADAPTER = TypeAdapter(dict[str, JsonValue])
 class OpenRouterTranscriptionProvider:
    """Adapter that performs image transcription through OpenRouter."""
-    def __init__(self, *, settings: Settings | None = None, client: OpenRouter | None = None):
+    def __init__(
        self,
        *,
        settings: Settings | None = None,
        client: OpenRouter | None = None,
        async_client: httpx.AsyncClient | None = None,
    ):
        self._settings = settings or get_settings()
        self._model = self._settings.provider_model or DEFAULT_OPENROUTER_MODEL
-        self._client = client or OpenRouter(api_key=self._settings.openrouter_api_key)
+        self._capturing_client: _CapturingAsyncClient | None = None
        self._current_request_manifest: RequestManifest | None = None
        self._current_transport_evidence: TransportEvidence | None = None
        if client is None:
            self._capturing_client = _CapturingAsyncClient(async_client or httpx.AsyncClient(follow_redirects=True))
            client = OpenRouter(
                api_key=self._settings.openrouter_api_key.get_secret_value(),
                async_client=self._capturing_client,
            )
        self._client = client
    @property
    def model(self) -> str:
        """Return the resolved OpenRouter model slug."""
        return self._model
-    async def transcribe(self, *, prompt_text: str, image_bytes: bytes, mime_type: str) -> TranscriptionResult:
+    @property
    def current_request_manifest(self) -> RequestManifest | None:
        return self._current_request_manifest
    @property
    def current_transport_evidence(self) -> TransportEvidence | None:
        if self._current_transport_evidence is not None:
            return self._current_transport_evidence
        if self._current_request_manifest is None:
            return None
        return self._captured_transport_evidence()
    async def aclose(self) -> None:
        if self._capturing_client is not None:
            await self._capturing_client.aclose()
    async def transcribe(
        self,
        *,
        prompt_text: str,
        image_bytes: bytes,
        mime_type: str,
        temperature: float | None = None,
        top_p: float | None = None,
        source_reference: SourceEvidenceReference | None = None,
    ) -> TranscriptionResult:
        """Send prompt + image to OpenRouter and return normalized text output."""
-        request = self._build_request(prompt_text=prompt_text, image_bytes=image_bytes, mime_type=mime_type)
+        request = self._build_request(
            prompt_text=prompt_text,
            image_bytes=image_bytes,
            mime_type=mime_type,
            temperature=temperature,
            top_p=top_p,
        )
        manifest = self._build_request_manifest(
            request=request,
            prompt_text=prompt_text,
            source_reference=source_reference,
            temperature=temperature,
            top_p=top_p,
        )
        self._current_request_manifest = manifest
        self._current_transport_evidence = None
        if self._capturing_client is not None:
            self._capturing_client.reset()
        try:
            response = await self._client.chat.send_async(
-                messages=cast(list[ChatMessagesTypedDict], request.messages),
+                **request.model_dump(mode="json", exclude_none=True),
-                model=request.model,
+                retries=None,
                http_referer=request.http_referer,
                x_open_router_title=request.x_open_router_title,
            )
        except Exception as exc:
-            message = str(exc).lower()
+            transport = self._captured_transport_evidence()
-            if "401" in message or "auth" in message or "api key" in message:
+            self._current_transport_evidence = transport
-                raise ProviderAuthError("OpenRouter authentication failed") from exc
+            if isinstance(exc, openrouter_errors.UnauthorizedResponseError):
-            raise ProviderError("OpenRouter request failed") from exc
+                raise ProviderAuthError(
                    "OpenRouter authentication failed",
                    request_manifest=manifest,
                    transport_evidence=transport,
                    failure_phase="http_response" if transport.response_received else "connection",
                ) from exc
            failure_phase = (
                "response_validation"
                if isinstance(exc, openrouter_errors.ResponseValidationError)
                else "http_response"
                if transport.response_received
                else "connection"
            )
            raise ProviderError(
                self._transport_error_message(transport),
                request_manifest=manifest,
                transport_evidence=transport,
                failure_phase=failure_phase,
            ) from exc
-        text = self._extract_text(response)
+        transport = self._captured_transport_evidence()
-        model = self._get_optional_attr(response, "model") or self.model
+        self._current_transport_evidence = transport
        raw_api_response = self._coerce_raw_response(response)
        try:
            validated_response = OpenRouterResponse.model_validate(raw_api_response)
        except ValidationError as exc:
            raise ProviderResponseError(
                "OpenRouter response failed schema validation",
                request_manifest=manifest,
                transport_evidence=transport,
                failure_phase="response_validation",
            ) from exc
        try:
            text = self._extract_text(validated_response)
        except ProviderResponseError as exc:
            raise ProviderResponseError(
                str(exc),
                request_manifest=manifest,
                transport_evidence=transport,
                failure_phase="response_validation",
            ) from exc
        model = validated_response.model or self.model
        metadata = self._build_metadata(validated_response)
        logger.info("OpenRouter transcription completed using model=%s", model)
-        return TranscriptionResult(text=text, provider="openrouter", prompt_name="", model=model)
+        return TranscriptionResult(
            text=text,
            provider="openrouter",
            prompt_name=None,
            prompt_hash=None,
            system_prompt=None,
            user_prompt=prompt_text,
            temperature=temperature,
            top_p=top_p,
            model=model,
            metadata=metadata,
            raw_api_response=raw_api_response,
            request_manifest=manifest,
            transport_evidence=transport,
        )
-    def _build_request(self, *, prompt_text: str, image_bytes: bytes, mime_type: str) -> OpenRouterRequest:
+    def _build_request_manifest(
        self,
        *,
        request: OpenRouterRequest,
        prompt_text: str,
        source_reference: SourceEvidenceReference | None,
        temperature: float | None,
        top_p: float | None,
    ) -> RequestManifest | None:
        if source_reference is None:
            return None
        request_payload = request.model_dump(mode="json", exclude_none=True)
        sanitized_request = self._replace_embedded_media(request_payload, source_reference=source_reference)
        explicit = tuple(name for name, value in (("temperature", temperature), ("top_p", top_p)) if value is not None)
        omitted = tuple(name for name in ("temperature", "top_p") if name not in explicit)
        return RequestManifest(
            provider="openrouter",
            requested_model=self.model,
            request=JSON_OBJECT_ADAPTER.validate_python(sanitized_request),
            source=source_reference,
            explicitly_supplied_parameters=explicit,
            omitted_optional_parameters=omitted,
            optional_parameter_states={
                "temperature": "value" if temperature is not None else "omitted",
                "top_p": "value" if top_p is not None else "omitted",
            },
            prompt_content=prompt_text,
            prompt_sha256=hashlib.sha256(prompt_text.encode("utf-8")).hexdigest(),
            timeout_seconds=self._settings.worker_provider_timeout_seconds,
            retry_policy="application-bounded; sdk-retries=0",
            software=build_software_context(
                adapter_name="openrouter",
                adapter_version=OPENROUTER_ADAPTER_VERSION,
                client_library="openrouter",
            ),
        )
    def _replace_embedded_media(
        self,
        value: Any,
        *,
        source_reference: SourceEvidenceReference,
    ) -> Any:
        if isinstance(value, str) and value.startswith("data:") and ";base64," in value:
            return {
                "source_reference": source_reference.model_dump(mode="json"),
                "embedded_media_omitted": True,
            }
        if isinstance(value, dict):
            return {
                str(key): self._replace_embedded_media(item, source_reference=source_reference)
                for key, item in value.items()
            }
        if isinstance(value, list | tuple):
            return [self._replace_embedded_media(item, source_reference=source_reference) for item in value]
        return value
    def _captured_transport_evidence(self) -> TransportEvidence:
        response = self._capturing_client.last_response if self._capturing_client is not None else None
        if response is None:
            return TransportEvidence(response_received=False)
        headers = filter_safe_response_headers(response.headers)
        body = self._capturing_client.last_body if self._capturing_client is not None else None
        return TransportEvidence(
            response_received=True,
            status_code=response.status_code,
            body=body,
            safe_headers=headers,
            content_type=headers.get("content-type"),
            content_encoding=headers.get("content-encoding"),
            request_id=headers.get("x-request-id"),
            generation_id=headers.get("x-openrouter-generation-id"),
        )
    @staticmethod
    def _transport_error_message(transport: TransportEvidence) -> str:
        message = "OpenRouter request failed"
        if transport.status_code is not None:
            message += f" with HTTP {transport.status_code}"
        if transport.body is None:
            return message
        try:
            payload = json.loads(transport.body)
        except (UnicodeDecodeError, json.JSONDecodeError):
            return message
        if not isinstance(payload, dict):
            return message
        error = payload.get("error")
        detail = error.get("message") if isinstance(error, dict) else None
        if isinstance(detail, str) and detail.strip():
            return f"{message}: {detail.strip()[:500]}"
        return message
    def _build_metadata(self, response: OpenRouterResponse) -> TranscriptionMetadata:
        choice = response.choices[0]
        finish_reason = choice.finish_reason.strip() if choice.finish_reason and choice.finish_reason.strip() else None
        normalized_usage = None
        if response.usage is not None:
            try:
                usage = ResponseUsage.model_validate(response.usage)
            except ValidationError as exc:
                logger.warning("Ignoring invalid OpenRouter usage metadata: %s", exc)
            else:
                normalized_usage = ProviderUsage(
                    input_tokens=usage.prompt_tokens if usage.prompt_tokens is not None else usage.input_tokens,
                    output_tokens=usage.completion_tokens
                    if usage.completion_tokens is not None
                    else usage.output_tokens,
                    total_tokens=usage.total_tokens if usage.total_tokens is not None else usage.total,
                )
                if normalized_usage.model_dump(exclude_none=True) == {}:
                    normalized_usage = None
        return TranscriptionMetadata(finish_reason=finish_reason, usage=normalized_usage)
    def _coerce_raw_response(self, response: Any) -> dict[str, JsonValue]:
        payload = self._to_json_compatible(response)
        try:
            return JSON_OBJECT_ADAPTER.validate_python(payload)
        except ValidationError as exc:
            raise ProviderResponseError("OpenRouter response is not a JSON object") from exc
    def _to_json_compatible(self, value: Any) -> Any:
        if value is None or isinstance(value, str | int | float | bool):
            return value
        if isinstance(value, dict):
            return {str(key): self._to_json_compatible(item) for key, item in value.items()}
        if isinstance(value, list | tuple | set):
            return [self._to_json_compatible(item) for item in value]
        for method_name in ("model_dump", "to_dict"):
            serializer = getattr(value, method_name, None)
            if callable(serializer):
                try:
                    serialized = serializer(mode="json") if method_name == "model_dump" else serializer()
                    return self._to_json_compatible(serialized)
                except (TypeError, ValueError) as exc:
                    logger.debug("OpenRouter response serializer %s failed: %s", method_name, exc)
                    continue
        object_dict = getattr(value, "__dict__", None)
        if isinstance(object_dict, dict):
            return {
                str(key): self._to_json_compatible(item)
                for key, item in object_dict.items()
                if not str(key).startswith("_")
            }
        raise ProviderResponseError(f"OpenRouter response contains unsupported value type: {type(value).__name__}")
    def _build_request(
        self,
        *,
        prompt_text: str,
        image_bytes: bytes,
        mime_type: str,
        temperature: float | None,
        top_p: float | None,
    ) -> OpenRouterRequest:
        image_b64 = base64.b64encode(image_bytes).decode("ascii")
        data_url = f"data:{mime_type};base64,{image_b64}"
-
+        media_content: ImageContent | FileContent
-        messages: list[dict[str, Any]] = [
+        if mime_type == "application/pdf":
-            {
+            media_content = FileContent(file=FileData(filename="source.pdf", file_data=data_url))
-                "role": "user",
+        else:
-                "content": [
+            media_content = ImageContent(image_url=ImageUrl(url=data_url))
                    {"type": "text", "text": prompt_text},
                    {"type": "image_url", "image_url": {"url": data_url}},
                ],
            }
        ]
        return OpenRouterRequest(
            model=self.model,
-            messages=messages,
+            messages=(UserMessage(content=(TextContent(text=prompt_text), media_content)),),
            http_referer=self._settings.openrouter_http_referer,
            x_open_router_title=self._settings.openrouter_app_title,
            temperature=temperature,
            top_p=top_p,
        )
-    def _extract_text(self, response: Any) -> str:
+    def _extract_text(self, response: OpenRouterResponse) -> str:
-        choices = self._get_optional_attr(response, "choices")
+        content = response.choices[0].message.content
        if not choices:
            raise ProviderResponseError("OpenRouter response missing choices")
        first_choice = choices[0]
        message = self._get_optional_attr(first_choice, "message")
        if message is None:
            raise ProviderResponseError("OpenRouter response missing assistant message")
        content = self._get_optional_attr(message, "content")
        text = self._normalize_content(content)
        if not text:
            raise ProviderResponseError("OpenRouter response contained no transcription text")
        return text
-    def _normalize_content(self, content: Any) -> str:
+    def _normalize_content(self, content: str | tuple[ResponseContentPart, ...] | None) -> str:
        if isinstance(content, str):
            return content.strip()
-        if isinstance(content, list):
+        if isinstance(content, tuple):
-            parts: list[str] = []
+            parts = [item.text.strip() for item in content if item.text and item.text.strip()]
            for item in content:
                text_part = None
                text_part = item.get("text") if isinstance(item, dict) else self._get_optional_attr(item, "text")
                if isinstance(text_part, str) and text_part.strip():
                    parts.append(text_part.strip())
            return "\n".join(parts).strip()
        return ""
    @staticmethod
    def _get_optional_attr(obj: Any, key: str) -> Any:
        if obj is None:
            return None
        if isinstance(obj, dict):
            return obj.get(key)
        return getattr(obj, key, None)
@@ -5,9 +5,11 @@ from dataclasses import field
 from .documents import DocumentService
 from .jobs import JobService
-from .transcription import TranscriptionService
+from .people import PeopleService
 from .prompts import PromptStore
 from .sources import SourceService
-__all__ = ["DocumentService", "JobService", "ServiceBundle", "TranscriptionService"]
+__all__ = ["DocumentService", "JobService", "PeopleService", "PromptStore", "ServiceBundle", "SourceService"]
@dataclass(frozen=True, slots=True)
@@ -15,5 +17,6 @@ class ServiceBundle:
    """Container for all service instances."""
    documents: DocumentService = field(default_factory=DocumentService)
    sources: SourceService = field(default_factory=SourceService)
    jobs: JobService = field(default_factory=JobService)
-    transcriptions: TranscriptionService = field(default_factory=TranscriptionService)
+    people: PeopleService = field(default_factory=PeopleService)
@@ -8,7 +8,8 @@ from sqlmodel.ext.asyncio.session import AsyncSession
 from ..config import Settings
 from ..config import get_settings
-from ..db.runtime import get_session_factory
+from ..db.session import resolve_session_factory
 from ..db.session import session_scope
 class ServiceBase(ABC):
@@ -22,21 +23,20 @@ class ServiceBase(ABC):
        self,
        session_factory: async_sessionmaker[AsyncSession] | None = None,
        queue: asyncio.Queue | None = None,
        settings: Settings | None = None,
    ):
-        self.settings = get_settings()
+        self.settings = settings or get_settings()
-        self.session_factory = session_factory or get_session_factory()
+        self.session_factory = session_factory or resolve_session_factory(settings=self.settings)
        self.queue = queue or asyncio.Queue()
    @asynccontextmanager
    async def _session_scope(self, session: AsyncSession | None = None):
        """Provide a transactional scope around a series of operations."""
-        if session is not None:
+        async with session_scope(
-            # Reuse the provided session if one is passed in
+            session_factory=self.session_factory,
-            yield session
+            session=session,
-        else:
+        ) as active_session:
-            # Otherwise, create a new session for this scope
+            yield active_session
            async with self.session_factory() as new_session:
                yield new_session
    async def _finalize(
        self,
@@ -1,17 +1,23 @@
 import logging
 import shutil
 from collections.abc import Sequence
 from dataclasses import dataclass
-from pathlib import Path
+from datetime import UTC
 from datetime import datetime
 from uuid import UUID
 from sqlalchemy import func
 from sqlalchemy.exc import IntegrityError
 from sqlalchemy.orm import selectinload
 from sqlmodel import col
 from sqlmodel import select
 from sqlmodel.ext.asyncio.session import AsyncSession
 from ..db.models import Document
 from ..db.models import DocumentPerson
 from ..db.models import DocumentType
 from ..errors import AppError
 from ..errors import ErrorCategory
 from ..models import Document
 from .base import ServiceBase
 logger = logging.getLogger(__name__)
@@ -21,31 +27,72 @@ class DocumentError(AppError):
    """Raised when document operations fail."""
-class MissingImageError(DocumentError):
+class MissingSourceError(DocumentError):
-    """Raised when a required image is missing."""
+    """Raised when a document has no associated sources."""
 class UploadError(DocumentError):
    """Raised when uploaded content cannot be persisted safely."""
 class DocumentAlreadyExistsError(DocumentError):
-    """Raised when a document with the same filename already exists in the database."""
+    """Raised when a document with the same name already exists in the database."""
-@dataclass(frozen=True)
+class DocumentDeleteBlockedError(DocumentError):
-class UploadJobResult:
+    """Raised when a document delete is blocked by dependent records."""
    """Summary of created upload records."""
-    document_id: UUID
+
-    job_id: UUID
+class DocumentTypeError(DocumentError):
-    stored_path: Path
+    """Raised when Document Type maintenance fails."""
-    original_filename: str
+
 def _normalize_registry_label(label: str) -> str:
    normalized = label.strip()
    if not normalized:
        raise DocumentTypeError(
            "Document Type label is required",
            category=ErrorCategory.VALIDATION,
            suggestion="Enter a user-facing label and retry.",
        )
    return normalized
 def _document_type_label_key(label: str) -> str:
    return _normalize_registry_label(label).casefold()
@dataclass(frozen=True, slots=True)
 class DocumentTypeSummary:
    """Settings read model for a Document Type and its usage count."""
    id: UUID
    label: str
    is_active: bool
    document_count: int
 class DocumentService(ServiceBase):
    """Thin service class for managing documents in the database."""
    async def _validate_document_type(self, *, session: AsyncSession, document: Document) -> None:
        """Validate the UUID-backed Document Type reference."""
        if document.document_type_id is None:
            return
        if await session.get(DocumentType, document.document_type_id) is None:
            raise DocumentError(
                f"Document type with id {document.document_type_id} not found",
                category=ErrorCategory.VALIDATION,
                suggestion="Select a valid document type and retry.",
            )
    async def _get_document_or_raise(self, *, session: AsyncSession, document_id: UUID) -> Document:
        """Get a document by id or raise a not-found service error."""
        document = await session.get(Document, document_id)
        if document is None:
            raise DocumentError(
                f"Document with id {document_id} not found",
                category=ErrorCategory.NOT_FOUND,
                suggestion="Verify the document id and retry.",
            )
        return document
    #
    # CRUD Operations
    #
@@ -58,6 +105,7 @@ class DocumentService(ServiceBase):
    ) -> Document:
        """Create a new document in the database."""
        async with self._session_scope(session) as _session:
            await self._validate_document_type(session=_session, document=document)
            _session.add(document)
            try:
                await self._finalize(session=_session, caller_session=session, refresh=(document,))
@@ -72,13 +120,16 @@ class DocumentService(ServiceBase):
    async def read_document(self, document_id: UUID, *, session: AsyncSession | None = None) -> Document:
        """Read an existing document from the database.
-        The selectinload option is used to eagerly load related jobs for the document.
+        The selectinload option is used to eagerly load related jobs and sources.
        """
        async with self._session_scope(session) as _session:
            document = await _session.get(
                Document,
                document_id,
-                options=(selectinload(Document.jobs),),  # pyright: ignore[reportArgumentType]
+                options=(
                    selectinload(Document.jobs),  # pyright: ignore[reportArgumentType]
                    selectinload(Document.sources),  # pyright: ignore[reportArgumentType]
                ),
            )
        if document is None:
            raise DocumentError(
@@ -86,42 +137,304 @@ class DocumentService(ServiceBase):
                category=ErrorCategory.NOT_FOUND,
                suggestion="Re-upload the source document and retry.",
            )
-        elif not Path(document.file_path).exists():
+        elif not document.sources:
-            raise MissingImageError(
+            raise MissingSourceError(
-                f"Document with id {document_id} is missing its image file in {self.settings.upload_dir:!s}",
+                f"Document with id {document_id} has no associated source records",
                category=ErrorCategory.NOT_FOUND,
-                suggestion="Re-upload the source document and retry.",
+                suggestion="Upload at least one source for this document and retry.",
            )
        return document
    async def update_document(self, document: Document, *, session: AsyncSession | None = None) -> Document:
        """Update an existing document in the database."""
        async with self._session_scope(session) as _session:
            await self._validate_document_type(session=_session, document=document)
            document.updated_at = datetime.now(UTC)
            merged = await _session.merge(document)
            await self._finalize(session=_session, caller_session=session, refresh=(merged,))
            return merged
    async def delete_document(self, document: Document, *, session: AsyncSession | None = None) -> None:
        """Delete a document from the database."""
        document_id = document.id
        async with self._session_scope(session) as _session:
-            await _session.delete(document)
+            existing = await _session.get(
                Document,
                document.id,
                options=(
                    selectinload(Document.jobs),  # pyright: ignore[reportArgumentType]
                    selectinload(Document.sources),  # pyright: ignore[reportArgumentType]
                    selectinload(Document.document_people),  # pyright: ignore[reportArgumentType]
                ),
            )
            if existing is None:
                raise DocumentError(
                    f"Document with id {document.id} not found",
                    category=ErrorCategory.NOT_FOUND,
                    suggestion="Verify the document id and retry.",
                )
            has_jobs = bool(existing.jobs)
            has_sources = bool(existing.sources)
            if has_jobs or has_sources:
                blocked_by: list[str] = []
                if has_sources:
                    blocked_by.append("Sources")
                if has_jobs:
                    blocked_by.append("Jobs")
                raise DocumentDeleteBlockedError(
                    f"Document delete blocked by related records: {', '.join(blocked_by)}",
                    category=ErrorCategory.VALIDATION,
                    suggestion="Remove related Sources and Jobs first, then retry deletion.",
                )
            for link in list(existing.document_people):
                await _session.delete(link)
            await _session.delete(existing)
            await self._finalize(session=_session, caller_session=session)
        self._delete_document_storage_folder(document_id=document_id)
    def _delete_document_storage_folder(self, *, document_id: UUID) -> None:
        """Best-effort cleanup for document-scoped source storage."""
        document_dir = self.settings.upload_dir / "documents" / str(document_id)
        if not document_dir.exists():
            return
        try:
            shutil.rmtree(document_dir)
            logger.info("Deleted document storage folder: %s", document_dir)
        except OSError:
            logger.warning("Failed to delete document storage folder: %s", document_dir)
    # Query Operations
    async def query_documents(
-        self, *, filename: str | None = None, session: AsyncSession | None = None
+        self, *, name: str | None = None, session: AsyncSession | None = None
    ) -> Sequence[Document]:
        """Query documents from the database based on provided filters."""
        async with self._session_scope(session) as _session:
            query = select(Document)
-            if filename is not None:
+            if name is not None:
-                query = query.where(Document.filename == filename)
+                query = query.where(Document.name == name)
            result = await _session.exec(query)
            return result.all()
    async def list_documents(self, *, session: AsyncSession | None = None) -> Sequence[Document]:
-        """List all documents in the database."""
+        """List documents with relations needed by the archival table."""
        async with self._session_scope(session) as _session:
-            result = await _session.exec(select(Document))
+            query = select(Document).options(
                selectinload(Document.document_people).selectinload(DocumentPerson.person),  # pyright: ignore[reportArgumentType]
                selectinload(Document.document_people).selectinload(DocumentPerson.role_ref),  # pyright: ignore[reportArgumentType]
                selectinload(Document.document_type_ref),  # pyright: ignore[reportArgumentType]
            )
            result = await _session.exec(query)
            return result.all()
    async def read_document_detail(self, document_id: UUID, *, session: AsyncSession | None = None) -> Document:
        """Read a document with eagerly loaded relations for UI detail rendering."""
        async with self._session_scope(session) as _session:
            query = (
                select(Document)
                .options(
                    selectinload(Document.jobs),  # pyright: ignore[reportArgumentType]
                    selectinload(Document.sources),  # pyright: ignore[reportArgumentType]
                    selectinload(Document.document_people).selectinload(DocumentPerson.person),  # pyright: ignore[reportArgumentType]
                    selectinload(Document.document_people).selectinload(DocumentPerson.role_ref),  # pyright: ignore[reportArgumentType]
                    selectinload(Document.document_type_ref),  # pyright: ignore[reportArgumentType]
                )
                .where(Document.id == document_id)
                .execution_options(populate_existing=True)
            )
            document = (await _session.exec(query)).first()
        if document is None:
            raise DocumentError(
                f"Document with id {document_id} not found",
                category=ErrorCategory.NOT_FOUND,
                suggestion="Verify the document id and retry.",
            )
        return document
    async def list_document_types(
        self,
        *,
        active_only: bool = True,
        session: AsyncSession | None = None,
    ) -> Sequence[DocumentType]:
        """List configured document types."""
        async with self._session_scope(session) as _session:
            query = select(DocumentType)
            if active_only:
                query = query.where(col(DocumentType.is_active).is_(True))
            query = query.order_by(col(DocumentType.normalized_label), col(DocumentType.id))
            result = await _session.exec(query)
            return result.all()
    async def list_document_type_summaries(
        self,
        *,
        session: AsyncSession | None = None,
    ) -> Sequence[DocumentTypeSummary]:
        """List Document Types alphabetically with current usage counts."""
        async with self._session_scope(session) as _session:
            query = (
                select(DocumentType, func.count(col(Document.id)))
                .outerjoin(Document, col(Document.document_type_id) == col(DocumentType.id))
                .group_by(col(DocumentType.id))
                .order_by(col(DocumentType.normalized_label), col(DocumentType.id))
            )
            rows = (await _session.exec(query)).all()
            return [
                DocumentTypeSummary(
                    id=document_type.id,
                    label=document_type.label,
                    is_active=document_type.is_active,
                    document_count=int(document_count),
                )
                for document_type, document_count in rows
            ]
    async def create_document_type(
        self,
        *,
        label: str,
        is_active: bool = True,
        session: AsyncSession | None = None,
    ) -> DocumentType:
        """Create a UUID-identified Document Type with a unique label."""
        document_type = DocumentType(
            label=_normalize_registry_label(label),
            normalized_label=_document_type_label_key(label),
            is_active=is_active,
        )
        async with self._session_scope(session) as _session:
            _session.add(document_type)
            try:
                await self._finalize(session=_session, caller_session=session, refresh=(document_type,))
            except IntegrityError as exc:
                raise DocumentTypeError(
                    f"Document Type label {document_type.label!r} already exists",
                    category=ErrorCategory.CONFLICT,
                    suggestion="Choose a different label or edit the existing type.",
                ) from exc
            return document_type
    async def read_document_type(
        self,
        document_type_id: UUID,
        *,
        session: AsyncSession | None = None,
    ) -> DocumentType:
        """Read a Document Type by id."""
        async with self._session_scope(session) as _session:
            document_type = await _session.get(DocumentType, document_type_id)
        if document_type is None:
            raise DocumentTypeError(
                f"Document Type with id {document_type_id} not found",
                category=ErrorCategory.NOT_FOUND,
                suggestion="Refresh Settings and select an available Document Type.",
            )
        return document_type
    async def update_document_type(
        self,
        document_type_id: UUID,
        *,
        label: str,
        is_active: bool,
        session: AsyncSession | None = None,
    ) -> DocumentType:
        """Update a Document Type label and active state."""
        async with self._session_scope(session) as _session:
            document_type = await _session.get(DocumentType, document_type_id)
            if document_type is None:
                raise DocumentTypeError(
                    f"Document Type with id {document_type_id} not found",
                    category=ErrorCategory.NOT_FOUND,
                    suggestion="Refresh Settings and select an available Document Type.",
                )
            document_type.label = _normalize_registry_label(label)
            document_type.normalized_label = _document_type_label_key(label)
            document_type.is_active = is_active
            document_type.updated_at = datetime.now(UTC)
            try:
                await self._finalize(session=_session, caller_session=session, refresh=(document_type,))
            except IntegrityError as exc:
                raise DocumentTypeError(
                    f"Document Type label {document_type.label!r} already exists",
                    category=ErrorCategory.CONFLICT,
                    suggestion="Choose a different label or edit the existing type.",
                ) from exc
            return document_type
    async def delete_document_type(
        self,
        document_type_id: UUID,
        *,
        session: AsyncSession | None = None,
    ) -> None:
        """Delete an unreferenced Document Type without cascade behavior."""
        async with self._session_scope(session) as _session:
            document_type = await _session.get(DocumentType, document_type_id)
            if document_type is None:
                raise DocumentTypeError(
                    f"Document Type with id {document_type_id} not found",
                    category=ErrorCategory.NOT_FOUND,
                    suggestion="Refresh Settings and select an available Document Type.",
                )
            if await self._document_type_is_referenced(session=_session, document_type=document_type):
                raise DocumentTypeError(
                    f"Document Type {document_type.label!r} is referenced and cannot be deleted",
                    category=ErrorCategory.CONFLICT,
                    suggestion="Deactivate the type instead; historical Documents will retain it.",
                )
            await _session.delete(document_type)
            await self._finalize(session=_session, caller_session=session)
    async def is_document_type_referenced(
        self,
        document_type_id: UUID,
        *,
        session: AsyncSession | None = None,
    ) -> bool:
        """Return whether a Document references a Document Type."""
        async with self._session_scope(session) as _session:
            document_type = await _session.get(DocumentType, document_type_id)
            if document_type is None:
                raise DocumentTypeError(
                    f"Document Type with id {document_type_id} not found",
                    category=ErrorCategory.NOT_FOUND,
                    suggestion="Refresh Settings and select an available Document Type.",
                )
            return await self._document_type_is_referenced(
                session=_session,
                document_type=document_type,
            )
    @staticmethod
    async def _document_type_is_referenced(
        *,
        session: AsyncSession,
        document_type: DocumentType,
    ) -> bool:
        reference = (
            await session.exec(select(Document.id).where(Document.document_type_id == document_type.id))
        ).first()
        return reference is not None
    async def set_document_type(
        self,
        *,
        document_id: UUID,
        document_type_id: UUID,
        session: AsyncSession | None = None,
    ) -> Document:
        """Set a Document Type by UUID."""
        async with self._session_scope(session) as _session:
            document = await self._get_document_or_raise(session=_session, document_id=document_id)
            document.document_type_id = document_type_id
            await self._validate_document_type(session=_session, document=document)
            document.updated_at = datetime.now(UTC)
            await self._finalize(session=_session, caller_session=session, refresh=(document,))
            return document
@@ -1,16 +1,44 @@
 import logging
 from collections.abc import Sequence
 from datetime import UTC
 from datetime import datetime
 from pathlib import Path
 from uuid import UUID
 from sqlalchemy import func
 from sqlalchemy.orm import selectinload
 from sqlmodel import select
 from sqlmodel.ext.asyncio.session import AsyncSession
-from ..models import Job
+from ..db.models import ExecutionAttempt
-from ..models import JobStatus
+from ..db.models import Job
 from ..db.models import JobSource
 from ..db.models import JobSourceStatus
 from ..db.models import JobStatus
 from ..db.models import ProcessingArtifact
 from ..db.models import Source
 from ..errors import AppError
 from ..errors import ErrorCategory
 from .base import ServiceBase
 logger = logging.getLogger(__name__)
 class JobDeleteBlockedError(AppError):
    """Raised when a job delete operation is blocked by lifecycle policy."""
 class JobCancelBlockedError(AppError):
    """Raised when a job cancel operation is blocked by lifecycle policy."""
 class JobResubmitBlockedError(AppError):
    """Raised when a job resubmit operation is blocked by lifecycle policy."""
 class JobNotFoundError(AppError):
    """Raised when a requested Job does not exist."""
 class JobService(ServiceBase):
    """Thin service class for managing jobs in the database."""
@@ -37,14 +65,14 @@ class JobService(ServiceBase):
                select(Job)
                .options(
                    selectinload(Job.document),  # pyright: ignore[reportArgumentType]
-                    selectinload(Job.transcripts),  # pyright: ignore[reportArgumentType]
+                    selectinload(Job.job_sources).selectinload(JobSource.source),  # pyright: ignore[reportArgumentType]
                )
                .where(Job.id == job_id)
                .execution_options(populate_existing=True)
            )
            job = (await _session.exec(query)).first()
        if job is None:
-            raise ValueError(f"Job with id {job_id} not found")
+            raise self._not_found(job_id)
        return job
    async def update_job(self, job: Job, session: AsyncSession | None = None) -> Job:
@@ -71,11 +99,14 @@ class JobService(ServiceBase):
    ) -> Sequence[Job]:
        """Query jobs from the database based on provided filters."""
        async with self._session_scope(session) as _session:
-            query = select(Job).options(selectinload(Job.document))  # pyright: ignore[reportArgumentType]
+            query = select(Job).options(
                selectinload(Job.document),  # pyright: ignore[reportArgumentType]
                selectinload(Job.job_sources).selectinload(JobSource.source),  # pyright: ignore[reportArgumentType]
            )
            if status is not None:
                query = query.where(Job.status == status)
            if filename is not None:
-                query = query.where(Job.document.filename == filename)
+                query = query.where(Job.job_sources.any(JobSource.source.has(Source.filename == filename)))
            result = await _session.exec(query)
            return result.all()
@@ -88,7 +119,10 @@ class JobService(ServiceBase):
        """List all jobs in the database with eagerly loaded documents."""
        _ = load_docs
        async with self._session_scope(session) as _session:
-            query = select(Job).options(selectinload(Job.document))  # pyright: ignore[reportArgumentType]
+            query = select(Job).options(
                selectinload(Job.document),  # pyright: ignore[reportArgumentType]
                selectinload(Job.job_sources).selectinload(JobSource.source),  # pyright: ignore[reportArgumentType]
            )
            result = await _session.exec(query)
            return result.all()
@@ -125,11 +159,11 @@ class JobService(ServiceBase):
            )
            job = (await _session.exec(query)).first()
            if job is None:
-                raise ValueError(f"Job with id {job_id} not found")
+                raise self._not_found(job_id)
            job.status = status
            if retry_count_increment:
                job.retry_count += retry_count_increment
-            job.updated_at = datetime.now(UTC)
+            job.date_updated = datetime.now(UTC)
            await self._finalize(session=_session, caller_session=session, refresh=(job,))
            return job
@@ -142,8 +176,247 @@ class JobService(ServiceBase):
        async with self._session_scope(session) as _session:
            query = (
                select(Job)
-                .options(selectinload(Job.document))  # pyright: ignore[reportArgumentType]
+                .options(
                    selectinload(Job.document),  # pyright: ignore[reportArgumentType]
                    selectinload(Job.job_sources).selectinload(JobSource.source),  # pyright: ignore[reportArgumentType]
                )
                .where(Job.status == JobStatus.QUEUED)
-                .order_by(Job.created_at)  # pyright: ignore[reportArgumentType]
+                # Break ties by id so "next" is stable when two rows share close timestamps.
                .order_by(Job.date_created, Job.id)  # pyright: ignore[reportArgumentType]
            )
            return (await _session.exec(query)).first()
    async def requeue_stale_processing_jobs(
        self,
        *,
        stale_before: datetime,
        session: AsyncSession | None = None,
    ) -> int:
        """Move stale processing jobs back to queued state.
        Jobs with ``status=PROCESSING`` and ``date_updated`` older than
        ``stale_before`` are considered stale and re-queued.
        """
        async with self._session_scope(session) as _session:
            query = select(Job).where(Job.status == JobStatus.PROCESSING).where(Job.date_updated < stale_before)
            stale_jobs = (await _session.exec(query)).all()
            if not stale_jobs:
                return 0
            now = datetime.now(UTC)
            for job in stale_jobs:
                job.status = JobStatus.QUEUED
                job.date_updated = now
            await self._finalize(session=_session, caller_session=session, refresh=stale_jobs)
            return len(stale_jobs)
    async def delete_job_with_guardrails(self, *, job_id: UUID, session: AsyncSession | None = None) -> None:
        """Delete a job with lifecycle guardrails and dependent cleanup policy.
        Policy:
        - Block when the job is actively processing.
        - Otherwise remove related JobSource rows, then delete the job.
        """
        async with self._session_scope(session) as _session:
            query = (
                select(Job)
                .options(selectinload(Job.job_sources))
                .where(Job.id == job_id)
                .execution_options(populate_existing=True)
            )
            job = (await _session.exec(query)).first()
            if job is None:
                raise self._not_found(job_id)
            if job.status == JobStatus.PROCESSING:
                raise JobDeleteBlockedError(
                    "Job delete blocked while status is processing",
                    category=ErrorCategory.VALIDATION,
                    suggestion="Wait for processing to complete, or move the job out of processing before deleting.",
                )
            attempt_count = (
                await _session.exec(
                    select(func.count())
                    .select_from(ExecutionAttempt)
                    .where(ExecutionAttempt.job_id == job_id)
                )
            ).one()
            if attempt_count:
                raise JobDeleteBlockedError(
                    "Job delete blocked because immutable execution evidence exists",
                    category=ErrorCategory.CONFLICT,
                    suggestion=(
                        "Retain the Job as processing history. Evidence deletion requires "
                        "an explicit retention workflow."
                    ),
                )
            for job_source in list(job.job_sources):
                await _session.delete(job_source)
            await _session.delete(job)
            await self._finalize(session=_session, caller_session=session)
    async def delete_job_and_evidence(self, *, job_id: UUID) -> None:
        """Explicitly delete a terminal job and all evidence owned by its attempts."""
        external_references: list[str] = []
        async with self._session_scope() as session:
            job = (
                await session.exec(
                    select(Job)
                    .options(selectinload(Job.job_sources))
                    .where(Job.id == job_id)
                    .execution_options(populate_existing=True)
                )
            ).first()
            if job is None:
                raise self._not_found(job_id)
            if job.status == JobStatus.PROCESSING:
                raise JobDeleteBlockedError(
                    "Job delete blocked while status is processing",
                    category=ErrorCategory.VALIDATION,
                    suggestion="Wait for processing to complete, or cancel it before deleting evidence.",
                )
            attempts = list(
                (
                    await session.exec(
                        select(ExecutionAttempt).where(ExecutionAttempt.job_id == job_id)
                    )
                ).all()
            )
            if attempts:
                attempt_ids = [attempt.id for attempt in attempts]
                artifacts = list(
                    (
                        await session.exec(
                            select(ProcessingArtifact).where(
                                ProcessingArtifact.execution_attempt_id.in_(attempt_ids)
                            )
                        )
                    ).all()
                )
                external_references = [
                    artifact.external_reference
                    for artifact in artifacts
                    if artifact.external_reference is not None
                ]
                for artifact in artifacts:
                    await session.delete(artifact)
                await session.flush()
                for attempt in attempts:
                    await session.delete(attempt)
                await session.flush()
            for job_source in list(job.job_sources):
                await session.delete(job_source)
            await session.flush()
            await session.delete(job)
            await self._finalize(session=session, caller_session=None)
        for external_reference in external_references:
            self._delete_external_artifact(external_reference)
    def _delete_external_artifact(self, external_reference: str) -> None:
        relative_path = Path(external_reference)
        if relative_path.is_absolute() or ".." in relative_path.parts:
            logger.warning("Skipped unsafe external artifact reference during job deletion: %s", external_reference)
            return
        artifact_root = self.settings.artifact_dir.resolve()
        artifact_path = (artifact_root / relative_path).resolve()
        if artifact_root not in artifact_path.parents:
            logger.warning("Skipped external artifact outside configured root: %s", external_reference)
            return
        try:
            artifact_path.unlink(missing_ok=True)
        except OSError:
            logger.warning("Failed to delete external artifact: %s", artifact_path)
    async def cancel_job(self, *, job_id: UUID, session: AsyncSession | None = None) -> Job:
        """Cancel a queued/processing job and stop remaining source work."""
        async with self._session_scope(session) as _session:
            query = (
                select(Job)
                .options(
                    selectinload(Job.job_sources).selectinload(JobSource.source),  # pyright: ignore[reportArgumentType]
                )
                .where(Job.id == job_id)
                .execution_options(populate_existing=True)
            )
            job = (await _session.exec(query)).first()
            if job is None:
                raise self._not_found(job_id)
            if job.status in {JobStatus.TRANSCRIBED, JobStatus.COMPLETED}:
                raise JobCancelBlockedError(
                    "Job cancel is not allowed for transcribed/completed jobs",
                    category=ErrorCategory.VALIDATION,
                    suggestion="Use resubmit for reprocessing needs, or leave the terminal job unchanged.",
                )
            now = datetime.now(UTC)
            job.status = JobStatus.FAILED
            job.date_updated = now
            for job_source in job.job_sources:
                if job_source.status == JobSourceStatus.TRANSCRIBED:
                    continue
                job_source.status = JobSourceStatus.FAILED
                job_source.raw_transcription = None
                job_source.error_detail = "Cancelled by user"
                job_source.executed_at = now
            await self._finalize(session=_session, caller_session=session, refresh=(job,))
            return job
    async def resubmit_failed_sources(self, *, job_id: UUID, session: AsyncSession | None = None) -> int:
        """Reset failed source executions and queue the job for reprocessing."""
        async with self._session_scope(session) as _session:
            query = (
                select(Job)
                .options(
                    selectinload(Job.job_sources).selectinload(JobSource.source),  # pyright: ignore[reportArgumentType]
                )
                .where(Job.id == job_id)
                .execution_options(populate_existing=True)
            )
            job = (await _session.exec(query)).first()
            if job is None:
                raise self._not_found(job_id)
            if job.status == JobStatus.PROCESSING:
                raise JobResubmitBlockedError(
                    "Job resubmit is blocked while processing is active",
                    category=ErrorCategory.VALIDATION,
                    suggestion="Cancel processing first, then resubmit remaining sources.",
                )
            candidates = [job_source for job_source in job.job_sources if job_source.status == JobSourceStatus.FAILED]
            if not candidates:
                raise JobResubmitBlockedError(
                    "Job has no failed sources to resubmit",
                    category=ErrorCategory.VALIDATION,
                    suggestion="Only failed sources can be resubmitted.",
                )
            now = datetime.now(UTC)
            for job_source in candidates:
                job_source.status = JobSourceStatus.PENDING
                job_source.raw_transcription = None
                job_source.error_detail = None
                job_source.executed_at = now
            job.status = JobStatus.QUEUED
            job.date_updated = now
            await self._finalize(session=_session, caller_session=session, refresh=(job,))
            return len(candidates)
    @staticmethod
    def _not_found(job_id: UUID) -> JobNotFoundError:
        return JobNotFoundError(
            f"Job with id {job_id} not found",
            category=ErrorCategory.NOT_FOUND,
            suggestion="Verify the Job id and retry.",
        )
@@ -0,0 +1,529 @@
 """People, relationship role, and document-person link services."""
 from __future__ import annotations
 import logging
 import re
 from collections.abc import Sequence
 from datetime import UTC
 from datetime import datetime
 from pathlib import Path
 from uuid import UUID
 from uuid import uuid4
 from sqlalchemy.exc import IntegrityError
 from sqlalchemy.orm import selectinload
 from sqlmodel import select
 from sqlmodel.ext.asyncio.session import AsyncSession
 from ..config import Settings
 from ..config import get_settings
 from ..db.models import Document
 from ..db.models import DocumentPerson
 from ..db.models import DocumentPersonRole
 from ..db.models import Person
 from ..db.models import PersonRole
 from ..errors import AppError
 from ..errors import ErrorCategory
 from .base import ServiceBase
 logger = logging.getLogger(__name__)
 PORTRAIT_EXTENSIONS = frozenset({".jpg", ".jpeg", ".png", ".gif", ".webp", ".bmp", ".tif", ".tiff"})
 FAMILY_SEARCH_ID_PATTERN = re.compile(r"^[A-Z0-9]{4}-[A-Z0-9]{3}$")
 class PeopleError(AppError):
    """Raised when a Person or document-person relationship operation fails."""
 class PersonMediaError(PeopleError):
    """Raised when Person portrait media cannot be validated or persisted."""
 class PersonRoleError(PeopleError):
    """Raised when Person Role maintenance fails."""
 REGISTRY_CODE_PATTERN = re.compile(r"^[a-z0-9_]+$")
 def _normalize_role_code(code: str) -> str:
    normalized = code.strip().lower()
    if not normalized or not REGISTRY_CODE_PATTERN.fullmatch(normalized):
        raise PersonRoleError(
            "Person Role code must contain only lowercase letters, numbers, and underscores",
            category=ErrorCategory.VALIDATION,
            suggestion="Enter a stable code such as witness or record_keeper.",
        )
    return normalized
 def _normalize_role_label(label: str) -> str:
    normalized = label.strip()
    if not normalized:
        raise PersonRoleError(
            "Person Role label is required",
            category=ErrorCategory.VALIDATION,
            suggestion="Enter a user-facing label and retry.",
        )
    return normalized
 def normalize_family_search_id(value: str | None) -> str | None:
    """Normalize and validate a FamilySearch tree person identifier."""
    normalized = (value or "").strip().upper()
    if not normalized:
        return None
    if not FAMILY_SEARCH_ID_PATTERN.fullmatch(normalized):
        raise PeopleError(
            "FamilySearch ID must use the format XXXX-XXX",
            category=ErrorCategory.VALIDATION,
            suggestion="Enter the seven-character FamilySearch person ID, including its hyphen.",
        )
    return normalized
 class PeopleService(ServiceBase):
    """Manage People, relationship roles, and document-person links."""
    async def create_person(self, person: Person, *, session: AsyncSession | None = None) -> Person:
        async with self._session_scope(session) as _session:
            person.family_search_id = normalize_family_search_id(person.family_search_id)
            _session.add(person)
            try:
                await self._finalize(session=_session, caller_session=session, refresh=(person,))
            except IntegrityError as exc:
                raise self._family_search_conflict(person.family_search_id) from exc
            return person
    async def read_person(self, person_id: UUID, *, session: AsyncSession | None = None) -> Person:
        async with self._session_scope(session) as _session:
            person = await _session.get(Person, person_id)
        if person is None:
            raise self._not_found(f"Person with id {person_id} not found")
        return person
    async def update_person(self, person: Person, *, session: AsyncSession | None = None) -> Person:
        async with self._session_scope(session) as _session:
            person.family_search_id = normalize_family_search_id(person.family_search_id)
            person.updated_at = datetime.now(UTC)
            merged = await _session.merge(person)
            try:
                await self._finalize(session=_session, caller_session=session, refresh=(merged,))
            except IntegrityError as exc:
                raise self._family_search_conflict(person.family_search_id) from exc
            return merged
    async def delete_person(self, person: Person, *, session: AsyncSession | None = None) -> None:
        async with self._session_scope(session) as _session:
            existing = await _session.get(
                Person,
                person.id,
                options=(selectinload(Person.document_people),),  # pyright: ignore[reportArgumentType]
            )
            if existing is None:
                raise self._not_found(f"Person with id {person.id} not found")
            for link in list(existing.document_people):
                await _session.delete(link)
            await _session.delete(existing)
            await self._finalize(session=_session, caller_session=session)
    async def create_document_person(
        self,
        document_person: DocumentPerson,
        *,
        session: AsyncSession | None = None,
    ) -> DocumentPerson:
        async with self._session_scope(session) as _session:
            await self._sync_role_fields(session=_session, link=document_person)
            _session.add(document_person)
            return await self._finalize_link(session=_session, caller_session=session, link=document_person)
    async def read_document_person(
        self,
        document_person_id: UUID,
        *,
        session: AsyncSession | None = None,
    ) -> DocumentPerson:
        async with self._session_scope(session) as _session:
            link = await _session.get(DocumentPerson, document_person_id)
        if link is None:
            raise self._not_found(f"DocumentPerson with id {document_person_id} not found")
        return link
    async def update_document_person(
        self,
        document_person: DocumentPerson,
        *,
        session: AsyncSession | None = None,
    ) -> DocumentPerson:
        async with self._session_scope(session) as _session:
            await self._sync_role_fields(session=_session, link=document_person)
            document_person.updated_at = datetime.now(UTC)
            merged = await _session.merge(document_person)
            return await self._finalize_link(session=_session, caller_session=session, link=merged)
    async def delete_document_person(
        self,
        document_person: DocumentPerson,
        *,
        session: AsyncSession | None = None,
    ) -> None:
        async with self._session_scope(session) as _session:
            await _session.delete(document_person)
            await self._finalize(session=_session, caller_session=session)
    async def read_person_detail(self, person_id: UUID, *, session: AsyncSession | None = None) -> Person:
        async with self._session_scope(session) as _session:
            query = (
                select(Person)
                .options(
                    selectinload(Person.document_people).selectinload(DocumentPerson.document),  # pyright: ignore[reportArgumentType]
                    selectinload(Person.document_people).selectinload(DocumentPerson.role_ref),  # pyright: ignore[reportArgumentType]
                )
                .where(Person.id == person_id)
                .execution_options(populate_existing=True)
            )
            person = (await _session.exec(query)).first()
        if person is None:
            raise self._not_found(f"Person with id {person_id} not found")
        return person
    async def list_people(self, *, session: AsyncSession | None = None) -> Sequence[Person]:
        async with self._session_scope(session) as _session:
            return (await _session.exec(select(Person))).all()
    async def list_person_roles(
        self,
        *,
        active_only: bool = True,
        session: AsyncSession | None = None,
    ) -> Sequence[PersonRole]:
        async with self._session_scope(session) as _session:
            query = select(PersonRole)
            if active_only:
                query = query.where(PersonRole.is_active.is_(True))
            return (await _session.exec(query.order_by(PersonRole.label, PersonRole.code))).all()
    async def create_person_role(
        self,
        *,
        code: str,
        label: str,
        is_active: bool = True,
        session: AsyncSession | None = None,
    ) -> PersonRole:
        """Create a Person Role with an immutable normalized code."""
        role = PersonRole(
            code=_normalize_role_code(code),
            label=_normalize_role_label(label),
            is_active=is_active,
        )
        async with self._session_scope(session) as _session:
            _session.add(role)
            try:
                await self._finalize(session=_session, caller_session=session, refresh=(role,))
            except IntegrityError as exc:
                raise PersonRoleError(
                    f"Person Role code {role.code!r} already exists",
                    category=ErrorCategory.CONFLICT,
                    suggestion="Choose a different stable code or edit the existing role.",
                ) from exc
            return role
    async def read_person_role(
        self,
        person_role_id: UUID,
        *,
        session: AsyncSession | None = None,
    ) -> PersonRole:
        """Read a Person Role by id."""
        async with self._session_scope(session) as _session:
            role = await _session.get(PersonRole, person_role_id)
        if role is None:
            raise PersonRoleError(
                f"Person Role with id {person_role_id} not found",
                category=ErrorCategory.NOT_FOUND,
                suggestion="Refresh Settings and select an available Person Role.",
            )
        return role
    async def update_person_role(
        self,
        person_role_id: UUID,
        *,
        label: str,
        is_active: bool,
        session: AsyncSession | None = None,
    ) -> PersonRole:
        """Update mutable Person Role fields without changing its code."""
        async with self._session_scope(session) as _session:
            role = await _session.get(PersonRole, person_role_id)
            if role is None:
                raise PersonRoleError(
                    f"Person Role with id {person_role_id} not found",
                    category=ErrorCategory.NOT_FOUND,
                    suggestion="Refresh Settings and select an available Person Role.",
                )
            role.label = _normalize_role_label(label)
            role.is_active = is_active
            role.updated_at = datetime.now(UTC)
            await self._finalize(session=_session, caller_session=session, refresh=(role,))
            return role
    async def delete_person_role(
        self,
        person_role_id: UUID,
        *,
        session: AsyncSession | None = None,
    ) -> None:
        """Delete an unreferenced Person Role without cascade behavior."""
        async with self._session_scope(session) as _session:
            role = await _session.get(PersonRole, person_role_id)
            if role is None:
                raise PersonRoleError(
                    f"Person Role with id {person_role_id} not found",
                    category=ErrorCategory.NOT_FOUND,
                    suggestion="Refresh Settings and select an available Person Role.",
                )
            if await self._person_role_is_referenced(session=_session, role=role):
                raise PersonRoleError(
                    f"Person Role {role.label!r} is referenced and cannot be deleted",
                    category=ErrorCategory.CONFLICT,
                    suggestion="Deactivate the role instead; historical relationships will retain it.",
                )
            await _session.delete(role)
            await self._finalize(session=_session, caller_session=session)
    async def is_person_role_referenced(
        self,
        person_role_id: UUID,
        *,
        session: AsyncSession | None = None,
    ) -> bool:
        """Return whether canonical or compatibility data references a Person Role."""
        async with self._session_scope(session) as _session:
            role = await _session.get(PersonRole, person_role_id)
            if role is None:
                raise PersonRoleError(
                    f"Person Role with id {person_role_id} not found",
                    category=ErrorCategory.NOT_FOUND,
                    suggestion="Refresh Settings and select an available Person Role.",
                )
            return await self._person_role_is_referenced(session=_session, role=role)
    @staticmethod
    async def _person_role_is_referenced(
        *,
        session: AsyncSession,
        role: PersonRole,
    ) -> bool:
        reference = (
            await session.exec(
                select(DocumentPerson.id).where(
                    (DocumentPerson.role_id == role.id) | (DocumentPerson.role == role.code)
                )
            )
        ).first()
        return reference is not None
    async def list_document_people(
        self,
        *,
        document_id: UUID | None = None,
        person_id: UUID | None = None,
        session: AsyncSession | None = None,
    ) -> Sequence[DocumentPerson]:
        async with self._session_scope(session) as _session:
            query = select(DocumentPerson).options(
                selectinload(DocumentPerson.document),  # pyright: ignore[reportArgumentType]
                selectinload(DocumentPerson.person),  # pyright: ignore[reportArgumentType]
                selectinload(DocumentPerson.role_ref),  # pyright: ignore[reportArgumentType]
            )
            if document_id is not None:
                query = query.where(DocumentPerson.document_id == document_id)
            if person_id is not None:
                query = query.where(DocumentPerson.person_id == person_id)
            return (await _session.exec(query)).all()
    async def add_document_person_link(
        self,
        *,
        document_id: UUID,
        person_id: UUID,
        role_id: UUID | None = None,
        role_code: str | None = None,
        session: AsyncSession | None = None,
    ) -> DocumentPerson:
        async with self._session_scope(session) as _session:
            await self._require_document(session=_session, document_id=document_id)
            if await _session.get(Person, person_id) is None:
                raise self._not_found(f"Person with id {person_id} not found")
            link = DocumentPerson(
                document_id=document_id,
                person_id=person_id,
                role=DocumentPersonRole.AUTHOR,
                role_id=role_id,
            )
            if role_code and role_code.strip():
                link.role = self._legacy_role(role_code)
            await self._sync_role_fields(session=_session, link=link)
            _session.add(link)
            return await self._finalize_link(session=_session, caller_session=session, link=link)
    async def set_document_person_role(
        self,
        *,
        document_person_id: UUID,
        role_id: UUID | None = None,
        role_code: str | None = None,
        session: AsyncSession | None = None,
    ) -> DocumentPerson:
        async with self._session_scope(session) as _session:
            link = await _session.get(DocumentPerson, document_person_id)
            if link is None:
                raise self._not_found(f"DocumentPerson with id {document_person_id} not found")
            if role_id is None and not (role_code or "").strip():
                raise PeopleError(
                    "Either role_id or role_code is required",
                    category=ErrorCategory.VALIDATION,
                    suggestion="Provide a valid role id or code and retry.",
                )
            if role_id is not None and (role_code or "").strip():
                raise PeopleError(
                    "Provide role_id or role_code, not both",
                    category=ErrorCategory.VALIDATION,
                    suggestion="Send only one relationship role selector and retry.",
                )
            link.role_id = role_id
            if role_code and role_code.strip():
                link.role = self._legacy_role(role_code)
            await self._sync_role_fields(session=_session, link=link)
            link.updated_at = datetime.now(UTC)
            return await self._finalize_link(session=_session, caller_session=session, link=link)
    async def remove_document_person_link(
        self,
        *,
        document_person_id: UUID,
        session: AsyncSession | None = None,
    ) -> None:
        async with self._session_scope(session) as _session:
            link = await _session.get(DocumentPerson, document_person_id)
            if link is None:
                raise self._not_found(f"DocumentPerson with id {document_person_id} not found")
            await _session.delete(link)
            await self._finalize(session=_session, caller_session=session)
    async def _resolve_or_create_role(
        self,
        *,
        session: AsyncSession,
        role_id: UUID | None,
        role_code: str,
    ) -> PersonRole:
        if role_id is not None:
            role = await session.get(PersonRole, role_id)
            if role is None:
                raise PeopleError(
                    f"Person role with id {role_id} not found",
                    category=ErrorCategory.VALIDATION,
                    suggestion="Select a valid relationship role and retry.",
                )
            return role
        normalized_code = role_code.strip().lower() or DocumentPersonRole.AUTHOR.value
        role = (await session.exec(select(PersonRole).where(PersonRole.code == normalized_code))).first()
        if role is None:
            role = PersonRole(code=normalized_code, label=normalized_code.replace("_", " ").title())
            session.add(role)
            await session.flush()
        return role
    async def _sync_role_fields(self, *, session: AsyncSession, link: DocumentPerson) -> None:
        role_code = link.role.value if isinstance(link.role, DocumentPersonRole) else str(link.role)
        role = await self._resolve_or_create_role(session=session, role_id=link.role_id, role_code=role_code)
        link.role_id = role.id
        link.role = self._legacy_role(role.code)
    async def _finalize_link(
        self,
        *,
        session: AsyncSession,
        caller_session: AsyncSession | None,
        link: DocumentPerson,
    ) -> DocumentPerson:
        try:
            await self._finalize(session=session, caller_session=caller_session, refresh=(link,))
        except IntegrityError as exc:
            raise PeopleError(
                "Duplicate relationship link for document/person/role",
                category=ErrorCategory.CONFLICT,
                suggestion="Remove the existing relationship link or choose a different role.",
            ) from exc
        return link
    async def _require_document(self, *, session: AsyncSession, document_id: UUID) -> None:
        if await session.get(Document, document_id) is None:
            raise self._not_found(f"Document with id {document_id} not found")
    @staticmethod
    def _family_search_conflict(family_search_id: str | None) -> PeopleError:
        return PeopleError(
            f"FamilySearch ID {family_search_id} is already assigned to another person",
            category=ErrorCategory.CONFLICT,
            suggestion="Open the existing person record or enter a different FamilySearch ID.",
        )
    @staticmethod
    def _legacy_role(role_code: str) -> str:
        return _normalize_role_code(role_code)
    @staticmethod
    def _not_found(message: str) -> PeopleError:
        return PeopleError(
            message,
            category=ErrorCategory.NOT_FOUND,
            suggestion="Verify the requested Person or relationship id and retry.",
        )
 def store_person_portrait(
    *,
    person_id: UUID,
    filename: str,
    file_bytes: bytes,
    settings: Settings | None = None,
 ) -> Path:
    """Persist Person portrait media under persons/<person_id>."""
    if not file_bytes:
        raise PersonMediaError(
            "Person portrait content is empty",
            category=ErrorCategory.VALIDATION,
            suggestion="Select a non-empty portrait file and retry.",
        )
    suffix = Path(filename).suffix.lower()
    if suffix not in PORTRAIT_EXTENSIONS:
        raise PersonMediaError(
            f"Unsupported portrait format: {suffix or '<none>'}",
            category=ErrorCategory.USER_INPUT,
            suggestion="Use JPG, JPEG, PNG, GIF, WEBP, BMP, or TIFF portrait media.",
        )
    runtime_settings = settings or get_settings()
    target_dir = runtime_settings.upload_dir / "persons" / str(person_id)
    target_dir.mkdir(parents=True, exist_ok=True)
    stored_path = target_dir / f"{uuid4()}{suffix}"
    try:
        stored_path.write_bytes(file_bytes)
    except OSError as exc:
        raise PersonMediaError(
            "Failed to persist Person portrait media",
            category=ErrorCategory.INFRA_PERSISTENT,
            suggestion="Check media directory permissions and available disk space, then retry.",
        ) from exc
    logger.info("Stored Person portrait media: %s", stored_path)
    return stored_path
@@ -0,0 +1,188 @@
 """Constrained storage for mutable prompt Markdown artifacts."""
 from __future__ import annotations
 import os
 from dataclasses import dataclass
 from pathlib import Path
 from uuid import uuid4
 from ..config import Settings
 from ..config import get_settings
 from ..errors import AppError
 from ..errors import ErrorCategory
 PROMPT_EXTENSION = ".md"
 BACKUP_SUFFIX = ".bak"
 class PromptStoreError(AppError):
    """Raised when prompt storage validation or persistence fails."""
@dataclass(frozen=True, slots=True)
 class PromptSummary:
    """Read model for one editable prompt artifact."""
    name: str
    is_default: bool
    has_backup: bool
 class PromptStore:
    """List, read, atomically update, and recover existing prompt files."""
    def __init__(self, settings: Settings | None = None) -> None:
        self.settings = settings or get_settings()
    def list_prompts(self) -> tuple[PromptSummary, ...]:
        """List editable direct-child Markdown prompts by filename."""
        root = self._prompt_root()
        try:
            candidates = tuple(root.iterdir())
        except OSError as exc:
            raise self._filesystem_error("Prompt directory could not be read", exc) from exc
        summaries: list[PromptSummary] = []
        for candidate in candidates:
            if candidate.suffix.lower() != PROMPT_EXTENSION or not candidate.is_file():
                continue
            resolved = candidate.resolve()
            if resolved.parent != root:
                continue
            summaries.append(
                PromptSummary(
                    name=candidate.name,
                    is_default=candidate.name == self.settings.default_prompt_name,
                    has_backup=self._backup_path(candidate).is_file(),
                )
            )
        return tuple(sorted(summaries, key=lambda item: item.name.casefold()))
    def read_prompt(self, name: str) -> str:
        """Read one existing UTF-8 prompt."""
        path = self._resolve_existing_prompt(name)
        return self._read_nonempty_text(path, description="Prompt")
    def write_prompt(self, name: str, content: str) -> None:
        """Atomically replace an existing prompt and retain one prior version."""
        path = self._resolve_existing_prompt(name)
        normalized_content = content.strip()
        if not normalized_content:
            raise PromptStoreError(
                "Prompt content cannot be empty",
                category=ErrorCategory.VALIDATION,
                suggestion="Enter prompt text before saving.",
            )
        self._atomic_write(path=path, content=f"{normalized_content}\n", preserve_current=True)
    def recover_prompt(self, name: str) -> None:
        """Restore the sole previous-version backup as an explicit operation."""
        path = self._resolve_existing_prompt(name)
        backup_path = self._backup_path(path)
        if not backup_path.is_file():
            raise PromptStoreError(
                f"No previous version is available for {path.name}",
                category=ErrorCategory.NOT_FOUND,
                suggestion="Save a prompt edit before attempting recovery.",
            )
        backup_content = self._read_nonempty_text(backup_path, description="Prompt backup")
        self._atomic_write(path=path, content=backup_content, preserve_current=True)
    def _prompt_root(self) -> Path:
        try:
            root = self.settings.prompt_dir.resolve()
        except OSError as exc:
            raise self._filesystem_error("Prompt directory could not be resolved", exc) from exc
        if not root.is_dir():
            raise PromptStoreError(
                f"Prompt directory is unavailable: {root}",
                category=ErrorCategory.INFRA_PERSISTENT,
                suggestion="Restore the configured prompt directory and its permissions.",
            )
        return root
    def _resolve_existing_prompt(self, name: str) -> Path:
        normalized_name = name.strip()
        if (
            not normalized_name
            or Path(normalized_name).name != normalized_name
            or Path(normalized_name).suffix.lower() != PROMPT_EXTENSION
        ):
            raise PromptStoreError(
                "Prompt name must be a direct-child Markdown filename",
                category=ErrorCategory.VALIDATION,
                suggestion="Select an existing .md prompt from Settings.",
            )
        root = self._prompt_root()
        try:
            path = (root / normalized_name).resolve()
        except OSError as exc:
            raise self._filesystem_error("Prompt path could not be resolved", exc) from exc
        if path.parent != root:
            raise PromptStoreError(
                "Prompt path must remain inside the configured prompt directory",
                category=ErrorCategory.VALIDATION,
                suggestion="Select an existing prompt from Settings.",
            )
        if not path.is_file():
            raise PromptStoreError(
                f"Prompt file not found: {normalized_name}",
                category=ErrorCategory.NOT_FOUND,
                suggestion="Refresh Settings and select an existing prompt.",
            )
        return path
    def _read_nonempty_text(self, path: Path, *, description: str) -> str:
        try:
            content = path.read_text(encoding="utf-8")
        except UnicodeError as exc:
            raise PromptStoreError(
                f"{description} is not valid UTF-8: {path.name}",
                category=ErrorCategory.VALIDATION,
                suggestion="Restore a valid UTF-8 Markdown prompt.",
            ) from exc
        except OSError as exc:
            raise self._filesystem_error(f"{description} could not be read", exc) from exc
        if not content.strip():
            raise PromptStoreError(
                f"{description} is empty: {path.name}",
                category=ErrorCategory.VALIDATION,
                suggestion="Restore non-empty prompt content.",
            )
        return content
    def _atomic_write(self, *, path: Path, content: str, preserve_current: bool) -> None:
        temporary_path = path.with_name(f".{path.name}.{uuid4().hex}.tmp")
        backup_path = self._backup_path(path)
        backup_temporary_path = backup_path.with_name(f".{backup_path.name}.{uuid4().hex}.tmp")
        try:
            self._write_synced(temporary_path, content.encode("utf-8"))
            if preserve_current:
                self._write_synced(backup_temporary_path, path.read_bytes())
                backup_temporary_path.replace(backup_path)
            temporary_path.replace(path)
        except (OSError, UnicodeError) as exc:
            raise self._filesystem_error(f"Prompt {path.name} could not be saved", exc) from exc
        finally:
            temporary_path.unlink(missing_ok=True)
            backup_temporary_path.unlink(missing_ok=True)
    @staticmethod
    def _write_synced(path: Path, content: bytes) -> None:
        with path.open("wb") as stream:
            stream.write(content)
            stream.flush()
            os.fsync(stream.fileno())
    @staticmethod
    def _backup_path(path: Path) -> Path:
        return path.with_name(f"{path.name}{BACKUP_SUFFIX}")
    @staticmethod
    def _filesystem_error(message: str, exc: Exception) -> PromptStoreError:
        return PromptStoreError(
            f"{message}: {exc}",
            category=ErrorCategory.INFRA_PERSISTENT,
            suggestion="Check prompt directory permissions and available disk space, then retry.",
        )
@@ -1,9 +1,14 @@
 from __future__ import annotations
 import hashlib
 import logging
 from collections.abc import Sequence
 from dataclasses import dataclass
 from pathlib import Path
 from uuid import UUID
 from uuid import uuid4
 from sqlmodel import select
 from sqlmodel.ext.asyncio.session import AsyncSession
 from transcription.config import Settings
@@ -11,50 +16,97 @@ from transcription.config import get_settings
 from transcription.errors import AppError
 from transcription.errors import ErrorCategory
-from ..models import Document
+from ..db.models import Document
-from ..models import Job
+from ..db.models import Job
-from .documents import UploadJobResult
+from ..db.models import JobSource
 from ..db.models import JobSourceStatus
 from ..db.models import Source
 from .sources import TranscriptionError
 from .sources import validate_source_content
 from .transcription import build_prompt_execution
 logger = logging.getLogger(__name__)
-SUPPORTED_UPLOAD_EXTENSIONS = {".jpg", ".jpeg", ".png", ".tif", ".tiff", ".pdf"}
+
 class SourceStorageError(AppError):
    """Raised when Source content cannot be validated or persisted safely."""
-class UploadError(AppError):
+UploadError = SourceStorageError
    """Raised when uploaded content cannot be persisted safely."""
-async def create_upload_job(
+@dataclass(frozen=True)
 class JobCreateResult:
    """Summary of explicit Job create records."""
    document_id: UUID
    job_id: UUID
    source_ids: tuple[UUID, ...]
@dataclass(frozen=True)
 class DocumentJobResult:
    """Summary of a Document, Source, and Job created together."""
    document_id: UUID
    job_id: UUID
    stored_path: Path
    original_filename: str
@dataclass(frozen=True)
 class PendingStoredSource:
    """Pre-staged Source artifact tied to a Source id."""
    source_id: UUID
    original_filename: str
    stored_path: Path
    file_hash: str
    file_size_bytes: int
 async def create_document_job(
    *,
    filename: str,
    file_bytes: bytes,
    session: AsyncSession,
    settings: Settings | None = None,
-) -> UploadJobResult:
+) -> DocumentJobResult:
-    """Create upload-backed document and queued job records."""
+    """Create a Document, its first Source, and a queued Job."""
    runtime_settings = settings or get_settings()
-    stored_path = store_file(
+    prompt_execution = build_prompt_execution(settings=runtime_settings)
    document_id = uuid4()
    source_id = uuid4()
    stored_path = store_source_file(
        filename=filename,
        file_bytes=file_bytes,
        settings=runtime_settings,
        relative_directory=Path("documents") / str(document_id),
        filename_stem=str(source_id),
    )
    file_hash, file_size_bytes = _compute_file_metadata(file_bytes)
    try:
-        document, job = await _create_upload_records(
+        document, job = await _create_document_job_records(
            session=session,
            document_id=document_id,
            source_id=source_id,
            original_filename=filename,
            stored_path=stored_path,
            file_hash=file_hash,
            file_size_bytes=file_size_bytes,
            prompt_execution=prompt_execution,
        )
    except Exception as exc:
        _best_effort_delete(stored_path)
-        raise UploadError(
+        raise SourceStorageError(
-            "Failed to create upload database records",
+            "Failed to create Document, Source, and Job records",
            category=ErrorCategory.INFRA_TRANSIENT,
            suggestion="Retry upload. If this keeps happening, verify database availability.",
            retriable=True,
        ) from exc
-    logger.info("Created upload job document_id=%s job_id=%s", document.id, job.id)
+    logger.info("Created document job document_id=%s job_id=%s", document.id, job.id)
-    return UploadJobResult(
+    return DocumentJobResult(
        document_id=document.id,
        job_id=job.id,
        stored_path=stored_path,
@@ -62,84 +114,270 @@ async def create_upload_job(
    )
-async def _create_upload_records(
+async def create_job_for_document(
    *,
    document_id: UUID,
    source_files: Sequence[tuple[str, bytes]],
    session: AsyncSession,
    provider: str | None = None,
    model: str | None = None,
    settings: Settings | None = None,
 ) -> JobCreateResult:
    """Create a queued Job for an existing Document with one or more Sources."""
    if not source_files:
        raise SourceStorageError(
            "At least one Source file is required to create a Job",
            category=ErrorCategory.VALIDATION,
            suggestion="Upload one or more files and try again.",
        )
    runtime_settings = settings or get_settings()
    prompt_execution = build_prompt_execution(settings=runtime_settings)
    sorted_source_files = sorted(source_files, key=lambda item: Path(item[0]).name.casefold())
    stored_sources: list[PendingStoredSource] = []
    for filename, file_bytes in sorted_source_files:
        source_id = uuid4()
        stored_sources.append(
            PendingStoredSource(
                source_id=source_id,
                original_filename=filename,
                stored_path=store_source_file(
                    filename=filename,
                    file_bytes=file_bytes,
                    settings=runtime_settings,
                    relative_directory=Path("documents") / str(document_id),
                    filename_stem=str(source_id),
                ),
                file_hash=_compute_file_hash(file_bytes),
                file_size_bytes=len(file_bytes),
            )
        )
    try:
        job, source_ids = await _create_job_for_document_records(
            session=session,
            document_id=document_id,
            stored_sources=stored_sources,
            provider=provider,
            model=model,
            prompt_execution=prompt_execution,
        )
    except Exception as exc:
        for source in stored_sources:
            _best_effort_delete(source.stored_path)
        raise SourceStorageError(
            "Failed to create Job records from Source files",
            category=ErrorCategory.INFRA_TRANSIENT,
            suggestion="Retry creation. If this keeps happening, verify database availability.",
            retriable=True,
        ) from exc
    logger.info("Created explicit job document_id=%s job_id=%s sources=%s", document_id, job.id, len(source_ids))
    return JobCreateResult(
        document_id=document_id,
        job_id=job.id,
        source_ids=tuple(source_ids),
    )
 async def _create_document_job_records(
    *,
    session: AsyncSession,
    document_id: UUID,
    source_id: UUID,
    original_filename: str,
    stored_path: Path,
    file_hash: str,
    file_size_bytes: int,
    prompt_execution,
 ) -> tuple[Document, Job]:
    document = Document(
-        filename=Path(original_filename).name,
+        id=document_id,
-        file_path=str(stored_path),
+        name=Path(original_filename).name,
    )
    session.add(document)
    await session.flush()
-    job = Job(document_id=document.id)
+    job = Job(
        document_id=document.id,
        prompt_name=prompt_execution.prompt_name,
        prompt_hash=prompt_execution.prompt_hash,
        system_prompt=prompt_execution.system_prompt,
        user_prompt=prompt_execution.user_prompt,
        temperature=prompt_execution.temperature,
        top_p=prompt_execution.top_p,
    )
    session.add(job)
    await session.flush()
    source = Source(
        id=source_id,
        document_id=document.id,
        page_number=1,
        upload_name=Path(original_filename).name,
        filename=stored_path.name,
        file_path=str(stored_path),
        file_hash=file_hash,
        file_size_bytes=file_size_bytes,
    )
    session.add(source)
    await session.flush()
    session.add(
        JobSource(
            job_id=job.id,
            source_id=source.id,
            status=JobSourceStatus.PENDING,
        )
    )
    await session.commit()
    await session.refresh(document)
    await session.refresh(job)
    return document, job
 async def _create_job_for_document_records(
    *,
    session: AsyncSession,
    document_id: UUID,
    stored_sources: Sequence[PendingStoredSource],
    provider: str | None,
    model: str | None,
    prompt_execution,
 ) -> tuple[Job, list[UUID]]:
    document = await session.get(Document, document_id)
    if document is None:
        raise SourceStorageError(
            f"Document with id {document_id} not found",
            category=ErrorCategory.NOT_FOUND,
            suggestion="Select an existing document and retry.",
        )
    existing_sources = (await session.exec(select(Source).where(Source.document_id == document_id))).all()
    next_page_number = max((source.page_number for source in existing_sources), default=0) + 1
    job = Job(
        document_id=document_id,
        provider=(provider or None),
        model=(model or None),
        prompt_name=prompt_execution.prompt_name,
        prompt_hash=prompt_execution.prompt_hash,
        system_prompt=prompt_execution.system_prompt,
        user_prompt=prompt_execution.user_prompt,
        temperature=prompt_execution.temperature,
        top_p=prompt_execution.top_p,
    )
    session.add(job)
    await session.flush()
    source_ids: list[UUID] = []
    for page_offset, stored_source in enumerate(stored_sources):
        source = Source(
            id=stored_source.source_id,
            document_id=document_id,
            page_number=next_page_number + page_offset,
            upload_name=Path(stored_source.original_filename).name,
            filename=stored_source.stored_path.name,
            file_path=str(stored_source.stored_path),
            file_hash=stored_source.file_hash,
            file_size_bytes=stored_source.file_size_bytes,
        )
        session.add(source)
        await session.flush()
        source_ids.append(source.id)
        session.add(
            JobSource(
                job_id=job.id,
                source_id=source.id,
                status=JobSourceStatus.PENDING,
            )
        )
    await session.commit()
    await session.refresh(job)
    return job, source_ids
 def _best_effort_delete(path: Path) -> None:
    try:
        if path.exists():
            path.unlink()
    except OSError:
-        logger.warning("Failed to clean up upload file after DB error: %s", path)
+        logger.warning("Failed to clean up Source file after database error: %s", path)
-def store_file(*, filename: str, file_bytes: bytes, settings: Settings | None = None) -> Path:
+def _compute_file_hash(file_bytes: bytes) -> str:
-    """Persist an uploaded file to the configured upload directory."""
+    return hashlib.sha256(file_bytes).hexdigest()
 def _compute_file_metadata(file_bytes: bytes) -> tuple[str, int]:
    return _compute_file_hash(file_bytes), len(file_bytes)
 def store_source_file(
    *,
    filename: str,
    file_bytes: bytes,
    settings: Settings | None = None,
    relative_directory: Path | None = None,
    filename_stem: str | None = None,
 ) -> Path:
    """Validate and persist a Source file to configured media storage."""
    runtime_settings = settings or get_settings()
-    _validate_upload(filename=filename, file_bytes=file_bytes)
+    try:
        validate_source_content(filename=filename, content=file_bytes)
    except TranscriptionError as exc:
        raise SourceStorageError(
            exc.message,
            category=exc.category,
            suggestion=exc.suggestion,
            retriable=exc.retriable,
        ) from exc
    return _store_file_bytes(
        filename=filename,
        file_bytes=file_bytes,
        settings=runtime_settings,
        relative_directory=relative_directory,
        filename_stem=filename_stem,
    )
    upload_dir = runtime_settings.upload_dir
    upload_dir.mkdir(parents=True, exist_ok=True)
-    stored_name = _build_stored_filename(filename)
+def _store_file_bytes(
-    stored_path = upload_dir / stored_name
+    *,
    filename: str,
    file_bytes: bytes,
    settings: Settings,
    relative_directory: Path | None = None,
    filename_stem: str | None = None,
 ) -> Path:
    upload_dir = settings.upload_dir
    target_dir = upload_dir if relative_directory is None else upload_dir / relative_directory
    target_dir.mkdir(parents=True, exist_ok=True)
    stored_name = _build_stored_filename(filename=filename, filename_stem=filename_stem)
    stored_path = target_dir / stored_name
    try:
        stored_path.write_bytes(file_bytes)
    except OSError as exc:
-        raise UploadError(
+        raise SourceStorageError(
-            "Failed to persist upload file",
+            "Failed to persist Source file",
            category=ErrorCategory.INFRA_PERSISTENT,
            suggestion="Check upload directory permissions and available disk space, then retry.",
        ) from exc
-    logger.info("Stored uploaded file: %s", stored_path)
+    logger.info("Stored Source file: %s", stored_path)
    return stored_path
-def _validate_upload(*, filename: str, file_bytes: bytes) -> None:
+def _build_stored_filename(*, filename: str, filename_stem: str | None = None) -> str:
    if not file_bytes:
        raise UploadError(
            "Upload payload is empty",
            category=ErrorCategory.VALIDATION,
            suggestion="Select a non-empty file and try again.",
        )
    safe_name = Path(filename).name
    if not safe_name:
        raise UploadError(
            "Upload filename is required",
            category=ErrorCategory.VALIDATION,
            suggestion="Choose a file with a valid filename and retry.",
        )
    suffix = Path(safe_name).suffix.lower()
-    if suffix not in SUPPORTED_UPLOAD_EXTENSIONS:
+    stem = filename_stem or str(uuid4())
-        raise UploadError(
+    return f"{stem}{suffix}"
            f"Unsupported upload extension: {suffix}",
            category=ErrorCategory.USER_INPUT,
            suggestion="Upload JPG, JPEG, PNG, TIFF, or PDF files only.",
        )
-def _build_stored_filename(filename: str) -> str:
+create_upload_job = create_document_job
-    safe_name = Path(filename).name
+store_file = store_source_file
    return f"{uuid4()}_{safe_name}"
@@ -1,303 +1,44 @@
-"""Prompt loading and provider-backed transcription service."""
+"""Compatibility exports for the Source-owned transcription implementation."""
-from __future__ import annotations
+from .sources import DEFAULT_PROMPT_FILE
 from .sources import SOURCE_EXTENSIONS
 from .sources import SOURCE_MIME_TYPES
 from .sources import PromptExecution
 from .sources import PromptLoadError
 from .sources import SourceDeleteBlockedError
 from .sources import SourceService
 from .sources import TranscriptionError
 from .sources import TranscriptionNotFoundError
 from .sources import build_prompt_execution
 from .sources import handle_transcription_errors
 from .sources import load_prompt_text
 from .sources import load_source_payload
 from .sources import source_mime_type
 from .sources import transcribe_document_image
 from .sources import validate_source_content
-import logging
+TranscriptionService = SourceService
-import mimetypes
+SUPPORTED_EXTENSIONS = SOURCE_EXTENSIONS
-from collections.abc import Sequence
+load_image_payload = load_source_payload
 from contextlib import contextmanager
 from pathlib import Path
 from uuid import UUID
-from sqlalchemy import func
+__all__ = [
-from sqlalchemy.ext.asyncio import async_sessionmaker
+    "DEFAULT_PROMPT_FILE",
-from sqlalchemy.orm import selectinload
+    "SOURCE_EXTENSIONS",
-from sqlmodel import select
+    "SOURCE_MIME_TYPES",
-from sqlmodel.ext.asyncio.session import AsyncSession
+    "SUPPORTED_EXTENSIONS",
-
+    "PromptExecution",
-from transcription.config import Settings
+    "PromptLoadError",
-from transcription.config import get_settings
+    "SourceDeleteBlockedError",
-from transcription.errors import AppError
+    "SourceService",
-from transcription.errors import ErrorCategory
+    "TranscriptionError",
-from transcription.models import Transcript
+    "TranscriptionNotFoundError",
-from transcription.providers import ProviderAuthError
+    "TranscriptionService",
-from transcription.providers import ProviderError
+    "build_prompt_execution",
-from transcription.providers import ProviderResponseError
+    "handle_transcription_errors",
-from transcription.providers import TranscriptionProvider
+    "load_image_payload",
-from transcription.providers import TranscriptionResult
+    "load_prompt_text",
-from transcription.providers import get_transcription_provider
+    "load_source_payload",
-
+    "source_mime_type",
-from .base import ServiceBase
+    "transcribe_document_image",
-
+    "validate_source_content",
-logger = logging.getLogger(__name__)
+]
 DEFAULT_PROMPT_FILE = "transcribe_document.md"
 SUPPORTED_EXTENSIONS = {".jpg", ".jpeg", ".png", ".tif", ".tiff", ".pdf"}
 class PromptLoadError(AppError):
    """Raised when prompt artifacts cannot be loaded safely."""
 class TranscriptionError(AppError):
    """Raised when transcription execution fails."""
 class TranscriptionNotFoundError(TranscriptionError):
    """Raised when a transcription is not found in the database."""
 class TranscriptionService(ServiceBase):
    """Service class for managing transcription operations.
    This is the top-level service that composes functionality from the other services."""
    provider: TranscriptionProvider
    def __init__(self, session_factory: async_sessionmaker[AsyncSession] | None = None):
        super().__init__(session_factory=session_factory)
        self.provider = get_transcription_provider(settings=self.settings)
    async def create_transcript(self, transcript: Transcript, *, session: AsyncSession | None = None) -> Transcript:
        """Create a new transcript in the database."""
        async with self._session_scope(session) as _session:
            _session.add(transcript)
            await self._finalize(session=_session, caller_session=session, refresh=(transcript,))
            return transcript
    async def read_transcript(self, transcript_id: UUID, *, session: AsyncSession | None = None) -> Transcript:
        """Read an existing transcript from the database."""
        async with self._session_scope(session) as _session:
            transcript = await _session.get(
                Transcript,
                transcript_id,
                # Makes the full Job model object available in the return Transcript object
                options=(selectinload(Transcript.job),),  # pyright: ignore[reportArgumentType]
            )
        if transcript is None:
            raise TranscriptionNotFoundError(
                f"Transcript with id {transcript_id} not found",
                category=ErrorCategory.NOT_FOUND,
                suggestion="Verify the transcript id and retry.",
            )
        return transcript
    async def update_transcript(self, transcript: Transcript, *, session: AsyncSession | None = None) -> Transcript:
        """Update an existing transcript in the database."""
        async with self._session_scope(session) as _session:
            merged = await _session.merge(transcript)
            await self._finalize(session=_session, caller_session=session, refresh=(merged,))
            return merged
    async def delete_transcript(self, transcript: Transcript, *, session: AsyncSession | None = None) -> None:
        """Delete a transcript from the database."""
        async with self._session_scope(session) as _session:
            await _session.delete(transcript)
            await self._finalize(session=_session, caller_session=session)
    async def transcribe_document(
        self,
        image_path: str | Path,
        job_id: UUID,
        *,
        prompt_name: str = DEFAULT_PROMPT_FILE,
        session: AsyncSession | None = None,
    ):
        """Transcribe a local image using the configured prompt and provider."""
        result = await transcribe_document_image(
            image_path=image_path,
            prompt_name=prompt_name,
            settings=self.settings,
            provider=self.provider,
        )
        await self.create_transcript_for_job(
            job_id=job_id,
            text=result.text,
            provider=result.provider,
            model=result.model,
            prompt_name=result.prompt_name,
            session=session,
        )
    async def create_transcript_for_job(
        self,
        *,
        job_id: UUID,
        text: str | None,
        error_detail: str | None = None,
        provider: str | None = None,
        model: str | None = None,
        prompt_name: str = DEFAULT_PROMPT_FILE,
        session: AsyncSession | None = None,
    ) -> Transcript:
        """Create a new transcript revision for a job id."""
        async with self._session_scope(session) as _session:
            rev_query = select(func.max(Transcript.revision)).where(Transcript.job_id == job_id)
            rev_result = await _session.exec(rev_query)
            max_revision = -1 if (rev := rev_result.one_or_none()) is None else rev
            next_revision = max_revision + 1
            transcript = Transcript(
                job_id=job_id,
                revision=next_revision,
                provider=provider or self.settings.provider.value,
                model=model or _resolve_transcript_model(provider=self.provider, settings=self.settings),
                prompt_name=prompt_name,
                text=text,
                error_detail=error_detail,
            )
            _session.add(transcript)
            await self._finalize(session=_session, caller_session=session, refresh=(transcript,))
            return transcript
    async def read_latest_transcript_by_job(
        self,
        job_id: UUID,
        *,
        session: AsyncSession | None = None,
    ) -> Transcript | None:
        """Read the latest transcript revision for a job id."""
        async with self._session_scope(session) as _session:
            query = _transcript_job_query(job_id=job_id).limit(1)
            result = await _session.exec(query)
            return result.one_or_none()
    async def list_transcripts_by_job(
        self,
        job_id: UUID,
        *,
        session: AsyncSession | None = None,
    ) -> Sequence[Transcript]:
        """List transcript revisions for a job id in ascending revision order."""
        async with self._session_scope(session) as _session:
            query = _transcript_job_query(job_id=job_id)
            result = await _session.exec(query)
            return result.all()
 def _transcript_job_query(job_id: UUID):
    return (
        select(Transcript)
        .where(Transcript.job_id == job_id)
        .options(selectinload(Transcript.job))  # pyright: ignore[reportArgumentType]
        .order_by(Transcript.revision)  # pyright: ignore[reportArgumentType]
    )  # fmt: skip
 def _resolve_transcript_model(*, provider: TranscriptionProvider, settings: Settings) -> str:
    provider_model = getattr(provider, "model", None)
    if isinstance(provider_model, str) and provider_model.strip():
        return provider_model
    if settings.provider_model and settings.provider_model.strip():
        return settings.provider_model
    return "unknown"
 async def transcribe_document_image(
    image_path: str | Path,
    *,
    prompt_name: str = DEFAULT_PROMPT_FILE,
    settings: Settings | None = None,
    provider: TranscriptionProvider | None = None,
 ) -> TranscriptionResult:
    """Transcribe a local image using the configured prompt and provider."""
    runtime_settings = settings or get_settings()
    prompt_text = load_prompt_text(prompt_name=prompt_name, settings=runtime_settings)
    image_bytes, mime_type = load_image_payload(image_path)
    adapter = provider or get_transcription_provider(settings=runtime_settings)
    logger.info("Starting transcription for image=%s mime_type=%s", image_path, mime_type)
    with handle_transcription_errors():
        result = await adapter.transcribe(
            prompt_text=prompt_text,
            image_bytes=image_bytes,
            mime_type=mime_type,
        )
    logger.info("Transcription completed for image=%s provider=%s", image_path, result.provider)
    return result
 def load_prompt_text(*, prompt_name: str = DEFAULT_PROMPT_FILE, settings: Settings | None = None) -> str:
    """Load and validate prompt text from PROMPT_DIR."""
    runtime_settings = settings or get_settings()
    prompt_path = runtime_settings.prompt_dir / prompt_name
    if not prompt_path.exists() or not prompt_path.is_file():
        raise PromptLoadError(
            f"Prompt file not found: {prompt_path}",
            category=ErrorCategory.INFRA_PERSISTENT,
            suggestion="Verify PROMPT_DIR and prompt file configuration, then retry.",
        )
    prompt_text = prompt_path.read_text(encoding="utf-8").strip()
    if not prompt_text:
        raise PromptLoadError(
            f"Prompt file is empty: {prompt_path}",
            category=ErrorCategory.VALIDATION,
            suggestion="Populate the prompt file with valid instructions and retry.",
        )
    logger.info("Loaded prompt artifact: %s", prompt_path)
    return prompt_text
 def load_image_payload(image_path: str | Path) -> tuple[bytes, str]:
    """Read image bytes and detect mime type for supported uploads."""
    path = Path(image_path)
    if not path.exists() or not path.is_file():
        raise TranscriptionError(
            f"Image file not found: {path}",
            category=ErrorCategory.NOT_FOUND,
            suggestion="Verify the uploaded file exists and retry from the jobs page.",
        )
    suffix = path.suffix.lower()
    if suffix not in SUPPORTED_EXTENSIONS:
        raise TranscriptionError(
            f"Unsupported file type: {suffix}",
            category=ErrorCategory.USER_INPUT,
            suggestion="Use JPG, JPEG, PNG, TIFF, or PDF files.",
        )
    mime_type, _ = mimetypes.guess_type(path.name)
    if suffix in {".tif", ".tiff"}:
        mime_type = "image/tiff"
    if not mime_type:
        raise TranscriptionError(
            f"Unable to determine MIME type for: {path}",
            category=ErrorCategory.VALIDATION,
            suggestion="Re-save the file in a supported format and retry.",
        )
    return path.read_bytes(), mime_type
@contextmanager
 def handle_transcription_errors():
    """Context manager to handle transcription errors."""
    try:
        yield
    except ProviderAuthError as exc:
        raise TranscriptionError(
            "Provider authentication failed",
            category=ErrorCategory.INFRA_PERSISTENT,
            suggestion="Verify provider API credentials and retry.",
        ) from exc
    except ProviderResponseError as exc:
        raise TranscriptionError(
            "Provider returned an invalid response",
            category=ErrorCategory.EXTERNAL_PROVIDER,
            suggestion="Retry once. If it persists, try a different provider model or inspect provider status.",
            retriable=True,
        ) from exc
    except ProviderError as exc:
        raise TranscriptionError(
            "Provider transcription failed",
            category=ErrorCategory.EXTERNAL_PROVIDER,
            suggestion="Retry the transcription from jobs. If repeated, check provider availability.",
            retriable=True,
        ) from exc
@@ -1,23 +1,63 @@
 import asyncio
 import inspect
 import logging
 from dataclasses import dataclass
 from datetime import UTC
 from datetime import datetime
 from sqlmodel.ext.asyncio.session import AsyncSession
 from ..config import Settings
 from ..config import get_settings
 from ..db.models import Job
 from ..db.models import JobSourceStatus
 from ..db.models import JobStatus
 from ..db.models import Source
 from ..errors import AppError
 from ..errors import ErrorCategory
 from ..errors import classify_unexpected_error
 from ..errors import format_error_detail
-from ..models import Job
+from ..providers import ProviderError
-from ..models import JobStatus
+from ..providers import RequestManifest
 from ..providers import SourceEvidenceReference
 from ..providers import TranscriptionProvider
 from ..providers import TranscriptionResult
 from ..providers import TransportEvidence
 from . import ServiceBundle
-from .transcription import DEFAULT_PROMPT_FILE
+from .sources import PromptExecution
-from .transcription import transcribe_document_image
+from .sources import build_prompt_execution
 from .sources import hash_prompt_text
 from .sources import source_mime_type
 from .sources import transcribe_document_image
 logger = logging.getLogger(__name__)
@dataclass(frozen=True)
 class _SuccessfulPage:
    source: Source
    result: TranscriptionResult
    started_at: datetime
    finished_at: datetime
    duration_ms: int
@dataclass(frozen=True)
 class _FailedPage:
    source: Source
    error: AppError
    started_at: datetime
    finished_at: datetime
    duration_ms: int
    request_manifest: RequestManifest | None = None
    transport_evidence: TransportEvidence | None = None
    failure_phase: str | None = None
    sdk_response_snapshot: dict | None = None
    normalized_metadata: dict | None = None
    provider: str | None = None
    model: str | None = None
 async def advance_job(
    job: Job,
    services: ServiceBundle,
@@ -26,9 +66,13 @@ async def advance_job(
 ) -> Job | None:
    """Advance a single job by lifecycle status."""
    settings = settings or get_settings()
-    match job.status:
+    current_status = _coerce_job_status(job.status)
    match current_status:
        case JobStatus.QUEUED:
-            return await process_queued_job(job=job, services=services, session=session)
+            return await process_queued_job(job=job, services=services, settings=settings, session=session)
        case JobStatus.PROCESSING:
            # Recover mid-flight jobs by continuing the queued processing path.
            return await process_queued_job(job=job, services=services, settings=settings, session=session)
        case JobStatus.FAILED:
            if job.retry_count < settings.worker_max_retries:
                return await services.jobs.update_job_state(
@@ -44,56 +88,227 @@ async def advance_job(
            return
-async def process_queued_job(
+async def process_queued_job(  # noqa: PLR0915
    *,
    job: Job,
    services: ServiceBundle,
    settings: Settings | None = None,
    session: AsyncSession | None = None,
 ) -> Job | None:
    """Process one complete transcription attempt for a queued job."""
-    if job.status != JobStatus.QUEUED:
+    runtime_settings = settings or get_settings()
    current_status = _coerce_job_status(job.status)
    if current_status not in {JobStatus.QUEUED, JobStatus.PROCESSING}:
        logger.warning(f"Job {job.id} is not queued. Current status: {job.status}")
        return
    # Transaction A: claim job for processing.
-    if session is None:
+    if current_status == JobStatus.QUEUED:
-        job = await services.jobs.mark_job_status(job.id, JobStatus.PROCESSING)
+        if session is None:
-    else:
+            job = await services.jobs.mark_job_status(job.id, JobStatus.PROCESSING)
-        # If we're sharing the session, need to make sure setting the Job to PROCESSING is committed before we start
+        else:
-        # the transcription, otherwise other workers may see the job as still QUEUED and try to process it.
+            # If we're sharing the session, need to make sure setting the Job to PROCESSING is committed before we start
-        job = await services.jobs.mark_job_status(job.id, JobStatus.PROCESSING, session=session)
+            # the transcription, otherwise other workers may see the job as still QUEUED and try to process it.
-        await session.commit()
+            job = await services.jobs.mark_job_status(job.id, JobStatus.PROCESSING, session=session)
            await session.commit()
-    document = job.document
+    source_job = await services.jobs.read_job(job_id=job.id, session=session)
-    assert document is not None, (
+    sources = _resolve_job_sources(source_job)
-        f"Job {job.id} has no associated document or the document failed to be loaded by the job service."
+    if not sources and not source_job.job_sources:
        candidate_sources = await services.sources.list_sources(document_id=job.document_id, session=session)
        sources = sorted(candidate_sources, key=lambda item: (item.page_number, item.upload_name.casefold()))
    if not sources:
        return await services.jobs.mark_job_status(job.id, JobStatus.TRANSCRIBED, session=session)
    successful_pages: list[_SuccessfulPage] = []
    failed_pages: list[_FailedPage] = []
    externally_stopped = False
    prompt_execution = _resolve_job_prompt_execution(source_job=source_job, settings=runtime_settings)
    for source in sources:
        if await _job_no_longer_processing(job_id=job.id, services=services, session=session):
            externally_stopped = True
            break
        started_at = datetime.now(UTC)
        monotonic_started_at = asyncio.get_running_loop().time()
        result: TranscriptionResult | None = None
        page_outcome: _SuccessfulPage | _FailedPage
        try:
            source_reference = SourceEvidenceReference(
                source_id=source.id,
                digest_sha256=source.file_hash.lower(),
                byte_size=source.file_size_bytes,
                media_type=source_mime_type(source.file_path),
                page_number=source.page_number,
            )
            result = await asyncio.wait_for(
                _call_transcriber(
                    source=source,
                    prompt_execution=prompt_execution,
                    settings=runtime_settings,
                    provider=services.sources.provider,
                    source_reference=source_reference,
                ),
                timeout=runtime_settings.worker_provider_timeout_seconds,
            )
            elapsed_seconds = asyncio.get_running_loop().time() - monotonic_started_at
            logger.info(
                "Provider response diagnostics operation=worker.provider_response "
                "job_id=%s document_id=%s source_id=%s provider=%s model=%s "
                "finish_reason=%s usage_input_tokens=%s usage_output_tokens=%s usage_total_tokens=%s "
                "latency_seconds=%.3f text_chars=%s text_lines=%s",
                job.id,
                job.document_id,
                source.id,
                result.provider,
                result.model,
                result.finish_reason or "unknown",
                result.usage_input_tokens,
                result.usage_output_tokens,
                result.usage_total_tokens,
                elapsed_seconds,
                len(result.text),
                _line_count(result.text),
            )
            _validate_transcription_quality(result=result, settings=runtime_settings)
            finished_at = datetime.now(UTC)
            page_outcome = _SuccessfulPage(
                source=source,
                result=result,
                started_at=started_at,
                finished_at=finished_at,
                duration_ms=max(0, int(elapsed_seconds * 1000)),
            )
            successful_pages.append(page_outcome)
        except TimeoutError:
            error = AppError(
                f"Provider call timed out after {runtime_settings.worker_provider_timeout_seconds:.1f}s",
                category=ErrorCategory.EXTERNAL_PROVIDER,
                suggestion="Retry the job. If this repeats, verify provider latency and request payload size.",
                retriable=True,
            )
            finished_at = datetime.now(UTC)
            page_outcome = _FailedPage(
                source=source,
                error=error,
                started_at=started_at,
                finished_at=finished_at,
                duration_ms=max(
                    0,
                    int((asyncio.get_running_loop().time() - monotonic_started_at) * 1000),
                ),
                request_manifest=getattr(services.sources.provider, "current_request_manifest", None),
                transport_evidence=getattr(
                    services.sources.provider,
                    "current_transport_evidence",
                    None,
                ),
                failure_phase="local_timeout",
            )
            failed_pages.append(page_outcome)
            logger.error(
                "Source failed operation=worker.process_job job_id=%s document_id=%s "
                "source_id=%s error_id=%s category=%s",
                job.id,
                job.document_id,
                source.id,
                error.error_id,
                error.category.value,
            )
        except Exception as exc:  # noqa: BLE001
            match exc:
                case AppError() as error:
                    pass
                case _:
                    error = classify_unexpected_error(exc, operation="worker.process_job")
            finished_at = datetime.now(UTC)
            provider_error = _find_provider_error(exc)
            page_outcome = _FailedPage(
                source=source,
                error=error,
                started_at=started_at,
                finished_at=finished_at,
                duration_ms=max(
                    0,
                    int((asyncio.get_running_loop().time() - monotonic_started_at) * 1000),
                ),
                request_manifest=(
                    result.request_manifest
                    if result is not None
                    else provider_error.request_manifest
                    if provider_error is not None
                    else None
                ),
                transport_evidence=(
                    result.transport_evidence
                    if result is not None
                    else provider_error.transport_evidence
                    if provider_error is not None
                    else None
                ),
                failure_phase=(
                    "transcription_quality"
                    if result is not None
                    else provider_error.failure_phase
                    if provider_error is not None
                    else "application"
                ),
                sdk_response_snapshot=result.raw_api_response if result is not None else None,
                normalized_metadata=result.metadata_payload() if result is not None else None,
                provider=result.provider if result is not None else None,
                model=result.model if result is not None else None,
            )
            failed_pages.append(page_outcome)
            logger.error(
                "Source failed operation=worker.process_job job_id=%s document_id=%s "
                "source_id=%s error_id=%s category=%s",
                job.id,
                job.document_id,
                source.id,
                error.error_id,
                error.category.value,
            )
        await _persist_page_outcome_durably(
            job=job,
            services=services,
            page=page_outcome,
            session=session,
        )
        if await _job_no_longer_processing(job_id=job.id, services=services, session=session):
            externally_stopped = True
            break
    terminal_status = JobStatus.TRANSCRIBED
    if externally_stopped:
        terminal_status = JobStatus.FAILED
    elif failed_pages and successful_pages:
        terminal_status = JobStatus.PARTIAL_SUCCESS
    elif failed_pages and not successful_pages:
        terminal_status = JobStatus.FAILED
    updated_job = await _finalize_batch_outcome(
        job=job,
        services=services,
        status=terminal_status,
        session=session,
    )
-    try:
+    logger.info(
-        result = await transcribe_document_image(document.file_path)
+        "Job finished operation=worker.process_job job_id=%s document_id=%s status=%s success_pages=%s failed_pages=%s",
-        job = await _finalize_transcribed(job=job, services=services, result=result, session=session)
+        updated_job.id,
-        logger.info(
+        updated_job.document_id,
-            "Job transcribed operation=worker.process_job job_id=%s document_id=%s provider=%s",
+        updated_job.status.value,
-            job.id,
+        len(successful_pages),
-            document.id,
+        len(failed_pages),
-            result.provider,
+    )
-        )
+    return updated_job
    except Exception as exc:  # noqa: BLE001
        match exc:
            case AppError() as error:
                pass
            case _:
                error = classify_unexpected_error(exc, operation="worker.process_job")
        job = await _finalize_failed(job=job, services=services, error=error, session=session)
        logger.error(
            "Job failed operation=worker.process_job job_id=%s document_id=%s error_id=%s category=%s",
            job.id,
            document.id,
            error.error_id,
            error.category.value,
        )
    return job
 async def process_next_queued_job(
@@ -104,6 +319,7 @@ async def process_next_queued_job(
 ) -> bool:
    """Process the next queued job if one exists."""
    job = await services.jobs.read_next_queued_job(session=session)
    if job is None:
        return False
@@ -111,133 +327,244 @@ async def process_next_queued_job(
    return True
-async def _finalize_transcribed(
+def _resolve_job_sources(job: Job) -> list[Source]:
    """Resolve non-transcribed linked sources for a job in deterministic page order."""
    if not job.job_sources:
        return []
    sources = [
        job_source.source
        for job_source in job.job_sources
        if job_source.source is not None and job_source.status != JobSourceStatus.TRANSCRIBED
    ]
    return sorted(sources, key=lambda item: (item.page_number, item.upload_name.casefold()))
 def _resolve_job_prompt_execution(*, source_job: Job, settings: Settings) -> PromptExecution:
    if source_job.user_prompt and source_job.prompt_name:
        return PromptExecution(
            prompt_name=source_job.prompt_name,
            prompt_hash=source_job.prompt_hash or hash_prompt_text(source_job.user_prompt),
            system_prompt=source_job.system_prompt,
            user_prompt=source_job.user_prompt,
            temperature=source_job.temperature,
            top_p=source_job.top_p,
        )
    return build_prompt_execution(settings=settings)
 async def _job_no_longer_processing(
    *,
    job_id,
    services: ServiceBundle,
    session: AsyncSession | None = None,
 ) -> bool:
    """Return True when job status changed externally from PROCESSING."""
    latest_job = await services.jobs.read_job(job_id=job_id, session=session)
    return latest_job.status != JobStatus.PROCESSING
 async def _finalize_batch_outcome(
    *,
    job: Job,
    services: ServiceBundle,
-    result: TranscriptionResult,
+    status: JobStatus,
    session: AsyncSession | None = None,
 ) -> Job:
-    """Transaction B: transcript + TRANSCRIBED in one commit."""
+    """Persist the terminal aggregate status after all page outcomes are durable."""
    if session is None:
        async with services.jobs._session_scope() as local_session:
-            await services.transcriptions.create_transcript_for_job(
+            updated_job = await services.jobs.mark_job_status(job.id, status, session=local_session)
                job_id=job.id,
                text=result.text,
                error_detail=None,
                provider=result.provider,
                model=result.model,
                prompt_name=result.prompt_name,
                session=local_session,
            )
            updated_job = await services.jobs.mark_job_status(
                job.id,
                JobStatus.TRANSCRIBED,
                session=local_session,
            )
            await local_session.commit()
            return updated_job
-    await services.transcriptions.create_transcript_for_job(
+    updated_job = await services.jobs.mark_job_status(job.id, status, session=session)
        job_id=job.id,
        text=result.text,
        error_detail=None,
        provider=result.provider,
        model=result.model,
        prompt_name=result.prompt_name,
        session=session,
    )
    updated_job = await services.jobs.mark_job_status(
        job.id,
        JobStatus.TRANSCRIBED,
        session=session,
    )
    await session.commit()
    return updated_job
-async def _finalize_retry(
+async def _persist_page_outcome_durably(
    *,
    job: Job,
    services: ServiceBundle,
-    error: AppError,
+    page: _SuccessfulPage | _FailedPage,
-    settings: Settings,
+    session: AsyncSession | None,
-    session: AsyncSession | None = None,
+) -> None:
-) -> Job:
+    """Commit one completed provider call before processing the next source."""
-    """Transaction C: transcript error + QUEUED + retry increment in one commit."""
+    task = asyncio.create_task(
-    if session is None:
+        _persist_page_outcome(job=job, services=services, page=page, session=session)
-        async with services.jobs._session_scope() as local_session:
+    )
-            await services.transcriptions.create_transcript_for_job(
+    try:
-                job_id=job.id,
+        await asyncio.shield(task)
-                text=None,
+    except asyncio.CancelledError:
-                error_detail=format_error_detail(error),
+        await task
-                prompt_name=DEFAULT_PROMPT_FILE,
+        raise
                session=local_session,
            )
            updated_job = await services.jobs.update_job_state(
                job_id=job.id,
                status=JobStatus.QUEUED,
                retry_count_increment=1,
                session=local_session,
            )
            await local_session.commit()
    else:
        await services.transcriptions.create_transcript_for_job(
            job_id=job.id,
            text=None,
            error_detail=format_error_detail(error),
            prompt_name=DEFAULT_PROMPT_FILE,
            session=session,
        )
        updated_job = await services.jobs.update_job_state(
            job_id=job.id,
            status=JobStatus.QUEUED,
            retry_count_increment=1,
            session=session,
        )
        await session.commit()
    if settings.worker_retry_backoff_seconds > 0:
        await asyncio.sleep(settings.worker_retry_backoff_seconds)
    return updated_job
-async def _finalize_failed(
+async def _persist_page_outcome(
    *,
    job: Job,
    services: ServiceBundle,
-    error: AppError,
+    page: _SuccessfulPage | _FailedPage,
-    session: AsyncSession | None = None,
+    session: AsyncSession | None,
-) -> Job:
+) -> None:
    """Transaction B: transcript error + FAILED in one commit."""
    if session is None:
-        async with services.jobs._session_scope() as local_session:
+        async with services.sources._session_scope() as local_session:
-            await services.transcriptions.create_transcript_for_job(
+            await _write_page_outcome(job=job, services=services, page=page, session=local_session)
                job_id=job.id,
                text=None,
                error_detail=format_error_detail(error),
                prompt_name=DEFAULT_PROMPT_FILE,
                session=local_session,
            )
            updated_job = await services.jobs.mark_job_status(
                job.id,
                JobStatus.FAILED,
                session=local_session,
            )
            await local_session.commit()
-            return updated_job
+        return
-    await services.transcriptions.create_transcript_for_job(
+    await _write_page_outcome(job=job, services=services, page=page, session=session)
    await session.commit()
 async def _write_page_outcome(
    *,
    job: Job,
    services: ServiceBundle,
    page: _SuccessfulPage | _FailedPage,
    session: AsyncSession,
 ) -> None:
    if isinstance(page, _SuccessfulPage):
        source = page.source
        result = page.result
        await services.sources.update_job_source_transcription(
            job_id=job.id,
            source_id=source.id,
            text=result.text,
            error_detail=None,
            ai_metadata=result.metadata_payload(),
            raw_api_response=result.raw_api_response,
            provider=result.provider,
            model=result.model,
            request_manifest=result.request_manifest,
            transport_evidence=result.transport_evidence,
            started_at=page.started_at,
            finished_at=page.finished_at,
            duration_ms=page.duration_ms,
            session=session,
        )
        return
    await services.sources.update_job_source_transcription(
        job_id=job.id,
        source_id=page.source.id,
        text=None,
-        error_detail=format_error_detail(error),
+        error_detail=format_error_detail(page.error),
-        prompt_name=DEFAULT_PROMPT_FILE,
+        ai_metadata=page.normalized_metadata,
        raw_api_response=page.sdk_response_snapshot,
        provider=page.provider,
        model=page.model,
        request_manifest=page.request_manifest,
        transport_evidence=page.transport_evidence,
        failure_phase=page.failure_phase,
        error_category=page.error.category.value,
        started_at=page.started_at,
        finished_at=page.finished_at,
        duration_ms=page.duration_ms,
        session=session,
    )
-    updated_job = await services.jobs.mark_job_status(
+
-        job.id,
+
-        JobStatus.FAILED,
+def _validate_transcription_quality(*, result: TranscriptionResult, settings: Settings) -> None:
-        session=session,
+    text_chars = len(result.text)
    text_lines = _line_count(result.text)
    if settings.worker_fail_on_finish_reason_length and (result.finish_reason or "").lower() == "length":
        raise AppError(
            "Provider output appears truncated (finish_reason=length)",
            category=ErrorCategory.EXTERNAL_PROVIDER,
            suggestion=(
                "Retry the job. If this repeats, use a faster model, reduce input complexity, "
                "or increase provider output budget."
            ),
            retriable=True,
        )
    if settings.worker_min_transcription_chars > 0 and text_chars < settings.worker_min_transcription_chars:
        raise AppError(
            (
                "Transcription output below configured minimum character threshold "
                f"({text_chars} < {settings.worker_min_transcription_chars})"
            ),
            category=ErrorCategory.EXTERNAL_PROVIDER,
            suggestion=(
                "Retry the job. If this repeats, switch model or raise minimum thresholds based on document type."
            ),
            retriable=True,
        )
    if settings.worker_min_transcription_lines > 0 and text_lines < settings.worker_min_transcription_lines:
        raise AppError(
            (
                "Transcription output below configured minimum line threshold "
                f"({text_lines} < {settings.worker_min_transcription_lines})"
            ),
            category=ErrorCategory.EXTERNAL_PROVIDER,
            suggestion=(
                "Retry the job. If this repeats, switch model or raise minimum thresholds based on document type."
            ),
            retriable=True,
        )
 def _line_count(text: str) -> int:
    stripped = text.strip()
    if not stripped:
        return 0
    return sum(1 for line in stripped.splitlines() if line.strip())
 def _coerce_job_status(value: object) -> JobStatus | None:
    if isinstance(value, JobStatus):
        return value
    if isinstance(value, str):
        lowered = value.strip().lower()
        for member in JobStatus:
            if lowered in {member.value.lower(), member.name.lower()}:
                return member
    return None
 def _find_provider_error(exc: BaseException) -> ProviderError | None:
    """Find provider evidence carried through application error translation."""
    current: BaseException | None = exc
    while current is not None:
        if isinstance(current, ProviderError):
            return current
        current = current.__cause__ or current.__context__
    return None
 async def _call_transcriber(
    *,
    source: Source,
    prompt_execution: PromptExecution,
    settings: Settings,
    provider: TranscriptionProvider,
    source_reference: SourceEvidenceReference,
 ) -> TranscriptionResult:
    """Call the current transcriber while supporting legacy injected test doubles."""
    if "source_reference" in inspect.signature(transcribe_document_image).parameters:
        return await transcribe_document_image(
            source.file_path,
            prompt_name=prompt_execution.prompt_name,
            prompt_text=prompt_execution.user_prompt,
            temperature=prompt_execution.temperature,
            top_p=prompt_execution.top_p,
            settings=settings,
            provider=provider,
            source_reference=source_reference,
        )
    return await transcribe_document_image(
        source.file_path,
        prompt_name=prompt_execution.prompt_name,
        prompt_text=prompt_execution.user_prompt,
        temperature=prompt_execution.temperature,
        top_p=prompt_execution.top_p,
        settings=settings,
        provider=provider,
    )
    await session.commit()
    return updated_job
@@ -1,38 +1,32 @@
 """UI page registration exports."""
-from pathlib import Path
+from contextlib import suppress
 from fastapi import FastAPI
 from nicegui import app as nicegui_app
 from nicegui import ui
 from transcription.config import get_settings
 from transcription.ui.pages.documents_page import register_page as register_documents_page
 from transcription.ui.pages.home_page import register_page as register_home_page
 from transcription.ui.pages.jobs_page import register_page as register_jobs_page
-from transcription.ui.pages.upload_page import register_page as register_upload_page
+from transcription.ui.pages.people_page import register_page as register_people_page
 from transcription.ui.pages.settings_page import register_page as register_settings_page
 from transcription.ui.pages.sources_page import register_page as register_sources_page
 from transcription.ui.resources import read_css
 from transcription.ui.theme import apply_archival_theme
 _THEME_REGISTERED_STATE_KEY = "transcription_ui_theme_registered"
 _THEME_COLORS: dict[str, str] = {
    "primary": "#6f97e8",
    "secondary": "#92b5f5",
    "accent": "#7fc0de",
    "dark": "#22304a",
    "dark_page": "#1a2538",
    "positive": "#86c8ad",
    "negative": "#d98a9a",
    "info": "#7ebdda",
    "warning": "#e2c083",
 }
 def _register_global_styles(app: FastAPI) -> None:
    if getattr(app.state, _THEME_REGISTERED_STATE_KEY, False):
        return
-    nicegui_app.colors(**_THEME_COLORS)
+    apply_archival_theme()
-
+    with suppress(RuntimeError):
-    css_path = Path(__file__).resolve().parent / "static" / "colors.css"
+        ui.add_css(read_css("theme.css"), shared=True)
-    if css_path.exists():
+        # NiceGUI shared style registration can raise when the app is constructed
-        ui.add_css(css_path, shared=True)
+        # outside an active client context, which happens in unit tests.
    setattr(app.state, _THEME_REGISTERED_STATE_KEY, True)
@@ -40,6 +34,10 @@ def _register_global_styles(app: FastAPI) -> None:
 def register_pages(app: FastAPI) -> None:
    """Register all NiceGUI pages and mount them onto the FastAPI app."""
    _register_global_styles(app)
-    register_upload_page()
+    register_home_page()
    register_documents_page()
    register_people_page()
    register_sources_page()
    register_jobs_page()
-    ui.run_with(app, mount_path="/ui", show_welcome_message=False, dark=True)
+    register_settings_page(settings=getattr(app.state, "settings", None) or get_settings())
    ui.run_with(app, mount_path="/ui", show_welcome_message=False, dark=False)
@@ -1,7 +1,19 @@
 """Reusable UI component exports."""
 from transcription.ui.components.app_shell import NAV_ITEMS
 from transcription.ui.components.app_shell import render_app_shell
 from transcription.ui.components.app_shell import render_navigation_header
 from transcription.ui.components.document_panzoom import render_document_panzoom
 from transcription.ui.components.primitives import destructive_button
 from transcription.ui.components.primitives import render_empty_state
 from transcription.ui.components.primitives import section_header_row
-__all__ = ["NAV_ITEMS", "render_document_panzoom", "render_navigation_header"]
+__all__ = [
    "NAV_ITEMS",
    "destructive_button",
    "render_app_shell",
    "render_document_panzoom",
    "render_empty_state",
    "render_navigation_header",
    "section_header_row",
 ]
@@ -4,56 +4,69 @@ from __future__ import annotations
 from nicegui import ui
 from transcription.ui.theme import VIBESCRIBE_LOGO_SVG
 NAV_ITEMS: tuple[tuple[str, str, str], ...] = (
-    ("Upload", "/upload", "upload_file"),
+    ("Documents", "/documents", "description"),
    ("People", "/people", "group"),
    ("Sources", "/sources", "folder"),
    ("Jobs", "/jobs", "work_history"),
    ("Settings", "/settings", "settings"),
 )
 def _is_active_path(*, current_path: str, item_path: str) -> bool:
    if item_path == "/jobs":
        return current_path == "/jobs" or current_path.startswith("/jobs/")
    if item_path == "/documents":
        return current_path == "/documents" or current_path.startswith("/documents/")
    if item_path == "/people":
        return current_path == "/people" or current_path.startswith("/people/")
    if item_path == "/sources":
        return current_path == "/sources" or current_path.startswith("/sources/")
    if item_path == "/settings":
        return current_path == "/settings" or current_path.startswith("/settings/")
    return current_path == item_path
 def _button_props(*, icon: str, is_active: bool) -> str:
    if is_active:
        return f"icon={icon} no-caps unelevated color=primary text-color=white"
    return f"icon={icon} no-caps outline color=secondary text-color=secondary"
 def _button_classes(*, is_active: bool) -> str:
    base = "w-full sm:w-auto min-h-[40px] px-3 rounded-lg text-body2 text-weight-medium"
    if is_active:
        return f"{base}"
    return f"{base}"
 def _render_nav_button(*, label: str, path: str, icon: str, current_path: str) -> None:
    is_active = _is_active_path(current_path=current_path, item_path=path)
-    button = ui.button(
+    classes = "app-shell__nav-item"
    if is_active:
        classes = f"{classes} app-shell__nav-item--active"
    ui.button(
        label,
        icon=icon,
        on_click=lambda _=None, route=path: ui.navigate.to(route),
-    )
+    ).props("flat no-caps").classes(classes)
    button.props(_button_props(icon=icon, is_active=is_active)).classes(_button_classes(is_active=is_active))
 def _normalize_path(current_path: str | None) -> str:
    normalized = (current_path or "").strip()
    if not normalized:
-        return "/upload"
+        return "/homepage"
    return normalized.rstrip("/") or "/"
-def render_navigation_header(*, current_path: str | None = None) -> None:
+def render_app_shell(*, current_path: str | None = None) -> None:
-    """Render a shared app header with links for top-level pages."""
+    """Render the shared application shell header."""
    normalized_path = _normalize_path(current_path)
-    with (
+    with ui.header().classes("app-shell"), ui.element("div").classes("app-shell__inner"):
-        ui.header(elevated=True).props("bordered").classes("bg-dark text-white q-px-sm q-py-xs"),
+        with ui.element("a").props('href="/ui/homepage"').classes("app-shell__brand no-wrap"):
-        ui.row().classes("w-full items-center justify-end q-gutter-xs"),
+            ui.html(VIBESCRIBE_LOGO_SVG).classes("app-shell__brand-mark")
-        ui.element("div").classes("w-full grid grid-cols-2 gap-2 sm:flex sm:justify-start sm:gap-2"),
+            ui.label("VibeScribe").classes("app-shell__brand-name")
-    ):
+
-        for label, path, icon in NAV_ITEMS:
+        with ui.element("nav").props('aria-label="Primary navigation"').classes("app-shell__nav"):
-            _render_nav_button(label=label, path=path, icon=icon, current_path=normalized_path)
+            for label, path, icon in NAV_ITEMS:
                _render_nav_button(label=label, path=path, icon=icon, current_path=normalized_path)
        with ui.row().classes("app-shell__actions no-wrap"):
            ui.label("Saved").classes("app-shell__save-state")
            ui.button(icon="more_horiz").props("flat round dense").tooltip("More actions")
 def render_navigation_header(*, current_path: str | None = None) -> None:
    """Render the app shell using the legacy page-level entry point."""
    render_app_shell(current_path=current_path)
@@ -0,0 +1,15 @@
 # transcription/ui/components/cards.py
 from contextlib import contextmanager
 from nicegui import ui
@contextmanager
 def archival_card(title: str | None = None, extra_classes: str = ""):
    """Reusable container for Flat 2.0 Bento Grid cards."""
    with ui.card().classes(f"w-full ui-card-surface p-4 {extra_classes}") as card:
        if title:
            ui.label(title.upper()).classes(
                "text-xs font-bold ui-text-muted tracking-wider mb-3 ui-header-divider pb-1"
            )
        yield card
@@ -0,0 +1,14 @@
 # transcription/ui/components/data_display.py
 from nicegui import ui
 def metadata_row(label: str, value: str):
    """Render a high-density, low-contrast key-value pair."""
    with ui.row().classes("justify-between w-full border-b ui-border-subtle pb-1 text-xs"):
        ui.label(label).classes("ui-text-muted")
        ui.label(value).classes("font-semibold ui-text-primary")
 def archival_badge(text: str):
    """Standardized Aged Sepia badge."""
    return ui.badge(text).classes("text-[10px] ui-badge-secondary")
@@ -10,43 +10,38 @@ from uuid import uuid4
 from nicegui import ui
 from transcription.config import get_settings
-from transcription.models import Document
+from transcription.db.models import Source
 PANGOZOOM_CDN_URL = "https://unpkg.com/@panzoom/[email protected]/dist/panzoom.min.js"
 UPLOADS_URL_PREFIX = "/uploads"
-def render_document_panzoom(*, document: Document) -> None:
+def render_document_panzoom(*, source: Source) -> None:
-    """Render a document preview with pan and zoom interactions."""
+    """Render a source preview with pan and zoom interactions."""
    _register_panzoom_assets()
    host_id = f"document-panzoom-{uuid4().hex}"
-    document_url = _document_url(document)
+    document_url = _document_url(source)
-    document_kind = _document_kind(document)
+    document_kind = _document_kind(source)
-    with ui.card().classes("w-full q-pa-md"):
+    with ui.card().classes("w-full q-pa-md ui-card-surface"):
        with ui.row().classes("w-full items-center justify-between no-wrap"):
            ui.label("Document preview").classes("text-subtitle1 text-weight-medium")
-            ui.label(document.filename).classes("text-caption text-grey-4 ellipsis").style(
+            ui.label(source.filename).classes("text-caption ui-text-muted ellipsis document-panzoom-filename")
                "max-width: 60%; text-align: right;"
            )
-        with (
+        with ui.element("div").classes("w-full document-panzoom-host q-mt-md") as host:
            ui.element("div").classes("w-full document-panzoom-host rounded-borders q-mt-md")
            # .style(f"height: {height};")
        ) as host:
            host.props(f"id={host_id}")
            with ui.element("div").classes("document-panzoom-surface"):
                if document_kind == "pdf":
                    ui.html(
                        f'<iframe class="document-panzoom-iframe" '
-                        f'src="{document_url}" title="{document.filename}" '
+                        f'src="{document_url}" title="{source.filename}" '
                        "data-panzoom-target></iframe>"
                    )
                else:
                    ui.html(
                        f'<img class="document-panzoom-media" '
-                        f'src="{document_url}" alt="{document.filename}" '
+                        f'src="{document_url}" alt="{source.filename}" '
                        "data-panzoom-target data-panzoom-media />"
                    )
@@ -59,47 +54,10 @@ def _register_panzoom_assets() -> None:
        f'<script src="{PANGOZOOM_CDN_URL}"></script>',
        shared=True,
    )
    ui.add_head_html(
        """
        <style>
            .document-panzoom-host {
                overflow: hidden;
                touch-action: none;
            }
            .document-panzoom-surface {
                width: 100%;
                height: 100%;
                display: flex;
                align-items: flex-start;
                justify-content: flex-start;
            }
            .document-panzoom-media {
                width: auto;
                height: auto;
                display: block;
                max-width: 100%;
                max-height: 100%;
                user-select: none;
                -webkit-user-drag: none;
            }
            .document-panzoom-iframe {
                width: 100%;
                height: 100%;
                border: 0;
                pointer-events: none;
                background: white;
            }
        </style>
        """,
        shared=True,
    )
-def _document_url(document: Document) -> str:
+def _document_url(source: Source) -> str:
-    file_path = Path(document.file_path)
+    file_path = Path(source.file_path)
    upload_dir = get_settings().upload_dir
    relative_path: Path
@@ -117,8 +75,8 @@ def _document_url(document: Document) -> str:
    return f"{UPLOADS_URL_PREFIX}/{encoded_relative_path}"
-def _document_kind(document: Document) -> str:
+def _document_kind(source: Source) -> str:
-    suffix = Path(document.file_path).suffix.lower()
+    suffix = Path(source.file_path).suffix.lower()
    if suffix == ".pdf":
        return "pdf"
    return "image"
@@ -26,7 +26,7 @@ def show_error(exc: Exception, *, title: str, operation: str) -> None:
        close_button="Dismiss",
    )
-    with ui.card().classes("bg-red-1 text-red-10 q-mt-md q-pa-md"):
+    with ui.card().classes("ui-card-error q-mt-md q-pa-md"):
        ui.label(title).classes("text-subtitle1")
        ui.label(error.message)
        ui.label(f"Suggested action: {error.suggestion}").classes("text-weight-medium")
@@ -0,0 +1,35 @@
 """Presentation-only formatting shared by archival UI surfaces."""
 import re
 from datetime import date
 from transcription.db.models import Person
 YEAR_PATTERN = re.compile(r"\b[12]\d{3}\b")
 def compact_date(exact: date | None, approximate: str | None) -> str:
    """Prefer an exact date, then an approximate value, then an unknown marker."""
    if exact is not None:
        return exact.isoformat()
    return (approximate or "").strip() or "Unknown"
 def person_selector_label(person: Person) -> str:
    """Build a readable selector label without treating names as identity."""
    preferred = (person.display_name or "").strip()
    full_name = person.full_name.strip()
    label = preferred if not preferred or preferred == full_name else f"{preferred} - {full_name}"
    if not label:
        label = full_name
    if person.birth_date is not None:
        return f"{label} ({person.birth_date.year})"
    approximate_year = YEAR_PATTERN.search(person.birth_date_raw or "")
    if approximate_year is not None:
        return f"{label} ({approximate_year.group(0)})"
    return label
 def family_search_url(family_search_id: str) -> str:
    """Build the fixed FamilySearch details URL for a validated identifier."""
    return f"https://www.familysearch.org/tree/person/details/{family_search_id}"
@@ -1,103 +0,0 @@
 """Reusable job detail rendering helpers."""
 from __future__ import annotations
 import logging
 from collections.abc import Sequence
 from nicegui import ui
 from transcription.models import Document
 from transcription.models import Job
 from transcription.models import Transcript
 from transcription.ui.components.document_panzoom import render_document_panzoom
 logger = logging.getLogger(__name__)
 def _status_chip_classes(status: str) -> str:
    if status == "queued":
        return "bg-blue-1 text-blue-10"
    if status == "processing":
        return "bg-amber-1 text-amber-10"
    if status == "transcribed":
        return "bg-green-1 text-green-10"
    if status == "failed":
        return "bg-red-1 text-red-10"
    return "bg-grey-2 text-grey-9"
 def _metadata_row(label: str, value: str) -> None:
    with ui.row().classes("w-full items-start justify-between q-gutter-x-md"):
        ui.label(label).classes("text-caption text-grey-5 text-uppercase w-28")
        ui.label(value).classes("text-body2 text-right text-grey-1 break-all")
 def _render_document_section(document: Document) -> None:
    with ui.card().classes("w-full q-pa-md"):
        ui.label("Document").classes("text-subtitle1 text-weight-medium")
        ui.separator().classes("q-my-sm")
        with ui.column().classes("w-full q-gutter-y-xs"):
            _metadata_row("Filename", document.filename)
            _metadata_row("File path", document.file_path)
        ui.separator().classes("q-my-md")
        render_document_panzoom(document=document)
 def _render_transcript_section(transcripts: Sequence[Transcript]) -> None:
    with ui.card().classes("w-full q-pa-md"):
        ui.label("Transcripts").classes("text-subtitle1 text-weight-medium")
        ui.separator().classes("q-my-sm")
        if not transcripts:
            ui.label("Transcript history is not available yet.").classes("text-body2 text-grey-3")
            return
        for transcript in transcripts:
            with ui.column().classes("w-full q-gutter-y-xs"):
                _metadata_row("Revision", str(transcript.revision))
                _metadata_row("Provider", transcript.provider)
                _metadata_row("Prompt", transcript.prompt_name)
                _metadata_row("Created", transcript.created_at.isoformat())
            if transcript.text:
                ui.separator().classes("q-my-sm")
                with ui.card().classes("w-fullq-pa-sm"):
                    ui.markdown(transcript.text).classes("text-grey-1")
            elif transcript.error_detail:
                ui.separator().classes("q-my-sm")
                with ui.card().classes("w-full bg-red-1 text-red-10 q-pa-sm"):
                    ui.label("Failure detail").classes("text-caption text-uppercase")
                    ui.label(transcript.error_detail).classes("text-body2")
            ui.separator().classes("q-my-md bg-blue-grey-7")
 def render_job_detail(*, job: Job, document: Document | None, transcripts: Sequence[Transcript]) -> None:
    """Render all sections for the job detail page."""
    logger.debug("Rendering job detail for job ID %s with %d transcripts", job.id, len(transcripts))
    status_text = job.status.value
    with ui.column().classes("w-full max-w-4xl q-gutter-md"):
        with ui.card().classes("w-full q-pa-lg"):
            with ui.row().classes("w-full items-center justify-between q-gutter-md"):
                with ui.column().classes("q-gutter-none"):
                    ui.label("Job overview").classes("text-h6 text-weight-bold")
                    ui.label(str(job.id)).classes("text-caption text-grey-5")
                status_chip_classes = (
                    "q-px-sm q-py-xs rounded-borders "
                    "text-weight-medium text-capitalize "
                    f"{_status_chip_classes(status_text)}"
                )
                ui.label(status_text).classes(status_chip_classes)
            ui.separator().classes("q-my-md bg-blue-grey-7")
            with ui.column().classes("w-full q-gutter-y-xs"):
                _metadata_row("Created", job.created_at.isoformat())
                _metadata_row("Updated", job.updated_at.isoformat())
                _metadata_row("Retries", str(job.retry_count))
        if document is not None:
            _render_document_section(document)
        _render_transcript_section(transcripts)
@@ -0,0 +1,38 @@
 from contextlib import contextmanager
 from nicegui import ui
@contextmanager
 def section_header_row(*, classes: str = ""):
    """Render a standardized section header row container."""
    base_classes = "w-full items-center justify-between pb-2 ui-header-divider"
    with ui.row().classes(f"{base_classes} {classes}".strip()) as row:
        yield row
 def render_empty_state(message: str, *, italic: bool = False, extra_classes: str = "") -> None:
    """Render standardized empty-state helper text."""
    classes = "text-xs ui-text-muted"
    if italic:
        classes = f"{classes} italic"
    ui.label(message).classes(f"{classes} {extra_classes}".strip())
 def destructive_button(
    label: str,
    *,
    on_click,
    icon: str,
    variant: str = "outlined",
    extra_classes: str = "",
 ):
    """Render a standardized destructive action button."""
    button = ui.button(label, on_click=on_click, icon=icon)
    if variant == "solid":
        button.props("unelevated color=negative")
    else:
        button.props("outlined color=negative")
    if extra_classes:
        button.classes(extra_classes)
    return button
@@ -11,19 +11,15 @@ logger = logging.getLogger(__name__)
 def _extract_row_id(args: Any) -> str | None:
-    if isinstance(args, dict):
+    if isinstance(args, list) and len(args) > 1 and isinstance(args[1], dict):
-        if isinstance(args.get("row"), dict):
+        return str(args[1].get("id")) if args[1].get("id") is not None else None
            row_id = args["row"].get("id")
            return str(row_id) if row_id is not None else None
        row_id = args.get("id")
        return str(row_id) if row_id is not None else None
-    if isinstance(args, list):
+    if isinstance(args, dict):
-        for value in args:
+        row = args.get("row")
-            if isinstance(value, dict):
+        if isinstance(row, dict) and "id" in row:
-                row_id = value.get("id")
+            return str(row["id"])
-                if row_id is not None:
+        if "id" in args:
-                    return str(row_id)
+            return str(args["id"])
    return None
@@ -40,7 +36,6 @@ def _bind_row_click_handler(
        on_row_click_id(row_id)
    table.on("rowClick", handle_row_click)
    logger.debug("Row click handler bound to table")
 def build_table(
@@ -49,25 +44,47 @@ def build_table(
    *,
    default_sort_by: str | None = None,
    default_descending: bool = False,
-    classes: str = "app-table",
+    classes: str = "",
    show_search: bool = True,
    search_placeholder: str = "Search records...",
    on_row_click_id: Callable[[str], None] | None = None,
 ) -> Any:
    """Build a styled Quasar table widget with optional client-side filtering and row-click handlers."""
    pagination: dict[str, Any] = {"rowsPerPage": 25}
    if default_sort_by is not None:
        pagination["sortBy"] = default_sort_by
        pagination["descending"] = default_descending
-    table = (
+    # Use a parent container to hold both the search bar and the table seamlessly
-        ui.table(
+    with ui.column().classes("w-full gap-2"):
-            rows=rows,
+        if show_search:
-            columns=columns,
+            with ui.row().classes("w-full items-center justify-end"):
-            row_key="id",
+                search_input = (
-            pagination=pagination,
+                    ui.input(placeholder=search_placeholder)
                    .props("dense outlined clearable icon=search")
                    .classes("w-64 text-xs ui-form-surface")
                )
        table = (
            ui.table(
                rows=rows,
                columns=columns,
                row_key="id",
                pagination=pagination,
            )
            .classes(f"w-full ui-table {classes}".strip())
            .props(
                'flat square binary-state-sort table-style="table-layout: fixed; width: 100%;" '
                'header-cell-class="ui-table-header text-xs uppercase tracking-wider" '
                'table-class="ui-table-body text-xs"'
            )
        )
-        .classes(classes)
+
-        .props('table-style="table-layout: fixed; width: 100%;"')
+        # Bind client-side text filter if search input is active
-    )
+        if show_search:
-    logger.info("Table built with %d rows and %d columns", len(rows), len(columns))
+            table.bind_filter_from(search_input, "value")
    if on_row_click_id is not None:
        _bind_row_click_handler(table, on_row_click_id=on_row_click_id)
    return table
@@ -0,0 +1,119 @@
 """Documents table rendering helpers."""
 from __future__ import annotations
 from collections.abc import Sequence
 from dataclasses import dataclass
 from typing import Any
 from uuid import UUID
 from nicegui import ui
 from transcription.ui.components.cards import archival_card
 from transcription.ui.components.primitives import render_empty_state
 from transcription.ui.components.table.common import build_table
@dataclass(frozen=True, slots=True)
 class DocumentTableRow:
    """Read model consumed by the documents table component."""
    id: UUID
    name: str
    document_type: str
    authors: str
    document_date: str
    archive_identifier: str
 def _serialize_rows(rows: Sequence[DocumentTableRow]) -> list[dict[str, Any]]:
    return [
        {
            "id": str(row.id),
            "name": row.name,
            "document_type": row.document_type or "Unspecified",
            "authors": row.authors or "Not set",
            "document_date": row.document_date,
            "archive_identifier": row.archive_identifier or "N/A",
        }
        for row in rows
    ]
 def render_documents_table(rows: Sequence[DocumentTableRow]) -> None:
    """Render documents table with search filtering and custom type chips."""
    if not rows:
        with archival_card(extra_classes="p-8 text-center"):
            render_empty_state("No documents found in repository.")
        return
    table = build_table(
        rows=_serialize_rows(rows),
        columns=[
            {
                "name": "name",
                "label": "Document Title",
                "field": "name",
                "sortable": True,
                "classes": "font-serif font-semibold text-left ui-table-cell-wrap",
                "align": "left",
                "style": "width: 30%;",
            },
            {
                "name": "document_type",
                "label": "Type",
                "field": "document_type",
                "sortable": True,
                "classes": "ui-table-cell-wrap",
                "align": "center",
                "style": "width: 14%;",
            },
            {
                "name": "authors",
                "label": "Author",
                "field": "authors",
                "sortable": True,
                "classes": "ui-table-cell-wrap",
                "align": "center",
                "style": "width: 22%;",
            },
            {
                "name": "document_date",
                "label": "Document Date",
                "field": "document_date",
                "sortable": True,
                "classes": "font-mono text-xs",
                "align": "center",
                "style": "width: 14%;",
            },
            {
                "name": "archive_identifier",
                "label": "Archive Ref",
                "field": "archive_identifier",
                "sortable": True,
                "classes": "font-mono text-xs",
                "align": "center",
                "style": "width: 20%;",
            },
        ],
        default_sort_by="name",
        search_placeholder="Search documents by title, type, or reference...",
        on_row_click_id=lambda doc_id: ui.navigate.to(f"/documents/{doc_id}"),
    )
    # Render document type using a subtle Quasar badge
    table.add_slot(
        "body-cell-document_type",
        r"""
        <q-td :props="props">
            <q-chip
                dense
                square
                size="sm"
                class="ui-chip-primary"
            >
                {{ props.value }}
            </q-chip>
        </q-td>
        """,
    )
@@ -11,6 +11,9 @@ from uuid import UUID
 from nicegui import ui
 from transcription.ui.components.cards import archival_card
 from transcription.ui.components.primitives import render_empty_state
 from .common import build_table
@@ -22,12 +25,12 @@ class JobTableRow:
    status: str
    filename: str
    retry_count: int
-    created_at: str
+    date_created: str
-    updated_at: str
+    date_updated: str
 def _format_timestamp(value: str) -> str:
-    """Return a friendly UTC timestamp for table display."""
+    """Return a friendly local timestamp for table display."""
    try:
        parsed = datetime.fromisoformat(value)
    except ValueError:
@@ -40,36 +43,87 @@ def _serialize_rows(rows: Sequence[JobTableRow]) -> list[dict[str, Any]]:
    return [
        {
            "id": str(row.id),
-            "status": row.status,
+            "status": row.status.lower(),
            "filename": row.filename,
            "retry_count": row.retry_count,
-            "created_at": _format_timestamp(row.created_at),
+            "date_created": _format_timestamp(row.date_created),
-            "updated_at": _format_timestamp(row.updated_at),
+            "date_updated": _format_timestamp(row.date_updated),
-            "created_sort": row.created_at,
+            "created_sort": row.date_created,
-            "updated_sort": row.updated_at,
+            "updated_sort": row.date_updated,
        }
        for row in rows
    ]
 def render_jobs_table(rows: Sequence[JobTableRow]) -> None:
-    """Render jobs table and open a detail page when clicking a row."""
+    """Render jobs table with search filtering and custom status chips."""
    if not rows:
-        ui.label("No jobs yet.")
+        with archival_card(extra_classes="p-8 text-center"):
            render_empty_state("No job records found in repository.")
        return
-    build_table(
+    table = build_table(
        rows=_serialize_rows(rows),
        columns=[
-            {"name": "id", "label": "Job ID", "field": "id", "sortable": True},
+            {
-            {"name": "status", "label": "Status", "field": "status", "sortable": True},
+                "name": "id",
-            {"name": "filename", "label": "Filename", "field": "filename", "sortable": True},
+                "label": "Job ID",
-            {"name": "retry_count", "label": "Retries", "field": "retry_count", "sortable": True},
+                "field": "id",
-            {"name": "created_at", "label": "Created", "field": "created_at", "sortable": True},
+                "sortable": True,
-            {"name": "updated_at", "label": "Updated", "field": "updated_at", "sortable": True},
+                "classes": "font-mono text-xs",
            },
            {
                "name": "status",
                "label": "Status",
                "field": "status",
                "sortable": True,
                "classes": "font-mono",
            },
            {
                "name": "filename",
                "label": "Source Filename",
                "field": "filename",
                "sortable": True,
                "classes": "font-mono text-xs",
            },
            {
                "name": "retry_count",
                "label": "Retries",
                "field": "retry_count",
                "sortable": True,
            },
            {
                "name": "date_created",
                "label": "Created",
                "field": "created_sort",
                "sortable": True,
            },
            {
                "name": "date_updated",
                "label": "Updated",
                "field": "updated_sort",
                "sortable": True,
            },
        ],
        default_sort_by="created_sort",
        default_descending=True,
-        classes="app-table w-full",
+        search_placeholder="Search jobs by ID, filename, or status...",
        on_row_click_id=lambda job_id: ui.navigate.to(f"/jobs/{job_id}"),
    )
    # Render job execution status using themed Quasar chips
    table.add_slot(
        "body-cell-status",
        r"""
        <q-td :props="props">
            <q-chip
                dense
                square
                size="sm"
                :class="`ui-status ui-status--${props.value}`"
            >
                {{ props.value.toUpperCase() }}
            </q-chip>
        </q-td>
        """,
    )
@@ -0,0 +1,109 @@
 """People table rendering helpers."""
 from __future__ import annotations
 from collections.abc import Sequence
 from dataclasses import dataclass
 from typing import Any
 from uuid import UUID
 from nicegui import ui
 from transcription.ui.components.cards import archival_card
 from transcription.ui.components.primitives import render_empty_state
 from transcription.ui.components.table.common import build_table
@dataclass(frozen=True, slots=True)
 class PersonTableRow:
    """Read model consumed by the people table component."""
    id: UUID
    full_name: str
    display_name: str
    maiden_name: str
    birth_date: str
    death_date: str
 def _serialize_rows(rows: Sequence[PersonTableRow]) -> list[dict[str, Any]]:
    return [
        {
            "id": str(row.id),
            "full_name": row.full_name,
            "display_name": row.display_name or "Not set",
            "maiden_name": row.maiden_name or "N/A",
            "birth_date": row.birth_date or "Unknown",
            "death_date": row.death_date or "Unknown",
        }
        for row in rows
    ]
 def render_people_table(rows: Sequence[PersonTableRow]) -> None:
    """Render people table with search filtering and custom typography."""
    if not rows:
        with archival_card(extra_classes="p-8 text-center"):
            render_empty_state("No person records found in repository.")
        return
    table = build_table(
        rows=_serialize_rows(rows),
        columns=[
            {
                "name": "full_name",
                "label": "Full Name",
                "field": "full_name",
                "sortable": True,
                "classes": "font-serif font-semibold text-left ui-table-cell-wrap",
            },
            {
                "name": "display_name",
                "label": "Display Name",
                "field": "display_name",
                "sortable": True,
                "classes": "ui-table-cell-wrap",
                "align": "center",
            },
            {
                "name": "maiden_name",
                "label": "Maiden Name",
                "field": "maiden_name",
                "sortable": True,
                "classes": "ui-table-cell-wrap",
                "align": "center",
            },
            {
                "name": "birth_date",
                "label": "Birth Date",
                "field": "birth_date",
                "sortable": True,
                "classes": "font-mono text-xs",
                "align": "center",
            },
            {
                "name": "death_date",
                "label": "Death Date",
                "field": "death_date",
                "sortable": True,
                "classes": "font-mono text-xs",
                "align": "center",
            },
        ],
        default_sort_by="full_name",
        search_placeholder="Search people by name, birth date, or death date...",
        on_row_click_id=lambda person_id: ui.navigate.to(f"/people/{person_id}"),
    )
    # Custom column template adding an archival entity icon next to person's name
    table.add_slot(
        "body-cell-full_name",
        r"""
        <q-td :props="props">
            <div class="row items-center q-gutter-x-xs">
                <q-icon name="person" size="xs" color="primary" />
                <span class="font-serif font-semibold">{{ props.value }}</span>
            </div>
        </q-td>
        """,
    )
@@ -0,0 +1,118 @@
 """Sources table rendering helpers."""
 from __future__ import annotations
 from collections.abc import Sequence
 from dataclasses import dataclass
 from typing import Any
 from uuid import UUID
 from nicegui import ui
 from transcription.ui.components.cards import archival_card
 from transcription.ui.components.primitives import render_empty_state
 from transcription.ui.components.table.common import build_table
@dataclass(frozen=True, slots=True)
 class SourceTableRow:
    """Read model consumed by the sources table component."""
    id: UUID
    page_number: int
    upload_name: str
    document_id: UUID
    document_name: str | None = None
    job_source_status: str | None = None
    job_source_error_detail: str | None = None
 def _serialize_rows(rows: Sequence[SourceTableRow]) -> list[dict[str, Any]]:
    return [
        {
            "id": str(row.id),
            "page_number": row.page_number,
            "upload_name": row.upload_name,
            "document_id": str(row.document_id),
            "document_name": row.document_name or "-",
            "job_source_status": row.job_source_status or "-",
            "job_source_error_detail": row.job_source_error_detail or "-",
        }
        for row in rows
    ]
 def render_sources_table(rows: Sequence[SourceTableRow]) -> None:
    """Render sources table and open detail page when clicking a row."""
    if not rows:
        with archival_card(extra_classes="p-8 text-center"):
            render_empty_state("No source asset records found in repository.")
        return
    table = build_table(
        rows=_serialize_rows(rows),
        columns=[
            {
                "name": "document_name",
                "label": "Document Name",
                "field": "document_name",
                "sortable": True,
                "classes": "font-serif text-left ui-table-cell-wrap",
                "align": "left",
                "style": "width: 27%;",
            },
            {
                "name": "page_number",
                "label": "Page Number",
                "field": "page_number",
                "sortable": True,
                "style": "width: 10%;",
            },
            {
                "name": "upload_name",
                "label": "Upload Title",
                "field": "upload_name",
                "sortable": True,
                "classes": "font-serif text-left ui-table-cell-wrap",
                "align": "left",
                "style": "width: 25%;",
            },
            {
                "name": "job_source_status",
                "label": "Status",
                "field": "job_source_status",
                "sortable": True,
                "classes": "font-mono",
                "align": "center",
                "style": "width: 14%;",
            },
            {
                "name": "job_source_error_detail",
                "label": "Error Detail",
                "field": "job_source_error_detail",
                "sortable": False,
                "classes": "font-mono text-xs text-left ui-text-muted ui-table-cell-wrap",
                "align": "left",
                "style": "width: 24%;",
            },
        ],
        default_sort_by="page_number",
        on_row_click_id=lambda source_id: ui.navigate.to(f"/sources/{source_id}"),
    )
    # Render job execution status using themed Quasar chips
    table.add_slot(
        "body-cell-job_source_status",
        r"""
        <q-td :props="props">
            <q-chip
                dense
                square
                size="sm"
                :class="`ui-status ui-status--${props.value}`"
            >
                {{ props.value }}
            </q-chip>
        </q-td>
        """,
    )
@@ -9,36 +9,67 @@ from typing import Any
 from nicegui import ui
-from transcription.models import Transcript
+from transcription.db.models import Job
 from transcription.db.models import Source
-type TranscriptAction = Callable[[Transcript], Awaitable[None] | None]
+type RevisionAction = Callable[[Source], Awaitable[None] | None]
-def render_transcript_revision_row(
+def render_original_transcription_card(*, job: Job, classes: str = "w-full") -> Any:
    """Render the immutable original job transcription output."""
    latest_error_detail = _latest_job_error_detail(job)
    status_label = "Failed" if latest_error_detail else "Transcribed"
    header = f"Original Transcription | {status_label}"
    provider = job.provider or "unknown"
    model = job.model or "unknown"
    caption = f"{provider} | {model} | {_format_created_at(job.date_updated)}"
    card = ui.card().classes(f"{classes} q-pa-md ui-card-surface")
    with card, ui.column().classes("w-full q-gutter-y-sm"):
        ui.label(header).classes("text-subtitle1 text-weight-medium")
        ui.label(caption).classes("text-caption ui-text-muted")
        _metadata_row(label="Prompt", value=_latest_job_prompt(job) or "unknown")
        _metadata_row(label="Updated", value=_format_created_at(job.date_updated))
        latest_transcription = _latest_job_transcription(job)
        if latest_transcription:
            with ui.card().classes("w-full q-pa-sm ui-card-surface"):
                ui.markdown(latest_transcription)
        if latest_error_detail:
            with ui.card().classes("w-full ui-card-error q-pa-sm"):
                ui.label("Failure detail").classes("text-caption text-uppercase")
                ui.label(latest_error_detail).classes("text-body2")
    return card
 def render_revision_row(
    *,
-    transcript: Transcript,
+    revision: Source | None,
    initially_expanded: bool = False,
    classes: str = "w-full",
-    on_delete: TranscriptAction | None = None,
+    on_delete: RevisionAction | None = None,
 ) -> Any:
-    """Render one collapsible row for a single transcript revision."""
+    """Render a collapsible row for the single optional source revision."""
-    status_label = "Failed" if transcript.error_detail else "Transcribed"
+    if revision is None:
-    header = f"Revision {transcript.revision} | {status_label}"
+        return None
    caption = f"{transcript.provider} | {transcript.model} | {_format_created_at(transcript.created_at)}"
-    expansion = ui.expansion(value=initially_expanded, group="group").classes(
+    header = "Source revision | User-authored"
-        f"{classes} rounded-borders bg-blue-grey-10"
+    caption = _format_created_at(revision.date_revised or revision.date_uploaded)
-    )
+
    expansion = ui.expansion(value=initially_expanded, group="group").classes(f"{classes} ui-card-surface")
    with expansion, ui.column().classes("w-full q-gutter-y-sm q-pa-sm"):
        with expansion.add_slot("header"), ui.row().classes("w-full items-start justify-between q-gutter-md"):
            with ui.column().classes("q-gutter-none"):
                ui.label(header).classes("text-subtitle1 text-weight-medium")
-                ui.label(caption).classes("text-caption text-grey-5")
+                ui.label(caption).classes("text-caption ui-text-muted")
            if on_delete is not None:
-                with ui.dialog() as delete_dialog, ui.card().classes("q-pa-md"):
+                with ui.dialog() as delete_dialog, ui.card().classes("q-pa-md ui-card-surface"):
-                    ui.label("Delete this transcript revision?").classes("text-body1")
+                    ui.label("Delete this source revision?").classes("text-body1")
                    with ui.row().classes("w-full justify-end q-gutter-sm"):
                        ui.button("Cancel", on_click=lambda: delete_dialog.submit(False)).props("flat")
                        ui.button("Delete", on_click=lambda: delete_dialog.submit(True)).props(
@@ -51,7 +82,7 @@ def render_transcript_revision_row(
                    if not confirmed:
                        return
-                    maybe_awaitable = on_delete(transcript)
+                    maybe_awaitable = on_delete(revision)
                    if isinstance(maybe_awaitable, Awaitable):
                        await maybe_awaitable
@@ -59,22 +90,36 @@ def render_transcript_revision_row(
                    ui.button(icon="delete", on_click=delete_current_transcript).props(
                        'flat round dense color="negative"'
                    )
-        _metadata_row(label="Provider", value=transcript.provider)
+        _metadata_row(label="Created", value=_format_created_at(revision.date_revised or revision.date_uploaded))
        _metadata_row(label="Model", value=transcript.model)
        _metadata_row(label="Created", value=_format_created_at(transcript.created_at))
-        if transcript.text:
+        if revision.revised_text:
-            with ui.card().classes("w-full q-pa-sm"):
+            with ui.card().classes("w-full q-pa-sm ui-card-surface"):
-                ui.markdown(transcript.text)
+                ui.markdown(revision.revised_text)
        if transcript.error_detail:
            with ui.card().classes("w-full bg-red-1 text-red-10 q-pa-sm"):
                ui.label("Failure detail").classes("text-caption text-uppercase")
                ui.label(transcript.error_detail).classes("text-body2")
    return expansion
 def _latest_job_transcription(job: Job) -> str | None:
    for job_source in sorted(job.job_sources, key=lambda item: item.executed_at, reverse=True):
        if job_source.raw_transcription:
            return job_source.raw_transcription
    return None
 def _latest_job_error_detail(job: Job) -> str | None:
    for job_source in sorted(job.job_sources, key=lambda item: item.executed_at, reverse=True):
        if job_source.error_detail:
            return job_source.error_detail
    return None
 def _latest_job_prompt(job: Job) -> str | None:
    for job_source in sorted(job.job_sources, key=lambda item: item.executed_at, reverse=True):
        if job_source.job and job_source.job.prompt_name:
            return job_source.job.prompt_name
    return None
 def _format_created_at(value: datetime) -> str:
    """Return a compact UTC-like timestamp for row captions."""
    return value.strftime("%Y-%m-%d %H:%M:%S %Z")
@@ -82,5 +127,5 @@ def _format_created_at(value: datetime) -> str:
 def _metadata_row(*, label: str, value: str) -> None:
    with ui.row().classes("w-md items-start justify-between q-gutter-x-md"):
-        ui.label(label).classes("text-caption text-grey-5 text-uppercase")
+        ui.label(label).classes("text-caption ui-text-muted text-uppercase")
-        ui.label(value).classes("text-body2 text-right text-grey-1 break-all")
+        ui.label(value).classes("text-body2 text-right break-all")
@@ -1,66 +0,0 @@
 """Reusable upload widget for document submission."""
 from __future__ import annotations
 from collections.abc import Awaitable
 from collections.abc import Callable
 from nicegui import ui
 from nicegui.binding import bindable_dataclass
 from nicegui.events import UploadEventArguments
 from transcription.errors import AppError
 from transcription.services.documents import UploadJobResult
 from transcription.ui.components.error_presenter import show_error
 from transcription.ui.components.error_presenter import summarize_error
 from transcription.worker import WorkerNotifier
 type UploadSubmitter = Callable[[str, bytes], Awaitable[UploadJobResult]]
@bindable_dataclass
 class UploadWidgetState:
    """Simple state container for upload feedback."""
    loading: bool = False
    message: str = ""
 def render_upload_widget(*, submitter: UploadSubmitter, notifier: WorkerNotifier | None = None) -> None:
    """Render upload controls and common status/error handling."""
    state = UploadWidgetState()
    status_label = ui.label("Upload a document to start transcription.")
    status_label.bind_text(state, "message")
    async def on_upload(event: UploadEventArguments) -> None:
        if state.loading:
            ui.notify("Upload already in progress. Please wait.", type="warning")
            return
        state.loading = True
        status_label.text = "Uploading..."
        try:
            payload = await event.file.read()
            result = await submitter(event.file.name, payload)
            job_id = result.job_id
            state.message = f"Created job {job_id}" if job_id is not None else "Upload complete"
            status_label.text = state.message
            if notifier is not None:
                notifier.notify()
            ui.notify(state.message, type="positive")
        except AppError as exc:
            state.message = summarize_error(exc, operation="upload.submit")
            status_label.text = f"Upload failed: {state.message}"
            show_error(exc, title="Upload failed", operation="upload.submit")
        except Exception as exc:  # noqa: BLE001
            state.message = summarize_error(exc, operation="upload.submit")
            status_label.text = f"Upload failed: {state.message}"
            show_error(exc, title="Upload failed", operation="upload.submit")
        finally:
            state.loading = False
    ui.upload(
        on_upload=on_upload,
        auto_upload=True,
        label="Select document file",
    ).props('accept=".jpg,.jpeg,.png,.tif,.tiff,.pdf"')
@@ -0,0 +1,21 @@
 """Simple media display components."""
 from nicegui import ui
 def dark_room_viewer(
    image_path: str | None,
    count_label: str = "1 Source Linked",
 ) -> None:
    """Render a plain responsive image that fills available width."""
    del count_label
    if image_path:
        ui.image(image_path).classes("w-full rounded-sm block ui-media-image")
        return
    with ui.column().classes(
        "w-full items-center justify-center border ui-border-viewer "
        "ui-bg-viewer-overlay-soft rounded-sm p-8 ui-media-placeholder"
    ):
        ui.label("No source media available for inspection.").classes("ui-text-muted text-xs italic")
@@ -0,0 +1,62 @@
 """File-backed storage helpers for the homepage content."""
 from __future__ import annotations
 from pathlib import Path
 HOME_PAGE_DIR = Path(__file__).resolve().parents[3] / "data" / "homepage"
 HOME_PAGE_MARKDOWN_PATH = HOME_PAGE_DIR / "homepage.md"
 SUPPORTED_IMAGE_SUFFIXES = {".jpg", ".jpeg", ".png", ".gif", ".webp", ".bmp", ".tif", ".tiff"}
 def ensure_homepage_storage() -> None:
    """Create the homepage storage directory when needed."""
    HOME_PAGE_DIR.mkdir(parents=True, exist_ok=True)
 def read_homepage_markdown() -> str:
    """Read the saved homepage markdown text."""
    ensure_homepage_storage()
    if not HOME_PAGE_MARKDOWN_PATH.exists():
        return ""
    return HOME_PAGE_MARKDOWN_PATH.read_text(encoding="utf-8")
 def save_homepage_markdown(markdown_text: str) -> None:
    """Persist the homepage markdown text."""
    ensure_homepage_storage()
    HOME_PAGE_MARKDOWN_PATH.write_text(markdown_text, encoding="utf-8")
 def store_homepage_image(*, filename: str, file_bytes: bytes) -> Path:
    """Persist an uploaded homepage image in the shared homepage folder."""
    ensure_homepage_storage()
    safe_name = Path(filename).name
    if not safe_name:
        msg = "Homepage image filename is required"
        raise ValueError(msg)
    stored_path = HOME_PAGE_DIR / safe_name
    stored_path.write_bytes(file_bytes)
    return stored_path
 def list_homepage_images() -> list[Path]:
    """List stored homepage images in the order they were last updated."""
    ensure_homepage_storage()
    image_paths = [
        path
        for path in HOME_PAGE_DIR.iterdir()
        if path.is_file() and path.suffix.lower() in SUPPORTED_IMAGE_SUFFIXES
    ]
    return sorted(image_paths, key=lambda path: (path.stat().st_mtime, path.name))
 def latest_homepage_image() -> Path | None:
    """Return the most recently updated homepage image, if one exists."""
    image_paths = list_homepage_images()
    if not image_paths:
        return None
    return image_paths[-1]
--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
Jim Lancaster	63373bf24d	V4.4 Scope defined (part 2)	2026-08-15 14:04:58 -05:00
Jim Lancaster	936af9b1d3	V4.4 Scope defined	2026-08-15 14:04:42 -05:00
Jim Lancaster	a78b58ff40	V4.3 revision to Document Types	2026-08-15 13:29:53 -05:00
Jim Lancaster	aed827babe	Finalized v4.3 scope	2026-08-14 16:17:23 -05:00
Jim Lancaster	178347e086	Updated v4.3 scope	2026-08-14 16:08:07 -05:00
Jim Lancaster	c9f5dca064	V4.2 complete	2026-08-14 15:59:38 -05:00
Jim Lancaster	6bd4cbb0a7	V4.2 Updated what ai_raw_response data is being captured. The changes were more extensive than I expected.	2026-08-14 07:21:15 -05:00
Jim Lancaster	28811d79ce	V4.1 major revision to docs. Removed all obsolete documents, updated v4.2 implementation scope and plan.	2026-08-13 15:32:40 -05:00
Jim Lancaster	171132919d	V4.1 revisions in preparation for v4.2. AI data capture now better defined.	2026-08-12 18:51:48 -05:00
Jim Lancaster	89cf69f8a2	V4.1 Mostly UI adjustments by GC	2026-08-12 13:11:50 -05:00
Jim Lancaster	1e8d8572d4	Continue GC code review: Pydantic	2026-08-12 01:35:50 -05:00
Jim Lancaster	888a8c380a	Continue GC code review: UI	2026-08-11 16:54:03 -05:00
Jim Lancaster	b8be27f0c9	Continue GC code review and cleanup	2026-08-11 16:42:09 -05:00
Jim Lancaster	8d5aec4301	Github Copilot service realignment & cleanup	2026-08-11 16:02:19 -05:00
Jim Lancaster	0ace10269f	V4 implemented. Some tweaking left, but it is working	2026-08-11 12:07:19 -05:00
Jim Lancaster	ccf2c78ff4	V4 final docs	2026-08-10 12:34:36 -05:00
Jim Lancaster	4b3baf5a3e	Revised and simplified V4 Plan and core documents.	2026-08-10 10:53:13 -05:00
Jim Lancaster	9b4d6f0340	Delete V3 duplicates	2026-08-09 13:34:36 -05:00
Jim Lancaster	5753eb0135	V4 plan created: Adding many-to-many links between Documents & People in the UI. Also adding some new tables for Document Type, Person role.	2026-08-09 13:29:50 -05:00
Jim Lancaster	e6549277c6	V3 Updated V3 core documents. Added data folder backup/restore before/after running destructive tests.	2026-08-09 10:51:30 -05:00
Jim Lancaster	b59d3da23e	V3 fix document delete issue	2026-08-08 21:58:19 -05:00
Jim Lancaster	4bf6c9e2f3	V3 post step 2 refinement: add temperature & top-p settings to config (and .env), add prompt fields back to job table so that the prompt settings get frozen at runtime for all sources being processed.	2026-08-08 18:21:44 -05:00
Jim Lancaster	4dac9349c1	V3 step 1 update models.py and step 2 implement service/worker, and raw API response persistence	2026-08-08 18:08:04 -05:00
Jim Lancaster	5a741de0a9	Updated documentation to v3 which will focus on capturing prompt/response interactions with AI.	2026-08-08 16:16:32 -05:00
Jim Lancaster	58faa00d7b	AI metadata and api prompt results data capture now fixed	2026-08-08 14:59:45 -05:00
Jim Lancaster	89cac3c378	Removed the image viewer which wasn't working anyway.	2026-08-08 09:22:26 -05:00
Jim Lancaster	4b5e7ac23e	Fixed Source Detail page	2026-08-08 08:21:35 -05:00
Jim Lancaster	21a7e83563	Tweak empty page settings to make them more uniform.	2026-08-07 09:10:03 -05:00
Jim Lancaster	fce7107863	Remove all references to "uploads" page	2026-08-06 16:53:21 -05:00
Jim Lancaster	5090e238ff	GLobal theme cleanup	2026-08-06 15:12:20 -05:00
Jim Lancaster	75f263c2b6	Unit testing fixed? So says Copilot	2026-08-05 19:52:40 -05:00
Jim Lancaster	be152a028e	Continue troubleshooting unit tests. I think it is time to let Copilot have a crack at it.	2026-08-05 18:33:44 -05:00
Jim Lancaster	9219adaf0c	Fix unit test errors	2026-08-05 13:27:30 -05:00
Jim Lancaster	fd3ca60008	Revamped the Documents, People, & Jobs too.	2026-08-05 13:05:41 -05:00
Jim Lancaster	72bc96ab3a	Revamped Sources related pages with the help of Gemini, which had a lot to say.	2026-08-05 12:35:54 -05:00
Jim Lancaster	4eeb552273	Added Home page	2026-08-04 19:20:33 -05:00
Jim Lancaster	d9f5fbb1a4	Delete Document & Delete Source buttons now delete the underlying files	2026-08-04 18:34:35 -05:00
Jim Lancaster	271633d1d5	Jobs: jobs still stuck in queue. Fixes from testing.	2026-08-04 18:09:31 -05:00
Jim Lancaster	6c6589d8ff	Jobs: big jobs stuck in queue. Added Cancel, Resubmit	2026-08-04 09:03:51 -05:00
Jim Lancaster	759d4c2434	UI style refresh: Very close!!!	2026-08-03 19:46:32 -05:00
Jim Lancaster	323f12d911	UI style refresh: Final cleanup	2026-08-03 16:27:02 -05:00
Jim Lancaster	f80834d589	UI style refresh: extract reusable components and refactor	2026-08-03 15:28:58 -05:00
Jim Lancaster	752346025b	UI style refresh continued	2026-08-03 15:15:58 -05:00
Jim Lancaster	4f6e1fd913	UI style refresh with Gemini's help	2026-08-03 13:54:24 -05:00
Jim Lancaster	47aef0e26e	UI slog grinds on	2026-08-02 23:44:53 -05:00
Jim Lancaster	c098013a68	UI slog continues	2026-08-02 20:03:02 -05:00
Jim Lancaster	49e2e48df1	UI updates continue. Focus on Sources	2026-08-02 18:41:09 -05:00
Jim Lancaster	0ab7ad50f2	UI updates, changes sync'd to UI docs	2026-08-02 18:20:38 -05:00
Jim Lancaster	9653060c2a	UI update complete?	2026-08-02 13:33:09 -05:00
Jim Lancaster	ed6f9dfe25	UI update planning complete	2026-08-02 11:33:19 -05:00
Jim Lancaster	5946867ff3	UI update initial phase complete. Still need to create schema-mapping for the two many-to-many tables.	2026-08-02 11:23:04 -05:00
Jim Lancaster	646a360aca	UI update planning continued	2026-08-02 10:00:03 -05:00
Jim Lancaster	2b3d33e50e	Begin UI update starting with Document table.	2026-08-02 08:02:58 -05:00
Jim Lancaster	dfe6f121ff	V2 (new) sStep 4 complete. (untested, unreviewed, cross my fingers)	2026-08-01 18:28:00 -05:00
Jim Lancaster	51ac2d0b98	V2 step 3 complete	2026-08-01 16:24:44 -05:00
Jim Lancaster	61cc8a200b	V2 step 2 complete	2026-08-01 16:17:38 -05:00
Jim Lancaster	c46d1bd0bc	V2 implementation step 1	2026-08-01 15:34:06 -05:00
Jim Lancaster	00ed176ac1	Merge branch 'session-engine' of https://gitea.john-stream.com/bbchops/transcription into session-engine	2026-08-01 14:47:39 -05:00
Jim Lancaster	99a128e981	Reorganized docs, checked docs for internal consistency and made adjustments	2026-08-01 14:47:29 -05:00
John Lancaster	aa94f34de4	pruned ddl	2026-08-01 10:35:21 -05:00
John Lancaster	661e2b1bec	smoothed readme and startup	2026-08-01 09:51:21 -05:00
John Lancaster	d75083a666	shutdown fixes	2026-08-01 09:36:34 -05:00
John Lancaster	d0a3ca0289	WIP theming	2026-07-31 19:34:12 -05:00
John Lancaster	209c48987c	separated cli settings	2026-07-31 19:23:12 -05:00
John Lancaster	4ed1f43eda	ui instructions	2026-07-31 16:01:23 -05:00
John Lancaster	ce8fcce6b0	Merge commit '4ae8e5be4f60059ba611ceea0cf553b009e6a88e' into session-engine	2026-07-31 15:40:20 -05:00
Jim Lancaster	4ae8e5be4f	Test UI diagrams	2026-07-31 11:35:37 -05:00
John Lancaster	6b5b0500b3	unified implementation plan	2026-07-31 10:28:47 -05:00
John Lancaster	bbf7fe28c2	cleanup	2026-07-31 10:20:02 -05:00
John Lancaster	c4d25c1be8	Merge remote-tracking branch 'origin/doc_update' into session-engine	2026-07-31 10:16:17 -05:00
John Lancaster	1fa5eb1127	doc updates for pydantic	2026-07-31 10:12:34 -05:00
Jim Lancaster	3eefc36239	Update V1 & V2 core documents and reorganize docs folder	2026-07-31 10:04:07 -05:00
John Lancaster	ec6617a1c4	updates	2026-07-30 23:16:17 -05:00
John Lancaster	9eb0f40c08	session and engine	2026-07-30 22:33:49 -05:00
John Lancaster	f769d29da1	uv.lock update	2026-07-30 21:17:09 -05:00
John Lancaster	8afc462a6d	startup	2026-07-30 21:16:48 -05:00
John Lancaster	3d6daec561	moved models to db pkg	2026-07-30 21:10:13 -05:00
John Lancaster	1cc2f319d5	uvicorn startup	2026-07-30 21:09:51 -05:00
Jim Lancaster	d2b793ea69	Begin planning V2	2026-07-30 19:39:12 -05:00
Jim Lancaster	a975ca299a	Trouble shooting PDF transcriptions	2026-07-29 18:58:52 -05:00
Jim Lancaster	a3b3bab571	V1 mostly complete except for some testing. Linting in the last step changed nearly every file which is why this commit is so larger.	2026-07-29 17:27:21 -05:00
Jim Lancaster	bc21a97019	Updated test suite	2026-07-29 16:20:46 -05:00
Jim Lancaster	0973311d9f	Update documentation for consistency and refactor the code. An unresolved error in testing still exists.	2026-07-29 14:12:18 -05:00
Jim Lancaster	eaf9805121	Updates to docs. Added new transcription_methodology, revised approach to revisions: 1 revision per document (that can be updated)	2026-07-29 13:29:44 -05:00
Jim Lancaster	ec61013b47	Used co-pilot for complete review of all documentation, including extensive revision of v1.md	2026-07-02 12:37:32 -05:00
Jim Lancaster	97cb7055d4	Updated models.py	2026-07-02 10:23:06 -05:00
Jim Lancaster	f975e25093	minor update to docs	2026-07-01 12:59:50 -05:00
Jim Lancaster	90ba8fefdd	Update docs after db restructure	2026-07-01 12:57:13 -05:00