V4 plan created: Adding many-to-many links between Documents & People in the UI. Also adding some new tables for Document Type, Person role.

docs/ver3/architecture_v3.md

@@ -0,0 +1,142 @@
+# System Architecture (Version 3)
+
+This document describes the V3 production architecture of the personal historical-document transcription system.
+
+## Architecture Objectives
+
+* Preserve source material as immutable transcribed text alongside page-level spatial AI metadata and complete provider API envelopes.
+* Support batching multi-image and folder uploads cleanly into sequential pages (`page_number`).
+* Capture complete input prompt provenance (`system_prompt`, `user_prompt`, `prompt_hash`) and execution parameters (`temperature`, `top_p`) at submission time on `Job`.
+* Leverage asynchronous worker pools (`asyncio`) for parallel single-image API execution bounded by rate limiters (`asyncio.Semaphore`).
+* Maintain relational data-model portability across the supported backends by using SQLModel/SQLAlchemy and compatibility types so the same domain schema works in SQLite for local development/testing and PostgreSQL in production.
+* Keep operator tooling and local maintenance workflows OS-independent by using Python or other cross-platform interfaces for canonical project automation.
+* Verify image asset integrity via SHA-256 file hashing (`file_hash`) while storing binary assets on the local filesystem.
+* Standardize all data validation, API parsing, and database models on **Pydantic V2** and **SQLModel**.
+* Support rich historical attribution (multi-author and multi-recipient relationships via `DocumentPerson`).
+
+## Runtime Topology
+
+The V3 runtime operates as an asynchronous Python application:
+
+* FastAPI + NiceGUI web application process.
+* In-process `asyncio` background task orchestrator for parallel API execution.
+* Relational persistence via SQLModel / SQLAlchemy, using SQLite for local development/testing and PostgreSQL as the production persistence target.
+* Pydantic V2 validation layer wrapping API payloads, prompt configurations, and JSON metadata schemas.
+* Cross-platform operator workflows implemented in Python so core local operations run consistently on Windows, Linux, and macOS.
+
+^^^mermaid
+flowchart LR
+U[Browser User] --> A[FastAPI + NiceGUI App]
+A --> W[Asyncio Worker Engine]
+A --> DB[(Relational DB\nSQLite / PostgreSQL)]
+W --> P[Vision Provider APIs\nOpenAI / Claude / OpenRouter]
+W --> DB
+^^^
+
+## Lifecycle Ownership
+
+Application lifespan owns runtime setup/teardown:
+
+* Initialize environment logging, directory paths, and Pydantic configuration.
+* Manage asynchronous database engine connection pools (`aiosqlite` or `asyncpg`).
+* Execute database bootstrap (`SQLModel.metadata.create_all()`) or migrations.
+* Recover stale or interrupted processing jobs on startup.
+* Manage graceful shutdown of active `asyncio` worker pools.
+
+## Layered Module Structure
+
+### Interface Layer
+
+* `src/transcription/ui/**` (NiceGUI pages, multi-page renderers, person cards)
+* `src/transcription/api/**` (FastAPI routes and JSON error handlers)
+
+### Application & Async Worker Layer
+
+* `src/transcription/services/workflows.py`
+* `src/transcription/worker.py`
+
+Responsibilities:
+
+* Batch orchestration and status transitions (`queued` -> `processing` -> `transcribed` | `partial_success` | `failed`).
+* Parallel single-image API execution using `asyncio.gather` bounded by `asyncio.Semaphore`.
+* Resolve prompt configuration at submission time and persist frozen snapshot fields on `Job`.
+* Pydantic schema parsing and validation prior to database storage.
+
+### Domain & Service Layer
+
+* `src/transcription/db/models.py` (SQLModel schema definitions for Document, Source, Job, JobSource, Person, DocumentPerson)
+* `src/transcription/services/*.py` (Transactional operations for `Document`, `Person`, `Source`, `Job`, and `JobSource`)
+
+### Infrastructure Layer
+
+* `src/transcription/db/**` (Async database session factory, engine creation, and JSON dialect abstractions)
+* `src/transcription/providers/**` (OpenAI, Anthropic, and OpenRouter Vision SDK adapters)
+
+## Processing Workflow
+
+1. User uploads a folder or batch of images for a `Document`.
+2. System hashes each image file (SHA-256), writes image files to filesystem storage, and creates `Document`, `Job(status='queued')`, and ordered `Source` pages (`page_number = 1..N`).
+3. Worker claims job, sets `Job.status = 'processing'`, and spawns parallel `asyncio` tasks bounded by semaphore.
+4. Each task reads the frozen prompt snapshot from `Job` and calls Vision API for a **single** `Source` image.
+5. On task completion:
+* Writes a `JobSource` record containing `status='transcribed'`, `raw_transcription`, operational `ai_metadata`, and complete unedited `raw_api_response`.
+* Caches active output text to `Source.raw_transcription`.
+
+
+6. On page failure:
+* Writes `JobSource` record with `status='failed'` and `error_detail`.
+
+
+7. Once all page tasks resolve:
+* Marks `Job.status` as `transcribed` (100% success), `partial_success` (at least 1 success, 1 failure), or `failed` (all failed).
+
+
+
+## Domain Ownership & Invariants
+
+* **Immutable AI Outputs:** `source.raw_transcription` and `job_source.raw_transcription` store original, point-in-time machine output and are immutable.
+* **Complete Input & Output Provenance:** Every `job` stores the exact frozen input configuration sent to the model, and every `job_source` stores per-page output evidence including the complete REST response envelope returned.
+* **Inlined Revisions:** Human corrections occur on `source.revised_text`. UI renders `COALESCE(revised_text, raw_transcription)`.
+* **Sequential Integrity:** Multi-page documents are strictly ordered by `source.page_number ASC`.
+* **Page Execution Isolation:** A failure on one page image does not invalidate successful transcriptions on sister pages in the same batch job.
+
+## Data Model Summary
+
+* `Document` has many `Source` pages, many `Job` runs, and many `Person` records via `DocumentPerson` junction (`author` or `recipient`).
+* `Source` belongs to one `Document` and can be processed across many `JobSource` executions.
+* `Job` has many `JobSource` execution records.
+* `JobSource` holds page-level execution status, output text, and raw response JSON.
+
+## Test Strategy
+
+* Unit tests for SQLModel/Pydantic V2 models, JSON cross-dialect serialization, and file hashing functions.
+* Integration tests for async database connection handling, session management, and queries.
+* Async workflow tests using mock AI providers to verify `partial_success`, page-level failure isolation, and retry logic.
+* UI integration tests for multi-page rendering and person attribution management.
+
+---
+
+## Technology References
+
+* [FastAPI documentation](https://fastapi.tiangolo.com/)
+* [NiceGUI documentation](https://nicegui.io/documentation)
+* [SQLModel documentation](https://sqlmodel.tiangolo.com/)
+* [SQLAlchemy Async I/O documentation](https://docs.sqlalchemy.org/en/20/orm/extensions/asyncio.html)
+* [Python asyncio](https://www.google.com/search?q=https://docs.python.org/3/library/asyncio.html%23module-asyncio)
+* [Pydantic Validation](https://pydantic.dev/docs/validation/latest/get-started/)
+
+## Related Local References
+
+- [System Overview](index_v3.md)
+- [System Design Intent](invariant/intent.md)
+- [Transcription Methodology](invariant/transcription_methodology.md)
+- System Architecture (this document)
+- [System Requirements](requirements_v3.md)
+- [Data model](schema_v3.md)
+- [Error Handling Policy](error_handling_v3.md)
+- [Implementation Plan](implementation_plan_v3.md)
+
+
+
+
+

docs/ver3/error_handling_v3.md

@@ -0,0 +1,87 @@
+# Error Handling Policy (Version 3)
+
+This document defines the canonical error-handling policy for the v3 document transcription system.
+
+## Error Handling Objectives
+
+* Make failures visible in clear, actionable language at both the document and individual page levels.
+* Support **isolated failure handling** in multi-image batches so single page errors do not crash an entire batch job.
+* Preserve diagnostic detail (Pydantic validation errors, raw provider REST envelopes, exact input prompts) in generic database JSON structures for fast troubleshooting.
+* Ensure consistent error envelope structure across API, UI, and async worker boundaries.
+
+## Scope And Authority
+
+Governs error behavior across NiceGUI pages, FastAPI routes, service orchestration, `asyncio` background tasks, database interactions, and AI provider adapters.
+
+## Error Taxonomy
+
+| Category | Definition | Retriable |
+| --- | --- | --- |
+| `validation_error` | Pydantic payload or parameter schema validation failure | no |
+| `user_input_error` | Unacceptable user file (unsupported image type, corrupt file) | no |
+| `not_found_error` | Requested resource (`Document`, `Source`, `Person`, `Job`) missing | no |
+| `conflict_error` | Operation violates state constraints (e.g., duplicate `document_person` role) | no |
+| `external_provider_error` | AI Provider API failure (rate limit, vision execution error) | yes |
+| `infrastructure_transient_error` | Temporary DB connection reset or HTTP timeout | yes |
+| `infrastructure_persistent_error` | Database down, missing API credentials, misconfiguration | no |
+| `internal_unexpected_error` | Uncaught Python exception or logic defect | no |
+
+## Async Batch & Page-Level Error Behavior
+
+In multi-image `asyncio` batch processing:
+
+1. **Page Isolation:** Exceptions caught during individual page calls are trapped within the `asyncio` task wrapper.
+2. **Page Record Logging:** Page failure details, along with the prompt inputs and hyperparameters attempted, are written directly to `job_source.error_detail` and `job_source.status = 'failed'`.
+3. **Batch Aggregate State:**
+* If **all** page tasks succeed -> `job.status = 'completed'`.
+* If **some** page tasks fail -> `job.status = 'partial_success'`.
+* If **all** page tasks fail -> `job.status = 'failed'`.
+
+
+4. **Retry Strategy:** The UI exposes a "Retry Failed Pages" option for `partial_success` jobs, which spawns a new targeted `Job` containing *only* the `Source` IDs marked as `failed`.
+
+## API Error Response Contract
+
+API error responses return a structured JSON envelope:
+^^^json
+{
+"error_id": "err_uuid_12345",
+"category": "validation_error",
+"message": "The uploaded payload failed schema validation.",
+"suggestion": "Check file format and metadata fields, then try again.",
+"details": {
+"pydantic_errors": [...]
+},
+"timestamp": "2026-08-08T15:00:00Z"
+}
+^^^
+
+HTTP Status Mappings:
+
+* `validation_error`, `user_input_error` -> `400`
+* `not_found_error` -> `404`
+* `conflict_error` -> `409`
+* `external_provider_error` -> `502` / `503`
+* `infrastructure_transient_error` -> `503`
+* `infrastructure_persistent_error`, `internal_unexpected_error` -> `500`
+
+---
+
+## Technology References
+
+* [FastAPI documentation](https://fastapi.tiangolo.com/)
+* [NiceGUI documentation](https://nicegui.io/documentation)
+* [SQLModel documentation](https://sqlmodel.tiangolo.com/)
+* [Python asyncio](https://www.google.com/search?q=https://docs.python.org/3/library/asyncio.html%23module-asyncio)
+* [Pydantic Validation](https://pydantic.dev/docs/validation/latest/get-started/)
+
+## Related Local References
+
+- [System Overview](index_v3.md)
+- [System Design Intent](invariant/intent.md)
+- [Transcription Methodology](invariant/transcription_methodology.md)
+- [System Architecture](architecture_v3.md)
+- [System Requirements](requirements_v3.md)
+- [Data model](schema_v3.md)
+- Error Handling Policy (this document)
+- [Implementation Plan](implementation_plan_v3.md)

docs/ver3/implementation_plan_v3.md

@@ -0,0 +1,81 @@
+# Implementation Plan (Version 3)
+
+## Goal
+
+Replace the current v2 SQLModel schema with the approved v3 schema and make sure every database operation works through the existing async SQLAlchemy/SQLModel session layer.
+
+Use a fresh database. There will be no migrations, data conversion, legacy compatibility shims, or parallel v2/v3 code paths.
+
+## Current Project Impact
+
+* `src/transcription/db/models.py` defines the SQLModel tables. It must be updated to match the approved v3 schema (`Document`, `Person`, `DocumentPerson`, `Source`, `Job`, `JobSource`).
+* The v3 target adds frozen submission-time prompt snapshot fields (`prompt_name`, `prompt_hash`, `system_prompt`, `user_prompt`, `temperature`, `top_p`) to `Job` and full output payloads (`raw_api_response`, `ai_metadata`) to `JobSource`.
+* The v3 target adds image asset verification fields (`file_hash`, `file_size_bytes`) to `Source`.
+* Database operations must utilize `JSONBCompat` and the existing SQLModel/SQLAlchemy abstractions to preserve the same logical schema and JSON behavior across the supported backends, while keeping PostgreSQL as the intended production database.
+* Async CRUD lives in `DocumentService`, `JobService`, `TranscriptionService`, and upload helpers. Their queries and relationship loading must be updated for v3 fields.
+* Canonical operator tooling must remain OS-independent; safety workflows such as destructive-test backup and restore should run through Python or other cross-platform entry points rather than platform-specific shells.
+
+## Implementation
+
+### 1. Update the Schema and Domain Models
+
+* Replace the models in `src/transcription/db/models.py` with the approved v3 tables, enums, relationships, foreign keys, constraints, and indexes.
+* Ensure all JSON fields use `JSONBCompat` for dialect portability across SQLite and PostgreSQL.
+* Keep `SQLModel.metadata.create_all()` as the schema bootstrap for fresh databases.
+* Delete `_ensure_sqlite_compat_columns()` and all legacy schema patching from `src/transcription/db/operations.py`.
+* Keep the Python models and `docs/schema_v3.md` perfectly synchronized.
+
+### 2. Update Data Services and Async Worker Layer
+
+* Update job creation and worker orchestration so prompt configuration is resolved at submission and frozen onto `Job` (`prompt_name`, `prompt_hash`, `system_prompt`, `user_prompt`, `temperature`, `top_p`) before execution starts.
+* Update `TranscriptionService` and provider adapters to store the complete unedited API REST response dictionary into `job_source.raw_api_response` alongside operational metrics in `job_source.ai_metadata`.
+* Update upload handlers to calculate and store file metadata (`file_hash` via SHA-256, `file_size_bytes`) on `Source` records during file ingestion.
+* Remove legacy single-source compatibility flows so worker paths persist per-page outcomes only through `JobSource` updates.
+
+### 3. Update Integration Tests and Mock AI Providers
+
+* Update mock provider fixtures in test suites to return realistic complete API response envelopes.
+* Verify test coverage for `JSONBCompat` field writes and reads under SQLite in-memory test databases.
+* Add assertions in async workflow tests to verify frozen prompt snapshot fields on `Job`, plus per-page failure isolation and output evidence on `JobSource`.
+
+### 4. Update the UI for the v3 Schema
+
+* Review the UI components and views displaying document, job, person, and source data so they reference v3 schema properties instead of v2 relationships.
+* Ensure the UI correctly renders `COALESCE(revised_text, raw_transcription)` for page viewing and inline editing.
+* Ensure resubmit actions only queue failed pages and preserve frozen prompt snapshot behavior on the existing `Job`.
+* Consider the guidance in `docs/ui_style_guide.md` when making UI changes so updated views remain consistent with the project’s visual conventions.
+
+### 5. Keep Operational Tooling Portable
+
+* Implement destructive-test backup and restore workflows in Python so the canonical path runs on Windows, Linux, and macOS.
+* Avoid making core developer or recovery procedures depend on PowerShell-only or shell-specific semantics.
+* Keep operational documentation aligned with the cross-platform command path used by the repository.
+
+## Done When
+
+* A fresh database is created directly from the v3 SQLModel metadata.
+* Frozen prompt input provenance is captured on `Job` for each submission, and full per-page output evidence is captured on `JobSource` for every AI execution task.
+* The focused tests and full test suite pass on both SQLite and PostgreSQL backends.
+* Canonical operator workflows required for development and destructive-test recovery run without a Windows-only shell dependency.
+
+## Out of Scope
+
+* Database migrations or preservation of v2 data
+* Legacy compatibility code
+* UI redesign, batch orchestration, worker concurrency, deployment, and operational runbooks
+ 
+---
+
+## Related Local References
+
+- [System Overview](index_v3.md)
+- [System Design Intent](invariant/intent.md)
+- [Transcription Methodology](invariant/transcription_methodology.md)
+- [System Architecture](architecture_v3.md)
+- [System Requirements](requirements_v3.md)
+- [Data model](schema_v3.md)
+- [Error Handling Policy](error_handling_v3.md)
+- Implementation Plan (this document)
+
+
+

docs/ver3/index_v3.md

@@ -0,0 +1,49 @@
+# Document Transcription System Overview (Version 3)
+
+This project is a personal-scale application for transcribing, indexing, and preserving historical family documents, letters, postcards, and journals.
+
+## Start Here
+
+Read [architecture_v3.md](https://www.google.com/search?q=architecture_v3.md) first for technical overview and system design.
+
+## Core V3 Capabilities
+
+* **Folder & Multi-Image Ingestion:** Upload one or more images that map sequentially (`page_number`) under a single `Document`.
+* **Parallel Async AI Vision Engine:** Concurrently process single-page image transcriptions using Python `asyncio` bounded by rate limiters.
+* **Portable Relational Storage:** SQLModel and SQLAlchemy preserve a portable relational model across the supported backends, with SQLite for local development/testing and PostgreSQL as the production database target.
+* **Cross-Platform Operations:** Canonical developer and recovery workflows run through Python-based, OS-independent tooling rather than platform-specific shell scripts.
+* **Complete Auditability & Provenance:** Capture frozen submission-time input prompts (`system_prompt`, `user_prompt`) and hyperparameters (`temperature`, `top_p`) on `Job`, plus per-page operational metrics (`ai_metadata`) and full provider response envelopes (`raw_api_response`) on `JobSource`.
+* **Asset Integrity Tracking:** Calculate and store cryptographic hashes (SHA-256) and file sizes on `Source` image records while preserving clean filesystem storage.
+* **Pydantic V2 Validation:** End-to-end type safety, DB row mapping, and JSON payload validation.
+* **Historical Person Management:** Track authors and recipients across documents with rich biographical entities (`Person`).
+* **Page-Level Execution Auditing & Revisions:** Store immutable point-in-time machine output per run while enabling inline human corrections (`revised_text`).
+* **Partial Failure Recovery:** Bounded batch execution that isolates single-page API errors (`partial_success`) for simple retries.
+
+## Technical Stack
+
+* **Application Web Framework:** FastAPI + NiceGUI
+* **Persistence Engine:** SQLModel / SQLAlchemy (SQLite for development/testing, PostgreSQL for production)
+* **Data Validation & Schemas:** Pydantic V2
+* **Concurrency & Workers:** Python `asyncio` worker pool with `asyncio.Semaphore`
+* **Vision Providers:** OpenAI, Anthropic, and OpenRouter Vision models via native SDK adapters
+
+---
+
+## Technology References
+
+* [FastAPI documentation](https://fastapi.tiangolo.com/)
+* [NiceGUI documentation](https://nicegui.io/documentation)
+* [SQLModel documentation](https://sqlmodel.tiangolo.com/)
+* [Python asyncio](https://www.google.com/search?q=https://docs.python.org/3/library/asyncio.html%23module-asyncio)
+* [Pydantic Validation](https://pydantic.dev/docs/validation/latest/get-started/)
+
+## Documentation Index
+
+- System Overview (this document)
+- [System Design Intent](invariant/intent.md)
+- [Transcription Methodology](invariant/transcription_methodology.md)
+- [System Architecture](architecture_v3.md)
+- [System Requirements](requirements_v3.md)
+- [Data model](schema_v3.md)
+- [Error Handling Policy](error_handling_v3.md)
+- [Implementation Plan](implementation_plan_v3.md)

docs/ver3/requirements_v3.md

@@ -0,0 +1,45 @@
+# Document Transcription System Requirements (Version 3)
+
+This document captures the **Version 3 baseline requirements** for the production implementation.
+
+## Requirements Model
+
+| ID | Category | Requirement | Verify Method |
+| --- | --- | --- | --- |
+| REQ-0 | System | Provide end-to-end multi-page document transcription with persistent, inspectable async job states. | demonstration |
+| REQ-1 | Functional | Allow users to upload multi-image batches as sequential `Source` pages under a `Document`. | test |
+| REQ-2 | Functional | Process multi-page jobs asynchronously using an `asyncio` worker pool bounded by rate limits. | test |
+| REQ-3 | Functional | Persist frozen submission-time execution parameters and full input prompts (`system_prompt`, `user_prompt`, `prompt_name`, `prompt_hash`, `temperature`, `top_p`) on `Job`, and persist page-level output responses (`raw_transcription`, `ai_metadata`, `raw_api_response`) on `JobSource`. | test |
+| REQ-4 | Functional | Support job states (`queued`, `processing`, `completed`, `partial_success`, `failed`) and page states (`pending`, `transcribed`, `failed`). | inspection |
+| REQ-5 | Functional | Allow users to manage historical `Person` records and link multiple authors/recipients to a `Document` via `DocumentPerson`. | test |
+| REQ-6 | Functional | Maintain immutable original machine output on `Source.raw_transcription` while permitting inline human edits on `Source.revised_text`. | test |
+| REQ-7 | Data Constraint | Use SQLModel/SQLAlchemy to preserve a portable relational domain model and compatible data shape across the supported backends, with SQLite for local development/testing and PostgreSQL as the production system of record. | inspection |
+| REQ-8 | Data Constraint | Validate all API requests, database rows, and JSON structures using Pydantic V2 schemas and SQLModel. | test |
+| REQ-9 | Interface | Render multi-page transcriptions sequentially by `page_number` in the web UI with author/recipient metadata. | demonstration |
+| REQ-10 | Operations | Allow operators to resubmit only failed pages for queued reprocessing while preserving the frozen prompt snapshot on the existing `Job`. | test |
+| REQ-11 | Data Constraint | Calculate and store cryptographic file hashes (SHA-256) and file sizes for uploaded source images to track asset integrity. | test |
+| REQ-12 | Operations Constraint | Keep core development, testing, restore, and recovery workflows OS-independent across Windows, Linux, and macOS; do not require a platform-specific shell for canonical project processes. | inspection |
+
+## Element Satisfaction Mapping
+
+* **UI (NiceGUI):** Satisfies REQ-1, REQ-5, REQ-6, REQ-9, REQ-10.
+* **API (FastAPI):** Satisfies REQ-1, REQ-4, REQ-5, REQ-8.
+* **WORKER (asyncio):** Satisfies REQ-2, REQ-3, REQ-4, REQ-10.
+* **PERSISTENCE (SQLModel/SQLAlchemy):** Satisfies REQ-3, REQ-6, REQ-7, REQ-11.
+* **MODELS (Pydantic V2 / SQLModel):** Satisfies REQ-8.
+* **OPERATIONS TOOLING (Python / OS-neutral automation):** Satisfies REQ-12.
+
+---
+
+## Related Local References
+
+- [System Overview](index_v3.md)
+- [System Design Intent](invariant/intent.md)
+- [Transcription Methodology](invariant/transcription_methodology.md)
+- [System Architecture](architecture_v3.md)
+- System Requirements (this document)
+- [Data model](schema_v3.md)
+- [Error Handling Policy](error_handling_v3.md)
+- [Implementation Plan](implementation_plan_v3.md)
+
+

docs/ver3/schema_v3.md

@@ -0,0 +1,137 @@
+# Database Schema (Version 3)
+
+This document describes the relational schema for the transcription platform. It incorporates multi-image batch orchestration, page-level execution tracking, many-to-many author/recipient attribution, submission-time prompt snapshot capture, and raw API payload evidence for archival auditing.
+
+The schema uses generic JSON columns compatible with SQLite in local development and PostgreSQL native JSONB/UUID types in production.
+
+## Entity Relationship Diagram
+
+```mermaid
+erDiagram
+PERSON {
+UUID id PK
+TEXT full_name
+TEXT display_name
+TEXT maiden_name
+DATE birth_date
+TEXT birth_date_raw
+TEXT birth_place
+DATE death_date
+TEXT death_date_raw
+TEXT death_place
+TEXT biography
+TEXT portrait_path
+JSONB metadata
+TIMESTAMPTZ created_at
+TIMESTAMPTZ updated_at
+}
+
+DOCUMENT {
+    UUID id PK
+    TEXT name
+    TEXT document_type
+    DATE document_date
+    TEXT document_date_raw
+    TEXT location_created
+    TEXT notes
+    TEXT archive_identifier
+    TIMESTAMPTZ created_at
+    TIMESTAMPTZ updated_at
+}
+
+DOCUMENT_PERSON {
+    UUID id PK
+    UUID document_id FK
+    UUID person_id FK
+    VARCHAR role "author | recipient"
+    TIMESTAMPTZ created_at
+}
+
+JOB {
+    UUID id PK
+    UUID document_id FK
+    VARCHAR status "queued | processing | transcribed | completed | partial_success | failed"
+    INTEGER retry_count
+    TEXT provider
+    TEXT model
+    TEXT prompt_name
+    TEXT prompt_hash
+    TEXT system_prompt
+    TEXT user_prompt
+    FLOAT temperature
+    FLOAT top_p
+    TIMESTAMPTZ date_created
+    TIMESTAMPTZ date_updated
+}
+
+SOURCE {
+    UUID id PK
+    UUID document_id FK
+    INTEGER page_number
+    TEXT upload_name
+    TEXT filename
+    TEXT file_path
+    TEXT file_hash
+    BIGINT file_size_bytes
+    TEXT raw_transcription
+    TEXT revised_text
+    TIMESTAMPTZ date_uploaded
+    TIMESTAMPTZ date_revised
+}
+
+JOB_SOURCE {
+    UUID id PK
+    UUID job_id FK
+    UUID source_id FK
+    VARCHAR status "pending | transcribed | failed"
+    TEXT raw_transcription
+    JSONB ai_metadata
+    JSONB raw_api_response
+    TEXT error_detail
+    TIMESTAMPTZ executed_at
+}
+
+DOCUMENT ||--o{ DOCUMENT_PERSON : "has_people"
+PERSON ||--o{ DOCUMENT_PERSON : "participates_in"
+DOCUMENT ||--o{ JOB : "has_jobs"
+DOCUMENT ||--o{ SOURCE : "contains_pages"
+JOB ||--o{ JOB_SOURCE : "executes"
+SOURCE ||--o{ JOB_SOURCE : "processed_in"
+```
+
+## Domain Invariants & Provenance Rules
+
+### Page-Level Execution & AI Outputs
+
+* **Execution Granularity:** Every single image execution attempt by an AI model produces a dedicated record in `job_source`.
+* **Submission Snapshot Provenance:** Every `job` captures the frozen prompt identifier details (`prompt_name`, `prompt_hash`), full prompt text strings (`system_prompt`, `user_prompt`), and hyperparameters (`temperature`, `top_p`) at submission time.
+* **Point-in-Time Output Auditability:** `job_source.raw_api_response` stores the complete, unedited provider REST response envelope for that specific image page call. `job_source.ai_metadata` stores spatial bounding boxes, normalized token usage, latency, and cost details for fast querying.
+* **Active Output Caching:** Upon successful completion of an image call, `source.raw_transcription` is updated with the latest output string from `job_source.raw_transcription` for fast UI rendering.
+
+### Image Storage & Integrity
+
+* **Filesystem Storage:** Binary images are stored on disk in the local file system. The `source` table holds the relative `file_path`.
+* **File Integrity Tracking:** `source` captures `file_hash` (SHA-256) and `file_size_bytes` at upload time to guarantee document file integrity and duplicate checking over long-term preservation.
+
+### Page Ordering & Revisions
+
+* **Sequential Integrity:** `source.page_number` dictates page ordering within a document. Reads assembling full documents must query `ORDER BY source.document_id, source.page_number ASC`.
+* **Inlined Human Corrections:** User edits occur at the page level inside `source.revised_text`. `source.raw_transcription` remains immutable. If `source.revised_text` is non-null, application frontends must render `source.revised_text`.
+
+### Async Job Lifecycle & Failure Isolation
+
+* **Batch Orchestrator:** A job represents an overarching execution run across one or more source images belonging to a document.
+* **Isolated Failures:** API requests run concurrently (e.g., using `asyncio`). A failure on page 3 does not invalidate successful transcriptions on page 1 or 2.
+* **Job States:**
+* `queued`: Created, awaiting worker execution.
+* `processing`: Concurrent HTTP tasks actively running.
+* `completed`: 100% of linked `job_source` tasks succeeded (`transcribed`).
+* `partial_success`: At least one `job_source` succeeded and at least one failed.
+* `failed`: All linked `job_source` tasks failed or a job-level runtime error occurred.
+
+
+
+### Attribution & Person Roles
+
+* **Multi-Person Roles:** Documents support zero, one, or many authors and recipients linked via `document_person`.
+* **Role Uniqueness:** `(document_id, person_id, role)` must be unique to prevent duplicate role tagging.

docs/ver4/architecture_v4.md

@@ -0,0 +1,137 @@
+# System Architecture (Version 4)
+
+This document describes the V4 architecture changes for expanded `Document`-`Person` relationship management.
+
+V4 extends V3 with role extensibility, assisted suggestion review, and role-policy enforcement while preserving the existing transcription execution topology.
+
+## Architecture Objectives
+
+- Keep V3 transcription pipeline behavior stable unless relationship evidence extraction requires additive integration.
+- Support many people per role for each document.
+- Support extensible relationship roles without hardcoding UI and API behavior to two values.
+- Support non-canonical suggestion intake with explicit human promotion to canonical asserted links.
+- Enforce role exclusivity policy consistently at service and persistence boundaries.
+- Preserve auditability for suggestion provenance and review actions.
+
+## Runtime Topology
+
+V4 keeps the existing runtime shape:
+
+- FastAPI + NiceGUI web app process.
+- Async service layer with SQLModel/SQLAlchemy persistence.
+- Existing worker execution path for transcription jobs.
+
+Additive V4 components:
+
+- Relationship policy evaluator (exclusivity checks).
+- Suggestion lifecycle service (propose/list/accept/reject).
+- Optional suggestion extraction adapter (rule/NLP or LLM-backed).
+
+^^^mermaid
+flowchart LR
+U[Browser User] --> UI[NiceGUI Pages]
+UI --> API[FastAPI Routes]
+API --> DS[Document Service]
+API --> SS[Suggestion Service]
+DS --> PE[Policy Evaluator]
+SS --> PE
+DS --> DB[(Relational DB)]
+SS --> DB
+W[Async Worker Engine] --> DB
+W --> E[Optional Suggestion Extraction]
+E --> SS
+^^^
+
+## Layer Responsibilities
+
+### UI Layer (`src/transcription/ui/**`)
+
+- Render per-role grouped relationships for document and person views.
+- Provide multi-select role assignment controls in document create/edit flows.
+- Provide suggestion review controls (accept/reject) for pending suggestions.
+- Surface policy conflict errors from API/service layer clearly.
+
+### API Layer (`src/transcription/api/**`)
+
+- Expose role-aware and state-aware read contracts.
+- Expose suggestion lifecycle write contracts.
+- Return deterministic validation/conflict errors for exclusivity and duplicate semantics.
+- Evolve endpoints additively, with explicit deprecations and short-lived transition windows.
+
+### Service Layer (`src/transcription/services/**`)
+
+- Implement set-based relationship sync (delta add/remove) to avoid destructive replacement behavior.
+- Apply policy evaluator before persistence commits.
+- Orchestrate suggestion acceptance/rejection transitions.
+- Ensure accepted suggestions result in asserted link creation/confirmation.
+
+### Policy Evaluator (New logical component)
+
+- Evaluate role exclusivity matrix for `(document_id, person_id)` writes.
+- Provide deterministic conflict reason payloads suitable for UI/API display.
+- Stay stateless and reusable across create/update/sync code paths.
+
+### Persistence Layer (`src/transcription/db/**`)
+
+- Store asserted links and suggestion records according to selected schema option.
+- Persist provenance metadata for suggestion records.
+- Enforce uniqueness and support performant role/state filtering.
+
+## Core V4 Workflows
+
+### 1) Manual Relationship Management
+
+1. User opens document edit view.
+2. UI loads asserted links grouped by role.
+3. User adds/removes people per role.
+4. Service computes delta and runs policy checks.
+5. Persistence applies adds/removes atomically.
+
+### 2) Assisted Suggestion Review
+
+1. Suggestion records are created (`pending`) by extraction logic or manual propose action.
+2. UI displays pending suggestions with evidence metadata.
+3. User accepts or rejects each suggestion.
+4. Accept path creates/confirms asserted relationship and marks suggestion `accepted`.
+5. Reject path marks suggestion `rejected`.
+
+### 3) Exclusivity Conflict Handling
+
+1. Incoming write attempts role assignment.
+2. Policy evaluator checks role pair conflicts for same `(document_id, person_id)`.
+3. If conflict exists, write is rejected with structured conflict details.
+4. UI presents actionable message without partial updates.
+
+## Invariants
+
+- Suggested links are never canonical until accepted.
+- Asserted links must satisfy exclusivity rules.
+- Relationship mutations are set-based and deterministic.
+- Relationship views remain consistent between document detail and person detail pages.
+- Existing V3 links remain valid under asserted semantics after migration.
+
+## Backward Compatibility
+
+- V3 author/recipient links are interpreted as asserted links in V4.
+- Existing document and person flows continue to function where unaffected by new role/state dimensions.
+- During development revisions, the current API contract is authoritative; long-lived legacy runtime compatibility layers are not required.
+
+## Observability and Auditability
+
+- Persist suggestion provenance fields sufficient for operator review.
+- Record review decision outcomes (`accepted`/`rejected`) with timestamps and reviewer identity where available.
+- Emit structured service-level logs for exclusivity conflicts and review actions.
+
+## Test Strategy Additions
+
+- Service tests for delta sync and exclusivity matrix enforcement.
+- API tests for role/state filtering and suggestion transitions.
+- UI tests for multi-role selection and suggestion review interactions.
+- Migration tests for V3-to-V4 asserted mapping and conflict scans.
+
+## Related Local References
+
+- [V4 Scope Boundary](scope_boundary_v4.md)
+- [V4 Requirements](requirements_v4.md)
+- [V4 Schema](schema_v4.md)
+- [V3 Architecture](../architecture_v3.md)

docs/ver4/error_handling_v4.md

@@ -0,0 +1,104 @@
+# Error Handling Policy (Version 4)
+
+This document defines canonical error-handling behavior for V4 document-person relationship expansion.
+
+V4 keeps V3 transcription error behavior and adds policy/conflict handling for role extensibility, suggestion lifecycle transitions, and exclusivity enforcement.
+
+## Error Handling Objectives
+
+- Provide clear, actionable conflict and validation feedback for relationship write operations.
+- Prevent partial, silent, or destructive relationship mutations when policy checks fail.
+- Preserve suggestion review auditability with deterministic accept/reject outcomes.
+- Keep consistent API/UI/service error envelopes across relationship workflows.
+
+## Scope and Authority
+
+Governs relationship-related error behavior in:
+
+- NiceGUI document/person relationship views,
+- FastAPI relationship and suggestion endpoints,
+- domain services for relationship sync and suggestion review,
+- persistence constraints for role, state, and exclusivity invariants.
+
+## Relationship Error Taxonomy
+
+| Category | Definition | Retriable |
+| --- | --- | --- |
+| `validation_error` | Payload shape/type invalid, unknown role/state, malformed IDs | no |
+| `not_found_error` | Target `Document`, `Person`, role, or suggestion record does not exist | no |
+| `conflict_error` | Write violates uniqueness or exclusivity policy | no |
+| `suggestion_state_error` | Invalid suggestion transition (for example accept after reject) | no |
+| `policy_violation_error` | Action blocked by configured role matrix or governance rule | no |
+| `infrastructure_transient_error` | Temporary DB or network instability during relationship operation | yes |
+| `infrastructure_persistent_error` | Persistent DB/configuration failure | no |
+| `internal_unexpected_error` | Unhandled exception/logic defect | no |
+
+## Deterministic Conflict Behavior
+
+When relationship writes fail policy checks:
+
+1. Reject the full write operation (no partial apply).
+2. Return structured conflict details including conflicting role pair and target identifiers.
+3. Preserve existing canonical relationships unchanged.
+
+When suggestion transitions fail:
+
+1. Reject invalid state transition.
+2. Return current state and allowed next actions.
+3. Preserve suggestion record integrity.
+
+## API Error Response Contract
+
+Relationship endpoints return a structured envelope:
+
+^^^json
+{
+  "error_id": "err_uuid_12345",
+  "category": "conflict_error",
+  "message": "Role assignment violates exclusivity policy.",
+  "suggestion": "Remove recipient role before assigning author for this person on this document.",
+  "details": {
+    "document_id": "...",
+    "person_id": "...",
+    "attempted_role": "author",
+    "conflicting_role": "recipient",
+    "policy_rule": "author+recipient exclusive"
+  },
+  "timestamp": "2026-08-09T15:00:00Z"
+}
+^^^
+
+HTTP status mappings:
+
+- `validation_error` -> `400`
+- `not_found_error` -> `404`
+- `conflict_error`, `suggestion_state_error`, `policy_violation_error` -> `409`
+- `infrastructure_transient_error` -> `503`
+- `infrastructure_persistent_error`, `internal_unexpected_error` -> `500`
+
+## UI Error Presentation Rules
+
+- Display concise conflict summary with actionable next step.
+- Keep user edits in context (do not discard form state when feasible).
+- Differentiate between validation issues, policy conflicts, and infrastructure failures.
+- For bulk role sync operations, show per-item conflict context when multiple failures occur.
+
+## Logging and Audit Expectations
+
+- Log relationship write failures with correlation IDs.
+- Log suggestion acceptance/rejection outcomes with actor and timestamp where available.
+- Log policy matrix violations with deterministic machine-readable context.
+
+## Relationship-Specific Retry Guidance
+
+- Do not auto-retry policy or conflict failures.
+- Permit user-driven retry only after input changes.
+- Retry infrastructure transient failures with bounded policy in service layer if operation is idempotent.
+
+## Related Local References
+
+- [V4 Scope Boundary](scope_boundary_v4.md)
+- [V4 Requirements](requirements_v4.md)
+- [V4 Schema](schema_v4.md)
+- [V4 Architecture](architecture_v4.md)
+- [V3 Error Handling](../error_handling_v3.md)

docs/ver4/implementation_plan_v4.md

@@ -0,0 +1,107 @@
+# Implementation Plan (Version 4)
+
+## Goal
+
+Implement V4 document-person relationship expansion and document type governance with extensible registries, assisted suggestion lifecycle, and policy-enforced exclusivity while preserving V3 transcription behavior.
+
+## Current Project Impact
+
+- `src/transcription/db/models.py` will require relationship schema evolution for role extensibility and suggestion lifecycle support.
+- `src/transcription/db/models.py` will require document type registry entities and document type reference updates.
+- `src/transcription/services/documents.py` will require set-based relationship sync and policy checks.
+- `src/transcription/services/documents.py` will require registry-based document type lookup, validation, and normalization helpers.
+- API modules under `src/transcription/api/**` will require role/state-aware contracts and suggestion lifecycle endpoints.
+- API modules under `src/transcription/api/**` will require additive document type catalog and code-based selection contracts.
+- UI pages under `src/transcription/ui/pages/**` will require multi-role, multi-person editing, suggestion review controls, and registry-backed type selectors.
+- Existing tests under `tests/services`, `tests/api`, and `tests/ui` need expanded coverage for V4 behavior and regression safety.
+
+## Implementation Phases
+
+### 1. Finalize V4 Schema Decisions
+
+- Role extensibility mechanism: use role registry tables.
+- Document type extensibility mechanism: use `document_type` registry tables.
+- Suggestion storage model: use a separate `document_person_suggestion` table.
+- Exclusivity baseline: `author` vs `recipient` exclusive, `mentioned` non-exclusive.
+- Enforce exclusivity on asserted links; evaluate conflicts on suggestion acceptance.
+
+### 2. Evolve Persistence Layer
+
+- Implement selected schema model in SQLModel.
+- Add constraints and indexes for dedupe, filtering, and policy support.
+- Add migration/backfill logic for V3 links to V4 asserted semantics.
+- Add conflict scan tooling for historical records violating exclusivity rules.
+- Perform a one-time manual mapping of existing document type values to registry-backed type references (`document_type_id`) for the current small corpus.
+
+### 3. Implement Service-Layer Policy and Sync Semantics
+
+- Add relationship delta sync operations (set-based add/remove).
+- Implement centralized policy evaluator for exclusivity checks.
+- Add suggestion lifecycle operations (propose/list/accept/reject).
+- Ensure accepted suggestions create/confirm asserted links atomically.
+- Add document type resolution operations by stable `code` and active/inactive state handling.
+
+### 4. Implement API Contract Changes
+
+- Add role-aware and state-aware query parameters/filters.
+- Add suggestion lifecycle endpoints and response models.
+- Add deterministic conflict/error payloads aligned to V4 error policy.
+- Use additive endpoint evolution for V4; during development mode, deprecate then remove without maintaining long-lived legacy runtime compatibility layers.
+- Add document type catalog endpoints with active-only filtering and code-based selection for document writes.
+
+### 5. Update UI Workflows
+
+- Replace single-author controls with grouped multi-role selectors.
+- Replace unconstrained document type free-text entry with registry-backed type selection.
+- Add pending suggestion review panel and accept/reject actions.
+- Update document/person detail cards to group links by role and state.
+- Preserve edit-state ergonomics on validation/conflict failures.
+
+### 6. Verification and Hardening
+
+- Add service tests for:
+  - many-per-role behavior,
+  - exclusivity enforcement,
+  - set-based sync correctness,
+  - suggestion transition validity,
+  - document type code resolution and inactive-type handling.
+- Add API tests for role/state filtering and conflict response shapes.
+- Add API tests for document type catalog retrieval and code-based write validation.
+- Add UI tests/manual walkthroughs for create/edit/review workflows.
+- Add UI tests/manual walkthroughs for registry-backed document type selection and validation messaging.
+- Add regression tests for document/person delete cleanup semantics.
+
+## Done When
+
+- V4 relationship schema and contracts are implemented and validated.
+- V4 document type registry schema and contracts are implemented and validated.
+- Suggestions remain non-canonical until explicit acceptance.
+- Exclusivity policy is enforced deterministically across service/API boundaries.
+- Existing V3 links are migrated to V4 asserted semantics without data loss.
+- Existing document type strings are manually normalized to registry references for the current corpus.
+- Test suite includes V4-specific coverage and passes on supported backends.
+
+## Out of Scope
+
+- Automatic acceptance of suggestions.
+- Global person entity-resolution/merge engine.
+- Core transcription execution redesign unrelated to relationship expansion.
+- Automated semantic document type classification.
+
+## Delivery Order Recommendation
+
+1. Requirements freeze (`requirements_v4.md`).
+2. Schema decision freeze (`schema_v4.md`).
+3. Error policy freeze (`error_handling_v4.md`).
+4. Implementation of persistence and service layer.
+5. API and UI changes.
+6. Final integration and regression validation.
+
+## Related Local References
+
+- [V4 Scope Boundary](scope_boundary_v4.md)
+- [V4 Requirements](requirements_v4.md)
+- [V4 Schema](schema_v4.md)
+- [V4 Architecture](architecture_v4.md)
+- [V4 Error Handling](error_handling_v4.md)
+- [V3 Implementation Plan](../implementation_plan_v3.md)

docs/ver4/index_v4.md

@@ -0,0 +1,49 @@
+# Document-Person Expansion Overview (Version 4)
+
+Version 4 defines the relationship-model evolution track for linking `Document` and `Person` entities with extensible roles, suggestion review, and policy enforcement.
+
+## Start Here
+
+Read [scope_boundary_v4.md](scope_boundary_v4.md) first to confirm scope and non-scope before implementation work begins.
+
+## Core V4 Capabilities
+
+- Extensible relationship role taxonomy.
+- Many-people-per-role linking for documents.
+- Explicit distinction between canonical asserted links and pending suggested links.
+- Human-in-the-loop suggestion accept/reject workflow.
+- Role exclusivity policy enforcement for configured role pairs.
+- Role/state-aware API and UI retrieval/presentation behavior.
+- Minimal document type governance rollout for the current corpus, with one-time manual mapping and no alias helper table.
+
+## V4 Documentation Index
+
+- [Scope Boundary](scope_boundary_v4.md)
+- [System Requirements](requirements_v4.md)
+- [Data Model](schema_v4.md)
+- [System Architecture](architecture_v4.md)
+- [Error Handling Policy](error_handling_v4.md)
+- [Implementation Plan](implementation_plan_v4.md)
+
+## Relationship To V3
+
+V3 remains the baseline production architecture and requirements set for transcription pipeline behavior. V4 is an additive evolution track focused on document-person relationship semantics and workflows.
+
+## Decision Status
+
+Locked decisions:
+
+1. Role extensibility uses registry tables.
+2. Suggestion storage uses a separate suggestion table.
+3. Exclusivity baseline is `author` vs `recipient` exclusive, with `mentioned` non-exclusive.
+4. API evolution is additive in development mode with explicit deprecate-then-remove behavior.
+
+Remaining decision:
+
+1. Suggestion generation strategy (deterministic rules/NLP vs LLM extraction).
+
+## Related Local References
+
+- [V3 System Overview](../index_v3.md)
+- [V3 Requirements](../requirements_v3.md)
+- [V3 Schema](../schema_v3.md)

docs/ver4/requirements_v4.md

@@ -0,0 +1,70 @@
+# Relationship and Document Type Governance Requirements (Version 4)
+
+This document defines Version 4 baseline requirements for expanding relationships between `Document` and `Person` and introducing governed document type classification.
+
+V4 preserves all applicable V3 capabilities and adds role extensibility, assisted suggestion workflows, explicit relationship policy enforcement, and registry-driven `Document` type governance.
+
+## Requirements Model
+
+| ID | Category | Requirement | Verify Method |
+| --- | --- | --- | --- |
+| REQ-0 | System | Provide end-to-end, reviewable, policy-enforced document-person relationship management supporting asserted and suggested links. | demonstration |
+| REQ-1 | Functional | Preserve many-to-many `Document` ↔ `Person` linking, and allow multiple people per role on a single document. | test |
+| REQ-2 | Functional | Support an extensible role taxonomy for document-person relationships beyond fixed `author`/`recipient`. | inspection |
+| REQ-3 | Functional | Represent relationship assertion state explicitly (`asserted` and `suggested`) and keep machine suggestions non-canonical until human acceptance. | test |
+| REQ-4 | Functional | Provide assisted suggestion lifecycle operations: create/list/filter suggestions, accept suggestion, reject suggestion, and promote accepted suggestions to asserted links. | test |
+| REQ-5 | Policy Constraint | Enforce a role exclusivity matrix for a single `(document_id, person_id)` pair on asserted links; initial rule set must block `author` + `recipient` coexistence while allowing `mentioned` to coexist with other roles. | test |
+| REQ-6 | Data Constraint | Store canonical asserted links in `document_person` and lifecycle-managed suggestions in a separate `document_person_suggestion` table. | test |
+| REQ-7 | Functional | Provide set-based synchronization behavior for relationship mutations (add/remove delta), replacing single-value replacement patterns that can drop unrelated links. | test |
+| REQ-8 | Interface | Render grouped relationship metadata by role and assertion state on document detail and person detail views. | demonstration |
+| REQ-9 | Interface | Document create/edit UI must support selecting multiple people per role and reviewing pending suggestions with explicit accept/reject controls. | demonstration |
+| REQ-10 | API Constraint | Expose additive, role-aware and state-aware retrieval/filtering in API contracts for documents, people, and relationship records. | test |
+| REQ-11 | Data Provenance | Capture suggestion provenance metadata sufficient for operator review (for example source mechanism, confidence, and evidence reference) without mutating canonical asserted links implicitly. | inspection |
+| REQ-12 | Operations | Provide migration/backfill validation that identifies and resolves historical records violating newly enforced exclusivity policies before hard enforcement. | test |
+| REQ-13 | Revision Upgrade | Support deterministic in-place revision upgrade behavior by mapping existing author/recipient links into V4 asserted semantics without requiring long-lived runtime legacy compatibility layers. | test |
+| REQ-14 | Reliability | Ensure document/person deletion and cleanup workflows remain safe and deterministic with expanded relationship semantics, including suggestion records. | test |
+| REQ-15 | Quality | Add automated test coverage for role extensibility, exclusivity enforcement, suggestion lifecycle transitions, and regression scenarios across service/API/UI flows. | test |
+| REQ-16 | Functional | Support a registry-driven document type taxonomy (`document_type`) with stable machine-readable codes, mutable display labels, and active/inactive lifecycle control. | test |
+| REQ-17 | Data Constraint | Replace unconstrained free-text document type assignment with controlled type references or deterministic code mapping governed by the document type registry. | test |
+| REQ-18 | Interface | Document create/edit UI must present type selection from active registry entries and prevent invalid type assignment. | demonstration |
+| REQ-19 | API Constraint | Provide additive API contracts for document type catalog retrieval, including active-only filtering and stable code-based selection for document writes. | test |
+| REQ-20 | Operations | Provide deterministic one-time manual normalization/backfill for existing document type strings in the current small corpus, assigning each document to a canonical registry type before strict write enforcement. | test |
+
+## Clarifying Constraints
+
+1. Suggestions are advisory only until accepted by a human operator.
+2. Relationship acceptance/rejection must be explicit and auditable.
+3. Role-policy enforcement must occur consistently across service and API boundaries.
+4. Many-per-role behavior is required for both asserted and suggested states where applicable.
+5. Suggestions that would violate asserted exclusivity may exist as `pending`, but acceptance must fail until the exclusivity conflict is resolved.
+6. Document type codes are stable identifiers; display labels may evolve without changing canonical type identity.
+
+## Assumptions
+
+1. V4 scope is limited to document-person relationship expansion and does not redesign the core transcription job execution model.
+2. Existing V3 data remains the starting corpus and is transformed via deterministic backfill/validation rules.
+3. V4 uses a role registry model and separate suggestion storage; enum-first role expansion is out of scope for this revision.
+4. V4 introduces document type registry governance and deprecates unconstrained free-text typing as an authoring-time default.
+
+## Element Satisfaction Mapping
+
+- **UI (NiceGUI):** Satisfies REQ-0, REQ-1, REQ-3, REQ-4, REQ-8, REQ-9, REQ-18.
+- **API (FastAPI):** Satisfies REQ-0, REQ-3, REQ-4, REQ-5, REQ-7, REQ-10, REQ-19.
+- **PERSISTENCE (SQLModel/SQLAlchemy):** Satisfies REQ-1, REQ-2, REQ-5, REQ-6, REQ-11, REQ-12, REQ-13, REQ-14, REQ-16, REQ-17, REQ-20.
+- **SERVICES (Domain Layer):** Satisfies REQ-4, REQ-5, REQ-7, REQ-10, REQ-14, REQ-17, REQ-20.
+- **TEST SUITE:** Satisfies REQ-15 and verifies all test-marked requirements.
+
+## Change Classification vs V3
+
+- **Semantic expansion:** role model and assertion state model.
+- **Policy expansion:** exclusivity matrix enforcement.
+- **Workflow expansion:** suggestion review and promotion lifecycle.
+- **Contract expansion:** role/state-aware read and write behavior.
+- **Governance expansion:** registry-managed document type taxonomy and normalization workflow.
+
+## Related Local References
+
+- [V4 Scope Boundary](scope_boundary_v4.md)
+- [System Overview V3](../index_v3.md)
+- [System Requirements V3](../requirements_v3.md)
+- [Data Model V3](../schema_v3.md)

docs/ver4/schema_v4.md

@@ -0,0 +1,214 @@
+# Database Schema (Version 4)
+
+This document defines the selected schema direction for V4 document-person relationship expansion.
+
+V4 goals are:
+
+- extensible role taxonomy,
+- extensible document type taxonomy,
+- explicit assertion state (`asserted`, `suggested`),
+- policy-driven exclusivity,
+- deterministic migration from V3 links.
+
+## Scope
+
+This specification focuses on relationship and document-type governance persistence changes. Existing `Person`, `Source`, `Job`, and `JobSource` core structures remain as in V3 unless explicitly noted.
+
+## New/Expanded Concepts
+
+- **Relationship role:** semantic label such as `author`, `recipient`, `mentioned`.
+- **Assertion state:** whether the link is canonical (`asserted`) or pending review (`suggested`).
+- **Exclusivity matrix:** configurable role-pair conflicts for same `(document_id, person_id)`.
+- **Suggestion provenance:** evidence fields enabling review decisions.
+- **Document type registry:** controlled taxonomy for `Document` classification with stable code identity.
+
+## Selected Model: Role Registry + Separate Suggestion Table
+
+This option cleanly separates canonical links from pending suggestions and enables fully data-driven role expansion.
+
+### Tables
+
+#### `person_role`
+
+| Column | Type | Notes |
+| --- | --- | --- |
+| `id` | UUID PK | Stable key |
+| `code` | TEXT UNIQUE | Canonical role code, for example `author`, `recipient`, `mentioned` |
+| `label` | TEXT | UI label |
+| `is_active` | BOOLEAN | Soft-enable/disable role |
+| `created_at` | TIMESTAMPTZ | Audit timestamp |
+| `updated_at` | TIMESTAMPTZ | Audit timestamp |
+
+#### `document_person` (asserted links only)
+
+| Column | Type | Notes |
+| --- | --- | --- |
+| `id` | UUID PK | Stable key |
+| `document_id` | UUID FK | -> `document.id` |
+| `person_id` | UUID FK | -> `person.id` |
+| `role_id` | UUID FK | -> `person_role.id` |
+| `created_at` | TIMESTAMPTZ | Audit timestamp |
+| `updated_at` | TIMESTAMPTZ | Audit timestamp |
+
+Constraints:
+
+- `UNIQUE(document_id, person_id, role_id)`
+
+#### `document_person_suggestion`
+
+| Column | Type | Notes |
+| --- | --- | --- |
+| `id` | UUID PK | Stable key |
+| `document_id` | UUID FK | -> `document.id` |
+| `person_id` | UUID FK | -> `person.id` |
+| `role_id` | UUID FK | -> `person_role.id` |
+| `status` | TEXT | `pending`, `accepted`, `rejected` |
+| `confidence` | FLOAT NULL | Optional score |
+| `source_mechanism` | TEXT NULL | For example `rule`, `llm` |
+| `evidence_ref` | TEXT NULL | Pointer or excerpt ID |
+| `evidence_span` | JSON NULL | Optional text span payload |
+| `note` | TEXT NULL | Reviewer note |
+| `created_at` | TIMESTAMPTZ | Suggestion creation time |
+| `reviewed_at` | TIMESTAMPTZ NULL | Decision time |
+| `reviewed_by` | TEXT NULL | Operator identifier |
+
+Constraints:
+
+- `UNIQUE(document_id, person_id, role_id, status)` with policy for multiple pending rows defined in service layer.
+- Optional stricter rule: one active pending suggestion per `(document_id, person_id, role_id)`.
+
+#### `role_exclusivity`
+
+| Column | Type | Notes |
+| --- | --- | --- |
+| `id` | UUID PK | Stable key |
+| `left_role_id` | UUID FK | -> `person_role.id` |
+| `right_role_id` | UUID FK | -> `person_role.id` |
+| `created_at` | TIMESTAMPTZ | Audit timestamp |
+
+Constraints:
+
+- Canonical ordering rule to avoid duplicate pairs (`left_role_id < right_role_id` enforced in service/DB).
+- `UNIQUE(left_role_id, right_role_id)`
+
+Initial seed:
+
+- Exclusivity pair: (`author`, `recipient`)
+
+## Deferred Alternative (Not Selected for V4)
+
+An enum-based shared table model was considered but is intentionally not selected for V4 because it couples canonical and provisional states in one table and increases invariant complexity.
+
+## Assertion-State Semantics
+
+- `asserted`: canonical relationship used for document/person metadata and business logic.
+- `suggested`: non-canonical proposal requiring explicit review.
+- Accept action:
+  - creates asserted link (or confirms existing),
+  - marks suggestion `accepted`.
+- Reject action:
+  - marks suggestion `rejected`.
+
+## Exclusivity Enforcement
+
+Policy target:
+
+- For a single `(document_id, person_id)`, disallow coexistence of role pairs configured as exclusive.
+
+Enforcement layers:
+
+1. Service-level pre-check for clear API errors.
+2. Database-level guard where feasible (constraints/triggers or deterministic write path).
+
+Initial configured rule:
+
+- `author` and `recipient` are exclusive.
+
+Enforcement semantics:
+
+- Exclusivity is enforced for asserted links.
+- Pending suggestions may exist even if they would conflict when asserted.
+- Accepting a suggestion must run exclusivity checks and fail deterministically on conflict.
+
+## Migration From V3
+
+### Data Mapping
+
+- Existing V3 `document_person` rows map to V4 `asserted` semantics.
+- Existing V3 role values:
+  - `author` -> role `author`
+  - `recipient` -> role `recipient`
+
+### Backfill Steps
+
+1. Seed role rows (`author`, `recipient`, `mentioned`) if using Option A.
+2. Migrate current links into asserted table/state.
+3. Run conflict scan for exclusivity violations.
+4. Apply deterministic conflict policy for any violations.
+5. Enable hard enforcement after data passes validation.
+
+## Indexing Guidance
+
+Recommended indexes:
+
+- `document_person(document_id)`
+- `document_person(person_id)`
+- `document_person(role_id)` (Option A) or `document_person(role)` (Option B)
+- `document_person_suggestion(document_id, status)` (Option A)
+- `document_person_suggestion(person_id, status)` (Option A)
+
+## Selection Rationale
+
+V4 selects role registry plus separate suggestion storage for clearer provenance boundaries, cleaner lifecycle transitions, and long-term extensibility.
+
+## Document Type Registry Model (Selected for V4)
+
+V4 applies the same registry governance pattern to document classification.
+
+### Tables
+
+#### `document_type`
+
+| Column | Type | Notes |
+| --- | --- | --- |
+| `id` | UUID PK | Stable key |
+| `code` | TEXT UNIQUE | Canonical type code, for example `letter`, `diary`, `book`, `postcard` |
+| `label` | TEXT | UI display label |
+| `is_active` | BOOLEAN | Soft-enable/disable type |
+| `sort_order` | INTEGER NULL | Optional UI ordering |
+| `created_at` | TIMESTAMPTZ | Audit timestamp |
+| `updated_at` | TIMESTAMPTZ | Audit timestamp |
+
+#### `document` update
+
+| Column | Type | Notes |
+| --- | --- | --- |
+| `document_type_id` | UUID FK NULL | -> `document_type.id` |
+
+### Constraints and Governance
+
+- `document_type.code` must be stable and unique.
+- `document_type.label` may change without changing canonical type identity.
+- Inactive types remain valid for historical records but are excluded from default create/edit selectors.
+
+### Initial Seeds
+
+- Seed baseline type codes from current V3 usage set (for example `letter`, `diary`, `book`, `postcard`, `record`, `memo`) and refine labels as needed.
+
+### Migration and Normalization (Small Corpus)
+
+1. Seed canonical `document_type` rows.
+2. Manually assign each existing document (10 total) to a canonical type via `document_type_id`.
+3. Resolve any outlier values directly during this one-time pass.
+4. Enforce registry-backed write validation after manual assignment is complete.
+
+### Indexing Guidance (Document Type)
+
+- `document_type(code)` unique index.
+- `document(document_type_id)` index.
+
+## Related Local References
+
+- [V4 Scope Boundary](scope_boundary_v4.md)
+- [V4 Requirements](requirements_v4.md)
+- [V3 Schema](../schema_v3.md)

docs/ver4/scope_boundary_v4.md

@@ -0,0 +1,134 @@
+# V4 Scope Boundary (Version 4)
+
+This document defines what is and is not included in Version 4 for expanding `Document`-`Person` relationships.
+
+## Purpose
+
+Create a clear implementation boundary before updating full V4 architecture, requirements, schema, and plan documents.
+
+## Why This Is V4 (Not V3.1)
+
+V4 is required because the change is semantic and cross-cutting:
+
+- Expands relationship meaning beyond fixed `author`/`recipient`.
+- Introduces a suggestion lifecycle (`suggested` vs `asserted`).
+- Introduces policy constraints (role exclusivity matrix).
+- Impacts persistence, service contracts, API behavior, UI workflows, and test strategy.
+
+A V3.1 patch would only be appropriate for non-semantic quality-of-life fixes inside existing role semantics.
+
+## In Scope for V4
+
+### 1) Relationship Semantics
+
+- Role taxonomy becomes extensible (not hardcoded to only two role values).
+- A `Document` can link to many `Person` records per role.
+- Relationship states are explicit:
+  - `asserted`: human-confirmed canonical link.
+  - `suggested`: machine- or heuristic-proposed link pending review.
+
+### 2) Policy Rules
+
+- Enforce role exclusivity for the same `(document, person)` pair where configured.
+- Initial policy decision:
+  - `author` + `recipient` are mutually exclusive.
+  - `mentioned` may coexist with other roles.
+
+### 3) Persistence and Contracts
+
+- Evolve persistence model to support:
+  - extensible roles,
+  - suggestion lifecycle metadata,
+  - deterministic conflict handling.
+- Replace single-value relationship mutation patterns with set-based sync behavior.
+
+### 4) Human-in-the-Loop Workflow
+
+- Add assisted suggestion review flow:
+  - list suggestions,
+  - accept,
+  - reject,
+  - promote accepted suggestion to asserted link.
+- No silent auto-promotion from suggestion to asserted.
+
+### 5) UI and API Behavior
+
+- Document create/edit/detail workflows support multi-person per role.
+- UI surfaces grouped role links and suggestion status.
+- API supports role-aware retrieval and suggestion lifecycle operations.
+
+### 6) Verification
+
+- Add tests for:
+  - many-per-role behavior,
+  - exclusivity enforcement,
+  - suggestion lifecycle,
+  - migration/conflict detection,
+  - regression on delete/link cleanup behavior.
+
+## Out of Scope for V4
+
+- Automatic acceptance of suggested links without human review.
+- Full entity resolution/identity merge pipeline across all `Person` records.
+- Historical provenance graph redesign beyond relationship-level evidence fields.
+- Large-scale NLP research features unrelated to document-person linking.
+- Changes to core transcription execution model (`Job`, `JobSource`) except where needed to expose suggestion evidence inputs.
+
+## Locked Design Decisions
+
+### A) Role Extensibility Mechanism
+
+- Adopt role registry tables (data-driven roles).
+- Do not use enum-first role expansion for V4.
+
+### B) Suggestion Storage Model
+
+- Adopt a separate `document_person_suggestion` table.
+- Keep `document_person` focused on canonical asserted links.
+- Revisit only if operational complexity proves materially higher than expected.
+
+### C) Exclusivity Matrix Baseline
+
+- `author` + `recipient` are mutually exclusive for the same `(document, person)` pair.
+- `mentioned` remains non-exclusive.
+- Hard exclusivity enforcement applies to asserted links.
+- Suggestions may be stored even if they would conflict at assert time; promotion to asserted must enforce exclusivity.
+
+### D) API Compatibility Strategy
+
+- Use additive API evolution in V4.
+- In development mode, the current revision is authoritative; long-lived legacy compatibility layers are not required.
+- Deprecations should be explicit and short-lived, with removals performed in subsequent revisions.
+
+### E) Document Type Rollout Strategy
+
+- Use a minimal registry rollout for the current corpus: no `document_type_alias` helper table.
+- Perform a one-time manual mapping of existing document types to canonical registry types.
+
+### F) Suggestion Generation Strategy (Still Open)
+
+- Option A: deterministic rules/NLP over transcribed text.
+- Option B: LLM extraction with confidence/evidence spans.
+
+## Compatibility and Rollout
+
+- Existing V3 author links migrate to `asserted` behavior.
+- Backfill validation identifies policy conflicts before constraints are enforced.
+- Preserve existing V3 core behavior where unaffected by role/suggestion evolution.
+
+## Exit Criteria for Scope Freeze
+
+V4 scope is considered frozen when:
+
+- Suggestion generation strategy is chosen.
+- Suggestion acceptance workflow details are approved.
+- Additive API change list and deprecation schedule are approved.
+
+## Core V4 Documents (Current Set)
+
+1. `docs/ver4/index_v4.md`
+2. `docs/ver4/requirements_v4.md`
+3. `docs/ver4/schema_v4.md`
+4. `docs/ver4/architecture_v4.md`
+5. `docs/ver4/error_handling_v4.md`
+6. `docs/ver4/implementation_plan_v4.md`