Updated documentation to v3 which will focus on capturing prompt/response interactions with AI.

2026-08-08 16:16:32 -05:00
parent 58faa00d7b
commit 5a741de0a9
12 changed files with 527 additions and 0 deletions
@@ -0,0 +1,136 @@
+# System Architecture (Version 2)
+
+This document describes the V2 production architecture of the personal historical-document transcription system.
+
+## Architecture Objectives
+
+* Preserve source material as immutable transcribed text alongside page-level spatial AI metadata.
+* Support batching multi-image and folder uploads cleanly into sequential pages (`page_number`).
+* Leverage asynchronous worker pools (`asyncio`) for parallel single-image API execution bounded by rate limiters (`asyncio.Semaphore`).
+* Migrate persistence to PostgreSQL using native `UUID`, `TIMESTAMPTZ`, and `JSONB` document storage.
+* Standardize all data validation, API parsing, and database models on **Pydantic V2**.
+* Support rich historical attribution (multi-author and multi-recipient relationships).
+
+## Runtime Topology
+
+The V2 runtime operates as an asynchronous Python application:
+
+* FastAPI + NiceGUI web application process.
+* In-process `asyncio` background task orchestrator for parallel API execution.
+* Relational persistence via PostgreSQL (using `asyncpg` or `psycopg3`).
+* Pydantic V2 validation layer wrapping API payloads and PostgreSQL `JSONB` schemas.
+
+```mermaid
+flowchart LR
+U[Browser User] --> A[FastAPI + NiceGUI App]
+A --> W[Asyncio Worker Engine]
+A --> DB[(PostgreSQL Database)]
+W --> P[Vision Provider APIs\nOpenAI / Claude]
+W --> DB
+```
+
+## Lifecycle Ownership
+
+Application lifespan owns runtime setup/teardown:
+
+* Initialize environment logging and Pydantic configuration.
+* Manage asynchronous PostgreSQL connection pools (`asyncpg` / `psycopg3`).
+* Execute database migrations and index initialization.
+* Recover stale processing jobs on startup.
+* Manage graceful shutdown of active `asyncio` worker pools.
+
+## Layered Module Structure
+
+### Interface Layer
+
+* `src/transcription/ui/**` (NiceGUI pages, multi-page renderers, person cards)
+* `src/transcription/api/**` (FastAPI routes and JSON error handlers)
+
+### Application & Async Worker Layer
+
+* `src/transcription/services/workflows.py`
+* `src/transcription/worker.py`
+
+Responsibilities:
+
+* Batch orchestration and status transitions (`queued` -> `processing` -> `completed` | `partial_success` | `failed`).
+* Parallel single-image API execution using `asyncio.gather` bounded by `asyncio.Semaphore`.
+* Pydantic schema parsing (`PageAIMetadata`) and validation prior to database storage.
+
+### Domain & Service Layer
+
+* `src/transcription/db/models.py` (SQLModel/Pydantic V2 schema definitions for the current implementation)
+* `src/transcription/services/*.py` (Transactional operations for `Document`, `Person`, `Source`, `Job`, and `JobSource`)
+
+### Infrastructure Layer
+
+* `src/transcription/db/**` (PostgreSQL connection pooling and raw parameterized SQL execution)
+* `src/transcription/providers/**` (OpenAI & Anthropic Vision SDK adapters)
+
+## Processing Workflow
+
+1. User uploads a folder or batch of images for a `Document`.
+2. System creates `Document`, `Job(status='queued')`, and ordered `Source` pages (`page_number = 1..N`).
+3. Worker claims job, sets `Job.status = 'processing'`, and spawns parallel `asyncio` tasks bounded by semaphore.
+4. Each task calls Vision API for a **single** `Source` image.
+5. On task completion:
+* Writes a `JobSource` record containing `status='transcribed'`, `raw_transcription`, `ai_metadata` (bounding boxes/confidence), and `raw_api_response`.
+* Caches active text to `Source.raw_transcription`.
+
+
+6. On page failure:
+* Writes `JobSource` record with `status='failed'` and `error_detail`.
+
+
+7. Once all page tasks resolve:
+* Marks `Job.status` as `completed` (100% success), `partial_success` (at least 1 success, 1 failure), or `failed` (all failed).
+
+
+
+## Domain Ownership & Invariants
+
+* **Immutable AI Outputs:** `source.raw_transcription` and `job_source.raw_transcription` store original, point-in-time machine output and are immutable.
+* **Inlined Revisions:** Human corrections occur on `source.revised_text`. UI renders `COALESCE(revised_text, raw_transcription)`.
+* **Sequential Integrity:** Multi-page documents are strictly ordered by `source.page_number ASC`.
+* **Page Execution Isolation:** A failure on one page image does not invalidate successful transcriptions on sister pages in the same batch job.
+
+## Data Model Summary
+
+* `Document` has many `Source` pages, many `Job` runs, and many `Person` records via `DocumentPerson` junction (`author` or `recipient`).
+* `Source` belongs to one `Document` and can be processed across many `JobSource` executions.
+* `Job` has many `JobSource` execution records.
+
+## Test Strategy
+
+* Unit tests for Pydantic V2 schemas, custom validators, and JSONB serialization.
+* Integration tests for async PostgreSQL connection handling and parameterized queries.
+* Async workflow tests using mock AI providers to verify `partial_success` and retry logic.
+* UI integration tests for multi-page rendering and person management.
+
+---
+
+## Technology References
+
+- [FastAPI documentation](https://fastapi.tiangolo.com/)
+- [NiceGUI documentation](https://nicegui.io/documentation)
+- [PostgreSQL documentation](https://www.postgresql.org/docs/)
+- [Python asyncio](https://docs.python.org/3/library/asyncio.html#module-asyncio)
+- [Pydantic Validation](https://pydantic.dev/docs/validation/latest/get-started/)
+- [Pydantic AI](https://pydantic.dev/docs/ai/overview/)
+ 
+
+## Related Local References
+
+- [System Overview](index_v2.md)
+- [System Design Intent](invariant/intent.md)
+- [Transcription Methodology](invariant/transcription_methodology.md)
+- System Architecture (this document)
+- [System Requirements](requirements_v2.md)
+- [Data model](schema_v2.md)
+- [Error Handling Policy](error_handling_v2.md)
+- [Implementation Plan](implementation_plan_v2.md)
+
+
+
+
+
@@ -0,0 +1,88 @@
+# Error Handling Policy (Version 2)
+
+This document defines the canonical error-handling policy for the V2 document transcription system.
+
+## Error Handling Objectives
+
+* Make failures visible in clear, actionable language at both the document and individual page levels.
+* Support **isolated failure handling** in multi-image batches so single page errors do not crash an entire batch job.
+* Preserve diagnostic detail (Pydantic validation errors, raw provider responses) in PostgreSQL `JSONB` for fast troubleshooting.
+* Ensure consistent error envelope structure across API, UI, and async worker boundaries.
+
+## Scope And Authority
+
+Governs error behavior across NiceGUI pages, FastAPI routes, service orchestration, `asyncio` background tasks, PostgreSQL interactions, and AI provider adapters.
+
+## Error Taxonomy
+
+| Category | Definition | Retriable |
+| --- | --- | --- |
+| `validation_error` | Pydantic payload or parameter schema validation failure | no |
+| `user_input_error` | Unacceptable user file (unsupported image type, corrupt file) | no |
+| `not_found_error` | Requested resource (`Document`, `Source`, `Person`, `Job`) missing | no |
+| `conflict_error` | Operation violates state constraints (e.g., duplicate `document_person` role) | no |
+| `external_provider_error` | AI Provider API failure (rate limit, vision execution error) | yes |
+| `infrastructure_transient_error` | Temporary DB connection reset or HTTP timeout | yes |
+| `infrastructure_persistent_error` | Database down, missing API credentials, misconfiguration | no |
+| `internal_unexpected_error` | Uncaught Python exception or logic defect | no |
+
+## Async Batch & Page-Level Error Behavior
+
+In multi-image `asyncio` batch processing:
+
+1. **Page Isolation:** Exceptions caught during individual page calls are caught within the `asyncio` task wrapper.
+2. **Page Record Logging:** Page failure detail is written directly to `job_source.error_detail` and `job_source.status = 'failed'`.
+3. **Batch Aggregate State:**
+* If **all** page tasks succeed -> `job.status = 'completed'`.
+* If **some** page tasks fail -> `job.status = 'partial_success'`.
+* If **all** page tasks fail -> `job.status = 'failed'`.
+
+
+4. **Retry Strategy:** The UI exposes a "Retry Failed Pages" option for `partial_success` jobs, which spawns a new targeted `Job` containing *only* the `Source` IDs marked as `failed`.
+
+## API Error Response Contract
+
+API error responses return a structured JSON envelope:
+```json
+{
+"error_id": "err_uuid_12345",
+"category": "validation_error",
+"message": "The uploaded payload failed schema validation.",
+"suggestion": "Check file format and metadata fields, then try again.",
+"details": {
+"pydantic_errors": [...]
+},
+"timestamp": "2026-07-31T07:55:00Z"
+}
+```
+
+HTTP Status Mappings:
+
+* `validation_error`, `user_input_error` -> `400`
+* `not_found_error` -> `404`
+* `conflict_error` -> `409`
+* `external_provider_error` -> `502` / `503`
+* `infrastructure_transient_error` -> `503`
+* `infrastructure_persistent_error`, `internal_unexpected_error` -> `500`
+
+---
+
+## Technology References
+
+- [FastAPI documentation](https://fastapi.tiangolo.com/)
+- [NiceGUI documentation](https://nicegui.io/documentation)
+- [PostgreSQL documentation](https://www.postgresql.org/docs/)
+- [Python asyncio](https://docs.python.org/3/library/asyncio.html#module-asyncio)
+- [Pydantic Validation](https://pydantic.dev/docs/validation/latest/get-started/)
+- [Pydantic AI](https://pydantic.dev/docs/ai/overview/)
+
+## Related Local References
+
+- [System Overview](index_v2.md)
+- [System Design Intent](invariant/intent.md)
+- [Transcription Methodology](invariant/transcription_methodology.md)
+- [System Architecture](architecture_v2.md)
+- [System Requirements](requirements_v2.md)
+- [Data model](schema_v2.md)
+- Error Handling Policy (this document)
+- [Implementation Plan](implementation_plan_v2.md)
@@ -0,0 +1,64 @@
+# implementation_plan_v2
+
+## Goal
+
+Replace the current V1 SQLModel schema with the approved V2 schema and make sure every database operation works through the existing async SQLAlchemy/SQLModel session layer.
+
+Use a fresh database. There will be no migrations, data conversion, legacy compatibility shims, or parallel V1/V2 code paths.
+
+## Current Project Impact
+
+- `src/transcription/db/models.py` still defines the V1 `Document`, `Source`, `Job`, and `Revision` tables.
+- The V2 target adds `Person`, `DocumentPerson`, and `JobSource`, moves revisions onto `Source`, and removes the direct `Source.job_id` relationship.
+- The engine, session factory, transaction handling, and PostgreSQL async support already exist and do not need to be rewritten.
+- Async CRUD currently lives in `DocumentService`, `JobService`, `TranscriptionService`, and the upload record helper. Their queries and eager-loading options depend on V1 relationships.
+- Existing tests cover only part of the schema and CRUD surface.
+
+## Implementation
+
+### 1. Update the schema
+
+- Replace the models in `src/transcription/db/models.py` with the approved V2 tables, enums, relationships, foreign keys, constraints, and indexes.
+- Remove `Revision`, `Source.job_id`, and the transcription fields that no longer belong on `Job`.
+- Keep `create_all()` as the schema bootstrap for a fresh database.
+- Delete `_ensure_sqlite_compat_columns()` and all schema patching from `src/transcription/db/operations.py`.
+- Keep the Python models and `docs/schema_v2.md` consistent.
+
+### 2. Align the async CRUD methods
+
+- Keep the existing `ServiceBase` session and transaction pattern.
+- Update document CRUD to load and manage its ordered `Source` rows and `DocumentPerson` links.
+- Update job CRUD and queue queries to use `JobSource` instead of `Source.job_id`.
+- Add the missing async CRUD operations for `Person`, `Source`, `DocumentPerson`, and `JobSource` using the existing service style. Do not add another repository abstraction.
+- Replace revision CRUD with direct updates to `Source.revised_text` and `Source.date_revised`.
+- Remove the temporary transcript compatibility aliases instead of redirecting them.
+- Update only direct database call sites that construct or query these records; UI and worker feature changes are not part of this work.
+
+### 3. Verify the schema and CRUD
+
+- Update the schema bootstrap test to expect `person`, `document`, `document_person`, `source`, `job`, and `job_source`, with no `revision` table.
+- Add async create, read, update, delete, list, and filtered-query tests for each entity that exposes those operations.
+- Test relationship loading, page ordering, uniqueness constraints, delete behavior, status values, and `JobSource` JSON fields.
+- Test both service-owned sessions and caller-provided sessions so flush/commit behavior remains correct.
+- Run the focused database and service tests, then the full suite with `uv run pytest`.
+
+### 4. Update the UI for the V2 schema
+
+- Review the UI components and views that display document, job, person, and source data so they reference the V2 schema instead of V1 relationships.
+- Update upload, detail, and listing screens to show the new person and source associations, revised-source fields, and the revised status values.
+- Keep the UI behavior aligned with the updated service layer and ensure the existing UI tests continue to pass with the V2 data model.
+- Consider the guidance in `docs/ui_style_guide.md` when making UI changes so the updated views remain consistent with the project’s visual and interaction conventions.
+
+## Done When
+
+- A fresh database is created directly from the V2 SQLModel metadata.
+- All async CRUD methods pass against the V2 relationships and fields.
+- No code references `Revision`, `Source.job_id`, removed `Job` transcription fields, or compatibility aliases.
+- The focused tests and full test suite pass.
+
+## Out of Scope
+
+- Database migrations or preservation of V1 data
+- Legacy compatibility code
+- Database engine or session-layer rewrites
+- UI redesign, batch orchestration, worker concurrency, deployment, and operational runbooks
@@ -0,0 +1,47 @@
+# Document Transcription System Overview (Version 2)
+
+This project is a personal-scale application for transcribing, indexing, and preserving historical family documents, letters, postcards, and journals.
+
+## Start Here
+
+Read [architecture_v2.md](architecture_v2.md) first for technical overview and system design.
+
+## Core V2 Capabilities
+
+* **Folder & Multi-Image Ingestion:** Upload whole folders or image batches that map sequentially (`page_number`) under a single `Document`.
+* **Parallel Async AI Vision Engine:** Concurrently process single-page image transcriptions using Python `asyncio` bounded by rate limiters.
+* **Robust PostgreSQL Storage:** Relational storage for entities with native `UUID`, `TIMESTAMPTZ`, and `JSONB` for deep AI spatial metadata and raw envelopes.
+* **Pydantic V2 Validation:** End-to-end type safety, DB row mapping, and JSONB payload validation.
+* **Historical Person Management:** Track authors and recipients across documents with rich biographical entities (`Person`).
+* **Page-Level Execution Auditing & Revisions:** Store immutable point-in-time machine output per run while enabling inline human corrections (`revised_text`).
+* **Partial Failure Recovery:** Bounded batch execution that isolates single-page API errors (`partial_success`) for simple retries.
+
+## Technical Stack
+
+* **Application Web Framework:** FastAPI + NiceGUI
+* **Persistence Engine:** PostgreSQL 18+
+* **Data Validation & Schemas:** Pydantic V2
+* **Concurrency & Workers:** Python `asyncio` worker pool with `asyncio.Semaphore`
+* **Vision Providers:** OpenAI (GPT-4o) and Anthropic (Claude 3.5 Sonnet) via native SDKs
+
+---
+
+## Technology References
+
+- [FastAPI documentation](https://fastapi.tiangolo.com/)
+- [NiceGUI documentation](https://nicegui.io/documentation)
+- [PostgreSQL documentation](https://www.postgresql.org/docs/)
+- [Python asyncio](https://docs.python.org/3/library/asyncio.html#module-asyncio)
+- [Pydantic Validation](https://pydantic.dev/docs/validation/latest/get-started/)
+- [Pydantic AI](https://pydantic.dev/docs/ai/overview/)
+ 
+## Documentation Index
+
+- System Overview (this document)
+- [System Design Intent](invariant/intent.md)
+- [Transcription Methodology](invariant/transcription_methodology.md)
+- [System Architecture](architecture_v2.md)
+- [System Requirements](requirements_v2.md)
+- [Data model](schema_v2.md)
+- [Error Handling Policy](error_handling_v2.md)
+- [Implementation Plan](implementation_plan_v2.md)
@@ -0,0 +1,42 @@
+# Document Transcription System Requirements (Version 2)
+
+This document captures the **Version 2 baseline requirements** for the production implementation.
+
+## Requirements Model
+
+| ID | Category | Requirement | Verify Method |
+| --- | --- | --- | --- |
+| REQ-0 | System | Provide end-to-end multi-page document transcription with persistent, inspectable async job states. | demonstration |
+| REQ-1 | Functional | Allow users to upload folders or multi-image batches as sequential `Source` pages under a `Document`. | test |
+| REQ-2 | Functional | Process multi-page jobs asynchronously using an `asyncio` worker pool bounded by rate limits. | test |
+| REQ-3 | Functional | Persist page-level execution outputs (`raw_transcription`, `ai_metadata`, `raw_api_response`) on `JobSource`. | test |
+| REQ-4 | Functional | Support job states (`queued`, `processing`, `completed`, `partial_success`, `failed`) and page states (`pending`, `transcribed`, `failed`). | inspection |
+| REQ-5 | Functional | Allow users to manage historical `Person` records and link multiple authors/recipients to a `Document` via `DocumentPerson`. | test |
+| REQ-6 | Functional | Maintain immutable original machine output on `Source.raw_transcription` while permitting inline human edits on `Source.revised_text`. | test |
+| REQ-7 | Data Constraint | Store all persistent domain data in PostgreSQL using native `UUID`, `TIMESTAMPTZ`, and `JSONB` columns. | inspection |
+| REQ-8 | Data Constraint | Validate all API requests, database rows, and JSONB structures using Pydantic V2 schemas. | test |
+| REQ-9 | Interface | Render multi-page transcriptions sequentially by `page_number` in the web UI with author/recipient metadata. | demonstration |
+| REQ-10 | Operations | Allow operators to retry only failed pages for jobs in a `partial_success` state. | test |
+
+## Element Satisfaction Mapping
+
+* **UI (NiceGUI):** Satisfies REQ-1, REQ-5, REQ-6, REQ-9, REQ-10.
+* **API (FastAPI):** Satisfies REQ-1, REQ-4, REQ-5, REQ-8.
+* **WORKER (asyncio):** Satisfies REQ-2, REQ-3, REQ-4, REQ-10.
+* **PERSISTENCE (PostgreSQL):** Satisfies REQ-3, REQ-6, REQ-7.
+* **MODELS (Pydantic V2):** Satisfies REQ-8.
+
+---
+
+## Related Local References
+
+- [System Overview](index_v2.md)
+- [System Design Intent](invariant/intent.md)
+- [Transcription Methodology](invariant/transcription_methodology.md)
+- [System Architecture](architecture_v2.md)
+- System Requirements (this document)
+- [Data model](schema_v2.md)
+- [Error Handling Policy](error_handling_v2.md)
+- [Implementation Plan](implementation_plan_v2.md)
+
+
@@ -0,0 +1,137 @@
+# Database Schema (Version 2)
+
+This document describes the PostgreSQL relational schema for the transcription platform. It incorporates multi-image batch orchestration via `asyncio`, page-level execution tracking, many-to-many author/recipient attribution, and JSONB document storage for AI vision outputs.
+
+All primary and foreign keys are PostgreSQL native UUIDs (`gen_random_uuid()`).
+
+## Entity Relationship Diagram
+
+```mermaid
+erDiagram
+    PERSON {
+        UUID id PK
+        TEXT full_name
+        TEXT display_name
+        TEXT maiden_name
+        DATE birth_date
+        TEXT birth_date_raw
+        TEXT birth_place
+        DATE death_date
+        TEXT death_date_raw
+        TEXT death_place
+        TEXT biography
+        TEXT portrait_path
+        JSONB metadata
+        TIMESTAMPTZ created_at
+        TIMESTAMPTZ updated_at
+    }
+
+    DOCUMENT {
+        UUID id PK
+        TEXT name
+        TEXT document_type
+        DATE document_date
+        TEXT document_date_raw
+        TEXT location_created
+        TEXT notes
+        TEXT archive_identifier
+        TIMESTAMPTZ created_at
+        TIMESTAMPTZ updated_at
+    }
+
+    DOCUMENT_PERSON {
+        UUID id PK
+        UUID document_id FK
+        UUID person_id FK
+        VARCHAR role "author | recipient"
+        TIMESTAMPTZ created_at
+    }
+
+    JOB {
+        UUID id PK
+        UUID document_id FK
+        VARCHAR status "queued | processing | transcribed | completed | partial_success | failed"
+        INTEGER retry_count
+        TEXT provider
+        TEXT model
+        TEXT prompt_name
+        TIMESTAMPTZ date_created
+        TIMESTAMPTZ date_updated
+    }
+
+    SOURCE {
+        UUID id PK
+        UUID document_id FK
+        INTEGER page_number
+        TEXT upload_name
+        TEXT filename
+        TEXT file_path
+        TEXT raw_transcription
+        TEXT revised_text
+        TIMESTAMPTZ date_uploaded
+        TIMESTAMPTZ date_revised
+    }
+
+    JOB_SOURCE {
+        UUID id PK
+        UUID job_id FK
+        UUID source_id FK
+        VARCHAR status "pending | transcribed | failed"
+        TEXT raw_transcription
+        JSONB ai_metadata
+        JSONB raw_api_response
+        TEXT error_detail
+        TIMESTAMPTZ executed_at
+    }
+
+    DOCUMENT ||--o{ DOCUMENT_PERSON : "has_people"
+    PERSON ||--o{ DOCUMENT_PERSON : "participates_in"
+    DOCUMENT ||--o{ JOB : "has_jobs"
+    DOCUMENT ||--o{ SOURCE : "contains_pages"
+    JOB ||--o{ JOB_SOURCE : "executes"
+    SOURCE ||--o{ JOB_SOURCE : "processed_in"
+```
+
+## Domain Invariants & Rules
+
+### Page-Level Execution & AI Outputs
+
+* Execution Granularity: Every single image execution by an AI model produces a dedicated record in job_source.
+* Source vs Execution Status: `source` does not carry a `status` column. Per-source execution state is tracked in `job_source.status` (`pending`, `transcribed`, `failed`).
+* Point-in-Time Auditability: job_source.raw_api_response stores the unparsed REST response envelope for that specific image page call. job_source.ai_metadata stores spatial bounding boxes, token usage, and layout details for that specific image page call.
+* Active Output Caching: Upon successful completion of an image call, source.raw_transcription is updated with the latest output string from job_source.raw_transcription for fast UI rendering.
+
+### Page Ordering & Revisions
+
+* Sequential Integrity: source.page_number dictates page ordering within a document. Reads assembling full documents must query ORDER BY source.document_id, source.page_number ASC.
+* Inlined Human Corrections: User edits occur at the page level inside source.revised_text. source.raw_transcription remains immutable. If source.revised_text is non-null, application frontends must render source.revised_text.
+
+### Async Job Lifecycle & Failure Isolation
+
+* Batch Orchestrator: A job represents an overarching execution run across one or more source images belonging to a document.
+* Isolated Failures: API requests run concurrently (e.g., using asyncio). A failure on page 3 does not invalidate successful transcriptions on page 1 or 2.
+* Job States:
+    - queued: Created, awaiting worker execution.
+    - processing: Concurrent HTTP tasks actively running.
+    - completed: 100% of linked job_source tasks succeeded (transcribed).
+    - partial_success: At least one job_source succeeded and at least one failed.
+    - failed: All linked job_source tasks failed or a job-level runtime error occurred.
+
+### Attribution & Person Roles
+
+* Multi-Person Roles: Documents support zero, one, or many authors and recipients linked via document_person.
+* Role Uniqueness: (document_id, person_id, role) must be unique to prevent duplicate role tagging.
+
+---
+
+## Related Local References
+
+- [System Overview](index_v2.md)
+- [System Design Intent](invariant/intent.md)
+- [Transcription Methodology](invariant/transcription_methodology.md)
+- [System Architecture](architecture_v2.md)
+- [System Requirements](requirements_v2.md)
+- Data model (this document)
+- [Error Handling Policy](error_handling_v2.md)
+- [Implementation Plan](implementation_plan_v2.md)
+