unified implementation plan

docs/ver2/ver2.md

@@ -1,183 +0,0 @@
-# Version 2 Plan
-
-## Purpose
-
-Version 2 updates the existing SQLModel domain schema to support multi-page documents, page-level transcription results, richer document metadata, and author/recipient attribution.
-
-PostgreSQL support is already present in the database runtime. V2 does not require a database-layer rewrite or a general SQLite-to-PostgreSQL migration system. PostgreSQL adoption consists primarily of selecting the existing PostgreSQL settings, provisioning the database, creating the V2 schema, and verifying the application against it.
-
-The main implementation effort is the schema update and the application changes that depend on it.
-
----
-
-## Current State
-
-- The application uses Python 3.12, Pydantic v2, SQLModel, and async SQLAlchemy sessions.
-- The database engine already supports both SQLite and PostgreSQL through `SqliteSettings` and `PostgresSettings`.
-- The PostgreSQL async driver is installed and the engine already builds `postgresql+asyncpg` connections.
-- Schema bootstrap currently uses `SQLModel.metadata.create_all()`.
-- SQLite remains the default local configuration and the current Compose configuration still selects SQLite.
-- The current V1 domain contains `Document`, `Source`, `Job`, and `Revision` tables.
-- The V2 target is defined in [V2 DB Schema](V2%20DB%20Schema.md) and [V2 PostgreSQL DDL Specification](V2%20PostgreSQL%20DDL%20Specification.md).
-
----
-
-## V2 Outcomes
-
-1. **V2 schema implemented**
-    - SQLModel models, relationships, enums, constraints, and indexes match the approved V2 schema.
-1. **Page-level batch processing supported**
-    - A job can process multiple sources and retain an independent result for each source through `JobSource`.
-1. **Document metadata expanded**
-    - Documents support ordered pages, descriptive metadata, and multiple authors and recipients.
-1. **Raw provider data retained**
-    - Complete provider payloads are stored in PostgreSQL `JSONB` without flattening or discarding fields.
-    - Stored documents remain suitable for a future MongoDB import if one is ever needed.
-1. **PostgreSQL enabled through configuration**
-    - The application starts against a provisioned PostgreSQL database using the existing runtime path.
-1. **Existing workflows remain reliable**
-    - Upload, worker execution, status inspection, and transcription revision work with the new schema.
-
----
-
-## Non-Goals
-
-- Rewriting the database engine or session layer
-- Building a general-purpose SQLite-to-PostgreSQL migration utility
-- Rehearsing a production database cutover when no production dataset requires preservation
-- Running or integrating MongoDB in V2
-- Building MongoDB projections, synchronization, or fallback behavior
-- Replacing Python, Pydantic, SQLModel, or SQLAlchemy
-- Supporting more than one active human revision per source
-
-If an existing SQLite dataset must be retained, define a small one-time import task separately. It is not part of the default V2 implementation path.
-
----
-
-## Scope
-
-### A) SQLModel schema update (primary)
-
-- Add `Person`, `DocumentPerson`, and `JobSource` models.
-- Expand `Document` with type, date, location, notes, archive identifier, and timestamps.
-- Update `Source` with page ordering, active raw transcription, revised text, and revision timestamp.
-- Update `Job` for batch execution and the `partial_success` terminal state.
-- Replace the standalone `Revision` table with revision fields on `Source`.
-- Remove the direct `Source.job_id` relationship; connect sources to jobs through `JobSource`.
-- Add role, job status, and job-source status enums.
-- Add required uniqueness constraints, foreign-key delete behavior, lookup indexes, and PostgreSQL JSON indexes.
-- Keep model definitions aligned with [V2 Python Pydantic Models](V2%20Python%20Pydantic%20Models.md).
-
-### B) Raw document storage
-
-- Store structured AI metadata in `job_source.ai_metadata` as `JSONB`.
-- Store the complete raw provider response in `job_source.raw_api_response` as `JSONB`.
-- Preserve the original document structure, field names, nested values, and unknown fields in the raw response.
-- Keep validation of extracted application fields separate from retention of the raw response.
-- Serialize UUIDs and datetimes using portable string representations.
-- Use MongoDB Extended JSON representations only if a provider value cannot be represented faithfully in standard JSON.
-- Do not store opaque BSON bytes in PostgreSQL unless a future payload contains BSON-only values that cannot be preserved in `JSONB`.
-
-### C) Schema creation and verification
-
-- Use a fresh V2 database during development unless preservation of existing data becomes a requirement.
-- Create the schema from SQLModel metadata and verify it against the approved DDL.
-- Keep SQLite available for fast unit tests where its behavior is equivalent.
-- Add focused PostgreSQL integration tests for native UUIDs, JSON storage, constraints, indexes, and transactions.
-- Introduce migration tooling only if V2 must update a populated deployed database in place.
-
-### D) Service and worker alignment
-
-- Update document, source, job, and store operations for the new relationships.
-- Create one `JobSource` row per source included in a job.
-- Persist page-level status, transcription, AI metadata, raw provider response, and errors on `JobSource`.
-- Derive the parent job status from its page results:
-  - `completed` when all pages succeed
-  - `partial_success` when successful and failed pages are mixed
-  - `failed` when all pages fail or a job-level failure prevents execution
-- Update `Source.raw_transcription` after a successful page result while keeping the original `JobSource.raw_transcription` immutable.
-- Read `Source.revised_text` in preference to `Source.raw_transcription` when presenting active text.
-
-### E) Document and multi-image workflows
-
-- Require a document before associating uploaded sources.
-- Support uploading multiple images into one document.
-- Preserve page order through `Source.page_number`.
-- Allow a job to include one or more sources from the same document.
-- Update job details to show the document, each source filename, page order, page status, and page-level errors.
-- Define a practical upload limit and split oversized selections into manageable batches if needed.
-
-### F) PostgreSQL configuration
-
-- Provision PostgreSQL for local and deployed environments.
-- Configure the existing `Settings.database` field with PostgreSQL host, port, database, user, and password values.
-- Update Compose and environment configuration to stop selecting SQLite.
-- Decide whether schema bootstrap is enabled for local development or performed as a separate deployment step.
-- Run a connectivity and schema smoke test against PostgreSQL.
-- Keep uploaded files on a persistent, backup-capable path outside the application image.
-
----
-
-## Milestones
-
-### M1 - Schema models
-
-- Implement the V2 SQLModel models and enums.
-- Implement relationships, constraints, indexes, and JSON column types.
-- Update the Pydantic data contracts where model decisions change.
-- Add schema-focused tests.
-
-**Exit criteria:** SQLModel metadata represents the approved V2 schema and schema tests pass.
-
-### M2 - Persistence and worker behavior
-
-- Update database operations and services for the V2 entities.
-- Implement page-level `JobSource` execution records.
-- Preserve complete raw provider responses in `JSONB`.
-- Implement aggregate job status calculation.
-- Add transaction, partial-success, and failure-isolation tests.
-
-**Exit criteria:** single-page and multi-page jobs persist correct page, raw payload, and aggregate states.
-
-### M3 - Document and upload workflows
-
-- Update document creation and source association flows.
-- Add ordered multi-image upload.
-- Update job and document detail views for page-level results.
-- Add focused UI and service tests.
-
-**Exit criteria:** a user can create a document, upload ordered pages, run a job, and inspect each result.
-
-### M4 - PostgreSQL verification and release
-
-- Switch local or test configuration to the existing PostgreSQL runtime path.
-- Create the V2 schema in a fresh PostgreSQL database.
-- Run PostgreSQL-specific schema and workflow tests.
-- Document startup, backup, and recovery settings.
-- Run the final regression suite.
-
-**Exit criteria:** V2 workflows pass against PostgreSQL and release checks are complete.
-
----
-
-## Risks and Mitigations
-
-- **Model and DDL drift** -> compare generated metadata with the approved schema and test named constraints and indexes.
-- **Raw payload loss** -> retain the complete provider response separately from validated and extracted fields.
-- **Cross-database differences** -> retain fast SQLite tests but verify PostgreSQL-native UUID, JSON, and index behavior in integration tests.
-- **Batch state errors** -> test all-success, mixed-result, and all-failed jobs explicitly.
-- **Page ordering errors** -> enforce uniqueness and ordering rules for document pages.
-- **Unexpected data-preservation need** -> confirm whether existing SQLite data matters before implementation; add a one-time importer only when required.
-- **Worker regressions** -> preserve terminal-state and retry reliability tests while changing persistence ownership.
-
----
-
-## Suggested First Tasks
-
-1. Update `src/transcription/db/models.py` to represent the approved V2 schema.
-2. Add schema tests for tables, columns, relationships, constraints, indexes, and enums.
-3. Define and test lossless raw provider response storage in `job_source.raw_api_response`.
-4. Update database operations and services to use `JobSource` and source-level revisions.
-5. Add page-result aggregation tests before changing the worker workflow.
-6. Update document and multi-image upload flows.
-7. Select PostgreSQL in configuration and run the integration suite against a fresh V2 database.