transcription/docs/ver2/ver2.md

Version 2 Plan

Purpose

Version 2 updates the existing SQLModel domain schema to support multi-page documents, page-level transcription results, richer document metadata, and author/recipient attribution.

PostgreSQL support is already present in the database runtime. V2 does not require a database-layer rewrite or a general SQLite-to-PostgreSQL migration system. PostgreSQL adoption consists primarily of selecting the existing PostgreSQL settings, provisioning the database, creating the V2 schema, and verifying the application against it.

The main implementation effort is the schema update and the application changes that depend on it.

---

Current State

• The application uses Python 3.12, Pydantic v2, SQLModel, and async SQLAlchemy sessions.
• The database engine already supports both SQLite and PostgreSQL through SqliteSettings and PostgresSettings.
• The PostgreSQL async driver is installed and the engine already builds postgresql+asyncpg connections.
• Schema bootstrap currently uses SQLModel.metadata.create_all().
• SQLite remains the default local configuration and the current Compose configuration still selects SQLite.
• The current V1 domain contains Document, Source, Job, and Revision tables.
• The V2 target is defined in V2 DB Schema and V2 PostgreSQL DDL Specification.

---

V2 Outcomes

1. V2 schema implemented
   • SQLModel models, relationships, enums, constraints, and indexes match the approved V2 schema.
2. Page-level batch processing supported
   • A job can process multiple sources and retain an independent result for each source through JobSource.
3. Document metadata expanded
   • Documents support ordered pages, descriptive metadata, and multiple authors and recipients.
4. Raw provider data retained
   • Complete provider payloads are stored in PostgreSQL JSONB without flattening or discarding fields.
   • Stored documents remain suitable for a future MongoDB import if one is ever needed.
5. PostgreSQL enabled through configuration
   • The application starts against a provisioned PostgreSQL database using the existing runtime path.
6. Existing workflows remain reliable
   • Upload, worker execution, status inspection, and transcription revision work with the new schema.

---

Non-Goals

• Rewriting the database engine or session layer
• Building a general-purpose SQLite-to-PostgreSQL migration utility
• Rehearsing a production database cutover when no production dataset requires preservation
• Running or integrating MongoDB in V2
• Building MongoDB projections, synchronization, or fallback behavior
• Replacing Python, Pydantic, SQLModel, or SQLAlchemy
• Supporting more than one active human revision per source

If an existing SQLite dataset must be retained, define a small one-time import task separately. It is not part of the default V2 implementation path.

---

Scope

A) SQLModel schema update (primary)

• Add Person, DocumentPerson, and JobSource models.
• Expand Document with type, date, location, notes, archive identifier, and timestamps.
• Update Source with page ordering, active raw transcription, revised text, and revision timestamp.
• Update Job for batch execution and the partial_success terminal state.
• Replace the standalone Revision table with revision fields on Source.
• Remove the direct Source.job_id relationship; connect sources to jobs through JobSource.
• Add role, job status, and job-source status enums.
• Add required uniqueness constraints, foreign-key delete behavior, lookup indexes, and PostgreSQL JSON indexes.
• Keep model definitions aligned with V2 Python Pydantic Models.

B) Raw document storage

• Store structured AI metadata in job_source.ai_metadata as JSONB.
• Store the complete raw provider response in job_source.raw_api_response as JSONB.
• Preserve the original document structure, field names, nested values, and unknown fields in the raw response.
• Keep validation of extracted application fields separate from retention of the raw response.
• Serialize UUIDs and datetimes using portable string representations.
• Use MongoDB Extended JSON representations only if a provider value cannot be represented faithfully in standard JSON.
• Do not store opaque BSON bytes in PostgreSQL unless a future payload contains BSON-only values that cannot be preserved in JSONB.

C) Schema creation and verification

• Use a fresh V2 database during development unless preservation of existing data becomes a requirement.
• Create the schema from SQLModel metadata and verify it against the approved DDL.
• Keep SQLite available for fast unit tests where its behavior is equivalent.
• Add focused PostgreSQL integration tests for native UUIDs, JSON storage, constraints, indexes, and transactions.
• Introduce migration tooling only if V2 must update a populated deployed database in place.

D) Service and worker alignment

• Update document, source, job, and store operations for the new relationships.
• Create one JobSource row per source included in a job.
• Persist page-level status, transcription, AI metadata, raw provider response, and errors on JobSource.
• Derive the parent job status from its page results:
  • completed when all pages succeed
  • partial_success when successful and failed pages are mixed
  • failed when all pages fail or a job-level failure prevents execution
• Update Source.raw_transcription after a successful page result while keeping the original JobSource.raw_transcription immutable.
• Read Source.revised_text in preference to Source.raw_transcription when presenting active text.

E) Document and multi-image workflows

• Require a document before associating uploaded sources.
• Support uploading multiple images into one document.
• Preserve page order through Source.page_number.
• Allow a job to include one or more sources from the same document.
• Update job details to show the document, each source filename, page order, page status, and page-level errors.
• Define a practical upload limit and split oversized selections into manageable batches if needed.

F) PostgreSQL configuration

• Provision PostgreSQL for local and deployed environments.
• Configure the existing Settings.database field with PostgreSQL host, port, database, user, and password values.
• Update Compose and environment configuration to stop selecting SQLite.
• Decide whether schema bootstrap is enabled for local development or performed as a separate deployment step.
• Run a connectivity and schema smoke test against PostgreSQL.
• Keep uploaded files on a persistent, backup-capable path outside the application image.

---

Milestones

M1 - Schema models

• Implement the V2 SQLModel models and enums.
• Implement relationships, constraints, indexes, and JSON column types.
• Update the Pydantic data contracts where model decisions change.
• Add schema-focused tests.

Exit criteria: SQLModel metadata represents the approved V2 schema and schema tests pass.

M2 - Persistence and worker behavior

• Update database operations and services for the V2 entities.
• Implement page-level JobSource execution records.
• Preserve complete raw provider responses in JSONB.
• Implement aggregate job status calculation.
• Add transaction, partial-success, and failure-isolation tests.

Exit criteria: single-page and multi-page jobs persist correct page, raw payload, and aggregate states.

M3 - Document and upload workflows

• Update document creation and source association flows.
• Add ordered multi-image upload.
• Update job and document detail views for page-level results.
• Add focused UI and service tests.

Exit criteria: a user can create a document, upload ordered pages, run a job, and inspect each result.

M4 - PostgreSQL verification and release

• Switch local or test configuration to the existing PostgreSQL runtime path.
• Create the V2 schema in a fresh PostgreSQL database.
• Run PostgreSQL-specific schema and workflow tests.
• Document startup, backup, and recovery settings.
• Run the final regression suite.

Exit criteria: V2 workflows pass against PostgreSQL and release checks are complete.

---

Risks and Mitigations

• Model and DDL drift -> compare generated metadata with the approved schema and test named constraints and indexes.
• Raw payload loss -> retain the complete provider response separately from validated and extracted fields.
• Cross-database differences -> retain fast SQLite tests but verify PostgreSQL-native UUID, JSON, and index behavior in integration tests.
• Batch state errors -> test all-success, mixed-result, and all-failed jobs explicitly.
• Page ordering errors -> enforce uniqueness and ordering rules for document pages.
• Unexpected data-preservation need -> confirm whether existing SQLite data matters before implementation; add a one-time importer only when required.
• Worker regressions -> preserve terminal-state and retry reliability tests while changing persistence ownership.

---

Suggested First Tasks

1. Update src/transcription/db/models.py to represent the approved V2 schema.
2. Add schema tests for tables, columns, relationships, constraints, indexes, and enums.
3. Define and test lossless raw provider response storage in job_source.raw_api_response.
4. Update database operations and services to use JobSource and source-level revisions.
5. Add page-result aggregation tests before changing the worker workflow.
6. Update document and multi-image upload flows.
7. Select PostgreSQL in configuration and run the integration suite against a fresh V2 database.