transcription/docs/implementation_plan_v2.md

Implementation Plan (Version 2)

This plan defines the path from the V1 baseline to Version 2 complete, aligned to the updated multi-image and multi-person relational domain model:

• Document acts as a logical parent container for physical artifacts, supporting multi-author and multi-recipient relationships via DocumentPerson.
• Source represents an individual image page within a document, maintaining sequential order (page_number), cached active machine output (raw_transcription), and inline single user revisions (revised_text).
• Job acts as an overarching batch orchestrator for multi-page async processing tasks.
• JobSource records individual point-in-time API executions per image page, storing Pydantic-validated ai_metadata and raw REST envelopes (raw_api_response).
• Pydantic V2 acts as the single source of truth for runtime validation, API payload parsing, and PostgreSQL JSONB serialization.

The objective is to complete the V2 scope with production readiness while keeping non-V2 enhancements out of active delivery.

---

V2 Completion Definition

V2 is complete when all of the following are true:

1. Functional complete

• Multi-image and whole-folder uploads assign sequential page numbers to Source records under a single Document.
• Batch jobs process pages concurrently using an asyncio worker pool with semaphore rate limiting.
• Partial job failures resolve cleanly to partial_success, allowing single-page retries without re-running successful pages.
• Multi-author and multi-recipient tagging is supported on Document.

2. Data-model complete

• SQLite is fully replaced with PostgreSQL (using asyncpg or psycopg3).
• Pydantic V2 models validate all API payloads, database row mappings, and JSONB structures.

3. Operational complete

• Concurrency controls, worker pool metrics, and database connections operate safely under batch load.

4. Documentation complete

• schema_v2.md, DDL_v2.sql, Pydantic model contracts are updated and consistent.

---

Phase 1 — Data Contract Stabilization & Pydantic Baseline

Goal: Lock the PostgreSQL schema, DDL, and Pydantic V2 models before refactoring service logic.

Tasks

1. Finalize DDL for PostgreSQL native types (UUID, TIMESTAMPTZ, JSONB) and junction tables (document_person, job_source).
2. Build core Pydantic V2 schemas (Person, Document, Source, Job, JobSource, PageAIMetadata).
3. Confirm and document data invariants:

• source.raw_transcription and job_source.raw_transcription are immutable machine outputs.
• source.revised_text holds user edits. UI renders COALESCE(revised_text, raw_transcription).
• Page sequence is strictly ordered by source.page_number ASC.

4. Freeze V2 job status values (queued, processing, completed, partial_success, failed) and page execution status values (pending, transcribed, failed).

Deliverables

• Canonical docs/schema_v2.md and docs/DDL_v2.sql.
• Centralized Pydantic validation suite in models/schemas_v2.py.

Exit Criteria

• All database tables, relationships, and JSONB structures have corresponding Pydantic V2 models passing unit validation tests.

---

Phase 2 — Persistence Layer Transition (SQLite to PostgreSQL)

Goal: Replace the SQLite storage layer with an asynchronous PostgreSQL driver (asyncpg or psycopg3).

Tasks

1. Configure PostgreSQL database connection pooling and environment configuration.
2. Refactor services/store.py / repository layers to execute parameterized async SQL queries ($1, $2).
3. Implement JSONB serialization and deserialization helpers using Pydantic's .model_dump_json() and .model_validate().
4. Implement database bootstrap routines for PostgreSQL table creation and index initialization.

Deliverables

• PostgreSQL-native database connection and query service modules.
• Integration test suite confirming connection pooling and JSONB CRUD operations.

Exit Criteria

• All database reads/writes run asynchronously against PostgreSQL with zero remaining SQLite driver dependencies.

---

Phase 3 — Service Layer & asyncio Engine Refactor

Goal: Implement batch orchestration and parallel single-image API execution.

Tasks

1. Refactor upload service to process folder/multi-image input:

• Group files into a single Document.
• Create ordered Source rows (page_number = 1..N).

2. Refactor services/workflows.py with asyncio worker pools:

• Use asyncio.Semaphore to enforce API provider rate limits.
• Issue parallel single-image requests to Vision APIs (OpenAI/Claude).
• Parse API responses directly into Pydantic models (PageAIMetadata).

3. Update execution tracking:

• Create a JobSource row per page call to record raw_transcription, ai_metadata, and raw_api_response.
• Update active source.raw_transcription upon task completion.
• Calculate aggregate batch status (completed, partial_success, failed) on the parent Job.

4. Refactor services/person.py and services/documents.py to handle multi-person roles via document_person.

Deliverables

• Asynchronous batch execution engine in services/workflows.py.
• Service routines for multi-person tagging and page-level retries.

Exit Criteria

• Executing a folder upload of 10+ images processes concurrently, populates page-level JobSource entries, and handles partial worker errors without crashing the batch.

---

Phase 4 — UI & API Contract Alignment

Goal: Update API endpoints and frontend/UI views to render multi-page documents and person roles.

Tasks

1. Update document and job API endpoints to accept batch file arrays and multi-person ID payloads.
2. Update UI document views:

• Render multi-page document transcriptions sequentially by page_number.
• Display author and recipient chips/cards linked from document_person.

3. Update job detail UI to show page-level execution statuses (transcribed vs. failed) and provide a "Retry Failed Pages" action for partial_success jobs.
4. Align inline page editing controls to update source.revised_text and source.date_revised.

Deliverables

• Refactored API routes and UI components supporting multi-page rendering and person management.

Exit Criteria

• UI successfully displays multi-page document text, allows per-page human revisions, and shows author/recipient metadata.

---

Phase 5 — Test Suite Realignment & Concurrency Testing

Goal: Ensure end-to-end system stability under concurrent async execution and load.

Tasks

1. Write unit tests for Pydantic models, custom validators, and JSONB conversions.
2. Write integration tests for async database operations:

• CRUD for Document, Person, DocumentPerson, Source, Job, and JobSource.

3. Write mock-backed async workflow tests:

• Verify asyncio.Semaphore bounds concurrent tasks properly.
• Validate state transition logic for completed, partial_success, and failed jobs.
• Confirm retry routines process only targeted JobSource records marked as failed.

4. Re-enable CI quality gates (linting, type checking with Pyright/mypy, pytest).

Deliverables

• Passing asynchronous test suite covering core workflows, edge cases, and failure recoveries.

Exit Criteria

• CI pipeline is green with comprehensive coverage across database operations, Pydantic models, and worker queues.

---

Phase 6 — Reliability, Operations, and Release Readiness

Goal: Prepare V2 for production deployment and operator management.

Tasks

1. Verify structured logging includes job_id, document_id, source_id, and person_id.
2. Tune PostgreSQL connection pool limits and asyncio concurrency thresholds for production infrastructure.
3. Update operational documentation:

• Review and update docs/schema_v2.md as needed.
• Create docs/runbook_v2.md detailing PostgreSQL maintenance, JSONB index management, and worker queue monitoring.
• Create docs/release_checklist_v2.md for launch sign-off.

Deliverables

• Updated project documentation and operational runbooks.
• V2 release sign-off checklist.

Exit Criteria

• All documentation reflects V2 architecture; launch checklist is fully verified.

---

Requirement Traceability Focus

Maintain evidence against these V2 requirement groups:

• Batch & Multi-Image Pipeline: Folder ingestion, page ordering, async worker execution.
• Database & Persistence: PostgreSQL, native UUIDs, JSONB execution storage, asyncpg pooling.
• Validation & Schemas: Pydantic V2 models for DB rows, API requests, and AI vision responses.
• Attribution & Metadata: Multi-author and multi-recipient tagging, biographical entity management.
• Error Recovery: Partial success states, page-level status flags, isolated retry execution.

---

Scope Discipline Rule (V2 Focus)

• Only tasks required for V2 scope (PostgreSQL, Pydantic V2, folder/async processing, multi-person roles) enter this plan.
• V3 candidate features (such as side-by-side multi-provider model output comparison) remain strictly in the future backlog.
• Any schema adjustments during implementation require immediate updates to DDL_v2.sql, Pydantic models, and schema_v2.md.

---

Technology References

• FastAPI documentation
• NiceGUI documentation
• PostgreSQL documentation
• Python asyncio
• Pydantic Validation
• Pydantic AI

Related Local References

• System Overview
• System Design Intent
• Transcription Methodology
• System Architecture
• System Requirements
• Data model
• Error Handling Policy
• Implementation Plan (this document)