Update V1 & V2 core documents and reorganize docs folder

2026-07-31 10:04:07 -05:00
parent d2b793ea69
commit 3eefc36239
25 changed files with 671 additions and 783 deletions
@@ -0,0 +1,248 @@
+# Implementation Plan (Version 2)
+
+This plan defines the path from the V1 baseline to **Version 2 complete**, aligned to the updated multi-image and multi-person relational domain model:
+
+* `Document` acts as a logical parent container for physical artifacts, supporting multi-author and multi-recipient relationships via `DocumentPerson`.
+* `Source` represents an individual image page within a document, maintaining sequential order (`page_number`), cached active machine output (`raw_transcription`), and inline single user revisions (`revised_text`).
+* `Job` acts as an overarching batch orchestrator for multi-page async processing tasks.
+* `JobSource` records individual point-in-time API executions per image page, storing Pydantic-validated `ai_metadata` and raw REST envelopes (`raw_api_response`).
+* **Pydantic V2** acts as the single source of truth for runtime validation, API payload parsing, and PostgreSQL JSONB serialization.
+
+The objective is to complete the V2 scope with production readiness while keeping non-V2 enhancements out of active delivery.
+
+---
+
+## V2 Completion Definition
+
+V2 is complete when all of the following are true:
+
+1. **Functional complete**
+* Multi-image and whole-folder uploads assign sequential page numbers to `Source` records under a single `Document`.
+* Batch jobs process pages concurrently using an `asyncio` worker pool with semaphore rate limiting.
+* Partial job failures resolve cleanly to `partial_success`, allowing single-page retries without re-running successful pages.
+* Multi-author and multi-recipient tagging is supported on `Document`.
+
+
+2. **Data-model complete**
+* SQLite is fully replaced with PostgreSQL (using `asyncpg` or `psycopg3`).
+* Pydantic V2 models validate all API payloads, database row mappings, and `JSONB` structures.
+
+
+3. **Operational complete**
+* Concurrency controls, worker pool metrics, and database connections operate safely under batch load.
+
+
+4. **Documentation complete**
+* `schema_v2.md`, `DDL_v2.sql`, Pydantic model contracts are updated and consistent.
+
+
+
+---
+
+## Phase 1 — Data Contract Stabilization & Pydantic Baseline
+
+**Goal:** Lock the PostgreSQL schema, DDL, and Pydantic V2 models before refactoring service logic.
+
+### Tasks
+
+1. Finalize DDL for PostgreSQL native types (`UUID`, `TIMESTAMPTZ`, `JSONB`) and junction tables (`document_person`, `job_source`).
+2. Build core Pydantic V2 schemas (`Person`, `Document`, `Source`, `Job`, `JobSource`, `PageAIMetadata`).
+3. Confirm and document data invariants:
+* `source.raw_transcription` and `job_source.raw_transcription` are immutable machine outputs.
+* `source.revised_text` holds user edits. UI renders `COALESCE(revised_text, raw_transcription)`.
+* Page sequence is strictly ordered by `source.page_number ASC`.
+
+
+4. Freeze V2 job status values (`queued`, `processing`, `completed`, `partial_success`, `failed`) and page execution status values (`pending`, `transcribed`, `failed`).
+
+### Deliverables
+
+* Canonical `docs/schema_v2.md` and `docs/DDL_v2.sql`.
+* Centralized Pydantic validation suite in `models/schemas_v2.py`.
+
+### Exit Criteria
+
+* All database tables, relationships, and JSONB structures have corresponding Pydantic V2 models passing unit validation tests.
+
+---
+
+## Phase 2 — Persistence Layer Transition (SQLite to PostgreSQL)
+
+**Goal:** Replace the SQLite storage layer with an asynchronous PostgreSQL driver (`asyncpg` or `psycopg3`).
+
+### Tasks
+
+1. Configure PostgreSQL database connection pooling and environment configuration.
+2. Refactor `services/store.py` / repository layers to execute parameterized async SQL queries (`$1`, `$2`).
+3. Implement JSONB serialization and deserialization helpers using Pydantic's `.model_dump_json()` and `.model_validate()`.
+4. Implement database bootstrap routines for PostgreSQL table creation and index initialization.
+
+### Deliverables
+
+* PostgreSQL-native database connection and query service modules.
+* Integration test suite confirming connection pooling and JSONB CRUD operations.
+
+### Exit Criteria
+
+* All database reads/writes run asynchronously against PostgreSQL with zero remaining SQLite driver dependencies.
+
+---
+
+## Phase 3 — Service Layer & `asyncio` Engine Refactor
+
+**Goal:** Implement batch orchestration and parallel single-image API execution.
+
+### Tasks
+
+1. Refactor upload service to process folder/multi-image input:
+* Group files into a single `Document`.
+* Create ordered `Source` rows (`page_number = 1..N`).
+
+
+2. Refactor `services/workflows.py` with `asyncio` worker pools:
+* Use `asyncio.Semaphore` to enforce API provider rate limits.
+* Issue parallel single-image requests to Vision APIs (OpenAI/Claude).
+* Parse API responses directly into Pydantic models (`PageAIMetadata`).
+
+
+3. Update execution tracking:
+* Create a `JobSource` row per page call to record `raw_transcription`, `ai_metadata`, and `raw_api_response`.
+* Update active `source.raw_transcription` upon task completion.
+* Calculate aggregate batch status (`completed`, `partial_success`, `failed`) on the parent `Job`.
+
+
+4. Refactor `services/person.py` and `services/documents.py` to handle multi-person roles via `document_person`.
+
+### Deliverables
+
+* Asynchronous batch execution engine in `services/workflows.py`.
+* Service routines for multi-person tagging and page-level retries.
+
+### Exit Criteria
+
+* Executing a folder upload of 10+ images processes concurrently, populates page-level `JobSource` entries, and handles partial worker errors without crashing the batch.
+
+---
+
+## Phase 4 — UI & API Contract Alignment
+
+**Goal:** Update API endpoints and frontend/UI views to render multi-page documents and person roles.
+
+### Tasks
+
+1. Update document and job API endpoints to accept batch file arrays and multi-person ID payloads.
+2. Update UI document views:
+* Render multi-page document transcriptions sequentially by `page_number`.
+* Display author and recipient chips/cards linked from `document_person`.
+
+
+3. Update job detail UI to show page-level execution statuses (`transcribed` vs. `failed`) and provide a "Retry Failed Pages" action for `partial_success` jobs.
+4. Align inline page editing controls to update `source.revised_text` and `source.date_revised`.
+
+### Deliverables
+
+* Refactored API routes and UI components supporting multi-page rendering and person management.
+
+### Exit Criteria
+
+* UI successfully displays multi-page document text, allows per-page human revisions, and shows author/recipient metadata.
+
+---
+
+## Phase 5 — Test Suite Realignment & Concurrency Testing
+
+**Goal:** Ensure end-to-end system stability under concurrent async execution and load.
+
+### Tasks
+
+1. Write unit tests for Pydantic models, custom validators, and JSONB conversions.
+2. Write integration tests for async database operations:
+* CRUD for `Document`, `Person`, `DocumentPerson`, `Source`, `Job`, and `JobSource`.
+
+
+3. Write mock-backed async workflow tests:
+* Verify `asyncio.Semaphore` bounds concurrent tasks properly.
+* Validate state transition logic for `completed`, `partial_success`, and `failed` jobs.
+* Confirm retry routines process only targeted `JobSource` records marked as `failed`.
+
+
+4. Re-enable CI quality gates (linting, type checking with Pyright/mypy, pytest).
+
+### Deliverables
+
+* Passing asynchronous test suite covering core workflows, edge cases, and failure recoveries.
+
+### Exit Criteria
+
+* CI pipeline is green with comprehensive coverage across database operations, Pydantic models, and worker queues.
+
+---
+
+## Phase 6 — Reliability, Operations, and Release Readiness
+
+**Goal:** Prepare V2 for production deployment and operator management.
+
+### Tasks
+
+1. Verify structured logging includes `job_id`, `document_id`, `source_id`, and `person_id`.
+2. Tune PostgreSQL connection pool limits and `asyncio` concurrency thresholds for production infrastructure.
+3. Update operational documentation:
+* Review and update `docs/schema_v2.md` as needed.
+* Create `docs/runbook_v2.md` detailing PostgreSQL maintenance, JSONB index management, and worker queue monitoring.
+* Create `docs/release_checklist_v2.md` for launch sign-off.
+
+
+
+### Deliverables
+
+* Updated project documentation and operational runbooks.
+* V2 release sign-off checklist.
+
+### Exit Criteria
+
+* All documentation reflects V2 architecture; launch checklist is fully verified.
+
+---
+
+## Requirement Traceability Focus
+
+Maintain evidence against these V2 requirement groups:
+
+* **Batch & Multi-Image Pipeline:** Folder ingestion, page ordering, async worker execution.
+* **Database & Persistence:** PostgreSQL, native UUIDs, JSONB execution storage, `asyncpg` pooling.
+* **Validation & Schemas:** Pydantic V2 models for DB rows, API requests, and AI vision responses.
+* **Attribution & Metadata:** Multi-author and multi-recipient tagging, biographical entity management.
+* **Error Recovery:** Partial success states, page-level status flags, isolated retry execution.
+
+---
+
+## Scope Discipline Rule (V2 Focus)
+
+* Only tasks required for V2 scope (PostgreSQL, Pydantic V2, folder/async processing, multi-person roles) enter this plan.
+* V3 candidate features (such as side-by-side multi-provider model output comparison) remain strictly in the future backlog.
+* Any schema adjustments during implementation require immediate updates to `DDL_v2.sql`, Pydantic models, and `schema_v2.md`.
+
+---
+
+## Technology References
+
+- [FastAPI documentation](https://fastapi.tiangolo.com/)
+- [NiceGUI documentation](https://nicegui.io/documentation)
+- [PostgreSQL documentation](https://www.postgresql.org/docs/)
+- [Python asyncio](https://docs.python.org/3/library/asyncio.html#module-asyncio)
+- [Pydantic Validation](https://pydantic.dev/docs/validation/latest/get-started/)
+- [Pydantic AI](https://pydantic.dev/docs/ai/overview/)
+
+## Related Local References
+
+- [System Overview](index_v2.md)
+- [System Design Intent](intent.md)
+- [Transcription Methodology](transcription_methodology.md)
+- [System Architecture](architecture_v2.md)
+- [System Requirements](requirements_v2.md)
+- [Data model](schema_v2.md)
+- [Error Handling Policy](error_handling_v2.md)
+- Implementation Plan (this document)
+ 
+
+