generated from john/python-template
unified implementation plan
This commit is contained in:
John Lancaster
+41
-232
@@ -1,248 +1,57 @@
|
||||
# Implementation Plan (Version 2)
|
||||
# implementation_plan_v2
|
||||
|
||||
This plan defines the path from the V1 baseline to **Version 2 complete**, aligned to the updated multi-image and multi-person relational domain model:
|
||||
## Goal
|
||||
|
||||
* `Document` acts as a logical parent container for physical artifacts, supporting multi-author and multi-recipient relationships via `DocumentPerson`.
|
||||
* `Source` represents an individual image page within a document, maintaining sequential order (`page_number`), cached active machine output (`raw_transcription`), and inline single user revisions (`revised_text`).
|
||||
* `Job` acts as an overarching batch orchestrator for multi-page async processing tasks.
|
||||
* `JobSource` records individual point-in-time API executions per image page, storing Pydantic-validated `ai_metadata` and raw REST envelopes (`raw_api_response`).
|
||||
* **Pydantic V2** acts as the single source of truth for runtime validation, API payload parsing, and PostgreSQL JSONB serialization.
|
||||
Replace the current V1 SQLModel schema with the approved V2 schema and make sure every database operation works through the existing async SQLAlchemy/SQLModel session layer.
|
||||
|
||||
The objective is to complete the V2 scope with production readiness while keeping non-V2 enhancements out of active delivery.
|
||||
Use a fresh database. There will be no migrations, data conversion, legacy compatibility shims, or parallel V1/V2 code paths.
|
||||
|
||||
---
|
||||
## Current Project Impact
|
||||
|
||||
## V2 Completion Definition
|
||||
- `src/transcription/db/models.py` still defines the V1 `Document`, `Source`, `Job`, and `Revision` tables.
|
||||
- The V2 target adds `Person`, `DocumentPerson`, and `JobSource`, moves revisions onto `Source`, and removes the direct `Source.job_id` relationship.
|
||||
- The engine, session factory, transaction handling, and PostgreSQL async support already exist and do not need to be rewritten.
|
||||
- Async CRUD currently lives in `DocumentService`, `JobService`, `TranscriptionService`, and the upload record helper. Their queries and eager-loading options depend on V1 relationships.
|
||||
- Existing tests cover only part of the schema and CRUD surface.
|
||||
|
||||
V2 is complete when all of the following are true:
|
||||
## Implementation
|
||||
|
||||
1. **Functional complete**
|
||||
* Multi-image and whole-folder uploads assign sequential page numbers to `Source` records under a single `Document`.
|
||||
* Batch jobs process pages concurrently using an `asyncio` worker pool with semaphore rate limiting.
|
||||
* Partial job failures resolve cleanly to `partial_success`, allowing single-page retries without re-running successful pages.
|
||||
* Multi-author and multi-recipient tagging is supported on `Document`.
|
||||
### 1. Update the schema
|
||||
|
||||
- Replace the models in `src/transcription/db/models.py` with the approved V2 tables, enums, relationships, foreign keys, constraints, and indexes.
|
||||
- Remove `Revision`, `Source.job_id`, and the transcription fields that no longer belong on `Job`.
|
||||
- Keep `create_all()` as the schema bootstrap for a fresh database.
|
||||
- Delete `_ensure_sqlite_compat_columns()` and all schema patching from `src/transcription/db/operations.py`.
|
||||
- Keep the Python models, `docs/schema_v2.md`, and `docs/ddl_v2.sql` consistent.
|
||||
|
||||
2. **Data-model complete**
|
||||
* SQLite is fully replaced with PostgreSQL (using `asyncpg` or `psycopg3`).
|
||||
* Pydantic V2 models validate all API payloads, database row mappings, and `JSONB` structures.
|
||||
### 2. Align the async CRUD methods
|
||||
|
||||
- Keep the existing `ServiceBase` session and transaction pattern.
|
||||
- Update document CRUD to load and manage its ordered `Source` rows and `DocumentPerson` links.
|
||||
- Update job CRUD and queue queries to use `JobSource` instead of `Source.job_id`.
|
||||
- Add the missing async CRUD operations for `Person`, `Source`, `DocumentPerson`, and `JobSource` using the existing service style. Do not add another repository abstraction.
|
||||
- Replace revision CRUD with direct updates to `Source.revised_text` and `Source.date_revised`.
|
||||
- Remove the temporary transcript compatibility aliases instead of redirecting them.
|
||||
- Update only direct database call sites that construct or query these records; UI and worker feature changes are not part of this work.
|
||||
|
||||
3. **Operational complete**
|
||||
* Concurrency controls, worker pool metrics, and database connections operate safely under batch load.
|
||||
### 3. Verify the schema and CRUD
|
||||
|
||||
- Update the schema bootstrap test to expect `person`, `document`, `document_person`, `source`, `job`, and `job_source`, with no `revision` table.
|
||||
- Add async create, read, update, delete, list, and filtered-query tests for each entity that exposes those operations.
|
||||
- Test relationship loading, page ordering, uniqueness constraints, delete behavior, status values, and `JobSource` JSON fields.
|
||||
- Test both service-owned sessions and caller-provided sessions so flush/commit behavior remains correct.
|
||||
- Run the focused database and service tests, then the full suite with `uv run pytest`.
|
||||
|
||||
4. **Documentation complete**
|
||||
* `schema_v2.md`, `DDL_v2.sql`, Pydantic model contracts are updated and consistent.
|
||||
## Done When
|
||||
|
||||
- A fresh database is created directly from the V2 SQLModel metadata.
|
||||
- All async CRUD methods pass against the V2 relationships and fields.
|
||||
- No code references `Revision`, `Source.job_id`, removed `Job` transcription fields, or compatibility aliases.
|
||||
- The focused tests and full test suite pass.
|
||||
|
||||
## Out of Scope
|
||||
|
||||
---
|
||||
|
||||
## Phase 1 — Data Contract Stabilization & Pydantic Baseline
|
||||
|
||||
**Goal:** Lock the PostgreSQL schema, DDL, and Pydantic V2 models before refactoring service logic.
|
||||
|
||||
### Tasks
|
||||
|
||||
1. Finalize DDL for PostgreSQL native types (`UUID`, `TIMESTAMPTZ`, `JSONB`) and junction tables (`document_person`, `job_source`).
|
||||
2. Build core Pydantic V2 schemas (`Person`, `Document`, `Source`, `Job`, `JobSource`, `PageAIMetadata`).
|
||||
3. Confirm and document data invariants:
|
||||
* `source.raw_transcription` and `job_source.raw_transcription` are immutable machine outputs.
|
||||
* `source.revised_text` holds user edits. UI renders `COALESCE(revised_text, raw_transcription)`.
|
||||
* Page sequence is strictly ordered by `source.page_number ASC`.
|
||||
|
||||
|
||||
4. Freeze V2 job status values (`queued`, `processing`, `completed`, `partial_success`, `failed`) and page execution status values (`pending`, `transcribed`, `failed`).
|
||||
|
||||
### Deliverables
|
||||
|
||||
* Canonical `docs/schema_v2.md` and `docs/DDL_v2.sql`.
|
||||
* Centralized Pydantic validation suite in `models/schemas_v2.py`.
|
||||
|
||||
### Exit Criteria
|
||||
|
||||
* All database tables, relationships, and JSONB structures have corresponding Pydantic V2 models passing unit validation tests.
|
||||
|
||||
---
|
||||
|
||||
## Phase 2 — Persistence Layer Transition (SQLite to PostgreSQL)
|
||||
|
||||
**Goal:** Replace the SQLite storage layer with an asynchronous PostgreSQL driver (`asyncpg` or `psycopg3`).
|
||||
|
||||
### Tasks
|
||||
|
||||
1. Configure PostgreSQL database connection pooling and environment configuration.
|
||||
2. Refactor `services/store.py` / repository layers to execute parameterized async SQL queries (`$1`, `$2`).
|
||||
3. Implement JSONB serialization and deserialization helpers using Pydantic's `.model_dump_json()` and `.model_validate()`.
|
||||
4. Implement database bootstrap routines for PostgreSQL table creation and index initialization.
|
||||
|
||||
### Deliverables
|
||||
|
||||
* PostgreSQL-native database connection and query service modules.
|
||||
* Integration test suite confirming connection pooling and JSONB CRUD operations.
|
||||
|
||||
### Exit Criteria
|
||||
|
||||
* All database reads/writes run asynchronously against PostgreSQL with zero remaining SQLite driver dependencies.
|
||||
|
||||
---
|
||||
|
||||
## Phase 3 — Service Layer & `asyncio` Engine Refactor
|
||||
|
||||
**Goal:** Implement batch orchestration and parallel single-image API execution.
|
||||
|
||||
### Tasks
|
||||
|
||||
1. Refactor upload service to process folder/multi-image input:
|
||||
* Group files into a single `Document`.
|
||||
* Create ordered `Source` rows (`page_number = 1..N`).
|
||||
|
||||
|
||||
2. Refactor `services/workflows.py` with `asyncio` worker pools:
|
||||
* Use `asyncio.Semaphore` to enforce API provider rate limits.
|
||||
* Issue parallel single-image requests to Vision APIs (OpenAI/Claude).
|
||||
* Parse API responses directly into Pydantic models (`PageAIMetadata`).
|
||||
|
||||
|
||||
3. Update execution tracking:
|
||||
* Create a `JobSource` row per page call to record `raw_transcription`, `ai_metadata`, and `raw_api_response`.
|
||||
* Update active `source.raw_transcription` upon task completion.
|
||||
* Calculate aggregate batch status (`completed`, `partial_success`, `failed`) on the parent `Job`.
|
||||
|
||||
|
||||
4. Refactor `services/person.py` and `services/documents.py` to handle multi-person roles via `document_person`.
|
||||
|
||||
### Deliverables
|
||||
|
||||
* Asynchronous batch execution engine in `services/workflows.py`.
|
||||
* Service routines for multi-person tagging and page-level retries.
|
||||
|
||||
### Exit Criteria
|
||||
|
||||
* Executing a folder upload of 10+ images processes concurrently, populates page-level `JobSource` entries, and handles partial worker errors without crashing the batch.
|
||||
|
||||
---
|
||||
|
||||
## Phase 4 — UI & API Contract Alignment
|
||||
|
||||
**Goal:** Update API endpoints and frontend/UI views to render multi-page documents and person roles.
|
||||
|
||||
### Tasks
|
||||
|
||||
1. Update document and job API endpoints to accept batch file arrays and multi-person ID payloads.
|
||||
2. Update UI document views:
|
||||
* Render multi-page document transcriptions sequentially by `page_number`.
|
||||
* Display author and recipient chips/cards linked from `document_person`.
|
||||
|
||||
|
||||
3. Update job detail UI to show page-level execution statuses (`transcribed` vs. `failed`) and provide a "Retry Failed Pages" action for `partial_success` jobs.
|
||||
4. Align inline page editing controls to update `source.revised_text` and `source.date_revised`.
|
||||
|
||||
### Deliverables
|
||||
|
||||
* Refactored API routes and UI components supporting multi-page rendering and person management.
|
||||
|
||||
### Exit Criteria
|
||||
|
||||
* UI successfully displays multi-page document text, allows per-page human revisions, and shows author/recipient metadata.
|
||||
|
||||
---
|
||||
|
||||
## Phase 5 — Test Suite Realignment & Concurrency Testing
|
||||
|
||||
**Goal:** Ensure end-to-end system stability under concurrent async execution and load.
|
||||
|
||||
### Tasks
|
||||
|
||||
1. Write unit tests for Pydantic models, custom validators, and JSONB conversions.
|
||||
2. Write integration tests for async database operations:
|
||||
* CRUD for `Document`, `Person`, `DocumentPerson`, `Source`, `Job`, and `JobSource`.
|
||||
|
||||
|
||||
3. Write mock-backed async workflow tests:
|
||||
* Verify `asyncio.Semaphore` bounds concurrent tasks properly.
|
||||
* Validate state transition logic for `completed`, `partial_success`, and `failed` jobs.
|
||||
* Confirm retry routines process only targeted `JobSource` records marked as `failed`.
|
||||
|
||||
|
||||
4. Re-enable CI quality gates (linting, type checking with Pyright/mypy, pytest).
|
||||
|
||||
### Deliverables
|
||||
|
||||
* Passing asynchronous test suite covering core workflows, edge cases, and failure recoveries.
|
||||
|
||||
### Exit Criteria
|
||||
|
||||
* CI pipeline is green with comprehensive coverage across database operations, Pydantic models, and worker queues.
|
||||
|
||||
---
|
||||
|
||||
## Phase 6 — Reliability, Operations, and Release Readiness
|
||||
|
||||
**Goal:** Prepare V2 for production deployment and operator management.
|
||||
|
||||
### Tasks
|
||||
|
||||
1. Verify structured logging includes `job_id`, `document_id`, `source_id`, and `person_id`.
|
||||
2. Tune PostgreSQL connection pool limits and `asyncio` concurrency thresholds for production infrastructure.
|
||||
3. Update operational documentation:
|
||||
* Review and update `docs/schema_v2.md` as needed.
|
||||
* Create `docs/runbook_v2.md` detailing PostgreSQL maintenance, JSONB index management, and worker queue monitoring.
|
||||
* Create `docs/release_checklist_v2.md` for launch sign-off.
|
||||
|
||||
|
||||
|
||||
### Deliverables
|
||||
|
||||
* Updated project documentation and operational runbooks.
|
||||
* V2 release sign-off checklist.
|
||||
|
||||
### Exit Criteria
|
||||
|
||||
* All documentation reflects V2 architecture; launch checklist is fully verified.
|
||||
|
||||
---
|
||||
|
||||
## Requirement Traceability Focus
|
||||
|
||||
Maintain evidence against these V2 requirement groups:
|
||||
|
||||
* **Batch & Multi-Image Pipeline:** Folder ingestion, page ordering, async worker execution.
|
||||
* **Database & Persistence:** PostgreSQL, native UUIDs, JSONB execution storage, `asyncpg` pooling.
|
||||
* **Validation & Schemas:** Pydantic V2 models for DB rows, API requests, and AI vision responses.
|
||||
* **Attribution & Metadata:** Multi-author and multi-recipient tagging, biographical entity management.
|
||||
* **Error Recovery:** Partial success states, page-level status flags, isolated retry execution.
|
||||
|
||||
---
|
||||
|
||||
## Scope Discipline Rule (V2 Focus)
|
||||
|
||||
* Only tasks required for V2 scope (PostgreSQL, Pydantic V2, folder/async processing, multi-person roles) enter this plan.
|
||||
* V3 candidate features (such as side-by-side multi-provider model output comparison) remain strictly in the future backlog.
|
||||
* Any schema adjustments during implementation require immediate updates to `DDL_v2.sql`, Pydantic models, and `schema_v2.md`.
|
||||
|
||||
---
|
||||
|
||||
## Technology References
|
||||
|
||||
- [FastAPI documentation](https://fastapi.tiangolo.com/)
|
||||
- [NiceGUI documentation](https://nicegui.io/documentation)
|
||||
- [PostgreSQL documentation](https://www.postgresql.org/docs/)
|
||||
- [Python asyncio](https://docs.python.org/3/library/asyncio.html#module-asyncio)
|
||||
- [Pydantic Validation](https://pydantic.dev/docs/validation/latest/get-started/)
|
||||
- [Pydantic AI](https://pydantic.dev/docs/ai/overview/)
|
||||
|
||||
## Related Local References
|
||||
|
||||
- [System Overview](index_v2.md)
|
||||
- [System Design Intent](intent.md)
|
||||
- [Transcription Methodology](transcription_methodology.md)
|
||||
- [System Architecture](architecture_v2.md)
|
||||
- [System Requirements](requirements_v2.md)
|
||||
- [Data model](schema_v2.md)
|
||||
- [Error Handling Policy](error_handling_v2.md)
|
||||
- Implementation Plan (this document)
|
||||
|
||||
|
||||
|
||||
- Database migrations or preservation of V1 data
|
||||
- Legacy compatibility code
|
||||
- Database engine or session-layer rewrites
|
||||
- UI redesign, batch orchestration, worker concurrency, deployment, and operational runbooks
|
||||
|
||||
Reference in New Issue
Block a user