generated from john/python-template
151 lines
5.2 KiB
Markdown
151 lines
5.2 KiB
Markdown
# Version 2 Plan
|
|
|
|
Desired Enhancements:
|
|
1. Data store
|
|
* Upgrade db to PostgresSQL
|
|
* Begin capturing JSONB data (which will allow future migration to MongoDB if desired)
|
|
* Relocate db and uploaded images to a location outside of the project folder that can be backed up. (This needs to be done for all projects.) (c:/github/data/transcription?)
|
|
2. Add ability to upload multiple images (or a folder of images)
|
|
* How many is too many?
|
|
* If there is a practical max image count, can I break a block of images up into smaller batches automatically?
|
|
3. UI
|
|
* Introduce the concept of "documents" to the UI.
|
|
* Before an image can be uploaded a "document" needs to be created/defined.
|
|
* As part of the upload process, document images need to be associated with a document.
|
|
* Multiple image upload
|
|
* Refine the job detail/log screen
|
|
* Is document id + original filename the best name for uploaded images?
|
|
* How to present multiple images within one job?
|
|
* Add document name, original filename to job detail.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
## Purpose
|
|
|
|
Version 2 extends the V1 baseline by introducing a production-oriented persistence architecture while preserving current user workflows.
|
|
|
|
Primary target changes:
|
|
|
|
- Migrate relational persistence from SQLite to PostgreSQL
|
|
- Introduce optional MongoDB for document-oriented adjunct data (non-canonical)
|
|
|
|
V1 behavior remains the functional baseline unless explicitly superseded by approved V2 requirements.
|
|
|
|
---
|
|
|
|
## V2 Goals
|
|
|
|
1. **Relational migration complete**
|
|
- PostgreSQL becomes the default system of record for `Document`, `Source`, `Job`, and `Revision`.
|
|
1. **Operational maturity**
|
|
- Repeatable migrations, rollback paths, and environment-specific deployment procedures are documented and tested.
|
|
1. **Optional document store integration**
|
|
- MongoDB is introduced only for clearly scoped use cases that do not replace canonical relational ownership.
|
|
1. **No regression of V1 workflows**
|
|
- Upload, queue/worker processing, status inspection, original transcription, and optional single revision remain stable.
|
|
|
|
---
|
|
|
|
## Non-Goals (V2)
|
|
|
|
- Replacing SQLModel domain ownership with MongoDB
|
|
- Introducing breaking UI behavior for existing V1 flows
|
|
- Expanding revision cardinality beyond current `0..1` without explicit requirements update
|
|
|
|
---
|
|
|
|
## Proposed Scope
|
|
|
|
### A) PostgreSQL migration (required)
|
|
|
|
- Add PostgreSQL runtime profile for local/dev/prod
|
|
- Introduce migration toolchain and migration history
|
|
- Convert bootstrap strategy from compatibility patching to explicit migrations
|
|
- Validate model constraints and indexes against PostgreSQL
|
|
- Add operational checks (connectivity, pool, transaction behavior)
|
|
|
|
### B) MongoDB integration (optional, gated)
|
|
|
|
- Define approved use cases (for example: denormalized read models, audit/event projections, or search-oriented materializations)
|
|
- Keep canonical write path in relational store
|
|
- Add feature flag/config gate to enable or disable Mongo features
|
|
- Document consistency model and failure behavior
|
|
|
|
---
|
|
|
|
## Milestones
|
|
|
|
## M1 — Requirements and architecture baseline
|
|
|
|
- Create V2 requirements delta from V1 baseline
|
|
- Define relational/document ownership boundaries
|
|
- Approve migration strategy and cutover approach
|
|
|
|
**Exit criteria:** signed architecture decision and updated traceability map.
|
|
|
|
## M2 — PostgreSQL foundation
|
|
|
|
- Add PostgreSQL environment wiring and secrets strategy
|
|
- Add migration framework and initial schema migration
|
|
- Add CI path using PostgreSQL service container
|
|
|
|
**Exit criteria:** test suite green on PostgreSQL in CI.
|
|
|
|
## M3 — Data migration and cutover rehearsal
|
|
|
|
- Build SQLite -> PostgreSQL migration utility/playbook
|
|
- Rehearse migration on representative datasets
|
|
- Validate rollback/recovery procedures
|
|
|
|
**Exit criteria:** successful dry-run migration with measured rollback test.
|
|
|
|
## M4 — MongoDB optional integration
|
|
|
|
- Implement scoped Mongo use case(s)
|
|
- Add fallback behavior when Mongo unavailable
|
|
- Add tests and operational runbook updates
|
|
|
|
**Exit criteria:** feature-gated Mongo behavior validated with no V1 flow regressions.
|
|
|
|
## M5 — Release readiness
|
|
|
|
- Final regression suite (functional + reliability)
|
|
- Performance and failure-mode checks
|
|
- Production release checklist and sign-off
|
|
|
|
**Exit criteria:** V2 release approval.
|
|
|
|
---
|
|
|
|
## Risks and Mitigations
|
|
|
|
- **Schema drift risk** -> enforce migration-first policy and CI migration checks.
|
|
- **Dual-store consistency risk** -> keep relational source of truth and explicit projection contracts.
|
|
- **Operational complexity** -> staged rollout, runbooks, and feature flags.
|
|
- **Regression risk in worker lifecycle** -> keep dedicated reliability tests around terminal-state guarantees.
|
|
|
|
---
|
|
|
|
## Traceability and Evidence
|
|
|
|
Maintain a V2 table with:
|
|
|
|
- requirement/change ID
|
|
- status (`not started` / `in progress` / `done`)
|
|
- implementation PR
|
|
- validation evidence (test names, migration rehearsal logs, runbook references)
|
|
|
|
---
|
|
|
|
## Suggested first implementation tasks
|
|
|
|
1. Create `docs/ver2/adr/` and draft ADR for persistence ownership boundaries.
|
|
2. Add PostgreSQL compose profile and env contract.
|
|
3. Introduce migration tooling and generate initial migration from current schema.
|
|
4. Add CI job for PostgreSQL-backed `pytest -m "not external"`.
|