transcription/docs/ver2/ver2.md

Version 2 Plan

Desired Enhancements:

1. Data store
   • Upgrade db to PostgresSQL
   • Begin capturing JSONB data (which will allow future migration to MongoDB if desired)
   • Relocate db and uploaded images to a location outside of the project folder that can be backed up. (This needs to be done for all projects.) (c:/github/data/transcription?)
2. Add ability to upload multiple images (or a folder of images)
   • How many is too many?
   • If there is a practical max image count, can I break a block of images up into smaller batches automatically?
3. UI
   • Introduce the concept of "documents" to the UI.
     • Before an image can be uploaded a "document" needs to be created/defined.
     • As part of the upload process, document images need to be associated with a document.
   • Multiple image upload
   • Refine the job detail/log screen
     • Is document id + original filename the best name for uploaded images?
     • How to present multiple images within one job?
     • Add document name, original filename to job detail.

Purpose

Version 2 extends the V1 baseline by introducing a production-oriented persistence architecture while preserving current user workflows.

Primary target changes:

• Migrate relational persistence from SQLite to PostgreSQL
• Introduce optional MongoDB for document-oriented adjunct data (non-canonical)

V1 behavior remains the functional baseline unless explicitly superseded by approved V2 requirements.

---

V2 Goals

1. Relational migration complete
   • PostgreSQL becomes the default system of record for Document, Source, Job, and Revision.
2. Operational maturity
   • Repeatable migrations, rollback paths, and environment-specific deployment procedures are documented and tested.
3. Optional document store integration
   • MongoDB is introduced only for clearly scoped use cases that do not replace canonical relational ownership.
4. No regression of V1 workflows
   • Upload, queue/worker processing, status inspection, original transcription, and optional single revision remain stable.

---

Non-Goals (V2)

• Replacing SQLModel domain ownership with MongoDB
• Introducing breaking UI behavior for existing V1 flows
• Expanding revision cardinality beyond current 0..1 without explicit requirements update

---

Proposed Scope

A) PostgreSQL migration (required)

• Add PostgreSQL runtime profile for local/dev/prod
• Introduce migration toolchain and migration history
• Convert bootstrap strategy from compatibility patching to explicit migrations
• Validate model constraints and indexes against PostgreSQL
• Add operational checks (connectivity, pool, transaction behavior)

B) MongoDB integration (optional, gated)

• Define approved use cases (for example: denormalized read models, audit/event projections, or search-oriented materializations)
• Keep canonical write path in relational store
• Add feature flag/config gate to enable or disable Mongo features
• Document consistency model and failure behavior

---

Milestones

M1 — Requirements and architecture baseline

• Create V2 requirements delta from V1 baseline
• Define relational/document ownership boundaries
• Approve migration strategy and cutover approach

Exit criteria: signed architecture decision and updated traceability map.

M2 — PostgreSQL foundation

• Add PostgreSQL environment wiring and secrets strategy
• Add migration framework and initial schema migration
• Add CI path using PostgreSQL service container

Exit criteria: test suite green on PostgreSQL in CI.

M3 — Data migration and cutover rehearsal

• Build SQLite -> PostgreSQL migration utility/playbook
• Rehearse migration on representative datasets
• Validate rollback/recovery procedures

Exit criteria: successful dry-run migration with measured rollback test.

M4 — MongoDB optional integration

• Implement scoped Mongo use case(s)
• Add fallback behavior when Mongo unavailable
• Add tests and operational runbook updates

Exit criteria: feature-gated Mongo behavior validated with no V1 flow regressions.

M5 — Release readiness

• Final regression suite (functional + reliability)
• Performance and failure-mode checks
• Production release checklist and sign-off

Exit criteria: V2 release approval.

---

Risks and Mitigations

• Schema drift risk -> enforce migration-first policy and CI migration checks.
• Dual-store consistency risk -> keep relational source of truth and explicit projection contracts.
• Operational complexity -> staged rollout, runbooks, and feature flags.
• Regression risk in worker lifecycle -> keep dedicated reliability tests around terminal-state guarantees.

---

Traceability and Evidence

Maintain a V2 table with:

• requirement/change ID
• status (not started / in progress / done)
• implementation PR
• validation evidence (test names, migration rehearsal logs, runbook references)

---

Suggested first implementation tasks

1. Create docs/ver2/adr/ and draft ADR for persistence ownership boundaries.
2. Add PostgreSQL compose profile and env contract.
3. Introduce migration tooling and generate initial migration from current schema.
4. Add CI job for PostgreSQL-backed pytest -m "not external".