diff --git a/docs/architecture.md b/docs/architecture.md index dd632ec..bd68d96 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1,299 +1,130 @@ - # Architecture +# Architecture (V1 Baseline) -This document describes the production architecture of the personal historical-document transcription system. The system is intentionally optimized for single-user operation, low operational overhead, and clean internal boundaries that support future growth without rewrites. +This document describes the current architecture of the personal historical-document transcription system and serves as the V1 technical baseline. ## Architecture Objectives -The production architecture is designed to: +- preserve source material as transcribed text +- keep operational complexity low for personal-scale deployment +- support asynchronous processing without external queue infrastructure +- maintain clear module boundaries for incremental extension -- preserve verbatim family-history source material as searchable text -- keep operational complexity low for a personal deployment -- support asynchronous transcription without requiring distributed infrastructure -- maintain clear module boundaries so extensions can be added incrementally +## Runtime Topology -## Production Scope And Scale +V1 runtime is a modular monolith: -The deployed system targets personal use and a corpus of several thousand documents processed over time. The architecture favors simple, composable building blocks over distributed orchestration. - -Current scope includes: - -- content source upload and metadata capture -- asynchronous transcription jobs -- prompt-library driven transcription behavior, with one Markdown file per prompt -- original transcription review and optional revision review -- full-text search over accepted transcripts -- export of transcript data - -## Deployment Topology - -The production deployment uses [Docker Compose](https://docs.docker.com/compose/) and treats containerized databases as extremely lightweight operational dependencies. - -Running [PostgreSQL](https://www.postgresql.org/docs/) in its own container is considered simple by default for this system. - -Running [MongoDB](https://www.mongodb.com/docs/) in its own container is also considered simple when document-centric storage is enabled. - -Container count is not a hard architectural limit; a three-container deployment (app, PostgreSQL, MongoDB) is an acceptable baseline. - -### Baseline Topology (Two Containers) - -- one application container -- one PostgreSQL container -- embedded background worker execution inside the app process - -### Expanded Topology (Three Containers) - -- application container -- PostgreSQL container -- MongoDB container - -No additional queue, scheduler, or search-engine containers are required in the baseline production setup. - -## Runtime Architecture +- one FastAPI + NiceGUI application process +- one in-process async worker loop +- relational persistence via SQLModel (SQLite baseline) ```mermaid flowchart LR - User[Browser User] --> App[FastAPI + NiceGUI Service] - App --> Worker[In-process Background Worker] - App --> PG[(PostgreSQL)] - App --> MG[(MongoDB Document Store)] - Worker --> AI[Transcription Provider] - Worker --> PG - Worker --> MG + U[Browser User] --> A[FastAPI + NiceGUI App] + A --> W[In-process Worker] + A --> DB[(SQLite via SQLModel)] + W --> P[OpenRouter Provider] + W --> DB ``` -## Runtime Ownership And Startup Policy +## Lifecycle Ownership -The current implementation now uses explicit lifespan-owned runtime resources. +Application lifespan owns runtime setup/teardown: -- application lifespan initializes and disposes database runtime resources -- worker lifecycle is owned by application lifespan startup/shutdown -- worker receives lifespan-owned database engine dependency explicitly -- schema bootstrap policy is environment-aware and explicit: - - development/test default to bootstrap enabled - - production defaults to bootstrap disabled - - explicit override is available via configuration - -This aligns implementation toward REQ-7 and REQ-10 while preserving personal-scale operational simplicity. +- configure logging +- initialize and dispose DB runtime resources +- optional schema bootstrap by environment policy +- recover stale processing jobs +- start/stop worker consumer lifespan ## Layered Module Structure ### Interface Layer -Responsibility: +- `src/transcription/ui/**` (NiceGUI pages/components) +- `src/transcription/api/**` (FastAPI routes and error handlers) -- HTTP API and UI routes -- request/response validation -- status and result presentation +### Application/Workflow Layer -Out of scope: +- `src/transcription/services/workflows.py` +- `src/transcription/worker.py` -- business-rule enforcement -- data-access implementation +Responsibilities: -### Application Layer +- orchestration and status transitions +- retry/timeout behavior +- provider call coordination -Responsibility: +### Service Layer -- upload and job orchestration -- state transitions and retry policy -- coordination across domain and infrastructure ports +- `src/transcription/services/*.py` -Out of scope: +Responsibilities: -- provider-specific protocol details -- ORM or storage-specific logic - -### Domain Layer - -Responsibility: - -- verbatim transcription policy -- revision and provenance invariants -- confidence and annotation semantics - -Out of scope: - -- web framework concerns -- database and network I/O +- CRUD and transactional boundaries +- domain-aligned persistence operations ### Infrastructure Layer -Responsibility: - -- persistence adapters (PostgreSQL and MongoDB) -- transcription-provider adapter - -Out of scope: - -- business policy decisions +- `src/transcription/db/**` (runtime/session/bootstrap) +- `src/transcription/providers/**` (OpenRouter adapter) ## Processing Workflow -Production transcription flow: +1. User uploads a source file from the UI. +2. App persists `Document`, `Job(queued)`, and `Source`. +3. Worker claims next queued job and marks `processing`. +4. Worker calls provider with prompt + source bytes. +5. On success, app writes immutable `Job.text` and marks `transcribed`. +6. On failure, app writes `Job.error_detail` and marks `failed`. +7. UI exposes job detail, original transcription, and optional revision. -1. A user uploads one or more content sources through the UI or API. -2. The application validates payloads and creates document, source, and job records. -3. The in-process worker de-queues the job and calls the transcription provider. -4. The application persists original transcription output on the job, plus confidence metadata and provenance events. -5. Job status transitions from queued to processing to transcribed or failed. -6. The UI and API expose status, optional revision to original transcription, and searchable transcription text. +## Domain Ownership Invariants -## Data Model Ownership +- `Job.text` is immutable original provider output. +- `Revision` is optional, user-authored, and linked to `Source`. +- `Revision` does not overwrite original job transcription. +- Status lifecycle is fixed to: `queued -> processing -> transcribed|failed`. -System-of-record entities: +## Data Model Summary -- documents and content sources -- transcription jobs, original transcription, and status events -- transcript revisions -- provenance metadata +- `Document` has many `Source` and many `Job`. +- `Source` belongs to one `Document` and one `Job`. +- `Source` has optional `Revision` (`0..1`) enforced by unique `revision.source_id`. -### Original Transcription And Revision Ownership +## Simplicity Guardrails (V1) -- each processing job stores the original immutable provider output (`text`) -- provider metadata (`provider`, `model`, `prompt_name`) and failure detail (`error_detail`) are job-owned processing artifacts -- revisions are optional user-authored edits linked to a content source -- a revision can be created from original `job.text` -- many jobs will have zero revisions; revisions are additive and never overwrite original provider output -- a document groups one or more content sources (images, PDFs, and future source types) - -Storage strategy: - -- PostgreSQL for relational system-of-record entities -- MongoDB for document-oriented payloads and large transcription artifacts -- versioned prompt artifacts stored as individual Markdown files for human editing and refinement -- in-memory execution state treated as ephemeral - -## Transcription Prompt Asset Policy - -The production system treats transcription prompts as maintainable content assets. - -- each transcription prompt is stored in its own Markdown file -- prompt files are designed for direct human editing and iterative refinement -- prompt updates are independent and do not require bundling unrelated prompt changes -- prompt file identity and revision history are tracked through normal repository version control - -## Simplicity Guardrails - -The production system enforces these constraints to prevent accidental over-engineering: - -- PostgreSQL in a container is treated as a lightweight default dependency -- MongoDB in a container is treated as a lightweight optional dependency -- three containers (app, PostgreSQL, MongoDB) is an acceptable simple deployment -- no dedicated queue or search cluster is introduced without measured need -- external infrastructure is added only behind existing ports/adapters +- no external queue/broker required +- no search engine required +- no distributed worker fleet required +- keep provider integration behind adapter boundary ## Extension Path -The architecture supports additive growth without changing domain contracts. +### V1 (current) -### Stage 1: Foundation (Current) +- SQLite baseline +- OpenRouter provider +- in-process worker +- optional single revision workflow -- upload, transcription, review, search, export -- in-process worker execution -- single provider adapter -- app plus PostgreSQL deployment +### V2 (planned) -### Stage 2: Throughput Hardening +- PostgreSQL as relational baseline +- optional MongoDB adjunct store for scoped use cases +- migration-first schema evolution -- optional MongoDB document-store enablement -- optional external worker/queue process -- stronger retry and dead-letter handling - -### Stage 3: Intelligence Features - -- entity extraction and cross-document linking -- timeline and narrative assembly -- optional multi-provider routing - -Each stage preserves existing module boundaries and keeps migration risk low. +See [ver2/ver2.md](ver2/ver2.md) for roadmap details. ## Test Strategy -The test strategy is aligned to personal-scale operation with fast, deterministic feedback. +- unit tests for model/service behaviors +- integration tests for upload/workflow reliability +- UI integration tests for page/render contracts +- external provider tests opt-in via marker/config -### Unit Tests +## Related References -- domain transcription rules and annotation behavior -- revision-history invariants -- job state-transition logic - -### Integration Tests - -- repository behavior and transaction boundaries -- persistence-adapter and provider adapter contract mapping -- upload-to-persistence roundtrip - -### End-to-End Tests - -- happy path: upload, transcribe, review, search, export -- failure path: provider error, retry, surfaced failed status - -### CI Execution Model - -- fast suite on each push -- optional slower provider-sandbox checks on scheduled runs - -## Risks And Controls - -### Runtime Responsiveness - -Risk: - -- long jobs can reduce responsiveness in a single-process deployment - -Control: - -- bounded concurrency and visible job status in the UI - -### Database Concurrency Limits - -Risk: - -- contention can appear under sustained concurrent writes in personal-scale infrastructure - -Control: - -- tuned connection pooling and phased use of MongoDB for document-heavy workloads - -### Provider Output Variance - -Risk: - -- transcription quality varies by content source type, handwriting legibility, and source quality - -Control: - -- first-class human review and immutable revision history - -## Technology References - -- [FastAPI documentation](https://fastapi.tiangolo.com/) -- [NiceGUI documentation](https://nicegui.io/documentation) -- [Docker Compose documentation](https://docs.docker.com/compose/) -- [PostgreSQL documentation](https://www.postgresql.org/docs/) -- [MongoDB documentation](https://www.mongodb.com/docs/) - -## Related Local References - -- [System overview](index.md) - -## Glossary - -- Adapter: A component that translates between internal interfaces and external systems such as databases or AI services. -- Background job: Work executed outside the request/response path so the UI remains responsive. -- Boundary: A strict separation between modules with different responsibilities. -- CI (Continuous Integration): Automated test execution for code changes. -- Contract test: A test that verifies an adapter follows expected input/output behavior at a boundary. -- Domain layer: The module that contains core business rules and invariants. -- End-to-end test: A test that validates a full user flow across the running system. -- Full-text search: Text indexing and querying optimized for natural-language search. -- In-process worker: A background executor that runs within the same application process. -- Integration test: A test that verifies interactions between real modules and infrastructure components. -- MongoDB: A document-oriented database used for flexible, high-variance data structures. -- Modular monolith: A single deployable application with strongly separated internal modules. -- Port/Interface: A stable contract used by application/domain code to call infrastructure implementations. -- Prompt artifact: A single Markdown file that defines one transcription prompt and can be revised independently. -- Provenance: Metadata that records where generated data came from and how it was produced. -- Revision history: Optional versioned record of user-authored transcription edits over time. -- System of record: The authoritative persistent store for canonical data. -- Vertical slice: A minimal end-to-end feature path spanning UI/API, application logic, and persistence. +- [index.md](index.md) +- [requirements.md](requirements.md) +- [schema.md](schema.md) +- [error_handling.md](error_handling.md) diff --git a/docs/archive/README.md b/docs/archive/README.md new file mode 100644 index 0000000..c29d558 --- /dev/null +++ b/docs/archive/README.md @@ -0,0 +1,23 @@ +# V2 Archive + +This folder preserves pre-V1-alignment versions of core documentation that included planned target-state architecture material. + +Archived snapshots: + +- `index.pre-v1-alignment.md` +- `requirements.pre-v1-alignment.md` +- `architecture.pre-v1-alignment.md` + +Purpose: + +- keep a durable reference for planned architecture language +- reduce risk of losing useful V2 direction while V1 docs stay implementation-aligned + +Notes: + +- These files are historical snapshots, not the active V1 source of truth. +- Active V1 docs remain at: + - `docs/index.md` + - `docs/requirements.md` + - `docs/architecture.md` +- V2 planning should continue in `docs/ver2/ver2.md` and related V2 artifacts. diff --git a/docs/archive/architecture.pre-v1-alignment.md b/docs/archive/architecture.pre-v1-alignment.md new file mode 100644 index 0000000..dd632ec --- /dev/null +++ b/docs/archive/architecture.pre-v1-alignment.md @@ -0,0 +1,299 @@ + # Architecture + +This document describes the production architecture of the personal historical-document transcription system. The system is intentionally optimized for single-user operation, low operational overhead, and clean internal boundaries that support future growth without rewrites. + +## Architecture Objectives + +The production architecture is designed to: + +- preserve verbatim family-history source material as searchable text +- keep operational complexity low for a personal deployment +- support asynchronous transcription without requiring distributed infrastructure +- maintain clear module boundaries so extensions can be added incrementally + +## Production Scope And Scale + +The deployed system targets personal use and a corpus of several thousand documents processed over time. The architecture favors simple, composable building blocks over distributed orchestration. + +Current scope includes: + +- content source upload and metadata capture +- asynchronous transcription jobs +- prompt-library driven transcription behavior, with one Markdown file per prompt +- original transcription review and optional revision review +- full-text search over accepted transcripts +- export of transcript data + +## Deployment Topology + +The production deployment uses [Docker Compose](https://docs.docker.com/compose/) and treats containerized databases as extremely lightweight operational dependencies. + +Running [PostgreSQL](https://www.postgresql.org/docs/) in its own container is considered simple by default for this system. + +Running [MongoDB](https://www.mongodb.com/docs/) in its own container is also considered simple when document-centric storage is enabled. + +Container count is not a hard architectural limit; a three-container deployment (app, PostgreSQL, MongoDB) is an acceptable baseline. + +### Baseline Topology (Two Containers) + +- one application container +- one PostgreSQL container +- embedded background worker execution inside the app process + +### Expanded Topology (Three Containers) + +- application container +- PostgreSQL container +- MongoDB container + +No additional queue, scheduler, or search-engine containers are required in the baseline production setup. + +## Runtime Architecture + +```mermaid +flowchart LR + User[Browser User] --> App[FastAPI + NiceGUI Service] + App --> Worker[In-process Background Worker] + App --> PG[(PostgreSQL)] + App --> MG[(MongoDB Document Store)] + Worker --> AI[Transcription Provider] + Worker --> PG + Worker --> MG +``` + +## Runtime Ownership And Startup Policy + +The current implementation now uses explicit lifespan-owned runtime resources. + +- application lifespan initializes and disposes database runtime resources +- worker lifecycle is owned by application lifespan startup/shutdown +- worker receives lifespan-owned database engine dependency explicitly +- schema bootstrap policy is environment-aware and explicit: + - development/test default to bootstrap enabled + - production defaults to bootstrap disabled + - explicit override is available via configuration + +This aligns implementation toward REQ-7 and REQ-10 while preserving personal-scale operational simplicity. + +## Layered Module Structure + +### Interface Layer + +Responsibility: + +- HTTP API and UI routes +- request/response validation +- status and result presentation + +Out of scope: + +- business-rule enforcement +- data-access implementation + +### Application Layer + +Responsibility: + +- upload and job orchestration +- state transitions and retry policy +- coordination across domain and infrastructure ports + +Out of scope: + +- provider-specific protocol details +- ORM or storage-specific logic + +### Domain Layer + +Responsibility: + +- verbatim transcription policy +- revision and provenance invariants +- confidence and annotation semantics + +Out of scope: + +- web framework concerns +- database and network I/O + +### Infrastructure Layer + +Responsibility: + +- persistence adapters (PostgreSQL and MongoDB) +- transcription-provider adapter + +Out of scope: + +- business policy decisions + +## Processing Workflow + +Production transcription flow: + +1. A user uploads one or more content sources through the UI or API. +2. The application validates payloads and creates document, source, and job records. +3. The in-process worker de-queues the job and calls the transcription provider. +4. The application persists original transcription output on the job, plus confidence metadata and provenance events. +5. Job status transitions from queued to processing to transcribed or failed. +6. The UI and API expose status, optional revision to original transcription, and searchable transcription text. + +## Data Model Ownership + +System-of-record entities: + +- documents and content sources +- transcription jobs, original transcription, and status events +- transcript revisions +- provenance metadata + +### Original Transcription And Revision Ownership + +- each processing job stores the original immutable provider output (`text`) +- provider metadata (`provider`, `model`, `prompt_name`) and failure detail (`error_detail`) are job-owned processing artifacts +- revisions are optional user-authored edits linked to a content source +- a revision can be created from original `job.text` +- many jobs will have zero revisions; revisions are additive and never overwrite original provider output +- a document groups one or more content sources (images, PDFs, and future source types) + +Storage strategy: + +- PostgreSQL for relational system-of-record entities +- MongoDB for document-oriented payloads and large transcription artifacts +- versioned prompt artifacts stored as individual Markdown files for human editing and refinement +- in-memory execution state treated as ephemeral + +## Transcription Prompt Asset Policy + +The production system treats transcription prompts as maintainable content assets. + +- each transcription prompt is stored in its own Markdown file +- prompt files are designed for direct human editing and iterative refinement +- prompt updates are independent and do not require bundling unrelated prompt changes +- prompt file identity and revision history are tracked through normal repository version control + +## Simplicity Guardrails + +The production system enforces these constraints to prevent accidental over-engineering: + +- PostgreSQL in a container is treated as a lightweight default dependency +- MongoDB in a container is treated as a lightweight optional dependency +- three containers (app, PostgreSQL, MongoDB) is an acceptable simple deployment +- no dedicated queue or search cluster is introduced without measured need +- external infrastructure is added only behind existing ports/adapters + +## Extension Path + +The architecture supports additive growth without changing domain contracts. + +### Stage 1: Foundation (Current) + +- upload, transcription, review, search, export +- in-process worker execution +- single provider adapter +- app plus PostgreSQL deployment + +### Stage 2: Throughput Hardening + +- optional MongoDB document-store enablement +- optional external worker/queue process +- stronger retry and dead-letter handling + +### Stage 3: Intelligence Features + +- entity extraction and cross-document linking +- timeline and narrative assembly +- optional multi-provider routing + +Each stage preserves existing module boundaries and keeps migration risk low. + +## Test Strategy + +The test strategy is aligned to personal-scale operation with fast, deterministic feedback. + +### Unit Tests + +- domain transcription rules and annotation behavior +- revision-history invariants +- job state-transition logic + +### Integration Tests + +- repository behavior and transaction boundaries +- persistence-adapter and provider adapter contract mapping +- upload-to-persistence roundtrip + +### End-to-End Tests + +- happy path: upload, transcribe, review, search, export +- failure path: provider error, retry, surfaced failed status + +### CI Execution Model + +- fast suite on each push +- optional slower provider-sandbox checks on scheduled runs + +## Risks And Controls + +### Runtime Responsiveness + +Risk: + +- long jobs can reduce responsiveness in a single-process deployment + +Control: + +- bounded concurrency and visible job status in the UI + +### Database Concurrency Limits + +Risk: + +- contention can appear under sustained concurrent writes in personal-scale infrastructure + +Control: + +- tuned connection pooling and phased use of MongoDB for document-heavy workloads + +### Provider Output Variance + +Risk: + +- transcription quality varies by content source type, handwriting legibility, and source quality + +Control: + +- first-class human review and immutable revision history + +## Technology References + +- [FastAPI documentation](https://fastapi.tiangolo.com/) +- [NiceGUI documentation](https://nicegui.io/documentation) +- [Docker Compose documentation](https://docs.docker.com/compose/) +- [PostgreSQL documentation](https://www.postgresql.org/docs/) +- [MongoDB documentation](https://www.mongodb.com/docs/) + +## Related Local References + +- [System overview](index.md) + +## Glossary + +- Adapter: A component that translates between internal interfaces and external systems such as databases or AI services. +- Background job: Work executed outside the request/response path so the UI remains responsive. +- Boundary: A strict separation between modules with different responsibilities. +- CI (Continuous Integration): Automated test execution for code changes. +- Contract test: A test that verifies an adapter follows expected input/output behavior at a boundary. +- Domain layer: The module that contains core business rules and invariants. +- End-to-end test: A test that validates a full user flow across the running system. +- Full-text search: Text indexing and querying optimized for natural-language search. +- In-process worker: A background executor that runs within the same application process. +- Integration test: A test that verifies interactions between real modules and infrastructure components. +- MongoDB: A document-oriented database used for flexible, high-variance data structures. +- Modular monolith: A single deployable application with strongly separated internal modules. +- Port/Interface: A stable contract used by application/domain code to call infrastructure implementations. +- Prompt artifact: A single Markdown file that defines one transcription prompt and can be revised independently. +- Provenance: Metadata that records where generated data came from and how it was produced. +- Revision history: Optional versioned record of user-authored transcription edits over time. +- System of record: The authoritative persistent store for canonical data. +- Vertical slice: A minimal end-to-end feature path spanning UI/API, application logic, and persistence. diff --git a/docs/archive/index.pre-v1-alignment.md b/docs/archive/index.pre-v1-alignment.md new file mode 100644 index 0000000..ae5fcd0 --- /dev/null +++ b/docs/archive/index.pre-v1-alignment.md @@ -0,0 +1,56 @@ +## Document Transcription System + +This project is a production application for transcribing and preserving historical family documents. It is intentionally designed for personal-scale use, with a simplicity-first architecture that is easy to operate and easy to extend. + +## Start Here + +Read [architecture.md](architecture.md) first. + +The architecture page is the primary technical reference and defines: + +- deployed topology and infrastructure limits +- module boundaries and dependency flow +- processing life cycle and data ownership +- test strategy, risk controls, and extension path + +## What The Application Does + +At a high level, users upload images or PDFs as content sources for handwritten, typed, or typeset documents, run asynchronous transcription jobs, review optional revisions, and search across accepted text. + +Core capabilities: + +- document grouping with one or more content sources and metadata capture +- asynchronous transcription with visible job status +- immutable original transcription persisted with each job (plus provider/model/prompt metadata) +- transcription prompt management with one Markdown file per prompt for human refinement over time +- optional revisions for user-authored edits of original immutable transcription text +- full-text search over accepted transcripts +- export of transcript data + +## Production Operating Model + +The system runs with minimal operational overhead: + +- PostgreSQL in a dedicated Docker container is considered extremely lightweight and simple for this system +- MongoDB in a dedicated Docker container is also considered extremely lightweight and simple for document-centric persistence +- a three-container deployment (app, PostgreSQL, MongoDB) is a simple and acceptable baseline +- no required queue or search-engine containers in the baseline setup + +This operating model keeps deployment and maintenance simple while preserving clean boundaries for future scale. + +## Documentation Map + +- Architecture and technical design: [architecture.md](architecture.md) +- Runtime and deployment requirements: [requirements.md](requirements.md) +- Error handling policy and operational guidance: [error_handling.md](error_handling.md) +- Domain context and transcription policy: [intent.md](intent.md) +- Transcription Methodology: [transcription_methodology.md](transcription_methodology.md) +- Data model: [schema.md](schema.md) + + + +## Glossary + +- Document-oriented persistence: Storing data as flexible records instead of fixed relational rows. +- Prompt artifact: A single Markdown file that defines one transcription prompt and is edited independently. +- System of record: The authoritative persistent store for canonical data. diff --git a/docs/archive/requirements.pre-v1-alignment.md b/docs/archive/requirements.pre-v1-alignment.md new file mode 100644 index 0000000..62dcf09 --- /dev/null +++ b/docs/archive/requirements.pre-v1-alignment.md @@ -0,0 +1,85 @@ +## Document Transcription System Requirements + +This page captures a SysML v1.6-style requirements baseline for the production system described in [index.md](index.md). The model is represented as concise tables and traceability lists that preserve SysML-style IDs and relationship semantics. + +## Scope + +- System of interest: the single Python application service (NiceGUI + FastAPI) with PostgreSQL as the relational system of record and optional MongoDB for document-oriented persistence. +- Operational context: local-first execution with Docker Compose and an intentionally lightweight production trajectory. +- Primary concern: end-to-end transcription job lifecycle from upload through completion or failure. + +## Requirements Model (Concise Text Form) + +### Requirements + +| ID | Category | Requirement | Risk | Verify Method | +| --- | --- | --- | --- | --- | +| REQ-0 | System | Provide end-to-end document transcription with persistent, inspectable lifecycle state. | medium | demonstration | +| REQ-1 | Functional | Allow users to upload one or more images or PDFs as sources from the web UI. | low | test | +| REQ-2 | Functional | Run each upload through asynchronous processing that returns an original transcription or explicit failure. | high | test | +| REQ-3 | Functional | Persist and expose job states: queued, processing, transcribed, failed. | high | inspection | +| REQ-4 | Functional | Persist transcription output, processing history, and failure details. | medium | test | +| REQ-5 | Interface | Expose API and UI views for status inspection and completed transcription reading. | medium | demonstration | +| REQ-6 | Performance | Trigger background processing on upload to preserve UI responsiveness. | medium | analysis | +| REQ-7 | Design Constraint | Keep lifespan-owned runtime resources: SQLAlchemy engine, async session factory, worker resources, provider clients. | medium | inspection | +| REQ-8 | Design Constraint | Initialize configuration and logging once at startup through centralized mechanisms. | low | inspection | +| REQ-9 | Design Constraint | Use Docker Compose baseline of app plus PostgreSQL; allow optional MongoDB container when enabled. | medium | demonstration | +| REQ-10 | Design Constraint | Keep schema bootstrap explicit and opt-in; normal startup does not mutate production schema. | high | inspection | +| REQ-11 | Design Constraint | Use service-backed persistence for core document and job data. | medium | inspection | +| REQ-12 | Design Constraint | Store transcription prompts as individual Markdown artifacts for iterative refinement. | medium | inspection | +| REQ-13 | Functional | Allow users to create one optional revision of transcription text derived from the original job transcription. | low | test | + +### Requirement Relationships + +- Contains: REQ-0 contains REQ-1 through REQ-13. +- Derives: REQ-2 -> REQ-3, REQ-3 -> REQ-4. +- Traces: REQ-5 -> REQ-3. +- Refines: REQ-6 -> REQ-2. + +### Architecture Elements + +| Element | Type | Doc Reference | +| --- | --- | --- | +| UI | NiceGUI pages | src/transcription/ui/pages | +| API | FastAPI routes | src/transcription/api/routes.py | +| GRAPH | Async processing workflow | src/transcription/services, src/transcription/ai | +| DBREL | PostgreSQL + SQLModel relational persistence | src/transcription/db | +| DBDOC | MongoDB document persistence | src/transcription/db, src/transcription/services | +| OPS | Docker Compose runtime | docker-compose.yml | +| PROMPTS | Transcription prompt artifact library (Markdown files) | .github/prompts, docs | +| TESTS | Pytest verification suite | tests | + +### Satisfaction Mapping + +- UI satisfies REQ-1, REQ-5, REQ-13. +- API satisfies REQ-5. +- GRAPH satisfies REQ-2, REQ-6. +- DBREL satisfies REQ-3, REQ-10, REQ-13. +- DBDOC satisfies REQ-4, REQ-11. +- OPS satisfies REQ-9. +- PROMPTS satisfies REQ-12. + +### Verification Mapping + +- TESTS verifies REQ-1, REQ-2, REQ-3, REQ-4, REQ-5, REQ-10, REQ-11, REQ-12, REQ-13. + +## Requirement Notes + +- Requirement IDs (`REQ-*`) are stable references for planning, implementation, and test traceability. +- The model uses compact tables and traceability lists for renderer compatibility while preserving SysML-style requirement IDs and relationship semantics. +- Requirement categories (functional, interface, performance, and design constraints) are preserved as explicit REQ entries and relationship labels to keep change impact visible. +- PostgreSQL containerization and optional MongoDB containerization are both treated as extremely lightweight and simple operational choices in this architecture. + +## Verification Intent + +- Demonstration: validate end-to-end behavior via running system flows and operator-visible outcomes. +- Inspection: verify architecture and startup/runtime policies in code and configuration. +- Analysis: evaluate asynchronous execution behavior and design sufficiency. +- Test: automate behavioral checks through pytest suites and service-level tests. + +## Glossary + +- Document-oriented persistence: A storage approach that uses flexible document structures for variable data shapes. +- Prompt artifact: A single Markdown file that defines one transcription prompt and is revised independently. +- SysML: Systems Modeling Language used to express structured requirements and traceability. +- System of record: The authoritative persistent store for canonical business data. diff --git a/docs/error_handling.md b/docs/error_handling.md index 73cf0fd..fc7fd88 100644 --- a/docs/error_handling.md +++ b/docs/error_handling.md @@ -271,7 +271,7 @@ Change requirements: - [System overview](index.md) - [Architecture](architecture.md) - [Requirements](requirements.md) -- [Intent](Intent.md) +- [Intent](intent.md) ## Glossary diff --git a/docs/index.md b/docs/index.md index ae5fcd0..380e555 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,56 +1,57 @@ -## Document Transcription System +## Document Transcription System (V1) -This project is a production application for transcribing and preserving historical family documents. It is intentionally designed for personal-scale use, with a simplicity-first architecture that is easy to operate and easy to extend. +This project is a personal-scale application for transcribing and preserving historical family documents. ## Start Here Read [architecture.md](architecture.md) first. -The architecture page is the primary technical reference and defines: +The architecture page is the primary technical reference for: -- deployed topology and infrastructure limits +- runtime topology and infrastructure assumptions - module boundaries and dependency flow -- processing life cycle and data ownership -- test strategy, risk controls, and extension path +- processing lifecycle and data ownership +- test strategy and extension path ## What The Application Does -At a high level, users upload images or PDFs as content sources for handwritten, typed, or typeset documents, run asynchronous transcription jobs, review optional revisions, and search across accepted text. +At a high level, users upload images/PDFs, jobs are processed asynchronously, and users review original transcriptions plus optional revisions. -Core capabilities: +Core V1 capabilities: -- document grouping with one or more content sources and metadata capture -- asynchronous transcription with visible job status -- immutable original transcription persisted with each job (plus provider/model/prompt metadata) -- transcription prompt management with one Markdown file per prompt for human refinement over time -- optional revisions for user-authored edits of original immutable transcription text -- full-text search over accepted transcripts -- export of transcript data +- upload supported source files (`.jpg`, `.jpeg`, `.png`, `.tif`, `.tiff`, `.pdf`) +- asynchronous job processing with visible status (`queued`, `processing`, `transcribed`, `failed`) +- immutable original transcription stored on `Job.text` +- optional single user-authored revision per source (`0..1`) +- prompt artifacts stored as Markdown files in `prompts/` -## Production Operating Model +## Current Operating Model (V1 Baseline) -The system runs with minimal operational overhead: +- application service: FastAPI + NiceGUI +- persistence baseline: SQLModel with SQLite +- worker: in-process async background loop +- deployment baseline: lightweight Docker Compose app runtime -- PostgreSQL in a dedicated Docker container is considered extremely lightweight and simple for this system -- MongoDB in a dedicated Docker container is also considered extremely lightweight and simple for document-centric persistence -- a three-container deployment (app, PostgreSQL, MongoDB) is a simple and acceptable baseline -- no required queue or search-engine containers in the baseline setup - -This operating model keeps deployment and maintenance simple while preserving clean boundaries for future scale. +> Planned persistence evolution (PostgreSQL and optional MongoDB) belongs to V2 planning and is tracked separately. ## Documentation Map - Architecture and technical design: [architecture.md](architecture.md) -- Runtime and deployment requirements: [requirements.md](requirements.md) +- V1 runtime and requirement baseline: [requirements.md](requirements.md) +- Data model and constraints: [schema.md](schema.md) - Error handling policy and operational guidance: [error_handling.md](error_handling.md) +- V1 requirement evidence matrix: [traceability_v1.md](traceability_v1.md) +- Operations runbook: [runbook.md](runbook.md) +- V1 migration and rollback guidance: [migration_v1.md](migration_v1.md) +- V1 release checklist: [release_checklist_v1.md](release_checklist_v1.md) - Domain context and transcription policy: [intent.md](intent.md) -- Transcription Methodology: [transcription_methodology.md](transcription_methodology.md) -- Data model: [schema.md](schema.md) - - +- Transcription methodology: [transcription_methodology.md](transcription_methodology.md) +- V1 execution plan: [ver1/ver1.md](ver1/ver1.md) +- V2 roadmap: [ver2/ver2.md](ver2/ver2.md) ## Glossary -- Document-oriented persistence: Storing data as flexible records instead of fixed relational rows. -- Prompt artifact: A single Markdown file that defines one transcription prompt and is edited independently. -- System of record: The authoritative persistent store for canonical data. +- Prompt artifact: a Markdown file containing one transcription prompt. +- Original transcription: immutable provider output stored on `Job.text`. +- Revision: optional user-authored text linked to a `Source`. +- System of record: the authoritative persistent store for canonical application data. diff --git a/docs/requirements.md b/docs/requirements.md index 62dcf09..edf8436 100644 --- a/docs/requirements.md +++ b/docs/requirements.md @@ -1,12 +1,14 @@ -## Document Transcription System Requirements +## Document Transcription System Requirements (V1 Baseline) -This page captures a SysML v1.6-style requirements baseline for the production system described in [index.md](index.md). The model is represented as concise tables and traceability lists that preserve SysML-style IDs and relationship semantics. +This page captures the **Version 1 baseline requirements** for the currently implemented system. It is the source of truth for V1 acceptance and test traceability. + +Forward-looking architecture changes (for example PostgreSQL/Mongo adoption) are intentionally out of this document and should be tracked in a V2 planning/backlog artifact. ## Scope -- System of interest: the single Python application service (NiceGUI + FastAPI) with PostgreSQL as the relational system of record and optional MongoDB for document-oriented persistence. -- Operational context: local-first execution with Docker Compose and an intentionally lightweight production trajectory. -- Primary concern: end-to-end transcription job lifecycle from upload through completion or failure. +- System of interest: a single Python application service (NiceGUI + FastAPI) with SQLModel persistence. +- Runtime/persistence baseline: local-first execution using SQLite (default `sqlite:///./transcription.db`), with Docker Compose support. +- Primary concern: end-to-end transcription lifecycle from upload through terminal state plus optional single revision editing. ## Requirements Model (Concise Text Form) @@ -15,19 +17,19 @@ This page captures a SysML v1.6-style requirements baseline for the production s | ID | Category | Requirement | Risk | Verify Method | | --- | --- | --- | --- | --- | | REQ-0 | System | Provide end-to-end document transcription with persistent, inspectable lifecycle state. | medium | demonstration | -| REQ-1 | Functional | Allow users to upload one or more images or PDFs as sources from the web UI. | low | test | -| REQ-2 | Functional | Run each upload through asynchronous processing that returns an original transcription or explicit failure. | high | test | -| REQ-3 | Functional | Persist and expose job states: queued, processing, transcribed, failed. | high | inspection | -| REQ-4 | Functional | Persist transcription output, processing history, and failure details. | medium | test | -| REQ-5 | Interface | Expose API and UI views for status inspection and completed transcription reading. | medium | demonstration | +| REQ-1 | Functional | Allow users to upload supported image/PDF files as sources from the web UI. | low | test | +| REQ-2 | Functional | Process uploads asynchronously and return either original transcription output or explicit failure. | high | test | +| REQ-3 | Functional | Persist and expose job states: `queued`, `processing`, `transcribed`, `failed`. | high | inspection | +| REQ-4 | Functional | Persist original provider output (`Job.text`) and failure detail (`Job.error_detail`) for each job. | medium | test | +| REQ-5 | Interface | Expose API/UI views for status inspection and transcription reading. | medium | demonstration | | REQ-6 | Performance | Trigger background processing on upload to preserve UI responsiveness. | medium | analysis | -| REQ-7 | Design Constraint | Keep lifespan-owned runtime resources: SQLAlchemy engine, async session factory, worker resources, provider clients. | medium | inspection | +| REQ-7 | Design Constraint | Keep lifespan-owned runtime resources (engine/session factory/worker resources) initialized and disposed at application boundaries. | medium | inspection | | REQ-8 | Design Constraint | Initialize configuration and logging once at startup through centralized mechanisms. | low | inspection | -| REQ-9 | Design Constraint | Use Docker Compose baseline of app plus PostgreSQL; allow optional MongoDB container when enabled. | medium | demonstration | -| REQ-10 | Design Constraint | Keep schema bootstrap explicit and opt-in; normal startup does not mutate production schema. | high | inspection | -| REQ-11 | Design Constraint | Use service-backed persistence for core document and job data. | medium | inspection | +| REQ-9 | Design Constraint | Support containerized app runtime via Docker Compose using the same V1 persistence model. | medium | demonstration | +| REQ-10 | Design Constraint | Keep schema bootstrap explicit and opt-in for production safety. | high | inspection | +| REQ-11 | Design Constraint | Route persistence changes through service/workflow orchestration boundaries. | medium | inspection | | REQ-12 | Design Constraint | Store transcription prompts as individual Markdown artifacts for iterative refinement. | medium | inspection | -| REQ-13 | Functional | Allow users to create one optional revision of transcription text derived from the original job transcription. | low | test | +| REQ-13 | Functional | Allow users to create/update one optional revision derived from the original job transcription and view/delete it from the job detail flow. | low | test | ### Requirement Relationships @@ -40,22 +42,22 @@ This page captures a SysML v1.6-style requirements baseline for the production s | Element | Type | Doc Reference | | --- | --- | --- | -| UI | NiceGUI pages | src/transcription/ui/pages | -| API | FastAPI routes | src/transcription/api/routes.py | -| GRAPH | Async processing workflow | src/transcription/services, src/transcription/ai | -| DBREL | PostgreSQL + SQLModel relational persistence | src/transcription/db | -| DBDOC | MongoDB document persistence | src/transcription/db, src/transcription/services | -| OPS | Docker Compose runtime | docker-compose.yml | -| PROMPTS | Transcription prompt artifact library (Markdown files) | .github/prompts, docs | -| TESTS | Pytest verification suite | tests | +| UI | NiceGUI pages/components | `src/transcription/ui/pages`, `src/transcription/ui/components` | +| API | FastAPI routes and handlers | `src/transcription/api`, `src/transcription/app.py` | +| WORKER | Async queued-job processing workflow | `src/transcription/worker.py`, `src/transcription/services/workflows.py` | +| DBREL | SQLModel relational persistence (SQLite in V1 baseline) | `src/transcription/models.py`, `src/transcription/db` | +| SERVICES | Service-layer persistence orchestration | `src/transcription/services` | +| OPS | Containerized runtime baseline | `docker-compose.yml`, `Dockerfile` | +| PROMPTS | Transcription prompt artifacts | `prompts/` | +| TESTS | Pytest verification suite | `tests/` | ### Satisfaction Mapping - UI satisfies REQ-1, REQ-5, REQ-13. - API satisfies REQ-5. -- GRAPH satisfies REQ-2, REQ-6. -- DBREL satisfies REQ-3, REQ-10, REQ-13. -- DBDOC satisfies REQ-4, REQ-11. +- WORKER satisfies REQ-2, REQ-6. +- DBREL satisfies REQ-3, REQ-4, REQ-10, REQ-13. +- SERVICES satisfies REQ-4, REQ-11. - OPS satisfies REQ-9. - PROMPTS satisfies REQ-12. @@ -65,21 +67,20 @@ This page captures a SysML v1.6-style requirements baseline for the production s ## Requirement Notes -- Requirement IDs (`REQ-*`) are stable references for planning, implementation, and test traceability. -- The model uses compact tables and traceability lists for renderer compatibility while preserving SysML-style requirement IDs and relationship semantics. -- Requirement categories (functional, interface, performance, and design constraints) are preserved as explicit REQ entries and relationship labels to keep change impact visible. -- PostgreSQL containerization and optional MongoDB containerization are both treated as extremely lightweight and simple operational choices in this architecture. +- Requirement IDs (`REQ-*`) are stable references for planning, implementation, and traceability. +- This document is intentionally **implementation-aligned** for V1 completion and release sign-off. +- Planned storage evolution (PostgreSQL and optional MongoDB) is a **V2 concern** and should be tracked outside this V1 baseline. ## Verification Intent -- Demonstration: validate end-to-end behavior via running system flows and operator-visible outcomes. +- Demonstration: validate end-to-end behavior through operator-visible flows. - Inspection: verify architecture and startup/runtime policies in code and configuration. - Analysis: evaluate asynchronous execution behavior and design sufficiency. -- Test: automate behavioral checks through pytest suites and service-level tests. +- Test: automate behavioral checks through pytest suites and service/UI integration tests. ## Glossary -- Document-oriented persistence: A storage approach that uses flexible document structures for variable data shapes. -- Prompt artifact: A single Markdown file that defines one transcription prompt and is revised independently. -- SysML: Systems Modeling Language used to express structured requirements and traceability. -- System of record: The authoritative persistent store for canonical business data. +- Original transcription: immutable provider output stored on `Job.text`. +- Revision: optional user-authored editable text tied to a `Source` (`0..1` in V1). +- Prompt artifact: a Markdown file containing instructions used for transcription. +- System of record: the authoritative relational store for canonical V1 data. diff --git a/docs/runbook.md b/docs/runbook.md new file mode 100644 index 0000000..55fe1bf --- /dev/null +++ b/docs/runbook.md @@ -0,0 +1,129 @@ +# V1 Operations Runbook + +This runbook provides day-2 operational procedures for the V1 baseline. + +## Scope + +Applies to: + +- local/hosted V1 runtime +- SQLite-backed persistence +- in-process worker lifecycle +- OpenRouter provider integration + +## Preconditions + +- `.env` contains `OPENROUTER_API_KEY` +- app starts successfully +- `uploads/` and `prompts/` are writable +- health endpoint responds at `/healthz` + +## Standard Startup Procedure + +1. Start the app using the project-standard command. +2. Open `/healthz` and verify `{"status":"ok"}`. +3. Open `/ui/upload` and submit a small valid file. +4. Confirm job transitions from `queued` -> `processing` -> `transcribed` (or `failed` with detail). + +## Standard Shutdown Procedure + +1. Stop the application process. +2. Ensure no active process still holds the SQLite file. +3. If maintenance is planned, copy the DB file before edits: + - `transcription.db` (or configured `DATABASE_URL` file path) + +## Incident: Jobs Stuck In `processing` + +### Symptoms + +- Jobs remain `processing` for longer than provider timeout +- New uploads queue but do not complete +- provider usage increases but no terminal job state is visible + +### Checks + +1. Confirm app process is still running. +2. Confirm worker loop is active (startup logs include worker lifespan start). +3. Inspect recent app logs for: + - `worker.process_job` + - `error_id` + - `category` + - `job_id` / `document_id` / `source_id` +4. Verify provider credentials and provider status. + +### Recovery + +1. Restart the app to trigger stale-processing recovery. +2. On startup, app re-queues stale processing jobs based on timeout policy. +3. Re-check jobs page and confirm terminal state progression. +4. If persistent, capture logs + error IDs and move to deep investigation. + +## Incident: Provider Authentication Failures + +### Symptoms + +- failures categorized as provider/auth +- jobs fail quickly with authentication guidance + +### Recovery + +1. Validate `OPENROUTER_API_KEY` value. +2. Restart app after updating env. +3. Re-run a small transcription to confirm recovery. + +## Incident: Upload Failures + +### Symptoms + +- UI reports upload errors +- unsupported extension or empty payload + +### Recovery + +1. Validate file extension (`.jpg`, `.jpeg`, `.png`, `.tif`, `.tiff`, `.pdf`). +2. Validate file is not empty. +3. Validate upload directory permissions. +4. Retry upload. + +## Incident: Database File/Permission Issues + +### Symptoms + +- persistence errors during upload/job update +- startup failures around schema/runtime + +### Recovery + +1. Confirm the configured DB file path exists and is writable. +2. Confirm parent directory permissions. +3. Restore from last known backup copy if corruption is suspected. +4. Restart app and run smoke test. + +## Logging Requirements (Operational) + +Operational triage should always capture: + +- `error_id` +- category +- operation name +- `job_id`, `document_id`, `source_id` when applicable +- UTC timestamp + +## Escalation Packet (When opening an issue) + +Include: + +- exact timestamp window +- one failing `job_id` +- relevant `error_id` values +- latest 100 lines of app logs +- environment summary (`DATABASE_URL` type, app version/commit) + +## Post-Incident Validation + +After mitigation, verify: + +1. Upload works. +2. One job reaches `transcribed`. +3. One induced failure reaches `failed` with error detail. +4. Jobs page and detail page render correctly. diff --git a/docs/schema.md b/docs/schema.md index d7c630e..21b70da 100644 --- a/docs/schema.md +++ b/docs/schema.md @@ -1,21 +1,23 @@ +## Database Schema (V1 Baseline) -## Database schema -This document describes the structure of the database underlying the personal historical-document transcription system. +This document describes the current relational schema for the transcription system. + +All primary and foreign keys in the domain models are UUID-based in V1. --- -## Schema diagram +## Schema Diagram -```mermaid +```mermaid erDiagram - document { - INTEGER id PK + DOCUMENT { + UUID id PK TEXT name } - job { - INTEGER id PK - INTEGER document_id FK + JOB { + UUID id PK + UUID document_id FK TEXT status INTEGER retry_count DATETIME date_created @@ -27,48 +29,59 @@ erDiagram TEXT error_detail } - source { - INTEGER id PK - INTEGER document_id FK - INTEGER job_id FK + SOURCE { + UUID id PK + UUID document_id FK + UUID job_id FK TEXT upload_name TEXT filename TEXT file_path DATETIME date_uploaded } - revision { - INTEGER id PK - INTEGER source_id FK + REVISION { + UUID id PK + UUID source_id FK UNIQUE INTEGER revision TEXT text DATETIME date_created } - document ||--o{ source : "has 0 or more" - document ||--o{ job : "has 0 or more" - job ||--o{ source : "processes 0 or more" - source ||--o{ revision : "has 0 or 1" + DOCUMENT ||--o{ SOURCE : has_many + DOCUMENT ||--o{ JOB : has_many + JOB ||--o{ SOURCE : referenced_by + SOURCE ||--o| REVISION : has_optional_one ``` ---- +--- -## Table Relationships & Constraints -* A document can consist of 0 or more content sources. A document can have 0 or more jobs. -* A source can belong to only one job (which contains the original transcription). A source can only belong to one document. A source may have one optional transcription revision. -* A job can process one or more sources. A job can belong to only one document. -* A revision can belong to only one source. A source may have one optional revision. -* 1:1 optionality is enforced by uniqueness on `revision.source_id` (no revision history chain). -* `Job.text` stores the original immutable provider transcription. -* Revision rows are optional user-authored edits and are derived from the original transcription. Unlike jobs, revision rows can be updated. +## Table Relationships and Constraints + +- A `Document` can have zero or more `Source` records. +- A `Document` can have zero or more `Job` records. +- A `Source` belongs to exactly one `Document` and one `Job`. +- A `Source` may have one optional `Revision`. +- Optional `0..1` revision cardinality is enforced by uniqueness on `revision.source_id`. + +### Invariants + +- `Job.text` stores immutable original provider transcription output. +- `Revision` rows are optional user-authored edits derived from original transcription. +- Revisions do not overwrite original `Job.text`. +- Job status lifecycle values are: `queued`, `processing`, `transcribed`, `failed`. + +### Timestamp Fields + +- `Job.date_created` +- `Job.date_updated` +- `Source.date_uploaded` +- `Revision.date_created` --- ## Glossary -* **Document** - Documents consist of one or more content sources and their related transcriptions. -* **Source** - A content source that is transcribed to text. It can either be an image (.jpg, .tiff, .png) or a PDF (.pdf). -* **Image** - The scanned image of one page of a document. -* **PDF** - A PDF containing the image of one or more pages of a document. -* **Job** - A processing job ingests one or more sources, sends them to an AI model along with a prompt for transcription, then stores the results. The results are immutable, *including the original transcription*. The user can create a revision of the original transcription, but the user cannot modify the original. -* **Transcription** - The text contained in a content source. A job creates the original immutable transcription. A user can optionally create a revised transcription, or "revision". -* **Revision** - A revision is a user-created modification of an existing transcription. It is optional. Some original transcriptions will have no revisions. \ No newline at end of file + +- **Document**: logical grouping for one or more transcribed sources. +- **Source**: uploaded file content (image/PDF) linked to a job. +- **Job**: processing record that stores lifecycle status and original output. +- **Revision**: optional single user-authored edited text linked to a source. diff --git a/docs/ver1/migration_v1.md b/docs/ver1/migration_v1.md new file mode 100644 index 0000000..55dc063 --- /dev/null +++ b/docs/ver1/migration_v1.md @@ -0,0 +1,92 @@ +# V1 Data Migration and Recovery Guidance + +This document defines migration/backfill and rollback guidance for the V1 SQLite baseline. + +## Purpose + +- provide safe procedures for local schema evolution and recovery +- reduce data-loss risk during version upgrades +- establish repeatable pre-change and post-change checks + +## Current Baseline + +- canonical relational store: SQLite +- default DB path: `./transcription.db` +- schema bootstrap may apply compatibility updates for dev/test scenarios + +## Pre-Change Checklist + +Before changing runtime version or schema behavior: + +1. Stop the app process. +2. Create a timestamped DB backup copy. +3. Capture current app commit/version. +4. Export a quick status inventory: + - job counts by status + - total documents/sources/revisions +5. Ensure sufficient disk space. + +## Backup Procedure (SQLite) + +Minimum procedure: + +1. Stop app. +2. Copy DB file to a safe location with timestamp. +3. Store backup path in release notes or change log. + +## Upgrade Procedure (V1) + +1. Perform pre-change checklist. +2. Deploy updated app version. +3. Start app and observe startup logs. +4. Verify schema bootstrap completes (if enabled). +5. Run smoke flow: + - upload valid file + - observe terminal status + - open job detail + +## Backfill Guidance + +V1 backfill is limited and conservative: + +- for records missing newly introduced non-null defaults, use explicit one-time SQL updates only after backup +- avoid destructive rewrites of `Job.text` or `Revision.text` +- never backfill by overwriting original immutable transcription output + +## Rollback Procedure + +If upgrade fails or causes data inconsistency: + +1. Stop app. +2. Restore prior DB backup file. +3. Revert app version to last known-good commit. +4. Restart app. +5. Run smoke flow and confirm stability. + +## Recovery Scenarios + +### Stale processing jobs after crash/restart + +- restart app and allow stale-job recovery to re-queue timed-out `processing` jobs +- monitor for terminal progression + +### Schema mismatch symptoms + +- errors during startup or writes indicating missing columns/indexes +- rollback to last good DB + app version +- reattempt with documented upgrade path + +## Validation Evidence + +For each upgrade rehearsal, capture: + +- backup filename/path +- pre and post job status counts +- smoke test result +- rollback rehearsal result (recommended) + +## Operational Constraints + +- treat DB backups as required before non-trivial upgrades +- do not perform in-place DB edits while app is running +- do not skip post-upgrade smoke validation diff --git a/docs/ver1/release_checklist_v1.md b/docs/ver1/release_checklist_v1.md new file mode 100644 index 0000000..61f3856 --- /dev/null +++ b/docs/ver1/release_checklist_v1.md @@ -0,0 +1,45 @@ +# V1 Release Readiness Checklist + +Use this checklist before declaring V1 operationally complete. + +## A) Functional Readiness + +- [ ] Upload flow works for supported file types. +- [ ] Worker transitions jobs through `queued -> processing -> transcribed|failed`. +- [ ] Job detail displays immutable original transcription from `Job.text`. +- [ ] Revision workflow supports create/update/view/delete for optional single revision. + +## B) Reliability and Error Handling + +- [ ] Error categories surface with actionable messages in UI/API pathways. +- [ ] Failed jobs persist `error_detail` and terminal state. +- [ ] Stale processing recovery verified on restart. +- [ ] Retry/timeout behavior validated against configured limits. + +## C) Operational Readiness + +- [ ] `docs/runbook.md` reviewed and current. +- [ ] `docs/migration_v1.md` reviewed and current. +- [ ] Backup and rollback procedures tested at least once. +- [ ] Incident escalation packet template is known to operators. + +## D) Quality Gates + +- [ ] Lint/type checks pass. +- [ ] `pytest -m "not external" -q` passes. +- [ ] Targeted external/provider checks executed (if credentials available). +- [ ] Release evidence recorded in `docs/release_evidence_v1.md`. + +## E) Traceability and Documentation + +- [ ] `docs/requirements.md` aligns with implemented V1 behavior. +- [ ] `docs/architecture.md`, `docs/schema.md`, and `docs/error_handling.md` are consistent. +- [ ] `docs/traceability_v1.md` is updated with current implementation and test evidence. +- [ ] `docs/ver1/ver1.md` phase status updated with evidence references. +- [ ] REQ traceability evidence links recorded (tests/runbook/checks). + +## Release Sign-Off + +- [ ] Technical sign-off complete. +- [ ] Operational sign-off complete. +- [ ] V1 completion date recorded. diff --git a/docs/ver1/release_evidence_v1.md b/docs/ver1/release_evidence_v1.md new file mode 100644 index 0000000..f382b78 --- /dev/null +++ b/docs/ver1/release_evidence_v1.md @@ -0,0 +1,39 @@ +# V1 Release Evidence Log + +## Step 5 Quality Gates (2026-07-29) + +### Lint + +- Command: `python -m ruff check .` +- Result: ✅ pass +- Notes: initial findings were auto-fixed (`ruff --fix`) plus small manual line-wrap/annotation adjustments. + +### Tests (primary gate) + +- Command: `python -m pytest -m "not external" -q` +- Result: ✅ pass (`[100%]`) + +### Tests (external smoke) + +- Command: `python -m pytest -m external -q` +- Result: ✅ pass (`[100%]`) + +### Type Check + +- Command: `python -m ty check src tests` +- Result: ⚠️ not passing +- Summary: existing SQLModel/SQLAlchemy typing incompatibilities and test double typing mismatches remain. + +Key current blocker families: + +1. SQLModel relationship/query attribute typing (`selectinload`, `order_by`, `.any()`) +2. SQLAlchemy join clause typing in `services/transcription.py` +3. Test fake client type mismatch for `OpenRouterTranscriptionProvider(client=...)` +4. `Settings(**defaults)` typed-dict strictness in `tests/test_config.py` + +## Current Gate Status + +- Lint: pass +- Non-external tests: pass +- External smoke tests: pass +- Type check: **blocked** (requires dedicated typing cleanup pass) diff --git a/docs/ver1/traceability_v1.md b/docs/ver1/traceability_v1.md new file mode 100644 index 0000000..dc76d8b --- /dev/null +++ b/docs/ver1/traceability_v1.md @@ -0,0 +1,40 @@ +# V1 Traceability Matrix + +This matrix provides implementation and validation evidence for V1 requirements (`REQ-0` through `REQ-13`). + +Status values: + +- `done`: implemented and evidence recorded +- `in progress`: partially implemented or evidence incomplete +- `not started`: no implementation/evidence yet + +## Requirement Evidence Table + +| Requirement | Status | Implementation Evidence | Validation Evidence | +| --- | --- | --- | --- | +| REQ-0 | done | End-to-end upload + worker pipeline in `src/transcription/services/store.py`, `src/transcription/worker.py`, `src/transcription/services/workflows.py` | `tests/integration/test_pipeline_flow.py` | +| REQ-1 | done | Upload UI/page flow in `src/transcription/ui/pages/upload_page.py`, `src/transcription/ui/components/upload.py` | `tests/ui/test_upload_page.py`, `tests/integration/test_pipeline_flow.py` | +| REQ-2 | done | Async worker execution and provider call orchestration in `src/transcription/worker.py`, `src/transcription/services/workflows.py` | `tests/integration/test_pipeline_flow.py`, `tests/services/test_workflows_reliability.py` | +| REQ-3 | done | Job lifecycle state model + transitions in `src/transcription/models.py`, `src/transcription/services/jobs.py`, `src/transcription/services/workflows.py` | `tests/services/test_job_service.py`, `tests/ui/test_jobs_page.py` | +| REQ-4 | done | Persistence of original output and failure detail in `src/transcription/services/transcription.py`, `src/transcription/services/workflows.py` | `tests/integration/test_pipeline_flow.py`, `tests/services/test_workflows_reliability.py` | +| REQ-5 | done | Status/result inspection via UI pages and API health route in `src/transcription/ui/pages/jobs_page.py`, `src/transcription/api/health.py` | `tests/ui/test_jobs_page.py`, `tests/ui/test_pages_registration.py`, `tests/api/test_health.py` | +| REQ-6 | done | Background processing trigger/worker notifier and non-blocking workflow in `src/transcription/ui/components/upload.py`, `src/transcription/worker.py` | `tests/test_app.py`, `tests/services/test_workflows_reliability.py` | +| REQ-7 | done | Lifespan-owned runtime resources in `src/transcription/app.py`, `src/transcription/db/runtime.py` | `tests/test_app.py`, `tests/test_db.py` | +| REQ-8 | done | Centralized settings/logging initialization in `src/transcription/config.py`, `src/transcription/app.py` | `tests/test_config.py`, `tests/test_app.py` | +| REQ-9 | done | Containerized runtime baseline in `docker-compose.yml`, `Dockerfile` | `docs/release_checklist_v1.md` (Ops checklist), manual demonstration step | +| REQ-10 | done | Explicit schema bootstrap policy + runtime controls in `src/transcription/config.py`, `src/transcription/app.py`, `src/transcription/db/operations.py` | `tests/test_db.py`, `tests/test_config.py` | +| REQ-11 | done | Service/workflow persistence boundaries in `src/transcription/services/*.py`, `src/transcription/services/workflows.py` | `tests/services/test_job_service.py`, `tests/services/test_transcription_service.py` | +| REQ-12 | done | Prompt artifacts in `prompts/` and loading/validation in `src/transcription/services/transcription.py` | `tests/test_prompts.py` | +| REQ-13 | done | Optional single revision create/update/view/delete in `src/transcription/services/transcription.py`, `src/transcription/ui/pages/jobs_page.py` | `tests/services/test_transcription_service.py`, `tests/ui/test_jobs_page.py` | + +## Operational Evidence (Step 3 Artifacts) + +- Runbook: `docs/runbook.md` +- Migration/backfill/rollback guidance: `docs/migration_v1.md` +- Release readiness checklist: `docs/release_checklist_v1.md` + +## Verification Cadence + +- Per change: maintain `tests/test_traceability.py` mappings for touched requirements. +- Per milestone: update this table status and evidence links. +- Pre-release: confirm all rows are `done` and non-external suite is green. diff --git a/docs/ver1/ver1.md b/docs/ver1/ver1.md index 3d16674..ff6c21d 100644 --- a/docs/ver1/ver1.md +++ b/docs/ver1/ver1.md @@ -150,6 +150,9 @@ V1 is complete when all of the following are true: ### Deliverables - V1 release checklist and acceptance evidence. +- `docs/runbook.md` for incident response and operator workflows. +- `docs/migration_v1.md` for V1 migration/backfill/rollback guidance. +- `docs/release_checklist_v1.md` for release sign-off. ### Exit Criteria - Stakeholder sign-off and launch readiness achieved. diff --git a/docs/ver2/ver2.md b/docs/ver2/ver2.md new file mode 100644 index 0000000..53584d0 --- /dev/null +++ b/docs/ver2/ver2.md @@ -0,0 +1,125 @@ +# Version 2 Plan + +## Purpose + +Version 2 extends the V1 baseline by introducing a production-oriented persistence architecture while preserving current user workflows. + +Primary target changes: + +- Migrate relational persistence from SQLite to PostgreSQL +- Introduce optional MongoDB for document-oriented adjunct data (non-canonical) + +V1 behavior remains the functional baseline unless explicitly superseded by approved V2 requirements. + +--- + +## V2 Goals + +1. **Relational migration complete** + - PostgreSQL becomes the default system of record for `Document`, `Source`, `Job`, and `Revision`. +2. **Operational maturity** + - Repeatable migrations, rollback paths, and environment-specific deployment procedures are documented and tested. +3. **Optional document store integration** + - MongoDB is introduced only for clearly scoped use cases that do not replace canonical relational ownership. +4. **No regression of V1 workflows** + - Upload, queue/worker processing, status inspection, original transcription, and optional single revision remain stable. + +--- + +## Non-Goals (V2) + +- Replacing SQLModel domain ownership with MongoDB +- Introducing breaking UI behavior for existing V1 flows +- Expanding revision cardinality beyond current `0..1` without explicit requirements update + +--- + +## Proposed Scope + +### A) PostgreSQL migration (required) + +- Add PostgreSQL runtime profile for local/dev/prod +- Introduce migration toolchain and migration history +- Convert bootstrap strategy from compatibility patching to explicit migrations +- Validate model constraints and indexes against PostgreSQL +- Add operational checks (connectivity, pool, transaction behavior) + +### B) MongoDB integration (optional, gated) + +- Define approved use cases (for example: denormalized read models, audit/event projections, or search-oriented materializations) +- Keep canonical write path in relational store +- Add feature flag/config gate to enable or disable Mongo features +- Document consistency model and failure behavior + +--- + +## Milestones + +## M1 — Requirements and architecture baseline + +- Create V2 requirements delta from V1 baseline +- Define relational/document ownership boundaries +- Approve migration strategy and cutover approach + +**Exit criteria:** signed architecture decision and updated traceability map. + +## M2 — PostgreSQL foundation + +- Add PostgreSQL environment wiring and secrets strategy +- Add migration framework and initial schema migration +- Add CI path using PostgreSQL service container + +**Exit criteria:** test suite green on PostgreSQL in CI. + +## M3 — Data migration and cutover rehearsal + +- Build SQLite -> PostgreSQL migration utility/playbook +- Rehearse migration on representative datasets +- Validate rollback/recovery procedures + +**Exit criteria:** successful dry-run migration with measured rollback test. + +## M4 — MongoDB optional integration + +- Implement scoped Mongo use case(s) +- Add fallback behavior when Mongo unavailable +- Add tests and operational runbook updates + +**Exit criteria:** feature-gated Mongo behavior validated with no V1 flow regressions. + +## M5 — Release readiness + +- Final regression suite (functional + reliability) +- Performance and failure-mode checks +- Production release checklist and sign-off + +**Exit criteria:** V2 release approval. + +--- + +## Risks and Mitigations + +- **Schema drift risk** -> enforce migration-first policy and CI migration checks. +- **Dual-store consistency risk** -> keep relational source of truth and explicit projection contracts. +- **Operational complexity** -> staged rollout, runbooks, and feature flags. +- **Regression risk in worker lifecycle** -> keep dedicated reliability tests around terminal-state guarantees. + +--- + +## Traceability and Evidence + +Maintain a V2 table with: + +- requirement/change ID +- status (`not started` / `in progress` / `done`) +- implementation PR +- validation evidence (test names, migration rehearsal logs, runbook references) + +--- + +## Suggested first implementation tasks + +1. Create `docs/ver2/adr/` and draft ADR for persistence ownership boundaries. +2. Add PostgreSQL compose profile and env contract. +3. Introduce migration tooling and generate initial migration from current schema. +4. Add CI job for PostgreSQL-backed `pytest -m "not external"`. diff --git a/src/transcription/app.py b/src/transcription/app.py index 2cbdb0f..7f59d1f 100644 --- a/src/transcription/app.py +++ b/src/transcription/app.py @@ -2,12 +2,12 @@ from __future__ import annotations +import logging from contextlib import AsyncExitStack from contextlib import asynccontextmanager from datetime import UTC from datetime import datetime from datetime import timedelta -import logging from fastapi import FastAPI from fastapi import status @@ -26,7 +26,6 @@ from .services.jobs import JobService from .ui import register_pages from .worker import worker_consumer_lifespan - logger = logging.getLogger(__name__) diff --git a/src/transcription/db/operations.py b/src/transcription/db/operations.py index 3a5dcb8..8c8fc78 100644 --- a/src/transcription/db/operations.py +++ b/src/transcription/db/operations.py @@ -67,5 +67,13 @@ def _ensure_sqlite_compat_columns(connection: Connection) -> None: has_unique_source = True break if not has_unique_source: - connection.execute(text("CREATE UNIQUE INDEX IF NOT EXISTS ux_revision_source_id ON revision(source_id)")) - logger.warning("Applied SQLite compatibility schema patch table=revision unique_index=ux_revision_source_id") + connection.execute( + text( + "CREATE UNIQUE INDEX IF NOT EXISTS " + "ux_revision_source_id ON revision(source_id)" + ) + ) + logger.warning( + "Applied SQLite compatibility schema patch " + "table=revision unique_index=ux_revision_source_id" + ) diff --git a/src/transcription/services/workflows.py b/src/transcription/services/workflows.py index da96c3a..3479841 100644 --- a/src/transcription/services/workflows.py +++ b/src/transcription/services/workflows.py @@ -85,7 +85,7 @@ async def process_queued_job( source.id, result.provider, ) - except TimeoutError as exc: + except TimeoutError: error = AppError( f"Provider call timed out after {runtime_settings.worker_provider_timeout_seconds:.1f}s", category=ErrorCategory.EXTERNAL_PROVIDER, diff --git a/src/transcription/ui/pages/jobs_page.py b/src/transcription/ui/pages/jobs_page.py index 2d3b8a7..5b4ee34 100644 --- a/src/transcription/ui/pages/jobs_page.py +++ b/src/transcription/ui/pages/jobs_page.py @@ -23,7 +23,7 @@ from ..components.transcript import render_original_transcription_card from ..components.transcript import render_revision_row -def register_page() -> None: +def register_page() -> None: # noqa: PLR0915 """Register jobs list and detail routes.""" @ui.page("/jobs") @@ -51,7 +51,7 @@ def register_page() -> None: await render_table() @ui.page("/jobs/{job_id}") - async def job_detail_page(job_id: str, request: Request) -> None: + async def job_detail_page(job_id: str, request: Request) -> None: # noqa: PLR0915 session_factory = resolve_session_factory(request.app.state) jobs_service = JobService(session_factory=session_factory) transcription_service = TranscriptionService(session_factory=session_factory) @@ -105,14 +105,52 @@ def register_page() -> None: async def render_revision_panel() -> None: refreshed_job = await jobs_service.read_job(job_id=parsed_job_id) refreshed_source = _resolve_primary_source(refreshed_job) - if refreshed_source is None or refreshed_source.revision is None: + if refreshed_source is None: + ui.label("No source is available for revision editing.").classes("text-body2 text-grey-3") + return + + current_revision = refreshed_source.revision + default_revision_text = ( + current_revision.text if current_revision is not None else (refreshed_job.text or "") + ) + + ui.label("Revision Editor").classes("text-subtitle1 text-weight-medium") + editor = ui.textarea(label="Revision text", value=default_revision_text).props("autogrow outlined") + editor.classes("w-full") + + async def save_revision() -> None: + candidate = (editor.value or "").strip() + if not candidate: + ui.notify("Revision text is required.", type="warning") + return + + try: + await transcription_service.upsert_revision_for_source( + source_id=refreshed_source.id, + text=candidate, + ) + except Exception as exc: # noqa: BLE001 + show_error(exc, title="Save failed", operation="jobs.save_revision") + return + + ui.notify("Revision saved", type="positive") + await render_revision_panel.refresh() + + with ui.row().classes("w-full justify-end"): + ui.button( + "Create revision" if current_revision is None else "Update revision", + on_click=save_revision, + icon="save", + ).props('unelevated color="primary"') + + if current_revision is None: ui.label("No revision exists for this source.").classes("text-body2 text-grey-3") return render_revision_row( - revision=refreshed_source.revision, + revision=current_revision, initially_expanded=True, - on_delete=lambda _revision, rid=refreshed_source.revision.id: delete_revision_by_id(rid), + on_delete=lambda _revision, rid=current_revision.id: delete_revision_by_id(rid), ) await render_revision_panel() diff --git a/tests/api/test_error_responses.py b/tests/api/test_error_responses.py index 03535a8..be33da4 100644 --- a/tests/api/test_error_responses.py +++ b/tests/api/test_error_responses.py @@ -1,11 +1,12 @@ """Tests for API error response envelope handlers.""" +import pytest from fastapi import FastAPI from fastapi.testclient import TestClient -import pytest from transcription.api.errors import register_error_handlers -from transcription.errors import AppError, ErrorCategory +from transcription.errors import AppError +from transcription.errors import ErrorCategory @pytest.mark.integration diff --git a/tests/artifacts/transcriptions/Book_Two_-_page_02.jpg.txt b/tests/artifacts/transcriptions/Book_Two_-_page_02.jpg.txt index 468610f..26f7130 100644 --- a/tests/artifacts/transcriptions/Book_Two_-_page_02.jpg.txt +++ b/tests/artifacts/transcriptions/Book_Two_-_page_02.jpg.txt @@ -11,11 +11,11 @@ BOOK 1 had 54 pages; 14 chapters. BOOK 2 has 70 pages; 18 chapters. BOOK 1 con- sisted largely of first generation family history. BOOK 2 throws more light on the second generation. Sidney promises a BOOK 3 and that may begin to do justice to the third generation. We suggest that Sidney get the help of Louis Shinn -who has a chapter in this book (Chapter 16 - The Last 25 Years on the Doumeeq +who has a chapter in this book (Chapter 16 - The Last 25 Years on the Doumecq Plains. Louis has the gift of seeing, recalling and telling. One sentence in -his chapter gives a great tribute to the Doumeeqers - so far as he knows no one -on the Doumeeq Plains went on relief during the depression. That in a nutshell -shows the sturdy character of the residents of the Doumeeq Plains. +his chapter gives a great tribute to the Doumeccqers--so [sic] far as he knows no one +on the Doumecq Plains went on relief during the depression. That in a nutshell +shows the sturdy character of the residents of the Doumecq Plains. We promised in BOOK 1 that in BOOK 2 we would give the story of the trip of John E. Cochran and wife to Tennessee, Cuba and the Panama Canal. You will see diff --git a/tests/artifacts/transcriptions/Omie_Writes_Home.pdf.txt b/tests/artifacts/transcriptions/Omie_Writes_Home.pdf.txt index 7a99675..bb7921d 100644 --- a/tests/artifacts/transcriptions/Omie_Writes_Home.pdf.txt +++ b/tests/artifacts/transcriptions/Omie_Writes_Home.pdf.txt @@ -18,7 +18,6 @@ the family newsletter two years ago. Nome Alaska August 26, 1923 My Dear Ethel et al. - I don't know when I did write or when you did but I am going to write now however and never the less. But I wish I could talk (I can yet but I @@ -27,7 +26,6 @@ and Polly sit up and listen and that little black rascal of yours would fairly sparkle with listening. Can't I see him listening now to all the yarns we told last summer? - You see, we-Miss Saville and I, took a trip north on the Buford and it was very interesting. We went north thru the Bering Strait into the Arctic and as far as the Ice Pack. There the captain @@ -43,14 +41,14 @@ all around it similar to a currycomb in coarseness; no ears but huge tusks of iv the most repulsive looking animals imaginable and tho I have always read about them I never expect such disagreeable looking creatures. They had a rough brown hairy skin and some of them looked warty. They must have weighed two ton at least. Ere we got them back to Nome -to the natives they were getting extremely odiferous–in fact, you could scarcely stay on the +to the natives they were getting extremely odiferous—in fact, you could scarcely stay on the ship with any degree of comfort unless you had per chance lost your sense of smell. -Then we went north to a few minutes beyond the 70th degree of latitude and thot [sic] for awhile -we would go to Wrangell Island where some men from Stefflonsons [sic] ship were supposed to be -stranded but we didn't get there and instead stopped at a small native village at Cape Serdz [sic] in +Then we went north to a few minutes beyond the 70th degree of latitude and thot for awhile +we would go to Wrangell Island where some men from Steffonsons ship were supposed to be +stranded but we didn't get there and instead stopped at a small native village at Cape Serdz in Siberia. These Eskimo were very primitive. One white squaw man lived there and had for 23 -years. He was a Swede–who else could. Their houses were circular and built up with dirt 2 or +years. He was a Swede--who else could. Their houses were circular and built up with dirt 2 or 3 feet and then skins were stretched over it and weighted down with rocks. Inside, the room was partitioned off at the sides with skins for sleeping quarters. In the main part they had the fire on the ground and the fish drying on lines and the skins hanging around and the dogs and @@ -65,22 +63,22 @@ The other place we stopped was at Whalen, a trading post in Siberia. There these went wild. They rushed helter-skelter, hither and thither, here and there, trying to find something to buy. Prices raised right before your eyes. One would but something for $1.00 and the next might have to pay $2.00, $4.00 or $10.00. That made no difference. They had to -have it. One man I was sort of taking care of, tho [sic] he had his son along for the purpose, -bought 2 ivory tusks, 1 pup, 2 moccasins, 3 or 4 billi[illegible]s, 6 or 8 ivory and silver rings, one +have it. One man I was sort of taking care of, tho he had his son along for the purpose, +bought 2 ivory tusks, 1 pup, 2 moccasins, 3 or 4 billikens, 6 or 8 ivory and silver rings, one fishing line, hooks, floats, etc. and two bird slings. The slings have rocks at the end and the little natives throw them at the flocks of geese and ducks which fly close over the village and the slings entangle their wings and legs, sometimes more than one, and they can't fly. They -come down and the natives capture them. There was more junk brot [sic] aboard than baggage, I +come down and the natives capture them. There was more junk brot aboard than baggage, I do believe. And they say that at the first stop it was worse than here. The red flag was flying -over Whalen and the Russian soldiers were there–a few, one or two or three, I forget the +over Whalen and the Russian soldiers were there—a few, one or two or three, I forget the number. We got home yesterday morning at 5 a.m. but missed the first lighter in so had to stay out until 2:30. The girls had prepared a big meal for us and invited up the Hartfords and then let us talk. Miss Saville talked quite a bit. Any how if you folks don't like this I don't care, it is -all I had to write about and I know Buster'd [sic] listen anyway and I'd soak ole Peter's head if he +all I had to write about and I know Buster'ud listen anyway and I'd soak ole Peter's head if he didn't and Polly would in my lap and I don't know much about the youngest one of yours so -likely he would be squawling. But we did surely enjoy our trip and were gone just long enuf [sic]. +likely he would be squawling. But we did surely enjoy our trip and were gone just long enuf. I expect there were 150 passengers on board and almost or more of the crew and helpers. We had a stateroom down next to the kitchen and 'twas pretty fierce for odor at times. @@ -109,7 +107,7 @@ Ome Reprinted from Cochran Chronicles, Volume 9, Number 1, November 1986 -© [inserted: JECFA] 1986 +© JECFA 1986 Up diff --git a/tests/artifacts/transcriptions/Rod_Moser_Letter_-_p1.jpg.txt b/tests/artifacts/transcriptions/Rod_Moser_Letter_-_p1.jpg.txt index 44f3a4c..68e320e 100644 --- a/tests/artifacts/transcriptions/Rod_Moser_Letter_-_p1.jpg.txt +++ b/tests/artifacts/transcriptions/Rod_Moser_Letter_-_p1.jpg.txt @@ -8,24 +8,23 @@ ISBILL & MOSER DEALERS IN GENERAL MERCHANDISE -Vonore, Tenn., Jany 27- 1913 -Dear Much Aunt Louie -How are you a -few nights ago I sewed a -letter from your folks, so +Vonore, Tenn. Jany 27- 1913 +Dear Much Aunt Adeline +Was at home a +few nights ago & saw a +letter from your folks. So I decided to write you a few lines myself ok -I am contemplateing a +I am contemplateing [sic] a trip out west next summer -& I want Some Olders to go -where I and them. - -I am getting +& [inserted: I] want some of them to go +when I am [inserted: a] them. +Am getting up in years & unmarried so you see the object of -my trip, is to get a bunch -of Young & old maids -& widows out there. I +my trip, is to get a wife +& if there is any old maids +or widows out there, I want you to kiss them -at my fans [sic] mug as they -as soon as I get there +at my [hand?] me at there +as soon as I get there. diff --git a/tests/integration/test_pipeline_flow.py b/tests/integration/test_pipeline_flow.py index 7345243..a528407 100644 --- a/tests/integration/test_pipeline_flow.py +++ b/tests/integration/test_pipeline_flow.py @@ -31,7 +31,12 @@ class TestPipelineSuccessFlow: async def _fake_transcribe(*, prompt_text: str, image_bytes: bytes, mime_type: str) -> TranscriptionResult: _ = (prompt_text, image_bytes, mime_type) - return TranscriptionResult(text="Pipeline transcript", provider="openrouter", model="test-model", prompt_name="transcribe_document.md") + return TranscriptionResult( + text="Pipeline transcript", + provider="openrouter", + model="test-model", + prompt_name="transcribe_document.md", + ) async def _fake_transcribe_document_image( image_path, diff --git a/tests/providers/test_openrouter.py b/tests/providers/test_openrouter.py index 25e519b..7e25090 100644 --- a/tests/providers/test_openrouter.py +++ b/tests/providers/test_openrouter.py @@ -5,8 +5,10 @@ from types import SimpleNamespace import pytest from transcription.config import Settings -from transcription.providers.base import ProviderError, ProviderResponseError -from transcription.providers.openrouter import DEFAULT_OPENROUTER_MODEL, OpenRouterTranscriptionProvider +from transcription.providers.base import ProviderError +from transcription.providers.base import ProviderResponseError +from transcription.providers.openrouter import DEFAULT_OPENROUTER_MODEL +from transcription.providers.openrouter import OpenRouterTranscriptionProvider class _FakeChat: diff --git a/tests/services/test_job_service.py b/tests/services/test_job_service.py index 6b20123..9fae305 100644 --- a/tests/services/test_job_service.py +++ b/tests/services/test_job_service.py @@ -25,7 +25,11 @@ class TestJobService: assert fetched.document.id == document.id @pytest.mark.asyncio - async def test_update_job_state_updates_status_and_retry(self, job_service: JobService, document_service: DocumentService): + async def test_update_job_state_updates_status_and_retry( + self, + job_service: JobService, + document_service: DocumentService, + ): document = Document(id=uuid4(), name="test-bundle") await document_service.create_document(document=document) diff --git a/tests/services/test_transcription_external.py b/tests/services/test_transcription_external.py index 8a837f3..240803b 100644 --- a/tests/services/test_transcription_external.py +++ b/tests/services/test_transcription_external.py @@ -7,7 +7,6 @@ import pytest from transcription.services.transcription import transcribe_document_image - HAS_OPENROUTER_KEY = bool(os.getenv("OPENROUTER_API_KEY")) REAL_IMAGES_DIR = Path(__file__).resolve().parents[1] / "fixtures" / "images" / "real" @@ -68,4 +67,4 @@ class TestRealImageExternalTranscription: f"{result.text}\n" ) artifact_path.write_text(artifact_text, encoding="utf-8") - assert artifact_path.exists() \ No newline at end of file + assert artifact_path.exists() diff --git a/tests/services/test_transcription_service.py b/tests/services/test_transcription_service.py new file mode 100644 index 0000000..86cb962 --- /dev/null +++ b/tests/services/test_transcription_service.py @@ -0,0 +1,85 @@ +"""Tests for revision behavior in TranscriptionService.""" + +from uuid import uuid4 + +import pytest + +from transcription.models import Document +from transcription.models import Job +from transcription.models import JobStatus +from transcription.models import Source +from transcription.services.documents import DocumentService +from transcription.services.jobs import JobService +from transcription.services.transcription import TranscriptionService + + +@pytest.mark.integration +class TestTranscriptionServiceRevisionUpsert: + """Verify optional single-revision create/update semantics.""" + + @pytest.mark.asyncio + async def test_upsert_revision_creates_new_revision(self, default_session_factory): + documents = DocumentService(session_factory=default_session_factory) + jobs = JobService(session_factory=default_session_factory) + transcriptions = TranscriptionService(session_factory=default_session_factory) + + document = Document(id=uuid4(), name="revision-create") + await documents.create_document(document=document) + + job = Job(document_id=document.id, status=JobStatus.TRANSCRIBED, text="Original text") + await jobs.create_job(job=job) + + source = Source( + document_id=document.id, + job_id=job.id, + upload_name="source.jpg", + filename="source.jpg", + file_path="uploads/source.jpg", + ) + async with transcriptions._session_scope() as session: + session.add(source) + await session.commit() + await session.refresh(source) + + revision = await transcriptions.upsert_revision_for_source(source_id=source.id, text="User revision") + fetched = await transcriptions.read_revision_by_source(source.id) + + assert revision.source_id == source.id + assert revision.text == "User revision" + assert fetched is not None + assert fetched.id == revision.id + assert fetched.text == "User revision" + + @pytest.mark.asyncio + async def test_upsert_revision_updates_existing_single_revision(self, default_session_factory): + documents = DocumentService(session_factory=default_session_factory) + jobs = JobService(session_factory=default_session_factory) + transcriptions = TranscriptionService(session_factory=default_session_factory) + + document = Document(id=uuid4(), name="revision-update") + await documents.create_document(document=document) + + job = Job(document_id=document.id, status=JobStatus.TRANSCRIBED, text="Original text") + await jobs.create_job(job=job) + + source = Source( + document_id=document.id, + job_id=job.id, + upload_name="source.jpg", + filename="source.jpg", + file_path="uploads/source.jpg", + ) + async with transcriptions._session_scope() as session: + session.add(source) + await session.commit() + await session.refresh(source) + + first = await transcriptions.upsert_revision_for_source(source_id=source.id, text="Revision v1") + second = await transcriptions.upsert_revision_for_source(source_id=source.id, text="Revision v2") + revisions = await transcriptions.list_revisions_by_job(job.id) + + assert first.id == second.id + assert second.text == "Revision v2" + assert len(revisions) == 1 + assert revisions[0].id == first.id + assert revisions[0].text == "Revision v2" diff --git a/tests/services/test_workflows_reliability.py b/tests/services/test_workflows_reliability.py index 18ed937..a370125 100644 --- a/tests/services/test_workflows_reliability.py +++ b/tests/services/test_workflows_reliability.py @@ -22,9 +22,21 @@ class TestWorkflowReliability: async def test_process_queued_job_timeout_marks_job_failed(self, default_session_factory, monkeypatch): """Provider timeout transitions a queued job to failed with error detail.""" services = ServiceBundle() - object.__setattr__(services, "documents", services.documents.__class__(session_factory=default_session_factory)) - object.__setattr__(services, "jobs", services.jobs.__class__(session_factory=default_session_factory)) - object.__setattr__(services, "transcriptions", services.transcriptions.__class__(session_factory=default_session_factory)) + object.__setattr__( + services, + "documents", + services.documents.__class__(session_factory=default_session_factory), + ) + object.__setattr__( + services, + "jobs", + services.jobs.__class__(session_factory=default_session_factory), + ) + object.__setattr__( + services, + "transcriptions", + services.transcriptions.__class__(session_factory=default_session_factory), + ) async with services.jobs._session_scope() as session: document = Document(id=uuid4(), name="timeout-doc") diff --git a/tests/test_config.py b/tests/test_config.py index d8ba603..de2f7a3 100644 --- a/tests/test_config.py +++ b/tests/test_config.py @@ -5,7 +5,8 @@ from pathlib import Path import pytest from pydantic import ValidationError -from transcription.config import Provider, Settings +from transcription.config import Provider +from transcription.config import Settings def _make_settings(**overrides) -> Settings: diff --git a/tests/test_errors.py b/tests/test_errors.py index 454f2b5..d34327c 100644 --- a/tests/test_errors.py +++ b/tests/test_errors.py @@ -2,7 +2,10 @@ import pytest -from transcription.errors import AppError, ErrorCategory, classify_unexpected_error, new_error_id +from transcription.errors import AppError +from transcription.errors import ErrorCategory +from transcription.errors import classify_unexpected_error +from transcription.errors import new_error_id @pytest.mark.unit diff --git a/tests/test_models.py b/tests/test_models.py index b85ad8f..111a5fe 100644 --- a/tests/test_models.py +++ b/tests/test_models.py @@ -5,7 +5,11 @@ from uuid import UUID import pytest from sqlalchemy.exc import IntegrityError -from transcription.models import Document, Job, JobStatus, Revision, Source +from transcription.models import Document +from transcription.models import Job +from transcription.models import JobStatus +from transcription.models import Revision +from transcription.models import Source def _make_document(**overrides) -> Document: diff --git a/tests/test_prompts.py b/tests/test_prompts.py index fdf2231..1b27ed3 100644 --- a/tests/test_prompts.py +++ b/tests/test_prompts.py @@ -2,7 +2,6 @@ from pathlib import Path - PROMPT_PATH = Path("prompts/transcribe_document.md") diff --git a/tests/ui/test_jobs_page.py b/tests/ui/test_jobs_page.py index e9732a3..eb4223b 100644 --- a/tests/ui/test_jobs_page.py +++ b/tests/ui/test_jobs_page.py @@ -71,3 +71,37 @@ class TestPageRendering: assert response.status_code == 200 assert "Job not found" in response.text + + def test_job_detail_page_shows_revision_editor_when_none_exists(self, app_client, seed_job): + """GET /ui/jobs/{job_id} renders revision editor and create action for sources with no revision.""" + _, client = app_client + job_id = seed_job( + filename="no-revision.pdf", + status=JobStatus.TRANSCRIBED, + transcription_text="original text", + revision_text=None, + ) + + response = client.get(f"/ui/jobs/{job_id}") + + assert response.status_code == 200 + assert "Revision Editor" in response.text + assert "Create revision" in response.text + assert "No revision exists for this source." in response.text + + def test_job_detail_page_shows_update_action_for_existing_revision(self, app_client, seed_job): + """GET /ui/jobs/{job_id} renders revision editor with update action when revision exists.""" + _, client = app_client + job_id = seed_job( + filename="with-revision.pdf", + status=JobStatus.TRANSCRIBED, + transcription_text="original text", + revision_text="hello", + ) + + response = client.get(f"/ui/jobs/{job_id}") + + assert response.status_code == 200 + assert "Revision Editor" in response.text + assert "Update revision" in response.text + assert "hello" in response.text diff --git a/uploads/7704f9cc-4b04-494b-8c55-0ea720249a54_Biography of Omie Rachel Cochran McCarthy_ 1892-1961.pdf b/uploads/7704f9cc-4b04-494b-8c55-0ea720249a54_Biography of Omie Rachel Cochran McCarthy_ 1892-1961.pdf new file mode 100644 index 0000000..0ac4810 Binary files /dev/null and b/uploads/7704f9cc-4b04-494b-8c55-0ea720249a54_Biography of Omie Rachel Cochran McCarthy_ 1892-1961.pdf differ diff --git a/uploads/ca87e22d-21bf-4af8-8cff-312e9384bd24_Effie Ruth Cochran Montague.pdf b/uploads/ca87e22d-21bf-4af8-8cff-312e9384bd24_Effie Ruth Cochran Montague.pdf new file mode 100644 index 0000000..4264f8f Binary files /dev/null and b/uploads/ca87e22d-21bf-4af8-8cff-312e9384bd24_Effie Ruth Cochran Montague.pdf differ