bbchops/transcription
1
0
Fork 0
generated from john/python-template
Code Issues 9 Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: bbchops/transcription:d2b793ea69471f5ac274b9014c106b73a58a53ec
Branches Tags
bbchops/transcription:main bbchops/transcription:traumatized bbchops/transcription:session-engine bbchops/transcription:doc_update bbchops/transcription:going-back bbchops/transcription:job-detail
..
compare: bbchops/transcription:job-detail
Branches Tags
bbchops/transcription:traumatized bbchops/transcription:session-engine bbchops/transcription:doc_update bbchops/transcription:going-back bbchops/transcription:job-detail bbchops/transcription:main
1 Commits
d2b793ea69 .. job-detail
Author SHA1 Message Date
Jim Lancaster 761765636a Update job detail page to track revisions and display document image next to transcription text. 2026-06-28 18:02:19 -05:00
93 changed files with 4516 additions and 3756 deletions
Expand all files Collapse all files
.github/instructions/ui.instructions.md
-6
View File
@@ -1,6 +0,0 @@
---
description: Copilot rules for modifying the UI
applyTo: 'src/transcription/ui/**/*.py'
---
The UI is forbidden from directly touching the database. All interactions should be done using the [services](../../src/transcription/services/)
.gitignore
-3
View File
@@ -14,6 +14,3 @@ wheels/
# SQLite database
*.db
# Document images
uploads/*
README.md
+6 -8
View File
@@ -10,6 +10,7 @@ The app lets you upload a document image/PDF, queues a background transcription
- Persist document + job records in SQLite
- Process jobs in a background worker (`queued -> processing -> transcribed/failed`)
- Store transcript text (or failure detail)
- Track transcript revisions (AI-generated and manual updates)
- Show status and results in the NiceGUI interface
## Quick start
@@ -34,12 +35,6 @@ Optional settings (defaults shown):
DATABASE_URL=sqlite:///./transcription.db
UPLOAD_DIR=./uploads
PROMPT_DIR=./prompts
WORKER_MAX_RETRIES=0
WORKER_RETRY_BACKOFF_SECONDS=0
WORKER_PROVIDER_TIMEOUT_SECONDS=20
WORKER_MIN_TRANSCRIPTION_CHARS=0
WORKER_MIN_TRANSCRIPTION_LINES=0
WORKER_FAIL_ON_FINISH_REASON_LENGTH=false
```
### 3) Run the app
@@ -67,8 +62,11 @@ uv run uvicorn transcription.app:create_app --factory --reload
- **Job detail page** (`/ui/jobs/{job_id}`)
- Shows job metadata and status.
- Displays transcript text when successful.
- Displays failure detail when transcription fails.
- Shows transcript metadata, including provider and model.
- Shows a version table with `Created` and `Version`.
- Displays latest version text in an editable textbox.
- **Update** creates a new transcript version.
- Displays failure detail for failed revisions.
## Prompt artifacts
docs/Intent.md
+24 -7
View File
@@ -1,23 +1,40 @@
# Historical Document Transcription
I have several thousand pages of family history told through letters, postcards, books, and other documents that I want to transcribe to text.
---
## Goals
1. Preserve our family history
2. Unburden my family (and descendants) from having to store and care for the physical media. Once the documents have been transcribed and organized, they can be donated (or kept by a family member that wants to retain them).
3. Make the text easily available and searchable by family members, as well as AI (which may have different requirements).
4. Ability create timelines or assemble the historical record of the family or specific individuals from across the complete document archive. Perhaps use AI to create the timelines in a more narrative form.
---
## Source material
1. **letters, cards, diaries** - handwritten; mostly stored in tubs, with little organization
2. **books** - typed or typeset; mostly self-published books 50-100 pages in length. This may be expanded to include selected pages from other publications.
3. **newspaper clippings, event programs, invitations, and other ephemera**
---
## Methodology
### Verbatim vs. Clean Copy
Transcriptions should be Verbatim and follow scholarly research guidelines, with no modifications to the original text.
## Methodology
### Prompt Curation Policy
Transcription behavior should be implemented with prompt assets that are human-maintainable over time.
See [transcription_methodology.md](transcription_methodology.md) for details on the transcription methodology.
1. Each transcription prompt is stored as an individual Markdown file.
2. Prompt files are refined iteratively as document quality and edge cases are discovered.
3. Prompt changes should be scoped to one prompt file at a time whenever possible to keep review history clear.
### Potential Document Issues
| Document Issue | How to Handle It | Example |
| :--- | :--- | :--- |
| **Misspellings & Errors** | Retain original spelling and insert italicized `[sic]` directly after the error. | `The weather was very cold and publick [sic] business delayed.` |
| **Missing Words / Slips** | Insert the missing word inside square brackets to restore basic readability. | `We went [to] the store to buy supplies.` |
| **Uncertain / Guesswork** | Place your best hypothesis followed by a question mark inside square brackets. | `He went to [Boston?] yesterday to meet the governor.` |
| **Completely Illegible** | Use a clear descriptive term like `[illegible]` or specify the reason (e.g., `[torn]`, `[ink blot]`). | `The total cost was [illegible] dollars.` or `The letter ends here [remainder of page torn].` |
| **Crossed-out Text** | Wrap the removed word or phrase in a deleted tag to preserve the author's edits. | `We left at [deleted: noon] one o'clock instead.` |
| **Squeezed-in Text** | Wrap text that was added above the line or in a tight space in an inserted tag. | `The [inserted: red] house on the hill was abandoned.` |
| **Superscripts & Abbreviations** | Bring raised letters down to the main line, or optionally expand them in brackets. | `Change Gen^l to Genl` OR `Change to Gen[era]l depending on project preference.` |
| **Images / Seals / Signs** | Describe the non-textual element using italicized text inside square brackets. | `[wax notary seal attached here]` or `[sketch of a fort layout]` |
| **Marginalia / Notes** | Note the spatial transition clearly before transcribing the note itself. | `[written in left margin:] Do not share this with anyone.` |
| **Line Breaks / Hyphens** | Rejoin words split across a page margin silently, dropping the line-break hyphen. | `Original: "estab- / lishment" becomes "establishment"` |
| **Ambiguous Capitalization** | Default to modern capitalization rules unless an archaic uppercase letter is clearly intentional. | `If a standard noun like 'Farm' looks randomly capitalized, type 'farm'.` |
**Hierarchical Outlines** | Preserve exact numbering characters (including lowercase Roman numerals or terminal 'j'). Replicate indentation levels using spaces/tabs. Do not correct math or sequence errors silently. | `I. Main Topic`<br>`&nbsp;&nbsp;a. Sub-point`<br>`&nbsp;&nbsp;b. Next point`<br>`III. [sic] Third Topic` |
docs/architecture.md
+243 -79
View File
@@ -1,130 +1,294 @@
# Architecture (V1 Baseline)
# Architecture
This document describes the current architecture of the personal historical-document transcription system and serves as the V1 technical baseline.
This document describes the production architecture of the personal historical-document transcription system. The system is intentionally optimized for single-user operation, low operational overhead, and clean internal boundaries that support future growth without rewrites.
## Architecture Objectives
- preserve source material as transcribed text
- keep operational complexity low for personal-scale deployment
- support asynchronous processing without external queue infrastructure
- maintain clear module boundaries for incremental extension
The production architecture is designed to:
## Runtime Topology
- preserve verbatim family-history source material as searchable text
- keep operational complexity low for a personal deployment
- support asynchronous transcription without requiring distributed infrastructure
- maintain clear module boundaries so extensions can be added incrementally
V1 runtime is a modular monolith:
## Production Scope And Scale
- one FastAPI + NiceGUI application process
- one in-process async worker loop
- relational persistence via SQLModel (SQLite baseline)
The deployed system targets personal use and a corpus of several thousand documents processed over time. The architecture favors simple, composable building blocks over distributed orchestration.
Current scope includes:
- document upload and metadata capture
- asynchronous transcription jobs
- prompt-library driven transcription behavior, with one Markdown file per prompt
- transcript review and revision history
- full-text search over accepted transcripts
- export of transcript data
## Deployment Topology
The production deployment uses [Docker Compose](https://docs.docker.com/compose/) and treats containerized databases as extremely lightweight operational dependencies.
Running [PostgreSQL](https://www.postgresql.org/docs/) in its own container is considered simple by default for this system.
Running [MongoDB](https://www.mongodb.com/docs/) in its own container is also considered simple when document-centric storage is enabled.
Container count is not a hard architectural limit; a three-container deployment (app, PostgreSQL, MongoDB) is an acceptable baseline.
### Baseline Topology (Two Containers)
- one application container
- one PostgreSQL container
- embedded background worker execution inside the app process
### Expanded Topology (Three Containers)
- application container
- PostgreSQL container
- MongoDB container
No additional queue, scheduler, or search-engine containers are required in the baseline production setup.
## Runtime Architecture
```mermaid
flowchart LR
U[Browser User] --> A[FastAPI + NiceGUI App]
A --> W[In-process Worker]
A --> DB[(SQLite via SQLModel)]
W --> P[OpenRouter Provider]
W --> DB
User[Browser User] --> App[FastAPI + NiceGUI Service]
App --> Worker[In-process Background Worker]
App --> PG[(PostgreSQL)]
App --> MG[(MongoDB Document Store)]
Worker --> AI[Transcription Provider]
Worker --> PG
Worker --> MG
```
## Lifecycle Ownership
## Runtime Ownership And Startup Policy (V1 Step 1)
Application lifespan owns runtime setup/teardown:
The current implementation now uses explicit lifespan-owned runtime resources.
- configure logging
- initialize and dispose DB runtime resources
- optional schema bootstrap by environment policy
- recover stale processing jobs
- start/stop worker consumer lifespan
- application lifespan initializes and disposes database runtime resources
- worker lifecycle is owned by application lifespan startup/shutdown
- worker receives lifespan-owned database engine dependency explicitly
- schema bootstrap policy is environment-aware and explicit:
- development/test default to bootstrap enabled
- production defaults to bootstrap disabled
- explicit override is available via configuration
This aligns implementation toward REQ-7 and REQ-10 while preserving personal-scale operational simplicity.
## Layered Module Structure
### Interface Layer
- `src/transcription/ui/**` (NiceGUI pages/components)
- `src/transcription/api/**` (FastAPI routes and error handlers)
Responsibility:
### Application/Workflow Layer
- HTTP API and UI routes
- request/response validation
- status and result presentation
- `src/transcription/services/workflows.py`
- `src/transcription/worker.py`
Out of scope:
Responsibilities:
- business-rule enforcement
- data-access implementation
- orchestration and status transitions
- retry/timeout behavior
- provider call coordination
### Application Layer
### Service Layer
Responsibility:
- `src/transcription/services/*.py`
- upload and job orchestration
- state transitions and retry policy
- coordination across domain and infrastructure ports
Responsibilities:
Out of scope:
- CRUD and transactional boundaries
- domain-aligned persistence operations
- provider-specific protocol details
- ORM or storage-specific logic
### Domain Layer
Responsibility:
- verbatim transcription policy
- revision and provenance invariants
- confidence and annotation semantics
Out of scope:
- web framework concerns
- database and network I/O
### Infrastructure Layer
- `src/transcription/db/**` (runtime/session/bootstrap)
- `src/transcription/providers/**` (OpenRouter adapter)
Responsibility:
- persistence adapters (PostgreSQL and MongoDB)
- transcription-provider adapter
Out of scope:
- business policy decisions
## Processing Workflow
1. User uploads a source file from the UI.
2. App persists `Document`, `Job(queued)`, and `Source`.
3. Worker claims next queued job and marks `processing`.
4. Worker calls provider with prompt + source bytes.
5. On success, app writes immutable `Job.text` and marks `transcribed`.
6. On failure, app writes `Job.error_detail` and marks `failed`.
7. UI exposes job detail, original transcription, and optional revision.
Production transcription flow:
## Domain Ownership Invariants
1. A user uploads an image or PDF through the UI or API.
2. The application validates payloads and creates document and job records.
3. The in-process worker dequeues the job and calls the transcription provider.
4. The application persists transcript output, confidence metadata, and provenance events.
5. Job status transitions from queued to processing to transcribed or failed.
6. The UI and API expose status, revision history, and searchable transcript text.
- `Job.text` is immutable original provider output.
- `Revision` is optional, user-authored, and linked to `Source`.
- `Revision` does not overwrite original job transcription.
- Status lifecycle is fixed to: `queued -> processing -> transcribed|failed`.
## Data Model Ownership
## Data Model Summary
System-of-record entities:
- `Document` has many `Source` and many `Job`.
- `Source` belongs to one `Document` and one `Job`.
- `Source` has optional `Revision` (`0..1`) enforced by unique `revision.source_id`.
- documents and pages
- transcription jobs and status events
- transcript revisions
- provenance metadata
## Simplicity Guardrails (V1)
Storage strategy:
- no external queue/broker required
- no search engine required
- no distributed worker fleet required
- keep provider integration behind adapter boundary
- PostgreSQL for relational system-of-record entities
- MongoDB for document-oriented payloads and large transcription artifacts
- versioned prompt artifacts stored as individual Markdown files for human editing and refinement
- in-memory execution state treated as ephemeral
## Transcription Prompt Asset Policy
The production system treats transcription prompts as maintainable content assets.
- each transcription prompt is stored in its own Markdown file
- prompt files are designed for direct human editing and iterative refinement
- prompt updates are independent and do not require bundling unrelated prompt changes
- prompt file identity and revision history are tracked through normal repository version control
## Simplicity Guardrails
The production system enforces these constraints to prevent accidental over-engineering:
- PostgreSQL in a container is treated as a lightweight default dependency
- MongoDB in a container is treated as a lightweight optional dependency
- three containers (app, PostgreSQL, MongoDB) is an acceptable simple deployment
- no dedicated queue or search cluster is introduced without measured need
- external infrastructure is added only behind existing ports/adapters
## Extension Path
### V1 (current)
The architecture supports additive growth without changing domain contracts.
- SQLite baseline
- OpenRouter provider
- in-process worker
- optional single revision workflow
### Stage 1: Foundation (Current)
### V2 (planned)
- upload, transcription, review, search, export
- in-process worker execution
- single provider adapter
- app plus PostgreSQL deployment
- PostgreSQL as relational baseline
- optional MongoDB adjunct store for scoped use cases
- migration-first schema evolution
### Stage 2: Throughput Hardening
See [ver2/ver2.md](ver2/ver2.md) for roadmap details.
- optional MongoDB document-store enablement
- optional external worker/queue process
- stronger retry and dead-letter handling
### Stage 3: Intelligence Features
- entity extraction and cross-document linking
- timeline and narrative assembly
- optional multi-provider routing
Each stage preserves existing module boundaries and keeps migration risk low.
## Test Strategy
- unit tests for model/service behaviors
- integration tests for upload/workflow reliability
- UI integration tests for page/render contracts
- external provider tests opt-in via marker/config
The test strategy is aligned to personal-scale operation with fast, deterministic feedback.
## Related References
### Unit Tests
- [index.md](index.md)
- [requirements.md](requirements.md)
- [schema.md](schema.md)
- [error_handling.md](error_handling.md)
- domain transcription rules and annotation behavior
- revision-history invariants
- job state-transition logic
### Integration Tests
- repository behavior and transaction boundaries
- persistence-adapter and provider adapter contract mapping
- upload-to-persistence roundtrip
### End-to-End Tests
- happy path: upload, transcribe, review, search, export
- failure path: provider error, retry, surfaced failed status
### CI Execution Model
- fast suite on each push
- optional slower provider-sandbox checks on scheduled runs
## Risks And Controls
### Runtime Responsiveness
Risk:
- long jobs can reduce responsiveness in a single-process deployment
Control:
- bounded concurrency and visible job status in the UI
### Database Concurrency Limits
Risk:
- contention can appear under sustained concurrent writes in personal-scale infrastructure
Control:
- tuned connection pooling and phased use of MongoDB for document-heavy workloads
### Provider Output Variance
Risk:
- transcription quality varies by document type, handwriting legibility, and image quality
Control:
- first-class human review and immutable revision history
## Technology References
- [FastAPI documentation](https://fastapi.tiangolo.com/)
- [NiceGUI documentation](https://nicegui.io/documentation)
- [Docker Compose documentation](https://docs.docker.com/compose/)
- [PostgreSQL documentation](https://www.postgresql.org/docs/)
- [MongoDB documentation](https://www.mongodb.com/docs/)
## Related Pages
- [System overview](index.md)
- [Version 1 plan](ver1/ver1.md)
- [Version 1 Step 1 plan](ver1/ver1-step1.md)
- [Version 1 Step 1 results](ver1/ver1-step1-results.md)
- [Architecture decision records index](adr/README.md)
## Glossary
- Adapter: A component that translates between internal interfaces and external systems such as databases or AI services.
- Background job: Work executed outside the request/response path so the UI remains responsive.
- Boundary: A strict separation between modules with different responsibilities.
- CI (Continuous Integration): Automated test execution for code changes.
- Contract test: A test that verifies an adapter follows expected input/output behavior at a boundary.
- Domain layer: The module that contains core business rules and invariants.
- End-to-end test: A test that validates a full user flow across the running system.
- Full-text search: Text indexing and querying optimized for natural-language search.
- In-process worker: A background executor that runs within the same application process.
- Integration test: A test that verifies interactions between real modules and infrastructure components.
- MongoDB: A document-oriented database used for flexible, high-variance data structures.
- Modular monolith: A single deployable application with strongly separated internal modules.
- Port/Interface: A stable contract used by application/domain code to call infrastructure implementations.
- Prompt artifact: A single Markdown file that defines one transcription prompt and can be revised independently.
- Provenance: Metadata that records where generated data came from and how it was produced.
- Revision history: Versioned record of transcript edits over time.
- System of record: The authoritative persistent store for canonical data.
- Vertical slice: A minimal end-to-end feature path spanning UI/API, application logic, and persistence.
docs/archive/README.md
-23
View File
@@ -1,23 +0,0 @@
# V2 Archive
This folder preserves pre-V1-alignment versions of core documentation that included planned target-state architecture material.
Archived snapshots:
- `index.pre-v1-alignment.md`
- `requirements.pre-v1-alignment.md`
- `architecture.pre-v1-alignment.md`
Purpose:
- keep a durable reference for planned architecture language
- reduce risk of losing useful V2 direction while V1 docs stay implementation-aligned
Notes:
- These files are historical snapshots, not the active V1 source of truth.
- Active V1 docs remain at:
- `docs/index.md`
- `docs/requirements.md`
- `docs/architecture.md`
- V2 planning should continue in `docs/ver2/ver2.md` and related V2 artifacts.
docs/archive/architecture.pre-v1-alignment.md
-299
View File
@@ -1,299 +0,0 @@
# Architecture
This document describes the production architecture of the personal historical-document transcription system. The system is intentionally optimized for single-user operation, low operational overhead, and clean internal boundaries that support future growth without rewrites.
## Architecture Objectives
The production architecture is designed to:
- preserve verbatim family-history source material as searchable text
- keep operational complexity low for a personal deployment
- support asynchronous transcription without requiring distributed infrastructure
- maintain clear module boundaries so extensions can be added incrementally
## Production Scope And Scale
The deployed system targets personal use and a corpus of several thousand documents processed over time. The architecture favors simple, composable building blocks over distributed orchestration.
Current scope includes:
- content source upload and metadata capture
- asynchronous transcription jobs
- prompt-library driven transcription behavior, with one Markdown file per prompt
- original transcription review and optional revision review
- full-text search over accepted transcripts
- export of transcript data
## Deployment Topology
The production deployment uses [Docker Compose](https://docs.docker.com/compose/) and treats containerized databases as extremely lightweight operational dependencies.
Running [PostgreSQL](https://www.postgresql.org/docs/) in its own container is considered simple by default for this system.
Running [MongoDB](https://www.mongodb.com/docs/) in its own container is also considered simple when document-centric storage is enabled.
Container count is not a hard architectural limit; a three-container deployment (app, PostgreSQL, MongoDB) is an acceptable baseline.
### Baseline Topology (Two Containers)
- one application container
- one PostgreSQL container
- embedded background worker execution inside the app process
### Expanded Topology (Three Containers)
- application container
- PostgreSQL container
- MongoDB container
No additional queue, scheduler, or search-engine containers are required in the baseline production setup.
## Runtime Architecture
```mermaid
flowchart LR
User[Browser User] --> App[FastAPI + NiceGUI Service]
App --> Worker[In-process Background Worker]
App --> PG[(PostgreSQL)]
App --> MG[(MongoDB Document Store)]
Worker --> AI[Transcription Provider]
Worker --> PG
Worker --> MG
```
## Runtime Ownership And Startup Policy
The current implementation now uses explicit lifespan-owned runtime resources.
- application lifespan initializes and disposes database runtime resources
- worker lifecycle is owned by application lifespan startup/shutdown
- worker receives lifespan-owned database engine dependency explicitly
- schema bootstrap policy is environment-aware and explicit:
- development/test default to bootstrap enabled
- production defaults to bootstrap disabled
- explicit override is available via configuration
This aligns implementation toward REQ-7 and REQ-10 while preserving personal-scale operational simplicity.
## Layered Module Structure
### Interface Layer
Responsibility:
- HTTP API and UI routes
- request/response validation
- status and result presentation
Out of scope:
- business-rule enforcement
- data-access implementation
### Application Layer
Responsibility:
- upload and job orchestration
- state transitions and retry policy
- coordination across domain and infrastructure ports
Out of scope:
- provider-specific protocol details
- ORM or storage-specific logic
### Domain Layer
Responsibility:
- verbatim transcription policy
- revision and provenance invariants
- confidence and annotation semantics
Out of scope:
- web framework concerns
- database and network I/O
### Infrastructure Layer
Responsibility:
- persistence adapters (PostgreSQL and MongoDB)
- transcription-provider adapter
Out of scope:
- business policy decisions
## Processing Workflow
Production transcription flow:
1. A user uploads one or more content sources through the UI or API.
2. The application validates payloads and creates document, source, and job records.
3. The in-process worker de-queues the job and calls the transcription provider.
4. The application persists original transcription output on the job, plus confidence metadata and provenance events.
5. Job status transitions from queued to processing to transcribed or failed.
6. The UI and API expose status, optional revision to original transcription, and searchable transcription text.
## Data Model Ownership
System-of-record entities:
- documents and content sources
- transcription jobs, original transcription, and status events
- transcript revisions
- provenance metadata
### Original Transcription And Revision Ownership
- each processing job stores the original immutable provider output (`text`)
- provider metadata (`provider`, `model`, `prompt_name`) and failure detail (`error_detail`) are job-owned processing artifacts
- revisions are optional user-authored edits linked to a content source
- a revision can be created from original `job.text`
- many jobs will have zero revisions; revisions are additive and never overwrite original provider output
- a document groups one or more content sources (images, PDFs, and future source types)
Storage strategy:
- PostgreSQL for relational system-of-record entities
- MongoDB for document-oriented payloads and large transcription artifacts
- versioned prompt artifacts stored as individual Markdown files for human editing and refinement
- in-memory execution state treated as ephemeral
## Transcription Prompt Asset Policy
The production system treats transcription prompts as maintainable content assets.
- each transcription prompt is stored in its own Markdown file
- prompt files are designed for direct human editing and iterative refinement
- prompt updates are independent and do not require bundling unrelated prompt changes
- prompt file identity and revision history are tracked through normal repository version control
## Simplicity Guardrails
The production system enforces these constraints to prevent accidental over-engineering:
- PostgreSQL in a container is treated as a lightweight default dependency
- MongoDB in a container is treated as a lightweight optional dependency
- three containers (app, PostgreSQL, MongoDB) is an acceptable simple deployment
- no dedicated queue or search cluster is introduced without measured need
- external infrastructure is added only behind existing ports/adapters
## Extension Path
The architecture supports additive growth without changing domain contracts.
### Stage 1: Foundation (Current)
- upload, transcription, review, search, export
- in-process worker execution
- single provider adapter
- app plus PostgreSQL deployment
### Stage 2: Throughput Hardening
- optional MongoDB document-store enablement
- optional external worker/queue process
- stronger retry and dead-letter handling
### Stage 3: Intelligence Features
- entity extraction and cross-document linking
- timeline and narrative assembly
- optional multi-provider routing
Each stage preserves existing module boundaries and keeps migration risk low.
## Test Strategy
The test strategy is aligned to personal-scale operation with fast, deterministic feedback.
### Unit Tests
- domain transcription rules and annotation behavior
- revision-history invariants
- job state-transition logic
### Integration Tests
- repository behavior and transaction boundaries
- persistence-adapter and provider adapter contract mapping
- upload-to-persistence roundtrip
### End-to-End Tests
- happy path: upload, transcribe, review, search, export
- failure path: provider error, retry, surfaced failed status
### CI Execution Model
- fast suite on each push
- optional slower provider-sandbox checks on scheduled runs
## Risks And Controls
### Runtime Responsiveness
Risk:
- long jobs can reduce responsiveness in a single-process deployment
Control:
- bounded concurrency and visible job status in the UI
### Database Concurrency Limits
Risk:
- contention can appear under sustained concurrent writes in personal-scale infrastructure
Control:
- tuned connection pooling and phased use of MongoDB for document-heavy workloads
### Provider Output Variance
Risk:
- transcription quality varies by content source type, handwriting legibility, and source quality
Control:
- first-class human review and immutable revision history
## Technology References
- [FastAPI documentation](https://fastapi.tiangolo.com/)
- [NiceGUI documentation](https://nicegui.io/documentation)
- [Docker Compose documentation](https://docs.docker.com/compose/)
- [PostgreSQL documentation](https://www.postgresql.org/docs/)
- [MongoDB documentation](https://www.mongodb.com/docs/)
## Related Local References
- [System overview](index.md)
## Glossary
- Adapter: A component that translates between internal interfaces and external systems such as databases or AI services.
- Background job: Work executed outside the request/response path so the UI remains responsive.
- Boundary: A strict separation between modules with different responsibilities.
- CI (Continuous Integration): Automated test execution for code changes.
- Contract test: A test that verifies an adapter follows expected input/output behavior at a boundary.
- Domain layer: The module that contains core business rules and invariants.
- End-to-end test: A test that validates a full user flow across the running system.
- Full-text search: Text indexing and querying optimized for natural-language search.
- In-process worker: A background executor that runs within the same application process.
- Integration test: A test that verifies interactions between real modules and infrastructure components.
- MongoDB: A document-oriented database used for flexible, high-variance data structures.
- Modular monolith: A single deployable application with strongly separated internal modules.
- Port/Interface: A stable contract used by application/domain code to call infrastructure implementations.
- Prompt artifact: A single Markdown file that defines one transcription prompt and can be revised independently.
- Provenance: Metadata that records where generated data came from and how it was produced.
- Revision history: Optional versioned record of user-authored transcription edits over time.
- System of record: The authoritative persistent store for canonical data.
- Vertical slice: A minimal end-to-end feature path spanning UI/API, application logic, and persistence.
docs/archive/index.pre-v1-alignment.md
-56
View File
@@ -1,56 +0,0 @@
## Document Transcription System
This project is a production application for transcribing and preserving historical family documents. It is intentionally designed for personal-scale use, with a simplicity-first architecture that is easy to operate and easy to extend.
## Start Here
Read [architecture.md](architecture.md) first.
The architecture page is the primary technical reference and defines:
- deployed topology and infrastructure limits
- module boundaries and dependency flow
- processing life cycle and data ownership
- test strategy, risk controls, and extension path
## What The Application Does
At a high level, users upload images or PDFs as content sources for handwritten, typed, or typeset documents, run asynchronous transcription jobs, review optional revisions, and search across accepted text.
Core capabilities:
- document grouping with one or more content sources and metadata capture
- asynchronous transcription with visible job status
- immutable original transcription persisted with each job (plus provider/model/prompt metadata)
- transcription prompt management with one Markdown file per prompt for human refinement over time
- optional revisions for user-authored edits of original immutable transcription text
- full-text search over accepted transcripts
- export of transcript data
## Production Operating Model
The system runs with minimal operational overhead:
- PostgreSQL in a dedicated Docker container is considered extremely lightweight and simple for this system
- MongoDB in a dedicated Docker container is also considered extremely lightweight and simple for document-centric persistence
- a three-container deployment (app, PostgreSQL, MongoDB) is a simple and acceptable baseline
- no required queue or search-engine containers in the baseline setup
This operating model keeps deployment and maintenance simple while preserving clean boundaries for future scale.
## Documentation Map
- Architecture and technical design: [architecture.md](architecture.md)
- Runtime and deployment requirements: [requirements.md](requirements.md)
- Error handling policy and operational guidance: [error_handling.md](error_handling.md)
- Domain context and transcription policy: [intent.md](intent.md)
- Transcription Methodology: [transcription_methodology.md](transcription_methodology.md)
- Data model: [schema.md](schema.md)
## Glossary
- Document-oriented persistence: Storing data as flexible records instead of fixed relational rows.
- Prompt artifact: A single Markdown file that defines one transcription prompt and is edited independently.
- System of record: The authoritative persistent store for canonical data.
docs/archive/requirements.pre-v1-alignment.md
-85
View File
@@ -1,85 +0,0 @@
## Document Transcription System Requirements
This page captures a SysML v1.6-style requirements baseline for the production system described in [index.md](index.md). The model is represented as concise tables and traceability lists that preserve SysML-style IDs and relationship semantics.
## Scope
- System of interest: the single Python application service (NiceGUI + FastAPI) with PostgreSQL as the relational system of record and optional MongoDB for document-oriented persistence.
- Operational context: local-first execution with Docker Compose and an intentionally lightweight production trajectory.
- Primary concern: end-to-end transcription job lifecycle from upload through completion or failure.
## Requirements Model (Concise Text Form)
### Requirements
| ID | Category | Requirement | Risk | Verify Method |
| --- | --- | --- | --- | --- |
| REQ-0 | System | Provide end-to-end document transcription with persistent, inspectable lifecycle state. | medium | demonstration |
| REQ-1 | Functional | Allow users to upload one or more images or PDFs as sources from the web UI. | low | test |
| REQ-2 | Functional | Run each upload through asynchronous processing that returns an original transcription or explicit failure. | high | test |
| REQ-3 | Functional | Persist and expose job states: queued, processing, transcribed, failed. | high | inspection |
| REQ-4 | Functional | Persist transcription output, processing history, and failure details. | medium | test |
| REQ-5 | Interface | Expose API and UI views for status inspection and completed transcription reading. | medium | demonstration |
| REQ-6 | Performance | Trigger background processing on upload to preserve UI responsiveness. | medium | analysis |
| REQ-7 | Design Constraint | Keep lifespan-owned runtime resources: SQLAlchemy engine, async session factory, worker resources, provider clients. | medium | inspection |
| REQ-8 | Design Constraint | Initialize configuration and logging once at startup through centralized mechanisms. | low | inspection |
| REQ-9 | Design Constraint | Use Docker Compose baseline of app plus PostgreSQL; allow optional MongoDB container when enabled. | medium | demonstration |
| REQ-10 | Design Constraint | Keep schema bootstrap explicit and opt-in; normal startup does not mutate production schema. | high | inspection |
| REQ-11 | Design Constraint | Use service-backed persistence for core document and job data. | medium | inspection |
| REQ-12 | Design Constraint | Store transcription prompts as individual Markdown artifacts for iterative refinement. | medium | inspection |
| REQ-13 | Functional | Allow users to create one optional revision of transcription text derived from the original job transcription. | low | test |
### Requirement Relationships
- Contains: REQ-0 contains REQ-1 through REQ-13.
- Derives: REQ-2 -> REQ-3, REQ-3 -> REQ-4.
- Traces: REQ-5 -> REQ-3.
- Refines: REQ-6 -> REQ-2.
### Architecture Elements
| Element | Type | Doc Reference |
| --- | --- | --- |
| UI | NiceGUI pages | src/transcription/ui/pages |
| API | FastAPI routes | src/transcription/api/routes.py |
| GRAPH | Async processing workflow | src/transcription/services, src/transcription/ai |
| DBREL | PostgreSQL + SQLModel relational persistence | src/transcription/db |
| DBDOC | MongoDB document persistence | src/transcription/db, src/transcription/services |
| OPS | Docker Compose runtime | docker-compose.yml |
| PROMPTS | Transcription prompt artifact library (Markdown files) | .github/prompts, docs |
| TESTS | Pytest verification suite | tests |
### Satisfaction Mapping
- UI satisfies REQ-1, REQ-5, REQ-13.
- API satisfies REQ-5.
- GRAPH satisfies REQ-2, REQ-6.
- DBREL satisfies REQ-3, REQ-10, REQ-13.
- DBDOC satisfies REQ-4, REQ-11.
- OPS satisfies REQ-9.
- PROMPTS satisfies REQ-12.
### Verification Mapping
- TESTS verifies REQ-1, REQ-2, REQ-3, REQ-4, REQ-5, REQ-10, REQ-11, REQ-12, REQ-13.
## Requirement Notes
- Requirement IDs (`REQ-*`) are stable references for planning, implementation, and test traceability.
- The model uses compact tables and traceability lists for renderer compatibility while preserving SysML-style requirement IDs and relationship semantics.
- Requirement categories (functional, interface, performance, and design constraints) are preserved as explicit REQ entries and relationship labels to keep change impact visible.
- PostgreSQL containerization and optional MongoDB containerization are both treated as extremely lightweight and simple operational choices in this architecture.
## Verification Intent
- Demonstration: validate end-to-end behavior via running system flows and operator-visible outcomes.
- Inspection: verify architecture and startup/runtime policies in code and configuration.
- Analysis: evaluate asynchronous execution behavior and design sufficiency.
- Test: automate behavioral checks through pytest suites and service-level tests.
## Glossary
- Document-oriented persistence: A storage approach that uses flexible document structures for variable data shapes.
- Prompt artifact: A single Markdown file that defines one transcription prompt and is revised independently.
- SysML: Systems Modeling Language used to express structured requirements and traceability.
- System of record: The authoritative persistent store for canonical business data.
docs/error_handling.md
+2 -2
View File
@@ -41,7 +41,7 @@ The system uses stable, implementation-independent categories:
| --- | --- | --- | --- |
| `validation_error` | Payload or parameter shape/content is invalid | UI/API input validation, service guards | no |
| `user_input_error` | User-provided artifact is unacceptable though structurally valid | unsupported file type, empty file, oversized upload | sometimes |
| `not_found_error` | Requested resource does not exist | missing job/document/source/revision | no |
| `not_found_error` | Requested resource does not exist | missing job/document/transcript | no |
| `conflict_error` | Requested operation violates current state constraints | invalid state transition | no |
| `external_provider_error` | External AI/provider call fails | upstream HTTP/API/provider failures | sometimes |
| `infrastructure_transient_error` | Temporary environment issue | network timeout, DB connection reset | yes |
@@ -111,7 +111,7 @@ All logged errors must include, where available:
- `category`
- `operation` (e.g., `upload.submit`, `worker.process_job`, `jobs.refresh`)
- `exception_type`
- `job_id`, `document_id`, `source_id` (when relevant)
- `job_id`, `document_id` (when relevant)
- UTC timestamp
Rules:
docs/index.md
+32 -32
View File
@@ -1,57 +1,57 @@
## Document Transcription System (V1)
## Document Transcription System
This project is a personal-scale application for transcribing and preserving historical family documents.
This project is a production application for transcribing and preserving historical family documents. It is intentionally designed for personal-scale use, with a simplicity-first architecture that is easy to operate and easy to extend.
## Start Here
Read [architecture.md](architecture.md) first.
The architecture page is the primary technical reference for:
Then review [ver1/ver1.md](ver1/ver1.md) for completion scope and [ver1/ver1-step1-results.md](ver1/ver1-step1-results.md) for current architecture-consolidation status.
- runtime topology and infrastructure assumptions
The architecture page is the primary technical reference and defines:
- deployed topology and infrastructure limits
- module boundaries and dependency flow
- processing lifecycle and data ownership
- test strategy and extension path
- processing life cycle and data ownership
- test strategy, risk controls, and extension path
## What The Application Does
At a high level, users upload images/PDFs, jobs are processed asynchronously, and users review original transcriptions plus optional revisions.
At a high level, users upload images of handwritten, typed, or typeset documents, run asynchronous transcription jobs, review and edit transcript revisions, and search across accepted text.
Core V1 capabilities:
Core capabilities:
- upload supported source files (`.jpg`, `.jpeg`, `.png`, `.tif`, `.tiff`, `.pdf`)
- asynchronous job processing with visible status (`queued`, `processing`, `transcribed`, `failed`)
- immutable original transcription stored on `Job.text`
- optional single user-authored revision per source (`0..1`)
- prompt artifacts stored as Markdown files in `prompts/`
- document upload and metadata capture
- asynchronous transcription with visible job status
- transcription prompt management with one Markdown file per prompt for human refinement over time
- revision history for transcript edits
- full-text search over accepted transcripts
- export of transcript data
## Current Operating Model (V1 Baseline)
## Production Operating Model
- application service: FastAPI + NiceGUI
- persistence baseline: SQLModel with SQLite
- worker: in-process async background loop
- deployment baseline: lightweight Docker Compose app runtime
The system runs with minimal operational overhead:
> Planned persistence evolution (PostgreSQL and optional MongoDB) belongs to V2 planning and is tracked separately.
- PostgreSQL in a dedicated Docker container is considered extremely lightweight and simple for this system
- MongoDB in a dedicated Docker container is also considered extremely lightweight and simple for document-centric persistence
- a three-container deployment (app, PostgreSQL, MongoDB) is a simple and acceptable baseline
- no required queue or search-engine containers in the baseline setup
This operating model keeps deployment and maintenance simple while preserving clean boundaries for future scale.
## Documentation Map
- Architecture and technical design: [architecture.md](architecture.md)
- V1 runtime and requirement baseline: [requirements.md](requirements.md)
- Data model and constraints: [schema.md](schema.md)
- Version 1 implementation plan: [ver1/ver1.md](ver1/ver1.md)
- Version 1 Step 1 plan: [ver1/ver1-step1.md](ver1/ver1-step1.md)
- Version 1 Step 1 results: [ver1/ver1-step1-results.md](ver1/ver1-step1-results.md)
- Architecture decision records (ADR index): [adr/README.md](adr/README.md)
- Runtime and deployment requirements: [requirements.md](requirements.md)
- Error handling policy and operational guidance: [error_handling.md](error_handling.md)
- V1 requirement evidence matrix: [traceability_v1.md](traceability_v1.md)
- Operations runbook: [runbook.md](runbook.md)
- V1 migration and rollback guidance: [migration_v1.md](migration_v1.md)
- V1 release checklist: [release_checklist_v1.md](release_checklist_v1.md)
- Domain context and transcription policy: [intent.md](intent.md)
- Transcription methodology: [transcription_methodology.md](transcription_methodology.md)
- V1 execution plan: [ver1/ver1.md](ver1/ver1.md)
- V2 roadmap: [ver2/ver2.md](ver2/ver2.md)
## Glossary
- Prompt artifact: a Markdown file containing one transcription prompt.
- Original transcription: immutable provider output stored on `Job.text`.
- Revision: optional user-authored text linked to a `Source`.
- System of record: the authoritative persistent store for canonical application data.
- Document-oriented persistence: Storing data as flexible records instead of fixed relational rows.
- Prompt artifact: A single Markdown file that defines one transcription prompt and is edited independently.
- System of record: The authoritative persistent store for canonical data.
docs/mvp/mvp-step1.md
+572
View File
@@ -0,0 +1,572 @@
# Step 1 Implementation Plan: `config.py` + `models.py` + `db.py`
## Purpose
Establish the foundational data layer and configuration system that every subsequent MVP step builds on. At the end of this step, the project has a runnable Python package with a validated schema, typed configuration, and a test suite proving the data layer works — before any UI, worker, or AI provider code exists.
---
## 1. Prerequisite: Project Structure Scaffolding
Before writing any logic, create the package skeleton so imports work correctly.
### Files to create (empty `__init__.py` stubs)
```
src/
└── transcription/
├── __init__.py
├── providers/
│ └── __init__.py
├── services/
│ └── __init__.py
└── ui/
└── __init__.py
```
### Files to create (with logic — the Step 1 deliverables)
```
src/transcription/config.py
src/transcription/models.py
src/transcription/db.py
```
### Test files to create
```
tests/
├── __init__.py
├── conftest.py
├── test_config.py
├── test_models.py
└── test_db.py
```
### Update `pyproject.toml`
Add the dependencies that Step 1 requires and won't change later:
```toml pyproject.toml
[project]
name = "transcription"
version = "0.1.0"
description = "Historical document transcription system"
readme = "README.md"
requires-python = ">=3.12"
dependencies = [
"openrouter>=0.7.0",
"pydantic>=2.13.4",
"pydantic-settings>=2.9.1",
"sqlmodel>=0.0.25",
]
[project.optional-dependencies]
dev = [
"pytest>=8.0",
"pytest-asyncio>=0.25",
]
[tool.pytest.ini_options]
addopts = "--strict-markers -q"
markers = [
"unit: pure logic tests with no external dependencies",
"integration: tests that touch framework or database contracts",
"external: tests that call external services (slow, requires credentials)",
]
```
Key additions:
- **`openrouter`** — official OpenRouter Python SDK used for model calls
- **`pydantic-settings`** — for `BaseSettings` with env-var loading (this was split out of `pydantic` core in v2)
- **`sqlmodel`** — provides SQLModel (which bundles SQLAlchemy + Pydantic model integration) and the SQLite driver
- **`pytest` + `pytest-asyncio`** — in `dev` extras for test execution
- **`[tool.pytest.ini_options]`** — strict marker checking enabled from the start; markers registered upfront per pytesting skill conventions
### Delete `hello.py`
The placeholder file is no longer needed.
---
## 2. `config.py` — Centralized Configuration
**Satisfies:** REQ-8 (centralized config and logging at startup)
### Design Decisions
| Decision | Rationale |
|----------|-----------|
| Use `pydantic-settings` `BaseSettings` | Type-safe, validates on construction, loads from env vars and `.env` files automatically |
| `PROVIDER` constrained to `openrouter` for MVP | Keeps configuration explicit while avoiding premature multi-provider complexity |
| `OPENROUTER_API_KEY` required | Matches official SDK docs and avoids ambiguous provider-agnostic naming |
| `PROVIDER_MODEL` defaults to `None` | OpenRouter adapter (Step 3) supplies a sensible default when `None` |
| `OPENROUTER_HTTP_REFERER` and `OPENROUTER_APP_TITLE` optional | Matches SDK optional app-attribution fields |
| `DATABASE_URL` defaults to SQLite | Zero-setup local development; PostgreSQL swap is a single env-var change post-MVP |
| `UPLOAD_DIR` and `PROMPT_DIR` as `Path` objects | Enables `.mkdir(parents=True, exist_ok=True)` and path validation at startup |
| Logging configured via `logging.config.dictConfig` in `setup_logging()` | Centralized, explicit formatter/handler/root logger topology; called once at startup with `disable_existing_loggers=False` |
### Proposed Implementation
```python src/transcription/config.py
"""Centralized application configuration.
All settings are loaded from environment variables (or a .env file)
once at startup. Provider-specific defaults (model names, base URLs)
are resolved by the provider adapters, not here.
"""
from enum import StrEnum
from functools import lru_cache
from pathlib import Path
import logging
import logging.config
from pydantic_settings import BaseSettings, SettingsConfigDict
class Provider(StrEnum):
OPENROUTER = "openrouter"
class Settings(BaseSettings):
model_config = SettingsConfigDict(
env_file=".env",
env_file_encoding="utf-8",
extra="ignore",
)
# --- AI provider ---
provider: Provider = Provider.OPENROUTER
openrouter_api_key: str
provider_model: str | None = None
openrouter_http_referer: str | None = None
openrouter_app_title: str | None = None
# --- persistence ---
database_url: str = "sqlite:///./transcription.db"
# --- filesystem paths ---
upload_dir: Path = Path("./uploads")
prompt_dir: Path = Path("./prompts")
LOGGING_CONFIG: dict[str, object] = {
"version": 1,
"disable_existing_loggers": False,
"formatters": {
"standard": {
"format": "%(asctime)s | %(levelname)-8s | %(name)s | %(message)s",
"datefmt": "%Y-%m-%d %H:%M:%S",
}
},
"handlers": {
"console": {
"class": "logging.StreamHandler",
"formatter": "standard",
"stream": "ext://sys.stdout",
}
},
"root": {
"level": "INFO",
"handlers": ["console"],
},
}
@lru_cache(maxsize=1)
def get_settings() -> Settings:
"""Return the singleton Settings instance.
Cached so the entire application shares one validated config.
"""
return Settings()
def setup_logging() -> None:
"""Configure root logging once at startup."""
logging.config.dictConfig(LOGGING_CONFIG)
```
### Key Behaviors
- **Startup validation**: If `OPENROUTER_API_KEY` is missing from the environment, `Settings()` raises a `ValidationError` immediately — the app won't start with a missing key.
- **`.env` support**: Developers can create a `.env` file in the project root for local keys; it's never committed (already covered by the existing `.gitignore` pattern or a new entry).
- **`extra="ignore"`**: Unknown env vars don't cause errors, keeping the config resilient to unrelated environment variables.
- **`lru_cache`**: `get_settings()` is the single access point. All modules import and call this function rather than constructing `Settings` directly.
- **Centralized logging**: `setup_logging()` calls `dictConfig` exactly once at startup; all modules should use `logging.getLogger(__name__)` and avoid `basicConfig`.
### `.env` template (not committed — add to `.gitignore`)
```bash .env.example
PROVIDER=openrouter
OPENROUTER_API_KEY=sk-or-...
# PROVIDER_MODEL= # optional: OpenRouter adapter supplies default
# OPENROUTER_HTTP_REFERER=https://example.com
# OPENROUTER_APP_TITLE=Historical Transcription MVP
# DATABASE_URL=sqlite:///./transcription.db
# UPLOAD_DIR=./uploads
# PROMPT_DIR=./prompts
```
### `.gitignore` addition
```gitignore .gitignore
# ... existing entries ...
# Environment secrets
.env
```
---
## 3. `models.py` — SQLModel Domain Models
**Satisfies:** REQ-3 (persist and expose job states), REQ-4 (persist transcription output and failure details)
### Design Decisions
| Decision | Rationale |
|----------|-----------|
| Three models: `Document`, `Job`, `Transcript` | Minimal set from MVP Feature 5. One-to-many from Document→Job and one-to-one from Job→Transcript |
| `JobStatus` as a `StrEnum` | Readable in the database (`"queued"` not `1`), type-safe in Python, trivially serializable to JSON for the UI |
| Status values: `queued`, `processing`, `transcribed`, `failed` | Matches MVP Feature 2 lifecycle. REQ-3 also lists `upload` and `completed` — these are deferred to post-MVP when revision/review workflows exist |
| UUIDs for primary keys | Avoids auto-increment collision concerns if we later move to PostgreSQL; safe for distributed ID generation; `uuid4` is simple |
| `uploaded_at`, `created_at`, `updated_at` as UTC `datetime` | Timezone-naive UTC by convention for MVP. Sufficient for single-user, single-timezone operation |
| `Transcript.text` is nullable | A failed job creates a Transcript with `text=None` and `error_detail` populated, keeping the query model uniform |
| Relationships via SQLModel `Relationship` | Enables `document.jobs` and `job.transcript` navigation in service code without manual joins |
### Proposed Implementation
- `resource://skills/fastapi-async-sqlalchemy-modernization/document`
```python src/transcription/models.py
"""SQLModel domain models for the transcription system.
Three models capture the MVP lifecycle:
Document → one-to-many → Job → one-to-one → Transcript
"""
from datetime import datetime, timezone
from enum import StrEnum
from uuid import UUID, uuid4
from sqlmodel import Field, Relationship, SQLModel
class JobStatus(StrEnum):
QUEUED = "queued"
PROCESSING = "processing"
TRANSCRIBED = "transcribed"
FAILED = "failed"
class Document(SQLModel, table=True):
"""An uploaded document image."""
id: UUID = Field(default_factory=uuid4, primary_key=True)
filename: str
file_path: str
uploaded_at: datetime = Field(
default_factory=lambda: datetime.now(timezone.utc),
)
# --- relationships ---
jobs: list["Job"] = Relationship(back_populates="document")
class Job(SQLModel, table=True):
"""A transcription job tied to a single document."""
id: UUID = Field(default_factory=uuid4, primary_key=True)
document_id: UUID = Field(foreign_key="document.id")
status: JobStatus = Field(default=JobStatus.QUEUED)
created_at: datetime = Field(
default_factory=lambda: datetime.now(timezone.utc),
)
updated_at: datetime = Field(
default_factory=lambda: datetime.now(timezone.utc),
)
# --- relationships ---
document: Document = Relationship(back_populates="jobs")
transcript: "Transcript | None" = Relationship(back_populates="job")
class Transcript(SQLModel, table=True):
"""The output of a transcription job."""
id: UUID = Field(default_factory=uuid4, primary_key=True)
job_id: UUID = Field(foreign_key="job.id", unique=True)
text: str | None = None
error_detail: str | None = None
created_at: datetime = Field(
default_factory=lambda: datetime.now(timezone.utc),
)
# --- relationships ---
job: Job = Relationship(back_populates="transcript")
```
### Entity-Relationship Summary
```
┌──────────┐ ┌──────────┐ ┌─────────────┐
│ Document │ 1───* │ Job │ 1───1 │ Transcript │
├──────────┤ ├──────────┤ ├─────────────┤
│ id (PK) │ │ id (PK) │ │ id (PK) │
│ filename │ │ doc_id │──FK──▶│ job_id (FK) │
│ file_path│ │ status │ │ text │
│ uploaded │ │ created │ │ error_detail│
│ │ │ updated │ │ created │
└──────────┘ └──────────┘ └─────────────┘
```
### Why Only Four Status Values
REQ-3 lists six states: `upload`, `queued`, `processing`, `transcribed`, `failed`, `completed`. The MVP simplifies this:
| REQ-3 State | MVP Treatment |
|-------------|---------------|
| `upload` | Implicit — the Document record exists before a Job is created. No separate job state needed. |
| `queued` | ✅ Included — job created, waiting for worker pickup |
| `processing` | ✅ Included — worker is actively transcribing |
| `transcribed` | ✅ Included — AI output received and stored |
| `failed` | ✅ Included — error captured |
| `completed` | Deferred — implies human review/acceptance. In MVP, `transcribed` is the terminal success state. |
---
## 4. `db.py` — Database Engine and Session Management
**Satisfies:** MVP Feature 5 (SQLite auto-created on first startup)
### Design Decisions
| Decision | Rationale |
|----------|-----------|
| Module-level `create_engine` + `Session` factory | REQ-7 (lifespan-owned resources) is deferred. A module-level engine is adequate for MVP's single-process, single-user operation |
| `create_all()` as an explicit function | Called at app startup. MVP auto-creates tables (REQ-10 deferred), but the function is isolated so it's easy to gate behind a flag later |
| `get_session()` as a generator | Standard FastAPI/SQLModel pattern — yields a session, ensures cleanup. Compatible with `Depends()` when the API layer arrives in Step 5 |
| `echo=False` default | Keeps logs clean. Can be toggled for debugging |
### Proposed Implementation
```python src/transcription/db.py
"""Database engine, session factory, and schema bootstrap.
MVP uses SQLite with auto-create-tables at startup.
PostgreSQL migration is a post-MVP configuration change.
"""
import contextlib
from collections.abc import Generator
from sqlmodel import Session, SQLModel, create_engine
from transcription.config import get_settings
def _build_engine():
settings = get_settings()
connect_args = {}
if settings.database_url.startswith("sqlite"):
connect_args["check_same_thread"] = False
return create_engine(
settings.database_url,
echo=False,
connect_args=connect_args,
)
engine = _build_engine()
def create_all() -> None:
"""Create all tables. Called once at application startup."""
SQLModel.metadata.create_all(engine)
@contextlib.contextmanager
def get_session() -> Generator[Session]:
"""Yield a database session and ensure cleanup."""
with Session(engine) as session:
yield session
```
### SQLite-Specific Note
`check_same_thread=False` is required for SQLite when the session may be accessed from different threads (e.g., a background worker on a different thread than the request handler). This setting is harmless and ignored for PostgreSQL connection strings.
---
## 5. Test Plan
Refer to these resources for rules and guidelines about structure:
- `resource://skills/pytesting/document`
- `resource://catalog/prompts/pytest-scaffold`
- `resource://catalog/prompts/pytest-fill-scaffold`
Hierarchy pattern used in this step:
```text
tests/
conftest.py
test_config.py
TestSettingsLoading
test_loads_from_env
test_requires_api_key
TestProviderSettings
test_defaults_to_openrouter
test_rejects_invalid_value
test_optional_fields_default_to_none
TestPathSettings
test_path_fields_are_path_objects
test_models.py
TestDocumentModel
test_can_be_persisted
test_defaults_are_populated
TestJobModel
test_can_be_created_for_document
test_defaults_are_populated
test_transitions_to_transcribed
test_transitions_to_failed
TestTranscriptModel
test_success_record_persists
test_failure_record_persists
test_job_id_is_unique
TestRelationships
test_document_exposes_jobs
test_job_exposes_transcript
test_db.py
TestSchemaBootstrap
test_create_all_creates_expected_tables
TestSessionFactory
test_get_session_yields_session
test_session_is_closed_after_generator_exit
```
### `tests/conftest.py` — Shared Fixtures
```python tests/conftest.py
"""Shared test fixtures.
Every test gets a fresh in-memory SQLite database so tests are
isolated, fast, and leave no artifacts on disk.
"""
import pytest
from sqlmodel import Session, SQLModel, create_engine
from sqlmodel.pool import StaticPool
@pytest.fixture
def session():
"""Provide a clean database session for each test."""
engine = create_engine(
"sqlite://",
connect_args={"check_same_thread": False},
poolclass=StaticPool,
)
SQLModel.metadata.create_all(engine)
with Session(engine) as session:
yield session
```
`StaticPool` ensures a single in-memory SQLite connection is shared across threads, which is required when `TestClient` (Step 5) spawns threads that would otherwise get separate in-memory databases. Establishing it now keeps the fixture stable across all future steps.
### `tests/test_config.py` — Configuration Hierarchy
| Class | Method | What It Verifies |
|------|--------|------------------|
| `TestSettingsLoading` | `test_loads_from_env` | `Settings` constructs successfully when `OPENROUTER_API_KEY` is set via env var |
| `TestSettingsLoading` | `test_requires_api_key` | `Settings()` raises `ValidationError` when `OPENROUTER_API_KEY` is missing |
| `TestProviderSettings` | `test_defaults_to_openrouter` | Default provider is `openrouter` when not explicitly set |
| `TestProviderSettings` | `test_rejects_invalid_value` | Setting `PROVIDER=invalid` raises `ValidationError` |
| `TestProviderSettings` | `test_optional_fields_default_to_none` | `provider_model`, `openrouter_http_referer`, and `openrouter_app_title` are `None` when unset |
| `TestPathSettings` | `test_path_fields_are_path_objects` | `upload_dir` and `prompt_dir` are `Path` instances |
### `tests/test_models.py` — Model & Relationship Hierarchy
| Class | Method | What It Verifies |
|------|--------|------------------|
| `TestDocumentModel` | `test_can_be_persisted` | A `Document` can be persisted and read back with correct fields |
| `TestDocumentModel` | `test_defaults_are_populated` | `id` is auto-generated UUID, `uploaded_at` is populated |
| `TestJobModel` | `test_can_be_created_for_document` | A `Job` linked to a `Document` via FK persists correctly |
| `TestJobModel` | `test_defaults_are_populated` | Default status is `queued`, `created_at` and `updated_at` are populated |
| `TestJobModel` | `test_transitions_to_transcribed` | Status can be updated from `queued` → `processing` → `transcribed` |
| `TestJobModel` | `test_transitions_to_failed` | Status can be updated from `processing` → `failed` |
| `TestTranscriptModel` | `test_success_record_persists` | A `Transcript` with `text` set and `error_detail=None` persists correctly |
| `TestTranscriptModel` | `test_failure_record_persists` | A `Transcript` with `text=None` and `error_detail` set persists correctly |
| `TestRelationships` | `test_document_exposes_jobs` | `document.jobs` returns the linked `Job` list |
| `TestRelationships` | `test_job_exposes_transcript` | `job.transcript` returns the linked `Transcript` |
| `TestTranscriptModel` | `test_job_id_is_unique` | Inserting two transcripts with the same `job_id` raises an integrity error |
### `tests/test_db.py` — Database Bootstrap Hierarchy
| Class | Method | What It Verifies |
|------|--------|------------------|
| `TestSchemaBootstrap` | `test_create_all_creates_expected_tables` | After `create_all()`, the expected tables (`document`, `job`, `transcript`) exist in the database |
| `TestSessionFactory` | `test_get_session_yields_session` | `get_session()` yields a usable `Session` object |
| `TestSessionFactory` | `test_session_is_closed_after_generator_exit` | After the generator is exhausted, the session is closed |
### Marker Strategy (Step 1)
- Markers (`unit`, `integration`, `external`) are registered upfront in `pyproject.toml` with `--strict-markers` enabled, per pytesting skill conventions.
- All Step 1 tests are unmarked — they run in the default lane since they are fast, deterministic, and have no external dependencies.
- When slower integration or external tests are introduced in later steps, apply explicit markers and keep test names unchanged.
### Test Workflow
Follow the two-phase approach from `resource://catalog/prompts/pytest-scaffold` and `resource://catalog/prompts/pytest-fill-scaffold`:
1. **Scaffold phase**: Create test files with class hierarchy, method names, and one-line docstrings only. Validate collection:
- `uv run pytest --collect-only -q`
2. **Fill phase**: Implement assertions, fixtures, and minimal test data. Treat scaffolded names and docstrings as locked. Validate execution:
- `uv run pytest -q`
Scaffolded structure is treated as a stable baseline — do not rename, move, merge, split, or re-nest tests once the scaffold is reviewed.
---
## 6. Step 1 Completion Checklist
When all of the following are true, Step 1 is done and Step 2 can begin:
| # | Criterion | How to Verify |
|---|-----------|---------------|
| 1 | `src/transcription/` package exists with `config.py`, `models.py`, `db.py` | `ls` / file inspection |
| 2 | Empty `__init__.py` stubs exist for `providers/`, `services/`, `ui/` | `ls` / file inspection |
| 3 | `Settings` loads from environment and validates `OPENROUTER_API_KEY` is present | `test_config.py` passes |
| 4 | `Document`, `Job`, `Transcript` models create tables in SQLite | `test_models.py` passes |
| 5 | `JobStatus` enum has exactly four values: `queued`, `processing`, `transcribed`, `failed` | `test_models.py` passes |
| 6 | Foreign key relationships work: Document→Job→Transcript | `test_models.py` passes |
| 7 | `create_all()` bootstraps the schema; `get_session()` yields a working session | `test_db.py` passes |
| 8 | All tests pass: `uv run pytest -q` | CI / local run |
| 9 | `hello.py` is deleted | File inspection |
| 10 | `pyproject.toml` includes `openrouter`, `sqlmodel`, `pydantic-settings`, `pytest`, `pytest-asyncio` | File inspection |
| 10a | `pyproject.toml` has `[tool.pytest.ini_options]` with `--strict-markers` and registered markers | File inspection |
| 11 | `.env.example` documents all config vars; `.env` is in `.gitignore` | File inspection |
| 12 | `setup_logging()` uses `logging.config.dictConfig` with centralized formatter/handler/root config | File inspection |
| 13 | `uv run pytest --collect-only -q` shows expected test hierarchy | Local run |
| 14 | `uv run pytest -q` passes all tests | Local run |
---
## 7. What This Step Does NOT Include
Explicitly out of scope to prevent scope creep:
| Excluded | Reason |
|----------|--------|
| FastAPI / NiceGUI app entrypoint | Step 5 |
| Additional provider adapters beyond OpenRouter | Post-MVP |
| Upload service logic | Step 4 |
| Worker / background processing | Step 4 |
| Transcription prompt files | Step 2 |
| Alembic or migration tooling | Post-MVP (REQ-10 deferred) |
| Async session factory | Post-MVP (REQ-7 deferred) |
---
This plan produces a fully tested, importable data foundation. Every subsequent step imports from `transcription.config`, `transcription.models`, and `transcription.db` without modification.
docs/mvp/mvp-step2.md
+278
View File
@@ -0,0 +1,278 @@
## Step 2: prompts/transcribe_document.md
### Goal
Implement the MVP prompt artifact system by creating a curated transcription prompt file:
- `prompts/transcribe_document.md`
This step primarily satisfies:
- **REQ-12**: prompts stored as individual Markdown artifacts
- MVP Feature 3: prompt-driven verbatim transcription behavior grounded in `docs/intent.md`
---
## Scope for Step 2
### In scope
1. Create prompt artifact directory and first prompt file.
2. Encode transcription rules from `docs/intent.md` into a model-facing prompt.
3. Define stable prompt structure so future revisions are easy to diff/review.
4. Add lightweight tests that validate artifact presence and baseline quality constraints.
5. Update docs/README references so Step 3 can consume prompt file directly.
### Out of scope
- Provider integration logic (Step 3)
- Worker/job orchestration (Step 4)
- UI behavior (Step 5)
---
## Proposed Deliverables
1. **`prompts/transcribe_document.md`**
- production prompt text for historical document transcription
2. **`prompts/README.md`** (recommended)
- conventions for prompt files, revision policy, naming
3. **`tests/test_prompts.py`** (recommended)
- artifact existence + structure checks
4. **Small docs update** (README or docs reference)
- indicate that prompts are file-based and loaded from `PROMPT_DIR`
---
## Detailed Work Breakdown
### 1) Create prompt artifact folder and canonical file
- Add `prompts/` at repo root.
- Add `transcribe_document.md` as the first curated artifact.
- Keep filename stable; this becomes the default in Step 3 unless overridden.
### 2) Author prompt content using a strict, sectioned format
Use section headers so future diffs are clean and policy changes are isolated.
Suggested sections:
1. **Purpose**
- verbatim scholarly transcription of historical documents
2. **Output requirements**
- plain text only
- no summaries, no paraphrasing
- preserve reading order and meaningful structure
3. **Core fidelity rules**
- preserve original wording and punctuation
- don’t silently normalize grammar/spelling
- no invented content
4. **Issue-handling rules (mapped from Intent table)**
- misspellings with `[sic]`
- missing words with `[word]`
- uncertainty with `[guess?]`
- illegible with `[illegible]` / reason tags
- crossed-out text as `[deleted: ...]`
- inserted text as `[inserted: ...]`
- superscripts handling guidance
- non-text elements as `[description]`
- marginalia format `[written in left margin: ...]`
- line-break hyphen rejoin behavior
- capitalization policy
- hierarchical outline preservation (including unusual numbering)
5. **Confidence/ambiguity policy**
- prefer explicit uncertainty markers over hallucination
6. **Final self-checklist for model**
- did I preserve structure?
- did I mark uncertain text?
- did I avoid silent corrections?
### 3) Add prompt-library conventions (`prompts/README.md`)
Recommended conventions:
- one prompt per file
- snake_case names
- each file starts with purpose + behavior contract
- iterative edits, one prompt per PR where possible
- no secrets in prompt files
### 4) Add tests for prompt assets (`tests/test_prompts.py`)
Keep tests robust but not brittle.
Recommended tests:
1. `test_prompt_file_exists`
2. `test_prompt_file_is_not_empty`
3. `test_prompt_mentions_verbatim_behavior`
4. `test_prompt_includes_uncertainty_and_illegible_markers`
5. `test_prompt_includes_deleted_and_inserted_conventions`
Avoid exact full-text matching; verify key semantic anchors only.
### 5) Optional config alignment check
Current config already has:
- `prompt_dir: Path = Path("./prompts")`
In Step 2, ensure docs reflect this and that Step 3 will resolve:
- `PROMPT_DIR / "transcribe_document.md"`
---
## Task-by-Task Execution Checklist
## Phase A — Scaffold files
- [ ] **A1. Create prompt directory**
- Path: `prompts/`
- Verify: directory exists at repo root
- [ ] **A2. Create canonical prompt file**
- Path: `prompts/transcribe_document.md`
- Verify: file exists and is non-empty
- [ ] **A3. (Recommended) Create prompt library README**
- Path: `prompts/README.md`
- Verify: includes naming + revision conventions
---
## Phase B — Author prompt content (core work)
- [ ] **B1. Add Purpose section**
- States verbatim historical transcription objective
- Explicitly disallows summarization/paraphrase
- [ ] **B2. Add Output Contract section**
- Plain text output expectation
- Preserve meaningful structure and reading order
- No fabricated text
- [ ] **B3. Add Rule Set from `docs/intent.md`**
- Misspellings/errors: `[sic]`
- Missing words: `[word]`
- Uncertain readings: `[guess?]`
- Illegible regions: `[illegible]` / reason labels
- Crossed-out text: `[deleted: ...]`
- Squeezed-in text: `[inserted: ...]`
- Superscripts/abbrev handling guidance
- Non-text visuals: bracketed descriptive labels
- Marginalia formatting cue
- Rejoin line-break hyphenated words silently
- Ambiguous capitalization policy
- Hierarchical outline numbering preservation
- [ ] **B4. Add Ambiguity and Confidence policy**
- “Mark uncertainty instead of guessing”
- “Never silently normalize uncertain passages”
- [ ] **B5. Add Final Self-Check section**
- Checklist for fidelity, uncertainty labeling, and format compliance
---
## Phase C — Add validations (tests)
- [ ] **C1. Create prompt tests file**
- Path: `tests/test_prompts.py`
- [ ] **C2. Add existence/health checks**
- Prompt file exists
- Prompt file has content (non-whitespace)
- [ ] **C3. Add semantic anchor checks**
- Mentions verbatim behavior
- Mentions uncertainty marker pattern (`?` in brackets conceptually)
- Mentions illegible handling
- Mentions deleted/inserted conventions
- [ ] **C4. Keep tests resilient**
- Avoid exact full-file snapshot assertions
- Assert required concepts, not precise phrasing
---
## Phase D — Documentation alignment
- [ ] **D1. Update top-level docs/README reference**
- Mention that prompts live in `prompts/`
- Mention Step 3 loads from `PROMPT_DIR`
- [ ] **D2. Confirm config compatibility**
- `src/transcription/config.py` already uses `prompt_dir = Path("./prompts")`
- No code change needed unless naming/path mismatch appears
---
## Phase E — Verification
- [ ] **E1. Run targeted test file**
- `uv run pytest tests/test_prompts.py -q`
- [ ] **E2. Run full suite**
- `uv run pytest -q`
- [ ] **E3. Confirm no regressions**
- All existing tests still green (expected: previous 20 + new prompt tests)
---
## Phase F — Commit plan (recommended granularity)
- [ ] **F1. Commit 1: scaffold**
- `prompts/transcribe_document.md` (initial structure)
- `prompts/README.md` (if included)
- [ ] **F2. Commit 2: finalized prompt content**
- full rule-complete prompt text
- [ ] **F3. Commit 3: tests + docs alignment**
- `tests/test_prompts.py`
- README/docs mention of prompt artifact pattern
---
## Done Criteria (quick gate)
- [ ] Canonical prompt exists and is curated for verbatim transcription.
- [ ] Prompt encodes all high-value handling rules from `docs/intent.md`.
- [ ] Prompt tests pass.
- [ ] Full project tests pass with `uv`.
- [ ] Ready for Step 3 provider integration.
---
## Acceptance Criteria (Definition of Done)
Step 2 is complete when all are true:
1. `prompts/transcribe_document.md` exists and is committed.
2. Prompt includes all critical handling rules from `docs/intent.md`.
3. Prompt is structured with stable section headings for future curation.
4. Prompt tests pass under `uv run pytest -q`.
5. Existing tests remain green (total suite still passes).
6. Docs indicate prompt artifact location and curation policy.
---
## Risks and Mitigations
1. **Risk: prompt too vague → hallucinated reconstructions**
- Mitigation: explicit uncertainty/illegible conventions and “no invention” rule.
2. **Risk: prompt too rigid for mixed document types**
- Mitigation: include neutral defaults + clear annotation formats.
3. **Risk: brittle tests block iterative prompt tuning**
- Mitigation: test semantic anchors, not exact wording.
---
## Handoff to Step 3
After Step 2, Step 3 can immediately:
1. Load `transcribe_document.md` from `PROMPT_DIR`
2. Inject prompt into OpenRouter request
3. Start validating real transcription behavior with minimal glue code
docs/mvp/mvp-step3.md
+236
View File
@@ -0,0 +1,236 @@
## Step 3: services/transcription.py + providers/
### Objective
Implement the **AI transcription integration layer** so the app can:
1. Read the curated prompt from `PROMPT_DIR`
2. Send prompt + image to the configured provider (OpenRouter)
3. Return normalized transcription output (or structured failure)
This corresponds to MVP Step 3 from `docs/mvp.md`:
- `services/transcription.py`
- `providers/` adapter(s)
---
## Scope for Step 3
### In scope
- Provider abstraction and OpenRouter adapter
- Prompt file loading utility in service layer
- Image payload preparation
- One high-level transcription service function usable by Step 4 worker
- Unit tests (mocked provider SDK, no external calls)
### Out of scope
- Job polling/background loop (Step 4)
- DB status transition orchestration in worker loop (Step 4)
- UI invocation/wiring (Step 5)
---
## Planned Deliverables
### Source files
- `src/transcription/providers/base.py`
- `src/transcription/providers/openrouter.py`
- `src/transcription/providers/__init__.py` (exports + factory)
- `src/transcription/services/transcription.py`
- `src/transcription/services/__init__.py` (optional export)
### Tests
- `tests/providers/test_openrouter.py`
- `tests/services/test_transcription.py`
### Test directory convention
- Mirror source domains under `tests/`.
- Provider adapter tests live under `tests/providers/`.
- Service-layer tests live under `tests/services/`.
- Prefer one focused test module per production module (for Step 3: `test_openrouter.py`, `test_transcription.py`).
---
## Design Decisions (before coding)
1. **Provider interface first**
- Define a stable contract independent of SDK specifics.
- Prevent Step 4 from depending on raw SDK response shapes.
2. **Service returns normalized result object**
- Include: `text`, `provider`, `model`, `raw_error`/exception metadata.
- Worker can map this cleanly to `Transcript` and `JobStatus`.
3. **Prompt loaded from file at call time**
- Uses `get_settings().prompt_dir / "transcribe_document.md"`.
- Keeps prompt edits hot-swappable without code changes.
4. **Clear exception boundary**
- SDK/network/model failures become predictable domain exceptions:
- `ProviderError`
- `PromptLoadError`
- `TranscriptionError` (optional top-level wrapper)
5. **Model resolution policy**
- Use `settings.provider_model` if set
- Otherwise use adapter default constant (e.g., vision-capable model slug)
---
## Task-by-Task Execution Checklist
## Phase A — Provider contract
- [ ] Create `src/transcription/providers/base.py`
- [ ] Define protocol/ABC for transcription providers:
- [ ] method signature accepts prompt text + image bytes (or data URL) + mime type
- [ ] returns normalized text result (and optional metadata)
- [ ] Define shared provider exceptions:
- [ ] `ProviderError`
- [ ] optional subclasses (`ProviderAuthError`, `ProviderResponseError`)
---
## Phase B — OpenRouter adapter
- [ ] Create `src/transcription/providers/openrouter.py`
- [ ] Implement `OpenRouterTranscriptionProvider` with:
- [ ] config-driven API key usage
- [ ] optional referer/title attribution headers
- [ ] model resolution fallback when `provider_model` is unset
- [ ] Implement request building:
- [ ] prompt included as instruction content
- [ ] image included in supported format for vision call
- [ ] Implement response parsing:
- [ ] extract final transcript text from SDK response
- [ ] validate non-empty text
- [ ] Wrap SDK failures into `ProviderError` with clean message
---
## Phase C — Provider factory
- [ ] Update `src/transcription/providers/__init__.py`
- [ ] Add `get_transcription_provider()` factory:
- [ ] reads `settings.provider`
- [ ] returns OpenRouter adapter for `openrouter`
- [ ] raises explicit error for unsupported provider values
---
## Phase D — Transcription service (Step 3 core)
- [ ] Create `src/transcription/services/transcription.py`
- [ ] Add prompt loader function:
- [ ] default file: `transcribe_document.md`
- [ ] raises `PromptLoadError` on missing/empty file
- [ ] Add image loader/validator:
- [ ] path existence check
- [ ] allowed mime detection (`.jpg/.jpeg/.png/.tiff/.pdf` policy aligned to MVP)
- [ ] Add high-level function (name example):
- [ ] `transcribe_document_image(image_path, prompt_name="transcribe_document.md")`
- [ ] loads prompt + image
- [ ] calls provider from factory
- [ ] returns normalized transcription result object
- [ ] Add structured logging at key boundaries:
- [ ] prompt loaded
- [ ] provider invoked
- [ ] success/failure outcome (no sensitive data in logs)
---
## Phase E — Tests (two-phase scaffold -> fill)
### Required execution resources
Load and reference these directly during test planning/implementation so the two-phase flow is enforced:
- [ ] `resource://catalog/prompts/pytest-scaffold`
- [ ] `resource://prompts/pytest-scaffold/document`
- [ ] `resource://catalog/prompts/pytest-fill-scaffold`
- [ ] `resource://prompts/pytest-fill-scaffold/document`
### Phase E1 — Scaffold test structure first
Prompt: `resource://catalog/prompts/pytest-scaffold`
Suggested arguments:
- [ ] `target_modules` = `src/transcription/providers/openrouter.py`, `src/transcription/services/transcription.py`
- [ ] `mode` = `scaffold`
- [ ] `path_strategy` = `src-to-tests-mirror`
- [ ] `naming_style` = `concise-behavior`
Expected scaffold outcomes:
- [ ] `tests/providers/test_openrouter.py` exists with class/method skeletons and one-line docstrings
- [ ] `tests/services/test_transcription.py` exists with class/method skeletons and one-line docstrings
- [ ] collection succeeds on scaffold-only tests
Scaffold coverage targets:
- [ ] adapter initializes from settings
- [ ] model fallback when `provider_model is None`
- [ ] referer/title options included when set
- [ ] successful SDK response parses transcript text
- [ ] SDK exception maps to `ProviderError`
- [ ] empty/invalid response maps to `ProviderError`
- [ ] prompt loader reads canonical prompt file
- [ ] missing prompt raises `PromptLoadError`
- [ ] transcription function loads file and calls provider once
- [ ] image path missing raises clear error
- [ ] provider error is propagated/wrapped predictably
- [ ] returned result includes transcript text and metadata
### Phase E2 — Fill scaffolded tests with assertions
Prompt: `resource://catalog/prompts/pytest-fill-scaffold`
Suggested arguments:
- [ ] `target_files` = `tests/providers/test_openrouter.py`, `tests/services/test_transcription.py`
- [ ] `stack` = `pure-python`
- [ ] `strategy` = `minimal`
- [ ] `marker_lane` = `unit`
Fill constraints:
- [ ] preserve scaffold class/method names and one-line docstrings
- [ ] keep mocks to an absolute minimum; mock only network boundaries and non-deterministic failures
- [ ] keep one behavior target per test method
> Default suite should remain deterministic and fast, but mocking should be minimal and intentional.
### Optional real-endpoint validation lane
- [ ] Add an opt-in integration lane for real provider calls (for example `@pytest.mark.integration` and `@pytest.mark.live_api`).
- [ ] Gate live tests behind explicit env vars (for example `OPENROUTER_API_KEY`, optional `RUN_LIVE_API_TESTS=1`).
- [ ] Exclude live tests from default CI/local runs unless explicitly requested.
- [ ] Keep at least one thin smoke path that can validate request/response compatibility against the real endpoint.
---
## Phase F — Verification commands
- [ ] E1 scaffold validation: `uv run pytest --collect-only -q`
- [ ] E2 fill validation (unit lane): `uv run pytest -m unit -q`
- [ ] E2 targeted provider file: `uv run pytest tests/providers/test_openrouter.py -q`
- [ ] E2 targeted service file: `uv run pytest tests/services/test_transcription.py -q`
- [ ] E2 final full-suite check: `uv run pytest -q`
---
## Implementation Notes / Guardrails
- Avoid coupling Step 3 service to DB models directly (that belongs in Step 4 orchestration).
- Do not silently swallow provider errors.
- Keep prompt filename stable (`transcribe_document.md`) unless explicitly parameterized.
- Keep request/response normalization inside provider adapter, not worker/UI layers.
---
## Definition of Done (Step 3)
Step 3 is done when:
1. Provider abstraction exists and OpenRouter adapter is implemented.
2. Service can transcribe a local image using prompt file content.
3. Failures are returned as structured exceptions, not raw SDK traceback noise.
4. Unit tests for provider and service pass.
5. Full suite remains green under `uv run pytest -q`.
6. Step 4 can call a single service function to process queued jobs.
docs/mvp/mvp-step4.md
+262
View File
@@ -0,0 +1,262 @@
## Step 4: `services/upload.py` + `worker.py`
### Objective
Implement the MVP upload and background-processing pipeline so the system can:
1. Save uploaded files into `UPLOAD_DIR`
2. Create `Document` + `Job(status="queued")`
3. Process queued jobs in a worker loop:
- `queued -> processing`
- call Step 3 transcription service
- persist `Transcript`
- finalize as `transcribed` or `failed`
This step advances MVP Feature 1 + Feature 2 and supports REQ-1, REQ-2, REQ-3, REQ-4, REQ-6.
---
## Scope
### In scope
- `src/transcription/services/upload.py`
- `src/transcription/worker.py`
- Upload persistence logic and initial job creation
- Worker polling and single-job lifecycle execution
- Deterministic test coverage for upload + worker (default suite)
### Out of scope
- UI integration and pages (Step 5)
- Queue infrastructure beyond in-process loop
- Async DB/session architecture refactor
- Broad production hardening beyond MVP needs
---
## Planned Deliverables
### Source files
- `src/transcription/services/upload.py`
- `src/transcription/worker.py`
- `src/transcription/services/__init__.py` (export updates as needed)
### Test files
- `tests/services/test_upload.py`
- `tests/services/test_worker.py`
### Optional external lane (already present pattern)
- reuse `external` marker for live-provider checks where appropriate
- keep external out of default lane
---
## Required MCP Prompt References (for test workflow)
Apply these resources directly during Step 4 test creation:
1. `resource://catalog/prompts/pytest-scaffold`
2. `resource://prompts/pytest-scaffold/document`
3. `resource://catalog/prompts/pytest-fill-scaffold`
4. `resource://prompts/pytest-fill-scaffold/document`
And (as referenced by those prompts) apply relevant pytest skill references for:
- naming/hierarchy
- marker defaults
- SQLAlchemy sync testing behavior where applicable
---
## Design Decisions
1. **Upload service owns initial file + record creation**
- Writes file, creates `Document`, creates queued `Job`, returns IDs/path.
2. **Worker owns lifecycle transitions**
- Worker is the single owner of `queued -> processing -> terminal` job state changes.
3. **Worker uses Step 3 service boundary**
- Worker calls `transcribe_document_image(...)`; no provider-specific SDK logic in worker.
4. **Failure information is always persisted**
- On failure: store `Transcript(text=None, error_detail=...)` and set `Job.status=failed`.
5. **Loop remains simple and stoppable**
- In-process polling loop with stop event and poll interval for MVP simplicity and testability.
---
## Task-by-Task Execution Checklist
## Phase A — Implement upload service (`src/transcription/services/upload.py`)
- [ ] Create `UploadError` exception
- [ ] Create `UploadJobResult` dataclass with:
- [ ] `document_id`
- [ ] `job_id`
- [ ] `stored_path`
- [ ] `original_filename`
- [ ] Add filename safety handling:
- [ ] normalize to basename
- [ ] avoid path traversal
- [ ] collision-safe stored name (e.g., UUID prefix/suffix)
- [ ] Validate upload payload:
- [ ] non-empty bytes required
- [ ] extension in supported set (`.jpg/.jpeg/.png/.tif/.tiff/.pdf`)
- [ ] Ensure upload directory exists (`mkdir(parents=True, exist_ok=True)`)
- [ ] Write file bytes to `UPLOAD_DIR`
- [ ] Persist DB records in one transaction:
- [ ] `Document(filename, file_path)`
- [ ] `Job(document_id=..., status=queued)`
- [ ] Return `UploadJobResult`
- [ ] Add logging for success/failure boundaries
---
## Phase B — Implement worker core (`src/transcription/worker.py`)
- [ ] Add `process_next_queued_job(...) -> bool`
- [ ] Fetch oldest queued job
- [ ] Return `False` when no queued jobs exist
- [ ] Transition picked job to `processing` and update timestamp
- [ ] Resolve associated `Document.file_path`
- [ ] Call `transcribe_document_image(image_path=...)`
- [ ] On success:
- [ ] insert/update transcript text
- [ ] clear error detail
- [ ] mark job `transcribed`
- [ ] update timestamp
- [ ] On failure:
- [ ] insert/update transcript with `text=None`, `error_detail=...`
- [ ] mark job `failed`
- [ ] update timestamp
- [ ] Commit terminal state and return `True`
- [ ] Add logs around job pickup, transition, and terminal outcome
---
## Phase C — Implement worker loop (`src/transcription/worker.py`)
- [ ] Add `run_worker_loop(...)`
- [ ] Accept configurable stop event/signal
- [ ] Accept configurable poll interval
- [ ] Repeatedly call `process_next_queued_job`
- [ ] Sleep only when queue is empty
- [ ] Exit cleanly when stop event is set
---
## Phase D — Exports
- [ ] Update `src/transcription/services/__init__.py` to expose upload APIs
- [ ] Keep existing transcription exports intact
---
## Phase E — Tests via MCP scaffold -> fill flow
## E1 Scaffold (structure only)
Use scaffold prompt workflow first for:
- `src/transcription/services/upload.py`
- `src/transcription/worker.py`
Expected scaffold targets:
- `tests/services/test_upload.py`
- `tests/services/test_worker.py`
Scaffold rules:
- [ ] Class hierarchy + method names + one-line docstrings only
- [ ] No assertions or implementation details in scaffold phase
- [ ] Keep method names concise and behavior-focused
Validation:
- [ ] `uv run pytest --collect-only -q`
## E2 Fill scaffold (implementation)
Use fill prompt workflow for:
- `tests/services/test_upload.py`
- `tests/services/test_worker.py`
- stack: `sqlalchemy-sync` (or `mixed` if combining pure + DB behaviors)
- marker lane preference: `unit` and `integration` as appropriate
- strategy: minimal deterministic implementation
Fill rules (invariants):
- [ ] Preserve scaffold class names, method names, and one-line docstrings
- [ ] Do not rename/re-nest scaffolded tests unless explicitly approved
- [ ] One behavior target per test
- [ ] Minimal mocking; mock only network/nondeterministic boundaries
Suggested test coverage:
### `tests/services/test_upload.py`
- [ ] creates file + document + queued job (`integration`)
- [ ] rejects empty bytes (`unit`)
- [ ] rejects unsupported extension (`unit`)
- [ ] writes collision-safe unique filename (`integration`)
- [ ] persisted job status is `queued` (`integration`)
### `tests/services/test_worker.py`
- [ ] returns `False` when queue empty (`integration`)
- [ ] transitions `queued -> processing -> transcribed` on success (`integration`)
- [ ] stores transcript text on success (`integration`)
- [ ] transitions to `failed` and stores `error_detail` on failure (`integration`)
- [ ] updates existing transcript instead of duplicate create (`integration`)
- [ ] worker loop exits when stop event set (`unit`)
---
## Marker Strategy
- `unit`: pure logic tests (filename handling, loop stop behavior, validation logic)
- `integration`: DB + service orchestration tests (SQLite/session/contracts)
- `external`: opt-in live provider tests only (not part of default Step 4 lane)
No new marker needed; reuse existing marker registration.
---
## Validation Sequence (strict order)
- [ ] `uv run pytest --collect-only -q`
- [ ] `uv run pytest -m unit -q` *(if unit tests touched)*
- [ ] `uv run pytest tests/services/test_upload.py -q`
- [ ] `uv run pytest tests/services/test_worker.py -q`
- [ ] `uv run pytest -q`
---
## Reporting Requirements (after implementation)
Implementation report must include:
1. Files created/updated
2. Fixture and marker decisions
3. MCP references used and why
4. Validation command results
5. Remaining risks/open questions (only blockers)
---
## Guardrails
- Keep Step 4 independent from UI concerns.
- Do not call provider SDK directly from worker.
- Do not silently swallow exceptions.
- Always persist terminal job outcome.
- Keep default suite deterministic and fast.
- Preserve scaffold invariants during fill phase.
---
## Definition of Done (Step 4)
Step 4 is complete when:
- [ ] Upload service writes file and creates `Document` + queued `Job`
- [ ] Worker processes queued jobs end-to-end using Step 3 transcription service
- [ ] Success path persists transcript text and sets `transcribed`
- [ ] Failure path persists error detail and sets `failed`
- [ ] Queue-empty path returns cleanly
- [ ] New tests pass and full suite is green (`uv run pytest -q`)
- [ ] Output report includes MCP reference usage + validation evidence
docs/mvp/mvp-step5-results.md
+70
View File
@@ -0,0 +1,70 @@
Step 5 is implemented and passing ✅
### What I implemented
#### App/bootstrap
- Added `src/transcription/app.py`:
- `create_app()` factory
- FastAPI lifespan startup/shutdown:
- `setup_logging()`
- `create_all()`
- ensure `upload_dir` and `prompt_dir`
- start/stop worker thread (`run_worker_loop`)
- includes health router
- registers NiceGUI pages
#### API
- Added `src/transcription/api/__init__.py`
- Added `src/transcription/api/health.py`:
- `GET /healthz` returns `{"status": "ok"}`
#### UI
- Added `src/transcription/ui/upload_page.py`:
- upload page route
- accepted extensions helper
- upload submit helper to Step 4 service (`create_upload_job`)
- success/error state messaging
- Added `src/transcription/ui/jobs_page.py`:
- jobs list page
- job detail page
- `fetch_jobs()` and `fetch_job_detail()` helpers
- Updated `src/transcription/ui/__init__.py`:
- explicit page registration
- NiceGUI mounted via `ui.run_with(app, mount_path="/ui", ...)`
#### Dependencies
- Added runtime dependencies:
- `fastapi`
- `nicegui`
---
### Tests created and filled (scaffold → fill)
- `tests/test_app.py`
- `tests/api/test_health.py`
- `tests/ui/test_pages_registration.py`
- `tests/ui/test_upload_page.py`
- `tests/ui/test_jobs_page.py`
All scaffolded method names/docstrings were preserved and filled with executable assertions.
---
### Validation results (strict sequence)
- `uv run pytest --collect-only -q` ✅
- `uv run pytest -m unit -q` ✅
- `uv run pytest tests/api/test_health.py -q` ✅
- `uv run pytest tests/ui/test_pages_registration.py -q` ✅
- `uv run pytest tests/test_app.py -q` ✅
- `uv run pytest tests/ui/test_upload_page.py -q` ✅
- `uv run pytest tests/ui/test_jobs_page.py -q` ✅
- `uv run pytest -q` ✅
Full suite is green.
---
### Note
You’ll see one warning from FastAPI/Starlette test client about `httpx` deprecation; it does not affect correctness and all tests pass.
docs/mvp/mvp-step5.md
+310
View File
@@ -0,0 +1,310 @@
## Step 5: `app.py` + UI Pages (NiceGUI + FastAPI composition)
## Objective
Implement the MVP user-facing application layer so users can:
1. Upload a document from the UI
2. Trigger Step 4 upload/job creation flow
3. See live job lifecycle status (`queued`, `processing`, `transcribed`, `failed`)
4. Open a job detail view to read transcript text or failure details
This step composes Steps 1–4 into a usable UI.
---
## Architecture Summary (NiceGUI-aligned)
Step 5 uses a **FastAPI app factory + lifespan orchestration** and mounts/registers NiceGUI pages via explicit page modules.
Reference baseline: `resource://skills/nicegui/document`
### Core architecture decisions
- **App factory:** `create_app()`
- **Lifespan-managed resources:** worker start/stop managed in startup/shutdown
- **Modular pages:** upload and jobs pages in separate modules (no monolithic UI file)
- **Health endpoint:** FastAPI-side `/healthz`
- **UI composition:** route pages stay modular and reusable shared shell/components live under `ui/components` as needed
- **Styling architecture:** shared CSS loaded once at startup; avoid ad-hoc per-page styling drift
- **Dependency direction (one-way):**
- `app` -> `config/logging/db/worker/ui/api`
- `ui/pages` -> `ui/components` + `services`
- `services` -> `db/models/providers`
- no reverse imports from services into UI/API
### DB and AI stance (explicit)
- **DB:** already enabled (SQLModel + SQLite), session lifecycle remains request/service-scoped as built in prior steps.
- **AI workflow:** already in place via Step 3 transcription service + Step 4 worker; UI does not call provider SDK directly.
- **Mounted docs:** not in Step 5 scope; docs mounting remains disabled for MVP.
### Async and responsiveness stance
- Prefer `async def` for page handlers and service boundaries when I/O is involved.
- Keep UI handlers non-blocking (no blocking sleeps or synchronous long I/O calls).
- For long-running user actions, always provide explicit loading/progress/error states.
- Keep cancellation/timeout behavior explicit for refresh/poll operations where applicable.
---
## Scope
### In scope
- `src/transcription/app.py`
- `src/transcription/ui/upload_page.py`
- `src/transcription/ui/jobs_page.py`
- `src/transcription/ui/__init__.py`
- `src/transcription/api/health.py` (or equivalent FastAPI health route module)
- UI/app tests with MCP scaffold->fill flow
### Out of scope
- Auth
- advanced filtering/search UX
- batch upload UX beyond MVP
- deployment/container hardening
---
## Planned Deliverables
### Source files
- `src/transcription/app.py` (app factory + lifespan wiring)
- `src/transcription/api/health.py` (GET `/healthz`)
- `src/transcription/ui/upload_page.py` (upload flow)
- `src/transcription/ui/jobs_page.py` (status list + detail)
- `src/transcription/ui/__init__.py` (explicit `register_pages(...)` export)
- `src/transcription/ui/components/*` (shared shell/navigation/status components if introduced)
- `src/transcription/ui/static/*.css` (optional shared CSS loaded once at startup)
### Test files
- `tests/test_app.py`
- `tests/api/test_health.py`
- `tests/ui/test_pages_registration.py`
- `tests/ui/test_upload_page.py`
- `tests/ui/test_jobs_page.py`
---
## Implementation Plan + Checklist
Plan baseline and guardrails source: `resource://skills/nicegui/document`
## Phase A — App factory and lifespan orchestration
- [ ] Create `create_app()` in `src/transcription/app.py`
- [ ] Add FastAPI lifespan startup/shutdown handlers
- [ ] Startup responsibilities:
- [ ] `setup_logging()`
- [ ] `create_all()`
- [ ] ensure directories exist (`upload_dir`, `prompt_dir`)
- [ ] create worker stop event
- [ ] start worker background thread/task
- [ ] Shutdown responsibilities:
- [ ] signal stop event
- [ ] join/cleanup worker thread/task cleanly
- [ ] Register API router(s), including health route
- [ ] Register NiceGUI pages via explicit page registration function
- [ ] Load shared CSS once at startup (if present)
## Phase B — FastAPI health endpoint
- [ ] Create `src/transcription/api/health.py`
- [ ] Add `GET /healthz` returning simple healthy payload
- [ ] Wire route into app factory
## Phase C — Upload page (`ui/upload_page.py`)
- [ ] Add upload route/page registration function
- [ ] Render file input accepting supported extensions
- [ ] On submit:
- [ ] show loading/progress state
- [ ] call `create_upload_job(filename, file_bytes, ...)`
- [ ] show success state with job reference/link
- [ ] On error:
- [ ] show user-safe error message
- [ ] restore ready UI state
- [ ] Ensure non-blocking I/O in UI event handlers; offload CPU-heavy work to worker path
- [ ] Make timeout/cancellation behavior explicit for any long-running action
## Phase D — Jobs page (`ui/jobs_page.py`)
- [ ] Add jobs list route/page registration function
- [ ] Display jobs with status + timestamps
- [ ] Add job detail route/view
- [ ] Show transcript on success, error detail on failure
- [ ] Include explicit refresh action and loading state
- [ ] Ensure error states are surfaced to user and logged
- [ ] Keep refresh path async and bounded to avoid UI freeze
## Phase E — UI registration module
- [ ] Update `src/transcription/ui/__init__.py`
- [ ] Export `register_pages(...)`
- [ ] Ensure each page module exports `register_page(...)`
- [ ] Keep page registration explicit and modular
## Phase F — Shared components and style consistency
- [ ] Add `ui/components` module only for reusable shell elements (header/nav/status chips), not page-local logic
- [ ] Keep structural layout in Python; keep visual polish in shared CSS
- [ ] Avoid one-off styling duplication across upload/jobs pages
---
## MCP Testing Workflow (Required)
Use these resources directly:
- `resource://catalog/prompts/pytest-scaffold`
- `resource://prompts/pytest-scaffold/document`
- `resource://catalog/prompts/pytest-fill-scaffold`
- `resource://prompts/pytest-fill-scaffold/document`
## E1 — Scaffold tests first (structure only)
Target modules:
- `src/transcription/app.py`
- `src/transcription/api/health.py`
- `src/transcription/ui/upload_page.py`
- `src/transcription/ui/jobs_page.py`
Scaffold test files:
- `tests/test_app.py`
- `tests/api/test_health.py`
- `tests/ui/test_pages_registration.py`
- `tests/ui/test_upload_page.py`
- `tests/ui/test_jobs_page.py`
Scaffold constraints:
- [ ] class/method skeletons only
- [ ] one-line docstrings
- [ ] concise behavior-focused names
- [ ] no implementation assertions yet
Validation:
- [ ] `uv run pytest --collect-only -q`
## E2 — Fill scaffold tests
Fill constraints from MCP guidance:
- [ ] preserve scaffold class/method names and docstrings (locked baseline)
- [ ] one behavior target per method
- [ ] deterministic tests preferred
- [ ] minimal mocking; only nondeterministic boundaries
Stack:
- [ ] `fastapi` (or `mixed` if needed for UI+DB fixture combination)
Suggested coverage:
### `tests/api/test_health.py`
- [ ] `/healthz` returns success status and expected payload shape
### `tests/ui/test_pages_registration.py`
- [ ] page registration wiring succeeds
- [ ] expected routes are present
### `tests/test_app.py`
- [ ] startup path initializes runtime dependencies
- [ ] worker start is invoked on startup
- [ ] worker shutdown signal/cleanup is invoked on shutdown
### `tests/ui/test_upload_page.py`
- [ ] upload action calls upload service
- [ ] success feedback displayed
- [ ] error feedback displayed for `UploadError`
- [ ] loading/progress state behavior covered
- [ ] timeout/cancellation behavior covered (if implemented)
### `tests/ui/test_jobs_page.py`
- [ ] list renders job statuses
- [ ] detail shows transcript text for successful job
- [ ] detail shows error detail for failed job
- [ ] refresh/loading state behavior covered
Marker strategy:
- [ ] `unit` for pure helpers/state formatting
- [ ] `integration` for app/page/service+DB contracts
- [ ] `external` not required for default Step 5 lane
Async behavior assertions:
- [ ] long-running actions keep button/inputs in expected disabled state
- [ ] completion/failure returns controls to ready state
---
## Validation Sequence (strict)
- [ ] `uv run pytest --collect-only -q`
- [ ] `uv run pytest -m unit -q` *(if unit tests touched)*
- [ ] `uv run pytest tests/api/test_health.py -q`
- [ ] `uv run pytest tests/ui/test_pages_registration.py -q`
- [ ] `uv run pytest tests/test_app.py -q`
- [ ] `uv run pytest tests/ui/test_upload_page.py -q`
- [ ] `uv run pytest tests/ui/test_jobs_page.py -q`
- [ ] `uv run pytest -q`
---
## Guardrails (NiceGUI + MVP)
- [ ] Do not collapse pages into one file.
- [ ] Do not use implicit global side effects for runtime wiring.
- [ ] Keep UI responsive with explicit loading/progress/error states.
- [ ] Do not block UI handlers with synchronous long I/O.
- [ ] Do not place provider SDK calls in UI handlers.
- [ ] Keep dependency direction one-way and maintainable.
- [ ] Keep shared UI in `ui/components`; keep service logic out of page modules.
---
## Definition of Done
- [ ] App factory + lifespan are in place
- [ ] Health endpoint exists and is tested
- [ ] Upload page creates queued jobs through service boundary
- [ ] Jobs list/detail pages render status/transcript/failure data
- [ ] Worker lifecycle is started/stopped by app lifespan
- [ ] Async UI states (loading/success/error) are deterministic and tested
- [ ] Scaffold->fill testing flow completed and validated
- [ ] Full suite passes: `uv run pytest -q`
## Completion Checks (NiceGUI skill aligned)
- [ ] Uses app factory and FastAPI lifespan
- [ ] Pages are modularized (not single-file UI)
- [ ] Health endpoint exists on FastAPI side
- [ ] Dependency direction is clean and one-way
- [ ] Async-first guidance is applied where I/O exists, with explicit non-blocking UX states
- [ ] DB/AI/docs decisions are explicit and reflected in structure
- [ ] Plan references baseline URI: `resource://skills/nicegui/document`
---
## PR Checklist (Integrated)
### Implementation
- [ ] `app.py` app factory + lifespan implemented
- [ ] FastAPI health route (`/healthz`) implemented
- [ ] `ui/upload_page.py` implemented
- [ ] `ui/jobs_page.py` implemented
- [ ] `ui/__init__.py` explicit page registration implemented
- [ ] Worker startup/shutdown managed by lifespan
### Testing (MCP-compliant)
- [ ] Scaffold phase completed first for all Step 5 tests
- [ ] `--collect-only` passed on scaffolds
- [ ] Fill phase completed without renaming/re-nesting scaffolded tests
- [ ] Marker decisions documented (`unit` vs `integration`)
- [ ] Targeted tests passed
- [ ] Full suite passed
### Evidence
- [ ] Validation command outputs captured
- [ ] Files created/updated listed
- [ ] MCP prompt resources referenced in implementation notes
- [ ] Any residual risks/questions documented
---
docs/mvp/mvp-step6-results.md
+50
View File
@@ -0,0 +1,50 @@
Step 6 implemented ✅
### Files added
- `tests/integration/test_pipeline_flow.py`
- `tests/test_traceability.py`
### What was implemented
#### 1) End-to-end pipeline integration coverage
`tests/integration/test_pipeline_flow.py` now verifies:
- **Success path**
- upload creates queued job
- worker processes job
- job becomes `transcribed`
- transcript text is persisted
- error detail remains `None`
- **Failure path**
- upload creates queued job
- worker processing failure is handled
- job becomes `failed`
- transcript has `text=None`
- `error_detail` contains provider failure reason
#### 2) MVP requirement traceability check
`tests/test_traceability.py` adds a concrete map:
- REQ-0, REQ-1, REQ-2, REQ-3, REQ-4, REQ-5, REQ-6, REQ-8, REQ-12
- each REQ maps to one or more existing test files
- test asserts each mapped path exists (guards against drift)
---
### Validation run results
Executed and passed:
1. `uv run pytest --collect-only -q` ✅
2. `uv run pytest -m unit -q` ✅
3. `uv run pytest -m integration -q` ✅
4. `uv run pytest -m "not external" -q` ✅
5. `uv run pytest tests/integration/test_pipeline_flow.py -q` ✅
6. `uv run pytest tests/ui/test_upload_page.py -q` ✅
7. `uv run pytest tests/ui/test_jobs_page.py -q` ✅
8. `uv run pytest -q` ✅
All green.
(Only existing non-blocking FastAPI TestClient deprecation warning remains.)
docs/mvp/mvp-step6.md
+229
View File
@@ -0,0 +1,229 @@
## Step 6: Test and Verification Hardening (MVP closeout)
## Objective
Complete MVP verification by building a **requirements-traceable, deterministic test strategy** across unit/integration/external lanes, then enforcing stable validation commands and reporting.
This step finalizes the MVP implementation sequence from `docs/mvp.md` (Step 6 in the build order: tests and automated verification).
---
## MCP Resource Integration (what was applied)
I reviewed all top-level skills/prompts from `john-stream-mcp` and integrated the relevant guidance into this plan:
### Directly applied
- `resource://skills/pytesting/document`
- `resource://catalog/prompts/pytest-scaffold`
- `resource://prompts/pytest-scaffold/document`
- `resource://catalog/prompts/pytest-fill-scaffold`
- `resource://prompts/pytest-fill-scaffold/document`
- `resource://skills/nicegui/document`
- `resource://skills/nicegui-ui-customization/document`
- `resource://skills/fastapi-uv-docker/document`
- `resource://skills/python-logging-dictconfig/document`
- `resource://skills/python-typing/document`
- `resource://skills/ruff-linting-formating/document`
### Reviewed but informational/non-blocking for Step 6
- `copilot-customization`, `mcp-details`, `vscode-configuration`, `zensical-docs`, and authoring/shim prompts.
- These are primarily customization/documentation tooling resources, not core MVP test-lane blockers.
- Step 6 includes optional workflow follow-ups where relevant (e.g., VS Code task conveniences).
---
## Scope
### In scope
- Strengthen and complete test coverage for the shipped MVP slice (Steps 1–5)
- Add requirement-to-test traceability for REQ-0..REQ-12 (MVP subset emphasized)
- Enforce deterministic default lanes (`unit`, `integration`)
- Keep `external` lane opt-in and isolated
- Validate app/UI/service/worker contracts end-to-end at test level
### Out of scope
- Major architecture rewrites (async SQLAlchemy migration, queue system, etc.)
- Full production deployment rollout
- Post-MVP feature expansion (revision history, search, export)
---
## Planned Deliverables
### Test files (new/updated)
- `tests/test_traceability.py` *(or docs-based traceability matrix if preferred)*
- `tests/integration/test_pipeline_flow.py` *(upload -> queued -> worker -> transcript/failed)*
- `tests/ui/test_upload_page.py` (augment loading/error/ready-state checks as practical)
- `tests/ui/test_jobs_page.py` (augment refresh/error behavior checks as practical)
- Existing tests touched only when needed; preserve naming/hierarchy unless explicitly approved.
### Optional docs output
- `docs/tests.md` or `docs/verification.md` with lane definitions and command matrix
- REQ-to-test mapping table
---
## Design and Policy Decisions (MCP-aligned)
1. **Scaffold-first, fill-second workflow is mandatory**
- First create/adjust skeletons and collect.
- Then fill test bodies.
- Preserve scaffold names/docstrings during fill.
2. **Deterministic-first default lanes**
- `unit` and `integration` run by default.
- `external` remains explicit opt-in.
3. **One behavior target per test**
- Short, behavior-focused names.
- Precise assertions on observable outcomes.
4. **Test double discipline (from pytesting skill)**
- Prefer real-input/real-object paths first.
- If monkeypatch/mocks/fakes are needed for a boundary, keep narrowly scoped.
- Avoid call-only assertions.
5. **NiceGUI responsiveness expectations**
- Verify loading/success/error state transitions where testable.
- Ensure user-facing feedback behavior is covered.
6. **FastAPI/ops baseline checks**
- Keep `/healthz` route validation in default lanes.
- Keep startup/shutdown lifecycle assertions present.
---
## Implementation Plan + Checklist
## Phase A — Coverage and traceability audit
- [ ] Build a REQ-to-test matrix for MVP requirements:
- [ ] REQ-0, REQ-1, REQ-2, REQ-3, REQ-4, REQ-5, REQ-6, REQ-8, REQ-12
- [ ] Identify weak spots:
- [ ] full pipeline integration (service + worker + persistence)
- [ ] UI state transition assertions (loading/error/ready)
- [ ] failure-path persistence verification robustness
- [ ] Record current baseline command results before edits
## Phase B — Scaffold phase (pytest-scaffold resources)
Target modules/areas:
- pipeline integration flow
- UI behavior augmentations
- traceability checks/document validators (if test-backed)
- [ ] Scaffold new/adjusted test files/classes/methods only
- [ ] Keep one-line intent docstrings
- [ ] Keep behavior-focused names
- [ ] Run: `uv run pytest --collect-only -q`
## Phase C — Fill phase (pytest-fill-scaffold resources)
- [ ] Fill scaffolded methods with deterministic setup/assertions
- [ ] Preserve scaffold names/hierarchy/docstrings
- [ ] Add/adjust fixtures at nearest useful scope
- [ ] Keep DB tests in `integration`; pure helper tests in `unit`
### Required coverage additions
#### Pipeline integration
- [ ] Upload service creates document/job and file path persists
- [ ] Worker success path creates transcript and terminal status
- [ ] Worker failure path persists error detail and terminal failed status
- [ ] Queue-empty behavior remains stable (`False` return / no side effects)
#### UI behavior (practical, testable boundaries)
- [ ] Upload helper flow success and UploadError surfacing
- [ ] Jobs data helpers return stable normalized view models
- [ ] Refresh/detail fallback behavior for missing/invalid job IDs
#### Traceability
- [ ] Every in-scope MVP REQ has at least one mapped test/assertion point
- [ ] Document and/or enforce mapping consistency
## Phase D — External lane stability
- [ ] Keep real-image external tests isolated under `@pytest.mark.external`
- [ ] Ensure no external test leaks into default runs
- [ ] Confirm artifact capture behavior remains stable
## Phase E — Quality gates and workflow
- [ ] Confirm logging/lifecycle startup tests still pass after changes
- [ ] (If enabled) add/update lint/type check commands in docs:
- [ ] Ruff lane (if configured)
- [ ] typing lane (if configured)
- [ ] Optionally add VS Code task aliases for test lanes (non-blocking)
---
## Marker and Fixture Strategy
- `unit`: pure logic, helper behavior, formatting/normalization
- `integration`: DB + service + app lifecycle contracts
- `external`: live provider/real image checks only
Fixture policy:
- Prefer reusable fixtures in `tests/conftest.py` only when broadly shared
- Use subtree/local fixtures for domain-specific setup
- Keep setup explicit and readable
---
## Validation Sequence (strict)
- [ ] `uv run pytest --collect-only -q`
- [ ] `uv run pytest -m unit -q`
- [ ] `uv run pytest -m integration -q`
- [ ] `uv run pytest -m "not external" -q`
- [ ] `uv run pytest tests/integration/test_pipeline_flow.py -q` *(if added)*
- [ ] `uv run pytest tests/ui/test_upload_page.py -q`
- [ ] `uv run pytest tests/ui/test_jobs_page.py -q`
- [ ] `uv run pytest -q`
Optional external verification:
- [ ] `uv run pytest -m external -q`
---
## Guardrails
- Do not rename/re-nest scaffolded tests during fill unless explicitly requested.
- Do not broaden external dependencies in default lane.
- Do not add flaky timing-based assertions; keep deterministic boundaries.
- Keep business logic out of UI tests; test through service/helper boundaries.
- Preserve one-way dependency direction in test setup patterns.
---
## Definition of Done (Step 6)
- [ ] MVP requirement coverage is explicitly traceable
- [ ] Deterministic lanes (`unit` + `integration`) are stable and green
- [ ] External lane remains opt-in and green when enabled
- [ ] Pipeline success/failure lifecycle paths are verified end-to-end
- [ ] UI helper/state behavior has explicit success/error assertions
- [ ] Full suite passes with `uv run pytest -q`
- [ ] Verification evidence is captured in implementation report
---
## PR Checklist (Step 6)
### Implementation
- [ ] Added/updated test files per scoped gaps
- [ ] Added REQ traceability mapping
- [ ] Kept default lanes deterministic
- [ ] Preserved scaffold invariants during fill
### Testing (MCP-compliant)
- [ ] Used scaffold prompt flow first
- [ ] Used fill prompt flow second
- [ ] Preserved naming/docstrings/hierarchy
- [ ] Marker usage documented (`unit`, `integration`, `external`)
### Evidence
- [ ] Collected command outputs in strict order
- [ ] Listed files changed
- [ ] Listed MCP resources used and why
- [ ] Noted residual risks/open questions (if any)
docs/mvp/mvp-step7-results.md
+134
View File
@@ -0,0 +1,134 @@
## Step 7 Results: Error Handling Standardization and Operational Visibility
## Summary
Step 7 was implemented across the MVP runtime boundaries with a shared error taxonomy, actionable UI error surfacing, worker failure normalization, and API error envelope handling.
All required validation gates in `docs/step7.md` were executed and passed.
---
## Scope Delivered
### Implemented
- Shared application error contract and taxonomy
- Service-layer error normalization (upload + transcription)
- UI error presentation helpers with suggested actions and error references
- Worker failure persistence format with category/suggestion/error_id markers
- API exception handlers for structured error responses
- Targeted tests for new error contract behavior
### Not implemented in this step
- External lane execution (`-m external`) was not required for Step 7 completion and was not run in this pass.
---
## Files Added
- `src/transcription/errors.py`
- `src/transcription/api/errors.py`
- `src/transcription/ui/error_presenter.py`
- `tests/test_errors.py`
- `tests/api/test_error_responses.py`
- `docs/step7.md`
## Files Updated
- `src/transcription/app.py`
- `src/transcription/services/upload.py`
- `src/transcription/services/transcription.py`
- `src/transcription/ui/upload_page.py`
- `src/transcription/ui/jobs_page.py`
- `src/transcription/worker.py`
- `tests/services/test_upload.py`
- `tests/services/test_transcription.py`
- `tests/services/test_worker.py`
- `tests/integration/test_pipeline_flow.py`
- `uv.lock`
---
## Implementation Notes by Phase
### Phase A/B (Foundation)
- Added `ErrorCategory` enum and `AppError` base type in `src/transcription/errors.py`.
- Added helper utilities:
- `new_error_id()`
- `build_error_envelope(...)`
- `classify_unexpected_error(...)`
- `format_error_detail(...)`
### Phase C (Service/Provider normalization)
- `UploadError` now extends `AppError` and includes category/suggestion/retriable metadata.
- `PromptLoadError` and `TranscriptionError` now extend `AppError`.
- Provider failures are mapped with deterministic category semantics (auth/payload/provider-failure cases).
### Phase D (UI visibility)
- Added `src/transcription/ui/error_presenter.py`.
- Upload and jobs pages now use centralized UI error rendering and summary helpers.
- UI error paths now include more visible/actionable guidance and reference IDs.
### Phase E (Worker failure handling)
- Worker now normalizes exception handling into structured persisted `error_detail` strings with:
- category marker
- suggestion marker
- error_id marker
- Logging now includes category/error_id context in failure paths.
### Phase F (API envelope)
- Added `src/transcription/api/errors.py` and registered handlers in app factory.
- AppError and unexpected exceptions now serialize to stable API envelopes with mapped status codes.
---
## Validation Commands and Outcomes
All commands were executed with `uv run python -m pytest ...` and completed successfully.
1. `uv run python -m pytest tests/test_errors.py -q` ✅
2. `uv run python -m pytest tests/services/test_upload.py -q` ✅
3. `uv run python -m pytest tests/services/test_transcription.py -q` ✅
4. `uv run python -m pytest tests/providers/test_openrouter.py -q` ✅
5. `uv run python -m pytest tests/services/test_worker.py -q` ✅
6. `uv run python -m pytest tests/integration/test_pipeline_flow.py -q` ✅
7. `uv run python -m pytest tests/api/test_error_responses.py -q` ✅
8. `uv run python -m pytest tests/ui/test_upload_page.py -q` ✅
9. `uv run python -m pytest tests/ui/test_jobs_page.py -q` ✅
10. `uv run python -m pytest -m "not external" -q` ✅
11. `uv run python -m pytest --collect-only -q` ✅
12. `uv run python -m pytest -m unit -q` ✅
13. `uv run python -m pytest -m integration -q` ✅
14. `uv run python -m pytest tests/integration/test_pipeline_flow.py -q` ✅
15. `uv run python -m pytest tests/ui/test_upload_page.py -q` ✅
16. `uv run python -m pytest tests/ui/test_jobs_page.py -q` ✅
17. `uv run python -m pytest -q` ✅
Observed warning (non-blocking): Starlette/FastAPI TestClient deprecation warning related to `httpx` package naming.
---
## Policy Alignment Check (`docs/error_handling.md`)
Aligned items:
- Stable taxonomy categories are implemented.
- Unexpected errors are normalized.
- User-facing UI paths include actionable guidance and references.
- Worker persistence includes trace-friendly failure detail.
- API error responses are structured and category-aware.
Follow-up candidates:
- Add richer UI tests that validate rendered suggested-action content end-to-end (current tests focus helper/service contracts).
- Consider typed storage fields for error metadata instead of packed `error_detail` strings in a future schema revision.
---
## Step 7 Definition of Done Status
- [x] Shared error taxonomy implemented across MVP layers
- [x] GUI error paths upgraded for visibility/actionability
- [x] Worker failure persistence and log context standardized
- [x] API error envelope handling added and tested
- [x] Phase-level and full-suite validation gates passed
- [x] Results documented in this report
Step 7 is complete.
docs/mvp/mvp-step7.md
+267
View File
@@ -0,0 +1,267 @@
## Step 7: Error Handling Standardization and Operational Visibility
## Objective
Apply the canonical error policy from `docs/error_handling.md` to the MVP implementation so failures are:
- consistently classified
- visibly surfaced in the GUI
- paired with suggested corrective actions
- traceable through logs via error reference IDs
- validated through deterministic tests after each phase
This step extends MVP hardening by converting current ad hoc exception behavior into a stable cross-layer contract.
---
## Scope
### In scope
- Introduce a shared application error contract and taxonomy implementation
- Normalize service/provider exceptions into taxonomy categories
- Improve GUI error visibility and suggested-action UX
- Standardize worker failure persistence and logging context
- Add API error-envelope policy hooks for current/future endpoints
- Add targeted tests and phase-level/full-suite validation gates
### Out of scope
- Major architecture rewrites (distributed queue, multi-service decomposition)
- Post-MVP feature expansion unrelated to error handling
- Full observability platform rollout (tracing backends, APM)
---
## Policy Source of Truth
- Canonical policy document: `docs/error_handling.md`
- If implementation and policy diverge, policy is authoritative and code/tests must be updated.
---
## Planned Deliverables
### Runtime code
- `src/transcription/errors.py` *(new shared contract module)*
- `src/transcription/ui/error_presenter.py` *(new UI error rendering helper)*
- Updates to:
- `src/transcription/services/upload.py`
- `src/transcription/services/transcription.py`
- `src/transcription/providers/openrouter.py`
- `src/transcription/worker.py`
- `src/transcription/ui/upload_page.py`
- `src/transcription/ui/jobs_page.py`
- `src/transcription/api/*` *(as needed for envelope/handlers)*
### Tests
- `tests/test_errors.py` *(new shared error contract tests)*
- updates/additions in:
- `tests/services/test_upload.py`
- `tests/services/test_transcription.py` *(add if missing)*
- `tests/providers/test_openrouter.py`
- `tests/services/test_worker.py`
- `tests/ui/test_upload_page.py`
- `tests/ui/test_jobs_page.py`
- `tests/api/test_error_responses.py` *(new, if API handlers added)*
### Documentation
- Update `docs/error_handling.md` only if implementation reveals policy gaps
- Capture validation evidence in a Step 7 results artifact (`docs/step7-results.md`)
---
## Design and Policy Decisions
1. **Stable taxonomy contract**
- Use policy categories as stable identifiers (`validation_error`, `user_input_error`, etc.).
2. **Actionable UX is mandatory**
- User-visible errors must include a suggested course of action.
3. **Traceability by default**
- Non-trivial errors include an `error_id` in both logs and user-facing output.
4. **Safe surface / rich logs**
- UI/API show safe summaries; logs retain diagnostic detail and traceback.
5. **Deterministic verification cadence**
- Targeted tests after each change batch, then phase-level regression gates.
---
## Implementation Plan + Checklist
## Phase A — Baseline Validation and Gap Confirmation
- [ ] Run baseline tests before changes
- [ ] Record baseline outputs and any known flaky behavior
- [ ] Confirm current behavior against `docs/error_handling.md` requirements
### Validation gate
- [ ] `uv run pytest -m "not external" -q`
- [ ] `uv run pytest -q`
## Phase B — Shared Error Contract Foundation
- [ ] Add `src/transcription/errors.py` with:
- [ ] stable category enum
- [ ] base `AppError` (category/message/suggestion/error_id/retriable)
- [ ] helpers for error-id generation and fallback classification
- [ ] Keep category names aligned with `docs/error_handling.md`
### Tests
- [ ] Add `tests/test_errors.py`
- [ ] category stability assertions
- [ ] error_id creation behavior
- [ ] fallback classification for unexpected exceptions
### Validation gate
- [ ] `uv run pytest tests/test_errors.py -q`
- [ ] `uv run pytest -m "not external" -q`
## Phase C — Service and Provider Normalization
- [ ] Refactor upload service exceptions to shared taxonomy
- [ ] Refactor transcription service exceptions to shared taxonomy
- [ ] Normalize provider adapter failures into deterministic categories
- [ ] Preserve causal chaining (`raise ... from exc`)
### Tests
- [ ] Extend `tests/services/test_upload.py`:
- [ ] empty payload category/suggestion
- [ ] unsupported extension category/suggestion
- [ ] persistence failure category mapping
- [ ] Add/extend `tests/services/test_transcription.py`:
- [ ] missing/empty prompt behavior
- [ ] unsupported file type behavior
- [ ] provider failure mapping behavior
- [ ] Extend `tests/providers/test_openrouter.py`:
- [ ] auth error mapping
- [ ] malformed response mapping
### Validation gate
- [ ] `uv run pytest tests/services/test_upload.py -q`
- [ ] `uv run pytest tests/services/test_transcription.py -q`
- [ ] `uv run pytest tests/providers/test_openrouter.py -q`
- [ ] `uv run pytest -m "not external" -q`
## Phase D — GUI Visibility and Suggested Actions
- [ ] Add `src/transcription/ui/error_presenter.py`
- [ ] Update upload/jobs pages to use centralized error presentation
- [ ] Ensure GUI surfaces:
- [ ] user-safe message
- [ ] suggested action
- [ ] error reference ID
- [ ] optional technical details panel
- [ ] Replace raw `str(exc)` UX where policy requires safer messaging
### Tests
- [ ] Extend `tests/ui/test_upload_page.py` for actionable error UX paths
- [ ] Extend `tests/ui/test_jobs_page.py` for refresh/detail error guidance
- [ ] Add `tests/ui/test_error_presenter.py` *(optional but recommended)*
### Validation gate
- [ ] `uv run pytest tests/ui/test_upload_page.py -q`
- [ ] `uv run pytest tests/ui/test_jobs_page.py -q`
- [ ] `uv run pytest -m "not external" -q`
## Phase E — Worker Failure Persistence and Logging Context
- [ ] Update worker failure handling to classify errors before persistence
- [ ] Ensure failed jobs persist actionable, structured error detail
- [ ] Add log context fields where available (`error_id`, `category`, `operation`, `job_id`)
- [ ] Ensure retry semantics are explicit and bounded (or clearly documented as deferred)
### Tests
- [ ] Extend `tests/services/test_worker.py`:
- [ ] missing document failure contract
- [ ] provider/transcription failure contract
- [ ] persisted error detail includes category/suggestion/error_id markers
- [ ] Validate integration failure flow in `tests/integration/test_pipeline_flow.py`
### Validation gate
- [ ] `uv run pytest tests/services/test_worker.py -q`
- [ ] `uv run pytest tests/integration/test_pipeline_flow.py -q`
- [ ] `uv run pytest -m "not external" -q`
## Phase F — API Error Envelope Alignment (Current + Future Routes)
- [ ] Add shared API error serialization utilities/handlers (as needed)
- [ ] Ensure API responses can include:
- [ ] `error_id`
- [ ] `category`
- [ ] `message`
- [ ] `suggestion`
- [ ] `timestamp`
- [ ] Map categories to HTTP status guidance from `docs/error_handling.md`
### Tests
- [ ] Add `tests/api/test_error_responses.py` *(if handlers added)*
- [ ] Keep `tests/api/test_health.py` passing
### Validation gate
- [ ] `uv run pytest tests/api/test_error_responses.py -q` *(if added)*
- [ ] `uv run pytest tests/api/test_health.py -q`
- [ ] `uv run pytest -m "not external" -q`
## Phase G — Final Regression and Documentation Closure
- [ ] Reconcile implementation details with `docs/error_handling.md`
- [ ] Update policy doc only where required by confirmed implementation learning
- [ ] Capture execution evidence in `docs/step7-results.md`
### Final validation sequence (strict)
- [ ] `uv run pytest --collect-only -q`
- [ ] `uv run pytest -m unit -q`
- [ ] `uv run pytest -m integration -q`
- [ ] `uv run pytest -m "not external" -q`
- [ ] `uv run pytest tests/integration/test_pipeline_flow.py -q`
- [ ] `uv run pytest tests/ui/test_upload_page.py -q`
- [ ] `uv run pytest tests/ui/test_jobs_page.py -q`
- [ ] `uv run pytest -q`
Optional:
- [ ] `uv run pytest -m external -q`
---
## Guardrails
- Do not weaken user-facing clarity to expose raw internals.
- Do not introduce silent exception swallowing.
- Do not break category-name stability without policy update.
- Do not merge phase changes without passing that phase validation gate.
- Keep targeted tests fast and deterministic; isolate external-provider tests under `external`.
---
## Definition of Done (Step 7)
- [ ] Shared error taxonomy is implemented and used across MVP layers
- [ ] GUI error experiences are visible, actionable, and traceable
- [ ] Worker persists and logs failure context consistently
- [ ] API error contract path is aligned for current/future endpoints
- [ ] Phase-by-phase test gates pass
- [ ] Full suite remains green (`uv run pytest -q`)
- [ ] Step 7 results are documented with evidence
---
## PR Checklist (Step 7)
### Implementation
- [ ] Added shared error contract module
- [ ] Updated service/provider/worker/UI error handling paths
- [ ] Added actionable GUI guidance for user-visible failures
- [ ] Added error reference IDs for traceability
### Testing
- [ ] Added/updated tests per phase scope
- [ ] Ran targeted phase tests after each change batch
- [ ] Ran `not external` regression at each phase boundary
- [ ] Ran full suite before closeout
### Documentation and Evidence
- [ ] `docs/error_handling.md` reviewed for alignment
- [ ] `docs/step7-results.md` includes executed command outputs
- [ ] Residual risks and deferred items explicitly recorded
docs/mvp/mvp.md
+209
View File
@@ -0,0 +1,209 @@
## MVP Definition: Historical Document Transcription System
### 1. MVP Objective
Deliver the thinnest possible end-to-end vertical slice — a user uploads an image of a document, the system transcribes it via the OpenRouter Python SDK, and the user reads the resulting transcript — with just enough persistence and structure to validate the core value proposition: *can AI-driven transcription, guided by curated prompts, produce useful verbatim transcripts of historical family documents?*
The MVP deliberately defers full-text search, export, revision history, MongoDB, and timeline assembly. These are additive features that don't need validation before the core transcription loop is proven.
---
### 2. Core User Story
*As a family historian, I can upload a photo of a historical document, wait for it to be transcribed, and read the verbatim transcript — so I can evaluate whether this system will work for my thousands of documents.*
---
### 3. In-Scope Requirements (from ```requirements.md```)
| Requirement | ID | MVP Rationale |
| --- | --- | --- |
| End-to-end transcription with lifecycle state | REQ-0 | This is the MVP. |
| Upload one or more images from the web UI | REQ-1 | Core entry point. MVP supports single-image upload (multi-image is a stretch goal). |
| Asynchronous processing → transcription or failure | REQ-2 | Validates the AI transcription pipeline. |
| Persist and expose job states (queued → processing → transcribed/failed) | REQ-3 | Minimum feedback loop for the user. |
| Persist transcription output and failure details | REQ-4 | User must be able to read the result. |
| UI views for status and transcript reading | REQ-5 | The user needs to see what happened. |
| Background processing to keep UI responsive | REQ-6 | Essential for usability during long AI calls. |
| Centralized config and logging at startup | REQ-8 | Small effort, high payoff for debugging. |
| Store transcription prompts as Markdown files | REQ-12 | Core to the Prompt Curation Policy in intent.md. Start with a single prompt file. |
### Deferred to Post-MVP
| Requirement | ID | Why Deferred |
| --- | --- | --- |
| Lifespan-owned runtime resources (engine, session factory, etc.) | REQ-7 | Important for production robustness, but a simple global or module-level setup is adequate for MVP validation. |
| Docker Compose (app + PostgreSQL + optional MongoDB) | REQ-9 | MVP runs locally with SQLite to eliminate container overhead during rapid iteration. PostgreSQL migration is Stage 1 hardening. |
| Explicit, opt-in schema bootstrap | REQ-10 | MVP uses auto-create-tables at startup (SQLModel create_all). Production schema discipline comes after the model stabilizes. |
| Service-backed persistence for core data | REQ-11 | MVP uses a thin repository layer over SQLite. Full service abstraction follows once the domain model is proven. |
---
### 4. MVP Feature Set
#### Feature 1: Document Upload (UI)
* A single NiceGUI page with a file-upload widget (accepts .jpg, .png, .tiff, .pdf).
* On upload: save the file to a local uploads/ directory, create a Document record, create a Job record with status queued.
* Minimal metadata capture: original filename, upload timestamp.
#### Feature 2: Asynchronous Transcription Worker
* An in-process background worker (Python asyncio task or BackgroundTasks) that:
1. Picks up queued jobs.
2. Transitions status to processing.
3. Sends the image + the curated Markdown prompt to an AI vision model via OpenRouter.
4. On success: saves the transcript text, transitions to transcribed.
5. On failure: saves the error detail, transitions to failed.
#### Feature 3: Transcription Prompt (Markdown Asset)
* A single Markdown file (prompts/transcribe_document.md) encoding the verbatim transcription rules from intent.md (the Document Issues table, scholarly guidelines, etc.).
* The worker reads this file at invocation time and injects it as the system/user prompt.
#### Feature 4: Job Status & Transcript Viewer (UI)
* A job list page showing all jobs with their current status (queued / processing / transcribed / failed).
* A transcript detail page showing:
* The original uploaded image (rendered inline).
* The transcription text (or the failure reason).
* Timestamp metadata.
#### Feature 5: Minimal Persistence (SQLite + SQLModel)
* Three tables/models:
* Document: id, filename, file_path, uploaded_at.
* Job: id, document_id (FK), status, created_at, updated_at.
* Transcript: id, job_id (FK), text, error_detail, created_at.
* SQLite database file stored locally. Auto-created on first startup.
#### Feature 6: Centralized Configuration
* A single config.py (or Pydantic BaseSettings) loading:
* PROVIDER (fixed to openrouter for MVP)
* OPENROUTER_API_KEY (required)
* PROVIDER_MODEL (default: OpenRouter model slug for vision transcription)
* OPENROUTER_HTTP_REFERER (optional; app attribution)
* OPENROUTER_APP_TITLE (optional; app attribution)
* DATABASE_URL (default: sqlite:///./transcription.db)
* UPLOAD_DIR (default: ./uploads)
* PROMPT_DIR (default: ./prompts)
#### Feature 7: MVP Dependency Baseline (OpenRouter-Centric)
* Runtime dependencies:
* openrouter (official OpenRouter Python SDK)
* pydantic
* pydantic-settings
* sqlmodel
* Explicitly out of MVP runtime dependencies:
* google-genai (deferred until/if Gemini is introduced post-MVP)
---
### 5. MVP Architecture (Simplified)
```Apply
┌─────────────────────────────────────────────┐
│ NiceGUI Web UI │
│ ┌──────────────┐ ┌───────────────────┐ │
│ │ Upload Page │ │ Jobs / Transcript │ │
│ └──────┬───────┘ └───────┬───────────┘ │
│ │ │ │
│ ▼ ▼ │
│ ┌───────────────────────────┐ │
│ │ Application Service │ │
│ │ (upload, job lifecycle) │ │
│ └─────┬─────────────┬───────┘ │
│ │ │ │
│ ┌─────▼─────┐ ┌─────▼───────────────┐ │
│ │ SQLite DB │ │ Background Worker │ │
│ │ (SQLModel)│ │ → AI Vision Provider│ │
│ └───────────┘ └─────────────────────┘ │
│ │ │
│ ┌─────▼──────┐ │
│ │ prompts/ │ │
│ │ *.md files │ │
│ └────────────┘ │
└─────────────────────────────────────────────┘
```
---
#### 6. Proposed File Structure
```Apply
project-root/
├── docs/ # (existing)
├── prompts/
│ └── transcribe_document.md # curated transcription prompt
├── src/
│ └── transcription/
│ ├── __init__.py
│ ├── app.py # FastAPI + NiceGUI app entrypoint
│ ├── config.py # Pydantic BaseSettings
│ ├── models.py # SQLModel: Document, Job, Transcript
│ ├── db.py # engine, session, create_all
│ ├── providers/
│ │ ├── __init__.py
│ │ ├── base.py # provider interface (transcribe contract)
│ │ ├── openrouter.py # OpenRouter via official Python SDK
│ ├── services/
│ │ ├── __init__.py
│ │ ├── upload.py # save file + create records
│ │ └── transcription.py # call provider, update job
│ ├── worker.py # background job loop
│ └── ui/
│ ├── __init__.py
│ ├── upload_page.py # NiceGUI upload page
│ └── jobs_page.py # NiceGUI job list + detail
├── tests/
│ ├── test_models.py
│ ├── test_upload.py
│ └── test_transcription.py
├── pyproject.toml
└── README.md
```
---
#### 7. MVP Validation Criteria
The MVP is considered validated when:
1. ✅ A user can upload an image of a document through the browser.
2. ✅ The system asynchronously sends the image to the configured AI vision model with the curated prompt.
3. ✅ The transcript (or failure reason) is persisted and visible in the UI.
4. ✅ The transcription follows verbatim scholarly rules defined in intent.md (spot-checked by the user on real family documents).
5. ✅ The transcription prompt is stored as a standalone Markdown file and can be edited without code changes.
6. ✅ Job status transitions are visible: queued → processing → transcribed/failed.
---
### 8. Key Feedback Questions the MVP Should Answer
These are the real unknowns this MVP exists to resolve:
| # | Question | How We Learn |
| --- | --- | --- |
| 1 | Is AI transcription quality good enough for this document corpus? | User reviews 20–50 real transcriptions against originals. |
| 2 | Does the verbatim prompt produce scholarly-quality output, or does it need major rework? | Compare output to the Document Issues table rules in intent.md. |
| 3 | What document types are hardest (old cursive, faded ink, pencil, postcards)? | Track which uploads produce failed or low-quality results. |
| 4 | Is single-image upload sufficient, or is batch upload needed early? | User friction during real scanning sessions. |
| 5 | What metadata is missing that the user wishes they could capture at upload time? | User feedback after processing real batches. |
---
#### 9. What Comes After MVP (Immediate Post-MVP)
Once the core transcription loop is validated, the next priorities (aligned to Architecture Stage 1) are:
1. **Multi-image upload** — process a batch from a scanning session.
2. **PostgreSQL migration** — swap SQLite for containerized PostgreSQL (REQ-9, REQ-10).
3. **Revision history** — allow the user to edit/correct transcripts with immutable version tracking.
4. **Full-text search** — search across all accepted transcripts.
5. **Repository/service layer formalization** — proper ports/adapters as the domain model stabilizes.
6. **Docker Compose deployment** — containerize the app for reproducible operation.
---
#### 10. Implementation Approach
Recommended build order for the MVP (each step produces a testable increment):
| Step | Deliverable | Validates |
| --- | --- | --- |
| 1 | config.py + models.py + db.py — data layer with SQLite | Schema and config foundation |
| 2 | prompts/transcribe_document.md — curated prompt from intent.md | Prompt asset pattern |
| 3 | services/transcription.py + providers/ — call AI vision provider with prompt + image | Core AI integration |
| 4 | services/upload.py + worker.py — upload handling + background job loop | End-to-end pipeline (CLI-testable) |
| 5 | ui/upload_page.py + ui/jobs_page.py — NiceGUI pages | User-facing interface |
| 6 | tests/ — unit + integration tests Automated verification |
This MVP is deliberately narrow: **one prompt, one provider (OpenRouter), one user, one image at a time, SQLite, no containers**. Every omission is intentional — the goal is to get real family documents through the transcription pipeline as fast as possible and let the quality of the output guide every subsequent decision.
docs/requirements.md
+38 -40
View File
@@ -1,14 +1,12 @@
## Document Transcription System Requirements (V1 Baseline)
## Document Transcription System Requirements
This page captures the **Version 1 baseline requirements** for the currently implemented system. It is the source of truth for V1 acceptance and test traceability.
Forward-looking architecture changes (for example PostgreSQL/Mongo adoption) are intentionally out of this document and should be tracked in a V2 planning/backlog artifact.
This page captures a SysML v1.6-style requirements baseline for the production system described in [index.md](index.md). The model is represented as concise tables and traceability lists that preserve SysML-style IDs and relationship semantics.
## Scope
- System of interest: a single Python application service (NiceGUI + FastAPI) with SQLModel persistence.
- Runtime/persistence baseline: local-first execution using SQLite (default `sqlite:///./transcription.db`), with Docker Compose support.
- Primary concern: end-to-end transcription lifecycle from upload through terminal state plus optional single revision editing.
- System of interest: the single Python application service (NiceGUI + FastAPI) with PostgreSQL as the relational system of record and optional MongoDB for document-oriented persistence.
- Operational context: local-first execution with Docker Compose and an intentionally lightweight production trajectory.
- Primary concern: end-to-end transcription job lifecycle from upload through completion or failure.
## Requirements Model (Concise Text Form)
@@ -17,23 +15,22 @@ Forward-looking architecture changes (for example PostgreSQL/Mongo adoption) are
| ID | Category | Requirement | Risk | Verify Method |
| --- | --- | --- | --- | --- |
| REQ-0 | System | Provide end-to-end document transcription with persistent, inspectable lifecycle state. | medium | demonstration |
| REQ-1 | Functional | Allow users to upload supported image/PDF files as sources from the web UI. | low | test |
| REQ-2 | Functional | Process uploads asynchronously and return either original transcription output or explicit failure. | high | test |
| REQ-3 | Functional | Persist and expose job states: `queued`, `processing`, `transcribed`, `failed`. | high | inspection |
| REQ-4 | Functional | Persist original provider output (`Job.text`) and failure detail (`Job.error_detail`) for each job. | medium | test |
| REQ-5 | Interface | Expose API/UI views for status inspection and transcription reading. | medium | demonstration |
| REQ-1 | Functional | Allow users to upload one or more document images from the web UI. | low | test |
| REQ-2 | Functional | Run each upload through asynchronous processing that returns a transcription or explicit failure. | high | test |
| REQ-3 | Functional | Persist and expose job states: upload, queued, processing, transcribed, failed, completed. | high | inspection |
| REQ-4 | Functional | Persist transcription output, processing history, and failure details. | medium | test |
| REQ-5 | Interface | Expose API and UI views for status inspection and completed transcription reading. | medium | demonstration |
| REQ-6 | Performance | Trigger background processing on upload to preserve UI responsiveness. | medium | analysis |
| REQ-7 | Design Constraint | Keep lifespan-owned runtime resources (engine/session factory/worker resources) initialized and disposed at application boundaries. | medium | inspection |
| REQ-7 | Design Constraint | Keep lifespan-owned runtime resources: SQLAlchemy engine, async session factory, worker resources, provider clients. | medium | inspection |
| REQ-8 | Design Constraint | Initialize configuration and logging once at startup through centralized mechanisms. | low | inspection |
| REQ-9 | Design Constraint | Support containerized app runtime via Docker Compose using the same V1 persistence model. | medium | demonstration |
| REQ-10 | Design Constraint | Keep schema bootstrap explicit and opt-in for production safety. | high | inspection |
| REQ-11 | Design Constraint | Route persistence changes through service/workflow orchestration boundaries. | medium | inspection |
| REQ-9 | Design Constraint | Use Docker Compose baseline of app plus PostgreSQL; allow optional MongoDB container when enabled. | medium | demonstration |
| REQ-10 | Design Constraint | Keep schema bootstrap explicit and opt-in; normal startup does not mutate production schema. | high | inspection |
| REQ-11 | Design Constraint | Use service-backed persistence for core document and job data. | medium | inspection |
| REQ-12 | Design Constraint | Store transcription prompts as individual Markdown artifacts for iterative refinement. | medium | inspection |
| REQ-13 | Functional | Allow users to create/update one optional revision derived from the original job transcription and view/delete it from the job detail flow. | low | test |
### Requirement Relationships
- Contains: REQ-0 contains REQ-1 through REQ-13.
- Contains: REQ-0 contains REQ-1 through REQ-12.
- Derives: REQ-2 -> REQ-3, REQ-3 -> REQ-4.
- Traces: REQ-5 -> REQ-3.
- Refines: REQ-6 -> REQ-2.
@@ -42,45 +39,46 @@ Forward-looking architecture changes (for example PostgreSQL/Mongo adoption) are
| Element | Type | Doc Reference |
| --- | --- | --- |
| UI | NiceGUI pages/components | `src/transcription/ui/pages`, `src/transcription/ui/components` |
| API | FastAPI routes and handlers | `src/transcription/api`, `src/transcription/app.py` |
| WORKER | Async queued-job processing workflow | `src/transcription/worker.py`, `src/transcription/services/workflows.py` |
| DBREL | SQLModel relational persistence (SQLite in V1 baseline) | `src/transcription/models.py`, `src/transcription/db` |
| SERVICES | Service-layer persistence orchestration | `src/transcription/services` |
| OPS | Containerized runtime baseline | `docker-compose.yml`, `Dockerfile` |
| PROMPTS | Transcription prompt artifacts | `prompts/` |
| TESTS | Pytest verification suite | `tests/` |
| UI | NiceGUI pages | src/transcription/ui/pages |
| API | FastAPI routes | src/transcription/api/routes.py |
| GRAPH | Async processing workflow | src/transcription/services, src/transcription/ai |
| DBREL | PostgreSQL + SQLModel relational persistence | src/transcription/db |
| DBDOC | MongoDB document persistence | src/transcription/db, src/transcription/services |
| OPS | Docker Compose runtime | docker-compose.yml |
| PROMPTS | Transcription prompt artifact library (Markdown files) | .github/prompts, docs |
| TESTS | Pytest verification suite | tests |
### Satisfaction Mapping
- UI satisfies REQ-1, REQ-5, REQ-13.
- UI satisfies REQ-1, REQ-5.
- API satisfies REQ-5.
- WORKER satisfies REQ-2, REQ-6.
- DBREL satisfies REQ-3, REQ-4, REQ-10, REQ-13.
- SERVICES satisfies REQ-4, REQ-11.
- GRAPH satisfies REQ-2, REQ-6.
- DBREL satisfies REQ-3, REQ-10.
- DBDOC satisfies REQ-4, REQ-11.
- OPS satisfies REQ-9.
- PROMPTS satisfies REQ-12.
### Verification Mapping
- TESTS verifies REQ-1, REQ-2, REQ-3, REQ-4, REQ-5, REQ-10, REQ-11, REQ-12, REQ-13.
- TESTS verifies REQ-1, REQ-2, REQ-3, REQ-4, REQ-5, REQ-10, REQ-11, REQ-12.
## Requirement Notes
- Requirement IDs (`REQ-*`) are stable references for planning, implementation, and traceability.
- This document is intentionally **implementation-aligned** for V1 completion and release sign-off.
- Planned storage evolution (PostgreSQL and optional MongoDB) is a **V2 concern** and should be tracked outside this V1 baseline.
- Requirement IDs (`REQ-*`) are stable references for planning, implementation, and test traceability.
- The model uses compact tables and traceability lists for renderer compatibility while preserving SysML-style requirement IDs and relationship semantics.
- Requirement categories (functional, interface, performance, and design constraints) are preserved as explicit REQ entries and relationship labels to keep change impact visible.
- PostgreSQL containerization and optional MongoDB containerization are both treated as extremely lightweight and simple operational choices in this architecture.
## Verification Intent
- Demonstration: validate end-to-end behavior through operator-visible flows.
- Demonstration: validate end-to-end behavior via running system flows and operator-visible outcomes.
- Inspection: verify architecture and startup/runtime policies in code and configuration.
- Analysis: evaluate asynchronous execution behavior and design sufficiency.
- Test: automate behavioral checks through pytest suites and service/UI integration tests.
- Test: automate behavioral checks through pytest suites and service-level tests.
## Glossary
- Original transcription: immutable provider output stored on `Job.text`.
- Revision: optional user-authored editable text tied to a `Source` (`0..1` in V1).
- Prompt artifact: a Markdown file containing instructions used for transcription.
- System of record: the authoritative relational store for canonical V1 data.
- Document-oriented persistence: A storage approach that uses flexible document structures for variable data shapes.
- Prompt artifact: A single Markdown file that defines one transcription prompt and is revised independently.
- SysML: Systems Modeling Language used to express structured requirements and traceability.
- System of record: The authoritative persistent store for canonical business data.
docs/runbook.md
-129
View File
@@ -1,129 +0,0 @@
# V1 Operations Runbook
This runbook provides day-2 operational procedures for the V1 baseline.
## Scope
Applies to:
- local/hosted V1 runtime
- SQLite-backed persistence
- in-process worker lifecycle
- OpenRouter provider integration
## Preconditions
- `.env` contains `OPENROUTER_API_KEY`
- app starts successfully
- `uploads/` and `prompts/` are writable
- health endpoint responds at `/healthz`
## Standard Startup Procedure
1. Start the app using the project-standard command.
2. Open `/healthz` and verify `{"status":"ok"}`.
3. Open `/ui/upload` and submit a small valid file.
4. Confirm job transitions from `queued` -> `processing` -> `transcribed` (or `failed` with detail).
## Standard Shutdown Procedure
1. Stop the application process.
2. Ensure no active process still holds the SQLite file.
3. If maintenance is planned, copy the DB file before edits:
- `transcription.db` (or configured `DATABASE_URL` file path)
## Incident: Jobs Stuck In `processing`
### Symptoms
- Jobs remain `processing` for longer than provider timeout
- New uploads queue but do not complete
- provider usage increases but no terminal job state is visible
### Checks
1. Confirm app process is still running.
2. Confirm worker loop is active (startup logs include worker lifespan start).
3. Inspect recent app logs for:
- `worker.process_job`
- `error_id`
- `category`
- `job_id` / `document_id` / `source_id`
4. Verify provider credentials and provider status.
### Recovery
1. Restart the app to trigger stale-processing recovery.
2. On startup, app re-queues stale processing jobs based on timeout policy.
3. Re-check jobs page and confirm terminal state progression.
4. If persistent, capture logs + error IDs and move to deep investigation.
## Incident: Provider Authentication Failures
### Symptoms
- failures categorized as provider/auth
- jobs fail quickly with authentication guidance
### Recovery
1. Validate `OPENROUTER_API_KEY` value.
2. Restart app after updating env.
3. Re-run a small transcription to confirm recovery.
## Incident: Upload Failures
### Symptoms
- UI reports upload errors
- unsupported extension or empty payload
### Recovery
1. Validate file extension (`.jpg`, `.jpeg`, `.png`, `.tif`, `.tiff`, `.pdf`).
2. Validate file is not empty.
3. Validate upload directory permissions.
4. Retry upload.
## Incident: Database File/Permission Issues
### Symptoms
- persistence errors during upload/job update
- startup failures around schema/runtime
### Recovery
1. Confirm the configured DB file path exists and is writable.
2. Confirm parent directory permissions.
3. Restore from last known backup copy if corruption is suspected.
4. Restart app and run smoke test.
## Logging Requirements (Operational)
Operational triage should always capture:
- `error_id`
- category
- operation name
- `job_id`, `document_id`, `source_id` when applicable
- UTC timestamp
## Escalation Packet (When opening an issue)
Include:
- exact timestamp window
- one failing `job_id`
- relevant `error_id` values
- latest 100 lines of app logs
- environment summary (`DATABASE_URL` type, app version/commit)
## Post-Incident Validation
After mitigation, verify:
1. Upload works.
2. One job reaches `transcribed`.
3. One induced failure reaches `failed` with error detail.
4. Jobs page and detail page render correctly.
docs/schema.md
-87
View File
@@ -1,87 +0,0 @@
## Database Schema (V1 Baseline)
This document describes the current relational schema for the transcription system.
All primary and foreign keys in the domain models are UUID-based in V1.
---
## Schema Diagram
```mermaid
erDiagram
DOCUMENT {
UUID id PK
TEXT name
}
JOB {
UUID id PK
UUID document_id FK
TEXT status
INTEGER retry_count
DATETIME date_created
DATETIME date_updated
TEXT provider
TEXT model
TEXT prompt_name
TEXT text
TEXT error_detail
}
SOURCE {
UUID id PK
UUID document_id FK
UUID job_id FK
TEXT upload_name
TEXT filename
TEXT file_path
DATETIME date_uploaded
}
REVISION {
UUID id PK
UUID source_id "FK, UK"
INTEGER revision
TEXT text
DATETIME date_created
}
DOCUMENT ||--o{ SOURCE : has_many
DOCUMENT ||--o{ JOB : has_many
JOB ||--o{ SOURCE : referenced_by
SOURCE ||--o| REVISION : has_optional_one
```
---
## Table Relationships and Constraints
- A `Document` can have zero or more `Source` records.
- A `Document` can have zero or more `Job` records.
- A `Source` belongs to exactly one `Document` and one `Job`.
- A `Source` may have one optional `Revision`.
- Optional `0..1` revision cardinality is enforced by uniqueness on `revision.source_id`.
### Invariants
- `Job.text` stores immutable original provider transcription output.
- `Revision` rows are optional user-authored edits derived from original transcription.
- Revisions do not overwrite original `Job.text`.
- Job status lifecycle values are: `queued`, `processing`, `transcribed`, `failed`.
### Timestamp Fields
- `Job.date_created`
- `Job.date_updated`
- `Source.date_uploaded`
- `Revision.date_created`
---
## Glossary
- **Document**: logical grouping for one or more transcribed sources.
- **Source**: uploaded file content (image/PDF) linked to a job.
- **Job**: processing record that stores lifecycle status and original output.
- **Revision**: optional single user-authored edited text linked to a source.
docs/transcription_methodology.md
-54
View File
@@ -1,54 +0,0 @@
# Transcription Methodology & Style Guide
## 1. Overview & Core Philosophy
This document defines the formal transcription standard for processing historical manuscripts, letters, diaries, and printed ephemera.
Following the principles established by Mary-Jo Kline in A Guide to Documentary Editing, this project adheres to a Strict Literal Transcription (Verbatim) model as its foundational layer. The primary goal is total textual fidelity—capturing what the author wrote, not what they intended to write—while ensuring the output remains machine-readable and indexable for downstream digital query and search systems.
## 2. Textual Policy
Transcribers (human or AI) must record the exact text of the source document without silent corrections, modernizations, or stylistic smoothing except where explicitly instructed in this guide.
* **Substantives:** Words, letter forms, structural layout, and semantic content must be recorded strictly as presented in the original document.
* **Accidentals:** Punctuation, capitalization, misspellings, and archaic character representations must be preserved unless an explicit rule below allows for standardization.
## 3. Standard Transcription Rules & Markup
The following rules map directly to editorial conventions for handling common manuscript anomalies and physical document features.
### 3.1 Textual Anomalies & Corrections
| Document Feature | Rule | Standard Markup Format | Output Example |
| --- | --- | --- | --- |
| **Misspellings & Errors** | Retain original spelling verbatim. Insert an italicized [sic] immediately following the error. Do not correct spelling silently. | [sic] | The weather was very cold and publick [sic] business delayed. |
| **Missing Words / Omissions** | Insert necessary words required to restore basic grammatical sense inside square brackets. | [word] | We went [to] the store to buy supplies. |
| **Uncertain / Conjectural** | Place best hypothesis followed by a question mark inside square brackets when handwriting is doubtful. | [word?] | He went to [Boston?] yesterday to meet the governor. |
| **Completely Illegible** | Use [illegible] for unreadable script. Use explicit damage descriptors when physical impairment prevents reading. | [illegible] or [reason] | The total cost was [illegible] dollars. or The letter ends here [remainder of page torn]. |
| **Canceled / Struck-through** | Wrap text removed by the author inside a [deleted: ...] tag to preserve authorial revisions. | [deleted: text] | We left at [deleted: noon] one o'clock instead. |
| **Interlineations / Additions** | Wrap text inserted above, below, or in margins into the narrative flow inside an [inserted: ...] tag. | [inserted: text] | The [inserted: red] house on the hill was abandoned. |
### 3.2 Typography, Characters & Layout
| Document Feature | Rule | Standard Markup Format | Output Example |
| --- | --- | --- | --- |
| **Superscripts & Abbreviations** | Bring raised letters down to the main line. Optionally expand abbreviations within square brackets based on project configuration. | [expanded] | Gen^l becomes Genl or Gen[era]l. |
| **Line-End Hyphenation** | Rejoin words split across a page or line boundary silently, dropping the soft hyphen. | Silently rejoin | Original: "estab- / lishment" becomes establishment |
| **Capitalization** | Preserve explicit capitalization. Default to modern capitalization rules only when authorial intent is ambiguous or archaic forms confuse sentence structure. | Literal / Contextual | If a standard noun like 'Farm' is clearly capitalized, record 'Farm'. If ambiguous, default to 'farm'. |
| **Hierarchical Outlines** | Preserve exact numbering characters (including lowercase Roman numerals or terminal 'j'). Replicate indentation levels using standard spacing. Do not correct sequence or mathematical errors. | Preserve syntax | I. Main Topic a. Sub-point b. Next pointIII. [sic] Third Topic |
### 3.3 Visual & Spatial Elements
| Document Feature | Rule | Standard Markup Format | Output Example |
| --- | --- | --- | --- |
| **Non-Textual Artifacts** | Record non-textual elements (seals, stamps, sketches, physical damage) using brief descriptive text inside square brackets. | [description] | [wax notary seal attached here] or [sketch of a fort layout] |
| **Marginalia & Addenda** | Explicitly indicate spatial transitions before transcribing content located in margins or non-standard orientations. | [location:] | [written in left margin:] Do not share this with anyone. |
## 4. Prompt Asset Integration
When executing programmatic transcriptions via LLM APIs or local models, processing instructions must be packaged into single-purpose system prompts aligned with these rules.
1. **Isolation:** Each transcription prompt file exists as an independent Markdown asset in the repository.
2. **Deterministic Output:** Prompts must explicitly instruct models to follow the markup standards in Section 3 without introducing conversational wrappers, extra prose, or structural markdown outside the source document's native layout.
3. **Iterative Scoping:** Rule modifications or edge-case additions must be submitted as isolated delta commits to individual prompt files to maintain clean revision tracking.
docs/ver1/migration_v1.md
-92
View File
@@ -1,92 +0,0 @@
# V1 Data Migration and Recovery Guidance
This document defines migration/backfill and rollback guidance for the V1 SQLite baseline.
## Purpose
- provide safe procedures for local schema evolution and recovery
- reduce data-loss risk during version upgrades
- establish repeatable pre-change and post-change checks
## Current Baseline
- canonical relational store: SQLite
- default DB path: `./transcription.db`
- schema bootstrap may apply compatibility updates for dev/test scenarios
## Pre-Change Checklist
Before changing runtime version or schema behavior:
1. Stop the app process.
2. Create a timestamped DB backup copy.
3. Capture current app commit/version.
4. Export a quick status inventory:
- job counts by status
- total documents/sources/revisions
5. Ensure sufficient disk space.
## Backup Procedure (SQLite)
Minimum procedure:
1. Stop app.
2. Copy DB file to a safe location with timestamp.
3. Store backup path in release notes or change log.
## Upgrade Procedure (V1)
1. Perform pre-change checklist.
2. Deploy updated app version.
3. Start app and observe startup logs.
4. Verify schema bootstrap completes (if enabled).
5. Run smoke flow:
- upload valid file
- observe terminal status
- open job detail
## Backfill Guidance
V1 backfill is limited and conservative:
- for records missing newly introduced non-null defaults, use explicit one-time SQL updates only after backup
- avoid destructive rewrites of `Job.text` or `Revision.text`
- never backfill by overwriting original immutable transcription output
## Rollback Procedure
If upgrade fails or causes data inconsistency:
1. Stop app.
2. Restore prior DB backup file.
3. Revert app version to last known-good commit.
4. Restart app.
5. Run smoke flow and confirm stability.
## Recovery Scenarios
### Stale processing jobs after crash/restart
- restart app and allow stale-job recovery to re-queue timed-out `processing` jobs
- monitor for terminal progression
### Schema mismatch symptoms
- errors during startup or writes indicating missing columns/indexes
- rollback to last good DB + app version
- reattempt with documented upgrade path
## Validation Evidence
For each upgrade rehearsal, capture:
- backup filename/path
- pre and post job status counts
- smoke test result
- rollback rehearsal result (recommended)
## Operational Constraints
- treat DB backups as required before non-trivial upgrades
- do not perform in-place DB edits while app is running
- do not skip post-upgrade smoke validation
docs/ver1/release_checklist_v1.md
-45
View File
@@ -1,45 +0,0 @@
# V1 Release Readiness Checklist
Use this checklist before declaring V1 operationally complete.
## A) Functional Readiness
- [ ] Upload flow works for supported file types.
- [ ] Worker transitions jobs through `queued -> processing -> transcribed|failed`.
- [ ] Job detail displays immutable original transcription from `Job.text`.
- [ ] Revision workflow supports create/update/view/delete for optional single revision.
## B) Reliability and Error Handling
- [ ] Error categories surface with actionable messages in UI/API pathways.
- [ ] Failed jobs persist `error_detail` and terminal state.
- [ ] Stale processing recovery verified on restart.
- [ ] Retry/timeout behavior validated against configured limits.
## C) Operational Readiness
- [ ] `docs/runbook.md` reviewed and current.
- [ ] `docs/migration_v1.md` reviewed and current.
- [ ] Backup and rollback procedures tested at least once.
- [ ] Incident escalation packet template is known to operators.
## D) Quality Gates
- [ ] Lint/type checks pass.
- [ ] `pytest -m "not external" -q` passes.
- [ ] Targeted external/provider checks executed (if credentials available).
- [ ] Release evidence recorded in `docs/release_evidence_v1.md`.
## E) Traceability and Documentation
- [ ] `docs/requirements.md` aligns with implemented V1 behavior.
- [ ] `docs/architecture.md`, `docs/schema.md`, and `docs/error_handling.md` are consistent.
- [ ] `docs/traceability_v1.md` is updated with current implementation and test evidence.
- [ ] `docs/ver1/ver1.md` phase status updated with evidence references.
- [ ] REQ traceability evidence links recorded (tests/runbook/checks).
## Release Sign-Off
- [ ] Technical sign-off complete.
- [ ] Operational sign-off complete.
- [ ] V1 completion date recorded.
docs/ver1/release_evidence_v1.md
-39
View File
@@ -1,39 +0,0 @@
# V1 Release Evidence Log
## Step 5 Quality Gates (2026-07-29)
### Lint
- Command: `python -m ruff check .`
- Result: ✅ pass
- Notes: initial findings were auto-fixed (`ruff --fix`) plus small manual line-wrap/annotation adjustments.
### Tests (primary gate)
- Command: `python -m pytest -m "not external" -q`
- Result: ✅ pass (`[100%]`)
### Tests (external smoke)
- Command: `python -m pytest -m external -q`
- Result: ✅ pass (`[100%]`)
### Type Check
- Command: `python -m ty check src tests`
- Result: ⚠️ not passing
- Summary: existing SQLModel/SQLAlchemy typing incompatibilities and test double typing mismatches remain.
Key current blocker families:
1. SQLModel relationship/query attribute typing (`selectinload`, `order_by`, `.any()`)
2. SQLAlchemy join clause typing in `services/transcription.py`
3. Test fake client type mismatch for `OpenRouterTranscriptionProvider(client=...)`
4. `Settings(**defaults)` typed-dict strictness in `tests/test_config.py`
## Current Gate Status
- Lint: pass
- Non-external tests: pass
- External smoke tests: pass
- Type check: **blocked** (requires dedicated typing cleanup pass)
docs/ver1/traceability_v1.md
-40
View File
@@ -1,40 +0,0 @@
# V1 Traceability Matrix
This matrix provides implementation and validation evidence for V1 requirements (`REQ-0` through `REQ-13`).
Status values:
- `done`: implemented and evidence recorded
- `in progress`: partially implemented or evidence incomplete
- `not started`: no implementation/evidence yet
## Requirement Evidence Table
| Requirement | Status | Implementation Evidence | Validation Evidence |
| --- | --- | --- | --- |
| REQ-0 | done | End-to-end upload + worker pipeline in `src/transcription/services/store.py`, `src/transcription/worker.py`, `src/transcription/services/workflows.py` | `tests/integration/test_pipeline_flow.py` |
| REQ-1 | done | Upload UI/page flow in `src/transcription/ui/pages/upload_page.py`, `src/transcription/ui/components/upload.py` | `tests/ui/test_upload_page.py`, `tests/integration/test_pipeline_flow.py` |
| REQ-2 | done | Async worker execution and provider call orchestration in `src/transcription/worker.py`, `src/transcription/services/workflows.py` | `tests/integration/test_pipeline_flow.py`, `tests/services/test_workflows_reliability.py` |
| REQ-3 | done | Job lifecycle state model + transitions in `src/transcription/models.py`, `src/transcription/services/jobs.py`, `src/transcription/services/workflows.py` | `tests/services/test_job_service.py`, `tests/ui/test_jobs_page.py` |
| REQ-4 | done | Persistence of original output and failure detail in `src/transcription/services/transcription.py`, `src/transcription/services/workflows.py` | `tests/integration/test_pipeline_flow.py`, `tests/services/test_workflows_reliability.py` |
| REQ-5 | done | Status/result inspection via UI pages and API health route in `src/transcription/ui/pages/jobs_page.py`, `src/transcription/api/health.py` | `tests/ui/test_jobs_page.py`, `tests/ui/test_pages_registration.py`, `tests/api/test_health.py` |
| REQ-6 | done | Background processing trigger/worker notifier and non-blocking workflow in `src/transcription/ui/components/upload.py`, `src/transcription/worker.py` | `tests/test_app.py`, `tests/services/test_workflows_reliability.py` |
| REQ-7 | done | Lifespan-owned runtime resources in `src/transcription/app.py`, `src/transcription/db/runtime.py` | `tests/test_app.py`, `tests/test_db.py` |
| REQ-8 | done | Centralized settings/logging initialization in `src/transcription/config.py`, `src/transcription/app.py` | `tests/test_config.py`, `tests/test_app.py` |
| REQ-9 | done | Containerized runtime baseline in `docker-compose.yml`, `Dockerfile` | `docs/release_checklist_v1.md` (Ops checklist), manual demonstration step |
| REQ-10 | done | Explicit schema bootstrap policy + runtime controls in `src/transcription/config.py`, `src/transcription/app.py`, `src/transcription/db/operations.py` | `tests/test_db.py`, `tests/test_config.py` |
| REQ-11 | done | Service/workflow persistence boundaries in `src/transcription/services/*.py`, `src/transcription/services/workflows.py` | `tests/services/test_job_service.py`, `tests/services/test_transcription_service.py` |
| REQ-12 | done | Prompt artifacts in `prompts/` and loading/validation in `src/transcription/services/transcription.py` | `tests/test_prompts.py` |
| REQ-13 | done | Optional single revision create/update/view/delete in `src/transcription/services/transcription.py`, `src/transcription/ui/pages/jobs_page.py` | `tests/services/test_transcription_service.py`, `tests/ui/test_jobs_page.py` |
## Operational Evidence (Step 3 Artifacts)
- Runbook: `docs/runbook.md`
- Migration/backfill/rollback guidance: `docs/migration_v1.md`
- Release readiness checklist: `docs/release_checklist_v1.md`
## Verification Cadence
- Per change: maintain `tests/test_traceability.py` mappings for touched requirements.
- Per milestone: update this table status and evidence links.
- Pre-release: confirm all rows are `done` and non-external suite is green.
docs/ver1/ver1.md
+248 -118
View File
@@ -1,190 +1,320 @@
# Version 1 Implementation Plan
This plan defines the path from current implementation to **Version 1 complete**, aligned to the updated domain model:
- `Document` groups one or more content `Source` records
- `Job` owns original immutable provider output (`text`) and processing metadata
- `Revision` stores optional user-authored edits linked to a `Source`
The objective is to complete V1 scope with production readiness while keeping non-V1 enhancements out of active delivery.
This plan defines the path from MVP to **Version 1 complete**.
The objective is to deliver the full scoped product with production readiness, while explicitly separating refinements/enhancements into a future document.
---
## V1 Completion Definition
## 0) Plan Governance & Scope Control (Foundation)
V1 is complete when all of the following are true:
**Goal:** Keep execution focused on V1 completion, not optimization/perfection.
1. **Functional complete**
- Upload, queue, processing, status display, and transcription result inspection work end-to-end.
- Optional revision workflow is implemented (create/view/update single revision).
2. **Data-model complete**
- Runtime behavior, persistence, and tests all align to `Document` / `Source` / `Job` / `Revision`.
3. **Operational complete**
- Error handling, logs, and runbooks support reliable operation.
4. **Documentation complete**
- Architecture, requirements, schema, error handling, and index are consistent and current.
---
## Phase 1 — Data Contract Stabilization (Schema-First)
**Goal:** Lock a single canonical contract before further feature work.
### Tasks
1. Confirm and document invariants:
- `Job.text` is original immutable transcription output.
- `Revision` is optional and user-authored.
- Revisions are derived from the original `Job.text`.
2. Verify relationship cardinality assumptions:
- `Document` -> many `Source`
- `Document` -> many `Job`
- `Source` -> one `Job`
- `Source` -> one `Revision`
3. Ensure field naming consistency (`date_created`, `date_updated`, `date_uploaded`) across code and docs.
4. Freeze V1 status lifecycle to current implementation (`queued`, `processing`, `transcribed`, `failed`).
### Implementation Steps
1. Create and maintain a **V1 Traceability Matrix**:
- Requirement ID
- Current status (`done`, `partial`, `not started`)
- Owner
- Validation method
2. Define V1 completion gates:
- Functional complete
- Operationally complete
- Production-ready complete
3. Snapshot the MVP baseline (tag/changelog reference).
4. Create a standing rule: any non-V1 idea is logged to a separate enhancements backlog document (to be named later), not added to active V1 scope unless explicitly approved.
### Deliverables
- Updated `docs/schema.md` and `docs/requirements.md` traceability alignment.
- Explicit V1 data invariants section in architecture docs.
- `docs/ver1/ver1.md` (this plan)
- V1 traceability artifact (linked from here when created)
### Exit Criteria
- No conflicting definitions of ownership/cardinality/status remain in docs.
- Every in-scope requirement has explicit ownership and status.
- Scope-change process is agreed and followed.
---
## Phase 2 — Service Layer Refactor To New Model
## 1) Architecture Consolidation
**Goal:** Remove all obsolete `Transcript` assumptions from service/workflow code.
**Goal:** Align implementation with the intended architecture and reduce MVP shortcuts.
### Tasks
1. Refactor `services/transcription.py`:
- Replace transcript CRUD assumptions with job-output + revision operations.
2. Refactor `services/jobs.py`:
- Replace old timestamp/relationship accessors with current model fields.
3. Refactor `services/documents.py` and `services/store.py`:
- Ensure upload creates and links `Document`, `Source`, and `Job` correctly.
4. Refactor `services/workflows.py`:
- Persist original provider output to `Job`.
- Persist failure detail to `Job.error_detail`.
- Use `Revision` only for user-authored edits.
### Implementation Steps
1. Compare implemented modules/components with architecture documentation.
2. Identify and classify architectural debt:
- Temporary coupling
- Missing interfaces
- Placeholder services/components
3. Resolve high-risk architectural gaps first.
4. Record key decisions and tradeoffs in ADRs.
### Deliverables
- Service layer fully aligned with new schema.
- Updated architecture diagrams and boundaries
- ADR entries for major decisions
### Exit Criteria
- No service module imports or persists `Transcript` model artifacts.
- Architecture documentation reflects system reality.
- Critical architecture risks are addressed or scheduled with owners/dates.
---
## Phase 3 — UI Contract Alignment
## 2) Error Handling & Reliability Hardening
**Goal:** Align pages/components to source/job/revision semantics.
**Goal:** Ensure predictable, safe behavior under failure conditions.
### Tasks
1. Update job detail and related UI components:
- Display original immutable transcription from `Job.text`.
- Display optional revision sourced from `Source.revision` (0 or 1).
2. Align date fields with new schema naming.
3. Preserve clear user messaging when no revisions exist.
### Implementation Steps
1. Standardize error taxonomy and envelope format across all layers.
2. Ensure clear distinction between:
- User-facing errors
- Internal/system errors
- Retryable vs non-retryable failures
3. Add resilience controls where needed:
- Timeouts
- Retries with backoff
- Circuit breaking / fallback logic
4. Add failure-path tests for critical workflows.
### Deliverables
- Updated jobs page and detail components.
- Error code catalog/reference
- Failure mode test coverage for critical paths
### Exit Criteria
- UI behavior and labels match documentation and domain model.
- Error behavior is consistent across major flows.
- Known failure scenarios are tested and pass.
---
## Phase 4 — Database Bootstrap, Migration, and Safety
## 3) Functional Completion by Requirement Domain
**Goal:** Make schema transition safe in dev/test and repeatable for deployment.
**Goal:** Complete all V1 functional requirements in a risk-aware order.
### Tasks
1. Update bootstrap compatibility logic in `db/operations.py`:
- Remove obsolete transcript-table assumptions.
- Add forward-compatible patches for current tables only.
2. Define migration/backfill approach for existing local data.
3. Document rollback and recovery steps.
4. Rehearse migration path against representative data.
### Recommended Order
1. Business-critical end-user flows
2. Data integrity and consistency capabilities
3. Admin/operational controls
4. Lower-priority UX and quality-of-life items that are in V1 scope
### Implementation Steps
For each requirement slice:
1. Finalize contract/schema
2. Implement domain logic
3. Implement persistence/state changes
4. Integrate API/UI
5. Add automated tests
6. Update docs
### Deliverables
- Migration/upgrade runbook.
- Validated bootstrap behavior for dev/test.
- Requirement completion report with validation evidence
### Exit Criteria
- Migration path is documented and tested with no unresolved data-loss risk.
- All V1 “must-have” requirements are complete and validated.
---
## Phase 5 — Test Suite Realignment
## 4) Data Model, Migration, and Backfill Safety
**Goal:** Restore full confidence after the schema redesign.
**Goal:** Ensure data model and migrations are production-safe.
### Tasks
1. Rewrite model tests for:
- `Document`, `Source`, `Job`, `Revision` relationships and invariants.
2. Rewrite service/integration tests:
- Worker success/failure paths using `Job.text` / `Job.error_detail`.
- Optional single-revision creation/update behavior.
3. Update UI tests for new job-detail/revision rendering behavior.
4. Re-enable strict CI quality gates (lint, type, tests).
### Implementation Steps
1. Validate schema against final V1 domain needs.
2. Implement forward-safe migrations.
3. Define rollback/mitigation plans for migration failures.
4. Build and verify backfill scripts (if needed).
5. Add migration rehearsal in staging with representative data.
### Deliverables
- Updated test matrix and passing CI.
- Migration runbook
- Backfill verification checklist
### Exit Criteria
- Critical user flows and failure paths are covered and green.
- Migration plan validated in staging.
- No unresolved data-loss risk for V1 rollout.
---
## Phase 6 — Reliability, Operations, and Release Readiness
## 5) Security, Access Control, and Compliance Baseline
**Goal:** Ensure V1 is operable and launch-safe.
**Goal:** Close MVP security gaps and establish V1 baseline controls.
### Tasks
1. Verify error taxonomy behavior across UI/API/service/worker.
2. Confirm structured logging includes relevant identifiers (`job_id`, `document_id`, `source_id` when applicable).
3. Validate retry behavior and terminal failure handling.
4. Finalize release checklist, deployment steps, and rollback procedure.
5. Execute final acceptance run against requirements traceability.
### Implementation Steps
1. Complete authn/authz coverage for all routes/actions.
2. Enforce input validation and output sanitization.
3. Verify secret management and credential rotation process.
4. Add audit logging for sensitive operations.
5. Run dependency/security scanning in CI and remediate findings.
### Deliverables
- V1 release checklist and acceptance evidence.
- `docs/runbook.md` for incident response and operator workflows.
- `docs/migration_v1.md` for V1 migration/backfill/rollback guidance.
- `docs/release_checklist_v1.md` for release sign-off.
- Security checklist with status
- Threat/risk update for V1 scope
### Exit Criteria
- Stakeholder sign-off and launch readiness achieved.
- No unresolved critical/high vulnerabilities for V1 launch.
- Access control behavior verified by tests.
---
## Requirement Traceability Focus
## 6) Observability & Operability
The plan must keep clear evidence against these requirement groups:
**Goal:** Make system behavior observable and supportable in production.
- **Core flow:** REQ-0 to REQ-6
- **Runtime and operations constraints:** REQ-7 to REQ-12
- **Revision workflow:** REQ-13
### Implementation Steps
1. Standardize structured logging and correlation IDs.
2. Add core metrics:
- Latency
- Throughput
- Error rates
- Resource saturation
3. Add tracing for critical request/workflow paths.
4. Define SLOs/SLIs and alert thresholds.
5. Prepare incident response and rollback runbooks.
A lightweight traceability table should be maintained with:
### Deliverables
- Dashboards and alerts
- Operations runbooks
- requirement ID
- implementation status (`not started` / `in progress` / `done`)
- validation evidence (test name, screenshot, or runbook step)
### Exit Criteria
- Team can detect, triage, and remediate incidents quickly.
- Core production signals are available and reliable.
---
## Suggested Execution Rhythm
## 7) Test Strategy Expansion & Quality Gates
- **Weekly:** requirement status and risk review
- **Per PR:** contract checks (model names, field names, lifecycle values)
- **Milestone checks:** end of Phases 2, 4, and 6
**Goal:** Raise confidence for repeatable, low-risk releases.
### Implementation Steps
1. Expand unit and integration tests across V1 features.
2. Add contract tests between key components/services.
3. Add end-to-end tests for critical user journeys.
4. Add non-functional tests where relevant:
- Performance/load
- Soak
- Failure-injection scenarios
5. Enforce CI quality gates (tests, lint, type checks, security scans).
### Deliverables
- Test matrix with ownership
- CI gate definition and thresholds
### Exit Criteria
- Critical-path regressions are blocked automatically.
- Test coverage and reliability thresholds meet V1 targets.
---
## 8) Performance & Scalability Validation
**Goal:** Meet expected V1 performance at projected load.
### Implementation Steps
1. Define performance budgets per key flow.
2. Benchmark current behavior in staging.
3. Optimize bottlenecks (queries, caching, concurrency, etc.).
4. Re-test after each optimization and compare against budget.
5. Document known limits and safe operating bounds.
### Deliverables
- Performance benchmark report
- Optimization log
### Exit Criteria
- V1 performance targets met for expected usage profile.
---
## 9) Release Engineering & Environment Readiness
**Goal:** Make deployment repeatable, controlled, and reversible.
### Implementation Steps
1. Harden CI/CD pipeline with clear promotion gates.
2. Ensure config parity and consistency across environments.
3. Define rollout strategy (phased/canary/limited release as applicable).
4. Validate rollback procedures in staging.
5. Produce release checklist and ownership model.
### Deliverables
- Release playbook
- Environment readiness checklist
### Exit Criteria
- Deployment and rollback are rehearsed and reliable.
- Release process is executable without tribal knowledge.
---
## 10) Documentation Completion
**Goal:** Ensure V1 can be built, operated, and supported from documentation.
### Implementation Steps
1. Update core project docs to match final V1 behavior:
- Architecture
- Error handling
- Requirements status
- Index/navigation
- Intent alignment summary
2. Add operator troubleshooting guides.
3. Add integration/API examples for consumers.
4. Publish changelog/version notes for V1.
### Deliverables
- Updated documentation set for V1
- V1 release notes
### Exit Criteria
- A new team member can run/support the system using docs alone.
---
## 11) Final Validation, UAT, and Launch
**Goal:** Confirm readiness and launch V1 safely.
### Implementation Steps
1. Run full-system acceptance validation against the V1 traceability matrix.
2. Conduct stakeholder UAT and capture sign-off.
3. Execute production readiness review.
4. Launch in controlled phases and monitor key signals.
### Deliverables
- UAT/PRR sign-off records
- Launch checklist and monitoring plan
### Exit Criteria
- Stakeholder approval achieved.
- Launch metrics are stable within defined thresholds.
---
## 12) Post-Launch Stabilization (30–60 Days)
**Goal:** Consolidate V1 in production before major expansion.
### Implementation Steps
1. Track incidents, defects, and user feedback.
2. Prioritize stabilization fixes with short cycle times.
3. Remove temporary flags/mitigations introduced during launch.
4. Produce post-launch retrospective and handoff to standard roadmap cadence.
### Deliverables
- Stabilization report
- Prioritized backlog update
### Exit Criteria
- Incident/error rates converge to steady-state targets.
- V1 transitions from launch mode to normal operations.
---
## Recommended Execution Rhythm
- **Weekly:** Requirement closure + risk review
- **Biweekly:** Release train with quality gates
- **Milestone reviews:** After phases 2, 6, 9, and 11
---
## Scope Discipline Rule (V1 Focus)
- Only work required to satisfy V1 requirements enters this plan.
- Nice-to-have enhancements are captured in a separate backlog document.
- Schema or contract changes after Phase 1 require explicit approval and traceability impact review.
To preserve delivery focus:
- V1 execution prioritizes completion of scoped requirements.
- Refinements/enhancements are captured in a separate future document and backlog.
- Only explicitly approved scope changes may enter this plan.
docs/ver2/V2 Context for DB Updates.md
-35
View File
@@ -1,35 +0,0 @@
# AI Coding Assistant Project Briefing & Context
## Project Mission
This application is a family history archival and transcription platform. Its primary goal is to accept scanned document images (letters, postcards, logbooks, diaries), execute OCR and structured transcription via AI vision models (OpenAI GPT-4o, Anthropic Claude 3.5 Sonnet), and manage historical metadata (authors, recipients, dates, and locations).
---
## Technical Stack & Architecture
* **Database:** PostgreSQL 13+ with native `UUID` (`gen_random_uuid()`) and `JSONB` columns.
* **Backend Runtime / Concurrency:** Python utilizing `asyncio` for concurrent HTTP API calls to AI providers, with strict rate-limiting via `asyncio.Semaphore`.
* **Validation & Types:** TypeScript with **Zod** schema definitions. Incoming AI responses must be parsed and validated with Zod schemas *before* database insertion.
* **ORM / Database Access:** Raw parameterized SQL queries or lightweight query builders (e.g., Kysely/Prisma) respecting PostgreSQL native types.
---
## Core System Directives for AI Code Generation
### 1. Data Immutability vs. Human Corrections
* `job_source.raw_transcription` and `source.raw_transcription` represent original, point-in-time machine outputs and are **immutable**.
* Human corrections occur on `source.revised_text`.
* When fetching text for the UI, always display `COALESCE(source.revised_text, source.raw_transcription)`.
### 2. Async Execution & Batching Rules
* A `job` represents an overarching execution run for a folder/group of images belonging to a single `document`.
* Images are submitted to AI APIs **one at a time in rapid succession** using `asyncio` worker pools.
* Each single-image API call populates a row in `job_source` with its own `status`, `raw_transcription`, `ai_metadata`, and `raw_api_response`.
* If 9 of 10 pages succeed and 1 fails, `job_source.status` for the failed image becomes `'failed'`, while `job.status` becomes `'partial_success'`. Do not mark the entire batch as failed if partial results exist.
### 3. Entity Relationships
* **Authors/Recipients:** A `document` can have multiple authors and recipients. Do NOT put direct `author_id` foreign keys on `document`. Query authors/recipients via `document_person` where `role = 'author'` or `role = 'recipient'`.
* **Page Ordering:** Multi-page documents must always be queried using `ORDER BY page_number ASC`.
### 4. Database Mutations
* Always use parameterized SQL queries (`$1`, `$2`) to prevent SQL injection.
* Store datetimes using UTC ISO 8601 strings or native PostgreSQL `TIMESTAMPTZ`.
docs/ver2/V2 DB Schema.md
-122
View File
@@ -1,122 +0,0 @@
# Database Schema (V2 Architecture)
This document describes the PostgreSQL relational schema for the transcription platform. It incorporates multi-image batch orchestration via `asyncio`, page-level execution tracking, many-to-many author/recipient attribution, and JSONB document storage for AI vision outputs.
All primary and foreign keys are PostgreSQL native UUIDs (`gen_random_uuid()`).
## Entity Relationship Diagram
```mermaid
erDiagram
PERSON {
UUID id PK
TEXT full_name
TEXT display_name
TEXT maiden_name
DATE birth_date
TEXT birth_date_raw
TEXT birth_place
DATE death_date
TEXT death_date_raw
TEXT death_place
TEXT biography
TEXT portrait_path
JSONB metadata
TIMESTAMPTZ created_at
TIMESTAMPTZ updated_at
}
DOCUMENT {
UUID id PK
TEXT name
TEXT document_type
DATE document_date
TEXT document_date_raw
TEXT location_created
TEXT notes
TEXT archive_identifier
TIMESTAMPTZ created_at
TIMESTAMPTZ updated_at
}
DOCUMENT_PERSON {
UUID id PK
UUID document_id FK
UUID person_id FK
VARCHAR role "author | recipient"
TIMESTAMPTZ created_at
}
JOB {
UUID id PK
UUID document_id FK
VARCHAR status "queued | processing | completed | partial_success | failed"
INTEGER retry_count
TEXT provider
TEXT model
TEXT prompt_name
TIMESTAMPTZ date_created
TIMESTAMPTZ date_updated
}
SOURCE {
UUID id PK
UUID document_id FK
INTEGER page_number
TEXT upload_name
TEXT filename
TEXT file_path
TEXT raw_transcription
TEXT revised_text
TIMESTAMPTZ date_uploaded
TIMESTAMPTZ date_revised
}
JOB_SOURCE {
UUID id PK
UUID job_id FK
UUID source_id FK
VARCHAR status "pending | transcribed | failed"
TEXT raw_transcription
JSONB ai_metadata
JSONB raw_api_response
TEXT error_detail
TIMESTAMPTZ executed_at
}
DOCUMENT ||--o{ DOCUMENT_PERSON : "has_people"
PERSON ||--o{ DOCUMENT_PERSON : "participates_in"
DOCUMENT ||--o{ JOB : "has_jobs"
DOCUMENT ||--o{ SOURCE : "contains_pages"
JOB ||--o{ JOB_SOURCE : "executes"
SOURCE ||--o{ JOB_SOURCE : "processed_in"
```
## Domain Invariants & Rules
### Page-Level Execution & AI Outputs
* Execution Granularity: Every single image execution by an AI model produces a dedicated record in job_source.
* Point-in-Time Auditability: job_source.raw_api_response stores the unparsed REST response envelope for that specific image page call. job_source.ai_metadata stores spatial bounding boxes, token usage, and layout details for that specific image page call.
* Active Output Caching: Upon successful completion of an image call, source.raw_transcription is updated with the latest output string from job_source.raw_transcription for fast UI rendering.
### Page Ordering & Revisions
* Sequential Integrity: source.page_number dictates page ordering within a document. Reads assembling full documents must query ORDER BY source.document_id, source.page_number ASC.
* Inlined Human Corrections: User edits occur at the page level inside source.revised_text. source.raw_transcription remains immutable. If source.revised_text is non-null, application frontends must render source.revised_text.
### Async Job Lifecycle & Failure Isolation
* Batch Orchestrator: A job represents an overarching execution run across one or more source images belonging to a document.
* Isolated Failures: API requests run concurrently (e.g., using asyncio). A failure on page 3 does not invalidate successful transcriptions on page 1 or 2.
* Job States:
- queued: Created, awaiting worker execution.
- processing: Concurrent HTTP tasks actively running.
- completed: 100% of linked job_source tasks succeeded (transcribed).
- partial_success: At least one job_source succeeded and at least one failed.
- failed: All linked job_source tasks failed or a job-level runtime error occurred.
### Attribution & Person Roles
* Multi-Person Roles: Documents support zero, one, or many authors and recipients linked via document_person.
* Role Uniqueness: (document_id, person_id, role) must be unique to prevent duplicate role tagging.
docs/ver2/V2 PostgreSQL DDL Specification.md
-102
View File
@@ -1,102 +0,0 @@
## PostgreSQL DDL Specification
```sql
-- Enable pgcrypto for UUID generation if on PostgreSQL < 13
CREATE EXTENSION IF NOT EXISTS "pgcrypto";
-- 1. PERSON TABLE
CREATE TABLE person (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
full_name TEXT NOT NULL,
display_name TEXT,
maiden_name TEXT,
birth_date DATE,
birth_date_raw TEXT,
birth_place TEXT,
death_date DATE,
death_date_raw TEXT,
death_place TEXT,
biography TEXT,
portrait_path TEXT,
metadata JSONB DEFAULT '{}'::jsonb,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
-- 2. DOCUMENT TABLE
CREATE TABLE document (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
name TEXT NOT NULL,
document_type TEXT,
document_date DATE,
document_date_raw TEXT,
location_created TEXT,
notes TEXT,
archive_identifier TEXT,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
-- 3. DOCUMENT_PERSON (Junction Table for Multi-Author / Multi-Recipient)
CREATE TABLE document_person (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
document_id UUID NOT NULL REFERENCES document(id) ON DELETE CASCADE,
person_id UUID NOT NULL REFERENCES person(id) ON DELETE CASCADE,
role VARCHAR(20) NOT NULL, -- 'author' or 'recipient'
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
CONSTRAINT unique_document_person_role UNIQUE (document_id, person_id, role)
);
-- 4. JOB TABLE (Batch-level orchestrator)
CREATE TABLE job (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
document_id UUID NOT NULL REFERENCES document(id) ON DELETE CASCADE,
status VARCHAR(50) NOT NULL DEFAULT 'queued', -- 'queued', 'processing', 'completed', 'partial_success', 'failed'
retry_count INTEGER NOT NULL DEFAULT 0,
provider TEXT NOT NULL, -- e.g., 'openai', 'anthropic'
model TEXT NOT NULL, -- e.g., 'gpt-4o', 'claude-3-5-sonnet'
prompt_name TEXT,
date_created TIMESTAMPTZ NOT NULL DEFAULT now(),
date_updated TIMESTAMPTZ NOT NULL DEFAULT now()
);
-- 5. SOURCE TABLE (Physical image files & active state)
CREATE TABLE source (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
document_id UUID NOT NULL REFERENCES document(id) ON DELETE CASCADE,
page_number INTEGER NOT NULL DEFAULT 1,
upload_name TEXT NOT NULL,
filename TEXT NOT NULL,
file_path TEXT NOT NULL,
raw_transcription TEXT, -- Cached active AI text output
revised_text TEXT, -- Active human edited text
date_uploaded TIMESTAMPTZ NOT NULL DEFAULT now(),
date_revised TIMESTAMPTZ
);
-- 6. JOB_SOURCE (Junction Table: Per-Image Execution Record)
CREATE TABLE job_source (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
job_id UUID NOT NULL REFERENCES job(id) ON DELETE CASCADE,
source_id UUID NOT NULL REFERENCES source(id) ON DELETE CASCADE,
status VARCHAR(50) NOT NULL DEFAULT 'pending', -- 'pending', 'transcribed', 'failed'
raw_transcription TEXT, -- Point-in-time raw AI text output
ai_metadata JSONB, -- Page-level bounding boxes, tokens, confidence
raw_api_response JSONB, -- Complete REST response envelope
error_detail TEXT,
executed_at TIMESTAMPTZ NOT NULL DEFAULT now(),
CONSTRAINT unique_job_source UNIQUE (job_id, source_id)
);
-- INDEXES FOR FAST LOOKUPS & QUERY PERFORMANCE
CREATE INDEX idx_person_full_name ON person(full_name);
CREATE INDEX idx_document_date ON document(document_date);
CREATE INDEX idx_document_person_doc ON document_person(document_id);
CREATE INDEX idx_document_person_per ON document_person(person_id);
CREATE INDEX idx_source_document ON source(document_id);
CREATE INDEX idx_source_page_order ON source(document_id, page_number);
CREATE INDEX idx_job_document ON job(document_id);
CREATE INDEX idx_job_source_job ON job_source(job_id);
CREATE INDEX idx_job_source_source ON job_source(source_id);
CREATE INDEX idx_job_source_ai_metadata ON job_source USING GIN (ai_metadata);
```
docs/ver2/V2 TypeScript Zod Schemas.md
-169
View File
@@ -1,169 +0,0 @@
## TypeScript Zod schemas
Here are the TypeScript Zod schemas matching your V2 PostgreSQL database definition.
These schemas cover:
1. Database Entities: Pure runtime validators representing rows fetched directly from PostgreSQL.
2. AI Payload Extensions: The structured document output stored inside job.ai_metadata.
3. Insert/Create Schemas: Utility types derived with .omit() for creating new records where auto-generated columns (id, created_at, updated_at, etc.) are handled by PostgreSQL defaults.
---
```
import { z } from "zod";
// ==========================================
// 1. ATOMIC & REUSABLE SCHEMAS
// ==========================================
export const UUIDSchema = z.string().uuid();
export const ISODateTimeSchema = z.coerce.date();
export const BoundingBoxSchema = z.object({
ymin: z.number().min(0).max(1000),
xmin: z.number().min(0).max(1000),
ymax: z.number().min(0).max(1000),
xmax: z.number().min(0).max(1000),
});
export const BlockTypeSchema = z.enum([
"heading",
"paragraph",
"table",
"margin_note",
"signature",
"footnote",
"header",
]);
// ==========================================
// 2. PAGE-LEVEL AI METADATA SCHEMA (job_source.ai_metadata)
// ==========================================
export const TranscribedBlockSchema = z.object({
text: z.string(),
confidence: z.number().min(0).max(1),
blockType: BlockTypeSchema,
boundingBox: BoundingBoxSchema.optional(),
});
export const PageAIMetadataSchema = z.object({
detectedLanguage: z.string().optional(),
overallConfidence: z.number().min(0).max(1),
blocks: z.array(TranscribedBlockSchema),
inputTokens: z.number().optional(),
outputTokens: z.number().optional(),
extractedEntities: z.record(z.string(), z.unknown()).optional(),
});
export type PageAIMetadata = z.infer<typeof PageAIMetadataSchema>;
// ==========================================
// 3. TABLE ENTITY SCHEMAS
// ==========================================
// --- PERSON TABLE ---
export const PersonSchema = z.object({
id: UUIDSchema,
fullName: z.string().min(1),
displayName: z.string().nullable().optional(),
maidenName: z.string().nullable().optional(),
birthDate: z.string().nullable().optional(),
birthDateRaw: z.string().nullable().optional(),
birthPlace: z.string().nullable().optional(),
deathDate: z.string().nullable().optional(),
deathDateRaw: z.string().nullable().optional(),
deathPlace: z.string().nullable().optional(),
biography: z.string().nullable().optional(),
portraitPath: z.string().nullable().optional(),
metadata: z.record(z.string(), z.unknown()).default({}),
createdAt: ISODateTimeSchema,
updatedAt: ISODateTimeSchema,
});
// --- DOCUMENT TABLE ---
export const DocumentSchema = z.object({
id: UUIDSchema,
name: z.string().min(1),
documentType: z.string().nullable().optional(),
documentDate: z.string().nullable().optional(),
documentDateRaw: z.string().nullable().optional(),
locationCreated: z.string().nullable().optional(),
notes: z.string().nullable().optional(),
archiveIdentifier: z.string().nullable().optional(),
createdAt: ISODateTimeSchema,
updatedAt: ISODateTimeSchema,
});
// --- DOCUMENT_PERSON JUNCTION ---
export const PersonRoleSchema = z.enum(["author", "recipient"]);
export const DocumentPersonSchema = z.object({
id: UUIDSchema,
documentId: UUIDSchema,
personId: UUIDSchema,
role: PersonRoleSchema,
createdAt: ISODateTimeSchema,
});
// --- JOB TABLE ---
export const JobStatusSchema = z.enum([
"queued",
"processing",
"completed",
"partial_success",
"failed",
]);
export const JobSchema = z.object({
id: UUIDSchema,
documentId: UUIDSchema,
status: JobStatusSchema.default("queued"),
retryCount: z.number().int().nonnegative().default(0),
provider: z.string(),
model: z.string(),
promptName: z.string().nullable().optional(),
dateCreated: ISODateTimeSchema,
dateUpdated: ISODateTimeSchema,
});
// --- SOURCE TABLE ---
export const SourceSchema = z.object({
id: UUIDSchema,
documentId: UUIDSchema,
pageNumber: z.number().int().positive().default(1),
uploadName: z.string(),
filename: z.string(),
filePath: z.string(),
rawTranscription: z.string().nullable().optional(),
revisedText: z.string().nullable().optional(),
dateUploaded: ISODateTimeSchema,
dateRevised: ISODateTimeSchema.nullable().optional(),
});
// --- JOB_SOURCE JUNCTION (Page Execution Output) ---
export const JobSourceStatusSchema = z.enum([
"pending",
"transcribed",
"failed",
]);
export const JobSourceSchema = z.object({
id: UUIDSchema,
jobId: UUIDSchema,
sourceId: UUIDSchema,
status: JobSourceStatusSchema.default("pending"),
rawTranscription: z.string().nullable().optional(),
aiMetadata: PageAIMetadataSchema.nullable().optional(),
rawApiResponse: z.record(z.string(), z.unknown()).nullable().optional(),
errorDetail: z.string().nullable().optional(),
executedAt: ISODateTimeSchema,
});
export type Person = z.infer<typeof PersonSchema>;
export type Document = z.infer<typeof DocumentSchema>;
export type DocumentPerson = z.infer<typeof DocumentPersonSchema>;
export type Job = z.infer<typeof JobSchema>;
export type Source = z.infer<typeof SourceSchema>;
export type JobSource = z.infer<typeof JobSourceSchema>;
```
docs/ver2/ver2.md
-150
View File
@@ -1,150 +0,0 @@
# Version 2 Plan
Desired Enhancements:
1. Data store
* Upgrade db to PostgresSQL
* Begin capturing JSONB data (which will allow future migration to MongoDB if desired)
* Relocate db and uploaded images to a location outside of the project folder that can be backed up. (This needs to be done for all projects.) (c:/github/data/transcription?)
2. Add ability to upload multiple images (or a folder of images)
* How many is too many?
* If there is a practical max image count, can I break a block of images up into smaller batches automatically?
3. UI
* Introduce the concept of "documents" to the UI.
* Before an image can be uploaded a "document" needs to be created/defined.
* As part of the upload process, document images need to be associated with a document.
* Multiple image upload
* Refine the job detail/log screen
* Is document id + original filename the best name for uploaded images?
* How to present multiple images within one job?
* Add document name, original filename to job detail.
## Purpose
Version 2 extends the V1 baseline by introducing a production-oriented persistence architecture while preserving current user workflows.
Primary target changes:
- Migrate relational persistence from SQLite to PostgreSQL
- Introduce optional MongoDB for document-oriented adjunct data (non-canonical)
V1 behavior remains the functional baseline unless explicitly superseded by approved V2 requirements.
---
## V2 Goals
1. **Relational migration complete**
- PostgreSQL becomes the default system of record for `Document`, `Source`, `Job`, and `Revision`.
1. **Operational maturity**
- Repeatable migrations, rollback paths, and environment-specific deployment procedures are documented and tested.
1. **Optional document store integration**
- MongoDB is introduced only for clearly scoped use cases that do not replace canonical relational ownership.
1. **No regression of V1 workflows**
- Upload, queue/worker processing, status inspection, original transcription, and optional single revision remain stable.
---
## Non-Goals (V2)
- Replacing SQLModel domain ownership with MongoDB
- Introducing breaking UI behavior for existing V1 flows
- Expanding revision cardinality beyond current `0..1` without explicit requirements update
---
## Proposed Scope
### A) PostgreSQL migration (required)
- Add PostgreSQL runtime profile for local/dev/prod
- Introduce migration toolchain and migration history
- Convert bootstrap strategy from compatibility patching to explicit migrations
- Validate model constraints and indexes against PostgreSQL
- Add operational checks (connectivity, pool, transaction behavior)
### B) MongoDB integration (optional, gated)
- Define approved use cases (for example: denormalized read models, audit/event projections, or search-oriented materializations)
- Keep canonical write path in relational store
- Add feature flag/config gate to enable or disable Mongo features
- Document consistency model and failure behavior
---
## Milestones
## M1 — Requirements and architecture baseline
- Create V2 requirements delta from V1 baseline
- Define relational/document ownership boundaries
- Approve migration strategy and cutover approach
**Exit criteria:** signed architecture decision and updated traceability map.
## M2 — PostgreSQL foundation
- Add PostgreSQL environment wiring and secrets strategy
- Add migration framework and initial schema migration
- Add CI path using PostgreSQL service container
**Exit criteria:** test suite green on PostgreSQL in CI.
## M3 — Data migration and cutover rehearsal
- Build SQLite -> PostgreSQL migration utility/playbook
- Rehearse migration on representative datasets
- Validate rollback/recovery procedures
**Exit criteria:** successful dry-run migration with measured rollback test.
## M4 — MongoDB optional integration
- Implement scoped Mongo use case(s)
- Add fallback behavior when Mongo unavailable
- Add tests and operational runbook updates
**Exit criteria:** feature-gated Mongo behavior validated with no V1 flow regressions.
## M5 — Release readiness
- Final regression suite (functional + reliability)
- Performance and failure-mode checks
- Production release checklist and sign-off
**Exit criteria:** V2 release approval.
---
## Risks and Mitigations
- **Schema drift risk** -> enforce migration-first policy and CI migration checks.
- **Dual-store consistency risk** -> keep relational source of truth and explicit projection contracts.
- **Operational complexity** -> staged rollout, runbooks, and feature flags.
- **Regression risk in worker lifecycle** -> keep dedicated reliability tests around terminal-state guarantees.
---
## Traceability and Evidence
Maintain a V2 table with:
- requirement/change ID
- status (`not started` / `in progress` / `done`)
- implementation PR
- validation evidence (test names, migration rehearsal logs, runbook references)
---
## Suggested first implementation tasks
1. Create `docs/ver2/adr/` and draft ADR for persistence ownership boundaries.
2. Add PostgreSQL compose profile and env contract.
3. Introduce migration tooling and generate initial migration from current schema.
4. Add CI job for PostgreSQL-backed `pytest -m "not external"`.
pyproject.toml
-4
View File
@@ -39,10 +39,6 @@ dev = [
[tool.pytest.ini_options]
addopts = "--strict-markers -q"
asyncio_mode = "strict"
filterwarnings = [
"error:coroutine .* was never awaited:RuntimeWarning",
]
markers = [
"unit: pure logic tests with no external dependencies",
"integration: tests that touch framework or database contracts",
src/transcription/app.py
+1 -32
View File
@@ -2,17 +2,12 @@
from __future__ import annotations
import logging
from contextlib import AsyncExitStack
from contextlib import asynccontextmanager
from datetime import UTC
from datetime import datetime
from datetime import timedelta
from fastapi import FastAPI
from fastapi import status
from fastapi.responses import RedirectResponse
from fastapi.staticfiles import StaticFiles
from .api.errors import register_error_handlers
from .api.health import router as health_router
@@ -22,18 +17,15 @@ from .db import create_all
from .db import dispose_database_runtime
from .db import initialize_database_runtime
from .services import ServiceBundle
from .services.jobs import JobService
from .ui import register_pages
from .worker import worker_consumer_lifespan
logger = logging.getLogger(__name__)
@asynccontextmanager
async def _lifespan(app: FastAPI):
configure_logging()
settings = getattr(app.state, "settings", None) or get_settings()
settings = get_settings()
app.state.settings = settings
app.state.services = ServiceBundle()
app.state.runtime = initialize_database_runtime(settings=settings)
@@ -44,8 +36,6 @@ async def _lifespan(app: FastAPI):
settings.upload_dir.mkdir(parents=True, exist_ok=True)
settings.prompt_dir.mkdir(parents=True, exist_ok=True)
await _recover_stale_processing_jobs(app)
async with AsyncExitStack() as stack:
stack.push_async_callback(dispose_database_runtime)
stop_event, worker_notifier = await stack.enter_async_context(
@@ -59,30 +49,9 @@ async def _lifespan(app: FastAPI):
yield
async def _recover_stale_processing_jobs(app: FastAPI) -> None:
"""Re-queue stale processing jobs at startup.
Any job left in PROCESSING longer than the configured provider timeout is
assumed orphaned and moved back to QUEUED before the worker starts.
"""
settings = app.state.settings
stale_before = datetime.now(UTC) - timedelta(seconds=settings.worker_provider_timeout_seconds)
job_service = JobService(session_factory=app.state.runtime.session_factory)
recovered = await job_service.requeue_stale_processing_jobs(stale_before=stale_before)
if recovered > 0:
logger.warning("Recovered %s stale processing job(s) at startup", recovered)
def create_app() -> FastAPI:
"""Create and configure the FastAPI application."""
app = FastAPI(title="Transcription", lifespan=_lifespan)
settings = get_settings()
app.state.settings = settings
app.mount(
"/uploads",
StaticFiles(directory=settings.upload_dir, check_dir=False),
name="uploads",
)
@app.get("/", include_in_schema=False)
async def root_redirect() -> RedirectResponse:
src/transcription/config.py
-5
View File
@@ -11,7 +11,6 @@ from enum import StrEnum
from pathlib import Path
from typing import Literal
from pydantic import Field
from pydantic_settings import BaseSettings
from pydantic_settings import SettingsConfigDict
@@ -51,10 +50,6 @@ class Settings(BaseSettings):
# --- worker reliability ---
worker_max_retries: int = 0
worker_retry_backoff_seconds: float = 0.0
worker_provider_timeout_seconds: float = Field(default=20.0, gt=0.0, le=20.0)
worker_min_transcription_chars: int = Field(default=0, ge=0)
worker_min_transcription_lines: int = Field(default=0, ge=0)
worker_fail_on_finish_reason_length: bool = False
@property
def should_bootstrap_schema(self) -> bool:
src/transcription/db/operations.py
+10 -23
View File
@@ -22,7 +22,7 @@ async def get_next_queued_job(*, session: AsyncSession) -> Job | None:
result = await session.exec(
select(Job)
.where(Job.status == JobStatus.QUEUED)
.order_by(Job.date_created) # pyright: ignore[reportArgumentType]
.order_by(Job.created_at) # pyright: ignore[reportArgumentType]
.limit(1)
) # fmt: skip
return result.first()
@@ -51,29 +51,16 @@ def _ensure_sqlite_compat_columns(connection: Connection) -> None:
inspector = inspect(connection)
table_names = set(inspector.get_table_names())
if "job" not in table_names:
return
if "job" in table_names:
job_columns = {column["name"] for column in inspector.get_columns("job")}
if "retry_count" not in job_columns:
columns = {column["name"] for column in inspector.get_columns("job")}
if "retry_count" not in columns:
connection.execute(text("ALTER TABLE job ADD COLUMN retry_count INTEGER NOT NULL DEFAULT 0"))
logger.warning("Applied SQLite compatibility schema patch table=job column=retry_count default=0")
if "revision" in table_names:
revision_columns = {column["name"] for column in inspector.get_columns("revision")}
if "source_id" in revision_columns:
has_unique_source = False
for index in inspector.get_indexes("revision"):
if index.get("unique") and index.get("column_names") == ["source_id"]:
has_unique_source = True
break
if not has_unique_source:
connection.execute(
text(
"CREATE UNIQUE INDEX IF NOT EXISTS "
"ux_revision_source_id ON revision(source_id)"
)
)
logger.warning(
"Applied SQLite compatibility schema patch "
"table=revision unique_index=ux_revision_source_id"
)
if "transcript" in table_names:
transcript_columns = {column["name"] for column in inspector.get_columns("transcript")}
if "model" not in transcript_columns:
connection.execute(text("ALTER TABLE transcript ADD COLUMN model VARCHAR"))
logger.warning("Applied SQLite compatibility schema patch table=transcript column=model")
src/transcription/models.py
+50 -59
View File
@@ -1,9 +1,9 @@
"""SQLModel domain models for the transcription system.
Core V1 lifecycle:
Document -> one-to-many -> Source
Document -> one-to-many -> Job
Source -> one-to-one? -> Revision (optional)
Core models capture the MVP lifecycle:
Document (1) -> (many) Job
Job (1) -> (1) Transcript
Job (1) -> (many) TranscriptRevision
"""
from datetime import UTC
@@ -27,37 +27,15 @@ class JobStatus(StrEnum):
class Document(SQLModel, table=True):
"""An historical document."""
"""An uploaded document image."""
id: UUID = Field(default_factory=uuid4, primary_key=True)
name: str
# Relationships
jobs: list["Job"] = Relationship(back_populates="document")
sources: list["Source"] = Relationship(back_populates="document")
class Source(SQLModel, table=True):
"""A document source (image or PDF)."""
id: UUID = Field(default_factory=uuid4, primary_key=True)
document_id: UUID = Field(foreign_key="document.id")
job_id: UUID = Field(foreign_key="job.id")
upload_name: str
"""The filename of the source that was uploaded for transcription."""
filename: str
"""The system generated unique source name."""
file_path: str
"""The location where the sources are stored on the local filesystem."""
date_uploaded: datetime = Field(default_factory=lambda: datetime.now(UTC))
uploaded_at: datetime = Field(default_factory=lambda: datetime.now(UTC))
# Relationships
document: Optional["Document"] = Relationship(back_populates="sources")
job: Optional["Job"] = Relationship(back_populates="sources")
revision: Optional["Revision"] = Relationship(
back_populates="source",
sa_relationship_kwargs={"uselist": False},
)
# --- relationships ---
jobs: list["Job"] = Relationship(back_populates="document")
class Job(SQLModel, table=True):
@@ -67,44 +45,57 @@ class Job(SQLModel, table=True):
document_id: UUID = Field(foreign_key="document.id")
status: JobStatus = Field(default=JobStatus.QUEUED)
retry_count: int = Field(default=0, ge=0)
date_created: datetime = Field(default_factory=lambda: datetime.now(UTC))
date_updated: datetime = Field(default_factory=lambda: datetime.now(UTC))
provider: str | None = None
created_at: datetime = Field(default_factory=lambda: datetime.now(UTC))
updated_at: datetime = Field(default_factory=lambda: datetime.now(UTC))
# --- relationships ---
document: Document = Relationship(back_populates="jobs")
transcript: Optional["Transcript"] = Relationship(back_populates="job")
transcript_revisions: list["TranscriptRevision"] = Relationship(back_populates="job")
@property
def filename(self) -> str:
"""Return the filename of the associated document."""
return self.document.filename if self.document else "unknown"
class Transcript(SQLModel, table=True):
"""The output of a transcription job."""
id: UUID = Field(default_factory=uuid4, primary_key=True)
job_id: UUID = Field(foreign_key="job.id", unique=True)
"""ID for the associated job. There's a 1-1 relationship bewteen transcripts and jobs."""
provider: str
"""Name of the transcription provider used to generate this transcript."""
model: str | None = None
"""Model identifier used to generate this transcript."""
prompt_name: str | None = None
prompt_name: str
"""Name of the prompt used to generate this transcript."""
model: str | None = None
"""Provider model that generated the original AI transcript."""
text: str | None = None
"""The transcribed text. This may be None if the job failed or is still in progress."""
error_detail: str | None = None
"""Details of any error that occurred during transcription."""
created_at: datetime = Field(default_factory=lambda: datetime.now(UTC))
# Relationships
document: Optional["Document"] = Relationship(back_populates="jobs")
sources: list["Source"] = Relationship(back_populates="job")
@property
def filename(self) -> str:
"""Return the filename of the associated source, when available."""
if not self.sources:
return "unknown"
return self.sources[0].filename
# --- relationships ---
job: Job = Relationship(back_populates="transcript")
class Revision(SQLModel, table=True):
"""A revision of a transcription text."""
class TranscriptRevision(SQLModel, table=True):
"""Version history entries for a transcription job."""
__table_args__ = (UniqueConstraint("job_id", "version_number", name="uq_transcript_revision_job_version"),)
id: UUID = Field(default_factory=uuid4, primary_key=True)
source_id: UUID = Field(foreign_key="source.id")
"""ID for the associated source."""
revision: int = Field(default=1, ge=1)
"""Revision number of this transcription revision, starting at 1."""
text: str
"""The revised text."""
date_created: datetime = Field(default_factory=lambda: datetime.now(UTC))
job_id: UUID = Field(foreign_key="job.id", index=True)
version_number: int = Field(ge=1)
provider: str
prompt_name: str
model: str | None = None
source: str = Field(default="ai")
text: str | None = None
error_detail: str | None = None
created_at: datetime = Field(default_factory=lambda: datetime.now(UTC))
__table_args__ = (UniqueConstraint("source_id", name="uq_revision_source_id"),)
# Relationships
source: Optional["Source"] = Relationship(back_populates="revision")
# --- relationships ---
job: Job = Relationship(back_populates="transcript_revisions")
src/transcription/providers/base.py
+13 -4
View File
@@ -2,6 +2,9 @@
from dataclasses import dataclass
from typing import Protocol
from uuid import UUID
from ..models import Transcript
class ProviderError(RuntimeError):
@@ -24,10 +27,16 @@ class TranscriptionResult:
provider: str
prompt_name: str
model: str
finish_reason: str | None = None
usage_input_tokens: int | None = None
usage_output_tokens: int | None = None
usage_total_tokens: int | None = None
def to_transcript(self, job_id: UUID) -> Transcript:
"""Convert a TranscriptionResult to a Transcript model instance."""
return Transcript(
job_id=job_id,
provider=self.provider,
prompt_name=self.prompt_name,
model=self.model,
text=self.text,
)
class TranscriptionProvider(Protocol):
src/transcription/providers/openrouter.py
+1 -46
View File
@@ -64,19 +64,8 @@ class OpenRouterTranscriptionProvider:
text = self._extract_text(response)
model = self._get_optional_attr(response, "model") or self.model
finish_reason = self._extract_finish_reason(response)
usage_input_tokens, usage_output_tokens, usage_total_tokens = self._extract_usage(response)
logger.info("OpenRouter transcription completed using model=%s", model)
return TranscriptionResult(
text=text,
provider="openrouter",
prompt_name="",
model=model,
finish_reason=finish_reason,
usage_input_tokens=usage_input_tokens,
usage_output_tokens=usage_output_tokens,
usage_total_tokens=usage_total_tokens,
)
return TranscriptionResult(text=text, provider="openrouter", prompt_name="", model=model)
def _build_request(self, *, prompt_text: str, image_bytes: bytes, mime_type: str) -> OpenRouterRequest:
image_b64 = base64.b64encode(image_bytes).decode("ascii")
@@ -115,34 +104,6 @@ class OpenRouterTranscriptionProvider:
raise ProviderResponseError("OpenRouter response contained no transcription text")
return text
def _extract_finish_reason(self, response: Any) -> str | None:
choices = self._get_optional_attr(response, "choices")
if not choices:
return None
first_choice = choices[0]
finish_reason = self._get_optional_attr(first_choice, "finish_reason")
if isinstance(finish_reason, str) and finish_reason.strip():
return finish_reason.strip()
return None
def _extract_usage(self, response: Any) -> tuple[int | None, int | None, int | None]:
usage = self._get_optional_attr(response, "usage")
if usage is None:
return None, None, None
input_tokens = self._as_int(self._get_optional_attr(usage, "prompt_tokens"))
output_tokens = self._as_int(self._get_optional_attr(usage, "completion_tokens"))
total_tokens = self._as_int(self._get_optional_attr(usage, "total_tokens"))
if input_tokens is None:
input_tokens = self._as_int(self._get_optional_attr(usage, "input_tokens"))
if output_tokens is None:
output_tokens = self._as_int(self._get_optional_attr(usage, "output_tokens"))
if total_tokens is None:
total_tokens = self._as_int(self._get_optional_attr(usage, "total"))
return input_tokens, output_tokens, total_tokens
def _normalize_content(self, content: Any) -> str:
if isinstance(content, str):
return content.strip()
@@ -166,9 +127,3 @@ class OpenRouterTranscriptionProvider:
if isinstance(obj, dict):
return obj.get(key)
return getattr(obj, key, None)
@staticmethod
def _as_int(value: Any) -> int | None:
if isinstance(value, int):
return value
return None
src/transcription/services/documents.py
+12 -15
View File
@@ -21,8 +21,8 @@ class DocumentError(AppError):
"""Raised when document operations fail."""
class MissingSourceError(DocumentError):
"""Raised when a document has no associated sources."""
class MissingImageError(DocumentError):
"""Raised when a required image is missing."""
class UploadError(DocumentError):
@@ -30,7 +30,7 @@ class UploadError(DocumentError):
class DocumentAlreadyExistsError(DocumentError):
"""Raised when a document with the same name already exists in the database."""
"""Raised when a document with the same filename already exists in the database."""
@dataclass(frozen=True)
@@ -72,16 +72,13 @@ class DocumentService(ServiceBase):
async def read_document(self, document_id: UUID, *, session: AsyncSession | None = None) -> Document:
"""Read an existing document from the database.
The selectinload option is used to eagerly load related jobs and sources.
The selectinload option is used to eagerly load related jobs for the document.
"""
async with self._session_scope(session) as _session:
document = await _session.get(
Document,
document_id,
options=(
selectinload(Document.jobs), # pyright: ignore[reportArgumentType]
selectinload(Document.sources), # pyright: ignore[reportArgumentType]
),
options=(selectinload(Document.jobs),), # pyright: ignore[reportArgumentType]
)
if document is None:
raise DocumentError(
@@ -89,11 +86,11 @@ class DocumentService(ServiceBase):
category=ErrorCategory.NOT_FOUND,
suggestion="Re-upload the source document and retry.",
)
elif not document.sources:
raise MissingSourceError(
f"Document with id {document_id} has no associated source records",
elif not Path(document.file_path).exists():
raise MissingImageError(
f"Document with id {document_id} is missing its image file in {self.settings.upload_dir:!s}",
category=ErrorCategory.NOT_FOUND,
suggestion="Upload at least one source for this document and retry.",
suggestion="Re-upload the source document and retry.",
)
return document
@@ -113,13 +110,13 @@ class DocumentService(ServiceBase):
# Query Operations
async def query_documents(
self, *, name: str | None = None, session: AsyncSession | None = None
self, *, filename: str | None = None, session: AsyncSession | None = None
) -> Sequence[Document]:
"""Query documents from the database based on provided filters."""
async with self._session_scope(session) as _session:
query = select(Document)
if name is not None:
query = query.where(Document.name == name)
if filename is not None:
query = query.where(Document.filename == filename)
result = await _session.exec(query)
return result.all()
src/transcription/services/jobs.py
+7 -49
View File
@@ -9,7 +9,6 @@ from sqlmodel.ext.asyncio.session import AsyncSession
from ..models import Job
from ..models import JobStatus
from ..models import Source
from .base import ServiceBase
@@ -36,10 +35,7 @@ class JobService(ServiceBase):
async with self._session_scope(session) as _session:
query = (
select(Job)
.options(
selectinload(Job.document), # pyright: ignore[reportArgumentType]
selectinload(Job.sources).selectinload(Source.revision), # pyright: ignore[reportArgumentType]
)
.options(selectinload(Job.document)) # pyright: ignore[reportArgumentType]
.where(Job.id == job_id)
.execution_options(populate_existing=True)
)
@@ -72,14 +68,11 @@ class JobService(ServiceBase):
) -> Sequence[Job]:
"""Query jobs from the database based on provided filters."""
async with self._session_scope(session) as _session:
query = select(Job).options(
selectinload(Job.document), # pyright: ignore[reportArgumentType]
selectinload(Job.sources), # pyright: ignore[reportArgumentType]
)
query = select(Job).options(selectinload(Job.document)) # pyright: ignore[reportArgumentType]
if status is not None:
query = query.where(Job.status == status)
if filename is not None:
query = query.where(Job.sources.any(Source.filename == filename))
query = query.where(Job.document.filename == filename)
result = await _session.exec(query)
return result.all()
@@ -92,10 +85,7 @@ class JobService(ServiceBase):
"""List all jobs in the database with eagerly loaded documents."""
_ = load_docs
async with self._session_scope(session) as _session:
query = select(Job).options(
selectinload(Job.document), # pyright: ignore[reportArgumentType]
selectinload(Job.sources), # pyright: ignore[reportArgumentType]
)
query = select(Job).options(selectinload(Job.document)) # pyright: ignore[reportArgumentType]
result = await _session.exec(query)
return result.all()
@@ -136,7 +126,7 @@ class JobService(ServiceBase):
job.status = status
if retry_count_increment:
job.retry_count += retry_count_increment
job.date_updated = datetime.now(UTC)
job.updated_at = datetime.now(UTC)
await self._finalize(session=_session, caller_session=session, refresh=(job,))
return job
@@ -149,40 +139,8 @@ class JobService(ServiceBase):
async with self._session_scope(session) as _session:
query = (
select(Job)
.options(
selectinload(Job.document), # pyright: ignore[reportArgumentType]
selectinload(Job.sources).selectinload(Source.revision), # pyright: ignore[reportArgumentType]
)
.options(selectinload(Job.document)) # pyright: ignore[reportArgumentType]
.where(Job.status == JobStatus.QUEUED)
.order_by(Job.date_created) # pyright: ignore[reportArgumentType]
.order_by(Job.created_at) # pyright: ignore[reportArgumentType]
)
return (await _session.exec(query)).first()
async def requeue_stale_processing_jobs(
self,
*,
stale_before: datetime,
session: AsyncSession | None = None,
) -> int:
"""Move stale processing jobs back to queued state.
Jobs with ``status=PROCESSING`` and ``date_updated`` older than
``stale_before`` are considered stale and re-queued.
"""
async with self._session_scope(session) as _session:
query = (
select(Job)
.where(Job.status == JobStatus.PROCESSING)
.where(Job.date_updated < stale_before)
)
stale_jobs = (await _session.exec(query)).all()
if not stale_jobs:
return 0
now = datetime.now(UTC)
for job in stale_jobs:
job.status = JobStatus.QUEUED
job.date_updated = now
await self._finalize(session=_session, caller_session=session, refresh=stale_jobs)
return len(stale_jobs)
src/transcription/services/store.py
+2 -13
View File
@@ -13,7 +13,6 @@ from transcription.errors import ErrorCategory
from ..models import Document
from ..models import Job
from ..models import Source
from .documents import UploadJobResult
logger = logging.getLogger(__name__)
@@ -70,24 +69,14 @@ async def _create_upload_records(
stored_path: Path,
) -> tuple[Document, Job]:
document = Document(
name=Path(original_filename).name,
filename=Path(original_filename).name,
file_path=str(stored_path),
)
session.add(document)
await session.flush()
job = Job(document_id=document.id)
session.add(job)
await session.flush()
source = Source(
document_id=document.id,
job_id=job.id,
upload_name=Path(original_filename).name,
filename=stored_path.name,
file_path=str(stored_path),
)
session.add(source)
await session.commit()
await session.refresh(document)
await session.refresh(job)
src/transcription/services/transcription.py
+150 -127
View File
@@ -4,10 +4,7 @@ from __future__ import annotations
import logging
import mimetypes
from collections.abc import Sequence
from contextlib import contextmanager
from datetime import UTC
from datetime import datetime
from pathlib import Path
from uuid import UUID
@@ -20,9 +17,8 @@ from transcription.config import Settings
from transcription.config import get_settings
from transcription.errors import AppError
from transcription.errors import ErrorCategory
from transcription.models import Job
from transcription.models import Revision
from transcription.models import Source
from transcription.models import Transcript
from transcription.models import TranscriptRevision
from transcription.providers import ProviderAuthError
from transcription.providers import ProviderError
from transcription.providers import ProviderResponseError
@@ -47,11 +43,13 @@ class TranscriptionError(AppError):
class TranscriptionNotFoundError(TranscriptionError):
"""Raised when a transcription-related resource is not found."""
"""Raised when a transcription is not found in the database."""
class TranscriptionService(ServiceBase):
"""Service class for job transcription output and optional source revisions."""
"""Service class for managing transcription operations.
This is the top-level service that composes functionality from the other services."""
provider: TranscriptionProvider
@@ -59,51 +57,98 @@ class TranscriptionService(ServiceBase):
super().__init__(session_factory=session_factory)
self.provider = get_transcription_provider(settings=self.settings)
async def create_revision(self, revision: Revision, *, session: AsyncSession | None = None) -> Revision:
"""Create a new revision in the database."""
async def create_transcript(self, transcript: Transcript, *, session: AsyncSession | None = None) -> Transcript:
"""Create a new transcript in the database."""
async with self._session_scope(session) as _session:
_session.add(revision)
await self._finalize(session=_session, caller_session=session, refresh=(revision,))
return revision
_session.add(transcript)
await self._finalize(session=_session, caller_session=session, refresh=(transcript,))
return transcript
async def read_revision(self, revision_id: UUID, *, session: AsyncSession | None = None) -> Revision:
"""Read an existing revision from the database."""
async def create_transcript_revision(
self,
transcript_revision: TranscriptRevision,
*,
session: AsyncSession | None = None,
) -> TranscriptRevision:
"""Create a new transcript revision in the database."""
async with self._session_scope(session) as _session:
revision = await _session.get(
Revision,
revision_id,
options=(selectinload(Revision.source),), # pyright: ignore[reportArgumentType]
_session.add(transcript_revision)
await self._finalize(session=_session, caller_session=session, refresh=(transcript_revision,))
return transcript_revision
async def read_transcript(self, transcript_id: UUID, *, session: AsyncSession | None = None) -> Transcript:
"""Read an existing transcript from the database."""
async with self._session_scope(session) as _session:
transcript = await _session.get(
Transcript,
transcript_id,
# Makes the full Job model object available in the return Transcript object
options=(selectinload(Transcript.job),), # pyright: ignore[reportArgumentType]
)
if revision is None:
if transcript is None:
raise TranscriptionNotFoundError(
f"Revision with id {revision_id} not found",
f"Transcript with id {transcript_id} not found",
category=ErrorCategory.NOT_FOUND,
suggestion="Verify the revision id and retry.",
suggestion="Verify the transcript id and retry.",
)
return revision
return transcript
async def update_revision(self, revision: Revision, *, session: AsyncSession | None = None) -> Revision:
"""Update an existing revision in the database."""
async def read_transcript_revision(
self,
transcript_revision_id: UUID,
*,
session: AsyncSession | None = None,
) -> TranscriptRevision:
"""Read an existing transcript revision from the database."""
async with self._session_scope(session) as _session:
merged = await _session.merge(revision)
transcript_revision = await _session.get(
TranscriptRevision,
transcript_revision_id,
options=(selectinload(TranscriptRevision.job),), # pyright: ignore[reportArgumentType]
)
if transcript_revision is None:
raise TranscriptionNotFoundError(
f"Transcript revision with id {transcript_revision_id} not found",
category=ErrorCategory.NOT_FOUND,
suggestion="Verify the transcript revision id and retry.",
)
return transcript_revision
async def update_transcript(self, transcript: Transcript, *, session: AsyncSession | None = None) -> Transcript:
"""Update an existing transcript in the database."""
async with self._session_scope(session) as _session:
merged = await _session.merge(transcript)
await self._finalize(session=_session, caller_session=session, refresh=(merged,))
return merged
async def delete_revision(self, revision: Revision, *, session: AsyncSession | None = None) -> None:
"""Delete a revision from the database."""
async def update_transcript_revision(
self,
transcript_revision: TranscriptRevision,
*,
session: AsyncSession | None = None,
) -> TranscriptRevision:
"""Update an existing transcript revision in the database."""
async with self._session_scope(session) as _session:
await _session.delete(revision)
merged = await _session.merge(transcript_revision)
await self._finalize(session=_session, caller_session=session, refresh=(merged,))
return merged
async def delete_transcript(self, transcript: Transcript, *, session: AsyncSession | None = None) -> None:
"""Delete a transcript from the database."""
async with self._session_scope(session) as _session:
await _session.delete(transcript)
await self._finalize(session=_session, caller_session=session)
# Temporary compatibility methods for callers still using transcript naming.
async def read_transcript(self, transcript_id: UUID, *, session: AsyncSession | None = None) -> Revision:
"""Backward-compatible alias for read_revision."""
return await self.read_revision(transcript_id, session=session)
async def delete_transcript(self, transcript: Revision, *, session: AsyncSession | None = None) -> None:
"""Backward-compatible alias for delete_revision."""
await self.delete_revision(transcript, session=session)
async def delete_transcript_revision(
self,
transcript_revision: TranscriptRevision,
*,
session: AsyncSession | None = None,
) -> None:
"""Delete a transcript revision from the database."""
async with self._session_scope(session) as _session:
await _session.delete(transcript_revision)
await self._finalize(session=_session, caller_session=session)
async def transcribe_document(
self,
@@ -112,7 +157,7 @@ class TranscriptionService(ServiceBase):
*,
prompt_name: str = DEFAULT_PROMPT_FILE,
session: AsyncSession | None = None,
) -> None:
):
"""Transcribe a local image using the configured prompt and provider."""
result = await transcribe_document_image(
image_path=image_path,
@@ -120,118 +165,96 @@ class TranscriptionService(ServiceBase):
settings=self.settings,
provider=self.provider,
)
await self.update_job_transcription(
job_id=job_id,
text=result.text,
error_detail=None,
provider=result.provider,
model=result.model,
prompt_name=result.prompt_name,
session=session,
)
await self.create_transcript(transcript=result.to_transcript(job_id=job_id), session=session)
async def update_job_transcription(
async def upsert_transcript_by_job(
self,
*,
job_id: UUID,
text: str | None,
error_detail: str | None = None,
error_detail: str | None,
provider: str | None = None,
model: str | None = None,
prompt_name: str = DEFAULT_PROMPT_FILE,
model: str | None = None,
session: AsyncSession | None = None,
) -> Job:
"""Persist original transcription output fields on a job."""
) -> Transcript:
"""Create or update a transcript for a job id."""
async with self._session_scope(session) as _session:
job = await _session.get(Job, job_id)
if job is None:
raise TranscriptionNotFoundError(
f"Job with id {job_id} not found",
category=ErrorCategory.NOT_FOUND,
suggestion="Verify the job id and retry.",
transcript = (await _session.exec(select(Transcript).where(Transcript.job_id == job_id))).first()
if transcript is None:
transcript = Transcript(
job_id=job_id,
provider=provider or self.settings.provider.value,
prompt_name=prompt_name,
)
job.text = text
job.error_detail = error_detail
job.provider = provider or job.provider or self.settings.provider.value
job.model = model or job.model or _resolve_transcript_model(provider=self.provider, settings=self.settings)
job.prompt_name = prompt_name or job.prompt_name or DEFAULT_PROMPT_FILE
job.date_updated = datetime.now(UTC)
transcript.text = text
transcript.error_detail = error_detail
if provider is not None:
transcript.provider = provider
transcript.prompt_name = prompt_name
if model is not None:
transcript.model = model
await self._finalize(session=_session, caller_session=session, refresh=(job,))
return job
_session.add(transcript)
await self._finalize(session=_session, caller_session=session, refresh=(transcript,))
return transcript
async def upsert_revision_for_source(
async def list_transcript_revisions_by_job(
self,
*,
source_id: UUID,
text: str,
job_id: UUID,
session: AsyncSession | None = None,
) -> Revision:
"""Create or replace the single optional revision for a source."""
) -> list[TranscriptRevision]:
"""Return transcript revisions for a job ordered by version number."""
async with self._session_scope(session) as _session:
source = await _session.get(Source, source_id)
if source is None:
raise TranscriptionNotFoundError(
f"Source with id {source_id} not found",
category=ErrorCategory.NOT_FOUND,
suggestion="Verify the source id and retry.",
revisions = (
await _session.exec(
select(TranscriptRevision)
.where(TranscriptRevision.job_id == job_id)
.order_by(TranscriptRevision.version_number)
)
).all()
return list(revisions)
query = select(Revision).where(Revision.source_id == source_id)
existing = (await _session.exec(query)).one_or_none()
async def append_transcript_revision(
self,
*,
job_id: UUID,
text: str | None,
error_detail: str | None,
provider: str,
prompt_name: str,
model: str | None,
source: str,
session: AsyncSession | None = None,
) -> TranscriptRevision:
"""Append a new transcript revision and allocate the next version number."""
async with self._session_scope(session) as _session:
latest_version = (
await _session.exec(
select(TranscriptRevision.version_number)
.where(TranscriptRevision.job_id == job_id)
.order_by(TranscriptRevision.version_number.desc())
.limit(1)
)
).first()
next_version = 1 if latest_version is None else latest_version + 1
if existing is None:
revision = Revision(source_id=source_id, text=text)
revision = TranscriptRevision(
job_id=job_id,
version_number=next_version,
provider=provider,
prompt_name=prompt_name,
model=model,
source=source,
text=text,
error_detail=error_detail,
)
_session.add(revision)
await self._finalize(session=_session, caller_session=session, refresh=(revision,))
return revision
existing.text = text
merged = await _session.merge(existing)
await self._finalize(session=_session, caller_session=session, refresh=(merged,))
return merged
async def read_revision_by_source(
self,
source_id: UUID,
*,
session: AsyncSession | None = None,
) -> Revision | None:
"""Read the single optional revision for a source."""
async with self._session_scope(session) as _session:
query = select(Revision).where(Revision.source_id == source_id)
result = await _session.exec(query)
return result.one_or_none()
async def list_revisions_by_job(
self,
job_id: UUID,
*,
session: AsyncSession | None = None,
) -> Sequence[Revision]:
"""List revisions connected to all sources for a job."""
async with self._session_scope(session) as _session:
query = (
select(Revision)
.join(Source, Source.id == Revision.source_id)
.where(Source.job_id == job_id)
.order_by(Revision.date_created) # pyright: ignore[reportArgumentType]
)
result = await _session.exec(query)
return result.all()
def _resolve_transcript_model(*, provider: TranscriptionProvider, settings: Settings) -> str:
provider_model = getattr(provider, "model", None)
if isinstance(provider_model, str) and provider_model.strip():
return provider_model
if settings.provider_model and settings.provider_model.strip():
return settings.provider_model
return "unknown"
async def transcribe_document_image(
image_path: str | Path,
src/transcription/services/workflows.py
+95 -123
View File
@@ -6,12 +6,10 @@ from sqlmodel.ext.asyncio.session import AsyncSession
from ..config import Settings
from ..config import get_settings
from ..errors import AppError
from ..errors import ErrorCategory
from ..errors import classify_unexpected_error
from ..errors import format_error_detail
from ..models import Job
from ..models import JobStatus
from ..models import Source
from ..providers import TranscriptionResult
from . import ServiceBundle
from .transcription import DEFAULT_PROMPT_FILE
@@ -30,7 +28,7 @@ async def advance_job(
settings = settings or get_settings()
match job.status:
case JobStatus.QUEUED:
return await process_queued_job(job=job, services=services, settings=settings, session=session)
return await process_queued_job(job=job, services=services, session=session)
case JobStatus.FAILED:
if job.retry_count < settings.worker_max_retries:
return await services.jobs.update_job_state(
@@ -50,11 +48,9 @@ async def process_queued_job(
*,
job: Job,
services: ServiceBundle,
settings: Settings | None = None,
session: AsyncSession | None = None,
) -> Job | None:
"""Process one complete transcription attempt for a queued job."""
runtime_settings = settings or get_settings()
if job.status != JobStatus.QUEUED:
logger.warning(f"Job {job.id} is not queued. Current status: {job.status}")
return
@@ -68,62 +64,20 @@ async def process_queued_job(
job = await services.jobs.mark_job_status(job.id, JobStatus.PROCESSING, session=session)
await session.commit()
source_job = await services.jobs.read_job(job_id=job.id, session=session)
source = _resolve_primary_source(source_job)
assert source is not None, f"Job {job.id} has no associated source record."
started_at = asyncio.get_running_loop().time()
document = job.document
assert document is not None, (
f"Job {job.id} has no associated document or the document failed to be loaded by the job service."
)
try:
result = await asyncio.wait_for(
transcribe_document_image(source.file_path),
timeout=runtime_settings.worker_provider_timeout_seconds,
)
elapsed_seconds = asyncio.get_running_loop().time() - started_at
logger.info(
"Provider response diagnostics operation=worker.provider_response "
"job_id=%s document_id=%s source_id=%s provider=%s model=%s "
"finish_reason=%s usage_input_tokens=%s usage_output_tokens=%s usage_total_tokens=%s "
"latency_seconds=%.3f text_chars=%s text_lines=%s",
job.id,
job.document_id,
source.id,
result.provider,
result.model,
result.finish_reason or "unknown",
result.usage_input_tokens,
result.usage_output_tokens,
result.usage_total_tokens,
elapsed_seconds,
len(result.text),
_line_count(result.text),
)
_validate_transcription_quality(result=result, settings=runtime_settings)
result = await transcribe_document_image(document.file_path)
job = await _finalize_transcribed(job=job, services=services, result=result, session=session)
logger.info(
"Job transcribed operation=worker.process_job job_id=%s document_id=%s source_id=%s provider=%s",
"Job transcribed operation=worker.process_job job_id=%s document_id=%s provider=%s",
job.id,
job.document_id,
source.id,
document.id,
result.provider,
)
except TimeoutError:
error = AppError(
f"Provider call timed out after {runtime_settings.worker_provider_timeout_seconds:.1f}s",
category=ErrorCategory.EXTERNAL_PROVIDER,
suggestion="Retry the job. If this repeats, verify provider latency and request payload size.",
retriable=True,
)
job = await _finalize_failed(job=job, services=services, error=error, session=session)
logger.error(
"Job failed operation=worker.process_job job_id=%s document_id=%s source_id=%s error_id=%s category=%s",
job.id,
job.document_id,
source.id,
error.error_id,
error.category.value,
)
except Exception as exc: # noqa: BLE001
match exc:
case AppError() as error:
@@ -133,10 +87,9 @@ async def process_queued_job(
job = await _finalize_failed(job=job, services=services, error=error, session=session)
logger.error(
"Job failed operation=worker.process_job job_id=%s document_id=%s source_id=%s error_id=%s category=%s",
"Job failed operation=worker.process_job job_id=%s document_id=%s error_id=%s category=%s",
job.id,
job.document_id,
source.id,
document.id,
error.error_id,
error.category.value,
)
@@ -165,16 +118,27 @@ async def _finalize_transcribed(
result: TranscriptionResult,
session: AsyncSession | None = None,
) -> Job:
"""Transaction B: job transcription output + TRANSCRIBED in one commit."""
"""Transaction B: transcript + TRANSCRIBED in one commit."""
if session is None:
async with services.jobs._session_scope() as local_session:
await services.transcriptions.update_job_transcription(
prompt_name = result.prompt_name or DEFAULT_PROMPT_FILE
await services.transcriptions.upsert_transcript_by_job(
job_id=job.id,
text=result.text,
error_detail=None,
provider=result.provider,
prompt_name=prompt_name,
model=result.model,
prompt_name=result.prompt_name,
session=local_session,
)
await services.transcriptions.append_transcript_revision(
job_id=job.id,
text=result.text,
error_detail=None,
provider=result.provider,
prompt_name=prompt_name,
model=result.model,
source="ai",
session=local_session,
)
updated_job = await services.jobs.mark_job_status(
@@ -185,13 +149,24 @@ async def _finalize_transcribed(
await local_session.commit()
return updated_job
await services.transcriptions.update_job_transcription(
prompt_name = result.prompt_name or DEFAULT_PROMPT_FILE
await services.transcriptions.upsert_transcript_by_job(
job_id=job.id,
text=result.text,
error_detail=None,
provider=result.provider,
prompt_name=prompt_name,
model=result.model,
prompt_name=result.prompt_name,
session=session,
)
await services.transcriptions.append_transcript_revision(
job_id=job.id,
text=result.text,
error_detail=None,
provider=result.provider,
prompt_name=prompt_name,
model=result.model,
source="ai",
session=session,
)
updated_job = await services.jobs.mark_job_status(
@@ -211,14 +186,27 @@ async def _finalize_retry(
settings: Settings,
session: AsyncSession | None = None,
) -> Job:
"""Transaction C: job error detail + QUEUED + retry increment in one commit."""
"""Transaction C: transcript error + QUEUED + retry increment in one commit."""
if session is None:
async with services.jobs._session_scope() as local_session:
await services.transcriptions.update_job_transcription(
provider_name = services.transcriptions.settings.provider.value
await services.transcriptions.upsert_transcript_by_job(
job_id=job.id,
text=None,
error_detail=format_error_detail(error),
provider=provider_name,
prompt_name=DEFAULT_PROMPT_FILE,
model=services.transcriptions.settings.provider_model,
session=local_session,
)
await services.transcriptions.append_transcript_revision(
job_id=job.id,
text=None,
error_detail=format_error_detail(error),
provider=provider_name,
prompt_name=DEFAULT_PROMPT_FILE,
model=services.transcriptions.settings.provider_model,
source="ai",
session=local_session,
)
updated_job = await services.jobs.update_job_state(
@@ -229,11 +217,24 @@ async def _finalize_retry(
)
await local_session.commit()
else:
await services.transcriptions.update_job_transcription(
provider_name = services.transcriptions.settings.provider.value
await services.transcriptions.upsert_transcript_by_job(
job_id=job.id,
text=None,
error_detail=format_error_detail(error),
provider=provider_name,
prompt_name=DEFAULT_PROMPT_FILE,
model=services.transcriptions.settings.provider_model,
session=session,
)
await services.transcriptions.append_transcript_revision(
job_id=job.id,
text=None,
error_detail=format_error_detail(error),
provider=provider_name,
prompt_name=DEFAULT_PROMPT_FILE,
model=services.transcriptions.settings.provider_model,
source="ai",
session=session,
)
updated_job = await services.jobs.update_job_state(
@@ -256,14 +257,27 @@ async def _finalize_failed(
error: AppError,
session: AsyncSession | None = None,
) -> Job:
"""Transaction B: job error detail + FAILED in one commit."""
"""Transaction B: transcript error + FAILED in one commit."""
if session is None:
async with services.jobs._session_scope() as local_session:
await services.transcriptions.update_job_transcription(
provider_name = services.transcriptions.settings.provider.value
await services.transcriptions.upsert_transcript_by_job(
job_id=job.id,
text=None,
error_detail=format_error_detail(error),
provider=provider_name,
prompt_name=DEFAULT_PROMPT_FILE,
model=services.transcriptions.settings.provider_model,
session=local_session,
)
await services.transcriptions.append_transcript_revision(
job_id=job.id,
text=None,
error_detail=format_error_detail(error),
provider=provider_name,
prompt_name=DEFAULT_PROMPT_FILE,
model=services.transcriptions.settings.provider_model,
source="ai",
session=local_session,
)
updated_job = await services.jobs.mark_job_status(
@@ -274,11 +288,24 @@ async def _finalize_failed(
await local_session.commit()
return updated_job
await services.transcriptions.update_job_transcription(
provider_name = services.transcriptions.settings.provider.value
await services.transcriptions.upsert_transcript_by_job(
job_id=job.id,
text=None,
error_detail=format_error_detail(error),
provider=provider_name,
prompt_name=DEFAULT_PROMPT_FILE,
model=services.transcriptions.settings.provider_model,
session=session,
)
await services.transcriptions.append_transcript_revision(
job_id=job.id,
text=None,
error_detail=format_error_detail(error),
provider=provider_name,
prompt_name=DEFAULT_PROMPT_FILE,
model=services.transcriptions.settings.provider_model,
source="ai",
session=session,
)
updated_job = await services.jobs.mark_job_status(
@@ -288,58 +315,3 @@ async def _finalize_failed(
)
await session.commit()
return updated_job
def _resolve_primary_source(job: Job) -> Source | None:
if not job.sources:
return None
return job.sources[0]
def _validate_transcription_quality(*, result: TranscriptionResult, settings: Settings) -> None:
text_chars = len(result.text)
text_lines = _line_count(result.text)
if settings.worker_fail_on_finish_reason_length and (result.finish_reason or "").lower() == "length":
raise AppError(
"Provider output appears truncated (finish_reason=length)",
category=ErrorCategory.EXTERNAL_PROVIDER,
suggestion=(
"Retry the job. If this repeats, use a faster model, reduce input complexity, "
"or increase provider output budget."
),
retriable=True,
)
if settings.worker_min_transcription_chars > 0 and text_chars < settings.worker_min_transcription_chars:
raise AppError(
(
"Transcription output below configured minimum character threshold "
f"({text_chars} < {settings.worker_min_transcription_chars})"
),
category=ErrorCategory.EXTERNAL_PROVIDER,
suggestion=(
"Retry the job. If this repeats, switch model or raise minimum thresholds based on document type."
),
retriable=True,
)
if settings.worker_min_transcription_lines > 0 and text_lines < settings.worker_min_transcription_lines:
raise AppError(
(
"Transcription output below configured minimum line threshold "
f"({text_lines} < {settings.worker_min_transcription_lines})"
),
category=ErrorCategory.EXTERNAL_PROVIDER,
suggestion=(
"Retry the job. If this repeats, switch model or raise minimum thresholds based on document type."
),
retriable=True,
)
def _line_count(text: str) -> int:
stripped = text.strip()
if not stripped:
return 0
return sum(1 for line in stripped.splitlines() if line.strip())
src/transcription/ui/__init__.py
+1 -32
View File
@@ -1,45 +1,14 @@
"""UI page registration exports."""
from pathlib import Path
from fastapi import FastAPI
from nicegui import app as nicegui_app
from nicegui import ui
from transcription.ui.pages.jobs_page import register_page as register_jobs_page
from transcription.ui.pages.upload_page import register_page as register_upload_page
_THEME_REGISTERED_STATE_KEY = "transcription_ui_theme_registered"
_THEME_COLORS: dict[str, str] = {
"primary": "#6f97e8",
"secondary": "#92b5f5",
"accent": "#7fc0de",
"dark": "#22304a",
"dark_page": "#1a2538",
"positive": "#86c8ad",
"negative": "#d98a9a",
"info": "#7ebdda",
"warning": "#e2c083",
}
def _register_global_styles(app: FastAPI) -> None:
if getattr(app.state, _THEME_REGISTERED_STATE_KEY, False):
return
nicegui_app.colors(**_THEME_COLORS)
css_path = Path(__file__).resolve().parent / "static" / "colors.css"
if css_path.exists():
ui.add_css(css_path, shared=True)
setattr(app.state, _THEME_REGISTERED_STATE_KEY, True)
def register_pages(app: FastAPI) -> None:
"""Register all NiceGUI pages and mount them onto the FastAPI app."""
_register_global_styles(app)
register_upload_page()
register_jobs_page()
ui.run_with(app, mount_path="/ui", show_welcome_message=False, dark=True)
ui.run_with(app, mount_path="/ui", show_welcome_message=False)
src/transcription/ui/components/__init__.py
-7
View File
@@ -1,7 +0,0 @@
"""Reusable UI component exports."""
from transcription.ui.components.app_shell import NAV_ITEMS
from transcription.ui.components.app_shell import render_navigation_header
from transcription.ui.components.document_panzoom import render_document_panzoom
__all__ = ["NAV_ITEMS", "render_document_panzoom", "render_navigation_header"]
src/transcription/ui/components/app_shell.py
-59
View File
@@ -1,59 +0,0 @@
"""Reusable app shell primitives for page-level layout."""
from __future__ import annotations
from nicegui import ui
NAV_ITEMS: tuple[tuple[str, str, str], ...] = (
("Upload", "/upload", "upload_file"),
("Jobs", "/jobs", "work_history"),
)
def _is_active_path(*, current_path: str, item_path: str) -> bool:
if item_path == "/jobs":
return current_path == "/jobs" or current_path.startswith("/jobs/")
return current_path == item_path
def _button_props(*, icon: str, is_active: bool) -> str:
if is_active:
return f"icon={icon} no-caps unelevated color=primary text-color=white"
return f"icon={icon} no-caps outline color=secondary text-color=secondary"
def _button_classes(*, is_active: bool) -> str:
base = "w-full sm:w-auto min-h-[40px] px-3 rounded-lg text-body2 text-weight-medium"
if is_active:
return f"{base}"
return f"{base}"
def _render_nav_button(*, label: str, path: str, icon: str, current_path: str) -> None:
is_active = _is_active_path(current_path=current_path, item_path=path)
button = ui.button(
label,
icon=icon,
on_click=lambda _=None, route=path: ui.navigate.to(route),
)
button.props(_button_props(icon=icon, is_active=is_active)).classes(_button_classes(is_active=is_active))
def _normalize_path(current_path: str | None) -> str:
normalized = (current_path or "").strip()
if not normalized:
return "/upload"
return normalized.rstrip("/") or "/"
def render_navigation_header(*, current_path: str | None = None) -> None:
"""Render a shared app header with links for top-level pages."""
normalized_path = _normalize_path(current_path)
with (
ui.header(elevated=True).props("bordered").classes("bg-dark text-white q-px-sm q-py-xs"),
ui.row().classes("w-full items-center justify-end q-gutter-xs"),
ui.element("div").classes("w-full grid grid-cols-2 gap-2 sm:flex sm:justify-start sm:gap-2"),
):
for label, path, icon in NAV_ITEMS:
_render_nav_button(label=label, path=path, icon=icon, current_path=normalized_path)
src/transcription/ui/components/document_panzoom.py
-211
View File
@@ -1,211 +0,0 @@
"""Panzoom-backed document preview component."""
from __future__ import annotations
from functools import lru_cache
from pathlib import Path
from urllib.parse import quote
from uuid import uuid4
from nicegui import ui
from transcription.config import get_settings
from transcription.models import Source
PANGOZOOM_CDN_URL = "https://unpkg.com/@panzoom/[email protected]/dist/panzoom.min.js"
UPLOADS_URL_PREFIX = "/uploads"
def render_document_panzoom(*, source: Source) -> None:
"""Render a source preview with pan and zoom interactions."""
_register_panzoom_assets()
host_id = f"document-panzoom-{uuid4().hex}"
document_url = _document_url(source)
document_kind = _document_kind(source)
with ui.card().classes("w-full q-pa-md"):
with ui.row().classes("w-full items-center justify-between no-wrap"):
ui.label("Document preview").classes("text-subtitle1 text-weight-medium")
ui.label(source.filename).classes("text-caption text-grey-4 ellipsis").style(
"max-width: 60%; text-align: right;"
)
with (
ui.element("div").classes("w-full document-panzoom-host rounded-borders q-mt-md")
# .style(f"height: {height};")
) as host:
host.props(f"id={host_id}")
with ui.element("div").classes("document-panzoom-surface"):
if document_kind == "pdf":
ui.html(
f'<iframe class="document-panzoom-iframe" '
f'src="{document_url}" title="{source.filename}" '
"data-panzoom-target></iframe>"
)
else:
ui.html(
f'<img class="document-panzoom-media" '
f'src="{document_url}" alt="{source.filename}" '
"data-panzoom-target data-panzoom-media />"
)
_attach_panzoom(host_id)
@lru_cache(maxsize=1)
def _register_panzoom_assets() -> None:
ui.add_head_html(
f'<script src="{PANGOZOOM_CDN_URL}"></script>',
shared=True,
)
ui.add_head_html(
"""
<style>
.document-panzoom-host {
overflow: hidden;
touch-action: none;
}
.document-panzoom-surface {
width: 100%;
height: 100%;
display: flex;
align-items: flex-start;
justify-content: flex-start;
}
.document-panzoom-media {
width: auto;
height: auto;
display: block;
max-width: 100%;
max-height: 100%;
user-select: none;
-webkit-user-drag: none;
}
.document-panzoom-iframe {
width: 100%;
height: 100%;
border: 0;
pointer-events: none;
background: white;
}
</style>
""",
shared=True,
)
def _document_url(source: Source) -> str:
file_path = Path(source.file_path)
upload_dir = get_settings().upload_dir
relative_path: Path
try:
relative_path = file_path.resolve().relative_to(upload_dir.resolve())
except ValueError:
parts = file_path.parts
if "uploads" in parts:
uploads_index = parts.index("uploads")
relative_path = Path(*parts[uploads_index + 1 :])
else:
relative_path = Path(file_path.name)
encoded_relative_path = "/".join(quote(part) for part in relative_path.parts)
return f"{UPLOADS_URL_PREFIX}/{encoded_relative_path}"
def _document_kind(source: Source) -> str:
suffix = Path(source.file_path).suffix.lower()
if suffix == ".pdf":
return "pdf"
return "image"
def _attach_panzoom(host_id: str) -> None:
ui.run_javascript(
f"""
(function() {{
if (!window.Panzoom) return;
window.__transcriptionPanzoom = window.__transcriptionPanzoom || {{}};
const host = document.getElementById({host_id!r});
if (!host) return;
const target = host.querySelector('[data-panzoom-target]');
const media = host.querySelector('[data-panzoom-media]');
if (!target) return;
const cleanup = () => {{
const existing = window.__transcriptionPanzoom[{host_id!r}];
if (existing?.resizeObserver) existing.resizeObserver.disconnect();
if (existing?.wheelHandler) host.removeEventListener('wheel', existing.wheelHandler);
if (existing?.instance) existing.instance.destroy();
}};
const computeFitScale = () => {{
const hostRect = host.getBoundingClientRect();
if (hostRect.width <= 0 || hostRect.height <= 0) return null;
return 1;
}};
const buildInstance = () => {{
cleanup();
const fitScale = computeFitScale();
if (fitScale === null) return false;
const minScale = Math.min(fitScale, 0.01);
const instance = Panzoom(target, {{
startX: 0,
startY: 0,
startScale: fitScale,
minScale: minScale,
maxScale: 256,
step: 0.2,
roundPixels: false,
panOnlyWhenZoomed: true,
overflow: 'hidden',
}});
const wheelHandler = (event) => instance.zoomWithWheel(event);
host.addEventListener('wheel', wheelHandler, {{ passive: false }});
requestAnimationFrame(() => {{
instance.reset({{ animate: false }});
}});
const resizeObserver = new ResizeObserver(() => {{
const nextFitScale = computeFitScale();
if (nextFitScale === null) return;
instance.setOptions({{
startScale: nextFitScale,
minScale: Math.min(nextFitScale, 0.01),
}});
instance.reset({{ animate: false }});
}});
resizeObserver.observe(host);
window.__transcriptionPanzoom[{host_id!r}] = {{
instance,
wheelHandler,
resizeObserver,
}};
return true;
}};
const initWhenReady = (retries = 15) => {{
if (buildInstance()) return;
if (retries <= 0) return;
requestAnimationFrame(() => initWhenReady(retries - 1));
}};
if (media && media.tagName === 'IMG' && !media.complete) {{
media.addEventListener('load', () => initWhenReady(), {{ once: true }});
return;
}}
initWhenReady();
}})();
"""
)
src/transcription/ui/components/job_detail.py
+251 -38
View File
@@ -2,18 +2,46 @@
from __future__ import annotations
import logging
import base64
import mimetypes
from collections.abc import Awaitable
from collections.abc import Callable
from dataclasses import dataclass
from pathlib import Path
from nicegui import ui
from transcription.models import Document
from transcription.models import Job
from transcription.models import Revision
from transcription.models import Source
from transcription.ui.components.document_panzoom import render_document_panzoom
from transcription.ui.components.transcript import render_original_transcription_card
from transcription.ui.components.transcript import render_revision_row
from transcription.models import Transcript
from transcription.models import TranscriptRevision
logger = logging.getLogger(__name__)
@dataclass(frozen=True)
class RevisionDisplayRow:
id: str
created: str
version: str
text: str
error_detail: str | None
def _extract_row_id(args: object) -> str | None:
if isinstance(args, dict):
if isinstance(args.get("row"), dict):
row_id = args["row"].get("id")
return str(row_id) if row_id is not None else None
row_id = args.get("id")
return str(row_id) if row_id is not None else None
if isinstance(args, list):
for value in args:
if isinstance(value, dict):
row_id = value.get("id")
if row_id is not None:
return str(row_id)
return None
def _status_chip_classes(status: str) -> str:
@@ -29,47 +57,229 @@ def _status_chip_classes(status: str) -> str:
def _metadata_row(label: str, value: str) -> None:
with ui.row().classes("w-full items-start justify-between q-gutter-x-md"):
ui.label(label).classes("text-caption text-grey-5 text-uppercase w-28")
ui.label(value).classes("text-body2 text-right text-grey-1 break-all")
with ui.row().classes("w-full items-start justify-between no-wrap q-gutter-x-md"):
ui.label(label).classes("text-caption text-grey-7 text-uppercase")
ui.label(value).classes("text-body2 text-right")
def _render_source_section(source: Source) -> None:
with ui.card().classes("w-full q-pa-md"):
ui.label("Source").classes("text-subtitle1 text-weight-medium")
def _render_document_section(document: Document) -> None:
with ui.card().classes("w-full bg-grey-1 q-pa-md"):
ui.label("Document").classes("text-subtitle1 text-weight-medium")
ui.separator().classes("q-my-sm")
with ui.column().classes("w-full q-gutter-y-xs"):
_metadata_row("Upload name", source.upload_name)
_metadata_row("Stored filename", source.filename)
_metadata_row("File path", source.file_path)
_metadata_row("Uploaded", source.date_uploaded.isoformat())
ui.separator().classes("q-my-md")
render_document_panzoom(source=source)
_metadata_row("Filename", document.filename)
_metadata_row("File path", document.file_path)
def _render_revision_section(revision: Revision | None) -> None:
with ui.card().classes("w-full q-pa-md"):
ui.label("Revision").classes("text-subtitle1 text-weight-medium")
ui.separator().classes("q-my-sm")
def _document_data_url(document: Document) -> tuple[str | None, str | None]:
path = Path(document.file_path)
if not path.exists() or not path.is_file():
return None, "Document preview unavailable: file not found"
if revision is None:
ui.label("No revision exists for this source.").classes("text-body2 text-grey-3")
suffix = path.suffix.lower()
mime_type, _ = mimetypes.guess_type(path.name)
if suffix in {".tif", ".tiff"}:
mime_type = "image/tiff"
if mime_type is None:
return None, "Document preview unavailable: unsupported MIME type"
encoded = base64.b64encode(path.read_bytes()).decode("ascii")
return f"data:{mime_type};base64,{encoded}", None
def _render_document_preview(document: Document) -> None:
source, error = _document_data_url(document)
if error is not None or source is None:
ui.label(error or "Document preview unavailable").classes("text-caption text-grey-7")
return
render_revision_row(revision=revision, initially_expanded=True)
suffix = Path(document.file_path).suffix.lower()
if suffix == ".pdf":
ui.html(
(
'<iframe title="Document preview" '
f'src="{source}" '
'style="width:100%;height:520px;border:1px solid #ddd;border-radius:8px;"></iframe>'
)
)
ui.label("Zoom controls are currently available for image files.").classes("text-caption text-grey-7 q-mt-sm")
return
zoom_percent = {"value": 100}
with ui.element("div").style(
"width:100%;height:520px;overflow:auto;border:1px solid #ddd;border-radius:8px;padding:8px;background:#fafafa;"
):
image = ui.image(source).classes("rounded-borders").style("width:100%;max-width:none;")
zoom_label = ui.label("Zoom: 100%").classes("text-caption text-grey-7 q-mt-sm")
def _apply_zoom() -> None:
image.style(f"width:{zoom_percent['value']}%;max-width:none;")
image.update()
zoom_label.text = f"Zoom: {zoom_percent['value']}%"
zoom_label.update()
def _zoom_in() -> None:
zoom_percent["value"] = min(300, zoom_percent["value"] + 25)
_apply_zoom()
def _zoom_out() -> None:
zoom_percent["value"] = max(50, zoom_percent["value"] - 25)
_apply_zoom()
def _zoom_reset() -> None:
zoom_percent["value"] = 100
_apply_zoom()
with ui.row().classes("q-gutter-sm q-mt-xs"):
ui.button("-", on_click=_zoom_out)
ui.button("+", on_click=_zoom_in)
ui.button("Reset", on_click=_zoom_reset)
def render_job_detail(*, job: Job, source: Source | None, revision: Revision | None) -> None:
def _build_display_rows(transcript: Transcript, revisions: list[TranscriptRevision]) -> list[RevisionDisplayRow]:
ordered = sorted(revisions, key=lambda revision: revision.version_number)
rows: list[RevisionDisplayRow] = []
if ordered:
first = ordered[0]
rows.append(
RevisionDisplayRow(
id="original",
created=first.created_at.isoformat(),
version="original",
text=first.text or "",
error_detail=first.error_detail,
)
)
for revision in ordered[1:]:
rows.append(
RevisionDisplayRow(
id=str(revision.version_number),
created=revision.created_at.isoformat(),
version=str(revision.version_number),
text=revision.text or "",
error_detail=revision.error_detail,
)
)
else:
rows.append(
RevisionDisplayRow(
id="original",
created=transcript.created_at.isoformat(),
version="original",
text=transcript.text or "",
error_detail=transcript.error_detail,
)
)
return rows
def _render_transcript_versioned_section(
*,
document: Document | None,
transcript: Transcript | None,
revisions: list[TranscriptRevision],
on_update: Callable[[str], Awaitable[None]] | None,
) -> None:
with ui.card().classes("w-full q-pa-md"):
ui.label("Transcript").classes("text-subtitle1 text-weight-medium")
ui.separator().classes("q-my-sm")
if transcript is None:
ui.label("Transcript not available yet.").classes("text-body2 text-grey-8")
return
with ui.column().classes("w-full q-gutter-y-xs"):
model_name = transcript.model
if model_name is None and revisions:
model_name = revisions[0].model
_metadata_row("Provider", transcript.provider)
_metadata_row("Model", model_name or "unknown")
_metadata_row("Prompt", transcript.prompt_name)
display_rows = _build_display_rows(transcript, revisions)
rows_by_id = {row.id: row for row in display_rows}
ui.separator().classes("q-my-sm")
ui.label("Versions").classes("text-subtitle2 text-weight-medium")
table = ui.table(
columns=[
{"name": "created", "label": "Created", "field": "created", "align": "left"},
{"name": "version", "label": "Version", "field": "version", "align": "left"},
],
rows=[
{
"id": row.id,
"created": row.created,
"version": row.version,
}
for row in display_rows
],
row_key="id",
).classes("w-full")
default_selected = display_rows[-1].id
selected_label = ui.label(f"Selected version: {rows_by_id[default_selected].version}").classes(
"text-caption text-grey-7"
)
ui.separator().classes("q-my-sm")
with ui.row().classes("w-full no-wrap items-start q-gutter-md"):
if document is not None:
with ui.column().classes("w-1/2"):
ui.label("Document Preview").classes("text-subtitle2 text-weight-medium")
_render_document_preview(document)
with ui.column().classes("w-1/2"):
editor = (
ui.textarea(label="Transcript text", value=rows_by_id[default_selected].text)
.props("autogrow outlined")
.classes("w-full")
)
error_label = ui.label("").classes("text-body2 text-red-10")
def _set_selected(version_id: str) -> None:
selected = rows_by_id.get(version_id)
if selected is None:
return
selected_label.text = f"Selected version: {selected.version}"
editor.value = selected.text
editor.update()
error_label.text = selected.error_detail or ""
error_label.update()
def _on_row_click(event) -> None: # noqa: ANN001
row_id = _extract_row_id(event.args)
if row_id is None:
return
_set_selected(row_id)
table.on("rowClick", _on_row_click)
_set_selected(default_selected)
if on_update is not None:
ui.button("Update", on_click=lambda: on_update(editor.value or ""))
def render_job_detail(
*,
job: Job,
document: Document | None,
transcript: Transcript | None,
revisions: list[TranscriptRevision],
on_update: Callable[[str], Awaitable[None]] | None = None,
) -> None:
"""Render all sections for the job detail page."""
logger.debug("Rendering job detail for job ID %s", job.id)
status_text = job.status.value
with ui.column().classes("w-full max-w-4xl q-gutter-md"):
with ui.card().classes("w-full q-pa-lg"):
with ui.row().classes("w-full items-center justify-between q-gutter-md"):
with ui.column().classes("q-gutter-none"):
ui.label("Job overview").classes("text-h6 text-weight-bold")
ui.label(str(job.id)).classes("text-caption text-grey-5")
ui.label(str(job.id)).classes("text-caption text-grey-7")
status_chip_classes = (
"q-px-sm q-py-xs rounded-borders "
"text-weight-medium text-capitalize "
@@ -77,15 +287,18 @@ def render_job_detail(*, job: Job, source: Source | None, revision: Revision | N
)
ui.label(status_text).classes(status_chip_classes)
ui.separator().classes("q-my-md bg-blue-grey-7")
ui.separator().classes("q-my-md")
with ui.column().classes("w-full q-gutter-y-xs"):
_metadata_row("Created", job.date_created.isoformat())
_metadata_row("Updated", job.date_updated.isoformat())
_metadata_row("Created", job.created_at.isoformat())
_metadata_row("Updated", job.updated_at.isoformat())
_metadata_row("Retries", str(job.retry_count))
render_original_transcription_card(job=job)
if document is not None:
_render_document_section(document)
if source is not None:
_render_source_section(source)
_render_revision_section(revision)
_render_transcript_versioned_section(
document=document,
transcript=transcript,
revisions=revisions,
on_update=on_update,
)
src/transcription/ui/components/table/common.py
-5
View File
@@ -1,14 +1,11 @@
"""Common logic for generating table widgets."""
import logging
from collections.abc import Callable
from typing import Any
from nicegui import events
from nicegui import ui
logger = logging.getLogger(__name__)
def _extract_row_id(args: Any) -> str | None:
if isinstance(args, dict):
@@ -40,7 +37,6 @@ def _bind_row_click_handler(
on_row_click_id(row_id)
table.on("rowClick", handle_row_click)
logger.debug("Row click handler bound to table")
def build_table(
@@ -67,7 +63,6 @@ def build_table(
.classes(classes)
.props('table-style="table-layout: fixed; width: 100%;"')
)
logger.info("Table built with %d rows and %d columns", len(rows), len(columns))
if on_row_click_id is not None:
_bind_row_click_handler(table, on_row_click_id=on_row_click_id)
return table
src/transcription/ui/components/table/jobs.py
+8 -8
View File
@@ -22,8 +22,8 @@ class JobTableRow:
status: str
filename: str
retry_count: int
date_created: str
date_updated: str
created_at: str
updated_at: str
def _format_timestamp(value: str) -> str:
@@ -43,10 +43,10 @@ def _serialize_rows(rows: Sequence[JobTableRow]) -> list[dict[str, Any]]:
"status": row.status,
"filename": row.filename,
"retry_count": row.retry_count,
"date_created": _format_timestamp(row.date_created),
"date_updated": _format_timestamp(row.date_updated),
"created_sort": row.date_created,
"updated_sort": row.date_updated,
"created_at": _format_timestamp(row.created_at),
"updated_at": _format_timestamp(row.updated_at),
"created_sort": row.created_at,
"updated_sort": row.updated_at,
}
for row in rows
]
@@ -65,8 +65,8 @@ def render_jobs_table(rows: Sequence[JobTableRow]) -> None:
{"name": "status", "label": "Status", "field": "status", "sortable": True},
{"name": "filename", "label": "Filename", "field": "filename", "sortable": True},
{"name": "retry_count", "label": "Retries", "field": "retry_count", "sortable": True},
{"name": "date_created", "label": "Created", "field": "date_created", "sortable": True},
{"name": "date_updated", "label": "Updated", "field": "date_updated", "sortable": True},
{"name": "created_at", "label": "Created", "field": "created_at", "sortable": True},
{"name": "updated_at", "label": "Updated", "field": "updated_at", "sortable": True},
],
default_sort_by="created_sort",
default_descending=True,
src/transcription/ui/components/transcript.py
-106
View File
@@ -1,106 +0,0 @@
"""Reusable transcript UI components."""
from __future__ import annotations
from collections.abc import Awaitable
from collections.abc import Callable
from datetime import datetime
from typing import Any
from nicegui import ui
from transcription.models import Job
from transcription.models import Revision
type RevisionAction = Callable[[Revision], Awaitable[None] | None]
def render_original_transcription_card(*, job: Job, classes: str = "w-full") -> Any:
"""Render the immutable original job transcription output."""
status_label = "Failed" if job.error_detail else "Transcribed"
header = f"Original Transcription | {status_label}"
provider = job.provider or "unknown"
model = job.model or "unknown"
caption = f"{provider} | {model} | {_format_created_at(job.date_updated)}"
card = ui.card().classes(f"{classes} q-pa-md bg-blue-grey-10")
with card, ui.column().classes("w-full q-gutter-y-sm"):
ui.label(header).classes("text-subtitle1 text-weight-medium")
ui.label(caption).classes("text-caption text-grey-5")
_metadata_row(label="Prompt", value=job.prompt_name or "unknown")
_metadata_row(label="Updated", value=_format_created_at(job.date_updated))
if job.text:
with ui.card().classes("w-full q-pa-sm"):
ui.markdown(job.text)
if job.error_detail:
with ui.card().classes("w-full bg-red-1 text-red-10 q-pa-sm"):
ui.label("Failure detail").classes("text-caption text-uppercase")
ui.label(job.error_detail).classes("text-body2")
return card
def render_revision_row(
*,
revision: Revision,
initially_expanded: bool = False,
classes: str = "w-full",
on_delete: RevisionAction | None = None,
) -> Any:
"""Render a collapsible row for the single optional source revision."""
header = "Revision | User-authored"
caption = _format_created_at(revision.date_created)
expansion = ui.expansion(value=initially_expanded, group="group").classes(
f"{classes} rounded-borders bg-blue-grey-10"
)
with expansion, ui.column().classes("w-full q-gutter-y-sm q-pa-sm"):
with expansion.add_slot("header"), ui.row().classes("w-full items-start justify-between q-gutter-md"):
with ui.column().classes("q-gutter-none"):
ui.label(header).classes("text-subtitle1 text-weight-medium")
ui.label(caption).classes("text-caption text-grey-5")
if on_delete is not None:
with ui.dialog() as delete_dialog, ui.card().classes("q-pa-md"):
ui.label("Delete this transcript revision?").classes("text-body1")
with ui.row().classes("w-full justify-end q-gutter-sm"):
ui.button("Cancel", on_click=lambda: delete_dialog.submit(False)).props("flat")
ui.button("Delete", on_click=lambda: delete_dialog.submit(True)).props(
'unelevated color="negative"'
)
async def delete_current_transcript() -> None:
delete_dialog.open()
confirmed = await delete_dialog
if not confirmed:
return
maybe_awaitable = on_delete(revision)
if isinstance(maybe_awaitable, Awaitable):
await maybe_awaitable
with ui.column(align_items="center").classes("self-center q-gutter-none"):
ui.button(icon="delete", on_click=delete_current_transcript).props(
'flat round dense color="negative"'
)
_metadata_row(label="Created", value=_format_created_at(revision.date_created))
if revision.text:
with ui.card().classes("w-full q-pa-sm"):
ui.markdown(revision.text)
return expansion
def _format_created_at(value: datetime) -> str:
"""Return a compact UTC-like timestamp for row captions."""
return value.strftime("%Y-%m-%d %H:%M:%S %Z")
def _metadata_row(*, label: str, value: str) -> None:
with ui.row().classes("w-md items-start justify-between q-gutter-x-md"):
ui.label(label).classes("text-caption text-grey-5 text-uppercase")
ui.label(value).classes("text-body2 text-right text-grey-1 break-all")
src/transcription/ui/pages/jobs_page.py
+126 -125
View File
@@ -4,159 +4,160 @@ from __future__ import annotations
from uuid import UUID
from fastapi import Request
from nicegui import ui
from sqlalchemy.orm import selectinload
from sqlmodel import desc
from sqlmodel import select
from transcription.app_state import resolve_session_factory
from transcription.db import get_session
from transcription.models import Document
from transcription.models import Job
from transcription.models import JobStatus
from transcription.models import Source
from transcription.services.jobs import JobService
from transcription.services.transcription import TranscriptionService
from transcription.ui.components.app_shell import render_navigation_header
from transcription.models import Transcript
from transcription.models import TranscriptRevision
from transcription.services import ServiceBundle
from transcription.ui.components.error_presenter import show_error
from transcription.ui.components.error_presenter import summarize_error
from transcription.ui.components.job_detail import render_job_detail
from transcription.ui.components.table.jobs import JobTableRow
from transcription.ui.components.table.jobs import render_jobs_table
from ..components.document_panzoom import render_document_panzoom
from ..components.table.jobs import JobTableRow
from ..components.transcript import render_original_transcription_card
from ..components.transcript import render_revision_row
def register_page() -> None: # noqa: PLR0915
"""Register jobs list and detail routes."""
@ui.page("/jobs")
async def jobs_page(request: Request) -> None:
session_factory = resolve_session_factory(request.app.state)
jobs_service = JobService(session_factory=session_factory)
render_navigation_header(current_path="/jobs")
@ui.refreshable
async def render_table() -> None:
jobs = [
async def fetch_job_rows() -> list[JobTableRow]:
"""Return jobs for display in most-recent-first order."""
async with get_session() as session:
jobs = (
await session.exec(
select(Job)
.options(selectinload(Job.document)) # pyright: ignore[reportArgumentType]
.order_by(desc(Job.created_at))
)
).all()
return [
JobTableRow(
id=job.id,
status=job.status.value,
filename=job.filename,
retry_count=job.retry_count,
date_created=job.date_created.isoformat(),
date_updated=job.date_updated.isoformat(),
created_at=job.created_at.isoformat(),
updated_at=job.updated_at.isoformat(),
)
for job in await jobs_service.list_jobs()
for job in jobs
]
render_jobs_table(jobs)
ui.button("Refresh", on_click=render_table.refresh, icon="refresh")
await render_table()
@ui.page("/jobs/{job_id}")
async def job_detail_page(job_id: str, request: Request) -> None: # noqa: PLR0915
session_factory = resolve_session_factory(request.app.state)
jobs_service = JobService(session_factory=session_factory)
transcription_service = TranscriptionService(session_factory=session_factory)
render_navigation_header(current_path="/jobs")
async def fetch_job_detail(job_id: UUID) -> tuple[Job | None, Document | None, Transcript | None, list[TranscriptRevision]]:
"""Return job, document, transcript snapshot, and revisions for detail view."""
async with get_session() as session:
job = await session.get(Job, job_id)
if job is None:
return None, None, None, []
document = await session.get(Document, job.document_id)
transcript = (await session.exec(select(Transcript).where(Transcript.job_id == job.id))).first()
revisions = (
await session.exec(
select(TranscriptRevision)
.where(TranscriptRevision.job_id == job.id)
.order_by(TranscriptRevision.version_number)
)
).all()
return job, document, transcript, list(revisions)
try:
parsed_job_id = UUID(job_id)
except ValueError:
ui.label("Invalid job id").classes("text-h6 text-negative")
return
try:
job = await jobs_service.read_job(job_id=parsed_job_id)
except ValueError:
ui.label("Job not found").classes("text-h6 text-negative")
return
def register_page() -> None:
"""Register jobs list and detail routes."""
source = _resolve_primary_source(job)
with ui.splitter(value=30).classes("w-full h-[calc(100vh-64px)]") as splitter:
with splitter.before, ui.column(align_items="stretch").classes("w-full h-full p-4 gap-3"):
if source is not None:
render_document_panzoom(source=source)
else:
ui.label("No source preview is available for this job.").classes("text-body2 text-grey-3")
with splitter.after, ui.column(align_items="stretch").classes("w-full h-full p-4 gap-3"):
with ui.row():
ui.button(icon="arrow_back", on_click=ui.navigate.back)
with ui.row().classes("w-full items-center justify-between"):
ui.label(f"{job.id}").classes("text-h6 text-weight-bold")
match job.status:
case JobStatus.TRANSCRIBED:
ui.chip(job.status.value.upper(), color="green", text_color="white").props("outline")
case _:
ui.label(f"{job.status.value}").classes("text-subtitle1 text-weight-medium")
render_original_transcription_card(job=job)
async def delete_revision_by_id(revision_id: UUID) -> None:
try:
revision = await transcription_service.read_revision(revision_id=revision_id)
await transcription_service.delete_revision(revision)
except Exception as exc: # noqa: BLE001
show_error(exc, title="Delete failed", operation="jobs.delete_revision")
return
ui.notify("Deleted revision", type="positive")
await render_revision_panel.refresh()
@ui.page("/jobs")
async def jobs_page() -> None:
ui.label("Transcription Jobs")
status = ui.label("Ready")
@ui.refreshable
async def render_revision_panel() -> None:
refreshed_job = await jobs_service.read_job(job_id=parsed_job_id)
refreshed_source = _resolve_primary_source(refreshed_job)
if refreshed_source is None:
ui.label("No source is available for revision editing.").classes("text-body2 text-grey-3")
return
current_revision = refreshed_source.revision
default_revision_text = (
current_revision.text if current_revision is not None else (refreshed_job.text or "")
)
ui.label("Revision Editor").classes("text-subtitle1 text-weight-medium")
editor = ui.textarea(label="Revision text", value=default_revision_text).props("autogrow outlined")
editor.classes("w-full")
async def save_revision() -> None:
candidate = (editor.value or "").strip()
if not candidate:
ui.notify("Revision text is required.", type="warning")
return
async def render_table() -> None:
jobs = await fetch_job_rows()
render_jobs_table(jobs)
async def refresh() -> None:
status.text = "Refreshing..."
try:
await transcription_service.upsert_revision_for_source(
source_id=refreshed_source.id,
text=candidate,
)
await render_table.refresh()
status.text = "Refreshed"
except Exception as exc: # noqa: BLE001
show_error(exc, title="Save failed", operation="jobs.save_revision")
status.text = f"Refresh failed: {summarize_error(exc, operation='jobs.refresh')}"
show_error(exc, title="Jobs refresh failed", operation="jobs.refresh")
ui.button("Refresh", on_click=refresh)
await render_table()
ui.link("Back to upload", "/upload")
@ui.page("/jobs/{job_id}")
async def job_detail_page(job_id: str) -> None:
ui.label("Job Detail")
content = ui.column().classes("w-full")
try:
parsed_id = UUID(job_id)
except ValueError:
ui.label("Invalid job id")
ui.link("Back to jobs", "/jobs")
return
ui.notify("Revision saved", type="positive")
await render_revision_panel.refresh()
with ui.row().classes("w-full justify-end"):
ui.button(
"Create revision" if current_revision is None else "Update revision",
on_click=save_revision,
icon="save",
).props('unelevated color="primary"')
if current_revision is None:
ui.label("No revision exists for this source.").classes("text-body2 text-grey-3")
async def refresh_content() -> None:
content.clear()
job, document, transcript, revisions = await fetch_job_detail(parsed_id)
if job is None:
with content:
ui.label("Job not found")
return
render_revision_row(
revision=current_revision,
initially_expanded=True,
on_delete=lambda _revision, rid=current_revision.id: delete_revision_by_id(rid),
services = ServiceBundle()
async def update_transcript_text(value: str) -> None:
try:
update_text = value.strip()
async with get_session() as session:
current_transcript = (
await session.exec(select(Transcript).where(Transcript.job_id == parsed_id))
).first()
provider_name = current_transcript.provider if current_transcript is not None else "openrouter"
prompt_name = (
current_transcript.prompt_name if current_transcript is not None else "transcribe_document.md"
)
model_name = current_transcript.model if current_transcript is not None else None
await services.transcriptions.upsert_transcript_by_job(
job_id=parsed_id,
text=update_text,
error_detail=None,
provider=provider_name,
prompt_name=prompt_name,
model=model_name,
session=session,
)
await services.transcriptions.append_transcript_revision(
job_id=parsed_id,
text=update_text,
error_detail=None,
provider=provider_name,
prompt_name=prompt_name,
model=model_name,
source="user",
session=session,
)
await session.commit()
ui.notify("Transcript updated", type="positive")
await refresh_content()
except Exception as exc: # noqa: BLE001
show_error(exc, title="Transcript update failed", operation="jobs.detail.update")
with content:
render_job_detail(
job=job,
document=document,
transcript=transcript,
revisions=revisions,
on_update=update_transcript_text,
)
await render_revision_panel()
await refresh_content()
def _resolve_primary_source(job: Job) -> Source | None:
if not job.sources:
return None
return job.sources[0]
ui.link("Back to jobs", "/jobs")
src/transcription/ui/pages/upload_page.py
+3 -2
View File
@@ -8,7 +8,6 @@ from nicegui import ui
from transcription.app_state import resolve_session_factory
from transcription.db import get_session
from transcription.services.store import create_upload_job
from transcription.ui.components.app_shell import render_navigation_header
from transcription.ui.components.upload import render_upload_widget
from transcription.worker import resolve_worker_notifier
@@ -18,7 +17,6 @@ def register_page() -> None:
@ui.page("/upload", title="Upload Document")
def upload_page(request: Request) -> None:
render_navigation_header(current_path="/upload")
session_factory = resolve_session_factory(request.app.state)
async def submit_upload(filename: str, file_bytes: bytes):
@@ -31,3 +29,6 @@ def register_page() -> None:
notify_worker = resolve_worker_notifier(request.app.state)
render_upload_widget(submitter=submit_upload, notifier=notify_worker)
with ui.row():
ui.link("View jobs", "/jobs")
src/transcription/ui/static/colors.css
-30
View File
@@ -1,30 +0,0 @@
:root {
/* Soft blue-night palette tokens */
--ctp-rosewater: #f2dde5;
--ctp-flamingo: #edcfd8;
--ctp-pink: #dcc7de;
--ctp-mauve: #a9bde5;
--ctp-red: #d98a9a;
--ctp-maroon: #d39aa5;
--ctp-peach: #d7af8c;
--ctp-yellow: #e2c083;
--ctp-green: #86c8ad;
--ctp-teal: #77bfbe;
--ctp-sky: #7ebdda;
--ctp-sapphire: #74aed0;
--ctp-blue: #92b5f5;
--ctp-lavender: #6f97e8;
--ctp-text: #d8e2f5;
--ctp-subtext1: #bfcae0;
--ctp-subtext0: #a9b6cf;
--ctp-overlay2: #95a3bf;
--ctp-overlay1: #7c8ca9;
--ctp-overlay0: #657490;
--ctp-surface2: #4d5f7c;
--ctp-surface1: #394a65;
--ctp-surface0: #2a3954;
--ctp-base: #1f2b42;
--ctp-mantle: #1a2538;
--ctp-crust: #141e30;
}
tests/api/test_error_responses.py
+2 -3
View File
@@ -1,12 +1,11 @@
"""Tests for API error response envelope handlers."""
import pytest
from fastapi import FastAPI
from fastapi.testclient import TestClient
import pytest
from transcription.api.errors import register_error_handlers
from transcription.errors import AppError
from transcription.errors import ErrorCategory
from transcription.errors import AppError, ErrorCategory
@pytest.mark.integration
tests/artifacts/transcriptions/Book_Two_-_page_02.jpg.txt
+4 -4
View File
@@ -11,11 +11,11 @@ BOOK 1 had 54 pages; 14 chapters. BOOK 2 has 70 pages; 18 chapters. BOOK 1 con-
sisted largely of first generation family history. BOOK 2 throws more light on
the second generation. Sidney promises a BOOK 3 and that may begin to do justice
to the third generation. We suggest that Sidney get the help of Louis Shinn
who has a chapter in this book (Chapter 16 - The Last 25 Years on the Doumecq
who has a chapter in this book (Chapter 16 - The Last 25 Years on the Doumeeq
Plains. Louis has the gift of seeing, recalling and telling. One sentence in
his chapter gives a great tribute to the Doumeccqers--so [sic] far as he knows no one
on the Doumecq Plains went on relief during the depression. That in a nutshell
shows the sturdy character of the residents of the Doumecq Plains.
his chapter gives a great tribute to the Doumeeqers - so far as he knows no one
on the Doumeeq Plains went on relief during the depression. That in a nutshell
shows the sturdy character of the residents of the Doumeeq Plains.
We promised in BOOK 1 that in BOOK 2 we would give the story of the trip of
John E. Cochran and wife to Tennessee, Cuba and the Panama Canal. You will see
tests/artifacts/transcriptions/Omie_Writes_Home.pdf.txt
+14 -12
View File
@@ -18,6 +18,7 @@ the family newsletter two years ago.
Nome Alaska August 26, 1923
My Dear Ethel et al.
I don't know when I did write or when you did
but I am going to write now however and never
the less. But I wish I could talk (I can yet but I
@@ -26,6 +27,7 @@ and Polly sit up and listen and that little black
rascal of yours would fairly sparkle with
listening. Can't I see him listening now to all the
yarns we told last summer?
You see, we-Miss Saville and I, took a trip north
on the Buford and it was very interesting. We
went north thru the Bering Strait into the Arctic and as far as the Ice Pack. There the captain
@@ -41,14 +43,14 @@ all around it similar to a currycomb in coarseness; no ears but huge tusks of iv
the most repulsive looking animals imaginable and tho I have always read about them I never
expect such disagreeable looking creatures. They had a rough brown hairy skin and some of
them looked warty. They must have weighed two ton at least. Ere we got them back to Nome
to the natives they were getting extremely odiferous—in fact, you could scarcely stay on the
to the natives they were getting extremely odiferous–in fact, you could scarcely stay on the
ship with any degree of comfort unless you had per chance lost your sense of smell.
Then we went north to a few minutes beyond the 70th degree of latitude and thot for awhile
we would go to Wrangell Island where some men from Steffonsons ship were supposed to be
stranded but we didn't get there and instead stopped at a small native village at Cape Serdz in
Then we went north to a few minutes beyond the 70th degree of latitude and thot [sic] for awhile
we would go to Wrangell Island where some men from Stefflonsons [sic] ship were supposed to be
stranded but we didn't get there and instead stopped at a small native village at Cape Serdz [sic] in
Siberia. These Eskimo were very primitive. One white squaw man lived there and had for 23
years. He was a Swede--who else could. Their houses were circular and built up with dirt 2 or
years. He was a Swede–who else could. Their houses were circular and built up with dirt 2 or
3 feet and then skins were stretched over it and weighted down with rocks. Inside, the room
was partitioned off at the sides with skins for sleeping quarters. In the main part they had the
fire on the ground and the fish drying on lines and the skins hanging around and the dogs and
@@ -63,22 +65,22 @@ The other place we stopped was at Whalen, a trading post in Siberia. There these
went wild. They rushed helter-skelter, hither and thither, here and there, trying to find
something to buy. Prices raised right before your eyes. One would but something for $1.00
and the next might have to pay $2.00, $4.00 or $10.00. That made no difference. They had to
have it. One man I was sort of taking care of, tho he had his son along for the purpose,
bought 2 ivory tusks, 1 pup, 2 moccasins, 3 or 4 billikens, 6 or 8 ivory and silver rings, one
have it. One man I was sort of taking care of, tho [sic] he had his son along for the purpose,
bought 2 ivory tusks, 1 pup, 2 moccasins, 3 or 4 billi[illegible]s, 6 or 8 ivory and silver rings, one
fishing line, hooks, floats, etc. and two bird slings. The slings have rocks at the end and the
little natives throw them at the flocks of geese and ducks which fly close over the village and
the slings entangle their wings and legs, sometimes more than one, and they can't fly. They
come down and the natives capture them. There was more junk brot aboard than baggage, I
come down and the natives capture them. There was more junk brot [sic] aboard than baggage, I
do believe. And they say that at the first stop it was worse than here. The red flag was flying
over Whalen and the Russian soldiers were there—a few, one or two or three, I forget the
over Whalen and the Russian soldiers were there–a few, one or two or three, I forget the
number.
We got home yesterday morning at 5 a.m. but missed the first lighter in so had to stay out
until 2:30. The girls had prepared a big meal for us and invited up the Hartfords and then let
us talk. Miss Saville talked quite a bit. Any how if you folks don't like this I don't care, it is
all I had to write about and I know Buster'ud listen anyway and I'd soak ole Peter's head if he
all I had to write about and I know Buster'd [sic] listen anyway and I'd soak ole Peter's head if he
didn't and Polly would in my lap and I don't know much about the youngest one of yours so
likely he would be squawling. But we did surely enjoy our trip and were gone just long enuf.
likely he would be squawling. But we did surely enjoy our trip and were gone just long enuf [sic].
I expect there were 150 passengers on board and almost or more of the crew and helpers. We
had a stateroom down next to the kitchen and 'twas pretty fierce for odor at times.
@@ -107,7 +109,7 @@ Ome
Reprinted from Cochran Chronicles, Volume 9, Number 1, November 1986
© JECFA 1986
© [inserted: JECFA] 1986
Up
tests/artifacts/transcriptions/Rod_Moser_Letter_-_p1.jpg.txt
+15 -14
View File
@@ -8,23 +8,24 @@ ISBILL & MOSER
DEALERS IN
GENERAL MERCHANDISE
Vonore, Tenn. Jany 27- 1913
Dear Much Aunt Adeline
Was at home a
few nights ago & saw a
letter from your folks. So
Vonore, Tenn., Jany 27- 1913
Dear Much Aunt Louie
How are you a
few nights ago I sewed a
letter from your folks, so
I decided to write you
a few lines myself ok
I am contemplateing [sic] a
I am contemplateing a
trip out west next summer
& [inserted: I] want some of them to go
when I am [inserted: a] them.
Am getting
& I want Some Olders to go
where I and them.
I am getting
up in years & unmarried
so you see the object of
my trip, is to get a wife
& if there is any old maids
or widows out there, I
my trip, is to get a bunch
of Young & old maids
& widows out there. I
want you to kiss them
at my [hand?] me at there
as soon as I get there.
at my fans [sic] mug as they
as soon as I get there
tests/integration/test_pipeline_flow.py
+44 -48
View File
@@ -3,10 +3,10 @@
from pathlib import Path
import pytest
from sqlmodel import select
from transcription.config import Settings
from transcription.models import Job
from transcription.models import JobStatus
from transcription.models import Job, JobStatus, Transcript, TranscriptRevision
from transcription.providers.base import TranscriptionResult
from transcription.services.store import create_upload_job
from transcription.worker import process_next_queued_job
@@ -17,10 +17,8 @@ class TestPipelineSuccessFlow:
"""Verify end-to-end success lifecycle behavior."""
@pytest.mark.asyncio
async def test_upload_then_worker_persists_transcribed_terminal_state(
self, async_session, tmp_path: Path, monkeypatch
):
"""Upload followed by worker processing persists job transcription and transcribed status."""
async def test_upload_then_worker_persists_transcribed_terminal_state(self, async_session, tmp_path: Path, monkeypatch):
"""Upload followed by worker processing persists transcript and transcribed status."""
settings = Settings(openrouter_api_key="test-key", upload_dir=tmp_path)
upload_result = await create_upload_job(
filename="pipeline.jpg",
@@ -29,43 +27,38 @@ class TestPipelineSuccessFlow:
settings=settings,
)
async def _fake_transcribe(*, prompt_text: str, image_bytes: bytes, mime_type: str) -> TranscriptionResult:
_ = (prompt_text, image_bytes, mime_type)
async def _fake_transcribe(_path: str) -> TranscriptionResult:
return TranscriptionResult(
text="Pipeline transcript",
provider="openrouter",
model="test-model",
prompt_name="transcribe_document.md",
model="test-model",
)
async def _fake_transcribe_document_image(
image_path,
*,
prompt_name="transcribe_document.md",
settings=None,
provider=None,
) -> TranscriptionResult:
_ = (image_path, prompt_name, settings, provider)
return TranscriptionResult(
text="Pipeline transcript",
provider="openrouter",
model="test-model",
prompt_name="transcribe_document.md",
)
monkeypatch.setattr(
"transcription.services.workflows.transcribe_document_image",
_fake_transcribe_document_image,
)
monkeypatch.setattr("transcription.services.workflows.transcribe_document_image", _fake_transcribe)
processed = await process_next_queued_job(session=async_session)
job = await async_session.get(Job, upload_result.job_id)
transcript = (await async_session.exec(select(Transcript).where(Transcript.job_id == upload_result.job_id))).first()
revisions = (
await async_session.exec(
select(TranscriptRevision)
.where(TranscriptRevision.job_id == upload_result.job_id)
.order_by(TranscriptRevision.version_number)
)
).all()
assert processed is True
assert job is not None
assert job.status == JobStatus.TRANSCRIBED
assert job.text == "Pipeline transcript"
assert job.error_detail is None
assert transcript is not None
assert transcript.text == "Pipeline transcript"
assert transcript.error_detail is None
assert transcript.model == "test-model"
assert len(revisions) == 1
assert revisions[0].version_number == 1
assert revisions[0].source == "ai"
assert revisions[0].text == "Pipeline transcript"
@pytest.mark.integration
@@ -74,7 +67,7 @@ class TestPipelineFailureFlow:
@pytest.mark.asyncio
async def test_upload_then_worker_persists_failed_terminal_state(self, async_session, tmp_path: Path, monkeypatch):
"""Upload followed by worker processing persists error detail and failed status on the job."""
"""Upload followed by worker processing persists error detail and failed status."""
settings = Settings(openrouter_api_key="test-key", upload_dir=tmp_path)
upload_result = await create_upload_job(
filename="pipeline.jpg",
@@ -83,29 +76,32 @@ class TestPipelineFailureFlow:
settings=settings,
)
async def _fake_transcribe_document_image(
image_path,
*,
prompt_name="transcribe_document.md",
settings=None,
provider=None,
) -> TranscriptionResult:
_ = (image_path, prompt_name, settings, provider)
async def _fake_transcribe(_path: str) -> TranscriptionResult:
raise RuntimeError("pipeline provider failure")
monkeypatch.setattr(
"transcription.services.workflows.transcribe_document_image",
_fake_transcribe_document_image,
)
monkeypatch.setattr("transcription.services.workflows.transcribe_document_image", _fake_transcribe)
processed = await process_next_queued_job(session=async_session)
job = await async_session.get(Job, upload_result.job_id)
transcript = (await async_session.exec(select(Transcript).where(Transcript.job_id == upload_result.job_id))).first()
revisions = (
await async_session.exec(
select(TranscriptRevision)
.where(TranscriptRevision.job_id == upload_result.job_id)
.order_by(TranscriptRevision.version_number)
)
).all()
assert processed is True
assert job is not None
assert job.status == JobStatus.FAILED
assert job.text is None
assert job.error_detail is not None
assert "pipeline provider failure" in job.error_detail
assert "[internal_unexpected_error]" in job.error_detail
assert "error_id=" in job.error_detail
assert transcript is not None
assert transcript.text is None
assert "pipeline provider failure" in transcript.error_detail
assert "[internal_unexpected_error]" in transcript.error_detail
assert "error_id=" in transcript.error_detail
assert len(revisions) == 1
assert revisions[0].version_number == 1
assert revisions[0].source == "ai"
assert revisions[0].text is None
assert "pipeline provider failure" in (revisions[0].error_detail or "")
tests/providers/test_openrouter.py
+11 -17
View File
@@ -5,10 +5,8 @@ from types import SimpleNamespace
import pytest
from transcription.config import Settings
from transcription.providers.base import ProviderError
from transcription.providers.base import ProviderResponseError
from transcription.providers.openrouter import DEFAULT_OPENROUTER_MODEL
from transcription.providers.openrouter import OpenRouterTranscriptionProvider
from transcription.providers.base import ProviderError, ProviderResponseError
from transcription.providers.openrouter import DEFAULT_OPENROUTER_MODEL, OpenRouterTranscriptionProvider
class _FakeChat:
@@ -17,7 +15,7 @@ class _FakeChat:
self._error = error
self.calls = []
async def send_async(self, **kwargs):
def send(self, **kwargs):
self.calls.append(kwargs)
if self._error:
raise self._error
@@ -50,8 +48,7 @@ class TestOpenRouterProviderInit:
class TestOpenRouterProviderTranscribe:
"""Verify OpenRouter request construction and response parsing."""
@pytest.mark.asyncio
async def test_includes_optional_referer_and_title_when_set(self):
def test_includes_optional_referer_and_title_when_set(self):
"""Transcribe sends app attribution fields when configured."""
response = {"model": "vendor/model-a", "choices": [{"message": {"content": "Transcript text"}}]}
client = _FakeClient(response=response)
@@ -62,7 +59,7 @@ class TestOpenRouterProviderTranscribe:
)
provider = OpenRouterTranscriptionProvider(settings=settings, client=client)
result = await provider.transcribe(
result = provider.transcribe(
prompt_text="Prompt body",
image_bytes=b"img-bytes",
mime_type="image/png",
@@ -73,8 +70,7 @@ class TestOpenRouterProviderTranscribe:
assert send_call["x_open_router_title"] == "Transcription App"
assert result.text == "Transcript text"
@pytest.mark.asyncio
async def test_parses_successful_response_text(self):
def test_parses_successful_response_text(self):
"""Transcribe returns normalized text from a valid response payload."""
response = {
"model": "vendor/model-b",
@@ -85,7 +81,7 @@ class TestOpenRouterProviderTranscribe:
client=_FakeClient(response=response),
)
result = await provider.transcribe(
result = provider.transcribe(
prompt_text="Prompt body",
image_bytes=b"img-bytes",
mime_type="image/jpeg",
@@ -95,8 +91,7 @@ class TestOpenRouterProviderTranscribe:
assert result.provider == "openrouter"
assert result.model == "vendor/model-b"
@pytest.mark.asyncio
async def test_maps_sdk_exception_to_provider_error(self):
def test_maps_sdk_exception_to_provider_error(self):
"""Transcribe converts SDK failures to ProviderError."""
provider = OpenRouterTranscriptionProvider(
settings=Settings(openrouter_api_key="test-key"),
@@ -104,14 +99,13 @@ class TestOpenRouterProviderTranscribe:
)
with pytest.raises(ProviderError):
await provider.transcribe(
provider.transcribe(
prompt_text="Prompt body",
image_bytes=b"img-bytes",
mime_type="image/png",
)
@pytest.mark.asyncio
async def test_raises_on_empty_or_invalid_response(self):
def test_raises_on_empty_or_invalid_response(self):
"""Transcribe raises ProviderResponseError for missing completion text."""
provider = OpenRouterTranscriptionProvider(
settings=Settings(openrouter_api_key="test-key"),
@@ -119,7 +113,7 @@ class TestOpenRouterProviderTranscribe:
)
with pytest.raises(ProviderResponseError):
await provider.transcribe(
provider.transcribe(
prompt_text="Prompt body",
image_bytes=b"img-bytes",
mime_type="image/png",
tests/services/test_job_service.py
+67 -79
View File
@@ -4,97 +4,85 @@ import pytest
from transcription.models import Document
from transcription.models import Job
from transcription.models import JobStatus
from transcription.models import Source
from transcription.services.documents import DocumentService
from transcription.services.jobs import JobService
from transcription.services.jobs import JobStatus
class TestJobService:
class TestBasicCRUD:
@pytest.mark.asyncio
async def test_create_and_read_job(self, job_service: JobService, document_service: DocumentService):
document = Document(id=uuid4(), name="test-bundle")
await document_service.create_document(document=document)
async def test_create_job(self, job_service: JobService):
"""Test creating a job."""
job = Job(document_id=document.id)
await job_service.create_job(job=job)
def fake_job_factory():
return Job(document_id=uuid4())
fetched = await job_service.read_job(job_id=job.id)
assert fetched.id == job.id
assert fetched.document is not None
assert fetched.document.id == document.id
@pytest.mark.asyncio
async def test_update_job_state_updates_status_and_retry(
self,
job_service: JobService,
document_service: DocumentService,
):
document = Document(id=uuid4(), name="test-bundle")
await document_service.create_document(document=document)
job = Job(document_id=document.id)
await job_service.create_job(job=job)
updated = await job_service.update_job_state(
job_id=job.id,
status=JobStatus.PROCESSING,
retry_count_increment=1,
)
assert updated.status == JobStatus.PROCESSING
assert updated.retry_count == 1
@pytest.mark.asyncio
async def test_query_jobs_by_status(self, job_service: JobService, document_service: DocumentService):
document = Document(id=uuid4(), name="query-doc")
await document_service.create_document(document=document)
await job_service.create_job(job=Job(document_id=document.id, status=JobStatus.PROCESSING))
await job_service.create_job(job=Job(document_id=document.id, status=JobStatus.QUEUED))
result = await job_service.query_jobs(status=JobStatus.PROCESSING)
assert len(result) == 1
assert result[0].status == JobStatus.PROCESSING
@pytest.mark.asyncio
async def test_query_jobs_by_source_filename(self, job_service: JobService, document_service: DocumentService):
document = Document(id=uuid4(), name="source-doc")
await document_service.create_document(document=document)
job = Job(document_id=document.id)
await job_service.create_job(job=job)
await job_service.create_job(job=fake_job_factory())
async with job_service._session_scope() as session:
session.add(
Source(
document_id=document.id,
job_id=job.id,
upload_name="letter.jpg",
filename="stored-letter.jpg",
file_path="/uploads/stored-letter.jpg",
)
)
await session.commit()
result = await job_service.query_jobs(filename="stored-letter.jpg")
assert len(result) == 1
assert result[0].id == job.id
for _ in range(10):
await job_service.create_job(job=fake_job_factory(), session=session)
@pytest.mark.asyncio
async def test_read_next_queued_job_orders_by_created_date(
self,
job_service: JobService,
document_service: DocumentService,
):
document = Document(id=uuid4(), name="ordered-doc")
async def test_backpropagation(self, job_service: JobService, document_service: DocumentService):
"""Test that creating a job backpropagates to the related document."""
doc_id = uuid4()
document = Document(
id=doc_id,
filename="test.txt",
file_path="/path/to/test.txt",
)
await document_service.create_document(document=document)
job = Job(document_id=doc_id)
await job_service.create_job(job=job)
first = Job(document_id=document.id, status=JobStatus.QUEUED)
second = Job(document_id=document.id, status=JobStatus.QUEUED)
await job_service.create_job(job=first)
await job_service.create_job(job=second)
read_job = await job_service.read_job(job_id=job.id)
assert isinstance(read_job.document, Document)
assert read_job.document.id == document.id
next_job = await job_service.read_next_queued_job()
assert next_job is not None
assert next_job.id == first.id
@pytest.mark.asyncio
async def test_reading_job(self, job_service: JobService):
"""Test reading a job."""
uuid = uuid4()
await job_service.create_job(job=Job(id=uuid, document_id=uuid4()))
job = await job_service.read_job(job_id=uuid)
assert job.id == uuid
@pytest.mark.asyncio
async def test_updating_job(self, job_service: JobService):
"""Test updating a job."""
uuid = uuid4()
job = Job(id=uuid, document_id=uuid4())
async with job_service._session_scope() as session:
await job_service.create_job(job=job, session=session)
job.status = JobStatus.PROCESSING
await job_service.update_job(job=job, session=session)
read_job = await job_service.read_job(job_id=uuid, session=session)
assert read_job == job
@pytest.mark.asyncio
async def test_deleting_job(self, job_service: JobService):
"""Test deleting a job."""
class TestServiceMethods:
@pytest.mark.asyncio
async def test_query_jobs(self, job_service: JobService):
"""Test querying jobs."""
await job_service.create_job(job=Job(document_id=uuid4(), status=JobStatus.PROCESSING))
result = await job_service.query_jobs(status=JobStatus.PROCESSING)
jobs = {str(job.id).split("-")[0]: job.status for job in result}
assert len(jobs) == 1
@pytest.mark.asyncio
async def test_list_jobs(self, job_service: JobService):
"""Test listing jobs."""
n = 5
for _ in range(n):
await job_service.create_job(job=Job(document_id=uuid4()))
jobs = await job_service.list_jobs()
assert len(jobs) == n
@pytest.mark.asyncio
async def test_mark_job_status(self, job_service: JobService):
"""Test marking a job with a new status."""
tests/services/test_transcription_external.py
+3 -3
View File
@@ -7,6 +7,7 @@ import pytest
from transcription.services.transcription import transcribe_document_image
HAS_OPENROUTER_KEY = bool(os.getenv("OPENROUTER_API_KEY"))
REAL_IMAGES_DIR = Path(__file__).resolve().parents[1] / "fixtures" / "images" / "real"
@@ -48,11 +49,10 @@ class TestRealImageExternalTranscription:
assert REAL_IMAGES_DIR.exists()
assert _real_image_paths()
@pytest.mark.asyncio
@pytest.mark.parametrize("image_path", _real_image_paths(), ids=lambda p: p.name)
async def test_transcribes_real_image_fixture(self, image_path: Path):
def test_transcribes_real_image_fixture(self, image_path: Path):
"""Real fixture image produces a non-empty transcription result."""
result = await transcribe_document_image(image_path)
result = transcribe_document_image(image_path)
assert result.provider == "openrouter"
assert isinstance(result.model, str) and result.model.strip()
assert isinstance(result.text, str) and result.text.strip()
tests/services/test_transcription_service.py
-85
View File
@@ -1,85 +0,0 @@
"""Tests for revision behavior in TranscriptionService."""
from uuid import uuid4
import pytest
from transcription.models import Document
from transcription.models import Job
from transcription.models import JobStatus
from transcription.models import Source
from transcription.services.documents import DocumentService
from transcription.services.jobs import JobService
from transcription.services.transcription import TranscriptionService
@pytest.mark.integration
class TestTranscriptionServiceRevisionUpsert:
"""Verify optional single-revision create/update semantics."""
@pytest.mark.asyncio
async def test_upsert_revision_creates_new_revision(self, default_session_factory):
documents = DocumentService(session_factory=default_session_factory)
jobs = JobService(session_factory=default_session_factory)
transcriptions = TranscriptionService(session_factory=default_session_factory)
document = Document(id=uuid4(), name="revision-create")
await documents.create_document(document=document)
job = Job(document_id=document.id, status=JobStatus.TRANSCRIBED, text="Original text")
await jobs.create_job(job=job)
source = Source(
document_id=document.id,
job_id=job.id,
upload_name="source.jpg",
filename="source.jpg",
file_path="uploads/source.jpg",
)
async with transcriptions._session_scope() as session:
session.add(source)
await session.commit()
await session.refresh(source)
revision = await transcriptions.upsert_revision_for_source(source_id=source.id, text="User revision")
fetched = await transcriptions.read_revision_by_source(source.id)
assert revision.source_id == source.id
assert revision.text == "User revision"
assert fetched is not None
assert fetched.id == revision.id
assert fetched.text == "User revision"
@pytest.mark.asyncio
async def test_upsert_revision_updates_existing_single_revision(self, default_session_factory):
documents = DocumentService(session_factory=default_session_factory)
jobs = JobService(session_factory=default_session_factory)
transcriptions = TranscriptionService(session_factory=default_session_factory)
document = Document(id=uuid4(), name="revision-update")
await documents.create_document(document=document)
job = Job(document_id=document.id, status=JobStatus.TRANSCRIBED, text="Original text")
await jobs.create_job(job=job)
source = Source(
document_id=document.id,
job_id=job.id,
upload_name="source.jpg",
filename="source.jpg",
file_path="uploads/source.jpg",
)
async with transcriptions._session_scope() as session:
session.add(source)
await session.commit()
await session.refresh(source)
first = await transcriptions.upsert_revision_for_source(source_id=source.id, text="Revision v1")
second = await transcriptions.upsert_revision_for_source(source_id=source.id, text="Revision v2")
revisions = await transcriptions.list_revisions_by_job(job.id)
assert first.id == second.id
assert second.text == "Revision v2"
assert len(revisions) == 1
assert revisions[0].id == first.id
assert revisions[0].text == "Revision v2"
tests/services/test_workflows_reliability.py
-75
View File
@@ -1,75 +0,0 @@
"""Reliability tests for worker workflow timeout behavior."""
from pathlib import Path
from uuid import uuid4
import pytest
from transcription.config import Settings
from transcription.models import Document
from transcription.models import Job
from transcription.models import JobStatus
from transcription.models import Source
from transcription.services import ServiceBundle
from transcription.services.workflows import process_queued_job
@pytest.mark.integration
class TestWorkflowReliability:
"""Verify timeout and terminal-state reliability behavior."""
@pytest.mark.asyncio
async def test_process_queued_job_timeout_marks_job_failed(self, default_session_factory, monkeypatch):
"""Provider timeout transitions a queued job to failed with error detail."""
services = ServiceBundle()
object.__setattr__(
services,
"documents",
services.documents.__class__(session_factory=default_session_factory),
)
object.__setattr__(
services,
"jobs",
services.jobs.__class__(session_factory=default_session_factory),
)
object.__setattr__(
services,
"transcriptions",
services.transcriptions.__class__(session_factory=default_session_factory),
)
async with services.jobs._session_scope() as session:
document = Document(id=uuid4(), name="timeout-doc")
session.add(document)
await session.flush()
job = Job(document_id=document.id, status=JobStatus.QUEUED)
session.add(job)
await session.flush()
source = Source(
document_id=document.id,
job_id=job.id,
upload_name="timeout.jpg",
filename="timeout.jpg",
file_path=str(Path("tests/fixtures/images/real/Book Two - page 02.jpg")),
)
session.add(source)
await session.commit()
loaded = await services.jobs.read_job(job_id=job.id, session=session)
async def _never_returns(image_path, *, prompt_name="transcribe_document.md", settings=None, provider=None):
_ = (image_path, prompt_name, settings, provider)
raise TimeoutError("simulated provider timeout")
monkeypatch.setattr("transcription.services.workflows.transcribe_document_image", _never_returns)
timeout_settings = Settings(openrouter_api_key="test-key", worker_provider_timeout_seconds=20.0)
result = await process_queued_job(job=loaded, services=services, settings=timeout_settings)
assert result is not None
assert result.status == JobStatus.FAILED
assert result.error_detail is not None
assert "timed out" in result.error_detail.lower()
assert "20.0s" in result.error_detail
tests/test_app.py
+29 -64
View File
@@ -1,7 +1,5 @@
"""Tests for transcription.app."""
from contextlib import asynccontextmanager
import pytest
from fastapi import FastAPI
from fastapi.testclient import TestClient
@@ -23,43 +21,28 @@ class TestAppFactory:
class TestAppLifespan:
"""Verify startup and shutdown lifecycle behavior."""
def test_startup_initializes_runtime_dependencies(self, monkeypatch, tmp_path):
def test_startup_initializes_runtime_dependencies(self, monkeypatch):
"""Startup initializes logging, schema, directories, and worker resources."""
calls = []
monkeypatch.setattr("transcription.app.configure_logging", lambda: calls.append("logging"))
async def _create_all(**_kwargs):
calls.append("schema")
monkeypatch.setattr("transcription.app.create_all", _create_all)
monkeypatch.setattr("transcription.app.setup_logging", lambda: calls.append("logging"))
monkeypatch.setattr("transcription.app.create_all", lambda **_kwargs: calls.append("schema"))
monkeypatch.setattr(
"transcription.app.initialize_database_runtime",
lambda **_kwargs: type("_Runtime", (), {"engine": object(), "session_factory": object()})(),
lambda **_kwargs: type("_Runtime", (), {"engine": object()})(),
)
monkeypatch.setattr("transcription.app.dispose_database_runtime", lambda: calls.append("dispose_db"))
monkeypatch.setattr("transcription.app.should_bootstrap_schema", lambda _settings: True)
monkeypatch.setattr("transcription.app._start_worker", lambda _app: calls.append("start_worker"))
monkeypatch.setattr("transcription.app._stop_worker", lambda _app: calls.append("stop_worker"))
async def _dispose_runtime():
calls.append("dispose_db")
monkeypatch.setattr("transcription.app.dispose_database_runtime", _dispose_runtime)
async def _recover_stale(_app):
calls.append("recover")
monkeypatch.setattr("transcription.app._recover_stale_processing_jobs", _recover_stale)
@asynccontextmanager
async def _worker_lifespan(**_kwargs):
calls.append("worker_start")
yield object(), object()
calls.append("worker_stop")
monkeypatch.setattr("transcription.app.worker_consumer_lifespan", _worker_lifespan)
class _Dir:
def mkdir(self, parents: bool, exist_ok: bool):
calls.append("mkdir")
class _Settings:
should_bootstrap_schema = True
upload_dir = tmp_path / "uploads"
prompt_dir = tmp_path / "prompts"
upload_dir = _Dir()
prompt_dir = _Dir()
monkeypatch.setattr("transcription.app.get_settings", lambda: _Settings())
@@ -69,50 +52,32 @@ class TestAppLifespan:
assert "logging" in calls
assert "schema" in calls
assert "recover" in calls
assert "worker_start" in calls
assert "worker_stop" in calls
assert "mkdir" in calls
assert "start_worker" in calls
assert "dispose_db" in calls
assert _Settings.upload_dir.exists()
assert _Settings.prompt_dir.exists()
def test_shutdown_stops_worker_resources(self, monkeypatch, tmp_path):
def test_shutdown_stops_worker_resources(self, monkeypatch):
"""Shutdown signals and stops worker resources cleanly."""
calls = []
monkeypatch.setattr("transcription.app.configure_logging", lambda: calls.append("logging"))
async def _create_all(**_kwargs):
calls.append("schema")
monkeypatch.setattr("transcription.app.create_all", _create_all)
monkeypatch.setattr("transcription.app.setup_logging", lambda: None)
monkeypatch.setattr("transcription.app.create_all", lambda **_kwargs: None)
monkeypatch.setattr(
"transcription.app.initialize_database_runtime",
lambda **_kwargs: type("_Runtime", (), {"engine": object(), "session_factory": object()})(),
lambda **_kwargs: type("_Runtime", (), {"engine": object()})(),
)
monkeypatch.setattr("transcription.app.dispose_database_runtime", lambda: calls.append("dispose_db"))
monkeypatch.setattr("transcription.app.should_bootstrap_schema", lambda _settings: True)
monkeypatch.setattr("transcription.app._start_worker", lambda _app: calls.append("start_worker"))
monkeypatch.setattr("transcription.app._stop_worker", lambda _app: calls.append("stop_worker"))
async def _dispose_runtime():
calls.append("dispose_db")
monkeypatch.setattr("transcription.app.dispose_database_runtime", _dispose_runtime)
async def _recover_stale(_app):
calls.append("recover")
monkeypatch.setattr("transcription.app._recover_stale_processing_jobs", _recover_stale)
@asynccontextmanager
async def _worker_lifespan(**_kwargs):
calls.append("worker_start")
yield object(), object()
calls.append("worker_stop")
monkeypatch.setattr("transcription.app.worker_consumer_lifespan", _worker_lifespan)
class _Dir:
def mkdir(self, parents: bool, exist_ok: bool):
return None
class _Settings:
should_bootstrap_schema = True
upload_dir = tmp_path / "uploads"
prompt_dir = tmp_path / "prompts"
upload_dir = _Dir()
prompt_dir = _Dir()
monkeypatch.setattr("transcription.app.get_settings", lambda: _Settings())
@@ -120,4 +85,4 @@ class TestAppLifespan:
with TestClient(app):
pass
assert calls == ["logging", "schema", "recover", "worker_start", "worker_stop", "dispose_db"]
assert calls == ["start_worker", "stop_worker", "dispose_db"]
tests/test_config.py
+1 -2
View File
@@ -5,8 +5,7 @@ from pathlib import Path
import pytest
from pydantic import ValidationError
from transcription.config import Provider
from transcription.config import Settings
from transcription.config import Provider, Settings
def _make_settings(**overrides) -> Settings:
tests/test_db.py
+44 -40
View File
@@ -1,64 +1,68 @@
"""Tests for transcription.db runtime and schema bootstrap behavior."""
"""Tests for transcription.db — async schema bootstrap/runtime behavior."""
import pytest
from sqlalchemy import inspect
from transcription.config import Settings
from transcription.db import create_all
from transcription.db import dispose_database_runtime
from transcription.db import get_session
from transcription.db import initialize_database_runtime
from sqlalchemy import text
import pytest
@pytest.mark.asyncio
async def test_create_all_creates_expected_tables(tmp_path):
settings = Settings(
openrouter_api_key="test-key",
database_url=f"sqlite:///{tmp_path / 'schema.db'}",
environment="test",
)
runtime = initialize_database_runtime(settings=settings)
class TestSchemaBootstrap:
"""Verify async create_all produces the expected table set."""
try:
await create_all(engine=runtime.engine)
async with runtime.engine.connect() as conn:
table_names = set(await conn.run_sync(lambda c: inspect(c).get_table_names()))
@pytest.mark.asyncio
async def test_create_all_creates_expected_tables(self, default_settings):
"""After async create_all(), document/job/transcript/revision tables exist."""
# Ensure models are imported so metadata is populated.
from transcription.models import Document, Job, Transcript, TranscriptRevision # noqa: F401
from transcription.db.operations import create_all
from transcription.db.runtime import get_engine
engine = get_engine(settings=default_settings)
await create_all(engine=engine)
async with engine.begin() as connection:
table_names = set(await connection.run_sync(lambda sync_conn: inspect(sync_conn).get_table_names()))
assert "document" in table_names
assert "job" in table_names
assert "source" in table_names
assert "revision" in table_names
finally:
await dispose_database_runtime()
assert "transcript" in table_names
assert "transcriptrevision" in table_names
@pytest.mark.asyncio
async def test_get_session_yields_async_session(tmp_path):
settings = Settings(
openrouter_api_key="test-key",
database_url=f"sqlite:///{tmp_path / 'session.db'}",
environment="test",
)
initialize_database_runtime(settings=settings)
class TestSessionFactory:
"""Verify async get_session yields a usable AsyncSession."""
try:
async with get_session(settings=settings) as session:
assert session is not None
finally:
await dispose_database_runtime()
@pytest.mark.asyncio
async def test_get_session_yields_session(self, default_settings):
"""get_session() yields an AsyncSession with a live connection."""
from transcription.db.runtime import get_session
async with get_session(settings=default_settings) as session:
result = await session.exec(text("SELECT 1"))
assert result.first()[0] == 1
def test_bootstrap_policy_production_defaults_false():
class TestBootstrapPolicy:
"""Verify startup schema bootstrap policy via Settings property."""
def test_production_defaults_to_no_bootstrap(self):
"""Production defaults to explicit non-bootstrap startup behavior."""
from transcription.config import Settings
settings = Settings(openrouter_api_key="test-key", environment="production")
assert settings.should_bootstrap_schema is False
def test_development_defaults_to_bootstrap(self):
"""Development defaults to schema bootstrap for local workflows."""
from transcription.config import Settings
def test_bootstrap_policy_development_defaults_true():
settings = Settings(openrouter_api_key="test-key", environment="development")
assert settings.should_bootstrap_schema is True
def test_explicit_override_wins(self):
"""Explicit bootstrap_schema_on_startup overrides environment default."""
from transcription.config import Settings
def test_bootstrap_policy_explicit_override_true():
settings = Settings(
openrouter_api_key="test-key",
environment="production",
tests/test_errors.py
+1 -4
View File
@@ -2,10 +2,7 @@
import pytest
from transcription.errors import AppError
from transcription.errors import ErrorCategory
from transcription.errors import classify_unexpected_error
from transcription.errors import new_error_id
from transcription.errors import AppError, ErrorCategory, classify_unexpected_error, new_error_id
@pytest.mark.unit
tests/test_models.py
+234 -96
View File
@@ -1,32 +1,31 @@
"""Tests for transcription.models — Document, Source, Job, Revision persistence and relationships."""
"""Tests for transcription.models — Document, Job, Transcript, TranscriptRevision models."""
from uuid import UUID
import pytest
from sqlalchemy.exc import IntegrityError
from transcription.models import Document
from transcription.models import Job
from transcription.models import JobStatus
from transcription.models import Revision
from transcription.models import Source
from transcription.models import Document, Job, JobStatus, Transcript, TranscriptRevision
def _make_document(**overrides) -> Document:
defaults = {"name": "letter bundle"}
"""Create a Document with sensible defaults."""
defaults = {"filename": "letter.jpg", "file_path": "/uploads/letter.jpg"}
defaults.update(overrides)
return Document(**defaults)
def _persist_document(session) -> Document:
document = _make_document()
session.add(document)
"""Create, persist, and return a Document."""
doc = _make_document()
session.add(doc)
session.commit()
session.refresh(document)
return document
session.refresh(doc)
return doc
def _persist_job(session, document: Document) -> Job:
"""Create, persist, and return a Job linked to a Document."""
job = Job(document_id=document.id)
session.add(job)
session.commit()
@@ -34,132 +33,271 @@ def _persist_job(session, document: Document) -> Job:
return job
def _persist_source(session, document: Document, job: Job, **overrides) -> Source:
defaults = {
"document_id": document.id,
"job_id": job.id,
"upload_name": "letter.jpg",
"filename": "stored-letter.jpg",
"file_path": "/uploads/stored-letter.jpg",
}
defaults.update(overrides)
source = Source(**defaults)
session.add(source)
session.commit()
session.refresh(source)
return source
class TestDocumentModel:
"""Verify Document creation and default field population."""
def test_can_be_persisted(self, session):
document = _persist_document(session)
fetched = session.get(Document, document.id)
"""A Document round-trips through the database with correct fields."""
doc = _persist_document(session)
fetched = session.get(Document, doc.id)
assert fetched is not None
assert fetched.name == "letter bundle"
assert fetched.filename == "letter.jpg"
assert fetched.file_path == "/uploads/letter.jpg"
def test_defaults_are_populated(self, session):
document = _persist_document(session)
assert isinstance(document.id, UUID)
"""id is a UUID and uploaded_at is populated on creation."""
doc = _persist_document(session)
assert isinstance(doc.id, UUID)
assert doc.uploaded_at is not None
class TestJobModel:
"""Verify Job creation, defaults, and status transitions."""
def test_can_be_created_for_document(self, session):
document = _persist_document(session)
job = _persist_job(session, document)
"""A Job linked to a Document via FK persists correctly."""
doc = _persist_document(session)
job = _persist_job(session, doc)
fetched = session.get(Job, job.id)
assert fetched is not None
assert fetched.document_id == document.id
assert fetched.document_id == doc.id
def test_defaults_are_populated(self, session):
document = _persist_document(session)
job = _persist_job(session, document)
"""Default status is queued; created_at and updated_at are populated."""
doc = _persist_document(session)
job = _persist_job(session, doc)
assert job.status == JobStatus.QUEUED
assert job.retry_count == 0
assert job.date_created is not None
assert job.date_updated is not None
assert job.created_at is not None
assert job.updated_at is not None
def test_transitions_to_transcribed(self, session):
document = _persist_document(session)
job = _persist_job(session, document)
"""Status updates from queued to processing to transcribed."""
doc = _persist_document(session)
job = _persist_job(session, doc)
assert job.status == JobStatus.QUEUED
job.status = JobStatus.PROCESSING
session.add(job)
session.commit()
session.refresh(job)
assert job.status == JobStatus.PROCESSING
job.status = JobStatus.TRANSCRIBED
session.add(job)
session.commit()
session.refresh(job)
assert job.status == JobStatus.TRANSCRIBED
def test_transitions_to_failed(self, session):
"""Status updates from processing to failed."""
doc = _persist_document(session)
job = _persist_job(session, doc)
job.status = JobStatus.PROCESSING
session.add(job)
session.commit()
session.refresh(job)
job.status = JobStatus.TRANSCRIBED
job.status = JobStatus.FAILED
session.add(job)
session.commit()
session.refresh(job)
assert job.status == JobStatus.TRANSCRIBED
assert job.status == JobStatus.FAILED
class TestSourceModel:
def test_can_be_created_for_document_and_job(self, session):
document = _persist_document(session)
job = _persist_job(session, document)
source = _persist_source(session, document, job)
class TestTranscriptModel:
"""Verify Transcript persistence for success and failure cases."""
fetched = session.get(Source, source.id)
assert fetched is not None
assert fetched.document_id == document.id
assert fetched.job_id == job.id
assert fetched.date_uploaded is not None
class TestRevisionModel:
def test_revision_persists_for_source(self, session):
document = _persist_document(session)
job = _persist_job(session, document)
source = _persist_source(session, document, job)
revision = Revision(source_id=source.id, text="Edited revision text")
session.add(revision)
def test_success_record_persists(self, session):
"""A Transcript with text set and error_detail None persists correctly."""
doc = _persist_document(session)
job = _persist_job(session, doc)
transcript = Transcript(job_id=job.id, provider="openrouter", prompt_name="transcribe_document.md", text="Dear Sir, ...")
session.add(transcript)
session.commit()
session.refresh(revision)
session.refresh(transcript)
fetched = session.get(Revision, revision.id)
fetched = session.get(Transcript, transcript.id)
assert fetched is not None
assert fetched.text == "Edited revision text"
assert fetched.date_created is not None
assert fetched.text == "Dear Sir, ..."
assert fetched.error_detail is None
def test_source_id_is_unique(self, session):
document = _persist_document(session)
job = _persist_job(session, document)
source = _persist_source(session, document, job)
def test_failure_record_persists(self, session):
"""A Transcript with text None and error_detail set persists correctly."""
doc = _persist_document(session)
job = _persist_job(session, doc)
transcript = Transcript(
job_id=job.id,
provider="openrouter",
prompt_name="transcribe_document.md",
error_detail="Provider timeout",
)
session.add(transcript)
session.commit()
session.refresh(transcript)
first = Revision(source_id=source.id, text="First")
session.add(first)
fetched = session.get(Transcript, transcript.id)
assert fetched is not None
assert fetched.text is None
assert fetched.error_detail == "Provider timeout"
def test_job_id_is_unique(self, session):
"""Inserting two transcripts with the same job_id raises an integrity error."""
doc = _persist_document(session)
job = _persist_job(session, doc)
t1 = Transcript(job_id=job.id, provider="openrouter", prompt_name="transcribe_document.md", text="First")
session.add(t1)
session.commit()
duplicate = Revision(source_id=source.id, text="Duplicate")
session.add(duplicate)
t2 = Transcript(job_id=job.id, provider="openrouter", prompt_name="transcribe_document.md", text="Duplicate")
session.add(t2)
with pytest.raises(IntegrityError):
session.commit()
class TestRelationships:
def test_document_exposes_jobs_and_sources(self, session):
document = _persist_document(session)
job = _persist_job(session, document)
_persist_source(session, document, job)
class TestTranscriptRevisionModel:
"""Verify TranscriptRevision persistence and version uniqueness constraints."""
session.refresh(document)
assert len(document.jobs) == 1
assert len(document.sources) == 1
def test_source_exposes_optional_single_revision(self, session):
document = _persist_document(session)
job = _persist_job(session, document)
source = _persist_source(session, document, job)
assert source.revision is None
revision = Revision(source_id=source.id, text="Edited")
def test_revision_record_persists(self, session):
"""A TranscriptRevision with version metadata persists correctly."""
doc = _persist_document(session)
job = _persist_job(session, doc)
revision = TranscriptRevision(
job_id=job.id,
version_number=1,
provider="openrouter",
prompt_name="transcribe_document.md",
model="google/gemini-2.5-flash",
source="ai",
text="Initial text",
)
session.add(revision)
session.commit()
session.refresh(revision)
session.refresh(source)
assert source.revision is not None
assert source.revision.text == "Edited"
fetched = session.get(TranscriptRevision, revision.id)
assert fetched is not None
assert fetched.version_number == 1
assert fetched.text == "Initial text"
assert fetched.source == "ai"
def test_job_version_pair_is_unique(self, session):
"""Duplicate version_number for same job raises integrity error."""
doc = _persist_document(session)
job = _persist_job(session, doc)
first = TranscriptRevision(
job_id=job.id,
version_number=1,
provider="openrouter",
prompt_name="transcribe_document.md",
source="ai",
text="Initial",
)
duplicate = TranscriptRevision(
job_id=job.id,
version_number=1,
provider="openrouter",
prompt_name="transcribe_document.md",
source="user",
text="Edited",
)
session.add(first)
session.commit()
session.add(duplicate)
with pytest.raises(IntegrityError):
session.commit()
def test_same_version_number_allowed_for_different_jobs(self, session):
"""Version numbers are scoped per job, not globally."""
doc1 = _persist_document(session)
job1 = _persist_job(session, doc1)
doc2 = _make_document(filename="letter2.jpg", file_path="/uploads/letter2.jpg")
session.add(doc2)
session.commit()
session.refresh(doc2)
job2 = _persist_job(session, doc2)
r1 = TranscriptRevision(
job_id=job1.id,
version_number=1,
provider="openrouter",
prompt_name="transcribe_document.md",
source="ai",
text="Job1 v1",
)
r2 = TranscriptRevision(
job_id=job2.id,
version_number=1,
provider="openrouter",
prompt_name="transcribe_document.md",
source="ai",
text="Job2 v1",
)
session.add(r1)
session.add(r2)
session.commit()
class TestRelationships:
"""Verify SQLModel relationship navigation between models."""
def test_document_exposes_jobs(self, session):
"""document.jobs returns the linked Job list."""
doc = _persist_document(session)
_persist_job(session, doc)
_persist_job(session, doc)
session.refresh(doc)
assert len(doc.jobs) == 2
assert all(isinstance(j, Job) for j in doc.jobs)
def test_job_exposes_transcript(self, session):
"""job.transcript returns the linked Transcript."""
doc = _persist_document(session)
job = _persist_job(session, doc)
transcript = Transcript(
job_id=job.id,
provider="openrouter",
prompt_name="transcribe_document.md",
text="Transcribed text",
)
session.add(transcript)
session.commit()
session.refresh(job)
assert job.transcript is not None
assert isinstance(job.transcript, Transcript)
assert job.transcript.text == "Transcribed text"
def test_job_exposes_transcript_revisions(self, session):
"""job.transcript_revisions returns all linked revisions."""
doc = _persist_document(session)
job = _persist_job(session, doc)
session.add(
TranscriptRevision(
job_id=job.id,
version_number=1,
provider="openrouter",
prompt_name="transcribe_document.md",
source="ai",
text="v1",
)
)
session.add(
TranscriptRevision(
job_id=job.id,
version_number=2,
provider="openrouter",
prompt_name="transcribe_document.md",
source="user",
text="v2",
)
)
session.commit()
session.refresh(job)
assert len(job.transcript_revisions) == 2
tests/test_prompts.py
+1
View File
@@ -2,6 +2,7 @@
from pathlib import Path
PROMPT_PATH = Path("prompts/transcribe_document.md")
tests/test_traceability.py
+6 -6
View File
@@ -9,19 +9,19 @@ MVP_REQUIREMENT_TEST_MAP: dict[str, list[str]] = {
"tests/integration/test_pipeline_flow.py",
],
"REQ-1": [
"tests/integration/test_pipeline_flow.py",
"tests/services/test_upload.py",
"tests/ui/test_upload_page.py",
],
"REQ-2": [
"tests/services/test_workflows_reliability.py",
"tests/services/test_worker.py",
"tests/integration/test_pipeline_flow.py",
],
"REQ-3": [
"tests/services/test_job_service.py",
"tests/services/test_worker.py",
"tests/ui/test_jobs_page.py",
],
"REQ-4": [
"tests/services/test_workflows_reliability.py",
"tests/services/test_worker.py",
"tests/integration/test_pipeline_flow.py",
],
"REQ-5": [
@@ -30,7 +30,7 @@ MVP_REQUIREMENT_TEST_MAP: dict[str, list[str]] = {
],
"REQ-6": [
"tests/test_app.py",
"tests/services/test_workflows_reliability.py",
"tests/services/test_worker.py",
],
"REQ-8": [
"tests/test_app.py",
@@ -38,7 +38,7 @@ MVP_REQUIREMENT_TEST_MAP: dict[str, list[str]] = {
],
"REQ-12": [
"tests/test_prompts.py",
"tests/services/test_transcription_external.py",
"tests/services/test_transcription.py",
],
}
tests/ui/conftest.py
-129
View File
@@ -1,129 +0,0 @@
"""Shared fixtures for UI integration tests."""
from __future__ import annotations
import asyncio
from collections.abc import Callable
from pathlib import Path
from uuid import UUID
import pytest
from fastapi import FastAPI
from fastapi.testclient import TestClient
from sqlmodel import delete
from transcription.app import create_app
from transcription.config import Settings
from transcription.config import _settings
from transcription.db import create_all
from transcription.db import get_session
from transcription.db import initialize_database_runtime
from transcription.models import Document
from transcription.models import Job
from transcription.models import JobStatus
from transcription.models import Revision
from transcription.models import Source
RevisionSeed = str
@pytest.fixture(scope="session")
def app_client(tmp_path_factory: pytest.TempPathFactory) -> tuple[FastAPI, TestClient]:
"""Provide a real application and test client backed by in-memory SQLite."""
tmp_path = tmp_path_factory.mktemp("ui")
settings = Settings(
openrouter_api_key="test-key",
database_url="sqlite:///:memory:",
environment="test",
bootstrap_schema_on_startup=True,
upload_dir=tmp_path / "uploads",
prompt_dir=tmp_path / "prompts",
)
_settings.set(settings)
app = create_app()
app.state.runtime = initialize_database_runtime(settings=settings)
asyncio.run(create_all(engine=app.state.runtime.engine))
with TestClient(app) as client:
yield app, client
@pytest.fixture(autouse=True)
def clear_ui_database(app_client: tuple[FastAPI, TestClient]) -> None:
"""Reset UI-facing tables before each test for isolation."""
app, _ = app_client
async def _clear() -> None:
async with get_session(session_factory=app.state.runtime.session_factory) as session:
await session.exec(delete(Revision))
await session.exec(delete(Source))
await session.exec(delete(Job))
await session.exec(delete(Document))
await session.commit()
asyncio.run(_clear())
@pytest.fixture
def seed_job(app_client: tuple[FastAPI, TestClient]) -> Callable[..., UUID]:
"""Return a helper for inserting a document/job/source/(optional revision) tuple."""
app, _ = app_client
fixtures_dir = Path(__file__).resolve().parents[1] / "fixtures" / "images" / "valid"
def _seed(
*,
filename: str = "sample.pdf",
status: JobStatus = JobStatus.TRANSCRIBED,
transcription_text: str | None = "Sample transcript text",
error_detail: str | None = None,
revision_text: RevisionSeed | None = None,
source_file: Path | None = None,
) -> UUID:
async def _insert() -> UUID:
async with get_session(session_factory=app.state.runtime.session_factory) as session:
stored_path = app.state.settings.upload_dir / filename
stored_path.parent.mkdir(parents=True, exist_ok=True)
source_path = source_file or fixtures_dir / "small_png.png"
stored_path.write_bytes(source_path.read_bytes())
document = Document(name=filename)
session.add(document)
await session.flush()
job = Job(
document_id=document.id,
status=status,
retry_count=0,
text=transcription_text,
error_detail=error_detail,
provider="openrouter",
model="google/gemini-2.5-flash",
prompt_name="transcribe_document.md",
)
session.add(job)
await session.flush()
source = Source(
document_id=document.id,
job_id=job.id,
upload_name=filename,
filename=filename,
file_path=str(stored_path),
)
session.add(source)
await session.flush()
if revision_text is not None:
session.add(
Revision(
source_id=source.id,
text=revision_text,
)
)
await session.commit()
return job.id
return asyncio.run(_insert())
return _seed
tests/ui/test_jobs_page.py
+25 -95
View File
@@ -1,107 +1,37 @@
"""Tests for the jobs page route."""
from pathlib import Path
from uuid import uuid4
import pytest
from fastapi import FastAPI
from fastapi.testclient import TestClient
from transcription.models import JobStatus
from transcription.ui import register_pages
from transcription.ui.pages import jobs_page
@pytest.fixture
def client(monkeypatch):
"""Provide a minimal app client with jobs data patched for rendering."""
async def _fetch_jobs_stub():
return []
monkeypatch.setattr(jobs_page, "fetch_jobs", _fetch_jobs_stub)
app = FastAPI()
register_pages(app)
with TestClient(app) as test_client:
yield test_client
@pytest.mark.integration
class TestPageRendering:
"""Verify jobs routes render correctly with real app wiring."""
"""Verify the jobs page is available and includes the main controls."""
def test_jobs_page_renders_empty_state(self, app_client):
"""GET /ui/jobs renders the page and empty-state text when no jobs exist."""
_, client = app_client
def test_jobs_page_renders_expected_controls(self, client):
"""GET /ui/jobs returns the page shell and jobs controls."""
response = client.get("/ui/jobs")
assert response.status_code == 200
assert "No jobs yet." in response.text
def test_jobs_page_lists_seeded_jobs(self, app_client, seed_job):
"""GET /ui/jobs lists seeded jobs from the in-memory database."""
_, client = app_client
seed_job(filename="sample.pdf", status=JobStatus.TRANSCRIBED, transcription_text="done")
response = client.get("/ui/jobs")
assert response.status_code == 200
assert "sample.pdf" in response.text
assert "transcribed" in response.text
def test_job_detail_page_renders_seeded_job(self, app_client, seed_job):
"""GET /ui/jobs/{job_id} renders detail content for a real seeded job."""
_, client = app_client
fixture_path = Path(__file__).resolve().parents[1] / "fixtures" / "images" / "valid" / "single_page_pdf.pdf"
job_id = seed_job(
filename="detail.pdf",
status=JobStatus.TRANSCRIBED,
transcription_text="original text",
revision_text="hello",
source_file=fixture_path,
)
response = client.get(f"/ui/jobs/{job_id}")
assert response.status_code == 200
assert "Original Transcription" in response.text
assert "detail.pdf" in response.text
assert "Revision" in response.text
assert "Revision" in response.text
assert "hello" in response.text
assert "original text" in response.text
assert "Document preview" in response.text
assert "/uploads/detail.pdf" in response.text
def test_job_detail_page_rejects_invalid_id(self, app_client):
"""GET /ui/jobs/{job_id} shows validation feedback for malformed IDs."""
_, client = app_client
response = client.get("/ui/jobs/not-a-uuid")
assert response.status_code == 200
assert "Invalid job id" in response.text
def test_job_detail_page_handles_missing_job(self, app_client):
"""GET /ui/jobs/{job_id} shows not-found state for unknown IDs."""
_, client = app_client
missing_id = uuid4()
response = client.get(f"/ui/jobs/{missing_id}")
assert response.status_code == 200
assert "Job not found" in response.text
def test_job_detail_page_shows_revision_editor_when_none_exists(self, app_client, seed_job):
"""GET /ui/jobs/{job_id} renders revision editor and create action for sources with no revision."""
_, client = app_client
job_id = seed_job(
filename="no-revision.pdf",
status=JobStatus.TRANSCRIBED,
transcription_text="original text",
revision_text=None,
)
response = client.get(f"/ui/jobs/{job_id}")
assert response.status_code == 200
assert "Revision Editor" in response.text
assert "Create revision" in response.text
assert "No revision exists for this source." in response.text
def test_job_detail_page_shows_update_action_for_existing_revision(self, app_client, seed_job):
"""GET /ui/jobs/{job_id} renders revision editor with update action when revision exists."""
_, client = app_client
job_id = seed_job(
filename="with-revision.pdf",
status=JobStatus.TRANSCRIBED,
transcription_text="original text",
revision_text="hello",
)
response = client.get(f"/ui/jobs/{job_id}")
assert response.status_code == 200
assert "Revision Editor" in response.text
assert "Update revision" in response.text
assert "hello" in response.text
assert "Transcription Jobs" in response.text
assert "Refresh" in response.text
assert "Back to upload" in response.text
tests/ui/test_pages_registration.py
+28 -8
View File
@@ -1,18 +1,38 @@
"""Tests for UI page registration wiring."""
import pytest
from fastapi import FastAPI
from transcription.ui import register_pages
@pytest.mark.integration
class TestPageRegistration:
"""Verify page registration and mounted UI routes."""
"""Verify page registration and route wiring."""
def test_ui_mount_serves_registered_pages(self, app_client):
"""Mounted UI routes respond successfully when the full app is created."""
_, client = app_client
def test_register_pages_wires_upload_jobs_and_mount(self, monkeypatch):
"""register_pages registers pages and mounts NiceGUI at /ui."""
calls: list[str] = []
upload_response = client.get("/ui/upload")
jobs_response = client.get("/ui/jobs")
def _record_upload() -> None:
calls.append("upload")
assert upload_response.status_code == 200
assert jobs_response.status_code == 200
def _record_jobs() -> None:
calls.append("jobs")
def _record_run_with(
_app: FastAPI,
*,
mount_path: str,
show_welcome_message: bool,
) -> None:
calls.append(f"run_with:{mount_path}:{show_welcome_message}")
monkeypatch.setattr("transcription.ui.register_upload_page", _record_upload)
monkeypatch.setattr("transcription.ui.register_jobs_page", _record_jobs)
monkeypatch.setattr("transcription.ui.ui.run_with", _record_run_with)
app = FastAPI()
register_pages(app)
assert calls == ["upload", "jobs", "run_with:/ui:False"]
tests/ui/test_upload_page.py
+30 -10
View File
@@ -1,35 +1,55 @@
"""Tests for upload and entry-point routes."""
"""Tests for the upload page route."""
from pathlib import Path
import pytest
from fastapi.testclient import TestClient
from transcription.app import create_app
from transcription.config import Settings
from transcription.config import _settings
@pytest.fixture
def client(tmp_path: Path):
"""Provide a real app client backed by in-memory SQLite."""
settings = Settings(
openrouter_api_key="test-key",
database_url="sqlite:///:memory:",
environment="test",
upload_dir=tmp_path / "uploads",
prompt_dir=tmp_path / "prompts",
)
_settings.set(settings)
app = create_app()
with TestClient(app) as test_client:
yield test_client
@pytest.mark.integration
class TestPageRendering:
"""Verify upload-related routes return working pages."""
"""Verify the upload page is available and includes the main controls."""
def test_root_redirects_to_ui(self, app_client):
def test_root_redirects_to_ui(self, client):
"""GET / redirects to the UI mount point."""
_, client = app_client
response = client.get("/", follow_redirects=False)
assert response.status_code == 307
assert response.headers["location"] == "/ui"
def test_ui_redirects_to_upload(self, app_client):
def test_ui_redirects_to_upload(self, client):
"""GET /ui redirects to the upload page."""
_, client = app_client
response = client.get("/ui", follow_redirects=False)
assert response.status_code == 307
assert response.headers["location"] == "/ui/upload"
def test_upload_page_renders_expected_controls(self, app_client):
def test_upload_page_renders_expected_controls(self, client):
"""GET /ui/upload returns the page shell and upload controls."""
_, client = app_client
response = client.get("/ui/upload")
assert response.status_code == 200
assert "Upload Document" in response.text
assert "Select document file" in response.text
assert "Upload" in response.text
assert "Jobs" in response.text
assert "View jobs" in response.text
uploads/01c483af-e590-4e12-b2e7-0183ef11e4fe_Lowry Record - page 16.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1.0 MiB

uploads/25148a77-bfce-40a6-a5f2-148008ede6fd_Rod Moser Letter - p2.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1.0 MiB

uploads/287bb88d-80c1-4553-aedc-35f2167c49b6_Book Two - page 61.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1.2 MiB

uploads/3b58a01a-8d97-4166-8a47-a99b75a34100_Rod Moser Letter - p3.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 969 KiB

uploads/7704f9cc-4b04-494b-8c55-0ea720249a54_Biography of Omie Rachel Cochran McCarthy_ 1892-1961.pdf → uploads/6365d9a7-402c-4d4c-a360-8824e9e6a936_Biography of Omie Rachel Cochran McCarthy_ 1892-1961.pdf
View File
uploads/7c6ee648-4171-4e70-b901-ba94ee227d6a_Time Rolls On - page 013.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1.8 MiB

uploads/8b799b25-7d4b-40eb-8cd4-69516343372f_Book Two - page 12.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1.1 MiB

uploads/aa9c92f7-5ad2-4c7d-864e-53ef5390b85d_Rod Moser Letter - p1.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1010 KiB

uploads/b04b640b-dce2-4543-b8da-74ea764bdae7_Time Rolls On - page 011.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1.7 MiB

uploads/bef10e09-d48d-40e9-85c4-9ec53735e85b_Book Two - page 18.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1.1 MiB

uploads/ca87e22d-21bf-4af8-8cff-312e9384bd24_Effie Ruth Cochran Montague.pdf
BIN
View File
Binary file not shown.
uploads/cc913a4f-77e3-4b77-aae3-0898f12bbc96_Hig's postcards to Zenna - November 24, 1924.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 6.0 MiB

uploads/f2984991-e861-4db3-8612-2da1a8ba23d2_Book Two - page 03.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1.2 MiB