generated from john/python-template
V1 mostly complete except for some testing. Linting in the last step changed nearly every file which is why this commit is so larger.
This commit is contained in:
Jim Lancaster
+79
-248
@@ -1,299 +1,130 @@
|
||||
# Architecture
|
||||
# Architecture (V1 Baseline)
|
||||
|
||||
This document describes the production architecture of the personal historical-document transcription system. The system is intentionally optimized for single-user operation, low operational overhead, and clean internal boundaries that support future growth without rewrites.
|
||||
This document describes the current architecture of the personal historical-document transcription system and serves as the V1 technical baseline.
|
||||
|
||||
## Architecture Objectives
|
||||
|
||||
The production architecture is designed to:
|
||||
- preserve source material as transcribed text
|
||||
- keep operational complexity low for personal-scale deployment
|
||||
- support asynchronous processing without external queue infrastructure
|
||||
- maintain clear module boundaries for incremental extension
|
||||
|
||||
- preserve verbatim family-history source material as searchable text
|
||||
- keep operational complexity low for a personal deployment
|
||||
- support asynchronous transcription without requiring distributed infrastructure
|
||||
- maintain clear module boundaries so extensions can be added incrementally
|
||||
## Runtime Topology
|
||||
|
||||
## Production Scope And Scale
|
||||
V1 runtime is a modular monolith:
|
||||
|
||||
The deployed system targets personal use and a corpus of several thousand documents processed over time. The architecture favors simple, composable building blocks over distributed orchestration.
|
||||
|
||||
Current scope includes:
|
||||
|
||||
- content source upload and metadata capture
|
||||
- asynchronous transcription jobs
|
||||
- prompt-library driven transcription behavior, with one Markdown file per prompt
|
||||
- original transcription review and optional revision review
|
||||
- full-text search over accepted transcripts
|
||||
- export of transcript data
|
||||
|
||||
## Deployment Topology
|
||||
|
||||
The production deployment uses [Docker Compose](https://docs.docker.com/compose/) and treats containerized databases as extremely lightweight operational dependencies.
|
||||
|
||||
Running [PostgreSQL](https://www.postgresql.org/docs/) in its own container is considered simple by default for this system.
|
||||
|
||||
Running [MongoDB](https://www.mongodb.com/docs/) in its own container is also considered simple when document-centric storage is enabled.
|
||||
|
||||
Container count is not a hard architectural limit; a three-container deployment (app, PostgreSQL, MongoDB) is an acceptable baseline.
|
||||
|
||||
### Baseline Topology (Two Containers)
|
||||
|
||||
- one application container
|
||||
- one PostgreSQL container
|
||||
- embedded background worker execution inside the app process
|
||||
|
||||
### Expanded Topology (Three Containers)
|
||||
|
||||
- application container
|
||||
- PostgreSQL container
|
||||
- MongoDB container
|
||||
|
||||
No additional queue, scheduler, or search-engine containers are required in the baseline production setup.
|
||||
|
||||
## Runtime Architecture
|
||||
- one FastAPI + NiceGUI application process
|
||||
- one in-process async worker loop
|
||||
- relational persistence via SQLModel (SQLite baseline)
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
User[Browser User] --> App[FastAPI + NiceGUI Service]
|
||||
App --> Worker[In-process Background Worker]
|
||||
App --> PG[(PostgreSQL)]
|
||||
App --> MG[(MongoDB Document Store)]
|
||||
Worker --> AI[Transcription Provider]
|
||||
Worker --> PG
|
||||
Worker --> MG
|
||||
U[Browser User] --> A[FastAPI + NiceGUI App]
|
||||
A --> W[In-process Worker]
|
||||
A --> DB[(SQLite via SQLModel)]
|
||||
W --> P[OpenRouter Provider]
|
||||
W --> DB
|
||||
```
|
||||
|
||||
## Runtime Ownership And Startup Policy
|
||||
## Lifecycle Ownership
|
||||
|
||||
The current implementation now uses explicit lifespan-owned runtime resources.
|
||||
Application lifespan owns runtime setup/teardown:
|
||||
|
||||
- application lifespan initializes and disposes database runtime resources
|
||||
- worker lifecycle is owned by application lifespan startup/shutdown
|
||||
- worker receives lifespan-owned database engine dependency explicitly
|
||||
- schema bootstrap policy is environment-aware and explicit:
|
||||
- development/test default to bootstrap enabled
|
||||
- production defaults to bootstrap disabled
|
||||
- explicit override is available via configuration
|
||||
|
||||
This aligns implementation toward REQ-7 and REQ-10 while preserving personal-scale operational simplicity.
|
||||
- configure logging
|
||||
- initialize and dispose DB runtime resources
|
||||
- optional schema bootstrap by environment policy
|
||||
- recover stale processing jobs
|
||||
- start/stop worker consumer lifespan
|
||||
|
||||
## Layered Module Structure
|
||||
|
||||
### Interface Layer
|
||||
|
||||
Responsibility:
|
||||
- `src/transcription/ui/**` (NiceGUI pages/components)
|
||||
- `src/transcription/api/**` (FastAPI routes and error handlers)
|
||||
|
||||
- HTTP API and UI routes
|
||||
- request/response validation
|
||||
- status and result presentation
|
||||
### Application/Workflow Layer
|
||||
|
||||
Out of scope:
|
||||
- `src/transcription/services/workflows.py`
|
||||
- `src/transcription/worker.py`
|
||||
|
||||
- business-rule enforcement
|
||||
- data-access implementation
|
||||
Responsibilities:
|
||||
|
||||
### Application Layer
|
||||
- orchestration and status transitions
|
||||
- retry/timeout behavior
|
||||
- provider call coordination
|
||||
|
||||
Responsibility:
|
||||
### Service Layer
|
||||
|
||||
- upload and job orchestration
|
||||
- state transitions and retry policy
|
||||
- coordination across domain and infrastructure ports
|
||||
- `src/transcription/services/*.py`
|
||||
|
||||
Out of scope:
|
||||
Responsibilities:
|
||||
|
||||
- provider-specific protocol details
|
||||
- ORM or storage-specific logic
|
||||
|
||||
### Domain Layer
|
||||
|
||||
Responsibility:
|
||||
|
||||
- verbatim transcription policy
|
||||
- revision and provenance invariants
|
||||
- confidence and annotation semantics
|
||||
|
||||
Out of scope:
|
||||
|
||||
- web framework concerns
|
||||
- database and network I/O
|
||||
- CRUD and transactional boundaries
|
||||
- domain-aligned persistence operations
|
||||
|
||||
### Infrastructure Layer
|
||||
|
||||
Responsibility:
|
||||
|
||||
- persistence adapters (PostgreSQL and MongoDB)
|
||||
- transcription-provider adapter
|
||||
|
||||
Out of scope:
|
||||
|
||||
- business policy decisions
|
||||
- `src/transcription/db/**` (runtime/session/bootstrap)
|
||||
- `src/transcription/providers/**` (OpenRouter adapter)
|
||||
|
||||
## Processing Workflow
|
||||
|
||||
Production transcription flow:
|
||||
1. User uploads a source file from the UI.
|
||||
2. App persists `Document`, `Job(queued)`, and `Source`.
|
||||
3. Worker claims next queued job and marks `processing`.
|
||||
4. Worker calls provider with prompt + source bytes.
|
||||
5. On success, app writes immutable `Job.text` and marks `transcribed`.
|
||||
6. On failure, app writes `Job.error_detail` and marks `failed`.
|
||||
7. UI exposes job detail, original transcription, and optional revision.
|
||||
|
||||
1. A user uploads one or more content sources through the UI or API.
|
||||
2. The application validates payloads and creates document, source, and job records.
|
||||
3. The in-process worker de-queues the job and calls the transcription provider.
|
||||
4. The application persists original transcription output on the job, plus confidence metadata and provenance events.
|
||||
5. Job status transitions from queued to processing to transcribed or failed.
|
||||
6. The UI and API expose status, optional revision to original transcription, and searchable transcription text.
|
||||
## Domain Ownership Invariants
|
||||
|
||||
## Data Model Ownership
|
||||
- `Job.text` is immutable original provider output.
|
||||
- `Revision` is optional, user-authored, and linked to `Source`.
|
||||
- `Revision` does not overwrite original job transcription.
|
||||
- Status lifecycle is fixed to: `queued -> processing -> transcribed|failed`.
|
||||
|
||||
System-of-record entities:
|
||||
## Data Model Summary
|
||||
|
||||
- documents and content sources
|
||||
- transcription jobs, original transcription, and status events
|
||||
- transcript revisions
|
||||
- provenance metadata
|
||||
- `Document` has many `Source` and many `Job`.
|
||||
- `Source` belongs to one `Document` and one `Job`.
|
||||
- `Source` has optional `Revision` (`0..1`) enforced by unique `revision.source_id`.
|
||||
|
||||
### Original Transcription And Revision Ownership
|
||||
## Simplicity Guardrails (V1)
|
||||
|
||||
- each processing job stores the original immutable provider output (`text`)
|
||||
- provider metadata (`provider`, `model`, `prompt_name`) and failure detail (`error_detail`) are job-owned processing artifacts
|
||||
- revisions are optional user-authored edits linked to a content source
|
||||
- a revision can be created from original `job.text`
|
||||
- many jobs will have zero revisions; revisions are additive and never overwrite original provider output
|
||||
- a document groups one or more content sources (images, PDFs, and future source types)
|
||||
|
||||
Storage strategy:
|
||||
|
||||
- PostgreSQL for relational system-of-record entities
|
||||
- MongoDB for document-oriented payloads and large transcription artifacts
|
||||
- versioned prompt artifacts stored as individual Markdown files for human editing and refinement
|
||||
- in-memory execution state treated as ephemeral
|
||||
|
||||
## Transcription Prompt Asset Policy
|
||||
|
||||
The production system treats transcription prompts as maintainable content assets.
|
||||
|
||||
- each transcription prompt is stored in its own Markdown file
|
||||
- prompt files are designed for direct human editing and iterative refinement
|
||||
- prompt updates are independent and do not require bundling unrelated prompt changes
|
||||
- prompt file identity and revision history are tracked through normal repository version control
|
||||
|
||||
## Simplicity Guardrails
|
||||
|
||||
The production system enforces these constraints to prevent accidental over-engineering:
|
||||
|
||||
- PostgreSQL in a container is treated as a lightweight default dependency
|
||||
- MongoDB in a container is treated as a lightweight optional dependency
|
||||
- three containers (app, PostgreSQL, MongoDB) is an acceptable simple deployment
|
||||
- no dedicated queue or search cluster is introduced without measured need
|
||||
- external infrastructure is added only behind existing ports/adapters
|
||||
- no external queue/broker required
|
||||
- no search engine required
|
||||
- no distributed worker fleet required
|
||||
- keep provider integration behind adapter boundary
|
||||
|
||||
## Extension Path
|
||||
|
||||
The architecture supports additive growth without changing domain contracts.
|
||||
### V1 (current)
|
||||
|
||||
### Stage 1: Foundation (Current)
|
||||
- SQLite baseline
|
||||
- OpenRouter provider
|
||||
- in-process worker
|
||||
- optional single revision workflow
|
||||
|
||||
- upload, transcription, review, search, export
|
||||
- in-process worker execution
|
||||
- single provider adapter
|
||||
- app plus PostgreSQL deployment
|
||||
### V2 (planned)
|
||||
|
||||
### Stage 2: Throughput Hardening
|
||||
- PostgreSQL as relational baseline
|
||||
- optional MongoDB adjunct store for scoped use cases
|
||||
- migration-first schema evolution
|
||||
|
||||
- optional MongoDB document-store enablement
|
||||
- optional external worker/queue process
|
||||
- stronger retry and dead-letter handling
|
||||
|
||||
### Stage 3: Intelligence Features
|
||||
|
||||
- entity extraction and cross-document linking
|
||||
- timeline and narrative assembly
|
||||
- optional multi-provider routing
|
||||
|
||||
Each stage preserves existing module boundaries and keeps migration risk low.
|
||||
See [ver2/ver2.md](ver2/ver2.md) for roadmap details.
|
||||
|
||||
## Test Strategy
|
||||
|
||||
The test strategy is aligned to personal-scale operation with fast, deterministic feedback.
|
||||
- unit tests for model/service behaviors
|
||||
- integration tests for upload/workflow reliability
|
||||
- UI integration tests for page/render contracts
|
||||
- external provider tests opt-in via marker/config
|
||||
|
||||
### Unit Tests
|
||||
## Related References
|
||||
|
||||
- domain transcription rules and annotation behavior
|
||||
- revision-history invariants
|
||||
- job state-transition logic
|
||||
|
||||
### Integration Tests
|
||||
|
||||
- repository behavior and transaction boundaries
|
||||
- persistence-adapter and provider adapter contract mapping
|
||||
- upload-to-persistence roundtrip
|
||||
|
||||
### End-to-End Tests
|
||||
|
||||
- happy path: upload, transcribe, review, search, export
|
||||
- failure path: provider error, retry, surfaced failed status
|
||||
|
||||
### CI Execution Model
|
||||
|
||||
- fast suite on each push
|
||||
- optional slower provider-sandbox checks on scheduled runs
|
||||
|
||||
## Risks And Controls
|
||||
|
||||
### Runtime Responsiveness
|
||||
|
||||
Risk:
|
||||
|
||||
- long jobs can reduce responsiveness in a single-process deployment
|
||||
|
||||
Control:
|
||||
|
||||
- bounded concurrency and visible job status in the UI
|
||||
|
||||
### Database Concurrency Limits
|
||||
|
||||
Risk:
|
||||
|
||||
- contention can appear under sustained concurrent writes in personal-scale infrastructure
|
||||
|
||||
Control:
|
||||
|
||||
- tuned connection pooling and phased use of MongoDB for document-heavy workloads
|
||||
|
||||
### Provider Output Variance
|
||||
|
||||
Risk:
|
||||
|
||||
- transcription quality varies by content source type, handwriting legibility, and source quality
|
||||
|
||||
Control:
|
||||
|
||||
- first-class human review and immutable revision history
|
||||
|
||||
## Technology References
|
||||
|
||||
- [FastAPI documentation](https://fastapi.tiangolo.com/)
|
||||
- [NiceGUI documentation](https://nicegui.io/documentation)
|
||||
- [Docker Compose documentation](https://docs.docker.com/compose/)
|
||||
- [PostgreSQL documentation](https://www.postgresql.org/docs/)
|
||||
- [MongoDB documentation](https://www.mongodb.com/docs/)
|
||||
|
||||
## Related Local References
|
||||
|
||||
- [System overview](index.md)
|
||||
|
||||
## Glossary
|
||||
|
||||
- Adapter: A component that translates between internal interfaces and external systems such as databases or AI services.
|
||||
- Background job: Work executed outside the request/response path so the UI remains responsive.
|
||||
- Boundary: A strict separation between modules with different responsibilities.
|
||||
- CI (Continuous Integration): Automated test execution for code changes.
|
||||
- Contract test: A test that verifies an adapter follows expected input/output behavior at a boundary.
|
||||
- Domain layer: The module that contains core business rules and invariants.
|
||||
- End-to-end test: A test that validates a full user flow across the running system.
|
||||
- Full-text search: Text indexing and querying optimized for natural-language search.
|
||||
- In-process worker: A background executor that runs within the same application process.
|
||||
- Integration test: A test that verifies interactions between real modules and infrastructure components.
|
||||
- MongoDB: A document-oriented database used for flexible, high-variance data structures.
|
||||
- Modular monolith: A single deployable application with strongly separated internal modules.
|
||||
- Port/Interface: A stable contract used by application/domain code to call infrastructure implementations.
|
||||
- Prompt artifact: A single Markdown file that defines one transcription prompt and can be revised independently.
|
||||
- Provenance: Metadata that records where generated data came from and how it was produced.
|
||||
- Revision history: Optional versioned record of user-authored transcription edits over time.
|
||||
- System of record: The authoritative persistent store for canonical data.
|
||||
- Vertical slice: A minimal end-to-end feature path spanning UI/API, application logic, and persistence.
|
||||
- [index.md](index.md)
|
||||
- [requirements.md](requirements.md)
|
||||
- [schema.md](schema.md)
|
||||
- [error_handling.md](error_handling.md)
|
||||
|
||||
@@ -0,0 +1,23 @@
|
||||
# V2 Archive
|
||||
|
||||
This folder preserves pre-V1-alignment versions of core documentation that included planned target-state architecture material.
|
||||
|
||||
Archived snapshots:
|
||||
|
||||
- `index.pre-v1-alignment.md`
|
||||
- `requirements.pre-v1-alignment.md`
|
||||
- `architecture.pre-v1-alignment.md`
|
||||
|
||||
Purpose:
|
||||
|
||||
- keep a durable reference for planned architecture language
|
||||
- reduce risk of losing useful V2 direction while V1 docs stay implementation-aligned
|
||||
|
||||
Notes:
|
||||
|
||||
- These files are historical snapshots, not the active V1 source of truth.
|
||||
- Active V1 docs remain at:
|
||||
- `docs/index.md`
|
||||
- `docs/requirements.md`
|
||||
- `docs/architecture.md`
|
||||
- V2 planning should continue in `docs/ver2/ver2.md` and related V2 artifacts.
|
||||
@@ -0,0 +1,299 @@
|
||||
# Architecture
|
||||
|
||||
This document describes the production architecture of the personal historical-document transcription system. The system is intentionally optimized for single-user operation, low operational overhead, and clean internal boundaries that support future growth without rewrites.
|
||||
|
||||
## Architecture Objectives
|
||||
|
||||
The production architecture is designed to:
|
||||
|
||||
- preserve verbatim family-history source material as searchable text
|
||||
- keep operational complexity low for a personal deployment
|
||||
- support asynchronous transcription without requiring distributed infrastructure
|
||||
- maintain clear module boundaries so extensions can be added incrementally
|
||||
|
||||
## Production Scope And Scale
|
||||
|
||||
The deployed system targets personal use and a corpus of several thousand documents processed over time. The architecture favors simple, composable building blocks over distributed orchestration.
|
||||
|
||||
Current scope includes:
|
||||
|
||||
- content source upload and metadata capture
|
||||
- asynchronous transcription jobs
|
||||
- prompt-library driven transcription behavior, with one Markdown file per prompt
|
||||
- original transcription review and optional revision review
|
||||
- full-text search over accepted transcripts
|
||||
- export of transcript data
|
||||
|
||||
## Deployment Topology
|
||||
|
||||
The production deployment uses [Docker Compose](https://docs.docker.com/compose/) and treats containerized databases as extremely lightweight operational dependencies.
|
||||
|
||||
Running [PostgreSQL](https://www.postgresql.org/docs/) in its own container is considered simple by default for this system.
|
||||
|
||||
Running [MongoDB](https://www.mongodb.com/docs/) in its own container is also considered simple when document-centric storage is enabled.
|
||||
|
||||
Container count is not a hard architectural limit; a three-container deployment (app, PostgreSQL, MongoDB) is an acceptable baseline.
|
||||
|
||||
### Baseline Topology (Two Containers)
|
||||
|
||||
- one application container
|
||||
- one PostgreSQL container
|
||||
- embedded background worker execution inside the app process
|
||||
|
||||
### Expanded Topology (Three Containers)
|
||||
|
||||
- application container
|
||||
- PostgreSQL container
|
||||
- MongoDB container
|
||||
|
||||
No additional queue, scheduler, or search-engine containers are required in the baseline production setup.
|
||||
|
||||
## Runtime Architecture
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
User[Browser User] --> App[FastAPI + NiceGUI Service]
|
||||
App --> Worker[In-process Background Worker]
|
||||
App --> PG[(PostgreSQL)]
|
||||
App --> MG[(MongoDB Document Store)]
|
||||
Worker --> AI[Transcription Provider]
|
||||
Worker --> PG
|
||||
Worker --> MG
|
||||
```
|
||||
|
||||
## Runtime Ownership And Startup Policy
|
||||
|
||||
The current implementation now uses explicit lifespan-owned runtime resources.
|
||||
|
||||
- application lifespan initializes and disposes database runtime resources
|
||||
- worker lifecycle is owned by application lifespan startup/shutdown
|
||||
- worker receives lifespan-owned database engine dependency explicitly
|
||||
- schema bootstrap policy is environment-aware and explicit:
|
||||
- development/test default to bootstrap enabled
|
||||
- production defaults to bootstrap disabled
|
||||
- explicit override is available via configuration
|
||||
|
||||
This aligns implementation toward REQ-7 and REQ-10 while preserving personal-scale operational simplicity.
|
||||
|
||||
## Layered Module Structure
|
||||
|
||||
### Interface Layer
|
||||
|
||||
Responsibility:
|
||||
|
||||
- HTTP API and UI routes
|
||||
- request/response validation
|
||||
- status and result presentation
|
||||
|
||||
Out of scope:
|
||||
|
||||
- business-rule enforcement
|
||||
- data-access implementation
|
||||
|
||||
### Application Layer
|
||||
|
||||
Responsibility:
|
||||
|
||||
- upload and job orchestration
|
||||
- state transitions and retry policy
|
||||
- coordination across domain and infrastructure ports
|
||||
|
||||
Out of scope:
|
||||
|
||||
- provider-specific protocol details
|
||||
- ORM or storage-specific logic
|
||||
|
||||
### Domain Layer
|
||||
|
||||
Responsibility:
|
||||
|
||||
- verbatim transcription policy
|
||||
- revision and provenance invariants
|
||||
- confidence and annotation semantics
|
||||
|
||||
Out of scope:
|
||||
|
||||
- web framework concerns
|
||||
- database and network I/O
|
||||
|
||||
### Infrastructure Layer
|
||||
|
||||
Responsibility:
|
||||
|
||||
- persistence adapters (PostgreSQL and MongoDB)
|
||||
- transcription-provider adapter
|
||||
|
||||
Out of scope:
|
||||
|
||||
- business policy decisions
|
||||
|
||||
## Processing Workflow
|
||||
|
||||
Production transcription flow:
|
||||
|
||||
1. A user uploads one or more content sources through the UI or API.
|
||||
2. The application validates payloads and creates document, source, and job records.
|
||||
3. The in-process worker de-queues the job and calls the transcription provider.
|
||||
4. The application persists original transcription output on the job, plus confidence metadata and provenance events.
|
||||
5. Job status transitions from queued to processing to transcribed or failed.
|
||||
6. The UI and API expose status, optional revision to original transcription, and searchable transcription text.
|
||||
|
||||
## Data Model Ownership
|
||||
|
||||
System-of-record entities:
|
||||
|
||||
- documents and content sources
|
||||
- transcription jobs, original transcription, and status events
|
||||
- transcript revisions
|
||||
- provenance metadata
|
||||
|
||||
### Original Transcription And Revision Ownership
|
||||
|
||||
- each processing job stores the original immutable provider output (`text`)
|
||||
- provider metadata (`provider`, `model`, `prompt_name`) and failure detail (`error_detail`) are job-owned processing artifacts
|
||||
- revisions are optional user-authored edits linked to a content source
|
||||
- a revision can be created from original `job.text`
|
||||
- many jobs will have zero revisions; revisions are additive and never overwrite original provider output
|
||||
- a document groups one or more content sources (images, PDFs, and future source types)
|
||||
|
||||
Storage strategy:
|
||||
|
||||
- PostgreSQL for relational system-of-record entities
|
||||
- MongoDB for document-oriented payloads and large transcription artifacts
|
||||
- versioned prompt artifacts stored as individual Markdown files for human editing and refinement
|
||||
- in-memory execution state treated as ephemeral
|
||||
|
||||
## Transcription Prompt Asset Policy
|
||||
|
||||
The production system treats transcription prompts as maintainable content assets.
|
||||
|
||||
- each transcription prompt is stored in its own Markdown file
|
||||
- prompt files are designed for direct human editing and iterative refinement
|
||||
- prompt updates are independent and do not require bundling unrelated prompt changes
|
||||
- prompt file identity and revision history are tracked through normal repository version control
|
||||
|
||||
## Simplicity Guardrails
|
||||
|
||||
The production system enforces these constraints to prevent accidental over-engineering:
|
||||
|
||||
- PostgreSQL in a container is treated as a lightweight default dependency
|
||||
- MongoDB in a container is treated as a lightweight optional dependency
|
||||
- three containers (app, PostgreSQL, MongoDB) is an acceptable simple deployment
|
||||
- no dedicated queue or search cluster is introduced without measured need
|
||||
- external infrastructure is added only behind existing ports/adapters
|
||||
|
||||
## Extension Path
|
||||
|
||||
The architecture supports additive growth without changing domain contracts.
|
||||
|
||||
### Stage 1: Foundation (Current)
|
||||
|
||||
- upload, transcription, review, search, export
|
||||
- in-process worker execution
|
||||
- single provider adapter
|
||||
- app plus PostgreSQL deployment
|
||||
|
||||
### Stage 2: Throughput Hardening
|
||||
|
||||
- optional MongoDB document-store enablement
|
||||
- optional external worker/queue process
|
||||
- stronger retry and dead-letter handling
|
||||
|
||||
### Stage 3: Intelligence Features
|
||||
|
||||
- entity extraction and cross-document linking
|
||||
- timeline and narrative assembly
|
||||
- optional multi-provider routing
|
||||
|
||||
Each stage preserves existing module boundaries and keeps migration risk low.
|
||||
|
||||
## Test Strategy
|
||||
|
||||
The test strategy is aligned to personal-scale operation with fast, deterministic feedback.
|
||||
|
||||
### Unit Tests
|
||||
|
||||
- domain transcription rules and annotation behavior
|
||||
- revision-history invariants
|
||||
- job state-transition logic
|
||||
|
||||
### Integration Tests
|
||||
|
||||
- repository behavior and transaction boundaries
|
||||
- persistence-adapter and provider adapter contract mapping
|
||||
- upload-to-persistence roundtrip
|
||||
|
||||
### End-to-End Tests
|
||||
|
||||
- happy path: upload, transcribe, review, search, export
|
||||
- failure path: provider error, retry, surfaced failed status
|
||||
|
||||
### CI Execution Model
|
||||
|
||||
- fast suite on each push
|
||||
- optional slower provider-sandbox checks on scheduled runs
|
||||
|
||||
## Risks And Controls
|
||||
|
||||
### Runtime Responsiveness
|
||||
|
||||
Risk:
|
||||
|
||||
- long jobs can reduce responsiveness in a single-process deployment
|
||||
|
||||
Control:
|
||||
|
||||
- bounded concurrency and visible job status in the UI
|
||||
|
||||
### Database Concurrency Limits
|
||||
|
||||
Risk:
|
||||
|
||||
- contention can appear under sustained concurrent writes in personal-scale infrastructure
|
||||
|
||||
Control:
|
||||
|
||||
- tuned connection pooling and phased use of MongoDB for document-heavy workloads
|
||||
|
||||
### Provider Output Variance
|
||||
|
||||
Risk:
|
||||
|
||||
- transcription quality varies by content source type, handwriting legibility, and source quality
|
||||
|
||||
Control:
|
||||
|
||||
- first-class human review and immutable revision history
|
||||
|
||||
## Technology References
|
||||
|
||||
- [FastAPI documentation](https://fastapi.tiangolo.com/)
|
||||
- [NiceGUI documentation](https://nicegui.io/documentation)
|
||||
- [Docker Compose documentation](https://docs.docker.com/compose/)
|
||||
- [PostgreSQL documentation](https://www.postgresql.org/docs/)
|
||||
- [MongoDB documentation](https://www.mongodb.com/docs/)
|
||||
|
||||
## Related Local References
|
||||
|
||||
- [System overview](index.md)
|
||||
|
||||
## Glossary
|
||||
|
||||
- Adapter: A component that translates between internal interfaces and external systems such as databases or AI services.
|
||||
- Background job: Work executed outside the request/response path so the UI remains responsive.
|
||||
- Boundary: A strict separation between modules with different responsibilities.
|
||||
- CI (Continuous Integration): Automated test execution for code changes.
|
||||
- Contract test: A test that verifies an adapter follows expected input/output behavior at a boundary.
|
||||
- Domain layer: The module that contains core business rules and invariants.
|
||||
- End-to-end test: A test that validates a full user flow across the running system.
|
||||
- Full-text search: Text indexing and querying optimized for natural-language search.
|
||||
- In-process worker: A background executor that runs within the same application process.
|
||||
- Integration test: A test that verifies interactions between real modules and infrastructure components.
|
||||
- MongoDB: A document-oriented database used for flexible, high-variance data structures.
|
||||
- Modular monolith: A single deployable application with strongly separated internal modules.
|
||||
- Port/Interface: A stable contract used by application/domain code to call infrastructure implementations.
|
||||
- Prompt artifact: A single Markdown file that defines one transcription prompt and can be revised independently.
|
||||
- Provenance: Metadata that records where generated data came from and how it was produced.
|
||||
- Revision history: Optional versioned record of user-authored transcription edits over time.
|
||||
- System of record: The authoritative persistent store for canonical data.
|
||||
- Vertical slice: A minimal end-to-end feature path spanning UI/API, application logic, and persistence.
|
||||
@@ -0,0 +1,56 @@
|
||||
## Document Transcription System
|
||||
|
||||
This project is a production application for transcribing and preserving historical family documents. It is intentionally designed for personal-scale use, with a simplicity-first architecture that is easy to operate and easy to extend.
|
||||
|
||||
## Start Here
|
||||
|
||||
Read [architecture.md](architecture.md) first.
|
||||
|
||||
The architecture page is the primary technical reference and defines:
|
||||
|
||||
- deployed topology and infrastructure limits
|
||||
- module boundaries and dependency flow
|
||||
- processing life cycle and data ownership
|
||||
- test strategy, risk controls, and extension path
|
||||
|
||||
## What The Application Does
|
||||
|
||||
At a high level, users upload images or PDFs as content sources for handwritten, typed, or typeset documents, run asynchronous transcription jobs, review optional revisions, and search across accepted text.
|
||||
|
||||
Core capabilities:
|
||||
|
||||
- document grouping with one or more content sources and metadata capture
|
||||
- asynchronous transcription with visible job status
|
||||
- immutable original transcription persisted with each job (plus provider/model/prompt metadata)
|
||||
- transcription prompt management with one Markdown file per prompt for human refinement over time
|
||||
- optional revisions for user-authored edits of original immutable transcription text
|
||||
- full-text search over accepted transcripts
|
||||
- export of transcript data
|
||||
|
||||
## Production Operating Model
|
||||
|
||||
The system runs with minimal operational overhead:
|
||||
|
||||
- PostgreSQL in a dedicated Docker container is considered extremely lightweight and simple for this system
|
||||
- MongoDB in a dedicated Docker container is also considered extremely lightweight and simple for document-centric persistence
|
||||
- a three-container deployment (app, PostgreSQL, MongoDB) is a simple and acceptable baseline
|
||||
- no required queue or search-engine containers in the baseline setup
|
||||
|
||||
This operating model keeps deployment and maintenance simple while preserving clean boundaries for future scale.
|
||||
|
||||
## Documentation Map
|
||||
|
||||
- Architecture and technical design: [architecture.md](architecture.md)
|
||||
- Runtime and deployment requirements: [requirements.md](requirements.md)
|
||||
- Error handling policy and operational guidance: [error_handling.md](error_handling.md)
|
||||
- Domain context and transcription policy: [intent.md](intent.md)
|
||||
- Transcription Methodology: [transcription_methodology.md](transcription_methodology.md)
|
||||
- Data model: [schema.md](schema.md)
|
||||
|
||||
|
||||
|
||||
## Glossary
|
||||
|
||||
- Document-oriented persistence: Storing data as flexible records instead of fixed relational rows.
|
||||
- Prompt artifact: A single Markdown file that defines one transcription prompt and is edited independently.
|
||||
- System of record: The authoritative persistent store for canonical data.
|
||||
@@ -0,0 +1,85 @@
|
||||
## Document Transcription System Requirements
|
||||
|
||||
This page captures a SysML v1.6-style requirements baseline for the production system described in [index.md](index.md). The model is represented as concise tables and traceability lists that preserve SysML-style IDs and relationship semantics.
|
||||
|
||||
## Scope
|
||||
|
||||
- System of interest: the single Python application service (NiceGUI + FastAPI) with PostgreSQL as the relational system of record and optional MongoDB for document-oriented persistence.
|
||||
- Operational context: local-first execution with Docker Compose and an intentionally lightweight production trajectory.
|
||||
- Primary concern: end-to-end transcription job lifecycle from upload through completion or failure.
|
||||
|
||||
## Requirements Model (Concise Text Form)
|
||||
|
||||
### Requirements
|
||||
|
||||
| ID | Category | Requirement | Risk | Verify Method |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| REQ-0 | System | Provide end-to-end document transcription with persistent, inspectable lifecycle state. | medium | demonstration |
|
||||
| REQ-1 | Functional | Allow users to upload one or more images or PDFs as sources from the web UI. | low | test |
|
||||
| REQ-2 | Functional | Run each upload through asynchronous processing that returns an original transcription or explicit failure. | high | test |
|
||||
| REQ-3 | Functional | Persist and expose job states: queued, processing, transcribed, failed. | high | inspection |
|
||||
| REQ-4 | Functional | Persist transcription output, processing history, and failure details. | medium | test |
|
||||
| REQ-5 | Interface | Expose API and UI views for status inspection and completed transcription reading. | medium | demonstration |
|
||||
| REQ-6 | Performance | Trigger background processing on upload to preserve UI responsiveness. | medium | analysis |
|
||||
| REQ-7 | Design Constraint | Keep lifespan-owned runtime resources: SQLAlchemy engine, async session factory, worker resources, provider clients. | medium | inspection |
|
||||
| REQ-8 | Design Constraint | Initialize configuration and logging once at startup through centralized mechanisms. | low | inspection |
|
||||
| REQ-9 | Design Constraint | Use Docker Compose baseline of app plus PostgreSQL; allow optional MongoDB container when enabled. | medium | demonstration |
|
||||
| REQ-10 | Design Constraint | Keep schema bootstrap explicit and opt-in; normal startup does not mutate production schema. | high | inspection |
|
||||
| REQ-11 | Design Constraint | Use service-backed persistence for core document and job data. | medium | inspection |
|
||||
| REQ-12 | Design Constraint | Store transcription prompts as individual Markdown artifacts for iterative refinement. | medium | inspection |
|
||||
| REQ-13 | Functional | Allow users to create one optional revision of transcription text derived from the original job transcription. | low | test |
|
||||
|
||||
### Requirement Relationships
|
||||
|
||||
- Contains: REQ-0 contains REQ-1 through REQ-13.
|
||||
- Derives: REQ-2 -> REQ-3, REQ-3 -> REQ-4.
|
||||
- Traces: REQ-5 -> REQ-3.
|
||||
- Refines: REQ-6 -> REQ-2.
|
||||
|
||||
### Architecture Elements
|
||||
|
||||
| Element | Type | Doc Reference |
|
||||
| --- | --- | --- |
|
||||
| UI | NiceGUI pages | src/transcription/ui/pages |
|
||||
| API | FastAPI routes | src/transcription/api/routes.py |
|
||||
| GRAPH | Async processing workflow | src/transcription/services, src/transcription/ai |
|
||||
| DBREL | PostgreSQL + SQLModel relational persistence | src/transcription/db |
|
||||
| DBDOC | MongoDB document persistence | src/transcription/db, src/transcription/services |
|
||||
| OPS | Docker Compose runtime | docker-compose.yml |
|
||||
| PROMPTS | Transcription prompt artifact library (Markdown files) | .github/prompts, docs |
|
||||
| TESTS | Pytest verification suite | tests |
|
||||
|
||||
### Satisfaction Mapping
|
||||
|
||||
- UI satisfies REQ-1, REQ-5, REQ-13.
|
||||
- API satisfies REQ-5.
|
||||
- GRAPH satisfies REQ-2, REQ-6.
|
||||
- DBREL satisfies REQ-3, REQ-10, REQ-13.
|
||||
- DBDOC satisfies REQ-4, REQ-11.
|
||||
- OPS satisfies REQ-9.
|
||||
- PROMPTS satisfies REQ-12.
|
||||
|
||||
### Verification Mapping
|
||||
|
||||
- TESTS verifies REQ-1, REQ-2, REQ-3, REQ-4, REQ-5, REQ-10, REQ-11, REQ-12, REQ-13.
|
||||
|
||||
## Requirement Notes
|
||||
|
||||
- Requirement IDs (`REQ-*`) are stable references for planning, implementation, and test traceability.
|
||||
- The model uses compact tables and traceability lists for renderer compatibility while preserving SysML-style requirement IDs and relationship semantics.
|
||||
- Requirement categories (functional, interface, performance, and design constraints) are preserved as explicit REQ entries and relationship labels to keep change impact visible.
|
||||
- PostgreSQL containerization and optional MongoDB containerization are both treated as extremely lightweight and simple operational choices in this architecture.
|
||||
|
||||
## Verification Intent
|
||||
|
||||
- Demonstration: validate end-to-end behavior via running system flows and operator-visible outcomes.
|
||||
- Inspection: verify architecture and startup/runtime policies in code and configuration.
|
||||
- Analysis: evaluate asynchronous execution behavior and design sufficiency.
|
||||
- Test: automate behavioral checks through pytest suites and service-level tests.
|
||||
|
||||
## Glossary
|
||||
|
||||
- Document-oriented persistence: A storage approach that uses flexible document structures for variable data shapes.
|
||||
- Prompt artifact: A single Markdown file that defines one transcription prompt and is revised independently.
|
||||
- SysML: Systems Modeling Language used to express structured requirements and traceability.
|
||||
- System of record: The authoritative persistent store for canonical business data.
|
||||
@@ -271,7 +271,7 @@ Change requirements:
|
||||
- [System overview](index.md)
|
||||
- [Architecture](architecture.md)
|
||||
- [Requirements](requirements.md)
|
||||
- [Intent](Intent.md)
|
||||
- [Intent](intent.md)
|
||||
|
||||
## Glossary
|
||||
|
||||
|
||||
+32
-31
@@ -1,56 +1,57 @@
|
||||
## Document Transcription System
|
||||
## Document Transcription System (V1)
|
||||
|
||||
This project is a production application for transcribing and preserving historical family documents. It is intentionally designed for personal-scale use, with a simplicity-first architecture that is easy to operate and easy to extend.
|
||||
This project is a personal-scale application for transcribing and preserving historical family documents.
|
||||
|
||||
## Start Here
|
||||
|
||||
Read [architecture.md](architecture.md) first.
|
||||
|
||||
The architecture page is the primary technical reference and defines:
|
||||
The architecture page is the primary technical reference for:
|
||||
|
||||
- deployed topology and infrastructure limits
|
||||
- runtime topology and infrastructure assumptions
|
||||
- module boundaries and dependency flow
|
||||
- processing life cycle and data ownership
|
||||
- test strategy, risk controls, and extension path
|
||||
- processing lifecycle and data ownership
|
||||
- test strategy and extension path
|
||||
|
||||
## What The Application Does
|
||||
|
||||
At a high level, users upload images or PDFs as content sources for handwritten, typed, or typeset documents, run asynchronous transcription jobs, review optional revisions, and search across accepted text.
|
||||
At a high level, users upload images/PDFs, jobs are processed asynchronously, and users review original transcriptions plus optional revisions.
|
||||
|
||||
Core capabilities:
|
||||
Core V1 capabilities:
|
||||
|
||||
- document grouping with one or more content sources and metadata capture
|
||||
- asynchronous transcription with visible job status
|
||||
- immutable original transcription persisted with each job (plus provider/model/prompt metadata)
|
||||
- transcription prompt management with one Markdown file per prompt for human refinement over time
|
||||
- optional revisions for user-authored edits of original immutable transcription text
|
||||
- full-text search over accepted transcripts
|
||||
- export of transcript data
|
||||
- upload supported source files (`.jpg`, `.jpeg`, `.png`, `.tif`, `.tiff`, `.pdf`)
|
||||
- asynchronous job processing with visible status (`queued`, `processing`, `transcribed`, `failed`)
|
||||
- immutable original transcription stored on `Job.text`
|
||||
- optional single user-authored revision per source (`0..1`)
|
||||
- prompt artifacts stored as Markdown files in `prompts/`
|
||||
|
||||
## Production Operating Model
|
||||
## Current Operating Model (V1 Baseline)
|
||||
|
||||
The system runs with minimal operational overhead:
|
||||
- application service: FastAPI + NiceGUI
|
||||
- persistence baseline: SQLModel with SQLite
|
||||
- worker: in-process async background loop
|
||||
- deployment baseline: lightweight Docker Compose app runtime
|
||||
|
||||
- PostgreSQL in a dedicated Docker container is considered extremely lightweight and simple for this system
|
||||
- MongoDB in a dedicated Docker container is also considered extremely lightweight and simple for document-centric persistence
|
||||
- a three-container deployment (app, PostgreSQL, MongoDB) is a simple and acceptable baseline
|
||||
- no required queue or search-engine containers in the baseline setup
|
||||
|
||||
This operating model keeps deployment and maintenance simple while preserving clean boundaries for future scale.
|
||||
> Planned persistence evolution (PostgreSQL and optional MongoDB) belongs to V2 planning and is tracked separately.
|
||||
|
||||
## Documentation Map
|
||||
|
||||
- Architecture and technical design: [architecture.md](architecture.md)
|
||||
- Runtime and deployment requirements: [requirements.md](requirements.md)
|
||||
- V1 runtime and requirement baseline: [requirements.md](requirements.md)
|
||||
- Data model and constraints: [schema.md](schema.md)
|
||||
- Error handling policy and operational guidance: [error_handling.md](error_handling.md)
|
||||
- V1 requirement evidence matrix: [traceability_v1.md](traceability_v1.md)
|
||||
- Operations runbook: [runbook.md](runbook.md)
|
||||
- V1 migration and rollback guidance: [migration_v1.md](migration_v1.md)
|
||||
- V1 release checklist: [release_checklist_v1.md](release_checklist_v1.md)
|
||||
- Domain context and transcription policy: [intent.md](intent.md)
|
||||
- Transcription Methodology: [transcription_methodology.md](transcription_methodology.md)
|
||||
- Data model: [schema.md](schema.md)
|
||||
|
||||
|
||||
- Transcription methodology: [transcription_methodology.md](transcription_methodology.md)
|
||||
- V1 execution plan: [ver1/ver1.md](ver1/ver1.md)
|
||||
- V2 roadmap: [ver2/ver2.md](ver2/ver2.md)
|
||||
|
||||
## Glossary
|
||||
|
||||
- Document-oriented persistence: Storing data as flexible records instead of fixed relational rows.
|
||||
- Prompt artifact: A single Markdown file that defines one transcription prompt and is edited independently.
|
||||
- System of record: The authoritative persistent store for canonical data.
|
||||
- Prompt artifact: a Markdown file containing one transcription prompt.
|
||||
- Original transcription: immutable provider output stored on `Job.text`.
|
||||
- Revision: optional user-authored text linked to a `Source`.
|
||||
- System of record: the authoritative persistent store for canonical application data.
|
||||
|
||||
+37
-36
@@ -1,12 +1,14 @@
|
||||
## Document Transcription System Requirements
|
||||
## Document Transcription System Requirements (V1 Baseline)
|
||||
|
||||
This page captures a SysML v1.6-style requirements baseline for the production system described in [index.md](index.md). The model is represented as concise tables and traceability lists that preserve SysML-style IDs and relationship semantics.
|
||||
This page captures the **Version 1 baseline requirements** for the currently implemented system. It is the source of truth for V1 acceptance and test traceability.
|
||||
|
||||
Forward-looking architecture changes (for example PostgreSQL/Mongo adoption) are intentionally out of this document and should be tracked in a V2 planning/backlog artifact.
|
||||
|
||||
## Scope
|
||||
|
||||
- System of interest: the single Python application service (NiceGUI + FastAPI) with PostgreSQL as the relational system of record and optional MongoDB for document-oriented persistence.
|
||||
- Operational context: local-first execution with Docker Compose and an intentionally lightweight production trajectory.
|
||||
- Primary concern: end-to-end transcription job lifecycle from upload through completion or failure.
|
||||
- System of interest: a single Python application service (NiceGUI + FastAPI) with SQLModel persistence.
|
||||
- Runtime/persistence baseline: local-first execution using SQLite (default `sqlite:///./transcription.db`), with Docker Compose support.
|
||||
- Primary concern: end-to-end transcription lifecycle from upload through terminal state plus optional single revision editing.
|
||||
|
||||
## Requirements Model (Concise Text Form)
|
||||
|
||||
@@ -15,19 +17,19 @@ This page captures a SysML v1.6-style requirements baseline for the production s
|
||||
| ID | Category | Requirement | Risk | Verify Method |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| REQ-0 | System | Provide end-to-end document transcription with persistent, inspectable lifecycle state. | medium | demonstration |
|
||||
| REQ-1 | Functional | Allow users to upload one or more images or PDFs as sources from the web UI. | low | test |
|
||||
| REQ-2 | Functional | Run each upload through asynchronous processing that returns an original transcription or explicit failure. | high | test |
|
||||
| REQ-3 | Functional | Persist and expose job states: queued, processing, transcribed, failed. | high | inspection |
|
||||
| REQ-4 | Functional | Persist transcription output, processing history, and failure details. | medium | test |
|
||||
| REQ-5 | Interface | Expose API and UI views for status inspection and completed transcription reading. | medium | demonstration |
|
||||
| REQ-1 | Functional | Allow users to upload supported image/PDF files as sources from the web UI. | low | test |
|
||||
| REQ-2 | Functional | Process uploads asynchronously and return either original transcription output or explicit failure. | high | test |
|
||||
| REQ-3 | Functional | Persist and expose job states: `queued`, `processing`, `transcribed`, `failed`. | high | inspection |
|
||||
| REQ-4 | Functional | Persist original provider output (`Job.text`) and failure detail (`Job.error_detail`) for each job. | medium | test |
|
||||
| REQ-5 | Interface | Expose API/UI views for status inspection and transcription reading. | medium | demonstration |
|
||||
| REQ-6 | Performance | Trigger background processing on upload to preserve UI responsiveness. | medium | analysis |
|
||||
| REQ-7 | Design Constraint | Keep lifespan-owned runtime resources: SQLAlchemy engine, async session factory, worker resources, provider clients. | medium | inspection |
|
||||
| REQ-7 | Design Constraint | Keep lifespan-owned runtime resources (engine/session factory/worker resources) initialized and disposed at application boundaries. | medium | inspection |
|
||||
| REQ-8 | Design Constraint | Initialize configuration and logging once at startup through centralized mechanisms. | low | inspection |
|
||||
| REQ-9 | Design Constraint | Use Docker Compose baseline of app plus PostgreSQL; allow optional MongoDB container when enabled. | medium | demonstration |
|
||||
| REQ-10 | Design Constraint | Keep schema bootstrap explicit and opt-in; normal startup does not mutate production schema. | high | inspection |
|
||||
| REQ-11 | Design Constraint | Use service-backed persistence for core document and job data. | medium | inspection |
|
||||
| REQ-9 | Design Constraint | Support containerized app runtime via Docker Compose using the same V1 persistence model. | medium | demonstration |
|
||||
| REQ-10 | Design Constraint | Keep schema bootstrap explicit and opt-in for production safety. | high | inspection |
|
||||
| REQ-11 | Design Constraint | Route persistence changes through service/workflow orchestration boundaries. | medium | inspection |
|
||||
| REQ-12 | Design Constraint | Store transcription prompts as individual Markdown artifacts for iterative refinement. | medium | inspection |
|
||||
| REQ-13 | Functional | Allow users to create one optional revision of transcription text derived from the original job transcription. | low | test |
|
||||
| REQ-13 | Functional | Allow users to create/update one optional revision derived from the original job transcription and view/delete it from the job detail flow. | low | test |
|
||||
|
||||
### Requirement Relationships
|
||||
|
||||
@@ -40,22 +42,22 @@ This page captures a SysML v1.6-style requirements baseline for the production s
|
||||
|
||||
| Element | Type | Doc Reference |
|
||||
| --- | --- | --- |
|
||||
| UI | NiceGUI pages | src/transcription/ui/pages |
|
||||
| API | FastAPI routes | src/transcription/api/routes.py |
|
||||
| GRAPH | Async processing workflow | src/transcription/services, src/transcription/ai |
|
||||
| DBREL | PostgreSQL + SQLModel relational persistence | src/transcription/db |
|
||||
| DBDOC | MongoDB document persistence | src/transcription/db, src/transcription/services |
|
||||
| OPS | Docker Compose runtime | docker-compose.yml |
|
||||
| PROMPTS | Transcription prompt artifact library (Markdown files) | .github/prompts, docs |
|
||||
| TESTS | Pytest verification suite | tests |
|
||||
| UI | NiceGUI pages/components | `src/transcription/ui/pages`, `src/transcription/ui/components` |
|
||||
| API | FastAPI routes and handlers | `src/transcription/api`, `src/transcription/app.py` |
|
||||
| WORKER | Async queued-job processing workflow | `src/transcription/worker.py`, `src/transcription/services/workflows.py` |
|
||||
| DBREL | SQLModel relational persistence (SQLite in V1 baseline) | `src/transcription/models.py`, `src/transcription/db` |
|
||||
| SERVICES | Service-layer persistence orchestration | `src/transcription/services` |
|
||||
| OPS | Containerized runtime baseline | `docker-compose.yml`, `Dockerfile` |
|
||||
| PROMPTS | Transcription prompt artifacts | `prompts/` |
|
||||
| TESTS | Pytest verification suite | `tests/` |
|
||||
|
||||
### Satisfaction Mapping
|
||||
|
||||
- UI satisfies REQ-1, REQ-5, REQ-13.
|
||||
- API satisfies REQ-5.
|
||||
- GRAPH satisfies REQ-2, REQ-6.
|
||||
- DBREL satisfies REQ-3, REQ-10, REQ-13.
|
||||
- DBDOC satisfies REQ-4, REQ-11.
|
||||
- WORKER satisfies REQ-2, REQ-6.
|
||||
- DBREL satisfies REQ-3, REQ-4, REQ-10, REQ-13.
|
||||
- SERVICES satisfies REQ-4, REQ-11.
|
||||
- OPS satisfies REQ-9.
|
||||
- PROMPTS satisfies REQ-12.
|
||||
|
||||
@@ -65,21 +67,20 @@ This page captures a SysML v1.6-style requirements baseline for the production s
|
||||
|
||||
## Requirement Notes
|
||||
|
||||
- Requirement IDs (`REQ-*`) are stable references for planning, implementation, and test traceability.
|
||||
- The model uses compact tables and traceability lists for renderer compatibility while preserving SysML-style requirement IDs and relationship semantics.
|
||||
- Requirement categories (functional, interface, performance, and design constraints) are preserved as explicit REQ entries and relationship labels to keep change impact visible.
|
||||
- PostgreSQL containerization and optional MongoDB containerization are both treated as extremely lightweight and simple operational choices in this architecture.
|
||||
- Requirement IDs (`REQ-*`) are stable references for planning, implementation, and traceability.
|
||||
- This document is intentionally **implementation-aligned** for V1 completion and release sign-off.
|
||||
- Planned storage evolution (PostgreSQL and optional MongoDB) is a **V2 concern** and should be tracked outside this V1 baseline.
|
||||
|
||||
## Verification Intent
|
||||
|
||||
- Demonstration: validate end-to-end behavior via running system flows and operator-visible outcomes.
|
||||
- Demonstration: validate end-to-end behavior through operator-visible flows.
|
||||
- Inspection: verify architecture and startup/runtime policies in code and configuration.
|
||||
- Analysis: evaluate asynchronous execution behavior and design sufficiency.
|
||||
- Test: automate behavioral checks through pytest suites and service-level tests.
|
||||
- Test: automate behavioral checks through pytest suites and service/UI integration tests.
|
||||
|
||||
## Glossary
|
||||
|
||||
- Document-oriented persistence: A storage approach that uses flexible document structures for variable data shapes.
|
||||
- Prompt artifact: A single Markdown file that defines one transcription prompt and is revised independently.
|
||||
- SysML: Systems Modeling Language used to express structured requirements and traceability.
|
||||
- System of record: The authoritative persistent store for canonical business data.
|
||||
- Original transcription: immutable provider output stored on `Job.text`.
|
||||
- Revision: optional user-authored editable text tied to a `Source` (`0..1` in V1).
|
||||
- Prompt artifact: a Markdown file containing instructions used for transcription.
|
||||
- System of record: the authoritative relational store for canonical V1 data.
|
||||
|
||||
+129
@@ -0,0 +1,129 @@
|
||||
# V1 Operations Runbook
|
||||
|
||||
This runbook provides day-2 operational procedures for the V1 baseline.
|
||||
|
||||
## Scope
|
||||
|
||||
Applies to:
|
||||
|
||||
- local/hosted V1 runtime
|
||||
- SQLite-backed persistence
|
||||
- in-process worker lifecycle
|
||||
- OpenRouter provider integration
|
||||
|
||||
## Preconditions
|
||||
|
||||
- `.env` contains `OPENROUTER_API_KEY`
|
||||
- app starts successfully
|
||||
- `uploads/` and `prompts/` are writable
|
||||
- health endpoint responds at `/healthz`
|
||||
|
||||
## Standard Startup Procedure
|
||||
|
||||
1. Start the app using the project-standard command.
|
||||
2. Open `/healthz` and verify `{"status":"ok"}`.
|
||||
3. Open `/ui/upload` and submit a small valid file.
|
||||
4. Confirm job transitions from `queued` -> `processing` -> `transcribed` (or `failed` with detail).
|
||||
|
||||
## Standard Shutdown Procedure
|
||||
|
||||
1. Stop the application process.
|
||||
2. Ensure no active process still holds the SQLite file.
|
||||
3. If maintenance is planned, copy the DB file before edits:
|
||||
- `transcription.db` (or configured `DATABASE_URL` file path)
|
||||
|
||||
## Incident: Jobs Stuck In `processing`
|
||||
|
||||
### Symptoms
|
||||
|
||||
- Jobs remain `processing` for longer than provider timeout
|
||||
- New uploads queue but do not complete
|
||||
- provider usage increases but no terminal job state is visible
|
||||
|
||||
### Checks
|
||||
|
||||
1. Confirm app process is still running.
|
||||
2. Confirm worker loop is active (startup logs include worker lifespan start).
|
||||
3. Inspect recent app logs for:
|
||||
- `worker.process_job`
|
||||
- `error_id`
|
||||
- `category`
|
||||
- `job_id` / `document_id` / `source_id`
|
||||
4. Verify provider credentials and provider status.
|
||||
|
||||
### Recovery
|
||||
|
||||
1. Restart the app to trigger stale-processing recovery.
|
||||
2. On startup, app re-queues stale processing jobs based on timeout policy.
|
||||
3. Re-check jobs page and confirm terminal state progression.
|
||||
4. If persistent, capture logs + error IDs and move to deep investigation.
|
||||
|
||||
## Incident: Provider Authentication Failures
|
||||
|
||||
### Symptoms
|
||||
|
||||
- failures categorized as provider/auth
|
||||
- jobs fail quickly with authentication guidance
|
||||
|
||||
### Recovery
|
||||
|
||||
1. Validate `OPENROUTER_API_KEY` value.
|
||||
2. Restart app after updating env.
|
||||
3. Re-run a small transcription to confirm recovery.
|
||||
|
||||
## Incident: Upload Failures
|
||||
|
||||
### Symptoms
|
||||
|
||||
- UI reports upload errors
|
||||
- unsupported extension or empty payload
|
||||
|
||||
### Recovery
|
||||
|
||||
1. Validate file extension (`.jpg`, `.jpeg`, `.png`, `.tif`, `.tiff`, `.pdf`).
|
||||
2. Validate file is not empty.
|
||||
3. Validate upload directory permissions.
|
||||
4. Retry upload.
|
||||
|
||||
## Incident: Database File/Permission Issues
|
||||
|
||||
### Symptoms
|
||||
|
||||
- persistence errors during upload/job update
|
||||
- startup failures around schema/runtime
|
||||
|
||||
### Recovery
|
||||
|
||||
1. Confirm the configured DB file path exists and is writable.
|
||||
2. Confirm parent directory permissions.
|
||||
3. Restore from last known backup copy if corruption is suspected.
|
||||
4. Restart app and run smoke test.
|
||||
|
||||
## Logging Requirements (Operational)
|
||||
|
||||
Operational triage should always capture:
|
||||
|
||||
- `error_id`
|
||||
- category
|
||||
- operation name
|
||||
- `job_id`, `document_id`, `source_id` when applicable
|
||||
- UTC timestamp
|
||||
|
||||
## Escalation Packet (When opening an issue)
|
||||
|
||||
Include:
|
||||
|
||||
- exact timestamp window
|
||||
- one failing `job_id`
|
||||
- relevant `error_id` values
|
||||
- latest 100 lines of app logs
|
||||
- environment summary (`DATABASE_URL` type, app version/commit)
|
||||
|
||||
## Post-Incident Validation
|
||||
|
||||
After mitigation, verify:
|
||||
|
||||
1. Upload works.
|
||||
2. One job reaches `transcribed`.
|
||||
3. One induced failure reaches `failed` with error detail.
|
||||
4. Jobs page and detail page render correctly.
|
||||
+49
-36
@@ -1,21 +1,23 @@
|
||||
## Database Schema (V1 Baseline)
|
||||
|
||||
## Database schema
|
||||
This document describes the structure of the database underlying the personal historical-document transcription system.
|
||||
This document describes the current relational schema for the transcription system.
|
||||
|
||||
All primary and foreign keys in the domain models are UUID-based in V1.
|
||||
|
||||
---
|
||||
|
||||
## Schema diagram
|
||||
## Schema Diagram
|
||||
|
||||
```mermaid
|
||||
```mermaid
|
||||
erDiagram
|
||||
document {
|
||||
INTEGER id PK
|
||||
DOCUMENT {
|
||||
UUID id PK
|
||||
TEXT name
|
||||
}
|
||||
|
||||
job {
|
||||
INTEGER id PK
|
||||
INTEGER document_id FK
|
||||
JOB {
|
||||
UUID id PK
|
||||
UUID document_id FK
|
||||
TEXT status
|
||||
INTEGER retry_count
|
||||
DATETIME date_created
|
||||
@@ -27,48 +29,59 @@ erDiagram
|
||||
TEXT error_detail
|
||||
}
|
||||
|
||||
source {
|
||||
INTEGER id PK
|
||||
INTEGER document_id FK
|
||||
INTEGER job_id FK
|
||||
SOURCE {
|
||||
UUID id PK
|
||||
UUID document_id FK
|
||||
UUID job_id FK
|
||||
TEXT upload_name
|
||||
TEXT filename
|
||||
TEXT file_path
|
||||
DATETIME date_uploaded
|
||||
}
|
||||
|
||||
revision {
|
||||
INTEGER id PK
|
||||
INTEGER source_id FK
|
||||
REVISION {
|
||||
UUID id PK
|
||||
UUID source_id FK UNIQUE
|
||||
INTEGER revision
|
||||
TEXT text
|
||||
DATETIME date_created
|
||||
}
|
||||
|
||||
document ||--o{ source : "has 0 or more"
|
||||
document ||--o{ job : "has 0 or more"
|
||||
job ||--o{ source : "processes 0 or more"
|
||||
source ||--o{ revision : "has 0 or 1"
|
||||
DOCUMENT ||--o{ SOURCE : has_many
|
||||
DOCUMENT ||--o{ JOB : has_many
|
||||
JOB ||--o{ SOURCE : referenced_by
|
||||
SOURCE ||--o| REVISION : has_optional_one
|
||||
```
|
||||
|
||||
---
|
||||
---
|
||||
|
||||
## Table Relationships & Constraints
|
||||
* A document can consist of 0 or more content sources. A document can have 0 or more jobs.
|
||||
* A source can belong to only one job (which contains the original transcription). A source can only belong to one document. A source may have one optional transcription revision.
|
||||
* A job can process one or more sources. A job can belong to only one document.
|
||||
* A revision can belong to only one source. A source may have one optional revision.
|
||||
* 1:1 optionality is enforced by uniqueness on `revision.source_id` (no revision history chain).
|
||||
* `Job.text` stores the original immutable provider transcription.
|
||||
* Revision rows are optional user-authored edits and are derived from the original transcription. Unlike jobs, revision rows can be updated.
|
||||
## Table Relationships and Constraints
|
||||
|
||||
- A `Document` can have zero or more `Source` records.
|
||||
- A `Document` can have zero or more `Job` records.
|
||||
- A `Source` belongs to exactly one `Document` and one `Job`.
|
||||
- A `Source` may have one optional `Revision`.
|
||||
- Optional `0..1` revision cardinality is enforced by uniqueness on `revision.source_id`.
|
||||
|
||||
### Invariants
|
||||
|
||||
- `Job.text` stores immutable original provider transcription output.
|
||||
- `Revision` rows are optional user-authored edits derived from original transcription.
|
||||
- Revisions do not overwrite original `Job.text`.
|
||||
- Job status lifecycle values are: `queued`, `processing`, `transcribed`, `failed`.
|
||||
|
||||
### Timestamp Fields
|
||||
|
||||
- `Job.date_created`
|
||||
- `Job.date_updated`
|
||||
- `Source.date_uploaded`
|
||||
- `Revision.date_created`
|
||||
|
||||
---
|
||||
|
||||
## Glossary
|
||||
* **Document** - Documents consist of one or more content sources and their related transcriptions.
|
||||
* **Source** - A content source that is transcribed to text. It can either be an image (.jpg, .tiff, .png) or a PDF (.pdf).
|
||||
* **Image** - The scanned image of one page of a document.
|
||||
* **PDF** - A PDF containing the image of one or more pages of a document.
|
||||
* **Job** - A processing job ingests one or more sources, sends them to an AI model along with a prompt for transcription, then stores the results. The results are immutable, *including the original transcription*. The user can create a revision of the original transcription, but the user cannot modify the original.
|
||||
* **Transcription** - The text contained in a content source. A job creates the original immutable transcription. A user can optionally create a revised transcription, or "revision".
|
||||
* **Revision** - A revision is a user-created modification of an existing transcription. It is optional. Some original transcriptions will have no revisions.
|
||||
|
||||
- **Document**: logical grouping for one or more transcribed sources.
|
||||
- **Source**: uploaded file content (image/PDF) linked to a job.
|
||||
- **Job**: processing record that stores lifecycle status and original output.
|
||||
- **Revision**: optional single user-authored edited text linked to a source.
|
||||
|
||||
@@ -0,0 +1,92 @@
|
||||
# V1 Data Migration and Recovery Guidance
|
||||
|
||||
This document defines migration/backfill and rollback guidance for the V1 SQLite baseline.
|
||||
|
||||
## Purpose
|
||||
|
||||
- provide safe procedures for local schema evolution and recovery
|
||||
- reduce data-loss risk during version upgrades
|
||||
- establish repeatable pre-change and post-change checks
|
||||
|
||||
## Current Baseline
|
||||
|
||||
- canonical relational store: SQLite
|
||||
- default DB path: `./transcription.db`
|
||||
- schema bootstrap may apply compatibility updates for dev/test scenarios
|
||||
|
||||
## Pre-Change Checklist
|
||||
|
||||
Before changing runtime version or schema behavior:
|
||||
|
||||
1. Stop the app process.
|
||||
2. Create a timestamped DB backup copy.
|
||||
3. Capture current app commit/version.
|
||||
4. Export a quick status inventory:
|
||||
- job counts by status
|
||||
- total documents/sources/revisions
|
||||
5. Ensure sufficient disk space.
|
||||
|
||||
## Backup Procedure (SQLite)
|
||||
|
||||
Minimum procedure:
|
||||
|
||||
1. Stop app.
|
||||
2. Copy DB file to a safe location with timestamp.
|
||||
3. Store backup path in release notes or change log.
|
||||
|
||||
## Upgrade Procedure (V1)
|
||||
|
||||
1. Perform pre-change checklist.
|
||||
2. Deploy updated app version.
|
||||
3. Start app and observe startup logs.
|
||||
4. Verify schema bootstrap completes (if enabled).
|
||||
5. Run smoke flow:
|
||||
- upload valid file
|
||||
- observe terminal status
|
||||
- open job detail
|
||||
|
||||
## Backfill Guidance
|
||||
|
||||
V1 backfill is limited and conservative:
|
||||
|
||||
- for records missing newly introduced non-null defaults, use explicit one-time SQL updates only after backup
|
||||
- avoid destructive rewrites of `Job.text` or `Revision.text`
|
||||
- never backfill by overwriting original immutable transcription output
|
||||
|
||||
## Rollback Procedure
|
||||
|
||||
If upgrade fails or causes data inconsistency:
|
||||
|
||||
1. Stop app.
|
||||
2. Restore prior DB backup file.
|
||||
3. Revert app version to last known-good commit.
|
||||
4. Restart app.
|
||||
5. Run smoke flow and confirm stability.
|
||||
|
||||
## Recovery Scenarios
|
||||
|
||||
### Stale processing jobs after crash/restart
|
||||
|
||||
- restart app and allow stale-job recovery to re-queue timed-out `processing` jobs
|
||||
- monitor for terminal progression
|
||||
|
||||
### Schema mismatch symptoms
|
||||
|
||||
- errors during startup or writes indicating missing columns/indexes
|
||||
- rollback to last good DB + app version
|
||||
- reattempt with documented upgrade path
|
||||
|
||||
## Validation Evidence
|
||||
|
||||
For each upgrade rehearsal, capture:
|
||||
|
||||
- backup filename/path
|
||||
- pre and post job status counts
|
||||
- smoke test result
|
||||
- rollback rehearsal result (recommended)
|
||||
|
||||
## Operational Constraints
|
||||
|
||||
- treat DB backups as required before non-trivial upgrades
|
||||
- do not perform in-place DB edits while app is running
|
||||
- do not skip post-upgrade smoke validation
|
||||
@@ -0,0 +1,45 @@
|
||||
# V1 Release Readiness Checklist
|
||||
|
||||
Use this checklist before declaring V1 operationally complete.
|
||||
|
||||
## A) Functional Readiness
|
||||
|
||||
- [ ] Upload flow works for supported file types.
|
||||
- [ ] Worker transitions jobs through `queued -> processing -> transcribed|failed`.
|
||||
- [ ] Job detail displays immutable original transcription from `Job.text`.
|
||||
- [ ] Revision workflow supports create/update/view/delete for optional single revision.
|
||||
|
||||
## B) Reliability and Error Handling
|
||||
|
||||
- [ ] Error categories surface with actionable messages in UI/API pathways.
|
||||
- [ ] Failed jobs persist `error_detail` and terminal state.
|
||||
- [ ] Stale processing recovery verified on restart.
|
||||
- [ ] Retry/timeout behavior validated against configured limits.
|
||||
|
||||
## C) Operational Readiness
|
||||
|
||||
- [ ] `docs/runbook.md` reviewed and current.
|
||||
- [ ] `docs/migration_v1.md` reviewed and current.
|
||||
- [ ] Backup and rollback procedures tested at least once.
|
||||
- [ ] Incident escalation packet template is known to operators.
|
||||
|
||||
## D) Quality Gates
|
||||
|
||||
- [ ] Lint/type checks pass.
|
||||
- [ ] `pytest -m "not external" -q` passes.
|
||||
- [ ] Targeted external/provider checks executed (if credentials available).
|
||||
- [ ] Release evidence recorded in `docs/release_evidence_v1.md`.
|
||||
|
||||
## E) Traceability and Documentation
|
||||
|
||||
- [ ] `docs/requirements.md` aligns with implemented V1 behavior.
|
||||
- [ ] `docs/architecture.md`, `docs/schema.md`, and `docs/error_handling.md` are consistent.
|
||||
- [ ] `docs/traceability_v1.md` is updated with current implementation and test evidence.
|
||||
- [ ] `docs/ver1/ver1.md` phase status updated with evidence references.
|
||||
- [ ] REQ traceability evidence links recorded (tests/runbook/checks).
|
||||
|
||||
## Release Sign-Off
|
||||
|
||||
- [ ] Technical sign-off complete.
|
||||
- [ ] Operational sign-off complete.
|
||||
- [ ] V1 completion date recorded.
|
||||
@@ -0,0 +1,39 @@
|
||||
# V1 Release Evidence Log
|
||||
|
||||
## Step 5 Quality Gates (2026-07-29)
|
||||
|
||||
### Lint
|
||||
|
||||
- Command: `python -m ruff check .`
|
||||
- Result: ✅ pass
|
||||
- Notes: initial findings were auto-fixed (`ruff --fix`) plus small manual line-wrap/annotation adjustments.
|
||||
|
||||
### Tests (primary gate)
|
||||
|
||||
- Command: `python -m pytest -m "not external" -q`
|
||||
- Result: ✅ pass (`[100%]`)
|
||||
|
||||
### Tests (external smoke)
|
||||
|
||||
- Command: `python -m pytest -m external -q`
|
||||
- Result: ✅ pass (`[100%]`)
|
||||
|
||||
### Type Check
|
||||
|
||||
- Command: `python -m ty check src tests`
|
||||
- Result: ⚠️ not passing
|
||||
- Summary: existing SQLModel/SQLAlchemy typing incompatibilities and test double typing mismatches remain.
|
||||
|
||||
Key current blocker families:
|
||||
|
||||
1. SQLModel relationship/query attribute typing (`selectinload`, `order_by`, `.any()`)
|
||||
2. SQLAlchemy join clause typing in `services/transcription.py`
|
||||
3. Test fake client type mismatch for `OpenRouterTranscriptionProvider(client=...)`
|
||||
4. `Settings(**defaults)` typed-dict strictness in `tests/test_config.py`
|
||||
|
||||
## Current Gate Status
|
||||
|
||||
- Lint: pass
|
||||
- Non-external tests: pass
|
||||
- External smoke tests: pass
|
||||
- Type check: **blocked** (requires dedicated typing cleanup pass)
|
||||
@@ -0,0 +1,40 @@
|
||||
# V1 Traceability Matrix
|
||||
|
||||
This matrix provides implementation and validation evidence for V1 requirements (`REQ-0` through `REQ-13`).
|
||||
|
||||
Status values:
|
||||
|
||||
- `done`: implemented and evidence recorded
|
||||
- `in progress`: partially implemented or evidence incomplete
|
||||
- `not started`: no implementation/evidence yet
|
||||
|
||||
## Requirement Evidence Table
|
||||
|
||||
| Requirement | Status | Implementation Evidence | Validation Evidence |
|
||||
| --- | --- | --- | --- |
|
||||
| REQ-0 | done | End-to-end upload + worker pipeline in `src/transcription/services/store.py`, `src/transcription/worker.py`, `src/transcription/services/workflows.py` | `tests/integration/test_pipeline_flow.py` |
|
||||
| REQ-1 | done | Upload UI/page flow in `src/transcription/ui/pages/upload_page.py`, `src/transcription/ui/components/upload.py` | `tests/ui/test_upload_page.py`, `tests/integration/test_pipeline_flow.py` |
|
||||
| REQ-2 | done | Async worker execution and provider call orchestration in `src/transcription/worker.py`, `src/transcription/services/workflows.py` | `tests/integration/test_pipeline_flow.py`, `tests/services/test_workflows_reliability.py` |
|
||||
| REQ-3 | done | Job lifecycle state model + transitions in `src/transcription/models.py`, `src/transcription/services/jobs.py`, `src/transcription/services/workflows.py` | `tests/services/test_job_service.py`, `tests/ui/test_jobs_page.py` |
|
||||
| REQ-4 | done | Persistence of original output and failure detail in `src/transcription/services/transcription.py`, `src/transcription/services/workflows.py` | `tests/integration/test_pipeline_flow.py`, `tests/services/test_workflows_reliability.py` |
|
||||
| REQ-5 | done | Status/result inspection via UI pages and API health route in `src/transcription/ui/pages/jobs_page.py`, `src/transcription/api/health.py` | `tests/ui/test_jobs_page.py`, `tests/ui/test_pages_registration.py`, `tests/api/test_health.py` |
|
||||
| REQ-6 | done | Background processing trigger/worker notifier and non-blocking workflow in `src/transcription/ui/components/upload.py`, `src/transcription/worker.py` | `tests/test_app.py`, `tests/services/test_workflows_reliability.py` |
|
||||
| REQ-7 | done | Lifespan-owned runtime resources in `src/transcription/app.py`, `src/transcription/db/runtime.py` | `tests/test_app.py`, `tests/test_db.py` |
|
||||
| REQ-8 | done | Centralized settings/logging initialization in `src/transcription/config.py`, `src/transcription/app.py` | `tests/test_config.py`, `tests/test_app.py` |
|
||||
| REQ-9 | done | Containerized runtime baseline in `docker-compose.yml`, `Dockerfile` | `docs/release_checklist_v1.md` (Ops checklist), manual demonstration step |
|
||||
| REQ-10 | done | Explicit schema bootstrap policy + runtime controls in `src/transcription/config.py`, `src/transcription/app.py`, `src/transcription/db/operations.py` | `tests/test_db.py`, `tests/test_config.py` |
|
||||
| REQ-11 | done | Service/workflow persistence boundaries in `src/transcription/services/*.py`, `src/transcription/services/workflows.py` | `tests/services/test_job_service.py`, `tests/services/test_transcription_service.py` |
|
||||
| REQ-12 | done | Prompt artifacts in `prompts/` and loading/validation in `src/transcription/services/transcription.py` | `tests/test_prompts.py` |
|
||||
| REQ-13 | done | Optional single revision create/update/view/delete in `src/transcription/services/transcription.py`, `src/transcription/ui/pages/jobs_page.py` | `tests/services/test_transcription_service.py`, `tests/ui/test_jobs_page.py` |
|
||||
|
||||
## Operational Evidence (Step 3 Artifacts)
|
||||
|
||||
- Runbook: `docs/runbook.md`
|
||||
- Migration/backfill/rollback guidance: `docs/migration_v1.md`
|
||||
- Release readiness checklist: `docs/release_checklist_v1.md`
|
||||
|
||||
## Verification Cadence
|
||||
|
||||
- Per change: maintain `tests/test_traceability.py` mappings for touched requirements.
|
||||
- Per milestone: update this table status and evidence links.
|
||||
- Pre-release: confirm all rows are `done` and non-external suite is green.
|
||||
@@ -150,6 +150,9 @@ V1 is complete when all of the following are true:
|
||||
|
||||
### Deliverables
|
||||
- V1 release checklist and acceptance evidence.
|
||||
- `docs/runbook.md` for incident response and operator workflows.
|
||||
- `docs/migration_v1.md` for V1 migration/backfill/rollback guidance.
|
||||
- `docs/release_checklist_v1.md` for release sign-off.
|
||||
|
||||
### Exit Criteria
|
||||
- Stakeholder sign-off and launch readiness achieved.
|
||||
|
||||
@@ -0,0 +1,125 @@
|
||||
# Version 2 Plan
|
||||
|
||||
## Purpose
|
||||
|
||||
Version 2 extends the V1 baseline by introducing a production-oriented persistence architecture while preserving current user workflows.
|
||||
|
||||
Primary target changes:
|
||||
|
||||
- Migrate relational persistence from SQLite to PostgreSQL
|
||||
- Introduce optional MongoDB for document-oriented adjunct data (non-canonical)
|
||||
|
||||
V1 behavior remains the functional baseline unless explicitly superseded by approved V2 requirements.
|
||||
|
||||
---
|
||||
|
||||
## V2 Goals
|
||||
|
||||
1. **Relational migration complete**
|
||||
- PostgreSQL becomes the default system of record for `Document`, `Source`, `Job`, and `Revision`.
|
||||
2. **Operational maturity**
|
||||
- Repeatable migrations, rollback paths, and environment-specific deployment procedures are documented and tested.
|
||||
3. **Optional document store integration**
|
||||
- MongoDB is introduced only for clearly scoped use cases that do not replace canonical relational ownership.
|
||||
4. **No regression of V1 workflows**
|
||||
- Upload, queue/worker processing, status inspection, original transcription, and optional single revision remain stable.
|
||||
|
||||
---
|
||||
|
||||
## Non-Goals (V2)
|
||||
|
||||
- Replacing SQLModel domain ownership with MongoDB
|
||||
- Introducing breaking UI behavior for existing V1 flows
|
||||
- Expanding revision cardinality beyond current `0..1` without explicit requirements update
|
||||
|
||||
---
|
||||
|
||||
## Proposed Scope
|
||||
|
||||
### A) PostgreSQL migration (required)
|
||||
|
||||
- Add PostgreSQL runtime profile for local/dev/prod
|
||||
- Introduce migration toolchain and migration history
|
||||
- Convert bootstrap strategy from compatibility patching to explicit migrations
|
||||
- Validate model constraints and indexes against PostgreSQL
|
||||
- Add operational checks (connectivity, pool, transaction behavior)
|
||||
|
||||
### B) MongoDB integration (optional, gated)
|
||||
|
||||
- Define approved use cases (for example: denormalized read models, audit/event projections, or search-oriented materializations)
|
||||
- Keep canonical write path in relational store
|
||||
- Add feature flag/config gate to enable or disable Mongo features
|
||||
- Document consistency model and failure behavior
|
||||
|
||||
---
|
||||
|
||||
## Milestones
|
||||
|
||||
## M1 — Requirements and architecture baseline
|
||||
|
||||
- Create V2 requirements delta from V1 baseline
|
||||
- Define relational/document ownership boundaries
|
||||
- Approve migration strategy and cutover approach
|
||||
|
||||
**Exit criteria:** signed architecture decision and updated traceability map.
|
||||
|
||||
## M2 — PostgreSQL foundation
|
||||
|
||||
- Add PostgreSQL environment wiring and secrets strategy
|
||||
- Add migration framework and initial schema migration
|
||||
- Add CI path using PostgreSQL service container
|
||||
|
||||
**Exit criteria:** test suite green on PostgreSQL in CI.
|
||||
|
||||
## M3 — Data migration and cutover rehearsal
|
||||
|
||||
- Build SQLite -> PostgreSQL migration utility/playbook
|
||||
- Rehearse migration on representative datasets
|
||||
- Validate rollback/recovery procedures
|
||||
|
||||
**Exit criteria:** successful dry-run migration with measured rollback test.
|
||||
|
||||
## M4 — MongoDB optional integration
|
||||
|
||||
- Implement scoped Mongo use case(s)
|
||||
- Add fallback behavior when Mongo unavailable
|
||||
- Add tests and operational runbook updates
|
||||
|
||||
**Exit criteria:** feature-gated Mongo behavior validated with no V1 flow regressions.
|
||||
|
||||
## M5 — Release readiness
|
||||
|
||||
- Final regression suite (functional + reliability)
|
||||
- Performance and failure-mode checks
|
||||
- Production release checklist and sign-off
|
||||
|
||||
**Exit criteria:** V2 release approval.
|
||||
|
||||
---
|
||||
|
||||
## Risks and Mitigations
|
||||
|
||||
- **Schema drift risk** -> enforce migration-first policy and CI migration checks.
|
||||
- **Dual-store consistency risk** -> keep relational source of truth and explicit projection contracts.
|
||||
- **Operational complexity** -> staged rollout, runbooks, and feature flags.
|
||||
- **Regression risk in worker lifecycle** -> keep dedicated reliability tests around terminal-state guarantees.
|
||||
|
||||
---
|
||||
|
||||
## Traceability and Evidence
|
||||
|
||||
Maintain a V2 table with:
|
||||
|
||||
- requirement/change ID
|
||||
- status (`not started` / `in progress` / `done`)
|
||||
- implementation PR
|
||||
- validation evidence (test names, migration rehearsal logs, runbook references)
|
||||
|
||||
---
|
||||
|
||||
## Suggested first implementation tasks
|
||||
|
||||
1. Create `docs/ver2/adr/` and draft ADR for persistence ownership boundaries.
|
||||
2. Add PostgreSQL compose profile and env contract.
|
||||
3. Introduce migration tooling and generate initial migration from current schema.
|
||||
4. Add CI job for PostgreSQL-backed `pytest -m "not external"`.
|
||||
@@ -2,12 +2,12 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from contextlib import AsyncExitStack
|
||||
from contextlib import asynccontextmanager
|
||||
from datetime import UTC
|
||||
from datetime import datetime
|
||||
from datetime import timedelta
|
||||
import logging
|
||||
|
||||
from fastapi import FastAPI
|
||||
from fastapi import status
|
||||
@@ -26,7 +26,6 @@ from .services.jobs import JobService
|
||||
from .ui import register_pages
|
||||
from .worker import worker_consumer_lifespan
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
|
||||
@@ -67,5 +67,13 @@ def _ensure_sqlite_compat_columns(connection: Connection) -> None:
|
||||
has_unique_source = True
|
||||
break
|
||||
if not has_unique_source:
|
||||
connection.execute(text("CREATE UNIQUE INDEX IF NOT EXISTS ux_revision_source_id ON revision(source_id)"))
|
||||
logger.warning("Applied SQLite compatibility schema patch table=revision unique_index=ux_revision_source_id")
|
||||
connection.execute(
|
||||
text(
|
||||
"CREATE UNIQUE INDEX IF NOT EXISTS "
|
||||
"ux_revision_source_id ON revision(source_id)"
|
||||
)
|
||||
)
|
||||
logger.warning(
|
||||
"Applied SQLite compatibility schema patch "
|
||||
"table=revision unique_index=ux_revision_source_id"
|
||||
)
|
||||
|
||||
@@ -85,7 +85,7 @@ async def process_queued_job(
|
||||
source.id,
|
||||
result.provider,
|
||||
)
|
||||
except TimeoutError as exc:
|
||||
except TimeoutError:
|
||||
error = AppError(
|
||||
f"Provider call timed out after {runtime_settings.worker_provider_timeout_seconds:.1f}s",
|
||||
category=ErrorCategory.EXTERNAL_PROVIDER,
|
||||
|
||||
@@ -23,7 +23,7 @@ from ..components.transcript import render_original_transcription_card
|
||||
from ..components.transcript import render_revision_row
|
||||
|
||||
|
||||
def register_page() -> None:
|
||||
def register_page() -> None: # noqa: PLR0915
|
||||
"""Register jobs list and detail routes."""
|
||||
|
||||
@ui.page("/jobs")
|
||||
@@ -51,7 +51,7 @@ def register_page() -> None:
|
||||
await render_table()
|
||||
|
||||
@ui.page("/jobs/{job_id}")
|
||||
async def job_detail_page(job_id: str, request: Request) -> None:
|
||||
async def job_detail_page(job_id: str, request: Request) -> None: # noqa: PLR0915
|
||||
session_factory = resolve_session_factory(request.app.state)
|
||||
jobs_service = JobService(session_factory=session_factory)
|
||||
transcription_service = TranscriptionService(session_factory=session_factory)
|
||||
@@ -105,14 +105,52 @@ def register_page() -> None:
|
||||
async def render_revision_panel() -> None:
|
||||
refreshed_job = await jobs_service.read_job(job_id=parsed_job_id)
|
||||
refreshed_source = _resolve_primary_source(refreshed_job)
|
||||
if refreshed_source is None or refreshed_source.revision is None:
|
||||
if refreshed_source is None:
|
||||
ui.label("No source is available for revision editing.").classes("text-body2 text-grey-3")
|
||||
return
|
||||
|
||||
current_revision = refreshed_source.revision
|
||||
default_revision_text = (
|
||||
current_revision.text if current_revision is not None else (refreshed_job.text or "")
|
||||
)
|
||||
|
||||
ui.label("Revision Editor").classes("text-subtitle1 text-weight-medium")
|
||||
editor = ui.textarea(label="Revision text", value=default_revision_text).props("autogrow outlined")
|
||||
editor.classes("w-full")
|
||||
|
||||
async def save_revision() -> None:
|
||||
candidate = (editor.value or "").strip()
|
||||
if not candidate:
|
||||
ui.notify("Revision text is required.", type="warning")
|
||||
return
|
||||
|
||||
try:
|
||||
await transcription_service.upsert_revision_for_source(
|
||||
source_id=refreshed_source.id,
|
||||
text=candidate,
|
||||
)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
show_error(exc, title="Save failed", operation="jobs.save_revision")
|
||||
return
|
||||
|
||||
ui.notify("Revision saved", type="positive")
|
||||
await render_revision_panel.refresh()
|
||||
|
||||
with ui.row().classes("w-full justify-end"):
|
||||
ui.button(
|
||||
"Create revision" if current_revision is None else "Update revision",
|
||||
on_click=save_revision,
|
||||
icon="save",
|
||||
).props('unelevated color="primary"')
|
||||
|
||||
if current_revision is None:
|
||||
ui.label("No revision exists for this source.").classes("text-body2 text-grey-3")
|
||||
return
|
||||
|
||||
render_revision_row(
|
||||
revision=refreshed_source.revision,
|
||||
revision=current_revision,
|
||||
initially_expanded=True,
|
||||
on_delete=lambda _revision, rid=refreshed_source.revision.id: delete_revision_by_id(rid),
|
||||
on_delete=lambda _revision, rid=current_revision.id: delete_revision_by_id(rid),
|
||||
)
|
||||
|
||||
await render_revision_panel()
|
||||
|
||||
@@ -1,11 +1,12 @@
|
||||
"""Tests for API error response envelope handlers."""
|
||||
|
||||
import pytest
|
||||
from fastapi import FastAPI
|
||||
from fastapi.testclient import TestClient
|
||||
import pytest
|
||||
|
||||
from transcription.api.errors import register_error_handlers
|
||||
from transcription.errors import AppError, ErrorCategory
|
||||
from transcription.errors import AppError
|
||||
from transcription.errors import ErrorCategory
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
|
||||
@@ -11,11 +11,11 @@ BOOK 1 had 54 pages; 14 chapters. BOOK 2 has 70 pages; 18 chapters. BOOK 1 con-
|
||||
sisted largely of first generation family history. BOOK 2 throws more light on
|
||||
the second generation. Sidney promises a BOOK 3 and that may begin to do justice
|
||||
to the third generation. We suggest that Sidney get the help of Louis Shinn
|
||||
who has a chapter in this book (Chapter 16 - The Last 25 Years on the Doumeeq
|
||||
who has a chapter in this book (Chapter 16 - The Last 25 Years on the Doumecq
|
||||
Plains. Louis has the gift of seeing, recalling and telling. One sentence in
|
||||
his chapter gives a great tribute to the Doumeeqers - so far as he knows no one
|
||||
on the Doumeeq Plains went on relief during the depression. That in a nutshell
|
||||
shows the sturdy character of the residents of the Doumeeq Plains.
|
||||
his chapter gives a great tribute to the Doumeccqers--so [sic] far as he knows no one
|
||||
on the Doumecq Plains went on relief during the depression. That in a nutshell
|
||||
shows the sturdy character of the residents of the Doumecq Plains.
|
||||
|
||||
We promised in BOOK 1 that in BOOK 2 we would give the story of the trip of
|
||||
John E. Cochran and wife to Tennessee, Cuba and the Panama Canal. You will see
|
||||
|
||||
@@ -18,7 +18,6 @@ the family newsletter two years ago.
|
||||
|
||||
Nome Alaska August 26, 1923
|
||||
My Dear Ethel et al.
|
||||
|
||||
I don't know when I did write or when you did
|
||||
but I am going to write now however and never
|
||||
the less. But I wish I could talk (I can yet but I
|
||||
@@ -27,7 +26,6 @@ and Polly sit up and listen and that little black
|
||||
rascal of yours would fairly sparkle with
|
||||
listening. Can't I see him listening now to all the
|
||||
yarns we told last summer?
|
||||
|
||||
You see, we-Miss Saville and I, took a trip north
|
||||
on the Buford and it was very interesting. We
|
||||
went north thru the Bering Strait into the Arctic and as far as the Ice Pack. There the captain
|
||||
@@ -43,14 +41,14 @@ all around it similar to a currycomb in coarseness; no ears but huge tusks of iv
|
||||
the most repulsive looking animals imaginable and tho I have always read about them I never
|
||||
expect such disagreeable looking creatures. They had a rough brown hairy skin and some of
|
||||
them looked warty. They must have weighed two ton at least. Ere we got them back to Nome
|
||||
to the natives they were getting extremely odiferous–in fact, you could scarcely stay on the
|
||||
to the natives they were getting extremely odiferous—in fact, you could scarcely stay on the
|
||||
ship with any degree of comfort unless you had per chance lost your sense of smell.
|
||||
|
||||
Then we went north to a few minutes beyond the 70th degree of latitude and thot [sic] for awhile
|
||||
we would go to Wrangell Island where some men from Stefflonsons [sic] ship were supposed to be
|
||||
stranded but we didn't get there and instead stopped at a small native village at Cape Serdz [sic] in
|
||||
Then we went north to a few minutes beyond the 70th degree of latitude and thot for awhile
|
||||
we would go to Wrangell Island where some men from Steffonsons ship were supposed to be
|
||||
stranded but we didn't get there and instead stopped at a small native village at Cape Serdz in
|
||||
Siberia. These Eskimo were very primitive. One white squaw man lived there and had for 23
|
||||
years. He was a Swede–who else could. Their houses were circular and built up with dirt 2 or
|
||||
years. He was a Swede--who else could. Their houses were circular and built up with dirt 2 or
|
||||
3 feet and then skins were stretched over it and weighted down with rocks. Inside, the room
|
||||
was partitioned off at the sides with skins for sleeping quarters. In the main part they had the
|
||||
fire on the ground and the fish drying on lines and the skins hanging around and the dogs and
|
||||
@@ -65,22 +63,22 @@ The other place we stopped was at Whalen, a trading post in Siberia. There these
|
||||
went wild. They rushed helter-skelter, hither and thither, here and there, trying to find
|
||||
something to buy. Prices raised right before your eyes. One would but something for $1.00
|
||||
and the next might have to pay $2.00, $4.00 or $10.00. That made no difference. They had to
|
||||
have it. One man I was sort of taking care of, tho [sic] he had his son along for the purpose,
|
||||
bought 2 ivory tusks, 1 pup, 2 moccasins, 3 or 4 billi[illegible]s, 6 or 8 ivory and silver rings, one
|
||||
have it. One man I was sort of taking care of, tho he had his son along for the purpose,
|
||||
bought 2 ivory tusks, 1 pup, 2 moccasins, 3 or 4 billikens, 6 or 8 ivory and silver rings, one
|
||||
fishing line, hooks, floats, etc. and two bird slings. The slings have rocks at the end and the
|
||||
little natives throw them at the flocks of geese and ducks which fly close over the village and
|
||||
the slings entangle their wings and legs, sometimes more than one, and they can't fly. They
|
||||
come down and the natives capture them. There was more junk brot [sic] aboard than baggage, I
|
||||
come down and the natives capture them. There was more junk brot aboard than baggage, I
|
||||
do believe. And they say that at the first stop it was worse than here. The red flag was flying
|
||||
over Whalen and the Russian soldiers were there–a few, one or two or three, I forget the
|
||||
over Whalen and the Russian soldiers were there—a few, one or two or three, I forget the
|
||||
number.
|
||||
|
||||
We got home yesterday morning at 5 a.m. but missed the first lighter in so had to stay out
|
||||
until 2:30. The girls had prepared a big meal for us and invited up the Hartfords and then let
|
||||
us talk. Miss Saville talked quite a bit. Any how if you folks don't like this I don't care, it is
|
||||
all I had to write about and I know Buster'd [sic] listen anyway and I'd soak ole Peter's head if he
|
||||
all I had to write about and I know Buster'ud listen anyway and I'd soak ole Peter's head if he
|
||||
didn't and Polly would in my lap and I don't know much about the youngest one of yours so
|
||||
likely he would be squawling. But we did surely enjoy our trip and were gone just long enuf [sic].
|
||||
likely he would be squawling. But we did surely enjoy our trip and were gone just long enuf.
|
||||
|
||||
I expect there were 150 passengers on board and almost or more of the crew and helpers. We
|
||||
had a stateroom down next to the kitchen and 'twas pretty fierce for odor at times.
|
||||
@@ -109,7 +107,7 @@ Ome
|
||||
|
||||
Reprinted from Cochran Chronicles, Volume 9, Number 1, November 1986
|
||||
|
||||
© [inserted: JECFA] 1986
|
||||
© JECFA 1986
|
||||
|
||||
Up
|
||||
|
||||
|
||||
@@ -8,24 +8,23 @@ ISBILL & MOSER
|
||||
DEALERS IN
|
||||
GENERAL MERCHANDISE
|
||||
|
||||
Vonore, Tenn., Jany 27- 1913
|
||||
Dear Much Aunt Louie
|
||||
How are you a
|
||||
few nights ago I sewed a
|
||||
letter from your folks, so
|
||||
Vonore, Tenn. Jany 27- 1913
|
||||
Dear Much Aunt Adeline
|
||||
Was at home a
|
||||
few nights ago & saw a
|
||||
letter from your folks. So
|
||||
I decided to write you
|
||||
a few lines myself ok
|
||||
I am contemplateing a
|
||||
I am contemplateing [sic] a
|
||||
trip out west next summer
|
||||
& I want Some Olders to go
|
||||
where I and them.
|
||||
|
||||
I am getting
|
||||
& [inserted: I] want some of them to go
|
||||
when I am [inserted: a] them.
|
||||
Am getting
|
||||
up in years & unmarried
|
||||
so you see the object of
|
||||
my trip, is to get a bunch
|
||||
of Young & old maids
|
||||
& widows out there. I
|
||||
my trip, is to get a wife
|
||||
& if there is any old maids
|
||||
or widows out there, I
|
||||
want you to kiss them
|
||||
at my fans [sic] mug as they
|
||||
as soon as I get there
|
||||
at my [hand?] me at there
|
||||
as soon as I get there.
|
||||
|
||||
@@ -31,7 +31,12 @@ class TestPipelineSuccessFlow:
|
||||
|
||||
async def _fake_transcribe(*, prompt_text: str, image_bytes: bytes, mime_type: str) -> TranscriptionResult:
|
||||
_ = (prompt_text, image_bytes, mime_type)
|
||||
return TranscriptionResult(text="Pipeline transcript", provider="openrouter", model="test-model", prompt_name="transcribe_document.md")
|
||||
return TranscriptionResult(
|
||||
text="Pipeline transcript",
|
||||
provider="openrouter",
|
||||
model="test-model",
|
||||
prompt_name="transcribe_document.md",
|
||||
)
|
||||
|
||||
async def _fake_transcribe_document_image(
|
||||
image_path,
|
||||
|
||||
@@ -5,8 +5,10 @@ from types import SimpleNamespace
|
||||
import pytest
|
||||
|
||||
from transcription.config import Settings
|
||||
from transcription.providers.base import ProviderError, ProviderResponseError
|
||||
from transcription.providers.openrouter import DEFAULT_OPENROUTER_MODEL, OpenRouterTranscriptionProvider
|
||||
from transcription.providers.base import ProviderError
|
||||
from transcription.providers.base import ProviderResponseError
|
||||
from transcription.providers.openrouter import DEFAULT_OPENROUTER_MODEL
|
||||
from transcription.providers.openrouter import OpenRouterTranscriptionProvider
|
||||
|
||||
|
||||
class _FakeChat:
|
||||
|
||||
@@ -25,7 +25,11 @@ class TestJobService:
|
||||
assert fetched.document.id == document.id
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_update_job_state_updates_status_and_retry(self, job_service: JobService, document_service: DocumentService):
|
||||
async def test_update_job_state_updates_status_and_retry(
|
||||
self,
|
||||
job_service: JobService,
|
||||
document_service: DocumentService,
|
||||
):
|
||||
document = Document(id=uuid4(), name="test-bundle")
|
||||
await document_service.create_document(document=document)
|
||||
|
||||
|
||||
@@ -7,7 +7,6 @@ import pytest
|
||||
|
||||
from transcription.services.transcription import transcribe_document_image
|
||||
|
||||
|
||||
HAS_OPENROUTER_KEY = bool(os.getenv("OPENROUTER_API_KEY"))
|
||||
|
||||
REAL_IMAGES_DIR = Path(__file__).resolve().parents[1] / "fixtures" / "images" / "real"
|
||||
@@ -68,4 +67,4 @@ class TestRealImageExternalTranscription:
|
||||
f"{result.text}\n"
|
||||
)
|
||||
artifact_path.write_text(artifact_text, encoding="utf-8")
|
||||
assert artifact_path.exists()
|
||||
assert artifact_path.exists()
|
||||
|
||||
@@ -0,0 +1,85 @@
|
||||
"""Tests for revision behavior in TranscriptionService."""
|
||||
|
||||
from uuid import uuid4
|
||||
|
||||
import pytest
|
||||
|
||||
from transcription.models import Document
|
||||
from transcription.models import Job
|
||||
from transcription.models import JobStatus
|
||||
from transcription.models import Source
|
||||
from transcription.services.documents import DocumentService
|
||||
from transcription.services.jobs import JobService
|
||||
from transcription.services.transcription import TranscriptionService
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
class TestTranscriptionServiceRevisionUpsert:
|
||||
"""Verify optional single-revision create/update semantics."""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_upsert_revision_creates_new_revision(self, default_session_factory):
|
||||
documents = DocumentService(session_factory=default_session_factory)
|
||||
jobs = JobService(session_factory=default_session_factory)
|
||||
transcriptions = TranscriptionService(session_factory=default_session_factory)
|
||||
|
||||
document = Document(id=uuid4(), name="revision-create")
|
||||
await documents.create_document(document=document)
|
||||
|
||||
job = Job(document_id=document.id, status=JobStatus.TRANSCRIBED, text="Original text")
|
||||
await jobs.create_job(job=job)
|
||||
|
||||
source = Source(
|
||||
document_id=document.id,
|
||||
job_id=job.id,
|
||||
upload_name="source.jpg",
|
||||
filename="source.jpg",
|
||||
file_path="uploads/source.jpg",
|
||||
)
|
||||
async with transcriptions._session_scope() as session:
|
||||
session.add(source)
|
||||
await session.commit()
|
||||
await session.refresh(source)
|
||||
|
||||
revision = await transcriptions.upsert_revision_for_source(source_id=source.id, text="User revision")
|
||||
fetched = await transcriptions.read_revision_by_source(source.id)
|
||||
|
||||
assert revision.source_id == source.id
|
||||
assert revision.text == "User revision"
|
||||
assert fetched is not None
|
||||
assert fetched.id == revision.id
|
||||
assert fetched.text == "User revision"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_upsert_revision_updates_existing_single_revision(self, default_session_factory):
|
||||
documents = DocumentService(session_factory=default_session_factory)
|
||||
jobs = JobService(session_factory=default_session_factory)
|
||||
transcriptions = TranscriptionService(session_factory=default_session_factory)
|
||||
|
||||
document = Document(id=uuid4(), name="revision-update")
|
||||
await documents.create_document(document=document)
|
||||
|
||||
job = Job(document_id=document.id, status=JobStatus.TRANSCRIBED, text="Original text")
|
||||
await jobs.create_job(job=job)
|
||||
|
||||
source = Source(
|
||||
document_id=document.id,
|
||||
job_id=job.id,
|
||||
upload_name="source.jpg",
|
||||
filename="source.jpg",
|
||||
file_path="uploads/source.jpg",
|
||||
)
|
||||
async with transcriptions._session_scope() as session:
|
||||
session.add(source)
|
||||
await session.commit()
|
||||
await session.refresh(source)
|
||||
|
||||
first = await transcriptions.upsert_revision_for_source(source_id=source.id, text="Revision v1")
|
||||
second = await transcriptions.upsert_revision_for_source(source_id=source.id, text="Revision v2")
|
||||
revisions = await transcriptions.list_revisions_by_job(job.id)
|
||||
|
||||
assert first.id == second.id
|
||||
assert second.text == "Revision v2"
|
||||
assert len(revisions) == 1
|
||||
assert revisions[0].id == first.id
|
||||
assert revisions[0].text == "Revision v2"
|
||||
@@ -22,9 +22,21 @@ class TestWorkflowReliability:
|
||||
async def test_process_queued_job_timeout_marks_job_failed(self, default_session_factory, monkeypatch):
|
||||
"""Provider timeout transitions a queued job to failed with error detail."""
|
||||
services = ServiceBundle()
|
||||
object.__setattr__(services, "documents", services.documents.__class__(session_factory=default_session_factory))
|
||||
object.__setattr__(services, "jobs", services.jobs.__class__(session_factory=default_session_factory))
|
||||
object.__setattr__(services, "transcriptions", services.transcriptions.__class__(session_factory=default_session_factory))
|
||||
object.__setattr__(
|
||||
services,
|
||||
"documents",
|
||||
services.documents.__class__(session_factory=default_session_factory),
|
||||
)
|
||||
object.__setattr__(
|
||||
services,
|
||||
"jobs",
|
||||
services.jobs.__class__(session_factory=default_session_factory),
|
||||
)
|
||||
object.__setattr__(
|
||||
services,
|
||||
"transcriptions",
|
||||
services.transcriptions.__class__(session_factory=default_session_factory),
|
||||
)
|
||||
|
||||
async with services.jobs._session_scope() as session:
|
||||
document = Document(id=uuid4(), name="timeout-doc")
|
||||
|
||||
@@ -5,7 +5,8 @@ from pathlib import Path
|
||||
import pytest
|
||||
from pydantic import ValidationError
|
||||
|
||||
from transcription.config import Provider, Settings
|
||||
from transcription.config import Provider
|
||||
from transcription.config import Settings
|
||||
|
||||
|
||||
def _make_settings(**overrides) -> Settings:
|
||||
|
||||
@@ -2,7 +2,10 @@
|
||||
|
||||
import pytest
|
||||
|
||||
from transcription.errors import AppError, ErrorCategory, classify_unexpected_error, new_error_id
|
||||
from transcription.errors import AppError
|
||||
from transcription.errors import ErrorCategory
|
||||
from transcription.errors import classify_unexpected_error
|
||||
from transcription.errors import new_error_id
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
|
||||
@@ -5,7 +5,11 @@ from uuid import UUID
|
||||
import pytest
|
||||
from sqlalchemy.exc import IntegrityError
|
||||
|
||||
from transcription.models import Document, Job, JobStatus, Revision, Source
|
||||
from transcription.models import Document
|
||||
from transcription.models import Job
|
||||
from transcription.models import JobStatus
|
||||
from transcription.models import Revision
|
||||
from transcription.models import Source
|
||||
|
||||
|
||||
def _make_document(**overrides) -> Document:
|
||||
|
||||
@@ -2,7 +2,6 @@
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
PROMPT_PATH = Path("prompts/transcribe_document.md")
|
||||
|
||||
|
||||
|
||||
@@ -71,3 +71,37 @@ class TestPageRendering:
|
||||
|
||||
assert response.status_code == 200
|
||||
assert "Job not found" in response.text
|
||||
|
||||
def test_job_detail_page_shows_revision_editor_when_none_exists(self, app_client, seed_job):
|
||||
"""GET /ui/jobs/{job_id} renders revision editor and create action for sources with no revision."""
|
||||
_, client = app_client
|
||||
job_id = seed_job(
|
||||
filename="no-revision.pdf",
|
||||
status=JobStatus.TRANSCRIBED,
|
||||
transcription_text="original text",
|
||||
revision_text=None,
|
||||
)
|
||||
|
||||
response = client.get(f"/ui/jobs/{job_id}")
|
||||
|
||||
assert response.status_code == 200
|
||||
assert "Revision Editor" in response.text
|
||||
assert "Create revision" in response.text
|
||||
assert "No revision exists for this source." in response.text
|
||||
|
||||
def test_job_detail_page_shows_update_action_for_existing_revision(self, app_client, seed_job):
|
||||
"""GET /ui/jobs/{job_id} renders revision editor with update action when revision exists."""
|
||||
_, client = app_client
|
||||
job_id = seed_job(
|
||||
filename="with-revision.pdf",
|
||||
status=JobStatus.TRANSCRIBED,
|
||||
transcription_text="original text",
|
||||
revision_text="hello",
|
||||
)
|
||||
|
||||
response = client.get(f"/ui/jobs/{job_id}")
|
||||
|
||||
assert response.status_code == 200
|
||||
assert "Revision Editor" in response.text
|
||||
assert "Update revision" in response.text
|
||||
assert "hello" in response.text
|
||||
|
||||
BIN
Binary file not shown.
Binary file not shown.
Reference in New Issue
Block a user