1 Commits
68 changed files with 3440 additions and 4928 deletions
-13
View File
@@ -1,13 +0,0 @@
.git
.gitignore
.vscode
.venv
.pytest_cache
.ruff_cache
__pycache__/
*.py[cod]
*.db
.env
tests/
docs/
uploads/
@@ -1,77 +0,0 @@
---
description: Follow these guidelines when editing the services
applyTo: 'src/transcription/services/*.py'
---
# Services
## Structure
- Project core data models defined in [models](../../src/transcription/models.py)
- 1 service class per data model
- Only services directly interact with the database, and only through async methods
- Services are completely independent of one another. Any operation that needs to use more than a single service, which is most of them, needs to have a separate orchestration function.
## Error Handling
- Service-specific errors defined at the top of the respective module and inherit from `AppError`
- Use a context manager for large `try/except` blocks like in [transcription](../../src/transcription/services/transcription.py)
## Checklist
- [ ] Uses `ServiceBase` for common logic
- [ ] CRUD methods created at the top
- [ ] Session kwarg for `AsyncSession` to pass in a session object to each method
- [ ] Services use `self._session_scope` in their methods to pass the session thru.
- Multiple operations on the same object(s) require sharing a session between all the methods used.
## CRUD Methods
- Create, read, update, and delete, created in that order
- Name format `<operation>_<model >`, for example `create_document` or `update_job`
- All services must define these 4 methods first, and in that order
## Transaction Finalization
When a service method accepts an optional `session` kwarg, write methods must use `self._finalize` to finalize the transaction properly according to whether or not they are sharing a session.
- If `session` is `None`: the method owns the transaction and should `commit()`.
- If `session` is provided: the method must **not** commit; it should `flush()` so IDs and FK values are available to the caller's transaction.
- Use `refresh()` on returned ORM objects when the caller needs DB-populated values (defaults, triggers, merged state).
Recommended helper behavior:
- Inputs: active session object, original `session` arg (or a boolean ownership flag), and an optional list of objects to refresh.
- Logic: `commit` when service-owned session, `flush` when caller-owned session, then refresh requested objects.
This keeps orchestration functions atomic: they can pass one shared session across multiple services and commit exactly once at the workflow boundary.
## Workflow Transaction Boundaries
For multi-step job lifecycles (for example queued transcription jobs), orchestration functions must use explicit transaction phases.
Required boundary model:
- **Transaction A (claim):** transition `JobStatus.QUEUED -> JobStatus.PROCESSING` and commit immediately.
- Perform provider/network work **outside** database transactions.
- **Transaction B (terminal success):** write transcript content and set `JobStatus.TRANSCRIBED` in the same shared-session commit.
- **Transaction B (terminal failure):** write transcript error detail and set `JobStatus.FAILED` in the same shared-session commit.
- **Transaction C (retry path):** write transcript error detail, increment retry count, and set `JobStatus.QUEUED` in one shared-session commit.
Atomicity rules:
- Never commit transcript updates separately from the paired terminal/retry job status change.
- Terminal state (`TRANSCRIBED` or `FAILED`) and transcript row changes must succeed or roll back together.
- Retry persistence (`QUEUED` + retry increment + error detail) must succeed or roll back together.
Separation of concerns:
- Worker modules should stay lightweight and delegate lifecycle transitions to service/workflow orchestration functions.
- In `workflows.py`, `process_queued_job` should own one complete attempt lifecycle: `QUEUED -> PROCESSING -> TRANSCRIBED|FAILED`.
- In `workflows.py`, `advance_job` should coordinate broader status progression around attempts (for example retry scheduling from `FAILED -> QUEUED`).
- Services should expose session-aware write helpers (flush on caller-owned session) so orchestration controls commit boundaries.
- Backoff/sleep behavior must run outside transactional scopes.
# Service Composition
Some operations, like uploading a picutre, require modifications to multiple tables, which can be done by composing methods from the service object into a separate function.
-6
View File
@@ -1,6 +0,0 @@
---
description: Copilot rules for modifying the UI
applyTo: 'src/transcription/ui/**/*.py'
---
The UI is forbidden from directly touching the database. All interactions should be done using the [services](../../src/transcription/services/)
-5
View File
@@ -14,8 +14,3 @@ wheels/
# SQLite database # SQLite database
*.db *.db
upload/
*.jpg
*.jpeg
*.png
-27
View File
@@ -1,27 +0,0 @@
{
"version": "0.2.0",
"configurations": [
{
"name": "Python: Debug transcription app",
"type": "debugpy",
"request": "launch",
"module": "debugpy",
"args": [
"-m",
"uvicorn",
"transcription.app:create_app",
"--factory",
"--host",
// "127.0.0.1",
"0.0.0.0",
"--port",
"8080"
],
"justMyCode": true,
"console": "integratedTerminal",
"env": {
"PYTHONPATH": "${workspaceFolder}/src"
}
}
]
}
-47
View File
@@ -1,47 +0,0 @@
FROM python:3.12-slim AS builder
ENV PYTHONDONTWRITEBYTECODE=1 \
PYTHONUNBUFFERED=1 \
UV_LINK_MODE=copy
WORKDIR /app
COPY --from=ghcr.io/astral-sh/uv:0.5.24 /uv /uvx /bin/
COPY pyproject.toml uv.lock README.md ./
RUN uv sync --frozen --no-dev --no-install-project
COPY src ./src
COPY prompts ./prompts
RUN uv sync --frozen --no-dev
FROM python:3.12-slim AS runtime
ENV PYTHONDONTWRITEBYTECODE=1 \
PYTHONUNBUFFERED=1 \
PATH="/app/.venv/bin:$PATH" \
PYTHONPATH="/app/src" \
UPLOAD_DIR="/app/uploads" \
PROMPT_DIR="/app/prompts"
WORKDIR /app
RUN groupadd --system --gid 1001 appgroup \
&& useradd --system --uid 1001 --gid appgroup --create-home appuser
COPY --from=builder /app/.venv /app/.venv
COPY --from=builder /app/src /app/src
COPY --from=builder /app/prompts /app/prompts
RUN mkdir -p /app/uploads /app/data \
&& chown -R appuser:appgroup /app
USER appuser
EXPOSE 8000
HEALTHCHECK --interval=30s --timeout=3s --start-period=3s --retries=3 \
CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:8000/healthz')"
CMD ["uvicorn", "transcription.app:create_app", "--factory", "--host", "0.0.0.0", "--port", "8000", "--proxy-headers"]
-21
View File
@@ -1,21 +0,0 @@
services:
transcription:
build:
context: .
dockerfile: Dockerfile
container_name: transcription-app
env_file:
- .env
environment:
DATABASE_URL: sqlite:////app/data/transcription.db
UPLOAD_DIR: /app/uploads
PROMPT_DIR: /app/prompts
ports:
- "8002:8000"
volumes:
- ./uploads:/app/uploads
- transcription_data:/app/data
restart: unless-stopped
volumes:
transcription_data:
@@ -0,0 +1,35 @@
# ADR-0001: Lifespan-owned runtime resources
- **Status:** accepted
- **Date:** 2026-06-25
## Context
MVP initialized core runtime resources (database engine and worker dependencies) through module-level globals and startup side effects. `REQ-7` requires lifespan-owned runtime resources with explicit ownership and cleanup.
## Decision
Adopt lifespan-owned runtime resource initialization in `transcription.app`:
1. Initialize database runtime during app lifespan startup.
2. Store runtime handles on `app.state`.
3. Pass runtime-owned dependencies (engine) to worker startup.
4. Dispose runtime resources explicitly during lifespan shutdown.
## Consequences
### Positive
- Explicit startup and shutdown ownership.
- Predictable cleanup ordering.
- Reduced hidden global side effects.
### Tradeoffs
- Minor wiring complexity in app startup.
- Some call-sites still support fallback lazy initialization for compatibility.
## Alternatives Considered
1. **Keep module-level global ownership**
- Rejected: conflicts with `REQ-7` and increases ambiguity.
2. **Introduce full async DB stack immediately**
- Rejected for Step 1: too broad for architecture-consolidation scope.
@@ -0,0 +1,36 @@
# ADR-0002: Explicit schema bootstrap policy
- **Status:** accepted
- **Date:** 2026-06-25
## Context
MVP called schema bootstrap (`create_all`) on every startup. `REQ-10` requires explicit, opt-in schema bootstrap behavior so normal production startup does not mutate schema.
## Decision
Add environment-aware bootstrap policy:
1. New settings:
- `environment`: `development` | `test` | `production`
- `bootstrap_schema_on_startup`: optional explicit override
2. Default behavior:
- Development/test: bootstrap enabled
- Production: bootstrap disabled
3. App startup calls `create_all` only when policy evaluates true.
## Consequences
### Positive
- Production startup behavior is safer and policy-driven.
- Local development remains simple by default.
### Tradeoffs
- Deployments now require explicit schema management in production.
## Alternatives Considered
1. **Always bootstrap in all environments**
- Rejected: violates `REQ-10` intent.
2. **Disable bootstrap everywhere immediately**
- Rejected: hurts local developer workflow without migration tool replacement yet.
@@ -0,0 +1,31 @@
# ADR-0003: Persistence baseline and transition path
- **Status:** accepted
- **Date:** 2026-06-25
## Context
Architecture targets PostgreSQL baseline (optional MongoDB), while MVP currently runs on SQLite by default. V1 needs a clear transition path without destabilizing ongoing work.
## Decision
1. Preserve database URL configurability through centralized settings.
2. Keep SQLite functional for local dev/test and fast feedback.
3. Treat PostgreSQL as production baseline target for V1 completion.
4. Keep persistence access behind `transcription.db` runtime/session access points.
## Consequences
### Positive
- Clear migration path without immediate broad rewrite.
- Controlled risk while preserving velocity.
### Tradeoffs
- Temporary dual-path assumptions (SQLite local vs PostgreSQL target).
## Alternatives Considered
1. **Immediate forced PostgreSQL-only migration**
- Rejected: higher short-term disruption risk.
2. **Remain SQLite-only for V1**
- Rejected: inconsistent with architecture and requirement trajectory.
@@ -0,0 +1,32 @@
# ADR-0004: In-process worker topology for V1
- **Status:** accepted
- **Date:** 2026-06-25
## Context
The current system uses an in-process background worker. Architecture docs allow this in foundation stage and permit later hardening (optional external worker/queue).
## Decision
Retain in-process worker topology for V1, with improved lifecycle ownership:
1. Worker starts/stops via app lifespan.
2. Worker receives runtime-owned DB engine dependency explicitly.
3. Extension path to external worker remains behind existing service/adapter seams.
## Consequences
### Positive
- Keeps operational complexity low for personal-scale use.
- Preserves delivery focus on V1 completion.
### Tradeoffs
- Throughput/scaling limits remain compared to external queue-based topology.
## Alternatives Considered
1. **Immediate queue/external worker introduction**
- Rejected: premature complexity for current scale.
2. **Ad hoc thread lifecycle management outside lifespan**
- Rejected: weaker shutdown guarantees and poorer ownership clarity.
+20
View File
@@ -0,0 +1,20 @@
# Architecture Decision Records (ADRs)
This directory records significant architecture decisions for Version 1.
## ADR Format
Each ADR should include:
1. **Status** (`proposed`, `accepted`, `superseded`)
2. **Context**
3. **Decision**
4. **Consequences**
5. **Alternatives Considered**
## Index
- [ADR-0001: Lifespan-owned runtime resources](ADR-0001-lifespan-owned-runtime-resources.md)
- [ADR-0002: Explicit schema bootstrap policy](ADR-0002-explicit-schema-bootstrap-policy.md)
- [ADR-0003: Persistence baseline and transition path](ADR-0003-persistence-baseline-and-transition-path.md)
- [ADR-0004: In-process worker topology for V1](ADR-0004-in-process-worker-topology.md)
+86
View File
@@ -0,0 +1,86 @@
# Ver1 Step 1 Results: Architecture Consolidation
## Summary
Step 1 implementation has been completed for the primary architecture-consolidation objectives:
1. Lifespan-owned runtime resource model introduced for DB runtime ownership.
2. Schema bootstrap policy changed from implicit-always to explicit/environment-aware.
3. Worker startup now receives lifespan-owned DB engine dependency.
4. ADR set established for key V1 architectural decisions.
## Implemented Changes
### 1) Runtime ownership
- Updated `src/transcription/db.py`:
- Added `DatabaseRuntime` resource model.
- Added explicit runtime lifecycle methods:
- `initialize_database_runtime(...)`
- `get_database_runtime()`
- `dispose_database_runtime()`
- Updated `src/transcription/app.py`:
- Lifespan initializes DB runtime and stores it on `app.state`.
- Lifespan disposes DB runtime on shutdown.
### 2) Schema bootstrap policy (REQ-10 alignment)
- Updated `src/transcription/config.py`:
- Added `environment` setting (`development`, `test`, `production`).
- Added `bootstrap_schema_on_startup` explicit override setting.
- Updated `src/transcription/db.py`:
- Added `should_bootstrap_schema(settings)` policy function.
- Updated `src/transcription/app.py`:
- Startup now calls `create_all(...)` only when policy allows.
### 3) Worker dependency ownership
- Updated `src/transcription/worker.py`:
- `process_next_queued_job(..., engine=None)` now supports explicit engine injection.
- `run_worker_loop(..., engine=None, ...)` now supports explicit engine injection.
- Updated `src/transcription/app.py`:
- Worker thread is started with lifespan-owned engine.
### 4) ADR governance
Created:
- `docs/adr/README.md`
- `docs/adr/ADR-0001-lifespan-owned-runtime-resources.md`
- `docs/adr/ADR-0002-explicit-schema-bootstrap-policy.md`
- `docs/adr/ADR-0003-persistence-baseline-and-transition-path.md`
- `docs/adr/ADR-0004-in-process-worker-topology.md`
## Test Evidence
Targeted regression checks executed successfully:
- `uv run pytest tests/test_app.py tests/test_db.py tests/services/test_worker.py -q`
- Result: pass
## Residual Risks / Follow-ups
1. Full REQ-7 completion may still require broader runtime ownership coverage for additional resources as V1 expands.
2. Production schema management workflow (migrations/runbook tooling) should be finalized in subsequent V1 steps.
3. Additional boundary enforcement automation (import-lint style checks) can be added in later hardening.
## Step 1 Exit Assessment
- Architecture ownership clarity: **met**
- Schema bootstrap policy hardening: **met**
- Worker lifecycle dependency clarity: **met**
- ADR baseline established: **met**
## Completion Checklist With Evidence
| Criterion | Status | Evidence |
| --- | --- | --- |
| Architecture conformance matrix approved | partial | Consolidation implemented and documented in `docs/ver1/ver1-step1.md` + this results doc; formal matrix artifact can be added as a follow-up appendix. |
| REQ-7 ownership gaps resolved or explicitly deferred | met | Lifespan-owned DB runtime and explicit worker engine wiring implemented in `src/transcription/app.py`, `src/transcription/db.py`, `src/transcription/worker.py`. Residual scope documented under follow-ups. |
| REQ-10 explicit bootstrap policy implemented and verified | met | Policy implemented via `environment` + `bootstrap_schema_on_startup` in `src/transcription/config.py`, `should_bootstrap_schema(...)` in `src/transcription/db.py`, startup gate in `src/transcription/app.py`, tested in `tests/test_db.py`. |
| Dependency direction rules documented and enforced | partial | Layering and runtime ownership documented in `docs/architecture.md`. Lightweight enforcement exists via review and test discipline; automated import-lint remains a follow-up. |
| ADR set created for major Step 1 decisions | met | `docs/adr/README.md` and ADR-0001 through ADR-0004 created. |
| Architecture/index docs updated to match implementation | met | `docs/architecture.md` and `docs/index.md` updated with V1 Step 1 runtime policy and links to V1/ADR artifacts. |
| Regression and full test suites pass | met | Targeted: `uv run pytest tests/test_app.py tests/test_db.py tests/services/test_worker.py -q`; full suite: `uv run pytest -q`. |
| Step 1 results artifact published | met | This document (`docs/ver1/ver1-step1-results.md`) created and updated with summary, evidence, risks, and checklist. |
Step 1 is complete and ready to hand off to Ver1 Step 2.
+309
View File
@@ -0,0 +1,309 @@
# Step 1 Implementation Plan: Architecture Consolidation
## Purpose
Align the implemented MVP codebase with the production architecture and V1 constraints documented in:
- `docs/architecture.md`
- `docs/requirements.md`
- `docs/error_handling.md`
- `docs/index.md`
- `docs/intent.md`
- `docs/ver1/ver1.md` (Step 1)
This step hardens architecture boundaries and ownership without expanding product scope.
---
## MCP Skill and Guide Inputs Incorporated
This plan explicitly incorporates patterns and guardrails from john-stream-mcp resources:
1. `resource://skills/fastapi-uv-docker/document`
- App factory and lifespan ownership
- Health endpoint and cloud-native baseline expectations
- Environment-driven configuration and startup discipline
2. `resource://skills/fastapi-async-sqlalchemy-modernization/document`
- Current-state gap audit first
- Target runtime model before refactor
- Explicit resource lifecycle ownership
- Transaction/session boundary clarity
- Phased migration with rollback points
3. `resource://skills/nicegui/document`
- Clear dependency direction
- UI/page registration as composition, not business logic container
- Async responsiveness and boundary separation
4. `resource://prompts/greenfield-architecture/document`
- Pattern-comparison-first planning
- Explicit tradeoffs and staged implementation
- Output contract with risks, open questions, and next steps
---
## Current-State Gap Summary (Architecture vs Implementation)
Based on docs and current `src/transcription` code:
1. **REQ-7 gap (lifespan-owned resources)**
- DB engine/session factory are module globals in `db.py`, not app lifespan-owned.
- Worker thread lifecycle is owned by lifespan (good), but DB/provider resource ownership is mixed.
2. **REQ-10 gap (explicit opt-in schema bootstrap)**
- `create_all()` is executed unconditionally on startup in `app.py`.
3. **Data store target gap (REQ-9 + architecture baseline)**
- Runtime still defaults to SQLite MVP setup; production architecture targets PostgreSQL baseline with optional MongoDB.
4. **Layering clarity gap (architecture layer model)**
- Boundaries exist but are not yet formally enforced (interface/app/domain/infra dependency rules are implicit, not codified).
5. **Decision record gap**
- No ADR set documenting key V1 architectural decisions and deviations from MVP.
---
## Scope for Step 1
### In scope
1. Produce architecture conformance audit and decision records.
2. Define and implement target runtime ownership model for core resources.
3. Establish explicit schema bootstrap policy (opt-in in production paths).
4. Consolidate module boundaries and dependency direction rules.
5. Update architecture docs to reflect implemented reality and V1 trajectory.
### Out of scope
- Full async SQLAlchemy rewrite (plan and seams only if deferred)
- MongoDB feature implementation
- New user-facing features
- Major worker architecture replacement (in-process worker remains baseline)
---
## Target Architecture Decisions for V1
1. **Keep modular monolith topology** (FastAPI + NiceGUI + in-process worker).
2. **Preserve container-light simplicity guardrails** from `architecture.md`.
3. **Move runtime ownership to lifespan** for:
- DB engine/session factory lifecycle
- Worker runtime resources
- Provider client factory/config lifecycle
4. **Adopt explicit schema bootstrap policy**:
- Dev/test: opt-in auto-bootstrap allowed
- Production: startup must not mutate schema implicitly
5. **Formalize boundary map**:
- Interface (`api`, `ui`) -> Application (`services`) -> Domain (`models/rules`) -> Infrastructure (`db`, `providers`)
- No reverse imports
---
## Detailed Work Breakdown
## Phase A — Architecture Audit and Baseline Freeze
- [ ] **A1. Produce architecture conformance matrix**
- Map each architecture section to current modules/files.
- Classify each row: `aligned`, `partial`, `not aligned`.
- [ ] **A2. Produce REQ-7/REQ-9/REQ-10 focused gap report**
- Explicitly capture current vs required state.
- Include operational risk if left unresolved.
- [ ] **A3. Freeze MVP architecture baseline**
- Record current baseline behavior and known temporary shortcuts.
- Link this baseline from `docs/ver1/ver1.md`.
### Deliverables
- `docs/ver1/ver1-step1-audit.md` (or equivalent section in this doc)
- Architecture conformance table
### Exit Criteria
- No architecture changes begin before gap matrix and baseline are approved.
---
## Phase B — Resource Ownership Consolidation (Lifespan-Centric)
- [ ] **B1. Define runtime resource ownership contract**
- `app.py` lifespan owns resource initialization and cleanup order.
- `app.state` carries resource handles/factories.
- No hidden module-global side-effect initialization for runtime resources.
- [ ] **B2. Refactor DB ownership model**
- Replace module-global engine singleton pattern with lifespan-initialized resource model.
- Define one canonical session-factory access path for app/worker/services.
- [ ] **B3. Normalize worker dependencies**
- Ensure worker uses lifespan-owned resources/factories rather than implicit globals.
- Preserve deterministic startup/shutdown behavior.
- [ ] **B4. Define provider adapter ownership**
- Provider client creation strategy is centralized and lifecycle-aware.
- Avoid per-call hidden client construction when unnecessary.
### MCP-Guided Guardrails
- Use explicit lifecycle composition patterns from `fastapi-async-sqlalchemy-modernization`.
- Maintain app-factory + lifespan structure per `fastapi-uv-docker`.
- Keep UI registration as composition only per `nicegui`.
### Exit Criteria
- Core runtime resources have one owner and one cleanup path.
- No critical resource has ambiguous ownership.
---
## Phase C — Schema Bootstrap Policy (REQ-10 Alignment)
- [ ] **C1. Define environment-aware bootstrap policy**
- `auto_create_schema` (or equivalent) disabled in production by default.
- Startup schema mutation is explicit and intentional.
- [ ] **C2. Split startup responsibilities**
- App startup performs health-critical initialization only.
- Schema bootstrap path is moved to explicit command/flag workflow.
- [ ] **C3. Update deployment/runbook docs**
- Document migration/bootstrap flow for dev, staging, prod.
- Ensure policy is testable and auditable.
### Exit Criteria
- Normal production startup path does not call schema auto-create implicitly.
- Bootstrap behavior is explicit and documented.
---
## Phase D — Module Boundary Enforcement
- [ ] **D1. Publish dependency direction rules**
- Allowed import directions across `api`, `ui`, `services`, `models/domain`, `db/providers`.
- Explicitly disallow reverse dependencies.
- [ ] **D2. Reconcile package map with docs**
- Ensure docs architecture elements match real package layout and naming.
- Update docs where intentional deviations remain.
- [ ] **D3. Isolate cross-layer responsibilities**
- Keep API/UI presentation concerns out of services.
- Keep provider/DB specifics out of interface layer.
- [ ] **D4. Add lightweight architecture checks**
- Add static/import checks and/or review checklist in CI/review process.
### Exit Criteria
- Boundary rules are documented and applied.
- Architectural drift can be detected during review/CI.
---
## Phase E — Architecture Decision Records (ADRs)
- [ ] **E1. Create ADR index**
- Add `docs/adr/README.md` with template and status model.
- [ ] **E2. Record minimum V1 ADR set**
1. Runtime ownership model (lifespan-owned resources)
2. Schema bootstrap policy (explicit vs implicit)
3. Persistence baseline (PostgreSQL target; SQLite transition strategy)
4. Worker topology (in-process for V1, extension path preserved)
- [ ] **E3. Cross-link ADRs**
- Link from architecture and V1 docs.
### Exit Criteria
- Major architecture decisions are explicit, versioned, and discoverable.
---
## Phase F — Documentation Consolidation
- [ ] **F1. Update `docs/architecture.md`**
- Reflect real implementation and V1 target state separately.
- Mark transitional choices clearly.
- [ ] **F2. Update `docs/index.md` navigation consistency**
- Ensure architecture/readme references match actual docs/files.
- [ ] **F3. Update `docs/requirements.md` traceability notes**
- Mark REQ-7/REQ-10 status and verification approach after consolidation.
- [ ] **F4. Add Step 1 result summary**
- Create `docs/ver1/ver1-step1-results.md` after implementation.
### Exit Criteria
- Docs are internally consistent and match runtime architecture reality.
---
## Verification Plan
## Architecture Verification Matrix (Step 1)
1. **Inspection**
- Resource ownership map exists and matches code.
- Schema bootstrap policy is explicit and environment-aware.
- ADRs exist for each key architecture decision.
2. **Automated checks**
- Existing test suite remains green.
- New/updated tests validate startup policy (no implicit schema mutation in production mode).
- Import/dependency-direction checks pass (if introduced).
3. **Demonstration**
- App starts in dev mode with explicit expected behavior.
- App starts in production mode without mutating schema implicitly.
- Worker lifecycle starts/stops cleanly with app lifespan.
---
## Risks and Mitigations
1. **Risk:** Refactor destabilizes MVP behavior
**Mitigation:** Phase changes with small PRs and regression checks after each phase.
2. **Risk:** Over-rotation into premature async rewrite
**Mitigation:** Keep this step focused on lifecycle ownership and boundaries; defer full async migration unless required.
3. **Risk:** Schema policy changes break local DX
**Mitigation:** Keep explicit dev bootstrap path simple and documented.
4. **Risk:** Boundary rules become “doc only”
**Mitigation:** Add CI/review enforcement and architecture checklist.
---
## Recommended Implementation Order
1. Phase A — Audit and baseline freeze
2. Phase B — Resource ownership consolidation
3. Phase C — Schema bootstrap policy
4. Phase D — Boundary enforcement
5. Phase E — ADR authoring
6. Phase F — Documentation consolidation
This order minimizes risk: diagnose first, then refactor ownership, then lock policy, then enforce boundaries, and finally finalize docs.
---
## Step 1 Completion Checklist
- [ ] Architecture conformance matrix approved.
- [ ] REQ-7 ownership gaps resolved or explicitly deferred with owner/date.
- [ ] REQ-10 explicit bootstrap policy implemented and verified.
- [ ] Dependency direction rules documented and enforced.
- [ ] ADR set created for all major Step 1 decisions.
- [ ] Architecture and index docs updated to match implementation.
- [ ] Full test suite passes after consolidation.
- [ ] `docs/ver1/ver1-step1-results.md` created with evidence and residual risks.
---
## Handoff to Step 2
Once Step 1 completes, Step 2 (Error Handling & Reliability Hardening) can proceed on stable architecture seams:
- consistent lifecycle ownership,
- explicit startup policy,
- clear module boundaries,
- documented architecture decisions.
+80
View File
@@ -0,0 +1,80 @@
# Ver1 Step 2 Results: Error Handling & Reliability Hardening
## Summary
Step 2 implementation is complete for the planned reliability and error-handling hardening scope:
1. Worker retries are now explicit, bounded, and category-driven.
2. Error behavior is more consistent across worker/API/UI boundaries.
3. Logging now includes stronger boundary context in key failure paths.
4. Test coverage was expanded for retry policy and new reliability settings.
## Implemented Changes
### 1) Worker retry policy and terminal behavior
- Updated `src/transcription/models.py`:
- Added `Job.retry_count` with default `0`.
- Updated `src/transcription/config.py`:
- Added `worker_max_retries`.
- Added `worker_retry_backoff_seconds`.
- Updated `src/transcription/worker.py`:
- Added bounded retry decision path (`_should_retry`).
- Added requeue behavior (`_requeue_for_retry`) for retriable errors.
- Added deterministic terminal failure behavior (`_finalize_failed_job`).
- Preserved transcript failure detail persistence (`error_id`, `category`, suggestion).
### 2) API fallback normalization hardening
- Updated `src/transcription/api/errors.py`:
- Fallback handler now emits safe generic internal message for unhandled exceptions.
- Added structured boundary logging fields including operation and exception type.
### 3) UI interaction reliability guard
- Updated `src/transcription/ui/upload_page.py`:
- Added duplicate in-flight submission guard to prevent repeated upload handling while busy.
### 4) Observability/logging improvements
- Updated worker logs in `src/transcription/worker.py` to include operation and domain identifiers in key transitions:
- pick
- retry
- transcribed
- failed
## Test Coverage Added/Updated
- Updated `tests/test_models.py`:
- Assert `retry_count` default.
- Updated `tests/test_config.py`:
- Added worker retry settings default test.
- Updated `tests/services/test_worker.py`:
- Added retriable requeue test.
- Added retry-exhaustion terminal failure test.
- Updated existing tests for settings-driven worker behavior.
- Existing API error tests remained green with fallback behavior updates:
- `tests/api/test_error_responses.py`
## Verification Evidence
Executed and passing:
- `uv run pytest tests/services/test_worker.py tests/test_models.py tests/test_config.py tests/api/test_error_responses.py -q`
- `uv run pytest -q`
## Residual Risks / Follow-ups
1. Retry policy currently uses simple fixed backoff; richer strategy (exponential/jitter) can be added in later hardening.
2. Full cross-layer structured logging standardization can be expanded in Step 6 observability work.
3. A formal Step 2 error-path inventory artifact (`ver1-step2-audit.md`) is still recommended for governance completeness.
## Step 2 Exit Assessment
- Error taxonomy and envelope stability: **met**
- Bounded retry and terminal failure behavior: **met**
- Worker reliability controls: **met**
- UI interaction hardening for duplicate actions: **met**
- Test coverage expansion and full-suite regression safety: **met**
Step 2 is complete and ready to hand off to Ver1 Step 3.
+302
View File
@@ -0,0 +1,302 @@
# Step 2 Implementation Plan: Error Handling & Reliability Hardening
## Purpose
Implement **Ver1 Step 2** from `docs/ver1/ver1.md` by standardizing failure behavior and reliability controls so the system fails safely, predictably, and transparently across UI, API, services, worker, and provider boundaries.
Primary governing docs:
- `docs/error_handling.md` (authoritative contract)
- `docs/requirements.md` (REQ-2, REQ-3, REQ-4, REQ-5, REQ-6)
- `docs/architecture.md` (boundary ownership and worker lifecycle)
- `docs/ver1/ver1.md` (Step 2 objective)
---
## MCP Skill and Guide Inputs Incorporated
This plan integrates guidance from john-stream-mcp resources:
1. `resource://skills/python-logging-dictconfig/document`
- centralized `dictConfig` logging
- startup-only configuration
- stable named loggers and boundary-level logging discipline
2. `resource://skills/pytesting/document`
- deterministic, behavior-first tests
- explicit marker usage and fast/slow lane discipline
- integration checks for boundary behavior and error contracts
3. `resource://skills/fastapi-async-sqlalchemy-modernization/document`
- classify at source boundary
- explicit transaction/session behavior under failure
- phased rollout with quality gates and rollback awareness
4. `resource://skills/nicegui-ui-customization/document`
- explicit user-facing error feedback for each interaction
- prevent duplicate actions during in-flight operations
- preserve one-way dependency boundaries from UI -> services
5. `resource://skills/fastapi-uv-docker/document` (applied selectively)
- lifespan-safe startup/shutdown behavior
- health/readiness posture and cloud-native operational checks
---
## Current-State Gap Summary
The project already has a strong baseline (`AppError`, taxonomy enum, API envelope, worker persistence), but Step 2 needs completion-level hardening:
1. **Error contract consistency**
- API envelope exists, but consistency must be verified for all error pathways.
2. **Cross-boundary category normalization**
- Provider/service/worker mappings exist, but require stricter policy checks and tests.
3. **Retry policy implementation depth**
- Step 2 requires bounded retry policy and clear terminal behavior for retriable failures.
4. **Operational traceability**
- Logging exists; Step 2 requires consistent structured fields at critical boundaries.
5. **UI failure UX consistency**
- UI error handling exists; Step 2 requires explicit contract coverage and anti-duplication safeguards.
---
## Scope for Step 2
### In scope
1. Enforce canonical error taxonomy and envelope across all boundaries.
2. Standardize logging fields and boundary-level error traceability.
3. Implement/complete bounded retry and terminal failure behavior in worker paths.
4. Improve UI/API error presentation consistency and actionable guidance.
5. Add comprehensive Step 2 test coverage and verification matrix.
6. Update documentation to reflect final Step 2 policies and behavior.
### Out of scope
- Major architecture/topology changes (external queue, distributed worker)
- New end-user feature expansion outside reliability/error handling
- Full async ORM migration (unless required by bug fix)
---
## Target Decisions for Step 2
1. **Taxonomy stability is mandatory**
- `ErrorCategory` values remain stable contract identifiers.
2. **Classification occurs at source boundary**
- adapters/services normalize early; UI/API only present safely.
3. **User safety over internal detail leakage**
- expose safe message + suggestion + error_id; keep sensitive detail in logs.
4. **Retry is explicit and bounded**
- only retriable categories may retry; retries are capped; terminal failures persist reason.
5. **Boundary logs carry correlation fields**
- include `error_id`, `category`, `operation`, and domain identifiers where available.
---
## Detailed Work Breakdown
## Phase A — Error Contract Audit and Policy Lock
- [ ] **A1. Build error-path inventory**
- Enumerate all failure entry points across:
- `api/`
- `ui/`
- `services/`
- `worker.py`
- `providers/`
- [ ] **A2. Produce taxonomy mapping table**
- For each known exception path, map:
- source exception type
- target `ErrorCategory`
- retriable flag
- API status (if exposed)
- [ ] **A3. Reconcile with `docs/error_handling.md`**
- Resolve any mismatch in category semantics, status codes, or suggested actions.
### Deliverables
- `docs/ver1/ver1-step2-audit.md` (recommended)
- taxonomy mapping table
### Exit Criteria
- Every known failure path has explicit category + retriable policy.
---
## Phase B — API and Service Contract Hardening
- [ ] **B1. Enforce API envelope completeness**
- Ensure all API errors return:
- `error_id`, `category`, `message`, `suggestion`, `timestamp`
- [ ] **B2. Verify category-to-status mapping consistency**
- Confirm `api/errors.py` matches `docs/error_handling.md` mapping guidance.
- [ ] **B3. Normalize service exceptions at boundary**
- Services should raise `AppError` subclasses for known failures.
- Unknown exceptions must become `internal_unexpected_error` with traceable `error_id`.
- [ ] **B4. Ensure safe detail handling**
- API/UI messages remain safe.
- Diagnostic context remains in logs/persisted failure detail where appropriate.
### Exit Criteria
- No unstructured/unclassified exception escapes core boundaries.
- API responses are contract-stable for all tested failure modes.
---
## Phase C — Worker Retry and Terminal Failure Policy
- [ ] **C1. Define bounded retry policy**
- Add configurable retry settings (attempt limit/backoff policy).
- Limit retries to retriable categories.
- [ ] **C2. Implement terminal failure persistence**
- On retry exhaustion, persist clear terminal reason and `error_id`.
- Ensure job status transitions end deterministically at `failed`.
- [ ] **C3. Add duplicate-processing safety checks**
- Prevent duplicate terminal updates when job already resolved.
- [ ] **C4. Validate worker lifecycle under repeated transient failures**
- Ensure loop remains stable and responsive.
### Exit Criteria
- Retries are bounded and policy-driven.
- Exhausted retries produce deterministic failed state with evidence.
---
## Phase D — Logging and Observability Contract Enforcement
- [ ] **D1. Central logging conformance check**
- Confirm startup-only `dictConfig` use remains canonical.
- No module-level `basicConfig` use.
- [ ] **D2. Standardize error log fields**
- Require at minimum when available:
- `error_id`, `category`, `operation`, `exception_type`, `job_id`, `document_id`
- [ ] **D3. Boundary handoff logging**
- Add/normalize logs at transitions:
- UI action -> service
- service -> provider/db
- worker pickup -> terminal state
- [ ] **D4. Log noise control**
- Avoid duplicate stack-trace logging across layers for same exception.
### Exit Criteria
- Critical failure events are traceable end-to-end via logs and `error_id`.
---
## Phase E — UI Error UX Consistency and Interaction Hardening
- [ ] **E1. Standardize user error presentation**
- For upload/jobs interactions, ensure:
- clear title
- plain-language message
- suggested action
- visible error reference id
- [ ] **E2. Add in-flight interaction guards**
- Prevent duplicate submits/click storms during pending operations.
- [ ] **E3. Ensure deterministic UI state recovery**
- controls re-enable after failure
- status text remains actionable
- [ ] **E4. Keep UI boundary clean**
- no provider/protocol details leaked into page modules
### Exit Criteria
- All primary UI actions have consistent success/failure interaction behavior.
---
## Phase F — Test Expansion and Verification
Apply pytesting guidance: behavior-first assertions, deterministic fixtures, strict markers.
- [ ] **F1. API error contract tests**
- verify envelope fields and status mapping for each category class.
- [ ] **F2. Service classification tests**
- verify known failures map to expected `AppError` subclasses/categories.
- [ ] **F3. Worker retry policy tests**
- retriable failure retries and eventual success
- retriable failure exhaustion -> terminal failed
- non-retriable failure -> immediate failed
- [ ] **F4. UI error behavior tests**
- upload/jobs actions show actionable feedback on failures
- duplicate action guard behavior
- [ ] **F5. Regression guard tests**
- at least one test per previously observed production/real-world failure mode
### Validation Commands
- `uv run pytest --collect-only -q`
- `uv run pytest -m unit -q`
- `uv run pytest -m "not external" -q`
- `uv run pytest -q`
### Exit Criteria
- All Step 2 reliability/error contract tests pass.
- Existing suite remains green.
---
## Recommended Implementation Order
1. Phase A — audit and policy lock
2. Phase B — API/service contract hardening
3. Phase C — worker retry and terminal policy
4. Phase D — logging/traceability normalization
5. Phase E — UI consistency hardening
6. Phase F — test expansion and full verification
This order reduces risk by locking policy first, then applying behavior changes at core boundaries before UI polish.
---
## Risks and Mitigations
1. **Risk:** Overly broad retry policy causes hidden failure loops
**Mitigation:** strict category-based retry eligibility + hard cap + terminal persistence.
2. **Risk:** User-facing messages become too technical
**Mitigation:** enforce safe message + suggestion contract in tests.
3. **Risk:** Logging becomes noisy/redundant
**Mitigation:** boundary logging rules and single-trace ownership.
4. **Risk:** Reliability work introduces regressions in happy path
**Mitigation:** run full suite continuously; preserve integration pipeline tests.
---
## Step 2 Completion Checklist
- [ ] Error taxonomy mapping table completed and approved.
- [ ] API envelope and HTTP status behavior verified for all relevant failure categories.
- [ ] Service/provider exception normalization is consistent and tested.
- [ ] Worker retry behavior is bounded, explicit, and terminal-state safe.
- [ ] Structured error logging fields are present at boundary handoffs.
- [ ] UI failure flows provide clear, actionable, and traceable feedback.
- [ ] Full test suite passes with new Step 2 coverage included.
- [ ] `docs/ver1/ver1-step2-results.md` created with evidence and residual risks.
---
## Handoff to Step 3
After Step 2 completion, Step 3 (Functional Completion by Requirement Domain) proceeds on a hardened foundation:
- stable failure contracts,
- predictable retries and terminal behavior,
- actionable user/API error semantics,
- improved diagnostic traceability.
+130 -130
View File
@@ -1,40 +1,39 @@
# Version 1 Implementation Plan # Version 1 Implementation Plan
This plan defines the path from MVP to **Version 1 complete**. This plan defines the path from MVP to **Version 1 complete**.
The objective is to deliver the full scoped product with production readiness, while explicitly separating refinements/enhancements into a future document. The objective is to deliver the full scoped product with readiness for reliable personal-scale operation, while explicitly separating refinements/enhancements into a future document.
--- ---
## 0) Plan Governance & Scope Control (Foundation) ## 0) Plan Governance & Scope Control (Foundation)
**Goal:** Keep execution focused on V1 completion, not optimization/perfection. **Goal:** Keep execution focused on V1 completion and avoid unnecessary process overhead.
### Implementation Steps ### Implementation Steps
1. Create and maintain a **V1 Traceability Matrix**: 1. Create and maintain a **V1 Traceability Matrix**:
- Requirement ID - Requirement ID
- Current status (`done`, `partial`, `not started`) - Current status (`done`, `partial`, `not started`)
- Owner
- Validation method - Validation method
2. Define V1 completion gates: 2. Define V1 completion gates:
- Functional complete - Functional complete
- Operationally complete - Operationally complete
- Production-ready complete - Personal-deployment ready
3. Snapshot the MVP baseline (tag/changelog reference). 3. Snapshot the MVP baseline (tag/changelog reference).
4. Create a standing rule: any non-V1 idea is logged to a separate enhancements backlog document (to be named later), not added to active V1 scope unless explicitly approved. 4. Keep a standing rule: non-V1 ideas go to a separate enhancements backlog, and enter V1 only by explicit approval.
### Deliverables ### Deliverables
- `docs/ver1/ver1.md` (this plan) - `docs/ver1/ver1.md` (this plan)
- V1 traceability artifact (linked from here when created) - V1 traceability artifact (linked from here when created)
### Exit Criteria ### Exit Criteria
- Every in-scope requirement has explicit ownership and status. - Every in-scope requirement has explicit status and validation evidence.
- Scope-change process is agreed and followed. - Scope-change discipline is followed consistently.
--- ---
## 1) Architecture Consolidation ## 1) Architecture Consolidation
**Goal:** Align implementation with the intended architecture and reduce MVP shortcuts. **Goal:** Align implementation with intended architecture while preserving simplicity.
### Implementation Steps ### Implementation Steps
1. Compare implemented modules/components with architecture documentation. 1. Compare implemented modules/components with architecture documentation.
@@ -42,201 +41,202 @@ The objective is to deliver the full scoped product with production readiness, w
- Temporary coupling - Temporary coupling
- Missing interfaces - Missing interfaces
- Placeholder services/components - Placeholder services/components
3. Resolve high-risk architectural gaps first. 3. Resolve architecture gaps that threaten reliability, maintainability, or clear boundaries.
4. Record key decisions and tradeoffs in ADRs. 4. Record material decisions and tradeoffs in ADRs.
### Deliverables ### Deliverables
- Updated architecture diagrams and boundaries - Updated architecture diagrams and boundaries
- ADR entries for major decisions - ADR entries for material decisions
### Exit Criteria ### Exit Criteria
- Architecture documentation reflects system reality. - Architecture documentation reflects system reality.
- Critical architecture risks are addressed or scheduled with owners/dates. - High-impact architecture risks are addressed or explicitly scheduled.
--- ---
## 2) Error Handling & Reliability Hardening ## 2) Error Handling & Reliability Hardening
**Goal:** Ensure predictable, safe behavior under failure conditions. **Goal:** Ensure predictable, diagnosable behavior under expected failure conditions.
### Implementation Steps ### Implementation Steps
1. Standardize error taxonomy and envelope format across all layers. 1. Apply the canonical taxonomy and response model from `docs/error_handling.md` across UI/API/service/worker boundaries.
2. Ensure clear distinction between: 2. Ensure clear distinction between:
- User-facing errors - User-facing safe messages
- Internal/system errors - Internal diagnostic detail
- Retryable vs non-retryable failures - Retryable vs non-retryable failures
3. Add resilience controls where needed: 3. Implement practical resilience controls where needed:
- Timeouts - Timeouts
- Retries with backoff - Bounded retries with backoff
- Circuit breaking / fallback logic - Explicit terminal failure states
4. Add failure-path tests for critical workflows. 4. Add failure-path tests for critical workflows.
### Deliverables ### Deliverables
- Error code catalog/reference - Error handling reference aligned with `docs/error_handling.md`
- Failure mode test coverage for critical paths - Failure-mode test coverage for critical paths
### Exit Criteria ### Exit Criteria
- Error behavior is consistent across major flows. - Error behavior is consistent across major flows.
- Known failure scenarios are tested and pass. - Known failure scenarios are tested and pass.
- Failed jobs include actionable, traceable failure detail.
--- ---
## 3) Functional Completion by Requirement Domain ## 3) Functional Completion by Requirement Domain
**Goal:** Complete all V1 functional requirements in a risk-aware order. **Goal:** Complete all V1 requirements in a practical, user-first order.
### Recommended Order ### Recommended Order
1. Business-critical end-user flows 1. End-user core flows (upload → transcribe → review)
2. Data integrity and consistency capabilities 2. Data integrity and persistence behavior
3. Admin/operational controls 3. Minimal operator controls needed for personal use
4. Lower-priority UX and quality-of-life items that are in V1 scope 4. In-scope UX quality improvements
### Implementation Steps ### Implementation Steps
For each requirement slice: For each requirement slice:
1. Finalize contract/schema 1. Confirm contract/schema
2. Implement domain logic 2. Implement service/domain logic
3. Implement persistence/state changes 3. Implement persistence/state transitions
4. Integrate API/UI 4. Integrate API/UI behavior
5. Add automated tests 5. Add or update automated tests
6. Update docs 6. Update relevant docs
### Deliverables ### Deliverables
- Requirement completion report with validation evidence - Requirement completion report with validation evidence linked to REQ IDs
### Exit Criteria ### Exit Criteria
- All V1 must-have requirements are complete and validated. - All V1 must-have requirements are complete and verified.
--- ---
## 4) Data Model, Migration, and Backfill Safety ## 4) Data Model and Migration Safety
**Goal:** Ensure data model and migrations are production-safe. **Goal:** Keep schema evolution safe and simple for personal-scale deployment.
### Implementation Steps ### Implementation Steps
1. Validate schema against final V1 domain needs. 1. Validate schema against finalized V1 domain needs.
2. Implement forward-safe migrations. 2. Implement forward-safe migrations for expected upgrades.
3. Define rollback/mitigation plans for migration failures. 3. Define a simple rollback/mitigation path for migration failures.
4. Build and verify backfill scripts (if needed). 4. Add backfill scripts only where truly required.
5. Add migration rehearsal in staging with representative data. 5. Rehearse migration + rollback locally using representative sample data.
### Deliverables ### Deliverables
- Migration runbook - Migration and rollback runbook
- Backfill verification checklist - Backfill checklist (if applicable)
### Exit Criteria ### Exit Criteria
- Migration plan validated in staging. - Migration path is tested and documented.
- No unresolved data-loss risk for V1 rollout. - No unresolved data-loss risk for V1 upgrade.
--- ---
## 5) Security, Access Control, and Compliance Baseline ## 5) Private-Network Safety Baseline
**Goal:** Close MVP security gaps and establish V1 baseline controls. **Goal:** Apply right-sized security controls for a single-user system on a trusted private network.
### Implementation Steps ### Implementation Steps
1. Complete authn/authz coverage for all routes/actions. 1. Enforce private-network deployment assumptions in docs and configuration.
2. Enforce input validation and output sanitization. 2. Ensure basic single-operator access control for UI/API actions.
3. Verify secret management and credential rotation process. 3. Enforce input validation and safe error output behavior.
4. Add audit logging for sensitive operations. 4. Keep secrets out of source control; document local secret handling.
5. Run dependency/security scanning in CI and remediate findings. 5. Run lightweight dependency/security scanning and resolve high-risk findings.
### Deliverables ### Deliverables
- Security checklist with status - Security assumptions checklist (private network, single operator)
- Threat/risk update for V1 scope - Basic risk update for V1 scope
### Exit Criteria ### Exit Criteria
- No unresolved critical/high vulnerabilities for V1 launch. - No unresolved critical vulnerabilities.
- Access control behavior verified by tests. - Access behavior and validation rules are verified for intended operating model.
--- ---
## 6) Observability & Operability ## 6) Minimal Observability & Operability
**Goal:** Make system behavior observable and supportable in production. **Goal:** Keep operation and troubleshooting simple, clear, and reliable.
### Implementation Steps ### Implementation Steps
1. Standardize structured logging and correlation IDs. 1. Standardize structured logging across UI/API/service/worker boundaries.
2. Add core metrics: 2. Ensure logged errors include category and error reference IDs per `error_handling.md`.
- Latency 3. Add lightweight health/startup checks.
- Throughput 4. Document a concise operator runbook:
- Error rates - start/stop
- Resource saturation - log locations
3. Add tracing for critical request/workflow paths. - common failure patterns and recovery steps
4. Define SLOs/SLIs and alert thresholds. 5. Add minimal counters/timings only where they clearly improve diagnosis.
5. Prepare incident response and rollback runbooks.
### Deliverables ### Deliverables
- Dashboards and alerts - Logging and error-traceability baseline
- Operations runbooks - Operator runbook
### Exit Criteria ### Exit Criteria
- Team can detect, triage, and remediate incidents quickly. - Operator can diagnose common failures using logs + runbook.
- Core production signals are available and reliable. - System recovery procedures are documented and repeatable.
--- ---
## 7) Test Strategy Expansion & Quality Gates ## 7) Test Coverage and Practical Quality Gates
**Goal:** Raise confidence for repeatable, low-risk releases. **Goal:** Prevent regressions in critical flows without overbuilding test infrastructure.
### Implementation Steps ### Implementation Steps
1. Expand unit and integration tests across V1 features. 1. Expand unit and integration tests for all V1 requirement slices.
2. Add contract tests between key components/services. 2. Add end-to-end tests for critical journeys:
3. Add end-to-end tests for critical user journeys. - upload
4. Add non-functional tests where relevant: - process/transcribe
- Performance/load - view result
- Soak - failure visibility
- Failure-injection scenarios 3. Add targeted contract tests where adapter boundaries are error-prone.
5. Enforce CI quality gates (tests, lint, type checks, security scans). 4. Keep CI gates focused on high-value checks (tests, lint, type checks, dependency scan).
### Deliverables ### Deliverables
- Test matrix with ownership - V1 test matrix mapped to requirements and critical flows
- CI gate definition and thresholds - CI quality-gate checklist
### Exit Criteria ### Exit Criteria
- Critical-path regressions are blocked automatically. - Critical-path regressions are automatically detected.
- Test coverage and reliability thresholds meet V1 targets. - Test suite gives consistent release confidence for personal-scale operation.
--- ---
## 8) Performance & Scalability Validation ## 8) Performance Validation for Personal Scale
**Goal:** Meet expected V1 performance at projected load. **Goal:** Confirm acceptable responsiveness for expected personal-use workload.
### Implementation Steps ### Implementation Steps
1. Define performance budgets per key flow. 1. Define practical performance expectations for key flows.
2. Benchmark current behavior in staging. 2. Run representative tests using real document samples.
3. Optimize bottlenecks (queries, caching, concurrency, etc.). 3. Address obvious bottlenecks in queries, file handling, or worker concurrency.
4. Re-test after each optimization and compare against budget. 4. Document known limits and expected operating bounds.
5. Document known limits and safe operating bounds.
### Deliverables ### Deliverables
- Performance benchmark report - Short performance validation note
- Optimization log - Known-limits summary
### Exit Criteria ### Exit Criteria
- V1 performance targets met for expected usage profile. - Core flows remain responsive for expected corpus size and usage patterns.
--- ---
## 9) Release Engineering & Environment Readiness ## 9) Release Readiness and Environment Simplicity
**Goal:** Make deployment repeatable, controlled, and reversible. **Goal:** Make deployment and rollback repeatable for a single-operator Docker Compose setup.
### Implementation Steps ### Implementation Steps
1. Harden CI/CD pipeline with clear promotion gates. 1. Define a simple release checklist:
2. Ensure config parity and consistency across environments. - run tests
3. Define rollout strategy (phased/canary/limited release as applicable). - run one end-to-end transcription check
4. Validate rollback procedures in staging. - verify migration compatibility
5. Produce release checklist and ownership model. 2. Document environment configuration requirements clearly.
3. Validate deployment and rollback steps in a local rehearsal.
4. Add backup/restore verification for core persisted data.
### Deliverables ### Deliverables
- Release playbook - Release checklist
- Environment readiness checklist - Environment and rollback guide
### Exit Criteria ### Exit Criteria
- Deployment and rollback are rehearsed and reliable. - Deployment/rollback is rehearsed and documented.
- Release process is executable without tribal knowledge. - Operator can release safely without hidden steps.
--- ---
@@ -252,7 +252,7 @@ For each requirement slice:
- Index/navigation - Index/navigation
- Intent alignment summary - Intent alignment summary
2. Add operator troubleshooting guides. 2. Add operator troubleshooting guides.
3. Add integration/API examples for consumers. 3. Add integration/API examples for the operator and future maintainers.
4. Publish changelog/version notes for V1. 4. Publish changelog/version notes for V1.
### Deliverables ### Deliverables
@@ -260,55 +260,55 @@ For each requirement slice:
- V1 release notes - V1 release notes
### Exit Criteria ### Exit Criteria
- A new team member can run/support the system using docs alone. - A future maintainer can run and support the system using docs alone.
--- ---
## 11) Final Validation, UAT, and Launch ## 11) Final Validation and Launch
**Goal:** Confirm readiness and launch V1 safely. **Goal:** Confirm V1 readiness and launch with low operational risk.
### Implementation Steps ### Implementation Steps
1. Run full-system acceptance validation against the V1 traceability matrix. 1. Run end-to-end acceptance validation against the V1 traceability matrix.
2. Conduct stakeholder UAT and capture sign-off. 2. Complete operator acceptance checks on representative real documents.
3. Execute production readiness review. 3. Execute launch checklist (including backup, migration, and rollback readiness).
4. Launch in controlled phases and monitor key signals. 4. Launch and monitor logs/status closely during initial use.
### Deliverables ### Deliverables
- UAT/PRR sign-off records - Acceptance validation record
- Launch checklist and monitoring plan - Launch checklist completion record
### Exit Criteria ### Exit Criteria
- Stakeholder approval achieved. - V1 requirements are validated.
- Launch metrics are stable within defined thresholds. - Initial launch behavior is stable and recoverable.
--- ---
## 12) Post-Launch Stabilization (3060 Days) ## 12) Post-Launch Stabilization
**Goal:** Consolidate V1 in production before major expansion. **Goal:** Address early issues quickly and lock in a reliable V1 baseline.
### Implementation Steps ### Implementation Steps
1. Track incidents, defects, and user feedback. 1. Track defects and operational pain points observed after launch.
2. Prioritize stabilization fixes with short cycle times. 2. Prioritize short-cycle stabilization fixes.
3. Remove temporary flags/mitigations introduced during launch. 3. Remove temporary launch-only workarounds when safe.
4. Produce post-launch retrospective and handoff to standard roadmap cadence. 4. Capture a brief retrospective and update the next-phase backlog.
### Deliverables ### Deliverables
- Stabilization report - Stabilization summary
- Prioritized backlog update - Updated backlog for post-V1 enhancements
### Exit Criteria ### Exit Criteria
- Incident/error rates converge to steady-state targets. - Major launch issues are resolved.
- V1 transitions from launch mode to normal operations. - System transitions to steady personal-use operation.
--- ---
## Recommended Execution Rhythm ## Recommended Execution Rhythm
- **Weekly:** Requirement closure + risk review - **Weekly:** Requirement closure + risk review
- **Biweekly:** Release train with quality gates - **As needed (small batch releases):** Run release checklist and deploy
- **Milestone reviews:** After phases 2, 6, 9, and 11 - **Milestone check-ins:** After phases 2, 6, 9, and 11
--- ---
+1 -12
View File
@@ -12,29 +12,18 @@ description = "Historical document transcription system"
readme = "README.md" readme = "README.md"
requires-python = ">=3.12" requires-python = ">=3.12"
dependencies = [ dependencies = [
"aiosqlite>=0.21.0",
"asyncpg>=0.31.0",
"fastapi>=0.138.0", "fastapi>=0.138.0",
"nicegui==3.13.0", "nicegui==3.13.0",
"openrouter>=0.7.0", "openrouter>=0.7.0",
"psycopg2-binary>=2.9.12",
"pydantic>=2.13.4", "pydantic>=2.13.4",
"pydantic-settings>=2.9.1", "pydantic-settings>=2.9.1",
"sqlmodel>=0.0.25", "sqlmodel>=0.0.25",
] ]
[project.optional-dependencies]
[dependency-groups]
dev = [ dev = [
"pytest>=8.0", "pytest>=8.0",
"pytest-asyncio>=0.25", "pytest-asyncio>=0.25",
"httpx2>=2.5.0",
"ipykernel>=7.3.0",
"ipywidgets>=8.1.8",
"pre-commit>=4.6.0",
"rich>=15.0.0",
"ruff>=0.15.20",
"ty>=0.0.54",
] ]
[tool.pytest.ini_options] [tool.pytest.ini_options]
-62
View File
@@ -1,62 +0,0 @@
line-length = 120
indent-width = 4
target-version = "py313"
exclude = [
".venv",
".devenv",
".git",
".vscode",
"build",
"site",
"__pycache__",
]
[lint]
preview = true
extend-select = [
"ARG", # https://docs.astral.sh/ruff/rules/#flake8-unused-arguments-arg
"B", # https://docs.astral.sh/ruff/rules/#flake8-bugbear-b
"C4", # https://docs.astral.sh/ruff/rules/#flake8-comprehensions-c4
"DOC102", # https://docs.astral.sh/ruff/rules/docstring-extraneous-parameter/
"DOC202", # https://docs.astral.sh/ruff/rules/docstring-extraneous-returns/
"DOC403", # https://docs.astral.sh/ruff/rules/docstring-extraneous-yields/
"DOC502", # https://docs.astral.sh/ruff/rules/docstring-extraneous-exception/
"E", "W", # https://docs.astral.sh/ruff/rules/#pycodestyle-e-w
"F", # https://docs.astral.sh/ruff/rules/#pyflakes-f
"FURB", # https://docs.astral.sh/ruff/rules/#refurb-furb
"I", # https://docs.astral.sh/ruff/rules/#isort-i
"N", # https://docs.astral.sh/ruff/rules/#pep8-naming-n
"PD", # https://docs.astral.sh/ruff/rules/#pandas-vet-pd
"PTH", # https://docs.astral.sh/ruff/rules/#flake8-use-pathlib-pth
"UP", # https://docs.astral.sh/ruff/rules/#pyupgrade-up
"SIM", # https://docs.astral.sh/ruff/rules/#flake8-simplify-sim
"PLR0202", # https://docs.astral.sh/ruff/rules/no-classmethod-decorator/
"PLR0203", # https://docs.astral.sh/ruff/rules/no-staticmethod-decorator/
"PLR0206", # https://docs.astral.sh/ruff/rules/property-with-parameters/
"PLR0915", # https://docs.astral.sh/ruff/rules/too-many-statements/
"PLR1702", # https://docs.astral.sh/ruff/rules/too-many-nested-blocks/
"TRY002",
]
extend-fixable = ["ALL"]
ignore = [
"UP046",
"UP047",
]
[lint.extend-per-file-ignores]
"*.ipynb" = [
"F401", # unused imports
"F841", # unused local variable
"F821", # undefined name in exploratory notebook cells
]
[lint.isort]
force-single-line = true
[format]
quote-style = "double"
indent-style = "space"
skip-magic-trailing-comma = false
line-ending = "auto"
+2 -5
View File
@@ -4,13 +4,10 @@ from __future__ import annotations
import logging import logging
from fastapi import FastAPI from fastapi import FastAPI, Request
from fastapi import Request
from fastapi.responses import JSONResponse from fastapi.responses import JSONResponse
from transcription.errors import AppError from transcription.errors import AppError, ErrorCategory, build_error_envelope
from transcription.errors import ErrorCategory
from transcription.errors import build_error_envelope
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
+49 -46
View File
@@ -2,74 +2,77 @@
from __future__ import annotations from __future__ import annotations
from contextlib import AsyncExitStack
from contextlib import asynccontextmanager from contextlib import asynccontextmanager
from threading import Event, Thread
from fastapi import FastAPI from fastapi import FastAPI
from fastapi import status
from fastapi.responses import RedirectResponse
from fastapi.staticfiles import StaticFiles
from .api.errors import register_error_handlers from transcription.api.errors import register_error_handlers
from .api.health import router as health_router from transcription.api.health import router as health_router
from .config import configure_logging from transcription.config import get_settings, setup_logging
from .config import get_settings from transcription.db import (
from .db import create_all create_all,
from .db import dispose_database_runtime dispose_database_runtime,
from .db import initialize_database_runtime initialize_database_runtime,
from .services import ServiceBundle should_bootstrap_schema,
from .ui import register_pages )
from .worker import worker_consumer_lifespan from transcription.ui import register_pages
from transcription.worker import run_worker_loop
def _start_worker(app: FastAPI) -> None:
stop_event = Event()
worker_thread = Thread(
target=run_worker_loop,
kwargs={
"engine": app.state.db_runtime.engine,
"stop_event": stop_event,
"poll_interval_seconds": 1.0,
},
daemon=True,
)
worker_thread.start()
app.state.worker_stop_event = stop_event
app.state.worker_thread = worker_thread
def _stop_worker(app: FastAPI) -> None:
stop_event = getattr(app.state, "worker_stop_event", None)
worker_thread = getattr(app.state, "worker_thread", None)
if stop_event is not None:
stop_event.set()
if worker_thread is not None:
worker_thread.join(timeout=2.0)
@asynccontextmanager @asynccontextmanager
async def _lifespan(app: FastAPI): async def _lifespan(app: FastAPI):
configure_logging() setup_logging()
settings = getattr(app.state, "settings", None) or get_settings() settings = get_settings()
app.state.settings = settings app.state.settings = settings
app.state.services = ServiceBundle() app.state.db_runtime = initialize_database_runtime(settings=settings)
app.state.runtime = initialize_database_runtime(settings=settings)
if settings.should_bootstrap_schema: if should_bootstrap_schema(settings):
await create_all(engine=app.state.runtime.engine) create_all(engine=app.state.db_runtime.engine)
settings.upload_dir.mkdir(parents=True, exist_ok=True) settings.upload_dir.mkdir(parents=True, exist_ok=True)
settings.prompt_dir.mkdir(parents=True, exist_ok=True) settings.prompt_dir.mkdir(parents=True, exist_ok=True)
async with AsyncExitStack() as stack: _start_worker(app)
stack.push_async_callback(dispose_database_runtime) try:
stop_event, worker_notifier = await stack.enter_async_context(
worker_consumer_lifespan(
session_factory=app.state.runtime.session_factory,
poll_interval_seconds=1.0,
)
)
app.state.worker_stop_event = stop_event
app.state.worker_notifier = worker_notifier
yield yield
finally:
_stop_worker(app)
dispose_database_runtime()
def create_app() -> FastAPI: def create_app() -> FastAPI:
"""Create and configure the FastAPI application.""" """Create and configure the FastAPI application."""
app = FastAPI(title="Transcription", lifespan=_lifespan) app = FastAPI(title="Transcription", lifespan=_lifespan)
settings = get_settings()
app.state.settings = settings
app.mount(
"/uploads",
StaticFiles(directory=settings.upload_dir, check_dir=False),
name="uploads",
)
@app.get("/", include_in_schema=False)
async def root_redirect() -> RedirectResponse:
return RedirectResponse(url="/ui", status_code=status.HTTP_307_TEMPORARY_REDIRECT)
@app.get("/ui", include_in_schema=False)
async def ui_redirect() -> RedirectResponse:
return RedirectResponse(url="/ui/upload", status_code=status.HTTP_307_TEMPORARY_REDIRECT)
register_error_handlers(app) register_error_handlers(app)
register_pages(app) register_pages(app)
app.include_router(health_router) app.include_router(health_router)
return app return app
-39
View File
@@ -1,39 +0,0 @@
"""Helpers for accessing lifespan-owned application state resources."""
from __future__ import annotations
from fastapi import FastAPI
from sqlalchemy.ext.asyncio import async_sessionmaker
from sqlmodel.ext.asyncio.session import AsyncSession
from transcription.db.runtime import DatabaseRuntime
from transcription.db.runtime import get_session_factory
from transcription.worker import WorkerNotifier
from transcription.worker import resolve_worker_notifier
def resolve_database_runtime(state: object) -> DatabaseRuntime | None:
"""Return database runtime from app-like state objects when available."""
runtime = getattr(state, "runtime", None)
return runtime if isinstance(runtime, DatabaseRuntime) else None
def require_database_runtime(state: object) -> DatabaseRuntime:
"""Return database runtime or raise when app lifespan has not initialized it."""
runtime = resolve_database_runtime(state)
if runtime is None:
raise RuntimeError("Database runtime is not initialized on application state")
return runtime
def resolve_session_factory(state: object) -> async_sessionmaker[AsyncSession]:
"""Return DB session factory from state when available, otherwise shared runtime."""
runtime = resolve_database_runtime(state)
if runtime is not None:
return runtime.session_factory
return get_session_factory()
def get_worker_notifier(app: FastAPI) -> WorkerNotifier:
"""Return app worker notifier, or a no-op fallback when unavailable."""
return resolve_worker_notifier(app.state)
+14 -34
View File
@@ -5,16 +5,14 @@ once at startup. Provider-specific defaults (model names, base URLs)
are resolved by the provider adapters, not here. are resolved by the provider adapters, not here.
""" """
import logging
import logging.config import logging.config
from contextvars import ContextVar
from enum import StrEnum from enum import StrEnum
from functools import lru_cache
from pathlib import Path from pathlib import Path
from typing import Literal from typing import Literal
from pydantic_settings import BaseSettings from pydantic_settings import BaseSettings, SettingsConfigDict
from pydantic_settings import SettingsConfigDict
logger = logging.getLogger(__name__)
class Provider(StrEnum): class Provider(StrEnum):
@@ -41,7 +39,6 @@ class Settings(BaseSettings):
# --- persistence --- # --- persistence ---
database_url: str = "sqlite:///./transcription.db" database_url: str = "sqlite:///./transcription.db"
bootstrap_schema_on_startup: bool | None = None bootstrap_schema_on_startup: bool | None = None
sqlite_check_same_thread: bool = False
# --- filesystem paths --- # --- filesystem paths ---
upload_dir: Path = Path("./uploads") upload_dir: Path = Path("./uploads")
@@ -51,31 +48,13 @@ class Settings(BaseSettings):
worker_max_retries: int = 0 worker_max_retries: int = 0
worker_retry_backoff_seconds: float = 0.0 worker_retry_backoff_seconds: float = 0.0
@property
def should_bootstrap_schema(self) -> bool:
"""Return whether startup should auto-create schema for this environment."""
if self.bootstrap_schema_on_startup is not None:
return self.bootstrap_schema_on_startup
return self.environment in {"development", "test"}
_settings: ContextVar[Settings | None] = ContextVar("settings", default=None)
def get_settings(**kwargs) -> Settings:
settings = _settings.get()
if settings is None:
settings = Settings(**kwargs) # pyright: ignore[reportCallIssue]
_settings.set(settings)
return settings
LOGGING_CONFIG: dict[str, object] = { LOGGING_CONFIG: dict[str, object] = {
"version": 1, "version": 1,
"disable_existing_loggers": False, "disable_existing_loggers": False,
"formatters": { "formatters": {
"standard": { "standard": {
"format": "%(asctime)s %(levelname)-8s | %(message)s", "format": "%(asctime)s | %(levelname)-8s | %(name)s | %(message)s",
"datefmt": "%Y-%m-%d %H:%M:%S", "datefmt": "%Y-%m-%d %H:%M:%S",
} }
}, },
@@ -90,17 +69,18 @@ LOGGING_CONFIG: dict[str, object] = {
"level": "INFO", "level": "INFO",
"handlers": ["console"], "handlers": ["console"],
}, },
"loggers": {
"transcription": {
"level": "DEBUG",
"handlers": ["console"],
"propagate": False,
}
},
} }
def configure_logging() -> None: @lru_cache(maxsize=1)
def get_settings() -> Settings:
"""Return the singleton Settings instance.
Cached so the entire application shares one validated config.
"""
return Settings()
def setup_logging() -> None:
"""Configure root logging once at startup.""" """Configure root logging once at startup."""
logging.config.dictConfig(LOGGING_CONFIG) logging.config.dictConfig(LOGGING_CONFIG)
logger.debug("Logging configured")
+116
View File
@@ -0,0 +1,116 @@
"""Database runtime ownership, schema bootstrap, and session access.
V1 moves database resource ownership to explicit runtime initialization so
startup/shutdown behavior is predictable and lifespan-managed.
"""
import contextlib
import logging
from collections.abc import Generator
from dataclasses import dataclass
from sqlalchemy import inspect, text
from sqlalchemy.engine import Engine
from sqlmodel import Session, SQLModel, create_engine
from transcription.config import Settings, get_settings
logger = logging.getLogger(__name__)
@dataclass(frozen=True)
class DatabaseRuntime:
"""Process-level database runtime resources."""
engine: Engine
_runtime: DatabaseRuntime | None = None
def _build_engine(settings: Settings) -> Engine:
connect_args: dict[str, object] = {}
if settings.database_url.startswith("sqlite"):
connect_args["check_same_thread"] = False
return create_engine(
settings.database_url,
echo=False,
connect_args=connect_args,
)
def initialize_database_runtime(*, settings: Settings | None = None) -> DatabaseRuntime:
"""Initialize and cache the process database runtime once."""
global _runtime
if _runtime is not None:
return _runtime
runtime_settings = settings or get_settings()
_runtime = DatabaseRuntime(engine=_build_engine(runtime_settings))
return _runtime
def get_database_runtime() -> DatabaseRuntime:
"""Return initialized database runtime, creating it if needed."""
if _runtime is None:
return initialize_database_runtime()
return _runtime
def dispose_database_runtime() -> None:
"""Dispose process database runtime resources."""
global _runtime
if _runtime is not None:
_runtime.engine.dispose()
_runtime = None
def should_bootstrap_schema(settings: Settings) -> bool:
"""Return whether startup should auto-create schema for this environment."""
if settings.bootstrap_schema_on_startup is not None:
return settings.bootstrap_schema_on_startup
return settings.environment in {"development", "test"}
def create_all(*, engine: Engine | None = None) -> None:
"""Create all tables on the selected engine."""
# Import models so SQLModel metadata is fully registered before bootstrap.
from transcription import models as _models # noqa: F401
active_engine = engine or get_database_runtime().engine
SQLModel.metadata.create_all(active_engine)
_ensure_sqlite_compat_columns(active_engine)
def _ensure_sqlite_compat_columns(engine: Engine) -> None:
"""Apply lightweight dev/test SQLite compatibility column patches.
This keeps local bootstrap resilient when models evolve but no full
migration tooling is in place yet.
"""
if engine.url.get_backend_name() != "sqlite":
return
inspector = inspect(engine)
table_names = set(inspector.get_table_names())
if "job" not in table_names:
return
columns = {column["name"] for column in inspector.get_columns("job")}
if "retry_count" not in columns:
with engine.begin() as connection:
connection.execute(
text("ALTER TABLE job ADD COLUMN retry_count INTEGER NOT NULL DEFAULT 0")
)
logger.warning(
"Applied SQLite compatibility schema patch table=job column=retry_count default=0"
)
@contextlib.contextmanager
def get_session(*, engine: Engine | None = None) -> Generator[Session]:
"""Yield a database session and ensure cleanup."""
active_engine = engine or get_database_runtime().engine
with Session(active_engine) as session:
yield session
-6
View File
@@ -1,6 +0,0 @@
from .operations import create_all
from .runtime import dispose_database_runtime
from .runtime import get_session
from .runtime import initialize_database_runtime
__all__ = ["create_all", "dispose_database_runtime", "get_session", "initialize_database_runtime"]
-65
View File
@@ -1,65 +0,0 @@
from __future__ import annotations
import logging
from sqlalchemy import inspect
from sqlalchemy import text
from sqlalchemy.engine import Connection
from sqlalchemy.ext.asyncio import AsyncEngine
from sqlmodel import SQLModel
from sqlmodel import select
from sqlmodel.ext.asyncio.session import AsyncSession
from ..models import Job
from ..models import JobStatus
from .runtime import get_engine
logger = logging.getLogger(__name__)
async def get_next_queued_job(*, session: AsyncSession) -> Job | None:
"""Get the next queued job, if any."""
result = await session.exec(
select(Job)
.where(Job.status == JobStatus.QUEUED)
.order_by(Job.created_at) # pyright: ignore[reportArgumentType]
.limit(1)
) # fmt: skip
return result.first()
async def create_all(*, engine: AsyncEngine | None = None) -> None:
"""Create all tables on the selected engine."""
# Import models so SQLModel metadata is fully registered before bootstrap.
from transcription import models as _models # noqa: F401
active_engine = engine or get_engine()
async with active_engine.begin() as connection:
await connection.run_sync(SQLModel.metadata.create_all)
await connection.run_sync(_ensure_sqlite_compat_columns)
logger.debug("Database schema bootstrap complete for database_url=%s", active_engine.url)
def _ensure_sqlite_compat_columns(connection: Connection) -> None:
"""Apply lightweight dev/test SQLite compatibility column patches.
This keeps local bootstrap resilient when models evolve but no full
migration tooling is in place yet.
"""
if connection.engine.url.get_backend_name() != "sqlite":
return
inspector = inspect(connection)
table_names = set(inspector.get_table_names())
if "job" in table_names:
job_columns = {column["name"] for column in inspector.get_columns("job")}
if "retry_count" not in job_columns:
connection.execute(text("ALTER TABLE job ADD COLUMN retry_count INTEGER NOT NULL DEFAULT 0"))
logger.warning("Applied SQLite compatibility schema patch table=job column=retry_count default=0")
if "transcript" in table_names:
transcript_columns = {column["name"] for column in inspector.get_columns("transcript")}
if "model" not in transcript_columns:
connection.execute(text("ALTER TABLE transcript ADD COLUMN model VARCHAR NOT NULL DEFAULT 'unknown'"))
logger.warning("Applied SQLite compatibility schema patch table=transcript column=model default=unknown")
-103
View File
@@ -1,103 +0,0 @@
import logging
from collections.abc import AsyncGenerator
from contextlib import asynccontextmanager
from contextvars import ContextVar
from dataclasses import dataclass
from functools import partial
from sqlalchemy.ext.asyncio import AsyncEngine
from sqlalchemy.ext.asyncio import async_sessionmaker
from sqlalchemy.ext.asyncio import create_async_engine
from sqlmodel.ext.asyncio.session import AsyncSession
from sqlmodel.pool import StaticPool
from ..config import Settings
from ..config import get_settings
logger = logging.getLogger(__name__)
@dataclass(frozen=True)
class DatabaseRuntime:
"""Database runtime resources owned by app lifespan."""
engine: AsyncEngine
session_factory: async_sessionmaker[AsyncSession]
_runtime: ContextVar[DatabaseRuntime | None] = ContextVar("database_runtime", default=None)
async def dispose_database_runtime() -> None:
"""Dispose lifespan-owned async database resources."""
runtime = _runtime.get()
if runtime is None:
return
await runtime.engine.dispose()
_runtime.set(None)
def _to_async_database_url(database_url: str) -> str:
"""Normalize configured database URL to an async SQLAlchemy driver URL."""
if database_url.startswith("sqlite://") and not database_url.startswith("sqlite+aiosqlite://"):
return database_url.replace("sqlite://", "sqlite+aiosqlite://", 1)
if database_url.startswith("postgresql://") and not database_url.startswith("postgresql+asyncpg://"):
return database_url.replace("postgresql://", "postgresql+asyncpg://", 1)
return database_url
def _build_engine(settings: Settings) -> AsyncEngine:
database_url = _to_async_database_url(settings.database_url)
engine_factory = partial(
create_async_engine,
url=database_url,
echo=False,
pool_pre_ping=True,
)
if database_url.startswith("sqlite"):
sqlite_connect_settings = {"check_same_thread": settings.sqlite_check_same_thread}
engine_factory = partial(engine_factory, connect_args=sqlite_connect_settings)
if ":memory:" in database_url:
engine_factory = partial(engine_factory, poolclass=StaticPool)
return engine_factory()
def initialize_database_runtime(*, settings: Settings | None = None) -> DatabaseRuntime:
"""Initialize lifespan-owned async DB resources once per process."""
runtime = _runtime.get()
if runtime is not None:
return runtime
active_settings = settings or get_settings()
engine = _build_engine(active_settings)
session_factory = async_sessionmaker(engine, class_=AsyncSession, expire_on_commit=False)
runtime = DatabaseRuntime(engine=engine, session_factory=session_factory)
_runtime.set(runtime)
logger.debug("Initialized async database runtime for database_url=%s", engine.url)
return runtime
def get_engine(settings: Settings | None = None) -> AsyncEngine:
"""Return the current async SQLAlchemy engine."""
runtime = _runtime.get() or initialize_database_runtime(settings=settings)
return runtime.engine
def get_session_factory(settings: Settings | None = None) -> async_sessionmaker[AsyncSession]:
"""Return the shared async session factory."""
runtime = _runtime.get() or initialize_database_runtime(settings=settings)
return runtime.session_factory
@asynccontextmanager
async def get_session(
*,
settings: Settings | None = None,
session_factory: async_sessionmaker[AsyncSession] | None = None,
) -> AsyncGenerator[AsyncSession]:
"""Yield a database session and ensure cleanup."""
active_session_factory = session_factory or get_session_factory(settings)
async with active_session_factory() as session:
yield session
+6 -5
View File
@@ -3,8 +3,7 @@
from __future__ import annotations from __future__ import annotations
from dataclasses import dataclass from dataclasses import dataclass
from datetime import UTC from datetime import datetime, timezone
from datetime import datetime
from enum import StrEnum from enum import StrEnum
from uuid import uuid4 from uuid import uuid4
@@ -17,7 +16,6 @@ class ErrorCategory(StrEnum):
NOT_FOUND = "not_found_error" NOT_FOUND = "not_found_error"
CONFLICT = "conflict_error" CONFLICT = "conflict_error"
EXTERNAL_PROVIDER = "external_provider_error" EXTERNAL_PROVIDER = "external_provider_error"
PROCESSING = "processing_error"
INFRA_TRANSIENT = "infrastructure_transient_error" INFRA_TRANSIENT = "infrastructure_transient_error"
INFRA_PERSISTENT = "infrastructure_persistent_error" INFRA_PERSISTENT = "infrastructure_persistent_error"
INTERNAL_UNEXPECTED = "internal_unexpected_error" INTERNAL_UNEXPECTED = "internal_unexpected_error"
@@ -66,7 +64,7 @@ def build_error_envelope(error: AppError) -> ErrorEnvelope:
category=error.category.value, category=error.category.value,
message=error.message, message=error.message,
suggestion=error.suggestion, suggestion=error.suggestion,
timestamp=datetime.now(UTC).isoformat(), timestamp=datetime.now(timezone.utc).isoformat(),
) )
@@ -82,4 +80,7 @@ def classify_unexpected_error(exc: Exception, *, operation: str) -> AppError:
def format_error_detail(error: AppError) -> str: def format_error_detail(error: AppError) -> str:
"""Return a compact persisted failure string for transcript.error_detail.""" """Return a compact persisted failure string for transcript.error_detail."""
return f"[{error.category.value}] {error.message} | suggestion={error.suggestion} | error_id={error.error_id}" return (
f"[{error.category.value}] {error.message} | "
f"suggestion={error.suggestion} | error_id={error.error_id}"
)
+20 -34
View File
@@ -1,19 +1,15 @@
"""SQLModel domain models for the transcription system. """SQLModel domain models for the transcription system.
Three models capture the MVP lifecycle: Three models capture the MVP lifecycle:
Document -> one-to-many -> Job -> one-to-many -> Transcript Document -> one-to-many -> Job -> one-to-one -> Transcript
""" """
from datetime import UTC from datetime import datetime, timezone
from datetime import datetime
from enum import StrEnum from enum import StrEnum
from uuid import UUID from typing import Optional
from uuid import uuid4 from uuid import UUID, uuid4
from sqlalchemy import UniqueConstraint from sqlmodel import Field, Relationship, SQLModel
from sqlmodel import Field
from sqlmodel import Relationship
from sqlmodel import SQLModel
class JobStatus(StrEnum): class JobStatus(StrEnum):
@@ -29,7 +25,9 @@ class Document(SQLModel, table=True):
id: UUID = Field(default_factory=uuid4, primary_key=True) id: UUID = Field(default_factory=uuid4, primary_key=True)
filename: str filename: str
file_path: str file_path: str
uploaded_at: datetime = Field(default_factory=lambda: datetime.now(UTC)) uploaded_at: datetime = Field(
default_factory=lambda: datetime.now(timezone.utc),
)
# --- relationships --- # --- relationships ---
jobs: list["Job"] = Relationship(back_populates="document") jobs: list["Job"] = Relationship(back_populates="document")
@@ -42,40 +40,28 @@ class Job(SQLModel, table=True):
document_id: UUID = Field(foreign_key="document.id") document_id: UUID = Field(foreign_key="document.id")
status: JobStatus = Field(default=JobStatus.QUEUED) status: JobStatus = Field(default=JobStatus.QUEUED)
retry_count: int = Field(default=0, ge=0) retry_count: int = Field(default=0, ge=0)
created_at: datetime = Field(default_factory=lambda: datetime.now(UTC)) created_at: datetime = Field(
updated_at: datetime = Field(default_factory=lambda: datetime.now(UTC)) default_factory=lambda: datetime.now(timezone.utc),
)
updated_at: datetime = Field(
default_factory=lambda: datetime.now(timezone.utc),
)
# --- relationships --- # --- relationships ---
document: Document = Relationship(back_populates="jobs") document: Document = Relationship(back_populates="jobs")
transcripts: list["Transcript"] = Relationship(back_populates="job") transcript: Optional["Transcript"] = Relationship(back_populates="job")
@property
def filename(self) -> str:
"""Return the filename of the associated document."""
return self.document.filename if self.document else "unknown"
class Transcript(SQLModel, table=True): class Transcript(SQLModel, table=True):
"""The output of a transcription job.""" """The output of a transcription job."""
id: UUID = Field(default_factory=uuid4, primary_key=True) id: UUID = Field(default_factory=uuid4, primary_key=True)
job_id: UUID = Field(foreign_key="job.id") job_id: UUID = Field(foreign_key="job.id", unique=True)
"""ID for the associated job."""
revision: int = Field(default=0, ge=0)
"""Revision number for this job's transcript history, starting at 0."""
provider: str
"""Name of the transcription provider used to generate this transcript."""
model: str
"""Model identifier used to generate this transcript revision."""
prompt_name: str
"""Name of the prompt used to generate this transcript."""
text: str | None = None text: str | None = None
"""The transcribed text. This may be None if the job failed or is still in progress."""
error_detail: str | None = None error_detail: str | None = None
"""Details of any error that occurred during transcription.""" created_at: datetime = Field(
created_at: datetime = Field(default_factory=lambda: datetime.now(UTC)) default_factory=lambda: datetime.now(timezone.utc),
)
__table_args__ = (UniqueConstraint("job_id", "revision", name="uq_transcript_job_revision"),)
# --- relationships --- # --- relationships ---
job: Job = Relationship(back_populates="transcripts") job: Job = Relationship(back_populates="transcript")
+9 -9
View File
@@ -1,13 +1,13 @@
"""Provider exports and factory for transcription adapters.""" """Provider exports and factory for transcription adapters."""
from transcription.config import Provider from transcription.config import Provider, Settings, get_settings
from transcription.config import Settings from transcription.providers.base import (
from transcription.config import get_settings ProviderAuthError,
from transcription.providers.base import ProviderAuthError ProviderError,
from transcription.providers.base import ProviderError ProviderResponseError,
from transcription.providers.base import ProviderResponseError TranscriptionProvider,
from transcription.providers.base import TranscriptionProvider TranscriptionResult,
from transcription.providers.base import TranscriptionResult )
from transcription.providers.openrouter import OpenRouterTranscriptionProvider from transcription.providers.openrouter import OpenRouterTranscriptionProvider
@@ -21,11 +21,11 @@ def get_transcription_provider(*, settings: Settings | None = None) -> Transcrip
__all__ = [ __all__ = [
"OpenRouterTranscriptionProvider",
"ProviderAuthError", "ProviderAuthError",
"ProviderError", "ProviderError",
"ProviderResponseError", "ProviderResponseError",
"TranscriptionProvider", "TranscriptionProvider",
"TranscriptionResult", "TranscriptionResult",
"OpenRouterTranscriptionProvider",
"get_transcription_provider", "get_transcription_provider",
] ]
+1 -17
View File
@@ -2,9 +2,6 @@
from dataclasses import dataclass from dataclasses import dataclass
from typing import Protocol from typing import Protocol
from uuid import UUID
from ..models import Transcript
class ProviderError(RuntimeError): class ProviderError(RuntimeError):
@@ -25,24 +22,11 @@ class TranscriptionResult:
text: str text: str
provider: str provider: str
prompt_name: str
model: str model: str
def to_transcript(self, job_id: UUID, *, revision: int = 0) -> Transcript:
"""Convert a TranscriptionResult to a Transcript model instance."""
return Transcript(
job_id=job_id,
revision=revision,
provider=self.provider,
prompt_name=self.prompt_name,
model=self.model,
text=self.text,
)
class TranscriptionProvider(Protocol): class TranscriptionProvider(Protocol):
"""Contract every transcription provider adapter must satisfy.""" """Contract every transcription provider adapter must satisfy."""
async def transcribe(self, *, prompt_text: str, image_bytes: bytes, mime_type: str) -> TranscriptionResult: def transcribe(self, *, prompt_text: str, image_bytes: bytes, mime_type: str) -> TranscriptionResult:
"""Transcribe the provided image according to the prompt text.""" """Transcribe the provided image according to the prompt text."""
...
+16 -14
View File
@@ -6,17 +6,16 @@ import base64
import logging import logging
from dataclasses import dataclass from dataclasses import dataclass
from typing import Any from typing import Any
from typing import cast
from openrouter import OpenRouter from openrouter import OpenRouter
from openrouter.components.chatmessages import ChatMessagesTypedDict
from transcription.config import Settings from transcription.config import Settings, get_settings
from transcription.config import get_settings from transcription.providers.base import (
from transcription.providers.base import ProviderAuthError ProviderAuthError,
from transcription.providers.base import ProviderError ProviderError,
from transcription.providers.base import ProviderResponseError ProviderResponseError,
from transcription.providers.base import TranscriptionResult TranscriptionResult,
)
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@@ -46,17 +45,17 @@ class OpenRouterTranscriptionProvider:
"""Return the resolved OpenRouter model slug.""" """Return the resolved OpenRouter model slug."""
return self._model return self._model
async def transcribe(self, *, prompt_text: str, image_bytes: bytes, mime_type: str) -> TranscriptionResult: def transcribe(self, *, prompt_text: str, image_bytes: bytes, mime_type: str) -> TranscriptionResult:
"""Send prompt + image to OpenRouter and return normalized text output.""" """Send prompt + image to OpenRouter and return normalized text output."""
request = self._build_request(prompt_text=prompt_text, image_bytes=image_bytes, mime_type=mime_type) request = self._build_request(prompt_text=prompt_text, image_bytes=image_bytes, mime_type=mime_type)
try: try:
response = await self._client.chat.send_async( response = self._client.chat.send(
messages=cast(list[ChatMessagesTypedDict], request.messages), messages=request.messages,
model=request.model, model=request.model,
http_referer=request.http_referer, http_referer=request.http_referer,
x_open_router_title=request.x_open_router_title, x_open_router_title=request.x_open_router_title,
) )
except Exception as exc: except Exception as exc: # noqa: BLE001
message = str(exc).lower() message = str(exc).lower()
if "401" in message or "auth" in message or "api key" in message: if "401" in message or "auth" in message or "api key" in message:
raise ProviderAuthError("OpenRouter authentication failed") from exc raise ProviderAuthError("OpenRouter authentication failed") from exc
@@ -65,7 +64,7 @@ class OpenRouterTranscriptionProvider:
text = self._extract_text(response) text = self._extract_text(response)
model = self._get_optional_attr(response, "model") or self.model model = self._get_optional_attr(response, "model") or self.model
logger.info("OpenRouter transcription completed using model=%s", model) logger.info("OpenRouter transcription completed using model=%s", model)
return TranscriptionResult(text=text, provider="openrouter", prompt_name="", model=model) return TranscriptionResult(text=text, provider="openrouter", model=model)
def _build_request(self, *, prompt_text: str, image_bytes: bytes, mime_type: str) -> OpenRouterRequest: def _build_request(self, *, prompt_text: str, image_bytes: bytes, mime_type: str) -> OpenRouterRequest:
image_b64 = base64.b64encode(image_bytes).decode("ascii") image_b64 = base64.b64encode(image_bytes).decode("ascii")
@@ -112,7 +111,10 @@ class OpenRouterTranscriptionProvider:
parts: list[str] = [] parts: list[str] = []
for item in content: for item in content:
text_part = None text_part = None
text_part = item.get("text") if isinstance(item, dict) else self._get_optional_attr(item, "text") if isinstance(item, dict):
text_part = item.get("text")
else:
text_part = self._get_optional_attr(item, "text")
if isinstance(text_part, str) and text_part.strip(): if isinstance(text_part, str) and text_part.strip():
parts.append(text_part.strip()) parts.append(text_part.strip())
+26 -15
View File
@@ -1,19 +1,30 @@
"""Service layer exports.""" """Service layer exports."""
from dataclasses import dataclass from transcription.services.transcription import (
from dataclasses import field DEFAULT_PROMPT_FILE,
PromptLoadError,
TranscriptionError,
load_image_payload,
load_prompt_text,
transcribe_document_image,
)
from transcription.services.upload import (
SUPPORTED_UPLOAD_EXTENSIONS,
UploadError,
UploadJobResult,
create_upload_job,
)
from .documents import DocumentService __all__ = [
from .jobs import JobService "DEFAULT_PROMPT_FILE",
from .transcription import TranscriptionService "PromptLoadError",
"TranscriptionError",
"load_image_payload",
"load_prompt_text",
"transcribe_document_image",
"SUPPORTED_UPLOAD_EXTENSIONS",
"UploadError",
"UploadJobResult",
"create_upload_job",
]
__all__ = ["DocumentService", "JobService", "ServiceBundle", "TranscriptionService"]
@dataclass(frozen=True, slots=True)
class ServiceBundle:
"""Container for all service instances."""
documents: DocumentService = field(default_factory=DocumentService)
jobs: JobService = field(default_factory=JobService)
transcriptions: TranscriptionService = field(default_factory=TranscriptionService)
-60
View File
@@ -1,60 +0,0 @@
import asyncio
from abc import ABC
from collections.abc import Sequence
from contextlib import asynccontextmanager
from sqlalchemy.ext.asyncio import async_sessionmaker
from sqlmodel.ext.asyncio.session import AsyncSession
from ..config import Settings
from ..config import get_settings
from ..db.runtime import get_session_factory
class ServiceBase(ABC):
"""Thin service class for managing documents in the database."""
settings: Settings
session_factory: async_sessionmaker[AsyncSession]
queue: asyncio.Queue
def __init__(
self,
session_factory: async_sessionmaker[AsyncSession] | None = None,
queue: asyncio.Queue | None = None,
):
self.settings = get_settings()
self.session_factory = session_factory or get_session_factory()
self.queue = queue or asyncio.Queue()
@asynccontextmanager
async def _session_scope(self, session: AsyncSession | None = None):
"""Provide a transactional scope around a series of operations."""
if session is not None:
# Reuse the provided session if one is passed in
yield session
else:
# Otherwise, create a new session for this scope
async with self.session_factory() as new_session:
yield new_session
async def _finalize(
self,
*,
session: AsyncSession,
caller_session: AsyncSession | None,
refresh: Sequence[object] = (),
) -> None:
"""Finalize a write based on transaction ownership.
Service-owned sessions commit immediately. Caller-owned sessions flush so
orchestration code can commit once at a larger transaction boundary.
"""
should_commit = caller_session is None
if should_commit:
await session.commit()
else:
await session.flush()
for obj in refresh:
await session.refresh(obj)
-127
View File
@@ -1,127 +0,0 @@
import logging
from collections.abc import Sequence
from dataclasses import dataclass
from pathlib import Path
from uuid import UUID
from sqlalchemy.exc import IntegrityError
from sqlalchemy.orm import selectinload
from sqlmodel import select
from sqlmodel.ext.asyncio.session import AsyncSession
from ..errors import AppError
from ..errors import ErrorCategory
from ..models import Document
from .base import ServiceBase
logger = logging.getLogger(__name__)
class DocumentError(AppError):
"""Raised when document operations fail."""
class MissingImageError(DocumentError):
"""Raised when a required image is missing."""
class UploadError(DocumentError):
"""Raised when uploaded content cannot be persisted safely."""
class DocumentAlreadyExistsError(DocumentError):
"""Raised when a document with the same filename already exists in the database."""
@dataclass(frozen=True)
class UploadJobResult:
"""Summary of created upload records."""
document_id: UUID
job_id: UUID
stored_path: Path
original_filename: str
class DocumentService(ServiceBase):
"""Thin service class for managing documents in the database."""
#
# CRUD Operations
#
async def create_document(
self,
document: Document,
*,
session: AsyncSession | None = None,
) -> Document:
"""Create a new document in the database."""
async with self._session_scope(session) as _session:
_session.add(document)
try:
await self._finalize(session=_session, caller_session=session, refresh=(document,))
except IntegrityError as exc:
raise DocumentAlreadyExistsError(
f"Document with id {document.id} already exists",
category=ErrorCategory.VALIDATION,
suggestion="Rename the file and try again.",
) from exc
return document
async def read_document(self, document_id: UUID, *, session: AsyncSession | None = None) -> Document:
"""Read an existing document from the database.
The selectinload option is used to eagerly load related jobs for the document.
"""
async with self._session_scope(session) as _session:
document = await _session.get(
Document,
document_id,
options=(selectinload(Document.jobs),), # pyright: ignore[reportArgumentType]
)
if document is None:
raise DocumentError(
f"Document with id {document_id} not found",
category=ErrorCategory.NOT_FOUND,
suggestion="Re-upload the source document and retry.",
)
elif not Path(document.file_path).exists():
raise MissingImageError(
f"Document with id {document_id} is missing its image file in {self.settings.upload_dir:!s}",
category=ErrorCategory.NOT_FOUND,
suggestion="Re-upload the source document and retry.",
)
return document
async def update_document(self, document: Document, *, session: AsyncSession | None = None) -> Document:
"""Update an existing document in the database."""
async with self._session_scope(session) as _session:
merged = await _session.merge(document)
await self._finalize(session=_session, caller_session=session, refresh=(merged,))
return merged
async def delete_document(self, document: Document, *, session: AsyncSession | None = None) -> None:
"""Delete a document from the database."""
async with self._session_scope(session) as _session:
await _session.delete(document)
await self._finalize(session=_session, caller_session=session)
# Query Operations
async def query_documents(
self, *, filename: str | None = None, session: AsyncSession | None = None
) -> Sequence[Document]:
"""Query documents from the database based on provided filters."""
async with self._session_scope(session) as _session:
query = select(Document)
if filename is not None:
query = query.where(Document.filename == filename)
result = await _session.exec(query)
return result.all()
async def list_documents(self, *, session: AsyncSession | None = None) -> Sequence[Document]:
"""List all documents in the database."""
async with self._session_scope(session) as _session:
result = await _session.exec(select(Document))
return result.all()
-149
View File
@@ -1,149 +0,0 @@
from collections.abc import Sequence
from datetime import UTC
from datetime import datetime
from uuid import UUID
from sqlalchemy.orm import selectinload
from sqlmodel import select
from sqlmodel.ext.asyncio.session import AsyncSession
from ..models import Job
from ..models import JobStatus
from .base import ServiceBase
class JobService(ServiceBase):
"""Thin service class for managing jobs in the database."""
#
# CRUD Operations
#
async def create_job(self, job: Job, session: AsyncSession | None = None) -> Job:
"""Create a new job in the database."""
async with self._session_scope(session) as _session:
_session.add(job)
await self._finalize(session=_session, caller_session=session, refresh=(job,))
return job
async def read_job(self, job_id: UUID, session: AsyncSession | None = None) -> Job:
"""Read an existing job from the database.
The related document is always eagerly loaded so callers can safely
access ``job.document`` in async contexts without triggering lazy-load IO.
"""
async with self._session_scope(session) as _session:
query = (
select(Job)
.options(
selectinload(Job.document), # pyright: ignore[reportArgumentType]
selectinload(Job.transcripts), # pyright: ignore[reportArgumentType]
)
.where(Job.id == job_id)
.execution_options(populate_existing=True)
)
job = (await _session.exec(query)).first()
if job is None:
raise ValueError(f"Job with id {job_id} not found")
return job
async def update_job(self, job: Job, session: AsyncSession | None = None) -> Job:
"""Update an existing job in the database."""
async with self._session_scope(session) as _session:
merged = await _session.merge(job)
await self._finalize(session=_session, caller_session=session, refresh=(merged,))
return merged
async def delete_job(self, job: Job, session: AsyncSession | None = None) -> None:
"""Delete a job from the database."""
async with self._session_scope(session) as _session:
await _session.delete(job)
await self._finalize(session=_session, caller_session=session)
# Query Operations
async def query_jobs(
self,
*,
status: JobStatus | None = None,
filename: str | None = None,
session: AsyncSession | None = None,
) -> Sequence[Job]:
"""Query jobs from the database based on provided filters."""
async with self._session_scope(session) as _session:
query = select(Job).options(selectinload(Job.document)) # pyright: ignore[reportArgumentType]
if status is not None:
query = query.where(Job.status == status)
if filename is not None:
query = query.where(Job.document.filename == filename)
result = await _session.exec(query)
return result.all()
async def list_jobs(
self,
*,
load_docs: bool = False,
session: AsyncSession | None = None,
) -> Sequence[Job]:
"""List all jobs in the database with eagerly loaded documents."""
_ = load_docs
async with self._session_scope(session) as _session:
query = select(Job).options(selectinload(Job.document)) # pyright: ignore[reportArgumentType]
result = await _session.exec(query)
return result.all()
# Other Operations
async def mark_job_status(
self,
job_id: UUID,
status: JobStatus,
session: AsyncSession | None = None,
) -> Job:
"""Mark a job with a new status."""
return await self.update_job_state(job_id=job_id, status=status, session=session)
async def update_job_state(
self,
*,
job_id: UUID,
status: JobStatus,
retry_count_increment: int = 0,
session: AsyncSession | None = None,
) -> Job:
"""Update a job's lifecycle fields.
When ``session`` is provided, this method flushes so callers can commit
once at an orchestration boundary.
"""
async with self._session_scope(session) as _session:
query = (
select(Job)
.options(selectinload(Job.document)) # pyright: ignore[reportArgumentType]
.where(Job.id == job_id)
.execution_options(populate_existing=True)
)
job = (await _session.exec(query)).first()
if job is None:
raise ValueError(f"Job with id {job_id} not found")
job.status = status
if retry_count_increment:
job.retry_count += retry_count_increment
job.updated_at = datetime.now(UTC)
await self._finalize(session=_session, caller_session=session, refresh=(job,))
return job
async def read_next_queued_job(
self,
*,
session: AsyncSession | None = None,
) -> Job | None:
"""Read the next queued job ordered by creation time."""
async with self._session_scope(session) as _session:
query = (
select(Job)
.options(selectinload(Job.document)) # pyright: ignore[reportArgumentType]
.where(Job.status == JobStatus.QUEUED)
.order_by(Job.created_at) # pyright: ignore[reportArgumentType]
)
return (await _session.exec(query)).first()
+33 -203
View File
@@ -4,30 +4,18 @@ from __future__ import annotations
import logging import logging
import mimetypes import mimetypes
from collections.abc import Sequence
from contextlib import contextmanager
from pathlib import Path from pathlib import Path
from uuid import UUID
from sqlalchemy import func from transcription.config import Settings, get_settings
from sqlalchemy.ext.asyncio import async_sessionmaker from transcription.errors import AppError, ErrorCategory
from sqlalchemy.orm import selectinload from transcription.providers import (
from sqlmodel import select ProviderAuthError,
from sqlmodel.ext.asyncio.session import AsyncSession ProviderError,
ProviderResponseError,
from transcription.config import Settings TranscriptionProvider,
from transcription.config import get_settings TranscriptionResult,
from transcription.errors import AppError get_transcription_provider,
from transcription.errors import ErrorCategory )
from transcription.models import Transcript
from transcription.providers import ProviderAuthError
from transcription.providers import ProviderError
from transcription.providers import ProviderResponseError
from transcription.providers import TranscriptionProvider
from transcription.providers import TranscriptionResult
from transcription.providers import get_transcription_provider
from .base import ServiceBase
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@@ -43,183 +31,6 @@ class TranscriptionError(AppError):
"""Raised when transcription execution fails.""" """Raised when transcription execution fails."""
class TranscriptionNotFoundError(TranscriptionError):
"""Raised when a transcription is not found in the database."""
class TranscriptionService(ServiceBase):
"""Service class for managing transcription operations.
This is the top-level service that composes functionality from the other services."""
provider: TranscriptionProvider
def __init__(self, session_factory: async_sessionmaker[AsyncSession] | None = None):
super().__init__(session_factory=session_factory)
self.provider = get_transcription_provider(settings=self.settings)
async def create_transcript(self, transcript: Transcript, *, session: AsyncSession | None = None) -> Transcript:
"""Create a new transcript in the database."""
async with self._session_scope(session) as _session:
_session.add(transcript)
await self._finalize(session=_session, caller_session=session, refresh=(transcript,))
return transcript
async def read_transcript(self, transcript_id: UUID, *, session: AsyncSession | None = None) -> Transcript:
"""Read an existing transcript from the database."""
async with self._session_scope(session) as _session:
transcript = await _session.get(
Transcript,
transcript_id,
# Makes the full Job model object available in the return Transcript object
options=(selectinload(Transcript.job),), # pyright: ignore[reportArgumentType]
)
if transcript is None:
raise TranscriptionNotFoundError(
f"Transcript with id {transcript_id} not found",
category=ErrorCategory.NOT_FOUND,
suggestion="Verify the transcript id and retry.",
)
return transcript
async def update_transcript(self, transcript: Transcript, *, session: AsyncSession | None = None) -> Transcript:
"""Update an existing transcript in the database."""
async with self._session_scope(session) as _session:
merged = await _session.merge(transcript)
await self._finalize(session=_session, caller_session=session, refresh=(merged,))
return merged
async def delete_transcript(self, transcript: Transcript, *, session: AsyncSession | None = None) -> None:
"""Delete a transcript from the database."""
async with self._session_scope(session) as _session:
await _session.delete(transcript)
await self._finalize(session=_session, caller_session=session)
async def transcribe_document(
self,
image_path: str | Path,
job_id: UUID,
*,
prompt_name: str = DEFAULT_PROMPT_FILE,
session: AsyncSession | None = None,
):
"""Transcribe a local image using the configured prompt and provider."""
result = await transcribe_document_image(
image_path=image_path,
prompt_name=prompt_name,
settings=self.settings,
provider=self.provider,
)
await self.create_transcript_for_job(
job_id=job_id,
text=result.text,
provider=result.provider,
model=result.model,
prompt_name=result.prompt_name,
session=session,
)
async def create_transcript_for_job(
self,
*,
job_id: UUID,
text: str | None,
error_detail: str | None = None,
provider: str | None = None,
model: str | None = None,
prompt_name: str = DEFAULT_PROMPT_FILE,
session: AsyncSession | None = None,
) -> Transcript:
"""Create a new transcript revision for a job id."""
async with self._session_scope(session) as _session:
rev_query = select(func.max(Transcript.revision)).where(Transcript.job_id == job_id)
rev_result = await _session.exec(rev_query)
max_revision = -1 if (rev := rev_result.one_or_none()) is None else rev
next_revision = max_revision + 1
transcript = Transcript(
job_id=job_id,
revision=next_revision,
provider=provider or self.settings.provider.value,
model=model or _resolve_transcript_model(provider=self.provider, settings=self.settings),
prompt_name=prompt_name,
text=text,
error_detail=error_detail,
)
_session.add(transcript)
await self._finalize(session=_session, caller_session=session, refresh=(transcript,))
return transcript
async def read_latest_transcript_by_job(
self,
job_id: UUID,
*,
session: AsyncSession | None = None,
) -> Transcript | None:
"""Read the latest transcript revision for a job id."""
async with self._session_scope(session) as _session:
query = _transcript_job_query(job_id=job_id).limit(1)
result = await _session.exec(query)
return result.one_or_none()
async def list_transcripts_by_job(
self,
job_id: UUID,
*,
session: AsyncSession | None = None,
) -> Sequence[Transcript]:
"""List transcript revisions for a job id in ascending revision order."""
async with self._session_scope(session) as _session:
query = _transcript_job_query(job_id=job_id)
result = await _session.exec(query)
return result.all()
def _transcript_job_query(job_id: UUID):
return (
select(Transcript)
.where(Transcript.job_id == job_id)
.options(selectinload(Transcript.job)) # pyright: ignore[reportArgumentType]
.order_by(Transcript.revision) # pyright: ignore[reportArgumentType]
) # fmt: skip
def _resolve_transcript_model(*, provider: TranscriptionProvider, settings: Settings) -> str:
provider_model = getattr(provider, "model", None)
if isinstance(provider_model, str) and provider_model.strip():
return provider_model
if settings.provider_model and settings.provider_model.strip():
return settings.provider_model
return "unknown"
async def transcribe_document_image(
image_path: str | Path,
*,
prompt_name: str = DEFAULT_PROMPT_FILE,
settings: Settings | None = None,
provider: TranscriptionProvider | None = None,
) -> TranscriptionResult:
"""Transcribe a local image using the configured prompt and provider."""
runtime_settings = settings or get_settings()
prompt_text = load_prompt_text(prompt_name=prompt_name, settings=runtime_settings)
image_bytes, mime_type = load_image_payload(image_path)
adapter = provider or get_transcription_provider(settings=runtime_settings)
logger.info("Starting transcription for image=%s mime_type=%s", image_path, mime_type)
with handle_transcription_errors():
result = await adapter.transcribe(
prompt_text=prompt_text,
image_bytes=image_bytes,
mime_type=mime_type,
)
logger.info("Transcription completed for image=%s provider=%s", image_path, result.provider)
return result
def load_prompt_text(*, prompt_name: str = DEFAULT_PROMPT_FILE, settings: Settings | None = None) -> str: def load_prompt_text(*, prompt_name: str = DEFAULT_PROMPT_FILE, settings: Settings | None = None) -> str:
"""Load and validate prompt text from PROMPT_DIR.""" """Load and validate prompt text from PROMPT_DIR."""
runtime_settings = settings or get_settings() runtime_settings = settings or get_settings()
@@ -276,11 +87,27 @@ def load_image_payload(image_path: str | Path) -> tuple[bytes, str]:
return path.read_bytes(), mime_type return path.read_bytes(), mime_type
@contextmanager def transcribe_document_image(
def handle_transcription_errors(): image_path: str | Path,
"""Context manager to handle transcription errors.""" *,
prompt_name: str = DEFAULT_PROMPT_FILE,
settings: Settings | None = None,
provider: TranscriptionProvider | None = None,
) -> TranscriptionResult:
"""Transcribe a local image using the configured prompt and provider."""
runtime_settings = settings or get_settings()
prompt_text = load_prompt_text(prompt_name=prompt_name, settings=runtime_settings)
image_bytes, mime_type = load_image_payload(image_path)
adapter = provider or get_transcription_provider(settings=runtime_settings)
logger.info("Starting transcription for image=%s mime_type=%s", image_path, mime_type)
try: try:
yield result = adapter.transcribe(
prompt_text=prompt_text,
image_bytes=image_bytes,
mime_type=mime_type,
)
except ProviderAuthError as exc: except ProviderAuthError as exc:
raise TranscriptionError( raise TranscriptionError(
"Provider authentication failed", "Provider authentication failed",
@@ -301,3 +128,6 @@ def handle_transcription_errors():
suggestion="Retry the transcription from jobs. If repeated, check provider availability.", suggestion="Retry the transcription from jobs. If repeated, check provider availability.",
retriable=True, retriable=True,
) from exc ) from exc
logger.info("Transcription completed for image=%s provider=%s", image_path, result.provider)
return result
@@ -1,19 +1,18 @@
"""Upload service for storing files and creating queued transcription jobs."""
from __future__ import annotations from __future__ import annotations
import logging import logging
from dataclasses import dataclass
from pathlib import Path from pathlib import Path
from uuid import uuid4 from uuid import UUID, uuid4
from sqlmodel.ext.asyncio.session import AsyncSession from sqlmodel import Session
from transcription.config import Settings from transcription.config import Settings, get_settings
from transcription.config import get_settings from transcription.db import get_session
from transcription.errors import AppError from transcription.errors import AppError, ErrorCategory
from transcription.errors import ErrorCategory from transcription.models import Document, Job, JobStatus
from ..models import Document
from ..models import Job
from .documents import UploadJobResult
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@@ -24,75 +23,24 @@ class UploadError(AppError):
"""Raised when uploaded content cannot be persisted safely.""" """Raised when uploaded content cannot be persisted safely."""
async def create_upload_job( @dataclass(frozen=True)
class UploadJobResult:
"""Summary of created upload records."""
document_id: UUID
job_id: UUID
stored_path: Path
original_filename: str
def create_upload_job(
*, *,
filename: str, filename: str,
file_bytes: bytes, file_bytes: bytes,
session: AsyncSession, session: Session | None = None,
settings: Settings | None = None, settings: Settings | None = None,
) -> UploadJobResult: ) -> UploadJobResult:
"""Create upload-backed document and queued job records.""" """Persist an uploaded file and create document/job records."""
runtime_settings = settings or get_settings()
stored_path = store_file(
filename=filename,
file_bytes=file_bytes,
settings=runtime_settings,
)
try:
document, job = await _create_upload_records(
session=session,
original_filename=filename,
stored_path=stored_path,
)
except Exception as exc:
_best_effort_delete(stored_path)
raise UploadError(
"Failed to create upload database records",
category=ErrorCategory.INFRA_TRANSIENT,
suggestion="Retry upload. If this keeps happening, verify database availability.",
retriable=True,
) from exc
logger.info("Created upload job document_id=%s job_id=%s", document.id, job.id)
return UploadJobResult(
document_id=document.id,
job_id=job.id,
stored_path=stored_path,
original_filename=Path(filename).name,
)
async def _create_upload_records(
*,
session: AsyncSession,
original_filename: str,
stored_path: Path,
) -> tuple[Document, Job]:
document = Document(
filename=Path(original_filename).name,
file_path=str(stored_path),
)
session.add(document)
await session.flush()
job = Job(document_id=document.id)
session.add(job)
await session.commit()
await session.refresh(document)
await session.refresh(job)
return document, job
def _best_effort_delete(path: Path) -> None:
try:
if path.exists():
path.unlink()
except OSError:
logger.warning("Failed to clean up upload file after DB error: %s", path)
def store_file(*, filename: str, file_bytes: bytes, settings: Settings | None = None) -> Path:
"""Persist an uploaded file to the configured upload directory."""
runtime_settings = settings or get_settings() runtime_settings = settings or get_settings()
_validate_upload(filename=filename, file_bytes=file_bytes) _validate_upload(filename=filename, file_bytes=file_bytes)
@@ -111,8 +59,32 @@ def store_file(*, filename: str, file_bytes: bytes, settings: Settings | None =
suggestion="Check upload directory permissions and available disk space, then retry.", suggestion="Check upload directory permissions and available disk space, then retry.",
) from exc ) from exc
logger.info("Stored uploaded file: %s", stored_path) try:
return stored_path if session is not None:
document, job = _create_upload_records(session=session, original_filename=filename, stored_path=stored_path)
else:
with get_session() as local_session:
document, job = _create_upload_records(
session=local_session,
original_filename=filename,
stored_path=stored_path,
)
except Exception as exc: # noqa: BLE001
_best_effort_delete(stored_path)
raise UploadError(
"Failed to create upload database records",
category=ErrorCategory.INFRA_TRANSIENT,
suggestion="Retry upload. If this keeps happening, verify database availability.",
retriable=True,
) from exc
logger.info("Created upload job document_id=%s job_id=%s", document.id, job.id)
return UploadJobResult(
document_id=document.id,
job_id=job.id,
stored_path=stored_path,
original_filename=Path(filename).name,
)
def _validate_upload(*, filename: str, file_bytes: bytes) -> None: def _validate_upload(*, filename: str, file_bytes: bytes) -> None:
@@ -143,3 +115,30 @@ def _validate_upload(*, filename: str, file_bytes: bytes) -> None:
def _build_stored_filename(filename: str) -> str: def _build_stored_filename(filename: str) -> str:
safe_name = Path(filename).name safe_name = Path(filename).name
return f"{uuid4()}_{safe_name}" return f"{uuid4()}_{safe_name}"
def _create_upload_records(*, session: Session, original_filename: str, stored_path: Path) -> tuple[Document, Job]:
document = Document(
filename=Path(original_filename).name,
file_path=str(stored_path),
)
session.add(document)
session.flush()
job = Job(
document_id=document.id,
status=JobStatus.QUEUED,
)
session.add(job)
session.commit()
session.refresh(document)
session.refresh(job)
return document, job
def _best_effort_delete(path: Path) -> None:
try:
if path.exists():
path.unlink()
except OSError:
logger.warning("Failed to clean up upload file after DB error: %s", path)
-243
View File
@@ -1,243 +0,0 @@
import asyncio
import logging
from sqlmodel.ext.asyncio.session import AsyncSession
from ..config import Settings
from ..config import get_settings
from ..errors import AppError
from ..errors import classify_unexpected_error
from ..errors import format_error_detail
from ..models import Job
from ..models import JobStatus
from ..providers import TranscriptionResult
from . import ServiceBundle
from .transcription import DEFAULT_PROMPT_FILE
from .transcription import transcribe_document_image
logger = logging.getLogger(__name__)
async def advance_job(
job: Job,
services: ServiceBundle,
settings: Settings | None = None,
session: AsyncSession | None = None,
) -> Job | None:
"""Advance a single job by lifecycle status."""
settings = settings or get_settings()
match job.status:
case JobStatus.QUEUED:
return await process_queued_job(job=job, services=services, session=session)
case JobStatus.FAILED:
if job.retry_count < settings.worker_max_retries:
return await services.jobs.update_job_state(
job_id=job.id,
status=JobStatus.QUEUED,
retry_count_increment=1,
session=session,
)
else:
logger.error(f"Job {job.id} has failed and reached max retries.")
return
case _:
return
async def process_queued_job(
*,
job: Job,
services: ServiceBundle,
session: AsyncSession | None = None,
) -> Job | None:
"""Process one complete transcription attempt for a queued job."""
if job.status != JobStatus.QUEUED:
logger.warning(f"Job {job.id} is not queued. Current status: {job.status}")
return
# Transaction A: claim job for processing.
if session is None:
job = await services.jobs.mark_job_status(job.id, JobStatus.PROCESSING)
else:
# If we're sharing the session, need to make sure setting the Job to PROCESSING is committed before we start
# the transcription, otherwise other workers may see the job as still QUEUED and try to process it.
job = await services.jobs.mark_job_status(job.id, JobStatus.PROCESSING, session=session)
await session.commit()
document = job.document
assert document is not None, (
f"Job {job.id} has no associated document or the document failed to be loaded by the job service."
)
try:
result = await transcribe_document_image(document.file_path)
job = await _finalize_transcribed(job=job, services=services, result=result, session=session)
logger.info(
"Job transcribed operation=worker.process_job job_id=%s document_id=%s provider=%s",
job.id,
document.id,
result.provider,
)
except Exception as exc: # noqa: BLE001
match exc:
case AppError() as error:
pass
case _:
error = classify_unexpected_error(exc, operation="worker.process_job")
job = await _finalize_failed(job=job, services=services, error=error, session=session)
logger.error(
"Job failed operation=worker.process_job job_id=%s document_id=%s error_id=%s category=%s",
job.id,
document.id,
error.error_id,
error.category.value,
)
return job
async def process_next_queued_job(
*,
services: ServiceBundle,
settings: Settings | None = None,
session: AsyncSession | None = None,
) -> bool:
"""Process the next queued job if one exists."""
job = await services.jobs.read_next_queued_job(session=session)
if job is None:
return False
await advance_job(job=job, services=services, settings=settings, session=session)
return True
async def _finalize_transcribed(
*,
job: Job,
services: ServiceBundle,
result: TranscriptionResult,
session: AsyncSession | None = None,
) -> Job:
"""Transaction B: transcript + TRANSCRIBED in one commit."""
if session is None:
async with services.jobs._session_scope() as local_session:
await services.transcriptions.create_transcript_for_job(
job_id=job.id,
text=result.text,
error_detail=None,
provider=result.provider,
model=result.model,
prompt_name=result.prompt_name,
session=local_session,
)
updated_job = await services.jobs.mark_job_status(
job.id,
JobStatus.TRANSCRIBED,
session=local_session,
)
await local_session.commit()
return updated_job
await services.transcriptions.create_transcript_for_job(
job_id=job.id,
text=result.text,
error_detail=None,
provider=result.provider,
model=result.model,
prompt_name=result.prompt_name,
session=session,
)
updated_job = await services.jobs.mark_job_status(
job.id,
JobStatus.TRANSCRIBED,
session=session,
)
await session.commit()
return updated_job
async def _finalize_retry(
*,
job: Job,
services: ServiceBundle,
error: AppError,
settings: Settings,
session: AsyncSession | None = None,
) -> Job:
"""Transaction C: transcript error + QUEUED + retry increment in one commit."""
if session is None:
async with services.jobs._session_scope() as local_session:
await services.transcriptions.create_transcript_for_job(
job_id=job.id,
text=None,
error_detail=format_error_detail(error),
prompt_name=DEFAULT_PROMPT_FILE,
session=local_session,
)
updated_job = await services.jobs.update_job_state(
job_id=job.id,
status=JobStatus.QUEUED,
retry_count_increment=1,
session=local_session,
)
await local_session.commit()
else:
await services.transcriptions.create_transcript_for_job(
job_id=job.id,
text=None,
error_detail=format_error_detail(error),
prompt_name=DEFAULT_PROMPT_FILE,
session=session,
)
updated_job = await services.jobs.update_job_state(
job_id=job.id,
status=JobStatus.QUEUED,
retry_count_increment=1,
session=session,
)
await session.commit()
if settings.worker_retry_backoff_seconds > 0:
await asyncio.sleep(settings.worker_retry_backoff_seconds)
return updated_job
async def _finalize_failed(
*,
job: Job,
services: ServiceBundle,
error: AppError,
session: AsyncSession | None = None,
) -> Job:
"""Transaction B: transcript error + FAILED in one commit."""
if session is None:
async with services.jobs._session_scope() as local_session:
await services.transcriptions.create_transcript_for_job(
job_id=job.id,
text=None,
error_detail=format_error_detail(error),
prompt_name=DEFAULT_PROMPT_FILE,
session=local_session,
)
updated_job = await services.jobs.mark_job_status(
job.id,
JobStatus.FAILED,
session=local_session,
)
await local_session.commit()
return updated_job
await services.transcriptions.create_transcript_for_job(
job_id=job.id,
text=None,
error_detail=format_error_detail(error),
prompt_name=DEFAULT_PROMPT_FILE,
session=session,
)
updated_job = await services.jobs.mark_job_status(
job.id,
JobStatus.FAILED,
session=session,
)
await session.commit()
return updated_job
+5 -34
View File
@@ -1,45 +1,16 @@
"""UI page registration exports.""" """UI page registration exports."""
from pathlib import Path
from fastapi import FastAPI from fastapi import FastAPI
from nicegui import app as nicegui_app
from nicegui import ui from nicegui import ui
from transcription.ui.pages.jobs_page import register_page as register_jobs_page from transcription.ui.jobs_page import register_page as register_jobs_page
from transcription.ui.pages.upload_page import register_page as register_upload_page from transcription.ui.upload_page import register_page as register_upload_page
_THEME_REGISTERED_STATE_KEY = "transcription_ui_theme_registered"
_THEME_COLORS: dict[str, str] = {
"primary": "#6f97e8",
"secondary": "#92b5f5",
"accent": "#7fc0de",
"dark": "#22304a",
"dark_page": "#1a2538",
"positive": "#86c8ad",
"negative": "#d98a9a",
"info": "#7ebdda",
"warning": "#e2c083",
}
def _register_global_styles(app: FastAPI) -> None:
if getattr(app.state, _THEME_REGISTERED_STATE_KEY, False):
return
nicegui_app.colors(**_THEME_COLORS)
css_path = Path(__file__).resolve().parent / "static" / "colors.css"
if css_path.exists():
ui.add_css(css_path, shared=True)
setattr(app.state, _THEME_REGISTERED_STATE_KEY, True)
def register_pages(app: FastAPI) -> None: def register_pages(app: FastAPI) -> None:
"""Register all NiceGUI pages and mount them onto the FastAPI app.""" """Register all NiceGUI pages and mount them onto the FastAPI app."""
_register_global_styles(app)
register_upload_page() register_upload_page()
register_jobs_page() register_jobs_page()
ui.run_with(app, mount_path="/ui", show_welcome_message=False, dark=True) ui.run_with(app, mount_path="/ui", show_welcome_message=False)
@@ -1,7 +0,0 @@
"""Reusable UI component exports."""
from transcription.ui.components.app_shell import NAV_ITEMS
from transcription.ui.components.app_shell import render_navigation_header
from transcription.ui.components.document_panzoom import render_document_panzoom
__all__ = ["NAV_ITEMS", "render_document_panzoom", "render_navigation_header"]
@@ -1,59 +0,0 @@
"""Reusable app shell primitives for page-level layout."""
from __future__ import annotations
from nicegui import ui
NAV_ITEMS: tuple[tuple[str, str, str], ...] = (
("Upload", "/upload", "upload_file"),
("Jobs", "/jobs", "work_history"),
)
def _is_active_path(*, current_path: str, item_path: str) -> bool:
if item_path == "/jobs":
return current_path == "/jobs" or current_path.startswith("/jobs/")
return current_path == item_path
def _button_props(*, icon: str, is_active: bool) -> str:
if is_active:
return f"icon={icon} no-caps unelevated color=primary text-color=white"
return f"icon={icon} no-caps outline color=secondary text-color=secondary"
def _button_classes(*, is_active: bool) -> str:
base = "w-full sm:w-auto min-h-[40px] px-3 rounded-lg text-body2 text-weight-medium"
if is_active:
return f"{base}"
return f"{base}"
def _render_nav_button(*, label: str, path: str, icon: str, current_path: str) -> None:
is_active = _is_active_path(current_path=current_path, item_path=path)
button = ui.button(
label,
icon=icon,
on_click=lambda _=None, route=path: ui.navigate.to(route),
)
button.props(_button_props(icon=icon, is_active=is_active)).classes(_button_classes(is_active=is_active))
def _normalize_path(current_path: str | None) -> str:
normalized = (current_path or "").strip()
if not normalized:
return "/upload"
return normalized.rstrip("/") or "/"
def render_navigation_header(*, current_path: str | None = None) -> None:
"""Render a shared app header with links for top-level pages."""
normalized_path = _normalize_path(current_path)
with (
ui.header(elevated=True).props("bordered").classes("bg-dark text-white q-px-sm q-py-xs"),
ui.row().classes("w-full items-center justify-end q-gutter-xs"),
ui.element("div").classes("w-full grid grid-cols-2 gap-2 sm:flex sm:justify-start sm:gap-2"),
):
for label, path, icon in NAV_ITEMS:
_render_nav_button(label=label, path=path, icon=icon, current_path=normalized_path)
@@ -1,211 +0,0 @@
"""Panzoom-backed document preview component."""
from __future__ import annotations
from functools import lru_cache
from pathlib import Path
from urllib.parse import quote
from uuid import uuid4
from nicegui import ui
from transcription.config import get_settings
from transcription.models import Document
PANGOZOOM_CDN_URL = "https://unpkg.com/@panzoom/[email protected]/dist/panzoom.min.js"
UPLOADS_URL_PREFIX = "/uploads"
def render_document_panzoom(*, document: Document) -> None:
"""Render a document preview with pan and zoom interactions."""
_register_panzoom_assets()
host_id = f"document-panzoom-{uuid4().hex}"
document_url = _document_url(document)
document_kind = _document_kind(document)
with ui.card().classes("w-full q-pa-md"):
with ui.row().classes("w-full items-center justify-between no-wrap"):
ui.label("Document preview").classes("text-subtitle1 text-weight-medium")
ui.label(document.filename).classes("text-caption text-grey-4 ellipsis").style(
"max-width: 60%; text-align: right;"
)
with (
ui.element("div").classes("w-full document-panzoom-host rounded-borders q-mt-md")
# .style(f"height: {height};")
) as host:
host.props(f"id={host_id}")
with ui.element("div").classes("document-panzoom-surface"):
if document_kind == "pdf":
ui.html(
f'<iframe class="document-panzoom-iframe" '
f'src="{document_url}" title="{document.filename}" '
"data-panzoom-target></iframe>"
)
else:
ui.html(
f'<img class="document-panzoom-media" '
f'src="{document_url}" alt="{document.filename}" '
"data-panzoom-target data-panzoom-media />"
)
_attach_panzoom(host_id)
@lru_cache(maxsize=1)
def _register_panzoom_assets() -> None:
ui.add_head_html(
f'<script src="{PANGOZOOM_CDN_URL}"></script>',
shared=True,
)
ui.add_head_html(
"""
<style>
.document-panzoom-host {
overflow: hidden;
touch-action: none;
}
.document-panzoom-surface {
width: 100%;
height: 100%;
display: flex;
align-items: flex-start;
justify-content: flex-start;
}
.document-panzoom-media {
width: auto;
height: auto;
display: block;
max-width: 100%;
max-height: 100%;
user-select: none;
-webkit-user-drag: none;
}
.document-panzoom-iframe {
width: 100%;
height: 100%;
border: 0;
pointer-events: none;
background: white;
}
</style>
""",
shared=True,
)
def _document_url(document: Document) -> str:
file_path = Path(document.file_path)
upload_dir = get_settings().upload_dir
relative_path: Path
try:
relative_path = file_path.resolve().relative_to(upload_dir.resolve())
except ValueError:
parts = file_path.parts
if "uploads" in parts:
uploads_index = parts.index("uploads")
relative_path = Path(*parts[uploads_index + 1 :])
else:
relative_path = Path(file_path.name)
encoded_relative_path = "/".join(quote(part) for part in relative_path.parts)
return f"{UPLOADS_URL_PREFIX}/{encoded_relative_path}"
def _document_kind(document: Document) -> str:
suffix = Path(document.file_path).suffix.lower()
if suffix == ".pdf":
return "pdf"
return "image"
def _attach_panzoom(host_id: str) -> None:
ui.run_javascript(
f"""
(function() {{
if (!window.Panzoom) return;
window.__transcriptionPanzoom = window.__transcriptionPanzoom || {{}};
const host = document.getElementById({host_id!r});
if (!host) return;
const target = host.querySelector('[data-panzoom-target]');
const media = host.querySelector('[data-panzoom-media]');
if (!target) return;
const cleanup = () => {{
const existing = window.__transcriptionPanzoom[{host_id!r}];
if (existing?.resizeObserver) existing.resizeObserver.disconnect();
if (existing?.wheelHandler) host.removeEventListener('wheel', existing.wheelHandler);
if (existing?.instance) existing.instance.destroy();
}};
const computeFitScale = () => {{
const hostRect = host.getBoundingClientRect();
if (hostRect.width <= 0 || hostRect.height <= 0) return null;
return 1;
}};
const buildInstance = () => {{
cleanup();
const fitScale = computeFitScale();
if (fitScale === null) return false;
const minScale = Math.min(fitScale, 0.01);
const instance = Panzoom(target, {{
startX: 0,
startY: 0,
startScale: fitScale,
minScale: minScale,
maxScale: 256,
step: 0.2,
roundPixels: false,
panOnlyWhenZoomed: true,
overflow: 'hidden',
}});
const wheelHandler = (event) => instance.zoomWithWheel(event);
host.addEventListener('wheel', wheelHandler, {{ passive: false }});
requestAnimationFrame(() => {{
instance.reset({{ animate: false }});
}});
const resizeObserver = new ResizeObserver(() => {{
const nextFitScale = computeFitScale();
if (nextFitScale === null) return;
instance.setOptions({{
startScale: nextFitScale,
minScale: Math.min(nextFitScale, 0.01),
}});
instance.reset({{ animate: false }});
}});
resizeObserver.observe(host);
window.__transcriptionPanzoom[{host_id!r}] = {{
instance,
wheelHandler,
resizeObserver,
}};
return true;
}};
const initWhenReady = (retries = 15) => {{
if (buildInstance()) return;
if (retries <= 0) return;
requestAnimationFrame(() => initWhenReady(retries - 1));
}};
if (media && media.tagName === 'IMG' && !media.complete) {{
media.addEventListener('load', () => initWhenReady(), {{ once: true }});
return;
}}
initWhenReady();
}})();
"""
)
@@ -1,103 +0,0 @@
"""Reusable job detail rendering helpers."""
from __future__ import annotations
import logging
from collections.abc import Sequence
from nicegui import ui
from transcription.models import Document
from transcription.models import Job
from transcription.models import Transcript
from transcription.ui.components.document_panzoom import render_document_panzoom
logger = logging.getLogger(__name__)
def _status_chip_classes(status: str) -> str:
if status == "queued":
return "bg-blue-1 text-blue-10"
if status == "processing":
return "bg-amber-1 text-amber-10"
if status == "transcribed":
return "bg-green-1 text-green-10"
if status == "failed":
return "bg-red-1 text-red-10"
return "bg-grey-2 text-grey-9"
def _metadata_row(label: str, value: str) -> None:
with ui.row().classes("w-full items-start justify-between q-gutter-x-md"):
ui.label(label).classes("text-caption text-grey-5 text-uppercase w-28")
ui.label(value).classes("text-body2 text-right text-grey-1 break-all")
def _render_document_section(document: Document) -> None:
with ui.card().classes("w-full q-pa-md"):
ui.label("Document").classes("text-subtitle1 text-weight-medium")
ui.separator().classes("q-my-sm")
with ui.column().classes("w-full q-gutter-y-xs"):
_metadata_row("Filename", document.filename)
_metadata_row("File path", document.file_path)
ui.separator().classes("q-my-md")
render_document_panzoom(document=document)
def _render_transcript_section(transcripts: Sequence[Transcript]) -> None:
with ui.card().classes("w-full q-pa-md"):
ui.label("Transcripts").classes("text-subtitle1 text-weight-medium")
ui.separator().classes("q-my-sm")
if not transcripts:
ui.label("Transcript history is not available yet.").classes("text-body2 text-grey-3")
return
for transcript in transcripts:
with ui.column().classes("w-full q-gutter-y-xs"):
_metadata_row("Revision", str(transcript.revision))
_metadata_row("Provider", transcript.provider)
_metadata_row("Prompt", transcript.prompt_name)
_metadata_row("Created", transcript.created_at.isoformat())
if transcript.text:
ui.separator().classes("q-my-sm")
with ui.card().classes("w-fullq-pa-sm"):
ui.markdown(transcript.text).classes("text-grey-1")
elif transcript.error_detail:
ui.separator().classes("q-my-sm")
with ui.card().classes("w-full bg-red-1 text-red-10 q-pa-sm"):
ui.label("Failure detail").classes("text-caption text-uppercase")
ui.label(transcript.error_detail).classes("text-body2")
ui.separator().classes("q-my-md bg-blue-grey-7")
def render_job_detail(*, job: Job, document: Document | None, transcripts: Sequence[Transcript]) -> None:
"""Render all sections for the job detail page."""
logger.debug("Rendering job detail for job ID %s with %d transcripts", job.id, len(transcripts))
status_text = job.status.value
with ui.column().classes("w-full max-w-4xl q-gutter-md"):
with ui.card().classes("w-full q-pa-lg"):
with ui.row().classes("w-full items-center justify-between q-gutter-md"):
with ui.column().classes("q-gutter-none"):
ui.label("Job overview").classes("text-h6 text-weight-bold")
ui.label(str(job.id)).classes("text-caption text-grey-5")
status_chip_classes = (
"q-px-sm q-py-xs rounded-borders "
"text-weight-medium text-capitalize "
f"{_status_chip_classes(status_text)}"
)
ui.label(status_text).classes(status_chip_classes)
ui.separator().classes("q-my-md bg-blue-grey-7")
with ui.column().classes("w-full q-gutter-y-xs"):
_metadata_row("Created", job.created_at.isoformat())
_metadata_row("Updated", job.updated_at.isoformat())
_metadata_row("Retries", str(job.retry_count))
if document is not None:
_render_document_section(document)
_render_transcript_section(transcripts)
@@ -1,4 +0,0 @@
from .jobs import JobTableRow
from .jobs import render_jobs_table
__all__ = ["JobTableRow", "render_jobs_table"]
@@ -1,73 +0,0 @@
"""Common logic for generating table widgets."""
import logging
from collections.abc import Callable
from typing import Any
from nicegui import events
from nicegui import ui
logger = logging.getLogger(__name__)
def _extract_row_id(args: Any) -> str | None:
if isinstance(args, dict):
if isinstance(args.get("row"), dict):
row_id = args["row"].get("id")
return str(row_id) if row_id is not None else None
row_id = args.get("id")
return str(row_id) if row_id is not None else None
if isinstance(args, list):
for value in args:
if isinstance(value, dict):
row_id = value.get("id")
if row_id is not None:
return str(row_id)
return None
def _bind_row_click_handler(
table: Any,
*,
on_row_click_id: Callable[[str], None],
) -> None:
def handle_row_click(event: events.GenericEventArguments) -> None:
row_id = _extract_row_id(event.args)
if row_id is None:
return
on_row_click_id(row_id)
table.on("rowClick", handle_row_click)
logger.debug("Row click handler bound to table")
def build_table(
rows: list[dict[str, Any]],
columns: list[dict[str, Any]],
*,
default_sort_by: str | None = None,
default_descending: bool = False,
classes: str = "app-table",
on_row_click_id: Callable[[str], None] | None = None,
) -> Any:
pagination: dict[str, Any] = {"rowsPerPage": 25}
if default_sort_by is not None:
pagination["sortBy"] = default_sort_by
pagination["descending"] = default_descending
table = (
ui.table(
rows=rows,
columns=columns,
row_key="id",
pagination=pagination,
)
.classes(classes)
.props('table-style="table-layout: fixed; width: 100%;"')
)
logger.info("Table built with %d rows and %d columns", len(rows), len(columns))
if on_row_click_id is not None:
_bind_row_click_handler(table, on_row_click_id=on_row_click_id)
return table
@@ -1,75 +0,0 @@
"""Jobs table rendering helpers."""
from __future__ import annotations
from collections.abc import Sequence
from dataclasses import dataclass
from datetime import UTC
from datetime import datetime
from typing import Any
from uuid import UUID
from nicegui import ui
from .common import build_table
@dataclass(frozen=True, slots=True)
class JobTableRow:
"""Read model consumed by the jobs table component."""
id: UUID
status: str
filename: str
retry_count: int
created_at: str
updated_at: str
def _format_timestamp(value: str) -> str:
"""Return a friendly UTC timestamp for table display."""
try:
parsed = datetime.fromisoformat(value)
except ValueError:
return value
parsed = parsed.replace(tzinfo=UTC) if parsed.tzinfo is None else parsed.astimezone(UTC)
return parsed.astimezone().strftime("%b %d, %I:%M %p")
def _serialize_rows(rows: Sequence[JobTableRow]) -> list[dict[str, Any]]:
return [
{
"id": str(row.id),
"status": row.status,
"filename": row.filename,
"retry_count": row.retry_count,
"created_at": _format_timestamp(row.created_at),
"updated_at": _format_timestamp(row.updated_at),
"created_sort": row.created_at,
"updated_sort": row.updated_at,
}
for row in rows
]
def render_jobs_table(rows: Sequence[JobTableRow]) -> None:
"""Render jobs table and open a detail page when clicking a row."""
if not rows:
ui.label("No jobs yet.")
return
build_table(
rows=_serialize_rows(rows),
columns=[
{"name": "id", "label": "Job ID", "field": "id", "sortable": True},
{"name": "status", "label": "Status", "field": "status", "sortable": True},
{"name": "filename", "label": "Filename", "field": "filename", "sortable": True},
{"name": "retry_count", "label": "Retries", "field": "retry_count", "sortable": True},
{"name": "created_at", "label": "Created", "field": "created_at", "sortable": True},
{"name": "updated_at", "label": "Updated", "field": "updated_at", "sortable": True},
],
default_sort_by="created_sort",
default_descending=True,
classes="app-table w-full",
on_row_click_id=lambda job_id: ui.navigate.to(f"/jobs/{job_id}"),
)
@@ -1,86 +0,0 @@
"""Reusable transcript UI components."""
from __future__ import annotations
from collections.abc import Awaitable
from collections.abc import Callable
from datetime import datetime
from typing import Any
from nicegui import ui
from transcription.models import Transcript
type TranscriptAction = Callable[[Transcript], Awaitable[None] | None]
def render_transcript_revision_row(
*,
transcript: Transcript,
initially_expanded: bool = False,
classes: str = "w-full",
on_delete: TranscriptAction | None = None,
) -> Any:
"""Render one collapsible row for a single transcript revision."""
status_label = "Failed" if transcript.error_detail else "Transcribed"
header = f"Revision {transcript.revision} | {status_label}"
caption = f"{transcript.provider} | {transcript.model} | {_format_created_at(transcript.created_at)}"
expansion = ui.expansion(value=initially_expanded, group="group").classes(
f"{classes} rounded-borders bg-blue-grey-10"
)
with expansion, ui.column().classes("w-full q-gutter-y-sm q-pa-sm"):
with expansion.add_slot("header"), ui.row().classes("w-full items-start justify-between q-gutter-md"):
with ui.column().classes("q-gutter-none"):
ui.label(header).classes("text-subtitle1 text-weight-medium")
ui.label(caption).classes("text-caption text-grey-5")
if on_delete is not None:
with ui.dialog() as delete_dialog, ui.card().classes("q-pa-md"):
ui.label("Delete this transcript revision?").classes("text-body1")
with ui.row().classes("w-full justify-end q-gutter-sm"):
ui.button("Cancel", on_click=lambda: delete_dialog.submit(False)).props("flat")
ui.button("Delete", on_click=lambda: delete_dialog.submit(True)).props(
'unelevated color="negative"'
)
async def delete_current_transcript() -> None:
delete_dialog.open()
confirmed = await delete_dialog
if not confirmed:
return
maybe_awaitable = on_delete(transcript)
if isinstance(maybe_awaitable, Awaitable):
await maybe_awaitable
with ui.column(align_items="center").classes("self-center q-gutter-none"):
ui.button(icon="delete", on_click=delete_current_transcript).props(
'flat round dense color="negative"'
)
_metadata_row(label="Provider", value=transcript.provider)
_metadata_row(label="Model", value=transcript.model)
_metadata_row(label="Created", value=_format_created_at(transcript.created_at))
if transcript.text:
with ui.card().classes("w-full q-pa-sm"):
ui.markdown(transcript.text)
if transcript.error_detail:
with ui.card().classes("w-full bg-red-1 text-red-10 q-pa-sm"):
ui.label("Failure detail").classes("text-caption text-uppercase")
ui.label(transcript.error_detail).classes("text-body2")
return expansion
def _format_created_at(value: datetime) -> str:
"""Return a compact UTC-like timestamp for row captions."""
return value.strftime("%Y-%m-%d %H:%M:%S %Z")
def _metadata_row(*, label: str, value: str) -> None:
with ui.row().classes("w-md items-start justify-between q-gutter-x-md"):
ui.label(label).classes("text-caption text-grey-5 text-uppercase")
ui.label(value).classes("text-body2 text-right text-grey-1 break-all")
-66
View File
@@ -1,66 +0,0 @@
"""Reusable upload widget for document submission."""
from __future__ import annotations
from collections.abc import Awaitable
from collections.abc import Callable
from nicegui import ui
from nicegui.binding import bindable_dataclass
from nicegui.events import UploadEventArguments
from transcription.errors import AppError
from transcription.services.documents import UploadJobResult
from transcription.ui.components.error_presenter import show_error
from transcription.ui.components.error_presenter import summarize_error
from transcription.worker import WorkerNotifier
type UploadSubmitter = Callable[[str, bytes], Awaitable[UploadJobResult]]
@bindable_dataclass
class UploadWidgetState:
"""Simple state container for upload feedback."""
loading: bool = False
message: str = ""
def render_upload_widget(*, submitter: UploadSubmitter, notifier: WorkerNotifier | None = None) -> None:
"""Render upload controls and common status/error handling."""
state = UploadWidgetState()
status_label = ui.label("Upload a document to start transcription.")
status_label.bind_text(state, "message")
async def on_upload(event: UploadEventArguments) -> None:
if state.loading:
ui.notify("Upload already in progress. Please wait.", type="warning")
return
state.loading = True
status_label.text = "Uploading..."
try:
payload = await event.file.read()
result = await submitter(event.file.name, payload)
job_id = result.job_id
state.message = f"Created job {job_id}" if job_id is not None else "Upload complete"
status_label.text = state.message
if notifier is not None:
notifier.notify()
ui.notify(state.message, type="positive")
except AppError as exc:
state.message = summarize_error(exc, operation="upload.submit")
status_label.text = f"Upload failed: {state.message}"
show_error(exc, title="Upload failed", operation="upload.submit")
except Exception as exc: # noqa: BLE001
state.message = summarize_error(exc, operation="upload.submit")
status_label.text = f"Upload failed: {state.message}"
show_error(exc, title="Upload failed", operation="upload.submit")
finally:
state.loading = False
ui.upload(
on_upload=on_upload,
auto_upload=True,
label="Select document file",
).props('accept=".jpg,.jpeg,.png,.tif,.tiff,.pdf"')
@@ -4,9 +4,7 @@ from __future__ import annotations
from nicegui import ui from nicegui import ui
from transcription.errors import AppError from transcription.errors import AppError, ErrorCategory, classify_unexpected_error
from transcription.errors import ErrorCategory
from transcription.errors import classify_unexpected_error
def to_app_error(exc: Exception, *, operation: str) -> AppError: def to_app_error(exc: Exception, *, operation: str) -> AppError:
@@ -39,4 +37,4 @@ def summarize_error(exc: Exception, *, operation: str) -> str:
error = to_app_error(exc, operation=operation) error = to_app_error(exc, operation=operation)
if error.category == ErrorCategory.INTERNAL_UNEXPECTED: if error.category == ErrorCategory.INTERNAL_UNEXPECTED:
return f"Unexpected error (ref: {error.error_id})" return f"Unexpected error (ref: {error.error_id})"
return f"{error.message} (ref: {error.error_id})" return f"{error.message} (ref: {error.error_id})"
+137
View File
@@ -0,0 +1,137 @@
"""Jobs list and detail page registration."""
from __future__ import annotations
from dataclasses import dataclass
from uuid import UUID
from nicegui import ui
from sqlmodel import select
from transcription.db import get_session
from transcription.models import Document, Job, Transcript
from transcription.ui.error_presenter import show_error, summarize_error
@dataclass(frozen=True)
class JobView:
"""Read model for rendering job rows in the UI."""
id: UUID
status: str
created_at: str
updated_at: str
def fetch_jobs() -> list[JobView]:
"""Return jobs for display in most-recent-first order."""
with get_session() as session:
jobs = session.exec(select(Job).order_by(Job.created_at.desc())).all()
return [
JobView(
id=job.id,
status=job.status.value,
created_at=job.created_at.isoformat(),
updated_at=job.updated_at.isoformat(),
)
for job in jobs
]
def fetch_job_detail(job_id: UUID) -> tuple[Job | None, Document | None, Transcript | None]:
"""Return job, document, and transcript for detail view."""
with get_session() as session:
job = session.get(Job, job_id)
if job is None:
return None, None, None
document = session.get(Document, job.document_id)
transcript = session.exec(select(Transcript).where(Transcript.job_id == job.id)).first()
return job, document, transcript
def register_page() -> None:
"""Register jobs list and detail routes."""
@ui.page("/jobs")
def jobs_page() -> None:
ui.label("Transcription Jobs")
status = ui.label("Ready")
table_container = ui.column()
def render_table() -> None:
table_container.clear()
jobs = fetch_jobs()
with table_container:
if not jobs:
ui.label("No jobs yet.")
return
rows = [
{
"id": str(job.id),
"status": job.status,
"created_at": job.created_at,
"updated_at": job.updated_at,
}
for job in jobs
]
ui.table(
columns=[
{"name": "id", "label": "Job ID", "field": "id"},
{"name": "status", "label": "Status", "field": "status"},
{"name": "created_at", "label": "Created", "field": "created_at"},
{"name": "updated_at", "label": "Updated", "field": "updated_at"},
],
rows=rows,
row_key="id",
)
for row in rows:
ui.link(f"Open {row['id']}", f"/jobs/{row['id']}")
def refresh() -> None:
status.text = "Refreshing..."
try:
render_table()
status.text = "Refreshed"
except Exception as exc: # noqa: BLE001
status.text = f"Refresh failed: {summarize_error(exc, operation='jobs.refresh')}"
show_error(exc, title="Jobs refresh failed", operation="jobs.refresh")
ui.button("Refresh", on_click=refresh)
render_table()
ui.link("Back to upload", "/")
@ui.page("/jobs/{job_id}")
def job_detail_page(job_id: str) -> None:
ui.label("Job Detail")
try:
parsed_id = UUID(job_id)
except ValueError:
ui.label("Invalid job id")
ui.link("Back to jobs", "/jobs")
return
job, document, transcript = fetch_job_detail(parsed_id)
if job is None:
ui.label("Job not found")
ui.link("Back to jobs", "/jobs")
return
ui.label(f"Job ID: {job.id}")
ui.label(f"Status: {job.status.value}")
ui.label(f"Created: {job.created_at.isoformat()}")
ui.label(f"Updated: {job.updated_at.isoformat()}")
if document is not None:
ui.label(f"Filename: {document.filename}")
ui.label(f"File path: {document.file_path}")
if transcript is None:
ui.label("Transcript not available yet.")
elif transcript.text:
ui.label("Transcript:")
ui.markdown(transcript.text)
elif transcript.error_detail:
ui.label("Failure detail:")
ui.label(transcript.error_detail)
ui.link("Back to jobs", "/jobs")
-99
View File
@@ -1,99 +0,0 @@
"""Jobs list and detail page registration."""
from __future__ import annotations
from uuid import UUID
from fastapi import Request
from nicegui import ui
from transcription.app_state import resolve_session_factory
from transcription.models import JobStatus
from transcription.services.jobs import JobService
from transcription.services.transcription import TranscriptionService
from transcription.ui.components.app_shell import render_navigation_header
from transcription.ui.components.error_presenter import show_error
from transcription.ui.components.table.jobs import render_jobs_table
from ..components.document_panzoom import render_document_panzoom
from ..components.table.jobs import JobTableRow
from ..components.transcript import render_transcript_revision_row
def register_page() -> None:
"""Register jobs list and detail routes."""
@ui.page("/jobs")
async def jobs_page(request: Request) -> None:
session_factory = resolve_session_factory(request.app.state)
jobs_service = JobService(session_factory=session_factory)
render_navigation_header(current_path="/jobs")
@ui.refreshable
async def render_table() -> None:
jobs = [
JobTableRow(
id=job.id,
status=job.status.value,
filename=job.filename,
retry_count=job.retry_count,
created_at=job.created_at.isoformat(),
updated_at=job.updated_at.isoformat(),
)
for job in await jobs_service.list_jobs()
]
render_jobs_table(jobs)
ui.button("Refresh", on_click=render_table.refresh, icon="refresh")
await render_table()
@ui.page("/jobs/{job_id}")
async def job_detail_page(job_id: str, request: Request) -> None:
session_factory = resolve_session_factory(request.app.state)
jobs_service = JobService(session_factory=session_factory)
transcription_service = TranscriptionService(session_factory=session_factory)
render_navigation_header(current_path="/jobs")
job = await jobs_service.read_job(job_id=UUID(job_id))
with ui.splitter(value=30).classes("w-full h-[calc(100vh-64px)]") as splitter:
with splitter.before, ui.column(align_items="stretch").classes("w-full h-full p-4 gap-3"):
render_document_panzoom(document=job.document)
with splitter.after, ui.column(align_items="stretch").classes("w-full h-full p-4 gap-3"):
with ui.row():
ui.button(icon="arrow_back", on_click=ui.navigate.back)
with ui.row().classes("w-full items-center justify-between"):
ui.label(f"{job.id}").classes("text-h6 text-weight-bold")
match job.status:
case JobStatus.TRANSCRIBED:
ui.chip(job.status.value.upper(), color="green", text_color="white").props("outline")
case _:
ui.label(f"{job.status.value}").classes("text-subtitle1 text-weight-medium")
async def delete_transcript_by_id(transcript_id: UUID, revision: int) -> None:
try:
transcript = await transcription_service.read_transcript(transcript_id=transcript_id)
await transcription_service.delete_transcript(transcript)
except Exception as exc: # noqa: BLE001
show_error(exc, title="Delete failed", operation="jobs.delete_transcript")
return
ui.notify(f"Deleted revision {revision}", type="positive")
await render_transcript_list.refresh()
@ui.refreshable
async def render_transcript_list() -> None:
refreshed_job = await jobs_service.read_job(job_id=UUID(job_id))
for i, transcript in enumerate(refreshed_job.transcripts):
render_transcript_revision_row(
transcript=transcript,
initially_expanded=(i == 0),
on_delete=(
lambda _transcript, tid=transcript.id, rev=transcript.revision: delete_transcript_by_id(
tid,
rev,
)
),
)
await render_transcript_list()
-33
View File
@@ -1,33 +0,0 @@
"""Upload page registration and handlers."""
from __future__ import annotations
from fastapi import Request
from nicegui import ui
from transcription.app_state import resolve_session_factory
from transcription.db import get_session
from transcription.services.store import create_upload_job
from transcription.ui.components.app_shell import render_navigation_header
from transcription.ui.components.upload import render_upload_widget
from transcription.worker import resolve_worker_notifier
def register_page() -> None:
"""Register the upload page route."""
@ui.page("/upload", title="Upload Document")
def upload_page(request: Request) -> None:
render_navigation_header(current_path="/upload")
session_factory = resolve_session_factory(request.app.state)
async def submit_upload(filename: str, file_bytes: bytes):
async with get_session(session_factory=session_factory) as session:
return await create_upload_job(
filename=filename,
file_bytes=file_bytes,
session=session,
)
notify_worker = resolve_worker_notifier(request.app.state)
render_upload_widget(submitter=submit_upload, notifier=notify_worker)
-30
View File
@@ -1,30 +0,0 @@
:root {
/* Soft blue-night palette tokens */
--ctp-rosewater: #f2dde5;
--ctp-flamingo: #edcfd8;
--ctp-pink: #dcc7de;
--ctp-mauve: #a9bde5;
--ctp-red: #d98a9a;
--ctp-maroon: #d39aa5;
--ctp-peach: #d7af8c;
--ctp-yellow: #e2c083;
--ctp-green: #86c8ad;
--ctp-teal: #77bfbe;
--ctp-sky: #7ebdda;
--ctp-sapphire: #74aed0;
--ctp-blue: #92b5f5;
--ctp-lavender: #6f97e8;
--ctp-text: #d8e2f5;
--ctp-subtext1: #bfcae0;
--ctp-subtext0: #a9b6cf;
--ctp-overlay2: #95a3bf;
--ctp-overlay1: #7c8ca9;
--ctp-overlay0: #657490;
--ctp-surface2: #4d5f7c;
--ctp-surface1: #394a65;
--ctp-surface0: #2a3954;
--ctp-base: #1f2b42;
--ctp-mantle: #1a2538;
--ctp-crust: #141e30;
}
+71
View File
@@ -0,0 +1,71 @@
"""Upload page registration and handlers."""
from __future__ import annotations
from dataclasses import dataclass
from nicegui import ui
from nicegui.events import UploadEventArguments
from transcription.services.upload import UploadError, UploadJobResult, create_upload_job
from transcription.ui.error_presenter import show_error, summarize_error
@dataclass
class UploadPageState:
"""Simple state container for upload page feedback."""
loading: bool = False
message: str = ""
def accepted_upload_types() -> str:
"""Return accepted file type string for upload input."""
return ".jpg,.jpeg,.png,.tif,.tiff,.pdf"
def submit_upload(*, filename: str, file_bytes: bytes) -> UploadJobResult:
"""Create an upload job from incoming file data."""
return create_upload_job(filename=filename, file_bytes=file_bytes)
def register_page() -> None:
"""Register the upload page route."""
@ui.page("/")
def upload_page() -> None:
state = UploadPageState()
status_label = ui.label("Upload a document to start transcription.")
async def on_upload(event: UploadEventArguments) -> None:
if state.loading:
ui.notify("Upload already in progress. Please wait.", type="warning")
return
state.loading = True
status_label.text = "Uploading..."
try:
payload = await event.file.read()
result = submit_upload(filename=event.file.name, file_bytes=payload)
state.message = f"Created job {result.job_id}"
status_label.text = state.message
ui.notify(state.message, type="positive")
except UploadError as exc:
state.message = summarize_error(exc, operation="upload.submit")
status_label.text = f"Upload failed: {state.message}"
show_error(exc, title="Upload failed", operation="upload.submit")
except Exception as exc: # noqa: BLE001
state.message = summarize_error(exc, operation="upload.submit")
status_label.text = f"Upload failed: {state.message}"
show_error(exc, title="Upload failed", operation="upload.submit")
finally:
state.loading = False
ui.upload(
on_upload=on_upload,
auto_upload=True,
label="Select document file",
).props(f"accept={accepted_upload_types()}")
with ui.row():
ui.link("View jobs", "/jobs")
+129 -155
View File
@@ -2,183 +2,157 @@
from __future__ import annotations from __future__ import annotations
import asyncio
import logging import logging
from collections.abc import AsyncGenerator import time
from contextlib import asynccontextmanager from datetime import datetime, timezone
from contextlib import contextmanager from threading import Event
from contextlib import suppress
from typing import Protocol
from uuid import UUID
from sqlalchemy.ext.asyncio import async_sessionmaker from pydantic import ValidationError
from sqlmodel.ext.asyncio.session import AsyncSession from sqlalchemy.engine import Engine
from sqlmodel import Session, select
from transcription.config import Settings, get_settings
from transcription.db import get_session from transcription.db import get_session
from transcription.errors import AppError from transcription.errors import AppError, ErrorCategory, classify_unexpected_error, format_error_detail
from transcription.errors import classify_unexpected_error from transcription.models import Document, Job, JobStatus, Transcript
from transcription.services.transcription import transcribe_document_image
from .services import ServiceBundle
from .services.documents import DocumentService
from .services.jobs import JobService
from .services.transcription import TranscriptionService
from .services.workflows import advance_job
from .services.workflows import process_next_queued_job as process_next_queued_job_workflow
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
class WorkerNotifier(Protocol): def process_next_queued_job(*, session: Session | None = None, engine: Engine | None = None) -> bool:
"""Abstraction for signaling the worker loop about new work.""" """Process the next queued job and persist terminal outcome.
def notify(self) -> None: Returns True when a job was processed, False when no queued job exists.
"""Signal the worker loop that work may be available."""
class EventWorkerNotifier:
"""Worker notifier backed by an asyncio.Event."""
def __init__(self, wake_event: asyncio.Event):
self._wake_event = wake_event
def notify(self) -> None:
self._wake_event.set()
class NoopWorkerNotifier:
"""Fallback notifier used when worker signaling is unavailable."""
def notify(self) -> None:
return
def resolve_worker_notifier(state: object) -> WorkerNotifier:
"""Resolve notifier from app-like state objects with no-op fallback."""
notifier = getattr(state, "worker_notifier", None)
if isinstance(notifier, NoopWorkerNotifier):
return notifier
if notifier is None:
return NoopWorkerNotifier()
return notifier
@asynccontextmanager
async def worker_consumer_lifespan(
*,
session_factory: async_sessionmaker[AsyncSession] | None = None,
poll_interval_seconds: float = 1.0,
) -> AsyncGenerator[tuple[asyncio.Event, WorkerNotifier]]:
"""Start and stop the worker consumer loop for app lifespan."""
stop_event = asyncio.Event()
wake_event = asyncio.Event()
worker_notifier: WorkerNotifier = EventWorkerNotifier(wake_event)
worker_task = asyncio.create_task(
run_worker_loop(
session_factory=session_factory,
stop_event=stop_event,
wake_event=wake_event,
poll_interval_seconds=poll_interval_seconds,
)
)
worker_notifier.notify()
try:
yield stop_event, worker_notifier
finally:
stop_event.set()
worker_notifier.notify()
try:
await asyncio.wait_for(worker_task, timeout=2.0)
except TimeoutError:
worker_task.cancel()
with suppress(asyncio.CancelledError):
await worker_task
async def queue_consumer_loop(queue: asyncio.Queue[UUID], stop_event: asyncio.Event):
"""Main worker loop that consumes jobs from the queue and processes them.
The queue is for Job UUIDs, and the corresponding documents should already have been uploaded.
""" """
service = JobService() if session is None:
while not stop_event.is_set(): with get_session(engine=engine) as local_session:
with handle_worker_exceptions(): return _process_next_queued_job(session=local_session)
async with _get_queue_item(queue) as job_id: return _process_next_queued_job(session=session)
job = await service.read_job(job_id)
asyncio.create_task(advance_job(job=job, services=ServiceBundle()))
@contextmanager def _process_next_queued_job(*, session: Session) -> bool:
def handle_worker_exceptions(operation: str = "worker.loop"): job = session.exec(
"""Context manager to log and suppress exceptions in the worker loop.""" select(Job)
try: .where(Job.status == JobStatus.QUEUED)
yield .order_by(Job.created_at)
except Exception as exc: ).first()
error = exc if isinstance(exc, AppError) else classify_unexpected_error(exc, operation=operation)
logger.exception( if job is None:
"Worker loop exception error_id=%s category=%s", return False
logger.info("Picked queued job operation=worker.pick job_id=%s", job.id)
job.status = JobStatus.PROCESSING
job.updated_at = datetime.now(timezone.utc)
session.add(job)
session.commit()
session.refresh(job)
document = session.get(Document, job.document_id)
if document is None:
error = AppError(
"Document not found",
category=ErrorCategory.NOT_FOUND,
suggestion="Re-upload the source document and retry processing.",
)
_finalize_failed_job(session=session, job=job, error=error)
logger.error(
"Job failed operation=worker.process_job job_id=%s error_id=%s category=%s",
job.id,
error.error_id, error.error_id,
error.category.value, error.category.value,
) )
return True
try:
result = transcribe_document_image(document.file_path)
_upsert_transcript(session=session, job_id=job.id, text=result.text, error_detail=None)
job.status = JobStatus.TRANSCRIBED
job.updated_at = datetime.now(timezone.utc)
session.add(job)
session.commit()
logger.info(
"Job transcribed operation=worker.process_job job_id=%s document_id=%s provider=%s",
job.id,
document.id,
result.provider,
)
except Exception as exc: # noqa: BLE001
error = exc if isinstance(exc, AppError) else classify_unexpected_error(exc, operation="worker.process_job")
settings = _get_worker_settings()
if _should_retry(job=job, error=error, settings=settings):
_requeue_for_retry(session=session, job=job, error=error, settings=settings)
logger.warning(
"Job retried operation=worker.process_job job_id=%s document_id=%s retry_count=%s error_id=%s category=%s",
job.id,
document.id,
job.retry_count,
error.error_id,
error.category.value,
)
else:
_finalize_failed_job(session=session, job=job, error=error)
logger.exception(
"Job failed operation=worker.process_job job_id=%s document_id=%s error_id=%s category=%s",
job.id,
document.id,
error.error_id,
error.category.value,
)
return True
@asynccontextmanager def _upsert_transcript(*, session: Session, job_id, text: str | None, error_detail: str | None) -> Transcript:
async def _get_queue_item(queue: asyncio.Queue[UUID]) -> AsyncGenerator[UUID]: transcript = session.exec(select(Transcript).where(Transcript.job_id == job_id)).first()
"""Context manager to enqueue a job and ensure it is marked done.""" if transcript is None:
yield await queue.get() transcript = Transcript(job_id=job_id)
queue.task_done()
transcript.text = text
transcript.error_detail = error_detail
session.add(transcript)
session.commit()
session.refresh(transcript)
return transcript
async def run_worker_loop( def _get_worker_settings() -> Settings:
*, try:
session_factory: async_sessionmaker[AsyncSession] | None = None, return get_settings()
stop_event: asyncio.Event | None = None, except ValidationError:
wake_event: asyncio.Event | None = None, return Settings(openrouter_api_key="test-key")
poll_interval_seconds: float = 1.0,
) -> None:
"""Run worker loop until stop_event is set.
If wake_event is provided, signal activity wakes the loop immediately while
timeout-based wakeups preserve current polling behavior. def _should_retry(*, job: Job, error: AppError, settings: Settings) -> bool:
""" return error.retriable and job.retry_count < settings.worker_max_retries
def _requeue_for_retry(*, session: Session, job: Job, error: AppError, settings: Settings) -> None:
_upsert_transcript(session=session, job_id=job.id, text=None, error_detail=format_error_detail(error))
job.retry_count += 1
job.status = JobStatus.QUEUED
job.updated_at = datetime.now(timezone.utc)
session.add(job)
session.commit()
if settings.worker_retry_backoff_seconds > 0:
time.sleep(settings.worker_retry_backoff_seconds)
def _finalize_failed_job(*, session: Session, job: Job, error: AppError) -> None:
_upsert_transcript(session=session, job_id=job.id, text=None, error_detail=format_error_detail(error))
job.status = JobStatus.FAILED
job.updated_at = datetime.now(timezone.utc)
session.add(job)
session.commit()
def run_worker_loop(*, engine: Engine | None = None, stop_event: Event | None = None, poll_interval_seconds: float = 1.0) -> None:
"""Run worker polling loop until stop_event is set."""
while True: while True:
if stop_event is not None and stop_event.is_set(): if stop_event is not None and stop_event.is_set():
logger.info("Worker stop event received") logger.info("Worker stop event received")
return return
if wake_event is not None: processed = process_next_queued_job(engine=engine)
with suppress(TimeoutError): if not processed:
await asyncio.wait_for(wake_event.wait(), timeout=poll_interval_seconds) time.sleep(poll_interval_seconds)
wake_event.clear()
processed_any = False
while await process_next_queued_job(session_factory=session_factory):
processed_any = True
if wake_event is None and not processed_any:
await asyncio.sleep(poll_interval_seconds)
async def process_next_queued_job(
*,
session: AsyncSession | None = None,
session_factory: async_sessionmaker[AsyncSession] | None = None,
) -> bool:
"""Process the next queued job and persist terminal outcome.
Returns True when a job was processed, False when no queued job exists.
"""
if session_factory is None:
services = ServiceBundle()
else:
services = ServiceBundle(
documents=DocumentService(session_factory=session_factory),
jobs=JobService(session_factory=session_factory),
transcriptions=TranscriptionService(session_factory=session_factory),
)
if session is None:
async with get_session(session_factory=session_factory) as local_session:
return await process_next_queued_job_workflow(services=services, session=local_session)
return await process_next_queued_job_workflow(services=services, session=session)
+4 -53
View File
@@ -5,67 +5,18 @@ isolated, fast, and leave no artifacts on disk.
""" """
import pytest import pytest
import pytest_asyncio from sqlmodel import Session, SQLModel, create_engine
from sqlmodel import Session
from sqlmodel import SQLModel
from sqlmodel import create_engine
from sqlmodel.pool import StaticPool from sqlmodel.pool import StaticPool
from transcription.config import Settings
from transcription.config import get_settings
from transcription.db.operations import create_all
from transcription.db.runtime import dispose_database_runtime
from transcription.db.runtime import get_engine
from transcription.db.runtime import get_session
from transcription.db.runtime import get_session_factory
from transcription.services.documents import DocumentService
from transcription.services.jobs import JobService
@pytest.fixture @pytest.fixture
def session(): def session():
"""Provide a clean synchronous database session for sync tests.""" """Provide a clean database session for each test."""
engine = create_engine( engine = create_engine(
"sqlite://", "sqlite://",
connect_args={"check_same_thread": False}, connect_args={"check_same_thread": False},
poolclass=StaticPool, poolclass=StaticPool,
) )
SQLModel.metadata.create_all(engine) SQLModel.metadata.create_all(engine)
with Session(engine) as sync_session: with Session(engine) as session:
yield sync_session yield session
@pytest_asyncio.fixture
async def default_settings():
"""Provide default settings for tests."""
settings = get_settings(database_url="sqlite:///:memory:")
await create_all(engine=get_engine(settings=settings))
return settings
@pytest_asyncio.fixture
async def async_session(default_settings: Settings):
"""Provide a clean asynchronous database session for async tests."""
async with get_session(settings=default_settings) as async_session:
yield async_session
await dispose_database_runtime()
@pytest.fixture
def default_session_factory(default_settings: Settings):
"""Provide a base fixture for tests that require database access."""
session_factory = get_session_factory(settings=default_settings)
return session_factory
@pytest.fixture
def job_service(default_session_factory) -> JobService:
"""Provide a JobService instance for testing."""
return JobService(session_factory=default_session_factory)
@pytest.fixture
def document_service(default_session_factory) -> DocumentService:
"""Provide a DocumentService instance for testing."""
return DocumentService(session_factory=default_session_factory)
-88
View File
@@ -1,88 +0,0 @@
from uuid import uuid4
import pytest
from transcription.models import Document
from transcription.models import Job
from transcription.services.documents import DocumentService
from transcription.services.jobs import JobService
from transcription.services.jobs import JobStatus
class TestJobService:
class TestBasicCRUD:
@pytest.mark.asyncio
async def test_create_job(self, job_service: JobService):
"""Test creating a job."""
def fake_job_factory():
return Job(document_id=uuid4())
await job_service.create_job(job=fake_job_factory())
async with job_service._session_scope() as session:
for _ in range(10):
await job_service.create_job(job=fake_job_factory(), session=session)
@pytest.mark.asyncio
async def test_backpropagation(self, job_service: JobService, document_service: DocumentService):
"""Test that creating a job backpropagates to the related document."""
doc_id = uuid4()
document = Document(
id=doc_id,
filename="test.txt",
file_path="/path/to/test.txt",
)
await document_service.create_document(document=document)
job = Job(document_id=doc_id)
await job_service.create_job(job=job)
read_job = await job_service.read_job(job_id=job.id)
assert isinstance(read_job.document, Document)
assert read_job.document.id == document.id
@pytest.mark.asyncio
async def test_reading_job(self, job_service: JobService):
"""Test reading a job."""
uuid = uuid4()
await job_service.create_job(job=Job(id=uuid, document_id=uuid4()))
job = await job_service.read_job(job_id=uuid)
assert job.id == uuid
@pytest.mark.asyncio
async def test_updating_job(self, job_service: JobService):
"""Test updating a job."""
uuid = uuid4()
job = Job(id=uuid, document_id=uuid4())
async with job_service._session_scope() as session:
await job_service.create_job(job=job, session=session)
job.status = JobStatus.PROCESSING
await job_service.update_job(job=job, session=session)
read_job = await job_service.read_job(job_id=uuid, session=session)
assert read_job == job
@pytest.mark.asyncio
async def test_deleting_job(self, job_service: JobService):
"""Test deleting a job."""
class TestServiceMethods:
@pytest.mark.asyncio
async def test_query_jobs(self, job_service: JobService):
"""Test querying jobs."""
await job_service.create_job(job=Job(document_id=uuid4(), status=JobStatus.PROCESSING))
result = await job_service.query_jobs(status=JobStatus.PROCESSING)
jobs = {str(job.id).split("-")[0]: job.status for job in result}
assert len(jobs) == 1
@pytest.mark.asyncio
async def test_list_jobs(self, job_service: JobService):
"""Test listing jobs."""
n = 5
for _ in range(n):
await job_service.create_job(job=Job(document_id=uuid4()))
jobs = await job_service.list_jobs()
assert len(jobs) == n
@pytest.mark.asyncio
async def test_mark_job_status(self, job_service: JobService):
"""Test marking a job with a new status."""
-36
View File
@@ -1,36 +0,0 @@
import pytest
class TestServiceBase:
class TestInitialization:
def test_initializes_with_defaults(self):
"""Test initialization with default session factory and queue."""
def test_initializes_with_custom_session_factory(self):
"""Test initialization with a provided session factory."""
def test_initializes_with_custom_queue(self):
"""Test initialization with a provided queue."""
class TestSessionScope:
@pytest.mark.asyncio
async def test_uses_provided_session(self):
"""Test that session scope reuses a provided session."""
@pytest.mark.asyncio
async def test_creates_new_session_when_none_provided(self):
"""Test that session scope creates a new session when none is provided."""
class TestContextManagerBehavior:
@pytest.mark.asyncio
async def test_yields_session(self):
"""Test that session scope yields a usable session object."""
@pytest.mark.asyncio
async def test_multiple_operations(self):
"""Test multiple operations within a single session scope."""
class TestEdgeCases:
@pytest.mark.asyncio
async def test_handles_exception_propagation(self):
"""Test exception propagation behavior inside session scope."""
+137
View File
@@ -0,0 +1,137 @@
"""Tests for transcription.services.transcription."""
from pathlib import Path
import pytest
from transcription.config import Settings
from transcription.providers.base import ProviderError, TranscriptionResult
from transcription.services.transcription import (
PromptLoadError,
TranscriptionError,
load_image_payload,
load_prompt_text,
transcribe_document_image,
)
class _FakeProvider:
def __init__(self, *, result: TranscriptionResult | None = None, error: Exception | None = None):
self._result = result or TranscriptionResult(
text="Transcript output",
provider="openrouter",
model="test-model",
)
self._error = error
self.calls: list[dict[str, object]] = []
def transcribe(self, *, prompt_text: str, image_bytes: bytes, mime_type: str) -> TranscriptionResult:
self.calls.append(
{
"prompt_text": prompt_text,
"image_bytes": image_bytes,
"mime_type": mime_type,
}
)
if self._error:
raise self._error
return self._result
@pytest.mark.unit
class TestPromptLoading:
"""Verify prompt artifact loading and validation."""
def test_loads_prompt_text_from_prompt_dir(self, tmp_path: Path):
"""Prompt loader returns canonical prompt text from configured prompt directory."""
prompt_dir = tmp_path / "prompts"
prompt_dir.mkdir()
prompt_file = prompt_dir / "transcribe_document.md"
prompt_file.write_text("Prompt body", encoding="utf-8")
settings = Settings(openrouter_api_key="test-key", prompt_dir=prompt_dir)
text = load_prompt_text(settings=settings)
assert text == "Prompt body"
def test_missing_prompt_raises_error(self, tmp_path: Path):
"""Prompt loader raises PromptLoadError when the file is missing."""
prompt_dir = tmp_path / "prompts"
prompt_dir.mkdir()
settings = Settings(openrouter_api_key="test-key", prompt_dir=prompt_dir)
with pytest.raises(PromptLoadError) as exc_info:
load_prompt_text(settings=settings)
assert exc_info.value.category.value == "infrastructure_persistent_error"
assert "verify prompt_dir" in exc_info.value.suggestion.lower()
@pytest.mark.unit
class TestImageLoading:
"""Verify local image payload loading and mime detection."""
def test_load_image_payload_reads_bytes_and_mime_type(self, tmp_path: Path):
"""Image loader returns file bytes and a detected MIME type for supported files."""
image_path = tmp_path / "sample.png"
image_bytes = b"\x89PNG\r\n\x1a\n"
image_path.write_bytes(image_bytes)
loaded_bytes, mime_type = load_image_payload(image_path)
assert loaded_bytes == image_bytes
assert mime_type == "image/png"
def test_missing_image_raises_error(self, tmp_path: Path):
"""Image loader raises TranscriptionError when image file does not exist."""
missing = tmp_path / "missing.png"
with pytest.raises(TranscriptionError) as exc_info:
load_image_payload(missing)
assert exc_info.value.category.value == "not_found_error"
assert "verify" in exc_info.value.suggestion.lower()
@pytest.mark.unit
class TestTranscriptionService:
"""Verify service orchestration across prompt, image, and provider calls."""
def test_transcribe_document_image_calls_provider_once(self, tmp_path: Path):
"""Service loads prompt and image, then invokes provider exactly once."""
prompt_dir = tmp_path / "prompts"
prompt_dir.mkdir()
(prompt_dir / "transcribe_document.md").write_text("Prompt body", encoding="utf-8")
image_path = tmp_path / "document.jpg"
image_path.write_bytes(b"jpeg-bytes")
settings = Settings(openrouter_api_key="test-key", prompt_dir=prompt_dir)
provider = _FakeProvider()
result = transcribe_document_image(image_path, settings=settings, provider=provider)
assert result.text == "Transcript output"
assert len(provider.calls) == 1
assert provider.calls[0]["prompt_text"] == "Prompt body"
assert provider.calls[0]["image_bytes"] == b"jpeg-bytes"
assert provider.calls[0]["mime_type"] == "image/jpeg"
def test_provider_error_is_wrapped(self, tmp_path: Path):
"""Service wraps provider failures in TranscriptionError."""
prompt_dir = tmp_path / "prompts"
prompt_dir.mkdir()
(prompt_dir / "transcribe_document.md").write_text("Prompt body", encoding="utf-8")
image_path = tmp_path / "document.png"
image_path.write_bytes(b"png-bytes")
settings = Settings(openrouter_api_key="test-key", prompt_dir=prompt_dir)
provider = _FakeProvider(error=ProviderError("upstream failure"))
with pytest.raises(TranscriptionError) as exc_info:
transcribe_document_image(image_path, settings=settings, provider=provider)
assert exc_info.value.category.value == "external_provider_error"
assert exc_info.value.retriable is True
assert "retry" in exc_info.value.suggestion.lower()
+104
View File
@@ -0,0 +1,104 @@
"""Tests for transcription.services.upload."""
from pathlib import Path
import pytest
from transcription.config import Settings
from transcription.models import Document, Job, JobStatus
from transcription.services.upload import UploadError, create_upload_job
@pytest.mark.unit
class TestUploadValidation:
"""Verify upload validation behavior."""
def test_rejects_empty_bytes(self, session, tmp_path: Path):
"""create_upload_job rejects an empty upload payload."""
settings = Settings(openrouter_api_key="test-key", upload_dir=tmp_path)
with pytest.raises(UploadError) as exc_info:
create_upload_job(
filename="letter.jpg",
file_bytes=b"",
session=session,
settings=settings,
)
assert exc_info.value.category.value == "validation_error"
assert "non-empty" in exc_info.value.suggestion.lower()
def test_rejects_unsupported_extension(self, session, tmp_path: Path):
"""create_upload_job rejects unsupported filename extensions."""
settings = Settings(openrouter_api_key="test-key", upload_dir=tmp_path)
with pytest.raises(UploadError) as exc_info:
create_upload_job(
filename="notes.txt",
file_bytes=b"content",
session=session,
settings=settings,
)
assert exc_info.value.category.value == "user_input_error"
assert "jpg" in exc_info.value.suggestion.lower()
@pytest.mark.integration
class TestUploadPersistence:
"""Verify upload file and record persistence behavior."""
def test_writes_file_and_creates_records(self, session, tmp_path: Path):
"""create_upload_job writes file and creates document/job records."""
settings = Settings(openrouter_api_key="test-key", upload_dir=tmp_path)
result = create_upload_job(
filename="letter.jpg",
file_bytes=b"image-bytes",
session=session,
settings=settings,
)
assert result.stored_path.exists()
assert result.stored_path.read_bytes() == b"image-bytes"
document = session.get(Document, result.document_id)
job = session.get(Job, result.job_id)
assert document is not None
assert job is not None
assert document.filename == "letter.jpg"
assert document.file_path == str(result.stored_path)
def test_uses_unique_stored_filename(self, session, tmp_path: Path):
"""create_upload_job stores uploads with unique filenames."""
settings = Settings(openrouter_api_key="test-key", upload_dir=tmp_path)
first = create_upload_job(
filename="duplicate.jpg",
file_bytes=b"first",
session=session,
settings=settings,
)
second = create_upload_job(
filename="duplicate.jpg",
file_bytes=b"second",
session=session,
settings=settings,
)
assert first.stored_path != second.stored_path
assert first.stored_path.exists()
assert second.stored_path.exists()
def test_sets_job_status_queued(self, session, tmp_path: Path):
"""create_upload_job persists a job with queued status."""
settings = Settings(openrouter_api_key="test-key", upload_dir=tmp_path)
result = create_upload_job(
filename="queued.pdf",
file_bytes=b"%PDF-1.4",
session=session,
settings=settings,
)
job = session.get(Job, result.job_id)
assert job is not None
assert job.status == JobStatus.QUEUED
+232
View File
@@ -0,0 +1,232 @@
"""Tests for transcription.worker."""
from pathlib import Path
from threading import Event
import pytest
from sqlmodel import select
from transcription.config import Settings
from transcription.errors import AppError, ErrorCategory
from transcription.models import Document, Job, JobStatus, Transcript
from transcription.providers.base import TranscriptionResult
from transcription.worker import process_next_queued_job, run_worker_loop
def _create_queued_job(session, *, filename: str = "doc.jpg", file_path: str = "uploads/doc.jpg") -> Job:
document = Document(filename=filename, file_path=file_path)
session.add(document)
session.commit()
session.refresh(document)
job = Job(document_id=document.id, status=JobStatus.QUEUED)
session.add(job)
session.commit()
session.refresh(job)
return job
@pytest.mark.integration
class TestWorkerQueueBehavior:
"""Verify worker behavior when selecting queued jobs."""
def test_returns_false_when_queue_empty(self, session):
"""process_next_queued_job returns False when there are no queued jobs."""
processed = process_next_queued_job(session=session)
assert processed is False
@pytest.mark.integration
class TestWorkerSuccessPath:
"""Verify worker success-path lifecycle transitions and transcript persistence."""
def test_transitions_processing_to_transcribed(self, session, monkeypatch, tmp_path: Path):
"""process_next_queued_job transitions queued jobs to transcribed on success."""
job = _create_queued_job(session)
def _fake_transcribe(_path):
return TranscriptionResult(text="ok", provider="openrouter", model="test-model")
monkeypatch.setattr("transcription.worker.transcribe_document_image", _fake_transcribe)
monkeypatch.setattr(
"transcription.worker.get_settings",
lambda: Settings(openrouter_api_key="test-key", prompt_dir=tmp_path),
)
processed = process_next_queued_job(session=session)
session.refresh(job)
assert processed is True
assert job.status == JobStatus.TRANSCRIBED
def test_persists_transcript_text_on_success(self, session, monkeypatch, tmp_path: Path):
"""process_next_queued_job stores transcript text for successful jobs."""
job = _create_queued_job(session)
def _fake_transcribe(_path):
return TranscriptionResult(text="Transcript body", provider="openrouter", model="test-model")
monkeypatch.setattr("transcription.worker.transcribe_document_image", _fake_transcribe)
monkeypatch.setattr(
"transcription.worker.get_settings",
lambda: Settings(openrouter_api_key="test-key", prompt_dir=tmp_path),
)
process_next_queued_job(session=session)
transcript = session.exec(
select(Transcript).where(Transcript.job_id == job.id)
).first()
assert transcript is not None
assert transcript.text == "Transcript body"
assert transcript.error_detail is None
@pytest.mark.integration
class TestWorkerFailurePath:
"""Verify worker failure-path lifecycle transitions and error persistence."""
def test_sets_failed_and_error_detail_on_failure(self, session, monkeypatch, tmp_path: Path):
"""process_next_queued_job marks failed and stores error detail on exception."""
job = _create_queued_job(session)
def _fake_transcribe(_path):
raise RuntimeError("provider failure")
monkeypatch.setattr("transcription.worker.transcribe_document_image", _fake_transcribe)
monkeypatch.setattr(
"transcription.worker.get_settings",
lambda: Settings(openrouter_api_key="test-key", prompt_dir=tmp_path),
)
processed = process_next_queued_job(session=session)
session.refresh(job)
transcript = session.exec(
select(Transcript).where(Transcript.job_id == job.id)
).first()
assert processed is True
assert job.status == JobStatus.FAILED
assert transcript is not None
assert transcript.text is None
assert "provider failure" in transcript.error_detail
assert "[internal_unexpected_error]" in transcript.error_detail
assert "error_id=" in transcript.error_detail
assert "suggestion=" in transcript.error_detail
def test_updates_existing_transcript_if_present(self, session, monkeypatch, tmp_path: Path):
"""process_next_queued_job updates existing transcript instead of duplicating."""
job = _create_queued_job(session)
existing = Transcript(job_id=job.id, text="old", error_detail=None)
session.add(existing)
session.commit()
session.refresh(existing)
def _fake_transcribe(_path):
raise RuntimeError("provider failure")
monkeypatch.setattr("transcription.worker.transcribe_document_image", _fake_transcribe)
monkeypatch.setattr(
"transcription.worker.get_settings",
lambda: Settings(openrouter_api_key="test-key", prompt_dir=tmp_path),
)
process_next_queued_job(session=session)
transcripts = session.exec(
select(Transcript).where(Transcript.job_id == job.id)
).all()
assert len(transcripts) == 1
assert transcripts[0].id == existing.id
assert transcripts[0].text is None
assert "provider failure" in transcripts[0].error_detail
assert "[internal_unexpected_error]" in transcripts[0].error_detail
assert "error_id=" in transcripts[0].error_detail
@pytest.mark.integration
class TestWorkerRetryBehavior:
"""Verify worker retry and terminal failure policies."""
def test_retriable_failure_requeues_until_limit(self, session, monkeypatch, tmp_path: Path):
"""Retriable failures requeue jobs while retry budget remains."""
job = _create_queued_job(session)
def _fake_transcribe(_path):
raise AppError(
"temporary upstream outage",
category=ErrorCategory.EXTERNAL_PROVIDER,
suggestion="Retry from jobs page.",
retriable=True,
)
monkeypatch.setattr("transcription.worker.transcribe_document_image", _fake_transcribe)
monkeypatch.setattr(
"transcription.worker.get_settings",
lambda: Settings(
openrouter_api_key="test-key",
prompt_dir=tmp_path,
worker_max_retries=1,
worker_retry_backoff_seconds=0.0,
),
)
processed = process_next_queued_job(session=session)
session.refresh(job)
assert processed is True
assert job.status == JobStatus.QUEUED
assert job.retry_count == 1
def test_retriable_failure_exhaustion_sets_failed(self, session, monkeypatch, tmp_path: Path):
"""Retriable failures transition to failed when retry budget is exhausted."""
job = _create_queued_job(session)
job.retry_count = 1
session.add(job)
session.commit()
def _fake_transcribe(_path):
raise AppError(
"temporary upstream outage",
category=ErrorCategory.EXTERNAL_PROVIDER,
suggestion="Retry from jobs page.",
retriable=True,
)
monkeypatch.setattr("transcription.worker.transcribe_document_image", _fake_transcribe)
monkeypatch.setattr(
"transcription.worker.get_settings",
lambda: Settings(
openrouter_api_key="test-key",
prompt_dir=tmp_path,
worker_max_retries=1,
worker_retry_backoff_seconds=0.0,
),
)
process_next_queued_job(session=session)
session.refresh(job)
assert job.status == JobStatus.FAILED
assert job.retry_count == 1
@pytest.mark.unit
class TestWorkerLoopControl:
"""Verify worker loop start/stop behavior."""
def test_stops_when_stop_event_is_set(self, monkeypatch):
"""run_worker_loop exits when a stop event is set."""
stop_event = Event()
stop_event.set()
called = {"value": False}
def _fake_process_next_queued_job(**_kwargs):
called["value"] = True
return False
monkeypatch.setattr("transcription.worker.process_next_queued_job", _fake_process_next_queued_job)
run_worker_loop(stop_event=stop_event, poll_interval_seconds=0.01)
assert called["value"] is False
-118
View File
@@ -1,118 +0,0 @@
"""Shared fixtures for UI integration tests."""
from __future__ import annotations
import asyncio
from collections.abc import Callable
from pathlib import Path
from uuid import UUID
import pytest
from fastapi import FastAPI
from fastapi.testclient import TestClient
from sqlmodel import delete
from transcription.app import create_app
from transcription.config import Settings
from transcription.config import _settings
from transcription.db import create_all
from transcription.db import get_session
from transcription.db import initialize_database_runtime
from transcription.models import Document
from transcription.models import Job
from transcription.models import JobStatus
from transcription.models import Transcript
TranscriptSeed = tuple[int, str | None, str | None]
@pytest.fixture(scope="session")
def app_client(tmp_path_factory: pytest.TempPathFactory) -> tuple[FastAPI, TestClient]:
"""Provide a real application and test client backed by in-memory SQLite."""
tmp_path = tmp_path_factory.mktemp("ui")
settings = Settings(
openrouter_api_key="test-key",
database_url="sqlite:///:memory:",
environment="test",
bootstrap_schema_on_startup=True,
upload_dir=tmp_path / "uploads",
prompt_dir=tmp_path / "prompts",
)
_settings.set(settings)
app = create_app()
app.state.runtime = initialize_database_runtime(settings=settings)
asyncio.run(create_all(engine=app.state.runtime.engine))
with TestClient(app) as client:
yield app, client
@pytest.fixture(autouse=True)
def clear_ui_database(app_client: tuple[FastAPI, TestClient]) -> None:
"""Reset UI-facing tables before each test for isolation."""
app, _ = app_client
async def _clear() -> None:
async with get_session(session_factory=app.state.runtime.session_factory) as session:
await session.exec(delete(Transcript))
await session.exec(delete(Job))
await session.exec(delete(Document))
await session.commit()
asyncio.run(_clear())
@pytest.fixture
def seed_job(app_client: tuple[FastAPI, TestClient]) -> Callable[..., UUID]:
"""Return a helper for inserting a document/job/transcript trio."""
app, _ = app_client
fixtures_dir = Path(__file__).resolve().parents[1] / "fixtures" / "images" / "valid"
def _seed(
*,
filename: str = "sample.pdf",
status: JobStatus = JobStatus.TRANSCRIBED,
transcript_text: str | None = "Sample transcript text",
error_detail: str | None = None,
transcript_revisions: list[TranscriptSeed] | None = None,
source_file: Path | None = None,
) -> UUID:
async def _insert() -> UUID:
async with get_session(session_factory=app.state.runtime.session_factory) as session:
stored_path = app.state.settings.upload_dir / filename
stored_path.parent.mkdir(parents=True, exist_ok=True)
source_path = source_file or fixtures_dir / "small_png.png"
stored_path.write_bytes(source_path.read_bytes())
document = Document(filename=filename, file_path=str(stored_path))
session.add(document)
await session.flush()
job = Job(document_id=document.id, status=status, retry_count=0)
session.add(job)
await session.flush()
revisions = transcript_revisions
if revisions is None and (transcript_text is not None or error_detail is not None):
revisions = [(0, transcript_text, error_detail)]
if revisions is not None:
for revision, revision_text, revision_error in revisions:
session.add(
Transcript(
job_id=job.id,
revision=revision,
provider="openrouter",
model="google/gemini-2.5-flash",
prompt_name="transcribe_document",
text=revision_text,
error_detail=revision_error,
)
)
await session.commit()
return job.id
return asyncio.run(_insert())
return _seed
+75 -57
View File
@@ -1,77 +1,95 @@
"""Tests for the jobs page route.""" """Tests for transcription.ui.jobs_page."""
from pathlib import Path
from uuid import uuid4 from uuid import uuid4
import pytest import pytest
from transcription.models import JobStatus from transcription.models import Document, Job, Transcript
from transcription.ui.jobs_page import fetch_job_detail, fetch_jobs
@pytest.mark.integration @pytest.mark.integration
class TestPageRendering: class TestJobsListBehavior:
"""Verify jobs routes render correctly with real app wiring.""" """Verify job list data and rendering helpers."""
def test_jobs_page_renders_empty_state(self, app_client): def test_fetch_jobs_returns_job_view_rows(self, session, monkeypatch):
"""GET /ui/jobs renders the page and empty-state text when no jobs exist.""" """fetch_jobs returns normalized JobView rows for UI consumption."""
_, client = app_client document = Document(filename="letter.jpg", file_path="uploads/letter.jpg")
response = client.get("/ui/jobs") session.add(document)
session.commit()
session.refresh(document)
assert response.status_code == 200 job = Job(document_id=document.id)
assert "Transcription Jobs" in response.text session.add(job)
assert "No jobs yet." in response.text session.commit()
def test_jobs_page_lists_seeded_jobs(self, app_client, seed_job): class _SessionContext:
"""GET /ui/jobs lists seeded jobs from the in-memory database.""" def __enter__(self):
_, client = app_client return session
seed_job(filename="sample.pdf", status=JobStatus.TRANSCRIBED, transcript_text="done")
response = client.get("/ui/jobs") def __exit__(self, exc_type, exc, tb):
return False
assert response.status_code == 200 monkeypatch.setattr("transcription.ui.jobs_page.get_session", lambda: _SessionContext())
assert "sample.pdf" in response.text
assert "transcribed" in response.text
def test_job_detail_page_renders_seeded_job(self, app_client, seed_job): rows = fetch_jobs()
"""GET /ui/jobs/{job_id} renders detail content for a real seeded job."""
_, client = app_client
fixture_path = Path(__file__).resolve().parents[1] / "fixtures" / "images" / "valid" / "single_page_pdf.pdf"
job_id = seed_job(
filename="detail.pdf",
status=JobStatus.TRANSCRIBED,
transcript_revisions=[
(0, None, "first attempt failed"),
(1, "hello", None),
],
source_file=fixture_path,
)
response = client.get(f"/ui/jobs/{job_id}") assert len(rows) == 1
assert rows[0].id == job.id
assert rows[0].status == "queued"
assert response.status_code == 200
assert "Job Detail" in response.text
assert "Job overview" in response.text
assert "detail.pdf" in response.text
assert "Transcripts" in response.text
assert "Revision" in response.text
assert "first attempt failed" in response.text
assert "hello" in response.text
assert "Document preview" in response.text
assert "/uploads/detail.pdf" in response.text
def test_job_detail_page_rejects_invalid_id(self, app_client): @pytest.mark.integration
"""GET /ui/jobs/{job_id} shows validation feedback for malformed IDs.""" class TestJobDetailBehavior:
_, client = app_client """Verify job detail retrieval behavior."""
response = client.get("/ui/jobs/not-a-uuid")
assert response.status_code == 200 def test_fetch_job_detail_returns_related_records_when_present(self, session, monkeypatch):
assert "Invalid job id" in response.text """fetch_job_detail returns job, document, and transcript when available."""
document = Document(filename="typed.jpg", file_path="uploads/typed.jpg")
session.add(document)
session.commit()
session.refresh(document)
def test_job_detail_page_handles_missing_job(self, app_client): job = Job(document_id=document.id)
"""GET /ui/jobs/{job_id} shows not-found state for unknown IDs.""" session.add(job)
_, client = app_client session.commit()
missing_id = uuid4() session.refresh(job)
response = client.get(f"/ui/jobs/{missing_id}")
assert response.status_code == 200 transcript = Transcript(job_id=job.id, text="Transcript text")
assert "Job not found" in response.text session.add(transcript)
session.commit()
class _SessionContext:
def __enter__(self):
return session
def __exit__(self, exc_type, exc, tb):
return False
monkeypatch.setattr("transcription.ui.jobs_page.get_session", lambda: _SessionContext())
fetched_job, fetched_document, fetched_transcript = fetch_job_detail(job.id)
assert fetched_job is not None
assert fetched_document is not None
assert fetched_transcript is not None
assert fetched_job.id == job.id
assert fetched_document.id == document.id
assert fetched_transcript.job_id == job.id
def test_fetch_job_detail_returns_nones_for_missing_job(self, session, monkeypatch):
"""fetch_job_detail returns triple None when job does not exist."""
class _SessionContext:
def __enter__(self):
return session
def __exit__(self, exc_type, exc, tb):
return False
monkeypatch.setattr("transcription.ui.jobs_page.get_session", lambda: _SessionContext())
fetched_job, fetched_document, fetched_transcript = fetch_job_detail(uuid4())
assert fetched_job is None
assert fetched_document is None
assert fetched_transcript is None
+16 -8
View File
@@ -1,18 +1,26 @@
"""Tests for UI page registration wiring.""" """Tests for UI page registration wiring."""
from fastapi import FastAPI
from fastapi.testclient import TestClient
import pytest import pytest
from transcription.ui import register_pages
@pytest.mark.integration @pytest.mark.integration
class TestPageRegistration: class TestPageRegistration:
"""Verify page registration and mounted UI routes.""" """Verify page registration and route wiring."""
def test_ui_mount_serves_registered_pages(self, app_client): def test_register_pages_adds_expected_routes(self):
"""Mounted UI routes respond successfully when the full app is created.""" """register_pages wires upload and jobs routes into the app."""
_, client = app_client app = FastAPI()
register_pages(app)
app.add_api_route("/healthz", lambda: {"status": "ok"}, methods=["GET"])
upload_response = client.get("/ui/upload") client = TestClient(app)
jobs_response = client.get("/ui/jobs") ui_response = client.get("/ui")
health_response = client.get("/healthz")
assert upload_response.status_code == 200 assert ui_response.status_code == 200
assert jobs_response.status_code == 200 assert health_response.status_code == 200
assert health_response.json() == {"status": "ok"}
+43 -25
View File
@@ -1,35 +1,53 @@
"""Tests for upload and entry-point routes.""" """Tests for transcription.ui.upload_page."""
from pathlib import Path
from uuid import uuid4
import pytest import pytest
from transcription.services.upload import UploadError, UploadJobResult
from transcription.ui import upload_page
@pytest.mark.integration
class TestPageRendering:
"""Verify upload-related routes return working pages."""
def test_root_redirects_to_ui(self, app_client): @pytest.mark.unit
"""GET / redirects to the UI mount point.""" class TestUploadPageBehavior:
_, client = app_client """Verify upload page helper and submission behavior."""
response = client.get("/", follow_redirects=False)
assert response.status_code == 307 def test_accepted_upload_types_contains_supported_extensions(self):
assert response.headers["location"] == "/ui" """accepted_upload_types includes all MVP-supported upload extensions."""
accepted = upload_page.accepted_upload_types()
assert ".jpg" in accepted
assert ".jpeg" in accepted
assert ".png" in accepted
assert ".tif" in accepted
assert ".tiff" in accepted
assert ".pdf" in accepted
def test_ui_redirects_to_upload(self, app_client): def test_submit_upload_calls_upload_service(self, monkeypatch):
"""GET /ui redirects to the upload page.""" """submit_upload delegates file persistence and job creation to upload service."""
_, client = app_client expected = UploadJobResult(
response = client.get("/ui", follow_redirects=False) document_id=uuid4(),
job_id=uuid4(),
stored_path=Path("uploads/mock.jpg"),
original_filename="mock.jpg",
)
assert response.status_code == 307 def fake_create_upload_job(*, filename: str, file_bytes: bytes):
assert response.headers["location"] == "/ui/upload" assert filename == "mock.jpg"
assert file_bytes == b"bytes"
return expected
def test_upload_page_renders_expected_controls(self, app_client): monkeypatch.setattr(upload_page, "create_upload_job", fake_create_upload_job)
"""GET /ui/upload returns the page shell and upload controls."""
_, client = app_client
response = client.get("/ui/upload")
assert response.status_code == 200 result = upload_page.submit_upload(filename="mock.jpg", file_bytes=b"bytes")
assert "Upload Document" in response.text assert result == expected
assert "Select document file" in response.text
assert "Upload" in response.text def test_submit_upload_surfaces_upload_error(self, monkeypatch):
assert "Jobs" in response.text """submit_upload raises UploadError for invalid upload payloads."""
def fake_create_upload_job(*, filename: str, file_bytes: bytes):
raise UploadError("invalid payload")
monkeypatch.setattr(upload_page, "create_upload_job", fake_create_upload_job)
with pytest.raises(UploadError):
upload_page.submit_upload(filename="bad.jpg", file_bytes=b"")
Generated
+1056 -1854
View File
File diff suppressed because it is too large Load Diff