Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
761765636a | ||
|
|
e2e421835f | ||
|
|
57c1d22bb9 | ||
|
|
dba96e7a72 | ||
|
|
912cfd44de | ||
|
|
8d5fee886f | ||
|
|
34468c521f | ||
|
|
b682e092ea | ||
|
|
bd33e338b3 | ||
|
|
57a988973f | ||
|
|
9ad67d55e8 | ||
|
|
7bdc0c6f79 | ||
|
|
4e4bf50219 | ||
|
|
c409b42077 | ||
|
|
f1fb45e0d2 | ||
|
|
f0501d919e | ||
|
|
517d01abe2 | ||
|
|
d967f58358 | ||
|
|
2000f0096b | ||
|
|
aa93080d7d | ||
|
|
52dbf70304 | ||
|
|
f0b359edf8 | ||
|
|
e6f12fa993 | ||
|
|
d3ffb01e93 | ||
|
|
f0b0109b11 | ||
|
|
f4417a0f64 | ||
|
|
cbb91c4cf6 | ||
|
|
2d73065d63 | ||
|
|
e75ca4c79a | ||
|
|
96cbadd56e | ||
|
|
f2aadf7e53 | ||
|
|
755f908b6a | ||
|
|
a16c6f5ecd | ||
|
|
621f508c26 | ||
|
|
b719d95f4b | ||
|
|
5ef74ef33a | ||
|
|
e4889ba584 |
@@ -0,0 +1,13 @@
|
|||||||
|
.git
|
||||||
|
.gitignore
|
||||||
|
.vscode
|
||||||
|
.venv
|
||||||
|
.pytest_cache
|
||||||
|
.ruff_cache
|
||||||
|
__pycache__/
|
||||||
|
*.py[cod]
|
||||||
|
*.db
|
||||||
|
.env
|
||||||
|
tests/
|
||||||
|
docs/
|
||||||
|
uploads/
|
||||||
@@ -0,0 +1,77 @@
|
|||||||
|
---
|
||||||
|
description: Follow these guidelines when editing the services
|
||||||
|
applyTo: 'src/transcription/services/*.py'
|
||||||
|
---
|
||||||
|
|
||||||
|
# Services
|
||||||
|
|
||||||
|
## Structure
|
||||||
|
|
||||||
|
- Project core data models defined in [models](../../src/transcription/models.py)
|
||||||
|
- 1 service class per data model
|
||||||
|
- Only services directly interact with the database, and only through async methods
|
||||||
|
- Services are completely independent of one another. Any operation that needs to use more than a single service, which is most of them, needs to have a separate orchestration function.
|
||||||
|
|
||||||
|
## Error Handling
|
||||||
|
|
||||||
|
- Service-specific errors defined at the top of the respective module and inherit from `AppError`
|
||||||
|
- Use a context manager for large `try/except` blocks like in [transcription](../../src/transcription/services/transcription.py)
|
||||||
|
|
||||||
|
## Checklist
|
||||||
|
|
||||||
|
- [ ] Uses `ServiceBase` for common logic
|
||||||
|
- [ ] CRUD methods created at the top
|
||||||
|
- [ ] Session kwarg for `AsyncSession` to pass in a session object to each method
|
||||||
|
- [ ] Services use `self._session_scope` in their methods to pass the session thru.
|
||||||
|
- Multiple operations on the same object(s) require sharing a session between all the methods used.
|
||||||
|
|
||||||
|
## CRUD Methods
|
||||||
|
|
||||||
|
- Create, read, update, and delete, created in that order
|
||||||
|
- Name format `<operation>_<model >`, for example `create_document` or `update_job`
|
||||||
|
- All services must define these 4 methods first, and in that order
|
||||||
|
|
||||||
|
## Transaction Finalization
|
||||||
|
|
||||||
|
When a service method accepts an optional `session` kwarg, write methods must use `self._finalize` to finalize the transaction properly according to whether or not they are sharing a session.
|
||||||
|
|
||||||
|
- If `session` is `None`: the method owns the transaction and should `commit()`.
|
||||||
|
- If `session` is provided: the method must **not** commit; it should `flush()` so IDs and FK values are available to the caller's transaction.
|
||||||
|
- Use `refresh()` on returned ORM objects when the caller needs DB-populated values (defaults, triggers, merged state).
|
||||||
|
|
||||||
|
Recommended helper behavior:
|
||||||
|
|
||||||
|
- Inputs: active session object, original `session` arg (or a boolean ownership flag), and an optional list of objects to refresh.
|
||||||
|
- Logic: `commit` when service-owned session, `flush` when caller-owned session, then refresh requested objects.
|
||||||
|
|
||||||
|
This keeps orchestration functions atomic: they can pass one shared session across multiple services and commit exactly once at the workflow boundary.
|
||||||
|
|
||||||
|
## Workflow Transaction Boundaries
|
||||||
|
|
||||||
|
For multi-step job lifecycles (for example queued transcription jobs), orchestration functions must use explicit transaction phases.
|
||||||
|
|
||||||
|
Required boundary model:
|
||||||
|
|
||||||
|
- **Transaction A (claim):** transition `JobStatus.QUEUED -> JobStatus.PROCESSING` and commit immediately.
|
||||||
|
- Perform provider/network work **outside** database transactions.
|
||||||
|
- **Transaction B (terminal success):** write transcript content and set `JobStatus.TRANSCRIBED` in the same shared-session commit.
|
||||||
|
- **Transaction B (terminal failure):** write transcript error detail and set `JobStatus.FAILED` in the same shared-session commit.
|
||||||
|
- **Transaction C (retry path):** write transcript error detail, increment retry count, and set `JobStatus.QUEUED` in one shared-session commit.
|
||||||
|
|
||||||
|
Atomicity rules:
|
||||||
|
|
||||||
|
- Never commit transcript updates separately from the paired terminal/retry job status change.
|
||||||
|
- Terminal state (`TRANSCRIBED` or `FAILED`) and transcript row changes must succeed or roll back together.
|
||||||
|
- Retry persistence (`QUEUED` + retry increment + error detail) must succeed or roll back together.
|
||||||
|
|
||||||
|
Separation of concerns:
|
||||||
|
|
||||||
|
- Worker modules should stay lightweight and delegate lifecycle transitions to service/workflow orchestration functions.
|
||||||
|
- In `workflows.py`, `process_queued_job` should own one complete attempt lifecycle: `QUEUED -> PROCESSING -> TRANSCRIBED|FAILED`.
|
||||||
|
- In `workflows.py`, `advance_job` should coordinate broader status progression around attempts (for example retry scheduling from `FAILED -> QUEUED`).
|
||||||
|
- Services should expose session-aware write helpers (flush on caller-owned session) so orchestration controls commit boundaries.
|
||||||
|
- Backoff/sleep behavior must run outside transactional scopes.
|
||||||
|
|
||||||
|
# Service Composition
|
||||||
|
|
||||||
|
Some operations, like uploading a picutre, require modifications to multiple tables, which can be done by composing methods from the service object into a separate function.
|
||||||
@@ -0,0 +1,27 @@
|
|||||||
|
{
|
||||||
|
"version": "0.2.0",
|
||||||
|
"configurations": [
|
||||||
|
{
|
||||||
|
"name": "Python: Debug transcription app",
|
||||||
|
"type": "debugpy",
|
||||||
|
"request": "launch",
|
||||||
|
"module": "debugpy",
|
||||||
|
"args": [
|
||||||
|
"-m",
|
||||||
|
"uvicorn",
|
||||||
|
"transcription.app:create_app",
|
||||||
|
"--factory",
|
||||||
|
"--host",
|
||||||
|
// "127.0.0.1",
|
||||||
|
"0.0.0.0",
|
||||||
|
"--port",
|
||||||
|
"8080"
|
||||||
|
],
|
||||||
|
"justMyCode": true,
|
||||||
|
"console": "integratedTerminal",
|
||||||
|
"env": {
|
||||||
|
"PYTHONPATH": "${workspaceFolder}/src"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
@@ -0,0 +1,47 @@
|
|||||||
|
FROM python:3.12-slim AS builder
|
||||||
|
|
||||||
|
ENV PYTHONDONTWRITEBYTECODE=1 \
|
||||||
|
PYTHONUNBUFFERED=1 \
|
||||||
|
UV_LINK_MODE=copy
|
||||||
|
|
||||||
|
WORKDIR /app
|
||||||
|
|
||||||
|
COPY --from=ghcr.io/astral-sh/uv:0.5.24 /uv /uvx /bin/
|
||||||
|
|
||||||
|
COPY pyproject.toml uv.lock README.md ./
|
||||||
|
RUN uv sync --frozen --no-dev --no-install-project
|
||||||
|
|
||||||
|
COPY src ./src
|
||||||
|
COPY prompts ./prompts
|
||||||
|
RUN uv sync --frozen --no-dev
|
||||||
|
|
||||||
|
|
||||||
|
FROM python:3.12-slim AS runtime
|
||||||
|
|
||||||
|
ENV PYTHONDONTWRITEBYTECODE=1 \
|
||||||
|
PYTHONUNBUFFERED=1 \
|
||||||
|
PATH="/app/.venv/bin:$PATH" \
|
||||||
|
PYTHONPATH="/app/src" \
|
||||||
|
UPLOAD_DIR="/app/uploads" \
|
||||||
|
PROMPT_DIR="/app/prompts"
|
||||||
|
|
||||||
|
WORKDIR /app
|
||||||
|
|
||||||
|
RUN groupadd --system --gid 1001 appgroup \
|
||||||
|
&& useradd --system --uid 1001 --gid appgroup --create-home appuser
|
||||||
|
|
||||||
|
COPY --from=builder /app/.venv /app/.venv
|
||||||
|
COPY --from=builder /app/src /app/src
|
||||||
|
COPY --from=builder /app/prompts /app/prompts
|
||||||
|
|
||||||
|
RUN mkdir -p /app/uploads /app/data \
|
||||||
|
&& chown -R appuser:appgroup /app
|
||||||
|
|
||||||
|
USER appuser
|
||||||
|
|
||||||
|
EXPOSE 8000
|
||||||
|
|
||||||
|
HEALTHCHECK --interval=30s --timeout=3s --start-period=3s --retries=3 \
|
||||||
|
CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:8000/healthz')"
|
||||||
|
|
||||||
|
CMD ["uvicorn", "transcription.app:create_app", "--factory", "--host", "0.0.0.0", "--port", "8000", "--proxy-headers"]
|
||||||
@@ -10,6 +10,7 @@ The app lets you upload a document image/PDF, queues a background transcription
|
|||||||
- Persist document + job records in SQLite
|
- Persist document + job records in SQLite
|
||||||
- Process jobs in a background worker (`queued -> processing -> transcribed/failed`)
|
- Process jobs in a background worker (`queued -> processing -> transcribed/failed`)
|
||||||
- Store transcript text (or failure detail)
|
- Store transcript text (or failure detail)
|
||||||
|
- Track transcript revisions (AI-generated and manual updates)
|
||||||
- Show status and results in the NiceGUI interface
|
- Show status and results in the NiceGUI interface
|
||||||
|
|
||||||
## Quick start
|
## Quick start
|
||||||
@@ -61,8 +62,11 @@ uv run uvicorn transcription.app:create_app --factory --reload
|
|||||||
|
|
||||||
- **Job detail page** (`/ui/jobs/{job_id}`)
|
- **Job detail page** (`/ui/jobs/{job_id}`)
|
||||||
- Shows job metadata and status.
|
- Shows job metadata and status.
|
||||||
- Displays transcript text when successful.
|
- Shows transcript metadata, including provider and model.
|
||||||
- Displays failure detail when transcription fails.
|
- Shows a version table with `Created` and `Version`.
|
||||||
|
- Displays latest version text in an editable textbox.
|
||||||
|
- **Update** creates a new transcript version.
|
||||||
|
- Displays failure detail for failed revisions.
|
||||||
|
|
||||||
## Prompt artifacts
|
## Prompt artifacts
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,21 @@
|
|||||||
|
services:
|
||||||
|
transcription:
|
||||||
|
build:
|
||||||
|
context: .
|
||||||
|
dockerfile: Dockerfile
|
||||||
|
container_name: transcription-app
|
||||||
|
env_file:
|
||||||
|
- .env
|
||||||
|
environment:
|
||||||
|
DATABASE_URL: sqlite:////app/data/transcription.db
|
||||||
|
UPLOAD_DIR: /app/uploads
|
||||||
|
PROMPT_DIR: /app/prompts
|
||||||
|
ports:
|
||||||
|
- "8002:8000"
|
||||||
|
volumes:
|
||||||
|
- ./uploads:/app/uploads
|
||||||
|
- transcription_data:/app/data
|
||||||
|
restart: unless-stopped
|
||||||
|
|
||||||
|
volumes:
|
||||||
|
transcription_data:
|
||||||
@@ -1,35 +0,0 @@
|
|||||||
# ADR-0001: Lifespan-owned runtime resources
|
|
||||||
|
|
||||||
- **Status:** accepted
|
|
||||||
- **Date:** 2026-06-25
|
|
||||||
|
|
||||||
## Context
|
|
||||||
|
|
||||||
MVP initialized core runtime resources (database engine and worker dependencies) through module-level globals and startup side effects. `REQ-7` requires lifespan-owned runtime resources with explicit ownership and cleanup.
|
|
||||||
|
|
||||||
## Decision
|
|
||||||
|
|
||||||
Adopt lifespan-owned runtime resource initialization in `transcription.app`:
|
|
||||||
|
|
||||||
1. Initialize database runtime during app lifespan startup.
|
|
||||||
2. Store runtime handles on `app.state`.
|
|
||||||
3. Pass runtime-owned dependencies (engine) to worker startup.
|
|
||||||
4. Dispose runtime resources explicitly during lifespan shutdown.
|
|
||||||
|
|
||||||
## Consequences
|
|
||||||
|
|
||||||
### Positive
|
|
||||||
- Explicit startup and shutdown ownership.
|
|
||||||
- Predictable cleanup ordering.
|
|
||||||
- Reduced hidden global side effects.
|
|
||||||
|
|
||||||
### Tradeoffs
|
|
||||||
- Minor wiring complexity in app startup.
|
|
||||||
- Some call-sites still support fallback lazy initialization for compatibility.
|
|
||||||
|
|
||||||
## Alternatives Considered
|
|
||||||
|
|
||||||
1. **Keep module-level global ownership**
|
|
||||||
- Rejected: conflicts with `REQ-7` and increases ambiguity.
|
|
||||||
2. **Introduce full async DB stack immediately**
|
|
||||||
- Rejected for Step 1: too broad for architecture-consolidation scope.
|
|
||||||
@@ -1,36 +0,0 @@
|
|||||||
# ADR-0002: Explicit schema bootstrap policy
|
|
||||||
|
|
||||||
- **Status:** accepted
|
|
||||||
- **Date:** 2026-06-25
|
|
||||||
|
|
||||||
## Context
|
|
||||||
|
|
||||||
MVP called schema bootstrap (`create_all`) on every startup. `REQ-10` requires explicit, opt-in schema bootstrap behavior so normal production startup does not mutate schema.
|
|
||||||
|
|
||||||
## Decision
|
|
||||||
|
|
||||||
Add environment-aware bootstrap policy:
|
|
||||||
|
|
||||||
1. New settings:
|
|
||||||
- `environment`: `development` | `test` | `production`
|
|
||||||
- `bootstrap_schema_on_startup`: optional explicit override
|
|
||||||
2. Default behavior:
|
|
||||||
- Development/test: bootstrap enabled
|
|
||||||
- Production: bootstrap disabled
|
|
||||||
3. App startup calls `create_all` only when policy evaluates true.
|
|
||||||
|
|
||||||
## Consequences
|
|
||||||
|
|
||||||
### Positive
|
|
||||||
- Production startup behavior is safer and policy-driven.
|
|
||||||
- Local development remains simple by default.
|
|
||||||
|
|
||||||
### Tradeoffs
|
|
||||||
- Deployments now require explicit schema management in production.
|
|
||||||
|
|
||||||
## Alternatives Considered
|
|
||||||
|
|
||||||
1. **Always bootstrap in all environments**
|
|
||||||
- Rejected: violates `REQ-10` intent.
|
|
||||||
2. **Disable bootstrap everywhere immediately**
|
|
||||||
- Rejected: hurts local developer workflow without migration tool replacement yet.
|
|
||||||
@@ -1,31 +0,0 @@
|
|||||||
# ADR-0003: Persistence baseline and transition path
|
|
||||||
|
|
||||||
- **Status:** accepted
|
|
||||||
- **Date:** 2026-06-25
|
|
||||||
|
|
||||||
## Context
|
|
||||||
|
|
||||||
Architecture targets PostgreSQL baseline (optional MongoDB), while MVP currently runs on SQLite by default. V1 needs a clear transition path without destabilizing ongoing work.
|
|
||||||
|
|
||||||
## Decision
|
|
||||||
|
|
||||||
1. Preserve database URL configurability through centralized settings.
|
|
||||||
2. Keep SQLite functional for local dev/test and fast feedback.
|
|
||||||
3. Treat PostgreSQL as production baseline target for V1 completion.
|
|
||||||
4. Keep persistence access behind `transcription.db` runtime/session access points.
|
|
||||||
|
|
||||||
## Consequences
|
|
||||||
|
|
||||||
### Positive
|
|
||||||
- Clear migration path without immediate broad rewrite.
|
|
||||||
- Controlled risk while preserving velocity.
|
|
||||||
|
|
||||||
### Tradeoffs
|
|
||||||
- Temporary dual-path assumptions (SQLite local vs PostgreSQL target).
|
|
||||||
|
|
||||||
## Alternatives Considered
|
|
||||||
|
|
||||||
1. **Immediate forced PostgreSQL-only migration**
|
|
||||||
- Rejected: higher short-term disruption risk.
|
|
||||||
2. **Remain SQLite-only for V1**
|
|
||||||
- Rejected: inconsistent with architecture and requirement trajectory.
|
|
||||||
@@ -1,32 +0,0 @@
|
|||||||
# ADR-0004: In-process worker topology for V1
|
|
||||||
|
|
||||||
- **Status:** accepted
|
|
||||||
- **Date:** 2026-06-25
|
|
||||||
|
|
||||||
## Context
|
|
||||||
|
|
||||||
The current system uses an in-process background worker. Architecture docs allow this in foundation stage and permit later hardening (optional external worker/queue).
|
|
||||||
|
|
||||||
## Decision
|
|
||||||
|
|
||||||
Retain in-process worker topology for V1, with improved lifecycle ownership:
|
|
||||||
|
|
||||||
1. Worker starts/stops via app lifespan.
|
|
||||||
2. Worker receives runtime-owned DB engine dependency explicitly.
|
|
||||||
3. Extension path to external worker remains behind existing service/adapter seams.
|
|
||||||
|
|
||||||
## Consequences
|
|
||||||
|
|
||||||
### Positive
|
|
||||||
- Keeps operational complexity low for personal-scale use.
|
|
||||||
- Preserves delivery focus on V1 completion.
|
|
||||||
|
|
||||||
### Tradeoffs
|
|
||||||
- Throughput/scaling limits remain compared to external queue-based topology.
|
|
||||||
|
|
||||||
## Alternatives Considered
|
|
||||||
|
|
||||||
1. **Immediate queue/external worker introduction**
|
|
||||||
- Rejected: premature complexity for current scale.
|
|
||||||
2. **Ad hoc thread lifecycle management outside lifespan**
|
|
||||||
- Rejected: weaker shutdown guarantees and poorer ownership clarity.
|
|
||||||
@@ -1,20 +0,0 @@
|
|||||||
# Architecture Decision Records (ADRs)
|
|
||||||
|
|
||||||
This directory records significant architecture decisions for Version 1.
|
|
||||||
|
|
||||||
## ADR Format
|
|
||||||
|
|
||||||
Each ADR should include:
|
|
||||||
|
|
||||||
1. **Status** (`proposed`, `accepted`, `superseded`)
|
|
||||||
2. **Context**
|
|
||||||
3. **Decision**
|
|
||||||
4. **Consequences**
|
|
||||||
5. **Alternatives Considered**
|
|
||||||
|
|
||||||
## Index
|
|
||||||
|
|
||||||
- [ADR-0001: Lifespan-owned runtime resources](ADR-0001-lifespan-owned-runtime-resources.md)
|
|
||||||
- [ADR-0002: Explicit schema bootstrap policy](ADR-0002-explicit-schema-bootstrap-policy.md)
|
|
||||||
- [ADR-0003: Persistence baseline and transition path](ADR-0003-persistence-baseline-and-transition-path.md)
|
|
||||||
- [ADR-0004: In-process worker topology for V1](ADR-0004-in-process-worker-topology.md)
|
|
||||||
@@ -1,86 +0,0 @@
|
|||||||
# Ver1 Step 1 Results: Architecture Consolidation
|
|
||||||
|
|
||||||
## Summary
|
|
||||||
|
|
||||||
Step 1 implementation has been completed for the primary architecture-consolidation objectives:
|
|
||||||
|
|
||||||
1. Lifespan-owned runtime resource model introduced for DB runtime ownership.
|
|
||||||
2. Schema bootstrap policy changed from implicit-always to explicit/environment-aware.
|
|
||||||
3. Worker startup now receives lifespan-owned DB engine dependency.
|
|
||||||
4. ADR set established for key V1 architectural decisions.
|
|
||||||
|
|
||||||
## Implemented Changes
|
|
||||||
|
|
||||||
### 1) Runtime ownership
|
|
||||||
|
|
||||||
- Updated `src/transcription/db.py`:
|
|
||||||
- Added `DatabaseRuntime` resource model.
|
|
||||||
- Added explicit runtime lifecycle methods:
|
|
||||||
- `initialize_database_runtime(...)`
|
|
||||||
- `get_database_runtime()`
|
|
||||||
- `dispose_database_runtime()`
|
|
||||||
- Updated `src/transcription/app.py`:
|
|
||||||
- Lifespan initializes DB runtime and stores it on `app.state`.
|
|
||||||
- Lifespan disposes DB runtime on shutdown.
|
|
||||||
|
|
||||||
### 2) Schema bootstrap policy (REQ-10 alignment)
|
|
||||||
|
|
||||||
- Updated `src/transcription/config.py`:
|
|
||||||
- Added `environment` setting (`development`, `test`, `production`).
|
|
||||||
- Added `bootstrap_schema_on_startup` explicit override setting.
|
|
||||||
- Updated `src/transcription/db.py`:
|
|
||||||
- Added `should_bootstrap_schema(settings)` policy function.
|
|
||||||
- Updated `src/transcription/app.py`:
|
|
||||||
- Startup now calls `create_all(...)` only when policy allows.
|
|
||||||
|
|
||||||
### 3) Worker dependency ownership
|
|
||||||
|
|
||||||
- Updated `src/transcription/worker.py`:
|
|
||||||
- `process_next_queued_job(..., engine=None)` now supports explicit engine injection.
|
|
||||||
- `run_worker_loop(..., engine=None, ...)` now supports explicit engine injection.
|
|
||||||
- Updated `src/transcription/app.py`:
|
|
||||||
- Worker thread is started with lifespan-owned engine.
|
|
||||||
|
|
||||||
### 4) ADR governance
|
|
||||||
|
|
||||||
Created:
|
|
||||||
- `docs/adr/README.md`
|
|
||||||
- `docs/adr/ADR-0001-lifespan-owned-runtime-resources.md`
|
|
||||||
- `docs/adr/ADR-0002-explicit-schema-bootstrap-policy.md`
|
|
||||||
- `docs/adr/ADR-0003-persistence-baseline-and-transition-path.md`
|
|
||||||
- `docs/adr/ADR-0004-in-process-worker-topology.md`
|
|
||||||
|
|
||||||
## Test Evidence
|
|
||||||
|
|
||||||
Targeted regression checks executed successfully:
|
|
||||||
|
|
||||||
- `uv run pytest tests/test_app.py tests/test_db.py tests/services/test_worker.py -q`
|
|
||||||
- Result: pass
|
|
||||||
|
|
||||||
## Residual Risks / Follow-ups
|
|
||||||
|
|
||||||
1. Full REQ-7 completion may still require broader runtime ownership coverage for additional resources as V1 expands.
|
|
||||||
2. Production schema management workflow (migrations/runbook tooling) should be finalized in subsequent V1 steps.
|
|
||||||
3. Additional boundary enforcement automation (import-lint style checks) can be added in later hardening.
|
|
||||||
|
|
||||||
## Step 1 Exit Assessment
|
|
||||||
|
|
||||||
- Architecture ownership clarity: **met**
|
|
||||||
- Schema bootstrap policy hardening: **met**
|
|
||||||
- Worker lifecycle dependency clarity: **met**
|
|
||||||
- ADR baseline established: **met**
|
|
||||||
|
|
||||||
## Completion Checklist With Evidence
|
|
||||||
|
|
||||||
| Criterion | Status | Evidence |
|
|
||||||
| --- | --- | --- |
|
|
||||||
| Architecture conformance matrix approved | partial | Consolidation implemented and documented in `docs/ver1/ver1-step1.md` + this results doc; formal matrix artifact can be added as a follow-up appendix. |
|
|
||||||
| REQ-7 ownership gaps resolved or explicitly deferred | met | Lifespan-owned DB runtime and explicit worker engine wiring implemented in `src/transcription/app.py`, `src/transcription/db.py`, `src/transcription/worker.py`. Residual scope documented under follow-ups. |
|
|
||||||
| REQ-10 explicit bootstrap policy implemented and verified | met | Policy implemented via `environment` + `bootstrap_schema_on_startup` in `src/transcription/config.py`, `should_bootstrap_schema(...)` in `src/transcription/db.py`, startup gate in `src/transcription/app.py`, tested in `tests/test_db.py`. |
|
|
||||||
| Dependency direction rules documented and enforced | partial | Layering and runtime ownership documented in `docs/architecture.md`. Lightweight enforcement exists via review and test discipline; automated import-lint remains a follow-up. |
|
|
||||||
| ADR set created for major Step 1 decisions | met | `docs/adr/README.md` and ADR-0001 through ADR-0004 created. |
|
|
||||||
| Architecture/index docs updated to match implementation | met | `docs/architecture.md` and `docs/index.md` updated with V1 Step 1 runtime policy and links to V1/ADR artifacts. |
|
|
||||||
| Regression and full test suites pass | met | Targeted: `uv run pytest tests/test_app.py tests/test_db.py tests/services/test_worker.py -q`; full suite: `uv run pytest -q`. |
|
|
||||||
| Step 1 results artifact published | met | This document (`docs/ver1/ver1-step1-results.md`) created and updated with summary, evidence, risks, and checklist. |
|
|
||||||
|
|
||||||
Step 1 is complete and ready to hand off to Ver1 Step 2.
|
|
||||||
@@ -1,309 +0,0 @@
|
|||||||
# Step 1 Implementation Plan: Architecture Consolidation
|
|
||||||
|
|
||||||
## Purpose
|
|
||||||
|
|
||||||
Align the implemented MVP codebase with the production architecture and V1 constraints documented in:
|
|
||||||
|
|
||||||
- `docs/architecture.md`
|
|
||||||
- `docs/requirements.md`
|
|
||||||
- `docs/error_handling.md`
|
|
||||||
- `docs/index.md`
|
|
||||||
- `docs/intent.md`
|
|
||||||
- `docs/ver1/ver1.md` (Step 1)
|
|
||||||
|
|
||||||
This step hardens architecture boundaries and ownership without expanding product scope.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## MCP Skill and Guide Inputs Incorporated
|
|
||||||
|
|
||||||
This plan explicitly incorporates patterns and guardrails from john-stream-mcp resources:
|
|
||||||
|
|
||||||
1. `resource://skills/fastapi-uv-docker/document`
|
|
||||||
- App factory and lifespan ownership
|
|
||||||
- Health endpoint and cloud-native baseline expectations
|
|
||||||
- Environment-driven configuration and startup discipline
|
|
||||||
|
|
||||||
2. `resource://skills/fastapi-async-sqlalchemy-modernization/document`
|
|
||||||
- Current-state gap audit first
|
|
||||||
- Target runtime model before refactor
|
|
||||||
- Explicit resource lifecycle ownership
|
|
||||||
- Transaction/session boundary clarity
|
|
||||||
- Phased migration with rollback points
|
|
||||||
|
|
||||||
3. `resource://skills/nicegui/document`
|
|
||||||
- Clear dependency direction
|
|
||||||
- UI/page registration as composition, not business logic container
|
|
||||||
- Async responsiveness and boundary separation
|
|
||||||
|
|
||||||
4. `resource://prompts/greenfield-architecture/document`
|
|
||||||
- Pattern-comparison-first planning
|
|
||||||
- Explicit tradeoffs and staged implementation
|
|
||||||
- Output contract with risks, open questions, and next steps
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Current-State Gap Summary (Architecture vs Implementation)
|
|
||||||
|
|
||||||
Based on docs and current `src/transcription` code:
|
|
||||||
|
|
||||||
1. **REQ-7 gap (lifespan-owned resources)**
|
|
||||||
- DB engine/session factory are module globals in `db.py`, not app lifespan-owned.
|
|
||||||
- Worker thread lifecycle is owned by lifespan (good), but DB/provider resource ownership is mixed.
|
|
||||||
|
|
||||||
2. **REQ-10 gap (explicit opt-in schema bootstrap)**
|
|
||||||
- `create_all()` is executed unconditionally on startup in `app.py`.
|
|
||||||
|
|
||||||
3. **Data store target gap (REQ-9 + architecture baseline)**
|
|
||||||
- Runtime still defaults to SQLite MVP setup; production architecture targets PostgreSQL baseline with optional MongoDB.
|
|
||||||
|
|
||||||
4. **Layering clarity gap (architecture layer model)**
|
|
||||||
- Boundaries exist but are not yet formally enforced (interface/app/domain/infra dependency rules are implicit, not codified).
|
|
||||||
|
|
||||||
5. **Decision record gap**
|
|
||||||
- No ADR set documenting key V1 architectural decisions and deviations from MVP.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Scope for Step 1
|
|
||||||
|
|
||||||
### In scope
|
|
||||||
1. Produce architecture conformance audit and decision records.
|
|
||||||
2. Define and implement target runtime ownership model for core resources.
|
|
||||||
3. Establish explicit schema bootstrap policy (opt-in in production paths).
|
|
||||||
4. Consolidate module boundaries and dependency direction rules.
|
|
||||||
5. Update architecture docs to reflect implemented reality and V1 trajectory.
|
|
||||||
|
|
||||||
### Out of scope
|
|
||||||
- Full async SQLAlchemy rewrite (plan and seams only if deferred)
|
|
||||||
- MongoDB feature implementation
|
|
||||||
- New user-facing features
|
|
||||||
- Major worker architecture replacement (in-process worker remains baseline)
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Target Architecture Decisions for V1
|
|
||||||
|
|
||||||
1. **Keep modular monolith topology** (FastAPI + NiceGUI + in-process worker).
|
|
||||||
2. **Preserve container-light simplicity guardrails** from `architecture.md`.
|
|
||||||
3. **Move runtime ownership to lifespan** for:
|
|
||||||
- DB engine/session factory lifecycle
|
|
||||||
- Worker runtime resources
|
|
||||||
- Provider client factory/config lifecycle
|
|
||||||
4. **Adopt explicit schema bootstrap policy**:
|
|
||||||
- Dev/test: opt-in auto-bootstrap allowed
|
|
||||||
- Production: startup must not mutate schema implicitly
|
|
||||||
5. **Formalize boundary map**:
|
|
||||||
- Interface (`api`, `ui`) -> Application (`services`) -> Domain (`models/rules`) -> Infrastructure (`db`, `providers`)
|
|
||||||
- No reverse imports
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Detailed Work Breakdown
|
|
||||||
|
|
||||||
## Phase A — Architecture Audit and Baseline Freeze
|
|
||||||
|
|
||||||
- [ ] **A1. Produce architecture conformance matrix**
|
|
||||||
- Map each architecture section to current modules/files.
|
|
||||||
- Classify each row: `aligned`, `partial`, `not aligned`.
|
|
||||||
|
|
||||||
- [ ] **A2. Produce REQ-7/REQ-9/REQ-10 focused gap report**
|
|
||||||
- Explicitly capture current vs required state.
|
|
||||||
- Include operational risk if left unresolved.
|
|
||||||
|
|
||||||
- [ ] **A3. Freeze MVP architecture baseline**
|
|
||||||
- Record current baseline behavior and known temporary shortcuts.
|
|
||||||
- Link this baseline from `docs/ver1/ver1.md`.
|
|
||||||
|
|
||||||
### Deliverables
|
|
||||||
- `docs/ver1/ver1-step1-audit.md` (or equivalent section in this doc)
|
|
||||||
- Architecture conformance table
|
|
||||||
|
|
||||||
### Exit Criteria
|
|
||||||
- No architecture changes begin before gap matrix and baseline are approved.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Phase B — Resource Ownership Consolidation (Lifespan-Centric)
|
|
||||||
|
|
||||||
- [ ] **B1. Define runtime resource ownership contract**
|
|
||||||
- `app.py` lifespan owns resource initialization and cleanup order.
|
|
||||||
- `app.state` carries resource handles/factories.
|
|
||||||
- No hidden module-global side-effect initialization for runtime resources.
|
|
||||||
|
|
||||||
- [ ] **B2. Refactor DB ownership model**
|
|
||||||
- Replace module-global engine singleton pattern with lifespan-initialized resource model.
|
|
||||||
- Define one canonical session-factory access path for app/worker/services.
|
|
||||||
|
|
||||||
- [ ] **B3. Normalize worker dependencies**
|
|
||||||
- Ensure worker uses lifespan-owned resources/factories rather than implicit globals.
|
|
||||||
- Preserve deterministic startup/shutdown behavior.
|
|
||||||
|
|
||||||
- [ ] **B4. Define provider adapter ownership**
|
|
||||||
- Provider client creation strategy is centralized and lifecycle-aware.
|
|
||||||
- Avoid per-call hidden client construction when unnecessary.
|
|
||||||
|
|
||||||
### MCP-Guided Guardrails
|
|
||||||
- Use explicit lifecycle composition patterns from `fastapi-async-sqlalchemy-modernization`.
|
|
||||||
- Maintain app-factory + lifespan structure per `fastapi-uv-docker`.
|
|
||||||
- Keep UI registration as composition only per `nicegui`.
|
|
||||||
|
|
||||||
### Exit Criteria
|
|
||||||
- Core runtime resources have one owner and one cleanup path.
|
|
||||||
- No critical resource has ambiguous ownership.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Phase C — Schema Bootstrap Policy (REQ-10 Alignment)
|
|
||||||
|
|
||||||
- [ ] **C1. Define environment-aware bootstrap policy**
|
|
||||||
- `auto_create_schema` (or equivalent) disabled in production by default.
|
|
||||||
- Startup schema mutation is explicit and intentional.
|
|
||||||
|
|
||||||
- [ ] **C2. Split startup responsibilities**
|
|
||||||
- App startup performs health-critical initialization only.
|
|
||||||
- Schema bootstrap path is moved to explicit command/flag workflow.
|
|
||||||
|
|
||||||
- [ ] **C3. Update deployment/runbook docs**
|
|
||||||
- Document migration/bootstrap flow for dev, staging, prod.
|
|
||||||
- Ensure policy is testable and auditable.
|
|
||||||
|
|
||||||
### Exit Criteria
|
|
||||||
- Normal production startup path does not call schema auto-create implicitly.
|
|
||||||
- Bootstrap behavior is explicit and documented.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Phase D — Module Boundary Enforcement
|
|
||||||
|
|
||||||
- [ ] **D1. Publish dependency direction rules**
|
|
||||||
- Allowed import directions across `api`, `ui`, `services`, `models/domain`, `db/providers`.
|
|
||||||
- Explicitly disallow reverse dependencies.
|
|
||||||
|
|
||||||
- [ ] **D2. Reconcile package map with docs**
|
|
||||||
- Ensure docs’ architecture elements match real package layout and naming.
|
|
||||||
- Update docs where intentional deviations remain.
|
|
||||||
|
|
||||||
- [ ] **D3. Isolate cross-layer responsibilities**
|
|
||||||
- Keep API/UI presentation concerns out of services.
|
|
||||||
- Keep provider/DB specifics out of interface layer.
|
|
||||||
|
|
||||||
- [ ] **D4. Add lightweight architecture checks**
|
|
||||||
- Add static/import checks and/or review checklist in CI/review process.
|
|
||||||
|
|
||||||
### Exit Criteria
|
|
||||||
- Boundary rules are documented and applied.
|
|
||||||
- Architectural drift can be detected during review/CI.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Phase E — Architecture Decision Records (ADRs)
|
|
||||||
|
|
||||||
- [ ] **E1. Create ADR index**
|
|
||||||
- Add `docs/adr/README.md` with template and status model.
|
|
||||||
|
|
||||||
- [ ] **E2. Record minimum V1 ADR set**
|
|
||||||
1. Runtime ownership model (lifespan-owned resources)
|
|
||||||
2. Schema bootstrap policy (explicit vs implicit)
|
|
||||||
3. Persistence baseline (PostgreSQL target; SQLite transition strategy)
|
|
||||||
4. Worker topology (in-process for V1, extension path preserved)
|
|
||||||
|
|
||||||
- [ ] **E3. Cross-link ADRs**
|
|
||||||
- Link from architecture and V1 docs.
|
|
||||||
|
|
||||||
### Exit Criteria
|
|
||||||
- Major architecture decisions are explicit, versioned, and discoverable.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Phase F — Documentation Consolidation
|
|
||||||
|
|
||||||
- [ ] **F1. Update `docs/architecture.md`**
|
|
||||||
- Reflect real implementation and V1 target state separately.
|
|
||||||
- Mark transitional choices clearly.
|
|
||||||
|
|
||||||
- [ ] **F2. Update `docs/index.md` navigation consistency**
|
|
||||||
- Ensure architecture/readme references match actual docs/files.
|
|
||||||
|
|
||||||
- [ ] **F3. Update `docs/requirements.md` traceability notes**
|
|
||||||
- Mark REQ-7/REQ-10 status and verification approach after consolidation.
|
|
||||||
|
|
||||||
- [ ] **F4. Add Step 1 result summary**
|
|
||||||
- Create `docs/ver1/ver1-step1-results.md` after implementation.
|
|
||||||
|
|
||||||
### Exit Criteria
|
|
||||||
- Docs are internally consistent and match runtime architecture reality.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Verification Plan
|
|
||||||
|
|
||||||
## Architecture Verification Matrix (Step 1)
|
|
||||||
|
|
||||||
1. **Inspection**
|
|
||||||
- Resource ownership map exists and matches code.
|
|
||||||
- Schema bootstrap policy is explicit and environment-aware.
|
|
||||||
- ADRs exist for each key architecture decision.
|
|
||||||
|
|
||||||
2. **Automated checks**
|
|
||||||
- Existing test suite remains green.
|
|
||||||
- New/updated tests validate startup policy (no implicit schema mutation in production mode).
|
|
||||||
- Import/dependency-direction checks pass (if introduced).
|
|
||||||
|
|
||||||
3. **Demonstration**
|
|
||||||
- App starts in dev mode with explicit expected behavior.
|
|
||||||
- App starts in production mode without mutating schema implicitly.
|
|
||||||
- Worker lifecycle starts/stops cleanly with app lifespan.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Risks and Mitigations
|
|
||||||
|
|
||||||
1. **Risk:** Refactor destabilizes MVP behavior
|
|
||||||
**Mitigation:** Phase changes with small PRs and regression checks after each phase.
|
|
||||||
|
|
||||||
2. **Risk:** Over-rotation into premature async rewrite
|
|
||||||
**Mitigation:** Keep this step focused on lifecycle ownership and boundaries; defer full async migration unless required.
|
|
||||||
|
|
||||||
3. **Risk:** Schema policy changes break local DX
|
|
||||||
**Mitigation:** Keep explicit dev bootstrap path simple and documented.
|
|
||||||
|
|
||||||
4. **Risk:** Boundary rules become “doc only”
|
|
||||||
**Mitigation:** Add CI/review enforcement and architecture checklist.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Recommended Implementation Order
|
|
||||||
|
|
||||||
1. Phase A — Audit and baseline freeze
|
|
||||||
2. Phase B — Resource ownership consolidation
|
|
||||||
3. Phase C — Schema bootstrap policy
|
|
||||||
4. Phase D — Boundary enforcement
|
|
||||||
5. Phase E — ADR authoring
|
|
||||||
6. Phase F — Documentation consolidation
|
|
||||||
|
|
||||||
This order minimizes risk: diagnose first, then refactor ownership, then lock policy, then enforce boundaries, and finally finalize docs.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Step 1 Completion Checklist
|
|
||||||
|
|
||||||
- [ ] Architecture conformance matrix approved.
|
|
||||||
- [ ] REQ-7 ownership gaps resolved or explicitly deferred with owner/date.
|
|
||||||
- [ ] REQ-10 explicit bootstrap policy implemented and verified.
|
|
||||||
- [ ] Dependency direction rules documented and enforced.
|
|
||||||
- [ ] ADR set created for all major Step 1 decisions.
|
|
||||||
- [ ] Architecture and index docs updated to match implementation.
|
|
||||||
- [ ] Full test suite passes after consolidation.
|
|
||||||
- [ ] `docs/ver1/ver1-step1-results.md` created with evidence and residual risks.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Handoff to Step 2
|
|
||||||
|
|
||||||
Once Step 1 completes, Step 2 (Error Handling & Reliability Hardening) can proceed on stable architecture seams:
|
|
||||||
|
|
||||||
- consistent lifecycle ownership,
|
|
||||||
- explicit startup policy,
|
|
||||||
- clear module boundaries,
|
|
||||||
- documented architecture decisions.
|
|
||||||
@@ -1,80 +0,0 @@
|
|||||||
# Ver1 Step 2 Results: Error Handling & Reliability Hardening
|
|
||||||
|
|
||||||
## Summary
|
|
||||||
|
|
||||||
Step 2 implementation is complete for the planned reliability and error-handling hardening scope:
|
|
||||||
|
|
||||||
1. Worker retries are now explicit, bounded, and category-driven.
|
|
||||||
2. Error behavior is more consistent across worker/API/UI boundaries.
|
|
||||||
3. Logging now includes stronger boundary context in key failure paths.
|
|
||||||
4. Test coverage was expanded for retry policy and new reliability settings.
|
|
||||||
|
|
||||||
## Implemented Changes
|
|
||||||
|
|
||||||
### 1) Worker retry policy and terminal behavior
|
|
||||||
|
|
||||||
- Updated `src/transcription/models.py`:
|
|
||||||
- Added `Job.retry_count` with default `0`.
|
|
||||||
- Updated `src/transcription/config.py`:
|
|
||||||
- Added `worker_max_retries`.
|
|
||||||
- Added `worker_retry_backoff_seconds`.
|
|
||||||
- Updated `src/transcription/worker.py`:
|
|
||||||
- Added bounded retry decision path (`_should_retry`).
|
|
||||||
- Added requeue behavior (`_requeue_for_retry`) for retriable errors.
|
|
||||||
- Added deterministic terminal failure behavior (`_finalize_failed_job`).
|
|
||||||
- Preserved transcript failure detail persistence (`error_id`, `category`, suggestion).
|
|
||||||
|
|
||||||
### 2) API fallback normalization hardening
|
|
||||||
|
|
||||||
- Updated `src/transcription/api/errors.py`:
|
|
||||||
- Fallback handler now emits safe generic internal message for unhandled exceptions.
|
|
||||||
- Added structured boundary logging fields including operation and exception type.
|
|
||||||
|
|
||||||
### 3) UI interaction reliability guard
|
|
||||||
|
|
||||||
- Updated `src/transcription/ui/upload_page.py`:
|
|
||||||
- Added duplicate in-flight submission guard to prevent repeated upload handling while busy.
|
|
||||||
|
|
||||||
### 4) Observability/logging improvements
|
|
||||||
|
|
||||||
- Updated worker logs in `src/transcription/worker.py` to include operation and domain identifiers in key transitions:
|
|
||||||
- pick
|
|
||||||
- retry
|
|
||||||
- transcribed
|
|
||||||
- failed
|
|
||||||
|
|
||||||
## Test Coverage Added/Updated
|
|
||||||
|
|
||||||
- Updated `tests/test_models.py`:
|
|
||||||
- Assert `retry_count` default.
|
|
||||||
- Updated `tests/test_config.py`:
|
|
||||||
- Added worker retry settings default test.
|
|
||||||
- Updated `tests/services/test_worker.py`:
|
|
||||||
- Added retriable requeue test.
|
|
||||||
- Added retry-exhaustion terminal failure test.
|
|
||||||
- Updated existing tests for settings-driven worker behavior.
|
|
||||||
- Existing API error tests remained green with fallback behavior updates:
|
|
||||||
- `tests/api/test_error_responses.py`
|
|
||||||
|
|
||||||
## Verification Evidence
|
|
||||||
|
|
||||||
Executed and passing:
|
|
||||||
|
|
||||||
- `uv run pytest tests/services/test_worker.py tests/test_models.py tests/test_config.py tests/api/test_error_responses.py -q`
|
|
||||||
- `uv run pytest -q`
|
|
||||||
|
|
||||||
## Residual Risks / Follow-ups
|
|
||||||
|
|
||||||
1. Retry policy currently uses simple fixed backoff; richer strategy (exponential/jitter) can be added in later hardening.
|
|
||||||
2. Full cross-layer structured logging standardization can be expanded in Step 6 observability work.
|
|
||||||
3. A formal Step 2 error-path inventory artifact (`ver1-step2-audit.md`) is still recommended for governance completeness.
|
|
||||||
|
|
||||||
## Step 2 Exit Assessment
|
|
||||||
|
|
||||||
- Error taxonomy and envelope stability: **met**
|
|
||||||
- Bounded retry and terminal failure behavior: **met**
|
|
||||||
- Worker reliability controls: **met**
|
|
||||||
- UI interaction hardening for duplicate actions: **met**
|
|
||||||
- Test coverage expansion and full-suite regression safety: **met**
|
|
||||||
|
|
||||||
Step 2 is complete and ready to hand off to Ver1 Step 3.
|
|
||||||
@@ -1,302 +0,0 @@
|
|||||||
# Step 2 Implementation Plan: Error Handling & Reliability Hardening
|
|
||||||
|
|
||||||
## Purpose
|
|
||||||
|
|
||||||
Implement **Ver1 Step 2** from `docs/ver1/ver1.md` by standardizing failure behavior and reliability controls so the system fails safely, predictably, and transparently across UI, API, services, worker, and provider boundaries.
|
|
||||||
|
|
||||||
Primary governing docs:
|
|
||||||
|
|
||||||
- `docs/error_handling.md` (authoritative contract)
|
|
||||||
- `docs/requirements.md` (REQ-2, REQ-3, REQ-4, REQ-5, REQ-6)
|
|
||||||
- `docs/architecture.md` (boundary ownership and worker lifecycle)
|
|
||||||
- `docs/ver1/ver1.md` (Step 2 objective)
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## MCP Skill and Guide Inputs Incorporated
|
|
||||||
|
|
||||||
This plan integrates guidance from john-stream-mcp resources:
|
|
||||||
|
|
||||||
1. `resource://skills/python-logging-dictconfig/document`
|
|
||||||
- centralized `dictConfig` logging
|
|
||||||
- startup-only configuration
|
|
||||||
- stable named loggers and boundary-level logging discipline
|
|
||||||
|
|
||||||
2. `resource://skills/pytesting/document`
|
|
||||||
- deterministic, behavior-first tests
|
|
||||||
- explicit marker usage and fast/slow lane discipline
|
|
||||||
- integration checks for boundary behavior and error contracts
|
|
||||||
|
|
||||||
3. `resource://skills/fastapi-async-sqlalchemy-modernization/document`
|
|
||||||
- classify at source boundary
|
|
||||||
- explicit transaction/session behavior under failure
|
|
||||||
- phased rollout with quality gates and rollback awareness
|
|
||||||
|
|
||||||
4. `resource://skills/nicegui-ui-customization/document`
|
|
||||||
- explicit user-facing error feedback for each interaction
|
|
||||||
- prevent duplicate actions during in-flight operations
|
|
||||||
- preserve one-way dependency boundaries from UI -> services
|
|
||||||
|
|
||||||
5. `resource://skills/fastapi-uv-docker/document` (applied selectively)
|
|
||||||
- lifespan-safe startup/shutdown behavior
|
|
||||||
- health/readiness posture and cloud-native operational checks
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Current-State Gap Summary
|
|
||||||
|
|
||||||
The project already has a strong baseline (`AppError`, taxonomy enum, API envelope, worker persistence), but Step 2 needs completion-level hardening:
|
|
||||||
|
|
||||||
1. **Error contract consistency**
|
|
||||||
- API envelope exists, but consistency must be verified for all error pathways.
|
|
||||||
2. **Cross-boundary category normalization**
|
|
||||||
- Provider/service/worker mappings exist, but require stricter policy checks and tests.
|
|
||||||
3. **Retry policy implementation depth**
|
|
||||||
- Step 2 requires bounded retry policy and clear terminal behavior for retriable failures.
|
|
||||||
4. **Operational traceability**
|
|
||||||
- Logging exists; Step 2 requires consistent structured fields at critical boundaries.
|
|
||||||
5. **UI failure UX consistency**
|
|
||||||
- UI error handling exists; Step 2 requires explicit contract coverage and anti-duplication safeguards.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Scope for Step 2
|
|
||||||
|
|
||||||
### In scope
|
|
||||||
1. Enforce canonical error taxonomy and envelope across all boundaries.
|
|
||||||
2. Standardize logging fields and boundary-level error traceability.
|
|
||||||
3. Implement/complete bounded retry and terminal failure behavior in worker paths.
|
|
||||||
4. Improve UI/API error presentation consistency and actionable guidance.
|
|
||||||
5. Add comprehensive Step 2 test coverage and verification matrix.
|
|
||||||
6. Update documentation to reflect final Step 2 policies and behavior.
|
|
||||||
|
|
||||||
### Out of scope
|
|
||||||
- Major architecture/topology changes (external queue, distributed worker)
|
|
||||||
- New end-user feature expansion outside reliability/error handling
|
|
||||||
- Full async ORM migration (unless required by bug fix)
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Target Decisions for Step 2
|
|
||||||
|
|
||||||
1. **Taxonomy stability is mandatory**
|
|
||||||
- `ErrorCategory` values remain stable contract identifiers.
|
|
||||||
2. **Classification occurs at source boundary**
|
|
||||||
- adapters/services normalize early; UI/API only present safely.
|
|
||||||
3. **User safety over internal detail leakage**
|
|
||||||
- expose safe message + suggestion + error_id; keep sensitive detail in logs.
|
|
||||||
4. **Retry is explicit and bounded**
|
|
||||||
- only retriable categories may retry; retries are capped; terminal failures persist reason.
|
|
||||||
5. **Boundary logs carry correlation fields**
|
|
||||||
- include `error_id`, `category`, `operation`, and domain identifiers where available.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Detailed Work Breakdown
|
|
||||||
|
|
||||||
## Phase A — Error Contract Audit and Policy Lock
|
|
||||||
|
|
||||||
- [ ] **A1. Build error-path inventory**
|
|
||||||
- Enumerate all failure entry points across:
|
|
||||||
- `api/`
|
|
||||||
- `ui/`
|
|
||||||
- `services/`
|
|
||||||
- `worker.py`
|
|
||||||
- `providers/`
|
|
||||||
|
|
||||||
- [ ] **A2. Produce taxonomy mapping table**
|
|
||||||
- For each known exception path, map:
|
|
||||||
- source exception type
|
|
||||||
- target `ErrorCategory`
|
|
||||||
- retriable flag
|
|
||||||
- API status (if exposed)
|
|
||||||
|
|
||||||
- [ ] **A3. Reconcile with `docs/error_handling.md`**
|
|
||||||
- Resolve any mismatch in category semantics, status codes, or suggested actions.
|
|
||||||
|
|
||||||
### Deliverables
|
|
||||||
- `docs/ver1/ver1-step2-audit.md` (recommended)
|
|
||||||
- taxonomy mapping table
|
|
||||||
|
|
||||||
### Exit Criteria
|
|
||||||
- Every known failure path has explicit category + retriable policy.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Phase B — API and Service Contract Hardening
|
|
||||||
|
|
||||||
- [ ] **B1. Enforce API envelope completeness**
|
|
||||||
- Ensure all API errors return:
|
|
||||||
- `error_id`, `category`, `message`, `suggestion`, `timestamp`
|
|
||||||
|
|
||||||
- [ ] **B2. Verify category-to-status mapping consistency**
|
|
||||||
- Confirm `api/errors.py` matches `docs/error_handling.md` mapping guidance.
|
|
||||||
|
|
||||||
- [ ] **B3. Normalize service exceptions at boundary**
|
|
||||||
- Services should raise `AppError` subclasses for known failures.
|
|
||||||
- Unknown exceptions must become `internal_unexpected_error` with traceable `error_id`.
|
|
||||||
|
|
||||||
- [ ] **B4. Ensure safe detail handling**
|
|
||||||
- API/UI messages remain safe.
|
|
||||||
- Diagnostic context remains in logs/persisted failure detail where appropriate.
|
|
||||||
|
|
||||||
### Exit Criteria
|
|
||||||
- No unstructured/unclassified exception escapes core boundaries.
|
|
||||||
- API responses are contract-stable for all tested failure modes.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Phase C — Worker Retry and Terminal Failure Policy
|
|
||||||
|
|
||||||
- [ ] **C1. Define bounded retry policy**
|
|
||||||
- Add configurable retry settings (attempt limit/backoff policy).
|
|
||||||
- Limit retries to retriable categories.
|
|
||||||
|
|
||||||
- [ ] **C2. Implement terminal failure persistence**
|
|
||||||
- On retry exhaustion, persist clear terminal reason and `error_id`.
|
|
||||||
- Ensure job status transitions end deterministically at `failed`.
|
|
||||||
|
|
||||||
- [ ] **C3. Add duplicate-processing safety checks**
|
|
||||||
- Prevent duplicate terminal updates when job already resolved.
|
|
||||||
|
|
||||||
- [ ] **C4. Validate worker lifecycle under repeated transient failures**
|
|
||||||
- Ensure loop remains stable and responsive.
|
|
||||||
|
|
||||||
### Exit Criteria
|
|
||||||
- Retries are bounded and policy-driven.
|
|
||||||
- Exhausted retries produce deterministic failed state with evidence.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Phase D — Logging and Observability Contract Enforcement
|
|
||||||
|
|
||||||
- [ ] **D1. Central logging conformance check**
|
|
||||||
- Confirm startup-only `dictConfig` use remains canonical.
|
|
||||||
- No module-level `basicConfig` use.
|
|
||||||
|
|
||||||
- [ ] **D2. Standardize error log fields**
|
|
||||||
- Require at minimum when available:
|
|
||||||
- `error_id`, `category`, `operation`, `exception_type`, `job_id`, `document_id`
|
|
||||||
|
|
||||||
- [ ] **D3. Boundary handoff logging**
|
|
||||||
- Add/normalize logs at transitions:
|
|
||||||
- UI action -> service
|
|
||||||
- service -> provider/db
|
|
||||||
- worker pickup -> terminal state
|
|
||||||
|
|
||||||
- [ ] **D4. Log noise control**
|
|
||||||
- Avoid duplicate stack-trace logging across layers for same exception.
|
|
||||||
|
|
||||||
### Exit Criteria
|
|
||||||
- Critical failure events are traceable end-to-end via logs and `error_id`.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Phase E — UI Error UX Consistency and Interaction Hardening
|
|
||||||
|
|
||||||
- [ ] **E1. Standardize user error presentation**
|
|
||||||
- For upload/jobs interactions, ensure:
|
|
||||||
- clear title
|
|
||||||
- plain-language message
|
|
||||||
- suggested action
|
|
||||||
- visible error reference id
|
|
||||||
|
|
||||||
- [ ] **E2. Add in-flight interaction guards**
|
|
||||||
- Prevent duplicate submits/click storms during pending operations.
|
|
||||||
|
|
||||||
- [ ] **E3. Ensure deterministic UI state recovery**
|
|
||||||
- controls re-enable after failure
|
|
||||||
- status text remains actionable
|
|
||||||
|
|
||||||
- [ ] **E4. Keep UI boundary clean**
|
|
||||||
- no provider/protocol details leaked into page modules
|
|
||||||
|
|
||||||
### Exit Criteria
|
|
||||||
- All primary UI actions have consistent success/failure interaction behavior.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Phase F — Test Expansion and Verification
|
|
||||||
|
|
||||||
Apply pytesting guidance: behavior-first assertions, deterministic fixtures, strict markers.
|
|
||||||
|
|
||||||
- [ ] **F1. API error contract tests**
|
|
||||||
- verify envelope fields and status mapping for each category class.
|
|
||||||
|
|
||||||
- [ ] **F2. Service classification tests**
|
|
||||||
- verify known failures map to expected `AppError` subclasses/categories.
|
|
||||||
|
|
||||||
- [ ] **F3. Worker retry policy tests**
|
|
||||||
- retriable failure retries and eventual success
|
|
||||||
- retriable failure exhaustion -> terminal failed
|
|
||||||
- non-retriable failure -> immediate failed
|
|
||||||
|
|
||||||
- [ ] **F4. UI error behavior tests**
|
|
||||||
- upload/jobs actions show actionable feedback on failures
|
|
||||||
- duplicate action guard behavior
|
|
||||||
|
|
||||||
- [ ] **F5. Regression guard tests**
|
|
||||||
- at least one test per previously observed production/real-world failure mode
|
|
||||||
|
|
||||||
### Validation Commands
|
|
||||||
- `uv run pytest --collect-only -q`
|
|
||||||
- `uv run pytest -m unit -q`
|
|
||||||
- `uv run pytest -m "not external" -q`
|
|
||||||
- `uv run pytest -q`
|
|
||||||
|
|
||||||
### Exit Criteria
|
|
||||||
- All Step 2 reliability/error contract tests pass.
|
|
||||||
- Existing suite remains green.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Recommended Implementation Order
|
|
||||||
|
|
||||||
1. Phase A — audit and policy lock
|
|
||||||
2. Phase B — API/service contract hardening
|
|
||||||
3. Phase C — worker retry and terminal policy
|
|
||||||
4. Phase D — logging/traceability normalization
|
|
||||||
5. Phase E — UI consistency hardening
|
|
||||||
6. Phase F — test expansion and full verification
|
|
||||||
|
|
||||||
This order reduces risk by locking policy first, then applying behavior changes at core boundaries before UI polish.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Risks and Mitigations
|
|
||||||
|
|
||||||
1. **Risk:** Overly broad retry policy causes hidden failure loops
|
|
||||||
**Mitigation:** strict category-based retry eligibility + hard cap + terminal persistence.
|
|
||||||
|
|
||||||
2. **Risk:** User-facing messages become too technical
|
|
||||||
**Mitigation:** enforce safe message + suggestion contract in tests.
|
|
||||||
|
|
||||||
3. **Risk:** Logging becomes noisy/redundant
|
|
||||||
**Mitigation:** boundary logging rules and single-trace ownership.
|
|
||||||
|
|
||||||
4. **Risk:** Reliability work introduces regressions in happy path
|
|
||||||
**Mitigation:** run full suite continuously; preserve integration pipeline tests.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Step 2 Completion Checklist
|
|
||||||
|
|
||||||
- [ ] Error taxonomy mapping table completed and approved.
|
|
||||||
- [ ] API envelope and HTTP status behavior verified for all relevant failure categories.
|
|
||||||
- [ ] Service/provider exception normalization is consistent and tested.
|
|
||||||
- [ ] Worker retry behavior is bounded, explicit, and terminal-state safe.
|
|
||||||
- [ ] Structured error logging fields are present at boundary handoffs.
|
|
||||||
- [ ] UI failure flows provide clear, actionable, and traceable feedback.
|
|
||||||
- [ ] Full test suite passes with new Step 2 coverage included.
|
|
||||||
- [ ] `docs/ver1/ver1-step2-results.md` created with evidence and residual risks.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Handoff to Step 3
|
|
||||||
|
|
||||||
After Step 2 completion, Step 3 (Functional Completion by Requirement Domain) proceeds on a hardened foundation:
|
|
||||||
|
|
||||||
- stable failure contracts,
|
|
||||||
- predictable retries and terminal behavior,
|
|
||||||
- actionable user/API error semantics,
|
|
||||||
- improved diagnostic traceability.
|
|
||||||
@@ -12,18 +12,29 @@ description = "Historical document transcription system"
|
|||||||
readme = "README.md"
|
readme = "README.md"
|
||||||
requires-python = ">=3.12"
|
requires-python = ">=3.12"
|
||||||
dependencies = [
|
dependencies = [
|
||||||
|
"aiosqlite>=0.21.0",
|
||||||
|
"asyncpg>=0.31.0",
|
||||||
"fastapi>=0.138.0",
|
"fastapi>=0.138.0",
|
||||||
"nicegui==3.13.0",
|
"nicegui==3.13.0",
|
||||||
"openrouter>=0.7.0",
|
"openrouter>=0.7.0",
|
||||||
|
"psycopg2-binary>=2.9.12",
|
||||||
"pydantic>=2.13.4",
|
"pydantic>=2.13.4",
|
||||||
"pydantic-settings>=2.9.1",
|
"pydantic-settings>=2.9.1",
|
||||||
"sqlmodel>=0.0.25",
|
"sqlmodel>=0.0.25",
|
||||||
]
|
]
|
||||||
|
|
||||||
[project.optional-dependencies]
|
|
||||||
|
[dependency-groups]
|
||||||
dev = [
|
dev = [
|
||||||
"pytest>=8.0",
|
"pytest>=8.0",
|
||||||
"pytest-asyncio>=0.25",
|
"pytest-asyncio>=0.25",
|
||||||
|
"httpx2>=2.5.0",
|
||||||
|
"ipykernel>=7.3.0",
|
||||||
|
"ipywidgets>=8.1.8",
|
||||||
|
"pre-commit>=4.6.0",
|
||||||
|
"rich>=15.0.0",
|
||||||
|
"ruff>=0.15.20",
|
||||||
|
"ty>=0.0.54",
|
||||||
]
|
]
|
||||||
|
|
||||||
[tool.pytest.ini_options]
|
[tool.pytest.ini_options]
|
||||||
|
|||||||
@@ -0,0 +1,62 @@
|
|||||||
|
line-length = 120
|
||||||
|
indent-width = 4
|
||||||
|
target-version = "py313"
|
||||||
|
|
||||||
|
exclude = [
|
||||||
|
".venv",
|
||||||
|
".devenv",
|
||||||
|
".git",
|
||||||
|
".vscode",
|
||||||
|
"build",
|
||||||
|
"site",
|
||||||
|
"__pycache__",
|
||||||
|
]
|
||||||
|
|
||||||
|
[lint]
|
||||||
|
preview = true
|
||||||
|
|
||||||
|
extend-select = [
|
||||||
|
"ARG", # https://docs.astral.sh/ruff/rules/#flake8-unused-arguments-arg
|
||||||
|
"B", # https://docs.astral.sh/ruff/rules/#flake8-bugbear-b
|
||||||
|
"C4", # https://docs.astral.sh/ruff/rules/#flake8-comprehensions-c4
|
||||||
|
"DOC102", # https://docs.astral.sh/ruff/rules/docstring-extraneous-parameter/
|
||||||
|
"DOC202", # https://docs.astral.sh/ruff/rules/docstring-extraneous-returns/
|
||||||
|
"DOC403", # https://docs.astral.sh/ruff/rules/docstring-extraneous-yields/
|
||||||
|
"DOC502", # https://docs.astral.sh/ruff/rules/docstring-extraneous-exception/
|
||||||
|
"E", "W", # https://docs.astral.sh/ruff/rules/#pycodestyle-e-w
|
||||||
|
"F", # https://docs.astral.sh/ruff/rules/#pyflakes-f
|
||||||
|
"FURB", # https://docs.astral.sh/ruff/rules/#refurb-furb
|
||||||
|
"I", # https://docs.astral.sh/ruff/rules/#isort-i
|
||||||
|
"N", # https://docs.astral.sh/ruff/rules/#pep8-naming-n
|
||||||
|
"PD", # https://docs.astral.sh/ruff/rules/#pandas-vet-pd
|
||||||
|
"PTH", # https://docs.astral.sh/ruff/rules/#flake8-use-pathlib-pth
|
||||||
|
"UP", # https://docs.astral.sh/ruff/rules/#pyupgrade-up
|
||||||
|
"SIM", # https://docs.astral.sh/ruff/rules/#flake8-simplify-sim
|
||||||
|
"PLR0202", # https://docs.astral.sh/ruff/rules/no-classmethod-decorator/
|
||||||
|
"PLR0203", # https://docs.astral.sh/ruff/rules/no-staticmethod-decorator/
|
||||||
|
"PLR0206", # https://docs.astral.sh/ruff/rules/property-with-parameters/
|
||||||
|
"PLR0915", # https://docs.astral.sh/ruff/rules/too-many-statements/
|
||||||
|
"PLR1702", # https://docs.astral.sh/ruff/rules/too-many-nested-blocks/
|
||||||
|
"TRY002",
|
||||||
|
]
|
||||||
|
extend-fixable = ["ALL"]
|
||||||
|
ignore = [
|
||||||
|
"UP046",
|
||||||
|
"UP047",
|
||||||
|
]
|
||||||
|
|
||||||
|
[lint.extend-per-file-ignores]
|
||||||
|
"*.ipynb" = [
|
||||||
|
"F401", # unused imports
|
||||||
|
"F841", # unused local variable
|
||||||
|
"F821", # undefined name in exploratory notebook cells
|
||||||
|
]
|
||||||
|
|
||||||
|
[lint.isort]
|
||||||
|
force-single-line = true
|
||||||
|
|
||||||
|
[format]
|
||||||
|
quote-style = "double"
|
||||||
|
indent-style = "space"
|
||||||
|
skip-magic-trailing-comma = false
|
||||||
|
line-ending = "auto"
|
||||||
@@ -4,10 +4,13 @@ from __future__ import annotations
|
|||||||
|
|
||||||
import logging
|
import logging
|
||||||
|
|
||||||
from fastapi import FastAPI, Request
|
from fastapi import FastAPI
|
||||||
|
from fastapi import Request
|
||||||
from fastapi.responses import JSONResponse
|
from fastapi.responses import JSONResponse
|
||||||
|
|
||||||
from transcription.errors import AppError, ErrorCategory, build_error_envelope
|
from transcription.errors import AppError
|
||||||
|
from transcription.errors import ErrorCategory
|
||||||
|
from transcription.errors import build_error_envelope
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|||||||
@@ -2,77 +2,66 @@
|
|||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from contextlib import AsyncExitStack
|
||||||
from contextlib import asynccontextmanager
|
from contextlib import asynccontextmanager
|
||||||
from threading import Event, Thread
|
|
||||||
|
|
||||||
from fastapi import FastAPI
|
from fastapi import FastAPI
|
||||||
|
from fastapi import status
|
||||||
|
from fastapi.responses import RedirectResponse
|
||||||
|
|
||||||
from transcription.api.errors import register_error_handlers
|
from .api.errors import register_error_handlers
|
||||||
from transcription.api.health import router as health_router
|
from .api.health import router as health_router
|
||||||
from transcription.config import get_settings, setup_logging
|
from .config import configure_logging
|
||||||
from transcription.db import (
|
from .config import get_settings
|
||||||
create_all,
|
from .db import create_all
|
||||||
dispose_database_runtime,
|
from .db import dispose_database_runtime
|
||||||
initialize_database_runtime,
|
from .db import initialize_database_runtime
|
||||||
should_bootstrap_schema,
|
from .services import ServiceBundle
|
||||||
)
|
from .ui import register_pages
|
||||||
from transcription.ui import register_pages
|
from .worker import worker_consumer_lifespan
|
||||||
from transcription.worker import run_worker_loop
|
|
||||||
|
|
||||||
|
|
||||||
def _start_worker(app: FastAPI) -> None:
|
|
||||||
stop_event = Event()
|
|
||||||
worker_thread = Thread(
|
|
||||||
target=run_worker_loop,
|
|
||||||
kwargs={
|
|
||||||
"engine": app.state.db_runtime.engine,
|
|
||||||
"stop_event": stop_event,
|
|
||||||
"poll_interval_seconds": 1.0,
|
|
||||||
},
|
|
||||||
daemon=True,
|
|
||||||
)
|
|
||||||
worker_thread.start()
|
|
||||||
app.state.worker_stop_event = stop_event
|
|
||||||
app.state.worker_thread = worker_thread
|
|
||||||
|
|
||||||
|
|
||||||
def _stop_worker(app: FastAPI) -> None:
|
|
||||||
stop_event = getattr(app.state, "worker_stop_event", None)
|
|
||||||
worker_thread = getattr(app.state, "worker_thread", None)
|
|
||||||
|
|
||||||
if stop_event is not None:
|
|
||||||
stop_event.set()
|
|
||||||
if worker_thread is not None:
|
|
||||||
worker_thread.join(timeout=2.0)
|
|
||||||
|
|
||||||
|
|
||||||
@asynccontextmanager
|
@asynccontextmanager
|
||||||
async def _lifespan(app: FastAPI):
|
async def _lifespan(app: FastAPI):
|
||||||
setup_logging()
|
configure_logging()
|
||||||
|
|
||||||
settings = get_settings()
|
settings = get_settings()
|
||||||
app.state.settings = settings
|
app.state.settings = settings
|
||||||
app.state.db_runtime = initialize_database_runtime(settings=settings)
|
app.state.services = ServiceBundle()
|
||||||
|
app.state.runtime = initialize_database_runtime(settings=settings)
|
||||||
|
|
||||||
if should_bootstrap_schema(settings):
|
if settings.should_bootstrap_schema:
|
||||||
create_all(engine=app.state.db_runtime.engine)
|
await create_all(engine=app.state.runtime.engine)
|
||||||
|
|
||||||
settings.upload_dir.mkdir(parents=True, exist_ok=True)
|
settings.upload_dir.mkdir(parents=True, exist_ok=True)
|
||||||
settings.prompt_dir.mkdir(parents=True, exist_ok=True)
|
settings.prompt_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
|
||||||
_start_worker(app)
|
async with AsyncExitStack() as stack:
|
||||||
try:
|
stack.push_async_callback(dispose_database_runtime)
|
||||||
|
stop_event, worker_notifier = await stack.enter_async_context(
|
||||||
|
worker_consumer_lifespan(
|
||||||
|
session_factory=app.state.runtime.session_factory,
|
||||||
|
poll_interval_seconds=1.0,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
app.state.worker_stop_event = stop_event
|
||||||
|
app.state.worker_notifier = worker_notifier
|
||||||
yield
|
yield
|
||||||
finally:
|
|
||||||
_stop_worker(app)
|
|
||||||
dispose_database_runtime()
|
|
||||||
|
|
||||||
|
|
||||||
def create_app() -> FastAPI:
|
def create_app() -> FastAPI:
|
||||||
"""Create and configure the FastAPI application."""
|
"""Create and configure the FastAPI application."""
|
||||||
app = FastAPI(title="Transcription", lifespan=_lifespan)
|
app = FastAPI(title="Transcription", lifespan=_lifespan)
|
||||||
|
|
||||||
|
@app.get("/", include_in_schema=False)
|
||||||
|
async def root_redirect() -> RedirectResponse:
|
||||||
|
return RedirectResponse(url="/ui", status_code=status.HTTP_307_TEMPORARY_REDIRECT)
|
||||||
|
|
||||||
|
@app.get("/ui", include_in_schema=False)
|
||||||
|
async def ui_redirect() -> RedirectResponse:
|
||||||
|
return RedirectResponse(url="/ui/upload", status_code=status.HTTP_307_TEMPORARY_REDIRECT)
|
||||||
|
|
||||||
register_error_handlers(app)
|
register_error_handlers(app)
|
||||||
register_pages(app)
|
register_pages(app)
|
||||||
app.include_router(health_router)
|
app.include_router(health_router)
|
||||||
return app
|
return app
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,39 @@
|
|||||||
|
"""Helpers for accessing lifespan-owned application state resources."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from fastapi import FastAPI
|
||||||
|
from sqlalchemy.ext.asyncio import async_sessionmaker
|
||||||
|
from sqlmodel.ext.asyncio.session import AsyncSession
|
||||||
|
|
||||||
|
from transcription.db.runtime import DatabaseRuntime
|
||||||
|
from transcription.db.runtime import get_session_factory
|
||||||
|
from transcription.worker import WorkerNotifier
|
||||||
|
from transcription.worker import resolve_worker_notifier
|
||||||
|
|
||||||
|
|
||||||
|
def resolve_database_runtime(state: object) -> DatabaseRuntime | None:
|
||||||
|
"""Return database runtime from app-like state objects when available."""
|
||||||
|
runtime = getattr(state, "runtime", None)
|
||||||
|
return runtime if isinstance(runtime, DatabaseRuntime) else None
|
||||||
|
|
||||||
|
|
||||||
|
def require_database_runtime(state: object) -> DatabaseRuntime:
|
||||||
|
"""Return database runtime or raise when app lifespan has not initialized it."""
|
||||||
|
runtime = resolve_database_runtime(state)
|
||||||
|
if runtime is None:
|
||||||
|
raise RuntimeError("Database runtime is not initialized on application state")
|
||||||
|
return runtime
|
||||||
|
|
||||||
|
|
||||||
|
def resolve_session_factory(state: object) -> async_sessionmaker[AsyncSession]:
|
||||||
|
"""Return DB session factory from state when available, otherwise shared runtime."""
|
||||||
|
runtime = resolve_database_runtime(state)
|
||||||
|
if runtime is not None:
|
||||||
|
return runtime.session_factory
|
||||||
|
return get_session_factory()
|
||||||
|
|
||||||
|
|
||||||
|
def get_worker_notifier(app: FastAPI) -> WorkerNotifier:
|
||||||
|
"""Return app worker notifier, or a no-op fallback when unavailable."""
|
||||||
|
return resolve_worker_notifier(app.state)
|
||||||
@@ -5,14 +5,16 @@ once at startup. Provider-specific defaults (model names, base URLs)
|
|||||||
are resolved by the provider adapters, not here.
|
are resolved by the provider adapters, not here.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import logging
|
|
||||||
import logging.config
|
import logging.config
|
||||||
|
from contextvars import ContextVar
|
||||||
from enum import StrEnum
|
from enum import StrEnum
|
||||||
from functools import lru_cache
|
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import Literal
|
from typing import Literal
|
||||||
|
|
||||||
from pydantic_settings import BaseSettings, SettingsConfigDict
|
from pydantic_settings import BaseSettings
|
||||||
|
from pydantic_settings import SettingsConfigDict
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
class Provider(StrEnum):
|
class Provider(StrEnum):
|
||||||
@@ -39,6 +41,7 @@ class Settings(BaseSettings):
|
|||||||
# --- persistence ---
|
# --- persistence ---
|
||||||
database_url: str = "sqlite:///./transcription.db"
|
database_url: str = "sqlite:///./transcription.db"
|
||||||
bootstrap_schema_on_startup: bool | None = None
|
bootstrap_schema_on_startup: bool | None = None
|
||||||
|
sqlite_check_same_thread: bool = False
|
||||||
|
|
||||||
# --- filesystem paths ---
|
# --- filesystem paths ---
|
||||||
upload_dir: Path = Path("./uploads")
|
upload_dir: Path = Path("./uploads")
|
||||||
@@ -48,13 +51,31 @@ class Settings(BaseSettings):
|
|||||||
worker_max_retries: int = 0
|
worker_max_retries: int = 0
|
||||||
worker_retry_backoff_seconds: float = 0.0
|
worker_retry_backoff_seconds: float = 0.0
|
||||||
|
|
||||||
|
@property
|
||||||
|
def should_bootstrap_schema(self) -> bool:
|
||||||
|
"""Return whether startup should auto-create schema for this environment."""
|
||||||
|
if self.bootstrap_schema_on_startup is not None:
|
||||||
|
return self.bootstrap_schema_on_startup
|
||||||
|
return self.environment in {"development", "test"}
|
||||||
|
|
||||||
|
|
||||||
|
_settings: ContextVar[Settings | None] = ContextVar("settings", default=None)
|
||||||
|
|
||||||
|
|
||||||
|
def get_settings(**kwargs) -> Settings:
|
||||||
|
settings = _settings.get()
|
||||||
|
if settings is None:
|
||||||
|
settings = Settings(**kwargs) # pyright: ignore[reportCallIssue]
|
||||||
|
_settings.set(settings)
|
||||||
|
return settings
|
||||||
|
|
||||||
|
|
||||||
LOGGING_CONFIG: dict[str, object] = {
|
LOGGING_CONFIG: dict[str, object] = {
|
||||||
"version": 1,
|
"version": 1,
|
||||||
"disable_existing_loggers": False,
|
"disable_existing_loggers": False,
|
||||||
"formatters": {
|
"formatters": {
|
||||||
"standard": {
|
"standard": {
|
||||||
"format": "%(asctime)s | %(levelname)-8s | %(name)s | %(message)s",
|
"format": "%(asctime)s %(levelname)-8s | %(message)s",
|
||||||
"datefmt": "%Y-%m-%d %H:%M:%S",
|
"datefmt": "%Y-%m-%d %H:%M:%S",
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
@@ -69,18 +90,17 @@ LOGGING_CONFIG: dict[str, object] = {
|
|||||||
"level": "INFO",
|
"level": "INFO",
|
||||||
"handlers": ["console"],
|
"handlers": ["console"],
|
||||||
},
|
},
|
||||||
|
"loggers": {
|
||||||
|
"transcription": {
|
||||||
|
"level": "DEBUG",
|
||||||
|
"handlers": ["console"],
|
||||||
|
"propagate": False,
|
||||||
|
}
|
||||||
|
},
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
@lru_cache(maxsize=1)
|
def configure_logging() -> None:
|
||||||
def get_settings() -> Settings:
|
|
||||||
"""Return the singleton Settings instance.
|
|
||||||
|
|
||||||
Cached so the entire application shares one validated config.
|
|
||||||
"""
|
|
||||||
return Settings()
|
|
||||||
|
|
||||||
|
|
||||||
def setup_logging() -> None:
|
|
||||||
"""Configure root logging once at startup."""
|
"""Configure root logging once at startup."""
|
||||||
logging.config.dictConfig(LOGGING_CONFIG)
|
logging.config.dictConfig(LOGGING_CONFIG)
|
||||||
|
logger.debug("Logging configured")
|
||||||
|
|||||||
@@ -1,116 +0,0 @@
|
|||||||
"""Database runtime ownership, schema bootstrap, and session access.
|
|
||||||
|
|
||||||
V1 moves database resource ownership to explicit runtime initialization so
|
|
||||||
startup/shutdown behavior is predictable and lifespan-managed.
|
|
||||||
"""
|
|
||||||
|
|
||||||
import contextlib
|
|
||||||
import logging
|
|
||||||
from collections.abc import Generator
|
|
||||||
from dataclasses import dataclass
|
|
||||||
|
|
||||||
from sqlalchemy import inspect, text
|
|
||||||
from sqlalchemy.engine import Engine
|
|
||||||
from sqlmodel import Session, SQLModel, create_engine
|
|
||||||
|
|
||||||
from transcription.config import Settings, get_settings
|
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class DatabaseRuntime:
|
|
||||||
"""Process-level database runtime resources."""
|
|
||||||
|
|
||||||
engine: Engine
|
|
||||||
|
|
||||||
|
|
||||||
_runtime: DatabaseRuntime | None = None
|
|
||||||
|
|
||||||
|
|
||||||
def _build_engine(settings: Settings) -> Engine:
|
|
||||||
connect_args: dict[str, object] = {}
|
|
||||||
if settings.database_url.startswith("sqlite"):
|
|
||||||
connect_args["check_same_thread"] = False
|
|
||||||
return create_engine(
|
|
||||||
settings.database_url,
|
|
||||||
echo=False,
|
|
||||||
connect_args=connect_args,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def initialize_database_runtime(*, settings: Settings | None = None) -> DatabaseRuntime:
|
|
||||||
"""Initialize and cache the process database runtime once."""
|
|
||||||
global _runtime
|
|
||||||
|
|
||||||
if _runtime is not None:
|
|
||||||
return _runtime
|
|
||||||
|
|
||||||
runtime_settings = settings or get_settings()
|
|
||||||
_runtime = DatabaseRuntime(engine=_build_engine(runtime_settings))
|
|
||||||
return _runtime
|
|
||||||
|
|
||||||
|
|
||||||
def get_database_runtime() -> DatabaseRuntime:
|
|
||||||
"""Return initialized database runtime, creating it if needed."""
|
|
||||||
if _runtime is None:
|
|
||||||
return initialize_database_runtime()
|
|
||||||
return _runtime
|
|
||||||
|
|
||||||
|
|
||||||
def dispose_database_runtime() -> None:
|
|
||||||
"""Dispose process database runtime resources."""
|
|
||||||
global _runtime
|
|
||||||
if _runtime is not None:
|
|
||||||
_runtime.engine.dispose()
|
|
||||||
_runtime = None
|
|
||||||
|
|
||||||
|
|
||||||
def should_bootstrap_schema(settings: Settings) -> bool:
|
|
||||||
"""Return whether startup should auto-create schema for this environment."""
|
|
||||||
if settings.bootstrap_schema_on_startup is not None:
|
|
||||||
return settings.bootstrap_schema_on_startup
|
|
||||||
return settings.environment in {"development", "test"}
|
|
||||||
|
|
||||||
|
|
||||||
def create_all(*, engine: Engine | None = None) -> None:
|
|
||||||
"""Create all tables on the selected engine."""
|
|
||||||
# Import models so SQLModel metadata is fully registered before bootstrap.
|
|
||||||
from transcription import models as _models # noqa: F401
|
|
||||||
|
|
||||||
active_engine = engine or get_database_runtime().engine
|
|
||||||
SQLModel.metadata.create_all(active_engine)
|
|
||||||
_ensure_sqlite_compat_columns(active_engine)
|
|
||||||
|
|
||||||
|
|
||||||
def _ensure_sqlite_compat_columns(engine: Engine) -> None:
|
|
||||||
"""Apply lightweight dev/test SQLite compatibility column patches.
|
|
||||||
|
|
||||||
This keeps local bootstrap resilient when models evolve but no full
|
|
||||||
migration tooling is in place yet.
|
|
||||||
"""
|
|
||||||
if engine.url.get_backend_name() != "sqlite":
|
|
||||||
return
|
|
||||||
|
|
||||||
inspector = inspect(engine)
|
|
||||||
table_names = set(inspector.get_table_names())
|
|
||||||
if "job" not in table_names:
|
|
||||||
return
|
|
||||||
|
|
||||||
columns = {column["name"] for column in inspector.get_columns("job")}
|
|
||||||
if "retry_count" not in columns:
|
|
||||||
with engine.begin() as connection:
|
|
||||||
connection.execute(
|
|
||||||
text("ALTER TABLE job ADD COLUMN retry_count INTEGER NOT NULL DEFAULT 0")
|
|
||||||
)
|
|
||||||
logger.warning(
|
|
||||||
"Applied SQLite compatibility schema patch table=job column=retry_count default=0"
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
@contextlib.contextmanager
|
|
||||||
def get_session(*, engine: Engine | None = None) -> Generator[Session]:
|
|
||||||
"""Yield a database session and ensure cleanup."""
|
|
||||||
active_engine = engine or get_database_runtime().engine
|
|
||||||
with Session(active_engine) as session:
|
|
||||||
yield session
|
|
||||||
@@ -0,0 +1,6 @@
|
|||||||
|
from .operations import create_all
|
||||||
|
from .runtime import dispose_database_runtime
|
||||||
|
from .runtime import get_session
|
||||||
|
from .runtime import initialize_database_runtime
|
||||||
|
|
||||||
|
__all__ = ["create_all", "dispose_database_runtime", "get_session", "initialize_database_runtime"]
|
||||||
@@ -0,0 +1,66 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import logging
|
||||||
|
|
||||||
|
from sqlalchemy import inspect
|
||||||
|
from sqlalchemy import text
|
||||||
|
from sqlalchemy.engine import Connection
|
||||||
|
from sqlalchemy.ext.asyncio import AsyncEngine
|
||||||
|
from sqlmodel import SQLModel
|
||||||
|
from sqlmodel import select
|
||||||
|
from sqlmodel.ext.asyncio.session import AsyncSession
|
||||||
|
|
||||||
|
from ..models import Job
|
||||||
|
from ..models import JobStatus
|
||||||
|
from .runtime import get_engine
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
async def get_next_queued_job(*, session: AsyncSession) -> Job | None:
|
||||||
|
"""Get the next queued job, if any."""
|
||||||
|
result = await session.exec(
|
||||||
|
select(Job)
|
||||||
|
.where(Job.status == JobStatus.QUEUED)
|
||||||
|
.order_by(Job.created_at) # pyright: ignore[reportArgumentType]
|
||||||
|
.limit(1)
|
||||||
|
) # fmt: skip
|
||||||
|
return result.first()
|
||||||
|
|
||||||
|
|
||||||
|
async def create_all(*, engine: AsyncEngine | None = None) -> None:
|
||||||
|
"""Create all tables on the selected engine."""
|
||||||
|
# Import models so SQLModel metadata is fully registered before bootstrap.
|
||||||
|
from transcription import models as _models # noqa: F401
|
||||||
|
|
||||||
|
active_engine = engine or get_engine()
|
||||||
|
async with active_engine.begin() as connection:
|
||||||
|
await connection.run_sync(SQLModel.metadata.create_all)
|
||||||
|
await connection.run_sync(_ensure_sqlite_compat_columns)
|
||||||
|
logger.debug("Database schema bootstrap complete for database_url=%s", active_engine.url)
|
||||||
|
|
||||||
|
|
||||||
|
def _ensure_sqlite_compat_columns(connection: Connection) -> None:
|
||||||
|
"""Apply lightweight dev/test SQLite compatibility column patches.
|
||||||
|
|
||||||
|
This keeps local bootstrap resilient when models evolve but no full
|
||||||
|
migration tooling is in place yet.
|
||||||
|
"""
|
||||||
|
if connection.engine.url.get_backend_name() != "sqlite":
|
||||||
|
return
|
||||||
|
|
||||||
|
inspector = inspect(connection)
|
||||||
|
table_names = set(inspector.get_table_names())
|
||||||
|
if "job" not in table_names:
|
||||||
|
return
|
||||||
|
|
||||||
|
columns = {column["name"] for column in inspector.get_columns("job")}
|
||||||
|
if "retry_count" not in columns:
|
||||||
|
connection.execute(text("ALTER TABLE job ADD COLUMN retry_count INTEGER NOT NULL DEFAULT 0"))
|
||||||
|
logger.warning("Applied SQLite compatibility schema patch table=job column=retry_count default=0")
|
||||||
|
|
||||||
|
if "transcript" in table_names:
|
||||||
|
transcript_columns = {column["name"] for column in inspector.get_columns("transcript")}
|
||||||
|
if "model" not in transcript_columns:
|
||||||
|
connection.execute(text("ALTER TABLE transcript ADD COLUMN model VARCHAR"))
|
||||||
|
logger.warning("Applied SQLite compatibility schema patch table=transcript column=model")
|
||||||
@@ -0,0 +1,103 @@
|
|||||||
|
import logging
|
||||||
|
from collections.abc import AsyncGenerator
|
||||||
|
from contextlib import asynccontextmanager
|
||||||
|
from contextvars import ContextVar
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from functools import partial
|
||||||
|
|
||||||
|
from sqlalchemy.ext.asyncio import AsyncEngine
|
||||||
|
from sqlalchemy.ext.asyncio import async_sessionmaker
|
||||||
|
from sqlalchemy.ext.asyncio import create_async_engine
|
||||||
|
from sqlmodel.ext.asyncio.session import AsyncSession
|
||||||
|
from sqlmodel.pool import StaticPool
|
||||||
|
|
||||||
|
from ..config import Settings
|
||||||
|
from ..config import get_settings
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class DatabaseRuntime:
|
||||||
|
"""Database runtime resources owned by app lifespan."""
|
||||||
|
|
||||||
|
engine: AsyncEngine
|
||||||
|
session_factory: async_sessionmaker[AsyncSession]
|
||||||
|
|
||||||
|
|
||||||
|
_runtime: ContextVar[DatabaseRuntime | None] = ContextVar("database_runtime", default=None)
|
||||||
|
|
||||||
|
|
||||||
|
async def dispose_database_runtime() -> None:
|
||||||
|
"""Dispose lifespan-owned async database resources."""
|
||||||
|
runtime = _runtime.get()
|
||||||
|
if runtime is None:
|
||||||
|
return
|
||||||
|
await runtime.engine.dispose()
|
||||||
|
_runtime.set(None)
|
||||||
|
|
||||||
|
|
||||||
|
def _to_async_database_url(database_url: str) -> str:
|
||||||
|
"""Normalize configured database URL to an async SQLAlchemy driver URL."""
|
||||||
|
if database_url.startswith("sqlite://") and not database_url.startswith("sqlite+aiosqlite://"):
|
||||||
|
return database_url.replace("sqlite://", "sqlite+aiosqlite://", 1)
|
||||||
|
if database_url.startswith("postgresql://") and not database_url.startswith("postgresql+asyncpg://"):
|
||||||
|
return database_url.replace("postgresql://", "postgresql+asyncpg://", 1)
|
||||||
|
return database_url
|
||||||
|
|
||||||
|
|
||||||
|
def _build_engine(settings: Settings) -> AsyncEngine:
|
||||||
|
database_url = _to_async_database_url(settings.database_url)
|
||||||
|
engine_factory = partial(
|
||||||
|
create_async_engine,
|
||||||
|
url=database_url,
|
||||||
|
echo=False,
|
||||||
|
pool_pre_ping=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
if database_url.startswith("sqlite"):
|
||||||
|
sqlite_connect_settings = {"check_same_thread": settings.sqlite_check_same_thread}
|
||||||
|
engine_factory = partial(engine_factory, connect_args=sqlite_connect_settings)
|
||||||
|
if ":memory:" in database_url:
|
||||||
|
engine_factory = partial(engine_factory, poolclass=StaticPool)
|
||||||
|
|
||||||
|
return engine_factory()
|
||||||
|
|
||||||
|
|
||||||
|
def initialize_database_runtime(*, settings: Settings | None = None) -> DatabaseRuntime:
|
||||||
|
"""Initialize lifespan-owned async DB resources once per process."""
|
||||||
|
runtime = _runtime.get()
|
||||||
|
if runtime is not None:
|
||||||
|
return runtime
|
||||||
|
|
||||||
|
active_settings = settings or get_settings()
|
||||||
|
engine = _build_engine(active_settings)
|
||||||
|
session_factory = async_sessionmaker(engine, class_=AsyncSession, expire_on_commit=False)
|
||||||
|
runtime = DatabaseRuntime(engine=engine, session_factory=session_factory)
|
||||||
|
_runtime.set(runtime)
|
||||||
|
logger.debug("Initialized async database runtime for database_url=%s", engine.url)
|
||||||
|
return runtime
|
||||||
|
|
||||||
|
|
||||||
|
def get_engine(settings: Settings | None = None) -> AsyncEngine:
|
||||||
|
"""Return the current async SQLAlchemy engine."""
|
||||||
|
runtime = _runtime.get() or initialize_database_runtime(settings=settings)
|
||||||
|
return runtime.engine
|
||||||
|
|
||||||
|
|
||||||
|
def get_session_factory(settings: Settings | None = None) -> async_sessionmaker[AsyncSession]:
|
||||||
|
"""Return the shared async session factory."""
|
||||||
|
runtime = _runtime.get() or initialize_database_runtime(settings=settings)
|
||||||
|
return runtime.session_factory
|
||||||
|
|
||||||
|
|
||||||
|
@asynccontextmanager
|
||||||
|
async def get_session(
|
||||||
|
*,
|
||||||
|
settings: Settings | None = None,
|
||||||
|
session_factory: async_sessionmaker[AsyncSession] | None = None,
|
||||||
|
) -> AsyncGenerator[AsyncSession]:
|
||||||
|
"""Yield a database session and ensure cleanup."""
|
||||||
|
active_session_factory = session_factory or get_session_factory(settings)
|
||||||
|
async with active_session_factory() as session:
|
||||||
|
yield session
|
||||||
@@ -3,7 +3,8 @@
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
from dataclasses import dataclass
|
from dataclasses import dataclass
|
||||||
from datetime import datetime, timezone
|
from datetime import UTC
|
||||||
|
from datetime import datetime
|
||||||
from enum import StrEnum
|
from enum import StrEnum
|
||||||
from uuid import uuid4
|
from uuid import uuid4
|
||||||
|
|
||||||
@@ -16,6 +17,7 @@ class ErrorCategory(StrEnum):
|
|||||||
NOT_FOUND = "not_found_error"
|
NOT_FOUND = "not_found_error"
|
||||||
CONFLICT = "conflict_error"
|
CONFLICT = "conflict_error"
|
||||||
EXTERNAL_PROVIDER = "external_provider_error"
|
EXTERNAL_PROVIDER = "external_provider_error"
|
||||||
|
PROCESSING = "processing_error"
|
||||||
INFRA_TRANSIENT = "infrastructure_transient_error"
|
INFRA_TRANSIENT = "infrastructure_transient_error"
|
||||||
INFRA_PERSISTENT = "infrastructure_persistent_error"
|
INFRA_PERSISTENT = "infrastructure_persistent_error"
|
||||||
INTERNAL_UNEXPECTED = "internal_unexpected_error"
|
INTERNAL_UNEXPECTED = "internal_unexpected_error"
|
||||||
@@ -64,7 +66,7 @@ def build_error_envelope(error: AppError) -> ErrorEnvelope:
|
|||||||
category=error.category.value,
|
category=error.category.value,
|
||||||
message=error.message,
|
message=error.message,
|
||||||
suggestion=error.suggestion,
|
suggestion=error.suggestion,
|
||||||
timestamp=datetime.now(timezone.utc).isoformat(),
|
timestamp=datetime.now(UTC).isoformat(),
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@@ -80,7 +82,4 @@ def classify_unexpected_error(exc: Exception, *, operation: str) -> AppError:
|
|||||||
|
|
||||||
def format_error_detail(error: AppError) -> str:
|
def format_error_detail(error: AppError) -> str:
|
||||||
"""Return a compact persisted failure string for transcript.error_detail."""
|
"""Return a compact persisted failure string for transcript.error_detail."""
|
||||||
return (
|
return f"[{error.category.value}] {error.message} | suggestion={error.suggestion} | error_id={error.error_id}"
|
||||||
f"[{error.category.value}] {error.message} | "
|
|
||||||
f"suggestion={error.suggestion} | error_id={error.error_id}"
|
|
||||||
)
|
|
||||||
|
|||||||
@@ -1,15 +1,22 @@
|
|||||||
"""SQLModel domain models for the transcription system.
|
"""SQLModel domain models for the transcription system.
|
||||||
|
|
||||||
Three models capture the MVP lifecycle:
|
Core models capture the MVP lifecycle:
|
||||||
Document -> one-to-many -> Job -> one-to-one -> Transcript
|
Document (1) -> (many) Job
|
||||||
|
Job (1) -> (1) Transcript
|
||||||
|
Job (1) -> (many) TranscriptRevision
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from datetime import datetime, timezone
|
from datetime import UTC
|
||||||
|
from datetime import datetime
|
||||||
from enum import StrEnum
|
from enum import StrEnum
|
||||||
from typing import Optional
|
from typing import Optional
|
||||||
from uuid import UUID, uuid4
|
from uuid import UUID
|
||||||
|
from uuid import uuid4
|
||||||
|
|
||||||
from sqlmodel import Field, Relationship, SQLModel
|
from sqlalchemy import UniqueConstraint
|
||||||
|
from sqlmodel import Field
|
||||||
|
from sqlmodel import Relationship
|
||||||
|
from sqlmodel import SQLModel
|
||||||
|
|
||||||
|
|
||||||
class JobStatus(StrEnum):
|
class JobStatus(StrEnum):
|
||||||
@@ -25,9 +32,7 @@ class Document(SQLModel, table=True):
|
|||||||
id: UUID = Field(default_factory=uuid4, primary_key=True)
|
id: UUID = Field(default_factory=uuid4, primary_key=True)
|
||||||
filename: str
|
filename: str
|
||||||
file_path: str
|
file_path: str
|
||||||
uploaded_at: datetime = Field(
|
uploaded_at: datetime = Field(default_factory=lambda: datetime.now(UTC))
|
||||||
default_factory=lambda: datetime.now(timezone.utc),
|
|
||||||
)
|
|
||||||
|
|
||||||
# --- relationships ---
|
# --- relationships ---
|
||||||
jobs: list["Job"] = Relationship(back_populates="document")
|
jobs: list["Job"] = Relationship(back_populates="document")
|
||||||
@@ -40,16 +45,18 @@ class Job(SQLModel, table=True):
|
|||||||
document_id: UUID = Field(foreign_key="document.id")
|
document_id: UUID = Field(foreign_key="document.id")
|
||||||
status: JobStatus = Field(default=JobStatus.QUEUED)
|
status: JobStatus = Field(default=JobStatus.QUEUED)
|
||||||
retry_count: int = Field(default=0, ge=0)
|
retry_count: int = Field(default=0, ge=0)
|
||||||
created_at: datetime = Field(
|
created_at: datetime = Field(default_factory=lambda: datetime.now(UTC))
|
||||||
default_factory=lambda: datetime.now(timezone.utc),
|
updated_at: datetime = Field(default_factory=lambda: datetime.now(UTC))
|
||||||
)
|
|
||||||
updated_at: datetime = Field(
|
|
||||||
default_factory=lambda: datetime.now(timezone.utc),
|
|
||||||
)
|
|
||||||
|
|
||||||
# --- relationships ---
|
# --- relationships ---
|
||||||
document: Document = Relationship(back_populates="jobs")
|
document: Document = Relationship(back_populates="jobs")
|
||||||
transcript: Optional["Transcript"] = Relationship(back_populates="job")
|
transcript: Optional["Transcript"] = Relationship(back_populates="job")
|
||||||
|
transcript_revisions: list["TranscriptRevision"] = Relationship(back_populates="job")
|
||||||
|
|
||||||
|
@property
|
||||||
|
def filename(self) -> str:
|
||||||
|
"""Return the filename of the associated document."""
|
||||||
|
return self.document.filename if self.document else "unknown"
|
||||||
|
|
||||||
|
|
||||||
class Transcript(SQLModel, table=True):
|
class Transcript(SQLModel, table=True):
|
||||||
@@ -57,11 +64,38 @@ class Transcript(SQLModel, table=True):
|
|||||||
|
|
||||||
id: UUID = Field(default_factory=uuid4, primary_key=True)
|
id: UUID = Field(default_factory=uuid4, primary_key=True)
|
||||||
job_id: UUID = Field(foreign_key="job.id", unique=True)
|
job_id: UUID = Field(foreign_key="job.id", unique=True)
|
||||||
|
"""ID for the associated job. There's a 1-1 relationship bewteen transcripts and jobs."""
|
||||||
|
provider: str
|
||||||
|
"""Name of the transcription provider used to generate this transcript."""
|
||||||
|
prompt_name: str
|
||||||
|
"""Name of the prompt used to generate this transcript."""
|
||||||
|
model: str | None = None
|
||||||
|
"""Provider model that generated the original AI transcript."""
|
||||||
text: str | None = None
|
text: str | None = None
|
||||||
|
"""The transcribed text. This may be None if the job failed or is still in progress."""
|
||||||
error_detail: str | None = None
|
error_detail: str | None = None
|
||||||
created_at: datetime = Field(
|
"""Details of any error that occurred during transcription."""
|
||||||
default_factory=lambda: datetime.now(timezone.utc),
|
created_at: datetime = Field(default_factory=lambda: datetime.now(UTC))
|
||||||
)
|
|
||||||
|
|
||||||
# --- relationships ---
|
# --- relationships ---
|
||||||
job: Job = Relationship(back_populates="transcript")
|
job: Job = Relationship(back_populates="transcript")
|
||||||
|
|
||||||
|
|
||||||
|
class TranscriptRevision(SQLModel, table=True):
|
||||||
|
"""Version history entries for a transcription job."""
|
||||||
|
|
||||||
|
__table_args__ = (UniqueConstraint("job_id", "version_number", name="uq_transcript_revision_job_version"),)
|
||||||
|
|
||||||
|
id: UUID = Field(default_factory=uuid4, primary_key=True)
|
||||||
|
job_id: UUID = Field(foreign_key="job.id", index=True)
|
||||||
|
version_number: int = Field(ge=1)
|
||||||
|
provider: str
|
||||||
|
prompt_name: str
|
||||||
|
model: str | None = None
|
||||||
|
source: str = Field(default="ai")
|
||||||
|
text: str | None = None
|
||||||
|
error_detail: str | None = None
|
||||||
|
created_at: datetime = Field(default_factory=lambda: datetime.now(UTC))
|
||||||
|
|
||||||
|
# --- relationships ---
|
||||||
|
job: Job = Relationship(back_populates="transcript_revisions")
|
||||||
|
|||||||
@@ -1,13 +1,13 @@
|
|||||||
"""Provider exports and factory for transcription adapters."""
|
"""Provider exports and factory for transcription adapters."""
|
||||||
|
|
||||||
from transcription.config import Provider, Settings, get_settings
|
from transcription.config import Provider
|
||||||
from transcription.providers.base import (
|
from transcription.config import Settings
|
||||||
ProviderAuthError,
|
from transcription.config import get_settings
|
||||||
ProviderError,
|
from transcription.providers.base import ProviderAuthError
|
||||||
ProviderResponseError,
|
from transcription.providers.base import ProviderError
|
||||||
TranscriptionProvider,
|
from transcription.providers.base import ProviderResponseError
|
||||||
TranscriptionResult,
|
from transcription.providers.base import TranscriptionProvider
|
||||||
)
|
from transcription.providers.base import TranscriptionResult
|
||||||
from transcription.providers.openrouter import OpenRouterTranscriptionProvider
|
from transcription.providers.openrouter import OpenRouterTranscriptionProvider
|
||||||
|
|
||||||
|
|
||||||
@@ -21,11 +21,11 @@ def get_transcription_provider(*, settings: Settings | None = None) -> Transcrip
|
|||||||
|
|
||||||
|
|
||||||
__all__ = [
|
__all__ = [
|
||||||
|
"OpenRouterTranscriptionProvider",
|
||||||
"ProviderAuthError",
|
"ProviderAuthError",
|
||||||
"ProviderError",
|
"ProviderError",
|
||||||
"ProviderResponseError",
|
"ProviderResponseError",
|
||||||
"TranscriptionProvider",
|
"TranscriptionProvider",
|
||||||
"TranscriptionResult",
|
"TranscriptionResult",
|
||||||
"OpenRouterTranscriptionProvider",
|
|
||||||
"get_transcription_provider",
|
"get_transcription_provider",
|
||||||
]
|
]
|
||||||
|
|||||||
@@ -2,6 +2,9 @@
|
|||||||
|
|
||||||
from dataclasses import dataclass
|
from dataclasses import dataclass
|
||||||
from typing import Protocol
|
from typing import Protocol
|
||||||
|
from uuid import UUID
|
||||||
|
|
||||||
|
from ..models import Transcript
|
||||||
|
|
||||||
|
|
||||||
class ProviderError(RuntimeError):
|
class ProviderError(RuntimeError):
|
||||||
@@ -22,11 +25,23 @@ class TranscriptionResult:
|
|||||||
|
|
||||||
text: str
|
text: str
|
||||||
provider: str
|
provider: str
|
||||||
|
prompt_name: str
|
||||||
model: str
|
model: str
|
||||||
|
|
||||||
|
def to_transcript(self, job_id: UUID) -> Transcript:
|
||||||
|
"""Convert a TranscriptionResult to a Transcript model instance."""
|
||||||
|
return Transcript(
|
||||||
|
job_id=job_id,
|
||||||
|
provider=self.provider,
|
||||||
|
prompt_name=self.prompt_name,
|
||||||
|
model=self.model,
|
||||||
|
text=self.text,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
class TranscriptionProvider(Protocol):
|
class TranscriptionProvider(Protocol):
|
||||||
"""Contract every transcription provider adapter must satisfy."""
|
"""Contract every transcription provider adapter must satisfy."""
|
||||||
|
|
||||||
def transcribe(self, *, prompt_text: str, image_bytes: bytes, mime_type: str) -> TranscriptionResult:
|
async def transcribe(self, *, prompt_text: str, image_bytes: bytes, mime_type: str) -> TranscriptionResult:
|
||||||
"""Transcribe the provided image according to the prompt text."""
|
"""Transcribe the provided image according to the prompt text."""
|
||||||
|
...
|
||||||
|
|||||||
@@ -6,16 +6,17 @@ import base64
|
|||||||
import logging
|
import logging
|
||||||
from dataclasses import dataclass
|
from dataclasses import dataclass
|
||||||
from typing import Any
|
from typing import Any
|
||||||
|
from typing import cast
|
||||||
|
|
||||||
from openrouter import OpenRouter
|
from openrouter import OpenRouter
|
||||||
|
from openrouter.components.chatmessages import ChatMessagesTypedDict
|
||||||
|
|
||||||
from transcription.config import Settings, get_settings
|
from transcription.config import Settings
|
||||||
from transcription.providers.base import (
|
from transcription.config import get_settings
|
||||||
ProviderAuthError,
|
from transcription.providers.base import ProviderAuthError
|
||||||
ProviderError,
|
from transcription.providers.base import ProviderError
|
||||||
ProviderResponseError,
|
from transcription.providers.base import ProviderResponseError
|
||||||
TranscriptionResult,
|
from transcription.providers.base import TranscriptionResult
|
||||||
)
|
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
@@ -45,17 +46,17 @@ class OpenRouterTranscriptionProvider:
|
|||||||
"""Return the resolved OpenRouter model slug."""
|
"""Return the resolved OpenRouter model slug."""
|
||||||
return self._model
|
return self._model
|
||||||
|
|
||||||
def transcribe(self, *, prompt_text: str, image_bytes: bytes, mime_type: str) -> TranscriptionResult:
|
async def transcribe(self, *, prompt_text: str, image_bytes: bytes, mime_type: str) -> TranscriptionResult:
|
||||||
"""Send prompt + image to OpenRouter and return normalized text output."""
|
"""Send prompt + image to OpenRouter and return normalized text output."""
|
||||||
request = self._build_request(prompt_text=prompt_text, image_bytes=image_bytes, mime_type=mime_type)
|
request = self._build_request(prompt_text=prompt_text, image_bytes=image_bytes, mime_type=mime_type)
|
||||||
try:
|
try:
|
||||||
response = self._client.chat.send(
|
response = await self._client.chat.send_async(
|
||||||
messages=request.messages,
|
messages=cast(list[ChatMessagesTypedDict], request.messages),
|
||||||
model=request.model,
|
model=request.model,
|
||||||
http_referer=request.http_referer,
|
http_referer=request.http_referer,
|
||||||
x_open_router_title=request.x_open_router_title,
|
x_open_router_title=request.x_open_router_title,
|
||||||
)
|
)
|
||||||
except Exception as exc: # noqa: BLE001
|
except Exception as exc:
|
||||||
message = str(exc).lower()
|
message = str(exc).lower()
|
||||||
if "401" in message or "auth" in message or "api key" in message:
|
if "401" in message or "auth" in message or "api key" in message:
|
||||||
raise ProviderAuthError("OpenRouter authentication failed") from exc
|
raise ProviderAuthError("OpenRouter authentication failed") from exc
|
||||||
@@ -64,7 +65,7 @@ class OpenRouterTranscriptionProvider:
|
|||||||
text = self._extract_text(response)
|
text = self._extract_text(response)
|
||||||
model = self._get_optional_attr(response, "model") or self.model
|
model = self._get_optional_attr(response, "model") or self.model
|
||||||
logger.info("OpenRouter transcription completed using model=%s", model)
|
logger.info("OpenRouter transcription completed using model=%s", model)
|
||||||
return TranscriptionResult(text=text, provider="openrouter", model=model)
|
return TranscriptionResult(text=text, provider="openrouter", prompt_name="", model=model)
|
||||||
|
|
||||||
def _build_request(self, *, prompt_text: str, image_bytes: bytes, mime_type: str) -> OpenRouterRequest:
|
def _build_request(self, *, prompt_text: str, image_bytes: bytes, mime_type: str) -> OpenRouterRequest:
|
||||||
image_b64 = base64.b64encode(image_bytes).decode("ascii")
|
image_b64 = base64.b64encode(image_bytes).decode("ascii")
|
||||||
@@ -111,10 +112,7 @@ class OpenRouterTranscriptionProvider:
|
|||||||
parts: list[str] = []
|
parts: list[str] = []
|
||||||
for item in content:
|
for item in content:
|
||||||
text_part = None
|
text_part = None
|
||||||
if isinstance(item, dict):
|
text_part = item.get("text") if isinstance(item, dict) else self._get_optional_attr(item, "text")
|
||||||
text_part = item.get("text")
|
|
||||||
else:
|
|
||||||
text_part = self._get_optional_attr(item, "text")
|
|
||||||
|
|
||||||
if isinstance(text_part, str) and text_part.strip():
|
if isinstance(text_part, str) and text_part.strip():
|
||||||
parts.append(text_part.strip())
|
parts.append(text_part.strip())
|
||||||
|
|||||||
@@ -1,30 +1,19 @@
|
|||||||
"""Service layer exports."""
|
"""Service layer exports."""
|
||||||
|
|
||||||
from transcription.services.transcription import (
|
from dataclasses import dataclass
|
||||||
DEFAULT_PROMPT_FILE,
|
from dataclasses import field
|
||||||
PromptLoadError,
|
|
||||||
TranscriptionError,
|
|
||||||
load_image_payload,
|
|
||||||
load_prompt_text,
|
|
||||||
transcribe_document_image,
|
|
||||||
)
|
|
||||||
from transcription.services.upload import (
|
|
||||||
SUPPORTED_UPLOAD_EXTENSIONS,
|
|
||||||
UploadError,
|
|
||||||
UploadJobResult,
|
|
||||||
create_upload_job,
|
|
||||||
)
|
|
||||||
|
|
||||||
__all__ = [
|
from .documents import DocumentService
|
||||||
"DEFAULT_PROMPT_FILE",
|
from .jobs import JobService
|
||||||
"PromptLoadError",
|
from .transcription import TranscriptionService
|
||||||
"TranscriptionError",
|
|
||||||
"load_image_payload",
|
|
||||||
"load_prompt_text",
|
|
||||||
"transcribe_document_image",
|
|
||||||
"SUPPORTED_UPLOAD_EXTENSIONS",
|
|
||||||
"UploadError",
|
|
||||||
"UploadJobResult",
|
|
||||||
"create_upload_job",
|
|
||||||
]
|
|
||||||
|
|
||||||
|
__all__ = ["DocumentService", "JobService", "ServiceBundle", "TranscriptionService"]
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class ServiceBundle:
|
||||||
|
"""Container for all service instances."""
|
||||||
|
|
||||||
|
documents: DocumentService = field(default_factory=DocumentService)
|
||||||
|
jobs: JobService = field(default_factory=JobService)
|
||||||
|
transcriptions: TranscriptionService = field(default_factory=TranscriptionService)
|
||||||
|
|||||||
@@ -0,0 +1,60 @@
|
|||||||
|
import asyncio
|
||||||
|
from abc import ABC
|
||||||
|
from collections.abc import Sequence
|
||||||
|
from contextlib import asynccontextmanager
|
||||||
|
|
||||||
|
from sqlalchemy.ext.asyncio import async_sessionmaker
|
||||||
|
from sqlmodel.ext.asyncio.session import AsyncSession
|
||||||
|
|
||||||
|
from ..config import Settings
|
||||||
|
from ..config import get_settings
|
||||||
|
from ..db.runtime import get_session_factory
|
||||||
|
|
||||||
|
|
||||||
|
class ServiceBase(ABC):
|
||||||
|
"""Thin service class for managing documents in the database."""
|
||||||
|
|
||||||
|
settings: Settings
|
||||||
|
session_factory: async_sessionmaker[AsyncSession]
|
||||||
|
queue: asyncio.Queue
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
session_factory: async_sessionmaker[AsyncSession] | None = None,
|
||||||
|
queue: asyncio.Queue | None = None,
|
||||||
|
):
|
||||||
|
self.settings = get_settings()
|
||||||
|
self.session_factory = session_factory or get_session_factory()
|
||||||
|
self.queue = queue or asyncio.Queue()
|
||||||
|
|
||||||
|
@asynccontextmanager
|
||||||
|
async def _session_scope(self, session: AsyncSession | None = None):
|
||||||
|
"""Provide a transactional scope around a series of operations."""
|
||||||
|
if session is not None:
|
||||||
|
# Reuse the provided session if one is passed in
|
||||||
|
yield session
|
||||||
|
else:
|
||||||
|
# Otherwise, create a new session for this scope
|
||||||
|
async with self.session_factory() as new_session:
|
||||||
|
yield new_session
|
||||||
|
|
||||||
|
async def _finalize(
|
||||||
|
self,
|
||||||
|
*,
|
||||||
|
session: AsyncSession,
|
||||||
|
caller_session: AsyncSession | None,
|
||||||
|
refresh: Sequence[object] = (),
|
||||||
|
) -> None:
|
||||||
|
"""Finalize a write based on transaction ownership.
|
||||||
|
|
||||||
|
Service-owned sessions commit immediately. Caller-owned sessions flush so
|
||||||
|
orchestration code can commit once at a larger transaction boundary.
|
||||||
|
"""
|
||||||
|
should_commit = caller_session is None
|
||||||
|
if should_commit:
|
||||||
|
await session.commit()
|
||||||
|
else:
|
||||||
|
await session.flush()
|
||||||
|
|
||||||
|
for obj in refresh:
|
||||||
|
await session.refresh(obj)
|
||||||
@@ -0,0 +1,127 @@
|
|||||||
|
import logging
|
||||||
|
from collections.abc import Sequence
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from pathlib import Path
|
||||||
|
from uuid import UUID
|
||||||
|
|
||||||
|
from sqlalchemy.exc import IntegrityError
|
||||||
|
from sqlalchemy.orm import selectinload
|
||||||
|
from sqlmodel import select
|
||||||
|
from sqlmodel.ext.asyncio.session import AsyncSession
|
||||||
|
|
||||||
|
from ..errors import AppError
|
||||||
|
from ..errors import ErrorCategory
|
||||||
|
from ..models import Document
|
||||||
|
from .base import ServiceBase
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
class DocumentError(AppError):
|
||||||
|
"""Raised when document operations fail."""
|
||||||
|
|
||||||
|
|
||||||
|
class MissingImageError(DocumentError):
|
||||||
|
"""Raised when a required image is missing."""
|
||||||
|
|
||||||
|
|
||||||
|
class UploadError(DocumentError):
|
||||||
|
"""Raised when uploaded content cannot be persisted safely."""
|
||||||
|
|
||||||
|
|
||||||
|
class DocumentAlreadyExistsError(DocumentError):
|
||||||
|
"""Raised when a document with the same filename already exists in the database."""
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class UploadJobResult:
|
||||||
|
"""Summary of created upload records."""
|
||||||
|
|
||||||
|
document_id: UUID
|
||||||
|
job_id: UUID
|
||||||
|
stored_path: Path
|
||||||
|
original_filename: str
|
||||||
|
|
||||||
|
|
||||||
|
class DocumentService(ServiceBase):
|
||||||
|
"""Thin service class for managing documents in the database."""
|
||||||
|
|
||||||
|
#
|
||||||
|
# CRUD Operations
|
||||||
|
#
|
||||||
|
|
||||||
|
async def create_document(
|
||||||
|
self,
|
||||||
|
document: Document,
|
||||||
|
*,
|
||||||
|
session: AsyncSession | None = None,
|
||||||
|
) -> Document:
|
||||||
|
"""Create a new document in the database."""
|
||||||
|
async with self._session_scope(session) as _session:
|
||||||
|
_session.add(document)
|
||||||
|
try:
|
||||||
|
await self._finalize(session=_session, caller_session=session, refresh=(document,))
|
||||||
|
except IntegrityError as exc:
|
||||||
|
raise DocumentAlreadyExistsError(
|
||||||
|
f"Document with id {document.id} already exists",
|
||||||
|
category=ErrorCategory.VALIDATION,
|
||||||
|
suggestion="Rename the file and try again.",
|
||||||
|
) from exc
|
||||||
|
return document
|
||||||
|
|
||||||
|
async def read_document(self, document_id: UUID, *, session: AsyncSession | None = None) -> Document:
|
||||||
|
"""Read an existing document from the database.
|
||||||
|
|
||||||
|
The selectinload option is used to eagerly load related jobs for the document.
|
||||||
|
"""
|
||||||
|
async with self._session_scope(session) as _session:
|
||||||
|
document = await _session.get(
|
||||||
|
Document,
|
||||||
|
document_id,
|
||||||
|
options=(selectinload(Document.jobs),), # pyright: ignore[reportArgumentType]
|
||||||
|
)
|
||||||
|
if document is None:
|
||||||
|
raise DocumentError(
|
||||||
|
f"Document with id {document_id} not found",
|
||||||
|
category=ErrorCategory.NOT_FOUND,
|
||||||
|
suggestion="Re-upload the source document and retry.",
|
||||||
|
)
|
||||||
|
elif not Path(document.file_path).exists():
|
||||||
|
raise MissingImageError(
|
||||||
|
f"Document with id {document_id} is missing its image file in {self.settings.upload_dir:!s}",
|
||||||
|
category=ErrorCategory.NOT_FOUND,
|
||||||
|
suggestion="Re-upload the source document and retry.",
|
||||||
|
)
|
||||||
|
return document
|
||||||
|
|
||||||
|
async def update_document(self, document: Document, *, session: AsyncSession | None = None) -> Document:
|
||||||
|
"""Update an existing document in the database."""
|
||||||
|
async with self._session_scope(session) as _session:
|
||||||
|
merged = await _session.merge(document)
|
||||||
|
await self._finalize(session=_session, caller_session=session, refresh=(merged,))
|
||||||
|
return merged
|
||||||
|
|
||||||
|
async def delete_document(self, document: Document, *, session: AsyncSession | None = None) -> None:
|
||||||
|
"""Delete a document from the database."""
|
||||||
|
async with self._session_scope(session) as _session:
|
||||||
|
await _session.delete(document)
|
||||||
|
await self._finalize(session=_session, caller_session=session)
|
||||||
|
|
||||||
|
# Query Operations
|
||||||
|
|
||||||
|
async def query_documents(
|
||||||
|
self, *, filename: str | None = None, session: AsyncSession | None = None
|
||||||
|
) -> Sequence[Document]:
|
||||||
|
"""Query documents from the database based on provided filters."""
|
||||||
|
async with self._session_scope(session) as _session:
|
||||||
|
query = select(Document)
|
||||||
|
if filename is not None:
|
||||||
|
query = query.where(Document.filename == filename)
|
||||||
|
result = await _session.exec(query)
|
||||||
|
return result.all()
|
||||||
|
|
||||||
|
async def list_documents(self, *, session: AsyncSession | None = None) -> Sequence[Document]:
|
||||||
|
"""List all documents in the database."""
|
||||||
|
async with self._session_scope(session) as _session:
|
||||||
|
result = await _session.exec(select(Document))
|
||||||
|
return result.all()
|
||||||
@@ -0,0 +1,146 @@
|
|||||||
|
from collections.abc import Sequence
|
||||||
|
from datetime import UTC
|
||||||
|
from datetime import datetime
|
||||||
|
from uuid import UUID
|
||||||
|
|
||||||
|
from sqlalchemy.orm import selectinload
|
||||||
|
from sqlmodel import select
|
||||||
|
from sqlmodel.ext.asyncio.session import AsyncSession
|
||||||
|
|
||||||
|
from ..models import Job
|
||||||
|
from ..models import JobStatus
|
||||||
|
from .base import ServiceBase
|
||||||
|
|
||||||
|
|
||||||
|
class JobService(ServiceBase):
|
||||||
|
"""Thin service class for managing jobs in the database."""
|
||||||
|
|
||||||
|
#
|
||||||
|
# CRUD Operations
|
||||||
|
#
|
||||||
|
|
||||||
|
async def create_job(self, job: Job, session: AsyncSession | None = None) -> Job:
|
||||||
|
"""Create a new job in the database."""
|
||||||
|
async with self._session_scope(session) as _session:
|
||||||
|
_session.add(job)
|
||||||
|
await self._finalize(session=_session, caller_session=session, refresh=(job,))
|
||||||
|
return job
|
||||||
|
|
||||||
|
async def read_job(self, job_id: UUID, session: AsyncSession | None = None) -> Job:
|
||||||
|
"""Read an existing job from the database.
|
||||||
|
|
||||||
|
The related document is always eagerly loaded so callers can safely
|
||||||
|
access ``job.document`` in async contexts without triggering lazy-load IO.
|
||||||
|
"""
|
||||||
|
async with self._session_scope(session) as _session:
|
||||||
|
query = (
|
||||||
|
select(Job)
|
||||||
|
.options(selectinload(Job.document)) # pyright: ignore[reportArgumentType]
|
||||||
|
.where(Job.id == job_id)
|
||||||
|
.execution_options(populate_existing=True)
|
||||||
|
)
|
||||||
|
job = (await _session.exec(query)).first()
|
||||||
|
if job is None:
|
||||||
|
raise ValueError(f"Job with id {job_id} not found")
|
||||||
|
return job
|
||||||
|
|
||||||
|
async def update_job(self, job: Job, session: AsyncSession | None = None) -> Job:
|
||||||
|
"""Update an existing job in the database."""
|
||||||
|
async with self._session_scope(session) as _session:
|
||||||
|
merged = await _session.merge(job)
|
||||||
|
await self._finalize(session=_session, caller_session=session, refresh=(merged,))
|
||||||
|
return merged
|
||||||
|
|
||||||
|
async def delete_job(self, job: Job, session: AsyncSession | None = None) -> None:
|
||||||
|
"""Delete a job from the database."""
|
||||||
|
async with self._session_scope(session) as _session:
|
||||||
|
await _session.delete(job)
|
||||||
|
await self._finalize(session=_session, caller_session=session)
|
||||||
|
|
||||||
|
# Query Operations
|
||||||
|
|
||||||
|
async def query_jobs(
|
||||||
|
self,
|
||||||
|
*,
|
||||||
|
status: JobStatus | None = None,
|
||||||
|
filename: str | None = None,
|
||||||
|
session: AsyncSession | None = None,
|
||||||
|
) -> Sequence[Job]:
|
||||||
|
"""Query jobs from the database based on provided filters."""
|
||||||
|
async with self._session_scope(session) as _session:
|
||||||
|
query = select(Job).options(selectinload(Job.document)) # pyright: ignore[reportArgumentType]
|
||||||
|
if status is not None:
|
||||||
|
query = query.where(Job.status == status)
|
||||||
|
if filename is not None:
|
||||||
|
query = query.where(Job.document.filename == filename)
|
||||||
|
result = await _session.exec(query)
|
||||||
|
return result.all()
|
||||||
|
|
||||||
|
async def list_jobs(
|
||||||
|
self,
|
||||||
|
*,
|
||||||
|
load_docs: bool = False,
|
||||||
|
session: AsyncSession | None = None,
|
||||||
|
) -> Sequence[Job]:
|
||||||
|
"""List all jobs in the database with eagerly loaded documents."""
|
||||||
|
_ = load_docs
|
||||||
|
async with self._session_scope(session) as _session:
|
||||||
|
query = select(Job).options(selectinload(Job.document)) # pyright: ignore[reportArgumentType]
|
||||||
|
result = await _session.exec(query)
|
||||||
|
return result.all()
|
||||||
|
|
||||||
|
# Other Operations
|
||||||
|
|
||||||
|
async def mark_job_status(
|
||||||
|
self,
|
||||||
|
job_id: UUID,
|
||||||
|
status: JobStatus,
|
||||||
|
session: AsyncSession | None = None,
|
||||||
|
) -> Job:
|
||||||
|
"""Mark a job with a new status."""
|
||||||
|
return await self.update_job_state(job_id=job_id, status=status, session=session)
|
||||||
|
|
||||||
|
async def update_job_state(
|
||||||
|
self,
|
||||||
|
*,
|
||||||
|
job_id: UUID,
|
||||||
|
status: JobStatus,
|
||||||
|
retry_count_increment: int = 0,
|
||||||
|
session: AsyncSession | None = None,
|
||||||
|
) -> Job:
|
||||||
|
"""Update a job's lifecycle fields.
|
||||||
|
|
||||||
|
When ``session`` is provided, this method flushes so callers can commit
|
||||||
|
once at an orchestration boundary.
|
||||||
|
"""
|
||||||
|
async with self._session_scope(session) as _session:
|
||||||
|
query = (
|
||||||
|
select(Job)
|
||||||
|
.options(selectinload(Job.document)) # pyright: ignore[reportArgumentType]
|
||||||
|
.where(Job.id == job_id)
|
||||||
|
.execution_options(populate_existing=True)
|
||||||
|
)
|
||||||
|
job = (await _session.exec(query)).first()
|
||||||
|
if job is None:
|
||||||
|
raise ValueError(f"Job with id {job_id} not found")
|
||||||
|
job.status = status
|
||||||
|
if retry_count_increment:
|
||||||
|
job.retry_count += retry_count_increment
|
||||||
|
job.updated_at = datetime.now(UTC)
|
||||||
|
await self._finalize(session=_session, caller_session=session, refresh=(job,))
|
||||||
|
return job
|
||||||
|
|
||||||
|
async def read_next_queued_job(
|
||||||
|
self,
|
||||||
|
*,
|
||||||
|
session: AsyncSession | None = None,
|
||||||
|
) -> Job | None:
|
||||||
|
"""Read the next queued job ordered by creation time."""
|
||||||
|
async with self._session_scope(session) as _session:
|
||||||
|
query = (
|
||||||
|
select(Job)
|
||||||
|
.options(selectinload(Job.document)) # pyright: ignore[reportArgumentType]
|
||||||
|
.where(Job.status == JobStatus.QUEUED)
|
||||||
|
.order_by(Job.created_at) # pyright: ignore[reportArgumentType]
|
||||||
|
)
|
||||||
|
return (await _session.exec(query)).first()
|
||||||
@@ -1,18 +1,19 @@
|
|||||||
"""Upload service for storing files and creating queued transcription jobs."""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import logging
|
import logging
|
||||||
from dataclasses import dataclass
|
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from uuid import UUID, uuid4
|
from uuid import uuid4
|
||||||
|
|
||||||
from sqlmodel import Session
|
from sqlmodel.ext.asyncio.session import AsyncSession
|
||||||
|
|
||||||
from transcription.config import Settings, get_settings
|
from transcription.config import Settings
|
||||||
from transcription.db import get_session
|
from transcription.config import get_settings
|
||||||
from transcription.errors import AppError, ErrorCategory
|
from transcription.errors import AppError
|
||||||
from transcription.models import Document, Job, JobStatus
|
from transcription.errors import ErrorCategory
|
||||||
|
|
||||||
|
from ..models import Document
|
||||||
|
from ..models import Job
|
||||||
|
from .documents import UploadJobResult
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
@@ -23,24 +24,75 @@ class UploadError(AppError):
|
|||||||
"""Raised when uploaded content cannot be persisted safely."""
|
"""Raised when uploaded content cannot be persisted safely."""
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
async def create_upload_job(
|
||||||
class UploadJobResult:
|
|
||||||
"""Summary of created upload records."""
|
|
||||||
|
|
||||||
document_id: UUID
|
|
||||||
job_id: UUID
|
|
||||||
stored_path: Path
|
|
||||||
original_filename: str
|
|
||||||
|
|
||||||
|
|
||||||
def create_upload_job(
|
|
||||||
*,
|
*,
|
||||||
filename: str,
|
filename: str,
|
||||||
file_bytes: bytes,
|
file_bytes: bytes,
|
||||||
session: Session | None = None,
|
session: AsyncSession,
|
||||||
settings: Settings | None = None,
|
settings: Settings | None = None,
|
||||||
) -> UploadJobResult:
|
) -> UploadJobResult:
|
||||||
"""Persist an uploaded file and create document/job records."""
|
"""Create upload-backed document and queued job records."""
|
||||||
|
runtime_settings = settings or get_settings()
|
||||||
|
stored_path = store_file(
|
||||||
|
filename=filename,
|
||||||
|
file_bytes=file_bytes,
|
||||||
|
settings=runtime_settings,
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
document, job = await _create_upload_records(
|
||||||
|
session=session,
|
||||||
|
original_filename=filename,
|
||||||
|
stored_path=stored_path,
|
||||||
|
)
|
||||||
|
except Exception as exc:
|
||||||
|
_best_effort_delete(stored_path)
|
||||||
|
raise UploadError(
|
||||||
|
"Failed to create upload database records",
|
||||||
|
category=ErrorCategory.INFRA_TRANSIENT,
|
||||||
|
suggestion="Retry upload. If this keeps happening, verify database availability.",
|
||||||
|
retriable=True,
|
||||||
|
) from exc
|
||||||
|
|
||||||
|
logger.info("Created upload job document_id=%s job_id=%s", document.id, job.id)
|
||||||
|
return UploadJobResult(
|
||||||
|
document_id=document.id,
|
||||||
|
job_id=job.id,
|
||||||
|
stored_path=stored_path,
|
||||||
|
original_filename=Path(filename).name,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
async def _create_upload_records(
|
||||||
|
*,
|
||||||
|
session: AsyncSession,
|
||||||
|
original_filename: str,
|
||||||
|
stored_path: Path,
|
||||||
|
) -> tuple[Document, Job]:
|
||||||
|
document = Document(
|
||||||
|
filename=Path(original_filename).name,
|
||||||
|
file_path=str(stored_path),
|
||||||
|
)
|
||||||
|
session.add(document)
|
||||||
|
await session.flush()
|
||||||
|
|
||||||
|
job = Job(document_id=document.id)
|
||||||
|
session.add(job)
|
||||||
|
await session.commit()
|
||||||
|
await session.refresh(document)
|
||||||
|
await session.refresh(job)
|
||||||
|
return document, job
|
||||||
|
|
||||||
|
|
||||||
|
def _best_effort_delete(path: Path) -> None:
|
||||||
|
try:
|
||||||
|
if path.exists():
|
||||||
|
path.unlink()
|
||||||
|
except OSError:
|
||||||
|
logger.warning("Failed to clean up upload file after DB error: %s", path)
|
||||||
|
|
||||||
|
|
||||||
|
def store_file(*, filename: str, file_bytes: bytes, settings: Settings | None = None) -> Path:
|
||||||
|
"""Persist an uploaded file to the configured upload directory."""
|
||||||
runtime_settings = settings or get_settings()
|
runtime_settings = settings or get_settings()
|
||||||
_validate_upload(filename=filename, file_bytes=file_bytes)
|
_validate_upload(filename=filename, file_bytes=file_bytes)
|
||||||
|
|
||||||
@@ -59,32 +111,8 @@ def create_upload_job(
|
|||||||
suggestion="Check upload directory permissions and available disk space, then retry.",
|
suggestion="Check upload directory permissions and available disk space, then retry.",
|
||||||
) from exc
|
) from exc
|
||||||
|
|
||||||
try:
|
logger.info("Stored uploaded file: %s", stored_path)
|
||||||
if session is not None:
|
return stored_path
|
||||||
document, job = _create_upload_records(session=session, original_filename=filename, stored_path=stored_path)
|
|
||||||
else:
|
|
||||||
with get_session() as local_session:
|
|
||||||
document, job = _create_upload_records(
|
|
||||||
session=local_session,
|
|
||||||
original_filename=filename,
|
|
||||||
stored_path=stored_path,
|
|
||||||
)
|
|
||||||
except Exception as exc: # noqa: BLE001
|
|
||||||
_best_effort_delete(stored_path)
|
|
||||||
raise UploadError(
|
|
||||||
"Failed to create upload database records",
|
|
||||||
category=ErrorCategory.INFRA_TRANSIENT,
|
|
||||||
suggestion="Retry upload. If this keeps happening, verify database availability.",
|
|
||||||
retriable=True,
|
|
||||||
) from exc
|
|
||||||
|
|
||||||
logger.info("Created upload job document_id=%s job_id=%s", document.id, job.id)
|
|
||||||
return UploadJobResult(
|
|
||||||
document_id=document.id,
|
|
||||||
job_id=job.id,
|
|
||||||
stored_path=stored_path,
|
|
||||||
original_filename=Path(filename).name,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _validate_upload(*, filename: str, file_bytes: bytes) -> None:
|
def _validate_upload(*, filename: str, file_bytes: bytes) -> None:
|
||||||
@@ -115,30 +143,3 @@ def _validate_upload(*, filename: str, file_bytes: bytes) -> None:
|
|||||||
def _build_stored_filename(filename: str) -> str:
|
def _build_stored_filename(filename: str) -> str:
|
||||||
safe_name = Path(filename).name
|
safe_name = Path(filename).name
|
||||||
return f"{uuid4()}_{safe_name}"
|
return f"{uuid4()}_{safe_name}"
|
||||||
|
|
||||||
|
|
||||||
def _create_upload_records(*, session: Session, original_filename: str, stored_path: Path) -> tuple[Document, Job]:
|
|
||||||
document = Document(
|
|
||||||
filename=Path(original_filename).name,
|
|
||||||
file_path=str(stored_path),
|
|
||||||
)
|
|
||||||
session.add(document)
|
|
||||||
session.flush()
|
|
||||||
|
|
||||||
job = Job(
|
|
||||||
document_id=document.id,
|
|
||||||
status=JobStatus.QUEUED,
|
|
||||||
)
|
|
||||||
session.add(job)
|
|
||||||
session.commit()
|
|
||||||
session.refresh(document)
|
|
||||||
session.refresh(job)
|
|
||||||
return document, job
|
|
||||||
|
|
||||||
|
|
||||||
def _best_effort_delete(path: Path) -> None:
|
|
||||||
try:
|
|
||||||
if path.exists():
|
|
||||||
path.unlink()
|
|
||||||
except OSError:
|
|
||||||
logger.warning("Failed to clean up upload file after DB error: %s", path)
|
|
||||||
@@ -4,18 +4,29 @@ from __future__ import annotations
|
|||||||
|
|
||||||
import logging
|
import logging
|
||||||
import mimetypes
|
import mimetypes
|
||||||
|
from contextlib import contextmanager
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
from uuid import UUID
|
||||||
|
|
||||||
from transcription.config import Settings, get_settings
|
from sqlalchemy.ext.asyncio import async_sessionmaker
|
||||||
from transcription.errors import AppError, ErrorCategory
|
from sqlalchemy.orm import selectinload
|
||||||
from transcription.providers import (
|
from sqlmodel import select
|
||||||
ProviderAuthError,
|
from sqlmodel.ext.asyncio.session import AsyncSession
|
||||||
ProviderError,
|
|
||||||
ProviderResponseError,
|
from transcription.config import Settings
|
||||||
TranscriptionProvider,
|
from transcription.config import get_settings
|
||||||
TranscriptionResult,
|
from transcription.errors import AppError
|
||||||
get_transcription_provider,
|
from transcription.errors import ErrorCategory
|
||||||
)
|
from transcription.models import Transcript
|
||||||
|
from transcription.models import TranscriptRevision
|
||||||
|
from transcription.providers import ProviderAuthError
|
||||||
|
from transcription.providers import ProviderError
|
||||||
|
from transcription.providers import ProviderResponseError
|
||||||
|
from transcription.providers import TranscriptionProvider
|
||||||
|
from transcription.providers import TranscriptionResult
|
||||||
|
from transcription.providers import get_transcription_provider
|
||||||
|
|
||||||
|
from .base import ServiceBase
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
@@ -31,6 +42,245 @@ class TranscriptionError(AppError):
|
|||||||
"""Raised when transcription execution fails."""
|
"""Raised when transcription execution fails."""
|
||||||
|
|
||||||
|
|
||||||
|
class TranscriptionNotFoundError(TranscriptionError):
|
||||||
|
"""Raised when a transcription is not found in the database."""
|
||||||
|
|
||||||
|
|
||||||
|
class TranscriptionService(ServiceBase):
|
||||||
|
"""Service class for managing transcription operations.
|
||||||
|
|
||||||
|
This is the top-level service that composes functionality from the other services."""
|
||||||
|
|
||||||
|
provider: TranscriptionProvider
|
||||||
|
|
||||||
|
def __init__(self, session_factory: async_sessionmaker[AsyncSession] | None = None):
|
||||||
|
super().__init__(session_factory=session_factory)
|
||||||
|
self.provider = get_transcription_provider(settings=self.settings)
|
||||||
|
|
||||||
|
async def create_transcript(self, transcript: Transcript, *, session: AsyncSession | None = None) -> Transcript:
|
||||||
|
"""Create a new transcript in the database."""
|
||||||
|
async with self._session_scope(session) as _session:
|
||||||
|
_session.add(transcript)
|
||||||
|
await self._finalize(session=_session, caller_session=session, refresh=(transcript,))
|
||||||
|
return transcript
|
||||||
|
|
||||||
|
async def create_transcript_revision(
|
||||||
|
self,
|
||||||
|
transcript_revision: TranscriptRevision,
|
||||||
|
*,
|
||||||
|
session: AsyncSession | None = None,
|
||||||
|
) -> TranscriptRevision:
|
||||||
|
"""Create a new transcript revision in the database."""
|
||||||
|
async with self._session_scope(session) as _session:
|
||||||
|
_session.add(transcript_revision)
|
||||||
|
await self._finalize(session=_session, caller_session=session, refresh=(transcript_revision,))
|
||||||
|
return transcript_revision
|
||||||
|
|
||||||
|
async def read_transcript(self, transcript_id: UUID, *, session: AsyncSession | None = None) -> Transcript:
|
||||||
|
"""Read an existing transcript from the database."""
|
||||||
|
async with self._session_scope(session) as _session:
|
||||||
|
transcript = await _session.get(
|
||||||
|
Transcript,
|
||||||
|
transcript_id,
|
||||||
|
# Makes the full Job model object available in the return Transcript object
|
||||||
|
options=(selectinload(Transcript.job),), # pyright: ignore[reportArgumentType]
|
||||||
|
)
|
||||||
|
if transcript is None:
|
||||||
|
raise TranscriptionNotFoundError(
|
||||||
|
f"Transcript with id {transcript_id} not found",
|
||||||
|
category=ErrorCategory.NOT_FOUND,
|
||||||
|
suggestion="Verify the transcript id and retry.",
|
||||||
|
)
|
||||||
|
return transcript
|
||||||
|
|
||||||
|
async def read_transcript_revision(
|
||||||
|
self,
|
||||||
|
transcript_revision_id: UUID,
|
||||||
|
*,
|
||||||
|
session: AsyncSession | None = None,
|
||||||
|
) -> TranscriptRevision:
|
||||||
|
"""Read an existing transcript revision from the database."""
|
||||||
|
async with self._session_scope(session) as _session:
|
||||||
|
transcript_revision = await _session.get(
|
||||||
|
TranscriptRevision,
|
||||||
|
transcript_revision_id,
|
||||||
|
options=(selectinload(TranscriptRevision.job),), # pyright: ignore[reportArgumentType]
|
||||||
|
)
|
||||||
|
if transcript_revision is None:
|
||||||
|
raise TranscriptionNotFoundError(
|
||||||
|
f"Transcript revision with id {transcript_revision_id} not found",
|
||||||
|
category=ErrorCategory.NOT_FOUND,
|
||||||
|
suggestion="Verify the transcript revision id and retry.",
|
||||||
|
)
|
||||||
|
return transcript_revision
|
||||||
|
|
||||||
|
async def update_transcript(self, transcript: Transcript, *, session: AsyncSession | None = None) -> Transcript:
|
||||||
|
"""Update an existing transcript in the database."""
|
||||||
|
async with self._session_scope(session) as _session:
|
||||||
|
merged = await _session.merge(transcript)
|
||||||
|
await self._finalize(session=_session, caller_session=session, refresh=(merged,))
|
||||||
|
return merged
|
||||||
|
|
||||||
|
async def update_transcript_revision(
|
||||||
|
self,
|
||||||
|
transcript_revision: TranscriptRevision,
|
||||||
|
*,
|
||||||
|
session: AsyncSession | None = None,
|
||||||
|
) -> TranscriptRevision:
|
||||||
|
"""Update an existing transcript revision in the database."""
|
||||||
|
async with self._session_scope(session) as _session:
|
||||||
|
merged = await _session.merge(transcript_revision)
|
||||||
|
await self._finalize(session=_session, caller_session=session, refresh=(merged,))
|
||||||
|
return merged
|
||||||
|
|
||||||
|
async def delete_transcript(self, transcript: Transcript, *, session: AsyncSession | None = None) -> None:
|
||||||
|
"""Delete a transcript from the database."""
|
||||||
|
async with self._session_scope(session) as _session:
|
||||||
|
await _session.delete(transcript)
|
||||||
|
await self._finalize(session=_session, caller_session=session)
|
||||||
|
|
||||||
|
async def delete_transcript_revision(
|
||||||
|
self,
|
||||||
|
transcript_revision: TranscriptRevision,
|
||||||
|
*,
|
||||||
|
session: AsyncSession | None = None,
|
||||||
|
) -> None:
|
||||||
|
"""Delete a transcript revision from the database."""
|
||||||
|
async with self._session_scope(session) as _session:
|
||||||
|
await _session.delete(transcript_revision)
|
||||||
|
await self._finalize(session=_session, caller_session=session)
|
||||||
|
|
||||||
|
async def transcribe_document(
|
||||||
|
self,
|
||||||
|
image_path: str | Path,
|
||||||
|
job_id: UUID,
|
||||||
|
*,
|
||||||
|
prompt_name: str = DEFAULT_PROMPT_FILE,
|
||||||
|
session: AsyncSession | None = None,
|
||||||
|
):
|
||||||
|
"""Transcribe a local image using the configured prompt and provider."""
|
||||||
|
result = await transcribe_document_image(
|
||||||
|
image_path=image_path,
|
||||||
|
prompt_name=prompt_name,
|
||||||
|
settings=self.settings,
|
||||||
|
provider=self.provider,
|
||||||
|
)
|
||||||
|
await self.create_transcript(transcript=result.to_transcript(job_id=job_id), session=session)
|
||||||
|
|
||||||
|
async def upsert_transcript_by_job(
|
||||||
|
self,
|
||||||
|
*,
|
||||||
|
job_id: UUID,
|
||||||
|
text: str | None,
|
||||||
|
error_detail: str | None,
|
||||||
|
provider: str | None = None,
|
||||||
|
prompt_name: str = DEFAULT_PROMPT_FILE,
|
||||||
|
model: str | None = None,
|
||||||
|
session: AsyncSession | None = None,
|
||||||
|
) -> Transcript:
|
||||||
|
"""Create or update a transcript for a job id."""
|
||||||
|
async with self._session_scope(session) as _session:
|
||||||
|
transcript = (await _session.exec(select(Transcript).where(Transcript.job_id == job_id))).first()
|
||||||
|
if transcript is None:
|
||||||
|
transcript = Transcript(
|
||||||
|
job_id=job_id,
|
||||||
|
provider=provider or self.settings.provider.value,
|
||||||
|
prompt_name=prompt_name,
|
||||||
|
)
|
||||||
|
|
||||||
|
transcript.text = text
|
||||||
|
transcript.error_detail = error_detail
|
||||||
|
if provider is not None:
|
||||||
|
transcript.provider = provider
|
||||||
|
transcript.prompt_name = prompt_name
|
||||||
|
if model is not None:
|
||||||
|
transcript.model = model
|
||||||
|
|
||||||
|
_session.add(transcript)
|
||||||
|
await self._finalize(session=_session, caller_session=session, refresh=(transcript,))
|
||||||
|
return transcript
|
||||||
|
|
||||||
|
async def list_transcript_revisions_by_job(
|
||||||
|
self,
|
||||||
|
*,
|
||||||
|
job_id: UUID,
|
||||||
|
session: AsyncSession | None = None,
|
||||||
|
) -> list[TranscriptRevision]:
|
||||||
|
"""Return transcript revisions for a job ordered by version number."""
|
||||||
|
async with self._session_scope(session) as _session:
|
||||||
|
revisions = (
|
||||||
|
await _session.exec(
|
||||||
|
select(TranscriptRevision)
|
||||||
|
.where(TranscriptRevision.job_id == job_id)
|
||||||
|
.order_by(TranscriptRevision.version_number)
|
||||||
|
)
|
||||||
|
).all()
|
||||||
|
return list(revisions)
|
||||||
|
|
||||||
|
async def append_transcript_revision(
|
||||||
|
self,
|
||||||
|
*,
|
||||||
|
job_id: UUID,
|
||||||
|
text: str | None,
|
||||||
|
error_detail: str | None,
|
||||||
|
provider: str,
|
||||||
|
prompt_name: str,
|
||||||
|
model: str | None,
|
||||||
|
source: str,
|
||||||
|
session: AsyncSession | None = None,
|
||||||
|
) -> TranscriptRevision:
|
||||||
|
"""Append a new transcript revision and allocate the next version number."""
|
||||||
|
async with self._session_scope(session) as _session:
|
||||||
|
latest_version = (
|
||||||
|
await _session.exec(
|
||||||
|
select(TranscriptRevision.version_number)
|
||||||
|
.where(TranscriptRevision.job_id == job_id)
|
||||||
|
.order_by(TranscriptRevision.version_number.desc())
|
||||||
|
.limit(1)
|
||||||
|
)
|
||||||
|
).first()
|
||||||
|
next_version = 1 if latest_version is None else latest_version + 1
|
||||||
|
|
||||||
|
revision = TranscriptRevision(
|
||||||
|
job_id=job_id,
|
||||||
|
version_number=next_version,
|
||||||
|
provider=provider,
|
||||||
|
prompt_name=prompt_name,
|
||||||
|
model=model,
|
||||||
|
source=source,
|
||||||
|
text=text,
|
||||||
|
error_detail=error_detail,
|
||||||
|
)
|
||||||
|
_session.add(revision)
|
||||||
|
await self._finalize(session=_session, caller_session=session, refresh=(revision,))
|
||||||
|
return revision
|
||||||
|
|
||||||
|
|
||||||
|
async def transcribe_document_image(
|
||||||
|
image_path: str | Path,
|
||||||
|
*,
|
||||||
|
prompt_name: str = DEFAULT_PROMPT_FILE,
|
||||||
|
settings: Settings | None = None,
|
||||||
|
provider: TranscriptionProvider | None = None,
|
||||||
|
) -> TranscriptionResult:
|
||||||
|
"""Transcribe a local image using the configured prompt and provider."""
|
||||||
|
runtime_settings = settings or get_settings()
|
||||||
|
prompt_text = load_prompt_text(prompt_name=prompt_name, settings=runtime_settings)
|
||||||
|
image_bytes, mime_type = load_image_payload(image_path)
|
||||||
|
|
||||||
|
adapter = provider or get_transcription_provider(settings=runtime_settings)
|
||||||
|
logger.info("Starting transcription for image=%s mime_type=%s", image_path, mime_type)
|
||||||
|
|
||||||
|
with handle_transcription_errors():
|
||||||
|
result = await adapter.transcribe(
|
||||||
|
prompt_text=prompt_text,
|
||||||
|
image_bytes=image_bytes,
|
||||||
|
mime_type=mime_type,
|
||||||
|
)
|
||||||
|
logger.info("Transcription completed for image=%s provider=%s", image_path, result.provider)
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
def load_prompt_text(*, prompt_name: str = DEFAULT_PROMPT_FILE, settings: Settings | None = None) -> str:
|
def load_prompt_text(*, prompt_name: str = DEFAULT_PROMPT_FILE, settings: Settings | None = None) -> str:
|
||||||
"""Load and validate prompt text from PROMPT_DIR."""
|
"""Load and validate prompt text from PROMPT_DIR."""
|
||||||
runtime_settings = settings or get_settings()
|
runtime_settings = settings or get_settings()
|
||||||
@@ -87,27 +337,11 @@ def load_image_payload(image_path: str | Path) -> tuple[bytes, str]:
|
|||||||
return path.read_bytes(), mime_type
|
return path.read_bytes(), mime_type
|
||||||
|
|
||||||
|
|
||||||
def transcribe_document_image(
|
@contextmanager
|
||||||
image_path: str | Path,
|
def handle_transcription_errors():
|
||||||
*,
|
"""Context manager to handle transcription errors."""
|
||||||
prompt_name: str = DEFAULT_PROMPT_FILE,
|
|
||||||
settings: Settings | None = None,
|
|
||||||
provider: TranscriptionProvider | None = None,
|
|
||||||
) -> TranscriptionResult:
|
|
||||||
"""Transcribe a local image using the configured prompt and provider."""
|
|
||||||
runtime_settings = settings or get_settings()
|
|
||||||
prompt_text = load_prompt_text(prompt_name=prompt_name, settings=runtime_settings)
|
|
||||||
image_bytes, mime_type = load_image_payload(image_path)
|
|
||||||
|
|
||||||
adapter = provider or get_transcription_provider(settings=runtime_settings)
|
|
||||||
logger.info("Starting transcription for image=%s mime_type=%s", image_path, mime_type)
|
|
||||||
|
|
||||||
try:
|
try:
|
||||||
result = adapter.transcribe(
|
yield
|
||||||
prompt_text=prompt_text,
|
|
||||||
image_bytes=image_bytes,
|
|
||||||
mime_type=mime_type,
|
|
||||||
)
|
|
||||||
except ProviderAuthError as exc:
|
except ProviderAuthError as exc:
|
||||||
raise TranscriptionError(
|
raise TranscriptionError(
|
||||||
"Provider authentication failed",
|
"Provider authentication failed",
|
||||||
@@ -128,6 +362,3 @@ def transcribe_document_image(
|
|||||||
suggestion="Retry the transcription from jobs. If repeated, check provider availability.",
|
suggestion="Retry the transcription from jobs. If repeated, check provider availability.",
|
||||||
retriable=True,
|
retriable=True,
|
||||||
) from exc
|
) from exc
|
||||||
|
|
||||||
logger.info("Transcription completed for image=%s provider=%s", image_path, result.provider)
|
|
||||||
return result
|
|
||||||
|
|||||||
@@ -0,0 +1,317 @@
|
|||||||
|
import asyncio
|
||||||
|
import logging
|
||||||
|
|
||||||
|
from sqlmodel.ext.asyncio.session import AsyncSession
|
||||||
|
|
||||||
|
from ..config import Settings
|
||||||
|
from ..config import get_settings
|
||||||
|
from ..errors import AppError
|
||||||
|
from ..errors import classify_unexpected_error
|
||||||
|
from ..errors import format_error_detail
|
||||||
|
from ..models import Job
|
||||||
|
from ..models import JobStatus
|
||||||
|
from ..providers import TranscriptionResult
|
||||||
|
from . import ServiceBundle
|
||||||
|
from .transcription import DEFAULT_PROMPT_FILE
|
||||||
|
from .transcription import transcribe_document_image
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
async def advance_job(
|
||||||
|
job: Job,
|
||||||
|
services: ServiceBundle,
|
||||||
|
settings: Settings | None = None,
|
||||||
|
session: AsyncSession | None = None,
|
||||||
|
) -> Job | None:
|
||||||
|
"""Advance a single job by lifecycle status."""
|
||||||
|
settings = settings or get_settings()
|
||||||
|
match job.status:
|
||||||
|
case JobStatus.QUEUED:
|
||||||
|
return await process_queued_job(job=job, services=services, session=session)
|
||||||
|
case JobStatus.FAILED:
|
||||||
|
if job.retry_count < settings.worker_max_retries:
|
||||||
|
return await services.jobs.update_job_state(
|
||||||
|
job_id=job.id,
|
||||||
|
status=JobStatus.QUEUED,
|
||||||
|
retry_count_increment=1,
|
||||||
|
session=session,
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
logger.error(f"Job {job.id} has failed and reached max retries.")
|
||||||
|
return
|
||||||
|
case _:
|
||||||
|
return
|
||||||
|
|
||||||
|
|
||||||
|
async def process_queued_job(
|
||||||
|
*,
|
||||||
|
job: Job,
|
||||||
|
services: ServiceBundle,
|
||||||
|
session: AsyncSession | None = None,
|
||||||
|
) -> Job | None:
|
||||||
|
"""Process one complete transcription attempt for a queued job."""
|
||||||
|
if job.status != JobStatus.QUEUED:
|
||||||
|
logger.warning(f"Job {job.id} is not queued. Current status: {job.status}")
|
||||||
|
return
|
||||||
|
|
||||||
|
# Transaction A: claim job for processing.
|
||||||
|
if session is None:
|
||||||
|
job = await services.jobs.mark_job_status(job.id, JobStatus.PROCESSING)
|
||||||
|
else:
|
||||||
|
# If we're sharing the session, need to make sure setting the Job to PROCESSING is committed before we start
|
||||||
|
# the transcription, otherwise other workers may see the job as still QUEUED and try to process it.
|
||||||
|
job = await services.jobs.mark_job_status(job.id, JobStatus.PROCESSING, session=session)
|
||||||
|
await session.commit()
|
||||||
|
|
||||||
|
document = job.document
|
||||||
|
assert document is not None, (
|
||||||
|
f"Job {job.id} has no associated document or the document failed to be loaded by the job service."
|
||||||
|
)
|
||||||
|
|
||||||
|
try:
|
||||||
|
result = await transcribe_document_image(document.file_path)
|
||||||
|
job = await _finalize_transcribed(job=job, services=services, result=result, session=session)
|
||||||
|
logger.info(
|
||||||
|
"Job transcribed operation=worker.process_job job_id=%s document_id=%s provider=%s",
|
||||||
|
job.id,
|
||||||
|
document.id,
|
||||||
|
result.provider,
|
||||||
|
)
|
||||||
|
except Exception as exc: # noqa: BLE001
|
||||||
|
match exc:
|
||||||
|
case AppError() as error:
|
||||||
|
pass
|
||||||
|
case _:
|
||||||
|
error = classify_unexpected_error(exc, operation="worker.process_job")
|
||||||
|
|
||||||
|
job = await _finalize_failed(job=job, services=services, error=error, session=session)
|
||||||
|
logger.error(
|
||||||
|
"Job failed operation=worker.process_job job_id=%s document_id=%s error_id=%s category=%s",
|
||||||
|
job.id,
|
||||||
|
document.id,
|
||||||
|
error.error_id,
|
||||||
|
error.category.value,
|
||||||
|
)
|
||||||
|
return job
|
||||||
|
|
||||||
|
|
||||||
|
async def process_next_queued_job(
|
||||||
|
*,
|
||||||
|
services: ServiceBundle,
|
||||||
|
settings: Settings | None = None,
|
||||||
|
session: AsyncSession | None = None,
|
||||||
|
) -> bool:
|
||||||
|
"""Process the next queued job if one exists."""
|
||||||
|
job = await services.jobs.read_next_queued_job(session=session)
|
||||||
|
if job is None:
|
||||||
|
return False
|
||||||
|
|
||||||
|
await advance_job(job=job, services=services, settings=settings, session=session)
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
async def _finalize_transcribed(
|
||||||
|
*,
|
||||||
|
job: Job,
|
||||||
|
services: ServiceBundle,
|
||||||
|
result: TranscriptionResult,
|
||||||
|
session: AsyncSession | None = None,
|
||||||
|
) -> Job:
|
||||||
|
"""Transaction B: transcript + TRANSCRIBED in one commit."""
|
||||||
|
if session is None:
|
||||||
|
async with services.jobs._session_scope() as local_session:
|
||||||
|
prompt_name = result.prompt_name or DEFAULT_PROMPT_FILE
|
||||||
|
await services.transcriptions.upsert_transcript_by_job(
|
||||||
|
job_id=job.id,
|
||||||
|
text=result.text,
|
||||||
|
error_detail=None,
|
||||||
|
provider=result.provider,
|
||||||
|
prompt_name=prompt_name,
|
||||||
|
model=result.model,
|
||||||
|
session=local_session,
|
||||||
|
)
|
||||||
|
await services.transcriptions.append_transcript_revision(
|
||||||
|
job_id=job.id,
|
||||||
|
text=result.text,
|
||||||
|
error_detail=None,
|
||||||
|
provider=result.provider,
|
||||||
|
prompt_name=prompt_name,
|
||||||
|
model=result.model,
|
||||||
|
source="ai",
|
||||||
|
session=local_session,
|
||||||
|
)
|
||||||
|
updated_job = await services.jobs.mark_job_status(
|
||||||
|
job.id,
|
||||||
|
JobStatus.TRANSCRIBED,
|
||||||
|
session=local_session,
|
||||||
|
)
|
||||||
|
await local_session.commit()
|
||||||
|
return updated_job
|
||||||
|
|
||||||
|
prompt_name = result.prompt_name or DEFAULT_PROMPT_FILE
|
||||||
|
await services.transcriptions.upsert_transcript_by_job(
|
||||||
|
job_id=job.id,
|
||||||
|
text=result.text,
|
||||||
|
error_detail=None,
|
||||||
|
provider=result.provider,
|
||||||
|
prompt_name=prompt_name,
|
||||||
|
model=result.model,
|
||||||
|
session=session,
|
||||||
|
)
|
||||||
|
await services.transcriptions.append_transcript_revision(
|
||||||
|
job_id=job.id,
|
||||||
|
text=result.text,
|
||||||
|
error_detail=None,
|
||||||
|
provider=result.provider,
|
||||||
|
prompt_name=prompt_name,
|
||||||
|
model=result.model,
|
||||||
|
source="ai",
|
||||||
|
session=session,
|
||||||
|
)
|
||||||
|
updated_job = await services.jobs.mark_job_status(
|
||||||
|
job.id,
|
||||||
|
JobStatus.TRANSCRIBED,
|
||||||
|
session=session,
|
||||||
|
)
|
||||||
|
await session.commit()
|
||||||
|
return updated_job
|
||||||
|
|
||||||
|
|
||||||
|
async def _finalize_retry(
|
||||||
|
*,
|
||||||
|
job: Job,
|
||||||
|
services: ServiceBundle,
|
||||||
|
error: AppError,
|
||||||
|
settings: Settings,
|
||||||
|
session: AsyncSession | None = None,
|
||||||
|
) -> Job:
|
||||||
|
"""Transaction C: transcript error + QUEUED + retry increment in one commit."""
|
||||||
|
if session is None:
|
||||||
|
async with services.jobs._session_scope() as local_session:
|
||||||
|
provider_name = services.transcriptions.settings.provider.value
|
||||||
|
await services.transcriptions.upsert_transcript_by_job(
|
||||||
|
job_id=job.id,
|
||||||
|
text=None,
|
||||||
|
error_detail=format_error_detail(error),
|
||||||
|
provider=provider_name,
|
||||||
|
prompt_name=DEFAULT_PROMPT_FILE,
|
||||||
|
model=services.transcriptions.settings.provider_model,
|
||||||
|
session=local_session,
|
||||||
|
)
|
||||||
|
await services.transcriptions.append_transcript_revision(
|
||||||
|
job_id=job.id,
|
||||||
|
text=None,
|
||||||
|
error_detail=format_error_detail(error),
|
||||||
|
provider=provider_name,
|
||||||
|
prompt_name=DEFAULT_PROMPT_FILE,
|
||||||
|
model=services.transcriptions.settings.provider_model,
|
||||||
|
source="ai",
|
||||||
|
session=local_session,
|
||||||
|
)
|
||||||
|
updated_job = await services.jobs.update_job_state(
|
||||||
|
job_id=job.id,
|
||||||
|
status=JobStatus.QUEUED,
|
||||||
|
retry_count_increment=1,
|
||||||
|
session=local_session,
|
||||||
|
)
|
||||||
|
await local_session.commit()
|
||||||
|
else:
|
||||||
|
provider_name = services.transcriptions.settings.provider.value
|
||||||
|
await services.transcriptions.upsert_transcript_by_job(
|
||||||
|
job_id=job.id,
|
||||||
|
text=None,
|
||||||
|
error_detail=format_error_detail(error),
|
||||||
|
provider=provider_name,
|
||||||
|
prompt_name=DEFAULT_PROMPT_FILE,
|
||||||
|
model=services.transcriptions.settings.provider_model,
|
||||||
|
session=session,
|
||||||
|
)
|
||||||
|
await services.transcriptions.append_transcript_revision(
|
||||||
|
job_id=job.id,
|
||||||
|
text=None,
|
||||||
|
error_detail=format_error_detail(error),
|
||||||
|
provider=provider_name,
|
||||||
|
prompt_name=DEFAULT_PROMPT_FILE,
|
||||||
|
model=services.transcriptions.settings.provider_model,
|
||||||
|
source="ai",
|
||||||
|
session=session,
|
||||||
|
)
|
||||||
|
updated_job = await services.jobs.update_job_state(
|
||||||
|
job_id=job.id,
|
||||||
|
status=JobStatus.QUEUED,
|
||||||
|
retry_count_increment=1,
|
||||||
|
session=session,
|
||||||
|
)
|
||||||
|
await session.commit()
|
||||||
|
|
||||||
|
if settings.worker_retry_backoff_seconds > 0:
|
||||||
|
await asyncio.sleep(settings.worker_retry_backoff_seconds)
|
||||||
|
return updated_job
|
||||||
|
|
||||||
|
|
||||||
|
async def _finalize_failed(
|
||||||
|
*,
|
||||||
|
job: Job,
|
||||||
|
services: ServiceBundle,
|
||||||
|
error: AppError,
|
||||||
|
session: AsyncSession | None = None,
|
||||||
|
) -> Job:
|
||||||
|
"""Transaction B: transcript error + FAILED in one commit."""
|
||||||
|
if session is None:
|
||||||
|
async with services.jobs._session_scope() as local_session:
|
||||||
|
provider_name = services.transcriptions.settings.provider.value
|
||||||
|
await services.transcriptions.upsert_transcript_by_job(
|
||||||
|
job_id=job.id,
|
||||||
|
text=None,
|
||||||
|
error_detail=format_error_detail(error),
|
||||||
|
provider=provider_name,
|
||||||
|
prompt_name=DEFAULT_PROMPT_FILE,
|
||||||
|
model=services.transcriptions.settings.provider_model,
|
||||||
|
session=local_session,
|
||||||
|
)
|
||||||
|
await services.transcriptions.append_transcript_revision(
|
||||||
|
job_id=job.id,
|
||||||
|
text=None,
|
||||||
|
error_detail=format_error_detail(error),
|
||||||
|
provider=provider_name,
|
||||||
|
prompt_name=DEFAULT_PROMPT_FILE,
|
||||||
|
model=services.transcriptions.settings.provider_model,
|
||||||
|
source="ai",
|
||||||
|
session=local_session,
|
||||||
|
)
|
||||||
|
updated_job = await services.jobs.mark_job_status(
|
||||||
|
job.id,
|
||||||
|
JobStatus.FAILED,
|
||||||
|
session=local_session,
|
||||||
|
)
|
||||||
|
await local_session.commit()
|
||||||
|
return updated_job
|
||||||
|
|
||||||
|
provider_name = services.transcriptions.settings.provider.value
|
||||||
|
await services.transcriptions.upsert_transcript_by_job(
|
||||||
|
job_id=job.id,
|
||||||
|
text=None,
|
||||||
|
error_detail=format_error_detail(error),
|
||||||
|
provider=provider_name,
|
||||||
|
prompt_name=DEFAULT_PROMPT_FILE,
|
||||||
|
model=services.transcriptions.settings.provider_model,
|
||||||
|
session=session,
|
||||||
|
)
|
||||||
|
await services.transcriptions.append_transcript_revision(
|
||||||
|
job_id=job.id,
|
||||||
|
text=None,
|
||||||
|
error_detail=format_error_detail(error),
|
||||||
|
provider=provider_name,
|
||||||
|
prompt_name=DEFAULT_PROMPT_FILE,
|
||||||
|
model=services.transcriptions.settings.provider_model,
|
||||||
|
source="ai",
|
||||||
|
session=session,
|
||||||
|
)
|
||||||
|
updated_job = await services.jobs.mark_job_status(
|
||||||
|
job.id,
|
||||||
|
JobStatus.FAILED,
|
||||||
|
session=session,
|
||||||
|
)
|
||||||
|
await session.commit()
|
||||||
|
return updated_job
|
||||||
@@ -3,8 +3,8 @@
|
|||||||
from fastapi import FastAPI
|
from fastapi import FastAPI
|
||||||
from nicegui import ui
|
from nicegui import ui
|
||||||
|
|
||||||
from transcription.ui.jobs_page import register_page as register_jobs_page
|
from transcription.ui.pages.jobs_page import register_page as register_jobs_page
|
||||||
from transcription.ui.upload_page import register_page as register_upload_page
|
from transcription.ui.pages.upload_page import register_page as register_upload_page
|
||||||
|
|
||||||
|
|
||||||
def register_pages(app: FastAPI) -> None:
|
def register_pages(app: FastAPI) -> None:
|
||||||
@@ -12,5 +12,3 @@ def register_pages(app: FastAPI) -> None:
|
|||||||
register_upload_page()
|
register_upload_page()
|
||||||
register_jobs_page()
|
register_jobs_page()
|
||||||
ui.run_with(app, mount_path="/ui", show_welcome_message=False)
|
ui.run_with(app, mount_path="/ui", show_welcome_message=False)
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -4,7 +4,9 @@ from __future__ import annotations
|
|||||||
|
|
||||||
from nicegui import ui
|
from nicegui import ui
|
||||||
|
|
||||||
from transcription.errors import AppError, ErrorCategory, classify_unexpected_error
|
from transcription.errors import AppError
|
||||||
|
from transcription.errors import ErrorCategory
|
||||||
|
from transcription.errors import classify_unexpected_error
|
||||||
|
|
||||||
|
|
||||||
def to_app_error(exc: Exception, *, operation: str) -> AppError:
|
def to_app_error(exc: Exception, *, operation: str) -> AppError:
|
||||||
@@ -37,4 +39,4 @@ def summarize_error(exc: Exception, *, operation: str) -> str:
|
|||||||
error = to_app_error(exc, operation=operation)
|
error = to_app_error(exc, operation=operation)
|
||||||
if error.category == ErrorCategory.INTERNAL_UNEXPECTED:
|
if error.category == ErrorCategory.INTERNAL_UNEXPECTED:
|
||||||
return f"Unexpected error (ref: {error.error_id})"
|
return f"Unexpected error (ref: {error.error_id})"
|
||||||
return f"{error.message} (ref: {error.error_id})"
|
return f"{error.message} (ref: {error.error_id})"
|
||||||
@@ -0,0 +1,304 @@
|
|||||||
|
"""Reusable job detail rendering helpers."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import base64
|
||||||
|
import mimetypes
|
||||||
|
from collections.abc import Awaitable
|
||||||
|
from collections.abc import Callable
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from nicegui import ui
|
||||||
|
|
||||||
|
from transcription.models import Document
|
||||||
|
from transcription.models import Job
|
||||||
|
from transcription.models import Transcript
|
||||||
|
from transcription.models import TranscriptRevision
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class RevisionDisplayRow:
|
||||||
|
id: str
|
||||||
|
created: str
|
||||||
|
version: str
|
||||||
|
text: str
|
||||||
|
error_detail: str | None
|
||||||
|
|
||||||
|
|
||||||
|
def _extract_row_id(args: object) -> str | None:
|
||||||
|
if isinstance(args, dict):
|
||||||
|
if isinstance(args.get("row"), dict):
|
||||||
|
row_id = args["row"].get("id")
|
||||||
|
return str(row_id) if row_id is not None else None
|
||||||
|
row_id = args.get("id")
|
||||||
|
return str(row_id) if row_id is not None else None
|
||||||
|
|
||||||
|
if isinstance(args, list):
|
||||||
|
for value in args:
|
||||||
|
if isinstance(value, dict):
|
||||||
|
row_id = value.get("id")
|
||||||
|
if row_id is not None:
|
||||||
|
return str(row_id)
|
||||||
|
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _status_chip_classes(status: str) -> str:
|
||||||
|
if status == "queued":
|
||||||
|
return "bg-blue-1 text-blue-10"
|
||||||
|
if status == "processing":
|
||||||
|
return "bg-amber-1 text-amber-10"
|
||||||
|
if status == "transcribed":
|
||||||
|
return "bg-green-1 text-green-10"
|
||||||
|
if status == "failed":
|
||||||
|
return "bg-red-1 text-red-10"
|
||||||
|
return "bg-grey-2 text-grey-9"
|
||||||
|
|
||||||
|
|
||||||
|
def _metadata_row(label: str, value: str) -> None:
|
||||||
|
with ui.row().classes("w-full items-start justify-between no-wrap q-gutter-x-md"):
|
||||||
|
ui.label(label).classes("text-caption text-grey-7 text-uppercase")
|
||||||
|
ui.label(value).classes("text-body2 text-right")
|
||||||
|
|
||||||
|
|
||||||
|
def _render_document_section(document: Document) -> None:
|
||||||
|
with ui.card().classes("w-full bg-grey-1 q-pa-md"):
|
||||||
|
ui.label("Document").classes("text-subtitle1 text-weight-medium")
|
||||||
|
ui.separator().classes("q-my-sm")
|
||||||
|
with ui.column().classes("w-full q-gutter-y-xs"):
|
||||||
|
_metadata_row("Filename", document.filename)
|
||||||
|
_metadata_row("File path", document.file_path)
|
||||||
|
|
||||||
|
|
||||||
|
def _document_data_url(document: Document) -> tuple[str | None, str | None]:
|
||||||
|
path = Path(document.file_path)
|
||||||
|
if not path.exists() or not path.is_file():
|
||||||
|
return None, "Document preview unavailable: file not found"
|
||||||
|
|
||||||
|
suffix = path.suffix.lower()
|
||||||
|
mime_type, _ = mimetypes.guess_type(path.name)
|
||||||
|
if suffix in {".tif", ".tiff"}:
|
||||||
|
mime_type = "image/tiff"
|
||||||
|
if mime_type is None:
|
||||||
|
return None, "Document preview unavailable: unsupported MIME type"
|
||||||
|
|
||||||
|
encoded = base64.b64encode(path.read_bytes()).decode("ascii")
|
||||||
|
return f"data:{mime_type};base64,{encoded}", None
|
||||||
|
|
||||||
|
|
||||||
|
def _render_document_preview(document: Document) -> None:
|
||||||
|
source, error = _document_data_url(document)
|
||||||
|
if error is not None or source is None:
|
||||||
|
ui.label(error or "Document preview unavailable").classes("text-caption text-grey-7")
|
||||||
|
return
|
||||||
|
|
||||||
|
suffix = Path(document.file_path).suffix.lower()
|
||||||
|
if suffix == ".pdf":
|
||||||
|
ui.html(
|
||||||
|
(
|
||||||
|
'<iframe title="Document preview" '
|
||||||
|
f'src="{source}" '
|
||||||
|
'style="width:100%;height:520px;border:1px solid #ddd;border-radius:8px;"></iframe>'
|
||||||
|
)
|
||||||
|
)
|
||||||
|
ui.label("Zoom controls are currently available for image files.").classes("text-caption text-grey-7 q-mt-sm")
|
||||||
|
return
|
||||||
|
|
||||||
|
zoom_percent = {"value": 100}
|
||||||
|
|
||||||
|
with ui.element("div").style(
|
||||||
|
"width:100%;height:520px;overflow:auto;border:1px solid #ddd;border-radius:8px;padding:8px;background:#fafafa;"
|
||||||
|
):
|
||||||
|
image = ui.image(source).classes("rounded-borders").style("width:100%;max-width:none;")
|
||||||
|
|
||||||
|
zoom_label = ui.label("Zoom: 100%").classes("text-caption text-grey-7 q-mt-sm")
|
||||||
|
|
||||||
|
def _apply_zoom() -> None:
|
||||||
|
image.style(f"width:{zoom_percent['value']}%;max-width:none;")
|
||||||
|
image.update()
|
||||||
|
zoom_label.text = f"Zoom: {zoom_percent['value']}%"
|
||||||
|
zoom_label.update()
|
||||||
|
|
||||||
|
def _zoom_in() -> None:
|
||||||
|
zoom_percent["value"] = min(300, zoom_percent["value"] + 25)
|
||||||
|
_apply_zoom()
|
||||||
|
|
||||||
|
def _zoom_out() -> None:
|
||||||
|
zoom_percent["value"] = max(50, zoom_percent["value"] - 25)
|
||||||
|
_apply_zoom()
|
||||||
|
|
||||||
|
def _zoom_reset() -> None:
|
||||||
|
zoom_percent["value"] = 100
|
||||||
|
_apply_zoom()
|
||||||
|
|
||||||
|
with ui.row().classes("q-gutter-sm q-mt-xs"):
|
||||||
|
ui.button("-", on_click=_zoom_out)
|
||||||
|
ui.button("+", on_click=_zoom_in)
|
||||||
|
ui.button("Reset", on_click=_zoom_reset)
|
||||||
|
|
||||||
|
|
||||||
|
def _build_display_rows(transcript: Transcript, revisions: list[TranscriptRevision]) -> list[RevisionDisplayRow]:
|
||||||
|
ordered = sorted(revisions, key=lambda revision: revision.version_number)
|
||||||
|
rows: list[RevisionDisplayRow] = []
|
||||||
|
|
||||||
|
if ordered:
|
||||||
|
first = ordered[0]
|
||||||
|
rows.append(
|
||||||
|
RevisionDisplayRow(
|
||||||
|
id="original",
|
||||||
|
created=first.created_at.isoformat(),
|
||||||
|
version="original",
|
||||||
|
text=first.text or "",
|
||||||
|
error_detail=first.error_detail,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
for revision in ordered[1:]:
|
||||||
|
rows.append(
|
||||||
|
RevisionDisplayRow(
|
||||||
|
id=str(revision.version_number),
|
||||||
|
created=revision.created_at.isoformat(),
|
||||||
|
version=str(revision.version_number),
|
||||||
|
text=revision.text or "",
|
||||||
|
error_detail=revision.error_detail,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
rows.append(
|
||||||
|
RevisionDisplayRow(
|
||||||
|
id="original",
|
||||||
|
created=transcript.created_at.isoformat(),
|
||||||
|
version="original",
|
||||||
|
text=transcript.text or "",
|
||||||
|
error_detail=transcript.error_detail,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
return rows
|
||||||
|
|
||||||
|
|
||||||
|
def _render_transcript_versioned_section(
|
||||||
|
*,
|
||||||
|
document: Document | None,
|
||||||
|
transcript: Transcript | None,
|
||||||
|
revisions: list[TranscriptRevision],
|
||||||
|
on_update: Callable[[str], Awaitable[None]] | None,
|
||||||
|
) -> None:
|
||||||
|
with ui.card().classes("w-full q-pa-md"):
|
||||||
|
ui.label("Transcript").classes("text-subtitle1 text-weight-medium")
|
||||||
|
ui.separator().classes("q-my-sm")
|
||||||
|
|
||||||
|
if transcript is None:
|
||||||
|
ui.label("Transcript not available yet.").classes("text-body2 text-grey-8")
|
||||||
|
return
|
||||||
|
|
||||||
|
with ui.column().classes("w-full q-gutter-y-xs"):
|
||||||
|
model_name = transcript.model
|
||||||
|
if model_name is None and revisions:
|
||||||
|
model_name = revisions[0].model
|
||||||
|
_metadata_row("Provider", transcript.provider)
|
||||||
|
_metadata_row("Model", model_name or "unknown")
|
||||||
|
_metadata_row("Prompt", transcript.prompt_name)
|
||||||
|
|
||||||
|
display_rows = _build_display_rows(transcript, revisions)
|
||||||
|
rows_by_id = {row.id: row for row in display_rows}
|
||||||
|
|
||||||
|
ui.separator().classes("q-my-sm")
|
||||||
|
ui.label("Versions").classes("text-subtitle2 text-weight-medium")
|
||||||
|
table = ui.table(
|
||||||
|
columns=[
|
||||||
|
{"name": "created", "label": "Created", "field": "created", "align": "left"},
|
||||||
|
{"name": "version", "label": "Version", "field": "version", "align": "left"},
|
||||||
|
],
|
||||||
|
rows=[
|
||||||
|
{
|
||||||
|
"id": row.id,
|
||||||
|
"created": row.created,
|
||||||
|
"version": row.version,
|
||||||
|
}
|
||||||
|
for row in display_rows
|
||||||
|
],
|
||||||
|
row_key="id",
|
||||||
|
).classes("w-full")
|
||||||
|
|
||||||
|
default_selected = display_rows[-1].id
|
||||||
|
selected_label = ui.label(f"Selected version: {rows_by_id[default_selected].version}").classes(
|
||||||
|
"text-caption text-grey-7"
|
||||||
|
)
|
||||||
|
|
||||||
|
ui.separator().classes("q-my-sm")
|
||||||
|
with ui.row().classes("w-full no-wrap items-start q-gutter-md"):
|
||||||
|
if document is not None:
|
||||||
|
with ui.column().classes("w-1/2"):
|
||||||
|
ui.label("Document Preview").classes("text-subtitle2 text-weight-medium")
|
||||||
|
_render_document_preview(document)
|
||||||
|
|
||||||
|
with ui.column().classes("w-1/2"):
|
||||||
|
editor = (
|
||||||
|
ui.textarea(label="Transcript text", value=rows_by_id[default_selected].text)
|
||||||
|
.props("autogrow outlined")
|
||||||
|
.classes("w-full")
|
||||||
|
)
|
||||||
|
error_label = ui.label("").classes("text-body2 text-red-10")
|
||||||
|
|
||||||
|
def _set_selected(version_id: str) -> None:
|
||||||
|
selected = rows_by_id.get(version_id)
|
||||||
|
if selected is None:
|
||||||
|
return
|
||||||
|
selected_label.text = f"Selected version: {selected.version}"
|
||||||
|
editor.value = selected.text
|
||||||
|
editor.update()
|
||||||
|
error_label.text = selected.error_detail or ""
|
||||||
|
error_label.update()
|
||||||
|
|
||||||
|
def _on_row_click(event) -> None: # noqa: ANN001
|
||||||
|
row_id = _extract_row_id(event.args)
|
||||||
|
if row_id is None:
|
||||||
|
return
|
||||||
|
_set_selected(row_id)
|
||||||
|
|
||||||
|
table.on("rowClick", _on_row_click)
|
||||||
|
_set_selected(default_selected)
|
||||||
|
|
||||||
|
if on_update is not None:
|
||||||
|
ui.button("Update", on_click=lambda: on_update(editor.value or ""))
|
||||||
|
|
||||||
|
|
||||||
|
def render_job_detail(
|
||||||
|
*,
|
||||||
|
job: Job,
|
||||||
|
document: Document | None,
|
||||||
|
transcript: Transcript | None,
|
||||||
|
revisions: list[TranscriptRevision],
|
||||||
|
on_update: Callable[[str], Awaitable[None]] | None = None,
|
||||||
|
) -> None:
|
||||||
|
"""Render all sections for the job detail page."""
|
||||||
|
status_text = job.status.value
|
||||||
|
with ui.column().classes("w-full max-w-4xl q-gutter-md"):
|
||||||
|
with ui.card().classes("w-full q-pa-lg"):
|
||||||
|
with ui.row().classes("w-full items-center justify-between q-gutter-md"):
|
||||||
|
with ui.column().classes("q-gutter-none"):
|
||||||
|
ui.label("Job overview").classes("text-h6 text-weight-bold")
|
||||||
|
ui.label(str(job.id)).classes("text-caption text-grey-7")
|
||||||
|
status_chip_classes = (
|
||||||
|
"q-px-sm q-py-xs rounded-borders "
|
||||||
|
"text-weight-medium text-capitalize "
|
||||||
|
f"{_status_chip_classes(status_text)}"
|
||||||
|
)
|
||||||
|
ui.label(status_text).classes(status_chip_classes)
|
||||||
|
|
||||||
|
ui.separator().classes("q-my-md")
|
||||||
|
with ui.column().classes("w-full q-gutter-y-xs"):
|
||||||
|
_metadata_row("Created", job.created_at.isoformat())
|
||||||
|
_metadata_row("Updated", job.updated_at.isoformat())
|
||||||
|
_metadata_row("Retries", str(job.retry_count))
|
||||||
|
|
||||||
|
if document is not None:
|
||||||
|
_render_document_section(document)
|
||||||
|
|
||||||
|
_render_transcript_versioned_section(
|
||||||
|
document=document,
|
||||||
|
transcript=transcript,
|
||||||
|
revisions=revisions,
|
||||||
|
on_update=on_update,
|
||||||
|
)
|
||||||
@@ -0,0 +1,4 @@
|
|||||||
|
from .jobs import JobTableRow
|
||||||
|
from .jobs import render_jobs_table
|
||||||
|
|
||||||
|
__all__ = ["JobTableRow", "render_jobs_table"]
|
||||||
@@ -0,0 +1,68 @@
|
|||||||
|
"""Common logic for generating table widgets."""
|
||||||
|
|
||||||
|
from collections.abc import Callable
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from nicegui import events
|
||||||
|
from nicegui import ui
|
||||||
|
|
||||||
|
|
||||||
|
def _extract_row_id(args: Any) -> str | None:
|
||||||
|
if isinstance(args, dict):
|
||||||
|
if isinstance(args.get("row"), dict):
|
||||||
|
row_id = args["row"].get("id")
|
||||||
|
return str(row_id) if row_id is not None else None
|
||||||
|
row_id = args.get("id")
|
||||||
|
return str(row_id) if row_id is not None else None
|
||||||
|
|
||||||
|
if isinstance(args, list):
|
||||||
|
for value in args:
|
||||||
|
if isinstance(value, dict):
|
||||||
|
row_id = value.get("id")
|
||||||
|
if row_id is not None:
|
||||||
|
return str(row_id)
|
||||||
|
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _bind_row_click_handler(
|
||||||
|
table: Any,
|
||||||
|
*,
|
||||||
|
on_row_click_id: Callable[[str], None],
|
||||||
|
) -> None:
|
||||||
|
def handle_row_click(event: events.GenericEventArguments) -> None:
|
||||||
|
row_id = _extract_row_id(event.args)
|
||||||
|
if row_id is None:
|
||||||
|
return
|
||||||
|
on_row_click_id(row_id)
|
||||||
|
|
||||||
|
table.on("rowClick", handle_row_click)
|
||||||
|
|
||||||
|
|
||||||
|
def build_table(
|
||||||
|
rows: list[dict[str, Any]],
|
||||||
|
columns: list[dict[str, Any]],
|
||||||
|
*,
|
||||||
|
default_sort_by: str | None = None,
|
||||||
|
default_descending: bool = False,
|
||||||
|
classes: str = "app-table",
|
||||||
|
on_row_click_id: Callable[[str], None] | None = None,
|
||||||
|
) -> Any:
|
||||||
|
pagination: dict[str, Any] = {"rowsPerPage": 25}
|
||||||
|
if default_sort_by is not None:
|
||||||
|
pagination["sortBy"] = default_sort_by
|
||||||
|
pagination["descending"] = default_descending
|
||||||
|
|
||||||
|
table = (
|
||||||
|
ui.table(
|
||||||
|
rows=rows,
|
||||||
|
columns=columns,
|
||||||
|
row_key="id",
|
||||||
|
pagination=pagination,
|
||||||
|
)
|
||||||
|
.classes(classes)
|
||||||
|
.props('table-style="table-layout: fixed; width: 100%;"')
|
||||||
|
)
|
||||||
|
if on_row_click_id is not None:
|
||||||
|
_bind_row_click_handler(table, on_row_click_id=on_row_click_id)
|
||||||
|
return table
|
||||||
@@ -0,0 +1,75 @@
|
|||||||
|
"""Jobs table rendering helpers."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from collections.abc import Sequence
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from datetime import UTC
|
||||||
|
from datetime import datetime
|
||||||
|
from typing import Any
|
||||||
|
from uuid import UUID
|
||||||
|
|
||||||
|
from nicegui import ui
|
||||||
|
|
||||||
|
from .common import build_table
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class JobTableRow:
|
||||||
|
"""Read model consumed by the jobs table component."""
|
||||||
|
|
||||||
|
id: UUID
|
||||||
|
status: str
|
||||||
|
filename: str
|
||||||
|
retry_count: int
|
||||||
|
created_at: str
|
||||||
|
updated_at: str
|
||||||
|
|
||||||
|
|
||||||
|
def _format_timestamp(value: str) -> str:
|
||||||
|
"""Return a friendly UTC timestamp for table display."""
|
||||||
|
try:
|
||||||
|
parsed = datetime.fromisoformat(value)
|
||||||
|
except ValueError:
|
||||||
|
return value
|
||||||
|
parsed = parsed.replace(tzinfo=UTC) if parsed.tzinfo is None else parsed.astimezone(UTC)
|
||||||
|
return parsed.astimezone().strftime("%b %d, %I:%M %p")
|
||||||
|
|
||||||
|
|
||||||
|
def _serialize_rows(rows: Sequence[JobTableRow]) -> list[dict[str, Any]]:
|
||||||
|
return [
|
||||||
|
{
|
||||||
|
"id": str(row.id),
|
||||||
|
"status": row.status,
|
||||||
|
"filename": row.filename,
|
||||||
|
"retry_count": row.retry_count,
|
||||||
|
"created_at": _format_timestamp(row.created_at),
|
||||||
|
"updated_at": _format_timestamp(row.updated_at),
|
||||||
|
"created_sort": row.created_at,
|
||||||
|
"updated_sort": row.updated_at,
|
||||||
|
}
|
||||||
|
for row in rows
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def render_jobs_table(rows: Sequence[JobTableRow]) -> None:
|
||||||
|
"""Render jobs table and open a detail page when clicking a row."""
|
||||||
|
if not rows:
|
||||||
|
ui.label("No jobs yet.")
|
||||||
|
return
|
||||||
|
|
||||||
|
build_table(
|
||||||
|
rows=_serialize_rows(rows),
|
||||||
|
columns=[
|
||||||
|
{"name": "id", "label": "Job ID", "field": "id", "sortable": True},
|
||||||
|
{"name": "status", "label": "Status", "field": "status", "sortable": True},
|
||||||
|
{"name": "filename", "label": "Filename", "field": "filename", "sortable": True},
|
||||||
|
{"name": "retry_count", "label": "Retries", "field": "retry_count", "sortable": True},
|
||||||
|
{"name": "created_at", "label": "Created", "field": "created_at", "sortable": True},
|
||||||
|
{"name": "updated_at", "label": "Updated", "field": "updated_at", "sortable": True},
|
||||||
|
],
|
||||||
|
default_sort_by="created_sort",
|
||||||
|
default_descending=True,
|
||||||
|
classes="app-table w-full",
|
||||||
|
on_row_click_id=lambda job_id: ui.navigate.to(f"/jobs/{job_id}"),
|
||||||
|
)
|
||||||
@@ -0,0 +1,66 @@
|
|||||||
|
"""Reusable upload widget for document submission."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from collections.abc import Awaitable
|
||||||
|
from collections.abc import Callable
|
||||||
|
|
||||||
|
from nicegui import ui
|
||||||
|
from nicegui.binding import bindable_dataclass
|
||||||
|
from nicegui.events import UploadEventArguments
|
||||||
|
|
||||||
|
from transcription.errors import AppError
|
||||||
|
from transcription.services.documents import UploadJobResult
|
||||||
|
from transcription.ui.components.error_presenter import show_error
|
||||||
|
from transcription.ui.components.error_presenter import summarize_error
|
||||||
|
from transcription.worker import WorkerNotifier
|
||||||
|
|
||||||
|
type UploadSubmitter = Callable[[str, bytes], Awaitable[UploadJobResult]]
|
||||||
|
|
||||||
|
|
||||||
|
@bindable_dataclass
|
||||||
|
class UploadWidgetState:
|
||||||
|
"""Simple state container for upload feedback."""
|
||||||
|
|
||||||
|
loading: bool = False
|
||||||
|
message: str = ""
|
||||||
|
|
||||||
|
|
||||||
|
def render_upload_widget(*, submitter: UploadSubmitter, notifier: WorkerNotifier | None = None) -> None:
|
||||||
|
"""Render upload controls and common status/error handling."""
|
||||||
|
state = UploadWidgetState()
|
||||||
|
status_label = ui.label("Upload a document to start transcription.")
|
||||||
|
status_label.bind_text(state, "message")
|
||||||
|
|
||||||
|
async def on_upload(event: UploadEventArguments) -> None:
|
||||||
|
if state.loading:
|
||||||
|
ui.notify("Upload already in progress. Please wait.", type="warning")
|
||||||
|
return
|
||||||
|
|
||||||
|
state.loading = True
|
||||||
|
status_label.text = "Uploading..."
|
||||||
|
try:
|
||||||
|
payload = await event.file.read()
|
||||||
|
result = await submitter(event.file.name, payload)
|
||||||
|
job_id = result.job_id
|
||||||
|
state.message = f"Created job {job_id}" if job_id is not None else "Upload complete"
|
||||||
|
status_label.text = state.message
|
||||||
|
if notifier is not None:
|
||||||
|
notifier.notify()
|
||||||
|
ui.notify(state.message, type="positive")
|
||||||
|
except AppError as exc:
|
||||||
|
state.message = summarize_error(exc, operation="upload.submit")
|
||||||
|
status_label.text = f"Upload failed: {state.message}"
|
||||||
|
show_error(exc, title="Upload failed", operation="upload.submit")
|
||||||
|
except Exception as exc: # noqa: BLE001
|
||||||
|
state.message = summarize_error(exc, operation="upload.submit")
|
||||||
|
status_label.text = f"Upload failed: {state.message}"
|
||||||
|
show_error(exc, title="Upload failed", operation="upload.submit")
|
||||||
|
finally:
|
||||||
|
state.loading = False
|
||||||
|
|
||||||
|
ui.upload(
|
||||||
|
on_upload=on_upload,
|
||||||
|
auto_upload=True,
|
||||||
|
label="Select document file",
|
||||||
|
).props('accept=".jpg,.jpeg,.png,.tif,.tiff,.pdf"')
|
||||||
@@ -1,137 +0,0 @@
|
|||||||
"""Jobs list and detail page registration."""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
from dataclasses import dataclass
|
|
||||||
from uuid import UUID
|
|
||||||
|
|
||||||
from nicegui import ui
|
|
||||||
from sqlmodel import select
|
|
||||||
|
|
||||||
from transcription.db import get_session
|
|
||||||
from transcription.models import Document, Job, Transcript
|
|
||||||
from transcription.ui.error_presenter import show_error, summarize_error
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class JobView:
|
|
||||||
"""Read model for rendering job rows in the UI."""
|
|
||||||
|
|
||||||
id: UUID
|
|
||||||
status: str
|
|
||||||
created_at: str
|
|
||||||
updated_at: str
|
|
||||||
|
|
||||||
|
|
||||||
def fetch_jobs() -> list[JobView]:
|
|
||||||
"""Return jobs for display in most-recent-first order."""
|
|
||||||
with get_session() as session:
|
|
||||||
jobs = session.exec(select(Job).order_by(Job.created_at.desc())).all()
|
|
||||||
return [
|
|
||||||
JobView(
|
|
||||||
id=job.id,
|
|
||||||
status=job.status.value,
|
|
||||||
created_at=job.created_at.isoformat(),
|
|
||||||
updated_at=job.updated_at.isoformat(),
|
|
||||||
)
|
|
||||||
for job in jobs
|
|
||||||
]
|
|
||||||
|
|
||||||
|
|
||||||
def fetch_job_detail(job_id: UUID) -> tuple[Job | None, Document | None, Transcript | None]:
|
|
||||||
"""Return job, document, and transcript for detail view."""
|
|
||||||
with get_session() as session:
|
|
||||||
job = session.get(Job, job_id)
|
|
||||||
if job is None:
|
|
||||||
return None, None, None
|
|
||||||
document = session.get(Document, job.document_id)
|
|
||||||
transcript = session.exec(select(Transcript).where(Transcript.job_id == job.id)).first()
|
|
||||||
return job, document, transcript
|
|
||||||
|
|
||||||
|
|
||||||
def register_page() -> None:
|
|
||||||
"""Register jobs list and detail routes."""
|
|
||||||
|
|
||||||
@ui.page("/jobs")
|
|
||||||
def jobs_page() -> None:
|
|
||||||
ui.label("Transcription Jobs")
|
|
||||||
status = ui.label("Ready")
|
|
||||||
table_container = ui.column()
|
|
||||||
|
|
||||||
def render_table() -> None:
|
|
||||||
table_container.clear()
|
|
||||||
jobs = fetch_jobs()
|
|
||||||
with table_container:
|
|
||||||
if not jobs:
|
|
||||||
ui.label("No jobs yet.")
|
|
||||||
return
|
|
||||||
rows = [
|
|
||||||
{
|
|
||||||
"id": str(job.id),
|
|
||||||
"status": job.status,
|
|
||||||
"created_at": job.created_at,
|
|
||||||
"updated_at": job.updated_at,
|
|
||||||
}
|
|
||||||
for job in jobs
|
|
||||||
]
|
|
||||||
ui.table(
|
|
||||||
columns=[
|
|
||||||
{"name": "id", "label": "Job ID", "field": "id"},
|
|
||||||
{"name": "status", "label": "Status", "field": "status"},
|
|
||||||
{"name": "created_at", "label": "Created", "field": "created_at"},
|
|
||||||
{"name": "updated_at", "label": "Updated", "field": "updated_at"},
|
|
||||||
],
|
|
||||||
rows=rows,
|
|
||||||
row_key="id",
|
|
||||||
)
|
|
||||||
for row in rows:
|
|
||||||
ui.link(f"Open {row['id']}", f"/jobs/{row['id']}")
|
|
||||||
|
|
||||||
def refresh() -> None:
|
|
||||||
status.text = "Refreshing..."
|
|
||||||
try:
|
|
||||||
render_table()
|
|
||||||
status.text = "Refreshed"
|
|
||||||
except Exception as exc: # noqa: BLE001
|
|
||||||
status.text = f"Refresh failed: {summarize_error(exc, operation='jobs.refresh')}"
|
|
||||||
show_error(exc, title="Jobs refresh failed", operation="jobs.refresh")
|
|
||||||
|
|
||||||
ui.button("Refresh", on_click=refresh)
|
|
||||||
render_table()
|
|
||||||
ui.link("Back to upload", "/")
|
|
||||||
|
|
||||||
@ui.page("/jobs/{job_id}")
|
|
||||||
def job_detail_page(job_id: str) -> None:
|
|
||||||
ui.label("Job Detail")
|
|
||||||
try:
|
|
||||||
parsed_id = UUID(job_id)
|
|
||||||
except ValueError:
|
|
||||||
ui.label("Invalid job id")
|
|
||||||
ui.link("Back to jobs", "/jobs")
|
|
||||||
return
|
|
||||||
|
|
||||||
job, document, transcript = fetch_job_detail(parsed_id)
|
|
||||||
if job is None:
|
|
||||||
ui.label("Job not found")
|
|
||||||
ui.link("Back to jobs", "/jobs")
|
|
||||||
return
|
|
||||||
|
|
||||||
ui.label(f"Job ID: {job.id}")
|
|
||||||
ui.label(f"Status: {job.status.value}")
|
|
||||||
ui.label(f"Created: {job.created_at.isoformat()}")
|
|
||||||
ui.label(f"Updated: {job.updated_at.isoformat()}")
|
|
||||||
|
|
||||||
if document is not None:
|
|
||||||
ui.label(f"Filename: {document.filename}")
|
|
||||||
ui.label(f"File path: {document.file_path}")
|
|
||||||
|
|
||||||
if transcript is None:
|
|
||||||
ui.label("Transcript not available yet.")
|
|
||||||
elif transcript.text:
|
|
||||||
ui.label("Transcript:")
|
|
||||||
ui.markdown(transcript.text)
|
|
||||||
elif transcript.error_detail:
|
|
||||||
ui.label("Failure detail:")
|
|
||||||
ui.label(transcript.error_detail)
|
|
||||||
|
|
||||||
ui.link("Back to jobs", "/jobs")
|
|
||||||
@@ -0,0 +1,163 @@
|
|||||||
|
"""Jobs list and detail page registration."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from uuid import UUID
|
||||||
|
|
||||||
|
from nicegui import ui
|
||||||
|
from sqlalchemy.orm import selectinload
|
||||||
|
from sqlmodel import desc
|
||||||
|
from sqlmodel import select
|
||||||
|
|
||||||
|
from transcription.db import get_session
|
||||||
|
from transcription.models import Document
|
||||||
|
from transcription.models import Job
|
||||||
|
from transcription.models import Transcript
|
||||||
|
from transcription.models import TranscriptRevision
|
||||||
|
from transcription.services import ServiceBundle
|
||||||
|
from transcription.ui.components.error_presenter import show_error
|
||||||
|
from transcription.ui.components.error_presenter import summarize_error
|
||||||
|
from transcription.ui.components.job_detail import render_job_detail
|
||||||
|
from transcription.ui.components.table.jobs import JobTableRow
|
||||||
|
from transcription.ui.components.table.jobs import render_jobs_table
|
||||||
|
|
||||||
|
|
||||||
|
async def fetch_job_rows() -> list[JobTableRow]:
|
||||||
|
"""Return jobs for display in most-recent-first order."""
|
||||||
|
async with get_session() as session:
|
||||||
|
jobs = (
|
||||||
|
await session.exec(
|
||||||
|
select(Job)
|
||||||
|
.options(selectinload(Job.document)) # pyright: ignore[reportArgumentType]
|
||||||
|
.order_by(desc(Job.created_at))
|
||||||
|
)
|
||||||
|
).all()
|
||||||
|
return [
|
||||||
|
JobTableRow(
|
||||||
|
id=job.id,
|
||||||
|
status=job.status.value,
|
||||||
|
filename=job.filename,
|
||||||
|
retry_count=job.retry_count,
|
||||||
|
created_at=job.created_at.isoformat(),
|
||||||
|
updated_at=job.updated_at.isoformat(),
|
||||||
|
)
|
||||||
|
for job in jobs
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
async def fetch_job_detail(job_id: UUID) -> tuple[Job | None, Document | None, Transcript | None, list[TranscriptRevision]]:
|
||||||
|
"""Return job, document, transcript snapshot, and revisions for detail view."""
|
||||||
|
async with get_session() as session:
|
||||||
|
job = await session.get(Job, job_id)
|
||||||
|
if job is None:
|
||||||
|
return None, None, None, []
|
||||||
|
document = await session.get(Document, job.document_id)
|
||||||
|
transcript = (await session.exec(select(Transcript).where(Transcript.job_id == job.id))).first()
|
||||||
|
revisions = (
|
||||||
|
await session.exec(
|
||||||
|
select(TranscriptRevision)
|
||||||
|
.where(TranscriptRevision.job_id == job.id)
|
||||||
|
.order_by(TranscriptRevision.version_number)
|
||||||
|
)
|
||||||
|
).all()
|
||||||
|
return job, document, transcript, list(revisions)
|
||||||
|
|
||||||
|
|
||||||
|
def register_page() -> None:
|
||||||
|
"""Register jobs list and detail routes."""
|
||||||
|
|
||||||
|
@ui.page("/jobs")
|
||||||
|
async def jobs_page() -> None:
|
||||||
|
ui.label("Transcription Jobs")
|
||||||
|
status = ui.label("Ready")
|
||||||
|
|
||||||
|
@ui.refreshable
|
||||||
|
async def render_table() -> None:
|
||||||
|
jobs = await fetch_job_rows()
|
||||||
|
render_jobs_table(jobs)
|
||||||
|
|
||||||
|
async def refresh() -> None:
|
||||||
|
status.text = "Refreshing..."
|
||||||
|
try:
|
||||||
|
await render_table.refresh()
|
||||||
|
status.text = "Refreshed"
|
||||||
|
except Exception as exc: # noqa: BLE001
|
||||||
|
status.text = f"Refresh failed: {summarize_error(exc, operation='jobs.refresh')}"
|
||||||
|
show_error(exc, title="Jobs refresh failed", operation="jobs.refresh")
|
||||||
|
|
||||||
|
ui.button("Refresh", on_click=refresh)
|
||||||
|
await render_table()
|
||||||
|
ui.link("Back to upload", "/upload")
|
||||||
|
|
||||||
|
@ui.page("/jobs/{job_id}")
|
||||||
|
async def job_detail_page(job_id: str) -> None:
|
||||||
|
ui.label("Job Detail")
|
||||||
|
content = ui.column().classes("w-full")
|
||||||
|
try:
|
||||||
|
parsed_id = UUID(job_id)
|
||||||
|
except ValueError:
|
||||||
|
ui.label("Invalid job id")
|
||||||
|
ui.link("Back to jobs", "/jobs")
|
||||||
|
return
|
||||||
|
|
||||||
|
async def refresh_content() -> None:
|
||||||
|
content.clear()
|
||||||
|
job, document, transcript, revisions = await fetch_job_detail(parsed_id)
|
||||||
|
if job is None:
|
||||||
|
with content:
|
||||||
|
ui.label("Job not found")
|
||||||
|
return
|
||||||
|
|
||||||
|
services = ServiceBundle()
|
||||||
|
|
||||||
|
async def update_transcript_text(value: str) -> None:
|
||||||
|
try:
|
||||||
|
update_text = value.strip()
|
||||||
|
async with get_session() as session:
|
||||||
|
current_transcript = (
|
||||||
|
await session.exec(select(Transcript).where(Transcript.job_id == parsed_id))
|
||||||
|
).first()
|
||||||
|
provider_name = current_transcript.provider if current_transcript is not None else "openrouter"
|
||||||
|
prompt_name = (
|
||||||
|
current_transcript.prompt_name if current_transcript is not None else "transcribe_document.md"
|
||||||
|
)
|
||||||
|
model_name = current_transcript.model if current_transcript is not None else None
|
||||||
|
|
||||||
|
await services.transcriptions.upsert_transcript_by_job(
|
||||||
|
job_id=parsed_id,
|
||||||
|
text=update_text,
|
||||||
|
error_detail=None,
|
||||||
|
provider=provider_name,
|
||||||
|
prompt_name=prompt_name,
|
||||||
|
model=model_name,
|
||||||
|
session=session,
|
||||||
|
)
|
||||||
|
await services.transcriptions.append_transcript_revision(
|
||||||
|
job_id=parsed_id,
|
||||||
|
text=update_text,
|
||||||
|
error_detail=None,
|
||||||
|
provider=provider_name,
|
||||||
|
prompt_name=prompt_name,
|
||||||
|
model=model_name,
|
||||||
|
source="user",
|
||||||
|
session=session,
|
||||||
|
)
|
||||||
|
await session.commit()
|
||||||
|
|
||||||
|
ui.notify("Transcript updated", type="positive")
|
||||||
|
await refresh_content()
|
||||||
|
except Exception as exc: # noqa: BLE001
|
||||||
|
show_error(exc, title="Transcript update failed", operation="jobs.detail.update")
|
||||||
|
|
||||||
|
with content:
|
||||||
|
render_job_detail(
|
||||||
|
job=job,
|
||||||
|
document=document,
|
||||||
|
transcript=transcript,
|
||||||
|
revisions=revisions,
|
||||||
|
on_update=update_transcript_text,
|
||||||
|
)
|
||||||
|
|
||||||
|
await refresh_content()
|
||||||
|
|
||||||
|
ui.link("Back to jobs", "/jobs")
|
||||||
@@ -0,0 +1,34 @@
|
|||||||
|
"""Upload page registration and handlers."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from fastapi import Request
|
||||||
|
from nicegui import ui
|
||||||
|
|
||||||
|
from transcription.app_state import resolve_session_factory
|
||||||
|
from transcription.db import get_session
|
||||||
|
from transcription.services.store import create_upload_job
|
||||||
|
from transcription.ui.components.upload import render_upload_widget
|
||||||
|
from transcription.worker import resolve_worker_notifier
|
||||||
|
|
||||||
|
|
||||||
|
def register_page() -> None:
|
||||||
|
"""Register the upload page route."""
|
||||||
|
|
||||||
|
@ui.page("/upload", title="Upload Document")
|
||||||
|
def upload_page(request: Request) -> None:
|
||||||
|
session_factory = resolve_session_factory(request.app.state)
|
||||||
|
|
||||||
|
async def submit_upload(filename: str, file_bytes: bytes):
|
||||||
|
async with get_session(session_factory=session_factory) as session:
|
||||||
|
return await create_upload_job(
|
||||||
|
filename=filename,
|
||||||
|
file_bytes=file_bytes,
|
||||||
|
session=session,
|
||||||
|
)
|
||||||
|
|
||||||
|
notify_worker = resolve_worker_notifier(request.app.state)
|
||||||
|
render_upload_widget(submitter=submit_upload, notifier=notify_worker)
|
||||||
|
|
||||||
|
with ui.row():
|
||||||
|
ui.link("View jobs", "/jobs")
|
||||||
@@ -1,71 +0,0 @@
|
|||||||
"""Upload page registration and handlers."""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
from dataclasses import dataclass
|
|
||||||
|
|
||||||
from nicegui import ui
|
|
||||||
from nicegui.events import UploadEventArguments
|
|
||||||
|
|
||||||
from transcription.services.upload import UploadError, UploadJobResult, create_upload_job
|
|
||||||
from transcription.ui.error_presenter import show_error, summarize_error
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
|
||||||
class UploadPageState:
|
|
||||||
"""Simple state container for upload page feedback."""
|
|
||||||
|
|
||||||
loading: bool = False
|
|
||||||
message: str = ""
|
|
||||||
|
|
||||||
|
|
||||||
def accepted_upload_types() -> str:
|
|
||||||
"""Return accepted file type string for upload input."""
|
|
||||||
return ".jpg,.jpeg,.png,.tif,.tiff,.pdf"
|
|
||||||
|
|
||||||
|
|
||||||
def submit_upload(*, filename: str, file_bytes: bytes) -> UploadJobResult:
|
|
||||||
"""Create an upload job from incoming file data."""
|
|
||||||
return create_upload_job(filename=filename, file_bytes=file_bytes)
|
|
||||||
|
|
||||||
|
|
||||||
def register_page() -> None:
|
|
||||||
"""Register the upload page route."""
|
|
||||||
|
|
||||||
@ui.page("/")
|
|
||||||
def upload_page() -> None:
|
|
||||||
state = UploadPageState()
|
|
||||||
status_label = ui.label("Upload a document to start transcription.")
|
|
||||||
|
|
||||||
async def on_upload(event: UploadEventArguments) -> None:
|
|
||||||
if state.loading:
|
|
||||||
ui.notify("Upload already in progress. Please wait.", type="warning")
|
|
||||||
return
|
|
||||||
|
|
||||||
state.loading = True
|
|
||||||
status_label.text = "Uploading..."
|
|
||||||
try:
|
|
||||||
payload = await event.file.read()
|
|
||||||
result = submit_upload(filename=event.file.name, file_bytes=payload)
|
|
||||||
state.message = f"Created job {result.job_id}"
|
|
||||||
status_label.text = state.message
|
|
||||||
ui.notify(state.message, type="positive")
|
|
||||||
except UploadError as exc:
|
|
||||||
state.message = summarize_error(exc, operation="upload.submit")
|
|
||||||
status_label.text = f"Upload failed: {state.message}"
|
|
||||||
show_error(exc, title="Upload failed", operation="upload.submit")
|
|
||||||
except Exception as exc: # noqa: BLE001
|
|
||||||
state.message = summarize_error(exc, operation="upload.submit")
|
|
||||||
status_label.text = f"Upload failed: {state.message}"
|
|
||||||
show_error(exc, title="Upload failed", operation="upload.submit")
|
|
||||||
finally:
|
|
||||||
state.loading = False
|
|
||||||
|
|
||||||
ui.upload(
|
|
||||||
on_upload=on_upload,
|
|
||||||
auto_upload=True,
|
|
||||||
label="Select document file",
|
|
||||||
).props(f"accept={accepted_upload_types()}")
|
|
||||||
|
|
||||||
with ui.row():
|
|
||||||
ui.link("View jobs", "/jobs")
|
|
||||||
@@ -2,157 +2,183 @@
|
|||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import asyncio
|
||||||
import logging
|
import logging
|
||||||
import time
|
from collections.abc import AsyncGenerator
|
||||||
from datetime import datetime, timezone
|
from contextlib import asynccontextmanager
|
||||||
from threading import Event
|
from contextlib import contextmanager
|
||||||
|
from contextlib import suppress
|
||||||
|
from typing import Protocol
|
||||||
|
from uuid import UUID
|
||||||
|
|
||||||
from pydantic import ValidationError
|
from sqlalchemy.ext.asyncio import async_sessionmaker
|
||||||
from sqlalchemy.engine import Engine
|
from sqlmodel.ext.asyncio.session import AsyncSession
|
||||||
from sqlmodel import Session, select
|
|
||||||
|
|
||||||
from transcription.config import Settings, get_settings
|
|
||||||
from transcription.db import get_session
|
from transcription.db import get_session
|
||||||
from transcription.errors import AppError, ErrorCategory, classify_unexpected_error, format_error_detail
|
from transcription.errors import AppError
|
||||||
from transcription.models import Document, Job, JobStatus, Transcript
|
from transcription.errors import classify_unexpected_error
|
||||||
from transcription.services.transcription import transcribe_document_image
|
|
||||||
|
from .services import ServiceBundle
|
||||||
|
from .services.documents import DocumentService
|
||||||
|
from .services.jobs import JobService
|
||||||
|
from .services.transcription import TranscriptionService
|
||||||
|
from .services.workflows import advance_job
|
||||||
|
from .services.workflows import process_next_queued_job as process_next_queued_job_workflow
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
def process_next_queued_job(*, session: Session | None = None, engine: Engine | None = None) -> bool:
|
class WorkerNotifier(Protocol):
|
||||||
"""Process the next queued job and persist terminal outcome.
|
"""Abstraction for signaling the worker loop about new work."""
|
||||||
|
|
||||||
Returns True when a job was processed, False when no queued job exists.
|
def notify(self) -> None:
|
||||||
"""
|
"""Signal the worker loop that work may be available."""
|
||||||
if session is None:
|
|
||||||
with get_session(engine=engine) as local_session:
|
|
||||||
return _process_next_queued_job(session=local_session)
|
|
||||||
return _process_next_queued_job(session=session)
|
|
||||||
|
|
||||||
|
|
||||||
def _process_next_queued_job(*, session: Session) -> bool:
|
class EventWorkerNotifier:
|
||||||
job = session.exec(
|
"""Worker notifier backed by an asyncio.Event."""
|
||||||
select(Job)
|
|
||||||
.where(Job.status == JobStatus.QUEUED)
|
|
||||||
.order_by(Job.created_at)
|
|
||||||
).first()
|
|
||||||
|
|
||||||
if job is None:
|
def __init__(self, wake_event: asyncio.Event):
|
||||||
return False
|
self._wake_event = wake_event
|
||||||
|
|
||||||
logger.info("Picked queued job operation=worker.pick job_id=%s", job.id)
|
def notify(self) -> None:
|
||||||
job.status = JobStatus.PROCESSING
|
self._wake_event.set()
|
||||||
job.updated_at = datetime.now(timezone.utc)
|
|
||||||
session.add(job)
|
|
||||||
session.commit()
|
|
||||||
session.refresh(job)
|
|
||||||
|
|
||||||
document = session.get(Document, job.document_id)
|
|
||||||
if document is None:
|
class NoopWorkerNotifier:
|
||||||
error = AppError(
|
"""Fallback notifier used when worker signaling is unavailable."""
|
||||||
"Document not found",
|
|
||||||
category=ErrorCategory.NOT_FOUND,
|
def notify(self) -> None:
|
||||||
suggestion="Re-upload the source document and retry processing.",
|
return
|
||||||
|
|
||||||
|
|
||||||
|
def resolve_worker_notifier(state: object) -> WorkerNotifier:
|
||||||
|
"""Resolve notifier from app-like state objects with no-op fallback."""
|
||||||
|
notifier = getattr(state, "worker_notifier", None)
|
||||||
|
if isinstance(notifier, NoopWorkerNotifier):
|
||||||
|
return notifier
|
||||||
|
if notifier is None:
|
||||||
|
return NoopWorkerNotifier()
|
||||||
|
return notifier
|
||||||
|
|
||||||
|
|
||||||
|
@asynccontextmanager
|
||||||
|
async def worker_consumer_lifespan(
|
||||||
|
*,
|
||||||
|
session_factory: async_sessionmaker[AsyncSession] | None = None,
|
||||||
|
poll_interval_seconds: float = 1.0,
|
||||||
|
) -> AsyncGenerator[tuple[asyncio.Event, WorkerNotifier]]:
|
||||||
|
"""Start and stop the worker consumer loop for app lifespan."""
|
||||||
|
stop_event = asyncio.Event()
|
||||||
|
wake_event = asyncio.Event()
|
||||||
|
worker_notifier: WorkerNotifier = EventWorkerNotifier(wake_event)
|
||||||
|
worker_task = asyncio.create_task(
|
||||||
|
run_worker_loop(
|
||||||
|
session_factory=session_factory,
|
||||||
|
stop_event=stop_event,
|
||||||
|
wake_event=wake_event,
|
||||||
|
poll_interval_seconds=poll_interval_seconds,
|
||||||
)
|
)
|
||||||
_finalize_failed_job(session=session, job=job, error=error)
|
)
|
||||||
logger.error(
|
worker_notifier.notify()
|
||||||
"Job failed operation=worker.process_job job_id=%s error_id=%s category=%s",
|
|
||||||
job.id,
|
try:
|
||||||
|
yield stop_event, worker_notifier
|
||||||
|
finally:
|
||||||
|
stop_event.set()
|
||||||
|
worker_notifier.notify()
|
||||||
|
try:
|
||||||
|
await asyncio.wait_for(worker_task, timeout=2.0)
|
||||||
|
except TimeoutError:
|
||||||
|
worker_task.cancel()
|
||||||
|
with suppress(asyncio.CancelledError):
|
||||||
|
await worker_task
|
||||||
|
|
||||||
|
|
||||||
|
async def queue_consumer_loop(queue: asyncio.Queue[UUID], stop_event: asyncio.Event):
|
||||||
|
"""Main worker loop that consumes jobs from the queue and processes them.
|
||||||
|
|
||||||
|
The queue is for Job UUIDs, and the corresponding documents should already have been uploaded.
|
||||||
|
"""
|
||||||
|
service = JobService()
|
||||||
|
while not stop_event.is_set():
|
||||||
|
with handle_worker_exceptions():
|
||||||
|
async with _get_queue_item(queue) as job_id:
|
||||||
|
job = await service.read_job(job_id)
|
||||||
|
asyncio.create_task(advance_job(job=job, services=ServiceBundle()))
|
||||||
|
|
||||||
|
|
||||||
|
@contextmanager
|
||||||
|
def handle_worker_exceptions(operation: str = "worker.loop"):
|
||||||
|
"""Context manager to log and suppress exceptions in the worker loop."""
|
||||||
|
try:
|
||||||
|
yield
|
||||||
|
except Exception as exc:
|
||||||
|
error = exc if isinstance(exc, AppError) else classify_unexpected_error(exc, operation=operation)
|
||||||
|
logger.exception(
|
||||||
|
"Worker loop exception error_id=%s category=%s",
|
||||||
error.error_id,
|
error.error_id,
|
||||||
error.category.value,
|
error.category.value,
|
||||||
)
|
)
|
||||||
return True
|
|
||||||
|
|
||||||
try:
|
|
||||||
result = transcribe_document_image(document.file_path)
|
|
||||||
_upsert_transcript(session=session, job_id=job.id, text=result.text, error_detail=None)
|
|
||||||
job.status = JobStatus.TRANSCRIBED
|
|
||||||
job.updated_at = datetime.now(timezone.utc)
|
|
||||||
session.add(job)
|
|
||||||
session.commit()
|
|
||||||
logger.info(
|
|
||||||
"Job transcribed operation=worker.process_job job_id=%s document_id=%s provider=%s",
|
|
||||||
job.id,
|
|
||||||
document.id,
|
|
||||||
result.provider,
|
|
||||||
)
|
|
||||||
except Exception as exc: # noqa: BLE001
|
|
||||||
error = exc if isinstance(exc, AppError) else classify_unexpected_error(exc, operation="worker.process_job")
|
|
||||||
settings = _get_worker_settings()
|
|
||||||
if _should_retry(job=job, error=error, settings=settings):
|
|
||||||
_requeue_for_retry(session=session, job=job, error=error, settings=settings)
|
|
||||||
logger.warning(
|
|
||||||
"Job retried operation=worker.process_job job_id=%s document_id=%s retry_count=%s error_id=%s category=%s",
|
|
||||||
job.id,
|
|
||||||
document.id,
|
|
||||||
job.retry_count,
|
|
||||||
error.error_id,
|
|
||||||
error.category.value,
|
|
||||||
)
|
|
||||||
else:
|
|
||||||
_finalize_failed_job(session=session, job=job, error=error)
|
|
||||||
logger.exception(
|
|
||||||
"Job failed operation=worker.process_job job_id=%s document_id=%s error_id=%s category=%s",
|
|
||||||
job.id,
|
|
||||||
document.id,
|
|
||||||
error.error_id,
|
|
||||||
error.category.value,
|
|
||||||
)
|
|
||||||
|
|
||||||
return True
|
|
||||||
|
|
||||||
|
|
||||||
def _upsert_transcript(*, session: Session, job_id, text: str | None, error_detail: str | None) -> Transcript:
|
@asynccontextmanager
|
||||||
transcript = session.exec(select(Transcript).where(Transcript.job_id == job_id)).first()
|
async def _get_queue_item(queue: asyncio.Queue[UUID]) -> AsyncGenerator[UUID]:
|
||||||
if transcript is None:
|
"""Context manager to enqueue a job and ensure it is marked done."""
|
||||||
transcript = Transcript(job_id=job_id)
|
yield await queue.get()
|
||||||
|
queue.task_done()
|
||||||
transcript.text = text
|
|
||||||
transcript.error_detail = error_detail
|
|
||||||
session.add(transcript)
|
|
||||||
session.commit()
|
|
||||||
session.refresh(transcript)
|
|
||||||
return transcript
|
|
||||||
|
|
||||||
|
|
||||||
def _get_worker_settings() -> Settings:
|
async def run_worker_loop(
|
||||||
try:
|
*,
|
||||||
return get_settings()
|
session_factory: async_sessionmaker[AsyncSession] | None = None,
|
||||||
except ValidationError:
|
stop_event: asyncio.Event | None = None,
|
||||||
return Settings(openrouter_api_key="test-key")
|
wake_event: asyncio.Event | None = None,
|
||||||
|
poll_interval_seconds: float = 1.0,
|
||||||
|
) -> None:
|
||||||
|
"""Run worker loop until stop_event is set.
|
||||||
|
|
||||||
|
If wake_event is provided, signal activity wakes the loop immediately while
|
||||||
def _should_retry(*, job: Job, error: AppError, settings: Settings) -> bool:
|
timeout-based wakeups preserve current polling behavior.
|
||||||
return error.retriable and job.retry_count < settings.worker_max_retries
|
"""
|
||||||
|
|
||||||
|
|
||||||
def _requeue_for_retry(*, session: Session, job: Job, error: AppError, settings: Settings) -> None:
|
|
||||||
_upsert_transcript(session=session, job_id=job.id, text=None, error_detail=format_error_detail(error))
|
|
||||||
job.retry_count += 1
|
|
||||||
job.status = JobStatus.QUEUED
|
|
||||||
job.updated_at = datetime.now(timezone.utc)
|
|
||||||
session.add(job)
|
|
||||||
session.commit()
|
|
||||||
if settings.worker_retry_backoff_seconds > 0:
|
|
||||||
time.sleep(settings.worker_retry_backoff_seconds)
|
|
||||||
|
|
||||||
|
|
||||||
def _finalize_failed_job(*, session: Session, job: Job, error: AppError) -> None:
|
|
||||||
_upsert_transcript(session=session, job_id=job.id, text=None, error_detail=format_error_detail(error))
|
|
||||||
job.status = JobStatus.FAILED
|
|
||||||
job.updated_at = datetime.now(timezone.utc)
|
|
||||||
session.add(job)
|
|
||||||
session.commit()
|
|
||||||
|
|
||||||
|
|
||||||
def run_worker_loop(*, engine: Engine | None = None, stop_event: Event | None = None, poll_interval_seconds: float = 1.0) -> None:
|
|
||||||
"""Run worker polling loop until stop_event is set."""
|
|
||||||
while True:
|
while True:
|
||||||
if stop_event is not None and stop_event.is_set():
|
if stop_event is not None and stop_event.is_set():
|
||||||
logger.info("Worker stop event received")
|
logger.info("Worker stop event received")
|
||||||
return
|
return
|
||||||
|
|
||||||
processed = process_next_queued_job(engine=engine)
|
if wake_event is not None:
|
||||||
if not processed:
|
with suppress(TimeoutError):
|
||||||
time.sleep(poll_interval_seconds)
|
await asyncio.wait_for(wake_event.wait(), timeout=poll_interval_seconds)
|
||||||
|
wake_event.clear()
|
||||||
|
|
||||||
|
processed_any = False
|
||||||
|
while await process_next_queued_job(session_factory=session_factory):
|
||||||
|
processed_any = True
|
||||||
|
|
||||||
|
if wake_event is None and not processed_any:
|
||||||
|
await asyncio.sleep(poll_interval_seconds)
|
||||||
|
|
||||||
|
|
||||||
|
async def process_next_queued_job(
|
||||||
|
*,
|
||||||
|
session: AsyncSession | None = None,
|
||||||
|
session_factory: async_sessionmaker[AsyncSession] | None = None,
|
||||||
|
) -> bool:
|
||||||
|
"""Process the next queued job and persist terminal outcome.
|
||||||
|
|
||||||
|
Returns True when a job was processed, False when no queued job exists.
|
||||||
|
"""
|
||||||
|
if session_factory is None:
|
||||||
|
services = ServiceBundle()
|
||||||
|
else:
|
||||||
|
services = ServiceBundle(
|
||||||
|
documents=DocumentService(session_factory=session_factory),
|
||||||
|
jobs=JobService(session_factory=session_factory),
|
||||||
|
transcriptions=TranscriptionService(session_factory=session_factory),
|
||||||
|
)
|
||||||
|
|
||||||
|
if session is None:
|
||||||
|
async with get_session(session_factory=session_factory) as local_session:
|
||||||
|
return await process_next_queued_job_workflow(services=services, session=local_session)
|
||||||
|
|
||||||
|
return await process_next_queued_job_workflow(services=services, session=session)
|
||||||
|
|||||||
@@ -5,18 +5,67 @@ isolated, fast, and leave no artifacts on disk.
|
|||||||
"""
|
"""
|
||||||
|
|
||||||
import pytest
|
import pytest
|
||||||
from sqlmodel import Session, SQLModel, create_engine
|
import pytest_asyncio
|
||||||
|
from sqlmodel import Session
|
||||||
|
from sqlmodel import SQLModel
|
||||||
|
from sqlmodel import create_engine
|
||||||
from sqlmodel.pool import StaticPool
|
from sqlmodel.pool import StaticPool
|
||||||
|
|
||||||
|
from transcription.config import Settings
|
||||||
|
from transcription.config import get_settings
|
||||||
|
from transcription.db.operations import create_all
|
||||||
|
from transcription.db.runtime import dispose_database_runtime
|
||||||
|
from transcription.db.runtime import get_engine
|
||||||
|
from transcription.db.runtime import get_session
|
||||||
|
from transcription.db.runtime import get_session_factory
|
||||||
|
from transcription.services.documents import DocumentService
|
||||||
|
from transcription.services.jobs import JobService
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture
|
@pytest.fixture
|
||||||
def session():
|
def session():
|
||||||
"""Provide a clean database session for each test."""
|
"""Provide a clean synchronous database session for sync tests."""
|
||||||
engine = create_engine(
|
engine = create_engine(
|
||||||
"sqlite://",
|
"sqlite://",
|
||||||
connect_args={"check_same_thread": False},
|
connect_args={"check_same_thread": False},
|
||||||
poolclass=StaticPool,
|
poolclass=StaticPool,
|
||||||
)
|
)
|
||||||
SQLModel.metadata.create_all(engine)
|
SQLModel.metadata.create_all(engine)
|
||||||
with Session(engine) as session:
|
with Session(engine) as sync_session:
|
||||||
yield session
|
yield sync_session
|
||||||
|
|
||||||
|
|
||||||
|
@pytest_asyncio.fixture
|
||||||
|
async def default_settings():
|
||||||
|
"""Provide default settings for tests."""
|
||||||
|
settings = get_settings(database_url="sqlite:///:memory:")
|
||||||
|
await create_all(engine=get_engine(settings=settings))
|
||||||
|
return settings
|
||||||
|
|
||||||
|
|
||||||
|
@pytest_asyncio.fixture
|
||||||
|
async def async_session(default_settings: Settings):
|
||||||
|
"""Provide a clean asynchronous database session for async tests."""
|
||||||
|
async with get_session(settings=default_settings) as async_session:
|
||||||
|
yield async_session
|
||||||
|
|
||||||
|
await dispose_database_runtime()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def default_session_factory(default_settings: Settings):
|
||||||
|
"""Provide a base fixture for tests that require database access."""
|
||||||
|
session_factory = get_session_factory(settings=default_settings)
|
||||||
|
return session_factory
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def job_service(default_session_factory) -> JobService:
|
||||||
|
"""Provide a JobService instance for testing."""
|
||||||
|
return JobService(session_factory=default_session_factory)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def document_service(default_session_factory) -> DocumentService:
|
||||||
|
"""Provide a DocumentService instance for testing."""
|
||||||
|
return DocumentService(session_factory=default_session_factory)
|
||||||
|
|||||||
@@ -6,9 +6,9 @@ import pytest
|
|||||||
from sqlmodel import select
|
from sqlmodel import select
|
||||||
|
|
||||||
from transcription.config import Settings
|
from transcription.config import Settings
|
||||||
from transcription.models import Job, JobStatus, Transcript
|
from transcription.models import Job, JobStatus, Transcript, TranscriptRevision
|
||||||
from transcription.providers.base import TranscriptionResult
|
from transcription.providers.base import TranscriptionResult
|
||||||
from transcription.services.upload import create_upload_job
|
from transcription.services.store import create_upload_job
|
||||||
from transcription.worker import process_next_queued_job
|
from transcription.worker import process_next_queued_job
|
||||||
|
|
||||||
|
|
||||||
@@ -16,24 +16,37 @@ from transcription.worker import process_next_queued_job
|
|||||||
class TestPipelineSuccessFlow:
|
class TestPipelineSuccessFlow:
|
||||||
"""Verify end-to-end success lifecycle behavior."""
|
"""Verify end-to-end success lifecycle behavior."""
|
||||||
|
|
||||||
def test_upload_then_worker_persists_transcribed_terminal_state(self, session, tmp_path: Path, monkeypatch):
|
@pytest.mark.asyncio
|
||||||
|
async def test_upload_then_worker_persists_transcribed_terminal_state(self, async_session, tmp_path: Path, monkeypatch):
|
||||||
"""Upload followed by worker processing persists transcript and transcribed status."""
|
"""Upload followed by worker processing persists transcript and transcribed status."""
|
||||||
settings = Settings(openrouter_api_key="test-key", upload_dir=tmp_path)
|
settings = Settings(openrouter_api_key="test-key", upload_dir=tmp_path)
|
||||||
upload_result = create_upload_job(
|
upload_result = await create_upload_job(
|
||||||
filename="pipeline.jpg",
|
filename="pipeline.jpg",
|
||||||
file_bytes=b"pipeline-bytes",
|
file_bytes=b"pipeline-bytes",
|
||||||
session=session,
|
session=async_session,
|
||||||
settings=settings,
|
settings=settings,
|
||||||
)
|
)
|
||||||
|
|
||||||
def _fake_transcribe(_path: str) -> TranscriptionResult:
|
async def _fake_transcribe(_path: str) -> TranscriptionResult:
|
||||||
return TranscriptionResult(text="Pipeline transcript", provider="openrouter", model="test-model")
|
return TranscriptionResult(
|
||||||
|
text="Pipeline transcript",
|
||||||
|
provider="openrouter",
|
||||||
|
prompt_name="transcribe_document.md",
|
||||||
|
model="test-model",
|
||||||
|
)
|
||||||
|
|
||||||
monkeypatch.setattr("transcription.worker.transcribe_document_image", _fake_transcribe)
|
monkeypatch.setattr("transcription.services.workflows.transcribe_document_image", _fake_transcribe)
|
||||||
|
|
||||||
processed = process_next_queued_job(session=session)
|
processed = await process_next_queued_job(session=async_session)
|
||||||
job = session.get(Job, upload_result.job_id)
|
job = await async_session.get(Job, upload_result.job_id)
|
||||||
transcript = session.exec(select(Transcript).where(Transcript.job_id == upload_result.job_id)).first()
|
transcript = (await async_session.exec(select(Transcript).where(Transcript.job_id == upload_result.job_id))).first()
|
||||||
|
revisions = (
|
||||||
|
await async_session.exec(
|
||||||
|
select(TranscriptRevision)
|
||||||
|
.where(TranscriptRevision.job_id == upload_result.job_id)
|
||||||
|
.order_by(TranscriptRevision.version_number)
|
||||||
|
)
|
||||||
|
).all()
|
||||||
|
|
||||||
assert processed is True
|
assert processed is True
|
||||||
assert job is not None
|
assert job is not None
|
||||||
@@ -41,30 +54,43 @@ class TestPipelineSuccessFlow:
|
|||||||
assert transcript is not None
|
assert transcript is not None
|
||||||
assert transcript.text == "Pipeline transcript"
|
assert transcript.text == "Pipeline transcript"
|
||||||
assert transcript.error_detail is None
|
assert transcript.error_detail is None
|
||||||
|
assert transcript.model == "test-model"
|
||||||
|
assert len(revisions) == 1
|
||||||
|
assert revisions[0].version_number == 1
|
||||||
|
assert revisions[0].source == "ai"
|
||||||
|
assert revisions[0].text == "Pipeline transcript"
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.integration
|
@pytest.mark.integration
|
||||||
class TestPipelineFailureFlow:
|
class TestPipelineFailureFlow:
|
||||||
"""Verify end-to-end failure lifecycle behavior."""
|
"""Verify end-to-end failure lifecycle behavior."""
|
||||||
|
|
||||||
def test_upload_then_worker_persists_failed_terminal_state(self, session, tmp_path: Path, monkeypatch):
|
@pytest.mark.asyncio
|
||||||
|
async def test_upload_then_worker_persists_failed_terminal_state(self, async_session, tmp_path: Path, monkeypatch):
|
||||||
"""Upload followed by worker processing persists error detail and failed status."""
|
"""Upload followed by worker processing persists error detail and failed status."""
|
||||||
settings = Settings(openrouter_api_key="test-key", upload_dir=tmp_path)
|
settings = Settings(openrouter_api_key="test-key", upload_dir=tmp_path)
|
||||||
upload_result = create_upload_job(
|
upload_result = await create_upload_job(
|
||||||
filename="pipeline.jpg",
|
filename="pipeline.jpg",
|
||||||
file_bytes=b"pipeline-bytes",
|
file_bytes=b"pipeline-bytes",
|
||||||
session=session,
|
session=async_session,
|
||||||
settings=settings,
|
settings=settings,
|
||||||
)
|
)
|
||||||
|
|
||||||
def _fake_transcribe(_path: str) -> TranscriptionResult:
|
async def _fake_transcribe(_path: str) -> TranscriptionResult:
|
||||||
raise RuntimeError("pipeline provider failure")
|
raise RuntimeError("pipeline provider failure")
|
||||||
|
|
||||||
monkeypatch.setattr("transcription.worker.transcribe_document_image", _fake_transcribe)
|
monkeypatch.setattr("transcription.services.workflows.transcribe_document_image", _fake_transcribe)
|
||||||
|
|
||||||
processed = process_next_queued_job(session=session)
|
processed = await process_next_queued_job(session=async_session)
|
||||||
job = session.get(Job, upload_result.job_id)
|
job = await async_session.get(Job, upload_result.job_id)
|
||||||
transcript = session.exec(select(Transcript).where(Transcript.job_id == upload_result.job_id)).first()
|
transcript = (await async_session.exec(select(Transcript).where(Transcript.job_id == upload_result.job_id))).first()
|
||||||
|
revisions = (
|
||||||
|
await async_session.exec(
|
||||||
|
select(TranscriptRevision)
|
||||||
|
.where(TranscriptRevision.job_id == upload_result.job_id)
|
||||||
|
.order_by(TranscriptRevision.version_number)
|
||||||
|
)
|
||||||
|
).all()
|
||||||
|
|
||||||
assert processed is True
|
assert processed is True
|
||||||
assert job is not None
|
assert job is not None
|
||||||
@@ -74,3 +100,8 @@ class TestPipelineFailureFlow:
|
|||||||
assert "pipeline provider failure" in transcript.error_detail
|
assert "pipeline provider failure" in transcript.error_detail
|
||||||
assert "[internal_unexpected_error]" in transcript.error_detail
|
assert "[internal_unexpected_error]" in transcript.error_detail
|
||||||
assert "error_id=" in transcript.error_detail
|
assert "error_id=" in transcript.error_detail
|
||||||
|
assert len(revisions) == 1
|
||||||
|
assert revisions[0].version_number == 1
|
||||||
|
assert revisions[0].source == "ai"
|
||||||
|
assert revisions[0].text is None
|
||||||
|
assert "pipeline provider failure" in (revisions[0].error_detail or "")
|
||||||
|
|||||||
@@ -0,0 +1,88 @@
|
|||||||
|
from uuid import uuid4
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from transcription.models import Document
|
||||||
|
from transcription.models import Job
|
||||||
|
from transcription.services.documents import DocumentService
|
||||||
|
from transcription.services.jobs import JobService
|
||||||
|
from transcription.services.jobs import JobStatus
|
||||||
|
|
||||||
|
|
||||||
|
class TestJobService:
|
||||||
|
class TestBasicCRUD:
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_create_job(self, job_service: JobService):
|
||||||
|
"""Test creating a job."""
|
||||||
|
|
||||||
|
def fake_job_factory():
|
||||||
|
return Job(document_id=uuid4())
|
||||||
|
|
||||||
|
await job_service.create_job(job=fake_job_factory())
|
||||||
|
|
||||||
|
async with job_service._session_scope() as session:
|
||||||
|
for _ in range(10):
|
||||||
|
await job_service.create_job(job=fake_job_factory(), session=session)
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_backpropagation(self, job_service: JobService, document_service: DocumentService):
|
||||||
|
"""Test that creating a job backpropagates to the related document."""
|
||||||
|
doc_id = uuid4()
|
||||||
|
document = Document(
|
||||||
|
id=doc_id,
|
||||||
|
filename="test.txt",
|
||||||
|
file_path="/path/to/test.txt",
|
||||||
|
)
|
||||||
|
await document_service.create_document(document=document)
|
||||||
|
job = Job(document_id=doc_id)
|
||||||
|
await job_service.create_job(job=job)
|
||||||
|
|
||||||
|
read_job = await job_service.read_job(job_id=job.id)
|
||||||
|
assert isinstance(read_job.document, Document)
|
||||||
|
assert read_job.document.id == document.id
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_reading_job(self, job_service: JobService):
|
||||||
|
"""Test reading a job."""
|
||||||
|
uuid = uuid4()
|
||||||
|
await job_service.create_job(job=Job(id=uuid, document_id=uuid4()))
|
||||||
|
job = await job_service.read_job(job_id=uuid)
|
||||||
|
assert job.id == uuid
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_updating_job(self, job_service: JobService):
|
||||||
|
"""Test updating a job."""
|
||||||
|
uuid = uuid4()
|
||||||
|
job = Job(id=uuid, document_id=uuid4())
|
||||||
|
async with job_service._session_scope() as session:
|
||||||
|
await job_service.create_job(job=job, session=session)
|
||||||
|
job.status = JobStatus.PROCESSING
|
||||||
|
await job_service.update_job(job=job, session=session)
|
||||||
|
read_job = await job_service.read_job(job_id=uuid, session=session)
|
||||||
|
assert read_job == job
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_deleting_job(self, job_service: JobService):
|
||||||
|
"""Test deleting a job."""
|
||||||
|
|
||||||
|
class TestServiceMethods:
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_query_jobs(self, job_service: JobService):
|
||||||
|
"""Test querying jobs."""
|
||||||
|
await job_service.create_job(job=Job(document_id=uuid4(), status=JobStatus.PROCESSING))
|
||||||
|
result = await job_service.query_jobs(status=JobStatus.PROCESSING)
|
||||||
|
jobs = {str(job.id).split("-")[0]: job.status for job in result}
|
||||||
|
assert len(jobs) == 1
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_list_jobs(self, job_service: JobService):
|
||||||
|
"""Test listing jobs."""
|
||||||
|
n = 5
|
||||||
|
for _ in range(n):
|
||||||
|
await job_service.create_job(job=Job(document_id=uuid4()))
|
||||||
|
jobs = await job_service.list_jobs()
|
||||||
|
assert len(jobs) == n
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_mark_job_status(self, job_service: JobService):
|
||||||
|
"""Test marking a job with a new status."""
|
||||||
@@ -0,0 +1,36 @@
|
|||||||
|
import pytest
|
||||||
|
|
||||||
|
|
||||||
|
class TestServiceBase:
|
||||||
|
class TestInitialization:
|
||||||
|
def test_initializes_with_defaults(self):
|
||||||
|
"""Test initialization with default session factory and queue."""
|
||||||
|
|
||||||
|
def test_initializes_with_custom_session_factory(self):
|
||||||
|
"""Test initialization with a provided session factory."""
|
||||||
|
|
||||||
|
def test_initializes_with_custom_queue(self):
|
||||||
|
"""Test initialization with a provided queue."""
|
||||||
|
|
||||||
|
class TestSessionScope:
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_uses_provided_session(self):
|
||||||
|
"""Test that session scope reuses a provided session."""
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_creates_new_session_when_none_provided(self):
|
||||||
|
"""Test that session scope creates a new session when none is provided."""
|
||||||
|
|
||||||
|
class TestContextManagerBehavior:
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_yields_session(self):
|
||||||
|
"""Test that session scope yields a usable session object."""
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_multiple_operations(self):
|
||||||
|
"""Test multiple operations within a single session scope."""
|
||||||
|
|
||||||
|
class TestEdgeCases:
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_handles_exception_propagation(self):
|
||||||
|
"""Test exception propagation behavior inside session scope."""
|
||||||
@@ -1,137 +0,0 @@
|
|||||||
"""Tests for transcription.services.transcription."""
|
|
||||||
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
|
|
||||||
from transcription.config import Settings
|
|
||||||
from transcription.providers.base import ProviderError, TranscriptionResult
|
|
||||||
from transcription.services.transcription import (
|
|
||||||
PromptLoadError,
|
|
||||||
TranscriptionError,
|
|
||||||
load_image_payload,
|
|
||||||
load_prompt_text,
|
|
||||||
transcribe_document_image,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
class _FakeProvider:
|
|
||||||
def __init__(self, *, result: TranscriptionResult | None = None, error: Exception | None = None):
|
|
||||||
self._result = result or TranscriptionResult(
|
|
||||||
text="Transcript output",
|
|
||||||
provider="openrouter",
|
|
||||||
model="test-model",
|
|
||||||
)
|
|
||||||
self._error = error
|
|
||||||
self.calls: list[dict[str, object]] = []
|
|
||||||
|
|
||||||
def transcribe(self, *, prompt_text: str, image_bytes: bytes, mime_type: str) -> TranscriptionResult:
|
|
||||||
self.calls.append(
|
|
||||||
{
|
|
||||||
"prompt_text": prompt_text,
|
|
||||||
"image_bytes": image_bytes,
|
|
||||||
"mime_type": mime_type,
|
|
||||||
}
|
|
||||||
)
|
|
||||||
if self._error:
|
|
||||||
raise self._error
|
|
||||||
return self._result
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.unit
|
|
||||||
class TestPromptLoading:
|
|
||||||
"""Verify prompt artifact loading and validation."""
|
|
||||||
|
|
||||||
def test_loads_prompt_text_from_prompt_dir(self, tmp_path: Path):
|
|
||||||
"""Prompt loader returns canonical prompt text from configured prompt directory."""
|
|
||||||
prompt_dir = tmp_path / "prompts"
|
|
||||||
prompt_dir.mkdir()
|
|
||||||
prompt_file = prompt_dir / "transcribe_document.md"
|
|
||||||
prompt_file.write_text("Prompt body", encoding="utf-8")
|
|
||||||
|
|
||||||
settings = Settings(openrouter_api_key="test-key", prompt_dir=prompt_dir)
|
|
||||||
text = load_prompt_text(settings=settings)
|
|
||||||
|
|
||||||
assert text == "Prompt body"
|
|
||||||
|
|
||||||
def test_missing_prompt_raises_error(self, tmp_path: Path):
|
|
||||||
"""Prompt loader raises PromptLoadError when the file is missing."""
|
|
||||||
prompt_dir = tmp_path / "prompts"
|
|
||||||
prompt_dir.mkdir()
|
|
||||||
|
|
||||||
settings = Settings(openrouter_api_key="test-key", prompt_dir=prompt_dir)
|
|
||||||
with pytest.raises(PromptLoadError) as exc_info:
|
|
||||||
load_prompt_text(settings=settings)
|
|
||||||
|
|
||||||
assert exc_info.value.category.value == "infrastructure_persistent_error"
|
|
||||||
assert "verify prompt_dir" in exc_info.value.suggestion.lower()
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.unit
|
|
||||||
class TestImageLoading:
|
|
||||||
"""Verify local image payload loading and mime detection."""
|
|
||||||
|
|
||||||
def test_load_image_payload_reads_bytes_and_mime_type(self, tmp_path: Path):
|
|
||||||
"""Image loader returns file bytes and a detected MIME type for supported files."""
|
|
||||||
image_path = tmp_path / "sample.png"
|
|
||||||
image_bytes = b"\x89PNG\r\n\x1a\n"
|
|
||||||
image_path.write_bytes(image_bytes)
|
|
||||||
|
|
||||||
loaded_bytes, mime_type = load_image_payload(image_path)
|
|
||||||
|
|
||||||
assert loaded_bytes == image_bytes
|
|
||||||
assert mime_type == "image/png"
|
|
||||||
|
|
||||||
def test_missing_image_raises_error(self, tmp_path: Path):
|
|
||||||
"""Image loader raises TranscriptionError when image file does not exist."""
|
|
||||||
missing = tmp_path / "missing.png"
|
|
||||||
|
|
||||||
with pytest.raises(TranscriptionError) as exc_info:
|
|
||||||
load_image_payload(missing)
|
|
||||||
|
|
||||||
assert exc_info.value.category.value == "not_found_error"
|
|
||||||
assert "verify" in exc_info.value.suggestion.lower()
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.unit
|
|
||||||
class TestTranscriptionService:
|
|
||||||
"""Verify service orchestration across prompt, image, and provider calls."""
|
|
||||||
|
|
||||||
def test_transcribe_document_image_calls_provider_once(self, tmp_path: Path):
|
|
||||||
"""Service loads prompt and image, then invokes provider exactly once."""
|
|
||||||
prompt_dir = tmp_path / "prompts"
|
|
||||||
prompt_dir.mkdir()
|
|
||||||
(prompt_dir / "transcribe_document.md").write_text("Prompt body", encoding="utf-8")
|
|
||||||
|
|
||||||
image_path = tmp_path / "document.jpg"
|
|
||||||
image_path.write_bytes(b"jpeg-bytes")
|
|
||||||
|
|
||||||
settings = Settings(openrouter_api_key="test-key", prompt_dir=prompt_dir)
|
|
||||||
provider = _FakeProvider()
|
|
||||||
|
|
||||||
result = transcribe_document_image(image_path, settings=settings, provider=provider)
|
|
||||||
|
|
||||||
assert result.text == "Transcript output"
|
|
||||||
assert len(provider.calls) == 1
|
|
||||||
assert provider.calls[0]["prompt_text"] == "Prompt body"
|
|
||||||
assert provider.calls[0]["image_bytes"] == b"jpeg-bytes"
|
|
||||||
assert provider.calls[0]["mime_type"] == "image/jpeg"
|
|
||||||
|
|
||||||
def test_provider_error_is_wrapped(self, tmp_path: Path):
|
|
||||||
"""Service wraps provider failures in TranscriptionError."""
|
|
||||||
prompt_dir = tmp_path / "prompts"
|
|
||||||
prompt_dir.mkdir()
|
|
||||||
(prompt_dir / "transcribe_document.md").write_text("Prompt body", encoding="utf-8")
|
|
||||||
|
|
||||||
image_path = tmp_path / "document.png"
|
|
||||||
image_path.write_bytes(b"png-bytes")
|
|
||||||
|
|
||||||
settings = Settings(openrouter_api_key="test-key", prompt_dir=prompt_dir)
|
|
||||||
provider = _FakeProvider(error=ProviderError("upstream failure"))
|
|
||||||
|
|
||||||
with pytest.raises(TranscriptionError) as exc_info:
|
|
||||||
transcribe_document_image(image_path, settings=settings, provider=provider)
|
|
||||||
|
|
||||||
assert exc_info.value.category.value == "external_provider_error"
|
|
||||||
assert exc_info.value.retriable is True
|
|
||||||
assert "retry" in exc_info.value.suggestion.lower()
|
|
||||||
@@ -1,104 +0,0 @@
|
|||||||
"""Tests for transcription.services.upload."""
|
|
||||||
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
|
|
||||||
from transcription.config import Settings
|
|
||||||
from transcription.models import Document, Job, JobStatus
|
|
||||||
from transcription.services.upload import UploadError, create_upload_job
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.unit
|
|
||||||
class TestUploadValidation:
|
|
||||||
"""Verify upload validation behavior."""
|
|
||||||
|
|
||||||
def test_rejects_empty_bytes(self, session, tmp_path: Path):
|
|
||||||
"""create_upload_job rejects an empty upload payload."""
|
|
||||||
settings = Settings(openrouter_api_key="test-key", upload_dir=tmp_path)
|
|
||||||
with pytest.raises(UploadError) as exc_info:
|
|
||||||
create_upload_job(
|
|
||||||
filename="letter.jpg",
|
|
||||||
file_bytes=b"",
|
|
||||||
session=session,
|
|
||||||
settings=settings,
|
|
||||||
)
|
|
||||||
|
|
||||||
assert exc_info.value.category.value == "validation_error"
|
|
||||||
assert "non-empty" in exc_info.value.suggestion.lower()
|
|
||||||
|
|
||||||
def test_rejects_unsupported_extension(self, session, tmp_path: Path):
|
|
||||||
"""create_upload_job rejects unsupported filename extensions."""
|
|
||||||
settings = Settings(openrouter_api_key="test-key", upload_dir=tmp_path)
|
|
||||||
with pytest.raises(UploadError) as exc_info:
|
|
||||||
create_upload_job(
|
|
||||||
filename="notes.txt",
|
|
||||||
file_bytes=b"content",
|
|
||||||
session=session,
|
|
||||||
settings=settings,
|
|
||||||
)
|
|
||||||
|
|
||||||
assert exc_info.value.category.value == "user_input_error"
|
|
||||||
assert "jpg" in exc_info.value.suggestion.lower()
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.integration
|
|
||||||
class TestUploadPersistence:
|
|
||||||
"""Verify upload file and record persistence behavior."""
|
|
||||||
|
|
||||||
def test_writes_file_and_creates_records(self, session, tmp_path: Path):
|
|
||||||
"""create_upload_job writes file and creates document/job records."""
|
|
||||||
settings = Settings(openrouter_api_key="test-key", upload_dir=tmp_path)
|
|
||||||
|
|
||||||
result = create_upload_job(
|
|
||||||
filename="letter.jpg",
|
|
||||||
file_bytes=b"image-bytes",
|
|
||||||
session=session,
|
|
||||||
settings=settings,
|
|
||||||
)
|
|
||||||
|
|
||||||
assert result.stored_path.exists()
|
|
||||||
assert result.stored_path.read_bytes() == b"image-bytes"
|
|
||||||
|
|
||||||
document = session.get(Document, result.document_id)
|
|
||||||
job = session.get(Job, result.job_id)
|
|
||||||
assert document is not None
|
|
||||||
assert job is not None
|
|
||||||
assert document.filename == "letter.jpg"
|
|
||||||
assert document.file_path == str(result.stored_path)
|
|
||||||
|
|
||||||
def test_uses_unique_stored_filename(self, session, tmp_path: Path):
|
|
||||||
"""create_upload_job stores uploads with unique filenames."""
|
|
||||||
settings = Settings(openrouter_api_key="test-key", upload_dir=tmp_path)
|
|
||||||
|
|
||||||
first = create_upload_job(
|
|
||||||
filename="duplicate.jpg",
|
|
||||||
file_bytes=b"first",
|
|
||||||
session=session,
|
|
||||||
settings=settings,
|
|
||||||
)
|
|
||||||
second = create_upload_job(
|
|
||||||
filename="duplicate.jpg",
|
|
||||||
file_bytes=b"second",
|
|
||||||
session=session,
|
|
||||||
settings=settings,
|
|
||||||
)
|
|
||||||
|
|
||||||
assert first.stored_path != second.stored_path
|
|
||||||
assert first.stored_path.exists()
|
|
||||||
assert second.stored_path.exists()
|
|
||||||
|
|
||||||
def test_sets_job_status_queued(self, session, tmp_path: Path):
|
|
||||||
"""create_upload_job persists a job with queued status."""
|
|
||||||
settings = Settings(openrouter_api_key="test-key", upload_dir=tmp_path)
|
|
||||||
|
|
||||||
result = create_upload_job(
|
|
||||||
filename="queued.pdf",
|
|
||||||
file_bytes=b"%PDF-1.4",
|
|
||||||
session=session,
|
|
||||||
settings=settings,
|
|
||||||
)
|
|
||||||
|
|
||||||
job = session.get(Job, result.job_id)
|
|
||||||
assert job is not None
|
|
||||||
assert job.status == JobStatus.QUEUED
|
|
||||||
@@ -1,232 +0,0 @@
|
|||||||
"""Tests for transcription.worker."""
|
|
||||||
|
|
||||||
from pathlib import Path
|
|
||||||
from threading import Event
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
from sqlmodel import select
|
|
||||||
|
|
||||||
from transcription.config import Settings
|
|
||||||
from transcription.errors import AppError, ErrorCategory
|
|
||||||
from transcription.models import Document, Job, JobStatus, Transcript
|
|
||||||
from transcription.providers.base import TranscriptionResult
|
|
||||||
from transcription.worker import process_next_queued_job, run_worker_loop
|
|
||||||
|
|
||||||
|
|
||||||
def _create_queued_job(session, *, filename: str = "doc.jpg", file_path: str = "uploads/doc.jpg") -> Job:
|
|
||||||
document = Document(filename=filename, file_path=file_path)
|
|
||||||
session.add(document)
|
|
||||||
session.commit()
|
|
||||||
session.refresh(document)
|
|
||||||
|
|
||||||
job = Job(document_id=document.id, status=JobStatus.QUEUED)
|
|
||||||
session.add(job)
|
|
||||||
session.commit()
|
|
||||||
session.refresh(job)
|
|
||||||
return job
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.integration
|
|
||||||
class TestWorkerQueueBehavior:
|
|
||||||
"""Verify worker behavior when selecting queued jobs."""
|
|
||||||
|
|
||||||
def test_returns_false_when_queue_empty(self, session):
|
|
||||||
"""process_next_queued_job returns False when there are no queued jobs."""
|
|
||||||
processed = process_next_queued_job(session=session)
|
|
||||||
assert processed is False
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.integration
|
|
||||||
class TestWorkerSuccessPath:
|
|
||||||
"""Verify worker success-path lifecycle transitions and transcript persistence."""
|
|
||||||
|
|
||||||
def test_transitions_processing_to_transcribed(self, session, monkeypatch, tmp_path: Path):
|
|
||||||
"""process_next_queued_job transitions queued jobs to transcribed on success."""
|
|
||||||
job = _create_queued_job(session)
|
|
||||||
|
|
||||||
def _fake_transcribe(_path):
|
|
||||||
return TranscriptionResult(text="ok", provider="openrouter", model="test-model")
|
|
||||||
|
|
||||||
monkeypatch.setattr("transcription.worker.transcribe_document_image", _fake_transcribe)
|
|
||||||
monkeypatch.setattr(
|
|
||||||
"transcription.worker.get_settings",
|
|
||||||
lambda: Settings(openrouter_api_key="test-key", prompt_dir=tmp_path),
|
|
||||||
)
|
|
||||||
|
|
||||||
processed = process_next_queued_job(session=session)
|
|
||||||
session.refresh(job)
|
|
||||||
|
|
||||||
assert processed is True
|
|
||||||
assert job.status == JobStatus.TRANSCRIBED
|
|
||||||
|
|
||||||
def test_persists_transcript_text_on_success(self, session, monkeypatch, tmp_path: Path):
|
|
||||||
"""process_next_queued_job stores transcript text for successful jobs."""
|
|
||||||
job = _create_queued_job(session)
|
|
||||||
|
|
||||||
def _fake_transcribe(_path):
|
|
||||||
return TranscriptionResult(text="Transcript body", provider="openrouter", model="test-model")
|
|
||||||
|
|
||||||
monkeypatch.setattr("transcription.worker.transcribe_document_image", _fake_transcribe)
|
|
||||||
monkeypatch.setattr(
|
|
||||||
"transcription.worker.get_settings",
|
|
||||||
lambda: Settings(openrouter_api_key="test-key", prompt_dir=tmp_path),
|
|
||||||
)
|
|
||||||
|
|
||||||
process_next_queued_job(session=session)
|
|
||||||
|
|
||||||
transcript = session.exec(
|
|
||||||
select(Transcript).where(Transcript.job_id == job.id)
|
|
||||||
).first()
|
|
||||||
assert transcript is not None
|
|
||||||
assert transcript.text == "Transcript body"
|
|
||||||
assert transcript.error_detail is None
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.integration
|
|
||||||
class TestWorkerFailurePath:
|
|
||||||
"""Verify worker failure-path lifecycle transitions and error persistence."""
|
|
||||||
|
|
||||||
def test_sets_failed_and_error_detail_on_failure(self, session, monkeypatch, tmp_path: Path):
|
|
||||||
"""process_next_queued_job marks failed and stores error detail on exception."""
|
|
||||||
job = _create_queued_job(session)
|
|
||||||
|
|
||||||
def _fake_transcribe(_path):
|
|
||||||
raise RuntimeError("provider failure")
|
|
||||||
|
|
||||||
monkeypatch.setattr("transcription.worker.transcribe_document_image", _fake_transcribe)
|
|
||||||
monkeypatch.setattr(
|
|
||||||
"transcription.worker.get_settings",
|
|
||||||
lambda: Settings(openrouter_api_key="test-key", prompt_dir=tmp_path),
|
|
||||||
)
|
|
||||||
|
|
||||||
processed = process_next_queued_job(session=session)
|
|
||||||
session.refresh(job)
|
|
||||||
transcript = session.exec(
|
|
||||||
select(Transcript).where(Transcript.job_id == job.id)
|
|
||||||
).first()
|
|
||||||
|
|
||||||
assert processed is True
|
|
||||||
assert job.status == JobStatus.FAILED
|
|
||||||
assert transcript is not None
|
|
||||||
assert transcript.text is None
|
|
||||||
assert "provider failure" in transcript.error_detail
|
|
||||||
assert "[internal_unexpected_error]" in transcript.error_detail
|
|
||||||
assert "error_id=" in transcript.error_detail
|
|
||||||
assert "suggestion=" in transcript.error_detail
|
|
||||||
|
|
||||||
def test_updates_existing_transcript_if_present(self, session, monkeypatch, tmp_path: Path):
|
|
||||||
"""process_next_queued_job updates existing transcript instead of duplicating."""
|
|
||||||
job = _create_queued_job(session)
|
|
||||||
existing = Transcript(job_id=job.id, text="old", error_detail=None)
|
|
||||||
session.add(existing)
|
|
||||||
session.commit()
|
|
||||||
session.refresh(existing)
|
|
||||||
|
|
||||||
def _fake_transcribe(_path):
|
|
||||||
raise RuntimeError("provider failure")
|
|
||||||
|
|
||||||
monkeypatch.setattr("transcription.worker.transcribe_document_image", _fake_transcribe)
|
|
||||||
monkeypatch.setattr(
|
|
||||||
"transcription.worker.get_settings",
|
|
||||||
lambda: Settings(openrouter_api_key="test-key", prompt_dir=tmp_path),
|
|
||||||
)
|
|
||||||
|
|
||||||
process_next_queued_job(session=session)
|
|
||||||
|
|
||||||
transcripts = session.exec(
|
|
||||||
select(Transcript).where(Transcript.job_id == job.id)
|
|
||||||
).all()
|
|
||||||
assert len(transcripts) == 1
|
|
||||||
assert transcripts[0].id == existing.id
|
|
||||||
assert transcripts[0].text is None
|
|
||||||
assert "provider failure" in transcripts[0].error_detail
|
|
||||||
assert "[internal_unexpected_error]" in transcripts[0].error_detail
|
|
||||||
assert "error_id=" in transcripts[0].error_detail
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.integration
|
|
||||||
class TestWorkerRetryBehavior:
|
|
||||||
"""Verify worker retry and terminal failure policies."""
|
|
||||||
|
|
||||||
def test_retriable_failure_requeues_until_limit(self, session, monkeypatch, tmp_path: Path):
|
|
||||||
"""Retriable failures requeue jobs while retry budget remains."""
|
|
||||||
job = _create_queued_job(session)
|
|
||||||
|
|
||||||
def _fake_transcribe(_path):
|
|
||||||
raise AppError(
|
|
||||||
"temporary upstream outage",
|
|
||||||
category=ErrorCategory.EXTERNAL_PROVIDER,
|
|
||||||
suggestion="Retry from jobs page.",
|
|
||||||
retriable=True,
|
|
||||||
)
|
|
||||||
|
|
||||||
monkeypatch.setattr("transcription.worker.transcribe_document_image", _fake_transcribe)
|
|
||||||
monkeypatch.setattr(
|
|
||||||
"transcription.worker.get_settings",
|
|
||||||
lambda: Settings(
|
|
||||||
openrouter_api_key="test-key",
|
|
||||||
prompt_dir=tmp_path,
|
|
||||||
worker_max_retries=1,
|
|
||||||
worker_retry_backoff_seconds=0.0,
|
|
||||||
),
|
|
||||||
)
|
|
||||||
|
|
||||||
processed = process_next_queued_job(session=session)
|
|
||||||
session.refresh(job)
|
|
||||||
|
|
||||||
assert processed is True
|
|
||||||
assert job.status == JobStatus.QUEUED
|
|
||||||
assert job.retry_count == 1
|
|
||||||
|
|
||||||
def test_retriable_failure_exhaustion_sets_failed(self, session, monkeypatch, tmp_path: Path):
|
|
||||||
"""Retriable failures transition to failed when retry budget is exhausted."""
|
|
||||||
job = _create_queued_job(session)
|
|
||||||
job.retry_count = 1
|
|
||||||
session.add(job)
|
|
||||||
session.commit()
|
|
||||||
|
|
||||||
def _fake_transcribe(_path):
|
|
||||||
raise AppError(
|
|
||||||
"temporary upstream outage",
|
|
||||||
category=ErrorCategory.EXTERNAL_PROVIDER,
|
|
||||||
suggestion="Retry from jobs page.",
|
|
||||||
retriable=True,
|
|
||||||
)
|
|
||||||
|
|
||||||
monkeypatch.setattr("transcription.worker.transcribe_document_image", _fake_transcribe)
|
|
||||||
monkeypatch.setattr(
|
|
||||||
"transcription.worker.get_settings",
|
|
||||||
lambda: Settings(
|
|
||||||
openrouter_api_key="test-key",
|
|
||||||
prompt_dir=tmp_path,
|
|
||||||
worker_max_retries=1,
|
|
||||||
worker_retry_backoff_seconds=0.0,
|
|
||||||
),
|
|
||||||
)
|
|
||||||
|
|
||||||
process_next_queued_job(session=session)
|
|
||||||
session.refresh(job)
|
|
||||||
|
|
||||||
assert job.status == JobStatus.FAILED
|
|
||||||
assert job.retry_count == 1
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.unit
|
|
||||||
class TestWorkerLoopControl:
|
|
||||||
"""Verify worker loop start/stop behavior."""
|
|
||||||
|
|
||||||
def test_stops_when_stop_event_is_set(self, monkeypatch):
|
|
||||||
"""run_worker_loop exits when a stop event is set."""
|
|
||||||
stop_event = Event()
|
|
||||||
stop_event.set()
|
|
||||||
|
|
||||||
called = {"value": False}
|
|
||||||
|
|
||||||
def _fake_process_next_queued_job(**_kwargs):
|
|
||||||
called["value"] = True
|
|
||||||
return False
|
|
||||||
|
|
||||||
monkeypatch.setattr("transcription.worker.process_next_queued_job", _fake_process_next_queued_job)
|
|
||||||
|
|
||||||
run_worker_loop(stop_event=stop_event, poll_interval_seconds=0.01)
|
|
||||||
assert called["value"] is False
|
|
||||||
@@ -1,96 +1,71 @@
|
|||||||
"""Tests for transcription.db — schema bootstrap and session factory."""
|
"""Tests for transcription.db — async schema bootstrap/runtime behavior."""
|
||||||
|
|
||||||
from sqlalchemy import inspect, text
|
from sqlalchemy import inspect
|
||||||
from sqlmodel import Session, SQLModel, create_engine
|
from sqlalchemy import text
|
||||||
from sqlmodel.pool import StaticPool
|
import pytest
|
||||||
|
|
||||||
|
|
||||||
def _in_memory_engine():
|
|
||||||
"""Create a fresh in-memory SQLite engine for isolated db tests."""
|
|
||||||
return create_engine(
|
|
||||||
"sqlite://",
|
|
||||||
connect_args={"check_same_thread": False},
|
|
||||||
poolclass=StaticPool,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
class TestSchemaBootstrap:
|
class TestSchemaBootstrap:
|
||||||
"""Verify create_all produces the expected table set."""
|
"""Verify async create_all produces the expected table set."""
|
||||||
|
|
||||||
def test_create_all_creates_expected_tables(self):
|
@pytest.mark.asyncio
|
||||||
"""After create_all(), document, job, and transcript tables exist."""
|
async def test_create_all_creates_expected_tables(self, default_settings):
|
||||||
engine = _in_memory_engine()
|
"""After async create_all(), document/job/transcript/revision tables exist."""
|
||||||
# Ensure models are imported so metadata is populated
|
# Ensure models are imported so metadata is populated.
|
||||||
from transcription.models import Document, Job, Transcript # noqa: F401
|
from transcription.models import Document, Job, Transcript, TranscriptRevision # noqa: F401
|
||||||
|
|
||||||
import transcription.db as db_module
|
from transcription.db.operations import create_all
|
||||||
|
from transcription.db.runtime import get_engine
|
||||||
|
|
||||||
db_module.create_all(engine=engine)
|
engine = get_engine(settings=default_settings)
|
||||||
|
await create_all(engine=engine)
|
||||||
|
|
||||||
|
async with engine.begin() as connection:
|
||||||
|
table_names = set(await connection.run_sync(lambda sync_conn: inspect(sync_conn).get_table_names()))
|
||||||
|
|
||||||
inspector = inspect(engine)
|
|
||||||
table_names = set(inspector.get_table_names())
|
|
||||||
assert "document" in table_names
|
assert "document" in table_names
|
||||||
assert "job" in table_names
|
assert "job" in table_names
|
||||||
assert "transcript" in table_names
|
assert "transcript" in table_names
|
||||||
|
assert "transcriptrevision" in table_names
|
||||||
|
|
||||||
|
|
||||||
class TestSessionFactory:
|
class TestSessionFactory:
|
||||||
"""Verify get_session yields and cleans up sessions."""
|
"""Verify async get_session yields a usable AsyncSession."""
|
||||||
|
|
||||||
def test_get_session_yields_session(self):
|
@pytest.mark.asyncio
|
||||||
"""get_session() yields a usable Session object."""
|
async def test_get_session_yields_session(self, default_settings):
|
||||||
engine = _in_memory_engine()
|
"""get_session() yields an AsyncSession with a live connection."""
|
||||||
SQLModel.metadata.create_all(engine)
|
from transcription.db.runtime import get_session
|
||||||
|
|
||||||
import transcription.db as db_module
|
async with get_session(settings=default_settings) as session:
|
||||||
|
result = await session.exec(text("SELECT 1"))
|
||||||
with db_module.get_session(engine=engine) as session:
|
assert result.first()[0] == 1
|
||||||
assert isinstance(session, Session)
|
|
||||||
|
|
||||||
def test_session_is_closed_after_generator_exit(self):
|
|
||||||
"""After the context manager exits, the session is closed."""
|
|
||||||
engine = _in_memory_engine()
|
|
||||||
SQLModel.metadata.create_all(engine)
|
|
||||||
|
|
||||||
import transcription.db as db_module
|
|
||||||
|
|
||||||
with db_module.get_session(engine=engine) as session:
|
|
||||||
# Session is usable inside the context
|
|
||||||
session.execute(text("SELECT 1"))
|
|
||||||
captured = session
|
|
||||||
|
|
||||||
# After exiting, the session's internal connection is released
|
|
||||||
# (no active transaction bound to the session)
|
|
||||||
assert captured._transaction is None
|
|
||||||
|
|
||||||
|
|
||||||
class TestBootstrapPolicy:
|
class TestBootstrapPolicy:
|
||||||
"""Verify schema bootstrap policy defaults and overrides."""
|
"""Verify startup schema bootstrap policy via Settings property."""
|
||||||
|
|
||||||
def test_production_defaults_to_no_bootstrap(self):
|
def test_production_defaults_to_no_bootstrap(self):
|
||||||
"""Production defaults to explicit non-bootstrap startup behavior."""
|
"""Production defaults to explicit non-bootstrap startup behavior."""
|
||||||
from transcription.config import Settings
|
from transcription.config import Settings
|
||||||
from transcription.db import should_bootstrap_schema
|
|
||||||
|
|
||||||
settings = Settings(openrouter_api_key="test-key", environment="production")
|
settings = Settings(openrouter_api_key="test-key", environment="production")
|
||||||
assert should_bootstrap_schema(settings) is False
|
assert settings.should_bootstrap_schema is False
|
||||||
|
|
||||||
def test_development_defaults_to_bootstrap(self):
|
def test_development_defaults_to_bootstrap(self):
|
||||||
"""Development defaults to schema bootstrap for local workflows."""
|
"""Development defaults to schema bootstrap for local workflows."""
|
||||||
from transcription.config import Settings
|
from transcription.config import Settings
|
||||||
from transcription.db import should_bootstrap_schema
|
|
||||||
|
|
||||||
settings = Settings(openrouter_api_key="test-key", environment="development")
|
settings = Settings(openrouter_api_key="test-key", environment="development")
|
||||||
assert should_bootstrap_schema(settings) is True
|
assert settings.should_bootstrap_schema is True
|
||||||
|
|
||||||
def test_explicit_override_wins(self):
|
def test_explicit_override_wins(self):
|
||||||
"""Explicit bootstrap_schema_on_startup overrides environment default."""
|
"""Explicit bootstrap_schema_on_startup overrides environment default."""
|
||||||
from transcription.config import Settings
|
from transcription.config import Settings
|
||||||
from transcription.db import should_bootstrap_schema
|
|
||||||
|
|
||||||
settings = Settings(
|
settings = Settings(
|
||||||
openrouter_api_key="test-key",
|
openrouter_api_key="test-key",
|
||||||
environment="production",
|
environment="production",
|
||||||
bootstrap_schema_on_startup=True,
|
bootstrap_schema_on_startup=True,
|
||||||
)
|
)
|
||||||
assert should_bootstrap_schema(settings) is True
|
assert settings.should_bootstrap_schema is True
|
||||||
|
|||||||
@@ -1,11 +1,11 @@
|
|||||||
"""Tests for transcription.models — Document, Job, Transcript persistence and relationships."""
|
"""Tests for transcription.models — Document, Job, Transcript, TranscriptRevision models."""
|
||||||
|
|
||||||
from uuid import UUID
|
from uuid import UUID
|
||||||
|
|
||||||
import pytest
|
import pytest
|
||||||
from sqlalchemy.exc import IntegrityError
|
from sqlalchemy.exc import IntegrityError
|
||||||
|
|
||||||
from transcription.models import Document, Job, JobStatus, Transcript
|
from transcription.models import Document, Job, JobStatus, Transcript, TranscriptRevision
|
||||||
|
|
||||||
|
|
||||||
def _make_document(**overrides) -> Document:
|
def _make_document(**overrides) -> Document:
|
||||||
@@ -113,7 +113,7 @@ class TestTranscriptModel:
|
|||||||
"""A Transcript with text set and error_detail None persists correctly."""
|
"""A Transcript with text set and error_detail None persists correctly."""
|
||||||
doc = _persist_document(session)
|
doc = _persist_document(session)
|
||||||
job = _persist_job(session, doc)
|
job = _persist_job(session, doc)
|
||||||
transcript = Transcript(job_id=job.id, text="Dear Sir, ...")
|
transcript = Transcript(job_id=job.id, provider="openrouter", prompt_name="transcribe_document.md", text="Dear Sir, ...")
|
||||||
session.add(transcript)
|
session.add(transcript)
|
||||||
session.commit()
|
session.commit()
|
||||||
session.refresh(transcript)
|
session.refresh(transcript)
|
||||||
@@ -127,7 +127,12 @@ class TestTranscriptModel:
|
|||||||
"""A Transcript with text None and error_detail set persists correctly."""
|
"""A Transcript with text None and error_detail set persists correctly."""
|
||||||
doc = _persist_document(session)
|
doc = _persist_document(session)
|
||||||
job = _persist_job(session, doc)
|
job = _persist_job(session, doc)
|
||||||
transcript = Transcript(job_id=job.id, error_detail="Provider timeout")
|
transcript = Transcript(
|
||||||
|
job_id=job.id,
|
||||||
|
provider="openrouter",
|
||||||
|
prompt_name="transcribe_document.md",
|
||||||
|
error_detail="Provider timeout",
|
||||||
|
)
|
||||||
session.add(transcript)
|
session.add(transcript)
|
||||||
session.commit()
|
session.commit()
|
||||||
session.refresh(transcript)
|
session.refresh(transcript)
|
||||||
@@ -142,16 +147,101 @@ class TestTranscriptModel:
|
|||||||
doc = _persist_document(session)
|
doc = _persist_document(session)
|
||||||
job = _persist_job(session, doc)
|
job = _persist_job(session, doc)
|
||||||
|
|
||||||
t1 = Transcript(job_id=job.id, text="First")
|
t1 = Transcript(job_id=job.id, provider="openrouter", prompt_name="transcribe_document.md", text="First")
|
||||||
session.add(t1)
|
session.add(t1)
|
||||||
session.commit()
|
session.commit()
|
||||||
|
|
||||||
t2 = Transcript(job_id=job.id, text="Duplicate")
|
t2 = Transcript(job_id=job.id, provider="openrouter", prompt_name="transcribe_document.md", text="Duplicate")
|
||||||
session.add(t2)
|
session.add(t2)
|
||||||
with pytest.raises(IntegrityError):
|
with pytest.raises(IntegrityError):
|
||||||
session.commit()
|
session.commit()
|
||||||
|
|
||||||
|
|
||||||
|
class TestTranscriptRevisionModel:
|
||||||
|
"""Verify TranscriptRevision persistence and version uniqueness constraints."""
|
||||||
|
|
||||||
|
def test_revision_record_persists(self, session):
|
||||||
|
"""A TranscriptRevision with version metadata persists correctly."""
|
||||||
|
doc = _persist_document(session)
|
||||||
|
job = _persist_job(session, doc)
|
||||||
|
revision = TranscriptRevision(
|
||||||
|
job_id=job.id,
|
||||||
|
version_number=1,
|
||||||
|
provider="openrouter",
|
||||||
|
prompt_name="transcribe_document.md",
|
||||||
|
model="google/gemini-2.5-flash",
|
||||||
|
source="ai",
|
||||||
|
text="Initial text",
|
||||||
|
)
|
||||||
|
session.add(revision)
|
||||||
|
session.commit()
|
||||||
|
session.refresh(revision)
|
||||||
|
|
||||||
|
fetched = session.get(TranscriptRevision, revision.id)
|
||||||
|
assert fetched is not None
|
||||||
|
assert fetched.version_number == 1
|
||||||
|
assert fetched.text == "Initial text"
|
||||||
|
assert fetched.source == "ai"
|
||||||
|
|
||||||
|
def test_job_version_pair_is_unique(self, session):
|
||||||
|
"""Duplicate version_number for same job raises integrity error."""
|
||||||
|
doc = _persist_document(session)
|
||||||
|
job = _persist_job(session, doc)
|
||||||
|
|
||||||
|
first = TranscriptRevision(
|
||||||
|
job_id=job.id,
|
||||||
|
version_number=1,
|
||||||
|
provider="openrouter",
|
||||||
|
prompt_name="transcribe_document.md",
|
||||||
|
source="ai",
|
||||||
|
text="Initial",
|
||||||
|
)
|
||||||
|
duplicate = TranscriptRevision(
|
||||||
|
job_id=job.id,
|
||||||
|
version_number=1,
|
||||||
|
provider="openrouter",
|
||||||
|
prompt_name="transcribe_document.md",
|
||||||
|
source="user",
|
||||||
|
text="Edited",
|
||||||
|
)
|
||||||
|
session.add(first)
|
||||||
|
session.commit()
|
||||||
|
|
||||||
|
session.add(duplicate)
|
||||||
|
with pytest.raises(IntegrityError):
|
||||||
|
session.commit()
|
||||||
|
|
||||||
|
def test_same_version_number_allowed_for_different_jobs(self, session):
|
||||||
|
"""Version numbers are scoped per job, not globally."""
|
||||||
|
doc1 = _persist_document(session)
|
||||||
|
job1 = _persist_job(session, doc1)
|
||||||
|
doc2 = _make_document(filename="letter2.jpg", file_path="/uploads/letter2.jpg")
|
||||||
|
session.add(doc2)
|
||||||
|
session.commit()
|
||||||
|
session.refresh(doc2)
|
||||||
|
job2 = _persist_job(session, doc2)
|
||||||
|
|
||||||
|
r1 = TranscriptRevision(
|
||||||
|
job_id=job1.id,
|
||||||
|
version_number=1,
|
||||||
|
provider="openrouter",
|
||||||
|
prompt_name="transcribe_document.md",
|
||||||
|
source="ai",
|
||||||
|
text="Job1 v1",
|
||||||
|
)
|
||||||
|
r2 = TranscriptRevision(
|
||||||
|
job_id=job2.id,
|
||||||
|
version_number=1,
|
||||||
|
provider="openrouter",
|
||||||
|
prompt_name="transcribe_document.md",
|
||||||
|
source="ai",
|
||||||
|
text="Job2 v1",
|
||||||
|
)
|
||||||
|
session.add(r1)
|
||||||
|
session.add(r2)
|
||||||
|
session.commit()
|
||||||
|
|
||||||
|
|
||||||
class TestRelationships:
|
class TestRelationships:
|
||||||
"""Verify SQLModel relationship navigation between models."""
|
"""Verify SQLModel relationship navigation between models."""
|
||||||
|
|
||||||
@@ -169,7 +259,12 @@ class TestRelationships:
|
|||||||
"""job.transcript returns the linked Transcript."""
|
"""job.transcript returns the linked Transcript."""
|
||||||
doc = _persist_document(session)
|
doc = _persist_document(session)
|
||||||
job = _persist_job(session, doc)
|
job = _persist_job(session, doc)
|
||||||
transcript = Transcript(job_id=job.id, text="Transcribed text")
|
transcript = Transcript(
|
||||||
|
job_id=job.id,
|
||||||
|
provider="openrouter",
|
||||||
|
prompt_name="transcribe_document.md",
|
||||||
|
text="Transcribed text",
|
||||||
|
)
|
||||||
session.add(transcript)
|
session.add(transcript)
|
||||||
session.commit()
|
session.commit()
|
||||||
|
|
||||||
@@ -177,3 +272,32 @@ class TestRelationships:
|
|||||||
assert job.transcript is not None
|
assert job.transcript is not None
|
||||||
assert isinstance(job.transcript, Transcript)
|
assert isinstance(job.transcript, Transcript)
|
||||||
assert job.transcript.text == "Transcribed text"
|
assert job.transcript.text == "Transcribed text"
|
||||||
|
|
||||||
|
def test_job_exposes_transcript_revisions(self, session):
|
||||||
|
"""job.transcript_revisions returns all linked revisions."""
|
||||||
|
doc = _persist_document(session)
|
||||||
|
job = _persist_job(session, doc)
|
||||||
|
session.add(
|
||||||
|
TranscriptRevision(
|
||||||
|
job_id=job.id,
|
||||||
|
version_number=1,
|
||||||
|
provider="openrouter",
|
||||||
|
prompt_name="transcribe_document.md",
|
||||||
|
source="ai",
|
||||||
|
text="v1",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
session.add(
|
||||||
|
TranscriptRevision(
|
||||||
|
job_id=job.id,
|
||||||
|
version_number=2,
|
||||||
|
provider="openrouter",
|
||||||
|
prompt_name="transcribe_document.md",
|
||||||
|
source="user",
|
||||||
|
text="v2",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
session.commit()
|
||||||
|
|
||||||
|
session.refresh(job)
|
||||||
|
assert len(job.transcript_revisions) == 2
|
||||||
|
|||||||
@@ -1,95 +1,37 @@
|
|||||||
"""Tests for transcription.ui.jobs_page."""
|
"""Tests for the jobs page route."""
|
||||||
|
|
||||||
from uuid import uuid4
|
|
||||||
|
|
||||||
import pytest
|
import pytest
|
||||||
|
from fastapi import FastAPI
|
||||||
|
from fastapi.testclient import TestClient
|
||||||
|
|
||||||
from transcription.models import Document, Job, Transcript
|
from transcription.ui import register_pages
|
||||||
from transcription.ui.jobs_page import fetch_job_detail, fetch_jobs
|
from transcription.ui.pages import jobs_page
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def client(monkeypatch):
|
||||||
|
"""Provide a minimal app client with jobs data patched for rendering."""
|
||||||
|
|
||||||
|
async def _fetch_jobs_stub():
|
||||||
|
return []
|
||||||
|
|
||||||
|
monkeypatch.setattr(jobs_page, "fetch_jobs", _fetch_jobs_stub)
|
||||||
|
|
||||||
|
app = FastAPI()
|
||||||
|
register_pages(app)
|
||||||
|
with TestClient(app) as test_client:
|
||||||
|
yield test_client
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.integration
|
@pytest.mark.integration
|
||||||
class TestJobsListBehavior:
|
class TestPageRendering:
|
||||||
"""Verify job list data and rendering helpers."""
|
"""Verify the jobs page is available and includes the main controls."""
|
||||||
|
|
||||||
def test_fetch_jobs_returns_job_view_rows(self, session, monkeypatch):
|
def test_jobs_page_renders_expected_controls(self, client):
|
||||||
"""fetch_jobs returns normalized JobView rows for UI consumption."""
|
"""GET /ui/jobs returns the page shell and jobs controls."""
|
||||||
document = Document(filename="letter.jpg", file_path="uploads/letter.jpg")
|
response = client.get("/ui/jobs")
|
||||||
session.add(document)
|
|
||||||
session.commit()
|
|
||||||
session.refresh(document)
|
|
||||||
|
|
||||||
job = Job(document_id=document.id)
|
assert response.status_code == 200
|
||||||
session.add(job)
|
assert "Transcription Jobs" in response.text
|
||||||
session.commit()
|
assert "Refresh" in response.text
|
||||||
|
assert "Back to upload" in response.text
|
||||||
class _SessionContext:
|
|
||||||
def __enter__(self):
|
|
||||||
return session
|
|
||||||
|
|
||||||
def __exit__(self, exc_type, exc, tb):
|
|
||||||
return False
|
|
||||||
|
|
||||||
monkeypatch.setattr("transcription.ui.jobs_page.get_session", lambda: _SessionContext())
|
|
||||||
|
|
||||||
rows = fetch_jobs()
|
|
||||||
|
|
||||||
assert len(rows) == 1
|
|
||||||
assert rows[0].id == job.id
|
|
||||||
assert rows[0].status == "queued"
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.integration
|
|
||||||
class TestJobDetailBehavior:
|
|
||||||
"""Verify job detail retrieval behavior."""
|
|
||||||
|
|
||||||
def test_fetch_job_detail_returns_related_records_when_present(self, session, monkeypatch):
|
|
||||||
"""fetch_job_detail returns job, document, and transcript when available."""
|
|
||||||
document = Document(filename="typed.jpg", file_path="uploads/typed.jpg")
|
|
||||||
session.add(document)
|
|
||||||
session.commit()
|
|
||||||
session.refresh(document)
|
|
||||||
|
|
||||||
job = Job(document_id=document.id)
|
|
||||||
session.add(job)
|
|
||||||
session.commit()
|
|
||||||
session.refresh(job)
|
|
||||||
|
|
||||||
transcript = Transcript(job_id=job.id, text="Transcript text")
|
|
||||||
session.add(transcript)
|
|
||||||
session.commit()
|
|
||||||
|
|
||||||
class _SessionContext:
|
|
||||||
def __enter__(self):
|
|
||||||
return session
|
|
||||||
|
|
||||||
def __exit__(self, exc_type, exc, tb):
|
|
||||||
return False
|
|
||||||
|
|
||||||
monkeypatch.setattr("transcription.ui.jobs_page.get_session", lambda: _SessionContext())
|
|
||||||
|
|
||||||
fetched_job, fetched_document, fetched_transcript = fetch_job_detail(job.id)
|
|
||||||
|
|
||||||
assert fetched_job is not None
|
|
||||||
assert fetched_document is not None
|
|
||||||
assert fetched_transcript is not None
|
|
||||||
assert fetched_job.id == job.id
|
|
||||||
assert fetched_document.id == document.id
|
|
||||||
assert fetched_transcript.job_id == job.id
|
|
||||||
|
|
||||||
def test_fetch_job_detail_returns_nones_for_missing_job(self, session, monkeypatch):
|
|
||||||
"""fetch_job_detail returns triple None when job does not exist."""
|
|
||||||
class _SessionContext:
|
|
||||||
def __enter__(self):
|
|
||||||
return session
|
|
||||||
|
|
||||||
def __exit__(self, exc_type, exc, tb):
|
|
||||||
return False
|
|
||||||
|
|
||||||
monkeypatch.setattr("transcription.ui.jobs_page.get_session", lambda: _SessionContext())
|
|
||||||
|
|
||||||
fetched_job, fetched_document, fetched_transcript = fetch_job_detail(uuid4())
|
|
||||||
|
|
||||||
assert fetched_job is None
|
|
||||||
assert fetched_document is None
|
|
||||||
assert fetched_transcript is None
|
|
||||||
|
|||||||
@@ -1,8 +1,7 @@
|
|||||||
"""Tests for UI page registration wiring."""
|
"""Tests for UI page registration wiring."""
|
||||||
|
|
||||||
from fastapi import FastAPI
|
|
||||||
from fastapi.testclient import TestClient
|
|
||||||
import pytest
|
import pytest
|
||||||
|
from fastapi import FastAPI
|
||||||
|
|
||||||
from transcription.ui import register_pages
|
from transcription.ui import register_pages
|
||||||
|
|
||||||
@@ -11,16 +10,29 @@ from transcription.ui import register_pages
|
|||||||
class TestPageRegistration:
|
class TestPageRegistration:
|
||||||
"""Verify page registration and route wiring."""
|
"""Verify page registration and route wiring."""
|
||||||
|
|
||||||
def test_register_pages_adds_expected_routes(self):
|
def test_register_pages_wires_upload_jobs_and_mount(self, monkeypatch):
|
||||||
"""register_pages wires upload and jobs routes into the app."""
|
"""register_pages registers pages and mounts NiceGUI at /ui."""
|
||||||
|
calls: list[str] = []
|
||||||
|
|
||||||
|
def _record_upload() -> None:
|
||||||
|
calls.append("upload")
|
||||||
|
|
||||||
|
def _record_jobs() -> None:
|
||||||
|
calls.append("jobs")
|
||||||
|
|
||||||
|
def _record_run_with(
|
||||||
|
_app: FastAPI,
|
||||||
|
*,
|
||||||
|
mount_path: str,
|
||||||
|
show_welcome_message: bool,
|
||||||
|
) -> None:
|
||||||
|
calls.append(f"run_with:{mount_path}:{show_welcome_message}")
|
||||||
|
|
||||||
|
monkeypatch.setattr("transcription.ui.register_upload_page", _record_upload)
|
||||||
|
monkeypatch.setattr("transcription.ui.register_jobs_page", _record_jobs)
|
||||||
|
monkeypatch.setattr("transcription.ui.ui.run_with", _record_run_with)
|
||||||
|
|
||||||
app = FastAPI()
|
app = FastAPI()
|
||||||
register_pages(app)
|
register_pages(app)
|
||||||
app.add_api_route("/healthz", lambda: {"status": "ok"}, methods=["GET"])
|
|
||||||
|
|
||||||
client = TestClient(app)
|
assert calls == ["upload", "jobs", "run_with:/ui:False"]
|
||||||
ui_response = client.get("/ui")
|
|
||||||
health_response = client.get("/healthz")
|
|
||||||
|
|
||||||
assert ui_response.status_code == 200
|
|
||||||
assert health_response.status_code == 200
|
|
||||||
assert health_response.json() == {"status": "ok"}
|
|
||||||
|
|||||||
@@ -1,53 +1,55 @@
|
|||||||
"""Tests for transcription.ui.upload_page."""
|
"""Tests for the upload page route."""
|
||||||
|
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from uuid import uuid4
|
|
||||||
|
|
||||||
import pytest
|
import pytest
|
||||||
|
from fastapi.testclient import TestClient
|
||||||
|
|
||||||
from transcription.services.upload import UploadError, UploadJobResult
|
from transcription.app import create_app
|
||||||
from transcription.ui import upload_page
|
from transcription.config import Settings
|
||||||
|
from transcription.config import _settings
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.unit
|
@pytest.fixture
|
||||||
class TestUploadPageBehavior:
|
def client(tmp_path: Path):
|
||||||
"""Verify upload page helper and submission behavior."""
|
"""Provide a real app client backed by in-memory SQLite."""
|
||||||
|
settings = Settings(
|
||||||
|
openrouter_api_key="test-key",
|
||||||
|
database_url="sqlite:///:memory:",
|
||||||
|
environment="test",
|
||||||
|
upload_dir=tmp_path / "uploads",
|
||||||
|
prompt_dir=tmp_path / "prompts",
|
||||||
|
)
|
||||||
|
_settings.set(settings)
|
||||||
|
|
||||||
def test_accepted_upload_types_contains_supported_extensions(self):
|
app = create_app()
|
||||||
"""accepted_upload_types includes all MVP-supported upload extensions."""
|
with TestClient(app) as test_client:
|
||||||
accepted = upload_page.accepted_upload_types()
|
yield test_client
|
||||||
assert ".jpg" in accepted
|
|
||||||
assert ".jpeg" in accepted
|
|
||||||
assert ".png" in accepted
|
|
||||||
assert ".tif" in accepted
|
|
||||||
assert ".tiff" in accepted
|
|
||||||
assert ".pdf" in accepted
|
|
||||||
|
|
||||||
def test_submit_upload_calls_upload_service(self, monkeypatch):
|
|
||||||
"""submit_upload delegates file persistence and job creation to upload service."""
|
|
||||||
expected = UploadJobResult(
|
|
||||||
document_id=uuid4(),
|
|
||||||
job_id=uuid4(),
|
|
||||||
stored_path=Path("uploads/mock.jpg"),
|
|
||||||
original_filename="mock.jpg",
|
|
||||||
)
|
|
||||||
|
|
||||||
def fake_create_upload_job(*, filename: str, file_bytes: bytes):
|
@pytest.mark.integration
|
||||||
assert filename == "mock.jpg"
|
class TestPageRendering:
|
||||||
assert file_bytes == b"bytes"
|
"""Verify the upload page is available and includes the main controls."""
|
||||||
return expected
|
|
||||||
|
|
||||||
monkeypatch.setattr(upload_page, "create_upload_job", fake_create_upload_job)
|
def test_root_redirects_to_ui(self, client):
|
||||||
|
"""GET / redirects to the UI mount point."""
|
||||||
|
response = client.get("/", follow_redirects=False)
|
||||||
|
|
||||||
result = upload_page.submit_upload(filename="mock.jpg", file_bytes=b"bytes")
|
assert response.status_code == 307
|
||||||
assert result == expected
|
assert response.headers["location"] == "/ui"
|
||||||
|
|
||||||
def test_submit_upload_surfaces_upload_error(self, monkeypatch):
|
def test_ui_redirects_to_upload(self, client):
|
||||||
"""submit_upload raises UploadError for invalid upload payloads."""
|
"""GET /ui redirects to the upload page."""
|
||||||
def fake_create_upload_job(*, filename: str, file_bytes: bytes):
|
response = client.get("/ui", follow_redirects=False)
|
||||||
raise UploadError("invalid payload")
|
|
||||||
|
|
||||||
monkeypatch.setattr(upload_page, "create_upload_job", fake_create_upload_job)
|
assert response.status_code == 307
|
||||||
|
assert response.headers["location"] == "/ui/upload"
|
||||||
|
|
||||||
with pytest.raises(UploadError):
|
def test_upload_page_renders_expected_controls(self, client):
|
||||||
upload_page.submit_upload(filename="bad.jpg", file_bytes=b"")
|
"""GET /ui/upload returns the page shell and upload controls."""
|
||||||
|
response = client.get("/ui/upload")
|
||||||
|
|
||||||
|
assert response.status_code == 200
|
||||||
|
assert "Upload Document" in response.text
|
||||||
|
assert "Select document file" in response.text
|
||||||
|
assert "View jobs" in response.text
|
||||||
|
|||||||
|
After Width: | Height: | Size: 1.0 MiB |
|
After Width: | Height: | Size: 1.0 MiB |
|
After Width: | Height: | Size: 969 KiB |
|
After Width: | Height: | Size: 1.1 MiB |
|
After Width: | Height: | Size: 1010 KiB |
|
After Width: | Height: | Size: 1.1 MiB |
|
After Width: | Height: | Size: 6.0 MiB |