zoltan57andCopilot App
7b9715b3f1
Claim jobs atomically [CRIT-01] - Replace JobService.read_next_queued_job with claim_next_queued_job, which selects and transitions QUEUED -> PROCESSING inside one transaction. The old read-then-write sequence left a window in which two workers could observe the same QUEUED row. - Add the missing .limit(1). The poll previously ordered the entire queued set and discarded all but the first row. - Drop the eager loads from the hot poll entirely. They were pure waste: process_queued_job immediately re-reads the job through read_job with the relationships it actually needs. - Guard the row with with_for_update(skip_locked=True) on PostgreSQL so the claim stays correct once more than one worker exists. On SQLite the claim is a bounded single-writer transaction. - Correct the comment at the remaining direct-call claim site, which described the hazard rather than the guarantee. Reuse the provider connection [HIGH-02] - Build the ServiceBundle once per worker loop instead of once per job, and close it at loop shutdown. Every job previously constructed a new SourceService, and with it a new provider adapter and a new httpx.AsyncClient, paying a full TLS handshake per page and discarding the connection pool. - process_next_queued_job now accepts an optional caller-owned bundle and only closes bundles it created itself. Uncap the provider timeout [HIGH-03] - Remove le=20.0 from worker_provider_timeout_seconds. The cap equalled the default, so the ceiling could never be raised, and dense-page vision transcription routinely needs longer. Default raised to 180s. - Pass an explicit httpx.Timeout to the OpenRouter AsyncClient. httpx defaults every phase to 5 seconds, so the real read budget was 5s regardless of the configured value; the outer asyncio.wait_for could never be the binding constraint. Connect stays at 10s. Tighten the provider boundary [MED-03] - Declare model, current_request_manifest, current_transport_evidence, and aclose on the TranscriptionProvider Protocol. - Delete the per-call inspect.signature(adapter.transcribe).parameters reflection and the untyped kwargs dict it fed. The Protocol had declared requested_model all along, so the reflection was dead defensive weight on the hot path. - Replace the three getattr probes for aclose and the evidence attributes with direct typed access. Deduplicate bundle construction [MED-06] - Add ServiceBundle.from_session_factory and ServiceBundle.aclose, replacing the duplicated four-service instantiation blocks in app.py and worker.py. - _recover_stale_processing_jobs now uses the bundle built moments earlier instead of constructing a second JobService. Tests - Claiming returns the oldest job, marks it PROCESSING, never hands the same job out twice, and emits exactly one unadorned SELECT carrying LIMIT and no JOIN. - The worker loop threads one bundle through consecutive jobs and closes it once at shutdown; a caller-owned bundle is left open. - Settings accepts a timeout above 20 seconds and still rejects zero. - The OpenRouter client's read, write, and pool timeouts track the configured budget rather than the httpx default. Note: .env in this checkout still pins WORKER_PROVIDER_TIMEOUT_SECONDS=20 and should be raised to pick up this fix. Verified: 268 passed, 4 skipped; ruff check clean. Co-authored-by: Copilot App <[email protected]>
Transcription
Historical document transcription system for family-history documents.
The app lets you upload a document image/PDF, queues a background transcription job, and then shows job status and results in a web UI.
What the app does
- Upload document files (
.jpg,.jpeg,.png,.tif,.tiff,.pdf) - Persist document + job records in SQLite
- Process jobs in a background worker (
queued -> processing -> transcribed/failed) - Store transcript text (or failure detail)
- Show status and results in the NiceGUI interface
Quick start
1) Install dependencies
uv sync
2) Configure environment
Create a .env file in the project root with the required OpenRouter API key:
OPENROUTER_API_KEY=your_openrouter_api_key
Settings are read from CLI arguments first, then environment variables, then .env, then the defaults below.
Configuration Source Precedence
When the same setting is provided in multiple places, the value is chosen in this order (highest priority first):
- CLI arguments (for example
--port 8000) - Settings constructor arguments (used mainly in tests)
- Environment variables
.envfile values- Model defaults in code
Practical examples:
--port 8000overrides bothPORT=8000in the shell andPORT=7000in.env.DATABASE__PATH=prod.dbin the shell overridesDATABASE__PATH=dev.dbin.env.
Server and runtime
| Environment variable | Default | Description |
|---|---|---|
HOST |
0.0.0.0 |
Address on which the server listens. |
PORT |
8000 |
Server port. |
LOG_LEVEL |
info |
Uvicorn and application log level. |
RELOAD |
false |
Restart the development server when source files change. |
ENVIRONMENT |
development |
Runtime environment: development, test, or production. |
Provider
| Environment variable | Default | Description |
|---|---|---|
PROVIDER |
openrouter |
Transcription provider. |
OPENROUTER_API_KEY |
Required | OpenRouter API key. |
PROVIDER_MODEL |
Provider default | Optional model override. |
OPENROUTER_HTTP_REFERER |
Unset | Optional OpenRouter attribution URL. |
OPENROUTER_APP_TITLE |
Unset | Optional OpenRouter attribution title. |
Database and files
Use nested env vars for database settings (recommended):
DATABASE__DRIVER=sqlite
DATABASE__PATH=app.db
# BOOTSTRAP_SCHEMA_ON_STARTUP=true
SQLITE_CHECK_SAME_THREAD=false
UPLOAD_DIR=./uploads
PROMPT_DIR=./prompts
DEFAULT_PROMPT_NAME=transcribe_document.md
# TRANSCRIPTION_TEMPERATURE=0.2 # range: 0.0-2.0
# TRANSCRIPTION_TOP_P=0.9 # range: 0.0-1.0
For PostgreSQL:
DATABASE__DRIVER=postgres
DATABASE__HOST=localhost
DATABASE__PORT=5432
DATABASE__DATABASE=transcription
DATABASE__USER=postgres
DATABASE__PASSWORD=change-me
This uses Pydantic nested settings (env_nested_delimiter='__') and avoids JSON blobs in .env. A top-level DATABASE={...} JSON value is still supported as a fallback, and nested keys such as DATABASE__PATH take precedence over conflicting JSON keys.
BOOTSTRAP_SCHEMA_ON_STARTUP creates missing tables when the app starts. When unset, it is enabled in development and test, and disabled in production; set it explicitly to override that policy. SQLITE_CHECK_SAME_THREAD defaults to false.
Worker
WORKER_MAX_RETRIES=0
WORKER_RETRY_BACKOFF_SECONDS=0
WORKER_PROVIDER_TIMEOUT_SECONDS=20
WORKER_MIN_TRANSCRIPTION_CHARS=0
WORKER_MIN_TRANSCRIPTION_LINES=0
WORKER_FAIL_ON_FINISH_REASON_LENGTH=false
3) Run the app
uv run python -m transcription --port 8000 --reload --database.driver sqlite --bootstrap-schema-on-startup
This starts the development server with SQLite, creates missing tables, and enables automatic reload. Run uv run python -m transcription --help for all CLI options; CLI names use kebab case and nested database options use dot notation, such as --database.path ./data/transcription.db.
4) Open in browser
- GUI: http://localhost:8000/ui
- Health check: http://localhost:8000/healthz
Replace localhost with the server's hostname or IP address when connecting from another machine.
How to navigate the GUI
-
Upload page (
/ui)- Select a supported file to upload.
- The app creates a queued transcription job.
- Use the View jobs link to inspect progress.
-
Jobs page (
/ui/jobs)- See all jobs and their status.
- Use Refresh to reload current states.
- Open a specific job to see details.
-
Job detail page (
/ui/jobs/{job_id})- Shows job metadata and status.
- Displays transcript text when successful.
- Displays failure detail when transcription fails.
Prompt artifacts
Prompt files are stored directly in PROMPT_DIR (default: ./prompts). DEFAULT_PROMPT_NAME must be a filename,
not a path. Each job snapshots the validated prompt text, SHA-256 hash, and sampling values for reproducibility.
The canonical MVP prompt is:
prompts/transcribe_document.md
Destructive test procedure (with data backup)
AI execution policy: before the first unit-test run in a test/fix cycle, create one backup of ./data. Reuse that same backup for every subsequent test run in the cycle. After tests succeed, always pause and ask whether to restore now.
Use the cross-platform Python wrapper below whenever an AI agent runs tests against this repository.
- Create one backup of
./dataand mark it as the active test-cycle backup. - Run your test command.
- On failure, fix the errors and run the wrapper again; it reuses the active backup and never backs up post-test data.
- On success, always prompt whether to restore now (do not auto-restore unless explicitly approved).
- Close the cycle only by restoring the active backup or explicitly accepting the current data.
Preflight behavior:
- Backup preflight is warning-only when
data/transcription.dbappears in use. - Restore preflight is blocking: the script prompts you to close conflicting applications, then type
retryto re-check orcancelto skip restore.
Run with confirmation-gated restore (default)
uv run python tools/run_destructive_tests.py -- pytest tests/services/test_job_service.py tests/ui/test_jobs_page.py
After tests pass, the script asks whether to restore backup immediately.
This is the required default mode for AI-assisted test runs because it gives time to verify and accept code changes before any restoration happens.
Run with automatic restore (non-interactive)
uv run python tools/run_destructive_tests.py --auto-restore -- pytest
Run without terminal prompt (decide restore later)
uv run python tools/run_destructive_tests.py --skip-restore-prompt -- pytest
This keeps both the current post-test state and the backup, so restore can be decided explicitly later.
Repeated wrapper invocations reuse the backup recorded in .test-backups/.active-backup. If that backup is missing, the wrapper stops rather than creating a replacement from potentially destructive post-test data.
Restore later from a saved backup
uv run python tools/run_destructive_tests.py --restore-from data-backup-YYYYMMDD-HHMMSS
To keep the current data and close the active cycle without restoring:
uv run python tools/run_destructive_tests.py --accept-current-data
Backups are stored in .test-backups/ and ignored by git.