57 Commits
Author SHA1 Message Date
Jim Lancaster be3d0b7ee0 Some tweaks to accommodate relocating development to loca
Quality Gate / gate (push) Successful in 1m13s
2026-09-10 11:33:52 -05:00
Jim Lancaster 7eca9fe7dc V6.2 Add GEDCOM data
Quality Gate / gate (push) Successful in 1m27s
2026-09-03 05:34:13 -05:00
Jim Lancaster e54c2d9f26 Update the roadmap_plan.md
Quality Gate / gate (push) Successful in 2m15s
2026-09-02 19:03:35 -05:00
Jim Lancaster 065acad125 Scrub references to older versions
Quality Gate / gate (push) Successful in 2m39s
2026-09-02 17:02:48 -05:00
Jim Lancaster eeb1888aa3 Update instructions - epilog (Code review, remediation now complete)
Quality Gate / gate (push) Successful in 2m35s
2026-09-02 16:48:00 -05:00
Jim Lancaster 321c454a4f Implementat Rec - PHase 5 complete
Quality Gate / gate (push) Successful in 2m37s
2026-09-02 16:36:06 -05:00
Jim Lancaster 6f4accf275 Implement rec - Phase 4 complete
Quality Gate / gate (push) Successful in 2m38s
2026-09-02 16:23:09 -05:00
Jim Lancaster 27ec81ca5b Implement Rec - Phase 3 complete
Quality Gate / gate (push) Successful in 2m37s
2026-09-02 16:15:33 -05:00
Jim Lancaster 4410d23f5c Implement Rec - Phase 2
Quality Gate / gate (push) Successful in 2m36s
2026-09-02 16:05:18 -05:00
Jim Lancaster d5e798825d Implement Rec - Phase 1
Quality Gate / gate (push) Successful in 2m31s
2026-09-02 15:59:56 -05:00
Jim Lancaster 02dca888d2 Code-review complete
Quality Gate / gate (push) Failing after 14s
2026-09-02 15:47:53 -05:00
Jim Lancaster a896a11d2e Update instructions - final
Quality Gate / gate (push) Successful in 2m32s
2026-09-02 14:54:37 -05:00
Jim Lancaster 0b48c80d87 Update instructions - Part 2, phase 3
Quality Gate / gate (push) Successful in 2m34s
2026-09-02 14:42:20 -05:00
Jim Lancaster 70f8d6182e Update instructions - part 2, phase 2
Quality Gate / gate (push) Successful in 2m33s
2026-09-02 14:36:45 -05:00
Jim Lancaster fc4288ff88 Update instructions - part 2
Quality Gate / gate (push) Successful in 2m34s
2026-09-02 14:31:40 -05:00
Jim Lancaster e5ef4d4422 Update instructions, agents, skills - part 1
Quality Gate / gate (push) Successful in 2m41s
2026-09-02 14:05:41 -05:00
Jim Lancaster 88cef169c4 V6.1 update roadmap_plan.md
Quality Gate / gate (push) Successful in 2m35s
2026-09-02 13:42:15 -05:00
Jim Lancaster 15a4814e23 V6.1 runtime settings ->.env.production fix
Quality Gate / gate (push) Successful in 2m29s
2026-09-02 12:31:04 -05:00
Jim Lancaster 16391463d6 V6.1 yet more backup/.env cleanup
Quality Gate / gate (push) Successful in 2m33s
2026-09-02 11:43:07 -05:00
Jim Lancaster 96bb80d91f V6.1 backups: once more into the breach.
Quality Gate / gate (push) Successful in 2m29s
2026-09-02 10:59:28 -05:00
Jim Lancaster 494f378e48 V6.1 continue beating on the backup
Quality Gate / gate (push) Failing after 2m34s
2026-09-02 10:04:38 -05:00
Jim Lancaster 75dc946123 V6.1 continue refining backup: switch to Synology Drive Client for remote backup
Quality Gate / gate (push) Failing after 1m10s
2026-09-01 18:47:03 -05:00
Jim Lancaster 929f8d6de9 V6.1 Fix backup script
Quality Gate / gate (push) Failing after 2m30s
2026-09-01 16:08:25 -05:00
Jim Lancaster 72909c1fd5 V6.1 Another dockerfile fix
Quality Gate / gate (push) Failing after 1m1s
2026-09-01 15:59:40 -05:00
Jim Lancaster a5ec9f40a8 V6.1 dockerfile fix.
Quality Gate / gate (push) Failing after 1m0s
2026-09-01 15:36:34 -05:00
Jim Lancaster 0d5206fe97 V6.1 update database for maintenance runs
Quality Gate / gate (push) Failing after 2m29s
2026-09-01 13:51:38 -05:00
Jim Lancaster 32b6b5f29a V6.1 refinements - add link to source image gallery on Document detail page. Fix Maintenance tab actions (again).
Quality Gate / gate (push) Failing after 2m30s
2026-09-01 13:32:46 -05:00
Jim Lancaster 9b6bb9ae66 V6.1 more fixes
Quality Gate / gate (push) Failing after 2m35s
2026-09-01 12:36:59 -05:00
Jim Lancaster 4c877dd6a2 V6.1 fixes
Quality Gate / gate (push) Failing after 2m59s
2026-09-01 11:54:34 -05:00
Jim Lancaster 9990583345 V6.1 UI refinements, add Maintenance jobs to Settings
Quality Gate / gate (push) Failing after 2m57s
2026-08-31 11:31:25 -05:00
Jim Lancaster daa1642933 Cloudflare-related tweak
Quality Gate / gate (push) Failing after 50s
2026-08-27 12:23:08 -05:00
Jim Lancaster d198cc0c68 Separate CLoudflare from transcribe
Quality Gate / gate (push) Failing after 49s
2026-08-27 10:19:34 -05:00
Jim LancasterandCopilot App 34c7b16675 Refine Synology backup workflow
Quality Gate / gate (push) Failing after 47s
Co-authored-by: Copilot App <[email protected]>
2026-08-26 17:32:20 -05:00
Jim Lancaster ca29bc8b74 Continued work on backup script, methodology
Quality Gate / gate (push) Failing after 48s
2026-08-26 14:26:15 -05:00
Jim Lancaster 89ed83239e Update backup script to include everything needed to do a bare metal restore of the new V6 server
Quality Gate / gate (push) Failing after 49s
2026-08-26 12:21:40 -05:00
Jim Lancaster 2c6ef46f5f V6 complete
Quality Gate / gate (push) Failing after 50s
2026-08-26 12:04:43 -05:00
Jim Lancaster c2ed98c16d V6 Post migration cleanup
Quality Gate / gate (push) Failing after 50s
2026-08-26 11:08:10 -05:00
Jim Lancaster c85cc6be20 V6 Phase 4 - Postgres backup to Synology
Quality Gate / gate (push) Failing after 49s
2026-08-25 13:09:14 -05:00
Jim Lancaster 234476ba6d V6.0 Phase 3 - Add cloudflare tunnel. Remote access to app via the tunnel now working.
Quality Gate / gate (push) Failing after 49s
2026-08-25 13:05:47 -05:00
Jim Lancaster faa30fd27b V6 Phase 2 complete
Quality Gate / gate (push) Failing after 50s
2026-08-25 11:44:59 -05:00
Jim Lancaster 867cc9eb78 V6 Phase 1 complete
Quality Gate / gate (push) Failing after 49s
2026-08-25 10:38:45 -05:00
Jim Lancaster 0e43094b80 Refine Settings page
Quality Gate / gate (push) Failing after 49s
2026-08-24 17:16:40 -05:00
Jim Lancaster c05c054b94 Refine Settings page
Quality Gate / gate (push) Failing after 47s
2026-08-24 17:08:09 -05:00
Jim Lancaster c0112c2714 Add ability to edit safe .env.example options to Settings
Quality Gate / gate (push) Failing after 47s
2026-08-24 16:14:49 -05:00
Jim Lancaster 96af7b2dc0 Minor UI refinement
Quality Gate / gate (push) Failing after 49s
2026-08-24 12:58:34 -05:00
Jim Lancaster 9a7970c533 UI refinement: Back buttons
Quality Gate / gate (push) Failing after 49s
2026-08-24 12:44:02 -05:00
Jim LancasterandCopilot App f15c9834e4 Remove completed remediation handoff and mark the review closed
Quality Gate / gate (push) Failing after 47s
The handoff brief was a work order for phases 2-5. That work is done, so the
document now describes a future that already happened and would misdirect
anyone who found it.

The review report itself had the same problem in weaker form: its findings read
as open. Adds a status banner marking it closed and retained for reasoning only.

The banner also records that two of its recommendations were wrong on contact.
The HIGH-03 fix as written would have stripped root-cause data from
ExecutionAttempt provenance, and the HIGH-01 fix had to preserve per-page
durability the report never mentioned. Leaving that unstated invites someone to
'restore' the report's version later.

Co-authored-by: Copilot App <[email protected]>
2026-08-23 19:25:39 -05:00
Jim LancasterandCopilot App 2c59cbd2c7 Add AGENTS.md to make repo context auto-discoverable
Nothing in this repo was auto-read by an agent at the start of a session.
The .github/instructions files only attach once a matching file is edited,
which is too late to steer strategy, and the canonical docs set is not
discoverable without already knowing to look for it.

AGENTS.md routes rather than duplicates: it states the authority order
(docs/* first, docs/reviews/** explicitly non-canonical), the uv-only
command set, the test-enforced boundaries, and the change protocol.

The Traps section records failure modes this codebase has actually produced
rather than generic advice: the message/detail split that leaked paths in one
direction and degraded provenance in the other, the two competing atomicity
invariants in workflows.py where the obvious simplification breaks multi-page
durability, and the habit of changing a shared symbol without enumerating its
consumers.

Verified: cited test paths exist, full suite passes, ruff/ty clean.

Co-authored-by: Copilot App <[email protected]>
2026-08-23 19:23:26 -05:00
Jim LancasterandCopilot App 5ff66a8c40 Align python-reviewer agent with the python-code-reviewer skill
The agent file had drifted from the skill it delegates to, in two ways that
would corrupt a review run.

Report target: the agent said write reports to ./docs, but the skill targets
./docs/reviews/<date>-code-review.md and explicitly marks docs/reviews/** as
non-canonical. Following the agent would place a dated, opinionated review
inside the canonical authority set that findings are supposed to resolve
against.

Verification commands: the agent said run 'ruff check', 'pytest', and 'ty'.
None are on PATH in this uv project, so an agent following its own instruction
gets command-not-found and is pushed toward guessing instead of verifying.

The agent now defers to the skill for all specifics rather than restating them,
which is what let the two copies drift apart. Also carries forward the
consumer-tracing rule and states the read-only scope explicitly.

Co-authored-by: Copilot App <[email protected]>
2026-08-23 19:17:33 -05:00
Jim LancasterandCopilot App 626b5d4b10 Harden python-code-reviewer skill with lessons from executing its own review
Three gaps surfaced by implementing the 2026-08-23 review's recommendations.

1. Recommendations were never verified the way claims were. The report's fix for
   the error path leak would have stripped root-cause data from evidence records,
   because the review traced one consumer of AppError.message and missed that
   format_error_detail writes it to ExecutionAttempt.error_detail. Adds workflow
   step 9 (validate recommendations against consumers), a Blast Radius field on
   findings, and the worked example so the failure mode is concrete.

2. Fixes that sit between competing invariants were not flagged. The atomicity
   recommendation did not note that per-page durability and terminal-status
   atomicity pull in opposite directions, so the obvious simplification silently
   breaks multi-page durability. Recommendations must now name both invariants,
   the test guarding each, and the over-correction to avoid.

3. Severity could not express reachability. Two findings were latent behind a
   default setting and a single-instance deployment, which is a sequencing
   constraint: they must be fixed before the change that makes them live. Adds an
   explicit Reachability field with Live / Latent / Theoretical.

Verified: meta contract guards and traceability tests pass.

Co-authored-by: Copilot App <[email protected]>
2026-08-23 19:14:35 -05:00
Jim LancasterandCopilot App 12f125761a test: invert UI boundary guard to allowlist
Quality Gate / gate (push) Failing after 47s
Co-authored-by: Copilot App <[email protected]>
2026-08-23 18:57:13 -05:00
Jim LancasterandCopilot App 803237371e fix: log omitted OpenRouter request manifests
Co-authored-by: Copilot App <[email protected]>
2026-08-23 18:55:33 -05:00
Jim LancasterandCopilot App d4ae97c1b1 chore: remove uncertain orphaned definitions
Co-authored-by: Copilot App <[email protected]>
2026-08-23 18:53:54 -05:00
Jim LancasterandCopilot App c52d41ec33 test: extend orphan sweep to public class methods
Co-authored-by: Copilot App <[email protected]>
2026-08-23 18:52:06 -05:00
Jim LancasterandCopilot App 6c3eac0a44 test: prune low-signal assertions and tighten guards
Quality Gate / gate (push) Failing after 47s
Co-authored-by: Copilot App <[email protected]>
2026-08-23 18:48:16 -05:00
Jim LancasterandCopilot App e5410708e4 refactor: extract blocking and sequence retry helpers
Co-authored-by: Copilot App <[email protected]>
2026-08-23 18:46:19 -05:00
Jim LancasterandCopilot App efbae26f16 refactor: route workflow sessions through unit of work
Co-authored-by: Copilot App <[email protected]>
2026-08-23 18:43:07 -05:00
130 changed files with 7910 additions and 1601 deletions
-66
View File
@@ -1,66 +0,0 @@
# Canonical settings mirror for src/transcription/config.py (Settings).
# Any value here overrides the in-code default.
# --- NiceGUI Server ---
HOST=0.0.0.0
PORT=8000
# LOG_LEVEL: critical | error | warning | info | debug | trace
LOG_LEVEL=info
RELOAD=false
LOG_DIR=./data/logs
LOG_FILE_NAME=transcription.log
LOG_FILE_MAX_BYTES=10485760
LOG_FILE_BACKUP_COUNT=5
# --- AI provider ---
# PROVIDER: openrouter
PROVIDER=openrouter
# Required.
OPENROUTER_API_KEY=your-api-key-goes-here
PROVIDER_MODEL=google/gemini-2.5-flash
# PROVIDER_MODELS default: derived from PROVIDER_MODEL when omitted.
# If provided, use a non-empty JSON array.
# PROVIDER_MODELS=["google/gemini-2.5-flash","anthropic/claude-sonnet-4"]
# OPENROUTER_HTTP_REFERER=
# OPENROUTER_APP_TITLE=
DEFAULT_PROMPT_NAME=transcribe_document.md
# TRANSCRIPTION_TEMPERATURE default: unset (optional range 0.0..2.0)
# TRANSCRIPTION_TEMPERATURE=
# TRANSCRIPTION_TOP_P default: unset (optional range 0.0..1.0)
# TRANSCRIPTION_TOP_P=
# --- runtime environment ---
# ENVIRONMENT: development | test | production
ENVIRONMENT=development
# TRANSCRIPTION_COMMIT default: unset (optional build/commit identifier for provenance evidence)
# TRANSCRIPTION_COMMIT=
# --- persistence ---
# Use nested keys (env_nested_delimiter="__").
DATABASE__DRIVER=sqlite
DATABASE__PATH=./data/transcription.db
# Postgres example:
# DATABASE__DRIVER=postgres
# DATABASE__HOST=localhost
# DATABASE__PORT=5432
# DATABASE__DATABASE=transcription
# DATABASE__USER=postgres
# DATABASE__PASSWORD=change-me
BOOTSTRAP_SCHEMA_ON_STARTUP=false
SQLITE_CHECK_SAME_THREAD=false
# --- filesystem paths ---
UPLOAD_DIR=./data
PROMPT_DIR=./prompts
DATABASE_BACKUP_DIR=./data/backups
# --- worker reliability ---
WORKER_MAX_RETRIES=0
WORKER_PROVIDER_TIMEOUT_SECONDS=30.0
WORKER_STALE_JOB_SECONDS=30.0
WORKER_RETRY_BACKOFF_SECONDS=1.0
WORKER_SHUTDOWN_GRACE_SECONDS=5.0
WORKER_POLL_INTERVAL_SECONDS=1.0
WORKER_MIN_TRANSCRIPTION_CHARS=0
WORKER_MIN_TRANSCRIPTION_LINES=0
WORKER_FAIL_ON_FINISH_REASON_LENGTH=false
+67
View File
@@ -0,0 +1,67 @@
# Production environment example for docker-compose.production.yml
# --- NiceGUI Server ---
HOST=0.0.0.0
PORT=8000
LOG_LEVEL=info
RELOAD=false
ENVIRONMENT=production
# TRANSCRIPTION_COMMIT=
RUN_EMBEDDED_WORKER=false
LOG_DIR=/app/data/logs
LOG_FILE_NAME=transcription.log
LOG_FILE_MAX_BYTES=10485760
LOG_FILE_BACKUP_COUNT=5
# --- AI provider ---
PROVIDER=openrouter
OPENROUTER_API_KEY=replace-with-real-key
PROVIDER_MODEL=google/gemini-2.5-flash
# PROVIDER_MODELS=["google/gemini-2.5-flash","anthropic/claude-sonnet-4"]
# OPENROUTER_HTTP_REFERER=
# OPENROUTER_APP_TITLE=
DEFAULT_PROMPT_NAME=transcribe_document.md
# TRANSCRIPTION_TEMPERATURE=
# TRANSCRIPTION_TOP_P=
# --- persistence ---
# Common database settings:
DATABASE__DRIVER=postgres
DATABASE__DATABASE=transcription
DATABASE__USER=transcription
DATABASE__PASSWORD=replace-with-strong-password
BOOTSTRAP_SCHEMA_ON_STARTUP=false
# SQLite-specific settings:
# DATABASE__PATH=./data/transcription.db
# SQLITE_CHECK_SAME_THREAD=false
# Postgres-specific settings:
DATABASE__HOST=postgres
DATABASE__PORT=5432
# --- filesystem paths ---
UPLOAD_DIR=/app/uploads
PROMPT_DIR=/app/prompts
# --- backup workflow helpers (not Runtime Settings model fields) ---
BACKUP_DIR=/backup
BACKUP_RETENTION_DAYS=14
# --- worker reliability ---
WORKER_MAX_RETRIES=0
WORKER_PROVIDER_TIMEOUT_SECONDS=30.0
WORKER_STALE_JOB_SECONDS=90.0
WORKER_RETRY_BACKOFF_SECONDS=1.0
WORKER_SHUTDOWN_GRACE_SECONDS=5.0
WORKER_POLL_INTERVAL_SECONDS=1.0
WORKER_MIN_TRANSCRIPTION_CHARS=0
WORKER_MIN_TRANSCRIPTION_LINES=0
WORKER_FAIL_ON_FINISH_REASON_LENGTH=false
# --- cloudflare tunnel ---
# Required for token-based tunnel startup.
CLOUDFLARE_TUNNEL_TOKEN=replace-with-cloudflare-tunnel-token
# --- deployment wiring helpers ---
# Runtime Settings writes target this file path inside the app container.
RUNTIME_SETTINGS_ENV_FILE=/app/.env.production
+67
View File
@@ -0,0 +1,67 @@
# Production environment
# --- NiceGUI Server ---
HOST=0.0.0.0
PORT=8000
LOG_LEVEL=info
RELOAD=false
ENVIRONMENT=production
# TRANSCRIPTION_COMMIT=
RUN_EMBEDDED_WORKER=false
LOG_DIR=/app/data/logs
LOG_FILE_NAME=transcription.log
LOG_FILE_MAX_BYTES=10485760
LOG_FILE_BACKUP_COUNT=5
# --- AI provider ---
PROVIDER=openrouter
OPENROUTER_API_KEY=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
PROVIDER_MODEL=google/gemini-2.5-flash
PROVIDER_MODELS=["google/gemini-2.5-pro","google/gemini-2.5-flash","anthropic/claude-opus-5","anthropic/claude-sonnet-4","openai/gpt-5.6","openai/gpt-4o"]
# OPENROUTER_HTTP_REFERER=
# OPENROUTER_APP_TITLE=
DEFAULT_PROMPT_NAME=transcribe_document.md
# TRANSCRIPTION_TEMPERATURE=
# TRANSCRIPTION_TOP_P=
# --- persistence ---
# Common database settings:
DATABASE__DRIVER=postgres
DATABASE__DATABASE=transcription
DATABASE__USER=transcription
DATABASE__PASSWORD=<password>
BOOTSTRAP_SCHEMA_ON_STARTUP=false
# SQLite-specific settings:
# DATABASE__PATH=./data/transcription.db
# SQLITE_CHECK_SAME_THREAD=false
# Postgres-specific settings:
DATABASE__HOST=postgres
DATABASE__PORT=5432
# --- filesystem paths ---
UPLOAD_DIR=/app/uploads
PROMPT_DIR=/app/prompts
# --- backup workflow helpers (not Runtime Settings model fields) ---
BACKUP_DIR=/backup
BACKUP_RETENTION_DAYS=14
# --- worker reliability ---
WORKER_MAX_RETRIES=0
WORKER_PROVIDER_TIMEOUT_SECONDS=30.0
WORKER_STALE_JOB_SECONDS=30.0
WORKER_RETRY_BACKOFF_SECONDS=1.0
WORKER_SHUTDOWN_GRACE_SECONDS=5.0
WORKER_POLL_INTERVAL_SECONDS=1.0
WORKER_MIN_TRANSCRIPTION_CHARS=0
WORKER_MIN_TRANSCRIPTION_LINES=0
WORKER_FAIL_ON_FINISH_REASON_LENGTH=false
# --- cloudflare tunnel ---
# Required for token-based tunnel startup.
CLOUDFLARE_TUNNEL_TOKEN=CLOUDFLARE_TUNNEL_TOKEN=eyJhIjoiYTRhNjM0NzNhNzBiZjhhYmY3OWUyNjE4ZTcyNjgwZmMiLCJ0IjoiZWY1MjFkNWItYzY1ZS00ZGFmLTlmYTMtMzQyOGYzMGUyMDY4IiwicyI6IlpHWm1NVGRsTVRVdFpEYzNaaTAwWkRJeUxXRmhPRFV0TmpKallXRmhPRFJrWXpSaSJ9
# --- deployment wiring helpers ---
# Runtime Settings writes target this file path inside the app container.
RUNTIME_SETTINGS_ENV_FILE=/app/.env.production
+67
View File
@@ -0,0 +1,67 @@
# Production environment
# --- NiceGUI Server ---
HOST=0.0.0.0
PORT=8000
LOG_LEVEL=info
RELOAD=false
ENVIRONMENT=production
# TRANSCRIPTION_COMMIT=
RUN_EMBEDDED_WORKER=false
LOG_DIR=/app/data/logs
LOG_FILE_NAME=transcription.log
LOG_FILE_MAX_BYTES=10485760
LOG_FILE_BACKUP_COUNT=5
# --- AI provider ---
PROVIDER=openrouter
OPENROUTER_API_KEY=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
PROVIDER_MODEL=google/gemini-2.5-flash
PROVIDER_MODELS=["google/gemini-2.5-pro","google/gemini-2.5-flash","anthropic/claude-opus-5","anthropic/claude-sonnet-4","openai/gpt-5.6","openai/gpt-4o"]
# OPENROUTER_HTTP_REFERER=
# OPENROUTER_APP_TITLE=
DEFAULT_PROMPT_NAME=transcribe_document.md
# TRANSCRIPTION_TEMPERATURE=
# TRANSCRIPTION_TOP_P=
# --- persistence ---
# Common database settings:
DATABASE__DRIVER=sqlite
# DATABASE__DATABASE=transcription
# DATABASE__USER=transcription-local
# DATABASE__PASSWORD=My!3sons
# BOOTSTRAP_SCHEMA_ON_STARTUP=false
# SQLite-specific settings:
DATABASE__PATH=./data-local/transcription-local.db
SQLITE_CHECK_SAME_THREAD=false
# Postgres-specific settings:
# DATABASE__HOST=postgres
# DATABASE__PORT=5432
# --- filesystem paths ---
UPLOAD_DIR=./data-local
PROMPT_DIR=/data/prompts
# --- backup workflow helpers (not Runtime Settings model fields) ---
BACKUP_DIR=/backup
BACKUP_RETENTION_DAYS=14
# --- worker reliability ---
WORKER_MAX_RETRIES=0
WORKER_PROVIDER_TIMEOUT_SECONDS=30.0
WORKER_STALE_JOB_SECONDS=30.0
WORKER_RETRY_BACKOFF_SECONDS=1.0
WORKER_SHUTDOWN_GRACE_SECONDS=5.0
WORKER_POLL_INTERVAL_SECONDS=1.0
WORKER_MIN_TRANSCRIPTION_CHARS=0
WORKER_MIN_TRANSCRIPTION_LINES=0
WORKER_FAIL_ON_FINISH_REASON_LENGTH=false
# --- cloudflare tunnel ---
# Required for token-based tunnel startup.
CLOUDFLARE_TUNNEL_TOKEN=CLOUDFLARE_TUNNEL_TOKEN=eyJhIjoiYTRhNjM0NzNhNzBiZjhhYmY3OWUyNjE4ZTcyNjgwZmMiLCJ0IjoiZWY1MjFkNWItYzY1ZS00ZGFmLTlmYTMtMzQyOGYzMGUyMDY4IiwicyI6IlpHWm1NVGRsTVRVdFpEYzNaaTAwWkRJeUxXRmhPRFV0TmpKallXRmhPRFJrWXpSaSJ9
# --- deployment wiring helpers ---
# Runtime Settings writes target this file path inside the app container.
RUNTIME_SETTINGS_ENV_FILE=/app/.env.production
+8 -9
View File
@@ -1,12 +1,6 @@
---
name: Python Architect Reviewer
description: Evidence-based senior architect reviewer for FastAPI, NiceGUI, and SQLModel codebases.
tools:
- read_file
- list_dir
- file_search
- grep_search
- run_in_terminal
skills:
- python-code-reviewer
---
@@ -15,10 +9,15 @@ skills:
You are a Senior Python Architect performing an evidence-based, read-only code review.
> No `tools:` allowlist is declared here on purpose. Tool identifiers differ between the runtimes
> this agent is invoked from, so a hard-coded list silently under-tools the agent in one of them.
> Read-only discipline is enforced by the **Read-Only Scope** rule below, not by the frontmatter.
## Operating Principles
- **Stack Context:** Python 3.12+, FastAPI, NiceGUI, SQLModel, SQLAlchemy (SQLite/PostgreSQL), Pydantic V2, asyncio workers, and OpenRouter adapters.
- **Evidence-Based:** Always inspect real files. Every finding must reference concrete file paths and line numbers (e.g., `app/services/worker.py:45-78`). Do not speculate.
- **Tool Verification:** Run linters and tests via the terminal (`ruff check`, `pytest`, `ty`) to verify issues before reporting.
- **Skill Execution:** Adhere strictly to the review dimensions, duplication analysis, and report scaffolding defined in the `python-code-reviewer` skill.
- **Report Target:** Output all complete review reports as Markdown files written to `./docs`.
- **Tool Verification:** This is a `uv` project; the toolchain is not on `PATH`. Verify with `uv run ruff check .`, `uv run ty check`, and `uv run pytest -q -m "not external"`, and record the exact commands and outcomes. Never report a lint, type, or test claim you did not run.
- **Verify Recommendations, Not Just Findings:** Before recommending a change to a shared symbol, enumerate its consumers and confirm the fix is safe for each. See the skill's consumer-tracing step and `Blast Radius` field.
- **Skill Is Canonical:** The `python-code-reviewer` skill defines the review workflow, deterministic checks, severity and reachability rubrics, report location, and report template. Follow it exactly. Where this file and the skill disagree, the skill wins — do not restate its specifics here.
- **Read-Only Scope:** Do not modify source, tests, docs, instructions, or configuration. The review report is the only artifact you produce.
@@ -7,6 +7,11 @@ applyTo: 'src/transcription/**/*.py'
Keep docs in sync in the same change whenever implementation alters a documented contract, behavior, or roadmap decision.
Documentation targets below always refer to the **current** baseline. `docs/index.md` states which
baseline that is; resolve any version-specific document from there. Never cite a superseded version
tree by name in this file or in the docs you update — retired revision trees are not authority, and
`tests/test_meta_contract_guards.py` fails active contract files that route authority through them.
## Update documentation when any of these change
1. **Schema/Data contract**
@@ -15,7 +20,7 @@ Keep docs in sync in the same change whenever implementation alters a documented
2. **Configuration contract**
- `Settings` keys, defaults, required/optional environment values.
- **Required doc update:** `.env.example` and any directly related setup docs.
- **Required doc update:** `.env.production.example` and any directly related setup docs.
3. **User-visible UI behavior**
- Page flow, routes, button/action behavior, labels, status wording, empty/error states.
@@ -27,7 +32,8 @@ Keep docs in sync in the same change whenever implementation alters a documented
5. **Roadmap/scope decisions**
- Version targets, sequencing, deferrals, and accepted alternatives.
- **Required doc update:** `docs/roadmap_plan.md` and related backlog docs (for example `docs/ver4.8/feature_backlog_v4_8.md`).
- **Required doc update:** `docs/roadmap_plan.md`, plus any backlog or feature document for the
current baseline. Locate it through `docs/index.md` rather than assuming a version-named path.
## Working rule
@@ -75,6 +75,30 @@ Required internal -> canonical mapping:
- Include actionable remediation guidance aligned to category.
- Keep envelope structure consistent across API endpoints.
### `AppError.message` vs `AppError.detail`
`AppError` carries two texts with different audiences, and they must not be collapsed. Getting this
wrong has already caused a real defect in this repository, in both directions.
| Attribute | Audience | Reaches | Rule |
| :--- | :--- | :--- | :--- |
| `message` | User and API clients | `ErrorEnvelope.message`, UI notifications | Stays generic. Never embed exception text, provider payloads, or filesystem paths. |
| `detail` | Internal only | Logs, and `format_error_detail` -> `ExecutionAttempt.error_detail` and `MaintenanceRun.error_detail` | Carries the root cause. Never rendered to users or serialized into an envelope without a sanitizing projection. |
- Putting root-cause data in `message` leaks infrastructure detail to users.
- Omitting it from `detail` silently degrades the provenance record this system exists to preserve —
a failed attempt whose `error_detail` says nothing is an attempt that cannot be diagnosed later.
- Any render boundary that displays persisted `error_detail` must apply the same no-local-path rule
as `message`: sanitize machine-local absolute paths before the text becomes user-visible.
- When you raise from a caught exception, populate **both**: a generic `message` and a `detail`
carrying `type(exc).__name__` and the exception text, with `raise ... from exc`.
- `detail` is optional (`None`). A read path that assumes it is populated must handle its absence.
- Before changing either attribute, or any helper that formats them, enumerate every consumer —
evidence writes, maintenance runs, logging, API envelopes, and UI presentation all read these
fields, and tests assert on the persisted text.
Canonical definitions live in `src/transcription/errors.py`; see also `docs/error_handling.md`.
## Logging and Diagnostics
- Log operation identifiers and error IDs where available.
@@ -0,0 +1,122 @@
---
description: Provider adapter rules for evidence capture, secret safety, and client lifecycle.
applyTo: 'src/transcription/providers/**/*.py'
---
# Provider Adapters
Primary references:
- `docs/invariant/ai_evidence_and_provenance.md` (canonical; provider adapters own provider-boundary evidence capture)
- `docs/architecture.md`
- `docs/schema.md`
The provider layer is where an external API becomes application data. It is also the only place
that can capture what actually crossed the wire — once a response reaches a service, the evidence
it did not preserve is gone permanently. Treat capture correctness as the primary job of this
layer and text extraction as secondary.
## Layer Boundary
- Adapters may depend on `transcription.config`, `transcription.providers.*`, the HTTP client, and
the provider SDK. They must not import `services`, `db`, `ui`, or `api`.
- Provider specifics — headers, model slugs, payload shapes, SDK types, error classes — stop here.
Callers receive `TranscriptionResult` and `ProviderError` subclasses only.
- Adapters raise provider/domain exceptions. They must not emit user-facing text, notifications,
or remediation wording; that translation belongs to services and UI. See
[error-handling instructions](./error-handling.instructions.md).
- Adapters do not persist. They return evidence; services decide what is written and when.
Enforced by `tests/test_provider_boundaries.py`.
## Contract Surface
- Every adapter satisfies the `TranscriptionProvider` protocol in `base.py`. Failed-call evidence is
returned through the caller-owned `ProviderCallEvidence` sink passed to `transcribe()`, so
evidence stays scoped to one invocation instead of living on mutable adapter instance state.
- `TranscriptionResult`, `RequestManifest`, and `TransportEvidence` are `extra="forbid"` and frozen.
Add a field to the contract rather than smuggling data through an untyped dict.
- Evidence contracts in `evidence.py` are versioned (`schema_name` + `schema_version`). A change to
the meaning or shape of a captured field requires a version bump, not a silent redefinition —
stored evidence must keep its original meaning.
## Transport Evidence
The rules below implement `docs/invariant/ai_evidence_and_provenance.md` §3.4-3.5. That document
wins if this file drifts from it.
- Capture the response body **at the HTTP boundary, before SDK parsing**, so fields the SDK does
not model are not lost. `_CapturingAsyncClient` exists for this; do not replace it with a
post-parse `model_dump()` and call the result transport evidence.
- Reset per-call capture state at the start of every call. Without it, a connection failure can
attach the *previous* call's response as evidence for this one. Guarded by
`tests/test_evidence_provenance.py::test_openrouter_does_not_reuse_prior_response_on_connection_failure`.
- Keep transport capture scoped to the call, not the adapter instance. Concurrent `transcribe()`
calls on one adapter must not be able to overwrite each other's response evidence.
- Handle the streamed-body case (`httpx.ResponseNotRead`) rather than assuming `response.content`
is always available.
- When no response arrives — timeout, DNS, connection reset — emit
`TransportEvidence(response_received=False)`. Absence of a response is itself evidence and must
be explicit, never an empty body or a missing record.
- Preserve safe response evidence for **unsuccessful** calls too, whenever a response was received.
- Never relabel an SDK snapshot or normalized metadata as transport evidence, and never backfill
it into an execution that predates capture.
## Secret Safety
- Persist response headers only through `filter_safe_response_headers` and the
`SAFE_RESPONSE_HEADERS` allowlist. Allowlist, never denylist: capture-then-redact is prohibited,
because an unknown header is unsafe by default.
- Adding a header to the allowlist is a deliberate evidence decision. Confirm it carries no
credential, cookie, or session material, and state why it is needed for correlation, content
interpretation, rate-limit diagnosis, or audit.
- API keys, `Authorization`, and cookies must never appear in a manifest, evidence record, log
line, or exception message.
- The request manifest references source content by identity (digest, size, media type, page).
Do not duplicate base64 source bytes into it — `_replace_embedded_media` exists for this.
## Execution Specification
The manifest must let a reader reconstruct what was asked, per invariant §3.3:
- Provider, requested model, full effective prompt text, and prompt digest.
- Every explicitly supplied parameter, and — separately — which optional parameters were
**omitted**. Omission is not the same as a null value or an assumed provider default; the
`optional_parameter_states` distinction between `omitted`, `null`, and `value` is deliberate.
- Timeout budget, retry policy, source reference, and `SoftwareContext` versions.
- Manifest digests use `canonical_json_bytes`. Do not hash a plain `json.dumps()`; key order and
separators must stay deterministic or digests become uncomparable.
## Client Lifecycle and Async Safety
- Reuse one pooled `AsyncClient` per adapter instance; do not construct a client per request.
- Accept an injected client so tests can drive the adapter without network access.
- Derive timeouts from `Settings` (`worker_provider_timeout_seconds`) rather than hard-coding, and
keep the client timeout aligned with the configured budget so the SDK cannot expire first and
hide the real failure.
- Implement `aclose()` and release pooled resources. An adapter that creates a client owns closing
it; one given a client must not close a caller-owned resource it did not create.
- Never block the event loop. Offload CPU-bound work (hashing large payloads, image encoding) with
`asyncio.to_thread`.
- Propagate `asyncio.CancelledError` untouched — do not convert cancellation into a provider error.
## Failure Handling
- Raise `ProviderAuthError` for authentication, `ProviderResponseError` for malformed or unusable
responses, and `ProviderError` otherwise.
- Always attach `request_manifest`, `transport_evidence`, and an accurate `failure_phase` to raised
errors. `failure_phase` must distinguish a received-but-failed response from a call that never
reached the provider.
- Validate responses with Pydantic rather than indexing into raw dicts.
- Invalid *optional* metadata (for example unparsable token counts) must not discard an otherwise
valid transcript. Degrade the metadata, not the result.
## Contract Sync Rule
If capture behavior, evidence schema, or the header allowlist changes:
1. Update `docs/invariant/ai_evidence_and_provenance.md` only if the durable preservation contract
itself is changing — that revision is deliberate and reviewed, not incidental.
2. Update `docs/schema.md` when persisted evidence fields change.
3. Update or add tests in the same change (`tests/providers/`, `tests/test_evidence_provenance.py`).
4. Bump the affected evidence `schema_version` when a field's meaning changes.
+59 -8
View File
@@ -17,9 +17,17 @@ applyTo: 'src/transcription/services/*.py'
- **A service module must not import another service module.** This is enforced by
[test_service_boundaries](../../tests/test_service_boundaries.py). Shared types go in a
neutral module that defines no service class (see [errors](../../src/transcription/services/errors.py)).
- Not every module in this package is a service. Helper modules that define no `*Service`
class (`base`, `errors`, `normalization`, `prompts`, `quality`, `media_storage`,
`source_media`) are free-function modules and are exempt from the service rules below.
- Not every module in this package is a service. Modules fall into three kinds:
- **Aggregate services** own models and define a `*Service` class: `documents.py`, `sources.py`,
`jobs.py`, `people.py`, `photos.py`, `maintenance.py`, and `evidence.py` (read/projection only,
owns nothing).
- **Orchestration modules** define no service class and compose writes across aggregates:
`store.py`, `workflows.py`. They are the sanctioned place to create or delete rows owned by more
than one service — see [Service Composition](#service-composition).
- **Shared infrastructure and free-function helpers** are exempt from the service rules below:
`base.py` (`ServiceBase`), `registry.py` (`RegistryService`, a generic base for lookup tables —
not an aggregate owner itself), `unit_of_work.py`, `errors.py`, `normalization.py`, `prompts.py`,
`quality.py`, `media_storage.py`, `source_media.py`. `__init__.py` exposes `ServiceBundle`.
- Cross-cutting error behavior must follow
[error-handling instructions](./error-handling.instructions.md).
@@ -30,11 +38,38 @@ is the only service that may **create or delete** its rows.
| Model | Owner |
| --- | --- |
| `Document`, `DocumentType` | `DocumentService` |
| `Document`, `DocumentType`, `DocumentTag` | `DocumentService` |
| `Source`, `JobSource` | `SourceService` |
| `Job` | `JobService` |
| `Person`, `PersonRole`, `DocumentPerson` | `PeopleService` |
| `Person`, `PersonRole`, `DocumentPerson`, `PersonTag` | `PeopleService` |
| `GenealogyPerson`, `GenealogyFamily`, `GenealogyFamilyChild`, `GenealogyCitation` | `MaintenanceService` |
| `Photo` | `PhotosService` |
| `MaintenanceRun` | `MaintenanceService` |
| `ExecutionAttempt` | `SourceService` |
| `Tag` | shared — see below |
Keep this table complete: every table in `src/transcription/db/models.py` appears exactly once,
except `Tag`. When you add a model, add its owner here in the same change.
### `Tag` is deliberately shared
`Tag` is one table reached through two `RegistryService[Tag]` facades that differ only in the
reference model they count usage through: `TagRegistry` (`documents.py`, via `DocumentTag`) and
`PersonTagRegistry` (`people.py`, via `PersonTag`). Both create and delete `Tag` rows through the
generic registry. This is the single sanctioned exception to one-owner-per-model — do not "fix" it by
assigning `Tag` to one service, because the other facade would then be creating rows it does not own.
Any change to `Tag` semantics, labels, or normalization must be validated against **both** facades
and the junction table each one counts.
`DocumentTag` and `PersonTag` follow the junction rule below: each is created and deleted only by the
service on its own side.
### Registries
`DocumentTypeRegistry`, `TagRegistry`, `PersonRoleRegistry`, and `PersonTagRegistry` are
`RegistryService` subclasses, not independent services. A registry belongs to the aggregate service
whose module declares it and shares that service's ownership. Registry CRUD uses `<operation>_entry`
naming (see [CRUD Methods](#crud-methods)).
### Junction tables
@@ -50,8 +85,14 @@ lifecycle owner. The service on the other side may read through the junction (vi
Two consequences follow, and both are deliberate:
- **Cascade deletion is not a violation.** A service deleting the aggregate root it owns
may delete junction rows referencing that root, because they cannot outlive it
(`JobService.delete_job_with_guardrails`).
may delete rows referencing that root which cannot outlive it
(`JobService.delete_job_with_guardrails` deletes the job's `job_source` rows).
- **Evidence deletion is an explicit workflow, not a runtime path.** `JobService.delete_job_and_evidence`
deletes `ExecutionAttempt` rows owned by `SourceService`. That is sanctioned because it is the
named retention workflow that `delete_job_with_guardrails` refuses to perform implicitly — that
method *blocks* deletion when attempts exist. Append-only means runtime code never rewrites or
removes history to represent a new outcome; it does not forbid a deliberate, operator-invoked
retention operation. Do not add a second path that deletes attempts.
- **Ownership governs creation and deletion, not every state transition.** `job_source` is
both a link and the transcription work queue. `JobService.cancel_job` and
`resubmit_failed_sources` transition `job_source.status` across a whole job, because that
@@ -133,6 +174,16 @@ pages before it. Enforced by `tests/integration/test_pipeline_atomicity.py`; per
durability is separately enforced by
`tests/services/test_workflows_reliability.py::TestWorkflowReliability::test_transcribed_page_is_committed_before_next_provider_call_finishes`.
### Stale-reclaim safety
- `WORKER_STALE_JOB_SECONDS` must remain **greater than** `WORKER_PROVIDER_TIMEOUT_SECONDS`; stale
recovery must not be able to fire before one provider call can legitimately finish.
- Long-running multi-page orchestration must refresh job liveness explicitly between intermediate
page commits. Do not rely on incidental row updates or provider metadata writes to keep
`Job.date_updated` fresh.
- Enforced by `tests/test_config.py` and
`tests/services/test_workflows_reliability.py::TestWorkflowReliability::test_intermediate_page_commit_advances_job_liveness_timestamp`.
## Contract Alignment
- Treat `docs/` as the active architecture and requirements baseline.
@@ -145,7 +196,7 @@ durability is separately enforced by
- `Source.raw_transcription` is a projection, not authoritative history.
- Service/UI read paths that touch relationships must be eager-loaded for `lazy="raise"` compatibility.
- If model fields, enums, constraints, indexes, or relationship-loading semantics change, update `docs/schema.md` in the same change.
- If `Settings` fields or defaults change in `src/transcription/config.py`, update `.env.example` in the same change so keys/defaults remain synchronized and no stale settings remain documented.
- If `Settings` fields or defaults change in `src/transcription/config.py`, update `.env.production.example` in the same change so keys/defaults remain synchronized and no stale settings remain documented.
## Schema Drift and Legacy Compatibility Policy
+119
View File
@@ -0,0 +1,119 @@
---
description: Authoring rules for the test suite, including markers, async discipline, and guard-test design.
applyTo: 'tests/**/*.py'
---
# Tests
Primary references:
- `AGENTS.md` (Change Protocol — failing test first)
- `docs/index.md` and `docs/invariant/*`
- `.github/skills/test-effectiveness-auditor/skill.md` (periodic audit of this suite)
The suite is not only regression protection here — it is where several architectural rules are
*defined*. `tests/test_service_boundaries.py`, `tests/test_ui_boundaries.py`,
`tests/test_provider_boundaries.py`, `tests/test_model_contract_guards.py`, and
`tests/test_meta_contract_guards.py` are the enforcement layer named in the `AGENTS.md` authority
order. A weak test in this repository does not merely fail to catch a bug; it can silently repeal a
documented invariant.
The baseline is green. `uv run pytest -q -m "not external"` must report zero failures and zero
errors, and there is no tolerated set of known-failing tests.
## Write the Failing Test First
For any behavioral fix, write the test before the fix and confirm it fails *for the reason you
expect*. A test that passes against the broken code proves nothing, and several defects in this
repository were subtle enough that a test written afterward would have done exactly that. If the
new test passes immediately, you have not reproduced the defect yet.
## Runner Configuration
Configured in `pyproject.toml`; do not work around these:
- `--strict-markers` — an unregistered marker is an error. Register new markers in
`[tool.pytest.ini_options] markers` with a description rather than inventing one at the call site.
- `asyncio_mode = "strict"` — every async test needs an explicit `@pytest.mark.asyncio`, and async
fixtures use `@pytest_asyncio.fixture`. There is no implicit promotion.
- `filterwarnings = ["error:coroutine .* was never awaited:RuntimeWarning"]` — an un-awaited
coroutine is an error, not a warning. This usually means a mock replaced an async callable with a
sync one, or an `await` was dropped. Fix the call; never silence the warning.
## Markers and Layout
- `unit` — pure logic, no framework or database.
- `integration` — touches framework, database, or multi-component contracts.
- `external` — calls live services; slow and credential-dependent.
`external` tests must also carry their own `skipif` so the suite stays green without credentials
(see `tests/services/test_transcription_external.py`). Local and documented runs use
`-m "not external"`; CI intentionally runs unfiltered, which is equivalent because those tests skip
themselves. Never let an unmarked test reach the network.
Place tests by the layer under test: `tests/services/`, `tests/ui/`, `tests/api/`,
`tests/providers/`, `tests/integration/`, with cross-cutting guards at the top level.
## Fixtures and Isolation
- Prefer the shared fixtures in `tests/conftest.py` (`default_settings`, `async_session`,
`default_session_factory`, and the per-aggregate service fixtures) over building settings or
engines by hand.
- `Settings` is isolated suite-wide by the session-scoped autouse fixture in `conftest.py`, because
`env_file` resolves against the working directory. Tests that need env-file loading pass
`_env_file=` explicitly; tests asserting declared defaults need nothing. Do not reintroduce
reliance on a developer's local env file. Guarded by `tests/test_config_isolation.py`.
- Database fixtures refuse to run against anything but the per-test path, and that refusal is
deliberate. Never relax it to point a destructive fixture at a real database.
- Tests must not leave artifacts outside `tmp_path`.
## Assertion Strength
Assert on the domain effect, not on the fact that code ran.
- Prefer persisted state, status transitions, error categories, and evidence records over
"no exception raised", "not None", or a bare status code.
- **Read committed state through a separate session.** Asserting against the same session that
performed the write can pass on unflushed in-memory state and prove nothing about durability.
This is how the atomicity guarantees in `tests/services/test_workflows_reliability.py` and
`tests/integration/test_pipeline_atomicity.py` are made real.
- Critical paths need negative-path coverage — timeouts, provider failures, validation errors,
cancellation. Happy-path-only coverage of a critical module is a gap, not a suite.
- Avoid count-threshold assertions as a proxy for correctness. A test asserting "at least N items
were discovered" passes indefinitely while the thing it was meant to protect rots; assert on a
specific known member instead.
## Guard Tests
Structural guards carry extra obligations, because they are cited as proof that a rule holds.
- **Guard the guard.** Every scanning guard needs a companion assertion that the scan actually found
something, following the existing `test_*_are_discovered` pattern. A guard that silently scans an
empty set passes forever.
- **Scope must match the claim.** A guard's name and docstring must describe only what it actually
verifies. A test covering one function while appearing to enforce a repo-wide rule is worse than
no test, because it stops anyone from writing the real one.
- **Prove non-vacuity by injected fault.** Temporarily introduce the violation, confirm the guard
fails with a comprehensible message, then revert. Do this whenever you add or materially change a
guard. Revert with an explicit edit if the file has uncommitted changes — `git checkout --` will
discard them.
- **Prefer structural analysis to substring matching.** AST inspection of imports and definitions is
resistant to false negatives; a bare-name search across the repository is not, since an unrelated
mention anywhere makes dead code look reachable.
- Failure messages should name the offending file, symbol, and the remedy. These fire for people who
did not write the guard.
- Any new file under `.github/**` must be added to `ACTIVE_CONTRACT_FILES` in
`tests/test_meta_contract_guards.py`, or the completeness guard fails by design.
## Redundancy
Duplicate coverage across layers costs runtime and dilutes signal. Pick the canonical layer for a
behavior — unit for logic, integration for wiring — and let the other layer assert only what is
unique to it. Retire tests superseded by a stronger guard instead of accumulating both, and record
deliberate retentions with a rationale rather than leaving them unexplained.
## Contract Sync Rule
When a test encodes or relaxes a documented rule, update the corresponding instruction file or
`docs/*` page in the same change. When a guard test is the enforcement for a rule stated in
`AGENTS.md` or an instruction file, cite the test by name there so the link survives refactoring.
@@ -9,7 +9,7 @@ agent: Python Architect Reviewer
Execute a comprehensive, evidence-based code review of the target codebase.
## Target Scope
- **Review Target:** ${{input:target_path:./}}
- **Review Target:** the repository root, unless the invoker names a narrower path; review that path instead.
- **Source Root:** `src/`
- **Docs Root:** `docs/`
- **Focus Areas:** FastAPI endpoints, NiceGUI components, SQLModel persistence, asyncio workers, Pydantic V2 models, and OpenRouter provider adapters.
@@ -17,7 +17,7 @@ Execute a comprehensive, evidence-based code review of the target codebase.
## Execution Rules
1. Map repository layout, dependency manifests, and configuration files from the project root before inspecting modules.
2. Read real code modules under `src/` (or the specified target path); cite exact file paths and line ranges for every finding.
3. Validate issues using terminal tools (`ruff check`, `pytest`, `ty`) where appropriate.
3. Validate issues by running `uv run ruff check .`, `uv run ty check`, and `uv run pytest -q -m "not external"`. Nothing is on `PATH` in this `uv` project, so bare `ruff`/`ty`/`pytest` will fail.
4. Check for duplication, divergent implementations, and extractable helpers.
5. Format the entire review following the standardized 6-section template defined in the `python-code-reviewer` skill.
6. Write the final report as a Markdown file to `./docs/code-review-${{current_date}}.md`.
5. Format the entire review following the standardized 10-section template defined in the `python-code-reviewer` skill.
6. Write the final report to `./docs/reviews/<YYYY-MM-DD>-code-review.md`, using today's date. This path is defined by the skill; do not write the report anywhere else.
+55 -9
View File
@@ -31,7 +31,7 @@ upgrade policy") — do not report it as a defect or recommend widening it.
## Review Workflow
1. **Map the Repository First:** Inspect entry points, package layout, configurations, dependency manifests, and any project-specific rule files (`AGENTS.md`, `CLAUDE.md`, `.github/instructions/`). Project-specific conventions override generic advice.
1. **Map the Repository First:** Inspect entry points, package layout, configurations, dependency manifests, and any project-specific rule files (`AGENTS.md`, `.github/instructions/`, `.github/skills/`). Project-specific conventions override generic advice.
2. **Establish Canonical Authority First:** Read architecture/contracts (`docs/*`, `docs/invariant/*`, UI docs) and active instructions/skills before evaluating source behavior.
3. **Read Representative Modules:** Sample across all layers (routes/pages, UI components, services, workers, persistence, provider adapters, settings, tests) before drawing conclusions.
4. **Run Drift Analysis:** Compare documented intended behavior versus repository ground truth; identify both implementation drift and undocumented-but-repeatable conventions that should be formalized.
@@ -39,13 +39,30 @@ upgrade policy") — do not report it as a defect or recommend widening it.
6. **Assess Boundary and Coupling Health:** Evaluate UI/service/persistence/provider dependency flow, identify circular dependencies, leaky abstractions, and transaction ownership ambiguity.
7. **Assess Invariant Placement:** For each hard rule, decide whether it belongs in docs (rationale), instructions (active steering), skills (periodic audit procedure), or deterministic tests (enforcement).
8. **Verify Claims:** This is a `uv` project (`uv.lock`, root `ruff.toml`). Run `uv run ruff check .`, `uv run ty check`, and `uv run pytest -q -m "not external"` rather than guessing, and record the exact commands and their outcomes in the report.
9. **Prioritize Hot Paths:** Focus deeply on request handling, database sessions, background workers, and external API calls.
10. **Enforce Read-Only Safety:** Do not modify code unless explicitly instructed.
11. **Escalate Provenance Audits:** For evidence/provenance-heavy changes, apply invariant checks from `.github/skills/evidence-provenance-auditor/skill.md` and include pass/fail outcomes in the report.
12. **Escalate Test-Suite Audits:** When findings touch test coverage, redundancy, or assertion strength, apply `.github/skills/test-effectiveness-auditor/skill.md` and include its outcomes alongside the provenance results.
9. **Validate Recommendations Against Consumers:** A recommendation is a claim about the future and must be verified like any other. Before recommending a change to a shared symbol — a model field, an exception attribute, a helper's return value, a function signature — enumerate **every** consumer of that symbol (`grep` the whole repo, including tests) and confirm the fix is safe for each one. Record the consumers in the finding's **Blast Radius**. A fix that is correct for the path that produced the finding can silently break a second consumer, and evidence/provenance and logging paths are the usual casualties because they read the same fields the UI does.
10. **Prioritize Hot Paths:** Focus deeply on request handling, database sessions, background workers, and external API calls.
11. **Enforce Read-Only Safety:** Do not modify code unless explicitly instructed.
12. **Escalate Provenance Audits:** For evidence/provenance-heavy changes, apply invariant checks from `.github/skills/evidence-provenance-auditor/skill.md` and include pass/fail outcomes in the report.
13. **Escalate Test-Suite Audits:** When findings touch test coverage, redundancy, or assertion strength, apply `.github/skills/test-effectiveness-auditor/skill.md` and include its outcomes alongside the provenance results.
### Worked example: why step 9 exists
The 2026-08-23 review recommended fixing a filesystem-path leak in
`classify_unexpected_error` by making `AppError.message` generic and logging the exception
detail instead. The analysis of the leak was correct, and the fix was implemented as written.
It was wrong. `AppError.message` had a second consumer the review never traced:
`format_error_detail`, which writes `ExecutionAttempt.error_detail` — a **provenance record**.
The recommended fix closed a privacy leak by silently stripping root-cause data from the
evidence history this system exists to preserve. It was caught only because an unrelated
integration test asserted on the persisted error text.
The correct fix separated the audiences — a user-safe `message` and an internal-only `detail`
that still reaches evidence and logs. One `grep` for consumers of `.message` during the review
would have found this. Treat any recommendation that changes a widely-read field as unverified
until its consumers are enumerated.
## Repo-Specific Deterministic Checks (Transcription)
When reviewing this repository, always include explicit pass/fail checks for the following.
Where **Enforced by** reads *unenforced*, recommending a deterministic test is itself a finding.
@@ -54,7 +71,7 @@ Where **Enforced by** reads *unenforced*, recommending a deterministic test is i
| 1 | **Service boundary rule:** no service-to-service imports | `tests/test_service_boundaries.py` |
| 2 | **UI boundary rule:** pages/components do not perform persistence access | `tests/test_ui_boundaries.py` |
| 3 | **Status vocabulary conformance:** `JobStatus`/`JobSourceStatus`/`JobPurpose` usage matches current enums in `src/transcription/db/models.py`; no stringly-typed status literals | `tests/test_model_contract_guards.py` |
| 4 | **Evidence ownership conformance:** append-only attempt history is preserved and projection writes are not mistaken for history mutation (`src/transcription/services/sources.py`, `src/transcription/services/evidence.py`) | `tests/test_v42_evidence.py::test_attempts_are_append_only_and_exported_with_integrity` |
| 4 | **Evidence ownership conformance:** append-only attempt history is preserved and projection writes are not mistaken for history mutation (`src/transcription/services/sources.py`, `src/transcription/services/evidence.py`) | `tests/test_evidence_provenance.py::test_attempts_are_append_only_and_exported_with_integrity` |
| 5 | **Canonical authority:** findings must resolve against `docs/*` first | `tests/test_meta_contract_guards.py::test_canonical_authority_references_are_present` |
| 6 | **Schema contract fidelity:** when model/persistence behavior changes, `docs/schema.md` remains field-accurate with `src/transcription/db/models.py` | `tests/test_model_contract_guards.py` (field names, ordering, enum members, table coverage), `tests/test_meta_contract_guards.py` (presence and references) |
| 7 | **Media boundary conformance:** print/export media is record-validated and UI media URL generation uses controlled resolver paths | `tests/test_media_path_safety.py`, `tests/ui/test_media_urls.py` |
@@ -127,6 +144,30 @@ Severity reflects concrete consequence, never style preference or effort to fix.
- **Medium:** Correctness risk under load or edge conditions (N+1, missing eager load, leaked task, missing timeout); drift between docs and code with no immediate runtime impact.
- **Low:** Maintainability, typing completeness, duplication, naming, or dead code with no behavioral risk.
### Reachability
Severity states how bad the consequence is; **Reachability** states whether it can happen today.
They are independent, and a finding is not complete without both. Record one of:
- **Live:** reachable in the current configuration and deployment.
- **Latent:** the defective code is present but unreachable because of a current setting, single-
instance deployment, or absent caller. **State the exact condition that unblocks it.**
- **Theoretical:** requires a combination the project has explicitly ruled out.
Latent findings carry a scheduling constraint that severity alone cannot express: a latent defect
must usually be fixed *before* the change that makes it live, not after. Say so explicitly in the
finding and reflect the ordering in the §9 action plan — for example, "fix the retry-category gate
before raising `worker_max_retries` above 0," or "handle this `IntegrityError` before deploying a
second worker replica." Do not downgrade severity merely because a finding is latent.
### Conflicting invariants
When a fix sits between two invariants that pull in opposite directions, say so in the
**Recommendation** and name both, along with the test that guards each. Flag explicitly what the
over-correction would be, because the simplest-looking fix usually satisfies one invariant by
silently destroying the other. A recommendation that resolves one side without naming the other is
incomplete and will be implemented incorrectly.
## Output Report Structure & Template
Generate Markdown reports at `./docs/reviews/<YYYY-MM-DD>-code-review.md` following this exact
@@ -157,8 +198,13 @@ template structure. Reports are dated, non-canonical artifacts: `docs/reviews/**
### Critical Severity
#### [CRIT-01] Title
- **Location:** `path/to/file.py:lines`
- **Reachability:** Live / Latent (state the exact condition that unblocks it) / Theoretical
- **Problem & Consequence:** Concrete consequence, not a style opinion.
- **Recommendation:** Fix with before/after sketch.
- **Blast Radius:** Every consumer of the symbols the recommendation changes, each confirmed
safe. Write `None — change is local` only after actually searching. If the fix touches a
shared field or helper, list the call sites (including tests and evidence/logging paths).
- **Recommendation:** Fix with before/after sketch. If two invariants conflict here, name both,
name the test guarding each, and state what the over-correction would be.
- **Effort:** S / M / L
### High Severity
@@ -213,7 +259,7 @@ template structure. Reports are dated, non-canonical artifacts: `docs/reviews/**
---
## 8. Meta-Tooling & Instruction Update Recommendations
- Required updates to docs/instructions/skills/tests to keep enforcement current.
- Required updates to docs, instructions, skills, or tests to keep enforcement current.
---
@@ -64,7 +64,9 @@ Run a deterministic audit of test usefulness. Focus on whether tests catch real
## Output Format
Produce a Markdown report in `docs/`:
Produce a Markdown report at `docs/reviews/<YYYY-MM-DD>-test-effectiveness.md`, using today's date.
Like code review reports, it is a dated, non-canonical artifact: `docs/reviews/**` is not part of the
canonical authority set.
```markdown
# Test Effectiveness Audit Report
+9 -4
View File
@@ -1,7 +1,7 @@
name: Quality Gate
# V4.7 Phase 6 / review log [40]. Before this, ruff, ty and pytest were enforced
# only by .pre-commit-config.yaml, and only for developers who had actually run
# Repository quality gate. Before this workflow existed, ruff, ty, and pytest were
# enforced only by .pre-commit-config.yaml for developers who had run
# `pre-commit install`.
on:
@@ -27,13 +27,13 @@ jobs:
- name: Write placeholder configuration
# Settings requires openrouter_api_key and 115 tests cannot construct
# Settings without it. This is written to a .env file rather than exported
# Settings without it. This is written to .env.production rather than exported
# as an environment variable on purpose: the external tests guard on
# os.getenv("OPENROUTER_API_KEY"), which reads the process environment and
# not the file, so writing the file reproduces the local result exactly -
# the 4 external tests skip instead of running against a fake key and
# failing. Exporting it instead produces 3 failures.
run: echo "OPENROUTER_API_KEY=ci-placeholder-not-a-real-key" > .env
run: echo "OPENROUTER_API_KEY=ci-placeholder-not-a-real-key" > .env.production
- name: Lint and type check
# Runs the hooks defined in .pre-commit-config.yaml instead of repeating
@@ -42,4 +42,9 @@ jobs:
run: uv run pre-commit run --all-files --show-diff-on-failure
- name: Tests
# Deliberately unfiltered, unlike the "-m 'not external'" form the guidance files
# use for local runs. Tests marked "external" skip themselves when live-service
# credentials are absent, so CI gets the same effective set plus a real run of any
# external test whose credentials are configured. Not drift -- do not "fix" this to
# match the local command without also giving those tests a way to run.
run: uv run pytest
+14 -8
View File
@@ -11,19 +11,25 @@ wheels/
# Environment secrets
.env
.env.production
# SQLite database
*.db
# Document images
uploads/*
# All data including db, backups, document images, photos, and logs:
data/*
data/backups/*
data/documents/*
data/logs/*
data/photos/*
data-local/*
# Local destructive-test backups
.test-backups/
# Temporary migration files
.migration-bundle-v51
data.pre-v50-20260823/*
data.pre-v51-20260823-120434/*
# Migration tests
data-migration-test/*
.migration-bundle/*
# Cloudflare tunnel local runtime files
deploy/cloudflared/config.yml
deploy/cloudflared/config.yaml
deploy/cloudflared/credentials.json
+1 -1
View File
@@ -1,4 +1,4 @@
# Quality gate for V4.6 [HIGH-06]. `ruff check`, `ruff format --check`, and `ty check`
# Quality gate: `ruff check`, `ruff format --check`, and `ty check`
# are blocking once known `ty` false positives are suppressed inline with rationale.
#
# Both tools are uv-managed dev dependencies and are not on PATH, so each entry must
+117
View File
@@ -0,0 +1,117 @@
# AGENTS.md
Orientation for AI agents working in this repository. This file is a **router**, not a spec:
it points at canonical authority and flags the traps that are expensive to discover by trial.
Where this file and `docs/*` disagree, `docs/*` wins.
## What This Is
A document transcription system that preserves durable archival records (Documents, Sources,
People) and executes page transcription asynchronously through vision/LLM providers. Its
defining constraint is **evidence**: every machine attempt is recorded append-only with
request/response provenance. Features that would lose, mutate, or obscure that history are
wrong regardless of how convenient they are.
Stack: Python 3.12+ · FastAPI + NiceGUI · SQLModel/SQLAlchemy (SQLite-first, PostgreSQL-
compatible) · Pydantic V2 · asyncio worker · OpenRouter adapter.
## Commands
This is a `uv` project. **Nothing is on `PATH`**`ruff`, `ty`, and `pytest` all require
`uv run`. Bare invocations fail with command-not-found.
```bash
uv run ruff check . # lint (blocking in pre-commit)
uv run ruff format --check . # format (blocking in pre-commit)
uv run ty check # types (blocking in pre-commit)
uv run pytest -q -m "not external" # default verification run
```
`external` marks tests that hit live services; always exclude it unless explicitly asked.
All four commands are expected to pass clean — there is no tolerated baseline of failures.
If `ty` reports something, fix it or suppress it inline *with a rationale comment*; a bare
`ignore` will not survive review.
## Authority Order
Resolve every question in this order, and stop at the first that answers it:
1. **`docs/*`** — canonical. Start at [`docs/index.md`](docs/index.md), which defines the
reading order. `docs/invariant/*` holds cross-version rules that outlive any release.
2. **`.github/instructions/*.md`** — active steering, auto-attached when you edit matching
paths. Covers services, UI, providers, tests, error handling, and documentation sync.
3. **`.github/skills/*`** — periodic audit procedures (code review, provenance, test
effectiveness).
4. **`tests/`** — deterministic enforcement. A guard test is the ground truth for whatever
rule it encodes.
`.github/agents/` and `.github/prompts/` hold named workflows that are loaded only when
invoked explicitly, so they never override the order above. They are how a review or audit
is *started*, not a source of rules.
`docs/reviews/**` is **not** canonical. Those are dated, opinionated snapshots that were
accurate when written and may since have been fixed, superseded, or found wrong.
## Layout
| Path | Role |
| :--- | :--- |
| `src/transcription/ui/**`, `api/**` | Interface. No direct persistence access. |
| `src/transcription/services/**` | Domain logic and transaction ownership. |
| `src/transcription/db/**` | Models and persistence. |
| `src/transcription/providers/**` | Provider adapters; provider details stop here. |
| `src/transcription/worker.py` | Asyncio worker loop. |
| `tests/` | Includes boundary/contract guards, not just behavior tests. |
## Enforced Boundaries
These are not conventions — a test fails if you break them:
- **No service-to-service imports** (`test_service_boundaries.py`). Compose in the caller.
- **No persistence access from pages/components** (`test_ui_boundaries.py`, allowlist-based).
- **No hand-rolled error notifications in UI** — use the shared error presenter.
- **No stringly-typed status literals** — use the enums (`test_model_contract_guards.py`).
- **Attempt history is append-only** (`test_evidence_provenance.py`).
- **`docs/schema.md` stays field-accurate** with `db/models.py`.
- **Orphans are tracked, not tolerated** — `test_orphan_sweep.py` records each retained
orphan with rationale in `KNOWN_ORPHANS`.
## Traps
Non-obvious things that have already caused real bugs here:
- **`AppError.message` vs `AppError.detail`.** `message` is user/API-facing and must stay
generic — never put exception text or filesystem paths in it. `detail` is internal-only and
is what reaches logs and `ExecutionAttempt.error_detail`. Putting root-cause data in
`message` leaks; removing it from `detail` silently degrades provenance. See
`docs/error_handling.md`.
- **Two competing atomicity invariants in `services/workflows.py`.** Intermediate pages must
commit individually (durability across a long multi-page job); the *final* page must commit
atomically with the terminal job status. Collapsing the batch into one transaction satisfies
the second and destroys the first. Both are guarded — `test_workflows_reliability.py` and
`tests/integration/test_pipeline_atomicity.py`.
- **Shared symbols have more consumers than the obvious one.** Before changing a model field,
exception attribute, or helper return value, grep for every consumer including tests.
Evidence and logging paths frequently read the same fields the UI does.
- **`Tag` is owned by two services, on purpose.** Every other model has exactly one owning
service, so the ownership rule reads as absolute — it isn't. `Tag` is a single table reached
through two `RegistryService[Tag]` facades, `TagRegistry` (documents) and `PersonTagRegistry`
(people), which count usage through `DocumentTag` and `PersonTag` respectively. Changing tag
semantics through one facade silently changes the other. Consolidating them under one service
is not a cleanup; it makes the other side a cross-aggregate writer.
- **Import style:** ruff `isort` runs with `force-single-line = true`. One import per line.
- **Latent defects have ordering constraints.** Some code is unreachable only because of a
current setting or single-instance deployment. Fix it *before* the change that unblocks it,
not after.
## Change Protocol
- **Write the failing test first** for behavioral fixes, and confirm it actually fails for the
reason you think. Several bugs here were subtle enough that a test written afterward would
have passed against the broken code.
- **Update docs in the same change** when you alter a contract, behavior, or scope — see
`.github/instructions/documentation-sync.instructions.md`.
- **Do not commit unless asked.** Making a requested change is not consent to commit it.
- **Do not push or open PRs on your own initiative.**
- **Scope discipline:** fix what was asked plus what your change genuinely breaks. Pre-existing
unrelated issues are a separate conversation.
+8
View File
@@ -13,6 +13,8 @@ RUN uv sync --frozen --no-dev --no-install-project
COPY src ./src
COPY prompts ./prompts
COPY tools ./tools
COPY deploy ./deploy
RUN uv sync --frozen --no-dev
@@ -27,12 +29,18 @@ ENV PYTHONDONTWRITEBYTECODE=1 \
WORKDIR /app
RUN apt-get update \
&& apt-get install -y --no-install-recommends postgresql-client \
&& rm -rf /var/lib/apt/lists/*
RUN groupadd --system --gid 1001 appgroup \
&& useradd --system --uid 1001 --gid appgroup --create-home appuser
COPY --from=builder /app/.venv /app/.venv
COPY --from=builder /app/src /app/src
COPY --from=builder /app/prompts /app/prompts
COPY --from=builder /app/tools /app/tools
COPY --from=builder /app/deploy /app/deploy
RUN mkdir -p /app/uploads /app/data \
&& chown -R appuser:appgroup /app
+38 -6
View File
@@ -22,13 +22,13 @@ uv sync
### 2) Configure environment
Create a `.env` file in the project root with the required OpenRouter API key:
Create a `.env.production` file in the project root with the required OpenRouter API key:
```env
OPENROUTER_API_KEY=your_openrouter_api_key
```
Settings are read from CLI arguments first, then environment variables, then `.env`, then the defaults below.
Settings are read from CLI arguments first, then environment variables, then `.env.production`, then the defaults below.
### Configuration Source Precedence
@@ -37,13 +37,13 @@ When the same setting is provided in multiple places, the value is chosen in thi
1. CLI arguments (for example `--port 8000`)
2. Settings constructor arguments (used mainly in tests)
3. Environment variables
4. `.env` file values
4. `.env.production` file values
5. Model defaults in code
Practical examples:
- `--port 8000` overrides both `PORT=8000` in the shell and `PORT=7000` in `.env`.
- `DATABASE__PATH=prod.db` in the shell overrides `DATABASE__PATH=dev.db` in `.env`.
- `--port 8000` overrides both `PORT=8000` in the shell and `PORT=7000` in `.env.production`.
- `DATABASE__PATH=prod.db` in the shell overrides `DATABASE__PATH=dev.db` in `.env.production`.
#### Server and runtime
@@ -54,6 +54,7 @@ Practical examples:
| `LOG_LEVEL` | `info` | Uvicorn and application log level. |
| `RELOAD` | `false` | Restart the development server when source files change. |
| `ENVIRONMENT` | `development` | Runtime environment: `development`, `test`, or `production`. |
| `RUN_EMBEDDED_WORKER` | `true` | Run worker loop inside web app process. Set `false` when using a dedicated worker service. |
#### Provider
@@ -92,7 +93,7 @@ DATABASE__USER=postgres
DATABASE__PASSWORD=change-me
```
This uses Pydantic nested settings (`env_nested_delimiter='__'`) and avoids JSON blobs in `.env`. A top-level `DATABASE={...}` JSON value is still supported as a fallback, and nested keys such as `DATABASE__PATH` take precedence over conflicting JSON keys.
This uses Pydantic nested settings (`env_nested_delimiter='__'`) and avoids JSON blobs in env files. A top-level `DATABASE={...}` JSON value is still supported as a fallback, and nested keys such as `DATABASE__PATH` take precedence over conflicting JSON keys.
`BOOTSTRAP_SCHEMA_ON_STARTUP` creates missing tables when the app starts. When unset, it is enabled in `development` and `test`, and disabled in `production`; set it explicitly to override that policy. `SQLITE_CHECK_SAME_THREAD` defaults to `false`.
@@ -122,6 +123,33 @@ This starts the development server with SQLite, creates missing tables, and enab
Replace `localhost` with the server's hostname or IP address when connecting from another machine.
## Production stack (Phase 1)
Use the production compose profile for split app/worker deployment with PostgreSQL and Cloudflare Tunnel:
```bash
copy .env.production.example .env.production
docker compose -f docker-compose.production.yml up -d --build
```
Services:
- `app`: FastAPI + NiceGUI runtime (`RUN_EMBEDDED_WORKER=false`)
- `worker`: standalone queue processor (`python -m transcription.worker_service`)
- `postgres`: primary datastore
- `cloudflared`: tunnel client using mounted ingress config + `CLOUDFLARE_TUNNEL_TOKEN`
Operational defaults in the production compose file:
- worker healthcheck is disabled (the worker process has no HTTP `/healthz` endpoint)
- cloudflared is pinned to HTTP/2 with explicit DNS resolvers (`1.1.1.1`, `1.0.0.1`) for restricted LXC/container DNS environments
Cloudflare setup files:
1. `copy deploy\cloudflared\config.yml.example deploy\cloudflared\config.yml`
2. set `CLOUDFLARE_TUNNEL_TOKEN` in `.env.production`
3. update ingress hostnames in `deploy\cloudflared\config.yml`
## How to navigate the GUI
- **Upload page** (`/ui`)
@@ -152,6 +180,10 @@ The canonical MVP prompt is:
Schema upgrades use an explicit export/import rebuild flow (no runtime legacy write compatibility).
See `docs/data_migration.md` for commands and cutover steps.
## Backup and restore workflow
Production backup/restore (PostgreSQL + uploads + deployment config) steps are documented in `docs/backup_restore.md`.
## Destructive test procedure (with data backup)
AI execution policy: before the first unit-test run in a test/fix cycle, create one backup of `./data`. Reuse that same backup for every subsequent test run in the cycle. After tests succeed, always pause and ask whether to restore now.
+69
View File
@@ -0,0 +1,69 @@
#!/usr/bin/env sh
set -eu
BACKUP_DIR="${BACKUP_DIR:-./data/backups}"
RETENTION_DAYS="${BACKUP_RETENTION_DAYS:-14}"
UPLOAD_DIR="${UPLOAD_DIR:-/app/uploads}"
PROMPT_DIR="${PROMPT_DIR:-/app/prompts}"
DATABASE_DRIVER="${DATABASE__DRIVER:-postgres}"
DATABASE_HOST="${DATABASE__HOST:-postgres}"
DATABASE_PORT="${DATABASE__PORT:-5432}"
DATABASE_NAME="${DATABASE__DATABASE:-}"
DATABASE_USER="${DATABASE__USER:-}"
DATABASE_PASSWORD="${DATABASE__PASSWORD:-}"
timestamp="$(date -u +%Y%m%d-%H%M%S)"
postgres_file="postgres-${timestamp}.dump"
manifest_file="backup-${timestamp}.manifest"
mkdir -p "${BACKUP_DIR}"
if [ "${DATABASE_DRIVER}" != "postgres" ]; then
echo "create_postgres_backup.sh requires DATABASE__DRIVER=postgres." >&2
exit 1
fi
if [ -z "${DATABASE_NAME}" ] || [ -z "${DATABASE_USER}" ] || [ -z "${DATABASE_PASSWORD}" ]; then
echo "DATABASE__DATABASE, DATABASE__USER, and DATABASE__PASSWORD must be set." >&2
exit 1
fi
if ! command -v pg_dump >/dev/null 2>&1; then
echo "pg_dump is not installed in this environment." >&2
exit 1
fi
PGPASSWORD="${DATABASE_PASSWORD}" pg_dump \
-h "${DATABASE_HOST}" \
-p "${DATABASE_PORT}" \
-U "${DATABASE_USER}" \
-d "${DATABASE_NAME}" \
-Fc \
> "${BACKUP_DIR}/${postgres_file}"
cat > "${BACKUP_DIR}/${manifest_file}" <<EOF
created_at_utc=${timestamp}
postgres_dump=${postgres_file}
backup_dir=${BACKUP_DIR}
uploads_backup_dir=${BACKUP_DIR}/uploads
prompts_backup_dir=${BACKUP_DIR}/prompts
EOF
find "${BACKUP_DIR}" -type f \( \
-name 'postgres-*.dump' -o \
-name 'backup-*.manifest' \
\) -mtime +"${RETENTION_DAYS}" -delete
uploads_backup_dir="${BACKUP_DIR}/uploads"
mkdir -p "${uploads_backup_dir}"
if [ -d "${UPLOAD_DIR}" ]; then
cp -an "${UPLOAD_DIR}/." "${uploads_backup_dir}/"
fi
prompts_backup_dir="${BACKUP_DIR}/prompts"
mkdir -p "${prompts_backup_dir}"
if [ -d "${PROMPT_DIR}" ]; then
cp -a "${PROMPT_DIR}/." "${prompts_backup_dir}/"
fi
echo "Created backup set:"
echo " ${BACKUP_DIR}/${postgres_file}"
echo " ${BACKUP_DIR}/${manifest_file}"
echo " ${uploads_backup_dir}/ (incremental uploads mirror)"
echo " ${prompts_backup_dir}/ (prompts mirror)"
@@ -0,0 +1,36 @@
#!/usr/bin/env sh
set -eu
# Example only. Copy to a local script and replace placeholder values.
# Do NOT commit secrets.
SHARE="//nas-host-or-ip/share-name"
MOUNT_POINT="/mnt/nas-backups"
CREDENTIALS_FILE="/etc/samba/credentials/nas-share-credentials"
USERNAME="replace-with-nas-user"
PASSWORD="replace-with-nas-password"
if [ "${USERNAME}" = "replace-with-nas-user" ] || [ "${PASSWORD}" = "replace-with-nas-password" ]; then
echo "Edit USERNAME and PASSWORD placeholders before running this script."
exit 1
fi
mkdir -p "${MOUNT_POINT}"
mkdir -p "$(dirname "${CREDENTIALS_FILE}")"
cat > "${CREDENTIALS_FILE}" <<'EOF'
username=__USERNAME__
password=__PASSWORD__
EOF
sed -i "s|__USERNAME__|${USERNAME}|g" "${CREDENTIALS_FILE}"
sed -i "s|__PASSWORD__|${PASSWORD}|g" "${CREDENTIALS_FILE}"
chmod 600 "${CREDENTIALS_FILE}"
mount -t cifs "${SHARE}" "${MOUNT_POINT}" \
-o "credentials=${CREDENTIALS_FILE},vers=3.0,iocharset=utf8,uid=0,gid=0,file_mode=0600,dir_mode=0700"
echo ""
echo "Mounted ${SHARE} at ${MOUNT_POINT}"
echo ""
echo "To persist across reboot, add this line to /etc/fstab:"
echo "${SHARE} ${MOUNT_POINT} cifs credentials=${CREDENTIALS_FILE},vers=3.0,iocharset=utf8,uid=0,gid=0,file_mode=0600,dir_mode=0700,_netdev,nofail,x-systemd.automount 0 0"
+86
View File
@@ -0,0 +1,86 @@
#!/usr/bin/env sh
set -eu
if [ "$#" -lt 1 ]; then
echo "Usage: $0 <path-to-postgres-dump>"
exit 1
fi
dump_file="$1"
COMPOSE_FILE="${COMPOSE_FILE:-docker-compose.production.yml}"
ENV_FILE="${ENV_FILE:-.env.production}"
SYNOLOGY_BACKUP_DIR="${SYNOLOGY_BACKUP_DIR:-}"
if [ ! -f "${dump_file}" ]; then
echo "Backup file not found: ${dump_file}"
exit 1
fi
backup_dir="$(dirname "${dump_file}")"
backup_name="$(basename "${dump_file}")"
timestamp="$(printf '%s' "${backup_name}" | sed -n 's/^postgres-\([0-9]\{8\}-[0-9]\{6\}\)\.dump$/\1/p')"
uploads_file=""
config_file=""
# If SYNOLOGY_BACKUP_DIR wasn't exported in the shell, read it from ENV_FILE.
if [ -z "${SYNOLOGY_BACKUP_DIR}" ] && [ -f "${ENV_FILE}" ]; then
SYNOLOGY_BACKUP_DIR="$(
sed -n 's/^SYNOLOGY_BACKUP_DIR=//p' "${ENV_FILE}" | tail -n 1
)"
fi
if [ -n "${timestamp}" ]; then
# Legacy local full-archive naming.
candidate_uploads="${backup_dir}/uploads-${timestamp}.tar.gz"
candidate_config="${backup_dir}/config-${timestamp}.tar.gz"
if [ -f "${candidate_uploads}" ]; then
uploads_file="${candidate_uploads}"
fi
if [ -f "${candidate_config}" ]; then
config_file="${candidate_config}"
fi
fi
docker compose --env-file "${ENV_FILE}" -f "${COMPOSE_FILE}" exec -T postgres sh -lc \
"PGPASSWORD=\"\$POSTGRES_PASSWORD\" psql -U \"\$POSTGRES_USER\" -d postgres -c \"DROP DATABASE IF EXISTS \\\"\$POSTGRES_DB\\\";\""
docker compose --env-file "${ENV_FILE}" -f "${COMPOSE_FILE}" exec -T postgres sh -lc \
"PGPASSWORD=\"\$POSTGRES_PASSWORD\" psql -U \"\$POSTGRES_USER\" -d postgres -c \"CREATE DATABASE \\\"\$POSTGRES_DB\\\";\""
cat "${dump_file}" | docker compose --env-file "${ENV_FILE}" -f "${COMPOSE_FILE}" exec -T postgres sh -lc \
"PGPASSWORD=\"\$POSTGRES_PASSWORD\" pg_restore -U \"\$POSTGRES_USER\" -d \"\$POSTGRES_DB\" --clean --if-exists --no-owner --no-privileges"
if [ -n "${SYNOLOGY_BACKUP_DIR}" ] && [ -d "${SYNOLOGY_BACKUP_DIR}/uploads" ]; then
docker compose --env-file "${ENV_FILE}" -f "${COMPOSE_FILE}" run --rm --no-deps \
-v "${SYNOLOGY_BACKUP_DIR}:/backup" \
--entrypoint sh app -lc \
"mkdir -p /app/uploads/documents /app/uploads/photos && \
find /app/uploads/documents -mindepth 1 -delete && \
find /app/uploads/photos -mindepth 1 -delete && \
if [ -d /backup/uploads/documents ]; then cp -a /backup/uploads/documents/. /app/uploads/documents/; fi && \
if [ -d /backup/uploads/photos ]; then cp -a /backup/uploads/photos/. /app/uploads/photos/; fi && \
if [ -f /backup/uploads/homepage.md ]; then cp /backup/uploads/homepage.md /app/uploads/homepage.md; else rm -f /app/uploads/homepage.md; fi"
elif [ -n "${uploads_file}" ]; then
# Legacy local full-archive restore.
cat "${uploads_file}" | docker compose --env-file "${ENV_FILE}" -f "${COMPOSE_FILE}" run --rm --no-deps --entrypoint sh app -lc \
"mkdir -p /app/uploads && find /app/uploads -mindepth 1 -delete && tar -xzf - -C /app/uploads"
fi
if [ -n "${SYNOLOGY_BACKUP_DIR}" ] && [ -n "${timestamp}" ] && [ -f "${SYNOLOGY_BACKUP_DIR}/config-${timestamp}.tar.gz" ]; then
tar -xzf "${SYNOLOGY_BACKUP_DIR}/config-${timestamp}.tar.gz" -C .
elif [ -n "${config_file}" ]; then
# Legacy local full-archive restore.
tar -xzf "${config_file}" -C .
fi
echo "Restore complete from: ${dump_file}"
if [ -n "${SYNOLOGY_BACKUP_DIR}" ] && [ -d "${SYNOLOGY_BACKUP_DIR}/uploads" ]; then
echo "Restored uploads mirror from: ${SYNOLOGY_BACKUP_DIR}/uploads"
elif [ -n "${uploads_file}" ]; then
echo "Restored uploads archive: ${uploads_file}"
fi
if [ -n "${SYNOLOGY_BACKUP_DIR}" ] && [ -n "${timestamp}" ] && [ -f "${SYNOLOGY_BACKUP_DIR}/config-${timestamp}.tar.gz" ]; then
echo "Restored config archive: ${SYNOLOGY_BACKUP_DIR}/config-${timestamp}.tar.gz"
elif [ -n "${config_file}" ]; then
echo "Restored config archive: ${config_file}"
fi
+14
View File
@@ -0,0 +1,14 @@
ingress:
# Primary transcription app endpoint.
- hostname: transcription.example.com
service: http://app:8000
# Optional: generic remote access endpoints for other internal services.
# Replace hostnames and targets for your LAN.
- hostname: homeassistant.example.com
service: http://192.168.1.50:8123
- hostname: pihole.example.com
service: http://192.168.1.60:80
# Required catch-all.
- service: http_status:404
+70
View File
@@ -0,0 +1,70 @@
services:
app:
build:
context: .
dockerfile: Dockerfile
image: transcription:prod
env_file:
- .env.production
environment:
RUN_EMBEDDED_WORKER: "false"
RUNTIME_SETTINGS_ENV_FILE: "/app/.env.production"
depends_on:
postgres:
condition: service_healthy
ports:
- "8000:8000"
volumes:
- app_uploads:/app/uploads
- app_data:/app/data
- ./backup:/backup
- ./.env.production:/app/.env.production
- ./prompts:/app/prompts:ro
restart: unless-stopped
healthcheck:
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8000/healthz')"]
interval: 30s
timeout: 5s
retries: 5
start_period: 20s
worker:
image: transcription:prod
env_file:
- .env.production
command: ["python", "-m", "transcription.worker_service"]
depends_on:
postgres:
condition: service_healthy
volumes:
- app_uploads:/app/uploads
- app_data:/app/data
- ./backup:/backup
- ./.env.production:/app/.env.production:ro
- ./prompts:/app/prompts:ro
healthcheck:
disable: true
restart: unless-stopped
postgres:
image: postgres:16-alpine
env_file:
- .env.production
environment:
POSTGRES_DB: ${DATABASE__DATABASE}
POSTGRES_USER: ${DATABASE__USER}
POSTGRES_PASSWORD: ${DATABASE__PASSWORD}
volumes:
- postgres_data:/var/lib/postgresql/data
restart: unless-stopped
healthcheck:
test: ["CMD-SHELL", "pg_isready -U $$POSTGRES_USER -d $$POSTGRES_DB"]
interval: 10s
timeout: 5s
retries: 10
start_period: 10s
volumes:
postgres_data:
app_uploads:
app_data:
+1 -1
View File
@@ -5,7 +5,7 @@ services:
dockerfile: Dockerfile
container_name: transcription-app
env_file:
- .env
- .env.production
environment:
# Database configuration uses nested settings names (env_nested_delimiter="__").
# DATABASE_URL is NOT read by the application and must not be used here.
+66 -5
View File
@@ -1,6 +1,6 @@
# System Architecture (Current Baseline: V5.1)
# System Architecture (Current Baseline: V6.1)
This document defines the current V5.1 architecture baseline.
This document defines the current V6.1 architecture baseline.
## Architecture Objectives
@@ -13,10 +13,11 @@ This document defines the current V5.1 architecture baseline.
- **Runtime:** Python 3.12+
- **Web application:** FastAPI + NiceGUI
- **Persistence:** SQLModel / SQLAlchemy (SQLite-first, PostgreSQL-compatible model design)
- **Persistence:** SQLModel / SQLAlchemy — PostgreSQL in production, SQLite for local development and tests
- **Validation and settings:** Pydantic V2 + pydantic-settings
- **Concurrency:** asyncio worker loop
- **Provider integration:** OpenRouter adapter behind provider interface
- **Deployment:** Docker Compose (app, worker, PostgreSQL, Cloudflare Tunnel)
- **Quality and tests:** Ruff, ty, pytest, pytest-asyncio
## Runtime Topology
@@ -25,11 +26,32 @@ This document defines the current V5.1 architecture baseline.
flowchart LR
U[Browser User] --> A[FastAPI + NiceGUI App]
A --> W[Asyncio Worker]
A --> DB[(SQLite/PostgreSQL Model)]
A --> DB[(PostgreSQL / SQLite)]
W --> P[Provider Adapter]
W --> DB
```
The worker loop drains two queues in the same pass: queued transcription Jobs and queued
`MaintenanceRun` records. When neither has work, it idles.
### Production deployment
Production runs as a Docker Compose stack with the app and worker as separate services, so the app
process runs with `RUN_EMBEDDED_WORKER=false` and the worker process owns queue draining. Local
development runs a single process with the worker embedded.
```mermaid
flowchart LR
I[Internet] --> CF[cloudflared tunnel + Access]
CF --> APP[app service]
APP --> PG[(postgres service)]
WK[worker service] --> PG
WK --> PROV[OpenRouter]
```
Deployment, rollback, and recovery procedures are in [Production Runbook](production-runbook.md);
backup configuration and restore are in [Backup and Restore](backup_restore.md).
## Layered Boundaries
### Interface Layer
@@ -45,11 +67,18 @@ Responsibilities:
### Service and Orchestration Layer
Aggregate services:
- `src/transcription/services/documents.py`
- `src/transcription/services/people.py`
- `src/transcription/services/jobs.py`
- `src/transcription/services/sources.py`
- `src/transcription/services/evidence.py`
- `src/transcription/services/photos.py`
- `src/transcription/services/maintenance.py`
- `src/transcription/services/evidence.py` (read/projection only)
Orchestration modules:
- `src/transcription/services/store.py`
- `src/transcription/services/workflows.py`
@@ -58,6 +87,11 @@ Responsibilities:
- Aggregate ownership and invariants.
- Transaction-aware write helpers.
- Cross-service workflows in orchestration modules (`store.py`, `workflows.py`).
- Lookup-table CRUD through the generic `RegistryService` base (`registry.py`), which is not an
aggregate owner itself.
Module classification and per-model ownership are defined in
[services instructions](../.github/instructions/services.instructions.md).
### Persistence Layer
@@ -85,7 +119,13 @@ Responsibilities:
- `Job` is an aggregate processing run with status and frozen prompt/runtime settings.
- `JobSource` is queue/membership state for one `(job, source)` pair.
- `ExecutionAttempt` is append-only evidence for each provider call.
- `Photo` is person imagery owned by `PhotosService`.
- `MaintenanceRun` is one queued or executed operational maintenance run.
- `GenealogyPerson`, `GenealogyFamily`, `GenealogyFamilyChild`, and `GenealogyCitation` store
imported GEDCOM genealogy data and citation provenance.
- `DocumentType` and `PersonRole` are UUID-backed registries with optional protected `semantic_key`.
- `Tag` is a shared registry reached through both document and person tagging, linked by
`DocumentTag` and `PersonTag`.
## Processing and Evidence Workflow
@@ -106,6 +146,25 @@ Responsibilities:
- **Job statuses:** `queued`, `processing`, `transcribed`, `partial_success`, `failed`
- Operational success path resolves to `transcribed`.
- **JobSource statuses:** `pending`, `transcribed`, `failed`, `cancelled`
- **MaintenanceRun statuses:** `queued`, `processing`, `succeeded`, `failed`
- Maintenance uses `succeeded` rather than `transcribed`; the transcription vocabulary does not
apply to operational runs.
- **Maintenance job types:** `backup`, `storage_reconciliation`, `gedcom_import`
## Maintenance Execution
Operational maintenance is queue-backed rather than run inline from the UI, so it survives request
lifetime and is recorded:
1. Settings enqueues a `MaintenanceRun` with `status=queued` and a `triggered_by` marker.
2. The worker claims the oldest queued run with a conditional update, moving it to `processing`.
3. `backup` runs the deploy backup script; `storage_reconciliation` compares stored media against
`Document`/`Source` records; `gedcom_import` parses the latest uploaded `.ged` file and upserts
genealogy records.
4. The run finalizes to `succeeded` or `failed` with summary, timing, log path, and `error_detail`.
`MaintenanceRun` records operational history and is not evidence in the `ExecutionAttempt` sense;
append-only guarantees apply to transcription attempts.
## Security and Path Handling Boundaries
@@ -165,5 +224,7 @@ Current architecture rules live in `docs/*`.
- [System Requirements](requirements.md)
- [Data Model](schema.md)
- [Error Handling Policy](error_handling.md)
- [Production Runbook](production-runbook.md)
- [Backup and Restore](backup_restore.md)
- [Error Handling invariant](./invariant/error_handling.md)
- [AI evidence invariant](./invariant/ai_evidence_and_provenance.md)
+60
View File
@@ -0,0 +1,60 @@
# Backup and Restore (V6.1)
This guide defines operational backup/restore for clean-slate recovery of the Docker runtime using a host-visible backup folder.
## 1. Backup artifacts
- Backup target root: `BACKUP_DIR` (recommended production value: `/backup`)
- Database artifact per run:
- `postgres-YYYYMMDD-HHMMSS.dump` (PostgreSQL custom dump via `pg_dump -Fc`)
- `backup-YYYYMMDD-HHMMSS.manifest` (run manifest)
- Media/config mirrors under `BACKUP_DIR`:
- `uploads/**` (incremental copy: new files only)
- `prompts/**` (prompt directory mirror)
Retention:
- `BACKUP_RETENTION_DAYS` applies to `postgres-*.dump` and `backup-*.manifest` files.
## 2. Creating backups
Run from repository root:
```bash
sh deploy/backup/create_postgres_backup.sh
```
Environment variables used by the backup script:
- `BACKUP_DIR` (default `./data/backups`)
- `BACKUP_RETENTION_DAYS` (default `14`)
- `UPLOAD_DIR` (default `/app/uploads`)
- `PROMPT_DIR` (default `/app/prompts`)
- `DATABASE__DRIVER` (must be `postgres`)
- `DATABASE__HOST` (default `postgres`)
- `DATABASE__PORT` (default `5432`)
- `DATABASE__DATABASE` (required)
- `DATABASE__USER` (required)
- `DATABASE__PASSWORD` (required)
Recommended production setup:
- Mount a host-visible folder into `/backup` for both `app` and `worker`.
- Set `BACKUP_DIR=/backup` in `.env.production`.
- Use host-level tooling (for example Synology Drive Client on the host) to replicate that folder externally.
## 3. Restoring from backup
Restore requires downtime for app + worker writes.
1. Stop app and worker:
- `docker compose --env-file .env.production -f docker-compose.production.yml stop app worker`
2. Restore database:
- `sh deploy/backup/restore_postgres_backup.sh /backup/postgres-YYYYMMDD-HHMMSS.dump`
3. Start app and worker:
- `docker compose --env-file .env.production -f docker-compose.production.yml start app worker`
4. Validate `/healthz` and run one smoke workflow.
Notes:
- `restore_postgres_backup.sh` still supports legacy archive restore paths for older backup sets.
+66
View File
@@ -0,0 +1,66 @@
# Cloudflare Tunnel and Access Setup
This guide defines the repository-supported setup for exposing app and selected LAN services through Cloudflare Tunnel with Cloudflare Access protection.
## 1. Files used by this deployment
1. `deploy/cloudflared/config.yml` (local copy from `config.yml.example`)
2. `.env.production` (`CLOUDFLARE_TUNNEL_TOKEN`)
3. `docker-compose.production.yml` (`cloudflared` service reads token + mounts config)
Do not commit `config.yml` or `.env.production`.
## 2. Configure cloudflared
1. Copy `deploy/cloudflared/config.yml.example` to `deploy/cloudflared/config.yml`.
2. Update hostname -> service mappings in `ingress`.
3. Keep the final catch-all ingress `http_status:404`.
4. Set `CLOUDFLARE_TUNNEL_TOKEN` in `.env.production`.
Example app route:
- `transcription.example.com` -> `http://app:8000`
Optional generic remote-access routes:
- `homeassistant.example.com` -> `http://<home-assistant-lan-ip>:8123`
- `pihole.example.com` -> `http://<pihole-lan-ip>:80`
## 3. Cloudflare Access policy baseline
Create one Access app policy per exposed hostname:
1. Include: your allowed identities/groups only.
2. Exclude: none by default.
3. Require: identity provider login (and MFA if available).
Recommended baseline:
- App endpoint (`transcription.*`): your admin identity set.
- Other internal endpoints (`homeassistant.*`, `pihole.*`, etc.): explicit least-privilege groups.
## 4. Startup
Start production stack:
```bash
docker compose --env-file .env.production -f docker-compose.production.yml up -d --build
```
Validate tunnel container:
```bash
docker compose --env-file .env.production -f docker-compose.production.yml logs cloudflared
```
LXC/proxied-network note:
- The `cloudflared` service is pinned to `--protocol http2` with explicit DNS resolvers (`1.1.1.1`, `1.0.0.1`) in `docker-compose.production.yml`.
- This avoids environments where Docker's embedded resolver (`127.0.0.11`) cannot resolve `region*.v2.argotunnel.com`, which causes connector precheck failure and tunnel shutdown.
- If tunnel status is still down, verify host/container egress for DNS and TCP 443 to `api.cloudflare.com` and `*.argotunnel.com`.
## 5. Security notes
- Keep `postgres` and other internal-only services off public hostnames unless required.
- Use distinct hostnames per service; avoid path-based multiplexing for unrelated admin surfaces.
- Rotate `CLOUDFLARE_TUNNEL_TOKEN` and Access policy memberships on a regular schedule.
+42 -6
View File
@@ -19,13 +19,36 @@ Optional source overrides:
- `--source-db <path-or-sqlalchemy-url>`
- `--source-upload-dir <path>`
### 2) Import bundle into a fresh DB + uploads root
### 2) Import bundle into a fresh target (SQLite or PostgreSQL)
```bash
uv run python tools/export_import_migration.py import --bundle-dir .migration-bundle --target-db .\data\transcription-new.db --target-upload-dir .\data-new
```
### 3) One-shot export+import
PostgreSQL target example:
```bash
uv run python tools/export_import_migration.py import --bundle-dir .migration-bundle --target-db postgresql://transcription:change-me@localhost:5432/transcription --target-upload-dir .\data-new
```
### 3) Verify migration parity and integrity
```bash
uv run python tools/export_import_migration.py verify --source-db .\data\transcription.db --target-db postgresql://transcription:change-me@localhost:5432/transcription
```
The verify command checks:
- row-count parity across migration tables
- orphan-reference checks for `source`, `job`, `job_source`, and `execution_attempt`
- duplicate `(job_id, source_id, attempt_number)` in `execution_attempt`
Exit code:
- `0` when counts and integrity checks pass
- `1` when mismatches or integrity violations are detected
### 4) One-shot export+import
```bash
uv run python tools/export_import_migration.py migrate --bundle-dir .migration-bundle --target-db .\data\transcription-new.db --target-upload-dir .\data-new
@@ -50,9 +73,22 @@ Legacy V4.x portrait/homepage backfill in the export step:
- Legacy homepage markdown is relocated from `UPLOAD_DIR/homepage/homepage.md` to `UPLOAD_DIR/homepage.md`.
- Legacy `person.full_name` values are split into `given_names` + `last_name` for V5.1 schema compatibility.
## Cutover
## Cutover (SQLite -> PostgreSQL)
After importing to a fresh target:
1. Stop the app.
2. Point `DATABASE__*` and `UPLOAD_DIR` to the new targets.
3. Start the app and run smoke checks (`/healthz`, create/upload/process one job).
1. Stop app and worker services to freeze writes.
2. Export a migration bundle from the last SQLite state.
3. Import bundle to PostgreSQL target.
4. Run `verify` against source and target before switching runtime.
5. Switch runtime config to PostgreSQL (`DATABASE__DRIVER=postgres` and related `DATABASE__*` values).
6. Start app and worker services.
7. Run smoke checks (`/healthz`, create/upload/process one job).
## Rollback
If verify or smoke checks fail:
1. Stop app and worker services.
2. Revert runtime config to SQLite.
3. Start app and worker against pre-cutover SQLite database.
4. Preserve failed migration bundle and logs for analysis.
+9 -5
View File
@@ -1,6 +1,6 @@
# Error Handling Policy (Current Baseline: V5.1)
# Error Handling Policy (Current Baseline: V6.1)
This policy defines the active V5.1 error taxonomy, translation boundaries, and retry semantics.
This policy defines the active V6.1 error taxonomy, translation boundaries, and retry semantics.
## Error Categories
@@ -107,12 +107,16 @@ user-facing envelopes must not carry it. `AppError` therefore separates the two
| Field | Audience | Carries root cause | Surfaces |
| --- | --- | --- | --- |
| `message` | User-facing and API-facing | No | `show_error`, `build_error_envelope` |
| `detail` | Internal only | Yes | `format_error_detail` (evidence), logs |
| `detail` | Internal only | Yes | `format_error_detail` (evidence), logs, sanitized UI projection only |
`classify_unexpected_error` builds a generic `message` and puts the exception type and
text on `detail`. Anything rendered to a user or serialized into an API envelope must
read `message`; anything persisted as provenance or logged may read `detail`.
Enforced by `tests/test_errors.py::test_unexpected_error_does_not_leak_filesystem_paths`.
read `message`; anything persisted as provenance or logged may read `detail`. When a UI
surface needs to show persisted `error_detail`, it must route through a sanitizing
projection that preserves the category, suggestion, and error reference while reducing
machine-local absolute paths to basenames only.
Enforced by `tests/test_errors.py::test_unexpected_error_does_not_leak_filesystem_paths`
and `tests/test_error_message_safety.py`.
## Operator Recovery Guidance
+16 -3
View File
@@ -1,6 +1,6 @@
# Document Transcription System Overview (Current Baseline: V5.1)
# Document Transcription System Overview (Current Baseline: V6.1)
This directory is the single source of truth for current V5.1 behavior and architecture.
This directory is the single source of truth for current V6.1 behavior and architecture.
## Canonical Reading Order
@@ -17,7 +17,20 @@ This directory is the single source of truth for current V5.1 behavior and archi
- [Digital Evidence and AI Processing Provenance](./invariant/ai_evidence_and_provenance.md)
- [UI Style Guide](./invariant/ui_style_guide.md)
## Deployment and Operations
- [Production Runbook](production-runbook.md) for deploy, rollback, and recovery.
- [Backup and Restore](backup_restore.md) for backup configuration and restore procedure.
- [Data Migration](data_migration.md) for the SQLite to PostgreSQL migration path.
- [Cloudflare Tunnel and Access](cloudflare_tunnel_access.md) for remote exposure and access control.
## Baseline Statement
The current V5.1 baseline includes behavior delivered through the architectural cleanup phases and person-schema redesign.
The current V6.1 baseline includes the architectural cleanup, person-schema redesign,
containerized PostgreSQL deployment, and the navigation, Document Detail, and worker-backed
maintenance refinements reflected across this canonical document set.
Use this `docs/*` canonical set for active design and implementation decisions.
Every canonical document above states this same baseline; `tests/test_meta_contract_guards.py`
fails if one of them falls behind. Forward-looking work is tracked in
[`roadmap_plan.md`](roadmap_plan.md) and is not part of the baseline.
+5 -1
View File
@@ -123,11 +123,15 @@ Evaluation should:
5. Preserve the exact model, endpoint or route, parameters, prompt, source digest, and scoring method for every comparison.
6. Treat model rankings as corpus- and version-specific, not permanent declarations of a universal “best” model.
The deterministic scorer for these comparisons lives in `src/transcription/benchmarking.py`; it is
retained as evaluation-policy infrastructure even though application runtime paths do not call it
directly.
Benchmark material containing family records remains private application data unless explicitly approved for publication.
## 6. Ownership and Change Policy
1. Canonical V4 architecture, schema, requirements, and error-policy documents define how current behavior satisfies this invariant.
1. Canonical V6.1 architecture, schema, requirements, and error-policy documents define how current behavior satisfies this invariant.
2. Provider adapters own the capture of provider-boundary evidence.
3. Services own validation, persistence, retention, and export behavior.
4. UI pages may inspect evidence through service contracts but do not define evidence semantics.
+32 -1
View File
@@ -12,6 +12,7 @@ This runbook is the operational checklist for releasing and monitoring the trans
- `OPENROUTER_API_KEY`
- `DATABASE__*`
- filesystem paths for data/logs/backups.
- `CLOUDFLARE_TUNNEL_TOKEN`
5. Confirm schema contract alignment is current:
- `src/transcription/db/models.py`
- `docs/schema.md`
@@ -19,9 +20,17 @@ This runbook is the operational checklist for releasing and monitoring the trans
## 2. Release execution steps
1. Deploy artifact/config to target environment.
- Production stack: `docker compose -f docker-compose.production.yml up -d --build`
- `Settings` loads from explicit `_env_file`, then `ENV_FILE`, then the repository-root `.env.production`; it does not resolve relative to the process working directory.
- For Runtime Settings writes in production, mount `.env.production` into the app container and set `RUNTIME_SETTINGS_ENV_FILE=/app/.env.production`.
- If deployment uses a non-default env-file location, set both `ENV_FILE` and `RUNTIME_SETTINGS_ENV_FILE` to that absolute path so startup reads and Settings-page writes stay aligned.
- For SQLite -> PostgreSQL cutover, run `uv run python tools/export_import_migration.py verify --source-db <sqlite-path-or-url> --target-db <postgres-url>` before switching runtime.
2. Validate service startup:
- `/healthz` responds `200`
- `worker.state` is `running`
- if `RUN_EMBEDDED_WORKER=true`, `worker.state` is `running`
- if `RUN_EMBEDDED_WORKER=false`, validate `worker` container is running in Compose
(worker healthcheck is intentionally disabled because it does not expose `/healthz`)
- validate `cloudflared` logs show active tunnel routes and no ingress errors
3. Execute one smoke workflow:
- create a document/job with at least one source
- verify terminal job outcome updates
@@ -29,6 +38,8 @@ This runbook is the operational checklist for releasing and monitoring the trans
4. Verify log flow:
- stdout aggregation receives events
- file logs are written under `./data/logs`
5. Create a fresh PostgreSQL backup after successful deployment:
- `sh deploy/backup/create_postgres_backup.sh` (creates DB dump plus uploads/prompts backups under `BACKUP_DIR`)
## 3. Rollback triggers and actions
@@ -46,6 +57,9 @@ This runbook is the operational checklist for releasing and monitoring the trans
4. Preserve incident evidence:
- `./data/logs`
- relevant DB rows (`job`, `job_source`, `execution_attempt`)
5. If persistence regression is confirmed, restore the latest valid DB dump:
- `sh deploy/backup/restore_postgres_backup.sh <dump-file>`
- Synology media mirror and paired config snapshot (same timestamp) are restored automatically when present.
## 4. Post-release monitoring checklist
@@ -83,6 +97,23 @@ This runbook is the operational checklist for releasing and monitoring the trans
2. Confirm failures are category-aligned (`external`/`timeout`/`internal`).
3. Triage whether issue is source quality, provider, or runtime regression.
### Cloudflare ingress/access failure
1. Check `cloudflared` container logs for ingress parse, DNS, or auth failures.
2. Confirm `deploy/cloudflared/config.yml` hostname mappings are correct.
3. Confirm `CLOUDFLARE_TUNNEL_TOKEN` in `.env.production` matches the tunnel configured in Cloudflare.
4. Confirm Cloudflare Access app policy includes the intended identity/group for that hostname.
5. If logs show SRV/DNS failures via `127.0.0.11`, use the compose-defined resolver override
(`dns: 1.1.1.1, 1.0.0.1`) and ensure outbound TCP 443 is allowed.
### Backup or restore failure
1. Verify `postgres` container is healthy and accepting connections.
2. Confirm dump file exists and is non-zero size.
3. Re-run backup/restore scripts with explicit `ENV_FILE` and `COMPOSE_FILE` if using non-default paths.
4. If direct Synology copy fails, keep local backup and resolve mount/network before next backup cycle.
- For LXC setups, use `deploy/backup/mount_synology_cifs.example.sh` as the persistent mount template.
## 6. Dependency upgrade policy
Dependencies are declared in `pyproject.toml` and resolved through the committed
+35 -2
View File
@@ -1,6 +1,9 @@
# System Requirements (Current Baseline: V5.1)
# System Requirements (Current Baseline: V6.1)
These requirements define the active V5.1 contract and align to current implementation.
These requirements define the active V6.1 contract and align to current implementation.
Requirement IDs encode the baseline that introduced them (`REQ-4-*` from V4, `REQ-6-*` from V6) and
are stable. Never renumber an existing ID; retire it explicitly instead.
## Functional Requirements
@@ -41,6 +44,31 @@ These requirements define the active V5.1 contract and align to current implemen
- **REQ-4-041 Partial Failure Visibility:** Mixed page outcomes must be visible at job and page level.
- **REQ-4-042 Retry Support:** Failed and cancelled pages must support targeted retranscription without requiring full document recreation.
### Person Imagery
- **REQ-6-001 Photo Records:** The system must store reusable `Photo` records that are either owned by a `Person` or unowned for homepage gallery use.
- **REQ-6-002 Primary Photo:** At most one photo per owning `Person` may be marked `is_primary`; setting a new primary must clear the previous one.
- **REQ-6-003 Primary Reassignment:** Deleting an owner's primary photo must promote a remaining photo of that owner rather than leaving the owner without a primary.
### Operational Maintenance
- **REQ-6-010 Queued Maintenance Runs:** Settings-initiated maintenance must persist a `MaintenanceRun` and execute in the worker, not inline in the request that started it.
- **REQ-6-011 Maintenance Run Types:** `MaintenanceRun.job_type` must use one of `backup`, `storage_reconciliation`, `gedcom_import`.
- **REQ-6-012 Maintenance Status Lifecycle:** `MaintenanceRun.status` must use one of `queued`, `processing`, `succeeded`, `failed`.
- **REQ-6-013 Single Claim:** A queued run must be claimed by at most one worker, using a conditional status update rather than read-then-write.
- **REQ-6-014 Run History:** Completed runs must retain status, timing, summary, log reference, and error detail, and expose the log for viewing and download.
- **REQ-6-015 GEDCOM Upload Import:** Settings must support manual `.ged` upload and queue-backed import into genealogy tables.
- **REQ-6-016 GEDCOM Idempotent Upsert:** GEDCOM import must upsert `GenealogyPerson` and `GenealogyFamily` by FamilySearch IDs and avoid duplicate imported citations on re-run.
### Deployment and Runtime Configuration
- **REQ-6-020 Production Persistence:** Production must run against PostgreSQL; SQLite remains supported for local development and tests.
- **REQ-6-021 Split Worker Deployment:** Production must support running the worker as its own process with the app started at `RUN_EMBEDDED_WORKER=false`.
- **REQ-6-022 Runtime Settings Persistence:** Runtime settings edits must persist to the mounted production environment file and survive container restart.
- **REQ-6-023 Configuration Contract Sync:** `.env.production.example` must stay synchronized with `Settings` keys and production-safe defaults.
- **REQ-6-024 Health Reporting:** The deployed stack must report app and worker health through `/healthz`.
- **REQ-6-025 Backup and Restore:** Database and media/config backups must be produced on a host-visible path with a tested restore procedure.
## Non-Functional Requirements
- **REQ-4-100 Boundary Integrity:** UI pages/components must not access persistence directly and must call service APIs.
@@ -73,6 +101,11 @@ These requirements define the active V5.1 contract and align to current implemen
- UI boundary enforcement: `tests/test_ui_boundaries.py`
- Job lifecycle reliability and terminal status behavior: `tests/services/test_workflows_reliability.py`
- Evidence append-only and projection behavior: `tests/services/test_store.py`, `tests/services/test_transcription_service.py`
- Person photo ownership and primary selection: `tests/services/test_photo_service.py`
- Maintenance run lifecycle and worker execution: `tests/services/test_maintenance_service.py`
- Runtime settings persistence: `tests/ui/test_runtime_settings_store.py`, `tests/services/test_settings_services.py`
- Configuration contract synchronization: `tests/test_meta_contract_guards.py`
- Deployment health reporting: `tests/api/test_health.py`
## Traceability Notes
+10
View File
@@ -8,6 +8,16 @@
**Escalations applied:** `.github/skills/evidence-provenance-auditor/skill.md`, `.github/skills/test-effectiveness-auditor/skill.md`
**Scope:** 77 Python modules / ~13k LOC under `src/transcription`, 57 test files (377 collected non-external tests), 23 documents under `docs/`, 9 active rule files.
> **Status: closed.** Every finding below was remediated in the phases following this
> review. This document is retained as a record of the reasoning, **not** as a list of
> open work, and it is not canonical authority.
>
> Two recommendations were wrong on contact and were corrected during implementation:
> the HIGH-03 fix as written would have stripped root-cause data from `ExecutionAttempt`
> provenance, and the HIGH-01 fix needed to preserve per-page durability that the report
> did not mention. Where this text and the current code or guard tests disagree, the code
> and tests are correct.
### Verification commands and outcomes
| Command | Outcome |
@@ -1,258 +0,0 @@
# Handoff Brief — Phases 2-5 of the 2026-08-23 Code Review
**Repo:** `C:\Github\transcription` · **Branch:** `traumatized` · **Baseline commit:** `de18c2e`
**Source of truth:** `docs/reviews/2026-08-23-code-review.md` (§9 Prioritized Action Plan)
Phase 1 is **done and committed**. This brief covers everything after it.
> **Status note.** This is a dated, non-canonical artifact, like everything under
> `docs/reviews/**`. It records a plan, not a contract. Where it disagrees with
> `.github/instructions/**` or the canonical docs, **they win**.
---
## 1. Environment — read this first
- `uv` project on **Windows / PowerShell**. `ruff`, `ty`, and `pytest` are **not on PATH**. Always prefix with `uv run`.
- PowerShell has **no heredoc**. Don't write `python - <<'PY'`. Use `python -c "..."` or pipe a single-quoted here-string (`@'``'@ | python -`).
- `&&` only chains *external* commands in PowerShell. Use `;` before PowerShell keywords.
- Ruff config (`ruff.toml`): line length **120**, `force-single-line = true`**one import per line**. Never combine imports.
- `# noqa: PLR0915` / `PLR1702` is established repo convention; don't strip existing ones.
## 2. Mandatory reading before editing `src/transcription/**`
The repo's instruction table requires these before source edits. They are contracts, not suggestions:
- `.github/instructions/services.instructions.md` — transaction boundaries, model ownership, no service-to-service imports
- `.github/instructions/error-handling.instructions.md` — error categories, retriability, user-safe messaging
- `.github/instructions/ui.instructions.md` — page/component boundaries
- `.github/instructions/documentation-sync.instructions.md`**docs must be updated in the same change** when contracts or behavior change
## 3. Verification commands
```powershell
uv run ruff check . # must be clean
uv run pytest -q -m "not external" # must be 381+ passing
uv run ty check # baseline is exactly 10 diagnostics
```
**The `ty` baseline is 10, and all 10 are false positives** — SQLModel/SQLAlchemy column
descriptors typed as `UUID`/`datetime`/`bool`, so `.is_()`, `.asc()`, `func.count()`, and
`group_by()` appear invalid. They are in `services/photos.py` (8) and
`tests/test_storage_reconciliation.py` (2). **Do not "fix" these by changing code.** Handling
them is task P2-1 below, and the fix is suppression comments, not code edits.
Hard-won gotcha: `typing.Mapping` trips ruff's `deprecated-import`, and `collections.abc.Mapping`
doesn't satisfy `ty` for SQLAlchemy row results. `Sequence[RowMapping]` + `RowMapping` is the
only spelling that satisfies both. Don't rediscover this.
---
## 4. What Phase 1 changed (context you need)
Commit `de18c2e`. Four things, all with tests:
1. **`workflows.py` commit boundary.** `process_queued_job` now commits every page except the
last individually, then defers the final page's write into `_finalize_batch_outcome` so it
shares the terminal-status transaction.
- **`_finalize_batch_outcome` gained a `final_page` kwarg.** If you touch this function, that
parameter is load-bearing.
- **Two invariants are in tension here — preserve both.** Intermediate pages must stay
individually durable (guarded by
`test_workflows_reliability.py::...::test_transcribed_page_is_committed_before_next_provider_call_finishes`),
and the final page must be atomic with the terminal status (guarded by
`tests/integration/test_pipeline_atomicity.py`). **Do not collapse the whole batch into one
transaction** to simplify things — that breaks multi-page durability.
2. **`AppError` gained an internal-only `detail` field.** `message` is user/API-facing and must
stay generic; `detail` carries the root cause and flows into evidence records via
`format_error_detail` and into logs. Documented in `docs/error_handling.md` §"Message vs
detail split". **When adding error paths: never put exception text into `message`.**
3. **8 `ui.notify` error sites replaced with `show_error`** in `home_page.py` / `people_page.py`.
4. **New AST guard** `test_ui_boundaries.py::test_no_page_hand_rolls_error_notifications`.
---
## 5. Phase 2 — Enforcement hardening
### P2-1 · `ty` suppression policy, then make the hook blocking
**Report ref:** LOW-05 · **Effort:** M
`.pre-commit-config.yaml` currently runs `ty` in **advisory** mode (a Python subprocess wrapper
that forces `sys.exit(0)`) because of the 10 known false positives. Net effect: a genuine new
type error prints alongside the known 10 and **blocks nothing**.
1. Add a targeted `# ty: ignore[<rule>]` at each of the 10 sites, each with a one-line comment
explaining it's a SQLAlchemy descriptor false positive.
2. Confirm `uv run ty check` reports **0**.
3. Flip the pre-commit hook to blocking (drop the `sys.exit(0)` wrapper).
4. Document the policy in `docs/` — when a suppression is acceptable and what the comment must say.
**Acceptance:** `uv run ty check` → 0 diagnostics; introducing a deliberate type error fails
`git commit`; revert the deliberate error afterward.
### P2-2 · `ruff format` enforcement
**Report ref:** LOW-06 · **Effort:** S
~35 files have formatting drift. **Two separate commits, in this order:**
1. `uv run ruff format .` — formatting only, **no other changes in this commit**.
2. Add `ruff format --check` to `.pre-commit-config.yaml`.
Keeping these separate matters: a mixed commit makes the formatting noise unreviewable.
**Acceptance:** `uv run ruff format --check .` clean; full suite still 381+.
### P2-3 · Enable ruff ruleset `G` (flake8-logging-format)
**Report ref:** LOW-09 · **Effort:** S
f-strings in logging calls format eagerly regardless of level and break template grouping in
structured backends. Known instance: `workflows.py:193`. Enable `G` in `ruff.toml`, then convert
offenders to `%s` lazy args: `logger.error("Job %s failed.", job.id)`.
**Acceptance:** `uv run ruff check .` clean with `G` enabled.
---
## 6. Phase 3 — Reliability & concurrency
### P3-1 · Periodic stale-job recovery
**Report ref:** MED-01 · **Effort:** M
`requeue_stale_processing_jobs` has exactly one caller — `app.py:79`, in the lifespan startup
handler. There is no runtime re-check. The threshold reuses `worker_provider_timeout_seconds`
(**30.0s**, `config.py:116`).
The failure mode: a job orphaned <30s before a fast restart fails the staleness predicate at the
only moment recovery runs, so it stays `PROCESSING` forever (the worker only claims `QUEUED`).
Restarts are exactly when orphans are created, so the recovery window is systematically
misaligned with the failure it exists to handle.
1. Add `worker_stale_job_seconds` to `Settings` (don't keep overloading the provider timeout —
they need independent tuning). Update `.env.example` in the same change (required by
`services.instructions.md`).
2. Run the sweep periodically in the worker loop, **in addition to** the startup call.
3. Test: a job left `PROCESSING` past the threshold is requeued **without** a restart.
### P3-2 · Gate retries on error category
**Report ref:** MED-02 · **Effort:** S
`workflows.py:184-194` gates only on `job.retry_count < settings.worker_max_retries`. It never
consults `error_category` or `AppError.retriable`, so `validation` / `not_found` / `conflict`
failures would retry to exhaustion, burning provider quota on calls that cannot succeed.
**Latent today** because `worker_max_retries` defaults to `0` — which is exactly why this must be
fixed *before* anyone raises that value. Add backoff too; retries currently requeue immediately.
**Acceptance:** a `validation`-category failure is not requeued even with `worker_max_retries=1`.
### P3-3 · Shutdown budget derived from provider timeout
**Report ref:** MED-04 · **Effort:** S
`worker.py:146` waits `2.0s` for the worker task, but an in-flight provider call may run 30s and
the stop event is only checked *between* jobs. Derive the budget from
`worker_provider_timeout_seconds` plus a small grace. Document the relationship to the container
termination grace period in `docs/production-runbook.md`.
### P3-4 · Handle `IntegrityError` on the attempt-number flush
**Report ref:** MED-03 · **Effort:** M
`sources.py:540-546` computes `MAX(attempt_number) + 1`; `uq_execution_attempt_number` enforces
uniqueness. The sibling `JobSource` insert catches `IntegrityError` at `sources.py:531-534`, but
the attempt `flush()` at `sources.py:587` does **not** — a race loses an evidence row.
Not reachable today (single worker, sequential sources). **It becomes reachable the moment a
second worker replica is deployed** — treat this as a hard precondition for horizontal scaling
and note that in `docs/production-runbook.md`.
Mirror the `JobSource` handling: catch, recompute, retry bounded, raise a domain error on
exhaustion.
### P3-5 · Move blocking work off the event loop
**Report ref:** LOW-01, LOW-02, LOW-03 · **Effort:** S
- `store.py:401``hashlib.sha256(file_bytes)` is CPU-bound on the loop. Wrap in `asyncio.to_thread`.
- `homepage_store.py:25,32` — sync file I/O called from async page handlers. Same fix.
- `app.py:62``poll_interval_seconds=1.0` hardcoded. Move to `Settings`; update `.env.example`.
Every other I/O path already uses `to_thread` (`media_storage.py:43`, `normalization.py:117`,
`photos.py:176`, `sources.py:740,753`) — follow those.
---
## 7. Phase 4 — Consolidation
### P4-1 · Single transaction entry point
**Report ref:** MED-05 · **Effort:** M
`workflows.py` opens transactions via `services.jobs._session_scope()` and
`services.sources._session_scope()`**private members of two different services**. This is the
mechanism that made HIGH-01 easy to introduce: nothing in the design signals that two scopes are
being opened for one logical unit of work.
Introduce `unit_of_work(services, session)` (proposed signature in review §7), migrate
`workflows.py` onto it, then add an AST guard to `test_service_boundaries.py` forbidding
`_session_scope` access outside its owning module. Note the existing test checks *imports*, not
attribute access, so it can't currently see this.
**Do not attempt this before Phase 1's tests are green in your working tree** — it touches the
same functions.
### P4-2 · Extract shared helpers
**Report ref:** §7 · **Effort:** M
`run_blocking` and `insert_with_sequence_retry`, per the review's proposed signatures. Do this
*after* P3-4 and P3-5, so the call sites exist.
### P4-3 · Prune low-signal tests
**Report ref:** LOW-10, LOW-11 · **Effort:** S
Full table in review §6 "Prune/strengthen backlog". Highlights:
- `test_traceability.py:54-57` — asserts properties of dict literals in the same file.
- `test_pipeline_flow.py:135-140,446-452``assert processed is True` is unfalsifiable
(`read_job` raises rather than returning `None`).
- `test_orphan_sweep.py:119``>= 200` snapshot threshold tolerates ±40 drift.
- `test_workflows_reliability.py:157-196` — real `time.sleep(0.40)` with only 10% slack; flaky
under CI load.
---
## 8. Phase 5 — Governance
- **P5-1** — Extend `test_orphan_sweep.py` beyond module-level definitions to public methods
(LOW-08); seed `KNOWN_ORPHANS` with current results to keep it non-breaking.
- **P5-2** — Resolve the 4 "uncertain" orphans (LOW-07). Note `summarize_error` should now be
reachable — consider using it, or delete it.
- **P5-3** — Log incomplete request manifests instead of silently returning `None`
(`openrouter.py:347`, LOW-04).
- **P5-4** — Consider inverting the UI boundary check from a forbidden-list to an allowlist; a
future persistence helper under a new name currently escapes it.
---
## 9. Ground rules
1. **Red first.** For any behavioral fix, write the test, *run it, observe it fail*, then fix.
That's how Phase 1 caught that its own first HIGH-03 attempt was wrong.
2. **One phase per commit series.** Don't mix P2-2's formatting sweep with logic changes.
3. **Docs in the same change.** Required by `documentation-sync.instructions.md` whenever
contracts, behavior, or `Settings` change. `Settings` changes additionally require
`.env.example` updates in the same commit.
4. **Don't widen the NiceGUI pin.** `nicegui==3.13.0` is a deliberate release-stability decision
recorded in `docs/production-runbook.md`. It is explicitly **not** a defect.
5. **`docs/reviews/**` is not canonical.** It's a dated artifact; don't treat it as a contract
the way `docs/schema.md` or the instruction files are.
6. **Ask before scope-expanding.** If a fix seems to require restructuring beyond its task,
stop and confirm — that's the signal a Phase boundary is being crossed.
## 10. Suggested first command
```powershell
cd C:\Github\transcription
git log --oneline -3
uv run ruff check . ; uv run pytest -q -m "not external" ; uv run ty check
```
Confirm the baseline (clean ruff, 381+ passing, exactly 10 `ty` diagnostics) before changing
anything. If that doesn't reproduce, stop and report rather than proceeding.
+896
View File
@@ -0,0 +1,896 @@
# Architecture & Code Review Report
**Repository Target:** `C:\Github\transcription\`
**Target Stack:** Python 3.12+ | FastAPI | NiceGUI | SQLModel/SQLAlchemy | Pydantic V2 | asyncio | OpenRouter
**Review Date:** 2026-09-02
**Canonical Baseline:** V6.1 (`docs/index.md`)
---
## 0. Verification Commands and Outcomes
All four commands were executed in this checkout before any finding was written. This report
records the exact outcomes rather than assuming them.
| Command | Outcome |
| :--- | :--- |
| `uv run pytest -q -m "not external"` | **410 passed, 0 failed, 0 errors** (exit 0) |
| `uv run ruff check .` | **All checks passed!** |
| `uv run ruff format --check .` | **191 files already formatted** |
| `uv run ty check` | **All checks passed!** |
The stated green baseline is real. No finding below is a test failure; every finding is a
behavior, contract, or guard-coverage defect that the passing suite does not detect.
---
## 1. Executive Summary
- **The system's core evidence guarantees hold.** `ExecutionAttempt` is genuinely append-only,
attempt numbering is allocated with bounded conflict retry, transport evidence is captured at
the HTTP boundary before SDK parsing, and header persistence uses a true allowlist. Provenance
invariant families AE and G pass.
- **Both competing atomicity invariants in `services/workflows.py` are real and both guards
genuinely enforce them.** I injected-fault-verified the tests rather than trusting the
docstrings: `test_pipeline_atomicity.py` fails on a split final-page commit, and
`test_workflows_reliability.py:318` reads intermediate attempts through a *separate session*,
so it would fail if intermediate pages stopped committing individually.
- **The most significant defect is a privacy leak that a prior review believed it had closed.**
The 2026-08-23 review moved root-cause text out of `AppError.message` into `AppError.detail`
to keep filesystem paths away from users. That text now reaches users anyway, because the UI
renders `ExecutionAttempt.error_detail` verbatim (HIGH-01). The leak was relocated, not closed.
- **A second, independent path leak exists in five explicit `raise` sites** that the existing
guard never covered — it tests only `classify_unexpected_error` (HIGH-02).
- **Provenance invariant family F (path safety) fails**, and it fails *inconsistently within one
file*: `sources_page.py:443` carefully sanitizes a stored path through
`public_media_path_label`, then `sources_page.py:484` dumps raw `error_detail` forty lines later.
- **The orphan sweep does not do what its docstring claims.** It matches definitions by bare name,
so an entirely dead *module* passes whenever its function names collide with live ones.
`ui/pages/tags_page.py` is the proof: 93 lines never imported by anything (MED-01/LOW-01).
- **On the three flagged open items:** the V4/V6.1 doc drift is confirmed (MED-02); the
`.env.production` coupling is real but currently correct and loud-failing, so Medium not High
(MED-03); and the Tags roadmap is **right** — the route is genuinely not registered, so the
module is dead code rather than a live retired route.
- **Two latent concurrency defects carry ordering constraints** and must be fixed *before* the
changes that would make them live (MED-04, MED-05), not after.
- **Guidance-file accuracy:** the recently revised `.github/instructions/*` files were verified
against code rather than trusted. They are accurate as written; the code is what diverges from
them. The one exception is that `error-handling.instructions.md` states a `detail` rule the UI
layer has never followed, which makes it an unenforced claim rather than a wrong one.
---
## 2. Executive Architecture Assessment
**Verdict: architecturally sound, with a concentrated failure in the *last mile* of error
presentation.**
Domain cohesion and dependency direction are good and, unusually, mechanically enforced.
`test_service_boundaries.py` and `test_ui_boundaries.py` AST-scan for violations using
*allowlists* rather than blocklists, which is the correct choice — a newly added persistence
helper cannot slip through under an unlisted name. `workflows.py` imports only the abstract
`providers` types and never `openrouter`, so provider details genuinely stop at the adapter.
Transaction ownership is explicit and well-reasoned: `ServiceBase._finalize` commits for
service-owned sessions and flushes for caller-owned ones, which is what lets orchestration
modules compose multi-aggregate writes without services importing each other.
The evidence layer is the strongest part of the system and shows real care. The distinction
between transport response, SDK-parsed response, and normalized metadata is maintained in code,
not just in prose — `_CapturingAsyncClient` exists specifically to retain the exact wire body
before the SDK can discard unknown fields, and `TransportEvidence(response_received=False)`
explicitly represents "no response was received" rather than conflating it with an empty one.
The weakness is at the boundary where internal diagnostic text becomes pixels. Every layer
*below* the UI respects the message/detail split; the UI layer reads the internal field directly
and renders it. The architecture defines the contract correctly and then has no enforcement at
the one layer that violates it.
**Top systemic risks:**
1. **Internal diagnostic text reaches users through the evidence display path** (HIGH-01). The
rule is documented in three places and enforced in none of them at the UI boundary.
2. **Path-safety discipline is applied per-call-site rather than structurally** (HIGH-02, HIGH-01).
It is correct wherever someone remembered; there is no guard that makes forgetting fail.
3. **Guard coverage is narrower than guard docstrings claim.** Two guards
(`test_orphan_sweep.py`, `test_errors.py`) assert something meaningfully weaker than the
invariant they are named for, which converts them into a false sense of enforcement.
4. **Worker safety currently rests on single-process sequential execution, not on configuration**
(MED-04, MED-05). Nothing is wrong today; two plausible future changes each make something wrong.
---
## 3. Findings by Severity
### Critical Severity
*None.* No evidence loss, append-only violation, secret leakage, or silent-wrong-output defect
was found. The candidates in this class (provider evidence mis-attribution, stale-job double
processing) are latent and are reported at High/Medium with their unblocking conditions.
---
### High Severity
#### [HIGH-01] Internal-only `error_detail` is rendered directly to users, reopening the leak the 2026-08-23 fix was meant to close
- **Location:**
- Write side: `src/transcription/errors.py:99-139` (`classify_unexpected_error``detail`, `format_error_detail` → persisted text)
- Persist: `src/transcription/services/workflows.py:720` (`error_detail=format_error_detail(page.error)`)
- **Render (Source Detail):** `src/transcription/ui/pages/sources_page.py:481-484`
- **Render (Sources list):** `src/transcription/ui/pages/sources_page.py:121``src/transcription/ui/components/table/sources.py:39,90-95` ("Error Detail" column)
- **Render (Maintenance):** `src/transcription/ui/pages/settings_page.py:562`, written by `src/transcription/services/maintenance.py:206`
- Contract violated: `docs/error_handling.md:107-114`; `.github/instructions/error-handling.instructions.md:86`; `docs/invariant/error_handling.md:59`; `docs/invariant/ai_evidence_and_provenance.md:103`
- **Reachability:** **Live.** Concrete path, no configuration required: a page fails with any
non-`AppError` exception → `workflows.py:388` calls `classify_unexpected_error(exc)`
`errors.py:118` sets `detail=f"{type(exc).__name__}: {exc}"``format_error_detail`
(`errors.py:135-139`) emits `... | detail=OSError: [Errno 13] Permission denied: '/app/uploads/documents/<uuid>/page-1.jpg' | ...`
→ persisted to `ExecutionAttempt.error_detail` → rendered verbatim at
`sources_page.py:484` and in the `/sources` table column. A SQLAlchemy `OperationalError`
carries the database path by the same route.
- **Problem & Consequence:** `docs/error_handling.md:110` states `detail` is *"Internal only"* and
that its only surfaces are `format_error_detail` (evidence) and logs;
`error-handling.instructions.md:86` says *"Never rendered to users or serialized into an
envelope."* The UI reads it anyway. The consequence is not hypothetical drift — it is the
precise defect the previous review's fix existed to prevent. That fix made `message` generic and
moved the root cause to `detail` on the stated grounds that `detail` never reaches users. That
premise was never true: `error_detail` had a UI consumer the whole time. The result is that the
filesystem-path leak was relocated from the notification banner to the Source Detail card and
the Sources table, while the test suite records the leak as fixed
(`tests/test_errors.py:56-78`).
The inconsistency is visible inside a single file: `sources_page.py:443` deliberately routes a
stored path through `public_media_path_label` (`ui/components/media_urls.py:58-72`), which
correctly degrades an absolute path to its bare filename — and then `sources_page.py:484`
renders unsanitized text that may contain an absolute path.
- **Blast Radius:** Enumerated by grepping every reader of `.detail` and `error_detail`:
- `errors.py:137``format_error_detail`, the only reader of `AppError.detail`. **Must keep the root cause.**
- `services/workflows.py:720` — the only writer of `ExecutionAttempt.error_detail`.
- `services/maintenance.py:206` — the only writer of `MaintenanceRun.error_detail`.
- `services/evidence.py:195``build_evidence_export` emits `error_detail`. Export is an
operator-initiated evidence artifact; per invariant 3.7.1 it **must** retain it.
- `db/models.py:508-522``Source.latest_error_detail` projection, consumed only by `sources_page.py:121`.
- `ui/pages/sources_page.py:481-484`, `ui/components/table/sources.py`, `ui/pages/settings_page.py:562` — the three render sites.
- Tests asserting on persisted text: `tests/test_v42_evidence.py:284`,
`tests/services/test_workflows_reliability.py` (timeout detail),
`tests/services/test_maintenance_service.py`. A fix that changes *what is stored* breaks these;
a fix that changes *what is displayed* does not.
- **Recommendation — two invariants conflict here; both must be named.**
**Invariant 1 (evidence):** `ExecutionAttempt.error_detail` must retain the root cause.
`docs/requirements.md:30` (REQ-4-021) and `docs/invariant/ai_evidence_and_provenance.md:33`
require it; guarded by `tests/test_v42_evidence.py::test_attempts_are_append_only_and_exported_with_integrity`
and `tests/services/test_workflows_reliability.py`.
**Invariant 2 (privacy):** user-facing surfaces must not expose local filesystem details.
`docs/invariant/error_handling.md:59`; guarded (partially) by
`tests/test_errors.py::test_unexpected_error_does_not_leak_filesystem_paths`.
**The over-correction to avoid is stripping root-cause text out of `detail` or
`format_error_detail` to make the UI safe.** That is exactly the mistake documented in the
reviewer skill's worked example, and it would silently destroy the provenance record this
system exists to preserve while making every guard still pass.
Fix at the **render** boundary, not the write boundary. Add a presentation-layer projection and
route all three UI sites through it, leaving the persisted evidence untouched:
```python
# src/transcription/ui/components/error_presenter.py (new)
def display_failure_detail(error_detail: str | None) -> str | None:
"""Render persisted failure detail without machine-local paths.
`ExecutionAttempt.error_detail` is provenance and keeps the full root cause
(docs/error_handling.md). This projection is the only thing a page may show.
"""
```
It should preserve the `[category]`, `suggestion=`, and `error_id=` segments (which are what
make the display actionable) and reduce any absolute path inside `detail=` to its basename,
mirroring `public_media_path_label`. The operator keeps diagnosability — required by
`docs/ui/pages/sources.md:43` and `docs/requirements.md:59` (REQ-6-014) — without the container
filesystem layout being published to the browser.
Then decide and record which resolution was chosen: either the UI shows the sanitized
projection (recommended), or `docs/error_handling.md:107-114` and
`error-handling.instructions.md:86` are revised to state that operator-facing evidence displays
may render `error_detail` **and** that the guarantee moves to "no machine-local detail ever
enters `detail`" — which would be a much harder guarantee to keep. Do not leave the current
state, where the docs claim one thing and three pages do another.
- **Effort:** M
---
#### [HIGH-02] Absolute filesystem paths are embedded in user-facing `AppError.message` at five explicit raise sites
- **Location:**
- `src/transcription/services/sources.py:856``f"Prompt file not found: {prompt_path}"`
- `src/transcription/services/sources.py:864``f"Prompt file is empty: {prompt_path}"`
- `src/transcription/services/sources.py:914``f"Source file not found: {path}"`
- `src/transcription/services/prompts.py:99``f"Prompt directory is unavailable: {root}"`
- `src/transcription/services/prompts.py:186-191``_filesystem_error` builds `f"{message}: {exc}"`
- Contract violated: `.github/instructions/error-handling.instructions.md:74,85`; `docs/invariant/error_handling.md:59`
- **Reachability:** **Live**, on an ordinary user path. `sources.py:845` resolves
`prompt_root = runtime_settings.prompt_dir.resolve()`, so `prompt_path` is absolute
(`/app/prompts/transcribe_document.md` in the container). `load_prompt_text` is invoked by
`build_prompt_execution` (`sources.py:829-831`), which runs on **every document upload** via
`services/store.py:94` and `store.py:162`. The resulting `PromptLoadError` is an `AppError`
subclass, so it flows through `run_ui_action``show_error`
(`ui/components/error_presenter.py:51-66`), which renders `error.message` into both a
`ui.notify` banner and a card label, and through `build_error_envelope` (`errors.py:88-96`)
into API responses.
- **Problem & Consequence:** `error-handling.instructions.md:85` requires `message` to *"Stay
generic. Never embed exception text, provider payloads, or filesystem paths."* These five sites
embed exactly that. `prompts.py:186-191` violates the rule in **both** directions at once: it
puts `{exc}` — an `OSError` whose `str()` includes the offending filename — into `message`, and
it sets **no `detail=`**, so the internal field that is supposed to carry the root cause is
empty while the user-facing field carries all of it.
This is not a new regression; it is coverage that the existing guard never had.
`tests/test_errors.py:56-78` verifies only that `classify_unexpected_error` — the *catch-all*
path — does not leak. Every deliberate `raise SomeError(f"... {path}")` in the codebase is
outside its scope, so the suite reports the invariant as enforced while five live sites violate it.
- **Blast Radius:** Verified by grepping all consumers of these exception types.
`PromptLoadError`/`PromptStoreError`/`TranscriptionError` messages are consumed by:
`ui/components/error_presenter.py:55,63` (render), `errors.py:92` (API envelope),
`errors.py:135` (`format_error_detail` → evidence). Because the recommended change *adds* a
`detail` and *shortens* `message`, `format_error_detail` output still contains the path — so
evidence value is preserved, not reduced. Tests asserting on these messages:
`tests/test_prompts.py`, `tests/services/test_prompt_store.py`,
`tests/services/test_transcription_service.py`. These assert on message prefixes
(`"Prompt file not found"`), not on the interpolated path, and were checked to survive the change —
but re-run them, since `prompts.py:186` currently produces a message whose suffix some
assertion could depend on.
- **Recommendation:** Apply the pattern `errors.py:113-119` already establishes — generic
`message`, root cause on `detail`, `raise ... from exc`. Use `path.name` when a filename is
genuinely useful to the user.
```python
# sources.py:855 — before
raise PromptLoadError(f"Prompt file not found: {prompt_path}", ...)
# after
raise PromptLoadError(
f"Prompt file not found: {prompt_path.name}",
category=ErrorCategory.INFRA_PERSISTENT,
suggestion="Verify PROMPT_DIR and prompt file configuration, then retry.",
detail=f"Prompt file missing at {prompt_path}",
)
# prompts.py:186 — before
return PromptStoreError(f"{message}: {exc}", category=..., suggestion=...)
# after
return PromptStoreError(
message,
category=ErrorCategory.INFRA_PERSISTENT,
suggestion="Check prompt directory permissions and available disk space, then retry.",
detail=f"{type(exc).__name__}: {exc}",
)
```
Then widen the guard so this class cannot recur — see MED-07. Note the dependency: HIGH-02 and
HIGH-01 must be fixed **together**, because moving the path from `message` to `detail` while the
UI still renders `error_detail` relocates the leak instead of closing it. That is the same
mistake that produced HIGH-01.
- **Effort:** S (fix) / M (with the guard)
---
### Medium Severity
#### [MED-01] The orphan sweep matches by bare name and therefore cannot detect a dead module
- **Location:** `tests/test_orphan_sweep.py:101-163` (`_public_definitions`, `_orphans`)
- **Reachability:** **Live** — the guard is running now and reporting a clean sweep that is not clean.
- **Problem & Consequence:** `_public_definitions()` keys definitions by bare name
(`definitions[node.name]`, line 111) and `_orphans()` marks a definition referenced if that
bare name appears **anywhere** in `src/`, `tests/`, or `tools/` (lines 157-162). Two different
modules that define the same public name are therefore indistinguishable, and neither can ever
be reported as an orphan.
`src/transcription/ui/pages/tags_page.py` demonstrates the consequence. Its only public
definition is `register_page` (line 18). Seven live page modules define a function of the same
name and `ui/__init__.py:37-43` calls all seven — so `register_page` is heavily referenced and
`tags_page.register_page` is scored as reachable. In fact **nothing imports `tags_page` at all**
(verified: the only repo-wide references to the module are the file itself and
`tests/ui/test_tags_page.py`, which merely asserts the route 404s). 93 lines of code, including a
lazy-load-unsafe relationship traversal at `tags_page.py:71-74`, sit outside the sweep's reach.
The sweep also never asks whether a *module* is imported, only whether its definitions' names
appear somewhere — so this is a structural gap, not a one-off miss.
- **Blast Radius:** `tests/test_orphan_sweep.py` only; `KNOWN_ORPHANS` entries are keyed by the
same bare/dotted names and would need re-keying if qualification is added. Expect the stricter
sweep to surface additional true orphans on first run — triage them into `KNOWN_ORPHANS` with
rationales rather than weakening the check.
- **Recommendation:** Qualify definitions by module (`f"{module_path}:{name}"`) and add a separate,
cheap module-reachability pass: a module under `src/transcription/` is reachable if any other
module imports it, or it is a declared entrypoint (`app.py`, `__main__.py`, `worker_service.py`).
Report unreachable modules as orphans in their own right. Also fix
`test_public_definitions_are_discovered` (line 169), whose `>= 420` snapshot threshold is a
weak assertion that drifts upward silently — the 2026-08-23 review already flagged the same
pattern at the then-current `>= 200` and it was raised rather than replaced.
- **Effort:** M
---
#### [MED-02] Canonical invariant document declares a V4 baseline while the canonical baseline is V6.1
- **Location:** `docs/invariant/ai_evidence_and_provenance.md:130`
- **Reachability:** **Live** (documentation), no runtime impact.
- **Problem & Consequence:** Section 6.1 reads *"Canonical V4 architecture, schema, requirements,
and error-policy documents define how current behavior satisfies this invariant."*
`docs/index.md:1,29-32` establishes V6.1 as the baseline and states that every canonical document
asserts the same baseline. This is the **ownership clause of the invariant that governs the
entire evidence model** — the clause that tells a reader which documents are authoritative — and
it points at a superseded generation. A reader following it lands on stale authority precisely
when resolving an evidence question, which is the highest-stakes case.
A baseline-currency guard **does** exist —
`tests/test_meta_contract_guards.py::test_canonical_docs_declare_one_consistent_baseline`
(lines 89-112) — and `docs/invariant/ai_evidence_and_provenance.md` is **not** in
`BASELINE_SCAN_EXCLUSIONS` (lines 56-64), so the file is scanned. The claim escapes for two
independent reasons, either of which alone would be sufficient:
1. `_CURRENT_VERSION_CLAIM` (line 67) matches only the words `current` or `active` before a
version. This line says "**Canonical** V4", a third phrasing the pattern does not know.
2. Both patterns require `V(\d+\.\d+)` — a mandatory minor version. The bare token `V4` cannot
match either regex under any phrasing.
The guard is therefore not absent but *phrase-shaped*: it enforces currency only for the two
sentence forms someone thought of, against version strings that carry a minor. That is a weaker
property than its docstring implies ("Every canonical doc that names the current baseline must
name the same one").
- **Blast Radius:** Documentation only; no code reads this string. Widening the guard's patterns
will re-scan all canonical docs — expect it to surface further stale mentions on first run
(`docs/architecture.md`, `docs/schema.md`, `docs/requirements.md`, and `docs/error_handling.md`
each contain 2-3 version tokens), which should be triaged rather than excluded.
- **Recommendation:** Two parts, and the second matters more than the first.
1. Change "Canonical V4" to "Canonical V6.1" at line 130.
2. Fix the guard's shape rather than adding a third phrase to the list. Accept an optional minor
(`V(\d+)(?:\.(\d+))?`) and invert the matching: flag **every** `V<n>` token in a scanned
canonical doc that is not the declared baseline, rather than only those preceded by an
approved adjective. Phrase-list matching fails open — each new phrasing silently reopens the
hole — whereas token matching fails closed and forces an explicit exclusion.
See §8.1 for the alternative the maintainer is considering: dropping version labels from
canonical docs entirely, which removes the failure mode instead of guarding it.
- **Effort:** S
---
#### [MED-03] Settings resolve `.env.production` relative to the process working directory, and the isolation fix exists only in the test harness
- **Location:** `src/transcription/config.py:66-75` (`env_file=".env.production"`);
workaround at `tests/conftest.py:27-50`; guarded by `tests/test_config_isolation.py`;
depended on by `.github/workflows/quality-gate.yml` and `docker-compose.production.yml`
- **Reachability:** **Live but currently correct.** I verified the production path rather than
assuming it: `Dockerfile` sets `WORKDIR /app` in the runtime stage, and
`docker-compose.production.yml` mounts `./.env.production` to `/app/.env.production` for both the
`app` and `worker` services, so the relative path resolves correctly today.
- **Problem & Consequence:** Correct configuration loading depends on an **implicit, undocumented
contract between `config.py` and the process working directory.** Nothing in `config.py` states
it, and nothing tests it. The failure mode is not silent — `openrouter_api_key` is required with
no default, so a wrong cwd produces a `ValidationError` at startup rather than a partially
configured process — which is why this is Medium rather than High.
The more telling symptom is what the coupling forced on the test harness. `conftest.py:45-50`
cannot escape it by passing an argument; it must **mutate the Pydantic class-level
`model_config` dict at runtime** and restore it in a `finally`. That is a global, order-sensitive
side effect adopted because the module offers no seam. It also silently repairs a second
consumer: `ui/runtime_settings_store.py:402` reads the same `Settings.model_config["env_file"]`
to decide where the Settings page writes. Two subsystems are coupled through a mutable class
attribute.
- **Blast Radius:** Every `Settings` construction. Consumers of `model_config["env_file"]`:
`ui/runtime_settings_store.py:402` (write-target resolution, contract documented at
`docs/ui/pages/settings.md:27`) and `tests/conftest.py:45-50`. A change must preserve the
documented three-step resolution order — explicit override, `RUNTIME_SETTINGS_ENV_FILE`, then the
configured default — or `docs/ui/pages/settings.md:27` becomes wrong.
- **Recommendation:** Introduce one explicit resolution function that both `Settings` construction
and `runtime_settings_store` call, honoring an `ENV_FILE` environment variable and falling back
to a path anchored to a known root rather than to `os.getcwd()`. Tests then pass a path instead of
mutating class state, and `tests/test_config_isolation.py` can assert against the seam rather
than against the monkeypatch. If instead the cwd contract is accepted as deliberate, document it
in `config.py` and in `docs/production-runbook.md` and add a guard asserting `WORKDIR`/cwd
alignment — an implicit contract with a container image is exactly the kind of rule the invariant
routing table exists to place.
- **Effort:** M
---
#### [MED-04] Stale-job reclaim threshold is not derived from maximum job duration; safety currently comes from single-process sequencing
- **Location:** `src/transcription/config.py:116-117`
(`worker_provider_timeout_seconds=30.0`, `worker_stale_job_seconds=30.0`);
sweep at `src/transcription/worker.py:222-228`; reclaim at
`src/transcription/services/jobs.py:242-268`
- **Reachability:** **Latent.** Unblocked by *either* of: (a) running more than one worker replica
(adding `deploy.replicas > 1` to the `worker` service in `docker-compose.production.yml`), or
(b) setting `RUN_EMBEDDED_WORKER=true` on the `app` service while the standalone `worker`
container is also running. It is safe today only because
`docker-compose.production.yml` sets `RUN_EMBEDDED_WORKER: "false"` on `app` and defines exactly
one `worker`, and because within a single loop `run_worker_loop` awaits
`process_next_queued_job` to completion before returning to the stale sweep — so the sweep can
never observe a job that this same process is actively working.
- **Problem & Consequence:** The stale threshold (30s) **equals** the per-page provider timeout
(30s), leaving zero margin even for a single-page job. A multi-page document is legitimately
`PROCESSING` for up to N × 30s. `Job.date_updated` carries an `onupdate`
(`db/models.py:372-375`), but between the initial claim and the terminal write the only touch
is `sources.py:522-523` reassigning `job.provider`/`job.model` to values they usually already
hold, which SQLAlchemy resolves to no net change and therefore no `UPDATE`. I did not empirically
confirm the no-`UPDATE` behavior, so treat that specific step as unverified — but the finding does
not depend on it, because even a per-page refresh leaves only a 30s margin against a 30s timeout.
With a second concurrent worker, the sweep would requeue a job that is mid-provider-call. Both
workers then process the same job, producing duplicate `ExecutionAttempt` rows for the same
logical work and racing terminal status writes. Append-only history would be *preserved* but no
longer *faithful*: the evidence would show attempts that do not correspond to distinct
application decisions.
This is worth flagging because `jobs.py:191-197` explicitly implements and documents
`SKIP LOCKED` row locking "so concurrent workers never contend for the same job." The claim path
is built for multi-worker operation; the reclaim path is not. A reader who trusts the claim
docstring would reasonably scale the worker.
- **Blast Radius:** `requeue_stale_processing_jobs` has one production caller (`worker.py:226`) and
tests in `tests/test_worker.py` and `tests/services/test_job_service.py`. Changing the *default*
affects `tests/test_config.py` declared-defaults assertions — check those before editing the default.
- **Recommendation:** **Fix before adding a second worker replica, not after.** Two parts:
(1) Make the threshold a function of the real bound rather than a coincidental peer of the
page timeout — at minimum default `worker_stale_job_seconds` to a multiple of
`worker_provider_timeout_seconds` with headroom, and add a model validator rejecting a stale
threshold at or below the provider timeout.
(2) Preferably make reclaim heartbeat-based: have `_persist_page_outcome` bump `Job.date_updated`
explicitly so liveness reflects progress rather than elapsed time since claim.
Add a guard asserting a multi-page job in flight is not reclaimed by a concurrently-invoked sweep.
- **Effort:** M
---
#### [MED-05] Provider evidence capture is per-instance mutable state, making the adapter non-reentrant by contract
- **Location:** `src/transcription/providers/openrouter.py:197-199, 264-267, 274-275, 297-298, 397-412`;
`_CapturingAsyncClient.last_response`/`last_body` at `openrouter.py:66-94`;
contract at `src/transcription/providers/base.py:110-118`
(`current_request_manifest`, `current_transport_evidence`)
- **Reachability:** **Latent.** Unblocked by any concurrent `transcribe()` on a single adapter
instance — most plausibly by processing a job's pages in parallel (`workflows.py:274` is
currently a sequential `for` loop) or by any second consumer sharing one
`SourceService.provider`. Verified safe today: `workflows.py:272` resolves one provider for the
loop and awaits each page; the worker's `ServiceBundle` (`worker.py:206`) is distinct from
`app.state.services` (`app.py:43`), so the UI cannot share the worker's adapter instance, and
the UI only enqueues jobs (`ui/pages/jobs_page.py:186-208`).
- **Problem & Consequence:** The `TranscriptionProvider` protocol defines evidence retrieval as
"the most recent call" state read *after* the fact. `workflows.py:369-370` relies on this on the
timeout path, reading `provider.current_request_manifest` / `current_transport_evidence` when no
result object exists. Under concurrency, page B's response overwrites
`_CapturingAsyncClient.last_response` before page A's timeout handler reads it, and page A's
`ExecutionAttempt` is written with page B's transport evidence.
The consequence is **evidence mis-attribution** — a provenance-integrity failure, which this
project's own rubric treats as its most serious class. It would also be near-undetectable after
the fact: the attempt row would be well-formed, internally consistent, and wrong. The
application-level design that makes this safe (sequential pages) is not expressed in the
provider contract, so the constraint lives only in `workflows.py`'s loop structure.
- **Blast Radius:** Changing the protocol touches `providers/base.py:102-136`,
`providers/openrouter.py:221-231`, the two read sites at `workflows.py:369-370`, and the fakes in
`tests/providers/test_openrouter.py`, `tests/services/test_workflows_reliability.py`, and
`tests/test_provider_boundaries.py`, all of which implement or assert the current property-based
contract.
- **Recommendation:** **Fix before introducing any intra-job page concurrency.** The durable fix is
to stop returning evidence through instance state: attach `request_manifest` and
`transport_evidence` to the raised exception on every failure path — which `ProviderError`
already supports (`providers/base.py:18-29`) and which the timeout path cannot currently use
because `asyncio.wait_for` raises `TimeoutError` from outside the adapter. A narrower option is
to have `transcribe()` accept a caller-owned capture sink so evidence is scoped to the call
rather than to the adapter. As an immediate, near-zero-cost step, document the non-reentrancy on
the protocol in `providers/base.py` so the constraint is visible where it is depended upon.
- **Effort:** M
---
#### [MED-06] Provider error bodies reach user-facing text while three provider failure paths persist no `detail`
- **Location:** `src/transcription/services/sources.py:923-947` (`handle_transcription_errors`);
message construction at `src/transcription/providers/openrouter.py:414-431`
(`_transport_error_message`)
- **Reachability:** **Live** for the message half (any provider failure during a UI-initiated
transcription surfaces through `show_error`).
- **Problem & Consequence:** Two mirrored halves of the same rule are broken in one function.
- `sources.py:943` builds `f"Provider transcription failed: {exc}"`, and `exc` is a
`ProviderError` whose message may embed up to 500 characters of the provider's error body
(`openrouter.py:430`). That is a provider payload in `message`, which
`error-handling.instructions.md:85` explicitly forbids.
- None of the three handlers (lines 929, 935, 942) passes `detail=`. Per
`error-handling.instructions.md:89-92`, omitting it degrades the provenance record.
I checked whether the provenance half is actually harmful before reporting it, and it is
**substantially mitigated**: `workflows.py:391` calls `_find_provider_error`, which walks
`__cause__`/`__context__` (`workflows.py:806-813`) to recover the original `ProviderError` and
persists its `transport_evidence` — status code, safe headers, and the exact response body — onto
the attempt. So the root cause is preserved in transport evidence even though `error_detail` is
thin. This is why the finding is Medium rather than High. The residual cost is that the
human-readable failure summary is uninformative for the two paths (`ProviderAuthError`,
`ProviderResponseError`) whose messages are entirely generic.
- **Blast Radius:** `handle_transcription_errors` is used on the transcription path in
`sources.py`; `TranscriptionError.message` is consumed by `error_presenter.show_error`,
`build_error_envelope`, and `format_error_detail`. Assertions on these messages live in
`tests/services/test_transcription_service.py` and `tests/providers/test_openrouter.py`.
- **Recommendation:** Move the interpolated provider text from `message` to `detail` on all three
handlers, keeping the generic message the other two already use:
```python
except ProviderError as exc:
raise TranscriptionError(
"Provider transcription failed",
category=ErrorCategory.EXTERNAL_PROVIDER,
suggestion="Retry the transcription from jobs. If repeated, check provider availability.",
retriable=True,
detail=f"{type(exc).__name__}: {exc}",
) from exc
```
Apply the same `detail=` addition to the `ProviderAuthError` and `ProviderResponseError`
handlers. Note the interaction with HIGH-01: until the render boundary is sanitized, moving text
into `detail` still reaches users through the `error_detail` display. Sequence accordingly.
- **Effort:** S
---
#### [MED-07] No deterministic guard covers the `message`/`detail` split at explicit raise sites
- **Location:** `tests/test_errors.py:56-78`; rule at
`.github/instructions/error-handling.instructions.md:78-98`; canonical statement at
`docs/error_handling.md:102-115`
- **Reachability:** **Live** — this coverage gap is what allowed HIGH-02 and MED-06 to exist in a
fully green suite.
- **Problem & Consequence:** `docs/error_handling.md:115` names
`tests/test_errors.py::test_unexpected_error_does_not_leak_filesystem_paths` as the enforcement
for the message/detail split. That test exercises exactly one function,
`classify_unexpected_error`. Every direct `raise SomeAppError(...)` in `src/` — roughly 50 sites
by grep — is unenforced. The documentation therefore overstates the enforcement, which is worse
than having no guard: a contributor reading `error_handling.md:115` reasonably concludes the rule
is mechanically protected.
Per the reviewer skill, where a check is unenforced, recommending the deterministic test is
itself a finding.
- **Blast Radius:** Tests only.
- **Recommendation:** Add an AST guard, `tests/test_error_message_safety.py`, that scans `src/`
for `raise <AppError subclass>(...)` and fails when the first positional argument is an f-string
containing a formatted value whose name matches a path-like or exception-like identifier
(`path`, `_path`, `root`, `dir`, `exc`, `err`, `e`). Model it on the existing AST guards, which
are the established pattern here (`test_ui_boundaries.py`, `test_service_boundaries.py`,
`test_orphan_sweep.py`). Pair it with a second guard asserting that no UI module reads
`error_detail` without routing through the sanitizing projection from HIGH-01 — that one closes
the render side, which is where the real leak is.
- **Effort:** M
---
### Low Severity
#### [LOW-01] `ui/pages/tags_page.py` is dead code; the V6.1 roadmap is correct
- **Location:** `src/transcription/ui/pages/tags_page.py` (93 lines);
registration list at `src/transcription/ui/__init__.py:37-43`
- **Reachability:** **Not reachable.** This resolves the flagged open item: the route is genuinely
**not** registered. `register_pages` calls seven page registrars and `tags_page` is not among
them; nothing anywhere imports the module. `docs/roadmap_plan.md:47` ("Retire the Tags page") is
accurate, and `tests/ui/test_tags_page.py` correctly asserts `/ui/tags` returns 404 — though it
passes trivially, since an unimported module cannot register anything.
- **Problem & Consequence:** No runtime risk; purely stranded code. It is worth noting that if it
*were* ever re-registered, `tags_page.py:71-74` traverses `document.document_tags` and
`link.tag_ref` inside a page render, and those relationships are configured `lazy="raise"`
(`docs/architecture.md:200-203`) — so re-enabling this module without adding eager loads to
`list_documents` would raise on first render.
- **Recommendation:** Delete `src/transcription/ui/pages/tags_page.py`. Retain
`tests/ui/test_tags_page.py` as the retirement guard. Fixing MED-01 first would make this
finding reproducible by the suite rather than by manual inspection.
- **Effort:** S
#### [LOW-02] `benchmarking.py` ships in the runtime package but is referenced only by tests
- **Location:** `src/transcription/benchmarking.py` (69 lines); sole consumers
`tests/test_v42_evidence.py:15-16` (`EditorialAssessment`, `score_transcription`)
- **Reachability:** Live as importable API; never invoked by application code.
- **Problem & Consequence:** No defect. It supports the model-evaluation policy in
`docs/invariant/ai_evidence_and_provenance.md:113-126`, which is legitimate, but it currently has
no production caller and no tooling entrypoint, so it is indistinguishable from drift.
- **Recommendation:** Either move it under `tools/` alongside the other operator utilities, or add
a `KNOWN_ORPHANS`-style rationale recording that it is retained as the evaluation-policy
implementation. Do not silently keep it unlabeled.
- **Effort:** S
#### [LOW-03] Two overlapping prompt error types split across modules
- **Location:** `src/transcription/services/errors.py:15-16` (`PromptLoadError`) and
`src/transcription/services/prompts.py:19` (`PromptStoreError`)
- **Reachability:** Live; no misbehavior observed.
- **Problem & Consequence:** `services/errors.py:1-8` documents itself as the neutral home for
exceptions raised by more than one service, precisely so a caller's `except` clause does not
change when an operation moves. `PromptStoreError` is defined outside that module and covers an
overlapping domain (prompt file access), so a caller wanting to handle "any prompt failure" must
import from two modules and know which is which. `sources.py:855` raises `PromptLoadError` for a
missing prompt file while `prompts.py:132` raises `PromptStoreError` for the same condition
reached through the Settings page.
- **Recommendation:** Move `PromptStoreError` into `services/errors.py` next to `PromptLoadError`,
or make one a subclass of the other so a single `except` covers prompt failures. Low urgency; do
it opportunistically when HIGH-02 touches both files anyway.
- **Effort:** S
---
## 4. Architectural Drift & Gap Analysis
| Area / Component | Direction | Documented / Intended Rule | Actual Implementation State | Severity | Recommended Resolution |
| :--- | :--- | :--- | :--- | :--- | :--- |
| Error presentation | `doc->code` | `docs/error_handling.md:110``detail` is internal only, surfaced by `format_error_detail` and logs | `sources_page.py:484`, `table/sources.py:90`, `settings_page.py:562` render `error_detail` verbatim to users | High | Sanitizing render projection (HIGH-01); do **not** strip `detail` |
| User-facing messages | `doc->code` | `invariant/error_handling.md:59` — no local filesystem detail in user-facing messages | 5 live sites interpolate absolute paths into `AppError.message` | High | Generic `message`, path on `detail` (HIGH-02) |
| Evidence invariant ownership | `doc->doc` | `docs/index.md:1` — baseline is V6.1 | `invariant/ai_evidence_and_provenance.md:130` names "Canonical V4"; the currency guard scans the file but its regexes match neither the phrasing nor a minor-less `V4` | Medium | Update text; make the guard token-based, or drop version labels entirely (MED-02, §8.1) |
| Enforcement claim | `doc->code` | `docs/error_handling.md:115` — split "Enforced by `tests/test_errors.py::…`" | That test covers only `classify_unexpected_error`; explicit raises unguarded | Medium | Add AST guard (MED-07) |
| Orphan sweep | `doc->code` | `test_orphan_sweep.py:1-13` — sweep is "deterministic" and "conservative" | Bare-name matching; cannot see a dead module (`tags_page.py`) | Medium | Qualify by module + module-reachability pass (MED-01) |
| Worker scaling | `code->doc` | `jobs.py:191-197``SKIP LOCKED` so "concurrent workers never contend" | Claim path is multi-worker-safe; stale-reclaim path is not | Medium | Derive stale threshold from job duration; document single-worker constraint until fixed (MED-04) |
| Provider adapter contract | `code->doc` | `providers/base.py:110-118` — evidence read as "most recent call" state | Contract is silently non-reentrant; safety lives in `workflows.py`'s sequential loop | Medium | Scope evidence to the call; document non-reentrancy (MED-05) |
| Settings env file | `code->doc` | `config.py:66-75``env_file=".env.production"` | Correctness depends on an undocumented cwd contract with `Dockerfile` `WORKDIR /app` | Medium | Explicit resolver seam, or document + guard the contract (MED-03) |
| Tags page | *(no drift)* | `roadmap_plan.md:47` — Tags page retired | Route genuinely unregistered; module is stranded code | Low | Delete the module (LOW-01) |
---
## 5. Invariant Inventory & Routing Recommendations
| Invariant / Constraint | Current Location | Recommended Target Layer | Rationale |
| :--- | :--- | :--- | :--- |
| `detail`/`error_detail` never rendered to users | docs + instructions | **Deterministic test** + sanitizing projection | Stated in three documents and violated in three files; prose has demonstrably failed to hold it |
| `message` carries no paths or exception text | instructions; partial test | **Deterministic test** (AST, all raise sites) | Existing guard covers one function; the gap produced HIGH-02 |
| `ExecutionAttempt.error_detail` retains root cause | docs + `test_v42_evidence.py` | **Keep in tests** — already correct | Counterweight to the above; must be named in any fix so it is not over-corrected |
| Intermediate pages commit individually | `workflows.py` docstring + `test_workflows_reliability.py:318` | **Keep in tests** — verified genuine | Cross-session read makes it a real durability assertion |
| Final page atomic with terminal status | `services.instructions.md` + `test_pipeline_atomicity.py` | **Keep in tests** — verified genuine | Fault injection makes a split commit fail |
| Canonical baseline version consistency | `docs/index.md` + `test_meta_contract_guards.py:89` | **Repair existing test, or remove the labels** | Guard exists but matches by approved phrase and requires a minor version, so it fails open on new phrasings (MED-02) |
| Module-level reachability / dead modules | `test_orphan_sweep.py` (ineffective) | **Deterministic test** (repair existing) | Guard exists but cannot detect the case (MED-01) |
| Stale threshold > max job duration | *(unenforced)* | **Config validator + test** | Currently a coincidence of two equal defaults (MED-04) |
| Provider adapter non-reentrancy | *(unenforced, implicit)* | **Instructions** + protocol docstring | A design constraint callers must know before adding concurrency (MED-05) |
| Env-file resolution independent of cwd | `tests/conftest.py` monkeypatch | **Code seam** + `docs/production-runbook.md` | A test-only fix for a production coupling is misrouted enforcement (MED-03) |
---
## 6. Stack-Specific Analysis
**Python 3.12+.** Modern and consistent. PEP 695 generics are used correctly and non-trivially
(`RegistryService[ModelT: RegistryEntry]` in `services/registry.py:58`, `UiActionOutcome[T]`,
`_get_or_raise[ModelT]`), `type` statements appear in `db/session.py:15,50`, and `X | None` is
used throughout. `structural Protocol` bounds (`RegistryEntry`, `WorkerNotifier`,
`TranscriptionProvider`) are used to avoid type suppressions rather than to decorate. `ty` passes
clean with no suppressions found. The two `# noqa` uses (`workflows.py:228` `PLR0915`,
`workflows.py:383` `BLE001`) are both justified in context — the broad catch is a deliberate
per-page containment boundary that immediately classifies and re-records.
**FastAPI.** Lifespan is handled via `@asynccontextmanager` (`app.py:36`), not the deprecated
`@app.on_event`. Session factories are injected through `Depends` (`SessionFactoryDep`,
`db/session.py:50`) rather than reached as globals from routes. `api/errors.py` centralizes
envelope translation. One residual: `get_settings` is `@cache`d and read as a module-level
fallback in ~10 modules; this is acceptable given the documented restart-to-apply contract
(`docs/ui/pages/settings.md:28`) but means the cache is process-lifetime and unclearable.
**NiceGUI (pinned `3.13.0`).** The pin is a recorded release-stability decision and is not
reported as a defect. Boundaries are enforced structurally: `test_ui_boundaries.py` uses an
import **allowlist**, which is the right polarity. Blocking work is dispatched off the event loop
via `run_blocking` (`settings_page.py:818,822`). The one boundary that is *not* enforced is
presentation of internal fields (HIGH-01) — pages are prevented from touching persistence but not
from rendering internal-only text.
**SQLModel / SQLAlchemy.** Strong. `lazy="raise"` on relationships forces explicit eager loading;
read paths declare `selectinload` chains with comments explaining *why* each is needed
(`sources.py:309-316` is a good example). `expire_on_commit=False` (`db/session.py:28`) is set
deliberately, which is what makes post-commit attribute access in `evidence.py:159-202` safe.
`claim_next_queued_job` (`jobs.py:186-241`) branches correctly on dialect — `SKIP LOCKED` on
PostgreSQL, conditional `UPDATE ... RETURNING` on SQLite — rather than assuming one engine.
Attempt-number allocation uses `begin_nested()` with bounded retry (`sources.py:598-616`), the
right pattern for a monotonic per-parent sequence. No N+1 patterns were found in the read paths
sampled.
**Pydantic V2 & Settings.** Fully V2; no `@validator`, `class Config`, `.dict()`, or `parse_obj`
anywhere. Evidence contracts use `ConfigDict(extra="forbid", frozen=True)` (`providers/evidence.py:47`),
which is exactly right for persisted provenance — an unexpected field fails loudly rather than
being silently dropped. `SecretStr` guards the API key. The discriminated
`SqliteSettings | PostgresSettings` union is clean. `normalize_provider_models` correctly runs
`mode="before"` so the derived tuple is produced by construction rather than by mutating a frozen
model — a subtlety that is easy to get wrong. Sole issue: the cwd-coupled `env_file` (MED-03).
**Asyncio Workers.** Notably careful. `asyncio.shield` wraps both the per-page commit and the
terminal commit (`workflows.py:601-614`, `650-663`), with the `except CancelledError: await task;
raise` pattern that actually completes the shielded work rather than merely deferring cancellation —
a detail most implementations get wrong. `handle_worker_exceptions` (`worker.py:157-182`)
distinguishes retriable from non-retriable faults and stops the loop rather than spinning.
`_advance_job_with_containment` (`worker.py`/`workflows.py:507-540`) guarantees a claimed job
cannot strand in `PROCESSING`. `worker_consumer_lifespan` has a bounded shutdown with escalation to
`cancel()`. Gaps are MED-04 and MED-05, both latent and both with stated unblocking conditions.
**OpenRouter / Adapter Boundary.** Encapsulation holds: `test_provider_boundaries.py` enforces it,
and `workflows.py` imports only `providers` abstractions. `_CapturingAsyncClient` is a
well-judged design — it captures the exact transport body before SDK parsing without altering what
the SDK consumes, including the streamed case. Timeout construction (`openrouter.py:200-206`)
correctly overrides httpx's 5s per-phase default that would otherwise silently cap the configured
budget. `SAFE_RESPONSE_HEADERS` (`providers/evidence.py:29-41`) was reviewed field-by-field:
all nine entries are non-secret correlation, content, or rate-limit headers, and
`filter_safe_response_headers` is a true allowlist filter with no redaction-after-capture — this
satisfies invariant 3.8.2 exactly. `_replace_embedded_media` correctly substitutes a source
reference for base64 payloads, satisfying 3.8.3. The one structural weakness is MED-05.
**Testing & Quality Tooling.** 410 tests, all green, with genuinely strong contract guards
(boundaries, model contract, media path safety, evidence append-only, atomicity). Marker strictness
and `asyncio_mode = "strict"` are configured, and no unawaited-coroutine warnings appeared. Two
guards, however, assert meaningfully less than their names and docstrings claim
(`test_orphan_sweep.py` — MED-01; `test_errors.py` path-leak coverage — MED-07), and the
`>= 420` snapshot threshold at `test_orphan_sweep.py:169` repeats a weak-assertion pattern the
2026-08-23 review already flagged at `>= 200`; it was raised rather than replaced with
set-membership.
---
## 7. Duplication & Consolidation Report
| Pattern / Duplication | Locations | Proposed Canonical Home | Estimated Lines Removed |
| :--- | :--- | :--- | :--- |
| `f"{type(exc).__name__}: {exc}"` detail construction | `errors.py:118`, `maintenance.py:71,95,135,206`, `runtime_settings_store.py:388,476,554` | `errors.py::exception_detail(exc)` | ~8 (consistency > line count) |
| Filesystem `AppError` construction from `OSError` | `prompts.py:186-191`, `runtime_settings_store.py:384-389,472-477,550-555` | `errors.py::filesystem_error(message, exc, *, suggestion)` | ~20 |
| Overlapping prompt error types | `services/errors.py:15`, `services/prompts.py:19` | `services/errors.py` (LOW-03) | ~5 |
| Duplicated `provider_duration_ms` / `processing_duration_ms` max-clamp arithmetic | `workflows.py:341-345, 361-368, 397-404` | `workflows.py::_page_durations(started_at, finished_at, monotonic_started_at)` | ~20 |
| `_utc_now_naive` defined per module | `workflows.py:51`, `jobs.py:29`, `db/models.py`, `sources.py` | Single helper in `db/models.py`, imported | ~12 |
### Proposed Canonical Abstractions
```python
# src/transcription/errors.py
def exception_detail(exc: BaseException) -> str:
"""Internal-only root-cause text for AppError.detail. Never user-facing."""
def filesystem_error[E: AppError](error_type: type[E], message: str, exc: OSError, *, suggestion: str) -> E:
"""Build a filesystem AppError with a generic message and the path on detail."""
# src/transcription/ui/components/error_presenter.py
def display_failure_detail(error_detail: str | None) -> str | None:
"""Sanitize persisted failure detail for UI rendering (HIGH-01)."""
```
---
## 8. Meta-Tooling & Instruction Update Recommendations
1. **`docs/invariant/ai_evidence_and_provenance.md:130`** — resolve the V4 label. Two viable
routes, and the maintainer has proposed the second:
- **(a) Repair the guard.** Fix the text to V6.1 and make
`test_canonical_docs_declare_one_consistent_baseline` token-based rather than phrase-based
(MED-02). Keeps version labels as navigational anchors.
- **(b) Remove version labels from canonical docs.** While the project has a single principal
user and no released versions to support, "canonical" and "current" are the same thing, so the
label carries no information a reader can act on — it only creates a second thing to keep in
sync. Retain the baseline declaration in `docs/index.md` alone as the release marker, keep
version language in `docs/roadmap_plan.md` and the migration/deployment docs (already
excluded from the scan for exactly this reason), and replace in-body references with
unversioned phrasing ("the canonical architecture, schema, requirements, and error-policy
documents"). The guard then inverts: assert that no canonical doc outside the exclusion set
contains a version token at all, which is a stricter and much cheaper property to hold than
agreement between many labels. Requirement IDs (`REQ-4-021`, `REQ-6-014`) are stable
identifiers, not currency claims, and should be left alone.
2. **`docs/error_handling.md:107-115`** — either add the sanitizing-projection rule for UI display
of `error_detail`, or revise the `detail` "Surfaces" row to admit operator-facing evidence
displays. Update the "Enforced by" line once MED-07's guard lands, since it currently overstates
coverage.
3. **`.github/instructions/error-handling.instructions.md`** — add an explicit clause under
"User-Safe Messaging" stating that *persisted* `error_detail` is subject to the same no-paths
rule at any render boundary. The current table (line 86) states the rule for `AppError.detail`
and stops there, so the persisted-then-rendered path falls between the lines.
4. **`.github/instructions/providers.instructions.md`** — record the adapter non-reentrancy
constraint (MED-05); it is currently an undocumented precondition of `workflows.py`.
5. **`.github/instructions/services.instructions.md`** — the two competing atomicity invariants are
well described and both guards verified; no change needed. Worth adding the stale-reclaim
threshold constraint (MED-04) alongside them, since it is a third worker-lifecycle rule with no
documented home.
6. **`tests/test_orphan_sweep.py`** — repair per MED-01 and replace the `>= 420` threshold with
set-membership assertions.
7. **New `tests/test_error_message_safety.py`** — AST guard per MED-07, covering both the raise
sites and the UI render sites.
8. **`docs/production-runbook.md`** — document the cwd/`WORKDIR` contract for `.env.production`
resolution if MED-03 is resolved by documentation rather than by a code seam.
---
## 9. Prioritized Dependency-Ordered Action Plan
**Phase 1 — Blocking fixes (privacy; ordered, HIGH-01 first)**
1. **HIGH-01** — add `display_failure_detail` and route `sources_page.py:484`,
`table/sources.py:90-95`, and `settings_page.py:562` through it. Do this **first**: it closes
the render boundary, so the Phase-1.2 fix cannot relocate a leak again.
2. **HIGH-02** — move paths from `message` to `detail` at the five sites, including the
`prompts.py:186` double violation.
3. **MED-06** — move provider payload text to `detail`; add `detail=` to all three
`handle_transcription_errors` handlers.
**Phase 2 — Enforcement hardening (make Phase 1 permanent)**
4. **MED-07** — AST guard for raise-site `message` safety **and** for UI reads of `error_detail`.
5. **MED-01** — qualify orphan definitions by module; add module-reachability; replace the
snapshot threshold.
6. **MED-02** — fix the V4/V6.1 text and extend the meta-contract guard to baseline-version currency.
**Phase 3 — Reliability & concurrency (latent; each must precede its unblocking change)**
7. **MED-04** — derive `worker_stale_job_seconds` from `worker_provider_timeout_seconds` with a
rejecting validator, ideally plus a progress heartbeat. **Must land before any second worker
replica.**
8. **MED-05** — scope provider evidence to the call rather than the instance. **Must land before
any intra-job page concurrency.** Document non-reentrancy immediately as an interim step.
**Phase 4 — Consolidation & refactoring**
9. **MED-03** — explicit env-file resolution seam shared by `Settings` and `runtime_settings_store`;
remove the `model_config` monkeypatch from `conftest.py`.
10. **LOW-01** — delete `tags_page.py` (after MED-01, so the suite reproduces the finding).
11. **LOW-03** and the §7 consolidations — fold in opportunistically while Phase 1 touches these files.
**Phase 5 — Non-blocking governance/documentation depth**
12. **LOW-02** — relocate or annotate `benchmarking.py`.
13. Instruction/doc updates §8.3–§8.5, §8.8.
---
## 10. Preserved Strengths
- **Append-only evidence is real, not aspirational.** Every provider call produces a distinct
`ExecutionAttempt`; no runtime path mutates a historical row. Projection writes onto
`Source.raw_transcription` are clearly separated from history, and `promote_machine_attempt`
(`evidence.py:117-146`) repoints the projection without rewriting evidence — with a docstring
that explains exactly why that one write lives in a read-oriented service.
- **Transport-layer terminology is honored in code.** `_CapturingAsyncClient` exists specifically so
the stored body is the application-boundary capture rather than an SDK-parsed object, and
`TransportEvidence(response_received=False)` explicitly represents "no response" instead of
conflating it with an empty one. This is invariant 3.4/3.5 implemented rather than asserted.
- **Header allowlisting is done the hard, correct way** — filter-before-store with an explicit
frozenset, never capture-then-redact (`providers/evidence.py:29-41,130-134`).
- **Boundaries are enforced by allowlist, not blocklist.** `test_ui_boundaries.py:20-25` states the
reasoning explicitly; it means a newly added persistence helper cannot slip through under an
unlisted name.
- **The two competing atomicity invariants are both correctly implemented and both genuinely
guarded**, with the tests structured so that the naive over-correction fails.
- **Cancellation safety in the worker is unusually well handled**`asyncio.shield` plus
`await task` on `CancelledError` actually completes the commit rather than merely deferring
cancellation.
- **Comments explain rationale, not mechanics.** `workflows.py:269-271`, `openrouter.py:200-202`,
`config.py:114-115`, and `jobs.py:191-197` each record *why* a non-obvious choice was made,
several citing the review log entry that motivated it. This is what made verifying the atomicity
and timeout invariants tractable in this review.
- **Documentation-to-code traceability is strong overall.** Page contracts, schema field tables,
and requirement IDs are maintained and guarded; the drift found in this review is narrow and
specific rather than systemic.
---
## Appendix A — Repo-Specific Deterministic Checks
| # | Check | Result | Evidence |
| :-- | :--- | :--- | :--- |
| 1 | Service boundary rule: no service-to-service imports | **Pass** | `tests/test_service_boundaries.py` green; AST scan, allowlist-based; `workflows.py` composes via `ServiceBundle` |
| 2 | UI boundary rule: no persistence access from pages/components | **Pass (structurally)** | `tests/test_ui_boundaries.py` green. Caveat: it guards *data access*, not presentation of internal-only fields — see HIGH-01 |
| 3 | Status vocabulary conformance; no stringly-typed literals | **Pass** | `tests/test_model_contract_guards.py` green; enum members verified against `db/models.py` |
| 4 | Evidence ownership: append-only history, projections not history mutation | **Pass** | `test_v42_evidence.py::test_attempts_are_append_only_and_exported_with_integrity` verified non-vacuous (asserts both retained attempts and export integrity at lines 281-290) |
| 5 | Canonical authority: findings resolve against `docs/*` first | **Pass with defect** | `test_canonical_authority_references_are_present` green. The companion baseline-currency guard (`test_canonical_docs_declare_one_consistent_baseline`) scans the offending file but fails open on its phrasing and on minor-less version tokens — MED-02 |
| 6 | Schema contract fidelity: `docs/schema.md` field-accurate | **Pass** | `test_model_contract_guards.py` + `test_meta_contract_guards.py` green |
| 7 | Media boundary: record-validated media, controlled URL resolver | **Pass** | `test_media_path_safety.py`, `tests/ui/test_media_urls.py` green; `public_media_path_label` verified path-safe |
| 8 | Eager-loading conformance vs `lazy="raise"` | **Pass** | Declaration-side guard green; sampled read paths declare explicit `selectinload` chains. Note: dead `tags_page.py:71-74` would violate it if re-registered (LOW-01) |
| 9 | Cross-cutting error conformance | **FAIL** | Guards green but coverage is narrower than documented: HIGH-01, HIGH-02, MED-06, MED-07 |
| 10 | Orphan/dead-code conformance | **FAIL** | Guard green but structurally unable to detect a dead module: MED-01, proven by LOW-01 |
## Appendix B — Evidence & Provenance Auditor Families
| Family | Subject | Result | Evidence |
| :--- | :--- | :--- | :--- |
| A | Attempt history append-only | **Pass** | No update/delete path to `ExecutionAttempt`; insert-only with `begin_nested` + bounded sequence retry (`sources.py:556-616`) |
| B | Attempt numbering monotonic per source | **Pass** | `insert_with_sequence_retry`; uniqueness constraint plus retry on conflict |
| C | Transport evidence captured at the transport boundary | **Pass** | `_CapturingAsyncClient` retains the exact wire body pre-SDK-parse (`openrouter.py:66-94`) |
| D | Absent response distinguished from empty response | **Pass** | `TransportEvidence.response_received` is explicit, not inferred |
| E | Response header persistence is allowlist-based | **Pass** | `SAFE_RESPONSE_HEADERS` (`providers/evidence.py:29-41`) — all nine entries verified non-secret; filter-before-store |
| F | No machine-local detail on user-facing surfaces | **FAIL** | `error_detail` rendered verbatim at three UI sites (HIGH-01); paths in `message` at five sites (HIGH-02) |
| G | Request manifest excludes embedded media payloads | **Pass** | `_replace_embedded_media` (`openrouter.py:377-395`) substitutes a source reference for base64 data |
| H | Evidence attribution is correct under concurrency | **Pass today / at risk** | Correct in the current sequential single-worker deployment; the contract itself is non-reentrant (MED-05) and reclaim has no margin (MED-04) |
+4 -5
View File
@@ -15,8 +15,7 @@ Naming: `<YYYY-MM-DD>-code-review.md` for review reports, and
## Current
- [`2026-08-23-code-review.md`](./2026-08-23-code-review.md) — full review. 0 critical,
4 high, 5 medium, 11 low.
- [`2026-08-23-remediation-handoff.md`](./2026-08-23-remediation-handoff.md) — **start here
to continue the remediation work.** Phase 1 (all 4 high findings) is complete as of commit
`de18c2e`; the handoff covers Phases 2-5 with per-task acceptance criteria, the verification
baseline, and the environment gotchas needed to avoid re-deriving them.
4 high, 5 medium, 11 low. **All findings remediated.** Retained as a record of the
reasoning, not as a list of open work. Note that a few of its recommendations were
wrong on contact and were corrected during implementation; the code and the guard
tests are authoritative over the report text.
+149 -25
View File
@@ -5,6 +5,9 @@ This roadmap starts at **V6.0** and tracks forward-looking work only.
## V6.0 - Hosting Migration
Objective: move from local-only operation to secure, stable remote hosting.
Status: **Completed**
Detailed plan: [`v6_0_hosting_migration_plan.md`](v6_0_hosting_migration_plan.md)
### Scope
1. Containerize app runtime for production deployment.
@@ -23,26 +26,151 @@ Objective: move from local-only operation to secure, stable remote hosting.
- One end-to-end document -> source -> job workflow succeeds remotely.
- Backup and restore procedure is tested.
## V6.1 - Reporting Features
### Accomplished
1. Delivered production Docker deployment with split `app`/`worker`, `postgres`, and `cloudflared`.
2. Landed SQLite -> PostgreSQL migration tooling and runbook coverage.
3. Added production health/reliability wiring and operational runbooks for deploy/rollback/recovery.
4. Established host-visible backup workflow and restore path for PostgreSQL plus media/config assets.
Objective: improve research value with person-centric outputs.
## V6.1 - Testing and Refinement
Objective: improve navigation and operational workflows after user feedback.
Status: **Completed**
### Scope
1. Person timeline views using document dates and linked records.
2. AI-assisted biography/family-history generation from curated sources.
3. Exportable report views (human-readable, print-oriented).
1. Make Document Detail the primary source-page workspace:
- Use Source-style pan/zoom + previous/next page controls.
- Move editable revision controls into Document Detail.
- Move archival/system metadata to dedicated Document Info route.
2. Simplify top navigation:
- Remove top-level Tags and Sources entries.
- Retire the Tags page and the global Source Asset Records entry flow.
3. Improve list/detail clarity:
- Add Document transcription status to Archival Documents list.
- Add Document Date in People Detail -> Linked Documents table.
4. Add worker-backed Settings maintenance runs:
- Add `maintenance_run` persistence (`id`, `job_type`, `status`, `started_at`, `finished_at`, `triggered_by`, `summary`, `log_path`, `error_detail`).
- Add Run Backup and Run Storage Reconciliation actions that enqueue runs and execute in the worker.
- Add run history with status, duration, summary, and log view/download.
- Defer daily/weekly scheduling controls to V6.2.
### Accomplished
1. Refactored Document Detail into the primary source-page workspace (pan/zoom viewer, previous/next page navigation, editable revision flow) and moved archival/system metadata to Document Info.
2. Simplified top navigation by removing Tags/Sources entries and retiring the Tags page/global Source Asset Records flow.
3. Improved data clarity with document transcription status in Archival Documents and Document Date in People Detail linked documents.
4. Implemented queue-backed maintenance operations (`maintenance_run` model/service/worker/UI) with run history and log view/download.
5. Hardened runtime settings operations in production:
- runtime settings writes target mounted `.env.production`,
- fallback write path for single-file bind mounts,
- explicit hidden/deployment-key disclosure in Settings UI.
6. Simplified backup configuration and behavior:
- standardized on `BACKUP_DIR` + `BACKUP_RETENTION_DAYS`,
- backup script uses `DATABASE__*` persistence keys,
- compose maps Postgres container init values from `DATABASE__*`,
- env contract drift tests now guard `.env.production.example`.
## V6.2 - GEDCOM Data Layer
Objective: introduce a genealogical data layer sourced from GEDCOM exports, bridged to
existing `Person` records via FamilySearch ID, without disrupting document-focused Person
workflows.
### Scope
1. Manual `.ged` file upload only. No FamilySearch credentials are stored or used by the
app; the user runs the third-party `getmyancestors` tool themselves and uploads the
resulting export.
2. Four new tables: `genealogy_person`, `genealogy_family`, `genealogy_family_child`, and
`genealogy_citation` (raw GEDCOM `SOUR` citations, reusable in a later version to record
when a transcribed document itself becomes citation evidence for FamilySearch).
3. Upsert-based import keyed on FamilySearch ID (`fs_id`) so repeat imports update existing
records in place without breaking existing `Person.family_search_id` links or duplicating
surrogate keys.
4. Reuse the existing V6.1 worker-backed `maintenance_run` pattern for import runs (run
history, status, summary, log view/download) rather than new infrastructure.
### Deliverables
- Timeline UI and service queries with clear ordering/filters.
- Prompted narrative generation workflow using existing evidence-safe patterns.
- Saved/printable report presentation for review and sharing.
- GEDCOM parser/importer producing the four genealogy tables.
- `MaintenanceJobType` entry for GEDCOM import with upsert semantics and a run summary
(records added/updated).
- Settings UI entry to upload a `.ged` file, trigger an import run, and view history.
### Exit Criteria
- Importing the same `.ged` file twice does not duplicate or orphan data.
- Existing `Person.family_search_id` values continue to resolve to the correct
`genealogy_person` row after import.
- Import run history is visible with status, duration, and summary, consistent with other
maintenance runs.
## V6.3 - Reporting and Genealogy-Enriched Features
Objective: improve research value with person-centric outputs, grounded in both archival
documents and the V6.2 genealogical data layer.
This version is broken into five sequential sub-versions because of real dependency
ordering: entity linking must exist before GEDCOM data can be targeted per-person; the
Facts/Events mechanism must exist before timelines or reconciliation have anything
meaningful to consume.
### V6.3.1 - Manual Entity Linking
- Search/browse UI over `genealogy_person` to find and link a candidate match to an
application `Person`, setting `family_search_id`. Linking is reversible (unlink).
- Once linked, GEDCOM vitals display alongside the `Person` record without requiring any
schema change to `Person`.
### V6.3.2 - Person Facts and Events
- New fact/event table capturing: person, fact type (birth/death/event/free-form), date
(+raw), place, free-text description, and a link to the source document as evidence.
- Manual tagging UI while reviewing a transcribed document: select a passage, choose the
person and fact type, record the date/description.
- One-time migration of existing `Person.birth_date`/`birth_date_raw`/`birth_place`/
`death_date`/`death_date_raw`/`death_place` values into fact/event rows (tagged as
legacy/no-document-evidence where no source document is known), followed by retiring those
six columns from `Person`. Birth/death become Facts/Events like any other locally-known
fact, for both linked and unlinked people. `Person` permanently keeps `last_name`,
`given_names`, `biography`, `family_search_id`, `metadata_`, tags, photos, and document
associations.
### V6.3.3 - Person Timelines
- Timeline query merging GEDCOM milestones (birth, marriage, children's births, death) for
linked persons with locally recorded Facts/Events.
- Timeline UI on Person Detail with clear ordering/filters; entries link back to their
originating document or GEDCOM record.
### V6.3.4 - Reconciliation
- Compares Facts/Events (the real, document-evidenced local signal) against corresponding
`genealogy_person` fields for linked persons.
- Persisted reconciliation record: person, field, local value with evidence-document link,
GEDCOM value, and status (open / submitted / dismissed).
- Re-evaluated automatically as part of each GEDCOM import maintenance run: opens new
discrepancies, auto-resolves ones where GEDCOM now matches, leaves others unchanged.
- Reconciliation review UI functions as a manual to-do list for updating FamilySearch; the
app does not write back to FamilySearch itself.
### V6.3.5 - AI-Assisted Biography Generation
- Prompted narrative generation grounded in GEDCOM facts, Facts/Events, and relevant
document snippets as structured input, using existing evidence-safe prompting patterns.
- Output cites back to source documents and FamilySearch records.
- Saved/printable report presentation for review; reports do not modify archival source
data.
### Exit Criteria (applies across V6.3.1-V6.3.5)
- Entity links are reversible and do not alter document associations.
- Timelines are reproducible from persisted records.
- Reconciliation items always carry a link to the document evidence justifying the local
value, and re-running GEDCOM import correctly opens, resolves, or leaves items unchanged.
- Narrative generation is traceable to source records and prompts.
- Reports can be reviewed without modifying archival source data.
## V6.2 - Access Control and Multi-User Readiness
## V6.4 - Access Control and Multi-User Readiness
[ *More thoughts on user accounts:*
* *Create a generic "view only" user that does not have the rights to alter any of the data*
* *Limit user accounts access to data by Tag. I have distant family members that I would want to share the transcribed data with, but they would only be interested in a subset of it. For example my Cochran cousins would have no interest in Lancaster documents, so limit the Cochra Clan cousins to view-only access to documents tagged "cochran clan"* ]
Objective: prepare for managed collaboration beyond single-user operation.
@@ -61,26 +189,22 @@ Objective: prepare for managed collaboration beyond single-user operation.
- Role policies are enforced by deterministic tests.
- User-attributed changes are visible for audit/review.
## V6.3 - Scalability and Multi-Tenant Direction (Optional)
## Deferred / Future Ideas (not committed scope)
Objective: keep architecture ready for broader deployment footprints.
### Scope
1. Evaluate per-tenant or per-user data partitioning strategy.
2. Formalize connection/runtime strategy for tenant-aware DB selection.
3. Expand operational telemetry for throughput and cost monitoring.
### Deliverables
- Decision document for tenancy model and migration strategy.
- Prototype-safe runtime boundary for selecting data targets.
- Monitoring baseline for queue depth, job latency, and provider cost trends.
### Exit Criteria
- Selected tenancy strategy is documented and testable.
- Operational metrics support capacity planning.
Captured for later consideration, not yet scheduled to a version:
* AI-assisted entity disambiguation (kinship co-occurrence, chronological plausibility
filtering) when linking document mentions to people.
* Kinship-aware `@mention` tagging while transcribing.
* Relationship-calculator badges (e.g., "3rd Great-Grandmother") in the document viewer.
* Interactive migration/geography mapping from GEDCOM and document place mentions.
* AI-suggested document discovery by date/location overlap with known persons.
* Ability to search within a document to find potential people to add to the People table.
## Planning Notes
- Keep architecture, schema, and UI contracts synchronized in `docs/` as each version lands.
- Prefer explicit schema migration over runtime compatibility write paths.
- Preserve evidence/provenance guarantees when adding new AI-powered features.
- GEDCOM/FamilySearch data is external, collaborative, and mutable; treat it as a managed
cache bridged via `fs_id`, never as a replacement for archival evidence recorded from
transcribed documents.
+101 -4
View File
@@ -1,14 +1,16 @@
# Data Model and Persistence Schema (Current Baseline: V5.1)
# Data Model and Persistence Schema (Current Baseline: V6.1)
This document is the field-accurate V5.1 schema contract aligned to `src/transcription/db/models.py`.
This document is the field-accurate V6.1 schema contract aligned to `src/transcription/db/models.py`.
## Source of Truth Anchors
- `src/transcription/db/models.py:60-78` (status and purpose enums)
- `src/transcription/db/models.py` (status and purpose enums, including maintenance lifecycle enums)
- `src/transcription/db/models.py:80-120` (`DocumentType`, `PersonRole`)
- `src/transcription/db/models.py:122-172` (`Tag`, `Document`)
- `src/transcription/db/models.py:175-281` (`Person`, `Photo`, `DocumentPerson`, `DocumentTag`)
- `src/transcription/db/models.py` (`Person`, `GenealogyPerson`, `GenealogyFamily`, `GenealogyFamilyChild`, `GenealogyCitation`)
- `src/transcription/db/models.py` (`Photo`, `DocumentPerson`, `DocumentTag`)
- `src/transcription/db/models.py:285-347` (`Job`)
- `src/transcription/db/models.py` (`MaintenanceRun`)
- `src/transcription/db/models.py:350-462` (`Source`, `JobSource`)
- `src/transcription/db/models.py:465-522` (`ExecutionAttempt`)
@@ -24,12 +26,22 @@ erDiagram
Person ||--o{ DocumentPerson : links
Person ||--o{ PersonTag : tagged
Person ||--o{ Photo : owns
GenealogyPerson ||--o{ GenealogyFamily : husband
GenealogyPerson ||--o{ GenealogyFamily : wife
GenealogyPerson ||--o{ GenealogyFamilyChild : child
GenealogyFamily ||--o{ GenealogyFamilyChild : includes
GenealogyPerson ||--o{ GenealogyCitation : cited
GenealogyFamily ||--o{ GenealogyCitation : cited
Document ||--o{ GenealogyCitation : evidence
PersonRole ||--o{ DocumentPerson : labels
Tag ||--o{ DocumentTag : labels
Tag ||--o{ PersonTag : labels
Job ||--o{ JobSource : includes
Source ||--o{ JobSource : participates
JobSource ||--o{ ExecutionAttempt : attempts
MaintenanceRun {
uuid id PK
}
```
## Authoritative Enumerations
@@ -54,6 +66,19 @@ erDiagram
- `transcription`
- `retranscription`
### MaintenanceJobType
- `backup`
- `storage_reconciliation`
- `gedcom_import`
### MaintenanceRunStatus
- `queued`
- `processing`
- `succeeded`
- `failed`
## Field-Accurate Table Contracts
### `DocumentType`
@@ -126,6 +151,62 @@ erDiagram
| `created_at` | `datetime` | default now |
| `updated_at` | `datetime` | default now, onupdate |
### `GenealogyPerson`
| Field | Type | Notes |
| :--- | :--- | :--- |
| `id` | `UUID` | PK |
| `fs_id` | `str` | unique, indexed FamilySearch identifier |
| `full_name` | `str` | required |
| `birth_date` | `date \| None` | optional |
| `birth_date_raw` | `str \| None` | optional |
| `birth_place` | `str \| None` | optional |
| `death_date` | `date \| None` | optional |
| `death_date_raw` | `str \| None` | optional |
| `death_place` | `str \| None` | optional |
| `created_at` | `datetime` | default now |
| `updated_at` | `datetime` | default now, onupdate |
### `GenealogyFamily`
| Field | Type | Notes |
| :--- | :--- | :--- |
| `id` | `UUID` | PK |
| `fs_family_id` | `str` | unique, indexed FamilySearch family identifier |
| `husband_id` | `UUID \| None` | nullable FK -> `genealogy_person.id`, indexed |
| `wife_id` | `UUID \| None` | nullable FK -> `genealogy_person.id`, indexed |
| `marriage_date` | `date \| None` | optional |
| `marriage_date_raw` | `str \| None` | optional |
| `marriage_place` | `str \| None` | optional |
| `created_at` | `datetime` | default now |
| `updated_at` | `datetime` | default now, onupdate |
### `GenealogyFamilyChild`
| Field | Type | Notes |
| :--- | :--- | :--- |
| `id` | `UUID` | PK |
| `family_id` | `UUID` | FK -> `genealogy_family.id`, indexed |
| `child_id` | `UUID` | FK -> `genealogy_person.id`, indexed |
| `relationship_type` | `str \| None` | optional |
| `created_at` | `datetime` | default now |
Constraint:
- `UniqueConstraint(family_id, child_id)` named `uq_genealogy_family_child`
### `GenealogyCitation`
| Field | Type | Notes |
| :--- | :--- | :--- |
| `id` | `UUID` | PK |
| `genealogy_person_id` | `UUID \| None` | nullable FK -> `genealogy_person.id`, indexed |
| `genealogy_family_id` | `UUID \| None` | nullable FK -> `genealogy_family.id`, indexed |
| `fact_type` | `GenealogyCitationFactType` | enum: `birth`, `death`, `marriage`, `other` |
| `raw_citation_text` | `str` | required raw GEDCOM citation text |
| `source_kind` | `GenealogyCitationSourceKind` | enum: `familysearch_imported`, `transcription_evidence` |
| `document_id` | `UUID \| None` | nullable FK -> `document.id`, indexed |
| `created_at` | `datetime` | default now |
### `Photo`
| Field | Type | Notes |
@@ -201,6 +282,22 @@ Constraint:
Index:
- `Index("ix_job_status_date_created", "status", "date_created")`
### `MaintenanceRun`
| Field | Type | Notes |
| :--- | :--- | :--- |
| `id` | `UUID` | PK |
| `job_type` | `MaintenanceJobType` | non-null enum |
| `status` | `MaintenanceRunStatus` | non-null enum, default `queued` |
| `started_at` | `datetime \| None` | optional |
| `finished_at` | `datetime \| None` | optional |
| `triggered_by` | `str \| None` | optional |
| `summary` | `str \| None` | optional |
| `log_path` | `str \| None` | optional, log-root-relative POSIX path |
| `error_detail` | `str \| None` | optional internal failure detail |
| `created_at` | `datetime` | default now |
| `updated_at` | `datetime` | default now, onupdate |
### `Source`
| Field | Type | Notes |
+2 -1
View File
@@ -13,6 +13,7 @@ These documents are written for maintainers and AI contributors. They are behavi
- [People](pages/people.md)
- [Jobs](pages/jobs.md)
- [Sources](pages/sources.md)
- [Settings](pages/settings.md)
NiceGUI registers the routes shown in each contract without the `/ui` prefix. The application mounts NiceGUI under `/ui`, so `/documents` in page code is served to a browser as `/ui/documents`.
@@ -56,4 +57,4 @@ Each page contract contains:
## Current Baseline
These contracts describe the current V5.1 baseline.
These contracts describe the current V6.1 baseline.
+24 -8
View File
@@ -11,10 +11,11 @@ Documents manages the archival record for each historical artifact independently
| `/documents` | Searchable archival Document list. |
| `/documents/new` | Create a Document. |
| `/documents/{document_id}` | View one Document and its related records. |
| `/documents/{document_id}/info` | View archival metadata and system logistics for one Document. |
| `/documents/{document_id}/edit` | Edit metadata and the complete Linked People set. |
| `/documents/{document_id}/delete` | Confirm or block deletion. |
| `/documents/{document_id}/jobs` | Show Jobs belonging to the Document. |
| `/documents/{document_id}/sources` | Redirect to the Document-filtered Sources list. |
| `/documents/{document_id}/sources` | Source-image gallery for the Document. |
| `/documents/{document_id}/print` | Preview and browser-print the persisted Document. |
## List Behavior
@@ -22,12 +23,14 @@ Documents manages the archival record for each historical artifact independently
- The title is **Archival Documents**.
- **Create new document** opens the create route.
- The table defaults to Document Title order and supports search and column sorting.
- Columns are Document Title, Author, Tags, Document Date, Type, and # Sources.
- Columns are Document Title, Author, Tags, Document Date, Type, # Sources, and Transcription Status.
- Document Title is left-aligned; the remaining columns are centered.
- Author lists all linked people in the `author` role.
- # Sources reflects the count of linked Source rows for each Document.
- Transcription Status reflects the most recent Job status for that Document; documents with no Jobs show a blank marker.
- Date display prefers exact date, then approximate date, then `Unknown`.
- Selecting a row opens Document Detail.
- Row navigation includes list context so Document Detail provides **Back to Documents**.
- No records displays `No documents found in repository.`
## Create and Edit Behavior
@@ -68,14 +71,27 @@ Rules:
## Detail Behavior
- The heading shows name, type, and internal ID.
- The first Source, when present, appears in the dark-room viewer.
- Archival Metadata shows authors, Document Type, tags, Document date (`MM-DD-YYYY` for exact dates), location, and archive identifier. Notes appear in a separate archival-notes block within the same card.
- System Logistics shows created and updated timestamps.
- The header includes a contextual back action: **Back to Documents** by default, **Back to Person** when opened from Person Detail, and **Back to Job** when opened from Job Detail.
- The detail workspace shows a Source-style pan/zoom media viewer with **Previous Page** / **Next Page** navigation for document source pages.
- The center column is **Editable Revision** for the active source page.
- Related People are grouped by role and link to Person Detail.
- **Sources & Pipeline Jobs** shows counts and actions for filtered Sources, Document Jobs, and adding a Job.
- **Edit Document**, **Print**, and **Delete** are available from the header.
- **Source Pages & Transcriptions** shows source/job counts and actions for source-image gallery, document jobs, and adding a Job.
- **Edit Document**, **Print**, **Document Details**, **View Source Detail**, and **Delete** are available from the header.
- Invalid IDs and missing Documents produce explicit states without rendering a partial page.
## Document Source Images Behavior
- `/documents/{document_id}/sources` shows the current Document's source pages in a thumbnail gallery.
- Each card shows the page number, stored filename, and an **Open Source Detail** action.
- The page includes a **Back to Document** action.
- No source pages displays an explicit empty state.
## Document Info Behavior
- `/documents/{document_id}/info` contains **Archival Metadata** and **System Logistics**.
- It includes a **Back to Document** action.
- Archival metadata includes authors, document type, tags, document date, location (linked when present), archive identifier, and notes.
## Print Behavior
- Print opens a dedicated preview for persisted Document data.
@@ -130,5 +146,5 @@ Rules:
## Known Limitations and Deferred Work
- Source page ordering remains read-only in V4.4.
- Source page ordering remains read-only.
- Printing other entities, batch printing, and server-side export formats are deferred.
+3 -1
View File
@@ -24,6 +24,7 @@ Jobs manages transcription processing runs. A Job belongs to one Document, links
- Search covers Job ID, document name, and status.
- Status is displayed as a semantic status chip.
- Selecting a row opens Job Detail.
- Global Job-list row navigation includes list context so Job Detail provides **Back to Jobs**.
- No records displays `No job records found in repository.`
## Create Behavior
@@ -43,8 +44,9 @@ Jobs manages transcription processing runs. A Job belongs to one Document, links
## Detail and Lifecycle Behavior
- The heading shows Job ID and a status badge.
- Job Detail includes a contextual back action: **Back to Jobs** by default and **Back to Document** when opened from a Document-filtered Job list.
- Execution Logistics shows provider, model, prompt, retry count, and last update.
- Document Links show a clickable Document Name, Sources count, and a single **View Sources** action using document filtering.
- Document Links show a clickable Document Name (with Job context), Sources count, and a single **View Sources** action using job filtering.
- Queued and processing Jobs show an auto-refresh notice and reload every four seconds.
- Polling stops when the Job becomes terminal or a refresh fails.
- Queued and processing Jobs expose **Cancel**.
+3 -1
View File
@@ -25,6 +25,7 @@ People manages reusable historical-person records. A Person may appear in many D
- # Documents reflects how many linked Documents each Person is connected to.
- Birth and death values independently prefer exact date, then approximate date, then `Unknown`.
- Selecting a row opens Person Detail.
- Row navigation includes list context so Person Detail provides **Back to People**.
- No records displays `No person records found in repository.`
## Create and Edit Behavior
@@ -55,6 +56,7 @@ Rules:
## Detail Behavior
- The header provides **New Document**, **Edit Person**, **Edit Photo(s)**, and **Delete**.
- The header includes a contextual back action: **Back to People** by default, and **Back to Document** when opened from Document Detail.
- **New Document** opens Document creation with this Person requested for author preselection.
- Person Detail shows a single-photo viewer with **Previous/Next** navigation; the page-level **Edit Photo(s)** header action opens photo management.
- Photo management (upload, description edit, set-primary, delete) is intentionally moved to `/people/{person_id}/photos`.
@@ -62,7 +64,7 @@ Rules:
- Birth and death place values are clickable links to Google Maps when present.
- FamilySearch ID is shown as a metadata value and is clickable to the FamilySearch person details route when present.
- Biography has an explicit empty value.
- Linked Documents render as a table with **Document Name**, **Role**, and **Number of Pages**; selecting a row opens Document Detail.
- Linked Documents render as a table with **Document Name**, **Document Date**, **Role**, and **Number of Pages**; selecting a row opens Document Detail.
- No links shows both an empty state and guidance to link from a Document workflow.
- System Logistics shows created and updated timestamps.
+22 -3
View File
@@ -2,13 +2,13 @@
## Purpose
Settings manages installation-local registries and editable text assets from one route.
Settings manages installation-local registries, safe runtime .env settings, and editable text assets from one route.
## Route
| Route | Purpose |
| --- | --- |
| `/settings` | Manage Document Types, Person Roles, Tags, Prompts, and Home Page Text. |
| `/settings` | Manage Runtime Settings, Document Types, Person Roles, Tags, Prompts, Home Page Text, and Maintenance runs. |
## Behavior
@@ -19,18 +19,37 @@ Settings manages installation-local registries and editable text assets from one
- **Tags**
- **Prompts**
- **Home Page Text**
- **Maintenance**
- **Runtime Settings**
- Runtime Settings exposes an allowlisted set of non-secret fields synchronized with `Settings` model fields except excluded secret/unsafe fields.
- Runtime Settings is rendered as a compact two-column editor (**Setting**, **Value**) in a centered, narrower responsive container.
- Runtime Settings persists changes to the resolved runtime env file, validates by constructing a `Settings` instance, and reports validation failures through the shared UI error presenter.
- `Settings` resolves its env file in this order: explicit `_env_file`, `ENV_FILE`, then the repository-root `.env.production`.
- Runtime Settings resolves its write target in this order: explicit function override (tests/tools), `RUNTIME_SETTINGS_ENV_FILE` environment variable (deployment override), `ENV_FILE`, then the repository-root `.env.production`.
- Runtime Settings changes require application restart to take effect.
- Runtime Settings renders a host-side restart command (`docker compose -f docker-compose.production.yml up -d --force-recreate app worker`) so operators can apply saved values without granting Docker control to the app container.
- Runtime Settings includes an explicit "Other settings not shown here" markdown table listing:
- secrets (`OPENROUTER_API_KEY`, `DATABASE__PASSWORD`)
- high-risk database connection settings (`DATABASE__DRIVER`, `DATABASE__PATH`, `DATABASE__HOST`, `DATABASE__PORT`, `DATABASE__DATABASE`, `DATABASE__USER`)
and deployment/helper keys (`CLOUDFLARE_TUNNEL_TOKEN`, `BACKUP_DIR`, `BACKUP_RETENTION_DAYS`, `RUNTIME_SETTINGS_ENV_FILE`, `ENV_FILE`, `COMPOSE_FILE`) plus legacy/deprecated keys (`POSTGRES_*`, `DATABASE_BACKUP_DIR`, `APP_DATA_BACKUP_DIR`, `UPLOADS_BACKUP_DIR`, `SYNOLOGY_BACKUP_DIR`), and directs edits for those keys to the resolved runtime env file path.
- Document Types, Person Roles, and Tags support Add/Edit/Delete with existing guardrails.
- Prompts exposes only `transcribe_document.md` for editing and restore-from-backup.
- Home Page Text edits the same Markdown content rendered on `/homepage`.
- Maintenance provides queue-backed **Run Backup** and **Run Storage Reconciliation** actions.
- Maintenance also provides GEDCOM upload and **Run GEDCOM Import** actions, using the same queue-backed `MaintenanceRun` history/log flow.
- Maintenance run history shows job type, status, started/finished timestamps, duration, summary, and log view/download actions.
- Maintenance actions enqueue work and signal the worker; the page itself does not execute shell commands directly.
## Acceptance Checklist
- `/ui/settings` renders all five tabs.
- `/ui/settings` renders all seven tabs.
- Registry and prompt workflows keep existing validation and error handling.
- Runtime Settings excludes secret fields and rejects invalid values.
- Saving Home Page Text persists content for the homepage view.
## Implementation Anchors
- `src/transcription/ui/pages/settings_page.py`
- `src/transcription/ui/runtime_settings_store.py`
- `src/transcription/ui/homepage_store.py`
- `tests/ui/test_pages_registration.py`
+6 -6
View File
@@ -8,7 +8,7 @@ Sources manages individual archived page/file records. It provides source-media
| Route | Purpose |
| --- | --- |
| `/sources` | Global or filtered Source list. |
| `/sources` | Document-filtered or Job-filtered Source list; global route redirects to Documents. |
| `/sources/{source_id}` | View media, transcription, revision, metadata, and evidence. |
| `/sources/{source_id}/delete` | Confirm or block deletion. |
@@ -16,11 +16,11 @@ The list accepts optional `document_id` and `job_id` query parameters. Document
## List Behavior
- The title is **Source Asset Records**, **Sources for Document**, or **Sources for Job** according to context.
- Global context provides **Create Job**.
- The global `/sources` route redirects to `/documents`.
- Filtered list titles are **Sources for Document** and **Sources for Job**.
- Filtered context provides **Back to Document** or **Back to Job**.
- Rows are ordered by page number and then upload name.
- Columns are Document Name, Page Number, Upload Title, Status, and Error Detail.
- Columns are Upload Title, Page Number, Document Name, Status, and Error Detail.
- Document Name, Upload Title, and Error Detail are left-aligned; Status is centered.
- Status labels are presented in uppercase for consistency with Jobs.
- Stored Filename is intentionally absent from the list.
@@ -30,7 +30,7 @@ The list accepts optional `document_id` and `job_id` query parameters. Document
## Detail Behavior
- The heading shows page number, upload name, and Source ID.
- **Back to Sources** returns to the global list.
- **Back to Document** returns to Document Detail for the active source page.
- **Retranscribe Source** opens Create Processing Job with this Source and its Document locked.
- **Delete Source** opens the guarded delete route.
- Previous and Next navigate only among Sources belonging to the same Document in page order; unavailable boundary actions are disabled.
@@ -96,4 +96,4 @@ The list accepts optional `document_id` and `job_id` query parameters. Document
## Planned Changes
- Source page reordering is deferred beyond V4.3 and may be reconsidered if a demonstrated workflow need emerges.
- Source page reordering remains deferred unless a demonstrated workflow need emerges.
-31
View File
@@ -1,31 +0,0 @@
# Tags Page Contract
## Purpose
Tags provides a dedicated browse/filter entry point for document tagging workflows.
## Route
| Route | Purpose |
| --- | --- |
| `/tags` | Browse Documents grouped by Tag and filter to one Tag. |
## Behavior
- The page title is **Tags**.
- When no tags exist, the page shows `No tags are configured yet.`
- A Tag filter select allows narrowing to one tag.
- Each rendered group header includes the tag label and document count.
- Document names are clickable and open Document Detail.
## Acceptance Checklist
- `/ui/tags` renders successfully from the main navigation.
- Group counts match the number of linked Documents per Tag.
- Filtering hides non-matching tag groups.
## Implementation Anchors
- `src/transcription/ui/pages/tags_page.py`
- `src/transcription/services/documents.py`
- `tests/ui/test_tags_page.py`
+1
View File
@@ -25,6 +25,7 @@ dependencies = [
"psycopg2-binary>=2.9.12",
"pydantic>=2.13.4",
"pydantic-settings>=2.9.1",
"python-gedcom>=1.1.0",
"sqlmodel>=0.0.25",
]
+1 -1
View File
@@ -50,12 +50,12 @@ async def _lifespan(app: FastAPI):
settings.upload_dir.mkdir(parents=True, exist_ok=True)
settings.prompt_dir.mkdir(parents=True, exist_ok=True)
settings.log_dir.mkdir(parents=True, exist_ok=True)
settings.database_backup_dir.mkdir(parents=True, exist_ok=True)
await _recover_stale_processing_jobs(app)
async with AsyncExitStack() as stack:
stack.push_async_callback(dispose_database_runtime)
if settings.run_embedded_worker:
stop_event, worker_notifier, worker_health = await stack.enter_async_context(
worker_consumer_lifespan(
session_factory=app.state.runtime.session_factory,
+8 -19
View File
@@ -1,4 +1,11 @@
"""Private-corpus benchmark contracts and deterministic text scoring."""
"""Private-corpus benchmark contracts and deterministic text scoring.
This module is retained as the implementation of the evaluation policy in
`docs/invariant/ai_evidence_and_provenance.md` §5. Application runtime paths do
not call it directly, but preserved execution attempts and manually reviewed
references need a deterministic scorer that remains importable for tests and
operator tooling.
"""
from __future__ import annotations
@@ -13,24 +20,6 @@ class BenchmarkModel(BaseModel):
model_config = ConfigDict(extra="forbid", frozen=True)
class BenchmarkItem(BenchmarkModel):
"""One private benchmark item referenced by archival identity."""
source_id: UUID
source_digest_sha256: str = Field(pattern=r"^[0-9a-f]{64}$")
categories: frozenset[str] = Field(min_length=1)
reference_transcription: str = Field(min_length=1)
class BenchmarkManifest(BenchmarkModel):
"""Versioned private benchmark definition without copied source media."""
schema_name: str = "transcription.private-benchmark"
schema_version: str = "1"
name: str = Field(min_length=1)
items: tuple[BenchmarkItem, ...] = Field(min_length=1)
class EditorialAssessment(BenchmarkModel):
"""Manually reviewed errors not represented adequately by CER or WER."""
+49 -4
View File
@@ -1,12 +1,13 @@
"""Centralized application configuration.
All settings are loaded from environment variables (or a .env file)
All settings are loaded from environment variables (or an env file)
once at startup. Provider-specific defaults (model names, base URLs)
are resolved by the provider adapters, not here.
"""
import copy
import logging.config
import os
from collections.abc import Sequence
from enum import StrEnum
from functools import cache
@@ -26,6 +27,16 @@ from pydantic_settings import BaseSettings
from pydantic_settings import SettingsConfigDict
logger = logging.getLogger(__name__)
PROJECT_ROOT = Path(__file__).resolve().parents[2]
DEFAULT_ENV_FILE_NAME = ".env.production"
def resolve_settings_env_file_path() -> Path:
"""Resolve the runtime env file independent of the process working directory."""
override = os.getenv("ENV_FILE", "").strip()
if override:
return Path(override)
return PROJECT_ROOT / DEFAULT_ENV_FILE_NAME
class Provider(StrEnum):
@@ -37,6 +48,7 @@ PromptFilename = Annotated[str, StringConstraints(strip_whitespace=True, min_len
Probability = Annotated[float, Field(ge=0.0, le=1.0)]
Temperature = Annotated[float, Field(ge=0.0, le=2.0)]
DEFAULT_PROVIDER_MODEL = "google/gemini-2.5-flash"
WORKER_STALE_TIMEOUT_MULTIPLIER = 3.0
class SqliteSettings(BaseModel):
@@ -65,7 +77,7 @@ DatabaseSettings = Annotated[
class Settings(BaseSettings):
model_config = SettingsConfigDict(
env_file=".env",
env_file=None,
env_file_encoding="utf-8",
extra="ignore",
env_nested_delimiter="__",
@@ -74,6 +86,11 @@ class Settings(BaseSettings):
frozen=True,
)
def __init__(self, /, **values: Any) -> None:
if "_env_file" not in values:
values["_env_file"] = resolve_settings_env_file_path()
super().__init__(**values)
# --- NiceGUI Server ---
host: str = "0.0.0.0"
port: int = 8000
@@ -98,6 +115,7 @@ class Settings(BaseSettings):
# --- runtime environment ---
environment: Literal["development", "test", "production"] = "development"
transcription_commit: NonEmptyStr | None = None
run_embedded_worker: bool = True
# --- persistence ---
database: DatabaseSettings = Field(default_factory=SqliteSettings)
@@ -107,14 +125,13 @@ class Settings(BaseSettings):
# --- filesystem paths ---
upload_dir: Path = Path("./data")
prompt_dir: Path = Path("./prompts")
database_backup_dir: Path = Path("./data/backups")
# --- worker reliability ---
worker_max_retries: int = Field(default=0, ge=0)
# Bounded only from below. Vision transcription of a dense page routinely runs
# well past twenty seconds, so an upper cap here would silently fail real work.
worker_provider_timeout_seconds: float = Field(default=30.0, gt=0.0)
worker_stale_job_seconds: float = Field(default=30.0, gt=0.0)
worker_stale_job_seconds: float = Field(default=90.0, gt=0.0)
worker_retry_backoff_seconds: float = Field(default=1.0, ge=0.0)
worker_shutdown_grace_seconds: float = Field(default=5.0, ge=0.0)
worker_poll_interval_seconds: float = Field(default=1.0, gt=0.0)
@@ -169,6 +186,34 @@ class Settings(BaseSettings):
return {**data, "provider_model": default_model, "provider_models": tuple(deduplicated)}
@model_validator(mode="before")
@classmethod
def _derive_worker_stale_job_seconds(cls, data: object) -> object:
"""Default stale-job recovery with margin over one provider timeout."""
if not isinstance(data, dict):
return data
if data.get("worker_stale_job_seconds") is not None:
return data
timeout = data.get("worker_provider_timeout_seconds", 30.0)
if not isinstance(timeout, (str, int, float)):
return data
try:
timeout_seconds = float(timeout)
except ValueError:
return data
return {
**data,
"worker_stale_job_seconds": timeout_seconds * WORKER_STALE_TIMEOUT_MULTIPLIER,
}
@model_validator(mode="after")
def _validate_worker_stale_job_seconds(self) -> "Settings":
"""Reject stale recovery that can fire before one provider timeout expires."""
if self.worker_stale_job_seconds <= self.worker_provider_timeout_seconds:
raise ValueError("WORKER_STALE_JOB_SECONDS must exceed WORKER_PROVIDER_TIMEOUT_SECONDS")
return self
@property
def should_bootstrap_schema(self) -> bool:
"""Return whether startup should auto-create schema for this environment."""
-11
View File
@@ -73,14 +73,3 @@ async def dispose_engine(database_url: str) -> None:
engine = _ENGINES.pop(database_url, None)
if engine is not None:
await engine.dispose()
async def dispose_all_engines() -> None:
while _ENGINES:
_, engine = _ENGINES.popitem()
await engine.dispose()
async def refresh_engine(database_url: str) -> AsyncEngine:
await dispose_engine(database_url)
return get_engine(database_url)
+172
View File
@@ -16,9 +16,12 @@ from uuid import uuid4
from sqlalchemy import URL
from sqlalchemy import MetaData
from sqlalchemy import Table
from sqlalchemy import bindparam
from sqlalchemy import create_engine
from sqlalchemy import func
from sqlalchemy import inspect as sqlalchemy_inspect
from sqlalchemy import select
from sqlalchemy import text
from sqlalchemy.engine import RowMapping
from sqlalchemy.engine import make_url
from sqlmodel import SQLModel
@@ -36,6 +39,8 @@ EXPORT_TABLE_ORDER = (
"tag",
"document",
"person",
"genealogy_person",
"genealogy_family",
"photo",
"document_person",
"document_tag",
@@ -44,9 +49,12 @@ EXPORT_TABLE_ORDER = (
"source",
"job_source",
"execution_attempt",
"genealogy_family_child",
"genealogy_citation",
)
BYTES_FIELDS = {"transport_body"}
VERIFICATION_TABLES = EXPORT_TABLE_ORDER
@dataclass(frozen=True)
@@ -142,21 +150,74 @@ def import_bundle(*, target_db_url: str, target_upload_dir: Path, bundle_dir: Pa
engine = create_engine(target_db_url)
try:
SQLModel.metadata.create_all(engine)
execution_attempt_ids = _collect_execution_attempt_ids(payload)
with engine.begin() as connection:
for table_name in reversed(EXPORT_TABLE_ORDER):
table = SQLModel.metadata.tables[table_name]
connection.execute(table.delete())
deferred_source_preferred_attempt_updates: list[dict[str, Any]] = []
for table_name in EXPORT_TABLE_ORDER:
rows = payload.get("tables", {}).get(table_name, [])
if not rows:
continue
table = SQLModel.metadata.tables[table_name]
if table_name == "source":
prepared_source_rows, updates = _prepare_source_rows_for_import(
rows=rows,
source_table=table,
execution_attempt_ids=execution_attempt_ids,
)
deferred_source_preferred_attempt_updates.extend(updates)
connection.execute(table.insert(), prepared_source_rows)
continue
connection.execute(table.insert(), [_deserialize_row(row, table) for row in rows])
if deferred_source_preferred_attempt_updates:
source_table = SQLModel.metadata.tables["source"]
connection.execute(
source_table.update()
.where(source_table.c.id == bindparam("source_id"))
.values(preferred_execution_attempt_id=bindparam("preferred_execution_attempt_id")),
deferred_source_preferred_attempt_updates,
)
finally:
engine.dispose()
def _collect_execution_attempt_ids(payload: dict[str, Any]) -> set[str]:
execution_attempt_rows = payload.get("tables", {}).get("execution_attempt", [])
return {_normalize_uuid_like(row.get("id")) for row in execution_attempt_rows if row.get("id") is not None}
def _prepare_source_rows_for_import(
*,
rows: list[dict[str, Any]],
source_table: Table,
execution_attempt_ids: set[str],
) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
prepared_source_rows: list[dict[str, Any]] = []
updates: list[dict[str, Any]] = []
for row in rows:
source_row = _deserialize_row(row, source_table)
source_id = source_row.get("id")
preferred_attempt_id = source_row.get("preferred_execution_attempt_id")
if (
source_id is not None
and preferred_attempt_id is not None
and _normalize_uuid_like(preferred_attempt_id) in execution_attempt_ids
):
updates.append(
{
"source_id": source_id,
"preferred_execution_attempt_id": preferred_attempt_id,
}
)
source_row["preferred_execution_attempt_id"] = None
prepared_source_rows.append(source_row)
return prepared_source_rows, updates
def _ensure_sqlite_target_parent_exists(target_db_url: str) -> None:
parsed = make_url(target_db_url)
if not parsed.drivername.startswith("sqlite"):
@@ -192,6 +253,24 @@ def migrate_via_bundle(paths: MigrationPaths) -> None:
)
@dataclass(frozen=True)
class MigrationVerificationReport:
source_counts: dict[str, int]
target_counts: dict[str, int]
mismatched_tables: dict[str, dict[str, int]]
integrity_violations: dict[str, int]
success: bool
def to_dict(self) -> dict[str, Any]:
return {
"success": self.success,
"source_counts": self.source_counts,
"target_counts": self.target_counts,
"mismatched_tables": self.mismatched_tables,
"integrity_violations": self.integrity_violations,
}
def sqlite_url_from_path(path: Path) -> str:
return URL.create(drivername="sqlite", database=str(path)).render_as_string(hide_password=False)
@@ -201,6 +280,87 @@ def default_sync_db_url(settings: Settings | None = None) -> str:
return get_database_url(runtime_settings).replace("+aiosqlite", "").replace("+asyncpg", "").replace("+psycopg", "")
def verify_migration(*, source_db_url: str, target_db_url: str) -> MigrationVerificationReport:
source_counts = _table_counts(source_db_url)
target_counts = _table_counts(target_db_url)
mismatched_tables = {
table_name: {"source": source_counts[table_name], "target": target_counts[table_name]}
for table_name in VERIFICATION_TABLES
if source_counts[table_name] != target_counts[table_name]
}
integrity_violations = _integrity_violations(target_db_url)
success = not mismatched_tables and all(count == 0 for count in integrity_violations.values())
return MigrationVerificationReport(
source_counts=source_counts,
target_counts=target_counts,
mismatched_tables=mismatched_tables,
integrity_violations=integrity_violations,
success=success,
)
def _table_counts(db_url: str) -> dict[str, int]:
engine = create_engine(db_url)
try:
metadata = MetaData()
metadata.reflect(bind=engine)
counts: dict[str, int] = {}
with engine.connect() as connection:
for table_name in VERIFICATION_TABLES:
table = metadata.tables.get(table_name)
if table is None:
counts[table_name] = 0
continue
counts[table_name] = int(connection.execute(select(func.count()).select_from(table)).scalar_one())
return counts
finally:
engine.dispose()
def _integrity_violations(db_url: str) -> dict[str, int]:
checks = {
"orphan_source_document": (
"select count(*) from source s left join document d on d.id = s.document_id where d.id is null"
),
"orphan_job_document": (
"select count(*) from job j left join document d on d.id = j.document_id where d.id is null"
),
"orphan_job_source_job": (
"select count(*) from job_source js left join job j on j.id = js.job_id where j.id is null"
),
"orphan_job_source_source": (
"select count(*) from job_source js left join source s on s.id = js.source_id where s.id is null"
),
"orphan_attempt_job_source": (
"select count(*) from execution_attempt ea "
"left join job_source js on js.id = ea.job_source_id "
"where js.id is null"
),
"orphan_attempt_job": (
"select count(*) from execution_attempt ea left join job j on j.id = ea.job_id where j.id is null"
),
"orphan_attempt_source": (
"select count(*) from execution_attempt ea left join source s on s.id = ea.source_id where s.id is null"
),
"duplicate_attempt_numbers": (
"select count(*) from ("
" select job_id, source_id, attempt_number, count(*) as c"
" from execution_attempt"
" group by job_id, source_id, attempt_number"
" having count(*) > 1"
") x"
),
}
engine = create_engine(db_url)
try:
with engine.connect() as connection:
return {
check_name: int(connection.execute(text(query)).scalar_one()) for check_name, query in checks.items()
}
finally:
engine.dispose()
def _serialize_row(row: RowMapping, *, table_name: str, source_upload_dir: Path) -> dict[str, Any]:
serialized: dict[str, Any] = {}
for raw_key, value in row.items():
@@ -283,6 +443,18 @@ def _deserialize_value(python_type: type[Any], value: Any) -> Any:
return value
def _normalize_uuid_like(value: Any) -> str:
if isinstance(value, UUID):
return str(value)
if isinstance(value, str):
text_value = value.strip()
try:
return str(UUID(text_value))
except ValueError:
return text_value
return str(value)
def _canonical_media_relative_path(value: str, *, source_upload_dir: Path, preferred_prefix: str) -> str:
normalized = value.strip().replace("\\", "/")
lowered = normalized.casefold()
+243 -32
View File
@@ -46,6 +46,11 @@ def _loaded_attribute(instance: object, attribute: str) -> Any | None:
return state.dict.get(attribute)
def _utc_now_naive() -> datetime:
"""Return a UTC timestamp stored as a naive datetime."""
return datetime.now(UTC).replace(tzinfo=None)
class JSONBCompat(TypeDecorator):
"""JSONB for PostgreSQL and JSON for SQLite/testing backends."""
@@ -77,6 +82,31 @@ class JobPurpose(StrEnum):
RETRANSCRIPTION = "retranscription"
class MaintenanceJobType(StrEnum):
BACKUP = "backup"
STORAGE_RECONCILIATION = "storage_reconciliation"
GEDCOM_IMPORT = "gedcom_import"
class MaintenanceRunStatus(StrEnum):
QUEUED = "queued"
PROCESSING = "processing"
SUCCEEDED = "succeeded"
FAILED = "failed"
class GenealogyCitationFactType(StrEnum):
BIRTH = "birth"
DEATH = "death"
MARRIAGE = "marriage"
OTHER = "other"
class GenealogyCitationSourceKind(StrEnum):
FAMILYSEARCH_IMPORTED = "familysearch_imported"
TRANSCRIPTION_EVIDENCE = "transcription_evidence"
class DocumentType(SQLModel, table=True):
"""Registry of allowed document types."""
@@ -87,10 +117,10 @@ class DocumentType(SQLModel, table=True):
label: str
normalized_label: str = Field(index=True, unique=True)
is_active: bool = True
created_at: datetime = Field(default_factory=lambda: datetime.now(UTC))
created_at: datetime = Field(default_factory=_utc_now_naive)
updated_at: datetime = Field(
default_factory=lambda: datetime.now(UTC),
sa_column_kwargs={"onupdate": lambda: datetime.now(UTC)},
default_factory=_utc_now_naive,
sa_column_kwargs={"onupdate": _utc_now_naive},
)
documents: list["Document"] = Relationship(
@@ -108,10 +138,10 @@ class PersonRole(SQLModel, table=True):
label: str
normalized_label: str = Field(index=True, unique=True)
is_active: bool = True
created_at: datetime = Field(default_factory=lambda: datetime.now(UTC))
created_at: datetime = Field(default_factory=_utc_now_naive)
updated_at: datetime = Field(
default_factory=lambda: datetime.now(UTC),
sa_column_kwargs={"onupdate": lambda: datetime.now(UTC)},
default_factory=_utc_now_naive,
sa_column_kwargs={"onupdate": _utc_now_naive},
)
document_people: list["DocumentPerson"] = Relationship(
@@ -129,10 +159,10 @@ class Tag(SQLModel, table=True):
label: str
normalized_label: str = Field(index=True, unique=True)
is_active: bool = True
created_at: datetime = Field(default_factory=lambda: datetime.now(UTC))
created_at: datetime = Field(default_factory=_utc_now_naive)
updated_at: datetime = Field(
default_factory=lambda: datetime.now(UTC),
sa_column_kwargs={"onupdate": lambda: datetime.now(UTC)},
default_factory=_utc_now_naive,
sa_column_kwargs={"onupdate": _utc_now_naive},
)
document_tags: list["DocumentTag"] = Relationship(
@@ -156,10 +186,10 @@ class Document(SQLModel, table=True):
location_created: str | None = None
notes: str | None = None
archive_identifier: str | None = None
created_at: datetime = Field(default_factory=lambda: datetime.now(UTC))
created_at: datetime = Field(default_factory=_utc_now_naive)
updated_at: datetime = Field(
default_factory=lambda: datetime.now(UTC),
sa_column_kwargs={"onupdate": lambda: datetime.now(UTC)},
default_factory=_utc_now_naive,
sa_column_kwargs={"onupdate": _utc_now_naive},
)
jobs: list["Job"] = Relationship(back_populates="document", sa_relationship_kwargs={"lazy": "raise"})
@@ -194,10 +224,10 @@ class Person(SQLModel, table=True):
default=None,
sa_column=Column("metadata", JSONBCompat(), nullable=True),
)
created_at: datetime = Field(default_factory=lambda: datetime.now(UTC))
created_at: datetime = Field(default_factory=_utc_now_naive)
updated_at: datetime = Field(
default_factory=lambda: datetime.now(UTC),
sa_column_kwargs={"onupdate": lambda: datetime.now(UTC)},
default_factory=_utc_now_naive,
sa_column_kwargs={"onupdate": _utc_now_naive},
)
document_people: list["DocumentPerson"] = Relationship(
@@ -218,6 +248,146 @@ class Person(SQLModel, table=True):
return f"{self.given_names} {self.last_name}".strip()
class GenealogyPerson(SQLModel, table=True):
"""An individual imported from a GEDCOM export."""
__tablename__ = "genealogy_person"
id: UUID = Field(default_factory=uuid4, primary_key=True)
fs_id: str = Field(index=True, unique=True)
full_name: str
birth_date: date | None = None
birth_date_raw: str | None = None
birth_place: str | None = None
death_date: date | None = None
death_date_raw: str | None = None
death_place: str | None = None
created_at: datetime = Field(default_factory=_utc_now_naive)
updated_at: datetime = Field(
default_factory=_utc_now_naive,
sa_column_kwargs={"onupdate": _utc_now_naive},
)
husband_families: list["GenealogyFamily"] = Relationship(
back_populates="husband",
sa_relationship_kwargs={"lazy": "raise", "foreign_keys": "[GenealogyFamily.husband_id]"},
)
wife_families: list["GenealogyFamily"] = Relationship(
back_populates="wife",
sa_relationship_kwargs={"lazy": "raise", "foreign_keys": "[GenealogyFamily.wife_id]"},
)
child_family_memberships: list["GenealogyFamilyChild"] = Relationship(
back_populates="child",
sa_relationship_kwargs={"lazy": "raise"},
)
citations: list["GenealogyCitation"] = Relationship(
back_populates="genealogy_person",
sa_relationship_kwargs={"lazy": "raise"},
)
class GenealogyFamily(SQLModel, table=True):
"""A family linking two GenealogyPerson records."""
__tablename__ = "genealogy_family"
id: UUID = Field(default_factory=uuid4, primary_key=True)
fs_family_id: str = Field(index=True, unique=True)
husband_id: UUID | None = Field(default=None, foreign_key="genealogy_person.id", index=True)
wife_id: UUID | None = Field(default=None, foreign_key="genealogy_person.id", index=True)
marriage_date: date | None = None
marriage_date_raw: str | None = None
marriage_place: str | None = None
created_at: datetime = Field(default_factory=_utc_now_naive)
updated_at: datetime = Field(
default_factory=_utc_now_naive,
sa_column_kwargs={"onupdate": _utc_now_naive},
)
husband: Optional["GenealogyPerson"] = Relationship(
back_populates="husband_families",
sa_relationship_kwargs={"lazy": "raise", "foreign_keys": "[GenealogyFamily.husband_id]"},
)
wife: Optional["GenealogyPerson"] = Relationship(
back_populates="wife_families",
sa_relationship_kwargs={"lazy": "raise", "foreign_keys": "[GenealogyFamily.wife_id]"},
)
children: list["GenealogyFamilyChild"] = Relationship(
back_populates="family",
sa_relationship_kwargs={"lazy": "raise"},
)
citations: list["GenealogyCitation"] = Relationship(
back_populates="genealogy_family",
sa_relationship_kwargs={"lazy": "raise"},
)
class GenealogyFamilyChild(SQLModel, table=True):
"""Junction table for child membership within a genealogy family."""
__tablename__ = "genealogy_family_child"
id: UUID = Field(default_factory=uuid4, primary_key=True)
family_id: UUID = Field(foreign_key="genealogy_family.id", index=True)
child_id: UUID = Field(foreign_key="genealogy_person.id", index=True)
relationship_type: str | None = None
created_at: datetime = Field(default_factory=_utc_now_naive)
__table_args__ = (UniqueConstraint("family_id", "child_id", name="uq_genealogy_family_child"),)
family: Optional["GenealogyFamily"] = Relationship(
back_populates="children",
sa_relationship_kwargs={"lazy": "raise"},
)
child: Optional["GenealogyPerson"] = Relationship(
back_populates="child_family_memberships",
sa_relationship_kwargs={"lazy": "raise"},
)
class GenealogyCitation(SQLModel, table=True):
"""A source citation attached to a genealogical fact."""
__tablename__ = "genealogy_citation"
id: UUID = Field(default_factory=uuid4, primary_key=True)
genealogy_person_id: UUID | None = Field(default=None, foreign_key="genealogy_person.id", index=True)
genealogy_family_id: UUID | None = Field(default=None, foreign_key="genealogy_family.id", index=True)
fact_type: GenealogyCitationFactType = Field(
sa_column=Column(
SAEnum(
GenealogyCitationFactType,
values_callable=lambda enum_cls: [item.value for item in enum_cls],
native_enum=False,
),
nullable=False,
)
)
raw_citation_text: str
source_kind: GenealogyCitationSourceKind = Field(
sa_column=Column(
SAEnum(
GenealogyCitationSourceKind,
values_callable=lambda enum_cls: [item.value for item in enum_cls],
native_enum=False,
),
nullable=False,
)
)
document_id: UUID | None = Field(default=None, foreign_key="document.id", index=True)
created_at: datetime = Field(default_factory=_utc_now_naive)
genealogy_person: Optional["GenealogyPerson"] = Relationship(
back_populates="citations",
sa_relationship_kwargs={"lazy": "raise"},
)
genealogy_family: Optional["GenealogyFamily"] = Relationship(
back_populates="citations",
sa_relationship_kwargs={"lazy": "raise"},
)
document: Optional["Document"] = Relationship(sa_relationship_kwargs={"lazy": "raise"})
class Photo(SQLModel, table=True):
"""A reusable image record for Person and homepage galleries."""
@@ -228,10 +398,10 @@ class Photo(SQLModel, table=True):
path: str
description: str | None = None
is_primary: bool = False
created_at: datetime = Field(default_factory=lambda: datetime.now(UTC))
created_at: datetime = Field(default_factory=_utc_now_naive)
updated_at: datetime = Field(
default_factory=lambda: datetime.now(UTC),
sa_column_kwargs={"onupdate": lambda: datetime.now(UTC)},
default_factory=_utc_now_naive,
sa_column_kwargs={"onupdate": _utc_now_naive},
)
person: Optional["Person"] = Relationship(
@@ -249,10 +419,10 @@ class DocumentPerson(SQLModel, table=True):
document_id: UUID = Field(foreign_key="document.id", index=True)
person_id: UUID = Field(foreign_key="person.id", index=True)
role_id: UUID = Field(foreign_key="person_role.id", index=True)
created_at: datetime = Field(default_factory=lambda: datetime.now(UTC))
created_at: datetime = Field(default_factory=_utc_now_naive)
updated_at: datetime = Field(
default_factory=lambda: datetime.now(UTC),
sa_column_kwargs={"onupdate": lambda: datetime.now(UTC)},
default_factory=_utc_now_naive,
sa_column_kwargs={"onupdate": _utc_now_naive},
)
__table_args__ = (UniqueConstraint("document_id", "person_id", name="uq_document_person"),)
@@ -276,10 +446,10 @@ class DocumentTag(SQLModel, table=True):
id: UUID = Field(default_factory=uuid4, primary_key=True)
document_id: UUID = Field(foreign_key="document.id", index=True)
tag_id: UUID = Field(foreign_key="tag.id", index=True)
created_at: datetime = Field(default_factory=lambda: datetime.now(UTC))
created_at: datetime = Field(default_factory=_utc_now_naive)
updated_at: datetime = Field(
default_factory=lambda: datetime.now(UTC),
sa_column_kwargs={"onupdate": lambda: datetime.now(UTC)},
default_factory=_utc_now_naive,
sa_column_kwargs={"onupdate": _utc_now_naive},
)
__table_args__ = (UniqueConstraint("document_id", "tag_id", name="uq_document_tag"),)
@@ -302,10 +472,10 @@ class PersonTag(SQLModel, table=True):
id: UUID = Field(default_factory=uuid4, primary_key=True)
person_id: UUID = Field(foreign_key="person.id", index=True)
tag_id: UUID = Field(foreign_key="tag.id", index=True)
created_at: datetime = Field(default_factory=lambda: datetime.now(UTC))
created_at: datetime = Field(default_factory=_utc_now_naive)
updated_at: datetime = Field(
default_factory=lambda: datetime.now(UTC),
sa_column_kwargs={"onupdate": lambda: datetime.now(UTC)},
default_factory=_utc_now_naive,
sa_column_kwargs={"onupdate": _utc_now_naive},
)
__table_args__ = (UniqueConstraint("person_id", "tag_id", name="uq_person_tag"),)
@@ -351,10 +521,10 @@ class Job(SQLModel, table=True):
default=JobPurpose.TRANSCRIPTION.value,
),
)
date_created: datetime = Field(default_factory=lambda: datetime.now(UTC))
date_created: datetime = Field(default_factory=_utc_now_naive)
date_updated: datetime = Field(
default_factory=lambda: datetime.now(UTC),
sa_column_kwargs={"onupdate": lambda: datetime.now(UTC)},
default_factory=_utc_now_naive,
sa_column_kwargs={"onupdate": _utc_now_naive},
)
provider: str | None = None
model: str | None = None
@@ -385,6 +555,47 @@ class Job(SQLModel, table=True):
return "unknown"
class MaintenanceRun(SQLModel, table=True):
"""A queued/processed maintenance task execution record."""
__tablename__ = "maintenance_run"
id: UUID = Field(default_factory=uuid4, primary_key=True)
job_type: MaintenanceJobType = Field(
sa_column=Column(
SAEnum(
MaintenanceJobType,
values_callable=lambda enum_cls: [item.value for item in enum_cls],
native_enum=False,
),
nullable=False,
)
)
status: MaintenanceRunStatus = Field(
default=MaintenanceRunStatus.QUEUED,
sa_column=Column(
SAEnum(
MaintenanceRunStatus,
values_callable=lambda enum_cls: [item.value for item in enum_cls],
native_enum=False,
),
nullable=False,
default=MaintenanceRunStatus.QUEUED.value,
),
)
started_at: datetime | None = None
finished_at: datetime | None = None
triggered_by: str | None = None
summary: str | None = None
log_path: str | None = None
error_detail: str | None = None
created_at: datetime = Field(default_factory=_utc_now_naive)
updated_at: datetime = Field(
default_factory=_utc_now_naive,
sa_column_kwargs={"onupdate": _utc_now_naive},
)
class Source(SQLModel, table=True):
"""A document source image or PDF page."""
@@ -413,7 +624,7 @@ class Source(SQLModel, table=True):
),
)
revised_text: str | None = None
date_uploaded: datetime = Field(default_factory=lambda: datetime.now(UTC))
date_uploaded: datetime = Field(default_factory=_utc_now_naive)
date_revised: datetime | None = None
document: Optional["Document"] = Relationship(
@@ -551,7 +762,7 @@ class ExecutionAttempt(SQLModel, table=True):
started_at: datetime
finished_at: datetime
duration_ms: int = Field(ge=0)
created_at: datetime = Field(default_factory=lambda: datetime.now(UTC))
created_at: datetime = Field(default_factory=_utc_now_naive)
job_source: Optional["JobSource"] = Relationship(
back_populates="execution_attempts", sa_relationship_kwargs={"lazy": "raise"}
+6 -1
View File
@@ -32,6 +32,11 @@ def new_error_id() -> str:
return uuid4().hex[:8]
def exception_detail(exc: BaseException) -> str:
"""Return internal-only root-cause text for persisted diagnostics."""
return f"{type(exc).__name__}: {exc}"
class AppError(RuntimeError):
"""Base application error carrying user-safe handling metadata."""
@@ -115,7 +120,7 @@ def classify_unexpected_error(exc: Exception, *, operation: str) -> AppError:
category=ErrorCategory.INTERNAL_UNEXPECTED,
suggestion="Retry once. If it persists, review logs and report the error reference id.",
retriable=False,
detail=f"{type(exc).__name__}: {exc}",
detail=exception_detail(exc),
)
logger.error(
"Unexpected error operation=%s error_id=%s",
+2
View File
@@ -4,6 +4,7 @@ from transcription.config import Provider
from transcription.config import Settings
from transcription.config import get_settings
from transcription.providers.base import ProviderAuthError
from transcription.providers.base import ProviderCallEvidence
from transcription.providers.base import ProviderError
from transcription.providers.base import ProviderResponseError
from transcription.providers.base import TranscriptionMetadata
@@ -27,6 +28,7 @@ def get_transcription_provider(*, settings: Settings | None = None) -> Transcrip
__all__ = [
"OpenRouterTranscriptionProvider",
"ProviderAuthError",
"ProviderCallEvidence",
"ProviderError",
"ProviderResponseError",
"RequestManifest",
+11 -11
View File
@@ -1,5 +1,6 @@
"""Provider interfaces and validated shared contracts for transcription adapters."""
from dataclasses import dataclass
from typing import Protocol
from pydantic import BaseModel
@@ -37,6 +38,14 @@ class ProviderResponseError(ProviderError):
"""Raised when provider responses are malformed or unusable."""
@dataclass(slots=True)
class ProviderCallEvidence:
"""Caller-owned evidence sink for one provider invocation."""
request_manifest: RequestManifest | None = None
transport_evidence: TransportEvidence | None = None
class ProviderUsage(BaseModel):
"""Normalized provider token accounting."""
@@ -107,16 +116,6 @@ class TranscriptionProvider(Protocol):
"""Return the resolved model slug this adapter will call."""
...
@property
def current_request_manifest(self) -> RequestManifest | None:
"""Return the manifest for the most recent call, for failure evidence."""
...
@property
def current_transport_evidence(self) -> TransportEvidence | None:
"""Return transport-level evidence for the most recent call."""
...
async def transcribe(
self,
*,
@@ -127,8 +126,9 @@ class TranscriptionProvider(Protocol):
top_p: float | None = None,
source_reference: SourceEvidenceReference | None = None,
requested_model: str | None = None,
evidence_capture: ProviderCallEvidence | None = None,
) -> TranscriptionResult:
"""Transcribe the provided image according to the prompt text."""
"""Transcribe one source and write failure evidence into the provided capture sink."""
...
async def aclose(self) -> None:
+48 -36
View File
@@ -3,11 +3,13 @@
from __future__ import annotations
import base64
import contextvars
import hashlib
import json
import logging
from collections.abc import AsyncIterator
from collections.abc import Callable
from dataclasses import dataclass
from typing import Annotated
from typing import Any
from typing import Literal
@@ -25,6 +27,7 @@ from pydantic import ValidationError
from transcription.config import Settings
from transcription.config import get_settings
from transcription.providers.base import ProviderAuthError
from transcription.providers.base import ProviderCallEvidence
from transcription.providers.base import ProviderError
from transcription.providers.base import ProviderResponseError
from transcription.providers.base import ProviderUsage
@@ -65,19 +68,24 @@ class _CapturingAsyncClient:
def __init__(self, client: httpx.AsyncClient):
self._client = client
self.last_response: httpx.Response | None = None
self.last_body: bytes | None = None
self._active_capture: contextvars.ContextVar[_TransportCapture | None] = contextvars.ContextVar(
"openrouter_transport_capture",
default=None,
)
async def send(self, request: httpx.Request, **kwargs: Any) -> httpx.Response:
capture = self._active_capture.get()
response = await self._client.send(request, **kwargs)
self.last_response = response
if capture is None:
return response
capture.response = response
try:
self.last_body = response.content
capture.body = response.content
except httpx.ResponseNotRead:
stream = response.stream
if not isinstance(stream, httpx.AsyncByteStream):
raise
response.stream = _CapturingAsyncByteStream(stream, self._capture_body)
response.stream = _CapturingAsyncByteStream(stream, lambda body: self._capture_body(capture, body))
return response
def build_request(self, *args: Any, **kwargs: Any) -> httpx.Request:
@@ -86,12 +94,21 @@ class _CapturingAsyncClient:
async def aclose(self) -> None:
await self._client.aclose()
def reset(self) -> None:
self.last_response = None
self.last_body = None
def begin_capture(self, capture: _TransportCapture) -> contextvars.Token[_TransportCapture | None]:
return self._active_capture.set(capture)
def _capture_body(self, body: bytes) -> None:
self.last_body = body
def end_capture(self, token: contextvars.Token[_TransportCapture | None]) -> None:
self._active_capture.reset(token)
@staticmethod
def _capture_body(capture: _TransportCapture, body: bytes) -> None:
capture.body = body
@dataclass(slots=True)
class _TransportCapture:
response: httpx.Response | None = None
body: bytes | None = None
class _ProviderModel(BaseModel):
@@ -195,8 +212,6 @@ class OpenRouterTranscriptionProvider:
self._settings = settings or get_settings()
self._model = self._settings.provider_model or DEFAULT_OPENROUTER_MODEL
self._capturing_client: _CapturingAsyncClient | None = None
self._current_request_manifest: RequestManifest | None = None
self._current_transport_evidence: TransportEvidence | None = None
if client is None:
# httpx defaults every phase to 5s, which silently caps provider calls far
# below worker_provider_timeout_seconds. Track the configured budget instead.
@@ -218,18 +233,6 @@ class OpenRouterTranscriptionProvider:
"""Return the resolved OpenRouter model slug."""
return self._model
@property
def current_request_manifest(self) -> RequestManifest | None:
return self._current_request_manifest
@property
def current_transport_evidence(self) -> TransportEvidence | None:
if self._current_transport_evidence is not None:
return self._current_transport_evidence
if self._current_request_manifest is None:
return None
return self._captured_transport_evidence()
async def aclose(self) -> None:
if self._capturing_client is not None:
await self._capturing_client.aclose()
@@ -244,6 +247,7 @@ class OpenRouterTranscriptionProvider:
top_p: float | None = None,
source_reference: SourceEvidenceReference | None = None,
requested_model: str | None = None,
evidence_capture: ProviderCallEvidence | None = None,
) -> TranscriptionResult:
"""Send prompt + image to OpenRouter and return normalized text output."""
request = self._build_request(
@@ -261,18 +265,21 @@ class OpenRouterTranscriptionProvider:
temperature=temperature,
top_p=top_p,
)
self._current_request_manifest = manifest
self._current_transport_evidence = None
if self._capturing_client is not None:
self._capturing_client.reset()
if evidence_capture is not None:
evidence_capture.request_manifest = manifest
evidence_capture.transport_evidence = None
transport_capture = _TransportCapture()
token = self._capturing_client.begin_capture(transport_capture) if self._capturing_client is not None else None
try:
try:
response = await self._client.chat.send_async(
**request.model_dump(mode="json", exclude_none=True),
retries=None,
)
except Exception as exc:
transport = self._captured_transport_evidence()
self._current_transport_evidence = transport
transport = self._captured_transport_evidence(transport_capture)
if evidence_capture is not None:
evidence_capture.transport_evidence = transport
if isinstance(exc, openrouter_errors.UnauthorizedResponseError):
raise ProviderAuthError(
"OpenRouter authentication failed",
@@ -294,8 +301,9 @@ class OpenRouterTranscriptionProvider:
failure_phase=failure_phase,
) from exc
transport = self._captured_transport_evidence()
self._current_transport_evidence = transport
transport = self._captured_transport_evidence(transport_capture)
if evidence_capture is not None:
evidence_capture.transport_evidence = transport
raw_api_response = self._coerce_raw_response(response)
try:
validated_response = OpenRouterResponse.model_validate(raw_api_response)
@@ -334,6 +342,10 @@ class OpenRouterTranscriptionProvider:
request_manifest=manifest,
transport_evidence=transport,
)
finally:
client = self._capturing_client
if token is not None and client is not None:
client.end_capture(token)
def _build_request_manifest(
self,
@@ -345,6 +357,7 @@ class OpenRouterTranscriptionProvider:
top_p: float | None,
) -> RequestManifest | None:
if source_reference is None:
logger.warning("OpenRouter request manifest omitted because source evidence reference is missing.")
return None
request_payload = request.model_dump(mode="json", exclude_none=True)
sanitized_request = self._replace_embedded_media(request_payload, source_reference=source_reference)
@@ -393,16 +406,15 @@ class OpenRouterTranscriptionProvider:
return [self._replace_embedded_media(item, source_reference=source_reference) for item in value]
return value
def _captured_transport_evidence(self) -> TransportEvidence:
response = self._capturing_client.last_response if self._capturing_client is not None else None
def _captured_transport_evidence(self, capture: _TransportCapture) -> TransportEvidence:
response = capture.response
if response is None:
return TransportEvidence(response_received=False)
headers = filter_safe_response_headers(response.headers)
body = self._capturing_client.last_body if self._capturing_client is not None else None
return TransportEvidence(
response_received=True,
status_code=response.status_code,
body=body,
body=capture.body,
safe_headers=headers,
content_type=headers.get("content-type"),
content_encoding=headers.get("content-encoding"),
+37
View File
@@ -0,0 +1,37 @@
from __future__ import annotations
import asyncio
from collections.abc import Awaitable
from collections.abc import Callable
from typing import TypeVar
from sqlalchemy.exc import IntegrityError
ResultT = TypeVar("ResultT")
async def run_blocking(func: Callable[..., ResultT], /, *args, **kwargs) -> ResultT:
"""Run blocking CPU/filesystem work on a worker thread."""
return await asyncio.to_thread(func, *args, **kwargs)
async def insert_with_sequence_retry(
*,
max_retries: int,
operation: Callable[[int], Awaitable[ResultT]],
on_conflict: Callable[[int, IntegrityError], None] | None = None,
) -> ResultT:
"""Retry a sequence-based insert operation on unique-key conflicts."""
if max_retries < 1:
raise ValueError("max_retries must be at least 1")
for retry in range(1, max_retries + 1):
try:
return await operation(retry)
except IntegrityError as exc:
if on_conflict is not None:
on_conflict(retry, exc)
if retry == max_retries:
raise
raise RuntimeError("insert_with_sequence_retry exhausted retries without returning or raising")
+4
View File
@@ -11,6 +11,7 @@ from ..config import Settings
from .documents import DocumentService
from .evidence import EvidenceService
from .jobs import JobService
from .maintenance import MaintenanceService
from .people import PeopleService
from .photos import PhotosService
from .prompts import PromptStore
@@ -20,6 +21,7 @@ __all__ = [
"DocumentService",
"EvidenceService",
"JobService",
"MaintenanceService",
"PeopleService",
"PhotosService",
"PromptStore",
@@ -35,6 +37,7 @@ class ServiceBundle:
documents: DocumentService = field(default_factory=DocumentService)
sources: SourceService = field(default_factory=SourceService)
jobs: JobService = field(default_factory=JobService)
maintenance: MaintenanceService = field(default_factory=MaintenanceService)
people: PeopleService = field(default_factory=PeopleService)
photos: PhotosService = field(default_factory=PhotosService)
evidence: EvidenceService = field(default_factory=EvidenceService)
@@ -53,6 +56,7 @@ class ServiceBundle:
documents=DocumentService(session_factory=session_factory, settings=settings),
sources=SourceService(session_factory=session_factory, settings=settings),
jobs=JobService(session_factory=session_factory, settings=settings),
maintenance=MaintenanceService(session_factory=session_factory, settings=settings),
people=PeopleService(session_factory=session_factory, settings=settings),
photos=PhotosService(session_factory=session_factory, settings=settings),
evidence=EvidenceService(session_factory=session_factory, settings=settings),
+1
View File
@@ -303,6 +303,7 @@ class DocumentService(ServiceBase):
selectinload(Document.document_type_ref),
selectinload(Document.document_tags).selectinload(orm_attribute(DocumentTag.tag_ref)),
selectinload(Document.sources),
selectinload(Document.jobs),
)
result = await _session.exec(query)
return result.all()
+4
View File
@@ -16,6 +16,10 @@ class PromptLoadError(AppError):
"""Raised when prompt artifacts cannot be loaded safely."""
class PromptStoreError(PromptLoadError):
"""Raised when prompt storage validation or persistence fails."""
class TranscriptionError(AppError):
"""Raised when transcription execution fails."""
+599
View File
@@ -0,0 +1,599 @@
"""GEDCOM parsing and import helpers for maintenance-driven genealogy sync."""
from __future__ import annotations
from collections.abc import Mapping
from dataclasses import dataclass
from datetime import date
from pathlib import Path
from uuid import UUID
from uuid import uuid4
from gedcom.element.element import Element
from gedcom.parser import Parser
from sqlalchemy import or_
from sqlmodel import col
from sqlmodel import delete
from sqlmodel import select
from sqlmodel.ext.asyncio.session import AsyncSession
from transcription.db.models import GenealogyCitation
from transcription.db.models import GenealogyCitationFactType
from transcription.db.models import GenealogyCitationSourceKind
from transcription.db.models import GenealogyFamily
from transcription.db.models import GenealogyFamilyChild
from transcription.db.models import GenealogyPerson
from transcription.errors import AppError
from transcription.errors import ErrorCategory
_MONTHS = {
"JAN": 1,
"FEB": 2,
"MAR": 3,
"APR": 4,
"MAY": 5,
"JUN": 6,
"JUL": 7,
"AUG": 8,
"SEP": 9,
"OCT": 10,
"NOV": 11,
"DEC": 12,
}
@dataclass(frozen=True, slots=True)
class ParsedCitation:
fact_type: GenealogyCitationFactType
raw_citation_text: str
@dataclass(frozen=True, slots=True)
class ParsedPerson:
pointer: str
fs_id: str | None
full_name: str
birth_date: date | None
birth_date_raw: str | None
birth_place: str | None
death_date: date | None
death_date_raw: str | None
death_place: str | None
citations: tuple[ParsedCitation, ...]
@dataclass(frozen=True, slots=True)
class ParsedFamilyChild:
child_pointer: str
relationship_type: str | None
@dataclass(frozen=True, slots=True)
class ParsedFamily:
fs_family_id: str | None
husband_pointer: str | None
wife_pointer: str | None
marriage_date: date | None
marriage_date_raw: str | None
marriage_place: str | None
children: tuple[ParsedFamilyChild, ...]
citations: tuple[ParsedCitation, ...]
@dataclass(frozen=True, slots=True)
class ParsedGedcom:
people: tuple[ParsedPerson, ...]
families: tuple[ParsedFamily, ...]
@dataclass(frozen=True, slots=True)
class GedcomImportResult:
new_people: int
updated_people: int
skipped_people_without_fs_id: int
new_families: int
updated_families: int
skipped_families_without_fs_id: int
family_children: int
citations: int
class GedcomImportError(AppError):
"""Raised when GEDCOM content cannot be parsed or imported."""
def parse_gedcom(*, file_path: Path) -> ParsedGedcom:
parser = Parser()
try:
parser.parse_file(str(file_path))
except Exception as exc:
raise GedcomImportError(
"GEDCOM file could not be parsed.",
category=ErrorCategory.VALIDATION,
suggestion="Upload a GEDCOM 5.5.1-compatible export and retry.",
detail=f"{type(exc).__name__}: {exc}",
) from exc
people: list[ParsedPerson] = []
families: list[ParsedFamily] = []
for element in parser.get_root_child_elements():
tag = element.get_tag()
if tag == "INDI":
people.append(_parse_person(element))
elif tag == "FAM":
families.append(_parse_family(element))
return ParsedGedcom(people=tuple(people), families=tuple(families))
async def import_gedcom_file(*, session: AsyncSession, file_path: Path) -> GedcomImportResult:
parsed = parse_gedcom(file_path=file_path)
pointer_to_fs_id = _pointer_to_fs_id_map(parsed=parsed)
people_by_fs_id, new_people, updated_people, skipped_people_without_fs_id = await _upsert_people(
session=session,
parsed=parsed,
)
(
families_by_fs_id,
new_families,
updated_families,
skipped_families_without_fs_id,
family_children,
) = await _upsert_families(
session=session,
parsed=parsed,
pointer_to_fs_id=pointer_to_fs_id,
people_by_fs_id=people_by_fs_id,
)
citation_rows = _citation_rows(
parsed=parsed,
pointer_to_fs_id=pointer_to_fs_id,
people=people_by_fs_id,
families=families_by_fs_id,
)
await _replace_imported_citations(
session=session,
person_ids={item.id for item in people_by_fs_id.values()},
family_ids={item.id for item in families_by_fs_id.values()},
citations=citation_rows,
)
await session.commit()
return GedcomImportResult(
new_people=new_people,
updated_people=updated_people,
skipped_people_without_fs_id=skipped_people_without_fs_id,
new_families=new_families,
updated_families=updated_families,
skipped_families_without_fs_id=skipped_families_without_fs_id,
family_children=family_children,
citations=len(citation_rows),
)
async def _load_people_by_fs_id(*, session: AsyncSession, fs_ids: set[str]) -> dict[str, GenealogyPerson]:
if not fs_ids:
return {}
query = select(GenealogyPerson).where(col(GenealogyPerson.fs_id).in_(fs_ids))
return {person.fs_id: person for person in (await session.exec(query)).all()}
async def _load_families_by_fs_id(*, session: AsyncSession, fs_family_ids: set[str]) -> dict[str, GenealogyFamily]:
if not fs_family_ids:
return {}
query = select(GenealogyFamily).where(col(GenealogyFamily.fs_family_id).in_(fs_family_ids))
return {family.fs_family_id: family for family in (await session.exec(query)).all()}
def _pointer_to_fs_id_map(*, parsed: ParsedGedcom) -> dict[str, str]:
return {person.pointer: person.fs_id for person in parsed.people if person.pointer and person.fs_id is not None}
async def _upsert_people(
*,
session: AsyncSession,
parsed: ParsedGedcom,
) -> tuple[dict[str, GenealogyPerson], int, int, int]:
people_with_fs_id = [person for person in parsed.people if person.fs_id is not None]
fs_ids = {person.fs_id for person in people_with_fs_id if person.fs_id is not None}
people_by_fs_id = await _load_people_by_fs_id(session=session, fs_ids=fs_ids)
new_people = 0
updated_people = 0
for person in people_with_fs_id:
assert person.fs_id is not None
existing = people_by_fs_id.get(person.fs_id)
if existing is None:
existing = GenealogyPerson(
fs_id=person.fs_id,
full_name=person.full_name,
birth_date=person.birth_date,
birth_date_raw=person.birth_date_raw,
birth_place=person.birth_place,
death_date=person.death_date,
death_date_raw=person.death_date_raw,
death_place=person.death_place,
)
session.add(existing)
await session.flush()
people_by_fs_id[person.fs_id] = existing
new_people += 1
continue
if _apply_person_updates(existing=existing, person=person):
updated_people += 1
skipped = len(parsed.people) - len(people_with_fs_id)
return people_by_fs_id, new_people, updated_people, skipped
def _apply_person_updates(*, existing: GenealogyPerson, person: ParsedPerson) -> bool:
changed = False
fields = (
("full_name", person.full_name),
("birth_date", person.birth_date),
("birth_date_raw", person.birth_date_raw),
("birth_place", person.birth_place),
("death_date", person.death_date),
("death_date_raw", person.death_date_raw),
("death_place", person.death_place),
)
for name, value in fields:
if getattr(existing, name) != value:
setattr(existing, name, value)
changed = True
return changed
async def _upsert_families(
*,
session: AsyncSession,
parsed: ParsedGedcom,
pointer_to_fs_id: Mapping[str, str],
people_by_fs_id: dict[str, GenealogyPerson],
) -> tuple[dict[str, GenealogyFamily], int, int, int, int]:
families_with_fs_id = [family for family in parsed.families if family.fs_family_id is not None]
fs_family_ids = {family.fs_family_id for family in families_with_fs_id if family.fs_family_id is not None}
families_by_fs_id = await _load_families_by_fs_id(session=session, fs_family_ids=fs_family_ids)
new_families = 0
updated_families = 0
family_children = 0
for family in families_with_fs_id:
assert family.fs_family_id is not None
husband_id = _person_id_from_pointer(
pointer=family.husband_pointer,
pointer_to_fs_id=pointer_to_fs_id,
people=people_by_fs_id,
)
wife_id = _person_id_from_pointer(
pointer=family.wife_pointer,
pointer_to_fs_id=pointer_to_fs_id,
people=people_by_fs_id,
)
target = families_by_fs_id.get(family.fs_family_id)
if target is None:
target = GenealogyFamily(
fs_family_id=family.fs_family_id,
husband_id=husband_id,
wife_id=wife_id,
marriage_date=family.marriage_date,
marriage_date_raw=family.marriage_date_raw,
marriage_place=family.marriage_place,
)
session.add(target)
await session.flush()
families_by_fs_id[family.fs_family_id] = target
new_families += 1
elif _apply_family_updates(existing=target, family=family, husband_id=husband_id, wife_id=wife_id):
updated_families += 1
family_children += await _replace_family_children(
session=session,
family=family,
family_id=target.id,
pointer_to_fs_id=pointer_to_fs_id,
people=people_by_fs_id,
)
skipped = len(parsed.families) - len(families_with_fs_id)
return families_by_fs_id, new_families, updated_families, skipped, family_children
def _apply_family_updates(
*,
existing: GenealogyFamily,
family: ParsedFamily,
husband_id: UUID | None,
wife_id: UUID | None,
) -> bool:
changed = False
fields = (
("husband_id", husband_id),
("wife_id", wife_id),
("marriage_date", family.marriage_date),
("marriage_date_raw", family.marriage_date_raw),
("marriage_place", family.marriage_place),
)
for name, value in fields:
if getattr(existing, name) != value:
setattr(existing, name, value)
changed = True
return changed
async def _replace_family_children(
*,
session: AsyncSession,
family: ParsedFamily,
family_id: UUID,
pointer_to_fs_id: Mapping[str, str],
people: dict[str, GenealogyPerson],
) -> int:
await session.exec(delete(GenealogyFamilyChild).where(col(GenealogyFamilyChild.family_id) == family_id))
child_rows = _family_child_rows(
family=family,
family_id=family_id,
pointer_to_fs_id=pointer_to_fs_id,
people=people,
)
for child_row in child_rows:
session.add(child_row)
return len(child_rows)
def _person_id_from_pointer(
*,
pointer: str | None,
pointer_to_fs_id: Mapping[str, str],
people: dict[str, GenealogyPerson],
) -> UUID | None:
if pointer is None:
return None
fs_id = pointer_to_fs_id.get(pointer)
if fs_id is None:
return None
person = people.get(fs_id)
return person.id if person is not None else None
def _family_child_rows(
*,
family: ParsedFamily,
family_id: UUID,
pointer_to_fs_id: Mapping[str, str],
people: dict[str, GenealogyPerson],
) -> list[GenealogyFamilyChild]:
rows: list[GenealogyFamilyChild] = []
seen_child_ids: set[UUID] = set()
for child in family.children:
child_id = _person_id_from_pointer(
pointer=child.child_pointer,
pointer_to_fs_id=pointer_to_fs_id,
people=people,
)
if child_id is None or child_id in seen_child_ids:
continue
seen_child_ids.add(child_id)
rows.append(
GenealogyFamilyChild(
id=uuid4(),
family_id=family_id,
child_id=child_id,
relationship_type=child.relationship_type,
)
)
return rows
def _citation_rows(
*,
parsed: ParsedGedcom,
pointer_to_fs_id: Mapping[str, str],
people: dict[str, GenealogyPerson],
families: dict[str, GenealogyFamily],
) -> list[GenealogyCitation]:
rows: list[GenealogyCitation] = []
seen: set[tuple[UUID | None, UUID | None, str, str]] = set()
for person in parsed.people:
fs_id = pointer_to_fs_id.get(person.pointer)
if fs_id is None:
continue
person_row = people.get(fs_id)
if person_row is None:
continue
for citation in person.citations:
key = (person_row.id, None, citation.fact_type.value, citation.raw_citation_text)
if key in seen:
continue
seen.add(key)
rows.append(
GenealogyCitation(
id=uuid4(),
genealogy_person_id=person_row.id,
genealogy_family_id=None,
fact_type=citation.fact_type,
raw_citation_text=citation.raw_citation_text,
source_kind=GenealogyCitationSourceKind.FAMILYSEARCH_IMPORTED,
document_id=None,
)
)
for family in parsed.families:
if family.fs_family_id is None:
continue
family_row = families.get(family.fs_family_id)
if family_row is None:
continue
for citation in family.citations:
key = (None, family_row.id, citation.fact_type.value, citation.raw_citation_text)
if key in seen:
continue
seen.add(key)
rows.append(
GenealogyCitation(
id=uuid4(),
genealogy_person_id=None,
genealogy_family_id=family_row.id,
fact_type=citation.fact_type,
raw_citation_text=citation.raw_citation_text,
source_kind=GenealogyCitationSourceKind.FAMILYSEARCH_IMPORTED,
document_id=None,
)
)
return rows
async def _replace_imported_citations(
*,
session: AsyncSession,
person_ids: set[UUID],
family_ids: set[UUID],
citations: list[GenealogyCitation],
) -> None:
where_clauses = []
if person_ids:
where_clauses.append(col(GenealogyCitation.genealogy_person_id).in_(person_ids))
if family_ids:
where_clauses.append(col(GenealogyCitation.genealogy_family_id).in_(family_ids))
if where_clauses:
target_scope = where_clauses[0] if len(where_clauses) == 1 else or_(*where_clauses)
await session.exec(
delete(GenealogyCitation).where(
col(GenealogyCitation.source_kind) == GenealogyCitationSourceKind.FAMILYSEARCH_IMPORTED,
target_scope,
)
)
for citation in citations:
session.add(citation)
def _parse_person(element: Element) -> ParsedPerson:
birth_event = _first_child(element, "BIRT")
death_event = _first_child(element, "DEAT")
birth_date_raw = _child_value(birth_event, "DATE")
death_date_raw = _child_value(death_event, "DATE")
return ParsedPerson(
pointer=element.get_pointer() or "",
fs_id=_extract_fs_identifier(element),
full_name=_person_name(element),
birth_date=_parse_exact_date(birth_date_raw),
birth_date_raw=birth_date_raw,
birth_place=_child_value(birth_event, "PLAC"),
death_date=_parse_exact_date(death_date_raw),
death_date_raw=death_date_raw,
death_place=_child_value(death_event, "PLAC"),
citations=(
*_fact_citations(fact_element=birth_event, fact_type=GenealogyCitationFactType.BIRTH),
*_fact_citations(fact_element=death_event, fact_type=GenealogyCitationFactType.DEATH),
),
)
def _parse_family(element: Element) -> ParsedFamily:
marriage_event = _first_child(element, "MARR")
marriage_date_raw = _child_value(marriage_event, "DATE")
children = tuple(
ParsedFamilyChild(
child_pointer=(child.get_value() or "").strip(),
relationship_type=_child_value(child, "PEDI"),
)
for child in _children(element, "CHIL")
if (child.get_value() or "").strip()
)
return ParsedFamily(
fs_family_id=_extract_fs_identifier(element),
husband_pointer=(_child_value(element, "HUSB") or "").strip() or None,
wife_pointer=(_child_value(element, "WIFE") or "").strip() or None,
marriage_date=_parse_exact_date(marriage_date_raw),
marriage_date_raw=marriage_date_raw,
marriage_place=_child_value(marriage_event, "PLAC"),
children=children,
citations=_fact_citations(fact_element=marriage_event, fact_type=GenealogyCitationFactType.MARRIAGE),
)
def _fact_citations(
*, fact_element: Element | None, fact_type: GenealogyCitationFactType
) -> tuple[ParsedCitation, ...]:
if fact_element is None:
return ()
citations: list[ParsedCitation] = []
for source in _children(fact_element, "SOUR"):
raw_citation = _flatten_tag_values(source)
if raw_citation is None:
continue
citations.append(ParsedCitation(fact_type=fact_type, raw_citation_text=raw_citation))
return tuple(citations)
def _extract_fs_identifier(element: Element) -> str | None:
for tag in ("_FSFTID", "FSFTID"):
value = _child_value(element, tag)
if value:
return value
for refn in _children(element, "REFN"):
refn_value = (refn.get_value() or "").strip()
refn_type = (_child_value(refn, "TYPE") or "").strip().casefold()
if refn_value and ("fsftid" in refn_type or "familysearch" in refn_type):
return refn_value
return None
def _person_name(element: Element) -> str:
raw_name = _child_value(element, "NAME")
if raw_name is None:
return "Unknown"
cleaned = raw_name.replace("/", " ").strip()
return " ".join(part for part in cleaned.split() if part) or "Unknown"
def _parse_exact_date(raw: str | None) -> date | None:
if raw is None:
return None
tokens = [token for token in raw.strip().upper().split() if token]
if len(tokens) != 3:
return None
day_token, month_token, year_token = tokens
if month_token not in _MONTHS:
return None
try:
return date(year=int(year_token), month=_MONTHS[month_token], day=int(day_token))
except ValueError:
return None
def _first_child(element: Element | None, tag: str) -> Element | None:
if element is None:
return None
for child in element.get_child_elements():
if child.get_tag() == tag:
return child
return None
def _children(element: Element | None, tag: str) -> list[Element]:
if element is None:
return []
return [child for child in element.get_child_elements() if child.get_tag() == tag]
def _child_value(element: Element | None, tag: str) -> str | None:
child = _first_child(element, tag)
if child is None:
return None
value = (child.get_value() or "").strip()
return value or None
def _flatten_tag_values(element: Element) -> str | None:
lines: list[str] = []
def walk(node: Element) -> None:
value = (node.get_value() or "").strip()
if value:
lines.append(f"{node.get_tag()}: {value}")
for child in node.get_child_elements():
walk(child)
walk(element)
return " | ".join(lines) if lines else None
+30 -5
View File
@@ -26,6 +26,18 @@ from .base import ServiceBase
logger = logging.getLogger(__name__)
def _utc_now_naive() -> datetime:
"""Return current UTC as naive datetime for DB timestamp columns."""
return datetime.now(UTC).replace(tzinfo=None)
def _as_naive_utc(value: datetime) -> datetime:
"""Normalize datetimes to naive UTC for DB comparisons/binds."""
if value.tzinfo is None:
return value
return value.astimezone(UTC).replace(tzinfo=None)
class JobDeleteBlockedError(AppError):
"""Raised when a job delete operation is blocked by lifecycle policy."""
@@ -171,6 +183,16 @@ class JobService(ServiceBase):
await self._finalize(session=_session, caller_session=session, refresh=(job,))
return job
async def note_processing_progress(self, *, job_id: UUID, session: AsyncSession | None = None) -> Job:
"""Refresh job liveness while a multi-page batch is still in progress."""
async with self._session_scope(session) as _session:
job = await _session.get(Job, job_id)
if job is None:
raise self._not_found(job_id)
job.date_updated = _utc_now_naive()
await self._finalize(session=_session, caller_session=session, refresh=(job,))
return job
async def claim_next_queued_job(
self,
*,
@@ -201,7 +223,7 @@ class JobService(ServiceBase):
await self._finalize(session=_session, caller_session=session, refresh=(job,))
return job
now = datetime.now(UTC)
now = _utc_now_naive()
queued_job_id = (
select(col(Job.id))
.where(col(Job.status) == JobStatus.QUEUED)
@@ -239,12 +261,15 @@ class JobService(ServiceBase):
``stale_before`` are considered stale and re-queued.
"""
async with self._session_scope(session) as _session:
query = select(Job).where(Job.status == JobStatus.PROCESSING).where(Job.date_updated < stale_before)
normalized_stale_before = _as_naive_utc(stale_before)
query = (
select(Job).where(Job.status == JobStatus.PROCESSING).where(Job.date_updated < normalized_stale_before)
)
stale_jobs = (await _session.exec(query)).all()
if not stale_jobs:
return 0
now = datetime.now(UTC)
now = _utc_now_naive()
for job in stale_jobs:
job.status = JobStatus.QUEUED
job.date_updated = now
@@ -352,7 +377,7 @@ class JobService(ServiceBase):
suggestion="Use resubmit for reprocessing needs, or leave the terminal job unchanged.",
)
now = datetime.now(UTC)
now = _utc_now_naive()
job.status = JobStatus.FAILED
job.date_updated = now
@@ -398,7 +423,7 @@ class JobService(ServiceBase):
suggestion="Only failed or cancelled sources can be resubmitted.",
)
now = datetime.now(UTC)
now = _utc_now_naive()
for job_source in candidates:
job_source.status = JobSourceStatus.PENDING
+444
View File
@@ -0,0 +1,444 @@
"""Queue-backed maintenance operations executed by the worker loop."""
from __future__ import annotations
import asyncio
from dataclasses import dataclass
from datetime import UTC
from datetime import datetime
from pathlib import Path
from uuid import UUID
from uuid import uuid4
from sqlalchemy import update
from sqlalchemy.exc import SQLAlchemyError
from sqlmodel import col
from sqlmodel import func
from sqlmodel import select
from sqlmodel.ext.asyncio.session import AsyncSession
from transcription.db.models import Document
from transcription.db.models import MaintenanceJobType
from transcription.db.models import MaintenanceRun
from transcription.db.models import MaintenanceRunStatus
from transcription.db.models import Source
from transcription.errors import AppError
from transcription.errors import ErrorCategory
from transcription.errors import classify_unexpected_error
from .base import ServiceBase
from .gedcom_import import GedcomImportError
from .gedcom_import import import_gedcom_file
from .media_storage import persist_named_media
def _utc_now_naive() -> datetime:
return datetime.now(UTC).replace(tzinfo=None)
@dataclass(frozen=True, slots=True)
class MaintenanceExecution:
"""In-memory result for one executed maintenance run."""
status: MaintenanceRunStatus
summary: str
output: str
error_detail: str | None = None
class MaintenanceError(AppError):
"""Raised when maintenance operations cannot be enqueued or executed."""
class MaintenanceService(ServiceBase):
"""Persist and execute background maintenance runs."""
async def store_gedcom_upload(self, *, filename: str, file_bytes: bytes) -> str:
if not file_bytes:
raise MaintenanceError(
"GEDCOM upload is empty.",
category=ErrorCategory.VALIDATION,
suggestion="Upload a non-empty .ged file and retry.",
)
if Path(filename).suffix.casefold() != ".ged":
raise MaintenanceError(
"Unsupported GEDCOM upload format.",
category=ErrorCategory.USER_INPUT,
suggestion="Upload a file with a .ged extension.",
)
stored_path = await persist_named_media(
root=self.settings.upload_dir,
namespace=Path("genealogy"),
filename=filename,
file_bytes=file_bytes,
filename_stem=str(uuid4()),
error=MaintenanceError,
failure_message="GEDCOM file could not be persisted.",
failure_suggestion="Check upload directory permissions and retry.",
log_label="gedcom file",
)
return str(stored_path.resolve().relative_to(self.settings.upload_dir.resolve()).as_posix())
def latest_gedcom_upload_path(self) -> str | None:
latest = self._latest_gedcom_upload()
if latest is None:
return None
return str(latest.resolve().relative_to(self.settings.upload_dir.resolve()).as_posix())
async def list_runs(
self,
*,
limit: int = 100,
session: AsyncSession | None = None,
) -> list[MaintenanceRun]:
try:
async with self._session_scope(session) as _session:
query = (
select(MaintenanceRun)
.order_by(col(MaintenanceRun.created_at).desc(), col(MaintenanceRun.id).desc())
.limit(limit)
)
return list((await _session.exec(query)).all())
except SQLAlchemyError as exc:
raise MaintenanceError(
"Maintenance runs are unavailable.",
category=ErrorCategory.INFRA_PERSISTENT,
suggestion="Verify database schema access and retry.",
detail=f"Failed to list maintenance runs: {type(exc).__name__}: {exc}",
) from exc
async def enqueue_run(
self,
*,
job_type: MaintenanceJobType,
triggered_by: str = "ui.settings",
session: AsyncSession | None = None,
) -> MaintenanceRun:
run = MaintenanceRun(
job_type=job_type,
status=MaintenanceRunStatus.QUEUED,
triggered_by=triggered_by,
)
try:
async with self._session_scope(session) as _session:
_session.add(run)
await self._finalize(session=_session, caller_session=session, refresh=(run,))
except SQLAlchemyError as exc:
raise MaintenanceError(
"Maintenance run could not be queued.",
category=ErrorCategory.INFRA_PERSISTENT,
suggestion="Run the V6.1 schema migration, then retry.",
detail=f"Failed to enqueue maintenance run: {type(exc).__name__}: {exc}",
) from exc
return run
async def claim_next_queued_run(self, *, session: AsyncSession | None = None) -> MaintenanceRun | None:
try:
async with self._session_scope(session) as _session:
now = _utc_now_naive()
queued_run_id = (
select(col(MaintenanceRun.id))
.where(col(MaintenanceRun.status) == MaintenanceRunStatus.QUEUED)
.order_by(col(MaintenanceRun.created_at), col(MaintenanceRun.id))
.limit(1)
.scalar_subquery()
)
claim_statement = (
update(MaintenanceRun)
.where(col(MaintenanceRun.id) == queued_run_id)
.where(col(MaintenanceRun.status) == MaintenanceRunStatus.QUEUED)
.values(
status=MaintenanceRunStatus.PROCESSING,
started_at=now,
updated_at=now,
)
.returning(col(MaintenanceRun.id))
)
claimed_row = (await _session.exec(claim_statement)).first()
if claimed_row is None:
return None
claimed_run_id = claimed_row if isinstance(claimed_row, UUID) else claimed_row[0]
run = await _session.get(MaintenanceRun, claimed_run_id)
if run is None:
return None
await self._finalize(session=_session, caller_session=session, refresh=(run,))
return run
except SQLAlchemyError as exc:
raise MaintenanceError(
"Maintenance queue claim failed.",
category=ErrorCategory.INFRA_PERSISTENT,
suggestion="Verify database schema access and retry.",
detail=f"Failed to claim queued maintenance run: {type(exc).__name__}: {exc}",
) from exc
async def process_next_queued_run(self, *, session: AsyncSession | None = None) -> bool:
run = await self.claim_next_queued_run(session=session)
if run is None:
return False
execution = await self._execute_run(run)
await self._finalize_run(run_id=run.id, execution=execution, session=session)
return True
async def _finalize_run(
self,
*,
run_id: UUID,
execution: MaintenanceExecution,
session: AsyncSession | None = None,
) -> None:
now = _utc_now_naive()
log_path = self._write_log(run_id=run_id, output=execution.output)
async with self._session_scope(session) as _session:
run = await _session.get(MaintenanceRun, run_id)
if run is None:
raise MaintenanceError(
"Maintenance run not found while finalizing.",
category=ErrorCategory.NOT_FOUND,
suggestion="Refresh the page and retry.",
)
run.status = execution.status
run.summary = execution.summary
run.error_detail = execution.error_detail
run.log_path = log_path
run.finished_at = now
run.updated_at = now
await self._finalize(session=_session, caller_session=session, refresh=(run,))
async def _execute_run(self, run: MaintenanceRun) -> MaintenanceExecution:
if run.job_type == MaintenanceJobType.BACKUP:
return await self._execute_backup()
if run.job_type == MaintenanceJobType.STORAGE_RECONCILIATION:
return await self._execute_storage_reconciliation()
if run.job_type == MaintenanceJobType.GEDCOM_IMPORT:
return await self._execute_gedcom_import()
raise MaintenanceError(
"Unsupported maintenance job type.",
category=ErrorCategory.VALIDATION,
suggestion="Choose a supported maintenance action and retry.",
)
async def _execute_backup(self) -> MaintenanceExecution:
script_path = Path("deploy") / "backup" / "create_postgres_backup.sh"
if not script_path.is_file():
return MaintenanceExecution(
status=MaintenanceRunStatus.FAILED,
summary="Backup script is unavailable in this environment.",
output="Backup script not found.",
error_detail=f"Missing script: {script_path}",
)
command = ["sh", str(script_path)]
try:
process = await asyncio.create_subprocess_exec(
*command,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.STDOUT,
)
stdout, _ = await process.communicate()
except OSError as exc:
return MaintenanceExecution(
status=MaintenanceRunStatus.FAILED,
summary="Backup command failed to start.",
output=f"Failed to execute {' '.join(command)}",
error_detail=f"{type(exc).__name__}: {exc}",
)
output = stdout.decode("utf-8", errors="replace")
if process.returncode == 0:
return MaintenanceExecution(
status=MaintenanceRunStatus.SUCCEEDED,
summary="Backup completed successfully.",
output=output,
)
return MaintenanceExecution(
status=MaintenanceRunStatus.FAILED,
summary="Backup command failed.",
output=output,
error_detail=f"Exit code: {process.returncode}",
)
async def _execute_storage_reconciliation(self) -> MaintenanceExecution:
try:
mismatches = await self._collect_storage_mismatches()
except Exception as exc: # noqa: BLE001
error = classify_unexpected_error(exc, operation="maintenance.storage_reconciliation")
return MaintenanceExecution(
status=MaintenanceRunStatus.FAILED,
summary="Storage reconciliation failed.",
output="Storage reconciliation failed before completion.",
error_detail=error.detail,
)
if mismatches:
report = "\n".join(f"- {item}" for item in mismatches)
return MaintenanceExecution(
status=MaintenanceRunStatus.FAILED,
summary=f"Storage reconciliation found {len(mismatches)} issue(s).",
output=report,
error_detail="Reconciliation mismatches were detected.",
)
return MaintenanceExecution(
status=MaintenanceRunStatus.SUCCEEDED,
summary="Storage reconciliation found no mismatches.",
output="No storage reconciliation mismatches detected.",
)
async def _execute_gedcom_import(self) -> MaintenanceExecution:
latest_upload = self._latest_gedcom_upload()
if latest_upload is None:
return MaintenanceExecution(
status=MaintenanceRunStatus.FAILED,
summary="No GEDCOM upload is available.",
output="No .ged file found under uploads/genealogy.",
error_detail="Missing GEDCOM upload in uploads/genealogy.",
)
try:
async with self._session_scope() as session:
result = await import_gedcom_file(session=session, file_path=latest_upload)
except GedcomImportError as exc:
return MaintenanceExecution(
status=MaintenanceRunStatus.FAILED,
summary="GEDCOM import failed.",
output=f"GEDCOM import failed for {latest_upload.name}.",
error_detail=exc.detail,
)
except Exception as exc: # noqa: BLE001
error = classify_unexpected_error(exc, operation="maintenance.gedcom_import")
return MaintenanceExecution(
status=MaintenanceRunStatus.FAILED,
summary="GEDCOM import failed.",
output=f"GEDCOM import failed for {latest_upload.name}.",
error_detail=error.detail,
)
summary = (
f"Imported {result.new_people + result.updated_people} people "
f"({result.new_people} new, {result.updated_people} updated) and "
f"{result.new_families + result.updated_families} families "
f"({result.new_families} new, {result.updated_families} updated)."
)
output_lines = [
f"GEDCOM file: {latest_upload.as_posix()}",
summary,
f"Skipped people without FamilySearch ID: {result.skipped_people_without_fs_id}",
f"Skipped families without FamilySearch ID: {result.skipped_families_without_fs_id}",
f"Family child links written: {result.family_children}",
f"Imported citations: {result.citations}",
]
return MaintenanceExecution(
status=MaintenanceRunStatus.SUCCEEDED,
summary=summary,
output="\n".join(output_lines),
)
async def _collect_storage_mismatches(self) -> list[str]:
upload_root = self.settings.upload_dir
folder_ids = _document_folder_ids(upload_root)
doc_ids = await self._document_ids()
source_counts = await self._source_counts_by_document()
folder_by_normalized = {_normalize_identifier(folder_id): folder_id for folder_id in folder_ids}
doc_by_normalized = {_normalize_identifier(doc_id): doc_id for doc_id in doc_ids}
source_counts_by_normalized = {
_normalize_identifier(document_id): count for document_id, count in source_counts.items()
}
mismatches: list[str] = []
missing_in_table = sorted(set(folder_by_normalized) - set(doc_by_normalized))
for folder_key in missing_in_table:
folder_name = folder_by_normalized[folder_key]
mismatches.append(f"document-folder-without-row: documents/{folder_name}")
missing_in_folders = sorted(set(doc_by_normalized) - set(folder_by_normalized))
for doc_key in missing_in_folders:
doc_id = doc_by_normalized[doc_key]
source_count = source_counts_by_normalized.get(doc_key, 0)
mismatches.append(f"document-row-without-folder: {doc_id} (source rows: {source_count})")
for doc_key in sorted(doc_by_normalized):
doc_id = doc_by_normalized[doc_key]
folder_name = folder_by_normalized.get(doc_key)
db_count = source_counts_by_normalized.get(doc_key, 0)
file_count = _source_file_count_for_document(upload_root, folder_name) if folder_name is not None else 0
if db_count != file_count:
path_label = f"documents/{folder_name}" if folder_name is not None else "documents/<missing-folder>"
mismatches.append(
f"source-count-mismatch: {doc_id} -> source rows: {db_count}, files in {path_label}: {file_count}"
)
return mismatches
def _latest_gedcom_upload(self) -> Path | None:
genealogy_root = self.settings.upload_dir / "genealogy"
if not genealogy_root.exists():
return None
candidates = [path for path in genealogy_root.rglob("*.ged") if path.is_file()]
if not candidates:
return None
return max(candidates, key=lambda path: (path.stat().st_mtime_ns, path.name.casefold()))
async def _document_ids(self) -> set[str]:
async with self._session_scope() as session:
rows = await session.exec(select(Document.id))
return {str(item) for item in rows.all()}
async def _source_counts_by_document(self) -> dict[str, int]:
async with self._session_scope() as session:
rows = await session.exec(
select(
Source.document_id,
func.count(Source.id), # ty: ignore[invalid-argument-type] - SQLAlchemy descriptor false positive.
).group_by(
Source.document_id # ty: ignore[invalid-argument-type] - SQLAlchemy descriptor false positive.
)
)
return {str(document_id): int(count) for document_id, count in rows}
def read_log_bytes(self, *, log_path: str) -> bytes:
candidate = (self.settings.log_dir / Path(log_path)).resolve()
base = self.settings.log_dir.resolve()
try:
candidate.relative_to(base)
except ValueError as exc:
raise MaintenanceError(
"Maintenance log path is invalid.",
category=ErrorCategory.VALIDATION,
suggestion="Refresh and retry.",
detail=f"Requested path outside log root: {candidate}",
) from exc
if not candidate.is_file():
raise MaintenanceError(
"Maintenance log file is unavailable.",
category=ErrorCategory.NOT_FOUND,
suggestion="Refresh and retry.",
)
return candidate.read_bytes()
def _write_log(self, *, run_id: UUID, output: str) -> str:
timestamp = _utc_now_naive().strftime("%Y%m%d-%H%M%S")
logs_dir = self.settings.log_dir / "maintenance"
logs_dir.mkdir(parents=True, exist_ok=True)
file_path = logs_dir / f"maintenance-{run_id}-{timestamp}.log"
file_path.write_text(output, encoding="utf-8")
return str(file_path.relative_to(self.settings.log_dir).as_posix())
def _normalize_identifier(value: str) -> str:
return value.replace("-", "").strip().lower()
def _document_folder_ids(root: Path) -> set[str]:
documents_root = root / "documents"
if not documents_root.exists():
return set()
return {entry.name for entry in documents_root.iterdir() if entry.is_dir()}
def _source_file_count_for_document(root: Path, document_id: str | None) -> int:
if document_id is None:
return 0
directory = root / "documents" / document_id
if not directory.exists():
return 0
return sum(1 for entry in directory.iterdir() if entry.is_file())
+6 -7
View File
@@ -9,17 +9,14 @@ from uuid import uuid4
from ..config import Settings
from ..config import get_settings
from ..errors import AppError
from ..errors import ErrorCategory
from ..errors import exception_detail
from .errors import PromptStoreError
PROMPT_EXTENSION = ".md"
BACKUP_SUFFIX = ".bak"
class PromptStoreError(AppError):
"""Raised when prompt storage validation or persistence fails."""
@dataclass(frozen=True, slots=True)
class PromptSummary:
"""Read model for one editable prompt artifact."""
@@ -96,9 +93,10 @@ class PromptStore:
raise self._filesystem_error("Prompt directory could not be resolved", exc) from exc
if not root.is_dir():
raise PromptStoreError(
f"Prompt directory is unavailable: {root}",
"Prompt directory is unavailable.",
category=ErrorCategory.INFRA_PERSISTENT,
suggestion="Restore the configured prompt directory and its permissions.",
detail=f"Prompt directory is unavailable: {root}",
)
return root
@@ -185,7 +183,8 @@ class PromptStore:
@staticmethod
def _filesystem_error(message: str, exc: Exception) -> PromptStoreError:
return PromptStoreError(
f"{message}: {exc}",
message,
category=ErrorCategory.INFRA_PERSISTENT,
suggestion="Check prompt directory permissions and available disk space, then retry.",
detail=exception_detail(exc),
)
+41 -18
View File
@@ -38,6 +38,7 @@ from transcription.db.models import JobSourceStatus
from transcription.db.models import Source
from transcription.errors import ErrorCategory
from transcription.providers import ProviderAuthError
from transcription.providers import ProviderCallEvidence
from transcription.providers import ProviderError
from transcription.providers import ProviderResponseError
from transcription.providers import RequestManifest
@@ -47,6 +48,7 @@ from transcription.providers import TranscriptionProvider
from transcription.providers import TranscriptionResult
from transcription.providers import TransportEvidence
from transcription.providers import get_transcription_provider
from transcription.runtime_helpers import insert_with_sequence_retry
from ..db.loading import orm_attribute
from ..db.loading import selectinload
@@ -65,6 +67,18 @@ JSON_OBJECT_ADAPTER = TypeAdapter(dict[str, JsonValue])
MAX_EXECUTION_ATTEMPT_NUMBER_RETRIES = 3
def _utc_now_naive() -> datetime:
"""Return current UTC as naive datetime for DB timestamp columns."""
return datetime.now(UTC).replace(tzinfo=None)
def _as_naive_utc(value: datetime) -> datetime:
"""Normalize aware or naive datetimes to naive UTC."""
if value.tzinfo is None:
return value
return value.astimezone(UTC).replace(tzinfo=None)
class PromptExecution(BaseModel):
"""Resolved prompt inputs captured for one page execution."""
@@ -532,15 +546,15 @@ class SourceService(ServiceBase):
else:
job_source.status = outcome
finish_time = finished_at or datetime.now(UTC)
start_time = started_at or finish_time
finish_time = _as_naive_utc(finished_at) if finished_at is not None else _utc_now_naive()
start_time = _as_naive_utc(started_at) if started_at is not None else finish_time
transport = transport_evidence or TransportEvidence(response_received=False)
manifest_payload = request_manifest.model_dump(mode="json") if request_manifest is not None else None
software_payload = (
request_manifest.software.model_dump(mode="json") if request_manifest is not None else None
)
attempt: ExecutionAttempt | None = None
for attempt_retry in range(1, MAX_EXECUTION_ATTEMPT_NUMBER_RETRIES + 1):
async def _insert_execution_attempt(_attempt_retry: int) -> ExecutionAttempt:
latest_attempt_number = (
await _session.exec(
select(func.max(ExecutionAttempt.attempt_number))
@@ -582,24 +596,25 @@ class SourceService(ServiceBase):
if duration_ms is not None
else max(0, int((finish_time - start_time).total_seconds() * 1000)),
)
try:
async with _session.begin_nested():
_session.add(candidate)
await _session.flush()
attempt = candidate
break
except IntegrityError:
logger.warning(
return candidate
try:
attempt = await insert_with_sequence_retry(
max_retries=MAX_EXECUTION_ATTEMPT_NUMBER_RETRIES,
operation=_insert_execution_attempt,
on_conflict=lambda attempt_retry, _exc: logger.warning(
"Execution attempt number conflict job_id=%s source_id=%s retry=%s/%s",
job_id,
source_id,
attempt_retry,
MAX_EXECUTION_ATTEMPT_NUMBER_RETRIES,
),
)
continue
if attempt is None:
raise self._execution_attempt_conflict(job_id=job_id, source_id=source_id)
except IntegrityError as exc:
raise self._execution_attempt_conflict(job_id=job_id, source_id=source_id) from exc
if text is not None and source.raw_transcription is None and source.preferred_execution_attempt_id is None:
source.raw_transcription = text
@@ -639,7 +654,7 @@ class SourceService(ServiceBase):
source = await self._read_source(session=_session, source_id=source_id)
source.revised_text = text
source.date_revised = datetime.now(UTC)
source.date_revised = _utc_now_naive()
await self._finalize(session=_session, caller_session=session, refresh=(source,))
return source
@@ -755,6 +770,7 @@ async def transcribe_document_image(
provider: TranscriptionProvider | None = None,
source_reference: SourceEvidenceReference | None = None,
requested_model: str | None = None,
evidence_capture: ProviderCallEvidence | None = None,
) -> TranscriptionResult:
"""Transcribe a local image using the configured prompt and provider."""
runtime_settings = settings or get_settings()
@@ -788,6 +804,7 @@ async def transcribe_document_image(
top_p=prompt_execution.top_p,
source_reference=source_reference,
requested_model=requested_model,
evidence_capture=evidence_capture,
)
finally:
if owns_adapter:
@@ -839,17 +856,19 @@ def load_prompt_text(*, prompt_name: str = DEFAULT_PROMPT_FILE, settings: Settin
if not prompt_path.exists() or not prompt_path.is_file():
raise PromptLoadError(
f"Prompt file not found: {prompt_path}",
f"Prompt file not found: {prompt_path.name}",
category=ErrorCategory.INFRA_PERSISTENT,
suggestion="Verify PROMPT_DIR and prompt file configuration, then retry.",
detail=f"Prompt file missing at {prompt_path}",
)
prompt_text = prompt_path.read_text(encoding="utf-8").strip()
if not prompt_text:
raise PromptLoadError(
f"Prompt file is empty: {prompt_path}",
f"Prompt file is empty: {prompt_path.name}",
category=ErrorCategory.VALIDATION,
suggestion="Populate the prompt file with valid instructions and retry.",
detail=f"Prompt file is empty at {prompt_path}",
)
logger.info("Loaded prompt artifact: %s", prompt_path)
@@ -897,9 +916,10 @@ def load_source_payload(source_path: str | Path) -> tuple[bytes, str]:
if not path.exists() or not path.is_file():
raise TranscriptionError(
f"Source file not found: {path}",
f"Source file not found: {path.name}",
category=ErrorCategory.NOT_FOUND,
suggestion="Verify the Source file exists and retry from the jobs page.",
detail=f"Source file not found at {path}",
)
content = path.read_bytes()
@@ -916,6 +936,7 @@ def handle_transcription_errors():
"Provider authentication failed",
category=ErrorCategory.INFRA_PERSISTENT,
suggestion="Verify provider API credentials and retry.",
detail=f"{type(exc).__name__}: {exc}",
) from exc
except ProviderResponseError as exc:
raise TranscriptionError(
@@ -923,11 +944,13 @@ def handle_transcription_errors():
category=ErrorCategory.EXTERNAL_PROVIDER,
suggestion="Retry once. If it persists, try a different provider model or inspect provider status.",
retriable=True,
detail=f"{type(exc).__name__}: {exc}",
) from exc
except ProviderError as exc:
raise TranscriptionError(
f"Provider transcription failed: {exc}",
"Provider transcription failed",
category=ErrorCategory.EXTERNAL_PROVIDER,
suggestion="Retry the transcription from jobs. If repeated, check provider availability.",
retriable=True,
detail=f"{type(exc).__name__}: {exc}",
) from exc
+2 -2
View File
@@ -1,6 +1,5 @@
from __future__ import annotations
import asyncio
import hashlib
import logging
from collections.abc import Sequence
@@ -16,6 +15,7 @@ from transcription.config import Settings
from transcription.config import get_settings
from transcription.errors import AppError
from transcription.errors import ErrorCategory
from transcription.runtime_helpers import run_blocking
from ..db.models import Document
from ..db.models import Job
@@ -399,7 +399,7 @@ async def store_source_file(
)
return StoredSourceFile(
path=stored_path,
file_hash=await asyncio.to_thread(_sha256_hexdigest, file_bytes),
file_hash=await run_blocking(_sha256_hexdigest, file_bytes),
file_size_bytes=len(file_bytes),
)
@@ -0,0 +1,24 @@
from __future__ import annotations
from collections.abc import AsyncIterator
from contextlib import asynccontextmanager
from sqlmodel.ext.asyncio.session import AsyncSession
from ..db.session import transaction_scope
from . import ServiceBundle
@asynccontextmanager
async def unit_of_work(
*,
services: ServiceBundle,
session: AsyncSession | None = None,
) -> AsyncIterator[AsyncSession]:
"""Yield one shared transactional session for orchestration paths."""
if session is not None:
yield session
return
async with transaction_scope(session_factory=services.jobs.session_factory) as local_session:
yield local_session
+19 -8
View File
@@ -21,6 +21,7 @@ from ..errors import AppError
from ..errors import ErrorCategory
from ..errors import classify_unexpected_error
from ..errors import format_error_detail
from ..providers import ProviderCallEvidence
from ..providers import ProviderError
from ..providers import RequestManifest
from ..providers import SourceEvidenceReference
@@ -37,6 +38,7 @@ from .sources import build_prompt_execution
from .sources import build_provider_input
from .sources import hash_prompt_text
from .sources import transcribe_document_image
from .unit_of_work import unit_of_work
logger = logging.getLogger(__name__)
@@ -47,6 +49,11 @@ _RETRIABLE_FAILED_JOB_ERROR_CATEGORIES = {
}
def _utc_now_naive() -> datetime:
"""Return current UTC as naive datetime for DB timestamp columns."""
return datetime.now(UTC).replace(tzinfo=None)
async def create_document_with_people(
*,
document: Document,
@@ -271,13 +278,14 @@ async def process_queued_job( # noqa: PLR0915
externally_stopped = True
break
started_at = datetime.now(UTC)
started_at = _utc_now_naive()
# Fallback start for failures raised before the provider call; reset to the
# true call boundary immediately before the wait_for below.
monotonic_started_at = asyncio.get_running_loop().time()
result: TranscriptionResult | None = None
provider_input = None
page_outcome: _SuccessfulPage | _FailedPage
provider_call_evidence = ProviderCallEvidence()
try:
provider_input = build_provider_input(source, upload_dir=runtime_settings.upload_dir)
source_reference = SourceEvidenceReference(
@@ -302,6 +310,7 @@ async def process_queued_job( # noqa: PLR0915
provider=provider,
source_reference=source_reference,
requested_model=source_job.model,
evidence_capture=provider_call_evidence,
),
timeout=runtime_settings.worker_provider_timeout_seconds,
)
@@ -326,7 +335,7 @@ async def process_queued_job( # noqa: PLR0915
)
_validate_transcription_quality(result=result, settings=runtime_settings)
finished_at = datetime.now(UTC)
finished_at = _utc_now_naive()
page_outcome = _SuccessfulPage(
source=source,
result=result,
@@ -346,7 +355,7 @@ async def process_queued_job( # noqa: PLR0915
suggestion="Retry the job. If this repeats, verify provider latency and request payload size.",
retriable=True,
)
finished_at = datetime.now(UTC)
finished_at = _utc_now_naive()
page_outcome = _FailedPage(
source=source,
error=error,
@@ -360,8 +369,8 @@ async def process_queued_job( # noqa: PLR0915
_duration_ms_between(started_at, finished_at),
max(0, int((asyncio.get_running_loop().time() - monotonic_started_at) * 1000)),
),
request_manifest=provider.current_request_manifest,
transport_evidence=provider.current_transport_evidence,
request_manifest=provider_call_evidence.request_manifest,
transport_evidence=provider_call_evidence.transport_evidence,
failure_phase="local_timeout",
)
failed_pages.append(page_outcome)
@@ -381,7 +390,7 @@ async def process_queued_job( # noqa: PLR0915
case _:
error = classify_unexpected_error(exc, operation="worker.process_job")
finished_at = datetime.now(UTC)
finished_at = _utc_now_naive()
provider_error = _find_provider_error(exc)
page_outcome = _FailedPage(
source=source,
@@ -627,7 +636,7 @@ async def _finalize_batch_outcome(
status change (no sources, or the batch stopped before the last page).
"""
if session is None:
async with services.jobs._session_scope() as local_session:
async with unit_of_work(services=services, session=session) as local_session:
if final_page is not None:
await _write_page_outcome(job=job, services=services, page=final_page, session=local_session)
updated_job = await services.jobs.mark_job_status(job.id, status, session=local_session)
@@ -665,12 +674,14 @@ async def _persist_page_outcome(
session: AsyncSession | None,
) -> None:
if session is None:
async with services.sources._session_scope() as local_session:
async with unit_of_work(services=services, session=session) as local_session:
await _write_page_outcome(job=job, services=services, page=page, session=local_session)
await services.jobs.note_processing_progress(job_id=job.id, session=local_session)
await local_session.commit()
return
await _write_page_outcome(job=job, services=services, page=page, session=session)
await services.jobs.note_processing_progress(job_id=job.id, session=session)
await session.commit()
-2
View File
@@ -13,7 +13,6 @@ from transcription.ui.pages.people_page import register_page as register_people_
from transcription.ui.pages.print_preview_page import register_page as register_print_preview_page
from transcription.ui.pages.settings_page import register_page as register_settings_page
from transcription.ui.pages.sources_page import register_page as register_sources_page
from transcription.ui.pages.tags_page import register_page as register_tags_page
from transcription.ui.resources import read_css
from transcription.ui.theme import apply_archival_theme
@@ -37,7 +36,6 @@ def register_pages(app: FastAPI) -> None:
"""Register all NiceGUI pages and mount them onto the FastAPI app."""
register_home_page()
register_documents_page()
register_tags_page()
register_people_page()
register_print_preview_page()
register_sources_page()
@@ -8,9 +8,7 @@ from transcription.ui.resources import read_svg
NAV_ITEMS: tuple[tuple[str, str, str], ...] = (
("Documents", "/documents", "description"),
("Tags", "/tags", "sell"),
("People", "/people", "group"),
("Sources", "/sources", "folder"),
("Jobs", "/jobs", "work_history"),
("Settings", "/settings", "settings"),
)
@@ -23,10 +21,6 @@ def _is_active_path(*, current_path: str, item_path: str) -> bool:
return current_path == "/documents" or current_path.startswith("/documents/")
if item_path == "/people":
return current_path == "/people" or current_path.startswith("/people/")
if item_path == "/tags":
return current_path == "/tags" or current_path.startswith("/tags/")
if item_path == "/sources":
return current_path == "/sources" or current_path.startswith("/sources/")
if item_path == "/settings":
return current_path == "/settings" or current_path.startswith("/settings/")
return current_path == item_path
@@ -2,20 +2,28 @@
from __future__ import annotations
import re
from collections.abc import Awaitable
from collections.abc import Callable
from dataclasses import dataclass
from pathlib import PurePosixPath
from pathlib import PureWindowsPath
from typing import TypeVar
from nicegui import ui
from transcription.errors import AppError
from transcription.errors import ErrorCategory
from transcription.errors import canonical_error_category
from transcription.errors import classify_unexpected_error
T = TypeVar("T")
_QUOTED_ABSOLUTE_PATH_RE = re.compile(
r"""(?P<quote>['"])(?P<path>(?:[A-Za-z]:[\\/][^'"]+|/(?!uploads/)[^'"]+))(?P=quote)"""
)
_UNQUOTED_WINDOWS_PATH_RE = re.compile(r"""(?P<path>[A-Za-z]:[\\/][^\s|]+)""")
_UNQUOTED_POSIX_PATH_RE = re.compile(r"""(?P<path>(?<![A-Za-z0-9:])/(?!uploads/)[^\s|]+)""")
@dataclass(frozen=True, slots=True)
class UiActionOutcome[T]:
@@ -67,14 +75,35 @@ def show_error(exc: Exception, *, title: str, operation: str) -> None:
ui.label(f"Category: {display_error_category(error)}").classes("text-caption")
def display_failure_detail(error_detail: str | None) -> str | None:
"""Render persisted failure detail without machine-local paths."""
candidate = (error_detail or "").strip()
if not candidate:
return None
sanitized = _QUOTED_ABSOLUTE_PATH_RE.sub(_replace_quoted_absolute_path, candidate)
sanitized = _UNQUOTED_WINDOWS_PATH_RE.sub(_replace_unquoted_absolute_path, sanitized)
sanitized = _UNQUOTED_POSIX_PATH_RE.sub(_replace_unquoted_absolute_path, sanitized)
return sanitized
def display_error_category(error: AppError) -> str:
"""Return the canonical UI-facing category label for an AppError."""
return canonical_error_category(error)
def summarize_error(exc: Exception, *, operation: str) -> str:
"""Return short one-line summary for status labels."""
error = to_app_error(exc, operation=operation)
if error.category == ErrorCategory.INTERNAL_UNEXPECTED:
return f"Unexpected error (ref: {error.error_id})"
return f"{error.message} (ref: {error.error_id})"
def _replace_quoted_absolute_path(match: re.Match[str]) -> str:
quote = match.group("quote")
return f"{quote}{_basename_for_absolute_path(match.group('path'))}{quote}"
def _replace_unquoted_absolute_path(match: re.Match[str]) -> str:
return _basename_for_absolute_path(match.group("path"))
def _basename_for_absolute_path(path: str) -> str:
if path.startswith("/uploads/"):
return path
if re.match(r"^[A-Za-z]:[\\/]", path):
return PureWindowsPath(path).name or "unknown"
return PurePosixPath(path).name or "unknown"
@@ -25,6 +25,7 @@ class DocumentTableRow:
tags: str
document_date: str
source_count: int
transcription_status: str | None = None
def _serialize_rows(rows: Sequence[DocumentTableRow]) -> list[dict[str, Any]]:
@@ -37,6 +38,7 @@ def _serialize_rows(rows: Sequence[DocumentTableRow]) -> list[dict[str, Any]]:
"tags": row.tags or "Not tagged",
"document_date": row.document_date,
"source_count": row.source_count,
"transcription_status": (row.transcription_status or "").lower(),
}
for row in rows
]
@@ -106,10 +108,19 @@ def render_documents_table(rows: Sequence[DocumentTableRow]) -> None:
"align": "center",
"style": "width: 10%;",
},
{
"name": "transcription_status",
"label": "Transcription Status",
"field": "transcription_status",
"sortable": True,
"classes": "font-mono",
"align": "center",
"style": "width: 15%;",
},
],
default_sort_by="name",
search_placeholder="Search documents by title, type, or author...",
on_row_click_id=lambda doc_id: ui.navigate.to(f"/documents/{doc_id}"),
on_row_click_id=lambda doc_id: ui.navigate.to(f"/documents/{doc_id}?from=documents"),
)
# Render document type using a subtle Quasar badge
@@ -128,3 +139,20 @@ def render_documents_table(rows: Sequence[DocumentTableRow]) -> None:
</q-td>
""",
)
table.add_slot(
"body-cell-transcription_status",
r"""
<q-td :props="props">
<span v-if="!props.value">-</span>
<q-chip
v-else
dense
square
size="sm"
:class="`ui-status ui-status--${props.value}`"
>
{{ props.value.toUpperCase() }}
</q-chip>
</q-td>
""",
)
@@ -57,13 +57,18 @@ def _serialize_rows(rows: Sequence[JobTableRow]) -> list[dict[str, Any]]:
]
def render_jobs_table(rows: Sequence[JobTableRow]) -> None:
def render_jobs_table(rows: Sequence[JobTableRow], *, document_context_id: str | None = None) -> None:
"""Render jobs table with search filtering and custom status chips."""
if not rows:
with archival_card(extra_classes="p-8 text-center"):
render_empty_state("No job records found in repository.")
return
def detail_target(job_id: str) -> str:
if document_context_id is not None:
return f"/jobs/{job_id}?from=document&document_id={document_context_id}"
return f"/jobs/{job_id}?from=jobs"
table = build_table(
rows=_serialize_rows(rows),
columns=[
@@ -117,7 +122,7 @@ def render_jobs_table(rows: Sequence[JobTableRow]) -> None:
default_sort_by="updated_sort",
default_descending=True,
search_placeholder="Search jobs by ID, document, or status...",
on_row_click_id=lambda job_id: ui.navigate.to(f"/jobs/{job_id}"),
on_row_click_id=lambda job_id: ui.navigate.to(detail_target(job_id)),
)
# Render job execution status using themed Quasar chips
@@ -94,7 +94,7 @@ def render_people_table(rows: Sequence[PersonTableRow]) -> None:
],
default_sort_by="name",
search_placeholder="Search people by name, tags, FamilySearch ID, or dates...",
on_row_click_id=lambda person_id: ui.navigate.to(f"/people/{person_id}"),
on_row_click_id=lambda person_id: ui.navigate.to(f"/people/{person_id}?from=people"),
)
# Custom column template adding an archival entity icon next to person's name
@@ -53,9 +53,9 @@ def render_sources_table(rows: Sequence[SourceTableRow]) -> None:
rows=_serialize_rows(rows),
columns=[
{
"name": "document_name",
"label": "Document Name",
"field": "document_name",
"name": "upload_name",
"label": "Upload Title",
"field": "upload_name",
"sortable": True,
"classes": "font-serif text-left ui-table-cell-wrap",
"align": "left",
@@ -69,9 +69,9 @@ def render_sources_table(rows: Sequence[SourceTableRow]) -> None:
"style": "width: 10%;",
},
{
"name": "upload_name",
"label": "Upload Title",
"field": "upload_name",
"name": "document_name",
"label": "Document Name",
"field": "document_name",
"sortable": True,
"classes": "font-serif text-left ui-table-cell-wrap",
"align": "left",
@@ -12,6 +12,7 @@ from nicegui import ui
from transcription.services.source_media import SOURCE_EXTENSIONS
SOURCE_UPLOAD_EXTENSIONS: tuple[str, ...] = tuple(sorted(SOURCE_EXTENSIONS))
GEDCOM_UPLOAD_EXTENSIONS: tuple[str, ...] = (".ged",)
IMAGE_UPLOAD_EXTENSIONS: tuple[str, ...] = (
".bmp",
".gif",
+288 -22
View File
@@ -13,12 +13,14 @@ from nicegui import ui
from transcription.config import Settings
from transcription.db.models import Document
from transcription.db.models import Source
from transcription.db.registries import AUTHOR_ROLE_SEMANTIC_KEY
from transcription.errors import ErrorCategory
from transcription.services.documents import DocumentDeleteBlockedError
from transcription.services.documents import DocumentError
from transcription.services.documents import DocumentService
from transcription.services.people import PeopleService
from transcription.services.sources import SourceService
from transcription.services.workflows import create_document_with_people
from transcription.services.workflows import update_document_with_people
from transcription.ui.components.app_shell import render_navigation_header
@@ -27,10 +29,13 @@ from transcription.ui.components.confirm_delete import dependency_summary
from transcription.ui.components.confirm_delete import render_delete_actions
from transcription.ui.components.confirm_delete import render_delete_blocked_notice
from transcription.ui.components.data_display import archival_badge
from transcription.ui.components.data_display import metadata_link_row
from transcription.ui.components.data_display import metadata_row
from transcription.ui.components.document_panzoom import render_document_panzoom
from transcription.ui.components.error_presenter import run_ui_action
from transcription.ui.components.error_presenter import show_error
from transcription.ui.components.formatters import compact_date
from transcription.ui.components.formatters import google_maps_search_url
from transcription.ui.components.formatters import parse_iso_date
from transcription.ui.components.formatters import parse_uuid
from transcription.ui.components.guards import parsed_record_id
@@ -43,7 +48,6 @@ from transcription.ui.components.primitives import render_empty_state
from transcription.ui.components.primitives import section_header_row
from transcription.ui.components.table.documents import DocumentTableRow
from transcription.ui.components.table.documents import render_documents_table
from transcription.ui.components.viewers import dark_room_viewer
from transcription.ui.runtime import resolve_runtime_settings
from transcription.ui.theme import page_header
@@ -199,6 +203,7 @@ def register_page() -> None: # noqa: PLR0915
document_date=compact_date(doc.document_date, doc.document_date_raw),
document_type=(doc.document_type_ref.label if doc.document_type_ref is not None else ""),
source_count=len(doc.sources),
transcription_status=_latest_job_status(doc),
)
for doc in documents
]
@@ -207,8 +212,22 @@ def register_page() -> None: # noqa: PLR0915
@ui.page("/documents/{document_id}")
async def document_detail_page(request: Request, document_id: str, session_factory: SessionFactoryDep) -> None:
document_service = DocumentService(session_factory=session_factory)
sources_service = SourceService(session_factory=session_factory)
render_navigation_header(current_path="/documents")
settings = resolve_runtime_settings(request)
back_label = "Back to Documents"
back_target = "/documents"
from_context = request.query_params.get("from")
if from_context == "person":
person_id = parse_uuid(request.query_params.get("person_id"))
if person_id is not None:
back_label = "Back to Person"
back_target = f"/people/{person_id}"
elif from_context == "job":
job_id = parse_uuid(request.query_params.get("job_id"))
if job_id is not None:
back_label = "Back to Job"
back_target = f"/jobs/{job_id}"
parsed_doc_id = parsed_record_id(document_id, noun="Document")
if parsed_doc_id is None:
@@ -223,12 +242,16 @@ def register_page() -> None: # noqa: PLR0915
show_error(exc, title="Load failed", operation="documents.read")
return
active_source = _resolve_active_source(document, parse_uuid(request.query_params.get("source_id")))
with ui.column().classes("w-full max-w-[1800px] mx-auto p-4 gap-4"):
type_display = document.document_type_ref.label if document.document_type_ref is not None else "Unspecified"
first_source = _resolve_active_source(document, None)
with section_header_row():
page_header(document.name, subtitle=f"Type: {type_display} | ID: {document.id}")
with ui.row().classes("items-center gap-2"):
ui.button(back_label, on_click=lambda: ui.navigate.to(back_target), icon="arrow_back").props("flat")
ui.button(
"Print",
on_click=lambda: ui.navigate.to(f"/documents/{document.id}/print"),
@@ -239,6 +262,24 @@ def register_page() -> None: # noqa: PLR0915
on_click=lambda: ui.navigate.to(f"/documents/{document.id}/edit"),
icon="edit",
).classes("ui-btn-primary text-xs")
ui.button(
"Document Details",
on_click=lambda: ui.navigate.to(f"/documents/{document.id}/info"),
icon="info",
).props("flat").classes("text-xs")
ui.button(
"View Source Detail",
on_click=(
(
lambda: ui.navigate.to(
f"/sources/{first_source.id}?from=document&document_id={document.id}"
)
)
if first_source is not None
else (lambda: ui.notify("No source pages are linked yet.", type="warning"))
),
icon="description",
).props("flat").classes("text-xs")
destructive_button(
"Delete",
on_click=lambda: ui.navigate.to(f"/documents/{document.id}/delete"),
@@ -247,19 +288,105 @@ def register_page() -> None: # noqa: PLR0915
)
with ui.grid().classes("w-full grid-cols-12 gap-4"):
_render_bento_viewer_zone(document, base_url=str(request.base_url), settings=settings)
_render_bento_metadata_zone(document)
_render_document_detail_viewer_zone(
document=document,
active_source=active_source,
base_url=str(request.base_url),
settings=settings,
)
_render_document_detail_revision_zone(
source=active_source,
sources_service=sources_service,
)
_render_bento_relations_zone(document)
@ui.page("/documents/{document_id}/info")
async def document_info_page(document_id: str, session_factory: SessionFactoryDep) -> None:
document_service = DocumentService(session_factory=session_factory)
render_navigation_header(current_path="/documents")
parsed_doc_id = parsed_record_id(document_id, noun="Document")
if parsed_doc_id is None:
return
try:
document = await document_service.read_document_detail(document_id=parsed_doc_id)
except DocumentError:
render_record_not_found("Document")
return
except Exception as exc: # noqa: BLE001
show_error(exc, title="Load failed", operation="documents.info.read")
return
with ui.column().classes("w-full max-w-6xl mx-auto p-4 gap-4"):
with section_header_row():
page_header("Document Info", subtitle=f"{document.name} ({document.id})")
ui.button(
"Back to Document",
on_click=lambda: ui.navigate.to(f"/documents/{document.id}"),
icon="arrow_back",
).props("flat")
_render_bento_metadata_zone(document)
@ui.page("/documents/{document_id}/jobs")
async def document_jobs_page(document_id: str, session_factory: SessionFactoryDep) -> RedirectResponse:
_ = session_factory
return RedirectResponse(url=f"/ui/jobs?document_id={document_id}")
@ui.page("/documents/{document_id}/sources")
async def document_sources_page(document_id: str, session_factory: SessionFactoryDep) -> RedirectResponse:
_ = session_factory
return RedirectResponse(url=f"/ui/sources?document_id={document_id}")
async def document_sources_page(request: Request, document_id: str, session_factory: SessionFactoryDep) -> None:
document_service = DocumentService(session_factory=session_factory)
render_navigation_header(current_path="/documents")
settings = resolve_runtime_settings(request)
parsed_doc_id = parsed_record_id(document_id, noun="Document")
if parsed_doc_id is None:
return
try:
document = await document_service.read_document_detail(document_id=parsed_doc_id)
except DocumentError:
render_record_not_found("Document")
return
except Exception as exc: # noqa: BLE001
show_error(exc, title="Load failed", operation="documents.sources.read")
return
ordered_sources = _sorted_document_sources(document)
with ui.column().classes("w-full max-w-7xl mx-auto p-4 gap-4"):
with section_header_row():
page_header("Source Images", subtitle=f"{document.name} ({len(ordered_sources)} pages)")
ui.button(
"Back to Document",
on_click=lambda: ui.navigate.to(f"/documents/{document.id}"),
icon="arrow_back",
).props("flat")
if not ordered_sources:
render_empty_state("No source pages are linked yet.")
return
with ui.grid().classes("w-full grid-cols-2 md:grid-cols-3 xl:grid-cols-4 gap-3"):
for source in ordered_sources:
source_url = resolve_media_url(
source.file_path,
upload_dir=settings.upload_dir,
base_url=str(request.base_url),
)
with archival_card(extra_classes="gap-2"):
if source_url is None:
render_empty_state("Image unavailable.", extra_classes="text-xs")
else:
ui.image(source_url).classes("w-full aspect-[3/4] object-contain rounded-sm bg-black/5")
ui.label(f"Page {source.page_number}").classes("text-xs font-semibold")
ui.label(source.filename).classes("text-[11px] ui-text-muted break-all")
ui.button(
"Open Source Detail",
on_click=lambda _=None, source_id=source.id: ui.navigate.to(
f"/sources/{source_id}?from=document&document_id={document.id}"
),
icon="description",
).props("flat dense").classes("text-xs self-start")
@ui.page("/documents/{document_id}/edit")
async def document_edit_page(document_id: str, session_factory: SessionFactoryDep) -> None:
@@ -523,17 +650,134 @@ def _render_document_form_fields(
)
def _render_bento_viewer_zone(document: Document, *, base_url: str, settings: Settings) -> None:
with ui.column().classes("col-span-12 lg:col-span-4"):
source_path = document.sources[0].file_path if document.sources else None
source_url = resolve_media_url(source_path, upload_dir=settings.upload_dir, base_url=base_url)
dark_room_viewer(source_url, count_label=f"{len(document.sources)} Source(s) Linked")
def _resolve_active_source(document: Document, requested_source_id: UUID | None) -> Source | None:
ordered = _sorted_document_sources(document)
if not ordered:
return None
if requested_source_id is None:
return ordered[0]
for source in ordered:
if source.id == requested_source_id:
return source
return ordered[0]
def _sorted_document_sources(document: Document) -> list[Source]:
return sorted(
document.sources,
key=lambda source: (source.page_number, source.upload_name.casefold(), source.filename.casefold()),
)
def _render_document_detail_viewer_zone(
*,
document: Document,
active_source: Source | None,
base_url: str,
settings: Settings,
) -> None:
with ui.column().classes("col-span-12 lg:col-span-4 gap-2"):
_render_document_source_navigation(document=document, active_source=active_source)
if active_source is None:
render_document_panzoom(media_url=None, filename="No source pages", count_label="0 Source Pages")
return
source_url = resolve_media_url(active_source.file_path, upload_dir=settings.upload_dir, base_url=base_url)
render_document_panzoom(
media_url=source_url,
filename=active_source.filename,
count_label=f"Page {active_source.page_number}",
)
def _render_document_source_navigation(*, document: Document, active_source: Source | None) -> None:
ordered = sorted(document.sources, key=lambda source: (source.page_number, source.id))
if not ordered or active_source is None:
with ui.row().classes("w-full justify-between items-center"):
ui.button("Previous Page", icon="chevron_left").props("flat dense disable")
ui.button("Next Page", icon="chevron_right").props("flat dense icon-right disable")
return
active_index = next((index for index, source in enumerate(ordered) if source.id == active_source.id), 0)
previous_source = ordered[active_index - 1] if active_index > 0 else None
next_source = ordered[active_index + 1] if active_index < len(ordered) - 1 else None
previous_target = f"/documents/{document.id}?source_id={previous_source.id}" if previous_source is not None else "#"
next_target = f"/documents/{document.id}?source_id={next_source.id}" if next_source is not None else "#"
with ui.row().classes("w-full justify-between items-center"):
previous = ui.button(
"Previous Page",
on_click=lambda: ui.navigate.to(previous_target),
icon="chevron_left",
).props("flat dense")
if previous_source is None:
previous.props("disable")
following = ui.button(
"Next Page",
on_click=lambda: ui.navigate.to(next_target),
icon="chevron_right",
).props("flat dense icon-right")
if next_source is None:
following.props("disable")
def _render_document_detail_revision_zone(*, source: Source | None, sources_service: SourceService) -> None:
with ui.column().classes("col-span-12 lg:col-span-6 gap-4"), archival_card(title="Editable Revision"):
if source is None:
render_empty_state("No source pages are linked yet.", italic=True)
return
seed_revision = source.revised_text if source.revised_text is not None else (source.raw_transcription or "")
revision_input = (
ui.textarea(
label="Revised transcription",
value=seed_revision,
)
.props("outlined autogrow")
.classes("w-full ui-form-surface")
)
save_state = ui.label(
f"Last saved: {source.date_revised.isoformat()}"
if source.date_revised is not None
else "No revision saved yet."
).classes("text-xs ui-text-muted")
async def submit_revision() -> None:
revised_text = (revision_input.value or "").strip()
if not revised_text:
ui.notify("Revised transcription cannot be empty.", type="warning")
return
save_outcome = await run_ui_action(
operation="documents.revision.save",
title="Save failed",
action=lambda: sources_service.upsert_revision_for_source(source_id=source.id, text=revised_text),
)
if not save_outcome.ok or save_outcome.value is None:
return
updated = save_outcome.value
source.revised_text = updated.revised_text
source.date_revised = updated.date_revised
save_state.text = (
f"Last saved: {updated.date_revised.isoformat()}"
if updated.date_revised is not None
else "Revision saved."
)
ui.notify("Revision saved", type="positive")
with ui.row().classes("w-full items-center gap-2 mt-2"):
ui.button("Save revision", on_click=submit_revision, icon="save").classes("ui-btn-primary")
ui.button(
"Reset",
on_click=lambda: _reset_document_revision_text(revision_input, source),
icon="refresh",
).props("flat")
def _first_source_path(document: Document) -> str | None:
first_source = _resolve_active_source(document, None)
return first_source.file_path if first_source is not None else None
def _render_bento_metadata_zone(document: Document) -> None:
author_names = _author_names(document)
with ui.column().classes("col-span-12 lg:col-span-4 gap-4"):
with ui.column().classes("w-full gap-4"):
with archival_card(title="Archival Metadata"):
metadata_row("Author(s):", ", ".join(author_names) if author_names else "Not set")
metadata_row(
@@ -546,7 +790,14 @@ def _render_bento_metadata_zone(document: Document) -> None:
)
metadata_row("Tags:", ", ".join(tags) if tags else "Not set")
metadata_row("Document Date:", _detail_document_date(document.document_date, document.document_date_raw))
metadata_row("Location Created:", document.location_created or "Not set")
if document.location_created:
metadata_link_row(
"Location Created:",
document.location_created,
google_maps_search_url(document.location_created),
)
else:
metadata_row("Location Created:", "Not set")
metadata_row("Archive Identifier:", document.archive_identifier or "Not set")
with ui.column().classes("w-full mt-2"):
@@ -565,7 +816,7 @@ def _detail_document_date(exact: date | None, approximate: str | None) -> str:
def _render_bento_relations_zone(document: Document) -> None:
with ui.column().classes("col-span-12 lg:col-span-4 gap-4"):
with ui.column().classes("col-span-12 lg:col-span-2 gap-4"):
_render_related_people_card(document)
_render_document_processing_card(document)
@@ -584,23 +835,27 @@ def _render_related_people_card(document: Document) -> None:
for person in grouped[role_label]:
ui.button(
person.full_name,
on_click=lambda _=None, person_id=person.id: ui.navigate.to(f"/people/{person_id}"),
on_click=lambda _=None, person_id=person.id: ui.navigate.to(
f"/people/{person_id}?from=document&document_id={document.id}"
),
icon="person",
).props("flat dense no-caps").classes("self-start text-xs font-semibold ui-link-primary")
def _render_document_processing_card(document: Document) -> None:
with archival_card(title="Sources & Pipeline Jobs"):
metadata_row("Sources:", str(len(document.sources)))
metadata_row("Jobs:", str(len(document.jobs)))
with archival_card(title="Source Pages & Transcriptions"):
metadata_row("Source pages:", str(len(document.sources)))
metadata_row("Transcription Jobs:", str(len(document.jobs)))
with ui.row().classes("w-full gap-2 mt-2 flex-wrap"):
ui.button(
"View Sources",
on_click=lambda: ui.navigate.to(f"/sources?document_id={document.id}"),
icon="description",
"View Source Images",
on_click=lambda: ui.navigate.to(f"/documents/{document.id}/sources"),
icon="photo_library",
).props("flat dense text-xs").classes("ui-link-primary")
ui.button(
"View Jobs", on_click=lambda: ui.navigate.to(f"/documents/{document.id}/jobs"), icon="work_history"
"View Transcription Jobs",
on_click=lambda: ui.navigate.to(f"/documents/{document.id}/jobs"),
icon="work_history",
).props("flat dense text-xs").classes("ui-link-primary")
ui.button(
"+ Add Job", on_click=lambda: ui.navigate.to(f"/jobs/new?document_id={document.id}"), icon="add"
@@ -670,3 +925,14 @@ def _resolve_selected_tag_labels(value: object) -> list[str]:
labels = [candidate.strip() for candidate in flatten(value) if candidate and candidate.strip()]
return list(dict.fromkeys(labels))
def _latest_job_status(document: Document) -> str | None:
if not document.jobs:
return None
latest = max(document.jobs, key=lambda job: (job.date_created, str(job.id)))
return latest.status.value
def _reset_document_revision_text(revision_input: ui.textarea, source: Source) -> None:
revision_input.value = source.revised_text if source.revised_text is not None else (source.raw_transcription or "")
+4 -4
View File
@@ -2,7 +2,6 @@
from __future__ import annotations
import asyncio
from collections.abc import Callable
from fastapi import Request
@@ -11,6 +10,7 @@ from nicegui import ui
from transcription.config import Settings
from transcription.db.models import Photo
from transcription.runtime_helpers import run_blocking
from transcription.services.photos import PhotoError
from transcription.services.photos import PhotosService
from transcription.ui.components.app_shell import render_navigation_header
@@ -184,7 +184,7 @@ def register_page() -> None: # noqa: PLR0915
render_home_content()
homepage_markdown[0] = await asyncio.to_thread(read_homepage_markdown, settings)
homepage_markdown[0] = await run_blocking(read_homepage_markdown, settings)
render_home_content.refresh()
@ui.page("/homepage/edit", title="Edit Home Page")
@@ -278,7 +278,7 @@ def register_page() -> None: # noqa: PLR0915
ui.navigate.to("/homepage/edit")
async def save_homepage() -> None:
await asyncio.to_thread(
await run_blocking(
save_homepage_markdown,
(markdown_input[0].value if markdown_input[0] is not None else "") or "",
settings,
@@ -300,6 +300,6 @@ def register_page() -> None: # noqa: PLR0915
initial_markdown="",
)
loaded_markdown = await asyncio.to_thread(read_homepage_markdown, settings)
loaded_markdown = await run_blocking(read_homepage_markdown, settings)
if markdown_input[0] is not None:
markdown_input[0].value = loaded_markdown
+21 -6
View File
@@ -68,6 +68,12 @@ def register_page() -> None: # noqa: PLR0915
"ui-btn-primary"
)
ui.button("Refresh", on_click=lambda: render_table.refresh(), icon="refresh").props("flat")
elif parsed_document_id is not None:
ui.button(
"Back to Document",
on_click=lambda: ui.navigate.to(f"/documents/{parsed_document_id}"),
icon="arrow_back",
).props("flat")
@ui.refreshable
async def render_table() -> None:
@@ -86,7 +92,7 @@ def register_page() -> None: # noqa: PLR0915
)
for job in jobs
]
render_jobs_table(rows)
render_jobs_table(rows, document_context_id=str(parsed_document_id) if parsed_document_id else None)
await render_table()
@@ -241,9 +247,17 @@ def register_page() -> None: # noqa: PLR0915
ui.button("Back to Jobs", on_click=lambda: ui.navigate.to("/jobs"), icon="arrow_back").props("flat")
@ui.page("/jobs/{job_id}")
async def job_detail_page(job_id: str, session_factory: SessionFactoryDep) -> None:
async def job_detail_page(job_id: str, request: Request, session_factory: SessionFactoryDep) -> None:
jobs_service = JobService(session_factory=session_factory)
render_navigation_header(current_path="/jobs")
back_label = "Back to Jobs"
back_target = "/jobs"
from_context = request.query_params.get("from")
if from_context == "document":
document_id = parse_uuid(request.query_params.get("document_id"))
if document_id is not None:
back_label = "Back to Document"
back_target = f"/documents/{document_id}"
parsed_job_id = parsed_record_id(job_id, noun="Job")
if parsed_job_id is None:
@@ -262,7 +276,7 @@ def register_page() -> None: # noqa: PLR0915
@ui.refreshable
def render_detail() -> None:
active_job = current_job[0]
_render_job_detail_header(active_job)
_render_job_detail_header(active_job, back_label=back_label, back_target=back_target)
with ui.grid().classes("w-full grid-cols-1 md:grid-cols-2 gap-4"):
_render_job_logistics(active_job)
_render_job_document_links(active_job)
@@ -540,10 +554,11 @@ def _render_upload_section(uploaded_files: list[tuple[str, bytes]]) -> None:
render_upload_list()
def _render_job_detail_header(job: Job) -> None:
def _render_job_detail_header(job: Job, *, back_label: str, back_target: str) -> None:
with section_header_row(classes="justify-between items-center"):
page_header(f"Job Record: {job.id}")
with ui.row().classes("items-center gap-2"):
ui.button(back_label, on_click=lambda: ui.navigate.to(back_target), icon="arrow_back").props("flat")
archival_badge(job.status.value.upper())
if job.status in {JobStatus.QUEUED, JobStatus.PROCESSING}:
@@ -584,7 +599,7 @@ def _render_job_document_links(job: Job) -> None:
ui.label("Document Name:").classes("text-xs font-semibold ui-text-primary")
ui.button(
document_name,
on_click=lambda: ui.navigate.to(f"/documents/{job.document_id}"),
on_click=lambda: ui.navigate.to(f"/documents/{job.document_id}?from=job&job_id={job.id}"),
icon="description",
).props("flat dense no-caps").classes("ui-link-primary text-xs")
@@ -592,7 +607,7 @@ def _render_job_document_links(job: Job) -> None:
with ui.row().classes("w-full gap-2 mt-2"):
ui.button(
"View Sources",
on_click=lambda: ui.navigate.to(f"/sources?document_id={job.document_id}"),
on_click=lambda: ui.navigate.to(f"/sources?job_id={job.id}"),
icon="description",
).props("flat dense text-xs").classes("ui-link-primary")
+29 -1
View File
@@ -31,6 +31,7 @@ from transcription.ui.components.formatters import compact_date
from transcription.ui.components.formatters import family_search_url
from transcription.ui.components.formatters import google_maps_search_url
from transcription.ui.components.formatters import parse_iso_date
from transcription.ui.components.formatters import parse_uuid
from transcription.ui.components.guards import parsed_record_id
from transcription.ui.components.guards import render_record_not_found
from transcription.ui.components.media_urls import resolve_media_url
@@ -178,6 +179,14 @@ def register_page() -> None: # noqa: PLR0915
people_service = PeopleService(session_factory=session_factory)
photos_service = PhotosService(session_factory=session_factory)
render_navigation_header(current_path="/people")
back_label = "Back to People"
back_target = "/people"
from_context = request.query_params.get("from")
if from_context == "document":
document_id = parse_uuid(request.query_params.get("document_id"))
if document_id is not None:
back_label = "Back to Document"
back_target = f"/documents/{document_id}"
parsed_person_id = parsed_record_id(person_id, noun="Person")
if parsed_person_id is None:
@@ -197,6 +206,7 @@ def register_page() -> None: # noqa: PLR0915
page_header(person.full_name, subtitle=f"Person ID: {person.id}")
with ui.row().classes("items-center gap-2"):
ui.button(back_label, on_click=lambda: ui.navigate.to(back_target), icon="arrow_back").props("flat")
ui.button(
"New Document",
on_click=lambda: ui.navigate.to(f"/documents/new?person_id={person.id}"),
@@ -645,14 +655,23 @@ async def _render_person_photo_zone(
active_index = [0]
with ui.column().classes("col-span-12 lg:col-span-4"), archival_card(title="Photos", extra_classes="gap-3"):
@ui.refreshable
def render_photo_viewer() -> None:
def refresh_photo_viewer() -> None:
render_photo_viewer.refresh()
_render_photo_viewer_with_navigation(
photos=photos,
active_index=active_index,
settings=settings,
request=request,
empty_message="No portrait photo uploaded yet.",
on_change=refresh_photo_viewer,
)
render_photo_viewer()
def _shift_gallery_index(*, photos: list, active_index: list[int], step: int) -> None:
if len(photos) < 2:
@@ -770,6 +789,7 @@ def _render_linked_documents(person: Person) -> None:
{
"id": str(link.document.id),
"document_name": link.document.name,
"document_date": compact_date(link.document.document_date, link.document.document_date_raw),
"role": link.role_ref.label if link.role_ref is not None else "Unknown role",
"page_count": len(link.document.sources),
}
@@ -795,6 +815,14 @@ def _render_linked_documents(person: Person) -> None:
"classes": "text-left ui-table-cell-wrap",
"align": "left",
},
{
"name": "document_date",
"label": "Document Date",
"field": "document_date",
"sortable": True,
"classes": "font-mono",
"align": "center",
},
{
"name": "role",
"label": "Role",
@@ -812,7 +840,7 @@ def _render_linked_documents(person: Person) -> None:
},
],
default_sort_by="document_name",
on_row_click_id=lambda doc_id: ui.navigate.to(f"/documents/{doc_id}"),
on_row_click_id=lambda doc_id: ui.navigate.to(f"/documents/{doc_id}?from=person&person_id={person.id}"),
show_search=False,
)
table.add_slot(
+399 -4
View File
@@ -2,26 +2,41 @@
from __future__ import annotations
import asyncio
from datetime import UTC
from datetime import datetime
from typing import Any
from uuid import UUID
from fastapi import Request
from nicegui import events
from nicegui import ui
from transcription.config import Settings
from transcription.db.models import MaintenanceJobType
from transcription.db.models import MaintenanceRunStatus
from transcription.runtime_helpers import run_blocking
from transcription.services.documents import DocumentService
from transcription.services.maintenance import MaintenanceService
from transcription.services.people import PeopleService
from transcription.services.prompts import PromptStore
from transcription.ui.components.app_shell import render_navigation_header
from transcription.ui.components.cards import archival_card
from transcription.ui.components.error_presenter import display_failure_detail
from transcription.ui.components.error_presenter import run_ui_action
from transcription.ui.components.primitives import destructive_button
from transcription.ui.components.primitives import render_empty_state
from transcription.ui.components.primitives import section_header_row
from transcription.ui.components.table.common import build_table
from transcription.ui.components.table.registry import render_registry_table
from transcription.ui.components.upload_panel import GEDCOM_UPLOAD_EXTENSIONS
from transcription.ui.components.upload_panel import render_upload_picker
from transcription.ui.homepage_store import read_homepage_markdown
from transcription.ui.homepage_store import save_homepage_markdown
from transcription.ui.runtime_settings_store import HIDDEN_SETTINGS_CATEGORIES
from transcription.ui.runtime_settings_store import read_runtime_settings_snapshot
from transcription.ui.runtime_settings_store import save_runtime_settings
from transcription.ui.theme import page_header
from transcription.worker import resolve_worker_notifier
from ...db.session import SessionFactoryDep
@@ -30,9 +45,10 @@ def register_page(*, settings: Settings) -> None: # noqa: PLR0915
"""Register the constrained Settings route."""
@ui.page("/settings")
async def settings_page(session_factory: SessionFactoryDep) -> None: # noqa: PLR0915
async def settings_page(request: Request, session_factory: SessionFactoryDep) -> None: # noqa: PLR0915
documents = DocumentService(session_factory=session_factory)
people = PeopleService(session_factory=session_factory)
maintenance = MaintenanceService(session_factory=session_factory)
prompts = PromptStore(settings=settings)
render_navigation_header(current_path="/settings")
@@ -497,12 +513,260 @@ def register_page(*, settings: Settings) -> None: # noqa: PLR0915
with ui.row().classes("items-center gap-2"):
ui.button("Save home text", icon="save", on_click=save_home_text).classes("ui-btn-primary")
@ui.refreshable
async def render_maintenance() -> None:
with archival_card("Maintenance"):
ui.label(
"Queue maintenance tasks for worker execution. Runs are persisted with summary and logs."
).classes("text-xs ui-text-muted mb-3")
await _render_gedcom_import_controls(
maintenance=maintenance,
request=request,
refresh=render_maintenance.refresh,
)
with ui.row().classes("w-full items-center gap-2"):
ui.button(
"Run Backup",
icon="save",
on_click=lambda: _enqueue_maintenance_run(
maintenance=maintenance,
job_type=MaintenanceJobType.BACKUP,
request=request,
refresh=render_maintenance.refresh,
),
).props("flat")
ui.button(
"Run Storage Reconciliation",
icon="rule",
on_click=lambda: _enqueue_maintenance_run(
maintenance=maintenance,
job_type=MaintenanceJobType.STORAGE_RECONCILIATION,
request=request,
refresh=render_maintenance.refresh,
),
).props("flat")
runs_outcome = await run_ui_action(
operation="settings.maintenance.list",
title="Maintenance runs unavailable",
action=maintenance.list_runs,
)
if not runs_outcome.ok:
return
runs = list(runs_outcome.value or ())
if not runs:
render_empty_state("No maintenance runs recorded yet.", extra_classes="mt-3")
return
rows = [
{
"id": str(run.id),
"job_type": run.job_type.value.replace("_", " ").title(),
"status": run.status.value,
"started_at": _format_timestamp(run.started_at),
"finished_at": _format_timestamp(run.finished_at),
"duration": _format_duration(started_at=run.started_at, finished_at=run.finished_at),
"summary": run.summary or "-",
"log_path": run.log_path or "",
"error_detail": display_failure_detail(run.error_detail) or "",
}
for run in runs
]
table = build_table(
rows=rows,
columns=[
{"name": "job_type", "label": "Job Type", "field": "job_type", "sortable": True},
{"name": "status", "label": "Status", "field": "status", "sortable": True},
{"name": "started_at", "label": "Started", "field": "started_at", "sortable": True},
{"name": "finished_at", "label": "Finished", "field": "finished_at", "sortable": True},
{"name": "duration", "label": "Duration", "field": "duration", "sortable": False},
{"name": "summary", "label": "Summary", "field": "summary", "sortable": False},
],
row_key="id",
selection="single",
show_search=False,
)
table.add_slot(
"body-cell-status",
r"""
<q-td :props="props">
<q-chip
dense
square
size="sm"
:class="`ui-status ui-status--${props.value}`"
>
{{ props.value.toUpperCase() }}
</q-chip>
</q-td>
""",
)
run_by_id = {str(run.id): run for run in runs}
def selected_run_id() -> str | None:
selected = table.selected or []
if len(selected) != 1:
return None
return str(selected[0].get("id") or "")
async def view_log() -> None:
run_id = selected_run_id()
if not run_id:
ui.notify("Select one run first.", type="warning")
return
run = run_by_id.get(run_id)
if run is None or not run.log_path:
ui.notify("Log unavailable for this run.", type="warning")
return
log_path = run.log_path
log_outcome = await run_ui_action(
operation="settings.maintenance.log.read",
title="Maintenance log unavailable",
action=lambda: _read_maintenance_log(maintenance=maintenance, log_path=log_path),
)
if not log_outcome.ok or log_outcome.value is None:
return
with ui.dialog() as dialog, ui.card().classes("w-full max-w-4xl"):
ui.label(f"Log: {run.log_path}").classes("text-sm font-semibold")
ui.code(log_outcome.value.decode("utf-8", errors="replace"), language="text").classes(
"w-full text-xs max-h-[65vh] overflow-auto"
)
with ui.row().classes("w-full justify-end"):
ui.button("Close", on_click=dialog.close).props("flat")
dialog.open()
async def download_log() -> None:
run_id = selected_run_id()
if not run_id:
ui.notify("Select one run first.", type="warning")
return
run = run_by_id.get(run_id)
if run is None or not run.log_path:
ui.notify("Log unavailable for this run.", type="warning")
return
log_path = run.log_path
log_outcome = await run_ui_action(
operation="settings.maintenance.log.download",
title="Maintenance log unavailable",
action=lambda: _read_maintenance_log(maintenance=maintenance, log_path=log_path),
)
if not log_outcome.ok or log_outcome.value is None:
return
ui.download(log_outcome.value, filename=f"{run.id}.log", media_type="text/plain")
with ui.row().classes("w-full justify-end items-center gap-2 mt-2"):
ui.button("View Log", icon="visibility", on_click=view_log).props("flat")
ui.button("Download Log", icon="download", on_click=download_log).props("flat")
_schedule_maintenance_refresh_if_active(runs=runs, refresh=render_maintenance.refresh)
@ui.refreshable
async def render_runtime_settings() -> None:
with archival_card("Runtime Settings"):
ui.label(
"Edit non-secret .env settings. Secret fields are intentionally excluded. "
"Changes are persisted to .env and apply after restart."
).classes("text-xs ui-text-muted mb-3")
snapshot_outcome = await run_ui_action(
operation="settings.runtime.read",
title="Runtime settings unavailable",
action=lambda: _read_runtime_settings_snapshot(settings),
)
if not snapshot_outcome.ok or snapshot_outcome.value is None:
return
snapshot = snapshot_outcome.value
field_controls: dict[str, Any] = {}
with ui.column().classes("w-full max-w-3xl mx-auto gap-0.5"):
with ui.row().classes("w-full items-center px-2 py-1 border-b ui-border-subtle"):
ui.label("Setting").classes("w-56 text-xs font-semibold ui-text-muted")
ui.label("Value").classes("text-xs font-semibold ui-text-muted")
for field in snapshot.fields:
with ui.row().classes("w-full items-start gap-3 px-2 py-1 border-b ui-border-subtle"):
with ui.column().classes("w-56 gap-0"):
ui.label(field.label).classes("text-xs font-semibold")
ui.label(f"{field.env_key} · {field.description}").classes(
"text-[11px] ui-text-muted"
)
with ui.column().classes("flex-1 min-w-0"):
if field.control == "bool":
control = ui.checkbox("", value=bool(field.value)).props("dense")
elif field.control == "select":
control = (
ui.select(
list(field.options),
label="",
value=str(field.value),
)
.props("outlined dense")
.classes("w-full")
)
else:
control = (
ui.input("", value=str(field.value))
.props("outlined dense")
.classes("w-full")
)
field_controls[field.field_name] = control
with ui.column().classes("w-full max-w-3xl mx-auto gap-2 mt-3"):
ui.label("Other settings not shown here").classes("text-sm font-semibold")
ui.label(f"Edit these directly in {snapshot.env_file_path}:").classes("text-xs ui-text-muted")
table_rows = [
f"| {category.title} | {', '.join(f'`{key}`' for key in category.env_keys)} |"
for category in HIDDEN_SETTINGS_CATEGORIES
]
ui.markdown(
"\n".join(
[
"| **Category** | **Settings** |",
"|---|---|",
*table_rows,
]
)
).classes("w-full text-xs")
async def save_runtime() -> None:
updates: dict[str, str | bool] = {}
for field in snapshot.fields:
control = field_controls[field.field_name]
if field.control == "bool":
updates[field.field_name] = bool(control.value)
else:
updates[field.field_name] = str(control.value or "")
save_outcome = await run_ui_action(
operation="settings.runtime.write",
title="Runtime settings save failed",
action=lambda: _write_runtime_settings(settings=settings, updates=updates),
)
if not save_outcome.ok or save_outcome.value is None:
return
target_path = save_outcome.value.env_file_path
ui.notify(
f"Runtime settings saved to {target_path}. Restart app/worker to apply.",
type="positive",
)
render_runtime_settings.refresh()
with ui.row().classes("items-center gap-2 mt-3"):
ui.button("Save runtime settings", icon="save", on_click=save_runtime).classes("ui-btn-primary")
with ui.column().classes("w-full max-w-3xl mx-auto gap-1 mt-2"):
ui.label("Apply saved runtime settings").classes("text-xs font-semibold")
ui.label(
"Run this on the host from the deployment repo to recreate app/worker with new env values:"
).classes("text-[11px] ui-text-muted")
ui.code(_runtime_restart_command(), language="bash").classes("w-full text-xs")
with ui.tabs().classes("w-full") as tabs:
document_types_tab = ui.tab("Document Types")
person_roles_tab = ui.tab("Person Roles")
tags_tab = ui.tab("Tags")
prompts_tab = ui.tab("Prompts")
home_page_text_tab = ui.tab("Home Page Text")
maintenance_tab = ui.tab("Maintenance")
runtime_settings_tab = ui.tab("Runtime Settings")
with ui.tab_panels(tabs, value=document_types_tab).classes("w-full"):
with ui.tab_panel(document_types_tab):
@@ -515,6 +779,10 @@ def register_page(*, settings: Settings) -> None: # noqa: PLR0915
await render_prompts()
with ui.tab_panel(home_page_text_tab):
await render_home_page_text()
with ui.tab_panel(maintenance_tab):
await render_maintenance()
with ui.tab_panel(runtime_settings_tab):
await render_runtime_settings()
def _selected_table_row(table: Any) -> dict[str, Any] | None:
@@ -541,8 +809,135 @@ async def _recover_prompt(prompts: PromptStore, name: str) -> None:
async def _read_home_page_text(settings: Settings) -> str:
return await asyncio.to_thread(read_homepage_markdown, settings=settings)
return await run_blocking(read_homepage_markdown, settings=settings)
async def _write_home_page_text(settings: Settings, markdown_text: str) -> None:
await asyncio.to_thread(save_homepage_markdown, markdown_text, settings=settings)
await run_blocking(save_homepage_markdown, markdown_text, settings=settings)
async def _read_runtime_settings_snapshot(settings: Settings):
return await run_blocking(read_runtime_settings_snapshot, settings=settings)
async def _write_runtime_settings(*, settings: Settings, updates: dict[str, str | bool]):
return await run_blocking(save_runtime_settings, settings=settings, updates=updates)
async def _enqueue_maintenance_run(
*,
maintenance: MaintenanceService,
job_type: MaintenanceJobType,
request: Request,
refresh,
) -> None:
created_outcome = await run_ui_action(
operation="settings.maintenance.enqueue",
title="Maintenance run failed",
action=lambda: maintenance.enqueue_run(job_type=job_type),
)
if not created_outcome.ok or created_outcome.value is None:
return
resolve_worker_notifier(request.app.state).notify()
ui.notify(f"Queued {job_type.value.replace('_', ' ')} run", type="positive")
refresh()
async def _render_gedcom_import_controls(
*,
maintenance: MaintenanceService,
request: Request,
refresh,
) -> None:
ui.label("GEDCOM Import").classes("text-sm font-semibold")
ui.label("Upload a FamilySearch GEDCOM export (.ged), then queue a GEDCOM import run.").classes(
"text-xs ui-text-muted mb-2"
)
async def on_gedcom_upload(event: events.UploadEventArguments) -> None:
payload = await event.file.read()
upload_outcome = await run_ui_action(
operation="settings.maintenance.gedcom.upload",
title="GEDCOM upload failed",
action=lambda: maintenance.store_gedcom_upload(
filename=event.file.name,
file_bytes=payload,
),
)
if not upload_outcome.ok or upload_outcome.value is None:
return
ui.notify(f"Uploaded GEDCOM file: {event.file.name}", type="positive")
refresh()
render_upload_picker(
on_upload=on_gedcom_upload,
label="Upload GEDCOM file",
extensions=GEDCOM_UPLOAD_EXTENSIONS,
)
latest_gedcom_outcome = await run_ui_action(
operation="settings.maintenance.gedcom.latest",
title="GEDCOM uploads unavailable",
action=lambda: _latest_gedcom_upload_path(maintenance=maintenance),
)
latest_gedcom_path = latest_gedcom_outcome.value if latest_gedcom_outcome.ok else None
ui.label(
f"Latest GEDCOM upload: {latest_gedcom_path}" if latest_gedcom_path else "Latest GEDCOM upload: none"
).classes("text-xs ui-text-muted mb-2")
with ui.row().classes("w-full items-center gap-2"):
ui.button(
"Run GEDCOM Import",
icon="upload_file",
on_click=lambda: _enqueue_maintenance_run(
maintenance=maintenance,
job_type=MaintenanceJobType.GEDCOM_IMPORT,
request=request,
refresh=refresh,
),
).classes("ui-btn-primary")
async def _latest_gedcom_upload_path(*, maintenance: MaintenanceService) -> str | None:
return maintenance.latest_gedcom_upload_path()
def _schedule_maintenance_refresh_if_active(*, runs: list[Any], refresh) -> None:
active_statuses = frozenset(
{
MaintenanceRunStatus.QUEUED,
MaintenanceRunStatus.PROCESSING,
}
)
if any(run.status in active_statuses for run in runs):
ui.timer(4.0, refresh, once=True)
def _format_timestamp(value: datetime | None) -> str:
if value is None:
return "-"
parsed = value.replace(tzinfo=UTC) if value.tzinfo is None else value.astimezone(UTC)
return parsed.astimezone().strftime("%b %d, %I:%M %p")
def _format_duration(*, started_at: datetime | None, finished_at: datetime | None) -> str:
if started_at is None:
return "-"
if finished_at is None:
return "in progress"
elapsed = finished_at - started_at
seconds = int(elapsed.total_seconds())
if seconds < 1:
return "<1s"
minutes, remainder = divmod(seconds, 60)
if minutes:
return f"{minutes}m {remainder}s"
return f"{remainder}s"
async def _read_maintenance_log(*, maintenance: MaintenanceService, log_path: str) -> bytes:
return await run_blocking(maintenance.read_log_bytes, log_path=log_path)
def _runtime_restart_command() -> str:
return "docker compose -f docker-compose.production.yml up -d --force-recreate app worker"
+21 -18
View File
@@ -9,6 +9,7 @@ from pathlib import Path
from uuid import UUID
from fastapi import Request
from fastapi.responses import RedirectResponse
from nicegui import ui
from PIL import Image
from PIL import UnidentifiedImageError
@@ -32,6 +33,7 @@ from transcription.ui.components.confirm_delete import render_delete_blocked_not
from transcription.ui.components.data_display import archival_badge
from transcription.ui.components.data_display import metadata_row
from transcription.ui.components.document_panzoom import render_document_panzoom
from transcription.ui.components.error_presenter import display_failure_detail
from transcription.ui.components.error_presenter import run_ui_action
from transcription.ui.components.error_presenter import show_error
from transcription.ui.components.formatters import parse_uuid
@@ -62,12 +64,15 @@ def register_page() -> None: # noqa: PLR0915
session_factory: SessionFactoryDep,
document_id: str | None = None,
job_id: str | None = None,
) -> None:
) -> RedirectResponse | None:
sources_service = SourceService(session_factory=session_factory)
parsed_doc_id = parse_uuid(document_id)
parsed_job_id = parse_uuid(job_id)
header_title = "Source Asset Records"
if parsed_doc_id is None and parsed_job_id is None:
return RedirectResponse(url="/ui/documents")
header_title = ""
if parsed_doc_id is not None:
header_title = "Sources for Document"
elif parsed_job_id is not None:
@@ -101,10 +106,10 @@ def register_page() -> None: # noqa: PLR0915
).props("flat")
else:
ui.button(
"Create Job",
on_click=lambda: ui.navigate.to("/jobs/new"),
icon="add",
).classes("ui-btn-primary")
"Back to Documents",
on_click=lambda: ui.navigate.to("/documents"),
icon="arrow_back",
).props("flat")
rows = [
SourceTableRow(
@@ -114,17 +119,12 @@ def register_page() -> None: # noqa: PLR0915
document_id=source.document_id,
document_name=source.document_name,
job_source_status=source.latest_status.value if source.latest_status else "unprocessed",
job_source_error_detail=source.latest_error_detail,
job_source_error_detail=display_failure_detail(source.latest_error_detail),
)
for source in sorted(sources, key=lambda item: (item.page_number, item.upload_name.casefold()))
]
render_sources_table(rows)
if parsed_doc_id is None and parsed_job_id is None:
ui.label("Open a source row to inspect AI output and add human revisions.").classes(
"text-xs ui-text-muted"
)
@ui.page("/sources/{source_id}")
async def source_detail_page(source_id: str, request: Request, session_factory: SessionFactoryDep) -> None:
sources_service = SourceService(session_factory=session_factory)
@@ -164,8 +164,8 @@ def register_page() -> None: # noqa: PLR0915
)
with ui.row().classes("items-center gap-2"):
ui.button(
"Back to Sources",
on_click=lambda: ui.navigate.to("/sources"),
"Back to Document",
on_click=lambda: ui.navigate.to(f"/documents/{source.document_id}?source_id={source.id}"),
icon="arrow_back",
).props("flat")
ui.button(
@@ -267,14 +267,14 @@ def register_page() -> None: # noqa: PLR0915
return
except TranscriptionNotFoundError:
ui.notify("Source not found.", type="warning")
ui.navigate.to("/sources")
ui.navigate.to("/documents")
return
except Exception as exc: # noqa: BLE001
show_error(exc, title="Delete failed", operation="sources.delete")
return
ui.notify("Source deleted", type="positive")
ui.navigate.to("/sources")
ui.navigate.to("/documents")
render_delete_actions(
confirm_label="Delete source permanently",
@@ -479,10 +479,13 @@ def _render_source_job_metadata_zone(
else "unknown",
)
if latest_attempt is not None and latest_attempt.attempt.error_detail:
failure_detail = (
display_failure_detail(latest_attempt.attempt.error_detail) if latest_attempt is not None else None
)
if failure_detail:
with ui.column().classes("w-full mt-2"):
ui.label("Failure Detail:").classes("ui-text-muted text-xs mb-1")
ui.label(latest_attempt.attempt.error_detail).classes("p-2 ui-note-box text-xs")
ui.label(failure_detail).classes("p-2 ui-note-box text-xs")
_render_provider_evidence(latest_attempt=latest_attempt)
-92
View File
@@ -1,92 +0,0 @@
"""Tags browse and filter page registration."""
from __future__ import annotations
from nicegui import ui
from transcription.services.documents import DocumentService
from transcription.ui.components.app_shell import render_navigation_header
from transcription.ui.components.cards import archival_card
from transcription.ui.components.error_presenter import run_ui_action
from transcription.ui.components.primitives import render_empty_state
from transcription.ui.components.primitives import section_header_row
from transcription.ui.theme import page_header
from ...db.session import SessionFactoryDep
def register_page() -> None:
"""Register the tags browse/filter route."""
@ui.page("/tags")
async def tags_page(session_factory: SessionFactoryDep) -> None:
document_service = DocumentService(session_factory=session_factory)
render_navigation_header(current_path="/tags")
tags_outcome = await run_ui_action(
operation="tags.list",
title="Tags unavailable",
action=document_service.list_tag_summaries,
)
if not tags_outcome.ok:
return
tag_summaries = tags_outcome.value or ()
tag_labels = [item.label for item in tag_summaries]
documents_outcome = await run_ui_action(
operation="documents.list",
title="Documents unavailable",
action=document_service.list_documents,
)
if not documents_outcome.ok:
return
documents = documents_outcome.value or ()
with ui.column().classes("w-full max-w-7xl mx-auto p-4 gap-4"):
with section_header_row():
page_header("Tags", subtitle="Browse documents by tag.")
if not tag_summaries:
with archival_card(extra_classes="p-8 text-center"):
render_empty_state("No tags are configured yet.")
return
selected_tag = (
ui.select(tag_labels, label="Filter by tag")
.props("outlined clearable use-input")
.classes("w-full md:w-96 ui-form-surface")
)
@ui.refreshable
def render_groups() -> None:
selected = str(selected_tag.value or "").strip()
with ui.column().classes("w-full gap-3"):
rendered_any = False
for summary in tag_summaries:
if selected and summary.label != selected:
continue
tagged_documents = [
document
for document in documents
if any(
link.tag_ref is not None and link.tag_ref.id == summary.id
for link in document.document_tags
)
]
if not tagged_documents:
continue
rendered_any = True
with archival_card(title=f"{summary.label} ({len(tagged_documents)})"):
for document in sorted(tagged_documents, key=lambda item: item.name.casefold()):
ui.button(
document.name,
on_click=lambda _=None, doc_id=document.id: ui.navigate.to(f"/documents/{doc_id}"),
icon="description",
).props("flat dense no-caps").classes("self-start ui-link-primary text-xs")
if not rendered_any:
with archival_card(extra_classes="p-6"):
render_empty_state("No documents match this tag filter.", italic=True)
selected_tag.on_value_change(lambda _event: render_groups.refresh())
render_groups()
@@ -0,0 +1,552 @@
"""Safe runtime settings catalog and .env persistence helpers."""
from __future__ import annotations
import os
import re
import tempfile
from dataclasses import dataclass
from errno import EBUSY
from errno import EPERM
from errno import EXDEV
from pathlib import Path
from typing import Literal
from pydantic import ValidationError
from transcription.config import Provider
from transcription.config import Settings
from transcription.config import resolve_settings_env_file_path
from transcription.errors import AppError
from transcription.errors import ErrorCategory
from transcription.errors import exception_detail
FieldControl = Literal["text", "bool", "select"]
@dataclass(frozen=True, slots=True)
class RuntimeSettingField:
"""UI metadata and value for a safe, editable runtime setting."""
field_name: str
env_key: str
label: str
description: str
control: FieldControl
value: str | bool
options: tuple[str, ...] = ()
@dataclass(frozen=True, slots=True)
class RuntimeSettingsSnapshot:
"""Typed snapshot of runtime settings editable in UI."""
env_file_path: Path
fields: tuple[RuntimeSettingField, ...]
@dataclass(frozen=True, slots=True)
class RuntimeSettingDescriptor:
field_name: str
env_key: str
label: str
description: str
control: FieldControl
options: tuple[str, ...] = ()
@dataclass(frozen=True, slots=True)
class HiddenSettingsCategory:
"""Settings excluded from the UI editor and why they are excluded."""
title: str
reason: str
env_keys: tuple[str, ...]
SETTINGS_UI_EXCLUDED_FIELDS = frozenset(
{
"openrouter_api_key",
"database",
}
)
HIDDEN_SETTINGS_CATEGORIES: tuple[HiddenSettingsCategory, ...] = (
HiddenSettingsCategory(
title="Not safe to expose/edit as plain text in UI (secrets)",
reason="These values are credentials and must remain hidden from page rendering and client responses.",
env_keys=("OPENROUTER_API_KEY", "DATABASE__PASSWORD"),
),
HiddenSettingsCategory(
title="Non-secret but high-risk (can break runtime/connectivity; still editable with guardrails)",
reason="These values control database connectivity and can make the app unavailable if changed incorrectly.",
env_keys=(
"DATABASE__DRIVER",
"DATABASE__PATH",
"DATABASE__HOST",
"DATABASE__PORT",
"DATABASE__DATABASE",
"DATABASE__USER",
),
),
HiddenSettingsCategory(
title="Deployment and backup/tunnel helpers (not Runtime Settings model fields)",
reason=(
"These keys are consumed by docker-compose, backup scripts, or deployment adapters and are "
"intentionally edited outside the Runtime Settings UI."
),
env_keys=(
"CLOUDFLARE_TUNNEL_TOKEN",
"RUNTIME_SETTINGS_ENV_FILE",
"BACKUP_DIR",
"BACKUP_RETENTION_DAYS",
"ENV_FILE",
"COMPOSE_FILE",
"POSTGRES_DB",
"POSTGRES_USER",
"POSTGRES_PASSWORD",
"POSTGRES_HOST",
"POSTGRES_PORT",
"DATABASE_BACKUP_DIR",
"APP_DATA_BACKUP_DIR",
"UPLOADS_BACKUP_DIR",
"SYNOLOGY_BACKUP_DIR",
),
),
)
RUNTIME_SETTINGS_CATALOG: tuple[RuntimeSettingDescriptor, ...] = (
RuntimeSettingDescriptor("host", "HOST", "Host", "Server bind host.", "text"),
RuntimeSettingDescriptor("port", "PORT", "Port", "Server bind port.", "text"),
RuntimeSettingDescriptor(
"log_level",
"LOG_LEVEL",
"Log level",
"Application log verbosity.",
"select",
options=("critical", "error", "warning", "info", "debug", "trace"),
),
RuntimeSettingDescriptor("reload", "RELOAD", "Reload", "Auto-reload on code changes.", "bool"),
RuntimeSettingDescriptor("log_dir", "LOG_DIR", "Log directory", "Directory for rotating log files.", "text"),
RuntimeSettingDescriptor(
"log_file_name",
"LOG_FILE_NAME",
"Log file name",
"Active rotating log filename.",
"text",
),
RuntimeSettingDescriptor(
"log_file_max_bytes",
"LOG_FILE_MAX_BYTES",
"Log file max bytes",
"Maximum bytes per log file before rotation.",
"text",
),
RuntimeSettingDescriptor(
"log_file_backup_count",
"LOG_FILE_BACKUP_COUNT",
"Log file backup count",
"Number of rotated log files to keep.",
"text",
),
RuntimeSettingDescriptor(
"provider",
"PROVIDER",
"Provider",
"Transcription provider key.",
"select",
options=tuple(member.value for member in Provider),
),
RuntimeSettingDescriptor(
"provider_model",
"PROVIDER_MODEL",
"Provider model",
"Default model id used for transcription jobs.",
"text",
),
RuntimeSettingDescriptor(
"provider_models",
"PROVIDER_MODELS",
"Provider models (JSON array)",
"Allowed model ids as a JSON array.",
"text",
),
RuntimeSettingDescriptor(
"openrouter_http_referer",
"OPENROUTER_HTTP_REFERER",
"OpenRouter HTTP referer",
"Optional header value sent to OpenRouter.",
"text",
),
RuntimeSettingDescriptor(
"openrouter_app_title",
"OPENROUTER_APP_TITLE",
"OpenRouter app title",
"Optional app title sent to OpenRouter.",
"text",
),
RuntimeSettingDescriptor(
"default_prompt_name",
"DEFAULT_PROMPT_NAME",
"Default prompt file",
"Prompt filename used for new transcription requests.",
"text",
),
RuntimeSettingDescriptor(
"transcription_temperature",
"TRANSCRIPTION_TEMPERATURE",
"Temperature",
"Optional sampling temperature (0.0 to 2.0).",
"text",
),
RuntimeSettingDescriptor(
"transcription_top_p",
"TRANSCRIPTION_TOP_P",
"Top P",
"Optional nucleus sampling probability (0.0 to 1.0).",
"text",
),
RuntimeSettingDescriptor(
"environment",
"ENVIRONMENT",
"Environment",
"Runtime environment mode.",
"select",
options=("development", "test", "production"),
),
RuntimeSettingDescriptor(
"transcription_commit",
"TRANSCRIPTION_COMMIT",
"Transcription commit",
"Optional build or commit identifier for evidence provenance.",
"text",
),
RuntimeSettingDescriptor(
"run_embedded_worker",
"RUN_EMBEDDED_WORKER",
"Run embedded worker",
"Run the in-process worker loop in the web app process.",
"bool",
),
RuntimeSettingDescriptor(
"bootstrap_schema_on_startup",
"BOOTSTRAP_SCHEMA_ON_STARTUP",
"Bootstrap schema on startup",
"Create schema on startup when enabled.",
"bool",
),
RuntimeSettingDescriptor(
"sqlite_check_same_thread",
"SQLITE_CHECK_SAME_THREAD",
"SQLite check same thread",
"SQLite engine same-thread flag.",
"bool",
),
RuntimeSettingDescriptor(
"upload_dir",
"UPLOAD_DIR",
"Upload directory",
"Root directory for uploaded media and generated files.",
"text",
),
RuntimeSettingDescriptor(
"prompt_dir",
"PROMPT_DIR",
"Prompt directory",
"Directory containing editable markdown prompts.",
"text",
),
RuntimeSettingDescriptor(
"worker_max_retries",
"WORKER_MAX_RETRIES",
"Worker max retries",
"Maximum retries per source attempt.",
"text",
),
RuntimeSettingDescriptor(
"worker_provider_timeout_seconds",
"WORKER_PROVIDER_TIMEOUT_SECONDS",
"Worker provider timeout seconds",
"Provider call timeout budget in seconds.",
"text",
),
RuntimeSettingDescriptor(
"worker_stale_job_seconds",
"WORKER_STALE_JOB_SECONDS",
"Worker stale job seconds",
"Seconds before queued jobs are considered stale.",
"text",
),
RuntimeSettingDescriptor(
"worker_retry_backoff_seconds",
"WORKER_RETRY_BACKOFF_SECONDS",
"Worker retry backoff seconds",
"Base backoff delay between retries.",
"text",
),
RuntimeSettingDescriptor(
"worker_shutdown_grace_seconds",
"WORKER_SHUTDOWN_GRACE_SECONDS",
"Worker shutdown grace seconds",
"Grace period before forced worker shutdown.",
"text",
),
RuntimeSettingDescriptor(
"worker_poll_interval_seconds",
"WORKER_POLL_INTERVAL_SECONDS",
"Worker poll interval seconds",
"Worker polling interval for queued jobs.",
"text",
),
RuntimeSettingDescriptor(
"worker_min_transcription_chars",
"WORKER_MIN_TRANSCRIPTION_CHARS",
"Worker min transcription chars",
"Minimum character threshold for successful transcription.",
"text",
),
RuntimeSettingDescriptor(
"worker_min_transcription_lines",
"WORKER_MIN_TRANSCRIPTION_LINES",
"Worker min transcription lines",
"Minimum line threshold for successful transcription.",
"text",
),
RuntimeSettingDescriptor(
"worker_fail_on_finish_reason_length",
"WORKER_FAIL_ON_FINISH_REASON_LENGTH",
"Fail on finish reason length",
"Treat provider finish_reason=length as failure when enabled.",
"bool",
),
)
def settings_catalog_field_names() -> frozenset[str]:
return frozenset(item.field_name for item in RUNTIME_SETTINGS_CATALOG)
def read_runtime_settings_snapshot(*, settings: Settings, env_file_path: Path | None = None) -> RuntimeSettingsSnapshot:
resolved_env_path = _resolve_env_file_path(settings=settings, env_file_path=env_file_path)
fields: list[RuntimeSettingField] = []
for descriptor in RUNTIME_SETTINGS_CATALOG:
value = getattr(settings, descriptor.field_name)
fields.append(
RuntimeSettingField(
field_name=descriptor.field_name,
env_key=descriptor.env_key,
label=descriptor.label,
description=descriptor.description,
control=descriptor.control,
value=_display_value(value),
options=descriptor.options,
)
)
return RuntimeSettingsSnapshot(
env_file_path=resolved_env_path,
fields=tuple(fields),
)
def save_runtime_settings(
*,
settings: Settings,
updates: dict[str, str | bool],
env_file_path: Path | None = None,
) -> RuntimeSettingsSnapshot:
descriptors = {item.field_name: item for item in RUNTIME_SETTINGS_CATALOG}
unknown_keys = sorted(set(updates) - set(descriptors))
if unknown_keys:
raise AppError(
"One or more settings fields cannot be edited from this page.",
category=ErrorCategory.VALIDATION,
suggestion="Refresh the page and submit only editable fields.",
detail=f"Unknown settings keys: {unknown_keys}",
)
resolved_env_path = _resolve_env_file_path(settings=settings, env_file_path=env_file_path)
original_lines = _read_env_lines(resolved_env_path)
env_map = _parse_env_map(original_lines)
mutable_lines = list(original_lines)
for field_name, raw_value in updates.items():
descriptor = descriptors[field_name]
encoded = _encode_field_value(raw_value)
if encoded == "":
_delete_env_key(mutable_lines, descriptor.env_key)
env_map.pop(descriptor.env_key, None)
continue
_upsert_env_key(mutable_lines, descriptor.env_key, encoded)
env_map[descriptor.env_key] = encoded
_validate_candidate_env(env_map=env_map, env_file_path=resolved_env_path)
try:
_write_env_lines_atomic(path=resolved_env_path, lines=mutable_lines)
except OSError as exc:
raise AppError(
"Runtime settings file is not writable.",
category=ErrorCategory.INFRA_PERSISTENT,
suggestion="Verify file path and write permissions, then retry.",
detail=f"Failed writing runtime env file {resolved_env_path}: {exception_detail(exc)}",
) from exc
refreshed = Settings(_env_file=resolved_env_path, _cli_parse_args=False)
return read_runtime_settings_snapshot(settings=refreshed, env_file_path=resolved_env_path)
def _resolve_env_file_path(*, settings: Settings, env_file_path: Path | None) -> Path:
_ = settings
if env_file_path is not None:
return env_file_path
override = os.getenv("RUNTIME_SETTINGS_ENV_FILE", "").strip()
if override:
return Path(override)
return resolve_settings_env_file_path()
def _display_value(value: object) -> str | bool:
if isinstance(value, bool):
return value
if value is None:
return ""
if isinstance(value, tuple):
values = [str(item) for item in value]
escaped = ",".join(f'"{_escape_json_string(item)}"' for item in values)
return f"[{escaped}]"
if isinstance(value, Path):
return value.as_posix()
if isinstance(value, Provider):
return value.value
return str(value)
def _encode_field_value(raw_value: str | bool) -> str:
if isinstance(raw_value, bool):
return "true" if raw_value else "false"
text = str(raw_value).strip()
if text == "":
return ""
if text.startswith("[") and text.endswith("]"):
return text
if " " in text or "#" in text:
escaped = text.replace("\\", "\\\\").replace('"', '\\"')
return f'"{escaped}"'
return text
def _validate_candidate_env(*, env_map: dict[str, str], env_file_path: Path) -> None:
serialized = "\n".join(f"{key}={value}" for key, value in env_map.items()) + "\n"
with tempfile.NamedTemporaryFile(mode="w", delete=False, encoding="utf-8", newline="\n", suffix=".env") as handle:
temp_path = Path(handle.name)
handle.write(serialized)
try:
Settings(_env_file=temp_path, _cli_parse_args=False)
except ValidationError as exc:
raise AppError(
"One or more settings values are invalid.",
category=ErrorCategory.VALIDATION,
suggestion="Review the highlighted values and use the same formats shown in .env.production.example.",
detail=f"Invalid runtime settings for {env_file_path}: {exc}",
) from exc
finally:
temp_path.unlink(missing_ok=True)
def _read_env_lines(path: Path) -> list[str]:
if not path.exists():
return []
try:
return path.read_text(encoding="utf-8").splitlines()
except OSError as exc:
raise AppError(
"Runtime settings file is unreadable.",
category=ErrorCategory.INFRA_PERSISTENT,
suggestion="Verify file path and read permissions, then retry.",
detail=f"Failed reading runtime env file {path}: {exception_detail(exc)}",
) from exc
def _parse_env_map(lines: list[str]) -> dict[str, str]:
env_map: dict[str, str] = {}
for line in lines:
parsed = _parse_env_line(line)
if parsed is None:
continue
key, value = parsed
env_map[key] = value
return env_map
def _parse_env_line(line: str) -> tuple[str, str] | None:
if not line or line.lstrip().startswith("#"):
return None
match = re.match(r"^\s*([A-Za-z_][A-Za-z0-9_]*(?:__[A-Za-z0-9_]+)*)\s*=\s*(.*)$", line)
if not match:
return None
return match.group(1), match.group(2)
def _delete_env_key(lines: list[str], key: str) -> None:
for index, line in enumerate(lines):
parsed = _parse_env_line(line)
if parsed is None:
continue
current_key, _ = parsed
if current_key == key:
del lines[index]
return
def _upsert_env_key(lines: list[str], key: str, value: str) -> None:
for index, line in enumerate(lines):
parsed = _parse_env_line(line)
if parsed is None:
continue
current_key, _ = parsed
if current_key == key:
lines[index] = f"{key}={value}"
return
lines.append(f"{key}={value}")
def _write_env_lines_atomic(*, path: Path, lines: list[str]) -> None:
temp_path: Path | None = None
try:
path.parent.mkdir(parents=True, exist_ok=True)
content = "\n".join(lines).rstrip("\n")
if content:
content += "\n"
with tempfile.NamedTemporaryFile(
mode="w",
delete=False,
dir=path.parent,
encoding="utf-8",
newline="\n",
suffix=".env.tmp",
) as handle:
temp_path = Path(handle.name)
handle.write(content)
try:
temp_path.replace(path)
except OSError as exc:
# Single-file bind mounts can reject replace() (cross-device or busy mountpoint).
# Fallback to direct write so Runtime Settings can persist to mounted env files.
if exc.errno not in {EXDEV, EBUSY, EPERM}:
raise
with path.open("w", encoding="utf-8", newline="\n") as handle:
handle.write(content)
except OSError as exc:
raise AppError(
"Runtime settings file is not writable.",
category=ErrorCategory.INFRA_PERSISTENT,
suggestion="Verify file path and write permissions, then retry.",
detail=f"Failed writing runtime env file {path}: {exception_detail(exc)}",
) from exc
finally:
if temp_path is not None:
temp_path.unlink(missing_ok=True)
def _escape_json_string(value: str) -> str:
return value.replace("\\", "\\\\").replace('"', '\\"')
+2 -1
View File
@@ -299,7 +299,8 @@ input:focus-visible,
}
.ui-status--partial_success,
.ui-status--transcribed {
.ui-status--transcribed,
.ui-status--succeeded {
color: var(--theme-text);
background: var(--theme-secondary);
}
+8 -5
View File
@@ -229,17 +229,20 @@ async def run_worker_loop(
processed_any = False
while True:
processed_job = False
with handle_worker_exceptions(operation="worker.process_next_queued_job"):
processed = await process_next_queued_job(
processed_job = await process_next_queued_job(
session_factory=session_factory,
services=services,
)
if not processed:
processed_maintenance = False
if session_factory is not None:
with handle_worker_exceptions(operation="worker.process_next_queued_maintenance"):
processed_maintenance = await services.maintenance.process_next_queued_run()
if not processed_job and not processed_maintenance:
break
processed_any = True
continue
break
if wake_event is None and not processed_any:
await asyncio.sleep(poll_interval_seconds)
+43
View File
@@ -0,0 +1,43 @@
"""Standalone worker process entrypoint for production deployments."""
from __future__ import annotations
import asyncio
import logging
from .config import configure_logging
from .config import parse_cli_settings
from .db import create_all
from .db import dispose_database_runtime
from .db import initialize_database_runtime
from .worker import run_worker_loop
logger = logging.getLogger(__name__)
async def _run() -> None:
settings = parse_cli_settings()
configure_logging(settings)
runtime = initialize_database_runtime(settings=settings)
if settings.should_bootstrap_schema:
await create_all(engine=runtime.engine)
try:
await run_worker_loop(
session_factory=runtime.session_factory,
poll_interval_seconds=settings.worker_poll_interval_seconds,
)
finally:
await dispose_database_runtime()
def main() -> None:
try:
asyncio.run(_run())
except KeyboardInterrupt:
logger.info("Worker service received shutdown signal")
if __name__ == "__main__":
main()
+30
View File
@@ -4,6 +4,7 @@ Every test gets a fresh in-memory SQLite database so tests are
isolated, fast, and leave no artifacts on disk.
"""
import os
from pathlib import Path
import pytest
@@ -24,6 +25,35 @@ from transcription.services.documents import DocumentService
from transcription.services.jobs import JobService
@pytest.fixture(autouse=True, scope="session")
def isolate_settings_from_local_env_files(tmp_path_factory):
"""Point `Settings` at a controlled stub env file instead of a developer one.
`Settings()` resolves its env file through the shared config seam, so a repository-root
pytest run would otherwise read a developer's real `.env.production` into tests that
assert declared defaults.
The stub mirrors what `.github/workflows/quality-gate.yml` writes in CI: only
`OPENROUTER_API_KEY`, which is required and which many tests need `get_settings()` to
find. Everything else falls back to declared defaults, so local and CI runs agree. This
stays a file rather than a process environment variable because
`test_config.py::test_requires_api_key` asserts the missing-key failure via
`_env_file=None`. Guarded by `tests/test_config_isolation.py`.
"""
stub = tmp_path_factory.mktemp("settings-env") / ".env.test"
stub.write_text("OPENROUTER_API_KEY=test-placeholder-not-a-real-key\n", encoding="utf-8")
original = os.environ.get("ENV_FILE")
os.environ["ENV_FILE"] = str(stub)
try:
yield
finally:
if original is None:
os.environ.pop("ENV_FILE", None)
else:
os.environ["ENV_FILE"] = original
@pytest.fixture
def session():
"""Provide a clean synchronous database session for sync tests."""
+42
View File
@@ -0,0 +1,42 @@
0 HEAD
1 SOUR getmyancestors
1 GEDC
2 VERS 5.5.1
1 CHAR UTF-8
0 @I1@ INDI
1 NAME John /Doe/
1 REFN KWC1-ABC
2 TYPE FSFTID
1 BIRT
2 DATE 1 JAN 1900
2 PLAC Springfield, Illinois
2 SOUR Birth Register
3 PAGE p. 12
1 DEAT
2 DATE 5 FEB 1970
2 PLAC Shelbyville, Illinois
2 SOUR Death Certificate
3 TEXT County archive
0 @I2@ INDI
1 NAME Jane /Smith/
1 _FSFTID LMN2-XYZ
1 BIRT
2 DATE 12 MAR 1905
2 PLAC Capital City, Illinois
0 @I3@ INDI
1 NAME Child /Doe/
1 REFN CHD3-123
2 TYPE FSFTID
0 @F1@ FAM
1 REFN FAM-001
2 TYPE FSFTID
1 HUSB @I1@
1 WIFE @I2@
1 CHIL @I3@
2 PEDI adopted
1 MARR
2 DATE 4 APR 1925
2 PLAC Springfield, Illinois
2 SOUR Marriage License
3 PAGE Book 9
0 TRLR
+23 -8
View File
@@ -24,7 +24,7 @@ from transcription.services.workflows import advance_job
async def _attempts_for_job(session, job) -> list[ExecutionAttempt]:
"""Load execution attempts for a job; V4.7 moved evidence off JobSource."""
"""Load execution attempts for a job; evidence lives on ExecutionAttempt."""
job_source_ids = [job_source.id for job_source in job.job_sources]
result = await session.exec(select(ExecutionAttempt).where(col(ExecutionAttempt.job_source_id).in_(job_source_ids)))
return list(result.all())
@@ -99,6 +99,7 @@ class TestPipelineSuccessFlow:
provider=None,
source_reference=None,
requested_model=None,
evidence_capture=None,
) -> TranscriptionResult:
_ = (
image_path,
@@ -110,6 +111,7 @@ class TestPipelineSuccessFlow:
provider,
source_reference,
requested_model,
evidence_capture,
)
return TranscriptionResult(
text="Pipeline transcript",
@@ -130,12 +132,10 @@ class TestPipelineSuccessFlow:
services = _build_services(default_session_factory)
queued_job = await services.jobs.read_job(job_id=upload_result.job_id, session=async_session)
processed = queued_job is not None
if queued_job is not None:
assert queued_job is not None
await advance_job(job=queued_job, services=services, settings=settings, session=async_session)
job = await services.jobs.read_job(job_id=upload_result.job_id, session=async_session)
assert processed is True
assert job is not None
assert job.status == JobStatus.TRANSCRIBED
attempts = await _attempts_for_job(async_session, job)
@@ -193,9 +193,20 @@ class TestPipelineSuccessFlow:
provider=None,
source_reference=None,
requested_model=None,
evidence_capture=None,
) -> TranscriptionResult:
page_name = Path(image_path).name
_ = (prompt_name, prompt_text, temperature, top_p, settings, provider, source_reference, requested_model)
_ = (
prompt_name,
prompt_text,
temperature,
top_p,
settings,
provider,
source_reference,
requested_model,
evidence_capture,
)
return TranscriptionResult(
text=f"Transcript for {page_name}",
provider="openrouter",
@@ -262,6 +273,7 @@ class TestPipelineSuccessFlow:
provider=None,
source_reference=None,
requested_model=None,
evidence_capture=None,
) -> TranscriptionResult:
nonlocal call_count
call_count += 1
@@ -275,6 +287,7 @@ class TestPipelineSuccessFlow:
provider,
source_reference,
requested_model,
evidence_capture,
)
if call_count == 2:
raise RuntimeError("simulated page failure")
@@ -353,6 +366,7 @@ class TestPipelineSuccessFlow:
provider=None,
source_reference=None,
requested_model=None,
evidence_capture=None,
) -> TranscriptionResult:
nonlocal call_count
_ = (
@@ -365,6 +379,7 @@ class TestPipelineSuccessFlow:
provider,
source_reference,
requested_model,
evidence_capture,
)
call_count += 1
return TranscriptionResult(
@@ -421,6 +436,7 @@ class TestPipelineFailureFlow:
provider=None,
source_reference=None,
requested_model=None,
evidence_capture=None,
) -> TranscriptionResult:
_ = (
image_path,
@@ -432,6 +448,7 @@ class TestPipelineFailureFlow:
provider,
source_reference,
requested_model,
evidence_capture,
)
raise RuntimeError("pipeline provider failure")
@@ -442,12 +459,10 @@ class TestPipelineFailureFlow:
services = _build_services(default_session_factory)
queued_job = await services.jobs.read_job(job_id=upload_result.job_id, session=async_session)
processed = queued_job is not None
if queued_job is not None:
assert queued_job is not None
await advance_job(job=queued_job, services=services, settings=settings, session=async_session)
job = await services.jobs.read_job(job_id=upload_result.job_id, session=async_session)
assert processed is True
assert job is not None
assert job.status == JobStatus.FAILED
attempts = await _attempts_for_job(async_session, job)
+97
View File
@@ -1,15 +1,21 @@
"""Tests for transcription.providers.openrouter."""
import asyncio
import json
import logging
from types import SimpleNamespace
from typing import cast
from uuid import uuid4
import httpx
import pytest
from openrouter import OpenRouter
from transcription.config import Settings
from transcription.providers.base import ProviderCallEvidence
from transcription.providers.base import ProviderError
from transcription.providers.base import ProviderResponseError
from transcription.providers.evidence import SourceEvidenceReference
from transcription.providers.openrouter import DEFAULT_OPENROUTER_MODEL
from transcription.providers.openrouter import OpenRouterTranscriptionProvider
@@ -226,3 +232,94 @@ class TestOpenRouterProviderTranscribe:
assert result.text == "Transcript text"
assert result.metadata_payload() is None
assert result.raw_api_response == response
@pytest.mark.asyncio
async def test_logs_when_request_manifest_is_omitted_without_source_reference(self, caplog):
response = {"model": "vendor/model-a", "choices": [{"message": {"content": "Transcript text"}}]}
provider = OpenRouterTranscriptionProvider(
settings=Settings(openrouter_api_key="test-key"),
client=_fake_client(response=response),
)
with caplog.at_level(logging.WARNING):
result = await provider.transcribe(
prompt_text="Prompt body",
image_bytes=b"img-bytes",
mime_type="image/png",
)
assert result.request_manifest is None
assert "request manifest omitted" in caplog.text.lower()
@pytest.mark.asyncio
async def test_concurrent_calls_keep_evidence_scoped_to_their_own_capture(self):
first_release = asyncio.Event()
async def handler(request: httpx.Request) -> httpx.Response:
payload = json.loads(request.content.decode("utf-8"))
prompt = payload["messages"][0]["content"][0]["text"]
if prompt == "First prompt":
await first_release.wait()
body = b'{"error":{"message":"first failure"}}'
else:
body = b'{"error":{"message":"second failure"}}'
return httpx.Response(
500,
content=body,
headers={"Content-Type": "application/json"},
request=request,
)
provider = OpenRouterTranscriptionProvider(
settings=Settings(openrouter_api_key="test-key"),
async_client=httpx.AsyncClient(transport=httpx.MockTransport(handler)),
)
first_capture = ProviderCallEvidence()
second_capture = ProviderCallEvidence()
first_task = asyncio.create_task(
provider.transcribe(
prompt_text="First prompt",
image_bytes=b"one",
mime_type="image/png",
evidence_capture=first_capture,
source_reference=SourceEvidenceReference(
source_id=uuid4(),
digest_sha256="1" * 64,
byte_size=3,
media_type="image/png",
page_number=1,
),
)
)
await asyncio.sleep(0)
second_task = asyncio.create_task(
provider.transcribe(
prompt_text="Second prompt",
image_bytes=b"two",
mime_type="image/png",
evidence_capture=second_capture,
source_reference=SourceEvidenceReference(
source_id=uuid4(),
digest_sha256="2" * 64,
byte_size=3,
media_type="image/png",
page_number=2,
),
)
)
with pytest.raises(ProviderError):
await second_task
first_release.set()
with pytest.raises(ProviderError):
await first_task
assert first_capture.request_manifest is not None
assert first_capture.request_manifest.prompt_content == "First prompt"
assert first_capture.transport_evidence is not None
assert first_capture.transport_evidence.body == b'{"error":{"message":"first failure"}}'
assert second_capture.request_manifest is not None
assert second_capture.request_manifest.prompt_content == "Second prompt"
assert second_capture.transport_evidence is not None
assert second_capture.transport_evidence.body == b'{"error":{"message":"second failure"}}'
@@ -1,4 +1,4 @@
"""V4.5 retranscription candidate and promotion tests."""
"""Retranscription candidate and promotion tests."""
from uuid import uuid4
@@ -20,7 +20,7 @@ def _services(default_session_factory, settings: Settings) -> ServiceBundle:
async def _seed_source(services: ServiceBundle) -> Source:
document = await services.documents.create_document(Document(name="V4.5 source"))
document = await services.documents.create_document(Document(name="candidate source"))
source = Source(
document_id=document.id,
page_number=1,
+69
View File
@@ -0,0 +1,69 @@
from __future__ import annotations
from datetime import date
from pathlib import Path
import pytest
from sqlmodel import select
from transcription.db.models import GenealogyCitation
from transcription.db.models import GenealogyFamily
from transcription.db.models import GenealogyFamilyChild
from transcription.db.models import GenealogyPerson
from transcription.services.gedcom_import import import_gedcom_file
from transcription.services.gedcom_import import parse_gedcom
_FIXTURE_PATH = Path(__file__).resolve().parents[1] / "fixtures" / "gedcom" / "sample_familysearch.ged"
def test_parse_gedcom_extracts_people_families_and_citations():
parsed = parse_gedcom(file_path=_FIXTURE_PATH)
assert len(parsed.people) == 3
john = next(person for person in parsed.people if person.fs_id == "KWC1-ABC")
assert john.full_name == "John Doe"
assert john.birth_date == date(1900, 1, 1)
assert john.birth_place == "Springfield, Illinois"
assert john.death_date == date(1970, 2, 5)
assert len(john.citations) == 2
assert "Birth Register" in john.citations[0].raw_citation_text
assert len(parsed.families) == 1
family = parsed.families[0]
assert family.fs_family_id == "FAM-001"
assert family.marriage_date == date(1925, 4, 4)
assert family.marriage_place == "Springfield, Illinois"
assert len(family.children) == 1
assert family.children[0].relationship_type == "adopted"
assert len(family.citations) == 1
assert "Marriage License" in family.citations[0].raw_citation_text
@pytest.mark.asyncio
async def test_import_gedcom_file_is_idempotent(default_session_factory):
async with default_session_factory() as session:
first = await import_gedcom_file(session=session, file_path=_FIXTURE_PATH)
async with default_session_factory() as session:
second = await import_gedcom_file(session=session, file_path=_FIXTURE_PATH)
async with default_session_factory() as session:
people = (await session.exec(select(GenealogyPerson))).all()
families = (await session.exec(select(GenealogyFamily))).all()
children = (await session.exec(select(GenealogyFamilyChild))).all()
citations = (await session.exec(select(GenealogyCitation))).all()
assert first.new_people == 3
assert first.new_families == 1
assert first.family_children == 1
assert first.citations == 3
assert second.new_people == 0
assert second.updated_people == 0
assert second.new_families == 0
assert second.updated_families == 0
assert second.family_children == 1
assert second.citations == 3
assert len(people) == 3
assert len(families) == 1
assert len(children) == 1
assert len(citations) == 3
+15
View File
@@ -20,9 +20,24 @@ from transcription.services.jobs import JobDeleteBlockedError
from transcription.services.jobs import JobNotFoundError
from transcription.services.jobs import JobResubmitBlockedError
from transcription.services.jobs import JobService
from transcription.services.jobs import _as_naive_utc
from transcription.services.jobs import _utc_now_naive
from transcription.services.sources import SourceService
def test_utc_now_naive_returns_naive_datetime():
now = _utc_now_naive()
assert now.tzinfo is None
def test_as_naive_utc_normalizes_timezone_aware_datetime():
aware = datetime.now(UTC)
normalized = _as_naive_utc(aware)
assert normalized.tzinfo is None
assert normalized == aware.replace(tzinfo=None)
class TestJobService:
@pytest.mark.asyncio
async def test_create_and_read_job(self, job_service: JobService, document_service: DocumentService):
@@ -0,0 +1,93 @@
"""Tests for maintenance run persistence and execution lifecycle."""
from __future__ import annotations
from pathlib import Path
import pytest
from sqlalchemy import text
from transcription.config import Settings
from transcription.db.models import MaintenanceJobType
from transcription.db.models import MaintenanceRunStatus
from transcription.errors import ErrorCategory
from transcription.services.maintenance import MaintenanceError
from transcription.services.maintenance import MaintenanceExecution
from transcription.services.maintenance import MaintenanceService
@pytest.mark.asyncio
async def test_enqueue_and_list_runs(default_session_factory, default_settings):
service = MaintenanceService(session_factory=default_session_factory, settings=default_settings)
first = await service.enqueue_run(job_type=MaintenanceJobType.BACKUP, triggered_by="test")
second = await service.enqueue_run(job_type=MaintenanceJobType.STORAGE_RECONCILIATION, triggered_by="test")
third = await service.enqueue_run(job_type=MaintenanceJobType.GEDCOM_IMPORT, triggered_by="test")
runs = await service.list_runs(limit=10)
assert len(runs) == 3
assert runs[0].id == third.id
assert runs[1].id == second.id
assert runs[2].id == first.id
assert runs[0].status == MaintenanceRunStatus.QUEUED
@pytest.mark.asyncio
async def test_process_next_queued_run_persists_terminal_result(
default_session_factory,
default_settings,
tmp_path,
monkeypatch,
):
settings = default_settings.model_copy(update={"log_dir": tmp_path / "logs"})
settings = Settings.model_validate(settings.model_dump())
service = MaintenanceService(session_factory=default_session_factory, settings=settings)
queued = await service.enqueue_run(job_type=MaintenanceJobType.BACKUP, triggered_by="test")
async def _fake_execute(_run):
return MaintenanceExecution(
status=MaintenanceRunStatus.SUCCEEDED,
summary="Synthetic success",
output="stdout line\nstderr line",
)
monkeypatch.setattr(service, "_execute_run", _fake_execute)
processed = await service.process_next_queued_run()
assert processed is True
runs = await service.list_runs(limit=10)
updated = next(run for run in runs if run.id == queued.id)
assert updated.status == MaintenanceRunStatus.SUCCEEDED
assert updated.summary == "Synthetic success"
assert updated.log_path is not None
assert (settings.log_dir / Path(updated.log_path)).is_file()
@pytest.mark.asyncio
async def test_list_runs_raises_when_table_is_missing(default_session_factory, default_settings):
service = MaintenanceService(session_factory=default_session_factory, settings=default_settings)
async with default_session_factory() as session:
await session.exec(text("DROP TABLE maintenance_run"))
await session.commit()
with pytest.raises(MaintenanceError) as exc:
await service.list_runs(limit=10)
assert exc.value.category == ErrorCategory.INFRA_PERSISTENT
@pytest.mark.asyncio
async def test_execute_run_dispatches_gedcom_import(default_session_factory, default_settings, monkeypatch):
service = MaintenanceService(session_factory=default_session_factory, settings=default_settings)
run = await service.enqueue_run(job_type=MaintenanceJobType.GEDCOM_IMPORT, triggered_by="test")
async def _fake_gedcom_import():
return MaintenanceExecution(
status=MaintenanceRunStatus.SUCCEEDED,
summary="GEDCOM imported",
output="ok",
)
monkeypatch.setattr(service, "_execute_gedcom_import", _fake_gedcom_import)
execution = await service._execute_run(run)
assert execution.summary == "GEDCOM imported"
+8
View File
@@ -4,6 +4,7 @@ import pytest
from transcription.config import Settings
from transcription.errors import ErrorCategory
from transcription.services.errors import PromptLoadError
from transcription.services.prompts import PromptStore
from transcription.services.prompts import PromptStoreError
@@ -79,6 +80,13 @@ def test_prompt_creation_and_empty_content_are_rejected(prompt_store):
assert empty.value.category == ErrorCategory.VALIDATION
def test_prompt_store_failures_are_catchable_as_prompt_load_errors(prompt_store):
store, _ = prompt_store
with pytest.raises(PromptLoadError):
store.read_prompt("missing.md")
def test_recovery_requires_a_backup(prompt_store):
store, _ = prompt_store
+2 -2
View File
@@ -1,4 +1,4 @@
"""Tests for deterministic V4.5 transcription warnings."""
"""Tests for deterministic transcription warnings."""
import pytest
@@ -8,7 +8,7 @@ from transcription.services.quality import quality_warning_payload
@pytest.mark.unit
def test_quality_analysis_reports_each_v45_warning_without_mutating_text():
def test_quality_analysis_reports_each_quality_warning_without_mutating_text():
text = (
"[document body handwritten]\n"
"[document body typewritten]\n"
+65 -5
View File
@@ -86,6 +86,7 @@ class TestWorkflowReliability:
provider=None,
source_reference=None,
requested_model=None,
evidence_capture=None,
):
_ = (
image_path,
@@ -97,6 +98,7 @@ class TestWorkflowReliability:
provider,
source_reference,
requested_model,
evidence_capture,
)
raise TimeoutError("simulated provider timeout")
@@ -154,8 +156,8 @@ class TestWorkflowReliability:
await session.commit()
loaded = await services.jobs.read_job(job_id=job.id, session=session)
setup_seconds = 0.40
budget_seconds = 0.20
setup_seconds = 0.25
budget_seconds = 0.15
real_build = workflows_module.build_provider_input
def _slow_build(source_arg, **kwargs):
@@ -188,9 +190,9 @@ class TestWorkflowReliability:
assert len(attempts) == 1
duration_ms = attempts[0].duration_ms
# At or just above the budget, and well clear of budget + setup.
assert duration_ms >= int(budget_seconds * 1000 * 0.9)
assert duration_ms < int((budget_seconds + setup_seconds) * 1000 * 0.9)
# At or above the timeout budget, and still well below setup + timeout.
assert duration_ms >= int(budget_seconds * 1000 * 0.7)
assert duration_ms < int((budget_seconds + setup_seconds) * 1000 * 0.75)
@pytest.mark.asyncio
async def test_attempt_metadata_persists_provider_and_processing_durations(
@@ -370,6 +372,64 @@ class TestWorkflowReliability:
assert result is not None
assert result.status == JobStatus.TRANSCRIBED
@pytest.mark.asyncio
async def test_intermediate_page_commit_advances_job_liveness_timestamp(
self,
default_session_factory,
monkeypatch,
):
services = ServiceBundle.from_session_factory(default_session_factory)
async with services.jobs._session_scope() as session:
document = Document(id=uuid4(), name="heartbeat-doc")
session.add(document)
await session.flush()
job = Job(document_id=document.id, status=JobStatus.QUEUED)
session.add(job)
await session.flush()
for page_number in (1, 2, 3):
source = Source(
document_id=document.id,
page_number=page_number,
upload_name=f"page-{page_number}.jpg",
filename=f"page-{page_number}.jpg",
file_path=str(Path("tests/fixtures/images/real/Book Two - page 02.jpg")),
file_hash=str(page_number) * 64,
file_size_bytes=1,
)
session.add(source)
await session.flush()
session.add(JobSource(job_id=job.id, source_id=source.id))
await session.commit()
loaded = await services.jobs.read_job(job_id=job.id, session=session)
third_started = asyncio.Event()
release_third = asyncio.Event()
call_count = 0
async def _transcribe(image_path, **kwargs):
nonlocal call_count
_ = (image_path, kwargs)
call_count += 1
if call_count == 3:
third_started.set()
await release_third.wait()
return TranscriptionResult(text=f"page {call_count}", provider="fixture", model="model")
monkeypatch.setattr("transcription.services.workflows.transcribe_document_image", _transcribe)
task = asyncio.create_task(process_queued_job(job=loaded, services=services))
await asyncio.wait_for(third_started.wait(), timeout=2)
async with services.jobs._session_scope() as session:
current_job = await session.get(Job, loaded.id)
assert current_job is not None
attempts = await services.evidence.list_execution_attempts(job_id=loaded.id, session=session)
assert len(attempts) == 2
assert current_job.date_updated >= attempts[1].finished_at
release_third.set()
await task
@pytest.mark.asyncio
async def test_failed_job_with_validation_category_is_not_requeued(self, default_session_factory):
services = ServiceBundle.from_session_factory(default_session_factory)

Some files were not shown because too many files have changed in this diff Show More