Review log [8]. classify_unexpected_error already returned retriable=False and
the verdict was logged and then thrown away. Measured across src/: retriable
was assigned in 9 places and read in none.
The plan asks for a test that a programming error "does not silently retry".
Probing with an injected AttributeError showed that is not what happens, and
the two real failure modes need different fixes.
Mode A, raised after the claim commits (inside advance_job): raised exactly
once, job left at PROCESSING, retry_count 0, never re-claimed, because
claim_next_queued_job filters status == QUEUED. A permanently stranded job
with one swallowed log line, not a retry. advance_job's PROCESSING branch,
commented "Recover mid-flight jobs", is unreachable from the worker for the
same reason.
Mode B, raised before or during the claim: 20 raises in 1.2s, an unbounded hot
spin at the poll interval. It never reaches the per-job retry machinery, so
WORKER_MAX_RETRIES does not cap it and the plan's 60s worst case understates
this path.
services/workflows.py
_advance_job_with_containment wraps advance_job. Any escaping exception is
classified and the job driven to terminal FAILED, which is visible in the UI
and resubmittable. The caller session is rolled back first and the terminal
write runs in its own transaction, so it stays atomic even when the failure
left that session dirty (plan task 3). The loop continues, so one poison job
cannot halt transcription for every other job.
worker.py
handle_worker_exceptions re-raises non-retriable faults rather than
suppressing them; retriable ones are still suppressed so transient
conditions do not stop work. run_worker_loop catches that, logs CRITICAL and
returns cleanly. Returning rather than propagating matters: the exception
would otherwise surface only at app shutdown, through the wait_for in
worker_consumer_lifespan.
tests
test_run_worker_loop_survives_process_next_exception asserted the loop
SURVIVES a RuntimeError and continues, which is the Mode B defect written
down as an expectation. Replaced by
test_run_worker_loop_stops_on_non_retriable_exception, with a new
test_run_worker_loop_survives_retriable_exception so suppression of genuinely
transient faults stays covered, and
test_error_after_claim_fails_the_job_instead_of_stranding_it for Mode A.
All three were verified to fail on pre-fix code. The Mode B guard fails by
timing out, which is the infinite spin made visible.
Verified: 295 passed, 4 skipped, 0 ruff, 0 ty.
Co-authored-by: Copilot App <[email protected]>
Baseline was 207 diagnostics. Two real bugs were hiding in the noise:
- tools/run_destructive_tests.py imported ctypes.wintypes at module scope,
which raises on non-Windows, and called fcntl unconditionally. The Windows
and POSIX implementations now live under a module-level sys.platform split.
- tests/ui/test_sources_page.py constructed Source(...) without document_id.
Structural fixes, not suppressions:
- New src/transcription/db/loading.py owns the SQLModel-field to
QueryableAttribute reinterpretation via orm_attribute()/selectinload()/
defer(). This removed 42 "# pyright: ignore[reportArgumentType]" comments
across documents/jobs/people/sources. Its docstring records that
selectinload(A.b, B.c) is NOT equivalent to the chained form: varargs
applies the selectin strategy only to the last path element, which under
lazy="raise" raises InvalidRequestError at render time.
- db/session.py transaction_scope no longer accepts or yields
AsyncSessionTransaction. No caller ever passed one, sessionmaker.begin()
yields an AsyncSession, and the dead branch was latently buggy because
services call .exec(). Cleared 7 workflows.py diagnostics.
- services/registry.py RegistryService is bound by a new RegistryEntry
Protocol instead of bare SQLModel, so the shared implementation can read
id/label/normalized_label/is_active. Cleared 9 diagnostics.
- Column expressions in sources.py/jobs.py/test_store.py wrap in sqlmodel
col(), the idiom already used in registry.py.
- read_source_navigation wraps its literal tuple bounds in literal().
- normalization.py narrows with isinstance(image, TiffImageFile) rather than
comparing image.format, since tag_v2 is TIFF-only.
- linked_people.render uses @ui.refreshable_method, the NiceGUI API for bound
methods.
- The OpenRouter capturing client re-raises ResponseNotRead when the response
stream is not async rather than mis-wrapping it.
Tooling gate:
- New .pre-commit-config.yaml runs ruff check and ty check as blocking hooks.
No pre-commit config previously existed. Negative-tested: injecting a type
error fails both hooks.
- The last two "# pyright: ignore" comments (config.py) are removed; ty does
not honor pyright directives. One "# ty: ignore" remains, in
tests/test_prompts.py, where the test deliberately assigns to a frozen
field to assert ValidationError.
- asyncio_default_fixture_loop_scope is pinned to "function" so
pytest-asyncio behavior does not shift on upgrade.
Verification: ruff check clean, ty check reports 0 diagnostics, 292 passed
and 4 skipped, pre-commit passes and demonstrably fails on a regression, and
tools/run_destructive_tests.py runs on Windows.
Co-authored-by: Copilot App <[email protected]>
Claim jobs atomically [CRIT-01]
- Replace JobService.read_next_queued_job with claim_next_queued_job, which
selects and transitions QUEUED -> PROCESSING inside one transaction. The old
read-then-write sequence left a window in which two workers could observe the
same QUEUED row.
- Add the missing .limit(1). The poll previously ordered the entire queued set
and discarded all but the first row.
- Drop the eager loads from the hot poll entirely. They were pure waste:
process_queued_job immediately re-reads the job through read_job with the
relationships it actually needs.
- Guard the row with with_for_update(skip_locked=True) on PostgreSQL so the
claim stays correct once more than one worker exists. On SQLite the claim is a
bounded single-writer transaction.
- Correct the comment at the remaining direct-call claim site, which described
the hazard rather than the guarantee.
Reuse the provider connection [HIGH-02]
- Build the ServiceBundle once per worker loop instead of once per job, and
close it at loop shutdown. Every job previously constructed a new
SourceService, and with it a new provider adapter and a new httpx.AsyncClient,
paying a full TLS handshake per page and discarding the connection pool.
- process_next_queued_job now accepts an optional caller-owned bundle and only
closes bundles it created itself.
Uncap the provider timeout [HIGH-03]
- Remove le=20.0 from worker_provider_timeout_seconds. The cap equalled the
default, so the ceiling could never be raised, and dense-page vision
transcription routinely needs longer. Default raised to 180s.
- Pass an explicit httpx.Timeout to the OpenRouter AsyncClient. httpx defaults
every phase to 5 seconds, so the real read budget was 5s regardless of the
configured value; the outer asyncio.wait_for could never be the binding
constraint. Connect stays at 10s.
Tighten the provider boundary [MED-03]
- Declare model, current_request_manifest, current_transport_evidence, and
aclose on the TranscriptionProvider Protocol.
- Delete the per-call inspect.signature(adapter.transcribe).parameters
reflection and the untyped kwargs dict it fed. The Protocol had declared
requested_model all along, so the reflection was dead defensive weight on the
hot path.
- Replace the three getattr probes for aclose and the evidence attributes with
direct typed access.
Deduplicate bundle construction [MED-06]
- Add ServiceBundle.from_session_factory and ServiceBundle.aclose, replacing the
duplicated four-service instantiation blocks in app.py and worker.py.
- _recover_stale_processing_jobs now uses the bundle built moments earlier
instead of constructing a second JobService.
Tests
- Claiming returns the oldest job, marks it PROCESSING, never hands the same job
out twice, and emits exactly one unadorned SELECT carrying LIMIT and no JOIN.
- The worker loop threads one bundle through consecutive jobs and closes it once
at shutdown; a caller-owned bundle is left open.
- Settings accepts a timeout above 20 seconds and still rejects zero.
- The OpenRouter client's read, write, and pool timeouts track the configured
budget rather than the httpx default.
Note: .env in this checkout still pins WORKER_PROVIDER_TIMEOUT_SECONDS=20 and
should be raised to pick up this fix.
Verified: 268 passed, 4 skipped; ruff check clean.
Co-authored-by: Copilot App <[email protected]>
Jim LancasterandCopilot App