V4.6 Phase 3: worker and provider reliability

Claim jobs atomically [CRIT-01]
- Replace JobService.read_next_queued_job with claim_next_queued_job, which
  selects and transitions QUEUED -> PROCESSING inside one transaction. The old
  read-then-write sequence left a window in which two workers could observe the
  same QUEUED row.
- Add the missing .limit(1). The poll previously ordered the entire queued set
  and discarded all but the first row.
- Drop the eager loads from the hot poll entirely. They were pure waste:
  process_queued_job immediately re-reads the job through read_job with the
  relationships it actually needs.
- Guard the row with with_for_update(skip_locked=True) on PostgreSQL so the
  claim stays correct once more than one worker exists. On SQLite the claim is a
  bounded single-writer transaction.
- Correct the comment at the remaining direct-call claim site, which described
  the hazard rather than the guarantee.

Reuse the provider connection [HIGH-02]
- Build the ServiceBundle once per worker loop instead of once per job, and
  close it at loop shutdown. Every job previously constructed a new
  SourceService, and with it a new provider adapter and a new httpx.AsyncClient,
  paying a full TLS handshake per page and discarding the connection pool.
- process_next_queued_job now accepts an optional caller-owned bundle and only
  closes bundles it created itself.

Uncap the provider timeout [HIGH-03]
- Remove le=20.0 from worker_provider_timeout_seconds. The cap equalled the
  default, so the ceiling could never be raised, and dense-page vision
  transcription routinely needs longer. Default raised to 180s.
- Pass an explicit httpx.Timeout to the OpenRouter AsyncClient. httpx defaults
  every phase to 5 seconds, so the real read budget was 5s regardless of the
  configured value; the outer asyncio.wait_for could never be the binding
  constraint. Connect stays at 10s.

Tighten the provider boundary [MED-03]
- Declare model, current_request_manifest, current_transport_evidence, and
  aclose on the TranscriptionProvider Protocol.
- Delete the per-call inspect.signature(adapter.transcribe).parameters
  reflection and the untyped kwargs dict it fed. The Protocol had declared
  requested_model all along, so the reflection was dead defensive weight on the
  hot path.
- Replace the three getattr probes for aclose and the evidence attributes with
  direct typed access.

Deduplicate bundle construction [MED-06]
- Add ServiceBundle.from_session_factory and ServiceBundle.aclose, replacing the
  duplicated four-service instantiation blocks in app.py and worker.py.
- _recover_stale_processing_jobs now uses the bundle built moments earlier
  instead of constructing a second JobService.

Tests
- Claiming returns the oldest job, marks it PROCESSING, never hands the same job
  out twice, and emits exactly one unadorned SELECT carrying LIMIT and no JOIN.
- The worker loop threads one bundle through consecutive jobs and closes it once
  at shutdown; a caller-owned bundle is left open.
- Settings accepts a timeout above 20 seconds and still rejects zero.
- The OpenRouter client's read, write, and pool timeouts track the configured
  budget rather than the httpx default.

Note: .env in this checkout still pins WORKER_PROVIDER_TIMEOUT_SECONDS=20 and
should be raised to pick up this fix.

Verified: 268 passed, 4 skipped; ruff check clean.

Co-authored-by: Copilot App <[email protected]>
This commit is contained in:
zoltan57
2026-08-17 16:26:31 -05:00
co-authored by Copilot App
parent 3e418a0889
commit 7b9715b3f1
13 changed files with 287 additions and 98 deletions
+38 -4
View File
@@ -4,6 +4,7 @@ from datetime import timedelta
from uuid import uuid4
import pytest
from sqlalchemy import event
from transcription.db.models import Document
from transcription.db.models import Job
@@ -101,7 +102,7 @@ class TestJobService:
assert result[0].id == job.id
@pytest.mark.asyncio
async def test_read_next_queued_job_orders_by_created_date(
async def test_claim_next_queued_job_claims_oldest_and_marks_processing(
self,
job_service: JobService,
document_service: DocumentService,
@@ -119,9 +120,42 @@ class TestJobService:
await job_service.create_job(job=first)
await job_service.create_job(job=second)
next_job = await job_service.read_next_queued_job()
assert next_job is not None
assert next_job.id == first.id
claimed = await job_service.claim_next_queued_job()
assert claimed is not None
assert claimed.id == first.id
assert claimed.status == JobStatus.PROCESSING
# The claim is exclusive: the same job is never handed out twice.
next_claim = await job_service.claim_next_queued_job()
assert next_claim is not None
assert next_claim.id == second.id
assert await job_service.claim_next_queued_job() is None
@pytest.mark.asyncio
async def test_claim_next_queued_job_emits_a_bounded_unadorned_query(
self,
job_service: JobService,
):
"""CRIT-01: the hot poll must not select a subgraph or scan the queue."""
statements: list[str] = []
async with job_service._session_scope() as session:
bind = session.get_bind()
def capture(_conn, _cursor, statement, *_rest):
statements.append(statement)
event.listen(bind, "before_cursor_execute", capture)
try:
await job_service.claim_next_queued_job(session=session)
finally:
event.remove(bind, "before_cursor_execute", capture)
selects = [item for item in statements if item.lstrip().upper().startswith("SELECT")]
assert len(selects) == 1, selects
assert "LIMIT" in selects[0].upper()
assert "JOIN" not in selects[0].upper()
@pytest.mark.asyncio
async def test_create_job_persists_provider_and_model(