generated from john/python-template
V4.6 Phase 3: worker and provider reliability
Claim jobs atomically [CRIT-01] - Replace JobService.read_next_queued_job with claim_next_queued_job, which selects and transitions QUEUED -> PROCESSING inside one transaction. The old read-then-write sequence left a window in which two workers could observe the same QUEUED row. - Add the missing .limit(1). The poll previously ordered the entire queued set and discarded all but the first row. - Drop the eager loads from the hot poll entirely. They were pure waste: process_queued_job immediately re-reads the job through read_job with the relationships it actually needs. - Guard the row with with_for_update(skip_locked=True) on PostgreSQL so the claim stays correct once more than one worker exists. On SQLite the claim is a bounded single-writer transaction. - Correct the comment at the remaining direct-call claim site, which described the hazard rather than the guarantee. Reuse the provider connection [HIGH-02] - Build the ServiceBundle once per worker loop instead of once per job, and close it at loop shutdown. Every job previously constructed a new SourceService, and with it a new provider adapter and a new httpx.AsyncClient, paying a full TLS handshake per page and discarding the connection pool. - process_next_queued_job now accepts an optional caller-owned bundle and only closes bundles it created itself. Uncap the provider timeout [HIGH-03] - Remove le=20.0 from worker_provider_timeout_seconds. The cap equalled the default, so the ceiling could never be raised, and dense-page vision transcription routinely needs longer. Default raised to 180s. - Pass an explicit httpx.Timeout to the OpenRouter AsyncClient. httpx defaults every phase to 5 seconds, so the real read budget was 5s regardless of the configured value; the outer asyncio.wait_for could never be the binding constraint. Connect stays at 10s. Tighten the provider boundary [MED-03] - Declare model, current_request_manifest, current_transport_evidence, and aclose on the TranscriptionProvider Protocol. - Delete the per-call inspect.signature(adapter.transcribe).parameters reflection and the untyped kwargs dict it fed. The Protocol had declared requested_model all along, so the reflection was dead defensive weight on the hot path. - Replace the three getattr probes for aclose and the evidence attributes with direct typed access. Deduplicate bundle construction [MED-06] - Add ServiceBundle.from_session_factory and ServiceBundle.aclose, replacing the duplicated four-service instantiation blocks in app.py and worker.py. - _recover_stale_processing_jobs now uses the bundle built moments earlier instead of constructing a second JobService. Tests - Claiming returns the oldest job, marks it PROCESSING, never hands the same job out twice, and emits exactly one unadorned SELECT carrying LIMIT and no JOIN. - The worker loop threads one bundle through consecutive jobs and closes it once at shutdown; a caller-owned bundle is left open. - Settings accepts a timeout above 20 seconds and still rejects zero. - The OpenRouter client's read, write, and pool timeouts track the configured budget rather than the httpx default. Note: .env in this checkout still pins WORKER_PROVIDER_TIMEOUT_SECONDS=20 and should be raised to pick up this fix. Verified: 268 passed, 4 skipped; ruff check clean. Co-authored-by: Copilot App <[email protected]>
This commit is contained in:
@@ -149,3 +149,28 @@ class TestWorkerReliabilitySettings:
|
||||
"""worker retry settings default to no retries."""
|
||||
settings = _make_settings()
|
||||
assert settings.worker_max_retries == 0
|
||||
|
||||
|
||||
def test_provider_timeout_is_not_capped_at_twenty_seconds():
|
||||
"""HIGH-03: vision transcription regularly runs past the old le=20.0 ceiling."""
|
||||
settings = Settings(openrouter_api_key="test-key", worker_provider_timeout_seconds=300.0)
|
||||
assert settings.worker_provider_timeout_seconds == 300.0
|
||||
|
||||
|
||||
def test_provider_timeout_must_still_be_positive():
|
||||
with pytest.raises(ValidationError):
|
||||
Settings(openrouter_api_key="test-key", worker_provider_timeout_seconds=0.0)
|
||||
|
||||
|
||||
def test_openrouter_client_timeout_tracks_the_configured_budget():
|
||||
"""HIGH-03: httpx defaults every phase to 5s, silently capping the provider call."""
|
||||
from transcription.providers.openrouter import OpenRouterTranscriptionProvider
|
||||
|
||||
settings = Settings(openrouter_api_key="test-key", worker_provider_timeout_seconds=123.0)
|
||||
provider = OpenRouterTranscriptionProvider(settings=settings)
|
||||
timeout = provider._capturing_client._client.timeout
|
||||
|
||||
assert timeout.read == 123.0
|
||||
assert timeout.write == 123.0
|
||||
assert timeout.pool == 123.0
|
||||
assert timeout.connect == 10.0
|
||||
|
||||
Reference in New Issue
Block a user