bbchops/transcription
1
0
Fork 0
generated from john/python-template
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

V4.6 Phase 3: worker and provider reliability

Browse Source 
Claim jobs atomically [CRIT-01]
- Replace JobService.read_next_queued_job with claim_next_queued_job, which
  selects and transitions QUEUED -> PROCESSING inside one transaction. The old
  read-then-write sequence left a window in which two workers could observe the
  same QUEUED row.
- Add the missing .limit(1). The poll previously ordered the entire queued set
  and discarded all but the first row.
- Drop the eager loads from the hot poll entirely. They were pure waste:
  process_queued_job immediately re-reads the job through read_job with the
  relationships it actually needs.
- Guard the row with with_for_update(skip_locked=True) on PostgreSQL so the
  claim stays correct once more than one worker exists. On SQLite the claim is a
  bounded single-writer transaction.
- Correct the comment at the remaining direct-call claim site, which described
  the hazard rather than the guarantee.

Reuse the provider connection [HIGH-02]
- Build the ServiceBundle once per worker loop instead of once per job, and
  close it at loop shutdown. Every job previously constructed a new
  SourceService, and with it a new provider adapter and a new httpx.AsyncClient,
  paying a full TLS handshake per page and discarding the connection pool.
- process_next_queued_job now accepts an optional caller-owned bundle and only
  closes bundles it created itself.

Uncap the provider timeout [HIGH-03]
- Remove le=20.0 from worker_provider_timeout_seconds. The cap equalled the
  default, so the ceiling could never be raised, and dense-page vision
  transcription routinely needs longer. Default raised to 180s.
- Pass an explicit httpx.Timeout to the OpenRouter AsyncClient. httpx defaults
  every phase to 5 seconds, so the real read budget was 5s regardless of the
  configured value; the outer asyncio.wait_for could never be the binding
  constraint. Connect stays at 10s.

Tighten the provider boundary [MED-03]
- Declare model, current_request_manifest, current_transport_evidence, and
  aclose on the TranscriptionProvider Protocol.
- Delete the per-call inspect.signature(adapter.transcribe).parameters
  reflection and the untyped kwargs dict it fed. The Protocol had declared
  requested_model all along, so the reflection was dead defensive weight on the
  hot path.
- Replace the three getattr probes for aclose and the evidence attributes with
  direct typed access.

Deduplicate bundle construction [MED-06]
- Add ServiceBundle.from_session_factory and ServiceBundle.aclose, replacing the
  duplicated four-service instantiation blocks in app.py and worker.py.
- _recover_stale_processing_jobs now uses the bundle built moments earlier
  instead of constructing a second JobService.

Tests
- Claiming returns the oldest job, marks it PROCESSING, never hands the same job
  out twice, and emits exactly one unadorned SELECT carrying LIMIT and no JOIN.
- The worker loop threads one bundle through consecutive jobs and closes it once
  at shutdown; a caller-owned bundle is left open.
- Settings accepts a timeout above 20 seconds and still rejects zero.
- The OpenRouter client's read, write, and pool timeouts track the configured
  budget rather than the httpx default.

Note: .env in this checkout still pins WORKER_PROVIDER_TIMEOUT_SECONDS=20 and
should be raised to pick up this fix.

Verified: 268 passed, 4 skipped; ruff check clean.

Co-authored-by: Copilot App <[email protected]>
This commit is contained in:
zoltan57
2026-08-17 16:26:31 -05:00
co-authored by Copilot App
parent 3e418a0889
commit 7b9715b3f1
13 changed files with 287 additions and 98 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/transcription/app.py
+2 -12
View File
@@ -25,10 +25,6 @@ from .db import create_all
from .db import dispose_database_runtime
from .db import initialize_database_runtime
from .services import ServiceBundle
from .services.documents import DocumentService
from .services.jobs import JobService
from .services.people import PeopleService
from .services.sources import SourceService
from .ui import register_pages
from .worker import worker_consumer_lifespan
@@ -42,12 +38,7 @@ async def _lifespan(app: FastAPI):
app.state.settings = settings
app.state.runtime = initialize_database_runtime(settings=settings)
session_factory = app.state.runtime.session_factory
app.state.services = ServiceBundle(
documents=DocumentService(session_factory=session_factory, settings=settings),
sources=SourceService(session_factory=session_factory, settings=settings),
jobs=JobService(session_factory=session_factory, settings=settings),
people=PeopleService(session_factory=session_factory, settings=settings),
)
app.state.services = ServiceBundle.from_session_factory(session_factory, settings=settings)
if settings.should_bootstrap_schema:
await create_all(engine=app.state.runtime.engine)
@@ -78,8 +69,7 @@ async def _recover_stale_processing_jobs(app: FastAPI) -> None:
"""
settings = app.state.settings
stale_before = datetime.now(UTC) - timedelta(seconds=settings.worker_provider_timeout_seconds)
job_service = JobService(session_factory=app.state.runtime.session_factory)
recovered = await job_service.requeue_stale_processing_jobs(stale_before=stale_before)
recovered = await app.state.services.jobs.requeue_stale_processing_jobs(stale_before=stale_before)
if recovered > 0:
logger.warning("Recovered %s stale processing job(s) at startup", recovered)
src/transcription/config.py
+3 -1
View File
@@ -106,7 +106,9 @@ class Settings(BaseSettings):
# --- worker reliability ---
worker_max_retries: int = Field(default=0, ge=0)
worker_provider_timeout_seconds: float = Field(default=20.0, gt=0.0, le=20.0)
# Bounded only from below. Vision transcription of a dense page routinely runs
# well past twenty seconds, so an upper cap here would silently fail real work.
worker_provider_timeout_seconds: float = Field(default=180.0, gt=0.0)
worker_min_transcription_chars: int = Field(default=0, ge=0)
worker_min_transcription_lines: int = Field(default=0, ge=0)
worker_fail_on_finish_reason_length: bool = False
src/transcription/providers/base.py
+19
View File
@@ -102,6 +102,21 @@ class TranscriptionResult(BaseModel):
class TranscriptionProvider(Protocol):
"""Contract every transcription provider adapter must satisfy."""
@property
def model(self) -> str:
"""Return the resolved model slug this adapter will call."""
...
@property
def current_request_manifest(self) -> RequestManifest | None:
"""Return the manifest for the most recent call, for failure evidence."""
...
@property
def current_transport_evidence(self) -> TransportEvidence | None:
"""Return transport-level evidence for the most recent call."""
...
async def transcribe(
self,
*,
@@ -115,3 +130,7 @@ class TranscriptionProvider(Protocol):
) -> TranscriptionResult:
"""Transcribe the provided image according to the prompt text."""
...
async def aclose(self) -> None:
"""Release any pooled network resources held by the adapter."""
...
src/transcription/providers/openrouter.py
+9 -1
View File
@@ -195,7 +195,15 @@ class OpenRouterTranscriptionProvider:
self._current_request_manifest: RequestManifest | None = None
self._current_transport_evidence: TransportEvidence | None = None
if client is None:
self._capturing_client = _CapturingAsyncClient(async_client or httpx.AsyncClient(follow_redirects=True))
# httpx defaults every phase to 5s, which silently caps provider calls far
# below worker_provider_timeout_seconds. Track the configured budget instead.
timeout = httpx.Timeout(
self._settings.worker_provider_timeout_seconds,
connect=10.0,
)
self._capturing_client = _CapturingAsyncClient(
async_client or httpx.AsyncClient(follow_redirects=True, timeout=timeout)
)
client = OpenRouter(
api_key=self._settings.openrouter_api_key.get_secret_value(),
async_client=self._capturing_client,
src/transcription/services/__init__.py
+26
View File
@@ -2,7 +2,12 @@
from dataclasses import dataclass
from dataclasses import field
from typing import Self
from sqlalchemy.ext.asyncio import async_sessionmaker
from sqlmodel.ext.asyncio.session import AsyncSession
from ..config import Settings
from .documents import DocumentService
from .jobs import JobService
from .people import PeopleService
@@ -20,3 +25,24 @@ class ServiceBundle:
sources: SourceService = field(default_factory=SourceService)
jobs: JobService = field(default_factory=JobService)
people: PeopleService = field(default_factory=PeopleService)
@classmethod
def from_session_factory(
cls,
session_factory: async_sessionmaker[AsyncSession] | None = None,
*,
settings: Settings | None = None,
) -> Self:
"""Build a bundle whose services all share one session factory and settings."""
if session_factory is None:
return cls()
return cls(
documents=DocumentService(session_factory=session_factory, settings=settings),
sources=SourceService(session_factory=session_factory, settings=settings),
jobs=JobService(session_factory=session_factory, settings=settings),
people=PeopleService(session_factory=session_factory, settings=settings),
)
async def aclose(self) -> None:
"""Release provider resources held by the bundle."""
await self.sources.aclose()
src/transcription/services/jobs.py
+20 -7
View File
@@ -168,24 +168,37 @@ class JobService(ServiceBase):
await self._finalize(session=_session, caller_session=session, refresh=(job,))
return job
async def read_next_queued_job(
async def claim_next_queued_job(
self,
*,
session: AsyncSession | None = None,
) -> Job | None:
"""Read the next queued job ordered by creation time."""
"""Atomically claim the oldest queued job by transitioning it to PROCESSING.
The selection is deliberately unadorned: no eager loads are applied to the
hot poll, because callers re-read the claimed job with the relationships
they actually need. On PostgreSQL the row is locked with ``SKIP LOCKED`` so
concurrent workers never contend for the same job.
"""
async with self._session_scope(session) as _session:
query = (
select(Job)
.options(
selectinload(Job.document), # pyright: ignore[reportArgumentType]
selectinload(Job.job_sources).selectinload(JobSource.source), # pyright: ignore[reportArgumentType]
)
.where(Job.status == JobStatus.QUEUED)
# Break ties by id so "next" is stable when two rows share close timestamps.
.order_by(Job.date_created, Job.id) # pyright: ignore[reportArgumentType]
.limit(1)
)
return (await _session.exec(query)).first()
if _session.get_bind().dialect.name == "postgresql":
query = query.with_for_update(skip_locked=True)
job = (await _session.exec(query)).first()
if job is None:
return None
job.status = JobStatus.PROCESSING
job.date_updated = datetime.now(UTC)
await self._finalize(session=_session, caller_session=session, refresh=(job,))
return job
async def requeue_stale_processing_jobs(
self,
src/transcription/services/sources.py
+12 -17
View File
@@ -4,7 +4,6 @@ from __future__ import annotations
import base64
import hashlib
import inspect
import logging
import os
from collections.abc import Sequence
@@ -1142,8 +1141,8 @@ class SourceService(ServiceBase):
def _resolve_transcript_model(*, provider: TranscriptionProvider, settings: Settings) -> str:
provider_model = getattr(provider, "model", None)
if isinstance(provider_model, str) and provider_model.strip():
provider_model = provider.model
if provider_model and provider_model.strip():
return provider_model
if settings.provider_model and settings.provider_model.strip():
@@ -1226,22 +1225,18 @@ async def transcribe_document_image(
try:
with handle_transcription_errors():
transcribe_kwargs = {
"prompt_text": prompt_execution.user_prompt,
"image_bytes": image_bytes,
"mime_type": mime_type,
"temperature": prompt_execution.temperature,
"top_p": prompt_execution.top_p,
"source_reference": source_reference,
}
if "requested_model" in inspect.signature(adapter.transcribe).parameters:
transcribe_kwargs["requested_model"] = requested_model
result = await adapter.transcribe(**transcribe_kwargs)
result = await adapter.transcribe(
prompt_text=prompt_execution.user_prompt,
image_bytes=image_bytes,
mime_type=mime_type,
temperature=prompt_execution.temperature,
top_p=prompt_execution.top_p,
source_reference=source_reference,
requested_model=requested_model,
)
finally:
if owns_adapter:
close = getattr(adapter, "aclose", None)
if close is not None:
await close()
await adapter.aclose()
logger.info("Transcription completed for image=%s provider=%s", image_path, result.provider)
return TranscriptionResult(
text=result.text,
src/transcription/services/workflows.py
+13 -10
View File
@@ -185,13 +185,15 @@ async def process_queued_job( # noqa: PLR0915
logger.warning(f"Job {job.id} is not queued. Current status: {job.status}")
return
# Transaction A: claim job for processing.
# Transaction A: claim job for processing. Reached only when a caller hands us a
# still-QUEUED job directly; the worker path already claimed it atomically in
# JobService.claim_next_queued_job.
if current_status == JobStatus.QUEUED:
if session is None:
job = await services.jobs.mark_job_status(job.id, JobStatus.PROCESSING)
else:
# If we're sharing the session, need to make sure setting the Job to PROCESSING is committed before we start
# the transcription, otherwise other workers may see the job as still QUEUED and try to process it.
# Commit the PROCESSING transition before transcription starts so the claim
# is durable and visible to any other worker before the long provider call.
job = await services.jobs.mark_job_status(job.id, JobStatus.PROCESSING, session=session)
await session.commit()
@@ -294,12 +296,8 @@ async def process_queued_job( # noqa: PLR0915
0,
int((asyncio.get_running_loop().time() - monotonic_started_at) * 1000),
),
request_manifest=getattr(services.sources.provider, "current_request_manifest", None),
transport_evidence=getattr(
services.sources.provider,
"current_transport_evidence",
None,
),
request_manifest=services.sources.provider.current_request_manifest,
transport_evidence=services.sources.provider.current_transport_evidence,
failure_phase="local_timeout",
model_input_artifact_id=(
provider_input.derivative_id if provider_input is not None else None
@@ -417,11 +415,16 @@ async def process_next_queued_job(
session: AsyncSession | None = None,
) -> bool:
"""Process the next queued job if one exists."""
job = await services.jobs.read_next_queued_job(session=session)
job = await services.jobs.claim_next_queued_job(session=session)
if job is None:
return False
# The claim must be durable before the provider call starts, otherwise another
# worker could observe the job as still QUEUED and process it a second time.
if session is not None:
await session.commit()
await advance_job(job=job, services=services, settings=settings, session=session)
return True
src/transcription/worker.py
+53 -39
View File
@@ -19,10 +19,6 @@ from transcription.errors import AppError
from transcription.errors import classify_unexpected_error
from .services import ServiceBundle
from .services.documents import DocumentService
from .services.jobs import JobService
from .services.people import PeopleService
from .services.sources import SourceService
from .services.workflows import process_next_queued_job as process_next_queued_job_workflow
logger = logging.getLogger(__name__)
@@ -121,56 +117,74 @@ async def run_worker_loop(
If wake_event is provided, signal activity wakes the loop immediately while
timeout-based wakeups preserve current polling behavior.
The service bundle — and with it the provider's pooled HTTP client — is built
once for the lifetime of the loop, so consecutive jobs reuse one connection
instead of paying a fresh TLS handshake each time.
"""
while True:
if stop_event is not None and stop_event.is_set():
logger.info("Worker stop event received")
return
if wake_event is not None:
with suppress(TimeoutError):
await asyncio.wait_for(wake_event.wait(), timeout=poll_interval_seconds)
wake_event.clear()
processed_any = False
services = ServiceBundle.from_session_factory(session_factory)
try:
while True:
with handle_worker_exceptions(operation="worker.process_next_queued_job"):
processed = await process_next_queued_job(session_factory=session_factory)
if not processed:
break
processed_any = True
continue
if stop_event is not None and stop_event.is_set():
logger.info("Worker stop event received")
return
break
if wake_event is not None:
with suppress(TimeoutError):
await asyncio.wait_for(wake_event.wait(), timeout=poll_interval_seconds)
wake_event.clear()
if wake_event is None and not processed_any:
await asyncio.sleep(poll_interval_seconds)
processed_any = False
while True:
with handle_worker_exceptions(operation="worker.process_next_queued_job"):
processed = await process_next_queued_job(
session_factory=session_factory,
services=services,
)
if not processed:
break
processed_any = True
continue
break
if wake_event is None and not processed_any:
await asyncio.sleep(poll_interval_seconds)
finally:
await services.aclose()
async def process_next_queued_job(
*,
session: AsyncSession | None = None,
session_factory: async_sessionmaker[AsyncSession] | None = None,
services: ServiceBundle | None = None,
) -> bool:
"""Process the next queued job and persist terminal outcome.
Returns True when a job was processed, False when no queued job exists.
When ``services`` is supplied the caller owns its lifecycle; otherwise a
bundle is created and closed here.
"""
if session_factory is None:
services = ServiceBundle()
else:
services = ServiceBundle(
documents=DocumentService(session_factory=session_factory),
sources=SourceService(session_factory=session_factory),
jobs=JobService(session_factory=session_factory),
people=PeopleService(session_factory=session_factory),
)
if services is not None:
return await _process_next_queued_job(services=services, session=session, session_factory=session_factory)
owned = ServiceBundle.from_session_factory(session_factory)
try:
if session is None:
async with session_scope(session_factory=session_factory) as local_session:
return await process_next_queued_job_workflow(services=services, session=local_session)
return await process_next_queued_job_workflow(services=services, session=session)
return await _process_next_queued_job(services=owned, session=session, session_factory=session_factory)
finally:
await services.sources.aclose()
await owned.aclose()
async def _process_next_queued_job(
*,
services: ServiceBundle,
session: AsyncSession | None,
session_factory: async_sessionmaker[AsyncSession] | None,
) -> bool:
if session is None:
async with session_scope(session_factory=session_factory) as local_session:
return await process_next_queued_job_workflow(services=services, session=local_session)
return await process_next_queued_job_workflow(services=services, session=session)