Commit Graph
38 Commits
Author SHA1 Message Date
zoltan57andCopilot App 7b9715b3f1 V4.6 Phase 3: worker and provider reliability
Claim jobs atomically [CRIT-01]
- Replace JobService.read_next_queued_job with claim_next_queued_job, which
  selects and transitions QUEUED -> PROCESSING inside one transaction. The old
  read-then-write sequence left a window in which two workers could observe the
  same QUEUED row.
- Add the missing .limit(1). The poll previously ordered the entire queued set
  and discarded all but the first row.
- Drop the eager loads from the hot poll entirely. They were pure waste:
  process_queued_job immediately re-reads the job through read_job with the
  relationships it actually needs.
- Guard the row with with_for_update(skip_locked=True) on PostgreSQL so the
  claim stays correct once more than one worker exists. On SQLite the claim is a
  bounded single-writer transaction.
- Correct the comment at the remaining direct-call claim site, which described
  the hazard rather than the guarantee.

Reuse the provider connection [HIGH-02]
- Build the ServiceBundle once per worker loop instead of once per job, and
  close it at loop shutdown. Every job previously constructed a new
  SourceService, and with it a new provider adapter and a new httpx.AsyncClient,
  paying a full TLS handshake per page and discarding the connection pool.
- process_next_queued_job now accepts an optional caller-owned bundle and only
  closes bundles it created itself.

Uncap the provider timeout [HIGH-03]
- Remove le=20.0 from worker_provider_timeout_seconds. The cap equalled the
  default, so the ceiling could never be raised, and dense-page vision
  transcription routinely needs longer. Default raised to 180s.
- Pass an explicit httpx.Timeout to the OpenRouter AsyncClient. httpx defaults
  every phase to 5 seconds, so the real read budget was 5s regardless of the
  configured value; the outer asyncio.wait_for could never be the binding
  constraint. Connect stays at 10s.

Tighten the provider boundary [MED-03]
- Declare model, current_request_manifest, current_transport_evidence, and
  aclose on the TranscriptionProvider Protocol.
- Delete the per-call inspect.signature(adapter.transcribe).parameters
  reflection and the untyped kwargs dict it fed. The Protocol had declared
  requested_model all along, so the reflection was dead defensive weight on the
  hot path.
- Replace the three getattr probes for aclose and the evidence attributes with
  direct typed access.

Deduplicate bundle construction [MED-06]
- Add ServiceBundle.from_session_factory and ServiceBundle.aclose, replacing the
  duplicated four-service instantiation blocks in app.py and worker.py.
- _recover_stale_processing_jobs now uses the bundle built moments earlier
  instead of constructing a second JobService.

Tests
- Claiming returns the oldest job, marks it PROCESSING, never hands the same job
  out twice, and emits exactly one unadorned SELECT carrying LIMIT and no JOIN.
- The worker loop threads one bundle through consecutive jobs and closes it once
  at shutdown; a caller-owned bundle is left open.
- Settings accepts a timeout above 20 seconds and still rejects zero.
- The OpenRouter client's read, write, and pool timeouts track the configured
  budget rather than the httpx default.

Note: .env in this checkout still pins WORKER_PROVIDER_TIMEOUT_SECONDS=20 and
should be raised to pick up this fix.

Verified: 268 passed, 4 skipped; ruff check clean.

Co-authored-by: Copilot App <[email protected]>
2026-08-17 16:26:31 -05:00
zoltan57andCopilot App 2ccea77520 V4.6 Phase 1: deletions and quick wins
Pure remediation; no behavior change. Every item traces to a finding in
docs/architecture_code_review_2026-08-17.md.

Deletions
- Delete app_state.py, which had zero importers and whose get_session_factory
  raised TypeError at runtime [HIGH-01].
- Delete services/transcription.py and point build_prompt_execution imports at
  services/sources.py; drop the store.py compatibility aliases [MED-05].
- Delete ServiceBase.queue and its unparameterized asyncio.Queue [MED-07].
- Delete db/operations.get_next_queued_job, a divergent duplicate [CRIT-01].
- Drop the discarded load_docs parameter from list_jobs [LOW-03].

Config
- Delete worker_retry_backoff_seconds; no backoff behavior existed anywhere, so
  wiring it would have been a new feature [MED-02].
- Wire sqlite_check_same_thread through get_engine. The engine hardcoded the
  setting's own default, so this preserves behavior exactly [MED-02].
- Replace DATABASE_URL in docker-compose.yml with the nested DATABASE__DRIVER /
  DATABASE__PATH names. Settings uses env_nested_delimiter with extra="ignore",
  so DATABASE_URL was silently discarded [MED-10].

UI
- Move the 23KB inline VIBESCRIBE_LOGO_SVG to ui/static/vibescribe_logo.svg and
  load it through a cached read_svg sibling of read_css [MED-09].
- Route the portrait upload failure through error_presenter.show_error [LOW-07].
- Cancel the job detail auto-refresh timer instead of only deactivating it, and
  name its interval constant [LOW-06].

Worker
- Make WorkerNotifier runtime_checkable and validate the resolved object in
  resolve_worker_notifier, which previously returned any non-None attribute
  unchecked [LOW-04].

Docs and lint
- Fix two stale paths in services.instructions.md, one of which pointed at the
  module deleted here [LOW-02].
- ruff check --fix to zero [LOW-01].

Verified: 264 passed, 4 skipped; ruff check clean.

Co-authored-by: Copilot App <[email protected]>
2026-08-17 16:06:08 -05:00
Jim Lancaster 7054cd8af9 V4.5 Complete - Enhanced trancription context, added option to restranscribe source under different models. 2026-08-16 09:06:56 -05:00
Jim Lancaster 5b97c759fe V4.4 revision to facsimile print format 2026-08-15 23:25:12 -05:00
Jim Lancaster 7db4df1729 V4.4 Complete 2026-08-15 14:30:33 -05:00
Jim Lancaster a78b58ff40 V4.3 revision to Document Types 2026-08-15 13:29:53 -05:00
Jim Lancaster c9f5dca064 V4.2 complete 2026-08-14 15:59:38 -05:00
Jim Lancaster 6bd4cbb0a7 V4.2 Updated what ai_raw_response data is being captured. The changes were more extensive than I expected. 2026-08-14 07:21:15 -05:00
Jim Lancaster 89cf69f8a2 V4.1 Mostly UI adjustments by GC 2026-08-12 13:11:50 -05:00
Jim Lancaster 1e8d8572d4 Continue GC code review: Pydantic 2026-08-12 01:35:50 -05:00
Jim Lancaster b8be27f0c9 Continue GC code review and cleanup 2026-08-11 16:42:09 -05:00
Jim Lancaster 8d5aec4301 Github Copilot service realignment & cleanup 2026-08-11 16:02:19 -05:00
Jim Lancaster 0ace10269f V4 implemented. Some tweaking left, but it is working 2026-08-11 12:07:19 -05:00
Jim Lancaster e6549277c6 V3 Updated V3 core documents. Added data folder backup/restore before/after running destructive tests. 2026-08-09 10:51:30 -05:00
Jim Lancaster b59d3da23e V3 fix document delete issue 2026-08-08 21:58:19 -05:00
Jim Lancaster 4bf6c9e2f3 V3 post step 2 refinement: add temperature & top-p settings to config (and .env), add prompt fields back to job table so that the prompt settings get frozen at runtime for all sources being processed. 2026-08-08 18:21:44 -05:00
Jim Lancaster 4dac9349c1 V3 step 1 update models.py and step 2 implement service/worker, and raw API response persistence 2026-08-08 18:08:04 -05:00
Jim Lancaster 58faa00d7b AI metadata and api prompt results data capture now fixed 2026-08-08 14:59:45 -05:00
Jim Lancaster d9f5fbb1a4 Delete Document & Delete Source buttons now delete the underlying files 2026-08-04 18:34:35 -05:00
Jim Lancaster 271633d1d5 Jobs: jobs still stuck in queue. Fixes from testing. 2026-08-04 18:09:31 -05:00
Jim Lancaster 6c6589d8ff Jobs: big jobs stuck in queue. Added Cancel, Resubmit 2026-08-04 09:03:51 -05:00
Jim Lancaster 752346025b UI style refresh continued 2026-08-03 15:15:58 -05:00
Jim Lancaster 9653060c2a UI update complete? 2026-08-02 13:33:09 -05:00
Jim Lancaster 51ac2d0b98 V2 step 3 complete 2026-08-01 16:24:44 -05:00
Jim Lancaster 61cc8a200b V2 step 2 complete 2026-08-01 16:17:38 -05:00
John Lancaster 3d6daec561 moved models to db pkg 2026-07-30 21:10:13 -05:00
Jim Lancaster a3b3bab571 V1 mostly complete except for some testing. Linting in the last step changed nearly every file which is why this commit is so larger. 2026-07-29 17:27:21 -05:00
Jim Lancaster bc21a97019 Updated test suite 2026-07-29 16:20:46 -05:00
John Lancaster 2000f0096b pruning 2026-06-27 22:13:55 -05:00
John Lancaster e6f12fa993 job service tests 2026-06-27 19:29:51 -05:00
John Lancaster d3ffb01e93 test settings 2026-06-27 19:15:56 -05:00
John Lancaster f0b0109b11 started test_job_service 2026-06-27 19:04:21 -05:00
John Lancaster f4417a0f64 test stubs 2026-06-27 18:05:10 -05:00
Jim Lancaster e61f7e7518 ver1-step2 implemented 2026-06-25 19:22:44 -05:00
Jim Lancaster e291ffc907 Error handling added to MVP according to error_handling.md guideline 2026-06-25 12:56:35 -05:00
Jim Lancaster fb1bc7ea16 Step 4 implemented 2026-06-24 19:12:42 -05:00
Jim Lancaster 3749355b19 Step 3 tested with real data. Added Step 4 implementation plan 2026-06-24 18:48:23 -05:00
Jim Lancaster 865cca39e6 Step 3 complete 2026-06-24 17:33:51 -05:00