transcription/docs/ver4.7/scope_boundary_v4_7.md

V4.7 Scope Boundary

This document defines the frozen boundary for V4.7, an architectural cleanup and evidence-model re-alignment release. V4.6 remains the behavioral baseline. V4.7 introduces no new user-facing features; it completes the structural work V4.6 deferred, simplifies the evidence model down to what the application actually uses, and closes the correctness items opened during V4.6 implementation.

Every item in scope is traceable either to a review finding ID in Architecture & Code Review Report or to a numbered entry in the V4.6 implementation review log. Any change that cannot be traced to one of those is out of scope.

All image presentation, media, and telemetry-presentation work is deferred to V4.8. See ver4.8/feature_backlog_v4_8.md.

Purpose

• Collapse the duplicated evidence model so that job_source records membership and queue state and execution_attempt records evidence, with no overlap.
• Remove the ProcessingArtifact subsystem, which has executed exactly once in the application's history, and move orientation normalization to ingest where it belongs.
• Complete [MED-14] by decomposing SourceService, which still owns four domain models.
• Correct the run-time measurement window so provider latency can be trusted before anything is built on top of it.
• Stop the worker from silently swallowing programming errors.
• End the dual-spelling persistence of JobSourceStatus.
• Make the ruff / ty gate enforceable in CI rather than only on a developer machine that has run pre-commit install.

Confirmed Operating Context

These answers are frozen for V4.7 and govern every decision below.

Question	Answer
Database	SQLite only. PostgreSQL remains the intended destination. The V4.6 re-level already resolved the FK cycle with use_alter=True.
Topology	Single user, single process, single worker. Unchanged from V4.6.
Schema evolution	Re-level from current metadata, exactly as V4.6. No Alembic, no _upgrade_* chain.
Existing data	The live database is populated. V4.7 is schema-affecting: three structural changes plus a one-time image backfill, delivered by a single tools/migrate_v46_to_v47.py.
Release character	Architectural cleanup and evidence-model re-alignment. No new features.
Image fidelity	Visually lossless is sufficient. Measured at 51.5-55.0 dB PSNR for a single re-encode generation. Bit-exact preservation was considered and rejected as unnecessary complexity.
Provider settings	WORKER_PROVIDER_TIMEOUT_SECONDS=30.0, WORKER_MAX_RETRIES=1, calibrated 2026-08-18. Not revisited in V4.7.

Evidence Gathered

The decisions below rest on measurements taken against the live database on 2026-08-18, not on inspection alone.

Measurement	Result
job_source.raw_transcription vs latest attempt	77/77 identical
job_source.ai_metadata vs normalized_metadata	77/77 identical
job_source.raw_api_response vs sdk_response_snapshot	77/77 identical
job_source.error_detail vs attempt error_detail	2/2 identical
job_source.status vs execution_attempt.status	0/79 textually identical - the dual-spelling defect [45]
job_source rows with more than one attempt	1 of 79
processing_artifact rows in existence	2, both from one job on 2026-08-16, against 77 successful transcriptions
Source images carrying EXIF orientation 3	58 of 79, of which only 1 was ever normalized

In Scope

1. Evidence Model Simplification

job_source began as the many-to-many link between job and source and accreted response-capture fields over time. execution_attempt, added later in V4.2 (commit 6bd4cbb), captures the same information in more detail. The measurements above show the overlap is total, not partial.

job_source is stripped, not deleted. It cannot be folded into execution_attempt, because it carries state that exists when no provider call has occurred:

• store.py:249,313 create rows with status=PENDING at job creation, before any call.
• workflows.py:432-442 selects work by status != TRANSCRIBED on job.job_sources.
• jobs.py:378-384 cancel writes a terminal state with no provider call at all, so no attempt row could carry it.

An append-only evidence table cannot express "queued, not yet attempted" or "cancelled before any call". The junction survives; the duplicated evidence does not.

Retained: id, job_id, source_id, status.

Removed: raw_transcription, ai_metadata, raw_api_response, executed_at, error_detail.

Added: JobSourceStatus.CANCELLED, so cancellation stops overloading FAILED plus the free-text string "Cancelled by user". This is what retires error_detail.

Unchanged: the retry reset at jobs.py:411-424. Flipping FAILED back to PENDING loses no history, because ExecutionAttempt's UniqueConstraint(job_id, source_id, attempt_number) (models.py:389) already preserves every prior attempt. This is confirmed in live data: attempt 1 FAILED/local_timeout and attempt 2 TRANSCRIBED are both retained. Adding a second job_source row per retry would duplicate that mechanism and break the one-row-per-(job, page) assumption in read_job_source_for_job and sources.py:570-574 - where uniqueness is enforced in code, not by a database constraint.

This item absorbs review log [45], since both changes rewrite JobSourceStatus persistence and must land as one migration.

2. ProcessingArtifact Removal and Ingest Normalization

ProcessingArtifact is a generic container for derived data products, with a CheckConstraint enforcing that content is either inline JSON or an external file, never both. Two rows exist. The quality-warnings path at workflows.py:560-571 writes one on every successful page, yet 77 successful transcriptions produced a single row, so the subsystem postdates nearly all data and has effectively never run.

Orientation normalization itself is not dispensable and was not a red herring. 58 of 79 stored images carry EXIF orientation 3, and the raw decoded pixels of the page that prompted the original investigation are genuinely upside down. Sending those bytes unrotated sends an inverted page to the model.

The fix is to normalize at ingest rather than derive at transcription time:

• Rotate on upload, in media_storage, before the image is stored. Every stored byte is then already upright and no derivative needs to exist.
• Use Pillow with qtables=im.quantization, subsampling=JpegImagePlugin.get_sampling(im), optimize=True. Measured against the current quality=95, subsampling=0 settings at normalization.py:84-85, this is better on both axes: 51.5-55.0 dB PSNR versus 50.0-53.5 dB, and roughly 6% smaller output versus 38% larger.
• Strip the EXIF orientation tag after rotating.
• No archival master is retained. No external jpegtran dependency is introduced. No MCU-alignment rejection path is needed, because Pillow handles any dimensions - including the single 2306x2019 outlier.

Then delete: the processing_artifact table, the ProcessingArtifact model, the ~283-line artifact cluster in sources.py (lines 732-1015), resolve_provider_input, and the artifact branch of build_evidence_export. The transcription_quality_warnings payload folds into execution_attempt.normalized_metadata.

Deleting stored images is not involved; the 58 already-ingested rotated images are rotated in place by the migration. No live integrity check is invalidated: Source has no digest column, and the only stored digests are ExecutionAttempt.request_manifest_sha256 - a hash of the request manifest, correct as history - and ProcessingArtifact.payload_sha256, which is removed with the table.

3. SourceService Decomposition ([MED-14])

services/sources.py is 1,389 lines and SourceService owns Source, JobSource, ExecutionAttempt, and ProcessingArtifact.

Item 2 removes the ProcessingArtifact responsibility by deletion rather than extraction. The previously planned services/artifacts.py is therefore cancelled - extracting ~283 lines into a new module and then deleting that module would be wasted work.

What remains is the ExecutionAttempt cluster, moved to services/evidence.py (~174 lines): read_latest_execution_attempt (216-245) with its LatestExecutionAttempt read model, promote_machine_attempt (679-710), list_execution_attempts (710-732), and build_evidence_export (1015-1107).

update_job_source_transcription stays in sources.py. The V4.6 deferral note proposed moving it to workflows.py as orchestration; that proposal is not adopted. The method writes JobSource and ExecutionAttempt inside one session scope and derives attempt_number at lines 595-600, and services.instructions.md:63-65 requires the transcript update and the paired terminal status change to commit or roll back together. services.instructions.md:72 assigns session-aware write helpers to services and commit-boundary control to orchestration, so the current placement already satisfies the instruction file. Splitting the two writes across modules is the most plausible way that atomicity later gets broken. The method will shrink under item 1, since several of the fields it writes cease to exist.

Expected result: sources.py lands near 900 lines.

4. Run-Time Measurement Window (review log [55])

services/workflows.py:221 sets monotonic_started_at before provider-input preparation and the session.commit() at line 228. Line 251 computes elapsed_seconds from it. But the asyncio.wait_for timeout at lines 240-249 wraps only _call_transcriber.

duration_ms therefore measures a strictly wider window than the budget that governs it. This is observable in the migrated data: three historical local_timeout rows recorded 20.4 / 20.8 / 22.0 s against a 20.0 s timeout.

In scope: either record provider latency as a distinct value, or move monotonic_started_at to immediately before the wait_for. Whichever is chosen, the resulting figure must be the quantity the timeout actually governs. Item 2 also removes normalization from this window entirely, which shrinks the discrepancy but does not by itself fix it.

This item must land before any V4.8 telemetry presentation work.

5. Worker Exception Handling (review log [8])

worker.py:96-106, handle_worker_exceptions, catches bare Exception, logs it, and suppresses it. A programming error inside the worker loop is therefore indistinguishable from a transient provider fault and is retried silently with no UI signal.

In scope: distinguish genuinely retriable faults from programming errors, and ensure a non-retriable error surfaces rather than looping. Retry counting and terminal-state transitions remain governed by services.instructions.md:63-65.

6. CI Enforcement of the Quality Gate ([HIGH-06], review log [40])

.github/workflows/ is empty. The ruff check and ty check gate established in V4.6 Phase 7 exists only in .pre-commit-config.yaml, which is inert until a developer runs pre-commit install.

In scope: a CI workflow running ruff check, ty check, and pytest on push and pull request, using the same commands as the local hooks so the two cannot drift.

Out of Scope

• All image and media presentation work. Pan and zoom on Source Detail, the homepage gallery, multi-portrait support, image descriptions, and background wallpaper are V4.8.
• The model-performance rollup (review log [54]). It depends on item 4 and is a new user-facing view.
• Reducing update_job_source_transcription. See section 3.
• Bit-exact image preservation. Considered and rejected; see Confirmed Operating Context.
• PostgreSQL cutover.
• Re-tuning WORKER_PROVIDER_TIMEOUT_SECONDS or WORKER_MAX_RETRIES. Calibrated 2026-08-18 against measured per-model durations.
• Removing slow models from PROVIDER_MODELS (review log [53]). A configuration judgement, deliberately left with the operator.
• Any new feature.

Locked Design Decisions

A. Cleanup Only

V4.7 changes structure and correctness. It does not change what the application does for a user. If a change would be visible on a page as new capability, it belongs in V4.8.

B. One Home Per Fact

After V4.7, any given piece of evidence is stored in exactly one place. job_source holds membership and state; execution_attempt holds evidence. Denormalized convenience copies are not reintroduced, and if a read becomes awkward the fix is a query or a read model, not a duplicated column.

C. Delete Before Refactor

Item 2 deletes the artifact subsystem before item 3 restructures what remains. Code scheduled for deletion is never extracted, renamed, or moved first.

D. Simplicity Over Edge-Case Management

Where two approaches both satisfy the requirement, the one with fewer moving parts wins. This is why rotation uses Pillow rather than a lossless DCT transform, and why no archival master is kept.

E. Measurement Before Presentation

Item 4 precedes all V4.8 telemetry work. A dashboard built on a conflated metric looks authoritative and quietly misleads.

F. One Migration, Backed Up

All schema and data changes land in a single tools/migrate_v46_to_v47.py: idempotent, never invoked at startup, never run by the test suite, following the tools/migrate_v45_to_v46.py conventions. data/transcription.db and data/documents/ are backed up before it runs, because the image backfill rewrites files in place.

G. The Instruction Files Are the Standard

.github/instructions/services.instructions.md and ui.instructions.md govern. Where this document and an instruction file disagree, the instruction file wins.

Acceptance Criteria

• job_source carries exactly id, job_id, source_id, status; every evidence read resolves through execution_attempt.
• JobSourceStatus.CANCELLED exists and cancellation no longer writes free text into a removed column.
• job_source.status and execution_attempt.status persist with one spelling, and existing rows are consistent.
• The processing_artifact table, its model, and its service cluster no longer exist.
• Newly uploaded images are stored upright with no EXIF orientation tag, and the 58 pre-existing rotated images have been backfilled.
• services/sources.py is materially smaller, with ExecutionAttempt responsibilities in services/evidence.py and update_job_source_transcription unmoved.
• The recorded duration reflects only the operation the timeout governs.
• A programming error in the worker loop is distinguishable from a provider fault.
• ruff check reports no findings; ty check reports 0 diagnostics, the V4.6 exit state.
• The full test suite passes.
• CI runs the same ruff / ty / pytest gate as the local hooks.
• No new user-facing behavior.

Scope Freeze Gate

This boundary is frozen. Adding an item requires a finding ID or a review-log entry, and an explicit note recording the addition.

Related Local References

• V4.7 Implementation Plan
• Architecture & Code Review Report - finding IDs
• V4.6 Review Log - resolves the review log [N] citations used throughout this document
• V4.6 Scope Boundary - the baseline this release builds on
• V4.6 Implementation Plan
• V4.8 Feature Backlog - where deferred feature work is parked
• .github/instructions/services.instructions.md
• .github/instructions/ui.instructions.md