transcription/docs/ver4.7/scope_boundary_v4_7.md

# V4.7 Scope Boundary

This document defines the frozen boundary for V4.7, an **architectural cleanup and evidence-model re-alignment release**. V4.6 remains the behavioral baseline. V4.7 introduces **no new user-facing features**; it completes the structural work V4.6 deferred, simplifies the evidence model down to what the application actually uses, and closes the correctness items opened during V4.6 implementation.

Every item in scope is traceable either to a review finding ID in [Architecture & Code Review Report](../architecture_code_review_2026-08-17.md) or to a numbered entry in the V4.6 implementation review log. Any change that cannot be traced to one of those is out of scope.

All image *presentation*, media, and telemetry-presentation work is deferred to V4.8. See [`ver4.8/feature_backlog_v4_8.md`](../ver4.8/feature_backlog_v4_8.md).

## Purpose

- Collapse the duplicated evidence model so that `job_source` records **membership and queue state** and `execution_attempt` records **evidence**, with no overlap.
- Remove the `ProcessingArtifact` subsystem, which has executed exactly once in the application's history, and move orientation normalization to ingest where it belongs.
- Complete [MED-14] by decomposing `SourceService`, which still owns four domain models.
- Correct the run-time measurement window so provider latency can be trusted before anything is built on top of it.
- Stop the worker from silently swallowing programming errors.
- End the dual-spelling persistence of `JobSourceStatus`.
- Make the `ruff` / `ty` gate enforceable in CI rather than only on a developer machine that has run `pre-commit install`.

## Confirmed Operating Context

These answers are frozen for V4.7 and govern every decision below.

| Question | Answer |
| :--- | :--- |
| Database | **SQLite only.** PostgreSQL remains the intended destination. The V4.6 re-level already resolved the FK cycle with `use_alter=True`. |
| Topology | **Single user, single process, single worker.** Unchanged from V4.6. |
| Schema evolution | **Re-level from current metadata**, exactly as V4.6. No Alembic, no `_upgrade_*` chain. |
| Existing data | The live database is populated. V4.7 is **schema-affecting**: three structural changes plus a one-time image backfill, delivered by a single `tools/migrate_v46_to_v47.py`. |
| Release character | **Architectural cleanup and evidence-model re-alignment.** No new features. |
| Image fidelity | **Visually lossless is sufficient.** Measured at 51.5-55.0 dB PSNR for a single re-encode generation. Bit-exact preservation was considered and rejected as unnecessary complexity. |
| Provider settings | `WORKER_PROVIDER_TIMEOUT_SECONDS=30.0`, `WORKER_MAX_RETRIES=1`, calibrated 2026-08-18. Not revisited in V4.7. |

## Evidence Gathered

The decisions below rest on measurements taken against the live database on 2026-08-18, not on inspection alone.

| Measurement | Result |
| :--- | :--- |
| `job_source.raw_transcription` vs latest attempt | **77/77 identical** |
| `job_source.ai_metadata` vs `normalized_metadata` | **77/77 identical** |
| `job_source.raw_api_response` vs `sdk_response_snapshot` | **77/77 identical** |
| `job_source.error_detail` vs attempt `error_detail` | 2/2 identical |
| `job_source.status` vs `execution_attempt.status` | 0/79 textually identical - the dual-spelling defect [45] |
| `job_source` rows with more than one attempt | 1 of 79 |
| `processing_artifact` rows in existence | **2**, both from one job on 2026-08-16, against 77 successful transcriptions |
| Source images carrying EXIF orientation 3 | **58 of 79**, of which only 1 was ever normalized |

## In Scope

### 1. Evidence Model Simplification

`job_source` began as the many-to-many link between `job` and `source` and accreted response-capture fields over time. `execution_attempt`, added later in V4.2 (commit `6bd4cbb`), captures the same information in more detail. The measurements above show the overlap is total, not partial.

**`job_source` is stripped, not deleted.** It cannot be folded into `execution_attempt`, because it carries state that exists when no provider call has occurred:

- `store.py:249,313` create rows with `status=PENDING` **at job creation**, before any call.
- `workflows.py:432-442` selects work by `status != TRANSCRIBED` on `job.job_sources`.
- `jobs.py:378-384` cancel writes a terminal state with **no provider call at all**, so no attempt row could carry it.

An append-only evidence table cannot express "queued, not yet attempted" or "cancelled before any call". The junction survives; the duplicated evidence does not.

**Retained:** `id`, `job_id`, `source_id`, `status`.

**Removed:** `raw_transcription`, `ai_metadata`, `raw_api_response`, `executed_at`, `error_detail`.

**Added:** `JobSourceStatus.CANCELLED`, so cancellation stops overloading `FAILED` plus the free-text string `"Cancelled by user"`. This is what retires `error_detail`.

**Unchanged:** the retry reset at `jobs.py:411-424`. Flipping `FAILED` back to `PENDING` loses no history, because `ExecutionAttempt`'s `UniqueConstraint(job_id, source_id, attempt_number)` (`models.py:389`) already preserves every prior attempt. This is confirmed in live data: attempt 1 `FAILED`/`local_timeout` and attempt 2 `TRANSCRIBED` are both retained. Adding a second `job_source` row per retry would duplicate that mechanism and break the one-row-per-`(job, page)` assumption in `read_job_source_for_job` and `sources.py:570-574` - where uniqueness is enforced **in code, not by a database constraint**.

This item absorbs review log [45], since both changes rewrite `JobSourceStatus` persistence and must land as one migration.

### 2. ProcessingArtifact Removal and Ingest Normalization

`ProcessingArtifact` is a generic container for derived data products, with a `CheckConstraint` enforcing that content is either inline JSON or an external file, never both. Two rows exist. The quality-warnings path at `workflows.py:560-571` writes one on **every** successful page, yet 77 successful transcriptions produced a single row, so the subsystem postdates nearly all data and has effectively never run.

Orientation normalization itself is **not** dispensable and was **not** a red herring. 58 of 79 stored images carry EXIF orientation 3, and the raw decoded pixels of the page that prompted the original investigation are genuinely upside down. Sending those bytes unrotated sends an inverted page to the model.

The fix is to normalize at ingest rather than derive at transcription time:

- Rotate on upload, in `media_storage`, before the image is stored. Every stored byte is then already upright and no derivative needs to exist.
- Use Pillow with `qtables=im.quantization`, `subsampling=JpegImagePlugin.get_sampling(im)`, `optimize=True`. Measured against the current `quality=95, subsampling=0` settings at `normalization.py:84-85`, this is **better on both axes**: 51.5-55.0 dB PSNR versus 50.0-53.5 dB, and roughly 6% smaller output versus 38% larger.
- Strip the EXIF orientation tag after rotating.
- No archival master is retained. No external `jpegtran` dependency is introduced. No MCU-alignment rejection path is needed, because Pillow handles any dimensions - including the single 2306x2019 outlier.

Then delete: the `processing_artifact` table, the `ProcessingArtifact` model, the ~283-line artifact cluster in `sources.py` (lines 732-1015), `resolve_provider_input`, and the artifact branch of `build_evidence_export`. The `transcription_quality_warnings` payload folds into `execution_attempt.normalized_metadata`.

Deleting stored images is not involved; the 58 already-ingested rotated images are rotated **in place** by the migration. No live integrity check is invalidated: `Source` has no digest column, and the only stored digests are `ExecutionAttempt.request_manifest_sha256` - a hash of the request manifest, correct as history - and `ProcessingArtifact.payload_sha256`, which is removed with the table.

### 3. SourceService Decomposition ([MED-14])

`services/sources.py` is **1,389 lines** and `SourceService` owns `Source`, `JobSource`, `ExecutionAttempt`, and `ProcessingArtifact`.

Item 2 removes the `ProcessingArtifact` responsibility by **deletion rather than extraction**. The previously planned `services/artifacts.py` is therefore cancelled - extracting ~283 lines into a new module and then deleting that module would be wasted work.

What remains is the `ExecutionAttempt` cluster, moved to **`services/evidence.py` (~174 lines)**: `read_latest_execution_attempt` (216-245) with its `LatestExecutionAttempt` read model, `promote_machine_attempt` (679-710), `list_execution_attempts` (710-732), and `build_evidence_export` (1015-1107).

**`update_job_source_transcription` stays in `sources.py`.** The V4.6 deferral note proposed moving it to `workflows.py` as orchestration; that proposal is not adopted. The method writes `JobSource` and `ExecutionAttempt` inside one session scope and derives `attempt_number` at lines 595-600, and `services.instructions.md:63-65` requires the transcript update and the paired terminal status change to commit or roll back together. `services.instructions.md:72` assigns session-aware write helpers to services and commit-boundary control to orchestration, so the current placement already satisfies the instruction file. Splitting the two writes across modules is the most plausible way that atomicity later gets broken. The method will shrink under item 1, since several of the fields it writes cease to exist.

Expected result: `sources.py` lands near **900 lines**.

### 4. Run-Time Measurement Window (review log [55])

`services/workflows.py:221` sets `monotonic_started_at` **before** provider-input preparation and the `session.commit()` at line 228. Line 251 computes `elapsed_seconds` from it. But the `asyncio.wait_for` timeout at lines 240-249 wraps **only** `_call_transcriber`.

`duration_ms` therefore measures a strictly wider window than the budget that governs it. This is observable in the migrated data: three historical `local_timeout` rows recorded 20.4 / 20.8 / 22.0 s against a 20.0 s timeout.

In scope: either record provider latency as a distinct value, or move `monotonic_started_at` to immediately before the `wait_for`. Whichever is chosen, the resulting figure must be the quantity the timeout actually governs. Item 2 also removes normalization from this window entirely, which shrinks the discrepancy but does not by itself fix it.

This item **must land before any V4.8 telemetry presentation work**.

### 5. Worker Exception Handling (review log [8])

`worker.py:96-106`, `handle_worker_exceptions`, catches bare `Exception`, logs it, and suppresses it. A programming error inside the worker loop is therefore indistinguishable from a transient provider fault and is retried silently with no UI signal.

In scope: distinguish genuinely retriable faults from programming errors, and ensure a non-retriable error surfaces rather than looping. Retry counting and terminal-state transitions remain governed by `services.instructions.md:63-65`.

### 6. CI Enforcement of the Quality Gate ([HIGH-06], review log [40])

`.github/workflows/` is empty. The `ruff check` and `ty check` gate established in V4.6 Phase 7 exists only in `.pre-commit-config.yaml`, which is inert until a developer runs `pre-commit install`.

In scope: a CI workflow running `ruff check`, `ty check`, and `pytest` on push and pull request, using the same commands as the local hooks so the two cannot drift.

## Out of Scope

- **All image and media presentation work.** Pan and zoom on Source Detail, the homepage gallery, multi-portrait support, image descriptions, and background wallpaper are V4.8.
- **The model-performance rollup** (review log [54]). It depends on item 4 and is a new user-facing view.
- **Reducing `update_job_source_transcription`.** See section 3.
- **Bit-exact image preservation.** Considered and rejected; see Confirmed Operating Context.
- **PostgreSQL cutover.**
- **Re-tuning `WORKER_PROVIDER_TIMEOUT_SECONDS` or `WORKER_MAX_RETRIES`.** Calibrated 2026-08-18 against measured per-model durations.
- **Removing slow models from `PROVIDER_MODELS`** (review log [53]). A configuration judgement, deliberately left with the operator.
- **Any new feature.**

## Locked Design Decisions

### A. Cleanup Only

V4.7 changes structure and correctness. It does not change what the application does for a user. If a change would be visible on a page as new capability, it belongs in V4.8.

### B. One Home Per Fact

After V4.7, any given piece of evidence is stored in exactly one place. `job_source` holds membership and state; `execution_attempt` holds evidence. Denormalized convenience copies are not reintroduced, and if a read becomes awkward the fix is a query or a read model, not a duplicated column.

### C. Delete Before Refactor

Item 2 deletes the artifact subsystem before item 3 restructures what remains. Code scheduled for deletion is never extracted, renamed, or moved first.

### D. Simplicity Over Edge-Case Management

Where two approaches both satisfy the requirement, the one with fewer moving parts wins. This is why rotation uses Pillow rather than a lossless DCT transform, and why no archival master is kept.

### E. Measurement Before Presentation

Item 4 precedes all V4.8 telemetry work. A dashboard built on a conflated metric looks authoritative and quietly misleads.

### F. One Migration, Backed Up

All schema and data changes land in a single `tools/migrate_v46_to_v47.py`: idempotent, never invoked at startup, never run by the test suite, following the `tools/migrate_v45_to_v46.py` conventions. `data/transcription.db` **and** `data/documents/` are backed up before it runs, because the image backfill rewrites files in place.

### G. The Instruction Files Are the Standard

`.github/instructions/services.instructions.md` and `ui.instructions.md` govern. Where this document and an instruction file disagree, the instruction file wins.

## Acceptance Criteria

- `job_source` carries exactly `id`, `job_id`, `source_id`, `status`; every evidence read resolves through `execution_attempt`.
- `JobSourceStatus.CANCELLED` exists and cancellation no longer writes free text into a removed column.
- `job_source.status` and `execution_attempt.status` persist with one spelling, and existing rows are consistent.
- The `processing_artifact` table, its model, and its service cluster no longer exist.
- Newly uploaded images are stored upright with no EXIF orientation tag, and the 58 pre-existing rotated images have been backfilled.
- `services/sources.py` is materially smaller, with `ExecutionAttempt` responsibilities in `services/evidence.py` and `update_job_source_transcription` unmoved.
- The recorded duration reflects only the operation the timeout governs.
- A programming error in the worker loop is distinguishable from a provider fault.
- `ruff check` reports no findings; `ty check` reports **0 diagnostics**, the V4.6 exit state.
- The full test suite passes.
- CI runs the same `ruff` / `ty` / `pytest` gate as the local hooks.
- No new user-facing behavior.

## Scope Freeze Gate

This boundary is frozen. Adding an item requires a finding ID or a review-log entry, and an explicit note recording the addition.

## Related Local References

- [V4.7 Implementation Plan](implementation_plan_v4_7.md)
- [Architecture & Code Review Report](../architecture_code_review_2026-08-17.md) - finding IDs
- [V4.6 Review Log](../ver4.6/review_log_v4_6.md) - resolves the `review log [N]` citations used throughout this document
- [V4.6 Scope Boundary](../ver4.6/scope_boundary_v4_6.md) - the baseline this release builds on
- [V4.6 Implementation Plan](../ver4.6/implementation_plan_v4_6.md)
- [V4.8 Feature Backlog](../ver4.8/feature_backlog_v4_8.md) - where deferred feature work is parked
- `.github/instructions/services.instructions.md`
- `.github/instructions/ui.instructions.md`