transcription/docs/ver4.6/scope_boundary_v4_6.md

# V4.6 Scope Boundary

This document defines the frozen boundary for architecture conformance, worker reliability, and Source-service
maintainability after the completed V4.5 revision. V4 through V4.5 remain the behavioral and evidence baseline.
V4.6 corrects implementation divergence and reorganizes internal ownership without replacing the V4 data model,
public identity rules, or preservation contracts.

## Purpose

- Freeze complete Job request provenance before work is queued.
- Make job claiming, attempt persistence, and first-success selection safe under concurrency.
- Implement the bounded page-processing concurrency required by the V4 architecture.
- Restore UI, workflow, service, transaction, provider, and filesystem ownership boundaries.
- Separate cohesive Source-domain responsibilities behind compatibility-preserving service contracts.
- Avoid loading large evidence fields for list and candidate-summary views.
- Correct documentation that no longer describes the implemented lifecycle accurately.

## In Scope

### 1. Frozen Job Submission Specification

- Every newly queued Job records its requested provider, model, prompt name, prompt hash, full prompt content,
  and explicitly configured generation parameters before the Job transaction commits.
- Normal transcription and Source retranscription use the same typed submission-specification builder.
- Provider and model values are resolved from validated application configuration.
- Every model submitted through the UI or another application workflow is validated against the configured model
  allowlist.
- A provider response may report a resolved model that differs from the requested model. The resolved provider and
  model belong to the immutable ExecutionAttempt and do not rewrite the Job request specification.
- Existing Jobs with null or incomplete historical fields remain readable and are not backfilled with facts that
  were not frozen at submission.
- Prompt edits continue to affect future Jobs only and never rewrite stored Job or attempt provenance.

### 2. Source Ingestion Workflow Ownership

- Source ingestion is an application workflow, not a UI transaction and not a generic file utility.
- The ingestion workflow owns:
  - Source-media validation.
  - Original-file staging and hashing.
  - Document lookup or creation when applicable.
  - Job creation from a frozen submission specification.
  - Ordered Source creation.
  - JobSource creation.
  - One database commit for the related records.
  - Compensating cleanup when persistence fails before the workflow completes.
- UI pages resolve input, invoke the workflow, notify the worker only after successful commit, and navigate or
  display errors.
- UI pages do not import session scopes, open transactions, execute queries, or commit and roll back persistence.
- Workflow code coordinates service contracts rather than inserting Document, Job, Source, or JobSource rows
  directly.

### 3. Source Media Validation

- Filename extension remains part of the accepted-format policy but is not sufficient proof of media type.
- New Source uploads are validated as non-empty, structurally recognizable JPEG, PNG, TIFF, or PDF content.
- The validated content type must agree with the supported filename extension.
- Raster validation may decode enough content to establish that the file is a supported image, but validation does
  not transform, re-encode, or replace the original bytes.
- PDF validation establishes that the content is a recognizable PDF without rendering, rewriting, or normalizing it.
- Original uploaded bytes, SHA-256 digest, byte size, upload name, and evidence identity remain unchanged.
- Historical Sources are not rejected merely because they predate content-aware validation.

### 4. Atomic Job Claiming

- Claiming a queued Job is one persistence operation that conditionally transitions
  `queued -> processing`.
- Two workers or application instances cannot both successfully claim the same queued Job.
- Claim ordering remains deterministic by creation time and UUID.
- A worker that loses a claim performs no provider call and creates no execution evidence for that Job.
- Claim behavior remains portable across supported SQLite and PostgreSQL deployments.
- Startup recovery may requeue stale processing Jobs according to existing policy, but recovery does not bypass the
  normal claim operation.

### 5. Stateless Provider Execution Evidence

- Request manifest and transport evidence belong to one provider call and are returned through that call's result or
  exception.
- Provider adapters do not expose mutable "current request" or "current response" state for worker persistence.
- Success, HTTP failure, connection failure, local timeout, cancellation, and response-validation failure retain
  their own request and transport evidence without cross-attribution.
- Provider adapters remain responsible for exact provider-boundary capture and safe-header allowlisting.
- Provider calls occur outside database transactions.
- Provider adapter instances may be reused only when their per-call state is concurrency-safe.

### 6. Bounded Page Concurrency

- One claimed multi-page Job may process page provider calls concurrently.
- A validated `worker_concurrency` setting defines the maximum number of in-flight page calls.
- The default concurrency is `2`; allowed values are positive and bounded to a documented safe maximum.
- The worker continues to orchestrate one claimed Job at a time within one application worker loop. V4.6 does not add
  distributed scheduling or simultaneous aggregate processing of multiple Jobs in one process.
- Each page task uses its own database session for preparation and outcome persistence; an AsyncSession is never
  shared across concurrent page tasks.
- Provider-input preparation completes and commits before that page's provider call begins.
- Each completed page outcome is committed durably without waiting for every sibling page.
- One page failure does not cancel successful independent pages.
- Aggregate Job status is computed after every started page task reaches a durable outcome or explicit cancellation.
- Cancellation prevents unstarted work and preserves durable outcomes from calls that already completed.
- Concurrency does not change provider cost policy: quality warnings never trigger automatic provider calls.

### 7. Conflict-Safe Attempts and Machine Selection

- Every actual provider call creates exactly one immutable ExecutionAttempt, including failure and timeout outcomes.
- Attempt numbering remains unique for `(job_id, source_id, attempt_number)` and does not use an unprotected
  application-only `max + 1` assumption.
- A duplicate persistence race fails deterministically without overwriting existing evidence.
- The first successful attempt may establish preferred machine output only when the Source has no selected machine
  provenance at the atomic write boundary.
- If another success wins first-selection concurrently, the losing successful attempt remains an unselected
  candidate.
- Candidate promotion continues to update `Source.preferred_execution_attempt_id` and
  `Source.raw_transcription` atomically.
- Human revision remains independent and is never cleared or changed by execution or promotion.

### 8. Processing Artifact Lifecycle

- Processing-artifact content and database metadata have one explicit creation owner.
- When artifact persistence fails or its enclosing commit fails, files created by that operation are removed when
  safely possible and the failure remains visible.
- Temporary artifact files use constrained application-managed paths and are recoverable or removable after
  interruption.
- Artifact integrity verification continues to distinguish unavailable content, malformed metadata, digest mismatch,
  and unsafe external references.
- Deleting artifact metadata and deleting external content is an explicit partial-failure-capable operation.
- A failed external-file deletion is logged with correlation identity and reported as incomplete cleanup rather than
  silent success.
- No cleanup path deletes an artifact still referenced by retained immutable evidence.

### 9. Source Service Responsibility Boundaries

- `SourceService` remains the public Source aggregate facade during V4.6.
- Internally cohesive Source-domain collaborators may own:
  - Source persistence, queries, navigation, and revisions.
  - Source media policy and managed original-file access.
  - Prompt execution snapshots.
  - Provider-input resolution and normalization-artifact association.
  - ExecutionAttempt and JobSource projection persistence.
  - Processing-artifact storage and integrity.
  - Evidence export.
  - Provider execution and error translation.
- Candidate promotion remains in the Source aggregate persistence boundary because it atomically changes the Source's
  selected provenance and compatibility text projection.
- Orientation transformation remains in the focused normalization module.
- Deterministic warning analysis remains in the focused quality module.
- Extraction does not create a generic `utils.py`, `helpers.py`, or `common.py` service dumping ground.
- Services remain independent; multi-service behavior stays in workflow functions or workflow objects.

### 10. Read Projections and Query Efficiency

- Source list, Source Detail, candidate summary, and evidence export use purpose-specific read contracts.
- Candidate and list views do not load exact transport bodies, full SDK snapshots, or other large evidence fields
  that they do not render.
- Full evidence bodies remain available through explicit evidence inspection or export.
- Job-filtered Source queries filter in SQL rather than loading every Source and filtering JobSource relationships in
  application memory.
- Source navigation preserves deterministic `(page_number, UUID)` ordering without requiring all sibling Source
  records to be materialized.
- Query optimization does not introduce lazy-loading behavior into detached UI models.

### 11. Lifecycle and Terminology Alignment

- `transcribed` is the canonical successful terminal Job status for the current transcription workflow.
- `completed` remains a readable historical/compatibility status but is not newly emitted by the V4.6 transcription
  worker.
- `partial_success` and `failed` retain their existing aggregate meanings.
- Architecture, schema, requirements, error-handling, UI, and model documentation use the same status terminology.
- JobSource remains the mutable queue and compatibility projection.
- ExecutionAttempt remains the authoritative append-only execution history.

### 12. Compatibility Adapter Isolation

- Existing imports from `transcription.services.transcription` continue to resolve during V4.6.
- Existing upload/storage aliases may remain temporarily at documented compatibility boundaries.
- Compatibility adapters contain no new business logic, persistence, provider calls, or filesystem policy.
- New application code imports the domain-owned implementation rather than adding more compatibility aliases.
- Removal of compatibility imports is deferred until usage is measured and a separate deprecation decision is made.

## Out of Scope

- Replacing JobSource with an event store or removing its compatibility fields.
- Rewriting, deleting, or relabeling historical ExecutionAttempt evidence.
- Changing Source candidate, promotion, or human-revision precedence.
- Changing the V4.5 orientation transformation or quality-warning rules.
- Reprocessing existing Sources solely to apply new media validation.
- Supporting additional media formats.
- Multiple selectable providers in the UI.
- Distributed queues, external brokers, distributed locks, or multi-host worker coordination.
- Concurrent processing of multiple aggregate Jobs in one application worker loop.
- Automatic paid retry triggered by quality warnings.
- Source page reordering or movement between Documents.
- A generic repository framework, generic service container framework, or generic filesystem utility layer.
- Destructive schema replacement or a V5 API redesign.

## Locked Design Decisions

### A. V4.6 Preserves the V4 Evidence Contract

- Original Source bytes remain primary evidence.
- ExecutionAttempt remains append-only and authoritative.
- JobSource remains a compatibility projection.
- Human revisions remain separate from all machine output.

### B. Requested and Resolved Model Identity Are Different Facts

- Job stores the requested provider and model frozen before queueing.
- ExecutionAttempt stores the provider and model observed for the individual execution.
- Worker completion never rewrites the Job request specification.

### C. Concurrency Is Page-Bounded, Not Job-Distributed

- V4.6 adds bounded page concurrency within one claimed Job.
- One worker loop continues to own one Job aggregate at a time.
- Every page task owns its own session and durable outcome transaction.

### D. The Source Facade Remains Stable

- Source internals may be extracted, but callers retain a stable SourceService contract during migration.
- Candidate promotion stays transactionally close to Source persistence.
- Provider, prompt, artifact, and export concerns become replaceable collaborators rather than additional aggregates.

### E. Filesystem and Database Partial Failure Is Explicit

- Cross-resource operations cannot rely on a relational transaction alone.
- Observable commit failures receive compensation.
- Unavoidable cleanup failure is reported and remains diagnosable rather than being represented as complete success.

### F. Refactoring Does Not Rewrite History

- Existing nulls remain null unless the operator performs a separately specified maintenance action.
- Historical SDK snapshots are not reclassified as transport evidence.
- Existing preferred-machine projections and human revisions remain unchanged.

## Data and Compatibility Policy

- V4.6 prefers no schema change where conditional writes and focused read projections are sufficient.
- Any required schema addition is additive and portable across SQLite and PostgreSQL.
- Existing Job, Source, JobSource, ExecutionAttempt, ProcessingArtifact, Document, and Person identifiers remain valid.
- Existing Source files and artifact references retain their stored meaning.
- Existing `completed` Job rows remain readable.
- Existing API response fields are not removed by this revision.
- Compatibility adapters remain read- and call-compatible while internal ownership moves.
- Database, integration, and UI tests use confirmed isolated data and never modify `data/transcription.db`.
- Potentially destructive tests run only through `tools/run_destructive_tests.py`.

## Acceptance Criteria

1. Every newly queued Job has complete requested provider, model, prompt, prompt hash, and parameter provenance before
   worker execution.
2. A provider-reported resolved model is stored on the ExecutionAttempt and does not change the Job's requested model.
3. Normal and retranscription Job creation reject models outside the configured allowlist.
4. Upload UI code opens no database session and performs no direct persistence.
5. A failed ingestion transaction removes files staged by that failed operation and leaves no partial database rows.
6. Supported Source content is validated beyond its extension without transforming original bytes.
7. Two concurrent claimers cannot both claim or execute the same queued Job.
8. Overlapping provider calls retain distinct request manifests and transport evidence.
9. Page execution concurrency never exceeds the configured bound and uses no shared AsyncSession.
10. Completed pages are durable while slower sibling calls remain in flight.
11. Every actual provider call creates one immutable attempt and no duplicate attempt number.
12. Concurrent first successes result in one preferred attempt and retained candidates for all other successes.
13. Commit failure during external artifact creation does not silently leave an untracked artifact.
14. External cleanup failure is visible and does not delete retained evidence metadata deceptively.
15. Source list and candidate-summary views do not load exact transport bodies.
16. Job-filtered Source lists filter in the database and preserve deterministic ordering.
17. SourceService callers remain compatible while focused collaborators become independently testable.
18. The worker emits `transcribed`, `partial_success`, or `failed` consistently, and historical `completed` rows remain
    readable.
19. Previous Jobs, attempts, artifacts, Source projections, and human revisions remain unchanged by the upgrade.
20. Verification uses isolated databases, fake providers, and does not modify curated Source or evidence files.

## Scope Freeze Gate

V4.6 is sufficiently defined to begin implementation:

- Submission-time versus execution-time provenance ownership is resolved.
- UI, workflow, service, and transaction boundaries are resolved.
- Claiming, attempt persistence, and first-selection concurrency rules are resolved.
- Page-level concurrency scope and default bound are resolved.
- Source extraction seams and compatibility policy are resolved.
- Media validation, artifact compensation, and query-projection expectations are resolved.
- Successful Job status terminology is resolved.
- V4.2 and V4.5 evidence, candidate, and revision contracts remain unchanged.

Any expansion into distributed work scheduling, additional providers or media types, evidence-model replacement,
candidate-policy changes, or incompatible API/schema redesign requires an explicit V4.6 scope amendment or V5.

## Related Local References

- [V4.6 Implementation Plan](implementation_plan_v4_6.md)
- [V4.5 Scope Boundary](../ver4.5/scope_boundary_v4_5.md)
- [V4.5 Implementation Plan](../ver4.5/implementation_plan_v4_5.md)
- [V4 Architecture](../ver4/architecture_v4.md)
- [V4 Schema](../ver4/schema_v4.md)
- [V4 Requirements](../ver4/requirements_v4.md)
- [V4 Error Handling Policy](../ver4/error_handling_v4.md)
- [AI Evidence and Provenance Invariant](../invariant/ai_evidence_and_provenance.md)
- [Error Handling Invariant](../invariant/error_handling.md)