# Implementation Plan (Version 4.6)

## Goal

Bring the completed V4.5 implementation into conformance with the documented V4 service, provenance, transaction,
worker-concurrency, and evidence boundaries while preserving all historical Source and execution behavior.

## Planning Status

- V4.5 is the completed implementation baseline.
- The V4.6 scope is frozen and sufficiently detailed to begin implementation.
- V4.6 is a compatibility-preserving architecture and reliability revision, not a V5 redesign.
- Scope additions require an explicit amendment or a later revision.

## Planning Constraints

- Original Source bytes and historical evidence remain immutable.
- Existing JobSource compatibility projections remain supported.
- Human revisions remain independent and retain display and print precedence.
- Candidate promotion semantics remain unchanged.
- Provider network work occurs outside database transactions.
- Concurrent page tasks never share an AsyncSession.
- UI pages do not own persistence transactions.
- Refactoring proceeds behind stable public service contracts.
- New shared code has a cohesive domain or infrastructure owner; no generic helper dumping grounds are introduced.
- Database, integration, and UI tests use confirmed isolated data and never modify `data/transcription.db`.
- Potentially destructive tests run only through `tools/run_destructive_tests.py`.

## Expected Project Impact

| Area | Expected impact |
| --- | --- |
| Job submission | Add one typed frozen request specification used by normal and retranscription workflows. |
| Upload workflow | Move transaction and compensation ownership from the UI and direct persistence functions into orchestration. |
| Source media | Validate content and extension consistently while preserving original bytes. |
| Job service | Add atomic claim behavior and stable lifecycle transitions. |
| Provider adapters | Remove mutable per-call evidence state and carry evidence through results and errors. |
| Worker workflows | Add bounded concurrent page execution with isolated sessions and durable page outcomes. |
| Source service | Extract cohesive collaborators behind the existing SourceService facade. |
| Evidence persistence | Make attempt numbering, first selection, artifact creation, and cleanup conflict-safe. |
| Read paths | Add lightweight Source and attempt projections that defer large evidence fields. |
| Compatibility | Isolate legacy transcription and storage aliases without adding new behavior. |
| Documentation | Align status terminology, ownership, concurrency, and provenance descriptions. |
| Tests | Add failure-injection, concurrency, query-shape, compatibility, and preservation coverage. |

## Implementation Phases

### 1. Add Characterization and Failure-Injection Coverage

- Record the current externally visible Source, Job, JobSource, attempt, artifact, revision, candidate, and export
  behavior before extraction.
- Add tests proving:
  - Historical Jobs with missing request fields remain readable.
  - JobSource SDK snapshots retain their existing evidence label.
  - Source preferred-machine fields and human revisions survive failed processing.
  - Per-page outcomes remain durable before sibling pages finish.
  - Candidate promotion updates both Source projection fields atomically.
- Add reusable test doubles for:
  - Provider success, response failure, connection failure, timeout, and cancellation.
  - Database flush and commit failure at named workflow phases.
  - Filesystem write, replace, and unlink failure.
  - Controlled overlapping page calls.
- Keep all providers fake and all databases and filesystem roots isolated.
- Use these tests as behavior gates for every later extraction.

### 2. Define the Frozen Job Submission Specification

- Move runtime prompt loading, prompt hashing, requested provider/model resolution, and explicit parameter-state capture
  into a focused prompt/job-submission module.
- Define one immutable typed value containing:
  - Requested provider.
  - Requested model.
  - Prompt name and SHA-256.
  - System and user prompt content.
  - Temperature and top-p values.
  - Explicitly supplied versus omitted state where applicable.
- Reuse the constrained PromptStore's path policy rather than maintaining a second incompatible prompt-path policy.
- Keep prompt editing, backup, and recovery behavior in the existing prompt-storage owner.
- Validate model selection against `provider_models` for both normal transcription and retranscription.
- Populate all new Job request fields before adding the Job to the session.
- Stop mutating Job provider and model during attempt persistence.
- Store provider-reported resolved identity on ExecutionAttempt only.
- Retain compatibility reads for historical Jobs whose request fields are null.
- Add tests for pre-queue completeness, allowlist rejection, prompt-edit isolation, requested/resolved model differences, and
  historical null behavior.

### 3. Establish Content-Aware Source Media Policy

- Move accepted Source extensions, canonical MIME mapping, validation, and managed payload loading into one Source-media
  module.
- Validate filename and non-empty content first.
- Validate JPEG, PNG, and TIFF using bounded raster decoding appropriate to the existing Pillow dependency.
- Validate PDF structure without rendering or modifying the file.
- Reject a supported extension whose content is corrupt or belongs to a different supported type.
- Return a typed validated-media result containing canonical media type, digest, byte size, and original bytes or a
  stable staged reference.
- Ensure hashing uses the exact bytes later persisted as the original Source.
- Keep orientation normalization separate and downstream of original-media validation.
- Preserve compatibility aliases for existing MIME and payload-loading imports.
- Add fixtures for valid, corrupt, truncated, mismatched, uppercase-extension, empty, and unsupported content.

### 4. Replace Direct Upload Persistence with an Ingestion Workflow

- Introduce a Source-ingestion workflow that accepts validated inputs and service dependencies.
- Generate Document and Source identifiers before file placement so managed relative paths remain deterministic.
- Stage all Source files beneath the configured application root.
- Build the frozen Job submission specification before starting database mutation.
- Open one workflow-owned transaction.
- Use DocumentService, JobService, and SourceService session-aware writes to create:
  - The Document when required.
  - The queued Job.
  - Ordered Sources.
  - Pending JobSource projections.
- Commit once after every record has flushed successfully.
- Delete files staged by the operation on validation, flush, or commit failure.
- Return a typed result only after commit succeeds.
- Move worker notification after successful workflow return.
- Remove session-scope imports and direct persistence ownership from the Jobs UI.
- Keep old storage entry points as thin compatibility adapters that invoke the workflow.
- Add tests for deterministic ordering, page numbering, duplicate names, every failure phase, cleanup, commit-before-notify,
  and no direct UI persistence.

### 5. Add Atomic Job Claiming and Canonical Lifecycle Semantics

- Add `claim_next_queued_job` to JobService or a Job-owned persistence collaborator.
- Select candidates deterministically by creation time and UUID.
- Transition one candidate using a conditional write that succeeds only while status is queued.
- Retry candidate selection when another claimant wins, without issuing provider work for the lost claim.
- Commit the processing transition before provider-input preparation.
- Route worker processing exclusively through the claim operation.
- Preserve stale-processing recovery and require recovered Jobs to be claimed again.
- Define `transcribed` as the newly emitted all-success terminal status.
- Continue reading `completed` as a historical terminal status.
- Add tests with concurrent claimers on SQLite and the portable SQL path used for PostgreSQL.
- Add a lifecycle transition-table test covering queued, processing, transcribed, partial success, failed, cancellation,
  resubmission, and stale recovery.

### 6. Make Provider Calls Stateless

- Extend provider success and error contracts so each call directly carries:
  - RequestManifest.
  - TransportEvidence or explicit no-response evidence.
  - Failure phase.
  - Parsed SDK snapshot when available.
  - Normalized metadata when available.
- Remove `current_request_manifest` and `current_transport_evidence` from provider adapters and worker persistence.
- Ensure local timeout and cancellation retain the call-specific request manifest without reading mutable adapter state.
- Keep response-body capture and safe-header filtering in the OpenRouter adapter.
- Preserve the distinction between exact OpenRouter-boundary response bytes, SDK snapshot, and normalized fields.
- Decide adapter reuse by concurrency safety; create per-call adapters if the SDK client cannot safely support concurrent
  calls.
- Add overlapping-call tests in which success, HTTP failure, timeout, and no-response failure complete in different
  orders.
- Assert that no call receives another call's source digest, prompt hash, response body, or identifiers.

### 7. Extract Source-Owned Evidence and Artifact Collaborators

- Keep SourceService as the public facade while moving cohesive implementation behind it.
- Extract Source persistence and query behavior first without changing method contracts.
- Extract ExecutionAttempt and JobSource compatibility persistence into a Source-owned evidence writer.
- Extract processing-artifact file, integrity, and metadata behavior into a Source-owned artifact store.
- Extract evidence export into a read-only projection builder.
- Extract provider-input preparation so it coordinates:
  - Managed original Source access.
  - Orientation normalization.
  - Derivative artifact creation.
  - Exact SourceEvidenceReference construction.
- Keep candidate promotion in Source persistence.
- Keep normalization and quality analysis in their existing focused modules.
- Make dependencies explicit through typed constructor parameters rather than importing service globals.
- Do not allow extracted collaborators to call unrelated aggregate services.
- Add contract tests showing SourceService behavior is unchanged while collaborators can be tested independently.

### 8. Make Artifact Creation and Cleanup Explicit

- Define one artifact-creation operation that owns temporary path creation, atomic file replacement, database metadata,
  and compensation.
- Track every path created before commit within the owning workflow.
- On flush or commit failure, remove newly created content and surface cleanup failure separately when removal also
  fails.
- Constrain temporary and final paths beneath the artifact root and preserve safe relative references.
- Add startup or explicit maintenance cleanup for abandoned temporary files without deleting final referenced
  artifacts.
- For deletion:
  - Verify retention policy and references before mutation.
  - Commit metadata changes according to the explicit deletion contract.
  - Attempt external cleanup.
  - Report incomplete cleanup with correlation identity when unlink fails.
- Do not silently skip unsafe references; classify them as validation or conflict failures as appropriate.
- Add tests for write, replace, flush, commit, unlink, cancellation, missing file, digest mismatch, and unsafe reference
  failures.

### 9. Make Attempt Persistence and First Selection Conflict-Safe

- Stop relying solely on an unprotected `max(attempt_number) + 1` calculation.
- Allocate or insert attempt numbers inside the protected Job processing boundary.
- Treat the database uniqueness constraint as a final integrity guard and translate conflicts deterministically.
- Ensure every completed provider call either persists one immutable attempt or leaves a visible persistence failure
  that does not masquerade as provider success.
- Perform first-success Source selection with a conditional update requiring both preferred provenance and machine
  projection to remain unset.
- When first-selection loses a race, retain the new successful attempt as a candidate.
- Keep explicit candidate promotion as a separate atomic command.
- Never modify earlier attempt text, transport evidence, warnings, or artifact associations.
- Add concurrency tests for duplicate attempt allocation, simultaneous successes, success/failure overlap, candidate
  promotion, and transaction rollback.

### 10. Add Bounded Concurrent Page Processing

- Add `worker_concurrency` to Settings with default `2`, positive validation, and a documented safe upper bound.
- Refactor the single-page body of `process_queued_job` into one typed page-attempt operation.
- Prepare each page's provider input and persist any derivative using its own session and committed preparation phase.
- Release all database transactions before awaiting provider network work.
- Use an asyncio semaphore or TaskGroup-based worker pattern to cap in-flight page calls.
- Give every page outcome persistence operation its own session and transaction.
- Shield only the minimum durable outcome write required to preserve a completed provider call during cancellation.
- Collect typed success, failure, and cancellation outcomes without sharing mutable lists across tasks.
- Compute the aggregate status after all started tasks reach durable outcomes.
- Stop scheduling unstarted pages after cancellation or an externally terminal Job transition.
- Preserve deterministic source ordering for scheduling and deterministic aggregate reporting independent of completion
  order.
- Add tests for concurrency bounds `1`, `2`, and a larger configured value; out-of-order completion; mixed outcomes;
  timeout; cancellation; external stop; durable early completion; and no shared session.

### 11. Introduce Purpose-Specific Read Projections

- Add a Source-list projection containing only fields needed by the Sources table.
- Filter Document and Job scopes in SQL.
- Add a Source-detail projection or explicit eager-loading contract for the relationships rendered by the page.
- Add an attempt-summary projection containing candidate metadata, transcription text or an explicit preview policy,
  warning summary, and preferred-selection identity.
- Defer transport body, SDK snapshot, request manifest, and other large evidence values from summaries.
- Keep one explicit full-attempt evidence read for detailed inspection.
- Keep evidence export complete and integrity-verified.
- Replace all-sibling materialization in Source navigation with predecessor/successor queries using page number and UUID.
- Add query-count and unloaded-attribute assertions so future changes cannot reintroduce accidental evidence loading.
- Preserve deterministic ordering and avoid lazy database access from UI components.

### 12. Isolate Compatibility Adapters and Cohesive Shared Policies

- Keep `services/transcription.py` as a documented compatibility facade with imports and aliases only.
- Keep legacy upload/storage aliases as thin calls into the new ingestion and media owners.
- Mark compatibility entry points in tests so removal can be evaluated separately.
- Consolidate Document Type and Person Role label normalization in a registry-domain policy module.
- Share canonical evidence digest primitives only where serialization and digest meaning are identical.
- Do not merge prompt backup writing, original Source storage, and immutable artifact storage into one generic helper;
  their lifecycle and recovery semantics remain domain-owned.
- Remove unused service-base state only after confirming no external caller depends on it.
- Add import-compatibility tests and direct tests for each extracted policy.

### 13. Align UI and API Boundaries

- Remove machine-local stored paths from routine Source Detail presentation.
- Keep stored filename and evidence-safe identifiers available.
- Ensure normal Job creation uses the configured provider and allowlisted model selector rather than arbitrary text input.
- Keep retranscription Source and Document context locked.
- Keep evidence export explicit and avoid loading export-only content during page render.
- Continue translating AppError through established UI and API presenters without parsing error messages.
- Keep sessions, session factories beyond dependency construction, and persistence operations out of reusable components.
- Add UI tests for allowlisted selection, no path disclosure, unchanged candidate actions, and error preservation.

### 14. Align Authoritative Documentation

- Update V4 architecture for:
  - Frozen Job request versus resolved attempt identity.
  - Atomic Job claims.
  - Bounded page concurrency and isolated page sessions.
  - Source facade and internal collaborators.
  - Explicit artifact lifecycle ownership.
- Update V4 schema documentation only for actual additive schema changes.
- Update V4 requirements and verification mapping for atomic claims, media validation, and concurrency tests.
- Update V4 error handling for filesystem/database partial cleanup.
- Define `transcribed` as the current all-success terminal status and `completed` as historical compatibility.
- Correct stale V3 module descriptions and UI documentation that exposes or promises machine-local paths.
- Update documentation together with the behavior it describes; do not prestate unimplemented behavior as complete.

### 15. Verification and Preservation Audit

- Run focused pure tests for submission specs, media validation, provider contracts, query projections, and policies.
- Run isolated service tests for claims, attempts, artifacts, ingestion, candidate selection, and compatibility.
- Run worker tests with fake providers and controlled concurrent completion.
- Run UI and API tests against isolated application state.
- Run schema tests only through the required destructive-test wrapper when applicable.
- Run the broader non-external regression suite after focused targets pass.
- Verify that tests did not change:
  - `data/transcription.db`.
  - Curated Source files.
  - Existing artifact files.
  - Prompt files outside temporary fixtures.
- Perform an explicit preservation audit showing that upgrading does not rewrite existing Job, Source, JobSource,
  ExecutionAttempt, ProcessingArtifact, or revision content.

## Delivery Order

1. Characterization and failure-injection tests.
2. Frozen submission specification and prompt ownership.
3. Content-aware Source media validation.
4. Ingestion workflow and UI transaction-boundary correction.
5. Atomic Job claiming and lifecycle terminology.
6. Stateless provider evidence.
7. Source evidence/artifact/provider-input extraction behind SourceService.
8. Artifact compensation and conflict-safe attempt/selection persistence.
9. Purpose-specific read projections.
10. Bounded concurrent page processing.
11. Compatibility isolation, UI/API alignment, and documentation.
12. Full preservation and regression verification.

## Done Criteria

- All V4.6 acceptance criteria are implemented and testable.
- New Jobs contain complete request provenance before queueing and are not rewritten by execution.
- Every submitted model is allowlisted and requested/resolved identities remain distinct.
- Upload UI code owns no database transaction.
- Source content validation verifies supported media without modifying originals.
- One queued Job can be claimed only once.
- Provider evidence is call-local and safe under overlapping execution.
- Page calls run concurrently within the configured bound and never share sessions.
- Every actual provider call has one immutable attempt.
- Concurrent first successes preserve one preferred result and every other success as a candidate.
- Artifact commit and cleanup failures are visible and compensated where safely possible.
- Source list and candidate views avoid loading full evidence bodies.
- SourceService remains compatible while extracted responsibilities are independently testable.
- Human revisions, preferred-machine projections, prior attempts, JobSource snapshots, and artifacts retain their original
  meaning.
- The worker consistently emits transcribed, partial_success, or failed for new processing.
- Verification uses isolated data, fake providers, and does not modify operator or curated evidence data.

## Related Local References

- [V4.6 Scope Boundary](scope_boundary_v4_6.md)
- [V4.5 Scope Boundary](../ver4.5/scope_boundary_v4_5.md)
- [V4.5 Implementation Plan](../ver4.5/implementation_plan_v4_5.md)
- [V4 Architecture](../ver4/architecture_v4.md)
- [V4 Schema](../ver4/schema_v4.md)
- [V4 Requirements](../ver4/requirements_v4.md)
- [V4 Error Handling Policy](../ver4/error_handling_v4.md)
- [AI Evidence and Provenance Invariant](../invariant/ai_evidence_and_provenance.md)
- [Error Handling Invariant](../invariant/error_handling.md)