transcription/docs/ver4.6/implementation_plan_v4_6.md

Implementation Plan (Version 4.6)

Goal

Bring the completed V4.5 implementation into conformance with the documented V4 service, provenance, transaction, worker-concurrency, and evidence boundaries while preserving all historical Source and execution behavior.

Planning Status

• V4.5 is the completed implementation baseline.
• The V4.6 scope is frozen and sufficiently detailed to begin implementation.
• V4.6 is a compatibility-preserving architecture and reliability revision, not a V5 redesign.
• Scope additions require an explicit amendment or a later revision.

Planning Constraints

• Original Source bytes and historical evidence remain immutable.
• Existing JobSource compatibility projections remain supported.
• Human revisions remain independent and retain display and print precedence.
• Candidate promotion semantics remain unchanged.
• Provider network work occurs outside database transactions.
• Concurrent page tasks never share an AsyncSession.
• UI pages do not own persistence transactions.
• Refactoring proceeds behind stable public service contracts.
• New shared code has a cohesive domain or infrastructure owner; no generic helper dumping grounds are introduced.
• Database, integration, and UI tests use confirmed isolated data and never modify data/transcription.db.
• Potentially destructive tests run only through tools/run_destructive_tests.py.

Expected Project Impact

Area	Expected impact
Job submission	Add one typed frozen request specification used by normal and retranscription workflows.
Upload workflow	Move transaction and compensation ownership from the UI and direct persistence functions into orchestration.
Source media	Validate content and extension consistently while preserving original bytes.
Job service	Add atomic claim behavior and stable lifecycle transitions.
Provider adapters	Remove mutable per-call evidence state and carry evidence through results and errors.
Worker workflows	Add bounded concurrent page execution with isolated sessions and durable page outcomes.
Source service	Extract cohesive collaborators behind the existing SourceService facade.
Evidence persistence	Make attempt numbering, first selection, artifact creation, and cleanup conflict-safe.
Read paths	Add lightweight Source and attempt projections that defer large evidence fields.
Compatibility	Isolate legacy transcription and storage aliases without adding new behavior.
Documentation	Align status terminology, ownership, concurrency, and provenance descriptions.
Tests	Add failure-injection, concurrency, query-shape, compatibility, and preservation coverage.

Implementation Phases

1. Add Characterization and Failure-Injection Coverage

• Record the current externally visible Source, Job, JobSource, attempt, artifact, revision, candidate, and export behavior before extraction.
• Add tests proving:
  • Historical Jobs with missing request fields remain readable.
  • JobSource SDK snapshots retain their existing evidence label.
  • Source preferred-machine fields and human revisions survive failed processing.
  • Per-page outcomes remain durable before sibling pages finish.
  • Candidate promotion updates both Source projection fields atomically.
• Add reusable test doubles for:
  • Provider success, response failure, connection failure, timeout, and cancellation.
  • Database flush and commit failure at named workflow phases.
  • Filesystem write, replace, and unlink failure.
  • Controlled overlapping page calls.
• Keep all providers fake and all databases and filesystem roots isolated.
• Use these tests as behavior gates for every later extraction.

2. Define the Frozen Job Submission Specification

• Move runtime prompt loading, prompt hashing, requested provider/model resolution, and explicit parameter-state capture into a focused prompt/job-submission module.
• Define one immutable typed value containing:
  • Requested provider.
  • Requested model.
  • Prompt name and SHA-256.
  • System and user prompt content.
  • Temperature and top-p values.
  • Explicitly supplied versus omitted state where applicable.
• Reuse the constrained PromptStore's path policy rather than maintaining a second incompatible prompt-path policy.
• Keep prompt editing, backup, and recovery behavior in the existing prompt-storage owner.
• Validate model selection against provider_models for both normal transcription and retranscription.
• Populate all new Job request fields before adding the Job to the session.
• Stop mutating Job provider and model during attempt persistence.
• Store provider-reported resolved identity on ExecutionAttempt only.
• Retain compatibility reads for historical Jobs whose request fields are null.
• Add tests for pre-queue completeness, allowlist rejection, prompt-edit isolation, requested/resolved model differences, and historical null behavior.

3. Establish Content-Aware Source Media Policy

• Move accepted Source extensions, canonical MIME mapping, validation, and managed payload loading into one Source-media module.
• Validate filename and non-empty content first.
• Validate JPEG, PNG, and TIFF using bounded raster decoding appropriate to the existing Pillow dependency.
• Validate PDF structure without rendering or modifying the file.
• Reject a supported extension whose content is corrupt or belongs to a different supported type.
• Return a typed validated-media result containing canonical media type, digest, byte size, and original bytes or a stable staged reference.
• Ensure hashing uses the exact bytes later persisted as the original Source.
• Keep orientation normalization separate and downstream of original-media validation.
• Preserve compatibility aliases for existing MIME and payload-loading imports.
• Add fixtures for valid, corrupt, truncated, mismatched, uppercase-extension, empty, and unsupported content.

4. Replace Direct Upload Persistence with an Ingestion Workflow

• Introduce a Source-ingestion workflow that accepts validated inputs and service dependencies.
• Generate Document and Source identifiers before file placement so managed relative paths remain deterministic.
• Stage all Source files beneath the configured application root.
• Build the frozen Job submission specification before starting database mutation.
• Open one workflow-owned transaction.
• Use DocumentService, JobService, and SourceService session-aware writes to create:
  • The Document when required.
  • The queued Job.
  • Ordered Sources.
  • Pending JobSource projections.
• Commit once after every record has flushed successfully.
• Delete files staged by the operation on validation, flush, or commit failure.
• Return a typed result only after commit succeeds.
• Move worker notification after successful workflow return.
• Remove session-scope imports and direct persistence ownership from the Jobs UI.
• Keep old storage entry points as thin compatibility adapters that invoke the workflow.
• Add tests for deterministic ordering, page numbering, duplicate names, every failure phase, cleanup, commit-before-notify, and no direct UI persistence.

5. Add Atomic Job Claiming and Canonical Lifecycle Semantics

• Add claim_next_queued_job to JobService or a Job-owned persistence collaborator.
• Select candidates deterministically by creation time and UUID.
• Transition one candidate using a conditional write that succeeds only while status is queued.
• Retry candidate selection when another claimant wins, without issuing provider work for the lost claim.
• Commit the processing transition before provider-input preparation.
• Route worker processing exclusively through the claim operation.
• Preserve stale-processing recovery and require recovered Jobs to be claimed again.
• Define transcribed as the newly emitted all-success terminal status.
• Continue reading completed as a historical terminal status.
• Add tests with concurrent claimers on SQLite and the portable SQL path used for PostgreSQL.
• Add a lifecycle transition-table test covering queued, processing, transcribed, partial success, failed, cancellation, resubmission, and stale recovery.

6. Make Provider Calls Stateless

• Extend provider success and error contracts so each call directly carries:
  • RequestManifest.
  • TransportEvidence or explicit no-response evidence.
  • Failure phase.
  • Parsed SDK snapshot when available.
  • Normalized metadata when available.
• Remove current_request_manifest and current_transport_evidence from provider adapters and worker persistence.
• Ensure local timeout and cancellation retain the call-specific request manifest without reading mutable adapter state.
• Keep response-body capture and safe-header filtering in the OpenRouter adapter.
• Preserve the distinction between exact OpenRouter-boundary response bytes, SDK snapshot, and normalized fields.
• Decide adapter reuse by concurrency safety; create per-call adapters if the SDK client cannot safely support concurrent calls.
• Add overlapping-call tests in which success, HTTP failure, timeout, and no-response failure complete in different orders.
• Assert that no call receives another call's source digest, prompt hash, response body, or identifiers.

7. Extract Source-Owned Evidence and Artifact Collaborators

• Keep SourceService as the public facade while moving cohesive implementation behind it.
• Extract Source persistence and query behavior first without changing method contracts.
• Extract ExecutionAttempt and JobSource compatibility persistence into a Source-owned evidence writer.
• Extract processing-artifact file, integrity, and metadata behavior into a Source-owned artifact store.
• Extract evidence export into a read-only projection builder.
• Extract provider-input preparation so it coordinates:
  • Managed original Source access.
  • Orientation normalization.
  • Derivative artifact creation.
  • Exact SourceEvidenceReference construction.
• Keep candidate promotion in Source persistence.
• Keep normalization and quality analysis in their existing focused modules.
• Make dependencies explicit through typed constructor parameters rather than importing service globals.
• Do not allow extracted collaborators to call unrelated aggregate services.
• Add contract tests showing SourceService behavior is unchanged while collaborators can be tested independently.

8. Make Artifact Creation and Cleanup Explicit

• Define one artifact-creation operation that owns temporary path creation, atomic file replacement, database metadata, and compensation.
• Track every path created before commit within the owning workflow.
• On flush or commit failure, remove newly created content and surface cleanup failure separately when removal also fails.
• Constrain temporary and final paths beneath the artifact root and preserve safe relative references.
• Add startup or explicit maintenance cleanup for abandoned temporary files without deleting final referenced artifacts.
• For deletion:
  • Verify retention policy and references before mutation.
  • Commit metadata changes according to the explicit deletion contract.
  • Attempt external cleanup.
  • Report incomplete cleanup with correlation identity when unlink fails.
• Do not silently skip unsafe references; classify them as validation or conflict failures as appropriate.
• Add tests for write, replace, flush, commit, unlink, cancellation, missing file, digest mismatch, and unsafe reference failures.

9. Make Attempt Persistence and First Selection Conflict-Safe

• Stop relying solely on an unprotected max(attempt_number) + 1 calculation.
• Allocate or insert attempt numbers inside the protected Job processing boundary.
• Treat the database uniqueness constraint as a final integrity guard and translate conflicts deterministically.
• Ensure every completed provider call either persists one immutable attempt or leaves a visible persistence failure that does not masquerade as provider success.
• Perform first-success Source selection with a conditional update requiring both preferred provenance and machine projection to remain unset.
• When first-selection loses a race, retain the new successful attempt as a candidate.
• Keep explicit candidate promotion as a separate atomic command.
• Never modify earlier attempt text, transport evidence, warnings, or artifact associations.
• Add concurrency tests for duplicate attempt allocation, simultaneous successes, success/failure overlap, candidate promotion, and transaction rollback.

10. Add Bounded Concurrent Page Processing

• Add worker_concurrency to Settings with default 2, positive validation, and a documented safe upper bound.
• Refactor the single-page body of process_queued_job into one typed page-attempt operation.
• Prepare each page's provider input and persist any derivative using its own session and committed preparation phase.
• Release all database transactions before awaiting provider network work.
• Use an asyncio semaphore or TaskGroup-based worker pattern to cap in-flight page calls.
• Give every page outcome persistence operation its own session and transaction.
• Shield only the minimum durable outcome write required to preserve a completed provider call during cancellation.
• Collect typed success, failure, and cancellation outcomes without sharing mutable lists across tasks.
• Compute the aggregate status after all started tasks reach durable outcomes.
• Stop scheduling unstarted pages after cancellation or an externally terminal Job transition.
• Preserve deterministic source ordering for scheduling and deterministic aggregate reporting independent of completion order.
• Add tests for concurrency bounds 1, 2, and a larger configured value; out-of-order completion; mixed outcomes; timeout; cancellation; external stop; durable early completion; and no shared session.

11. Introduce Purpose-Specific Read Projections

• Add a Source-list projection containing only fields needed by the Sources table.
• Filter Document and Job scopes in SQL.
• Add a Source-detail projection or explicit eager-loading contract for the relationships rendered by the page.
• Add an attempt-summary projection containing candidate metadata, transcription text or an explicit preview policy, warning summary, and preferred-selection identity.
• Defer transport body, SDK snapshot, request manifest, and other large evidence values from summaries.
• Keep one explicit full-attempt evidence read for detailed inspection.
• Keep evidence export complete and integrity-verified.
• Replace all-sibling materialization in Source navigation with predecessor/successor queries using page number and UUID.
• Add query-count and unloaded-attribute assertions so future changes cannot reintroduce accidental evidence loading.
• Preserve deterministic ordering and avoid lazy database access from UI components.

12. Isolate Compatibility Adapters and Cohesive Shared Policies

• Keep services/transcription.py as a documented compatibility facade with imports and aliases only.
• Keep legacy upload/storage aliases as thin calls into the new ingestion and media owners.
• Mark compatibility entry points in tests so removal can be evaluated separately.
• Consolidate Document Type and Person Role label normalization in a registry-domain policy module.
• Share canonical evidence digest primitives only where serialization and digest meaning are identical.
• Do not merge prompt backup writing, original Source storage, and immutable artifact storage into one generic helper; their lifecycle and recovery semantics remain domain-owned.
• Remove unused service-base state only after confirming no external caller depends on it.
• Add import-compatibility tests and direct tests for each extracted policy.

13. Align UI and API Boundaries

• Remove machine-local stored paths from routine Source Detail presentation.
• Keep stored filename and evidence-safe identifiers available.
• Ensure normal Job creation uses the configured provider and allowlisted model selector rather than arbitrary text input.
• Keep retranscription Source and Document context locked.
• Keep evidence export explicit and avoid loading export-only content during page render.
• Continue translating AppError through established UI and API presenters without parsing error messages.
• Keep sessions, session factories beyond dependency construction, and persistence operations out of reusable components.
• Add UI tests for allowlisted selection, no path disclosure, unchanged candidate actions, and error preservation.

14. Align Authoritative Documentation

• Update V4 architecture for:
  • Frozen Job request versus resolved attempt identity.
  • Atomic Job claims.
  • Bounded page concurrency and isolated page sessions.
  • Source facade and internal collaborators.
  • Explicit artifact lifecycle ownership.
• Update V4 schema documentation only for actual additive schema changes.
• Update V4 requirements and verification mapping for atomic claims, media validation, and concurrency tests.
• Update V4 error handling for filesystem/database partial cleanup.
• Define transcribed as the current all-success terminal status and completed as historical compatibility.
• Correct stale V3 module descriptions and UI documentation that exposes or promises machine-local paths.
• Update documentation together with the behavior it describes; do not prestate unimplemented behavior as complete.

15. Verification and Preservation Audit

• Run focused pure tests for submission specs, media validation, provider contracts, query projections, and policies.
• Run isolated service tests for claims, attempts, artifacts, ingestion, candidate selection, and compatibility.
• Run worker tests with fake providers and controlled concurrent completion.
• Run UI and API tests against isolated application state.
• Run schema tests only through the required destructive-test wrapper when applicable.
• Run the broader non-external regression suite after focused targets pass.
• Verify that tests did not change:
  • data/transcription.db.
  • Curated Source files.
  • Existing artifact files.
  • Prompt files outside temporary fixtures.
• Perform an explicit preservation audit showing that upgrading does not rewrite existing Job, Source, JobSource, ExecutionAttempt, ProcessingArtifact, or revision content.

Delivery Order

1. Characterization and failure-injection tests.
2. Frozen submission specification and prompt ownership.
3. Content-aware Source media validation.
4. Ingestion workflow and UI transaction-boundary correction.
5. Atomic Job claiming and lifecycle terminology.
6. Stateless provider evidence.
7. Source evidence/artifact/provider-input extraction behind SourceService.
8. Artifact compensation and conflict-safe attempt/selection persistence.
9. Purpose-specific read projections.
10. Bounded concurrent page processing.
11. Compatibility isolation, UI/API alignment, and documentation.
12. Full preservation and regression verification.

Done Criteria

• All V4.6 acceptance criteria are implemented and testable.
• New Jobs contain complete request provenance before queueing and are not rewritten by execution.
• Every submitted model is allowlisted and requested/resolved identities remain distinct.
• Upload UI code owns no database transaction.
• Source content validation verifies supported media without modifying originals.
• One queued Job can be claimed only once.
• Provider evidence is call-local and safe under overlapping execution.
• Page calls run concurrently within the configured bound and never share sessions.
• Every actual provider call has one immutable attempt.
• Concurrent first successes preserve one preferred result and every other success as a candidate.
• Artifact commit and cleanup failures are visible and compensated where safely possible.
• Source list and candidate views avoid loading full evidence bodies.
• SourceService remains compatible while extracted responsibilities are independently testable.
• Human revisions, preferred-machine projections, prior attempts, JobSource snapshots, and artifacts retain their original meaning.
• The worker consistently emits transcribed, partial_success, or failed for new processing.
• Verification uses isolated data, fake providers, and does not modify operator or curated evidence data.

Related Local References

• V4.6 Scope Boundary
• V4.5 Scope Boundary
• V4.5 Implementation Plan
• V4 Architecture
• V4 Schema
• V4 Requirements
• V4 Error Handling Policy
• AI Evidence and Provenance Invariant
• Error Handling Invariant