From aec89b3a7afdf5ed4edcd9ddec27fedb7c0730f8 Mon Sep 17 00:00:00 2001 From: zoltan57 <40281233+zoltan57@users.noreply.github.com> Date: Mon, 17 Aug 2026 12:45:03 -0500 Subject: [PATCH] Hide V4.6 recommendations after code review --- README.md | 10 +- docs/ver4.6/implementation_plan_v4_6.md | 338 ------------------------ docs/ver4.6/scope_boundary_v4_6.md | 296 --------------------- 3 files changed, 5 insertions(+), 639 deletions(-) delete mode 100644 docs/ver4.6/implementation_plan_v4_6.md delete mode 100644 docs/ver4.6/scope_boundary_v4_6.md diff --git a/README.md b/README.md index 800cbe8..d6a95a3 100644 --- a/README.md +++ b/README.md @@ -34,7 +34,7 @@ Settings are read from CLI arguments first, then environment variables, then `.e When the same setting is provided in multiple places, the value is chosen in this order (highest priority first): -1. CLI arguments (for example `--port 9999`) +1. CLI arguments (for example `--port 8000`) 2. Settings constructor arguments (used mainly in tests) 3. Environment variables 4. `.env` file values @@ -42,7 +42,7 @@ When the same setting is provided in multiple places, the value is chosen in thi Practical examples: -- `--port 9999` overrides both `PORT=8000` in the shell and `PORT=7000` in `.env`. +- `--port 8000` overrides both `PORT=8000` in the shell and `PORT=7000` in `.env`. - `DATABASE__PATH=prod.db` in the shell overrides `DATABASE__PATH=dev.db` in `.env`. #### Server and runtime @@ -110,15 +110,15 @@ WORKER_FAIL_ON_FINISH_REASON_LENGTH=false ### 3) Run the app ```bash -uv run python -m transcription --port 9999 --reload --database.driver sqlite --bootstrap-schema-on-startup +uv run python -m transcription --port 8000 --reload --database.driver sqlite --bootstrap-schema-on-startup ``` This starts the development server with SQLite, creates missing tables, and enables automatic reload. Run `uv run python -m transcription --help` for all CLI options; CLI names use kebab case and nested database options use dot notation, such as `--database.path ./data/transcription.db`. ### 4) Open in browser -- GUI: [http://localhost:9999/ui](http://localhost:9999/ui) -- Health check: [http://localhost:9999/healthz](http://localhost:9999/healthz) +- GUI: [http://localhost:8000/ui](http://localhost:8000/ui) +- Health check: [http://localhost:8000/healthz](http://localhost:8000/healthz) Replace `localhost` with the server's hostname or IP address when connecting from another machine. diff --git a/docs/ver4.6/implementation_plan_v4_6.md b/docs/ver4.6/implementation_plan_v4_6.md deleted file mode 100644 index 9c8d7e5..0000000 --- a/docs/ver4.6/implementation_plan_v4_6.md +++ /dev/null @@ -1,338 +0,0 @@ -# Implementation Plan (Version 4.6) - -## Goal - -Bring the completed V4.5 implementation into conformance with the documented V4 service, provenance, transaction, -worker-concurrency, and evidence boundaries while preserving all historical Source and execution behavior. - -## Planning Status - -- V4.5 is the completed implementation baseline. -- The V4.6 scope is frozen and sufficiently detailed to begin implementation. -- V4.6 is a compatibility-preserving architecture and reliability revision, not a V5 redesign. -- Scope additions require an explicit amendment or a later revision. - -## Planning Constraints - -- Original Source bytes and historical evidence remain immutable. -- Existing JobSource compatibility projections remain supported. -- Human revisions remain independent and retain display and print precedence. -- Candidate promotion semantics remain unchanged. -- Provider network work occurs outside database transactions. -- Concurrent page tasks never share an AsyncSession. -- UI pages do not own persistence transactions. -- Refactoring proceeds behind stable public service contracts. -- New shared code has a cohesive domain or infrastructure owner; no generic helper dumping grounds are introduced. -- Database, integration, and UI tests use confirmed isolated data and never modify `data/transcription.db`. -- Potentially destructive tests run only through `tools/run_destructive_tests.py`. - -## Expected Project Impact - -| Area | Expected impact | -| --- | --- | -| Job submission | Add one typed frozen request specification used by normal and retranscription workflows. | -| Upload workflow | Move transaction and compensation ownership from the UI and direct persistence functions into orchestration. | -| Source media | Validate content and extension consistently while preserving original bytes. | -| Job service | Add atomic claim behavior and stable lifecycle transitions. | -| Provider adapters | Remove mutable per-call evidence state and carry evidence through results and errors. | -| Worker workflows | Add bounded concurrent page execution with isolated sessions and durable page outcomes. | -| Source service | Extract cohesive collaborators behind the existing SourceService facade. | -| Evidence persistence | Make attempt numbering, first selection, artifact creation, and cleanup conflict-safe. | -| Read paths | Add lightweight Source and attempt projections that defer large evidence fields. | -| Compatibility | Isolate legacy transcription and storage aliases without adding new behavior. | -| Documentation | Align status terminology, ownership, concurrency, and provenance descriptions. | -| Tests | Add failure-injection, concurrency, query-shape, compatibility, and preservation coverage. | - -## Implementation Phases - -### 1. Add Characterization and Failure-Injection Coverage - -- Record the current externally visible Source, Job, JobSource, attempt, artifact, revision, candidate, and export - behavior before extraction. -- Add tests proving: - - Historical Jobs with missing request fields remain readable. - - JobSource SDK snapshots retain their existing evidence label. - - Source preferred-machine fields and human revisions survive failed processing. - - Per-page outcomes remain durable before sibling pages finish. - - Candidate promotion updates both Source projection fields atomically. -- Add reusable test doubles for: - - Provider success, response failure, connection failure, timeout, and cancellation. - - Database flush and commit failure at named workflow phases. - - Filesystem write, replace, and unlink failure. - - Controlled overlapping page calls. -- Keep all providers fake and all databases and filesystem roots isolated. -- Use these tests as behavior gates for every later extraction. - -### 2. Define the Frozen Job Submission Specification - -- Move runtime prompt loading, prompt hashing, requested provider/model resolution, and explicit parameter-state capture - into a focused prompt/job-submission module. -- Define one immutable typed value containing: - - Requested provider. - - Requested model. - - Prompt name and SHA-256. - - System and user prompt content. - - Temperature and top-p values. - - Explicitly supplied versus omitted state where applicable. -- Reuse the constrained PromptStore's path policy rather than maintaining a second incompatible prompt-path policy. -- Keep prompt editing, backup, and recovery behavior in the existing prompt-storage owner. -- Validate model selection against `provider_models` for both normal transcription and retranscription. -- Populate all new Job request fields before adding the Job to the session. -- Stop mutating Job provider and model during attempt persistence. -- Store provider-reported resolved identity on ExecutionAttempt only. -- Retain compatibility reads for historical Jobs whose request fields are null. -- Add tests for pre-queue completeness, allowlist rejection, prompt-edit isolation, requested/resolved model differences, and - historical null behavior. - -### 3. Establish Content-Aware Source Media Policy - -- Move accepted Source extensions, canonical MIME mapping, validation, and managed payload loading into one Source-media - module. -- Validate filename and non-empty content first. -- Validate JPEG, PNG, and TIFF using bounded raster decoding appropriate to the existing Pillow dependency. -- Validate PDF structure without rendering or modifying the file. -- Reject a supported extension whose content is corrupt or belongs to a different supported type. -- Return a typed validated-media result containing canonical media type, digest, byte size, and original bytes or a - stable staged reference. -- Ensure hashing uses the exact bytes later persisted as the original Source. -- Keep orientation normalization separate and downstream of original-media validation. -- Preserve compatibility aliases for existing MIME and payload-loading imports. -- Add fixtures for valid, corrupt, truncated, mismatched, uppercase-extension, empty, and unsupported content. - -### 4. Replace Direct Upload Persistence with an Ingestion Workflow - -- Introduce a Source-ingestion workflow that accepts validated inputs and service dependencies. -- Generate Document and Source identifiers before file placement so managed relative paths remain deterministic. -- Stage all Source files beneath the configured application root. -- Build the frozen Job submission specification before starting database mutation. -- Open one workflow-owned transaction. -- Use DocumentService, JobService, and SourceService session-aware writes to create: - - The Document when required. - - The queued Job. - - Ordered Sources. - - Pending JobSource projections. -- Commit once after every record has flushed successfully. -- Delete files staged by the operation on validation, flush, or commit failure. -- Return a typed result only after commit succeeds. -- Move worker notification after successful workflow return. -- Remove session-scope imports and direct persistence ownership from the Jobs UI. -- Keep old storage entry points as thin compatibility adapters that invoke the workflow. -- Add tests for deterministic ordering, page numbering, duplicate names, every failure phase, cleanup, commit-before-notify, - and no direct UI persistence. - -### 5. Add Atomic Job Claiming and Canonical Lifecycle Semantics - -- Add `claim_next_queued_job` to JobService or a Job-owned persistence collaborator. -- Select candidates deterministically by creation time and UUID. -- Transition one candidate using a conditional write that succeeds only while status is queued. -- Retry candidate selection when another claimant wins, without issuing provider work for the lost claim. -- Commit the processing transition before provider-input preparation. -- Route worker processing exclusively through the claim operation. -- Preserve stale-processing recovery and require recovered Jobs to be claimed again. -- Define `transcribed` as the newly emitted all-success terminal status. -- Continue reading `completed` as a historical terminal status. -- Add tests with concurrent claimers on SQLite and the portable SQL path used for PostgreSQL. -- Add a lifecycle transition-table test covering queued, processing, transcribed, partial success, failed, cancellation, - resubmission, and stale recovery. - -### 6. Make Provider Calls Stateless - -- Extend provider success and error contracts so each call directly carries: - - RequestManifest. - - TransportEvidence or explicit no-response evidence. - - Failure phase. - - Parsed SDK snapshot when available. - - Normalized metadata when available. -- Remove `current_request_manifest` and `current_transport_evidence` from provider adapters and worker persistence. -- Ensure local timeout and cancellation retain the call-specific request manifest without reading mutable adapter state. -- Keep response-body capture and safe-header filtering in the OpenRouter adapter. -- Preserve the distinction between exact OpenRouter-boundary response bytes, SDK snapshot, and normalized fields. -- Decide adapter reuse by concurrency safety; create per-call adapters if the SDK client cannot safely support concurrent - calls. -- Add overlapping-call tests in which success, HTTP failure, timeout, and no-response failure complete in different - orders. -- Assert that no call receives another call's source digest, prompt hash, response body, or identifiers. - -### 7. Extract Source-Owned Evidence and Artifact Collaborators - -- Keep SourceService as the public facade while moving cohesive implementation behind it. -- Extract Source persistence and query behavior first without changing method contracts. -- Extract ExecutionAttempt and JobSource compatibility persistence into a Source-owned evidence writer. -- Extract processing-artifact file, integrity, and metadata behavior into a Source-owned artifact store. -- Extract evidence export into a read-only projection builder. -- Extract provider-input preparation so it coordinates: - - Managed original Source access. - - Orientation normalization. - - Derivative artifact creation. - - Exact SourceEvidenceReference construction. -- Keep candidate promotion in Source persistence. -- Keep normalization and quality analysis in their existing focused modules. -- Make dependencies explicit through typed constructor parameters rather than importing service globals. -- Do not allow extracted collaborators to call unrelated aggregate services. -- Add contract tests showing SourceService behavior is unchanged while collaborators can be tested independently. - -### 8. Make Artifact Creation and Cleanup Explicit - -- Define one artifact-creation operation that owns temporary path creation, atomic file replacement, database metadata, - and compensation. -- Track every path created before commit within the owning workflow. -- On flush or commit failure, remove newly created content and surface cleanup failure separately when removal also - fails. -- Constrain temporary and final paths beneath the artifact root and preserve safe relative references. -- Add startup or explicit maintenance cleanup for abandoned temporary files without deleting final referenced - artifacts. -- For deletion: - - Verify retention policy and references before mutation. - - Commit metadata changes according to the explicit deletion contract. - - Attempt external cleanup. - - Report incomplete cleanup with correlation identity when unlink fails. -- Do not silently skip unsafe references; classify them as validation or conflict failures as appropriate. -- Add tests for write, replace, flush, commit, unlink, cancellation, missing file, digest mismatch, and unsafe reference - failures. - -### 9. Make Attempt Persistence and First Selection Conflict-Safe - -- Stop relying solely on an unprotected `max(attempt_number) + 1` calculation. -- Allocate or insert attempt numbers inside the protected Job processing boundary. -- Treat the database uniqueness constraint as a final integrity guard and translate conflicts deterministically. -- Ensure every completed provider call either persists one immutable attempt or leaves a visible persistence failure - that does not masquerade as provider success. -- Perform first-success Source selection with a conditional update requiring both preferred provenance and machine - projection to remain unset. -- When first-selection loses a race, retain the new successful attempt as a candidate. -- Keep explicit candidate promotion as a separate atomic command. -- Never modify earlier attempt text, transport evidence, warnings, or artifact associations. -- Add concurrency tests for duplicate attempt allocation, simultaneous successes, success/failure overlap, candidate - promotion, and transaction rollback. - -### 10. Add Bounded Concurrent Page Processing - -- Add `worker_concurrency` to Settings with default `2`, positive validation, and a documented safe upper bound. -- Refactor the single-page body of `process_queued_job` into one typed page-attempt operation. -- Prepare each page's provider input and persist any derivative using its own session and committed preparation phase. -- Release all database transactions before awaiting provider network work. -- Use an asyncio semaphore or TaskGroup-based worker pattern to cap in-flight page calls. -- Give every page outcome persistence operation its own session and transaction. -- Shield only the minimum durable outcome write required to preserve a completed provider call during cancellation. -- Collect typed success, failure, and cancellation outcomes without sharing mutable lists across tasks. -- Compute the aggregate status after all started tasks reach durable outcomes. -- Stop scheduling unstarted pages after cancellation or an externally terminal Job transition. -- Preserve deterministic source ordering for scheduling and deterministic aggregate reporting independent of completion - order. -- Add tests for concurrency bounds `1`, `2`, and a larger configured value; out-of-order completion; mixed outcomes; - timeout; cancellation; external stop; durable early completion; and no shared session. - -### 11. Introduce Purpose-Specific Read Projections - -- Add a Source-list projection containing only fields needed by the Sources table. -- Filter Document and Job scopes in SQL. -- Add a Source-detail projection or explicit eager-loading contract for the relationships rendered by the page. -- Add an attempt-summary projection containing candidate metadata, transcription text or an explicit preview policy, - warning summary, and preferred-selection identity. -- Defer transport body, SDK snapshot, request manifest, and other large evidence values from summaries. -- Keep one explicit full-attempt evidence read for detailed inspection. -- Keep evidence export complete and integrity-verified. -- Replace all-sibling materialization in Source navigation with predecessor/successor queries using page number and UUID. -- Add query-count and unloaded-attribute assertions so future changes cannot reintroduce accidental evidence loading. -- Preserve deterministic ordering and avoid lazy database access from UI components. - -### 12. Isolate Compatibility Adapters and Cohesive Shared Policies - -- Keep `services/transcription.py` as a documented compatibility facade with imports and aliases only. -- Keep legacy upload/storage aliases as thin calls into the new ingestion and media owners. -- Mark compatibility entry points in tests so removal can be evaluated separately. -- Consolidate Document Type and Person Role label normalization in a registry-domain policy module. -- Share canonical evidence digest primitives only where serialization and digest meaning are identical. -- Do not merge prompt backup writing, original Source storage, and immutable artifact storage into one generic helper; - their lifecycle and recovery semantics remain domain-owned. -- Remove unused service-base state only after confirming no external caller depends on it. -- Add import-compatibility tests and direct tests for each extracted policy. - -### 13. Align UI and API Boundaries - -- Remove machine-local stored paths from routine Source Detail presentation. -- Keep stored filename and evidence-safe identifiers available. -- Ensure normal Job creation uses the configured provider and allowlisted model selector rather than arbitrary text input. -- Keep retranscription Source and Document context locked. -- Keep evidence export explicit and avoid loading export-only content during page render. -- Continue translating AppError through established UI and API presenters without parsing error messages. -- Keep sessions, session factories beyond dependency construction, and persistence operations out of reusable components. -- Add UI tests for allowlisted selection, no path disclosure, unchanged candidate actions, and error preservation. - -### 14. Align Authoritative Documentation - -- Update V4 architecture for: - - Frozen Job request versus resolved attempt identity. - - Atomic Job claims. - - Bounded page concurrency and isolated page sessions. - - Source facade and internal collaborators. - - Explicit artifact lifecycle ownership. -- Update V4 schema documentation only for actual additive schema changes. -- Update V4 requirements and verification mapping for atomic claims, media validation, and concurrency tests. -- Update V4 error handling for filesystem/database partial cleanup. -- Define `transcribed` as the current all-success terminal status and `completed` as historical compatibility. -- Correct stale V3 module descriptions and UI documentation that exposes or promises machine-local paths. -- Update documentation together with the behavior it describes; do not prestate unimplemented behavior as complete. - -### 15. Verification and Preservation Audit - -- Run focused pure tests for submission specs, media validation, provider contracts, query projections, and policies. -- Run isolated service tests for claims, attempts, artifacts, ingestion, candidate selection, and compatibility. -- Run worker tests with fake providers and controlled concurrent completion. -- Run UI and API tests against isolated application state. -- Run schema tests only through the required destructive-test wrapper when applicable. -- Run the broader non-external regression suite after focused targets pass. -- Verify that tests did not change: - - `data/transcription.db`. - - Curated Source files. - - Existing artifact files. - - Prompt files outside temporary fixtures. -- Perform an explicit preservation audit showing that upgrading does not rewrite existing Job, Source, JobSource, - ExecutionAttempt, ProcessingArtifact, or revision content. - -## Delivery Order - -1. Characterization and failure-injection tests. -2. Frozen submission specification and prompt ownership. -3. Content-aware Source media validation. -4. Ingestion workflow and UI transaction-boundary correction. -5. Atomic Job claiming and lifecycle terminology. -6. Stateless provider evidence. -7. Source evidence/artifact/provider-input extraction behind SourceService. -8. Artifact compensation and conflict-safe attempt/selection persistence. -9. Purpose-specific read projections. -10. Bounded concurrent page processing. -11. Compatibility isolation, UI/API alignment, and documentation. -12. Full preservation and regression verification. - -## Done Criteria - -- All V4.6 acceptance criteria are implemented and testable. -- New Jobs contain complete request provenance before queueing and are not rewritten by execution. -- Every submitted model is allowlisted and requested/resolved identities remain distinct. -- Upload UI code owns no database transaction. -- Source content validation verifies supported media without modifying originals. -- One queued Job can be claimed only once. -- Provider evidence is call-local and safe under overlapping execution. -- Page calls run concurrently within the configured bound and never share sessions. -- Every actual provider call has one immutable attempt. -- Concurrent first successes preserve one preferred result and every other success as a candidate. -- Artifact commit and cleanup failures are visible and compensated where safely possible. -- Source list and candidate views avoid loading full evidence bodies. -- SourceService remains compatible while extracted responsibilities are independently testable. -- Human revisions, preferred-machine projections, prior attempts, JobSource snapshots, and artifacts retain their original - meaning. -- The worker consistently emits transcribed, partial_success, or failed for new processing. -- Verification uses isolated data, fake providers, and does not modify operator or curated evidence data. - -## Related Local References - -- [V4.6 Scope Boundary](scope_boundary_v4_6.md) -- [V4.5 Scope Boundary](../ver4.5/scope_boundary_v4_5.md) -- [V4.5 Implementation Plan](../ver4.5/implementation_plan_v4_5.md) -- [V4 Architecture](../ver4/architecture_v4.md) -- [V4 Schema](../ver4/schema_v4.md) -- [V4 Requirements](../ver4/requirements_v4.md) -- [V4 Error Handling Policy](../ver4/error_handling_v4.md) -- [AI Evidence and Provenance Invariant](../invariant/ai_evidence_and_provenance.md) -- [Error Handling Invariant](../invariant/error_handling.md) diff --git a/docs/ver4.6/scope_boundary_v4_6.md b/docs/ver4.6/scope_boundary_v4_6.md deleted file mode 100644 index 0a4c6c3..0000000 --- a/docs/ver4.6/scope_boundary_v4_6.md +++ /dev/null @@ -1,296 +0,0 @@ -# V4.6 Scope Boundary - -This document defines the frozen boundary for architecture conformance, worker reliability, and Source-service -maintainability after the completed V4.5 revision. V4 through V4.5 remain the behavioral and evidence baseline. -V4.6 corrects implementation divergence and reorganizes internal ownership without replacing the V4 data model, -public identity rules, or preservation contracts. - -## Purpose - -- Freeze complete Job request provenance before work is queued. -- Make job claiming, attempt persistence, and first-success selection safe under concurrency. -- Implement the bounded page-processing concurrency required by the V4 architecture. -- Restore UI, workflow, service, transaction, provider, and filesystem ownership boundaries. -- Separate cohesive Source-domain responsibilities behind compatibility-preserving service contracts. -- Avoid loading large evidence fields for list and candidate-summary views. -- Correct documentation that no longer describes the implemented lifecycle accurately. - -## In Scope - -### 1. Frozen Job Submission Specification - -- Every newly queued Job records its requested provider, model, prompt name, prompt hash, full prompt content, - and explicitly configured generation parameters before the Job transaction commits. -- Normal transcription and Source retranscription use the same typed submission-specification builder. -- Provider and model values are resolved from validated application configuration. -- Every model submitted through the UI or another application workflow is validated against the configured model - allowlist. -- A provider response may report a resolved model that differs from the requested model. The resolved provider and - model belong to the immutable ExecutionAttempt and do not rewrite the Job request specification. -- Existing Jobs with null or incomplete historical fields remain readable and are not backfilled with facts that - were not frozen at submission. -- Prompt edits continue to affect future Jobs only and never rewrite stored Job or attempt provenance. - -### 2. Source Ingestion Workflow Ownership - -- Source ingestion is an application workflow, not a UI transaction and not a generic file utility. -- The ingestion workflow owns: - - Source-media validation. - - Original-file staging and hashing. - - Document lookup or creation when applicable. - - Job creation from a frozen submission specification. - - Ordered Source creation. - - JobSource creation. - - One database commit for the related records. - - Compensating cleanup when persistence fails before the workflow completes. -- UI pages resolve input, invoke the workflow, notify the worker only after successful commit, and navigate or - display errors. -- UI pages do not import session scopes, open transactions, execute queries, or commit and roll back persistence. -- Workflow code coordinates service contracts rather than inserting Document, Job, Source, or JobSource rows - directly. - -### 3. Source Media Validation - -- Filename extension remains part of the accepted-format policy but is not sufficient proof of media type. -- New Source uploads are validated as non-empty, structurally recognizable JPEG, PNG, TIFF, or PDF content. -- The validated content type must agree with the supported filename extension. -- Raster validation may decode enough content to establish that the file is a supported image, but validation does - not transform, re-encode, or replace the original bytes. -- PDF validation establishes that the content is a recognizable PDF without rendering, rewriting, or normalizing it. -- Original uploaded bytes, SHA-256 digest, byte size, upload name, and evidence identity remain unchanged. -- Historical Sources are not rejected merely because they predate content-aware validation. - -### 4. Atomic Job Claiming - -- Claiming a queued Job is one persistence operation that conditionally transitions - `queued -> processing`. -- Two workers or application instances cannot both successfully claim the same queued Job. -- Claim ordering remains deterministic by creation time and UUID. -- A worker that loses a claim performs no provider call and creates no execution evidence for that Job. -- Claim behavior remains portable across supported SQLite and PostgreSQL deployments. -- Startup recovery may requeue stale processing Jobs according to existing policy, but recovery does not bypass the - normal claim operation. - -### 5. Stateless Provider Execution Evidence - -- Request manifest and transport evidence belong to one provider call and are returned through that call's result or - exception. -- Provider adapters do not expose mutable "current request" or "current response" state for worker persistence. -- Success, HTTP failure, connection failure, local timeout, cancellation, and response-validation failure retain - their own request and transport evidence without cross-attribution. -- Provider adapters remain responsible for exact provider-boundary capture and safe-header allowlisting. -- Provider calls occur outside database transactions. -- Provider adapter instances may be reused only when their per-call state is concurrency-safe. - -### 6. Bounded Page Concurrency - -- One claimed multi-page Job may process page provider calls concurrently. -- A validated `worker_concurrency` setting defines the maximum number of in-flight page calls. -- The default concurrency is `2`; allowed values are positive and bounded to a documented safe maximum. -- The worker continues to orchestrate one claimed Job at a time within one application worker loop. V4.6 does not add - distributed scheduling or simultaneous aggregate processing of multiple Jobs in one process. -- Each page task uses its own database session for preparation and outcome persistence; an AsyncSession is never - shared across concurrent page tasks. -- Provider-input preparation completes and commits before that page's provider call begins. -- Each completed page outcome is committed durably without waiting for every sibling page. -- One page failure does not cancel successful independent pages. -- Aggregate Job status is computed after every started page task reaches a durable outcome or explicit cancellation. -- Cancellation prevents unstarted work and preserves durable outcomes from calls that already completed. -- Concurrency does not change provider cost policy: quality warnings never trigger automatic provider calls. - -### 7. Conflict-Safe Attempts and Machine Selection - -- Every actual provider call creates exactly one immutable ExecutionAttempt, including failure and timeout outcomes. -- Attempt numbering remains unique for `(job_id, source_id, attempt_number)` and does not use an unprotected - application-only `max + 1` assumption. -- A duplicate persistence race fails deterministically without overwriting existing evidence. -- The first successful attempt may establish preferred machine output only when the Source has no selected machine - provenance at the atomic write boundary. -- If another success wins first-selection concurrently, the losing successful attempt remains an unselected - candidate. -- Candidate promotion continues to update `Source.preferred_execution_attempt_id` and - `Source.raw_transcription` atomically. -- Human revision remains independent and is never cleared or changed by execution or promotion. - -### 8. Processing Artifact Lifecycle - -- Processing-artifact content and database metadata have one explicit creation owner. -- When artifact persistence fails or its enclosing commit fails, files created by that operation are removed when - safely possible and the failure remains visible. -- Temporary artifact files use constrained application-managed paths and are recoverable or removable after - interruption. -- Artifact integrity verification continues to distinguish unavailable content, malformed metadata, digest mismatch, - and unsafe external references. -- Deleting artifact metadata and deleting external content is an explicit partial-failure-capable operation. -- A failed external-file deletion is logged with correlation identity and reported as incomplete cleanup rather than - silent success. -- No cleanup path deletes an artifact still referenced by retained immutable evidence. - -### 9. Source Service Responsibility Boundaries - -- `SourceService` remains the public Source aggregate facade during V4.6. -- Internally cohesive Source-domain collaborators may own: - - Source persistence, queries, navigation, and revisions. - - Source media policy and managed original-file access. - - Prompt execution snapshots. - - Provider-input resolution and normalization-artifact association. - - ExecutionAttempt and JobSource projection persistence. - - Processing-artifact storage and integrity. - - Evidence export. - - Provider execution and error translation. -- Candidate promotion remains in the Source aggregate persistence boundary because it atomically changes the Source's - selected provenance and compatibility text projection. -- Orientation transformation remains in the focused normalization module. -- Deterministic warning analysis remains in the focused quality module. -- Extraction does not create a generic `utils.py`, `helpers.py`, or `common.py` service dumping ground. -- Services remain independent; multi-service behavior stays in workflow functions or workflow objects. - -### 10. Read Projections and Query Efficiency - -- Source list, Source Detail, candidate summary, and evidence export use purpose-specific read contracts. -- Candidate and list views do not load exact transport bodies, full SDK snapshots, or other large evidence fields - that they do not render. -- Full evidence bodies remain available through explicit evidence inspection or export. -- Job-filtered Source queries filter in SQL rather than loading every Source and filtering JobSource relationships in - application memory. -- Source navigation preserves deterministic `(page_number, UUID)` ordering without requiring all sibling Source - records to be materialized. -- Query optimization does not introduce lazy-loading behavior into detached UI models. - -### 11. Lifecycle and Terminology Alignment - -- `transcribed` is the canonical successful terminal Job status for the current transcription workflow. -- `completed` remains a readable historical/compatibility status but is not newly emitted by the V4.6 transcription - worker. -- `partial_success` and `failed` retain their existing aggregate meanings. -- Architecture, schema, requirements, error-handling, UI, and model documentation use the same status terminology. -- JobSource remains the mutable queue and compatibility projection. -- ExecutionAttempt remains the authoritative append-only execution history. - -### 12. Compatibility Adapter Isolation - -- Existing imports from `transcription.services.transcription` continue to resolve during V4.6. -- Existing upload/storage aliases may remain temporarily at documented compatibility boundaries. -- Compatibility adapters contain no new business logic, persistence, provider calls, or filesystem policy. -- New application code imports the domain-owned implementation rather than adding more compatibility aliases. -- Removal of compatibility imports is deferred until usage is measured and a separate deprecation decision is made. - -## Out of Scope - -- Replacing JobSource with an event store or removing its compatibility fields. -- Rewriting, deleting, or relabeling historical ExecutionAttempt evidence. -- Changing Source candidate, promotion, or human-revision precedence. -- Changing the V4.5 orientation transformation or quality-warning rules. -- Reprocessing existing Sources solely to apply new media validation. -- Supporting additional media formats. -- Multiple selectable providers in the UI. -- Distributed queues, external brokers, distributed locks, or multi-host worker coordination. -- Concurrent processing of multiple aggregate Jobs in one application worker loop. -- Automatic paid retry triggered by quality warnings. -- Source page reordering or movement between Documents. -- A generic repository framework, generic service container framework, or generic filesystem utility layer. -- Destructive schema replacement or a V5 API redesign. - -## Locked Design Decisions - -### A. V4.6 Preserves the V4 Evidence Contract - -- Original Source bytes remain primary evidence. -- ExecutionAttempt remains append-only and authoritative. -- JobSource remains a compatibility projection. -- Human revisions remain separate from all machine output. - -### B. Requested and Resolved Model Identity Are Different Facts - -- Job stores the requested provider and model frozen before queueing. -- ExecutionAttempt stores the provider and model observed for the individual execution. -- Worker completion never rewrites the Job request specification. - -### C. Concurrency Is Page-Bounded, Not Job-Distributed - -- V4.6 adds bounded page concurrency within one claimed Job. -- One worker loop continues to own one Job aggregate at a time. -- Every page task owns its own session and durable outcome transaction. - -### D. The Source Facade Remains Stable - -- Source internals may be extracted, but callers retain a stable SourceService contract during migration. -- Candidate promotion stays transactionally close to Source persistence. -- Provider, prompt, artifact, and export concerns become replaceable collaborators rather than additional aggregates. - -### E. Filesystem and Database Partial Failure Is Explicit - -- Cross-resource operations cannot rely on a relational transaction alone. -- Observable commit failures receive compensation. -- Unavoidable cleanup failure is reported and remains diagnosable rather than being represented as complete success. - -### F. Refactoring Does Not Rewrite History - -- Existing nulls remain null unless the operator performs a separately specified maintenance action. -- Historical SDK snapshots are not reclassified as transport evidence. -- Existing preferred-machine projections and human revisions remain unchanged. - -## Data and Compatibility Policy - -- V4.6 prefers no schema change where conditional writes and focused read projections are sufficient. -- Any required schema addition is additive and portable across SQLite and PostgreSQL. -- Existing Job, Source, JobSource, ExecutionAttempt, ProcessingArtifact, Document, and Person identifiers remain valid. -- Existing Source files and artifact references retain their stored meaning. -- Existing `completed` Job rows remain readable. -- Existing API response fields are not removed by this revision. -- Compatibility adapters remain read- and call-compatible while internal ownership moves. -- Database, integration, and UI tests use confirmed isolated data and never modify `data/transcription.db`. -- Potentially destructive tests run only through `tools/run_destructive_tests.py`. - -## Acceptance Criteria - -1. Every newly queued Job has complete requested provider, model, prompt, prompt hash, and parameter provenance before - worker execution. -2. A provider-reported resolved model is stored on the ExecutionAttempt and does not change the Job's requested model. -3. Normal and retranscription Job creation reject models outside the configured allowlist. -4. Upload UI code opens no database session and performs no direct persistence. -5. A failed ingestion transaction removes files staged by that failed operation and leaves no partial database rows. -6. Supported Source content is validated beyond its extension without transforming original bytes. -7. Two concurrent claimers cannot both claim or execute the same queued Job. -8. Overlapping provider calls retain distinct request manifests and transport evidence. -9. Page execution concurrency never exceeds the configured bound and uses no shared AsyncSession. -10. Completed pages are durable while slower sibling calls remain in flight. -11. Every actual provider call creates one immutable attempt and no duplicate attempt number. -12. Concurrent first successes result in one preferred attempt and retained candidates for all other successes. -13. Commit failure during external artifact creation does not silently leave an untracked artifact. -14. External cleanup failure is visible and does not delete retained evidence metadata deceptively. -15. Source list and candidate-summary views do not load exact transport bodies. -16. Job-filtered Source lists filter in the database and preserve deterministic ordering. -17. SourceService callers remain compatible while focused collaborators become independently testable. -18. The worker emits `transcribed`, `partial_success`, or `failed` consistently, and historical `completed` rows remain - readable. -19. Previous Jobs, attempts, artifacts, Source projections, and human revisions remain unchanged by the upgrade. -20. Verification uses isolated databases, fake providers, and does not modify curated Source or evidence files. - -## Scope Freeze Gate - -V4.6 is sufficiently defined to begin implementation: - -- Submission-time versus execution-time provenance ownership is resolved. -- UI, workflow, service, and transaction boundaries are resolved. -- Claiming, attempt persistence, and first-selection concurrency rules are resolved. -- Page-level concurrency scope and default bound are resolved. -- Source extraction seams and compatibility policy are resolved. -- Media validation, artifact compensation, and query-projection expectations are resolved. -- Successful Job status terminology is resolved. -- V4.2 and V4.5 evidence, candidate, and revision contracts remain unchanged. - -Any expansion into distributed work scheduling, additional providers or media types, evidence-model replacement, -candidate-policy changes, or incompatible API/schema redesign requires an explicit V4.6 scope amendment or V5. - -## Related Local References - -- [V4.6 Implementation Plan](implementation_plan_v4_6.md) -- [V4.5 Scope Boundary](../ver4.5/scope_boundary_v4_5.md) -- [V4.5 Implementation Plan](../ver4.5/implementation_plan_v4_5.md) -- [V4 Architecture](../ver4/architecture_v4.md) -- [V4 Schema](../ver4/schema_v4.md) -- [V4 Requirements](../ver4/requirements_v4.md) -- [V4 Error Handling Policy](../ver4/error_handling_v4.md) -- [AI Evidence and Provenance Invariant](../invariant/ai_evidence_and_provenance.md) -- [Error Handling Invariant](../invariant/error_handling.md)