diff --git a/docs/ver4.6/implementation_plan_v4_6.md b/docs/ver4.6/implementation_plan_v4_6.md new file mode 100644 index 0000000..9c8d7e5 --- /dev/null +++ b/docs/ver4.6/implementation_plan_v4_6.md @@ -0,0 +1,338 @@ +# Implementation Plan (Version 4.6) + +## Goal + +Bring the completed V4.5 implementation into conformance with the documented V4 service, provenance, transaction, +worker-concurrency, and evidence boundaries while preserving all historical Source and execution behavior. + +## Planning Status + +- V4.5 is the completed implementation baseline. +- The V4.6 scope is frozen and sufficiently detailed to begin implementation. +- V4.6 is a compatibility-preserving architecture and reliability revision, not a V5 redesign. +- Scope additions require an explicit amendment or a later revision. + +## Planning Constraints + +- Original Source bytes and historical evidence remain immutable. +- Existing JobSource compatibility projections remain supported. +- Human revisions remain independent and retain display and print precedence. +- Candidate promotion semantics remain unchanged. +- Provider network work occurs outside database transactions. +- Concurrent page tasks never share an AsyncSession. +- UI pages do not own persistence transactions. +- Refactoring proceeds behind stable public service contracts. +- New shared code has a cohesive domain or infrastructure owner; no generic helper dumping grounds are introduced. +- Database, integration, and UI tests use confirmed isolated data and never modify `data/transcription.db`. +- Potentially destructive tests run only through `tools/run_destructive_tests.py`. + +## Expected Project Impact + +| Area | Expected impact | +| --- | --- | +| Job submission | Add one typed frozen request specification used by normal and retranscription workflows. | +| Upload workflow | Move transaction and compensation ownership from the UI and direct persistence functions into orchestration. | +| Source media | Validate content and extension consistently while preserving original bytes. | +| Job service | Add atomic claim behavior and stable lifecycle transitions. | +| Provider adapters | Remove mutable per-call evidence state and carry evidence through results and errors. | +| Worker workflows | Add bounded concurrent page execution with isolated sessions and durable page outcomes. | +| Source service | Extract cohesive collaborators behind the existing SourceService facade. | +| Evidence persistence | Make attempt numbering, first selection, artifact creation, and cleanup conflict-safe. | +| Read paths | Add lightweight Source and attempt projections that defer large evidence fields. | +| Compatibility | Isolate legacy transcription and storage aliases without adding new behavior. | +| Documentation | Align status terminology, ownership, concurrency, and provenance descriptions. | +| Tests | Add failure-injection, concurrency, query-shape, compatibility, and preservation coverage. | + +## Implementation Phases + +### 1. Add Characterization and Failure-Injection Coverage + +- Record the current externally visible Source, Job, JobSource, attempt, artifact, revision, candidate, and export + behavior before extraction. +- Add tests proving: + - Historical Jobs with missing request fields remain readable. + - JobSource SDK snapshots retain their existing evidence label. + - Source preferred-machine fields and human revisions survive failed processing. + - Per-page outcomes remain durable before sibling pages finish. + - Candidate promotion updates both Source projection fields atomically. +- Add reusable test doubles for: + - Provider success, response failure, connection failure, timeout, and cancellation. + - Database flush and commit failure at named workflow phases. + - Filesystem write, replace, and unlink failure. + - Controlled overlapping page calls. +- Keep all providers fake and all databases and filesystem roots isolated. +- Use these tests as behavior gates for every later extraction. + +### 2. Define the Frozen Job Submission Specification + +- Move runtime prompt loading, prompt hashing, requested provider/model resolution, and explicit parameter-state capture + into a focused prompt/job-submission module. +- Define one immutable typed value containing: + - Requested provider. + - Requested model. + - Prompt name and SHA-256. + - System and user prompt content. + - Temperature and top-p values. + - Explicitly supplied versus omitted state where applicable. +- Reuse the constrained PromptStore's path policy rather than maintaining a second incompatible prompt-path policy. +- Keep prompt editing, backup, and recovery behavior in the existing prompt-storage owner. +- Validate model selection against `provider_models` for both normal transcription and retranscription. +- Populate all new Job request fields before adding the Job to the session. +- Stop mutating Job provider and model during attempt persistence. +- Store provider-reported resolved identity on ExecutionAttempt only. +- Retain compatibility reads for historical Jobs whose request fields are null. +- Add tests for pre-queue completeness, allowlist rejection, prompt-edit isolation, requested/resolved model differences, and + historical null behavior. + +### 3. Establish Content-Aware Source Media Policy + +- Move accepted Source extensions, canonical MIME mapping, validation, and managed payload loading into one Source-media + module. +- Validate filename and non-empty content first. +- Validate JPEG, PNG, and TIFF using bounded raster decoding appropriate to the existing Pillow dependency. +- Validate PDF structure without rendering or modifying the file. +- Reject a supported extension whose content is corrupt or belongs to a different supported type. +- Return a typed validated-media result containing canonical media type, digest, byte size, and original bytes or a + stable staged reference. +- Ensure hashing uses the exact bytes later persisted as the original Source. +- Keep orientation normalization separate and downstream of original-media validation. +- Preserve compatibility aliases for existing MIME and payload-loading imports. +- Add fixtures for valid, corrupt, truncated, mismatched, uppercase-extension, empty, and unsupported content. + +### 4. Replace Direct Upload Persistence with an Ingestion Workflow + +- Introduce a Source-ingestion workflow that accepts validated inputs and service dependencies. +- Generate Document and Source identifiers before file placement so managed relative paths remain deterministic. +- Stage all Source files beneath the configured application root. +- Build the frozen Job submission specification before starting database mutation. +- Open one workflow-owned transaction. +- Use DocumentService, JobService, and SourceService session-aware writes to create: + - The Document when required. + - The queued Job. + - Ordered Sources. + - Pending JobSource projections. +- Commit once after every record has flushed successfully. +- Delete files staged by the operation on validation, flush, or commit failure. +- Return a typed result only after commit succeeds. +- Move worker notification after successful workflow return. +- Remove session-scope imports and direct persistence ownership from the Jobs UI. +- Keep old storage entry points as thin compatibility adapters that invoke the workflow. +- Add tests for deterministic ordering, page numbering, duplicate names, every failure phase, cleanup, commit-before-notify, + and no direct UI persistence. + +### 5. Add Atomic Job Claiming and Canonical Lifecycle Semantics + +- Add `claim_next_queued_job` to JobService or a Job-owned persistence collaborator. +- Select candidates deterministically by creation time and UUID. +- Transition one candidate using a conditional write that succeeds only while status is queued. +- Retry candidate selection when another claimant wins, without issuing provider work for the lost claim. +- Commit the processing transition before provider-input preparation. +- Route worker processing exclusively through the claim operation. +- Preserve stale-processing recovery and require recovered Jobs to be claimed again. +- Define `transcribed` as the newly emitted all-success terminal status. +- Continue reading `completed` as a historical terminal status. +- Add tests with concurrent claimers on SQLite and the portable SQL path used for PostgreSQL. +- Add a lifecycle transition-table test covering queued, processing, transcribed, partial success, failed, cancellation, + resubmission, and stale recovery. + +### 6. Make Provider Calls Stateless + +- Extend provider success and error contracts so each call directly carries: + - RequestManifest. + - TransportEvidence or explicit no-response evidence. + - Failure phase. + - Parsed SDK snapshot when available. + - Normalized metadata when available. +- Remove `current_request_manifest` and `current_transport_evidence` from provider adapters and worker persistence. +- Ensure local timeout and cancellation retain the call-specific request manifest without reading mutable adapter state. +- Keep response-body capture and safe-header filtering in the OpenRouter adapter. +- Preserve the distinction between exact OpenRouter-boundary response bytes, SDK snapshot, and normalized fields. +- Decide adapter reuse by concurrency safety; create per-call adapters if the SDK client cannot safely support concurrent + calls. +- Add overlapping-call tests in which success, HTTP failure, timeout, and no-response failure complete in different + orders. +- Assert that no call receives another call's source digest, prompt hash, response body, or identifiers. + +### 7. Extract Source-Owned Evidence and Artifact Collaborators + +- Keep SourceService as the public facade while moving cohesive implementation behind it. +- Extract Source persistence and query behavior first without changing method contracts. +- Extract ExecutionAttempt and JobSource compatibility persistence into a Source-owned evidence writer. +- Extract processing-artifact file, integrity, and metadata behavior into a Source-owned artifact store. +- Extract evidence export into a read-only projection builder. +- Extract provider-input preparation so it coordinates: + - Managed original Source access. + - Orientation normalization. + - Derivative artifact creation. + - Exact SourceEvidenceReference construction. +- Keep candidate promotion in Source persistence. +- Keep normalization and quality analysis in their existing focused modules. +- Make dependencies explicit through typed constructor parameters rather than importing service globals. +- Do not allow extracted collaborators to call unrelated aggregate services. +- Add contract tests showing SourceService behavior is unchanged while collaborators can be tested independently. + +### 8. Make Artifact Creation and Cleanup Explicit + +- Define one artifact-creation operation that owns temporary path creation, atomic file replacement, database metadata, + and compensation. +- Track every path created before commit within the owning workflow. +- On flush or commit failure, remove newly created content and surface cleanup failure separately when removal also + fails. +- Constrain temporary and final paths beneath the artifact root and preserve safe relative references. +- Add startup or explicit maintenance cleanup for abandoned temporary files without deleting final referenced + artifacts. +- For deletion: + - Verify retention policy and references before mutation. + - Commit metadata changes according to the explicit deletion contract. + - Attempt external cleanup. + - Report incomplete cleanup with correlation identity when unlink fails. +- Do not silently skip unsafe references; classify them as validation or conflict failures as appropriate. +- Add tests for write, replace, flush, commit, unlink, cancellation, missing file, digest mismatch, and unsafe reference + failures. + +### 9. Make Attempt Persistence and First Selection Conflict-Safe + +- Stop relying solely on an unprotected `max(attempt_number) + 1` calculation. +- Allocate or insert attempt numbers inside the protected Job processing boundary. +- Treat the database uniqueness constraint as a final integrity guard and translate conflicts deterministically. +- Ensure every completed provider call either persists one immutable attempt or leaves a visible persistence failure + that does not masquerade as provider success. +- Perform first-success Source selection with a conditional update requiring both preferred provenance and machine + projection to remain unset. +- When first-selection loses a race, retain the new successful attempt as a candidate. +- Keep explicit candidate promotion as a separate atomic command. +- Never modify earlier attempt text, transport evidence, warnings, or artifact associations. +- Add concurrency tests for duplicate attempt allocation, simultaneous successes, success/failure overlap, candidate + promotion, and transaction rollback. + +### 10. Add Bounded Concurrent Page Processing + +- Add `worker_concurrency` to Settings with default `2`, positive validation, and a documented safe upper bound. +- Refactor the single-page body of `process_queued_job` into one typed page-attempt operation. +- Prepare each page's provider input and persist any derivative using its own session and committed preparation phase. +- Release all database transactions before awaiting provider network work. +- Use an asyncio semaphore or TaskGroup-based worker pattern to cap in-flight page calls. +- Give every page outcome persistence operation its own session and transaction. +- Shield only the minimum durable outcome write required to preserve a completed provider call during cancellation. +- Collect typed success, failure, and cancellation outcomes without sharing mutable lists across tasks. +- Compute the aggregate status after all started tasks reach durable outcomes. +- Stop scheduling unstarted pages after cancellation or an externally terminal Job transition. +- Preserve deterministic source ordering for scheduling and deterministic aggregate reporting independent of completion + order. +- Add tests for concurrency bounds `1`, `2`, and a larger configured value; out-of-order completion; mixed outcomes; + timeout; cancellation; external stop; durable early completion; and no shared session. + +### 11. Introduce Purpose-Specific Read Projections + +- Add a Source-list projection containing only fields needed by the Sources table. +- Filter Document and Job scopes in SQL. +- Add a Source-detail projection or explicit eager-loading contract for the relationships rendered by the page. +- Add an attempt-summary projection containing candidate metadata, transcription text or an explicit preview policy, + warning summary, and preferred-selection identity. +- Defer transport body, SDK snapshot, request manifest, and other large evidence values from summaries. +- Keep one explicit full-attempt evidence read for detailed inspection. +- Keep evidence export complete and integrity-verified. +- Replace all-sibling materialization in Source navigation with predecessor/successor queries using page number and UUID. +- Add query-count and unloaded-attribute assertions so future changes cannot reintroduce accidental evidence loading. +- Preserve deterministic ordering and avoid lazy database access from UI components. + +### 12. Isolate Compatibility Adapters and Cohesive Shared Policies + +- Keep `services/transcription.py` as a documented compatibility facade with imports and aliases only. +- Keep legacy upload/storage aliases as thin calls into the new ingestion and media owners. +- Mark compatibility entry points in tests so removal can be evaluated separately. +- Consolidate Document Type and Person Role label normalization in a registry-domain policy module. +- Share canonical evidence digest primitives only where serialization and digest meaning are identical. +- Do not merge prompt backup writing, original Source storage, and immutable artifact storage into one generic helper; + their lifecycle and recovery semantics remain domain-owned. +- Remove unused service-base state only after confirming no external caller depends on it. +- Add import-compatibility tests and direct tests for each extracted policy. + +### 13. Align UI and API Boundaries + +- Remove machine-local stored paths from routine Source Detail presentation. +- Keep stored filename and evidence-safe identifiers available. +- Ensure normal Job creation uses the configured provider and allowlisted model selector rather than arbitrary text input. +- Keep retranscription Source and Document context locked. +- Keep evidence export explicit and avoid loading export-only content during page render. +- Continue translating AppError through established UI and API presenters without parsing error messages. +- Keep sessions, session factories beyond dependency construction, and persistence operations out of reusable components. +- Add UI tests for allowlisted selection, no path disclosure, unchanged candidate actions, and error preservation. + +### 14. Align Authoritative Documentation + +- Update V4 architecture for: + - Frozen Job request versus resolved attempt identity. + - Atomic Job claims. + - Bounded page concurrency and isolated page sessions. + - Source facade and internal collaborators. + - Explicit artifact lifecycle ownership. +- Update V4 schema documentation only for actual additive schema changes. +- Update V4 requirements and verification mapping for atomic claims, media validation, and concurrency tests. +- Update V4 error handling for filesystem/database partial cleanup. +- Define `transcribed` as the current all-success terminal status and `completed` as historical compatibility. +- Correct stale V3 module descriptions and UI documentation that exposes or promises machine-local paths. +- Update documentation together with the behavior it describes; do not prestate unimplemented behavior as complete. + +### 15. Verification and Preservation Audit + +- Run focused pure tests for submission specs, media validation, provider contracts, query projections, and policies. +- Run isolated service tests for claims, attempts, artifacts, ingestion, candidate selection, and compatibility. +- Run worker tests with fake providers and controlled concurrent completion. +- Run UI and API tests against isolated application state. +- Run schema tests only through the required destructive-test wrapper when applicable. +- Run the broader non-external regression suite after focused targets pass. +- Verify that tests did not change: + - `data/transcription.db`. + - Curated Source files. + - Existing artifact files. + - Prompt files outside temporary fixtures. +- Perform an explicit preservation audit showing that upgrading does not rewrite existing Job, Source, JobSource, + ExecutionAttempt, ProcessingArtifact, or revision content. + +## Delivery Order + +1. Characterization and failure-injection tests. +2. Frozen submission specification and prompt ownership. +3. Content-aware Source media validation. +4. Ingestion workflow and UI transaction-boundary correction. +5. Atomic Job claiming and lifecycle terminology. +6. Stateless provider evidence. +7. Source evidence/artifact/provider-input extraction behind SourceService. +8. Artifact compensation and conflict-safe attempt/selection persistence. +9. Purpose-specific read projections. +10. Bounded concurrent page processing. +11. Compatibility isolation, UI/API alignment, and documentation. +12. Full preservation and regression verification. + +## Done Criteria + +- All V4.6 acceptance criteria are implemented and testable. +- New Jobs contain complete request provenance before queueing and are not rewritten by execution. +- Every submitted model is allowlisted and requested/resolved identities remain distinct. +- Upload UI code owns no database transaction. +- Source content validation verifies supported media without modifying originals. +- One queued Job can be claimed only once. +- Provider evidence is call-local and safe under overlapping execution. +- Page calls run concurrently within the configured bound and never share sessions. +- Every actual provider call has one immutable attempt. +- Concurrent first successes preserve one preferred result and every other success as a candidate. +- Artifact commit and cleanup failures are visible and compensated where safely possible. +- Source list and candidate views avoid loading full evidence bodies. +- SourceService remains compatible while extracted responsibilities are independently testable. +- Human revisions, preferred-machine projections, prior attempts, JobSource snapshots, and artifacts retain their original + meaning. +- The worker consistently emits transcribed, partial_success, or failed for new processing. +- Verification uses isolated data, fake providers, and does not modify operator or curated evidence data. + +## Related Local References + +- [V4.6 Scope Boundary](scope_boundary_v4_6.md) +- [V4.5 Scope Boundary](../ver4.5/scope_boundary_v4_5.md) +- [V4.5 Implementation Plan](../ver4.5/implementation_plan_v4_5.md) +- [V4 Architecture](../ver4/architecture_v4.md) +- [V4 Schema](../ver4/schema_v4.md) +- [V4 Requirements](../ver4/requirements_v4.md) +- [V4 Error Handling Policy](../ver4/error_handling_v4.md) +- [AI Evidence and Provenance Invariant](../invariant/ai_evidence_and_provenance.md) +- [Error Handling Invariant](../invariant/error_handling.md) diff --git a/docs/ver4.6/scope_boundary_v4_6.md b/docs/ver4.6/scope_boundary_v4_6.md new file mode 100644 index 0000000..0a4c6c3 --- /dev/null +++ b/docs/ver4.6/scope_boundary_v4_6.md @@ -0,0 +1,296 @@ +# V4.6 Scope Boundary + +This document defines the frozen boundary for architecture conformance, worker reliability, and Source-service +maintainability after the completed V4.5 revision. V4 through V4.5 remain the behavioral and evidence baseline. +V4.6 corrects implementation divergence and reorganizes internal ownership without replacing the V4 data model, +public identity rules, or preservation contracts. + +## Purpose + +- Freeze complete Job request provenance before work is queued. +- Make job claiming, attempt persistence, and first-success selection safe under concurrency. +- Implement the bounded page-processing concurrency required by the V4 architecture. +- Restore UI, workflow, service, transaction, provider, and filesystem ownership boundaries. +- Separate cohesive Source-domain responsibilities behind compatibility-preserving service contracts. +- Avoid loading large evidence fields for list and candidate-summary views. +- Correct documentation that no longer describes the implemented lifecycle accurately. + +## In Scope + +### 1. Frozen Job Submission Specification + +- Every newly queued Job records its requested provider, model, prompt name, prompt hash, full prompt content, + and explicitly configured generation parameters before the Job transaction commits. +- Normal transcription and Source retranscription use the same typed submission-specification builder. +- Provider and model values are resolved from validated application configuration. +- Every model submitted through the UI or another application workflow is validated against the configured model + allowlist. +- A provider response may report a resolved model that differs from the requested model. The resolved provider and + model belong to the immutable ExecutionAttempt and do not rewrite the Job request specification. +- Existing Jobs with null or incomplete historical fields remain readable and are not backfilled with facts that + were not frozen at submission. +- Prompt edits continue to affect future Jobs only and never rewrite stored Job or attempt provenance. + +### 2. Source Ingestion Workflow Ownership + +- Source ingestion is an application workflow, not a UI transaction and not a generic file utility. +- The ingestion workflow owns: + - Source-media validation. + - Original-file staging and hashing. + - Document lookup or creation when applicable. + - Job creation from a frozen submission specification. + - Ordered Source creation. + - JobSource creation. + - One database commit for the related records. + - Compensating cleanup when persistence fails before the workflow completes. +- UI pages resolve input, invoke the workflow, notify the worker only after successful commit, and navigate or + display errors. +- UI pages do not import session scopes, open transactions, execute queries, or commit and roll back persistence. +- Workflow code coordinates service contracts rather than inserting Document, Job, Source, or JobSource rows + directly. + +### 3. Source Media Validation + +- Filename extension remains part of the accepted-format policy but is not sufficient proof of media type. +- New Source uploads are validated as non-empty, structurally recognizable JPEG, PNG, TIFF, or PDF content. +- The validated content type must agree with the supported filename extension. +- Raster validation may decode enough content to establish that the file is a supported image, but validation does + not transform, re-encode, or replace the original bytes. +- PDF validation establishes that the content is a recognizable PDF without rendering, rewriting, or normalizing it. +- Original uploaded bytes, SHA-256 digest, byte size, upload name, and evidence identity remain unchanged. +- Historical Sources are not rejected merely because they predate content-aware validation. + +### 4. Atomic Job Claiming + +- Claiming a queued Job is one persistence operation that conditionally transitions + `queued -> processing`. +- Two workers or application instances cannot both successfully claim the same queued Job. +- Claim ordering remains deterministic by creation time and UUID. +- A worker that loses a claim performs no provider call and creates no execution evidence for that Job. +- Claim behavior remains portable across supported SQLite and PostgreSQL deployments. +- Startup recovery may requeue stale processing Jobs according to existing policy, but recovery does not bypass the + normal claim operation. + +### 5. Stateless Provider Execution Evidence + +- Request manifest and transport evidence belong to one provider call and are returned through that call's result or + exception. +- Provider adapters do not expose mutable "current request" or "current response" state for worker persistence. +- Success, HTTP failure, connection failure, local timeout, cancellation, and response-validation failure retain + their own request and transport evidence without cross-attribution. +- Provider adapters remain responsible for exact provider-boundary capture and safe-header allowlisting. +- Provider calls occur outside database transactions. +- Provider adapter instances may be reused only when their per-call state is concurrency-safe. + +### 6. Bounded Page Concurrency + +- One claimed multi-page Job may process page provider calls concurrently. +- A validated `worker_concurrency` setting defines the maximum number of in-flight page calls. +- The default concurrency is `2`; allowed values are positive and bounded to a documented safe maximum. +- The worker continues to orchestrate one claimed Job at a time within one application worker loop. V4.6 does not add + distributed scheduling or simultaneous aggregate processing of multiple Jobs in one process. +- Each page task uses its own database session for preparation and outcome persistence; an AsyncSession is never + shared across concurrent page tasks. +- Provider-input preparation completes and commits before that page's provider call begins. +- Each completed page outcome is committed durably without waiting for every sibling page. +- One page failure does not cancel successful independent pages. +- Aggregate Job status is computed after every started page task reaches a durable outcome or explicit cancellation. +- Cancellation prevents unstarted work and preserves durable outcomes from calls that already completed. +- Concurrency does not change provider cost policy: quality warnings never trigger automatic provider calls. + +### 7. Conflict-Safe Attempts and Machine Selection + +- Every actual provider call creates exactly one immutable ExecutionAttempt, including failure and timeout outcomes. +- Attempt numbering remains unique for `(job_id, source_id, attempt_number)` and does not use an unprotected + application-only `max + 1` assumption. +- A duplicate persistence race fails deterministically without overwriting existing evidence. +- The first successful attempt may establish preferred machine output only when the Source has no selected machine + provenance at the atomic write boundary. +- If another success wins first-selection concurrently, the losing successful attempt remains an unselected + candidate. +- Candidate promotion continues to update `Source.preferred_execution_attempt_id` and + `Source.raw_transcription` atomically. +- Human revision remains independent and is never cleared or changed by execution or promotion. + +### 8. Processing Artifact Lifecycle + +- Processing-artifact content and database metadata have one explicit creation owner. +- When artifact persistence fails or its enclosing commit fails, files created by that operation are removed when + safely possible and the failure remains visible. +- Temporary artifact files use constrained application-managed paths and are recoverable or removable after + interruption. +- Artifact integrity verification continues to distinguish unavailable content, malformed metadata, digest mismatch, + and unsafe external references. +- Deleting artifact metadata and deleting external content is an explicit partial-failure-capable operation. +- A failed external-file deletion is logged with correlation identity and reported as incomplete cleanup rather than + silent success. +- No cleanup path deletes an artifact still referenced by retained immutable evidence. + +### 9. Source Service Responsibility Boundaries + +- `SourceService` remains the public Source aggregate facade during V4.6. +- Internally cohesive Source-domain collaborators may own: + - Source persistence, queries, navigation, and revisions. + - Source media policy and managed original-file access. + - Prompt execution snapshots. + - Provider-input resolution and normalization-artifact association. + - ExecutionAttempt and JobSource projection persistence. + - Processing-artifact storage and integrity. + - Evidence export. + - Provider execution and error translation. +- Candidate promotion remains in the Source aggregate persistence boundary because it atomically changes the Source's + selected provenance and compatibility text projection. +- Orientation transformation remains in the focused normalization module. +- Deterministic warning analysis remains in the focused quality module. +- Extraction does not create a generic `utils.py`, `helpers.py`, or `common.py` service dumping ground. +- Services remain independent; multi-service behavior stays in workflow functions or workflow objects. + +### 10. Read Projections and Query Efficiency + +- Source list, Source Detail, candidate summary, and evidence export use purpose-specific read contracts. +- Candidate and list views do not load exact transport bodies, full SDK snapshots, or other large evidence fields + that they do not render. +- Full evidence bodies remain available through explicit evidence inspection or export. +- Job-filtered Source queries filter in SQL rather than loading every Source and filtering JobSource relationships in + application memory. +- Source navigation preserves deterministic `(page_number, UUID)` ordering without requiring all sibling Source + records to be materialized. +- Query optimization does not introduce lazy-loading behavior into detached UI models. + +### 11. Lifecycle and Terminology Alignment + +- `transcribed` is the canonical successful terminal Job status for the current transcription workflow. +- `completed` remains a readable historical/compatibility status but is not newly emitted by the V4.6 transcription + worker. +- `partial_success` and `failed` retain their existing aggregate meanings. +- Architecture, schema, requirements, error-handling, UI, and model documentation use the same status terminology. +- JobSource remains the mutable queue and compatibility projection. +- ExecutionAttempt remains the authoritative append-only execution history. + +### 12. Compatibility Adapter Isolation + +- Existing imports from `transcription.services.transcription` continue to resolve during V4.6. +- Existing upload/storage aliases may remain temporarily at documented compatibility boundaries. +- Compatibility adapters contain no new business logic, persistence, provider calls, or filesystem policy. +- New application code imports the domain-owned implementation rather than adding more compatibility aliases. +- Removal of compatibility imports is deferred until usage is measured and a separate deprecation decision is made. + +## Out of Scope + +- Replacing JobSource with an event store or removing its compatibility fields. +- Rewriting, deleting, or relabeling historical ExecutionAttempt evidence. +- Changing Source candidate, promotion, or human-revision precedence. +- Changing the V4.5 orientation transformation or quality-warning rules. +- Reprocessing existing Sources solely to apply new media validation. +- Supporting additional media formats. +- Multiple selectable providers in the UI. +- Distributed queues, external brokers, distributed locks, or multi-host worker coordination. +- Concurrent processing of multiple aggregate Jobs in one application worker loop. +- Automatic paid retry triggered by quality warnings. +- Source page reordering or movement between Documents. +- A generic repository framework, generic service container framework, or generic filesystem utility layer. +- Destructive schema replacement or a V5 API redesign. + +## Locked Design Decisions + +### A. V4.6 Preserves the V4 Evidence Contract + +- Original Source bytes remain primary evidence. +- ExecutionAttempt remains append-only and authoritative. +- JobSource remains a compatibility projection. +- Human revisions remain separate from all machine output. + +### B. Requested and Resolved Model Identity Are Different Facts + +- Job stores the requested provider and model frozen before queueing. +- ExecutionAttempt stores the provider and model observed for the individual execution. +- Worker completion never rewrites the Job request specification. + +### C. Concurrency Is Page-Bounded, Not Job-Distributed + +- V4.6 adds bounded page concurrency within one claimed Job. +- One worker loop continues to own one Job aggregate at a time. +- Every page task owns its own session and durable outcome transaction. + +### D. The Source Facade Remains Stable + +- Source internals may be extracted, but callers retain a stable SourceService contract during migration. +- Candidate promotion stays transactionally close to Source persistence. +- Provider, prompt, artifact, and export concerns become replaceable collaborators rather than additional aggregates. + +### E. Filesystem and Database Partial Failure Is Explicit + +- Cross-resource operations cannot rely on a relational transaction alone. +- Observable commit failures receive compensation. +- Unavoidable cleanup failure is reported and remains diagnosable rather than being represented as complete success. + +### F. Refactoring Does Not Rewrite History + +- Existing nulls remain null unless the operator performs a separately specified maintenance action. +- Historical SDK snapshots are not reclassified as transport evidence. +- Existing preferred-machine projections and human revisions remain unchanged. + +## Data and Compatibility Policy + +- V4.6 prefers no schema change where conditional writes and focused read projections are sufficient. +- Any required schema addition is additive and portable across SQLite and PostgreSQL. +- Existing Job, Source, JobSource, ExecutionAttempt, ProcessingArtifact, Document, and Person identifiers remain valid. +- Existing Source files and artifact references retain their stored meaning. +- Existing `completed` Job rows remain readable. +- Existing API response fields are not removed by this revision. +- Compatibility adapters remain read- and call-compatible while internal ownership moves. +- Database, integration, and UI tests use confirmed isolated data and never modify `data/transcription.db`. +- Potentially destructive tests run only through `tools/run_destructive_tests.py`. + +## Acceptance Criteria + +1. Every newly queued Job has complete requested provider, model, prompt, prompt hash, and parameter provenance before + worker execution. +2. A provider-reported resolved model is stored on the ExecutionAttempt and does not change the Job's requested model. +3. Normal and retranscription Job creation reject models outside the configured allowlist. +4. Upload UI code opens no database session and performs no direct persistence. +5. A failed ingestion transaction removes files staged by that failed operation and leaves no partial database rows. +6. Supported Source content is validated beyond its extension without transforming original bytes. +7. Two concurrent claimers cannot both claim or execute the same queued Job. +8. Overlapping provider calls retain distinct request manifests and transport evidence. +9. Page execution concurrency never exceeds the configured bound and uses no shared AsyncSession. +10. Completed pages are durable while slower sibling calls remain in flight. +11. Every actual provider call creates one immutable attempt and no duplicate attempt number. +12. Concurrent first successes result in one preferred attempt and retained candidates for all other successes. +13. Commit failure during external artifact creation does not silently leave an untracked artifact. +14. External cleanup failure is visible and does not delete retained evidence metadata deceptively. +15. Source list and candidate-summary views do not load exact transport bodies. +16. Job-filtered Source lists filter in the database and preserve deterministic ordering. +17. SourceService callers remain compatible while focused collaborators become independently testable. +18. The worker emits `transcribed`, `partial_success`, or `failed` consistently, and historical `completed` rows remain + readable. +19. Previous Jobs, attempts, artifacts, Source projections, and human revisions remain unchanged by the upgrade. +20. Verification uses isolated databases, fake providers, and does not modify curated Source or evidence files. + +## Scope Freeze Gate + +V4.6 is sufficiently defined to begin implementation: + +- Submission-time versus execution-time provenance ownership is resolved. +- UI, workflow, service, and transaction boundaries are resolved. +- Claiming, attempt persistence, and first-selection concurrency rules are resolved. +- Page-level concurrency scope and default bound are resolved. +- Source extraction seams and compatibility policy are resolved. +- Media validation, artifact compensation, and query-projection expectations are resolved. +- Successful Job status terminology is resolved. +- V4.2 and V4.5 evidence, candidate, and revision contracts remain unchanged. + +Any expansion into distributed work scheduling, additional providers or media types, evidence-model replacement, +candidate-policy changes, or incompatible API/schema redesign requires an explicit V4.6 scope amendment or V5. + +## Related Local References + +- [V4.6 Implementation Plan](implementation_plan_v4_6.md) +- [V4.5 Scope Boundary](../ver4.5/scope_boundary_v4_5.md) +- [V4.5 Implementation Plan](../ver4.5/implementation_plan_v4_5.md) +- [V4 Architecture](../ver4/architecture_v4.md) +- [V4 Schema](../ver4/schema_v4.md) +- [V4 Requirements](../ver4/requirements_v4.md) +- [V4 Error Handling Policy](../ver4/error_handling_v4.md) +- [AI Evidence and Provenance Invariant](../invariant/ai_evidence_and_provenance.md) +- [Error Handling Invariant](../invariant/error_handling.md) diff --git a/docs/ver4/index_v4.md b/docs/ver4/index_v4.md index 58ab01e..281c36b 100644 --- a/docs/ver4/index_v4.md +++ b/docs/ver4/index_v4.md @@ -29,3 +29,4 @@ Version 4 is the architecture baseline for the personal-scale application used t - [V4.3 Settings Scope](../ver4.3/scope_boundary_v4_3.md) and [Implementation Plan](../ver4.3/implementation_plan_v4_3.md) - [V4.4 Semantic Registries, Linked People, and Printing Scope](../ver4.4/scope_boundary_v4_4.md) and [Implementation Plan](../ver4.4/implementation_plan_v4_4.md) - [V4.5 Transcription Input Normalization and Quality Scope](../ver4.5/scope_boundary_v4_5.md) and [Implementation Plan](../ver4.5/implementation_plan_v4_5.md) +- [V4.6 Architecture Conformance and Reliability Scope](../ver4.6/scope_boundary_v4_6.md) and [Implementation Plan](../ver4.6/implementation_plan_v4_6.md)