generated from john/python-template
V4.6 Code review: V4 architecture-conformance and reliability release
This commit is contained in:
Jim Lancaster
@@ -0,0 +1,338 @@
|
||||
# Implementation Plan (Version 4.6)
|
||||
|
||||
## Goal
|
||||
|
||||
Bring the completed V4.5 implementation into conformance with the documented V4 service, provenance, transaction,
|
||||
worker-concurrency, and evidence boundaries while preserving all historical Source and execution behavior.
|
||||
|
||||
## Planning Status
|
||||
|
||||
- V4.5 is the completed implementation baseline.
|
||||
- The V4.6 scope is frozen and sufficiently detailed to begin implementation.
|
||||
- V4.6 is a compatibility-preserving architecture and reliability revision, not a V5 redesign.
|
||||
- Scope additions require an explicit amendment or a later revision.
|
||||
|
||||
## Planning Constraints
|
||||
|
||||
- Original Source bytes and historical evidence remain immutable.
|
||||
- Existing JobSource compatibility projections remain supported.
|
||||
- Human revisions remain independent and retain display and print precedence.
|
||||
- Candidate promotion semantics remain unchanged.
|
||||
- Provider network work occurs outside database transactions.
|
||||
- Concurrent page tasks never share an AsyncSession.
|
||||
- UI pages do not own persistence transactions.
|
||||
- Refactoring proceeds behind stable public service contracts.
|
||||
- New shared code has a cohesive domain or infrastructure owner; no generic helper dumping grounds are introduced.
|
||||
- Database, integration, and UI tests use confirmed isolated data and never modify `data/transcription.db`.
|
||||
- Potentially destructive tests run only through `tools/run_destructive_tests.py`.
|
||||
|
||||
## Expected Project Impact
|
||||
|
||||
| Area | Expected impact |
|
||||
| --- | --- |
|
||||
| Job submission | Add one typed frozen request specification used by normal and retranscription workflows. |
|
||||
| Upload workflow | Move transaction and compensation ownership from the UI and direct persistence functions into orchestration. |
|
||||
| Source media | Validate content and extension consistently while preserving original bytes. |
|
||||
| Job service | Add atomic claim behavior and stable lifecycle transitions. |
|
||||
| Provider adapters | Remove mutable per-call evidence state and carry evidence through results and errors. |
|
||||
| Worker workflows | Add bounded concurrent page execution with isolated sessions and durable page outcomes. |
|
||||
| Source service | Extract cohesive collaborators behind the existing SourceService facade. |
|
||||
| Evidence persistence | Make attempt numbering, first selection, artifact creation, and cleanup conflict-safe. |
|
||||
| Read paths | Add lightweight Source and attempt projections that defer large evidence fields. |
|
||||
| Compatibility | Isolate legacy transcription and storage aliases without adding new behavior. |
|
||||
| Documentation | Align status terminology, ownership, concurrency, and provenance descriptions. |
|
||||
| Tests | Add failure-injection, concurrency, query-shape, compatibility, and preservation coverage. |
|
||||
|
||||
## Implementation Phases
|
||||
|
||||
### 1. Add Characterization and Failure-Injection Coverage
|
||||
|
||||
- Record the current externally visible Source, Job, JobSource, attempt, artifact, revision, candidate, and export
|
||||
behavior before extraction.
|
||||
- Add tests proving:
|
||||
- Historical Jobs with missing request fields remain readable.
|
||||
- JobSource SDK snapshots retain their existing evidence label.
|
||||
- Source preferred-machine fields and human revisions survive failed processing.
|
||||
- Per-page outcomes remain durable before sibling pages finish.
|
||||
- Candidate promotion updates both Source projection fields atomically.
|
||||
- Add reusable test doubles for:
|
||||
- Provider success, response failure, connection failure, timeout, and cancellation.
|
||||
- Database flush and commit failure at named workflow phases.
|
||||
- Filesystem write, replace, and unlink failure.
|
||||
- Controlled overlapping page calls.
|
||||
- Keep all providers fake and all databases and filesystem roots isolated.
|
||||
- Use these tests as behavior gates for every later extraction.
|
||||
|
||||
### 2. Define the Frozen Job Submission Specification
|
||||
|
||||
- Move runtime prompt loading, prompt hashing, requested provider/model resolution, and explicit parameter-state capture
|
||||
into a focused prompt/job-submission module.
|
||||
- Define one immutable typed value containing:
|
||||
- Requested provider.
|
||||
- Requested model.
|
||||
- Prompt name and SHA-256.
|
||||
- System and user prompt content.
|
||||
- Temperature and top-p values.
|
||||
- Explicitly supplied versus omitted state where applicable.
|
||||
- Reuse the constrained PromptStore's path policy rather than maintaining a second incompatible prompt-path policy.
|
||||
- Keep prompt editing, backup, and recovery behavior in the existing prompt-storage owner.
|
||||
- Validate model selection against `provider_models` for both normal transcription and retranscription.
|
||||
- Populate all new Job request fields before adding the Job to the session.
|
||||
- Stop mutating Job provider and model during attempt persistence.
|
||||
- Store provider-reported resolved identity on ExecutionAttempt only.
|
||||
- Retain compatibility reads for historical Jobs whose request fields are null.
|
||||
- Add tests for pre-queue completeness, allowlist rejection, prompt-edit isolation, requested/resolved model differences, and
|
||||
historical null behavior.
|
||||
|
||||
### 3. Establish Content-Aware Source Media Policy
|
||||
|
||||
- Move accepted Source extensions, canonical MIME mapping, validation, and managed payload loading into one Source-media
|
||||
module.
|
||||
- Validate filename and non-empty content first.
|
||||
- Validate JPEG, PNG, and TIFF using bounded raster decoding appropriate to the existing Pillow dependency.
|
||||
- Validate PDF structure without rendering or modifying the file.
|
||||
- Reject a supported extension whose content is corrupt or belongs to a different supported type.
|
||||
- Return a typed validated-media result containing canonical media type, digest, byte size, and original bytes or a
|
||||
stable staged reference.
|
||||
- Ensure hashing uses the exact bytes later persisted as the original Source.
|
||||
- Keep orientation normalization separate and downstream of original-media validation.
|
||||
- Preserve compatibility aliases for existing MIME and payload-loading imports.
|
||||
- Add fixtures for valid, corrupt, truncated, mismatched, uppercase-extension, empty, and unsupported content.
|
||||
|
||||
### 4. Replace Direct Upload Persistence with an Ingestion Workflow
|
||||
|
||||
- Introduce a Source-ingestion workflow that accepts validated inputs and service dependencies.
|
||||
- Generate Document and Source identifiers before file placement so managed relative paths remain deterministic.
|
||||
- Stage all Source files beneath the configured application root.
|
||||
- Build the frozen Job submission specification before starting database mutation.
|
||||
- Open one workflow-owned transaction.
|
||||
- Use DocumentService, JobService, and SourceService session-aware writes to create:
|
||||
- The Document when required.
|
||||
- The queued Job.
|
||||
- Ordered Sources.
|
||||
- Pending JobSource projections.
|
||||
- Commit once after every record has flushed successfully.
|
||||
- Delete files staged by the operation on validation, flush, or commit failure.
|
||||
- Return a typed result only after commit succeeds.
|
||||
- Move worker notification after successful workflow return.
|
||||
- Remove session-scope imports and direct persistence ownership from the Jobs UI.
|
||||
- Keep old storage entry points as thin compatibility adapters that invoke the workflow.
|
||||
- Add tests for deterministic ordering, page numbering, duplicate names, every failure phase, cleanup, commit-before-notify,
|
||||
and no direct UI persistence.
|
||||
|
||||
### 5. Add Atomic Job Claiming and Canonical Lifecycle Semantics
|
||||
|
||||
- Add `claim_next_queued_job` to JobService or a Job-owned persistence collaborator.
|
||||
- Select candidates deterministically by creation time and UUID.
|
||||
- Transition one candidate using a conditional write that succeeds only while status is queued.
|
||||
- Retry candidate selection when another claimant wins, without issuing provider work for the lost claim.
|
||||
- Commit the processing transition before provider-input preparation.
|
||||
- Route worker processing exclusively through the claim operation.
|
||||
- Preserve stale-processing recovery and require recovered Jobs to be claimed again.
|
||||
- Define `transcribed` as the newly emitted all-success terminal status.
|
||||
- Continue reading `completed` as a historical terminal status.
|
||||
- Add tests with concurrent claimers on SQLite and the portable SQL path used for PostgreSQL.
|
||||
- Add a lifecycle transition-table test covering queued, processing, transcribed, partial success, failed, cancellation,
|
||||
resubmission, and stale recovery.
|
||||
|
||||
### 6. Make Provider Calls Stateless
|
||||
|
||||
- Extend provider success and error contracts so each call directly carries:
|
||||
- RequestManifest.
|
||||
- TransportEvidence or explicit no-response evidence.
|
||||
- Failure phase.
|
||||
- Parsed SDK snapshot when available.
|
||||
- Normalized metadata when available.
|
||||
- Remove `current_request_manifest` and `current_transport_evidence` from provider adapters and worker persistence.
|
||||
- Ensure local timeout and cancellation retain the call-specific request manifest without reading mutable adapter state.
|
||||
- Keep response-body capture and safe-header filtering in the OpenRouter adapter.
|
||||
- Preserve the distinction between exact OpenRouter-boundary response bytes, SDK snapshot, and normalized fields.
|
||||
- Decide adapter reuse by concurrency safety; create per-call adapters if the SDK client cannot safely support concurrent
|
||||
calls.
|
||||
- Add overlapping-call tests in which success, HTTP failure, timeout, and no-response failure complete in different
|
||||
orders.
|
||||
- Assert that no call receives another call's source digest, prompt hash, response body, or identifiers.
|
||||
|
||||
### 7. Extract Source-Owned Evidence and Artifact Collaborators
|
||||
|
||||
- Keep SourceService as the public facade while moving cohesive implementation behind it.
|
||||
- Extract Source persistence and query behavior first without changing method contracts.
|
||||
- Extract ExecutionAttempt and JobSource compatibility persistence into a Source-owned evidence writer.
|
||||
- Extract processing-artifact file, integrity, and metadata behavior into a Source-owned artifact store.
|
||||
- Extract evidence export into a read-only projection builder.
|
||||
- Extract provider-input preparation so it coordinates:
|
||||
- Managed original Source access.
|
||||
- Orientation normalization.
|
||||
- Derivative artifact creation.
|
||||
- Exact SourceEvidenceReference construction.
|
||||
- Keep candidate promotion in Source persistence.
|
||||
- Keep normalization and quality analysis in their existing focused modules.
|
||||
- Make dependencies explicit through typed constructor parameters rather than importing service globals.
|
||||
- Do not allow extracted collaborators to call unrelated aggregate services.
|
||||
- Add contract tests showing SourceService behavior is unchanged while collaborators can be tested independently.
|
||||
|
||||
### 8. Make Artifact Creation and Cleanup Explicit
|
||||
|
||||
- Define one artifact-creation operation that owns temporary path creation, atomic file replacement, database metadata,
|
||||
and compensation.
|
||||
- Track every path created before commit within the owning workflow.
|
||||
- On flush or commit failure, remove newly created content and surface cleanup failure separately when removal also
|
||||
fails.
|
||||
- Constrain temporary and final paths beneath the artifact root and preserve safe relative references.
|
||||
- Add startup or explicit maintenance cleanup for abandoned temporary files without deleting final referenced
|
||||
artifacts.
|
||||
- For deletion:
|
||||
- Verify retention policy and references before mutation.
|
||||
- Commit metadata changes according to the explicit deletion contract.
|
||||
- Attempt external cleanup.
|
||||
- Report incomplete cleanup with correlation identity when unlink fails.
|
||||
- Do not silently skip unsafe references; classify them as validation or conflict failures as appropriate.
|
||||
- Add tests for write, replace, flush, commit, unlink, cancellation, missing file, digest mismatch, and unsafe reference
|
||||
failures.
|
||||
|
||||
### 9. Make Attempt Persistence and First Selection Conflict-Safe
|
||||
|
||||
- Stop relying solely on an unprotected `max(attempt_number) + 1` calculation.
|
||||
- Allocate or insert attempt numbers inside the protected Job processing boundary.
|
||||
- Treat the database uniqueness constraint as a final integrity guard and translate conflicts deterministically.
|
||||
- Ensure every completed provider call either persists one immutable attempt or leaves a visible persistence failure
|
||||
that does not masquerade as provider success.
|
||||
- Perform first-success Source selection with a conditional update requiring both preferred provenance and machine
|
||||
projection to remain unset.
|
||||
- When first-selection loses a race, retain the new successful attempt as a candidate.
|
||||
- Keep explicit candidate promotion as a separate atomic command.
|
||||
- Never modify earlier attempt text, transport evidence, warnings, or artifact associations.
|
||||
- Add concurrency tests for duplicate attempt allocation, simultaneous successes, success/failure overlap, candidate
|
||||
promotion, and transaction rollback.
|
||||
|
||||
### 10. Add Bounded Concurrent Page Processing
|
||||
|
||||
- Add `worker_concurrency` to Settings with default `2`, positive validation, and a documented safe upper bound.
|
||||
- Refactor the single-page body of `process_queued_job` into one typed page-attempt operation.
|
||||
- Prepare each page's provider input and persist any derivative using its own session and committed preparation phase.
|
||||
- Release all database transactions before awaiting provider network work.
|
||||
- Use an asyncio semaphore or TaskGroup-based worker pattern to cap in-flight page calls.
|
||||
- Give every page outcome persistence operation its own session and transaction.
|
||||
- Shield only the minimum durable outcome write required to preserve a completed provider call during cancellation.
|
||||
- Collect typed success, failure, and cancellation outcomes without sharing mutable lists across tasks.
|
||||
- Compute the aggregate status after all started tasks reach durable outcomes.
|
||||
- Stop scheduling unstarted pages after cancellation or an externally terminal Job transition.
|
||||
- Preserve deterministic source ordering for scheduling and deterministic aggregate reporting independent of completion
|
||||
order.
|
||||
- Add tests for concurrency bounds `1`, `2`, and a larger configured value; out-of-order completion; mixed outcomes;
|
||||
timeout; cancellation; external stop; durable early completion; and no shared session.
|
||||
|
||||
### 11. Introduce Purpose-Specific Read Projections
|
||||
|
||||
- Add a Source-list projection containing only fields needed by the Sources table.
|
||||
- Filter Document and Job scopes in SQL.
|
||||
- Add a Source-detail projection or explicit eager-loading contract for the relationships rendered by the page.
|
||||
- Add an attempt-summary projection containing candidate metadata, transcription text or an explicit preview policy,
|
||||
warning summary, and preferred-selection identity.
|
||||
- Defer transport body, SDK snapshot, request manifest, and other large evidence values from summaries.
|
||||
- Keep one explicit full-attempt evidence read for detailed inspection.
|
||||
- Keep evidence export complete and integrity-verified.
|
||||
- Replace all-sibling materialization in Source navigation with predecessor/successor queries using page number and UUID.
|
||||
- Add query-count and unloaded-attribute assertions so future changes cannot reintroduce accidental evidence loading.
|
||||
- Preserve deterministic ordering and avoid lazy database access from UI components.
|
||||
|
||||
### 12. Isolate Compatibility Adapters and Cohesive Shared Policies
|
||||
|
||||
- Keep `services/transcription.py` as a documented compatibility facade with imports and aliases only.
|
||||
- Keep legacy upload/storage aliases as thin calls into the new ingestion and media owners.
|
||||
- Mark compatibility entry points in tests so removal can be evaluated separately.
|
||||
- Consolidate Document Type and Person Role label normalization in a registry-domain policy module.
|
||||
- Share canonical evidence digest primitives only where serialization and digest meaning are identical.
|
||||
- Do not merge prompt backup writing, original Source storage, and immutable artifact storage into one generic helper;
|
||||
their lifecycle and recovery semantics remain domain-owned.
|
||||
- Remove unused service-base state only after confirming no external caller depends on it.
|
||||
- Add import-compatibility tests and direct tests for each extracted policy.
|
||||
|
||||
### 13. Align UI and API Boundaries
|
||||
|
||||
- Remove machine-local stored paths from routine Source Detail presentation.
|
||||
- Keep stored filename and evidence-safe identifiers available.
|
||||
- Ensure normal Job creation uses the configured provider and allowlisted model selector rather than arbitrary text input.
|
||||
- Keep retranscription Source and Document context locked.
|
||||
- Keep evidence export explicit and avoid loading export-only content during page render.
|
||||
- Continue translating AppError through established UI and API presenters without parsing error messages.
|
||||
- Keep sessions, session factories beyond dependency construction, and persistence operations out of reusable components.
|
||||
- Add UI tests for allowlisted selection, no path disclosure, unchanged candidate actions, and error preservation.
|
||||
|
||||
### 14. Align Authoritative Documentation
|
||||
|
||||
- Update V4 architecture for:
|
||||
- Frozen Job request versus resolved attempt identity.
|
||||
- Atomic Job claims.
|
||||
- Bounded page concurrency and isolated page sessions.
|
||||
- Source facade and internal collaborators.
|
||||
- Explicit artifact lifecycle ownership.
|
||||
- Update V4 schema documentation only for actual additive schema changes.
|
||||
- Update V4 requirements and verification mapping for atomic claims, media validation, and concurrency tests.
|
||||
- Update V4 error handling for filesystem/database partial cleanup.
|
||||
- Define `transcribed` as the current all-success terminal status and `completed` as historical compatibility.
|
||||
- Correct stale V3 module descriptions and UI documentation that exposes or promises machine-local paths.
|
||||
- Update documentation together with the behavior it describes; do not prestate unimplemented behavior as complete.
|
||||
|
||||
### 15. Verification and Preservation Audit
|
||||
|
||||
- Run focused pure tests for submission specs, media validation, provider contracts, query projections, and policies.
|
||||
- Run isolated service tests for claims, attempts, artifacts, ingestion, candidate selection, and compatibility.
|
||||
- Run worker tests with fake providers and controlled concurrent completion.
|
||||
- Run UI and API tests against isolated application state.
|
||||
- Run schema tests only through the required destructive-test wrapper when applicable.
|
||||
- Run the broader non-external regression suite after focused targets pass.
|
||||
- Verify that tests did not change:
|
||||
- `data/transcription.db`.
|
||||
- Curated Source files.
|
||||
- Existing artifact files.
|
||||
- Prompt files outside temporary fixtures.
|
||||
- Perform an explicit preservation audit showing that upgrading does not rewrite existing Job, Source, JobSource,
|
||||
ExecutionAttempt, ProcessingArtifact, or revision content.
|
||||
|
||||
## Delivery Order
|
||||
|
||||
1. Characterization and failure-injection tests.
|
||||
2. Frozen submission specification and prompt ownership.
|
||||
3. Content-aware Source media validation.
|
||||
4. Ingestion workflow and UI transaction-boundary correction.
|
||||
5. Atomic Job claiming and lifecycle terminology.
|
||||
6. Stateless provider evidence.
|
||||
7. Source evidence/artifact/provider-input extraction behind SourceService.
|
||||
8. Artifact compensation and conflict-safe attempt/selection persistence.
|
||||
9. Purpose-specific read projections.
|
||||
10. Bounded concurrent page processing.
|
||||
11. Compatibility isolation, UI/API alignment, and documentation.
|
||||
12. Full preservation and regression verification.
|
||||
|
||||
## Done Criteria
|
||||
|
||||
- All V4.6 acceptance criteria are implemented and testable.
|
||||
- New Jobs contain complete request provenance before queueing and are not rewritten by execution.
|
||||
- Every submitted model is allowlisted and requested/resolved identities remain distinct.
|
||||
- Upload UI code owns no database transaction.
|
||||
- Source content validation verifies supported media without modifying originals.
|
||||
- One queued Job can be claimed only once.
|
||||
- Provider evidence is call-local and safe under overlapping execution.
|
||||
- Page calls run concurrently within the configured bound and never share sessions.
|
||||
- Every actual provider call has one immutable attempt.
|
||||
- Concurrent first successes preserve one preferred result and every other success as a candidate.
|
||||
- Artifact commit and cleanup failures are visible and compensated where safely possible.
|
||||
- Source list and candidate views avoid loading full evidence bodies.
|
||||
- SourceService remains compatible while extracted responsibilities are independently testable.
|
||||
- Human revisions, preferred-machine projections, prior attempts, JobSource snapshots, and artifacts retain their original
|
||||
meaning.
|
||||
- The worker consistently emits transcribed, partial_success, or failed for new processing.
|
||||
- Verification uses isolated data, fake providers, and does not modify operator or curated evidence data.
|
||||
|
||||
## Related Local References
|
||||
|
||||
- [V4.6 Scope Boundary](scope_boundary_v4_6.md)
|
||||
- [V4.5 Scope Boundary](../ver4.5/scope_boundary_v4_5.md)
|
||||
- [V4.5 Implementation Plan](../ver4.5/implementation_plan_v4_5.md)
|
||||
- [V4 Architecture](../ver4/architecture_v4.md)
|
||||
- [V4 Schema](../ver4/schema_v4.md)
|
||||
- [V4 Requirements](../ver4/requirements_v4.md)
|
||||
- [V4 Error Handling Policy](../ver4/error_handling_v4.md)
|
||||
- [AI Evidence and Provenance Invariant](../invariant/ai_evidence_and_provenance.md)
|
||||
- [Error Handling Invariant](../invariant/error_handling.md)
|
||||
@@ -0,0 +1,296 @@
|
||||
# V4.6 Scope Boundary
|
||||
|
||||
This document defines the frozen boundary for architecture conformance, worker reliability, and Source-service
|
||||
maintainability after the completed V4.5 revision. V4 through V4.5 remain the behavioral and evidence baseline.
|
||||
V4.6 corrects implementation divergence and reorganizes internal ownership without replacing the V4 data model,
|
||||
public identity rules, or preservation contracts.
|
||||
|
||||
## Purpose
|
||||
|
||||
- Freeze complete Job request provenance before work is queued.
|
||||
- Make job claiming, attempt persistence, and first-success selection safe under concurrency.
|
||||
- Implement the bounded page-processing concurrency required by the V4 architecture.
|
||||
- Restore UI, workflow, service, transaction, provider, and filesystem ownership boundaries.
|
||||
- Separate cohesive Source-domain responsibilities behind compatibility-preserving service contracts.
|
||||
- Avoid loading large evidence fields for list and candidate-summary views.
|
||||
- Correct documentation that no longer describes the implemented lifecycle accurately.
|
||||
|
||||
## In Scope
|
||||
|
||||
### 1. Frozen Job Submission Specification
|
||||
|
||||
- Every newly queued Job records its requested provider, model, prompt name, prompt hash, full prompt content,
|
||||
and explicitly configured generation parameters before the Job transaction commits.
|
||||
- Normal transcription and Source retranscription use the same typed submission-specification builder.
|
||||
- Provider and model values are resolved from validated application configuration.
|
||||
- Every model submitted through the UI or another application workflow is validated against the configured model
|
||||
allowlist.
|
||||
- A provider response may report a resolved model that differs from the requested model. The resolved provider and
|
||||
model belong to the immutable ExecutionAttempt and do not rewrite the Job request specification.
|
||||
- Existing Jobs with null or incomplete historical fields remain readable and are not backfilled with facts that
|
||||
were not frozen at submission.
|
||||
- Prompt edits continue to affect future Jobs only and never rewrite stored Job or attempt provenance.
|
||||
|
||||
### 2. Source Ingestion Workflow Ownership
|
||||
|
||||
- Source ingestion is an application workflow, not a UI transaction and not a generic file utility.
|
||||
- The ingestion workflow owns:
|
||||
- Source-media validation.
|
||||
- Original-file staging and hashing.
|
||||
- Document lookup or creation when applicable.
|
||||
- Job creation from a frozen submission specification.
|
||||
- Ordered Source creation.
|
||||
- JobSource creation.
|
||||
- One database commit for the related records.
|
||||
- Compensating cleanup when persistence fails before the workflow completes.
|
||||
- UI pages resolve input, invoke the workflow, notify the worker only after successful commit, and navigate or
|
||||
display errors.
|
||||
- UI pages do not import session scopes, open transactions, execute queries, or commit and roll back persistence.
|
||||
- Workflow code coordinates service contracts rather than inserting Document, Job, Source, or JobSource rows
|
||||
directly.
|
||||
|
||||
### 3. Source Media Validation
|
||||
|
||||
- Filename extension remains part of the accepted-format policy but is not sufficient proof of media type.
|
||||
- New Source uploads are validated as non-empty, structurally recognizable JPEG, PNG, TIFF, or PDF content.
|
||||
- The validated content type must agree with the supported filename extension.
|
||||
- Raster validation may decode enough content to establish that the file is a supported image, but validation does
|
||||
not transform, re-encode, or replace the original bytes.
|
||||
- PDF validation establishes that the content is a recognizable PDF without rendering, rewriting, or normalizing it.
|
||||
- Original uploaded bytes, SHA-256 digest, byte size, upload name, and evidence identity remain unchanged.
|
||||
- Historical Sources are not rejected merely because they predate content-aware validation.
|
||||
|
||||
### 4. Atomic Job Claiming
|
||||
|
||||
- Claiming a queued Job is one persistence operation that conditionally transitions
|
||||
`queued -> processing`.
|
||||
- Two workers or application instances cannot both successfully claim the same queued Job.
|
||||
- Claim ordering remains deterministic by creation time and UUID.
|
||||
- A worker that loses a claim performs no provider call and creates no execution evidence for that Job.
|
||||
- Claim behavior remains portable across supported SQLite and PostgreSQL deployments.
|
||||
- Startup recovery may requeue stale processing Jobs according to existing policy, but recovery does not bypass the
|
||||
normal claim operation.
|
||||
|
||||
### 5. Stateless Provider Execution Evidence
|
||||
|
||||
- Request manifest and transport evidence belong to one provider call and are returned through that call's result or
|
||||
exception.
|
||||
- Provider adapters do not expose mutable "current request" or "current response" state for worker persistence.
|
||||
- Success, HTTP failure, connection failure, local timeout, cancellation, and response-validation failure retain
|
||||
their own request and transport evidence without cross-attribution.
|
||||
- Provider adapters remain responsible for exact provider-boundary capture and safe-header allowlisting.
|
||||
- Provider calls occur outside database transactions.
|
||||
- Provider adapter instances may be reused only when their per-call state is concurrency-safe.
|
||||
|
||||
### 6. Bounded Page Concurrency
|
||||
|
||||
- One claimed multi-page Job may process page provider calls concurrently.
|
||||
- A validated `worker_concurrency` setting defines the maximum number of in-flight page calls.
|
||||
- The default concurrency is `2`; allowed values are positive and bounded to a documented safe maximum.
|
||||
- The worker continues to orchestrate one claimed Job at a time within one application worker loop. V4.6 does not add
|
||||
distributed scheduling or simultaneous aggregate processing of multiple Jobs in one process.
|
||||
- Each page task uses its own database session for preparation and outcome persistence; an AsyncSession is never
|
||||
shared across concurrent page tasks.
|
||||
- Provider-input preparation completes and commits before that page's provider call begins.
|
||||
- Each completed page outcome is committed durably without waiting for every sibling page.
|
||||
- One page failure does not cancel successful independent pages.
|
||||
- Aggregate Job status is computed after every started page task reaches a durable outcome or explicit cancellation.
|
||||
- Cancellation prevents unstarted work and preserves durable outcomes from calls that already completed.
|
||||
- Concurrency does not change provider cost policy: quality warnings never trigger automatic provider calls.
|
||||
|
||||
### 7. Conflict-Safe Attempts and Machine Selection
|
||||
|
||||
- Every actual provider call creates exactly one immutable ExecutionAttempt, including failure and timeout outcomes.
|
||||
- Attempt numbering remains unique for `(job_id, source_id, attempt_number)` and does not use an unprotected
|
||||
application-only `max + 1` assumption.
|
||||
- A duplicate persistence race fails deterministically without overwriting existing evidence.
|
||||
- The first successful attempt may establish preferred machine output only when the Source has no selected machine
|
||||
provenance at the atomic write boundary.
|
||||
- If another success wins first-selection concurrently, the losing successful attempt remains an unselected
|
||||
candidate.
|
||||
- Candidate promotion continues to update `Source.preferred_execution_attempt_id` and
|
||||
`Source.raw_transcription` atomically.
|
||||
- Human revision remains independent and is never cleared or changed by execution or promotion.
|
||||
|
||||
### 8. Processing Artifact Lifecycle
|
||||
|
||||
- Processing-artifact content and database metadata have one explicit creation owner.
|
||||
- When artifact persistence fails or its enclosing commit fails, files created by that operation are removed when
|
||||
safely possible and the failure remains visible.
|
||||
- Temporary artifact files use constrained application-managed paths and are recoverable or removable after
|
||||
interruption.
|
||||
- Artifact integrity verification continues to distinguish unavailable content, malformed metadata, digest mismatch,
|
||||
and unsafe external references.
|
||||
- Deleting artifact metadata and deleting external content is an explicit partial-failure-capable operation.
|
||||
- A failed external-file deletion is logged with correlation identity and reported as incomplete cleanup rather than
|
||||
silent success.
|
||||
- No cleanup path deletes an artifact still referenced by retained immutable evidence.
|
||||
|
||||
### 9. Source Service Responsibility Boundaries
|
||||
|
||||
- `SourceService` remains the public Source aggregate facade during V4.6.
|
||||
- Internally cohesive Source-domain collaborators may own:
|
||||
- Source persistence, queries, navigation, and revisions.
|
||||
- Source media policy and managed original-file access.
|
||||
- Prompt execution snapshots.
|
||||
- Provider-input resolution and normalization-artifact association.
|
||||
- ExecutionAttempt and JobSource projection persistence.
|
||||
- Processing-artifact storage and integrity.
|
||||
- Evidence export.
|
||||
- Provider execution and error translation.
|
||||
- Candidate promotion remains in the Source aggregate persistence boundary because it atomically changes the Source's
|
||||
selected provenance and compatibility text projection.
|
||||
- Orientation transformation remains in the focused normalization module.
|
||||
- Deterministic warning analysis remains in the focused quality module.
|
||||
- Extraction does not create a generic `utils.py`, `helpers.py`, or `common.py` service dumping ground.
|
||||
- Services remain independent; multi-service behavior stays in workflow functions or workflow objects.
|
||||
|
||||
### 10. Read Projections and Query Efficiency
|
||||
|
||||
- Source list, Source Detail, candidate summary, and evidence export use purpose-specific read contracts.
|
||||
- Candidate and list views do not load exact transport bodies, full SDK snapshots, or other large evidence fields
|
||||
that they do not render.
|
||||
- Full evidence bodies remain available through explicit evidence inspection or export.
|
||||
- Job-filtered Source queries filter in SQL rather than loading every Source and filtering JobSource relationships in
|
||||
application memory.
|
||||
- Source navigation preserves deterministic `(page_number, UUID)` ordering without requiring all sibling Source
|
||||
records to be materialized.
|
||||
- Query optimization does not introduce lazy-loading behavior into detached UI models.
|
||||
|
||||
### 11. Lifecycle and Terminology Alignment
|
||||
|
||||
- `transcribed` is the canonical successful terminal Job status for the current transcription workflow.
|
||||
- `completed` remains a readable historical/compatibility status but is not newly emitted by the V4.6 transcription
|
||||
worker.
|
||||
- `partial_success` and `failed` retain their existing aggregate meanings.
|
||||
- Architecture, schema, requirements, error-handling, UI, and model documentation use the same status terminology.
|
||||
- JobSource remains the mutable queue and compatibility projection.
|
||||
- ExecutionAttempt remains the authoritative append-only execution history.
|
||||
|
||||
### 12. Compatibility Adapter Isolation
|
||||
|
||||
- Existing imports from `transcription.services.transcription` continue to resolve during V4.6.
|
||||
- Existing upload/storage aliases may remain temporarily at documented compatibility boundaries.
|
||||
- Compatibility adapters contain no new business logic, persistence, provider calls, or filesystem policy.
|
||||
- New application code imports the domain-owned implementation rather than adding more compatibility aliases.
|
||||
- Removal of compatibility imports is deferred until usage is measured and a separate deprecation decision is made.
|
||||
|
||||
## Out of Scope
|
||||
|
||||
- Replacing JobSource with an event store or removing its compatibility fields.
|
||||
- Rewriting, deleting, or relabeling historical ExecutionAttempt evidence.
|
||||
- Changing Source candidate, promotion, or human-revision precedence.
|
||||
- Changing the V4.5 orientation transformation or quality-warning rules.
|
||||
- Reprocessing existing Sources solely to apply new media validation.
|
||||
- Supporting additional media formats.
|
||||
- Multiple selectable providers in the UI.
|
||||
- Distributed queues, external brokers, distributed locks, or multi-host worker coordination.
|
||||
- Concurrent processing of multiple aggregate Jobs in one application worker loop.
|
||||
- Automatic paid retry triggered by quality warnings.
|
||||
- Source page reordering or movement between Documents.
|
||||
- A generic repository framework, generic service container framework, or generic filesystem utility layer.
|
||||
- Destructive schema replacement or a V5 API redesign.
|
||||
|
||||
## Locked Design Decisions
|
||||
|
||||
### A. V4.6 Preserves the V4 Evidence Contract
|
||||
|
||||
- Original Source bytes remain primary evidence.
|
||||
- ExecutionAttempt remains append-only and authoritative.
|
||||
- JobSource remains a compatibility projection.
|
||||
- Human revisions remain separate from all machine output.
|
||||
|
||||
### B. Requested and Resolved Model Identity Are Different Facts
|
||||
|
||||
- Job stores the requested provider and model frozen before queueing.
|
||||
- ExecutionAttempt stores the provider and model observed for the individual execution.
|
||||
- Worker completion never rewrites the Job request specification.
|
||||
|
||||
### C. Concurrency Is Page-Bounded, Not Job-Distributed
|
||||
|
||||
- V4.6 adds bounded page concurrency within one claimed Job.
|
||||
- One worker loop continues to own one Job aggregate at a time.
|
||||
- Every page task owns its own session and durable outcome transaction.
|
||||
|
||||
### D. The Source Facade Remains Stable
|
||||
|
||||
- Source internals may be extracted, but callers retain a stable SourceService contract during migration.
|
||||
- Candidate promotion stays transactionally close to Source persistence.
|
||||
- Provider, prompt, artifact, and export concerns become replaceable collaborators rather than additional aggregates.
|
||||
|
||||
### E. Filesystem and Database Partial Failure Is Explicit
|
||||
|
||||
- Cross-resource operations cannot rely on a relational transaction alone.
|
||||
- Observable commit failures receive compensation.
|
||||
- Unavoidable cleanup failure is reported and remains diagnosable rather than being represented as complete success.
|
||||
|
||||
### F. Refactoring Does Not Rewrite History
|
||||
|
||||
- Existing nulls remain null unless the operator performs a separately specified maintenance action.
|
||||
- Historical SDK snapshots are not reclassified as transport evidence.
|
||||
- Existing preferred-machine projections and human revisions remain unchanged.
|
||||
|
||||
## Data and Compatibility Policy
|
||||
|
||||
- V4.6 prefers no schema change where conditional writes and focused read projections are sufficient.
|
||||
- Any required schema addition is additive and portable across SQLite and PostgreSQL.
|
||||
- Existing Job, Source, JobSource, ExecutionAttempt, ProcessingArtifact, Document, and Person identifiers remain valid.
|
||||
- Existing Source files and artifact references retain their stored meaning.
|
||||
- Existing `completed` Job rows remain readable.
|
||||
- Existing API response fields are not removed by this revision.
|
||||
- Compatibility adapters remain read- and call-compatible while internal ownership moves.
|
||||
- Database, integration, and UI tests use confirmed isolated data and never modify `data/transcription.db`.
|
||||
- Potentially destructive tests run only through `tools/run_destructive_tests.py`.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
1. Every newly queued Job has complete requested provider, model, prompt, prompt hash, and parameter provenance before
|
||||
worker execution.
|
||||
2. A provider-reported resolved model is stored on the ExecutionAttempt and does not change the Job's requested model.
|
||||
3. Normal and retranscription Job creation reject models outside the configured allowlist.
|
||||
4. Upload UI code opens no database session and performs no direct persistence.
|
||||
5. A failed ingestion transaction removes files staged by that failed operation and leaves no partial database rows.
|
||||
6. Supported Source content is validated beyond its extension without transforming original bytes.
|
||||
7. Two concurrent claimers cannot both claim or execute the same queued Job.
|
||||
8. Overlapping provider calls retain distinct request manifests and transport evidence.
|
||||
9. Page execution concurrency never exceeds the configured bound and uses no shared AsyncSession.
|
||||
10. Completed pages are durable while slower sibling calls remain in flight.
|
||||
11. Every actual provider call creates one immutable attempt and no duplicate attempt number.
|
||||
12. Concurrent first successes result in one preferred attempt and retained candidates for all other successes.
|
||||
13. Commit failure during external artifact creation does not silently leave an untracked artifact.
|
||||
14. External cleanup failure is visible and does not delete retained evidence metadata deceptively.
|
||||
15. Source list and candidate-summary views do not load exact transport bodies.
|
||||
16. Job-filtered Source lists filter in the database and preserve deterministic ordering.
|
||||
17. SourceService callers remain compatible while focused collaborators become independently testable.
|
||||
18. The worker emits `transcribed`, `partial_success`, or `failed` consistently, and historical `completed` rows remain
|
||||
readable.
|
||||
19. Previous Jobs, attempts, artifacts, Source projections, and human revisions remain unchanged by the upgrade.
|
||||
20. Verification uses isolated databases, fake providers, and does not modify curated Source or evidence files.
|
||||
|
||||
## Scope Freeze Gate
|
||||
|
||||
V4.6 is sufficiently defined to begin implementation:
|
||||
|
||||
- Submission-time versus execution-time provenance ownership is resolved.
|
||||
- UI, workflow, service, and transaction boundaries are resolved.
|
||||
- Claiming, attempt persistence, and first-selection concurrency rules are resolved.
|
||||
- Page-level concurrency scope and default bound are resolved.
|
||||
- Source extraction seams and compatibility policy are resolved.
|
||||
- Media validation, artifact compensation, and query-projection expectations are resolved.
|
||||
- Successful Job status terminology is resolved.
|
||||
- V4.2 and V4.5 evidence, candidate, and revision contracts remain unchanged.
|
||||
|
||||
Any expansion into distributed work scheduling, additional providers or media types, evidence-model replacement,
|
||||
candidate-policy changes, or incompatible API/schema redesign requires an explicit V4.6 scope amendment or V5.
|
||||
|
||||
## Related Local References
|
||||
|
||||
- [V4.6 Implementation Plan](implementation_plan_v4_6.md)
|
||||
- [V4.5 Scope Boundary](../ver4.5/scope_boundary_v4_5.md)
|
||||
- [V4.5 Implementation Plan](../ver4.5/implementation_plan_v4_5.md)
|
||||
- [V4 Architecture](../ver4/architecture_v4.md)
|
||||
- [V4 Schema](../ver4/schema_v4.md)
|
||||
- [V4 Requirements](../ver4/requirements_v4.md)
|
||||
- [V4 Error Handling Policy](../ver4/error_handling_v4.md)
|
||||
- [AI Evidence and Provenance Invariant](../invariant/ai_evidence_and_provenance.md)
|
||||
- [Error Handling Invariant](../invariant/error_handling.md)
|
||||
Reference in New Issue
Block a user