generated from john/python-template
Hide V4.6 recommendations after code review
This commit is contained in:
zoltan57
@@ -1,296 +0,0 @@
|
||||
# V4.6 Scope Boundary
|
||||
|
||||
This document defines the frozen boundary for architecture conformance, worker reliability, and Source-service
|
||||
maintainability after the completed V4.5 revision. V4 through V4.5 remain the behavioral and evidence baseline.
|
||||
V4.6 corrects implementation divergence and reorganizes internal ownership without replacing the V4 data model,
|
||||
public identity rules, or preservation contracts.
|
||||
|
||||
## Purpose
|
||||
|
||||
- Freeze complete Job request provenance before work is queued.
|
||||
- Make job claiming, attempt persistence, and first-success selection safe under concurrency.
|
||||
- Implement the bounded page-processing concurrency required by the V4 architecture.
|
||||
- Restore UI, workflow, service, transaction, provider, and filesystem ownership boundaries.
|
||||
- Separate cohesive Source-domain responsibilities behind compatibility-preserving service contracts.
|
||||
- Avoid loading large evidence fields for list and candidate-summary views.
|
||||
- Correct documentation that no longer describes the implemented lifecycle accurately.
|
||||
|
||||
## In Scope
|
||||
|
||||
### 1. Frozen Job Submission Specification
|
||||
|
||||
- Every newly queued Job records its requested provider, model, prompt name, prompt hash, full prompt content,
|
||||
and explicitly configured generation parameters before the Job transaction commits.
|
||||
- Normal transcription and Source retranscription use the same typed submission-specification builder.
|
||||
- Provider and model values are resolved from validated application configuration.
|
||||
- Every model submitted through the UI or another application workflow is validated against the configured model
|
||||
allowlist.
|
||||
- A provider response may report a resolved model that differs from the requested model. The resolved provider and
|
||||
model belong to the immutable ExecutionAttempt and do not rewrite the Job request specification.
|
||||
- Existing Jobs with null or incomplete historical fields remain readable and are not backfilled with facts that
|
||||
were not frozen at submission.
|
||||
- Prompt edits continue to affect future Jobs only and never rewrite stored Job or attempt provenance.
|
||||
|
||||
### 2. Source Ingestion Workflow Ownership
|
||||
|
||||
- Source ingestion is an application workflow, not a UI transaction and not a generic file utility.
|
||||
- The ingestion workflow owns:
|
||||
- Source-media validation.
|
||||
- Original-file staging and hashing.
|
||||
- Document lookup or creation when applicable.
|
||||
- Job creation from a frozen submission specification.
|
||||
- Ordered Source creation.
|
||||
- JobSource creation.
|
||||
- One database commit for the related records.
|
||||
- Compensating cleanup when persistence fails before the workflow completes.
|
||||
- UI pages resolve input, invoke the workflow, notify the worker only after successful commit, and navigate or
|
||||
display errors.
|
||||
- UI pages do not import session scopes, open transactions, execute queries, or commit and roll back persistence.
|
||||
- Workflow code coordinates service contracts rather than inserting Document, Job, Source, or JobSource rows
|
||||
directly.
|
||||
|
||||
### 3. Source Media Validation
|
||||
|
||||
- Filename extension remains part of the accepted-format policy but is not sufficient proof of media type.
|
||||
- New Source uploads are validated as non-empty, structurally recognizable JPEG, PNG, TIFF, or PDF content.
|
||||
- The validated content type must agree with the supported filename extension.
|
||||
- Raster validation may decode enough content to establish that the file is a supported image, but validation does
|
||||
not transform, re-encode, or replace the original bytes.
|
||||
- PDF validation establishes that the content is a recognizable PDF without rendering, rewriting, or normalizing it.
|
||||
- Original uploaded bytes, SHA-256 digest, byte size, upload name, and evidence identity remain unchanged.
|
||||
- Historical Sources are not rejected merely because they predate content-aware validation.
|
||||
|
||||
### 4. Atomic Job Claiming
|
||||
|
||||
- Claiming a queued Job is one persistence operation that conditionally transitions
|
||||
`queued -> processing`.
|
||||
- Two workers or application instances cannot both successfully claim the same queued Job.
|
||||
- Claim ordering remains deterministic by creation time and UUID.
|
||||
- A worker that loses a claim performs no provider call and creates no execution evidence for that Job.
|
||||
- Claim behavior remains portable across supported SQLite and PostgreSQL deployments.
|
||||
- Startup recovery may requeue stale processing Jobs according to existing policy, but recovery does not bypass the
|
||||
normal claim operation.
|
||||
|
||||
### 5. Stateless Provider Execution Evidence
|
||||
|
||||
- Request manifest and transport evidence belong to one provider call and are returned through that call's result or
|
||||
exception.
|
||||
- Provider adapters do not expose mutable "current request" or "current response" state for worker persistence.
|
||||
- Success, HTTP failure, connection failure, local timeout, cancellation, and response-validation failure retain
|
||||
their own request and transport evidence without cross-attribution.
|
||||
- Provider adapters remain responsible for exact provider-boundary capture and safe-header allowlisting.
|
||||
- Provider calls occur outside database transactions.
|
||||
- Provider adapter instances may be reused only when their per-call state is concurrency-safe.
|
||||
|
||||
### 6. Bounded Page Concurrency
|
||||
|
||||
- One claimed multi-page Job may process page provider calls concurrently.
|
||||
- A validated `worker_concurrency` setting defines the maximum number of in-flight page calls.
|
||||
- The default concurrency is `2`; allowed values are positive and bounded to a documented safe maximum.
|
||||
- The worker continues to orchestrate one claimed Job at a time within one application worker loop. V4.6 does not add
|
||||
distributed scheduling or simultaneous aggregate processing of multiple Jobs in one process.
|
||||
- Each page task uses its own database session for preparation and outcome persistence; an AsyncSession is never
|
||||
shared across concurrent page tasks.
|
||||
- Provider-input preparation completes and commits before that page's provider call begins.
|
||||
- Each completed page outcome is committed durably without waiting for every sibling page.
|
||||
- One page failure does not cancel successful independent pages.
|
||||
- Aggregate Job status is computed after every started page task reaches a durable outcome or explicit cancellation.
|
||||
- Cancellation prevents unstarted work and preserves durable outcomes from calls that already completed.
|
||||
- Concurrency does not change provider cost policy: quality warnings never trigger automatic provider calls.
|
||||
|
||||
### 7. Conflict-Safe Attempts and Machine Selection
|
||||
|
||||
- Every actual provider call creates exactly one immutable ExecutionAttempt, including failure and timeout outcomes.
|
||||
- Attempt numbering remains unique for `(job_id, source_id, attempt_number)` and does not use an unprotected
|
||||
application-only `max + 1` assumption.
|
||||
- A duplicate persistence race fails deterministically without overwriting existing evidence.
|
||||
- The first successful attempt may establish preferred machine output only when the Source has no selected machine
|
||||
provenance at the atomic write boundary.
|
||||
- If another success wins first-selection concurrently, the losing successful attempt remains an unselected
|
||||
candidate.
|
||||
- Candidate promotion continues to update `Source.preferred_execution_attempt_id` and
|
||||
`Source.raw_transcription` atomically.
|
||||
- Human revision remains independent and is never cleared or changed by execution or promotion.
|
||||
|
||||
### 8. Processing Artifact Lifecycle
|
||||
|
||||
- Processing-artifact content and database metadata have one explicit creation owner.
|
||||
- When artifact persistence fails or its enclosing commit fails, files created by that operation are removed when
|
||||
safely possible and the failure remains visible.
|
||||
- Temporary artifact files use constrained application-managed paths and are recoverable or removable after
|
||||
interruption.
|
||||
- Artifact integrity verification continues to distinguish unavailable content, malformed metadata, digest mismatch,
|
||||
and unsafe external references.
|
||||
- Deleting artifact metadata and deleting external content is an explicit partial-failure-capable operation.
|
||||
- A failed external-file deletion is logged with correlation identity and reported as incomplete cleanup rather than
|
||||
silent success.
|
||||
- No cleanup path deletes an artifact still referenced by retained immutable evidence.
|
||||
|
||||
### 9. Source Service Responsibility Boundaries
|
||||
|
||||
- `SourceService` remains the public Source aggregate facade during V4.6.
|
||||
- Internally cohesive Source-domain collaborators may own:
|
||||
- Source persistence, queries, navigation, and revisions.
|
||||
- Source media policy and managed original-file access.
|
||||
- Prompt execution snapshots.
|
||||
- Provider-input resolution and normalization-artifact association.
|
||||
- ExecutionAttempt and JobSource projection persistence.
|
||||
- Processing-artifact storage and integrity.
|
||||
- Evidence export.
|
||||
- Provider execution and error translation.
|
||||
- Candidate promotion remains in the Source aggregate persistence boundary because it atomically changes the Source's
|
||||
selected provenance and compatibility text projection.
|
||||
- Orientation transformation remains in the focused normalization module.
|
||||
- Deterministic warning analysis remains in the focused quality module.
|
||||
- Extraction does not create a generic `utils.py`, `helpers.py`, or `common.py` service dumping ground.
|
||||
- Services remain independent; multi-service behavior stays in workflow functions or workflow objects.
|
||||
|
||||
### 10. Read Projections and Query Efficiency
|
||||
|
||||
- Source list, Source Detail, candidate summary, and evidence export use purpose-specific read contracts.
|
||||
- Candidate and list views do not load exact transport bodies, full SDK snapshots, or other large evidence fields
|
||||
that they do not render.
|
||||
- Full evidence bodies remain available through explicit evidence inspection or export.
|
||||
- Job-filtered Source queries filter in SQL rather than loading every Source and filtering JobSource relationships in
|
||||
application memory.
|
||||
- Source navigation preserves deterministic `(page_number, UUID)` ordering without requiring all sibling Source
|
||||
records to be materialized.
|
||||
- Query optimization does not introduce lazy-loading behavior into detached UI models.
|
||||
|
||||
### 11. Lifecycle and Terminology Alignment
|
||||
|
||||
- `transcribed` is the canonical successful terminal Job status for the current transcription workflow.
|
||||
- `completed` remains a readable historical/compatibility status but is not newly emitted by the V4.6 transcription
|
||||
worker.
|
||||
- `partial_success` and `failed` retain their existing aggregate meanings.
|
||||
- Architecture, schema, requirements, error-handling, UI, and model documentation use the same status terminology.
|
||||
- JobSource remains the mutable queue and compatibility projection.
|
||||
- ExecutionAttempt remains the authoritative append-only execution history.
|
||||
|
||||
### 12. Compatibility Adapter Isolation
|
||||
|
||||
- Existing imports from `transcription.services.transcription` continue to resolve during V4.6.
|
||||
- Existing upload/storage aliases may remain temporarily at documented compatibility boundaries.
|
||||
- Compatibility adapters contain no new business logic, persistence, provider calls, or filesystem policy.
|
||||
- New application code imports the domain-owned implementation rather than adding more compatibility aliases.
|
||||
- Removal of compatibility imports is deferred until usage is measured and a separate deprecation decision is made.
|
||||
|
||||
## Out of Scope
|
||||
|
||||
- Replacing JobSource with an event store or removing its compatibility fields.
|
||||
- Rewriting, deleting, or relabeling historical ExecutionAttempt evidence.
|
||||
- Changing Source candidate, promotion, or human-revision precedence.
|
||||
- Changing the V4.5 orientation transformation or quality-warning rules.
|
||||
- Reprocessing existing Sources solely to apply new media validation.
|
||||
- Supporting additional media formats.
|
||||
- Multiple selectable providers in the UI.
|
||||
- Distributed queues, external brokers, distributed locks, or multi-host worker coordination.
|
||||
- Concurrent processing of multiple aggregate Jobs in one application worker loop.
|
||||
- Automatic paid retry triggered by quality warnings.
|
||||
- Source page reordering or movement between Documents.
|
||||
- A generic repository framework, generic service container framework, or generic filesystem utility layer.
|
||||
- Destructive schema replacement or a V5 API redesign.
|
||||
|
||||
## Locked Design Decisions
|
||||
|
||||
### A. V4.6 Preserves the V4 Evidence Contract
|
||||
|
||||
- Original Source bytes remain primary evidence.
|
||||
- ExecutionAttempt remains append-only and authoritative.
|
||||
- JobSource remains a compatibility projection.
|
||||
- Human revisions remain separate from all machine output.
|
||||
|
||||
### B. Requested and Resolved Model Identity Are Different Facts
|
||||
|
||||
- Job stores the requested provider and model frozen before queueing.
|
||||
- ExecutionAttempt stores the provider and model observed for the individual execution.
|
||||
- Worker completion never rewrites the Job request specification.
|
||||
|
||||
### C. Concurrency Is Page-Bounded, Not Job-Distributed
|
||||
|
||||
- V4.6 adds bounded page concurrency within one claimed Job.
|
||||
- One worker loop continues to own one Job aggregate at a time.
|
||||
- Every page task owns its own session and durable outcome transaction.
|
||||
|
||||
### D. The Source Facade Remains Stable
|
||||
|
||||
- Source internals may be extracted, but callers retain a stable SourceService contract during migration.
|
||||
- Candidate promotion stays transactionally close to Source persistence.
|
||||
- Provider, prompt, artifact, and export concerns become replaceable collaborators rather than additional aggregates.
|
||||
|
||||
### E. Filesystem and Database Partial Failure Is Explicit
|
||||
|
||||
- Cross-resource operations cannot rely on a relational transaction alone.
|
||||
- Observable commit failures receive compensation.
|
||||
- Unavoidable cleanup failure is reported and remains diagnosable rather than being represented as complete success.
|
||||
|
||||
### F. Refactoring Does Not Rewrite History
|
||||
|
||||
- Existing nulls remain null unless the operator performs a separately specified maintenance action.
|
||||
- Historical SDK snapshots are not reclassified as transport evidence.
|
||||
- Existing preferred-machine projections and human revisions remain unchanged.
|
||||
|
||||
## Data and Compatibility Policy
|
||||
|
||||
- V4.6 prefers no schema change where conditional writes and focused read projections are sufficient.
|
||||
- Any required schema addition is additive and portable across SQLite and PostgreSQL.
|
||||
- Existing Job, Source, JobSource, ExecutionAttempt, ProcessingArtifact, Document, and Person identifiers remain valid.
|
||||
- Existing Source files and artifact references retain their stored meaning.
|
||||
- Existing `completed` Job rows remain readable.
|
||||
- Existing API response fields are not removed by this revision.
|
||||
- Compatibility adapters remain read- and call-compatible while internal ownership moves.
|
||||
- Database, integration, and UI tests use confirmed isolated data and never modify `data/transcription.db`.
|
||||
- Potentially destructive tests run only through `tools/run_destructive_tests.py`.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
1. Every newly queued Job has complete requested provider, model, prompt, prompt hash, and parameter provenance before
|
||||
worker execution.
|
||||
2. A provider-reported resolved model is stored on the ExecutionAttempt and does not change the Job's requested model.
|
||||
3. Normal and retranscription Job creation reject models outside the configured allowlist.
|
||||
4. Upload UI code opens no database session and performs no direct persistence.
|
||||
5. A failed ingestion transaction removes files staged by that failed operation and leaves no partial database rows.
|
||||
6. Supported Source content is validated beyond its extension without transforming original bytes.
|
||||
7. Two concurrent claimers cannot both claim or execute the same queued Job.
|
||||
8. Overlapping provider calls retain distinct request manifests and transport evidence.
|
||||
9. Page execution concurrency never exceeds the configured bound and uses no shared AsyncSession.
|
||||
10. Completed pages are durable while slower sibling calls remain in flight.
|
||||
11. Every actual provider call creates one immutable attempt and no duplicate attempt number.
|
||||
12. Concurrent first successes result in one preferred attempt and retained candidates for all other successes.
|
||||
13. Commit failure during external artifact creation does not silently leave an untracked artifact.
|
||||
14. External cleanup failure is visible and does not delete retained evidence metadata deceptively.
|
||||
15. Source list and candidate-summary views do not load exact transport bodies.
|
||||
16. Job-filtered Source lists filter in the database and preserve deterministic ordering.
|
||||
17. SourceService callers remain compatible while focused collaborators become independently testable.
|
||||
18. The worker emits `transcribed`, `partial_success`, or `failed` consistently, and historical `completed` rows remain
|
||||
readable.
|
||||
19. Previous Jobs, attempts, artifacts, Source projections, and human revisions remain unchanged by the upgrade.
|
||||
20. Verification uses isolated databases, fake providers, and does not modify curated Source or evidence files.
|
||||
|
||||
## Scope Freeze Gate
|
||||
|
||||
V4.6 is sufficiently defined to begin implementation:
|
||||
|
||||
- Submission-time versus execution-time provenance ownership is resolved.
|
||||
- UI, workflow, service, and transaction boundaries are resolved.
|
||||
- Claiming, attempt persistence, and first-selection concurrency rules are resolved.
|
||||
- Page-level concurrency scope and default bound are resolved.
|
||||
- Source extraction seams and compatibility policy are resolved.
|
||||
- Media validation, artifact compensation, and query-projection expectations are resolved.
|
||||
- Successful Job status terminology is resolved.
|
||||
- V4.2 and V4.5 evidence, candidate, and revision contracts remain unchanged.
|
||||
|
||||
Any expansion into distributed work scheduling, additional providers or media types, evidence-model replacement,
|
||||
candidate-policy changes, or incompatible API/schema redesign requires an explicit V4.6 scope amendment or V5.
|
||||
|
||||
## Related Local References
|
||||
|
||||
- [V4.6 Implementation Plan](implementation_plan_v4_6.md)
|
||||
- [V4.5 Scope Boundary](../ver4.5/scope_boundary_v4_5.md)
|
||||
- [V4.5 Implementation Plan](../ver4.5/implementation_plan_v4_5.md)
|
||||
- [V4 Architecture](../ver4/architecture_v4.md)
|
||||
- [V4 Schema](../ver4/schema_v4.md)
|
||||
- [V4 Requirements](../ver4/requirements_v4.md)
|
||||
- [V4 Error Handling Policy](../ver4/error_handling_v4.md)
|
||||
- [AI Evidence and Provenance Invariant](../invariant/ai_evidence_and_provenance.md)
|
||||
- [Error Handling Invariant](../invariant/error_handling.md)
|
||||
Reference in New Issue
Block a user