generated from john/python-template
Hide V4.6 recommendations after code review
This commit is contained in:
zoltan57
@@ -1,338 +0,0 @@
|
||||
# Implementation Plan (Version 4.6)
|
||||
|
||||
## Goal
|
||||
|
||||
Bring the completed V4.5 implementation into conformance with the documented V4 service, provenance, transaction,
|
||||
worker-concurrency, and evidence boundaries while preserving all historical Source and execution behavior.
|
||||
|
||||
## Planning Status
|
||||
|
||||
- V4.5 is the completed implementation baseline.
|
||||
- The V4.6 scope is frozen and sufficiently detailed to begin implementation.
|
||||
- V4.6 is a compatibility-preserving architecture and reliability revision, not a V5 redesign.
|
||||
- Scope additions require an explicit amendment or a later revision.
|
||||
|
||||
## Planning Constraints
|
||||
|
||||
- Original Source bytes and historical evidence remain immutable.
|
||||
- Existing JobSource compatibility projections remain supported.
|
||||
- Human revisions remain independent and retain display and print precedence.
|
||||
- Candidate promotion semantics remain unchanged.
|
||||
- Provider network work occurs outside database transactions.
|
||||
- Concurrent page tasks never share an AsyncSession.
|
||||
- UI pages do not own persistence transactions.
|
||||
- Refactoring proceeds behind stable public service contracts.
|
||||
- New shared code has a cohesive domain or infrastructure owner; no generic helper dumping grounds are introduced.
|
||||
- Database, integration, and UI tests use confirmed isolated data and never modify `data/transcription.db`.
|
||||
- Potentially destructive tests run only through `tools/run_destructive_tests.py`.
|
||||
|
||||
## Expected Project Impact
|
||||
|
||||
| Area | Expected impact |
|
||||
| --- | --- |
|
||||
| Job submission | Add one typed frozen request specification used by normal and retranscription workflows. |
|
||||
| Upload workflow | Move transaction and compensation ownership from the UI and direct persistence functions into orchestration. |
|
||||
| Source media | Validate content and extension consistently while preserving original bytes. |
|
||||
| Job service | Add atomic claim behavior and stable lifecycle transitions. |
|
||||
| Provider adapters | Remove mutable per-call evidence state and carry evidence through results and errors. |
|
||||
| Worker workflows | Add bounded concurrent page execution with isolated sessions and durable page outcomes. |
|
||||
| Source service | Extract cohesive collaborators behind the existing SourceService facade. |
|
||||
| Evidence persistence | Make attempt numbering, first selection, artifact creation, and cleanup conflict-safe. |
|
||||
| Read paths | Add lightweight Source and attempt projections that defer large evidence fields. |
|
||||
| Compatibility | Isolate legacy transcription and storage aliases without adding new behavior. |
|
||||
| Documentation | Align status terminology, ownership, concurrency, and provenance descriptions. |
|
||||
| Tests | Add failure-injection, concurrency, query-shape, compatibility, and preservation coverage. |
|
||||
|
||||
## Implementation Phases
|
||||
|
||||
### 1. Add Characterization and Failure-Injection Coverage
|
||||
|
||||
- Record the current externally visible Source, Job, JobSource, attempt, artifact, revision, candidate, and export
|
||||
behavior before extraction.
|
||||
- Add tests proving:
|
||||
- Historical Jobs with missing request fields remain readable.
|
||||
- JobSource SDK snapshots retain their existing evidence label.
|
||||
- Source preferred-machine fields and human revisions survive failed processing.
|
||||
- Per-page outcomes remain durable before sibling pages finish.
|
||||
- Candidate promotion updates both Source projection fields atomically.
|
||||
- Add reusable test doubles for:
|
||||
- Provider success, response failure, connection failure, timeout, and cancellation.
|
||||
- Database flush and commit failure at named workflow phases.
|
||||
- Filesystem write, replace, and unlink failure.
|
||||
- Controlled overlapping page calls.
|
||||
- Keep all providers fake and all databases and filesystem roots isolated.
|
||||
- Use these tests as behavior gates for every later extraction.
|
||||
|
||||
### 2. Define the Frozen Job Submission Specification
|
||||
|
||||
- Move runtime prompt loading, prompt hashing, requested provider/model resolution, and explicit parameter-state capture
|
||||
into a focused prompt/job-submission module.
|
||||
- Define one immutable typed value containing:
|
||||
- Requested provider.
|
||||
- Requested model.
|
||||
- Prompt name and SHA-256.
|
||||
- System and user prompt content.
|
||||
- Temperature and top-p values.
|
||||
- Explicitly supplied versus omitted state where applicable.
|
||||
- Reuse the constrained PromptStore's path policy rather than maintaining a second incompatible prompt-path policy.
|
||||
- Keep prompt editing, backup, and recovery behavior in the existing prompt-storage owner.
|
||||
- Validate model selection against `provider_models` for both normal transcription and retranscription.
|
||||
- Populate all new Job request fields before adding the Job to the session.
|
||||
- Stop mutating Job provider and model during attempt persistence.
|
||||
- Store provider-reported resolved identity on ExecutionAttempt only.
|
||||
- Retain compatibility reads for historical Jobs whose request fields are null.
|
||||
- Add tests for pre-queue completeness, allowlist rejection, prompt-edit isolation, requested/resolved model differences, and
|
||||
historical null behavior.
|
||||
|
||||
### 3. Establish Content-Aware Source Media Policy
|
||||
|
||||
- Move accepted Source extensions, canonical MIME mapping, validation, and managed payload loading into one Source-media
|
||||
module.
|
||||
- Validate filename and non-empty content first.
|
||||
- Validate JPEG, PNG, and TIFF using bounded raster decoding appropriate to the existing Pillow dependency.
|
||||
- Validate PDF structure without rendering or modifying the file.
|
||||
- Reject a supported extension whose content is corrupt or belongs to a different supported type.
|
||||
- Return a typed validated-media result containing canonical media type, digest, byte size, and original bytes or a
|
||||
stable staged reference.
|
||||
- Ensure hashing uses the exact bytes later persisted as the original Source.
|
||||
- Keep orientation normalization separate and downstream of original-media validation.
|
||||
- Preserve compatibility aliases for existing MIME and payload-loading imports.
|
||||
- Add fixtures for valid, corrupt, truncated, mismatched, uppercase-extension, empty, and unsupported content.
|
||||
|
||||
### 4. Replace Direct Upload Persistence with an Ingestion Workflow
|
||||
|
||||
- Introduce a Source-ingestion workflow that accepts validated inputs and service dependencies.
|
||||
- Generate Document and Source identifiers before file placement so managed relative paths remain deterministic.
|
||||
- Stage all Source files beneath the configured application root.
|
||||
- Build the frozen Job submission specification before starting database mutation.
|
||||
- Open one workflow-owned transaction.
|
||||
- Use DocumentService, JobService, and SourceService session-aware writes to create:
|
||||
- The Document when required.
|
||||
- The queued Job.
|
||||
- Ordered Sources.
|
||||
- Pending JobSource projections.
|
||||
- Commit once after every record has flushed successfully.
|
||||
- Delete files staged by the operation on validation, flush, or commit failure.
|
||||
- Return a typed result only after commit succeeds.
|
||||
- Move worker notification after successful workflow return.
|
||||
- Remove session-scope imports and direct persistence ownership from the Jobs UI.
|
||||
- Keep old storage entry points as thin compatibility adapters that invoke the workflow.
|
||||
- Add tests for deterministic ordering, page numbering, duplicate names, every failure phase, cleanup, commit-before-notify,
|
||||
and no direct UI persistence.
|
||||
|
||||
### 5. Add Atomic Job Claiming and Canonical Lifecycle Semantics
|
||||
|
||||
- Add `claim_next_queued_job` to JobService or a Job-owned persistence collaborator.
|
||||
- Select candidates deterministically by creation time and UUID.
|
||||
- Transition one candidate using a conditional write that succeeds only while status is queued.
|
||||
- Retry candidate selection when another claimant wins, without issuing provider work for the lost claim.
|
||||
- Commit the processing transition before provider-input preparation.
|
||||
- Route worker processing exclusively through the claim operation.
|
||||
- Preserve stale-processing recovery and require recovered Jobs to be claimed again.
|
||||
- Define `transcribed` as the newly emitted all-success terminal status.
|
||||
- Continue reading `completed` as a historical terminal status.
|
||||
- Add tests with concurrent claimers on SQLite and the portable SQL path used for PostgreSQL.
|
||||
- Add a lifecycle transition-table test covering queued, processing, transcribed, partial success, failed, cancellation,
|
||||
resubmission, and stale recovery.
|
||||
|
||||
### 6. Make Provider Calls Stateless
|
||||
|
||||
- Extend provider success and error contracts so each call directly carries:
|
||||
- RequestManifest.
|
||||
- TransportEvidence or explicit no-response evidence.
|
||||
- Failure phase.
|
||||
- Parsed SDK snapshot when available.
|
||||
- Normalized metadata when available.
|
||||
- Remove `current_request_manifest` and `current_transport_evidence` from provider adapters and worker persistence.
|
||||
- Ensure local timeout and cancellation retain the call-specific request manifest without reading mutable adapter state.
|
||||
- Keep response-body capture and safe-header filtering in the OpenRouter adapter.
|
||||
- Preserve the distinction between exact OpenRouter-boundary response bytes, SDK snapshot, and normalized fields.
|
||||
- Decide adapter reuse by concurrency safety; create per-call adapters if the SDK client cannot safely support concurrent
|
||||
calls.
|
||||
- Add overlapping-call tests in which success, HTTP failure, timeout, and no-response failure complete in different
|
||||
orders.
|
||||
- Assert that no call receives another call's source digest, prompt hash, response body, or identifiers.
|
||||
|
||||
### 7. Extract Source-Owned Evidence and Artifact Collaborators
|
||||
|
||||
- Keep SourceService as the public facade while moving cohesive implementation behind it.
|
||||
- Extract Source persistence and query behavior first without changing method contracts.
|
||||
- Extract ExecutionAttempt and JobSource compatibility persistence into a Source-owned evidence writer.
|
||||
- Extract processing-artifact file, integrity, and metadata behavior into a Source-owned artifact store.
|
||||
- Extract evidence export into a read-only projection builder.
|
||||
- Extract provider-input preparation so it coordinates:
|
||||
- Managed original Source access.
|
||||
- Orientation normalization.
|
||||
- Derivative artifact creation.
|
||||
- Exact SourceEvidenceReference construction.
|
||||
- Keep candidate promotion in Source persistence.
|
||||
- Keep normalization and quality analysis in their existing focused modules.
|
||||
- Make dependencies explicit through typed constructor parameters rather than importing service globals.
|
||||
- Do not allow extracted collaborators to call unrelated aggregate services.
|
||||
- Add contract tests showing SourceService behavior is unchanged while collaborators can be tested independently.
|
||||
|
||||
### 8. Make Artifact Creation and Cleanup Explicit
|
||||
|
||||
- Define one artifact-creation operation that owns temporary path creation, atomic file replacement, database metadata,
|
||||
and compensation.
|
||||
- Track every path created before commit within the owning workflow.
|
||||
- On flush or commit failure, remove newly created content and surface cleanup failure separately when removal also
|
||||
fails.
|
||||
- Constrain temporary and final paths beneath the artifact root and preserve safe relative references.
|
||||
- Add startup or explicit maintenance cleanup for abandoned temporary files without deleting final referenced
|
||||
artifacts.
|
||||
- For deletion:
|
||||
- Verify retention policy and references before mutation.
|
||||
- Commit metadata changes according to the explicit deletion contract.
|
||||
- Attempt external cleanup.
|
||||
- Report incomplete cleanup with correlation identity when unlink fails.
|
||||
- Do not silently skip unsafe references; classify them as validation or conflict failures as appropriate.
|
||||
- Add tests for write, replace, flush, commit, unlink, cancellation, missing file, digest mismatch, and unsafe reference
|
||||
failures.
|
||||
|
||||
### 9. Make Attempt Persistence and First Selection Conflict-Safe
|
||||
|
||||
- Stop relying solely on an unprotected `max(attempt_number) + 1` calculation.
|
||||
- Allocate or insert attempt numbers inside the protected Job processing boundary.
|
||||
- Treat the database uniqueness constraint as a final integrity guard and translate conflicts deterministically.
|
||||
- Ensure every completed provider call either persists one immutable attempt or leaves a visible persistence failure
|
||||
that does not masquerade as provider success.
|
||||
- Perform first-success Source selection with a conditional update requiring both preferred provenance and machine
|
||||
projection to remain unset.
|
||||
- When first-selection loses a race, retain the new successful attempt as a candidate.
|
||||
- Keep explicit candidate promotion as a separate atomic command.
|
||||
- Never modify earlier attempt text, transport evidence, warnings, or artifact associations.
|
||||
- Add concurrency tests for duplicate attempt allocation, simultaneous successes, success/failure overlap, candidate
|
||||
promotion, and transaction rollback.
|
||||
|
||||
### 10. Add Bounded Concurrent Page Processing
|
||||
|
||||
- Add `worker_concurrency` to Settings with default `2`, positive validation, and a documented safe upper bound.
|
||||
- Refactor the single-page body of `process_queued_job` into one typed page-attempt operation.
|
||||
- Prepare each page's provider input and persist any derivative using its own session and committed preparation phase.
|
||||
- Release all database transactions before awaiting provider network work.
|
||||
- Use an asyncio semaphore or TaskGroup-based worker pattern to cap in-flight page calls.
|
||||
- Give every page outcome persistence operation its own session and transaction.
|
||||
- Shield only the minimum durable outcome write required to preserve a completed provider call during cancellation.
|
||||
- Collect typed success, failure, and cancellation outcomes without sharing mutable lists across tasks.
|
||||
- Compute the aggregate status after all started tasks reach durable outcomes.
|
||||
- Stop scheduling unstarted pages after cancellation or an externally terminal Job transition.
|
||||
- Preserve deterministic source ordering for scheduling and deterministic aggregate reporting independent of completion
|
||||
order.
|
||||
- Add tests for concurrency bounds `1`, `2`, and a larger configured value; out-of-order completion; mixed outcomes;
|
||||
timeout; cancellation; external stop; durable early completion; and no shared session.
|
||||
|
||||
### 11. Introduce Purpose-Specific Read Projections
|
||||
|
||||
- Add a Source-list projection containing only fields needed by the Sources table.
|
||||
- Filter Document and Job scopes in SQL.
|
||||
- Add a Source-detail projection or explicit eager-loading contract for the relationships rendered by the page.
|
||||
- Add an attempt-summary projection containing candidate metadata, transcription text or an explicit preview policy,
|
||||
warning summary, and preferred-selection identity.
|
||||
- Defer transport body, SDK snapshot, request manifest, and other large evidence values from summaries.
|
||||
- Keep one explicit full-attempt evidence read for detailed inspection.
|
||||
- Keep evidence export complete and integrity-verified.
|
||||
- Replace all-sibling materialization in Source navigation with predecessor/successor queries using page number and UUID.
|
||||
- Add query-count and unloaded-attribute assertions so future changes cannot reintroduce accidental evidence loading.
|
||||
- Preserve deterministic ordering and avoid lazy database access from UI components.
|
||||
|
||||
### 12. Isolate Compatibility Adapters and Cohesive Shared Policies
|
||||
|
||||
- Keep `services/transcription.py` as a documented compatibility facade with imports and aliases only.
|
||||
- Keep legacy upload/storage aliases as thin calls into the new ingestion and media owners.
|
||||
- Mark compatibility entry points in tests so removal can be evaluated separately.
|
||||
- Consolidate Document Type and Person Role label normalization in a registry-domain policy module.
|
||||
- Share canonical evidence digest primitives only where serialization and digest meaning are identical.
|
||||
- Do not merge prompt backup writing, original Source storage, and immutable artifact storage into one generic helper;
|
||||
their lifecycle and recovery semantics remain domain-owned.
|
||||
- Remove unused service-base state only after confirming no external caller depends on it.
|
||||
- Add import-compatibility tests and direct tests for each extracted policy.
|
||||
|
||||
### 13. Align UI and API Boundaries
|
||||
|
||||
- Remove machine-local stored paths from routine Source Detail presentation.
|
||||
- Keep stored filename and evidence-safe identifiers available.
|
||||
- Ensure normal Job creation uses the configured provider and allowlisted model selector rather than arbitrary text input.
|
||||
- Keep retranscription Source and Document context locked.
|
||||
- Keep evidence export explicit and avoid loading export-only content during page render.
|
||||
- Continue translating AppError through established UI and API presenters without parsing error messages.
|
||||
- Keep sessions, session factories beyond dependency construction, and persistence operations out of reusable components.
|
||||
- Add UI tests for allowlisted selection, no path disclosure, unchanged candidate actions, and error preservation.
|
||||
|
||||
### 14. Align Authoritative Documentation
|
||||
|
||||
- Update V4 architecture for:
|
||||
- Frozen Job request versus resolved attempt identity.
|
||||
- Atomic Job claims.
|
||||
- Bounded page concurrency and isolated page sessions.
|
||||
- Source facade and internal collaborators.
|
||||
- Explicit artifact lifecycle ownership.
|
||||
- Update V4 schema documentation only for actual additive schema changes.
|
||||
- Update V4 requirements and verification mapping for atomic claims, media validation, and concurrency tests.
|
||||
- Update V4 error handling for filesystem/database partial cleanup.
|
||||
- Define `transcribed` as the current all-success terminal status and `completed` as historical compatibility.
|
||||
- Correct stale V3 module descriptions and UI documentation that exposes or promises machine-local paths.
|
||||
- Update documentation together with the behavior it describes; do not prestate unimplemented behavior as complete.
|
||||
|
||||
### 15. Verification and Preservation Audit
|
||||
|
||||
- Run focused pure tests for submission specs, media validation, provider contracts, query projections, and policies.
|
||||
- Run isolated service tests for claims, attempts, artifacts, ingestion, candidate selection, and compatibility.
|
||||
- Run worker tests with fake providers and controlled concurrent completion.
|
||||
- Run UI and API tests against isolated application state.
|
||||
- Run schema tests only through the required destructive-test wrapper when applicable.
|
||||
- Run the broader non-external regression suite after focused targets pass.
|
||||
- Verify that tests did not change:
|
||||
- `data/transcription.db`.
|
||||
- Curated Source files.
|
||||
- Existing artifact files.
|
||||
- Prompt files outside temporary fixtures.
|
||||
- Perform an explicit preservation audit showing that upgrading does not rewrite existing Job, Source, JobSource,
|
||||
ExecutionAttempt, ProcessingArtifact, or revision content.
|
||||
|
||||
## Delivery Order
|
||||
|
||||
1. Characterization and failure-injection tests.
|
||||
2. Frozen submission specification and prompt ownership.
|
||||
3. Content-aware Source media validation.
|
||||
4. Ingestion workflow and UI transaction-boundary correction.
|
||||
5. Atomic Job claiming and lifecycle terminology.
|
||||
6. Stateless provider evidence.
|
||||
7. Source evidence/artifact/provider-input extraction behind SourceService.
|
||||
8. Artifact compensation and conflict-safe attempt/selection persistence.
|
||||
9. Purpose-specific read projections.
|
||||
10. Bounded concurrent page processing.
|
||||
11. Compatibility isolation, UI/API alignment, and documentation.
|
||||
12. Full preservation and regression verification.
|
||||
|
||||
## Done Criteria
|
||||
|
||||
- All V4.6 acceptance criteria are implemented and testable.
|
||||
- New Jobs contain complete request provenance before queueing and are not rewritten by execution.
|
||||
- Every submitted model is allowlisted and requested/resolved identities remain distinct.
|
||||
- Upload UI code owns no database transaction.
|
||||
- Source content validation verifies supported media without modifying originals.
|
||||
- One queued Job can be claimed only once.
|
||||
- Provider evidence is call-local and safe under overlapping execution.
|
||||
- Page calls run concurrently within the configured bound and never share sessions.
|
||||
- Every actual provider call has one immutable attempt.
|
||||
- Concurrent first successes preserve one preferred result and every other success as a candidate.
|
||||
- Artifact commit and cleanup failures are visible and compensated where safely possible.
|
||||
- Source list and candidate views avoid loading full evidence bodies.
|
||||
- SourceService remains compatible while extracted responsibilities are independently testable.
|
||||
- Human revisions, preferred-machine projections, prior attempts, JobSource snapshots, and artifacts retain their original
|
||||
meaning.
|
||||
- The worker consistently emits transcribed, partial_success, or failed for new processing.
|
||||
- Verification uses isolated data, fake providers, and does not modify operator or curated evidence data.
|
||||
|
||||
## Related Local References
|
||||
|
||||
- [V4.6 Scope Boundary](scope_boundary_v4_6.md)
|
||||
- [V4.5 Scope Boundary](../ver4.5/scope_boundary_v4_5.md)
|
||||
- [V4.5 Implementation Plan](../ver4.5/implementation_plan_v4_5.md)
|
||||
- [V4 Architecture](../ver4/architecture_v4.md)
|
||||
- [V4 Schema](../ver4/schema_v4.md)
|
||||
- [V4 Requirements](../ver4/requirements_v4.md)
|
||||
- [V4 Error Handling Policy](../ver4/error_handling_v4.md)
|
||||
- [AI Evidence and Provenance Invariant](../invariant/ai_evidence_and_provenance.md)
|
||||
- [Error Handling Invariant](../invariant/error_handling.md)
|
||||
Reference in New Issue
Block a user