transcription/docs/ver4.5/implementation_plan_v4_5.md

Implementation Plan (Version 4.5)

Goal

Normalize metadata-directed image orientation for provider input, improve transcription-medium instructions and deterministic quality warnings, and support user-initiated single-Source retranscription with approved alternate models and explicit candidate promotion.

Planning Status

• V4.4 is the completed implementation baseline.
• The V4.5 scope is frozen and sufficiently detailed to begin implementation.
• Scope additions require an explicit amendment or a later revision.

Planning Constraints

• Original uploaded Source files remain immutable.
• Provider input must remain traceable to the original Source and any normalized derivative.
• Normalization is limited to recognized orientation metadata; no enhancement pipeline is introduced.
• Quality warnings never mutate transcription text or trigger paid requests automatically.
• Retranscription creates new immutable Job and execution evidence.
• A retranscription result remains a candidate until explicitly promoted.
• Human revision remains separate from and takes precedence over machine selection.
• Provider work occurs outside database transactions.
• Database, integration, and UI tests use confirmed isolated data and never modify data/transcription.db.
• Potentially destructive tests run only through tools/run_destructive_tests.py.

Expected Project Impact

Area	Expected impact
Configuration	Add a validated provider-model allowlist while retaining one default model.
Image processing	Add metadata-directed orientation normalization and model-input artifact creation.
Evidence model	Link each request to the exact model-input artifact and record preferred machine-attempt provenance.
Prompt contract	Add explicit body-medium classification and structured-layout rules.
Quality service	Add deterministic, non-mutating warnings for known output defects.
Job creation	Support a Source-locked retranscription Job and approved model selection.
Worker workflows	Preserve candidates without automatically replacing preferred machine text.
Source Detail	Add Retranscribe Source, candidate summaries, comparison, promotion, and normalization indicators.
Tests and documentation	Add isolated normalization, configuration, warning, retranscription, promotion, and UI coverage.

Implementation Phases

1. Align Configuration Contracts

• Add provider_models as an immutable validated collection in Settings.
• Parse PROVIDER_MODELS using the standard Pydantic-settings JSON representation.
• Preserve PROVIDER_MODEL as the default.
• If the allowlist is omitted, derive a one-entry list from the default.
• Normalize whitespace, reject empty values, and deduplicate while preserving the relative order of non-default values.
• Ensure the default appears exactly once and first in the effective selector order.
• Validate a submitted model against the allowlist in the service or workflow boundary, not only in the UI.
• Document .env.example behavior without adding real credentials.
• Add configuration tests for omitted, valid, duplicate, malformed, and empty model lists.

2. Define Orientation-Normalized Artifacts

• Reuse ProcessingArtifact for the model-input derivative and transformation metadata.
• Define a versioned orientation-normalization artifact schema containing:
  • Original Source identity and digest.
  • Original orientation value.
  • Applied rotation.
  • Original and derivative dimensions, media types, byte sizes, and digests.
  • Processor name and version.
• Add one domain-owned orientation-normalization service or adapter; keep image-library details out of UI and provider modules.
• Apply recognized metadata orientation physically to raster pixels.
• Reset or remove orientation metadata on the derivative.
• Store derivatives under application-managed artifact storage with safe relative references.
• Avoid creating a derivative when no supported transformation is required.
• Return a typed provider-input reference that identifies whether the request uses the original or a derivative.
• Never mutate or delete the original Source as part of normalization.

3. Integrate Normalization with Provider Input

• Resolve the exact model input before constructing the request manifest.
• Use the normalized derivative when orientation metadata requires it; otherwise use the original Source.
• Extend the existing SourceEvidenceReference or associated artifact reference so the manifest identifies:
  • Original Source.
  • Derivative artifact when present.
  • Transformation schema and digest.
• Ensure normalization completes and persists before provider network work begins.
• Bind the immutable model-input artifact reference to the exact ExecutionAttempt that consumed it before terminal attempt persistence completes.
• If normalized bytes are reused, retain attempt-specific association while preserving one content identity and digest.
• If normalization fails, do not send a provider request.
• Preserve safe failure evidence and an actionable error category.
• Confirm provider payload loading and evidence hashing read the same resolved bytes.

4. Revise the Transcription Prompt

• Align the prompt with the durable medium rules in docs/invariant/transcription_methodology.md.
• Add the four frozen document-body markers.
• Define operational differences among handwritten, typewritten, typeset, and mixed content.
• State that mechanical typewriter variation is not handwriting.
• Require exactly one body marker.
• Prohibit repeated whole-line handwriting wrappers after a whole-body handwritten marker.
• Permit localized handwriting markers only for actual annotations, signatures, or mixed-body portions.
• Add layout instructions for tables of contents, tables, forms, columns, captions, marginalia, page numbers, dotted leaders, and associated references.
• Require plain-text characters rather than HTML entities.
• Retain verbatim, uncertainty, damage, deletion, insertion, and line-break-hyphenation rules.
• Update prompt fixtures and prompt-hash expectations without rewriting historical Job prompt evidence.

5. Add Deterministic Quality Analysis

• Introduce a small typed warning model with stable warning codes and human-readable detail.
• Analyze successful output without modifying it.
• Implement warnings for:
  • Unicode replacement characters.
  • Multiple document-body markers.
  • Whole-body handwritten plus repeated line-level handwriting wrappers.
  • Likely unresolved HTML entities.
• Keep warning rules deterministic and provider-independent.
• Persist warnings once as an immutable, versioned ProcessingArtifact tied to the successful ExecutionAttempt.
• Source Detail renders stored warnings and never recomputes historical attempts under newer warning rules.
• Make warning analysis idempotent and versioned so newer rules apply only to newly analyzed attempts unless a separate future reanalysis workflow is introduced.
• Do not implement confidence scoring or automatic retries.

6. Model Retranscription and Selection State

• Add a durable Job purpose or equivalent discriminator for normal transcription versus Source retranscription.
• Ensure a retranscription Job contains exactly one JobSource for the locked Source.
• Add durable preferred-machine-attempt provenance for each Source.
• Retain Source.raw_transcription as the preferred-machine-text projection for compatibility.
• Define legacy behavior for Sources whose current projection predates ExecutionAttempt provenance.
• On the first successful result with no preferred machine output, select the successful attempt automatically regardless of Job purpose.
• Once preferred provenance exists, preserve every later successful result as an unselected candidate regardless of Job purpose.
• Prevent normal and retranscription workflows from writing Source.raw_transcription directly when preferred provenance already exists.
• Add a promotion command that:
  • Loads the Source and successful ExecutionAttempt.
  • Verifies ownership and successful text.
  • Updates preferred-attempt provenance and Source.raw_transcription in one transaction.
  • Leaves Source.revised_text and all attempts unchanged.
• Reject failed, unrelated, missing, or textless candidates deterministically.

7. Add the Retranscribe Source Workflow

• Add a Source Detail Retranscribe Source action.
• Navigate to Create Processing Job with an explicit source_id query parameter.
• Load the Source and derive its Document server-side.
• Render Source identity and filename as locked context.
• Render Provider from Settings as read-only.
• Render Model as a selector using the effective allowlist and default.
• Reuse the frozen default prompt unless a later scope addition explicitly allows prompt selection.
• Create a new queued retranscription Job and one pending JobSource atomically.
• Notify the worker only after the transaction commits.
• Preserve current preferred machine text and human revision throughout queueing, processing, success, and failure.
• Return to the new Job Detail after successful creation.

8. Adapt Worker Success Semantics

• Select the first successful result automatically for Sources without preferred machine output, including a successful retranscription after earlier failures.
• For every later success, persist JobSource and ExecutionAttempt text without replacing the Source projection, regardless of normal or retranscription Job purpose.
• Run deterministic quality analysis after successful normalization of provider output.
• Persist candidate warnings with the attempt.
• Keep all terminal Job status and execution evidence updates atomic according to the existing workflow boundary.
• Ensure a failed retranscription cannot clear or change preferred machine text.

9. Build Candidate Review and Promotion UI

• Extend Source Detail with concise machine-output sections:
  • Preferred machine transcription.
  • Human revision.
  • Candidate machine transcriptions.
• List candidates with creation date, provider, model, Job ID, status, and warning indicator.
• Default to a compact candidate list; do not render every full transcript simultaneously.
• When no successful machine result exists, render an explicit empty state without comparison controls.
• When preferred output exists without candidates, render an explicit no-candidates state.
• Allow one candidate to be opened for comparison with the preferred machine transcription.
• Label both sides with provider, model, Job ID, and date.
• Add Use this transcription only for a successful unselected candidate.
• Require explicit confirmation before promotion.
• After promotion, refresh Source Detail and retain the former preferred result in attempt history.
• If a human revision exists, explain that promotion changes machine selection but not the human-preferred displayed/printed text.
• When orientation normalization occurred, show a compact indicator and link to transformation evidence; do not require routine side-by-side image display.

10. Align API and Service Contracts

• Keep arbitrary provider and model identifiers out of public write contracts.
• If an API is added for retranscription, accept Source UUID and one configured model identifier and validate both server-side.
• If an API is added for promotion, accept Source UUID and ExecutionAttempt UUID and validate their relationship.
• Return stable validation, conflict, not-found, provider, and persistence errors through the existing taxonomy.
• Keep filesystem paths, raw credentials, and unrestricted artifact references out of responses.

11. Verification

• Add pure unit tests for:
  • Orientation metadata interpretation.
  • No-op versus transformed input selection.
  • Derivative metadata and hashing.
  • Model allowlist normalization.
  • Prompt marker rules.
  • Every quality-warning code.
• Add isolated service and workflow tests for:
  • Original Source immutability.
  • Normalization failure before provider invocation.
  • Request manifests referencing exact provider-input bytes.
  • Single-Source retranscription Job creation.
  • Candidate preservation.
  • Initial automatic selection.
  • Explicit candidate promotion.
  • Atomic rollback on invalid promotion.
  • Human revision preservation.
• Add UI tests for:
  • Retranscribe Source navigation.
  • Locked Source and Document context.
  • Read-only Provider and allowlisted Model selector.
  • Candidate list, warnings, comparison, confirmation, and promotion.
  • Normalization indicator without mandatory image comparison.
• Use fixed image fixtures with known EXIF orientation and hashes.
• Use fake providers only; no focused or regression test sends an external provider request.
• Confirm all database, integration, and UI targets use isolated databases.
• Run potentially destructive schema tests only through tools/run_destructive_tests.py.
• Run the broader non-external regression suite after focused coverage passes.
• Verify that data/transcription.db and curated Source files were not changed by test execution.

12. Align Authoritative Documentation

• Update V4 architecture, requirements, and schema for:
  • Original versus model-input artifacts.
  • Preferred machine-attempt provenance.
  • Retranscription Job purpose.
  • Candidate and promotion semantics.
  • Quality-warning evidence.
• Update Source and Job UI contracts after implementation behavior is accepted.
• Update the transcription methodology and evidence invariant only where V4.5 establishes a durable cross-version rule.
• Keep historical prompt and execution evidence immutable.

Delivery Order

1. Configuration and model allowlist.
2. Orientation artifact schema and normalization adapter.
3. Provider-input and evidence integration.
4. Prompt revision and deterministic warning analysis.
5. Retranscription and preferred-attempt persistence.
6. Worker candidate semantics.
7. Retranscription creation UI.
8. Candidate comparison and promotion UI.
9. Authoritative documentation and regression verification.

Done Criteria

• All frozen V4.5 acceptance criteria are implemented and testable.
• EXIF-oriented images are physically upright for provider processing while originals remain byte-for-byte unchanged.
• Every provider request identifies its exact original and normalized inputs.
• The prompt classifies body medium consistently and avoids redundant handwriting wrappers.
• Quality defects produce deterministic warnings without silent rewriting or automatic cost.
• Source Detail can create one-Source retranscription Jobs using configured alternate models.
• Retranscription results remain candidates until explicitly promoted.
• Promotion updates exact preferred-attempt provenance and the compatibility projection atomically.
• Human revisions remain unchanged and retain display/print precedence.
• Previous Jobs and attempts remain immutable and inspectable.
• No manual image editor, visual orientation inference, confidence percentage, automatic retry, or arbitrary model entry is introduced.
• Verification uses isolated data, fake providers, and does not modify data/transcription.db or curated Source files.

Related Local References

• V4.5 Scope Boundary
• V4.4 Scope Boundary
• V4.4 Implementation Plan
• V4.2 Evidence and Provenance Scope
• V4 Architecture
• V4 Schema
• V4 Requirements
• V4 Error Handling Policy
• Transcription Methodology
• AI Evidence and Provenance Invariant