transcription/docs/ver4.2/implementation_plan_v4_2.md

Implementation Plan (Version 4.2)

Goal

Make processing evidence precise, append-only, secret-safe, and exportable while preserving every existing record and creating a provider-neutral home for future OCR/layout artifacts.

Implementation Principles

• Implement the Digital Evidence and AI Processing Provenance invariant, not a provider-specific approximation of it.
• Capture transport evidence before SDK parsing.
• Keep exact evidence separate from parsed and normalized representations.
• Prefer additive schema evolution and explicit compatibility behavior.
• Reference source content by digest rather than duplicating it in request JSON.
• Use allowlists for safe metadata capture.
• Keep persistence and evidence semantics behind service boundaries.
• Do not change the default model until a representative benchmark supports that decision.

Current-State Gaps

Current behavior	Gap to close
Source stores original file path, digest, and size.	Media type and image/page geometry used for an execution are not frozen with that execution.
Job stores prompt text/hash, requested model after resolution, temperature, and top_p.	The complete effective request structure, omitted-versus-explicit parameter state, routing constraints, and software versions are not frozen.
JobSource.raw_api_response stores model_dump() output from the OpenRouter SDK.	The exact HTTP body can be normalized by OpenRouter and filtered again by the SDK before persistence.
JobSource.ai_metadata stores finish reason and basic token counts.	Detailed accounting remains only in the SDK snapshot and is not a substitute for exact evidence.
Provider exceptions become application errors.	Safe HTTP error bodies, statuses, headers, and no-response distinctions are not persisted.
Worker logs elapsed time.	Execution duration is not stored on JobSource.
Source Detail displays AI metadata and the SDK snapshot.	The UI does not identify evidence layers or expose request/transport/software provenance.
No generic processing-artifact model exists.	Future OCR geometry would require ad hoc provider fields or an unrelated schema.

Expected Project Impact

Area	Expected impact
Database models and upgrades	Add execution-specification, transport-evidence, timing, software-context, and generic artifact storage without removing existing columns.
OpenRouter adapter	Introduce a transport boundary that can capture exact body/status/safe headers before typed SDK parsing, or use supported SDK hooks that expose the unparsed response reliably.
Provider contract	Return structured evidence for success and failure without leaking provider-specific transport concerns into workflow orchestration.
Source and workflow services	Persist one append-only execution outcome and its artifacts transactionally; retain compatibility projections.
UI	Label and inspect evidence layers; export safe evidence packages through service operations.
Benchmarking	Add a private manifest and repeatable evaluator using the literal-transcription methodology.
Tests and documentation	Add compatibility, capture, security, integrity, export, and benchmark-scoring coverage; correct overstated V4 evidence language.

Proposed Data Design

Exact names should be confirmed against existing conventions before migration code is written. The design should provide the following logical records.

1. Execution Evidence

Extend JobSource or associate it one-to-one with a new execution-evidence record containing:

• Request manifest JSON and manifest schema version.
• Transport status, body bytes or exact decoded body plus encoding/content type, and safe headers.
• Parsed SDK snapshot retained separately from transport content.
• Application, adapter, SDK, and runtime version metadata.
• Start, finish, and duration values.
• Router/provider request and generation identifiers when available.
• Failure phase and whether an HTTP response was received.

The implementation should evaluate a companion table rather than continuing to widen JobSource. A companion record better isolates large/optional evidence and permits clear one-to-one compatibility semantics.

2. Generic Processing Artifact

Add a one-to-many artifact model associated with a source and, when applicable, a producing execution:

• Stable artifact UUID.
• source_id and optional execution/job_source_id.
• Semantic artifact type.
• Media/serialization format.
• Schema name and version.
• Producer and producer version.
• Inline JSON payload or external location.
• Payload digest and byte size.
• Coordinate-system metadata when relevant.
• Creation timestamp.

Enforce exactly one content location: inline payload or external reference. An external artifact must be written durably and hashed before its database record commits.

3. Compatibility Projections

• Keep JobSource.raw_api_response unchanged for existing and new compatibility reads until a later deprecation decision.
• Keep JobSource.ai_metadata for indexed/display-ready normalized values.
• Keep Source.raw_transcription as the latest successful machine-output projection while treating per-execution JobSource.raw_transcription as history.
• Document that older rows have an SDK snapshot but no exact transport capture.

Implementation Phases

1. Correct Terminology and Define Typed Contracts

• Add typed domain models for request manifests, software context, transport metadata, failure phase, and artifact descriptors.
• Version every persisted JSON contract from its first release.
• Define the safe response-header allowlist. Begin with correlation, content type/encoding, date, retry/rate-limit, and router-specific generation identifiers only when documented and non-secret.
• Define size limits and external-storage thresholds for exact bodies and artifacts.
• Correct docs/ver4/schema_v4.md under “Page-Level Execution and AI Outputs” so the existing column is described as an SDK-serialized OpenRouter response snapshot, not a complete provider envelope, exact HTTP body, or native upstream-provider response. Apply the same terminology to architecture and UI schema references.
• Add serialization and secret-rejection unit tests before provider changes.

2. Add Additive Persistence and Upgrade Behavior

• Add the selected execution-evidence and artifact models.
• Add foreign keys, uniqueness constraints, and indexes for source/execution lookup.
• Implement idempotent upgrades following the repository's existing schema-upgrade policy.
• Do not populate exact response fields for historical rows.
• Do not write a capture-time classification onto historical rows during migration. Compatibility reads may describe a populated legacy raw_api_response as an SDK snapshot, but exports must identify that description as a later compatibility interpretation rather than execution-time metadata.
• Verify JSON portability and large-payload behavior for SQLite and PostgreSQL.
• Add upgrade tests starting from a representative pre-V4.2 schema.

3. Build Secret-Safe Request Manifests

• Build the manifest from the concrete outgoing request body immediately before transport, not from a narrower typed projection that may discard unrecognized request fields.
• Replace each image payload in that concrete representation with a source reference containing source UUID, digest, byte size, media type, dimensions, and transformation identity.
• Store exact prompt content and preserve omitted-versus-explicit parameter state.
• Include requested model, routing preferences, response-format requirements, and timeout/retry policy.
• Record application version/commit when available, adapter contract version, SDK package/version, and request-manifest schema version.
• Hash the canonical manifest representation for integrity checks.
• Test that credentials and embedded image data cannot enter the persisted manifest.
• Test that every field actually sent to the provider, including routing and future provider options, is represented or explicitly excluded by the manifest transform.

4. Capture OpenRouter Transport Evidence

• Evaluate the installed OpenRouter SDK hooks/client injection first.
• If hooks cannot expose an exact stable response before typed parsing, implement the non-streaming OpenRouter call through the existing async HTTP client boundary while retaining typed validation in the adapter.
• Read the response body once, preserve it exactly, then parse and normalize it.
• Store status, content type/encoding, allowlisted headers, request/generation ID, and timing.
• Maintain current authentication, referer/title headers, timeout behavior, and error classification.
• Explicitly document that the captured body is the OpenRouter-normalized transport response, not Gemini/Anthropic/OpenAI native upstream JSON.
• Add fixture-based tests proving unknown response fields survive transport capture even if a typed parser ignores them.

5. Preserve Failure Evidence

• Return or raise a typed provider failure that carries safe evidence separately from its user-facing error.
• Persist non-success status/body/allowlisted headers before marking an execution failed.
• Represent DNS/connect/TLS/local timeout failures as no-response outcomes with a failure phase and safe diagnostic category.
• Preserve response-validation failures with both the exact body and validation details.
• Keep transcription-quality rejection distinct from provider failure because a valid provider response was received.
• Ensure error strings and logs do not contain authorization data or embedded image payloads.
• Add tests for 4xx, 5xx, malformed JSON, schema mismatch, timeout, connection failure, and quality rejection.

6. Make Execution History Reliably Append-Only

• Confirm retry behavior creates a distinct execution attempt rather than reusing and overwriting a completed evidence record.
• Separate queue linkage from execution-attempt identity; the current update-in-place behavior cannot serve as append-only execution history.
• Assign each attempt a deterministic, monotonically increasing attempt number scoped to its Job and Source, enforced by a database uniqueness constraint.
• Update the latest-transcription projection only after a successful attempt.
• Never update prior response bodies, manifests, timings, or artifacts during a retry.
• Select the latest attempt and latest successful attempt by the persisted attempt number with a stable identifier as a defensive secondary key, never by timestamp alone.
• Add service/workflow tests covering retries, partial success, interrupted jobs, and historical projection behavior.

7. Add Generic Artifact Persistence

• Implement service operations to create, read, list, verify, export, and, only under explicit retention policy, delete artifacts.
• Validate semantic type, schema/version, digest, media type, and coordinate metadata.
• Support JSON artifacts inline initially when within the agreed size threshold.
• Support external artifacts through a constrained application-data root with atomic write, digest verification, and explicit missing-file errors.
• Add a provider-neutral example fixture representing OCR words/lines with polygons and confidence values.
• Do not integrate a live OCR vendor in this phase.

8. Add Evidence Inspection and Export

• Rename the current Source Detail label to identify historical values as an OpenRouter SDK Response Snapshot.
• Add separate sections for Request Manifest, Transport Response, Normalized Metadata, Software Context, and Derived Artifacts.
• Show an explicit “not captured for this historical execution” state instead of an empty object.
• Keep large bodies collapsed by default and avoid rendering embedded source data.
• Add a service-owned export that packages a versioned manifest, evidence JSON/body files, artifact content or references, and digest inventory.
• Exclude secrets and machine-local paths that are not required to interpret the evidence.
• Add UI and export tests for new, historical, failed, and large-evidence records.

9. Establish the Private Benchmark

• Select a small initial corpus, then expand only when it exposes meaningful differences.
• Stratify examples by printed/typed text, handwriting style, degradation, layout complexity, language, and editorial anomaly.
• Reference existing Source UUIDs and digests in a private manifest; do not copy family documents into public test fixtures.
• Create manually reviewed reference transcriptions following the invariant methodology.
• Implement or adopt existing project-compatible CER/WER calculations without changing dependencies unless justified.
• Score omissions, inventions, silent modernization, uncertainty markup, and layout fidelity separately from CER/WER.
• Record cost and latency from preserved execution evidence.
• Run the current google/gemini-2.5-flash configuration as the baseline before testing alternatives.
• Treat results as model-version/route/corpus specific and preserve each comparison run.

10. Verify, Migrate, and Align Documentation

• Run the smallest focused model, provider, service, workflow, UI, upgrade, and export test groups first.
• Run broader regression tests only after focused validation passes.
• Execute all destructive tests through tools/run_destructive_tests.py.
• Verify backup creation and required restoration behavior before any test touching real application data.
• Confirm existing Source Detail records remain readable after upgrade.
• Update V4 architecture, schema, requirements, and UI schema mappings to point to V4.2 semantics.
• Record any deliberate deviation from this plan in the V4.2 scope before release.

Recommended Delivery Order

1. Typed/versioned evidence contracts and terminology.
2. Additive execution-evidence persistence.
3. Secret-safe request manifests.
4. Exact OpenRouter transport capture.
5. Failure evidence and append-only retry semantics.
6. Generic artifact persistence.
7. Inspection and export.
8. Private benchmark tooling and baseline run.
9. Migration, regression verification, and documentation alignment.

Key Implementation Decisions to Resolve

1. Whether execution evidence is a one-to-one companion to JobSource or part of a new execution-attempt model required for append-only retries.
2. Whether exact response bodies remain database values at expected sizes or move to hashed external files above a threshold.
3. The canonical JSON algorithm used to hash request manifests.
4. The safe-header allowlist supported by OpenRouter and future adapters.
5. The application version identity available in local, packaged, and uncommitted development builds.
6. The initial inline/external artifact size threshold and application-data root.
7. Whether evidence exports include original source binaries by default, optionally, or only by reference.
8. The minimum private benchmark corpus size and review process before model comparisons influence defaults.

These decisions must be settled before their corresponding implementation phase; they do not weaken the invariant or expand V4.2 into live OCR integration.

Resolved Implementation Decisions

1. JobSource remains queue linkage and a compatibility projection; immutable retries use a one-to-many ExecutionAttempt model with a unique (job_id, source_id, attempt_number) constraint.
2. Exact OpenRouter response bytes remain database values for V4.2. Generic artifacts use inline canonical JSON up to 1 MiB by default and constrained, atomically written external files above that threshold.
3. Request manifests use transcription-canonical-json-v1: UTF-8 JSON with sorted keys, compact separators, preserved Unicode, and non-finite numbers rejected.
4. Safe response headers are explicitly allowlisted in the evidence contract; all others are discarded before persistence.
5. Software identity records the package version, optional TRANSCRIPTION_COMMIT, adapter contract version, OpenRouter SDK version, and Python version.
6. The artifact root defaults to data/artifacts and stores source-scoped relative references.
7. Evidence exports include source identity and digest by reference, not original source binaries.
8. The benchmark manifest is private and digest-referenced. Corpus size remains archive-dependent, but every run uses preserved execution-attempt identity and the fixed literal scoring contract.

Done When

• Every V4.2 acceptance criterion is satisfied by focused tests or an explicit demonstration.
• Existing SDK snapshots retain their content and are labeled accurately.
• New successful and failed calls preserve secret-safe provider-boundary evidence.
• Unknown transport fields survive even when the typed SDK/parser does not recognize them.
• Retries cannot overwrite prior execution evidence.
• A generic versioned artifact can represent OCR geometry and pass integrity verification.
• Evidence can be safely inspected and exported with schema identities and digests.
• The current model has a reproducible private benchmark baseline.
• No credential or embedded source payload appears in persisted manifests, safe headers, logs, or exports.
• Existing V4.1 behavior remains compatible.

Out of Scope

• Live OCR/document-AI provider integration.
• Automatic model switching.
• Archive-wide reprocessing.
• Native upstream-provider response capture through OpenRouter when OpenRouter does not expose it.
• Guarantees of deterministic hosted-model output.

Related Local References

• V4.2 Scope Boundary
• Digital Evidence and AI Processing Provenance
• V4 Architecture
• V4 Schema
• V4 Requirements
• Draft V4.3 Implementation Plan