V4.1 revisions in preparation for v4.2. AI data capture now better defined.

docs/ver4.3/implementation_plan_v4_3.md

@@ -0,0 +1,241 @@
+# Implementation Plan (Version 4.3)
+
+## Goal
+
+Make processing evidence precise, append-only, secret-safe, and exportable while preserving every existing record and creating a provider-neutral home for future OCR/layout artifacts.
+
+## Implementation Principles
+
+- Implement the [Digital Evidence and AI Processing Provenance invariant](../invariant/ai_evidence_and_provenance.md), not a provider-specific approximation of it.
+- Capture transport evidence before SDK parsing.
+- Keep exact evidence separate from parsed and normalized representations.
+- Prefer additive schema evolution and explicit compatibility behavior.
+- Reference source content by digest rather than duplicating it in request JSON.
+- Use allowlists for safe metadata capture.
+- Keep persistence and evidence semantics behind service boundaries.
+- Do not change the default model until a representative benchmark supports that decision.
+
+## Current-State Gaps
+
+| Current behavior | Gap to close |
+| --- | --- |
+| `Source` stores original file path, digest, and size. | Media type and image/page geometry used for an execution are not frozen with that execution. |
+| `Job` stores prompt text/hash, requested model after resolution, temperature, and `top_p`. | The complete effective request structure, omitted-versus-explicit parameter state, routing constraints, and software versions are not frozen. |
+| `JobSource.raw_api_response` stores `model_dump()` output from the OpenRouter SDK. | The exact HTTP body can be normalized by OpenRouter and filtered again by the SDK before persistence. |
+| `JobSource.ai_metadata` stores finish reason and basic token counts. | Detailed accounting remains only in the SDK snapshot and is not a substitute for exact evidence. |
+| Provider exceptions become application errors. | Safe HTTP error bodies, statuses, headers, and no-response distinctions are not persisted. |
+| Worker logs elapsed time. | Execution duration is not stored on `JobSource`. |
+| Source Detail displays AI metadata and the SDK snapshot. | The UI does not identify evidence layers or expose request/transport/software provenance. |
+| No generic processing-artifact model exists. | Future OCR geometry would require ad hoc provider fields or an unrelated schema. |
+
+## Expected Project Impact
+
+| Area | Expected impact |
+| --- | --- |
+| Database models and upgrades | Add execution-specification, transport-evidence, timing, software-context, and generic artifact storage without removing existing columns. |
+| OpenRouter adapter | Introduce a transport boundary that can capture exact body/status/safe headers before typed SDK parsing, or use supported SDK hooks that expose the unparsed response reliably. |
+| Provider contract | Return structured evidence for success and failure without leaking provider-specific transport concerns into workflow orchestration. |
+| Source and workflow services | Persist one append-only execution outcome and its artifacts transactionally; retain compatibility projections. |
+| UI | Label and inspect evidence layers; export safe evidence packages through service operations. |
+| Benchmarking | Add a private manifest and repeatable evaluator using the literal-transcription methodology. |
+| Tests and documentation | Add compatibility, capture, security, integrity, export, and benchmark-scoring coverage; correct overstated V4 evidence language. |
+
+## Proposed Data Design
+
+Exact names should be confirmed against existing conventions before migration code is written. The design should provide the following logical records.
+
+### 1. Execution Evidence
+
+Extend `JobSource` or associate it one-to-one with a new execution-evidence record containing:
+
+- Request manifest JSON and manifest schema version.
+- Transport status, body bytes or exact decoded body plus encoding/content type, and safe headers.
+- Parsed SDK snapshot retained separately from transport content.
+- Application, adapter, SDK, and runtime version metadata.
+- Start, finish, and duration values.
+- Router/provider request and generation identifiers when available.
+- Failure phase and whether an HTTP response was received.
+
+The implementation should evaluate a companion table rather than continuing to widen `JobSource`. A companion record better isolates large/optional evidence and permits clear one-to-one compatibility semantics.
+
+### 2. Generic Processing Artifact
+
+Add a one-to-many artifact model associated with a source and, when applicable, a producing execution:
+
+- Stable artifact UUID.
+- `source_id` and optional execution/`job_source_id`.
+- Semantic artifact type.
+- Media/serialization format.
+- Schema name and version.
+- Producer and producer version.
+- Inline JSON payload or external location.
+- Payload digest and byte size.
+- Coordinate-system metadata when relevant.
+- Creation timestamp.
+
+Enforce exactly one content location: inline payload or external reference. An external artifact must be written durably and hashed before its database record commits.
+
+### 3. Compatibility Projections
+
+- Keep `JobSource.raw_api_response` unchanged for existing and new compatibility reads until a later deprecation decision.
+- Keep `JobSource.ai_metadata` for indexed/display-ready normalized values.
+- Keep `Source.raw_transcription` as the latest successful machine-output projection while treating per-execution `JobSource.raw_transcription` as history.
+- Document that older rows have an SDK snapshot but no exact transport capture.
+
+## Implementation Phases
+
+### 1. Correct Terminology and Define Typed Contracts
+
+- Add typed domain models for request manifests, software context, transport metadata, failure phase, and artifact descriptors.
+- Version every persisted JSON contract from its first release.
+- Define the safe response-header allowlist. Begin with correlation, content type/encoding, date, retry/rate-limit, and router-specific generation identifiers only when documented and non-secret.
+- Define size limits and external-storage thresholds for exact bodies and artifacts.
+- Correct `docs/ver4/schema_v4.md` under “Page-Level Execution and AI Outputs” so the existing column is described as an SDK-serialized OpenRouter response snapshot, not a complete provider envelope, exact HTTP body, or native upstream-provider response. Apply the same terminology to architecture and UI schema references.
+- Add serialization and secret-rejection unit tests before provider changes.
+
+### 2. Add Additive Persistence and Upgrade Behavior
+
+- Add the selected execution-evidence and artifact models.
+- Add foreign keys, uniqueness constraints, and indexes for source/execution lookup.
+- Implement idempotent upgrades following the repository's existing schema-upgrade policy.
+- Do not populate exact response fields for historical rows.
+- Do not write a capture-time classification onto historical rows during migration. Compatibility reads may describe a populated legacy `raw_api_response` as an SDK snapshot, but exports must identify that description as a later compatibility interpretation rather than execution-time metadata.
+- Verify JSON portability and large-payload behavior for SQLite and PostgreSQL.
+- Add upgrade tests starting from a representative pre-V4.3 schema.
+
+### 3. Build Secret-Safe Request Manifests
+
+- Build the manifest from the concrete outgoing request body immediately before transport, not from a narrower typed projection that may discard unrecognized request fields.
+- Replace each image payload in that concrete representation with a source reference containing source UUID, digest, byte size, media type, dimensions, and transformation identity.
+- Store exact prompt content and preserve omitted-versus-explicit parameter state.
+- Include requested model, routing preferences, response-format requirements, and timeout/retry policy.
+- Record application version/commit when available, adapter contract version, SDK package/version, and request-manifest schema version.
+- Hash the canonical manifest representation for integrity checks.
+- Test that credentials and embedded image data cannot enter the persisted manifest.
+- Test that every field actually sent to the provider, including routing and future provider options, is represented or explicitly excluded by the manifest transform.
+
+### 4. Capture OpenRouter Transport Evidence
+
+- Evaluate the installed OpenRouter SDK hooks/client injection first.
+- If hooks cannot expose an exact stable response before typed parsing, implement the non-streaming OpenRouter call through the existing async HTTP client boundary while retaining typed validation in the adapter.
+- Read the response body once, preserve it exactly, then parse and normalize it.
+- Store status, content type/encoding, allowlisted headers, request/generation ID, and timing.
+- Maintain current authentication, referer/title headers, timeout behavior, and error classification.
+- Explicitly document that the captured body is the OpenRouter-normalized transport response, not Gemini/Anthropic/OpenAI native upstream JSON.
+- Add fixture-based tests proving unknown response fields survive transport capture even if a typed parser ignores them.
+
+### 5. Preserve Failure Evidence
+
+- Return or raise a typed provider failure that carries safe evidence separately from its user-facing error.
+- Persist non-success status/body/allowlisted headers before marking an execution failed.
+- Represent DNS/connect/TLS/local timeout failures as no-response outcomes with a failure phase and safe diagnostic category.
+- Preserve response-validation failures with both the exact body and validation details.
+- Keep transcription-quality rejection distinct from provider failure because a valid provider response was received.
+- Ensure error strings and logs do not contain authorization data or embedded image payloads.
+- Add tests for 4xx, 5xx, malformed JSON, schema mismatch, timeout, connection failure, and quality rejection.
+
+### 6. Make Execution History Reliably Append-Only
+
+- Confirm retry behavior creates a distinct execution attempt rather than reusing and overwriting a completed evidence record.
+- Separate queue linkage from execution-attempt identity; the current update-in-place behavior cannot serve as append-only execution history.
+- Assign each attempt a deterministic, monotonically increasing attempt number scoped to its Job and Source, enforced by a database uniqueness constraint.
+- Update the latest-transcription projection only after a successful attempt.
+- Never update prior response bodies, manifests, timings, or artifacts during a retry.
+- Select the latest attempt and latest successful attempt by the persisted attempt number with a stable identifier as a defensive secondary key, never by timestamp alone.
+- Add service/workflow tests covering retries, partial success, interrupted jobs, and historical projection behavior.
+
+### 7. Add Generic Artifact Persistence
+
+- Implement service operations to create, read, list, verify, export, and, only under explicit retention policy, delete artifacts.
+- Validate semantic type, schema/version, digest, media type, and coordinate metadata.
+- Support JSON artifacts inline initially when within the agreed size threshold.
+- Support external artifacts through a constrained application-data root with atomic write, digest verification, and explicit missing-file errors.
+- Add a provider-neutral example fixture representing OCR words/lines with polygons and confidence values.
+- Do not integrate a live OCR vendor in this phase.
+
+### 8. Add Evidence Inspection and Export
+
+- Rename the current Source Detail label to identify historical values as an OpenRouter SDK Response Snapshot.
+- Add separate sections for Request Manifest, Transport Response, Normalized Metadata, Software Context, and Derived Artifacts.
+- Show an explicit “not captured for this historical execution” state instead of an empty object.
+- Keep large bodies collapsed by default and avoid rendering embedded source data.
+- Add a service-owned export that packages a versioned manifest, evidence JSON/body files, artifact content or references, and digest inventory.
+- Exclude secrets and machine-local paths that are not required to interpret the evidence.
+- Add UI and export tests for new, historical, failed, and large-evidence records.
+
+### 9. Establish the Private Benchmark
+
+- Select a small initial corpus, then expand only when it exposes meaningful differences.
+- Stratify examples by printed/typed text, handwriting style, degradation, layout complexity, language, and editorial anomaly.
+- Reference existing Source UUIDs and digests in a private manifest; do not copy family documents into public test fixtures.
+- Create manually reviewed reference transcriptions following the invariant methodology.
+- Implement or adopt existing project-compatible CER/WER calculations without changing dependencies unless justified.
+- Score omissions, inventions, silent modernization, uncertainty markup, and layout fidelity separately from CER/WER.
+- Record cost and latency from preserved execution evidence.
+- Run the current `google/gemini-2.5-flash` configuration as the baseline before testing alternatives.
+- Treat results as model-version/route/corpus specific and preserve each comparison run.
+
+### 10. Verify, Migrate, and Align Documentation
+
+- Run the smallest focused model, provider, service, workflow, UI, upgrade, and export test groups first.
+- Run broader regression tests only after focused validation passes.
+- Execute all destructive tests through `tools/run_destructive_tests.py`.
+- Verify backup creation and required restoration behavior before any test touching real application data.
+- Confirm existing Source Detail records remain readable after upgrade.
+- Update V4 architecture, schema, requirements, and UI schema mappings to point to V4.3 semantics.
+- Record any deliberate deviation from this plan in the V4.3 scope before release.
+
+## Recommended Delivery Order
+
+1. Typed/versioned evidence contracts and terminology.
+2. Additive execution-evidence persistence.
+3. Secret-safe request manifests.
+4. Exact OpenRouter transport capture.
+5. Failure evidence and append-only retry semantics.
+6. Generic artifact persistence.
+7. Inspection and export.
+8. Private benchmark tooling and baseline run.
+9. Migration, regression verification, and documentation alignment.
+
+## Key Implementation Decisions to Resolve
+
+1. Whether execution evidence is a one-to-one companion to `JobSource` or part of a new execution-attempt model required for append-only retries.
+2. Whether exact response bodies remain database values at expected sizes or move to hashed external files above a threshold.
+3. The canonical JSON algorithm used to hash request manifests.
+4. The safe-header allowlist supported by OpenRouter and future adapters.
+5. The application version identity available in local, packaged, and uncommitted development builds.
+6. The initial inline/external artifact size threshold and application-data root.
+7. Whether evidence exports include original source binaries by default, optionally, or only by reference.
+8. The minimum private benchmark corpus size and review process before model comparisons influence defaults.
+
+These decisions must be settled before their corresponding implementation phase; they do not weaken the invariant or expand V4.3 into live OCR integration.
+
+## Done When
+
+- Every V4.3 acceptance criterion is satisfied by focused tests or an explicit demonstration.
+- Existing SDK snapshots retain their content and are labeled accurately.
+- New successful and failed calls preserve secret-safe provider-boundary evidence.
+- Unknown transport fields survive even when the typed SDK/parser does not recognize them.
+- Retries cannot overwrite prior execution evidence.
+- A generic versioned artifact can represent OCR geometry and pass integrity verification.
+- Evidence can be safely inspected and exported with schema identities and digests.
+- The current model has a reproducible private benchmark baseline.
+- No credential or embedded source payload appears in persisted manifests, safe headers, logs, or exports.
+- Existing V4.1 and V4.2 behavior remains compatible.
+
+## Out of Scope
+
+- Live OCR/document-AI provider integration.
+- Automatic model switching.
+- Archive-wide reprocessing.
+- Native upstream-provider response capture through OpenRouter when OpenRouter does not expose it.
+- Guarantees of deterministic hosted-model output.
+
+## Related Local References
+
+- [V4.3 Scope Boundary](scope_boundary_v4_3.md)
+- [Digital Evidence and AI Processing Provenance](../invariant/ai_evidence_and_provenance.md)
+- [V4 Architecture](../ver4/architecture_v4.md)
+- [V4 Schema](../ver4/schema_v4.md)
+- [V4 Requirements](../ver4/requirements_v4.md)
+- [V4.2 Implementation Plan](../ver4.2/implementation_plan_v4_2.md)