V4.1 major revision to docs. Removed all obsolete documents, updated v4.2 implementation scope and plan.

docs/ver4.2/implementation_plan_v4_2.md

@@ -1,137 +1,241 @@
-# Draft Implementation Plan (Version 4.2)
+# Implementation Plan (Version 4.2)
 
 ## Goal
 
-Prepare a safe implementation path for Source page reordering and constrained application settings. This plan remains provisional until the V4.2 scope-freeze decisions are resolved.
+Make processing evidence precise, append-only, secret-safe, and exportable while preserving every existing record and creating a provider-neutral home for future OCR/layout artifacts.
 
-## Planning Constraints
+## Implementation Principles
 
-- V4 and V4.1 remain the behavioral baseline.
-- Reordering must be atomic and service-owned.
-- Settings must use explicit domain operations rather than direct database, environment-file, or arbitrary filesystem access from UI pages.
-- Prompt changes must preserve historical Job provenance and use a defined safe-write policy.
+- Implement the [Digital Evidence and AI Processing Provenance invariant](../invariant/ai_evidence_and_provenance.md), not a provider-specific approximation of it.
+- Capture transport evidence before SDK parsing.
+- Keep exact evidence separate from parsed and normalized representations.
+- Prefer additive schema evolution and explicit compatibility behavior.
+- Reference source content by digest rather than duplicating it in request JSON.
+- Use allowlists for safe metadata capture.
+- Keep persistence and evidence semantics behind service boundaries.
+- Do not change the default model until a representative benchmark supports that decision.
+
+## Current-State Gaps
+
+| Current behavior | Gap to close |
+| --- | --- |
+| `Source` stores original file path, digest, and size. | Media type and image/page geometry used for an execution are not frozen with that execution. |
+| `Job` stores prompt text/hash, requested model after resolution, temperature, and `top_p`. | The complete effective request structure, omitted-versus-explicit parameter state, routing constraints, and software versions are not frozen. |
+| `JobSource.raw_api_response` stores `model_dump()` output from the OpenRouter SDK. | The exact HTTP body can be normalized by OpenRouter and filtered again by the SDK before persistence. |
+| `JobSource.ai_metadata` stores finish reason and basic token counts. | Detailed accounting remains only in the SDK snapshot and is not a substitute for exact evidence. |
+| Provider exceptions become application errors. | Safe HTTP error bodies, statuses, headers, and no-response distinctions are not persisted. |
+| Worker logs elapsed time. | Execution duration is not stored on `JobSource`. |
+| Source Detail displays AI metadata and the SDK snapshot. | The UI does not identify evidence layers or expose request/transport/software provenance. |
+| No generic processing-artifact model exists. | Future OCR geometry would require ad hoc provider fields or an unrelated schema. |
 
 ## Expected Project Impact
 
 | Area | Expected impact |
 | --- | --- |
-| Sources service | Add validated, transactional set-based page reordering. |
-| Documents/Sources UI | Add a reorder entry point and interaction for one Document. |
-| Documents service | Expand controlled Document Type maintenance operations. |
-| People service | Expand controlled Person Role maintenance operations. |
-| Prompt adapter/service | Add constrained listing, reading, validation, and safe writing of prompt artifacts. |
-| UI composition/navigation | Register Settings routes and navigation without moving persistence into UI code. |
-| Tests | Add transaction, conflict, registry lifecycle, prompt safety, and UI workflow coverage. |
+| Database models and upgrades | Add execution-specification, transport-evidence, timing, software-context, and generic artifact storage without removing existing columns. |
+| OpenRouter adapter | Introduce a transport boundary that can capture exact body/status/safe headers before typed SDK parsing, or use supported SDK hooks that expose the unparsed response reliably. |
+| Provider contract | Return structured evidence for success and failure without leaking provider-specific transport concerns into workflow orchestration. |
+| Source and workflow services | Persist one append-only execution outcome and its artifacts transactionally; retain compatibility projections. |
+| UI | Label and inspect evidence layers; export safe evidence packages through service operations. |
+| Benchmarking | Add a private manifest and repeatable evaluator using the literal-transcription methodology. |
+| Tests and documentation | Add compatibility, capture, security, integrity, export, and benchmark-scoring coverage; correct overstated V4 evidence language. |
 
-## Proposed Implementation Phases
+## Proposed Data Design
 
-### 1. Resolve Scope-Freeze Decisions
+Exact names should be confirmed against existing conventions before migration code is written. The design should provide the following logical records.
 
-- Select and document the reorder interaction.
-- Define whether active Jobs block reorder.
-- Define prompt atomic-write, backup, and recovery policy.
-- Decide whether prompt creation/deletion is excluded.
-- Finalize registry ordering requirements.
-- Remove the Draft designation only after these decisions are reflected in scope and acceptance criteria.
+### 1. Execution Evidence
 
-### 2. Define Service Contracts
+Extend `JobSource` or associate it one-to-one with a new execution-evidence record containing:
 
-- Define a Source reorder command containing `document_id`, the complete ordered Source ID list, and a concurrency token or equivalent stale-write guard if supported by the current model.
-- Define Document Type maintenance commands for create, relabel, sort, activate, and deactivate.
-- Define Person Role maintenance commands for create, relabel, activate, and deactivate.
-- Define a Prompt Store interface for constrained list/read/write behavior.
-- Map validation, conflict, not-found, dependency, and filesystem failures to existing `AppError` categories.
+- Request manifest JSON and manifest schema version.
+- Transport status, body bytes or exact decoded body plus encoding/content type, and safe headers.
+- Parsed SDK snapshot retained separately from transport content.
+- Application, adapter, SDK, and runtime version metadata.
+- Start, finish, and duration values.
+- Router/provider request and generation identifiers when available.
+- Failure phase and whether an HTTP response was received.
 
-### 3. Implement Transactional Source Reordering
+The implementation should evaluate a companion table rather than continuing to widen `JobSource`. A companion record better isolates large/optional evidence and permits clear one-to-one compatibility semantics.
 
-- Load all Sources for the target Document in the same transaction.
-- Reject missing, extra, duplicate, or foreign Source IDs.
-- Reject stale writes using the selected concurrency policy.
-- Apply a collision-safe renumbering strategy suitable for both SQLite and PostgreSQL.
-- Finish with contiguous `page_number` values beginning at 1.
-- Roll back the entire operation on any failure.
-- Add service tests for valid reorder, no-op, reverse order, invalid membership, duplicates, stale submissions, rollback, and backend-compatible SQL behavior.
+### 2. Generic Processing Artifact
 
-### 4. Implement the Reorder UI
+Add a one-to-many artifact model associated with a source and, when applicable, a producing execution:
 
-- Add a Reorder Pages action from a Document-scoped Source view or Document Detail.
-- Render Source labels/previews sufficient to identify each page.
-- Capture the complete intended order.
-- Require explicit Save and provide Cancel without mutation.
-- Surface validation and conflict errors through the shared error presenter.
-- Return to a Document-scoped ordered view after success.
-- Verify keyboard-accessible controls for any drag-and-drop interaction.
+- Stable artifact UUID.
+- `source_id` and optional execution/`job_source_id`.
+- Semantic artifact type.
+- Media/serialization format.
+- Schema name and version.
+- Producer and producer version.
+- Inline JSON payload or external location.
+- Payload digest and byte size.
+- Coordinate-system metadata when relevant.
+- Creation timestamp.
 
-### 5. Expand Registry Maintenance Services
+Enforce exactly one content location: inline payload or external reference. An external artifact must be written durably and hashed before its database record commits.
 
-- Reuse existing Document and People service ownership.
-- Add explicit write methods rather than passing UI-mutated ORM objects directly where practical.
-- Normalize and validate new stable codes.
-- Reject duplicate codes deterministically.
-- Block deletion or omit deletion entirely; use activation state for lifecycle management.
-- Preserve inactive entries for historical reads.
-- Add service tests for create, relabel, activation, deactivation, duplicates, immutable codes, and referenced records.
+### 3. Compatibility Projections
 
-### 6. Add Constrained Prompt Storage
+- Keep `JobSource.raw_api_response` unchanged for existing and new compatibility reads until a later deprecation decision.
+- Keep `JobSource.ai_metadata` for indexed/display-ready normalized values.
+- Keep `Source.raw_transcription` as the latest successful machine-output projection while treating per-execution `JobSource.raw_transcription` as history.
+- Document that older rows have an SDK snapshot but no exact transport capture.
 
-- Place filesystem access behind a dedicated Prompt Store/service boundary.
-- Resolve all filenames directly beneath the configured prompt root and reject traversal.
-- Permit only the agreed markdown extension and reject empty content.
-- Implement the approved safe-write strategy, including flush/replace behavior and backup/recovery if selected.
-- Preserve file encoding and provide explicit failures for read-only or unavailable storage.
-- Do not modify any Job row when prompt defaults change.
-- Add unit tests for valid reads/writes, traversal, invalid names, empty content, filesystem failures, and unchanged Job provenance.
+## Implementation Phases
 
-### 7. Build the Settings UI
+### 1. Correct Terminology and Define Typed Contracts
 
-- Register a Settings landing page and navigation entry.
-- Add separate pages or panels for Document Types, Person Roles, and Prompts.
-- Keep pages responsible for orchestration and notifications only.
-- Use service callbacks for all mutations.
-- Explain stable codes, inactive historical entries, and future-only prompt effects in the UI.
-- Do not render raw environment values or secrets.
+- Add typed domain models for request manifests, software context, transport metadata, failure phase, and artifact descriptors.
+- Version every persisted JSON contract from its first release.
+- Define the safe response-header allowlist. Begin with correlation, content type/encoding, date, retry/rate-limit, and router-specific generation identifiers only when documented and non-secret.
+- Define size limits and external-storage thresholds for exact bodies and artifacts.
+- Correct `docs/ver4/schema_v4.md` under “Page-Level Execution and AI Outputs” so the existing column is described as an SDK-serialized OpenRouter response snapshot, not a complete provider envelope, exact HTTP body, or native upstream-provider response. Apply the same terminology to architecture and UI schema references.
+- Add serialization and secret-rejection unit tests before provider changes.
 
-### 8. Verification and Rollout
+### 2. Add Additive Persistence and Upgrade Behavior
 
-- Run focused service tests before UI integration tests.
-- Verify reorder behavior against Documents with one and many Sources.
-- Verify ordered transcription rendering and V4.1 previous/next navigation after reorder.
-- Verify inactive registry behavior in both historical display and create/edit selectors.
-- Verify prompt changes are picked up by newly created Jobs while historical Jobs retain frozen content/hash.
-- Run the relevant regression suite.
+- Add the selected execution-evidence and artifact models.
+- Add foreign keys, uniqueness constraints, and indexes for source/execution lookup.
+- Implement idempotent upgrades following the repository's existing schema-upgrade policy.
+- Do not populate exact response fields for historical rows.
+- Do not write a capture-time classification onto historical rows during migration. Compatibility reads may describe a populated legacy `raw_api_response` as an SDK snapshot, but exports must identify that description as a later compatibility interpretation rather than execution-time metadata.
+- Verify JSON portability and large-payload behavior for SQLite and PostgreSQL.
+- Add upgrade tests starting from a representative pre-V4.2 schema.
 
-## Migration and Compatibility Notes
+### 3. Build Secret-Safe Request Manifests
 
-- No new table is expected solely for reordering; `Source.page_number` remains authoritative.
-- A uniqueness constraint on `(document_id, page_number)` should be evaluated before scope freeze. If added, migration and collision-safe update behavior must be designed for both supported databases.
-- Existing registry records remain valid.
-- Prompt editing changes mutable application files, not database provenance already captured on Jobs.
-- V4.2 must not require users to recreate existing Sources, Documents, People, roles, or types.
+- Build the manifest from the concrete outgoing request body immediately before transport, not from a narrower typed projection that may discard unrecognized request fields.
+- Replace each image payload in that concrete representation with a source reference containing source UUID, digest, byte size, media type, dimensions, and transformation identity.
+- Store exact prompt content and preserve omitted-versus-explicit parameter state.
+- Include requested model, routing preferences, response-format requirements, and timeout/retry policy.
+- Record application version/commit when available, adapter contract version, SDK package/version, and request-manifest schema version.
+- Hash the canonical manifest representation for integrity checks.
+- Test that credentials and embedded image data cannot enter the persisted manifest.
+- Test that every field actually sent to the provider, including routing and future provider options, is represented or explicitly excluded by the manifest transform.
 
-## Proposed Delivery Order
+### 4. Capture OpenRouter Transport Evidence
 
-1. Freeze the remaining decisions.
-2. Implement and verify Source reorder service semantics.
-3. Build the reorder UI.
-4. Implement registry maintenance service operations.
-5. Implement the Prompt Store and safety policy.
-6. Build Settings pages.
-7. Run integration and regression verification.
+- Evaluate the installed OpenRouter SDK hooks/client injection first.
+- If hooks cannot expose an exact stable response before typed parsing, implement the non-streaming OpenRouter call through the existing async HTTP client boundary while retaining typed validation in the adapter.
+- Read the response body once, preserve it exactly, then parse and normalize it.
+- Store status, content type/encoding, allowlisted headers, request/generation ID, and timing.
+- Maintain current authentication, referer/title headers, timeout behavior, and error classification.
+- Explicitly document that the captured body is the OpenRouter-normalized transport response, not Gemini/Anthropic/OpenAI native upstream JSON.
+- Add fixture-based tests proving unknown response fields survive transport capture even if a typed parser ignores them.
 
-## Draft Done Criteria
+### 5. Preserve Failure Evidence
 
-- All V4.2 acceptance criteria are testable and satisfied.
-- Reordering is atomic, conflict-aware, contiguous, and cross-database compatible.
-- Settings mutations cross explicit service or adapter boundaries.
-- Registry codes cannot be accidentally changed.
-- Prompt writes cannot escape the configured directory or rewrite historical provenance.
-- No secret or raw environment editor exists.
-- V4.1 workflows remain intact.
+- Return or raise a typed provider failure that carries safe evidence separately from its user-facing error.
+- Persist non-success status/body/allowlisted headers before marking an execution failed.
+- Represent DNS/connect/TLS/local timeout failures as no-response outcomes with a failure phase and safe diagnostic category.
+- Preserve response-validation failures with both the exact body and validation details.
+- Keep transcription-quality rejection distinct from provider failure because a valid provider response was received.
+- Ensure error strings and logs do not contain authorization data or embedded image payloads.
+- Add tests for 4xx, 5xx, malformed JSON, schema mismatch, timeout, connection failure, and quality rejection.
+
+### 6. Make Execution History Reliably Append-Only
+
+- Confirm retry behavior creates a distinct execution attempt rather than reusing and overwriting a completed evidence record.
+- Separate queue linkage from execution-attempt identity; the current update-in-place behavior cannot serve as append-only execution history.
+- Assign each attempt a deterministic, monotonically increasing attempt number scoped to its Job and Source, enforced by a database uniqueness constraint.
+- Update the latest-transcription projection only after a successful attempt.
+- Never update prior response bodies, manifests, timings, or artifacts during a retry.
+- Select the latest attempt and latest successful attempt by the persisted attempt number with a stable identifier as a defensive secondary key, never by timestamp alone.
+- Add service/workflow tests covering retries, partial success, interrupted jobs, and historical projection behavior.
+
+### 7. Add Generic Artifact Persistence
+
+- Implement service operations to create, read, list, verify, export, and, only under explicit retention policy, delete artifacts.
+- Validate semantic type, schema/version, digest, media type, and coordinate metadata.
+- Support JSON artifacts inline initially when within the agreed size threshold.
+- Support external artifacts through a constrained application-data root with atomic write, digest verification, and explicit missing-file errors.
+- Add a provider-neutral example fixture representing OCR words/lines with polygons and confidence values.
+- Do not integrate a live OCR vendor in this phase.
+
+### 8. Add Evidence Inspection and Export
+
+- Rename the current Source Detail label to identify historical values as an OpenRouter SDK Response Snapshot.
+- Add separate sections for Request Manifest, Transport Response, Normalized Metadata, Software Context, and Derived Artifacts.
+- Show an explicit “not captured for this historical execution” state instead of an empty object.
+- Keep large bodies collapsed by default and avoid rendering embedded source data.
+- Add a service-owned export that packages a versioned manifest, evidence JSON/body files, artifact content or references, and digest inventory.
+- Exclude secrets and machine-local paths that are not required to interpret the evidence.
+- Add UI and export tests for new, historical, failed, and large-evidence records.
+
+### 9. Establish the Private Benchmark
+
+- Select a small initial corpus, then expand only when it exposes meaningful differences.
+- Stratify examples by printed/typed text, handwriting style, degradation, layout complexity, language, and editorial anomaly.
+- Reference existing Source UUIDs and digests in a private manifest; do not copy family documents into public test fixtures.
+- Create manually reviewed reference transcriptions following the invariant methodology.
+- Implement or adopt existing project-compatible CER/WER calculations without changing dependencies unless justified.
+- Score omissions, inventions, silent modernization, uncertainty markup, and layout fidelity separately from CER/WER.
+- Record cost and latency from preserved execution evidence.
+- Run the current `google/gemini-2.5-flash` configuration as the baseline before testing alternatives.
+- Treat results as model-version/route/corpus specific and preserve each comparison run.
+
+### 10. Verify, Migrate, and Align Documentation
+
+- Run the smallest focused model, provider, service, workflow, UI, upgrade, and export test groups first.
+- Run broader regression tests only after focused validation passes.
+- Execute all destructive tests through `tools/run_destructive_tests.py`.
+- Verify backup creation and required restoration behavior before any test touching real application data.
+- Confirm existing Source Detail records remain readable after upgrade.
+- Update V4 architecture, schema, requirements, and UI schema mappings to point to V4.2 semantics.
+- Record any deliberate deviation from this plan in the V4.2 scope before release.
+
+## Recommended Delivery Order
+
+1. Typed/versioned evidence contracts and terminology.
+2. Additive execution-evidence persistence.
+3. Secret-safe request manifests.
+4. Exact OpenRouter transport capture.
+5. Failure evidence and append-only retry semantics.
+6. Generic artifact persistence.
+7. Inspection and export.
+8. Private benchmark tooling and baseline run.
+9. Migration, regression verification, and documentation alignment.
+
+## Key Implementation Decisions to Resolve
+
+1. Whether execution evidence is a one-to-one companion to `JobSource` or part of a new execution-attempt model required for append-only retries.
+2. Whether exact response bodies remain database values at expected sizes or move to hashed external files above a threshold.
+3. The canonical JSON algorithm used to hash request manifests.
+4. The safe-header allowlist supported by OpenRouter and future adapters.
+5. The application version identity available in local, packaged, and uncommitted development builds.
+6. The initial inline/external artifact size threshold and application-data root.
+7. Whether evidence exports include original source binaries by default, optionally, or only by reference.
+8. The minimum private benchmark corpus size and review process before model comparisons influence defaults.
+
+These decisions must be settled before their corresponding implementation phase; they do not weaken the invariant or expand V4.2 into live OCR integration.
+
+## Done When
+
+- Every V4.2 acceptance criterion is satisfied by focused tests or an explicit demonstration.
+- Existing SDK snapshots retain their content and are labeled accurately.
+- New successful and failed calls preserve secret-safe provider-boundary evidence.
+- Unknown transport fields survive even when the typed SDK/parser does not recognize them.
+- Retries cannot overwrite prior execution evidence.
+- A generic versioned artifact can represent OCR geometry and pass integrity verification.
+- Evidence can be safely inspected and exported with schema identities and digests.
+- The current model has a reproducible private benchmark baseline.
+- No credential or embedded source payload appears in persisted manifests, safe headers, logs, or exports.
+- Existing V4.1 behavior remains compatible.
+
+## Out of Scope
+
+- Live OCR/document-AI provider integration.
+- Automatic model switching.
+- Archive-wide reprocessing.
+- Native upstream-provider response capture through OpenRouter when OpenRouter does not expose it.
+- Guarantees of deterministic hosted-model output.
 
 ## Related Local References
 
-- [Draft V4.2 Scope Boundary](scope_boundary_v4_2.md)
-- [V4.1 Implementation Plan](../ver4.1/implementation_plan_v4_1.md)
-- [V4 Implementation Plan](../ver4/implementation_plan_v4.md)
-- [V4 Error Handling Policy](../ver4/error_handling_v4.md)
-
+- [V4.2 Scope Boundary](scope_boundary_v4_2.md)
+- [Digital Evidence and AI Processing Provenance](../invariant/ai_evidence_and_provenance.md)
+- [V4 Architecture](../ver4/architecture_v4.md)
+- [V4 Schema](../ver4/schema_v4.md)
+- [V4 Requirements](../ver4/requirements_v4.md)
+- [Draft V4.3 Implementation Plan](../ver4.3/implementation_plan_v4_3.md)

docs/ver4.2/scope_boundary_v4_2.md

@@ -1,130 +1,163 @@
-# Draft V4.2 Scope Boundary
+# V4.2 Scope Boundary
 
-This document defines the proposed boundary for the second incremental revision to Version 4. It is intentionally a draft until V4.1 has been used and the remaining workflows have been validated.
+This document defines the boundary for the digital-evidence and AI-provenance revision that follows V4.1 and precedes the planned V4.3 page-reordering and settings work. V4 remains the architecture baseline; V4.2 makes the existing evidence claims precise and adds a provider-neutral foundation for future processing artifacts.
 
 ## Purpose
 
-- Allow correction of Source page order after import.
-- Provide a constrained Settings area for safe maintenance of selected application-managed configuration.
-- Avoid exposing secrets, restart-sensitive settings, or unrestricted filesystem editing through the UI.
+- Align the application with the [Digital Evidence and AI Processing Provenance invariant](../invariant/ai_evidence_and_provenance.md).
+- Preserve provider-boundary evidence before SDK parsing can remove unknown fields.
+- Make successful and failed processing attempts inspectable without storing secrets.
+- Support future OCR and layout outputs without coupling the database to one vendor.
+- Establish a repeatable method for comparing transcription models against this archive.
 
-## Proposed In Scope
+## In Scope
 
-### 1. Source Page Reordering
+### 1. Evidence Terminology and Existing-Data Compatibility
 
-- Allow Sources within one Document to be reordered after import.
-- Present the current order using page number and a recognizable source label or preview.
-- Persist the complete intended order atomically.
-- Renumber the affected Document's Sources to a contiguous sequence beginning at 1.
-- Keep all Sources attached to their existing Document.
-- Ensure transcription rendering and previous/next navigation use the updated order.
-- Detect stale or invalid reorder submissions and fail without partial mutation.
+- Define transport response, router-normalized response, SDK response, normalized metadata, and derived artifact consistently in code, schema documentation, and UI labels.
+- Treat existing `JobSource.raw_api_response` values as historical SDK response snapshots.
+- Preserve every existing `Job`, `Source`, and `JobSource` row.
+- Use additive migrations and compatibility reads; do not reinterpret previously stored values as exact transport captures.
+- Correct the “Page-Level Execution and AI Outputs” rule in `docs/ver4/schema_v4.md` that currently describes `JOB_SOURCE` as storing a complete provider response envelope. The corrected rule must identify `raw_api_response` as an SDK-serialized OpenRouter response snapshot and state that it is neither the exact HTTP body nor the native upstream-provider response.
 
-### 2. Settings Navigation
+### 2. Secret-Safe Request Manifests
 
-- Add a Settings entry to application navigation.
-- Provide separate, clearly described settings areas rather than a raw configuration editor.
-- Restrict V4.2 settings to application-managed values that can be validated and safely changed at runtime.
+- Persist the effective request specification for each page execution without storing credentials or duplicate base64 media.
+- Include requested provider/model, routing constraints, prompt content and hash, explicitly supplied parameters, source digest, media type, dimensions when known, and page identity.
+- Distinguish an omitted optional parameter from an explicitly supplied null or value.
+- Record application, provider-adapter, Python client, and relevant schema versions.
+- Use source or derivative references in place of embedded media bytes.
 
-### 3. Document Type Maintenance
+### 3. Provider-Boundary Response Capture
 
-- List active and inactive Document Types.
-- Add new types with a stable unique code and user-facing label.
-- Edit mutable labels and sort order.
-- Activate or deactivate types without invalidating historical Documents.
-- Do not allow changing a stable code after creation.
-- Do not delete types that are referenced by Documents.
+- Capture the exact HTTP response body before OpenRouter SDK parsing for non-streaming transcription calls.
+- Store HTTP status and an explicit allowlist of safe response headers.
+- Store router request/generation identifiers and resolved model/provider-routing metadata when exposed.
+- Preserve the current parsed SDK snapshot and normalized metadata where useful.
+- Keep exact body, parsed representation, and normalized fields distinguishable.
 
-### 4. Person Role Maintenance
+### 4. Failure Evidence and Timing
 
-- List active and inactive Person Roles.
-- Add new roles with a stable unique code and user-facing label.
-- Edit mutable labels.
-- Activate or deactivate roles without invalidating historical links.
-- Do not allow changing a stable code after creation.
-- Do not delete roles that are referenced by document-person links.
+- Create or update a page execution record for every attempted provider call.
+- Persist safe response evidence for non-success HTTP responses.
+- Distinguish HTTP response failures, connection failures, local timeouts, response-validation failures, and transcription-quality failures.
+- Store execution start/end times or duration using a clearly defined clock policy.
+- Do not collapse a provider error body into only a generic user-facing message.
 
-### 5. Prompt Maintenance
+### 5. Generic Processing Artifacts
 
-- List prompt markdown files from the configured prompt directory.
-- View a prompt with a concise explanation of its purpose and use.
-- Edit an existing prompt as plain markdown text.
-- Validate the filename boundary and reject empty prompt content.
-- Save changes explicitly and report filesystem failures.
-- Preserve submission-time prompt text and hash already frozen on existing Jobs.
-- Define a safe-write and recovery approach before this feature is considered final scope.
+- Add a provider-neutral representation for versioned derived artifacts.
+- Support inline JSON and externally stored payloads with a digest and stable reference.
+- Record artifact type, format, schema/version, producer/version, source, producing execution, and creation time.
+- Define coordinate-system metadata sufficient for word, line, block, or page geometry.
+- Permit future OCR/layout/confidence results without implementing a vendor-specific table for each provider.
 
-## Proposed Out of Scope
+### 6. Evidence Inspection and Export
 
-- Viewing or editing raw `.env` files.
-- Displaying or changing provider API keys and other secrets.
-- Editing host, port, database connection, upload paths, or other restart-sensitive runtime settings.
-- Arbitrary file browsing or arbitrary prompt paths.
-- Runtime theme/CSS editing.
-- Installing themes or plugins.
-- Source movement between Documents as part of reordering.
-- Automatic ordering based on filenames, OCR, or image content.
-- FamilySearch API synchronization.
-- A generic external-reference registry.
-- Ancestry references and Google Maps links.
+- Expand Source Detail and/or Job Detail to identify the evidence layer being displayed.
+- Provide readable JSON inspection for request manifests, transport metadata, parsed responses, normalized metadata, and derived artifacts.
+- Provide a safe export containing evidence content or references, relationships, schema versions, and digests.
+- Clearly label evidence that was not captured for historical records.
+- Do not display or export credentials, unrestricted headers, or embedded base64 source media.
 
-## Proposed Design Decisions
+### 7. Representative-Corpus Benchmark Protocol
 
-### A. Reordering Is Set-Based
+- Define a private benchmark manifest referencing source digests rather than duplicating archival media.
+- Include representative printed, typed, handwritten, degraded, tabular, and spatially complex pages.
+- Pair each benchmark item with a manually reviewed literal transcription.
+- Score character error rate, word error rate, omissions, inventions, silent normalization, uncertainty handling, layout fidelity, cost, and latency.
+- Preserve the complete execution provenance for every benchmark run.
+- Keep the current model as a baseline; do not change the application default solely from vendor benchmarks.
 
-- The client submits the full ordered list of Source IDs for one Document.
-- The service validates membership, completeness, duplicates, and authorization/context before writing.
-- All page-number updates occur in one transaction.
+### 8. Migration, Integrity, and Verification
 
-### B. Registry Codes Are Immutable
+- Provide non-destructive upgrade behavior for supported SQLite and PostgreSQL deployments.
+- Backfill only facts that can be derived reliably from existing records.
+- Mark unavailable historical evidence as unavailable rather than fabricating it.
+- Add digest, serialization, header-allowlist, failure-path, compatibility, artifact, export, and UI inspection tests.
+- Run destructive tests only through the repository's required backup-and-restore wrapper.
 
-- Document Type and Person Role codes are stable identifiers.
-- Labels and active state remain mutable.
-- Historical references remain valid when a registry entry is inactive.
+## Out of Scope
 
-### C. No Raw Environment Editor
+- Selecting or declaring a permanent best transcription model.
+- Changing the default transcription model without benchmark evidence and a separate decision.
+- Integrating Azure Document Intelligence, Google Document AI, Transkribus, Mistral OCR, or another OCR provider in V4.2.
+- Generating bounding boxes retroactively for existing transcriptions.
+- Bulk reprocessing the archive.
+- Packet capture, TLS evidence, full unrestricted request/response headers, or credential retention.
+- Storing duplicate base64 source images in request manifests.
+- Guaranteeing byte-identical reproduction from nondeterministic or updated hosted models.
+- Automatic entity extraction, biography generation, or genealogical inference.
+- Replacing the relational database with an event store or content-addressed object store.
+- Destructive renaming or removal of `raw_api_response`.
 
-- `.env` may contain secrets and values that are not safely reloadable.
-- V4.2 exposes only purpose-built forms backed by explicit validation and service methods.
+## Locked Design Decisions
 
-### D. Prompt Editing Is Constrained
+### A. The Original Source Is Primary Evidence
 
-- Prompt maintenance is limited to direct children of the configured prompt directory.
-- Existing Job provenance is never rewritten when a prompt file changes.
-- The UI must distinguish editing the default for future submissions from inspecting historical Job prompts.
+- Original uploaded bytes and their digest remain authoritative.
+- Processing derivatives and outputs are independently identified derived evidence.
+- Future OCR/layout work reuses the original or a documented derivative.
 
-## Decisions Required Before Scope Freeze
+### B. Evidence Is Layered
 
-1. Choose the reorder interaction: move-up/down controls, drag-and-drop, or both.
-2. Decide whether reordering is allowed while the Document has a queued or processing Job.
-3. Define prompt backup, atomic-write, and recovery behavior.
-4. Decide whether prompt creation and deletion are needed or whether V4.2 edits existing prompts only.
-5. Confirm whether registry sort-order maintenance is needed for Person Roles as well as Document Types.
-6. Confirm that settings changes remain local to the current installation and do not require an API surface.
+- Exact transport evidence, SDK-parsed objects, normalized metadata, and transcription text serve different purposes.
+- One representation must not silently stand in for another.
+- UI and export labels name the stored evidence layer.
 
-## Draft Acceptance Criteria
+### C. History Is Append-Only
 
-1. Reordering a Document's Sources produces contiguous page numbers and updates every ordered view consistently.
-2. Invalid, incomplete, duplicate, cross-Document, or stale reorder requests make no changes.
-3. Document Type and Person Role maintenance preserves stable codes and historical references.
-4. Inactive registry entries remain visible on historical records but are excluded from default create selectors.
-5. Prompt edits are restricted to valid markdown files in the configured prompt directory.
-6. A prompt edit affects future Jobs only and leaves stored Job provenance unchanged.
-7. No Settings page exposes secrets or unrestricted filesystem access.
-8. Focused service and UI tests pass without regressing V4.1 workflows.
+- A retry or reprocessing attempt creates new execution evidence.
+- Convenience caches may change, but historical execution output does not.
+- Human revisions remain separate from machine output.
 
-## Scope Freeze Gate
+### D. Capture Is Secret-Safe by Construction
 
-V4.2 implementation should not begin until:
+- Safe headers are allowlisted.
+- Authorization, cookies, API keys, and unrestricted headers are never persisted.
+- Request manifests reference source digests instead of embedding source bytes.
 
-- V4.1 has been used sufficiently to validate priorities.
-- The six open decisions above are resolved.
-- The prompt-write safety policy is documented.
-- The implementation plan is revised from draft to committed delivery plan.
+### E. Derived Artifacts Are Generic and Versioned
+
+- Artifact storage is not limited to bounding boxes.
+- Coordinate metadata declares units, origin, dimensions, and transformations.
+- Provider-specific payloads may be retained without making provider-specific fields the durable application contract.
+
+### F. Existing Evidence Keeps Its Original Meaning
+
+- Existing `raw_api_response` data remains an SDK response snapshot.
+- A migration may label or classify it but may not claim that missing transport data was captured.
+- Historical nulls and absent fields remain distinguishable from new explicitly captured values.
+
+## Data and Compatibility Policy
+
+- All schema changes are additive in V4.2.
+- Existing source files, hashes, transcriptions, revisions, prompts, jobs, and relationships remain valid.
+- Compatibility reads continue to display historical SDK snapshots.
+- Large derived artifacts may be stored outside the database when the database retains a stable reference, digest, media type, and schema identity.
+- JSON evidence must remain portable across SQLite and PostgreSQL.
+- Exports use explicit schema versions so later releases can interpret older packages.
+
+## Acceptance Criteria
+
+1. A new execution can be traced from its source digest through its frozen request manifest, transport response, parsed/normalized data, and derived outputs.
+2. Exact response content is captured before SDK parsing and is clearly distinguished from the existing SDK snapshot.
+3. Failed HTTP calls retain safe provider evidence; calls with no response record that fact explicitly.
+4. Omitted parameters remain distinguishable from explicit values.
+5. No persisted request, header set, UI display, log, or export contains API credentials.
+6. Retrying or reprocessing does not overwrite prior execution evidence.
+7. Historical records remain readable and are not mislabeled as exact transport captures.
+8. A versioned generic artifact can represent OCR/layout JSON and its coordinate system without a provider-specific schema change.
+9. Evidence exports include relationships, schema identities, and digests sufficient for independent integrity checks.
+10. The benchmark protocol can compare the current baseline with another model on the same private corpus and scoring rules.
+11. Additive migrations and focused tests work across the supported persistence model.
+12. All destructive-test runs comply with the backup-and-restore protocol.
 
 ## Related Local References
 
-- [Draft V4.2 Implementation Plan](implementation_plan_v4_2.md)
-- [V4.1 Scope Boundary](../ver4.1/scope_boundary_v4_1.md)
+- [V4.2 Implementation Plan](implementation_plan_v4_2.md)
+- [Digital Evidence and AI Processing Provenance](../invariant/ai_evidence_and_provenance.md)
 - [V4 Architecture](../ver4/architecture_v4.md)
 - [V4 Schema](../ver4/schema_v4.md)
+- [V4 Requirements](../ver4/requirements_v4.md)
+- [Transcription Methodology](../invariant/transcription_methodology.md)