transcription/docs/ver4.2/scope_boundary_v4_2.md

V4.2 Scope Boundary

This document defines the boundary for the digital-evidence and AI-provenance revision that follows V4.1 and precedes the planned V4.3 page-reordering and settings work. V4 remains the architecture baseline; V4.2 makes the existing evidence claims precise and adds a provider-neutral foundation for future processing artifacts.

Purpose

• Align the application with the Digital Evidence and AI Processing Provenance invariant.
• Preserve provider-boundary evidence before SDK parsing can remove unknown fields.
• Make successful and failed processing attempts inspectable without storing secrets.
• Support future OCR and layout outputs without coupling the database to one vendor.
• Establish a repeatable method for comparing transcription models against this archive.

In Scope

1. Evidence Terminology and Existing-Data Compatibility

• Define transport response, router-normalized response, SDK response, normalized metadata, and derived artifact consistently in code, schema documentation, and UI labels.
• Treat existing JobSource.raw_api_response values as historical SDK response snapshots.
• Preserve every existing Job, Source, and JobSource row.
• Use additive migrations and compatibility reads; do not reinterpret previously stored values as exact transport captures.
• Correct the “Page-Level Execution and AI Outputs” rule in docs/ver4/schema_v4.md that currently describes JOB_SOURCE as storing a complete provider response envelope. The corrected rule must identify raw_api_response as an SDK-serialized OpenRouter response snapshot and state that it is neither the exact HTTP body nor the native upstream-provider response.

2. Secret-Safe Request Manifests

• Persist the effective request specification for each page execution without storing credentials or duplicate base64 media.
• Include requested provider/model, routing constraints, prompt content and hash, explicitly supplied parameters, source digest, media type, dimensions when known, and page identity.
• Distinguish an omitted optional parameter from an explicitly supplied null or value.
• Record application, provider-adapter, Python client, and relevant schema versions.
• Use source or derivative references in place of embedded media bytes.

3. Provider-Boundary Response Capture

• Capture the exact HTTP response body before OpenRouter SDK parsing for non-streaming transcription calls.
• Store HTTP status and an explicit allowlist of safe response headers.
• Store router request/generation identifiers and resolved model/provider-routing metadata when exposed.
• Preserve the current parsed SDK snapshot and normalized metadata where useful.
• Keep exact body, parsed representation, and normalized fields distinguishable.

4. Failure Evidence and Timing

• Create or update a page execution record for every attempted provider call.
• Persist safe response evidence for non-success HTTP responses.
• Distinguish HTTP response failures, connection failures, local timeouts, response-validation failures, and transcription-quality failures.
• Store execution start/end times or duration using a clearly defined clock policy.
• Do not collapse a provider error body into only a generic user-facing message.

5. Generic Processing Artifacts

• Add a provider-neutral representation for versioned derived artifacts.
• Support inline JSON and externally stored payloads with a digest and stable reference.
• Record artifact type, format, schema/version, producer/version, source, producing execution, and creation time.
• Define coordinate-system metadata sufficient for word, line, block, or page geometry.
• Permit future OCR/layout/confidence results without implementing a vendor-specific table for each provider.

6. Evidence Inspection and Export

• Expand Source Detail and/or Job Detail to identify the evidence layer being displayed.
• Provide readable JSON inspection for request manifests, transport metadata, parsed responses, normalized metadata, and derived artifacts.
• Provide a safe export containing evidence content or references, relationships, schema versions, and digests.
• Clearly label evidence that was not captured for historical records.
• Do not display or export credentials, unrestricted headers, or embedded base64 source media.

7. Representative-Corpus Benchmark Protocol

• Define a private benchmark manifest referencing source digests rather than duplicating archival media.
• Include representative printed, typed, handwritten, degraded, tabular, and spatially complex pages.
• Pair each benchmark item with a manually reviewed literal transcription.
• Score character error rate, word error rate, omissions, inventions, silent normalization, uncertainty handling, layout fidelity, cost, and latency.
• Preserve the complete execution provenance for every benchmark run.
• Keep the current model as a baseline; do not change the application default solely from vendor benchmarks.

8. Migration, Integrity, and Verification

• Provide non-destructive upgrade behavior for supported SQLite and PostgreSQL deployments.
• Backfill only facts that can be derived reliably from existing records.
• Mark unavailable historical evidence as unavailable rather than fabricating it.
• Add digest, serialization, header-allowlist, failure-path, compatibility, artifact, export, and UI inspection tests.
• Run destructive tests only through the repository's required backup-and-restore wrapper.

Out of Scope

• Selecting or declaring a permanent best transcription model.
• Changing the default transcription model without benchmark evidence and a separate decision.
• Integrating Azure Document Intelligence, Google Document AI, Transkribus, Mistral OCR, or another OCR provider in V4.2.
• Generating bounding boxes retroactively for existing transcriptions.
• Bulk reprocessing the archive.
• Packet capture, TLS evidence, full unrestricted request/response headers, or credential retention.
• Storing duplicate base64 source images in request manifests.
• Guaranteeing byte-identical reproduction from nondeterministic or updated hosted models.
• Automatic entity extraction, biography generation, or genealogical inference.
• Replacing the relational database with an event store or content-addressed object store.
• Destructive renaming or removal of raw_api_response.

Locked Design Decisions

A. The Original Source Is Primary Evidence

• Original uploaded bytes and their digest remain authoritative.
• Processing derivatives and outputs are independently identified derived evidence.
• Future OCR/layout work reuses the original or a documented derivative.

B. Evidence Is Layered

• Exact transport evidence, SDK-parsed objects, normalized metadata, and transcription text serve different purposes.
• One representation must not silently stand in for another.
• UI and export labels name the stored evidence layer.

C. History Is Append-Only

• A retry or reprocessing attempt creates new execution evidence.
• Convenience caches may change, but historical execution output does not.
• Human revisions remain separate from machine output.

D. Capture Is Secret-Safe by Construction

• Safe headers are allowlisted.
• Authorization, cookies, API keys, and unrestricted headers are never persisted.
• Request manifests reference source digests instead of embedding source bytes.

E. Derived Artifacts Are Generic and Versioned

• Artifact storage is not limited to bounding boxes.
• Coordinate metadata declares units, origin, dimensions, and transformations.
• Provider-specific payloads may be retained without making provider-specific fields the durable application contract.

F. Existing Evidence Keeps Its Original Meaning

• Existing raw_api_response data remains an SDK response snapshot.
• A migration may label or classify it but may not claim that missing transport data was captured.
• Historical nulls and absent fields remain distinguishable from new explicitly captured values.

Data and Compatibility Policy

• All schema changes are additive in V4.2.
• Existing source files, hashes, transcriptions, revisions, prompts, jobs, and relationships remain valid.
• Compatibility reads continue to display historical SDK snapshots.
• Large derived artifacts may be stored outside the database when the database retains a stable reference, digest, media type, and schema identity.
• JSON evidence must remain portable across SQLite and PostgreSQL.
• Exports use explicit schema versions so later releases can interpret older packages.

Acceptance Criteria

1. A new execution can be traced from its source digest through its frozen request manifest, transport response, parsed/normalized data, and derived outputs.
2. Exact response content is captured before SDK parsing and is clearly distinguished from the existing SDK snapshot.
3. Failed HTTP calls retain safe provider evidence; calls with no response record that fact explicitly.
4. Omitted parameters remain distinguishable from explicit values.
5. No persisted request, header set, UI display, log, or export contains API credentials.
6. Retrying or reprocessing does not overwrite prior execution evidence.
7. Historical records remain readable and are not mislabeled as exact transport captures.
8. A versioned generic artifact can represent OCR/layout JSON and its coordinate system without a provider-specific schema change.
9. Evidence exports include relationships, schema identities, and digests sufficient for independent integrity checks.
10. The benchmark protocol can compare the current baseline with another model on the same private corpus and scoring rules.
11. Additive migrations and focused tests work across the supported persistence model.
12. All destructive-test runs comply with the backup-and-restore protocol.

Related Local References

• V4.2 Implementation Plan
• Digital Evidence and AI Processing Provenance
• V4 Architecture
• V4 Schema
• V4 Requirements
• Transcription Methodology