transcription/docs/invariant/ai_evidence_and_provenance.md

Digital Evidence and AI Processing Provenance (Invariant)

1. Purpose

This document defines non-negotiable evidence and provenance rules for the transcription application.

The application exists to preserve historical source material and produce useful transcriptions without losing the ability to inspect, reinterpret, or reprocess the evidence later. Provider integrations, model names, schemas, and user interfaces may change; the principles below must remain true.

2. Evidence Model

The application distinguishes five kinds of information:

1. Source evidence: the original uploaded media and the facts needed to identify and verify it.
2. Execution specification: the frozen instructions, parameters, source identity, and software context for one processing attempt.
3. Transport evidence: the response received at the application/provider boundary, including safe protocol metadata.
4. Normalized data: selected fields extracted for search, display, accounting, and workflow behavior.
5. Derived artifacts: outputs produced from source evidence, such as transcription text, OCR geometry, confidence data, layout analysis, or entity extraction.

Normalized data and derived artifacts never replace source or transport evidence.

3. Core Invariants

3.1 Original Source Preservation

1. The original uploaded bytes are the primary evidence and must be preserved without transformation.
2. Each source must have a cryptographic content digest, byte size, and stable identity.
3. Processing may use transformed derivatives, but those derivatives must not overwrite the original.
4. A derivative used for processing must record its relationship to the original, its transformation, and its own digest.
5. Moving or renaming a stored file must not change its evidence identity.

3.2 Append-Only Processing History

1. Every processing attempt must have a distinct execution record, whether it succeeds, partially succeeds, times out, or fails.
2. A later attempt must not overwrite the evidence from an earlier attempt.
3. A convenient “latest transcription” value may be maintained as a cache or projection, but it is not the authoritative execution history.
4. Human revisions must remain distinguishable from all machine-generated outputs.
5. Reprocessing a source must create new evidence rather than rewriting historical evidence.

3.3 Frozen Execution Specification

Each execution must preserve enough information to understand what the application asked the processor to do:

1. Requested provider, model, and provider-routing constraints.
2. Full effective system and user instructions.
3. Prompt asset name and content digest when a prompt asset is used.
4. Every explicitly supplied generation or processing parameter.
5. Whether an optional parameter was explicitly set or omitted.
6. Source and derivative digests, media type, dimensions or page geometry when known, and page identity.
7. A secret-safe representation of the request structure.
8. Application, provider-adapter, and client-library versions sufficient to interpret the execution.

The execution specification must not contain credentials, authorization headers, secret query values, or unnecessary duplicate source binaries.

3.4 Evidence-Layer Terminology

The following terms are not interchangeable:

• Transport response: the status, safe headers, and exact response body received by the application at its HTTP boundary.
• Router-normalized response: a response transformed by an intermediary into its common schema.
• SDK-parsed response: an object created when a client library validates or filters a response.
• Normalized metadata: application-selected fields derived from a response.
• Native provider response: the upstream provider's own response before any intermediary transformation.

The application and its documentation must identify which layer is stored. A response must not be described as “raw,” “complete,” or “native” without naming the boundary at which that claim is true.

3.5 Transport Evidence

1. Preserve the exact successful response body received at the application's transport boundary before SDK model parsing can discard unknown fields.
2. Preserve the response status and an allowlisted set of non-secret headers needed for correlation, content interpretation, rate-limit diagnosis, or audit.
3. Preserve provider/router request and generation identifiers when available.
4. Preserve safe response evidence for unsuccessful calls when a response was received.
5. Record explicitly when no response was received, such as a local timeout or connection failure.
6. Retain parsed and normalized forms only as additional representations of the preserved response.

Wire-level packet capture, TLS session data, credentials, and unrestricted headers are neither required nor permitted.

These requirements apply to executions performed after transport capture is implemented. For earlier executions, the absence of transport evidence must be represented explicitly. An SDK snapshot or normalized record must never be relabeled or backfilled as transport evidence.

3.6 Derived Artifact Provenance

1. Every derived artifact must identify its source evidence and producing execution.
2. Each artifact must declare its semantic type, media/serialization format, schema name and version, producer, producer version, and creation time.
3. Artifact content must be stored directly or referenced by a stable path or object identifier and protected by a cryptographic digest.
4. Coordinates must declare their coordinate system, units, origin, page/image dimensions, and transformation history.
5. Confidence values must identify the producer and scope to which they apply; values from different producers must not be treated as directly comparable without validation.
6. Provider-specific payloads may be retained, but durable application behavior must not depend on undocumented provider fields.

This model must accommodate future OCR text, word or line polygons, layout regions, confidence data, alternate transcriptions, and structured extraction without adding a dedicated column for every possible feature.

3.7 Integrity and Auditability

1. Stored evidence must be exportable with enough identifiers and metadata to verify relationships and digests outside the application.
2. Evidence mutation, deletion, and retention behavior must be explicit and testable.
3. Schema upgrades must preserve existing evidence and its original meaning.
4. Backfills must be identified as backfills; they must not imply that previously uncaptured evidence existed.
5. Integrity verification must distinguish a missing file, digest mismatch, unavailable external artifact, and malformed metadata.

3.8 Security and Privacy

1. API keys, authorization headers, cookies, and credentials must never be persisted as provenance.
2. Persist only headers and metadata fields that appear on an explicit allowlist of known-safe fields. Discard all other fields before storage; never persist an unrestricted capture and attempt to redact it afterward.
3. Request manifests should reference source content by identity instead of duplicating base64 source data.
4. Diagnostic displays and exports must avoid exposing secrets or machine-local details that are not necessary for evidence interpretation.

4. Reproducibility Limits

Provenance supports explanation, comparison, and best-effort reproduction; it does not guarantee identical output.

Identical requests may produce different results because of model updates, provider routing, nondeterministic computation, undocumented defaults, safety systems, or retired endpoints. The application must preserve whether a parameter was omitted rather than pretending to know the provider default used at that time.

Likewise, preserving a general vision-model response does not create OCR coordinates that were never returned. Future coordinate extraction remains possible because the original source evidence is preserved and can be processed again by a suitable system.

5. Model Evaluation Policy

Model selection must be based on a representative sample of the actual archive rather than vendor claims alone.

Evaluation should:

1. Use manually reviewed reference transcriptions following the project's Transcription Methodology.
2. Represent printed, typed, handwritten, degraded, tabular, multilingual, and spatially complex material present in the archive.
3. Measure character and word error rates where appropriate.
4. Separately record silent corrections, invented text, omitted text, uncertainty handling, layout fidelity, cost, and latency.
5. Preserve the exact model, endpoint or route, parameters, prompt, source digest, and scoring method for every comparison.
6. Treat model rankings as corpus- and version-specific, not permanent declarations of a universal “best” model.

Benchmark material containing family records remains private application data unless explicitly approved for publication.

6. Ownership and Change Policy

1. Versioned architecture, schema, scope, and implementation documents define how a release satisfies this invariant.
2. Provider adapters own the capture of provider-boundary evidence.
3. Services own validation, persistence, retention, and export behavior.
4. UI pages may inspect evidence through service contracts but do not define evidence semantics.
5. If implementation conflicts with this invariant, either correct the implementation or explicitly revise this document before accepting the behavior.
6. Revisions to this document require deliberate review because they change the long-term preservation contract.

7. Related Invariants

• Historical Document Transcription Design Intent
• Transcription Methodology & Style Guide
• UI Style Guide