V4.1 major revision to docs. Removed all obsolete documents, updated v4.2 implementation scope and plan.

docs/ver4.2/scope_boundary_v4_2.md

@@ -1,130 +1,163 @@
-# Draft V4.2 Scope Boundary
+# V4.2 Scope Boundary
 
-This document defines the proposed boundary for the second incremental revision to Version 4. It is intentionally a draft until V4.1 has been used and the remaining workflows have been validated.
+This document defines the boundary for the digital-evidence and AI-provenance revision that follows V4.1 and precedes the planned V4.3 page-reordering and settings work. V4 remains the architecture baseline; V4.2 makes the existing evidence claims precise and adds a provider-neutral foundation for future processing artifacts.
 
 ## Purpose
 
-- Allow correction of Source page order after import.
-- Provide a constrained Settings area for safe maintenance of selected application-managed configuration.
-- Avoid exposing secrets, restart-sensitive settings, or unrestricted filesystem editing through the UI.
+- Align the application with the [Digital Evidence and AI Processing Provenance invariant](../invariant/ai_evidence_and_provenance.md).
+- Preserve provider-boundary evidence before SDK parsing can remove unknown fields.
+- Make successful and failed processing attempts inspectable without storing secrets.
+- Support future OCR and layout outputs without coupling the database to one vendor.
+- Establish a repeatable method for comparing transcription models against this archive.
 
-## Proposed In Scope
+## In Scope
 
-### 1. Source Page Reordering
+### 1. Evidence Terminology and Existing-Data Compatibility
 
-- Allow Sources within one Document to be reordered after import.
-- Present the current order using page number and a recognizable source label or preview.
-- Persist the complete intended order atomically.
-- Renumber the affected Document's Sources to a contiguous sequence beginning at 1.
-- Keep all Sources attached to their existing Document.
-- Ensure transcription rendering and previous/next navigation use the updated order.
-- Detect stale or invalid reorder submissions and fail without partial mutation.
+- Define transport response, router-normalized response, SDK response, normalized metadata, and derived artifact consistently in code, schema documentation, and UI labels.
+- Treat existing `JobSource.raw_api_response` values as historical SDK response snapshots.
+- Preserve every existing `Job`, `Source`, and `JobSource` row.
+- Use additive migrations and compatibility reads; do not reinterpret previously stored values as exact transport captures.
+- Correct the “Page-Level Execution and AI Outputs” rule in `docs/ver4/schema_v4.md` that currently describes `JOB_SOURCE` as storing a complete provider response envelope. The corrected rule must identify `raw_api_response` as an SDK-serialized OpenRouter response snapshot and state that it is neither the exact HTTP body nor the native upstream-provider response.
 
-### 2. Settings Navigation
+### 2. Secret-Safe Request Manifests
 
-- Add a Settings entry to application navigation.
-- Provide separate, clearly described settings areas rather than a raw configuration editor.
-- Restrict V4.2 settings to application-managed values that can be validated and safely changed at runtime.
+- Persist the effective request specification for each page execution without storing credentials or duplicate base64 media.
+- Include requested provider/model, routing constraints, prompt content and hash, explicitly supplied parameters, source digest, media type, dimensions when known, and page identity.
+- Distinguish an omitted optional parameter from an explicitly supplied null or value.
+- Record application, provider-adapter, Python client, and relevant schema versions.
+- Use source or derivative references in place of embedded media bytes.
 
-### 3. Document Type Maintenance
+### 3. Provider-Boundary Response Capture
 
-- List active and inactive Document Types.
-- Add new types with a stable unique code and user-facing label.
-- Edit mutable labels and sort order.
-- Activate or deactivate types without invalidating historical Documents.
-- Do not allow changing a stable code after creation.
-- Do not delete types that are referenced by Documents.
+- Capture the exact HTTP response body before OpenRouter SDK parsing for non-streaming transcription calls.
+- Store HTTP status and an explicit allowlist of safe response headers.
+- Store router request/generation identifiers and resolved model/provider-routing metadata when exposed.
+- Preserve the current parsed SDK snapshot and normalized metadata where useful.
+- Keep exact body, parsed representation, and normalized fields distinguishable.
 
-### 4. Person Role Maintenance
+### 4. Failure Evidence and Timing
 
-- List active and inactive Person Roles.
-- Add new roles with a stable unique code and user-facing label.
-- Edit mutable labels.
-- Activate or deactivate roles without invalidating historical links.
-- Do not allow changing a stable code after creation.
-- Do not delete roles that are referenced by document-person links.
+- Create or update a page execution record for every attempted provider call.
+- Persist safe response evidence for non-success HTTP responses.
+- Distinguish HTTP response failures, connection failures, local timeouts, response-validation failures, and transcription-quality failures.
+- Store execution start/end times or duration using a clearly defined clock policy.
+- Do not collapse a provider error body into only a generic user-facing message.
 
-### 5. Prompt Maintenance
+### 5. Generic Processing Artifacts
 
-- List prompt markdown files from the configured prompt directory.
-- View a prompt with a concise explanation of its purpose and use.
-- Edit an existing prompt as plain markdown text.
-- Validate the filename boundary and reject empty prompt content.
-- Save changes explicitly and report filesystem failures.
-- Preserve submission-time prompt text and hash already frozen on existing Jobs.
-- Define a safe-write and recovery approach before this feature is considered final scope.
+- Add a provider-neutral representation for versioned derived artifacts.
+- Support inline JSON and externally stored payloads with a digest and stable reference.
+- Record artifact type, format, schema/version, producer/version, source, producing execution, and creation time.
+- Define coordinate-system metadata sufficient for word, line, block, or page geometry.
+- Permit future OCR/layout/confidence results without implementing a vendor-specific table for each provider.
 
-## Proposed Out of Scope
+### 6. Evidence Inspection and Export
 
-- Viewing or editing raw `.env` files.
-- Displaying or changing provider API keys and other secrets.
-- Editing host, port, database connection, upload paths, or other restart-sensitive runtime settings.
-- Arbitrary file browsing or arbitrary prompt paths.
-- Runtime theme/CSS editing.
-- Installing themes or plugins.
-- Source movement between Documents as part of reordering.
-- Automatic ordering based on filenames, OCR, or image content.
-- FamilySearch API synchronization.
-- A generic external-reference registry.
-- Ancestry references and Google Maps links.
+- Expand Source Detail and/or Job Detail to identify the evidence layer being displayed.
+- Provide readable JSON inspection for request manifests, transport metadata, parsed responses, normalized metadata, and derived artifacts.
+- Provide a safe export containing evidence content or references, relationships, schema versions, and digests.
+- Clearly label evidence that was not captured for historical records.
+- Do not display or export credentials, unrestricted headers, or embedded base64 source media.
 
-## Proposed Design Decisions
+### 7. Representative-Corpus Benchmark Protocol
 
-### A. Reordering Is Set-Based
+- Define a private benchmark manifest referencing source digests rather than duplicating archival media.
+- Include representative printed, typed, handwritten, degraded, tabular, and spatially complex pages.
+- Pair each benchmark item with a manually reviewed literal transcription.
+- Score character error rate, word error rate, omissions, inventions, silent normalization, uncertainty handling, layout fidelity, cost, and latency.
+- Preserve the complete execution provenance for every benchmark run.
+- Keep the current model as a baseline; do not change the application default solely from vendor benchmarks.
 
-- The client submits the full ordered list of Source IDs for one Document.
-- The service validates membership, completeness, duplicates, and authorization/context before writing.
-- All page-number updates occur in one transaction.
+### 8. Migration, Integrity, and Verification
 
-### B. Registry Codes Are Immutable
+- Provide non-destructive upgrade behavior for supported SQLite and PostgreSQL deployments.
+- Backfill only facts that can be derived reliably from existing records.
+- Mark unavailable historical evidence as unavailable rather than fabricating it.
+- Add digest, serialization, header-allowlist, failure-path, compatibility, artifact, export, and UI inspection tests.
+- Run destructive tests only through the repository's required backup-and-restore wrapper.
 
-- Document Type and Person Role codes are stable identifiers.
-- Labels and active state remain mutable.
-- Historical references remain valid when a registry entry is inactive.
+## Out of Scope
 
-### C. No Raw Environment Editor
+- Selecting or declaring a permanent best transcription model.
+- Changing the default transcription model without benchmark evidence and a separate decision.
+- Integrating Azure Document Intelligence, Google Document AI, Transkribus, Mistral OCR, or another OCR provider in V4.2.
+- Generating bounding boxes retroactively for existing transcriptions.
+- Bulk reprocessing the archive.
+- Packet capture, TLS evidence, full unrestricted request/response headers, or credential retention.
+- Storing duplicate base64 source images in request manifests.
+- Guaranteeing byte-identical reproduction from nondeterministic or updated hosted models.
+- Automatic entity extraction, biography generation, or genealogical inference.
+- Replacing the relational database with an event store or content-addressed object store.
+- Destructive renaming or removal of `raw_api_response`.
 
-- `.env` may contain secrets and values that are not safely reloadable.
-- V4.2 exposes only purpose-built forms backed by explicit validation and service methods.
+## Locked Design Decisions
 
-### D. Prompt Editing Is Constrained
+### A. The Original Source Is Primary Evidence
 
-- Prompt maintenance is limited to direct children of the configured prompt directory.
-- Existing Job provenance is never rewritten when a prompt file changes.
-- The UI must distinguish editing the default for future submissions from inspecting historical Job prompts.
+- Original uploaded bytes and their digest remain authoritative.
+- Processing derivatives and outputs are independently identified derived evidence.
+- Future OCR/layout work reuses the original or a documented derivative.
 
-## Decisions Required Before Scope Freeze
+### B. Evidence Is Layered
 
-1. Choose the reorder interaction: move-up/down controls, drag-and-drop, or both.
-2. Decide whether reordering is allowed while the Document has a queued or processing Job.
-3. Define prompt backup, atomic-write, and recovery behavior.
-4. Decide whether prompt creation and deletion are needed or whether V4.2 edits existing prompts only.
-5. Confirm whether registry sort-order maintenance is needed for Person Roles as well as Document Types.
-6. Confirm that settings changes remain local to the current installation and do not require an API surface.
+- Exact transport evidence, SDK-parsed objects, normalized metadata, and transcription text serve different purposes.
+- One representation must not silently stand in for another.
+- UI and export labels name the stored evidence layer.
 
-## Draft Acceptance Criteria
+### C. History Is Append-Only
 
-1. Reordering a Document's Sources produces contiguous page numbers and updates every ordered view consistently.
-2. Invalid, incomplete, duplicate, cross-Document, or stale reorder requests make no changes.
-3. Document Type and Person Role maintenance preserves stable codes and historical references.
-4. Inactive registry entries remain visible on historical records but are excluded from default create selectors.
-5. Prompt edits are restricted to valid markdown files in the configured prompt directory.
-6. A prompt edit affects future Jobs only and leaves stored Job provenance unchanged.
-7. No Settings page exposes secrets or unrestricted filesystem access.
-8. Focused service and UI tests pass without regressing V4.1 workflows.
+- A retry or reprocessing attempt creates new execution evidence.
+- Convenience caches may change, but historical execution output does not.
+- Human revisions remain separate from machine output.
 
-## Scope Freeze Gate
+### D. Capture Is Secret-Safe by Construction
 
-V4.2 implementation should not begin until:
+- Safe headers are allowlisted.
+- Authorization, cookies, API keys, and unrestricted headers are never persisted.
+- Request manifests reference source digests instead of embedding source bytes.
 
-- V4.1 has been used sufficiently to validate priorities.
-- The six open decisions above are resolved.
-- The prompt-write safety policy is documented.
-- The implementation plan is revised from draft to committed delivery plan.
+### E. Derived Artifacts Are Generic and Versioned
+
+- Artifact storage is not limited to bounding boxes.
+- Coordinate metadata declares units, origin, dimensions, and transformations.
+- Provider-specific payloads may be retained without making provider-specific fields the durable application contract.
+
+### F. Existing Evidence Keeps Its Original Meaning
+
+- Existing `raw_api_response` data remains an SDK response snapshot.
+- A migration may label or classify it but may not claim that missing transport data was captured.
+- Historical nulls and absent fields remain distinguishable from new explicitly captured values.
+
+## Data and Compatibility Policy
+
+- All schema changes are additive in V4.2.
+- Existing source files, hashes, transcriptions, revisions, prompts, jobs, and relationships remain valid.
+- Compatibility reads continue to display historical SDK snapshots.
+- Large derived artifacts may be stored outside the database when the database retains a stable reference, digest, media type, and schema identity.
+- JSON evidence must remain portable across SQLite and PostgreSQL.
+- Exports use explicit schema versions so later releases can interpret older packages.
+
+## Acceptance Criteria
+
+1. A new execution can be traced from its source digest through its frozen request manifest, transport response, parsed/normalized data, and derived outputs.
+2. Exact response content is captured before SDK parsing and is clearly distinguished from the existing SDK snapshot.
+3. Failed HTTP calls retain safe provider evidence; calls with no response record that fact explicitly.
+4. Omitted parameters remain distinguishable from explicit values.
+5. No persisted request, header set, UI display, log, or export contains API credentials.
+6. Retrying or reprocessing does not overwrite prior execution evidence.
+7. Historical records remain readable and are not mislabeled as exact transport captures.
+8. A versioned generic artifact can represent OCR/layout JSON and its coordinate system without a provider-specific schema change.
+9. Evidence exports include relationships, schema identities, and digests sufficient for independent integrity checks.
+10. The benchmark protocol can compare the current baseline with another model on the same private corpus and scoring rules.
+11. Additive migrations and focused tests work across the supported persistence model.
+12. All destructive-test runs comply with the backup-and-restore protocol.
 
 ## Related Local References
 
-- [Draft V4.2 Implementation Plan](implementation_plan_v4_2.md)
-- [V4.1 Scope Boundary](../ver4.1/scope_boundary_v4_1.md)
+- [V4.2 Implementation Plan](implementation_plan_v4_2.md)
+- [Digital Evidence and AI Processing Provenance](../invariant/ai_evidence_and_provenance.md)
 - [V4 Architecture](../ver4/architecture_v4.md)
 - [V4 Schema](../ver4/schema_v4.md)
+- [V4 Requirements](../ver4/requirements_v4.md)
+- [Transcription Methodology](../invariant/transcription_methodology.md)