V4.2 Updated what ai_raw_response data is being captured. The changes were more extensive than I expected.

This commit is contained in:
Jim Lancaster
2026-08-14 07:21:15 -05:00
parent 28811d79ce
commit 6bd4cbb0a7
29 changed files with 2091 additions and 130 deletions
+2 -1
View File
@@ -57,4 +57,5 @@ Each page contract contains:
## Current Baseline
These contracts describe the V4 baseline with completed V4.1 UI behavior. Planned V4.2 evidence/provenance changes and draft V4.3 Settings/page-reordering changes are not described as current behavior.
These contracts describe the V4 baseline with completed V4.1 behavior and V4.2 evidence/provenance behavior.
Draft V4.3 Settings/page-reordering changes are not described as current behavior.
+3 -6
View File
@@ -58,9 +58,10 @@ Jobs manages transcription processing runs. A Job belongs to one Document, links
## Resubmit Behavior
- The page shows current status and failed Source count.
- The page explicitly states: `Resubmit queues only failed linked sources. New results overwrite prior page-level results.`
- The page explains that resubmission queues failed linked Sources while preserving immutable prior attempt evidence.
- The service blocks submission while processing is active or when no failed Sources exist.
- Current behavior updates the existing page-level result when new output arrives.
- `JobSource` remains the latest compatibility projection, while every provider call appends an `ExecutionAttempt`.
- The latest successful `Source.raw_transcription` projection remains available while a retry is pending or fails.
- Success reports the number of resubmitted Sources and returns to Job Detail.
## Delete Behavior
@@ -88,7 +89,3 @@ Jobs manages transcription processing runs. A Job belongs to one Document, links
- `tests/ui/test_jobs_page.py`
- `tests/services/test_job_service.py`
- `tests/services/test_store.py`
## Planned Change
V4.2 replaces update-in-place retry evidence with append-only processing attempts and adds exact transport evidence. Until implemented, the current overwrite behavior must be labeled accurately rather than described as archival history. See the [V4.2 scope](../../ver4.2/scope_boundary_v4_2.md).
+6 -3
View File
@@ -42,9 +42,13 @@ The list accepts optional `document_id` and `job_id` query parameters. Document
## Provider Evidence
- Provider Evidence is associated with the latest JobSource execution.
- AI Metadata and the current `raw_api_response` value are displayed as expandable formatted JSON.
- New attempts display separate expandable Request Manifest, Transport Response, OpenRouter SDK Response Snapshot,
Normalized Metadata, Software Context, and Derived Artifacts sections.
- Historical `raw_api_response` values are labeled as OpenRouter SDK response snapshots.
- Missing evidence has an explicit empty state.
- Under the current V4 implementation, `raw_api_response` is an OpenRouter SDK response snapshot, not an exact HTTP or native upstream-provider response.
- Historical executions explicitly state that exact transport evidence was not captured.
- **Export Evidence** downloads a versioned package containing source identity, attempts, artifacts, relationships,
schema versions, and integrity digests without source binaries, credentials, or machine-local source paths.
## Revision Behavior
@@ -82,5 +86,4 @@ The list accepts optional `document_id` and `job_id` query parameters. Document
## Planned Changes
- V4.2 will rename and separate evidence layers, add exact OpenRouter transport capture, and preserve append-only attempts. See the [V4.2 scope](../../ver4.2/scope_boundary_v4_2.md).
- Source page reordering is deferred to the [draft V4.3 scope](../../ver4.3/scope_boundary_v4_3.md).
+17
View File
@@ -210,6 +210,23 @@ Enforce exactly one content location: inline payload or external reference. An e
These decisions must be settled before their corresponding implementation phase; they do not weaken the invariant or expand V4.2 into live OCR integration.
## Resolved Implementation Decisions
1. `JobSource` remains queue linkage and a compatibility projection; immutable retries use a one-to-many
`ExecutionAttempt` model with a unique `(job_id, source_id, attempt_number)` constraint.
2. Exact OpenRouter response bytes remain database values for V4.2. Generic artifacts use inline canonical JSON up
to 1 MiB by default and constrained, atomically written external files above that threshold.
3. Request manifests use `transcription-canonical-json-v1`: UTF-8 JSON with sorted keys, compact separators,
preserved Unicode, and non-finite numbers rejected.
4. Safe response headers are explicitly allowlisted in the evidence contract; all others are discarded before
persistence.
5. Software identity records the package version, optional `TRANSCRIPTION_COMMIT`, adapter contract version,
OpenRouter SDK version, and Python version.
6. The artifact root defaults to `data/artifacts` and stores source-scoped relative references.
7. Evidence exports include source identity and digest by reference, not original source binaries.
8. The benchmark manifest is private and digest-referenced. Corpus size remains archive-dependent, but every run
uses preserved execution-attempt identity and the fixed literal scoring contract.
## Done When
- Every V4.2 acceptance criterion is satisfied by focused tests or an explicit demonstration.
+12 -3
View File
@@ -26,7 +26,10 @@ This document describes the production architecture of the document transcriptio
- Isolate page failures so multi-page jobs can complete with partial success.
- Operate across supported platforms through Python-based application and maintenance tooling.
V4.2 extends this baseline with exact OpenRouter transport evidence and provider-neutral derived-artifact provenance. See the [V4.2 Scope Boundary](../ver4.2/scope_boundary_v4_2.md).
V4.2 extends this baseline with immutable execution attempts, exact OpenRouter transport evidence, safe
versioned exports, and provider-neutral derived-artifact provenance. `JobSource` remains the mutable queue and
compatibility projection; `ExecutionAttempt` is the authoritative append-only processing history. See the
[V4.2 Scope Boundary](../ver4.2/scope_boundary_v4_2.md).
## Technical Stack
@@ -135,8 +138,11 @@ Responsibilities:
1. User uploads one or more images for a `Document`.
2. System stores files, hashes them, creates ordered `Source` rows, and creates a `Job`.
3. Worker claims the job, marks it `processing`, and executes page calls concurrently.
4. Each page writes a `JobSource` result with machine output, normalized metadata, and an SDK-serialized OpenRouter response snapshot.
5. Aggregate status becomes `completed`, `partial_success`, or `failed`.
4. Each provider call appends an `ExecutionAttempt` with its request manifest, transport evidence, SDK snapshot,
normalized metadata, timing, and outcome.
5. The linked `JobSource` is updated as a compatibility projection, and a successful attempt updates the
`Source.raw_transcription` latest-success projection.
6. Aggregate status becomes `completed`, `partial_success`, or `failed`.
### 2. Document-Person Relationship Management
@@ -160,6 +166,9 @@ Responsibilities:
- Human corrections occur only in `Source.revised_text`.
- Prompt and parameter provenance is frozen on `Job` at submission time.
- The SDK-serialized OpenRouter response snapshot is stored on `JobSource` for each successful page execution.
- Every V4.2 provider call appends a distinct `ExecutionAttempt`; retries never rewrite earlier attempts.
- Exact response bytes identify the OpenRouter HTTP boundary and are not labeled as native upstream-provider JSON.
- Generic `ProcessingArtifact` records use versioned schemas, digests, and one inline or external content location.
- `DocumentPerson` links are unique for `(document_id, person_id, role_id)`.
- Relationship mutations are deterministic and set-based.
- `DocumentType.code` is stable; `DocumentType.label` may evolve.
+1 -1
View File
@@ -9,7 +9,7 @@ This document defines the baseline requirements for the document transcription s
| REQ-0 | System | Provide end-to-end multi-page document transcription with persistent, inspectable async job states. | demonstration |
| REQ-1 | Functional | Allow users to upload one or more images as ordered `Source` pages under a `Document`. | test |
| REQ-2 | Functional | Process page transcription asynchronously using an `asyncio` worker pool bounded by rate limits. | test |
| REQ-3 | Functional | Persist submission-time prompt configuration and full page-level provider response evidence for every job execution. | test |
| REQ-3 | Functional | Persist submission-time request provenance and accurately labeled page-level SDK evidence; V4.2 adds exact OpenRouter-boundary transport evidence for new attempts. | test |
| REQ-4 | Functional | Support job states `queued`, `processing`, `completed`, `partial_success`, and `failed`, plus page states `pending`, `transcribed`, and `failed`. | inspection |
| REQ-5 | Functional | Allow users to manage historical `Person` records and link multiple people per role to a `Document`. | test |
| REQ-6 | Functional | Support an extensible role taxonomy for document-person relationships. | inspection |
+58 -1
View File
@@ -109,6 +109,46 @@ TEXT error_detail
TIMESTAMPTZ executed_at
}
EXECUTION_ATTEMPT {
UUID id PK
UUID job_source_id FK
UUID job_id FK
UUID source_id FK
INTEGER attempt_number
VARCHAR status
JSONB request_manifest
TEXT request_manifest_sha256
INTEGER transport_status_code
BINARY transport_body
JSONB transport_safe_headers
JSONB sdk_response_snapshot
JSONB normalized_metadata
JSONB software_context
TEXT raw_transcription
TEXT failure_phase
TIMESTAMPTZ started_at
TIMESTAMPTZ finished_at
INTEGER duration_ms
}
PROCESSING_ARTIFACT {
UUID id PK
UUID source_id FK
UUID execution_attempt_id FK
TEXT artifact_type
TEXT media_type
TEXT schema_name
TEXT schema_version
TEXT producer
TEXT producer_version
JSONB inline_payload
TEXT external_reference
TEXT payload_sha256
BIGINT byte_size
JSONB coordinate_metadata
TIMESTAMPTZ created_at
}
DOCUMENT_TYPE ||--o{ DOCUMENT : classifies
DOCUMENT ||--o{ DOCUMENT_PERSON : has_people
PERSON ||--o{ DOCUMENT_PERSON : appears_in
@@ -117,6 +157,9 @@ DOCUMENT ||--o{ JOB : has_jobs
DOCUMENT ||--o{ SOURCE : contains_pages
JOB ||--o{ JOB_SOURCE : executes
SOURCE ||--o{ JOB_SOURCE : processed_in
JOB_SOURCE ||--o{ EXECUTION_ATTEMPT : projects
SOURCE ||--o{ PROCESSING_ARTIFACT : derives
EXECUTION_ATTEMPT ||--o{ PROCESSING_ARTIFACT : produces
```
## Domain Invariants and Provenance Rules
@@ -125,9 +168,23 @@ SOURCE ||--o{ JOB_SOURCE : processed_in
- Every single page execution by an AI model produces a dedicated `JOB_SOURCE` record.
- Every `JOB` stores the frozen prompt identifier, prompt text, and hyperparameters used at submission time.
- Every `JOB_SOURCE` stores the complete provider response envelope and page-level operational metadata.
- `JOB_SOURCE.raw_api_response` is a compatibility projection containing an SDK-serialized OpenRouter response
snapshot. It is neither the exact HTTP body nor the native upstream-provider response.
- Every new provider call creates an immutable `EXECUTION_ATTEMPT` containing the frozen request manifest,
exact OpenRouter-boundary response bytes when received, safe transport metadata, SDK snapshot, normalized
metadata, timing, and outcome.
- `EXECUTION_ATTEMPT(job_id, source_id, attempt_number)` is unique; retries increment the persisted attempt number.
- Historical `JOB_SOURCE` rows without an `EXECUTION_ATTEMPT` remain SDK snapshots and are explicitly labeled as
lacking transport evidence.
- `SOURCE.raw_transcription` caches the latest successful machine output for that page.
### Generic Processing Artifacts
- `PROCESSING_ARTIFACT` stores provider-neutral versioned derived outputs.
- Exactly one of `inline_payload` and `external_reference` is populated.
- Externally stored artifacts use application-managed relative references and are verified by SHA-256 and byte size.
- Coordinate metadata declares units, origin, dimensions, and transformations when geometry is present.
### Image Storage and Integrity
- Binary images are stored on disk; `SOURCE.file_path` stores the persisted path.