From bdb1b31b0a24e2aa4d73142ca27ce35469867767 Mon Sep 17 00:00:00 2001 From: Jim Lancaster <40281233+zoltan57@users.noreply.github.com> Date: Sun, 16 Aug 2026 00:13:34 -0500 Subject: [PATCH] V4.5 Scope defined --- docs/invariant/transcription_methodology.md | 18 ++ docs/ui/README.md | 4 +- docs/ver4.5/implementation_plan_v4_5.md | 263 ++++++++++++++++++++ docs/ver4.5/scope_boundary_v4_5.md | 226 +++++++++++++++++ docs/ver4/index_v4.md | 1 + 5 files changed, 510 insertions(+), 2 deletions(-) create mode 100644 docs/ver4.5/implementation_plan_v4_5.md create mode 100644 docs/ver4.5/scope_boundary_v4_5.md diff --git a/docs/invariant/transcription_methodology.md b/docs/invariant/transcription_methodology.md index dd01967..a207390 100644 --- a/docs/invariant/transcription_methodology.md +++ b/docs/invariant/transcription_methodology.md @@ -45,6 +45,24 @@ The following rules map directly to editorial conventions for handling common ma | **Non-Textual Artifacts** | Record non-textual elements (seals, stamps, sketches, physical damage) using brief descriptive text inside square brackets. | [description] | [wax notary seal attached here] or [sketch of a fort layout] | | **Marginalia & Addenda** | Explicitly indicate spatial transitions before transcribing content located in margins or non-standard orientations. | [location:] | [written in left margin:] Do not share this with anyone. | +### 3.4 Document-Body Medium + +Every transcript must identify the predominant document-body medium exactly once at the beginning: + +| Medium | Use | Standard Markup | +| --- | --- | --- | +| **Handwritten** | The main body was written by hand. | `[document body handwritten]` | +| **Typewritten** | The main body was produced with a typewriter. Uneven impressions, monospaced characters, and mechanical defects remain typewritten rather than handwritten. | `[document body typewritten]` | +| **Typeset** | The main body was composed for printing or produced as printed text rather than with a typewriter. | `[document body typeset]` | +| **Mixed** | No single medium predominates, or handwritten and printed/typewritten content are structurally interleaved. | `[document body mixed]` | + +- Use exactly one document-body marker. +- Do not wrap each line in `[handwritten: ...]` after declaring the body handwritten. +- In typewritten or typeset documents, use localized handwriting markers only for genuinely handwritten annotations, insertions, or signatures. +- In mixed documents, identify handwritten portions locally while preserving their reading context. +- Preserve tables of contents, tables, forms, columns, captions, marginalia, page numbers, dotted leaders, and associated references in their logical reading order. +- Produce plain text characters rather than HTML entities for ordinary transcription content. + ## 4. Prompt Asset Integration When executing programmatic transcriptions via LLM APIs or local models, processing instructions must be packaged into single-purpose system prompts aligned with these rules. diff --git a/docs/ui/README.md b/docs/ui/README.md index e9ab165..b562156 100644 --- a/docs/ui/README.md +++ b/docs/ui/README.md @@ -57,5 +57,5 @@ Each page contract contains: ## Current Baseline -These contracts describe the V4 baseline with completed V4.1 behavior and V4.2 evidence/provenance behavior. -Draft V4.3 Settings changes are not described as current behavior. +These contracts describe the completed V4 through V4.4 behavior. +V4.5 planned behavior remains defined by its frozen scope and implementation plan until implementation updates the applicable page contracts. diff --git a/docs/ver4.5/implementation_plan_v4_5.md b/docs/ver4.5/implementation_plan_v4_5.md new file mode 100644 index 0000000..5349dcb --- /dev/null +++ b/docs/ver4.5/implementation_plan_v4_5.md @@ -0,0 +1,263 @@ +# Implementation Plan (Version 4.5) + +## Goal + +Normalize metadata-directed image orientation for provider input, improve transcription-medium instructions and deterministic quality warnings, and support user-initiated single-Source retranscription with approved alternate models and explicit candidate promotion. + +## Planning Status + +- V4.4 is the completed implementation baseline. +- The V4.5 scope is frozen and sufficiently detailed to begin implementation. +- Scope additions require an explicit amendment or a later revision. + +## Planning Constraints + +- Original uploaded Source files remain immutable. +- Provider input must remain traceable to the original Source and any normalized derivative. +- Normalization is limited to recognized orientation metadata; no enhancement pipeline is introduced. +- Quality warnings never mutate transcription text or trigger paid requests automatically. +- Retranscription creates new immutable Job and execution evidence. +- A retranscription result remains a candidate until explicitly promoted. +- Human revision remains separate from and takes precedence over machine selection. +- Provider work occurs outside database transactions. +- Database, integration, and UI tests use confirmed isolated data and never modify `data/transcription.db`. +- Potentially destructive tests run only through `tools/run_destructive_tests.py`. + +## Expected Project Impact + +| Area | Expected impact | +| --- | --- | +| Configuration | Add a validated provider-model allowlist while retaining one default model. | +| Image processing | Add metadata-directed orientation normalization and model-input artifact creation. | +| Evidence model | Link each request to the exact model-input artifact and record preferred machine-attempt provenance. | +| Prompt contract | Add explicit body-medium classification and structured-layout rules. | +| Quality service | Add deterministic, non-mutating warnings for known output defects. | +| Job creation | Support a Source-locked retranscription Job and approved model selection. | +| Worker workflows | Preserve candidates without automatically replacing preferred machine text. | +| Source Detail | Add Retranscribe Source, candidate summaries, comparison, promotion, and normalization indicators. | +| Tests and documentation | Add isolated normalization, configuration, warning, retranscription, promotion, and UI coverage. | + +## Implementation Phases + +### 1. Align Configuration Contracts + +- Add `provider_models` as an immutable validated collection in Settings. +- Parse `PROVIDER_MODELS` using the standard Pydantic-settings JSON representation. +- Preserve `PROVIDER_MODEL` as the default. +- If the allowlist is omitted, derive a one-entry list from the default. +- Normalize whitespace, reject empty values, and deduplicate while preserving the relative order of non-default values. +- Ensure the default appears exactly once and first in the effective selector order. +- Validate a submitted model against the allowlist in the service or workflow boundary, not only in the UI. +- Document `.env.example` behavior without adding real credentials. +- Add configuration tests for omitted, valid, duplicate, malformed, and empty model lists. + +### 2. Define Orientation-Normalized Artifacts + +- Reuse `ProcessingArtifact` for the model-input derivative and transformation metadata. +- Define a versioned orientation-normalization artifact schema containing: + - Original Source identity and digest. + - Original orientation value. + - Applied rotation. + - Original and derivative dimensions, media types, byte sizes, and digests. + - Processor name and version. +- Add one domain-owned orientation-normalization service or adapter; keep image-library details out of UI and provider modules. +- Apply recognized metadata orientation physically to raster pixels. +- Reset or remove orientation metadata on the derivative. +- Store derivatives under application-managed artifact storage with safe relative references. +- Avoid creating a derivative when no supported transformation is required. +- Return a typed provider-input reference that identifies whether the request uses the original or a derivative. +- Never mutate or delete the original Source as part of normalization. + +### 3. Integrate Normalization with Provider Input + +- Resolve the exact model input before constructing the request manifest. +- Use the normalized derivative when orientation metadata requires it; otherwise use the original Source. +- Extend the existing `SourceEvidenceReference` or associated artifact reference so the manifest identifies: + - Original Source. + - Derivative artifact when present. + - Transformation schema and digest. +- Ensure normalization completes and persists before provider network work begins. +- Bind the immutable model-input artifact reference to the exact ExecutionAttempt that consumed it before terminal attempt persistence completes. +- If normalized bytes are reused, retain attempt-specific association while preserving one content identity and digest. +- If normalization fails, do not send a provider request. +- Preserve safe failure evidence and an actionable error category. +- Confirm provider payload loading and evidence hashing read the same resolved bytes. + +### 4. Revise the Transcription Prompt + +- Align the prompt with the durable medium rules in `docs/invariant/transcription_methodology.md`. +- Add the four frozen document-body markers. +- Define operational differences among handwritten, typewritten, typeset, and mixed content. +- State that mechanical typewriter variation is not handwriting. +- Require exactly one body marker. +- Prohibit repeated whole-line handwriting wrappers after a whole-body handwritten marker. +- Permit localized handwriting markers only for actual annotations, signatures, or mixed-body portions. +- Add layout instructions for tables of contents, tables, forms, columns, captions, marginalia, page numbers, dotted leaders, and associated references. +- Require plain-text characters rather than HTML entities. +- Retain verbatim, uncertainty, damage, deletion, insertion, and line-break-hyphenation rules. +- Update prompt fixtures and prompt-hash expectations without rewriting historical Job prompt evidence. + +### 5. Add Deterministic Quality Analysis + +- Introduce a small typed warning model with stable warning codes and human-readable detail. +- Analyze successful output without modifying it. +- Implement warnings for: + - Unicode replacement characters. + - Multiple document-body markers. + - Whole-body handwritten plus repeated line-level handwriting wrappers. + - Likely unresolved HTML entities. +- Keep warning rules deterministic and provider-independent. +- Persist warnings once as an immutable, versioned `ProcessingArtifact` tied to the successful ExecutionAttempt. +- Source Detail renders stored warnings and never recomputes historical attempts under newer warning rules. +- Make warning analysis idempotent and versioned so newer rules apply only to newly analyzed attempts unless a separate future reanalysis workflow is introduced. +- Do not implement confidence scoring or automatic retries. + +### 6. Model Retranscription and Selection State + +- Add a durable Job purpose or equivalent discriminator for normal transcription versus Source retranscription. +- Ensure a retranscription Job contains exactly one JobSource for the locked Source. +- Add durable preferred-machine-attempt provenance for each Source. +- Retain `Source.raw_transcription` as the preferred-machine-text projection for compatibility. +- Define legacy behavior for Sources whose current projection predates ExecutionAttempt provenance. +- On the first successful result with no preferred machine output, select the successful attempt automatically regardless of Job purpose. +- Once preferred provenance exists, preserve every later successful result as an unselected candidate regardless of Job purpose. +- Prevent normal and retranscription workflows from writing `Source.raw_transcription` directly when preferred provenance already exists. +- Add a promotion command that: + - Loads the Source and successful ExecutionAttempt. + - Verifies ownership and successful text. + - Updates preferred-attempt provenance and `Source.raw_transcription` in one transaction. + - Leaves `Source.revised_text` and all attempts unchanged. +- Reject failed, unrelated, missing, or textless candidates deterministically. + +### 7. Add the Retranscribe Source Workflow + +- Add a Source Detail **Retranscribe Source** action. +- Navigate to Create Processing Job with an explicit `source_id` query parameter. +- Load the Source and derive its Document server-side. +- Render Source identity and filename as locked context. +- Render Provider from Settings as read-only. +- Render Model as a selector using the effective allowlist and default. +- Reuse the frozen default prompt unless a later scope addition explicitly allows prompt selection. +- Create a new queued retranscription Job and one pending JobSource atomically. +- Notify the worker only after the transaction commits. +- Preserve current preferred machine text and human revision throughout queueing, processing, success, and failure. +- Return to the new Job Detail after successful creation. + +### 8. Adapt Worker Success Semantics + +- Select the first successful result automatically for Sources without preferred machine output, including a successful retranscription after earlier failures. +- For every later success, persist JobSource and ExecutionAttempt text without replacing the Source projection, regardless of normal or retranscription Job purpose. +- Run deterministic quality analysis after successful normalization of provider output. +- Persist candidate warnings with the attempt. +- Keep all terminal Job status and execution evidence updates atomic according to the existing workflow boundary. +- Ensure a failed retranscription cannot clear or change preferred machine text. + +### 9. Build Candidate Review and Promotion UI + +- Extend Source Detail with concise machine-output sections: + - Preferred machine transcription. + - Human revision. + - Candidate machine transcriptions. +- List candidates with creation date, provider, model, Job ID, status, and warning indicator. +- Default to a compact candidate list; do not render every full transcript simultaneously. +- When no successful machine result exists, render an explicit empty state without comparison controls. +- When preferred output exists without candidates, render an explicit no-candidates state. +- Allow one candidate to be opened for comparison with the preferred machine transcription. +- Label both sides with provider, model, Job ID, and date. +- Add **Use this transcription** only for a successful unselected candidate. +- Require explicit confirmation before promotion. +- After promotion, refresh Source Detail and retain the former preferred result in attempt history. +- If a human revision exists, explain that promotion changes machine selection but not the human-preferred displayed/printed text. +- When orientation normalization occurred, show a compact indicator and link to transformation evidence; do not require routine side-by-side image display. + +### 10. Align API and Service Contracts + +- Keep arbitrary provider and model identifiers out of public write contracts. +- If an API is added for retranscription, accept Source UUID and one configured model identifier and validate both server-side. +- If an API is added for promotion, accept Source UUID and ExecutionAttempt UUID and validate their relationship. +- Return stable validation, conflict, not-found, provider, and persistence errors through the existing taxonomy. +- Keep filesystem paths, raw credentials, and unrestricted artifact references out of responses. + +### 11. Verification + +- Add pure unit tests for: + - Orientation metadata interpretation. + - No-op versus transformed input selection. + - Derivative metadata and hashing. + - Model allowlist normalization. + - Prompt marker rules. + - Every quality-warning code. +- Add isolated service and workflow tests for: + - Original Source immutability. + - Normalization failure before provider invocation. + - Request manifests referencing exact provider-input bytes. + - Single-Source retranscription Job creation. + - Candidate preservation. + - Initial automatic selection. + - Explicit candidate promotion. + - Atomic rollback on invalid promotion. + - Human revision preservation. +- Add UI tests for: + - Retranscribe Source navigation. + - Locked Source and Document context. + - Read-only Provider and allowlisted Model selector. + - Candidate list, warnings, comparison, confirmation, and promotion. + - Normalization indicator without mandatory image comparison. +- Use fixed image fixtures with known EXIF orientation and hashes. +- Use fake providers only; no focused or regression test sends an external provider request. +- Confirm all database, integration, and UI targets use isolated databases. +- Run potentially destructive schema tests only through `tools/run_destructive_tests.py`. +- Run the broader non-external regression suite after focused coverage passes. +- Verify that `data/transcription.db` and curated Source files were not changed by test execution. + +### 12. Align Authoritative Documentation + +- Update V4 architecture, requirements, and schema for: + - Original versus model-input artifacts. + - Preferred machine-attempt provenance. + - Retranscription Job purpose. + - Candidate and promotion semantics. + - Quality-warning evidence. +- Update Source and Job UI contracts after implementation behavior is accepted. +- Update the transcription methodology and evidence invariant only where V4.5 establishes a durable cross-version rule. +- Keep historical prompt and execution evidence immutable. + +## Delivery Order + +1. Configuration and model allowlist. +2. Orientation artifact schema and normalization adapter. +3. Provider-input and evidence integration. +4. Prompt revision and deterministic warning analysis. +5. Retranscription and preferred-attempt persistence. +6. Worker candidate semantics. +7. Retranscription creation UI. +8. Candidate comparison and promotion UI. +9. Authoritative documentation and regression verification. + +## Done Criteria + +- All frozen V4.5 acceptance criteria are implemented and testable. +- EXIF-oriented images are physically upright for provider processing while originals remain byte-for-byte unchanged. +- Every provider request identifies its exact original and normalized inputs. +- The prompt classifies body medium consistently and avoids redundant handwriting wrappers. +- Quality defects produce deterministic warnings without silent rewriting or automatic cost. +- Source Detail can create one-Source retranscription Jobs using configured alternate models. +- Retranscription results remain candidates until explicitly promoted. +- Promotion updates exact preferred-attempt provenance and the compatibility projection atomically. +- Human revisions remain unchanged and retain display/print precedence. +- Previous Jobs and attempts remain immutable and inspectable. +- No manual image editor, visual orientation inference, confidence percentage, automatic retry, or arbitrary model entry is introduced. +- Verification uses isolated data, fake providers, and does not modify `data/transcription.db` or curated Source files. + +## Related Local References + +- [V4.5 Scope Boundary](scope_boundary_v4_5.md) +- [V4.4 Scope Boundary](../ver4.4/scope_boundary_v4_4.md) +- [V4.4 Implementation Plan](../ver4.4/implementation_plan_v4_4.md) +- [V4.2 Evidence and Provenance Scope](../ver4.2/scope_boundary_v4_2.md) +- [V4 Architecture](../ver4/architecture_v4.md) +- [V4 Schema](../ver4/schema_v4.md) +- [V4 Requirements](../ver4/requirements_v4.md) +- [V4 Error Handling Policy](../ver4/error_handling_v4.md) +- [Transcription Methodology](../invariant/transcription_methodology.md) +- [AI Evidence and Provenance Invariant](../invariant/ai_evidence_and_provenance.md) diff --git a/docs/ver4.5/scope_boundary_v4_5.md b/docs/ver4.5/scope_boundary_v4_5.md new file mode 100644 index 0000000..da916d0 --- /dev/null +++ b/docs/ver4.5/scope_boundary_v4_5.md @@ -0,0 +1,226 @@ +# V4.5 Scope Boundary + +This document defines the frozen boundary for transcription input normalization and selective transcription-quality improvement after the completed V4.4 revision. V4 through V4.4 remain the architecture and behavioral baseline except where this document explicitly changes Source processing, transcription selection, or Source Detail behavior. + +## Purpose + +- Ensure model inputs are physically upright when curated image files rely on orientation metadata. +- Distinguish typewritten, typeset, handwritten, and mixed document bodies consistently. +- Let the user selectively retranscribe an unsatisfactory Source with an approved alternate vision model. +- Preserve every machine result while allowing the user to choose which result is the preferred machine transcription. + +## In Scope + +### 1. Metadata-Driven Orientation Normalization + +- The original uploaded Source remains immutable archival evidence. +- Before a supported raster image is sent to a transcription provider, the application reads recognized orientation metadata. +- When the metadata requires rotation, the application creates a physically upright model-input derivative and resets or removes the derivative's orientation metadata. +- The provider receives the normalized derivative rather than upside-down stored pixels. +- When no supported orientation transformation is required, the original Source may remain the provider input. +- The derivative records: + - Source UUID. + - Original and derivative SHA-256 digests and byte sizes. + - Original and derivative dimensions and media types. + - Applied orientation transformation. + - Transformation implementation and version. + - Creation timestamp. +- The derivative uses the existing processing-artifact and evidence architecture rather than replacing the Source file. +- PDF orientation, visual orientation inference, manual rotation controls, deskewing, cropping, contrast changes, and general image enhancement are not part of V4.5. + +### 2. Transcription Medium Contract + +- The durable document-medium rules are defined in the cross-version [Transcription Methodology](../invariant/transcription_methodology.md). +- The transcription prompt distinguishes these document-body media: + - `[document body handwritten]` + - `[document body typewritten]` + - `[document body typeset]` + - `[document body mixed]` +- A typewriter's uneven impressions, monospaced characters, or mechanical defects do not by themselves indicate handwriting. +- The transcript contains exactly one applicable document-body marker. +- A wholly handwritten body uses the one body marker rather than wrapping every line in `[handwritten: ...]`. +- Typewritten and typeset bodies do not use handwriting wrappers unless a genuinely handwritten annotation or signature appears. +- A mixed body may use localized handwriting markers only for the handwritten portions. +- Stored transcription output remains plain text. Model-generated HTML entities are not required for ordinary characters. +- The prompt includes layout guidance for tables of contents, tables, forms, columns, captions, marginalia, page numbers, dotted leaders, and associated page references. +- Line-break hyphenation rules continue to preserve intentional hyphens while rejoining words split only by line wrapping. + +### 3. Quality Warnings + +- The application evaluates successful machine output for deterministic warning conditions, including: + - Unicode replacement characters such as `�`. + - A whole-body handwritten marker combined with repeated whole-line handwriting wrappers. + - More than one document-body medium marker. + - Unresolved HTML entities in otherwise plain transcription text. +- Warnings do not silently rewrite model output. +- Warnings do not automatically trigger another paid provider request. +- Source Detail displays warnings with the relevant machine result so the user can decide whether to revise or retranscribe it. +- V4.5 does not assign or display a transcription-confidence percentage. Provider self-assessments and token probabilities are not treated as calibrated transcription confidence. + +### 4. Source Retranscription Entry Point + +- Source Detail adds a **Retranscribe Source** action. +- The action opens Create Processing Job with the Source preselected and locked. +- The Source's existing Document is derived from its relationship and cannot be changed in this flow. +- The new Job contains a `JobSource` only for the selected Source; it does not retranscribe every Source in the Document. +- Provider is populated from the configured `PROVIDER` value and is read-only while only one provider is configured. +- Model is selected from an operator-configured allowlist. +- The configured default model is initially selected. +- Creating the Job freezes the selected provider, model, prompt, prompt hash, parameters, and Source evidence according to the existing provenance contract. +- Retranscription creates a new Job and new execution evidence. It does not reuse, mutate, or erase a previous Job. + +### 5. Configured Vision-Model Allowlist + +- `PROVIDER_MODEL` remains the default transcription model. +- `PROVIDER_MODELS` defines the models available in the Create Processing Job model selector. +- The environment representation is a JSON array, for example: + +```dotenv +PROVIDER=openrouter +PROVIDER_MODEL=google/gemini-2.5-flash +PROVIDER_MODELS=["google/gemini-2.5-flash","google/gemini-2.5-pro","anthropic/claude-sonnet-4"] +``` + +- If `PROVIDER_MODELS` is omitted, the selector contains only `PROVIDER_MODEL`. +- The default model appears exactly once and first; remaining configured models retain their relative order. +- Empty, malformed, or duplicate values produce deterministic configuration validation. +- The UI never accepts an arbitrary model identifier outside the configured allowlist. +- The allowlist controls availability, not claims of quality, price, or provider compatibility. The operator is responsible for configuring models supported by the selected provider. + +### 6. Candidate Machine Transcriptions + +- The first successful transcription becomes preferred automatically whenever the Source has no preferred machine output, regardless of whether it came from an initial or retranscription Job. +- Once a Source has a preferred machine transcription, every later successful result is stored as a candidate regardless of Job purpose and cannot replace the preferred result automatically. +- Every candidate remains associated with its immutable Job, JobSource, ExecutionAttempt, provider, model, prompt, parameters, timestamps, warnings, and normalized-input evidence. +- Source Detail presents: + - The current preferred machine transcription. + - Available successful candidates with date, provider, model, Job ID, and warning state. + - A comparison between the current preferred machine transcription and one selected candidate. + - A **Use this transcription** action for a successful candidate. +- Before any successful result exists, Source Detail displays an explicit no-machine-transcription state. +- When a preferred result exists but no candidates exist, Source Detail omits comparison controls and displays an explicit no-candidates state. +- Promoting a candidate: + - Verifies that the successful execution belongs to the Source. + - Records the selected execution as the preferred machine-output provenance. + - Updates `Source.raw_transcription` as the preferred-machine-text projection. + - Does not modify `Source.revised_text`. + - Does not delete or alter any previous machine result. +- If a human revision exists, it remains the current human-preferred text used by normal display and printing after a machine candidate is promoted. + +### 7. Evidence and Failure Behavior + +- The original Source and every normalized derivative are content-addressed and traceable. +- Every provider request identifies the exact original Source and model-input artifact used. +- Every execution attempt records the exact immutable model-input artifact it consumed, including when normalized bytes are reused. +- Every retranscription attempt follows the V4.2 immutable execution-attempt contract. +- A normalization failure prevents the provider request and produces an explicit actionable error. +- A provider or persistence failure leaves the current preferred machine transcription and human revision unchanged. +- Candidate promotion is atomic: provenance selection and the preferred-machine-text projection either both commit or both remain unchanged. +- Provider network work occurs outside database transactions. + +### 8. Source Detail Terminology + +- **Original Source** means the immutable uploaded file. +- **Model input** means the original Source or normalized derivative actually sent to the provider. +- **Machine attempt** means one immutable provider execution. +- **Candidate transcription** means a successful machine result not currently selected as preferred. +- **Preferred machine transcription** means the selected machine result projected through `Source.raw_transcription`. +- **Human revision** means `Source.revised_text`, which remains independent of every machine result. +- These distinctions use concise labels and progressive disclosure; routine satisfactory Sources do not display a mandatory side-by-side original/normalized image comparison. +- When normalization occurred, Source Detail displays an orientation-normalized indicator and makes transformation evidence inspectable through the existing evidence UI. + +## Out of Scope + +- Manual image rotation or image-editing controls. +- Visual orientation detection when metadata is absent or incorrect. +- Deskewing, cropping, contrast normalization, denoising, sharpening, or restoration. +- Replacing or modifying original Source files. +- Automatically retranscribing every Source. +- Automatic provider retries triggered by quality warnings. +- Arbitrary model identifiers entered by users. +- Multiple provider selection in the UI. +- Model benchmarking, pricing recommendations, or automatic model ranking. +- A provider-independent transcription-confidence percentage. +- Silent cleanup or rewriting of model output. +- Deleting unsuccessful, superseded, or unselected machine attempts. +- Promoting a machine candidate over a human revision. +- Source page renumbering or reordering. + +## Locked Design Decisions + +### A. Curated Originals Remain Authoritative + +- V4.5 corrects metadata-directed orientation only for model processing. +- The archival upload is never replaced by the normalized derivative. + +### B. Orientation Is Automatic and Metadata-Driven + +- No manual orientation workflow is introduced. +- V4.5 does not guess orientation from page content. + +### C. Selective Retranscription Replaces Automatic Escalation + +- The normal default model remains efficient for satisfactory Sources. +- The user explicitly chooses when an alternate approved model is worth another provider request. +- Quality warnings inform that choice but never incur cost automatically. + +### D. Retranscription Produces Candidates + +- Alternate results remain immutable and comparable. +- The user explicitly promotes the preferred machine result. +- Human revision remains a separate, higher-precedence layer. + +### E. Model Choice Is Operator-Controlled + +- Environment configuration defines the finite allowed model set. +- Job records freeze the actual selected model and request parameters. + +### F. Confidence Is Evidence-Based, Not Invented + +- V4.5 does not present model self-rating as objective confidence. +- Review uses visible output, deterministic warnings, provenance, and human judgment. + +## Acceptance Criteria + +1. A JPEG with EXIF Orientation 3 produces an upright model-input derivative while the original bytes remain unchanged. +2. A Source requiring no recognized orientation transformation is not unnecessarily altered. +3. Orientation transformation metadata and hashes identify the exact provider input. +4. The prompt distinguishes handwritten, typewritten, typeset, and mixed bodies with exactly one body marker. +5. Typewritten text is not wrapped line by line as handwriting. +6. Tables of contents retain row associations and page references without handwriting wrappers. +7. Deterministic warnings identify replacement characters and contradictory body markers without changing output. +8. Source Detail provides Retranscribe Source for an existing Source. +9. Create Processing Job locks the Source and Document, uses configured Provider, and restricts Model to the configured allowlist. +10. Retranscription creates a new single-Source Job with complete frozen request and execution evidence. +11. The first successful result is selected automatically; every later successful result remains a candidate and cannot replace the preferred machine transcription automatically. +12. Source Detail can compare the preferred machine transcription with one candidate and promote that candidate explicitly. +13. Candidate promotion records exact successful-execution provenance and updates the machine-text projection atomically. +14. Candidate promotion never changes or clears a human revision. +15. Earlier machine attempts remain inspectable after retranscription and promotion. +16. No confidence percentage, manual image editor, automatic quality retry, or arbitrary model input is introduced. +17. Database, integration, and UI verification uses isolated test data and never modifies `data/transcription.db`. + +## Scope Freeze Gate + +V4.5 is sufficiently frozen to begin implementation: + +- Orientation behavior and preservation rules are resolved. +- Prompt medium categories and marker behavior are resolved. +- Warning behavior and the absence of automatic retry are resolved. +- Retranscription entry point, single-Source scope, and model configuration are resolved. +- Candidate preservation, comparison, promotion, and human-revision precedence are resolved. +- The broader future-feature list has been reviewed, and no additional V4.5 features are required. + +Any expansion into manual image editing, visual orientation inference, automatic retries, multiple providers, confidence scoring, or additional processing features requires an explicit V4.5 scope amendment or a later revision. + +## Related Local References + +- [V4.5 Implementation Plan](implementation_plan_v4_5.md) +- [V4.4 Scope Boundary](../ver4.4/scope_boundary_v4_4.md) +- [V4.4 Implementation Plan](../ver4.4/implementation_plan_v4_4.md) +- [V4.2 Evidence and Provenance Scope](../ver4.2/scope_boundary_v4_2.md) +- [V4 Architecture](../ver4/architecture_v4.md) +- [V4 Schema](../ver4/schema_v4.md) +- [V4 Requirements](../ver4/requirements_v4.md) +- [Transcription Methodology](../invariant/transcription_methodology.md) +- [AI Evidence and Provenance Invariant](../invariant/ai_evidence_and_provenance.md) diff --git a/docs/ver4/index_v4.md b/docs/ver4/index_v4.md index a098265..58ab01e 100644 --- a/docs/ver4/index_v4.md +++ b/docs/ver4/index_v4.md @@ -28,3 +28,4 @@ Version 4 is the architecture baseline for the personal-scale application used t - [V4.2 Evidence and Provenance Scope](../ver4.2/scope_boundary_v4_2.md) and [Implementation Plan](../ver4.2/implementation_plan_v4_2.md) - [V4.3 Settings Scope](../ver4.3/scope_boundary_v4_3.md) and [Implementation Plan](../ver4.3/implementation_plan_v4_3.md) - [V4.4 Semantic Registries, Linked People, and Printing Scope](../ver4.4/scope_boundary_v4_4.md) and [Implementation Plan](../ver4.4/implementation_plan_v4_4.md) +- [V4.5 Transcription Input Normalization and Quality Scope](../ver4.5/scope_boundary_v4_5.md) and [Implementation Plan](../ver4.5/implementation_plan_v4_5.md)