gpt-5.3-codex review phase 1 - Flatten the documentation

docs/ver4/requirements_v4.md

@@ -1,60 +1,61 @@
-# Document Transcription System Requirements (Version 4)
+# System Requirements (Version 4)
 
-This document defines the baseline requirements for the document transcription system.
+These requirements define the active V4 contract and align to current implementation.
 
-## Requirements Model
+## Functional Requirements
 
-| ID | Category | Requirement | Verify Method |
-| --- | --- | --- | --- |
-| REQ-0 | System | Provide end-to-end multi-page document transcription with persistent, inspectable async job states. | demonstration |
-| REQ-1 | Functional | Allow users to upload one or more images as ordered `Source` pages under a `Document`. | test |
-| REQ-2 | Functional | Process page transcription asynchronously using an `asyncio` worker pool bounded by rate limits. | test |
-| REQ-3 | Functional | Persist submission-time request provenance and accurately labeled page-level SDK evidence; V4.2 adds exact OpenRouter-boundary transport evidence for new attempts. | test |
-| REQ-4 | Functional | Support job states `queued`, `processing`, `completed`, `partial_success`, and `failed`, plus page states `pending`, `transcribed`, and `failed`. | inspection |
-| REQ-5 | Functional | Allow users to manage historical `Person` records and link each Person to a Document once with exactly one role. | test |
-| REQ-6 | Functional | Support an extensible role taxonomy for document-person relationships. | inspection |
-| REQ-7 | Policy Constraint | Enforce deterministic relationship-role writes with uniqueness on `(document_id, person_id)` and explicit conflict responses for duplicate Person links. | test |
-| REQ-8 | Functional | Use set-based synchronization for document-person mutations so updates add and remove only the intended links. | test |
-| REQ-9 | Functional | Maintain selected machine output and exact attempt provenance on `Source` while permitting independent human edits on `Source.revised_text`. | test |
-| REQ-10 | Functional | Support a UUID-identified `DocumentType` taxonomy with unique user-facing labels and active/inactive lifecycle control. | test |
-| REQ-11 | Data Constraint | Store `Document` type as a controlled reference to `DocumentType`. | test |
-| REQ-12 | Interface | Render multi-page transcriptions sequentially by `page_number` with document, people, and document-type metadata. | demonstration |
-| REQ-13 | Interface | Document create/edit UI must provide one staged Linked People table and select active registry entries by UUID and label. | demonstration |
-| REQ-14 | API Constraint | Expose additive, role-aware retrieval and write behavior for document-person links and UUID-based selection for document types. | test |
-| REQ-15 | Data Constraint | Calculate and store cryptographic file hashes (SHA-256) and file sizes for uploaded source images. | test |
-| REQ-16 | Data Constraint | Preserve a portable relational model across supported backends using SQLModel, SQLAlchemy, SQLite, and PostgreSQL. | inspection |
-| REQ-17 | Reliability | Ensure delete and update flows for documents, people, and relationship links remain deterministic and safe. | test |
-| REQ-18 | Operations Constraint | Keep canonical development, testing, restore, and recovery workflows OS-independent; for AI-run unit tests, require a pre-test backup of `./data` and an always-shown post-success confirmation prompt before any restore action. | inspection |
-| REQ-19 | Quality | Provide automated coverage for async transcription workflows, relationship-role enforcement, document-type selection, and regression behavior. | test |
-| REQ-20 | Data Constraint | Permit hidden immutable semantic keys only on protected built-in Document Types and Person Roles while retaining UUID as relationship identity. | test |
-| REQ-21 | Reliability | Persist Document fields and their complete Linked People set atomically. | test |
-| REQ-22 | Interface | Provide safe browser-native Facsimile and Text-only print views from persisted Document Detail. | demonstration |
-| REQ-23 | Security | Escape stored print text and serve Source images through record-validated application routes without disclosing local paths. | test |
-| REQ-24 | Functional | Print current human-preferred Source text, semantic Author metadata, deterministic Source order, and oldest-to-newest Job metadata. | test |
-| REQ-25 | Quality | Physically apply recognized raster orientation metadata to provider-input derivatives without changing original Source bytes. | test |
-| REQ-26 | Quality | Persist deterministic, non-mutating output warnings without automatic paid retries. | test |
-| REQ-27 | Functional | Create one-Source retranscription Jobs from a configured model allowlist and preserve later successes as candidates until explicit promotion. | test |
+### Domain and Record Management
 
-## Clarifying Constraints
+- **REQ-4-001 Document Registry:** The system must create and update `Document` records with title, type, language, comments, date metadata, and optional location.
+- **REQ-4-002 Source Registry:** The system must create and update `Source` records linked to exactly one `Document`.
+- **REQ-4-003 People Registry:** The system must create and update `Person` records and support many-to-many links to `Document` with role and confidence.
+- **REQ-4-004 Registry Semantics:** Document types and person roles must support optional immutable semantic keys and hard-delete only when unreferenced.
 
-1. `DocumentType.id` and `PersonRole.id` are their public and relationship identities; labels are unique ignoring case and surrounding whitespace.
-2. Nullable `semantic_key` values identify protected application built-ins, remain internal, and never change.
-3. Relationship-write policy and conflict handling must be consistent across UI, API, services, and persistence.
-4. One Person may appear only once per Document and every link has exactly one role.
-5. Relationship conflicts must fail deterministically without partial Document or link mutation.
-6. Source page reordering and server-generated PDF files remain outside this revision.
+### Job and Workflow Behavior
 
-## Element Satisfaction Mapping
+- **REQ-4-010 Job Creation:** The system must create `Job` records from uploaded sources and from retranscription of existing sources.
+- **REQ-4-011 Prompt Snapshotting:** Job creation must persist effective prompt and runtime settings as immutable per-job snapshots.
+- **REQ-4-012 Queue Membership:** Each `(job, source)` pair must be represented by one `JobSource` row.
+- **REQ-4-013 Job Status Lifecycle:** `Job.status` must use one of `queued`, `processing`, `transcribed`, `completed`, `partial_success`, `failed`.
+- **REQ-4-014 JobSource Status Lifecycle:** `JobSource.status` must use one of `pending`, `transcribed`, `failed`, `cancelled`.
+- **REQ-4-015 Terminal Job Resolution:** Job terminal status must derive from page outcomes as `transcribed`, `partial_success`, or `failed`.
+- **REQ-4-016 Cancellation Semantics:** Job cancellation must set remaining `pending` page entries to `cancelled`.
 
-- UI (NiceGUI): Satisfies REQ-0, REQ-1, REQ-5, REQ-9, REQ-12, REQ-13, REQ-22, REQ-24.
-- API (FastAPI): Satisfies REQ-1, REQ-4, REQ-5, REQ-7, REQ-8, REQ-14, REQ-23.
-- Worker (`asyncio`): Satisfies REQ-2, REQ-3, REQ-4.
-- Persistence (SQLModel / SQLAlchemy): Satisfies REQ-3, REQ-7, REQ-9, REQ-10, REQ-11, REQ-15, REQ-16, REQ-17, REQ-20, REQ-21.
-- Test Suite: Verifies all test-marked requirements and satisfies REQ-19.
+### Transcription and Evidence
 
-## Related Local References
+- **REQ-4-020 Attempt Evidence:** Each provider call must emit one append-only `ExecutionAttempt` record.
+- **REQ-4-021 Attempt Payload:** `ExecutionAttempt` must retain request manifest/hash, outcome, timing, model/provider fields, and error details when present.
+- **REQ-4-022 Transport Evidence:** Provider response evidence must be attached to the attempt when a response is available.
+- **REQ-4-023 Source Projection Rule:** `Source.raw_transcription` is a projection chosen from attempt outcomes and can be repointed by explicit promotion.
+- **REQ-4-024 Candidate Visibility:** UI must expose candidate attempts with metadata needed for comparative review and selection.
 
-- [System Overview](index_v4.md)
-- [System Architecture](architecture_v4.md)
-- [Data Model](schema_v4.md)
-- [Error Handling Policy](error_handling_v4.md)
+### Media and Access
+
+- **REQ-4-030 Ingest Canonicalization:** Stored source bytes may be normalized at ingest (for example orientation correction); stored bytes are the canonical processing source.
+- **REQ-4-031 Path Safety:** Client-facing media URLs must be generated from controlled application paths only.
+- **REQ-4-032 Print Media Validation:** Print/export source media must be served through record-validated API routes.
+
+### Error and UX Contracts
+
+- **REQ-4-040 Error Envelope:** Service/API errors must map to structured, user-safe error categories and messages.
+- **REQ-4-041 Partial Failure Visibility:** Mixed page outcomes must be visible at job and page level.
+- **REQ-4-042 Retry Support:** Failed and cancelled pages must support targeted retranscription without requiring full document recreation.
+
+## Non-Functional Requirements
+
+- **REQ-4-100 Boundary Integrity:** UI pages/components must not access persistence directly and must call service APIs.
+- **REQ-4-101 Service Ownership:** Aggregate writes must occur in owning service/workflow modules, not in UI handlers.
+- **REQ-4-102 Deterministic Loading:** ORM relationship reads in service/UI code must use explicit eager loading compatible with `lazy="raise"`.
+- **REQ-4-103 Async Safety:** Long-running provider calls must not block UI event handlers directly.
+- **REQ-4-104 Evidence Durability:** Attempt evidence must survive process restart once the transaction commits.
+- **REQ-4-105 Test Guardrails:** Architecture boundary tests must remain in place for services and UI boundaries.
+
+## Traceability Notes
+
+- Source of truth for status enums:
+  - `src/transcription/db/models.py`
+- Source of truth for workflow transitions:
+  - `src/transcription/services/workflows.py`
+  - `src/transcription/services/jobs.py`
+- Source of truth for attempt evidence writes:
+  - `src/transcription/services/sources.py`