gpt-5.3-codex review phase 1 - Flatten the documentation

2026-08-19 14:54:24 -05:00
parent 2c26177d0c
commit 5404224079
32 changed files with 358 additions and 4467 deletions
@@ -1,258 +1,130 @@
-# Database Schema (Version 4)
+# Data Model and Persistence Schema (Version 4)

-This document defines the relational schema for the document transcription system.
+This schema reflects the current V4 persistence contract.

-## Entity Relationship Diagram
+## Entity Relationship Overview

 ```mermaid
 erDiagram
-DOCUMENT_TYPE {
-UUID id PK
-TEXT semantic_key UK
-TEXT label
-TEXT normalized_label
-BOOLEAN is_active
-TIMESTAMPTZ created_at
-TIMESTAMPTZ updated_at
-}
+    Document ||--o{ Source : has
+    Document ||--o{ Job : has
+    Document ||--o{ DocumentPerson : links
+    Person ||--o{ DocumentPerson : links
+    Job ||--o{ JobSource : includes
+    Source ||--o{ JobSource : participates
+    Source ||--o{ ExecutionAttempt : records

-PERSON_ROLE {
-UUID id PK
-TEXT semantic_key UK
-TEXT label
-TEXT normalized_label
-BOOLEAN is_active
-TIMESTAMPTZ created_at
-TIMESTAMPTZ updated_at
-}
+    Document {
+      uuid id PK
+      string title
+      uuid type_id FK
+      string language
+      datetime doc_date
+      string date_note
+      string comments
+      string location
+      datetime created_at
+      datetime updated_at
+    }

-PERSON {
-UUID id PK
-TEXT full_name
-TEXT display_name
-TEXT maiden_name
-DATE birth_date
-TEXT birth_date_raw
-TEXT birth_place
-DATE death_date
-TEXT death_date_raw
-TEXT death_place
-TEXT biography
-TEXT portrait_path
-JSONB metadata
-TIMESTAMPTZ created_at
-TIMESTAMPTZ updated_at
-}
+    Source {
+      uuid id PK
+      uuid document_id FK
+      string original_name
+      string media_type
+      string storage_path
+      int file_size
+      string file_hash
+      string raw_transcription
+      datetime created_at
+      datetime updated_at
+    }

-DOCUMENT {
-UUID id PK
-UUID document_type_id FK
-TEXT name
-DATE document_date
-TEXT document_date_raw
-TEXT location_created
-TEXT notes
-TEXT archive_identifier
-TIMESTAMPTZ created_at
-TIMESTAMPTZ updated_at
-}
+    Job {
+      uuid id PK
+      uuid document_id FK
+      enum status
+      string prompt
+      json model_settings_json
+      datetime created_at
+      datetime updated_at
+    }

-DOCUMENT_PERSON {
-UUID id PK
-UUID document_id FK
-UUID person_id FK
-UUID role_id FK
-TIMESTAMPTZ created_at
-TIMESTAMPTZ updated_at
-}
+    JobSource {
+      uuid job_id FK
+      uuid source_id FK
+      enum status
+      uuid selected_attempt_id FK
+      string error_message
+      datetime created_at
+      datetime updated_at
+    }

-JOB {
-UUID id PK
-UUID document_id FK
-VARCHAR status
-INTEGER retry_count
-VARCHAR purpose
-TEXT provider
-TEXT model
-TEXT prompt_name
-TEXT prompt_hash
-TEXT system_prompt
-TEXT user_prompt
-FLOAT temperature
-FLOAT top_p
-TIMESTAMPTZ date_created
-TIMESTAMPTZ date_updated
-}
-
-SOURCE {
-UUID id PK
-UUID document_id FK
-INTEGER page_number
-TEXT upload_name
-TEXT filename
-TEXT file_path
-TEXT file_hash
-BIGINT file_size_bytes
-TEXT raw_transcription
-UUID preferred_execution_attempt_id FK
-TEXT revised_text
-TIMESTAMPTZ date_uploaded
-TIMESTAMPTZ date_revised
-}
-
-JOB_SOURCE {
-UUID id PK
-UUID job_id FK
-UUID source_id FK
-VARCHAR status
-TEXT raw_transcription
-JSONB ai_metadata
-JSONB raw_api_response
-TEXT error_detail
-TIMESTAMPTZ executed_at
-}
-
-EXECUTION_ATTEMPT {
-UUID id PK
-UUID job_source_id FK
-UUID job_id FK
-UUID source_id FK
-INTEGER attempt_number
-VARCHAR status
-JSONB request_manifest
-TEXT request_manifest_sha256
-INTEGER transport_status_code
-BINARY transport_body
-JSONB transport_safe_headers
-JSONB sdk_response_snapshot
-JSONB normalized_metadata
-JSONB software_context
-TEXT raw_transcription
-TEXT failure_phase
-TIMESTAMPTZ started_at
-TIMESTAMPTZ finished_at
-INTEGER duration_ms
-}
-
-PROCESSING_ARTIFACT {
-UUID id PK
-UUID source_id FK
-UUID execution_attempt_id FK
-TEXT artifact_type
-TEXT media_type
-TEXT schema_name
-TEXT schema_version
-TEXT producer
-TEXT producer_version
-JSONB inline_payload
-TEXT external_reference
-TEXT payload_sha256
-BIGINT byte_size
-JSONB coordinate_metadata
-TIMESTAMPTZ created_at
-}
-
-DOCUMENT_TYPE ||--o{ DOCUMENT : classifies
-DOCUMENT ||--o{ DOCUMENT_PERSON : has_people
-PERSON ||--o{ DOCUMENT_PERSON : appears_in
-PERSON_ROLE ||--o{ DOCUMENT_PERSON : labels
-DOCUMENT ||--o{ JOB : has_jobs
-DOCUMENT ||--o{ SOURCE : contains_pages
-JOB ||--o{ JOB_SOURCE : executes
-SOURCE ||--o{ JOB_SOURCE : processed_in
-JOB_SOURCE ||--o{ EXECUTION_ATTEMPT : projects
-SOURCE ||--o{ PROCESSING_ARTIFACT : derives
-EXECUTION_ATTEMPT ||--o{ PROCESSING_ARTIFACT : produces
+    ExecutionAttempt {
+      uuid id PK
+      uuid source_id FK
+      uuid job_id FK
+      enum outcome
+      string provider_name
+      string provider_model
+      string request_manifest_hash
+      json request_manifest_json
+      json transport_evidence_json
+      string transcript_text
+      string error_category
+      string error_message
+      float duration_seconds
+      datetime started_at
+      datetime completed_at
+      datetime created_at
+    }
 ```

-## Domain Invariants and Provenance Rules
+## Authoritative Enumerations

-### Page-Level Execution and AI Outputs
+### JobStatus

- Every single page execution by an AI model produces a dedicated `JOB_SOURCE` record.
- Every `JOB` stores the frozen prompt identifier, prompt text, and hyperparameters used at submission time.
- `JOB_SOURCE.raw_api_response` is a compatibility projection containing an SDK-serialized OpenRouter response
-  snapshot. It is neither the exact HTTP body nor the native upstream-provider response.
- Every new provider call creates an immutable `EXECUTION_ATTEMPT` containing the frozen request manifest,
-  exact OpenRouter-boundary response bytes when received, safe transport metadata, SDK snapshot, normalized
-  metadata, timing, and outcome.
- `EXECUTION_ATTEMPT(job_id, source_id, attempt_number)` is unique; retries increment the persisted attempt number.
- Historical `JOB_SOURCE` rows without an `EXECUTION_ATTEMPT` remain SDK snapshots and are explicitly labeled as
-  lacking transport evidence.
- `SOURCE.raw_transcription` caches the explicitly selected preferred machine output for that page.
- `SOURCE.preferred_execution_attempt_id` records exact successful-attempt provenance. Legacy projections may remain
-  null until a new successful result is selected.
+- `queued`
+- `processing`
+- `transcribed`
+- `completed` (legacy-compatible)
+- `partial_success`
+- `failed`

-### Generic Processing Artifacts
+### JobSourceStatus

- `PROCESSING_ARTIFACT` stores provider-neutral versioned derived outputs.
- Exactly one of `inline_payload` and `external_reference` is populated.
- Externally stored artifacts use application-managed relative references and are verified by SHA-256 and byte size.
- Coordinate metadata declares units, origin, dimensions, and transformations when geometry is present.
- Orientation-normalized binary model inputs and JSON quality-warning results use distinct versioned artifact types
-  and are attached to the exact consuming `EXECUTION_ATTEMPT`.
+- `pending`
+- `transcribed`
+- `failed`
+- `cancelled`

-### Image Storage and Integrity
+## Aggregate Ownership

- Binary images are stored on disk; `SOURCE.file_path` stores the persisted path.
- `SOURCE.file_hash` stores a SHA-256 digest.
- `SOURCE.file_size_bytes` stores the original file size.
+- `Document` aggregate: `Document`, linked `Source`, linked `DocumentPerson`.
+- `Job` aggregate: `Job`, `JobSource` rows, selected-attempt pointers.
+- Evidence aggregate: append-only `ExecutionAttempt` rows keyed by `source_id` + `job_id`.

-### Page Ordering and Revisions
+## Persistence Invariants

- `SOURCE.page_number` dictates page ordering within a document.
- `SOURCE.raw_transcription` changes only through first-success selection or explicit candidate promotion.
- `SOURCE.revised_text` stores human edits and is the preferred display value when present.
+1. `ExecutionAttempt` rows are immutable after creation, except explicit support fields reserved for compatibility migrations.
+2. `JobSource.status` is queue/projection state; it does not duplicate full attempt payload.
+3. `Source.raw_transcription` is a projection, not the complete evidence record.
+4. `Job` terminal status is derived from `JobSource` outcomes.
+5. Registry semantic keys, when present, are immutable once created.

-### Semantic Registry Governance
+## Media Storage Semantics

- `DOCUMENT_TYPE.id` and `PERSON_ROLE.id` are the only relationship and public API identities.
- Nullable unique `semantic_key` values identify application-defined built-ins and are immutable after creation.
- Semantic keys are internal and are never accepted from Settings or public relationship APIs.
- A non-null semantic key marks a protected built-in; built-ins may be relabeled or disabled but not deleted.
- Custom entries have null semantic keys and may be deleted only when unreferenced.
- Labels are mutable display text and are unique after trimming and case normalization.
- Inactive entries remain valid for historical rows but are excluded from new-assignment selectors.
+1. `Source.storage_path` references canonical stored bytes used by processing.
+2. Canonical stored bytes may reflect ingest-time normalization.
+3. File hash and size fields describe canonical stored bytes.

-### Document-Person Role Governance
+## Query and Loading Requirements

- Documents support zero or one relationship for each Person.
- Relationship roles are defined by `PERSON_ROLE` rather than hardcoded columns.
- `DOCUMENT_PERSON.role_id` is required.
- `DOCUMENT_PERSON` must be unique for `(document_id, person_id)`.
- Complete link sets and Document fields are validated and persisted in one atomic transaction.
- Existing inactive roles may remain unchanged; new or changed assignments require active roles.
+- Relationship access from service/UI layers must use explicit eager loading patterns compatible with `lazy="raise"`.
+- Candidate-attempt views should select latest/selected attempts explicitly; do not rely on implicit lazy traversal.

-### Document Type Governance
+## Cross-Reference

- Every document type is defined by `DOCUMENT_TYPE`.
- `DOCUMENT_TYPE.id` is the relationship identity; hidden semantic keys identify protected built-in meaning.
- `DOCUMENT_TYPE.label` is mutable display text and is unique after trimming and case normalization.
- `DOCUMENT_TYPE.normalized_label` stores the normalized uniqueness key.
- Inactive types remain valid for historical rows but should be excluded from default selection UIs.
-
-## Constraint Summary
-
- `DOCUMENT_TYPE.normalized_label` is unique.
- `DOCUMENT_TYPE.semantic_key` is nullable and unique.
- `PERSON_ROLE.normalized_label` is unique.
- `PERSON_ROLE.semantic_key` is nullable and unique.
- `DOCUMENT_PERSON(document_id, person_id)` is unique.
-
-## Indexing Guidance
-
- `document(document_type_id)`
- `document_person(document_id)`
- `document_person(person_id)`
- `document_person(role_id)`
- `source(document_id, page_number)`
- `job(document_id, status)`
- `job_source(job_id)`
- `job_source(source_id)`
-
-## Related Local References
-
- [System Overview](index_v4.md)
 - [System Architecture](architecture_v4.md)
 - [System Requirements](requirements_v4.md)
 - [Error Handling Policy](error_handling_v4.md)