diff --git a/docs/ver4/schema_v4.md b/docs/ver4/schema_v4.md index 72f9c06..b4b91a7 100644 --- a/docs/ver4/schema_v4.md +++ b/docs/ver4/schema_v4.md @@ -1,83 +1,29 @@ # Data Model and Persistence Schema (Version 4) -This schema reflects the current V4 persistence contract. +This document is the field-accurate V4 schema contract aligned to `src/transcription/db/models.py`. + +## Source of Truth Anchors + +- `src/transcription/db/models.py:60-78` (status and purpose enums) +- `src/transcription/db/models.py:80-120` (`DocumentType`, `PersonRole`) +- `src/transcription/db/models.py:122-207` (`Document`, `Person`, `DocumentPerson`) +- `src/transcription/db/models.py:208-255` (`Job`) +- `src/transcription/db/models.py:273-386` (`Source`, `JobSource`) +- `src/transcription/db/models.py:387-445` (`ExecutionAttempt`) ## Entity Relationship Overview ```mermaid erDiagram - Document ||--o{ Source : has + DocumentType ||--o{ Document : classifies Document ||--o{ Job : has + Document ||--o{ Source : has Document ||--o{ DocumentPerson : links Person ||--o{ DocumentPerson : links + PersonRole ||--o{ DocumentPerson : labels Job ||--o{ JobSource : includes Source ||--o{ JobSource : participates - Source ||--o{ ExecutionAttempt : records - - Document { - uuid id PK - string title - uuid type_id FK - string language - datetime doc_date - string date_note - string comments - string location - datetime created_at - datetime updated_at - } - - Source { - uuid id PK - uuid document_id FK - string original_name - string media_type - string storage_path - int file_size - string file_hash - string raw_transcription - datetime created_at - datetime updated_at - } - - Job { - uuid id PK - uuid document_id FK - enum status - string prompt - json model_settings_json - datetime created_at - datetime updated_at - } - - JobSource { - uuid job_id FK - uuid source_id FK - enum status - uuid selected_attempt_id FK - string error_message - datetime created_at - datetime updated_at - } - - ExecutionAttempt { - uuid id PK - uuid source_id FK - uuid job_id FK - enum outcome - string provider_name - string provider_model - string request_manifest_hash - json request_manifest_json - json transport_evidence_json - string transcript_text - string error_category - string error_message - float duration_seconds - datetime started_at - datetime completed_at - datetime created_at - } + JobSource ||--o{ ExecutionAttempt : attempts ``` ## Authoritative Enumerations @@ -97,64 +43,192 @@ erDiagram - `failed` - `cancelled` -## Aggregate Ownership +### JobPurpose -- `Document` aggregate: `Document`, linked `Source`, linked `DocumentPerson`. -- `Job` aggregate: `Job`, `JobSource` rows, selected-attempt pointers. -- Evidence aggregate: append-only `ExecutionAttempt` rows keyed by `source_id` + `job_id`. +- `transcription` +- `retranscription` -## Persistence Invariants +## Field-Accurate Table Contracts -1. `ExecutionAttempt` rows are immutable after creation, except explicit support fields reserved for compatibility migrations. -2. `JobSource.status` is queue/projection state; it does not duplicate full attempt payload. -3. `Source.raw_transcription` is a projection, not the complete evidence record. -4. `Job` terminal status is derived from `JobSource` outcomes. -5. Registry semantic keys, when present, are immutable once created. +### `DocumentType` -## Schema Design Rationale +| Field | Type | Notes | +| :--- | :--- | :--- | +| `id` | `UUID` | PK | +| `semantic_key` | `str \| None` | nullable unique, indexed | +| `label` | `str` | required | +| `normalized_label` | `str` | unique, indexed | +| `is_active` | `bool` | default `True` | +| `created_at` | `datetime` | default now | +| `updated_at` | `datetime` | default now, onupdate | -### Why `ExecutionAttempt` exists alongside `JobSource` +### `PersonRole` -- `JobSource` is the mutable queue/projection row for workflow control and current-facing page outcome state. -- `ExecutionAttempt` is the durable, append-only evidence timeline for each provider call. -- Keeping both avoids overloading one table with competing concerns (queue state vs immutable audit history). +| Field | Type | Notes | +| :--- | :--- | :--- | +| `id` | `UUID` | PK | +| `semantic_key` | `str \| None` | nullable unique, indexed | +| `label` | `str` | required | +| `normalized_label` | `str` | unique, indexed | +| `is_active` | `bool` | default `True` | +| `created_at` | `datetime` | default now | +| `updated_at` | `datetime` | default now, onupdate | -### Why `Source.raw_transcription` remains on `Source` +### `Document` -- `Source` needs a stable, current projection for UI and print behavior. -- Projection reads are fast and direct, while deep historical inspection remains available through attempts. -- Explicit candidate promotion updates the projection pointer without rewriting history. +| Field | Type | Notes | +| :--- | :--- | :--- | +| `id` | `UUID` | PK | +| `name` | `str` | required | +| `document_type_id` | `UUID \| None` | FK -> `document_type.id`, indexed | +| `document_date` | `date \| None` | optional | +| `document_date_raw` | `str \| None` | optional | +| `location_created` | `str \| None` | optional | +| `notes` | `str \| None` | optional | +| `archive_identifier` | `str \| None` | optional | +| `created_at` | `datetime` | default now | +| `updated_at` | `datetime` | default now, onupdate | -### Why semantic registries use UUID identity plus optional semantic keys +### `Person` -- UUIDs are durable relationship identifiers for public/domain links. -- Optional immutable semantic keys support protected built-ins without exposing internal meaning as external API identity. -- Labels can evolve without breaking relationships. +| Field | Type | Notes | +| :--- | :--- | :--- | +| `id` | `UUID` | PK | +| `full_name` | `str` | required | +| `display_name` | `str \| None` | optional | +| `maiden_name` | `str \| None` | optional | +| `birth_date` | `date \| None` | optional | +| `birth_date_raw` | `str \| None` | optional | +| `birth_place` | `str \| None` | optional | +| `death_date` | `date \| None` | optional | +| `death_date_raw` | `str \| None` | optional | +| `death_place` | `str \| None` | optional | +| `biography` | `str \| None` | optional | +| `portrait_path` | `str \| None` | optional | +| `family_search_id` | `str \| None` | nullable unique | +| `metadata_` | `dict[str, JsonValue] \| None` | stored as DB column `metadata` (`JSONBCompat`) | +| `created_at` | `datetime` | default now | +| `updated_at` | `datetime` | default now, onupdate | -### Why status enums are narrow +### `DocumentPerson` -- Restricting `JobStatus` and `JobSourceStatus` keeps lifecycle transitions explicit and testable. -- Queue/state transitions and terminal derivation logic remain deterministic across worker and UI flows. +| Field | Type | Notes | +| :--- | :--- | :--- | +| `id` | `UUID` | PK | +| `document_id` | `UUID` | FK -> `document.id`, indexed | +| `person_id` | `UUID` | FK -> `person.id`, indexed | +| `role_id` | `UUID` | FK -> `person_role.id`, indexed | +| `created_at` | `datetime` | default now | +| `updated_at` | `datetime` | default now, onupdate | -## Media Storage Semantics +Constraint: +- `UniqueConstraint(document_id, person_id)` named `uq_document_person` -1. `Source.storage_path` references canonical stored bytes used by processing. -2. Canonical stored bytes may reflect ingest-time normalization. -3. File hash and size fields describe canonical stored bytes. +### `Job` -## Query and Loading Requirements +| Field | Type | Notes | +| :--- | :--- | :--- | +| `id` | `UUID` | PK | +| `document_id` | `UUID` | FK -> `document.id`, indexed | +| `status` | `JobStatus` | non-null enum (stored as enum values) | +| `retry_count` | `int` | default `0`, `ge=0` | +| `purpose` | `JobPurpose` | non-null enum, default `transcription` | +| `date_created` | `datetime` | default now | +| `date_updated` | `datetime` | default now, onupdate | +| `provider` | `str \| None` | optional | +| `model` | `str \| None` | optional | +| `prompt_name` | `str \| None` | optional | +| `prompt_hash` | `str \| None` | optional | +| `system_prompt` | `str \| None` | optional | +| `user_prompt` | `str \| None` | optional | +| `temperature` | `float \| None` | optional | +| `top_p` | `float \| None` | optional | -- Relationship access from service/UI layers must use explicit eager loading patterns compatible with `lazy="raise"`. -- Candidate-attempt views should select latest/selected attempts explicitly; do not rely on implicit lazy traversal. +Index: +- `Index("ix_job_status_date_created", "status", "date_created")` -## Evolution and Migration Policy +### `Source` -- Additive schema evolution is preferred for evidence-bearing records. -- Deprecated semantics should be removed only when model enums, service logic, tests, and docs are updated together. -- Historical V4.x schema discussions are archived under tag `docs-v4x-archive`; this file is the active contract. +| Field | Type | Notes | +| :--- | :--- | :--- | +| `id` | `UUID` | PK | +| `document_id` | `UUID` | FK -> `document.id`, indexed | +| `page_number` | `int` | default `1`, `ge=1` | +| `upload_name` | `str` | required | +| `filename` | `str` | required | +| `file_path` | `str` | required | +| `file_hash` | `str` | required | +| `file_size_bytes` | `int` | `BigInteger`, non-null | +| `raw_transcription` | `str \| None` | projection field | +| `preferred_execution_attempt_id` | `UUID \| None` | nullable FK -> `execution_attempt.id`, indexed (`use_alter`) | +| `revised_text` | `str \| None` | optional human revision | +| `date_uploaded` | `datetime` | default now | +| `date_revised` | `datetime \| None` | optional | + +### `JobSource` + +| Field | Type | Notes | +| :--- | :--- | :--- | +| `id` | `UUID` | PK | +| `job_id` | `UUID` | FK -> `job.id`, indexed | +| `source_id` | `UUID` | FK -> `source.id`, indexed | +| `status` | `JobSourceStatus` | non-null enum, default `pending` | + +### `ExecutionAttempt` + +| Field | Type | Notes | +| :--- | :--- | :--- | +| `id` | `UUID` | PK | +| `job_source_id` | `UUID` | FK -> `job_source.id`, indexed | +| `job_id` | `UUID` | FK -> `job.id`, indexed | +| `source_id` | `UUID` | FK -> `source.id`, indexed | +| `attempt_number` | `int` | `ge=1` | +| `status` | `JobSourceStatus` | non-null enum, value-stable with `JobSource.status` | +| `provider` | `str` | required | +| `model` | `str \| None` | optional | +| `request_manifest` | `dict[str, JsonValue] \| None` | JSONBCompat | +| `request_manifest_sha256` | `str \| None` | optional | +| `request_manifest_schema_version` | `str \| None` | optional | +| `response_received` | `bool` | default `False` | +| `transport_status_code` | `int \| None` | optional | +| `transport_body` | `bytes \| None` | LargeBinary | +| `transport_content_type` | `str \| None` | optional | +| `transport_content_encoding` | `str \| None` | optional | +| `transport_safe_headers` | `dict[str, JsonValue] \| None` | JSONBCompat | +| `router_request_id` | `str \| None` | optional | +| `router_generation_id` | `str \| None` | optional | +| `sdk_response_snapshot` | `dict[str, JsonValue] \| None` | JSONBCompat | +| `normalized_metadata` | `dict[str, JsonValue] \| None` | JSONBCompat | +| `software_context` | `dict[str, JsonValue] \| None` | JSONBCompat | +| `raw_transcription` | `str \| None` | optional | +| `error_category` | `str \| None` | optional | +| `error_detail` | `str \| None` | optional | +| `failure_phase` | `str \| None` | optional | +| `started_at` | `datetime` | required | +| `finished_at` | `datetime` | required | +| `duration_ms` | `int` | `ge=0` | +| `created_at` | `datetime` | default now | + +Constraint: +- `UniqueConstraint(job_id, source_id, attempt_number)` named `uq_execution_attempt_number` + +## Relationship Loading Contract + +- Most ORM relationships are configured with `lazy="raise"`. +- `JobSource.execution_attempts` is intentionally `lazy="noload"` with ordered attempts. +- Service/UI read paths must explicitly eager-load required relationships before access. + +## Persistence Invariants (Ground Truth) + +1. `ExecutionAttempt` is append-only runtime evidence. +2. `JobSource.status` represents queue/projection execution state and is not a full evidence container. +3. `Source.raw_transcription` is a mutable projection and not authoritative attempt history. +4. `Job` terminal status derives from page outcomes (`JobSource` state), not from a separate summary table. +5. `DocumentType.semantic_key` and `PersonRole.semantic_key` are nullable-unique semantic identifiers. ## Cross-Reference - [System Architecture](architecture_v4.md) - [System Requirements](requirements_v4.md) - [Error Handling Policy](error_handling_v4.md) +- [AI Evidence and Provenance Invariant](../invariant/ai_evidence_and_provenance.md)