generated from john/python-template
295 lines
10 KiB
Markdown
295 lines
10 KiB
Markdown
# Data Model and Persistence Schema (Current Baseline: V5.1)
|
|
|
|
This document is the field-accurate V5.1 schema contract aligned to `src/transcription/db/models.py`.
|
|
|
|
## Source of Truth Anchors
|
|
|
|
- `src/transcription/db/models.py:60-78` (status and purpose enums)
|
|
- `src/transcription/db/models.py:80-120` (`DocumentType`, `PersonRole`)
|
|
- `src/transcription/db/models.py:122-172` (`Tag`, `Document`)
|
|
- `src/transcription/db/models.py:175-281` (`Person`, `Photo`, `DocumentPerson`, `DocumentTag`)
|
|
- `src/transcription/db/models.py:285-347` (`Job`)
|
|
- `src/transcription/db/models.py:350-462` (`Source`, `JobSource`)
|
|
- `src/transcription/db/models.py:465-522` (`ExecutionAttempt`)
|
|
|
|
## Entity Relationship Overview
|
|
|
|
```mermaid
|
|
erDiagram
|
|
DocumentType ||--o{ Document : classifies
|
|
Document ||--o{ Job : has
|
|
Document ||--o{ Source : has
|
|
Document ||--o{ DocumentPerson : links
|
|
Document ||--o{ DocumentTag : tagged
|
|
Person ||--o{ DocumentPerson : links
|
|
Person ||--o{ PersonTag : tagged
|
|
Person ||--o{ Photo : owns
|
|
PersonRole ||--o{ DocumentPerson : labels
|
|
Tag ||--o{ DocumentTag : labels
|
|
Tag ||--o{ PersonTag : labels
|
|
Job ||--o{ JobSource : includes
|
|
Source ||--o{ JobSource : participates
|
|
JobSource ||--o{ ExecutionAttempt : attempts
|
|
```
|
|
|
|
## Authoritative Enumerations
|
|
|
|
### JobStatus
|
|
|
|
- `queued`
|
|
- `processing`
|
|
- `transcribed`
|
|
- `partial_success`
|
|
- `failed`
|
|
|
|
### JobSourceStatus
|
|
|
|
- `pending`
|
|
- `transcribed`
|
|
- `failed`
|
|
- `cancelled`
|
|
|
|
### JobPurpose
|
|
|
|
- `transcription`
|
|
- `retranscription`
|
|
|
|
## Field-Accurate Table Contracts
|
|
|
|
### `DocumentType`
|
|
|
|
| Field | Type | Notes |
|
|
| :--- | :--- | :--- |
|
|
| `id` | `UUID` | PK |
|
|
| `semantic_key` | `str \| None` | nullable unique, indexed |
|
|
| `label` | `str` | required |
|
|
| `normalized_label` | `str` | unique, indexed |
|
|
| `is_active` | `bool` | default `True` |
|
|
| `created_at` | `datetime` | default now |
|
|
| `updated_at` | `datetime` | default now, onupdate |
|
|
|
|
### `PersonRole`
|
|
|
|
| Field | Type | Notes |
|
|
| :--- | :--- | :--- |
|
|
| `id` | `UUID` | PK |
|
|
| `semantic_key` | `str \| None` | nullable unique, indexed |
|
|
| `label` | `str` | required |
|
|
| `normalized_label` | `str` | unique, indexed |
|
|
| `is_active` | `bool` | default `True` |
|
|
| `created_at` | `datetime` | default now |
|
|
| `updated_at` | `datetime` | default now, onupdate |
|
|
|
|
### `Tag`
|
|
|
|
| Field | Type | Notes |
|
|
| :--- | :--- | :--- |
|
|
| `id` | `UUID` | PK |
|
|
| `semantic_key` | `str \| None` | nullable unique, indexed |
|
|
| `label` | `str` | required |
|
|
| `normalized_label` | `str` | unique, indexed |
|
|
| `is_active` | `bool` | default `True` |
|
|
| `created_at` | `datetime` | default now |
|
|
| `updated_at` | `datetime` | default now, onupdate |
|
|
|
|
### `Document`
|
|
|
|
| Field | Type | Notes |
|
|
| :--- | :--- | :--- |
|
|
| `id` | `UUID` | PK |
|
|
| `name` | `str` | required |
|
|
| `document_type_id` | `UUID \| None` | FK -> `document_type.id`, indexed |
|
|
| `document_date` | `date \| None` | optional |
|
|
| `document_date_raw` | `str \| None` | optional |
|
|
| `location_created` | `str \| None` | optional |
|
|
| `notes` | `str \| None` | optional |
|
|
| `archive_identifier` | `str \| None` | optional |
|
|
| `created_at` | `datetime` | default now |
|
|
| `updated_at` | `datetime` | default now, onupdate |
|
|
|
|
### `Person`
|
|
|
|
| Field | Type | Notes |
|
|
| :--- | :--- | :--- |
|
|
| `id` | `UUID` | PK |
|
|
| `last_name` | `str` | required |
|
|
| `given_names` | `str` | required |
|
|
| `birth_date` | `date \| None` | optional |
|
|
| `birth_date_raw` | `str \| None` | optional |
|
|
| `birth_place` | `str \| None` | optional |
|
|
| `death_date` | `date \| None` | optional |
|
|
| `death_date_raw` | `str \| None` | optional |
|
|
| `death_place` | `str \| None` | optional |
|
|
| `biography` | `str \| None` | optional |
|
|
| `family_search_id` | `str \| None` | nullable unique |
|
|
| `metadata_` | `dict[str, JsonValue] \| None` | stored as DB column `metadata` (`JSONBCompat`) |
|
|
| `created_at` | `datetime` | default now |
|
|
| `updated_at` | `datetime` | default now, onupdate |
|
|
|
|
### `Photo`
|
|
|
|
| Field | Type | Notes |
|
|
| :--- | :--- | :--- |
|
|
| `id` | `UUID` | PK |
|
|
| `person_id` | `UUID \| None` | nullable FK -> `person.id`, indexed (`NULL` = homepage photo) |
|
|
| `path` | `str` | required upload-root-relative POSIX path (`photos/...`) |
|
|
| `description` | `str \| None` | optional |
|
|
| `is_primary` | `bool` | default `False`; owner-level "featured/primary" marker |
|
|
| `created_at` | `datetime` | default now |
|
|
| `updated_at` | `datetime` | default now, onupdate |
|
|
|
|
### `DocumentPerson`
|
|
|
|
| Field | Type | Notes |
|
|
| :--- | :--- | :--- |
|
|
| `id` | `UUID` | PK |
|
|
| `document_id` | `UUID` | FK -> `document.id`, indexed |
|
|
| `person_id` | `UUID` | FK -> `person.id`, indexed |
|
|
| `role_id` | `UUID` | FK -> `person_role.id`, indexed |
|
|
| `created_at` | `datetime` | default now |
|
|
| `updated_at` | `datetime` | default now, onupdate |
|
|
|
|
Constraint:
|
|
- `UniqueConstraint(document_id, person_id)` named `uq_document_person`
|
|
|
|
### `DocumentTag`
|
|
|
|
| Field | Type | Notes |
|
|
| :--- | :--- | :--- |
|
|
| `id` | `UUID` | PK |
|
|
| `document_id` | `UUID` | FK -> `document.id`, indexed |
|
|
| `tag_id` | `UUID` | FK -> `tag.id`, indexed |
|
|
| `created_at` | `datetime` | default now |
|
|
| `updated_at` | `datetime` | default now, onupdate |
|
|
|
|
Constraint:
|
|
- `UniqueConstraint(document_id, tag_id)` named `uq_document_tag`
|
|
|
|
### `PersonTag`
|
|
|
|
| Field | Type | Notes |
|
|
| :--- | :--- | :--- |
|
|
| `id` | `UUID` | PK |
|
|
| `person_id` | `UUID` | FK -> `person.id`, indexed |
|
|
| `tag_id` | `UUID` | FK -> `tag.id`, indexed |
|
|
| `created_at` | `datetime` | default now |
|
|
| `updated_at` | `datetime` | default now, onupdate |
|
|
|
|
Constraint:
|
|
- `UniqueConstraint(person_id, tag_id)` named `uq_person_tag`
|
|
|
|
### `Job`
|
|
|
|
| Field | Type | Notes |
|
|
| :--- | :--- | :--- |
|
|
| `id` | `UUID` | PK |
|
|
| `document_id` | `UUID` | FK -> `document.id`, indexed |
|
|
| `status` | `JobStatus` | non-null enum (stored as enum values) |
|
|
| `retry_count` | `int` | default `0`, `ge=0` |
|
|
| `purpose` | `JobPurpose` | non-null enum, default `transcription` |
|
|
| `date_created` | `datetime` | default now |
|
|
| `date_updated` | `datetime` | default now, onupdate |
|
|
| `provider` | `str \| None` | optional |
|
|
| `model` | `str \| None` | optional |
|
|
| `prompt_name` | `str \| None` | optional |
|
|
| `prompt_hash` | `str \| None` | optional |
|
|
| `system_prompt` | `str \| None` | optional |
|
|
| `user_prompt` | `str \| None` | optional |
|
|
| `temperature` | `float \| None` | optional |
|
|
| `top_p` | `float \| None` | optional |
|
|
|
|
Index:
|
|
- `Index("ix_job_status_date_created", "status", "date_created")`
|
|
|
|
### `Source`
|
|
|
|
| Field | Type | Notes |
|
|
| :--- | :--- | :--- |
|
|
| `id` | `UUID` | PK |
|
|
| `document_id` | `UUID` | FK -> `document.id`, indexed |
|
|
| `page_number` | `int` | default `1`, `ge=1` |
|
|
| `upload_name` | `str` | required |
|
|
| `filename` | `str` | required |
|
|
| `file_path` | `str` | required upload-root-relative POSIX path (`documents/...`) |
|
|
| `file_hash` | `str` | required |
|
|
| `file_size_bytes` | `int` | `BigInteger`, non-null |
|
|
| `raw_transcription` | `str \| None` | projection field |
|
|
| `preferred_execution_attempt_id` | `UUID \| None` | nullable FK -> `execution_attempt.id`, indexed (`use_alter`) |
|
|
| `revised_text` | `str \| None` | optional human revision |
|
|
| `date_uploaded` | `datetime` | default now |
|
|
| `date_revised` | `datetime \| None` | optional |
|
|
|
|
### `JobSource`
|
|
|
|
| Field | Type | Notes |
|
|
| :--- | :--- | :--- |
|
|
| `id` | `UUID` | PK |
|
|
| `job_id` | `UUID` | FK -> `job.id`, indexed |
|
|
| `source_id` | `UUID` | FK -> `source.id`, indexed |
|
|
| `status` | `JobSourceStatus` | non-null enum, default `pending` |
|
|
|
|
Constraint:
|
|
- `UniqueConstraint(job_id, source_id)` named `uq_job_source_job_source`
|
|
|
|
Runtime reconciliation:
|
|
- Startup database operations remove retired V4.6 `job_source` evidence columns (`raw_transcription`, `ai_metadata`, `raw_api_response`, `error_detail`, `executed_at`) when present so persisted schema matches this contract.
|
|
|
|
### `ExecutionAttempt`
|
|
|
|
| Field | Type | Notes |
|
|
| :--- | :--- | :--- |
|
|
| `id` | `UUID` | PK |
|
|
| `job_source_id` | `UUID` | FK -> `job_source.id`, indexed |
|
|
| `job_id` | `UUID` | FK -> `job.id`, indexed |
|
|
| `source_id` | `UUID` | FK -> `source.id`, indexed |
|
|
| `attempt_number` | `int` | `ge=1` |
|
|
| `status` | `JobSourceStatus` | non-null enum, value-stable with `JobSource.status` |
|
|
| `provider` | `str` | required |
|
|
| `model` | `str \| None` | optional |
|
|
| `request_manifest` | `dict[str, JsonValue] \| None` | JSONBCompat |
|
|
| `request_manifest_sha256` | `str \| None` | optional |
|
|
| `request_manifest_schema_version` | `str \| None` | optional |
|
|
| `response_received` | `bool` | default `False` |
|
|
| `transport_status_code` | `int \| None` | optional |
|
|
| `transport_body` | `bytes \| None` | LargeBinary |
|
|
| `transport_content_type` | `str \| None` | optional |
|
|
| `transport_content_encoding` | `str \| None` | optional |
|
|
| `transport_safe_headers` | `dict[str, JsonValue] \| None` | JSONBCompat |
|
|
| `router_request_id` | `str \| None` | optional |
|
|
| `router_generation_id` | `str \| None` | optional |
|
|
| `sdk_response_snapshot` | `dict[str, JsonValue] \| None` | JSONBCompat |
|
|
| `normalized_metadata` | `dict[str, JsonValue] \| None` | JSONBCompat; may include app-namespaced `processing_timing` (`provider_call_duration_ms`, `processing_duration_ms`) |
|
|
| `software_context` | `dict[str, JsonValue] \| None` | JSONBCompat |
|
|
| `raw_transcription` | `str \| None` | optional |
|
|
| `error_category` | `str \| None` | optional |
|
|
| `error_detail` | `str \| None` | optional |
|
|
| `failure_phase` | `str \| None` | optional |
|
|
| `started_at` | `datetime` | required |
|
|
| `finished_at` | `datetime` | required |
|
|
| `duration_ms` | `int` | `ge=0` |
|
|
| `created_at` | `datetime` | default now |
|
|
|
|
Constraint:
|
|
- `UniqueConstraint(job_id, source_id, attempt_number)` named `uq_execution_attempt_number`
|
|
|
|
## Relationship Loading Contract
|
|
|
|
- Most ORM relationships are configured with `lazy="raise"`.
|
|
- `JobSource.execution_attempts` is intentionally `lazy="noload"` with ordered attempts.
|
|
- Service/UI read paths must explicitly eager-load required relationships before access.
|
|
|
|
## Persistence Invariants (Ground Truth)
|
|
|
|
1. `ExecutionAttempt` is append-only runtime evidence.
|
|
2. `JobSource.status` represents queue/projection execution state and is not a full evidence container.
|
|
3. `Source.raw_transcription` is a mutable projection and not authoritative attempt history.
|
|
4. `Job` terminal status derives from page outcomes (`JobSource` state), not from a separate summary table.
|
|
5. `DocumentType.semantic_key` and `PersonRole.semantic_key` are nullable-unique semantic identifiers.
|
|
|
|
## Cross-Reference
|
|
|
|
- [System Architecture](architecture.md)
|
|
- [System Requirements](requirements.md)
|
|
- [Error Handling Policy](error_handling.md)
|
|
- [AI Evidence and Provenance Invariant](./invariant/ai_evidence_and_provenance.md)
|