bbchops/transcription
1
0
Fork 0
generated from john/python-template
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
be3d0b7ee052de238fde8c7589f99d7c79c21665
transcription/docs/schema.md
T
Jim Lancaster 7eca9fe7dc
Quality Gate / gate (push) Successful in 1m27s
Details
V6.2 Add GEDCOM data
2026-09-03 05:34:13 -05:00

392 lines
14 KiB
Markdown
Raw Blame History

# Data Model and Persistence Schema (Current Baseline: V6.1)
This document is the field-accurate V6.1 schema contract aligned to `src/transcription/db/models.py`.
## Source of Truth Anchors
- `src/transcription/db/models.py` (status and purpose enums, including maintenance lifecycle enums)
- `src/transcription/db/models.py:80-120` (`DocumentType`, `PersonRole`)
- `src/transcription/db/models.py:122-172` (`Tag`, `Document`)
- `src/transcription/db/models.py` (`Person`, `GenealogyPerson`, `GenealogyFamily`, `GenealogyFamilyChild`, `GenealogyCitation`)
- `src/transcription/db/models.py` (`Photo`, `DocumentPerson`, `DocumentTag`)
- `src/transcription/db/models.py:285-347` (`Job`)
- `src/transcription/db/models.py` (`MaintenanceRun`)
- `src/transcription/db/models.py:350-462` (`Source`, `JobSource`)
- `src/transcription/db/models.py:465-522` (`ExecutionAttempt`)
## Entity Relationship Overview
```mermaid
erDiagram
DocumentType ||--o{ Document : classifies
Document ||--o{ Job : has
Document ||--o{ Source : has
Document ||--o{ DocumentPerson : links
Document ||--o{ DocumentTag : tagged
Person ||--o{ DocumentPerson : links
Person ||--o{ PersonTag : tagged
Person ||--o{ Photo : owns
GenealogyPerson ||--o{ GenealogyFamily : husband
GenealogyPerson ||--o{ GenealogyFamily : wife
GenealogyPerson ||--o{ GenealogyFamilyChild : child
GenealogyFamily ||--o{ GenealogyFamilyChild : includes
GenealogyPerson ||--o{ GenealogyCitation : cited
GenealogyFamily ||--o{ GenealogyCitation : cited
Document ||--o{ GenealogyCitation : evidence
PersonRole ||--o{ DocumentPerson : labels
Tag ||--o{ DocumentTag : labels
Tag ||--o{ PersonTag : labels
Job ||--o{ JobSource : includes
Source ||--o{ JobSource : participates
JobSource ||--o{ ExecutionAttempt : attempts
MaintenanceRun {
uuid id PK
}
```
## Authoritative Enumerations
### JobStatus
- `queued`
- `processing`
- `transcribed`
- `partial_success`
- `failed`
### JobSourceStatus
- `pending`
- `transcribed`
- `failed`
- `cancelled`
### JobPurpose
- `transcription`
- `retranscription`
### MaintenanceJobType
- `backup`
- `storage_reconciliation`
- `gedcom_import`
### MaintenanceRunStatus
- `queued`
- `processing`
- `succeeded`
- `failed`
## Field-Accurate Table Contracts
### `DocumentType`
| Field | Type | Notes |
| :--- | :--- | :--- |
| `id` | `UUID` | PK |
| `semantic_key` | `str \| None` | nullable unique, indexed |
| `label` | `str` | required |
| `normalized_label` | `str` | unique, indexed |
| `is_active` | `bool` | default `True` |
| `created_at` | `datetime` | default now |
| `updated_at` | `datetime` | default now, onupdate |
### `PersonRole`
| Field | Type | Notes |
| :--- | :--- | :--- |
| `id` | `UUID` | PK |
| `semantic_key` | `str \| None` | nullable unique, indexed |
| `label` | `str` | required |
| `normalized_label` | `str` | unique, indexed |
| `is_active` | `bool` | default `True` |
| `created_at` | `datetime` | default now |
| `updated_at` | `datetime` | default now, onupdate |
### `Tag`
| Field | Type | Notes |
| :--- | :--- | :--- |
| `id` | `UUID` | PK |
| `semantic_key` | `str \| None` | nullable unique, indexed |
| `label` | `str` | required |
| `normalized_label` | `str` | unique, indexed |
| `is_active` | `bool` | default `True` |
| `created_at` | `datetime` | default now |
| `updated_at` | `datetime` | default now, onupdate |
### `Document`
| Field | Type | Notes |
| :--- | :--- | :--- |
| `id` | `UUID` | PK |
| `name` | `str` | required |
| `document_type_id` | `UUID \| None` | FK -> `document_type.id`, indexed |
| `document_date` | `date \| None` | optional |
| `document_date_raw` | `str \| None` | optional |
| `location_created` | `str \| None` | optional |
| `notes` | `str \| None` | optional |
| `archive_identifier` | `str \| None` | optional |
| `created_at` | `datetime` | default now |
| `updated_at` | `datetime` | default now, onupdate |
### `Person`
| Field | Type | Notes |
| :--- | :--- | :--- |
| `id` | `UUID` | PK |
| `last_name` | `str` | required |
| `given_names` | `str` | required |
| `birth_date` | `date \| None` | optional |
| `birth_date_raw` | `str \| None` | optional |
| `birth_place` | `str \| None` | optional |
| `death_date` | `date \| None` | optional |
| `death_date_raw` | `str \| None` | optional |
| `death_place` | `str \| None` | optional |
| `biography` | `str \| None` | optional |
| `family_search_id` | `str \| None` | nullable unique |
| `metadata_` | `dict[str, JsonValue] \| None` | stored as DB column `metadata` (`JSONBCompat`) |
| `created_at` | `datetime` | default now |
| `updated_at` | `datetime` | default now, onupdate |
### `GenealogyPerson`
| Field | Type | Notes |
| :--- | :--- | :--- |
| `id` | `UUID` | PK |
| `fs_id` | `str` | unique, indexed FamilySearch identifier |
| `full_name` | `str` | required |
| `birth_date` | `date \| None` | optional |
| `birth_date_raw` | `str \| None` | optional |
| `birth_place` | `str \| None` | optional |
| `death_date` | `date \| None` | optional |
| `death_date_raw` | `str \| None` | optional |
| `death_place` | `str \| None` | optional |
| `created_at` | `datetime` | default now |
| `updated_at` | `datetime` | default now, onupdate |
### `GenealogyFamily`
| Field | Type | Notes |
| :--- | :--- | :--- |
| `id` | `UUID` | PK |
| `fs_family_id` | `str` | unique, indexed FamilySearch family identifier |
| `husband_id` | `UUID \| None` | nullable FK -> `genealogy_person.id`, indexed |
| `wife_id` | `UUID \| None` | nullable FK -> `genealogy_person.id`, indexed |
| `marriage_date` | `date \| None` | optional |
| `marriage_date_raw` | `str \| None` | optional |
| `marriage_place` | `str \| None` | optional |
| `created_at` | `datetime` | default now |
| `updated_at` | `datetime` | default now, onupdate |
### `GenealogyFamilyChild`
| Field | Type | Notes |
| :--- | :--- | :--- |
| `id` | `UUID` | PK |
| `family_id` | `UUID` | FK -> `genealogy_family.id`, indexed |
| `child_id` | `UUID` | FK -> `genealogy_person.id`, indexed |
| `relationship_type` | `str \| None` | optional |
| `created_at` | `datetime` | default now |
Constraint:
- `UniqueConstraint(family_id, child_id)` named `uq_genealogy_family_child`
### `GenealogyCitation`
| Field | Type | Notes |
| :--- | :--- | :--- |
| `id` | `UUID` | PK |
| `genealogy_person_id` | `UUID \| None` | nullable FK -> `genealogy_person.id`, indexed |
| `genealogy_family_id` | `UUID \| None` | nullable FK -> `genealogy_family.id`, indexed |
| `fact_type` | `GenealogyCitationFactType` | enum: `birth`, `death`, `marriage`, `other` |
| `raw_citation_text` | `str` | required raw GEDCOM citation text |
| `source_kind` | `GenealogyCitationSourceKind` | enum: `familysearch_imported`, `transcription_evidence` |
| `document_id` | `UUID \| None` | nullable FK -> `document.id`, indexed |
| `created_at` | `datetime` | default now |
### `Photo`
| Field | Type | Notes |
| :--- | :--- | :--- |
| `id` | `UUID` | PK |
| `person_id` | `UUID \| None` | nullable FK -> `person.id`, indexed (`NULL` = homepage photo) |
| `path` | `str` | required upload-root-relative POSIX path (`photos/...`) |
| `description` | `str \| None` | optional |
| `is_primary` | `bool` | default `False`; owner-level "featured/primary" marker |
| `created_at` | `datetime` | default now |
| `updated_at` | `datetime` | default now, onupdate |
### `DocumentPerson`
| Field | Type | Notes |
| :--- | :--- | :--- |
| `id` | `UUID` | PK |
| `document_id` | `UUID` | FK -> `document.id`, indexed |
| `person_id` | `UUID` | FK -> `person.id`, indexed |
| `role_id` | `UUID` | FK -> `person_role.id`, indexed |
| `created_at` | `datetime` | default now |
| `updated_at` | `datetime` | default now, onupdate |
Constraint:
- `UniqueConstraint(document_id, person_id)` named `uq_document_person`
### `DocumentTag`
| Field | Type | Notes |
| :--- | :--- | :--- |
| `id` | `UUID` | PK |
| `document_id` | `UUID` | FK -> `document.id`, indexed |
| `tag_id` | `UUID` | FK -> `tag.id`, indexed |
| `created_at` | `datetime` | default now |
| `updated_at` | `datetime` | default now, onupdate |
Constraint:
- `UniqueConstraint(document_id, tag_id)` named `uq_document_tag`
### `PersonTag`
| Field | Type | Notes |
| :--- | :--- | :--- |
| `id` | `UUID` | PK |
| `person_id` | `UUID` | FK -> `person.id`, indexed |
| `tag_id` | `UUID` | FK -> `tag.id`, indexed |
| `created_at` | `datetime` | default now |
| `updated_at` | `datetime` | default now, onupdate |
Constraint:
- `UniqueConstraint(person_id, tag_id)` named `uq_person_tag`
### `Job`
| Field | Type | Notes |
| :--- | :--- | :--- |
| `id` | `UUID` | PK |
| `document_id` | `UUID` | FK -> `document.id`, indexed |
| `status` | `JobStatus` | non-null enum (stored as enum values) |
| `retry_count` | `int` | default `0`, `ge=0` |
| `purpose` | `JobPurpose` | non-null enum, default `transcription` |
| `date_created` | `datetime` | default now |
| `date_updated` | `datetime` | default now, onupdate |
| `provider` | `str \| None` | optional |
| `model` | `str \| None` | optional |
| `prompt_name` | `str \| None` | optional |
| `prompt_hash` | `str \| None` | optional |
| `system_prompt` | `str \| None` | optional |
| `user_prompt` | `str \| None` | optional |
| `temperature` | `float \| None` | optional |
| `top_p` | `float \| None` | optional |
Index:
- `Index("ix_job_status_date_created", "status", "date_created")`
### `MaintenanceRun`
| Field | Type | Notes |
| :--- | :--- | :--- |
| `id` | `UUID` | PK |
| `job_type` | `MaintenanceJobType` | non-null enum |
| `status` | `MaintenanceRunStatus` | non-null enum, default `queued` |
| `started_at` | `datetime \| None` | optional |
| `finished_at` | `datetime \| None` | optional |
| `triggered_by` | `str \| None` | optional |
| `summary` | `str \| None` | optional |
| `log_path` | `str \| None` | optional, log-root-relative POSIX path |
| `error_detail` | `str \| None` | optional internal failure detail |
| `created_at` | `datetime` | default now |
| `updated_at` | `datetime` | default now, onupdate |
### `Source`
| Field | Type | Notes |
| :--- | :--- | :--- |
| `id` | `UUID` | PK |
| `document_id` | `UUID` | FK -> `document.id`, indexed |
| `page_number` | `int` | default `1`, `ge=1` |
| `upload_name` | `str` | required |
| `filename` | `str` | required |
| `file_path` | `str` | required upload-root-relative POSIX path (`documents/...`) |
| `file_hash` | `str` | required |
| `file_size_bytes` | `int` | `BigInteger`, non-null |
| `raw_transcription` | `str \| None` | projection field |
| `preferred_execution_attempt_id` | `UUID \| None` | nullable FK -> `execution_attempt.id`, indexed (`use_alter`) |
| `revised_text` | `str \| None` | optional human revision |
| `date_uploaded` | `datetime` | default now |
| `date_revised` | `datetime \| None` | optional |
### `JobSource`
| Field | Type | Notes |
| :--- | :--- | :--- |
| `id` | `UUID` | PK |
| `job_id` | `UUID` | FK -> `job.id`, indexed |
| `source_id` | `UUID` | FK -> `source.id`, indexed |
| `status` | `JobSourceStatus` | non-null enum, default `pending` |
Constraint:
- `UniqueConstraint(job_id, source_id)` named `uq_job_source_job_source`
Runtime reconciliation:
- Startup database operations remove retired V4.6 `job_source` evidence columns (`raw_transcription`, `ai_metadata`, `raw_api_response`, `error_detail`, `executed_at`) when present so persisted schema matches this contract.
### `ExecutionAttempt`
| Field | Type | Notes |
| :--- | :--- | :--- |
| `id` | `UUID` | PK |
| `job_source_id` | `UUID` | FK -> `job_source.id`, indexed |
| `job_id` | `UUID` | FK -> `job.id`, indexed |
| `source_id` | `UUID` | FK -> `source.id`, indexed |
| `attempt_number` | `int` | `ge=1` |
| `status` | `JobSourceStatus` | non-null enum, value-stable with `JobSource.status` |
| `provider` | `str` | required |
| `model` | `str \| None` | optional |
| `request_manifest` | `dict[str, JsonValue] \| None` | JSONBCompat |
| `request_manifest_sha256` | `str \| None` | optional |
| `request_manifest_schema_version` | `str \| None` | optional |
| `response_received` | `bool` | default `False` |
| `transport_status_code` | `int \| None` | optional |
| `transport_body` | `bytes \| None` | LargeBinary |
| `transport_content_type` | `str \| None` | optional |
| `transport_content_encoding` | `str \| None` | optional |
| `transport_safe_headers` | `dict[str, JsonValue] \| None` | JSONBCompat |
| `router_request_id` | `str \| None` | optional |
| `router_generation_id` | `str \| None` | optional |
| `sdk_response_snapshot` | `dict[str, JsonValue] \| None` | JSONBCompat |
| `normalized_metadata` | `dict[str, JsonValue] \| None` | JSONBCompat; may include app-namespaced `processing_timing` (`provider_call_duration_ms`, `processing_duration_ms`) |
| `software_context` | `dict[str, JsonValue] \| None` | JSONBCompat |
| `raw_transcription` | `str \| None` | optional |
| `error_category` | `str \| None` | optional |
| `error_detail` | `str \| None` | optional |
| `failure_phase` | `str \| None` | optional |
| `started_at` | `datetime` | required |
| `finished_at` | `datetime` | required |
| `duration_ms` | `int` | `ge=0` |
| `created_at` | `datetime` | default now |
Constraint:
- `UniqueConstraint(job_id, source_id, attempt_number)` named `uq_execution_attempt_number`
## Relationship Loading Contract
- Most ORM relationships are configured with `lazy="raise"`.
- `JobSource.execution_attempts` is intentionally `lazy="noload"` with ordered attempts.
- Service/UI read paths must explicitly eager-load required relationships before access.
## Persistence Invariants (Ground Truth)
1. `ExecutionAttempt` is append-only runtime evidence.
2. `JobSource.status` represents queue/projection execution state and is not a full evidence container.
3. `Source.raw_transcription` is a mutable projection and not authoritative attempt history.
4. `Job` terminal status derives from page outcomes (`JobSource` state), not from a separate summary table.
5. `DocumentType.semantic_key` and `PersonRole.semantic_key` are nullable-unique semantic identifiers.
## Cross-Reference
- [System Architecture](architecture.md)
- [System Requirements](requirements.md)
- [Error Handling Policy](error_handling.md)
- [AI Evidence and Provenance Invariant](./invariant/ai_evidence_and_provenance.md)
Reference in New Issue View Git Blame Copy Permalink