bbchops/transcription
1
0
Fork 0
generated from john/python-template
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

Phase 1 of Phase 1 results (I'm losing track of the phases) - Update the schema doc
Quality Gate / gate (push) Successful in 35s
Details

Browse Source
This commit is contained in:
Jim Lancaster
2026-08-19 18:29:20 -05:00
parent b6a5a89a84
commit de8cdb6e1a
1 changed files with 179 additions and 105 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/ver4/schema_v4.md
+179 -105
View File
@@ -1,83 +1,29 @@
# Data Model and Persistence Schema (Version 4) # Data Model and Persistence Schema (Version 4)
This schema reflects the current V4 persistence contract. This document is the field-accurate V4 schema contract aligned to `src/transcription/db/models.py`.
## Source of Truth Anchors
- `src/transcription/db/models.py:60-78` (status and purpose enums)
- `src/transcription/db/models.py:80-120` (`DocumentType`, `PersonRole`)
- `src/transcription/db/models.py:122-207` (`Document`, `Person`, `DocumentPerson`)
- `src/transcription/db/models.py:208-255` (`Job`)
- `src/transcription/db/models.py:273-386` (`Source`, `JobSource`)
- `src/transcription/db/models.py:387-445` (`ExecutionAttempt`)
## Entity Relationship Overview ## Entity Relationship Overview
```mermaid ```mermaid
erDiagram erDiagram
Document ||--o{ Source : has DocumentType ||--o{ Document : classifies
Document ||--o{ Job : has Document ||--o{ Job : has
Document ||--o{ Source : has
Document ||--o{ DocumentPerson : links Document ||--o{ DocumentPerson : links
Person ||--o{ DocumentPerson : links Person ||--o{ DocumentPerson : links
PersonRole ||--o{ DocumentPerson : labels
Job ||--o{ JobSource : includes Job ||--o{ JobSource : includes
Source ||--o{ JobSource : participates Source ||--o{ JobSource : participates
Source ||--o{ ExecutionAttempt : records JobSource ||--o{ ExecutionAttempt : attempts
Document {
uuid id PK
string title
uuid type_id FK
string language
datetime doc_date
string date_note
string comments
string location
datetime created_at
datetime updated_at
}
Source {
uuid id PK
uuid document_id FK
string original_name
string media_type
string storage_path
int file_size
string file_hash
string raw_transcription
datetime created_at
datetime updated_at
}
Job {
uuid id PK
uuid document_id FK
enum status
string prompt
json model_settings_json
datetime created_at
datetime updated_at
}
JobSource {
uuid job_id FK
uuid source_id FK
enum status
uuid selected_attempt_id FK
string error_message
datetime created_at
datetime updated_at
}
ExecutionAttempt {
uuid id PK
uuid source_id FK
uuid job_id FK
enum outcome
string provider_name
string provider_model
string request_manifest_hash
json request_manifest_json
json transport_evidence_json
string transcript_text
string error_category
string error_message
float duration_seconds
datetime started_at
datetime completed_at
datetime created_at
}
``` ```
## Authoritative Enumerations ## Authoritative Enumerations
@@ -97,64 +43,192 @@ erDiagram
- `failed` - `failed`
- `cancelled` - `cancelled`
## Aggregate Ownership ### JobPurpose
- `Document` aggregate: `Document`, linked `Source`, linked `DocumentPerson`. - `transcription`
- `Job` aggregate: `Job`, `JobSource` rows, selected-attempt pointers. - `retranscription`
- Evidence aggregate: append-only `ExecutionAttempt` rows keyed by `source_id` + `job_id`.
## Persistence Invariants ## Field-Accurate Table Contracts
1. `ExecutionAttempt` rows are immutable after creation, except explicit support fields reserved for compatibility migrations. ### `DocumentType`
2. `JobSource.status` is queue/projection state; it does not duplicate full attempt payload.
3. `Source.raw_transcription` is a projection, not the complete evidence record.
4. `Job` terminal status is derived from `JobSource` outcomes.
5. Registry semantic keys, when present, are immutable once created.
## Schema Design Rationale | Field | Type | Notes |
| :--- | :--- | :--- |
| `id` | `UUID` | PK |
| `semantic_key` | `str \| None` | nullable unique, indexed |
| `label` | `str` | required |
| `normalized_label` | `str` | unique, indexed |
| `is_active` | `bool` | default `True` |
| `created_at` | `datetime` | default now |
| `updated_at` | `datetime` | default now, onupdate |
### Why `ExecutionAttempt` exists alongside `JobSource` ### `PersonRole`
- `JobSource` is the mutable queue/projection row for workflow control and current-facing page outcome state. | Field | Type | Notes |
- `ExecutionAttempt` is the durable, append-only evidence timeline for each provider call. | :--- | :--- | :--- |
- Keeping both avoids overloading one table with competing concerns (queue state vs immutable audit history). | `id` | `UUID` | PK |
| `semantic_key` | `str \| None` | nullable unique, indexed |
| `label` | `str` | required |
| `normalized_label` | `str` | unique, indexed |
| `is_active` | `bool` | default `True` |
| `created_at` | `datetime` | default now |
| `updated_at` | `datetime` | default now, onupdate |
### Why `Source.raw_transcription` remains on `Source` ### `Document`
- `Source` needs a stable, current projection for UI and print behavior. | Field | Type | Notes |
- Projection reads are fast and direct, while deep historical inspection remains available through attempts. | :--- | :--- | :--- |
- Explicit candidate promotion updates the projection pointer without rewriting history. | `id` | `UUID` | PK |
| `name` | `str` | required |
| `document_type_id` | `UUID \| None` | FK -> `document_type.id`, indexed |
| `document_date` | `date \| None` | optional |
| `document_date_raw` | `str \| None` | optional |
| `location_created` | `str \| None` | optional |
| `notes` | `str \| None` | optional |
| `archive_identifier` | `str \| None` | optional |
| `created_at` | `datetime` | default now |
| `updated_at` | `datetime` | default now, onupdate |
### Why semantic registries use UUID identity plus optional semantic keys ### `Person`
- UUIDs are durable relationship identifiers for public/domain links. | Field | Type | Notes |
- Optional immutable semantic keys support protected built-ins without exposing internal meaning as external API identity. | :--- | :--- | :--- |
- Labels can evolve without breaking relationships. | `id` | `UUID` | PK |
| `full_name` | `str` | required |
| `display_name` | `str \| None` | optional |
| `maiden_name` | `str \| None` | optional |
| `birth_date` | `date \| None` | optional |
| `birth_date_raw` | `str \| None` | optional |
| `birth_place` | `str \| None` | optional |
| `death_date` | `date \| None` | optional |
| `death_date_raw` | `str \| None` | optional |
| `death_place` | `str \| None` | optional |
| `biography` | `str \| None` | optional |
| `portrait_path` | `str \| None` | optional |
| `family_search_id` | `str \| None` | nullable unique |
| `metadata_` | `dict[str, JsonValue] \| None` | stored as DB column `metadata` (`JSONBCompat`) |
| `created_at` | `datetime` | default now |
| `updated_at` | `datetime` | default now, onupdate |
### Why status enums are narrow ### `DocumentPerson`
- Restricting `JobStatus` and `JobSourceStatus` keeps lifecycle transitions explicit and testable. | Field | Type | Notes |
- Queue/state transitions and terminal derivation logic remain deterministic across worker and UI flows. | :--- | :--- | :--- |
| `id` | `UUID` | PK |
| `document_id` | `UUID` | FK -> `document.id`, indexed |
| `person_id` | `UUID` | FK -> `person.id`, indexed |
| `role_id` | `UUID` | FK -> `person_role.id`, indexed |
| `created_at` | `datetime` | default now |
| `updated_at` | `datetime` | default now, onupdate |
## Media Storage Semantics Constraint:
- `UniqueConstraint(document_id, person_id)` named `uq_document_person`
1. `Source.storage_path` references canonical stored bytes used by processing. ### `Job`
2. Canonical stored bytes may reflect ingest-time normalization.
3. File hash and size fields describe canonical stored bytes.
## Query and Loading Requirements | Field | Type | Notes |
| :--- | :--- | :--- |
| `id` | `UUID` | PK |
| `document_id` | `UUID` | FK -> `document.id`, indexed |
| `status` | `JobStatus` | non-null enum (stored as enum values) |
| `retry_count` | `int` | default `0`, `ge=0` |
| `purpose` | `JobPurpose` | non-null enum, default `transcription` |
| `date_created` | `datetime` | default now |
| `date_updated` | `datetime` | default now, onupdate |
| `provider` | `str \| None` | optional |
| `model` | `str \| None` | optional |
| `prompt_name` | `str \| None` | optional |
| `prompt_hash` | `str \| None` | optional |
| `system_prompt` | `str \| None` | optional |
| `user_prompt` | `str \| None` | optional |
| `temperature` | `float \| None` | optional |
| `top_p` | `float \| None` | optional |
- Relationship access from service/UI layers must use explicit eager loading patterns compatible with `lazy="raise"`. Index:
- Candidate-attempt views should select latest/selected attempts explicitly; do not rely on implicit lazy traversal. - `Index("ix_job_status_date_created", "status", "date_created")`
## Evolution and Migration Policy ### `Source`
- Additive schema evolution is preferred for evidence-bearing records. | Field | Type | Notes |
- Deprecated semantics should be removed only when model enums, service logic, tests, and docs are updated together. | :--- | :--- | :--- |
- Historical V4.x schema discussions are archived under tag `docs-v4x-archive`; this file is the active contract. | `id` | `UUID` | PK |
| `document_id` | `UUID` | FK -> `document.id`, indexed |
| `page_number` | `int` | default `1`, `ge=1` |
| `upload_name` | `str` | required |
| `filename` | `str` | required |
| `file_path` | `str` | required |
| `file_hash` | `str` | required |
| `file_size_bytes` | `int` | `BigInteger`, non-null |
| `raw_transcription` | `str \| None` | projection field |
| `preferred_execution_attempt_id` | `UUID \| None` | nullable FK -> `execution_attempt.id`, indexed (`use_alter`) |
| `revised_text` | `str \| None` | optional human revision |
| `date_uploaded` | `datetime` | default now |
| `date_revised` | `datetime \| None` | optional |
### `JobSource`
| Field | Type | Notes |
| :--- | :--- | :--- |
| `id` | `UUID` | PK |
| `job_id` | `UUID` | FK -> `job.id`, indexed |
| `source_id` | `UUID` | FK -> `source.id`, indexed |
| `status` | `JobSourceStatus` | non-null enum, default `pending` |
### `ExecutionAttempt`
| Field | Type | Notes |
| :--- | :--- | :--- |
| `id` | `UUID` | PK |
| `job_source_id` | `UUID` | FK -> `job_source.id`, indexed |
| `job_id` | `UUID` | FK -> `job.id`, indexed |
| `source_id` | `UUID` | FK -> `source.id`, indexed |
| `attempt_number` | `int` | `ge=1` |
| `status` | `JobSourceStatus` | non-null enum, value-stable with `JobSource.status` |
| `provider` | `str` | required |
| `model` | `str \| None` | optional |
| `request_manifest` | `dict[str, JsonValue] \| None` | JSONBCompat |
| `request_manifest_sha256` | `str \| None` | optional |
| `request_manifest_schema_version` | `str \| None` | optional |
| `response_received` | `bool` | default `False` |
| `transport_status_code` | `int \| None` | optional |
| `transport_body` | `bytes \| None` | LargeBinary |
| `transport_content_type` | `str \| None` | optional |
| `transport_content_encoding` | `str \| None` | optional |
| `transport_safe_headers` | `dict[str, JsonValue] \| None` | JSONBCompat |
| `router_request_id` | `str \| None` | optional |
| `router_generation_id` | `str \| None` | optional |
| `sdk_response_snapshot` | `dict[str, JsonValue] \| None` | JSONBCompat |
| `normalized_metadata` | `dict[str, JsonValue] \| None` | JSONBCompat |
| `software_context` | `dict[str, JsonValue] \| None` | JSONBCompat |
| `raw_transcription` | `str \| None` | optional |
| `error_category` | `str \| None` | optional |
| `error_detail` | `str \| None` | optional |
| `failure_phase` | `str \| None` | optional |
| `started_at` | `datetime` | required |
| `finished_at` | `datetime` | required |
| `duration_ms` | `int` | `ge=0` |
| `created_at` | `datetime` | default now |
Constraint:
- `UniqueConstraint(job_id, source_id, attempt_number)` named `uq_execution_attempt_number`
## Relationship Loading Contract
- Most ORM relationships are configured with `lazy="raise"`.
- `JobSource.execution_attempts` is intentionally `lazy="noload"` with ordered attempts.
- Service/UI read paths must explicitly eager-load required relationships before access.
## Persistence Invariants (Ground Truth)
1. `ExecutionAttempt` is append-only runtime evidence.
2. `JobSource.status` represents queue/projection execution state and is not a full evidence container.
3. `Source.raw_transcription` is a mutable projection and not authoritative attempt history.
4. `Job` terminal status derives from page outcomes (`JobSource` state), not from a separate summary table.
5. `DocumentType.semantic_key` and `PersonRole.semantic_key` are nullable-unique semantic identifiers.
## Cross-Reference ## Cross-Reference
- [System Architecture](architecture_v4.md) - [System Architecture](architecture_v4.md)
- [System Requirements](requirements_v4.md) - [System Requirements](requirements_v4.md)
- [Error Handling Policy](error_handling_v4.md) - [Error Handling Policy](error_handling_v4.md)
- [AI Evidence and Provenance Invariant](../invariant/ai_evidence_and_provenance.md)