generated from john/python-template
161 lines
4.9 KiB
Markdown
161 lines
4.9 KiB
Markdown
# Data Model and Persistence Schema (Version 4)
|
|
|
|
This schema reflects the current V4 persistence contract.
|
|
|
|
## Entity Relationship Overview
|
|
|
|
```mermaid
|
|
erDiagram
|
|
Document ||--o{ Source : has
|
|
Document ||--o{ Job : has
|
|
Document ||--o{ DocumentPerson : links
|
|
Person ||--o{ DocumentPerson : links
|
|
Job ||--o{ JobSource : includes
|
|
Source ||--o{ JobSource : participates
|
|
Source ||--o{ ExecutionAttempt : records
|
|
|
|
Document {
|
|
uuid id PK
|
|
string title
|
|
uuid type_id FK
|
|
string language
|
|
datetime doc_date
|
|
string date_note
|
|
string comments
|
|
string location
|
|
datetime created_at
|
|
datetime updated_at
|
|
}
|
|
|
|
Source {
|
|
uuid id PK
|
|
uuid document_id FK
|
|
string original_name
|
|
string media_type
|
|
string storage_path
|
|
int file_size
|
|
string file_hash
|
|
string raw_transcription
|
|
datetime created_at
|
|
datetime updated_at
|
|
}
|
|
|
|
Job {
|
|
uuid id PK
|
|
uuid document_id FK
|
|
enum status
|
|
string prompt
|
|
json model_settings_json
|
|
datetime created_at
|
|
datetime updated_at
|
|
}
|
|
|
|
JobSource {
|
|
uuid job_id FK
|
|
uuid source_id FK
|
|
enum status
|
|
uuid selected_attempt_id FK
|
|
string error_message
|
|
datetime created_at
|
|
datetime updated_at
|
|
}
|
|
|
|
ExecutionAttempt {
|
|
uuid id PK
|
|
uuid source_id FK
|
|
uuid job_id FK
|
|
enum outcome
|
|
string provider_name
|
|
string provider_model
|
|
string request_manifest_hash
|
|
json request_manifest_json
|
|
json transport_evidence_json
|
|
string transcript_text
|
|
string error_category
|
|
string error_message
|
|
float duration_seconds
|
|
datetime started_at
|
|
datetime completed_at
|
|
datetime created_at
|
|
}
|
|
```
|
|
|
|
## Authoritative Enumerations
|
|
|
|
### JobStatus
|
|
|
|
- `queued`
|
|
- `processing`
|
|
- `transcribed`
|
|
- `partial_success`
|
|
- `failed`
|
|
|
|
### JobSourceStatus
|
|
|
|
- `pending`
|
|
- `transcribed`
|
|
- `failed`
|
|
- `cancelled`
|
|
|
|
## Aggregate Ownership
|
|
|
|
- `Document` aggregate: `Document`, linked `Source`, linked `DocumentPerson`.
|
|
- `Job` aggregate: `Job`, `JobSource` rows, selected-attempt pointers.
|
|
- Evidence aggregate: append-only `ExecutionAttempt` rows keyed by `source_id` + `job_id`.
|
|
|
|
## Persistence Invariants
|
|
|
|
1. `ExecutionAttempt` rows are immutable after creation, except explicit support fields reserved for compatibility migrations.
|
|
2. `JobSource.status` is queue/projection state; it does not duplicate full attempt payload.
|
|
3. `Source.raw_transcription` is a projection, not the complete evidence record.
|
|
4. `Job` terminal status is derived from `JobSource` outcomes.
|
|
5. Registry semantic keys, when present, are immutable once created.
|
|
|
|
## Schema Design Rationale
|
|
|
|
### Why `ExecutionAttempt` exists alongside `JobSource`
|
|
|
|
- `JobSource` is the mutable queue/projection row for workflow control and current-facing page outcome state.
|
|
- `ExecutionAttempt` is the durable, append-only evidence timeline for each provider call.
|
|
- Keeping both avoids overloading one table with competing concerns (queue state vs immutable audit history).
|
|
|
|
### Why `Source.raw_transcription` remains on `Source`
|
|
|
|
- `Source` needs a stable, current projection for UI and print behavior.
|
|
- Projection reads are fast and direct, while deep historical inspection remains available through attempts.
|
|
- Explicit candidate promotion updates the projection pointer without rewriting history.
|
|
|
|
### Why semantic registries use UUID identity plus optional semantic keys
|
|
|
|
- UUIDs are durable relationship identifiers for public/domain links.
|
|
- Optional immutable semantic keys support protected built-ins without exposing internal meaning as external API identity.
|
|
- Labels can evolve without breaking relationships.
|
|
|
|
### Why status enums are narrow
|
|
|
|
- Restricting `JobStatus` and `JobSourceStatus` keeps lifecycle transitions explicit and testable.
|
|
- Queue/state transitions and terminal derivation logic remain deterministic across worker and UI flows.
|
|
|
|
## Media Storage Semantics
|
|
|
|
1. `Source.storage_path` references canonical stored bytes used by processing.
|
|
2. Canonical stored bytes may reflect ingest-time normalization.
|
|
3. File hash and size fields describe canonical stored bytes.
|
|
|
|
## Query and Loading Requirements
|
|
|
|
- Relationship access from service/UI layers must use explicit eager loading patterns compatible with `lazy="raise"`.
|
|
- Candidate-attempt views should select latest/selected attempts explicitly; do not rely on implicit lazy traversal.
|
|
|
|
## Evolution and Migration Policy
|
|
|
|
- Additive schema evolution is preferred for evidence-bearing records.
|
|
- Deprecated semantics should be removed only when model enums, service logic, tests, and docs are updated together.
|
|
- Historical V4.x schema discussions are archived under tag `docs-v4x-archive`; this file is the active contract.
|
|
|
|
## Cross-Reference
|
|
|
|
- [System Architecture](architecture_v4.md)
|
|
- [System Requirements](requirements_v4.md)
|
|
- [Error Handling Policy](error_handling_v4.md)
|