generated from john/python-template
138 lines
5.0 KiB
Markdown
138 lines
5.0 KiB
Markdown
# Database Schema (Version 3)
|
|
|
|
This document describes the relational schema for the transcription platform. It incorporates multi-image batch orchestration, page-level execution tracking, many-to-many author/recipient attribution, submission-time prompt snapshot capture, and raw API payload evidence for archival auditing.
|
|
|
|
The schema uses generic JSON columns compatible with SQLite in local development and PostgreSQL native JSONB/UUID types in production.
|
|
|
|
## Entity Relationship Diagram
|
|
|
|
```mermaid
|
|
erDiagram
|
|
PERSON {
|
|
UUID id PK
|
|
TEXT full_name
|
|
TEXT display_name
|
|
TEXT maiden_name
|
|
DATE birth_date
|
|
TEXT birth_date_raw
|
|
TEXT birth_place
|
|
DATE death_date
|
|
TEXT death_date_raw
|
|
TEXT death_place
|
|
TEXT biography
|
|
TEXT portrait_path
|
|
JSONB metadata
|
|
TIMESTAMPTZ created_at
|
|
TIMESTAMPTZ updated_at
|
|
}
|
|
|
|
DOCUMENT {
|
|
UUID id PK
|
|
TEXT name
|
|
TEXT document_type
|
|
DATE document_date
|
|
TEXT document_date_raw
|
|
TEXT location_created
|
|
TEXT notes
|
|
TEXT archive_identifier
|
|
TIMESTAMPTZ created_at
|
|
TIMESTAMPTZ updated_at
|
|
}
|
|
|
|
DOCUMENT_PERSON {
|
|
UUID id PK
|
|
UUID document_id FK
|
|
UUID person_id FK
|
|
VARCHAR role "author | recipient"
|
|
TIMESTAMPTZ created_at
|
|
}
|
|
|
|
JOB {
|
|
UUID id PK
|
|
UUID document_id FK
|
|
VARCHAR status "queued | processing | transcribed | completed | partial_success | failed"
|
|
INTEGER retry_count
|
|
TEXT provider
|
|
TEXT model
|
|
TEXT prompt_name
|
|
TEXT prompt_hash
|
|
TEXT system_prompt
|
|
TEXT user_prompt
|
|
FLOAT temperature
|
|
FLOAT top_p
|
|
TIMESTAMPTZ date_created
|
|
TIMESTAMPTZ date_updated
|
|
}
|
|
|
|
SOURCE {
|
|
UUID id PK
|
|
UUID document_id FK
|
|
INTEGER page_number
|
|
TEXT upload_name
|
|
TEXT filename
|
|
TEXT file_path
|
|
TEXT file_hash
|
|
BIGINT file_size_bytes
|
|
TEXT raw_transcription
|
|
TEXT revised_text
|
|
TIMESTAMPTZ date_uploaded
|
|
TIMESTAMPTZ date_revised
|
|
}
|
|
|
|
JOB_SOURCE {
|
|
UUID id PK
|
|
UUID job_id FK
|
|
UUID source_id FK
|
|
VARCHAR status "pending | transcribed | failed"
|
|
TEXT raw_transcription
|
|
JSONB ai_metadata
|
|
JSONB raw_api_response
|
|
TEXT error_detail
|
|
TIMESTAMPTZ executed_at
|
|
}
|
|
|
|
DOCUMENT ||--o{ DOCUMENT_PERSON : "has_people"
|
|
PERSON ||--o{ DOCUMENT_PERSON : "participates_in"
|
|
DOCUMENT ||--o{ JOB : "has_jobs"
|
|
DOCUMENT ||--o{ SOURCE : "contains_pages"
|
|
JOB ||--o{ JOB_SOURCE : "executes"
|
|
SOURCE ||--o{ JOB_SOURCE : "processed_in"
|
|
```
|
|
|
|
## Domain Invariants & Provenance Rules
|
|
|
|
### Page-Level Execution & AI Outputs
|
|
|
|
* **Execution Granularity:** Every single image execution attempt by an AI model produces a dedicated record in `job_source`.
|
|
* **Submission Snapshot Provenance:** Every `job` captures the frozen prompt identifier details (`prompt_name`, `prompt_hash`), full prompt text strings (`system_prompt`, `user_prompt`), and hyperparameters (`temperature`, `top_p`) at submission time.
|
|
* **Point-in-Time Output Auditability:** `job_source.raw_api_response` stores the complete, unedited provider REST response envelope for that specific image page call. `job_source.ai_metadata` stores spatial bounding boxes, normalized token usage, latency, and cost details for fast querying.
|
|
* **Active Output Caching:** Upon successful completion of an image call, `source.raw_transcription` is updated with the latest output string from `job_source.raw_transcription` for fast UI rendering.
|
|
|
|
### Image Storage & Integrity
|
|
|
|
* **Filesystem Storage:** Binary images are stored on disk in the local file system. The `source` table holds the relative `file_path`.
|
|
* **File Integrity Tracking:** `source` captures `file_hash` (SHA-256) and `file_size_bytes` at upload time to guarantee document file integrity and duplicate checking over long-term preservation.
|
|
|
|
### Page Ordering & Revisions
|
|
|
|
* **Sequential Integrity:** `source.page_number` dictates page ordering within a document. Reads assembling full documents must query `ORDER BY source.document_id, source.page_number ASC`.
|
|
* **Inlined Human Corrections:** User edits occur at the page level inside `source.revised_text`. `source.raw_transcription` remains immutable. If `source.revised_text` is non-null, application frontends must render `source.revised_text`.
|
|
|
|
### Async Job Lifecycle & Failure Isolation
|
|
|
|
* **Batch Orchestrator:** A job represents an overarching execution run across one or more source images belonging to a document.
|
|
* **Isolated Failures:** API requests run concurrently (e.g., using `asyncio`). A failure on page 3 does not invalidate successful transcriptions on page 1 or 2.
|
|
* **Job States:**
|
|
* `queued`: Created, awaiting worker execution.
|
|
* `processing`: Concurrent HTTP tasks actively running.
|
|
* `completed`: 100% of linked `job_source` tasks succeeded (`transcribed`).
|
|
* `partial_success`: At least one `job_source` succeeded and at least one failed.
|
|
* `failed`: All linked `job_source` tasks failed or a job-level runtime error occurred.
|
|
|
|
|
|
|
|
### Attribution & Person Roles
|
|
|
|
* **Multi-Person Roles:** Documents support zero, one, or many authors and recipients linked via `document_person`.
|
|
* **Role Uniqueness:** `(document_id, person_id, role)` must be unique to prevent duplicate role tagging.
|