# Database Schema (Version 3)

This document describes the relational schema for the transcription platform. It incorporates multi-image batch orchestration, page-level execution tracking, many-to-many author/recipient attribution, submission-time prompt snapshot capture, and raw API payload evidence for archival auditing.

The schema uses generic JSON columns compatible with SQLite in local development and PostgreSQL native JSONB/UUID types in production.

## Entity Relationship Diagram

```mermaid
erDiagram
PERSON {
UUID id PK
TEXT full_name
TEXT display_name
TEXT maiden_name
DATE birth_date
TEXT birth_date_raw
TEXT birth_place
DATE death_date
TEXT death_date_raw
TEXT death_place
TEXT biography
TEXT portrait_path
JSONB metadata
TIMESTAMPTZ created_at
TIMESTAMPTZ updated_at
}

DOCUMENT {
    UUID id PK
    TEXT name
    TEXT document_type
    DATE document_date
    TEXT document_date_raw
    TEXT location_created
    TEXT notes
    TEXT archive_identifier
    TIMESTAMPTZ created_at
    TIMESTAMPTZ updated_at
}

DOCUMENT_PERSON {
    UUID id PK
    UUID document_id FK
    UUID person_id FK
    VARCHAR role "author | recipient"
    TIMESTAMPTZ created_at
}

JOB {
    UUID id PK
    UUID document_id FK
    VARCHAR status "queued | processing | transcribed | completed | partial_success | failed"
    INTEGER retry_count
    TEXT provider
    TEXT model
    TEXT prompt_name
    TEXT prompt_hash
    TEXT system_prompt
    TEXT user_prompt
    FLOAT temperature
    FLOAT top_p
    TIMESTAMPTZ date_created
    TIMESTAMPTZ date_updated
}

SOURCE {
    UUID id PK
    UUID document_id FK
    INTEGER page_number
    TEXT upload_name
    TEXT filename
    TEXT file_path
    TEXT file_hash
    BIGINT file_size_bytes
    TEXT raw_transcription
    TEXT revised_text
    TIMESTAMPTZ date_uploaded
    TIMESTAMPTZ date_revised
}

JOB_SOURCE {
    UUID id PK
    UUID job_id FK
    UUID source_id FK
    VARCHAR status "pending | transcribed | failed"
    TEXT raw_transcription
    JSONB ai_metadata
    JSONB raw_api_response
    TEXT error_detail
    TIMESTAMPTZ executed_at
}

DOCUMENT ||--o{ DOCUMENT_PERSON : "has_people"
PERSON ||--o{ DOCUMENT_PERSON : "participates_in"
DOCUMENT ||--o{ JOB : "has_jobs"
DOCUMENT ||--o{ SOURCE : "contains_pages"
JOB ||--o{ JOB_SOURCE : "executes"
SOURCE ||--o{ JOB_SOURCE : "processed_in"
```

## Domain Invariants & Provenance Rules

### Page-Level Execution & AI Outputs

* **Execution Granularity:** Every single image execution attempt by an AI model produces a dedicated record in `job_source`.
* **Submission Snapshot Provenance:** Every `job` captures the frozen prompt identifier details (`prompt_name`, `prompt_hash`), full prompt text strings (`system_prompt`, `user_prompt`), and hyperparameters (`temperature`, `top_p`) at submission time.
* **Point-in-Time Output Auditability:** `job_source.raw_api_response` stores the complete, unedited provider REST response envelope for that specific image page call. `job_source.ai_metadata` stores spatial bounding boxes, normalized token usage, latency, and cost details for fast querying.
* **Active Output Caching:** Upon successful completion of an image call, `source.raw_transcription` is updated with the latest output string from `job_source.raw_transcription` for fast UI rendering.

### Image Storage & Integrity

* **Filesystem Storage:** Binary images are stored on disk in the local file system. The `source` table holds the relative `file_path`.
* **File Integrity Tracking:** `source` captures `file_hash` (SHA-256) and `file_size_bytes` at upload time to guarantee document file integrity and duplicate checking over long-term preservation.

### Page Ordering & Revisions

* **Sequential Integrity:** `source.page_number` dictates page ordering within a document. Reads assembling full documents must query `ORDER BY source.document_id, source.page_number ASC`.
* **Inlined Human Corrections:** User edits occur at the page level inside `source.revised_text`. `source.raw_transcription` remains immutable. If `source.revised_text` is non-null, application frontends must render `source.revised_text`.

### Async Job Lifecycle & Failure Isolation

* **Batch Orchestrator:** A job represents an overarching execution run across one or more source images belonging to a document.
* **Isolated Failures:** API requests run concurrently (e.g., using `asyncio`). A failure on page 3 does not invalidate successful transcriptions on page 1 or 2.
* **Job States:**
* `queued`: Created, awaiting worker execution.
* `processing`: Concurrent HTTP tasks actively running.
* `completed`: 100% of linked `job_source` tasks succeeded (`transcribed`).
* `partial_success`: At least one `job_source` succeeded and at least one failed.
* `failed`: All linked `job_source` tasks failed or a job-level runtime error occurred.



### Attribution & Person Roles

* **Multi-Person Roles:** Documents support zero, one, or many authors and recipients linked via `document_person`.
* **Role Uniqueness:** `(document_id, person_id, role)` must be unique to prevent duplicate role tagging.