transcription/docs/schema_v3.md

Database Schema (Version 3)

This document describes the relational schema for the transcription platform. It incorporates multi-image batch orchestration, page-level execution tracking, many-to-many author/recipient attribution, input prompt provenance capture, and raw API payload evidence for archival auditing.

The schema uses generic JSON columns compatible with SQLite in local development and PostgreSQL native JSONB/UUID types in production.

Entity Relationship Diagram

erDiagram
PERSON {
UUID id PK
TEXT full_name
TEXT display_name
TEXT maiden_name
DATE birth_date
TEXT birth_date_raw
TEXT birth_place
DATE death_date
TEXT death_date_raw
TEXT death_place
TEXT biography
TEXT portrait_path
JSONB metadata
TIMESTAMPTZ created_at
TIMESTAMPTZ updated_at
}

DOCUMENT {
    UUID id PK
    TEXT name
    TEXT document_type
    DATE document_date
    TEXT document_date_raw
    TEXT location_created
    TEXT notes
    TEXT archive_identifier
    TIMESTAMPTZ created_at
    TIMESTAMPTZ updated_at
}

DOCUMENT_PERSON {
    UUID id PK
    UUID document_id FK
    UUID person_id FK
    VARCHAR role "author | recipient"
    TIMESTAMPTZ created_at
}

JOB {
    UUID id PK
    UUID document_id FK
    VARCHAR status "queued | processing | transcribed | completed | partial_success | failed"
    INTEGER retry_count
    TEXT provider
    TEXT model
    TIMESTAMPTZ date_created
    TIMESTAMPTZ date_updated
}

SOURCE {
    UUID id PK
    UUID document_id FK
    INTEGER page_number
    TEXT upload_name
    TEXT filename
    TEXT file_path
    TEXT file_hash
    BIGINT file_size_bytes
    TEXT raw_transcription
    TEXT revised_text
    TIMESTAMPTZ date_uploaded
    TIMESTAMPTZ date_revised
}

JOB_SOURCE {
    UUID id PK
    UUID job_id FK
    UUID source_id FK
    VARCHAR status "pending | transcribed | failed"
    TEXT raw_transcription
    TEXT prompt_name
    TEXT prompt_hash
    TEXT system_prompt
    TEXT user_prompt
    FLOAT temperature
    FLOAT top_p
    JSONB ai_metadata
    JSONB raw_api_response
    TEXT error_detail
    TIMESTAMPTZ executed_at
}

DOCUMENT ||--o{ DOCUMENT_PERSON : "has_people"
PERSON ||--o{ DOCUMENT_PERSON : "participates_in"
DOCUMENT ||--o{ JOB : "has_jobs"
DOCUMENT ||--o{ SOURCE : "contains_pages"
JOB ||--o{ JOB_SOURCE : "executes"
SOURCE ||--o{ JOB_SOURCE : "processed_in"

Domain Invariants & Provenance Rules

Page-Level Execution & AI Outputs

• Execution Granularity: Every single image execution attempt by an AI model produces a dedicated record in job_source.
• Page-Level Input Provenance: Every job_source execution captures its exact hyperparameters (temperature, top_p), prompt identifier details (prompt_name, prompt_hash), and full prompt text strings (system_prompt, user_prompt) used for that specific page call.
• Point-in-Time Output Auditability: job_source.raw_api_response stores the complete, unedited provider REST response envelope for that specific image page call. job_source.ai_metadata stores spatial bounding boxes, normalized token usage, latency, and cost details for fast querying.
• Active Output Caching: Upon successful completion of an image call, source.raw_transcription is updated with the latest output string from job_source.raw_transcription for fast UI rendering.

Image Storage & Integrity

• Filesystem Storage: Binary images are stored on disk in the local file system. The source table holds the relative file_path.
• File Integrity Tracking: source captures file_hash (SHA-256) and file_size_bytes at upload time to guarantee document file integrity and duplicate checking over long-term preservation.

Page Ordering & Revisions

• Sequential Integrity: source.page_number dictates page ordering within a document. Reads assembling full documents must query ORDER BY source.document_id, source.page_number ASC.
• Inlined Human Corrections: User edits occur at the page level inside source.revised_text. source.raw_transcription remains immutable. If source.revised_text is non-null, application frontends must render source.revised_text.

Async Job Lifecycle & Failure Isolation

• Batch Orchestrator: A job represents an overarching execution run across one or more source images belonging to a document.
• Isolated Failures: API requests run concurrently (e.g., using asyncio). A failure on page 3 does not invalidate successful transcriptions on page 1 or 2.
• Job States:
• queued: Created, awaiting worker execution.
• processing: Concurrent HTTP tasks actively running.
• completed: 100% of linked job_source tasks succeeded (transcribed).
• partial_success: At least one job_source succeeded and at least one failed.
• failed: All linked job_source tasks failed or a job-level runtime error occurred.

Attribution & Person Roles

• Multi-Person Roles: Documents support zero, one, or many authors and recipients linked via document_person.
• Role Uniqueness: (document_id, person_id, role) must be unique to prevent duplicate role tagging.