V4 plan created: Adding many-to-many links between Documents & People in the UI. Also adding some new tables for Document Type, Person role.

2026-08-09 13:29:50 -05:00
parent e6549277c6
commit 5753eb0135
13 changed files with 1356 additions and 0 deletions
@@ -0,0 +1,137 @@
+# Database Schema (Version 3)
+
+This document describes the relational schema for the transcription platform. It incorporates multi-image batch orchestration, page-level execution tracking, many-to-many author/recipient attribution, submission-time prompt snapshot capture, and raw API payload evidence for archival auditing.
+
+The schema uses generic JSON columns compatible with SQLite in local development and PostgreSQL native JSONB/UUID types in production.
+
+## Entity Relationship Diagram
+
+```mermaid
+erDiagram
+PERSON {
+UUID id PK
+TEXT full_name
+TEXT display_name
+TEXT maiden_name
+DATE birth_date
+TEXT birth_date_raw
+TEXT birth_place
+DATE death_date
+TEXT death_date_raw
+TEXT death_place
+TEXT biography
+TEXT portrait_path
+JSONB metadata
+TIMESTAMPTZ created_at
+TIMESTAMPTZ updated_at
+}
+
+DOCUMENT {
+    UUID id PK
+    TEXT name
+    TEXT document_type
+    DATE document_date
+    TEXT document_date_raw
+    TEXT location_created
+    TEXT notes
+    TEXT archive_identifier
+    TIMESTAMPTZ created_at
+    TIMESTAMPTZ updated_at
+}
+
+DOCUMENT_PERSON {
+    UUID id PK
+    UUID document_id FK
+    UUID person_id FK
+    VARCHAR role "author | recipient"
+    TIMESTAMPTZ created_at
+}
+
+JOB {
+    UUID id PK
+    UUID document_id FK
+    VARCHAR status "queued | processing | transcribed | completed | partial_success | failed"
+    INTEGER retry_count
+    TEXT provider
+    TEXT model
+    TEXT prompt_name
+    TEXT prompt_hash
+    TEXT system_prompt
+    TEXT user_prompt
+    FLOAT temperature
+    FLOAT top_p
+    TIMESTAMPTZ date_created
+    TIMESTAMPTZ date_updated
+}
+
+SOURCE {
+    UUID id PK
+    UUID document_id FK
+    INTEGER page_number
+    TEXT upload_name
+    TEXT filename
+    TEXT file_path
+    TEXT file_hash
+    BIGINT file_size_bytes
+    TEXT raw_transcription
+    TEXT revised_text
+    TIMESTAMPTZ date_uploaded
+    TIMESTAMPTZ date_revised
+}
+
+JOB_SOURCE {
+    UUID id PK
+    UUID job_id FK
+    UUID source_id FK
+    VARCHAR status "pending | transcribed | failed"
+    TEXT raw_transcription
+    JSONB ai_metadata
+    JSONB raw_api_response
+    TEXT error_detail
+    TIMESTAMPTZ executed_at
+}
+
+DOCUMENT ||--o{ DOCUMENT_PERSON : "has_people"
+PERSON ||--o{ DOCUMENT_PERSON : "participates_in"
+DOCUMENT ||--o{ JOB : "has_jobs"
+DOCUMENT ||--o{ SOURCE : "contains_pages"
+JOB ||--o{ JOB_SOURCE : "executes"
+SOURCE ||--o{ JOB_SOURCE : "processed_in"
+```
+
+## Domain Invariants & Provenance Rules
+
+### Page-Level Execution & AI Outputs
+
+* **Execution Granularity:** Every single image execution attempt by an AI model produces a dedicated record in `job_source`.
+* **Submission Snapshot Provenance:** Every `job` captures the frozen prompt identifier details (`prompt_name`, `prompt_hash`), full prompt text strings (`system_prompt`, `user_prompt`), and hyperparameters (`temperature`, `top_p`) at submission time.
+* **Point-in-Time Output Auditability:** `job_source.raw_api_response` stores the complete, unedited provider REST response envelope for that specific image page call. `job_source.ai_metadata` stores spatial bounding boxes, normalized token usage, latency, and cost details for fast querying.
+* **Active Output Caching:** Upon successful completion of an image call, `source.raw_transcription` is updated with the latest output string from `job_source.raw_transcription` for fast UI rendering.
+
+### Image Storage & Integrity
+
+* **Filesystem Storage:** Binary images are stored on disk in the local file system. The `source` table holds the relative `file_path`.
+* **File Integrity Tracking:** `source` captures `file_hash` (SHA-256) and `file_size_bytes` at upload time to guarantee document file integrity and duplicate checking over long-term preservation.
+
+### Page Ordering & Revisions
+
+* **Sequential Integrity:** `source.page_number` dictates page ordering within a document. Reads assembling full documents must query `ORDER BY source.document_id, source.page_number ASC`.
+* **Inlined Human Corrections:** User edits occur at the page level inside `source.revised_text`. `source.raw_transcription` remains immutable. If `source.revised_text` is non-null, application frontends must render `source.revised_text`.
+
+### Async Job Lifecycle & Failure Isolation
+
+* **Batch Orchestrator:** A job represents an overarching execution run across one or more source images belonging to a document.
+* **Isolated Failures:** API requests run concurrently (e.g., using `asyncio`). A failure on page 3 does not invalidate successful transcriptions on page 1 or 2.
+* **Job States:**
+* `queued`: Created, awaiting worker execution.
+* `processing`: Concurrent HTTP tasks actively running.
+* `completed`: 100% of linked `job_source` tasks succeeded (`transcribed`).
+* `partial_success`: At least one `job_source` succeeded and at least one failed.
+* `failed`: All linked `job_source` tasks failed or a job-level runtime error occurred.
+
+
+
+### Attribution & Person Roles
+
+* **Multi-Person Roles:** Documents support zero, one, or many authors and recipients linked via `document_person`.
+* **Role Uniqueness:** `(document_id, person_id, role)` must be unique to prevent duplicate role tagging.