generated from john/python-template
137 lines
4.8 KiB
Markdown
137 lines
4.8 KiB
Markdown
# Database Schema (Version 2)
|
|
|
|
This document describes the PostgreSQL relational schema for the transcription platform. It incorporates multi-image batch orchestration via `asyncio`, page-level execution tracking, many-to-many author/recipient attribution, and JSONB document storage for AI vision outputs.
|
|
|
|
All primary and foreign keys are PostgreSQL native UUIDs (`gen_random_uuid()`).
|
|
|
|
## Entity Relationship Diagram
|
|
|
|
```mermaid
|
|
erDiagram
|
|
PERSON {
|
|
UUID id PK
|
|
TEXT full_name
|
|
TEXT display_name
|
|
TEXT maiden_name
|
|
DATE birth_date
|
|
TEXT birth_date_raw
|
|
TEXT birth_place
|
|
DATE death_date
|
|
TEXT death_date_raw
|
|
TEXT death_place
|
|
TEXT biography
|
|
TEXT portrait_path
|
|
JSONB metadata
|
|
TIMESTAMPTZ created_at
|
|
TIMESTAMPTZ updated_at
|
|
}
|
|
|
|
DOCUMENT {
|
|
UUID id PK
|
|
TEXT name
|
|
TEXT document_type
|
|
DATE document_date
|
|
TEXT document_date_raw
|
|
TEXT location_created
|
|
TEXT notes
|
|
TEXT archive_identifier
|
|
TIMESTAMPTZ created_at
|
|
TIMESTAMPTZ updated_at
|
|
}
|
|
|
|
DOCUMENT_PERSON {
|
|
UUID id PK
|
|
UUID document_id FK
|
|
UUID person_id FK
|
|
VARCHAR role "author | recipient"
|
|
TIMESTAMPTZ created_at
|
|
}
|
|
|
|
JOB {
|
|
UUID id PK
|
|
UUID document_id FK
|
|
VARCHAR status "queued | processing | completed | partial_success | failed"
|
|
INTEGER retry_count
|
|
TEXT provider
|
|
TEXT model
|
|
TEXT prompt_name
|
|
TIMESTAMPTZ date_created
|
|
TIMESTAMPTZ date_updated
|
|
}
|
|
|
|
SOURCE {
|
|
UUID id PK
|
|
UUID document_id FK
|
|
INTEGER page_number
|
|
TEXT upload_name
|
|
TEXT filename
|
|
TEXT file_path
|
|
TEXT raw_transcription
|
|
TEXT revised_text
|
|
TIMESTAMPTZ date_uploaded
|
|
TIMESTAMPTZ date_revised
|
|
}
|
|
|
|
JOB_SOURCE {
|
|
UUID id PK
|
|
UUID job_id FK
|
|
UUID source_id FK
|
|
VARCHAR status "pending | transcribed | failed"
|
|
TEXT raw_transcription
|
|
JSONB ai_metadata
|
|
JSONB raw_api_response
|
|
TEXT error_detail
|
|
TIMESTAMPTZ executed_at
|
|
}
|
|
|
|
DOCUMENT ||--o{ DOCUMENT_PERSON : "has_people"
|
|
PERSON ||--o{ DOCUMENT_PERSON : "participates_in"
|
|
DOCUMENT ||--o{ JOB : "has_jobs"
|
|
DOCUMENT ||--o{ SOURCE : "contains_pages"
|
|
JOB ||--o{ JOB_SOURCE : "executes"
|
|
SOURCE ||--o{ JOB_SOURCE : "processed_in"
|
|
```
|
|
|
|
## Domain Invariants & Rules
|
|
|
|
### Page-Level Execution & AI Outputs
|
|
|
|
* Execution Granularity: Every single image execution by an AI model produces a dedicated record in job_source.
|
|
* Point-in-Time Auditability: job_source.raw_api_response stores the unparsed REST response envelope for that specific image page call. job_source.ai_metadata stores spatial bounding boxes, token usage, and layout details for that specific image page call.
|
|
* Active Output Caching: Upon successful completion of an image call, source.raw_transcription is updated with the latest output string from job_source.raw_transcription for fast UI rendering.
|
|
|
|
### Page Ordering & Revisions
|
|
|
|
* Sequential Integrity: source.page_number dictates page ordering within a document. Reads assembling full documents must query ORDER BY source.document_id, source.page_number ASC.
|
|
* Inlined Human Corrections: User edits occur at the page level inside source.revised_text. source.raw_transcription remains immutable. If source.revised_text is non-null, application frontends must render source.revised_text.
|
|
|
|
### Async Job Lifecycle & Failure Isolation
|
|
|
|
* Batch Orchestrator: A job represents an overarching execution run across one or more source images belonging to a document.
|
|
* Isolated Failures: API requests run concurrently (e.g., using asyncio). A failure on page 3 does not invalidate successful transcriptions on page 1 or 2.
|
|
* Job States:
|
|
- queued: Created, awaiting worker execution.
|
|
- processing: Concurrent HTTP tasks actively running.
|
|
- completed: 100% of linked job_source tasks succeeded (transcribed).
|
|
- partial_success: At least one job_source succeeded and at least one failed.
|
|
- failed: All linked job_source tasks failed or a job-level runtime error occurred.
|
|
|
|
### Attribution & Person Roles
|
|
|
|
* Multi-Person Roles: Documents support zero, one, or many authors and recipients linked via document_person.
|
|
* Role Uniqueness: (document_id, person_id, role) must be unique to prevent duplicate role tagging.
|
|
|
|
---
|
|
|
|
## Related Local References
|
|
|
|
- [System Overview](index_v2.md)
|
|
- [System Design Intent](intent.md)
|
|
- [Transcription Methodology](transcription_methodology.md)
|
|
- [System Architecture](architecture_v2.md)
|
|
- [System Requirements](requirements_v2.md)
|
|
- Data model (this document)
|
|
- [Error Handling Policy](error_handling_v2.md)
|
|
- [Implementation Plan](implementation_plan_v2.md)
|
|
|