bbchops/transcription
1
0
Fork 0
generated from john/python-template
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

Revised and simplified V4 Plan and core documents.

Browse Source
This commit is contained in:
Jim Lancaster
2026-08-10 10:53:13 -05:00
parent 9b4d6f0340
commit 4b3baf5a3e
7 changed files with 495 additions and 578 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/ver4/schema_v4.md
+158 -180
View File
@@ -1,214 +1,192 @@
# Database Schema (Version 4)
This document defines the selected schema direction for V4 document-person relationship expansion.
This document defines the relational schema for the document transcription system.
V4 goals are:
## Entity Relationship Diagram
- extensible role taxonomy,
- extensible document type taxonomy,
- explicit assertion state (`asserted`, `suggested`),
- policy-driven exclusivity,
- deterministic migration from V3 links.
```mermaid
erDiagram
DOCUMENT_TYPE {
UUID id PK
TEXT code
TEXT label
BOOLEAN is_active
INTEGER sort_order
TIMESTAMPTZ created_at
TIMESTAMPTZ updated_at
}
## Scope
PERSON_ROLE {
UUID id PK
TEXT code
TEXT label
BOOLEAN is_active
TIMESTAMPTZ created_at
TIMESTAMPTZ updated_at
}
This specification focuses on relationship and document-type governance persistence changes. Existing `Person`, `Source`, `Job`, and `JobSource` core structures remain as in V3 unless explicitly noted.
ROLE_EXCLUSIVITY {
UUID id PK
UUID left_role_id FK
UUID right_role_id FK
TIMESTAMPTZ created_at
}
## New/Expanded Concepts
PERSON {
UUID id PK
TEXT full_name
TEXT display_name
TEXT maiden_name
DATE birth_date
TEXT birth_date_raw
TEXT birth_place
DATE death_date
TEXT death_date_raw
TEXT death_place
TEXT biography
TEXT portrait_path
JSONB metadata
TIMESTAMPTZ created_at
TIMESTAMPTZ updated_at
}
- **Relationship role:** semantic label such as `author`, `recipient`, `mentioned`.
- **Assertion state:** whether the link is canonical (`asserted`) or pending review (`suggested`).
- **Exclusivity matrix:** configurable role-pair conflicts for same `(document_id, person_id)`.
- **Suggestion provenance:** evidence fields enabling review decisions.
- **Document type registry:** controlled taxonomy for `Document` classification with stable code identity.
DOCUMENT {
UUID id PK
UUID document_type_id FK
TEXT name
DATE document_date
TEXT document_date_raw
TEXT location_created
TEXT notes
TEXT archive_identifier
TIMESTAMPTZ created_at
TIMESTAMPTZ updated_at
}
## Selected Model: Role Registry + Separate Suggestion Table
DOCUMENT_PERSON {
UUID id PK
UUID document_id FK
UUID person_id FK
UUID role_id FK
TIMESTAMPTZ created_at
TIMESTAMPTZ updated_at
}
This option cleanly separates canonical links from pending suggestions and enables fully data-driven role expansion.
JOB {
UUID id PK
UUID document_id FK
VARCHAR status
INTEGER retry_count
TEXT provider
TEXT model
TEXT prompt_name
TEXT prompt_hash
TEXT system_prompt
TEXT user_prompt
FLOAT temperature
FLOAT top_p
TIMESTAMPTZ date_created
TIMESTAMPTZ date_updated
}
### Tables
SOURCE {
UUID id PK
UUID document_id FK
INTEGER page_number
TEXT upload_name
TEXT filename
TEXT file_path
TEXT file_hash
BIGINT file_size_bytes
TEXT raw_transcription
TEXT revised_text
TIMESTAMPTZ date_uploaded
TIMESTAMPTZ date_revised
}
#### `person_role`
JOB_SOURCE {
UUID id PK
UUID job_id FK
UUID source_id FK
VARCHAR status
TEXT raw_transcription
JSONB ai_metadata
JSONB raw_api_response
TEXT error_detail
TIMESTAMPTZ executed_at
}
| Column | Type | Notes |
| --- | --- | --- |
| `id` | UUID PK | Stable key |
| `code` | TEXT UNIQUE | Canonical role code, for example `author`, `recipient`, `mentioned` |
| `label` | TEXT | UI label |
| `is_active` | BOOLEAN | Soft-enable/disable role |
| `created_at` | TIMESTAMPTZ | Audit timestamp |
| `updated_at` | TIMESTAMPTZ | Audit timestamp |
DOCUMENT_TYPE ||--o{ DOCUMENT : classifies
DOCUMENT ||--o{ DOCUMENT_PERSON : has_people
PERSON ||--o{ DOCUMENT_PERSON : appears_in
PERSON_ROLE ||--o{ DOCUMENT_PERSON : labels
PERSON_ROLE ||--o{ ROLE_EXCLUSIVITY : left_rule
PERSON_ROLE ||--o{ ROLE_EXCLUSIVITY : right_rule
DOCUMENT ||--o{ JOB : has_jobs
DOCUMENT ||--o{ SOURCE : contains_pages
JOB ||--o{ JOB_SOURCE : executes
SOURCE ||--o{ JOB_SOURCE : processed_in
```
#### `document_person` (asserted links only)
## Domain Invariants and Provenance Rules
| Column | Type | Notes |
| --- | --- | --- |
| `id` | UUID PK | Stable key |
| `document_id` | UUID FK | -> `document.id` |
| `person_id` | UUID FK | -> `person.id` |
| `role_id` | UUID FK | -> `person_role.id` |
| `created_at` | TIMESTAMPTZ | Audit timestamp |
| `updated_at` | TIMESTAMPTZ | Audit timestamp |
### Page-Level Execution and AI Outputs
Constraints:
- Every single page execution by an AI model produces a dedicated `JOB_SOURCE` record.
- Every `JOB` stores the frozen prompt identifier, prompt text, and hyperparameters used at submission time.
- Every `JOB_SOURCE` stores the complete provider response envelope and page-level operational metadata.
- `SOURCE.raw_transcription` caches the latest successful machine output for that page.
- `UNIQUE(document_id, person_id, role_id)`
### Image Storage and Integrity
#### `document_person_suggestion`
- Binary images are stored on disk; `SOURCE.file_path` stores the persisted path.
- `SOURCE.file_hash` stores a SHA-256 digest.
- `SOURCE.file_size_bytes` stores the original file size.
| Column | Type | Notes |
| --- | --- | --- |
| `id` | UUID PK | Stable key |
| `document_id` | UUID FK | -> `document.id` |
| `person_id` | UUID FK | -> `person.id` |
| `role_id` | UUID FK | -> `person_role.id` |
| `status` | TEXT | `pending`, `accepted`, `rejected` |
| `confidence` | FLOAT NULL | Optional score |
| `source_mechanism` | TEXT NULL | For example `rule`, `llm` |
| `evidence_ref` | TEXT NULL | Pointer or excerpt ID |
| `evidence_span` | JSON NULL | Optional text span payload |
| `note` | TEXT NULL | Reviewer note |
| `created_at` | TIMESTAMPTZ | Suggestion creation time |
| `reviewed_at` | TIMESTAMPTZ NULL | Decision time |
| `reviewed_by` | TEXT NULL | Operator identifier |
### Page Ordering and Revisions
Constraints:
- `SOURCE.page_number` dictates page ordering within a document.
- `SOURCE.raw_transcription` remains immutable machine output.
- `SOURCE.revised_text` stores human edits and is the preferred display value when present.
- `UNIQUE(document_id, person_id, role_id, status)` with policy for multiple pending rows defined in service layer.
- Optional stricter rule: one active pending suggestion per `(document_id, person_id, role_id)`.
### Document-Person Role Governance
#### `role_exclusivity`
- Documents support zero, one, or many people per relationship role.
- Relationship roles are defined by `PERSON_ROLE` rather than hardcoded columns.
- `DOCUMENT_PERSON` must be unique for `(document_id, person_id, role_id)`.
- Configured exclusive role pairs from `ROLE_EXCLUSIVITY` cannot coexist for the same `(document_id, person_id)`.
- Initial exclusivity seed blocks `author` and `recipient` for the same person-document pair.
| Column | Type | Notes |
| --- | --- | --- |
| `id` | UUID PK | Stable key |
| `left_role_id` | UUID FK | -> `person_role.id` |
| `right_role_id` | UUID FK | -> `person_role.id` |
| `created_at` | TIMESTAMPTZ | Audit timestamp |
### Document Type Governance
Constraints:
- Every document type is defined by `DOCUMENT_TYPE`.
- `DOCUMENT_TYPE.code` is a stable machine identifier.
- `DOCUMENT_TYPE.label` is mutable display text.
- Inactive types remain valid for historical rows but should be excluded from default selection UIs.
- Canonical ordering rule to avoid duplicate pairs (`left_role_id < right_role_id` enforced in service/DB).
- `UNIQUE(left_role_id, right_role_id)`
## Constraint Summary
Initial seed:
- Exclusivity pair: (`author`, `recipient`)
## Deferred Alternative (Not Selected for V4)
An enum-based shared table model was considered but is intentionally not selected for V4 because it couples canonical and provisional states in one table and increases invariant complexity.
## Assertion-State Semantics
- `asserted`: canonical relationship used for document/person metadata and business logic.
- `suggested`: non-canonical proposal requiring explicit review.
- Accept action:
- creates asserted link (or confirms existing),
- marks suggestion `accepted`.
- Reject action:
- marks suggestion `rejected`.
## Exclusivity Enforcement
Policy target:
- For a single `(document_id, person_id)`, disallow coexistence of role pairs configured as exclusive.
Enforcement layers:
1. Service-level pre-check for clear API errors.
2. Database-level guard where feasible (constraints/triggers or deterministic write path).
Initial configured rule:
- `author` and `recipient` are exclusive.
Enforcement semantics:
- Exclusivity is enforced for asserted links.
- Pending suggestions may exist even if they would conflict when asserted.
- Accepting a suggestion must run exclusivity checks and fail deterministically on conflict.
## Migration From V3
### Data Mapping
- Existing V3 `document_person` rows map to V4 `asserted` semantics.
- Existing V3 role values:
- `author` -> role `author`
- `recipient` -> role `recipient`
### Backfill Steps
1. Seed role rows (`author`, `recipient`, `mentioned`) if using Option A.
2. Migrate current links into asserted table/state.
3. Run conflict scan for exclusivity violations.
4. Apply deterministic conflict policy for any violations.
5. Enable hard enforcement after data passes validation.
- `DOCUMENT_TYPE.code` is unique.
- `PERSON_ROLE.code` is unique.
- `DOCUMENT_PERSON(document_id, person_id, role_id)` is unique.
- `ROLE_EXCLUSIVITY(left_role_id, right_role_id)` is unique.
- `ROLE_EXCLUSIVITY` must use canonical ordering to avoid duplicate mirrored pairs.
## Indexing Guidance
Recommended indexes:
- `document(document_type_id)`
- `document_person(document_id)`
- `document_person(person_id)`
- `document_person(role_id)` (Option A) or `document_person(role)` (Option B)
- `document_person_suggestion(document_id, status)` (Option A)
- `document_person_suggestion(person_id, status)` (Option A)
## Selection Rationale
V4 selects role registry plus separate suggestion storage for clearer provenance boundaries, cleaner lifecycle transitions, and long-term extensibility.
## Document Type Registry Model (Selected for V4)
V4 applies the same registry governance pattern to document classification.
### Tables
#### `document_type`
| Column | Type | Notes |
| --- | --- | --- |
| `id` | UUID PK | Stable key |
| `code` | TEXT UNIQUE | Canonical type code, for example `letter`, `diary`, `book`, `postcard` |
| `label` | TEXT | UI display label |
| `is_active` | BOOLEAN | Soft-enable/disable type |
| `sort_order` | INTEGER NULL | Optional UI ordering |
| `created_at` | TIMESTAMPTZ | Audit timestamp |
| `updated_at` | TIMESTAMPTZ | Audit timestamp |
#### `document` update
| Column | Type | Notes |
| --- | --- | --- |
| `document_type_id` | UUID FK NULL | -> `document_type.id` |
### Constraints and Governance
- `document_type.code` must be stable and unique.
- `document_type.label` may change without changing canonical type identity.
- Inactive types remain valid for historical records but are excluded from default create/edit selectors.
### Initial Seeds
- Seed baseline type codes from current V3 usage set (for example `letter`, `diary`, `book`, `postcard`, `record`, `memo`) and refine labels as needed.
### Migration and Normalization (Small Corpus)
1. Seed canonical `document_type` rows.
2. Manually assign each existing document (10 total) to a canonical type via `document_type_id`.
3. Resolve any outlier values directly during this one-time pass.
4. Enforce registry-backed write validation after manual assignment is complete.
### Indexing Guidance (Document Type)
- `document_type(code)` unique index.
- `document(document_type_id)` index.
- `document_person(role_id)`
- `role_exclusivity(left_role_id, right_role_id)`
- `source(document_id, page_number)`
- `job(document_id, status)`
- `job_source(job_id)`
- `job_source(source_id)`
## Related Local References
- [V4 Scope Boundary](scope_boundary_v4.md)
- [V4 Requirements](requirements_v4.md)
- [V3 Schema](../schema_v3.md)
- [System Overview](index_v4.md)
- [System Architecture](architecture_v4.md)
- [System Requirements](requirements_v4.md)
- [Error Handling Policy](error_handling_v4.md)