bbchops/transcription
1
0
Fork 0
generated from john/python-template
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Begin planning V2

Browse Source
This commit is contained in:
Jim Lancaster
2026-07-30 19:39:12 -05:00
parent a975ca299a
commit d2b793ea69
6 changed files with 457 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/schema.md
+1 -1
View File
@@ -41,7 +41,7 @@ erDiagram
REVISION {
UUID id PK
UUID source_id FK UNIQUE
UUID source_id "FK, UK"
INTEGER revision
TEXT text
DATETIME date_created
docs/ver2/V2 Context for DB Updates.md
+35
View File
@@ -0,0 +1,35 @@
# AI Coding Assistant Project Briefing & Context
## Project Mission
This application is a family history archival and transcription platform. Its primary goal is to accept scanned document images (letters, postcards, logbooks, diaries), execute OCR and structured transcription via AI vision models (OpenAI GPT-4o, Anthropic Claude 3.5 Sonnet), and manage historical metadata (authors, recipients, dates, and locations).
---
## Technical Stack & Architecture
* **Database:** PostgreSQL 13+ with native `UUID` (`gen_random_uuid()`) and `JSONB` columns.
* **Backend Runtime / Concurrency:** Python utilizing `asyncio` for concurrent HTTP API calls to AI providers, with strict rate-limiting via `asyncio.Semaphore`.
* **Validation & Types:** TypeScript with **Zod** schema definitions. Incoming AI responses must be parsed and validated with Zod schemas *before* database insertion.
* **ORM / Database Access:** Raw parameterized SQL queries or lightweight query builders (e.g., Kysely/Prisma) respecting PostgreSQL native types.
---
## Core System Directives for AI Code Generation
### 1. Data Immutability vs. Human Corrections
* `job_source.raw_transcription` and `source.raw_transcription` represent original, point-in-time machine outputs and are **immutable**.
* Human corrections occur on `source.revised_text`.
* When fetching text for the UI, always display `COALESCE(source.revised_text, source.raw_transcription)`.
### 2. Async Execution & Batching Rules
* A `job` represents an overarching execution run for a folder/group of images belonging to a single `document`.
* Images are submitted to AI APIs **one at a time in rapid succession** using `asyncio` worker pools.
* Each single-image API call populates a row in `job_source` with its own `status`, `raw_transcription`, `ai_metadata`, and `raw_api_response`.
* If 9 of 10 pages succeed and 1 fails, `job_source.status` for the failed image becomes `'failed'`, while `job.status` becomes `'partial_success'`. Do not mark the entire batch as failed if partial results exist.
### 3. Entity Relationships
* **Authors/Recipients:** A `document` can have multiple authors and recipients. Do NOT put direct `author_id` foreign keys on `document`. Query authors/recipients via `document_person` where `role = 'author'` or `role = 'recipient'`.
* **Page Ordering:** Multi-page documents must always be queried using `ORDER BY page_number ASC`.
### 4. Database Mutations
* Always use parameterized SQL queries (`$1`, `$2`) to prevent SQL injection.
* Store datetimes using UTC ISO 8601 strings or native PostgreSQL `TIMESTAMPTZ`.
docs/ver2/V2 DB Schema.md
+122
View File
@@ -0,0 +1,122 @@
# Database Schema (V2 Architecture)
This document describes the PostgreSQL relational schema for the transcription platform. It incorporates multi-image batch orchestration via `asyncio`, page-level execution tracking, many-to-many author/recipient attribution, and JSONB document storage for AI vision outputs.
All primary and foreign keys are PostgreSQL native UUIDs (`gen_random_uuid()`).
## Entity Relationship Diagram
```mermaid
erDiagram
PERSON {
UUID id PK
TEXT full_name
TEXT display_name
TEXT maiden_name
DATE birth_date
TEXT birth_date_raw
TEXT birth_place
DATE death_date
TEXT death_date_raw
TEXT death_place
TEXT biography
TEXT portrait_path
JSONB metadata
TIMESTAMPTZ created_at
TIMESTAMPTZ updated_at
}
DOCUMENT {
UUID id PK
TEXT name
TEXT document_type
DATE document_date
TEXT document_date_raw
TEXT location_created
TEXT notes
TEXT archive_identifier
TIMESTAMPTZ created_at
TIMESTAMPTZ updated_at
}
DOCUMENT_PERSON {
UUID id PK
UUID document_id FK
UUID person_id FK
VARCHAR role "author | recipient"
TIMESTAMPTZ created_at
}
JOB {
UUID id PK
UUID document_id FK
VARCHAR status "queued | processing | completed | partial_success | failed"
INTEGER retry_count
TEXT provider
TEXT model
TEXT prompt_name
TIMESTAMPTZ date_created
TIMESTAMPTZ date_updated
}
SOURCE {
UUID id PK
UUID document_id FK
INTEGER page_number
TEXT upload_name
TEXT filename
TEXT file_path
TEXT raw_transcription
TEXT revised_text
TIMESTAMPTZ date_uploaded
TIMESTAMPTZ date_revised
}
JOB_SOURCE {
UUID id PK
UUID job_id FK
UUID source_id FK
VARCHAR status "pending | transcribed | failed"
TEXT raw_transcription
JSONB ai_metadata
JSONB raw_api_response
TEXT error_detail
TIMESTAMPTZ executed_at
}
DOCUMENT ||--o{ DOCUMENT_PERSON : "has_people"
PERSON ||--o{ DOCUMENT_PERSON : "participates_in"
DOCUMENT ||--o{ JOB : "has_jobs"
DOCUMENT ||--o{ SOURCE : "contains_pages"
JOB ||--o{ JOB_SOURCE : "executes"
SOURCE ||--o{ JOB_SOURCE : "processed_in"
```
## Domain Invariants & Rules
### Page-Level Execution & AI Outputs
* Execution Granularity: Every single image execution by an AI model produces a dedicated record in job_source.
* Point-in-Time Auditability: job_source.raw_api_response stores the unparsed REST response envelope for that specific image page call. job_source.ai_metadata stores spatial bounding boxes, token usage, and layout details for that specific image page call.
* Active Output Caching: Upon successful completion of an image call, source.raw_transcription is updated with the latest output string from job_source.raw_transcription for fast UI rendering.
### Page Ordering & Revisions
* Sequential Integrity: source.page_number dictates page ordering within a document. Reads assembling full documents must query ORDER BY source.document_id, source.page_number ASC.
* Inlined Human Corrections: User edits occur at the page level inside source.revised_text. source.raw_transcription remains immutable. If source.revised_text is non-null, application frontends must render source.revised_text.
### Async Job Lifecycle & Failure Isolation
* Batch Orchestrator: A job represents an overarching execution run across one or more source images belonging to a document.
* Isolated Failures: API requests run concurrently (e.g., using asyncio). A failure on page 3 does not invalidate successful transcriptions on page 1 or 2.
* Job States:
- queued: Created, awaiting worker execution.
- processing: Concurrent HTTP tasks actively running.
- completed: 100% of linked job_source tasks succeeded (transcribed).
- partial_success: At least one job_source succeeded and at least one failed.
- failed: All linked job_source tasks failed or a job-level runtime error occurred.
### Attribution & Person Roles
* Multi-Person Roles: Documents support zero, one, or many authors and recipients linked via document_person.
* Role Uniqueness: (document_id, person_id, role) must be unique to prevent duplicate role tagging.
docs/ver2/V2 PostgreSQL DDL Specification.md
+102
View File
@@ -0,0 +1,102 @@
## PostgreSQL DDL Specification
```sql
-- Enable pgcrypto for UUID generation if on PostgreSQL < 13
CREATE EXTENSION IF NOT EXISTS "pgcrypto";
-- 1. PERSON TABLE
CREATE TABLE person (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
full_name TEXT NOT NULL,
display_name TEXT,
maiden_name TEXT,
birth_date DATE,
birth_date_raw TEXT,
birth_place TEXT,
death_date DATE,
death_date_raw TEXT,
death_place TEXT,
biography TEXT,
portrait_path TEXT,
metadata JSONB DEFAULT '{}'::jsonb,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
-- 2. DOCUMENT TABLE
CREATE TABLE document (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
name TEXT NOT NULL,
document_type TEXT,
document_date DATE,
document_date_raw TEXT,
location_created TEXT,
notes TEXT,
archive_identifier TEXT,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
-- 3. DOCUMENT_PERSON (Junction Table for Multi-Author / Multi-Recipient)
CREATE TABLE document_person (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
document_id UUID NOT NULL REFERENCES document(id) ON DELETE CASCADE,
person_id UUID NOT NULL REFERENCES person(id) ON DELETE CASCADE,
role VARCHAR(20) NOT NULL, -- 'author' or 'recipient'
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
CONSTRAINT unique_document_person_role UNIQUE (document_id, person_id, role)
);
-- 4. JOB TABLE (Batch-level orchestrator)
CREATE TABLE job (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
document_id UUID NOT NULL REFERENCES document(id) ON DELETE CASCADE,
status VARCHAR(50) NOT NULL DEFAULT 'queued', -- 'queued', 'processing', 'completed', 'partial_success', 'failed'
retry_count INTEGER NOT NULL DEFAULT 0,
provider TEXT NOT NULL, -- e.g., 'openai', 'anthropic'
model TEXT NOT NULL, -- e.g., 'gpt-4o', 'claude-3-5-sonnet'
prompt_name TEXT,
date_created TIMESTAMPTZ NOT NULL DEFAULT now(),
date_updated TIMESTAMPTZ NOT NULL DEFAULT now()
);
-- 5. SOURCE TABLE (Physical image files & active state)
CREATE TABLE source (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
document_id UUID NOT NULL REFERENCES document(id) ON DELETE CASCADE,
page_number INTEGER NOT NULL DEFAULT 1,
upload_name TEXT NOT NULL,
filename TEXT NOT NULL,
file_path TEXT NOT NULL,
raw_transcription TEXT, -- Cached active AI text output
revised_text TEXT, -- Active human edited text
date_uploaded TIMESTAMPTZ NOT NULL DEFAULT now(),
date_revised TIMESTAMPTZ
);
-- 6. JOB_SOURCE (Junction Table: Per-Image Execution Record)
CREATE TABLE job_source (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
job_id UUID NOT NULL REFERENCES job(id) ON DELETE CASCADE,
source_id UUID NOT NULL REFERENCES source(id) ON DELETE CASCADE,
status VARCHAR(50) NOT NULL DEFAULT 'pending', -- 'pending', 'transcribed', 'failed'
raw_transcription TEXT, -- Point-in-time raw AI text output
ai_metadata JSONB, -- Page-level bounding boxes, tokens, confidence
raw_api_response JSONB, -- Complete REST response envelope
error_detail TEXT,
executed_at TIMESTAMPTZ NOT NULL DEFAULT now(),
CONSTRAINT unique_job_source UNIQUE (job_id, source_id)
);
-- INDEXES FOR FAST LOOKUPS & QUERY PERFORMANCE
CREATE INDEX idx_person_full_name ON person(full_name);
CREATE INDEX idx_document_date ON document(document_date);
CREATE INDEX idx_document_person_doc ON document_person(document_id);
CREATE INDEX idx_document_person_per ON document_person(person_id);
CREATE INDEX idx_source_document ON source(document_id);
CREATE INDEX idx_source_page_order ON source(document_id, page_number);
CREATE INDEX idx_job_document ON job(document_id);
CREATE INDEX idx_job_source_job ON job_source(job_id);
CREATE INDEX idx_job_source_source ON job_source(source_id);
CREATE INDEX idx_job_source_ai_metadata ON job_source USING GIN (ai_metadata);
```
docs/ver2/V2 TypeScript Zod Schemas.md
+169
View File
@@ -0,0 +1,169 @@
## TypeScript Zod schemas
Here are the TypeScript Zod schemas matching your V2 PostgreSQL database definition.
These schemas cover:
1. Database Entities: Pure runtime validators representing rows fetched directly from PostgreSQL.
2. AI Payload Extensions: The structured document output stored inside job.ai_metadata.
3. Insert/Create Schemas: Utility types derived with .omit() for creating new records where auto-generated columns (id, created_at, updated_at, etc.) are handled by PostgreSQL defaults.
---
```
import { z } from "zod";
// ==========================================
// 1. ATOMIC & REUSABLE SCHEMAS
// ==========================================
export const UUIDSchema = z.string().uuid();
export const ISODateTimeSchema = z.coerce.date();
export const BoundingBoxSchema = z.object({
ymin: z.number().min(0).max(1000),
xmin: z.number().min(0).max(1000),
ymax: z.number().min(0).max(1000),
xmax: z.number().min(0).max(1000),
});
export const BlockTypeSchema = z.enum([
"heading",
"paragraph",
"table",
"margin_note",
"signature",
"footnote",
"header",
]);
// ==========================================
// 2. PAGE-LEVEL AI METADATA SCHEMA (job_source.ai_metadata)
// ==========================================
export const TranscribedBlockSchema = z.object({
text: z.string(),
confidence: z.number().min(0).max(1),
blockType: BlockTypeSchema,
boundingBox: BoundingBoxSchema.optional(),
});
export const PageAIMetadataSchema = z.object({
detectedLanguage: z.string().optional(),
overallConfidence: z.number().min(0).max(1),
blocks: z.array(TranscribedBlockSchema),
inputTokens: z.number().optional(),
outputTokens: z.number().optional(),
extractedEntities: z.record(z.string(), z.unknown()).optional(),
});
export type PageAIMetadata = z.infer<typeof PageAIMetadataSchema>;
// ==========================================
// 3. TABLE ENTITY SCHEMAS
// ==========================================
// --- PERSON TABLE ---
export const PersonSchema = z.object({
id: UUIDSchema,
fullName: z.string().min(1),
displayName: z.string().nullable().optional(),
maidenName: z.string().nullable().optional(),
birthDate: z.string().nullable().optional(),
birthDateRaw: z.string().nullable().optional(),
birthPlace: z.string().nullable().optional(),
deathDate: z.string().nullable().optional(),
deathDateRaw: z.string().nullable().optional(),
deathPlace: z.string().nullable().optional(),
biography: z.string().nullable().optional(),
portraitPath: z.string().nullable().optional(),
metadata: z.record(z.string(), z.unknown()).default({}),
createdAt: ISODateTimeSchema,
updatedAt: ISODateTimeSchema,
});
// --- DOCUMENT TABLE ---
export const DocumentSchema = z.object({
id: UUIDSchema,
name: z.string().min(1),
documentType: z.string().nullable().optional(),
documentDate: z.string().nullable().optional(),
documentDateRaw: z.string().nullable().optional(),
locationCreated: z.string().nullable().optional(),
notes: z.string().nullable().optional(),
archiveIdentifier: z.string().nullable().optional(),
createdAt: ISODateTimeSchema,
updatedAt: ISODateTimeSchema,
});
// --- DOCUMENT_PERSON JUNCTION ---
export const PersonRoleSchema = z.enum(["author", "recipient"]);
export const DocumentPersonSchema = z.object({
id: UUIDSchema,
documentId: UUIDSchema,
personId: UUIDSchema,
role: PersonRoleSchema,
createdAt: ISODateTimeSchema,
});
// --- JOB TABLE ---
export const JobStatusSchema = z.enum([
"queued",
"processing",
"completed",
"partial_success",
"failed",
]);
export const JobSchema = z.object({
id: UUIDSchema,
documentId: UUIDSchema,
status: JobStatusSchema.default("queued"),
retryCount: z.number().int().nonnegative().default(0),
provider: z.string(),
model: z.string(),
promptName: z.string().nullable().optional(),
dateCreated: ISODateTimeSchema,
dateUpdated: ISODateTimeSchema,
});
// --- SOURCE TABLE ---
export const SourceSchema = z.object({
id: UUIDSchema,
documentId: UUIDSchema,
pageNumber: z.number().int().positive().default(1),
uploadName: z.string(),
filename: z.string(),
filePath: z.string(),
rawTranscription: z.string().nullable().optional(),
revisedText: z.string().nullable().optional(),
dateUploaded: ISODateTimeSchema,
dateRevised: ISODateTimeSchema.nullable().optional(),
});
// --- JOB_SOURCE JUNCTION (Page Execution Output) ---
export const JobSourceStatusSchema = z.enum([
"pending",
"transcribed",
"failed",
]);
export const JobSourceSchema = z.object({
id: UUIDSchema,
jobId: UUIDSchema,
sourceId: UUIDSchema,
status: JobSourceStatusSchema.default("pending"),
rawTranscription: z.string().nullable().optional(),
aiMetadata: PageAIMetadataSchema.nullable().optional(),
rawApiResponse: z.record(z.string(), z.unknown()).nullable().optional(),
errorDetail: z.string().nullable().optional(),
executedAt: ISODateTimeSchema,
});
export type Person = z.infer<typeof PersonSchema>;
export type Document = z.infer<typeof DocumentSchema>;
export type DocumentPerson = z.infer<typeof DocumentPersonSchema>;
export type Job = z.infer<typeof JobSchema>;
export type Source = z.infer<typeof SourceSchema>;
export type JobSource = z.infer<typeof JobSourceSchema>;
```
docs/ver2/ver2.md
+28 -3
View File
@@ -1,5 +1,30 @@
# Version 2 Plan
Desired Enhancements:
1. Data store
* Upgrade db to PostgresSQL
* Begin capturing JSONB data (which will allow future migration to MongoDB if desired)
* Relocate db and uploaded images to a location outside of the project folder that can be backed up. (This needs to be done for all projects.) (c:/github/data/transcription?)
2. Add ability to upload multiple images (or a folder of images)
* How many is too many?
* If there is a practical max image count, can I break a block of images up into smaller batches automatically?
3. UI
* Introduce the concept of "documents" to the UI.
* Before an image can be uploaded a "document" needs to be created/defined.
* As part of the upload process, document images need to be associated with a document.
* Multiple image upload
* Refine the job detail/log screen
* Is document id + original filename the best name for uploaded images?
* How to present multiple images within one job?
* Add document name, original filename to job detail.
## Purpose
Version 2 extends the V1 baseline by introducing a production-oriented persistence architecture while preserving current user workflows.
@@ -17,11 +42,11 @@ V1 behavior remains the functional baseline unless explicitly superseded by appr
1. **Relational migration complete**
- PostgreSQL becomes the default system of record for `Document`, `Source`, `Job`, and `Revision`.
2. **Operational maturity**
1. **Operational maturity**
- Repeatable migrations, rollback paths, and environment-specific deployment procedures are documented and tested.
3. **Optional document store integration**
1. **Optional document store integration**
- MongoDB is introduced only for clearly scoped use cases that do not replace canonical relational ownership.
4. **No regression of V1 workflows**
1. **No regression of V1 workflows**
- Upload, queue/worker processing, status inspection, original transcription, and optional single revision remain stable.
---