From d2b793ea69471f5ac274b9014c106b73a58a53ec Mon Sep 17 00:00:00 2001 From: Jim Lancaster <40281233+zoltan57@users.noreply.github.com> Date: Thu, 30 Jul 2026 19:39:12 -0500 Subject: [PATCH] Begin planning V2 --- docs/schema.md | 2 +- docs/ver2/V2 Context for DB Updates.md | 35 ++++ docs/ver2/V2 DB Schema.md | 122 +++++++++++++ docs/ver2/V2 PostgreSQL DDL Specification.md | 102 +++++++++++ docs/ver2/V2 TypeScript Zod Schemas.md | 169 +++++++++++++++++++ docs/ver2/ver2.md | 31 +++- 6 files changed, 457 insertions(+), 4 deletions(-) create mode 100644 docs/ver2/V2 Context for DB Updates.md create mode 100644 docs/ver2/V2 DB Schema.md create mode 100644 docs/ver2/V2 PostgreSQL DDL Specification.md create mode 100644 docs/ver2/V2 TypeScript Zod Schemas.md diff --git a/docs/schema.md b/docs/schema.md index 21b70da..00bb673 100644 --- a/docs/schema.md +++ b/docs/schema.md @@ -41,7 +41,7 @@ erDiagram REVISION { UUID id PK - UUID source_id FK UNIQUE + UUID source_id "FK, UK" INTEGER revision TEXT text DATETIME date_created diff --git a/docs/ver2/V2 Context for DB Updates.md b/docs/ver2/V2 Context for DB Updates.md new file mode 100644 index 0000000..88c20e2 --- /dev/null +++ b/docs/ver2/V2 Context for DB Updates.md @@ -0,0 +1,35 @@ +# AI Coding Assistant Project Briefing & Context + +## Project Mission +This application is a family history archival and transcription platform. Its primary goal is to accept scanned document images (letters, postcards, logbooks, diaries), execute OCR and structured transcription via AI vision models (OpenAI GPT-4o, Anthropic Claude 3.5 Sonnet), and manage historical metadata (authors, recipients, dates, and locations). + +--- + +## Technical Stack & Architecture +* **Database:** PostgreSQL 13+ with native `UUID` (`gen_random_uuid()`) and `JSONB` columns. +* **Backend Runtime / Concurrency:** Python utilizing `asyncio` for concurrent HTTP API calls to AI providers, with strict rate-limiting via `asyncio.Semaphore`. +* **Validation & Types:** TypeScript with **Zod** schema definitions. Incoming AI responses must be parsed and validated with Zod schemas *before* database insertion. +* **ORM / Database Access:** Raw parameterized SQL queries or lightweight query builders (e.g., Kysely/Prisma) respecting PostgreSQL native types. + +--- + +## Core System Directives for AI Code Generation + +### 1. Data Immutability vs. Human Corrections +* `job_source.raw_transcription` and `source.raw_transcription` represent original, point-in-time machine outputs and are **immutable**. +* Human corrections occur on `source.revised_text`. +* When fetching text for the UI, always display `COALESCE(source.revised_text, source.raw_transcription)`. + +### 2. Async Execution & Batching Rules +* A `job` represents an overarching execution run for a folder/group of images belonging to a single `document`. +* Images are submitted to AI APIs **one at a time in rapid succession** using `asyncio` worker pools. +* Each single-image API call populates a row in `job_source` with its own `status`, `raw_transcription`, `ai_metadata`, and `raw_api_response`. +* If 9 of 10 pages succeed and 1 fails, `job_source.status` for the failed image becomes `'failed'`, while `job.status` becomes `'partial_success'`. Do not mark the entire batch as failed if partial results exist. + +### 3. Entity Relationships +* **Authors/Recipients:** A `document` can have multiple authors and recipients. Do NOT put direct `author_id` foreign keys on `document`. Query authors/recipients via `document_person` where `role = 'author'` or `role = 'recipient'`. +* **Page Ordering:** Multi-page documents must always be queried using `ORDER BY page_number ASC`. + +### 4. Database Mutations +* Always use parameterized SQL queries (`$1`, `$2`) to prevent SQL injection. +* Store datetimes using UTC ISO 8601 strings or native PostgreSQL `TIMESTAMPTZ`. \ No newline at end of file diff --git a/docs/ver2/V2 DB Schema.md b/docs/ver2/V2 DB Schema.md new file mode 100644 index 0000000..e812c01 --- /dev/null +++ b/docs/ver2/V2 DB Schema.md @@ -0,0 +1,122 @@ +# Database Schema (V2 Architecture) + +This document describes the PostgreSQL relational schema for the transcription platform. It incorporates multi-image batch orchestration via `asyncio`, page-level execution tracking, many-to-many author/recipient attribution, and JSONB document storage for AI vision outputs. + +All primary and foreign keys are PostgreSQL native UUIDs (`gen_random_uuid()`). + +## Entity Relationship Diagram + +```mermaid +erDiagram + PERSON { + UUID id PK + TEXT full_name + TEXT display_name + TEXT maiden_name + DATE birth_date + TEXT birth_date_raw + TEXT birth_place + DATE death_date + TEXT death_date_raw + TEXT death_place + TEXT biography + TEXT portrait_path + JSONB metadata + TIMESTAMPTZ created_at + TIMESTAMPTZ updated_at + } + + DOCUMENT { + UUID id PK + TEXT name + TEXT document_type + DATE document_date + TEXT document_date_raw + TEXT location_created + TEXT notes + TEXT archive_identifier + TIMESTAMPTZ created_at + TIMESTAMPTZ updated_at + } + + DOCUMENT_PERSON { + UUID id PK + UUID document_id FK + UUID person_id FK + VARCHAR role "author | recipient" + TIMESTAMPTZ created_at + } + + JOB { + UUID id PK + UUID document_id FK + VARCHAR status "queued | processing | completed | partial_success | failed" + INTEGER retry_count + TEXT provider + TEXT model + TEXT prompt_name + TIMESTAMPTZ date_created + TIMESTAMPTZ date_updated + } + + SOURCE { + UUID id PK + UUID document_id FK + INTEGER page_number + TEXT upload_name + TEXT filename + TEXT file_path + TEXT raw_transcription + TEXT revised_text + TIMESTAMPTZ date_uploaded + TIMESTAMPTZ date_revised + } + + JOB_SOURCE { + UUID id PK + UUID job_id FK + UUID source_id FK + VARCHAR status "pending | transcribed | failed" + TEXT raw_transcription + JSONB ai_metadata + JSONB raw_api_response + TEXT error_detail + TIMESTAMPTZ executed_at + } + + DOCUMENT ||--o{ DOCUMENT_PERSON : "has_people" + PERSON ||--o{ DOCUMENT_PERSON : "participates_in" + DOCUMENT ||--o{ JOB : "has_jobs" + DOCUMENT ||--o{ SOURCE : "contains_pages" + JOB ||--o{ JOB_SOURCE : "executes" + SOURCE ||--o{ JOB_SOURCE : "processed_in" +``` + +## Domain Invariants & Rules + +### Page-Level Execution & AI Outputs + +* Execution Granularity: Every single image execution by an AI model produces a dedicated record in job_source. +* Point-in-Time Auditability: job_source.raw_api_response stores the unparsed REST response envelope for that specific image page call. job_source.ai_metadata stores spatial bounding boxes, token usage, and layout details for that specific image page call. +* Active Output Caching: Upon successful completion of an image call, source.raw_transcription is updated with the latest output string from job_source.raw_transcription for fast UI rendering. + +### Page Ordering & Revisions + +* Sequential Integrity: source.page_number dictates page ordering within a document. Reads assembling full documents must query ORDER BY source.document_id, source.page_number ASC. +* Inlined Human Corrections: User edits occur at the page level inside source.revised_text. source.raw_transcription remains immutable. If source.revised_text is non-null, application frontends must render source.revised_text. + +### Async Job Lifecycle & Failure Isolation + +* Batch Orchestrator: A job represents an overarching execution run across one or more source images belonging to a document. +* Isolated Failures: API requests run concurrently (e.g., using asyncio). A failure on page 3 does not invalidate successful transcriptions on page 1 or 2. +* Job States: + - queued: Created, awaiting worker execution. + - processing: Concurrent HTTP tasks actively running. + - completed: 100% of linked job_source tasks succeeded (transcribed). + - partial_success: At least one job_source succeeded and at least one failed. + - failed: All linked job_source tasks failed or a job-level runtime error occurred. + +### Attribution & Person Roles + +* Multi-Person Roles: Documents support zero, one, or many authors and recipients linked via document_person. +* Role Uniqueness: (document_id, person_id, role) must be unique to prevent duplicate role tagging. \ No newline at end of file diff --git a/docs/ver2/V2 PostgreSQL DDL Specification.md b/docs/ver2/V2 PostgreSQL DDL Specification.md new file mode 100644 index 0000000..9eaf912 --- /dev/null +++ b/docs/ver2/V2 PostgreSQL DDL Specification.md @@ -0,0 +1,102 @@ +## PostgreSQL DDL Specification + +```sql +-- Enable pgcrypto for UUID generation if on PostgreSQL < 13 +CREATE EXTENSION IF NOT EXISTS "pgcrypto"; + +-- 1. PERSON TABLE +CREATE TABLE person ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + full_name TEXT NOT NULL, + display_name TEXT, + maiden_name TEXT, + birth_date DATE, + birth_date_raw TEXT, + birth_place TEXT, + death_date DATE, + death_date_raw TEXT, + death_place TEXT, + biography TEXT, + portrait_path TEXT, + metadata JSONB DEFAULT '{}'::jsonb, + created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + updated_at TIMESTAMPTZ NOT NULL DEFAULT now() +); + +-- 2. DOCUMENT TABLE +CREATE TABLE document ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + name TEXT NOT NULL, + document_type TEXT, + document_date DATE, + document_date_raw TEXT, + location_created TEXT, + notes TEXT, + archive_identifier TEXT, + created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + updated_at TIMESTAMPTZ NOT NULL DEFAULT now() +); + +-- 3. DOCUMENT_PERSON (Junction Table for Multi-Author / Multi-Recipient) +CREATE TABLE document_person ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + document_id UUID NOT NULL REFERENCES document(id) ON DELETE CASCADE, + person_id UUID NOT NULL REFERENCES person(id) ON DELETE CASCADE, + role VARCHAR(20) NOT NULL, -- 'author' or 'recipient' + created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + CONSTRAINT unique_document_person_role UNIQUE (document_id, person_id, role) +); + +-- 4. JOB TABLE (Batch-level orchestrator) +CREATE TABLE job ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + document_id UUID NOT NULL REFERENCES document(id) ON DELETE CASCADE, + status VARCHAR(50) NOT NULL DEFAULT 'queued', -- 'queued', 'processing', 'completed', 'partial_success', 'failed' + retry_count INTEGER NOT NULL DEFAULT 0, + provider TEXT NOT NULL, -- e.g., 'openai', 'anthropic' + model TEXT NOT NULL, -- e.g., 'gpt-4o', 'claude-3-5-sonnet' + prompt_name TEXT, + date_created TIMESTAMPTZ NOT NULL DEFAULT now(), + date_updated TIMESTAMPTZ NOT NULL DEFAULT now() +); + +-- 5. SOURCE TABLE (Physical image files & active state) +CREATE TABLE source ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + document_id UUID NOT NULL REFERENCES document(id) ON DELETE CASCADE, + page_number INTEGER NOT NULL DEFAULT 1, + upload_name TEXT NOT NULL, + filename TEXT NOT NULL, + file_path TEXT NOT NULL, + raw_transcription TEXT, -- Cached active AI text output + revised_text TEXT, -- Active human edited text + date_uploaded TIMESTAMPTZ NOT NULL DEFAULT now(), + date_revised TIMESTAMPTZ +); + +-- 6. JOB_SOURCE (Junction Table: Per-Image Execution Record) +CREATE TABLE job_source ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + job_id UUID NOT NULL REFERENCES job(id) ON DELETE CASCADE, + source_id UUID NOT NULL REFERENCES source(id) ON DELETE CASCADE, + status VARCHAR(50) NOT NULL DEFAULT 'pending', -- 'pending', 'transcribed', 'failed' + raw_transcription TEXT, -- Point-in-time raw AI text output + ai_metadata JSONB, -- Page-level bounding boxes, tokens, confidence + raw_api_response JSONB, -- Complete REST response envelope + error_detail TEXT, + executed_at TIMESTAMPTZ NOT NULL DEFAULT now(), + CONSTRAINT unique_job_source UNIQUE (job_id, source_id) +); + +-- INDEXES FOR FAST LOOKUPS & QUERY PERFORMANCE +CREATE INDEX idx_person_full_name ON person(full_name); +CREATE INDEX idx_document_date ON document(document_date); +CREATE INDEX idx_document_person_doc ON document_person(document_id); +CREATE INDEX idx_document_person_per ON document_person(person_id); +CREATE INDEX idx_source_document ON source(document_id); +CREATE INDEX idx_source_page_order ON source(document_id, page_number); +CREATE INDEX idx_job_document ON job(document_id); +CREATE INDEX idx_job_source_job ON job_source(job_id); +CREATE INDEX idx_job_source_source ON job_source(source_id); +CREATE INDEX idx_job_source_ai_metadata ON job_source USING GIN (ai_metadata); +``` \ No newline at end of file diff --git a/docs/ver2/V2 TypeScript Zod Schemas.md b/docs/ver2/V2 TypeScript Zod Schemas.md new file mode 100644 index 0000000..8620c84 --- /dev/null +++ b/docs/ver2/V2 TypeScript Zod Schemas.md @@ -0,0 +1,169 @@ +## TypeScript Zod schemas + +Here are the TypeScript Zod schemas matching your V2 PostgreSQL database definition. + +These schemas cover: +1. Database Entities: Pure runtime validators representing rows fetched directly from PostgreSQL. +2. AI Payload Extensions: The structured document output stored inside job.ai_metadata. +3. Insert/Create Schemas: Utility types derived with .omit() for creating new records where auto-generated columns (id, created_at, updated_at, etc.) are handled by PostgreSQL defaults. + +--- + +``` +import { z } from "zod"; + +// ========================================== +// 1. ATOMIC & REUSABLE SCHEMAS +// ========================================== + +export const UUIDSchema = z.string().uuid(); +export const ISODateTimeSchema = z.coerce.date(); + +export const BoundingBoxSchema = z.object({ + ymin: z.number().min(0).max(1000), + xmin: z.number().min(0).max(1000), + ymax: z.number().min(0).max(1000), + xmax: z.number().min(0).max(1000), +}); + +export const BlockTypeSchema = z.enum([ + "heading", + "paragraph", + "table", + "margin_note", + "signature", + "footnote", + "header", +]); + +// ========================================== +// 2. PAGE-LEVEL AI METADATA SCHEMA (job_source.ai_metadata) +// ========================================== + +export const TranscribedBlockSchema = z.object({ + text: z.string(), + confidence: z.number().min(0).max(1), + blockType: BlockTypeSchema, + boundingBox: BoundingBoxSchema.optional(), +}); + +export const PageAIMetadataSchema = z.object({ + detectedLanguage: z.string().optional(), + overallConfidence: z.number().min(0).max(1), + blocks: z.array(TranscribedBlockSchema), + inputTokens: z.number().optional(), + outputTokens: z.number().optional(), + extractedEntities: z.record(z.string(), z.unknown()).optional(), +}); + +export type PageAIMetadata = z.infer; + +// ========================================== +// 3. TABLE ENTITY SCHEMAS +// ========================================== + +// --- PERSON TABLE --- +export const PersonSchema = z.object({ + id: UUIDSchema, + fullName: z.string().min(1), + displayName: z.string().nullable().optional(), + maidenName: z.string().nullable().optional(), + birthDate: z.string().nullable().optional(), + birthDateRaw: z.string().nullable().optional(), + birthPlace: z.string().nullable().optional(), + deathDate: z.string().nullable().optional(), + deathDateRaw: z.string().nullable().optional(), + deathPlace: z.string().nullable().optional(), + biography: z.string().nullable().optional(), + portraitPath: z.string().nullable().optional(), + metadata: z.record(z.string(), z.unknown()).default({}), + createdAt: ISODateTimeSchema, + updatedAt: ISODateTimeSchema, +}); + +// --- DOCUMENT TABLE --- +export const DocumentSchema = z.object({ + id: UUIDSchema, + name: z.string().min(1), + documentType: z.string().nullable().optional(), + documentDate: z.string().nullable().optional(), + documentDateRaw: z.string().nullable().optional(), + locationCreated: z.string().nullable().optional(), + notes: z.string().nullable().optional(), + archiveIdentifier: z.string().nullable().optional(), + createdAt: ISODateTimeSchema, + updatedAt: ISODateTimeSchema, +}); + +// --- DOCUMENT_PERSON JUNCTION --- +export const PersonRoleSchema = z.enum(["author", "recipient"]); + +export const DocumentPersonSchema = z.object({ + id: UUIDSchema, + documentId: UUIDSchema, + personId: UUIDSchema, + role: PersonRoleSchema, + createdAt: ISODateTimeSchema, +}); + +// --- JOB TABLE --- +export const JobStatusSchema = z.enum([ + "queued", + "processing", + "completed", + "partial_success", + "failed", +]); + +export const JobSchema = z.object({ + id: UUIDSchema, + documentId: UUIDSchema, + status: JobStatusSchema.default("queued"), + retryCount: z.number().int().nonnegative().default(0), + provider: z.string(), + model: z.string(), + promptName: z.string().nullable().optional(), + dateCreated: ISODateTimeSchema, + dateUpdated: ISODateTimeSchema, +}); + +// --- SOURCE TABLE --- +export const SourceSchema = z.object({ + id: UUIDSchema, + documentId: UUIDSchema, + pageNumber: z.number().int().positive().default(1), + uploadName: z.string(), + filename: z.string(), + filePath: z.string(), + rawTranscription: z.string().nullable().optional(), + revisedText: z.string().nullable().optional(), + dateUploaded: ISODateTimeSchema, + dateRevised: ISODateTimeSchema.nullable().optional(), +}); + +// --- JOB_SOURCE JUNCTION (Page Execution Output) --- +export const JobSourceStatusSchema = z.enum([ + "pending", + "transcribed", + "failed", +]); + +export const JobSourceSchema = z.object({ + id: UUIDSchema, + jobId: UUIDSchema, + sourceId: UUIDSchema, + status: JobSourceStatusSchema.default("pending"), + rawTranscription: z.string().nullable().optional(), + aiMetadata: PageAIMetadataSchema.nullable().optional(), + rawApiResponse: z.record(z.string(), z.unknown()).nullable().optional(), + errorDetail: z.string().nullable().optional(), + executedAt: ISODateTimeSchema, +}); + +export type Person = z.infer; +export type Document = z.infer; +export type DocumentPerson = z.infer; +export type Job = z.infer; +export type Source = z.infer; +export type JobSource = z.infer; +``` diff --git a/docs/ver2/ver2.md b/docs/ver2/ver2.md index 53584d0..09e558f 100644 --- a/docs/ver2/ver2.md +++ b/docs/ver2/ver2.md @@ -1,5 +1,30 @@ # Version 2 Plan +Desired Enhancements: +1. Data store + * Upgrade db to PostgresSQL + * Begin capturing JSONB data (which will allow future migration to MongoDB if desired) + * Relocate db and uploaded images to a location outside of the project folder that can be backed up. (This needs to be done for all projects.) (c:/github/data/transcription?) +2. Add ability to upload multiple images (or a folder of images) + * How many is too many? + * If there is a practical max image count, can I break a block of images up into smaller batches automatically? +3. UI + * Introduce the concept of "documents" to the UI. + * Before an image can be uploaded a "document" needs to be created/defined. + * As part of the upload process, document images need to be associated with a document. + * Multiple image upload + * Refine the job detail/log screen + * Is document id + original filename the best name for uploaded images? + * How to present multiple images within one job? + * Add document name, original filename to job detail. + + + + + + + + ## Purpose Version 2 extends the V1 baseline by introducing a production-oriented persistence architecture while preserving current user workflows. @@ -17,11 +42,11 @@ V1 behavior remains the functional baseline unless explicitly superseded by appr 1. **Relational migration complete** - PostgreSQL becomes the default system of record for `Document`, `Source`, `Job`, and `Revision`. -2. **Operational maturity** +1. **Operational maturity** - Repeatable migrations, rollback paths, and environment-specific deployment procedures are documented and tested. -3. **Optional document store integration** +1. **Optional document store integration** - MongoDB is introduced only for clearly scoped use cases that do not replace canonical relational ownership. -4. **No regression of V1 workflows** +1. **No regression of V1 workflows** - Upload, queue/worker processing, status inspection, original transcription, and optional single revision remain stable. ---