bbchops/transcription
1
0
Fork 0
generated from john/python-template
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update V1 & V2 core documents and reorganize docs folder

Browse Source
This commit is contained in:
Jim Lancaster
2026-07-31 10:04:07 -05:00
parent d2b793ea69
commit 3eefc36239
25 changed files with 671 additions and 783 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/ver2/V2 Context for DB Updates.md
-35
View File
@@ -1,35 +0,0 @@
# AI Coding Assistant Project Briefing & Context
## Project Mission
This application is a family history archival and transcription platform. Its primary goal is to accept scanned document images (letters, postcards, logbooks, diaries), execute OCR and structured transcription via AI vision models (OpenAI GPT-4o, Anthropic Claude 3.5 Sonnet), and manage historical metadata (authors, recipients, dates, and locations).
---
## Technical Stack & Architecture
* **Database:** PostgreSQL 13+ with native `UUID` (`gen_random_uuid()`) and `JSONB` columns.
* **Backend Runtime / Concurrency:** Python utilizing `asyncio` for concurrent HTTP API calls to AI providers, with strict rate-limiting via `asyncio.Semaphore`.
* **Validation & Types:** TypeScript with **Zod** schema definitions. Incoming AI responses must be parsed and validated with Zod schemas *before* database insertion.
* **ORM / Database Access:** Raw parameterized SQL queries or lightweight query builders (e.g., Kysely/Prisma) respecting PostgreSQL native types.
---
## Core System Directives for AI Code Generation
### 1. Data Immutability vs. Human Corrections
* `job_source.raw_transcription` and `source.raw_transcription` represent original, point-in-time machine outputs and are **immutable**.
* Human corrections occur on `source.revised_text`.
* When fetching text for the UI, always display `COALESCE(source.revised_text, source.raw_transcription)`.
### 2. Async Execution & Batching Rules
* A `job` represents an overarching execution run for a folder/group of images belonging to a single `document`.
* Images are submitted to AI APIs **one at a time in rapid succession** using `asyncio` worker pools.
* Each single-image API call populates a row in `job_source` with its own `status`, `raw_transcription`, `ai_metadata`, and `raw_api_response`.
* If 9 of 10 pages succeed and 1 fails, `job_source.status` for the failed image becomes `'failed'`, while `job.status` becomes `'partial_success'`. Do not mark the entire batch as failed if partial results exist.
### 3. Entity Relationships
* **Authors/Recipients:** A `document` can have multiple authors and recipients. Do NOT put direct `author_id` foreign keys on `document`. Query authors/recipients via `document_person` where `role = 'author'` or `role = 'recipient'`.
* **Page Ordering:** Multi-page documents must always be queried using `ORDER BY page_number ASC`.
### 4. Database Mutations
* Always use parameterized SQL queries (`$1`, `$2`) to prevent SQL injection.
* Store datetimes using UTC ISO 8601 strings or native PostgreSQL `TIMESTAMPTZ`.
docs/ver2/V2 DB Schema.md
-122
View File
@@ -1,122 +0,0 @@
# Database Schema (V2 Architecture)
This document describes the PostgreSQL relational schema for the transcription platform. It incorporates multi-image batch orchestration via `asyncio`, page-level execution tracking, many-to-many author/recipient attribution, and JSONB document storage for AI vision outputs.
All primary and foreign keys are PostgreSQL native UUIDs (`gen_random_uuid()`).
## Entity Relationship Diagram
```mermaid
erDiagram
PERSON {
UUID id PK
TEXT full_name
TEXT display_name
TEXT maiden_name
DATE birth_date
TEXT birth_date_raw
TEXT birth_place
DATE death_date
TEXT death_date_raw
TEXT death_place
TEXT biography
TEXT portrait_path
JSONB metadata
TIMESTAMPTZ created_at
TIMESTAMPTZ updated_at
}
DOCUMENT {
UUID id PK
TEXT name
TEXT document_type
DATE document_date
TEXT document_date_raw
TEXT location_created
TEXT notes
TEXT archive_identifier
TIMESTAMPTZ created_at
TIMESTAMPTZ updated_at
}
DOCUMENT_PERSON {
UUID id PK
UUID document_id FK
UUID person_id FK
VARCHAR role "author | recipient"
TIMESTAMPTZ created_at
}
JOB {
UUID id PK
UUID document_id FK
VARCHAR status "queued | processing | completed | partial_success | failed"
INTEGER retry_count
TEXT provider
TEXT model
TEXT prompt_name
TIMESTAMPTZ date_created
TIMESTAMPTZ date_updated
}
SOURCE {
UUID id PK
UUID document_id FK
INTEGER page_number
TEXT upload_name
TEXT filename
TEXT file_path
TEXT raw_transcription
TEXT revised_text
TIMESTAMPTZ date_uploaded
TIMESTAMPTZ date_revised
}
JOB_SOURCE {
UUID id PK
UUID job_id FK
UUID source_id FK
VARCHAR status "pending | transcribed | failed"
TEXT raw_transcription
JSONB ai_metadata
JSONB raw_api_response
TEXT error_detail
TIMESTAMPTZ executed_at
}
DOCUMENT ||--o{ DOCUMENT_PERSON : "has_people"
PERSON ||--o{ DOCUMENT_PERSON : "participates_in"
DOCUMENT ||--o{ JOB : "has_jobs"
DOCUMENT ||--o{ SOURCE : "contains_pages"
JOB ||--o{ JOB_SOURCE : "executes"
SOURCE ||--o{ JOB_SOURCE : "processed_in"
```
## Domain Invariants & Rules
### Page-Level Execution & AI Outputs
* Execution Granularity: Every single image execution by an AI model produces a dedicated record in job_source.
* Point-in-Time Auditability: job_source.raw_api_response stores the unparsed REST response envelope for that specific image page call. job_source.ai_metadata stores spatial bounding boxes, token usage, and layout details for that specific image page call.
* Active Output Caching: Upon successful completion of an image call, source.raw_transcription is updated with the latest output string from job_source.raw_transcription for fast UI rendering.
### Page Ordering & Revisions
* Sequential Integrity: source.page_number dictates page ordering within a document. Reads assembling full documents must query ORDER BY source.document_id, source.page_number ASC.
* Inlined Human Corrections: User edits occur at the page level inside source.revised_text. source.raw_transcription remains immutable. If source.revised_text is non-null, application frontends must render source.revised_text.
### Async Job Lifecycle & Failure Isolation
* Batch Orchestrator: A job represents an overarching execution run across one or more source images belonging to a document.
* Isolated Failures: API requests run concurrently (e.g., using asyncio). A failure on page 3 does not invalidate successful transcriptions on page 1 or 2.
* Job States:
- queued: Created, awaiting worker execution.
- processing: Concurrent HTTP tasks actively running.
- completed: 100% of linked job_source tasks succeeded (transcribed).
- partial_success: At least one job_source succeeded and at least one failed.
- failed: All linked job_source tasks failed or a job-level runtime error occurred.
### Attribution & Person Roles
* Multi-Person Roles: Documents support zero, one, or many authors and recipients linked via document_person.
* Role Uniqueness: (document_id, person_id, role) must be unique to prevent duplicate role tagging.
docs/ver2/V2 PostgreSQL DDL Specification.md
-102
View File
@@ -1,102 +0,0 @@
## PostgreSQL DDL Specification
```sql
-- Enable pgcrypto for UUID generation if on PostgreSQL < 13
CREATE EXTENSION IF NOT EXISTS "pgcrypto";
-- 1. PERSON TABLE
CREATE TABLE person (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
full_name TEXT NOT NULL,
display_name TEXT,
maiden_name TEXT,
birth_date DATE,
birth_date_raw TEXT,
birth_place TEXT,
death_date DATE,
death_date_raw TEXT,
death_place TEXT,
biography TEXT,
portrait_path TEXT,
metadata JSONB DEFAULT '{}'::jsonb,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
-- 2. DOCUMENT TABLE
CREATE TABLE document (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
name TEXT NOT NULL,
document_type TEXT,
document_date DATE,
document_date_raw TEXT,
location_created TEXT,
notes TEXT,
archive_identifier TEXT,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
-- 3. DOCUMENT_PERSON (Junction Table for Multi-Author / Multi-Recipient)
CREATE TABLE document_person (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
document_id UUID NOT NULL REFERENCES document(id) ON DELETE CASCADE,
person_id UUID NOT NULL REFERENCES person(id) ON DELETE CASCADE,
role VARCHAR(20) NOT NULL, -- 'author' or 'recipient'
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
CONSTRAINT unique_document_person_role UNIQUE (document_id, person_id, role)
);
-- 4. JOB TABLE (Batch-level orchestrator)
CREATE TABLE job (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
document_id UUID NOT NULL REFERENCES document(id) ON DELETE CASCADE,
status VARCHAR(50) NOT NULL DEFAULT 'queued', -- 'queued', 'processing', 'completed', 'partial_success', 'failed'
retry_count INTEGER NOT NULL DEFAULT 0,
provider TEXT NOT NULL, -- e.g., 'openai', 'anthropic'
model TEXT NOT NULL, -- e.g., 'gpt-4o', 'claude-3-5-sonnet'
prompt_name TEXT,
date_created TIMESTAMPTZ NOT NULL DEFAULT now(),
date_updated TIMESTAMPTZ NOT NULL DEFAULT now()
);
-- 5. SOURCE TABLE (Physical image files & active state)
CREATE TABLE source (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
document_id UUID NOT NULL REFERENCES document(id) ON DELETE CASCADE,
page_number INTEGER NOT NULL DEFAULT 1,
upload_name TEXT NOT NULL,
filename TEXT NOT NULL,
file_path TEXT NOT NULL,
raw_transcription TEXT, -- Cached active AI text output
revised_text TEXT, -- Active human edited text
date_uploaded TIMESTAMPTZ NOT NULL DEFAULT now(),
date_revised TIMESTAMPTZ
);
-- 6. JOB_SOURCE (Junction Table: Per-Image Execution Record)
CREATE TABLE job_source (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
job_id UUID NOT NULL REFERENCES job(id) ON DELETE CASCADE,
source_id UUID NOT NULL REFERENCES source(id) ON DELETE CASCADE,
status VARCHAR(50) NOT NULL DEFAULT 'pending', -- 'pending', 'transcribed', 'failed'
raw_transcription TEXT, -- Point-in-time raw AI text output
ai_metadata JSONB, -- Page-level bounding boxes, tokens, confidence
raw_api_response JSONB, -- Complete REST response envelope
error_detail TEXT,
executed_at TIMESTAMPTZ NOT NULL DEFAULT now(),
CONSTRAINT unique_job_source UNIQUE (job_id, source_id)
);
-- INDEXES FOR FAST LOOKUPS & QUERY PERFORMANCE
CREATE INDEX idx_person_full_name ON person(full_name);
CREATE INDEX idx_document_date ON document(document_date);
CREATE INDEX idx_document_person_doc ON document_person(document_id);
CREATE INDEX idx_document_person_per ON document_person(person_id);
CREATE INDEX idx_source_document ON source(document_id);
CREATE INDEX idx_source_page_order ON source(document_id, page_number);
CREATE INDEX idx_job_document ON job(document_id);
CREATE INDEX idx_job_source_job ON job_source(job_id);
CREATE INDEX idx_job_source_source ON job_source(source_id);
CREATE INDEX idx_job_source_ai_metadata ON job_source USING GIN (ai_metadata);
```
docs/ver2/V2 TypeScript Zod Schemas.md
-169
View File
@@ -1,169 +0,0 @@
## TypeScript Zod schemas
Here are the TypeScript Zod schemas matching your V2 PostgreSQL database definition.
These schemas cover:
1. Database Entities: Pure runtime validators representing rows fetched directly from PostgreSQL.
2. AI Payload Extensions: The structured document output stored inside job.ai_metadata.
3. Insert/Create Schemas: Utility types derived with .omit() for creating new records where auto-generated columns (id, created_at, updated_at, etc.) are handled by PostgreSQL defaults.
---
```
import { z } from "zod";
// ==========================================
// 1. ATOMIC & REUSABLE SCHEMAS
// ==========================================
export const UUIDSchema = z.string().uuid();
export const ISODateTimeSchema = z.coerce.date();
export const BoundingBoxSchema = z.object({
ymin: z.number().min(0).max(1000),
xmin: z.number().min(0).max(1000),
ymax: z.number().min(0).max(1000),
xmax: z.number().min(0).max(1000),
});
export const BlockTypeSchema = z.enum([
"heading",
"paragraph",
"table",
"margin_note",
"signature",
"footnote",
"header",
]);
// ==========================================
// 2. PAGE-LEVEL AI METADATA SCHEMA (job_source.ai_metadata)
// ==========================================
export const TranscribedBlockSchema = z.object({
text: z.string(),
confidence: z.number().min(0).max(1),
blockType: BlockTypeSchema,
boundingBox: BoundingBoxSchema.optional(),
});
export const PageAIMetadataSchema = z.object({
detectedLanguage: z.string().optional(),
overallConfidence: z.number().min(0).max(1),
blocks: z.array(TranscribedBlockSchema),
inputTokens: z.number().optional(),
outputTokens: z.number().optional(),
extractedEntities: z.record(z.string(), z.unknown()).optional(),
});
export type PageAIMetadata = z.infer<typeof PageAIMetadataSchema>;
// ==========================================
// 3. TABLE ENTITY SCHEMAS
// ==========================================
// --- PERSON TABLE ---
export const PersonSchema = z.object({
id: UUIDSchema,
fullName: z.string().min(1),
displayName: z.string().nullable().optional(),
maidenName: z.string().nullable().optional(),
birthDate: z.string().nullable().optional(),
birthDateRaw: z.string().nullable().optional(),
birthPlace: z.string().nullable().optional(),
deathDate: z.string().nullable().optional(),
deathDateRaw: z.string().nullable().optional(),
deathPlace: z.string().nullable().optional(),
biography: z.string().nullable().optional(),
portraitPath: z.string().nullable().optional(),
metadata: z.record(z.string(), z.unknown()).default({}),
createdAt: ISODateTimeSchema,
updatedAt: ISODateTimeSchema,
});
// --- DOCUMENT TABLE ---
export const DocumentSchema = z.object({
id: UUIDSchema,
name: z.string().min(1),
documentType: z.string().nullable().optional(),
documentDate: z.string().nullable().optional(),
documentDateRaw: z.string().nullable().optional(),
locationCreated: z.string().nullable().optional(),
notes: z.string().nullable().optional(),
archiveIdentifier: z.string().nullable().optional(),
createdAt: ISODateTimeSchema,
updatedAt: ISODateTimeSchema,
});
// --- DOCUMENT_PERSON JUNCTION ---
export const PersonRoleSchema = z.enum(["author", "recipient"]);
export const DocumentPersonSchema = z.object({
id: UUIDSchema,
documentId: UUIDSchema,
personId: UUIDSchema,
role: PersonRoleSchema,
createdAt: ISODateTimeSchema,
});
// --- JOB TABLE ---
export const JobStatusSchema = z.enum([
"queued",
"processing",
"completed",
"partial_success",
"failed",
]);
export const JobSchema = z.object({
id: UUIDSchema,
documentId: UUIDSchema,
status: JobStatusSchema.default("queued"),
retryCount: z.number().int().nonnegative().default(0),
provider: z.string(),
model: z.string(),
promptName: z.string().nullable().optional(),
dateCreated: ISODateTimeSchema,
dateUpdated: ISODateTimeSchema,
});
// --- SOURCE TABLE ---
export const SourceSchema = z.object({
id: UUIDSchema,
documentId: UUIDSchema,
pageNumber: z.number().int().positive().default(1),
uploadName: z.string(),
filename: z.string(),
filePath: z.string(),
rawTranscription: z.string().nullable().optional(),
revisedText: z.string().nullable().optional(),
dateUploaded: ISODateTimeSchema,
dateRevised: ISODateTimeSchema.nullable().optional(),
});
// --- JOB_SOURCE JUNCTION (Page Execution Output) ---
export const JobSourceStatusSchema = z.enum([
"pending",
"transcribed",
"failed",
]);
export const JobSourceSchema = z.object({
id: UUIDSchema,
jobId: UUIDSchema,
sourceId: UUIDSchema,
status: JobSourceStatusSchema.default("pending"),
rawTranscription: z.string().nullable().optional(),
aiMetadata: PageAIMetadataSchema.nullable().optional(),
rawApiResponse: z.record(z.string(), z.unknown()).nullable().optional(),
errorDetail: z.string().nullable().optional(),
executedAt: ISODateTimeSchema,
});
export type Person = z.infer<typeof PersonSchema>;
export type Document = z.infer<typeof DocumentSchema>;
export type DocumentPerson = z.infer<typeof DocumentPersonSchema>;
export type Job = z.infer<typeof JobSchema>;
export type Source = z.infer<typeof SourceSchema>;
export type JobSource = z.infer<typeof JobSourceSchema>;
```
docs/ver2/ver2.md
-150
View File
@@ -1,150 +0,0 @@
# Version 2 Plan
Desired Enhancements:
1. Data store
* Upgrade db to PostgresSQL
* Begin capturing JSONB data (which will allow future migration to MongoDB if desired)
* Relocate db and uploaded images to a location outside of the project folder that can be backed up. (This needs to be done for all projects.) (c:/github/data/transcription?)
2. Add ability to upload multiple images (or a folder of images)
* How many is too many?
* If there is a practical max image count, can I break a block of images up into smaller batches automatically?
3. UI
* Introduce the concept of "documents" to the UI.
* Before an image can be uploaded a "document" needs to be created/defined.
* As part of the upload process, document images need to be associated with a document.
* Multiple image upload
* Refine the job detail/log screen
* Is document id + original filename the best name for uploaded images?
* How to present multiple images within one job?
* Add document name, original filename to job detail.
## Purpose
Version 2 extends the V1 baseline by introducing a production-oriented persistence architecture while preserving current user workflows.
Primary target changes:
- Migrate relational persistence from SQLite to PostgreSQL
- Introduce optional MongoDB for document-oriented adjunct data (non-canonical)
V1 behavior remains the functional baseline unless explicitly superseded by approved V2 requirements.
---
## V2 Goals
1. **Relational migration complete**
- PostgreSQL becomes the default system of record for `Document`, `Source`, `Job`, and `Revision`.
1. **Operational maturity**
- Repeatable migrations, rollback paths, and environment-specific deployment procedures are documented and tested.
1. **Optional document store integration**
- MongoDB is introduced only for clearly scoped use cases that do not replace canonical relational ownership.
1. **No regression of V1 workflows**
- Upload, queue/worker processing, status inspection, original transcription, and optional single revision remain stable.
---
## Non-Goals (V2)
- Replacing SQLModel domain ownership with MongoDB
- Introducing breaking UI behavior for existing V1 flows
- Expanding revision cardinality beyond current `0..1` without explicit requirements update
---
## Proposed Scope
### A) PostgreSQL migration (required)
- Add PostgreSQL runtime profile for local/dev/prod
- Introduce migration toolchain and migration history
- Convert bootstrap strategy from compatibility patching to explicit migrations
- Validate model constraints and indexes against PostgreSQL
- Add operational checks (connectivity, pool, transaction behavior)
### B) MongoDB integration (optional, gated)
- Define approved use cases (for example: denormalized read models, audit/event projections, or search-oriented materializations)
- Keep canonical write path in relational store
- Add feature flag/config gate to enable or disable Mongo features
- Document consistency model and failure behavior
---
## Milestones
## M1 — Requirements and architecture baseline
- Create V2 requirements delta from V1 baseline
- Define relational/document ownership boundaries
- Approve migration strategy and cutover approach
**Exit criteria:** signed architecture decision and updated traceability map.
## M2 — PostgreSQL foundation
- Add PostgreSQL environment wiring and secrets strategy
- Add migration framework and initial schema migration
- Add CI path using PostgreSQL service container
**Exit criteria:** test suite green on PostgreSQL in CI.
## M3 — Data migration and cutover rehearsal
- Build SQLite -> PostgreSQL migration utility/playbook
- Rehearse migration on representative datasets
- Validate rollback/recovery procedures
**Exit criteria:** successful dry-run migration with measured rollback test.
## M4 — MongoDB optional integration
- Implement scoped Mongo use case(s)
- Add fallback behavior when Mongo unavailable
- Add tests and operational runbook updates
**Exit criteria:** feature-gated Mongo behavior validated with no V1 flow regressions.
## M5 — Release readiness
- Final regression suite (functional + reliability)
- Performance and failure-mode checks
- Production release checklist and sign-off
**Exit criteria:** V2 release approval.
---
## Risks and Mitigations
- **Schema drift risk** -> enforce migration-first policy and CI migration checks.
- **Dual-store consistency risk** -> keep relational source of truth and explicit projection contracts.
- **Operational complexity** -> staged rollout, runbooks, and feature flags.
- **Regression risk in worker lifecycle** -> keep dedicated reliability tests around terminal-state guarantees.
---
## Traceability and Evidence
Maintain a V2 table with:
- requirement/change ID
- status (`not started` / `in progress` / `done`)
- implementation PR
- validation evidence (test names, migration rehearsal logs, runbook references)
---
## Suggested first implementation tasks
1. Create `docs/ver2/adr/` and draft ADR for persistence ownership boundaries.
2. Add PostgreSQL compose profile and env contract.
3. Introduce migration tooling and generate initial migration from current schema.
4. Add CI job for PostgreSQL-backed `pytest -m "not external"`.