bbchops/transcription
1
0
Fork 0
generated from john/python-template
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

V4.1 major revision to docs. Removed all obsolete documents, updated v4.2 implementation scope and plan.

Browse Source
This commit is contained in:
Jim Lancaster
2026-08-13 15:32:40 -05:00
parent 171132919d
commit 28811d79ce
60 changed files with 1170 additions and 7471 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/ver4/architecture_v4.md
+34 -6
View File
@@ -4,15 +4,40 @@ This document describes the production architecture of the document transcriptio
## Architecture Objectives
- Preserve original source material and immutable machine transcription output.
- Preserve original source material, per-execution machine output, and separate human revision.
- Support batching one or more images into ordered multi-page documents.
- Capture complete submission-time prompt provenance and per-page provider response evidence.
- Capture submission-time prompt provenance and a per-page OpenRouter SDK response snapshot.
- Execute page transcription concurrently with bounded `asyncio` workers.
- Maintain relational portability across SQLite and PostgreSQL.
- Keep operator workflows cross-platform and Python-driven.
- Support many-to-many document-person relationships with extensible roles.
- Support registry-driven document type classification.
## Core Capabilities
- Ingest one or more images into sequential `Source` pages under a `Document`.
- Execute asynchronous vision transcription with bounded worker concurrency.
- Preserve original source files with SHA-256 digests and byte sizes.
- Freeze prompt text, prompt hash, model, and explicitly configured sampling parameters on each `Job`.
- Preserve page-level machine output, normalized metadata, and an SDK-serialized OpenRouter response snapshot on `JobSource`.
- Organize historical `Person` records through many-to-many Document relationships and extensible roles.
- Classify Documents through a registry with stable type codes.
- Maintain human revision separately from machine-generated text.
- Isolate page failures so multi-page jobs can complete with partial success.
- Operate across supported platforms through Python-based application and maintenance tooling.
V4.2 extends this baseline with exact OpenRouter transport evidence and provider-neutral derived-artifact provenance. See the [V4.2 Scope Boundary](../ver4.2/scope_boundary_v4_2.md).
## Technical Stack
- **Runtime:** Python 3.12 or later.
- **Web application:** FastAPI and NiceGUI.
- **Persistence:** SQLModel and SQLAlchemy, with SQLite and PostgreSQL support.
- **Validation and settings:** Pydantic V2 and pydantic-settings.
- **Concurrency:** Python `asyncio` workers.
- **Vision integration:** OpenRouter through the application's provider adapter.
- **Testing and quality:** pytest, pytest-asyncio, Ruff, and ty.
## Runtime Topology
The runtime operates as an asynchronous Python application:
@@ -110,7 +135,7 @@ Responsibilities:
1. User uploads one or more images for a `Document`.
2. System stores files, hashes them, creates ordered `Source` rows, and creates a `Job`.
3. Worker claims the job, marks it `processing`, and executes page calls concurrently.
4. Each page writes a `JobSource` result with raw output, metadata, and full provider response evidence.
4. Each page writes a `JobSource` result with machine output, normalized metadata, and an SDK-serialized OpenRouter response snapshot.
5. Aggregate status becomes `completed`, `partial_success`, or `failed`.
### 2. Document-Person Relationship Management
@@ -128,12 +153,13 @@ Responsibilities:
3. Persistence stores the `document_type_id` reference.
4. Inactive types remain valid for historical rows but are excluded from default selectors.
## Domain Invariants
## V4 Domain Rules
- `Source.raw_transcription` stores immutable machine output.
- `JobSource.raw_transcription` preserves page output for its Job execution.
- `Source.raw_transcription` is the latest-success machine-output projection for a page.
- Human corrections occur only in `Source.revised_text`.
- Prompt and parameter provenance is frozen on `Job` at submission time.
- Provider output evidence is stored on `JobSource` for each page execution.
- The SDK-serialized OpenRouter response snapshot is stored on `JobSource` for each successful page execution.
- `DocumentPerson` links are unique for `(document_id, person_id, role_id)`.
- Relationship mutations are deterministic and set-based.
- `DocumentType.code` is stable; `DocumentType.label` may evolve.
@@ -158,3 +184,5 @@ Responsibilities:
- [System Requirements](requirements_v4.md)
- [Data Model](schema_v4.md)
- [Error Handling Policy](error_handling_v4.md)
- [Error Handling Invariant](../invariant/error_handling.md)
- [Digital Evidence and AI Processing Provenance](../invariant/ai_evidence_and_provenance.md)
docs/ver4/error_handling_v4.md
+11 -6
View File
@@ -1,13 +1,17 @@
# Error Handling Policy (Version 4)
This document defines the canonical error-handling policy for the document transcription system.
This document defines the Version 4 taxonomy, contracts, and framework behavior used to satisfy the cross-version [Error Handling invariant](../invariant/error_handling.md).
## Error Handling Objectives
## Invariant Alignment
- Make failures visible in clear, actionable language at both the document and page levels.
- Support isolated failure handling in multi-page jobs so one failing page does not invalidate successful pages.
- Preserve diagnostic detail for validation failures, provider failures, and policy conflicts.
- Ensure consistent error envelope structure across API, UI, service, and worker boundaries.
Version 4 implements the invariant through:
- The shared error taxonomy below.
- Structured error envelopes with correlation IDs.
- Page-level failure isolation and explicit aggregate job status.
- Atomic relationship and classification writes.
- Consistent translation across API, UI, service, worker, persistence, and provider boundaries.
- Bounded retry guidance based on category and idempotency.
## Scope and Authority
@@ -104,6 +108,7 @@ HTTP status mappings:
## Related Local References
- [Error Handling Invariant](../invariant/error_handling.md)
- [System Overview](index_v4.md)
- [System Requirements](requirements_v4.md)
- [Data Model](schema_v4.md)
docs/ver4/index_v4.md
+19 -30
View File
@@ -1,40 +1,29 @@
# Document Transcription System Overview (Version 4)
This project is a personal-scale application for transcribing, organizing, and preserving historical documents, images, and related people records.
Version 4 is the architecture baseline for the personal-scale application used to transcribe, organize, and preserve historical documents, source images, and related people records.
## Start Here
## Recommended Reading Order
Read [architecture_v4.md](architecture_v4.md) first for the technical overview and system design.
1. [System Architecture](architecture_v4.md) for capabilities, technical stack, runtime structure, workflows, and component ownership.
2. [System Requirements](requirements_v4.md) for the verifiable V4 contract.
3. [Data Model](schema_v4.md) for entities, relationships, constraints, and persistence rules.
4. [Error Handling Policy](error_handling_v4.md) for the V4 taxonomy and boundary contracts.
## Core Capabilities
## Cross-Version Invariants
- Folder and multi-image ingestion into sequential `Source` pages under a single `Document`.
- Parallel asynchronous AI vision transcription using Python `asyncio` bounded by rate limits.
- Portable relational storage using SQLModel and SQLAlchemy across SQLite and PostgreSQL.
- Complete prompt and response provenance for every transcription job and page execution.
- File-integrity tracking through SHA-256 hashing and stored file sizes.
- Historical `Person` management with many-to-many document links and extensible relationship roles.
- Registry-driven `DocumentType` classification with stable codes and controlled selection.
- Inline human revision of transcribed pages while preserving immutable machine output.
- Partial-failure recovery for multi-page jobs.
- Cross-platform operational workflows driven by Python-based tooling.
- [Historical Document Transcription Design Intent](../invariant/intent.md)
- [Transcription Methodology](../invariant/transcription_methodology.md)
- [Error Handling](../invariant/error_handling.md)
- [Digital Evidence and AI Processing Provenance](../invariant/ai_evidence_and_provenance.md)
- [UI Style Guide](../invariant/ui_style_guide.md)
## Technical Stack
- Application Web Framework: FastAPI + NiceGUI
- Persistence Engine: SQLModel / SQLAlchemy
- Data Validation and Schemas: Pydantic V2
- Concurrency and Workers: Python `asyncio`
- Vision Providers: OpenAI, Anthropic, and OpenRouter adapters
## Core Documentation Index
- [System Architecture](architecture_v4.md)
- [System Requirements](requirements_v4.md)
- [Data Model](schema_v4.md)
- [Error Handling Policy](error_handling_v4.md)
## Transition Documents
## V4 Transition Documents
- [Scope Boundary](scope_boundary_v4.md)
- [Implementation Plan](implementation_plan_v4.md)
## Incremental Revisions
- [V4.1 Scope](../ver4.1/scope_boundary_v4_1.md) and [Implementation Plan](../ver4.1/implementation_plan_v4_1.md)
- [V4.2 Evidence and Provenance Scope](../ver4.2/scope_boundary_v4_2.md) and [Implementation Plan](../ver4.2/implementation_plan_v4_2.md)
- [Draft V4.3 Page Reordering and Settings Scope](../ver4.3/scope_boundary_v4_3.md) and [Draft Implementation Plan](../ver4.3/implementation_plan_v4_3.md)