bbchops/transcription
1
0
Fork 0
generated from john/python-template
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

V1 mostly complete except for some testing. Linting in the last step changed nearly every file which is why this commit is so larger.

Browse Source
This commit is contained in:
Jim Lancaster
2026-07-29 17:27:21 -05:00
parent bc21a97019
commit a3b3bab571
37 changed files with 1383 additions and 410 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/archive/README.md
+23
View File
@@ -0,0 +1,23 @@
# V2 Archive
This folder preserves pre-V1-alignment versions of core documentation that included planned target-state architecture material.
Archived snapshots:
- `index.pre-v1-alignment.md`
- `requirements.pre-v1-alignment.md`
- `architecture.pre-v1-alignment.md`
Purpose:
- keep a durable reference for planned architecture language
- reduce risk of losing useful V2 direction while V1 docs stay implementation-aligned
Notes:
- These files are historical snapshots, not the active V1 source of truth.
- Active V1 docs remain at:
- `docs/index.md`
- `docs/requirements.md`
- `docs/architecture.md`
- V2 planning should continue in `docs/ver2/ver2.md` and related V2 artifacts.
docs/archive/architecture.pre-v1-alignment.md
+299
View File
@@ -0,0 +1,299 @@
# Architecture
This document describes the production architecture of the personal historical-document transcription system. The system is intentionally optimized for single-user operation, low operational overhead, and clean internal boundaries that support future growth without rewrites.
## Architecture Objectives
The production architecture is designed to:
- preserve verbatim family-history source material as searchable text
- keep operational complexity low for a personal deployment
- support asynchronous transcription without requiring distributed infrastructure
- maintain clear module boundaries so extensions can be added incrementally
## Production Scope And Scale
The deployed system targets personal use and a corpus of several thousand documents processed over time. The architecture favors simple, composable building blocks over distributed orchestration.
Current scope includes:
- content source upload and metadata capture
- asynchronous transcription jobs
- prompt-library driven transcription behavior, with one Markdown file per prompt
- original transcription review and optional revision review
- full-text search over accepted transcripts
- export of transcript data
## Deployment Topology
The production deployment uses [Docker Compose](https://docs.docker.com/compose/) and treats containerized databases as extremely lightweight operational dependencies.
Running [PostgreSQL](https://www.postgresql.org/docs/) in its own container is considered simple by default for this system.
Running [MongoDB](https://www.mongodb.com/docs/) in its own container is also considered simple when document-centric storage is enabled.
Container count is not a hard architectural limit; a three-container deployment (app, PostgreSQL, MongoDB) is an acceptable baseline.
### Baseline Topology (Two Containers)
- one application container
- one PostgreSQL container
- embedded background worker execution inside the app process
### Expanded Topology (Three Containers)
- application container
- PostgreSQL container
- MongoDB container
No additional queue, scheduler, or search-engine containers are required in the baseline production setup.
## Runtime Architecture
```mermaid
flowchart LR
User[Browser User] --> App[FastAPI + NiceGUI Service]
App --> Worker[In-process Background Worker]
App --> PG[(PostgreSQL)]
App --> MG[(MongoDB Document Store)]
Worker --> AI[Transcription Provider]
Worker --> PG
Worker --> MG
```
## Runtime Ownership And Startup Policy
The current implementation now uses explicit lifespan-owned runtime resources.
- application lifespan initializes and disposes database runtime resources
- worker lifecycle is owned by application lifespan startup/shutdown
- worker receives lifespan-owned database engine dependency explicitly
- schema bootstrap policy is environment-aware and explicit:
- development/test default to bootstrap enabled
- production defaults to bootstrap disabled
- explicit override is available via configuration
This aligns implementation toward REQ-7 and REQ-10 while preserving personal-scale operational simplicity.
## Layered Module Structure
### Interface Layer
Responsibility:
- HTTP API and UI routes
- request/response validation
- status and result presentation
Out of scope:
- business-rule enforcement
- data-access implementation
### Application Layer
Responsibility:
- upload and job orchestration
- state transitions and retry policy
- coordination across domain and infrastructure ports
Out of scope:
- provider-specific protocol details
- ORM or storage-specific logic
### Domain Layer
Responsibility:
- verbatim transcription policy
- revision and provenance invariants
- confidence and annotation semantics
Out of scope:
- web framework concerns
- database and network I/O
### Infrastructure Layer
Responsibility:
- persistence adapters (PostgreSQL and MongoDB)
- transcription-provider adapter
Out of scope:
- business policy decisions
## Processing Workflow
Production transcription flow:
1. A user uploads one or more content sources through the UI or API.
2. The application validates payloads and creates document, source, and job records.
3. The in-process worker de-queues the job and calls the transcription provider.
4. The application persists original transcription output on the job, plus confidence metadata and provenance events.
5. Job status transitions from queued to processing to transcribed or failed.
6. The UI and API expose status, optional revision to original transcription, and searchable transcription text.
## Data Model Ownership
System-of-record entities:
- documents and content sources
- transcription jobs, original transcription, and status events
- transcript revisions
- provenance metadata
### Original Transcription And Revision Ownership
- each processing job stores the original immutable provider output (`text`)
- provider metadata (`provider`, `model`, `prompt_name`) and failure detail (`error_detail`) are job-owned processing artifacts
- revisions are optional user-authored edits linked to a content source
- a revision can be created from original `job.text`
- many jobs will have zero revisions; revisions are additive and never overwrite original provider output
- a document groups one or more content sources (images, PDFs, and future source types)
Storage strategy:
- PostgreSQL for relational system-of-record entities
- MongoDB for document-oriented payloads and large transcription artifacts
- versioned prompt artifacts stored as individual Markdown files for human editing and refinement
- in-memory execution state treated as ephemeral
## Transcription Prompt Asset Policy
The production system treats transcription prompts as maintainable content assets.
- each transcription prompt is stored in its own Markdown file
- prompt files are designed for direct human editing and iterative refinement
- prompt updates are independent and do not require bundling unrelated prompt changes
- prompt file identity and revision history are tracked through normal repository version control
## Simplicity Guardrails
The production system enforces these constraints to prevent accidental over-engineering:
- PostgreSQL in a container is treated as a lightweight default dependency
- MongoDB in a container is treated as a lightweight optional dependency
- three containers (app, PostgreSQL, MongoDB) is an acceptable simple deployment
- no dedicated queue or search cluster is introduced without measured need
- external infrastructure is added only behind existing ports/adapters
## Extension Path
The architecture supports additive growth without changing domain contracts.
### Stage 1: Foundation (Current)
- upload, transcription, review, search, export
- in-process worker execution
- single provider adapter
- app plus PostgreSQL deployment
### Stage 2: Throughput Hardening
- optional MongoDB document-store enablement
- optional external worker/queue process
- stronger retry and dead-letter handling
### Stage 3: Intelligence Features
- entity extraction and cross-document linking
- timeline and narrative assembly
- optional multi-provider routing
Each stage preserves existing module boundaries and keeps migration risk low.
## Test Strategy
The test strategy is aligned to personal-scale operation with fast, deterministic feedback.
### Unit Tests
- domain transcription rules and annotation behavior
- revision-history invariants
- job state-transition logic
### Integration Tests
- repository behavior and transaction boundaries
- persistence-adapter and provider adapter contract mapping
- upload-to-persistence roundtrip
### End-to-End Tests
- happy path: upload, transcribe, review, search, export
- failure path: provider error, retry, surfaced failed status
### CI Execution Model
- fast suite on each push
- optional slower provider-sandbox checks on scheduled runs
## Risks And Controls
### Runtime Responsiveness
Risk:
- long jobs can reduce responsiveness in a single-process deployment
Control:
- bounded concurrency and visible job status in the UI
### Database Concurrency Limits
Risk:
- contention can appear under sustained concurrent writes in personal-scale infrastructure
Control:
- tuned connection pooling and phased use of MongoDB for document-heavy workloads
### Provider Output Variance
Risk:
- transcription quality varies by content source type, handwriting legibility, and source quality
Control:
- first-class human review and immutable revision history
## Technology References
- [FastAPI documentation](https://fastapi.tiangolo.com/)
- [NiceGUI documentation](https://nicegui.io/documentation)
- [Docker Compose documentation](https://docs.docker.com/compose/)
- [PostgreSQL documentation](https://www.postgresql.org/docs/)
- [MongoDB documentation](https://www.mongodb.com/docs/)
## Related Local References
- [System overview](index.md)
## Glossary
- Adapter: A component that translates between internal interfaces and external systems such as databases or AI services.
- Background job: Work executed outside the request/response path so the UI remains responsive.
- Boundary: A strict separation between modules with different responsibilities.
- CI (Continuous Integration): Automated test execution for code changes.
- Contract test: A test that verifies an adapter follows expected input/output behavior at a boundary.
- Domain layer: The module that contains core business rules and invariants.
- End-to-end test: A test that validates a full user flow across the running system.
- Full-text search: Text indexing and querying optimized for natural-language search.
- In-process worker: A background executor that runs within the same application process.
- Integration test: A test that verifies interactions between real modules and infrastructure components.
- MongoDB: A document-oriented database used for flexible, high-variance data structures.
- Modular monolith: A single deployable application with strongly separated internal modules.
- Port/Interface: A stable contract used by application/domain code to call infrastructure implementations.
- Prompt artifact: A single Markdown file that defines one transcription prompt and can be revised independently.
- Provenance: Metadata that records where generated data came from and how it was produced.
- Revision history: Optional versioned record of user-authored transcription edits over time.
- System of record: The authoritative persistent store for canonical data.
- Vertical slice: A minimal end-to-end feature path spanning UI/API, application logic, and persistence.
docs/archive/index.pre-v1-alignment.md
+56
View File
@@ -0,0 +1,56 @@
## Document Transcription System
This project is a production application for transcribing and preserving historical family documents. It is intentionally designed for personal-scale use, with a simplicity-first architecture that is easy to operate and easy to extend.
## Start Here
Read [architecture.md](architecture.md) first.
The architecture page is the primary technical reference and defines:
- deployed topology and infrastructure limits
- module boundaries and dependency flow
- processing life cycle and data ownership
- test strategy, risk controls, and extension path
## What The Application Does
At a high level, users upload images or PDFs as content sources for handwritten, typed, or typeset documents, run asynchronous transcription jobs, review optional revisions, and search across accepted text.
Core capabilities:
- document grouping with one or more content sources and metadata capture
- asynchronous transcription with visible job status
- immutable original transcription persisted with each job (plus provider/model/prompt metadata)
- transcription prompt management with one Markdown file per prompt for human refinement over time
- optional revisions for user-authored edits of original immutable transcription text
- full-text search over accepted transcripts
- export of transcript data
## Production Operating Model
The system runs with minimal operational overhead:
- PostgreSQL in a dedicated Docker container is considered extremely lightweight and simple for this system
- MongoDB in a dedicated Docker container is also considered extremely lightweight and simple for document-centric persistence
- a three-container deployment (app, PostgreSQL, MongoDB) is a simple and acceptable baseline
- no required queue or search-engine containers in the baseline setup
This operating model keeps deployment and maintenance simple while preserving clean boundaries for future scale.
## Documentation Map
- Architecture and technical design: [architecture.md](architecture.md)
- Runtime and deployment requirements: [requirements.md](requirements.md)
- Error handling policy and operational guidance: [error_handling.md](error_handling.md)
- Domain context and transcription policy: [intent.md](intent.md)
- Transcription Methodology: [transcription_methodology.md](transcription_methodology.md)
- Data model: [schema.md](schema.md)
## Glossary
- Document-oriented persistence: Storing data as flexible records instead of fixed relational rows.
- Prompt artifact: A single Markdown file that defines one transcription prompt and is edited independently.
- System of record: The authoritative persistent store for canonical data.
docs/archive/requirements.pre-v1-alignment.md
+85
View File
@@ -0,0 +1,85 @@
## Document Transcription System Requirements
This page captures a SysML v1.6-style requirements baseline for the production system described in [index.md](index.md). The model is represented as concise tables and traceability lists that preserve SysML-style IDs and relationship semantics.
## Scope
- System of interest: the single Python application service (NiceGUI + FastAPI) with PostgreSQL as the relational system of record and optional MongoDB for document-oriented persistence.
- Operational context: local-first execution with Docker Compose and an intentionally lightweight production trajectory.
- Primary concern: end-to-end transcription job lifecycle from upload through completion or failure.
## Requirements Model (Concise Text Form)
### Requirements
| ID | Category | Requirement | Risk | Verify Method |
| --- | --- | --- | --- | --- |
| REQ-0 | System | Provide end-to-end document transcription with persistent, inspectable lifecycle state. | medium | demonstration |
| REQ-1 | Functional | Allow users to upload one or more images or PDFs as sources from the web UI. | low | test |
| REQ-2 | Functional | Run each upload through asynchronous processing that returns an original transcription or explicit failure. | high | test |
| REQ-3 | Functional | Persist and expose job states: queued, processing, transcribed, failed. | high | inspection |
| REQ-4 | Functional | Persist transcription output, processing history, and failure details. | medium | test |
| REQ-5 | Interface | Expose API and UI views for status inspection and completed transcription reading. | medium | demonstration |
| REQ-6 | Performance | Trigger background processing on upload to preserve UI responsiveness. | medium | analysis |
| REQ-7 | Design Constraint | Keep lifespan-owned runtime resources: SQLAlchemy engine, async session factory, worker resources, provider clients. | medium | inspection |
| REQ-8 | Design Constraint | Initialize configuration and logging once at startup through centralized mechanisms. | low | inspection |
| REQ-9 | Design Constraint | Use Docker Compose baseline of app plus PostgreSQL; allow optional MongoDB container when enabled. | medium | demonstration |
| REQ-10 | Design Constraint | Keep schema bootstrap explicit and opt-in; normal startup does not mutate production schema. | high | inspection |
| REQ-11 | Design Constraint | Use service-backed persistence for core document and job data. | medium | inspection |
| REQ-12 | Design Constraint | Store transcription prompts as individual Markdown artifacts for iterative refinement. | medium | inspection |
| REQ-13 | Functional | Allow users to create one optional revision of transcription text derived from the original job transcription. | low | test |
### Requirement Relationships
- Contains: REQ-0 contains REQ-1 through REQ-13.
- Derives: REQ-2 -> REQ-3, REQ-3 -> REQ-4.
- Traces: REQ-5 -> REQ-3.
- Refines: REQ-6 -> REQ-2.
### Architecture Elements
| Element | Type | Doc Reference |
| --- | --- | --- |
| UI | NiceGUI pages | src/transcription/ui/pages |
| API | FastAPI routes | src/transcription/api/routes.py |
| GRAPH | Async processing workflow | src/transcription/services, src/transcription/ai |
| DBREL | PostgreSQL + SQLModel relational persistence | src/transcription/db |
| DBDOC | MongoDB document persistence | src/transcription/db, src/transcription/services |
| OPS | Docker Compose runtime | docker-compose.yml |
| PROMPTS | Transcription prompt artifact library (Markdown files) | .github/prompts, docs |
| TESTS | Pytest verification suite | tests |
### Satisfaction Mapping
- UI satisfies REQ-1, REQ-5, REQ-13.
- API satisfies REQ-5.
- GRAPH satisfies REQ-2, REQ-6.
- DBREL satisfies REQ-3, REQ-10, REQ-13.
- DBDOC satisfies REQ-4, REQ-11.
- OPS satisfies REQ-9.
- PROMPTS satisfies REQ-12.
### Verification Mapping
- TESTS verifies REQ-1, REQ-2, REQ-3, REQ-4, REQ-5, REQ-10, REQ-11, REQ-12, REQ-13.
## Requirement Notes
- Requirement IDs (`REQ-*`) are stable references for planning, implementation, and test traceability.
- The model uses compact tables and traceability lists for renderer compatibility while preserving SysML-style requirement IDs and relationship semantics.
- Requirement categories (functional, interface, performance, and design constraints) are preserved as explicit REQ entries and relationship labels to keep change impact visible.
- PostgreSQL containerization and optional MongoDB containerization are both treated as extremely lightweight and simple operational choices in this architecture.
## Verification Intent
- Demonstration: validate end-to-end behavior via running system flows and operator-visible outcomes.
- Inspection: verify architecture and startup/runtime policies in code and configuration.
- Analysis: evaluate asynchronous execution behavior and design sufficiency.
- Test: automate behavioral checks through pytest suites and service-level tests.
## Glossary
- Document-oriented persistence: A storage approach that uses flexible document structures for variable data shapes.
- Prompt artifact: A single Markdown file that defines one transcription prompt and is revised independently.
- SysML: Systems Modeling Language used to express structured requirements and traceability.
- System of record: The authoritative persistent store for canonical business data.