bbchops/transcription
1
0
Fork 0
generated from john/python-template
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update V1 & V2 core documents and reorganize docs folder

Browse Source
This commit is contained in:
Jim Lancaster
2026-07-31 10:04:07 -05:00
parent d2b793ea69
commit 3eefc36239
25 changed files with 671 additions and 783 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/ver1/architecture_v1.md
+308
View File
@@ -0,0 +1,308 @@
# System Architecture (Version 1)
This document describes the production architecture of the personal historical-document transcription system. The system is intentionally optimized for single-user operation, low operational overhead, and clean internal boundaries that support future growth without rewrites.
## Architecture Objectives
The production architecture is designed to:
- preserve verbatim family-history source material as searchable text
- keep operational complexity low for a personal deployment
- support asynchronous transcription without requiring distributed infrastructure
- maintain clear module boundaries so extensions can be added incrementally
## Production Scope And Scale
The deployed system targets personal use and a corpus of several thousand documents processed over time. The architecture favors simple, composable building blocks over distributed orchestration.
Current scope includes:
- content source upload and metadata capture
- asynchronous transcription jobs
- prompt-library driven transcription behavior, with one Markdown file per prompt
- original transcription review and optional revision review
- full-text search over accepted transcripts
- export of transcript data
## Deployment Topology
The production deployment uses [Docker Compose](https://docs.docker.com/compose/) and treats containerized databases as extremely lightweight operational dependencies.
Running [PostgreSQL](https://www.postgresql.org/docs/) in its own container is considered simple by default for this system.
Running [MongoDB](https://www.mongodb.com/docs/) in its own container is also considered simple when document-centric storage is enabled.
Container count is not a hard architectural limit; a three-container deployment (app, PostgreSQL, MongoDB) is an acceptable baseline.
### Baseline Topology (Two Containers)
- one application container
- one PostgreSQL container
- embedded background worker execution inside the app process
### Expanded Topology (Three Containers)
- application container
- PostgreSQL container
- MongoDB container
No additional queue, scheduler, or search-engine containers are required in the baseline production setup.
## Runtime Architecture
```mermaid
flowchart LR
User[Browser User] --> App[FastAPI + NiceGUI Service]
App --> Worker[In-process Background Worker]
App --> PG[(PostgreSQL)]
App --> MG[(MongoDB Document Store)]
Worker --> AI[Transcription Provider]
Worker --> PG
Worker --> MG
```
## Runtime Ownership And Startup Policy
The current implementation now uses explicit lifespan-owned runtime resources.
- application lifespan initializes and disposes database runtime resources
- worker lifecycle is owned by application lifespan startup/shutdown
- worker receives lifespan-owned database engine dependency explicitly
- schema bootstrap policy is environment-aware and explicit:
- development/test default to bootstrap enabled
- production defaults to bootstrap disabled
- explicit override is available via configuration
This aligns implementation toward REQ-7 and REQ-10 while preserving personal-scale operational simplicity.
## Layered Module Structure
### Interface Layer
Responsibility:
- HTTP API and UI routes
- request/response validation
- status and result presentation
Out of scope:
- business-rule enforcement
- data-access implementation
### Application Layer
Responsibility:
- upload and job orchestration
- state transitions and retry policy
- coordination across domain and infrastructure ports
Out of scope:
- provider-specific protocol details
- ORM or storage-specific logic
### Domain Layer
Responsibility:
- verbatim transcription policy
- revision and provenance invariants
- confidence and annotation semantics
Out of scope:
- web framework concerns
- database and network I/O
### Infrastructure Layer
Responsibility:
- persistence adapters (PostgreSQL and MongoDB)
- transcription-provider adapter
Out of scope:
- business policy decisions
## Processing Workflow
Production transcription flow:
1. A user uploads one or more content sources through the UI or API.
2. The application validates payloads and creates document, source, and job records.
3. The in-process worker de-queues the job and calls the transcription provider.
4. The application persists original transcription output on the job, plus confidence metadata and provenance events.
5. Job status transitions from queued to processing to transcribed or failed.
6. The UI and API expose status, optional revision to original transcription, and searchable transcription text.
## Data Model Ownership
System-of-record entities:
- documents and content sources
- transcription jobs, original transcription, and status events
- transcript revisions
- provenance metadata
### Original Transcription And Revision Ownership
- each processing job stores the original immutable provider output (`text`)
- provider metadata (`provider`, `model`, `prompt_name`) and failure detail (`error_detail`) are job-owned processing artifacts
- revisions are optional user-authored edits linked to a content source
- a revision can be created from original `job.text`
- many jobs will have zero revisions; revisions are additive and never overwrite original provider output
- a document groups one or more content sources (images, PDFs, and future source types)
Storage strategy:
- PostgreSQL for relational system-of-record entities
- MongoDB for document-oriented payloads and large transcription artifacts
- versioned prompt artifacts stored as individual Markdown files for human editing and refinement
- in-memory execution state treated as ephemeral
## Transcription Prompt Asset Policy
The production system treats transcription prompts as maintainable content assets.
- each transcription prompt is stored in its own Markdown file
- prompt files are designed for direct human editing and iterative refinement
- prompt updates are independent and do not require bundling unrelated prompt changes
- prompt file identity and revision history are tracked through normal repository version control
## Simplicity Guardrails
The production system enforces these constraints to prevent accidental over-engineering:
- PostgreSQL in a container is treated as a lightweight default dependency
- MongoDB in a container is treated as a lightweight optional dependency
- three containers (app, PostgreSQL, MongoDB) is an acceptable simple deployment
- no dedicated queue or search cluster is introduced without measured need
- external infrastructure is added only behind existing ports/adapters
## Extension Path
The architecture supports additive growth without changing domain contracts.
### Stage 1: Foundation (Current)
- upload, transcription, review, search, export
- in-process worker execution
- single provider adapter
- app plus PostgreSQL deployment
### Stage 2: Throughput Hardening
- optional MongoDB document-store enablement
- optional external worker/queue process
- stronger retry and dead-letter handling
### Stage 3: Intelligence Features
- entity extraction and cross-document linking
- timeline and narrative assembly
- optional multi-provider routing
Each stage preserves existing module boundaries and keeps migration risk low.
## Test Strategy
The test strategy is aligned to personal-scale operation with fast, deterministic feedback.
### Unit Tests
- domain transcription rules and annotation behavior
- revision-history invariants
- job state-transition logic
### Integration Tests
- repository behavior and transaction boundaries
- persistence-adapter and provider adapter contract mapping
- upload-to-persistence roundtrip
### End-to-End Tests
- happy path: upload, transcribe, review, search, export
- failure path: provider error, retry, surfaced failed status
### CI Execution Model
- fast suite on each push
- optional slower provider-sandbox checks on scheduled runs
## Risks And Controls
### Runtime Responsiveness
Risk:
- long jobs can reduce responsiveness in a single-process deployment
Control:
- bounded concurrency and visible job status in the UI
### Database Concurrency Limits
Risk:
- contention can appear under sustained concurrent writes in personal-scale infrastructure
Control:
- tuned connection pooling and phased use of MongoDB for document-heavy workloads
### Provider Output Variance
Risk:
- transcription quality varies by content source type, handwriting legibility, and source quality
Control:
- first-class human review and immutable revision history
---
## Technology References
- [FastAPI documentation](https://fastapi.tiangolo.com/)
- [NiceGUI documentation](https://nicegui.io/documentation)
- [Docker Compose documentation](https://docs.docker.com/compose/)
- [PostgreSQL documentation](https://www.postgresql.org/docs/)
- [MongoDB documentation](https://www.mongodb.com/docs/)
## Related Local References
- [System Overview](index_v1.md)
- [System Design Intent](intent.md)
- [Transcription Methodology](transcription_methodology.md)
- System Architecture (this document)
- [System Requirements](requirements_v1.md)
- [Data model](schema_v1.md)
- [Error Handling Policy](error_handling_v1.md)
- [Implementation Plan](implementation_plan_v1.md)
## Glossary
- Adapter: A component that translates between internal interfaces and external systems such as databases or AI services.
- Background job: Work executed outside the request/response path so the UI remains responsive.
- Boundary: A strict separation between modules with different responsibilities.
- CI (Continuous Integration): Automated test execution for code changes.
- Contract test: A test that verifies an adapter follows expected input/output behavior at a boundary.
- Domain layer: The module that contains core business rules and invariants.
- End-to-end test: A test that validates a full user flow across the running system.
- Full-text search: Text indexing and querying optimized for natural-language search.
- In-process worker: A background executor that runs within the same application process.
- Integration test: A test that verifies interactions between real modules and infrastructure components.
- MongoDB: A document-oriented database used for flexible, high-variance data structures.
- Modular monolith: A single deployable application with strongly separated internal modules.
- Port/Interface: A stable contract used by application/domain code to call infrastructure implementations.
- Prompt artifact: A single Markdown file that defines one transcription prompt and can be revised independently.
- Provenance: Metadata that records where generated data came from and how it was produced.
- Revision history: Optional versioned record of user-authored transcription edits over time.
- System of record: The authoritative persistent store for canonical data.
- Vertical slice: A minimal end-to-end feature path spanning UI/API, application logic, and persistence.
docs/ver1/error_handling_v1.md
+288
View File
@@ -0,0 +1,288 @@
# Error Handling Policy
This document defines the canonical error-handling policy for the document transcription system. It is the single source of truth for how errors are classified, surfaced to users, logged for diagnosis, and handled across UI, API, service, worker, and provider boundaries.
## Error Handling Objectives
The production error-handling model is designed to:
- make failures visible to the user in clear, actionable language
- preserve enough diagnostic detail for fast troubleshooting
- keep module behavior consistent across all boundaries
- distinguish expected domain failures from unexpected defects
- support safe retries for transient failures without hiding persistent faults
## Scope And Authority
This page governs error-handling behavior for:
- UI interactions (NiceGUI pages)
- API endpoints (FastAPI routes)
- application services and orchestration logic
- in-process background worker execution
- external provider adapters and persistence adapters
If implementation behavior conflicts with this document, this document is authoritative and implementation should be updated.
## Core Principles
- **Clarity first:** user-facing messages should explain what failed in plain language.
- **Actionability required:** each surfaced error should include a suggested next step.
- **Safety by default:** internal details are logged; sensitive details are not exposed by default in UI/API.
- **Consistency across boundaries:** category and structure should remain stable from source to surface.
- **Fail explicitly:** silent failure is prohibited.
- **Traceability:** every non-trivial error should be traceable with an error reference ID.
## Error Taxonomy
The system uses stable, implementation-independent categories:
| Category | Definition | Typical Source | Retriable |
| --- | --- | --- | --- |
| `validation_error` | Payload or parameter shape/content is invalid | UI/API input validation, service guards | no |
| `user_input_error` | User-provided artifact is unacceptable though structurally valid | unsupported file type, empty file, oversized upload | sometimes |
| `not_found_error` | Requested resource does not exist | missing job/document/source/revision | no |
| `conflict_error` | Requested operation violates current state constraints | invalid state transition | no |
| `external_provider_error` | External AI/provider call fails | upstream HTTP/API/provider failures | sometimes |
| `infrastructure_transient_error` | Temporary environment issue | network timeout, DB connection reset | yes |
| `infrastructure_persistent_error` | Non-transient environment issue | missing permissions, misconfiguration | no |
| `internal_unexpected_error` | Unhandled defect or unknown failure | uncaught exceptions, logic errors | unknown (default no) |
### Classification Rules
- Classification occurs as close as possible to the origin boundary.
- Provider-specific exceptions must be normalized into taxonomy categories before crossing service boundaries.
- Unknown exceptions are classified as `internal_unexpected_error` and logged with traceback.
- Category names are stable contracts and must not be changed casually.
## User-Facing Error Experience Contract
When an error is shown in the GUI, it must include:
1. **Title** (short context, e.g., “Upload failed”)
2. **Message** (plain-language explanation)
3. **Suggested action** (explicit next step)
4. **Error reference ID** (for support/debug traceability)
5. **Technical details** (optional/collapsible for advanced users)
### UI Message Rules
- Do not expose raw stack traces by default.
- Do not expose secrets, credentials, connection strings, or filesystem internals unless explicitly in debug tooling.
- Prefer domain language over implementation language.
- Use persistent visibility for important failures (dialog/card), not only transient toasts.
### Suggested Action Requirements
Every user-visible error must include a suggested course of action, such as:
- retry the operation
- check file type/size constraints
- refresh the jobs page
- verify environment configuration
- contact operator with error ID and timestamp
## API Error Response Contract
API errors should return a structured envelope with stable fields:
- `error_id`: short unique reference ID
- `category`: taxonomy category
- `message`: safe human-readable summary
- `suggestion`: recommended next step
- `details`: optional, only when safe and appropriate
- `timestamp`: UTC ISO-8601
HTTP status mapping guidance:
- `validation_error`, `user_input_error` -> `400`
- `not_found_error` -> `404`
- `conflict_error` -> `409`
- `external_provider_error` -> `502` or `503` (depending on failure mode)
- `infrastructure_transient_error` -> `503`
- `infrastructure_persistent_error` -> `500`
- `internal_unexpected_error` -> `500`
## Logging And Observability Contract
All logged errors must include, where available:
- `error_id`
- `category`
- `operation` (e.g., `upload.submit`, `worker.process_job`, `jobs.refresh`)
- `exception_type`
- `job_id`, `document_id`, `source_id` (when relevant)
- UTC timestamp
Rules:
- Use structured logging fields where practical.
- Use full traceback for unexpected errors (`internal_unexpected_error`).
- Log at boundary handoff points to preserve causal trail.
- Avoid duplicate noisy logging for the same exception at every layer.
## Recovery And Retry Policy
### Retriable Conditions
Retriable failures include:
- transient network/provider timeouts
- intermittent provider unavailability
- temporary DB/network interruptions
### Non-Retriable Conditions
Non-retriable failures include:
- invalid file formats
- missing required data
- permission/configuration failures
- deterministic domain conflicts
### Worker Behavior
- The worker must classify and persist failure details consistently.
- Retries should be bounded by configured limits.
- Exhausted retries must end in explicit failed status with recorded reason.
- No infinite retry loops are allowed.
## Boundary-Specific Responsibilities
### UI Layer
Responsibility:
- display user-safe error summaries and suggested actions
- show persistent error visibility for critical failures
- include error reference IDs in visible output
Out of scope:
- low-level exception parsing
- provider-specific protocol interpretation
### API Layer
Responsibility:
- map application exceptions into stable error envelopes and HTTP statuses
- preserve category and error_id continuity
Out of scope:
- domain-specific remediation logic
### Service Layer
Responsibility:
- classify domain and infrastructure exceptions
- convert adapter-specific failures into taxonomy categories
- return deterministic error types to callers
Out of scope:
- presentation formatting for UI
### Worker Layer
Responsibility:
- execute retry policy for retriable failures
- persist terminal failure details for jobs
- emit operational logs with category and identifiers
Out of scope:
- direct UI messaging
### Provider Adapter Layer
Responsibility:
- normalize provider SDK/HTTP failures into domain-neutral exceptions
- preserve raw provider context for logs (safely)
Out of scope:
- choosing user-facing wording
## Error Lifecycle Workflow
Standard lifecycle:
1. Failure occurs at a boundary or operation.
2. Exception is classified into taxonomy category.
3. `error_id` is created (or propagated).
4. Error is logged with required structured fields.
5. User/API receives safe message + suggested action.
6. Persistent job/resource state is updated when applicable.
7. Tests verify contract behavior for the pathway.
## Test Strategy For Error Handling
### Unit Tests
- category classification behavior
- retry eligibility decisions
- exception-to-message mapping safety
### Integration Tests
- UI pathways show clear message + suggested action for known failures
- API returns structured error envelope with expected status/category
- worker persists failed status and failure detail as required
### Regression Tests
- each previously observed production issue should have a guarding test
- contract tests must cover adapter error normalization behavior
## Known Failure Patterns And Prescribed Responses
| Pattern | Category | User Message | Suggested Action |
| --- | --- | --- | --- |
| Upload payload cannot be parsed by UI handler | `internal_unexpected_error` (until narrowed) | Upload failed due to unexpected processing error | Retry once; if repeated, report error ID and check runtime version compatibility |
| Unsupported extension | `user_input_error` | File type is not supported | Upload JPG, PNG, TIFF, or PDF |
| Empty file upload | `validation_error` | Uploaded file is empty | Choose a valid non-empty file and retry |
| Provider timeout | `external_provider_error` or `infrastructure_transient_error` | Transcription provider timed out | Retry from jobs page; if repeated, check provider status |
| Job lookup missing | `not_found_error` | Requested job was not found | Refresh jobs list and open a valid job |
## Governance And Update Process
This document is a living policy artifact.
Update this document when:
- new error categories are introduced
- handling behavior changes at any boundary
- a production incident reveals missing guidance
- API/UI error contracts change
Change requirements:
- update this document and associated tests in the same change set
- preserve taxonomy stability; if changed, document migration impact
- record noteworthy policy changes in project release notes or changelog
---
## Related Local References
- [System Overview](index_v1.md)
- [System Design Intent](intent.md)
- [Transcription Methodology](transcription_methodology.md)
- [System Architecture](architecture_v1.md)
- [System Requirements](requirements_v1.md)
- [Data model](schema_v1.md)
- Error Handling Policy (this document)
- [Implementation Plan](implementation_plan_v1.md)
## Glossary
- Error category: Stable classification used to drive handling, messaging, and status mapping.
- Error envelope: Structured API payload describing a failure.
- Error reference ID: Short identifier used to correlate user-visible failure with logs.
- Retriable error: Failure likely to succeed on a later attempt without code changes.
- Terminal failure: Failure state after retries are exhausted or retry is not allowed.
docs/ver1/ver1.md → docs/ver1/implementation_plan_v1.md
+18 -5
View File
@@ -44,7 +44,7 @@ V1 is complete when all of the following are true:
4. Freeze V1 status lifecycle to current implementation (`queued`, `processing`, `transcribed`, `failed`).
### Deliverables
- Updated `docs/schema.md` and `docs/requirements.md` traceability alignment.
- Updated `schema_v1.md` and `requirements.md` traceability alignment.
- Explicit V1 data invariants section in architecture docs.
### Exit Criteria
@@ -150,9 +150,8 @@ V1 is complete when all of the following are true:
### Deliverables
- V1 release checklist and acceptance evidence.
- `docs/runbook.md` for incident response and operator workflows.
- `docs/migration_v1.md` for V1 migration/backfill/rollback guidance.
- `docs/release_checklist_v1.md` for release sign-off.
- `runbook_v1.md` for incident response and operator workflows.
- `release_checklist_v1.md` for release sign-off.
### Exit Criteria
- Stakeholder sign-off and launch readiness achieved.
@@ -187,4 +186,18 @@ A lightweight traceability table should be maintained with:
- Only work required to satisfy V1 requirements enters this plan.
- Nice-to-have enhancements are captured in a separate backlog document.
- Schema or contract changes after Phase 1 require explicit approval and traceability impact review.
- Schema or contract changes after Phase 1 require explicit approval and traceability impact review.
---
## Related Local References
- [System Overview](index_v1.md)
- [System Design Intent](intent.md)
- [Transcription Methodology](transcription_methodology.md)
- [System Architecture](architecture_v1.md)
- [System Requirements](requirements_v1.md)
- [Data model](schema_v1.md)
- [Error Handling Policy](error_handling_v1.md)
- Implementation Plan (this document)
docs/ver1/index_v1.md
+58
View File
@@ -0,0 +1,58 @@
## Document Transcription System Overview
This project is a production application for transcribing and preserving historical family documents. It is intentionally designed for personal-scale use, with a simplicity-first architecture that is easy to operate and easy to extend.
## Start Here
Read [architecture_v1.md](architecture_v1.md) first.
The architecture page is the primary technical reference and defines:
- deployed topology and infrastructure limits
- module boundaries and dependency flow
- processing life cycle and data ownership
- test strategy, risk controls, and extension path
## What The Application Does
At a high level, users upload images or PDFs as content sources for handwritten, typed, or typeset documents, run asynchronous transcription jobs, review optional revisions, and search across accepted text.
### Core capabilities:
- document grouping with one or more content sources and metadata capture
- asynchronous transcription with visible job status
- immutable original transcription persisted with each job (plus provider/model/prompt metadata)
- transcription prompt management with one Markdown file per prompt for human refinement over time
- optional revisions for user-authored edits of original immutable transcription text
- full-text search over accepted transcripts
- export of transcript data
## Production Operating Model
The system runs with minimal operational overhead:
- PostgreSQL in a dedicated Docker container is considered extremely lightweight and simple for this system
- MongoDB in a dedicated Docker container is also considered extremely lightweight and simple for document-centric persistence
- a three-container deployment (app, PostgreSQL, MongoDB) is a simple and acceptable baseline
- no required queue or search-engine containers in the baseline setup
This operating model keeps deployment and maintenance simple while preserving clean boundaries for future scale.
---
## Documentation Map
- System Overview (this document)
- [System Design Intent](intent.md)
- [Transcription Methodology](transcription_methodology.md)
- [System Architecture](architecture_v1.md)
- [System Requirements](requirements_v1.md)
- [Data model](schema_v1.md)
- [Error Handling Policy](error_handling_v1.md)
- [Implementation Plan](implementation_plan_v1.md)
## Glossary
- Document-oriented persistence: Storing data as flexible records instead of fixed relational rows.
- Prompt artifact: A single Markdown file that defines one transcription prompt and is edited independently.
- System of record: The authoritative persistent store for canonical data.
docs/ver1/migration_v1.md
-92
View File
@@ -1,92 +0,0 @@
# V1 Data Migration and Recovery Guidance
This document defines migration/backfill and rollback guidance for the V1 SQLite baseline.
## Purpose
- provide safe procedures for local schema evolution and recovery
- reduce data-loss risk during version upgrades
- establish repeatable pre-change and post-change checks
## Current Baseline
- canonical relational store: SQLite
- default DB path: `./transcription.db`
- schema bootstrap may apply compatibility updates for dev/test scenarios
## Pre-Change Checklist
Before changing runtime version or schema behavior:
1. Stop the app process.
2. Create a timestamped DB backup copy.
3. Capture current app commit/version.
4. Export a quick status inventory:
- job counts by status
- total documents/sources/revisions
5. Ensure sufficient disk space.
## Backup Procedure (SQLite)
Minimum procedure:
1. Stop app.
2. Copy DB file to a safe location with timestamp.
3. Store backup path in release notes or change log.
## Upgrade Procedure (V1)
1. Perform pre-change checklist.
2. Deploy updated app version.
3. Start app and observe startup logs.
4. Verify schema bootstrap completes (if enabled).
5. Run smoke flow:
- upload valid file
- observe terminal status
- open job detail
## Backfill Guidance
V1 backfill is limited and conservative:
- for records missing newly introduced non-null defaults, use explicit one-time SQL updates only after backup
- avoid destructive rewrites of `Job.text` or `Revision.text`
- never backfill by overwriting original immutable transcription output
## Rollback Procedure
If upgrade fails or causes data inconsistency:
1. Stop app.
2. Restore prior DB backup file.
3. Revert app version to last known-good commit.
4. Restart app.
5. Run smoke flow and confirm stability.
## Recovery Scenarios
### Stale processing jobs after crash/restart
- restart app and allow stale-job recovery to re-queue timed-out `processing` jobs
- monitor for terminal progression
### Schema mismatch symptoms
- errors during startup or writes indicating missing columns/indexes
- rollback to last good DB + app version
- reattempt with documented upgrade path
## Validation Evidence
For each upgrade rehearsal, capture:
- backup filename/path
- pre and post job status counts
- smoke test result
- rollback rehearsal result (recommended)
## Operational Constraints
- treat DB backups as required before non-trivial upgrades
- do not perform in-place DB edits while app is running
- do not skip post-upgrade smoke validation
docs/ver1/release_checklist_v1.md
+7 -7
View File
@@ -18,8 +18,8 @@ Use this checklist before declaring V1 operationally complete.
## C) Operational Readiness
- [ ] `docs/runbook.md` reviewed and current.
- [ ] `docs/migration_v1.md` reviewed and current.
- [ ] `runbook_v1.md` reviewed and current.
- [ ] `migration_v1.md` reviewed and current.
- [ ] Backup and rollback procedures tested at least once.
- [ ] Incident escalation packet template is known to operators.
@@ -28,14 +28,14 @@ Use this checklist before declaring V1 operationally complete.
- [ ] Lint/type checks pass.
- [ ] `pytest -m "not external" -q` passes.
- [ ] Targeted external/provider checks executed (if credentials available).
- [ ] Release evidence recorded in `docs/release_evidence_v1.md`.
- [ ] Release evidence recorded in `release_evidence_v1.md`.
## E) Traceability and Documentation
- [ ] `docs/requirements.md` aligns with implemented V1 behavior.
- [ ] `docs/architecture.md`, `docs/schema.md`, and `docs/error_handling.md` are consistent.
- [ ] `docs/traceability_v1.md` is updated with current implementation and test evidence.
- [ ] `docs/ver1/ver1.md` phase status updated with evidence references.
- [ ] `requirements_v1.md` aligns with implemented V1 behavior.
- [ ] `architecture_v1.md`, `schema_v1.md`, and `error_handling_v1.md` are consistent.
- [ ] `traceability_v1.md` is updated with current implementation and test evidence.
- [ ] `implementation_plan_v1.md` phase status updated with evidence references.
- [ ] REQ traceability evidence links recorded (tests/runbook/checks).
## Release Sign-Off
docs/ver1/requirements_v1.md
+98
View File
@@ -0,0 +1,98 @@
## Document Transcription System Requirements
This page captures a SysML v1.6-style requirements baseline for the production system described in [index_v1.md](index_v1.md). The model is represented as concise tables and traceability lists that preserve SysML-style IDs and relationship semantics.
## Scope
- System of interest: the single Python application service (NiceGUI + FastAPI) with PostgreSQL as the relational system of record and optional MongoDB for document-oriented persistence.
- Operational context: local-first execution with Docker Compose and an intentionally lightweight production trajectory.
- Primary concern: end-to-end transcription job lifecycle from upload through completion or failure.
## Requirements Model (Concise Text Form)
### Requirements
| ID | Category | Requirement | Risk | Verify Method |
| --- | --- | --- | --- | --- |
| REQ-0 | System | Provide end-to-end document transcription with persistent, inspectable lifecycle state. | medium | demonstration |
| REQ-1 | Functional | Allow users to upload one or more images or PDFs as sources from the web UI. | low | test |
| REQ-2 | Functional | Run each upload through asynchronous processing that returns an original transcription or explicit failure. | high | test |
| REQ-3 | Functional | Persist and expose job states: queued, processing, transcribed, failed. | high | inspection |
| REQ-4 | Functional | Persist transcription output, processing history, and failure details. | medium | test |
| REQ-5 | Interface | Expose API and UI views for status inspection and completed transcription reading. | medium | demonstration |
| REQ-6 | Performance | Trigger background processing on upload to preserve UI responsiveness. | medium | analysis |
| REQ-7 | Design Constraint | Keep lifespan-owned runtime resources: SQLAlchemy engine, async session factory, worker resources, provider clients. | medium | inspection |
| REQ-8 | Design Constraint | Initialize configuration and logging once at startup through centralized mechanisms. | low | inspection |
| REQ-9 | Design Constraint | Use Docker Compose baseline of app plus PostgreSQL; allow optional MongoDB container when enabled. | medium | demonstration |
| REQ-10 | Design Constraint | Keep schema bootstrap explicit and opt-in; normal startup does not mutate production schema. | high | inspection |
| REQ-11 | Design Constraint | Use service-backed persistence for core document and job data. | medium | inspection |
| REQ-12 | Design Constraint | Store transcription prompts as individual Markdown artifacts for iterative refinement. | medium | inspection |
| REQ-13 | Functional | Allow users to create one optional revision of transcription text derived from the original job transcription. | low | test |
### Requirement Relationships
- Contains: REQ-0 contains REQ-1 through REQ-13.
- Derives: REQ-2 -> REQ-3, REQ-3 -> REQ-4.
- Traces: REQ-5 -> REQ-3.
- Refines: REQ-6 -> REQ-2.
### Architecture Elements
| Element | Type | Doc Reference |
| --- | --- | --- |
| UI | NiceGUI pages | src/transcription/ui/pages |
| API | FastAPI routes | src/transcription/api/routes.py |
| GRAPH | Async processing workflow | src/transcription/services, src/transcription/ai |
| DBREL | PostgreSQL + SQLModel relational persistence | src/transcription/db |
| DBDOC | MongoDB document persistence | src/transcription/db, src/transcription/services |
| OPS | Docker Compose runtime | docker-compose.yml |
| PROMPTS | Transcription prompt artifact library (Markdown files) | .github/prompts, docs |
| TESTS | Pytest verification suite | tests |
### Satisfaction Mapping
- UI satisfies REQ-1, REQ-5, REQ-13.
- API satisfies REQ-5.
- GRAPH satisfies REQ-2, REQ-6.
- DBREL satisfies REQ-3, REQ-10, REQ-13.
- DBDOC satisfies REQ-4, REQ-11.
- OPS satisfies REQ-9.
- PROMPTS satisfies REQ-12.
### Verification Mapping
- TESTS verifies REQ-1, REQ-2, REQ-3, REQ-4, REQ-5, REQ-10, REQ-11, REQ-12, REQ-13.
## Requirement Notes
- Requirement IDs (`REQ-*`) are stable references for planning, implementation, and test traceability.
- The model uses compact tables and traceability lists for renderer compatibility while preserving SysML-style requirement IDs and relationship semantics.
- Requirement categories (functional, interface, performance, and design constraints) are preserved as explicit REQ entries and relationship labels to keep change impact visible.
- PostgreSQL containerization and optional MongoDB containerization are both treated as extremely lightweight and simple operational choices in this architecture.
## Verification Intent
- Demonstration: validate end-to-end behavior via running system flows and operator-visible outcomes.
- Inspection: verify architecture and startup/runtime policies in code and configuration.
- Analysis: evaluate asynchronous execution behavior and design sufficiency.
- Test: automate behavioral checks through pytest suites and service-level tests.
---
## Related Local References
- [System Overview](index_v1.md)
- [System Design Intent](intent.md)
- [Transcription Methodology](transcription_methodology.md)
- [System Architecture](architecture_v1.md)
- System Requirements (this document)
- [Data model](schema_v1.md)
- [Error Handling Policy](error_handling_v1.md)
- [Implementation Plan](implementation_plan_v1.md)
## Glossary
- Document-oriented persistence: A storage approach that uses flexible document structures for variable data shapes.
- Prompt artifact: A single Markdown file that defines one transcription prompt and is revised independently.
- SysML: Systems Modeling Language used to express structured requirements and traceability.
- System of record: The authoritative persistent store for canonical business data.
docs/ver1/runbook_v1.md
+129
View File
@@ -0,0 +1,129 @@
# V1 Operations Runbook
This runbook provides day-2 operational procedures for the V1 baseline.
## Scope
Applies to:
- local/hosted V1 runtime
- SQLite-backed persistence
- in-process worker lifecycle
- OpenRouter provider integration
## Preconditions
- `.env` contains `OPENROUTER_API_KEY`
- app starts successfully
- `uploads/` and `prompts/` are writable
- health endpoint responds at `/healthz`
## Standard Startup Procedure
1. Start the app using the project-standard command.
2. Open `/healthz` and verify `{"status":"ok"}`.
3. Open `/ui/upload` and submit a small valid file.
4. Confirm job transitions from `queued` -> `processing` -> `transcribed` (or `failed` with detail).
## Standard Shutdown Procedure
1. Stop the application process.
2. Ensure no active process still holds the SQLite file.
3. If maintenance is planned, copy the DB file before edits:
- `transcription.db` (or configured `DATABASE_URL` file path)
## Incident: Jobs Stuck In `processing`
### Symptoms
- Jobs remain `processing` for longer than provider timeout
- New uploads queue but do not complete
- provider usage increases but no terminal job state is visible
### Checks
1. Confirm app process is still running.
2. Confirm worker loop is active (startup logs include worker lifespan start).
3. Inspect recent app logs for:
- `worker.process_job`
- `error_id`
- `category`
- `job_id` / `document_id` / `source_id`
4. Verify provider credentials and provider status.
### Recovery
1. Restart the app to trigger stale-processing recovery.
2. On startup, app re-queues stale processing jobs based on timeout policy.
3. Re-check jobs page and confirm terminal state progression.
4. If persistent, capture logs + error IDs and move to deep investigation.
## Incident: Provider Authentication Failures
### Symptoms
- failures categorized as provider/auth
- jobs fail quickly with authentication guidance
### Recovery
1. Validate `OPENROUTER_API_KEY` value.
2. Restart app after updating env.
3. Re-run a small transcription to confirm recovery.
## Incident: Upload Failures
### Symptoms
- UI reports upload errors
- unsupported extension or empty payload
### Recovery
1. Validate file extension (`.jpg`, `.jpeg`, `.png`, `.tif`, `.tiff`, `.pdf`).
2. Validate file is not empty.
3. Validate upload directory permissions.
4. Retry upload.
## Incident: Database File/Permission Issues
### Symptoms
- persistence errors during upload/job update
- startup failures around schema/runtime
### Recovery
1. Confirm the configured DB file path exists and is writable.
2. Confirm parent directory permissions.
3. Restore from last known backup copy if corruption is suspected.
4. Restart app and run smoke test.
## Logging Requirements (Operational)
Operational triage should always capture:
- `error_id`
- category
- operation name
- `job_id`, `document_id`, `source_id` when applicable
- UTC timestamp
## Escalation Packet (When opening an issue)
Include:
- exact timestamp window
- one failing `job_id`
- relevant `error_id` values
- latest 100 lines of app logs
- environment summary (`DATABASE_URL` type, app version/commit)
## Post-Incident Validation
After mitigation, verify:
1. Upload works.
2. One job reaches `transcribed`.
3. One induced failure reaches `failed` with error detail.
4. Jobs page and detail page render correctly.
docs/ver1/schema_v1.md
+98
View File
@@ -0,0 +1,98 @@
## Database Schema (V1 Baseline)
This document describes the current relational schema for the transcription system.
All primary and foreign keys in the domain models are UUID-based in V1.
---
## Schema Diagram
```mermaid
erDiagram
DOCUMENT {
UUID id PK
TEXT name
}
JOB {
UUID id PK
UUID document_id FK
TEXT status
INTEGER retry_count
DATETIME date_created
DATETIME date_updated
TEXT provider
TEXT model
TEXT prompt_name
TEXT text
TEXT error_detail
}
SOURCE {
UUID id PK
UUID document_id FK
UUID job_id FK
TEXT upload_name
TEXT filename
TEXT file_path
DATETIME date_uploaded
}
REVISION {
UUID id PK
UUID source_id "FK, UK"
INTEGER revision
TEXT text
DATETIME date_created
}
DOCUMENT ||--o{ SOURCE : has_many
DOCUMENT ||--o{ JOB : has_many
JOB ||--o{ SOURCE : referenced_by
SOURCE ||--o| REVISION : has_optional_one
```
---
## Table Relationships and Constraints
- A `Document` can have zero or more `Source` records.
- A `Document` can have zero or more `Job` records.
- A `Source` belongs to exactly one `Document` and one `Job`.
- A `Source` may have one optional `Revision`.
- Optional `0..1` revision cardinality is enforced by uniqueness on `revision.source_id`.
### Invariants
- `Job.text` stores immutable original provider transcription output.
- `Revision` rows are optional user-authored edits derived from original transcription.
- Revisions do not overwrite original `Job.text`.
- Job status lifecycle values are: `queued`, `processing`, `transcribed`, `failed`.
### Timestamp Fields
- `Job.date_created`
- `Job.date_updated`
- `Source.date_uploaded`
- `Revision.date_created`
---
## Related Local References
- [System Overview](index_v1.md)
- [System Design Intent](intent.md)
- [Transcription Methodology](transcription_methodology.md)
- [System Architecture](architecture_v1.md)
- [System Requirements](requirements_v1.md)
- Data model (this document)
- [Error Handling Policy](error_handling_v1.md)
- [Implementation Plan](implementation_plan_v1.md)
## Glossary
- **Document**: logical grouping for one or more transcribed sources.
- **Source**: uploaded file content (image/PDF) linked to a job.
- **Job**: processing record that stores lifecycle status and original output.
- **Revision**: optional single user-authored edited text linked to a source.
docs/ver1/traceability_v1.md
+4 -4
View File
@@ -21,7 +21,7 @@ Status values:
| REQ-6 | done | Background processing trigger/worker notifier and non-blocking workflow in `src/transcription/ui/components/upload.py`, `src/transcription/worker.py` | `tests/test_app.py`, `tests/services/test_workflows_reliability.py` |
| REQ-7 | done | Lifespan-owned runtime resources in `src/transcription/app.py`, `src/transcription/db/runtime.py` | `tests/test_app.py`, `tests/test_db.py` |
| REQ-8 | done | Centralized settings/logging initialization in `src/transcription/config.py`, `src/transcription/app.py` | `tests/test_config.py`, `tests/test_app.py` |
| REQ-9 | done | Containerized runtime baseline in `docker-compose.yml`, `Dockerfile` | `docs/release_checklist_v1.md` (Ops checklist), manual demonstration step |
| REQ-9 | done | Containerized runtime baseline in `docker-compose.yml`, `Dockerfile` | `release_checklist_v1.md` (Ops checklist), manual demonstration step |
| REQ-10 | done | Explicit schema bootstrap policy + runtime controls in `src/transcription/config.py`, `src/transcription/app.py`, `src/transcription/db/operations.py` | `tests/test_db.py`, `tests/test_config.py` |
| REQ-11 | done | Service/workflow persistence boundaries in `src/transcription/services/*.py`, `src/transcription/services/workflows.py` | `tests/services/test_job_service.py`, `tests/services/test_transcription_service.py` |
| REQ-12 | done | Prompt artifacts in `prompts/` and loading/validation in `src/transcription/services/transcription.py` | `tests/test_prompts.py` |
@@ -29,9 +29,9 @@ Status values:
## Operational Evidence (Step 3 Artifacts)
- Runbook: `docs/runbook.md`
- Migration/backfill/rollback guidance: `docs/migration_v1.md`
- Release readiness checklist: `docs/release_checklist_v1.md`
- Runbook: `runbook_v1.md`
- Migration/backfill/rollback guidance: `migration_v1.md`
- Release readiness checklist: `release_checklist_v1.md`
## Verification Cadence