bbchops/transcription
1
0
Fork 0
generated from john/python-template
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Creation of MVP.md

Browse Source
This commit is contained in:
Jim Lancaster
2026-06-23 18:56:29 -05:00
parent 187324d903
commit d0626c7653
3 changed files with 201 additions and 129 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/mvp.md
+193
View File
@@ -0,0 +1,193 @@
## MVP Definition: Historical Document Transcription System
### 1. MVP Objective
Deliver the thinnest possible end-to-end vertical slice — a user uploads an image of a handwritten document, the system transcribes it via an AI provider, and the user reads the resulting transcript — with just enough persistence and structure to validate the core value proposition: *can AI-driven transcription, guided by curated prompts, produce useful verbatim transcripts of historical family documents?*
The MVP deliberately defers full-text search, export, revision history, MongoDB, timeline assembly, and multi-provider routing. These are additive features that don't need validation before the core transcription loop is proven.
---
### 2. Core User Story
*As a family historian, I can upload a photo of a handwritten letter, wait for it to be transcribed, and read the verbatim transcript — so I can evaluate whether this system will work for my thousands of documents.*
---
### 3. In-Scope Requirements (from ```requirements.md```)
| Requirement | ID | MVP Rationale |
| --- | --- | --- |
| End-to-end transcription with lifecycle state | REQ-0 | This is the MVP. |
| Upload one or more images from the web UI | REQ-1 | Core entry point. MVP supports single-image upload (multi-image is a stretch goal). |
| Asynchronous processing → transcription or failure | REQ-2 | Validates the AI transcription pipeline. |
| Persist and expose job states (queued → processing → transcribed/failed) | REQ-3 | Minimum feedback loop for the user. |
| Persist transcription output and failure details | REQ-4 | User must be able to read the result. |
| UI views for status and transcript reading | REQ-5 | The user needs to see what happened. |
| Background processing to keep UI responsive | REQ-6 | Essential for usability during long AI calls. |
| Centralized config and logging at startup | REQ-8 | Small effort, high payoff for debugging. |
| Store transcription prompts as Markdown files | REQ-12 | Core to the Prompt Curation Policy in Intent.md. Start with a single prompt file. |
### Deferred to Post-MVP
| Requirement | ID | Why Deferred |
| --- | --- | --- |
| Lifespan-owned runtime resources (engine, session factory, etc.) | REQ-7 | Important for production robustness, but a simple global or module-level setup is adequate for MVP validation. |
| Docker Compose (app + PostgreSQL + optional MongoDB) | REQ-9 | MVP runs locally with SQLite to eliminate container overhead during rapid iteration. PostgreSQL migration is Stage 1 hardening. |
| Explicit, opt-in schema bootstrap | REQ-10 | MVP uses auto-create-tables at startup (SQLModel create_all). Production schema discipline comes after the model stabilizes. |
| Service-backed persistence for core data | REQ-11 | MVP uses a thin repository layer over SQLite. Full service abstraction follows once the domain model is proven. |
---
### 4. MVP Feature Set
#### Feature 1: Document Upload (UI)
* A single NiceGUI page with a file-upload widget (accepts .jpg, .png, .tiff, .pdf).
* On upload: save the file to a local uploads/ directory, create a Document record, create a Job record with status queued.
* Minimal metadata capture: original filename, upload timestamp.
#### Feature 2: Asynchronous Transcription Worker
* An in-process background worker (Python asyncio task or BackgroundTasks) that:
1. Picks up queued jobs.
2. Transitions status to processing.
3. Sends the image + the curated Markdown prompt to an AI vision model (OpenAI API — already a dependency).
4. On success: saves the transcript text, transitions to transcribed.
5. On failure: saves the error detail, transitions to failed.
#### Feature 3: Transcription Prompt (Markdown Asset)
* A single Markdown file (prompts/transcribe_handwriting.md) encoding the verbatim transcription rules from Intent.md (the Document Issues table, scholarly guidelines, etc.).
* The worker reads this file at invocation time and injects it as the system/user prompt.
#### Feature 4: Job Status & Transcript Viewer (UI)
* A job list page showing all jobs with their current status (queued / processing / transcribed / failed).
* A transcript detail page showing:
* The original uploaded image (rendered inline).
* The transcription text (or the failure reason).
* Timestamp metadata.
#### Feature 5: Minimal Persistence (SQLite + SQLModel)
* Three tables/models:
* Document: id, filename, file_path, uploaded_at.
* Job: id, document_id (FK), status, created_at, updated_at.
* Transcript: id, job_id (FK), text, error_detail, created_at.
* SQLite database file stored locally. Auto-created on first startup.
#### Feature 6: Centralized Configuration
* A single config.py (or Pydantic BaseSettings) loading:
* OPENAI_API_KEY (required)
* OPENAI_MODEL (default: gpt-4o)
* DATABASE_URL (default: sqlite:///./transcription.db)
* UPLOAD_DIR (default: ./uploads)
* PROMPT_DIR (default: ./prompts)
---
### 5. MVP Architecture (Simplified)
```Apply
┌─────────────────────────────────────────────┐
│ NiceGUI Web UI │
│ ┌──────────────┐ ┌───────────────────┐ │
│ │ Upload Page │ │ Jobs / Transcript │ │
│ └──────┬───────┘ └───────┬───────────┘ │
│ │ │ │
│ ▼ ▼ │
│ ┌───────────────────────────┐ │
│ │ Application Service │ │
│ │ (upload, job lifecycle) │ │
│ └─────┬─────────────┬───────┘ │
│ │ │ │
│ ┌─────▼─────┐ ┌─────▼─────────────┐ │
│ │ SQLite DB │ │ Background Worker │ │
│ │ (SQLModel)│ │ → OpenAI Vision │ │
│ └───────────┘ └───────────────────┘ │
│ │ │
│ ┌─────▼──────┐ │
│ │ prompts/ │ │
│ │ *.md files │ │
│ └────────────┘ │
└─────────────────────────────────────────────┘
```
---
#### 6. Proposed File Structure
```Apply
project-root/
├── docs/ # (existing)
├── prompts/
│ └── transcribe_handwriting.md # curated transcription prompt
├── src/
│ └── handwriting/
│ ├── __init__.py
│ ├── app.py # FastAPI + NiceGUI app entrypoint
│ ├── config.py # Pydantic BaseSettings
│ ├── models.py # SQLModel: Document, Job, Transcript
│ ├── db.py # engine, session, create_all
│ ├── services/
│ │ ├── __init__.py
│ │ ├── upload.py # save file + create records
│ │ └── transcription.py # call AI provider, update job
│ ├── worker.py # background job loop
│ └── ui/
│ ├── __init__.py
│ ├── upload_page.py # NiceGUI upload page
│ └── jobs_page.py # NiceGUI job list + detail
├── tests/
│ ├── test_models.py
│ ├── test_upload.py
│ └── test_transcription.py
├── pyproject.toml
└── README.md
```
---
#### 7. MVP Validation Criteria
The MVP is considered validated when:
1. ✅ A user can upload an image of a handwritten document through the browser.
2. ✅ The system asynchronously sends the image to OpenAI's vision model with the curated prompt.
3. ✅ The transcript (or failure reason) is persisted and visible in the UI.
4. ✅ The transcription follows verbatim scholarly rules defined in Intent.md (spot-checked by the user on real family documents).
5. ✅ The transcription prompt is stored as a standalone Markdown file and can be edited without code changes.
6. ✅ Job status transitions are visible: queued → processing → transcribed/failed.
---
### 8. Key Feedback Questions the MVP Should Answer
These are the real unknowns this MVP exists to resolve:
| # | Question | How We Learn |
| --- | --- | --- |
| 1 | Is AI transcription quality good enough for this handwriting corpus? | User reviews 20–50 real transcriptions against originals. |
| 2 | Does the verbatim prompt produce scholarly-quality output, or does it need major rework? | Compare output to the Document Issues table rules in Intent.md. |
| 3 | What document types are hardest (old cursive, faded ink, pencil, postcards)? | Track which uploads produce failed or low-quality results. |
| 4 | Is single-image upload sufficient, or is batch upload needed early? | User friction during real scanning sessions. |
| 5 | What metadata is missing that the user wishes they could capture at upload time? | User feedback after processing real batches. |
---
#### 9. What Comes After MVP (Immediate Post-MVP)
Once the core transcription loop is validated, the next priorities (aligned to Architecture Stage 1) are:
1. **Multi-image upload** — process a batch from a scanning session.
2. **PostgreSQL migration** — swap SQLite for containerized PostgreSQL (REQ-9, REQ-10).
3. **Revision history** — allow the user to edit/correct transcripts with immutable version tracking.
4. **Full-text search** — search across all accepted transcripts.
5. **Repository/service layer formalization** — proper ports/adapters as the domain model stabilizes.
6. **Docker Compose deployment** — containerize the app for reproducible operation.
---
#### 10. Implementation Approach
Recommended build order for the MVP (each step produces a testable increment):
| Step | Deliverable | Validates |
| --- | --- | --- |
| 1 | config.py + models.py + db.py — data layer with SQLite | Schema and config foundation |
| 2 | prompts/transcribe_handwriting.md — curated prompt from Intent.md | Prompt asset pattern |
| 3 | services/transcription.py — call OpenAI vision API with prompt + image | Core AI integration |
| 4 | services/upload.py + worker.py — upload handling + background job loop | End-to-end pipeline (CLI-testable) |
| 5 | ui/upload_page.py + ui/jobs_page.py — NiceGUI pages | User-facing interface |
| 6 | tests/ — unit + integration tests Automated verification |
This MVP is deliberately narrow: **one prompt, one provider, one user, one image at a time, SQLite, no containers**. Every omission is intentional — the goal is to get real family documents through the transcription pipeline as fast as possible and let the quality of the output guide every subsequent decision.
docs/requirements.md
+8 -8
View File
@@ -1,4 +1,4 @@
## Handwriting Transcription System Requirements
## Document Transcription System Requirements
This page captures a SysML v1.6-style requirements baseline for the production system described in [index.md](index.md). The model is represented as concise tables and traceability lists that preserve SysML-style IDs and relationship semantics.
@@ -14,8 +14,8 @@ This page captures a SysML v1.6-style requirements baseline for the production s
| ID | Category | Requirement | Risk | Verify Method |
| --- | --- | --- | --- | --- |
| REQ-0 | System | Provide end-to-end handwriting transcription with persistent, inspectable lifecycle state. | medium | demonstration |
| REQ-1 | Functional | Allow users to upload one or more handwriting images from the web UI. | low | test |
| REQ-0 | System | Provide end-to-end document transcription with persistent, inspectable lifecycle state. | medium | demonstration |
| REQ-1 | Functional | Allow users to upload one or more document images from the web UI. | low | test |
| REQ-2 | Functional | Run each upload through asynchronous processing that returns a transcription or explicit failure. | high | test |
| REQ-3 | Functional | Persist and expose job states: upload, queued, processing, transcribed, failed, completed. | high | inspection |
| REQ-4 | Functional | Persist transcription output, processing history, and failure details. | medium | test |
@@ -39,11 +39,11 @@ This page captures a SysML v1.6-style requirements baseline for the production s
| Element | Type | Doc Reference |
| --- | --- | --- |
| UI | NiceGUI pages | src/handwriting/ui/pages |
| API | FastAPI routes | src/handwriting/api/routes.py |
| GRAPH | Async processing workflow | src/handwriting/services, src/handwriting/ai |
| DBREL | PostgreSQL + SQLModel relational persistence | src/handwriting/db |
| DBDOC | MongoDB document persistence | src/handwriting/db, src/handwriting/services |
| UI | NiceGUI pages | src/transcription/ui/pages |
| API | FastAPI routes | src/transcription/api/routes.py |
| GRAPH | Async processing workflow | src/transcription/services, src/transcription/ai |
| DBREL | PostgreSQL + SQLModel relational persistence | src/transcription/db |
| DBDOC | MongoDB document persistence | src/transcription/db, src/transcription/services |
| OPS | Docker Compose runtime | docker-compose.yml |
| PROMPTS | Transcription prompt artifact library (Markdown files) | .github/prompts, docs |
| TESTS | Pytest verification suite | tests |
docs/tests.md
-121
View File
@@ -1,121 +0,0 @@
# Testing
The greenfield test structure mirrors the source tree at a high level and keeps the first pass shallow and easy to extend.
## Source To Test Map
- `src/handwriting/bootstrap.py`, `src/handwriting/config.py`, `src/handwriting/logging_buffer.py`, `src/handwriting/main.py` -> `tests/handwriting/test_core.py`
- `src/handwriting/ai/**` -> `tests/handwriting/ai/`
- `src/handwriting/api/**` -> `tests/handwriting/api/`
- `src/handwriting/db/**` -> `tests/handwriting/db/`
- `src/handwriting/services/**` -> `tests/handwriting/services/`
- `src/handwriting/ui/components/**` and `src/handwriting/ui/pages/**` -> `tests/handwriting/ui/`
- Shared test helpers and fixtures -> `tests/conftest.py` plus subtree `conftest.py` files where needed
## Major Sections
- `tests/handwriting/test_core.py`: bootstrap, config, logging, and entrypoint coverage.
- `tests/handwriting/ai/`: AI contracts, runtime orchestration, graph wiring, and node behavior.
- `tests/handwriting/api/`: HTTP routes and request/response contract checks.
- `tests/handwriting/db/`: session setup, models, and repositories.
- `tests/handwriting/services/`: service-layer orchestration and domain logic.
- `tests/handwriting/ui/`: NiceGUI components and pages.
- `tests/conftest.py`: shared lightweight fixtures and deterministic defaults.
- `tests/handwriting/**/conftest.py`: subtree-specific fixtures only where a package needs its own setup.
## Implemented In This Pass (Core + DB)
### Created Files
- `tests/handwriting/test_core.py`
- `tests/handwriting/conftest.py`
- `tests/handwriting/db/conftest.py`
- `tests/handwriting/db/test_session.py`
- `tests/handwriting/db/test_models.py`
- `tests/handwriting/db/test_job_repo.py`
- `tests/handwriting/db/test_service_job.py`
### Updated Files
- `tests/conftest.py`: added shared `settings_factory` fixture.
- `pyproject.toml`: registered strict markers and async pytest mode.
### Marker Taxonomy
- `unit`: fast deterministic unit tests.
- `db`: database-backed tests against PostgreSQL-backed fixtures.
- `document`: document-store tests for MongoDB-backed persistence behavior.
- `integration`: cross-layer integration tests (reserved for expanded next pass).
- `smoke`: high-value end-to-end surface checks.
- `slow`: long-running tests.
- `external`: tests that require external services/credentials.
### Fixture Ownership
- `tests/conftest.py`: global lightweight fixtures used across all sections.
- `tests/handwriting/conftest.py`: core package-level environment fixtures.
- `tests/handwriting/db/conftest.py`:
- PostgreSQL fixtures (`postgres_engine`, `db_session`) for default data-layer tests.
- optional MongoDB fixture (`mongo_client`) gated by `PYTEST_MONGO_URL` for document-store integration checks.
### Initial Coverage Added
- `tests/handwriting/test_core.py`
- settings identity and URL validation behavior.
- logging UI buffer wiring.
- buffer incremental read behavior.
- minimal main module export smoke check.
- `tests/handwriting/db/test_session.py`
- session scope, factory, and engine helper behavior.
- `tests/handwriting/db/test_models.py`
- model defaults and enum/value shape checks.
- `tests/handwriting/db/test_job_repo.py`
- repository list/detail happy path and not-found/missing-image edges.
- `tests/handwriting/db/test_service_job.py`
- job creation happy path and validation errors for unsupported type / oversized payload.
## Run Commands
- Collect only: `uv run pytest --collect-only -q`
- Fast local path: `uv run pytest -m unit -q`
- DB path: `uv run pytest -m "db or integration" -q`
- Full path: `uv run pytest -q`
## Current Verification Snapshot
- `uv run pytest --collect-only -q` -> 19 tests collected.
- `uv run pytest -m unit -q` -> 11 passed, 8 deselected.
- `uv run pytest -m "db or integration" -q` -> 7 passed, 12 deselected.
## Jobs Page Diagnostics
To investigate cases where `/jobs` renders without a table, the scaffold now includes focused service and UI coverage:
- `tests/handwriting/services/test_ui_jobs.py`
- verifies the `list_jobs_for_ui` contract used by the page.
- includes the join edge case where a job with a missing image relation does not appear in list output.
- `tests/handwriting/ui/test_jobs_page_runtime.py`
- verifies runtime guard behavior for missing app state (`settings`, `session_factory`).
- `tests/handwriting/ui/test_jobs_page_rendering.py`
- includes a detector test for initial render behavior on `/jobs`.
- verifies empty, table, and error rendering branches after refresh.
- `tests/handwriting/ui/test_jobs_page_smoke.py`
- confirms route registration and refresh button wiring.
Targeted commands:
- `uv run pytest tests/handwriting/ui/test_jobs_page_rendering.py -q`
- `uv run pytest -m "unit or integration" -q`
- `uv run pytest -m smoke -k jobs -q`
- `HANDWRITING_E2E_BASE_URL=http://localhost uv run pytest tests/handwriting/ui/test_jobs_page_browser_smoke.py -q`
## Next Pass
The next pass can expand each major section into markers, fixtures, and test case placeholders.
## Glossary
- Contract mapping: Verifying that adapter input/output shapes match expected boundaries.
- Document-store tests: Tests that validate behavior against document-oriented persistence components.
- Marker taxonomy: The test marker classification scheme used to select test slices.