generated from john/python-template
Creation of MVP.md
This commit is contained in:
Jim Lancaster
+193
@@ -0,0 +1,193 @@
|
||||
## MVP Definition: Historical Document Transcription System
|
||||
|
||||
### 1. MVP Objective
|
||||
Deliver the thinnest possible end-to-end vertical slice — a user uploads an image of a handwritten document, the system transcribes it via an AI provider, and the user reads the resulting transcript — with just enough persistence and structure to validate the core value proposition: *can AI-driven transcription, guided by curated prompts, produce useful verbatim transcripts of historical family documents?*
|
||||
|
||||
The MVP deliberately defers full-text search, export, revision history, MongoDB, timeline assembly, and multi-provider routing. These are additive features that don't need validation before the core transcription loop is proven.
|
||||
|
||||
---
|
||||
|
||||
### 2. Core User Story
|
||||
*As a family historian, I can upload a photo of a handwritten letter, wait for it to be transcribed, and read the verbatim transcript — so I can evaluate whether this system will work for my thousands of documents.*
|
||||
|
||||
---
|
||||
|
||||
### 3. In-Scope Requirements (from ```requirements.md```)
|
||||
|
||||
| Requirement | ID | MVP Rationale |
|
||||
| --- | --- | --- |
|
||||
| End-to-end transcription with lifecycle state | REQ-0 | This is the MVP. |
|
||||
| Upload one or more images from the web UI | REQ-1 | Core entry point. MVP supports single-image upload (multi-image is a stretch goal). |
|
||||
| Asynchronous processing → transcription or failure | REQ-2 | Validates the AI transcription pipeline. |
|
||||
| Persist and expose job states (queued → processing → transcribed/failed) | REQ-3 | Minimum feedback loop for the user. |
|
||||
| Persist transcription output and failure details | REQ-4 | User must be able to read the result. |
|
||||
| UI views for status and transcript reading | REQ-5 | The user needs to see what happened. |
|
||||
| Background processing to keep UI responsive | REQ-6 | Essential for usability during long AI calls. |
|
||||
| Centralized config and logging at startup | REQ-8 | Small effort, high payoff for debugging. |
|
||||
| Store transcription prompts as Markdown files | REQ-12 | Core to the Prompt Curation Policy in Intent.md. Start with a single prompt file. |
|
||||
|
||||
|
||||
### Deferred to Post-MVP
|
||||
| Requirement | ID | Why Deferred |
|
||||
| --- | --- | --- |
|
||||
| Lifespan-owned runtime resources (engine, session factory, etc.) | REQ-7 | Important for production robustness, but a simple global or module-level setup is adequate for MVP validation. |
|
||||
| Docker Compose (app + PostgreSQL + optional MongoDB) | REQ-9 | MVP runs locally with SQLite to eliminate container overhead during rapid iteration. PostgreSQL migration is Stage 1 hardening. |
|
||||
| Explicit, opt-in schema bootstrap | REQ-10 | MVP uses auto-create-tables at startup (SQLModel create_all). Production schema discipline comes after the model stabilizes. |
|
||||
| Service-backed persistence for core data | REQ-11 | MVP uses a thin repository layer over SQLite. Full service abstraction follows once the domain model is proven. |
|
||||
|
||||
---
|
||||
|
||||
### 4. MVP Feature Set
|
||||
#### Feature 1: Document Upload (UI)
|
||||
* A single NiceGUI page with a file-upload widget (accepts .jpg, .png, .tiff, .pdf).
|
||||
* On upload: save the file to a local uploads/ directory, create a Document record, create a Job record with status queued.
|
||||
* Minimal metadata capture: original filename, upload timestamp.
|
||||
|
||||
#### Feature 2: Asynchronous Transcription Worker
|
||||
* An in-process background worker (Python asyncio task or BackgroundTasks) that:
|
||||
1. Picks up queued jobs.
|
||||
2. Transitions status to processing.
|
||||
3. Sends the image + the curated Markdown prompt to an AI vision model (OpenAI API — already a dependency).
|
||||
4. On success: saves the transcript text, transitions to transcribed.
|
||||
5. On failure: saves the error detail, transitions to failed.
|
||||
|
||||
#### Feature 3: Transcription Prompt (Markdown Asset)
|
||||
* A single Markdown file (prompts/transcribe_handwriting.md) encoding the verbatim transcription rules from Intent.md (the Document Issues table, scholarly guidelines, etc.).
|
||||
* The worker reads this file at invocation time and injects it as the system/user prompt.
|
||||
|
||||
#### Feature 4: Job Status & Transcript Viewer (UI)
|
||||
* A job list page showing all jobs with their current status (queued / processing / transcribed / failed).
|
||||
* A transcript detail page showing:
|
||||
* The original uploaded image (rendered inline).
|
||||
* The transcription text (or the failure reason).
|
||||
* Timestamp metadata.
|
||||
|
||||
#### Feature 5: Minimal Persistence (SQLite + SQLModel)
|
||||
* Three tables/models:
|
||||
* Document: id, filename, file_path, uploaded_at.
|
||||
* Job: id, document_id (FK), status, created_at, updated_at.
|
||||
* Transcript: id, job_id (FK), text, error_detail, created_at.
|
||||
* SQLite database file stored locally. Auto-created on first startup.
|
||||
|
||||
#### Feature 6: Centralized Configuration
|
||||
* A single config.py (or Pydantic BaseSettings) loading:
|
||||
* OPENAI_API_KEY (required)
|
||||
* OPENAI_MODEL (default: gpt-4o)
|
||||
* DATABASE_URL (default: sqlite:///./transcription.db)
|
||||
* UPLOAD_DIR (default: ./uploads)
|
||||
* PROMPT_DIR (default: ./prompts)
|
||||
|
||||
---
|
||||
|
||||
### 5. MVP Architecture (Simplified)
|
||||
|
||||
```Apply
|
||||
┌─────────────────────────────────────────────┐
|
||||
│ NiceGUI Web UI │
|
||||
│ ┌──────────────┐ ┌───────────────────┐ │
|
||||
│ │ Upload Page │ │ Jobs / Transcript │ │
|
||||
│ └──────┬───────┘ └───────┬───────────┘ │
|
||||
│ │ │ │
|
||||
│ ▼ ▼ │
|
||||
│ ┌───────────────────────────┐ │
|
||||
│ │ Application Service │ │
|
||||
│ │ (upload, job lifecycle) │ │
|
||||
│ └─────┬─────────────┬───────┘ │
|
||||
│ │ │ │
|
||||
│ ┌─────▼─────┐ ┌─────▼─────────────┐ │
|
||||
│ │ SQLite DB │ │ Background Worker │ │
|
||||
│ │ (SQLModel)│ │ → OpenAI Vision │ │
|
||||
│ └───────────┘ └───────────────────┘ │
|
||||
│ │ │
|
||||
│ ┌─────▼──────┐ │
|
||||
│ │ prompts/ │ │
|
||||
│ │ *.md files │ │
|
||||
│ └────────────┘ │
|
||||
└─────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
#### 6. Proposed File Structure
|
||||
|
||||
```Apply
|
||||
project-root/
|
||||
├── docs/ # (existing)
|
||||
├── prompts/
|
||||
│ └── transcribe_handwriting.md # curated transcription prompt
|
||||
├── src/
|
||||
│ └── handwriting/
|
||||
│ ├── __init__.py
|
||||
│ ├── app.py # FastAPI + NiceGUI app entrypoint
|
||||
│ ├── config.py # Pydantic BaseSettings
|
||||
│ ├── models.py # SQLModel: Document, Job, Transcript
|
||||
│ ├── db.py # engine, session, create_all
|
||||
│ ├── services/
|
||||
│ │ ├── __init__.py
|
||||
│ │ ├── upload.py # save file + create records
|
||||
│ │ └── transcription.py # call AI provider, update job
|
||||
│ ├── worker.py # background job loop
|
||||
│ └── ui/
|
||||
│ ├── __init__.py
|
||||
│ ├── upload_page.py # NiceGUI upload page
|
||||
│ └── jobs_page.py # NiceGUI job list + detail
|
||||
├── tests/
|
||||
│ ├── test_models.py
|
||||
│ ├── test_upload.py
|
||||
│ └── test_transcription.py
|
||||
├── pyproject.toml
|
||||
└── README.md
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
#### 7. MVP Validation Criteria
|
||||
The MVP is considered validated when:
|
||||
|
||||
1. ✅ A user can upload an image of a handwritten document through the browser.
|
||||
2. ✅ The system asynchronously sends the image to OpenAI's vision model with the curated prompt.
|
||||
3. ✅ The transcript (or failure reason) is persisted and visible in the UI.
|
||||
4. ✅ The transcription follows verbatim scholarly rules defined in Intent.md (spot-checked by the user on real family documents).
|
||||
5. ✅ The transcription prompt is stored as a standalone Markdown file and can be edited without code changes.
|
||||
6. ✅ Job status transitions are visible: queued → processing → transcribed/failed.
|
||||
|
||||
---
|
||||
|
||||
### 8. Key Feedback Questions the MVP Should Answer
|
||||
These are the real unknowns this MVP exists to resolve:
|
||||
|
||||
| # | Question | How We Learn |
|
||||
| --- | --- | --- |
|
||||
| 1 | Is AI transcription quality good enough for this handwriting corpus? | User reviews 20–50 real transcriptions against originals. |
|
||||
| 2 | Does the verbatim prompt produce scholarly-quality output, or does it need major rework? | Compare output to the Document Issues table rules in Intent.md. |
|
||||
| 3 | What document types are hardest (old cursive, faded ink, pencil, postcards)? | Track which uploads produce failed or low-quality results. |
|
||||
| 4 | Is single-image upload sufficient, or is batch upload needed early? | User friction during real scanning sessions. |
|
||||
| 5 | What metadata is missing that the user wishes they could capture at upload time? | User feedback after processing real batches. |
|
||||
|
||||
---
|
||||
|
||||
#### 9. What Comes After MVP (Immediate Post-MVP)
|
||||
Once the core transcription loop is validated, the next priorities (aligned to Architecture Stage 1) are:
|
||||
|
||||
1. **Multi-image upload** — process a batch from a scanning session.
|
||||
2. **PostgreSQL migration** — swap SQLite for containerized PostgreSQL (REQ-9, REQ-10).
|
||||
3. **Revision history** — allow the user to edit/correct transcripts with immutable version tracking.
|
||||
4. **Full-text search** — search across all accepted transcripts.
|
||||
5. **Repository/service layer formalization** — proper ports/adapters as the domain model stabilizes.
|
||||
6. **Docker Compose deployment** — containerize the app for reproducible operation.
|
||||
|
||||
---
|
||||
|
||||
#### 10. Implementation Approach
|
||||
Recommended build order for the MVP (each step produces a testable increment):
|
||||
|
||||
| Step | Deliverable | Validates |
|
||||
| --- | --- | --- |
|
||||
| 1 | config.py + models.py + db.py — data layer with SQLite | Schema and config foundation |
|
||||
| 2 | prompts/transcribe_handwriting.md — curated prompt from Intent.md | Prompt asset pattern |
|
||||
| 3 | services/transcription.py — call OpenAI vision API with prompt + image | Core AI integration |
|
||||
| 4 | services/upload.py + worker.py — upload handling + background job loop | End-to-end pipeline (CLI-testable) |
|
||||
| 5 | ui/upload_page.py + ui/jobs_page.py — NiceGUI pages | User-facing interface |
|
||||
| 6 | tests/ — unit + integration tests Automated verification |
|
||||
|
||||
This MVP is deliberately narrow: **one prompt, one provider, one user, one image at a time, SQLite, no containers**. Every omission is intentional — the goal is to get real family documents through the transcription pipeline as fast as possible and let the quality of the output guide every subsequent decision.
|
||||
Reference in New Issue
Block a user