generated from john/python-template
12 KiB
12 KiB
MVP Definition: Historical Document Transcription System
1. MVP Objective
Deliver the thinnest possible end-to-end vertical slice — a user uploads an image of a document, the system transcribes it via the OpenRouter Python SDK, and the user reads the resulting transcript — with just enough persistence and structure to validate the core value proposition: can AI-driven transcription, guided by curated prompts, produce useful verbatim transcripts of historical family documents?
The MVP deliberately defers full-text search, export, revision history, MongoDB, and timeline assembly. These are additive features that don't need validation before the core transcription loop is proven.
2. Core User Story
As a family historian, I can upload a photo of a historical document, wait for it to be transcribed, and read the verbatim transcript — so I can evaluate whether this system will work for my thousands of documents.
3. In-Scope Requirements (from requirements.md)
| Requirement | ID | MVP Rationale |
|---|---|---|
| End-to-end transcription with lifecycle state | REQ-0 | This is the MVP. |
| Upload one or more images from the web UI | REQ-1 | Core entry point. MVP supports single-image upload (multi-image is a stretch goal). |
| Asynchronous processing → transcription or failure | REQ-2 | Validates the AI transcription pipeline. |
| Persist and expose job states (queued → processing → transcribed/failed) | REQ-3 | Minimum feedback loop for the user. |
| Persist transcription output and failure details | REQ-4 | User must be able to read the result. |
| UI views for status and transcript reading | REQ-5 | The user needs to see what happened. |
| Background processing to keep UI responsive | REQ-6 | Essential for usability during long AI calls. |
| Centralized config and logging at startup | REQ-8 | Small effort, high payoff for debugging. |
| Store transcription prompts as Markdown files | REQ-12 | Core to the Prompt Curation Policy in intent.md. Start with a single prompt file. |
Deferred to Post-MVP
| Requirement | ID | Why Deferred |
|---|---|---|
| Lifespan-owned runtime resources (engine, session factory, etc.) | REQ-7 | Important for production robustness, but a simple global or module-level setup is adequate for MVP validation. |
| Docker Compose (app + PostgreSQL + optional MongoDB) | REQ-9 | MVP runs locally with SQLite to eliminate container overhead during rapid iteration. PostgreSQL migration is Stage 1 hardening. |
| Explicit, opt-in schema bootstrap | REQ-10 | MVP uses auto-create-tables at startup (SQLModel create_all). Production schema discipline comes after the model stabilizes. |
| Service-backed persistence for core data | REQ-11 | MVP uses a thin repository layer over SQLite. Full service abstraction follows once the domain model is proven. |
4. MVP Feature Set
Feature 1: Document Upload (UI)
- A single NiceGUI page with a file-upload widget (accepts .jpg, .png, .tiff, .pdf).
- On upload: save the file to a local uploads/ directory, create a Document record, create a Job record with status queued.
- Minimal metadata capture: original filename, upload timestamp.
Feature 2: Asynchronous Transcription Worker
- An in-process background worker (Python asyncio task or BackgroundTasks) that:
- Picks up queued jobs.
- Transitions status to processing.
- Sends the image + the curated Markdown prompt to an AI vision model via OpenRouter.
- On success: saves the transcript text, transitions to transcribed.
- On failure: saves the error detail, transitions to failed.
Feature 3: Transcription Prompt (Markdown Asset)
- A single Markdown file (prompts/transcribe_document.md) encoding the verbatim transcription rules from intent.md (the Document Issues table, scholarly guidelines, etc.).
- The worker reads this file at invocation time and injects it as the system/user prompt.
Feature 4: Job Status & Transcript Viewer (UI)
- A job list page showing all jobs with their current status (queued / processing / transcribed / failed).
- A transcript detail page showing:
- The original uploaded image (rendered inline).
- The transcription text (or the failure reason).
- Timestamp metadata.
Feature 5: Minimal Persistence (SQLite + SQLModel)
- Three tables/models:
- Document: id, filename, file_path, uploaded_at.
- Job: id, document_id (FK), status, created_at, updated_at.
- Transcript: id, job_id (FK), text, error_detail, created_at.
- SQLite database file stored locally. Auto-created on first startup.
Feature 6: Centralized Configuration
- A single config.py (or Pydantic BaseSettings) loading:
- PROVIDER (fixed to openrouter for MVP)
- OPENROUTER_API_KEY (required)
- PROVIDER_MODEL (default: OpenRouter model slug for vision transcription)
- OPENROUTER_HTTP_REFERER (optional; app attribution)
- OPENROUTER_APP_TITLE (optional; app attribution)
- DATABASE_URL (default: sqlite:///./transcription.db)
- UPLOAD_DIR (default: ./uploads)
- PROMPT_DIR (default: ./prompts)
Feature 7: MVP Dependency Baseline (OpenRouter-Centric)
- Runtime dependencies:
- openrouter (official OpenRouter Python SDK)
- pydantic
- pydantic-settings
- sqlmodel
- Explicitly out of MVP runtime dependencies:
- google-genai (deferred until/if Gemini is introduced post-MVP)
5. MVP Architecture (Simplified)
┌─────────────────────────────────────────────┐
│ NiceGUI Web UI │
│ ┌──────────────┐ ┌───────────────────┐ │
│ │ Upload Page │ │ Jobs / Transcript │ │
│ └──────┬───────┘ └───────┬───────────┘ │
│ │ │ │
│ ▼ ▼ │
│ ┌───────────────────────────┐ │
│ │ Application Service │ │
│ │ (upload, job lifecycle) │ │
│ └─────┬─────────────┬───────┘ │
│ │ │ │
│ ┌─────▼─────┐ ┌─────▼───────────────┐ │
│ │ SQLite DB │ │ Background Worker │ │
│ │ (SQLModel)│ │ → AI Vision Provider│ │
│ └───────────┘ └─────────────────────┘ │
│ │ │
│ ┌─────▼──────┐ │
│ │ prompts/ │ │
│ │ *.md files │ │
│ └────────────┘ │
└─────────────────────────────────────────────┘
6. Proposed File Structure
project-root/
├── docs/ # (existing)
├── prompts/
│ └── transcribe_document.md # curated transcription prompt
├── src/
│ └── transcription/
│ ├── __init__.py
│ ├── app.py # FastAPI + NiceGUI app entrypoint
│ ├── config.py # Pydantic BaseSettings
│ ├── models.py # SQLModel: Document, Job, Transcript
│ ├── db.py # engine, session, create_all
│ ├── providers/
│ │ ├── __init__.py
│ │ ├── base.py # provider interface (transcribe contract)
│ │ ├── openrouter.py # OpenRouter via official Python SDK
│ ├── services/
│ │ ├── __init__.py
│ │ ├── upload.py # save file + create records
│ │ └── transcription.py # call provider, update job
│ ├── worker.py # background job loop
│ └── ui/
│ ├── __init__.py
│ ├── upload_page.py # NiceGUI upload page
│ └── jobs_page.py # NiceGUI job list + detail
├── tests/
│ ├── test_models.py
│ ├── test_upload.py
│ └── test_transcription.py
├── pyproject.toml
└── README.md
7. MVP Validation Criteria
The MVP is considered validated when:
- ✅ A user can upload an image of a document through the browser.
- ✅ The system asynchronously sends the image to the configured AI vision model with the curated prompt.
- ✅ The transcript (or failure reason) is persisted and visible in the UI.
- ✅ The transcription follows verbatim scholarly rules defined in intent.md (spot-checked by the user on real family documents).
- ✅ The transcription prompt is stored as a standalone Markdown file and can be edited without code changes.
- ✅ Job status transitions are visible: queued → processing → transcribed/failed.
8. Key Feedback Questions the MVP Should Answer
These are the real unknowns this MVP exists to resolve:
| # | Question | How We Learn |
|---|---|---|
| 1 | Is AI transcription quality good enough for this document corpus? | User reviews 20–50 real transcriptions against originals. |
| 2 | Does the verbatim prompt produce scholarly-quality output, or does it need major rework? | Compare output to the Document Issues table rules in intent.md. |
| 3 | What document types are hardest (old cursive, faded ink, pencil, postcards)? | Track which uploads produce failed or low-quality results. |
| 4 | Is single-image upload sufficient, or is batch upload needed early? | User friction during real scanning sessions. |
| 5 | What metadata is missing that the user wishes they could capture at upload time? | User feedback after processing real batches. |
9. What Comes After MVP (Immediate Post-MVP)
Once the core transcription loop is validated, the next priorities (aligned to Architecture Stage 1) are:
- Multi-image upload — process a batch from a scanning session.
- PostgreSQL migration — swap SQLite for containerized PostgreSQL (REQ-9, REQ-10).
- Revision history — allow the user to edit/correct transcripts with immutable version tracking.
- Full-text search — search across all accepted transcripts.
- Repository/service layer formalization — proper ports/adapters as the domain model stabilizes.
- Docker Compose deployment — containerize the app for reproducible operation.
10. Implementation Approach
Recommended build order for the MVP (each step produces a testable increment):
| Step | Deliverable | Validates |
|---|---|---|
| 1 | config.py + models.py + db.py — data layer with SQLite | Schema and config foundation |
| 2 | prompts/transcribe_document.md — curated prompt from intent.md | Prompt asset pattern |
| 3 | services/transcription.py + providers/ — call AI vision provider with prompt + image | Core AI integration |
| 4 | services/upload.py + worker.py — upload handling + background job loop | End-to-end pipeline (CLI-testable) |
| 5 | ui/upload_page.py + ui/jobs_page.py — NiceGUI pages | User-facing interface |
| 6 | tests/ — unit + integration tests Automated verification |
This MVP is deliberately narrow: one prompt, one provider (OpenRouter), one user, one image at a time, SQLite, no containers. Every omission is intentional — the goal is to get real family documents through the transcription pipeline as fast as possible and let the quality of the output guide every subsequent decision.