generated from john/python-template
209 lines
12 KiB
Markdown
209 lines
12 KiB
Markdown
## MVP Definition: Historical Document Transcription System
|
||
|
||
### 1. MVP Objective
|
||
Deliver the thinnest possible end-to-end vertical slice — a user uploads an image of a document, the system transcribes it via the OpenRouter Python SDK, and the user reads the resulting transcript — with just enough persistence and structure to validate the core value proposition: *can AI-driven transcription, guided by curated prompts, produce useful verbatim transcripts of historical family documents?*
|
||
|
||
The MVP deliberately defers full-text search, export, revision history, MongoDB, and timeline assembly. These are additive features that don't need validation before the core transcription loop is proven.
|
||
|
||
---
|
||
|
||
### 2. Core User Story
|
||
*As a family historian, I can upload a photo of a historical document, wait for it to be transcribed, and read the verbatim transcript — so I can evaluate whether this system will work for my thousands of documents.*
|
||
|
||
---
|
||
|
||
### 3. In-Scope Requirements (from ```requirements.md```)
|
||
|
||
| Requirement | ID | MVP Rationale |
|
||
| --- | --- | --- |
|
||
| End-to-end transcription with lifecycle state | REQ-0 | This is the MVP. |
|
||
| Upload one or more images from the web UI | REQ-1 | Core entry point. MVP supports single-image upload (multi-image is a stretch goal). |
|
||
| Asynchronous processing → transcription or failure | REQ-2 | Validates the AI transcription pipeline. |
|
||
| Persist and expose job states (queued → processing → transcribed/failed) | REQ-3 | Minimum feedback loop for the user. |
|
||
| Persist transcription output and failure details | REQ-4 | User must be able to read the result. |
|
||
| UI views for status and transcript reading | REQ-5 | The user needs to see what happened. |
|
||
| Background processing to keep UI responsive | REQ-6 | Essential for usability during long AI calls. |
|
||
| Centralized config and logging at startup | REQ-8 | Small effort, high payoff for debugging. |
|
||
| Store transcription prompts as Markdown files | REQ-12 | Core to the Prompt Curation Policy in intent.md. Start with a single prompt file. |
|
||
|
||
|
||
### Deferred to Post-MVP
|
||
| Requirement | ID | Why Deferred |
|
||
| --- | --- | --- |
|
||
| Lifespan-owned runtime resources (engine, session factory, etc.) | REQ-7 | Important for production robustness, but a simple global or module-level setup is adequate for MVP validation. |
|
||
| Docker Compose (app + PostgreSQL + optional MongoDB) | REQ-9 | MVP runs locally with SQLite to eliminate container overhead during rapid iteration. PostgreSQL migration is Stage 1 hardening. |
|
||
| Explicit, opt-in schema bootstrap | REQ-10 | MVP uses auto-create-tables at startup (SQLModel create_all). Production schema discipline comes after the model stabilizes. |
|
||
| Service-backed persistence for core data | REQ-11 | MVP uses a thin repository layer over SQLite. Full service abstraction follows once the domain model is proven. |
|
||
|
||
---
|
||
|
||
### 4. MVP Feature Set
|
||
#### Feature 1: Document Upload (UI)
|
||
* A single NiceGUI page with a file-upload widget (accepts .jpg, .png, .tiff, .pdf).
|
||
* On upload: save the file to a local uploads/ directory, create a Document record, create a Job record with status queued.
|
||
* Minimal metadata capture: original filename, upload timestamp.
|
||
|
||
#### Feature 2: Asynchronous Transcription Worker
|
||
* An in-process background worker (Python asyncio task or BackgroundTasks) that:
|
||
1. Picks up queued jobs.
|
||
2. Transitions status to processing.
|
||
3. Sends the image + the curated Markdown prompt to an AI vision model via OpenRouter.
|
||
4. On success: saves the transcript text, transitions to transcribed.
|
||
5. On failure: saves the error detail, transitions to failed.
|
||
|
||
#### Feature 3: Transcription Prompt (Markdown Asset)
|
||
* A single Markdown file (prompts/transcribe_document.md) encoding the verbatim transcription rules from intent.md (the Document Issues table, scholarly guidelines, etc.).
|
||
* The worker reads this file at invocation time and injects it as the system/user prompt.
|
||
|
||
#### Feature 4: Job Status & Transcript Viewer (UI)
|
||
* A job list page showing all jobs with their current status (queued / processing / transcribed / failed).
|
||
* A transcript detail page showing:
|
||
* The original uploaded image (rendered inline).
|
||
* The transcription text (or the failure reason).
|
||
* Timestamp metadata.
|
||
|
||
#### Feature 5: Minimal Persistence (SQLite + SQLModel)
|
||
* Three tables/models:
|
||
* Document: id, filename, file_path, uploaded_at.
|
||
* Job: id, document_id (FK), status, created_at, updated_at.
|
||
* Transcript: id, job_id (FK), text, error_detail, created_at.
|
||
* SQLite database file stored locally. Auto-created on first startup.
|
||
|
||
#### Feature 6: Centralized Configuration
|
||
* A single config.py (or Pydantic BaseSettings) loading:
|
||
* PROVIDER (fixed to openrouter for MVP)
|
||
* OPENROUTER_API_KEY (required)
|
||
* PROVIDER_MODEL (default: OpenRouter model slug for vision transcription)
|
||
* OPENROUTER_HTTP_REFERER (optional; app attribution)
|
||
* OPENROUTER_APP_TITLE (optional; app attribution)
|
||
* DATABASE_URL (default: sqlite:///./transcription.db)
|
||
* UPLOAD_DIR (default: ./uploads)
|
||
* PROMPT_DIR (default: ./prompts)
|
||
|
||
#### Feature 7: MVP Dependency Baseline (OpenRouter-Centric)
|
||
* Runtime dependencies:
|
||
* openrouter (official OpenRouter Python SDK)
|
||
* pydantic
|
||
* pydantic-settings
|
||
* sqlmodel
|
||
* Explicitly out of MVP runtime dependencies:
|
||
* google-genai (deferred until/if Gemini is introduced post-MVP)
|
||
|
||
---
|
||
|
||
### 5. MVP Architecture (Simplified)
|
||
|
||
```Apply
|
||
┌─────────────────────────────────────────────┐
|
||
│ NiceGUI Web UI │
|
||
│ ┌──────────────┐ ┌───────────────────┐ │
|
||
│ │ Upload Page │ │ Jobs / Transcript │ │
|
||
│ └──────┬───────┘ └───────┬───────────┘ │
|
||
│ │ │ │
|
||
│ ▼ ▼ │
|
||
│ ┌───────────────────────────┐ │
|
||
│ │ Application Service │ │
|
||
│ │ (upload, job lifecycle) │ │
|
||
│ └─────┬─────────────┬───────┘ │
|
||
│ │ │ │
|
||
│ ┌─────▼─────┐ ┌─────▼───────────────┐ │
|
||
│ │ SQLite DB │ │ Background Worker │ │
|
||
│ │ (SQLModel)│ │ → AI Vision Provider│ │
|
||
│ └───────────┘ └─────────────────────┘ │
|
||
│ │ │
|
||
│ ┌─────▼──────┐ │
|
||
│ │ prompts/ │ │
|
||
│ │ *.md files │ │
|
||
│ └────────────┘ │
|
||
└─────────────────────────────────────────────┘
|
||
```
|
||
|
||
---
|
||
|
||
#### 6. Proposed File Structure
|
||
|
||
```Apply
|
||
project-root/
|
||
├── docs/ # (existing)
|
||
├── prompts/
|
||
│ └── transcribe_document.md # curated transcription prompt
|
||
├── src/
|
||
│ └── transcription/
|
||
│ ├── __init__.py
|
||
│ ├── app.py # FastAPI + NiceGUI app entrypoint
|
||
│ ├── config.py # Pydantic BaseSettings
|
||
│ ├── models.py # SQLModel: Document, Job, Transcript
|
||
│ ├── db.py # engine, session, create_all
|
||
│ ├── providers/
|
||
│ │ ├── __init__.py
|
||
│ │ ├── base.py # provider interface (transcribe contract)
|
||
│ │ ├── openrouter.py # OpenRouter via official Python SDK
|
||
│ ├── services/
|
||
│ │ ├── __init__.py
|
||
│ │ ├── upload.py # save file + create records
|
||
│ │ └── transcription.py # call provider, update job
|
||
│ ├── worker.py # background job loop
|
||
│ └── ui/
|
||
│ ├── __init__.py
|
||
│ ├── upload_page.py # NiceGUI upload page
|
||
│ └── jobs_page.py # NiceGUI job list + detail
|
||
├── tests/
|
||
│ ├── test_models.py
|
||
│ ├── test_upload.py
|
||
│ └── test_transcription.py
|
||
├── pyproject.toml
|
||
└── README.md
|
||
```
|
||
|
||
---
|
||
|
||
#### 7. MVP Validation Criteria
|
||
The MVP is considered validated when:
|
||
|
||
1. ✅ A user can upload an image of a document through the browser.
|
||
2. ✅ The system asynchronously sends the image to the configured AI vision model with the curated prompt.
|
||
3. ✅ The transcript (or failure reason) is persisted and visible in the UI.
|
||
4. ✅ The transcription follows verbatim scholarly rules defined in intent.md (spot-checked by the user on real family documents).
|
||
5. ✅ The transcription prompt is stored as a standalone Markdown file and can be edited without code changes.
|
||
6. ✅ Job status transitions are visible: queued → processing → transcribed/failed.
|
||
|
||
---
|
||
|
||
### 8. Key Feedback Questions the MVP Should Answer
|
||
These are the real unknowns this MVP exists to resolve:
|
||
|
||
| # | Question | How We Learn |
|
||
| --- | --- | --- |
|
||
| 1 | Is AI transcription quality good enough for this document corpus? | User reviews 20–50 real transcriptions against originals. |
|
||
| 2 | Does the verbatim prompt produce scholarly-quality output, or does it need major rework? | Compare output to the Document Issues table rules in intent.md. |
|
||
| 3 | What document types are hardest (old cursive, faded ink, pencil, postcards)? | Track which uploads produce failed or low-quality results. |
|
||
| 4 | Is single-image upload sufficient, or is batch upload needed early? | User friction during real scanning sessions. |
|
||
| 5 | What metadata is missing that the user wishes they could capture at upload time? | User feedback after processing real batches. |
|
||
|
||
---
|
||
|
||
#### 9. What Comes After MVP (Immediate Post-MVP)
|
||
Once the core transcription loop is validated, the next priorities (aligned to Architecture Stage 1) are:
|
||
|
||
1. **Multi-image upload** — process a batch from a scanning session.
|
||
2. **PostgreSQL migration** — swap SQLite for containerized PostgreSQL (REQ-9, REQ-10).
|
||
3. **Revision history** — allow the user to edit/correct transcripts with immutable version tracking.
|
||
4. **Full-text search** — search across all accepted transcripts.
|
||
5. **Repository/service layer formalization** — proper ports/adapters as the domain model stabilizes.
|
||
6. **Docker Compose deployment** — containerize the app for reproducible operation.
|
||
|
||
---
|
||
|
||
#### 10. Implementation Approach
|
||
Recommended build order for the MVP (each step produces a testable increment):
|
||
|
||
| Step | Deliverable | Validates |
|
||
| --- | --- | --- |
|
||
| 1 | config.py + models.py + db.py — data layer with SQLite | Schema and config foundation |
|
||
| 2 | prompts/transcribe_document.md — curated prompt from intent.md | Prompt asset pattern |
|
||
| 3 | services/transcription.py + providers/ — call AI vision provider with prompt + image | Core AI integration |
|
||
| 4 | services/upload.py + worker.py — upload handling + background job loop | End-to-end pipeline (CLI-testable) |
|
||
| 5 | ui/upload_page.py + ui/jobs_page.py — NiceGUI pages | User-facing interface |
|
||
| 6 | tests/ — unit + integration tests Automated verification |
|
||
|
||
This MVP is deliberately narrow: **one prompt, one provider (OpenRouter), one user, one image at a time, SQLite, no containers**. Every omission is intentional — the goal is to get real family documents through the transcription pipeline as fast as possible and let the quality of the output guide every subsequent decision. |