From d0626c76537b341807713823bac187ac97b9e742 Mon Sep 17 00:00:00 2001 From: Jim Lancaster <40281233+zoltan57@users.noreply.github.com> Date: Tue, 23 Jun 2026 18:56:29 -0500 Subject: [PATCH] Creation of MVP.md --- docs/mvp.md | 193 +++++++++++++++++++++++++++++++++++++++++++ docs/requirements.md | 16 ++-- docs/tests.md | 121 --------------------------- 3 files changed, 201 insertions(+), 129 deletions(-) create mode 100644 docs/mvp.md delete mode 100644 docs/tests.md diff --git a/docs/mvp.md b/docs/mvp.md new file mode 100644 index 0000000..867716f --- /dev/null +++ b/docs/mvp.md @@ -0,0 +1,193 @@ +## MVP Definition: Historical Document Transcription System + +### 1. MVP Objective +Deliver the thinnest possible end-to-end vertical slice — a user uploads an image of a handwritten document, the system transcribes it via an AI provider, and the user reads the resulting transcript — with just enough persistence and structure to validate the core value proposition: *can AI-driven transcription, guided by curated prompts, produce useful verbatim transcripts of historical family documents?* + +The MVP deliberately defers full-text search, export, revision history, MongoDB, timeline assembly, and multi-provider routing. These are additive features that don't need validation before the core transcription loop is proven. + +--- + +### 2. Core User Story +*As a family historian, I can upload a photo of a handwritten letter, wait for it to be transcribed, and read the verbatim transcript — so I can evaluate whether this system will work for my thousands of documents.* + +--- + +### 3. In-Scope Requirements (from ```requirements.md```) + +| Requirement | ID | MVP Rationale | +| --- | --- | --- | +| End-to-end transcription with lifecycle state | REQ-0 | This is the MVP. | +| Upload one or more images from the web UI | REQ-1 | Core entry point. MVP supports single-image upload (multi-image is a stretch goal). | +| Asynchronous processing → transcription or failure | REQ-2 | Validates the AI transcription pipeline. | +| Persist and expose job states (queued → processing → transcribed/failed) | REQ-3 | Minimum feedback loop for the user. | +| Persist transcription output and failure details | REQ-4 | User must be able to read the result. | +| UI views for status and transcript reading | REQ-5 | The user needs to see what happened. | +| Background processing to keep UI responsive | REQ-6 | Essential for usability during long AI calls. | +| Centralized config and logging at startup | REQ-8 | Small effort, high payoff for debugging. | +| Store transcription prompts as Markdown files | REQ-12 | Core to the Prompt Curation Policy in Intent.md. Start with a single prompt file. | + + +### Deferred to Post-MVP +| Requirement | ID | Why Deferred | +| --- | --- | --- | +| Lifespan-owned runtime resources (engine, session factory, etc.) | REQ-7 | Important for production robustness, but a simple global or module-level setup is adequate for MVP validation. | +| Docker Compose (app + PostgreSQL + optional MongoDB) | REQ-9 | MVP runs locally with SQLite to eliminate container overhead during rapid iteration. PostgreSQL migration is Stage 1 hardening. | +| Explicit, opt-in schema bootstrap | REQ-10 | MVP uses auto-create-tables at startup (SQLModel create_all). Production schema discipline comes after the model stabilizes. | +| Service-backed persistence for core data | REQ-11 | MVP uses a thin repository layer over SQLite. Full service abstraction follows once the domain model is proven. | + +--- + +### 4. MVP Feature Set +#### Feature 1: Document Upload (UI) +* A single NiceGUI page with a file-upload widget (accepts .jpg, .png, .tiff, .pdf). +* On upload: save the file to a local uploads/ directory, create a Document record, create a Job record with status queued. +* Minimal metadata capture: original filename, upload timestamp. + +#### Feature 2: Asynchronous Transcription Worker +* An in-process background worker (Python asyncio task or BackgroundTasks) that: + 1. Picks up queued jobs. + 2. Transitions status to processing. + 3. Sends the image + the curated Markdown prompt to an AI vision model (OpenAI API — already a dependency). + 4. On success: saves the transcript text, transitions to transcribed. + 5. On failure: saves the error detail, transitions to failed. + +#### Feature 3: Transcription Prompt (Markdown Asset) +* A single Markdown file (prompts/transcribe_handwriting.md) encoding the verbatim transcription rules from Intent.md (the Document Issues table, scholarly guidelines, etc.). +* The worker reads this file at invocation time and injects it as the system/user prompt. + +#### Feature 4: Job Status & Transcript Viewer (UI) +* A job list page showing all jobs with their current status (queued / processing / transcribed / failed). +* A transcript detail page showing: + * The original uploaded image (rendered inline). + * The transcription text (or the failure reason). + * Timestamp metadata. + +#### Feature 5: Minimal Persistence (SQLite + SQLModel) +* Three tables/models: + * Document: id, filename, file_path, uploaded_at. + * Job: id, document_id (FK), status, created_at, updated_at. + * Transcript: id, job_id (FK), text, error_detail, created_at. +* SQLite database file stored locally. Auto-created on first startup. + +#### Feature 6: Centralized Configuration +* A single config.py (or Pydantic BaseSettings) loading: + * OPENAI_API_KEY (required) + * OPENAI_MODEL (default: gpt-4o) + * DATABASE_URL (default: sqlite:///./transcription.db) + * UPLOAD_DIR (default: ./uploads) + * PROMPT_DIR (default: ./prompts) + +--- + +### 5. MVP Architecture (Simplified) + +```Apply +┌─────────────────────────────────────────────┐ +│ NiceGUI Web UI │ +│ ┌──────────────┐ ┌───────────────────┐ │ +│ │ Upload Page │ │ Jobs / Transcript │ │ +│ └──────┬───────┘ └───────┬───────────┘ │ +│ │ │ │ +│ ▼ ▼ │ +│ ┌───────────────────────────┐ │ +│ │ Application Service │ │ +│ │ (upload, job lifecycle) │ │ +│ └─────┬─────────────┬───────┘ │ +│ │ │ │ +│ ┌─────▼─────┐ ┌─────▼─────────────┐ │ +│ │ SQLite DB │ │ Background Worker │ │ +│ │ (SQLModel)│ │ → OpenAI Vision │ │ +│ └───────────┘ └───────────────────┘ │ +│ │ │ +│ ┌─────▼──────┐ │ +│ │ prompts/ │ │ +│ │ *.md files │ │ +│ └────────────┘ │ +└─────────────────────────────────────────────┘ +``` + +--- + +#### 6. Proposed File Structure + +```Apply +project-root/ +├── docs/ # (existing) +├── prompts/ +│ └── transcribe_handwriting.md # curated transcription prompt +├── src/ +│ └── handwriting/ +│ ├── __init__.py +│ ├── app.py # FastAPI + NiceGUI app entrypoint +│ ├── config.py # Pydantic BaseSettings +│ ├── models.py # SQLModel: Document, Job, Transcript +│ ├── db.py # engine, session, create_all +│ ├── services/ +│ │ ├── __init__.py +│ │ ├── upload.py # save file + create records +│ │ └── transcription.py # call AI provider, update job +│ ├── worker.py # background job loop +│ └── ui/ +│ ├── __init__.py +│ ├── upload_page.py # NiceGUI upload page +│ └── jobs_page.py # NiceGUI job list + detail +├── tests/ +│ ├── test_models.py +│ ├── test_upload.py +│ └── test_transcription.py +├── pyproject.toml +└── README.md +``` + +--- + +#### 7. MVP Validation Criteria +The MVP is considered validated when: + +1. ✅ A user can upload an image of a handwritten document through the browser. +2. ✅ The system asynchronously sends the image to OpenAI's vision model with the curated prompt. +3. ✅ The transcript (or failure reason) is persisted and visible in the UI. +4. ✅ The transcription follows verbatim scholarly rules defined in Intent.md (spot-checked by the user on real family documents). +5. ✅ The transcription prompt is stored as a standalone Markdown file and can be edited without code changes. +6. ✅ Job status transitions are visible: queued → processing → transcribed/failed. + +--- + +### 8. Key Feedback Questions the MVP Should Answer +These are the real unknowns this MVP exists to resolve: + +| # | Question | How We Learn | +| --- | --- | --- | +| 1 | Is AI transcription quality good enough for this handwriting corpus? | User reviews 20–50 real transcriptions against originals. | +| 2 | Does the verbatim prompt produce scholarly-quality output, or does it need major rework? | Compare output to the Document Issues table rules in Intent.md. | +| 3 | What document types are hardest (old cursive, faded ink, pencil, postcards)? | Track which uploads produce failed or low-quality results. | +| 4 | Is single-image upload sufficient, or is batch upload needed early? | User friction during real scanning sessions. | +| 5 | What metadata is missing that the user wishes they could capture at upload time? | User feedback after processing real batches. | + +--- + +#### 9. What Comes After MVP (Immediate Post-MVP) +Once the core transcription loop is validated, the next priorities (aligned to Architecture Stage 1) are: + +1. **Multi-image upload** — process a batch from a scanning session. +2. **PostgreSQL migration** — swap SQLite for containerized PostgreSQL (REQ-9, REQ-10). +3. **Revision history** — allow the user to edit/correct transcripts with immutable version tracking. +4. **Full-text search** — search across all accepted transcripts. +5. **Repository/service layer formalization** — proper ports/adapters as the domain model stabilizes. +6. **Docker Compose deployment** — containerize the app for reproducible operation. + +--- + +#### 10. Implementation Approach +Recommended build order for the MVP (each step produces a testable increment): + +| Step | Deliverable | Validates | +| --- | --- | --- | +| 1 | config.py + models.py + db.py — data layer with SQLite | Schema and config foundation | +| 2 | prompts/transcribe_handwriting.md — curated prompt from Intent.md | Prompt asset pattern | +| 3 | services/transcription.py — call OpenAI vision API with prompt + image | Core AI integration | +| 4 | services/upload.py + worker.py — upload handling + background job loop | End-to-end pipeline (CLI-testable) | +| 5 | ui/upload_page.py + ui/jobs_page.py — NiceGUI pages | User-facing interface | +| 6 | tests/ — unit + integration tests Automated verification | + +This MVP is deliberately narrow: **one prompt, one provider, one user, one image at a time, SQLite, no containers**. Every omission is intentional — the goal is to get real family documents through the transcription pipeline as fast as possible and let the quality of the output guide every subsequent decision. \ No newline at end of file diff --git a/docs/requirements.md b/docs/requirements.md index 262b995..0481a3b 100644 --- a/docs/requirements.md +++ b/docs/requirements.md @@ -1,4 +1,4 @@ -## Handwriting Transcription System Requirements +## Document Transcription System Requirements This page captures a SysML v1.6-style requirements baseline for the production system described in [index.md](index.md). The model is represented as concise tables and traceability lists that preserve SysML-style IDs and relationship semantics. @@ -14,8 +14,8 @@ This page captures a SysML v1.6-style requirements baseline for the production s | ID | Category | Requirement | Risk | Verify Method | | --- | --- | --- | --- | --- | -| REQ-0 | System | Provide end-to-end handwriting transcription with persistent, inspectable lifecycle state. | medium | demonstration | -| REQ-1 | Functional | Allow users to upload one or more handwriting images from the web UI. | low | test | +| REQ-0 | System | Provide end-to-end document transcription with persistent, inspectable lifecycle state. | medium | demonstration | +| REQ-1 | Functional | Allow users to upload one or more document images from the web UI. | low | test | | REQ-2 | Functional | Run each upload through asynchronous processing that returns a transcription or explicit failure. | high | test | | REQ-3 | Functional | Persist and expose job states: upload, queued, processing, transcribed, failed, completed. | high | inspection | | REQ-4 | Functional | Persist transcription output, processing history, and failure details. | medium | test | @@ -39,11 +39,11 @@ This page captures a SysML v1.6-style requirements baseline for the production s | Element | Type | Doc Reference | | --- | --- | --- | -| UI | NiceGUI pages | src/handwriting/ui/pages | -| API | FastAPI routes | src/handwriting/api/routes.py | -| GRAPH | Async processing workflow | src/handwriting/services, src/handwriting/ai | -| DBREL | PostgreSQL + SQLModel relational persistence | src/handwriting/db | -| DBDOC | MongoDB document persistence | src/handwriting/db, src/handwriting/services | +| UI | NiceGUI pages | src/transcription/ui/pages | +| API | FastAPI routes | src/transcription/api/routes.py | +| GRAPH | Async processing workflow | src/transcription/services, src/transcription/ai | +| DBREL | PostgreSQL + SQLModel relational persistence | src/transcription/db | +| DBDOC | MongoDB document persistence | src/transcription/db, src/transcription/services | | OPS | Docker Compose runtime | docker-compose.yml | | PROMPTS | Transcription prompt artifact library (Markdown files) | .github/prompts, docs | | TESTS | Pytest verification suite | tests | diff --git a/docs/tests.md b/docs/tests.md deleted file mode 100644 index a3cc291..0000000 --- a/docs/tests.md +++ /dev/null @@ -1,121 +0,0 @@ -# Testing - -The greenfield test structure mirrors the source tree at a high level and keeps the first pass shallow and easy to extend. - -## Source To Test Map - -- `src/handwriting/bootstrap.py`, `src/handwriting/config.py`, `src/handwriting/logging_buffer.py`, `src/handwriting/main.py` -> `tests/handwriting/test_core.py` -- `src/handwriting/ai/**` -> `tests/handwriting/ai/` -- `src/handwriting/api/**` -> `tests/handwriting/api/` -- `src/handwriting/db/**` -> `tests/handwriting/db/` -- `src/handwriting/services/**` -> `tests/handwriting/services/` -- `src/handwriting/ui/components/**` and `src/handwriting/ui/pages/**` -> `tests/handwriting/ui/` -- Shared test helpers and fixtures -> `tests/conftest.py` plus subtree `conftest.py` files where needed - -## Major Sections - -- `tests/handwriting/test_core.py`: bootstrap, config, logging, and entrypoint coverage. -- `tests/handwriting/ai/`: AI contracts, runtime orchestration, graph wiring, and node behavior. -- `tests/handwriting/api/`: HTTP routes and request/response contract checks. -- `tests/handwriting/db/`: session setup, models, and repositories. -- `tests/handwriting/services/`: service-layer orchestration and domain logic. -- `tests/handwriting/ui/`: NiceGUI components and pages. -- `tests/conftest.py`: shared lightweight fixtures and deterministic defaults. -- `tests/handwriting/**/conftest.py`: subtree-specific fixtures only where a package needs its own setup. - -## Implemented In This Pass (Core + DB) - -### Created Files - -- `tests/handwriting/test_core.py` -- `tests/handwriting/conftest.py` -- `tests/handwriting/db/conftest.py` -- `tests/handwriting/db/test_session.py` -- `tests/handwriting/db/test_models.py` -- `tests/handwriting/db/test_job_repo.py` -- `tests/handwriting/db/test_service_job.py` - -### Updated Files - -- `tests/conftest.py`: added shared `settings_factory` fixture. -- `pyproject.toml`: registered strict markers and async pytest mode. - -### Marker Taxonomy - -- `unit`: fast deterministic unit tests. -- `db`: database-backed tests against PostgreSQL-backed fixtures. -- `document`: document-store tests for MongoDB-backed persistence behavior. -- `integration`: cross-layer integration tests (reserved for expanded next pass). -- `smoke`: high-value end-to-end surface checks. -- `slow`: long-running tests. -- `external`: tests that require external services/credentials. - -### Fixture Ownership - -- `tests/conftest.py`: global lightweight fixtures used across all sections. -- `tests/handwriting/conftest.py`: core package-level environment fixtures. -- `tests/handwriting/db/conftest.py`: - - PostgreSQL fixtures (`postgres_engine`, `db_session`) for default data-layer tests. - - optional MongoDB fixture (`mongo_client`) gated by `PYTEST_MONGO_URL` for document-store integration checks. - -### Initial Coverage Added - -- `tests/handwriting/test_core.py` - - settings identity and URL validation behavior. - - logging UI buffer wiring. - - buffer incremental read behavior. - - minimal main module export smoke check. -- `tests/handwriting/db/test_session.py` - - session scope, factory, and engine helper behavior. -- `tests/handwriting/db/test_models.py` - - model defaults and enum/value shape checks. -- `tests/handwriting/db/test_job_repo.py` - - repository list/detail happy path and not-found/missing-image edges. -- `tests/handwriting/db/test_service_job.py` - - job creation happy path and validation errors for unsupported type / oversized payload. - -## Run Commands - -- Collect only: `uv run pytest --collect-only -q` -- Fast local path: `uv run pytest -m unit -q` -- DB path: `uv run pytest -m "db or integration" -q` -- Full path: `uv run pytest -q` - -## Current Verification Snapshot - -- `uv run pytest --collect-only -q` -> 19 tests collected. -- `uv run pytest -m unit -q` -> 11 passed, 8 deselected. -- `uv run pytest -m "db or integration" -q` -> 7 passed, 12 deselected. - -## Jobs Page Diagnostics - -To investigate cases where `/jobs` renders without a table, the scaffold now includes focused service and UI coverage: - -- `tests/handwriting/services/test_ui_jobs.py` - - verifies the `list_jobs_for_ui` contract used by the page. - - includes the join edge case where a job with a missing image relation does not appear in list output. -- `tests/handwriting/ui/test_jobs_page_runtime.py` - - verifies runtime guard behavior for missing app state (`settings`, `session_factory`). -- `tests/handwriting/ui/test_jobs_page_rendering.py` - - includes a detector test for initial render behavior on `/jobs`. - - verifies empty, table, and error rendering branches after refresh. -- `tests/handwriting/ui/test_jobs_page_smoke.py` - - confirms route registration and refresh button wiring. - -Targeted commands: - -- `uv run pytest tests/handwriting/ui/test_jobs_page_rendering.py -q` -- `uv run pytest -m "unit or integration" -q` -- `uv run pytest -m smoke -k jobs -q` -- `HANDWRITING_E2E_BASE_URL=http://localhost uv run pytest tests/handwriting/ui/test_jobs_page_browser_smoke.py -q` - -## Next Pass - -The next pass can expand each major section into markers, fixtures, and test case placeholders. - -## Glossary - -- Contract mapping: Verifying that adapter input/output shapes match expected boundaries. -- Document-store tests: Tests that validate behavior against document-oriented persistence components. -- Marker taxonomy: The test marker classification scheme used to select test slices. -