From ec61013b47ce0fccd305bf11e2f11c51e38765c0 Mon Sep 17 00:00:00 2001 From: Jim Lancaster <40281233+zoltan57@users.noreply.github.com> Date: Thu, 2 Jul 2026 12:37:32 -0500 Subject: [PATCH] Used co-pilot for complete review of all documentation, including extensive revision of v1.md --- docs/architecture.md | 17 +- docs/error_handling.md | 6 +- docs/index.md | 15 +- docs/mvp/mvp-step1.md | 572 ---------------------------------- docs/mvp/mvp-step2.md | 278 ----------------- docs/mvp/mvp-step3.md | 236 -------------- docs/mvp/mvp-step4.md | 262 ---------------- docs/mvp/mvp-step5-results.md | 70 ----- docs/mvp/mvp-step5.md | 310 ------------------ docs/mvp/mvp-step6-results.md | 50 --- docs/mvp/mvp-step6.md | 229 -------------- docs/mvp/mvp-step7-results.md | 134 -------- docs/mvp/mvp-step7.md | 267 ---------------- docs/mvp/mvp.md | 209 ------------- docs/requirements.md | 15 +- docs/schema.md | 14 +- docs/ver1/ver1.md | 365 +++++++--------------- 17 files changed, 155 insertions(+), 2894 deletions(-) delete mode 100644 docs/mvp/mvp-step1.md delete mode 100644 docs/mvp/mvp-step2.md delete mode 100644 docs/mvp/mvp-step3.md delete mode 100644 docs/mvp/mvp-step4.md delete mode 100644 docs/mvp/mvp-step5-results.md delete mode 100644 docs/mvp/mvp-step5.md delete mode 100644 docs/mvp/mvp-step6-results.md delete mode 100644 docs/mvp/mvp-step6.md delete mode 100644 docs/mvp/mvp-step7-results.md delete mode 100644 docs/mvp/mvp-step7.md delete mode 100644 docs/mvp/mvp.md diff --git a/docs/architecture.md b/docs/architecture.md index 9fb442a..11016af 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -20,7 +20,7 @@ Current scope includes: - content source upload and metadata capture - asynchronous transcription jobs - prompt-library driven transcription behavior, with one Markdown file per prompt -- transcript review and revision history +- original transcription review and optional revision history - full-text search over accepted transcripts - export of transcript data @@ -134,9 +134,9 @@ Production transcription flow: 1. A user uploads one or more content sources through the UI or API. 2. The application validates payloads and creates document, source, and job records. 3. The in-process worker de-queues the job and calls the transcription provider. -4. The application persists transcript output, confidence metadata, and provenance events. +4. The application persists original transcription output on the job, plus confidence metadata and provenance events. 5. Job status transitions from queued to processing to transcribed or failed. -6. The UI and API expose status, revision history, and searchable transcript text. +6. The UI and API expose status, optional revision history, and searchable transcription text. ## Data Model Ownership @@ -147,6 +147,15 @@ System-of-record entities: - transcript revisions - provenance metadata +### Original Transcription And Revision Ownership + +- each processing job stores the original immutable provider output (`text`) +- provider metadata (`provider`, `model`, `prompt_name`) and failure detail (`error_detail`) are job-owned processing artifacts +- revisions are optional user-authored edits linked to a content source +- a revision can be created from either original `job.text` or a previous revision +- many jobs will have zero revisions; revision history is additive and never overwrites original provider output +- a document groups one or more content sources (images, PDFs, and future source types) + Storage strategy: - PostgreSQL for relational system-of-record entities @@ -285,6 +294,6 @@ Control: - Port/Interface: A stable contract used by application/domain code to call infrastructure implementations. - Prompt artifact: A single Markdown file that defines one transcription prompt and can be revised independently. - Provenance: Metadata that records where generated data came from and how it was produced. -- Revision history: Versioned record of transcript edits over time. +- Revision history: Optional versioned record of user-authored transcription edits over time. - System of record: The authoritative persistent store for canonical data. - Vertical slice: A minimal end-to-end feature path spanning UI/API, application logic, and persistence. diff --git a/docs/error_handling.md b/docs/error_handling.md index 1b65406..73cf0fd 100644 --- a/docs/error_handling.md +++ b/docs/error_handling.md @@ -41,7 +41,7 @@ The system uses stable, implementation-independent categories: | --- | --- | --- | --- | | `validation_error` | Payload or parameter shape/content is invalid | UI/API input validation, service guards | no | | `user_input_error` | User-provided artifact is unacceptable though structurally valid | unsupported file type, empty file, oversized upload | sometimes | -| `not_found_error` | Requested resource does not exist | missing job/document/transcript | no | +| `not_found_error` | Requested resource does not exist | missing job/document/source/revision | no | | `conflict_error` | Requested operation violates current state constraints | invalid state transition | no | | `external_provider_error` | External AI/provider call fails | upstream HTTP/API/provider failures | sometimes | | `infrastructure_transient_error` | Temporary environment issue | network timeout, DB connection reset | yes | @@ -111,7 +111,7 @@ All logged errors must include, where available: - `category` - `operation` (e.g., `upload.submit`, `worker.process_job`, `jobs.refresh`) - `exception_type` -- `job_id`, `document_id` (when relevant) +- `job_id`, `document_id`, `source_id` (when relevant) - UTC timestamp Rules: @@ -271,7 +271,7 @@ Change requirements: - [System overview](index.md) - [Architecture](architecture.md) - [Requirements](requirements.md) -- [Intent](intent.md) +- [Intent](Intent.md) ## Glossary diff --git a/docs/index.md b/docs/index.md index f7c4390..e78c1e6 100644 --- a/docs/index.md +++ b/docs/index.md @@ -4,9 +4,7 @@ This project is a production application for transcribing and preserving histori ## Start Here -Read [architecture.md](architecture.md) first. - -Then review [ver1/ver1.md](ver1/ver1.md) for completion scope and [ver1/ver1-step1-results.md](ver1/ver1-step1-results.md) for current architecture-consolidation status. +Read [architecture.md](architecture.md) first, then review [ver1/ver1.md](ver1/ver1.md). The architecture page is the primary technical reference and defines: @@ -17,14 +15,15 @@ The architecture page is the primary technical reference and defines: ## What The Application Does -At a high level, users upload images or pdfs as sources of handwritten, typed, or typeset documents, run asynchronous transcription jobs, review and edit transcript revisions, and search across accepted text. +At a high level, users upload images or PDFs as content sources for handwritten, typed, or typeset documents, run asynchronous transcription jobs, review optional revisions, and search across accepted text. Core capabilities: -- document upload and metadata capture +- document grouping with one or more content sources and metadata capture - asynchronous transcription with visible job status +- immutable original transcription persisted with each job (plus provider/model/prompt metadata) - transcription prompt management with one Markdown file per prompt for human refinement over time -- revision history for transcript edits +- optional revision history for user-authored edits (from original text or prior revisions) - full-text search over accepted transcripts - export of transcript data @@ -43,12 +42,10 @@ This operating model keeps deployment and maintenance simple while preserving cl - Architecture and technical design: [architecture.md](architecture.md) - Version 1 implementation plan: [ver1/ver1.md](ver1/ver1.md) -- Version 1 Step 1 plan: [ver1/ver1-step1.md](ver1/ver1-step1.md) -- Version 1 Step 1 results: [ver1/ver1-step1-results.md](ver1/ver1-step1-results.md) - Architecture decision records (ADR index): [adr/README.md](adr/README.md) - Runtime and deployment requirements: [requirements.md](requirements.md) - Error handling policy and operational guidance: [error_handling.md](error_handling.md) -- Domain context and transcription policy: [intent.md](intent.md) +- Domain context and transcription policy: [Intent.md](Intent.md) ## Glossary diff --git a/docs/mvp/mvp-step1.md b/docs/mvp/mvp-step1.md deleted file mode 100644 index c9693c2..0000000 --- a/docs/mvp/mvp-step1.md +++ /dev/null @@ -1,572 +0,0 @@ -# Step 1 Implementation Plan: `config.py` + `models.py` + `db.py` - -## Purpose - -Establish the foundational data layer and configuration system that every subsequent MVP step builds on. At the end of this step, the project has a runnable Python package with a validated schema, typed configuration, and a test suite proving the data layer works — before any UI, worker, or AI provider code exists. - ---- - -## 1. Prerequisite: Project Structure Scaffolding - -Before writing any logic, create the package skeleton so imports work correctly. - -### Files to create (empty `__init__.py` stubs) - -``` -src/ -└── transcription/ - ├── __init__.py - ├── providers/ - │ └── __init__.py - ├── services/ - │ └── __init__.py - └── ui/ - └── __init__.py -``` - -### Files to create (with logic — the Step 1 deliverables) - -``` -src/transcription/config.py -src/transcription/models.py -src/transcription/db.py -``` - -### Test files to create - -``` -tests/ -├── __init__.py -├── conftest.py -├── test_config.py -├── test_models.py -└── test_db.py -``` - -### Update `pyproject.toml` - -Add the dependencies that Step 1 requires and won't change later: - -```toml pyproject.toml -[project] -name = "transcription" -version = "0.1.0" -description = "Historical document transcription system" -readme = "README.md" -requires-python = ">=3.12" -dependencies = [ - "openrouter>=0.7.0", - "pydantic>=2.13.4", - "pydantic-settings>=2.9.1", - "sqlmodel>=0.0.25", -] - -[project.optional-dependencies] -dev = [ - "pytest>=8.0", - "pytest-asyncio>=0.25", -] - -[tool.pytest.ini_options] -addopts = "--strict-markers -q" -markers = [ - "unit: pure logic tests with no external dependencies", - "integration: tests that touch framework or database contracts", - "external: tests that call external services (slow, requires credentials)", -] -``` - -Key additions: -- **`openrouter`** — official OpenRouter Python SDK used for model calls -- **`pydantic-settings`** — for `BaseSettings` with env-var loading (this was split out of `pydantic` core in v2) -- **`sqlmodel`** — provides SQLModel (which bundles SQLAlchemy + Pydantic model integration) and the SQLite driver -- **`pytest` + `pytest-asyncio`** — in `dev` extras for test execution -- **`[tool.pytest.ini_options]`** — strict marker checking enabled from the start; markers registered upfront per pytesting skill conventions - -### Delete `hello.py` - -The placeholder file is no longer needed. - ---- - -## 2. `config.py` — Centralized Configuration - -**Satisfies:** REQ-8 (centralized config and logging at startup) - -### Design Decisions - -| Decision | Rationale | -|----------|-----------| -| Use `pydantic-settings` `BaseSettings` | Type-safe, validates on construction, loads from env vars and `.env` files automatically | -| `PROVIDER` constrained to `openrouter` for MVP | Keeps configuration explicit while avoiding premature multi-provider complexity | -| `OPENROUTER_API_KEY` required | Matches official SDK docs and avoids ambiguous provider-agnostic naming | -| `PROVIDER_MODEL` defaults to `None` | OpenRouter adapter (Step 3) supplies a sensible default when `None` | -| `OPENROUTER_HTTP_REFERER` and `OPENROUTER_APP_TITLE` optional | Matches SDK optional app-attribution fields | -| `DATABASE_URL` defaults to SQLite | Zero-setup local development; PostgreSQL swap is a single env-var change post-MVP | -| `UPLOAD_DIR` and `PROMPT_DIR` as `Path` objects | Enables `.mkdir(parents=True, exist_ok=True)` and path validation at startup | -| Logging configured via `logging.config.dictConfig` in `setup_logging()` | Centralized, explicit formatter/handler/root logger topology; called once at startup with `disable_existing_loggers=False` | - -### Proposed Implementation - -```python src/transcription/config.py -"""Centralized application configuration. - -All settings are loaded from environment variables (or a .env file) -once at startup. Provider-specific defaults (model names, base URLs) -are resolved by the provider adapters, not here. -""" - -from enum import StrEnum -from functools import lru_cache -from pathlib import Path -import logging -import logging.config - -from pydantic_settings import BaseSettings, SettingsConfigDict - - -class Provider(StrEnum): - OPENROUTER = "openrouter" - - -class Settings(BaseSettings): - model_config = SettingsConfigDict( - env_file=".env", - env_file_encoding="utf-8", - extra="ignore", - ) - - # --- AI provider --- - provider: Provider = Provider.OPENROUTER - openrouter_api_key: str - provider_model: str | None = None - openrouter_http_referer: str | None = None - openrouter_app_title: str | None = None - - # --- persistence --- - database_url: str = "sqlite:///./transcription.db" - - # --- filesystem paths --- - upload_dir: Path = Path("./uploads") - prompt_dir: Path = Path("./prompts") - - -LOGGING_CONFIG: dict[str, object] = { - "version": 1, - "disable_existing_loggers": False, - "formatters": { - "standard": { - "format": "%(asctime)s | %(levelname)-8s | %(name)s | %(message)s", - "datefmt": "%Y-%m-%d %H:%M:%S", - } - }, - "handlers": { - "console": { - "class": "logging.StreamHandler", - "formatter": "standard", - "stream": "ext://sys.stdout", - } - }, - "root": { - "level": "INFO", - "handlers": ["console"], - }, -} - - -@lru_cache(maxsize=1) -def get_settings() -> Settings: - """Return the singleton Settings instance. - - Cached so the entire application shares one validated config. - """ - return Settings() - - -def setup_logging() -> None: - """Configure root logging once at startup.""" - logging.config.dictConfig(LOGGING_CONFIG) -``` - -### Key Behaviors - -- **Startup validation**: If `OPENROUTER_API_KEY` is missing from the environment, `Settings()` raises a `ValidationError` immediately — the app won't start with a missing key. -- **`.env` support**: Developers can create a `.env` file in the project root for local keys; it's never committed (already covered by the existing `.gitignore` pattern or a new entry). -- **`extra="ignore"`**: Unknown env vars don't cause errors, keeping the config resilient to unrelated environment variables. -- **`lru_cache`**: `get_settings()` is the single access point. All modules import and call this function rather than constructing `Settings` directly. -- **Centralized logging**: `setup_logging()` calls `dictConfig` exactly once at startup; all modules should use `logging.getLogger(__name__)` and avoid `basicConfig`. - -### `.env` template (not committed — add to `.gitignore`) - -```bash .env.example -PROVIDER=openrouter -OPENROUTER_API_KEY=sk-or-... -# PROVIDER_MODEL= # optional: OpenRouter adapter supplies default -# OPENROUTER_HTTP_REFERER=https://example.com -# OPENROUTER_APP_TITLE=Historical Transcription MVP -# DATABASE_URL=sqlite:///./transcription.db -# UPLOAD_DIR=./uploads -# PROMPT_DIR=./prompts -``` - -### `.gitignore` addition - -```gitignore .gitignore -# ... existing entries ... - -# Environment secrets -.env -``` - ---- - -## 3. `models.py` — SQLModel Domain Models - -**Satisfies:** REQ-3 (persist and expose job states), REQ-4 (persist transcription output and failure details) - -### Design Decisions - -| Decision | Rationale | -|----------|-----------| -| Three models: `Document`, `Job`, `Transcript` | Minimal set from MVP Feature 5. One-to-many from Document→Job and one-to-one from Job→Transcript | -| `JobStatus` as a `StrEnum` | Readable in the database (`"queued"` not `1`), type-safe in Python, trivially serializable to JSON for the UI | -| Status values: `queued`, `processing`, `transcribed`, `failed` | Matches MVP Feature 2 lifecycle. REQ-3 also lists `upload` and `completed` — these are deferred to post-MVP when revision/review workflows exist | -| UUIDs for primary keys | Avoids auto-increment collision concerns if we later move to PostgreSQL; safe for distributed ID generation; `uuid4` is simple | -| `uploaded_at`, `created_at`, `updated_at` as UTC `datetime` | Timezone-naive UTC by convention for MVP. Sufficient for single-user, single-timezone operation | -| `Transcript.text` is nullable | A failed job creates a Transcript with `text=None` and `error_detail` populated, keeping the query model uniform | -| Relationships via SQLModel `Relationship` | Enables `document.jobs` and `job.transcript` navigation in service code without manual joins | - -### Proposed Implementation - -- `resource://skills/fastapi-async-sqlalchemy-modernization/document` - -```python src/transcription/models.py -"""SQLModel domain models for the transcription system. - -Three models capture the MVP lifecycle: - Document → one-to-many → Job → one-to-one → Transcript -""" - -from datetime import datetime, timezone -from enum import StrEnum -from uuid import UUID, uuid4 - -from sqlmodel import Field, Relationship, SQLModel - - -class JobStatus(StrEnum): - QUEUED = "queued" - PROCESSING = "processing" - TRANSCRIBED = "transcribed" - FAILED = "failed" - - -class Document(SQLModel, table=True): - """An uploaded document image.""" - - id: UUID = Field(default_factory=uuid4, primary_key=True) - filename: str - file_path: str - uploaded_at: datetime = Field( - default_factory=lambda: datetime.now(timezone.utc), - ) - - # --- relationships --- - jobs: list["Job"] = Relationship(back_populates="document") - - -class Job(SQLModel, table=True): - """A transcription job tied to a single document.""" - - id: UUID = Field(default_factory=uuid4, primary_key=True) - document_id: UUID = Field(foreign_key="document.id") - status: JobStatus = Field(default=JobStatus.QUEUED) - created_at: datetime = Field( - default_factory=lambda: datetime.now(timezone.utc), - ) - updated_at: datetime = Field( - default_factory=lambda: datetime.now(timezone.utc), - ) - - # --- relationships --- - document: Document = Relationship(back_populates="jobs") - transcript: "Transcript | None" = Relationship(back_populates="job") - - -class Transcript(SQLModel, table=True): - """The output of a transcription job.""" - - id: UUID = Field(default_factory=uuid4, primary_key=True) - job_id: UUID = Field(foreign_key="job.id", unique=True) - text: str | None = None - error_detail: str | None = None - created_at: datetime = Field( - default_factory=lambda: datetime.now(timezone.utc), - ) - - # --- relationships --- - job: Job = Relationship(back_populates="transcript") -``` - -### Entity-Relationship Summary - -``` -┌──────────┐ ┌──────────┐ ┌─────────────┐ -│ Document │ 1───* │ Job │ 1───1 │ Transcript │ -├──────────┤ ├──────────┤ ├─────────────┤ -│ id (PK) │ │ id (PK) │ │ id (PK) │ -│ filename │ │ doc_id │──FK──▶│ job_id (FK) │ -│ file_path│ │ status │ │ text │ -│ uploaded │ │ created │ │ error_detail│ -│ │ │ updated │ │ created │ -└──────────┘ └──────────┘ └─────────────┘ -``` - -### Why Only Four Status Values - -REQ-3 lists six states: `upload`, `queued`, `processing`, `transcribed`, `failed`, `completed`. The MVP simplifies this: - -| REQ-3 State | MVP Treatment | -|-------------|---------------| -| `upload` | Implicit — the Document record exists before a Job is created. No separate job state needed. | -| `queued` | ✅ Included — job created, waiting for worker pickup | -| `processing` | ✅ Included — worker is actively transcribing | -| `transcribed` | ✅ Included — AI output received and stored | -| `failed` | ✅ Included — error captured | -| `completed` | Deferred — implies human review/acceptance. In MVP, `transcribed` is the terminal success state. | - ---- - -## 4. `db.py` — Database Engine and Session Management - -**Satisfies:** MVP Feature 5 (SQLite auto-created on first startup) - -### Design Decisions - -| Decision | Rationale | -|----------|-----------| -| Module-level `create_engine` + `Session` factory | REQ-7 (lifespan-owned resources) is deferred. A module-level engine is adequate for MVP's single-process, single-user operation | -| `create_all()` as an explicit function | Called at app startup. MVP auto-creates tables (REQ-10 deferred), but the function is isolated so it's easy to gate behind a flag later | -| `get_session()` as a generator | Standard FastAPI/SQLModel pattern — yields a session, ensures cleanup. Compatible with `Depends()` when the API layer arrives in Step 5 | -| `echo=False` default | Keeps logs clean. Can be toggled for debugging | - -### Proposed Implementation - -```python src/transcription/db.py -"""Database engine, session factory, and schema bootstrap. - -MVP uses SQLite with auto-create-tables at startup. -PostgreSQL migration is a post-MVP configuration change. -""" -import contextlib -from collections.abc import Generator - -from sqlmodel import Session, SQLModel, create_engine - -from transcription.config import get_settings - - -def _build_engine(): - settings = get_settings() - connect_args = {} - if settings.database_url.startswith("sqlite"): - connect_args["check_same_thread"] = False - return create_engine( - settings.database_url, - echo=False, - connect_args=connect_args, - ) - - -engine = _build_engine() - - -def create_all() -> None: - """Create all tables. Called once at application startup.""" - SQLModel.metadata.create_all(engine) - - -@contextlib.contextmanager -def get_session() -> Generator[Session]: - """Yield a database session and ensure cleanup.""" - with Session(engine) as session: - yield session -``` - -### SQLite-Specific Note - -`check_same_thread=False` is required for SQLite when the session may be accessed from different threads (e.g., a background worker on a different thread than the request handler). This setting is harmless and ignored for PostgreSQL connection strings. - ---- - -## 5. Test Plan - -Refer to these resources for rules and guidelines about structure: - -- `resource://skills/pytesting/document` -- `resource://catalog/prompts/pytest-scaffold` -- `resource://catalog/prompts/pytest-fill-scaffold` - -Hierarchy pattern used in this step: - -```text -tests/ - conftest.py - test_config.py - TestSettingsLoading - test_loads_from_env - test_requires_api_key - TestProviderSettings - test_defaults_to_openrouter - test_rejects_invalid_value - test_optional_fields_default_to_none - TestPathSettings - test_path_fields_are_path_objects - test_models.py - TestDocumentModel - test_can_be_persisted - test_defaults_are_populated - TestJobModel - test_can_be_created_for_document - test_defaults_are_populated - test_transitions_to_transcribed - test_transitions_to_failed - TestTranscriptModel - test_success_record_persists - test_failure_record_persists - test_job_id_is_unique - TestRelationships - test_document_exposes_jobs - test_job_exposes_transcript - test_db.py - TestSchemaBootstrap - test_create_all_creates_expected_tables - TestSessionFactory - test_get_session_yields_session - test_session_is_closed_after_generator_exit -``` - -### `tests/conftest.py` — Shared Fixtures - -```python tests/conftest.py -"""Shared test fixtures. - -Every test gets a fresh in-memory SQLite database so tests are -isolated, fast, and leave no artifacts on disk. -""" - -import pytest -from sqlmodel import Session, SQLModel, create_engine -from sqlmodel.pool import StaticPool - - -@pytest.fixture -def session(): - """Provide a clean database session for each test.""" - engine = create_engine( - "sqlite://", - connect_args={"check_same_thread": False}, - poolclass=StaticPool, - ) - SQLModel.metadata.create_all(engine) - with Session(engine) as session: - yield session -``` - -`StaticPool` ensures a single in-memory SQLite connection is shared across threads, which is required when `TestClient` (Step 5) spawns threads that would otherwise get separate in-memory databases. Establishing it now keeps the fixture stable across all future steps. - -### `tests/test_config.py` — Configuration Hierarchy - -| Class | Method | What It Verifies | -|------|--------|------------------| -| `TestSettingsLoading` | `test_loads_from_env` | `Settings` constructs successfully when `OPENROUTER_API_KEY` is set via env var | -| `TestSettingsLoading` | `test_requires_api_key` | `Settings()` raises `ValidationError` when `OPENROUTER_API_KEY` is missing | -| `TestProviderSettings` | `test_defaults_to_openrouter` | Default provider is `openrouter` when not explicitly set | -| `TestProviderSettings` | `test_rejects_invalid_value` | Setting `PROVIDER=invalid` raises `ValidationError` | -| `TestProviderSettings` | `test_optional_fields_default_to_none` | `provider_model`, `openrouter_http_referer`, and `openrouter_app_title` are `None` when unset | -| `TestPathSettings` | `test_path_fields_are_path_objects` | `upload_dir` and `prompt_dir` are `Path` instances | - -### `tests/test_models.py` — Model & Relationship Hierarchy - -| Class | Method | What It Verifies | -|------|--------|------------------| -| `TestDocumentModel` | `test_can_be_persisted` | A `Document` can be persisted and read back with correct fields | -| `TestDocumentModel` | `test_defaults_are_populated` | `id` is auto-generated UUID, `uploaded_at` is populated | -| `TestJobModel` | `test_can_be_created_for_document` | A `Job` linked to a `Document` via FK persists correctly | -| `TestJobModel` | `test_defaults_are_populated` | Default status is `queued`, `created_at` and `updated_at` are populated | -| `TestJobModel` | `test_transitions_to_transcribed` | Status can be updated from `queued` → `processing` → `transcribed` | -| `TestJobModel` | `test_transitions_to_failed` | Status can be updated from `processing` → `failed` | -| `TestTranscriptModel` | `test_success_record_persists` | A `Transcript` with `text` set and `error_detail=None` persists correctly | -| `TestTranscriptModel` | `test_failure_record_persists` | A `Transcript` with `text=None` and `error_detail` set persists correctly | -| `TestRelationships` | `test_document_exposes_jobs` | `document.jobs` returns the linked `Job` list | -| `TestRelationships` | `test_job_exposes_transcript` | `job.transcript` returns the linked `Transcript` | -| `TestTranscriptModel` | `test_job_id_is_unique` | Inserting two transcripts with the same `job_id` raises an integrity error | - -### `tests/test_db.py` — Database Bootstrap Hierarchy - -| Class | Method | What It Verifies | -|------|--------|------------------| -| `TestSchemaBootstrap` | `test_create_all_creates_expected_tables` | After `create_all()`, the expected tables (`document`, `job`, `transcript`) exist in the database | -| `TestSessionFactory` | `test_get_session_yields_session` | `get_session()` yields a usable `Session` object | -| `TestSessionFactory` | `test_session_is_closed_after_generator_exit` | After the generator is exhausted, the session is closed | - -### Marker Strategy (Step 1) - -- Markers (`unit`, `integration`, `external`) are registered upfront in `pyproject.toml` with `--strict-markers` enabled, per pytesting skill conventions. -- All Step 1 tests are unmarked — they run in the default lane since they are fast, deterministic, and have no external dependencies. -- When slower integration or external tests are introduced in later steps, apply explicit markers and keep test names unchanged. - -### Test Workflow - -Follow the two-phase approach from `resource://catalog/prompts/pytest-scaffold` and `resource://catalog/prompts/pytest-fill-scaffold`: - -1. **Scaffold phase**: Create test files with class hierarchy, method names, and one-line docstrings only. Validate collection: - - `uv run pytest --collect-only -q` -2. **Fill phase**: Implement assertions, fixtures, and minimal test data. Treat scaffolded names and docstrings as locked. Validate execution: - - `uv run pytest -q` - -Scaffolded structure is treated as a stable baseline — do not rename, move, merge, split, or re-nest tests once the scaffold is reviewed. - ---- - -## 6. Step 1 Completion Checklist - -When all of the following are true, Step 1 is done and Step 2 can begin: - -| # | Criterion | How to Verify | -|---|-----------|---------------| -| 1 | `src/transcription/` package exists with `config.py`, `models.py`, `db.py` | `ls` / file inspection | -| 2 | Empty `__init__.py` stubs exist for `providers/`, `services/`, `ui/` | `ls` / file inspection | -| 3 | `Settings` loads from environment and validates `OPENROUTER_API_KEY` is present | `test_config.py` passes | -| 4 | `Document`, `Job`, `Transcript` models create tables in SQLite | `test_models.py` passes | -| 5 | `JobStatus` enum has exactly four values: `queued`, `processing`, `transcribed`, `failed` | `test_models.py` passes | -| 6 | Foreign key relationships work: Document→Job→Transcript | `test_models.py` passes | -| 7 | `create_all()` bootstraps the schema; `get_session()` yields a working session | `test_db.py` passes | -| 8 | All tests pass: `uv run pytest -q` | CI / local run | -| 9 | `hello.py` is deleted | File inspection | -| 10 | `pyproject.toml` includes `openrouter`, `sqlmodel`, `pydantic-settings`, `pytest`, `pytest-asyncio` | File inspection | -| 10a | `pyproject.toml` has `[tool.pytest.ini_options]` with `--strict-markers` and registered markers | File inspection | -| 11 | `.env.example` documents all config vars; `.env` is in `.gitignore` | File inspection | -| 12 | `setup_logging()` uses `logging.config.dictConfig` with centralized formatter/handler/root config | File inspection | -| 13 | `uv run pytest --collect-only -q` shows expected test hierarchy | Local run | -| 14 | `uv run pytest -q` passes all tests | Local run | - ---- - -## 7. What This Step Does NOT Include - -Explicitly out of scope to prevent scope creep: - -| Excluded | Reason | -|----------|--------| -| FastAPI / NiceGUI app entrypoint | Step 5 | -| Additional provider adapters beyond OpenRouter | Post-MVP | -| Upload service logic | Step 4 | -| Worker / background processing | Step 4 | -| Transcription prompt files | Step 2 | -| Alembic or migration tooling | Post-MVP (REQ-10 deferred) | -| Async session factory | Post-MVP (REQ-7 deferred) | - ---- - -This plan produces a fully tested, importable data foundation. Every subsequent step imports from `transcription.config`, `transcription.models`, and `transcription.db` without modification. \ No newline at end of file diff --git a/docs/mvp/mvp-step2.md b/docs/mvp/mvp-step2.md deleted file mode 100644 index 091e81c..0000000 --- a/docs/mvp/mvp-step2.md +++ /dev/null @@ -1,278 +0,0 @@ -## Step 2: prompts/transcribe_document.md - -### Goal - -Implement the MVP prompt artifact system by creating a curated transcription prompt file: - -- `prompts/transcribe_document.md` - -This step primarily satisfies: - -- **REQ-12**: prompts stored as individual Markdown artifacts -- MVP Feature 3: prompt-driven verbatim transcription behavior grounded in `docs/intent.md` - ---- - -## Scope for Step 2 - -### In scope -1. Create prompt artifact directory and first prompt file. -2. Encode transcription rules from `docs/intent.md` into a model-facing prompt. -3. Define stable prompt structure so future revisions are easy to diff/review. -4. Add lightweight tests that validate artifact presence and baseline quality constraints. -5. Update docs/README references so Step 3 can consume prompt file directly. - -### Out of scope -- Provider integration logic (Step 3) -- Worker/job orchestration (Step 4) -- UI behavior (Step 5) - ---- - -## Proposed Deliverables - -1. **`prompts/transcribe_document.md`** - - production prompt text for historical document transcription - -2. **`prompts/README.md`** (recommended) - - conventions for prompt files, revision policy, naming - -3. **`tests/test_prompts.py`** (recommended) - - artifact existence + structure checks - -4. **Small docs update** (README or docs reference) - - indicate that prompts are file-based and loaded from `PROMPT_DIR` - ---- - -## Detailed Work Breakdown - -### 1) Create prompt artifact folder and canonical file -- Add `prompts/` at repo root. -- Add `transcribe_document.md` as the first curated artifact. -- Keep filename stable; this becomes the default in Step 3 unless overridden. - -### 2) Author prompt content using a strict, sectioned format -Use section headers so future diffs are clean and policy changes are isolated. - -Suggested sections: - -1. **Purpose** - - verbatim scholarly transcription of historical documents - -2. **Output requirements** - - plain text only - - no summaries, no paraphrasing - - preserve reading order and meaningful structure - -3. **Core fidelity rules** - - preserve original wording and punctuation - - don’t silently normalize grammar/spelling - - no invented content - -4. **Issue-handling rules (mapped from Intent table)** - - misspellings with `[sic]` - - missing words with `[word]` - - uncertainty with `[guess?]` - - illegible with `[illegible]` / reason tags - - crossed-out text as `[deleted: ...]` - - inserted text as `[inserted: ...]` - - superscripts handling guidance - - non-text elements as `[description]` - - marginalia format `[written in left margin: ...]` - - line-break hyphen rejoin behavior - - capitalization policy - - hierarchical outline preservation (including unusual numbering) - -5. **Confidence/ambiguity policy** - - prefer explicit uncertainty markers over hallucination - -6. **Final self-checklist for model** - - did I preserve structure? - - did I mark uncertain text? - - did I avoid silent corrections? - -### 3) Add prompt-library conventions (`prompts/README.md`) -Recommended conventions: -- one prompt per file -- snake_case names -- each file starts with purpose + behavior contract -- iterative edits, one prompt per PR where possible -- no secrets in prompt files - -### 4) Add tests for prompt assets (`tests/test_prompts.py`) -Keep tests robust but not brittle. - -Recommended tests: -1. `test_prompt_file_exists` -2. `test_prompt_file_is_not_empty` -3. `test_prompt_mentions_verbatim_behavior` -4. `test_prompt_includes_uncertainty_and_illegible_markers` -5. `test_prompt_includes_deleted_and_inserted_conventions` - -Avoid exact full-text matching; verify key semantic anchors only. - -### 5) Optional config alignment check -Current config already has: -- `prompt_dir: Path = Path("./prompts")` - -In Step 2, ensure docs reflect this and that Step 3 will resolve: -- `PROMPT_DIR / "transcribe_document.md"` - ---- - -## Task-by-Task Execution Checklist - -## Phase A — Scaffold files - -- [ ] **A1. Create prompt directory** - - Path: `prompts/` - - Verify: directory exists at repo root - -- [ ] **A2. Create canonical prompt file** - - Path: `prompts/transcribe_document.md` - - Verify: file exists and is non-empty - -- [ ] **A3. (Recommended) Create prompt library README** - - Path: `prompts/README.md` - - Verify: includes naming + revision conventions - ---- - -## Phase B — Author prompt content (core work) - -- [ ] **B1. Add Purpose section** - - States verbatim historical transcription objective - - Explicitly disallows summarization/paraphrase - -- [ ] **B2. Add Output Contract section** - - Plain text output expectation - - Preserve meaningful structure and reading order - - No fabricated text - -- [ ] **B3. Add Rule Set from `docs/intent.md`** - - Misspellings/errors: `[sic]` - - Missing words: `[word]` - - Uncertain readings: `[guess?]` - - Illegible regions: `[illegible]` / reason labels - - Crossed-out text: `[deleted: ...]` - - Squeezed-in text: `[inserted: ...]` - - Superscripts/abbrev handling guidance - - Non-text visuals: bracketed descriptive labels - - Marginalia formatting cue - - Rejoin line-break hyphenated words silently - - Ambiguous capitalization policy - - Hierarchical outline numbering preservation - -- [ ] **B4. Add Ambiguity and Confidence policy** - - “Mark uncertainty instead of guessing” - - “Never silently normalize uncertain passages” - -- [ ] **B5. Add Final Self-Check section** - - Checklist for fidelity, uncertainty labeling, and format compliance - ---- - -## Phase C — Add validations (tests) - -- [ ] **C1. Create prompt tests file** - - Path: `tests/test_prompts.py` - -- [ ] **C2. Add existence/health checks** - - Prompt file exists - - Prompt file has content (non-whitespace) - -- [ ] **C3. Add semantic anchor checks** - - Mentions verbatim behavior - - Mentions uncertainty marker pattern (`?` in brackets conceptually) - - Mentions illegible handling - - Mentions deleted/inserted conventions - -- [ ] **C4. Keep tests resilient** - - Avoid exact full-file snapshot assertions - - Assert required concepts, not precise phrasing - ---- - -## Phase D — Documentation alignment - -- [ ] **D1. Update top-level docs/README reference** - - Mention that prompts live in `prompts/` - - Mention Step 3 loads from `PROMPT_DIR` - -- [ ] **D2. Confirm config compatibility** - - `src/transcription/config.py` already uses `prompt_dir = Path("./prompts")` - - No code change needed unless naming/path mismatch appears - ---- - -## Phase E — Verification - -- [ ] **E1. Run targeted test file** - - `uv run pytest tests/test_prompts.py -q` - -- [ ] **E2. Run full suite** - - `uv run pytest -q` - -- [ ] **E3. Confirm no regressions** - - All existing tests still green (expected: previous 20 + new prompt tests) - ---- - -## Phase F — Commit plan (recommended granularity) - -- [ ] **F1. Commit 1: scaffold** - - `prompts/transcribe_document.md` (initial structure) - - `prompts/README.md` (if included) - -- [ ] **F2. Commit 2: finalized prompt content** - - full rule-complete prompt text - -- [ ] **F3. Commit 3: tests + docs alignment** - - `tests/test_prompts.py` - - README/docs mention of prompt artifact pattern - ---- - -## Done Criteria (quick gate) - -- [ ] Canonical prompt exists and is curated for verbatim transcription. -- [ ] Prompt encodes all high-value handling rules from `docs/intent.md`. -- [ ] Prompt tests pass. -- [ ] Full project tests pass with `uv`. -- [ ] Ready for Step 3 provider integration. - ---- - -## Acceptance Criteria (Definition of Done) - -Step 2 is complete when all are true: - -1. `prompts/transcribe_document.md` exists and is committed. -2. Prompt includes all critical handling rules from `docs/intent.md`. -3. Prompt is structured with stable section headings for future curation. -4. Prompt tests pass under `uv run pytest -q`. -5. Existing tests remain green (total suite still passes). -6. Docs indicate prompt artifact location and curation policy. - ---- - -## Risks and Mitigations - -1. **Risk: prompt too vague → hallucinated reconstructions** - - Mitigation: explicit uncertainty/illegible conventions and “no invention” rule. - -2. **Risk: prompt too rigid for mixed document types** - - Mitigation: include neutral defaults + clear annotation formats. - -3. **Risk: brittle tests block iterative prompt tuning** - - Mitigation: test semantic anchors, not exact wording. - ---- - -## Handoff to Step 3 - -After Step 2, Step 3 can immediately: -1. Load `transcribe_document.md` from `PROMPT_DIR` -2. Inject prompt into OpenRouter request -3. Start validating real transcription behavior with minimal glue code diff --git a/docs/mvp/mvp-step3.md b/docs/mvp/mvp-step3.md deleted file mode 100644 index 7e24274..0000000 --- a/docs/mvp/mvp-step3.md +++ /dev/null @@ -1,236 +0,0 @@ -## Step 3: services/transcription.py + providers/ - -### Objective - -Implement the **AI transcription integration layer** so the app can: - -1. Read the curated prompt from `PROMPT_DIR` -2. Send prompt + image to the configured provider (OpenRouter) -3. Return normalized transcription output (or structured failure) - -This corresponds to MVP Step 3 from `docs/mvp.md`: -- `services/transcription.py` -- `providers/` adapter(s) - ---- - -## Scope for Step 3 - -### In scope -- Provider abstraction and OpenRouter adapter -- Prompt file loading utility in service layer -- Image payload preparation -- One high-level transcription service function usable by Step 4 worker -- Unit tests (mocked provider SDK, no external calls) - -### Out of scope -- Job polling/background loop (Step 4) -- DB status transition orchestration in worker loop (Step 4) -- UI invocation/wiring (Step 5) - ---- - -## Planned Deliverables - -### Source files -- `src/transcription/providers/base.py` -- `src/transcription/providers/openrouter.py` -- `src/transcription/providers/__init__.py` (exports + factory) -- `src/transcription/services/transcription.py` -- `src/transcription/services/__init__.py` (optional export) - -### Tests -- `tests/providers/test_openrouter.py` -- `tests/services/test_transcription.py` - -### Test directory convention -- Mirror source domains under `tests/`. -- Provider adapter tests live under `tests/providers/`. -- Service-layer tests live under `tests/services/`. -- Prefer one focused test module per production module (for Step 3: `test_openrouter.py`, `test_transcription.py`). - ---- - -## Design Decisions (before coding) - -1. **Provider interface first** - - Define a stable contract independent of SDK specifics. - - Prevent Step 4 from depending on raw SDK response shapes. - -2. **Service returns normalized result object** - - Include: `text`, `provider`, `model`, `raw_error`/exception metadata. - - Worker can map this cleanly to `Transcript` and `JobStatus`. - -3. **Prompt loaded from file at call time** - - Uses `get_settings().prompt_dir / "transcribe_document.md"`. - - Keeps prompt edits hot-swappable without code changes. - -4. **Clear exception boundary** - - SDK/network/model failures become predictable domain exceptions: - - `ProviderError` - - `PromptLoadError` - - `TranscriptionError` (optional top-level wrapper) - -5. **Model resolution policy** - - Use `settings.provider_model` if set - - Otherwise use adapter default constant (e.g., vision-capable model slug) - ---- - -## Task-by-Task Execution Checklist - -## Phase A — Provider contract - -- [ ] Create `src/transcription/providers/base.py` -- [ ] Define protocol/ABC for transcription providers: - - [ ] method signature accepts prompt text + image bytes (or data URL) + mime type - - [ ] returns normalized text result (and optional metadata) -- [ ] Define shared provider exceptions: - - [ ] `ProviderError` - - [ ] optional subclasses (`ProviderAuthError`, `ProviderResponseError`) - ---- - -## Phase B — OpenRouter adapter - -- [ ] Create `src/transcription/providers/openrouter.py` -- [ ] Implement `OpenRouterTranscriptionProvider` with: - - [ ] config-driven API key usage - - [ ] optional referer/title attribution headers - - [ ] model resolution fallback when `provider_model` is unset -- [ ] Implement request building: - - [ ] prompt included as instruction content - - [ ] image included in supported format for vision call -- [ ] Implement response parsing: - - [ ] extract final transcript text from SDK response - - [ ] validate non-empty text -- [ ] Wrap SDK failures into `ProviderError` with clean message - ---- - -## Phase C — Provider factory - -- [ ] Update `src/transcription/providers/__init__.py` -- [ ] Add `get_transcription_provider()` factory: - - [ ] reads `settings.provider` - - [ ] returns OpenRouter adapter for `openrouter` - - [ ] raises explicit error for unsupported provider values - ---- - -## Phase D — Transcription service (Step 3 core) - -- [ ] Create `src/transcription/services/transcription.py` -- [ ] Add prompt loader function: - - [ ] default file: `transcribe_document.md` - - [ ] raises `PromptLoadError` on missing/empty file -- [ ] Add image loader/validator: - - [ ] path existence check - - [ ] allowed mime detection (`.jpg/.jpeg/.png/.tiff/.pdf` policy aligned to MVP) -- [ ] Add high-level function (name example): - - [ ] `transcribe_document_image(image_path, prompt_name="transcribe_document.md")` - - [ ] loads prompt + image - - [ ] calls provider from factory - - [ ] returns normalized transcription result object -- [ ] Add structured logging at key boundaries: - - [ ] prompt loaded - - [ ] provider invoked - - [ ] success/failure outcome (no sensitive data in logs) - ---- - -## Phase E — Tests (two-phase scaffold -> fill) - -### Required execution resources - -Load and reference these directly during test planning/implementation so the two-phase flow is enforced: - -- [ ] `resource://catalog/prompts/pytest-scaffold` -- [ ] `resource://prompts/pytest-scaffold/document` -- [ ] `resource://catalog/prompts/pytest-fill-scaffold` -- [ ] `resource://prompts/pytest-fill-scaffold/document` - -### Phase E1 — Scaffold test structure first - -Prompt: `resource://catalog/prompts/pytest-scaffold` - -Suggested arguments: -- [ ] `target_modules` = `src/transcription/providers/openrouter.py`, `src/transcription/services/transcription.py` -- [ ] `mode` = `scaffold` -- [ ] `path_strategy` = `src-to-tests-mirror` -- [ ] `naming_style` = `concise-behavior` - -Expected scaffold outcomes: -- [ ] `tests/providers/test_openrouter.py` exists with class/method skeletons and one-line docstrings -- [ ] `tests/services/test_transcription.py` exists with class/method skeletons and one-line docstrings -- [ ] collection succeeds on scaffold-only tests - -Scaffold coverage targets: -- [ ] adapter initializes from settings -- [ ] model fallback when `provider_model is None` -- [ ] referer/title options included when set -- [ ] successful SDK response parses transcript text -- [ ] SDK exception maps to `ProviderError` -- [ ] empty/invalid response maps to `ProviderError` -- [ ] prompt loader reads canonical prompt file -- [ ] missing prompt raises `PromptLoadError` -- [ ] transcription function loads file and calls provider once -- [ ] image path missing raises clear error -- [ ] provider error is propagated/wrapped predictably -- [ ] returned result includes transcript text and metadata - -### Phase E2 — Fill scaffolded tests with assertions - -Prompt: `resource://catalog/prompts/pytest-fill-scaffold` - -Suggested arguments: -- [ ] `target_files` = `tests/providers/test_openrouter.py`, `tests/services/test_transcription.py` -- [ ] `stack` = `pure-python` -- [ ] `strategy` = `minimal` -- [ ] `marker_lane` = `unit` - -Fill constraints: -- [ ] preserve scaffold class/method names and one-line docstrings -- [ ] keep mocks to an absolute minimum; mock only network boundaries and non-deterministic failures -- [ ] keep one behavior target per test method - -> Default suite should remain deterministic and fast, but mocking should be minimal and intentional. - -### Optional real-endpoint validation lane - -- [ ] Add an opt-in integration lane for real provider calls (for example `@pytest.mark.integration` and `@pytest.mark.live_api`). -- [ ] Gate live tests behind explicit env vars (for example `OPENROUTER_API_KEY`, optional `RUN_LIVE_API_TESTS=1`). -- [ ] Exclude live tests from default CI/local runs unless explicitly requested. -- [ ] Keep at least one thin smoke path that can validate request/response compatibility against the real endpoint. - ---- - -## Phase F — Verification commands - -- [ ] E1 scaffold validation: `uv run pytest --collect-only -q` -- [ ] E2 fill validation (unit lane): `uv run pytest -m unit -q` -- [ ] E2 targeted provider file: `uv run pytest tests/providers/test_openrouter.py -q` -- [ ] E2 targeted service file: `uv run pytest tests/services/test_transcription.py -q` -- [ ] E2 final full-suite check: `uv run pytest -q` - ---- - -## Implementation Notes / Guardrails - -- Avoid coupling Step 3 service to DB models directly (that belongs in Step 4 orchestration). -- Do not silently swallow provider errors. -- Keep prompt filename stable (`transcribe_document.md`) unless explicitly parameterized. -- Keep request/response normalization inside provider adapter, not worker/UI layers. - ---- - -## Definition of Done (Step 3) - -Step 3 is done when: - -1. Provider abstraction exists and OpenRouter adapter is implemented. -2. Service can transcribe a local image using prompt file content. -3. Failures are returned as structured exceptions, not raw SDK traceback noise. -4. Unit tests for provider and service pass. -5. Full suite remains green under `uv run pytest -q`. -6. Step 4 can call a single service function to process queued jobs. diff --git a/docs/mvp/mvp-step4.md b/docs/mvp/mvp-step4.md deleted file mode 100644 index 45a0aa9..0000000 --- a/docs/mvp/mvp-step4.md +++ /dev/null @@ -1,262 +0,0 @@ -## Step 4: `services/upload.py` + `worker.py` - -### Objective - -Implement the MVP upload and background-processing pipeline so the system can: - -1. Save uploaded files into `UPLOAD_DIR` -2. Create `Document` + `Job(status="queued")` -3. Process queued jobs in a worker loop: - - `queued -> processing` - - call Step 3 transcription service - - persist `Transcript` - - finalize as `transcribed` or `failed` - -This step advances MVP Feature 1 + Feature 2 and supports REQ-1, REQ-2, REQ-3, REQ-4, REQ-6. - ---- - -## Scope - -### In scope -- `src/transcription/services/upload.py` -- `src/transcription/worker.py` -- Upload persistence logic and initial job creation -- Worker polling and single-job lifecycle execution -- Deterministic test coverage for upload + worker (default suite) - -### Out of scope -- UI integration and pages (Step 5) -- Queue infrastructure beyond in-process loop -- Async DB/session architecture refactor -- Broad production hardening beyond MVP needs - ---- - -## Planned Deliverables - -### Source files -- `src/transcription/services/upload.py` -- `src/transcription/worker.py` -- `src/transcription/services/__init__.py` (export updates as needed) - -### Test files -- `tests/services/test_upload.py` -- `tests/services/test_worker.py` - -### Optional external lane (already present pattern) -- reuse `external` marker for live-provider checks where appropriate -- keep external out of default lane - ---- - -## Required MCP Prompt References (for test workflow) - -Apply these resources directly during Step 4 test creation: - -1. `resource://catalog/prompts/pytest-scaffold` -2. `resource://prompts/pytest-scaffold/document` -3. `resource://catalog/prompts/pytest-fill-scaffold` -4. `resource://prompts/pytest-fill-scaffold/document` - -And (as referenced by those prompts) apply relevant pytest skill references for: -- naming/hierarchy -- marker defaults -- SQLAlchemy sync testing behavior where applicable - ---- - -## Design Decisions - -1. **Upload service owns initial file + record creation** - - Writes file, creates `Document`, creates queued `Job`, returns IDs/path. - -2. **Worker owns lifecycle transitions** - - Worker is the single owner of `queued -> processing -> terminal` job state changes. - -3. **Worker uses Step 3 service boundary** - - Worker calls `transcribe_document_image(...)`; no provider-specific SDK logic in worker. - -4. **Failure information is always persisted** - - On failure: store `Transcript(text=None, error_detail=...)` and set `Job.status=failed`. - -5. **Loop remains simple and stoppable** - - In-process polling loop with stop event and poll interval for MVP simplicity and testability. - ---- - -## Task-by-Task Execution Checklist - -## Phase A — Implement upload service (`src/transcription/services/upload.py`) - -- [ ] Create `UploadError` exception -- [ ] Create `UploadJobResult` dataclass with: - - [ ] `document_id` - - [ ] `job_id` - - [ ] `stored_path` - - [ ] `original_filename` -- [ ] Add filename safety handling: - - [ ] normalize to basename - - [ ] avoid path traversal - - [ ] collision-safe stored name (e.g., UUID prefix/suffix) -- [ ] Validate upload payload: - - [ ] non-empty bytes required - - [ ] extension in supported set (`.jpg/.jpeg/.png/.tif/.tiff/.pdf`) -- [ ] Ensure upload directory exists (`mkdir(parents=True, exist_ok=True)`) -- [ ] Write file bytes to `UPLOAD_DIR` -- [ ] Persist DB records in one transaction: - - [ ] `Document(filename, file_path)` - - [ ] `Job(document_id=..., status=queued)` -- [ ] Return `UploadJobResult` -- [ ] Add logging for success/failure boundaries - ---- - -## Phase B — Implement worker core (`src/transcription/worker.py`) - -- [ ] Add `process_next_queued_job(...) -> bool` -- [ ] Fetch oldest queued job -- [ ] Return `False` when no queued jobs exist -- [ ] Transition picked job to `processing` and update timestamp -- [ ] Resolve associated `Document.file_path` -- [ ] Call `transcribe_document_image(image_path=...)` -- [ ] On success: - - [ ] insert/update transcript text - - [ ] clear error detail - - [ ] mark job `transcribed` - - [ ] update timestamp -- [ ] On failure: - - [ ] insert/update transcript with `text=None`, `error_detail=...` - - [ ] mark job `failed` - - [ ] update timestamp -- [ ] Commit terminal state and return `True` -- [ ] Add logs around job pickup, transition, and terminal outcome - ---- - -## Phase C — Implement worker loop (`src/transcription/worker.py`) - -- [ ] Add `run_worker_loop(...)` -- [ ] Accept configurable stop event/signal -- [ ] Accept configurable poll interval -- [ ] Repeatedly call `process_next_queued_job` -- [ ] Sleep only when queue is empty -- [ ] Exit cleanly when stop event is set - ---- - -## Phase D — Exports - -- [ ] Update `src/transcription/services/__init__.py` to expose upload APIs -- [ ] Keep existing transcription exports intact - ---- - -## Phase E — Tests via MCP scaffold -> fill flow - -## E1 Scaffold (structure only) - -Use scaffold prompt workflow first for: -- `src/transcription/services/upload.py` -- `src/transcription/worker.py` - -Expected scaffold targets: -- `tests/services/test_upload.py` -- `tests/services/test_worker.py` - -Scaffold rules: -- [ ] Class hierarchy + method names + one-line docstrings only -- [ ] No assertions or implementation details in scaffold phase -- [ ] Keep method names concise and behavior-focused - -Validation: -- [ ] `uv run pytest --collect-only -q` - -## E2 Fill scaffold (implementation) - -Use fill prompt workflow for: -- `tests/services/test_upload.py` -- `tests/services/test_worker.py` -- stack: `sqlalchemy-sync` (or `mixed` if combining pure + DB behaviors) -- marker lane preference: `unit` and `integration` as appropriate -- strategy: minimal deterministic implementation - -Fill rules (invariants): -- [ ] Preserve scaffold class names, method names, and one-line docstrings -- [ ] Do not rename/re-nest scaffolded tests unless explicitly approved -- [ ] One behavior target per test -- [ ] Minimal mocking; mock only network/nondeterministic boundaries - -Suggested test coverage: - -### `tests/services/test_upload.py` -- [ ] creates file + document + queued job (`integration`) -- [ ] rejects empty bytes (`unit`) -- [ ] rejects unsupported extension (`unit`) -- [ ] writes collision-safe unique filename (`integration`) -- [ ] persisted job status is `queued` (`integration`) - -### `tests/services/test_worker.py` -- [ ] returns `False` when queue empty (`integration`) -- [ ] transitions `queued -> processing -> transcribed` on success (`integration`) -- [ ] stores transcript text on success (`integration`) -- [ ] transitions to `failed` and stores `error_detail` on failure (`integration`) -- [ ] updates existing transcript instead of duplicate create (`integration`) -- [ ] worker loop exits when stop event set (`unit`) - ---- - -## Marker Strategy - -- `unit`: pure logic tests (filename handling, loop stop behavior, validation logic) -- `integration`: DB + service orchestration tests (SQLite/session/contracts) -- `external`: opt-in live provider tests only (not part of default Step 4 lane) - -No new marker needed; reuse existing marker registration. - ---- - -## Validation Sequence (strict order) - -- [ ] `uv run pytest --collect-only -q` -- [ ] `uv run pytest -m unit -q` *(if unit tests touched)* -- [ ] `uv run pytest tests/services/test_upload.py -q` -- [ ] `uv run pytest tests/services/test_worker.py -q` -- [ ] `uv run pytest -q` - ---- - -## Reporting Requirements (after implementation) - -Implementation report must include: - -1. Files created/updated -2. Fixture and marker decisions -3. MCP references used and why -4. Validation command results -5. Remaining risks/open questions (only blockers) - ---- - -## Guardrails - -- Keep Step 4 independent from UI concerns. -- Do not call provider SDK directly from worker. -- Do not silently swallow exceptions. -- Always persist terminal job outcome. -- Keep default suite deterministic and fast. -- Preserve scaffold invariants during fill phase. - ---- - -## Definition of Done (Step 4) - -Step 4 is complete when: - -- [ ] Upload service writes file and creates `Document` + queued `Job` -- [ ] Worker processes queued jobs end-to-end using Step 3 transcription service -- [ ] Success path persists transcript text and sets `transcribed` -- [ ] Failure path persists error detail and sets `failed` -- [ ] Queue-empty path returns cleanly -- [ ] New tests pass and full suite is green (`uv run pytest -q`) -- [ ] Output report includes MCP reference usage + validation evidence \ No newline at end of file diff --git a/docs/mvp/mvp-step5-results.md b/docs/mvp/mvp-step5-results.md deleted file mode 100644 index 62ba5c4..0000000 --- a/docs/mvp/mvp-step5-results.md +++ /dev/null @@ -1,70 +0,0 @@ -Step 5 is implemented and passing ✅ - -### What I implemented - -#### App/bootstrap -- Added `src/transcription/app.py`: - - `create_app()` factory - - FastAPI lifespan startup/shutdown: - - `setup_logging()` - - `create_all()` - - ensure `upload_dir` and `prompt_dir` - - start/stop worker thread (`run_worker_loop`) - - includes health router - - registers NiceGUI pages - -#### API -- Added `src/transcription/api/__init__.py` -- Added `src/transcription/api/health.py`: - - `GET /healthz` returns `{"status": "ok"}` - -#### UI -- Added `src/transcription/ui/upload_page.py`: - - upload page route - - accepted extensions helper - - upload submit helper to Step 4 service (`create_upload_job`) - - success/error state messaging -- Added `src/transcription/ui/jobs_page.py`: - - jobs list page - - job detail page - - `fetch_jobs()` and `fetch_job_detail()` helpers -- Updated `src/transcription/ui/__init__.py`: - - explicit page registration - - NiceGUI mounted via `ui.run_with(app, mount_path="/ui", ...)` - -#### Dependencies -- Added runtime dependencies: - - `fastapi` - - `nicegui` - ---- - -### Tests created and filled (scaffold → fill) - -- `tests/test_app.py` -- `tests/api/test_health.py` -- `tests/ui/test_pages_registration.py` -- `tests/ui/test_upload_page.py` -- `tests/ui/test_jobs_page.py` - -All scaffolded method names/docstrings were preserved and filled with executable assertions. - ---- - -### Validation results (strict sequence) - -- `uv run pytest --collect-only -q` ✅ -- `uv run pytest -m unit -q` ✅ -- `uv run pytest tests/api/test_health.py -q` ✅ -- `uv run pytest tests/ui/test_pages_registration.py -q` ✅ -- `uv run pytest tests/test_app.py -q` ✅ -- `uv run pytest tests/ui/test_upload_page.py -q` ✅ -- `uv run pytest tests/ui/test_jobs_page.py -q` ✅ -- `uv run pytest -q` ✅ - -Full suite is green. - ---- - -### Note -You’ll see one warning from FastAPI/Starlette test client about `httpx` deprecation; it does not affect correctness and all tests pass. \ No newline at end of file diff --git a/docs/mvp/mvp-step5.md b/docs/mvp/mvp-step5.md deleted file mode 100644 index 06964ae..0000000 --- a/docs/mvp/mvp-step5.md +++ /dev/null @@ -1,310 +0,0 @@ -## Step 5: `app.py` + UI Pages (NiceGUI + FastAPI composition) - -## Objective - -Implement the MVP user-facing application layer so users can: - -1. Upload a document from the UI -2. Trigger Step 4 upload/job creation flow -3. See live job lifecycle status (`queued`, `processing`, `transcribed`, `failed`) -4. Open a job detail view to read transcript text or failure details - -This step composes Steps 1–4 into a usable UI. - ---- - -## Architecture Summary (NiceGUI-aligned) - -Step 5 uses a **FastAPI app factory + lifespan orchestration** and mounts/registers NiceGUI pages via explicit page modules. - -Reference baseline: `resource://skills/nicegui/document` - -### Core architecture decisions - -- **App factory:** `create_app()` -- **Lifespan-managed resources:** worker start/stop managed in startup/shutdown -- **Modular pages:** upload and jobs pages in separate modules (no monolithic UI file) -- **Health endpoint:** FastAPI-side `/healthz` -- **UI composition:** route pages stay modular and reusable shared shell/components live under `ui/components` as needed -- **Styling architecture:** shared CSS loaded once at startup; avoid ad-hoc per-page styling drift -- **Dependency direction (one-way):** - - `app` -> `config/logging/db/worker/ui/api` - - `ui/pages` -> `ui/components` + `services` - - `services` -> `db/models/providers` - - no reverse imports from services into UI/API - -### DB and AI stance (explicit) - -- **DB:** already enabled (SQLModel + SQLite), session lifecycle remains request/service-scoped as built in prior steps. -- **AI workflow:** already in place via Step 3 transcription service + Step 4 worker; UI does not call provider SDK directly. -- **Mounted docs:** not in Step 5 scope; docs mounting remains disabled for MVP. - -### Async and responsiveness stance - -- Prefer `async def` for page handlers and service boundaries when I/O is involved. -- Keep UI handlers non-blocking (no blocking sleeps or synchronous long I/O calls). -- For long-running user actions, always provide explicit loading/progress/error states. -- Keep cancellation/timeout behavior explicit for refresh/poll operations where applicable. - ---- - -## Scope - -### In scope -- `src/transcription/app.py` -- `src/transcription/ui/upload_page.py` -- `src/transcription/ui/jobs_page.py` -- `src/transcription/ui/__init__.py` -- `src/transcription/api/health.py` (or equivalent FastAPI health route module) -- UI/app tests with MCP scaffold->fill flow - -### Out of scope -- Auth -- advanced filtering/search UX -- batch upload UX beyond MVP -- deployment/container hardening - ---- - -## Planned Deliverables - -### Source files -- `src/transcription/app.py` (app factory + lifespan wiring) -- `src/transcription/api/health.py` (GET `/healthz`) -- `src/transcription/ui/upload_page.py` (upload flow) -- `src/transcription/ui/jobs_page.py` (status list + detail) -- `src/transcription/ui/__init__.py` (explicit `register_pages(...)` export) -- `src/transcription/ui/components/*` (shared shell/navigation/status components if introduced) -- `src/transcription/ui/static/*.css` (optional shared CSS loaded once at startup) - -### Test files -- `tests/test_app.py` -- `tests/api/test_health.py` -- `tests/ui/test_pages_registration.py` -- `tests/ui/test_upload_page.py` -- `tests/ui/test_jobs_page.py` - ---- - -## Implementation Plan + Checklist - -Plan baseline and guardrails source: `resource://skills/nicegui/document` - -## Phase A — App factory and lifespan orchestration - -- [ ] Create `create_app()` in `src/transcription/app.py` -- [ ] Add FastAPI lifespan startup/shutdown handlers -- [ ] Startup responsibilities: - - [ ] `setup_logging()` - - [ ] `create_all()` - - [ ] ensure directories exist (`upload_dir`, `prompt_dir`) - - [ ] create worker stop event - - [ ] start worker background thread/task -- [ ] Shutdown responsibilities: - - [ ] signal stop event - - [ ] join/cleanup worker thread/task cleanly -- [ ] Register API router(s), including health route -- [ ] Register NiceGUI pages via explicit page registration function -- [ ] Load shared CSS once at startup (if present) - -## Phase B — FastAPI health endpoint - -- [ ] Create `src/transcription/api/health.py` -- [ ] Add `GET /healthz` returning simple healthy payload -- [ ] Wire route into app factory - -## Phase C — Upload page (`ui/upload_page.py`) - -- [ ] Add upload route/page registration function -- [ ] Render file input accepting supported extensions -- [ ] On submit: - - [ ] show loading/progress state - - [ ] call `create_upload_job(filename, file_bytes, ...)` - - [ ] show success state with job reference/link -- [ ] On error: - - [ ] show user-safe error message - - [ ] restore ready UI state -- [ ] Ensure non-blocking I/O in UI event handlers; offload CPU-heavy work to worker path -- [ ] Make timeout/cancellation behavior explicit for any long-running action - -## Phase D — Jobs page (`ui/jobs_page.py`) - -- [ ] Add jobs list route/page registration function -- [ ] Display jobs with status + timestamps -- [ ] Add job detail route/view -- [ ] Show transcript on success, error detail on failure -- [ ] Include explicit refresh action and loading state -- [ ] Ensure error states are surfaced to user and logged -- [ ] Keep refresh path async and bounded to avoid UI freeze - -## Phase E — UI registration module - -- [ ] Update `src/transcription/ui/__init__.py` -- [ ] Export `register_pages(...)` -- [ ] Ensure each page module exports `register_page(...)` -- [ ] Keep page registration explicit and modular - -## Phase F — Shared components and style consistency - -- [ ] Add `ui/components` module only for reusable shell elements (header/nav/status chips), not page-local logic -- [ ] Keep structural layout in Python; keep visual polish in shared CSS -- [ ] Avoid one-off styling duplication across upload/jobs pages - ---- - -## MCP Testing Workflow (Required) - -Use these resources directly: - -- `resource://catalog/prompts/pytest-scaffold` -- `resource://prompts/pytest-scaffold/document` -- `resource://catalog/prompts/pytest-fill-scaffold` -- `resource://prompts/pytest-fill-scaffold/document` - -## E1 — Scaffold tests first (structure only) - -Target modules: -- `src/transcription/app.py` -- `src/transcription/api/health.py` -- `src/transcription/ui/upload_page.py` -- `src/transcription/ui/jobs_page.py` - -Scaffold test files: -- `tests/test_app.py` -- `tests/api/test_health.py` -- `tests/ui/test_pages_registration.py` -- `tests/ui/test_upload_page.py` -- `tests/ui/test_jobs_page.py` - -Scaffold constraints: -- [ ] class/method skeletons only -- [ ] one-line docstrings -- [ ] concise behavior-focused names -- [ ] no implementation assertions yet - -Validation: -- [ ] `uv run pytest --collect-only -q` - -## E2 — Fill scaffold tests - -Fill constraints from MCP guidance: -- [ ] preserve scaffold class/method names and docstrings (locked baseline) -- [ ] one behavior target per method -- [ ] deterministic tests preferred -- [ ] minimal mocking; only nondeterministic boundaries - -Stack: -- [ ] `fastapi` (or `mixed` if needed for UI+DB fixture combination) - -Suggested coverage: - -### `tests/api/test_health.py` -- [ ] `/healthz` returns success status and expected payload shape - -### `tests/ui/test_pages_registration.py` -- [ ] page registration wiring succeeds -- [ ] expected routes are present - -### `tests/test_app.py` -- [ ] startup path initializes runtime dependencies -- [ ] worker start is invoked on startup -- [ ] worker shutdown signal/cleanup is invoked on shutdown - -### `tests/ui/test_upload_page.py` -- [ ] upload action calls upload service -- [ ] success feedback displayed -- [ ] error feedback displayed for `UploadError` -- [ ] loading/progress state behavior covered -- [ ] timeout/cancellation behavior covered (if implemented) - -### `tests/ui/test_jobs_page.py` -- [ ] list renders job statuses -- [ ] detail shows transcript text for successful job -- [ ] detail shows error detail for failed job -- [ ] refresh/loading state behavior covered - -Marker strategy: -- [ ] `unit` for pure helpers/state formatting -- [ ] `integration` for app/page/service+DB contracts -- [ ] `external` not required for default Step 5 lane - -Async behavior assertions: -- [ ] long-running actions keep button/inputs in expected disabled state -- [ ] completion/failure returns controls to ready state - ---- - -## Validation Sequence (strict) - -- [ ] `uv run pytest --collect-only -q` -- [ ] `uv run pytest -m unit -q` *(if unit tests touched)* -- [ ] `uv run pytest tests/api/test_health.py -q` -- [ ] `uv run pytest tests/ui/test_pages_registration.py -q` -- [ ] `uv run pytest tests/test_app.py -q` -- [ ] `uv run pytest tests/ui/test_upload_page.py -q` -- [ ] `uv run pytest tests/ui/test_jobs_page.py -q` -- [ ] `uv run pytest -q` - ---- - -## Guardrails (NiceGUI + MVP) - -- [ ] Do not collapse pages into one file. -- [ ] Do not use implicit global side effects for runtime wiring. -- [ ] Keep UI responsive with explicit loading/progress/error states. -- [ ] Do not block UI handlers with synchronous long I/O. -- [ ] Do not place provider SDK calls in UI handlers. -- [ ] Keep dependency direction one-way and maintainable. -- [ ] Keep shared UI in `ui/components`; keep service logic out of page modules. - ---- - -## Definition of Done - -- [ ] App factory + lifespan are in place -- [ ] Health endpoint exists and is tested -- [ ] Upload page creates queued jobs through service boundary -- [ ] Jobs list/detail pages render status/transcript/failure data -- [ ] Worker lifecycle is started/stopped by app lifespan -- [ ] Async UI states (loading/success/error) are deterministic and tested -- [ ] Scaffold->fill testing flow completed and validated -- [ ] Full suite passes: `uv run pytest -q` - -## Completion Checks (NiceGUI skill aligned) - -- [ ] Uses app factory and FastAPI lifespan -- [ ] Pages are modularized (not single-file UI) -- [ ] Health endpoint exists on FastAPI side -- [ ] Dependency direction is clean and one-way -- [ ] Async-first guidance is applied where I/O exists, with explicit non-blocking UX states -- [ ] DB/AI/docs decisions are explicit and reflected in structure -- [ ] Plan references baseline URI: `resource://skills/nicegui/document` - ---- - -## PR Checklist (Integrated) - -### Implementation -- [ ] `app.py` app factory + lifespan implemented -- [ ] FastAPI health route (`/healthz`) implemented -- [ ] `ui/upload_page.py` implemented -- [ ] `ui/jobs_page.py` implemented -- [ ] `ui/__init__.py` explicit page registration implemented -- [ ] Worker startup/shutdown managed by lifespan - -### Testing (MCP-compliant) -- [ ] Scaffold phase completed first for all Step 5 tests -- [ ] `--collect-only` passed on scaffolds -- [ ] Fill phase completed without renaming/re-nesting scaffolded tests -- [ ] Marker decisions documented (`unit` vs `integration`) -- [ ] Targeted tests passed -- [ ] Full suite passed - -### Evidence -- [ ] Validation command outputs captured -- [ ] Files created/updated listed -- [ ] MCP prompt resources referenced in implementation notes -- [ ] Any residual risks/questions documented - ---- - diff --git a/docs/mvp/mvp-step6-results.md b/docs/mvp/mvp-step6-results.md deleted file mode 100644 index 8281995..0000000 --- a/docs/mvp/mvp-step6-results.md +++ /dev/null @@ -1,50 +0,0 @@ -Step 6 implemented ✅ - -### Files added -- `tests/integration/test_pipeline_flow.py` -- `tests/test_traceability.py` - -### What was implemented - -#### 1) End-to-end pipeline integration coverage -`tests/integration/test_pipeline_flow.py` now verifies: - -- **Success path** - - upload creates queued job - - worker processes job - - job becomes `transcribed` - - transcript text is persisted - - error detail remains `None` - -- **Failure path** - - upload creates queued job - - worker processing failure is handled - - job becomes `failed` - - transcript has `text=None` - - `error_detail` contains provider failure reason - -#### 2) MVP requirement traceability check -`tests/test_traceability.py` adds a concrete map: - -- REQ-0, REQ-1, REQ-2, REQ-3, REQ-4, REQ-5, REQ-6, REQ-8, REQ-12 -- each REQ maps to one or more existing test files -- test asserts each mapped path exists (guards against drift) - ---- - -### Validation run results - -Executed and passed: - -1. `uv run pytest --collect-only -q` ✅ -2. `uv run pytest -m unit -q` ✅ -3. `uv run pytest -m integration -q` ✅ -4. `uv run pytest -m "not external" -q` ✅ -5. `uv run pytest tests/integration/test_pipeline_flow.py -q` ✅ -6. `uv run pytest tests/ui/test_upload_page.py -q` ✅ -7. `uv run pytest tests/ui/test_jobs_page.py -q` ✅ -8. `uv run pytest -q` ✅ - -All green. -(Only existing non-blocking FastAPI TestClient deprecation warning remains.) - diff --git a/docs/mvp/mvp-step6.md b/docs/mvp/mvp-step6.md deleted file mode 100644 index 2ae4f1a..0000000 --- a/docs/mvp/mvp-step6.md +++ /dev/null @@ -1,229 +0,0 @@ -## Step 6: Test and Verification Hardening (MVP closeout) - -## Objective - -Complete MVP verification by building a **requirements-traceable, deterministic test strategy** across unit/integration/external lanes, then enforcing stable validation commands and reporting. - -This step finalizes the MVP implementation sequence from `docs/mvp.md` (Step 6 in the build order: tests and automated verification). - ---- - -## MCP Resource Integration (what was applied) - -I reviewed all top-level skills/prompts from `john-stream-mcp` and integrated the relevant guidance into this plan: - -### Directly applied -- `resource://skills/pytesting/document` -- `resource://catalog/prompts/pytest-scaffold` -- `resource://prompts/pytest-scaffold/document` -- `resource://catalog/prompts/pytest-fill-scaffold` -- `resource://prompts/pytest-fill-scaffold/document` -- `resource://skills/nicegui/document` -- `resource://skills/nicegui-ui-customization/document` -- `resource://skills/fastapi-uv-docker/document` -- `resource://skills/python-logging-dictconfig/document` -- `resource://skills/python-typing/document` -- `resource://skills/ruff-linting-formating/document` - -### Reviewed but informational/non-blocking for Step 6 -- `copilot-customization`, `mcp-details`, `vscode-configuration`, `zensical-docs`, and authoring/shim prompts. - - These are primarily customization/documentation tooling resources, not core MVP test-lane blockers. - - Step 6 includes optional workflow follow-ups where relevant (e.g., VS Code task conveniences). - ---- - -## Scope - -### In scope -- Strengthen and complete test coverage for the shipped MVP slice (Steps 1–5) -- Add requirement-to-test traceability for REQ-0..REQ-12 (MVP subset emphasized) -- Enforce deterministic default lanes (`unit`, `integration`) -- Keep `external` lane opt-in and isolated -- Validate app/UI/service/worker contracts end-to-end at test level - -### Out of scope -- Major architecture rewrites (async SQLAlchemy migration, queue system, etc.) -- Full production deployment rollout -- Post-MVP feature expansion (revision history, search, export) - ---- - -## Planned Deliverables - -### Test files (new/updated) -- `tests/test_traceability.py` *(or docs-based traceability matrix if preferred)* -- `tests/integration/test_pipeline_flow.py` *(upload -> queued -> worker -> transcript/failed)* -- `tests/ui/test_upload_page.py` (augment loading/error/ready-state checks as practical) -- `tests/ui/test_jobs_page.py` (augment refresh/error behavior checks as practical) -- Existing tests touched only when needed; preserve naming/hierarchy unless explicitly approved. - -### Optional docs output -- `docs/tests.md` or `docs/verification.md` with lane definitions and command matrix -- REQ-to-test mapping table - ---- - -## Design and Policy Decisions (MCP-aligned) - -1. **Scaffold-first, fill-second workflow is mandatory** - - First create/adjust skeletons and collect. - - Then fill test bodies. - - Preserve scaffold names/docstrings during fill. - -2. **Deterministic-first default lanes** - - `unit` and `integration` run by default. - - `external` remains explicit opt-in. - -3. **One behavior target per test** - - Short, behavior-focused names. - - Precise assertions on observable outcomes. - -4. **Test double discipline (from pytesting skill)** - - Prefer real-input/real-object paths first. - - If monkeypatch/mocks/fakes are needed for a boundary, keep narrowly scoped. - - Avoid call-only assertions. - -5. **NiceGUI responsiveness expectations** - - Verify loading/success/error state transitions where testable. - - Ensure user-facing feedback behavior is covered. - -6. **FastAPI/ops baseline checks** - - Keep `/healthz` route validation in default lanes. - - Keep startup/shutdown lifecycle assertions present. - ---- - -## Implementation Plan + Checklist - -## Phase A — Coverage and traceability audit - -- [ ] Build a REQ-to-test matrix for MVP requirements: - - [ ] REQ-0, REQ-1, REQ-2, REQ-3, REQ-4, REQ-5, REQ-6, REQ-8, REQ-12 -- [ ] Identify weak spots: - - [ ] full pipeline integration (service + worker + persistence) - - [ ] UI state transition assertions (loading/error/ready) - - [ ] failure-path persistence verification robustness -- [ ] Record current baseline command results before edits - -## Phase B — Scaffold phase (pytest-scaffold resources) - -Target modules/areas: -- pipeline integration flow -- UI behavior augmentations -- traceability checks/document validators (if test-backed) - -- [ ] Scaffold new/adjusted test files/classes/methods only -- [ ] Keep one-line intent docstrings -- [ ] Keep behavior-focused names -- [ ] Run: `uv run pytest --collect-only -q` - -## Phase C — Fill phase (pytest-fill-scaffold resources) - -- [ ] Fill scaffolded methods with deterministic setup/assertions -- [ ] Preserve scaffold names/hierarchy/docstrings -- [ ] Add/adjust fixtures at nearest useful scope -- [ ] Keep DB tests in `integration`; pure helper tests in `unit` - -### Required coverage additions - -#### Pipeline integration -- [ ] Upload service creates document/job and file path persists -- [ ] Worker success path creates transcript and terminal status -- [ ] Worker failure path persists error detail and terminal failed status -- [ ] Queue-empty behavior remains stable (`False` return / no side effects) - -#### UI behavior (practical, testable boundaries) -- [ ] Upload helper flow success and UploadError surfacing -- [ ] Jobs data helpers return stable normalized view models -- [ ] Refresh/detail fallback behavior for missing/invalid job IDs - -#### Traceability -- [ ] Every in-scope MVP REQ has at least one mapped test/assertion point -- [ ] Document and/or enforce mapping consistency - -## Phase D — External lane stability - -- [ ] Keep real-image external tests isolated under `@pytest.mark.external` -- [ ] Ensure no external test leaks into default runs -- [ ] Confirm artifact capture behavior remains stable - -## Phase E — Quality gates and workflow - -- [ ] Confirm logging/lifecycle startup tests still pass after changes -- [ ] (If enabled) add/update lint/type check commands in docs: - - [ ] Ruff lane (if configured) - - [ ] typing lane (if configured) -- [ ] Optionally add VS Code task aliases for test lanes (non-blocking) - ---- - -## Marker and Fixture Strategy - -- `unit`: pure logic, helper behavior, formatting/normalization -- `integration`: DB + service + app lifecycle contracts -- `external`: live provider/real image checks only - -Fixture policy: -- Prefer reusable fixtures in `tests/conftest.py` only when broadly shared -- Use subtree/local fixtures for domain-specific setup -- Keep setup explicit and readable - ---- - -## Validation Sequence (strict) - -- [ ] `uv run pytest --collect-only -q` -- [ ] `uv run pytest -m unit -q` -- [ ] `uv run pytest -m integration -q` -- [ ] `uv run pytest -m "not external" -q` -- [ ] `uv run pytest tests/integration/test_pipeline_flow.py -q` *(if added)* -- [ ] `uv run pytest tests/ui/test_upload_page.py -q` -- [ ] `uv run pytest tests/ui/test_jobs_page.py -q` -- [ ] `uv run pytest -q` - -Optional external verification: -- [ ] `uv run pytest -m external -q` - ---- - -## Guardrails - -- Do not rename/re-nest scaffolded tests during fill unless explicitly requested. -- Do not broaden external dependencies in default lane. -- Do not add flaky timing-based assertions; keep deterministic boundaries. -- Keep business logic out of UI tests; test through service/helper boundaries. -- Preserve one-way dependency direction in test setup patterns. - ---- - -## Definition of Done (Step 6) - -- [ ] MVP requirement coverage is explicitly traceable -- [ ] Deterministic lanes (`unit` + `integration`) are stable and green -- [ ] External lane remains opt-in and green when enabled -- [ ] Pipeline success/failure lifecycle paths are verified end-to-end -- [ ] UI helper/state behavior has explicit success/error assertions -- [ ] Full suite passes with `uv run pytest -q` -- [ ] Verification evidence is captured in implementation report - ---- - -## PR Checklist (Step 6) - -### Implementation -- [ ] Added/updated test files per scoped gaps -- [ ] Added REQ traceability mapping -- [ ] Kept default lanes deterministic -- [ ] Preserved scaffold invariants during fill - -### Testing (MCP-compliant) -- [ ] Used scaffold prompt flow first -- [ ] Used fill prompt flow second -- [ ] Preserved naming/docstrings/hierarchy -- [ ] Marker usage documented (`unit`, `integration`, `external`) - -### Evidence -- [ ] Collected command outputs in strict order -- [ ] Listed files changed -- [ ] Listed MCP resources used and why -- [ ] Noted residual risks/open questions (if any) diff --git a/docs/mvp/mvp-step7-results.md b/docs/mvp/mvp-step7-results.md deleted file mode 100644 index 7e3b075..0000000 --- a/docs/mvp/mvp-step7-results.md +++ /dev/null @@ -1,134 +0,0 @@ -## Step 7 Results: Error Handling Standardization and Operational Visibility - -## Summary - -Step 7 was implemented across the MVP runtime boundaries with a shared error taxonomy, actionable UI error surfacing, worker failure normalization, and API error envelope handling. - -All required validation gates in `docs/step7.md` were executed and passed. - ---- - -## Scope Delivered - -### Implemented -- Shared application error contract and taxonomy -- Service-layer error normalization (upload + transcription) -- UI error presentation helpers with suggested actions and error references -- Worker failure persistence format with category/suggestion/error_id markers -- API exception handlers for structured error responses -- Targeted tests for new error contract behavior - -### Not implemented in this step -- External lane execution (`-m external`) was not required for Step 7 completion and was not run in this pass. - ---- - -## Files Added - -- `src/transcription/errors.py` -- `src/transcription/api/errors.py` -- `src/transcription/ui/error_presenter.py` -- `tests/test_errors.py` -- `tests/api/test_error_responses.py` -- `docs/step7.md` - -## Files Updated - -- `src/transcription/app.py` -- `src/transcription/services/upload.py` -- `src/transcription/services/transcription.py` -- `src/transcription/ui/upload_page.py` -- `src/transcription/ui/jobs_page.py` -- `src/transcription/worker.py` -- `tests/services/test_upload.py` -- `tests/services/test_transcription.py` -- `tests/services/test_worker.py` -- `tests/integration/test_pipeline_flow.py` -- `uv.lock` - ---- - -## Implementation Notes by Phase - -### Phase A/B (Foundation) -- Added `ErrorCategory` enum and `AppError` base type in `src/transcription/errors.py`. -- Added helper utilities: - - `new_error_id()` - - `build_error_envelope(...)` - - `classify_unexpected_error(...)` - - `format_error_detail(...)` - -### Phase C (Service/Provider normalization) -- `UploadError` now extends `AppError` and includes category/suggestion/retriable metadata. -- `PromptLoadError` and `TranscriptionError` now extend `AppError`. -- Provider failures are mapped with deterministic category semantics (auth/payload/provider-failure cases). - -### Phase D (UI visibility) -- Added `src/transcription/ui/error_presenter.py`. -- Upload and jobs pages now use centralized UI error rendering and summary helpers. -- UI error paths now include more visible/actionable guidance and reference IDs. - -### Phase E (Worker failure handling) -- Worker now normalizes exception handling into structured persisted `error_detail` strings with: - - category marker - - suggestion marker - - error_id marker -- Logging now includes category/error_id context in failure paths. - -### Phase F (API envelope) -- Added `src/transcription/api/errors.py` and registered handlers in app factory. -- AppError and unexpected exceptions now serialize to stable API envelopes with mapped status codes. - ---- - -## Validation Commands and Outcomes - -All commands were executed with `uv run python -m pytest ...` and completed successfully. - -1. `uv run python -m pytest tests/test_errors.py -q` ✅ -2. `uv run python -m pytest tests/services/test_upload.py -q` ✅ -3. `uv run python -m pytest tests/services/test_transcription.py -q` ✅ -4. `uv run python -m pytest tests/providers/test_openrouter.py -q` ✅ -5. `uv run python -m pytest tests/services/test_worker.py -q` ✅ -6. `uv run python -m pytest tests/integration/test_pipeline_flow.py -q` ✅ -7. `uv run python -m pytest tests/api/test_error_responses.py -q` ✅ -8. `uv run python -m pytest tests/ui/test_upload_page.py -q` ✅ -9. `uv run python -m pytest tests/ui/test_jobs_page.py -q` ✅ -10. `uv run python -m pytest -m "not external" -q` ✅ -11. `uv run python -m pytest --collect-only -q` ✅ -12. `uv run python -m pytest -m unit -q` ✅ -13. `uv run python -m pytest -m integration -q` ✅ -14. `uv run python -m pytest tests/integration/test_pipeline_flow.py -q` ✅ -15. `uv run python -m pytest tests/ui/test_upload_page.py -q` ✅ -16. `uv run python -m pytest tests/ui/test_jobs_page.py -q` ✅ -17. `uv run python -m pytest -q` ✅ - -Observed warning (non-blocking): Starlette/FastAPI TestClient deprecation warning related to `httpx` package naming. - ---- - -## Policy Alignment Check (`docs/error_handling.md`) - -Aligned items: -- Stable taxonomy categories are implemented. -- Unexpected errors are normalized. -- User-facing UI paths include actionable guidance and references. -- Worker persistence includes trace-friendly failure detail. -- API error responses are structured and category-aware. - -Follow-up candidates: -- Add richer UI tests that validate rendered suggested-action content end-to-end (current tests focus helper/service contracts). -- Consider typed storage fields for error metadata instead of packed `error_detail` strings in a future schema revision. - ---- - -## Step 7 Definition of Done Status - -- [x] Shared error taxonomy implemented across MVP layers -- [x] GUI error paths upgraded for visibility/actionability -- [x] Worker failure persistence and log context standardized -- [x] API error envelope handling added and tested -- [x] Phase-level and full-suite validation gates passed -- [x] Results documented in this report - -Step 7 is complete. \ No newline at end of file diff --git a/docs/mvp/mvp-step7.md b/docs/mvp/mvp-step7.md deleted file mode 100644 index f8d0d0e..0000000 --- a/docs/mvp/mvp-step7.md +++ /dev/null @@ -1,267 +0,0 @@ -## Step 7: Error Handling Standardization and Operational Visibility - -## Objective - -Apply the canonical error policy from `docs/error_handling.md` to the MVP implementation so failures are: - -- consistently classified -- visibly surfaced in the GUI -- paired with suggested corrective actions -- traceable through logs via error reference IDs -- validated through deterministic tests after each phase - -This step extends MVP hardening by converting current ad hoc exception behavior into a stable cross-layer contract. - ---- - -## Scope - -### In scope -- Introduce a shared application error contract and taxonomy implementation -- Normalize service/provider exceptions into taxonomy categories -- Improve GUI error visibility and suggested-action UX -- Standardize worker failure persistence and logging context -- Add API error-envelope policy hooks for current/future endpoints -- Add targeted tests and phase-level/full-suite validation gates - -### Out of scope -- Major architecture rewrites (distributed queue, multi-service decomposition) -- Post-MVP feature expansion unrelated to error handling -- Full observability platform rollout (tracing backends, APM) - ---- - -## Policy Source of Truth - -- Canonical policy document: `docs/error_handling.md` -- If implementation and policy diverge, policy is authoritative and code/tests must be updated. - ---- - -## Planned Deliverables - -### Runtime code -- `src/transcription/errors.py` *(new shared contract module)* -- `src/transcription/ui/error_presenter.py` *(new UI error rendering helper)* -- Updates to: - - `src/transcription/services/upload.py` - - `src/transcription/services/transcription.py` - - `src/transcription/providers/openrouter.py` - - `src/transcription/worker.py` - - `src/transcription/ui/upload_page.py` - - `src/transcription/ui/jobs_page.py` - - `src/transcription/api/*` *(as needed for envelope/handlers)* - -### Tests -- `tests/test_errors.py` *(new shared error contract tests)* -- updates/additions in: - - `tests/services/test_upload.py` - - `tests/services/test_transcription.py` *(add if missing)* - - `tests/providers/test_openrouter.py` - - `tests/services/test_worker.py` - - `tests/ui/test_upload_page.py` - - `tests/ui/test_jobs_page.py` - - `tests/api/test_error_responses.py` *(new, if API handlers added)* - -### Documentation -- Update `docs/error_handling.md` only if implementation reveals policy gaps -- Capture validation evidence in a Step 7 results artifact (`docs/step7-results.md`) - ---- - -## Design and Policy Decisions - -1. **Stable taxonomy contract** - - Use policy categories as stable identifiers (`validation_error`, `user_input_error`, etc.). - -2. **Actionable UX is mandatory** - - User-visible errors must include a suggested course of action. - -3. **Traceability by default** - - Non-trivial errors include an `error_id` in both logs and user-facing output. - -4. **Safe surface / rich logs** - - UI/API show safe summaries; logs retain diagnostic detail and traceback. - -5. **Deterministic verification cadence** - - Targeted tests after each change batch, then phase-level regression gates. - ---- - -## Implementation Plan + Checklist - -## Phase A — Baseline Validation and Gap Confirmation - -- [ ] Run baseline tests before changes -- [ ] Record baseline outputs and any known flaky behavior -- [ ] Confirm current behavior against `docs/error_handling.md` requirements - -### Validation gate -- [ ] `uv run pytest -m "not external" -q` -- [ ] `uv run pytest -q` - -## Phase B — Shared Error Contract Foundation - -- [ ] Add `src/transcription/errors.py` with: - - [ ] stable category enum - - [ ] base `AppError` (category/message/suggestion/error_id/retriable) - - [ ] helpers for error-id generation and fallback classification -- [ ] Keep category names aligned with `docs/error_handling.md` - -### Tests -- [ ] Add `tests/test_errors.py` - - [ ] category stability assertions - - [ ] error_id creation behavior - - [ ] fallback classification for unexpected exceptions - -### Validation gate -- [ ] `uv run pytest tests/test_errors.py -q` -- [ ] `uv run pytest -m "not external" -q` - -## Phase C — Service and Provider Normalization - -- [ ] Refactor upload service exceptions to shared taxonomy -- [ ] Refactor transcription service exceptions to shared taxonomy -- [ ] Normalize provider adapter failures into deterministic categories -- [ ] Preserve causal chaining (`raise ... from exc`) - -### Tests -- [ ] Extend `tests/services/test_upload.py`: - - [ ] empty payload category/suggestion - - [ ] unsupported extension category/suggestion - - [ ] persistence failure category mapping -- [ ] Add/extend `tests/services/test_transcription.py`: - - [ ] missing/empty prompt behavior - - [ ] unsupported file type behavior - - [ ] provider failure mapping behavior -- [ ] Extend `tests/providers/test_openrouter.py`: - - [ ] auth error mapping - - [ ] malformed response mapping - -### Validation gate -- [ ] `uv run pytest tests/services/test_upload.py -q` -- [ ] `uv run pytest tests/services/test_transcription.py -q` -- [ ] `uv run pytest tests/providers/test_openrouter.py -q` -- [ ] `uv run pytest -m "not external" -q` - -## Phase D — GUI Visibility and Suggested Actions - -- [ ] Add `src/transcription/ui/error_presenter.py` -- [ ] Update upload/jobs pages to use centralized error presentation -- [ ] Ensure GUI surfaces: - - [ ] user-safe message - - [ ] suggested action - - [ ] error reference ID - - [ ] optional technical details panel -- [ ] Replace raw `str(exc)` UX where policy requires safer messaging - -### Tests -- [ ] Extend `tests/ui/test_upload_page.py` for actionable error UX paths -- [ ] Extend `tests/ui/test_jobs_page.py` for refresh/detail error guidance -- [ ] Add `tests/ui/test_error_presenter.py` *(optional but recommended)* - -### Validation gate -- [ ] `uv run pytest tests/ui/test_upload_page.py -q` -- [ ] `uv run pytest tests/ui/test_jobs_page.py -q` -- [ ] `uv run pytest -m "not external" -q` - -## Phase E — Worker Failure Persistence and Logging Context - -- [ ] Update worker failure handling to classify errors before persistence -- [ ] Ensure failed jobs persist actionable, structured error detail -- [ ] Add log context fields where available (`error_id`, `category`, `operation`, `job_id`) -- [ ] Ensure retry semantics are explicit and bounded (or clearly documented as deferred) - -### Tests -- [ ] Extend `tests/services/test_worker.py`: - - [ ] missing document failure contract - - [ ] provider/transcription failure contract - - [ ] persisted error detail includes category/suggestion/error_id markers -- [ ] Validate integration failure flow in `tests/integration/test_pipeline_flow.py` - -### Validation gate -- [ ] `uv run pytest tests/services/test_worker.py -q` -- [ ] `uv run pytest tests/integration/test_pipeline_flow.py -q` -- [ ] `uv run pytest -m "not external" -q` - -## Phase F — API Error Envelope Alignment (Current + Future Routes) - -- [ ] Add shared API error serialization utilities/handlers (as needed) -- [ ] Ensure API responses can include: - - [ ] `error_id` - - [ ] `category` - - [ ] `message` - - [ ] `suggestion` - - [ ] `timestamp` -- [ ] Map categories to HTTP status guidance from `docs/error_handling.md` - -### Tests -- [ ] Add `tests/api/test_error_responses.py` *(if handlers added)* -- [ ] Keep `tests/api/test_health.py` passing - -### Validation gate -- [ ] `uv run pytest tests/api/test_error_responses.py -q` *(if added)* -- [ ] `uv run pytest tests/api/test_health.py -q` -- [ ] `uv run pytest -m "not external" -q` - -## Phase G — Final Regression and Documentation Closure - -- [ ] Reconcile implementation details with `docs/error_handling.md` -- [ ] Update policy doc only where required by confirmed implementation learning -- [ ] Capture execution evidence in `docs/step7-results.md` - -### Final validation sequence (strict) -- [ ] `uv run pytest --collect-only -q` -- [ ] `uv run pytest -m unit -q` -- [ ] `uv run pytest -m integration -q` -- [ ] `uv run pytest -m "not external" -q` -- [ ] `uv run pytest tests/integration/test_pipeline_flow.py -q` -- [ ] `uv run pytest tests/ui/test_upload_page.py -q` -- [ ] `uv run pytest tests/ui/test_jobs_page.py -q` -- [ ] `uv run pytest -q` - -Optional: -- [ ] `uv run pytest -m external -q` - ---- - -## Guardrails - -- Do not weaken user-facing clarity to expose raw internals. -- Do not introduce silent exception swallowing. -- Do not break category-name stability without policy update. -- Do not merge phase changes without passing that phase validation gate. -- Keep targeted tests fast and deterministic; isolate external-provider tests under `external`. - ---- - -## Definition of Done (Step 7) - -- [ ] Shared error taxonomy is implemented and used across MVP layers -- [ ] GUI error experiences are visible, actionable, and traceable -- [ ] Worker persists and logs failure context consistently -- [ ] API error contract path is aligned for current/future endpoints -- [ ] Phase-by-phase test gates pass -- [ ] Full suite remains green (`uv run pytest -q`) -- [ ] Step 7 results are documented with evidence - ---- - -## PR Checklist (Step 7) - -### Implementation -- [ ] Added shared error contract module -- [ ] Updated service/provider/worker/UI error handling paths -- [ ] Added actionable GUI guidance for user-visible failures -- [ ] Added error reference IDs for traceability - -### Testing -- [ ] Added/updated tests per phase scope -- [ ] Ran targeted phase tests after each change batch -- [ ] Ran `not external` regression at each phase boundary -- [ ] Ran full suite before closeout - -### Documentation and Evidence -- [ ] `docs/error_handling.md` reviewed for alignment -- [ ] `docs/step7-results.md` includes executed command outputs -- [ ] Residual risks and deferred items explicitly recorded \ No newline at end of file diff --git a/docs/mvp/mvp.md b/docs/mvp/mvp.md deleted file mode 100644 index 696f4c2..0000000 --- a/docs/mvp/mvp.md +++ /dev/null @@ -1,209 +0,0 @@ -## MVP Definition: Historical Document Transcription System - -### 1. MVP Objective -Deliver the thinnest possible end-to-end vertical slice — a user uploads an image of a document, the system transcribes it via the OpenRouter Python SDK, and the user reads the resulting transcript — with just enough persistence and structure to validate the core value proposition: *can AI-driven transcription, guided by curated prompts, produce useful verbatim transcripts of historical family documents?* - -The MVP deliberately defers full-text search, export, revision history, MongoDB, and timeline assembly. These are additive features that don't need validation before the core transcription loop is proven. - ---- - -### 2. Core User Story -*As a family historian, I can upload a photo of a historical document, wait for it to be transcribed, and read the verbatim transcript — so I can evaluate whether this system will work for my thousands of documents.* - ---- - -### 3. In-Scope Requirements (from ```requirements.md```) - -| Requirement | ID | MVP Rationale | -| --- | --- | --- | -| End-to-end transcription with lifecycle state | REQ-0 | This is the MVP. | -| Upload one or more images from the web UI | REQ-1 | Core entry point. MVP supports single-image upload (multi-image is a stretch goal). | -| Asynchronous processing → transcription or failure | REQ-2 | Validates the AI transcription pipeline. | -| Persist and expose job states (queued → processing → transcribed/failed) | REQ-3 | Minimum feedback loop for the user. | -| Persist transcription output and failure details | REQ-4 | User must be able to read the result. | -| UI views for status and transcript reading | REQ-5 | The user needs to see what happened. | -| Background processing to keep UI responsive | REQ-6 | Essential for usability during long AI calls. | -| Centralized config and logging at startup | REQ-8 | Small effort, high payoff for debugging. | -| Store transcription prompts as Markdown files | REQ-12 | Core to the Prompt Curation Policy in intent.md. Start with a single prompt file. | - - -### Deferred to Post-MVP -| Requirement | ID | Why Deferred | -| --- | --- | --- | -| Lifespan-owned runtime resources (engine, session factory, etc.) | REQ-7 | Important for production robustness, but a simple global or module-level setup is adequate for MVP validation. | -| Docker Compose (app + PostgreSQL + optional MongoDB) | REQ-9 | MVP runs locally with SQLite to eliminate container overhead during rapid iteration. PostgreSQL migration is Stage 1 hardening. | -| Explicit, opt-in schema bootstrap | REQ-10 | MVP uses auto-create-tables at startup (SQLModel create_all). Production schema discipline comes after the model stabilizes. | -| Service-backed persistence for core data | REQ-11 | MVP uses a thin repository layer over SQLite. Full service abstraction follows once the domain model is proven. | - ---- - -### 4. MVP Feature Set -#### Feature 1: Document Upload (UI) -* A single NiceGUI page with a file-upload widget (accepts .jpg, .png, .tiff, .pdf). -* On upload: save the file to a local uploads/ directory, create a Document record, create a Job record with status queued. -* Minimal metadata capture: original filename, upload timestamp. - -#### Feature 2: Asynchronous Transcription Worker -* An in-process background worker (Python asyncio task or BackgroundTasks) that: - 1. Picks up queued jobs. - 2. Transitions status to processing. - 3. Sends the image + the curated Markdown prompt to an AI vision model via OpenRouter. - 4. On success: saves the transcript text, transitions to transcribed. - 5. On failure: saves the error detail, transitions to failed. - -#### Feature 3: Transcription Prompt (Markdown Asset) -* A single Markdown file (prompts/transcribe_document.md) encoding the verbatim transcription rules from intent.md (the Document Issues table, scholarly guidelines, etc.). -* The worker reads this file at invocation time and injects it as the system/user prompt. - -#### Feature 4: Job Status & Transcript Viewer (UI) -* A job list page showing all jobs with their current status (queued / processing / transcribed / failed). -* A transcript detail page showing: - * The original uploaded image (rendered inline). - * The transcription text (or the failure reason). - * Timestamp metadata. - -#### Feature 5: Minimal Persistence (SQLite + SQLModel) -* Three tables/models: - * Document: id, filename, file_path, uploaded_at. - * Job: id, document_id (FK), status, created_at, updated_at. - * Transcript: id, job_id (FK), text, error_detail, created_at. -* SQLite database file stored locally. Auto-created on first startup. - -#### Feature 6: Centralized Configuration -* A single config.py (or Pydantic BaseSettings) loading: - * PROVIDER (fixed to openrouter for MVP) - * OPENROUTER_API_KEY (required) - * PROVIDER_MODEL (default: OpenRouter model slug for vision transcription) - * OPENROUTER_HTTP_REFERER (optional; app attribution) - * OPENROUTER_APP_TITLE (optional; app attribution) - * DATABASE_URL (default: sqlite:///./transcription.db) - * UPLOAD_DIR (default: ./uploads) - * PROMPT_DIR (default: ./prompts) - -#### Feature 7: MVP Dependency Baseline (OpenRouter-Centric) -* Runtime dependencies: - * openrouter (official OpenRouter Python SDK) - * pydantic - * pydantic-settings - * sqlmodel -* Explicitly out of MVP runtime dependencies: - * google-genai (deferred until/if Gemini is introduced post-MVP) - ---- - -### 5. MVP Architecture (Simplified) - -```Apply -┌─────────────────────────────────────────────┐ -│ NiceGUI Web UI │ -│ ┌──────────────┐ ┌───────────────────┐ │ -│ │ Upload Page │ │ Jobs / Transcript │ │ -│ └──────┬───────┘ └───────┬───────────┘ │ -│ │ │ │ -│ ▼ ▼ │ -│ ┌───────────────────────────┐ │ -│ │ Application Service │ │ -│ │ (upload, job lifecycle) │ │ -│ └─────┬─────────────┬───────┘ │ -│ │ │ │ -│ ┌─────▼─────┐ ┌─────▼───────────────┐ │ -│ │ SQLite DB │ │ Background Worker │ │ -│ │ (SQLModel)│ │ → AI Vision Provider│ │ -│ └───────────┘ └─────────────────────┘ │ -│ │ │ -│ ┌─────▼──────┐ │ -│ │ prompts/ │ │ -│ │ *.md files │ │ -│ └────────────┘ │ -└─────────────────────────────────────────────┘ -``` - ---- - -#### 6. Proposed File Structure - -```Apply -project-root/ -├── docs/ # (existing) -├── prompts/ -│ └── transcribe_document.md # curated transcription prompt -├── src/ -│ └── transcription/ -│ ├── __init__.py -│ ├── app.py # FastAPI + NiceGUI app entrypoint -│ ├── config.py # Pydantic BaseSettings -│ ├── models.py # SQLModel: Document, Job, Transcript -│ ├── db.py # engine, session, create_all -│ ├── providers/ -│ │ ├── __init__.py -│ │ ├── base.py # provider interface (transcribe contract) -│ │ ├── openrouter.py # OpenRouter via official Python SDK -│ ├── services/ -│ │ ├── __init__.py -│ │ ├── upload.py # save file + create records -│ │ └── transcription.py # call provider, update job -│ ├── worker.py # background job loop -│ └── ui/ -│ ├── __init__.py -│ ├── upload_page.py # NiceGUI upload page -│ └── jobs_page.py # NiceGUI job list + detail -├── tests/ -│ ├── test_models.py -│ ├── test_upload.py -│ └── test_transcription.py -├── pyproject.toml -└── README.md -``` - ---- - -#### 7. MVP Validation Criteria -The MVP is considered validated when: - -1. ✅ A user can upload an image of a document through the browser. -2. ✅ The system asynchronously sends the image to the configured AI vision model with the curated prompt. -3. ✅ The transcript (or failure reason) is persisted and visible in the UI. -4. ✅ The transcription follows verbatim scholarly rules defined in intent.md (spot-checked by the user on real family documents). -5. ✅ The transcription prompt is stored as a standalone Markdown file and can be edited without code changes. -6. ✅ Job status transitions are visible: queued → processing → transcribed/failed. - ---- - -### 8. Key Feedback Questions the MVP Should Answer -These are the real unknowns this MVP exists to resolve: - -| # | Question | How We Learn | -| --- | --- | --- | -| 1 | Is AI transcription quality good enough for this document corpus? | User reviews 20–50 real transcriptions against originals. | -| 2 | Does the verbatim prompt produce scholarly-quality output, or does it need major rework? | Compare output to the Document Issues table rules in intent.md. | -| 3 | What document types are hardest (old cursive, faded ink, pencil, postcards)? | Track which uploads produce failed or low-quality results. | -| 4 | Is single-image upload sufficient, or is batch upload needed early? | User friction during real scanning sessions. | -| 5 | What metadata is missing that the user wishes they could capture at upload time? | User feedback after processing real batches. | - ---- - -#### 9. What Comes After MVP (Immediate Post-MVP) -Once the core transcription loop is validated, the next priorities (aligned to Architecture Stage 1) are: - -1. **Multi-image upload** — process a batch from a scanning session. -2. **PostgreSQL migration** — swap SQLite for containerized PostgreSQL (REQ-9, REQ-10). -3. **Revision history** — allow the user to edit/correct transcripts with immutable version tracking. -4. **Full-text search** — search across all accepted transcripts. -5. **Repository/service layer formalization** — proper ports/adapters as the domain model stabilizes. -6. **Docker Compose deployment** — containerize the app for reproducible operation. - ---- - -#### 10. Implementation Approach -Recommended build order for the MVP (each step produces a testable increment): - -| Step | Deliverable | Validates | -| --- | --- | --- | -| 1 | config.py + models.py + db.py — data layer with SQLite | Schema and config foundation | -| 2 | prompts/transcribe_document.md — curated prompt from intent.md | Prompt asset pattern | -| 3 | services/transcription.py + providers/ — call AI vision provider with prompt + image | Core AI integration | -| 4 | services/upload.py + worker.py — upload handling + background job loop | End-to-end pipeline (CLI-testable) | -| 5 | ui/upload_page.py + ui/jobs_page.py — NiceGUI pages | User-facing interface | -| 6 | tests/ — unit + integration tests Automated verification | - -This MVP is deliberately narrow: **one prompt, one provider (OpenRouter), one user, one image at a time, SQLite, no containers**. Every omission is intentional — the goal is to get real family documents through the transcription pipeline as fast as possible and let the quality of the output guide every subsequent decision. \ No newline at end of file diff --git a/docs/requirements.md b/docs/requirements.md index 64e8809..f616583 100644 --- a/docs/requirements.md +++ b/docs/requirements.md @@ -15,9 +15,9 @@ This page captures a SysML v1.6-style requirements baseline for the production s | ID | Category | Requirement | Risk | Verify Method | | --- | --- | --- | --- | --- | | REQ-0 | System | Provide end-to-end document transcription with persistent, inspectable lifecycle state. | medium | demonstration | -| REQ-1 | Functional | Allow users to upload one or more images or pdfs as sources from the web UI. | low | test | -| REQ-2 | Functional | Run each upload through asynchronous processing that returns a transcription or explicit failure. | high | test | -| REQ-3 | Functional | Persist and expose job states: upload, queued, processing, transcribed, failed, completed. | high | inspection | +| REQ-1 | Functional | Allow users to upload one or more images or PDFs as sources from the web UI. | low | test | +| REQ-2 | Functional | Run each upload through asynchronous processing that returns an original transcription or explicit failure. | high | test | +| REQ-3 | Functional | Persist and expose job states: queued, processing, transcribed, failed. | high | inspection | | REQ-4 | Functional | Persist transcription output, processing history, and failure details. | medium | test | | REQ-5 | Interface | Expose API and UI views for status inspection and completed transcription reading. | medium | demonstration | | REQ-6 | Performance | Trigger background processing on upload to preserve UI responsiveness. | medium | analysis | @@ -27,10 +27,11 @@ This page captures a SysML v1.6-style requirements baseline for the production s | REQ-10 | Design Constraint | Keep schema bootstrap explicit and opt-in; normal startup does not mutate production schema. | high | inspection | | REQ-11 | Design Constraint | Use service-backed persistence for core document and job data. | medium | inspection | | REQ-12 | Design Constraint | Store transcription prompts as individual Markdown artifacts for iterative refinement. | medium | inspection | +| REQ-13 | Functional | Allow users to revise transcription text from either original text or a previous revision. | low | test | ### Requirement Relationships -- Contains: REQ-0 contains REQ-1 through REQ-12. +- Contains: REQ-0 contains REQ-1 through REQ-13. - Derives: REQ-2 -> REQ-3, REQ-3 -> REQ-4. - Traces: REQ-5 -> REQ-3. - Refines: REQ-6 -> REQ-2. @@ -50,17 +51,17 @@ This page captures a SysML v1.6-style requirements baseline for the production s ### Satisfaction Mapping -- UI satisfies REQ-1, REQ-5. +- UI satisfies REQ-1, REQ-5, REQ-13. - API satisfies REQ-5. - GRAPH satisfies REQ-2, REQ-6. -- DBREL satisfies REQ-3, REQ-10. +- DBREL satisfies REQ-3, REQ-10, REQ-13. - DBDOC satisfies REQ-4, REQ-11. - OPS satisfies REQ-9. - PROMPTS satisfies REQ-12. ### Verification Mapping -- TESTS verifies REQ-1, REQ-2, REQ-3, REQ-4, REQ-5, REQ-10, REQ-11, REQ-12. +- TESTS verifies REQ-1, REQ-2, REQ-3, REQ-4, REQ-5, REQ-10, REQ-11, REQ-12, REQ-13. ## Requirement Notes diff --git a/docs/schema.md b/docs/schema.md index 19f3003..a5da226 100644 --- a/docs/schema.md +++ b/docs/schema.md @@ -55,17 +55,19 @@ erDiagram ## Table Relationships & Constraints * A document can consist of 0 or more content sources. A document can have 0 or more jobs. -* An source can belong to only one job (which contains the original transcription). A source can only belong to one document. A source can have 0 or more transcription revisions. +* A source can belong to only one job (which contains the original transcription). A source can only belong to one document. A source can have 0 or more transcription revisions. * A job can process one or more sources. A job can belong to only one document. -* A revision can belong to only one source. An source can have 0 or more revisions. +* A revision can belong to only one source. A source can have 0 or more revisions. +* `Job.text` stores the original immutable provider transcription. +* Revision rows are optional user-authored edits and may be derived from either the original transcription or a prior revision. --- ## Glossary * **Document** - Documents consist of one or more content sources and their related transcriptions. -* **Source** - A content source that is transcribed to text. It can either be an image (.jpg, .tiff, .png) or a pdf (.pdf) +* **Source** - A content source that is transcribed to text. It can either be an image (.jpg, .tiff, .png) or a PDF (.pdf). * **Image** - The scanned image of one page of a document. -* **PDF** - A pdf containing the image of one or more pages of a document. +* **PDF** - A PDF containing the image of one or more pages of a document. * **Job** - A processing job ingests one or more sources, sends them to an AI model along with a prompt for transcription, then stores the results. The results are immutable, *including the original transcription*. The user can create a revision of the original transcription, but the user cannot modify the original. -* Transcription - The text contained in a content source. A job creates the original transcription. -* Revision - A modification of a transcription. A job creates the original transcription. A user can create one or more transcription revisions. +* **Transcription** - The text contained in a content source. A job creates the original immutable transcription. A user can optionally create a revised transcription, or "revision". +* **Revision** - A revision is a user-created modification of an existing transcription. A user can create a revision from either the original transcription or a previous revision, but it is entirely optional. Some original transcriptions will have no revisions. \ No newline at end of file diff --git a/docs/ver1/ver1.md b/docs/ver1/ver1.md index 9bccb2e..0bdc028 100644 --- a/docs/ver1/ver1.md +++ b/docs/ver1/ver1.md @@ -1,320 +1,189 @@ # Version 1 Implementation Plan -This plan defines the path from MVP to **Version 1 complete**. -The objective is to deliver the full scoped product with production readiness, while explicitly separating refinements/enhancements into a future document. +This plan defines the path from current implementation to **Version 1 complete**, aligned to the updated domain model: + +- `Document` groups one or more content `Source` records +- `Job` owns original immutable provider output (`text`) and processing metadata +- `Revision` stores optional user-authored edits linked to a `Source` + +The objective is to complete V1 scope with production readiness while keeping non-V1 enhancements out of active delivery. --- -## 0) Plan Governance & Scope Control (Foundation) +## V1 Completion Definition -**Goal:** Keep execution focused on V1 completion, not optimization/perfection. +V1 is complete when all of the following are true: -### Implementation Steps -1. Create and maintain a **V1 Traceability Matrix**: - - Requirement ID - - Current status (`done`, `partial`, `not started`) - - Owner - - Validation method -2. Define V1 completion gates: - - Functional complete - - Operationally complete - - Production-ready complete -3. Snapshot the MVP baseline (tag/changelog reference). -4. Create a standing rule: any non-V1 idea is logged to a separate enhancements backlog document (to be named later), not added to active V1 scope unless explicitly approved. +1. **Functional complete** + - Upload, queue, processing, status display, and transcription result inspection work end-to-end. + - Optional revision workflow is implemented (create/list/view). +2. **Data-model complete** + - Runtime behavior, persistence, and tests all align to `Document` / `Source` / `Job` / `Revision`. +3. **Operational complete** + - Error handling, logs, and runbooks support reliable operation. +4. **Documentation complete** + - Architecture, requirements, schema, error handling, and index are consistent and current. + +--- + +## Phase 1 — Data Contract Stabilization (Schema-First) + +**Goal:** Lock a single canonical contract before further feature work. + +### Tasks +1. Confirm and document invariants: + - `Job.text` is original immutable transcription output. + - `Revision` is optional and user-authored. + - Revisions may derive from original `Job.text` or a previous revision. +2. Verify relationship cardinality assumptions: + - `Document` -> many `Source` + - `Document` -> many `Job` + - `Source` -> one `Job` + - `Source` -> many `Revision` +3. Ensure field naming consistency (`date_created`, `date_updated`, `date_uploaded`) across code and docs. +4. Freeze V1 status lifecycle to current implementation (`queued`, `processing`, `transcribed`, `failed`). ### Deliverables -- `docs/ver1/ver1.md` (this plan) -- V1 traceability artifact (linked from here when created) +- Updated `docs/schema.md` and `docs/requirements.md` traceability alignment. +- Explicit V1 data invariants section in architecture docs. ### Exit Criteria -- Every in-scope requirement has explicit ownership and status. -- Scope-change process is agreed and followed. +- No conflicting definitions of ownership/cardinality/status remain in docs. --- -## 1) Architecture Consolidation +## Phase 2 — Service Layer Refactor To New Model -**Goal:** Align implementation with the intended architecture and reduce MVP shortcuts. +**Goal:** Remove all obsolete `Transcript` assumptions from service/workflow code. -### Implementation Steps -1. Compare implemented modules/components with architecture documentation. -2. Identify and classify architectural debt: - - Temporary coupling - - Missing interfaces - - Placeholder services/components -3. Resolve high-risk architectural gaps first. -4. Record key decisions and tradeoffs in ADRs. +### Tasks +1. Refactor `services/transcription.py`: + - Replace transcript CRUD assumptions with job-output + revision operations. +2. Refactor `services/jobs.py`: + - Replace old timestamp/relationship accessors with current model fields. +3. Refactor `services/documents.py` and `services/store.py`: + - Ensure upload creates and links `Document`, `Source`, and `Job` correctly. +4. Refactor `services/workflows.py`: + - Persist original provider output to `Job`. + - Persist failure detail to `Job.error_detail`. + - Use `Revision` only for user-authored edits. ### Deliverables -- Updated architecture diagrams and boundaries -- ADR entries for major decisions +- Service layer fully aligned with new schema. ### Exit Criteria -- Architecture documentation reflects system reality. -- Critical architecture risks are addressed or scheduled with owners/dates. +- No service module imports or persists `Transcript` model artifacts. --- -## 2) Error Handling & Reliability Hardening +## Phase 3 — UI Contract Alignment -**Goal:** Ensure predictable, safe behavior under failure conditions. +**Goal:** Align pages/components to source/job/revision semantics. -### Implementation Steps -1. Standardize error taxonomy and envelope format across all layers. -2. Ensure clear distinction between: - - User-facing errors - - Internal/system errors - - Retryable vs non-retryable failures -3. Add resilience controls where needed: - - Timeouts - - Retries with backoff - - Circuit breaking / fallback logic -4. Add failure-path tests for critical workflows. +### Tasks +1. Update job detail and related UI components: + - Display original immutable transcription from `Job.text`. + - Display optional revisions sourced from `Source.revisions`. +2. Replace legacy terminology in UI copy: + - “transcript history” -> “revision history” where appropriate. +3. Align date fields with new schema naming. +4. Preserve clear user messaging when no revisions exist. ### Deliverables -- Error code catalog/reference -- Failure mode test coverage for critical paths +- Updated jobs page and detail components. ### Exit Criteria -- Error behavior is consistent across major flows. -- Known failure scenarios are tested and pass. +- UI behavior and labels match documentation and domain model. --- -## 3) Functional Completion by Requirement Domain +## Phase 4 — Database Bootstrap, Migration, and Safety -**Goal:** Complete all V1 functional requirements in a risk-aware order. +**Goal:** Make schema transition safe in dev/test and repeatable for deployment. -### Recommended Order -1. Business-critical end-user flows -2. Data integrity and consistency capabilities -3. Admin/operational controls -4. Lower-priority UX and quality-of-life items that are in V1 scope - -### Implementation Steps -For each requirement slice: -1. Finalize contract/schema -2. Implement domain logic -3. Implement persistence/state changes -4. Integrate API/UI -5. Add automated tests -6. Update docs +### Tasks +1. Update bootstrap compatibility logic in `db/operations.py`: + - Remove obsolete transcript-table assumptions. + - Add forward-compatible patches for current tables only. +2. Define migration/backfill approach for existing local data. +3. Document rollback and recovery steps. +4. Rehearse migration path against representative data. ### Deliverables -- Requirement completion report with validation evidence +- Migration/upgrade runbook. +- Validated bootstrap behavior for dev/test. ### Exit Criteria -- All V1 “must-have” requirements are complete and validated. +- Migration path is documented and tested with no unresolved data-loss risk. --- -## 4) Data Model, Migration, and Backfill Safety +## Phase 5 — Test Suite Realignment -**Goal:** Ensure data model and migrations are production-safe. +**Goal:** Restore full confidence after the schema redesign. -### Implementation Steps -1. Validate schema against final V1 domain needs. -2. Implement forward-safe migrations. -3. Define rollback/mitigation plans for migration failures. -4. Build and verify backfill scripts (if needed). -5. Add migration rehearsal in staging with representative data. +### Tasks +1. Rewrite model tests for: + - `Document`, `Source`, `Job`, `Revision` relationships and invariants. +2. Rewrite service/integration tests: + - Worker success/failure paths using `Job.text` / `Job.error_detail`. + - Optional revision creation and lineage behavior. +3. Update UI tests for new job-detail/revision rendering behavior. +4. Re-enable strict CI quality gates (lint, type, tests). ### Deliverables -- Migration runbook -- Backfill verification checklist +- Updated test matrix and passing CI. ### Exit Criteria -- Migration plan validated in staging. -- No unresolved data-loss risk for V1 rollout. +- Critical user flows and failure paths are covered and green. --- -## 5) Security, Access Control, and Compliance Baseline +## Phase 6 — Reliability, Operations, and Release Readiness -**Goal:** Close MVP security gaps and establish V1 baseline controls. +**Goal:** Ensure V1 is operable and launch-safe. -### Implementation Steps -1. Complete authn/authz coverage for all routes/actions. -2. Enforce input validation and output sanitization. -3. Verify secret management and credential rotation process. -4. Add audit logging for sensitive operations. -5. Run dependency/security scanning in CI and remediate findings. +### Tasks +1. Verify error taxonomy behavior across UI/API/service/worker. +2. Confirm structured logging includes relevant identifiers (`job_id`, `document_id`, `source_id` when applicable). +3. Validate retry behavior and terminal failure handling. +4. Finalize release checklist, deployment steps, and rollback procedure. +5. Execute final acceptance run against requirements traceability. ### Deliverables -- Security checklist with status -- Threat/risk update for V1 scope +- V1 release checklist and acceptance evidence. ### Exit Criteria -- No unresolved critical/high vulnerabilities for V1 launch. -- Access control behavior verified by tests. +- Stakeholder sign-off and launch readiness achieved. --- -## 6) Observability & Operability +## Requirement Traceability Focus -**Goal:** Make system behavior observable and supportable in production. +The plan must keep clear evidence against these requirement groups: -### Implementation Steps -1. Standardize structured logging and correlation IDs. -2. Add core metrics: - - Latency - - Throughput - - Error rates - - Resource saturation -3. Add tracing for critical request/workflow paths. -4. Define SLOs/SLIs and alert thresholds. -5. Prepare incident response and rollback runbooks. +- **Core flow:** REQ-0 to REQ-6 +- **Runtime and operations constraints:** REQ-7 to REQ-12 +- **Revision workflow:** REQ-13 -### Deliverables -- Dashboards and alerts -- Operations runbooks +A lightweight traceability table should be maintained with: -### Exit Criteria -- Team can detect, triage, and remediate incidents quickly. -- Core production signals are available and reliable. +- requirement ID +- implementation status (`not started` / `in progress` / `done`) +- validation evidence (test name, screenshot, or runbook step) --- -## 7) Test Strategy Expansion & Quality Gates +## Suggested Execution Rhythm -**Goal:** Raise confidence for repeatable, low-risk releases. - -### Implementation Steps -1. Expand unit and integration tests across V1 features. -2. Add contract tests between key components/services. -3. Add end-to-end tests for critical user journeys. -4. Add non-functional tests where relevant: - - Performance/load - - Soak - - Failure-injection scenarios -5. Enforce CI quality gates (tests, lint, type checks, security scans). - -### Deliverables -- Test matrix with ownership -- CI gate definition and thresholds - -### Exit Criteria -- Critical-path regressions are blocked automatically. -- Test coverage and reliability thresholds meet V1 targets. - ---- - -## 8) Performance & Scalability Validation - -**Goal:** Meet expected V1 performance at projected load. - -### Implementation Steps -1. Define performance budgets per key flow. -2. Benchmark current behavior in staging. -3. Optimize bottlenecks (queries, caching, concurrency, etc.). -4. Re-test after each optimization and compare against budget. -5. Document known limits and safe operating bounds. - -### Deliverables -- Performance benchmark report -- Optimization log - -### Exit Criteria -- V1 performance targets met for expected usage profile. - ---- - -## 9) Release Engineering & Environment Readiness - -**Goal:** Make deployment repeatable, controlled, and reversible. - -### Implementation Steps -1. Harden CI/CD pipeline with clear promotion gates. -2. Ensure config parity and consistency across environments. -3. Define rollout strategy (phased/canary/limited release as applicable). -4. Validate rollback procedures in staging. -5. Produce release checklist and ownership model. - -### Deliverables -- Release playbook -- Environment readiness checklist - -### Exit Criteria -- Deployment and rollback are rehearsed and reliable. -- Release process is executable without tribal knowledge. - ---- - -## 10) Documentation Completion - -**Goal:** Ensure V1 can be built, operated, and supported from documentation. - -### Implementation Steps -1. Update core project docs to match final V1 behavior: - - Architecture - - Error handling - - Requirements status - - Index/navigation - - Intent alignment summary -2. Add operator troubleshooting guides. -3. Add integration/API examples for consumers. -4. Publish changelog/version notes for V1. - -### Deliverables -- Updated documentation set for V1 -- V1 release notes - -### Exit Criteria -- A new team member can run/support the system using docs alone. - ---- - -## 11) Final Validation, UAT, and Launch - -**Goal:** Confirm readiness and launch V1 safely. - -### Implementation Steps -1. Run full-system acceptance validation against the V1 traceability matrix. -2. Conduct stakeholder UAT and capture sign-off. -3. Execute production readiness review. -4. Launch in controlled phases and monitor key signals. - -### Deliverables -- UAT/PRR sign-off records -- Launch checklist and monitoring plan - -### Exit Criteria -- Stakeholder approval achieved. -- Launch metrics are stable within defined thresholds. - ---- - -## 12) Post-Launch Stabilization (30–60 Days) - -**Goal:** Consolidate V1 in production before major expansion. - -### Implementation Steps -1. Track incidents, defects, and user feedback. -2. Prioritize stabilization fixes with short cycle times. -3. Remove temporary flags/mitigations introduced during launch. -4. Produce post-launch retrospective and handoff to standard roadmap cadence. - -### Deliverables -- Stabilization report -- Prioritized backlog update - -### Exit Criteria -- Incident/error rates converge to steady-state targets. -- V1 transitions from launch mode to normal operations. - ---- - -## Recommended Execution Rhythm - -- **Weekly:** Requirement closure + risk review -- **Biweekly:** Release train with quality gates -- **Milestone reviews:** After phases 2, 6, 9, and 11 +- **Weekly:** requirement status and risk review +- **Per PR:** contract checks (model names, field names, lifecycle values) +- **Milestone checks:** end of Phases 2, 4, and 6 --- ## Scope Discipline Rule (V1 Focus) -To preserve delivery focus: -- V1 execution prioritizes completion of scoped requirements. -- Refinements/enhancements are captured in a separate future document and backlog. -- Only explicitly approved scope changes may enter this plan. \ No newline at end of file +- Only work required to satisfy V1 requirements enters this plan. +- Nice-to-have enhancements are captured in a separate backlog document. +- Schema or contract changes after Phase 1 require explicit approval and traceability impact review. \ No newline at end of file