diff --git a/docs/step1.md b/docs/step1.md new file mode 100644 index 0000000..d123283 --- /dev/null +++ b/docs/step1.md @@ -0,0 +1,471 @@ +I now have a complete understanding of all the constraints. Here is the detailed implementation plan for Step 1: + +--- + +# Step 1 Implementation Plan: `config.py` + `models.py` + `db.py` + +## Purpose + +Establish the foundational data layer and configuration system that every subsequent MVP step builds on. At the end of this step, the project has a runnable Python package with a validated schema, typed configuration, and a test suite proving the data layer works — before any UI, worker, or AI provider code exists. + +--- + +## 1. Prerequisite: Project Structure Scaffolding + +Before writing any logic, create the package skeleton so imports work correctly. + +### Files to create (empty `__init__.py` stubs) + +``` +src/ +└── transcription/ + ├── __init__.py + ├── providers/ + │ └── __init__.py + ├── services/ + │ └── __init__.py + └── ui/ + └── __init__.py +``` + +### Files to create (with logic — the Step 1 deliverables) + +``` +src/transcription/config.py +src/transcription/models.py +src/transcription/db.py +``` + +### Test files to create + +``` +tests/ +├── __init__.py +├── conftest.py +├── test_config.py +├── test_models.py +└── test_db.py +``` + +### Update `pyproject.toml` + +Add the dependencies that Step 1 requires and won't change later: + +```toml pyproject.toml +[project] +name = "transcription" +version = "0.1.0" +description = "Historical document transcription system" +readme = "README.md" +requires-python = ">=3.12" +dependencies = [ + "google-genai>=1.0.0", + "openai>=2.43.0", + "pydantic>=2.13.4", + "pydantic-settings>=2.9.1", + "sqlmodel>=0.0.25", +] + +[project.optional-dependencies] +dev = [ + "pytest>=8.0", + "pytest-asyncio>=0.25", +] +``` + +Key additions: +- **`pydantic-settings`** — for `BaseSettings` with env-var loading (this was split out of `pydantic` core in v2) +- **`sqlmodel`** — provides SQLModel (which bundles SQLAlchemy + Pydantic model integration) and the SQLite driver +- **`pytest` + `pytest-asyncio`** — in `dev` extras for test execution + +### Delete `hello.py` + +The placeholder file is no longer needed. + +--- + +## 2. `config.py` — Centralized Configuration + +**Satisfies:** REQ-8 (centralized config and logging at startup) + +### Design Decisions + +| Decision | Rationale | +|----------|-----------| +| Use `pydantic-settings` `BaseSettings` | Type-safe, validates on construction, loads from env vars and `.env` files automatically | +| `PROVIDER` as a string enum (`openrouter`, `gemini`) | Drives provider factory in Step 3; validated at startup, not at first API call | +| `PROVIDER_MODEL` defaults to `None` | Each provider adapter (Step 3) supplies its own sensible default when `None`; avoids config knowing about provider-specific model names | +| `PROVIDER_BASE_URL` defaults to `None` | Only needed to override OpenRouter's base URL; Gemini ignores it. `None` means "use provider default" | +| `DATABASE_URL` defaults to SQLite | Zero-setup local development; PostgreSQL swap is a single env-var change post-MVP | +| `UPLOAD_DIR` and `PROMPT_DIR` as `Path` objects | Enables `.mkdir(parents=True, exist_ok=True)` and path validation at startup | +| Logging configured in a `setup_logging()` function | Called once at startup; uses stdlib `logging` with a simple format. No third-party logging library needed for MVP | + +### Proposed Implementation + +```python src/transcription/config.py +"""Centralized application configuration. + +All settings are loaded from environment variables (or a .env file) +once at startup. Provider-specific defaults (model names, base URLs) +are resolved by the provider adapters, not here. +""" + +from enum import StrEnum +from functools import lru_cache +from pathlib import Path +import logging + +from pydantic_settings import BaseSettings, SettingsConfigDict + + +class Provider(StrEnum): + OPENROUTER = "openrouter" + GEMINI = "gemini" + + +class Settings(BaseSettings): + model_config = SettingsConfigDict( + env_file=".env", + env_file_encoding="utf-8", + extra="ignore", + ) + + # --- AI provider --- + provider: Provider = Provider.OPENROUTER + provider_api_key: str + provider_model: str | None = None + provider_base_url: str | None = None + + # --- persistence --- + database_url: str = "sqlite:///./transcription.db" + + # --- filesystem paths --- + upload_dir: Path = Path("./uploads") + prompt_dir: Path = Path("./prompts") + + +@lru_cache(maxsize=1) +def get_settings() -> Settings: + """Return the singleton Settings instance. + + Cached so the entire application shares one validated config. + """ + return Settings() + + +def setup_logging() -> None: + """Configure root logging once at startup.""" + logging.basicConfig( + level=logging.INFO, + format="%(asctime)s | %(levelname)-8s | %(name)s | %(message)s", + datefmt="%Y-%m-%d %H:%M:%S", + ) +``` + +### Key Behaviors + +- **Startup validation**: If `PROVIDER_API_KEY` is missing from the environment, `Settings()` raises a `ValidationError` immediately — the app won't start with a missing key. +- **`.env` support**: Developers can create a `.env` file in the project root for local keys; it's never committed (already covered by the existing `.gitignore` pattern or a new entry). +- **`extra="ignore"`**: Unknown env vars don't cause errors, keeping the config resilient to unrelated environment variables. +- **`lru_cache`**: `get_settings()` is the single access point. All modules import and call this function rather than constructing `Settings` directly. + +### `.env` template (not committed — add to `.gitignore`) + +```bash .env.example +PROVIDER=openrouter +PROVIDER_API_KEY=sk-or-... +# PROVIDER_MODEL= # optional: provider adapter supplies default +# PROVIDER_BASE_URL= # optional: override provider endpoint +# DATABASE_URL=sqlite:///./transcription.db +# UPLOAD_DIR=./uploads +# PROMPT_DIR=./prompts +``` + +### `.gitignore` addition + +```gitignore .gitignore +# ... existing entries ... + +# Environment secrets +.env +``` + +--- + +## 3. `models.py` — SQLModel Domain Models + +**Satisfies:** REQ-3 (persist and expose job states), REQ-4 (persist transcription output and failure details) + +### Design Decisions + +| Decision | Rationale | +|----------|-----------| +| Three models: `Document`, `Job`, `Transcript` | Minimal set from MVP Feature 5. One-to-many from Document→Job and one-to-one from Job→Transcript | +| `JobStatus` as a `StrEnum` | Readable in the database (`"queued"` not `1`), type-safe in Python, trivially serializable to JSON for the UI | +| Status values: `queued`, `processing`, `transcribed`, `failed` | Matches MVP Feature 2 lifecycle. REQ-3 also lists `upload` and `completed` — these are deferred to post-MVP when revision/review workflows exist | +| UUIDs for primary keys | Avoids auto-increment collision concerns if we later move to PostgreSQL; safe for distributed ID generation; `uuid4` is simple | +| `uploaded_at`, `created_at`, `updated_at` as UTC `datetime` | Timezone-naive UTC by convention for MVP. Sufficient for single-user, single-timezone operation | +| `Transcript.text` is nullable | A failed job creates a Transcript with `text=None` and `error_detail` populated, keeping the query model uniform | +| Relationships via SQLModel `Relationship` | Enables `document.jobs` and `job.transcript` navigation in service code without manual joins | + +### Proposed Implementation + +```python src/transcription/models.py +"""SQLModel domain models for the transcription system. + +Three models capture the MVP lifecycle: + Document → one-to-many → Job → one-to-one → Transcript +""" + +from datetime import datetime, timezone +from enum import StrEnum +from uuid import UUID, uuid4 + +from sqlmodel import Field, Relationship, SQLModel + + +class JobStatus(StrEnum): + QUEUED = "queued" + PROCESSING = "processing" + TRANSCRIBED = "transcribed" + FAILED = "failed" + + +class Document(SQLModel, table=True): + """An uploaded document image.""" + + id: UUID = Field(default_factory=uuid4, primary_key=True) + filename: str + file_path: str + uploaded_at: datetime = Field( + default_factory=lambda: datetime.now(timezone.utc), + ) + + # --- relationships --- + jobs: list["Job"] = Relationship(back_populates="document") + + +class Job(SQLModel, table=True): + """A transcription job tied to a single document.""" + + id: UUID = Field(default_factory=uuid4, primary_key=True) + document_id: UUID = Field(foreign_key="document.id") + status: JobStatus = Field(default=JobStatus.QUEUED) + created_at: datetime = Field( + default_factory=lambda: datetime.now(timezone.utc), + ) + updated_at: datetime = Field( + default_factory=lambda: datetime.now(timezone.utc), + ) + + # --- relationships --- + document: Document = Relationship(back_populates="jobs") + transcript: "Transcript | None" = Relationship(back_populates="job") + + +class Transcript(SQLModel, table=True): + """The output of a transcription job.""" + + id: UUID = Field(default_factory=uuid4, primary_key=True) + job_id: UUID = Field(foreign_key="job.id", unique=True) + text: str | None = None + error_detail: str | None = None + created_at: datetime = Field( + default_factory=lambda: datetime.now(timezone.utc), + ) + + # --- relationships --- + job: Job = Relationship(back_populates="transcript") +``` + +### Entity-Relationship Summary + +``` +┌──────────┐ ┌──────────┐ ┌─────────────┐ +│ Document │ 1───* │ Job │ 1───1 │ Transcript │ +├──────────┤ ├──────────┤ ├─────────────┤ +│ id (PK) │ │ id (PK) │ │ id (PK) │ +│ filename │ │ doc_id │──FK──▶│ job_id (FK) │ +│ file_path│ │ status │ │ text │ +│ uploaded │ │ created │ │ error_detail│ +│ │ │ updated │ │ created │ +└──────────┘ └──────────┘ └─────────────┘ +``` + +### Why Only Four Status Values + +REQ-3 lists six states: `upload`, `queued`, `processing`, `transcribed`, `failed`, `completed`. The MVP simplifies this: + +| REQ-3 State | MVP Treatment | +|-------------|---------------| +| `upload` | Implicit — the Document record exists before a Job is created. No separate job state needed. | +| `queued` | ✅ Included — job created, waiting for worker pickup | +| `processing` | ✅ Included — worker is actively transcribing | +| `transcribed` | ✅ Included — AI output received and stored | +| `failed` | ✅ Included — error captured | +| `completed` | Deferred — implies human review/acceptance. In MVP, `transcribed` is the terminal success state. | + +--- + +## 4. `db.py` — Database Engine and Session Management + +**Satisfies:** MVP Feature 5 (SQLite auto-created on first startup) + +### Design Decisions + +| Decision | Rationale | +|----------|-----------| +| Module-level `create_engine` + `Session` factory | REQ-7 (lifespan-owned resources) is deferred. A module-level engine is adequate for MVP's single-process, single-user operation | +| `create_all()` as an explicit function | Called at app startup. MVP auto-creates tables (REQ-10 deferred), but the function is isolated so it's easy to gate behind a flag later | +| `get_session()` as a generator | Standard FastAPI/SQLModel pattern — yields a session, ensures cleanup. Compatible with `Depends()` when the API layer arrives in Step 5 | +| `echo=False` default | Keeps logs clean. Can be toggled for debugging | + +### Proposed Implementation + +```python src/transcription/db.py +"""Database engine, session factory, and schema bootstrap. + +MVP uses SQLite with auto-create-tables at startup. +PostgreSQL migration is a post-MVP configuration change. +""" + +from collections.abc import Generator + +from sqlmodel import Session, SQLModel, create_engine + +from transcription.config import get_settings + + +def _build_engine(): + settings = get_settings() + connect_args = {} + if settings.database_url.startswith("sqlite"): + connect_args["check_same_thread"] = False + return create_engine( + settings.database_url, + echo=False, + connect_args=connect_args, + ) + + +engine = _build_engine() + + +def create_all() -> None: + """Create all tables. Called once at application startup.""" + SQLModel.metadata.create_all(engine) + + +def get_session() -> Generator[Session, None, None]: + """Yield a database session and ensure cleanup.""" + with Session(engine) as session: + yield session +``` + +### SQLite-Specific Note + +`check_same_thread=False` is required for SQLite when the session may be accessed from different threads (e.g., a background worker on a different thread than the request handler). This setting is harmless and ignored for PostgreSQL connection strings. + +--- + +## 5. Test Plan + +### `tests/conftest.py` — Shared Fixtures + +```python tests/conftest.py +"""Shared test fixtures. + +Every test gets a fresh in-memory SQLite database so tests are +isolated, fast, and leave no artifacts on disk. +""" + +import pytest +from sqlmodel import Session, SQLModel, create_engine + + +@pytest.fixture +def session(): + """Provide a clean database session for each test.""" + engine = create_engine( + "sqlite://", # in-memory + connect_args={"check_same_thread": False}, + ) + SQLModel.metadata.create_all(engine) + with Session(engine) as session: + yield session +``` + +### `tests/test_config.py` — Configuration Tests + +| Test | What It Verifies | +|------|------------------| +| `test_settings_loads_from_env` | `Settings` constructs successfully when `PROVIDER_API_KEY` is set via env var | +| `test_settings_requires_api_key` | `Settings()` raises `ValidationError` when `PROVIDER_API_KEY` is missing | +| `test_provider_defaults_to_openrouter` | Default provider is `openrouter` when not explicitly set | +| `test_provider_rejects_invalid_value` | Setting `PROVIDER=invalid` raises `ValidationError` | +| `test_optional_fields_default_to_none` | `provider_model` and `provider_base_url` are `None` when unset | +| `test_path_fields_are_path_objects` | `upload_dir` and `prompt_dir` are `Path` instances | + +### `tests/test_models.py` — Model & Relationship Tests + +| Test | What It Verifies | +|------|------------------| +| `test_create_document` | A `Document` can be persisted and read back with correct fields | +| `test_document_defaults` | `id` is auto-generated UUID, `uploaded_at` is populated | +| `test_create_job_with_document` | A `Job` linked to a `Document` via FK persists correctly | +| `test_job_defaults` | Default status is `queued`, `created_at` and `updated_at` are populated | +| `test_job_status_transitions` | Status can be updated from `queued` → `processing` → `transcribed` | +| `test_job_status_to_failed` | Status can be updated from `processing` → `failed` | +| `test_create_transcript_success` | A `Transcript` with `text` set and `error_detail=None` persists correctly | +| `test_create_transcript_failure` | A `Transcript` with `text=None` and `error_detail` set persists correctly | +| `test_document_jobs_relationship` | `document.jobs` returns the linked `Job` list | +| `test_job_transcript_relationship` | `job.transcript` returns the linked `Transcript` | +| `test_transcript_job_id_unique` | Inserting two transcripts with the same `job_id` raises an integrity error | + +### `tests/test_db.py` — Database Bootstrap Tests + +| Test | What It Verifies | +|------|------------------| +| `test_create_all_creates_tables` | After `create_all()`, the expected tables (`document`, `job`, `transcript`) exist in the database | +| `test_get_session_yields_session` | `get_session()` yields a usable `Session` object | +| `test_session_cleanup_on_exit` | After the generator is exhausted, the session is closed | + +--- + +## 6. Step 1 Completion Checklist + +When all of the following are true, Step 1 is done and Step 2 can begin: + +| # | Criterion | How to Verify | +|---|-----------|---------------| +| 1 | `src/transcription/` package exists with `config.py`, `models.py`, `db.py` | `ls` / file inspection | +| 2 | Empty `__init__.py` stubs exist for `providers/`, `services/`, `ui/` | `ls` / file inspection | +| 3 | `Settings` loads from environment and validates `PROVIDER_API_KEY` is present | `test_config.py` passes | +| 4 | `Document`, `Job`, `Transcript` models create tables in SQLite | `test_models.py` passes | +| 5 | `JobStatus` enum has exactly four values: `queued`, `processing`, `transcribed`, `failed` | `test_models.py` passes | +| 6 | Foreign key relationships work: Document→Job→Transcript | `test_models.py` passes | +| 7 | `create_all()` bootstraps the schema; `get_session()` yields a working session | `test_db.py` passes | +| 8 | All tests pass: `uv run pytest tests/` | CI / local run | +| 9 | `hello.py` is deleted | File inspection | +| 10 | `pyproject.toml` includes `sqlmodel`, `pydantic-settings`, `pytest`, `pytest-asyncio` | File inspection | +| 11 | `.env.example` documents all config vars; `.env` is in `.gitignore` | File inspection | + +--- + +## 7. What This Step Does NOT Include + +Explicitly out of scope to prevent scope creep: + +| Excluded | Reason | +|----------|--------| +| FastAPI / NiceGUI app entrypoint | Step 5 | +| Provider adapters (`openrouter.py`, `gemini.py`) | Step 3 | +| Upload service logic | Step 4 | +| Worker / background processing | Step 4 | +| Transcription prompt files | Step 2 | +| Alembic or migration tooling | Post-MVP (REQ-10 deferred) | +| Async session factory | Post-MVP (REQ-7 deferred) | + +--- + +This plan produces a fully tested, importable data foundation. Every subsequent step imports from `transcription.config`, `transcription.models`, and `transcription.db` without modification. When you're ready, switch to **Agent mode** and I'll implement it. \ No newline at end of file