transcription/docs/step1.md

I now have a complete understanding of all the constraints. Here is the detailed implementation plan for Step 1:

---

# Step 1 Implementation Plan: `config.py` + `models.py` + `db.py`

## Purpose

Establish the foundational data layer and configuration system that every subsequent MVP step builds on. At the end of this step, the project has a runnable Python package with a validated schema, typed configuration, and a test suite proving the data layer works — before any UI, worker, or AI provider code exists.

---

## 1. Prerequisite: Project Structure Scaffolding

Before writing any logic, create the package skeleton so imports work correctly.

### Files to create (empty `__init__.py` stubs)

```
src/
└── transcription/
    ├── __init__.py
    ├── providers/
    │   └── __init__.py
    ├── services/
    │   └── __init__.py
    └── ui/
        └── __init__.py
```

### Files to create (with logic — the Step 1 deliverables)

```
src/transcription/config.py
src/transcription/models.py
src/transcription/db.py
```

### Test files to create

```
tests/
├── __init__.py
├── conftest.py
├── test_config.py
├── test_models.py
└── test_db.py
```

### Update `pyproject.toml`

Add the dependencies that Step 1 requires and won't change later:

```toml pyproject.toml
[project]
name = "transcription"
version = "0.1.0"
description = "Historical document transcription system"
readme = "README.md"
requires-python = ">=3.12"
dependencies = [
    "openrouter>=0.7.0",
    "pydantic>=2.13.4",
    "pydantic-settings>=2.9.1",
    "sqlmodel>=0.0.25",
]

[project.optional-dependencies]
dev = [
    "pytest>=8.0",
    "pytest-asyncio>=0.25",
]

[tool.pytest.ini_options]
addopts = "--strict-markers -q"
markers = [
    "unit: pure logic tests with no external dependencies",
    "integration: tests that touch framework or database contracts",
    "external: tests that call external services (slow, requires credentials)",
]
```

Key additions:
- **`openrouter`** — official OpenRouter Python SDK used for model calls
- **`pydantic-settings`** — for `BaseSettings` with env-var loading (this was split out of `pydantic` core in v2)
- **`sqlmodel`** — provides SQLModel (which bundles SQLAlchemy + Pydantic model integration) and the SQLite driver
- **`pytest` + `pytest-asyncio`** — in `dev` extras for test execution
- **`[tool.pytest.ini_options]`** — strict marker checking enabled from the start; markers registered upfront per pytesting skill conventions

### Delete `hello.py`

The placeholder file is no longer needed.

---

## 2. `config.py` — Centralized Configuration

**Satisfies:** REQ-8 (centralized config and logging at startup)

### Design Decisions

| Decision | Rationale |
|----------|-----------|
| Use `pydantic-settings` `BaseSettings` | Type-safe, validates on construction, loads from env vars and `.env` files automatically |
| `PROVIDER` constrained to `openrouter` for MVP | Keeps configuration explicit while avoiding premature multi-provider complexity |
| `OPENROUTER_API_KEY` required | Matches official SDK docs and avoids ambiguous provider-agnostic naming |
| `PROVIDER_MODEL` defaults to `None` | OpenRouter adapter (Step 3) supplies a sensible default when `None` |
| `OPENROUTER_HTTP_REFERER` and `OPENROUTER_APP_TITLE` optional | Matches SDK optional app-attribution fields |
| `DATABASE_URL` defaults to SQLite | Zero-setup local development; PostgreSQL swap is a single env-var change post-MVP |
| `UPLOAD_DIR` and `PROMPT_DIR` as `Path` objects | Enables `.mkdir(parents=True, exist_ok=True)` and path validation at startup |
| Logging configured via `logging.config.dictConfig` in `setup_logging()` | Centralized, explicit formatter/handler/root logger topology; called once at startup with `disable_existing_loggers=False` |

### Proposed Implementation

```python src/transcription/config.py
"""Centralized application configuration.

All settings are loaded from environment variables (or a .env file)
once at startup. Provider-specific defaults (model names, base URLs)
are resolved by the provider adapters, not here.
"""

from enum import StrEnum
from functools import lru_cache
from pathlib import Path
import logging
import logging.config

from pydantic_settings import BaseSettings, SettingsConfigDict


class Provider(StrEnum):
    OPENROUTER = "openrouter"


class Settings(BaseSettings):
    model_config = SettingsConfigDict(
        env_file=".env",
        env_file_encoding="utf-8",
        extra="ignore",
    )

    # --- AI provider ---
    provider: Provider = Provider.OPENROUTER
    openrouter_api_key: str
    provider_model: str | None = None
    openrouter_http_referer: str | None = None
    openrouter_app_title: str | None = None

    # --- persistence ---
    database_url: str = "sqlite:///./transcription.db"

    # --- filesystem paths ---
    upload_dir: Path = Path("./uploads")
    prompt_dir: Path = Path("./prompts")


LOGGING_CONFIG: dict[str, object] = {
    "version": 1,
    "disable_existing_loggers": False,
    "formatters": {
        "standard": {
            "format": "%(asctime)s | %(levelname)-8s | %(name)s | %(message)s",
            "datefmt": "%Y-%m-%d %H:%M:%S",
        }
    },
    "handlers": {
        "console": {
            "class": "logging.StreamHandler",
            "formatter": "standard",
            "stream": "ext://sys.stdout",
        }
    },
    "root": {
        "level": "INFO",
        "handlers": ["console"],
    },
}


@lru_cache(maxsize=1)
def get_settings() -> Settings:
    """Return the singleton Settings instance.

    Cached so the entire application shares one validated config.
    """
    return Settings()


def setup_logging() -> None:
    """Configure root logging once at startup."""
    logging.config.dictConfig(LOGGING_CONFIG)
```

### Key Behaviors

- **Startup validation**: If `OPENROUTER_API_KEY` is missing from the environment, `Settings()` raises a `ValidationError` immediately — the app won't start with a missing key.
- **`.env` support**: Developers can create a `.env` file in the project root for local keys; it's never committed (already covered by the existing `.gitignore` pattern or a new entry).
- **`extra="ignore"`**: Unknown env vars don't cause errors, keeping the config resilient to unrelated environment variables.
- **`lru_cache`**: `get_settings()` is the single access point. All modules import and call this function rather than constructing `Settings` directly.
- **Centralized logging**: `setup_logging()` calls `dictConfig` exactly once at startup; all modules should use `logging.getLogger(__name__)` and avoid `basicConfig`.

### `.env` template (not committed — add to `.gitignore`)

```bash .env.example
PROVIDER=openrouter
OPENROUTER_API_KEY=sk-or-...
# PROVIDER_MODEL=          # optional: OpenRouter adapter supplies default
# OPENROUTER_HTTP_REFERER=https://example.com
# OPENROUTER_APP_TITLE=Historical Transcription MVP
# DATABASE_URL=sqlite:///./transcription.db
# UPLOAD_DIR=./uploads
# PROMPT_DIR=./prompts
```

### `.gitignore` addition

```gitignore .gitignore
# ... existing entries ...

# Environment secrets
.env
```

---

## 3. `models.py` — SQLModel Domain Models

**Satisfies:** REQ-3 (persist and expose job states), REQ-4 (persist transcription output and failure details)

### Design Decisions

| Decision | Rationale |
|----------|-----------|
| Three models: `Document`, `Job`, `Transcript` | Minimal set from MVP Feature 5. One-to-many from Document→Job and one-to-one from Job→Transcript |
| `JobStatus` as a `StrEnum` | Readable in the database (`"queued"` not `1`), type-safe in Python, trivially serializable to JSON for the UI |
| Status values: `queued`, `processing`, `transcribed`, `failed` | Matches MVP Feature 2 lifecycle. REQ-3 also lists `upload` and `completed` — these are deferred to post-MVP when revision/review workflows exist |
| UUIDs for primary keys | Avoids auto-increment collision concerns if we later move to PostgreSQL; safe for distributed ID generation; `uuid4` is simple |
| `uploaded_at`, `created_at`, `updated_at` as UTC `datetime` | Timezone-naive UTC by convention for MVP. Sufficient for single-user, single-timezone operation |
| `Transcript.text` is nullable | A failed job creates a Transcript with `text=None` and `error_detail` populated, keeping the query model uniform |
| Relationships via SQLModel `Relationship` | Enables `document.jobs` and `job.transcript` navigation in service code without manual joins |

### Proposed Implementation

- `resource://skills/fastapi-async-sqlalchemy-modernization/document`

```python src/transcription/models.py
"""SQLModel domain models for the transcription system.

Three models capture the MVP lifecycle:
  Document  →  one-to-many  →  Job  →  one-to-one  →  Transcript
"""

from datetime import datetime, timezone
from enum import StrEnum
from uuid import UUID, uuid4

from sqlmodel import Field, Relationship, SQLModel


class JobStatus(StrEnum):
    QUEUED = "queued"
    PROCESSING = "processing"
    TRANSCRIBED = "transcribed"
    FAILED = "failed"


class Document(SQLModel, table=True):
    """An uploaded document image."""

    id: UUID = Field(default_factory=uuid4, primary_key=True)
    filename: str
    file_path: str
    uploaded_at: datetime = Field(
        default_factory=lambda: datetime.now(timezone.utc),
    )

    # --- relationships ---
    jobs: list["Job"] = Relationship(back_populates="document")


class Job(SQLModel, table=True):
    """A transcription job tied to a single document."""

    id: UUID = Field(default_factory=uuid4, primary_key=True)
    document_id: UUID = Field(foreign_key="document.id")
    status: JobStatus = Field(default=JobStatus.QUEUED)
    created_at: datetime = Field(
        default_factory=lambda: datetime.now(timezone.utc),
    )
    updated_at: datetime = Field(
        default_factory=lambda: datetime.now(timezone.utc),
    )

    # --- relationships ---
    document: Document = Relationship(back_populates="jobs")
    transcript: "Transcript | None" = Relationship(back_populates="job")


class Transcript(SQLModel, table=True):
    """The output of a transcription job."""

    id: UUID = Field(default_factory=uuid4, primary_key=True)
    job_id: UUID = Field(foreign_key="job.id", unique=True)
    text: str | None = None
    error_detail: str | None = None
    created_at: datetime = Field(
        default_factory=lambda: datetime.now(timezone.utc),
    )

    # --- relationships ---
    job: Job = Relationship(back_populates="transcript")
```

### Entity-Relationship Summary

```
┌──────────┐       ┌──────────┐       ┌─────────────┐
│ Document │ 1───* │   Job    │ 1───1 │ Transcript  │
├──────────┤       ├──────────┤       ├─────────────┤
│ id (PK)  │       │ id (PK)  │       │ id (PK)     │
│ filename │       │ doc_id   │──FK──▶│ job_id (FK) │
│ file_path│       │ status   │       │ text        │
│ uploaded │       │ created  │       │ error_detail│
│          │       │ updated  │       │ created     │
└──────────┘       └──────────┘       └─────────────┘
```

### Why Only Four Status Values

REQ-3 lists six states: `upload`, `queued`, `processing`, `transcribed`, `failed`, `completed`. The MVP simplifies this:

| REQ-3 State | MVP Treatment |
|-------------|---------------|
| `upload` | Implicit — the Document record exists before a Job is created. No separate job state needed. |
| `queued` | ✅ Included — job created, waiting for worker pickup |
| `processing` | ✅ Included — worker is actively transcribing |
| `transcribed` | ✅ Included — AI output received and stored |
| `failed` | ✅ Included — error captured |
| `completed` | Deferred — implies human review/acceptance. In MVP, `transcribed` is the terminal success state. |

---

## 4. `db.py` — Database Engine and Session Management

**Satisfies:** MVP Feature 5 (SQLite auto-created on first startup)

### Design Decisions

| Decision | Rationale |
|----------|-----------|
| Module-level `create_engine` + `Session` factory | REQ-7 (lifespan-owned resources) is deferred. A module-level engine is adequate for MVP's single-process, single-user operation |
| `create_all()` as an explicit function | Called at app startup. MVP auto-creates tables (REQ-10 deferred), but the function is isolated so it's easy to gate behind a flag later |
| `get_session()` as a generator | Standard FastAPI/SQLModel pattern — yields a session, ensures cleanup. Compatible with `Depends()` when the API layer arrives in Step 5 |
| `echo=False` default | Keeps logs clean. Can be toggled for debugging |

### Proposed Implementation

```python src/transcription/db.py
"""Database engine, session factory, and schema bootstrap.

MVP uses SQLite with auto-create-tables at startup.
PostgreSQL migration is a post-MVP configuration change.
"""
import contextlib
from collections.abc import Generator

from sqlmodel import Session, SQLModel, create_engine

from transcription.config import get_settings


def _build_engine():
    settings = get_settings()
    connect_args = {}
    if settings.database_url.startswith("sqlite"):
        connect_args["check_same_thread"] = False
    return create_engine(
        settings.database_url,
        echo=False,
        connect_args=connect_args,
    )


engine = _build_engine()


def create_all() -> None:
    """Create all tables. Called once at application startup."""
    SQLModel.metadata.create_all(engine)


@contextlib.contextmanager
def get_session() -> Generator[Session]:
    """Yield a database session and ensure cleanup."""
    with Session(engine) as session:
        yield session
```

### SQLite-Specific Note

`check_same_thread=False` is required for SQLite when the session may be accessed from different threads (e.g., a background worker on a different thread than the request handler). This setting is harmless and ignored for PostgreSQL connection strings.

---

## 5. Test Plan

Refer to these resources for rules and guidelines about structure:

- `resource://skills/pytesting/document`
- `resource://catalog/prompts/pytest-scaffold`
- `resource://catalog/prompts/pytest-fill-scaffold`

Hierarchy pattern used in this step:

```text
tests/
    conftest.py
    test_config.py
        TestSettingsLoading
            test_loads_from_env
            test_requires_api_key
        TestProviderSettings
            test_defaults_to_openrouter
            test_rejects_invalid_value
            test_optional_fields_default_to_none
        TestPathSettings
            test_path_fields_are_path_objects
    test_models.py
        TestDocumentModel
            test_can_be_persisted
            test_defaults_are_populated
        TestJobModel
            test_can_be_created_for_document
            test_defaults_are_populated
            test_transitions_to_transcribed
            test_transitions_to_failed
        TestTranscriptModel
            test_success_record_persists
            test_failure_record_persists
            test_job_id_is_unique
        TestRelationships
            test_document_exposes_jobs
            test_job_exposes_transcript
    test_db.py
        TestSchemaBootstrap
            test_create_all_creates_expected_tables
        TestSessionFactory
            test_get_session_yields_session
            test_session_is_closed_after_generator_exit
```

### `tests/conftest.py` — Shared Fixtures

```python tests/conftest.py
"""Shared test fixtures.

Every test gets a fresh in-memory SQLite database so tests are
isolated, fast, and leave no artifacts on disk.
"""

import pytest
from sqlmodel import Session, SQLModel, create_engine
from sqlmodel.pool import StaticPool


@pytest.fixture
def session():
    """Provide a clean database session for each test."""
    engine = create_engine(
        "sqlite://",
        connect_args={"check_same_thread": False},
        poolclass=StaticPool,
    )
    SQLModel.metadata.create_all(engine)
    with Session(engine) as session:
        yield session
```

`StaticPool` ensures a single in-memory SQLite connection is shared across threads, which is required when `TestClient` (Step 5) spawns threads that would otherwise get separate in-memory databases. Establishing it now keeps the fixture stable across all future steps.

### `tests/test_config.py` — Configuration Hierarchy

| Class | Method | What It Verifies |
|------|--------|------------------|
| `TestSettingsLoading` | `test_loads_from_env` | `Settings` constructs successfully when `OPENROUTER_API_KEY` is set via env var |
| `TestSettingsLoading` | `test_requires_api_key` | `Settings()` raises `ValidationError` when `OPENROUTER_API_KEY` is missing |
| `TestProviderSettings` | `test_defaults_to_openrouter` | Default provider is `openrouter` when not explicitly set |
| `TestProviderSettings` | `test_rejects_invalid_value` | Setting `PROVIDER=invalid` raises `ValidationError` |
| `TestProviderSettings` | `test_optional_fields_default_to_none` | `provider_model`, `openrouter_http_referer`, and `openrouter_app_title` are `None` when unset |
| `TestPathSettings` | `test_path_fields_are_path_objects` | `upload_dir` and `prompt_dir` are `Path` instances |

### `tests/test_models.py` — Model & Relationship Hierarchy

| Class | Method | What It Verifies |
|------|--------|------------------|
| `TestDocumentModel` | `test_can_be_persisted` | A `Document` can be persisted and read back with correct fields |
| `TestDocumentModel` | `test_defaults_are_populated` | `id` is auto-generated UUID, `uploaded_at` is populated |
| `TestJobModel` | `test_can_be_created_for_document` | A `Job` linked to a `Document` via FK persists correctly |
| `TestJobModel` | `test_defaults_are_populated` | Default status is `queued`, `created_at` and `updated_at` are populated |
| `TestJobModel` | `test_transitions_to_transcribed` | Status can be updated from `queued` → `processing` → `transcribed` |
| `TestJobModel` | `test_transitions_to_failed` | Status can be updated from `processing` → `failed` |
| `TestTranscriptModel` | `test_success_record_persists` | A `Transcript` with `text` set and `error_detail=None` persists correctly |
| `TestTranscriptModel` | `test_failure_record_persists` | A `Transcript` with `text=None` and `error_detail` set persists correctly |
| `TestRelationships` | `test_document_exposes_jobs` | `document.jobs` returns the linked `Job` list |
| `TestRelationships` | `test_job_exposes_transcript` | `job.transcript` returns the linked `Transcript` |
| `TestTranscriptModel` | `test_job_id_is_unique` | Inserting two transcripts with the same `job_id` raises an integrity error |

### `tests/test_db.py` — Database Bootstrap Hierarchy

| Class | Method | What It Verifies |
|------|--------|------------------|
| `TestSchemaBootstrap` | `test_create_all_creates_expected_tables` | After `create_all()`, the expected tables (`document`, `job`, `transcript`) exist in the database |
| `TestSessionFactory` | `test_get_session_yields_session` | `get_session()` yields a usable `Session` object |
| `TestSessionFactory` | `test_session_is_closed_after_generator_exit` | After the generator is exhausted, the session is closed |

### Marker Strategy (Step 1)

- Markers (`unit`, `integration`, `external`) are registered upfront in `pyproject.toml` with `--strict-markers` enabled, per pytesting skill conventions.
- All Step 1 tests are unmarked — they run in the default lane since they are fast, deterministic, and have no external dependencies.
- When slower integration or external tests are introduced in later steps, apply explicit markers and keep test names unchanged.

### Test Workflow

Follow the two-phase approach from `resource://catalog/prompts/pytest-scaffold` and `resource://catalog/prompts/pytest-fill-scaffold`:

1. **Scaffold phase**: Create test files with class hierarchy, method names, and one-line docstrings only. Validate collection:
   - `uv run pytest --collect-only -q`
2. **Fill phase**: Implement assertions, fixtures, and minimal test data. Treat scaffolded names and docstrings as locked. Validate execution:
   - `uv run pytest -q`

Scaffolded structure is treated as a stable baseline — do not rename, move, merge, split, or re-nest tests once the scaffold is reviewed.

---

## 6. Step 1 Completion Checklist

When all of the following are true, Step 1 is done and Step 2 can begin:

| # | Criterion | How to Verify |
|---|-----------|---------------|
| 1 | `src/transcription/` package exists with `config.py`, `models.py`, `db.py` | `ls` / file inspection |
| 2 | Empty `__init__.py` stubs exist for `providers/`, `services/`, `ui/` | `ls` / file inspection |
| 3 | `Settings` loads from environment and validates `OPENROUTER_API_KEY` is present | `test_config.py` passes |
| 4 | `Document`, `Job`, `Transcript` models create tables in SQLite | `test_models.py` passes |
| 5 | `JobStatus` enum has exactly four values: `queued`, `processing`, `transcribed`, `failed` | `test_models.py` passes |
| 6 | Foreign key relationships work: Document→Job→Transcript | `test_models.py` passes |
| 7 | `create_all()` bootstraps the schema; `get_session()` yields a working session | `test_db.py` passes |
| 8 | All tests pass: `uv run pytest -q` | CI / local run |
| 9 | `hello.py` is deleted | File inspection |
| 10 | `pyproject.toml` includes `openrouter`, `sqlmodel`, `pydantic-settings`, `pytest`, `pytest-asyncio` | File inspection |
| 10a | `pyproject.toml` has `[tool.pytest.ini_options]` with `--strict-markers` and registered markers | File inspection |
| 11 | `.env.example` documents all config vars; `.env` is in `.gitignore` | File inspection |
| 12 | `setup_logging()` uses `logging.config.dictConfig` with centralized formatter/handler/root config | File inspection |
| 13 | `uv run pytest --collect-only -q` shows expected test hierarchy | Local run |
| 14 | `uv run pytest -q` passes all tests | Local run |

---

## 7. What This Step Does NOT Include

Explicitly out of scope to prevent scope creep:

| Excluded | Reason |
|----------|--------|
| FastAPI / NiceGUI app entrypoint | Step 5 |
| Additional provider adapters beyond OpenRouter | Post-MVP |
| Upload service logic | Step 4 |
| Worker / background processing | Step 4 |
| Transcription prompt files | Step 2 |
| Alembic or migration tooling | Post-MVP (REQ-10 deferred) |
| Async session factory | Post-MVP (REQ-7 deferred) |

---

This plan produces a fully tested, importable data foundation. Every subsequent step imports from `transcription.config`, `transcription.models`, and `transcription.db` without modification. When you're ready, switch to **Agent mode** and I'll implement it.