diff --git a/docs/mvp.md b/docs/mvp.md index 352cedf..6a1bc94 100644 --- a/docs/mvp.md +++ b/docs/mvp.md @@ -1,7 +1,7 @@ ## MVP Definition: Historical Document Transcription System ### 1. MVP Objective -Deliver the thinnest possible end-to-end vertical slice — a user uploads an image of a document, the system transcribes it via a configurable AI provider, and the user reads the resulting transcript — with just enough persistence and structure to validate the core value proposition: *can AI-driven transcription, guided by curated prompts, produce useful verbatim transcripts of historical family documents?* +Deliver the thinnest possible end-to-end vertical slice — a user uploads an image of a document, the system transcribes it via the OpenRouter Python SDK, and the user reads the resulting transcript — with just enough persistence and structure to validate the core value proposition: *can AI-driven transcription, guided by curated prompts, produce useful verbatim transcripts of historical family documents?* The MVP deliberately defers full-text search, export, revision history, MongoDB, and timeline assembly. These are additive features that don't need validation before the core transcription loop is proven. @@ -47,7 +47,7 @@ The MVP deliberately defers full-text search, export, revision history, MongoDB, * An in-process background worker (Python asyncio task or BackgroundTasks) that: 1. Picks up queued jobs. 2. Transitions status to processing. - 3. Sends the image + the curated Markdown prompt to an AI vision model via the configured provider (OpenRouter or Gemini). + 3. Sends the image + the curated Markdown prompt to an AI vision model via OpenRouter. 4. On success: saves the transcript text, transitions to transcribed. 5. On failure: saves the error detail, transitions to failed. @@ -71,14 +71,24 @@ The MVP deliberately defers full-text search, export, revision history, MongoDB, #### Feature 6: Centralized Configuration * A single config.py (or Pydantic BaseSettings) loading: - * PROVIDER (default: openrouter; options: openrouter, gemini) - * PROVIDER_API_KEY (required) - * PROVIDER_MODEL (default: provider-appropriate default) - * PROVIDER_BASE_URL (default: provider-appropriate default; overridable) + * PROVIDER (fixed to openrouter for MVP) + * OPENROUTER_API_KEY (required) + * PROVIDER_MODEL (default: OpenRouter model slug for vision transcription) + * OPENROUTER_HTTP_REFERER (optional; app attribution) + * OPENROUTER_APP_TITLE (optional; app attribution) * DATABASE_URL (default: sqlite:///./transcription.db) * UPLOAD_DIR (default: ./uploads) * PROMPT_DIR (default: ./prompts) +#### Feature 7: MVP Dependency Baseline (OpenRouter-Centric) +* Runtime dependencies: + * openrouter (official OpenRouter Python SDK) + * pydantic + * pydantic-settings + * sqlmodel +* Explicitly out of MVP runtime dependencies: + * google-genai (deferred until/if Gemini is introduced post-MVP) + --- ### 5. MVP Architecture (Simplified) @@ -127,8 +137,7 @@ project-root/ │ ├── providers/ │ │ ├── __init__.py │ │ ├── base.py # provider interface (transcribe contract) -│ │ ├── openrouter.py # OpenRouter via openai client -│ │ └── gemini.py # Google Gemini +│ │ ├── openrouter.py # OpenRouter via official Python SDK │ ├── services/ │ │ ├── __init__.py │ │ ├── upload.py # save file + create records @@ -197,4 +206,4 @@ Recommended build order for the MVP (each step produces a testable increment): | 5 | ui/upload_page.py + ui/jobs_page.py — NiceGUI pages | User-facing interface | | 6 | tests/ — unit + integration tests Automated verification | -This MVP is deliberately narrow: **one prompt, one configurable provider, one user, one image at a time, SQLite, no containers**. Every omission is intentional — the goal is to get real family documents through the transcription pipeline as fast as possible and let the quality of the output guide every subsequent decision. \ No newline at end of file +This MVP is deliberately narrow: **one prompt, one provider (OpenRouter), one user, one image at a time, SQLite, no containers**. Every omission is intentional — the goal is to get real family documents through the transcription pipeline as fast as possible and let the quality of the output guide every subsequent decision. \ No newline at end of file diff --git a/docs/step1.md b/docs/step1.md index 1a9d3ce..7524498 100644 --- a/docs/step1.md +++ b/docs/step1.md @@ -59,8 +59,7 @@ description = "Historical document transcription system" readme = "README.md" requires-python = ">=3.12" dependencies = [ - "google-genai>=1.0.0", - "openai>=2.43.0", + "openrouter>=0.7.0", "pydantic>=2.13.4", "pydantic-settings>=2.9.1", "sqlmodel>=0.0.25", @@ -74,6 +73,7 @@ dev = [ ``` Key additions: +- **`openrouter`** — official OpenRouter Python SDK used for model calls - **`pydantic-settings`** — for `BaseSettings` with env-var loading (this was split out of `pydantic` core in v2) - **`sqlmodel`** — provides SQLModel (which bundles SQLAlchemy + Pydantic model integration) and the SQLite driver - **`pytest` + `pytest-asyncio`** — in `dev` extras for test execution @@ -93,12 +93,13 @@ The placeholder file is no longer needed. | Decision | Rationale | |----------|-----------| | Use `pydantic-settings` `BaseSettings` | Type-safe, validates on construction, loads from env vars and `.env` files automatically | -| `PROVIDER` as a string enum (`openrouter`, `gemini`) | Drives provider factory in Step 3; validated at startup, not at first API call | -| `PROVIDER_MODEL` defaults to `None` | Each provider adapter (Step 3) supplies its own sensible default when `None`; avoids config knowing about provider-specific model names | -| `PROVIDER_BASE_URL` defaults to `None` | Only needed to override OpenRouter's base URL; Gemini ignores it. `None` means "use provider default" | +| `PROVIDER` constrained to `openrouter` for MVP | Keeps configuration explicit while avoiding premature multi-provider complexity | +| `OPENROUTER_API_KEY` required | Matches official SDK docs and avoids ambiguous provider-agnostic naming | +| `PROVIDER_MODEL` defaults to `None` | OpenRouter adapter (Step 3) supplies a sensible default when `None` | +| `OPENROUTER_HTTP_REFERER` and `OPENROUTER_APP_TITLE` optional | Matches SDK optional app-attribution fields | | `DATABASE_URL` defaults to SQLite | Zero-setup local development; PostgreSQL swap is a single env-var change post-MVP | | `UPLOAD_DIR` and `PROMPT_DIR` as `Path` objects | Enables `.mkdir(parents=True, exist_ok=True)` and path validation at startup | -| Logging configured in a `setup_logging()` function | Called once at startup; uses stdlib `logging` with a simple format. No third-party logging library needed for MVP | +| Logging configured via `logging.config.dictConfig` in `setup_logging()` | Centralized, explicit formatter/handler/root logger topology; called once at startup with `disable_existing_loggers=False` | ### Proposed Implementation @@ -114,13 +115,13 @@ from enum import StrEnum from functools import lru_cache from pathlib import Path import logging +import logging.config from pydantic_settings import BaseSettings, SettingsConfigDict class Provider(StrEnum): OPENROUTER = "openrouter" - GEMINI = "gemini" class Settings(BaseSettings): @@ -132,9 +133,10 @@ class Settings(BaseSettings): # --- AI provider --- provider: Provider = Provider.OPENROUTER - provider_api_key: str + openrouter_api_key: str provider_model: str | None = None - provider_base_url: str | None = None + openrouter_http_referer: str | None = None + openrouter_app_title: str | None = None # --- persistence --- database_url: str = "sqlite:///./transcription.db" @@ -144,6 +146,29 @@ class Settings(BaseSettings): prompt_dir: Path = Path("./prompts") +LOGGING_CONFIG: dict[str, object] = { + "version": 1, + "disable_existing_loggers": False, + "formatters": { + "standard": { + "format": "%(asctime)s | %(levelname)-8s | %(name)s | %(message)s", + "datefmt": "%Y-%m-%d %H:%M:%S", + } + }, + "handlers": { + "console": { + "class": "logging.StreamHandler", + "formatter": "standard", + "stream": "ext://sys.stdout", + } + }, + "root": { + "level": "INFO", + "handlers": ["console"], + }, +} + + @lru_cache(maxsize=1) def get_settings() -> Settings: """Return the singleton Settings instance. @@ -155,27 +180,25 @@ def get_settings() -> Settings: def setup_logging() -> None: """Configure root logging once at startup.""" - logging.basicConfig( - level=logging.INFO, - format="%(asctime)s | %(levelname)-8s | %(name)s | %(message)s", - datefmt="%Y-%m-%d %H:%M:%S", - ) + logging.config.dictConfig(LOGGING_CONFIG) ``` ### Key Behaviors -- **Startup validation**: If `PROVIDER_API_KEY` is missing from the environment, `Settings()` raises a `ValidationError` immediately — the app won't start with a missing key. +- **Startup validation**: If `OPENROUTER_API_KEY` is missing from the environment, `Settings()` raises a `ValidationError` immediately — the app won't start with a missing key. - **`.env` support**: Developers can create a `.env` file in the project root for local keys; it's never committed (already covered by the existing `.gitignore` pattern or a new entry). - **`extra="ignore"`**: Unknown env vars don't cause errors, keeping the config resilient to unrelated environment variables. - **`lru_cache`**: `get_settings()` is the single access point. All modules import and call this function rather than constructing `Settings` directly. +- **Centralized logging**: `setup_logging()` calls `dictConfig` exactly once at startup; all modules should use `logging.getLogger(__name__)` and avoid `basicConfig`. ### `.env` template (not committed — add to `.gitignore`) ```bash .env.example PROVIDER=openrouter -PROVIDER_API_KEY=sk-or-... -# PROVIDER_MODEL= # optional: provider adapter supplies default -# PROVIDER_BASE_URL= # optional: override provider endpoint +OPENROUTER_API_KEY=sk-or-... +# PROVIDER_MODEL= # optional: OpenRouter adapter supplies default +# OPENROUTER_HTTP_REFERER=https://example.com +# OPENROUTER_APP_TITLE=Historical Transcription MVP # DATABASE_URL=sqlite:///./transcription.db # UPLOAD_DIR=./uploads # PROMPT_DIR=./prompts @@ -448,11 +471,11 @@ def session(): | Class | Method | What It Verifies | |------|--------|------------------| -| `TestSettingsLoading` | `test_loads_from_env` | `Settings` constructs successfully when `PROVIDER_API_KEY` is set via env var | -| `TestSettingsLoading` | `test_requires_api_key` | `Settings()` raises `ValidationError` when `PROVIDER_API_KEY` is missing | +| `TestSettingsLoading` | `test_loads_from_env` | `Settings` constructs successfully when `OPENROUTER_API_KEY` is set via env var | +| `TestSettingsLoading` | `test_requires_api_key` | `Settings()` raises `ValidationError` when `OPENROUTER_API_KEY` is missing | | `TestProviderSettings` | `test_defaults_to_openrouter` | Default provider is `openrouter` when not explicitly set | | `TestProviderSettings` | `test_rejects_invalid_value` | Setting `PROVIDER=invalid` raises `ValidationError` | -| `TestProviderSettings` | `test_optional_fields_default_to_none` | `provider_model` and `provider_base_url` are `None` when unset | +| `TestProviderSettings` | `test_optional_fields_default_to_none` | `provider_model`, `openrouter_http_referer`, and `openrouter_app_title` are `None` when unset | | `TestPathSettings` | `test_path_fields_are_path_objects` | `upload_dir` and `prompt_dir` are `Path` instances | ### `tests/test_models.py` — Model & Relationship Hierarchy @@ -494,15 +517,16 @@ When all of the following are true, Step 1 is done and Step 2 can begin: |---|-----------|---------------| | 1 | `src/transcription/` package exists with `config.py`, `models.py`, `db.py` | `ls` / file inspection | | 2 | Empty `__init__.py` stubs exist for `providers/`, `services/`, `ui/` | `ls` / file inspection | -| 3 | `Settings` loads from environment and validates `PROVIDER_API_KEY` is present | `test_config.py` passes | +| 3 | `Settings` loads from environment and validates `OPENROUTER_API_KEY` is present | `test_config.py` passes | | 4 | `Document`, `Job`, `Transcript` models create tables in SQLite | `test_models.py` passes | | 5 | `JobStatus` enum has exactly four values: `queued`, `processing`, `transcribed`, `failed` | `test_models.py` passes | | 6 | Foreign key relationships work: Document→Job→Transcript | `test_models.py` passes | | 7 | `create_all()` bootstraps the schema; `get_session()` yields a working session | `test_db.py` passes | | 8 | All tests pass: `uv run pytest tests/` | CI / local run | | 9 | `hello.py` is deleted | File inspection | -| 10 | `pyproject.toml` includes `sqlmodel`, `pydantic-settings`, `pytest`, `pytest-asyncio` | File inspection | +| 10 | `pyproject.toml` includes `openrouter`, `sqlmodel`, `pydantic-settings`, `pytest`, `pytest-asyncio` | File inspection | | 11 | `.env.example` documents all config vars; `.env` is in `.gitignore` | File inspection | +| 12 | `setup_logging()` uses `logging.config.dictConfig` with centralized formatter/handler/root config | File inspection | --- @@ -513,7 +537,7 @@ Explicitly out of scope to prevent scope creep: | Excluded | Reason | |----------|--------| | FastAPI / NiceGUI app entrypoint | Step 5 | -| Provider adapters (`openrouter.py`, `gemini.py`) | Step 3 | +| Additional provider adapters beyond OpenRouter | Post-MVP | | Upload service logic | Step 4 | | Worker / background processing | Step 4 | | Transcription prompt files | Step 2 | diff --git a/pyproject.toml b/pyproject.toml index d2a7ad4..34dbb82 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -5,7 +5,8 @@ description = "Add your description here" readme = "README.md" requires-python = ">=3.12" dependencies = [ - "google-genai>=1.0.0", - "openai>=2.43.0", + "openrouter>=0.7.0", "pydantic>=2.13.4", + "pydantic-settings>=2.9.1", + "sqlmodel>=0.0.25", ]