reformed around openrouter for mvp

This commit is contained in:
John Lancaster
2026-06-23 20:23:28 -05:00
parent 0030c521d3
commit 4f731cc293
3 changed files with 69 additions and 35 deletions
+18 -9
View File
@@ -1,7 +1,7 @@
## MVP Definition: Historical Document Transcription System ## MVP Definition: Historical Document Transcription System
### 1. MVP Objective ### 1. MVP Objective
Deliver the thinnest possible end-to-end vertical slice — a user uploads an image of a document, the system transcribes it via a configurable AI provider, and the user reads the resulting transcript — with just enough persistence and structure to validate the core value proposition: *can AI-driven transcription, guided by curated prompts, produce useful verbatim transcripts of historical family documents?* Deliver the thinnest possible end-to-end vertical slice — a user uploads an image of a document, the system transcribes it via the OpenRouter Python SDK, and the user reads the resulting transcript — with just enough persistence and structure to validate the core value proposition: *can AI-driven transcription, guided by curated prompts, produce useful verbatim transcripts of historical family documents?*
The MVP deliberately defers full-text search, export, revision history, MongoDB, and timeline assembly. These are additive features that don't need validation before the core transcription loop is proven. The MVP deliberately defers full-text search, export, revision history, MongoDB, and timeline assembly. These are additive features that don't need validation before the core transcription loop is proven.
@@ -47,7 +47,7 @@ The MVP deliberately defers full-text search, export, revision history, MongoDB,
* An in-process background worker (Python asyncio task or BackgroundTasks) that: * An in-process background worker (Python asyncio task or BackgroundTasks) that:
1. Picks up queued jobs. 1. Picks up queued jobs.
2. Transitions status to processing. 2. Transitions status to processing.
3. Sends the image + the curated Markdown prompt to an AI vision model via the configured provider (OpenRouter or Gemini). 3. Sends the image + the curated Markdown prompt to an AI vision model via OpenRouter.
4. On success: saves the transcript text, transitions to transcribed. 4. On success: saves the transcript text, transitions to transcribed.
5. On failure: saves the error detail, transitions to failed. 5. On failure: saves the error detail, transitions to failed.
@@ -71,14 +71,24 @@ The MVP deliberately defers full-text search, export, revision history, MongoDB,
#### Feature 6: Centralized Configuration #### Feature 6: Centralized Configuration
* A single config.py (or Pydantic BaseSettings) loading: * A single config.py (or Pydantic BaseSettings) loading:
* PROVIDER (default: openrouter; options: openrouter, gemini) * PROVIDER (fixed to openrouter for MVP)
* PROVIDER_API_KEY (required) * OPENROUTER_API_KEY (required)
* PROVIDER_MODEL (default: provider-appropriate default) * PROVIDER_MODEL (default: OpenRouter model slug for vision transcription)
* PROVIDER_BASE_URL (default: provider-appropriate default; overridable) * OPENROUTER_HTTP_REFERER (optional; app attribution)
* OPENROUTER_APP_TITLE (optional; app attribution)
* DATABASE_URL (default: sqlite:///./transcription.db) * DATABASE_URL (default: sqlite:///./transcription.db)
* UPLOAD_DIR (default: ./uploads) * UPLOAD_DIR (default: ./uploads)
* PROMPT_DIR (default: ./prompts) * PROMPT_DIR (default: ./prompts)
#### Feature 7: MVP Dependency Baseline (OpenRouter-Centric)
* Runtime dependencies:
* openrouter (official OpenRouter Python SDK)
* pydantic
* pydantic-settings
* sqlmodel
* Explicitly out of MVP runtime dependencies:
* google-genai (deferred until/if Gemini is introduced post-MVP)
--- ---
### 5. MVP Architecture (Simplified) ### 5. MVP Architecture (Simplified)
@@ -127,8 +137,7 @@ project-root/
│ ├── providers/ │ ├── providers/
│ │ ├── __init__.py │ │ ├── __init__.py
│ │ ├── base.py # provider interface (transcribe contract) │ │ ├── base.py # provider interface (transcribe contract)
│ │ ├── openrouter.py # OpenRouter via openai client │ │ ├── openrouter.py # OpenRouter via official Python SDK
│ │ └── gemini.py # Google Gemini
│ ├── services/ │ ├── services/
│ │ ├── __init__.py │ │ ├── __init__.py
│ │ ├── upload.py # save file + create records │ │ ├── upload.py # save file + create records
@@ -197,4 +206,4 @@ Recommended build order for the MVP (each step produces a testable increment):
| 5 | ui/upload_page.py + ui/jobs_page.py — NiceGUI pages | User-facing interface | | 5 | ui/upload_page.py + ui/jobs_page.py — NiceGUI pages | User-facing interface |
| 6 | tests/ — unit + integration tests Automated verification | | 6 | tests/ — unit + integration tests Automated verification |
This MVP is deliberately narrow: **one prompt, one configurable provider, one user, one image at a time, SQLite, no containers**. Every omission is intentional — the goal is to get real family documents through the transcription pipeline as fast as possible and let the quality of the output guide every subsequent decision. This MVP is deliberately narrow: **one prompt, one provider (OpenRouter), one user, one image at a time, SQLite, no containers**. Every omission is intentional — the goal is to get real family documents through the transcription pipeline as fast as possible and let the quality of the output guide every subsequent decision.
+48 -24
View File
@@ -59,8 +59,7 @@ description = "Historical document transcription system"
readme = "README.md" readme = "README.md"
requires-python = ">=3.12" requires-python = ">=3.12"
dependencies = [ dependencies = [
"google-genai>=1.0.0", "openrouter>=0.7.0",
"openai>=2.43.0",
"pydantic>=2.13.4", "pydantic>=2.13.4",
"pydantic-settings>=2.9.1", "pydantic-settings>=2.9.1",
"sqlmodel>=0.0.25", "sqlmodel>=0.0.25",
@@ -74,6 +73,7 @@ dev = [
``` ```
Key additions: Key additions:
- **`openrouter`** — official OpenRouter Python SDK used for model calls
- **`pydantic-settings`** — for `BaseSettings` with env-var loading (this was split out of `pydantic` core in v2) - **`pydantic-settings`** — for `BaseSettings` with env-var loading (this was split out of `pydantic` core in v2)
- **`sqlmodel`** — provides SQLModel (which bundles SQLAlchemy + Pydantic model integration) and the SQLite driver - **`sqlmodel`** — provides SQLModel (which bundles SQLAlchemy + Pydantic model integration) and the SQLite driver
- **`pytest` + `pytest-asyncio`** — in `dev` extras for test execution - **`pytest` + `pytest-asyncio`** — in `dev` extras for test execution
@@ -93,12 +93,13 @@ The placeholder file is no longer needed.
| Decision | Rationale | | Decision | Rationale |
|----------|-----------| |----------|-----------|
| Use `pydantic-settings` `BaseSettings` | Type-safe, validates on construction, loads from env vars and `.env` files automatically | | Use `pydantic-settings` `BaseSettings` | Type-safe, validates on construction, loads from env vars and `.env` files automatically |
| `PROVIDER` as a string enum (`openrouter`, `gemini`) | Drives provider factory in Step 3; validated at startup, not at first API call | | `PROVIDER` constrained to `openrouter` for MVP | Keeps configuration explicit while avoiding premature multi-provider complexity |
| `PROVIDER_MODEL` defaults to `None` | Each provider adapter (Step 3) supplies its own sensible default when `None`; avoids config knowing about provider-specific model names | | `OPENROUTER_API_KEY` required | Matches official SDK docs and avoids ambiguous provider-agnostic naming |
| `PROVIDER_BASE_URL` defaults to `None` | Only needed to override OpenRouter's base URL; Gemini ignores it. `None` means "use provider default" | | `PROVIDER_MODEL` defaults to `None` | OpenRouter adapter (Step 3) supplies a sensible default when `None` |
| `OPENROUTER_HTTP_REFERER` and `OPENROUTER_APP_TITLE` optional | Matches SDK optional app-attribution fields |
| `DATABASE_URL` defaults to SQLite | Zero-setup local development; PostgreSQL swap is a single env-var change post-MVP | | `DATABASE_URL` defaults to SQLite | Zero-setup local development; PostgreSQL swap is a single env-var change post-MVP |
| `UPLOAD_DIR` and `PROMPT_DIR` as `Path` objects | Enables `.mkdir(parents=True, exist_ok=True)` and path validation at startup | | `UPLOAD_DIR` and `PROMPT_DIR` as `Path` objects | Enables `.mkdir(parents=True, exist_ok=True)` and path validation at startup |
| Logging configured in a `setup_logging()` function | Called once at startup; uses stdlib `logging` with a simple format. No third-party logging library needed for MVP | | Logging configured via `logging.config.dictConfig` in `setup_logging()` | Centralized, explicit formatter/handler/root logger topology; called once at startup with `disable_existing_loggers=False` |
### Proposed Implementation ### Proposed Implementation
@@ -114,13 +115,13 @@ from enum import StrEnum
from functools import lru_cache from functools import lru_cache
from pathlib import Path from pathlib import Path
import logging import logging
import logging.config
from pydantic_settings import BaseSettings, SettingsConfigDict from pydantic_settings import BaseSettings, SettingsConfigDict
class Provider(StrEnum): class Provider(StrEnum):
OPENROUTER = "openrouter" OPENROUTER = "openrouter"
GEMINI = "gemini"
class Settings(BaseSettings): class Settings(BaseSettings):
@@ -132,9 +133,10 @@ class Settings(BaseSettings):
# --- AI provider --- # --- AI provider ---
provider: Provider = Provider.OPENROUTER provider: Provider = Provider.OPENROUTER
provider_api_key: str openrouter_api_key: str
provider_model: str | None = None provider_model: str | None = None
provider_base_url: str | None = None openrouter_http_referer: str | None = None
openrouter_app_title: str | None = None
# --- persistence --- # --- persistence ---
database_url: str = "sqlite:///./transcription.db" database_url: str = "sqlite:///./transcription.db"
@@ -144,6 +146,29 @@ class Settings(BaseSettings):
prompt_dir: Path = Path("./prompts") prompt_dir: Path = Path("./prompts")
LOGGING_CONFIG: dict[str, object] = {
"version": 1,
"disable_existing_loggers": False,
"formatters": {
"standard": {
"format": "%(asctime)s | %(levelname)-8s | %(name)s | %(message)s",
"datefmt": "%Y-%m-%d %H:%M:%S",
}
},
"handlers": {
"console": {
"class": "logging.StreamHandler",
"formatter": "standard",
"stream": "ext://sys.stdout",
}
},
"root": {
"level": "INFO",
"handlers": ["console"],
},
}
@lru_cache(maxsize=1) @lru_cache(maxsize=1)
def get_settings() -> Settings: def get_settings() -> Settings:
"""Return the singleton Settings instance. """Return the singleton Settings instance.
@@ -155,27 +180,25 @@ def get_settings() -> Settings:
def setup_logging() -> None: def setup_logging() -> None:
"""Configure root logging once at startup.""" """Configure root logging once at startup."""
logging.basicConfig( logging.config.dictConfig(LOGGING_CONFIG)
level=logging.INFO,
format="%(asctime)s | %(levelname)-8s | %(name)s | %(message)s",
datefmt="%Y-%m-%d %H:%M:%S",
)
``` ```
### Key Behaviors ### Key Behaviors
- **Startup validation**: If `PROVIDER_API_KEY` is missing from the environment, `Settings()` raises a `ValidationError` immediately — the app won't start with a missing key. - **Startup validation**: If `OPENROUTER_API_KEY` is missing from the environment, `Settings()` raises a `ValidationError` immediately — the app won't start with a missing key.
- **`.env` support**: Developers can create a `.env` file in the project root for local keys; it's never committed (already covered by the existing `.gitignore` pattern or a new entry). - **`.env` support**: Developers can create a `.env` file in the project root for local keys; it's never committed (already covered by the existing `.gitignore` pattern or a new entry).
- **`extra="ignore"`**: Unknown env vars don't cause errors, keeping the config resilient to unrelated environment variables. - **`extra="ignore"`**: Unknown env vars don't cause errors, keeping the config resilient to unrelated environment variables.
- **`lru_cache`**: `get_settings()` is the single access point. All modules import and call this function rather than constructing `Settings` directly. - **`lru_cache`**: `get_settings()` is the single access point. All modules import and call this function rather than constructing `Settings` directly.
- **Centralized logging**: `setup_logging()` calls `dictConfig` exactly once at startup; all modules should use `logging.getLogger(__name__)` and avoid `basicConfig`.
### `.env` template (not committed — add to `.gitignore`) ### `.env` template (not committed — add to `.gitignore`)
```bash .env.example ```bash .env.example
PROVIDER=openrouter PROVIDER=openrouter
PROVIDER_API_KEY=sk-or-... OPENROUTER_API_KEY=sk-or-...
# PROVIDER_MODEL= # optional: provider adapter supplies default # PROVIDER_MODEL= # optional: OpenRouter adapter supplies default
# PROVIDER_BASE_URL= # optional: override provider endpoint # OPENROUTER_HTTP_REFERER=https://example.com
# OPENROUTER_APP_TITLE=Historical Transcription MVP
# DATABASE_URL=sqlite:///./transcription.db # DATABASE_URL=sqlite:///./transcription.db
# UPLOAD_DIR=./uploads # UPLOAD_DIR=./uploads
# PROMPT_DIR=./prompts # PROMPT_DIR=./prompts
@@ -448,11 +471,11 @@ def session():
| Class | Method | What It Verifies | | Class | Method | What It Verifies |
|------|--------|------------------| |------|--------|------------------|
| `TestSettingsLoading` | `test_loads_from_env` | `Settings` constructs successfully when `PROVIDER_API_KEY` is set via env var | | `TestSettingsLoading` | `test_loads_from_env` | `Settings` constructs successfully when `OPENROUTER_API_KEY` is set via env var |
| `TestSettingsLoading` | `test_requires_api_key` | `Settings()` raises `ValidationError` when `PROVIDER_API_KEY` is missing | | `TestSettingsLoading` | `test_requires_api_key` | `Settings()` raises `ValidationError` when `OPENROUTER_API_KEY` is missing |
| `TestProviderSettings` | `test_defaults_to_openrouter` | Default provider is `openrouter` when not explicitly set | | `TestProviderSettings` | `test_defaults_to_openrouter` | Default provider is `openrouter` when not explicitly set |
| `TestProviderSettings` | `test_rejects_invalid_value` | Setting `PROVIDER=invalid` raises `ValidationError` | | `TestProviderSettings` | `test_rejects_invalid_value` | Setting `PROVIDER=invalid` raises `ValidationError` |
| `TestProviderSettings` | `test_optional_fields_default_to_none` | `provider_model` and `provider_base_url` are `None` when unset | | `TestProviderSettings` | `test_optional_fields_default_to_none` | `provider_model`, `openrouter_http_referer`, and `openrouter_app_title` are `None` when unset |
| `TestPathSettings` | `test_path_fields_are_path_objects` | `upload_dir` and `prompt_dir` are `Path` instances | | `TestPathSettings` | `test_path_fields_are_path_objects` | `upload_dir` and `prompt_dir` are `Path` instances |
### `tests/test_models.py` — Model & Relationship Hierarchy ### `tests/test_models.py` — Model & Relationship Hierarchy
@@ -494,15 +517,16 @@ When all of the following are true, Step 1 is done and Step 2 can begin:
|---|-----------|---------------| |---|-----------|---------------|
| 1 | `src/transcription/` package exists with `config.py`, `models.py`, `db.py` | `ls` / file inspection | | 1 | `src/transcription/` package exists with `config.py`, `models.py`, `db.py` | `ls` / file inspection |
| 2 | Empty `__init__.py` stubs exist for `providers/`, `services/`, `ui/` | `ls` / file inspection | | 2 | Empty `__init__.py` stubs exist for `providers/`, `services/`, `ui/` | `ls` / file inspection |
| 3 | `Settings` loads from environment and validates `PROVIDER_API_KEY` is present | `test_config.py` passes | | 3 | `Settings` loads from environment and validates `OPENROUTER_API_KEY` is present | `test_config.py` passes |
| 4 | `Document`, `Job`, `Transcript` models create tables in SQLite | `test_models.py` passes | | 4 | `Document`, `Job`, `Transcript` models create tables in SQLite | `test_models.py` passes |
| 5 | `JobStatus` enum has exactly four values: `queued`, `processing`, `transcribed`, `failed` | `test_models.py` passes | | 5 | `JobStatus` enum has exactly four values: `queued`, `processing`, `transcribed`, `failed` | `test_models.py` passes |
| 6 | Foreign key relationships work: Document→Job→Transcript | `test_models.py` passes | | 6 | Foreign key relationships work: Document→Job→Transcript | `test_models.py` passes |
| 7 | `create_all()` bootstraps the schema; `get_session()` yields a working session | `test_db.py` passes | | 7 | `create_all()` bootstraps the schema; `get_session()` yields a working session | `test_db.py` passes |
| 8 | All tests pass: `uv run pytest tests/` | CI / local run | | 8 | All tests pass: `uv run pytest tests/` | CI / local run |
| 9 | `hello.py` is deleted | File inspection | | 9 | `hello.py` is deleted | File inspection |
| 10 | `pyproject.toml` includes `sqlmodel`, `pydantic-settings`, `pytest`, `pytest-asyncio` | File inspection | | 10 | `pyproject.toml` includes `openrouter`, `sqlmodel`, `pydantic-settings`, `pytest`, `pytest-asyncio` | File inspection |
| 11 | `.env.example` documents all config vars; `.env` is in `.gitignore` | File inspection | | 11 | `.env.example` documents all config vars; `.env` is in `.gitignore` | File inspection |
| 12 | `setup_logging()` uses `logging.config.dictConfig` with centralized formatter/handler/root config | File inspection |
--- ---
@@ -513,7 +537,7 @@ Explicitly out of scope to prevent scope creep:
| Excluded | Reason | | Excluded | Reason |
|----------|--------| |----------|--------|
| FastAPI / NiceGUI app entrypoint | Step 5 | | FastAPI / NiceGUI app entrypoint | Step 5 |
| Provider adapters (`openrouter.py`, `gemini.py`) | Step 3 | | Additional provider adapters beyond OpenRouter | Post-MVP |
| Upload service logic | Step 4 | | Upload service logic | Step 4 |
| Worker / background processing | Step 4 | | Worker / background processing | Step 4 |
| Transcription prompt files | Step 2 | | Transcription prompt files | Step 2 |
+3 -2
View File
@@ -5,7 +5,8 @@ description = "Add your description here"
readme = "README.md" readme = "README.md"
requires-python = ">=3.12" requires-python = ">=3.12"
dependencies = [ dependencies = [
"google-genai>=1.0.0", "openrouter>=0.7.0",
"openai>=2.43.0",
"pydantic>=2.13.4", "pydantic>=2.13.4",
"pydantic-settings>=2.9.1",
"sqlmodel>=0.0.25",
] ]