bbchops/transcription
1
0
Fork 0
generated from john/python-template
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

reformed around openrouter for mvp

Browse Source
This commit is contained in:
John Lancaster
2026-06-23 20:23:28 -05:00
parent 0030c521d3
commit 4f731cc293
3 changed files with 69 additions and 35 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/mvp.md
+18 -9
View File
@@ -1,7 +1,7 @@
## MVP Definition: Historical Document Transcription System
### 1. MVP Objective
Deliver the thinnest possible end-to-end vertical slice — a user uploads an image of a document, the system transcribes it via a configurable AI provider, and the user reads the resulting transcript — with just enough persistence and structure to validate the core value proposition: *can AI-driven transcription, guided by curated prompts, produce useful verbatim transcripts of historical family documents?*
Deliver the thinnest possible end-to-end vertical slice — a user uploads an image of a document, the system transcribes it via the OpenRouter Python SDK, and the user reads the resulting transcript — with just enough persistence and structure to validate the core value proposition: *can AI-driven transcription, guided by curated prompts, produce useful verbatim transcripts of historical family documents?*
The MVP deliberately defers full-text search, export, revision history, MongoDB, and timeline assembly. These are additive features that don't need validation before the core transcription loop is proven.
@@ -47,7 +47,7 @@ The MVP deliberately defers full-text search, export, revision history, MongoDB,
* An in-process background worker (Python asyncio task or BackgroundTasks) that:
1. Picks up queued jobs.
2. Transitions status to processing.
3. Sends the image + the curated Markdown prompt to an AI vision model via the configured provider (OpenRouter or Gemini).
3. Sends the image + the curated Markdown prompt to an AI vision model via OpenRouter.
4. On success: saves the transcript text, transitions to transcribed.
5. On failure: saves the error detail, transitions to failed.
@@ -71,14 +71,24 @@ The MVP deliberately defers full-text search, export, revision history, MongoDB,
#### Feature 6: Centralized Configuration
* A single config.py (or Pydantic BaseSettings) loading:
* PROVIDER (default: openrouter; options: openrouter, gemini)
* PROVIDER_API_KEY (required)
* PROVIDER_MODEL (default: provider-appropriate default)
* PROVIDER_BASE_URL (default: provider-appropriate default; overridable)
* PROVIDER (fixed to openrouter for MVP)
* OPENROUTER_API_KEY (required)
* PROVIDER_MODEL (default: OpenRouter model slug for vision transcription)
* OPENROUTER_HTTP_REFERER (optional; app attribution)
* OPENROUTER_APP_TITLE (optional; app attribution)
* DATABASE_URL (default: sqlite:///./transcription.db)
* UPLOAD_DIR (default: ./uploads)
* PROMPT_DIR (default: ./prompts)
#### Feature 7: MVP Dependency Baseline (OpenRouter-Centric)
* Runtime dependencies:
* openrouter (official OpenRouter Python SDK)
* pydantic
* pydantic-settings
* sqlmodel
* Explicitly out of MVP runtime dependencies:
* google-genai (deferred until/if Gemini is introduced post-MVP)
---
### 5. MVP Architecture (Simplified)
@@ -127,8 +137,7 @@ project-root/
│ ├── providers/
│ │ ├── __init__.py
│ │ ├── base.py # provider interface (transcribe contract)
│ │ ├── openrouter.py # OpenRouter via openai client
│ │ └── gemini.py # Google Gemini
│ │ ├── openrouter.py # OpenRouter via official Python SDK
│ ├── services/
│ │ ├── __init__.py
│ │ ├── upload.py # save file + create records
@@ -197,4 +206,4 @@ Recommended build order for the MVP (each step produces a testable increment):
| 5 | ui/upload_page.py + ui/jobs_page.py — NiceGUI pages | User-facing interface |
| 6 | tests/ — unit + integration tests Automated verification |
This MVP is deliberately narrow: **one prompt, one configurable provider, one user, one image at a time, SQLite, no containers**. Every omission is intentional — the goal is to get real family documents through the transcription pipeline as fast as possible and let the quality of the output guide every subsequent decision.
This MVP is deliberately narrow: **one prompt, one provider (OpenRouter), one user, one image at a time, SQLite, no containers**. Every omission is intentional — the goal is to get real family documents through the transcription pipeline as fast as possible and let the quality of the output guide every subsequent decision.
docs/step1.md
+48 -24
View File
@@ -59,8 +59,7 @@ description = "Historical document transcription system"
readme = "README.md"
requires-python = ">=3.12"
dependencies = [
"google-genai>=1.0.0",
"openai>=2.43.0",
"openrouter>=0.7.0",
"pydantic>=2.13.4",
"pydantic-settings>=2.9.1",
"sqlmodel>=0.0.25",
@@ -74,6 +73,7 @@ dev = [
```
Key additions:
- **`openrouter`** — official OpenRouter Python SDK used for model calls
- **`pydantic-settings`** — for `BaseSettings` with env-var loading (this was split out of `pydantic` core in v2)
- **`sqlmodel`** — provides SQLModel (which bundles SQLAlchemy + Pydantic model integration) and the SQLite driver
- **`pytest` + `pytest-asyncio`** — in `dev` extras for test execution
@@ -93,12 +93,13 @@ The placeholder file is no longer needed.
| Decision | Rationale |
|----------|-----------|
| Use `pydantic-settings` `BaseSettings` | Type-safe, validates on construction, loads from env vars and `.env` files automatically |
| `PROVIDER` as a string enum (`openrouter`, `gemini`) | Drives provider factory in Step 3; validated at startup, not at first API call |
| `PROVIDER_MODEL` defaults to `None` | Each provider adapter (Step 3) supplies its own sensible default when `None`; avoids config knowing about provider-specific model names |
| `PROVIDER_BASE_URL` defaults to `None` | Only needed to override OpenRouter's base URL; Gemini ignores it. `None` means "use provider default" |
| `PROVIDER` constrained to `openrouter` for MVP | Keeps configuration explicit while avoiding premature multi-provider complexity |
| `OPENROUTER_API_KEY` required | Matches official SDK docs and avoids ambiguous provider-agnostic naming |
| `PROVIDER_MODEL` defaults to `None` | OpenRouter adapter (Step 3) supplies a sensible default when `None` |
| `OPENROUTER_HTTP_REFERER` and `OPENROUTER_APP_TITLE` optional | Matches SDK optional app-attribution fields |
| `DATABASE_URL` defaults to SQLite | Zero-setup local development; PostgreSQL swap is a single env-var change post-MVP |
| `UPLOAD_DIR` and `PROMPT_DIR` as `Path` objects | Enables `.mkdir(parents=True, exist_ok=True)` and path validation at startup |
| Logging configured in a `setup_logging()` function | Called once at startup; uses stdlib `logging` with a simple format. No third-party logging library needed for MVP |
| Logging configured via `logging.config.dictConfig` in `setup_logging()` | Centralized, explicit formatter/handler/root logger topology; called once at startup with `disable_existing_loggers=False` |
### Proposed Implementation
@@ -114,13 +115,13 @@ from enum import StrEnum
from functools import lru_cache
from pathlib import Path
import logging
import logging.config
from pydantic_settings import BaseSettings, SettingsConfigDict
class Provider(StrEnum):
OPENROUTER = "openrouter"
GEMINI = "gemini"
class Settings(BaseSettings):
@@ -132,9 +133,10 @@ class Settings(BaseSettings):
# --- AI provider ---
provider: Provider = Provider.OPENROUTER
provider_api_key: str
openrouter_api_key: str
provider_model: str | None = None
provider_base_url: str | None = None
openrouter_http_referer: str | None = None
openrouter_app_title: str | None = None
# --- persistence ---
database_url: str = "sqlite:///./transcription.db"
@@ -144,6 +146,29 @@ class Settings(BaseSettings):
prompt_dir: Path = Path("./prompts")
LOGGING_CONFIG: dict[str, object] = {
"version": 1,
"disable_existing_loggers": False,
"formatters": {
"standard": {
"format": "%(asctime)s | %(levelname)-8s | %(name)s | %(message)s",
"datefmt": "%Y-%m-%d %H:%M:%S",
}
},
"handlers": {
"console": {
"class": "logging.StreamHandler",
"formatter": "standard",
"stream": "ext://sys.stdout",
}
},
"root": {
"level": "INFO",
"handlers": ["console"],
},
}
@lru_cache(maxsize=1)
def get_settings() -> Settings:
"""Return the singleton Settings instance.
@@ -155,27 +180,25 @@ def get_settings() -> Settings:
def setup_logging() -> None:
"""Configure root logging once at startup."""
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s | %(levelname)-8s | %(name)s | %(message)s",
datefmt="%Y-%m-%d %H:%M:%S",
)
logging.config.dictConfig(LOGGING_CONFIG)
```
### Key Behaviors
- **Startup validation**: If `PROVIDER_API_KEY` is missing from the environment, `Settings()` raises a `ValidationError` immediately — the app won't start with a missing key.
- **Startup validation**: If `OPENROUTER_API_KEY` is missing from the environment, `Settings()` raises a `ValidationError` immediately — the app won't start with a missing key.
- **`.env` support**: Developers can create a `.env` file in the project root for local keys; it's never committed (already covered by the existing `.gitignore` pattern or a new entry).
- **`extra="ignore"`**: Unknown env vars don't cause errors, keeping the config resilient to unrelated environment variables.
- **`lru_cache`**: `get_settings()` is the single access point. All modules import and call this function rather than constructing `Settings` directly.
- **Centralized logging**: `setup_logging()` calls `dictConfig` exactly once at startup; all modules should use `logging.getLogger(__name__)` and avoid `basicConfig`.
### `.env` template (not committed — add to `.gitignore`)
```bash .env.example
PROVIDER=openrouter
PROVIDER_API_KEY=sk-or-...
# PROVIDER_MODEL= # optional: provider adapter supplies default
# PROVIDER_BASE_URL= # optional: override provider endpoint
OPENROUTER_API_KEY=sk-or-...
# PROVIDER_MODEL= # optional: OpenRouter adapter supplies default
# OPENROUTER_HTTP_REFERER=https://example.com
# OPENROUTER_APP_TITLE=Historical Transcription MVP
# DATABASE_URL=sqlite:///./transcription.db
# UPLOAD_DIR=./uploads
# PROMPT_DIR=./prompts
@@ -448,11 +471,11 @@ def session():
| Class | Method | What It Verifies |
|------|--------|------------------|
| `TestSettingsLoading` | `test_loads_from_env` | `Settings` constructs successfully when `PROVIDER_API_KEY` is set via env var |
| `TestSettingsLoading` | `test_requires_api_key` | `Settings()` raises `ValidationError` when `PROVIDER_API_KEY` is missing |
| `TestSettingsLoading` | `test_loads_from_env` | `Settings` constructs successfully when `OPENROUTER_API_KEY` is set via env var |
| `TestSettingsLoading` | `test_requires_api_key` | `Settings()` raises `ValidationError` when `OPENROUTER_API_KEY` is missing |
| `TestProviderSettings` | `test_defaults_to_openrouter` | Default provider is `openrouter` when not explicitly set |
| `TestProviderSettings` | `test_rejects_invalid_value` | Setting `PROVIDER=invalid` raises `ValidationError` |
| `TestProviderSettings` | `test_optional_fields_default_to_none` | `provider_model` and `provider_base_url` are `None` when unset |
| `TestProviderSettings` | `test_optional_fields_default_to_none` | `provider_model`, `openrouter_http_referer`, and `openrouter_app_title` are `None` when unset |
| `TestPathSettings` | `test_path_fields_are_path_objects` | `upload_dir` and `prompt_dir` are `Path` instances |
### `tests/test_models.py` — Model & Relationship Hierarchy
@@ -494,15 +517,16 @@ When all of the following are true, Step 1 is done and Step 2 can begin:
|---|-----------|---------------|
| 1 | `src/transcription/` package exists with `config.py`, `models.py`, `db.py` | `ls` / file inspection |
| 2 | Empty `__init__.py` stubs exist for `providers/`, `services/`, `ui/` | `ls` / file inspection |
| 3 | `Settings` loads from environment and validates `PROVIDER_API_KEY` is present | `test_config.py` passes |
| 3 | `Settings` loads from environment and validates `OPENROUTER_API_KEY` is present | `test_config.py` passes |
| 4 | `Document`, `Job`, `Transcript` models create tables in SQLite | `test_models.py` passes |
| 5 | `JobStatus` enum has exactly four values: `queued`, `processing`, `transcribed`, `failed` | `test_models.py` passes |
| 6 | Foreign key relationships work: Document→Job→Transcript | `test_models.py` passes |
| 7 | `create_all()` bootstraps the schema; `get_session()` yields a working session | `test_db.py` passes |
| 8 | All tests pass: `uv run pytest tests/` | CI / local run |
| 9 | `hello.py` is deleted | File inspection |
| 10 | `pyproject.toml` includes `sqlmodel`, `pydantic-settings`, `pytest`, `pytest-asyncio` | File inspection |
| 10 | `pyproject.toml` includes `openrouter`, `sqlmodel`, `pydantic-settings`, `pytest`, `pytest-asyncio` | File inspection |
| 11 | `.env.example` documents all config vars; `.env` is in `.gitignore` | File inspection |
| 12 | `setup_logging()` uses `logging.config.dictConfig` with centralized formatter/handler/root config | File inspection |
---
@@ -513,7 +537,7 @@ Explicitly out of scope to prevent scope creep:
| Excluded | Reason |
|----------|--------|
| FastAPI / NiceGUI app entrypoint | Step 5 |
| Provider adapters (`openrouter.py`, `gemini.py`) | Step 3 |
| Additional provider adapters beyond OpenRouter | Post-MVP |
| Upload service logic | Step 4 |
| Worker / background processing | Step 4 |
| Transcription prompt files | Step 2 |
pyproject.toml
+3 -2
View File
@@ -5,7 +5,8 @@ description = "Add your description here"
readme = "README.md"
requires-python = ">=3.12"
dependencies = [
"google-genai>=1.0.0",
"openai>=2.43.0",
"openrouter>=0.7.0",
"pydantic>=2.13.4",
"pydantic-settings>=2.9.1",
"sqlmodel>=0.0.25",
]