bbchops/transcription
1
0
Fork 0
generated from john/python-template
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Added Step 1 implentation plan

Browse Source
This commit is contained in:
Jim Lancaster
2026-06-23 19:53:43 -05:00
parent fdf931796e
commit 4040332e8f
1 changed files with 471 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/step1.md
+471
View File
@@ -0,0 +1,471 @@
I now have a complete understanding of all the constraints. Here is the detailed implementation plan for Step 1:
---
# Step 1 Implementation Plan: `config.py` + `models.py` + `db.py`
## Purpose
Establish the foundational data layer and configuration system that every subsequent MVP step builds on. At the end of this step, the project has a runnable Python package with a validated schema, typed configuration, and a test suite proving the data layer works — before any UI, worker, or AI provider code exists.
---
## 1. Prerequisite: Project Structure Scaffolding
Before writing any logic, create the package skeleton so imports work correctly.
### Files to create (empty `__init__.py` stubs)
```
src/
└── transcription/
├── __init__.py
├── providers/
│ └── __init__.py
├── services/
│ └── __init__.py
└── ui/
└── __init__.py
```
### Files to create (with logic — the Step 1 deliverables)
```
src/transcription/config.py
src/transcription/models.py
src/transcription/db.py
```
### Test files to create
```
tests/
├── __init__.py
├── conftest.py
├── test_config.py
├── test_models.py
└── test_db.py
```
### Update `pyproject.toml`
Add the dependencies that Step 1 requires and won't change later:
```toml pyproject.toml
[project]
name = "transcription"
version = "0.1.0"
description = "Historical document transcription system"
readme = "README.md"
requires-python = ">=3.12"
dependencies = [
"google-genai>=1.0.0",
"openai>=2.43.0",
"pydantic>=2.13.4",
"pydantic-settings>=2.9.1",
"sqlmodel>=0.0.25",
]
[project.optional-dependencies]
dev = [
"pytest>=8.0",
"pytest-asyncio>=0.25",
]
```
Key additions:
- **`pydantic-settings`** — for `BaseSettings` with env-var loading (this was split out of `pydantic` core in v2)
- **`sqlmodel`** — provides SQLModel (which bundles SQLAlchemy + Pydantic model integration) and the SQLite driver
- **`pytest` + `pytest-asyncio`** — in `dev` extras for test execution
### Delete `hello.py`
The placeholder file is no longer needed.
---
## 2. `config.py` — Centralized Configuration
**Satisfies:** REQ-8 (centralized config and logging at startup)
### Design Decisions
| Decision | Rationale |
|----------|-----------|
| Use `pydantic-settings` `BaseSettings` | Type-safe, validates on construction, loads from env vars and `.env` files automatically |
| `PROVIDER` as a string enum (`openrouter`, `gemini`) | Drives provider factory in Step 3; validated at startup, not at first API call |
| `PROVIDER_MODEL` defaults to `None` | Each provider adapter (Step 3) supplies its own sensible default when `None`; avoids config knowing about provider-specific model names |
| `PROVIDER_BASE_URL` defaults to `None` | Only needed to override OpenRouter's base URL; Gemini ignores it. `None` means "use provider default" |
| `DATABASE_URL` defaults to SQLite | Zero-setup local development; PostgreSQL swap is a single env-var change post-MVP |
| `UPLOAD_DIR` and `PROMPT_DIR` as `Path` objects | Enables `.mkdir(parents=True, exist_ok=True)` and path validation at startup |
| Logging configured in a `setup_logging()` function | Called once at startup; uses stdlib `logging` with a simple format. No third-party logging library needed for MVP |
### Proposed Implementation
```python src/transcription/config.py
"""Centralized application configuration.
All settings are loaded from environment variables (or a .env file)
once at startup. Provider-specific defaults (model names, base URLs)
are resolved by the provider adapters, not here.
"""
from enum import StrEnum
from functools import lru_cache
from pathlib import Path
import logging
from pydantic_settings import BaseSettings, SettingsConfigDict
class Provider(StrEnum):
OPENROUTER = "openrouter"
GEMINI = "gemini"
class Settings(BaseSettings):
model_config = SettingsConfigDict(
env_file=".env",
env_file_encoding="utf-8",
extra="ignore",
)
# --- AI provider ---
provider: Provider = Provider.OPENROUTER
provider_api_key: str
provider_model: str | None = None
provider_base_url: str | None = None
# --- persistence ---
database_url: str = "sqlite:///./transcription.db"
# --- filesystem paths ---
upload_dir: Path = Path("./uploads")
prompt_dir: Path = Path("./prompts")
@lru_cache(maxsize=1)
def get_settings() -> Settings:
"""Return the singleton Settings instance.
Cached so the entire application shares one validated config.
"""
return Settings()
def setup_logging() -> None:
"""Configure root logging once at startup."""
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s | %(levelname)-8s | %(name)s | %(message)s",
datefmt="%Y-%m-%d %H:%M:%S",
)
```
### Key Behaviors
- **Startup validation**: If `PROVIDER_API_KEY` is missing from the environment, `Settings()` raises a `ValidationError` immediately — the app won't start with a missing key.
- **`.env` support**: Developers can create a `.env` file in the project root for local keys; it's never committed (already covered by the existing `.gitignore` pattern or a new entry).
- **`extra="ignore"`**: Unknown env vars don't cause errors, keeping the config resilient to unrelated environment variables.
- **`lru_cache`**: `get_settings()` is the single access point. All modules import and call this function rather than constructing `Settings` directly.
### `.env` template (not committed — add to `.gitignore`)
```bash .env.example
PROVIDER=openrouter
PROVIDER_API_KEY=sk-or-...
# PROVIDER_MODEL= # optional: provider adapter supplies default
# PROVIDER_BASE_URL= # optional: override provider endpoint
# DATABASE_URL=sqlite:///./transcription.db
# UPLOAD_DIR=./uploads
# PROMPT_DIR=./prompts
```
### `.gitignore` addition
```gitignore .gitignore
# ... existing entries ...
# Environment secrets
.env
```
---
## 3. `models.py` — SQLModel Domain Models
**Satisfies:** REQ-3 (persist and expose job states), REQ-4 (persist transcription output and failure details)
### Design Decisions
| Decision | Rationale |
|----------|-----------|
| Three models: `Document`, `Job`, `Transcript` | Minimal set from MVP Feature 5. One-to-many from Document→Job and one-to-one from Job→Transcript |
| `JobStatus` as a `StrEnum` | Readable in the database (`"queued"` not `1`), type-safe in Python, trivially serializable to JSON for the UI |
| Status values: `queued`, `processing`, `transcribed`, `failed` | Matches MVP Feature 2 lifecycle. REQ-3 also lists `upload` and `completed` — these are deferred to post-MVP when revision/review workflows exist |
| UUIDs for primary keys | Avoids auto-increment collision concerns if we later move to PostgreSQL; safe for distributed ID generation; `uuid4` is simple |
| `uploaded_at`, `created_at`, `updated_at` as UTC `datetime` | Timezone-naive UTC by convention for MVP. Sufficient for single-user, single-timezone operation |
| `Transcript.text` is nullable | A failed job creates a Transcript with `text=None` and `error_detail` populated, keeping the query model uniform |
| Relationships via SQLModel `Relationship` | Enables `document.jobs` and `job.transcript` navigation in service code without manual joins |
### Proposed Implementation
```python src/transcription/models.py
"""SQLModel domain models for the transcription system.
Three models capture the MVP lifecycle:
Document → one-to-many → Job → one-to-one → Transcript
"""
from datetime import datetime, timezone
from enum import StrEnum
from uuid import UUID, uuid4
from sqlmodel import Field, Relationship, SQLModel
class JobStatus(StrEnum):
QUEUED = "queued"
PROCESSING = "processing"
TRANSCRIBED = "transcribed"
FAILED = "failed"
class Document(SQLModel, table=True):
"""An uploaded document image."""
id: UUID = Field(default_factory=uuid4, primary_key=True)
filename: str
file_path: str
uploaded_at: datetime = Field(
default_factory=lambda: datetime.now(timezone.utc),
)
# --- relationships ---
jobs: list["Job"] = Relationship(back_populates="document")
class Job(SQLModel, table=True):
"""A transcription job tied to a single document."""
id: UUID = Field(default_factory=uuid4, primary_key=True)
document_id: UUID = Field(foreign_key="document.id")
status: JobStatus = Field(default=JobStatus.QUEUED)
created_at: datetime = Field(
default_factory=lambda: datetime.now(timezone.utc),
)
updated_at: datetime = Field(
default_factory=lambda: datetime.now(timezone.utc),
)
# --- relationships ---
document: Document = Relationship(back_populates="jobs")
transcript: "Transcript | None" = Relationship(back_populates="job")
class Transcript(SQLModel, table=True):
"""The output of a transcription job."""
id: UUID = Field(default_factory=uuid4, primary_key=True)
job_id: UUID = Field(foreign_key="job.id", unique=True)
text: str | None = None
error_detail: str | None = None
created_at: datetime = Field(
default_factory=lambda: datetime.now(timezone.utc),
)
# --- relationships ---
job: Job = Relationship(back_populates="transcript")
```
### Entity-Relationship Summary
```
┌──────────┐ ┌──────────┐ ┌─────────────┐
│ Document │ 1───* │ Job │ 1───1 │ Transcript │
├──────────┤ ├──────────┤ ├─────────────┤
│ id (PK) │ │ id (PK) │ │ id (PK) │
│ filename │ │ doc_id │──FK──▶│ job_id (FK) │
│ file_path│ │ status │ │ text │
│ uploaded │ │ created │ │ error_detail│
│ │ │ updated │ │ created │
└──────────┘ └──────────┘ └─────────────┘
```
### Why Only Four Status Values
REQ-3 lists six states: `upload`, `queued`, `processing`, `transcribed`, `failed`, `completed`. The MVP simplifies this:
| REQ-3 State | MVP Treatment |
|-------------|---------------|
| `upload` | Implicit — the Document record exists before a Job is created. No separate job state needed. |
| `queued` | ✅ Included — job created, waiting for worker pickup |
| `processing` | ✅ Included — worker is actively transcribing |
| `transcribed` | ✅ Included — AI output received and stored |
| `failed` | ✅ Included — error captured |
| `completed` | Deferred — implies human review/acceptance. In MVP, `transcribed` is the terminal success state. |
---
## 4. `db.py` — Database Engine and Session Management
**Satisfies:** MVP Feature 5 (SQLite auto-created on first startup)
### Design Decisions
| Decision | Rationale |
|----------|-----------|
| Module-level `create_engine` + `Session` factory | REQ-7 (lifespan-owned resources) is deferred. A module-level engine is adequate for MVP's single-process, single-user operation |
| `create_all()` as an explicit function | Called at app startup. MVP auto-creates tables (REQ-10 deferred), but the function is isolated so it's easy to gate behind a flag later |
| `get_session()` as a generator | Standard FastAPI/SQLModel pattern — yields a session, ensures cleanup. Compatible with `Depends()` when the API layer arrives in Step 5 |
| `echo=False` default | Keeps logs clean. Can be toggled for debugging |
### Proposed Implementation
```python src/transcription/db.py
"""Database engine, session factory, and schema bootstrap.
MVP uses SQLite with auto-create-tables at startup.
PostgreSQL migration is a post-MVP configuration change.
"""
from collections.abc import Generator
from sqlmodel import Session, SQLModel, create_engine
from transcription.config import get_settings
def _build_engine():
settings = get_settings()
connect_args = {}
if settings.database_url.startswith("sqlite"):
connect_args["check_same_thread"] = False
return create_engine(
settings.database_url,
echo=False,
connect_args=connect_args,
)
engine = _build_engine()
def create_all() -> None:
"""Create all tables. Called once at application startup."""
SQLModel.metadata.create_all(engine)
def get_session() -> Generator[Session, None, None]:
"""Yield a database session and ensure cleanup."""
with Session(engine) as session:
yield session
```
### SQLite-Specific Note
`check_same_thread=False` is required for SQLite when the session may be accessed from different threads (e.g., a background worker on a different thread than the request handler). This setting is harmless and ignored for PostgreSQL connection strings.
---
## 5. Test Plan
### `tests/conftest.py` — Shared Fixtures
```python tests/conftest.py
"""Shared test fixtures.
Every test gets a fresh in-memory SQLite database so tests are
isolated, fast, and leave no artifacts on disk.
"""
import pytest
from sqlmodel import Session, SQLModel, create_engine
@pytest.fixture
def session():
"""Provide a clean database session for each test."""
engine = create_engine(
"sqlite://", # in-memory
connect_args={"check_same_thread": False},
)
SQLModel.metadata.create_all(engine)
with Session(engine) as session:
yield session
```
### `tests/test_config.py` — Configuration Tests
| Test | What It Verifies |
|------|------------------|
| `test_settings_loads_from_env` | `Settings` constructs successfully when `PROVIDER_API_KEY` is set via env var |
| `test_settings_requires_api_key` | `Settings()` raises `ValidationError` when `PROVIDER_API_KEY` is missing |
| `test_provider_defaults_to_openrouter` | Default provider is `openrouter` when not explicitly set |
| `test_provider_rejects_invalid_value` | Setting `PROVIDER=invalid` raises `ValidationError` |
| `test_optional_fields_default_to_none` | `provider_model` and `provider_base_url` are `None` when unset |
| `test_path_fields_are_path_objects` | `upload_dir` and `prompt_dir` are `Path` instances |
### `tests/test_models.py` — Model & Relationship Tests
| Test | What It Verifies |
|------|------------------|
| `test_create_document` | A `Document` can be persisted and read back with correct fields |
| `test_document_defaults` | `id` is auto-generated UUID, `uploaded_at` is populated |
| `test_create_job_with_document` | A `Job` linked to a `Document` via FK persists correctly |
| `test_job_defaults` | Default status is `queued`, `created_at` and `updated_at` are populated |
| `test_job_status_transitions` | Status can be updated from `queued` → `processing` → `transcribed` |
| `test_job_status_to_failed` | Status can be updated from `processing` → `failed` |
| `test_create_transcript_success` | A `Transcript` with `text` set and `error_detail=None` persists correctly |
| `test_create_transcript_failure` | A `Transcript` with `text=None` and `error_detail` set persists correctly |
| `test_document_jobs_relationship` | `document.jobs` returns the linked `Job` list |
| `test_job_transcript_relationship` | `job.transcript` returns the linked `Transcript` |
| `test_transcript_job_id_unique` | Inserting two transcripts with the same `job_id` raises an integrity error |
### `tests/test_db.py` — Database Bootstrap Tests
| Test | What It Verifies |
|------|------------------|
| `test_create_all_creates_tables` | After `create_all()`, the expected tables (`document`, `job`, `transcript`) exist in the database |
| `test_get_session_yields_session` | `get_session()` yields a usable `Session` object |
| `test_session_cleanup_on_exit` | After the generator is exhausted, the session is closed |
---
## 6. Step 1 Completion Checklist
When all of the following are true, Step 1 is done and Step 2 can begin:
| # | Criterion | How to Verify |
|---|-----------|---------------|
| 1 | `src/transcription/` package exists with `config.py`, `models.py`, `db.py` | `ls` / file inspection |
| 2 | Empty `__init__.py` stubs exist for `providers/`, `services/`, `ui/` | `ls` / file inspection |
| 3 | `Settings` loads from environment and validates `PROVIDER_API_KEY` is present | `test_config.py` passes |
| 4 | `Document`, `Job`, `Transcript` models create tables in SQLite | `test_models.py` passes |
| 5 | `JobStatus` enum has exactly four values: `queued`, `processing`, `transcribed`, `failed` | `test_models.py` passes |
| 6 | Foreign key relationships work: Document→Job→Transcript | `test_models.py` passes |
| 7 | `create_all()` bootstraps the schema; `get_session()` yields a working session | `test_db.py` passes |
| 8 | All tests pass: `uv run pytest tests/` | CI / local run |
| 9 | `hello.py` is deleted | File inspection |
| 10 | `pyproject.toml` includes `sqlmodel`, `pydantic-settings`, `pytest`, `pytest-asyncio` | File inspection |
| 11 | `.env.example` documents all config vars; `.env` is in `.gitignore` | File inspection |
---
## 7. What This Step Does NOT Include
Explicitly out of scope to prevent scope creep:
| Excluded | Reason |
|----------|--------|
| FastAPI / NiceGUI app entrypoint | Step 5 |
| Provider adapters (`openrouter.py`, `gemini.py`) | Step 3 |
| Upload service logic | Step 4 |
| Worker / background processing | Step 4 |
| Transcription prompt files | Step 2 |
| Alembic or migration tooling | Post-MVP (REQ-10 deferred) |
| Async session factory | Post-MVP (REQ-7 deferred) |
---
This plan produces a fully tested, importable data foundation. Every subsequent step imports from `transcription.config`, `transcription.models`, and `transcription.db` without modification. When you're ready, switch to **Agent mode** and I'll implement it.