Step 2 implemented

README.md

@@ -0,0 +1,8 @@
+# Transcription
+
+Historical document transcription system.
+
+## Prompt Artifacts
+
+Prompt files are stored in `prompts/` and loaded from `PROMPT_DIR` (default: `./prompts`).
+The canonical MVP prompt is `prompts/transcribe_document.md`.

docs/step2.md

@@ -1,6 +1,6 @@
-Great—here’s a **detailed implementation plan for Step 2** based on your docs and current project state.
+## Step 2: prompts/transcribe_document.md
 
-## Step 2 Goal
+### Goal
 
 Implement the MVP prompt artifact system by creating a curated transcription prompt file:
 
@@ -121,6 +121,129 @@ In Step 2, ensure docs reflect this and that Step 3 will resolve:
 
 ---
 
+## Task-by-Task Execution Checklist
+
+## Phase A — Scaffold files
+
+- [ ] **A1. Create prompt directory**
+  - Path: `prompts/`
+  - Verify: directory exists at repo root
+
+- [ ] **A2. Create canonical prompt file**
+  - Path: `prompts/transcribe_document.md`
+  - Verify: file exists and is non-empty
+
+- [ ] **A3. (Recommended) Create prompt library README**
+  - Path: `prompts/README.md`
+  - Verify: includes naming + revision conventions
+
+---
+
+## Phase B — Author prompt content (core work)
+
+- [ ] **B1. Add Purpose section**
+  - States verbatim historical transcription objective
+  - Explicitly disallows summarization/paraphrase
+
+- [ ] **B2. Add Output Contract section**
+  - Plain text output expectation
+  - Preserve meaningful structure and reading order
+  - No fabricated text
+
+- [ ] **B3. Add Rule Set from `docs/Intent.md`**
+  - Misspellings/errors: `[sic]`
+  - Missing words: `[word]`
+  - Uncertain readings: `[guess?]`
+  - Illegible regions: `[illegible]` / reason labels
+  - Crossed-out text: `[deleted: ...]`
+  - Squeezed-in text: `[inserted: ...]`
+  - Superscripts/abbrev handling guidance
+  - Non-text visuals: bracketed descriptive labels
+  - Marginalia formatting cue
+  - Rejoin line-break hyphenated words silently
+  - Ambiguous capitalization policy
+  - Hierarchical outline numbering preservation
+
+- [ ] **B4. Add Ambiguity and Confidence policy**
+  - “Mark uncertainty instead of guessing”
+  - “Never silently normalize uncertain passages”
+
+- [ ] **B5. Add Final Self-Check section**
+  - Checklist for fidelity, uncertainty labeling, and format compliance
+
+---
+
+## Phase C — Add validations (tests)
+
+- [ ] **C1. Create prompt tests file**
+  - Path: `tests/test_prompts.py`
+
+- [ ] **C2. Add existence/health checks**
+  - Prompt file exists
+  - Prompt file has content (non-whitespace)
+
+- [ ] **C3. Add semantic anchor checks**
+  - Mentions verbatim behavior
+  - Mentions uncertainty marker pattern (`?` in brackets conceptually)
+  - Mentions illegible handling
+  - Mentions deleted/inserted conventions
+
+- [ ] **C4. Keep tests resilient**
+  - Avoid exact full-file snapshot assertions
+  - Assert required concepts, not precise phrasing
+
+---
+
+## Phase D — Documentation alignment
+
+- [ ] **D1. Update top-level docs/README reference**
+  - Mention that prompts live in `prompts/`
+  - Mention Step 3 loads from `PROMPT_DIR`
+
+- [ ] **D2. Confirm config compatibility**
+  - `src/transcription/config.py` already uses `prompt_dir = Path("./prompts")`
+  - No code change needed unless naming/path mismatch appears
+
+---
+
+## Phase E — Verification
+
+- [ ] **E1. Run targeted test file**
+  - `uv run pytest tests/test_prompts.py -q`
+
+- [ ] **E2. Run full suite**
+  - `uv run pytest -q`
+
+- [ ] **E3. Confirm no regressions**
+  - All existing tests still green (expected: previous 20 + new prompt tests)
+
+---
+
+## Phase F — Commit plan (recommended granularity)
+
+- [ ] **F1. Commit 1: scaffold**
+  - `prompts/transcribe_document.md` (initial structure)
+  - `prompts/README.md` (if included)
+
+- [ ] **F2. Commit 2: finalized prompt content**
+  - full rule-complete prompt text
+
+- [ ] **F3. Commit 3: tests + docs alignment**
+  - `tests/test_prompts.py`
+  - README/docs mention of prompt artifact pattern
+
+---
+
+## Done Criteria (quick gate)
+
+- [ ] Canonical prompt exists and is curated for verbatim transcription.
+- [ ] Prompt encodes all high-value handling rules from `docs/Intent.md`.
+- [ ] Prompt tests pass.
+- [ ] Full project tests pass with `uv`.
+- [ ] Ready for Step 3 provider integration.
+
+---
+
 ## Acceptance Criteria (Definition of Done)
 
 Step 2 is complete when all are true:

docs/step3.md

@@ -0,0 +1,188 @@
+## Step 3: services/transcription.py + providers/ 
+
+### Objective
+
+Implement the **AI transcription integration layer** so the app can:
+
+1. Read the curated prompt from `PROMPT_DIR`
+2. Send prompt + image to the configured provider (OpenRouter)
+3. Return normalized transcription output (or structured failure)
+
+This corresponds to MVP Step 3 from `docs/mvp.md`:
+- `services/transcription.py`
+- `providers/` adapter(s)
+
+---
+
+## Scope for Step 3
+
+### In scope
+- Provider abstraction and OpenRouter adapter
+- Prompt file loading utility in service layer
+- Image payload preparation
+- One high-level transcription service function usable by Step 4 worker
+- Unit tests (mocked provider SDK, no external calls)
+
+### Out of scope
+- Job polling/background loop (Step 4)
+- DB status transition orchestration in worker loop (Step 4)
+- UI invocation/wiring (Step 5)
+
+---
+
+## Planned Deliverables
+
+### Source files
+- `src/transcription/providers/base.py`
+- `src/transcription/providers/openrouter.py`
+- `src/transcription/providers/__init__.py` (exports + factory)
+- `src/transcription/services/transcription.py`
+- `src/transcription/services/__init__.py` (optional export)
+
+### Tests
+- `tests/test_providers_openrouter.py`
+- `tests/test_transcription_service.py`
+
+---
+
+## Design Decisions (before coding)
+
+1. **Provider interface first**
+   - Define a stable contract independent of SDK specifics.
+   - Prevent Step 4 from depending on raw SDK response shapes.
+
+2. **Service returns normalized result object**
+   - Include: `text`, `provider`, `model`, `raw_error`/exception metadata.
+   - Worker can map this cleanly to `Transcript` and `JobStatus`.
+
+3. **Prompt loaded from file at call time**
+   - Uses `get_settings().prompt_dir / "transcribe_document.md"`.
+   - Keeps prompt edits hot-swappable without code changes.
+
+4. **Clear exception boundary**
+   - SDK/network/model failures become predictable domain exceptions:
+     - `ProviderError`
+     - `PromptLoadError`
+     - `TranscriptionError` (optional top-level wrapper)
+
+5. **Model resolution policy**
+   - Use `settings.provider_model` if set
+   - Otherwise use adapter default constant (e.g., vision-capable model slug)
+
+---
+
+## Task-by-Task Execution Checklist
+
+## Phase A — Provider contract
+
+- [ ] Create `src/transcription/providers/base.py`
+- [ ] Define protocol/ABC for transcription providers:
+  - [ ] method signature accepts prompt text + image bytes (or data URL) + mime type
+  - [ ] returns normalized text result (and optional metadata)
+- [ ] Define shared provider exceptions:
+  - [ ] `ProviderError`
+  - [ ] optional subclasses (`ProviderAuthError`, `ProviderResponseError`)
+
+---
+
+## Phase B — OpenRouter adapter
+
+- [ ] Create `src/transcription/providers/openrouter.py`
+- [ ] Implement `OpenRouterTranscriptionProvider` with:
+  - [ ] config-driven API key usage
+  - [ ] optional referer/title attribution headers
+  - [ ] model resolution fallback when `provider_model` is unset
+- [ ] Implement request building:
+  - [ ] prompt included as instruction content
+  - [ ] image included in supported format for vision call
+- [ ] Implement response parsing:
+  - [ ] extract final transcript text from SDK response
+  - [ ] validate non-empty text
+- [ ] Wrap SDK failures into `ProviderError` with clean message
+
+---
+
+## Phase C — Provider factory
+
+- [ ] Update `src/transcription/providers/__init__.py`
+- [ ] Add `get_transcription_provider()` factory:
+  - [ ] reads `settings.provider`
+  - [ ] returns OpenRouter adapter for `openrouter`
+  - [ ] raises explicit error for unsupported provider values
+
+---
+
+## Phase D — Transcription service (Step 3 core)
+
+- [ ] Create `src/transcription/services/transcription.py`
+- [ ] Add prompt loader function:
+  - [ ] default file: `transcribe_document.md`
+  - [ ] raises `PromptLoadError` on missing/empty file
+- [ ] Add image loader/validator:
+  - [ ] path existence check
+  - [ ] allowed mime detection (`.jpg/.jpeg/.png/.tiff/.pdf` policy aligned to MVP)
+- [ ] Add high-level function (name example):
+  - [ ] `transcribe_document_image(image_path, prompt_name="transcribe_document.md")`
+  - [ ] loads prompt + image
+  - [ ] calls provider from factory
+  - [ ] returns normalized transcription result object
+- [ ] Add structured logging at key boundaries:
+  - [ ] prompt loaded
+  - [ ] provider invoked
+  - [ ] success/failure outcome (no sensitive data in logs)
+
+---
+
+## Phase E — Tests (mocked, deterministic)
+
+## `tests/test_providers_openrouter.py`
+- [ ] test adapter initializes from settings
+- [ ] test model fallback when `provider_model is None`
+- [ ] test referer/title options are included when set
+- [ ] test successful SDK response parses transcript text
+- [ ] test SDK exception maps to `ProviderError`
+- [ ] test empty/invalid response maps to `ProviderError`
+
+## `tests/test_transcription_service.py`
+- [ ] test prompt loader reads canonical prompt file
+- [ ] test missing prompt raises `PromptLoadError`
+- [ ] test transcription function loads file and calls provider once
+- [ ] test image path missing raises clear error
+- [ ] test provider error is propagated/wrapped predictably
+- [ ] test returned result includes transcript text and metadata
+
+> Keep these unit tests mocked (no real OpenRouter calls in default suite).
+
+---
+
+## Phase F — Verification commands
+
+- [ ] `uv run pytest tests/test_providers_openrouter.py -q`
+- [ ] `uv run pytest tests/test_transcription_service.py -q`
+- [ ] `uv run pytest -q`
+
+---
+
+## Implementation Notes / Guardrails
+
+- Avoid coupling Step 3 service to DB models directly (that belongs in Step 4 orchestration).
+- Do not silently swallow provider errors.
+- Keep prompt filename stable (`transcribe_document.md`) unless explicitly parameterized.
+- Keep request/response normalization inside provider adapter, not worker/UI layers.
+
+---
+
+## Definition of Done (Step 3)
+
+Step 3 is done when:
+
+1. Provider abstraction exists and OpenRouter adapter is implemented.
+2. Service can transcribe a local image using prompt file content.
+3. Failures are returned as structured exceptions, not raw SDK traceback noise.
+4. Unit tests for provider and service pass.
+5. Full suite remains green under `uv run pytest -q`.
+6. Step 4 can call a single service function to process queued jobs.
+
+---
+
+If you want, I can now convert this into a **PR-ready markdown checklist** (same format as Step 2) and then implement it once you confirm.

prompts/README.md

@@ -0,0 +1,13 @@
+# Prompt Artifacts
+
+This directory stores transcription prompts as individual Markdown artifacts.
+
+## Conventions
+- Keep one prompt per file.
+- Use stable, descriptive snake_case file names.
+- Prefer incremental edits to a single prompt per change for clean history.
+- Keep prompts human-readable and policy-focused.
+- Do not store secrets in prompt files.
+
+## Current Prompt
+- `transcribe_document.md`: baseline verbatim transcription policy for historical documents.

prompts/transcribe_document.md

@@ -0,0 +1,72 @@
+# Historical Document Verbatim Transcription Prompt
+
+## Purpose
+Transcribe the provided historical document image as a faithful **verbatim** transcript.
+Do not summarize. Do not paraphrase. Do not modernize style.
+
+## Output Contract
+- Return only the transcription text.
+- Preserve original wording, punctuation, and meaningful structure.
+- Keep line/section flow readable while preserving intent and document organization.
+- Never invent missing content.
+
+## Rules for Ambiguous or Damaged Text
+
+### Misspellings and original errors
+- Preserve original spelling.
+- Add `[sic]` immediately after an evident original error.
+
+### Missing words or clear omissions
+- If a single missing word is obvious from context, insert it in square brackets.
+- Example form: `[to]`
+
+### Uncertain readings
+- If best-effort interpretation is uncertain, use bracketed guess with question mark.
+- Example form: `[Boston?]`
+
+### Completely illegible text
+- Use a clear bracketed label.
+- Preferred forms: `[illegible]`, `[torn]`, `[ink blot]`, `[remainder of page torn]`
+
+### Crossed-out text
+- Preserve it using: `[deleted: ...]`
+
+### Squeezed-in or above-line insertions
+- Preserve it using: `[inserted: ...]`
+
+### Superscripts and abbreviations
+- Bring superscript letters down to baseline text.
+- Expand only when clearly intended; if expanded, place added letters in brackets.
+
+### Non-text visual elements
+- Describe briefly in square brackets.
+- Example forms: `[wax notary seal attached here]`, `[sketch of a fort layout]`
+
+### Marginalia and side notes
+- Signal location before the note text.
+- Example form: `[written in left margin: ...]`
+
+### Line-break hyphenation
+- Rejoin words split across line breaks when they are clearly one word.
+- Remove only line-break hyphens used for wrapping.
+
+### Ambiguous capitalization
+- Prefer modern capitalization only when uncertainty is high.
+- Preserve clearly intentional archaic capitalization.
+
+### Hierarchical outlines and numbering
+- Preserve original numbering characters exactly (including roman numerals and unusual suffixes).
+- Preserve indentation levels.
+- Do not silently correct sequence mistakes; if clearly erroneous, preserve and use `[sic]` where appropriate.
+
+## Confidence and Integrity Policy
+- When uncertain, mark uncertainty explicitly rather than guessing silently.
+- If text cannot be read, use a bracketed illegibility label instead of fabrication.
+- Do not add commentary outside the transcription.
+
+## Final Self-Check
+Before finalizing, ensure:
+1. The transcript is verbatim and not summarized.
+2. Uncertain/illegible areas are explicitly marked.
+3. Crossed-out and inserted text are preserved with required tags.
+4. Structure/ordering is preserved as faithfully as possible.

tests/test_prompts.py

@@ -0,0 +1,42 @@
+"""Tests for prompt artifacts in prompts/."""
+
+from pathlib import Path
+
+
+PROMPT_PATH = Path("prompts/transcribe_document.md")
+
+
+def _prompt_text() -> str:
+    """Read prompt text from the canonical prompt file."""
+    return PROMPT_PATH.read_text(encoding="utf-8")
+
+
+class TestPromptArtifact:
+    """Verify prompt artifact presence and baseline semantics."""
+
+    def test_prompt_file_exists(self):
+        """Canonical transcription prompt file exists."""
+        assert PROMPT_PATH.exists()
+
+    def test_prompt_file_is_not_empty(self):
+        """Canonical prompt file has non-whitespace content."""
+        text = _prompt_text()
+        assert text.strip()
+
+    def test_prompt_mentions_verbatim_behavior(self):
+        """Prompt explicitly enforces verbatim transcription behavior."""
+        text = _prompt_text().lower()
+        assert "verbatim" in text
+        assert "do not summarize" in text
+
+    def test_prompt_includes_uncertainty_and_illegible_markers(self):
+        """Prompt contains conventions for uncertainty and illegible text."""
+        text = _prompt_text().lower()
+        assert "[boston?]" in text
+        assert "[illegible]" in text
+
+    def test_prompt_includes_deleted_and_inserted_conventions(self):
+        """Prompt contains conventions for deleted and inserted text."""
+        text = _prompt_text().lower()
+        assert "[deleted:" in text
+        assert "[inserted:" in text