generated from john/python-template
Ver1 Implementation Plan, and a detailed impl plan for step 1.
This commit is contained in:
Jim Lancaster
@@ -0,0 +1,278 @@
|
||||
## Step 2: prompts/transcribe_document.md
|
||||
|
||||
### Goal
|
||||
|
||||
Implement the MVP prompt artifact system by creating a curated transcription prompt file:
|
||||
|
||||
- `prompts/transcribe_document.md`
|
||||
|
||||
This step primarily satisfies:
|
||||
|
||||
- **REQ-12**: prompts stored as individual Markdown artifacts
|
||||
- MVP Feature 3: prompt-driven verbatim transcription behavior grounded in `docs/intent.md`
|
||||
|
||||
---
|
||||
|
||||
## Scope for Step 2
|
||||
|
||||
### In scope
|
||||
1. Create prompt artifact directory and first prompt file.
|
||||
2. Encode transcription rules from `docs/intent.md` into a model-facing prompt.
|
||||
3. Define stable prompt structure so future revisions are easy to diff/review.
|
||||
4. Add lightweight tests that validate artifact presence and baseline quality constraints.
|
||||
5. Update docs/README references so Step 3 can consume prompt file directly.
|
||||
|
||||
### Out of scope
|
||||
- Provider integration logic (Step 3)
|
||||
- Worker/job orchestration (Step 4)
|
||||
- UI behavior (Step 5)
|
||||
|
||||
---
|
||||
|
||||
## Proposed Deliverables
|
||||
|
||||
1. **`prompts/transcribe_document.md`**
|
||||
- production prompt text for historical document transcription
|
||||
|
||||
2. **`prompts/README.md`** (recommended)
|
||||
- conventions for prompt files, revision policy, naming
|
||||
|
||||
3. **`tests/test_prompts.py`** (recommended)
|
||||
- artifact existence + structure checks
|
||||
|
||||
4. **Small docs update** (README or docs reference)
|
||||
- indicate that prompts are file-based and loaded from `PROMPT_DIR`
|
||||
|
||||
---
|
||||
|
||||
## Detailed Work Breakdown
|
||||
|
||||
### 1) Create prompt artifact folder and canonical file
|
||||
- Add `prompts/` at repo root.
|
||||
- Add `transcribe_document.md` as the first curated artifact.
|
||||
- Keep filename stable; this becomes the default in Step 3 unless overridden.
|
||||
|
||||
### 2) Author prompt content using a strict, sectioned format
|
||||
Use section headers so future diffs are clean and policy changes are isolated.
|
||||
|
||||
Suggested sections:
|
||||
|
||||
1. **Purpose**
|
||||
- verbatim scholarly transcription of historical documents
|
||||
|
||||
2. **Output requirements**
|
||||
- plain text only
|
||||
- no summaries, no paraphrasing
|
||||
- preserve reading order and meaningful structure
|
||||
|
||||
3. **Core fidelity rules**
|
||||
- preserve original wording and punctuation
|
||||
- don’t silently normalize grammar/spelling
|
||||
- no invented content
|
||||
|
||||
4. **Issue-handling rules (mapped from Intent table)**
|
||||
- misspellings with `[sic]`
|
||||
- missing words with `[word]`
|
||||
- uncertainty with `[guess?]`
|
||||
- illegible with `[illegible]` / reason tags
|
||||
- crossed-out text as `[deleted: ...]`
|
||||
- inserted text as `[inserted: ...]`
|
||||
- superscripts handling guidance
|
||||
- non-text elements as `[description]`
|
||||
- marginalia format `[written in left margin: ...]`
|
||||
- line-break hyphen rejoin behavior
|
||||
- capitalization policy
|
||||
- hierarchical outline preservation (including unusual numbering)
|
||||
|
||||
5. **Confidence/ambiguity policy**
|
||||
- prefer explicit uncertainty markers over hallucination
|
||||
|
||||
6. **Final self-checklist for model**
|
||||
- did I preserve structure?
|
||||
- did I mark uncertain text?
|
||||
- did I avoid silent corrections?
|
||||
|
||||
### 3) Add prompt-library conventions (`prompts/README.md`)
|
||||
Recommended conventions:
|
||||
- one prompt per file
|
||||
- snake_case names
|
||||
- each file starts with purpose + behavior contract
|
||||
- iterative edits, one prompt per PR where possible
|
||||
- no secrets in prompt files
|
||||
|
||||
### 4) Add tests for prompt assets (`tests/test_prompts.py`)
|
||||
Keep tests robust but not brittle.
|
||||
|
||||
Recommended tests:
|
||||
1. `test_prompt_file_exists`
|
||||
2. `test_prompt_file_is_not_empty`
|
||||
3. `test_prompt_mentions_verbatim_behavior`
|
||||
4. `test_prompt_includes_uncertainty_and_illegible_markers`
|
||||
5. `test_prompt_includes_deleted_and_inserted_conventions`
|
||||
|
||||
Avoid exact full-text matching; verify key semantic anchors only.
|
||||
|
||||
### 5) Optional config alignment check
|
||||
Current config already has:
|
||||
- `prompt_dir: Path = Path("./prompts")`
|
||||
|
||||
In Step 2, ensure docs reflect this and that Step 3 will resolve:
|
||||
- `PROMPT_DIR / "transcribe_document.md"`
|
||||
|
||||
---
|
||||
|
||||
## Task-by-Task Execution Checklist
|
||||
|
||||
## Phase A — Scaffold files
|
||||
|
||||
- [ ] **A1. Create prompt directory**
|
||||
- Path: `prompts/`
|
||||
- Verify: directory exists at repo root
|
||||
|
||||
- [ ] **A2. Create canonical prompt file**
|
||||
- Path: `prompts/transcribe_document.md`
|
||||
- Verify: file exists and is non-empty
|
||||
|
||||
- [ ] **A3. (Recommended) Create prompt library README**
|
||||
- Path: `prompts/README.md`
|
||||
- Verify: includes naming + revision conventions
|
||||
|
||||
---
|
||||
|
||||
## Phase B — Author prompt content (core work)
|
||||
|
||||
- [ ] **B1. Add Purpose section**
|
||||
- States verbatim historical transcription objective
|
||||
- Explicitly disallows summarization/paraphrase
|
||||
|
||||
- [ ] **B2. Add Output Contract section**
|
||||
- Plain text output expectation
|
||||
- Preserve meaningful structure and reading order
|
||||
- No fabricated text
|
||||
|
||||
- [ ] **B3. Add Rule Set from `docs/intent.md`**
|
||||
- Misspellings/errors: `[sic]`
|
||||
- Missing words: `[word]`
|
||||
- Uncertain readings: `[guess?]`
|
||||
- Illegible regions: `[illegible]` / reason labels
|
||||
- Crossed-out text: `[deleted: ...]`
|
||||
- Squeezed-in text: `[inserted: ...]`
|
||||
- Superscripts/abbrev handling guidance
|
||||
- Non-text visuals: bracketed descriptive labels
|
||||
- Marginalia formatting cue
|
||||
- Rejoin line-break hyphenated words silently
|
||||
- Ambiguous capitalization policy
|
||||
- Hierarchical outline numbering preservation
|
||||
|
||||
- [ ] **B4. Add Ambiguity and Confidence policy**
|
||||
- “Mark uncertainty instead of guessing”
|
||||
- “Never silently normalize uncertain passages”
|
||||
|
||||
- [ ] **B5. Add Final Self-Check section**
|
||||
- Checklist for fidelity, uncertainty labeling, and format compliance
|
||||
|
||||
---
|
||||
|
||||
## Phase C — Add validations (tests)
|
||||
|
||||
- [ ] **C1. Create prompt tests file**
|
||||
- Path: `tests/test_prompts.py`
|
||||
|
||||
- [ ] **C2. Add existence/health checks**
|
||||
- Prompt file exists
|
||||
- Prompt file has content (non-whitespace)
|
||||
|
||||
- [ ] **C3. Add semantic anchor checks**
|
||||
- Mentions verbatim behavior
|
||||
- Mentions uncertainty marker pattern (`?` in brackets conceptually)
|
||||
- Mentions illegible handling
|
||||
- Mentions deleted/inserted conventions
|
||||
|
||||
- [ ] **C4. Keep tests resilient**
|
||||
- Avoid exact full-file snapshot assertions
|
||||
- Assert required concepts, not precise phrasing
|
||||
|
||||
---
|
||||
|
||||
## Phase D — Documentation alignment
|
||||
|
||||
- [ ] **D1. Update top-level docs/README reference**
|
||||
- Mention that prompts live in `prompts/`
|
||||
- Mention Step 3 loads from `PROMPT_DIR`
|
||||
|
||||
- [ ] **D2. Confirm config compatibility**
|
||||
- `src/transcription/config.py` already uses `prompt_dir = Path("./prompts")`
|
||||
- No code change needed unless naming/path mismatch appears
|
||||
|
||||
---
|
||||
|
||||
## Phase E — Verification
|
||||
|
||||
- [ ] **E1. Run targeted test file**
|
||||
- `uv run pytest tests/test_prompts.py -q`
|
||||
|
||||
- [ ] **E2. Run full suite**
|
||||
- `uv run pytest -q`
|
||||
|
||||
- [ ] **E3. Confirm no regressions**
|
||||
- All existing tests still green (expected: previous 20 + new prompt tests)
|
||||
|
||||
---
|
||||
|
||||
## Phase F — Commit plan (recommended granularity)
|
||||
|
||||
- [ ] **F1. Commit 1: scaffold**
|
||||
- `prompts/transcribe_document.md` (initial structure)
|
||||
- `prompts/README.md` (if included)
|
||||
|
||||
- [ ] **F2. Commit 2: finalized prompt content**
|
||||
- full rule-complete prompt text
|
||||
|
||||
- [ ] **F3. Commit 3: tests + docs alignment**
|
||||
- `tests/test_prompts.py`
|
||||
- README/docs mention of prompt artifact pattern
|
||||
|
||||
---
|
||||
|
||||
## Done Criteria (quick gate)
|
||||
|
||||
- [ ] Canonical prompt exists and is curated for verbatim transcription.
|
||||
- [ ] Prompt encodes all high-value handling rules from `docs/intent.md`.
|
||||
- [ ] Prompt tests pass.
|
||||
- [ ] Full project tests pass with `uv`.
|
||||
- [ ] Ready for Step 3 provider integration.
|
||||
|
||||
---
|
||||
|
||||
## Acceptance Criteria (Definition of Done)
|
||||
|
||||
Step 2 is complete when all are true:
|
||||
|
||||
1. `prompts/transcribe_document.md` exists and is committed.
|
||||
2. Prompt includes all critical handling rules from `docs/intent.md`.
|
||||
3. Prompt is structured with stable section headings for future curation.
|
||||
4. Prompt tests pass under `uv run pytest -q`.
|
||||
5. Existing tests remain green (total suite still passes).
|
||||
6. Docs indicate prompt artifact location and curation policy.
|
||||
|
||||
---
|
||||
|
||||
## Risks and Mitigations
|
||||
|
||||
1. **Risk: prompt too vague → hallucinated reconstructions**
|
||||
- Mitigation: explicit uncertainty/illegible conventions and “no invention” rule.
|
||||
|
||||
2. **Risk: prompt too rigid for mixed document types**
|
||||
- Mitigation: include neutral defaults + clear annotation formats.
|
||||
|
||||
3. **Risk: brittle tests block iterative prompt tuning**
|
||||
- Mitigation: test semantic anchors, not exact wording.
|
||||
|
||||
---
|
||||
|
||||
## Handoff to Step 3
|
||||
|
||||
After Step 2, Step 3 can immediately:
|
||||
1. Load `transcribe_document.md` from `PROMPT_DIR`
|
||||
2. Inject prompt into OpenRouter request
|
||||
3. Start validating real transcription behavior with minimal glue code
|
||||
Reference in New Issue
Block a user