bbchops/transcription
1
0
Fork 0
generated from john/python-template
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Implement Ver1 Stage3

Browse Source
This commit is contained in:
Jim Lancaster
2026-06-26 14:23:12 -05:00
parent 21478a904c
commit 06bb4290be
17 changed files with 1457 additions and 62 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/ver1/ver1-step1-2-carry-forward-checklist.md
+4 -4
View File
@@ -28,14 +28,14 @@ Historical records remain unchanged:
| ID | Carry-Forward Task | Source | Related REQ | Owning V1 Step(s) | Validation Method | Status | Evidence Link/Note |
| --- | --- | --- | --- | --- | --- | --- | --- |
| CF-A1 | Confirm remaining implicit/global runtime ownership and lift only high-impact resources to lifespan ownership | Step 1 residual follow-up | REQ-7 | Step 3, Step 9 | Inspection + test | not started | |
| CF-A1 | Confirm remaining implicit/global runtime ownership and lift only high-impact resources to lifespan ownership | Step 1 residual follow-up | REQ-7 | Step 3, Step 9 | Inspection + test | in progress | Step 3 added `services/library.py` and `api/routes.py` using existing service/session access patterns; no new module-global runtime resource ownership introduced. Reconfirm in Step 9 release readiness. |
| CF-A2 | Finalize migration + rollback runbook usage and rehearse on representative local data | Step 1 residual follow-up | REQ-10 | Step 4, Step 9 | Demonstration + test | not started | |
| CF-A3 | Maintain lightweight boundary enforcement (review checklist and/or simple import checks) | Step 1 residual follow-up | REQ-7, REQ-11 | Step 3, Step 7 | Inspection | not started | |
| CF-A3 | Maintain lightweight boundary enforcement (review checklist and/or simple import checks) | Step 1 residual follow-up | REQ-7, REQ-11 | Step 3, Step 7 | Inspection | in progress | Step 3 implementation keeps UI/API composition thin and pushes revision/search/export logic to `services/library.py`; continue with Step 7 checks. |
| CF-B1 | Build compact error-path inventory for major failure paths and category mapping | Step 2 governance follow-up | REQ-2, REQ-3, REQ-4, REQ-5 | Step 6, Step 7 | Inspection | not started | Use `docs/ver1/ver1-step2-error-path-inventory.md` |
| CF-B2 | Standardize required logging fields at critical boundary handoffs | Step 2 residual follow-up | REQ-3, REQ-4, REQ-8 | Step 6 | Inspection + test | not started | |
| CF-B3 | Revisit retry backoff strategy only if observed runtime behavior justifies extra complexity | Step 2 residual follow-up | REQ-2, REQ-6 | Step 6, Step 8 | Analysis + test | deferred | Keep fixed backoff unless evidence suggests change |
| CF-C1 | Integrate Step 1/2 completed outcomes and open follow-ups into V1 traceability tracking | Revision-plan workstream | REQ-0..REQ-12 (traceability) | Step 3, Step 10 | Inspection | in progress | This checklist is the initial integration artifact |
| CF-C2 | Keep carry-forward routing aligned with revised V1 plan (architecture via 3/4/9, reliability via 6/7) | Revision-plan workstream | REQ-0..REQ-12 (execution alignment) | Step 3+ | Inspection | in progress | Routing established in this matrix |
| CF-C1 | Integrate Step 1/2 completed outcomes and open follow-ups into V1 traceability tracking | Revision-plan workstream | REQ-0..REQ-12 (traceability) | Step 3, Step 10 | Inspection | done | Step 3 artifacts added: `docs/ver1/ver1-step3.md`, `docs/ver1/ver1-step3-results.md`, and this checklist updated with Step 3 evidence and routing. |
| CF-C2 | Keep carry-forward routing aligned with revised V1 plan (architecture via 3/4/9, reliability via 6/7) | Revision-plan workstream | REQ-0..REQ-12 (execution alignment) | Step 3+ | Inspection | in progress | Step 3 execution followed routing: functional features implemented in Step 3; migration/rollback items remain in Step 4/9; logging/error-path standardization remains Step 6/7. |
---
docs/ver1/ver1-step3-results.md
+160
View File
@@ -0,0 +1,160 @@
# Ver1 Step 3 Results: Functional Completion by Requirement Domain
## Summary
Step 3 implementation has been completed for the planned functional-completion scope in a practical personal-scale form.
Implemented in this step:
1. Revision history and acceptance workflows for transcripts.
2. Search over accepted transcript revisions.
3. Export of accepted transcript data.
4. API routes for jobs, revisions, search, and export.
5. UI pathways for revision management, search, and export.
6. Carry-forward integration updates for Step 1/2 follow-ups owned by Step 3.
---
## Implemented Changes
### 1) Data model expansion (functional domain)
Updated `src/transcription/models.py`:
- Added `JobStatus.COMPLETED`.
- Added `TranscriptRevision` table/model:
- `job_id`
- `revision_number`
- `text`
- `source`
- `accepted`
- `created_at`
- Added `Job.revisions` relationship.
This supports immutable revision history and accepted-transcript semantics for search/export.
### 2) Step 3 service layer
Created `src/transcription/services/library.py` with service-backed functional operations:
- `list_jobs(...)`
- `get_job_detail(...)`
- `add_revision(...)`
- `accept_revision(...)`
- `list_revisions(...)`
- `search_accepted_transcripts(...)`
- `export_transcripts(...)`
Key behavior:
- revisions are append-only and incrementing
- accepted revision is unique per job
- accepting a revision syncs canonical transcript and sets job to `completed`
- search scope is accepted revisions only
- export emits deterministic record payloads for archive workflows
### 3) Worker integration for revision provenance
Updated `src/transcription/worker.py`:
- Success path now calls `add_revision(..., source="worker", accepted=False)`.
- Worker still persists canonical transcript and `transcribed` job state.
- Initial machine transcription now appears in revision history.
### 4) API functional completion
Created `src/transcription/api/routes.py` and wired in `src/transcription/app.py`.
New endpoints:
- `GET /api/jobs`
- `GET /api/jobs/{job_id}`
- `GET /api/jobs/{job_id}/revisions`
- `POST /api/jobs/{job_id}/revisions`
- `POST /api/revisions/{revision_id}/accept`
- `GET /api/search?query=...`
- `GET /api/export?accepted_only=true|false`
### 5) UI functional completion
Updated `src/transcription/ui/jobs_page.py`:
- Job detail now includes revision history panel.
- Added user revision submission.
- Added revision accept action.
- Added `/search` page for accepted transcript search.
- Added `/export` page for accepted transcript export preview.
---
## Test Evidence
### Added/Updated Tests
1. `tests/services/test_library.py`
- revision append/accept behavior
- accepted-only search behavior
- export payload behavior
2. `tests/api/test_routes.py`
- jobs/revisions/search/export API serialization and contract behavior
3. `tests/test_models.py`
- `completed` status transition coverage
- `TranscriptRevision` persistence and relationship coverage
4. `tests/services/test_worker.py`
- success-path now verifies initial worker-generated revision persistence
### Full Validation Run
Executed and passing:
- `uv run pytest -q`
---
## Requirement Slice Coverage (Step 3)
| Slice | REQ Coverage | Status | Evidence |
| --- | --- | --- | --- |
| Core lifecycle completion and visibility | REQ-0, REQ-2, REQ-3, REQ-5, REQ-6 | met | worker integration + API/UI jobs routes + tests |
| Revision history and acceptance | REQ-3, REQ-4, REQ-5, REQ-11 | met | `TranscriptRevision`, `services/library.py`, UI revision panel, tests |
| Search over accepted transcripts | REQ-5, REQ-11 | met | `search_accepted_transcripts`, `/api/search`, `/ui/search`, tests |
| Export transcript data | REQ-4, REQ-5, REQ-11 | met | `export_transcripts`, `/api/export`, `/ui/export`, tests |
| Prompt and verbatim flow continuity | REQ-12 | met (continued) | worker transcription flow unchanged in prompt-loading contract |
---
## Carry-Forward Integration Updates
Updated:
- `docs/ver1/ver1-step1-2-carry-forward-checklist.md`
Step 3 updates recorded for:
- CF-A1: in progress with Step 3 inspection evidence
- CF-A3: in progress with boundary-discipline evidence
- CF-C1: done (Step 3 traceability artifacts integrated)
- CF-C2: in progress (routing preserved for later steps)
---
## Residual Follow-ups
1. Step 4: migration rehearsal and rollback runbook execution for schema changes.
2. Step 6/7: broader error-path inventory closure and logging field normalization.
3. Step 9: release readiness reconfirmation for runtime ownership and migration behavior.
---
## Step 3 Exit Assessment
- Requirement-domain functional completion: **met**
- Data integrity and state consistency for new flows: **met**
- API/UI parity for new Step 3 features: **met**
- Test and regression safety: **met**
- Carry-forward integration obligations (Step 3-owned): **met/in progress as routed**
Step 3 is complete and ready to hand off to Step 4.
docs/ver1/ver1-step3.md
+433
View File
@@ -0,0 +1,433 @@
# Step 3 Implementation Plan: Functional Completion by Requirement Domain
## Purpose
Implement **Ver1 Step 3** from `docs/ver1/ver1.md` by completing all in-scope V1 functional requirements in a practical, user-first order while preserving:
- personal-scale operation
- single-operator workflow
- private-network deployment assumptions
- low operational overhead
- clean architecture boundaries
Primary governing docs:
- `docs/ver1/ver1.md` (Step 3 objective and sequencing)
- `docs/architecture.md` (module boundaries, workflow, simplicity guardrails)
- `docs/requirements.md` (REQ-0 through REQ-12 traceability)
- `docs/error_handling.md` (error contract across boundaries)
- `docs/intent.md` (verbatim transcription policy and prompt curation)
- `docs/ver1/ver1-step1-2-carry-forward-checklist.md` (Step 1/2 carry-forward integration)
- `docs/ver1/ver1-step2-error-path-inventory.md` (failure-path coverage visibility)
---
## MCP Resources Reviewed and Applied
All resources on `john-stream-mcp` were reviewed. Step 3 applies the following guidance directly:
1. `resource://skills/nicegui/document`
- modular page registration
- one-way dependency flow (`ui/api -> services -> infra`)
- async-first UI responsiveness expectations
2. `resource://skills/nicegui-ui-customization/document`
- reusable UI component extraction for repeated patterns
- in-flight guards and explicit success/failure user feedback
- event-driven updates over ad-hoc polling
3. `resource://skills/fastapi-async-sqlalchemy-modernization/document`
- explicit transaction/session boundaries
- deterministic resource ownership and cleanup continuity from Step 1
- incremental migration strategy with rollback-aware checkpoints
4. `resource://skills/pydantic-settings/document`
- typed configuration as single source of runtime truth
- explicit source precedence and environment-safe defaults
5. `resource://skills/python-logging-dictconfig/document`
- centralized startup-only logging configuration
- named logger discipline and boundary-level structured fields
6. `resource://skills/pytesting/document`
- deterministic test structure and marker discipline
- behavior-first tests with clear fast-path and full-suite validation
7. `resource://skills/fastapi-uv-docker/document`
- health endpoint and runtime startup/shutdown hygiene
- compose/deployment readiness constraints relevant to functional completion
8. `resource://skills/python-typing/document`
- modern typing updates where touched by Step 3 work
9. `resource://skills/ruff-linting-formating/document`
- maintain lint/format consistency in all modified modules
10. `resource://prompts/greenfield-architecture/document`
- explicit staged delivery with tradeoff-aware sequencing and test strategy
11. `resource://prompts/pytest-scaffold/document`
12. `resource://prompts/pytest-fill-scaffold/document`
- structure-first test planning, then deterministic implementation fill-in
Resources reviewed but not directly in Step 3 execution scope (no changes required now):
- `copilot-customization`, `mcp-details`, `vscode-configuration`, `zensical-docs`
- prompts: `authoring`, `mcp-consumer-repo-shim`
---
## Step 3 Success Criteria
Step 3 is complete when:
1. All Step 3-targeted requirement slices are implemented and verified.
2. Functional behavior is available through UI/API where required.
3. Core data integrity and state transitions are deterministic.
4. Error behavior follows `docs/error_handling.md` contracts.
5. Carry-forward Step 1/2 items mapped to Step 3 are updated with evidence.
---
## Requirement-Slice Execution Model (Applied to Every Slice)
For each slice, execute this sequence:
1. Confirm contract/schema and boundary ownership.
2. Implement service/domain logic.
3. Implement persistence/state transitions.
4. Integrate API and/or UI behavior.
5. Add/update unit + integration + targeted end-to-end tests.
6. Update docs and traceability artifacts.
Definition of done per slice:
- behavior is functional
- tests pass in intended marker lanes
- error pathways are classified and surfaced correctly
- requirement traceability is updated with evidence
---
## Detailed Workstreams
## Workstream A — Functional Baseline Audit and Slice Backlog Lock
### Goals
- establish exact Step 3 functional delta from current implementation
- lock a practical slice backlog before coding
### Tasks
1. Build Step 3 requirement matrix (REQ -> current status -> gap -> target slice).
2. Map each gap to one of these domains:
- Upload and lifecycle integrity
- Review and revision history
- Search over accepted transcripts
- Export workflows
- Prompt asset management behavior
- API/UI parity and status visibility
3. Align each slice with architecture boundary ownership and persistence strategy.
4. Link open carry-forward items from checklist:
- CF-A1, CF-A3 (architecture continuity in Step 3)
- CF-C1, CF-C2 (traceability/execution continuity)
### Deliverables
- Step 3 requirement-slice matrix (appendix in this doc or separate artifact)
- prioritized slice backlog with owner and validation method
### Exit Criteria
- every Step 3 slice maps to REQ IDs and a validation method
- no ambiguous ownership remains for in-scope slices
---
## Workstream B — Core End-User Flows (Upload -> Transcribe -> Review)
### Related Requirements
- REQ-0, REQ-1, REQ-2, REQ-3, REQ-4, REQ-5, REQ-6, REQ-12
### Goals
- guarantee end-to-end reliability and usability of the primary user flow
- ensure review experience supports transcript acceptance and correction
### Tasks
1. Validate and close any lifecycle-state gaps:
- enforce valid transitions (`queued -> processing -> transcribed/failed/completed`)
- ensure transition visibility in UI/API
2. Review experience completion:
- transcript detail display stability
- failure detail readability and actionability
- acceptance/edit path for human review
3. Ensure prompt-asset integration remains file-based and auditable:
- one prompt per Markdown file
- prompt selection/usage traceability in job outcomes (if available in model)
4. Confirm worker/UI interactions remain responsive under long-running jobs:
- in-flight guards
- clear status refresh behavior
### Deliverables
- complete end-user flow behavior with stable lifecycle visibility
- test coverage for happy path and failure path
### Exit Criteria
- user can run upload -> process -> review reliably
- failed and successful outcomes are both actionable and traceable
---
## Workstream C — Revision History and Provenance Completion
### Related Requirements
- REQ-3, REQ-4, REQ-5, REQ-11
### Goals
- finalize immutable transcript revision behavior and provenance consistency
### Tasks
1. Define/confirm revision invariants:
- append-only revision history
- clear current/accepted revision indicator
2. Persist revision events consistently through service layer boundaries.
3. Ensure UI/API expose revision timeline and selected revision details.
4. Align error handling for revision conflicts and missing resources.
### Deliverables
- revision-history feature completeness
- provenance and history read-path coverage
### Exit Criteria
- transcript edits produce deterministic revision records
- previous revisions remain inspectable
---
## Workstream D — Search Completion (Accepted Transcript Scope)
### Related Requirements
- REQ-0, REQ-5, REQ-11
### Goals
- provide practical search over accepted transcripts for personal corpus usage
### Tasks
1. Finalize searchable scope and indexing rules (accepted/current text only).
2. Implement service-backed search query behavior.
3. Expose search in UI/API with clear result metadata (document/job/revision context).
4. Add guardrails for empty/no-result/error scenarios with actionable messaging.
### Deliverables
- functional search pathway with deterministic results for accepted text
### Exit Criteria
- operator can find transcripts reliably by text queries
- no-result and error states are clear and non-silent
---
## Workstream E — Export Completion
### Related Requirements
- REQ-0, REQ-4, REQ-5, REQ-11
### Goals
- deliver practical export of transcript data for personal archive use
### Tasks
1. Finalize export contract (format, included fields, scope filters).
2. Implement export service with deterministic data mapping.
3. Add UI/API trigger path and user-visible completion/failure feedback.
4. Validate export integrity against persisted source-of-record entities.
### Deliverables
- end-to-end export capability with operator-visible outcomes
### Exit Criteria
- export output is complete, consistent, and usable for downstream personal archive workflows
---
## Workstream F — API/UI Parity and Interaction Hardening
### Related Requirements
- REQ-5 plus cross-cutting REQ-2/3/4
### Goals
- ensure UI and API expose coherent feature behavior and error contracts
### Tasks
1. Verify API/UI parity matrix for each Step 3 slice.
2. Standardize interaction behavior:
- loading and in-flight states
- success/failure notifications
- stable error_id visibility where user-facing
3. Ensure route/page modules remain composition-focused (business logic in services).
### Deliverables
- API/UI parity checklist with resolved gaps
### Exit Criteria
- no major flow exists in one interface with conflicting semantics in the other
---
## Workstream G — Carry-Forward Integration During Step 3
### Goals
- close Step 1/2 follow-ups that are Step 3-owned
### Tasks
1. Update checklist item CF-A1 as Step 3 slices touch runtime resources.
2. Update checklist item CF-A3 with lightweight boundary enforcement evidence.
3. Update CF-C1/CF-C2 traceability mapping with Step 3 outcomes.
### Deliverables
- updated `docs/ver1/ver1-step1-2-carry-forward-checklist.md` evidence entries
### Exit Criteria
- Step 3-owned carry-forward items are either completed or explicitly routed with evidence
---
## Test and Validation Plan
Apply `pytesting` guidance with deterministic, behavior-focused coverage.
### Validation Lanes
1. Structure/collection:
- `uv run pytest --collect-only -q`
2. Fast feedback lane:
- `uv run pytest -m unit -q`
3. Main verification lane:
- `uv run pytest -m "not external" -q`
4. Full suite:
- `uv run pytest -q`
### Required Coverage Areas
- lifecycle transition invariants
- revision history invariants
- search query behavior and result mapping
- export integrity and failure handling
- UI interaction guards and actionable failure feedback
- API envelope and status consistency for new/changed flows
### Test Design Rules
- one behavior target per test
- minimize heavy mocking; prefer real-path behavior checks where practical
- keep markers explicit and strict
---
## Logging, Error, and Config Guardrails for Step 3 Changes
1. Logging
- keep centralized startup logging config (`dictConfig`) as canonical
- include required error fields at boundary failures (`error_id`, `category`, `operation`, identifiers where available)
2. Error handling
- preserve taxonomy stability from `docs/error_handling.md`
- map any new failure pathways into existing categories
- surface actionable suggestions in UI/API
3. Configuration
- use typed settings and avoid ad-hoc env reads in business modules
- keep environment behavior explicit and documented
---
## Implementation Order (Detailed)
1. Workstream A: audit and backlog lock
2. Workstream B: core flow completion
3. Workstream C: revision/provenance completion
4. Workstream D: search completion
5. Workstream E: export completion
6. Workstream F: API/UI parity hardening
7. Workstream G: carry-forward integration updates
8. Full validation pass + docs/traceability updates
---
## Deliverables
1. Step 3 requirement-slice matrix with REQ mapping and evidence links
2. implemented Step 3 functional slices across service/persistence/API/UI
3. updated tests and passing validation lanes
4. updated carry-forward checklist entries (`CF-A1`, `CF-A3`, `CF-C1`, `CF-C2` as applicable)
5. Step 3 results document (`docs/ver1/ver1-step3-results.md`)
---
## Risks and Mitigations
1. **Risk:** Scope creep from optional enhancements during feature completion
- **Mitigation:** enforce REQ-mapped slice backlog and defer non-REQ enhancements
2. **Risk:** Functional parity drift between UI and API
- **Mitigation:** maintain parity matrix and verify both surfaces per slice
3. **Risk:** Data-model changes introduce migration surprises
- **Mitigation:** coordinate with Step 4 runbook expectations early and test on representative data
4. **Risk:** Reliability regressions while adding functionality
- **Mitigation:** run full error-path regression checks and keep Step 2 contracts intact
---
## Step 3 Completion Checklist
- [ ] Step 3 requirement-slice matrix completed and linked to REQ IDs.
- [ ] Core end-user flow is functionally complete and verified.
- [ ] Revision history/provenance behavior is complete and test-covered.
- [ ] Search over accepted transcripts is complete and test-covered.
- [ ] Export flow is complete and test-covered.
- [ ] API/UI parity checklist has no unresolved high-impact gaps.
- [ ] Step 3-owned carry-forward items are updated with evidence.
- [ ] Validation lanes pass (`collect-only`, unit, non-external, full).
- [ ] `docs/ver1/ver1-step3-results.md` is created with evidence and residual follow-ups.
---
## Handoff to Step 4
Step 3 completion enables Step 4 (Data Model and Migration Safety) with:
- finalized functional domain behavior
- stable persistence expectations
- traceable requirement evidence
- clarified migration-impact surface