generated from john/python-template
Merge branch 'main' into diverging
This commit is contained in:
John Lancaster
@@ -34,19 +34,62 @@ Optional settings (defaults shown):
|
||||
DATABASE_URL=sqlite:///./transcription.db
|
||||
UPLOAD_DIR=./uploads
|
||||
PROMPT_DIR=./prompts
|
||||
MAX_UPLOAD_BYTES=15728640
|
||||
OPERATOR_ACCESS_ENABLED=false
|
||||
OPERATOR_USERNAME=operator
|
||||
# OPERATOR_PASSWORD=replace_with_secure_value
|
||||
```
|
||||
|
||||
|
||||
### 3) Run the app
|
||||
|
||||
```bash
|
||||
uv run uvicorn transcription.app:create_app --factory --reload
|
||||
```
|
||||
|
||||
### 4) Open in browser
|
||||
### 4) (Optional) Run explicit migrations/checks
|
||||
|
||||
Use the migration runner for Step 4 schema safety workflows:
|
||||
|
||||
```bash
|
||||
uv run python -m transcription.migration_runner --list
|
||||
uv run python -m transcription.migration_runner --apply
|
||||
uv run python -m transcription.migration_runner --check
|
||||
```
|
||||
|
||||
### 5) Open in browser
|
||||
|
||||
|
||||
- GUI: [http://[IP_ADDRESS]:8000/ui](http://[IP_ADDRESS]:8000/ui)
|
||||
- Health check: [http://[IP_ADDRESS]:8000/healthz](http://[IP_ADDRESS]:8000/healthz)
|
||||
|
||||
### Schema safety settings
|
||||
|
||||
Optional environment settings (defaults shown):
|
||||
|
||||
```env
|
||||
MIGRATION_AUTO_APPLY_ON_STARTUP=false
|
||||
VALIDATE_SCHEMA_ON_STARTUP=true
|
||||
```
|
||||
|
||||
### Step 5 security settings
|
||||
|
||||
Use this baseline for trusted private-network operation:
|
||||
|
||||
```env
|
||||
OPERATOR_ACCESS_ENABLED=true
|
||||
OPERATOR_USERNAME=operator
|
||||
OPERATOR_PASSWORD=replace_with_strong_local_secret
|
||||
MAX_UPLOAD_BYTES=15728640
|
||||
```
|
||||
|
||||
Notes:
|
||||
- `/healthz` remains unauthenticated for operational checks.
|
||||
- `/ui` and `/api` require HTTP Basic credentials when operator access is enabled.
|
||||
- Keep `OPERATOR_PASSWORD` in environment variables only (never commit secrets).
|
||||
|
||||
|
||||
|
||||
## How to navigate the GUI
|
||||
|
||||
- **Upload page** (`/ui`)
|
||||
|
||||
+2
-4
@@ -6,7 +6,7 @@ This project is a production application for transcribing and preserving histori
|
||||
|
||||
Read [architecture.md](architecture.md) first.
|
||||
|
||||
Then review [ver1/ver1.md](ver1/ver1.md) for completion scope and [ver1/ver1-step1-results.md](ver1/ver1-step1-results.md) for current architecture-consolidation status.
|
||||
Then review [ver1/ver1.md](ver1/ver1.md) for completion scope.
|
||||
|
||||
The architecture page is the primary technical reference and defines:
|
||||
|
||||
@@ -41,10 +41,8 @@ This operating model keeps deployment and maintenance simple while preserving cl
|
||||
|
||||
## Documentation Map
|
||||
|
||||
- Architecture and technical design: [architecture.md](architecture.md)
|
||||
- Version 1 implementation plan: [ver1/ver1.md](ver1/ver1.md)
|
||||
- Version 1 Step 1 plan: [ver1/ver1-step1.md](ver1/ver1-step1.md)
|
||||
- Version 1 Step 1 results: [ver1/ver1-step1-results.md](ver1/ver1-step1-results.md)
|
||||
- Architecture and technical design: [architecture.md](architecture.md)
|
||||
- Architecture decision records (ADR index): [adr/README.md](adr/README.md)
|
||||
- Runtime and deployment requirements: [requirements.md](requirements.md)
|
||||
- Error handling policy and operational guidance: [error_handling.md](error_handling.md)
|
||||
|
||||
@@ -0,0 +1,73 @@
|
||||
# Ver1 Step 1/2 Carry-Forward Checklist
|
||||
|
||||
## Purpose
|
||||
|
||||
Track open Step 1 and Step 2 follow-ups through later V1 steps, with lightweight verification evidence and requirement traceability.
|
||||
|
||||
This artifact implements the carry-forward approach defined in:
|
||||
- `docs/ver1/ver1-step1-2_revised.md`
|
||||
|
||||
Historical records remain unchanged:
|
||||
- `docs/ver1/ver1-step1.md`
|
||||
- `docs/ver1/ver1-step1-results.md`
|
||||
- `docs/ver1/ver1-step2.md`
|
||||
- `docs/ver1/ver1-step2-results.md`
|
||||
|
||||
---
|
||||
|
||||
## Status Legend
|
||||
|
||||
- `not started`
|
||||
- `in progress`
|
||||
- `done`
|
||||
- `deferred`
|
||||
|
||||
---
|
||||
|
||||
## Carry-Forward Mapping Matrix
|
||||
|
||||
| ID | Carry-Forward Task | Source | Related REQ | Owning V1 Step(s) | Validation Method | Status | Evidence Link/Note |
|
||||
| --- | --- | --- | --- | --- | --- | --- | --- |
|
||||
| CF-A1 | Confirm remaining implicit/global runtime ownership and lift only high-impact resources to lifespan ownership | Step 1 residual follow-up | REQ-7 | Step 3, Step 9 | Inspection + test | in progress | Step 3 added `services/library.py` and `api/routes.py` using existing service/session access patterns; no new module-global runtime resource ownership introduced. Reconfirm in Step 9 release readiness. |
|
||||
| CF-A2 | Finalize migration + rollback runbook usage and rehearse on representative local data | Step 1 residual follow-up | REQ-10 | Step 4, Step 9 | Demonstration + test | not started | |
|
||||
| CF-A3 | Maintain lightweight boundary enforcement (review checklist and/or simple import checks) | Step 1 residual follow-up | REQ-7, REQ-11 | Step 3, Step 7 | Inspection | in progress | Step 3 implementation keeps UI/API composition thin and pushes revision/search/export logic to `services/library.py`; continue with Step 7 checks. |
|
||||
| CF-B1 | Build compact error-path inventory for major failure paths and category mapping | Step 2 governance follow-up | REQ-2, REQ-3, REQ-4, REQ-5 | Step 6, Step 7 | Inspection | not started | Use `docs/ver1/ver1-step2-error-path-inventory.md` |
|
||||
| CF-B2 | Standardize required logging fields at critical boundary handoffs | Step 2 residual follow-up | REQ-3, REQ-4, REQ-8 | Step 6 | Inspection + test | not started | |
|
||||
| CF-B3 | Revisit retry backoff strategy only if observed runtime behavior justifies extra complexity | Step 2 residual follow-up | REQ-2, REQ-6 | Step 6, Step 8 | Analysis + test | deferred | Keep fixed backoff unless evidence suggests change |
|
||||
| CF-C1 | Integrate Step 1/2 completed outcomes and open follow-ups into V1 traceability tracking | Revision-plan workstream | REQ-0..REQ-12 (traceability) | Step 3, Step 10 | Inspection | done | Step 3 artifacts added: `docs/ver1/ver1-step3.md`, `docs/ver1/ver1-step3-results.md`, and this checklist updated with Step 3 evidence and routing. |
|
||||
| CF-C2 | Keep carry-forward routing aligned with revised V1 plan (architecture via 3/4/9, reliability via 6/7) | Revision-plan workstream | REQ-0..REQ-12 (execution alignment) | Step 3+ | Inspection | in progress | Step 3 execution followed routing: functional features implemented in Step 3; migration/rollback items remain in Step 4/9; logging/error-path standardization remains Step 6/7. |
|
||||
|
||||
---
|
||||
|
||||
## Execution Notes
|
||||
|
||||
### Step 3 (Functional Completion)
|
||||
- Use CF-A1 and CF-A3 during requirement-slice implementation reviews.
|
||||
- Record any discovered boundary/runtime ownership gaps in this checklist.
|
||||
|
||||
### Step 4 (Data Model and Migration Safety)
|
||||
- Execute CF-A2 rehearsal and link evidence (commands, runbook notes, outcomes).
|
||||
|
||||
### Step 6 (Minimal Observability & Operability)
|
||||
- Execute CF-B1 and CF-B2 with focused artifacts and log-field verification.
|
||||
|
||||
### Step 7 (Test Coverage and Practical Quality Gates)
|
||||
- Add/verify tests supporting CF-A3 and CF-B1/B2 where meaningful.
|
||||
|
||||
### Step 8 (Performance Validation)
|
||||
- Reassess CF-B3 only if retries/backoff are observed to cause practical issues.
|
||||
|
||||
### Step 9 (Release Readiness)
|
||||
- Reconfirm CF-A1/A2 readiness in release checklist and rollback drill.
|
||||
|
||||
### Step 10 (Documentation Completion)
|
||||
- Ensure final V1 docs reference outcomes from this checklist where relevant.
|
||||
|
||||
---
|
||||
|
||||
## Acceptance Check for Carry-Forward Completion
|
||||
|
||||
- [ ] Historical Step 1/2 documents remain unchanged.
|
||||
- [ ] Every open Step 1/2 follow-up has an owning V1 step and validation method.
|
||||
- [ ] Evidence links are recorded for each completed carry-forward item.
|
||||
- [ ] No carry-forward item introduces unnecessary complexity for personal-scale operation.
|
||||
@@ -0,0 +1,166 @@
|
||||
# Ver1 Step 1 & Step 2 Revision Plan (Additive)
|
||||
|
||||
## Purpose
|
||||
|
||||
Define a **targeted implementation follow-through plan** for Step 1 and Step 2 outcomes so remaining V1 work stays aligned with `docs/ver1/ver1.md`:
|
||||
|
||||
- personal-scale operation
|
||||
- single operator
|
||||
- private-network assumptions
|
||||
- low operational overhead
|
||||
- practical, testable controls
|
||||
|
||||
This document is additive and does **not** replace or revise historical Step 1/Step 2 records.
|
||||
|
||||
---
|
||||
|
||||
## Source Documents Reviewed
|
||||
|
||||
- `docs/ver1/ver1.md`
|
||||
- `docs/ver1/ver1-step1.md`
|
||||
- `docs/ver1/ver1-step1-results.md`
|
||||
- `docs/ver1/ver1-step2.md`
|
||||
- `docs/ver1/ver1-step2-results.md`
|
||||
- `docs/architecture.md`
|
||||
- `docs/error_handling.md`
|
||||
- `docs/requirements.md`
|
||||
- `docs/index.md`
|
||||
- `docs/intent.md`
|
||||
|
||||
---
|
||||
|
||||
## Revision Goals
|
||||
|
||||
1. Preserve all completed Step 1/Step 2 technical hardening work.
|
||||
2. Keep historical Step 1/Step 2 documents unchanged.
|
||||
3. Convert residual risks/follow-ups into concrete implementation tasks for subsequent V1 steps.
|
||||
4. Preserve traceability to requirements and implemented evidence.
|
||||
5. Maintain alignment with personal-scale architecture and operating model.
|
||||
|
||||
---
|
||||
|
||||
## Scope
|
||||
|
||||
### In Scope
|
||||
- Define carry-forward implementation tasks based on Step 1/2 residual risks and open items.
|
||||
- Map carry-forward tasks to later V1 steps (especially Steps 3, 4, 6, 7, and 9).
|
||||
- Define lightweight verification evidence expected for each carry-forward task.
|
||||
- Update V1 traceability references to include completed Step 1/2 outcomes and deferred follow-ups.
|
||||
|
||||
### Out of Scope
|
||||
- Simplifying tone/structure of existing Step 1/2 documents
|
||||
- Clarifying or rewriting historical Step 1/2 plan/results content
|
||||
- Editing `docs/ver1/ver1-step1.md`
|
||||
- Editing `docs/ver1/ver1-step1-results.md`
|
||||
- Editing `docs/ver1/ver1-step2.md`
|
||||
- Editing `docs/ver1/ver1-step2-results.md`
|
||||
- Re-implementing Step 1/2 code changes
|
||||
- Rewriting `docs/ver1/ver1.md`
|
||||
- Deleting historical sections/results
|
||||
- Altering requirements IDs or architecture principles
|
||||
|
||||
---
|
||||
|
||||
## Carry-Forward Implementation Plan
|
||||
|
||||
## Workstream A — Close Step 1 follow-ups through later V1 steps
|
||||
|
||||
### A1) Runtime ownership completion (REQ-7 continuity)
|
||||
- Confirm whether any remaining runtime resources still use implicit/global ownership.
|
||||
- Move only high-impact remaining resources to explicit lifespan ownership when needed.
|
||||
- Keep ownership model simple and documented.
|
||||
|
||||
### A2) Schema/migration operations readiness (REQ-10 continuity)
|
||||
- Finalize practical migration + rollback runbook usage in Step 4 execution.
|
||||
- Rehearse upgrade and rollback on representative local data.
|
||||
- Keep production startup free from implicit schema mutation.
|
||||
|
||||
### A3) Boundary enforcement (lightweight only)
|
||||
- Keep architecture boundary checks lightweight (review checklist and/or simple import checks).
|
||||
- Avoid heavy governance tooling unless clear recurring drift appears.
|
||||
|
||||
### Expected Outcome
|
||||
Step 1 architecture hardening remains intact and is completed pragmatically where open items remain.
|
||||
|
||||
---
|
||||
|
||||
## Workstream B — Close Step 2 follow-ups through later V1 steps
|
||||
|
||||
### B1) Error-path inventory and coverage visibility
|
||||
- Create a compact error-path inventory artifact (or equivalent matrix section) covering major failure paths.
|
||||
- Ensure each critical path maps to category, retriable policy, and surfaced behavior.
|
||||
|
||||
### B2) Logging field consistency at key boundaries
|
||||
- Standardize required fields at critical failure handoffs (`error_id`, `category`, `operation`, identifiers when available).
|
||||
- Prioritize worker/API/service boundaries first.
|
||||
|
||||
### B3) Retry policy refinement (only if needed)
|
||||
- Keep current bounded retry baseline.
|
||||
- Revisit richer backoff strategy only if observed behavior justifies added complexity.
|
||||
|
||||
### Expected Outcome
|
||||
Step 2 reliability behavior stays stable, diagnosable, and right-sized for personal-scale operation.
|
||||
|
||||
---
|
||||
|
||||
## Workstream C — Integrate Step 1/2 outputs into ongoing V1 governance
|
||||
|
||||
### C1) Traceability integration
|
||||
- Link completed Step 1/2 outcomes and deferred follow-ups to the V1 traceability matrix.
|
||||
- Ensure open follow-ups have owning step and validation method.
|
||||
|
||||
### C2) Execution alignment with revised V1 plan
|
||||
- Route architecture follow-ups primarily through Steps 3/4/9.
|
||||
- Route reliability/diagnostics follow-ups primarily through Steps 6/7.
|
||||
|
||||
### Expected Outcome
|
||||
Step 1/2 work is fully carried forward without revising historical documents.
|
||||
|
||||
## Deliverables
|
||||
|
||||
1. This document (`docs/ver1/ver1-step1-2_revised.md`) as the carry-forward implementation plan.
|
||||
2. A compact Step 1/2 carry-forward checklist linked to V1 steps and validation methods.
|
||||
3. Traceability updates showing where each open Step 1/2 follow-up will be closed.
|
||||
4. Optional new artifact for error-path inventory (if created during Step 6/7 execution).
|
||||
|
||||
---
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- Historical Step 1/Step 2 documents remain unchanged.
|
||||
- Open Step 1/2 follow-ups are explicitly mapped to later V1 steps with validation expectations.
|
||||
- No loss of core technical intent (REQ-7, REQ-10, error taxonomy, retry safety, traceability).
|
||||
- No conflicts introduced with `docs/architecture.md`, `docs/error_handling.md`, or `docs/ver1/ver1.md`.
|
||||
- Carry-forward tasks remain right-sized for personal-scale operation.
|
||||
|
||||
---
|
||||
|
||||
## Implementation Order
|
||||
|
||||
1. Keep existing Step 1/Step 2 docs unchanged as historical records.
|
||||
2. Define carry-forward tasks and owning V1 steps in this document.
|
||||
3. Create and maintain carry-forward traceability artifacts:
|
||||
- `docs/ver1/ver1-step1-2-carry-forward-checklist.md`
|
||||
- `docs/ver1/ver1-step2-error-path-inventory.md`
|
||||
4. Execute carry-forward tasks during Steps 3+ and capture evidence in step results docs.
|
||||
5. Perform final consistency pass across `docs/ver1/*` references.
|
||||
|
||||
---
|
||||
|
||||
## Risks and Mitigations
|
||||
|
||||
1. **Risk:** Open Step 1/2 items are forgotten as Step 3+ work proceeds.
|
||||
**Mitigation:** Track each follow-up in the V1 traceability matrix with owning step and evidence expectation.
|
||||
|
||||
2. **Risk:** Carry-forward work expands beyond personal-scale needs.
|
||||
**Mitigation:** Apply simplicity guardrails from `docs/architecture.md` before accepting additional hardening tasks.
|
||||
|
||||
3. **Risk:** Reliability follow-ups become fragmented across multiple steps.
|
||||
**Mitigation:** Keep one consolidated carry-forward checklist and update it at milestone check-ins.
|
||||
|
||||
---
|
||||
|
||||
## Notes
|
||||
|
||||
This revision effort is scope-alignment and implementation-follow-through focused.
|
||||
Historical Step 1/Step 2 documents are intentionally preserved as-is.
|
||||
@@ -0,0 +1,42 @@
|
||||
# Ver1 Step 2 Error-Path Inventory (Carry-Forward)
|
||||
|
||||
## Purpose
|
||||
|
||||
Provide a compact inventory of major failure paths with taxonomy mapping and retry behavior, aligned with:
|
||||
- `docs/error_handling.md`
|
||||
- `docs/ver1/ver1-step2-results.md`
|
||||
- `docs/ver1/ver1-step1-2-carry-forward-checklist.md` (CF-B1)
|
||||
|
||||
This is a lightweight operational artifact for Step 6/7 follow-through.
|
||||
|
||||
---
|
||||
|
||||
## Inventory Table
|
||||
|
||||
| Path ID | Boundary/Operation | Typical Failure Source | Category | Retriable | Surface Behavior | Current Coverage | Notes |
|
||||
| --- | --- | --- | --- | --- | --- | --- | --- |
|
||||
| EP-API-001 | API upload request validation | invalid payload / empty file metadata | `validation_error` | no | structured API error envelope (400) | partial | confirm all upload variants |
|
||||
| EP-API-002 | API resource lookup | missing job/document | `not_found_error` | no | structured API error envelope (404) | partial | verify consistency for all lookup routes |
|
||||
| EP-SVC-001 | Service provider-call mapping | provider SDK/HTTP failure | `external_provider_error` | sometimes | normalized AppError and safe message | partial | ensure consistent mapping in service boundary tests |
|
||||
| EP-WKR-001 | Worker provider timeout | timeout/unavailable upstream | `external_provider_error` or `infrastructure_transient_error` | yes | retry or terminal failed with persisted reason | partial | validate category mapping remains deterministic |
|
||||
| EP-WKR-002 | Worker non-retriable domain/input failure | deterministic invalid input/state | `user_input_error` or `conflict_error` | no | immediate terminal failed with persisted reason | partial | ensure no retry on non-retriable categories |
|
||||
| EP-WKR-003 | Worker retry exhaustion | repeated retriable failure | category from source; terminal state | capped then no | explicit failed status + error detail | met | implemented in Step 2; keep regression coverage |
|
||||
| EP-UI-001 | UI upload action failure | surfaced AppError or fallback exception | category-based safe user message | category-driven | title + message + suggestion + error id | partial | verify consistency on all primary UI actions |
|
||||
| EP-LOG-001 | Cross-boundary error logging | missing/uneven fields | n/a | n/a | logs include `error_id`, `category`, `operation`, ids when available | partial | complete in Step 6 (CF-B2) |
|
||||
|
||||
---
|
||||
|
||||
## Verification Targets (Step 6/7)
|
||||
|
||||
1. Every critical path has category + retriable policy defined.
|
||||
2. API/UI behavior remains safe and actionable.
|
||||
3. Worker terminal failures are explicit and persisted.
|
||||
4. Logging fields are consistent at critical handoffs.
|
||||
|
||||
---
|
||||
|
||||
## Evidence Links
|
||||
|
||||
- Step 2 implementation results: `docs/ver1/ver1-step2-results.md`
|
||||
- Carry-forward tracking: `docs/ver1/ver1-step1-2-carry-forward-checklist.md`
|
||||
- Canonical contract: `docs/error_handling.md`
|
||||
@@ -0,0 +1,160 @@
|
||||
# Ver1 Step 3 Results: Functional Completion by Requirement Domain
|
||||
|
||||
## Summary
|
||||
|
||||
Step 3 implementation has been completed for the planned functional-completion scope in a practical personal-scale form.
|
||||
|
||||
Implemented in this step:
|
||||
|
||||
1. Revision history and acceptance workflows for transcripts.
|
||||
2. Search over accepted transcript revisions.
|
||||
3. Export of accepted transcript data.
|
||||
4. API routes for jobs, revisions, search, and export.
|
||||
5. UI pathways for revision management, search, and export.
|
||||
6. Carry-forward integration updates for Step 1/2 follow-ups owned by Step 3.
|
||||
|
||||
---
|
||||
|
||||
## Implemented Changes
|
||||
|
||||
### 1) Data model expansion (functional domain)
|
||||
|
||||
Updated `src/transcription/models.py`:
|
||||
|
||||
- Added `JobStatus.COMPLETED`.
|
||||
- Added `TranscriptRevision` table/model:
|
||||
- `job_id`
|
||||
- `revision_number`
|
||||
- `text`
|
||||
- `source`
|
||||
- `accepted`
|
||||
- `created_at`
|
||||
- Added `Job.revisions` relationship.
|
||||
|
||||
This supports immutable revision history and accepted-transcript semantics for search/export.
|
||||
|
||||
### 2) Step 3 service layer
|
||||
|
||||
Created `src/transcription/services/library.py` with service-backed functional operations:
|
||||
|
||||
- `list_jobs(...)`
|
||||
- `get_job_detail(...)`
|
||||
- `add_revision(...)`
|
||||
- `accept_revision(...)`
|
||||
- `list_revisions(...)`
|
||||
- `search_accepted_transcripts(...)`
|
||||
- `export_transcripts(...)`
|
||||
|
||||
Key behavior:
|
||||
|
||||
- revisions are append-only and incrementing
|
||||
- accepted revision is unique per job
|
||||
- accepting a revision syncs canonical transcript and sets job to `completed`
|
||||
- search scope is accepted revisions only
|
||||
- export emits deterministic record payloads for archive workflows
|
||||
|
||||
### 3) Worker integration for revision provenance
|
||||
|
||||
Updated `src/transcription/worker.py`:
|
||||
|
||||
- Success path now calls `add_revision(..., source="worker", accepted=False)`.
|
||||
- Worker still persists canonical transcript and `transcribed` job state.
|
||||
- Initial machine transcription now appears in revision history.
|
||||
|
||||
### 4) API functional completion
|
||||
|
||||
Created `src/transcription/api/routes.py` and wired in `src/transcription/app.py`.
|
||||
|
||||
New endpoints:
|
||||
|
||||
- `GET /api/jobs`
|
||||
- `GET /api/jobs/{job_id}`
|
||||
- `GET /api/jobs/{job_id}/revisions`
|
||||
- `POST /api/jobs/{job_id}/revisions`
|
||||
- `POST /api/revisions/{revision_id}/accept`
|
||||
- `GET /api/search?query=...`
|
||||
- `GET /api/export?accepted_only=true|false`
|
||||
|
||||
### 5) UI functional completion
|
||||
|
||||
Updated `src/transcription/ui/jobs_page.py`:
|
||||
|
||||
- Job detail now includes revision history panel.
|
||||
- Added user revision submission.
|
||||
- Added revision accept action.
|
||||
- Added `/search` page for accepted transcript search.
|
||||
- Added `/export` page for accepted transcript export preview.
|
||||
|
||||
---
|
||||
|
||||
## Test Evidence
|
||||
|
||||
### Added/Updated Tests
|
||||
|
||||
1. `tests/services/test_library.py`
|
||||
- revision append/accept behavior
|
||||
- accepted-only search behavior
|
||||
- export payload behavior
|
||||
|
||||
2. `tests/api/test_routes.py`
|
||||
- jobs/revisions/search/export API serialization and contract behavior
|
||||
|
||||
3. `tests/test_models.py`
|
||||
- `completed` status transition coverage
|
||||
- `TranscriptRevision` persistence and relationship coverage
|
||||
|
||||
4. `tests/services/test_worker.py`
|
||||
- success-path now verifies initial worker-generated revision persistence
|
||||
|
||||
### Full Validation Run
|
||||
|
||||
Executed and passing:
|
||||
|
||||
- `uv run pytest -q`
|
||||
|
||||
---
|
||||
|
||||
## Requirement Slice Coverage (Step 3)
|
||||
|
||||
| Slice | REQ Coverage | Status | Evidence |
|
||||
| --- | --- | --- | --- |
|
||||
| Core lifecycle completion and visibility | REQ-0, REQ-2, REQ-3, REQ-5, REQ-6 | met | worker integration + API/UI jobs routes + tests |
|
||||
| Revision history and acceptance | REQ-3, REQ-4, REQ-5, REQ-11 | met | `TranscriptRevision`, `services/library.py`, UI revision panel, tests |
|
||||
| Search over accepted transcripts | REQ-5, REQ-11 | met | `search_accepted_transcripts`, `/api/search`, `/ui/search`, tests |
|
||||
| Export transcript data | REQ-4, REQ-5, REQ-11 | met | `export_transcripts`, `/api/export`, `/ui/export`, tests |
|
||||
| Prompt and verbatim flow continuity | REQ-12 | met (continued) | worker transcription flow unchanged in prompt-loading contract |
|
||||
|
||||
---
|
||||
|
||||
## Carry-Forward Integration Updates
|
||||
|
||||
Updated:
|
||||
|
||||
- `docs/ver1/ver1-step1-2-carry-forward-checklist.md`
|
||||
|
||||
Step 3 updates recorded for:
|
||||
|
||||
- CF-A1: in progress with Step 3 inspection evidence
|
||||
- CF-A3: in progress with boundary-discipline evidence
|
||||
- CF-C1: done (Step 3 traceability artifacts integrated)
|
||||
- CF-C2: in progress (routing preserved for later steps)
|
||||
|
||||
---
|
||||
|
||||
## Residual Follow-ups
|
||||
|
||||
1. Step 4: migration rehearsal and rollback runbook execution for schema changes.
|
||||
2. Step 6/7: broader error-path inventory closure and logging field normalization.
|
||||
3. Step 9: release readiness reconfirmation for runtime ownership and migration behavior.
|
||||
|
||||
---
|
||||
|
||||
## Step 3 Exit Assessment
|
||||
|
||||
- Requirement-domain functional completion: **met**
|
||||
- Data integrity and state consistency for new flows: **met**
|
||||
- API/UI parity for new Step 3 features: **met**
|
||||
- Test and regression safety: **met**
|
||||
- Carry-forward integration obligations (Step 3-owned): **met/in progress as routed**
|
||||
|
||||
Step 3 is complete and ready to hand off to Step 4.
|
||||
@@ -0,0 +1,433 @@
|
||||
# Step 3 Implementation Plan: Functional Completion by Requirement Domain
|
||||
|
||||
## Purpose
|
||||
|
||||
Implement **Ver1 Step 3** from `docs/ver1/ver1.md` by completing all in-scope V1 functional requirements in a practical, user-first order while preserving:
|
||||
|
||||
- personal-scale operation
|
||||
- single-operator workflow
|
||||
- private-network deployment assumptions
|
||||
- low operational overhead
|
||||
- clean architecture boundaries
|
||||
|
||||
Primary governing docs:
|
||||
|
||||
- `docs/ver1/ver1.md` (Step 3 objective and sequencing)
|
||||
- `docs/architecture.md` (module boundaries, workflow, simplicity guardrails)
|
||||
- `docs/requirements.md` (REQ-0 through REQ-12 traceability)
|
||||
- `docs/error_handling.md` (error contract across boundaries)
|
||||
- `docs/intent.md` (verbatim transcription policy and prompt curation)
|
||||
- `docs/ver1/ver1-step1-2-carry-forward-checklist.md` (Step 1/2 carry-forward integration)
|
||||
- `docs/ver1/ver1-step2-error-path-inventory.md` (failure-path coverage visibility)
|
||||
|
||||
---
|
||||
|
||||
## MCP Resources Reviewed and Applied
|
||||
|
||||
All resources on `john-stream-mcp` were reviewed. Step 3 applies the following guidance directly:
|
||||
|
||||
1. `resource://skills/nicegui/document`
|
||||
- modular page registration
|
||||
- one-way dependency flow (`ui/api -> services -> infra`)
|
||||
- async-first UI responsiveness expectations
|
||||
|
||||
2. `resource://skills/nicegui-ui-customization/document`
|
||||
- reusable UI component extraction for repeated patterns
|
||||
- in-flight guards and explicit success/failure user feedback
|
||||
- event-driven updates over ad-hoc polling
|
||||
|
||||
3. `resource://skills/fastapi-async-sqlalchemy-modernization/document`
|
||||
- explicit transaction/session boundaries
|
||||
- deterministic resource ownership and cleanup continuity from Step 1
|
||||
- incremental migration strategy with rollback-aware checkpoints
|
||||
|
||||
4. `resource://skills/pydantic-settings/document`
|
||||
- typed configuration as single source of runtime truth
|
||||
- explicit source precedence and environment-safe defaults
|
||||
|
||||
5. `resource://skills/python-logging-dictconfig/document`
|
||||
- centralized startup-only logging configuration
|
||||
- named logger discipline and boundary-level structured fields
|
||||
|
||||
6. `resource://skills/pytesting/document`
|
||||
- deterministic test structure and marker discipline
|
||||
- behavior-first tests with clear fast-path and full-suite validation
|
||||
|
||||
7. `resource://skills/fastapi-uv-docker/document`
|
||||
- health endpoint and runtime startup/shutdown hygiene
|
||||
- compose/deployment readiness constraints relevant to functional completion
|
||||
|
||||
8. `resource://skills/python-typing/document`
|
||||
- modern typing updates where touched by Step 3 work
|
||||
|
||||
9. `resource://skills/ruff-linting-formating/document`
|
||||
- maintain lint/format consistency in all modified modules
|
||||
|
||||
10. `resource://prompts/greenfield-architecture/document`
|
||||
- explicit staged delivery with tradeoff-aware sequencing and test strategy
|
||||
|
||||
11. `resource://prompts/pytest-scaffold/document`
|
||||
12. `resource://prompts/pytest-fill-scaffold/document`
|
||||
- structure-first test planning, then deterministic implementation fill-in
|
||||
|
||||
Resources reviewed but not directly in Step 3 execution scope (no changes required now):
|
||||
|
||||
- `copilot-customization`, `mcp-details`, `vscode-configuration`, `zensical-docs`
|
||||
- prompts: `authoring`, `mcp-consumer-repo-shim`
|
||||
|
||||
---
|
||||
|
||||
## Step 3 Success Criteria
|
||||
|
||||
Step 3 is complete when:
|
||||
|
||||
1. All Step 3-targeted requirement slices are implemented and verified.
|
||||
2. Functional behavior is available through UI/API where required.
|
||||
3. Core data integrity and state transitions are deterministic.
|
||||
4. Error behavior follows `docs/error_handling.md` contracts.
|
||||
5. Carry-forward Step 1/2 items mapped to Step 3 are updated with evidence.
|
||||
|
||||
---
|
||||
|
||||
## Requirement-Slice Execution Model (Applied to Every Slice)
|
||||
|
||||
For each slice, execute this sequence:
|
||||
|
||||
1. Confirm contract/schema and boundary ownership.
|
||||
2. Implement service/domain logic.
|
||||
3. Implement persistence/state transitions.
|
||||
4. Integrate API and/or UI behavior.
|
||||
5. Add/update unit + integration + targeted end-to-end tests.
|
||||
6. Update docs and traceability artifacts.
|
||||
|
||||
Definition of done per slice:
|
||||
|
||||
- behavior is functional
|
||||
- tests pass in intended marker lanes
|
||||
- error pathways are classified and surfaced correctly
|
||||
- requirement traceability is updated with evidence
|
||||
|
||||
---
|
||||
|
||||
## Detailed Workstreams
|
||||
|
||||
## Workstream A — Functional Baseline Audit and Slice Backlog Lock
|
||||
|
||||
### Goals
|
||||
|
||||
- establish exact Step 3 functional delta from current implementation
|
||||
- lock a practical slice backlog before coding
|
||||
|
||||
### Tasks
|
||||
|
||||
1. Build Step 3 requirement matrix (REQ -> current status -> gap -> target slice).
|
||||
2. Map each gap to one of these domains:
|
||||
- Upload and lifecycle integrity
|
||||
- Review and revision history
|
||||
- Search over accepted transcripts
|
||||
- Export workflows
|
||||
- Prompt asset management behavior
|
||||
- API/UI parity and status visibility
|
||||
3. Align each slice with architecture boundary ownership and persistence strategy.
|
||||
4. Link open carry-forward items from checklist:
|
||||
- CF-A1, CF-A3 (architecture continuity in Step 3)
|
||||
- CF-C1, CF-C2 (traceability/execution continuity)
|
||||
|
||||
### Deliverables
|
||||
|
||||
- Step 3 requirement-slice matrix (appendix in this doc or separate artifact)
|
||||
- prioritized slice backlog with owner and validation method
|
||||
|
||||
### Exit Criteria
|
||||
|
||||
- every Step 3 slice maps to REQ IDs and a validation method
|
||||
- no ambiguous ownership remains for in-scope slices
|
||||
|
||||
---
|
||||
|
||||
## Workstream B — Core End-User Flows (Upload -> Transcribe -> Review)
|
||||
|
||||
### Related Requirements
|
||||
|
||||
- REQ-0, REQ-1, REQ-2, REQ-3, REQ-4, REQ-5, REQ-6, REQ-12
|
||||
|
||||
### Goals
|
||||
|
||||
- guarantee end-to-end reliability and usability of the primary user flow
|
||||
- ensure review experience supports transcript acceptance and correction
|
||||
|
||||
### Tasks
|
||||
|
||||
1. Validate and close any lifecycle-state gaps:
|
||||
- enforce valid transitions (`queued -> processing -> transcribed/failed/completed`)
|
||||
- ensure transition visibility in UI/API
|
||||
2. Review experience completion:
|
||||
- transcript detail display stability
|
||||
- failure detail readability and actionability
|
||||
- acceptance/edit path for human review
|
||||
3. Ensure prompt-asset integration remains file-based and auditable:
|
||||
- one prompt per Markdown file
|
||||
- prompt selection/usage traceability in job outcomes (if available in model)
|
||||
4. Confirm worker/UI interactions remain responsive under long-running jobs:
|
||||
- in-flight guards
|
||||
- clear status refresh behavior
|
||||
|
||||
### Deliverables
|
||||
|
||||
- complete end-user flow behavior with stable lifecycle visibility
|
||||
- test coverage for happy path and failure path
|
||||
|
||||
### Exit Criteria
|
||||
|
||||
- user can run upload -> process -> review reliably
|
||||
- failed and successful outcomes are both actionable and traceable
|
||||
|
||||
---
|
||||
|
||||
## Workstream C — Revision History and Provenance Completion
|
||||
|
||||
### Related Requirements
|
||||
|
||||
- REQ-3, REQ-4, REQ-5, REQ-11
|
||||
|
||||
### Goals
|
||||
|
||||
- finalize immutable transcript revision behavior and provenance consistency
|
||||
|
||||
### Tasks
|
||||
|
||||
1. Define/confirm revision invariants:
|
||||
- append-only revision history
|
||||
- clear current/accepted revision indicator
|
||||
2. Persist revision events consistently through service layer boundaries.
|
||||
3. Ensure UI/API expose revision timeline and selected revision details.
|
||||
4. Align error handling for revision conflicts and missing resources.
|
||||
|
||||
### Deliverables
|
||||
|
||||
- revision-history feature completeness
|
||||
- provenance and history read-path coverage
|
||||
|
||||
### Exit Criteria
|
||||
|
||||
- transcript edits produce deterministic revision records
|
||||
- previous revisions remain inspectable
|
||||
|
||||
---
|
||||
|
||||
## Workstream D — Search Completion (Accepted Transcript Scope)
|
||||
|
||||
### Related Requirements
|
||||
|
||||
- REQ-0, REQ-5, REQ-11
|
||||
|
||||
### Goals
|
||||
|
||||
- provide practical search over accepted transcripts for personal corpus usage
|
||||
|
||||
### Tasks
|
||||
|
||||
1. Finalize searchable scope and indexing rules (accepted/current text only).
|
||||
2. Implement service-backed search query behavior.
|
||||
3. Expose search in UI/API with clear result metadata (document/job/revision context).
|
||||
4. Add guardrails for empty/no-result/error scenarios with actionable messaging.
|
||||
|
||||
### Deliverables
|
||||
|
||||
- functional search pathway with deterministic results for accepted text
|
||||
|
||||
### Exit Criteria
|
||||
|
||||
- operator can find transcripts reliably by text queries
|
||||
- no-result and error states are clear and non-silent
|
||||
|
||||
---
|
||||
|
||||
## Workstream E — Export Completion
|
||||
|
||||
### Related Requirements
|
||||
|
||||
- REQ-0, REQ-4, REQ-5, REQ-11
|
||||
|
||||
### Goals
|
||||
|
||||
- deliver practical export of transcript data for personal archive use
|
||||
|
||||
### Tasks
|
||||
|
||||
1. Finalize export contract (format, included fields, scope filters).
|
||||
2. Implement export service with deterministic data mapping.
|
||||
3. Add UI/API trigger path and user-visible completion/failure feedback.
|
||||
4. Validate export integrity against persisted source-of-record entities.
|
||||
|
||||
### Deliverables
|
||||
|
||||
- end-to-end export capability with operator-visible outcomes
|
||||
|
||||
### Exit Criteria
|
||||
|
||||
- export output is complete, consistent, and usable for downstream personal archive workflows
|
||||
|
||||
---
|
||||
|
||||
## Workstream F — API/UI Parity and Interaction Hardening
|
||||
|
||||
### Related Requirements
|
||||
|
||||
- REQ-5 plus cross-cutting REQ-2/3/4
|
||||
|
||||
### Goals
|
||||
|
||||
- ensure UI and API expose coherent feature behavior and error contracts
|
||||
|
||||
### Tasks
|
||||
|
||||
1. Verify API/UI parity matrix for each Step 3 slice.
|
||||
2. Standardize interaction behavior:
|
||||
- loading and in-flight states
|
||||
- success/failure notifications
|
||||
- stable error_id visibility where user-facing
|
||||
3. Ensure route/page modules remain composition-focused (business logic in services).
|
||||
|
||||
### Deliverables
|
||||
|
||||
- API/UI parity checklist with resolved gaps
|
||||
|
||||
### Exit Criteria
|
||||
|
||||
- no major flow exists in one interface with conflicting semantics in the other
|
||||
|
||||
---
|
||||
|
||||
## Workstream G — Carry-Forward Integration During Step 3
|
||||
|
||||
### Goals
|
||||
|
||||
- close Step 1/2 follow-ups that are Step 3-owned
|
||||
|
||||
### Tasks
|
||||
|
||||
1. Update checklist item CF-A1 as Step 3 slices touch runtime resources.
|
||||
2. Update checklist item CF-A3 with lightweight boundary enforcement evidence.
|
||||
3. Update CF-C1/CF-C2 traceability mapping with Step 3 outcomes.
|
||||
|
||||
### Deliverables
|
||||
|
||||
- updated `docs/ver1/ver1-step1-2-carry-forward-checklist.md` evidence entries
|
||||
|
||||
### Exit Criteria
|
||||
|
||||
- Step 3-owned carry-forward items are either completed or explicitly routed with evidence
|
||||
|
||||
---
|
||||
|
||||
## Test and Validation Plan
|
||||
|
||||
Apply `pytesting` guidance with deterministic, behavior-focused coverage.
|
||||
|
||||
### Validation Lanes
|
||||
|
||||
1. Structure/collection:
|
||||
- `uv run pytest --collect-only -q`
|
||||
2. Fast feedback lane:
|
||||
- `uv run pytest -m unit -q`
|
||||
3. Main verification lane:
|
||||
- `uv run pytest -m "not external" -q`
|
||||
4. Full suite:
|
||||
- `uv run pytest -q`
|
||||
|
||||
### Required Coverage Areas
|
||||
|
||||
- lifecycle transition invariants
|
||||
- revision history invariants
|
||||
- search query behavior and result mapping
|
||||
- export integrity and failure handling
|
||||
- UI interaction guards and actionable failure feedback
|
||||
- API envelope and status consistency for new/changed flows
|
||||
|
||||
### Test Design Rules
|
||||
|
||||
- one behavior target per test
|
||||
- minimize heavy mocking; prefer real-path behavior checks where practical
|
||||
- keep markers explicit and strict
|
||||
|
||||
---
|
||||
|
||||
## Logging, Error, and Config Guardrails for Step 3 Changes
|
||||
|
||||
1. Logging
|
||||
- keep centralized startup logging config (`dictConfig`) as canonical
|
||||
- include required error fields at boundary failures (`error_id`, `category`, `operation`, identifiers where available)
|
||||
|
||||
2. Error handling
|
||||
- preserve taxonomy stability from `docs/error_handling.md`
|
||||
- map any new failure pathways into existing categories
|
||||
- surface actionable suggestions in UI/API
|
||||
|
||||
3. Configuration
|
||||
- use typed settings and avoid ad-hoc env reads in business modules
|
||||
- keep environment behavior explicit and documented
|
||||
|
||||
---
|
||||
|
||||
## Implementation Order (Detailed)
|
||||
|
||||
1. Workstream A: audit and backlog lock
|
||||
2. Workstream B: core flow completion
|
||||
3. Workstream C: revision/provenance completion
|
||||
4. Workstream D: search completion
|
||||
5. Workstream E: export completion
|
||||
6. Workstream F: API/UI parity hardening
|
||||
7. Workstream G: carry-forward integration updates
|
||||
8. Full validation pass + docs/traceability updates
|
||||
|
||||
---
|
||||
|
||||
## Deliverables
|
||||
|
||||
1. Step 3 requirement-slice matrix with REQ mapping and evidence links
|
||||
2. implemented Step 3 functional slices across service/persistence/API/UI
|
||||
3. updated tests and passing validation lanes
|
||||
4. updated carry-forward checklist entries (`CF-A1`, `CF-A3`, `CF-C1`, `CF-C2` as applicable)
|
||||
5. Step 3 results document (`docs/ver1/ver1-step3-results.md`)
|
||||
|
||||
---
|
||||
|
||||
## Risks and Mitigations
|
||||
|
||||
1. **Risk:** Scope creep from optional enhancements during feature completion
|
||||
- **Mitigation:** enforce REQ-mapped slice backlog and defer non-REQ enhancements
|
||||
|
||||
2. **Risk:** Functional parity drift between UI and API
|
||||
- **Mitigation:** maintain parity matrix and verify both surfaces per slice
|
||||
|
||||
3. **Risk:** Data-model changes introduce migration surprises
|
||||
- **Mitigation:** coordinate with Step 4 runbook expectations early and test on representative data
|
||||
|
||||
4. **Risk:** Reliability regressions while adding functionality
|
||||
- **Mitigation:** run full error-path regression checks and keep Step 2 contracts intact
|
||||
|
||||
---
|
||||
|
||||
## Step 3 Completion Checklist
|
||||
|
||||
- [ ] Step 3 requirement-slice matrix completed and linked to REQ IDs.
|
||||
- [ ] Core end-user flow is functionally complete and verified.
|
||||
- [ ] Revision history/provenance behavior is complete and test-covered.
|
||||
- [ ] Search over accepted transcripts is complete and test-covered.
|
||||
- [ ] Export flow is complete and test-covered.
|
||||
- [ ] API/UI parity checklist has no unresolved high-impact gaps.
|
||||
- [ ] Step 3-owned carry-forward items are updated with evidence.
|
||||
- [ ] Validation lanes pass (`collect-only`, unit, non-external, full).
|
||||
- [ ] `docs/ver1/ver1-step3-results.md` is created with evidence and residual follow-ups.
|
||||
|
||||
---
|
||||
|
||||
## Handoff to Step 4
|
||||
|
||||
Step 3 completion enables Step 4 (Data Model and Migration Safety) with:
|
||||
|
||||
- finalized functional domain behavior
|
||||
- stable persistence expectations
|
||||
- traceable requirement evidence
|
||||
- clarified migration-impact surface
|
||||
@@ -0,0 +1,113 @@
|
||||
# Ver1 Step 4 Migration and Rollback Runbook
|
||||
|
||||
## Purpose
|
||||
|
||||
Provide a concise, operator-safe procedure for schema migration execution,
|
||||
compatibility validation, and rollback/mitigation for personal-scale deployments.
|
||||
|
||||
This runbook supports `docs/ver1/ver1-step4.md` and REQ-10 by keeping normal
|
||||
production startup non-mutating unless explicitly configured otherwise.
|
||||
|
||||
---
|
||||
|
||||
## Preconditions
|
||||
|
||||
1. Application version to deploy is known and checked out.
|
||||
2. `.env` values are configured for target environment.
|
||||
3. Database backup path is prepared.
|
||||
4. Application process is stopped before migration on production-like systems.
|
||||
|
||||
---
|
||||
|
||||
## Commands
|
||||
|
||||
Use explicit migration runner operations:
|
||||
|
||||
1. List pending migrations:
|
||||
- `uv run python -m transcription.migration_runner --list`
|
||||
2. Apply pending migrations:
|
||||
- `uv run python -m transcription.migration_runner --apply`
|
||||
3. Validate schema compatibility:
|
||||
- `uv run python -m transcription.migration_runner --check`
|
||||
|
||||
Recommended execution order:
|
||||
|
||||
1. `--list`
|
||||
2. backup database
|
||||
3. `--apply`
|
||||
4. `--check`
|
||||
5. start application
|
||||
|
||||
---
|
||||
|
||||
## Backup Procedure (SQLite Baseline)
|
||||
|
||||
For SQLite deployments, copy the DB file before migration:
|
||||
|
||||
- Example DB path default: `./transcription.db`
|
||||
- Keep timestamped backup copy in a safe location.
|
||||
|
||||
If the file is in active use, stop the app first.
|
||||
|
||||
---
|
||||
|
||||
## Verification Checklist
|
||||
|
||||
After migration apply:
|
||||
|
||||
1. `--check` exits successfully.
|
||||
2. `schema_migration_history` includes applied revisions.
|
||||
3. Application starts successfully.
|
||||
4. Health endpoint responds: `/healthz`.
|
||||
5. Critical flows smoke-check:
|
||||
- upload
|
||||
- job processing
|
||||
- revision listing/acceptance
|
||||
|
||||
---
|
||||
|
||||
## Rollback and Mitigation Decision Tree
|
||||
|
||||
1. If migration fails before changes commit:
|
||||
- fix issue
|
||||
- re-run apply
|
||||
2. If migration partially applied or compatibility check fails:
|
||||
- stop app
|
||||
- restore from backup
|
||||
- investigate and produce forward-fix migration if needed
|
||||
3. If app starts but functional invariants fail:
|
||||
- stop app
|
||||
- restore backup
|
||||
- add corrective migration/backfill and rehearse before retry
|
||||
|
||||
For this Step 4 baseline, backup restore is the primary rollback mechanism.
|
||||
|
||||
---
|
||||
|
||||
## Failure Classification Guidance
|
||||
|
||||
Classify migration failures using `docs/error_handling.md` categories:
|
||||
|
||||
- transient connection issues -> `infrastructure_transient_error`
|
||||
- permissions/misconfiguration -> `infrastructure_persistent_error`
|
||||
- unexpected migration logic defects -> `internal_unexpected_error`
|
||||
|
||||
Record failure details with operation context and timestamp.
|
||||
|
||||
---
|
||||
|
||||
## Operational Notes
|
||||
|
||||
- `migration_auto_apply_on_startup` defaults to `False`.
|
||||
- `validate_schema_on_startup` defaults to `True`.
|
||||
- Startup schema validation fails fast on incompatibility.
|
||||
|
||||
This protects production from accidental schema drift.
|
||||
|
||||
---
|
||||
|
||||
## Post-Step-4 Follow-Up
|
||||
|
||||
If migration complexity grows beyond lightweight revision scripts,
|
||||
introduce a dedicated migration framework in a future step while preserving
|
||||
this runbook structure and operator-first workflow.
|
||||
@@ -0,0 +1,153 @@
|
||||
# Ver1 Step 4 Results: Data Model and Migration Safety
|
||||
|
||||
## Summary
|
||||
|
||||
Step 4 implementation status: **complete (baseline scope)**.
|
||||
|
||||
This document records completed migration-safety work, validation evidence, and remaining follow-ups for Ver1 Step 4.
|
||||
|
||||
Implemented in this step:
|
||||
|
||||
1. Added explicit migration framework module with revision history tracking.
|
||||
2. Added schema compatibility validation and startup guardrails.
|
||||
3. Added migration runner CLI for list/apply/check operations.
|
||||
4. Added migration tests and Step 4 validation evidence.
|
||||
5. Added Step 4 migration/rollback runbook.
|
||||
---
|
||||
|
||||
## Implemented Changes
|
||||
|
||||
### 1) Schema audit and invariant lock
|
||||
|
||||
Implemented read-only compatibility checks in `src/transcription/db.py`:
|
||||
|
||||
- `validate_schema_compatibility(...)` verifies required V1 tables:
|
||||
- `document`
|
||||
- `job`
|
||||
- `transcript`
|
||||
- `transcriptrevision`
|
||||
- verifies required `job.retry_count` column
|
||||
- returns explicit issue identifiers (non-mutating check)
|
||||
|
||||
### 2) Migration policy/tooling lock
|
||||
|
||||
Added explicit migration revision model in `src/transcription/migrations.py`:
|
||||
|
||||
- `MigrationRevision` dataclass
|
||||
- ordered `MIGRATIONS` registry
|
||||
- migration history table: `schema_migration_history`
|
||||
- explicit pending-list and apply operations
|
||||
|
||||
### 3) Forward migration implementation
|
||||
|
||||
Implemented two baseline forward migrations:
|
||||
|
||||
1. `0001_add_retry_count_to_job`
|
||||
2. `0002_create_transcriptrevision_table`
|
||||
|
||||
Each migration is idempotent and recorded in migration history.
|
||||
|
||||
### 4) Rollback and mitigation runbook
|
||||
|
||||
Created `docs/ver1/ver1-step4-migration-runbook.md` with:
|
||||
|
||||
- preconditions
|
||||
- list/apply/check command sequence
|
||||
- backup-first procedure
|
||||
- verification checklist
|
||||
- rollback/mitigation decision tree
|
||||
- error classification guidance aligned to `docs/error_handling.md`
|
||||
|
||||
### 5) Backfill implementation or explicit no-backfill decision
|
||||
|
||||
No backfill required for this baseline Step 4 scope.
|
||||
|
||||
Rationale:
|
||||
|
||||
- additive migration operations only
|
||||
- default values and new-table creation do not require historical row rewrites for current V1 invariants
|
||||
- residual advanced backfill scenarios deferred unless future schema evolution introduces incompatible transforms
|
||||
---
|
||||
|
||||
## Test and Verification Evidence
|
||||
|
||||
### Added/Updated Tests
|
||||
|
||||
1. `tests/test_migrations.py`
|
||||
- pending migration discovery
|
||||
- migration apply + history recording
|
||||
- idempotent re-apply behavior
|
||||
2. `tests/test_db.py`
|
||||
- compatibility-check behavior on fresh schema
|
||||
- table expectation updates for `transcriptrevision`
|
||||
3. `tests/test_config.py`
|
||||
- migration safety setting defaults
|
||||
4. `tests/test_app.py`
|
||||
- lifespan test compatibility with migration/validation startup hooks
|
||||
### Validation Runs
|
||||
|
||||
Run and record outcomes:
|
||||
|
||||
- `uv run pytest --collect-only -q` -> passed
|
||||
- `uv run pytest -m unit -q` -> passed
|
||||
- `uv run pytest -m "not external" -q` -> passed
|
||||
- `uv run pytest -q` -> passed
|
||||
### Migration Rehearsal Evidence
|
||||
|
||||
Migration rehearsal details (test-based):
|
||||
|
||||
- baseline data set used: in-memory SQLite legacy-shaped schema fixture (`job` table missing Step 4 additions)
|
||||
- forward migration result: pending revisions applied successfully (`0001`, `0002`)
|
||||
- post-migration verification result: schema checks pass and migration history recorded
|
||||
- rollback/mitigation rehearsal result: runbook defined backup-restore primary rollback class for personal-scale SQLite deployment
|
||||
---
|
||||
|
||||
## Requirement Traceability (Step 4)
|
||||
|
||||
| Step 4 Area | REQ Coverage | Status | Evidence |
|
||||
| --- | --- | --- | --- |
|
||||
| Schema lifecycle and state persistence safety | REQ-3, REQ-4, REQ-11 | met | `src/transcription/migrations.py`, `tests/test_migrations.py`, `tests/test_db.py` |
|
||||
| Lifespan/runtime ownership continuity | REQ-7 | met | `src/transcription/app.py` startup checks + existing lifespan ownership model |
|
||||
| Explicit non-mutating production startup policy | REQ-10 | met | `migration_auto_apply_on_startup=False` default + explicit runner workflow + startup validation gate |
|
||||
| Prompt/data continuity constraints | REQ-12 | met (continued) | no prompt-contract mutation in Step 4 changes |
|
||||
---
|
||||
|
||||
## Operational Artifacts Produced
|
||||
|
||||
- `docs/ver1/ver1-step4.md`
|
||||
- `docs/ver1/ver1-step4-migration-runbook.md`
|
||||
- `src/transcription/migrations.py`
|
||||
- `src/transcription/migration_runner.py`
|
||||
- README migration workflow updates
|
||||
---
|
||||
|
||||
## Risks, Exceptions, and Follow-Ups
|
||||
|
||||
1. This lightweight migration system is appropriate for current personal-scale scope but may require a dedicated framework as schema complexity grows.
|
||||
2. Rollback remains backup-restore primary; reversible down-migration coverage is intentionally limited in this baseline.
|
||||
3. Startup compatibility checks currently fail fast with generic runtime error text and can be further normalized under API/operator error envelopes in later hardening.
|
||||
|
||||
Open follow-ups to carry forward:
|
||||
|
||||
- Evaluate migration framework escalation criteria in Step 9/10 readiness updates.
|
||||
- Add optional richer structured migration logging fields if observability scope expands.
|
||||
---
|
||||
|
||||
## Step 4 Exit Assessment
|
||||
|
||||
- Schema validation against finalized V1 domain: **met**
|
||||
- Forward migration path safety and repeatability: **met (baseline scope)**
|
||||
- Rollback/mitigation readiness: **met (backup-restore primary path)**
|
||||
- Backfill risk closure: **met (no backfill required for current deltas)**
|
||||
- Test and regression safety: **met**
|
||||
|
||||
Step 4 completion status: **complete (baseline scope)**
|
||||
---
|
||||
|
||||
## Handoff to Step 5
|
||||
|
||||
Once Step 4 is marked complete, Step 5 can proceed with:
|
||||
|
||||
- verified migration safety baseline
|
||||
- explicit rollback and recovery procedures
|
||||
- reduced data-integrity risk entering private-network safety hardening
|
||||
@@ -0,0 +1,378 @@
|
||||
# Step 4 Implementation Plan: Data Model and Migration Safety
|
||||
|
||||
## Purpose
|
||||
|
||||
Implement **Ver1 Step 4** from `docs/ver1/ver1.md` by making data-model evolution safe, explicit, and repeatable for personal-scale deployment.
|
||||
|
||||
Step 4 ensures schema changes are handled through deterministic migration workflows rather than implicit startup mutation, while preserving:
|
||||
|
||||
- personal-scale operational simplicity
|
||||
- single-operator deployment model
|
||||
- lifecycle-owned runtime resource boundaries
|
||||
- stable requirement traceability and low rollback risk
|
||||
|
||||
Primary governing docs:
|
||||
|
||||
- `docs/ver1/ver1.md` (Step 4 objective and sequencing)
|
||||
- `docs/architecture.md` (runtime ownership, persistence boundaries, simplicity guardrails)
|
||||
- `docs/requirements.md` (REQ-3, REQ-4, REQ-7, REQ-10, REQ-11, REQ-12 emphasis)
|
||||
- `docs/error_handling.md` (failure classification and safe error surfacing)
|
||||
- `docs/intent.md` (verbatim/transcription/revision domain behavior)
|
||||
|
||||
---
|
||||
|
||||
## MCP Resources Reviewed and Applied
|
||||
|
||||
All currently available resources on `john-stream-mcp` were reviewed. Step 4 applies the following guidance directly:
|
||||
|
||||
1. `resource://skills/fastapi-async-sqlalchemy-modernization/document`
|
||||
- explicit engine/session lifecycle ownership
|
||||
- transaction boundary clarity for schema transitions and backfills
|
||||
- phased rollout with rollback-aware checkpoints
|
||||
|
||||
2. `resource://skills/pydantic-settings/document`
|
||||
- typed migration/runtime safety settings
|
||||
- explicit source-precedence behavior for operational toggles
|
||||
- fail-fast config semantics for unsafe startup paths
|
||||
|
||||
3. `resource://skills/pytesting/document`
|
||||
- deterministic migration verification lanes
|
||||
- strict marker discipline
|
||||
- behavior-first test coverage for migration outcomes
|
||||
|
||||
4. `resource://skills/python-logging-dictconfig/document`
|
||||
- startup-centralized logging configuration
|
||||
- structured migration and rollback event traceability
|
||||
|
||||
5. `resource://skills/fastapi-uv-docker/document`
|
||||
- deployment and rehearsal discipline
|
||||
- startup/health posture validation during migration windows
|
||||
|
||||
6. `resource://skills/python-typing/document`
|
||||
- modern typing hygiene for touched migration/persistence modules
|
||||
|
||||
7. `resource://skills/ruff-linting-formating/document`
|
||||
- lint/format consistency for migration scripts and database modules
|
||||
|
||||
Planning methodology inputs also applied:
|
||||
|
||||
8. `resource://prompts/greenfield-architecture/document`
|
||||
- staged execution with explicit risk and extension handling
|
||||
|
||||
9. `resource://prompts/pytest-scaffold/document`
|
||||
10. `resource://prompts/pytest-fill-scaffold/document`
|
||||
- test-structure-first and deterministic fill-in sequencing
|
||||
|
||||
Reviewed but not directly Step 4 execution-critical:
|
||||
|
||||
- skills: `copilot-customization`, `mcp-details`, `nicegui`, `nicegui-ui-customization`, `vscode-configuration`, `zensical-docs`
|
||||
- prompts: `authoring`, `mcp-consumer-repo-shim`
|
||||
|
||||
---
|
||||
|
||||
## Current-State Gap Summary (Step 4 Scope)
|
||||
|
||||
Based on Step 1–3 outcomes and current docs/tests:
|
||||
|
||||
1. **Bootstrap policy baseline is present**
|
||||
- Environment-aware schema bootstrap policy exists and aligns with REQ-10 intent.
|
||||
2. **Functional model expanded in Step 3**
|
||||
- Revision/acceptance features introduce schema evolution requirements that need formal migration safety rehearsal.
|
||||
3. **Runbook maturity required**
|
||||
- Step 4 requires explicit migration + rollback procedures and evidence.
|
||||
4. **Backfill risk must be evaluated**
|
||||
- New/changed fields and semantics must be checked for historical data reconciliation needs.
|
||||
5. **Release-path integration needed**
|
||||
- Step 4 artifacts must feed Step 9 release readiness and Step 10 docs completion.
|
||||
|
||||
---
|
||||
|
||||
## Scope for Step 4
|
||||
|
||||
### In scope
|
||||
|
||||
1. Validate final V1 schema against implemented domain behavior (post-Step 3 reality).
|
||||
2. Define and implement forward-safe migration path for expected upgrades.
|
||||
3. Define and document rollback/mitigation strategy for migration failures.
|
||||
4. Implement backfill scripts only if required, with idempotent behavior.
|
||||
5. Rehearse migration + rollback locally using representative sample data.
|
||||
6. Add Step 4-specific verification tests and operational checks.
|
||||
7. Produce operator-facing migration/rollback runbook and Step 4 results evidence.
|
||||
|
||||
### Out of scope
|
||||
|
||||
- Distributed/externally orchestrated migration systems
|
||||
- Major persistence-architecture rewrites beyond V1 scope
|
||||
- Non-V1 enhancement migrations unrelated to implemented requirement slices
|
||||
|
||||
---
|
||||
|
||||
## Target Decisions for Step 4
|
||||
|
||||
1. **Production startup remains non-mutating by default**
|
||||
- Preserve REQ-10 posture and avoid implicit schema mutation at normal startup.
|
||||
|
||||
2. **Schema changes are explicit operator workflows**
|
||||
- Migrations run as deliberate operational actions, not hidden side effects.
|
||||
|
||||
3. **Migration safety beats migration speed**
|
||||
- Additive and reversible-first patterns are preferred where possible.
|
||||
|
||||
4. **Rollback policy is explicit per change**
|
||||
- Each migration must declare rollback class:
|
||||
- direct rollback supported
|
||||
- forward-fix required
|
||||
- backup restore required
|
||||
|
||||
5. **Backfills are optional and minimal**
|
||||
- Introduce only when required by correctness/invariants, never by convenience.
|
||||
|
||||
6. **Migration observability is mandatory**
|
||||
- Structured logs include operation, migration identifier, status, and failure classification.
|
||||
|
||||
---
|
||||
|
||||
## Detailed Work Breakdown
|
||||
|
||||
## Phase A — Schema and Domain Invariant Audit
|
||||
|
||||
- [ ] **A1. Build canonical V1 schema inventory**
|
||||
- Enumerate all persisted entities and key fields:
|
||||
- document records
|
||||
- jobs and statuses
|
||||
- transcripts
|
||||
- transcript revisions
|
||||
- failure/provenance fields
|
||||
- [ ] **A2. Validate invariants against implemented behavior**
|
||||
- Cross-check Step 3 functionality and current domain expectations:
|
||||
- append-only revision history
|
||||
- accepted revision semantics
|
||||
- canonical transcript synchronization behavior
|
||||
- [ ] **A3. Classify required schema deltas**
|
||||
- Categorize deltas:
|
||||
- additive and safe
|
||||
- compatibility-sensitive
|
||||
- potentially destructive (must be staged or deferred)
|
||||
|
||||
### Deliverables
|
||||
|
||||
- `docs/ver1/ver1-step4-schema-audit.md` (recommended)
|
||||
- schema-delta matrix with risk class and owning module
|
||||
|
||||
### Exit Criteria
|
||||
|
||||
- all required schema changes have explicit rationale and risk classification
|
||||
- no ambiguous domain invariant remains
|
||||
|
||||
---
|
||||
|
||||
## Phase B — Migration Policy and Tooling Lock
|
||||
|
||||
- [ ] **B1. Lock migration workflow policy**
|
||||
- Define canonical migration execution path and artifact conventions.
|
||||
- [ ] **B2. Define migration authoring checklist**
|
||||
- Include:
|
||||
- preconditions
|
||||
- forward steps
|
||||
- rollback class
|
||||
- post-verification checks
|
||||
- [ ] **B3. Align policy with runtime startup safeguards**
|
||||
- Ensure production startup remains explicit/non-mutating by default.
|
||||
- [ ] **B4. Define operator invocation standard**
|
||||
- One documented command path for local and production-like workflows.
|
||||
|
||||
### Deliverables
|
||||
|
||||
- migration policy section (this doc + runbook)
|
||||
- migration authoring/review checklist
|
||||
|
||||
### Exit Criteria
|
||||
|
||||
- one unambiguous migration process exists and is documented
|
||||
- startup policy and migration policy are consistent and non-conflicting
|
||||
|
||||
---
|
||||
|
||||
## Phase C — Forward Migration Implementation
|
||||
|
||||
- [ ] **C1. Implement required migration set**
|
||||
- Build migration artifacts for all approved Step 4 deltas.
|
||||
- [ ] **C2. Preserve compatibility where needed**
|
||||
- Use staged expand/contract strategy when direct cutover is unsafe.
|
||||
- [ ] **C3. Add migration logging checkpoints**
|
||||
- Log start, phase boundaries, completion, and failure details.
|
||||
- [ ] **C4. Verify post-migration schema state**
|
||||
- Confirm expected tables/columns/constraints/indexes are present.
|
||||
|
||||
### Deliverables
|
||||
|
||||
- migration artifacts/scripts for V1 target schema
|
||||
- schema verification checklist outputs
|
||||
|
||||
### Exit Criteria
|
||||
|
||||
- baseline-to-target forward migration executes successfully
|
||||
- post-migration checks pass deterministically
|
||||
|
||||
---
|
||||
|
||||
## Phase D — Rollback and Mitigation Strategy
|
||||
|
||||
- [ ] **D1. Define rollback classes per migration**
|
||||
- direct downgrade vs forward-fix vs backup-restore.
|
||||
- [ ] **D2. Create rollback decision tree**
|
||||
- trigger conditions, safe stop points, and recovery path.
|
||||
- [ ] **D3. Align failure classification with `error_handling.md`**
|
||||
- normalize migration failures into canonical categories:
|
||||
- `infrastructure_transient_error`
|
||||
- `infrastructure_persistent_error`
|
||||
- `internal_unexpected_error` (as needed)
|
||||
- [ ] **D4. Rehearse rollback flow**
|
||||
- run at least one migration failure simulation and execute chosen recovery path.
|
||||
|
||||
### Deliverables
|
||||
|
||||
- rollback/mitigation decision tree
|
||||
- rehearsal evidence notes
|
||||
|
||||
### Exit Criteria
|
||||
|
||||
- operator can execute rollback/mitigation without undocumented steps
|
||||
- migration failure paths are diagnosable and classified
|
||||
|
||||
---
|
||||
|
||||
## Phase E — Backfill Decision and Execution (Conditional)
|
||||
|
||||
- [ ] **E1. Determine backfill necessity**
|
||||
- inspect whether existing records violate new invariants.
|
||||
- [ ] **E2. If required, implement idempotent backfill**
|
||||
- resumable, batch-safe, and deterministic update semantics.
|
||||
- [ ] **E3. Add post-backfill verification**
|
||||
- validate:
|
||||
- revision sequencing integrity
|
||||
- accepted/current transcript consistency
|
||||
- job lifecycle consistency
|
||||
- [ ] **E4. If not required, record explicit “no backfill needed” evidence**
|
||||
|
||||
### Deliverables
|
||||
|
||||
- backfill script(s) and checklist (if applicable)
|
||||
- no-backfill rationale artifact (if not applicable)
|
||||
|
||||
### Exit Criteria
|
||||
|
||||
- required backfills completed and verified OR formally ruled out with evidence
|
||||
|
||||
---
|
||||
|
||||
## Phase F — Verification and Test Expansion
|
||||
|
||||
Apply `pytesting` guidance (deterministic, behavior-first, strict markers).
|
||||
|
||||
- [ ] **F1. Migration application tests**
|
||||
- verify forward migration from representative baseline.
|
||||
- [ ] **F2. Post-migration schema contract tests**
|
||||
- verify expected schema shape and key constraints.
|
||||
- [ ] **F3. Rollback/mitigation tests**
|
||||
- verify chosen rollback class behavior where practical.
|
||||
- [ ] **F4. Startup policy regression tests**
|
||||
- confirm production-mode startup does not mutate schema implicitly.
|
||||
- [ ] **F5. Backfill behavior tests (if applicable)**
|
||||
- idempotency and invariants after repeated execution.
|
||||
|
||||
### Validation Commands
|
||||
|
||||
- `uv run pytest --collect-only -q`
|
||||
- `uv run pytest -m unit -q`
|
||||
- `uv run pytest -m "not external" -q`
|
||||
- `uv run pytest -q`
|
||||
|
||||
### Exit Criteria
|
||||
|
||||
- all Step 4 migration-safety checks pass
|
||||
- no REQ-10 regression introduced
|
||||
|
||||
---
|
||||
|
||||
## Phase G — Runbook and Documentation Closure
|
||||
|
||||
- [ ] **G1. Create migration and rollback runbook**
|
||||
- include:
|
||||
- prerequisites
|
||||
- backup step
|
||||
- migration execution
|
||||
- verification
|
||||
- rollback/mitigation
|
||||
- [ ] **G2. Update traceability artifacts**
|
||||
- map Step 4 outcomes to REQ IDs and evidence.
|
||||
- [ ] **G3. Prepare Step 4 handoff artifacts**
|
||||
- ensure outputs feed Step 9 release readiness and Step 10 docs completion.
|
||||
|
||||
### Deliverables
|
||||
|
||||
- `docs/ver1/ver1-step4-migration-runbook.md` (recommended)
|
||||
- `docs/ver1/ver1-step4-results.md`
|
||||
- updated traceability references where needed
|
||||
|
||||
### Exit Criteria
|
||||
|
||||
- migration operations are executable using docs alone
|
||||
- Step 4 evidence is complete and auditable
|
||||
|
||||
---
|
||||
|
||||
## Recommended Implementation Order
|
||||
|
||||
1. Phase A — schema/invariant audit
|
||||
2. Phase B — migration policy and tooling lock
|
||||
3. Phase C — forward migration implementation
|
||||
4. Phase D — rollback/mitigation strategy + rehearsal
|
||||
5. Phase E — backfill decision and execution (conditional)
|
||||
6. Phase F — test and verification expansion
|
||||
7. Phase G — runbook + traceability closure
|
||||
|
||||
This sequence minimizes risk by locking policy and scope before irreversible data changes.
|
||||
|
||||
---
|
||||
|
||||
## Risks and Mitigations
|
||||
|
||||
1. **Risk:** Data loss from unsafe schema transitions
|
||||
- **Mitigation:** backup-first gate, staged migration strategies, post-check verification.
|
||||
|
||||
2. **Risk:** Startup policy drift reintroduces implicit schema mutation
|
||||
- **Mitigation:** explicit regression tests for production startup behavior (REQ-10 guard).
|
||||
|
||||
3. **Risk:** Rollback path is incomplete or untested
|
||||
- **Mitigation:** mandatory rollback class declaration + rehearsal evidence.
|
||||
|
||||
4. **Risk:** Backfill scripts cause partial/inconsistent state
|
||||
- **Mitigation:** idempotent design, batching, and invariant-focused verification.
|
||||
|
||||
5. **Risk:** Migration failure diagnostics are unclear
|
||||
- **Mitigation:** structured logging + error category mapping per `error_handling.md`.
|
||||
|
||||
---
|
||||
|
||||
## Step 4 Completion Checklist
|
||||
|
||||
- [ ] V1 schema audit completed and approved.
|
||||
- [ ] Migration workflow policy is locked and documented.
|
||||
- [ ] Required forward migrations are implemented and validated.
|
||||
- [ ] Rollback/mitigation decision tree is documented and rehearsed.
|
||||
- [ ] Backfill required/not-required decision is evidenced.
|
||||
- [ ] Migration-safety test coverage is added and passing.
|
||||
- [ ] Startup non-mutation policy remains verified in production mode.
|
||||
- [ ] Step 4 runbook and results artifacts are completed.
|
||||
|
||||
---
|
||||
|
||||
## Handoff to Step 5
|
||||
|
||||
Step 4 completion enables Step 5 (Private-Network Safety Baseline) with:
|
||||
|
||||
- stable, explicit schema evolution mechanics
|
||||
- reduced upgrade risk for single-operator deployments
|
||||
- migration/rollback procedures suitable for personal-scale production
|
||||
- traceable evidence for release-readiness gates
|
||||
@@ -0,0 +1,178 @@
|
||||
# Ver1 Step 5 Results: Private-Network Safety Baseline
|
||||
|
||||
## Summary
|
||||
|
||||
Step 5 implementation status: **complete**.
|
||||
|
||||
This document records completed private-network safety controls, validation evidence, and residual risks for Ver1 Step 5.
|
||||
|
||||
Implemented in this step:
|
||||
|
||||
1. Added private-network security assumptions and control matrix (`docs/ver1/ver1-step5-security-assumptions.md`).
|
||||
2. Implemented optional single-operator access control for `/ui*` and `/api*` via HTTP Basic auth.
|
||||
3. Added upload-size guardrails (`MAX_UPLOAD_BYTES`) and config fail-fast validation for operator credential requirements.
|
||||
4. Hardened unexpected-error user-facing messaging to reduce sensitive detail leakage.
|
||||
5. Added Step 5 tests for access control, security settings, and upload size boundaries.
|
||||
6. Executed dependency/security scans (`pip-audit`, `bandit`) with no critical/high findings.
|
||||
|
||||
---
|
||||
|
||||
## Implemented Changes
|
||||
|
||||
### 1) Security assumptions and threat model
|
||||
|
||||
Completed.
|
||||
|
||||
- Added `docs/ver1/ver1-step5-security-assumptions.md` defining:
|
||||
- trusted private-network deployment assumptions
|
||||
- single-operator usage model
|
||||
- explicit out-of-scope classes (enterprise IAM, internet-facing zero-trust, multi-tenant controls)
|
||||
- Added Step 5 control/ownership matrix and residual-risk notes.
|
||||
|
||||
### 2) Single-operator access control baseline
|
||||
|
||||
Completed.
|
||||
|
||||
- New module: `src/transcription/security.py`
|
||||
- `is_protected_path(...)` protects `/ui*` and `/api*`
|
||||
- `enforce_request_access(...)` enforces optional operator auth
|
||||
- robust Basic auth parsing and safe denial responses via `AccessDeniedError`
|
||||
- App middleware added in `src/transcription/app.py`:
|
||||
- enforces auth on protected paths
|
||||
- returns consistent `401` envelope and `WWW-Authenticate: Basic` for denied requests
|
||||
- Health endpoint `/healthz` remains intentionally unauthenticated.
|
||||
|
||||
### 3) Input validation and safe-output hardening
|
||||
|
||||
Completed baseline.
|
||||
|
||||
- `src/transcription/services/upload.py`
|
||||
- added size-based validation guard (`max_upload_bytes`)
|
||||
- emits `user_input_error` with actionable guidance on over-limit uploads
|
||||
- `src/transcription/errors.py`
|
||||
- `classify_unexpected_error(...)` now returns operation-only message without embedding raw exception text
|
||||
- preserves traceability via existing `error_id` and taxonomy while reducing accidental sensitive leak risk
|
||||
|
||||
### 4) Secret handling and configuration safety
|
||||
|
||||
Completed baseline.
|
||||
|
||||
- `src/transcription/config.py` additions:
|
||||
- `max_upload_bytes` (default `15 * 1024 * 1024`)
|
||||
- `operator_access_enabled` (default `False`)
|
||||
- `operator_username` (default `operator`)
|
||||
- `operator_password` (optional, required when auth enabled)
|
||||
- Added settings validator enforcing fail-fast config safety:
|
||||
- raises validation error if `OPERATOR_ACCESS_ENABLED=true` and `OPERATOR_PASSWORD` unset
|
||||
- `README.md` updated with Step 5 security env settings and explicit secret-handling guidance.
|
||||
|
||||
### 5) Dependency/security scanning baseline
|
||||
|
||||
Completed.
|
||||
|
||||
- Dependency vulnerability scan:
|
||||
- `uvx pip-audit`
|
||||
- Result: **No known vulnerabilities found**
|
||||
- Static security scan:
|
||||
- `uvx bandit -r src/transcription`
|
||||
- Result: **No issues identified** (0 low/medium/high)
|
||||
|
||||
---
|
||||
|
||||
## Test and Verification Evidence
|
||||
|
||||
### Added/Updated Tests
|
||||
|
||||
1. `tests/api/test_access_control.py`
|
||||
- unauthorized protected API denied (`401` + challenge)
|
||||
- invalid credentials denied
|
||||
- valid credentials accepted
|
||||
- `/ui` protected when auth enabled
|
||||
- `/healthz` remains unprotected
|
||||
2. `tests/services/test_upload.py`
|
||||
- added rejection test for payloads above `MAX_UPLOAD_BYTES`
|
||||
3. `tests/test_config.py`
|
||||
- added security defaults assertions
|
||||
- added fail-fast assertion for missing `OPERATOR_PASSWORD` when auth enabled
|
||||
4. `tests/test_errors.py`
|
||||
- updated expectations for sanitized unexpected-error message behavior
|
||||
5. Updated integration expectations where failure detail should no longer include raw exception text:
|
||||
- `tests/services/test_worker.py`
|
||||
- `tests/integration/test_pipeline_flow.py`
|
||||
6. `tests/test_app.py` updated for new middleware wiring.
|
||||
|
||||
### Validation Runs
|
||||
|
||||
Run and record outcomes:
|
||||
|
||||
- `uv run pytest --collect-only -q` -> passed
|
||||
- `uv run pytest -m unit -q` -> passed
|
||||
- `uv run pytest -m "not external" -q` -> passed
|
||||
- `uv run pytest -q` -> passed
|
||||
|
||||
### Security Scan Evidence
|
||||
|
||||
Record scan commands and outcomes:
|
||||
|
||||
- dependency scan command(s): `uvx pip-audit`
|
||||
- static/security lint command(s): `uvx bandit -r src/transcription`
|
||||
- critical/high findings: none
|
||||
- remediation/defer decisions: no remediations required for Step 5 baseline
|
||||
|
||||
---
|
||||
|
||||
## Requirement Traceability (Step 5)
|
||||
|
||||
| Step 5 Area | REQ Coverage | Status | Evidence |
|
||||
| --- | --- | --- | --- |
|
||||
| Private-network and single-operator safety posture | REQ-9 | met | `docs/ver1/ver1-step5-security-assumptions.md`, README security section |
|
||||
| Access control behavior at UI/API boundaries | REQ-5, REQ-7 | met | `src/transcription/security.py`, `src/transcription/app.py`, `tests/api/test_access_control.py` |
|
||||
| Input validation and safe user-facing error behavior | REQ-1, REQ-2, REQ-5 | met | `src/transcription/services/upload.py`, `src/transcription/errors.py`, updated tests |
|
||||
| Config and startup safety controls | REQ-8, REQ-10 | met | `src/transcription/config.py`, `tests/test_config.py`, `README.md` |
|
||||
| Persistence and domain integrity continuity | REQ-11, REQ-12 | met (no regressions) | full test lane pass including integration and worker flows |
|
||||
|
||||
---
|
||||
|
||||
## Operational Artifacts Produced
|
||||
|
||||
- `docs/ver1/ver1-step5.md`
|
||||
- `docs/ver1/ver1-step5-results.md`
|
||||
- `docs/ver1/ver1-step5-security-assumptions.md`
|
||||
- `src/transcription/security.py`
|
||||
- `tests/api/test_access_control.py`
|
||||
|
||||
---
|
||||
|
||||
## Risks, Exceptions, and Follow-Ups
|
||||
|
||||
1. Basic auth is intentionally right-sized for trusted private-network use; if deployment posture changes, stronger identity controls are required.
|
||||
2. Current model remains single shared operator credential (no per-user audit identity).
|
||||
3. No built-in brute-force/rate-limit controls in Step 5 scope; evaluate in future hardening if threat model expands.
|
||||
|
||||
Open follow-ups to carry forward:
|
||||
|
||||
- Consider stronger auth/session model if system becomes multi-user or internet-accessible.
|
||||
- Consider request throttling/rate limiting if threat model changes.
|
||||
|
||||
---
|
||||
|
||||
## Step 5 Exit Assessment
|
||||
|
||||
- Private-network assumptions and controls: **met**
|
||||
- Access-control baseline effectiveness: **met**
|
||||
- Validation and safe-output safety: **met (baseline)**
|
||||
- Secret handling and config safety: **met**
|
||||
- Dependency/security risk closure: **met (no critical/high findings)**
|
||||
- Test and regression safety: **met**
|
||||
|
||||
Step 5 completion status: **complete**
|
||||
|
||||
---
|
||||
|
||||
## Handoff to Step 6
|
||||
|
||||
Once Step 5 is marked complete, Step 6 can proceed with:
|
||||
|
||||
- clearer operational security assumptions for logs/runbooks
|
||||
- hardened boundary behavior for diagnosis and support
|
||||
- reduced risk posture for personal-scale ongoing operations
|
||||
@@ -0,0 +1,51 @@
|
||||
# Ver1 Step 5 Security Assumptions (Private-Network Baseline)
|
||||
|
||||
## Operating Model
|
||||
|
||||
This system is operated as:
|
||||
|
||||
1. single operator
|
||||
2. trusted private network
|
||||
3. non-public deployment (no direct internet exposure for UI/API)
|
||||
|
||||
Out of scope for Step 5:
|
||||
|
||||
- enterprise IAM/SSO/RBAC
|
||||
- internet-facing zero-trust edge controls
|
||||
- multi-tenant user isolation
|
||||
|
||||
## Step 5 Controls and Ownership
|
||||
|
||||
| Control | Boundary Owner | Verification |
|
||||
| --- | --- | --- |
|
||||
| Optional operator authentication for `/ui*` and `/api*` routes | `src/transcription/security.py`, `src/transcription/app.py` | `tests/api/test_access_control.py` |
|
||||
| Unauthorized contract (`401` + safe envelope + `WWW-Authenticate`) | `src/transcription/api/errors.py` | `tests/api/test_access_control.py` |
|
||||
| Upload size guard (`MAX_UPLOAD_BYTES`) | `src/transcription/services/upload.py`, `src/transcription/config.py` | `tests/services/test_upload.py` |
|
||||
| Fail-fast auth config when enabled | `src/transcription/config.py` | `tests/test_config.py` |
|
||||
| Safe unexpected error messaging (reduced leak surface) | `src/transcription/errors.py` | `tests/test_errors.py`, worker/integration failure tests |
|
||||
|
||||
## Access-Control Policy (Step 5)
|
||||
|
||||
- Health endpoint (`/healthz`) remains unauthenticated for operability checks.
|
||||
- When `OPERATOR_ACCESS_ENABLED=true`, protected paths require HTTP Basic auth:
|
||||
- `/ui`
|
||||
- `/ui/...`
|
||||
- `/api/...`
|
||||
- Credentials are runtime-configured:
|
||||
- `OPERATOR_USERNAME` (default `operator`)
|
||||
- `OPERATOR_PASSWORD` (required when access is enabled)
|
||||
|
||||
## Secrets Policy
|
||||
|
||||
- Secrets must be provided via runtime environment variables.
|
||||
- Secrets must not be committed to source control.
|
||||
- Secrets must not be logged.
|
||||
- Example secret values in docs must always be placeholders.
|
||||
|
||||
## Residual Risks (Accepted for Step 5)
|
||||
|
||||
1. HTTP Basic credentials are suitable only for trusted private-network deployment.
|
||||
2. No per-user identity model (single shared operator credential).
|
||||
3. No advanced brute-force/rate-limit controls in Step 5 scope.
|
||||
|
||||
These are carried forward for future hardening only if deployment posture changes.
|
||||
@@ -0,0 +1,459 @@
|
||||
# Step 5 Implementation Plan: Private-Network Safety Baseline
|
||||
|
||||
## Purpose
|
||||
|
||||
Implement **Ver1 Step 5** from `docs/ver1/ver1.md` by applying right-sized security controls for a single-user system running on a trusted private network.
|
||||
|
||||
Step 5 focuses on practical risk reduction without introducing unnecessary complexity, while preserving:
|
||||
|
||||
- personal-scale operational simplicity
|
||||
- single-operator workflow
|
||||
- explicit boundary ownership from `docs/architecture.md`
|
||||
- safety and diagnostics behavior defined in `docs/error_handling.md`
|
||||
|
||||
Primary governing docs:
|
||||
|
||||
- `docs/ver1/ver1.md` (Step 5 objective and sequencing)
|
||||
- `docs/architecture.md` (deployment model and module boundaries)
|
||||
- `docs/error_handling.md` (safe user output and diagnostic boundaries)
|
||||
- `docs/requirements.md` (REQ-1, REQ-2, REQ-5, REQ-7, REQ-8, REQ-9, REQ-10, REQ-11, REQ-12)
|
||||
- `docs/intent.md` (domain integrity priorities)
|
||||
|
||||
---
|
||||
|
||||
## MCP Resources Reviewed and Applied
|
||||
|
||||
All currently available resources on `john-stream-mcp` were reviewed. Step 5 applies the following guidance directly:
|
||||
|
||||
1. `resource://skills/pydantic-settings/document`
|
||||
- typed security-related runtime settings
|
||||
- explicit env/source precedence
|
||||
- fail-fast handling for missing/invalid required values
|
||||
|
||||
2. `resource://skills/fastapi-uv-docker/document`
|
||||
- environment and deployment safety defaults
|
||||
- startup/health posture and container hygiene assumptions
|
||||
- local secret handling expectations
|
||||
|
||||
3. `resource://skills/pytesting/document`
|
||||
- deterministic security-behavior test lanes
|
||||
- marker discipline and behavior-first assertions
|
||||
|
||||
4. `resource://skills/python-logging-dictconfig/document`
|
||||
- centralized logging discipline
|
||||
- avoid leaking sensitive values in logs
|
||||
|
||||
5. `resource://skills/nicegui-ui-customization/document`
|
||||
- user-safe failure messaging in UI
|
||||
- resilient interaction behavior and clear error feedback
|
||||
|
||||
6. `resource://skills/ruff-linting-formating/document`
|
||||
- keep lint quality baseline stable during safety changes
|
||||
|
||||
Planning methodology input:
|
||||
|
||||
7. `resource://prompts/greenfield-architecture/document`
|
||||
- explicit tradeoff-oriented staging
|
||||
- scope discipline for minimally sufficient security controls
|
||||
|
||||
Reviewed but not directly Step 5 execution-critical:
|
||||
|
||||
- skills: `copilot-customization`, `fastapi-async-sqlalchemy-modernization`, `mcp-details`, `nicegui`, `python-typing`, `vscode-configuration`, `zensical-docs`
|
||||
- prompts: `authoring`, `mcp-consumer-repo-shim`, `pytest-scaffold`, `pytest-fill-scaffold`
|
||||
|
||||
---
|
||||
|
||||
## Current-State Gap Summary (Step 5 Scope)
|
||||
|
||||
Based on current implementation and prior Step outputs:
|
||||
|
||||
1. **Private-network assumptions are implicit, not fully codified**
|
||||
- Need explicit, documented security posture and operator constraints.
|
||||
|
||||
2. **Access control for UI/API is minimal or absent**
|
||||
- Step 5 requires basic single-operator gating appropriate for private-network use.
|
||||
|
||||
3. **Input validation baseline exists but needs security-oriented audit closure**
|
||||
- Upload and API validation should be verified for abuse-resistant boundaries.
|
||||
|
||||
4. **Safe error output baseline exists (Step 2), but needs security confirmation pass**
|
||||
- Must ensure no sensitive internals leak through API/UI error payloads.
|
||||
|
||||
5. **Secret handling documentation needs formalization in Step 5 artifacts**
|
||||
- Local workflow should clearly prohibit secrets in repo-tracked files and logs.
|
||||
|
||||
6. **Dependency/security scanning is not yet formalized as a recurring gate**
|
||||
- Step 5 requires lightweight scanning and triage of high-risk findings.
|
||||
|
||||
---
|
||||
|
||||
## Scope for Step 5
|
||||
|
||||
### In scope
|
||||
|
||||
1. Codify private-network and single-operator security assumptions in docs and config.
|
||||
2. Add basic access control for UI/API actions (right-sized for trusted network model).
|
||||
3. Audit and harden input-validation boundaries (upload, API params/payloads, operational flags).
|
||||
4. Verify safe error surface behavior (UI/API) and prevent sensitive leak paths.
|
||||
5. Formalize local secret handling policy and usage examples.
|
||||
6. Add lightweight dependency/security scan workflow and triage policy.
|
||||
7. Add Step 5 verification tests and results artifact.
|
||||
|
||||
### Out of scope
|
||||
|
||||
- Internet-facing zero-trust security architecture
|
||||
- Enterprise IAM/SSO/role systems
|
||||
- Full cryptographic key-management infrastructure
|
||||
- Major security product integrations beyond lightweight V1 needs
|
||||
|
||||
---
|
||||
|
||||
## Target Decisions for Step 5
|
||||
|
||||
1. **Threat model is explicitly private-network + single operator**
|
||||
- Security controls are right-sized to this posture and documented as assumptions.
|
||||
|
||||
2. **Access control is required, even in private network mode**
|
||||
- Basic gate (single shared operator credential/token) protects UI/API mutation paths.
|
||||
|
||||
3. **Validation and output safety are strict defaults**
|
||||
- Reject invalid inputs early; never expose sensitive internals in user-facing outputs.
|
||||
|
||||
4. **Secrets are runtime-only**
|
||||
- No secrets committed to source control; no plaintext secret logging.
|
||||
|
||||
5. **Security scanning is lightweight but mandatory**
|
||||
- Add recurring dependency/security checks with high-risk triage and closure workflow.
|
||||
|
||||
6. **No security control may violate Step 1–4 operational simplicity guardrails**
|
||||
- Preserve deployability and maintainability for personal-scale use.
|
||||
|
||||
---
|
||||
|
||||
## Detailed Work Breakdown
|
||||
|
||||
## Phase A — Security Posture Definition and Gap Lock
|
||||
|
||||
- [ ] **A1. Define Step 5 threat model**
|
||||
- trusted private network
|
||||
- single operator
|
||||
- local deployment assumptions
|
||||
- explicit out-of-scope threat classes
|
||||
|
||||
- [ ] **A2. Produce security baseline checklist**
|
||||
- access control
|
||||
- validation boundaries
|
||||
- safe error behavior
|
||||
- secret handling
|
||||
- dependency risk checks
|
||||
|
||||
- [ ] **A3. Map controls to architecture boundaries**
|
||||
- UI
|
||||
- API
|
||||
- service
|
||||
- config/runtime
|
||||
- operator runbooks
|
||||
|
||||
### Deliverables
|
||||
|
||||
- `docs/ver1/ver1-step5-security-assumptions.md` (recommended)
|
||||
- Step 5 control matrix (control -> owner -> validation method)
|
||||
|
||||
### Exit Criteria
|
||||
|
||||
- private-network safety posture is explicit and approved
|
||||
- each in-scope control has boundary ownership and verification path
|
||||
|
||||
---
|
||||
|
||||
## Phase B — Basic Single-Operator Access Control
|
||||
|
||||
- [ ] **B1. Select access mechanism**
|
||||
- minimal approach suitable for private-network model
|
||||
- explicitly document tradeoffs and operator ergonomics
|
||||
|
||||
- [ ] **B2. Protect mutating operations first**
|
||||
- upload/create/accept/export-trigger endpoints
|
||||
- UI actions that trigger persistence changes
|
||||
|
||||
- [ ] **B3. Protect read operations as policy requires**
|
||||
- determine read-path gating expectations and apply consistently
|
||||
|
||||
- [ ] **B4. Add clear unauthorized behavior contract**
|
||||
- stable API status and safe message
|
||||
- UI feedback with actionable operator guidance
|
||||
|
||||
### Deliverables
|
||||
|
||||
- access-control policy and implementation notes
|
||||
- unauthorized behavior matrix (UI/API)
|
||||
|
||||
### Exit Criteria
|
||||
|
||||
- unauthorized actions are blocked consistently
|
||||
- authorized operator flows remain usable and deterministic
|
||||
|
||||
---
|
||||
|
||||
## Phase C — Input Validation and Safe Output Hardening
|
||||
|
||||
- [ ] **C1. Validation audit for all entry points**
|
||||
- file uploads (type/size/content guards)
|
||||
- route/query/body constraints
|
||||
- service-layer invariants
|
||||
|
||||
- [ ] **C2. Normalize validation failures to canonical taxonomy**
|
||||
- `validation_error` vs `user_input_error` consistency
|
||||
|
||||
- [ ] **C3. Confirm safe error output policy under security lens**
|
||||
- no stack traces/secrets/internal paths in UI/API default outputs
|
||||
- preserve error reference IDs for traceability
|
||||
|
||||
- [ ] **C4. Add abuse-resistant guardrails where practical**
|
||||
- basic request-size and payload-shape constraints
|
||||
- anti-duplication interaction safeguards (where missing)
|
||||
|
||||
### Deliverables
|
||||
|
||||
- validation-path inventory and hardening checklist
|
||||
- safe-output verification notes
|
||||
|
||||
### Exit Criteria
|
||||
|
||||
- input boundaries are deterministic and tested
|
||||
- user-facing error outputs remain safe and actionable
|
||||
|
||||
---
|
||||
|
||||
## Phase D — Secrets Handling and Configuration Safety
|
||||
|
||||
- [ ] **D1. Define secret handling policy**
|
||||
- where secrets are allowed (runtime env only)
|
||||
- where secrets are prohibited (source files, docs examples beyond placeholders)
|
||||
|
||||
- [ ] **D2. Enforce settings expectations**
|
||||
- required secret fields fail fast
|
||||
- avoid fallback defaults that silently weaken safety
|
||||
|
||||
- [ ] **D3. Add operator documentation for local secret workflow**
|
||||
- how to set environment values safely
|
||||
- how to rotate/update credentials locally
|
||||
|
||||
- [ ] **D4. Validate logging does not leak secret values**
|
||||
- startup/config logs
|
||||
- error logs for provider/config failures
|
||||
|
||||
### Deliverables
|
||||
|
||||
- secret-handling section in runbook/README/docs
|
||||
- settings and logging safety verification notes
|
||||
|
||||
### Exit Criteria
|
||||
|
||||
- no secret leakage paths remain in normal operations
|
||||
- operator can configure secrets safely using docs only
|
||||
|
||||
---
|
||||
|
||||
## Phase E — Dependency and Security Scanning Baseline
|
||||
|
||||
- [ ] **E1. Select lightweight scanning commands for V1**
|
||||
- dependency vulnerability scan
|
||||
- optional static security scan if practical
|
||||
|
||||
- [ ] **E2. Define triage policy for findings**
|
||||
- severity classification
|
||||
- required closure criteria for Step 5 completion
|
||||
|
||||
- [ ] **E3. Run scans and capture evidence**
|
||||
- record command outputs/summaries
|
||||
- remediate or formally defer with risk notes
|
||||
|
||||
- [ ] **E4. Add recurring execution guidance**
|
||||
- local pre-release checklist integration
|
||||
- future CI gate handoff for Step 7/9
|
||||
|
||||
### Deliverables
|
||||
|
||||
- Step 5 scan report artifact (recommended)
|
||||
- triage log of resolved/deferred findings
|
||||
|
||||
### Exit Criteria
|
||||
|
||||
- no unresolved critical vulnerabilities in Step 5 scope
|
||||
- high-risk findings are resolved or explicitly risk-accepted with rationale
|
||||
|
||||
---
|
||||
|
||||
## Phase F — Verification and Test Expansion
|
||||
|
||||
Apply `pytesting` guidance (deterministic, behavior-first, strict markers).
|
||||
|
||||
- [ ] **F1. Access-control tests**
|
||||
- unauthorized requests are rejected as expected
|
||||
- authorized operator requests succeed
|
||||
|
||||
- [ ] **F2. Validation and abuse-boundary tests**
|
||||
- invalid payloads rejected with stable category/status
|
||||
- file-type/size constraints enforced
|
||||
|
||||
- [ ] **F3. Safe-output tests**
|
||||
- API/UI error responses avoid sensitive details
|
||||
- error IDs and suggestions remain present
|
||||
|
||||
- [ ] **F4. Config/secret safety tests**
|
||||
- required secrets fail fast when missing
|
||||
- no unsafe fallback behavior introduced
|
||||
|
||||
### Validation Commands
|
||||
|
||||
- `uv run pytest --collect-only -q`
|
||||
- `uv run pytest -m unit -q`
|
||||
- `uv run pytest -m "not external" -q`
|
||||
- `uv run pytest -q`
|
||||
|
||||
### Exit Criteria
|
||||
|
||||
- Step 5 safety behavior is test-covered and passing
|
||||
- no regression in core upload/transcribe/review workflows
|
||||
|
||||
---
|
||||
|
||||
## Phase G — Documentation and Risk Closure
|
||||
|
||||
- [ ] **G1. Create Step 5 results artifact**
|
||||
- `docs/ver1/ver1-step5-results.md`
|
||||
|
||||
- [ ] **G2. Update operator-facing docs**
|
||||
- security assumptions and local deployment cautions
|
||||
- credential handling and recovery basics
|
||||
|
||||
- [ ] **G3. Update traceability and carry-forward notes**
|
||||
- map Step 5 controls to REQ and evidence
|
||||
|
||||
### Deliverables
|
||||
|
||||
- `docs/ver1/ver1-step5-results.md`
|
||||
- updated security assumptions checklist and risk summary
|
||||
|
||||
### Exit Criteria
|
||||
|
||||
- Step 5 controls and residual risks are fully documented
|
||||
- handoff is ready for Step 6 observability and Step 7 quality gates
|
||||
|
||||
---
|
||||
|
||||
## Recommended Implementation Order
|
||||
|
||||
1. Phase A — posture definition and gap lock
|
||||
2. Phase B — access control baseline
|
||||
3. Phase C — validation/output hardening
|
||||
4. Phase D — secrets and config safety
|
||||
5. Phase E — dependency/security scan baseline
|
||||
6. Phase F — test expansion and verification
|
||||
7. Phase G — docs and risk closure
|
||||
|
||||
This order reduces risk by locking assumptions first, then applying controls at highest-impact boundaries before final verification and documentation.
|
||||
|
||||
---
|
||||
|
||||
## Step 5 Execution Checklist (Phase-by-Phase)
|
||||
|
||||
Use this checklist to execute Step 5 in implementation order and record progress/evidence.
|
||||
|
||||
### Phase A — Security Posture Definition and Gap Lock
|
||||
|
||||
- [ ] Publish `docs/ver1/ver1-step5-security-assumptions.md`.
|
||||
- [ ] Record explicit in-scope and out-of-scope threat classes.
|
||||
- [ ] Produce Step 5 control matrix (control, owner, validation method).
|
||||
- [ ] Confirm boundary ownership for each control (UI/API/service/config/docs).
|
||||
|
||||
### Phase B — Basic Single-Operator Access Control
|
||||
|
||||
- [ ] Choose and document access mechanism (with rationale and tradeoffs).
|
||||
- [ ] Implement enforcement for mutating API operations.
|
||||
- [ ] Implement corresponding UI-side access behavior for protected actions.
|
||||
- [ ] Decide and enforce read-path protection policy.
|
||||
- [ ] Add unauthorized API/UI contract tests.
|
||||
|
||||
### Phase C — Input Validation and Safe Output Hardening
|
||||
|
||||
- [ ] Complete input-validation inventory for upload/API/service boundaries.
|
||||
- [ ] Tighten payload/file constraints where gaps are found.
|
||||
- [ ] Ensure validation failure categories match `docs/error_handling.md`.
|
||||
- [ ] Verify user-facing errors remain safe, actionable, and traceable.
|
||||
- [ ] Add regression tests for invalid/boundary inputs.
|
||||
|
||||
### Phase D — Secrets Handling and Configuration Safety
|
||||
|
||||
- [ ] Document secrets policy (runtime-only, no repo storage).
|
||||
- [ ] Verify required secret settings fail fast when missing.
|
||||
- [ ] Audit logs for accidental secret leakage risk paths.
|
||||
- [ ] Update operator docs for local secret setup/rotation workflow.
|
||||
- [ ] Add tests for config safety expectations where practical.
|
||||
|
||||
### Phase E — Dependency and Security Scanning Baseline
|
||||
|
||||
- [ ] Select scanning commands and record tool versions.
|
||||
- [ ] Run baseline scans and capture outputs.
|
||||
- [ ] Triage findings by severity and exploitability in private-network context.
|
||||
- [ ] Resolve/mitigate critical findings; document accepted residual risk.
|
||||
- [ ] Add recurring scan guidance for release workflow handoff.
|
||||
|
||||
### Phase F — Verification and Test Expansion
|
||||
|
||||
- [ ] Run `uv run pytest --collect-only -q`.
|
||||
- [ ] Run `uv run pytest -m unit -q`.
|
||||
- [ ] Run `uv run pytest -m "not external" -q`.
|
||||
- [ ] Run `uv run pytest -q`.
|
||||
- [ ] Confirm no regressions in upload/transcribe/review core flows.
|
||||
|
||||
### Phase G — Documentation and Risk Closure
|
||||
|
||||
- [ ] Complete `docs/ver1/ver1-step5-results.md` with evidence.
|
||||
- [ ] Update docs/README/runbooks with final Step 5 security posture.
|
||||
- [ ] Record REQ traceability updates and residual risks.
|
||||
- [ ] Confirm Step 5 completion checklist items are all closed.
|
||||
|
||||
---
|
||||
|
||||
## Risks and Mitigations
|
||||
|
||||
1. **Risk:** Over-engineering beyond private-network needs
|
||||
- **Mitigation:** enforce Step 5 scope discipline and threat-model constraints.
|
||||
|
||||
2. **Risk:** Access controls disrupt operator usability
|
||||
- **Mitigation:** keep mechanism minimal and test primary workflows thoroughly.
|
||||
|
||||
3. **Risk:** Sensitive details leak through errors/logging
|
||||
- **Mitigation:** apply safe-output and log-sanitization checks with tests.
|
||||
|
||||
4. **Risk:** Unpatched dependency vulnerabilities remain invisible
|
||||
- **Mitigation:** formalize scan + triage + evidence capture workflow.
|
||||
|
||||
5. **Risk:** Secret handling remains ad hoc
|
||||
- **Mitigation:** fail-fast settings + explicit operator documentation + review checks.
|
||||
|
||||
---
|
||||
|
||||
## Step 5 Completion Checklist
|
||||
|
||||
- [ ] Private-network and single-operator security assumptions are documented.
|
||||
- [ ] Basic single-operator access control is implemented and verified.
|
||||
- [ ] Input-validation boundaries are audited, hardened, and test-covered.
|
||||
- [ ] UI/API error output safety is confirmed under security tests.
|
||||
- [ ] Secret handling policy and local workflow docs are complete.
|
||||
- [ ] Dependency/security scans are run; critical findings are resolved.
|
||||
- [ ] Step 5 tests pass across all validation lanes.
|
||||
- [ ] `docs/ver1/ver1-step5-results.md` is completed with evidence and residual risks.
|
||||
|
||||
---
|
||||
|
||||
## Handoff to Step 6
|
||||
|
||||
Step 5 completion enables Step 6 (Minimal Observability & Operability) with:
|
||||
|
||||
- explicit security assumptions for operator context
|
||||
- access and validation controls suitable for private-network operation
|
||||
- safer runtime/configuration handling for ongoing operations
|
||||
- dependency-risk visibility feeding release-readiness gates
|
||||
@@ -0,0 +1,206 @@
|
||||
## Step 6 Goal (from `docs/ver1/ver1.md`)
|
||||
Implement **minimal observability & operability** so a single operator can quickly diagnose and recover from common failures.
|
||||
|
||||
---
|
||||
|
||||
## 1) Current-State Assessment (what already exists)
|
||||
|
||||
### Already in place
|
||||
- Central startup logging initialization via `setup_logging()` and `dictConfig` (`src/transcription/config.py`, `src/transcription/app.py`).
|
||||
- Error taxonomy and `error_id` envelope contract (`src/transcription/errors.py`) aligned with `docs/error_handling.md`.
|
||||
- Error handling for API and worker includes category + error IDs in some paths (`src/transcription/api/errors.py`, `src/transcription/worker.py`).
|
||||
- Basic health endpoint `/healthz` (`src/transcription/api/health.py`).
|
||||
- UI error display already shows actionable message + error reference (`src/transcription/ui/error_presenter.py`).
|
||||
|
||||
### Gaps to close for Step 6
|
||||
1. **Structured logging is inconsistent** (many logs are free-form text with embedded key/value; no enforced schema).
|
||||
2. **Boundary coverage is incomplete** (UI/service/API/worker don’t all emit consistent operation logs).
|
||||
3. `/healthz` is very basic; no lightweight readiness/startup diagnostics endpoint/reporting.
|
||||
4. No concise **operator runbook** yet (start/stop, log interpretation, recovery playbooks).
|
||||
5. Minimal counters/timings are not yet standardized.
|
||||
|
||||
---
|
||||
|
||||
## 2) MCP Guidance Incorporated (relevant items)
|
||||
|
||||
From `john-stream-mcp`, these are directly applied:
|
||||
|
||||
- **`python-logging-dictconfig`**: keep one centralized `dictConfig`, configure once at startup, named loggers in modules.
|
||||
- **`fastapi-async-sqlalchemy-modernization`**: include observability + health/readiness checks; explicit lifecycle and deterministic startup/shutdown checks.
|
||||
- **`fastapi-uv-docker`**: keep `/healthz`; add practical readiness/ops checks for deployment clarity.
|
||||
- **`pytesting`**: deterministic tests, concise structure, validation lanes (`collect-only`, `unit`, `not external`, full).
|
||||
- **`pydantic-settings`**: keep typed settings as single source for logging/health behavior flags.
|
||||
- **`nicegui` + `nicegui-ui-customization`**: preserve clear, actionable user-facing error feedback and non-blocking UI flows.
|
||||
- **`zensical-docs`**: produce focused, navigable operator docs.
|
||||
|
||||
(Other MCP resources were reviewed but are not core to Step 6 implementation scope.)
|
||||
|
||||
---
|
||||
|
||||
## 3) Detailed Implementation Plan for Step 6
|
||||
|
||||
## Workstream A — Structured Logging Contract
|
||||
|
||||
### A1. Define a canonical log event schema
|
||||
Create a project log schema (doc + code-level constants) with required keys:
|
||||
- `timestamp` (UTC)
|
||||
- `level`
|
||||
- `logger`
|
||||
- `operation`
|
||||
- `event`
|
||||
- `error_id` (when error)
|
||||
- `category` (when error)
|
||||
- `exception_type` (when error)
|
||||
- `job_id`, `document_id` (when relevant)
|
||||
- optional: `duration_ms`, `retry_count`, `status`
|
||||
|
||||
### A2. Standardize log emission helpers
|
||||
Add small logging helpers (or adapter utilities) to reduce drift:
|
||||
- `log_operation_start(...)`
|
||||
- `log_operation_success(...)`
|
||||
- `log_operation_error(...)`
|
||||
|
||||
Keep this minimal and avoid heavy observability frameworks.
|
||||
|
||||
### A3. Update formatter to structured output
|
||||
Use `dictConfig` to emit either:
|
||||
- JSON lines (preferred for structure), or
|
||||
- strict key-value line format with fixed fields.
|
||||
|
||||
**Recommendation:** JSON lines to satisfy “structured logging” unambiguously while still simple.
|
||||
|
||||
---
|
||||
|
||||
## Workstream B — Boundary-by-Boundary Instrumentation
|
||||
|
||||
### B1. API boundary (`src/transcription/api/*`)
|
||||
- Add request-level operation logs for key routes (`upload.submit`, `jobs.list`, `jobs.get`, etc.).
|
||||
- Ensure API exception handler logs always include `error_id`, `category`, `operation`, `exception_type`.
|
||||
|
||||
### B2. Service boundary (`src/transcription/services/*`)
|
||||
- Add operation logs around:
|
||||
- upload validation/persist,
|
||||
- transcription orchestration,
|
||||
- revision add/accept,
|
||||
- search/export.
|
||||
- Add timing (`duration_ms`) for high-value operations only.
|
||||
|
||||
### B3. Worker boundary (`src/transcription/worker.py`)
|
||||
- Standardize all worker log events to schema.
|
||||
- Ensure retry logs include: `retriable`, `retry_count`, `max_retries`, `backoff_seconds`.
|
||||
- Ensure terminal failure logs include error contract fields.
|
||||
|
||||
### B4. UI boundary (`src/transcription/ui/*`)
|
||||
- Keep user-safe UI messages as-is.
|
||||
- Add backend/UI logger events for user-triggered failures (operation + error_id + category) so UI-visible errors correlate to server logs.
|
||||
|
||||
---
|
||||
|
||||
## Workstream C — Health, Readiness, Startup Operability
|
||||
|
||||
### C1. Keep `/healthz` lightweight
|
||||
- Return “process is running” status quickly.
|
||||
|
||||
### C2. Add lightweight `/readyz`
|
||||
Include small checks:
|
||||
- DB connectivity ping.
|
||||
- Worker thread alive check.
|
||||
- Optional prompt directory existence check.
|
||||
|
||||
Return structured status payload with per-check pass/fail.
|
||||
|
||||
### C3. Startup self-check summary log
|
||||
At startup, emit one concise ops summary event:
|
||||
- environment
|
||||
- schema validation result
|
||||
- worker started
|
||||
- directories checked
|
||||
- bootstrap/migration mode flags
|
||||
|
||||
---
|
||||
|
||||
## Workstream D — Minimal Counters & Timings
|
||||
|
||||
Add only high-value diagnostics:
|
||||
1. `worker_jobs_processed_total`
|
||||
2. `worker_jobs_failed_total`
|
||||
3. `worker_retries_total`
|
||||
4. `transcription_duration_ms` (per job)
|
||||
5. `upload_persist_duration_ms` (per upload path)
|
||||
|
||||
Implementation can be log-derived counters (no external metrics backend required).
|
||||
|
||||
---
|
||||
|
||||
## Workstream E — Operator Runbook
|
||||
|
||||
Create concise runbook doc (recommended: `docs/ver1/ver1-step6-operator-runbook.md`) with:
|
||||
|
||||
1. **Start/Stop**
|
||||
- local `uv` run mode
|
||||
- docker compose mode (if applicable)
|
||||
|
||||
2. **Where logs are**
|
||||
- stdout, docker logs commands, filtering by `error_id` / `operation`.
|
||||
|
||||
3. **Common failure patterns → recovery**
|
||||
- provider timeout
|
||||
- auth denied
|
||||
- missing prompt dir
|
||||
- DB unavailable
|
||||
- job stuck/failed with retry exhausted
|
||||
|
||||
4. **Recovery procedures**
|
||||
- restart sequence
|
||||
- verify health/readiness
|
||||
- when to requeue/re-upload
|
||||
|
||||
5. **Escalation artifacts**
|
||||
- capture timestamp + error_id + operation + job_id/document_id
|
||||
|
||||
Also update `README.md` with short links to the runbook.
|
||||
|
||||
---
|
||||
|
||||
## Workstream F — Verification & Quality Gates
|
||||
|
||||
### Tests to add/update
|
||||
- `tests/api/test_health.py`
|
||||
- `/healthz` baseline
|
||||
- `/readyz` pass/fail behavior
|
||||
- `tests/api/test_error_responses.py` / `tests/api/test_routes.py`
|
||||
- logs include `error_id/category/operation` on failures
|
||||
- `tests/services/test_worker.py`
|
||||
- retry/failure log fields + timing presence
|
||||
- `tests/ui/*`
|
||||
- ensure UI error correlation path includes operation/ref id behavior
|
||||
|
||||
### Validation commands (per MCP pytest guidance)
|
||||
- `uv run pytest --collect-only -q`
|
||||
- `uv run pytest -m unit -q`
|
||||
- `uv run pytest -m "not external" -q`
|
||||
- `uv run pytest -q`
|
||||
|
||||
---
|
||||
|
||||
## 4) Traceability to Governing Docs
|
||||
|
||||
- **`docs/ver1/ver1.md` Step 6:** all 5 implementation bullets covered.
|
||||
- **`docs/error_handling.md`:** logging contract fields and error taxonomy continuity enforced.
|
||||
- **`docs/architecture.md`:** respects modular boundaries, in-process worker model, low-complexity ops.
|
||||
- **`docs/requirements.md`:**
|
||||
- REQ-8 (startup logging/config centralization) strengthened,
|
||||
- REQ-5 (status visibility) improved operationally,
|
||||
- REQ-7 lifecycle ownership observability improved.
|
||||
- **`docs/intent.md`:** keeps operation simple for personal-scale archival workflow.
|
||||
|
||||
---
|
||||
|
||||
## 5) Suggested Execution Order (low risk)
|
||||
|
||||
1. Logging schema + formatter + helpers
|
||||
2. Worker/API instrumentation (highest value)
|
||||
3. Service/UI instrumentation
|
||||
4. `/readyz` + startup summary check
|
||||
5. Runbook + README links
|
||||
6. Tests + Step 6 results artifact (`docs/ver1/ver1-step6-results.md`)
|
||||
+133
-131
@@ -1,40 +1,41 @@
|
||||
# Version 1 Implementation Plan
|
||||
|
||||
This plan defines the path from MVP to **Version 1 complete**.
|
||||
The objective is to deliver the full scoped product with production readiness, while explicitly separating refinements/enhancements into a future document.
|
||||
The objective is to deliver the full scoped product with readiness for reliable personal-scale operation, while explicitly separating refinements/enhancements into a future document.
|
||||
|
||||
---
|
||||
|
||||
## 0) Plan Governance & Scope Control (Foundation)
|
||||
|
||||
**Goal:** Keep execution focused on V1 completion, not optimization/perfection.
|
||||
**Goal:** Keep execution focused on V1 completion and avoid unnecessary process overhead.
|
||||
|
||||
### Implementation Steps
|
||||
1. Create and maintain a **V1 Traceability Matrix**:
|
||||
- Requirement ID
|
||||
- Current status (`done`, `partial`, `not started`)
|
||||
- Owner
|
||||
- Validation method
|
||||
2. Define V1 completion gates:
|
||||
- Functional complete
|
||||
- Operationally complete
|
||||
- Production-ready complete
|
||||
- Personal-deployment ready
|
||||
3. Snapshot the MVP baseline (tag/changelog reference).
|
||||
4. Create a standing rule: any non-V1 idea is logged to a separate enhancements backlog document (to be named later), not added to active V1 scope unless explicitly approved.
|
||||
4. Keep a standing rule: non-V1 ideas go to a separate enhancements backlog, and enter V1 only by explicit approval.
|
||||
|
||||
### Deliverables
|
||||
- `docs/ver1/ver1.md` (this plan)
|
||||
- V1 traceability artifact (linked from here when created)
|
||||
- V1 traceability artifact:
|
||||
- `docs/ver1/ver1-step1-2-carry-forward-checklist.md`
|
||||
- `docs/ver1/ver1-step2-error-path-inventory.md` (supporting artifact)
|
||||
|
||||
### Exit Criteria
|
||||
- Every in-scope requirement has explicit ownership and status.
|
||||
- Scope-change process is agreed and followed.
|
||||
- Every in-scope requirement has explicit status and validation evidence.
|
||||
- Scope-change discipline is followed consistently.
|
||||
|
||||
---
|
||||
|
||||
## 1) Architecture Consolidation
|
||||
|
||||
**Goal:** Align implementation with the intended architecture and reduce MVP shortcuts.
|
||||
**Goal:** Align implementation with intended architecture while preserving simplicity.
|
||||
|
||||
### Implementation Steps
|
||||
1. Compare implemented modules/components with architecture documentation.
|
||||
@@ -42,201 +43,202 @@ The objective is to deliver the full scoped product with production readiness, w
|
||||
- Temporary coupling
|
||||
- Missing interfaces
|
||||
- Placeholder services/components
|
||||
3. Resolve high-risk architectural gaps first.
|
||||
4. Record key decisions and tradeoffs in ADRs.
|
||||
3. Resolve architecture gaps that threaten reliability, maintainability, or clear boundaries.
|
||||
4. Record material decisions and tradeoffs in ADRs.
|
||||
|
||||
### Deliverables
|
||||
- Updated architecture diagrams and boundaries
|
||||
- ADR entries for major decisions
|
||||
- ADR entries for material decisions
|
||||
|
||||
### Exit Criteria
|
||||
- Architecture documentation reflects system reality.
|
||||
- Critical architecture risks are addressed or scheduled with owners/dates.
|
||||
- High-impact architecture risks are addressed or explicitly scheduled.
|
||||
|
||||
---
|
||||
|
||||
## 2) Error Handling & Reliability Hardening
|
||||
|
||||
**Goal:** Ensure predictable, safe behavior under failure conditions.
|
||||
**Goal:** Ensure predictable, diagnosable behavior under expected failure conditions.
|
||||
|
||||
### Implementation Steps
|
||||
1. Standardize error taxonomy and envelope format across all layers.
|
||||
1. Apply the canonical taxonomy and response model from `docs/error_handling.md` across UI/API/service/worker boundaries.
|
||||
2. Ensure clear distinction between:
|
||||
- User-facing errors
|
||||
- Internal/system errors
|
||||
- User-facing safe messages
|
||||
- Internal diagnostic detail
|
||||
- Retryable vs non-retryable failures
|
||||
3. Add resilience controls where needed:
|
||||
3. Implement practical resilience controls where needed:
|
||||
- Timeouts
|
||||
- Retries with backoff
|
||||
- Circuit breaking / fallback logic
|
||||
- Bounded retries with backoff
|
||||
- Explicit terminal failure states
|
||||
4. Add failure-path tests for critical workflows.
|
||||
|
||||
### Deliverables
|
||||
- Error code catalog/reference
|
||||
- Failure mode test coverage for critical paths
|
||||
- Error handling reference aligned with `docs/error_handling.md`
|
||||
- Failure-mode test coverage for critical paths
|
||||
|
||||
### Exit Criteria
|
||||
- Error behavior is consistent across major flows.
|
||||
- Known failure scenarios are tested and pass.
|
||||
- Failed jobs include actionable, traceable failure detail.
|
||||
|
||||
---
|
||||
|
||||
## 3) Functional Completion by Requirement Domain
|
||||
|
||||
**Goal:** Complete all V1 functional requirements in a risk-aware order.
|
||||
**Goal:** Complete all V1 requirements in a practical, user-first order.
|
||||
|
||||
### Recommended Order
|
||||
1. Business-critical end-user flows
|
||||
2. Data integrity and consistency capabilities
|
||||
3. Admin/operational controls
|
||||
4. Lower-priority UX and quality-of-life items that are in V1 scope
|
||||
1. End-user core flows (upload → transcribe → review)
|
||||
2. Data integrity and persistence behavior
|
||||
3. Minimal operator controls needed for personal use
|
||||
4. In-scope UX quality improvements
|
||||
|
||||
### Implementation Steps
|
||||
For each requirement slice:
|
||||
1. Finalize contract/schema
|
||||
2. Implement domain logic
|
||||
3. Implement persistence/state changes
|
||||
4. Integrate API/UI
|
||||
5. Add automated tests
|
||||
6. Update docs
|
||||
1. Confirm contract/schema
|
||||
2. Implement service/domain logic
|
||||
3. Implement persistence/state transitions
|
||||
4. Integrate API/UI behavior
|
||||
5. Add or update automated tests
|
||||
6. Update relevant docs
|
||||
|
||||
### Deliverables
|
||||
- Requirement completion report with validation evidence
|
||||
- Requirement completion report with validation evidence linked to REQ IDs
|
||||
|
||||
### Exit Criteria
|
||||
- All V1 “must-have” requirements are complete and validated.
|
||||
- All V1 must-have requirements are complete and verified.
|
||||
|
||||
---
|
||||
|
||||
## 4) Data Model, Migration, and Backfill Safety
|
||||
## 4) Data Model and Migration Safety
|
||||
|
||||
**Goal:** Ensure data model and migrations are production-safe.
|
||||
**Goal:** Keep schema evolution safe and simple for personal-scale deployment.
|
||||
|
||||
### Implementation Steps
|
||||
1. Validate schema against final V1 domain needs.
|
||||
2. Implement forward-safe migrations.
|
||||
3. Define rollback/mitigation plans for migration failures.
|
||||
4. Build and verify backfill scripts (if needed).
|
||||
5. Add migration rehearsal in staging with representative data.
|
||||
1. Validate schema against finalized V1 domain needs.
|
||||
2. Implement forward-safe migrations for expected upgrades.
|
||||
3. Define a simple rollback/mitigation path for migration failures.
|
||||
4. Add backfill scripts only where truly required.
|
||||
5. Rehearse migration + rollback locally using representative sample data.
|
||||
|
||||
### Deliverables
|
||||
- Migration runbook
|
||||
- Backfill verification checklist
|
||||
- Migration and rollback runbook
|
||||
- Backfill checklist (if applicable)
|
||||
|
||||
### Exit Criteria
|
||||
- Migration plan validated in staging.
|
||||
- No unresolved data-loss risk for V1 rollout.
|
||||
- Migration path is tested and documented.
|
||||
- No unresolved data-loss risk for V1 upgrade.
|
||||
|
||||
---
|
||||
|
||||
## 5) Security, Access Control, and Compliance Baseline
|
||||
## 5) Private-Network Safety Baseline
|
||||
|
||||
**Goal:** Close MVP security gaps and establish V1 baseline controls.
|
||||
**Goal:** Apply right-sized security controls for a single-user system on a trusted private network.
|
||||
|
||||
### Implementation Steps
|
||||
1. Complete authn/authz coverage for all routes/actions.
|
||||
2. Enforce input validation and output sanitization.
|
||||
3. Verify secret management and credential rotation process.
|
||||
4. Add audit logging for sensitive operations.
|
||||
5. Run dependency/security scanning in CI and remediate findings.
|
||||
1. Enforce private-network deployment assumptions in docs and configuration.
|
||||
2. Ensure basic single-operator access control for UI/API actions.
|
||||
3. Enforce input validation and safe error output behavior.
|
||||
4. Keep secrets out of source control; document local secret handling.
|
||||
5. Run lightweight dependency/security scanning and resolve high-risk findings.
|
||||
|
||||
### Deliverables
|
||||
- Security checklist with status
|
||||
- Threat/risk update for V1 scope
|
||||
- Security assumptions checklist (private network, single operator)
|
||||
- Basic risk update for V1 scope
|
||||
|
||||
### Exit Criteria
|
||||
- No unresolved critical/high vulnerabilities for V1 launch.
|
||||
- Access control behavior verified by tests.
|
||||
- No unresolved critical vulnerabilities.
|
||||
- Access behavior and validation rules are verified for intended operating model.
|
||||
|
||||
---
|
||||
|
||||
## 6) Observability & Operability
|
||||
## 6) Minimal Observability & Operability
|
||||
|
||||
**Goal:** Make system behavior observable and supportable in production.
|
||||
**Goal:** Keep operation and troubleshooting simple, clear, and reliable.
|
||||
|
||||
### Implementation Steps
|
||||
1. Standardize structured logging and correlation IDs.
|
||||
2. Add core metrics:
|
||||
- Latency
|
||||
- Throughput
|
||||
- Error rates
|
||||
- Resource saturation
|
||||
3. Add tracing for critical request/workflow paths.
|
||||
4. Define SLOs/SLIs and alert thresholds.
|
||||
5. Prepare incident response and rollback runbooks.
|
||||
1. Standardize structured logging across UI/API/service/worker boundaries.
|
||||
2. Ensure logged errors include category and error reference IDs per `error_handling.md`.
|
||||
3. Add lightweight health/startup checks.
|
||||
4. Document a concise operator runbook:
|
||||
- start/stop
|
||||
- log locations
|
||||
- common failure patterns and recovery steps
|
||||
5. Add minimal counters/timings only where they clearly improve diagnosis.
|
||||
|
||||
### Deliverables
|
||||
- Dashboards and alerts
|
||||
- Operations runbooks
|
||||
- Logging and error-traceability baseline
|
||||
- Operator runbook
|
||||
|
||||
### Exit Criteria
|
||||
- Team can detect, triage, and remediate incidents quickly.
|
||||
- Core production signals are available and reliable.
|
||||
- Operator can diagnose common failures using logs + runbook.
|
||||
- System recovery procedures are documented and repeatable.
|
||||
|
||||
---
|
||||
|
||||
## 7) Test Strategy Expansion & Quality Gates
|
||||
## 7) Test Coverage and Practical Quality Gates
|
||||
|
||||
**Goal:** Raise confidence for repeatable, low-risk releases.
|
||||
**Goal:** Prevent regressions in critical flows without overbuilding test infrastructure.
|
||||
|
||||
### Implementation Steps
|
||||
1. Expand unit and integration tests across V1 features.
|
||||
2. Add contract tests between key components/services.
|
||||
3. Add end-to-end tests for critical user journeys.
|
||||
4. Add non-functional tests where relevant:
|
||||
- Performance/load
|
||||
- Soak
|
||||
- Failure-injection scenarios
|
||||
5. Enforce CI quality gates (tests, lint, type checks, security scans).
|
||||
1. Expand unit and integration tests for all V1 requirement slices.
|
||||
2. Add end-to-end tests for critical journeys:
|
||||
- upload
|
||||
- process/transcribe
|
||||
- view result
|
||||
- failure visibility
|
||||
3. Add targeted contract tests where adapter boundaries are error-prone.
|
||||
4. Keep CI gates focused on high-value checks (tests, lint, type checks, dependency scan).
|
||||
|
||||
### Deliverables
|
||||
- Test matrix with ownership
|
||||
- CI gate definition and thresholds
|
||||
- V1 test matrix mapped to requirements and critical flows
|
||||
- CI quality-gate checklist
|
||||
|
||||
### Exit Criteria
|
||||
- Critical-path regressions are blocked automatically.
|
||||
- Test coverage and reliability thresholds meet V1 targets.
|
||||
- Critical-path regressions are automatically detected.
|
||||
- Test suite gives consistent release confidence for personal-scale operation.
|
||||
|
||||
---
|
||||
|
||||
## 8) Performance & Scalability Validation
|
||||
## 8) Performance Validation for Personal Scale
|
||||
|
||||
**Goal:** Meet expected V1 performance at projected load.
|
||||
**Goal:** Confirm acceptable responsiveness for expected personal-use workload.
|
||||
|
||||
### Implementation Steps
|
||||
1. Define performance budgets per key flow.
|
||||
2. Benchmark current behavior in staging.
|
||||
3. Optimize bottlenecks (queries, caching, concurrency, etc.).
|
||||
4. Re-test after each optimization and compare against budget.
|
||||
5. Document known limits and safe operating bounds.
|
||||
1. Define practical performance expectations for key flows.
|
||||
2. Run representative tests using real document samples.
|
||||
3. Address obvious bottlenecks in queries, file handling, or worker concurrency.
|
||||
4. Document known limits and expected operating bounds.
|
||||
|
||||
### Deliverables
|
||||
- Performance benchmark report
|
||||
- Optimization log
|
||||
- Short performance validation note
|
||||
- Known-limits summary
|
||||
|
||||
### Exit Criteria
|
||||
- V1 performance targets met for expected usage profile.
|
||||
- Core flows remain responsive for expected corpus size and usage patterns.
|
||||
|
||||
---
|
||||
|
||||
## 9) Release Engineering & Environment Readiness
|
||||
## 9) Release Readiness and Environment Simplicity
|
||||
|
||||
**Goal:** Make deployment repeatable, controlled, and reversible.
|
||||
**Goal:** Make deployment and rollback repeatable for a single-operator Docker Compose setup.
|
||||
|
||||
### Implementation Steps
|
||||
1. Harden CI/CD pipeline with clear promotion gates.
|
||||
2. Ensure config parity and consistency across environments.
|
||||
3. Define rollout strategy (phased/canary/limited release as applicable).
|
||||
4. Validate rollback procedures in staging.
|
||||
5. Produce release checklist and ownership model.
|
||||
1. Define a simple release checklist:
|
||||
- run tests
|
||||
- run one end-to-end transcription check
|
||||
- verify migration compatibility
|
||||
2. Document environment configuration requirements clearly.
|
||||
3. Validate deployment and rollback steps in a local rehearsal.
|
||||
4. Add backup/restore verification for core persisted data.
|
||||
|
||||
### Deliverables
|
||||
- Release playbook
|
||||
- Environment readiness checklist
|
||||
- Release checklist
|
||||
- Environment and rollback guide
|
||||
|
||||
### Exit Criteria
|
||||
- Deployment and rollback are rehearsed and reliable.
|
||||
- Release process is executable without tribal knowledge.
|
||||
- Deployment/rollback is rehearsed and documented.
|
||||
- Operator can release safely without hidden steps.
|
||||
|
||||
---
|
||||
|
||||
@@ -252,7 +254,7 @@ For each requirement slice:
|
||||
- Index/navigation
|
||||
- Intent alignment summary
|
||||
2. Add operator troubleshooting guides.
|
||||
3. Add integration/API examples for consumers.
|
||||
3. Add integration/API examples for the operator and future maintainers.
|
||||
4. Publish changelog/version notes for V1.
|
||||
|
||||
### Deliverables
|
||||
@@ -260,55 +262,55 @@ For each requirement slice:
|
||||
- V1 release notes
|
||||
|
||||
### Exit Criteria
|
||||
- A new team member can run/support the system using docs alone.
|
||||
- A future maintainer can run and support the system using docs alone.
|
||||
|
||||
---
|
||||
|
||||
## 11) Final Validation, UAT, and Launch
|
||||
## 11) Final Validation and Launch
|
||||
|
||||
**Goal:** Confirm readiness and launch V1 safely.
|
||||
**Goal:** Confirm V1 readiness and launch with low operational risk.
|
||||
|
||||
### Implementation Steps
|
||||
1. Run full-system acceptance validation against the V1 traceability matrix.
|
||||
2. Conduct stakeholder UAT and capture sign-off.
|
||||
3. Execute production readiness review.
|
||||
4. Launch in controlled phases and monitor key signals.
|
||||
1. Run end-to-end acceptance validation against the V1 traceability matrix.
|
||||
2. Complete operator acceptance checks on representative real documents.
|
||||
3. Execute launch checklist (including backup, migration, and rollback readiness).
|
||||
4. Launch and monitor logs/status closely during initial use.
|
||||
|
||||
### Deliverables
|
||||
- UAT/PRR sign-off records
|
||||
- Launch checklist and monitoring plan
|
||||
- Acceptance validation record
|
||||
- Launch checklist completion record
|
||||
|
||||
### Exit Criteria
|
||||
- Stakeholder approval achieved.
|
||||
- Launch metrics are stable within defined thresholds.
|
||||
- V1 requirements are validated.
|
||||
- Initial launch behavior is stable and recoverable.
|
||||
|
||||
---
|
||||
|
||||
## 12) Post-Launch Stabilization (30–60 Days)
|
||||
## 12) Post-Launch Stabilization
|
||||
|
||||
**Goal:** Consolidate V1 in production before major expansion.
|
||||
**Goal:** Address early issues quickly and lock in a reliable V1 baseline.
|
||||
|
||||
### Implementation Steps
|
||||
1. Track incidents, defects, and user feedback.
|
||||
2. Prioritize stabilization fixes with short cycle times.
|
||||
3. Remove temporary flags/mitigations introduced during launch.
|
||||
4. Produce post-launch retrospective and handoff to standard roadmap cadence.
|
||||
1. Track defects and operational pain points observed after launch.
|
||||
2. Prioritize short-cycle stabilization fixes.
|
||||
3. Remove temporary launch-only workarounds when safe.
|
||||
4. Capture a brief retrospective and update the next-phase backlog.
|
||||
|
||||
### Deliverables
|
||||
- Stabilization report
|
||||
- Prioritized backlog update
|
||||
- Stabilization summary
|
||||
- Updated backlog for post-V1 enhancements
|
||||
|
||||
### Exit Criteria
|
||||
- Incident/error rates converge to steady-state targets.
|
||||
- V1 transitions from launch mode to normal operations.
|
||||
- Major launch issues are resolved.
|
||||
- System transitions to steady personal-use operation.
|
||||
|
||||
---
|
||||
|
||||
## Recommended Execution Rhythm
|
||||
|
||||
- **Weekly:** Requirement closure + risk review
|
||||
- **Biweekly:** Release train with quality gates
|
||||
- **Milestone reviews:** After phases 2, 6, 9, and 11
|
||||
- **As needed (small batch releases):** Run release checklist and deploy
|
||||
- **Milestone check-ins:** After phases 2, 6, 9, and 11
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -34,6 +34,12 @@ def _status_for(error: AppError) -> int:
|
||||
def register_error_handlers(app: FastAPI) -> None:
|
||||
"""Register API exception handlers on the app."""
|
||||
|
||||
@app.exception_handler(AccessDeniedError)
|
||||
async def access_denied_handler(_request: Request, exc: AccessDeniedError) -> JSONResponse:
|
||||
envelope = build_error_envelope(exc)
|
||||
headers = {"WWW-Authenticate": "Basic"} if exc.should_challenge else None
|
||||
return JSONResponse(status_code=401, content=envelope.__dict__, headers=headers)
|
||||
|
||||
@app.exception_handler(AppError)
|
||||
async def app_error_handler(_request: Request, exc: AppError) -> JSONResponse:
|
||||
envelope = build_error_envelope(exc)
|
||||
|
||||
@@ -0,0 +1,161 @@
|
||||
"""Functional API routes for jobs, revisions, search, and export."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from uuid import UUID
|
||||
|
||||
from fastapi import APIRouter
|
||||
from pydantic import BaseModel
|
||||
from pydantic import Field
|
||||
|
||||
from transcription.services.library import accept_revision
|
||||
from transcription.services.library import add_revision
|
||||
from transcription.services.library import export_transcripts
|
||||
from transcription.services.library import get_job_detail
|
||||
from transcription.services.library import list_jobs
|
||||
from transcription.services.library import list_revisions
|
||||
from transcription.services.library import search_accepted_transcripts
|
||||
|
||||
router = APIRouter(prefix="/api", tags=["transcription"])
|
||||
|
||||
|
||||
class CreateRevisionRequest(BaseModel):
|
||||
text: str = Field(min_length=1)
|
||||
source: str = "user"
|
||||
accepted: bool = False
|
||||
|
||||
|
||||
@router.get("/jobs")
|
||||
def get_jobs() -> list[dict[str, str]]:
|
||||
jobs = list_jobs()
|
||||
return [
|
||||
{
|
||||
"id": str(job.id),
|
||||
"document_id": str(job.document_id),
|
||||
"status": job.status.value,
|
||||
"created_at": job.created_at.isoformat(),
|
||||
"updated_at": job.updated_at.isoformat(),
|
||||
}
|
||||
for job in jobs
|
||||
]
|
||||
|
||||
|
||||
@router.get("/jobs/{job_id}")
|
||||
def get_job(job_id: UUID) -> dict[str, object | None]:
|
||||
detail = get_job_detail(job_id=job_id)
|
||||
return {
|
||||
"job": {
|
||||
"id": str(detail.job.id),
|
||||
"document_id": str(detail.job.document_id),
|
||||
"status": detail.job.status.value,
|
||||
"created_at": detail.job.created_at.isoformat(),
|
||||
"updated_at": detail.job.updated_at.isoformat(),
|
||||
},
|
||||
"document": (
|
||||
{
|
||||
"id": str(detail.document.id),
|
||||
"filename": detail.document.filename,
|
||||
"file_path": detail.document.file_path,
|
||||
}
|
||||
if detail.document is not None
|
||||
else None
|
||||
),
|
||||
"transcript": (
|
||||
{
|
||||
"id": str(detail.transcript.id),
|
||||
"text": detail.transcript.text,
|
||||
"error_detail": detail.transcript.error_detail,
|
||||
"created_at": detail.transcript.created_at.isoformat(),
|
||||
}
|
||||
if detail.transcript is not None
|
||||
else None
|
||||
),
|
||||
"accepted_revision": (
|
||||
{
|
||||
"id": str(detail.accepted_revision.id),
|
||||
"revision_number": detail.accepted_revision.revision_number,
|
||||
"text": detail.accepted_revision.text,
|
||||
"source": detail.accepted_revision.source,
|
||||
"created_at": detail.accepted_revision.created_at.isoformat(),
|
||||
}
|
||||
if detail.accepted_revision is not None
|
||||
else None
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
@router.get("/jobs/{job_id}/revisions")
|
||||
def get_job_revisions(job_id: UUID) -> list[dict[str, object]]:
|
||||
revisions = list_revisions(job_id=job_id)
|
||||
return [
|
||||
{
|
||||
"id": str(revision.id),
|
||||
"job_id": str(revision.job_id),
|
||||
"revision_number": revision.revision_number,
|
||||
"text": revision.text,
|
||||
"source": revision.source,
|
||||
"accepted": revision.accepted,
|
||||
"created_at": revision.created_at.isoformat(),
|
||||
}
|
||||
for revision in revisions
|
||||
]
|
||||
|
||||
|
||||
@router.post("/jobs/{job_id}/revisions")
|
||||
def create_job_revision(job_id: UUID, payload: CreateRevisionRequest) -> dict[str, object]:
|
||||
revision = add_revision(
|
||||
job_id=job_id,
|
||||
text=payload.text,
|
||||
source=payload.source,
|
||||
accepted=payload.accepted,
|
||||
)
|
||||
return {
|
||||
"id": str(revision.id),
|
||||
"job_id": str(revision.job_id),
|
||||
"revision_number": revision.revision_number,
|
||||
"text": revision.text,
|
||||
"source": revision.source,
|
||||
"accepted": revision.accepted,
|
||||
"created_at": revision.created_at.isoformat(),
|
||||
}
|
||||
|
||||
|
||||
@router.post("/revisions/{revision_id}/accept")
|
||||
def accept_job_revision(revision_id: UUID) -> dict[str, object]:
|
||||
revision = accept_revision(revision_id=revision_id)
|
||||
return {
|
||||
"id": str(revision.id),
|
||||
"job_id": str(revision.job_id),
|
||||
"revision_number": revision.revision_number,
|
||||
"text": revision.text,
|
||||
"source": revision.source,
|
||||
"accepted": revision.accepted,
|
||||
"created_at": revision.created_at.isoformat(),
|
||||
}
|
||||
|
||||
|
||||
@router.get("/search")
|
||||
def search(query: str) -> list[dict[str, object]]:
|
||||
results = search_accepted_transcripts(query=query)
|
||||
return [
|
||||
{
|
||||
"revision_id": str(revision.id),
|
||||
"job_id": str(revision.job_id),
|
||||
"revision_number": revision.revision_number,
|
||||
"text": revision.text,
|
||||
"source": revision.source,
|
||||
"accepted": revision.accepted,
|
||||
"created_at": revision.created_at.isoformat(),
|
||||
}
|
||||
for revision in results
|
||||
]
|
||||
|
||||
|
||||
@router.get("/export")
|
||||
def export(accepted_only: bool = True) -> dict[str, object]:
|
||||
records = export_transcripts(accepted_only=accepted_only)
|
||||
return {
|
||||
"count": len(records),
|
||||
"accepted_only": accepted_only,
|
||||
"records": records,
|
||||
}
|
||||
@@ -75,7 +75,21 @@ async def _lifespan(app: FastAPI):
|
||||
def create_app() -> FastAPI:
|
||||
"""Create and configure the FastAPI application."""
|
||||
app = FastAPI(title="Transcription", lifespan=_lifespan)
|
||||
|
||||
@app.middleware("http")
|
||||
async def operator_access_middleware(request: Request, call_next):
|
||||
settings = get_settings()
|
||||
try:
|
||||
enforce_request_access(request=request, settings=settings)
|
||||
except AccessDeniedError as exc:
|
||||
envelope = build_error_envelope(exc)
|
||||
headers = {"WWW-Authenticate": "Basic"} if exc.should_challenge else None
|
||||
return JSONResponse(status_code=401, content=envelope.__dict__, headers=headers)
|
||||
|
||||
return await call_next(request)
|
||||
|
||||
register_error_handlers(app)
|
||||
register_pages(app)
|
||||
app.include_router(health_router)
|
||||
app.include_router(transcription_router)
|
||||
return app
|
||||
|
||||
@@ -41,11 +41,21 @@ class Settings(BaseSettings):
|
||||
# --- persistence ---
|
||||
database_url: str = "sqlite:///./transcription.db"
|
||||
bootstrap_schema_on_startup: bool | None = None
|
||||
migration_auto_apply_on_startup: bool = False
|
||||
validate_schema_on_startup: bool = True
|
||||
|
||||
# --- filesystem paths ---
|
||||
upload_dir: Path = Path("./uploads")
|
||||
prompt_dir: Path = Path("./prompts")
|
||||
|
||||
# --- upload safety ---
|
||||
max_upload_bytes: int = 15 * 1024 * 1024
|
||||
|
||||
# --- single-operator access control ---
|
||||
operator_access_enabled: bool = False
|
||||
operator_username: str = "operator"
|
||||
operator_password: str | None = None
|
||||
|
||||
# --- worker reliability ---
|
||||
worker_max_retries: int = 0
|
||||
worker_retry_backoff_seconds: float = 0.0
|
||||
|
||||
@@ -114,16 +114,18 @@ async def create_all(*, engine: AsyncEngine | None = None) -> None:
|
||||
def _ensure_sqlite_compat_columns(connection: Connection) -> None:
|
||||
"""Apply lightweight dev/test SQLite compatibility column patches.
|
||||
|
||||
This keeps local bootstrap resilient when models evolve but no full
|
||||
migration tooling is in place yet.
|
||||
This performs read-only validation and never mutates schema.
|
||||
"""
|
||||
if connection.engine.url.get_backend_name() != "sqlite":
|
||||
return
|
||||
|
||||
inspector = inspect(connection)
|
||||
table_names = set(inspector.get_table_names())
|
||||
if "job" not in table_names:
|
||||
return
|
||||
|
||||
required_tables = {"document", "job", "transcript", "transcriptrevision"}
|
||||
missing_tables = sorted(required_tables - table_names)
|
||||
for table_name in missing_tables:
|
||||
issues.append(f"missing_table:{table_name}")
|
||||
|
||||
columns = {column["name"] for column in inspector.get_columns("job")}
|
||||
if "retry_count" not in columns:
|
||||
|
||||
@@ -71,8 +71,9 @@ def build_error_envelope(error: AppError) -> ErrorEnvelope:
|
||||
|
||||
def classify_unexpected_error(exc: Exception, *, operation: str) -> AppError:
|
||||
"""Normalize unknown exceptions into internal_unexpected_error."""
|
||||
_ = exc
|
||||
return AppError(
|
||||
f"Unexpected error during {operation}: {exc}",
|
||||
f"Unexpected error during {operation}",
|
||||
category=ErrorCategory.INTERNAL_UNEXPECTED,
|
||||
suggestion="Retry once. If it persists, review logs and report the error reference id.",
|
||||
retriable=False,
|
||||
|
||||
@@ -0,0 +1,75 @@
|
||||
"""CLI entrypoint for explicit schema migration and compatibility checks."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
|
||||
from transcription.config import get_settings
|
||||
from transcription.db import initialize_database_runtime
|
||||
from transcription.db import validate_schema_compatibility
|
||||
from transcription.migrations import apply_pending_migrations
|
||||
from transcription.migrations import list_pending_migrations
|
||||
|
||||
|
||||
def _build_parser() -> argparse.ArgumentParser:
|
||||
parser = argparse.ArgumentParser(description="Transcription schema migration runner")
|
||||
parser.add_argument(
|
||||
"--apply",
|
||||
action="store_true",
|
||||
help="Apply all pending migrations.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--list",
|
||||
action="store_true",
|
||||
help="List pending migrations.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--check",
|
||||
action="store_true",
|
||||
help="Run schema compatibility check.",
|
||||
)
|
||||
return parser
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = _build_parser()
|
||||
args = parser.parse_args()
|
||||
|
||||
if not (args.apply or args.list or args.check):
|
||||
parser.error("Specify at least one action: --list, --apply, or --check")
|
||||
|
||||
runtime = initialize_database_runtime(settings=get_settings())
|
||||
engine = runtime.engine
|
||||
|
||||
if args.list:
|
||||
pending = list_pending_migrations(engine=engine)
|
||||
if not pending:
|
||||
print("No pending migrations.")
|
||||
else:
|
||||
print("Pending migrations:")
|
||||
for migration in pending:
|
||||
print(f"- {migration.revision_id}: {migration.description}")
|
||||
|
||||
if args.apply:
|
||||
applied = apply_pending_migrations(engine=engine)
|
||||
if not applied:
|
||||
print("No migrations applied.")
|
||||
else:
|
||||
print("Applied migrations:")
|
||||
for revision_id in applied:
|
||||
print(f"- {revision_id}")
|
||||
|
||||
if args.check:
|
||||
issues = validate_schema_compatibility(engine=engine)
|
||||
if issues:
|
||||
print("Schema compatibility check failed:")
|
||||
for issue in issues:
|
||||
print(f"- {issue}")
|
||||
return 1
|
||||
print("Schema compatibility check passed.")
|
||||
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,132 @@
|
||||
"""Lightweight schema migration helpers for V1 Step 4.
|
||||
|
||||
This module provides explicit, operator-invoked migration execution for
|
||||
personal-scale deployments without introducing heavyweight migration tooling.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from collections.abc import Callable
|
||||
from dataclasses import dataclass
|
||||
from datetime import UTC
|
||||
from datetime import datetime
|
||||
|
||||
from sqlalchemy import inspect
|
||||
from sqlalchemy import text
|
||||
from sqlalchemy.engine import Connection
|
||||
from sqlalchemy.engine import Engine
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class MigrationRevision:
|
||||
"""Represents one ordered schema migration revision."""
|
||||
|
||||
revision_id: str
|
||||
description: str
|
||||
apply: Callable[[Connection], None]
|
||||
|
||||
|
||||
def _ensure_history_table(connection: Connection) -> None:
|
||||
"""Create migration history table when missing."""
|
||||
connection.execute(
|
||||
text(
|
||||
"""
|
||||
CREATE TABLE IF NOT EXISTS schema_migration_history (
|
||||
revision_id VARCHAR(64) PRIMARY KEY,
|
||||
description VARCHAR(255) NOT NULL,
|
||||
applied_at VARCHAR(64) NOT NULL
|
||||
)
|
||||
"""
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def _get_applied_revisions(connection: Connection) -> set[str]:
|
||||
"""Return applied migration revision IDs."""
|
||||
_ensure_history_table(connection)
|
||||
rows = connection.execute(text("SELECT revision_id FROM schema_migration_history")).fetchall()
|
||||
return {row[0] for row in rows}
|
||||
|
||||
|
||||
def _record_revision(connection: Connection, revision: MigrationRevision) -> None:
|
||||
"""Persist one applied migration revision record."""
|
||||
connection.execute(
|
||||
text(
|
||||
"""
|
||||
INSERT INTO schema_migration_history (revision_id, description, applied_at)
|
||||
VALUES (:revision_id, :description, :applied_at)
|
||||
"""
|
||||
),
|
||||
{
|
||||
"revision_id": revision.revision_id,
|
||||
"description": revision.description,
|
||||
"applied_at": datetime.now(UTC).isoformat(),
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def _apply_0001_add_retry_count(connection: Connection) -> None:
|
||||
"""Ensure job.retry_count exists for legacy databases."""
|
||||
inspector = inspect(connection)
|
||||
table_names = set(inspector.get_table_names())
|
||||
if "job" not in table_names:
|
||||
return
|
||||
|
||||
columns = {column["name"] for column in inspector.get_columns("job")}
|
||||
if "retry_count" in columns:
|
||||
return
|
||||
|
||||
# Compatible with SQLite and PostgreSQL for this additive integer column.
|
||||
connection.execute(text("ALTER TABLE job ADD COLUMN retry_count INTEGER NOT NULL DEFAULT 0"))
|
||||
|
||||
|
||||
def _apply_0002_create_transcriptrevision(connection: Connection) -> None:
|
||||
"""Ensure transcriptrevision table exists."""
|
||||
# Import models lazily so metadata is fully populated.
|
||||
from sqlmodel import SQLModel
|
||||
|
||||
from transcription.models import TranscriptRevision # noqa: F401
|
||||
|
||||
table = SQLModel.metadata.tables["transcriptrevision"]
|
||||
table.create(bind=connection, checkfirst=True)
|
||||
|
||||
|
||||
MIGRATIONS: tuple[MigrationRevision, ...] = (
|
||||
MigrationRevision(
|
||||
revision_id="0001_add_retry_count_to_job",
|
||||
description="Add retry_count column to job table with default 0",
|
||||
apply=_apply_0001_add_retry_count,
|
||||
),
|
||||
MigrationRevision(
|
||||
revision_id="0002_create_transcriptrevision_table",
|
||||
description="Create transcriptrevision table for immutable transcript history",
|
||||
apply=_apply_0002_create_transcriptrevision,
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def list_pending_migrations(*, engine: Engine) -> list[MigrationRevision]:
|
||||
"""Return pending migrations ordered by revision."""
|
||||
with engine.begin() as connection:
|
||||
applied = _get_applied_revisions(connection)
|
||||
return [revision for revision in MIGRATIONS if revision.revision_id not in applied]
|
||||
|
||||
|
||||
def apply_pending_migrations(*, engine: Engine) -> list[str]:
|
||||
"""Apply all pending migrations and return applied revision IDs."""
|
||||
pending = list_pending_migrations(engine=engine)
|
||||
applied_ids: list[str] = []
|
||||
|
||||
for revision in pending:
|
||||
logger.info("Applying migration revision=%s", revision.revision_id)
|
||||
with engine.begin() as connection:
|
||||
_ensure_history_table(connection)
|
||||
revision.apply(connection)
|
||||
_record_revision(connection, revision)
|
||||
applied_ids.append(revision.revision_id)
|
||||
logger.info("Applied migration revision=%s", revision.revision_id)
|
||||
|
||||
return applied_ids
|
||||
+23
-18
@@ -1,8 +1,4 @@
|
||||
"""SQLModel domain models for the transcription system.
|
||||
|
||||
Three models capture the MVP lifecycle:
|
||||
Document -> one-to-many -> Job -> one-to-one -> Transcript
|
||||
"""
|
||||
"""SQLModel domain models for the transcription system."""
|
||||
|
||||
from datetime import UTC
|
||||
from datetime import datetime
|
||||
@@ -20,6 +16,7 @@ class JobStatus(StrEnum):
|
||||
QUEUED = "queued"
|
||||
PROCESSING = "processing"
|
||||
TRANSCRIBED = "transcribed"
|
||||
COMPLETED = "completed"
|
||||
FAILED = "failed"
|
||||
|
||||
|
||||
@@ -29,9 +26,7 @@ class Document(SQLModel, table=True):
|
||||
id: UUID = Field(default_factory=uuid4, primary_key=True)
|
||||
filename: str
|
||||
file_path: str
|
||||
uploaded_at: datetime = Field(
|
||||
default_factory=lambda: datetime.now(UTC),
|
||||
)
|
||||
uploaded_at: datetime = Field(default_factory=lambda: datetime.now(UTC))
|
||||
|
||||
# --- relationships ---
|
||||
jobs: list["Job"] = Relationship(back_populates="document")
|
||||
@@ -44,28 +39,38 @@ class Job(SQLModel, table=True):
|
||||
document_id: UUID = Field(foreign_key="document.id")
|
||||
status: JobStatus = Field(default=JobStatus.QUEUED)
|
||||
retry_count: int = Field(default=0, ge=0)
|
||||
created_at: datetime = Field(
|
||||
default_factory=lambda: datetime.now(UTC),
|
||||
)
|
||||
updated_at: datetime = Field(
|
||||
default_factory=lambda: datetime.now(UTC),
|
||||
)
|
||||
created_at: datetime = Field(default_factory=lambda: datetime.now(UTC))
|
||||
updated_at: datetime = Field(default_factory=lambda: datetime.now(UTC))
|
||||
|
||||
# --- relationships ---
|
||||
document: Document = Relationship(back_populates="jobs")
|
||||
transcript: Optional["Transcript"] = Relationship(back_populates="job")
|
||||
revisions: list["TranscriptRevision"] = Relationship(back_populates="job")
|
||||
|
||||
|
||||
class Transcript(SQLModel, table=True):
|
||||
"""The output of a transcription job."""
|
||||
"""Canonical transcript state for a job (latest text or failure detail)."""
|
||||
|
||||
id: UUID = Field(default_factory=uuid4, primary_key=True)
|
||||
job_id: UUID = Field(foreign_key="job.id", unique=True)
|
||||
text: str | None = None
|
||||
error_detail: str | None = None
|
||||
created_at: datetime = Field(
|
||||
default_factory=lambda: datetime.now(UTC),
|
||||
)
|
||||
created_at: datetime = Field(default_factory=lambda: datetime.now(UTC))
|
||||
|
||||
# --- relationships ---
|
||||
job: Job = Relationship(back_populates="transcript")
|
||||
|
||||
|
||||
class TranscriptRevision(SQLModel, table=True):
|
||||
"""Immutable transcript revision history for review/acceptance workflows."""
|
||||
|
||||
id: UUID = Field(default_factory=uuid4, primary_key=True)
|
||||
job_id: UUID = Field(foreign_key="job.id", index=True)
|
||||
revision_number: int = Field(ge=1)
|
||||
text: str
|
||||
source: str = Field(default="worker")
|
||||
accepted: bool = Field(default=False)
|
||||
created_at: datetime = Field(default_factory=lambda: datetime.now(UTC))
|
||||
|
||||
# --- relationships ---
|
||||
job: Job = Relationship(back_populates="revisions")
|
||||
|
||||
@@ -0,0 +1,82 @@
|
||||
"""Step 5 single-operator access control helpers."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import binascii
|
||||
import secrets
|
||||
|
||||
from fastapi import Request
|
||||
|
||||
from transcription.config import Settings
|
||||
from transcription.errors import AppError
|
||||
from transcription.errors import ErrorCategory
|
||||
|
||||
|
||||
class AccessDeniedError(AppError):
|
||||
"""Raised when a request is not authorized for operator actions."""
|
||||
|
||||
def __init__(self, message: str, *, suggestion: str, should_challenge: bool = True) -> None:
|
||||
super().__init__(message, category=ErrorCategory.USER_INPUT, suggestion=suggestion)
|
||||
self.should_challenge = should_challenge
|
||||
|
||||
|
||||
def is_protected_path(path: str) -> bool:
|
||||
"""Return True when a request path requires operator authentication."""
|
||||
return path == "/ui" or path.startswith(("/ui/", "/api"))
|
||||
|
||||
|
||||
def enforce_request_access(*, request: Request, settings: Settings) -> None:
|
||||
"""Enforce basic operator access control for protected paths."""
|
||||
if not settings.operator_access_enabled or not is_protected_path(request.url.path):
|
||||
return
|
||||
|
||||
if not settings.operator_password:
|
||||
raise AppError(
|
||||
"Operator authentication is enabled but credentials are not configured",
|
||||
category=ErrorCategory.INFRA_PERSISTENT,
|
||||
suggestion="Set OPERATOR_PASSWORD in the runtime environment and restart the app.",
|
||||
)
|
||||
|
||||
authorization = request.headers.get("Authorization")
|
||||
username, password = _parse_basic_authorization_header(authorization)
|
||||
|
||||
valid_username = secrets.compare_digest(username, settings.operator_username)
|
||||
valid_password = secrets.compare_digest(password, settings.operator_password)
|
||||
if not (valid_username and valid_password):
|
||||
raise AccessDeniedError(
|
||||
"Invalid operator credentials",
|
||||
suggestion="Provide valid operator credentials and retry.",
|
||||
)
|
||||
|
||||
|
||||
def _parse_basic_authorization_header(value: str | None) -> tuple[str, str]:
|
||||
if not value:
|
||||
raise AccessDeniedError(
|
||||
"Operator authentication required",
|
||||
suggestion="Provide HTTP Basic operator credentials and retry.",
|
||||
)
|
||||
|
||||
scheme, _, token = value.partition(" ")
|
||||
if scheme.lower() != "basic" or not token:
|
||||
raise AccessDeniedError(
|
||||
"Operator authentication required",
|
||||
suggestion="Provide HTTP Basic operator credentials and retry.",
|
||||
)
|
||||
|
||||
try:
|
||||
decoded = base64.b64decode(token, validate=True).decode("utf-8")
|
||||
except (binascii.Error, UnicodeDecodeError) as exc:
|
||||
raise AccessDeniedError(
|
||||
"Invalid authentication header",
|
||||
suggestion="Provide HTTP Basic operator credentials and retry.",
|
||||
) from exc
|
||||
|
||||
username, sep, password = decoded.partition(":")
|
||||
if not sep or not username:
|
||||
raise AccessDeniedError(
|
||||
"Invalid authentication header",
|
||||
suggestion="Provide HTTP Basic operator credentials and retry.",
|
||||
)
|
||||
|
||||
return username, password
|
||||
@@ -0,0 +1,264 @@
|
||||
"""Step 3 functional services: job detail, revisions, search, and export."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from datetime import UTC
|
||||
from datetime import datetime
|
||||
from uuid import UUID
|
||||
|
||||
from sqlmodel import Session
|
||||
from sqlmodel import select
|
||||
|
||||
from transcription.db import get_session
|
||||
from transcription.errors import AppError
|
||||
from transcription.errors import ErrorCategory
|
||||
from transcription.models import Document
|
||||
from transcription.models import Job
|
||||
from transcription.models import JobStatus
|
||||
from transcription.models import Transcript
|
||||
from transcription.models import TranscriptRevision
|
||||
|
||||
|
||||
class LibraryError(AppError):
|
||||
"""Base error for review/search/export service pathways."""
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class JobDetail:
|
||||
"""Job detail read model including latest transcript and accepted revision."""
|
||||
|
||||
job: Job
|
||||
document: Document | None
|
||||
transcript: Transcript | None
|
||||
accepted_revision: TranscriptRevision | None
|
||||
|
||||
|
||||
def list_jobs(*, session: Session | None = None) -> list[Job]:
|
||||
"""Return jobs in most-recent-first order."""
|
||||
if session is None:
|
||||
with get_session() as local_session:
|
||||
return list_jobs(session=local_session)
|
||||
|
||||
return list(session.exec(select(Job).order_by(Job.created_at.desc())).all())
|
||||
|
||||
|
||||
def get_job_detail(*, job_id: UUID, session: Session | None = None) -> JobDetail:
|
||||
"""Fetch job detail with related document/transcript and accepted revision."""
|
||||
if session is None:
|
||||
with get_session() as local_session:
|
||||
return get_job_detail(job_id=job_id, session=local_session)
|
||||
|
||||
job = session.get(Job, job_id)
|
||||
if job is None:
|
||||
raise LibraryError(
|
||||
f"Job not found: {job_id}",
|
||||
category=ErrorCategory.NOT_FOUND,
|
||||
suggestion="Refresh jobs list and open a valid job id.",
|
||||
)
|
||||
|
||||
document = session.get(Document, job.document_id)
|
||||
transcript = session.exec(select(Transcript).where(Transcript.job_id == job.id)).first()
|
||||
accepted_revision = session.exec(
|
||||
select(TranscriptRevision)
|
||||
.where(TranscriptRevision.job_id == job.id, TranscriptRevision.accepted.is_(True))
|
||||
.order_by(TranscriptRevision.revision_number.desc())
|
||||
).first()
|
||||
|
||||
return JobDetail(
|
||||
job=job,
|
||||
document=document,
|
||||
transcript=transcript,
|
||||
accepted_revision=accepted_revision,
|
||||
)
|
||||
|
||||
|
||||
def add_revision(
|
||||
*,
|
||||
job_id: UUID,
|
||||
text: str,
|
||||
source: str = "user",
|
||||
accepted: bool = False,
|
||||
session: Session | None = None,
|
||||
) -> TranscriptRevision:
|
||||
"""Append a transcript revision and optionally mark it as accepted."""
|
||||
if not text.strip():
|
||||
raise LibraryError(
|
||||
"Revision text cannot be empty",
|
||||
category=ErrorCategory.VALIDATION,
|
||||
suggestion="Provide non-empty transcript text and retry.",
|
||||
)
|
||||
|
||||
if session is None:
|
||||
with get_session() as local_session:
|
||||
return add_revision(
|
||||
job_id=job_id,
|
||||
text=text,
|
||||
source=source,
|
||||
accepted=accepted,
|
||||
session=local_session,
|
||||
)
|
||||
|
||||
job = session.get(Job, job_id)
|
||||
if job is None:
|
||||
raise LibraryError(
|
||||
f"Job not found: {job_id}",
|
||||
category=ErrorCategory.NOT_FOUND,
|
||||
suggestion="Refresh jobs list and retry with a valid job id.",
|
||||
)
|
||||
|
||||
revisions = list(
|
||||
session.exec(
|
||||
select(TranscriptRevision)
|
||||
.where(TranscriptRevision.job_id == job_id)
|
||||
.order_by(TranscriptRevision.revision_number)
|
||||
).all()
|
||||
)
|
||||
next_revision_number = (revisions[-1].revision_number + 1) if revisions else 1
|
||||
|
||||
if accepted:
|
||||
for existing in revisions:
|
||||
if existing.accepted:
|
||||
existing.accepted = False
|
||||
session.add(existing)
|
||||
|
||||
revision = TranscriptRevision(
|
||||
job_id=job_id,
|
||||
revision_number=next_revision_number,
|
||||
text=text,
|
||||
source=source,
|
||||
accepted=accepted,
|
||||
)
|
||||
session.add(revision)
|
||||
|
||||
transcript = session.exec(select(Transcript).where(Transcript.job_id == job_id)).first()
|
||||
if transcript is None:
|
||||
transcript = Transcript(job_id=job_id)
|
||||
|
||||
transcript.text = text
|
||||
transcript.error_detail = None
|
||||
session.add(transcript)
|
||||
|
||||
job.updated_at = datetime.now(UTC)
|
||||
if accepted:
|
||||
job.status = JobStatus.COMPLETED
|
||||
elif job.status == JobStatus.QUEUED:
|
||||
job.status = JobStatus.TRANSCRIBED
|
||||
session.add(job)
|
||||
|
||||
session.commit()
|
||||
session.refresh(revision)
|
||||
return revision
|
||||
|
||||
|
||||
def accept_revision(*, revision_id: UUID, session: Session | None = None) -> TranscriptRevision:
|
||||
"""Mark one revision as accepted and synchronize canonical transcript/job state."""
|
||||
if session is None:
|
||||
with get_session() as local_session:
|
||||
return accept_revision(revision_id=revision_id, session=local_session)
|
||||
|
||||
revision = session.get(TranscriptRevision, revision_id)
|
||||
if revision is None:
|
||||
raise LibraryError(
|
||||
f"Revision not found: {revision_id}",
|
||||
category=ErrorCategory.NOT_FOUND,
|
||||
suggestion="Refresh job detail and select a valid revision.",
|
||||
)
|
||||
|
||||
all_revisions = list(session.exec(select(TranscriptRevision).where(TranscriptRevision.job_id == revision.job_id)).all())
|
||||
for item in all_revisions:
|
||||
item.accepted = item.id == revision.id
|
||||
session.add(item)
|
||||
|
||||
transcript = session.exec(select(Transcript).where(Transcript.job_id == revision.job_id)).first()
|
||||
if transcript is None:
|
||||
transcript = Transcript(job_id=revision.job_id)
|
||||
|
||||
transcript.text = revision.text
|
||||
transcript.error_detail = None
|
||||
session.add(transcript)
|
||||
|
||||
job = session.get(Job, revision.job_id)
|
||||
if job is not None:
|
||||
job.status = JobStatus.COMPLETED
|
||||
job.updated_at = datetime.now(UTC)
|
||||
session.add(job)
|
||||
|
||||
session.commit()
|
||||
session.refresh(revision)
|
||||
return revision
|
||||
|
||||
|
||||
def list_revisions(*, job_id: UUID, session: Session | None = None) -> list[TranscriptRevision]:
|
||||
"""Return revision history for a job in ascending revision order."""
|
||||
if session is None:
|
||||
with get_session() as local_session:
|
||||
return list_revisions(job_id=job_id, session=local_session)
|
||||
|
||||
if session.get(Job, job_id) is None:
|
||||
raise LibraryError(
|
||||
f"Job not found: {job_id}",
|
||||
category=ErrorCategory.NOT_FOUND,
|
||||
suggestion="Refresh jobs list and open a valid job id.",
|
||||
)
|
||||
|
||||
return list(
|
||||
session.exec(
|
||||
select(TranscriptRevision)
|
||||
.where(TranscriptRevision.job_id == job_id)
|
||||
.order_by(TranscriptRevision.revision_number)
|
||||
).all()
|
||||
)
|
||||
|
||||
|
||||
def search_accepted_transcripts(*, query: str, session: Session | None = None) -> list[TranscriptRevision]:
|
||||
"""Search accepted transcript revisions using case-insensitive text containment."""
|
||||
if not query.strip():
|
||||
raise LibraryError(
|
||||
"Search query cannot be empty",
|
||||
category=ErrorCategory.VALIDATION,
|
||||
suggestion="Enter a non-empty search query and retry.",
|
||||
)
|
||||
|
||||
if session is None:
|
||||
with get_session() as local_session:
|
||||
return search_accepted_transcripts(query=query, session=local_session)
|
||||
|
||||
pattern = f"%{query.strip()}%"
|
||||
return list(
|
||||
session.exec(
|
||||
select(TranscriptRevision)
|
||||
.where(TranscriptRevision.accepted.is_(True), TranscriptRevision.text.ilike(pattern))
|
||||
.order_by(TranscriptRevision.created_at.desc())
|
||||
).all()
|
||||
)
|
||||
|
||||
|
||||
def export_transcripts(*, accepted_only: bool = True, session: Session | None = None) -> list[dict[str, str | int | None]]:
|
||||
"""Export transcript data as serializable records for archive workflows."""
|
||||
if session is None:
|
||||
with get_session() as local_session:
|
||||
return export_transcripts(accepted_only=accepted_only, session=local_session)
|
||||
|
||||
statement = select(TranscriptRevision).order_by(TranscriptRevision.created_at)
|
||||
if accepted_only:
|
||||
statement = statement.where(TranscriptRevision.accepted.is_(True))
|
||||
|
||||
revisions = list(session.exec(statement).all())
|
||||
payload: list[dict[str, str | int | None]] = []
|
||||
for revision in revisions:
|
||||
detail = get_job_detail(job_id=revision.job_id, session=session)
|
||||
payload.append(
|
||||
{
|
||||
"job_id": str(revision.job_id),
|
||||
"document_id": str(detail.job.document_id),
|
||||
"filename": detail.document.filename if detail.document else None,
|
||||
"revision_id": str(revision.id),
|
||||
"revision_number": revision.revision_number,
|
||||
"accepted": revision.accepted,
|
||||
"source": revision.source,
|
||||
"text": revision.text,
|
||||
"created_at": revision.created_at.isoformat(),
|
||||
}
|
||||
)
|
||||
return payload
|
||||
@@ -47,7 +47,11 @@ async def create_upload_job(
|
||||
) -> UploadJobResult:
|
||||
"""Persist an uploaded file and create document/job records."""
|
||||
runtime_settings = settings or get_settings()
|
||||
_validate_upload(filename=filename, file_bytes=file_bytes)
|
||||
_validate_upload(
|
||||
filename=filename,
|
||||
file_bytes=file_bytes,
|
||||
max_upload_bytes=runtime_settings.max_upload_bytes,
|
||||
)
|
||||
|
||||
upload_dir = runtime_settings.upload_dir
|
||||
upload_dir.mkdir(parents=True, exist_ok=True)
|
||||
@@ -96,7 +100,7 @@ async def create_upload_job(
|
||||
)
|
||||
|
||||
|
||||
def _validate_upload(*, filename: str, file_bytes: bytes) -> None:
|
||||
def _validate_upload(*, filename: str, file_bytes: bytes, max_upload_bytes: int) -> None:
|
||||
if not file_bytes:
|
||||
raise UploadError(
|
||||
"Upload payload is empty",
|
||||
@@ -104,6 +108,13 @@ def _validate_upload(*, filename: str, file_bytes: bytes) -> None:
|
||||
suggestion="Select a non-empty file and try again.",
|
||||
)
|
||||
|
||||
if len(file_bytes) > max_upload_bytes:
|
||||
raise UploadError(
|
||||
f"Upload exceeds maximum allowed size ({max_upload_bytes} bytes)",
|
||||
category=ErrorCategory.USER_INPUT,
|
||||
suggestion="Upload a smaller file or increase MAX_UPLOAD_BYTES for this deployment.",
|
||||
)
|
||||
|
||||
safe_name = Path(filename).name
|
||||
if not safe_name:
|
||||
raise UploadError(
|
||||
|
||||
@@ -0,0 +1,245 @@
|
||||
"""Jobs list and detail page registration."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from uuid import UUID
|
||||
|
||||
from nicegui import ui
|
||||
from sqlmodel import select
|
||||
|
||||
from transcription.db import get_session
|
||||
from transcription.models import Document
|
||||
from transcription.models import Job
|
||||
from transcription.models import Transcript
|
||||
from transcription.services.library import accept_revision
|
||||
from transcription.services.library import add_revision
|
||||
from transcription.services.library import export_transcripts
|
||||
from transcription.services.library import list_revisions
|
||||
from transcription.services.library import search_accepted_transcripts
|
||||
from transcription.ui.error_presenter import show_error
|
||||
from transcription.ui.error_presenter import summarize_error
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class JobView:
|
||||
"""Read model for rendering job rows in the UI."""
|
||||
|
||||
id: UUID
|
||||
status: str
|
||||
created_at: str
|
||||
updated_at: str
|
||||
|
||||
|
||||
def fetch_jobs() -> list[JobView]:
|
||||
"""Return jobs for display in most-recent-first order."""
|
||||
with get_session() as session:
|
||||
jobs = session.exec(select(Job).order_by(Job.created_at.desc())).all()
|
||||
return [
|
||||
JobView(
|
||||
id=job.id,
|
||||
status=job.status.value,
|
||||
created_at=job.created_at.isoformat(),
|
||||
updated_at=job.updated_at.isoformat(),
|
||||
)
|
||||
for job in jobs
|
||||
]
|
||||
|
||||
|
||||
def fetch_job_detail(job_id: UUID) -> tuple[Job | None, Document | None, Transcript | None]:
|
||||
"""Return job, document, and transcript for detail view."""
|
||||
with get_session() as session:
|
||||
job = session.get(Job, job_id)
|
||||
if job is None:
|
||||
return None, None, None
|
||||
document = session.get(Document, job.document_id)
|
||||
transcript = session.exec(select(Transcript).where(Transcript.job_id == job.id)).first()
|
||||
return job, document, transcript
|
||||
|
||||
|
||||
def register_page() -> None:
|
||||
"""Register jobs list and detail routes."""
|
||||
|
||||
@ui.page("/jobs")
|
||||
def jobs_page() -> None:
|
||||
ui.label("Transcription Jobs")
|
||||
status = ui.label("Ready")
|
||||
table_container = ui.column()
|
||||
|
||||
def render_table() -> None:
|
||||
table_container.clear()
|
||||
jobs = fetch_jobs()
|
||||
with table_container:
|
||||
if not jobs:
|
||||
ui.label("No jobs yet.")
|
||||
return
|
||||
rows = [
|
||||
{
|
||||
"id": str(job.id),
|
||||
"status": job.status,
|
||||
"created_at": job.created_at,
|
||||
"updated_at": job.updated_at,
|
||||
}
|
||||
for job in jobs
|
||||
]
|
||||
ui.table(
|
||||
columns=[
|
||||
{"name": "id", "label": "Job ID", "field": "id"},
|
||||
{"name": "status", "label": "Status", "field": "status"},
|
||||
{"name": "created_at", "label": "Created", "field": "created_at"},
|
||||
{"name": "updated_at", "label": "Updated", "field": "updated_at"},
|
||||
],
|
||||
rows=rows,
|
||||
row_key="id",
|
||||
)
|
||||
for row in rows:
|
||||
ui.link(f"Open {row['id']}", f"/jobs/{row['id']}")
|
||||
|
||||
def refresh() -> None:
|
||||
status.text = "Refreshing..."
|
||||
try:
|
||||
render_table()
|
||||
status.text = "Refreshed"
|
||||
except Exception as exc: # noqa: BLE001
|
||||
status.text = f"Refresh failed: {summarize_error(exc, operation='jobs.refresh')}"
|
||||
show_error(exc, title="Jobs refresh failed", operation="jobs.refresh")
|
||||
|
||||
ui.button("Refresh", on_click=refresh)
|
||||
render_table()
|
||||
ui.link("Back to upload", "/")
|
||||
|
||||
@ui.page("/jobs/{job_id}")
|
||||
def job_detail_page(job_id: str) -> None:
|
||||
ui.label("Job Detail")
|
||||
try:
|
||||
parsed_id = UUID(job_id)
|
||||
except ValueError:
|
||||
ui.label("Invalid job id")
|
||||
ui.link("Back to jobs", "/jobs")
|
||||
return
|
||||
|
||||
job, document, transcript = fetch_job_detail(parsed_id)
|
||||
if job is None:
|
||||
ui.label("Job not found")
|
||||
ui.link("Back to jobs", "/jobs")
|
||||
return
|
||||
|
||||
ui.label(f"Job ID: {job.id}")
|
||||
ui.label(f"Status: {job.status.value}")
|
||||
ui.label(f"Created: {job.created_at.isoformat()}")
|
||||
ui.label(f"Updated: {job.updated_at.isoformat()}")
|
||||
|
||||
if document is not None:
|
||||
ui.label(f"Filename: {document.filename}")
|
||||
ui.label(f"File path: {document.file_path}")
|
||||
|
||||
if transcript is None:
|
||||
ui.label("Transcript not available yet.")
|
||||
elif transcript.text:
|
||||
ui.label("Transcript:")
|
||||
ui.markdown(transcript.text)
|
||||
elif transcript.error_detail:
|
||||
ui.label("Failure detail:")
|
||||
ui.label(transcript.error_detail)
|
||||
|
||||
ui.separator()
|
||||
ui.label("Revision History")
|
||||
revisions_container = ui.column()
|
||||
|
||||
def render_revisions() -> None:
|
||||
revisions_container.clear()
|
||||
with revisions_container:
|
||||
revisions = list_revisions(job_id=parsed_id)
|
||||
if not revisions:
|
||||
ui.label("No revisions yet.")
|
||||
return
|
||||
|
||||
for revision in revisions:
|
||||
with ui.card().classes("w-full"):
|
||||
ui.label(
|
||||
f"Revision {revision.revision_number} | source={revision.source} | accepted={revision.accepted}"
|
||||
)
|
||||
ui.markdown(revision.text)
|
||||
|
||||
if not revision.accepted:
|
||||
ui.button(
|
||||
"Accept revision",
|
||||
on_click=lambda rev_id=revision.id: _accept_revision(rev_id),
|
||||
)
|
||||
|
||||
def _accept_revision(revision_id):
|
||||
try:
|
||||
accept_revision(revision_id=revision_id)
|
||||
ui.notify("Revision accepted", type="positive")
|
||||
render_revisions()
|
||||
except Exception as exc: # noqa: BLE001
|
||||
show_error(exc, title="Accept revision failed", operation="revisions.accept")
|
||||
|
||||
new_revision_text = ui.textarea("Add revision text").props("rows=6")
|
||||
|
||||
def _submit_revision() -> None:
|
||||
try:
|
||||
add_revision(job_id=parsed_id, text=new_revision_text.value or "", source="user", accepted=False)
|
||||
new_revision_text.value = ""
|
||||
ui.notify("Revision added", type="positive")
|
||||
render_revisions()
|
||||
except Exception as exc: # noqa: BLE001
|
||||
show_error(exc, title="Add revision failed", operation="revisions.create")
|
||||
|
||||
ui.button("Add revision", on_click=_submit_revision)
|
||||
|
||||
render_revisions()
|
||||
|
||||
ui.link("Search transcripts", "/search")
|
||||
ui.link("Export transcripts", "/export")
|
||||
ui.link("Back to jobs", "/jobs")
|
||||
|
||||
@ui.page("/search")
|
||||
def search_page() -> None:
|
||||
ui.label("Search Accepted Transcripts")
|
||||
query_input = ui.input("Search query")
|
||||
results_container = ui.column()
|
||||
|
||||
def run_search() -> None:
|
||||
results_container.clear()
|
||||
try:
|
||||
results = search_accepted_transcripts(query=query_input.value or "")
|
||||
except Exception as exc: # noqa: BLE001
|
||||
show_error(exc, title="Search failed", operation="search.run")
|
||||
return
|
||||
|
||||
with results_container:
|
||||
if not results:
|
||||
ui.label("No results.")
|
||||
return
|
||||
|
||||
for result in results:
|
||||
with ui.card().classes("w-full"):
|
||||
ui.label(f"Job {result.job_id} | Revision {result.revision_number}")
|
||||
ui.markdown(result.text)
|
||||
|
||||
ui.button("Search", on_click=run_search)
|
||||
ui.link("Back to jobs", "/jobs")
|
||||
|
||||
@ui.page("/export")
|
||||
def export_page() -> None:
|
||||
ui.label("Export Accepted Transcripts")
|
||||
results_container = ui.column()
|
||||
|
||||
def run_export() -> None:
|
||||
results_container.clear()
|
||||
try:
|
||||
records = export_transcripts(accepted_only=True)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
show_error(exc, title="Export failed", operation="export.run")
|
||||
return
|
||||
|
||||
with results_container:
|
||||
ui.label(f"Exported records: {len(records)}")
|
||||
for record in records:
|
||||
with ui.card().classes("w-full"):
|
||||
ui.label(f"{record['filename']} | Revision {record['revision_number']}")
|
||||
ui.markdown(str(record["text"]))
|
||||
|
||||
ui.button("Run export", on_click=run_export)
|
||||
ui.link("Back to jobs", "/jobs")
|
||||
@@ -81,10 +81,11 @@ async def _process_next_queued_job(*, session: AsyncSession) -> bool:
|
||||
session.add(job)
|
||||
await session.commit()
|
||||
logger.info(
|
||||
"Job transcribed operation=worker.process_job job_id=%s document_id=%s provider=%s",
|
||||
"Job transcribed operation=worker.process_job job_id=%s document_id=%s provider=%s revision_number=%s",
|
||||
job.id,
|
||||
document.id,
|
||||
result.provider,
|
||||
revision.revision_number,
|
||||
)
|
||||
except Exception as exc:
|
||||
error = exc if isinstance(exc, AppError) else classify_unexpected_error(exc, operation="worker.process_job")
|
||||
|
||||
@@ -0,0 +1,129 @@
|
||||
"""Tests for Step 5 operator access control behavior."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
from types import SimpleNamespace
|
||||
|
||||
import pytest
|
||||
from fastapi import FastAPI
|
||||
from fastapi.responses import JSONResponse
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from transcription.api.errors import register_error_handlers
|
||||
from transcription.errors import build_error_envelope
|
||||
from transcription.security import AccessDeniedError
|
||||
from transcription.security import enforce_request_access
|
||||
|
||||
|
||||
def _basic_header(username: str, password: str) -> str:
|
||||
token = base64.b64encode(f"{username}:{password}".encode()).decode("ascii")
|
||||
return f"Basic {token}"
|
||||
|
||||
|
||||
def _build_app(*, settings) -> FastAPI:
|
||||
app = FastAPI()
|
||||
register_error_handlers(app)
|
||||
|
||||
@app.middleware("http")
|
||||
async def operator_access_middleware(request, call_next):
|
||||
try:
|
||||
enforce_request_access(request=request, settings=settings)
|
||||
except AccessDeniedError as exc:
|
||||
envelope = build_error_envelope(exc)
|
||||
headers = {"WWW-Authenticate": "Basic"} if exc.should_challenge else None
|
||||
return JSONResponse(status_code=401, content=envelope.__dict__, headers=headers)
|
||||
return await call_next(request)
|
||||
|
||||
@app.get("/healthz")
|
||||
def healthz():
|
||||
return {"status": "ok"}
|
||||
|
||||
@app.get("/api/jobs")
|
||||
def get_jobs():
|
||||
return [{"id": "demo"}]
|
||||
|
||||
@app.get("/ui")
|
||||
def ui_root():
|
||||
return {"ok": True}
|
||||
|
||||
return app
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
class TestAccessControl:
|
||||
"""Verify protected routes enforce operator auth when enabled."""
|
||||
|
||||
def test_protected_api_requires_credentials(self):
|
||||
settings = SimpleNamespace(
|
||||
operator_access_enabled=True,
|
||||
operator_username="operator",
|
||||
operator_password="secret",
|
||||
)
|
||||
client = TestClient(_build_app(settings=settings), raise_server_exceptions=False)
|
||||
|
||||
response = client.get("/api/jobs")
|
||||
|
||||
assert response.status_code == 401
|
||||
assert response.headers.get("WWW-Authenticate") == "Basic"
|
||||
payload = response.json()
|
||||
assert payload["category"] == "user_input_error"
|
||||
assert payload["suggestion"]
|
||||
|
||||
def test_protected_api_rejects_invalid_credentials(self):
|
||||
settings = SimpleNamespace(
|
||||
operator_access_enabled=True,
|
||||
operator_username="operator",
|
||||
operator_password="secret",
|
||||
)
|
||||
client = TestClient(_build_app(settings=settings), raise_server_exceptions=False)
|
||||
|
||||
response = client.get(
|
||||
"/api/jobs",
|
||||
headers={"Authorization": _basic_header("operator", "wrong")},
|
||||
)
|
||||
|
||||
assert response.status_code == 401
|
||||
payload = response.json()
|
||||
assert payload["message"] == "Invalid operator credentials"
|
||||
|
||||
def test_protected_api_allows_valid_credentials(self):
|
||||
settings = SimpleNamespace(
|
||||
operator_access_enabled=True,
|
||||
operator_username="operator",
|
||||
operator_password="secret",
|
||||
)
|
||||
client = TestClient(_build_app(settings=settings), raise_server_exceptions=False)
|
||||
|
||||
response = client.get(
|
||||
"/api/jobs",
|
||||
headers={"Authorization": _basic_header("operator", "secret")},
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
assert response.json() == [{"id": "demo"}]
|
||||
|
||||
def test_protected_ui_path_requires_credentials(self):
|
||||
settings = SimpleNamespace(
|
||||
operator_access_enabled=True,
|
||||
operator_username="operator",
|
||||
operator_password="secret",
|
||||
)
|
||||
client = TestClient(_build_app(settings=settings), raise_server_exceptions=False)
|
||||
|
||||
response = client.get("/ui")
|
||||
|
||||
assert response.status_code == 401
|
||||
|
||||
def test_healthz_is_not_protected(self):
|
||||
settings = SimpleNamespace(
|
||||
operator_access_enabled=True,
|
||||
operator_username="operator",
|
||||
operator_password="secret",
|
||||
)
|
||||
client = TestClient(_build_app(settings=settings), raise_server_exceptions=False)
|
||||
|
||||
response = client.get("/healthz")
|
||||
|
||||
assert response.status_code == 200
|
||||
assert response.json() == {"status": "ok"}
|
||||
@@ -1,11 +1,12 @@
|
||||
"""Tests for API error response envelope handlers."""
|
||||
|
||||
import pytest
|
||||
from fastapi import FastAPI
|
||||
from fastapi.testclient import TestClient
|
||||
import pytest
|
||||
|
||||
from transcription.api.errors import register_error_handlers
|
||||
from transcription.errors import AppError, ErrorCategory
|
||||
from transcription.errors import AppError
|
||||
from transcription.errors import ErrorCategory
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
|
||||
@@ -0,0 +1,119 @@
|
||||
"""Tests for Step 3 functional API routes."""
|
||||
|
||||
from datetime import UTC
|
||||
from datetime import datetime
|
||||
from types import SimpleNamespace
|
||||
from uuid import uuid4
|
||||
|
||||
import pytest
|
||||
from fastapi import FastAPI
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from transcription.api.errors import register_error_handlers
|
||||
from transcription.api.routes import router
|
||||
|
||||
|
||||
def _build_app() -> FastAPI:
|
||||
app = FastAPI()
|
||||
register_error_handlers(app)
|
||||
app.include_router(router)
|
||||
return app
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
class TestFunctionalRoutes:
|
||||
"""Verify jobs/revisions/search/export route behavior."""
|
||||
|
||||
def test_get_jobs_returns_serialized_rows(self, monkeypatch):
|
||||
"""GET /api/jobs returns normalized job rows."""
|
||||
now = datetime.now(UTC)
|
||||
job = SimpleNamespace(
|
||||
id=uuid4(),
|
||||
document_id=uuid4(),
|
||||
status=SimpleNamespace(value="queued"),
|
||||
created_at=now,
|
||||
updated_at=now,
|
||||
)
|
||||
monkeypatch.setattr("transcription.api.routes.list_jobs", lambda: [job])
|
||||
|
||||
client = TestClient(_build_app())
|
||||
response = client.get("/api/jobs")
|
||||
|
||||
assert response.status_code == 200
|
||||
payload = response.json()
|
||||
assert len(payload) == 1
|
||||
assert payload[0]["id"] == str(job.id)
|
||||
assert payload[0]["status"] == "queued"
|
||||
|
||||
def test_create_revision_returns_revision_payload(self, monkeypatch):
|
||||
"""POST /api/jobs/{job_id}/revisions returns created revision fields."""
|
||||
revision = SimpleNamespace(
|
||||
id=uuid4(),
|
||||
job_id=uuid4(),
|
||||
revision_number=2,
|
||||
text="edited text",
|
||||
source="user",
|
||||
accepted=False,
|
||||
created_at=datetime.now(UTC),
|
||||
)
|
||||
monkeypatch.setattr("transcription.api.routes.add_revision", lambda **_kwargs: revision)
|
||||
|
||||
client = TestClient(_build_app())
|
||||
response = client.post(
|
||||
f"/api/jobs/{revision.job_id}/revisions",
|
||||
json={"text": "edited text", "source": "user", "accepted": False},
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
payload = response.json()
|
||||
assert payload["id"] == str(revision.id)
|
||||
assert payload["revision_number"] == 2
|
||||
assert payload["text"] == "edited text"
|
||||
|
||||
def test_search_returns_results(self, monkeypatch):
|
||||
"""GET /api/search returns accepted transcript matches."""
|
||||
result = SimpleNamespace(
|
||||
id=uuid4(),
|
||||
job_id=uuid4(),
|
||||
revision_number=1,
|
||||
text="family archive",
|
||||
source="user",
|
||||
accepted=True,
|
||||
created_at=datetime.now(UTC),
|
||||
)
|
||||
monkeypatch.setattr("transcription.api.routes.search_accepted_transcripts", lambda query: [result])
|
||||
|
||||
client = TestClient(_build_app())
|
||||
response = client.get("/api/search", params={"query": "archive"})
|
||||
|
||||
assert response.status_code == 200
|
||||
payload = response.json()
|
||||
assert len(payload) == 1
|
||||
assert payload[0]["revision_id"] == str(result.id)
|
||||
assert payload[0]["accepted"] is True
|
||||
|
||||
def test_export_returns_count_and_records(self, monkeypatch):
|
||||
"""GET /api/export returns record count and payload list."""
|
||||
records = [
|
||||
{
|
||||
"job_id": str(uuid4()),
|
||||
"document_id": str(uuid4()),
|
||||
"filename": "letter.jpg",
|
||||
"revision_id": str(uuid4()),
|
||||
"revision_number": 1,
|
||||
"accepted": True,
|
||||
"source": "user",
|
||||
"text": "exported",
|
||||
"created_at": datetime.now(UTC).isoformat(),
|
||||
}
|
||||
]
|
||||
monkeypatch.setattr("transcription.api.routes.export_transcripts", lambda accepted_only=True: records)
|
||||
|
||||
client = TestClient(_build_app())
|
||||
response = client.get("/api/export")
|
||||
|
||||
assert response.status_code == 200
|
||||
payload = response.json()
|
||||
assert payload["count"] == 1
|
||||
assert payload["accepted_only"] is True
|
||||
assert payload["records"] == records
|
||||
@@ -11,11 +11,11 @@ BOOK 1 had 54 pages; 14 chapters. BOOK 2 has 70 pages; 18 chapters. BOOK 1 con-
|
||||
sisted largely of first generation family history. BOOK 2 throws more light on
|
||||
the second generation. Sidney promises a BOOK 3 and that may begin to do justice
|
||||
to the third generation. We suggest that Sidney get the help of Louis Shinn
|
||||
who has a chapter in this book (Chapter 16 - The Last 25 Years on the Doumeeq
|
||||
who has a chapter in this book (Chapter 16 - The Last 25 Years on the Doumecq
|
||||
Plains. Louis has the gift of seeing, recalling and telling. One sentence in
|
||||
his chapter gives a great tribute to the Doumeeqers - so far as he knows no one
|
||||
on the Doumeeq Plains went on relief during the depression. That in a nutshell
|
||||
shows the sturdy character of the residents of the Doumeeq Plains.
|
||||
his chapter gives a great tribute to the Doumecqers--so far as he knows no one
|
||||
on the Doumecq Plains went on relief during the depression. That in a nutshell
|
||||
shows the sturdy character of the residents of the Doumecq Plains.
|
||||
|
||||
We promised in BOOK 1 that in BOOK 2 we would give the story of the trip of
|
||||
John E. Cochran and wife to Tennessee, Cuba and the Panama Canal. You will see
|
||||
@@ -32,7 +32,7 @@ enough pictures but we had to take only part of them. We think there are great
|
||||
possibilities in reproducing old pictures. We wish we had a Pickard group. Some
|
||||
Pickard descendant may wish to make a collection.
|
||||
|
||||
We are much impressed with the future possibilities of getting a complete geneol-
|
||||
We are much impressed with the future possibilities of getting a complete geneol-[sic]
|
||||
ogy of the Pickard family. Mr. Cochran has a fine chapter on the Pickards but
|
||||
to date we have not had the pleasure of finding all of the family dates. We had
|
||||
intended to give more family data in this book but it takes time to get the
|
||||
|
||||
@@ -6,9 +6,7 @@ JOHN E. COCHRAN
|
||||
FAMILY ASSOCIATION
|
||||
Family Only
|
||||
Home | Sibling's Stories | 1st Cousins | Ancestor's Stories | Reunion History | Next Reunion | JECMEF
|
||||
|
||||
OMIE WRITES HOME
|
||||
|
||||
Ed. Note: The following letter was written by Omie Cochran in Nome, Alaska and sent to her
|
||||
sister, Ethel Shinn, in Canfield, Idaho in 1923. It has been stored away these 63 years in the
|
||||
original envelope with its 2 cent stamp. The letter has a number of references to the Shinn
|
||||
@@ -17,6 +15,7 @@ Miss Saville was the nurse at the Nome Hospital that was mentioned in the articl
|
||||
the family newsletter two years ago.
|
||||
|
||||
Nome Alaska August 26, 1923
|
||||
|
||||
My Dear Ethel et al.
|
||||
|
||||
I don't know when I did write or when you did
|
||||
@@ -27,7 +26,7 @@ and Polly sit up and listen and that little black
|
||||
rascal of yours would fairly sparkle with
|
||||
listening. Can't I see him listening now to all the
|
||||
yarns we told last summer?
|
||||
|
||||
[photo of people on ice with kayak and dog sled]
|
||||
You see, we-Miss Saville and I, took a trip north
|
||||
on the Buford and it was very interesting. We
|
||||
went north thru the Bering Strait into the Arctic and as far as the Ice Pack. There the captain
|
||||
@@ -38,19 +37,18 @@ along the icebergs and the walrus were 'heisted' on board by the use of the cran
|
||||
tons of freight and the beasts were so huge that they made the pulleys just creak. They were
|
||||
over 12 feet long, as big around as three cows, had no feet but toenails on their flippers or
|
||||
flappers, no head but their body just suddenly ended with a hole for a mouth and big bristles
|
||||
|
||||
all around it similar to a currycomb in coarseness; no ears but huge tusks of ivory. They are
|
||||
the most repulsive looking animals imaginable and tho I have always read about them I never
|
||||
expect such disagreeable looking creatures. They had a rough brown hairy skin and some of
|
||||
them looked warty. They must have weighed two ton at least. Ere we got them back to Nome
|
||||
to the natives they were getting extremely odiferous–in fact, you could scarcely stay on the
|
||||
to the natives they were getting extremely odiferous—in fact, you could scarcely stay on the
|
||||
ship with any degree of comfort unless you had per chance lost your sense of smell.
|
||||
|
||||
Then we went north to a few minutes beyond the 70th degree of latitude and thot [sic] for awhile
|
||||
we would go to Wrangell Island where some men from Stefflonsons [sic] ship were supposed to be
|
||||
stranded but we didn't get there and instead stopped at a small native village at Cape Serdz [sic] in
|
||||
Then we went north to a few minutes beyond the 70th degree of latitude and thot for awhile
|
||||
we would go to Wrangell Island where some men from Steffonsons ship were supposed to be
|
||||
stranded but we didn't get there and instead stopped at a small native village at Cape Serdz in
|
||||
Siberia. These Eskimo were very primitive. One white squaw man lived there and had for 23
|
||||
years. He was a Swede–who else could. Their houses were circular and built up with dirt 2 or
|
||||
years. He was a Swede--who else could. Their houses were circular and built up with dirt 2 or
|
||||
3 feet and then skins were stretched over it and weighted down with rocks. Inside, the room
|
||||
was partitioned off at the sides with skins for sleeping quarters. In the main part they had the
|
||||
fire on the ground and the fish drying on lines and the skins hanging around and the dogs and
|
||||
@@ -65,23 +63,22 @@ The other place we stopped was at Whalen, a trading post in Siberia. There these
|
||||
went wild. They rushed helter-skelter, hither and thither, here and there, trying to find
|
||||
something to buy. Prices raised right before your eyes. One would but something for $1.00
|
||||
and the next might have to pay $2.00, $4.00 or $10.00. That made no difference. They had to
|
||||
have it. One man I was sort of taking care of, tho [sic] he had his son along for the purpose,
|
||||
bought 2 ivory tusks, 1 pup, 2 moccasins, 3 or 4 billi[illegible]s, 6 or 8 ivory and silver rings, one
|
||||
have it. One man I was sort of taking care of, tho he had his son along for the purpose,
|
||||
bought 2 ivory tusks, 1 pup, 2 moccasins, 3 or 4 billikens, 6 or 8 ivory and silver rings, one
|
||||
fishing line, hooks, floats, etc. and two bird slings. The slings have rocks at the end and the
|
||||
little natives throw them at the flocks of geese and ducks which fly close over the village and
|
||||
the slings entangle their wings and legs, sometimes more than one, and they can't fly. They
|
||||
come down and the natives capture them. There was more junk brot [sic] aboard than baggage, I
|
||||
come down and the natives capture them. There was more junk brot aboard than baggage, I
|
||||
do believe. And they say that at the first stop it was worse than here. The red flag was flying
|
||||
over Whalen and the Russian soldiers were there–a few, one or two or three, I forget the
|
||||
over Whalen and the Russian soldiers were there—a few, one or two or three, I forget the
|
||||
number.
|
||||
|
||||
We got home yesterday morning at 5 a.m. but missed the first lighter in so had to stay out
|
||||
until 2:30. The girls had prepared a big meal for us and invited up the Hartfords and then let
|
||||
us talk. Miss Saville talked quite a bit. Any how if you folks don't like this I don't care, it is
|
||||
all I had to write about and I know Buster'd [sic] listen anyway and I'd soak ole Peter's head if he
|
||||
all I had to write about and I know Buster'ud listen anyway and I'd soak ole Peter's head if he
|
||||
didn't and Polly would in my lap and I don't know much about the youngest one of yours so
|
||||
likely he would be squawling. But we did surely enjoy our trip and were gone just long enuf [sic].
|
||||
|
||||
likely he would be squawling. But we did surely enjoy our trip and were gone just long enuf.
|
||||
I expect there were 150 passengers on board and almost or more of the crew and helpers. We
|
||||
had a stateroom down next to the kitchen and 'twas pretty fierce for odor at times.
|
||||
|
||||
@@ -109,7 +106,7 @@ Ome
|
||||
|
||||
Reprinted from Cochran Chronicles, Volume 9, Number 1, November 1986
|
||||
|
||||
© [inserted: JECFA] 1986
|
||||
© JECFA 1986
|
||||
|
||||
Up
|
||||
|
||||
|
||||
@@ -2,30 +2,28 @@ source: Rod Moser Letter - p1.jpg
|
||||
provider: openrouter
|
||||
model: google/gemini-2.5-flash
|
||||
---
|
||||
JOHN ISBILL
|
||||
R. T. MOSER
|
||||
JOHN ISBILL R. T. MOSER
|
||||
ISBILL & MOSER
|
||||
DEALERS IN
|
||||
GENERAL MERCHANDISE
|
||||
|
||||
Vonore, Tenn., Jany 27- 1913
|
||||
Dear Much Aunt Louie
|
||||
How are you a
|
||||
few nights ago I sewed a
|
||||
letter from your folks, so
|
||||
Vonore, Tenn. January 27 - 1913
|
||||
Dear Uncle [sic] Aun[t Adeline?]
|
||||
Was at home a
|
||||
few nights ago & saw a
|
||||
letter from you folks, so
|
||||
I decided to write you
|
||||
a few lines myself ok
|
||||
I am contemplateing a
|
||||
trip out west next summer
|
||||
& I want Some Olders to go
|
||||
where I and them.
|
||||
|
||||
I am getting
|
||||
up in years & unmarried
|
||||
a few lines myself &
|
||||
I am contemplating a
|
||||
trip out west next summ[er]
|
||||
& I want both of fillers [sic] to go
|
||||
when I am [to] them.
|
||||
Am getting
|
||||
up in years & unmarried,
|
||||
so you see the object of
|
||||
my trip, is to get a bunch
|
||||
of Young & old maids
|
||||
my trip, is to get a wife
|
||||
& I hear is a lot old maids
|
||||
& widows out there. I
|
||||
want you to kiss them
|
||||
at my fans [sic] mug as they
|
||||
want you to see them
|
||||
at my land my [sic] at there [sic]
|
||||
as soon as I get there
|
||||
|
||||
+3
-1
@@ -5,7 +5,9 @@ isolated, fast, and leave no artifacts on disk.
|
||||
"""
|
||||
|
||||
import pytest
|
||||
from sqlmodel import Session, SQLModel, create_engine
|
||||
from sqlmodel import Session
|
||||
from sqlmodel import SQLModel
|
||||
from sqlmodel import create_engine
|
||||
from sqlmodel.pool import StaticPool
|
||||
|
||||
|
||||
|
||||
@@ -6,7 +6,9 @@ import pytest
|
||||
from sqlmodel import select
|
||||
|
||||
from transcription.config import Settings
|
||||
from transcription.models import Job, JobStatus, Transcript
|
||||
from transcription.models import Job
|
||||
from transcription.models import JobStatus
|
||||
from transcription.models import Transcript
|
||||
from transcription.providers.base import TranscriptionResult
|
||||
from transcription.services.upload import create_upload_job
|
||||
from transcription.worker import process_next_queued_job
|
||||
@@ -71,6 +73,6 @@ class TestPipelineFailureFlow:
|
||||
assert job.status == JobStatus.FAILED
|
||||
assert transcript is not None
|
||||
assert transcript.text is None
|
||||
assert "pipeline provider failure" in transcript.error_detail
|
||||
assert "pipeline provider failure" not in transcript.error_detail
|
||||
assert "[internal_unexpected_error]" in transcript.error_detail
|
||||
assert "error_id=" in transcript.error_detail
|
||||
|
||||
@@ -5,8 +5,10 @@ from types import SimpleNamespace
|
||||
import pytest
|
||||
|
||||
from transcription.config import Settings
|
||||
from transcription.providers.base import ProviderError, ProviderResponseError
|
||||
from transcription.providers.openrouter import DEFAULT_OPENROUTER_MODEL, OpenRouterTranscriptionProvider
|
||||
from transcription.providers.base import ProviderError
|
||||
from transcription.providers.base import ProviderResponseError
|
||||
from transcription.providers.openrouter import DEFAULT_OPENROUTER_MODEL
|
||||
from transcription.providers.openrouter import OpenRouterTranscriptionProvider
|
||||
|
||||
|
||||
class _FakeChat:
|
||||
|
||||
@@ -0,0 +1,98 @@
|
||||
"""Tests for Step 3 library services (revisions, search, export)."""
|
||||
|
||||
import pytest
|
||||
from sqlmodel import select
|
||||
|
||||
from transcription.models import Document
|
||||
from transcription.models import Job
|
||||
from transcription.models import JobStatus
|
||||
from transcription.models import Transcript
|
||||
from transcription.models import TranscriptRevision
|
||||
from transcription.services.library import accept_revision
|
||||
from transcription.services.library import add_revision
|
||||
from transcription.services.library import export_transcripts
|
||||
from transcription.services.library import list_revisions
|
||||
from transcription.services.library import search_accepted_transcripts
|
||||
|
||||
|
||||
def _create_job(session) -> Job:
|
||||
document = Document(filename="letter.jpg", file_path="uploads/letter.jpg")
|
||||
session.add(document)
|
||||
session.commit()
|
||||
session.refresh(document)
|
||||
|
||||
job = Job(document_id=document.id, status=JobStatus.TRANSCRIBED)
|
||||
session.add(job)
|
||||
session.commit()
|
||||
session.refresh(job)
|
||||
return job
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
class TestRevisionHistoryBehavior:
|
||||
"""Verify revision append/accept behavior."""
|
||||
|
||||
def test_add_revision_appends_incrementing_revision_numbers(self, session):
|
||||
"""add_revision creates immutable incrementing revisions per job."""
|
||||
job = _create_job(session)
|
||||
|
||||
r1 = add_revision(job_id=job.id, text="first", session=session)
|
||||
r2 = add_revision(job_id=job.id, text="second", session=session)
|
||||
|
||||
revisions = list_revisions(job_id=job.id, session=session)
|
||||
|
||||
assert r1.revision_number == 1
|
||||
assert r2.revision_number == 2
|
||||
assert [revision.revision_number for revision in revisions] == [1, 2]
|
||||
|
||||
def test_accept_revision_marks_selected_revision_and_sets_job_completed(self, session):
|
||||
"""accept_revision marks one revision accepted and transitions job to completed."""
|
||||
job = _create_job(session)
|
||||
r1 = add_revision(job_id=job.id, text="v1", session=session)
|
||||
r2 = add_revision(job_id=job.id, text="v2", session=session)
|
||||
|
||||
accepted = accept_revision(revision_id=r2.id, session=session)
|
||||
session.refresh(job)
|
||||
|
||||
all_revisions = list(session.exec(select(TranscriptRevision).where(TranscriptRevision.job_id == job.id)).all())
|
||||
accepted_flags = {revision.id: revision.accepted for revision in all_revisions}
|
||||
transcript = session.exec(select(Transcript).where(Transcript.job_id == job.id)).first()
|
||||
|
||||
assert accepted.id == r2.id
|
||||
assert accepted_flags[r1.id] is False
|
||||
assert accepted_flags[r2.id] is True
|
||||
assert job.status == JobStatus.COMPLETED
|
||||
assert transcript is not None
|
||||
assert transcript.text == "v2"
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
class TestSearchAndExportBehavior:
|
||||
"""Verify accepted-only search and export semantics."""
|
||||
|
||||
def test_search_returns_only_accepted_revisions(self, session):
|
||||
"""search_accepted_transcripts filters out non-accepted revisions."""
|
||||
job = _create_job(session)
|
||||
draft = add_revision(job_id=job.id, text="family archive draft", session=session)
|
||||
accepted = add_revision(job_id=job.id, text="family archive final", accepted=True, session=session)
|
||||
|
||||
results = search_accepted_transcripts(query="archive", session=session)
|
||||
|
||||
assert results
|
||||
result_ids = {result.id for result in results}
|
||||
assert accepted.id in result_ids
|
||||
assert draft.id not in result_ids
|
||||
|
||||
def test_export_returns_serializable_records_for_accepted_revisions(self, session):
|
||||
"""export_transcripts returns expected fields for accepted-only export."""
|
||||
job = _create_job(session)
|
||||
accepted = add_revision(job_id=job.id, text="export me", accepted=True, session=session)
|
||||
|
||||
records = export_transcripts(accepted_only=True, session=session)
|
||||
|
||||
assert len(records) == 1
|
||||
record = records[0]
|
||||
assert record["job_id"] == str(job.id)
|
||||
assert record["revision_id"] == str(accepted.id)
|
||||
assert record["accepted"] is True
|
||||
assert record["text"] == "export me"
|
||||
@@ -5,14 +5,13 @@ from pathlib import Path
|
||||
import pytest
|
||||
|
||||
from transcription.config import Settings
|
||||
from transcription.providers.base import ProviderError, TranscriptionResult
|
||||
from transcription.services.transcription import (
|
||||
PromptLoadError,
|
||||
TranscriptionError,
|
||||
load_image_payload,
|
||||
load_prompt_text,
|
||||
transcribe_document_image,
|
||||
)
|
||||
from transcription.providers.base import ProviderError
|
||||
from transcription.providers.base import TranscriptionResult
|
||||
from transcription.services.transcription import PromptLoadError
|
||||
from transcription.services.transcription import TranscriptionError
|
||||
from transcription.services.transcription import load_image_payload
|
||||
from transcription.services.transcription import load_prompt_text
|
||||
from transcription.services.transcription import transcribe_document_image
|
||||
|
||||
|
||||
class _FakeProvider:
|
||||
|
||||
@@ -7,7 +7,6 @@ import pytest
|
||||
|
||||
from transcription.services.transcription import transcribe_document_image
|
||||
|
||||
|
||||
HAS_OPENROUTER_KEY = bool(os.getenv("OPENROUTER_API_KEY"))
|
||||
|
||||
REAL_IMAGES_DIR = Path(__file__).resolve().parents[1] / "fixtures" / "images" / "real"
|
||||
|
||||
@@ -5,8 +5,11 @@ from pathlib import Path
|
||||
import pytest
|
||||
|
||||
from transcription.config import Settings
|
||||
from transcription.models import Document, Job, JobStatus
|
||||
from transcription.services.upload import UploadError, create_upload_job
|
||||
from transcription.models import Document
|
||||
from transcription.models import Job
|
||||
from transcription.models import JobStatus
|
||||
from transcription.services.upload import UploadError
|
||||
from transcription.services.upload import create_upload_job
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
@@ -41,6 +44,24 @@ class TestUploadValidation:
|
||||
assert exc_info.value.category.value == "user_input_error"
|
||||
assert "jpg" in exc_info.value.suggestion.lower()
|
||||
|
||||
def test_rejects_payload_exceeding_max_upload_bytes(self, session, tmp_path: Path):
|
||||
"""create_upload_job rejects payloads above configured size limit."""
|
||||
settings = Settings(
|
||||
openrouter_api_key="test-key",
|
||||
upload_dir=tmp_path,
|
||||
max_upload_bytes=3,
|
||||
)
|
||||
with pytest.raises(UploadError) as exc_info:
|
||||
create_upload_job(
|
||||
filename="scan.jpg",
|
||||
file_bytes=b"1234",
|
||||
session=session,
|
||||
settings=settings,
|
||||
)
|
||||
|
||||
assert exc_info.value.category.value == "user_input_error"
|
||||
assert "smaller file" in exc_info.value.suggestion.lower()
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
class TestUploadPersistence:
|
||||
|
||||
@@ -7,10 +7,16 @@ import pytest
|
||||
from sqlmodel import select
|
||||
|
||||
from transcription.config import Settings
|
||||
from transcription.errors import AppError, ErrorCategory
|
||||
from transcription.models import Document, Job, JobStatus, Transcript
|
||||
from transcription.errors import AppError
|
||||
from transcription.errors import ErrorCategory
|
||||
from transcription.models import Document
|
||||
from transcription.models import Job
|
||||
from transcription.models import JobStatus
|
||||
from transcription.models import Transcript
|
||||
from transcription.models import TranscriptRevision
|
||||
from transcription.providers.base import TranscriptionResult
|
||||
from transcription.worker import process_next_queued_job, run_worker_loop
|
||||
from transcription.worker import process_next_queued_job
|
||||
from transcription.worker import run_worker_loop
|
||||
|
||||
|
||||
def _create_queued_job(session, *, filename: str = "doc.jpg", file_path: str = "uploads/doc.jpg") -> Job:
|
||||
@@ -74,12 +80,21 @@ class TestWorkerSuccessPath:
|
||||
|
||||
process_next_queued_job(session=session)
|
||||
|
||||
transcript = session.exec(
|
||||
select(Transcript).where(Transcript.job_id == job.id)
|
||||
transcript = session.exec(select(Transcript).where(Transcript.job_id == job.id)).first()
|
||||
revision = session.exec(
|
||||
select(TranscriptRevision)
|
||||
.where(TranscriptRevision.job_id == job.id)
|
||||
.order_by(TranscriptRevision.revision_number)
|
||||
).first()
|
||||
|
||||
assert transcript is not None
|
||||
assert transcript.text == "Transcript body"
|
||||
assert transcript.error_detail is None
|
||||
assert revision is not None
|
||||
assert revision.revision_number == 1
|
||||
assert revision.text == "Transcript body"
|
||||
assert revision.source == "worker"
|
||||
assert revision.accepted is False
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
@@ -109,7 +124,7 @@ class TestWorkerFailurePath:
|
||||
assert job.status == JobStatus.FAILED
|
||||
assert transcript is not None
|
||||
assert transcript.text is None
|
||||
assert "provider failure" in transcript.error_detail
|
||||
assert "provider failure" not in transcript.error_detail
|
||||
assert "[internal_unexpected_error]" in transcript.error_detail
|
||||
assert "error_id=" in transcript.error_detail
|
||||
assert "suggestion=" in transcript.error_detail
|
||||
@@ -139,7 +154,7 @@ class TestWorkerFailurePath:
|
||||
assert len(transcripts) == 1
|
||||
assert transcripts[0].id == existing.id
|
||||
assert transcripts[0].text is None
|
||||
assert "provider failure" in transcripts[0].error_detail
|
||||
assert "provider failure" not in transcripts[0].error_detail
|
||||
assert "[internal_unexpected_error]" in transcripts[0].error_detail
|
||||
assert "error_id=" in transcripts[0].error_detail
|
||||
|
||||
|
||||
@@ -26,6 +26,8 @@ class TestAppLifespan:
|
||||
calls = []
|
||||
|
||||
monkeypatch.setattr("transcription.app.setup_logging", lambda: calls.append("logging"))
|
||||
monkeypatch.setattr("transcription.app.get_settings", lambda: object())
|
||||
monkeypatch.setattr("transcription.app.enforce_request_access", lambda **_kwargs: None)
|
||||
monkeypatch.setattr("transcription.app.create_all", lambda **_kwargs: calls.append("schema"))
|
||||
monkeypatch.setattr(
|
||||
"transcription.app.initialize_database_runtime",
|
||||
@@ -35,6 +37,8 @@ class TestAppLifespan:
|
||||
monkeypatch.setattr("transcription.app.should_bootstrap_schema", lambda _settings: True)
|
||||
monkeypatch.setattr("transcription.app._start_worker", lambda _app: calls.append("start_worker"))
|
||||
monkeypatch.setattr("transcription.app._stop_worker", lambda _app: calls.append("stop_worker"))
|
||||
monkeypatch.setattr("transcription.app.apply_pending_migrations", lambda **_kwargs: calls.append("migrate"))
|
||||
monkeypatch.setattr("transcription.app.validate_schema_compatibility", lambda **_kwargs: [])
|
||||
|
||||
class _Dir:
|
||||
def mkdir(self, parents: bool, exist_ok: bool):
|
||||
@@ -43,6 +47,8 @@ class TestAppLifespan:
|
||||
class _Settings:
|
||||
upload_dir = _Dir()
|
||||
prompt_dir = _Dir()
|
||||
migration_auto_apply_on_startup = False
|
||||
validate_schema_on_startup = True
|
||||
|
||||
monkeypatch.setattr("transcription.app.get_settings", lambda: _Settings())
|
||||
|
||||
@@ -61,6 +67,8 @@ class TestAppLifespan:
|
||||
calls = []
|
||||
|
||||
monkeypatch.setattr("transcription.app.setup_logging", lambda: None)
|
||||
monkeypatch.setattr("transcription.app.get_settings", lambda: object())
|
||||
monkeypatch.setattr("transcription.app.enforce_request_access", lambda **_kwargs: None)
|
||||
monkeypatch.setattr("transcription.app.create_all", lambda **_kwargs: None)
|
||||
monkeypatch.setattr(
|
||||
"transcription.app.initialize_database_runtime",
|
||||
@@ -70,6 +78,8 @@ class TestAppLifespan:
|
||||
monkeypatch.setattr("transcription.app.should_bootstrap_schema", lambda _settings: True)
|
||||
monkeypatch.setattr("transcription.app._start_worker", lambda _app: calls.append("start_worker"))
|
||||
monkeypatch.setattr("transcription.app._stop_worker", lambda _app: calls.append("stop_worker"))
|
||||
monkeypatch.setattr("transcription.app.apply_pending_migrations", lambda **_kwargs: calls.append("migrate"))
|
||||
monkeypatch.setattr("transcription.app.validate_schema_compatibility", lambda **_kwargs: [])
|
||||
|
||||
class _Dir:
|
||||
def mkdir(self, parents: bool, exist_ok: bool):
|
||||
@@ -78,6 +88,8 @@ class TestAppLifespan:
|
||||
class _Settings:
|
||||
upload_dir = _Dir()
|
||||
prompt_dir = _Dir()
|
||||
migration_auto_apply_on_startup = False
|
||||
validate_schema_on_startup = True
|
||||
|
||||
monkeypatch.setattr("transcription.app.get_settings", lambda: _Settings())
|
||||
|
||||
|
||||
+29
-1
@@ -5,7 +5,8 @@ from pathlib import Path
|
||||
import pytest
|
||||
from pydantic import ValidationError
|
||||
|
||||
from transcription.config import Provider, Settings
|
||||
from transcription.config import Provider
|
||||
from transcription.config import Settings
|
||||
|
||||
|
||||
def _make_settings(**overrides) -> Settings:
|
||||
@@ -63,6 +64,33 @@ class TestPathSettings:
|
||||
assert isinstance(settings.prompt_dir, Path)
|
||||
|
||||
|
||||
class TestMigrationSafetySettings:
|
||||
"""Verify migration safety settings defaults."""
|
||||
|
||||
def test_migration_safety_defaults(self):
|
||||
"""Migration auto-apply is off and startup schema validation is on by default."""
|
||||
settings = _make_settings()
|
||||
assert settings.migration_auto_apply_on_startup is False
|
||||
assert settings.validate_schema_on_startup is True
|
||||
|
||||
|
||||
class TestSecuritySettings:
|
||||
"""Verify Step 5 security-related settings behavior."""
|
||||
|
||||
def test_security_defaults(self):
|
||||
"""Security controls default to disabled auth and bounded upload size."""
|
||||
settings = _make_settings()
|
||||
assert settings.max_upload_bytes == 15 * 1024 * 1024
|
||||
assert settings.operator_access_enabled is False
|
||||
assert settings.operator_username == "operator"
|
||||
assert settings.operator_password is None
|
||||
|
||||
def test_operator_password_required_when_access_enabled(self):
|
||||
"""Enabling operator access requires OPERATOR_PASSWORD."""
|
||||
with pytest.raises(ValidationError):
|
||||
_make_settings(operator_access_enabled=True, operator_password=None)
|
||||
|
||||
|
||||
class TestWorkerReliabilitySettings:
|
||||
"""Verify worker retry settings defaults."""
|
||||
|
||||
|
||||
+22
-5
@@ -1,7 +1,10 @@
|
||||
"""Tests for transcription.db — schema bootstrap and session factory."""
|
||||
|
||||
from sqlalchemy import inspect, text
|
||||
from sqlmodel import Session, SQLModel, create_engine
|
||||
from sqlalchemy import inspect
|
||||
from sqlalchemy import text
|
||||
from sqlmodel import Session
|
||||
from sqlmodel import SQLModel
|
||||
from sqlmodel import create_engine
|
||||
from sqlmodel.pool import StaticPool
|
||||
|
||||
|
||||
@@ -18,12 +21,14 @@ class TestSchemaBootstrap:
|
||||
"""Verify create_all produces the expected table set."""
|
||||
|
||||
def test_create_all_creates_expected_tables(self):
|
||||
"""After create_all(), document, job, and transcript tables exist."""
|
||||
"""After create_all(), core V1 tables exist."""
|
||||
engine = _in_memory_engine()
|
||||
# Ensure models are imported so metadata is populated
|
||||
from transcription.models import Document, Job, Transcript # noqa: F401
|
||||
|
||||
import transcription.db as db_module
|
||||
from transcription.models import Document # noqa: F401
|
||||
from transcription.models import Job # noqa: F401
|
||||
from transcription.models import Transcript # noqa: F401
|
||||
from transcription.models import TranscriptRevision # noqa: F401
|
||||
|
||||
db_module.create_all(engine=engine)
|
||||
|
||||
@@ -32,6 +37,18 @@ class TestSchemaBootstrap:
|
||||
assert "document" in table_names
|
||||
assert "job" in table_names
|
||||
assert "transcript" in table_names
|
||||
assert "transcriptrevision" in table_names
|
||||
|
||||
def test_validate_schema_compatibility_returns_no_issues_for_fresh_schema(self):
|
||||
"""validate_schema_compatibility reports no issues on fresh schema."""
|
||||
engine = _in_memory_engine()
|
||||
|
||||
import transcription.db as db_module
|
||||
|
||||
db_module.create_all(engine=engine)
|
||||
issues = db_module.validate_schema_compatibility(engine=engine)
|
||||
|
||||
assert issues == []
|
||||
|
||||
|
||||
class TestSessionFactory:
|
||||
|
||||
@@ -2,7 +2,10 @@
|
||||
|
||||
import pytest
|
||||
|
||||
from transcription.errors import AppError, ErrorCategory, classify_unexpected_error, new_error_id
|
||||
from transcription.errors import AppError
|
||||
from transcription.errors import ErrorCategory
|
||||
from transcription.errors import classify_unexpected_error
|
||||
from transcription.errors import new_error_id
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
@@ -38,6 +41,6 @@ class TestAppErrorHelpers:
|
||||
assert isinstance(err, AppError)
|
||||
assert err.category == ErrorCategory.INTERNAL_UNEXPECTED
|
||||
assert "unit.test" in err.message
|
||||
assert "boom" in err.message
|
||||
assert "boom" not in err.message
|
||||
assert err.suggestion
|
||||
assert err.error_id
|
||||
|
||||
@@ -0,0 +1,108 @@
|
||||
"""Tests for transcription.migrations — explicit Step 4 migration safety behavior."""
|
||||
|
||||
from sqlalchemy import inspect
|
||||
from sqlalchemy import text
|
||||
from sqlmodel import create_engine
|
||||
from sqlmodel.pool import StaticPool
|
||||
|
||||
from transcription.migrations import apply_pending_migrations
|
||||
from transcription.migrations import list_pending_migrations
|
||||
|
||||
|
||||
def _in_memory_engine():
|
||||
"""Create isolated in-memory SQLite engine."""
|
||||
return create_engine(
|
||||
"sqlite://",
|
||||
connect_args={"check_same_thread": False},
|
||||
poolclass=StaticPool,
|
||||
)
|
||||
|
||||
|
||||
class TestMigrations:
|
||||
"""Verify migration listing and application behavior."""
|
||||
|
||||
def test_list_pending_returns_all_before_apply(self):
|
||||
"""All known migrations are pending on a fresh legacy-shaped database."""
|
||||
engine = _in_memory_engine()
|
||||
|
||||
with engine.begin() as connection:
|
||||
connection.execute(
|
||||
text(
|
||||
"""
|
||||
CREATE TABLE job (
|
||||
id TEXT PRIMARY KEY,
|
||||
document_id TEXT NOT NULL,
|
||||
status TEXT NOT NULL,
|
||||
created_at TEXT NOT NULL,
|
||||
updated_at TEXT NOT NULL
|
||||
)
|
||||
"""
|
||||
)
|
||||
)
|
||||
|
||||
pending = list_pending_migrations(engine=engine)
|
||||
assert [migration.revision_id for migration in pending] == [
|
||||
"0001_add_retry_count_to_job",
|
||||
"0002_create_transcriptrevision_table",
|
||||
]
|
||||
|
||||
def test_apply_pending_migrations_records_history_and_schema(self):
|
||||
"""Applying pending migrations mutates schema and records revision history."""
|
||||
engine = _in_memory_engine()
|
||||
|
||||
with engine.begin() as connection:
|
||||
connection.execute(
|
||||
text(
|
||||
"""
|
||||
CREATE TABLE job (
|
||||
id TEXT PRIMARY KEY,
|
||||
document_id TEXT NOT NULL,
|
||||
status TEXT NOT NULL,
|
||||
created_at TEXT NOT NULL,
|
||||
updated_at TEXT NOT NULL
|
||||
)
|
||||
"""
|
||||
)
|
||||
)
|
||||
|
||||
applied = apply_pending_migrations(engine=engine)
|
||||
assert applied == [
|
||||
"0001_add_retry_count_to_job",
|
||||
"0002_create_transcriptrevision_table",
|
||||
]
|
||||
|
||||
inspector = inspect(engine)
|
||||
job_columns = {column["name"] for column in inspector.get_columns("job")}
|
||||
assert "retry_count" in job_columns
|
||||
assert "transcriptrevision" in set(inspector.get_table_names())
|
||||
|
||||
with engine.begin() as connection:
|
||||
rows = connection.execute(
|
||||
text("SELECT revision_id FROM schema_migration_history ORDER BY revision_id")
|
||||
).fetchall()
|
||||
assert [row[0] for row in rows] == applied
|
||||
|
||||
def test_apply_pending_migrations_is_idempotent(self):
|
||||
"""Re-running apply_pending_migrations with no pending revisions is a no-op."""
|
||||
engine = _in_memory_engine()
|
||||
|
||||
with engine.begin() as connection:
|
||||
connection.execute(
|
||||
text(
|
||||
"""
|
||||
CREATE TABLE job (
|
||||
id TEXT PRIMARY KEY,
|
||||
document_id TEXT NOT NULL,
|
||||
status TEXT NOT NULL,
|
||||
created_at TEXT NOT NULL,
|
||||
updated_at TEXT NOT NULL
|
||||
)
|
||||
"""
|
||||
)
|
||||
)
|
||||
|
||||
first_apply = apply_pending_migrations(engine=engine)
|
||||
second_apply = apply_pending_migrations(engine=engine)
|
||||
|
||||
assert len(first_apply) == 2
|
||||
assert second_apply == []
|
||||
+44
-1
@@ -5,7 +5,11 @@ from uuid import UUID
|
||||
import pytest
|
||||
from sqlalchemy.exc import IntegrityError
|
||||
|
||||
from transcription.models import Document, Job, JobStatus, Transcript
|
||||
from transcription.models import Document
|
||||
from transcription.models import Job
|
||||
from transcription.models import JobStatus
|
||||
from transcription.models import Transcript
|
||||
from transcription.models import TranscriptRevision
|
||||
|
||||
|
||||
def _make_document(**overrides) -> Document:
|
||||
@@ -89,6 +93,12 @@ class TestJobModel:
|
||||
session.refresh(job)
|
||||
assert job.status == JobStatus.TRANSCRIBED
|
||||
|
||||
job.status = JobStatus.COMPLETED
|
||||
session.add(job)
|
||||
session.commit()
|
||||
session.refresh(job)
|
||||
assert job.status == JobStatus.COMPLETED
|
||||
|
||||
def test_transitions_to_failed(self, session):
|
||||
"""Status updates from processing to failed."""
|
||||
doc = _persist_document(session)
|
||||
@@ -152,6 +162,27 @@ class TestTranscriptModel:
|
||||
session.commit()
|
||||
|
||||
|
||||
class TestTranscriptRevisionModel:
|
||||
"""Verify transcript revision persistence and defaults."""
|
||||
|
||||
def test_revision_defaults_and_persistence(self, session):
|
||||
"""Revision records persist with revision metadata and defaults."""
|
||||
doc = _persist_document(session)
|
||||
job = _persist_job(session, doc)
|
||||
|
||||
revision = TranscriptRevision(job_id=job.id, revision_number=1, text="Rev text")
|
||||
session.add(revision)
|
||||
session.commit()
|
||||
session.refresh(revision)
|
||||
|
||||
fetched = session.get(TranscriptRevision, revision.id)
|
||||
assert fetched is not None
|
||||
assert fetched.revision_number == 1
|
||||
assert fetched.text == "Rev text"
|
||||
assert fetched.source == "worker"
|
||||
assert fetched.accepted is False
|
||||
|
||||
|
||||
class TestRelationships:
|
||||
"""Verify SQLModel relationship navigation between models."""
|
||||
|
||||
@@ -177,3 +208,15 @@ class TestRelationships:
|
||||
assert job.transcript is not None
|
||||
assert isinstance(job.transcript, Transcript)
|
||||
assert job.transcript.text == "Transcribed text"
|
||||
|
||||
def test_job_exposes_revisions(self, session):
|
||||
"""job.revisions returns revision history linked to the Job."""
|
||||
doc = _persist_document(session)
|
||||
job = _persist_job(session, doc)
|
||||
session.add(TranscriptRevision(job_id=job.id, revision_number=1, text="v1"))
|
||||
session.add(TranscriptRevision(job_id=job.id, revision_number=2, text="v2", accepted=True))
|
||||
session.commit()
|
||||
|
||||
session.refresh(job)
|
||||
assert len(job.revisions) == 2
|
||||
assert all(isinstance(revision, TranscriptRevision) for revision in job.revisions)
|
||||
|
||||
@@ -2,7 +2,6 @@
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
PROMPT_PATH = Path("prompts/transcribe_document.md")
|
||||
|
||||
|
||||
|
||||
@@ -19,14 +19,17 @@ MVP_REQUIREMENT_TEST_MAP: dict[str, list[str]] = {
|
||||
"REQ-3": [
|
||||
"tests/services/test_worker.py",
|
||||
"tests/ui/test_jobs_page.py",
|
||||
"tests/services/test_library.py",
|
||||
],
|
||||
"REQ-4": [
|
||||
"tests/services/test_worker.py",
|
||||
"tests/integration/test_pipeline_flow.py",
|
||||
"tests/services/test_library.py",
|
||||
],
|
||||
"REQ-5": [
|
||||
"tests/ui/test_jobs_page.py",
|
||||
"tests/ui/test_pages_registration.py",
|
||||
"tests/api/test_routes.py",
|
||||
],
|
||||
"REQ-6": [
|
||||
"tests/test_app.py",
|
||||
@@ -36,6 +39,9 @@ MVP_REQUIREMENT_TEST_MAP: dict[str, list[str]] = {
|
||||
"tests/test_app.py",
|
||||
"tests/test_config.py",
|
||||
],
|
||||
"REQ-11": [
|
||||
"tests/services/test_library.py",
|
||||
],
|
||||
"REQ-12": [
|
||||
"tests/test_prompts.py",
|
||||
"tests/services/test_transcription.py",
|
||||
|
||||
@@ -4,8 +4,11 @@ from uuid import uuid4
|
||||
|
||||
import pytest
|
||||
|
||||
from transcription.models import Document, Job, Transcript
|
||||
from transcription.ui.jobs_page import fetch_job_detail, fetch_jobs
|
||||
from transcription.models import Document
|
||||
from transcription.models import Job
|
||||
from transcription.models import Transcript
|
||||
from transcription.ui.jobs_page import fetch_job_detail
|
||||
from transcription.ui.jobs_page import fetch_jobs
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
|
||||
@@ -1,8 +1,8 @@
|
||||
"""Tests for UI page registration wiring."""
|
||||
|
||||
import pytest
|
||||
from fastapi import FastAPI
|
||||
from fastapi.testclient import TestClient
|
||||
import pytest
|
||||
|
||||
from transcription.ui import register_pages
|
||||
|
||||
|
||||
@@ -5,7 +5,8 @@ from uuid import uuid4
|
||||
|
||||
import pytest
|
||||
|
||||
from transcription.services.upload import UploadError, UploadJobResult
|
||||
from transcription.services.upload import UploadError
|
||||
from transcription.services.upload import UploadJobResult
|
||||
from transcription.ui import upload_page
|
||||
|
||||
|
||||
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 1.7 MiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 385 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 274 KiB |
Reference in New Issue
Block a user