generated from john/python-template
Step 6 implementation plan
This commit is contained in:
Jim Lancaster
+229
@@ -0,0 +1,229 @@
|
|||||||
|
## Step 6: Test and Verification Hardening (MVP closeout)
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
Complete MVP verification by building a **requirements-traceable, deterministic test strategy** across unit/integration/external lanes, then enforcing stable validation commands and reporting.
|
||||||
|
|
||||||
|
This step finalizes the MVP implementation sequence from `docs/mvp.md` (Step 6 in the build order: tests and automated verification).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## MCP Resource Integration (what was applied)
|
||||||
|
|
||||||
|
I reviewed all top-level skills/prompts from `john-stream-mcp` and integrated the relevant guidance into this plan:
|
||||||
|
|
||||||
|
### Directly applied
|
||||||
|
- `resource://skills/pytesting/document`
|
||||||
|
- `resource://catalog/prompts/pytest-scaffold`
|
||||||
|
- `resource://prompts/pytest-scaffold/document`
|
||||||
|
- `resource://catalog/prompts/pytest-fill-scaffold`
|
||||||
|
- `resource://prompts/pytest-fill-scaffold/document`
|
||||||
|
- `resource://skills/nicegui/document`
|
||||||
|
- `resource://skills/nicegui-ui-customization/document`
|
||||||
|
- `resource://skills/fastapi-uv-docker/document`
|
||||||
|
- `resource://skills/python-logging-dictconfig/document`
|
||||||
|
- `resource://skills/python-typing/document`
|
||||||
|
- `resource://skills/ruff-linting-formating/document`
|
||||||
|
|
||||||
|
### Reviewed but informational/non-blocking for Step 6
|
||||||
|
- `copilot-customization`, `mcp-details`, `vscode-configuration`, `zensical-docs`, and authoring/shim prompts.
|
||||||
|
- These are primarily customization/documentation tooling resources, not core MVP test-lane blockers.
|
||||||
|
- Step 6 includes optional workflow follow-ups where relevant (e.g., VS Code task conveniences).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
### In scope
|
||||||
|
- Strengthen and complete test coverage for the shipped MVP slice (Steps 1–5)
|
||||||
|
- Add requirement-to-test traceability for REQ-0..REQ-12 (MVP subset emphasized)
|
||||||
|
- Enforce deterministic default lanes (`unit`, `integration`)
|
||||||
|
- Keep `external` lane opt-in and isolated
|
||||||
|
- Validate app/UI/service/worker contracts end-to-end at test level
|
||||||
|
|
||||||
|
### Out of scope
|
||||||
|
- Major architecture rewrites (async SQLAlchemy migration, queue system, etc.)
|
||||||
|
- Full production deployment rollout
|
||||||
|
- Post-MVP feature expansion (revision history, search, export)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Planned Deliverables
|
||||||
|
|
||||||
|
### Test files (new/updated)
|
||||||
|
- `tests/test_traceability.py` *(or docs-based traceability matrix if preferred)*
|
||||||
|
- `tests/integration/test_pipeline_flow.py` *(upload -> queued -> worker -> transcript/failed)*
|
||||||
|
- `tests/ui/test_upload_page.py` (augment loading/error/ready-state checks as practical)
|
||||||
|
- `tests/ui/test_jobs_page.py` (augment refresh/error behavior checks as practical)
|
||||||
|
- Existing tests touched only when needed; preserve naming/hierarchy unless explicitly approved.
|
||||||
|
|
||||||
|
### Optional docs output
|
||||||
|
- `docs/tests.md` or `docs/verification.md` with lane definitions and command matrix
|
||||||
|
- REQ-to-test mapping table
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Design and Policy Decisions (MCP-aligned)
|
||||||
|
|
||||||
|
1. **Scaffold-first, fill-second workflow is mandatory**
|
||||||
|
- First create/adjust skeletons and collect.
|
||||||
|
- Then fill test bodies.
|
||||||
|
- Preserve scaffold names/docstrings during fill.
|
||||||
|
|
||||||
|
2. **Deterministic-first default lanes**
|
||||||
|
- `unit` and `integration` run by default.
|
||||||
|
- `external` remains explicit opt-in.
|
||||||
|
|
||||||
|
3. **One behavior target per test**
|
||||||
|
- Short, behavior-focused names.
|
||||||
|
- Precise assertions on observable outcomes.
|
||||||
|
|
||||||
|
4. **Test double discipline (from pytesting skill)**
|
||||||
|
- Prefer real-input/real-object paths first.
|
||||||
|
- If monkeypatch/mocks/fakes are needed for a boundary, keep narrowly scoped.
|
||||||
|
- Avoid call-only assertions.
|
||||||
|
|
||||||
|
5. **NiceGUI responsiveness expectations**
|
||||||
|
- Verify loading/success/error state transitions where testable.
|
||||||
|
- Ensure user-facing feedback behavior is covered.
|
||||||
|
|
||||||
|
6. **FastAPI/ops baseline checks**
|
||||||
|
- Keep `/healthz` route validation in default lanes.
|
||||||
|
- Keep startup/shutdown lifecycle assertions present.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Implementation Plan + Checklist
|
||||||
|
|
||||||
|
## Phase A — Coverage and traceability audit
|
||||||
|
|
||||||
|
- [ ] Build a REQ-to-test matrix for MVP requirements:
|
||||||
|
- [ ] REQ-0, REQ-1, REQ-2, REQ-3, REQ-4, REQ-5, REQ-6, REQ-8, REQ-12
|
||||||
|
- [ ] Identify weak spots:
|
||||||
|
- [ ] full pipeline integration (service + worker + persistence)
|
||||||
|
- [ ] UI state transition assertions (loading/error/ready)
|
||||||
|
- [ ] failure-path persistence verification robustness
|
||||||
|
- [ ] Record current baseline command results before edits
|
||||||
|
|
||||||
|
## Phase B — Scaffold phase (pytest-scaffold resources)
|
||||||
|
|
||||||
|
Target modules/areas:
|
||||||
|
- pipeline integration flow
|
||||||
|
- UI behavior augmentations
|
||||||
|
- traceability checks/document validators (if test-backed)
|
||||||
|
|
||||||
|
- [ ] Scaffold new/adjusted test files/classes/methods only
|
||||||
|
- [ ] Keep one-line intent docstrings
|
||||||
|
- [ ] Keep behavior-focused names
|
||||||
|
- [ ] Run: `uv run pytest --collect-only -q`
|
||||||
|
|
||||||
|
## Phase C — Fill phase (pytest-fill-scaffold resources)
|
||||||
|
|
||||||
|
- [ ] Fill scaffolded methods with deterministic setup/assertions
|
||||||
|
- [ ] Preserve scaffold names/hierarchy/docstrings
|
||||||
|
- [ ] Add/adjust fixtures at nearest useful scope
|
||||||
|
- [ ] Keep DB tests in `integration`; pure helper tests in `unit`
|
||||||
|
|
||||||
|
### Required coverage additions
|
||||||
|
|
||||||
|
#### Pipeline integration
|
||||||
|
- [ ] Upload service creates document/job and file path persists
|
||||||
|
- [ ] Worker success path creates transcript and terminal status
|
||||||
|
- [ ] Worker failure path persists error detail and terminal failed status
|
||||||
|
- [ ] Queue-empty behavior remains stable (`False` return / no side effects)
|
||||||
|
|
||||||
|
#### UI behavior (practical, testable boundaries)
|
||||||
|
- [ ] Upload helper flow success and UploadError surfacing
|
||||||
|
- [ ] Jobs data helpers return stable normalized view models
|
||||||
|
- [ ] Refresh/detail fallback behavior for missing/invalid job IDs
|
||||||
|
|
||||||
|
#### Traceability
|
||||||
|
- [ ] Every in-scope MVP REQ has at least one mapped test/assertion point
|
||||||
|
- [ ] Document and/or enforce mapping consistency
|
||||||
|
|
||||||
|
## Phase D — External lane stability
|
||||||
|
|
||||||
|
- [ ] Keep real-image external tests isolated under `@pytest.mark.external`
|
||||||
|
- [ ] Ensure no external test leaks into default runs
|
||||||
|
- [ ] Confirm artifact capture behavior remains stable
|
||||||
|
|
||||||
|
## Phase E — Quality gates and workflow
|
||||||
|
|
||||||
|
- [ ] Confirm logging/lifecycle startup tests still pass after changes
|
||||||
|
- [ ] (If enabled) add/update lint/type check commands in docs:
|
||||||
|
- [ ] Ruff lane (if configured)
|
||||||
|
- [ ] typing lane (if configured)
|
||||||
|
- [ ] Optionally add VS Code task aliases for test lanes (non-blocking)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Marker and Fixture Strategy
|
||||||
|
|
||||||
|
- `unit`: pure logic, helper behavior, formatting/normalization
|
||||||
|
- `integration`: DB + service + app lifecycle contracts
|
||||||
|
- `external`: live provider/real image checks only
|
||||||
|
|
||||||
|
Fixture policy:
|
||||||
|
- Prefer reusable fixtures in `tests/conftest.py` only when broadly shared
|
||||||
|
- Use subtree/local fixtures for domain-specific setup
|
||||||
|
- Keep setup explicit and readable
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Validation Sequence (strict)
|
||||||
|
|
||||||
|
- [ ] `uv run pytest --collect-only -q`
|
||||||
|
- [ ] `uv run pytest -m unit -q`
|
||||||
|
- [ ] `uv run pytest -m integration -q`
|
||||||
|
- [ ] `uv run pytest -m "not external" -q`
|
||||||
|
- [ ] `uv run pytest tests/integration/test_pipeline_flow.py -q` *(if added)*
|
||||||
|
- [ ] `uv run pytest tests/ui/test_upload_page.py -q`
|
||||||
|
- [ ] `uv run pytest tests/ui/test_jobs_page.py -q`
|
||||||
|
- [ ] `uv run pytest -q`
|
||||||
|
|
||||||
|
Optional external verification:
|
||||||
|
- [ ] `uv run pytest -m external -q`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Guardrails
|
||||||
|
|
||||||
|
- Do not rename/re-nest scaffolded tests during fill unless explicitly requested.
|
||||||
|
- Do not broaden external dependencies in default lane.
|
||||||
|
- Do not add flaky timing-based assertions; keep deterministic boundaries.
|
||||||
|
- Keep business logic out of UI tests; test through service/helper boundaries.
|
||||||
|
- Preserve one-way dependency direction in test setup patterns.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Definition of Done (Step 6)
|
||||||
|
|
||||||
|
- [ ] MVP requirement coverage is explicitly traceable
|
||||||
|
- [ ] Deterministic lanes (`unit` + `integration`) are stable and green
|
||||||
|
- [ ] External lane remains opt-in and green when enabled
|
||||||
|
- [ ] Pipeline success/failure lifecycle paths are verified end-to-end
|
||||||
|
- [ ] UI helper/state behavior has explicit success/error assertions
|
||||||
|
- [ ] Full suite passes with `uv run pytest -q`
|
||||||
|
- [ ] Verification evidence is captured in implementation report
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## PR Checklist (Step 6)
|
||||||
|
|
||||||
|
### Implementation
|
||||||
|
- [ ] Added/updated test files per scoped gaps
|
||||||
|
- [ ] Added REQ traceability mapping
|
||||||
|
- [ ] Kept default lanes deterministic
|
||||||
|
- [ ] Preserved scaffold invariants during fill
|
||||||
|
|
||||||
|
### Testing (MCP-compliant)
|
||||||
|
- [ ] Used scaffold prompt flow first
|
||||||
|
- [ ] Used fill prompt flow second
|
||||||
|
- [ ] Preserved naming/docstrings/hierarchy
|
||||||
|
- [ ] Marker usage documented (`unit`, `integration`, `external`)
|
||||||
|
|
||||||
|
### Evidence
|
||||||
|
- [ ] Collected command outputs in strict order
|
||||||
|
- [ ] Listed files changed
|
||||||
|
- [ ] Listed MCP resources used and why
|
||||||
|
- [ ] Noted residual risks/open questions (if any)
|
||||||
Reference in New Issue
Block a user