generated from john/python-template
Step 6 implementation plan
This commit is contained in:
Jim Lancaster
+229
@@ -0,0 +1,229 @@
|
||||
## Step 6: Test and Verification Hardening (MVP closeout)
|
||||
|
||||
## Objective
|
||||
|
||||
Complete MVP verification by building a **requirements-traceable, deterministic test strategy** across unit/integration/external lanes, then enforcing stable validation commands and reporting.
|
||||
|
||||
This step finalizes the MVP implementation sequence from `docs/mvp.md` (Step 6 in the build order: tests and automated verification).
|
||||
|
||||
---
|
||||
|
||||
## MCP Resource Integration (what was applied)
|
||||
|
||||
I reviewed all top-level skills/prompts from `john-stream-mcp` and integrated the relevant guidance into this plan:
|
||||
|
||||
### Directly applied
|
||||
- `resource://skills/pytesting/document`
|
||||
- `resource://catalog/prompts/pytest-scaffold`
|
||||
- `resource://prompts/pytest-scaffold/document`
|
||||
- `resource://catalog/prompts/pytest-fill-scaffold`
|
||||
- `resource://prompts/pytest-fill-scaffold/document`
|
||||
- `resource://skills/nicegui/document`
|
||||
- `resource://skills/nicegui-ui-customization/document`
|
||||
- `resource://skills/fastapi-uv-docker/document`
|
||||
- `resource://skills/python-logging-dictconfig/document`
|
||||
- `resource://skills/python-typing/document`
|
||||
- `resource://skills/ruff-linting-formating/document`
|
||||
|
||||
### Reviewed but informational/non-blocking for Step 6
|
||||
- `copilot-customization`, `mcp-details`, `vscode-configuration`, `zensical-docs`, and authoring/shim prompts.
|
||||
- These are primarily customization/documentation tooling resources, not core MVP test-lane blockers.
|
||||
- Step 6 includes optional workflow follow-ups where relevant (e.g., VS Code task conveniences).
|
||||
|
||||
---
|
||||
|
||||
## Scope
|
||||
|
||||
### In scope
|
||||
- Strengthen and complete test coverage for the shipped MVP slice (Steps 1–5)
|
||||
- Add requirement-to-test traceability for REQ-0..REQ-12 (MVP subset emphasized)
|
||||
- Enforce deterministic default lanes (`unit`, `integration`)
|
||||
- Keep `external` lane opt-in and isolated
|
||||
- Validate app/UI/service/worker contracts end-to-end at test level
|
||||
|
||||
### Out of scope
|
||||
- Major architecture rewrites (async SQLAlchemy migration, queue system, etc.)
|
||||
- Full production deployment rollout
|
||||
- Post-MVP feature expansion (revision history, search, export)
|
||||
|
||||
---
|
||||
|
||||
## Planned Deliverables
|
||||
|
||||
### Test files (new/updated)
|
||||
- `tests/test_traceability.py` *(or docs-based traceability matrix if preferred)*
|
||||
- `tests/integration/test_pipeline_flow.py` *(upload -> queued -> worker -> transcript/failed)*
|
||||
- `tests/ui/test_upload_page.py` (augment loading/error/ready-state checks as practical)
|
||||
- `tests/ui/test_jobs_page.py` (augment refresh/error behavior checks as practical)
|
||||
- Existing tests touched only when needed; preserve naming/hierarchy unless explicitly approved.
|
||||
|
||||
### Optional docs output
|
||||
- `docs/tests.md` or `docs/verification.md` with lane definitions and command matrix
|
||||
- REQ-to-test mapping table
|
||||
|
||||
---
|
||||
|
||||
## Design and Policy Decisions (MCP-aligned)
|
||||
|
||||
1. **Scaffold-first, fill-second workflow is mandatory**
|
||||
- First create/adjust skeletons and collect.
|
||||
- Then fill test bodies.
|
||||
- Preserve scaffold names/docstrings during fill.
|
||||
|
||||
2. **Deterministic-first default lanes**
|
||||
- `unit` and `integration` run by default.
|
||||
- `external` remains explicit opt-in.
|
||||
|
||||
3. **One behavior target per test**
|
||||
- Short, behavior-focused names.
|
||||
- Precise assertions on observable outcomes.
|
||||
|
||||
4. **Test double discipline (from pytesting skill)**
|
||||
- Prefer real-input/real-object paths first.
|
||||
- If monkeypatch/mocks/fakes are needed for a boundary, keep narrowly scoped.
|
||||
- Avoid call-only assertions.
|
||||
|
||||
5. **NiceGUI responsiveness expectations**
|
||||
- Verify loading/success/error state transitions where testable.
|
||||
- Ensure user-facing feedback behavior is covered.
|
||||
|
||||
6. **FastAPI/ops baseline checks**
|
||||
- Keep `/healthz` route validation in default lanes.
|
||||
- Keep startup/shutdown lifecycle assertions present.
|
||||
|
||||
---
|
||||
|
||||
## Implementation Plan + Checklist
|
||||
|
||||
## Phase A — Coverage and traceability audit
|
||||
|
||||
- [ ] Build a REQ-to-test matrix for MVP requirements:
|
||||
- [ ] REQ-0, REQ-1, REQ-2, REQ-3, REQ-4, REQ-5, REQ-6, REQ-8, REQ-12
|
||||
- [ ] Identify weak spots:
|
||||
- [ ] full pipeline integration (service + worker + persistence)
|
||||
- [ ] UI state transition assertions (loading/error/ready)
|
||||
- [ ] failure-path persistence verification robustness
|
||||
- [ ] Record current baseline command results before edits
|
||||
|
||||
## Phase B — Scaffold phase (pytest-scaffold resources)
|
||||
|
||||
Target modules/areas:
|
||||
- pipeline integration flow
|
||||
- UI behavior augmentations
|
||||
- traceability checks/document validators (if test-backed)
|
||||
|
||||
- [ ] Scaffold new/adjusted test files/classes/methods only
|
||||
- [ ] Keep one-line intent docstrings
|
||||
- [ ] Keep behavior-focused names
|
||||
- [ ] Run: `uv run pytest --collect-only -q`
|
||||
|
||||
## Phase C — Fill phase (pytest-fill-scaffold resources)
|
||||
|
||||
- [ ] Fill scaffolded methods with deterministic setup/assertions
|
||||
- [ ] Preserve scaffold names/hierarchy/docstrings
|
||||
- [ ] Add/adjust fixtures at nearest useful scope
|
||||
- [ ] Keep DB tests in `integration`; pure helper tests in `unit`
|
||||
|
||||
### Required coverage additions
|
||||
|
||||
#### Pipeline integration
|
||||
- [ ] Upload service creates document/job and file path persists
|
||||
- [ ] Worker success path creates transcript and terminal status
|
||||
- [ ] Worker failure path persists error detail and terminal failed status
|
||||
- [ ] Queue-empty behavior remains stable (`False` return / no side effects)
|
||||
|
||||
#### UI behavior (practical, testable boundaries)
|
||||
- [ ] Upload helper flow success and UploadError surfacing
|
||||
- [ ] Jobs data helpers return stable normalized view models
|
||||
- [ ] Refresh/detail fallback behavior for missing/invalid job IDs
|
||||
|
||||
#### Traceability
|
||||
- [ ] Every in-scope MVP REQ has at least one mapped test/assertion point
|
||||
- [ ] Document and/or enforce mapping consistency
|
||||
|
||||
## Phase D — External lane stability
|
||||
|
||||
- [ ] Keep real-image external tests isolated under `@pytest.mark.external`
|
||||
- [ ] Ensure no external test leaks into default runs
|
||||
- [ ] Confirm artifact capture behavior remains stable
|
||||
|
||||
## Phase E — Quality gates and workflow
|
||||
|
||||
- [ ] Confirm logging/lifecycle startup tests still pass after changes
|
||||
- [ ] (If enabled) add/update lint/type check commands in docs:
|
||||
- [ ] Ruff lane (if configured)
|
||||
- [ ] typing lane (if configured)
|
||||
- [ ] Optionally add VS Code task aliases for test lanes (non-blocking)
|
||||
|
||||
---
|
||||
|
||||
## Marker and Fixture Strategy
|
||||
|
||||
- `unit`: pure logic, helper behavior, formatting/normalization
|
||||
- `integration`: DB + service + app lifecycle contracts
|
||||
- `external`: live provider/real image checks only
|
||||
|
||||
Fixture policy:
|
||||
- Prefer reusable fixtures in `tests/conftest.py` only when broadly shared
|
||||
- Use subtree/local fixtures for domain-specific setup
|
||||
- Keep setup explicit and readable
|
||||
|
||||
---
|
||||
|
||||
## Validation Sequence (strict)
|
||||
|
||||
- [ ] `uv run pytest --collect-only -q`
|
||||
- [ ] `uv run pytest -m unit -q`
|
||||
- [ ] `uv run pytest -m integration -q`
|
||||
- [ ] `uv run pytest -m "not external" -q`
|
||||
- [ ] `uv run pytest tests/integration/test_pipeline_flow.py -q` *(if added)*
|
||||
- [ ] `uv run pytest tests/ui/test_upload_page.py -q`
|
||||
- [ ] `uv run pytest tests/ui/test_jobs_page.py -q`
|
||||
- [ ] `uv run pytest -q`
|
||||
|
||||
Optional external verification:
|
||||
- [ ] `uv run pytest -m external -q`
|
||||
|
||||
---
|
||||
|
||||
## Guardrails
|
||||
|
||||
- Do not rename/re-nest scaffolded tests during fill unless explicitly requested.
|
||||
- Do not broaden external dependencies in default lane.
|
||||
- Do not add flaky timing-based assertions; keep deterministic boundaries.
|
||||
- Keep business logic out of UI tests; test through service/helper boundaries.
|
||||
- Preserve one-way dependency direction in test setup patterns.
|
||||
|
||||
---
|
||||
|
||||
## Definition of Done (Step 6)
|
||||
|
||||
- [ ] MVP requirement coverage is explicitly traceable
|
||||
- [ ] Deterministic lanes (`unit` + `integration`) are stable and green
|
||||
- [ ] External lane remains opt-in and green when enabled
|
||||
- [ ] Pipeline success/failure lifecycle paths are verified end-to-end
|
||||
- [ ] UI helper/state behavior has explicit success/error assertions
|
||||
- [ ] Full suite passes with `uv run pytest -q`
|
||||
- [ ] Verification evidence is captured in implementation report
|
||||
|
||||
---
|
||||
|
||||
## PR Checklist (Step 6)
|
||||
|
||||
### Implementation
|
||||
- [ ] Added/updated test files per scoped gaps
|
||||
- [ ] Added REQ traceability mapping
|
||||
- [ ] Kept default lanes deterministic
|
||||
- [ ] Preserved scaffold invariants during fill
|
||||
|
||||
### Testing (MCP-compliant)
|
||||
- [ ] Used scaffold prompt flow first
|
||||
- [ ] Used fill prompt flow second
|
||||
- [ ] Preserved naming/docstrings/hierarchy
|
||||
- [ ] Marker usage documented (`unit`, `integration`, `external`)
|
||||
|
||||
### Evidence
|
||||
- [ ] Collected command outputs in strict order
|
||||
- [ ] Listed files changed
|
||||
- [ ] Listed MCP resources used and why
|
||||
- [ ] Noted residual risks/open questions (if any)
|
||||
Reference in New Issue
Block a user