generated from john/python-template
230 lines
8.3 KiB
Markdown
230 lines
8.3 KiB
Markdown
## Step 6: Test and Verification Hardening (MVP closeout)
|
||
|
||
## Objective
|
||
|
||
Complete MVP verification by building a **requirements-traceable, deterministic test strategy** across unit/integration/external lanes, then enforcing stable validation commands and reporting.
|
||
|
||
This step finalizes the MVP implementation sequence from `docs/mvp.md` (Step 6 in the build order: tests and automated verification).
|
||
|
||
---
|
||
|
||
## MCP Resource Integration (what was applied)
|
||
|
||
I reviewed all top-level skills/prompts from `john-stream-mcp` and integrated the relevant guidance into this plan:
|
||
|
||
### Directly applied
|
||
- `resource://skills/pytesting/document`
|
||
- `resource://catalog/prompts/pytest-scaffold`
|
||
- `resource://prompts/pytest-scaffold/document`
|
||
- `resource://catalog/prompts/pytest-fill-scaffold`
|
||
- `resource://prompts/pytest-fill-scaffold/document`
|
||
- `resource://skills/nicegui/document`
|
||
- `resource://skills/nicegui-ui-customization/document`
|
||
- `resource://skills/fastapi-uv-docker/document`
|
||
- `resource://skills/python-logging-dictconfig/document`
|
||
- `resource://skills/python-typing/document`
|
||
- `resource://skills/ruff-linting-formating/document`
|
||
|
||
### Reviewed but informational/non-blocking for Step 6
|
||
- `copilot-customization`, `mcp-details`, `vscode-configuration`, `zensical-docs`, and authoring/shim prompts.
|
||
- These are primarily customization/documentation tooling resources, not core MVP test-lane blockers.
|
||
- Step 6 includes optional workflow follow-ups where relevant (e.g., VS Code task conveniences).
|
||
|
||
---
|
||
|
||
## Scope
|
||
|
||
### In scope
|
||
- Strengthen and complete test coverage for the shipped MVP slice (Steps 1–5)
|
||
- Add requirement-to-test traceability for REQ-0..REQ-12 (MVP subset emphasized)
|
||
- Enforce deterministic default lanes (`unit`, `integration`)
|
||
- Keep `external` lane opt-in and isolated
|
||
- Validate app/UI/service/worker contracts end-to-end at test level
|
||
|
||
### Out of scope
|
||
- Major architecture rewrites (async SQLAlchemy migration, queue system, etc.)
|
||
- Full production deployment rollout
|
||
- Post-MVP feature expansion (revision history, search, export)
|
||
|
||
---
|
||
|
||
## Planned Deliverables
|
||
|
||
### Test files (new/updated)
|
||
- `tests/test_traceability.py` *(or docs-based traceability matrix if preferred)*
|
||
- `tests/integration/test_pipeline_flow.py` *(upload -> queued -> worker -> transcript/failed)*
|
||
- `tests/ui/test_upload_page.py` (augment loading/error/ready-state checks as practical)
|
||
- `tests/ui/test_jobs_page.py` (augment refresh/error behavior checks as practical)
|
||
- Existing tests touched only when needed; preserve naming/hierarchy unless explicitly approved.
|
||
|
||
### Optional docs output
|
||
- `docs/tests.md` or `docs/verification.md` with lane definitions and command matrix
|
||
- REQ-to-test mapping table
|
||
|
||
---
|
||
|
||
## Design and Policy Decisions (MCP-aligned)
|
||
|
||
1. **Scaffold-first, fill-second workflow is mandatory**
|
||
- First create/adjust skeletons and collect.
|
||
- Then fill test bodies.
|
||
- Preserve scaffold names/docstrings during fill.
|
||
|
||
2. **Deterministic-first default lanes**
|
||
- `unit` and `integration` run by default.
|
||
- `external` remains explicit opt-in.
|
||
|
||
3. **One behavior target per test**
|
||
- Short, behavior-focused names.
|
||
- Precise assertions on observable outcomes.
|
||
|
||
4. **Test double discipline (from pytesting skill)**
|
||
- Prefer real-input/real-object paths first.
|
||
- If monkeypatch/mocks/fakes are needed for a boundary, keep narrowly scoped.
|
||
- Avoid call-only assertions.
|
||
|
||
5. **NiceGUI responsiveness expectations**
|
||
- Verify loading/success/error state transitions where testable.
|
||
- Ensure user-facing feedback behavior is covered.
|
||
|
||
6. **FastAPI/ops baseline checks**
|
||
- Keep `/healthz` route validation in default lanes.
|
||
- Keep startup/shutdown lifecycle assertions present.
|
||
|
||
---
|
||
|
||
## Implementation Plan + Checklist
|
||
|
||
## Phase A — Coverage and traceability audit
|
||
|
||
- [ ] Build a REQ-to-test matrix for MVP requirements:
|
||
- [ ] REQ-0, REQ-1, REQ-2, REQ-3, REQ-4, REQ-5, REQ-6, REQ-8, REQ-12
|
||
- [ ] Identify weak spots:
|
||
- [ ] full pipeline integration (service + worker + persistence)
|
||
- [ ] UI state transition assertions (loading/error/ready)
|
||
- [ ] failure-path persistence verification robustness
|
||
- [ ] Record current baseline command results before edits
|
||
|
||
## Phase B — Scaffold phase (pytest-scaffold resources)
|
||
|
||
Target modules/areas:
|
||
- pipeline integration flow
|
||
- UI behavior augmentations
|
||
- traceability checks/document validators (if test-backed)
|
||
|
||
- [ ] Scaffold new/adjusted test files/classes/methods only
|
||
- [ ] Keep one-line intent docstrings
|
||
- [ ] Keep behavior-focused names
|
||
- [ ] Run: `uv run pytest --collect-only -q`
|
||
|
||
## Phase C — Fill phase (pytest-fill-scaffold resources)
|
||
|
||
- [ ] Fill scaffolded methods with deterministic setup/assertions
|
||
- [ ] Preserve scaffold names/hierarchy/docstrings
|
||
- [ ] Add/adjust fixtures at nearest useful scope
|
||
- [ ] Keep DB tests in `integration`; pure helper tests in `unit`
|
||
|
||
### Required coverage additions
|
||
|
||
#### Pipeline integration
|
||
- [ ] Upload service creates document/job and file path persists
|
||
- [ ] Worker success path creates transcript and terminal status
|
||
- [ ] Worker failure path persists error detail and terminal failed status
|
||
- [ ] Queue-empty behavior remains stable (`False` return / no side effects)
|
||
|
||
#### UI behavior (practical, testable boundaries)
|
||
- [ ] Upload helper flow success and UploadError surfacing
|
||
- [ ] Jobs data helpers return stable normalized view models
|
||
- [ ] Refresh/detail fallback behavior for missing/invalid job IDs
|
||
|
||
#### Traceability
|
||
- [ ] Every in-scope MVP REQ has at least one mapped test/assertion point
|
||
- [ ] Document and/or enforce mapping consistency
|
||
|
||
## Phase D — External lane stability
|
||
|
||
- [ ] Keep real-image external tests isolated under `@pytest.mark.external`
|
||
- [ ] Ensure no external test leaks into default runs
|
||
- [ ] Confirm artifact capture behavior remains stable
|
||
|
||
## Phase E — Quality gates and workflow
|
||
|
||
- [ ] Confirm logging/lifecycle startup tests still pass after changes
|
||
- [ ] (If enabled) add/update lint/type check commands in docs:
|
||
- [ ] Ruff lane (if configured)
|
||
- [ ] typing lane (if configured)
|
||
- [ ] Optionally add VS Code task aliases for test lanes (non-blocking)
|
||
|
||
---
|
||
|
||
## Marker and Fixture Strategy
|
||
|
||
- `unit`: pure logic, helper behavior, formatting/normalization
|
||
- `integration`: DB + service + app lifecycle contracts
|
||
- `external`: live provider/real image checks only
|
||
|
||
Fixture policy:
|
||
- Prefer reusable fixtures in `tests/conftest.py` only when broadly shared
|
||
- Use subtree/local fixtures for domain-specific setup
|
||
- Keep setup explicit and readable
|
||
|
||
---
|
||
|
||
## Validation Sequence (strict)
|
||
|
||
- [ ] `uv run pytest --collect-only -q`
|
||
- [ ] `uv run pytest -m unit -q`
|
||
- [ ] `uv run pytest -m integration -q`
|
||
- [ ] `uv run pytest -m "not external" -q`
|
||
- [ ] `uv run pytest tests/integration/test_pipeline_flow.py -q` *(if added)*
|
||
- [ ] `uv run pytest tests/ui/test_upload_page.py -q`
|
||
- [ ] `uv run pytest tests/ui/test_jobs_page.py -q`
|
||
- [ ] `uv run pytest -q`
|
||
|
||
Optional external verification:
|
||
- [ ] `uv run pytest -m external -q`
|
||
|
||
---
|
||
|
||
## Guardrails
|
||
|
||
- Do not rename/re-nest scaffolded tests during fill unless explicitly requested.
|
||
- Do not broaden external dependencies in default lane.
|
||
- Do not add flaky timing-based assertions; keep deterministic boundaries.
|
||
- Keep business logic out of UI tests; test through service/helper boundaries.
|
||
- Preserve one-way dependency direction in test setup patterns.
|
||
|
||
---
|
||
|
||
## Definition of Done (Step 6)
|
||
|
||
- [ ] MVP requirement coverage is explicitly traceable
|
||
- [ ] Deterministic lanes (`unit` + `integration`) are stable and green
|
||
- [ ] External lane remains opt-in and green when enabled
|
||
- [ ] Pipeline success/failure lifecycle paths are verified end-to-end
|
||
- [ ] UI helper/state behavior has explicit success/error assertions
|
||
- [ ] Full suite passes with `uv run pytest -q`
|
||
- [ ] Verification evidence is captured in implementation report
|
||
|
||
---
|
||
|
||
## PR Checklist (Step 6)
|
||
|
||
### Implementation
|
||
- [ ] Added/updated test files per scoped gaps
|
||
- [ ] Added REQ traceability mapping
|
||
- [ ] Kept default lanes deterministic
|
||
- [ ] Preserved scaffold invariants during fill
|
||
|
||
### Testing (MCP-compliant)
|
||
- [ ] Used scaffold prompt flow first
|
||
- [ ] Used fill prompt flow second
|
||
- [ ] Preserved naming/docstrings/hierarchy
|
||
- [ ] Marker usage documented (`unit`, `integration`, `external`)
|
||
|
||
### Evidence
|
||
- [ ] Collected command outputs in strict order
|
||
- [ ] Listed files changed
|
||
- [ ] Listed MCP resources used and why
|
||
- [ ] Noted residual risks/open questions (if any)
|