From 759c8c2739151721bc1a6053fc178d376a2c1de1 Mon Sep 17 00:00:00 2001 From: Jim Lancaster <40281233+zoltan57@users.noreply.github.com> Date: Thu, 25 Jun 2026 10:28:43 -0500 Subject: [PATCH] Step 6 implementation plan --- docs/step6.md | 229 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 229 insertions(+) create mode 100644 docs/step6.md diff --git a/docs/step6.md b/docs/step6.md new file mode 100644 index 0000000..2ae4f1a --- /dev/null +++ b/docs/step6.md @@ -0,0 +1,229 @@ +## Step 6: Test and Verification Hardening (MVP closeout) + +## Objective + +Complete MVP verification by building a **requirements-traceable, deterministic test strategy** across unit/integration/external lanes, then enforcing stable validation commands and reporting. + +This step finalizes the MVP implementation sequence from `docs/mvp.md` (Step 6 in the build order: tests and automated verification). + +--- + +## MCP Resource Integration (what was applied) + +I reviewed all top-level skills/prompts from `john-stream-mcp` and integrated the relevant guidance into this plan: + +### Directly applied +- `resource://skills/pytesting/document` +- `resource://catalog/prompts/pytest-scaffold` +- `resource://prompts/pytest-scaffold/document` +- `resource://catalog/prompts/pytest-fill-scaffold` +- `resource://prompts/pytest-fill-scaffold/document` +- `resource://skills/nicegui/document` +- `resource://skills/nicegui-ui-customization/document` +- `resource://skills/fastapi-uv-docker/document` +- `resource://skills/python-logging-dictconfig/document` +- `resource://skills/python-typing/document` +- `resource://skills/ruff-linting-formating/document` + +### Reviewed but informational/non-blocking for Step 6 +- `copilot-customization`, `mcp-details`, `vscode-configuration`, `zensical-docs`, and authoring/shim prompts. + - These are primarily customization/documentation tooling resources, not core MVP test-lane blockers. + - Step 6 includes optional workflow follow-ups where relevant (e.g., VS Code task conveniences). + +--- + +## Scope + +### In scope +- Strengthen and complete test coverage for the shipped MVP slice (Steps 1–5) +- Add requirement-to-test traceability for REQ-0..REQ-12 (MVP subset emphasized) +- Enforce deterministic default lanes (`unit`, `integration`) +- Keep `external` lane opt-in and isolated +- Validate app/UI/service/worker contracts end-to-end at test level + +### Out of scope +- Major architecture rewrites (async SQLAlchemy migration, queue system, etc.) +- Full production deployment rollout +- Post-MVP feature expansion (revision history, search, export) + +--- + +## Planned Deliverables + +### Test files (new/updated) +- `tests/test_traceability.py` *(or docs-based traceability matrix if preferred)* +- `tests/integration/test_pipeline_flow.py` *(upload -> queued -> worker -> transcript/failed)* +- `tests/ui/test_upload_page.py` (augment loading/error/ready-state checks as practical) +- `tests/ui/test_jobs_page.py` (augment refresh/error behavior checks as practical) +- Existing tests touched only when needed; preserve naming/hierarchy unless explicitly approved. + +### Optional docs output +- `docs/tests.md` or `docs/verification.md` with lane definitions and command matrix +- REQ-to-test mapping table + +--- + +## Design and Policy Decisions (MCP-aligned) + +1. **Scaffold-first, fill-second workflow is mandatory** + - First create/adjust skeletons and collect. + - Then fill test bodies. + - Preserve scaffold names/docstrings during fill. + +2. **Deterministic-first default lanes** + - `unit` and `integration` run by default. + - `external` remains explicit opt-in. + +3. **One behavior target per test** + - Short, behavior-focused names. + - Precise assertions on observable outcomes. + +4. **Test double discipline (from pytesting skill)** + - Prefer real-input/real-object paths first. + - If monkeypatch/mocks/fakes are needed for a boundary, keep narrowly scoped. + - Avoid call-only assertions. + +5. **NiceGUI responsiveness expectations** + - Verify loading/success/error state transitions where testable. + - Ensure user-facing feedback behavior is covered. + +6. **FastAPI/ops baseline checks** + - Keep `/healthz` route validation in default lanes. + - Keep startup/shutdown lifecycle assertions present. + +--- + +## Implementation Plan + Checklist + +## Phase A — Coverage and traceability audit + +- [ ] Build a REQ-to-test matrix for MVP requirements: + - [ ] REQ-0, REQ-1, REQ-2, REQ-3, REQ-4, REQ-5, REQ-6, REQ-8, REQ-12 +- [ ] Identify weak spots: + - [ ] full pipeline integration (service + worker + persistence) + - [ ] UI state transition assertions (loading/error/ready) + - [ ] failure-path persistence verification robustness +- [ ] Record current baseline command results before edits + +## Phase B — Scaffold phase (pytest-scaffold resources) + +Target modules/areas: +- pipeline integration flow +- UI behavior augmentations +- traceability checks/document validators (if test-backed) + +- [ ] Scaffold new/adjusted test files/classes/methods only +- [ ] Keep one-line intent docstrings +- [ ] Keep behavior-focused names +- [ ] Run: `uv run pytest --collect-only -q` + +## Phase C — Fill phase (pytest-fill-scaffold resources) + +- [ ] Fill scaffolded methods with deterministic setup/assertions +- [ ] Preserve scaffold names/hierarchy/docstrings +- [ ] Add/adjust fixtures at nearest useful scope +- [ ] Keep DB tests in `integration`; pure helper tests in `unit` + +### Required coverage additions + +#### Pipeline integration +- [ ] Upload service creates document/job and file path persists +- [ ] Worker success path creates transcript and terminal status +- [ ] Worker failure path persists error detail and terminal failed status +- [ ] Queue-empty behavior remains stable (`False` return / no side effects) + +#### UI behavior (practical, testable boundaries) +- [ ] Upload helper flow success and UploadError surfacing +- [ ] Jobs data helpers return stable normalized view models +- [ ] Refresh/detail fallback behavior for missing/invalid job IDs + +#### Traceability +- [ ] Every in-scope MVP REQ has at least one mapped test/assertion point +- [ ] Document and/or enforce mapping consistency + +## Phase D — External lane stability + +- [ ] Keep real-image external tests isolated under `@pytest.mark.external` +- [ ] Ensure no external test leaks into default runs +- [ ] Confirm artifact capture behavior remains stable + +## Phase E — Quality gates and workflow + +- [ ] Confirm logging/lifecycle startup tests still pass after changes +- [ ] (If enabled) add/update lint/type check commands in docs: + - [ ] Ruff lane (if configured) + - [ ] typing lane (if configured) +- [ ] Optionally add VS Code task aliases for test lanes (non-blocking) + +--- + +## Marker and Fixture Strategy + +- `unit`: pure logic, helper behavior, formatting/normalization +- `integration`: DB + service + app lifecycle contracts +- `external`: live provider/real image checks only + +Fixture policy: +- Prefer reusable fixtures in `tests/conftest.py` only when broadly shared +- Use subtree/local fixtures for domain-specific setup +- Keep setup explicit and readable + +--- + +## Validation Sequence (strict) + +- [ ] `uv run pytest --collect-only -q` +- [ ] `uv run pytest -m unit -q` +- [ ] `uv run pytest -m integration -q` +- [ ] `uv run pytest -m "not external" -q` +- [ ] `uv run pytest tests/integration/test_pipeline_flow.py -q` *(if added)* +- [ ] `uv run pytest tests/ui/test_upload_page.py -q` +- [ ] `uv run pytest tests/ui/test_jobs_page.py -q` +- [ ] `uv run pytest -q` + +Optional external verification: +- [ ] `uv run pytest -m external -q` + +--- + +## Guardrails + +- Do not rename/re-nest scaffolded tests during fill unless explicitly requested. +- Do not broaden external dependencies in default lane. +- Do not add flaky timing-based assertions; keep deterministic boundaries. +- Keep business logic out of UI tests; test through service/helper boundaries. +- Preserve one-way dependency direction in test setup patterns. + +--- + +## Definition of Done (Step 6) + +- [ ] MVP requirement coverage is explicitly traceable +- [ ] Deterministic lanes (`unit` + `integration`) are stable and green +- [ ] External lane remains opt-in and green when enabled +- [ ] Pipeline success/failure lifecycle paths are verified end-to-end +- [ ] UI helper/state behavior has explicit success/error assertions +- [ ] Full suite passes with `uv run pytest -q` +- [ ] Verification evidence is captured in implementation report + +--- + +## PR Checklist (Step 6) + +### Implementation +- [ ] Added/updated test files per scoped gaps +- [ ] Added REQ traceability mapping +- [ ] Kept default lanes deterministic +- [ ] Preserved scaffold invariants during fill + +### Testing (MCP-compliant) +- [ ] Used scaffold prompt flow first +- [ ] Used fill prompt flow second +- [ ] Preserved naming/docstrings/hierarchy +- [ ] Marker usage documented (`unit`, `integration`, `external`) + +### Evidence +- [ ] Collected command outputs in strict order +- [ ] Listed files changed +- [ ] Listed MCP resources used and why +- [ ] Noted residual risks/open questions (if any)