transcription/docs/step6.md

Step 6: Test and Verification Hardening (MVP closeout)

Objective

Complete MVP verification by building a requirements-traceable, deterministic test strategy across unit/integration/external lanes, then enforcing stable validation commands and reporting.

This step finalizes the MVP implementation sequence from docs/mvp.md (Step 6 in the build order: tests and automated verification).

---

MCP Resource Integration (what was applied)

I reviewed all top-level skills/prompts from john-stream-mcp and integrated the relevant guidance into this plan:

Directly applied

• resource://skills/pytesting/document
• resource://catalog/prompts/pytest-scaffold
• resource://prompts/pytest-scaffold/document
• resource://catalog/prompts/pytest-fill-scaffold
• resource://prompts/pytest-fill-scaffold/document
• resource://skills/nicegui/document
• resource://skills/nicegui-ui-customization/document
• resource://skills/fastapi-uv-docker/document
• resource://skills/python-logging-dictconfig/document
• resource://skills/python-typing/document
• resource://skills/ruff-linting-formating/document

Reviewed but informational/non-blocking for Step 6

• copilot-customization, mcp-details, vscode-configuration, zensical-docs, and authoring/shim prompts.
  • These are primarily customization/documentation tooling resources, not core MVP test-lane blockers.
  • Step 6 includes optional workflow follow-ups where relevant (e.g., VS Code task conveniences).

---

Scope

In scope

• Strengthen and complete test coverage for the shipped MVP slice (Steps 1–5)
• Add requirement-to-test traceability for REQ-0..REQ-12 (MVP subset emphasized)
• Enforce deterministic default lanes (unit, integration)
• Keep external lane opt-in and isolated
• Validate app/UI/service/worker contracts end-to-end at test level

Out of scope

• Major architecture rewrites (async SQLAlchemy migration, queue system, etc.)
• Full production deployment rollout
• Post-MVP feature expansion (revision history, search, export)

---

Planned Deliverables

Test files (new/updated)

• tests/test_traceability.py (or docs-based traceability matrix if preferred)
• tests/integration/test_pipeline_flow.py (upload -> queued -> worker -> transcript/failed)
• tests/ui/test_upload_page.py (augment loading/error/ready-state checks as practical)
• tests/ui/test_jobs_page.py (augment refresh/error behavior checks as practical)
• Existing tests touched only when needed; preserve naming/hierarchy unless explicitly approved.

Optional docs output

• docs/tests.md or docs/verification.md with lane definitions and command matrix
• REQ-to-test mapping table

---

Design and Policy Decisions (MCP-aligned)

1. Scaffold-first, fill-second workflow is mandatory

   • First create/adjust skeletons and collect.
   • Then fill test bodies.
   • Preserve scaffold names/docstrings during fill.

2. Deterministic-first default lanes

   • unit and integration run by default.
   • external remains explicit opt-in.

3. One behavior target per test

   • Short, behavior-focused names.
   • Precise assertions on observable outcomes.

4. Test double discipline (from pytesting skill)

   • Prefer real-input/real-object paths first.
   • If monkeypatch/mocks/fakes are needed for a boundary, keep narrowly scoped.
   • Avoid call-only assertions.

5. NiceGUI responsiveness expectations

   • Verify loading/success/error state transitions where testable.
   • Ensure user-facing feedback behavior is covered.

6. FastAPI/ops baseline checks

   • Keep /healthz route validation in default lanes.
   • Keep startup/shutdown lifecycle assertions present.

---

Implementation Plan + Checklist

Phase A — Coverage and traceability audit

• Build a REQ-to-test matrix for MVP requirements:
  • REQ-0, REQ-1, REQ-2, REQ-3, REQ-4, REQ-5, REQ-6, REQ-8, REQ-12
• Identify weak spots:
  • full pipeline integration (service + worker + persistence)
  • UI state transition assertions (loading/error/ready)
  • failure-path persistence verification robustness
• Record current baseline command results before edits

Phase B — Scaffold phase (pytest-scaffold resources)

Target modules/areas:

• pipeline integration flow

• UI behavior augmentations

• traceability checks/document validators (if test-backed)

• Scaffold new/adjusted test files/classes/methods only

• Keep one-line intent docstrings

• Keep behavior-focused names

• Run: uv run pytest --collect-only -q

Phase C — Fill phase (pytest-fill-scaffold resources)

• Fill scaffolded methods with deterministic setup/assertions
• Preserve scaffold names/hierarchy/docstrings
• Add/adjust fixtures at nearest useful scope
• Keep DB tests in integration; pure helper tests in unit

Required coverage additions

Pipeline integration

• Upload service creates document/job and file path persists
• Worker success path creates transcript and terminal status
• Worker failure path persists error detail and terminal failed status
• Queue-empty behavior remains stable (False return / no side effects)

UI behavior (practical, testable boundaries)

• Upload helper flow success and UploadError surfacing
• Jobs data helpers return stable normalized view models
• Refresh/detail fallback behavior for missing/invalid job IDs

Traceability

• Every in-scope MVP REQ has at least one mapped test/assertion point
• Document and/or enforce mapping consistency

Phase D — External lane stability

• Keep real-image external tests isolated under @pytest.mark.external
• Ensure no external test leaks into default runs
• Confirm artifact capture behavior remains stable

Phase E — Quality gates and workflow

• Confirm logging/lifecycle startup tests still pass after changes
• (If enabled) add/update lint/type check commands in docs:
  • Ruff lane (if configured)
  • typing lane (if configured)
• Optionally add VS Code task aliases for test lanes (non-blocking)

---

Marker and Fixture Strategy

• unit: pure logic, helper behavior, formatting/normalization
• integration: DB + service + app lifecycle contracts
• external: live provider/real image checks only

Fixture policy:

• Prefer reusable fixtures in tests/conftest.py only when broadly shared
• Use subtree/local fixtures for domain-specific setup
• Keep setup explicit and readable

---

Validation Sequence (strict)

• uv run pytest --collect-only -q
• uv run pytest -m unit -q
• uv run pytest -m integration -q
• uv run pytest -m "not external" -q
• uv run pytest tests/integration/test_pipeline_flow.py -q (if added)
• uv run pytest tests/ui/test_upload_page.py -q
• uv run pytest tests/ui/test_jobs_page.py -q
• uv run pytest -q

Optional external verification:

• uv run pytest -m external -q

---

Guardrails

• Do not rename/re-nest scaffolded tests during fill unless explicitly requested.
• Do not broaden external dependencies in default lane.
• Do not add flaky timing-based assertions; keep deterministic boundaries.
• Keep business logic out of UI tests; test through service/helper boundaries.
• Preserve one-way dependency direction in test setup patterns.

---

Definition of Done (Step 6)

• MVP requirement coverage is explicitly traceable
• Deterministic lanes (unit + integration) are stable and green
• External lane remains opt-in and green when enabled
• Pipeline success/failure lifecycle paths are verified end-to-end
• UI helper/state behavior has explicit success/error assertions
• Full suite passes with uv run pytest -q
• Verification evidence is captured in implementation report

---

PR Checklist (Step 6)

Implementation

• Added/updated test files per scoped gaps
• Added REQ traceability mapping
• Kept default lanes deterministic
• Preserved scaffold invariants during fill

Testing (MCP-compliant)

• Used scaffold prompt flow first
• Used fill prompt flow second
• Preserved naming/docstrings/hierarchy
• Marker usage documented (unit, integration, external)

Evidence

• Collected command outputs in strict order
• Listed files changed
• Listed MCP resources used and why
• Noted residual risks/open questions (if any)