## Step 6: Test and Verification Hardening (MVP closeout)

## Objective

Complete MVP verification by building a **requirements-traceable, deterministic test strategy** across unit/integration/external lanes, then enforcing stable validation commands and reporting.

This step finalizes the MVP implementation sequence from `docs/mvp.md` (Step 6 in the build order: tests and automated verification).

---

## MCP Resource Integration (what was applied)

I reviewed all top-level skills/prompts from `john-stream-mcp` and integrated the relevant guidance into this plan:

### Directly applied
- `resource://skills/pytesting/document`
- `resource://catalog/prompts/pytest-scaffold`
- `resource://prompts/pytest-scaffold/document`
- `resource://catalog/prompts/pytest-fill-scaffold`
- `resource://prompts/pytest-fill-scaffold/document`
- `resource://skills/nicegui/document`
- `resource://skills/nicegui-ui-customization/document`
- `resource://skills/fastapi-uv-docker/document`
- `resource://skills/python-logging-dictconfig/document`
- `resource://skills/python-typing/document`
- `resource://skills/ruff-linting-formating/document`

### Reviewed but informational/non-blocking for Step 6
- `copilot-customization`, `mcp-details`, `vscode-configuration`, `zensical-docs`, and authoring/shim prompts.
  - These are primarily customization/documentation tooling resources, not core MVP test-lane blockers.
  - Step 6 includes optional workflow follow-ups where relevant (e.g., VS Code task conveniences).

---

## Scope

### In scope
- Strengthen and complete test coverage for the shipped MVP slice (Steps 1–5)
- Add requirement-to-test traceability for REQ-0..REQ-12 (MVP subset emphasized)
- Enforce deterministic default lanes (`unit`, `integration`)
- Keep `external` lane opt-in and isolated
- Validate app/UI/service/worker contracts end-to-end at test level

### Out of scope
- Major architecture rewrites (async SQLAlchemy migration, queue system, etc.)
- Full production deployment rollout
- Post-MVP feature expansion (revision history, search, export)

---

## Planned Deliverables

### Test files (new/updated)
- `tests/test_traceability.py` *(or docs-based traceability matrix if preferred)*
- `tests/integration/test_pipeline_flow.py` *(upload -> queued -> worker -> transcript/failed)*
- `tests/ui/test_upload_page.py` (augment loading/error/ready-state checks as practical)
- `tests/ui/test_jobs_page.py` (augment refresh/error behavior checks as practical)
- Existing tests touched only when needed; preserve naming/hierarchy unless explicitly approved.

### Optional docs output
- `docs/tests.md` or `docs/verification.md` with lane definitions and command matrix
- REQ-to-test mapping table

---

## Design and Policy Decisions (MCP-aligned)

1. **Scaffold-first, fill-second workflow is mandatory**
   - First create/adjust skeletons and collect.
   - Then fill test bodies.
   - Preserve scaffold names/docstrings during fill.

2. **Deterministic-first default lanes**
   - `unit` and `integration` run by default.
   - `external` remains explicit opt-in.

3. **One behavior target per test**
   - Short, behavior-focused names.
   - Precise assertions on observable outcomes.

4. **Test double discipline (from pytesting skill)**
   - Prefer real-input/real-object paths first.
   - If monkeypatch/mocks/fakes are needed for a boundary, keep narrowly scoped.
   - Avoid call-only assertions.

5. **NiceGUI responsiveness expectations**
   - Verify loading/success/error state transitions where testable.
   - Ensure user-facing feedback behavior is covered.

6. **FastAPI/ops baseline checks**
   - Keep `/healthz` route validation in default lanes.
   - Keep startup/shutdown lifecycle assertions present.

---

## Implementation Plan + Checklist

## Phase A — Coverage and traceability audit

- [ ] Build a REQ-to-test matrix for MVP requirements:
  - [ ] REQ-0, REQ-1, REQ-2, REQ-3, REQ-4, REQ-5, REQ-6, REQ-8, REQ-12
- [ ] Identify weak spots:
  - [ ] full pipeline integration (service + worker + persistence)
  - [ ] UI state transition assertions (loading/error/ready)
  - [ ] failure-path persistence verification robustness
- [ ] Record current baseline command results before edits

## Phase B — Scaffold phase (pytest-scaffold resources)

Target modules/areas:
- pipeline integration flow
- UI behavior augmentations
- traceability checks/document validators (if test-backed)

- [ ] Scaffold new/adjusted test files/classes/methods only
- [ ] Keep one-line intent docstrings
- [ ] Keep behavior-focused names
- [ ] Run: `uv run pytest --collect-only -q`

## Phase C — Fill phase (pytest-fill-scaffold resources)

- [ ] Fill scaffolded methods with deterministic setup/assertions
- [ ] Preserve scaffold names/hierarchy/docstrings
- [ ] Add/adjust fixtures at nearest useful scope
- [ ] Keep DB tests in `integration`; pure helper tests in `unit`

### Required coverage additions

#### Pipeline integration
- [ ] Upload service creates document/job and file path persists
- [ ] Worker success path creates transcript and terminal status
- [ ] Worker failure path persists error detail and terminal failed status
- [ ] Queue-empty behavior remains stable (`False` return / no side effects)

#### UI behavior (practical, testable boundaries)
- [ ] Upload helper flow success and UploadError surfacing
- [ ] Jobs data helpers return stable normalized view models
- [ ] Refresh/detail fallback behavior for missing/invalid job IDs

#### Traceability
- [ ] Every in-scope MVP REQ has at least one mapped test/assertion point
- [ ] Document and/or enforce mapping consistency

## Phase D — External lane stability

- [ ] Keep real-image external tests isolated under `@pytest.mark.external`
- [ ] Ensure no external test leaks into default runs
- [ ] Confirm artifact capture behavior remains stable

## Phase E — Quality gates and workflow

- [ ] Confirm logging/lifecycle startup tests still pass after changes
- [ ] (If enabled) add/update lint/type check commands in docs:
  - [ ] Ruff lane (if configured)
  - [ ] typing lane (if configured)
- [ ] Optionally add VS Code task aliases for test lanes (non-blocking)

---

## Marker and Fixture Strategy

- `unit`: pure logic, helper behavior, formatting/normalization
- `integration`: DB + service + app lifecycle contracts
- `external`: live provider/real image checks only

Fixture policy:
- Prefer reusable fixtures in `tests/conftest.py` only when broadly shared
- Use subtree/local fixtures for domain-specific setup
- Keep setup explicit and readable

---

## Validation Sequence (strict)

- [ ] `uv run pytest --collect-only -q`
- [ ] `uv run pytest -m unit -q`
- [ ] `uv run pytest -m integration -q`
- [ ] `uv run pytest -m "not external" -q`
- [ ] `uv run pytest tests/integration/test_pipeline_flow.py -q` *(if added)*
- [ ] `uv run pytest tests/ui/test_upload_page.py -q`
- [ ] `uv run pytest tests/ui/test_jobs_page.py -q`
- [ ] `uv run pytest -q`

Optional external verification:
- [ ] `uv run pytest -m external -q`

---

## Guardrails

- Do not rename/re-nest scaffolded tests during fill unless explicitly requested.
- Do not broaden external dependencies in default lane.
- Do not add flaky timing-based assertions; keep deterministic boundaries.
- Keep business logic out of UI tests; test through service/helper boundaries.
- Preserve one-way dependency direction in test setup patterns.

---

## Definition of Done (Step 6)

- [ ] MVP requirement coverage is explicitly traceable
- [ ] Deterministic lanes (`unit` + `integration`) are stable and green
- [ ] External lane remains opt-in and green when enabled
- [ ] Pipeline success/failure lifecycle paths are verified end-to-end
- [ ] UI helper/state behavior has explicit success/error assertions
- [ ] Full suite passes with `uv run pytest -q`
- [ ] Verification evidence is captured in implementation report

---

## PR Checklist (Step 6)

### Implementation
- [ ] Added/updated test files per scoped gaps
- [ ] Added REQ traceability mapping
- [ ] Kept default lanes deterministic
- [ ] Preserved scaffold invariants during fill

### Testing (MCP-compliant)
- [ ] Used scaffold prompt flow first
- [ ] Used fill prompt flow second
- [ ] Preserved naming/docstrings/hierarchy
- [ ] Marker usage documented (`unit`, `integration`, `external`)

### Evidence
- [ ] Collected command outputs in strict order
- [ ] Listed files changed
- [ ] Listed MCP resources used and why
- [ ] Noted residual risks/open questions (if any)