john/prompts
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

pytest improvements

Browse Source
This commit is contained in:
John Lancaster
2026-06-19 17:22:34 -05:00
parent 75b0c8d192
commit ed6068f398
4 changed files with 606 additions and 108 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/skills/pytest-scaffolding/SKILL.md
+89 -108
View File
@@ -1,136 +1,117 @@
---
name: pytest-scaffolding
description: "Scaffold a maintainable, hierarchical pytest suite for core functionality first, then extend safely. Use when setting up tests, organizing fixtures by dependency, mirroring src structure in tests, or enforcing fast-by-default test runs."
argument-hint: "Target scope (for example: app/services/job, app/ai, or full repo)"
description: "Scaffold a maintainable, hierarchical pytest suite with fast defaults and clear escalation paths for FastAPI and SQLAlchemy tests. Use when creating or reorganizing tests, defining fixture/marker boundaries, or making test strategy progressively discoverable."
argument-hint: "Target scope plus stack details (pure Python, FastAPI, SQLAlchemy sync, SQLAlchemy async, or mixed)"
---
# Pytest Scaffolding
Create test scaffolding that is:
- Hierarchical: test layout roughly mirrors source layout.
- Fast by default: most tests run in under a second total for core units.
- Dependency-aware: slow/external dependencies are isolated behind markers and fixture scope.
- Extensible: minimal initial skeleton supports adding detailed tests later without refactors.
Create test scaffolding that stays fast for daily work and scales safely as dependencies increase.
This repository currently uses:
- `uv run pytest` as the canonical test invocation.
- `pyproject.toml` pytest config under `[tool.pytest.ini_options]`.
- strict marker checking (`--strict-markers`).
This skill is optimized for progressive discoverability:
1. Start with the shortest path in this file.
2. Load exactly one deeper reference only when a decision requires it.
3. Continue only as far as needed for the current task.
Load [pytest references](./references/pytest-docs.md) when you need detailed rules.
Repository defaults:
- `uv run pytest` is the canonical invocation.
- pytest settings live in `pyproject.toml` under `[tool.pytest.ini_options]`.
- strict marker checking is expected (`--strict-markers`).
## When To Use
- Bootstrapping tests for a new or existing Python repo.
- Reorganizing tests that have become flat, slow, or difficult to extend.
- Defining fixture boundaries before writing many assertions.
- Creating only the first-layer scaffold for core behavior (not exhaustive coverage yet).
## Discovery Ladder
## Inputs To Collect
1. Target test scope: full repo, package, or module.
2. Dependency profile: pure Python, DB, network/API, filesystem, UI/browser.
3. Runtime expectation: what must be instant vs allowed to be slower.
4. CI policy: which marker groups must block merges.
### Level 0: Scope And Stack Triage (always)
Collect:
1. Target scope (repo, package, module).
2. Stack shape (pure Python, FastAPI, SQLAlchemy sync, SQLAlchemy async, or mixed).
3. Speed target (what must stay instant).
4. CI gate policy (which marker groups block merge).
If these are missing, ask concise clarifying questions before editing.
If any are missing, ask concise clarifying questions before scaffolding.
## Workflow
1. Map source tree to test tree.
2. Classify tests by dependency cost.
3. Create minimal directories and placeholder test modules.
4. Create fixture layers (`tests/conftest.py` plus local `conftest.py` in subtrees only when needed).
5. Register markers and default selection behavior.
6. Run collection and fast path tests.
7. Report gaps and next extension points.
### Level 1: Core pytest scaffold (default)
Use this for all stacks first:
1. Mirror `src/` into `tests/` with one starter file per core module.
2. Classify test intent by cost:
- `unit`: no DB/network/filesystem side effects.
- `integration`: framework, DB, or multi-layer contracts.
- `smoke`: thin critical-path checks.
3. Scaffold each new module with:
- one happy-path test,
- one failure/edge test,
- TODO anchors for deeper assertions.
4. Keep fixtures layered:
- global lightweight fixtures in `tests/conftest.py`,
- domain fixtures in subtree `conftest.py` only when needed.
5. Register markers early: `unit`, `integration`, `smoke`, `slow`, `external`.
6. Validate in order:
- `uv run pytest --collect-only -q`
- `uv run pytest -m unit -q`
- `uv run pytest -q` when dependencies are available.
## Step 1: Map Source To Tests
Create a mirrored structure rooted at `tests/` that follows major source concepts.
Load next reference only if needed:
- Baseline details and rationale: [pytest-docs.md](./references/pytest-docs.md)
- Condensed quick path variant: [pytest-docs copy.md](./references/pytest-docs copy.md)
Example mapping pattern:
- `src/app/services/job.py` -> `tests/app/services/test_job.py`
- `src/app/ai/graphs/transcription.py` -> `tests/app/ai/graphs/test_transcription.py`
- `src/app/api/routes.py` -> `tests/app/api/test_routes.py`
### Level 2: FastAPI branch (only for HTTP/dependency/lifespan concerns)
Escalate here when testing API routes, dependency injection boundaries, or app lifespan behavior.
Rules:
- One initial test module per core source module.
- Prefer `test_<module>.py` naming.
- Keep directory mirrors shallow first; add deeper modules only where behavior is complex.
Apply these defaults:
1. Prefer `TestClient` with sync `def` tests for route behavior.
2. Use `AsyncClient` + `@pytest.mark.anyio` only when test logic must await other async work.
3. Prefer `app.dependency_overrides` over patching internals.
4. Reset dependency overrides in teardown after every test/fixture.
5. For startup/shutdown semantics:
- use `TestClient` as context manager, or
- use `LifespanManager` with async client.
## Step 2: Classify By Dependency Cost
Assign each test module to one initial class:
- `unit`: no DB/network/filesystem side effects; instant execution.
- `integration`: touches DB, HTTP stack, workflow runtime, or external services.
- `smoke`: thin end-to-end confidence checks.
Marker intent in FastAPI-heavy suites:
- `unit`: service logic without HTTP/DB.
- `integration`: route + DI + DB contract checks.
- `smoke`: one request per critical user path.
Decision logic:
- If logic can run with fakes/stubs, make it `unit`.
- If contract with framework/DB is essential, make it `integration`.
- If validating a user-critical path across layers, make it `smoke`.
Reference: [fastapi-testing.md](./references/fastapi-testing.md)
## Step 3: Scaffold Minimal Test Modules
For each target module, scaffold:
- import section
- one happy-path test function
- one error/edge test function
- TODO comments indicating detail expansion points
### Level 3: SQLAlchemy branch (only for DB transaction/session design)
Escalate here when session lifecycle, transaction isolation, or async ORM behavior matters.
Keep assertions minimal but behavior-focused. Avoid large fixtures in module files.
Apply these defaults:
1. Create engine once per test session.
2. Open connection + outer transaction per test.
3. Bind session with `join_transaction_mode="create_savepoint"`.
4. Allow code under test to call `commit()` safely; rollback outer transaction at test end.
5. Keep unit tests DB-free; DB tests belong under `integration`.
## Step 4: Fixture Layering Strategy
Use fixture scopes based on cost:
- `function` scope by default.
- broader scopes (`module`/`session`) only for expensive setup with clear teardown.
Async additions:
- use async fixtures and `@pytest.mark.anyio`.
- set `expire_on_commit=False` for `AsyncSession`.
- avoid implicit lazy IO; use eager loading (`selectinload`) or explicit refresh.
Layer fixtures by directory:
- `tests/conftest.py`: global, lightweight fixtures only (factories, deterministic defaults).
- subtree `conftest.py`: domain-specific fixtures (API client, DB session, AI runtime stubs).
SQLite in-memory with threaded test clients:
- use `StaticPool` when required by thread/connection sharing.
Guidelines:
- Prefer yield fixtures for setup/teardown.
- Keep fixtures atomic (one state-changing responsibility per fixture).
- Avoid autouse except for truly universal behavior.
Reference: [sqlalchemy-testing.md](./references/sqlalchemy-testing.md)
## Step 5: Marker Taxonomy And Config
Ensure marker names are explicit and registered in `pyproject.toml` because strict markers are enabled.
## Branching Logic Summary
- If pure logic can be faked cleanly, keep in `unit`.
- If framework/DB contract is the behavior under test, use `integration`.
- If external service credentials/network is required, gate behind `external`.
- If suite slows down, split by marker before broadening fixture scope.
- If async relationship access raises `MissingGreenlet`, switch to eager loading strategy.
Recommended baseline markers:
- `unit`
- `integration`
- `smoke`
- `slow`
- `external` (requires network/service credentials)
Default run strategy:
- Fast local path: run only `unit` by default in day-to-day iteration.
- Full validation path: run all markers in CI or pre-release checks.
## Step 6: Execution And Verification
Run commands in this order:
1. `uv run pytest --collect-only -q`
2. `uv run pytest -m unit -q`
3. `uv run pytest -q` (if dependencies are available)
Optional targeted runs:
- by node id for one test
- by `-k` expression for focused iteration
## Step 7: Completion Checks
## Completion Checks
A scaffold pass is complete when all are true:
1. Every core source area has at least one corresponding test module.
2. Unit tests run quickly and deterministically.
3. Integration/external tests are isolated by marker and fixture boundaries.
4. No unregistered marker warnings/errors.
5. `tests/` structure is understandable without extra documentation.
6. A clear TODO path exists for deepening assertions later.
1. Core source areas map to clear test modules.
2. Fast path (`-m unit`) is deterministic and quick.
3. Integration and external paths are isolated by fixtures and markers.
4. No unregistered-marker failures occur.
5. Structure is understandable without extra oral context.
6. Clear TODO extension points exist for deeper assertions.
## Branching Scenarios
- If external APIs are required: provide stubs/mocks for unit tests; guard real calls behind `external` marker.
- If DB is required: build a dedicated integration fixture layer and keep unit tests DB-free.
- If tests become slow: split slow tests via marker and widen fixture scope only where safe.
- If naming conflicts appear: keep unique test module names or package test directories explicitly.
## Output Format
When applying this skill, provide:
## Output Contract
When this skill is applied, return:
1. Proposed test tree diff.
2. Marker and fixture plan.
3. Exact commands for fast path and full path.
4. Risks/open questions before writing detailed assertions.
3. Exact fast-path and full-path commands.
4. Which reference level was loaded and why.
5. Risks or open questions before expanding assertions.