From ff7cd4a07f50cf6a8ca3c0e29fbbabb7923a873e Mon Sep 17 00:00:00 2001 From: John Lancaster <32917998+jsl12@users.noreply.github.com> Date: Sun, 21 Jun 2026 20:54:33 -0500 Subject: [PATCH] testing page updates --- docs/testing.md | 194 ++++++++++++++---------------------------------- 1 file changed, 56 insertions(+), 138 deletions(-) diff --git a/docs/testing.md b/docs/testing.md index 249966a..1177b78 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -1,177 +1,95 @@ +--- +icon: lucide/flask-conical +--- + # Testing -This page defines an initial, high-level test structure for this repository that is easy to grow over time while keeping local feedback fast. +This page describes the current test layout and execution model for this repository. Primary guidance sources: - [Pytest scaffolding skill](./skills/pytest-scaffolding/SKILL.md) -- [FastAPI testing reference](./skills/pytest-scaffolding/references/fastapi-testing.md) +- [Pytest docs reference](./skills/pytest-scaffolding/references/pytest-docs.md) - [FastAPI + uv + Docker skill](./skills/fastapi-uv-docker/SKILL.md) ## Goals -1. Keep the default developer test loop fast and deterministic. -2. Separate cheap tests from expensive tests with clear markers. -3. Mirror production code layout so new tests are easy to place. -4. Make FastAPI dependency and lifespan testing explicit. -5. Leave clear extension points for database and external integrations. +1. Keep local feedback fast with deterministic tests. +2. Mirror source modules with focused test groups. +3. Keep endpoint and MCP surface checks explicit. +4. Make marker usage strict and intentional. -## Naming And Layout Conventions +## Current Test Layout -Use these naming rules consistently: -- test files: `test_.py` -- test functions: `test__()` -- test classes (optional): `Test` -- fixtures: `_` (for example `session_engine`, `api_client`) - -Target end-state test tree: +Current tree: ```text tests/ - conftest.py - unit/ - test_main.py - test_mcp.py - catalog/ - test_server.py - skills/ - test_document_loader.py - test_document_loader_references.py - web/ - test_config.py - test_docs_mount.py - integration/ - api/ - test_health_endpoint.py - test_app_routes.py - catalog/ - test_prompt_loader_and_catalog.py - test_step6_tool_fallback.py - startup/ - test_lifespan.py - smoke/ - test_service_boot_and_health.py - fixtures/ - __init__.py - factories.py - payloads.py - resources/ - prompts/ - skills/ - catalog/ - configs/ + __init__.py + conftest.py + registry/ + ingest/ + conftest.py + test_current_docs.py + test_document.py + test_prompt.py + test_skill.py + models/ + test_document_validation.py + test_prompt_validation.py + test_registry_payload_models.py + test_skill_validation.py + web/ + conftest.py + test_endpoint_connections.py + test_mcp_skills.py ``` -Mapping rule: -- mirror `src/personal_mcp/` into `tests/unit/` first. -- add integration modules only where contracts exist (FastAPI routes, DB behavior, external adapters). +Source-to-test alignment today: +- `src/personal_mcp/registry/ingest/` -> `tests/registry/ingest/` +- `src/personal_mcp/registry/models/` -> `tests/registry/models/` +- `src/personal_mcp/web/` and MCP HTTP surface -> `tests/web/` -Current to target mapping in this repo: -- `tests/test_document_loader_references.py` -> `tests/unit/skills/test_document_loader_references.py` -- `tests/test_prompt_loader_and_catalog.py` -> `tests/integration/catalog/test_prompt_loader_and_catalog.py` -- `tests/test_step6_tool_fallback.py` -> `tests/integration/catalog/test_step6_tool_fallback.py` +## Markers And Strictness -## Marker Strategy +Configured markers in `pyproject.toml`: +- `unit`: fast deterministic tests with no external dependencies +- `integration`: framework or component integration tests +- `smoke`: thin critical-path checks -Use a small marker vocabulary from day one: -- `unit`: pure logic, no DB/network/filesystem side effects. -- `integration`: framework wiring and/or DB contract checks. -- `smoke`: thin checks for critical paths. -- `slow`: expensive tests that should not run in every local loop. -- `external`: real third-party calls; typically excluded in CI by default. - -Fast-path expectation: -- local default loop should prioritize `unit` (and optionally `smoke`). +Pytest runs with `--strict-markers`, so any unregistered marker fails the test run. ## Fixture Layering -1. Keep lightweight, global fixtures in `tests/conftest.py`. -2. Add subtree `conftest.py` files only when a test group needs dedicated setup. -3. Prefer fixture composition over large monolithic fixtures. -4. Always clean up FastAPI dependency overrides in fixture teardown. +Fixture placement follows test scope: +1. `tests/conftest.py` for cross-suite defaults. +2. `tests/registry/ingest/conftest.py` for ingest-specific setup. +3. `tests/web/conftest.py` for web and endpoint client setup. -Suggested early fixtures: -- `settings_override` -- `api_client` -- `tmp_workspace` - -Optional integration fixtures (add when needed): -- `session_engine` / `db_session` -- `async_engine` / `async_session` - -## FastAPI Testing Structure - -Default route tests: -- use `fastapi.testclient.TestClient` for standard route behavior tests. -- keep tests as sync `def` unless async behavior must be asserted. - -Async route tests: -- use `httpx.AsyncClient` with `ASGITransport` and `@pytest.mark.anyio`. - -Dependency testing: -- prefer `app.dependency_overrides` for `Depends(...)` seams. -- reset `app.dependency_overrides` after each test/fixture. - -Lifespan behavior: -- use `TestClient(app)` as a context manager for startup/shutdown checks. - -## Test Granularity By Layer - -`unit/`: -- parser and loader behavior -- catalog indexing and filtering logic -- utility functions and pure transformations - -`integration/api/`: -- endpoint request/response contracts -- dependency override behavior -- error mapping and status code assertions - -`integration/db/` (future-ready): -- transaction boundaries -- commit/rollback semantics -- async session lifecycle behavior - -`smoke/`: -- one request/assertion path per critical workflow +Prefer adding fixtures at the narrowest scope that serves more than one test. ## Command Baseline -Canonical invocation in this repository: +Canonical invocation: ```bash uv run pytest ``` -Recommended execution sequence: +Useful filtered runs: ```bash uv run pytest --collect-only -q uv run pytest -m unit -q -uv run pytest -m "unit or smoke" -q -uv run pytest -q +uv run pytest -m integration -q +uv run pytest -m smoke -q ``` -## Extension Plan +## Adding New Tests -When adding a new feature: -1. Add or update corresponding `unit` tests first. -2. Add an `integration` test only for real boundary/contracts. -3. Add/adjust fixtures at the narrowest useful scope. -4. Add a `smoke` test only for user-critical paths. +When adding coverage: +1. Place tests under the nearest existing module subtree (`registry/` or `web/`). +2. Mirror the source path where practical. +3. Reuse existing `conftest.py` files before adding new fixture layers. +4. Add markers only when they convey execution intent, and register new markers in `pyproject.toml` first. -When the suite grows: -1. Split slower groups behind `slow` and/or `external` markers. -2. Keep `unit` runtime bounded for rapid local feedback. -3. Promote shared setup into fixtures only after repeated duplication. - -## Rollout Order - -Use this order to reach the end-state structure with minimal disruption: -1. Create `tests/unit/` and `tests/integration/` subtrees. -2. Move the 3 existing top-level tests into the mapped target paths. -3. Add `tests/unit` modules mirroring uncovered source modules. -4. Add API and lifespan integration tests under `tests/integration/api/` and `tests/integration/startup/`. -5. Add one smoke test for boot + health path. -6. Add marker registration in `pyproject.toml` as marker usage expands. - -This gives a concrete, stable final organization that remains easy to extend as new modules and boundaries are added. \ No newline at end of file +This keeps the suite aligned with the current architecture while preserving a fast local test loop. \ No newline at end of file