diff --git a/docs/testing.md b/docs/testing.md index 94cfd7d..249966a 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -1 +1,177 @@ -# Testing \ No newline at end of file +# Testing + +This page defines an initial, high-level test structure for this repository that is easy to grow over time while keeping local feedback fast. + +Primary guidance sources: +- [Pytest scaffolding skill](./skills/pytest-scaffolding/SKILL.md) +- [FastAPI testing reference](./skills/pytest-scaffolding/references/fastapi-testing.md) +- [FastAPI + uv + Docker skill](./skills/fastapi-uv-docker/SKILL.md) + +## Goals + +1. Keep the default developer test loop fast and deterministic. +2. Separate cheap tests from expensive tests with clear markers. +3. Mirror production code layout so new tests are easy to place. +4. Make FastAPI dependency and lifespan testing explicit. +5. Leave clear extension points for database and external integrations. + +## Naming And Layout Conventions + +Use these naming rules consistently: +- test files: `test_.py` +- test functions: `test__()` +- test classes (optional): `Test` +- fixtures: `_` (for example `session_engine`, `api_client`) + +Target end-state test tree: + +```text +tests/ + conftest.py + unit/ + test_main.py + test_mcp.py + catalog/ + test_server.py + skills/ + test_document_loader.py + test_document_loader_references.py + web/ + test_config.py + test_docs_mount.py + integration/ + api/ + test_health_endpoint.py + test_app_routes.py + catalog/ + test_prompt_loader_and_catalog.py + test_step6_tool_fallback.py + startup/ + test_lifespan.py + smoke/ + test_service_boot_and_health.py + fixtures/ + __init__.py + factories.py + payloads.py + resources/ + prompts/ + skills/ + catalog/ + configs/ +``` + +Mapping rule: +- mirror `src/personal_mcp/` into `tests/unit/` first. +- add integration modules only where contracts exist (FastAPI routes, DB behavior, external adapters). + +Current to target mapping in this repo: +- `tests/test_document_loader_references.py` -> `tests/unit/skills/test_document_loader_references.py` +- `tests/test_prompt_loader_and_catalog.py` -> `tests/integration/catalog/test_prompt_loader_and_catalog.py` +- `tests/test_step6_tool_fallback.py` -> `tests/integration/catalog/test_step6_tool_fallback.py` + +## Marker Strategy + +Use a small marker vocabulary from day one: +- `unit`: pure logic, no DB/network/filesystem side effects. +- `integration`: framework wiring and/or DB contract checks. +- `smoke`: thin checks for critical paths. +- `slow`: expensive tests that should not run in every local loop. +- `external`: real third-party calls; typically excluded in CI by default. + +Fast-path expectation: +- local default loop should prioritize `unit` (and optionally `smoke`). + +## Fixture Layering + +1. Keep lightweight, global fixtures in `tests/conftest.py`. +2. Add subtree `conftest.py` files only when a test group needs dedicated setup. +3. Prefer fixture composition over large monolithic fixtures. +4. Always clean up FastAPI dependency overrides in fixture teardown. + +Suggested early fixtures: +- `settings_override` +- `api_client` +- `tmp_workspace` + +Optional integration fixtures (add when needed): +- `session_engine` / `db_session` +- `async_engine` / `async_session` + +## FastAPI Testing Structure + +Default route tests: +- use `fastapi.testclient.TestClient` for standard route behavior tests. +- keep tests as sync `def` unless async behavior must be asserted. + +Async route tests: +- use `httpx.AsyncClient` with `ASGITransport` and `@pytest.mark.anyio`. + +Dependency testing: +- prefer `app.dependency_overrides` for `Depends(...)` seams. +- reset `app.dependency_overrides` after each test/fixture. + +Lifespan behavior: +- use `TestClient(app)` as a context manager for startup/shutdown checks. + +## Test Granularity By Layer + +`unit/`: +- parser and loader behavior +- catalog indexing and filtering logic +- utility functions and pure transformations + +`integration/api/`: +- endpoint request/response contracts +- dependency override behavior +- error mapping and status code assertions + +`integration/db/` (future-ready): +- transaction boundaries +- commit/rollback semantics +- async session lifecycle behavior + +`smoke/`: +- one request/assertion path per critical workflow + +## Command Baseline + +Canonical invocation in this repository: + +```bash +uv run pytest +``` + +Recommended execution sequence: + +```bash +uv run pytest --collect-only -q +uv run pytest -m unit -q +uv run pytest -m "unit or smoke" -q +uv run pytest -q +``` + +## Extension Plan + +When adding a new feature: +1. Add or update corresponding `unit` tests first. +2. Add an `integration` test only for real boundary/contracts. +3. Add/adjust fixtures at the narrowest useful scope. +4. Add a `smoke` test only for user-critical paths. + +When the suite grows: +1. Split slower groups behind `slow` and/or `external` markers. +2. Keep `unit` runtime bounded for rapid local feedback. +3. Promote shared setup into fixtures only after repeated duplication. + +## Rollout Order + +Use this order to reach the end-state structure with minimal disruption: +1. Create `tests/unit/` and `tests/integration/` subtrees. +2. Move the 3 existing top-level tests into the mapped target paths. +3. Add `tests/unit` modules mirroring uncovered source modules. +4. Add API and lifespan integration tests under `tests/integration/api/` and `tests/integration/startup/`. +5. Add one smoke test for boot + health path. +6. Add marker registration in `pyproject.toml` as marker usage expands. + +This gives a concrete, stable final organization that remains easy to extend as new modules and boundaries are added. \ No newline at end of file