testing page updates
This commit is contained in:
John Lancaster
+56
-138
@@ -1,177 +1,95 @@
|
||||
---
|
||||
icon: lucide/flask-conical
|
||||
---
|
||||
|
||||
# Testing
|
||||
|
||||
This page defines an initial, high-level test structure for this repository that is easy to grow over time while keeping local feedback fast.
|
||||
This page describes the current test layout and execution model for this repository.
|
||||
|
||||
Primary guidance sources:
|
||||
- [Pytest scaffolding skill](./skills/pytest-scaffolding/SKILL.md)
|
||||
- [FastAPI testing reference](./skills/pytest-scaffolding/references/fastapi-testing.md)
|
||||
- [Pytest docs reference](./skills/pytest-scaffolding/references/pytest-docs.md)
|
||||
- [FastAPI + uv + Docker skill](./skills/fastapi-uv-docker/SKILL.md)
|
||||
|
||||
## Goals
|
||||
|
||||
1. Keep the default developer test loop fast and deterministic.
|
||||
2. Separate cheap tests from expensive tests with clear markers.
|
||||
3. Mirror production code layout so new tests are easy to place.
|
||||
4. Make FastAPI dependency and lifespan testing explicit.
|
||||
5. Leave clear extension points for database and external integrations.
|
||||
1. Keep local feedback fast with deterministic tests.
|
||||
2. Mirror source modules with focused test groups.
|
||||
3. Keep endpoint and MCP surface checks explicit.
|
||||
4. Make marker usage strict and intentional.
|
||||
|
||||
## Naming And Layout Conventions
|
||||
## Current Test Layout
|
||||
|
||||
Use these naming rules consistently:
|
||||
- test files: `test_<subject>.py`
|
||||
- test functions: `test_<behavior>_<expected_result>()`
|
||||
- test classes (optional): `Test<Subject>`
|
||||
- fixtures: `<scope>_<resource>` (for example `session_engine`, `api_client`)
|
||||
|
||||
Target end-state test tree:
|
||||
Current tree:
|
||||
|
||||
```text
|
||||
tests/
|
||||
conftest.py
|
||||
unit/
|
||||
test_main.py
|
||||
test_mcp.py
|
||||
catalog/
|
||||
test_server.py
|
||||
skills/
|
||||
test_document_loader.py
|
||||
test_document_loader_references.py
|
||||
web/
|
||||
test_config.py
|
||||
test_docs_mount.py
|
||||
integration/
|
||||
api/
|
||||
test_health_endpoint.py
|
||||
test_app_routes.py
|
||||
catalog/
|
||||
test_prompt_loader_and_catalog.py
|
||||
test_step6_tool_fallback.py
|
||||
startup/
|
||||
test_lifespan.py
|
||||
smoke/
|
||||
test_service_boot_and_health.py
|
||||
fixtures/
|
||||
__init__.py
|
||||
factories.py
|
||||
payloads.py
|
||||
resources/
|
||||
prompts/
|
||||
skills/
|
||||
catalog/
|
||||
configs/
|
||||
__init__.py
|
||||
conftest.py
|
||||
registry/
|
||||
ingest/
|
||||
conftest.py
|
||||
test_current_docs.py
|
||||
test_document.py
|
||||
test_prompt.py
|
||||
test_skill.py
|
||||
models/
|
||||
test_document_validation.py
|
||||
test_prompt_validation.py
|
||||
test_registry_payload_models.py
|
||||
test_skill_validation.py
|
||||
web/
|
||||
conftest.py
|
||||
test_endpoint_connections.py
|
||||
test_mcp_skills.py
|
||||
```
|
||||
|
||||
Mapping rule:
|
||||
- mirror `src/personal_mcp/` into `tests/unit/` first.
|
||||
- add integration modules only where contracts exist (FastAPI routes, DB behavior, external adapters).
|
||||
Source-to-test alignment today:
|
||||
- `src/personal_mcp/registry/ingest/` -> `tests/registry/ingest/`
|
||||
- `src/personal_mcp/registry/models/` -> `tests/registry/models/`
|
||||
- `src/personal_mcp/web/` and MCP HTTP surface -> `tests/web/`
|
||||
|
||||
Current to target mapping in this repo:
|
||||
- `tests/test_document_loader_references.py` -> `tests/unit/skills/test_document_loader_references.py`
|
||||
- `tests/test_prompt_loader_and_catalog.py` -> `tests/integration/catalog/test_prompt_loader_and_catalog.py`
|
||||
- `tests/test_step6_tool_fallback.py` -> `tests/integration/catalog/test_step6_tool_fallback.py`
|
||||
## Markers And Strictness
|
||||
|
||||
## Marker Strategy
|
||||
Configured markers in `pyproject.toml`:
|
||||
- `unit`: fast deterministic tests with no external dependencies
|
||||
- `integration`: framework or component integration tests
|
||||
- `smoke`: thin critical-path checks
|
||||
|
||||
Use a small marker vocabulary from day one:
|
||||
- `unit`: pure logic, no DB/network/filesystem side effects.
|
||||
- `integration`: framework wiring and/or DB contract checks.
|
||||
- `smoke`: thin checks for critical paths.
|
||||
- `slow`: expensive tests that should not run in every local loop.
|
||||
- `external`: real third-party calls; typically excluded in CI by default.
|
||||
|
||||
Fast-path expectation:
|
||||
- local default loop should prioritize `unit` (and optionally `smoke`).
|
||||
Pytest runs with `--strict-markers`, so any unregistered marker fails the test run.
|
||||
|
||||
## Fixture Layering
|
||||
|
||||
1. Keep lightweight, global fixtures in `tests/conftest.py`.
|
||||
2. Add subtree `conftest.py` files only when a test group needs dedicated setup.
|
||||
3. Prefer fixture composition over large monolithic fixtures.
|
||||
4. Always clean up FastAPI dependency overrides in fixture teardown.
|
||||
Fixture placement follows test scope:
|
||||
1. `tests/conftest.py` for cross-suite defaults.
|
||||
2. `tests/registry/ingest/conftest.py` for ingest-specific setup.
|
||||
3. `tests/web/conftest.py` for web and endpoint client setup.
|
||||
|
||||
Suggested early fixtures:
|
||||
- `settings_override`
|
||||
- `api_client`
|
||||
- `tmp_workspace`
|
||||
|
||||
Optional integration fixtures (add when needed):
|
||||
- `session_engine` / `db_session`
|
||||
- `async_engine` / `async_session`
|
||||
|
||||
## FastAPI Testing Structure
|
||||
|
||||
Default route tests:
|
||||
- use `fastapi.testclient.TestClient` for standard route behavior tests.
|
||||
- keep tests as sync `def` unless async behavior must be asserted.
|
||||
|
||||
Async route tests:
|
||||
- use `httpx.AsyncClient` with `ASGITransport` and `@pytest.mark.anyio`.
|
||||
|
||||
Dependency testing:
|
||||
- prefer `app.dependency_overrides` for `Depends(...)` seams.
|
||||
- reset `app.dependency_overrides` after each test/fixture.
|
||||
|
||||
Lifespan behavior:
|
||||
- use `TestClient(app)` as a context manager for startup/shutdown checks.
|
||||
|
||||
## Test Granularity By Layer
|
||||
|
||||
`unit/`:
|
||||
- parser and loader behavior
|
||||
- catalog indexing and filtering logic
|
||||
- utility functions and pure transformations
|
||||
|
||||
`integration/api/`:
|
||||
- endpoint request/response contracts
|
||||
- dependency override behavior
|
||||
- error mapping and status code assertions
|
||||
|
||||
`integration/db/` (future-ready):
|
||||
- transaction boundaries
|
||||
- commit/rollback semantics
|
||||
- async session lifecycle behavior
|
||||
|
||||
`smoke/`:
|
||||
- one request/assertion path per critical workflow
|
||||
Prefer adding fixtures at the narrowest scope that serves more than one test.
|
||||
|
||||
## Command Baseline
|
||||
|
||||
Canonical invocation in this repository:
|
||||
Canonical invocation:
|
||||
|
||||
```bash
|
||||
uv run pytest
|
||||
```
|
||||
|
||||
Recommended execution sequence:
|
||||
Useful filtered runs:
|
||||
|
||||
```bash
|
||||
uv run pytest --collect-only -q
|
||||
uv run pytest -m unit -q
|
||||
uv run pytest -m "unit or smoke" -q
|
||||
uv run pytest -q
|
||||
uv run pytest -m integration -q
|
||||
uv run pytest -m smoke -q
|
||||
```
|
||||
|
||||
## Extension Plan
|
||||
## Adding New Tests
|
||||
|
||||
When adding a new feature:
|
||||
1. Add or update corresponding `unit` tests first.
|
||||
2. Add an `integration` test only for real boundary/contracts.
|
||||
3. Add/adjust fixtures at the narrowest useful scope.
|
||||
4. Add a `smoke` test only for user-critical paths.
|
||||
When adding coverage:
|
||||
1. Place tests under the nearest existing module subtree (`registry/` or `web/`).
|
||||
2. Mirror the source path where practical.
|
||||
3. Reuse existing `conftest.py` files before adding new fixture layers.
|
||||
4. Add markers only when they convey execution intent, and register new markers in `pyproject.toml` first.
|
||||
|
||||
When the suite grows:
|
||||
1. Split slower groups behind `slow` and/or `external` markers.
|
||||
2. Keep `unit` runtime bounded for rapid local feedback.
|
||||
3. Promote shared setup into fixtures only after repeated duplication.
|
||||
|
||||
## Rollout Order
|
||||
|
||||
Use this order to reach the end-state structure with minimal disruption:
|
||||
1. Create `tests/unit/` and `tests/integration/` subtrees.
|
||||
2. Move the 3 existing top-level tests into the mapped target paths.
|
||||
3. Add `tests/unit` modules mirroring uncovered source modules.
|
||||
4. Add API and lifespan integration tests under `tests/integration/api/` and `tests/integration/startup/`.
|
||||
5. Add one smoke test for boot + health path.
|
||||
6. Add marker registration in `pyproject.toml` as marker usage expands.
|
||||
|
||||
This gives a concrete, stable final organization that remains easy to extend as new modules and boundaries are added.
|
||||
This keeps the suite aligned with the current architecture while preserving a fast local test loop.
|
||||
Reference in New Issue
Block a user