154 Commits
Author SHA1 Message Date
John Lancaster 3abafc4850 prune 2026-07-30 01:04:11 -05:00
John Lancaster d4c7952175 task tweak 2026-07-30 01:03:51 -05:00
John Lancaster da58e20b69 mounting docs 2026-07-30 01:03:34 -05:00
John Lancaster d79025538b extra javascript 2026-07-30 01:03:24 -05:00
John Lancaster 7b1e5fcacb focused styling 2026-07-30 01:02:26 -05:00
John Lancaster 37461fd880 mathjax 2026-07-30 00:41:54 -05:00
John Lancaster 226f19b2c6 more styling 2026-07-30 00:41:26 -05:00
John Lancaster a18c8456d3 pydantic-settings update 2026-07-30 00:08:48 -05:00
John Lancaster 34e6d693ab uvicorn startup 2026-07-30 00:01:18 -05:00
John Lancaster efba051cb5 css reference page 2026-07-29 23:47:24 -05:00
John Lancaster c9b6e137f2 mount change 2026-07-29 23:38:04 -05:00
John Lancaster 8f26051a52 nicegui consolidation 2026-07-29 23:37:48 -05:00
John Lancaster bc0d6ede49 simplified a bit 2026-07-26 20:11:25 -05:00
John Lancaster aed2e41ef0 started crud reference 2026-07-26 19:35:01 -05:00
John Lancaster d999a04144 improvements 2026-07-26 19:10:40 -05:00
John Lancaster 4818e86a1e improving async fastapi sqlmodel skill 2026-07-26 17:57:55 -05:00
John Lancaster b6393f1222 renamed async fastapi skill 2026-07-26 17:40:33 -05:00
John Lancaster 3897eabfbc authoring reference 2026-07-26 17:39:42 -05:00
John Lancaster 9e0097708c docstrings 2026-07-26 17:25:55 -05:00
John Lancaster 42ea105bee WIP simplifying load/startup 2026-07-26 17:23:39 -05:00
John Lancaster 5e20f69cfe settings 2026-07-26 14:09:08 -05:00
John Lancaster 007d823c0a ruff rules 2026-07-21 08:52:32 -05:00
John Lancaster 27f783fc90 app factory 2026-07-21 08:49:36 -05:00
John Lancaster 7970e76d4f config updates 2026-07-21 08:43:31 -05:00
John Lancaster 70dd0f45d9 declarative logging 2026-07-08 22:39:46 -05:00
John Lancaster 963805c551 logging skill updates 2026-07-08 21:03:38 -05:00
John Lancaster a3ca1a65c2 task updates 2026-07-02 23:23:11 -05:00
John Lancaster 94dd47cc19 logging references 2026-07-02 23:22:55 -05:00
John Lancaster b3d4e55a15 uv.lock update 2026-07-02 23:05:12 -05:00
John Lancaster 7b2b80ecf2 config updates 2026-07-02 23:05:05 -05:00
John Lancaster d4ca78dbfb renamed python-logging 2026-07-02 23:04:54 -05:00
John Lancaster eeeb6ecdbe async sqlmodel 2026-06-26 00:53:18 -05:00
John Lancaster 0177496fab pydantic-settings skill 2026-06-25 21:51:49 -05:00
John Lancaster 00498a2fed rename 2026-06-24 08:55:04 -05:00
John Lancaster 913ba66d8b pytest scaffold 2026-06-24 08:41:11 -05:00
John Lancaster e2c199c1b7 usage notes 2026-06-24 08:38:43 -05:00
John Lancaster 45a1e56d1c binding dataclass page 2026-06-23 19:46:35 -05:00
John Lancaster a6ccc14917 greenfield architecture 2026-06-22 10:00:21 -05:00
John Lancaster a0ae38d0cc authoring prompt 2026-06-22 09:39:57 -05:00
John Lancaster 34d3808bbb shim creation prompt 2026-06-22 08:50:26 -05:00
John Lancaster 1cfe9c8e40 docs 2026-06-22 08:39:18 -05:00
John Lancaster 313c4ecb1e authoring page 2026-06-22 08:15:35 -05:00
John Lancaster ea5450f6cb pytest principles 2026-06-22 07:59:18 -05:00
John Lancaster ab53c239bf rename 2026-06-21 22:55:58 -05:00
John Lancaster 35d7fa1718 tag conventions 2026-06-21 22:52:26 -05:00
John Lancaster e93462ec3b better tagging 2026-06-21 22:46:53 -05:00
John Lancaster 58a94ad9b6 doc updates 2026-06-21 22:35:05 -05:00
John Lancaster c893173fcc contract updates 2026-06-21 22:29:46 -05:00
John Lancaster 123c491413 tightening 2026-06-21 22:12:02 -05:00
John Lancaster 3c7f7e61b7 mcp details skills 2026-06-21 21:56:27 -05:00
John Lancaster 76ea9ebbda in process fixes 2026-06-21 21:24:37 -05:00
John Lancaster 0aa7ace272 asyncio testing 2026-06-21 21:05:10 -05:00
John Lancaster 4da2b0ac83 better web tests 2026-06-21 21:01:11 -05:00
John Lancaster 806bb15bcc link skill script 2026-06-21 21:00:55 -05:00
John Lancaster ff7cd4a07f testing page updates 2026-06-21 20:54:33 -05:00
John Lancaster 3603471699 fixes 2026-06-21 20:24:05 -05:00
John Lancaster dcbf570a13 ipywidgets 2026-06-21 20:22:50 -05:00
John Lancaster d3b336f1e3 started tasks 2026-06-21 18:15:56 -05:00
John Lancaster 1f7e63267a doc updates 2026-06-21 18:12:58 -05:00
John Lancaster 69cd9037a3 mcp tests 2026-06-21 18:04:46 -05:00
John Lancaster b9bb11ac02 web tests 2026-06-21 17:57:01 -05:00
John Lancaster 4958eeb3ef connection tests 2026-06-21 17:51:16 -05:00
John Lancaster 5a31ba6390 mcp endpoint test scaffold 2026-06-21 17:48:33 -05:00
John Lancaster 36347ff4a5 models 2026-06-21 17:35:01 -05:00
John Lancaster c189677717 started model tests 2026-06-21 17:09:03 -05:00
John Lancaster 37fa9b6c6f better data models 2026-06-21 16:53:43 -05:00
John Lancaster 34923b51d7 prune 2026-06-21 16:15:47 -05:00
John Lancaster 2d65d83162 docstrings 2026-06-21 16:15:37 -05:00
John Lancaster 3a6e2665dd pruning 2026-06-21 16:01:08 -05:00
John Lancaster b98d8b782a added current doc collection tests 2026-06-21 15:58:00 -05:00
John Lancaster 36032040ae added prompt ingestion 2026-06-21 15:51:51 -05:00
John Lancaster 4f05f13e45 better typing 2026-06-21 15:41:22 -05:00
John Lancaster 57347077a9 better pytest 2026-06-21 15:36:31 -05:00
John Lancaster 7fec3a4337 ty checking 2026-06-21 15:29:57 -05:00
John Lancaster c5b7733528 typing skill improvements 2026-06-21 15:20:00 -05:00
John Lancaster 3c5db37223 test content 2026-06-21 15:13:44 -05:00
John Lancaster 29130c3a0c registry ingest test scaffolding 2026-06-21 12:58:42 -05:00
John Lancaster aec3500370 typing 2026-06-21 12:41:14 -05:00
John Lancaster 9c8ab70c06 python typing skill 2026-06-21 12:37:19 -05:00
John Lancaster 9a9432cc55 frozen pydantic models 2026-06-21 12:26:12 -05:00
John Lancaster 4320a251f5 ruff workflow 2026-06-21 12:14:46 -05:00
John Lancaster caa4a5079a WIP loading 2026-06-21 11:34:07 -05:00
John Lancaster 993dc6a879 file loading 2026-06-21 10:06:25 -05:00
John Lancaster dab539489a started manual refactor 2026-06-21 09:13:30 -05:00
John Lancaster 197fa32f2c rename 2026-06-20 20:37:09 -05:00
John Lancaster c653c7024b structured tests 2026-06-20 20:36:44 -05:00
John Lancaster 82b50fb63b testing page 2026-06-20 20:34:10 -05:00
John Lancaster f8e0c14d46 started prompt mechanics 2026-06-20 20:27:32 -05:00
John Lancaster 098a2418ee shims 2026-06-20 20:03:54 -05:00
John Lancaster 7f672b9c8f pytest naming convention 2026-06-20 20:02:03 -05:00
John Lancaster 3c5efc6018 prune link 2026-06-20 19:44:11 -05:00
John Lancaster 0b2d45d419 pytest add 2026-06-20 19:44:02 -05:00
John Lancaster 406fd63a07 better copilot integration 2026-06-20 19:40:43 -05:00
John Lancaster 323f02102d step6 2026-06-20 19:30:37 -05:00
John Lancaster 906bba427b step 6 update 2026-06-20 18:20:43 -05:00
John Lancaster 06d5fc18f2 consolidated new-skill resource 2026-06-20 18:18:44 -05:00
John Lancaster 38edc4ac36 vscode config improvements 2026-06-20 18:05:05 -05:00
John Lancaster c73771c2f4 shim instructions 2026-06-20 17:52:03 -05:00
John Lancaster 33144da02f bootstrap prompt 2026-06-20 17:36:17 -05:00
John Lancaster 0a9dadd5a8 ruff skill 2026-06-20 17:25:47 -05:00
John Lancaster 660ca88e47 auto generating reference front-matter 2026-06-20 16:43:29 -05:00
John Lancaster e60fc4b27b update instructions to add links 2026-06-20 16:25:47 -05:00
John Lancaster 8817d2586f icons 2026-06-20 15:01:31 -05:00
John Lancaster bb7508cf65 doc updates 2026-06-20 14:56:25 -05:00
John Lancaster 467e1d3c35 sten 6 implementation 2026-06-20 14:31:24 -05:00
John Lancaster 3885774e5b step 6 2026-06-20 14:23:29 -05:00
John Lancaster f54cacd6cb ruffage 2026-06-20 14:13:22 -05:00
John Lancaster 19f3c1740a implemented steps 1-5 2026-06-20 14:08:59 -05:00
John Lancaster c273ecfc54 added doc updates to plan 2026-06-20 13:45:10 -05:00
John Lancaster fa4498cb78 phase 3 2026-06-20 13:39:25 -05:00
John Lancaster adaa4177fe adjustments 2026-06-20 13:33:18 -05:00
John Lancaster 85355a8509 added step 4 and 5 2026-06-20 13:22:03 -05:00
John Lancaster 127e56692e the plan 2026-06-20 12:50:38 -05:00
John Lancaster 85eb75d188 logging 2026-06-19 17:40:04 -05:00
John Lancaster 5c4de7b721 pytest skill 2026-06-19 17:39:57 -05:00
John Lancaster ed6068f398 pytest improvements 2026-06-19 17:22:34 -05:00
John Lancaster 75b0c8d192 vscode skill 2026-06-19 16:56:45 -05:00
John Lancaster 45d8beda8a index update 2026-06-19 16:40:28 -05:00
John Lancaster 7a9e4044f0 explanations 2026-06-19 08:38:10 -05:00
John Lancaster 3347443ca9 formatting 2026-06-19 01:29:05 -05:00
John Lancaster 964cd6f76d page organization 2026-06-19 01:22:08 -05:00
John Lancaster ef3255544f copilot instructions 2026-06-19 01:15:27 -05:00
John Lancaster be9551c76e docker compose 2026-06-19 00:47:25 -05:00
John Lancaster 9c3fafd2fe edits 2026-06-19 00:47:01 -05:00
John Lancaster 36abea5940 zensical docs skill 2026-06-19 00:24:08 -05:00
John Lancaster 07475f972f intent adjustments 2026-06-19 00:01:52 -05:00
John Lancaster 59c638c634 usage docs 2026-06-18 23:50:37 -05:00
John Lancaster 1254cc5432 better 2026-06-18 22:34:31 -05:00
John Lancaster a4db33531e zensical-docs skills started 2026-06-18 22:33:19 -05:00
John Lancaster 818de1b3f9 new skill meta 2026-06-18 22:22:30 -05:00
John Lancaster 9f34e12e08 completing move 2026-06-18 22:14:02 -05:00
John Lancaster e78383be1f move 2026-06-18 22:06:40 -05:00
John Lancaster 6c5fda9c3a styling 2026-06-18 22:04:23 -05:00
John Lancaster 99e741f2de docker implementation 2026-06-18 21:57:20 -05:00
John Lancaster c3dc66b20f mermaid diagram 2026-06-18 21:49:08 -05:00
John Lancaster 85ae1fe8eb redirects 2026-06-18 21:24:01 -05:00
John Lancaster b6010775ad web package 2026-06-18 21:15:54 -05:00
John Lancaster 4b1a261b2d doc first implementation 2026-06-18 20:41:06 -05:00
John Lancaster d1c80d4737 docs 2026-06-18 20:29:52 -05:00
John Lancaster c915c1846d docs 2026-06-18 20:01:47 -05:00
John Lancaster 485b93b3c9 catalog 2026-06-18 19:40:14 -05:00
John Lancaster d54f427112 resource pattern 2026-06-18 19:38:57 -05:00
John Lancaster dbaaad8df8 initial server 2026-06-18 19:16:06 -05:00
John Lancaster 9b02007216 more 2026-06-17 21:49:57 -05:00
John Lancaster 22e5357ffb initial fastapi-sqlalchemy 2026-06-17 21:45:54 -05:00
John Lancaster 6e32955533 uvicorn settings 2026-06-17 00:28:59 -05:00
John Lancaster 78fa9235d3 fastapi-uv-docker 2026-06-17 00:24:37 -05:00
John Lancaster 57da7e001e reformatted skill 2026-06-16 23:51:59 -05:00
John Lancaster 35a3a8dbed reformatted to be more skill-like 2026-06-16 23:46:05 -05:00
John Lancaster c8906cef7b reorg 2026-06-16 23:39:45 -05:00
John Lancaster 8ada18b7fd pruning 2026-06-16 23:11:11 -05:00
John Lancaster bbbf7e6c7d customization skill 2026-06-16 22:50:22 -05:00
John Lancaster 5649b9ace6 front matter 2026-06-16 22:33:21 -05:00
John Lancaster 0a4af6d6ee docs extension 2026-06-16 22:12:01 -05:00
148 changed files with 18765 additions and 504 deletions
+10
View File
@@ -0,0 +1,10 @@
.git
.github
.venv
__pycache__
.cache*
.mypy_cache
.pytest_cache
.ruff_cache
site/
.env
+18
View File
@@ -0,0 +1,18 @@
# Personal MCP HTTP runtime
PERSONAL_MCP_HOST=127.0.0.1
PERSONAL_MCP_PORT=8000
PERSONAL_MCP_LOG_LEVEL=info
PERSONAL_MCP_DEBUG=false
# Mounted routes
PERSONAL_MCP_MCP_ROUTE=/mcp
PERSONAL_MCP_DOCS_ROUTE=/docs
# Static docs output built by Zensical
PERSONAL_MCP_SITE_DIR=site
# Example local command:
# uv run uvicorn personal_mcp.main:app --host 127.0.0.1 --port 8000 --log-level info
# Example container-oriented command:
# uv run uvicorn personal_mcp.main:app --host 0.0.0.0 --port 8000 --proxy-headers --forwarded-allow-ips '*'
@@ -0,0 +1,17 @@
---
name: Authoring Content
description: "Use when editing Markdown under docs/. Routes authors to the canonical docs ownership, layout, and symlink guidance."
applyTo: 'docs/**/*.md'
---
For edits under `docs/`, use the [Authoring Guide](../../docs/authoring.md) as the entry point for content placement and contracts.
For source-tree ownership, symlink, packaging, or runtime questions, follow [Source Tree Ownership](../../docs/authoring.md). Treat that section as authoritative instead of restating its guidance here.
Primary references:
- [Skill contract](../../docs/contracts/skill_contract.md)
- [Prompt contract](../../docs/contracts/prompt.md)
- [Frontmatter contract](../../docs/contracts/frontmatter.md)
- [URI contract](../../docs/contracts/uris.md)
- [Zensical documentation authoring skill](../../docs/skills/zensical-docs/SKILL.md)
@@ -0,0 +1,18 @@
---
name: Pytest Scaffolding Guidance
description: Route tests edits to the Personal MCP pytesting resource.
applyTo: 'tests/**'
---
When editing files under `tests/`, use `resource://skills/pytesting/document` as the primary guidance source for test scaffolding and pytest authoring decisions.
Execution pattern:
1. Load `resource://skills/pytesting/document` first.
2. Apply only the portions relevant to the file being edited.
3. Keep tests focused, deterministic, and aligned with repository conventions.
4. Include source-document links for any feature-level recommendation.
If task intent is ambiguous, ask one clarifying question before editing.
Be sure to also refer to the [testing page](../../docs/testing.md) page for design detail
@@ -0,0 +1,19 @@
---
name: VS Code Configuration
description: Route .vscode edits to the Personal MCP VS Code configuration skill resource.
applyTo: '.vscode/**'
---
When editing files under `.vscode/`, use `resource://skills/vscode-configuration/document` as the primary guidance source.
Execution pattern:
1. Load `resource://skills/vscode-configuration/document` first.
2. Select only the matching reference page for the current file type:
- `launch.json` -> debug launch configurations.
- `tasks.json` -> tasks.json project tasks.
- `mcp.json` -> mcp.json MCP server configuration.
3. Prefer the smallest safe config change and keep settings explicit.
4. Include source-document links for any feature-level recommendation.
If task intent is ambiguous, ask one clarifying question before editing.
@@ -0,0 +1,13 @@
---
name: Zensical Docs Markdown Guidance
description: Use the Zensical docs MCP resource when editing Markdown documentation in this repository.
applyTo: '**/*.md'
---
When editing Markdown files in this repository, use the Zensical docs resource `resource://skills/zensical-docs/document` for relevant documentation authoring guidance.
Prefer Zensical-native documentation conventions when they cover the need cleanly, while preserving expected MkDocs compatibility unless the Zensical guidance intentionally diverges.
Always check to make sure the entries in `/home/john/Documents/prompts/zensical.toml` are up to date with any changes.
Also ensure that top-level pages specify icons in their front matter
@@ -0,0 +1,46 @@
## Plan: Docs-First FastMCP End State
Create a docs-first FastMCP architecture where all Markdown remains in docs/ as the only source of truth, each skill is Anthropic-compatible in its own directory, skill metadata lives in SKILL.md frontmatter, and packaged docs are served through importlib.resources so stdio deployments work from installed wheels.
**Steps**
1. Phase 1: Define the end-state content contract. Confirm canonical structure as docs/skills/<skill-id>/SKILL.md plus docs/skills/<skill-id>/references/..., with strict per-skill ownership and no metadata.yaml sidecar. Also define stable skill-id rules (kebab-case, immutable after release). Deliverable: update the current docs/ directory with the finalized end-state content contract from this step.
2. Phase 1: Define SKILL.md frontmatter schema with Pydantic-compatible fields: id, version, name, description, tags, capabilities, depends_on, and references manifest entries. The references manifest must map logical reference ids to relative paths so each skill can reorganize references internally without changing global server code. Depends on step 1. Deliverable: update the current docs/ directory with the finalized SKILL.md frontmatter schema from this step.
3. Phase 1: Define URI contract with explicit break-and-replace policy. Recommend resource://catalog/skills_index, resource://catalog/skills/{skill_id}, resource://skills/{skill_id}/document, resource://skills/{skill_id}/references/{ref_id}, and resource://docs/{path*}. Evolving URIs and reference ids requires direct replacement, with no aliases or compatibility shims. Depends on steps 1-2. Deliverable: update the current docs/ directory with the finalized URI contract and break-and-replace policy from this step.
4. Phase 2: Build a docs registry loader that reads packaged docs via importlib.resources.files(...) Traversable APIs, parses SKILL.md frontmatter, validates schema, and creates an in-memory registry keyed by skill_id. Fail fast for duplicate ids, missing files, broken reference mappings, or invalid depends_on. Depends on steps 2-3.
5. Phase 2: Register FastMCP resources from the registry using RFC6570 templates (including wildcard paths where appropriate), read-only/idempotent annotations, explicit mime types, and on_duplicate_resources="error" for startup safety. Depends on step 4.
6. Phase 2: Add discovery surfaces as resources first, then tool fallback. Keep catalog discovery in resources, then add ResourcesAsTools for tool-only clients. Add thin discovery tools only for parity and optional BM25/regex tool search when catalog/tool volume grows enough to affect token efficiency. Define canonical fallback tool names (`list_resources`, `read_resource`, `search_patterns`, `get_pattern_by_id`, `get_skill_document_by_id`), research host-specific naming behavior for GitHub Copilot, Cursor, Claude Desktop, and generic MCP clients, and require client-side name mapping or intentionally documented aliases when providers expose namespaced wrappers. Depends on step 5.
7. Phase 3: Implement packaging so docs/ is copied into package resource space at build time (wheel + sdist) while docs/ remains canonical in source control. Use importlib.resources at runtime only; avoid direct filesystem assumptions. Depends on steps 4-6.
8. Phase 3: Remove materialization coupling between skill source modules and docs. The website build reads docs/ directly, while MCP reads packaged docs resources from the installed package. This preserves one authored source with two distribution surfaces. Depends on step 7.
9. Phase 4: Add validation and CI gates: frontmatter schema checks, URI uniqueness checks, reference integrity checks, docs build check, package content check, and stdio smoke checks that read representative skill/document resources from an installed wheel. Depends on steps 5-8.
10. Phase 4: Add long-term maintainability guardrails: architecture decision record for URI and schema contracts, skill authoring checklist, and release checklist for evolving references safely within one skill. Parallel with step 9 after core architecture is stable.
**Relevant files**
- /home/john/Documents/prompts/docs/index.md — Keep top-level docs entry and explain docs-first architecture contract.
- /home/john/Documents/prompts/docs/skills — Canonical location for all skill content, including SKILL.md and references.
- /home/john/Documents/prompts/pyproject.toml — Build inclusion rules for packaged markdown resources in wheel/sdist.
- /home/john/Documents/prompts/src/personal_mcp/main.py — App/server startup wiring for resource registry initialization.
- /home/john/Documents/prompts/src/personal_mcp/mcp.py — FastMCP instance composition and transform registration.
- /home/john/Documents/prompts/src/personal_mcp/catalog/server.py — Catalog resource and fallback discovery behavior.
- /home/john/Documents/prompts/src/personal_mcp/skills/document_loader.py — Replace file-path assumptions with importlib.resources docs registry loading.
- /home/john/Documents/prompts/src/personal_mcp/web/materialize_skill_docs.py — De-scope or retire materialization once docs-first runtime is authoritative.
**Verification**
1. Run uv run zensical build to verify docs/ remains valid and site output is stable.
2. Run uv run pytest -q with tests that validate frontmatter parsing, URI generation, reference mapping, and catalog responses.
3. Run a packaging integrity check using importlib.resources.files(...) to confirm packaged docs resources exist and are readable from an installed wheel.
4. Run a stdio MCP smoke test that lists resources and reads at least one skill document and one reference document.
5. Run fallback-client smoke tests verifying list_resources/read_resource tools work and return expected metadata for both static and templated resources, and that GitHub Copilot, Cursor, Claude Desktop, and protocol-level SDK tests use canonical tool names or documented mapped aliases.
**Decisions**
- Anthropic compatibility: strict skill directory pattern with SKILL.md and references subtree.
- Metadata strategy: YAML frontmatter in SKILL.md (no separate metadata file).
- Discovery strategy: resource-first catalog with tool fallback for tool-only MCP clients.
- Included scope: ideal end-state architecture, contracts, validation, and packaging for stdio operation.
- Excluded scope: migration mechanics from current implementation, backward-compat shim details, and docs visual redesign.
**Further Considerations**
1. Prefer recursive references support under each skill plus frontmatter manifest ids, so skill teams can reorganize internal reference folders without URI churn.
2. Define a hard rule that skill_id and directory name must match exactly to eliminate namespace/slug drift classes of bugs.
3. Do not provide URI aliases; client updates must track canonical URI contract changes directly.
+169
View File
@@ -0,0 +1,169 @@
**Phase 3 Results: Packaging Contract and Surface Decoupling (Wheel/sdist Resources + Docs-Only Authoring)**
This section finalizes Phase 3 by defining how authored docs are packaged as runtime resources, how runtime loading avoids filesystem assumptions, and how website and MCP distribution surfaces are decoupled while sharing one authored source.
### Greenfield Framing (Normative)
This Phase 3 design assumes a full refactor with intentional break-and-replace behavior:
1. No compatibility shims, aliases, adapter layers, or dual-read runtime paths.
2. No runtime dependency on repository checkout layout.
3. Runtime docs access is package-resource-only.
4. Canonical authoring remains in `docs/` in source control.
### Research Baseline (Packaging + Runtime)
Authoritative references used for this phase:
1. Python `importlib.resources` docs (`files`, `Traversable`, and zip-safe behavior)
2. Python packaging guidance for wheel/sdist data inclusion
3. Hatchling build target configuration guidance for including non-code files
4. Existing repository constraints from Steps 4-5 (registry-first, deterministic startup, resource-first discovery)
Best-practice conclusions applied to this design:
1. Package docs as build artifacts so runtime reads work from installed wheels.
2. Keep docs source-of-truth in one place (`docs/`) and project into package resource space at build time.
3. Avoid `Path(__file__)`/repo-root probing in runtime paths.
4. Enforce parity across wheel and sdist so local/dev/prod behavior does not drift.
### Phase 3 Responsibilities (Normative)
Phase 3 MUST:
1. Ensure authored markdown under `docs/` is included in wheel and sdist artifacts.
2. Ensure runtime docs registry/document reads use `importlib.resources` only.
3. Ensure MCP runtime behavior is independent of current working directory or checkout structure.
4. Ensure website docs build continues to consume source `docs/` directly.
5. Remove materialization/path-probing coupling from runtime loader code.
6. Preserve deterministic packaged docs layout for registry/resource URI generation.
### Packaging Contract (Wheel + sdist)
Canonical packaging behavior:
1. Source-authored docs remain at repository root: `docs/`.
2. Build projects docs into package resource space under `personal_mcp/docs/` inside artifacts.
3. Runtime anchor for docs loading is `importlib.resources.files("personal_mcp").joinpath("docs")`.
4. Build artifacts MUST include:
- top-level docs pages used by discovery/overview
- `docs/skills/<skill-id>/SKILL.md`
- `docs/skills/<skill-id>/references/**`
Parity requirements:
1. Wheel and sdist contain equivalent docs content for runtime use.
2. Missing docs resources in either artifact is a hard validation failure.
### Build-System Plan (pyproject + build)
Primary target file:
1. `pyproject.toml`
Configuration goals:
1. Add explicit build inclusion rules so docs resources are shipped in wheel artifacts.
2. Add explicit sdist inclusion rules so docs are present for source builds.
3. Keep inclusion deterministic and auditable (no implicit glob side effects beyond intended docs content).
4. Ensure packaged destination path matches runtime anchor (`personal_mcp/docs`).
Implementation note:
1. Use Hatchling-native inclusion mapping (for example force-include or equivalent target-level include mapping) to project `docs/` into package resource space.
2. Prefer one clear packaging path over multiple fallback packaging mechanisms.
### Runtime Loader Contract (No Filesystem Assumptions)
Primary target file:
1. `src/personal_mcp/skills/document_loader.py`
Required runtime behavior:
1. Remove repository-root discovery helpers and path-probing candidates.
2. Remove metadata-based document path overrides that bypass canonical skill layout.
3. Resolve SKILL and reference documents via package-resource-relative paths only.
4. Keep reads UTF-8 and deterministic.
5. Raise explicit errors for missing packaged resources; no fallback probing.
Prohibited runtime behavior:
1. No `Path(__file__).resolve().parents[...]` lookup for docs.
2. No implicit fallback to source-tree `docs/` during runtime reads.
3. No slug-guessing or namespace substitution for path recovery.
### Surface Decoupling Contract (Website vs MCP)
Website surface:
1. Website build pipeline consumes source `docs/` directly (`uv run zensical build`).
2. Static output (`site/`) remains a build artifact served by web mounting logic.
MCP surface:
1. MCP runtime serves docs from packaged resources loaded by registry/resource handlers.
2. MCP does not read `site/` and does not depend on website build artifacts.
Decoupling guarantees:
1. One authored source (`docs/`), two distribution surfaces (website + MCP runtime).
2. Changes to website serving do not alter MCP resource loading semantics.
3. Changes to MCP runtime loader do not require website materialization logic.
### Integration Plan for Existing Modules
Primary integration targets:
1. `pyproject.toml`: add wheel/sdist docs inclusion mapping.
2. `src/personal_mcp/skills/document_loader.py`: replace filesystem probing with package-resource loading.
3. `src/personal_mcp/main.py`: keep startup composition deterministic once registry/resource registration is in place.
4. `src/personal_mcp/mcp.py`: maintain registry-driven resource composition as canonical runtime surface.
5. `src/personal_mcp/web/docs_mount.py`: continue static-site mount behavior without coupling to MCP runtime docs loading.
Cleanup targets:
1. Remove obsolete references to materialization-only modules if no longer present/used.
2. Remove dead code paths that attempt source-tree fallback loading.
### Validation and Test Plan (Phase 3 Scope)
Build/package validation:
1. Build wheel and sdist in CI/local.
2. Inspect artifacts to confirm `personal_mcp/docs/**` exists and includes representative skill/reference files.
3. Install built wheel in isolated environment and verify resource reads via `importlib.resources.files(...)`.
Runtime validation:
1. Run MCP in an environment where repo-root docs paths are unavailable and confirm reads still succeed.
2. Verify representative URIs resolve (skill document and reference document).
3. Confirm startup fails clearly if required packaged docs resources are missing.
Decoupling validation:
1. Run `uv run zensical build` to verify website pipeline still consumes source `docs/`.
2. Confirm MCP runtime does not require `site/` presence.
3. Confirm web static serving behavior is unchanged when docs are built.
Expected command path in this repo:
1. `uv run pytest -q`
2. `uv run zensical build`
### Acceptance Criteria for Phase 3 Completion
Phase 3 is complete when all are true:
1. Wheel and sdist include docs resources in deterministic package paths.
2. Runtime docs loading works from installed artifacts using `importlib.resources` only.
3. Runtime docs loading has no checkout-path dependency and no fallback probing.
4. Website docs build remains source-docs-driven and independent of MCP runtime loading.
5. No compatibility shims, aliases, or dual runtime loader paths exist.
### Non-goals for Phase 3
1. No Step 6 discovery-tool fallback implementation details.
2. No URI aliasing or backward-compat transition mechanics.
3. No redesign of skill frontmatter/schema contracts already finalized in earlier steps.
4. No web UI visual redesign or docs IA overhaul.
+84
View File
@@ -0,0 +1,84 @@
**Step 1 Results: End-State Content Contract**
This section finalizes Step 1 by defining the canonical authored content model.
### Step Deliverable
- Update the current `docs/` directory with the finalized Step 1 content contract from this document.
### Canonical source of truth
- All authored Markdown lives under `docs/`.
- MCP resources and static docs are two distribution surfaces of the same authored files.
- No parallel authored markdown is allowed in `src/` or other package-only paths.
### Canonical skill shape (Anthropic-compatible)
Each skill is one directory under `docs/skills/`:
```text
docs/
skills/
<skill-id>/
SKILL.md
references/
... (one or more markdown files, optional nested folders)
```
Rules:
- `SKILL.md` is required for every skill.
- `references/` is the only place for skill-specific supporting docs.
- Nested folders inside `references/` are allowed so a skill can reorganize internals without changing global architecture.
- Skill directories are independent ownership boundaries; no cross-skill file writes.
### File placement and ownership boundaries
- Top-level project docs stay in `docs/*.md`.
- Skill docs stay in `docs/skills/<skill-id>/...`.
- A skill may link to other skills, but must not store content inside another skill's directory.
- Server/runtime code may index and serve docs, but must not be the source of authored markdown.
### Metadata location constraint
- Skill metadata is embedded in YAML frontmatter in `SKILL.md`.
- No `metadata.yaml` sidecar in the end state.
- Reference lookup metadata (ids to relative paths) is declared from `SKILL.md` frontmatter, not inferred as a hidden global convention.
### Skill-id contract (change-friendly)
`skill-id` is the public identifier and SHOULD satisfy all rules below:
- Format: lowercase kebab-case only.
- Character set: `a-z`, `0-9`, and `-`.
- Must start with a letter.
- No underscores, spaces, dots, or uppercase characters.
- Directory name should equal `skill-id` in each committed revision.
- Frontmatter `id` should equal directory name in each committed revision.
- Treat `skill-id` as immutable after release; any rename is a breaking replacement and clients must move to the new id.
Example valid ids:
- `fastapi-uv-docker`
- `zensical-docs`
- `pytesting`
Example invalid ids:
- `fastapi_uv_docker` (underscore)
- `Zensical-Docs` (uppercase)
- `docs.zensical` (dot)
### Invariants this contract guarantees
- One authored source tree (`docs/`) for both website and MCP.
- One skill directory maps to one skill identity per revision.
- Namespace/slug drift is minimized by keeping directory and frontmatter ids aligned per revision.
- Per-skill reference structure can evolve without changing cross-skill architecture.
- Packaging for stdio is deterministic because authored content is path-stable.
### Non-goals for Step 1
- No URI versioning policy details yet (handled in Step 3).
- No full frontmatter schema details yet (handled in Step 2).
- No migration instructions from current architecture (out of scope for this plan).
+298
View File
@@ -0,0 +1,298 @@
**Step 2 Results: SKILL.md Frontmatter and FastMCP Metadata Contract**
This section finalizes Step 2 by defining the canonical SKILL.md frontmatter schema, separating Anthropic-supported fields from repository extension fields, and mapping frontmatter to FastMCP-native metadata surfaces for resources and tools.
### Step Deliverable
- Update the current `docs/` directory with the finalized Step 2 frontmatter and metadata contract content from this document.
### Anthropic Frontmatter Support (Research Baseline)
Across Anthropic API and Agent Skills specification surfaces:
- Required for custom skill bundles: `name`, `description`.
- `name` constraints (Agent Skills API docs): 1-64 chars, lowercase letters/numbers/hyphens, no XML tags, and must not use reserved words `anthropic` or `claude`.
- `description` constraints (Agent Skills API docs): 1-1024 chars, non-empty, no XML tags.
Portable optional fields from the Agent Skills specification:
- `license`
- `compatibility`
- `metadata`
- `allowed-tools` (experimental)
Claude Code-specific optional fields (supported by Claude Code skills docs):
- `when_to_use`, `argument-hint`, `arguments`
- `disable-model-invocation`, `user-invocable`
- `allowed-tools`, `disallowed-tools`
- `model`, `effort`, `context`, `agent`, `hooks`, `paths`, `shell`
Contract decision for this repository:
- Treat `name` and `description` as required in all SKILL.md files, even where a client could infer defaults.
- Keep Anthropic-facing semantics in standard fields and keep MCP indexing metadata in a namespaced extension block.
- Preserve forward compatibility by allowing additive optional metadata fields over time.
### Canonical Frontmatter Schema For This Repository
Use this exact two-layer pattern:
1. Anthropic layer (portable): top-level fields intended for Anthropic/Agent Skills behavior.
2. Repository layer (runtime indexing): one namespaced block, `x-personal-mcp`, for MCP catalog and routing metadata.
Canonical shape:
```yaml
---
name: <skill-id>
description: <what this skill does and when to use it>
# Optional Anthropic/Agent Skills fields (use only when needed)
when_to_use: <extra trigger guidance>
allowed-tools: <space-separated string or YAML list>
disable-model-invocation: false
user-invocable: true
license: <optional>
compatibility: <optional>
# Repository-specific metadata (authoritative for MCP indexing)
x-personal-mcp:
id: <skill-id>
version: <semver>
tags:
- <tag>
capabilities:
- resource://skills/<skill-id>/document
depends_on: []
references:
<ref-id>:
path: references/<file>.md
mime_type: text/markdown
title: <short title>
---
```
### Repository Metadata Field Rules (`x-personal-mcp`)
- `id` required: must follow Step 1 skill-id rules and equal directory name.
- `version` required: semantic version string.
- `tags` optional: list of kebab-case discovery labels.
- `capabilities` required: list of MCP URIs this skill publishes.
- `depends_on` optional: list of other skill ids.
- `references` optional map:
- key is `ref-id` (kebab-case).
- `path` is a skill-relative markdown path and must stay inside the same skill directory.
- nested folders under `references/` are allowed.
- `mime_type` defaults to `text/markdown` if omitted.
- `title` is an optional display label.
- renaming `ref-id` values is allowed when needed; optional aliases may be used during transitions.
### Pydantic Models For Frontmatter Validation
Define the Step 2 contract with Pydantic v2 models and change-friendly validation.
Normative model sketch:
```python
from __future__ import annotations
import re
from pathlib import PurePosixPath
from typing import Any
from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator
SKILL_ID_RE = re.compile(r"^[a-z][a-z0-9-]*$")
SEMVER_RE = re.compile(r"^(0|[1-9]\d*)\.(0|[1-9]\d*)\.(0|[1-9]\d*)(?:[-+][0-9A-Za-z.-]+)?$")
class ReferenceEntry(BaseModel):
model_config = ConfigDict(extra="ignore", str_strip_whitespace=True)
path: str
mime_type: str = "text/markdown"
title: str | None = None
@field_validator("path")
@classmethod
def validate_reference_path(cls, value: str) -> str:
p = PurePosixPath(value)
if p.is_absolute() or ".." in p.parts:
raise ValueError("reference path must be a relative in-skill path")
if not str(p).startswith("references/"):
raise ValueError("reference path must stay under references/")
if p.suffix.lower() != ".md":
raise ValueError("reference path must target a markdown file")
return str(p)
class PersonalMcpMetadata(BaseModel):
model_config = ConfigDict(extra="ignore", str_strip_whitespace=True)
id: str
version: str
tags: list[str] = Field(default_factory=list)
capabilities: list[str] = Field(min_length=1)
depends_on: list[str] = Field(default_factory=list)
references: dict[str, ReferenceEntry] = Field(default_factory=dict)
@field_validator("id")
@classmethod
def validate_id(cls, value: str) -> str:
if not SKILL_ID_RE.fullmatch(value):
raise ValueError("id must be lowercase kebab-case and start with a letter")
return value
@field_validator("version")
@classmethod
def validate_version(cls, value: str) -> str:
if not SEMVER_RE.fullmatch(value):
raise ValueError("version must be semver")
return value
@field_validator("depends_on")
@classmethod
def validate_depends_on(cls, value: list[str]) -> list[str]:
for dep in value:
if not SKILL_ID_RE.fullmatch(dep):
raise ValueError(f"invalid depends_on skill id: {dep}")
return value
@field_validator("references")
@classmethod
def validate_reference_ids(cls, value: dict[str, ReferenceEntry]) -> dict[str, ReferenceEntry]:
for ref_id in value:
if not SKILL_ID_RE.fullmatch(ref_id):
raise ValueError(f"invalid reference id: {ref_id}")
return value
@model_validator(mode="after")
def ensure_primary_capability(self) -> "PersonalMcpMetadata":
expected = f"resource://skills/{self.id}/document"
if expected not in self.capabilities:
raise ValueError(f"capabilities must include {expected}")
return self
class SkillFrontmatter(BaseModel):
model_config = ConfigDict(extra="ignore", str_strip_whitespace=True)
# Anthropic/Agent Skills fields
name: str = Field(min_length=1, max_length=64)
description: str = Field(min_length=1, max_length=1024)
when_to_use: str | None = None
allowed_tools: str | list[str] | None = Field(default=None, alias="allowed-tools")
disallowed_tools: str | list[str] | None = Field(default=None, alias="disallowed-tools")
disable_model_invocation: bool | None = Field(default=None, alias="disable-model-invocation")
user_invocable: bool | None = Field(default=None, alias="user-invocable")
argument_hint: str | None = Field(default=None, alias="argument-hint")
arguments: str | list[str] | None = None
license: str | None = None
compatibility: str | None = None
metadata: dict[str, str] | None = None
# Repository extension block
x_personal_mcp: PersonalMcpMetadata = Field(alias="x-personal-mcp")
@field_validator("name")
@classmethod
def validate_name(cls, value: str) -> str:
if not SKILL_ID_RE.fullmatch(value):
raise ValueError("name must be lowercase kebab-case and start with a letter")
if "anthropic" in value or "claude" in value:
raise ValueError("name must not contain reserved words anthropic or claude")
return value
@model_validator(mode="after")
def cross_validate(self) -> "SkillFrontmatter":
if self.x_personal_mcp.id != self.name:
raise ValueError("x-personal-mcp.id must exactly match name")
return self
def validate_skill_frontmatter(raw: dict[str, Any], skill_dir_name: str) -> SkillFrontmatter:
model = SkillFrontmatter.model_validate(raw)
if model.name != skill_dir_name:
raise ValueError("frontmatter name must exactly match skill directory name")
return model
```
Validation behavior contract:
- Validate required core fields and relationships during registry load before FastMCP resource/tool registration.
- Allow unknown additive fields so frontmatter can evolve without blocking startup.
- Treat hard contract violations (missing required fields, invalid ids, broken required mappings) as startup errors.
- Treat non-critical compatibility issues as warnings when possible.
- Error messages should include skill path and failing field for CI readability.
Projection mode contract (for Anthropic API upload pipelines):
- Parse with `SkillFrontmatter` first.
- Emit Anthropic-safe frontmatter with standard fields only.
- Serialize repository metadata into standard `metadata` as namespaced keys.
- Preserve the canonical authored source in `x-personal-mcp`; projection output is a build artifact.
### Anthropic Upload Compatibility Rule
- Anthropic documentation guarantees behavior for standard frontmatter fields but does not explicitly guarantee handling of arbitrary unknown top-level keys.
- Therefore, publishing pipelines that target strict API compatibility should support a projection mode that emits only standard frontmatter fields for upload.
- In projection mode, repository extension metadata is serialized into the standard `metadata` field (for example as namespaced keys or JSON-encoded values), while source-of-truth authoring remains in `x-personal-mcp`.
### FastMCP Native Metadata Surfaces (Research Baseline)
Resources (`@mcp.resource` and templates) support native definition metadata:
- `name`, `description`, `mime_type`, `tags`
- `annotations` (`readOnlyHint`, `idempotentHint`)
- `icons`
- `meta` (custom metadata passed through to the MCP client resource object)
- `version`
- `enabled` (deprecated in v3; prefer server-level `mcp.enable()` / `mcp.disable()`)
Resources support runtime metadata:
- `ResourceContent.meta` (item-level)
- `ResourceResult.meta` (result-level `_meta`)
Tools (`@mcp.tool`) support native definition metadata:
- `name`, `description`, `tags`
- `annotations` (`title`, `readOnlyHint`, `destructiveHint`, `idempotentHint`, `openWorldHint`)
- `icons`
- `meta` (custom metadata passed through to the MCP client tool object)
- `version`
- `timeout`, `output_schema`, `run_in_thread`
- `enabled` (deprecated in v3; prefer server-level `mcp.enable()` / `mcp.disable()`)
Tools support runtime metadata:
- `ToolResult.meta` (execution-level metadata for each call)
### Frontmatter To FastMCP Mapping Contract
At server startup, map `x-personal-mcp` fields into FastMCP registration as follows:
- `x-personal-mcp.id` -> canonical URI namespace and identity checks.
- `description` -> default `description` for the primary skill document resource.
- `x-personal-mcp.tags` -> `tags` on resources/tools.
- `x-personal-mcp.version` -> `version` on resources/tools.
- `x-personal-mcp.capabilities` -> registered URI list plus catalog exposure.
- `x-personal-mcp.references[*]` -> resource templates or concrete resources with:
- `mime_type` from reference entry (or default)
- `meta` including `skill_id`, `ref_id`, and source `path`
- read-only annotations for documentation resources
- `x-personal-mcp.depends_on` -> catalog dependency graph metadata and validation checks.
### Invariants This Contract Guarantees
- Anthropic-required frontmatter stays valid for custom skill upload and Claude Code loading.
- MCP-specific metadata remains embedded in SKILL.md frontmatter, with no `metadata.yaml` sidecar.
- FastMCP registration uses only native metadata fields for resources/tools.
- Reference ids and metadata can evolve with low-friction updates while internal file layout under `references/` stays refactor-friendly.
### Non-goals For Step 2
- No URI versioning/deprecation rollout policy details (handled in Step 3).
- No migration script design from existing `metadata.yaml` files.
- No runtime caching/indexing performance tuning details.
+114
View File
@@ -0,0 +1,114 @@
**Step 3 Results: URI Contract and Compatibility Policy**
This section finalizes Step 3 by defining the canonical resource URI contract, template parameter rules, and explicit compatibility/versioning policy for URIs and reference ids.
### Step Deliverable
- Update the current `docs/` directory with the finalized Step 3 URI contract and compatibility policy content from this document.
### Canonical URI Surface (Normative)
The public, preferred URIs are:
1. `resource://catalog/skills_index`
2. `resource://catalog/skills/{skill_id}`
3. `resource://skills/{skill_id}/document`
4. `resource://skills/{skill_id}/references/{ref_id}`
5. `resource://docs/{path*}`
Contract intent:
- Catalog URIs are discovery surfaces.
- Skill URIs are primary per-skill guidance surfaces.
- Docs wildcard URI is a direct authored-markdown access surface under `docs/`.
### URI Semantics
`resource://catalog/skills_index`
- Returns a compact list of skill records for discovery.
- One entry per `skill_id`.
- Must include enough metadata for client-side selection (at minimum id, name, description, tags, capabilities).
`resource://catalog/skills/{skill_id}`
- Returns one normalized record for `skill_id`.
- Must include canonical document URI and declared reference ids.
- Returns not-found when `skill_id` does not exist.
`resource://skills/{skill_id}/document`
- Returns the canonical `SKILL.md` authored content for that skill.
- `skill_id` must match Step 1 stable id rules.
`resource://skills/{skill_id}/references/{ref_id}`
- Returns one reference document declared in the skill frontmatter references manifest.
- `ref_id` is the stable public handle for that reference document.
`resource://docs/{path*}`
- Returns authored markdown at a normalized relative path under `docs/`.
- Supports nested paths via RFC6570 wildcard expansion.
- Typical examples: `index.md`, `usage.md`, `skills/<skill-id>/SKILL.md`, `skills/<skill-id>/references/<file>.md`.
### Template Parameter and Validation Rules
`skill_id`
- Lowercase kebab-case.
- Must satisfy Step 1 stable id rules.
`ref_id`
- Lowercase kebab-case.
- Must be declared in the skills references manifest.
`path*`
- Relative POSIX path only.
- No leading slash.
- No `..` traversal segments.
- Resolves only inside `docs/`.
- This surface is markdown-only in end state (`.md` files).
### URI Versioning Policy
Default rule:
- Keep URIs unversioned by default.
- Allow URI and payload updates when they improve clarity or implementation simplicity.
Breaking-change rule:
- Breaking changes use direct replacement of the canonical URI family.
- No compatibility aliases or dual URI families are maintained in this greenfield phase.
FastMCP version metadata usage:
- Resource `version` metadata MAY be used for implementation/version discovery.
- URI readability and maintainability remain the primary contract.
### Reference ID Compatibility Policy
`ref_id` is the public identifier for a reference document, separate from file path.
Rules:
- Prefer keeping `ref_id` stable when practical.
- File paths may change without URI churn as long as the mapped `ref_id` resolves.
- If a reference is renamed, introduce a new `ref_id` and treat the old one as retired.
- Avoid reusing retired `ref_id` values for unrelated content.
### Invariants This Contract Guarantees
- One canonical URI pattern per core capability surface.
- Fast, low-friction URI evolution through direct replacement of canonical URIs.
- A single canonical catalog URI family with no alias maintenance overhead.
- Reference mappings can evolve with minimal churn.
### Non-goals For Step 3
- No implementation-specific transform wiring details (`VersionFilter`, mounts, provider composition).
- No migration script mechanics for auto-generating aliases.
- No authorization policy design for URI-level access control.
+248
View File
@@ -0,0 +1,248 @@
**Step 4 Results: Docs Registry Loader Design (importlib.resources + Fail-Fast Validation)**
This section finalizes Step 4 by defining a production-ready docs registry loader that reads packaged docs through Python resource APIs, parses SKILL.md frontmatter, validates schema and cross-links, and builds an immutable in-memory registry keyed by skill_id.
### Greenfield Framing (Normative)
This Step 4 design is for the greenfield target state:
1. No legacy metadata sidecars (`metadata.yaml`) are part of the runtime contract.
2. No dual-loader compatibility path is required.
3. Registry loading from packaged resources is the only runtime source of truth.
4. Compatibility shims are prohibited.
### Research Baseline (Python + Design Guidance)
Authoritative references used for this step:
1. Python `importlib.resources` docs (`files`, `as_file`, `Traversable` APIs)
2. Python `importlib.resources.abc` docs (`Traversable`, path traversal semantics, joinpath compatibility notes)
3. Pydantic v2 model/validation docs (`model_validate`, `ValidationError`, strictness and extra handling)
4. Python packaging guidance for including package data in wheels/sdists
Best-practice conclusions applied to this design:
1. Prefer `importlib.resources.files(<package>).joinpath(...)` over filesystem assumptions so stdio deployments from installed wheels work.
2. Treat resources as potentially non-filesystem artifacts (zip-import compatible); only use `as_file(...)` when an actual OS path is required.
3. Validate metadata with explicit Pydantic models and fail startup on contract violations.
4. Keep registry load deterministic (sorted traversal, stable error messages, no hidden fallback mutations).
5. Resolve references via manifest ids declared in frontmatter, not by global file conventions.
### Loader Responsibilities (Normative)
The Step 4 loader MUST:
1. Read canonical docs from package resources (not repo-root paths).
2. Discover all skill directories under `docs/skills/` in packaged resources.
3. For each skill, read and parse `SKILL.md` frontmatter.
4. Validate frontmatter using the Step 2 schema contract.
5. Validate directory/id invariants from Step 1 (directory name equals frontmatter id).
6. Validate URI/reference semantics from Step 3 assumptions.
7. Build a single in-memory registry keyed by `skill_id`.
8. Fail fast on any integrity error before FastMCP resource registration.
9. Precompute compact discovery projections so index resources can be served without reading full markdown bodies at request time.
### Package Resource Contract
Runtime anchor:
1. The loader resolves content from an importable package anchor, for example `personal_mcp`.
2. Docs root is located as `files(anchor).joinpath("docs")` when docs are packaged at package root, or an equivalent configured subpath.
3. Skill root is `docs/skills`.
Resource assumptions:
1. `SKILL.md` is UTF-8 text.
2. Reference files declared in frontmatter are UTF-8 markdown by default unless otherwise declared.
3. Path resolution always remains inside the same skill directory.
### Registry Data Model
Build immutable runtime records with explicit structure:
1. `SkillRecord`
- `skill_id`
- `name`
- `description`
- `version`
- `tags`
- `capabilities`
- `depends_on`
- `document_uri`
- `document_relpath` (canonical resource-relative path)
- `references` map keyed by `ref_id`
2. `ReferenceRecord`
- `ref_id`
- `uri`
- `relpath`
- `mime_type`
- `title`
3. `DocsRegistry`
- `skills_by_id: dict[str, SkillRecord]`
- `skills_in_load_order: list[str]` (deterministic ordering)
- helper indexes for catalog payload generation
- `skills_summary_in_load_order: list[SkillSummaryRecord]` for progressive discovery responses
- filter indexes (for example by tag/capability) derived once at startup
4. `SkillSummaryRecord`
- `skill_id`
- `name`
- `description`
- `tags`
- `capabilities`
- `document_uri`
- optional `version`
Immutability rule:
1. Once built, registry records are treated as read-only for the process lifetime.
2. No runtime mutation during requests; refresh only via process restart.
### Frontmatter Parsing Contract
`SKILL.md` parse steps:
1. Read full markdown text from resource.
2. Parse YAML frontmatter block at file start (between the first two `---` delimiters).
3. Parse YAML with safe loader semantics.
4. Validate parsed object with Step 2 Pydantic model(s).
5. Preserve markdown body as document content payload.
Parsing failure behavior:
1. Missing frontmatter block: startup error.
2. Invalid YAML: startup error with skill path and YAML parser detail.
3. Missing required fields (`name`, `description`, `x-personal-mcp` contract fields): startup error.
### Validation Pipeline (Fail-Fast)
Validation happens in this order:
1. Structural discovery validation
- skill directory exists under `docs/skills`
- required `SKILL.md` exists for each discovered skill
2. Schema validation
- Pydantic frontmatter validation for all required and constrained fields
3. Identity validation
- frontmatter `name` equals `x-personal-mcp.id`
- frontmatter id equals skill directory name
4. Reference manifest validation
- unique `ref_id` keys per skill
- each manifest path is relative, in-skill, and under `references/`
- each manifest target exists and is a file
5. Dependency graph validation
- every `depends_on` target exists in discovered skill set
- no self-dependency
- cycle detection enabled (hard error on cycle)
6. Capability sanity checks
- required primary capability `resource://skills/{skill_id}/document` is present
7. Global uniqueness checks
- no duplicate `skill_id`
- no duplicate canonical resource URIs generated from registry
8. Discovery payload checks
- summary fields required by catalog index are present and non-empty
- summary generation does not require reading markdown body content during request handling
### Error Model and Reporting
Error handling contract:
1. Collect errors per validation phase for clarity, then raise one startup exception containing all findings.
2. Error messages must include:
- skill id (when known)
- packaged relative path
- violated rule
- actionable fix hint
3. If any error exists, registry is not published and FastMCP resource registration does not proceed.
Recommended exception shape:
1. `DocsRegistryValidationError(errors: list[RegistryIssue])`
2. `RegistryIssue` fields: `code`, `message`, `skill_id`, `path`, `hint`
### Determinism and Runtime Safety
Determinism rules:
1. Traverse directories in sorted order.
2. Normalize all stored relative paths to POSIX form.
3. Normalize ids/tags exactly once at parse boundary.
4. Produce stable catalog ordering to reduce client churn.
5. Produce stable summary projections and filter indexes from the same normalized source records.
Runtime safety rules:
1. No dependence on `Path(__file__)` or repository root.
2. No ad-hoc fallback probing across multiple locations.
3. No lazy validation deferred until first request.
### Integration Plan for Existing Modules
Primary integration target:
1. Implement the canonical package-resource-based registry loader in `src/personal_mcp/skills/document_loader.py` as the only supported runtime loader path.
Catalog integration:
1. Update `src/personal_mcp/catalog/server.py` to consume the shared in-memory registry as the only catalog data source.
2. Keep catalog payload normalization deterministic and sourced from registry records only.
Startup wiring:
1. Initialize registry once during app/server startup in `src/personal_mcp/main.py` or equivalent composition point.
2. Pass registry to resource registration step (Step 5).
### Proposed Loader API Surface
Use a small, testable API:
1. `load_docs_registry(*, package_anchor: str, docs_root: str = "docs") -> DocsRegistry`
2. `read_skill_document(registry: DocsRegistry, skill_id: str) -> DocumentPayload`
3. `read_skill_reference(registry: DocsRegistry, skill_id: str, ref_id: str) -> DocumentPayload`
Design constraints:
1. Loader functions are pure relative to package resources and input args.
2. No global mutable singleton required for unit tests.
3. Caching is explicit and owned by startup composition.
### Test and Validation Plan (Step 4 Scope)
Unit tests:
1. valid multi-skill registry load from packaged test fixtures
2. duplicate id detection
3. missing SKILL.md detection
4. invalid frontmatter field constraints
5. broken reference target detection
6. invalid depends_on target detection
7. cycle detection in depends_on graph
8. deterministic output ordering across runs
Packaging/runtime tests:
1. install built wheel in isolated env
2. load registry via `importlib.resources.files(...)`
3. assert representative skill document/reference are readable
Expected command path in this repo:
1. `uv run pytest -q`
### Acceptance Criteria for Step 4 Completion
Step 4 is complete when all are true:
1. Registry loads exclusively from packaged resources.
2. All Step 2 and Step 3 dependent validations are enforced at startup.
3. Invalid docs state blocks startup with actionable diagnostics.
4. Registry is deterministic and immutable for runtime use.
5. Catalog and later resource registration can consume registry without direct filesystem scanning.
### Non-goals for Step 4
1. No FastMCP resource registration wiring details (Step 5).
2. No discovery-tool fallback behavior design (Step 6).
3. No final packaging/build-system migration mechanics (Step 7).
4. No backward-compat alias rollout mechanics in the greenfield baseline.
5. No compatibility layer of any kind (URI aliases, dual reads, adapter shims, or legacy schema bridges).
+221
View File
@@ -0,0 +1,221 @@
**Step 5 Results: Registry-Driven FastMCP Resource Registration (RFC6570 + Startup Safety)**
This section finalizes Step 5 by defining how FastMCP resources are registered from the Step 4 docs registry using RFC6570 URI templates, explicit metadata, and strict duplicate-registration safety.
### Greenfield Framing (Normative)
This Step 5 design is for the greenfield target state:
1. Registry-driven resources are the primary and authoritative discovery/read surface.
2. No legacy per-skill hardcoded resource registration is retained.
3. Resource contracts are defined for net-new clients and replace prior contracts without transition shims.
4. Step 6 tool fallback layers on top of this resource contract, not as a competing source of truth.
5. Breaking changes are intentional in this full-refactor phase.
### Research Baseline (FastMCP + URI Templates)
Authoritative references used for this step:
1. FastMCP Resources and Templates docs (resource decorator, template behavior)
2. FastMCP RFC6570 support docs (simple params, wildcard params, query params)
3. FastMCP duplicate handling docs (`on_duplicate_resources`)
4. FastMCP annotations guidance (`readOnlyHint`, `idempotentHint`)
Best-practice conclusions applied to this design:
1. Use URI templates for parameterized resources instead of generating N static resource handlers.
2. Use wildcard template parameters (`{path*}`) for hierarchical docs paths.
3. Set startup duplicate policy to `on_duplicate_resources="error"` to fail fast on contract collisions.
4. Set explicit `mime_type` and resource annotations for all docs resources.
5. Keep registration deterministic and sourced only from the validated Step 4 registry.
### Registration Responsibilities (Normative)
The Step 5 registration layer MUST:
1. Consume only the validated in-memory registry produced by Step 4.
2. Register canonical resource discovery surfaces and skill document/reference surfaces.
3. Use RFC6570 templates where URI patterns are parameterized.
4. Use wildcard templates where path depth is variable.
5. Attach read-only/idempotent annotations to documentation resources.
6. Set explicit MIME types for all registered resources.
7. Fail startup if duplicate URI/template keys are encountered.
### Canonical Resource Surface (from Registry)
The preferred resources registered in this phase are:
1. `resource://catalog/skills_index`
2. `resource://catalog/skills_index{?q,tag,capability,cursor,limit}` (optional filtered/paginated discovery template)
3. `resource://catalog/skills/{skill_id}`
4. `resource://skills/{skill_id}/document`
5. `resource://skills/{skill_id}/references/{ref_id}`
6. `resource://docs/{path*}`
Registration decision rules:
1. Use static resource registration for fixed singleton endpoints (for example `skills_index`).
2. Use template registration for parameterized endpoints (`{skill_id}`, `{ref_id}`) and optional discovery query templates.
3. Use wildcard template registration for hierarchical docs routing (`{path*}`).
4. Keep the singleton and query-template discovery surfaces semantically equivalent (same schema, query template adds filtering/pagination only).
### Progressive Discovery Contract
Discovery-first behavior for Step 5 resources:
1. `skills_index` returns summaries only (no embedded full SKILL.md bodies).
2. Each summary includes canonical follow-up URIs so clients can progressively fetch detail (`catalog/skills/{skill_id}` then `skills/{skill_id}/document`).
3. Filtered/paginated discovery uses RFC6570 query params (`q`, `tag`, `capability`, `cursor`, `limit`) with deterministic ordering.
4. Handlers should enforce bounded page size and return explicit continuation metadata when pagination is active.
5. Errors for unsupported filter params or invalid cursor/limit are explicit and actionable.
### RFC6570 Template Contract
Path parameters:
1. `{skill_id}` and `{ref_id}` are single-segment template params.
2. `{path*}` is a wildcard param and may capture multi-segment paths separated by `/`.
Validation contract at resource-read time:
1. `skill_id` must exist in registry.
2. `ref_id` must exist in that skills reference manifest.
3. wildcard `path*` must normalize to an allowed docs-relative markdown path.
4. invalid params return explicit not-found or validation errors (no silent fallback).
Template function signature contract:
1. Required URI params must exist as function parameters.
2. Avoid hidden implicit params not represented in template.
3. Keep template handlers side-effect free.
### Metadata and Annotation Contract
Each docs/resource registration should specify explicit metadata:
1. `mime_type`
- skill docs and references: `text/markdown`
- catalog payloads: `application/json`
2. `annotations`
- `readOnlyHint: true`
- `idempotentHint: true`
3. `tags`
- include stable categories such as `catalog`, `skill-doc`, `reference`, `docs`
4. `version`
- project-defined version from registry metadata where applicable
5. `meta`
- include normalized identifiers (for example `skill_id`, `ref_id`, `source_relpath`) when useful
### Startup Safety and Duplicate Policy
FastMCP initialization contract for this phase:
1. Construct the root server with `on_duplicate_resources="error"`.
2. Register all Step 5 resources during startup composition before serving traffic.
3. Treat duplicate registration as a hard startup failure.
Duplicate conflict classes covered:
1. static URI vs static URI collision
2. static URI vs template key collision
3. template URI vs template URI collision
4. conflicting registrations introduced by future aliases without explicit migration handling
### Registration Architecture
Use one dedicated registration module that converts registry records into FastMCP resources.
Recommended API:
1. `register_docs_resources(mcp: FastMCP, registry: DocsRegistry) -> None`
Responsibilities of `register_docs_resources`:
1. register singleton catalog resources
2. register parameterized catalog/detail templates
3. register skill document and reference templates
4. register docs wildcard template
5. apply shared annotations and MIME defaults consistently
Separation of concerns:
1. Step 4 validates and normalizes docs state.
2. Step 5 only registers handlers and reads from validated registry state.
3. Request handlers do not re-discover filesystem/package structure.
### Handler Behavior Contract
Catalog handlers:
1. `skills_index` returns compact deterministic discovery payload (summary records only) and supports progressive follow-up links.
2. `skills/{skill_id}` returns one normalized detail record or not-found.
Skill document handlers:
1. `skills/{skill_id}/document` returns canonical SKILL markdown content.
2. MIME type is always `text/markdown`.
Reference handlers:
1. `skills/{skill_id}/references/{ref_id}` resolves via frontmatter manifest mapping.
2. MIME type is explicit from manifest or defaults to `text/markdown`.
Wildcard docs handler:
1. `docs/{path*}` serves markdown docs under canonical packaged docs tree.
2. traversal outside docs root is blocked.
### Integration Plan for Existing Modules
Primary composition updates:
1. Implement registry-driven registration in [src/personal_mcp/mcp.py](src/personal_mcp/mcp.py) as the canonical resource composition path.
2. Keep [src/personal_mcp/main.py](src/personal_mcp/main.py) responsible for startup wiring order (load registry first, then register resources).
3. Use [src/personal_mcp/catalog/server.py](src/personal_mcp/catalog/server.py) as registry-backed handlers only.
Lifecycle order (required):
1. load and validate registry (Step 4)
2. initialize FastMCP with duplicate error policy
3. register all Step 5 resources/templates
4. start server
### Testing Plan (Step 5 Scope)
Unit/integration tests:
1. resource registration succeeds with valid registry
2. duplicate resource registration fails at startup
3. `skills/{skill_id}` template resolves expected record
4. `skills/{skill_id}/document` returns markdown with correct MIME
5. `skills/{skill_id}/references/{ref_id}` resolves manifest-mapped file
6. `docs/{path*}` resolves nested docs paths and blocks traversal attempts
7. all registered docs resources include `readOnlyHint` and `idempotentHint`
8. catalog payload order is deterministic
9. filtered/paginated `skills_index{?q,tag,capability,cursor,limit}` responses are deterministic and schema-compatible with the singleton index response
10. catalog index payload excludes full markdown bodies and includes follow-up URIs for progressive reads
Smoke tests:
1. list resources includes singleton and template entries
2. read representative skill doc URI and reference URI successfully
3. read representative wildcard docs URI successfully
### Acceptance Criteria for Step 5 Completion
Step 5 is complete when all are true:
1. Resource registration is fully registry-driven (no per-skill hardcoded decorators required for core docs surfaces).
2. RFC6570 templates are used for parameterized URI families, including wildcard where needed.
3. All docs resources declare explicit MIME types and read-only/idempotent annotations.
4. `on_duplicate_resources="error"` is enabled and verified by tests.
5. Startup fails safely on registration conflicts.
### Non-goals for Step 5
1. No tool fallback discovery behavior implementation (Step 6).
2. No packaging build inclusion mechanics (Step 7).
3. No CI gate expansion details (Step 9).
4. No migration shims for legacy URI aliases in the greenfield baseline.
5. No ranking-strategy implementation for discovery tools beyond what is needed to preserve deterministic resource-first discovery contracts.
6. No backward-compat resource aliases, adapter handlers, or dual registration paths.
+243
View File
@@ -0,0 +1,243 @@
**Step 6 Results: Resource-First Discovery and Tool Fallback Contract**
This section finalizes Step 6 by defining discovery behavior for clients that can attach MCP resources and the fallback behavior for clients or chat surfaces that must rely on MCP tools.
### Step Deliverable
- Update the current `docs/` directory with the finalized Step 6 discovery and fallback contract content from this document.
### Primary Source Baseline (Repository Docs)
Step 6 is based on the current project contracts in:
1. `docs/architecture.md` (resource-first architecture and catalog role)
2. `docs/usage.md` (operating flows, bounded loading, and fallback sequence)
3. `docs/copilot.md` (client capability lanes and practical fallback behavior)
4. `docs/mcp_layout.md` (shared content source and thin-tool fallback position)
5. `docs/securing.md` (read-only/public-docs security invariant)
Normative conclusions from those sources:
1. Discovery stays resource-first.
2. Tool fallback is allowed, thin, and read-only.
3. Resources and tools must resolve to the same canonical authored markdown.
4. Fallback behavior should keep context bounded and deterministic.
### FastMCP Source Baseline (Authoritative References)
Step 6 fallback behavior and compatibility-layer expectations align with:
1. [FastMCP server concepts](https://gofastmcp.com/servers/server)
2. [FastMCP resources and resource templates](https://gofastmcp.com/servers/resources)
3. [FastMCP resources-as-tools transform](https://gofastmcp.com/servers/transforms/resources-as-tools)
4. [MCP specification: resources](https://modelcontextprotocol.io/specification/latest/server/resources)
Applied conclusions for this step:
1. Resource contracts remain canonical and should be surfaced directly when clients support resource attachment.
2. Tool-first compatibility layers should wrap canonical resource reads rather than creating alternate authored-content stores.
3. URI-template-backed resource identity remains stable across direct-resource and tool-compatibility access paths.
### Client Tool-Naming Research Baseline
Authoritative and client-specific references to verify during implementation:
1. [MCP specification: tools](https://modelcontextprotocol.io/specification/latest/server/tools)
2. [MCP client concepts](https://modelcontextprotocol.io/docs/learn/client-concepts)
3. [FastMCP tools](https://gofastmcp.com/servers/tools)
4. [FastMCP resources-as-tools transform](https://gofastmcp.com/servers/transforms/resources-as-tools)
5. [VS Code MCP servers](https://code.visualstudio.com/docs/agent-customization/mcp-servers)
6. [VS Code MCP configuration reference](https://code.visualstudio.com/docs/agents/reference/mcp-configuration)
7. [Cursor MCP documentation](https://docs.cursor.com/context/model-context-protocol)
8. [Claude Desktop local MCP server setup](https://support.anthropic.com/en/articles/10949351-getting-started-with-local-mcp-servers-on-claude-desktop)
Baseline naming conclusions:
1. MCP protocol tool identity is the server-advertised `name` returned by `tools/list` and used in `tools/call`.
2. FastMCP tool identity should be treated as the canonical server contract unless a tool is intentionally registered with an explicit alternate name.
3. Clients and host integrations may display, namespace, or internally route tool names with provider-specific prefixes, but those wrappers are not canonical server tool names.
4. Compatibility should be validated by observed `tools/list` and successful `tools/call` behavior in each target client rather than by assuming one global host naming convention.
### Discovery Priority Contract (Normative)
Preferred sequence for skill discovery and loading:
1. `resource://catalog/skills_index`
2. `resource://catalog/skills/{skill_id}`
3. `resource://skills/{skill_id}/document`
4. `resource://skills/{skill_id}/references/{ref_id}` only when needed
Rules:
1. Start from catalog discovery before loading any skill document.
2. Do not skip straight to broad document loading when catalog metadata can narrow choices first.
3. Use `resource://docs/{path*}` only for direct authored-doc access outside skill-specific flows.
### Fallback Activation Rule
Fallback is used only when the active client path cannot reliably attach MCP resources (for example, tool-only chat surfaces).
Rules:
1. Keep the same discovery order semantics as the resource path.
2. If resource attachment is available, prefer resources over tools.
3. Tool fallback must never become a second authoritative content source.
### Tool Fallback Surface (Normative)
The fallback tool surface includes:
1. `list_resources`
2. `read_resource`
3. `search_patterns`
4. `get_pattern_by_id`
5. `get_skill_document_by_id`
Canonical naming rule:
1. The server-level tool contract uses the exact registered FastMCP tool names above.
2. Clients that expose provider-prefixed names (for example, namespaced wrappers) must map those names to the canonical server tool name before invocation.
3. `catalog_get_skill_document_by_id` is not a canonical server tool name for this contract unless an explicit alias is intentionally registered.
Compatibility alias policy:
1. Prefer canonical server tool names over aliases.
2. Add server-side aliases only when a major client cannot reliably map its wrapper name back to the canonical name.
3. Any alias must be read-only, delegate to the same payload builder as the canonical tool, and be documented as compatibility-only.
4. If aliases are added, canonical and alias tools must return byte-for-byte equivalent payloads for the same input.
Fallback order:
1. call `list_resources` to inspect canonical static/template resource surfaces
2. call `read_resource` for catalog URIs and selected skill URIs
3. use thin catalog tools only when additional metadata-first narrowing is needed
Tool behavior requirements:
1. read-only and idempotent semantics
2. deterministic ordering and bounded pagination
3. explicit not-found responses (`found: false` style) where applicable
4. payloads remain schema-aligned with catalog resources
5. tool invocation examples and Copilot guidance must use canonical server tool names to avoid unknown-tool errors
### Major Client Compatibility Plan
Target clients and expected validation:
1. GitHub Copilot in VS Code
- primary path: attach MCP resources when `MCP Resources...` is available
- fallback path: call `list_resources`, `read_resource`, then canonical thin tools only when needed
- validation: confirm Copilot-visible tool inventory includes or can invoke `list_resources`, `read_resource`, `search_patterns`, `get_pattern_by_id`, and `get_skill_document_by_id`
- compatibility risk: host-generated wrapper names may differ from canonical FastMCP names; document any observed wrapper-to-canonical mapping
2. Cursor
- primary path: use the client MCP server configuration and resource/tool surfaces supported by the active Cursor version
- fallback path: prefer resource-backed tools first, then canonical thin tools
- validation: capture Cursor `tools/list` equivalent behavior and verify the canonical tool names or required host mappings
- compatibility risk: Cursor may present MCP tools through its own UI labels or internal routing names
3. Claude Desktop
- primary path: configure the local MCP server and inspect advertised tools/resources in Claude Desktop
- fallback path: invoke canonical server tool names exactly as returned by `tools/list`
- validation: run a local smoke prompt that reads `resource://catalog/skills_index` and loads one skill document through `read_resource` or `get_skill_document_by_id`
- compatibility risk: local server configuration and transport setup may fail before tool-name compatibility is tested
4. Generic MCP clients and SDK-based tests
- primary path: protocol-level `resources/list`, `resources/read`, `tools/list`, and `tools/call`
- fallback path: none beyond the canonical tool contract
- validation: automated smoke tests assert exact tool names returned by `tools/list` and successful calls for canonical names
- compatibility risk: SDK/client libraries may expose helper names that differ from raw protocol names
Implementation checklist:
1. Capture each target client's advertised tool names before adding aliases.
2. Prefer fixing documentation or client-side mapping when the server already advertises canonical names correctly.
3. Add a server-side alias only for a confirmed major-client incompatibility.
4. Add regression tests for canonical names, resource-backed tools, and any intentionally supported aliases.
5. Keep public examples centered on `list_resources`/`read_resource` and canonical thin tool names.
### Resources-As-Tools Compatibility Layer
Step 6 includes a resources-as-tools compatibility layer for clients that can call tools but not attach resources.
Rules:
1. It wraps canonical resource reads rather than re-implementing content transforms.
2. It preserves canonical URIs and metadata semantics.
3. It does not replace the minimal catalog tools listed above.
4. It is interoperability-driven and remains read-only.
### Resource/Tool Parity Contract
Resources and fallback tools must agree on identity and routing metadata.
Parity requirements:
1. `skill_id` and `ref_id` are identical across both paths.
2. canonical URIs in payloads match Step 3 URI rules.
3. skill metadata (`id`, `name`, `description`, `tags`, `capabilities`, `version`) remains consistent.
4. document payload returned by `get_skill_document_by_id` resolves to the same canonical `SKILL.md` content as `resource://skills/{skill_id}/document`.
### Relevance and Ranking Contract
Baseline matching behavior is metadata-first and deterministic.
Rules:
1. Search primarily over normalized skill metadata (id, name, description, tags).
2. Keep deterministic ordering and deterministic pagination behavior.
3. Keep ranking logic transparent and bounded for predictable client behavior.
Optional extension policy:
1. BM25/regex augmentation is allowed only when catalog/tool volume meaningfully harms token efficiency or precision.
2. Any augmentation must preserve canonical ids, URIs, and deterministic tie-breaking.
3. Any augmentation remains discovery-only and does not create alternate content payloads.
### Context-Bounding and Clarification Policy
To prevent context bloat and improve answer quality:
1. load only the most relevant skill document by default
2. load at most two skill documents in one pass unless the user explicitly asks for more
3. if confidence is low after catalog discovery, ask one clarifying question before loading additional skill documents
4. fetch references lazily and only when required
### Security and Safety Constraints
Fallback tools must preserve the project security invariant.
Rules:
1. tool surfaces stay documentation-only and read-only
2. no mutation, shell execution, secret access, or private filesystem exposure
3. all returned content remains safe to publish publicly
### Integration Boundaries
Step 6 integrates with prior steps as follows:
1. Step 4 provides the validated in-memory registry.
2. Step 5 provides canonical resource registration.
3. Step 6 adds fallback discovery/read behavior that reuses the same registry and canonical markdown sources.
Separation-of-concerns rule:
1. Catalog/resource contracts remain canonical.
2. Fallback tools are interoperability adapters, not a parallel architecture.
### Acceptance Criteria for Step 6 Completion
Step 6 is complete when all are true:
1. Resource-first discovery remains the documented and implemented default path.
2. `list_resources` and `read_resource` are available for tool-only clients.
3. Thin catalog tools remain minimal, read-only parity surfaces.
4. Fallback tool outputs map to canonical skill identities and URIs.
5. Context loading is bounded and clarifying-question behavior is documented for low-confidence cases.
6. No second content source is introduced; resources and tools resolve the same authored markdown.
### Non-goals for Step 6
1. No write or side-effecting tools.
2. No alternate authored markdown stores or duplicated skill content pipelines.
3. No guarantee that every client session exposes MCP resource attachment UI.
4. No packaging/build contract changes (handled in Step 7).
5. No CI gate expansion details (handled in later validation/governance steps).
+6
View File
@@ -0,0 +1,6 @@
.venv
__pycache__
.cache*
site/
*.log*
+86
View File
@@ -0,0 +1,86 @@
{
"version": "2.0.0",
"tasks": [
{
"label": "Ruff: Check",
"type": "shell",
"command": "uv",
"args": [
"run",
"ruff",
"check",
"."
],
"options": {
"cwd": "${workspaceFolder}"
},
"problemMatcher": []
},
{
"label": "Ty: Check",
"type": "shell",
"command": "uv",
"args": [
"run",
"ty",
"check"
],
"options": {
"cwd": "${workspaceFolder}"
},
"problemMatcher": []
},
{
"label": "Docs: Build",
"type": "shell",
"command": "uv",
"args": [
"run",
"zensical",
"build"
],
"options": {
"cwd": "${workspaceFolder}"
},
"group": "build",
"problemMatcher": []
},
{
"label": "Server: Run (uvicorn)",
"type": "shell",
"command": "uv",
"args": [
"run",
"uvicorn",
"personal_mcp.main:create_app",
"--factory",
"--host",
"127.0.0.1",
"--port",
"8000",
"--reload"
],
"options": {
"cwd": "${workspaceFolder}"
},
"isBackground": true,
"problemMatcher": []
},
{
"label": "Docker: Compose Up (Build)",
"type": "shell",
"command": "docker",
"args": [
"compose",
"up",
"--build",
"-d"
],
"options": {
"cwd": "${workspaceFolder}"
},
"isBackground": true,
"problemMatcher": []
}
]
}
+54
View File
@@ -0,0 +1,54 @@
FROM python:3.14-slim AS builder
COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/
ENV PYTHONDONTWRITEBYTECODE=1 \
PYTHONUNBUFFERED=1 \
UV_COMPILE_BYTECODE=1 \
UV_LINK_MODE=copy \
UV_LOCKED=1
WORKDIR /app
RUN --mount=type=cache,target=/root/.cache/uv \
--mount=type=bind,source=zensical.toml,target=zensical.toml \
--mount=type=bind,source=docs/,target=docs/ \
uvx zensical build
RUN --mount=type=cache,target=/root/.cache/uv \
--mount=type=bind,source=uv.lock,target=uv.lock \
--mount=type=bind,source=pyproject.toml,target=pyproject.toml \
uv sync --no-install-project
# COPY --chown=appuser:appuser . /app
# RUN --mount=type=cache,target=/root/.cache/uv \
# uv sync --no-editable
FROM python:3.14-slim AS runtime
ENV PYTHONDONTWRITEBYTECODE=1 \
PYTHONUNBUFFERED=1 \
PATH="/app/.venv/bin:$PATH" \
PERSONAL_MCP_SITE_DIR=/app/site
WORKDIR /app
EXPOSE 8765
RUN groupadd --system --gid 1001 appuser && \
useradd --system --uid 1001 --gid appuser appuser
COPY --from=ghcr.io/astral-sh/uv:latest --chown=appuser:appuser /uv /uvx /bin/
COPY --from=builder --chown=appuser:appuser /app/.venv /app/.venv
COPY --from=builder --chown=appuser:appuser /app/site /app/site
COPY --chown=appuser:appuser ./docs /app/docs
RUN --mount=type=cache,target=/root/.cache/uv \
--mount=type=bind,source=uv.lock,target=uv.lock \
--mount=type=bind,source=pyproject.toml,target=pyproject.toml \
--mount=type=bind,source=src/,target=src/ \
uv sync --no-editable
USER appuser
CMD ["uvicorn", "personal_mcp.main:create_app", "--factory", "--host", "0.0.0.0", "--port", "8765"]
+10
View File
@@ -0,0 +1,10 @@
services:
personal-mcp:
build:
context: .
dockerfile: Dockerfile
restart: unless-stopped
ports:
- "8765:8765"
volumes:
- ./docs:/app/src/personal_mcp/docs
+266
View File
@@ -0,0 +1,266 @@
---
icon: lucide/library
---
# Architecture
## Overview
The platform is implemented as a resource-first MCP system with an integrated static documentation surface. The same methodology content powers both MCP resources and the published docs site.
An MCP server is a runtime that exposes machine-readable resources and tools through stable interfaces so AI clients can discover and consume context consistently. Here, the server's role is intentionally narrow: publish canonical methodology documents as resources, keep discovery predictable through a catalog layer, and serve the same source material as pre-built static documentation.
The system is complete in three layers:
1. Canonical methodology is maintained in Markdown skill documents.
2. Catalog resources provide normalized discovery.
3. Zensical builds a static site from those same Markdown sources and the FastAPI app serves it in the FastMCP runtime process.
Prompt documents under `docs/prompts/` are also indexed and exposed as first-class catalog and prompt surfaces.
This architecture is anchored by three contracts:
1. Docs-first authored content contract under `docs/` with strict per-skill ownership.
2. `SKILL.md` frontmatter contract with Anthropic fields plus `x-personal-mcp` metadata.
3. Canonical resource URI contract with break-and-replace policy for contract changes.
Detailed contract pages:
1. [Content Contract](./contracts/index.md#content-contract)
2. [Frontmatter Contract](./contracts/frontmatter.md)
3. [URI Contract](./contracts/uris.md)
This architecture keeps authored content human-friendly while preserving machine-stable contracts.
## Intent
The architecture is designed to satisfy three long-term requirements:
1. Methodology must be editable as markdown by humans.
2. Agents must consume stable, discoverable resource contracts, with a minimal read-only catalog tool fallback for constrained clients.
3. Public documentation must be pre-built static output served from the application runtime without a separate docs service.
## System Model
### Pattern Modules
Each skill encapsulates one methodology domain in a docs-owned directory:
1. `docs/skills/<skill-id>/SKILL.md`
2. `docs/skills/<skill-id>/references/...`
The skill document and references are the authored source of truth; runtime code indexes and serves these files without becoming a second authored source.
Each skill publishes resource families:
1. document
The document resource returns canonical Markdown, while clients can perform any downstream section extraction they need.
### Prompt Modules
Prompt guidance can be authored in `docs/prompts/` using either canonical prompt directories (`docs/prompts/<prompt-id>/PROMPT.md`) or legacy markdown files during migration.
Prompt modules publish two additive surfaces:
1. prompt resources for catalog and document retrieval
2. MCP prompt objects for prompt-list/get-prompt style client workflows
This keeps authored markdown as source-of-truth while allowing clients to discover and invoke prompts directly.
### Catalog Module
The catalog is the canonical discovery layer and publishes normalized records for all modules. It may also expose a minimal set of read-only discovery tools that resolve back to the same canonical markdown content when a client chat surface does not expose MCP resource attachment.
Typical catalog resources:
1. resource://catalog/skills_index
2. resource://catalog/skills_index{?q,tag,capability,cursor,limit}
3. resource://catalog/skills/{skill_id}
4. resource://catalog/prompts_index
5. resource://catalog/prompts_index{?q,tag,cursor,limit}
6. resource://catalog/prompts/{prompt_id}
Only canonical catalog resources are part of the runtime contract in this phase.
### Registry Loader
Importing the package does not read or parse documentation. The MCP server and FastAPI application factories request the registry when constructing a runnable server, using packaged resources through `importlib.resources.files(...)` and `Traversable` APIs.
Loader responsibilities:
1. Parse SKILL.md frontmatter for each skill.
2. Validate schema and cross-field constraints before any resource registration.
3. Build an in-memory registry keyed by `skill_id`.
4. Fail fast for duplicate ids, missing markdown files, and broken reference mappings.
The immutable registry is cached for the process lifetime. Each Uvicorn worker constructs and retains its own registry because worker processes do not share Python objects. Registry load failure is a server-factory startup error, not a package-import error or partial runtime warning.
### Content Sources
Content is authored in markdown under `docs/` and managed as long-form reference material. Skill documents and companion references now live under `docs/skills/`, while project-authored pages remain alongside them in the docs tree. Resource handlers expose the same authored documents through stable resource URIs.
The repository root `docs/` directory is the only authored source. The `src/personal_mcp/docs` path is a relative symlink to that directory for source-checkout and editable-install workflows; it is not a second content tree and packaging does not depend on traversing it.
For wheel builds, [Hatchling forced inclusion](https://hatch.pypa.io/latest/config/build/#forced-inclusion) maps the root `docs/` tree to `personal_mcp/docs/`. The wheel therefore contains regular resource files at that destination rather than a symlink. Runtime registry loading uses [`importlib.resources.files`](https://docs.python.org/3/library/importlib.resources.html#importlib.resources.files) and `Traversable` operations from the `personal_mcp` package anchor, so it does not depend on the repository layout or current working directory.
### Static Docs Surface
Static docs are built directly from two markdown source streams:
1. Project-authored docs pages
2. Skill and reference markdown pages
The merged docs tree is built by Zensical into static files and served by the FastAPI app.
Generated `site/` files are deployment assets for the human-facing static site. They are separate from the authored Markdown resources packaged under `personal_mcp/docs/`.
## Data Flow
```mermaid
flowchart TD
A[Authored Markdown] --> C[Resource Handlers]
B[Pattern Metadata] --> D[Catalog Resources]
A --> E[Zensical Static Build]
E --> H[FastAPI Static Mount]
H --> I[Served Docs Site]
D --> I
```
## Contracts
### Metadata Contract
Each skill declares frontmatter in `docs/skills/<skill-id>/SKILL.md`.
For the full field-level contract, validation model, and FastMCP metadata mapping, see [Frontmatter Contract](./contracts/frontmatter.md).
Anthropic-facing required fields:
1. name
2. description
Repository indexing metadata is declared in `x-personal-mcp`:
1. id
2. version
3. tags
4. capabilities
5. optional references map (for nested entries, overrides, and aliases)
No `metadata.yaml` sidecar is part of the end-state contract.
### URI Contract
Canonical resource URIs are:
For the full URI semantics, parameter validation rules, and compatibility policy, see [URI Contract](./contracts/uris.md).
1. resource://skills/<skill_id>/document
2. resource://skills/<skill_id>/references/<ref_id>
3. resource://catalog/skills_index
4. resource://catalog/skills_index{?q,tag,capability,cursor,limit}
5. resource://catalog/skills/{skill_id}
6. resource://docs/{path*}
7. resource://catalog/prompts_index
8. resource://catalog/prompts_index{?q,tag,cursor,limit}
9. resource://catalog/prompts/{prompt_id}
10. resource://prompts/{prompt_id}/document
Validation rules:
1. `skill_id` is lowercase kebab-case and must satisfy the stable skill id contract.
2. `ref_id` is lowercase kebab-case and must resolve from either:
- top-level auto-discovery of `references/*.md` filename stems, or
- an explicit `x-personal-mcp.references` entry.
3. `path*` resolves only to normalized markdown paths under `docs/`.
### Resource Registration Contract
Resources are registered from the validated registry, not by ad hoc per-skill hardcoding.
Registration rules:
1. Use RFC6570 URI templates where appropriate.
2. Mark documentation resources as read-only and idempotent.
3. Set explicit mime types for resource responses.
4. Configure duplicate URI handling with `on_duplicate="error"` for startup safety.
This keeps runtime behavior deterministic and prevents accidental URI collisions.
### Versioning Rule
URIs are unversioned and canonical in this phase.
1. Breaking URI changes are handled as direct replacement.
2. No compatibility aliases or dual URI families are maintained.
## Static Hosting Pattern
The docs site is pre-built and served by the same FastAPI runtime process used by the MCP app.
Runtime behavior:
1. App starts.
2. FastAPI mounts the static docs output directory.
3. Requests to docs paths are served as static assets.
This provides a single deployment artifact with no runtime markdown rendering dependency.
## Advantages
### Single Source of Truth
Methodology is authored once and reused in both MCP resources and docs pages.
### High-Fidelity Agent Context
Resources expose the same canonical Markdown that humans author and review.
### Operational Simplicity
A single app process serves MCP and docs surfaces.
### Long-Term Maintainability
Markdown remains easy to review, while contracts remain stable for clients.
### Client Independence
Clients can use Ask, Edit, or Agent modes without requiring prompt-first orchestration. Prompt objects are available as an additive MCP surface, while resource retrieval remains the canonical source path. MCP affordances are still chat-surface-dependent: some clients or sessions expose resource attachment directly, while others make tool invocation the more reliable retrieval path.
## Authoring and Publishing Lifecycle
1. Update markdown reference content.
2. Update metadata if capability surface changes.
3. Build static docs with Zensical.
4. Serve built output through FastAPI static mount.
## Scope and Non-Goals
In-scope:
1. Resource-first methodology delivery
2. Catalog-based discovery
3. Pre-built static docs hosting in app runtime
Out-of-scope:
1. Prompt-first orchestration as the primary interface
2. Large tool inventories duplicating static guidance across skill modules
3. Separate dynamic docs service at runtime
Allowed exception:
1. A small catalog-level tool layer is acceptable when it improves client interoperability without creating a second source of truth for skill content.
## Example Content Inputs
Existing markdown reference sets are valid examples of authored source material for this architecture:
1. docs/skills/pytesting/references/pytest-docs.md
2. docs/skills/python-logging/references/python-logging-docs.md
3. docs/skills/python-logging/references/json-file-logging.md
4. docs/skills/fastapi-uv-docker/references/fastapi-best-practices.md
These inputs are treated as content sources, while resource URIs and catalog payloads remain the machine-facing contracts.
+262
View File
@@ -0,0 +1,262 @@
---
icon: lucide/pencil
---
# Authoring Guide
This page defines the practical authoring workflow for this repository so Markdown remains the single source of truth for both published docs and MCP resources.
Primary references:
- [Skill contract](./contracts/skill_contract.md)
- [Prompt contract](./contracts/prompt.md)
- [Frontmatter contract](./contracts/frontmatter.md)
- [URI contract](./contracts/uris.md)
- [Zensical documentation authoring skill](./skills/zensical-docs/SKILL.md)
## What You Author
This repository has two primary authored content types:
1. Skills under `docs/skills/<skill-id>/`.
2. Prompts under `docs/prompts/<prompt-id>/`.
Each module keeps one canonical document plus optional references:
```text
docs/
skills/<skill-id>/
SKILL.md
references/
*.md
prompts/<prompt-id>/
PROMPT.md
references/
*.md
```
## Source Tree Ownership
Edit content only under the repository root `docs/` directory. The `src/personal_mcp/docs` path is a relative symlink provided so package-oriented tooling and editable installs see the same files; do not replace it with copied content or author files through a second tree.
[Hatchling forced inclusion](https://hatch.pypa.io/latest/config/build/#forced-inclusion) projects root `docs/` into `personal_mcp/docs/` when building the wheel. Installed code reads that destination through [`importlib.resources`](https://docs.python.org/3/library/importlib.resources.html), while Zensical continues to build the human-facing site directly from root `docs/`.
Package import does not load these resources. A runnable MCP or FastAPI server loads and validates them when its factory runs, then caches the immutable registry for that process. Restart initialized development or worker processes after changing authored Markdown.
## Authoring Principles
1. Keep Markdown as the canonical source and avoid duplicating content into alternate metadata files.
2. Prefer resource-first discovery paths (`resource://catalog/...` then `resource://skills/...` or `resource://prompts/...`).
3. Keep pages focused and composable: overview in the primary doc, details in `references/`.
4. Use descriptive inline links for external sources instead of bare URLs.
5. Use stable ids and slugs; renames are breaking changes and should be intentional.
## Skill Authoring Workflow
When creating or updating a skill:
1. Confirm slug format is lowercase kebab-case.
2. Keep directory name, `name`, and `x-personal-mcp.id` aligned.
3. Ensure capabilities include `resource://skills/<skill-id>/document`.
4. Place supporting material under `references/`.
5. Use explicit frontmatter reference entries only when you need overrides or nested mappings.
Recommended sequence:
1. Draft `SKILL.md` intent and routing sections.
2. Add or refine `references/*.md`.
3. Verify links and example commands.
4. Run docs build and tests.
For exact metadata rules, see [Frontmatter contract](./contracts/frontmatter.md) and [Skill contract](./contracts/skill_contract.md).
## Prompt Authoring Workflow
When creating or updating a prompt module:
1. Keep one canonical `PROMPT.md`.
2. Keep `name`, `x-personal-mcp.id`, and directory slug aligned.
3. Include `resource://prompts/<prompt-id>/document` in capabilities.
4. Define prompt arguments in `x-personal-mcp.arguments` when inputs are required.
5. Keep long rationale and source notes in `references/` to preserve prompt clarity.
For exact structure, see [Prompt contract](./contracts/prompt.md).
## Prompt Argument Mechanics
When defining prompt inputs, keep argument metadata aligned with the prompt contract and runtime behavior.
1. Define arguments under `x-personal-mcp.arguments` as a map keyed by argument name.
2. Argument names must match Python identifier format: `^[A-Za-z_][A-Za-z0-9_]*$`.
3. Each argument entry supports only:
- `title` (optional)
- `description` (optional)
- `required` (optional, defaults to `false`)
4. Unknown argument fields are rejected by strict frontmatter validation.
5. Prompt argument metadata appears in `resource://catalog/prompts/{prompt_id}`, and MCP prompt objects expose the same arguments for prompt-list/get-prompt workflows.
6. Enum-like constraints are not a native argument field; encode allowed values in `description`.
### Frontmatter Safety Rules
Use these rules to avoid YAML parse failures in prompt and skill frontmatter:
1. Quote any scalar value that contains `:` (for example, `description: "Enum: skill | prompt | shim"`).
2. Prefer quoted scalars for values with reserved YAML characters such as `#`, `{}`, `[]`, or leading `*`.
3. If a description needs multiple lines, use a block scalar (`|`) instead of packing punctuation-heavy text into one line.
4. Keep frontmatter keys simple and contract-bound; do not add undeclared argument fields.
### Validation Timing
Run validation immediately after frontmatter edits, not only at the end of a task:
1. First pass after metadata changes: `uv run zensical build`
2. Prompt/skill load verification: `uv run pytest -q`
3. Final full pass before completion: run the full checklist in [Validation Checklist](#validation-checklist)
Example:
```yaml
x-personal-mcp:
arguments:
artifact_type:
title: Artifact type
description: Allowed values are skill, prompt, or shim.
required: true
scope_glob:
title: Scope glob
description: Optional applyTo glob for shim outputs.
required: false
```
References:
1. [Frontmatter contract](./contracts/frontmatter.md)
2. [URI contract](./contracts/uris.md)
3. [Resource-First Pattern Module Architecture](./architecture.md)
4. [Prompt objects concept docs](https://modelcontextprotocol.io/docs/learn/server-concepts#prompts)
## Writing Quality Rules
Apply these defaults to all docs pages:
1. Prefer short sections with strong headings over long unbroken prose.
2. Keep claims source-linked, especially for MCP, FastMCP, pytest, FastAPI, SQLAlchemy, and Zensical behavior.
3. Prefer relative links for internal docs paths.
4. Use code blocks for commands and configuration snippets.
5. Keep examples minimal and actionable.
Source examples:
- [Model Context Protocol docs](https://modelcontextprotocol.io/docs/getting-started/intro)
- [FastMCP docs](https://gofastmcp.com/getting-started/welcome)
- [Zensical docs](https://zensical.org/docs/)
## Authoring for GitHub Copilot
For resource selection or tool-based matching to work well, each skill should have:
1. precise `description`
2. focused `tags`
3. explicit `capabilities`
4. stable `id` and slug naming
Weak metadata reduces Copilot match quality and increases wrong context injection.
### Copilot Instruction Authoring Pattern
If you want Copilot to use `personal-mcp` skill content more reliably, instruction files should describe three things clearly:
1. when MCP-backed skill guidance is relevant
2. which retrieval path Copilot should prefer first
3. how much skill context it should load before answering
Instructions strongly steer discovery behavior, but they do not force VS Code to auto-attach MCP resources. Keep wording explicit about preferred path and fallback path.
Repository policy:
1. start from catalog discovery
2. prefer MCP resources when the current chat surface exposes resource attachment
3. fall back to catalog tools when resource attachment is unavailable
4. keep loaded skill context bounded
Suggested instruction text:
```md
When a task may match a documented implementation pattern from `personal-mcp`:
1. Start with catalog-first discovery.
2. Prefer MCP resources when the chat surface exposes resource attachment.
3. If MCP resource attachment is unavailable, use `list_resources`/`read_resource` first, then thin catalog tools if needed.
4. Load only the most relevant skill document, or at most 2 skill documents.
5. Reconcile loaded skill guidance with the actual repository code before making changes.
Preferred resource order:
1. `resource://catalog/skills_index`
2. `resource://catalog/skills/{skill_id}`
3. `resource://skills/<skill-id>/document`
4. `resource://skills/<skill-id>/references/<ref-id>` when needed
Preferred tool fallback order:
1. `list_resources`
2. `read_resource`
3. `search_patterns`
4. `get_pattern_by_id`
5. `get_skill_document_by_id`
Compatibility aliases for clients that use `catalog_*` naming are also available:
1. `catalog_search_patterns`
2. `catalog_get_pattern_by_id`
3. `catalog_get_skill_document_by_id`
4. `catalog_search_prompts`
5. `catalog_get_prompt_by_id`
Use canonical names first; aliases exist only to preserve interoperability when a client emits non-canonical names.
If confidence is low after discovery, ask one clarifying question before loading more context.
```
This is guidance, not a guarantee. It defines a reliable policy while preserving the resource-first architecture.
Thin shim path binding guidance for MCP consumers is covered in [Skill Usage Mechanics](./usage.md).
## Zensical Details
When adding or restructuring pages:
1. Update navigation in `zensical.toml`.
2. Ensure top-level pages include frontmatter with an icon.
3. Keep naming and labels concise so navigation remains scannable.
Top-level page pattern:
```yaml
---
icon: lucide/pencil
---
```
## Validation Checklist
Run these checks before considering authoring changes complete:
```bash
uv run zensical build
uv run ruff check .
uv run ty check
uv run pytest
```
Address any errors or warnings that result.
If a change only affects docs content, `uv run zensical build` is still required.
## Quick Authoring Checklist
1. Correct location (`skills/` or `prompts/`).
2. Frontmatter id and slug alignment.
3. Capability URI present.
4. Links valid and descriptive.
5. Navigation updated when needed.
6. Validation commands passed.
+234
View File
@@ -0,0 +1,234 @@
---
icon: lucide/braces
---
# Frontmatter Contract
This page defines the `SKILL.md` frontmatter and FastMCP metadata contract.
Prompt modules use the same contract style in `docs/prompts/<prompt-id>/PROMPT.md` with prompt-specific capability and MCP-aligned prompt argument metadata.
## Validated Frontmatter Surface
The registry runtime validates a strict, standard-only frontmatter surface:
1. Top-level fields accepted for skills: `name`, `description`, `x-personal-mcp`.
2. Top-level fields accepted for prompts: `name`, `description`, `x-personal-mcp`.
3. Unknown top-level fields are rejected during registry load.
Skill and prompt identifier rules:
1. `name` is required, 1-64 chars, lowercase kebab-case, and must not contain `anthropic` or `claude`.
2. `description` is required, 1-1024 chars.
3. `x-personal-mcp.id` must exactly match `name`.
4. Directory slug must exactly match `name`.
Capability invariants:
1. Skill capabilities must include `resource://skills/<skill-id>/document`.
2. Prompt capabilities must include `resource://prompts/<prompt-id>/document`.
Repository contract decisions:
1. Treat `name` and `description` as required in all `SKILL.md` files.
2. Keep only validated standard fields at top level.
3. Keep MCP indexing metadata in a namespaced extension block.
4. Reject unsupported optional top-level fields until explicit model support is added.
Reference specs:
1. MCP prompts data types: [Prompts](https://modelcontextprotocol.io/specification/latest/server/prompts)
2. MCP schema reference for `Prompt` and `PromptArgument`: [Schema](https://modelcontextprotocol.io/specification/latest/schema)
## Canonical Frontmatter Schema
Use this two-layer pattern:
1. Anthropic layer: top-level fields intended for Anthropic and Agent Skills behavior.
2. Repository layer: one namespaced block, `x-personal-mcp`, for MCP catalog and routing metadata.
Canonical shape:
```yaml
---
name: <skill-id>
description: <what this skill does and when to use it>
# Repository-specific metadata
x-personal-mcp:
id: <skill-id>
version: <semver>
tags:
- <tag>
capabilities:
- resource://skills/<skill-id>/document
# Optional: overrides and nested references only.
# Top-level references/*.md are auto-discovered.
references:
<ref-id>:
path: references/<file>.md
mime_type: text/markdown
title: <short title>
---
```
## Repository Metadata Field Rules
Rules for `x-personal-mcp`:
1. `id` is required, must follow the skill id rules from the content contract, and must equal the directory name.
2. `version` is required and must be a semantic version string.
3. `tags` is optional and should be a list of kebab-case discovery labels.
4. `capabilities` is required and lists the MCP URIs the skill publishes.
5. `references` is an optional map keyed by `ref-id` for overrides and nested entries.
Prompt-specific additions:
1. `arguments` is an optional map keyed by argument name.
2. Each argument supports optional `title`, optional `description`, and optional `required`.
3. This aligns with MCP `PromptArgument` shape (`name`, optional `title`, optional `description`, optional `required`) where `name` is represented by the map key.
4. Prompt `capabilities` must include `resource://prompts/<prompt-id>/document`.
Example prompt frontmatter:
```yaml
---
name: initial-test-structure
description: Generate a baseline pytest test layout for a target scope.
x-personal-mcp:
id: initial-test-structure
version: 1.0.0
tags:
- pytest
- testing
capabilities:
- resource://prompts/initial-test-structure/document
arguments:
target_scope:
title: Target scope
description: Target package or module under test.
required: true
---
```
Reference entry rules:
1. `ref-id` is lowercase kebab-case.
2. `path` is a skill-relative markdown path and must stay inside the same skill directory.
3. Top-level files under `references/*.md` are auto-discovered with `ref-id` derived from a normalized filename stem (lowercase kebab-case).
4. Nested folders under `references/` are not auto-discovered and must be declared explicitly.
5. `mime_type` defaults to `text/markdown` when omitted.
6. `title` is an optional display label.
7. Renaming `ref-id` values is allowed when needed; optional aliases may be used during transitions.
## Auto-Generated Reference IDs
Top-level markdown files directly under `references/` are auto-registered as MCP references even when `x-personal-mcp.references` is empty.
How `ref-id` is derived:
1. Start from the filename stem (without `.md`).
2. Normalize to lowercase kebab-case.
3. Publish at `resource://skills/<skill-id>/references/<ref-id>`.
Examples:
1. `references/ruff-docs.md` -> `ref-id: ruff-docs`
2. `references/Ruff Integrations.md` -> `ref-id: ruff-integrations`
3. `references/python_logging_docs.md` -> `ref-id: python-logging-docs`
When to use explicit `x-personal-mcp.references` entries:
1. The file is nested, for example `references/guides/ci.md`.
2. You need to override defaults (`title`, `mime_type`, or custom `ref-id`).
3. You need compatibility aliases during a rename.
## Validation Models
The normative runtime model uses strict Pydantic v2 validation:
1. Models are immutable (`frozen=True`) and reject unknown fields (`extra="forbid"`).
2. `SkillFrontmatter` accepts only `name`, `description`, and `x-personal-mcp`.
3. `PromptFrontmatter` accepts only `name`, `description`, and `x-personal-mcp`.
4. `PromptArgumentEntry` accepts only optional `title`, optional `description`, and optional `required`.
5. Skill and prompt metadata enforce semver, kebab-case ids, capability requirements, and id/name/directory consistency.
6. Reference paths are validated as markdown files under `references/`.
Validation behavior contract:
1. Validate required core fields and relationships during registry load before FastMCP resource or tool registration.
2. Reject unknown or unsupported fields at parse and model-validation time.
3. Treat hard contract violations, including missing required fields, invalid ids, and broken required mappings, as startup errors.
4. Keep failure messages path-aware and field-specific for CI readability.
Projection mode contract for Anthropic API upload pipelines:
1. Parse with `SkillFrontmatter` first.
2. Emit Anthropic-safe frontmatter with standard fields only.
3. Preserve `x-personal-mcp` in source-of-truth documents; projection output is a build artifact.
## Anthropic Upload Compatibility Rule
1. Anthropic documentation guarantees behavior for standard frontmatter fields but does not explicitly guarantee handling of arbitrary unknown top-level keys.
2. Publishing pipelines that target strict API compatibility should support a projection mode that emits only standard frontmatter fields for upload.
3. Source-of-truth authoring remains in `x-personal-mcp`; upload payload shape is an explicit build concern.
## FastMCP Native Metadata Surfaces
Resources support native definition metadata:
1. `name`
2. `description`
3. `mime_type`
4. `tags`
5. `annotations`, including `readOnlyHint` and `idempotentHint`
6. `icons`
7. `meta`
8. `version`
9. `enabled`, which is deprecated in FastMCP v3 in favor of server-level enable and disable controls
Resources also support runtime metadata through `ResourceContent.meta` and `ResourceResult.meta`.
Tools support native definition metadata:
1. `name`
2. `description`
3. `tags`
4. `annotations`, including `title`, `readOnlyHint`, `destructiveHint`, `idempotentHint`, and `openWorldHint`
5. `icons`
6. `meta`
7. `version`
8. `timeout`
9. `output_schema`
10. `run_in_thread`
11. `enabled`, which is deprecated in FastMCP v3 in favor of server-level enable and disable controls
Tools also support runtime metadata through `ToolResult.meta`.
## Frontmatter To FastMCP Mapping Contract
At server startup, map `x-personal-mcp` into FastMCP registration as follows:
1. `x-personal-mcp.id` defines the canonical URI namespace and identity checks.
2. `description` becomes the default description for the primary skill document resource.
3. `x-personal-mcp.tags` maps to resource and tool tags.
4. `x-personal-mcp.version` maps to resource and tool version metadata.
5. `x-personal-mcp.capabilities` becomes the registered URI list and catalog exposure.
6. `x-personal-mcp.references[*]` becomes resource templates or concrete resources with `mime_type`, read-only annotations, and `meta` that includes `skill_id`, `ref_id`, and source `path`.
## Invariants
This contract guarantees:
1. Anthropic-required frontmatter stays valid for custom skill upload and Claude Code loading.
2. MCP-specific metadata remains embedded in `SKILL.md` frontmatter, with no `metadata.yaml` sidecar.
3. FastMCP registration uses native metadata fields for resources and tools.
4. Reference ids and metadata can evolve with low-friction updates while internal file layout under `references/` stays refactor-friendly.
## Non-Goals
This contract does not define:
1. URI versioning and deprecation rollout policy details.
2. Migration script design from existing `metadata.yaml` files.
3. Runtime caching and indexing performance tuning.
+88
View File
@@ -0,0 +1,88 @@
---
icon: lucide/file-check-2
---
# Contracts
This section groups the core data and contract documents for the repository.
## Pages
1. [Prompt Contract](./prompt.md)
2. [Skill Contract](./skill_contract.md)
3. [Frontmatter Contract](./frontmatter.md)
4. [URI Contract](./uris.md)
Use these pages as the normative source for authored content layout, frontmatter schema, and canonical MCP URI semantics.
## Content Contract
This page defines the authored content contract for the docs-first MCP architecture.
## Canonical Source Of Truth
1. All authored Markdown lives under `docs/`.
2. MCP resources and static docs are two distribution surfaces of the same authored files.
3. No parallel authored markdown is allowed in `src/` or other package-only paths.
## Canonical Content Shape
Authored content is organized under `docs/`:
```mermaid
---
config:
treeView:
rowIndent: 20
lineThickness: 2
themeVariables:
treeView:
labelColor: '#FFFFFF'
lineColor: '#FFFFFF'
---
treeView-beta
"docs/"
"*.md (top-level docs pages)"
"contracts/"
"prompt.md"
"skill_contract.md"
"frontmatter.md"
"uris.md"
"prompts/"
"<prompt-id>/"
"PROMPT.md"
"skills/"
"<skill-id>/"
"SKILL.md"
"references/..."
```
## File Placement And Ownership Boundaries
1. Top-level project docs stay in `docs/*.md`.
2. Skill docs stay in `docs/skills/<skill-id>/...`.
3. Prompt docs stay in `docs/prompts/<prompt-id>/...`.
4. A skill or prompt may link across sections, but must not store content in another artifact's directory.
5. Server and runtime code may index and serve docs, but must not be the source of authored markdown.
## Delegated Contracts
1. Skill-specific directory, metadata, and id rules are defined in [Skill Contract](./skill_contract.md).
2. Prompt-specific directory, metadata, and id rules are defined in [Prompt Contract](./prompt.md).
## Invariants
This contract guarantees:
1. One authored source tree in `docs/` for both website and MCP.
2. Skill and prompt artifacts remain path-stable within their own sections.
3. Cross-surface publishing remains deterministic because authored content paths are canonical.
## Non-Goals
This contract does not define:
1. URI versioning policy details.
2. The full frontmatter schema.
3. Detailed skill rules (see [Skill Contract](./skill_contract.md)).
4. Detailed prompt rules (see [Prompt Contract](./prompt.md)).
+75
View File
@@ -0,0 +1,75 @@
---
icon: lucide/messages-square
---
# Prompt Contract
This page defines the canonical contract for prompts in the docs-first MCP architecture.
## Canonical Prompt Shape
Each prompt is one directory under `docs/prompts/`:
```mermaid
---
config:
treeView:
rowIndent: 20
lineThickness: 2
themeVariables:
treeView:
labelColor: '#FFFFFF'
lineColor: '#FFFFFF'
---
treeView-beta
"docs/"
"... (other docs)"
"prompts/"
"<prompt-id>/"
"PROMPT.md"
"references/"
"... (one or more markdown files, optional nested folders)"
```
Rules:
1. `PROMPT.md` is required for every prompt.
2. `references/` is the only place for prompt-specific supporting docs.
3. Nested folders inside `references/` are allowed so a prompt can reorganize internals without changing global architecture.
4. Prompt directories are independent ownership boundaries; no cross-prompt file writes.
## Metadata Location Constraint
1. Prompt metadata is embedded in YAML frontmatter in `PROMPT.md`.
2. No `metadata.yaml` sidecar exists in the end state.
3. Reference lookup metadata is documented and explicit: top-level `references/*.md` are auto-discovered from filenames, while `PROMPT.md` frontmatter declares overrides and nested mappings when needed.
## Prompt Id Contract
`prompt-id` is the public identifier and should satisfy all rules below:
1. Format: lowercase kebab-case only.
2. Character set: `a-z`, `0-9`, and `-`.
3. Must start with a letter.
4. No underscores, spaces, dots, or uppercase characters.
5. Directory name should equal `prompt-id` in each committed revision.
6. Frontmatter `id` should equal directory name in each committed revision.
7. Treat `prompt-id` as immutable after release; any rename is a breaking replacement and clients must move to the new id.
Valid examples:
1. `pytest-fill-scaffold`
2. `review-pr-comments`
3. `scaffold-fastapi-service`
Invalid examples:
1. `fill_pytest_scaffold`
2. `Prompt-Template`
3. `docs.prompt`
## Direct Documentation Inclusion
1. For direct API documentation, use mkdocstrings directives rather than pasting large code blocks.
2. Keep manually-authored code examples short and task-focused; large implementation excerpts are out of scope for this contract.
+75
View File
@@ -0,0 +1,75 @@
---
icon: lucide/brain-circuit
---
# Skill Contract
This page defines the canonical contract for skills in the docs-first MCP architecture.
## Canonical Skill Shape
Each skill is one directory under `docs/skills/`:
```mermaid
---
config:
treeView:
rowIndent: 20
lineThickness: 2
themeVariables:
treeView:
labelColor: '#FFFFFF'
lineColor: '#FFFFFF'
---
treeView-beta
"docs/"
"... (other docs)"
"skills/"
"<skill-id>/"
"SKILL.md"
"references/"
"... (one or more markdown files, optional nested folders)"
```
Rules:
1. `SKILL.md` is required for every skill.
2. `references/` is the only place for skill-specific supporting docs.
3. Nested folders inside `references/` are allowed so a skill can reorganize internals without changing global architecture.
4. Skill directories are independent ownership boundaries; no cross-skill file writes.
## Metadata Location Constraint
1. Skill metadata is embedded in YAML frontmatter in `SKILL.md`.
2. No `metadata.yaml` sidecar exists in the end state.
3. Reference lookup metadata is documented and explicit: top-level `references/*.md` are auto-discovered from filenames, while `SKILL.md` frontmatter declares overrides and nested mappings when needed.
## Skill Id Contract
`skill-id` is the public identifier and should satisfy all rules below:
1. Format: lowercase kebab-case only.
2. Character set: `a-z`, `0-9`, and `-`.
3. Must start with a letter.
4. No underscores, spaces, dots, or uppercase characters.
5. Directory name should equal `skill-id` in each committed revision.
6. Frontmatter `id` should equal directory name in each committed revision.
7. Treat `skill-id` as immutable after release; any rename is a breaking replacement and clients must move to the new id.
Valid examples:
1. `fastapi-uv-docker`
2. `zensical-docs`
3. `pytesting`
Invalid examples:
1. `fastapi_uv_docker`
2. `Zensical-Docs`
3. `docs.zensical`
## Direct Documentation Inclusion
1. For direct API documentation, use mkdocstrings directives rather than pasting large code blocks.
2. Keep manually-authored code examples short and task-focused; large implementation excerpts are out of scope for this contract.
+186
View File
@@ -0,0 +1,186 @@
---
icon: lucide/link
---
# URI Contract
This page defines the canonical resource URI contract, template parameter rules, and compatibility policy.
Conventions in this document follow [MCP resource semantics](https://modelcontextprotocol.io/docs/learn/server-concepts#resources), [URI generic syntax (RFC3986)](https://www.rfc-editor.org/rfc/rfc3986), and [URI templates (RFC6570)](https://www.rfc-editor.org/rfc/rfc6570).
## Canonical URI Surface
The public, preferred direct resource URIs are:
1. `resource://catalog/skills_index`
2. `resource://catalog/skills/{skill_id}`
3. `resource://skills/{skill_id}/document`
4. `resource://skills/{skill_id}/references/{ref_id}`
5. `resource://docs/{path*}`
6. `resource://catalog/prompts_index`
7. `resource://catalog/prompts/{prompt_id}`
8. `resource://prompts/{prompt_id}/document`
The public, preferred resource template URIs are:
1. `resource://catalog/skills_index{?q,tag,capability,cursor,limit}`
2. `resource://catalog/prompts_index{?q,tag,cursor,limit}`
Contract intent:
1. Catalog URIs are discovery surfaces.
2. Skill URIs are the primary per-skill guidance surfaces.
3. Catalog query templates are additive discovery helpers for filtering and pagination.
4. The docs wildcard URI is a direct authored-markdown access surface under `docs/`.
Best-practice alignment:
1. Resource identifiers are stable and noun-oriented.
2. Dynamic lookup variants are represented as RFC6570 templates.
3. Resources remain read-oriented and are described with explicit MIME types.
## URI Semantics
### `resource://catalog/skills_index`
1. Returns a compact list of skill records for discovery.
2. Contains one entry per `skill_id`.
3. Includes enough metadata for client-side selection, at minimum `id`, `name`, `description`, `tags`, and `capabilities`.
### `resource://catalog/skills/{skill_id}`
1. Returns one normalized record for `skill_id`.
2. Includes the canonical document URI and declared reference ids.
3. Returns not found when `skill_id` does not exist.
### `resource://skills/{skill_id}/document`
1. Returns the canonical `SKILL.md` authored content for that skill.
2. `skill_id` must satisfy the stable skill id rules from the content contract.
### `resource://skills/{skill_id}/references/{ref_id}`
1. Returns one reference document declared in the skill frontmatter references manifest.
2. `ref_id` is the stable public handle for that reference document.
### `resource://docs/{path*}`
1. Returns authored markdown at a normalized relative path under `docs/`.
2. Supports nested paths via [RFC6570 wildcard expansion](https://www.rfc-editor.org/rfc/rfc6570).
3. Typical examples include `index.md`, `usage.md`, `skills/<skill-id>/SKILL.md`, and `skills/<skill-id>/references/<file>.md`.
### `resource://catalog/prompts_index`
1. Returns a compact list of prompt records for discovery.
2. Contains one entry per `prompt_id`.
3. Includes `id`, `name`, `description`, `tags`, `version`, and canonical document URI.
### `resource://catalog/skills_index{?q,tag,capability,cursor,limit}`
1. Returns the same record family as `resource://catalog/skills_index` with optional filtering and pagination.
2. Query parameters are optional and composable.
3. Unknown query keys are ignored or rejected deterministically by server policy.
### `resource://catalog/prompts_index{?q,tag,cursor,limit}`
1. Returns the same record family as `resource://catalog/prompts_index` with optional filtering and pagination.
2. Query parameters are optional and composable.
3. Unknown query keys are ignored or rejected deterministically by server policy.
### `resource://catalog/prompts/{prompt_id}`
1. Returns one normalized record for `prompt_id`.
2. Includes prompt argument metadata when declared in frontmatter.
3. Returns not found when `prompt_id` does not exist.
### `resource://prompts/{prompt_id}/document`
1. Returns the canonical prompt markdown document.
2. `prompt_id` must satisfy lowercase kebab-case rules.
## Template Parameter And Validation Rules
### `skill_id`
1. Lowercase kebab-case.
2. Must satisfy the stable skill id rules from the content contract.
### `ref_id`
1. Lowercase kebab-case.
2. Must be declared in the skill's references manifest.
### `path*`
1. Relative POSIX path only, expressed as URI path segments under [RFC3986 path syntax](https://www.rfc-editor.org/rfc/rfc3986#section-3.3).
2. No leading slash.
3. No `..` traversal segments.
4. Resolves only inside `docs/`.
5. Markdown-only in the end state, meaning `.md` files.
6. Any reserved URI characters in path segments must be [percent-encoded](https://www.rfc-editor.org/rfc/rfc3986#section-2.1).
### `prompt_id`
1. Lowercase kebab-case.
2. Must be unique across prompt ids and must not collide with skill ids.
## URI Hygiene Rules
1. Use lowercase, human-readable path segments for stable discoverability.
2. Keep identifiers immutable once public whenever practical.
3. Keep template variables semantic (`skill_id`, `prompt_id`, `ref_id`, `path*`) and avoid overloading one variable for unrelated meanings.
4. Do not include secrets, tokens, or user-identifying data in URI paths or query strings.
5. Prefer additive query parameters for discovery over introducing parallel URI families, matching [MCP resource-template discovery patterns](https://modelcontextprotocol.io/docs/learn/server-concepts#resources).
6. Return clear not-found semantics for unknown ids and invalid template resolution.
## URI Versioning Policy
Default rule:
1. Keep URIs unversioned by default.
2. Allow URI and payload updates when they improve clarity or implementation simplicity.
Breaking-change rule:
1. Breaking changes use direct replacement of the canonical URI family.
2. No compatibility aliases or dual URI families are maintained.
FastMCP version metadata usage:
1. Resource `version` metadata may be used for implementation and version discovery.
2. URI readability and maintainability remain the primary contract.
## Reference Id Compatibility Policy
`ref_id` is the public identifier for a reference document, separate from file path.
Rules:
1. Prefer keeping `ref_id` stable when practical.
2. File paths may change without URI churn as long as the mapped `ref_id` still resolves.
3. If a reference is renamed, introduce a new `ref_id` and treat the old one as retired.
4. Avoid reusing retired `ref_id` values for unrelated content.
## Invariants
This contract guarantees:
1. One canonical URI pattern per core capability surface.
2. Fast, low-friction URI evolution through direct replacement of canonical URIs.
3. A single canonical catalog URI family with no alias maintenance overhead.
4. Reference mappings can evolve with minimal churn.
## Non-Goals
This contract does not define:
1. Implementation-specific transform wiring details, such as `VersionFilter`, mounts, or provider composition.
2. Migration script mechanics for auto-generating aliases.
3. Authorization policy design for URI-level access control.
## Sources
1. [MCP Server Concepts: Resources](https://modelcontextprotocol.io/docs/learn/server-concepts#resources)
2. [MCP Architecture Overview](https://modelcontextprotocol.io/docs/learn/architecture)
3. [MCP Specification Repository](https://github.com/modelcontextprotocol/spec)
4. [RFC6570 URI Template](https://www.rfc-editor.org/rfc/rfc6570)
+204
View File
@@ -0,0 +1,204 @@
---
icon: lucide/bot
---
# Copilot MCP Mechanics
## Purpose
This page explains how the GitHub Copilot extension in VS Code behaves as an MCP client when connected to `personal-mcp`, including why tools can appear while resource attachment appears unavailable.
## Core Model
Copilot interacts with MCP servers through separate capability lanes:
1. tools (invoked by the model during execution)
2. resources (attached as read-only context)
3. prompts (server-provided prompt templates)
These lanes are related but independently gated in the client.
Reliable paths are:
1. attach MCP resources explicitly through `Add Context > MCP Resources` or `MCP: Browse Resources`
2. let Copilot invoke MCP tools when the task and tool descriptions make that relevant
3. invoke MCP prompts explicitly with `/server.prompt` when your server exposes them
## What Actually Happens In VS Code
### MCP server side
Your server can advertise resources and serve them correctly. In this project that includes catalog resources and skill document resources.
### Copilot session side
The chat surface exposes tools, resources, and prompts through different UI paths. In practice, you can encounter sessions where tool use is available but MCP resource attachment is not exposed in `Add Context`.
That is why you can sometimes see MCP tools before you see `Add Context > MCP Resources`.
## Why The Picker Sometimes Shows Only Tools
`MCP Resources...` in Add Context requires at least:
1. at least one connected MCP server advertises resource capability
2. the current chat surface exposes MCP resource attachment
If the second condition is not met, resources can be available on the server while still being absent from the picker.
## Practical Workflow
Use this sequence to confirm behavior:
1. run `MCP: Browse Resources` and verify resources exist
2. use `MCP: List Servers` to verify the server is enabled and running
3. open Copilot Chat
4. check `Add Context` for `MCP Resources...`
5. if still missing, restart the server and reload VS Code window
## Recommended Usage Pattern
1. rely on canonical catalog resources for discovery (`skills_index`, then `skills/{skill_id}`)
2. fetch only selected skill documents for context
3. keep slash commands for deterministic fallback flows
When resource attachment is unavailable in the active session, use ResourcesAsTools first, then thin catalog discovery tools as parity fallback:
1. `list_resources`
2. `read_resource`
3. `search_patterns`
4. `get_pattern_by_id`
5. `get_skill_document_by_id`
Canonical naming policy:
1. Prefer the five canonical tool names above in prompts and instructions.
2. For compatibility with clients that emit `catalog_*` naming, the server also exposes:
- `catalog_search_patterns`
- `catalog_get_pattern_by_id`
- `catalog_get_skill_document_by_id`
3. Canonical and compatibility alias tools return equivalent payloads for the same input.
The first two are generated from the canonical resource surface and should be preferred in tool-only clients.
These should stay read-only, minimal, and schema-aligned with catalog resources.
For very large tool catalogs, server operators can optionally enable tool search mode (`regex` or `bm25`) while keeping `list_resources` and `read_resource` pinned as always-visible fallback tools.
## What To Type In Copilot Chat
Use prompts that tell Copilot which MCP feature path to take.
### If `MCP Resources...` is available
Use the resource attachment UI first, then ask Copilot to work from the attached material.
Example:
```text
I attached the catalog resources and the FastAPI async SQLAlchemy modernization skill document. Use that context to propose a migration plan for this repo.
```
If you want to keep the attachment sequence explicit, use:
```text
I attached personal-mcp catalog resources first. Use them to identify the best matching skill, then work only from the selected skill document.
```
### If only tools are available
Ask Copilot to explicitly use resource-backed tools first.
Example resource-backed prompt:
```text
Use personal-mcp tool fallback by first calling list_resources, then read_resource for resource://catalog/skills_index and the selected resource://skills/<skill-id>/document URI. Use only that loaded skill context in your answer.
```
If needed, use the thin catalog tools.
Example discovery prompt:
```text
Use the personal-mcp catalog tools to search for the most relevant skill for FastAPI async SQLAlchemy modernization. Then load the selected skill document and use it as context for your answer.
```
Example direct-load prompt:
```text
Call get_skill_document_by_id for async-fastapi-sqlmodel and use that document as the main context for this task.
```
Example bounded-selection prompt:
```text
Search personal-mcp skills for NiceGUI UI customization, select at most 2 strong matches, load the best skill document, and answer using only that material plus the workspace code.
```
## Repo Instructions Example
Repo instructions are the best place to teach Copilot when MCP content is relevant and which path to prefer.
If you add a repo-level `copilot-instructions.md`, keep the rule simple: prefer catalog-first discovery, keep loaded skill context small, and fall back to tools when resource attachment is unavailable.
Instructions can strongly steer behavior, but they do not guarantee that VS Code will auto-attach MCP resources for a request. For reliable resource use, either attach resources explicitly or prompt Copilot to use the fallback tools.
Example:
```md
# MCP Usage
When a task may benefit from personal-mcp skills, use this sequence:
1. Start with personal-mcp catalog discovery when the task appears to match documented implementation patterns.
2. Prefer MCP resources when the chat surface exposes resource attachment.
3. If MCP resource attachment is unavailable, use `list_resources`/`read_resource` first, then thin catalog tools if needed.
4. Load only the most relevant skill document or at most 2 skill documents.
5. Treat skill documents as guidance, then reconcile them with the actual repository code before making changes.
Preferred discovery order:
1. `resource://catalog/skills_index`
2. `resource://catalog/skills/{skill_id}`
3. `resource://skills/<skill-id>/document`
4. `resource://skills/<skill-id>/references/<ref-id>` when needed
Tool fallback order:
1. `list_resources`
2. `read_resource`
3. `search_patterns`
4. `get_pattern_by_id`
5. `get_skill_document_by_id`
If confidence is low after catalog discovery, ask one clarifying question before loading more skill documents.
```
That instruction style does two useful things:
1. it tells Copilot to prefer the MCP server when relevant without forcing it on every prompt
2. it keeps context size bounded so skill loading does not become noisy or expensive
If you want stronger behavior, add one more line that names the MCP server directly:
```md
Use the `personal-mcp` server for skill discovery whenever the task involves documented implementation patterns available from the catalog.
```
## Known Gotcha
A successful `resources/list` response from the server does not guarantee the resource picker appears in every Copilot session type. UI availability is session-capability-dependent.
## Further Reading
### VS Code docs
1. [Add and manage MCP servers](https://code.visualstudio.com/docs/agent-customization/mcp-servers)
2. [MCP configuration reference](https://code.visualstudio.com/docs/agents/reference/mcp-configuration)
3. [Manage context for AI](https://code.visualstudio.com/docs/chat/copilot-chat-context)
4. [AI features cheat sheet](https://code.visualstudio.com/docs/agents/reference/ai-features-cheat-sheet)
### Project docs
1. [Resource-First Pattern Module Architecture](./architecture.md)
2. [Static Docs Hosting Pattern](./mcp_layout.md)
3. [Skill Usage Mechanics](./usage.md)
+26 -151
View File
@@ -2,172 +2,47 @@
icon: lucide/rocket icon: lucide/rocket
--- ---
# Get started # Personal MCP
For full documentation visit [zensical.org](https://zensical.org/docs/). This project is a document library of software patterns, best practices, and structured references to external documentation. The same markdown files are published through two equivalent surfaces, so human-readable docs and MCP resources stay aligned.
## Commands ## MCP Server
* [`zensical new`][new] - Create a new project An [MCP server](https://modelcontextprotocol.io/docs/getting-started/intro) at `/mcp` provides context for AI systems. The markdown files are exposed as [resources](https://modelcontextprotocol.io/docs/learn/server-concepts#resources) and are structured to be easily consumed by [MCP clients](https://modelcontextprotocol.io/docs/learn/client-concepts), such as VS Code.
* [`zensical serve`][serve] - Start local web server
* [`zensical build`][build] - Build your site
[new]: https://zensical.org/docs/usage/new/ ## Docs
[serve]: https://zensical.org/docs/usage/preview/
[build]: https://zensical.org/docs/usage/build/
## Examples A website at `/docs` for humans to read and review.
### Admonitions ## Quick start
> Go to [documentation](https://zensical.org/docs/authoring/admonitions/) Install dependencies first:
!!! note ```bash
uv sync
This is a **note** admonition. Use it to provide helpful information.
!!! warning
This is a **warning** admonition. Be careful!
### Details
> Go to [documentation](https://zensical.org/docs/authoring/admonitions/#collapsible-blocks)
??? info "Click to expand for more info"
This content is hidden until you click to expand it.
Great for FAQs or long explanations.
## Code Blocks
> Go to [documentation](https://zensical.org/docs/authoring/code-blocks/)
``` python hl_lines="2" title="Code blocks"
def greet(name):
print(f"Hello, {name}!") # (1)!
greet("Python")
``` ```
1. > Go to [documentation](https://zensical.org/docs/authoring/code-blocks/#code-annotations) Run the app locally with the static docs rebuilt first, using [Uvicorn factory mode](https://www.uvicorn.org/settings/#application):
Code annotations allow to attach notes to lines of code. ```bash
uv run zensical build && uv run uvicorn personal_mcp.main:create_app --factory --host 127.0.0.1 --port 8765
Code can also be highlighted inline: `#!python print("Hello, Python!")`.
## Content tabs
> Go to [documentation](https://zensical.org/docs/authoring/content-tabs/)
=== "Python"
``` python
print("Hello from Python!")
``` ```
=== "Rust" Build and run the Docker image with the same exposed port:
``` rs ```bash
println!("Hello from Rust!"); docker build -t personal-mcp . && docker run --rm -p 8765:8765 personal-mcp
``` ```
## Diagrams When the server is running, the health check is available at `/healthz` and the generated docs are available at `/docs/`.
> Go to [documentation](https://zensical.org/docs/authoring/diagrams/) ## Architecture
``` mermaid - [Resource-First Pattern Module Architecture](./architecture.md)
graph LR - [Contracts](./contracts/index.md)
A[Start] --> B{Error?}; - [Content Contract](./contracts/index.md#content-contract)
B -->|Yes| C[Hmm...]; - [Frontmatter Contract](./contracts/frontmatter.md)
C --> D[Debug]; - [URI Contract](./contracts/uris.md)
D --> B; - [Static Docs Hosting Pattern](./mcp_layout.md)
B ---->|No| E[Yay!]; - [Skill Usage Mechanics](./usage.md)
``` - [Copilot MCP Mechanics](./copilot.md)
## Footnotes
> Go to [documentation](https://zensical.org/docs/authoring/footnotes/)
Here's a sentence with a footnote.[^1]
Hover it, to see a tooltip.
[^1]: This is the footnote.
## Formatting
> Go to [documentation](https://zensical.org/docs/authoring/formatting/)
- ==This was marked (highlight)==
- ^^This was inserted (underline)^^
- ~~This was deleted (strikethrough)~~
- H~2~O
- A^T^A
- ++ctrl+alt+del++
## Icons, Emojis
> Go to [documentation](https://zensical.org/docs/authoring/icons-emojis/)
* :sparkles: `:sparkles:`
* :rocket: `:rocket:`
* :tada: `:tada:`
* :memo: `:memo:`
* :eyes: `:eyes:`
## Maths
> Go to [documentation](https://zensical.org/docs/authoring/math/)
$$
\cos x=\sum_{k=0}^{\infty}\frac{(-1)^k}{(2k)!}x^{2k}
$$
!!! warning "Needs configuration"
Note that MathJax is included via a `script` tag on this page and is not
configured in the generated default configuration to avoid including it
in a pages that do not need it. See the documentation for details on how
to configure it on all your pages if they are more Maths-heavy than these
simple starter pages.
<script id="MathJax-script" src="https://unpkg.com/mathjax@3/es5/tex-mml-chtml.js"></script>
<script>
window.MathJax = {
tex: {
inlineMath: [["\\(", "\\)"]],
displayMath: [["\\[", "\\]"]],
processEscapes: true,
processEnvironments: true
},
options: {
ignoreHtmlClass: ".*|",
processHtmlClass: "arithmatex"
}
};
document$.subscribe(() => {
MathJax.startup.output.clearCache()
MathJax.typesetClear()
MathJax.texReset()
MathJax.typesetPromise()
})
</script>
## Task Lists
> Go to [documentation](https://zensical.org/docs/authoring/lists/#using-task-lists)
* [x] Install Zensical
* [x] Configure `zensical.toml`
* [x] Write amazing documentation
* [ ] Deploy anywhere
## Tooltips
> Go to [documentation](https://zensical.org/docs/authoring/tooltips/)
[Hover me][example]
[example]: https://example.com "I'm a tooltip!"
+25
View File
@@ -0,0 +1,25 @@
window.MathJax = {
tex: {
inlineMath: [['\\(', '\\)']],
displayMath: [['\\[', '\\]']],
processEscapes: true,
processEnvironments: true
},
options: {
ignoreHtmlClass: '.*|',
processHtmlClass: 'arithmatex'
}
};
document$.subscribe(() => {
MathJax.startup.output.clearCache();
MathJax.typesetClear();
MathJax.texReset();
MathJax.typesetPromise();
});
component$.subscribe(({ ref }) => {
if (ref.classList.contains('md-annotation')) {
MathJax.typesetPromise([ref]);
}
});
+98
View File
@@ -0,0 +1,98 @@
let mermaidPromise;
async function getMermaid() {
if (!mermaidPromise) {
mermaidPromise = import("https://cdn.jsdelivr.net/npm/mermaid@11/dist/mermaid.esm.min.mjs").then(
(module) => {
const mermaid = module.default ?? module;
mermaid.initialize({ startOnLoad: false });
return mermaid;
},
);
}
return mermaidPromise;
}
function readDiagramSource(block) {
const code = block.querySelector("code");
return (code?.textContent ?? block.textContent ?? "").trim();
}
async function renderBlock(block) {
if (block.dataset.mermaidOverrideState) {
return;
}
const source = readDiagramSource(block);
if (!source) {
block.dataset.mermaidOverrideState = "empty";
return;
}
block.dataset.mermaidOverrideState = "pending";
try {
const mermaid = await getMermaid();
const replacement = document.createElement("div");
replacement.className = "mermaid";
replacement.textContent = source;
replacement.dataset.mermaidOverrideState = "rendered";
block.replaceWith(replacement);
await mermaid.run({ nodes: [replacement] });
} catch (error) {
block.dataset.mermaidOverrideState = "failed";
console.warn("Mermaid override failed", error);
}
}
function findBlocks(root = document) {
const blocks = [];
if (root instanceof Element && root.matches("pre.mermaid")) {
blocks.push(root);
}
if (root instanceof Document || root instanceof Element) {
blocks.push(...root.querySelectorAll("pre.mermaid"));
}
return blocks;
}
function renderAll(root = document) {
for (const block of findBlocks(root)) {
void renderBlock(block);
}
}
function installObserver() {
if (!(document.body instanceof HTMLBodyElement)) {
return;
}
const observer = new MutationObserver((mutations) => {
for (const mutation of mutations) {
for (const node of mutation.addedNodes) {
if (node instanceof Element) {
renderAll(node);
}
}
}
});
observer.observe(document.body, { childList: true, subtree: true });
}
if (document.readyState === "loading") {
document.addEventListener("DOMContentLoaded", () => {
renderAll();
installObserver();
});
} else {
renderAll();
installObserver();
}
window.addEventListener("pageshow", () => renderAll());
window.addEventListener("popstate", () => renderAll());
+201
View File
@@ -0,0 +1,201 @@
---
icon: lucide/server
---
# Static Docs Hosting Pattern
## Purpose
This document describes the completed layout and runtime pattern used to host a pre-built static documentation site from the same FastAPI app process that runs the FastMCP server.
This design intentionally avoids runtime docs rendering and avoids a separate docs hosting service.
It also treats Markdown as the single source of truth for both MCP resources and published docs.
## Completed-State Layout
```mermaid
---
config:
treeView:
rowIndent: 40
lineThickness: 2
themeVariables:
treeView:
labelColor: '#FFFFFF'
lineColor: '#FFFFFF'
---
treeView-beta
"project-root"
"pyproject.toml"
"uv.lock"
"zensical.toml"
"docs"
"index.md"
"<project-docs>.md"
"contracts"
"index.md"
"<contract-pages>.md"
"mcp_layout.md"
"prompts"
"<prompt-id>"
"PROMPT.md"
"references"
"skills"
"<skill-id>"
"SKILL.md"
"references"
"<reference>.md"
"site"
"static build output"
"src"
"personal_mcp"
"__init__.py"
"main.py"
"mcp.py"
"catalog"
"<catalog-modules>.py"
"registry"
"<registry-modules>.py"
"web"
"<web-modules>.py"
"skills"
"<skills-modules>.py"
```
Notes:
1. docs contains both project-authored pages and the canonical skill Markdown tree.
2. site contains static build output only.
3. docs/skills contains canonical skill Markdown and reference Markdown.
4. docs/prompts contains canonical prompt Markdown used for prompt catalog and document surfaces.
5. MCP resources and docs site read from the same Markdown sources.
## Runtime Composition
The runtime process serves two surfaces:
1. MCP protocol surface from FastMCP
2. Static docs surface from FastAPI static mount
```mermaid
flowchart TD
A[Docs Registry Loader] --> B[Validated In-Memory Registry]
B --> C[FastMCP Resource Registration]
C --> D[MCP Transport]
C --> E[FastAPI Application]
E --> F[Static Mount /docs]
F --> G[Zensical site output directory]
```
Runtime guarantees:
1. Docs registry load and validation happen before resource exposure.
2. Duplicate resource and template registration fails startup (`on_duplicate="error"`).
3. Resource registration is metadata-driven from SKILL frontmatter and reference manifests.
4. Legacy per-skill Python servers and `metadata.yaml` sidecars are not part of the runtime.
## Build and Publish Flow
The docs flow is pre-build only.
1. Read authored docs pages and skill markdown sources.
2. Build static site with Zensical into site.
3. Start app and serve site directory as static files.
No runtime markdown conversion is required.
## Content Merge Pattern
The published docs site always contains both:
1. Project-authored docs pages
2. Skill Markdown content from docs/skills/*/SKILL.md and references
This ensures the public docs reflect architectural guidance and the exact Markdown served by MCP.
## Markdown-to-Resource Mapping
MCP resources map directly to canonical Markdown documents.
Example mapping model:
1. docs/skills/<skill-id>/SKILL.md -> resource://skills/<skill_id>/document
2. docs/skills/<skill-id>/references/<file>.md -> resource://skills/<skill_id>/references/<ref_id> (via frontmatter references manifest)
3. docs/<path>.md -> resource://docs/{path*}
Catalog discovery resources are:
1. resource://catalog/skills_index
2. resource://catalog/skills_index{?q,tag,capability,cursor,limit}
3. resource://catalog/skills/{skill_id}
4. resource://catalog/prompts_index
5. resource://catalog/prompts_index{?q,tag,cursor,limit}
6. resource://catalog/prompts/{prompt_id}
Registry-backed registration details:
1. `resource://skills/{skill_id}/document` resolves to each skill's SKILL.md.
2. `resource://skills/{skill_id}/references/{ref_id}` resolves through frontmatter reference manifests.
3. `resource://docs/{path*}` resolves normalized markdown paths under `docs/`.
4. Resource metadata includes explicit mime type and read-only/idempotent annotations.
When clients cannot attach MCP resources directly, thin catalog tools may retrieve the same underlying skill documents indirectly. This does not create a second content source.
## URI Compatibility Policy
1. Canonical URIs are the only supported URIs in this runtime.
2. No backward-compatibility aliases or dual registration paths are maintained.
3. Contract changes should update clients to canonical URIs directly.
## Why This Pattern
### Operational Simplicity
One application process serves both protocol and static docs surfaces.
### Deterministic Docs
Published docs are immutable static assets for a given build.
### Documentation Fidelity
The docs site and MCP resources resolve from the same Markdown sources.
### Maintainer Experience
Authors continue to work in markdown while resource contracts remain machine-consumable.
## FastAPI Static Mount Expectations
The FastAPI app is expected to:
1. Mount static directory containing Zensical output.
2. Serve index and asset files from that directory.
3. Keep docs route stable across releases.
Recommended route conventions:
1. /docs for static site root
2. /docs/* for static assets and page routes
## Update Lifecycle
For each documentation update:
1. Edit authored docs and skill markdown content.
2. Rebuild static site.
3. Restart runtime if needed.
This keeps docs publication explicit and predictable.
## Example Source Material
Existing reference docs remain valid content inputs in this pattern:
1. docs/skills/pytesting/references/pytest-docs.md
2. docs/skills/python-logging/references/python-logging-docs.md
3. docs/skills/python-logging/references/json-file-logging.md
4. docs/skills/fastapi-uv-docker/references/fastapi-best-practices.md
These are source documents, not deployment artifacts.
+88
View File
@@ -0,0 +1,88 @@
---
name: authoring
description: Provide a practical checklist and baseline template for authoring docs-first MCP modules and repository-specific Copilot instruction shims.
x-personal-mcp:
id: authoring
version: 1.0.0
tags:
- authoring
- mcp
- fastmcp
- copilot
- prompts
- scaffolding
capabilities:
- resource://prompts/authoring/document
arguments:
artifact_type:
title: Artifact type
description: "Enum (case-sensitive): skill | prompt | shim."
required: true
artifact_id:
title: Artifact id
description: Lowercase kebab-case id for the module or shim.
required: true
goal:
title: Goal
description: One-sentence capability statement describing what to create and when to use it.
required: true
scope_glob:
title: Scope glob
description: Optional applyTo glob for shim outputs.
required: false
---
# Authoring Bootstrap
Use this prompt to author or update docs-first MCP modules in this repository, including repository-specific Copilot thin shims.
## Inputs
1. artifact_type: one of skill, prompt, shim
2. artifact_id: lowercase kebab-case id
3. goal: one-sentence capability statement
4. optional scope_glob for shim outputs
## Required References
Load only what matches the requested artifact:
1. Authoring workflow and validation policy: [Authoring Guide](../../authoring.md)
2. Prompt metadata and structure: [Prompt Contract](../../contracts/prompt.md)
3. Skill metadata and structure (only for skill outputs): [Skill Contract](../../contracts/skill_contract.md)
4. Thin shim mechanics and path binding: [Skill Usage Mechanics](../../usage.md)
5. Copilot resource attachment and fallback behavior: [Copilot MCP Mechanics](../../copilot.md)
## Workflow
1. Validate required inputs and ask one clarifying question if any required input is missing.
2. Keep ids and slugs aligned with folder names and frontmatter ids.
3. Enforce artifact_type enum values exactly: skill, prompt, shim.
4. If artifact_type is outside the enum, ask one correction question and stop before generating output.
5. Apply YAML safety rules for frontmatter values:
- quote values containing `:`
- prefer quotes for punctuation-heavy scalars
- use block scalars for multiline descriptions
6. Run immediate validation after frontmatter edits:
- `uv run zensical build`
- `uv run pytest -q`
7. Produce only the requested artifact type.
8. Keep guidance deterministic and minimal, with explicit references to source docs.
9. If artifact_type is shim:
- bind one applyTo scope to one primary skill resource URI
- prefer MCP resource attachment first
- if resource attachment is unavailable, use fallback tool order:
1. list_resources
2. read_resource
3. search_patterns
4. get_pattern_by_id
5. get_skill_document_by_id
10. Return created or updated file paths and any validation commands that should be run.
## Output Contract
Return:
1. Files created or updated.
2. Which references were used.
3. Validation commands and outcomes (or commands to run if execution is not requested).
@@ -0,0 +1,111 @@
---
name: greenfield-architecture
description: Research established patterns and design a high-level architecture for a new app or library with explicit tradeoffs and test strategy.
x-personal-mcp:
id: greenfield-architecture
version: 1.0.0
tags:
- architecture
- planning
- greenfield
- design
- testing
- prompts
capabilities:
- resource://prompts/greenfield-architecture/document
arguments:
scope_type:
title: Scope type
description: "Scope type: app or library."
required: true
intent_document:
title: Intent document
description: Optional full document describing goals, context, and desired outcomes.
required: false
problem_domain:
title: Problem domain
description: Domain and business goal for the new app or library when no full intent document is provided.
required: false
constraints:
title: Constraints
description: Runtime, deployment, and non-functional constraints.
required: false
---
# Greenfield Architecture Planner
Use this prompt to design a new software app or library architecture in generic terms.
## Inputs
1. intent_document: optional full document that explains goals, context, constraints, and desired outcomes
2. problem_domain: concise domain and one-sentence business goal when no full intent document is provided
3. scope_type: app or library
4. optional constraints: runtime, deployment, scale, non-functional priorities
If both intent_document and problem_domain are provided, treat intent_document as the primary source and use problem_domain as a summary cross-check.
## Workflow
1. Validate required inputs.
- scope_type is required
- at least one of intent_document or problem_domain must be provided
- ask one concise clarification question if inputs are incomplete or contradictory
2. Start with research before proposing architecture:
- identify at least three established patterns or methodologies used for similar systems
- summarize what each pattern optimizes for
- compare strengths, risks, and implementation complexity
3. Ask which aspects of those patterns matter most for the user context.
4. Identify major libraries or frameworks commonly used for this problem space and explain tradeoffs for each:
- strengths and weaknesses
- ecosystem maturity
- performance profile
- operational complexity
- learning curve
5. Recommend one primary stack and one fallback stack, with rationale tied to stated priorities.
6. Produce the architecture deliverables:
- high-level concepts, features, and requirements
- intended use cases and key workflows
- high-level package/module structure
- conceptual boundaries for each module (what belongs there and what does not)
- dependency and data-flow direction between modules
7. Plan incremental delivery with explicit growth paths:
- define the initial prototype slice with the smallest valuable feature set
- identify which features are intentionally deferred from the prototype
- describe extension paths that add complexity in controlled stages
- ensure each stage preserves clean module boundaries and low migration risk
8. Design a test strategy aligned to the proposed structure and staged delivery plan:
- unit, integration, contract, and end-to-end layers
- what each layer should cover in prototype stage vs extension stages
- fixture and environment setup for fast, deterministic tests
- boundary seams for mocks/fakes and minimization of nondeterministic external I/O
- CI execution approach for fast feedback and confidence
9. Call out key risks, assumptions, and open questions.
## Output Contract
Return these sections in order:
1. Research Summary
2. Pattern Comparison
3. Library and Framework Tradeoffs
4. Recommended Stack
5. Architecture Overview
6. Concepts, Features, and Requirements
7. Intended Use Cases
8. Package and Module Layout
9. Conceptual Boundary Map
10. Initial Prototype Scope
11. Extension Roadmap
12. Test Strategy
13. Risks and Open Questions
14. Next Implementation Steps
## Quality Rules
1. Keep language generic and project-agnostic.
2. Prefer established patterns over novelty unless there is a strong reason to diverge.
3. Tie each recommendation to an explicit requirement or tradeoff.
4. Make assumptions explicit and concise.
5. Ask one focused clarifying question when confidence is low instead of over-speculating.
6. Prefer architecture decisions that support starting simple and growing complexity without major rewrites.
@@ -0,0 +1,111 @@
---
name: mcp-consumer-repo-shim
description: Create one repository-specific thin shim instruction file that binds a file scope to a user-selected Personal MCP skill resource and enforces resource-first Copilot retrieval behavior.
x-personal-mcp:
id: mcp-consumer-repo-shim
version: 1.0.0
tags:
- copilot
- mcp
- instructions
- shims
- prompts
capabilities:
- resource://prompts/mcp-consumer-repo-shim/document
arguments:
apply_to_glob:
description: File glob scope for the shim applyTo field, such as tests/** or **/*.md.
required: true
primary_skill_resource:
description: Primary skill resource URI, usually resource://skills/<skill-id>/document.
required: true
shim_title:
description: Human-readable name for the instruction shim frontmatter.
required: false
companion_docs_page:
description: Optional relative docs link for human-facing companion guidance.
required: false
---
# MCP Consumer Repository Shim
Use this prompt to generate exactly one repository-scoped Copilot instruction shim for an MCP consumer repository.
## Inputs
- Required:
- apply_to_glob
- primary_skill_resource
- Optional:
- shim_title
- companion_docs_page
## Required References
Load only sections relevant to the requested shim:
1. Thin shim pattern and scope guidance: [Skill Usage Mechanics](../../usage.md)
2. VS Code Copilot MCP behavior and fallback mechanics: [Copilot MCP Mechanics](../../copilot.md)
3. Authoring workflow and validation checklist: [Authoring Guide](../../authoring.md)
4. Instruction metadata expectations and examples: [Copilot customization skill](../../skills/copilot-customization/SKILL.md)
## Workflow
1. Validate that apply_to_glob and primary_skill_resource are present.
2. If either value is missing or ambiguous, ask exactly one clarifying question before generating output.
3. Generate one .instructions.md file content block only.
4. Keep the shim concise and deterministic:
- include YAML frontmatter with name, description, and applyTo
- include a primary rule that uses the selected primary_skill_resource first
- include a bounded execution pattern (load primary doc, apply only relevant sections, keep edits minimal)
5. Include VS Code/Copilot integration mechanics in the shim body:
- prefer MCP resource attachment when available
- if attachment is unavailable, use tool fallback order:
1. list_resources
2. read_resource
3. search_patterns
4. get_pattern_by_id
5. get_skill_document_by_id
- ask one clarifying question when confidence is low
6. If companion_docs_page is provided, include it as a companion docs link line.
7. Do not generate additional files, code changes, or batch shim packs.
## Output Format
Return exactly:
1. Suggested file path line under .github/instructions/.
2. One fenced markdown block containing the full .instructions.md content.
3. A brief note (max 3 lines) describing what the shim routes and why.
## Output Template
````md
Path: .github/instructions/<slug>.instructions.md
```md
---
name: <shim-title>
description: Route <scope> edits to the Personal MCP <skill-id> resource.
applyTo: '<apply_to_glob>'
---
When editing files matching <apply_to_glob>, use <primary_skill_resource> as the primary guidance source.
Execution pattern:
1. Load the primary skill document first.
2. Apply only sections relevant to the file being edited.
3. Keep edits minimal and aligned with repository conventions.
4. Prefer MCP resource attachment when available in the current chat surface.
5. If MCP resource attachment is unavailable, use tool fallback in this order:
1. list_resources
2. read_resource
3. search_patterns
4. get_pattern_by_id
5. get_skill_document_by_id
6. If confidence is low, ask one clarifying question before editing.
Companion docs page: <optional-relative-doc-link>
```
````
-320
View File
@@ -1,320 +0,0 @@
# NiceGUI
Build a production-ready, multi-page NiceGUI application that follows modern FastAPI project layout and architecture practices.
## Goal
Create a clean, maintainable app structure where:
- FastAPI is the underlying ASGI app.
- NiceGUI is mounted/initialized in a way that aligns with FastAPI best practices.
- Each page is implemented as a separate module in `pages/`.
- The project is easy to test, extend, and deploy.
## Requirements
- Prefer Pydantic settings with a `.env` example.
- Consider adding structured logging configuration.
- Use a `src/`-style layout with clear package boundaries.
- Use an app factory (`create_app`) and avoid global side effects at import time.
- Register NiceGUI pages from modules in `pages/`.
- Keep UI logic in page modules and shared UI helpers/components in a separate package.
- Include at least these pages:
- Home (`/`)
- Dashboard (`/dashboard`)
- About (`/about`)
- Include a health endpoint (`/healthz`) on the FastAPI side.
- Provide minimal test examples for FastAPI routes and page registration.
- Use FastAPI lifespan for startup/shutdown resource management.
## Recommendations
- Consider a shared navigation/header component and left drawer used by all pages.
- Consider including dev tooling basics (formatting/lint/test commands).
- Prefer clear separation of concerns:
- `db/` for engine/session/base/models/repositories (if needed)
- `api/` for HTTP endpoints
- `ui/` for NiceGUI pages/components
- `services/` for business logic
- If you need database access, a solid pattern is to use FastAPI + SQLAlchemy best practices:
- Use one engine per application process (do not create per request).
- Prefer a request-scoped session dependency using `yield`.
- Keep transaction boundaries explicit (commit/rollback) in service/repository flows.
- Avoid sharing a session across concurrent tasks.
- Consider production-minded pool settings (`pool_pre_ping=True` and sensible recycle/timeout values).
- Prefer Alembic for schema migrations.
## Suggested Project Structure
Base structure (no database required):
```text
.
├─ pyproject.toml
├─ .env.example
├─ README.md
├─ src/
│ └─ app/
│ ├─ __init__.py
│ ├─ main.py
│ ├─ config.py
│ ├─ logging.py
│ ├─ api/
│ │ ├─ __init__.py
│ │ └─ health.py
│ ├─ services/
│ │ ├─ __init__.py
│ │ └─ example_service.py
│ ├─ ui/
│ │ ├─ __init__.py
│ │ ├─ components/
│ │ │ ├─ __init__.py
│ │ │ └─ nav.py
│ │ └─ pages/
│ │ ├─ __init__.py
│ │ ├─ home.py
│ │ ├─ dashboard.py
│ │ └─ about.py
│ └─ bootstrap.py
└─ tests/
├─ test_health.py
└─ test_pages_registration.py
```
### Conceptual boundaries for the base structure
- `main.py`: application entrypoint only; wires `create_app()` and process startup concerns.
- `bootstrap.py`: app composition layer; owns app factory, router registration, page registration, and lifespan wiring.
- `config.py`: configuration boundary; centralizes settings parsing and typed config objects.
- `logging.py`: observability boundary; centralizes logging format/levels/handlers so modules do not configure logging ad hoc.
- `api/`: transport boundary for HTTP endpoints; validate/shape request-response objects and delegate business work to `services/`.
- `services/`: use-case/business boundary; orchestrates domain behavior and dependencies, independent of UI rendering.
- `ui/pages/`: route-level UI boundary; one module per page, focused on layout/event handling for that page.
- `ui/components/`: reusable presentation boundary; shared UI widgets/composition utilities with no business side effects.
- `tests/`: behavior boundary; verify public behavior (health endpoint, page registration, service outcomes), not private implementation details.
### Dependency direction to keep clear
- Prefer this dependency flow:
- `main/bootstrap` -> `config/logging` + `api` + `ui/pages` + `services`
- `api` -> `services`
- `ui/pages` -> `ui/components` + `services`
- `services` -> pure helpers/clients (and later `db/` if enabled)
- Avoid reverse imports (for example, `services` importing from `ui/pages` or `api`).
- Keep page modules and API handlers as thin adapters; keep business decisions in `services/`.
## Extending This Pattern with a Database (Optional)
If database access is needed, extend with:
```text
.
├─ alembic.ini
├─ alembic/
│ ├─ env.py
│ └─ versions/
├─ src/
│ └─ app/
│ └─ db/
│ ├─ __init__.py
│ ├─ base.py
│ ├─ session.py
│ ├─ models/
│ │ ├─ __init__.py
│ │ └─ example.py
│ └─ repositories/
│ ├─ __init__.py
│ └─ example_repo.py
└─ tests/
└─ test_db_session_dependency.py
```
### Recommended database layering
- Keep `db/` focused on persistence primitives:
- `session.py`: engine + session factory + dependency helpers
- `models/`: ORM table mappings only
- `repositories/`: query/persistence operations only
- Keep transaction orchestration in `services/`, not in UI pages.
- Keep `ui/pages/` free of SQLAlchemy details; pages call services.
- Keep API handlers thin and delegate data work to services/repositories.
### Session and transaction pattern to request
- Use one engine and one sessionmaker per process, initialized at app startup.
- Use a `get_db_session()` dependency with `yield` for request-scoped sessions.
- In write flows, always use context managers for commit/rollback safety.
- In read-only flows, avoid unnecessary transactions and keep queries narrowly scoped.
- Do not share a session across concurrent tasks; open a new session per task/unit of work.
### Migration and schema workflow
- Treat Alembic migrations as the source of truth for schema evolution.
- Prefer:
- `alembic revision --autogenerate -m "..."`
- review migration script
- `alembic upgrade head`
- Avoid relying on `metadata.create_all()` for production schema management.
- Include a lightweight startup check that logs current migration state (without mutating schema).
### Configuration and runtime concerns
- Read `DATABASE_URL` from settings (`pydantic-settings`) and keep secrets out of source control.
- Configure pool options appropriate for environment (for example: `pool_pre_ping`, timeout/recycle tuning).
- Keep SQL echo/debug logging disabled in production by default.
- Consider a readiness probe (`/readyz`) that verifies DB connectivity when your deployment needs it.
### Testing guidance for DB-enabled projects
- Unit test repositories against a disposable test database.
- Integration test `get_db_session()` lifecycle and rollback behavior on failures.
- Add migration tests that validate model changes are represented in Alembic revisions.
- Add API/service tests for critical read/write paths (including uniqueness and constraint errors).
- Keep DB tests isolated and deterministic (fresh schema + transactional cleanup per test module/session).
## Extending This Pattern with LangGraph (Optional)
If your app needs multi-step AI workflows, tool-calling loops, or human-in-the-loop approvals, a clean extension is to add a dedicated LangGraph domain layer.
### Recommended architecture shape
- Keep LangGraph logic out of `ui/pages/` and out of HTTP route handlers.
- Add an `ai/` package (or similar) and keep graph definition, state schema, tools, and orchestration there.
- Call the graph from `services/` so your UI and API both use the same orchestration entry points.
- Continue using FastAPI lifespan for shared resources (models, clients, stores/checkpointers).
### Suggested project extension
```text
.
└─ src/
└─ app/
├─ ai/
│ ├─ __init__.py
│ ├─ state.py # TypedDict / Pydantic state schemas + reducers
│ ├─ nodes/
│ │ ├─ __init__.py
│ │ ├─ llm.py
│ │ ├─ tools.py
│ │ └─ review.py # optional interrupt/human-review nodes
│ ├─ graphs/
│ │ ├─ __init__.py
│ │ └─ assistant_graph.py
│ ├─ runtime.py # compile() + checkpointer/store wiring
│ └─ contracts.py # input/output DTOs between services and graph
├─ services/
│ └─ ai_service.py # invoke/stream/resume wrappers
└─ api/
└─ ai.py # optional HTTP endpoints for run/stream/resume
```
### Idiomatic LangGraph practices to request
- Define explicit graph state schema (`TypedDict` or Pydantic) and use reducers for append/merge behavior where needed.
- Compile graphs with persistence primitives appropriate to environment:
- dev/testing: in-memory checkpointer/store
- production: durable checkpointer/store
- Always pass a stable `thread_id` in `configurable` for resumable sessions and conversation continuity.
- For interactive UX, prefer event streaming APIs and project-specific streaming adapters in service code.
- Use `interrupt()` for human approvals/reviews; resume with `Command(resume=...)` using the same `thread_id`.
- Keep interrupt payloads JSON-serializable.
- Place non-idempotent side effects after interrupts (or isolate them in separate nodes), because interrupted nodes re-run from the start.
- If using tools, prefer LangGraphs `ToolNode` (or an equivalent centralized tool-execution node) for consistent execution/error handling.
- Keep graph nodes deterministic and narrow in scope (single responsibility per node).
- Add observability with LangSmith tracing for graph runs, transitions, and latency debugging.
### Integration guidance for NiceGUI + FastAPI
- UI pages should call `ai_service` methods, not graph internals.
- If the UX needs live token/progress updates, expose streaming from service -> UI in a transport-appropriate way.
- For approval workflows, surface interrupt payloads in UI, collect user response, then resume via service with `Command(resume=...)`.
- Keep graph invocation boundaries typed (request/response contracts) so changes in graph internals do not leak into page modules.
### Testing recommendations for LangGraph additions
- Unit test node functions as pure transformations where possible.
- Integration test graph routes (happy path, tool path, interrupt/resume path).
- Add tests ensuring `thread_id` reuse resumes correctly and new IDs start fresh sessions.
- Add regression tests for state-schema evolution and serialization compatibility.
## Implementation Notes
- Prefer an explicit page registration function pattern, for example:
- `register_pages()` in `ui/pages/__init__.py`
- Each page module exports `register_page()`
- Keep imports one-way to avoid circular dependencies.
- Keep page modules cohesive: route declaration + page layout for that route.
- Prefer FastAPI lifespan hooks where appropriate for startup/shutdown behavior.
- Prefer type hints throughout.
### Database-specific notes (only if your app needs a DB)
- Prefer settings via `pydantic-settings` and read `DATABASE_URL` from environment/dotenv.
- Configure the SQLAlchemy engine once at app setup level; avoid creating engine/sessionmaker in endpoint functions.
- Provide `get_db_session()` via `yield` so sessions are consistently closed.
- Prefer a repository/service boundary to keep SQL details out of page modules.
- Prefer Alembic for schema changes and include commands for `revision --autogenerate` and `upgrade head`.
- Avoid `metadata.create_all()` in production startup flow; reserve it for optional local/dev bootstrap paths.
- Consider a light DB readiness check in health reporting (or a dedicated `/readyz`) when relevant.
- Ensure logs do not leak secrets (for example, avoid unsafe SQL echo in production).
## Output Format
The goal of the output is to produce an output
Return:
- A concise, high-level architecture explanation.
- An explanation of how the different parts of the concept will fit into this structure
- What are the core services?
- What are the main ui pages?
- What are the main ui components?
- Whether a database will be involved and whether the app owns the database
- Whether any AI will be used and what the workflow looks like
- A concrete implementation plan in the form of a checklist, organized into sections by concept which should roughly mirror the structure of sub-packages.
- Key functions/classes
- Configuration/settings available
- Migration or rollout notes (if relevant)
## Guardrails
- Do not put all pages in a single file.
- Never use globals, implicit or otherwise.
- Keep code minimal but production-minded.
- Being clear and concise is a higher priority than being verbose.
- Prefer clarity and maintainability over clever abstractions.
## Source Documentation (recommended references)
- FastAPI lifespan events:
- https://fastapi.tiangolo.com/advanced/events/
- FastAPI settings and environment variables:
- https://fastapi.tiangolo.com/advanced/settings/
- FastAPI dependencies with `yield` (session lifecycle pattern):
- https://fastapi.tiangolo.com/tutorial/dependencies/dependencies-with-yield/
- FastAPI SQL databases tutorial (when using a DB):
- https://fastapi.tiangolo.com/tutorial/sql-databases/
- SQLAlchemy engine configuration and pooling (when using a DB):
- https://docs.sqlalchemy.org/en/20/core/engines.html
- SQLAlchemy session basics and lifecycle guidance (when using a DB):
- https://docs.sqlalchemy.org/en/20/orm/session_basics.html
- Alembic tutorial and migration environment (when using a DB):
- https://alembic.sqlalchemy.org/en/latest/tutorial.html
- Pydantic settings management:
- https://pydantic.dev/docs/validation/latest/concepts/pydantic_settings/
- NiceGUI pages/routing and FastAPI integration:
- https://www.nicegui.io/documentation/section_pages_routing
- NiceGUI security best practices:
- https://www.nicegui.io/documentation/section_security
- LangGraph overview:
- https://docs.langchain.com/oss/python/langgraph/overview
- LangGraph quickstart:
- https://docs.langchain.com/oss/python/langgraph/quickstart
- LangGraph workflows and agents:
- https://docs.langchain.com/oss/python/langgraph/workflows-agents
- LangGraph persistence (checkpointer vs store):
- https://docs.langchain.com/oss/python/langgraph/persistence
- LangGraph memory concepts:
- https://docs.langchain.com/oss/python/concepts/memory
- LangGraph streaming:
- https://docs.langchain.com/oss/python/langgraph/streaming
- LangGraph interrupts / human-in-the-loop:
- https://docs.langchain.com/oss/python/langgraph/interrupts
@@ -0,0 +1,90 @@
---
name: pytest-fill-scaffold
description: Fill scaffolded pytest test methods with assertions, fixtures, and minimal test data while preserving concise test names and one-line intent docstrings.
x-personal-mcp:
id: pytest-fill-scaffold
version: 1.0.0
tags:
- pytest
- testing
- scaffolding
- prompts
capabilities:
- resource://prompts/pytest-fill-scaffold/document
arguments:
target_files:
description: Target test file paths under tests/.
required: true
stack:
description: Runtime stack type for fixture and marker choices.
required: true
strategy:
description: Balance between minimal and comprehensive implementation.
required: false
marker_lane:
description: Preferred marker lane when applicable.
required: false
---
# Pytest Fill Scaffold
Use this prompt after test scaffolding exists and method names/docstrings are already in place.
## Inputs
- Target test file(s) under tests/.
- Stack type:
- pure-python
- fastapi
- sqlalchemy-sync
- sqlalchemy-async
- mixed
- Optional constraints:
- keep implementation minimal vs comprehensive
- marker lane target (unit, integration, smoke)
## Required References
Load these in order and use only what matches the task:
1. Core defaults: [pytest scaffolding skill](../../skills/pytesting/SKILL.md)
2. Naming/hierarchy preservation: [naming and organization](../../skills/pytesting/references/naming-and-organization.md)
3. Baseline pytest fixtures/markers: [pytest docs notes](../../skills/pytesting/references/pytest-docs.md)
4. FastAPI-specific behavior (only when needed): [fastapi testing](../../skills/pytesting/references/fastapi-testing.md)
5. SQLAlchemy-specific behavior (only when needed): [sqlalchemy testing](../../skills/pytesting/references/sqlalchemy-testing.md)
## Workflow
1. Inspect target files and treat human-reviewed docstring-only scaffolds as invariant.
2. Convert each scaffolded method into an executable test with a single behavior focus.
3. Keep one-line docstrings for class and method intent.
4. Add or refine fixtures at the nearest useful scope:
- global in tests/conftest.py only when broadly reusable
- subtree conftest.py for domain-specific fixtures
5. Assign markers consistent with cost and dependencies:
- unit for pure logic
- integration for framework/DB contracts
- smoke for thin critical-path checks
6. Validate in this order:
- uv run pytest --collect-only -q
- uv run pytest -m unit -q when unit tests are touched
- uv run pytest -q if dependencies are available
## Authoring Rules
- Prefer deterministic tests and explicit setup/teardown.
- Keep assertions precise and readable.
- Do not overfit tests to private implementation details.
- If a scaffolded class or method has only a docstring body, treat its name and hierarchy as locked.
- Do not rename, move, merge, split, or re-nest docstring-only scaffolded tests unless explicitly requested.
- Preserve existing one-line docstrings on scaffolded classes and methods unless they are factually incorrect.
- If stack details are missing and would change fixture strategy, ask one concise clarifying question before editing.
## Output Format
Return:
1. Files updated.
2. Fixture and marker decisions.
3. Which references were used and why.
4. Validation command results.
5. Risks or open questions.
+97
View File
@@ -0,0 +1,97 @@
---
name: pytest-scaffold
description: Plan and optionally scaffold pytest file and class structure for selected Python modules while preserving concise behavior-focused test names and one-line intent docstrings.
x-personal-mcp:
id: pytest-scaffold
version: 1.0.0
tags:
- pytest
- testing
- scaffolding
- prompts
capabilities:
- resource://prompts/pytest-scaffold/document
arguments:
target_modules:
description: Target module path(s) under src/.
required: true
mode:
description: Execution mode, either plan-only or scaffold.
required: true
path_strategy:
description: Optional mapping preference for src to tests paths.
required: false
naming_style:
description: Optional preference for concise method naming style.
required: false
---
# Pytest Scaffold
Use this prompt to consistently plan and scaffold pytest test modules for selected Python source modules.
## Inputs
- Required:
- target_modules: one or more module paths under src/
- mode: one of plan-only or scaffold
- Optional:
- path_strategy: preference for how source paths map into tests/
- naming_style: preference for concise method naming style
## Required References
Load these in order and apply only the relevant sections:
1. Primary conventions: [Pytesting Skill](../../skills/pytesting/SKILL.md)
2. Hierarchy and naming: [Naming and Organization](../../skills/pytesting/references/naming-and-organization.md)
3. Marker and fixture defaults: [Pytest Docs Notes](../../skills/pytesting/references/pytest-docs.md)
## Workflow
1. Inspect the current tests/ layout and infer existing naming and grouping conventions.
2. Propose a concise hierarchy plan first:
- test file paths
- class hierarchy
- method naming pattern
- fixture placement choices (tests/conftest.py or subtree conftest.py)
3. If mode is scaffold, implement only the scaffold structure:
- create missing test modules
- create class hierarchy
- add one-line docstrings to each class and test method
- keep test method names short and behavior-focused
4. Treat docstring-only scaffolds as an intentionally stable baseline for later fill-in work.
5. Validate collection with:
- uv run pytest --collect-only -q
6. Report outcomes:
- files created or updated
- collection result
- ambiguities and follow-up choices
## Naming Defaults
- Class naming:
- Test<PrimarySubject> as a top-level subject class
- nested Test<MethodOrArea> classes where extra context improves readability
- Test<FunctionName> top-level classes for standalone module functions
- Method naming:
- test_<short_outcome>
- one behavior target per method
- one-line docstring for full intent
## Authoring Rules
1. Keep scope focused on structure and naming in this prompt.
2. Do not fill test implementation details unless explicitly requested.
3. Preserve established repository conventions when they are already present.
4. If input constraints conflict, ask one concise clarifying question before editing.
## Output Contract
Return:
1. Discovery summary and references used.
2. Proposed or applied test tree.
3. Class and method naming map.
4. Validation command result.
5. Open questions only when they block completion.
+139
View File
@@ -0,0 +1,139 @@
---
icon: lucide/shield-check
---
# Securing Remote Access
## Context
This project exposes two related surfaces from the same runtime:
1. a static documentation site under `/docs`
2. a Streamable HTTP MCP endpoint under `/mcp`
The same Markdown content backs both surfaces. For the current project shape, the MCP server is resource-first and primarily exposes public skill and documentation text. It is not intended to expose secrets, private data, shell access, filesystem access, or tools with side effects.
The expected deployment path is:
```text
Public internet
-> Cloudflare Tunnel
-> Caddy
-> personal-mcp container
```
## Decision
For the current use case, heavy application-level authentication is not required.
The recommended posture is:
1. Keep the service behind Cloudflare Tunnel and Caddy.
2. Do not expose the container port directly to the public internet.
3. Treat everything exposed through MCP as publishable public documentation.
4. Add stronger authentication only if the MCP surface later includes sensitive content or tools with meaningful side effects.
This keeps the deployment simple while preserving a clear upgrade path.
## Tradeoffs
### Leaving `/mcp` Public
This is acceptable if `/mcp` exposes only the same public Markdown already available through `/docs`.
Benefits:
1. lowest operational friction
2. fewer compatibility issues with MCP clients
3. no need to implement OAuth, mTLS, JWT validation, or custom auth middleware
4. consistent with the project assumption that documentation content is public
Risks:
1. random scraping, probing, or fuzzing of a machine endpoint
2. possible bandwidth or CPU nuisance traffic
3. accidental future exposure if new tools or private resources are added
4. less control over who can use the MCP endpoint
### Protecting `/mcp` With Cloudflare Access
Cloudflare Access can add a lightweight gate using GitHub, Google, one-time PIN, or service tokens.
Benefits:
1. reduces random internet traffic
2. requires little app code
3. works well for a small trusted team
4. provides logs and centralized access control
Costs:
1. browser-based login may not work with all MCP clients
2. non-browser MCP clients may need Cloudflare Access service tokens
3. adds operational configuration for a low-sensitivity endpoint
### Using mTLS
mTLS is useful when both client and server environments are tightly controlled.
Benefits:
1. strong client identity
2. good fit for service-to-service or private infrastructure
3. can be used between Cloudflare, Caddy, and the backend if desired
Costs:
1. harder certificate provisioning and rotation
2. weaker compatibility with normal MCP clients
3. unnecessary for public documentation-only content
For this project, mTLS is not the primary recommendation.
## Practical Recommendation
Use a simple public-docs posture unless the endpoint changes.
Recommended current setup:
```text
/docs public
/mcp public or lightly protected
```
If `/mcp` remains public, add only basic operational safeguards:
1. keep Cloudflare Tunnel and Caddy in front
2. avoid publishing `8765` directly
3. enable Cloudflare or Caddy rate limiting if traffic becomes noisy
4. monitor logs for unusual request volume
5. document that MCP resources must remain safe to publish
A slightly stricter setup is also reasonable:
```text
/docs public
/mcp Cloudflare Access or service token
```
This is the best option if the team wants to reduce drive-by MCP traffic without adding auth code to the application.
## Upgrade Trigger
Add real authentication before introducing any MCP capability that can:
1. read non-public files
2. access private notes or credentials
3. call upstream APIs
4. mutate data
5. run commands
6. expose environment details
7. perform expensive computation
At that point, prefer edge-level authentication first, such as Cloudflare Access, and consider proper OAuth 2.1 resource-server behavior only if broad public MCP client interoperability becomes a goal.
## Security Invariant
Everything exposed by the MCP server must be safe to publish publicly.
If that invariant stops being true, `/mcp` should be protected before the new capability is deployed.
+207
View File
@@ -0,0 +1,207 @@
---
name: async-fastapi-sqlmodel
description: 'Explain and apply async database principles for FastAPI, SQLAlchemy 2.x, and SQLModel. Use when: learning or reviewing AsyncEngine and AsyncSession lifecycles, FastAPI lifespan and yield dependencies, transaction boundaries, concurrency safety, implicit ORM I/O, AsyncExitStack, pooling, testing, or SQLModel integration.'
x-personal-mcp:
id: async-fastapi-sqlmodel
version: 1.1.0
tags:
- fastapi
- sqlalchemy
- sqlmodel
- async
- asyncio
- database
- transactions
- resource-lifecycle
- architecture
capabilities:
- resource://skills/async-fastapi-sqlmodel/document
---
# Async FastAPI, SQLAlchemy, and SQLModel
Use this skill to explain how an async database layer works, why the recommended patterns exist, and how to evaluate code against them. Teach the runtime model before suggesting implementation changes.
Primary targets: PostgreSQL with asyncpg and SQLite with aiosqlite.
## When to Use
- Explain an async engine, session factory, session, connection, or transaction.
- Review FastAPI lifespan or dependency-based database management.
- Diagnose shared-session concurrency, implicit I/O, cleanup, or transaction problems.
- Compare SQLModel's model conveniences with SQLAlchemy's async runtime APIs.
- Decide whether a context manager, `AsyncExitStack`, eager loading, pooling option, or explicit transaction is appropriate.
## Outcome
Produce a focused technical explanation that:
- Defines the objects involved and identifies who owns each one.
- Traces acquisition, use, transaction behavior, and cleanup.
- Separates required invariants from defaults and situational choices.
- Explains failure modes and concurrency consequences.
- Uses a minimal canonical pattern when code clarifies the mechanics.
- Links claims to the relevant reference and upstream documentation.
Do not default to producing a project plan. Give sequencing advice only when the user explicitly asks for implementation steps.
## Mental Model
Keep three ownership scopes distinct:
| Scope | Object | Purpose | Typical owner |
|---|---|---|---|
| Application process | `AsyncEngine` and `async_sessionmaker` | Dialect, connection pool, and repeatable session configuration | FastAPI lifespan |
| Request or concurrent task | `AsyncSession` | Mutable ORM identity map and transactional state | A `yield` dependency or explicit unit of work |
| Atomic operation | `SessionTransaction` | Commit all changes together or roll them back together | Service or use-case boundary |
The engine is a long-lived factory and pool, not a single database connection. The session is a mutable unit-of-work object, not a concurrency-safe global. A transaction is a consistency boundary, not merely a call to `commit()`.
## Core Principles
### Match lifetime to ownership
- Create one `AsyncEngine` per process and database configuration in the normal case.
- Dispose it explicitly in an awaitable shutdown path; garbage collection cannot reliably await async driver cleanup.
- Configure `async_sessionmaker` once and call it to create short-lived sessions.
- Close each session deterministically with `async with` or a FastAPI dependency that yields once.
See [engine lifecycle](references/engine.md) and [session management](references/session.md).
### Isolate mutable session state
An `AsyncSession` represents one stateful transaction in progress. Never use one session in multiple concurrent tasks, including branches of `asyncio.gather()`. Give each task its own session and pass sessions explicitly rather than relying on mutable scoped globals.
See [session management](references/session.md).
### Make I/O visible
Async ORM code must not unexpectedly issue SQL during ordinary attribute access. Load relationships and deferred columns explicitly with eager loader options such as `selectinload()`, use `awaitable_attrs` or `refresh()` for deliberate fallback loading, and consider `lazy="raise"` where accidental access should fail fast. `expire_on_commit=False` is a common async configuration because post-commit expiration can otherwise turn attribute reads into implicit I/O.
See [implicit ORM I/O](references/implicit_io.md).
### Put transactions around business invariants
Use `async with session.begin():` when several operations must commit or roll back as one unit. A successful exit flushes and commits; an exception rolls back. Reads still participate in SQLAlchemy's autobegin behavior unless the connection uses true DBAPI autocommit, so describe a path as read-only because of application intent and permissions, not because a session silently has no transaction.
Use `begin_nested()` only for a real SAVEPOINT requirement and account for backend-specific behavior. In SQLAlchemy 2.x, calling `session.commit()` commits the outermost transaction, not the current savepoint.
See [transaction boundaries](references/transactions.md).
### Keep framework boundaries explicit
FastAPI lifespan owns resources shared by many requests. A dependency with one `yield` owns request-scoped resources and runs cleanup after use. These are related context-manager mechanisms but solve different lifetime problems.
Use `AsyncExitStack` when lifespan acquires a variable, conditional, or mixed collection of context-managed resources. It records cleanup as resources are acquired and unwinds callbacks in reverse order. A single engine with one cleanup callback can use a plain `try/finally`; `AsyncExitStack` is a composition tool, not a requirement.
See [engine lifecycle](references/engine.md).
### Use SQLModel as the primary modeling layer
Default to SQLModel for table models and API data models in FastAPI applications. A SQLModel table model is also a SQLAlchemy model, and every SQLModel model is also a Pydantic model, so shared base models can reduce schema duplication while preserving access to SQLAlchemy's full ORM.
SQLModel does not replace SQLAlchemy's async engine, session, transaction, or loader mechanics. Its main tutorial currently demonstrates synchronous sessions and its advanced guide still lists comprehensive async documentation as future work. For async applications, combine SQLModel models and statements with SQLAlchemy's `AsyncSession` APIs. Use SQLAlchemy declarative models only when a concrete unsupported mapping or library constraint justifies the exception.
See [SQLModel integration](references/sqlmodel.md).
### Configure from evidence
Pool sizing, overflow, recycle, pre-ping, isolation, statement timeouts, and health checks depend on the driver, database, deployment concurrency, and failure model. Explain defaults and tradeoffs before recommending values. Avoid treating pool checkout as proof that a useful query can succeed.
See [observability and resilience](references/observability.md).
## Reference Map
| Concept | Reference |
|---|---|
| Engine lifecycle and ownership | [Engine lifecycle reference](references/engine.md) |
| Session factory and scope | [Session management reference](references/session.md) |
| Transaction boundaries | [Transaction boundaries reference](references/transactions.md) |
| Lifespan composition | [Engine lifecycle reference](references/engine.md) |
| Dependency injection | [Session management reference](references/session.md) |
| Implicit I/O control in ORM | [Implicit I/O reference](references/implicit_io.md) |
| Observability and resilience | [Observability reference](references/observability.md) |
| SQLModel-first modeling | [SQLModel integration reference](references/sqlmodel.md) |
| CRUD repository and standalone functions | [Basic CRUD reference](references/crud.md) |
## Canonical Composition Pattern
This example shows the ownership boundaries. Adapt state storage and dependency wiring to the application's conventions.
```python
from contextlib import AsyncExitStack, asynccontextmanager
from collections.abc import AsyncIterator
from fastapi import FastAPI
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker, create_async_engine
@asynccontextmanager
async def lifespan(app: FastAPI) -> AsyncIterator[None]:
async with AsyncExitStack() as stack:
engine = create_async_engine(settings.database_url)
stack.push_async_callback(engine.dispose)
session_factory = async_sessionmaker(engine, expire_on_commit=False)
app.state.session_factory = session_factory
yield
async def get_session() -> AsyncIterator[AsyncSession]:
async with app.state.session_factory() as session:
yield session
```
For direct construction without `AsyncExitStack`, put `await engine.dispose()` in a `finally` block. For background work that outlives a request, create a new session inside that task instead of retaining the request's session.
## Explanation Procedure
1. Identify the exact concept or observed behavior in question.
2. Name the owning scope: application, request/task, or transaction.
3. Trace what state the object holds and where actual database I/O can occur.
4. Explain normal entry, successful exit, exceptional exit, and concurrent use.
5. Distinguish an invariant from a recommended default or backend-specific choice.
6. Load only the matching reference documents and cite upstream sources.
7. Show the smallest useful code pattern or contrast when prose is insufficient.
8. End with concrete checks the reader can use to inspect their own code.
When reviewing code, verify:
- The URL uses an asyncio-compatible dialect.
- Engine creation and disposal have one clear owner.
- Every session has a bounded lifetime and is not shared across tasks.
- Transaction boundaries match business invariants and exception behavior.
- Relationship and deferred-column access cannot surprise the event loop with implicit I/O.
- Pool and timeout settings are justified by deployment behavior.
- Tests exercise rollback, cleanup, concurrency, and lifespan behavior where relevant.
## Anti-Patterns to Flag
- Creating engines inside request handlers.
- Sharing one AsyncSession across concurrent tasks.
- Implicit commit/rollback behavior with unclear ownership.
- Global mutable session state.
- Lifespan cleanup that depends on implicit garbage collection.
- Treating `AsyncExitStack` as mandatory for a fixed single resource.
- Treating SQLModel's synchronous tutorial examples as the async runtime pattern.
- Allowing lazy relationship access to hide database I/O.
- Copying pool settings without relating them to worker count and database capacity.
## Output Contract
Answer in the shape best suited to the question, usually:
1. Direct explanation.
2. Underlying lifecycle or transaction mechanics.
3. Required invariants and situational tradeoffs.
4. Minimal example or code-review findings when useful.
5. Verification questions and source links.
## References
!!! info "Primary sources"
- [SQLAlchemy asyncio extension](https://docs.sqlalchemy.org/en/21/orm/extensions/asyncio.html)
- [SQLAlchemy transaction management](https://docs.sqlalchemy.org/en/21/orm/session_transaction.html)
- [FastAPI lifespan events](https://fastapi.tiangolo.com/advanced/events/)
- [FastAPI dependencies with `yield`](https://fastapi.tiangolo.com/tutorial/dependencies/dependencies-with-yield/)
- [Python `AsyncExitStack`](https://docs.python.org/3/library/contextlib.html#contextlib.AsyncExitStack)
- [SQLModel session dependency pattern](https://sqlmodel.tiangolo.com/tutorial/fastapi/session-with-dependency/)
@@ -0,0 +1,334 @@
# Basic CRUD Repository and Functions
!!! info "Primary sources"
- [SQLModel create-data tutorial](https://sqlmodel.tiangolo.com/tutorial/fastapi/multiple-models/)
- [SQLModel update-data tutorial](https://sqlmodel.tiangolo.com/tutorial/fastapi/update-extra-data/)
- [SQLModel select tutorial](https://sqlmodel.tiangolo.com/tutorial/select/)
- [SQLAlchemy `AsyncSession` API](https://docs.sqlalchemy.org/en/21/orm/extensions/asyncio.html#sqlalchemy.ext.asyncio.AsyncSession)
??? abstract "Decision metadata"
- Status: adopted
- Decision level: advisory
- Applies to: api-runtime, workers, tests
- Last reviewed: 2026-07-26
---
## Purpose
Show a small SQLModel CRUD layer in two forms:
- independent functions for convenient standalone or composed operations;
- a repository object that groups those functions behind one domain-oriented interface.
Every public operation accepts an optional `AsyncSession`. When omitted, reads resolve the cached session factory and own a short-lived session, while writes resolve the same factory and own a complete session-and-transaction scope. When supplied, reads borrow the session and writes borrow its already-active caller-owned transaction. The repository stores configuration and delegates to the same functions without changing those semantics.
Use the same vocabulary at every layer:
| Operation | Function | Repository method | Scope when session is omitted | Missing-row result |
|---|---|---|---|---|
| Create | `create_widget()` | `create()` | Owned transaction | Not applicable |
| Read one | `get_widget()` | `get()` | Owned session | `None` |
| Read many | `list_widgets()` | `list()` | Owned session | Empty list |
| Update | `update_widget()` | `update()` | Owned transaction | `None` |
| Delete | `delete_widget()` | `delete()` | Owned transaction | `None` |
Functions and repository methods both put domain arguments first. Database configuration and sessions are keyword-only infrastructure arguments. This keeps call sites analogous and makes ownership choices visible.
---
## Models
Start with one table model when the application does not need distinct persistence and API schemas.
```python
from sqlmodel import Field
from sqlmodel import SQLModel
class Widget(SQLModel, table=True):
id: int | None = Field(default=None, primary_key=True)
name: str = Field(index=True)
description: str | None = None
```
This reference uses direct field arguments and full-update semantics to keep the CRUD mechanics visible. Introduce separate create, update, or public schemas only when an API boundary needs different validation, field visibility, or partial-update behavior. See [SQLModel integration](sqlmodel.md) for that larger modeling pattern.
---
## Independent CRUD Functions
Functions are the simplest default when grouping state or behavior in an object adds no value. Each function is a complete operation boundary: it can run standalone by resolving the cached factory from `database_url`, or compose into a caller-owned scope through `session`.
```python
from sqlalchemy.ext.asyncio import AsyncSession
from sqlmodel import select
from .session import session_scope
from .session import transaction_scope
async def create_widget(
name: str,
description: str | None = None,
*,
database_url: str,
session: AsyncSession | None = None,
) -> Widget:
async with transaction_scope(
database_url=database_url,
session=session,
) as active_session:
widget = Widget(name=name, description=description)
active_session.add(widget)
await active_session.flush()
return widget
async def get_widget(
widget_id: int,
*,
database_url: str,
session: AsyncSession | None = None,
) -> Widget | None:
async with session_scope(
database_url=database_url,
session=session,
) as active_session:
return await active_session.get(Widget, widget_id)
async def list_widgets(
*,
database_url: str,
offset: int = 0,
limit: int = 100,
session: AsyncSession | None = None,
) -> list[Widget]:
if offset < 0:
raise ValueError("offset must be non-negative")
if not 1 <= limit <= 100:
raise ValueError("limit must be between 1 and 100")
async with session_scope(
database_url=database_url,
session=session,
) as active_session:
statement = select(Widget).order_by(Widget.id).offset(offset).limit(limit)
return list(await active_session.scalars(statement))
async def update_widget(
widget_id: int,
name: str,
description: str | None,
*,
database_url: str,
session: AsyncSession | None = None,
) -> Widget | None:
async with transaction_scope(
database_url=database_url,
session=session,
) as active_session:
widget = await active_session.get(Widget, widget_id)
if widget is None:
return None
widget.name = name
widget.description = description
await active_session.flush()
return widget
async def delete_widget(
widget_id: int,
*,
database_url: str,
session: AsyncSession | None = None,
) -> Widget | None:
async with transaction_scope(
database_url=database_url,
session=session,
) as active_session:
widget = await active_session.get(Widget, widget_id)
if widget is None:
return None
await active_session.delete(widget)
await active_session.flush()
return widget
```
Update and delete load the row through the same session that mutates it. This avoids accepting detached instances from an earlier standalone read and gives both operations an explicit `None` result that the application layer can map to a domain or HTTP error. Delete returns the loaded object for callers that need its values, but that object represents a row scheduled for deletion and must not be reused as persistent state. List operations validate their bounds and order by the primary key so pagination is deterministic. Add a unique tiebreaker whenever ordering by a non-unique field.
`flush()` sends pending writes and populates ordinary generated primary keys. It does not itself commit. For a standalone write, the surrounding owned `transaction_scope()` commits after the function body succeeds. For a supplied session, the caller's outer transaction retains commit and rollback ownership. Use `await active_session.refresh(widget)` only when the operation deliberately needs database-generated state that was not returned during the flush; an unconditional refresh adds another query.
---
## Repository Object
A repository can provide a stable domain-facing interface when several callers need the same grouped operations. It stores repeatable database configuration, never a mutable session. Every method delegates to the analogous function and exposes the same optional-session contract.
```python
from sqlalchemy.ext.asyncio import AsyncSession
class WidgetRepository:
def __init__(self, database_url: str) -> None:
self.database_url = database_url
async def create(
self,
name: str,
description: str | None = None,
*,
session: AsyncSession | None = None,
) -> Widget:
return await create_widget(
name,
description,
database_url=self.database_url,
session=session,
)
async def get(
self,
widget_id: int,
*,
session: AsyncSession | None = None,
) -> Widget | None:
return await get_widget(
widget_id,
database_url=self.database_url,
session=session,
)
async def list(
self,
*,
offset: int = 0,
limit: int = 100,
session: AsyncSession | None = None,
) -> list[Widget]:
return await list_widgets(
database_url=self.database_url,
offset=offset,
limit=limit,
session=session,
)
async def update(
self,
widget_id: int,
name: str,
description: str | None,
*,
session: AsyncSession | None = None,
) -> Widget | None:
return await update_widget(
widget_id,
name,
description,
database_url=self.database_url,
session=session,
)
async def delete(
self,
widget_id: int,
*,
session: AsyncSession | None = None,
) -> Widget | None:
return await delete_widget(
widget_id,
database_url=self.database_url,
session=session,
)
```
The object is intentionally thin. Tests can construct it with a test database URL or pass a transaction-scoped test session to individual methods. A caller-provided session always wins and remains open after the method returns. A standalone operation closes its owned session before returning, so returned objects are detached; load every required scalar, deferred column, and relationship explicitly before the scope exits, and do not mutate those objects expecting persistence.
If a read participates in a later write, pass the same session and place both operations inside the explicit transaction. This avoids splitting one use case across sessions and keeps SQLAlchemy's autobegin behavior from obscuring transaction ownership. Add a repository only when its naming, shared query policy, dependency substitution, or domain boundary improves the application. Independent functions remain a valid and often clearer design.
---
## Transaction Ownership
Compose multiple calls under one use-case transaction. At this boundary, a supplied session joins its already-active caller-owned transaction, while omitting the session creates a standalone session and transaction. Each nested CRUD write receives `active_session`, detects that transaction, and borrows it instead of committing independently.
The scope names describe exactly what they own: `session_scope()` manages session lifetime but never commits, while `transaction_scope()` manages a complete transaction only when it also creates the session. Both yield the name `active_session` because downstream CRUD code does not need to know whether the session was borrowed or owned.
```python
from .session import transaction_scope
async def replace_widget(
repository: WidgetRepository,
widget_id: int,
replacement_name: str,
replacement_description: str | None = None,
*,
session: AsyncSession | None = None,
) -> Widget | None:
async with transaction_scope(
database_url=repository.database_url,
session=session,
) as active_session:
deleted_widget = await repository.delete(
widget_id,
session=active_session,
)
if deleted_widget is None:
return None
return await repository.create(
replacement_name,
replacement_description,
session=active_session,
)
```
If creation fails, deletion rolls back with it. For a caller-owned transaction, wrap the call in `async with session.begin():` and pass that session. For a standalone use case, omit the session; the outer `transaction_scope()` commits on successful exit, rolls back on exception, and closes its owned session. Do not add direct `commit()` calls to CRUD functions or repository methods because that prevents callers from composing several operations atomically. See [transaction boundaries](transactions.md) and [session management](session.md) for ownership details.
---
## Anti-Patterns
- Storing one mutable `AsyncSession` on a long-lived repository object.
- Constructing ad hoc factories or sessions instead of resolving the cached factory through the scope helpers.
- Using `session_scope()` for an optional write, which would close an owned session without committing.
- Accepting a supplied session for a write without requiring an active caller-owned transaction.
- Calling `commit()` or `rollback()` directly instead of expressing ownership through `transaction_scope()`.
- Accepting unbounded list queries.
- Accepting detached ORM instances for update or delete when an identifier can be resolved in the active session.
- Accessing unloaded attributes after a standalone repository read has closed its owned session.
---
## Operational Checks
- Every CRUD call receives a task-local `AsyncSession`.
- Standalone reads resolve the cached factory by database URL and close their owned session.
- Standalone writes resolve the cached factory and own commit, rollback, and session cleanup through `transaction_scope()`.
- Supplied write sessions already have an active caller-owned transaction.
- Each complete operation, service, or use-case boundary borrows an active transaction or owns a complete session-and-transaction scope.
- List operations have pagination and deterministic ordering where required.
- Update requires values for both mutable fields; passing `None` explicitly clears the nullable description.
- Get, update, and delete use the same identifier and missing-row semantics.
- Functions and repository methods use domain arguments first and keyword-only infrastructure arguments consistently.
- Standalone reads load all state needed after their owned session closes.
- Repository objects hold configuration or policy, never request-scoped session state.
---
## Testing Checks
- Create tests verify generated identifiers and persisted field values after commit.
- Get and list tests cover found, missing, pagination, and ordering behavior.
- List tests reject negative offsets and limits outside the supported range.
- Update tests cover replacement of both mutable fields, including clearing the nullable description.
- Update and delete tests cover missing identifiers without mutating the database.
- Delete tests verify the returned row and its absence after commit.
- Failure tests verify that a surrounding transaction rolls back all composed CRUD calls.
- Optional-session read tests verify borrowed sessions remain open and owned sessions close without committing.
- Optional-session write tests verify supplied transactions remain caller-owned and standalone transactions commit or roll back before closing.
- Composition tests pass one active session through several CRUD calls and verify one atomic commit or rollback.
@@ -0,0 +1,171 @@
# Async SQLAlchemy Engine
!!! info "Primary sources"
- [Python `functools.cache`](https://docs.python.org/3/library/functools.html#functools.cache)
- [SQLAlchemy connections](https://docs.sqlalchemy.org/en/21/core/connections.html)
- [SQLAlchemy asyncio extension](https://docs.sqlalchemy.org/en/21/orm/extensions/asyncio.html)
- [SQLAlchemy pooling and multiprocessing](https://docs.sqlalchemy.org/en/21/core/pooling.html#pooling-multiprocessing)
- [FastAPI lifespan events](https://fastapi.tiangolo.com/advanced/events/)
---
## Engine Ownership Model
Create one async engine per process per database URL and keep engine construction independent from FastAPI.
- SQLAlchemy guidance: the engine is intended as a long-lived, concurrent registry over pooled DB connections, not a per-request object.
- A cached function provides stable process-local engine identity without making framework state the only way to obtain it.
- FastAPI lifespan starts and stops that independently defined resource; it does not contain the construction policy.
!!! tip "Practical rule"
- Exactly one `create_async_engine(...)` call in the cached engine factory.
- Zero `create_async_engine(...)` calls in request handlers.
- Zero calls to the cached factory from repository code.
---
## Cached Engine Factory
Use [`functools.cache`](https://docs.python.org/3/library/functools.html#functools.cache) on a synchronous factory. Creating an `AsyncEngine` configures the dialect and pool; it does not need to await a database connection.
```python
from functools import cache
from sqlalchemy.ext.asyncio import AsyncEngine, create_async_engine
@cache
def get_engine(database_url: str) -> AsyncEngine:
return create_async_engine(
database_url,
pool_pre_ping=True,
)
async def dispose_engine(database_url: str) -> None:
engine = get_engine(database_url)
try:
await engine.dispose()
finally:
get_engine.cache_clear()
async def refresh_engine(database_url: str) -> AsyncEngine:
await dispose_engine(database_url)
return get_engine(database_url)
```
The database URL is an explicit, hashable cache key. Calls with the same URL return the same engine; a different URL receives a different engine. If engine options vary at runtime, make them explicit hashable arguments too.
Resolve settings at the composition boundary and call `get_engine(settings.database_url)`. Do not hide settings lookup or engine creation inside feature code.
## Thin FastAPI Lifespan Wrapper
The lifespan context manager only connects the cached resource to FastAPI ownership:
```python
from collections.abc import AsyncIterator
from contextlib import asynccontextmanager
from fastapi import FastAPI
@asynccontextmanager
async def lifespan(app: FastAPI) -> AsyncIterator[None]:
database_url = app.state.settings.database_url
engine = get_engine(database_url)
app.state.engine = engine
try:
yield
finally:
await dispose_engine(database_url)
app = FastAPI(lifespan=lifespan)
```
`dispose()` closes checked-in connections and replaces the pool, but it does not remove the Python object from `functools.cache`. `dispose_engine()` clears the cache even if driver cleanup raises, preventing a later lifespan run or test from retrieving that engine instance.
This simple cleanup assumes one configured database URL per process. If a process intentionally owns several cached engines, use a small registry with per-key removal instead of clearing the whole cache. For a fixed engine, `try/finally` is sufficient; use `AsyncExitStack` when lifespan composes multiple conditional or dynamically acquired resources.
When directly testing engine construction or lifespan behavior:
- Call `get_engine.cache_clear()` before the test to remove process-local state.
- Dispose any engine the test creates.
- Clear the cache again during teardown, even when the test fails.
---
## Driver URLs (Project Requirement: asyncpg + aiosqlite)
Use SQLAlchemy async driver URLs:
- PostgreSQL: `postgresql+asyncpg://user:pass@host:5432/dbname`
- SQLite: `sqlite+aiosqlite:///./app.db`
!!! warning "Driver compatibility"
- Do not mix sync drivers, for example `psycopg2`, with `create_async_engine()`.
- Keep URL construction centralized in settings/config, not in feature modules.
---
## Pooling Defaults and Tuning
Default behavior is usually correct first:
- Async engines use async-compatible pooling (`AsyncAdaptedQueuePool`) by default.
- Start with defaults, then tune from observed load (`pool_size`, `max_overflow`, `pool_timeout`, `pool_recycle`).
- Enable `pool_pre_ping=True` for safer stale-connection handling in long-running services.
When to switch pool strategy:
- `NullPool` if you explicitly need no pooling (special environments, some tests, or strict cross-loop constraints).
- Keep in mind this increases connect/disconnect churn.
---
## Disposal Semantics
`engine.dispose()` replaces/disposes the pool, but only checked-in connections are immediately closed.
Rules:
- Dispose when the app is shutting down.
- Dispose before reusing an engine across event loops.
- In forked child-process initialization, use `engine.dispose(close=False)` (sync API guidance) so child processes do not touch parent-held connections.
Avoid relying on garbage collection for engine cleanup in async code.
---
## Event Loop and Process Boundaries
Do not share pooled connections across boundaries:
- Multiple event loops: do not reuse the same pooled async engine across loops unless you intentionally disable pooling (`NullPool`) or dispose before handoff.
- Multiprocessing/fork: pooled connections must not be inherited for active use across process boundaries.
This prevents broken socket state and cross-process connection corruption.
---
## What Not to Do
- Create an engine inside every request dependency.
- Create/dispose engines inside repository methods.
- Call `get_engine()` from repositories instead of injecting their engine or session dependency.
- Keep engine creation as a hidden side effect of import-time module globals.
- Dispose a cached engine without clearing the cache during final teardown.
- Use deprecated FastAPI startup/shutdown events together with lifespan.
---
## Engine Design Checklist
- One engine per process per DB URL.
- Engine created by one cached, framework-independent factory.
- Lifespan only retrieves, exposes, disposes, and uncaches the engine.
- Async driver URL matches backend (`asyncpg` or `aiosqlite`).
- Pooling strategy is explicit for non-default needs.
- No request-path engine creation.
- Tests dispose engines and clear cached state deterministically.
@@ -0,0 +1,101 @@
# Preventing Implicit ORM I/O (Asyncio)
!!! info "Primary sources"
- [Preventing implicit I/O with AsyncSession](https://docs.sqlalchemy.org/en/21/orm/extensions/asyncio.html#preventing-implicit-io-when-using-asyncsession)
- [SQLAlchemy relationship loading](https://docs.sqlalchemy.org/en/21/orm/queryguide/relationships.html)
??? abstract "Decision metadata"
- Status: adopted
- Decision level: advisory
- Applies to: api-runtime, workers, tests
- Last reviewed: 2026-06-17
---
## Purpose
Minimize unexpected database round-trips caused by attribute access in async ORM code.
In asyncio applications, hidden lazy loads are easy to miss and can produce runtime surprises. This guide defines explicit-loading defaults and progressive enforcement practices.
---
## Scope and Non-Goals
- In scope: relationship loading strategy, post-commit attribute access, explicit refresh/awaitable access patterns.
- Out of scope: full ORM performance tuning and domain-specific query architecture.
---
## Rules
- Prefer explicit eager loading for data required by endpoint/service outputs.
- Avoid relying on implicit lazy-load behavior in request critical paths.
- Keep `expire_on_commit=False` unless strict expiration behavior is intentionally required.
- Use explicit refresh or awaitable-attribute access when loading deferred state is necessary.
---
## Recommended Patterns
### Pattern A: Eager-load what you need
```python
from sqlalchemy import select
from sqlalchemy.orm import selectinload
stmt = select(User).options(selectinload(User.roles))
users = (await session.scalars(stmt)).all()
```
### Pattern B: Explicit refresh of named attributes
```python
user = await session.get(User, user_id)
await session.refresh(user, ["roles"])
```
### Pattern C: Awaitable attribute access where needed
```python
# Requires AsyncAttrs mixin on mapped base or class.
roles = await user.awaitable_attrs.roles
```
---
## Practical Enforcement Model
Require explicit I/O behavior on every async ORM path:
1. Define loader options for relationships and deferred columns needed by the operation.
2. Use `refresh()` or awaitable attributes only when the additional query is deliberate and visible.
3. Add review checks that reject unplanned lazy-load paths.
This keeps event-loop behavior predictable and makes query boundaries reviewable from the code.
---
## Anti-Patterns
- Returning ORM objects from handlers and triggering lazy loads during serialization.
- Assuming post-commit attribute access will always be loaded without explicit strategy.
- Relying on broad expiration + implicit reload behavior in async request flows.
- Enabling relationship patterns that hide SQL behavior in critical code paths.
---
## Operational Checks
- Endpoint query blocks define loader options for returned related data.
- Critical handlers do not depend on incidental lazy loads.
- Known exceptions are documented with rationale and follow-up items.
---
## Testing Checks
- Integration tests cover endpoints that return related objects.
- Tests verify expected data is present without hidden secondary query surprises.
- Regression tests exist for routes previously affected by implicit-load failures.
@@ -0,0 +1,34 @@
# FastAPI Async SQLAlchemy References Index
Purpose: concept registry for the principles, mechanics, and implementation guidance used by this skill.
---
## Concepts
| Concept | File | Status | Decision Level | Owner | Last Reviewed |
|---|---|---|---|---|---|
| Engine lifecycle and ownership | [engine.md](engine.md) | adopted | mandatory | platform/backend | 2026-06-17 |
| Session factory and scope | [session.md](session.md) | adopted | mandatory | platform/backend | 2026-06-17 |
| Transaction boundaries | [transactions.md](transactions.md) | adopted | mandatory | platform/backend | 2026-06-17 |
| Implicit ORM I/O under asyncio | [implicit_io.md](implicit_io.md) | adopted | advisory | platform/backend | 2026-06-17 |
| Observability and resilience | [observability.md](observability.md) | adopted | mandatory | platform/backend | 2026-06-17 |
| SQLModel modeling and async boundaries | [sqlmodel.md](sqlmodel.md) | adopted | mandatory | platform/backend | 2026-07-26 |
| Basic CRUD repository and functions | [crud.md](crud.md) | adopted | advisory | platform/backend | 2026-07-26 |
---
## How to Use This Folder
- `SKILL.md` defines the explanatory workflow and shared mental model.
- Each concept doc defines policy-level guidance for one concern.
- Use the template in [template.md](template.md) for new concept docs.
- Keep references source-linked and implementation snippets minimal.
---
## Update Rules
- If a PR changes database lifecycle/session/ORM loading behavior, update the relevant concept file.
- Keep `Status`, `Decision Level`, and `Last Reviewed` current.
- Use `advisory` for recommendations that depend on application context; use `mandatory` for required runtime policy.
@@ -0,0 +1,107 @@
# DB Observability and Resilience
!!! info "Primary sources"
- [SQLAlchemy pooling](https://docs.sqlalchemy.org/en/21/core/pooling.html)
- [SQLAlchemy engine configuration](https://docs.sqlalchemy.org/en/21/core/engines.html)
- [SQLAlchemy events](https://docs.sqlalchemy.org/en/21/core/events.html)
- [FastAPI lifespan events](https://fastapi.tiangolo.com/advanced/events/)
??? abstract "Decision metadata"
- Status: adopted
- Decision level: mandatory
- Applies to: api-runtime, workers, tests
- Last reviewed: 2026-06-17
---
## Purpose
Define baseline observability and resilience practices for DB connectivity in async FastAPI + SQLAlchemy apps.
Goals:
- detect and recover from stale/disconnected connections,
- expose useful diagnostics for pool/engine behavior,
- make readiness/liveness signals meaningful.
---
## Scope and Non-Goals
- In scope: pool health, connection liveness, SQL/pool logging hygiene, readiness checks, failure handling.
- Out of scope: full APM stack design and vendor-specific monitoring platform setup.
---
## Rules
- Enable connection liveness strategy (`pool_pre_ping=True`) for long-running services.
- Keep DB health checks out of liveness; include dependency checks in readiness.
- Centralize engine options and logging configuration.
- Avoid noisy SQL debug logging in production defaults.
- Treat disconnect handling as a first-class test scenario.
---
## Recommended Baseline
```python
engine = create_async_engine(
settings.database_url,
pool_pre_ping=True,
# Tune only from measured behavior:
# pool_size=10,
# max_overflow=20,
# pool_timeout=30,
# pool_recycle=1800,
)
```
Operational guidance:
- `pool_pre_ping=True` for stale-connection resilience.
- Introduce `pool_recycle` where backend/network idle timeout behavior warrants it.
- Use structured app logs with request correlation and error context.
---
## Health Endpoint Policy
- `/healthz`: process is alive; no DB call required.
- `/readyz`: application can currently serve traffic; include DB connectivity verification.
Readiness checks should be lightweight and bounded (timeouts), not heavy diagnostic queries.
---
## Failure Handling Guidance
- Handle transient disconnects with pool invalidation/reconnect semantics.
- Keep one failed request from cascading into broad app instability.
- Capture and log contextual DB errors with enough metadata for debugging.
---
## Anti-Patterns
- No readiness check for DB-dependent services.
- Permanent debug SQL echo in production.
- Per-handler ad hoc pool settings.
- Assuming disconnect events are too rare to test.
---
## Operational Checks
- Engine creation is centralized and configured once.
- Liveness/readiness behavior is documented and validated.
- Pool settings are explicit, versioned, and reviewed.
- DB-related errors produce actionable logs.
---
## Testing Checks
- Readiness endpoint test covers healthy and unhealthy DB states.
- Integration test simulates disconnect/reconnect behavior.
- Load/concurrency tests validate pool behavior under stress.
@@ -0,0 +1,383 @@
# Async SQLAlchemy Session Management
!!! info "Primary sources"
- [Python `functools.cache`](https://docs.python.org/3/library/functools.html#functools.cache)
- [Python `asynccontextmanager`](https://docs.python.org/3/library/contextlib.html#contextlib.asynccontextmanager)
- [SQLAlchemy asyncio extension](https://docs.sqlalchemy.org/en/21/orm/extensions/asyncio.html)
- [SQLAlchemy session basics](https://docs.sqlalchemy.org/en/21/orm/session_basics.html)
- [FastAPI dependencies with yield](https://fastapi.tiangolo.com/tutorial/dependencies/dependencies-with-yield/)
---
## Purpose
Define one canonical session model for FastAPI + SQLAlchemy asyncio:
- configure one shared session factory,
- create one AsyncSession per request or per unit-of-work,
- never share one AsyncSession across concurrent tasks.
---
## Scope and Non-Goals
- In scope: session factory creation, FastAPI dependency wiring, request/task scoping, transaction demarcation.
- Out of scope: ORM model design, query optimization strategy, schema migration tooling.
---
## Rules
- Create one cached `async_sessionmaker` per app-owned AsyncEngine.
- Let repositories resolve the cached maker by database URL.
- Use a fresh AsyncSession for each request or explicit unit-of-work.
- Pass an `AsyncSession` directly to data-access functions.
- Borrow a caller-provided session without closing or committing it.
- Do not share AsyncSession across `asyncio.gather()` or parallel tasks.
- Prefer direct dependency injection over global scoped-session patterns in new code.
- Use explicit transaction boundaries (`async with session.begin():`) for writes.
- When a use case accepts an optional session, borrow only an active caller-owned transaction or own the complete session-and-transaction scope.
---
## Sessions and Transactions
A session and a transaction solve related but different problems:
| Concept | Responsibility | Typical lifetime |
| --- | --- | --- |
| `AsyncSession` | Provides the ORM workspace: executes queries, tracks loaded and changed objects in its identity map, and flushes pending changes. It also coordinates access to a database connection. | One request, task, or explicit unit of work. |
| Transaction | Defines the atomic database boundary: all work inside it commits together on success or rolls back together on failure. | One complete operation that must have a single outcome. |
A transaction belongs to a session; it is not an alternative to one. The session is the interface used by application and data-access code, while the transaction determines when that work becomes permanent. A session may coordinate sequential transactions during its lifetime, although short-lived application scopes commonly use one session for one transaction.
Use a session without a helper-owned commit boundary for independent reads or lower-level functions that must participate in whatever transaction their caller controls:
```python
async with session_factory() as session:
item = await find_item(session, item_id)
```
Use an explicit transaction for writes, read-modify-write operations, or several statements that must succeed or fail as one unit:
```python
async with session_factory.begin() as session:
order = await create_order(session, order_data)
await reserve_inventory(session, order)
```
SQLAlchemy sessions use [autobegin](https://docs.sqlalchemy.org/en/21/orm/session_basics.html#auto-begin), so the first database operation normally starts a transaction even for a read. Therefore, “session-only” means that the surrounding helper owns only session lifetime and does not promise to commit; it does not mean that no database transaction exists. Closing such a session releases its resources and rolls back any unfinished transaction. An explicit `begin()` is valuable when application code must make the atomic boundary and commit ownership visible.
For most read-only operations, a session context is sufficient. Use an explicit transaction for reads when they need a defined consistency boundary, participate in a larger atomic operation, or use locking such as `SELECT ... FOR UPDATE`.
---
## Session Factory Mechanics
An `async_sessionmaker[AsyncSession]` is a reusable configuration object and callable session producer. It stores how sessions should be created, including the engine binding and options such as `expire_on_commit=False`. It is not itself a session, connection, or transaction, and calling it does not make a shared global `AsyncSession`.
Cache it by the application-owned engine so repeated composition calls return the same maker:
```python
from functools import cache
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
from .engine import dispose_engine
from .engine import get_engine
@cache
def get_session_factory(database_url: str) -> async_sessionmaker[AsyncSession]:
return async_sessionmaker(
bind=get_engine(database_url),
class_=AsyncSession,
expire_on_commit=False,
)
async def dispose_session_factory(database_url: str) -> None:
get_session_factory.cache_clear()
await dispose_engine(database_url)
```
`functools.cache` caches by argument equality and requires hashable arguments. The database URL is an explicit string key shared with the cached engine factory. The cache retains the returned maker until `get_session_factory.cache_clear()` runs. Cache the synchronous maker function, never an async function and never a produced `AsyncSession`.
Each call to `session_factory()` creates a distinct `AsyncSession`. The caller that invokes the factory owns that session lifetime and must close it, normally with `async with`:
```python
async with session_factory() as session:
...
```
The factory can be shared across requests and tasks. Sessions produced by it cannot be shared across concurrent tasks.
An `async_sessionmaker` has no connection pool or async `dispose()` method of its own. `dispose_session_factory()` means "invalidate the cached maker, then dispose its engine." Clearing the maker first ensures no subsequent composition call can retrieve a maker bound to the engine being shut down.
Use the helper when shutting down or replacing the database resources:
```python
await dispose_session_factory(database_url)
```
Otherwise, a later call can return a maker that still references the old engine object. This matters in lifespan tests, application restarts within one process, and test suites that replace engines.
---
## Optional Session Ownership
A small [`asynccontextmanager`](https://docs.python.org/3/library/contextlib.html#contextlib.asynccontextmanager) can make repository methods composable. It borrows an existing session when supplied; otherwise it creates and closes one from a supplied factory:
```python
from collections.abc import AsyncIterator
from contextlib import asynccontextmanager
@asynccontextmanager
async def session_scope(
*,
database_url: str,
session: AsyncSession | None = None,
) -> AsyncIterator[AsyncSession]:
if session is not None:
yield session
return
async with get_session_factory(database_url)() as owned_session:
yield owned_session
```
The branch is intentionally explicit. Python's [`nullcontext`](https://docs.python.org/3/library/contextlib.html#contextlib.nullcontext) can express the same borrow-or-own idea, but the branch keeps ownership and typing obvious.
This helper manages session lifetime only:
- It does not close, commit, or roll back a supplied session; the caller owns it.
- It closes a session that it creates. Closing releases resources and rolls back an unfinished transaction; it does not commit.
- It does not start a transaction. Put `session.begin()` at the use-case boundary.
- A supplied session wins; the cached factory is not resolved.
- Otherwise, `database_url` selects the cached factory returned by `get_session_factory()`.
Do not turn this into an implicit unit-of-work helper that sometimes commits. Whether work joins an existing transaction or creates a new one must remain visible to the caller.
---
## Optional Transaction Ownership
Use a separate context manager when a service or use-case function must support both a caller-owned transaction and a standalone transaction. A supplied session must already be inside a transaction; otherwise the helper creates a session and transaction together with `async_sessionmaker.begin()`:
```python
@asynccontextmanager
async def transaction_scope(
*,
database_url: str,
session: AsyncSession | None = None,
) -> AsyncIterator[AsyncSession]:
if session is not None:
if not session.in_transaction():
raise RuntimeError("A supplied session must have an active transaction")
yield session
return
session_factory = get_session_factory(database_url)
async with session_factory.begin() as owned_session:
yield owned_session
```
Here, `begin()` is intentionally called on the [`async_sessionmaker`](https://docs.sqlalchemy.org/en/20/orm/extensions/asyncio.html#sqlalchemy.ext.asyncio.async_sessionmaker.begin), not on an existing `AsyncSession`. The related APIs have different ownership semantics:
- `session_factory()` creates a session whose lifetime the surrounding code must manage; it does not commit automatically.
- `session_factory.begin()` creates a new session and transaction together, commits on successful exit or rolls back on exceptional exit, and then closes the session.
- `session.begin()` manages a transaction on an existing session but does not own or close that session.
The factory form is equivalent in ownership terms to creating a session and then entering that session's transaction:
```python
async with session_factory() as owned_session:
async with owned_session.begin():
yield owned_session
```
This helper makes transaction ownership follow the same explicit borrow-or-own mechanics as session ownership:
- A supplied session and its active transaction remain caller-owned. The helper does not commit, roll back, or close them.
- Without a supplied session, the helper owns the session and transaction. Successful exit commits; exceptional exit rolls back; either path closes the session.
- Use this helper only at a complete operation, service, or use-case boundary. A public CRUD function or repository method may be such a boundary when its optional-session contract explicitly states that omitting the session owns and commits one transaction. Never use it inside a lower-level session-required helper.
- Do not silently begin a transaction on a supplied session. That would make commit ownership depend on hidden helper behavior.
Callers that supply a session make their ownership visible with an outer transaction:
```python
async with session_factory() as session:
async with session.begin():
await run_use_case(..., session=session)
```
Standalone callers omit the session and let the use case own the complete unit of work:
```python
await run_use_case(...)
```
---
## Repository and Function Boundaries
Pass the database URL to repository constructors. The repository stores repeatable database configuration, not mutable session state, and `session_scope()` resolves the cached factory when a standalone operation needs a session:
```python
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
async def find_item(session: AsyncSession, item_id: int) -> Item | None:
statement = select(Item).where(Item.id == item_id)
return await session.scalar(statement)
class ItemRepository:
def __init__(self, database_url: str) -> None:
self.database_url = database_url
async def find(
self,
item_id: int,
*,
session: AsyncSession | None = None,
) -> Item | None:
async with session_scope(
database_url=self.database_url,
session=session,
) as active_session:
return await find_item(active_session, item_id)
```
This split gives each layer one job:
- The repository object identifies its database configuration and creates a session only for a standalone call.
- Standalone calls reuse the cached factory selected by database URL.
- A caller can pass a session to join an existing unit of work; the repository borrows it.
- The access function owns only the query and requires an existing `AsyncSession`.
- Application wiring supplies the production factory.
- Tests can use a test database URL or call `find_item()` with a transaction-scoped test session.
When several repository operations must share one transaction, pass the same session through each call. Put the transaction at the use-case boundary:
```python
async with session_factory() as session:
async with session.begin():
item = await repository.find(item_id, session=session)
await update_item(session, item, changes)
```
This preserves atomicity without making repository objects hold mutable `AsyncSession` instances across calls.
---
## Canonical FastAPI Dependency Pattern
```python
from collections.abc import AsyncIterator
from fastapi import Depends
from fastapi import Request
from sqlalchemy.ext.asyncio import AsyncSession
from sqlalchemy.ext.asyncio import async_sessionmaker
type SessionFactory = async_sessionmaker[AsyncSession]
def resolve_session_factory(request: Request) -> SessionFactory:
return get_session_factory(request.app.state.settings.database_url)
async def get_db_session(
session_factory: SessionFactory = Depends(resolve_session_factory),
) -> AsyncIterator[AsyncSession]:
async with session_factory() as session:
yield session
```
Route usage:
```python
from fastapi import APIRouter, Depends
from sqlalchemy.ext.asyncio import AsyncSession
from .session import get_db_session
router = APIRouter()
@router.post("/items")
async def create_item(session: AsyncSession = Depends(get_db_session)) -> dict:
async with session.begin():
# write operations here
...
return {"status": "ok"}
```
---
## Configuration Guidance
- `expire_on_commit=False` is commonly preferred in asyncio applications to reduce accidental post-commit reload behavior.
- `AsyncSession.refresh()` is preferred over broad expiration patterns when state refresh is needed.
- [`async_sessionmaker.begin()`](https://docs.sqlalchemy.org/en/21/orm/extensions/asyncio.html#sqlalchemy.ext.asyncio.async_sessionmaker.begin) is a concise option when one scope must create a session, begin a transaction, commit on success, roll back on failure, and close. Do not use it when borrowing a caller's session.
## SQLModel Alignment
- Use SQLModel as the default model and statement layer while keeping the same session ownership model: one `async_sessionmaker`, one `AsyncSession` per request/unit-of-work.
- SQLModel does not replace SQLAlchemy async lifecycle primitives; it provides model declaration, validation, and typing ergonomics on top of them.
- Do not mix ad hoc session construction with the canonical async dependency.
---
## Concurrency Rules
- One session per concurrent task.
- If work fans out into parallel tasks, each task receives its own AsyncSession.
- Pass sessions explicitly to service functions; avoid mutable global session state.
---
## Anti-Patterns
- A singleton/global AsyncSession reused across requests.
- Sharing one AsyncSession across parallel tasks.
- Passing an application-global AsyncSession to a repository constructor.
- Caching an `AsyncSession` instead of caching `async_sessionmaker`.
- Leaving a cached maker pointing at a disposed or replaced engine.
- Calling the session factory inside low-level access functions such as `find_item()`.
- Hidden session creation in lower access functions with no caller control.
- Closing or committing a session supplied by the caller.
- Starting a new transaction inside a helper that may receive a session already in a transaction.
- Silently starting or committing a transaction on a supplied session.
- Mixing commit/rollback ownership across layers without a declared boundary.
---
## Operational Checks
- Exactly one cached `async_sessionmaker` exists per application engine.
- Session factory caches are cleared before their engines are disposed or replaced.
- Request handlers receive sessions from one canonical dependency.
- No code path creates AsyncSession in module import side effects.
- Background jobs and API handlers each create task-local sessions.
---
## Testing Checks
- Repository constructors accept a test database URL without FastAPI startup.
- Session-taking access functions accept a transaction-scoped test session directly.
- Optional-session tests verify that borrowed sessions remain open and created sessions close.
- Optional-session tests verify that neither path commits implicitly.
- Optional-transaction tests verify supplied sessions require an active transaction and remain caller-owned.
- Optional-transaction tests verify owned transactions commit on success, roll back on failure, and close their sessions.
- Cache tests clear `get_session_factory` before and after replacing engines.
- Dependency override exists for the FastAPI session factory.
- Rollback behavior is verified for failed write units.
- Parallel-task tests verify no shared AsyncSession instances.
- Lifespan tests confirm session factory is initialized and teardown-safe.
@@ -0,0 +1,127 @@
# SQLModel-First Modeling and Async Boundaries
!!! info "Primary sources"
- [SQLModel documentation](https://sqlmodel.tiangolo.com/)
- [SQLModel features](https://sqlmodel.tiangolo.com/features/)
- [SQLModel advanced guide](https://sqlmodel.tiangolo.com/advanced/)
- [SQLModel FastAPI session dependency tutorial](https://sqlmodel.tiangolo.com/tutorial/fastapi/session-with-dependency/)
- [SQLModel release notes](https://sqlmodel.tiangolo.com/release-notes/)
- [SQLAlchemy asyncio extension](https://docs.sqlalchemy.org/en/21/orm/extensions/asyncio.html)
??? abstract "Decision metadata"
- Status: adopted
- Decision level: mandatory
- Applies to: api-runtime, workers, tests
- Last reviewed: 2026-07-26
---
## Purpose
Define SQLModel as the primary model layer for async FastAPI applications and explain how it composes with SQLAlchemy's async runtime.
SQLModel is designed for FastAPI, built on Pydantic and SQLAlchemy, and intended to minimize duplication while preserving the capabilities of both. Async engine, session, transaction, and loading behavior still follow SQLAlchemy's asyncio contract.
---
## Scope and Non-Goals
- In scope: table models, API data models, SQLAlchemy interoperability, async session usage, and exception criteria.
- Out of scope: replacing SQLAlchemy's async runtime primitives or claiming that synchronous tutorial examples are async patterns.
---
## Rules
- Default to SQLModel for new table models and API data models.
- Keep SQLAlchemy async primitives as the runtime base: `create_async_engine`, `async_sessionmaker`, and `AsyncSession`.
- Keep transaction and session ownership policies identical whether models are SQLAlchemy Declarative or SQLModel.
- Use SQLModel inheritance to share validated fields while keeping table, create, update, and public contracts distinct where their semantics differ.
- Use SQLAlchemy declarative models only for a concrete unsupported mapping or third-party constraint; document the reason.
- Use SQLAlchemy relationship loading options explicitly on async paths.
---
## Recommended Patterns
### Pattern A: Data model split for API boundaries
Use distinct models for persistence and external contracts.
```python
from sqlmodel import Field, SQLModel
class UserBase(SQLModel):
email: str
display_name: str
class User(UserBase, table=True):
id: int | None = Field(default=None, primary_key=True)
class UserCreate(UserBase):
pass
class UserRead(UserBase):
id: int
```
### Pattern B: Keep SQLModel models with the async runtime
```python
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker, create_async_engine
from sqlmodel import select
engine = create_async_engine(settings.database_url, pool_pre_ping=True)
session_factory = async_sessionmaker(engine, class_=AsyncSession, expire_on_commit=False)
async with session_factory() as session:
users = (await session.scalars(select(User))).all()
```
`sqlmodel.select()` keeps SQLModel's typing-oriented statement construction, while `AsyncSession.scalars()` and the surrounding lifecycle come from SQLAlchemy.
---
## Interoperability Notes
- A SQLModel table model is a SQLAlchemy model and can participate in SQLAlchemy relationships, statements, loader options, and sessions.
- A SQLModel model is also a Pydantic model; non-table models are useful for request and response contracts.
- SQLModel's official FastAPI dependency tutorial currently uses synchronous `Session`; translate the ownership pattern, not the concrete session type, for async applications.
- SQLModel's advanced guide still lists dedicated async documentation as future work, so use SQLAlchemy's asyncio documentation as the authority for runtime mechanics.
- Prefer one query style per module to reduce cognitive overhead.
- Keep loader strategies explicit in async paths to avoid implicit I/O surprises.
---
## Anti-Patterns
- Treating SQLModel as an alternative to SQLAlchemy rather than a layer built on it.
- Copying a synchronous `Session` example into an async request path.
- Constructing sessions in handlers instead of using the application session factory.
- Mixing multiple query/session idioms within the same module without clear conventions.
---
## Operational Checks
- New model modules are SQLModel-first; exceptions state the unsupported need or constraint.
- Session/transaction ownership remains consistent across both model styles.
- Table, create, update, and public models share fields intentionally without exposing persistence-only data.
---
## Testing Checks
- Module-level tests verify CRUD semantics for SQLModel models through `AsyncSession`.
- API tests verify response/request model behavior for SQLModel-based endpoints.
- Relationship tests verify async loader strategies do not depend on implicit I/O.
---
## Version Checks
- Verify installed SQLModel, SQLAlchemy, and Pydantic versions together when using newly added typing or ORM features.
@@ -0,0 +1,59 @@
# <Concept Title>
!!! info "Primary sources"
- Primary source: `<primary source URL>`
- Secondary source: `<secondary source URL>`
??? abstract "Decision metadata"
- Status: draft|adopted|deprecated
- Decision level: advisory|mandatory
- Applies to: api-runtime|workers|tests
- Last reviewed: YYYY-MM-DD
---
## Purpose
Describe what this concept governs and why it exists.
## Scope and Non-Goals
- In scope:
- Out of scope:
---
## Rules
- Rule 1
- Rule 2
---
## Recommended Pattern
```python
# minimal example
```
---
## Anti-Patterns
- Anti-pattern 1
- Anti-pattern 2
---
## Operational Checks
- Check 1
- Check 2
---
## Testing Checks
- Test 1
- Test 2
@@ -0,0 +1,105 @@
# Async Transaction Boundaries
!!! info "Primary sources"
- [SQLAlchemy transactions](https://docs.sqlalchemy.org/en/21/orm/session_transaction.html)
- [SQLAlchemy asyncio extension](https://docs.sqlalchemy.org/en/21/orm/extensions/asyncio.html)
- [SQLAlchemy connections](https://docs.sqlalchemy.org/en/21/core/connections.html)
??? abstract "Decision metadata"
- Status: adopted
- Decision level: mandatory
- Applies to: api-runtime, workers, tests
- Last reviewed: 2026-06-17
---
## Purpose
Define consistent transaction demarcation for async SQLAlchemy so write behavior is predictable, rollback semantics are clear, and concurrent request flows remain safe.
---
## Scope and Non-Goals
- In scope: transaction ownership, write/read policy, exception and rollback behavior, nested transaction guidance.
- Out of scope: business-domain validation rules and cross-service distributed transactions.
---
## Rules
- Every mutating use case must run inside an explicit transaction boundary.
- Prefer `async with session.begin():` for write units.
- Keep transaction ownership at a service, use-case, or explicitly documented complete-operation boundary, not deep in helper internals.
- An optional-session write may own one transaction when omitting the session clearly means standalone execution; a supplied session must remain caller-owned.
- Read paths should not auto-upgrade into hidden write behavior.
- On exception in a transaction block, rely on rollback semantics and propagate or map exceptions intentionally.
---
## Recommended Patterns
### Pattern A: Single write unit
```python
async def create_order(session: AsyncSession, payload: OrderIn) -> Order:
async with session.begin():
order = Order(...)
session.add(order)
# additional writes...
return order
```
### Pattern B: Explicit read flow
```python
async def get_order(session: AsyncSession, order_id: UUID) -> Order | None:
stmt = select(Order).where(Order.id == order_id)
return await session.scalar(stmt)
```
### Pattern C: Nested transaction (only when required)
```python
async with session.begin():
# outer transaction
async with session.begin_nested():
# savepoint-scoped operation
...
```
Use nested transactions only when partial failure semantics are explicitly required.
---
## Exception and Rollback Policy
- Write block fails: transaction context rolls back.
- Caller decides whether to translate exception (for example to domain/API errors).
- Do not swallow DB exceptions silently; map or re-raise intentionally.
---
## Anti-Patterns
- Multiple commits scattered across one logical use case.
- Helper functions that commit or roll back without an explicit ownership contract.
- Mixing implicit and explicit transaction styles in confusing ways.
- Using savepoints as a default pattern rather than a targeted tool.
---
## Operational Checks
- All mutating services and complete operations declare one clear transaction boundary.
- No repository or helper performs hidden direct commit calls; standalone ownership is expressed through a documented transaction scope.
- Transaction style is consistent across handlers and workers.
---
## Testing Checks
- Success path test verifies expected durable writes.
- Failure path test verifies rollback behavior.
- Tests cover concurrency-sensitive write flows.
- Savepoint usage (if present) has dedicated behavior tests.
+138
View File
@@ -0,0 +1,138 @@
---
name: copilot-customization
description: 'Plan, create, review, and debug GitHub Copilot and VS Code agent customizations, including instructions, prompt files, skills, custom agents, hooks, MCP servers, and repo-specific personal-mcp skill integration.'
x-personal-mcp:
id: copilot-customization
version: 1.0.0
tags:
- copilot
- vscode
- customization
- instructions
- prompts
- agent-skills
- custom-agents
- hooks
- mcp
- personal-mcp
- skills
capabilities:
- resource://skills/copilot-customization/document
---
# Copilot Customization
Use this skill when a task is about changing how GitHub Copilot or VS Code agents behave through customization files or MCP-backed skill resources.
## When to Use
- Creating or updating:
- `.github/copilot-instructions.md`
- `AGENTS.md`
- `CLAUDE.md`
- `*.instructions.md` files
- `*.prompt.md` files
- Creating prompt files, custom agents, hooks, or Agent Skills.
- Deciding whether behavior belongs in instructions, prompts, skills, agents, hooks, MCP servers, or agent plugins.
- Debugging why a customization is not discovered, loaded, or invoked.
- Adding a new documentation-backed skill to this `personal-mcp` repository.
## Start With The Decision
Choose the smallest customization that matches the desired behavior:
1. Use always-on instructions for project-wide coding standards, architecture decisions, security rules, and documentation standards that should apply to most requests.
2. Use file-based instructions for conventions that only apply to matching files, folders, languages, frameworks, or documentation types.
3. Use prompt files for reusable slash commands that package a single recurring prompt.
4. Use Agent Skills for portable, task-specific workflows that may include references, scripts, examples, or templates.
5. Use custom agents for specialized personas, tool restrictions, model choices, or role-specific workflows.
6. Use hooks when a deterministic lifecycle action must enforce a policy, run a command, or block unsafe behavior.
7. Use MCP servers when the agent needs live external tools, structured resources, or discoverable data beyond static instruction files.
8. Use agent plugins when several related customizations should ship together as an installable package.
If the request is ambiguous, ask only for the missing axis that changes the file type: scope, trigger, expected output, required tools, or whether it must be portable beyond VS Code.
## Research Map
Use [VS Code customization references](./references/vscode-customization.md) for official-source details about locations, frontmatter, discovery behavior, priority, and troubleshooting.
## Repo Shim Pattern For Personal MCP
Use a shim when you want another repository to consume this server as a preference and documentation source without duplicating methodology content.
### What the shim does
1. Tells the agent when to consult this MCP server.
2. Tells the agent how to retrieve relevant guidance.
3. Keeps repo-local behavior thin while canonical guidance stays in Personal MCP resources.
### Shim formats
Use either:
1. A repo instruction file (`*.instructions.md`) for always-on or file-scoped behavior.
2. A prompt file (`*.prompt.md`) for explicit, on-demand guidance retrieval.
### Retrieval strategies
Choose one of these patterns:
1. Direct URI strategy:
- Reference known resources directly, such as:
- `resource://catalog/skills_index`
- `resource://catalog/skills/{skill_id}`
- `resource://skills/<skill-id>/document`
- `resource://skills/<skill-id>/references/<ref-id>`
2. Discovery-first strategy:
- Start at catalog discovery (`resource://catalog/skills_index`), select the best skill match, then load the skill document and only the minimal references needed.
### Authoring guidance for shims
1. Keep shim content short and procedural; avoid copying large guidance blocks from Personal MCP.
2. State trigger conditions clearly (for example: "when creating a new skill" or "when editing docs contracts").
3. Specify whether to use direct URIs or discovery for that repo's common workflows.
4. Prefer loading only the most relevant skill document first; expand to references only when needed.
5. For stable repeated workflows, use explicit URIs. For broader or ambiguous requests, use discovery-first.
### Minimal shim examples
Instruction-style shim intent:
1. "For markdown edits (`applyTo: '**/*.md'`), load `resource://skills/zensical-docs/document` and apply Zensical-native documentation conventions unless they conflict with expected MkDocs compatibility."
Prompt-style shim intent:
1. "For docs authoring tasks, consult `resource://skills/zensical-docs/document`, summarize the relevant authoring constraints, then propose the smallest markdown change for this repository."
### Validation for shim implementation
1. Confirm the shim triggers in expected contexts.
2. Confirm resource loading path is unambiguous (direct URI or discovery).
3. Confirm repo-local customization remains thin and references Personal MCP as source of truth.
## Workspace Customization Workflow
1. Identify the customization primitive and scope.
2. Check existing files before creating a new one.
3. Keep the description or frontmatter trigger specific and keyword-rich.
4. Keep instructions concise, focused, and self-contained.
5. Add examples only when they clarify a non-obvious convention.
6. For `*.instructions.md`, set `applyTo` only when automatic file matching is intended.
7. For skills, make the folder name match the `name` field exactly and reference any extra files from `SKILL.md` with relative links.
8. Validate placement, YAML frontmatter, discovery settings, and whether the customization should be workspace or user scoped.
## Quality Checks
Before finishing:
1. Confirm the customization file is in a supported location for its intended scope.
2. Confirm required frontmatter fields are present and valid.
3. Confirm names match directory names where VS Code requires it.
4. Confirm descriptions include the phrases users are likely to ask for.
5. Confirm extra skill resources are linked from `SKILL.md`.
6. Confirm repo skill metadata exposes the correct `resource://skills/<skill-id>/document` capability.
7. State any remaining ambiguity or user choice, such as personal vs workspace scope.
## Output Contract
Return the concrete customization created or changed, where it lives, how to invoke or trigger it, and any validation performed.
@@ -0,0 +1,83 @@
# VS Code Copilot Customization References
Use these notes as a source map before creating or debugging Copilot customizations.
## Official Sources
!!! info "Official sources"
- [Customization overview](https://code.visualstudio.com/docs/copilot/customization/overview)
- [Custom instructions](https://code.visualstudio.com/docs/copilot/customization/custom-instructions)
- [Agent skills](https://code.visualstudio.com/docs/copilot/customization/agent-skills)
- [Prompt files](https://code.visualstudio.com/docs/copilot/customization/prompt-files)
- [Custom agents](https://code.visualstudio.com/docs/copilot/customization/custom-agents)
- [MCP servers](https://code.visualstudio.com/docs/copilot/customization/mcp-servers)
- [Hooks](https://code.visualstudio.com/docs/copilot/customization/hooks)
## Customization Types
- Instructions describe standards and conventions that apply to every request or to matching files.
- Prompt files save reusable slash-command prompts for recurring tasks.
- Agent Skills package reusable workflows, scripts, examples, and resources that load on demand.
- Custom agents define specialized personas, tool access, model choices, and role-specific workflows.
- MCP servers connect the agent to external tools, resources, and data.
- Hooks run deterministic actions at defined lifecycle points.
- Agent plugins bundle related customization types into an installable package.
## Instructions
Use `.github/copilot-instructions.md` for workspace-wide rules that should be included in every chat request. Use `AGENTS.md` when multiple agents should share the same repository guidance, or when nested agent guidance is useful. Use `CLAUDE.md` for Claude-compatible instruction sharing.
Use `.github/instructions/**/*.instructions.md` for file-based or task-specific rules. Supported frontmatter fields include:
```yaml
---
name: Documentation Standards
description: Rules for documentation writing tasks
applyTo: '**/*.md'
---
```
`applyTo` is a workspace-relative glob. If it is omitted, the instruction file can still be manually attached but does not automatically apply by file match.
## Agent Skills
Skills live in a directory whose name must match the `name` field in `SKILL.md`. VS Code supports project skills in `.github/skills/`, `.claude/skills/`, and `.agents/skills/`, and personal skills under user-level skill folders.
Required skill frontmatter:
```yaml
---
name: skill-name
description: Description of what the skill does and when to use it.
---
```
Useful optional fields:
- `argument-hint`: shown when invoking the skill as a slash command.
- `user-invocable`: controls whether it appears in the slash menu.
- `disable-model-invocation`: controls whether the model can auto-load it.
- `context`: can use `fork` for a separate subagent context when supported.
Skills load progressively: discovery reads frontmatter, instruction loading reads `SKILL.md`, and extra resources load only when linked from the skill document.
## Priority And Discovery
When multiple instruction sources apply, personal instructions have higher priority than repository instructions, and repository instructions have higher priority than organization instructions. If multiple instruction files exist, VS Code combines them; do not rely on ordering between instruction files.
For monorepos, `chat.useCustomizationsInParentRepositories` can enable discovery from a trusted parent repository root. Skill locations can also be configured with `chat.agentSkillsLocations`, and instruction locations with `chat.instructionsFilesLocations`.
## Troubleshooting
If a customization is not applied:
1. Confirm the file is in a supported location.
2. Confirm frontmatter is valid YAML.
3. Confirm skill `name` matches the parent directory.
4. Confirm `applyTo` matches the file path when using `*.instructions.md`.
5. Confirm relevant settings are enabled, such as instruction inclusion, referenced instruction inclusion, or AGENTS/CLAUDE support.
6. Use the Chat customization diagnostics view or Agent Debug Logs to inspect what VS Code loaded.
## Writing Effective Instructions
Keep instructions short, self-contained, and focused on non-obvious rules. Include the reason for a rule when it helps with edge cases. Prefer concrete examples over abstract preferences. Split unrelated rules into separate targeted files when they have different triggers.
+435
View File
@@ -0,0 +1,435 @@
---
name: fastapi-uv-docker
description: 'Audit and migrate an existing Python project to best practices for a cloud-native ASGI FastAPI app managed with uv and run with uvicorn in Docker. Use when: conforming a project to production standards, setting up src layout, configuring pyproject.toml, writing multi-stage Dockerfiles, wiring lifespan and settings, adding health endpoints, enforcing non-root container user, migrating from requirements.txt to uv.'
x-personal-mcp:
id: fastapi-uv-docker
version: 1.0.0
tags:
- fastapi
- uv
- uvicorn
- docker
- architecture
capabilities:
- resource://skills/fastapi-uv-docker/document
---
# FastAPI Project Best Practices
Bring an existing Python project into full conformance with cloud-native best practices: **FastAPI + uv + uvicorn + Docker**. This skill audits the current state, produces a gap list, and walks through each conformance area in priority order.
## When to Use
- Migrating an existing app from `pip`/`requirements.txt` to `uv`
- Conforming a FastAPI app to src-layout and app-factory structure
- Writing a production-grade Dockerfile with multi-stage builds
- Wiring Pydantic settings from environment variables
- Adding health endpoints and graceful lifespan shutdown
- Enforcing non-root container user and proper signal handling
- Setting up `docker-compose.yml` for local dev and CI
## Progressive Loading References
Load these references only when needed:
- FastAPI patterns and app structure: [FastAPI best practices](./references/fastapi-best-practices.md)
- uv project layout and dependency management: [uv project layout](./references/uv-project-layout.md)
- uvicorn CLI settings reference: [uvicorn settings](./references/uvicorn-settings.md)
- Docker and cloud-native patterns: [Docker cloud-native patterns](./references/docker-cloud-native.md)
---
## Procedure
### Step 0: Audit the Project
Before making changes, map the current state across six areas. Produce a short gap list for each.
| Area | Check |
|------|-------|
| **Project manager** | Is `uv` used? Is `pyproject.toml` present? Is `uv.lock` committed? |
| **Package layout** | Is a `src/` layout used? Is the package installable? |
| **App structure** | Is `create_app()` factory used? Is lifespan wired? Are routers registered via `APIRouter`? |
| **Configuration** | Are settings loaded from env via Pydantic `BaseSettings`? Are secrets out of code? |
| **Container** | Is there a `Dockerfile`? Multi-stage? Non-root user? `.dockerignore` present? |
| **Cloud-native** | Is there a `/healthz` endpoint? Graceful shutdown? Structured logs? |
Load the [FastAPI best practices reference](./references/fastapi-best-practices.md) for structure rules.
Load the [uv project layout reference](./references/uv-project-layout.md) for uv migration rules.
Load the [uvicorn settings reference](./references/uvicorn-settings.md) for uvicorn CLI reference.
Completion check: You can name every gap before touching any file.
---
### Step 1: Migrate to uv and Establish pyproject.toml
**If the project uses `requirements.txt` / `setup.py` / `setup.cfg` / `pip`:**
1. Initialize uv if not present: `uv init` (or `uv init --lib` for importable package).
2. Import existing requirements: `uv add -r requirements.txt`.
3. Remove `requirements.txt`, `setup.py`, `setup.cfg`, and any `Pipfile`.
4. Ensure `.python-version` is committed with the target Python version.
5. Commit `uv.lock` — it is the source of truth for reproducible installs.
6. Add `.venv` to `.gitignore` and `.dockerignore`.
**Canonical `pyproject.toml` shape:**
```toml
[project]
name = "my-app"
version = "0.1.0"
requires-python = ">=3.12"
dependencies = [
"fastapi[standard]>=0.115",
"uvicorn[standard]>=0.34",
"pydantic-settings>=2.0",
]
[project.scripts]
my-app = "my_app.main:main"
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[tool.uv]
dev-dependencies = [
"pytest>=8",
"httpx>=0.27",
"pytest-asyncio>=0.24",
]
[tool.hatch.build.targets.wheel]
packages = ["src/my_app"]
```
**Commands:**
```bash
uv add fastapi[standard] uvicorn[standard] pydantic-settings
uv add --dev pytest httpx pytest-asyncio
uv sync # creates .venv and installs all deps
uv run pytest # run tests via uv
```
Completion check: `uv run python -c "import my_app"` succeeds.
---
### Step 2: Establish src Layout
Move the package under `src/` to prevent import confusion between installed and local code.
```
.
├── pyproject.toml
├── uv.lock
├── .python-version
├── .env.example
├── README.md
├── src/
│ └── my_app/
│ ├── __init__.py
│ ├── main.py # create_app() + entry point
│ ├── config.py # Pydantic BaseSettings
│ ├── lifespan.py # @asynccontextmanager lifespan
│ ├── api/
│ │ ├── __init__.py
│ │ ├── health.py # GET /healthz
│ │ └── v1/
│ │ └── __init__.py
│ └── services/
│ └── __init__.py
├── tests/
│ ├── conftest.py
│ └── test_health.py
├── Dockerfile
├── .dockerignore
└── docker-compose.yml
```
Completion check: `uv run python -m my_app` starts the server.
---
### Step 3: Wire the FastAPI App Factory
Load the [FastAPI best practices reference](./references/fastapi-best-practices.md) for the full patterns. Key rules:
**`src/my_app/main.py`:**
```python
from contextlib import asynccontextmanager
from fastapi import FastAPI
from my_app.api.health import router as health_router
from my_app.config import Settings
def create_app(settings: Settings | None = None) -> FastAPI:
if settings is None:
settings = Settings()
@asynccontextmanager
async def lifespan(app: FastAPI):
# startup: open DB pools, load models, etc.
app.state.settings = settings
yield
# shutdown: close connections
app = FastAPI(
title="My App",
lifespan=lifespan,
docs_url="/docs" if settings.debug else None,
redoc_url=None,
)
app.include_router(health_router)
return app
app = create_app()
```
**`src/my_app/api/health.py`:**
```python
from fastapi import APIRouter
from fastapi.responses import JSONResponse
router = APIRouter(tags=["ops"])
@router.get("/healthz", include_in_schema=False)
async def health() -> JSONResponse:
return JSONResponse({"status": "ok"})
```
**`src/my_app/config.py`:**
```python
from pydantic_settings import BaseSettings, SettingsConfigDict
class Settings(BaseSettings):
model_config = SettingsConfigDict(env_file=".env", extra="ignore")
debug: bool = False
host: str = "0.0.0.0"
port: int = 8000
log_level: str = "info"
```
Completion check: `uv run uvicorn my_app.main:app --reload` starts with no import errors.
---
### Step 4: uvicorn Production Configuration
Load the [uvicorn settings reference](./references/uvicorn-settings.md) for the full settings reference.
**Never** configure uvicorn inside application code. Pass all settings via CLI or environment variables (`UVICORN_*` prefix).
```bash
# Development — reload only; never in production
uv run uvicorn my_app.main:app \
--reload \
--host 127.0.0.1 \
--port 8000 \
--log-level debug
# Production — single process (orchestrator handles replication)
uv run uvicorn my_app.main:app \
--host 0.0.0.0 \
--port 8000 \
--workers 1 \
--loop auto \
--http auto \
--log-level info \
--proxy-headers \
--forwarded-allow-ips '*' \
--timeout-graceful-shutdown 30
```
**Key flags for production:**
| Flag | Value | Reason |
|------|-------|--------|
| `--host 0.0.0.0` | Required in containers | Bind to all interfaces, not just loopback |
| `--workers 1` | Kubernetes/Cloud Run | Orchestrator replicates containers |
| `--loop auto` | Default | Uses `uvloop` when available (install `uvicorn[standard]`) |
| `--http auto` | Default | Uses `httptools` when available |
| `--proxy-headers` | Behind any proxy | Trusts `X-Forwarded-For`, `X-Forwarded-Proto` |
| `--forwarded-allow-ips '*'` | Container/K8s | Trusts proxy headers from all IPs (safe when inside a trusted network) |
| `--timeout-graceful-shutdown 30` | Prod | Seconds to wait before force-closing requests on shutdown |
| `--no-access-log` | High-traffic prod | Disable per-request logs if using structured app-level logging |
**Environment variable equivalents** (useful in `docker-compose.yml` / K8s manifests):
```bash
UVICORN_HOST=0.0.0.0
UVICORN_PORT=8000
UVICORN_WORKERS=1
UVICORN_LOG_LEVEL=info
UVICORN_PROXY_HEADERS=true
UVICORN_FORWARDED_ALLOW_IPS=*
```
**`--reload` and `--workers` are mutually exclusive** — never combine them.
**When to use `--workers > 1`:** Only for Docker Compose on a single host where orchestrator-level replication is not available. For Kubernetes / Cloud Run / Fargate: always `--workers 1` and scale via replicas — this gives predictable per-container memory and cleaner crash isolation.
Completion check: `curl http://localhost:8000/healthz` returns `{"status":"ok"}`.
---
### Step 5: Write the Dockerfile
Load the [Docker cloud-native patterns reference](./references/docker-cloud-native.md) for the full template and cloud-native rules. Key requirements:
- Multi-stage build: `builder` stage installs deps; `runtime` stage is slim.
- Pin uv version (copy from official image, not `latest`).
- Use `uv sync --locked --no-editable` to install into `.venv`.
- Set `ENV PATH="/app/.venv/bin:$PATH"` — do **not** use `uv run` in production `CMD`.
- Run as non-root user.
- Use `CMD` exec form, never shell form.
- Add `HEALTHCHECK`.
**Canonical Dockerfile:**
```dockerfile
# syntax=docker/dockerfile:1
# ── builder ──────────────────────────────────────────────────────────────────
FROM python:3.12-slim-bookworm AS builder
# Pin uv version for reproducibility
COPY --from=ghcr.io/astral-sh/uv:0.5.27 /uv /uvx /bin/
WORKDIR /app
# Install dependencies first (cache layer)
RUN --mount=type=cache,target=/root/.cache/uv \
--mount=type=bind,source=uv.lock,target=uv.lock \
--mount=type=bind,source=pyproject.toml,target=pyproject.toml \
uv sync --locked --no-install-project --no-editable
# Copy source and install project
COPY . /app
RUN --mount=type=cache,target=/root/.cache/uv \
uv sync --locked --no-editable
# ── runtime ───────────────────────────────────────────────────────────────────
FROM python:3.12-slim-bookworm AS runtime
# Non-root user
RUN groupadd --system --gid 1001 appgroup && \
useradd --system --uid 1001 --gid appgroup --no-log-init appuser
WORKDIR /app
# Copy only the virtual environment (not source code)
COPY --from=builder --chown=appuser:appgroup /app/.venv /app/.venv
# Copy application source
COPY --chown=appuser:appgroup src/ /app/src/
ENV PATH="/app/.venv/bin:$PATH" \
PYTHONUNBUFFERED=1 \
PYTHONDONTWRITEBYTECODE=1
USER appuser
EXPOSE 8000
HEALTHCHECK --interval=30s --timeout=10s --start-period=10s --retries=3 \
CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:8000/healthz')"
# Exec form — required for graceful shutdown / lifespan events
CMD ["uvicorn", "my_app.main:app", \
"--host", "0.0.0.0", \
"--port", "8000", \
"--workers", "1", \
"--proxy-headers"]
```
Completion check: `docker build -t my-app . && docker run --rm -p 8000:8000 my-app` serves `/healthz`.
---
### Step 6: Write .dockerignore and docker-compose.yml
**`.dockerignore`:**
```
.venv/
.git/
.gitignore
.env
*.pyc
__pycache__/
.pytest_cache/
dist/
*.egg-info/
README.md
```
**`docker-compose.yml` (local dev):**
```yaml
services:
app:
build: .
ports:
- "8000:8000"
environment:
- DEBUG=true
- LOG_LEVEL=debug
env_file:
- .env
develop:
watch:
- action: sync
path: ./src
target: /app/src
- action: rebuild
path: ./pyproject.toml
- action: rebuild
path: ./uv.lock
```
Completion check: `docker compose up` starts the app; `docker compose watch` enables hot reload.
---
### Step 7: Cloud-Native Checklist
Run this final checklist before shipping:
- [ ] `GET /healthz` returns 200 with no auth required
- [ ] App reads all config from environment (`Settings` with no hardcoded values)
- [ ] `.env` is in `.gitignore` and `.dockerignore`; `.env.example` is committed
- [ ] `uv.lock` is committed
- [ ] `.python-version` is committed
- [ ] Dockerfile uses non-root user (`USER appuser`)
- [ ] `CMD` uses exec form (list, not string)
- [ ] `--proxy-headers` is set in the uvicorn `CMD` if behind a proxy
- [ ] `PYTHONUNBUFFERED=1` is set (logs flush immediately)
- [ ] `EXPOSE` declares the correct port
- [ ] `HEALTHCHECK` is defined
- [ ] Multi-stage build — final image contains no build tools or uv binary
- [ ] `.venv` is in `.dockerignore`
- [ ] No secrets hardcoded in `Dockerfile`, `pyproject.toml`, or source
- [ ] `uv sync --locked` in CI (fail if lock is stale)
- [ ] Tests pass via `uv run pytest`
---
## Anti-patterns to Fix
| Anti-pattern | Correct approach |
|---|---|
| `requirements.txt` | Use `uv` with `pyproject.toml` and `uv.lock` |
| `pip install` in Dockerfile | `uv sync --locked` |
| `ENV SECRET_KEY=abc123` in Dockerfile | Inject at runtime via env; never bake secrets |
| Shell form `CMD uvicorn ...` | Exec form `CMD ["uvicorn", ...]` |
| `FROM tiangolo/uvicorn-gunicorn-fastapi` | Build from scratch with `python:3.x-slim` |
| Multiple workers inside K8s container | `--workers 1`; scale via replicas |
| Running as root in container | `USER appuser` with explicit UID 1001 |
| Startup/shutdown in `@app.on_event` | Use `@asynccontextmanager` lifespan |
| Config loaded from `.env` directly in code | Pydantic `BaseSettings` with `env_file` |
@@ -0,0 +1,349 @@
# Docker and Cloud-Native Patterns
!!! info "Primary sources"
- [Docker build best practices](https://docs.docker.com/build/building/best-practices/)
- [uv Docker integration](https://docs.astral.sh/uv/guides/integration/docker/)
- [FastAPI Docker deployment](https://fastapi.tiangolo.com/deployment/docker/)
- [uvicorn deployment](https://uvicorn.dev/deployment/)
---
## Canonical Multi-Stage Dockerfile (uv + FastAPI + uvicorn)
```dockerfile
# syntax=docker/dockerfile:1
# ── builder stage ─────────────────────────────────────────────────────────────
FROM python:3.12-slim-bookworm AS builder
# Pin uv version — never use :latest in production
COPY --from=ghcr.io/astral-sh/uv:0.5.27 /uv /uvx /bin/
WORKDIR /app
# Layer: install dependencies only (cached until pyproject.toml or uv.lock changes)
RUN --mount=type=cache,target=/root/.cache/uv \
--mount=type=bind,source=uv.lock,target=uv.lock \
--mount=type=bind,source=pyproject.toml,target=pyproject.toml \
uv sync --locked --no-install-project --no-dev --no-editable
# Layer: copy source and install project
COPY src/ /app/src/
COPY pyproject.toml uv.lock /app/
RUN --mount=type=cache,target=/root/.cache/uv \
uv sync --locked --no-dev --no-editable
# ── runtime stage ─────────────────────────────────────────────────────────────
FROM python:3.12-slim-bookworm AS runtime
# Non-root user with explicit UID/GID
RUN groupadd --system --gid 1001 appgroup && \
useradd --system --uid 1001 --gid appgroup --no-log-init --home /app appuser
WORKDIR /app
# Copy only the venv (uv binary stays in builder)
COPY --from=builder --chown=appuser:appgroup /app/.venv /app/.venv
# Copy app source
COPY --from=builder --chown=appuser:appgroup /app/src /app/src
# Activate the venv via PATH — do not rely on `uv run` at runtime
ENV PATH="/app/.venv/bin:$PATH" \
PYTHONUNBUFFERED=1 \
PYTHONDONTWRITEBYTECODE=1 \
UV_PYTHON_DOWNLOADS=0
USER appuser
EXPOSE 8000
# Liveness probe — no extra tools needed; uses stdlib urllib
HEALTHCHECK --interval=30s --timeout=10s --start-period=15s --retries=3 \
CMD python -c \
"import urllib.request; urllib.request.urlopen('http://localhost:8000/healthz')" \
|| exit 1
# Exec form (NOT shell form) — required for SIGTERM → graceful shutdown
CMD ["uvicorn", "my_app.main:app", \
"--host", "0.0.0.0", \
"--port", "8000", \
"--workers", "1", \
"--proxy-headers", \
"--forwarded-allow-ips", "*"]
```
---
## .dockerignore
```
# Python artifacts
.venv/
__pycache__/
*.pyc
*.pyo
.pytest_cache/
.mypy_cache/
.ruff_cache/
dist/
*.egg-info/
# Environment and secrets
.env
.env.*
!.env.example
# Git
.git/
.gitignore
# Docs and local tooling
README.md
docs/
*.md
```
**Always add `.venv/` to `.dockerignore`** — it is platform-specific and will cause subtle failures if copied into the image.
---
## docker-compose.yml (Local Development)
```yaml
services:
app:
build:
context: .
target: runtime # build only up to the runtime stage
ports:
- "8000:8000"
environment:
DEBUG: "true"
LOG_LEVEL: "debug"
env_file:
- .env
restart: unless-stopped
# Docker Compose Watch — hot reload without rebuilding image
develop:
watch:
- action: sync
path: ./src
target: /app/src
ignore:
- .venv/
- __pycache__/
- action: rebuild
path: ./pyproject.toml
- action: rebuild
path: ./uv.lock
```
Run with:
```bash
docker compose up # start normally
docker compose watch # start with live sync
docker compose up --build # force rebuild
```
---
## Key Dockerfile Rules
### Multi-Stage Builds
- **Stage 1 (`builder`)**: Has uv, build tools, compiles `.pyc` files if needed.
- **Stage 2 (`runtime`)**: Minimal; contains only the venv and source. No uv, no build tools.
- Final image size: typically 150200 MB for a FastAPI app (vs 500+ MB with a single stage).
### Layer Caching Strategy
Copy files in order of change frequency (least → most):
```
1. uv.lock + pyproject.toml → install deps (cached for days)
2. Source code → install project (invalidated on every code change)
```
This means dependency installation is only re-run when `uv.lock` or `pyproject.toml` changes.
### CMD Exec Form (Critical)
```dockerfile
# ✅ Exec form — process receives SIGTERM directly → graceful shutdown
CMD ["uvicorn", "my_app.main:app", "--host", "0.0.0.0", "--port", "8000"]
# ❌ Shell form — process is a child of /bin/sh → SIGTERM goes to shell, not uvicorn
CMD uvicorn my_app.main:app --host 0.0.0.0 --port 8000
```
Shell form breaks graceful shutdown and FastAPI lifespan shutdown events.
### Non-Root User
```dockerfile
RUN groupadd --system --gid 1001 appgroup && \
useradd --system --uid 1001 --gid appgroup --no-log-init --home /app appuser
USER appuser
```
- Use `--system` for service accounts (no shell, no home by default).
- Use `--no-log-init` to avoid `/var/log/faillog` disk exhaustion (Go runtime bug in older kernels).
- Use explicit UID/GID (1001) — deterministic, scanners can reason about it.
- Never run as UID 0 (root) in production.
### HEALTHCHECK
```dockerfile
HEALTHCHECK --interval=30s --timeout=10s --start-period=15s --retries=3 \
CMD python -c \
"import urllib.request; urllib.request.urlopen('http://localhost:8000/healthz')" \
|| exit 1
```
- `--start-period`: grace period on startup before health checks begin.
- `--retries`: mark as unhealthy only after N consecutive failures.
- Uses stdlib `urllib` — no extra tools needed, not even `curl`.
---
## Cloud-Native Twelve-Factor Principles Applied
### Factor III — Config (Environment Variables)
- All config comes from environment variables, never from code.
- `pydantic-settings` reads from env automatically.
- Provide `.env.example` with documentation; never commit `.env`.
```bash
# Runtime injection
docker run -e DATABASE_URL="postgresql://..." -e SECRET_KEY="..." my-app
# or via env_file in compose / Kubernetes Secret
```
### Factor XI — Logs (Treat as Event Streams)
- Set `PYTHONUNBUFFERED=1` — logs are flushed immediately to stdout/stderr.
- Never write logs to files inside the container.
- Configure uvicorn to log JSON in production:
```python
# In create_app() lifespan or a logging setup module
import logging
import json
class JSONFormatter(logging.Formatter):
def format(self, record: logging.LogRecord) -> str:
return json.dumps({
"level": record.levelname,
"message": record.getMessage(),
"logger": record.name,
})
```
Or use `structlog` with the JSON renderer for production.
### Factor IX — Disposability (Fast Startup, Graceful Shutdown)
- FastAPI lifespan handles startup/shutdown.
- Uvicorn forwards `SIGTERM` to the Python process when using exec form `CMD`.
- The process should be fully ready to serve requests within 10 seconds.
- Kubernetes `terminationGracePeriodSeconds` should be >= your timeout.
---
## Kubernetes Readiness / Liveness Probes
Expose two endpoints:
```python
@router.get("/healthz", include_in_schema=False) # liveness
async def health():
return {"status": "ok"}
@router.get("/readyz", include_in_schema=False) # readiness
async def readiness(request: Request):
# Check DB, cache, etc.
return {"status": "ready"}
```
Kubernetes manifest snippet:
```yaml
livenessProbe:
httpGet:
path: /healthz
port: 8000
initialDelaySeconds: 10
periodSeconds: 30
readinessProbe:
httpGet:
path: /readyz
port: 8000
initialDelaySeconds: 5
periodSeconds: 10
```
---
## Scaling: Workers vs. Replicas
| Deployment | Workers setting | Reasoning |
|---|---|---|
| Kubernetes / Cloud Run | `--workers 1` | Orchestrator handles replication; 1 process per container for predictable memory |
| Docker Compose (single host) | `--workers 4` | No external replication; use CPU cores |
| Local dev | `--reload` (single process) | Hot reload only works with single process |
**Never use `--reload` in production.**
---
## Base Image Selection
| Use case | Recommended base |
|---|---|
| Standard production | `python:3.12-slim-bookworm` |
| Smallest possible image | `python:3.12-alpine3.20` (musl; watch for C extension compat) |
| uv-managed Python | `ghcr.io/astral-sh/uv:python3.12-bookworm-slim` |
| Security-hardened | `cgr.dev/chainguard/python:latest` (distroless) |
**Do not use:**
- `python:latest` — unpinned, breaks reproducibility
- `tiangolo/uvicorn-gunicorn-fastapi` — deprecated by FastAPI team
- Full `python:3.x` (non-slim) — 300+ MB unnecessary overhead
---
## Build Optimizations
```bash
# Enable BuildKit (default in Docker >= 23)
export DOCKER_BUILDKIT=1
# Build with cache mount (fastest for repeated local builds)
docker build -t my-app .
# CI: force fresh base image + no layer cache
docker build --pull --no-cache -t my-app .
# Multi-platform build (for ARM deployment from x86 CI)
docker buildx build --platform linux/amd64,linux/arm64 -t my-app .
```
---
## Security Hardening Checklist
- [ ] Non-root user with explicit UID/GID
- [ ] Read-only filesystem where possible (`--read-only` docker run flag or `securityContext.readOnlyRootFilesystem: true` in K8s)
- [ ] No secrets in `ENV` Dockerfile instructions
- [ ] Minimal base image (slim/alpine)
- [ ] Multi-stage build (no build tools in runtime image)
- [ ] Pin base image and uv version (not `:latest`)
- [ ] `.dockerignore` excludes `.env`, `.git`, `.venv`
- [ ] `EXPOSE` only the port actually used
- [ ] Regular base image updates in CI (`docker build --pull`)
@@ -0,0 +1,241 @@
# FastAPI Best Practices
!!! info "Primary sources"
- [FastAPI deployment](https://fastapi.tiangolo.com/deployment/)
- [FastAPI lifespan events](https://fastapi.tiangolo.com/advanced/events/)
---
## App Factory Pattern
Always use `create_app()` — it makes the app testable (inject a test `Settings`) and avoids module-level side effects.
```python
# src/my_app/main.py
from contextlib import asynccontextmanager
from fastapi import FastAPI
from my_app.config import Settings
from my_app.api import health, v1
def create_app(settings: Settings | None = None) -> FastAPI:
if settings is None:
settings = Settings()
@asynccontextmanager
async def lifespan(app: FastAPI):
# --- startup ---
app.state.settings = settings
# Open DB pool, warm caches, etc.
yield
# --- shutdown ---
# Close DB pool, flush buffers, etc.
app = FastAPI(
title=settings.app_name,
version="1.0.0",
lifespan=lifespan,
docs_url="/docs" if settings.debug else None,
redoc_url=None,
)
app.include_router(health.router)
app.include_router(v1.router, prefix="/api/v1")
return app
# Module-level instance for uvicorn
app = create_app()
```
!!! warning "Prefer lifespan handlers"
Never use `@app.on_event("startup")` / `@app.on_event("shutdown")`. These are deprecated. The `asynccontextmanager` lifespan is the canonical approach since FastAPI 0.95.
---
## Router Organization
```
src/my_app/api/
├── __init__.py
├── health.py # GET /healthz — no auth, no versioning
├── deps.py # Shared Depends() factories
└── v1/
├── __init__.py # APIRouter with prefix="/v1"
├── items.py
└── users.py
```
Each router file:
```python
from fastapi import APIRouter
router = APIRouter(prefix="/items", tags=["items"])
@router.get("/")
async def list_items() -> list[Item]:
...
```
Root registration:
```python
from fastapi import APIRouter
from my_app.api.v1 import items, users
router = APIRouter()
router.include_router(items.router)
router.include_router(users.router)
```
---
## Pydantic Settings (Configuration)
```python
# src/my_app/config.py
from pydantic_settings import BaseSettings, SettingsConfigDict
from functools import lru_cache
class Settings(BaseSettings):
model_config = SettingsConfigDict(
env_file=".env",
env_file_encoding="utf-8",
extra="ignore",
)
app_name: str = "My App"
debug: bool = False
host: str = "0.0.0.0"
port: int = 8000
log_level: str = "info"
# Add DB URL, secret keys, etc. here — never hardcode
# database_url: str # required — will raise if missing
@lru_cache
def get_settings() -> Settings:
return Settings()
```
Use `lru_cache` so the `.env` file is read once. In tests, override with:
```python
from my_app.config import get_settings
from my_app.main import create_app
app = create_app(settings=Settings(debug=True, database_url="sqlite://"))
```
**Never** import `settings` as a module-level singleton — it prevents test overrides.
---
## Health Endpoint
```python
# src/my_app/api/health.py
from fastapi import APIRouter
from fastapi.responses import JSONResponse
router = APIRouter(tags=["ops"])
@router.get("/healthz", include_in_schema=False)
async def health() -> JSONResponse:
"""Kubernetes/Docker liveness probe. No auth. No versioning."""
return JSONResponse({"status": "ok"})
@router.get("/readyz", include_in_schema=False)
async def readiness(request: Request) -> JSONResponse:
"""Readiness probe — check DB connectivity, etc."""
# Example: await request.app.state.db.execute("SELECT 1")
return JSONResponse({"status": "ready"})
```
Rules:
- No authentication required on `/healthz` and `/readyz`.
- `/healthz` — liveness: can the process respond?
- `/readyz` — readiness: are dependencies available?
- Keep them on the root path (not `/api/v1/healthz`).
---
## Dependency Injection
Use `Depends()` to share resources from app state:
```python
# src/my_app/api/deps.py
from fastapi import Depends, Request
from my_app.config import Settings
def get_settings(request: Request) -> Settings:
return request.app.state.settings
SettingsDep = Annotated[Settings, Depends(get_settings)]
```
In route handlers:
```python
@router.get("/config")
async def show_config(settings: SettingsDep) -> dict:
return {"debug": settings.debug}
```
---
## Error Handling
Register a global exception handler for unhandled errors:
```python
from fastapi import Request
from fastapi.responses import JSONResponse
@app.exception_handler(Exception)
async def unhandled_exception_handler(request: Request, exc: Exception) -> JSONResponse:
# Log the exception — never expose internal details to clients
logger.exception("Unhandled error", exc_info=exc)
return JSONResponse(status_code=500, content={"detail": "Internal server error"})
```
---
## CORS
Add CORS middleware only when needed (e.g., browser clients from a different origin):
```python
from fastapi.middleware.cors import CORSMiddleware
app.add_middleware(
CORSMiddleware,
allow_origins=settings.cors_origins, # list from config, never ["*"] in prod
allow_credentials=True,
allow_methods=["*"],
allow_headers=["*"],
)
```
---
## Response Models
Always declare `response_model` or return type annotations — FastAPI uses them for OpenAPI docs and response validation:
```python
@router.post("/items/", response_model=ItemOut, status_code=201)
async def create_item(item: ItemIn) -> ItemOut:
...
```
Use separate `In` / `Out` models when the write shape differs from the read shape (e.g., password hashing, computed fields).
---
## Security Checklist
- Never expose `docs_url` in production (set `docs_url=None` when `not settings.debug`).
- Validate all user input with Pydantic models — never pass raw request data to DB queries.
- Use `SecretStr` for passwords and API keys in `Settings`.
- Apply authentication globally via middleware or `app.include_router(..., dependencies=[Depends(verify_token)])`.
- Add `X-Content-Type-Options: nosniff` and `X-Frame-Options: DENY` headers in production middleware.
@@ -0,0 +1,255 @@
# uv Project Layout and Dependency Management
!!! info "Primary sources"
- [uv project guide](https://docs.astral.sh/uv/guides/projects/)
- [uv project layout](https://docs.astral.sh/uv/concepts/projects/layout/)
- [uv Docker integration](https://docs.astral.sh/uv/guides/integration/docker/)
---
## Core Files
| File | Purpose | Commit? |
|------|---------|---------|
| `pyproject.toml` | Project metadata, deps, tool config | Yes |
| `uv.lock` | Exact resolved versions, cross-platform | **Yes** |
| `.python-version` | Default Python version for the project | Yes |
| `.venv/` | Local virtual environment | No (`.gitignore`) |
!!! warning "Commit the lockfile"
`uv.lock` must be committed. It is the source of truth for reproducible installs in CI and Docker. Never edit it by hand.
---
## Essential Commands
```bash
# Initialize a new project (app, not library)
uv init --app
# Initialize a library (installable package with src layout)
uv init --lib
# Add a runtime dependency
uv add fastapi[standard]
# Add multiple dependencies at once
uv add uvicorn[standard] pydantic-settings
# Add dev-only dependency
uv add --dev pytest httpx pytest-asyncio ruff mypy
# Remove a dependency
uv remove requests
# Upgrade a specific package (keeps rest of lockfile intact)
uv lock --upgrade-package fastapi
# Upgrade all packages
uv lock --upgrade
# Sync env to lockfile (install/remove as needed)
uv sync
# Sync without dev deps (e.g., in CI or Docker)
uv sync --no-dev
# Sync and assert lockfile is up-to-date (for CI / Docker)
uv sync --locked
# Run a command in the project environment
uv run pytest
uv run uvicorn my_app.main:app --reload
# Run a one-off command without installing anything permanently
uv run --with httpx python -c "import httpx; print(httpx.__version__)"
```
---
## pyproject.toml Reference
```toml
[project]
name = "my-app"
version = "0.1.0"
description = "Production FastAPI service"
requires-python = ">=3.12"
license = { text = "MIT" }
readme = "README.md"
dependencies = [
"fastapi[standard]>=0.115",
"uvicorn[standard]>=0.34",
"pydantic-settings>=2.0",
]
[project.scripts]
# Creates `my-app` CLI entry point when installed
my-app = "my_app.main:main"
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[tool.hatch.build.targets.wheel]
packages = ["src/my_app"]
[tool.uv]
dev-dependencies = [
"pytest>=8",
"pytest-asyncio>=0.24",
"httpx>=0.27", # needed for FastAPI TestClient
"ruff>=0.6",
"mypy>=1.11",
]
[tool.pytest.ini_options]
testpaths = ["tests"]
asyncio_mode = "auto"
strict-markers = true
[tool.ruff.lint]
select = ["E", "F", "I", "UP", "B", "S"]
[tool.mypy]
strict = true
```
---
## src Layout with uv
Use `uv init --lib` or set up manually:
```
my-app/
├── pyproject.toml
├── uv.lock
├── .python-version # e.g., "3.12"
├── .env.example
├── README.md
├── src/
│ └── my_app/
│ └── __init__.py
└── tests/
└── conftest.py
```
The `src/` layout prevents the local directory from shadowing the installed package, which would otherwise cause silent test failures when testing the installed version.
**hatchling config for src layout:**
```toml
[tool.hatch.build.targets.wheel]
packages = ["src/my_app"]
```
---
## Migration from pip / requirements.txt
```bash
# 1. Initialize uv in an existing project
uv init --no-workspace # if already has pyproject.toml, skip this
# 2. Import from requirements.txt
uv add -r requirements.txt
# 3. Import dev requirements
uv add --dev -r requirements-dev.txt
# 4. Verify lockfile was created
cat uv.lock | head -20
# 5. Clean up old files
rm requirements.txt requirements-dev.txt setup.py setup.cfg Pipfile Pipfile.lock
# 6. Add .venv to .gitignore
echo ".venv/" >> .gitignore
```
---
## uv in Docker
The canonical pattern uses `--mount=type=cache` for fast rebuilds and `--no-install-project` for layer separation:
```dockerfile
# Copy uv binary (pin the version)
COPY --from=ghcr.io/astral-sh/uv:0.5.27 /uv /uvx /bin/
WORKDIR /app
# Layer 1: install dependencies (changes rarely → cached aggressively)
RUN --mount=type=cache,target=/root/.cache/uv \
--mount=type=bind,source=uv.lock,target=uv.lock \
--mount=type=bind,source=pyproject.toml,target=pyproject.toml \
uv sync --locked --no-install-project --no-editable
# Layer 2: copy source and install project (changes frequently)
COPY . /app
RUN --mount=type=cache,target=/root/.cache/uv \
uv sync --locked --no-editable
```
**Key flags:**
| Flag | Effect |
|------|--------|
| `--locked` | Fail if `uv.lock` is out of date with `pyproject.toml` |
| `--no-install-project` | Install deps but not the project itself (layer separation) |
| `--no-editable` | Install in non-editable mode (copy code into `.venv`, not symlink) |
| `--no-dev` | Skip dev dependencies (use in production images) |
| `--compile-bytecode` | Pre-compile `.pyc` files (faster startup, larger image) |
**After syncing, activate the venv via PATH (not `uv run`) in production:**
```dockerfile
ENV PATH="/app/.venv/bin:$PATH"
CMD ["uvicorn", "my_app.main:app", "--host", "0.0.0.0", "--port", "8000"]
```
Using `CMD ["uv", "run", "uvicorn", ...]` in production is fine but adds a small overhead and requires uv to be present in the final image.
---
## CI/CD with uv
```yaml
# GitHub Actions example
- name: Install uv
uses: astral-sh/setup-uv@v4
with:
version: "0.5.27" # pin for reproducibility
- name: Sync dependencies
run: uv sync --locked
- name: Run tests
run: uv run pytest --tb=short
- name: Lint
run: uv run ruff check .
- name: Type check
run: uv run mypy src/
```
---
## .gitignore Entries for uv Projects
```
.venv/
__pycache__/
*.pyc
.env
dist/
*.egg-info/
.mypy_cache/
.ruff_cache/
.pytest_cache/
```
**Do NOT ignore `uv.lock`** — it must be committed.
@@ -0,0 +1,195 @@
# uvicorn Settings Reference
!!! info "Primary sources"
- [uvicorn settings](https://uvicorn.dev/settings/)
- [uvicorn deployment](https://uvicorn.dev/deployment/)
---
## Configuration Methods
Three equivalent approaches (CLI takes precedence over env vars):
```bash
# 1. CLI flags
uvicorn main:app --host 0.0.0.0 --port 8000
# 2. UVICORN_* environment variables
export UVICORN_HOST=0.0.0.0
export UVICORN_PORT=8000
uvicorn main:app
# 3. Programmatic (dev/test only)
uvicorn.run("main:app", host="0.0.0.0", port=8000)
```
!!! note "Environment file scope"
`UVICORN_*` env vars cannot be used from within an `--env-file`. The `--env-file` flag is for the ASGI *application's* config, not uvicorn's own config.
---
## All Settings by Category
### Socket Binding
| Flag | Default | Notes |
|------|---------|-------|
| `--host <str>` | `127.0.0.1` | Use `0.0.0.0` in containers to bind all interfaces |
| `--port <int>` | `8000` | Use `0` to auto-pick an available port |
| `--uds <path>` | — | UNIX domain socket path (use behind Nginx) |
| `--fd <int>` | — | Inherit socket from file descriptor (use with Supervisor) |
### Production
| Flag | Default | Notes |
|------|---------|-------|
| `--workers <int>` | `1` (or `$WEB_CONCURRENCY`) | **Mutually exclusive with `--reload`** |
| `--env-file <path>` | — | Env file for the *application* (not uvicorn itself) |
| `--timeout-worker-healthcheck <int>` | `5` | Seconds before killing a stuck worker |
### Logging
| Flag | Default | Notes |
|------|---------|-------|
| `--log-level <str>` | `info` | `critical`, `error`, `warning`, `info`, `debug`, `trace` |
| `--log-config <path>` | — | `.json` or `.yaml` for `dictConfig()`; other formats use `fileConfig()` |
| `--no-access-log` | — | Disable access log without changing log level |
| `--use-colors / --no-use-colors` | auto | Force color on/off in log output |
### Implementation
| Flag | Default | Notes |
|------|---------|-------|
| `--loop <str>` | `auto` | `auto`, `asyncio`, `uvloop`. `uvloop` requires `uvicorn[standard]` |
| `--http <str>` | `auto` | `auto`, `h11`, `httptools`. `httptools` requires `uvicorn[standard]` |
| `--ws <str>` | `auto` | `auto`, `none`, `websockets`, `websockets-sansio`, `wsproto` |
| `--lifespan <str>` | `auto` | `auto`, `on`, `off` |
| `--ws-max-size <int>` | `16777216` | WebSocket max message size in bytes (16 MB) |
| `--ws-ping-interval <float>` | `20.0` | WebSocket ping interval in seconds |
| `--ws-ping-timeout <float>` | `20.0` | WebSocket ping timeout in seconds |
### HTTP / Proxy Headers
| Flag | Default | Notes |
|------|---------|-------|
| `--proxy-headers` | enabled | Trust `X-Forwarded-For`, `X-Forwarded-Proto` from trusted IPs |
| `--no-proxy-headers` | — | Disable proxy header trust entirely |
| `--forwarded-allow-ips <list>` | `127.0.0.1` | Comma-separated IPs/networks/literals to trust. Use `'*'` to trust all (safe in containers behind a trusted LB). **Security risk if exposed directly to internet.** |
| `--root-path <str>` | `""` | ASGI `root_path` for apps mounted below a URL prefix |
| `--server-header / --no-server-header` | enabled | Include/suppress `Server` response header |
| `--date-header / --no-date-header` | enabled | Include/suppress `Date` response header |
| `--header <name:value>` | — | Add custom default response headers (repeatable) |
### Resource Limits
| Flag | Default | Notes |
|------|---------|-------|
| `--limit-concurrency <int>` | — | Max concurrent connections/tasks; returns HTTP 503 above this |
| `--limit-max-requests <int>` | — | Restart worker after N requests (limits memory leak accumulation) |
| `--limit-max-requests-jitter <int>` | `0` | Random jitter added to `--limit-max-requests` to stagger worker restarts |
| `--backlog <int>` | `2048` | Max queued connections under high load |
### Timeouts
| Flag | Default | Notes |
|------|---------|-------|
| `--timeout-keep-alive <int>` | `5` | Close keep-alive connections after N seconds of inactivity |
| `--timeout-graceful-shutdown <int>` | — | Seconds to wait for in-flight requests to complete on SIGTERM before force-closing |
### Development
| Flag | Default | Notes |
|------|---------|-------|
| `--reload` | `False` | Auto-reload on file changes. **Never use in production.** Mutually exclusive with `--workers`. |
| `--reload-dir <path>` | `.` | Directory to watch for changes (repeatable) |
| `--reload-delay <float>` | `0.25` | Seconds between reload checks |
| `--reload-include <glob>` | `*.py` | Patterns to include in watch (requires `watchfiles`) |
| `--reload-exclude <glob>` | `.*, .py[cod], ...` | Patterns to exclude from watch (requires `watchfiles`) |
### Application
| Flag | Default | Notes |
|------|---------|-------|
| `--factory` | — | Treat `APP` as a `() -> ASGI app` callable (app factory pattern) |
| `--app-dir <path>` | `.` | Add to `PYTHONPATH` when resolving `APP` |
| `--reset-contextvars` | `False` | Run each request in a fresh `contextvars.Context` (asyncio only; workaround for a CPython context-leak bug) |
---
## Recommended Production CMD
```bash
uvicorn my_app.main:app \
--host 0.0.0.0 \
--port 8000 \
--workers 1 \
--loop auto \
--http auto \
--log-level info \
--proxy-headers \
--forwarded-allow-ips '*' \
--timeout-graceful-shutdown 30
```
In a Dockerfile (exec form):
```dockerfile
CMD ["uvicorn", "my_app.main:app",
"--host", "0.0.0.0",
"--port", "8000",
"--workers", "1",
"--proxy-headers",
"--forwarded-allow-ips", "*",
"--timeout-graceful-shutdown", "30"]
```
---
## Process Manager Options
### Built-in multi-worker (Docker Compose / single host)
```bash
uvicorn my_app.main:app --workers 4
```
The built-in manager spawns workers, monitors their health, and auto-restarts crashed workers. Signal support:
- `SIGHUP` — rolling graceful restart (deploy new code without dropping requests)
- `SIGTTIN` — add one worker
- `SIGTTOU` — remove one worker
### Behind Nginx (UNIX socket)
```bash
uvicorn my_app.main:app --uds /tmp/uvicorn.sock --proxy-headers
```
Nginx config headers to set:
```nginx
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
```
---
## `uvicorn[standard]` vs bare `uvicorn`
| Package | Extras included |
|---------|----------------|
| `uvicorn` | Pure Python h11 HTTP, asyncio event loop |
| `uvicorn[standard]` | `uvloop` (faster event loop), `httptools` (faster HTTP parser), `watchfiles` (better reload), `websockets`, `PyYAML` (for `--log-config`) |
Use `uvicorn[standard]` in all environments. The `[standard]` extras are also included when you install `fastapi[standard]`.
---
## Anti-patterns
| Anti-pattern | Fix |
|---|---|
| `--reload` in Dockerfile CMD | Remove it — `--reload` is dev-only |
| `--loop uvloop` explicitly | Use `--loop auto` — it selects uvloop automatically when available |
| `--http h11` explicitly in prod | Use `--http auto` — it selects httptools when available |
| `uvicorn.run()` at module level (no `if __name__ == '__main__':`) | Breaks multiprocessing workers; always guard it |
| Shell form `CMD uvicorn ...` | Exec form `CMD ["uvicorn", ...]` — required for SIGTERM to reach uvicorn |
+59
View File
@@ -0,0 +1,59 @@
---
name: mcp-details
description: "Reference hub for MCP and FastMCP source documentation links. Use when you need authoritative protocol, SDK, transport, and deployment docs without loading broad implementation guidance."
x-personal-mcp:
id: mcp-details
version: 1.0.0
tags:
- mcp
- model-context-protocol
- fastmcp
- references
- source-docs
capabilities:
- resource://skills/mcp-details/document
---
# MCP Details
This skill is a reference index only. It is optimized for fast retrieval of upstream documentation links for MCP and FastMCP.
## When to Use
- You need official MCP protocol and architecture docs.
- You need MCP SDK links for Python or TypeScript.
- You need FastMCP docs and source references.
- You need ecosystem links for tooling, inspection, and client configuration.
## How To Use This Skill
1. Classify the request by intent: protocol, SDK usage, FastMCP, or ecosystem integration.
2. Open only the matching reference page first.
3. Load at most one additional reference page if the request spans multiple areas.
4. Return links grouped by category, with a one-line reason for each group.
## Intent Router
1. MCP fundamentals, protocol architecture, resources, tools, prompts, transports, security: [mcp-protocol-and-spec.md](./references/mcp-protocol-and-spec.md)
2. MCP SDK and FastMCP implementation references for Python and TypeScript: [sdk-and-fastmcp.md](./references/sdk-and-fastmcp.md)
3. MCP client integration and operational tooling references: [ecosystem-and-tooling.md](./references/ecosystem-and-tooling.md)
## Load Order
1. Start with the single best-match reference page from the Intent Router.
2. If the question includes both protocol and implementation details, load [mcp-protocol-and-spec.md](./references/mcp-protocol-and-spec.md) then [sdk-and-fastmcp.md](./references/sdk-and-fastmcp.md).
3. Load [ecosystem-and-tooling.md](./references/ecosystem-and-tooling.md) only when the request includes client setup, inspector usage, or deployment/operations context.
## Load Budget
1. Single-focus request: 1 reference page.
2. Mixed protocol and implementation request: 2 reference pages.
3. Broad audit or migration planning request: up to 3 reference pages.
## Output Contract
When this skill is applied, return:
1. Which reference files were consulted.
2. The discovery path used (intent classification and load order).
3. Curated source-document links grouped by topic.
4. Any notable gaps or ambiguities in the currently indexed links.
@@ -0,0 +1,29 @@
# Ecosystem and Tooling
Use this page for MCP client setup, operational tools, and integration references.
## VS Code and Copilot MCP Integration
!!! info "VS Code MCP docs"
- [VS Code MCP servers overview](https://code.visualstudio.com/docs/agent-customization/mcp-servers)
- [VS Code MCP configuration reference](https://code.visualstudio.com/docs/agents/reference/mcp-configuration)
- [VS Code Copilot customization overview](https://code.visualstudio.com/docs/copilot/customization/overview)
## Debugging and Inspection
!!! info "Inspector and diagnostics"
- [MCP inspector repository](https://github.com/modelcontextprotocol/inspector)
- [MCP protocol repository issues](https://github.com/modelcontextprotocol/spec/issues)
- [Python logging configuration docs](https://docs.python.org/3/library/logging.config.html)
## Runtime and API Framework References
!!! info "Runtime references"
- [FastAPI documentation](https://fastapi.tiangolo.com/)
- [Uvicorn settings](https://www.uvicorn.org/settings/)
- [AnyIO documentation](https://anyio.readthedocs.io/en/stable/)
## Notes
- Use these links when tasks include IDE wiring, MCP server runtime setup, or production operations.
- Keep protocol and SDK references separate to avoid overloading implementation prompts.
@@ -0,0 +1,32 @@
# MCP Protocol and Specification
Use this page for authoritative links about MCP concepts, protocol shape, and official specification assets.
## Official Documentation
!!! info "MCP docs"
- [MCP introduction](https://modelcontextprotocol.io/docs/getting-started/intro)
- [Architecture overview](https://modelcontextprotocol.io/docs/learn/architecture)
- [Server concepts](https://modelcontextprotocol.io/docs/learn/server-concepts)
- [Client concepts](https://modelcontextprotocol.io/docs/learn/client-concepts)
- [Security overview](https://modelcontextprotocol.io/docs/learn/security-overview)
## Protocol and Schema Sources
!!! info "Specification repositories"
- [MCP specification repository](https://github.com/modelcontextprotocol/spec)
- [Specification schema directory](https://github.com/modelcontextprotocol/spec/tree/main/schema)
- [Specification issues and proposals](https://github.com/modelcontextprotocol/spec/issues)
## Core Capability References
!!! info "Capability details"
- [Resources concept docs](https://modelcontextprotocol.io/docs/learn/server-concepts#resources)
- [Tools concept docs](https://modelcontextprotocol.io/docs/learn/server-concepts#tools)
- [Prompt objects concept docs](https://modelcontextprotocol.io/docs/learn/server-concepts#prompts)
- [Sampling concept docs](https://modelcontextprotocol.io/docs/learn/client-concepts)
## Notes
- Prefer these links when the user asks about protocol correctness, transport semantics, capability naming, or compatibility.
- For implementation-level examples, use [sdk-and-fastmcp.md](./sdk-and-fastmcp.md).
@@ -0,0 +1,31 @@
# SDK and FastMCP
Use this page for implementation-oriented links across MCP SDKs and FastMCP.
## MCP SDKs
!!! info "SDK sources"
- [Python SDK repository](https://github.com/modelcontextprotocol/python-sdk)
- [TypeScript SDK repository](https://github.com/modelcontextprotocol/typescript-sdk)
- [Python SDK documentation](https://modelcontextprotocol.github.io/python-sdk/)
## FastMCP
!!! info "FastMCP sources"
- [FastMCP project documentation](https://gofastmcp.com/)
- [FastMCP GitHub repository](https://github.com/jlowin/fastmcp)
- [FastMCP examples directory](https://github.com/jlowin/fastmcp/tree/main/examples)
- [FastMCP PyPI package](https://pypi.org/project/fastmcp/)
## Server Implementation Patterns
!!! info "Implementation references"
- [MCP server concepts](https://modelcontextprotocol.io/docs/learn/server-concepts)
- [MCP architecture patterns](https://modelcontextprotocol.io/docs/learn/architecture)
- [Python packaging and resources](https://docs.python.org/3/library/importlib.resources.html)
## Notes
- Prefer official SDK repositories for API shape and compatibility checks.
- Use FastMCP references for rapid server scaffolding and implementation examples.
- For protocol-first questions, start from [mcp-protocol-and-spec.md](./mcp-protocol-and-spec.md).
+157
View File
@@ -0,0 +1,157 @@
---
name: nicegui
description: 'Reference hub for NiceGUI and FastAPI application structure, typed configuration, ASGI and Uvicorn startup, UI composition, styling, bindable state, interactions, troubleshooting, testing, and source documentation. Use when planning, implementing, reviewing, deploying, or debugging NiceGUI applications; load only the references relevant to the task.'
x-personal-mcp:
id: nicegui
version: 2.5.0
tags:
- nicegui
- fastapi
- asgi
- uvicorn
- pydantic-settings
- configuration
- deployment
- ui
- architecture
- scaffolding
- customization
- frontend
- testing
- source-docs
capabilities:
- resource://skills/nicegui/document
---
# NiceGUI Reference
Use this skill as a progressive reference for NiceGUI applications built with FastAPI. Start with the routing map, load only the material needed for the current question, and reconcile it with the target project's NiceGUI version and established conventions.
## When to Use
- Planning or reviewing NiceGUI application structure and FastAPI composition.
- Building or refactoring pages, components, layouts, and static assets.
- Modeling UI state with bindings or bindable dataclasses.
- Implementing forms, uploads, refreshes, live updates, or background work.
- Diagnosing UI state, concurrency, navigation, or asset problems.
- Verifying framework behavior against primary documentation.
## How to Use This Skill
1. Classify the request using the discovery map below.
2. Load the smallest relevant reference, or at most two references for a mixed concern.
3. Inspect the target repository before applying guidance; preserve its sound local patterns.
4. Check the pinned NiceGUI and integration versions before relying on version-specific APIs.
5. Validate the changed behavior with focused tests and, for UI work, relevant viewport checks.
## Progressive Discovery Map
### Application Architecture
Load [application architecture](./references/architecture.md) for:
- FastAPI app factories and lifespan ownership
- package boundaries and dependency direction
- page registration and health routes
- optional persistence, LangGraph, or mounted documentation
- async responsiveness and baseline tests
### FastAPI And Uvicorn Startup
Load [FastAPI and Uvicorn startup](./references/fastapi-uvicorn-startup.md) for:
- choosing between `ui.run()` and `ui.run_with()`
- understanding the parent FastAPI app and NiceGUI's internal app
- composing ASGI lifespan and mounted routes
- loading one typed settings snapshot for server and application configuration
- serving an app instance or factory with Uvicorn
- exposing programmatic startup through `[project.scripts]`
- reload, worker, and process-local state constraints
### Components And Styling
Load [architecture and styling](./references/architecture-and-styling.md) for:
- page, component, and service boundaries
- component extraction decisions
- Quasar props, Tailwind utilities, and custom CSS boundaries
- responsive layout and static asset conventions
- Tailwind and Quasar breakpoint scales, container queries, and responsive testing
- uniformly scaling dialogs on mobile
- preserving Quasar field proportions
- keeping detached `QSelect` menus anchored
- sizing scrollable dialog cards under CSS `zoom`
- validating zoomed controls with Playwright or a browser
### Bindable State
Load [bindable dataclasses](./references/binding-dataclasses.md) for:
- typed local UI state
- propagation and refresh behavior
- nested structures and strict bindings
- mutable defaults, performance, and version notes
### Interaction Patterns
Load [interaction patterns](./references/interaction-patterns.md) for:
- uploads and form submission
- explicit refreshes
- server-sent events and WebSockets
- background work and duplicate-submission guards
### Troubleshooting And Quality
Load [troubleshooting and quality gates](./references/troubleshooting-and-quality-gates.md) for:
- upload failures and UI race conditions
- stale assets and navigation drift
- responsiveness, accessibility, reliability, and maintainability checks
### Primary Sources
Load [source documentation](./references/source-documentation.md) when:
- behavior is version-sensitive or uncertain
- an integration recommendation needs verification
- upstream NiceGUI, FastAPI, Tailwind, Quasar, SQLAlchemy, Pydantic, or LangGraph documentation is required
## Common Discovery Paths
### New Application Or Architecture Review
1. Load [application architecture](./references/architecture.md).
2. Add [FastAPI and Uvicorn startup](./references/fastapi-uvicorn-startup.md) when FastAPI owns the application or startup must be exposed as a project command.
3. Add [architecture and styling](./references/architecture-and-styling.md) only when page and component design is in scope.
### Page Or Component Work
1. Load [architecture and styling](./references/architecture-and-styling.md).
2. Add [interaction patterns](./references/interaction-patterns.md) or [bindable dataclasses](./references/binding-dataclasses.md) according to the page behavior.
### Debugging Or Production Review
1. Start with [troubleshooting and quality gates](./references/troubleshooting-and-quality-gates.md).
2. Follow the symptom to one detailed reference.
3. Confirm uncertain behavior in [source documentation](./references/source-documentation.md).
## General Defaults
- Keep composition, transport, services, pages, and components directionally separated.
- Keep business logic out of UI components and event handlers.
- Avoid blocking I/O and CPU-heavy work in the UI event loop.
- Prefer event-driven updates and explicit refreshes over unrelated polling.
- Prefer Tailwind utilities, then Quasar props, then reusable component helpers; use minimal shared CSS when those are insufficient.
- Provide loading, success, and failure states for user-triggered work.
- Treat version-specific guidance as a prompt to verify the project's dependency version.
## Reference Use Contract
When applying this skill:
- return only guidance relevant to the current task
- distinguish repository facts from reference recommendations
- cite the appropriate source reference for framework-level claims
- state assumptions when application requirements are missing
- report the focused checks used to validate implementation changes
@@ -0,0 +1,289 @@
# NiceGUI Page Layout And Styling
Use this reference to structure NiceGUI pages, choose component boundaries, apply responsive layout, and introduce custom CSS without fighting Quasar's internal geometry.
## Ownership And Dependency Boundaries
Keep dependencies flowing in one direction:
- pages import components and services
- components contain presentation logic only
- services contain business logic and do not import UI
- bootstrap code mounts static assets and loads shared CSS once
Suggested module split:
```text
src/my_app/
ui/
pages/
components/
static/
services/
api/
```
Page modules should compose a route from reusable presentation and service calls. They should not own domain rules, persistence, or long-running synchronous work.
## Page Composition
Build the outer layout before styling individual controls:
1. Define the page shell and width constraints.
2. Establish responsive rows, columns, gaps, and wrapping.
3. Add semantic sections and repeated components.
4. Configure Quasar component appearance with props.
5. Add custom CSS only for behavior that props and utilities cannot express safely.
```python
with ui.column().classes("w-full max-w-6xl mx-auto gap-6 px-4"):
page_header(title="Inventory")
with ui.row().classes("w-full gap-4 flex-wrap lg:flex-nowrap items-start"):
filters_panel().classes("w-full lg:w-72 shrink-0")
item_grid().classes("w-full flex-1 min-w-0")
```
Use stable width, minimum-width, and flex constraints so labels, icons, validation messages, and loaded content do not shift the surrounding layout.
## Component Extraction
Extract a presentation pattern to `ui/components/` when it appears on two or more pages or when it owns a meaningful interaction boundary. Keep one-off route layout in the page module.
```python
def card_section(title: str, content: str) -> ui.card:
with ui.card().classes("w-full max-w-md") as card:
ui.label(title).classes("text-lg font-bold")
ui.label(content).classes("text-gray-600")
return card
```
Reusable components should accept data and event callbacks rather than import page state or business services implicitly.
## Styling Decision Order
NiceGUI wraps Quasar components. Choose the styling mechanism according to what it owns:
1. Use Quasar props for component appearance, density, labels, and popup behavior.
2. Use NiceGUI `.classes()` and Tailwind utilities for width, spacing, alignment, and responsive layout.
3. Use reusable component functions for repeated visual patterns.
4. Use `.style()` for genuinely dynamic inline values.
5. Use minimal shared CSS only when props and utilities are insufficient.
Common Quasar props include:
- `outlined`
- `dense`
- `stack-label`
- `popup-content-class`
- `input-class`
- `input-style`
Avoid overriding internal selectors such as:
- `.q-field__label`
- `.q-field__native`
- `.q-field__control`
- `.q-field__input`
Quasar coordinates field height, padding, labels, values, icons, and floating-label transforms. Changing only one internal part tends to cause clipping or overlap.
## Responsive Layout
Support these layouts only:
- mobile: a single-column layout with wrapping toolbars and full-width controls
- landscape desktop: $1920 \times 1080$ with side-by-side panels where they improve scanning
- portrait desktop: $1080 \times 1920 with stacked panels or a narrow fixed sidebar
Build the mobile layout first, then add one desktop breakpoint when a row or grid needs more space. Prefer flex wrapping and fluid grids before adding another breakpoint. Use Tailwind classes for page layout and Quasar props for component behavior.
```python
with ui.row().classes('w-full flex-wrap gap-4 lg:flex-nowrap items-start'):
filters_panel().classes('w-full lg:w-72 shrink-0')
item_grid().classes('w-full flex-1 min-w-0')
```
Use `min-w-0` for flexible children, `flex-wrap` for toolbars, and `max-w-* mx-auto` to keep portrait layouts readable. Do not add device-specific component trees, container queries, or custom breakpoints unless a supported layout demonstrates a concrete failure.
## Static Assets And Shared CSS
- Mount static assets from the composition layer.
- Load shared CSS once rather than injecting it from individual pages.
- Keep custom CSS tokenized with variables and scoped to application classes.
- Avoid broad rules against Quasar internals.
- Verify mount paths, reverse-proxy rewrites, and cache behavior.
```python
from pathlib import Path
from fastapi.staticfiles import StaticFiles
STATIC_DIR = Path(__file__).parent / "ui" / "static"
app.mount("/static", StaticFiles(directory=STATIC_DIR), name="static")
ui.add_css((STATIC_DIR / "css" / "base.css").read_text(encoding="utf-8"))
```
## Responsive Dialog Pattern
Use whole-card scaling when a form dialog must become uniformly larger on mobile while preserving Quasar's internal proportions. Keep detached select menus unscaled and make the card itself scrollable.
### Use Normal Field Density
Normal Quasar fields are approximately `56px` high, while dense fields are approximately `40px` high. Remove `dense` when larger controls are needed.
```python
ui.input("Name").props("outlined")
ui.number("Quantity").props("outlined")
ui.select(...).props(
"outlined popup-content-class=app-item-detail-menu"
)
ui.textarea("Description").props("outlined autogrow")
```
Add a scoped class to the dialog card:
```python
ui.card().classes("app-detail-card app-item-detail-card")
```
### Scale The Complete Card
```css
:root {
--item-dialog-scale: 1;
--item-dialog-max-height: calc(100dvh - 3rem);
}
.app-item-detail-card {
width: min(50rem, 50vw);
max-height: var(--item-dialog-max-height);
overflow-y: auto;
overscroll-behavior: contain;
zoom: var(--item-dialog-scale);
}
/* Restore Quasar's baseline if a global rule overrides it. */
.app-item-detail-card .q-field,
.app-item-detail-menu {
font-size: 14px;
}
@media (max-width: 599px) {
:root {
--item-dialog-scale: 1.2;
/* 75dvh becomes 90dvh after 1.2x zoom. */
--item-dialog-max-height: 75dvh;
}
.app-item-detail-card {
width: 80vw;
}
.app-item-detail-menu {
font-size: 16.8px;
}
}
```
The main mobile tuning knob is:
```css
--item-dialog-scale: 1.2;
```
### Keep Detached Popups Unscaled
Do not apply `zoom` or `transform: scale()` to a `QSelect` popup menu. Quasar renders menus outside the dialog and positions them from the unscaled anchor geometry. Scaling the menu container afterward separates it from its field.
Avoid:
```css
.app-item-detail-card,
.app-item-detail-menu {
zoom: 1.2;
}
```
Use:
```css
.app-item-detail-card {
zoom: 1.2;
}
.app-item-detail-menu {
font-size: 16.8px;
}
```
Use `popup-content-class=app-item-detail-menu` to target the detached menu and enlarge its text without changing its coordinate system.
### Account For Zoom When Scrolling
The card's pre-zoom maximum height must account for the scale:
\[
\begin{aligned}
h_{\mathrm{pre}} &= \frac{h_{\mathrm{visible}}}{s} \\
\text{where } s &= \text{the zoom scale}
\end{aligned}
\]
For a desired visual height of `90dvh` at \(1.2\times\):
\[
\frac{90\,\mathrm{dvh}}{1.2} = 75\,\mathrm{dvh}
\]
Therefore:
```css
--item-dialog-max-height: 75dvh;
```
Apply scrolling to the card itself:
```css
.app-item-detail-card {
max-height: var(--item-dialog-max-height);
overflow-y: auto;
overscroll-behavior: contain;
}
```
This keeps the dimmed page stationary while the form scrolls.
### Match The Quasar Breakpoint
Quasar's extra-small breakpoint ends at `599.98px`. A mobile-only rule can use:
```css
@media (max-width: 599px) {
/* Mobile rules. */
}
```
Confirm custom breakpoint values against the target application's Quasar configuration.
## Validation Checklist
Check each completed page at these three viewports:
1. A representative mobile viewport, such as $390 \times 844$.
2. Landscape desktop at $1920 \times 1080$.
3. Portrait desktop at $1080 \times 1920$.
Confirm that page sections do not overlap, toolbars wrap on mobile, desktop panels use the available space without becoming excessively wide, and dialogs remain visible and scroll to their final field.
## Sources
!!! info "Primary sources"
- [NiceGUI element styling and props](https://nicegui.io/documentation/element)
- [NiceGUI binding properties](https://nicegui.io/documentation/section_binding_properties)
- [Quasar components](https://quasar.dev/vue-components)
- [Quasar field](https://quasar.dev/vue-components/field/)
- [Quasar select](https://quasar.dev/vue-components/select/)
- [Tailwind responsive design](https://tailwindcss.com/docs/responsive-design)
- [MDN `zoom`](https://developer.mozilla.org/en-US/docs/Web/CSS/zoom)
@@ -0,0 +1,137 @@
# NiceGUI Application Architecture
Load this reference for application composition, package boundaries, and optional subsystem decisions.
## Baseline Package Boundaries
- `main.py`: process entry point and app factory exposure.
- `bootstrap.py`: app composition, router wiring, page registration, and lifespan orchestration.
- `config.py`: typed settings and environment parsing.
- `logging.py`: centralized logging setup.
- `api/`: HTTP transport that delegates to services.
- `services/`: business and use-case logic.
- `ui/pages/`: route-level NiceGUI pages.
- `ui/components/`: shared presentation building blocks.
Recommended base shape:
```text
.
├─ pyproject.toml
├─ .env.example
├─ src/
│ └─ app/
│ ├─ __init__.py
│ ├─ main.py
│ ├─ bootstrap.py
│ ├─ config.py
│ ├─ logging.py
│ ├─ api/
│ │ ├─ __init__.py
│ │ └─ health.py
│ ├─ services/
│ │ ├─ __init__.py
│ │ └─ example_service.py
│ └─ ui/
│ ├─ __init__.py
│ ├─ components/
│ │ ├─ __init__.py
│ │ └─ nav.py
│ └─ pages/
│ ├─ __init__.py
│ ├─ home.py
│ ├─ dashboard.py
│ └─ about.py
└─ tests/
├─ test_health.py
└─ test_pages_registration.py
```
## Required Baseline Behavior
- FastAPI is the base ASGI app.
- `create_app()` composes routes, resources, and NiceGUI.
- Lifespan owns startup and shutdown resources.
- NiceGUI pages are modular and explicitly registered.
- FastAPI exposes a health route such as `/healthz`.
- Imports do not trigger runtime global side effects.
For the ownership relationship between a caller-created FastAPI app, `nicegui.app`, `ui.run_with()`, Uvicorn, and a packaged startup command, load [FastAPI and Uvicorn startup](./fastapi-uvicorn-startup.md).
## Dependency Direction
Prefer:
- `main/bootstrap` -> `config/logging` + `api` + `ui/pages` + `services`
- `api` -> `services`
- `ui/pages` -> `ui/components` + `services`
- `services` -> helpers, clients, and `db/` when enabled
Avoid imports from services back into API or UI modules.
## Optional Persistence
Use only when the product requires durable data.
```text
src/app/db/
├─ __init__.py
├─ base.py
├─ session.py
├─ models/
└─ repositories/
```
- Create one engine and sessionmaker per process.
- Provide request- or operation-scoped sessions with `yield`.
- Keep transaction boundaries explicit in service or repository flows.
- Never share sessions across concurrent tasks.
- Use Alembic as the schema migration source of truth.
## Optional LangGraph AI
Use only for multi-step orchestration, resumable work, streaming, or human approval.
```text
src/app/ai/
├─ state.py
├─ nodes/
├─ graphs/
├─ runtime.py
└─ contracts.py
```
- Keep graph internals outside API and UI modules.
- Invoke graphs through a service such as `services/ai_service.py`.
- Use stable thread or session IDs for resumable flows.
- Keep interrupt payloads JSON-serializable.
## Optional Mounted Docs
Use only when generated docs must be served by the application.
Suggested settings:
- `docs_enabled`
- `docs_mount_path`
- `docs_site_dir`
- `docs_require_build`
Mount docs in the composition layer, normalize the mount path, avoid route conflicts, and define behavior for missing build artifacts.
## Async And Responsiveness
- Use `async def` where a handler or service path performs I/O.
- Prefer non-blocking clients and libraries.
- Offload CPU-heavy work to worker or background execution.
- Define progress, cancellation, timeout, completion, and error states for long actions.
- Stream or chunk results when workflows are long-running or multi-step.
## Testing Minimums
- Test the FastAPI health route.
- Test page registration wiring.
- If persistence is enabled, test session lifecycle and rollback behavior.
- If AI is enabled, test happy paths and interrupt/resume behavior.
- If docs are enabled, test the mounted index route.
- For long actions, test loading, completion, and error states.
@@ -0,0 +1,100 @@
# Binding Dataclasses Deep Dive
Use this reference to model NiceGUI state with bindable dataclasses and avoid common propagation and performance pitfalls.
## Primary Sources
- NiceGUI binding docs: [binding properties](https://www.nicegui.io/documentation/section_binding_properties)
- Python dataclass docs: [dataclasses module](https://docs.python.org/3/library/dataclasses.html)
- Data class design rationale: [PEP 557](https://peps.python.org/pep-0557/)
## Bindable Dataclass Behavior
`@binding.bindable_dataclass` extends standard dataclasses by turning fields into bindable properties, allowing UI bindings to propagate when a field is assigned.
```python
from nicegui import binding, ui
@binding.bindable_dataclass
class Profile:
name: str = "Ada"
age: int = 37
profile = Profile()
ui.input("Name").bind_value(profile, "name")
ui.number("Age", min=0).bind_value(profile, "age")
ui.label().bind_text_from(profile, "name", backward=lambda name: f"User: {name}")
```
## Propagation And Performance
NiceGUI distinguishes between two link types:
- Bindable properties propagate efficiently when values are assigned.
- Active links are checked in a refresh loop.
Prefer bindable dataclasses for frequently updated form state. Keep binding transforms pure and inexpensive. If an application has many active links, tune `binding_refresh_interval` in `ui.run(...)` only after measuring the impact.
## Dataclass Modeling Rules
- Use `field(default_factory=...)` for mutable defaults.
- Avoid `frozen=True` for models edited by UI controls.
- Use `slots=True` only after confirming compatibility with inheritance and extension needs.
- Keep UI-editable fields explicit and typed.
```python
from dataclasses import field
from nicegui import binding
@binding.bindable_dataclass
class Filters:
query: str = ""
tags: list[str] = field(default_factory=list)
```
## Nested Structures
NiceGUI supports tuple paths for nested data structures.
```python
from nicegui import ui
data = {"user": {"name": "Ada"}}
ui.input("Name").bind_value(data, ("user", "name"))
ui.label().bind_text_from(data, ("user", "name"))
```
Keep nested dataclass updates explicit and predictable at the field level.
## Strictness And Refactor Safety
- Object attributes are checked by default.
- Dictionary keys are not checked by default.
- Use `strict=True` when missing dictionary keys should produce warnings.
```python
from nicegui import app, ui
ui.input().bind_value(app.storage.user, "display_name", strict=True)
```
## Common Pitfalls
- In-place mutation may not produce immediate UI synchronization. Assign the updated value back to the bound field.
- Heavy binding transforms can degrade refresh performance. Move expensive work to event handlers or services.
- State shared across unrelated pages or users can leak data. Scope models to the appropriate page, client, or user context.
## Version Checks
- `bindable_dataclass` was added in NiceGUI 2.11.0.
- Depth-first binding propagation was documented in NiceGUI 2.16.0.
- Binding `strict` behavior was documented in NiceGUI 3.0.0.
- Tuple paths for nested properties were documented in NiceGUI 3.10.0.
Verify these behaviors against the NiceGUI version pinned by the target project.
@@ -0,0 +1,315 @@
# FastAPI And Uvicorn Startup
Use this reference when FastAPI owns the application and NiceGUI is one part of it. The central distinction is between **composing an ASGI application** and **starting an ASGI server**:
- [`ui.run_with()`](https://github.com/zauberzeug/nicegui/blob/main/nicegui/ui_run_with.py) composes NiceGUI with a caller-owned FastAPI application. It does not start Uvicorn.
- [`uvicorn.run()`](https://www.uvicorn.org/#running-programmatically) starts the server and tells it which ASGI application to serve.
## Ownership Model
```mermaid
flowchart TD
E["Project script: my-app"] --> M["main()"]
M --> S["get_settings()"]
M --> U["uvicorn.run()"]
U --> F["create_app()"]
F --> S
F --> P["Parent FastAPI app"]
P --> A["API routes and middleware"]
P -->|"mount_path=/gui"| N["NiceGUI App"]
U -->|"ASGI requests and lifespan"| P
```
The objects have separate responsibilities:
| Object | Owner | Responsibility |
| --- | --- | --- |
| Parent `FastAPI` instance | Application code | Root ASGI app, API routes, middleware, lifespan, and mounted applications |
| `Settings` instance | Application code | Immutable, process-local configuration snapshot shared by startup and composition |
| `nicegui.app` | NiceGUI | A process-local [`App`](https://github.com/zauberzeug/nicegui/blob/main/nicegui/app/app.py) instance that subclasses `FastAPI` |
| `ui.run_with(parent_app)` | NiceGUI integration | Configures NiceGUI, mounts `nicegui.app` into `parent_app`, and integrates lifecycle handling |
| Uvicorn | Server process | Imports or receives the root ASGI app, opens sockets, drives lifespan, and serves requests |
Uvicorn must serve the **parent FastAPI app** when using `ui.run_with()`. Passing `nicegui.app` to `ui.run_with()` is rejected because it would mount NiceGUI into itself and recurse on unmatched routes.
## Choose One Startup Mode
### Let NiceGUI Own Startup
Use `ui.run()` when NiceGUI is the main application. Add ordinary FastAPI routes to the exported `nicegui.app` object:
```python
from nicegui import app, ui
@app.get('/healthz')
def health() -> dict[str, str]:
return {'status': 'ok'}
@ui.page('/')
def home() -> None:
ui.label('Home')
ui.run()
```
In this mode, NiceGUI configures and starts its own [Uvicorn-derived server](https://github.com/zauberzeug/nicegui/blob/main/nicegui/server.py). Do not also call `uvicorn.run()`.
### Let FastAPI Own The Application
Use `ui.run_with()` when an existing FastAPI application owns middleware, API routers, OpenAPI configuration, lifespan resources, or deployment startup. The [official NiceGUI FastAPI example](https://github.com/zauberzeug/nicegui/blob/main/examples/fastapi/main.py) follows this model.
`mount_path` controls where the NiceGUI application appears externally. A NiceGUI page declared as `/` is reachable at `/gui/` when mounted at `/gui`, while parent routes such as `/healthz` remain at the root. A dedicated UI prefix usually makes ownership and route conflicts clearer than mounting both applications at `/`.
## Canonical Factory Layout
Keep application composition importable and server startup explicit:
```text
.
├─ pyproject.toml
└─ src/
└─ my_app/
├─ __init__.py
├─ config.py
└─ main.py
```
```python title="src/my_app/config.py"
from functools import cache
from typing import Literal
from pydantic import BaseModel, ConfigDict, Field, SecretStr
from pydantic_settings import BaseSettings, SettingsConfigDict
class ServerSettings(BaseModel):
model_config = ConfigDict(frozen=True)
host: str = '0.0.0.0'
port: int = 8000
log_level: Literal['critical', 'error', 'warning', 'info', 'debug', 'trace'] = (
'info'
)
reload: bool = False
class GuiSettings(BaseModel):
model_config = ConfigDict(frozen=True)
mount_path: str = '/gui'
storage_secret: SecretStr | None = None
class Settings(BaseSettings):
model_config = SettingsConfigDict(
env_prefix='MY_APP_',
env_nested_delimiter='__',
env_file='.env',
env_file_encoding='utf-8',
frozen=True,
)
server: ServerSettings = Field(default_factory=ServerSettings)
gui: GuiSettings = Field(default_factory=GuiSettings)
@cache
def get_settings() -> Settings:
return Settings()
```
`ServerSettings` and `GuiSettings` inherit from `BaseModel` because they share one application owner, source policy, and process lifecycle. The root `BaseSettings` reads the sources once and validates one atomic snapshot. Environment variables use names such as `MY_APP_SERVER__PORT`, `MY_APP_SERVER__RELOAD`, `MY_APP_GUI__MOUNT_PATH`, and `MY_APP_GUI__STORAGE_SECRET`.
The argument-free [`functools.cache`](https://docs.python.org/3/library/functools.html#functools.cache) provider is appropriate here because both the project entry point and Uvicorn's zero-argument factory need process-lifetime access. Each reload or worker process gets its own settings instance. Do not add override arguments to `get_settings()`; inject a `Settings` instance directly into `create_app()` in tests or alternate composition roots. See the [Pydantic settings implementation guide](../../pydantic-settings/SKILL.md) for source precedence, independent settings boundaries, cache clearing, and runtime reload guidance.
```python title="src/my_app/main.py"
from collections.abc import AsyncIterator
from contextlib import asynccontextmanager
import uvicorn
from fastapi import FastAPI
from nicegui import ui
from my_app.config import Settings, get_settings
@asynccontextmanager
async def lifespan(app: FastAPI) -> AsyncIterator[None]:
app.state.ready = True
try:
yield
finally:
app.state.ready = False
def register_pages() -> None:
@ui.page('/')
def dashboard() -> None:
ui.label('Dashboard')
def create_app(settings: Settings | None = None) -> FastAPI:
settings = settings or get_settings()
app = FastAPI(lifespan=lifespan)
app.state.settings = settings
@app.get('/healthz')
def health() -> dict[str, str]:
return {'status': 'ok'}
register_pages()
ui.run_with(
app,
mount_path=settings.gui.mount_path,
storage_secret=(
settings.gui.storage_secret.get_secret_value()
if settings.gui.storage_secret is not None
else None
),
)
return app
def main() -> None:
settings = get_settings()
uvicorn.run(
'my_app.main:create_app',
factory=True,
host=settings.server.host,
port=settings.server.port,
log_level=settings.server.log_level,
reload=settings.server.reload,
)
if __name__ == '__main__':
main()
```
The `storage_secret` is optional unless the application uses `ui.storage.user` or `ui.storage.browser`. `SecretStr` prevents accidental plaintext display in logs and model representations, while `get_secret_value()` unwraps it only at the NiceGUI integration boundary. Supply production secrets through environment variables or a supported settings secret source rather than committing them.
The example passes an [import string and `factory=True`](https://www.uvicorn.org/settings/#application) to Uvicorn. Uvicorn imports `my_app.main`, calls the zero-argument `create_app` factory, and serves the returned parent FastAPI app. Import strings are also required when Uvicorn creates reload or worker subprocesses; passing `create_app()` directly only supports the simple single-process case.
NiceGUI keeps framework state in its process-local app singleton. Treat `create_app()` as a once-per-worker factory. Calling it repeatedly in one interpreter can register the same pages and lifecycle handlers more than once; tests that create multiple apps must isolate or reset NiceGUI state.
## Lifespan Ordering
The [ASGI lifespan protocol](https://asgi.readthedocs.io/en/latest/specs/lifespan.html) is driven by the server. Uvicorn sends startup before accepting requests and sends shutdown while terminating the process. Lifespan runs once per event loop, including once in each worker process.
Current NiceGUI source integrates with the parent application by:
1. Capturing the parent FastAPI lifespan context.
2. Mounting NiceGUI's internal app on the parent.
3. Replacing the parent lifespan with a wrapper.
4. Starting NiceGUI before entering the original parent lifespan.
5. Exiting the original parent lifespan before shutting down NiceGUI.
This exact ordering comes from the current [`ui.run_with` implementation](https://github.com/zauberzeug/nicegui/blob/main/nicegui/ui_run_with.py) and is version-sensitive. Check the pinned NiceGUI version before making one startup handler depend on another framework's internal ordering.
Create database pools, HTTP clients, and similar resources in the parent [FastAPI lifespan](https://fastapi.tiangolo.com/advanced/events/), then close them after `yield`. Do not create event-loop-bound resources at import time or assume that globals are shared between workers.
## Expose The Server As A Project Script
Map a command name to the no-argument startup function:
```toml title="pyproject.toml"
[project]
name = "my-app"
version = "0.1.0"
requires-python = ">=3.12"
dependencies = [
"fastapi",
"nicegui",
"pydantic-settings",
"uvicorn[standard]",
]
[project.scripts]
my-app = "my_app.main:main"
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[tool.hatch.build.targets.wheel]
packages = ["src/my_app"]
```
Run the installed command through uv:
```bash
uv run my-app
```
The uv [project entry-point documentation](https://docs.astral.sh/uv/concepts/projects/config/#entry-points) requires a build system so uv installs the project and generates its command. The `[project.scripts]` target follows the [PyPA entry-point specification](https://packaging.python.org/en/latest/specifications/entry-points/#use-for-scripts): its generated wrapper imports `main`, calls it without arguments, and uses the return value as the process exit status. Returning `None` means successful completion.
The settings model now owns host, port, logging, reload, mount path, and storage-secret configuration. Add an explicit CLI settings source or another CLI parser only when the project command needs user-supplied arguments; the entry-point callable itself still receives no arguments.
## Development Reload
Because `main()` supplies an import string, it can enable Uvicorn reload for local development:
```dotenv title=".env"
MY_APP_SERVER__HOST=127.0.0.1
MY_APP_SERVER__RELOAD=true
```
The cached settings object is a process-start snapshot. Changing an environment variable or dotenv file does not mutate a running instance; restart the process, or let the development reloader create a new worker when a watched file changes. Keep reload disabled in production. Uvicorn documents [`reload` and `workers` as mutually exclusive](https://www.uvicorn.org/settings/#production), and each worker would have independent settings, NiceGUI state, lifespan resources, and WebSocket connections. Use one worker by default unless the application has explicitly validated session affinity and externalized every stateful dependency needed across processes.
## Anti-Patterns
| Anti-pattern | Why it fails | Preferred approach |
| --- | --- | --- |
| `ui.run_with(nicegui.app)` | Mounts NiceGUI into itself | Pass a separately created `FastAPI()` instance |
| Calling both `ui.run()` and `ui.run_with()` | Gives two paths responsibility for startup | Choose one ownership model |
| `uvicorn.run(create_app(), reload=True)` | Reload subprocesses cannot import the app object | Use an import string with `factory=True` |
| Calling `uvicorn.run()` at module import time | Importing the module starts a blocking server and breaks subprocess startup | Call it from `main()` |
| Top-level `ui.label(...)` with `ui.run_with()` | Script-mode elements are discarded by this integration | Register UI in `@ui.page` functions or a root callable |
| Multiple workers by default | Process-local UI state and WebSockets are not automatically shared | Start with one worker and validate a distributed design explicitly |
| Reconstructing `Settings()` throughout the app | Re-reads sources and obscures the active configuration lifecycle | Inject the startup snapshot or use the argument-free provider at framework boundaries |
| Adding kwargs to cached `get_settings()` | Retains one hidden process-lifetime instance per argument combination | Construct explicit `Settings(...)` overrides and inject them |
## Verification
Use `TestClient` as a context manager so the parent ASGI lifespan runs:
```python
from fastapi.testclient import TestClient
from my_app.config import GuiSettings, Settings
from my_app.main import create_app
def test_application_routes() -> None:
settings = Settings(
gui=GuiSettings(storage_secret='test-storage-secret'),
)
with TestClient(create_app(settings)) as client:
assert client.get('/healthz').json() == {'status': 'ok'}
assert client.get('/gui/').status_code == 200
```
Also verify:
- startup resources exist while the client context is active and are released afterward
- the mounted UI returns HTML and parent API failures retain FastAPI's JSON responses
- `uv run my-app` starts the server and responds on both the API and UI paths
- shutdown signals complete without orphaned background tasks
## Primary Sources
- [NiceGUI pages, routing, and FastAPI integration](https://www.nicegui.io/documentation/section_pages_routing)
- [NiceGUI `ui.run_with` implementation](https://github.com/zauberzeug/nicegui/blob/main/nicegui/ui_run_with.py)
- [NiceGUI FastAPI example](https://github.com/zauberzeug/nicegui/blob/main/examples/fastapi/main.py)
- [FastAPI lifespan events](https://fastapi.tiangolo.com/advanced/events/)
- [ASGI lifespan protocol](https://asgi.readthedocs.io/en/latest/specs/lifespan.html)
- [Uvicorn settings](https://www.uvicorn.org/settings/)
- [Uvicorn programmatic startup](https://www.uvicorn.org/#running-programmatically)
- [Pydantic settings management](https://pydantic.dev/docs/validation/latest/concepts/pydantic_settings/)
- [`functools.cache`](https://docs.python.org/3/library/functools.html#functools.cache)
- [uv project entry points](https://docs.astral.sh/uv/concepts/projects/config/#entry-points)
- [PyPA entry points specification](https://packaging.python.org/en/latest/specifications/entry-points/)
@@ -0,0 +1,110 @@
# Interaction Patterns Reference
## Reactive State
Use bindable dataclasses for local page state.
```python
from dataclasses import field
from nicegui import binding, ui
@binding.bindable_dataclass
class PageState:
selected_id: int | None = None
items: list = field(default_factory=list)
state = PageState()
ui.label().bind_text_from(state, "selected_id")
```
## File Upload Pattern
- Validate extension and size before storing.
- Delegate storage to a service method.
- Notify success and failure explicitly.
```python
async def handle_upload(e: ui.events.UploadEventArguments):
try:
if e.size > 10 * 1024 * 1024:
raise ValueError("File too large")
if not e.name.endswith(".pdf"):
raise ValueError("Only PDF allowed")
await file_service.store(e.content.read(), e.name)
ui.notify(f"Uploaded: {e.name}", type="positive")
except ValueError as err:
ui.notify(str(err), type="negative")
ui.upload(on_upload=handle_upload, auto_upload=True)
```
## Form Submission Pattern
- Bind UI inputs to dataclass fields.
- Perform validation in the service layer.
- Clear form state on success.
```python
@binding.bindable_dataclass
class FormData:
name: str = ""
email: str = ""
data = FormData()
ui.input("Name").bind_value(data, "name")
ui.input("Email").bind_value(data, "email")
async def on_submit():
try:
await user_service.create_user(name=data.name, email=data.email)
ui.notify("User created", type="positive")
data.name = data.email = ""
except ValueError as err:
ui.notify(str(err), type="negative")
ui.button("Submit").on_click(on_submit)
```
## Real-Time Updates Decision
Use SSE for one-way status streaming.
Use WebSocket for bidirectional messaging.
SSE endpoint example:
```python
@app.get("/events/status")
async def status_stream():
async def gen():
while True:
yield f"data: {await get_status()}\\n\\n"
await asyncio.sleep(1)
return StreamingResponse(gen(), media_type="text/event-stream")
```
## Background Work Pattern
- Start long jobs in FastAPI background tasks.
- Expose status via endpoint or streaming channel.
- Guard buttons against duplicate submissions during in-flight tasks.
## Explicit Refresh Pattern
Use @ui.refreshable and call refresh intentionally instead of polling unrelated state.
```python
@ui.refreshable
async def item_list():
items = await service.list()
for item in items:
ui.label(item.name)
ui.button("Refresh").on_click(lambda: item_list.refresh())
```
## Links
!!! info "Primary sources"
- [NiceGUI action events](https://nicegui.io/documentation/section_action_events)
- [FastAPI server-sent events](https://fastapi.tiangolo.com/advanced/server-sent-events/)
- [FastAPI WebSockets](https://fastapi.tiangolo.com/advanced/websockets/)
@@ -0,0 +1,70 @@
# Source Documentation
Use these links to verify framework-specific behavior before relying on version-sensitive or integration-specific guidance.
## NiceGUI
!!! info "NiceGUI sources"
- [Pages, routing, and FastAPI integration](https://www.nicegui.io/documentation/section_pages_routing)
- [`ui.run_with` implementation](https://github.com/zauberzeug/nicegui/blob/main/nicegui/ui_run_with.py)
- [FastAPI integration example](https://github.com/zauberzeug/nicegui/blob/main/examples/fastapi/main.py)
- [Binding properties and bindable dataclasses](https://www.nicegui.io/documentation/section_binding_properties)
- [Action events](https://www.nicegui.io/documentation/section_action_events)
- [Security best practices](https://www.nicegui.io/documentation/section_security)
## FastAPI
!!! info "FastAPI sources"
- [Lifespan events](https://fastapi.tiangolo.com/advanced/events/)
- [Settings and environment variables](https://fastapi.tiangolo.com/advanced/settings/)
- [Dependencies with yield](https://fastapi.tiangolo.com/tutorial/dependencies/dependencies-with-yield/)
- [Server-sent events](https://fastapi.tiangolo.com/advanced/server-sent-events/)
- [WebSockets](https://fastapi.tiangolo.com/advanced/websockets/)
## ASGI And Uvicorn
!!! info "Server and lifespan sources"
- [ASGI lifespan protocol](https://asgi.readthedocs.io/en/latest/specs/lifespan.html)
- [Uvicorn settings](https://www.uvicorn.org/settings/)
- [Uvicorn programmatic startup](https://www.uvicorn.org/#running-programmatically)
- [Uvicorn deployment](https://www.uvicorn.org/deployment/)
## uv And Project Scripts
!!! info "Packaging and command sources"
- [uv project entry points](https://docs.astral.sh/uv/concepts/projects/config/#entry-points)
- [uv project packaging](https://docs.astral.sh/uv/concepts/projects/config/#project-packaging)
- [PyPA entry points specification](https://packaging.python.org/en/latest/specifications/entry-points/)
## Styling
!!! info "Styling sources"
- [Tailwind utility-first styling](https://tailwindcss.com/docs/utility-first)
- [Tailwind responsive design and container queries](https://tailwindcss.com/docs/responsive-design)
- [Quasar components](https://quasar.dev/vue-components)
- [Quasar Screen plugin documentation source](https://github.com/quasarframework/quasar/blob/dev/docs/src/pages/options/screen-plugin.md)
- [CSS media queries](https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_media_queries/Using_media_queries)
- [CSS container queries](https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_containment/Container_queries)
## Persistence
!!! info "Persistence sources"
- [SQLAlchemy engine configuration and pooling](https://docs.sqlalchemy.org/en/20/core/engines.html)
- [SQLAlchemy session lifecycle](https://docs.sqlalchemy.org/en/20/orm/session_basics.html)
- [Alembic tutorial](https://alembic.sqlalchemy.org/en/latest/tutorial.html)
## Configuration And Dataclasses
!!! info "Python and Pydantic sources"
- [Pydantic settings management](https://docs.pydantic.dev/latest/concepts/pydantic_settings/)
- [Python dataclasses](https://docs.python.org/3/library/dataclasses.html)
- [PEP 557: Data Classes](https://peps.python.org/pep-0557/)
## LangGraph
!!! info "LangGraph sources"
- [Overview](https://docs.langchain.com/oss/python/langgraph/overview)
- [Workflows and agents](https://docs.langchain.com/oss/python/langgraph/workflows-agents)
- [Persistence](https://docs.langchain.com/oss/python/langgraph/persistence)
- [Streaming](https://docs.langchain.com/oss/python/langgraph/streaming)
- [Interrupts and human-in-the-loop](https://docs.langchain.com/oss/python/langgraph/interrupts)
@@ -0,0 +1,39 @@
# Troubleshooting and Quality Gates
## Troubleshooting
### Upload Errors
- Validate extension and size before storage.
- Catch expected exceptions and return negative notifications.
- Log unexpected exceptions with request context.
### UI Race Conditions
- Disable triggering controls during async work.
- Remove duplicate timers and listeners targeting the same state.
- Ensure service call ordering is deterministic before render updates.
### Asset Caching
- Confirm static mount and proxy rewrite correctness.
- Add cache-busting query strings for changed assets.
- Avoid per-page CSS injection.
### Navigation and State Drift
- Avoid global mutable UI state.
- Keep state request-scoped or service-managed.
- Rehydrate page data during route load.
## Production Readiness Gate
Pass all checks before shipping:
- Structure: one-way dependencies between pages, components, and services.
- Responsiveness: UI validated at both small and large viewport widths.
- Accessibility: labels and actions are clear and readable.
- Reliability: validation and exception paths surface user feedback.
- Maintainability: repeated UI patterns are extracted; business logic remains in services.
If any check fails, return to the workflow step that owns that concern.
+344
View File
@@ -0,0 +1,344 @@
---
name: pydantic-settings
description: "Practical guide for implementing typed application configuration with pydantic-settings. Use when designing BaseSettings models, choosing nested or independent settings boundaries, managing settings lifecycles, configuring dotenv or secrets, and customizing source priority safely."
x-personal-mcp:
id: pydantic-settings
version: 1.1.0
tags:
- python
- pydantic
- pydantic-settings
- configuration
- env-vars
- secrets
- dotenv
- source-priority
- caching
- lifecycle
capabilities:
- resource://skills/pydantic-settings/document
---
# Pydantic Settings Implementation Guide
Use this skill to implement robust, typed application configuration with `pydantic-settings` in production Python services.
## When to Use
- You need a single typed configuration model for app settings.
- You are migrating from ad-hoc `os.getenv(...)` calls.
- You need predictable precedence across init args, env vars, dotenv files, and secrets.
- You need nested settings models and reliable parsing behavior.
- You need to choose between one nested application settings object and independently owned settings objects.
- You need a deliberate construction, caching, or reload lifecycle.
- You need to customize settings sources or source order safely.
## Procedure
### 1. Baseline Model
Create a single settings model for the service boundary:
```python
from pydantic import BaseModel, Field
from pydantic_settings import BaseSettings, SettingsConfigDict
class DatabaseSettings(BaseModel):
host: str = "localhost"
port: int = 5432
user: str
password: str
class Settings(BaseSettings):
model_config = SettingsConfigDict(
env_prefix="APP_",
env_file=".env",
env_file_encoding="utf-8",
extra="ignore",
frozen=True,
)
debug: bool = False
log_level: str = "info"
database: DatabaseSettings
api_key: str = Field(validation_alias="MY_API_KEY")
```
Quality gate:
1. Required fields fail fast when missing.
2. Defaults are intentional and safe.
### 2. Pick Env Naming Rules
1. Choose one prefix and apply it consistently.
2. Use aliases only for compatibility or external contracts.
3. Document whether env names are case-sensitive.
Quality gate:
1. Team can derive env variable names without guessing.
2. Legacy names are supported only where needed.
### 3. Decide Nested Parsing
For nested models via env vars, configure delimiters intentionally:
```python
model_config = SettingsConfigDict(
env_prefix="APP_",
env_nested_delimiter="__",
env_nested_max_split=1,
)
```
Typical vars:
1. `APP_DATABASE={"host": "db", "port": 5432, "user": "svc", "password": "pw"}`
2. `APP_DATABASE__HOST=db.internal`
Quality gate:
1. Nested overrides behave as expected.
2. Delimiter choice does not collide with field names.
### 4. Confirm Source Priority
Default priority (higher first):
1. CLI args (if enabled)
2. init kwargs
3. env vars
4. dotenv
5. secrets dir
6. defaults
Only customize when required:
```python
from pydantic_settings import PydanticBaseSettingsSource
@classmethod
def settings_customise_sources(
cls,
settings_cls: type[BaseSettings],
init_settings: PydanticBaseSettingsSource,
env_settings: PydanticBaseSettingsSource,
dotenv_settings: PydanticBaseSettingsSource,
file_secret_settings: PydanticBaseSettingsSource,
) -> tuple[PydanticBaseSettingsSource, ...]:
return (init_settings, env_settings, dotenv_settings, file_secret_settings)
```
Quality gate:
1. Priority order is explicit in code.
2. Tests verify conflict resolution.
### 5. Add Secrets Strategy
1. In local development, dotenv is acceptable for non-production values.
2. In deployed environments, prefer env vars or secret managers.
3. For file-mounted secrets, use `secrets_dir`.
Example:
```python
model_config = SettingsConfigDict(
env_prefix="APP_",
env_file=".env",
secrets_dir="/run/secrets",
)
```
Quality gate:
1. No secret literals in repository code.
2. Missing secrets behavior is understood per environment.
### 6. Choose Nested Or Independent Settings Boundaries
Prefer one root `BaseSettings` object with nested `BaseModel` sections when the configuration belongs to one application lifecycle:
```python
from pydantic import BaseModel, Field
from pydantic_settings import BaseSettings, SettingsConfigDict
class DatabaseSettings(BaseModel):
host: str = "localhost"
port: int = 5432
class ObservabilitySettings(BaseModel):
log_level: str = "INFO"
json_logs: bool = True
class Settings(BaseSettings):
model_config = SettingsConfigDict(
env_prefix="APP_",
env_nested_delimiter="__",
frozen=True,
)
database: DatabaseSettings = Field(default_factory=DatabaseSettings)
observability: ObservabilitySettings = Field(
default_factory=ObservabilitySettings
)
```
This produces names such as `APP_DATABASE__HOST` and gives the application one validated, atomic configuration snapshot. Nested sections should normally inherit from `BaseModel`, not `BaseSettings`; otherwise each nested settings model can collect sources independently and produce surprising results.
Use independent `BaseSettings` classes when the objects have genuinely independent ownership:
1. Different packages or deployable components own the schemas.
2. Each object needs its own env prefix or source policy.
3. A component is optional or loaded lazily.
4. Components need different reload lifecycles.
5. The same component must run outside the application.
Construct independent objects explicitly at the composition root and inject each dependency. Do not nest one `BaseSettings` class inside another merely to reuse its fields. Extract a shared `BaseModel` schema when models need common structure.
Quality gate:
1. Nested sections share one source policy and lifecycle.
2. Independent settings have distinct owners, prefixes, or lifecycles.
3. The application does not repeatedly scan the same sources through accidental nested `BaseSettings` construction.
### 7. Own The Settings Lifecycle
For most applications, construct settings once at the composition root and pass the validated object to services:
```python
def main() -> None:
settings = Settings()
application = Application(settings=settings)
application.run()
```
This makes ownership, startup failure, and test overrides explicit. Treat the object as a snapshot: environment variables and files changing later do not update an existing instance. Prefer `frozen=True` for shared settings so consumers cannot silently mutate process-wide configuration.
Use [`functools.cache`](https://docs.python.org/3/library/functools.html#functools.cache) only when process-lifetime singleton access is intentional and explicit injection is awkward, such as a framework dependency provider:
```python
from functools import cache
@cache
def get_settings() -> Settings:
return Settings()
```
Keep the cached factory argument-free. Passing override kwargs creates one cached instance per argument combination, retains those values for the process lifetime, and obscures which configuration is active. In tests, instantiate `Settings(...)` directly or override the dependency; when a test must exercise the cached getter, isolate environment changes with `get_settings.cache_clear()` before and after the assertion.
`cache` is process-local. Every worker process gets its own instance, and concurrent first calls can construct more than one instance before the cache is populated. Settings construction must therefore be side-effect free; create engines, clients, and sessions in their own lifecycle-managed providers.
Quality gate:
1. Settings are created once per intended application or worker lifecycle.
2. Cached factories are argument-free and side-effect free.
3. Tests do not leak cached settings or environment changes.
4. Resource construction is separate from configuration parsing.
### 8. Reload Deliberately
Static service configuration should normally require a process restart. If runtime reload is a real requirement, construct a fresh settings instance and atomically replace the owned reference. Do not call `__init__()` on a shared instance: readers can observe mutation in progress, and resources derived from old values may remain alive.
Settings sources are synchronous. In an async application, construction or reload that reads dotenv, secrets, JSON, TOML, or YAML files should run in a worker thread:
```python
import asyncio
async def load_settings() -> Settings:
return await asyncio.to_thread(Settings)
```
Clearing `get_settings` is sufficient for controlled tests or single-threaded administration, but it is not an atomic live-reload protocol. Concurrent applications should own the current reference behind an application-specific lock or lifecycle manager, swap in a fully validated replacement, and then rebuild dependent resources.
Quality gate:
1. Reload creates and validates a replacement before publication.
2. Readers cannot observe a partially mutated object.
3. Dependent resources are recreated after the settings reference changes.
4. File-backed source reads do not block an async event loop.
### 9. Add Focused Lifecycle Tests
Do not add tests that re-validate baseline `pydantic-settings` functionality unless custom behavior is layered on top. Test the application-owned behavior instead:
1. Repeated cached getter calls return the same instance.
2. Cache clearing after an environment change returns a newly validated instance.
3. Explicitly injected settings bypass global cached state.
4. Reload swaps the settings snapshot and rebuilds dependent resources, when reload is supported.
Suggested invocation:
1. `uv run pytest -q`
## Completion Checks
1. Settings ownership matches the application or component lifecycle.
2. Source precedence is documented and tested.
3. Env naming conventions and aliases are explicit and stable.
4. Nested parsing behavior is tested when custom parsing behavior is added.
5. Secrets and dotenv usage are environment-appropriate and do not leak sensitive defaults.
6. Validation errors are actionable and fail fast for required values.
7. Cached factories are argument-free, process-local, and cleared deliberately in tests.
8. Nested models share one source policy; independent settings have an explicit ownership reason.
9. Runtime reload, if supported, replaces a validated snapshot and rebuilds dependent resources.
## Output Contract
When this skill is applied, return:
1. Which references were consulted.
2. The chosen source-precedence model and why.
3. The exact parsing and alias decisions made.
4. Any deferred choices and their risk.
5. The validation commands or tests run to confirm behavior.
Use these upstream docs when implementing or reviewing `pydantic-settings` behavior.
## Source Docs
### Primary
- [Settings Management](https://pydantic.dev/docs/validation/latest/concepts/pydantic_settings/)
- [pydantic-settings package repository](https://github.com/pydantic/pydantic-settings)
### Core Concepts
- [Field aliases](https://pydantic.dev/docs/validation/latest/concepts/fields/#field-aliases)
- [Alias choices](https://pydantic.dev/docs/validation/latest/concepts/alias#aliaspath-and-aliaschoices)
- [Validation default behavior](https://pydantic.dev/docs/validation/latest/concepts/fields#validate-default-values)
- [ImportString type](https://pydantic.dev/docs/validation/latest/api/pydantic/types/#pydantic.types.ImportString)
### Priority And Sources
- [Field value priority](https://pydantic.dev/docs/validation/latest/concepts/pydantic_settings/#field-value-priority)
- [Customise settings sources](https://pydantic.dev/docs/validation/latest/concepts/pydantic_settings/#customise-settings-sources)
- [Other settings source types](https://pydantic.dev/docs/validation/latest/concepts/pydantic_settings/#other-settings-source)
### Environment And Parsing
- [Environment variable names and prefix behavior](https://pydantic.dev/docs/validation/latest/concepts/pydantic_settings/#environment-variable-names)
- [Case sensitivity behavior](https://pydantic.dev/docs/validation/latest/concepts/pydantic_settings/#case-sensitivity)
- [Parsing environment variable values](https://pydantic.dev/docs/validation/latest/concepts/pydantic_settings/#parsing-environment-variable-values)
- [Nested model default partial updates](https://pydantic.dev/docs/validation/latest/concepts/pydantic_settings/#nested-model-default-partial-updates)
### Lifecycle And Reloading
- [In-place reloading](https://pydantic.dev/docs/validation/latest/concepts/pydantic_settings/#in-place-reloading)
- [Async environments](https://pydantic.dev/docs/validation/latest/concepts/pydantic_settings/#async-environments)
- [`functools.cache`](https://docs.python.org/3/library/functools.html#functools.cache)
### Dotenv And Secrets
- [Dotenv support](https://pydantic.dev/docs/validation/latest/concepts/pydantic_settings/#dotenv-env-support)
- [Secrets](https://pydantic.dev/docs/validation/latest/concepts/pydantic_settings/#secrets)
- [Nested secrets](https://pydantic.dev/docs/validation/latest/concepts/pydantic_settings/#nested-secrets)
+144
View File
@@ -0,0 +1,144 @@
---
name: pytesting
description: "Reference hub for pytest suite structure, naming, markers, and stack-specific testing patterns. Optimized for progressive discovery so naming and hierarchy guidance are loaded first when shaping or reorganizing tests."
x-personal-mcp:
id: pytesting
version: 1.0.0
tags:
- pytest
- testing
- python
- fastapi
- asyncio
- anyio
- deterministic
capabilities:
- resource://skills/pytesting/document
---
# Pytesting
This skill is a collection of preferences and links to source documentation for building and maintaining pytest suites.
Use it to quickly find the right guidance for:
1. Baseline pytest structure and marker strategy.
2. Naming conventions and test hierarchy organization.
3. FastAPI route, dependency override, and lifespan testing patterns.
4. SQLAlchemy transaction and session testing patterns.
5. AsyncIO loop-scope, fixture-lifecycle, and cancellation-safe testing patterns.
Repository defaults:
- `uv run pytest` is the canonical invocation.
- pytest settings live in `pyproject.toml` under `[tool.pytest.ini_options]`.
- strict marker checking is expected (`--strict-markers`).
## Progressive Discovery Start
Use this load order by default so guidance stays targeted and naming conventions are pulled in early:
1. Classify intent first: naming and organization, baseline pytest mechanics, FastAPI testing, SQLAlchemy testing, or mixed.
2. For create/restructure/rename tasks, load [naming-and-organization.md](./references/naming-and-organization.md) first.
3. Load [pytest-docs.md](./references/pytest-docs.md) next for fixture and marker defaults.
4. Load at most one stack-specific reference unless the request is explicitly mixed stack.
5. If confidence is low after two references, ask one clarifying question before loading more.
Load budget defaults:
1. Single-stack task: 1 to 2 references.
2. Mixed-stack task: up to 3 references.
3. Avoid loading all references unless the user explicitly asks for a broad audit.
## Intent Router
Open only the reference that matches the immediate task.
1. Naming, file layout, discovery prefixes, class/function naming: [naming-and-organization.md](./references/naming-and-organization.md)
2. Fixture layering, marker policy, collect-only and fast-path commands: [pytest-docs.md](./references/pytest-docs.md)
3. Route tests, dependency overrides, lifespan handling: [fastapi-testing.md](./references/fastapi-testing.md)
4. Session and transaction fixtures, async ORM behavior: [sqlalchemy-testing.md](./references/sqlalchemy-testing.md)
5. Async test mode selection, event loop scope, cancel-scope teardown issues: [asyncio-testing.md](./references/asyncio-testing.md)
## Naming Pull-In Triggers
Always consult [naming-and-organization.md](./references/naming-and-organization.md) before recommending structure when any of these are true:
1. New tests are being added.
2. Existing tests are being reorganized or renamed.
3. The request mentions conventions, readability, hierarchy, or discoverability.
4. The task introduces parametrization where case naming affects failure readability.
## Guiding Principles
These principles are abstract, but are the highest priority to follow.
- Much of testing is very well-trodden. In general, tests should follow whatever conventions there are.
- Tests will be run very frequently, so it's important that they run quickly and deterministically.
- When tests fail, it should be easy to determine what failed and fix it.
- Always be on guard against tests that are tautological. Every test should provide specific value by capturing something about the intent of the program.
## Pytest Best Practices
These are stable defaults regardless of stack:
1. Apply pytest naming and hierarchy conventions first so discovery and ownership stay predictable; see [naming-and-organization.md](./references/naming-and-organization.md).
2. Mirror `src/` into `tests/` so ownership and coverage are obvious.
3. Keep fixtures explicit and layered (`tests/conftest.py` globally, subtree `conftest.py` for domain-specific fixtures).
4. Register markers up front (`unit`, `integration`, `smoke`, `slow`, `external`) and keep strict marker checks enabled.
5. Separate fast feedback (`-m unit`) from broader integration/external lanes.
6. Validate structure early with collection checks before expanding assertions.
7. Keep test scope tight and count intentional; add tests only when each case protects a distinct behavior.
8. Start with the single core-intent behavior path, then add edge cases based on real risk.
9. Prefer parametrized tests for behavior variants instead of cloning near-identical test functions.
10. Reject low-signal assertions (for example `assert True` patterns) and avoid tests that only assert a mock was called.
11. Prefer behavior-first tests that exercise real code paths and concrete inputs over patching internals.
12. Use monkeypatching, mocks, and fakes extremely sparingly, only when no practical real-input alternative exists, and only after explicit user confirmation.
## Universal Test Double Policy (Repo-Local Placement)
To avoid over-using monkeypatching, mocks, fakes, etc, apply this policy whenever a test change introduces one of them:
1. Attempt a real-input, real-object test design first.
2. If that approach is impractical, explain why and request user confirmation before adding monkeypatching, mocks, or fakes.
3. Keep any approved test double narrowly scoped and document the exact boundary it replaces.
4. Do not treat call-only verification as sufficient; pair any test double with assertions on observable behavior or outputs.
5. Revisit approved test doubles when implementation seams improve so they can be removed.
## Stack-Specific Guidance
- For FastAPI, prefer dependency overrides and clear lifecycle handling; see [fastapi-testing.md](./references/fastapi-testing.md).
- For SQLAlchemy, prefer transaction-safe session fixtures and explicit async loading strategy; see [sqlalchemy-testing.md](./references/sqlalchemy-testing.md).
- For async fixtures, loop-scope selection, and cancellation-safe teardown, see [asyncio-testing.md](./references/asyncio-testing.md).
- For naming and tree organization, use the conventions in [naming-and-organization.md](./references/naming-and-organization.md).
## Source Documentation Entry Points
Primary upstream docs are curated in each reference page. Start with:
1. Pytest good practices: [pytest docs](https://docs.pytest.org/en/stable/explanation/goodpractices.html)
2. Pytest fixtures: [fixture how-to](https://docs.pytest.org/en/stable/how-to/fixtures.html)
3. Pytest markers: [marker examples](https://docs.pytest.org/en/stable/example/markers.html)
4. FastAPI testing: [FastAPI testing tutorial](https://fastapi.tiangolo.com/tutorial/testing/)
5. SQLAlchemy transaction testing: [SQLAlchemy external transaction pattern](https://docs.sqlalchemy.org/en/20/orm/session_transaction.html#joining-a-session-into-an-external-transaction-such-as-for-test-suites)
6. Pytest monkeypatch usage and limits: [monkeypatch how-to](https://docs.pytest.org/en/stable/how-to/monkeypatch.html)
7. pytest-asyncio configuration: [pytest-asyncio config](https://pytest-asyncio.readthedocs.io/en/stable/reference/configuration.html)
8. AnyIO cancellation semantics: [AnyIO cancellation and timeouts](https://anyio.readthedocs.io/en/stable/cancellation.html)
## Quick Validation Commands
Use these commands to check structure and execution lanes:
1. `uv run pytest --collect-only -q`
2. `uv run pytest -m unit -q`
3. `uv run pytest -m "not external" -q`
4. `uv run pytest -q`
## Output Contract
When this skill is applied, return:
1. Which references were consulted.
2. The discovery path used (intent classification, load order, and why).
3. Recommended structure, naming, fixture, and marker decisions.
4. Concrete naming outcomes: file/module naming pattern, class usage decision, and any parametrization `ids` conventions.
5. Exact validation commands.
6. Relevant source-doc links for any non-trivial recommendation.
7. Risks, assumptions, or open questions.
8. Explicit confirmation status if monkeypatching, mocks, or fakes were requested or used.
@@ -0,0 +1,108 @@
# AsyncIO Testing Patterns (Pytest, FastAPI, AnyIO)
!!! info "Primary sources"
- [pytest-asyncio configuration](https://pytest-asyncio.readthedocs.io/en/stable/reference/configuration.html)
- [pytest-asyncio concepts](https://pytest-asyncio.readthedocs.io/en/stable/concepts.html)
- [pytest-asyncio fixture loop scope how-to](https://pytest-asyncio.readthedocs.io/en/stable/how-to-guides/change_fixture_loop.html)
- [pytest-asyncio default fixture loop scope how-to](https://pytest-asyncio.readthedocs.io/en/stable/how-to-guides/change_default_fixture_loop.html)
- [AnyIO cancellation and cancel-scope safety](https://anyio.readthedocs.io/en/stable/cancellation.html)
- [FastAPI async tests](https://fastapi.tiangolo.com/advanced/async-tests/)
## Agent Quick Path
Use this reference when tests involve asynchronous fixtures, HTTP clients, task groups, or teardown failures.
1. Confirm async plugin mode in pytest config (`asyncio_mode`).
2. Keep async fixture loop scope predictable, defaulting to `function` unless there is a measured need to broaden it.
3. Prefer one async testing model per lane (pytest-asyncio or AnyIO-style markers), and keep it consistent.
4. Keep async fixtures small and isolate stateful resources to the narrowest useful scope.
5. If teardown errors mention cancel scopes or task groups, validate that setup and teardown run in the same task context.
## Baseline Configuration
Recommended defaults for most projects using `pytest-asyncio`:
```toml
[tool.pytest.ini_options]
asyncio_mode = "auto"
asyncio_default_fixture_loop_scope = "function"
```
Why:
- [Strict mode](https://pytest-asyncio.readthedocs.io/en/stable/concepts.html#test-discovery-modes) is safer for multi-plugin environments, but [auto mode](https://pytest-asyncio.readthedocs.io/en/stable/concepts.html#test-discovery-modes) is often simpler when the suite is primarily asyncio-based.
- [Function loop scope](https://pytest-asyncio.readthedocs.io/en/stable/reference/configuration.html#asyncio-default-fixture-loop-scope) minimizes cross-test coupling and avoids many lifecycle surprises.
If a fixture or test needs broader loop sharing, make it explicit instead of changing suite-wide defaults:
```python
import pytest
import pytest_asyncio
@pytest_asyncio.fixture(loop_scope="module")
async def shared_resource():
...
@pytest.mark.asyncio(loop_scope="module")
async def test_uses_shared_loop(shared_resource):
...
```
## FastAPI Endpoint Test Patterns
Use [FastAPI's async testing guidance](https://fastapi.tiangolo.com/advanced/async-tests/) as the default:
1. Use `httpx.AsyncClient` with `ASGITransport` for async endpoint tests.
2. Mark async tests with one consistent marker style for the suite.
3. If app lifespan hooks matter, add [LifespanManager](https://fastapi.tiangolo.com/advanced/async-tests/#httpx) support because `AsyncClient` alone does not trigger lifespan events.
Example:
```python
import pytest
from httpx import ASGITransport, AsyncClient
@pytest.mark.asyncio
async def test_healthz(app):
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as client:
response = await client.get("/healthz")
assert response.status_code == 200
```
## Fixture Design For Async Reliability
Apply these patterns first:
1. Keep async fixtures narrow (`function` scope by default).
2. Keep one responsibility per fixture when possible.
3. Prefer yield fixtures and pair each setup step with teardown in the same fixture.
4. Avoid mixing many independent event-loop lifecycles in one fixture chain.
When using transports that manage internal task groups (for example, streaming clients), avoid patterns that risk splitting lifecycle across different task contexts.
## Troubleshooting Cancel-Scope Teardown Failures
When you see errors like `Attempted to exit cancel scope in a different task than it was entered in`, treat it as an async lifecycle-ownership issue first.
Checklist:
1. Verify fixture and test loop scopes are compatible and explicit.
2. Confirm async resource setup and teardown are owned by the same fixture context.
3. Reduce fixture scope (`module` or `session` -> `function`) to test for loop/task ownership drift.
4. Ensure the suite uses one primary async plugin model for the failing lane.
5. Re-run with focused selection and skip reasons to isolate first failing fixture:
- `uv run --group test python -m pytest -m smoke tests/web -q -rs`
Relevant references:
- [Avoiding cancel scope stack corruption](https://anyio.readthedocs.io/en/stable/cancellation.html#avoiding-cancel-scope-stack-corruption)
- [pytest-asyncio configuration](https://pytest-asyncio.readthedocs.io/en/stable/reference/configuration.html)
- [pytest fixture teardown behavior](https://docs.pytest.org/en/stable/how-to/fixtures.html#teardown-cleanup-aka-fixture-finalization)
## Commands Worth Remembering
- `uv run --group test python -m pytest --collect-only -q`
- `uv run --group test python -m pytest -m smoke tests/web -q -rs`
- `uv run --group test python -m pytest -m integration -q`
- `uv run --group test python -m pytest -q`
@@ -0,0 +1,236 @@
# [FastAPI Testing](https://fastapi.tiangolo.com/tutorial/testing/)
Best practices for testing FastAPI applications with pytest.
## Agent Quick Path
Use this sequence before reading the full reference:
1. If test is pure route behavior, use `TestClient` and plain `def` tests.
2. If test must `await` other async work, use `AsyncClient` + `@pytest.mark.anyio`.
3. Prefer `app.dependency_overrides` over `mock.patch`.
4. Reset overrides after each test/fixture teardown.
5. For startup/shutdown logic, use `TestClient` as context manager or `LifespanManager` with async client.
Decision rules:
- Need DB contract verification: choose integration tests and override `get_db`/`get_session`.
- Need pure business logic checks: keep tests HTTP-free (`unit`).
- Need one critical path sanity check: one endpoint per `smoke` test.
---
## Tools
| Tool | Purpose |
|------|---------|
| `fastapi.testclient.TestClient` | Synchronous HTTP test client (wraps HTTPX, built on Starlette) |
| `httpx.AsyncClient` + `ASGITransport` | Async client for tests that `await` other async code |
| `app.dependency_overrides` | Replace any `Depends()` dependency for the duration of a test |
| `anyio` / `pytest-anyio` | Run async test functions with `@pytest.mark.anyio` |
Install deps: `httpx`, `anyio` (or `pytest-anyio`).
---
## Synchronous Tests (Preferred Default)
Use `TestClient` for route tests that don't need to `await` anything else.
Test functions are plain `def` — no `async def`, no `await`.
```python
from fastapi.testclient import TestClient
from myapp.main import app
client = TestClient(app)
def test_read_item_returns_200():
response = client.get("/items/foo", headers={"X-Token": "secret"})
assert response.status_code == 200
assert response.json()["id"] == "foo"
def test_read_item_bad_token_returns_400():
response = client.get("/items/foo", headers={"X-Token": "wrong"})
assert response.status_code == 400
```
Rules:
- One `TestClient` per test module is fine (stateless between calls).
- Pass headers, query params, JSON body, or form data the same way as HTTPX/requests.
- Do not pass Pydantic models directly; use `.model_dump()` or `jsonable_encoder`.
---
## Async Tests
Use `AsyncClient` only when the test itself needs to `await` other coroutines
(e.g. querying a real async DB after an API call to verify side effects).
```python
import pytest
from httpx import ASGITransport, AsyncClient
from myapp.main import app
@pytest.mark.anyio
async def test_root_async():
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as ac:
response = await ac.get("/")
assert response.status_code == 200
```
Rules:
- Mark with `@pytest.mark.anyio`; register the `anyio` marker in `pyproject.toml`.
- `AsyncClient` does **not** trigger lifespan events by default; use `asgi-lifespan`'s
`LifespanManager` when startup/shutdown matters.
- Instantiate objects that require an event loop (e.g. async DB clients) inside
async functions, not at module level.
---
## Dependency Overrides (Preferred Over Mocking)
`app.dependency_overrides` is the idiomatic FastAPI seam — use it instead of
patching internals with `unittest.mock`.
```python
from fastapi.testclient import TestClient
from myapp.main import app
from myapp.deps import get_current_user
client = TestClient(app)
def fake_user():
return {"id": 1, "name": "Test User"}
def test_protected_route_with_fake_user():
app.dependency_overrides[get_current_user] = fake_user
response = client.get("/me")
app.dependency_overrides = {} # always reset after the test
assert response.status_code == 200
assert response.json()["name"] == "Test User"
```
Or reset cleanly with `autouse=False` fixture teardown:
```python
import pytest
from myapp.main import app
@pytest.fixture()
def override_user():
app.dependency_overrides[get_current_user] = lambda: {"id": 1, "name": "Test User"}
yield
app.dependency_overrides = {}
```
Rules:
- Override at the lowest-level dependency that owns the external boundary
(e.g. `get_db`, `get_current_user`, `get_settings`).
- Always reset `app.dependency_overrides` after each test or fixture teardown.
- Prefer a real in-process fake (e.g. in-memory SQLite session) over a mock object.
---
## Database Testing
The preferred pattern is a real SQLite (or test Postgres) session injected via
dependency override, not a mock.
```python
# tests/conftest.py
import pytest
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from fastapi.testclient import TestClient
from myapp.main import app
from myapp.db.session import get_db
from myapp.db.models import Base
TEST_DB_URL = "sqlite:///./test.db"
@pytest.fixture(scope="session")
def engine():
e = create_engine(TEST_DB_URL, connect_args={"check_same_thread": False})
Base.metadata.create_all(bind=e)
yield e
Base.metadata.drop_all(bind=e)
@pytest.fixture()
def db_session(engine):
Session = sessionmaker(bind=engine)
session = Session()
yield session
session.rollback()
session.close()
@pytest.fixture()
def client(db_session):
app.dependency_overrides[get_db] = lambda: db_session
yield TestClient(app)
app.dependency_overrides = {}
```
Rules:
- Prefer `session`-scoped engine creation; `function`-scoped session with rollback per test.
- Keep unit tests DB-free; use this pattern only in `integration`-marked tests.
- For async SQLAlchemy, mirror the same pattern using `AsyncEngine` / `AsyncSession`.
---
## Lifespan and Startup Events
`TestClient` triggers lifespan events (startup/shutdown) when used as a context manager:
```python
def test_with_lifespan():
with TestClient(app) as client:
response = client.get("/health")
assert response.status_code == 200
```
For `AsyncClient`, use `asgi-lifespan`:
```python
from asgi_lifespan import LifespanManager
@pytest.mark.anyio
async def test_with_async_lifespan():
async with LifespanManager(app):
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as ac:
response = await ac.get("/health")
assert response.status_code == 200
```
---
## Marker Strategy for FastAPI Tests
| Marker | When to use |
|--------|------------|
| `unit` | Pure service/utility logic with no HTTP or DB calls |
| `integration` | `TestClient` + real DB session via dependency override |
| `smoke` | One `TestClient` call per critical user path, no DB reset |
| `external` | Tests that call real third-party APIs (skip in CI by default) |
---
## Quick Reference: Sending Data
| What to send | Parameter |
|---|---|
| Path / query param | Part of the URL string |
| JSON body | `json={"key": "value"}` |
| Form data | `data={"field": "value"}` |
| Headers | `headers={"X-Token": "..."}` |
| Cookies | `cookies={"session": "..."}` |
| File upload | `files={"file": ("name.txt", b"content", "text/plain")}` |
---
## Official Docs
- [Testing tutorial](https://fastapi.tiangolo.com/tutorial/testing/)
- [Async tests](https://fastapi.tiangolo.com/advanced/async-tests/)
- [Testing dependencies with overrides](https://fastapi.tiangolo.com/advanced/testing-dependencies/)
- [Testing a database (SQLModel)](https://fastapi.tiangolo.com/how-to/testing-database/)
- [Testing lifespan events](https://fastapi.tiangolo.com/advanced/testing-events/)
- [Testing WebSockets](https://fastapi.tiangolo.com/advanced/testing-websockets/)
- [HTTPX docs](https://www.python-httpx.org/)
@@ -0,0 +1,118 @@
# Pytest Naming Conventions and Test Organization
!!! info "Primary sources"
- [Good integration practices](https://docs.pytest.org/en/stable/explanation/goodpractices.html)
- [Changing standard (Python) test discovery](https://docs.pytest.org/en/stable/example/pythoncollection.html)
- [How to use fixtures](https://docs.pytest.org/en/stable/how-to/fixtures.html)
- [How to parametrize fixtures and test functions](https://docs.pytest.org/en/stable/how-to/parametrize.html)
- [Marker examples](https://docs.pytest.org/en/stable/example/markers.html)
## Agent Quick Path
Use this when creating or reorganizing test modules so naming and hierarchy stay predictable.
1. Mirror the product domain structure in `tests/` so ownership is obvious.
2. Encode broad context in module and class names (`test_*.py`, `Test*`).
3. Keep leaf test names short and behavior-focused (`test_*`).
4. Use `class Test<Subject>:` only for grouping related scenarios.
5. Place fixtures in the nearest `conftest.py` needed by scope.
6. Separate expensive tests with markers first, directories second.
## Naming Conventions
### File and directory naming
- Use lowercase snake_case for test file names: `test_user_service.py`.
- Keep directories domain-oriented and stable over time: `tests/orders/`, `tests/billing/`.
- Prefer descriptive test names over internal ticket numbers or implementation details.
### Test function naming
- Prefer hierarchical naming: put broad context in folder/module/class, and keep the function name focused on the final assertion.
- Keep pytest discovery prefixes intact:
- modules start with `test_`
- classes start with `Test`
- functions start with `test_`
- Start with user-visible behavior or contract, not private helper names.
Recommended pattern:
- module: `test_<subject>.py`
- class: `Test<Operation>` or `Test<Scenario>`
- function: `test_<expected_outcome>`
Examples:
- Flat (still valid): `test_create_order_rejects_invalid_currency`
- Class-context: `TestOrder -> TestCreate -> test_rejects_invalid_currency`
- Module-context: `test_order.py -> TestCreate -> test_rejects_invalid_currency`
- Module + class context can similarly shorten:
- `test_token.py -> TestRefresh -> test_rotates_session_id`
- `test_user_list.py -> TestListUsers -> test_returns_empty_for_new_tenant`
### Test class naming
- Use `class Test<SubjectOrScenario>:` for scenario grouping and context reduction.
- Keep class names noun-focused (`TestOrderService`) rather than action-focused.
- Avoid xUnit style setup inheritance when fixtures can express dependencies directly.
## Hierarchy and Organization Patterns
Two patterns work well; choose one and apply it consistently.
### Pattern A: Source-mirror hierarchy (default for product code ownership)
```text
src/
app/
orders/service.py
billing/invoice.py
tests/
app/
orders/test_service.py
billing/test_invoice.py
```
Use this when teams own modules by source path and want direct test-to-source mapping.
### Pattern B: Cost-lane hierarchy (default for CI policy clarity)
```text
tests/
unit/
orders/test_service.py
integration/
api/test_orders.py
persistence/test_order_repository.py
smoke/
test_health.py
```
Use this when CI gating is based on cost lanes and marker filtering.
### Hybrid rule (recommended)
- Keep a source-mirror tree for local ownership.
- Add markers (`unit`, `integration`, `smoke`, `external`) for runtime policy.
- Avoid duplicating both trees unless the repository already requires it.
## Fixture Placement Strategy
- Put universal lightweight fixtures in `tests/conftest.py`.
- Put domain fixtures in subtree `conftest.py` files close to where they are used.
- Keep fixtures composable and explicit; avoid large fixture "god objects".
- Use `yield` fixtures for teardown so cleanup is always paired with setup.
## Parametrize and ID Naming
- Use `pytest.mark.parametrize` for behavior matrices instead of copy/paste tests.
- Provide explicit `ids=` labels when case names are not obvious.
- Keep IDs business-meaningful (`"expired-token"`, `"zero-balance"`) so failures are readable.
## Collection and Structure Checks
Use these checks after introducing new test files or renaming modules:
- `uv run pytest --collect-only -q`
- `uv run pytest -m unit -q`
- `uv run pytest -m "not external" -q`
If collection surprises appear, verify file names, marker registration, and directory placement first.
## Common Anti-Patterns
- Mixed naming styles (`testFoo.py`, `test_foo.py`, `foo_test.py`) in one repository.
- Deep fixture chains that hide setup behavior.
- Test names that encode implementation details instead of behavior.
- Moving slow tests into `unit` directories without marker updates.
- Sharing mutable module-level state across tests.
@@ -0,0 +1,36 @@
# Pytest Documentation Notes
!!! info "Primary sources"
- [Good integration practices](https://docs.pytest.org/en/stable/explanation/goodpractices.html)
- [Fixture how-to](https://docs.pytest.org/en/stable/how-to/fixtures.html)
- [Marker examples](https://docs.pytest.org/en/stable/example/markers.html)
- [Configuration reference](https://docs.pytest.org/en/stable/reference/customize.html)
- [Flaky tests](https://docs.pytest.org/en/stable/explanation/flaky.html)
## Agent Quick Path
Use this file when you need fast pytest scaffolding defaults without framework-specific details.
1. Mirror source layout under `tests/`.
2. Keep fixtures small and explicit; default to `function` scope.
3. Register markers up front in `pyproject.toml`.
4. Validate structure first with `uv run pytest --collect-only -q`.
5. Run fast lane with `uv run pytest -m unit -q`.
Load other references only when needed:
- FastAPI routes/dependency injection/lifespan: `fastapi-testing.md`
- SQLAlchemy sessions/transactions/DB fixtures: `sqlalchemy-testing.md`
- Naming conventions and test hierarchy: `naming-and-organization.md`
## Practical Guidance For This Skill
- Use src-aligned test layout and keep test discovery conventional.
- Keep fixtures small, composable, and explicit; use `yield` for teardown.
- Register custom markers and keep strict marker validation on.
- Separate quick unit runs from slower integration/external runs.
- Minimize flakiness by controlling shared state and avoiding hidden dependencies.
- Use `--collect-only` and marker-filtered runs to validate scaffold quality early.
## Commands Worth Remembering
- `uv run pytest --collect-only -q`
- `uv run pytest -m unit -q`
- `uv run pytest -m "not external" -q`
- `uv run pytest -q`
@@ -0,0 +1,246 @@
# [SQLAlchemy 2.x Testing](https://docs.sqlalchemy.org/en/20/orm/session_transaction.html#joining-a-session-into-an-external-transaction-such-as-for-test-suites)
Best practices for testing SQLAlchemy ORM code (sync and async) with pytest.
## Agent Quick Path
Use this path first; read deeper sections only when needed.
1. Create engine once per test session.
2. Open connection + outer transaction per test function.
3. Bind `Session`/`AsyncSession` to that connection with `join_transaction_mode="create_savepoint"`.
4. Let code under test call `commit()` safely; rollback outer transaction after test.
5. Inject session into FastAPI via dependency override and always clear overrides.
Branching logic:
- Sync stack: use `create_engine` + `Session` fixtures.
- Async stack: use `create_async_engine` + `AsyncSession` fixtures + `pytest.mark.anyio`.
- SQLite in-memory with threaded client: use `StaticPool` when required by framework threading behavior.
- Async relationship access fails (`MissingGreenlet`): eager load (`selectinload`) or explicit refresh.
---
## Core Concepts
| Concept | Preferred approach |
|---|---|
| Isolate tests from production DB | Use an in-memory SQLite engine per test |
| Prevent data leaking between tests | Roll back at the connection level after each test |
| Inject test DB into FastAPI | Override the `get_db` / `get_session` dependency |
| Create schema for tests | Call `Base.metadata.create_all(engine)` once per session |
| Avoid lazy-load errors in async | Use `expire_on_commit=False`; use `selectinload()` for relationships |
---
## Sync SQLAlchemy + FastAPI (Recommended Pattern)
The canonical 2.0 pattern joins a test `Session` into an external transaction on a shared `Connection`, then rolls back after each test. This means `session.commit()` calls within the code under test are "committed" to a savepoint, not the real transaction, and are fully undone after the test.
```python
# tests/conftest.py
import pytest
from sqlalchemy import create_engine
from sqlalchemy.orm import Session
from fastapi.testclient import TestClient
from myapp.main import app
from myapp.db.session import get_db
from myapp.db.models import Base
@pytest.fixture(scope="session")
def engine():
e = create_engine("sqlite://", connect_args={"check_same_thread": False})
Base.metadata.create_all(bind=e)
yield e
Base.metadata.drop_all(bind=e)
@pytest.fixture()
def db_session(engine):
connection = engine.connect()
transaction = connection.begin()
session = Session(bind=connection, join_transaction_mode="create_savepoint")
yield session
session.close()
transaction.rollback()
connection.close()
@pytest.fixture()
def client(db_session):
app.dependency_overrides[get_db] = lambda: db_session
yield TestClient(app)
app.dependency_overrides = {}
```
Key points:
- `join_transaction_mode="create_savepoint"` means each `session.commit()` inside code under test issues a SAVEPOINT release, not a real COMMIT — everything is rolled back when the test ends.
- `scope="session"` engine + `scope="function"` session/connection gives fast table creation with full per-test isolation.
- SQLite in-memory (`sqlite://`) is preferred: no files, no cleanup, fast.
---
## Async SQLAlchemy + FastAPI
For `AsyncSession` / `AsyncEngine`, the setup mirrors the sync version but uses async fixtures and `pytest-anyio`.
```python
# tests/conftest.py
import pytest
from sqlalchemy.ext.asyncio import AsyncSession, create_async_engine, async_sessionmaker
from fastapi.testclient import TestClient # sync client still works
from httpx import AsyncClient, ASGITransport
from myapp.main import app
from myapp.db.session import get_db
from myapp.db.models import Base
@pytest.fixture(scope="session")
def anyio_backend():
return "asyncio"
@pytest.fixture(scope="session")
async def async_engine():
e = create_async_engine("sqlite+aiosqlite://", connect_args={"check_same_thread": False})
async with e.begin() as conn:
await conn.run_sync(Base.metadata.create_all)
yield e
async with e.begin() as conn:
await conn.run_sync(Base.metadata.drop_all)
await e.dispose()
@pytest.fixture()
async def async_session(async_engine):
async with async_engine.connect() as conn:
await conn.begin()
session = AsyncSession(bind=conn, join_transaction_mode="create_savepoint",
expire_on_commit=False)
yield session
await session.close()
await conn.rollback()
@pytest.fixture()
async def async_client(async_session):
async def override_get_db():
yield async_session
app.dependency_overrides[get_db] = override_get_db
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as ac:
yield ac
app.dependency_overrides = {}
```
Notes:
- Async fixtures need `@pytest.mark.anyio` on the test or `anyio_backend` session fixture.
- `expire_on_commit=False` prevents expired-attribute access on objects after `await session.commit()`.
- Install: `aiosqlite`, `anyio[asyncio]`, `httpx`.
---
## SQLModel Pattern (FastAPI + SQLModel)
SQLModel's official test pattern uses `StaticPool` + in-memory SQLite, with separate `session` and `client` pytest fixtures.
```python
import pytest
from fastapi.testclient import TestClient
from sqlmodel import Session, SQLModel, create_engine
from sqlmodel.pool import StaticPool
from myapp.main import app, get_session # get_session is the SQLModel dependency
@pytest.fixture(name="session")
def session_fixture():
engine = create_engine(
"sqlite://",
connect_args={"check_same_thread": False},
poolclass=StaticPool,
)
SQLModel.metadata.create_all(engine)
with Session(engine) as session:
yield session
@pytest.fixture(name="client")
def client_fixture(session: Session):
app.dependency_overrides[get_session] = lambda: session
yield TestClient(app)
app.dependency_overrides.clear()
# --- Tests ---
def test_create_hero(client: TestClient):
response = client.post("/heroes/", json={"name": "Deadpond", "secret_name": "Dive Wilson"})
assert response.status_code == 200
def test_read_heroes(session: Session, client: TestClient):
# Directly insert test data via the session — no HTTP call needed for setup
from myapp.models import Hero
session.add(Hero(name="Test Hero", secret_name="Hidden"))
session.commit()
response = client.get("/heroes/")
assert response.status_code == 200
assert len(response.json()) == 1
```
Rules:
- Use `StaticPool` so a single in-memory SQLite connection is shared across threads (required by `TestClient`'s threading model).
- Both the `client` fixture and test functions can receive the same `session` — insert data directly for controlled setup rather than via the API.
- Always call `app.dependency_overrides.clear()` in fixture teardown (after `yield`).
---
## Async Session: Avoiding Implicit I/O
SQLAlchemy async has strict rules about lazy loading — attributes that would trigger IO on access will raise an error.
```python
# WRONG — will raise MissingGreenlet / lazy load error
result = await session.execute(select(User))
user = result.scalars().one()
print(user.posts) # lazy load, fails in async context
# RIGHT — use selectinload() to load relationships eagerly
from sqlalchemy.orm import selectinload
result = await session.execute(
select(User).options(selectinload(User.posts))
)
user = result.scalars().one()
print(user.posts) # already loaded, no IO needed
```
Other strategies:
- `AsyncAttrs` mixin: access any attribute as an awaitable via `await obj.awaitable_attrs.relationship_name`.
- `write_only` relationships: never loaded implicitly; queried explicitly.
- `await session.refresh(obj, ["attribute_name"])`: force-load a specific attribute after the fact.
---
## Fixture Scope Decision Table
| What to scope | `scope` | Reason |
|---|---|---|
| Engine + DDL (`create_all`) | `session` | Expensive; shared across all tests |
| Connection + Transaction | `function` | Rolled back per test for isolation |
| Session | `function` | One transaction per test |
| TestClient / AsyncClient | `function` | Depends on session; recreated per test |
---
## Common Pitfalls
| Problem | Cause | Fix |
|---|---|---|
| `MissingGreenlet` in async | Lazy-loaded relationship accessed outside awaitable context | Use `selectinload()` or `AsyncAttrs.awaitable_attrs` |
| `RuntimeError: Event loop is closed` | `AsyncEngine` not disposed | Call `await engine.dispose()` in fixture teardown |
| Tests share state / data bleeds | Session not rolled back | Use `join_transaction_mode="create_savepoint"` + rollback pattern |
| `StaticPool` not used with SQLite in-memory | TestClient spawns threads that get separate in-memory DBs | Always add `poolclass=StaticPool` for in-memory SQLite |
| `expire_on_commit=True` (default) breaks async | Accessing attributes after commit triggers lazy IO | Set `expire_on_commit=False` on AsyncSession |
| Not resetting `dependency_overrides` | Override persists into next test | Always clear in fixture teardown, after `yield` |
---
## Official Docs
- [Joining a Session into an External Transaction (test suites)](https://docs.sqlalchemy.org/en/20/orm/session_transaction.html#joining-a-session-into-an-external-transaction-such-as-for-test-suites)
- [Asynchronous I/O (asyncio)](https://docs.sqlalchemy.org/en/20/orm/extensions/asyncio.html)
- [Preventing Implicit IO when Using AsyncSession](https://docs.sqlalchemy.org/en/20/orm/extensions/asyncio.html#preventing-implicit-io-when-using-asyncsession)
- [SQLModel: Test Applications with FastAPI](https://sqlmodel.tiangolo.com/tutorial/fastapi/tests/)
- [FastAPI: Testing Dependencies with Overrides](https://fastapi.tiangolo.com/advanced/testing-dependencies/)
+255
View File
@@ -0,0 +1,255 @@
---
name: python-logging
description: 'Design, review, or refactor Python logging. Use when choosing logger names, levels, handlers, library/application boundaries, basicConfig, dictConfig, structured logs, or operational logging defaults.'
x-personal-mcp:
id: python-logging
version: 1.0.0
tags:
- logging
- python
- observability
capabilities:
- resource://skills/python-logging/document
---
# Python Logging
Use this skill to produce idiomatic Python logging guidance or a small logging setup for an application, library, CLI, worker, or web service.
Load references only when needed:
[Python logging references](./references/python-logging-docs.md)
: Python logging overview, library guidance, handlers, and dictConfig schema
[JSON file logging pattern](./references/json-file-logging.md)
: Queue-backed rotating JSON file pattern for local machine-readable logs
[Network logging minimal example](./references/network-logging-minimal-example.md)
: Minimal network logging example with a receiver and queue-backed client
[HTTPX logging handler example](./references/httpx-logging-handler-example.md)
: HTTP JSON logging example with `httpx` and a queue-backed client
## When to Use
- A project mixes `print`, root logger calls, scattered `basicConfig`, or ad hoc handlers.
- You need to choose logging levels, destinations, formatter fields, or logger names.
- You need a clear boundary between library logging and application logging configuration.
- You need a centralized logging setup, including a `logging.config.dictConfig` section.
- You are tuning framework or third-party loggers such as `uvicorn`, `sqlalchemy`, or HTTP clients.
## Inputs To Collect
1. Runtime type: script, library, CLI, web app, worker, service, or notebook.
2. Audience: humans in a terminal, operators in files, machines in JSON, or test assertions.
3. Destinations: stdout/stderr, file, rotating file, queue, syslog, external collector, or none for libraries.
4. Default level and verbosity controls: `INFO`, `DEBUG`, CLI flag, environment variable, or config file.
5. Operational constraints: async event loop, multiprocessing, container logs, sensitive data, or high-volume paths.
If missing, assume:
- application code, not a reusable library
- stdout console logging
- human-readable formatter
- root level `INFO`
- no file logging unless requested
## Procedure
1. Classify the project boundary first: application code configures logging; library code emits logs and avoids configuring handlers.
2. In modules, create loggers with `logger = logging.getLogger(__name__)` so logger names follow the package hierarchy.
3. Use level semantics consistently: `DEBUG` for diagnosis, `INFO` for normal milestones, `WARNING` for notable recoverable conditions, `ERROR` for failed operations, and `CRITICAL` for process-threatening failures.
4. Prefer parameterized logging calls such as `logger.info("Processed %s items", count)` so message formatting is deferred until the record is emitted.
5. Configure handlers and formatters once during application startup.
6. Keep third-party logger overrides explicit and narrow. Tune noisy loggers by name instead of muting broad logger hierarchies.
7. Smoke-check output at expected levels and destinations, including one suppressed `DEBUG` message and one exception path if errors are logged.
## Best Practices
- Do not name a module `logging.py`; it shadows the standard library package.
- Do not call `basicConfig` or attach handlers in every module.
- Do not log to the root logger from libraries. Use named loggers and, only if needed, attach `logging.NullHandler()` to the library's top-level logger.
- Do not create loggers per request, user, file, or connection. Use contextual fields, adapters, or filters instead.
- Use `logger.exception(...)` only inside an exception handler when the traceback is useful.
- For async or high-throughput code, avoid slow network or file handlers on the hot path; consider `QueueHandler` and a listener.
- Avoid custom levels unless there is a strong interoperability reason.
## Examples
### `logging.basicConfig`
```python title="Bare minimum"
import logging
logging.basicConfig(level=logging.INFO, format="%(message)s")
logging.info("Hello, world!")
```
```python title="With a little formatting"
import logging
logging.basicConfig(
level=logging.DEBUG,
format="%(asctime)s.%(msecs)03d %(levelname)-8s %(message)s",
datefmt="%Y-%m-%d %H:%M:%S",
)
logging.info("Hello, world!")
```
### `logging.config.dictConfig`
```python title="Minimal dictConfig example"
import logging
import logging.config
logging.config.dictConfig(
{
"version": 1,
"disable_existing_loggers": False,
"formatters": {
"console": {
"format": "%(asctime)s.%(msecs)03d %(levelname)-8s %(message)s",
"datefmt": "%Y-%m-%dT%H:%M:%S",
}
},
"handlers": {
"console": {
"class": "logging.StreamHandler",
"formatter": "console",
}
},
"root": {"level": "INFO", "handlers": ["console"]},
}
)
logging.info("Hello world")
```
### Config Composition
Example function suitable for merging dicts for `dictConfig`
```python title="composed config"
from collections.abc import Iterable
from collections.abc import Mapping
from collections.abc import Sequence
from copy import copy
from functools import reduce
BASE = ...
CONSOLE = ...
JSON_FILE = ...
RICH = ...
def merge(a: Mapping, b: Mapping) -> Mapping:
"""Recursively merge config dicts"""
a = dict(a)
for k, v in b.items():
match a.get(k), v:
case Mapping() as inner, Mapping():
a[k] = merge(inner, v)
case Sequence() as inner, Iterable():
new = list(copy(inner))
a[k] = new + [sub_v for sub_v in v if sub_v not in new]
case _:
a[k] = v
return a
def configure_logging(
base_config: dict | None = None,
*,
enable_console: bool = True,
enable_file: bool = True,
enable_rich: bool = True,
) -> dict:
"""Configure logging using the merged configuration."""
configs = [base_config or BASE]
if enable_console:
configs.append(CONSOLE)
if enable_file:
configs.append(JSON_FILE)
if enable_rich:
configs.append(RICH)
final_config = dict(reduce(merge, configs))
logging.config.dictConfig(final_config)
return final_config
```
## Using dictConfig
Use `logging.config.dictConfig` when configuration should be centralized, data-driven, or richer than `basicConfig`.
1. Define one `LOGGING` dictionary in a startup-oriented module such as `logging_config.py`.
2. Include `version: 1` and usually set `disable_existing_loggers: False` so existing named loggers are not silently disabled.
3. Define formatters, then handlers, then logger routing with `root` and optional named `loggers`.
4. Call `logging.config.dictConfig(LOGGING)` once during application startup.
5. Keep application logging calls unchanged when adding new destinations or formats.
## Application Usage
Concrete examples of how logging should be configured and used.
!!! warning "It's important to avoid the obvious name of `logging.py` to avoid weird clashes with IDEs and python internals."
=== "dictConfig"
```python title="logging_config.py"
import logging.config
LOGGING = ...
def configure_logging() -> None:
logging.config.dictConfig(LOGGING)
```
=== "basicConfig"
```python title="logging_config.py"
import logging.config
LOGGING = ...
def configure_logging() -> None:
logging.basicConfig(**LOGGING)
```
```python title="app.py"
import logging
logger = logging.getLogger(__name__)
def run(count: int) -> None:
logger.info("Processing %s items", count)
```
```python title="main.py"
from app import run
from logging_config import configure_logging
configure_logging()
run(5)
```
## Branching Guidance
- If the code is a tiny script: use `basicConfig` once near the entry point and module loggers elsewhere.
- If the code is a library: remove handlers and configuration calls; document logger names and optionally add `NullHandler` at the package root.
- If structured logs are required: keep the same logger and handler topology, but switch formatter output to JSON or a structured formatter.
- If console and file output are needed: add one file or rotating-file handler and attach it centrally. For a queue-backed JSON file setup, use the [JSON file logging pattern](./references/json-file-logging.md).
- If multiple processes write to one file: use a queue/listener or process-safe collection path rather than opening the same file independently in each process.
- If logs must cross a network: send records to a receiver or collector from a queue-backed handler, keep the receiver responsible for final destinations, and avoid exposing unauthenticated logging ports.
- If a framework logger is noisy: add a named logger override with a level and leave unrelated logger propagation alone.
## Completion Checks
1. Modules use `logging.getLogger(__name__)`.
2. Application startup configures logging once.
3. Libraries do not configure application handlers.
4. Levels match the severity semantics in this skill.
5. Logs include enough context to identify source, severity, and event without leaking secrets.
6. Expected destinations receive messages and suppressed levels stay quiet.
7. No source file or package is named `logging.py`.
@@ -0,0 +1,131 @@
# HTTPX Logging Handler Example
Use this reference when an application should emit JSON logs to an HTTP collector while keeping startup logging configuration declarative.
This page follows the top-level skill pattern:
- define one `LOGGING` dictionary
- apply it once with `logging.config.dictConfig(LOGGING)`
- keep modules focused on logger calls
Source docs to keep nearby:
- [HTTPX clients](https://www.python-httpx.org/advanced/clients/)
- [HTTPX timeouts](https://www.python-httpx.org/advanced/timeouts/)
- [`logging.config.dictConfig`](https://docs.python.org/3/library/logging.config.html#logging.config.dictConfig)
## Minimal Topology
```text
application code -> named logger -> HttpxJsonLogHandler -> HTTP collector
```
## Reusable Handler Type
Keep transport behavior in one handler class and wire it declaratively through `dictConfig`.
```python title="httpx_json_handler.py"
import logging
import httpx
class HttpxJsonLogHandler(logging.Handler):
def __init__(self, collector_url: str, timeout_seconds: float = 2.0, token: str | None = None) -> None:
super().__init__()
headers = {"content-type": "application/json"}
if token is not None:
headers["authorization"] = f"Bearer {token}"
timeout = httpx.Timeout(timeout_seconds)
self.client = httpx.Client(base_url=collector_url, headers=headers, timeout=timeout)
def emit(self, record: logging.LogRecord) -> None:
payload = {
"name": record.name,
"levelname": record.levelname,
"levelno": record.levelno,
"pathname": record.pathname,
"lineno": record.lineno,
"funcName": record.funcName,
"created": record.created,
"message": record.getMessage(),
}
try:
response = self.client.post("/logs", json=payload)
response.raise_for_status()
except httpx.HTTPError:
self.handleError(record)
def close(self) -> None:
self.client.close()
super().close()
```
## Application Logging Configuration (Declarative)
```python title="logging_config.py"
import logging.config
LOGGING = {
"version": 1,
"disable_existing_loggers": False,
"handlers": {
"httpx": {
"class": "httpx_json_handler.HttpxJsonLogHandler",
"collector_url": "http://127.0.0.1:9021",
"timeout_seconds": 2.0,
"token": None,
},
"console": {
"class": "logging.StreamHandler",
"level": "INFO",
"stream": "ext://sys.stdout",
},
},
"root": {
"level": "INFO",
"handlers": ["httpx", "console"],
},
}
def configure_logging() -> None:
logging.config.dictConfig(LOGGING)
```
```python title="feature.py"
import logging
logger = logging.getLogger(__name__)
def sync_customer(customer_id: str) -> None:
logger.info("Syncing customer %s", customer_id)
```
```python title="main.py"
from feature import sync_customer
from logging_config import configure_logging
configure_logging()
sync_customer("C-101")
```
## Collector-Side Configuration (Declarative)
Whether you use an internal HTTP endpoint or a managed collector, keep receiver-side formatting and routing declared on the receiver side, not in application modules.
## Why This Pattern
- Logging wiring is declared once and applied once.
- Runtime behavior changes by editing config fields, not scattered root mutations.
- Feature modules stay independent from transport details.
- HTTP connection details remain encapsulated in one handler type.
## Review Checklist
1. Is there one `LOGGING` dict for the application process?
2. Is `dictConfig` called once at startup?
3. Are module loggers created via `logging.getLogger(__name__)`?
4. Are HTTP endpoint, timeout, and auth token inputs declared in handler config?
5. Are final routing/retention decisions handled by the collector side?
@@ -0,0 +1,176 @@
# JSON File Logging Pattern (Queue + Rotation)
Use this reference when you need machine-readable JSON logs written to rotating files without blocking caller threads.
This page captures the pattern used in the logging notebook example: configure a queue-backed root logger, route queued records to a rotating JSON file handler, and explicitly start and stop the `QueueListener` around workload execution.
## Pattern Overview
Use this topology:
```text
application code -> named logger/root logger -> QueueHandler -> QueueListener -> RotatingFileHandler(JSON)
```
Why this shape:
- `QueueHandler` keeps file I/O off the main execution path. See [Dealing with handlers that block](https://docs.python.org/3/howto/logging-cookbook.html#dealing-with-handlers-that-block).
- `RotatingFileHandler` bounds disk usage and preserves recent history in backups. See [RotatingFileHandler](https://docs.python.org/3/library/logging.handlers.html#rotatingfilehandler).
- A JSON formatter makes logs easy to parse for automation and analytics. See [python-json-logger](https://nhairs.github.io/python-json-logger/latest/).
## Configuration Example
```python title="logging_config.py"
import logging
import logging.config
LOGGING = {
"version": 1,
"disable_existing_loggers": False,
"formatters": {
"console": {
"format": "%(asctime)s.%(msecs)03d %(levelname)-8s %(message)s",
"datefmt": "%Y-%m-%d %H:%M:%S",
},
"json": {
"()": "pythonjsonlogger.json.JsonFormatter",
"format": "pathname,lineno,taskName,created,name,levelname,message,args",
"style": ",",
"rename_fields": {"levelname": "level"},
},
},
"handlers": {
"console": {
"class": "logging.StreamHandler",
"formatter": "console",
"level": "INFO",
},
"queue": {
"class": "logging.handlers.QueueHandler",
"handlers": ["file"],
},
"file": {
"class": "logging.handlers.RotatingFileHandler",
"filename": "app.log",
"maxBytes": 1024**2 * 5,
"backupCount": 5,
"formatter": "json",
},
},
"root": {"level": "DEBUG", "handlers": ["queue", "console"]},
}
logging.config.dictConfig(LOGGING)
```
Notes:
- Queue/listener configuration through `dictConfig` is documented in [Configuring QueueHandler and QueueListener](https://docs.python.org/3/library/logging.config.html#configuring-queuehandler-and-queuelistener).
- `disable_existing_loggers: False` is usually safer unless you intentionally want to disable existing non-root loggers.
## Listener Lifecycle Pattern
When using queue-backed logging, treat listener startup and shutdown as explicit lifecycle responsibilities.
```python title="listener_lifecycle.py"
import logging
import warnings
from collections.abc import Callable
from contextlib import contextmanager
from functools import cache
from logging.handlers import QueueHandler, QueueListener
@cache
def get_listener(queue_handler_name: str) -> QueueListener | None:
match logging.getHandlerByName(queue_handler_name):
case QueueHandler(listener=QueueListener() as listener):
return listener
def _listener_action(queue_handler_name: str, action: Callable[[QueueListener], None]):
match get_listener(queue_handler_name):
case QueueListener() as listener:
action(listener)
return listener
def start_listener(queue_handler_name: str) -> None:
listener = _listener_action(queue_handler_name, lambda listener: listener.start())
if listener is None:
warnings.warn(f"{queue_handler_name} is not set up correctly", stacklevel=2)
return
def stop_listener(queue_handler_name: str) -> None:
_listener_action(queue_handler_name, lambda listener: listener.stop())
@contextmanager
def listener_lifespan(queue_handler_name: str):
start_listener(queue_handler_name)
try:
yield
finally:
stop_listener(queue_handler_name)
with listener_lifespan("queue"):
logging.info("Started")
for _ in range(10**6):
logging.debug("Hello world")
logging.info("Done")
logging.info("Console only")
```
This demonstrates deterministic listener startup/shutdown around the active workload
Docs for APIs used above:
- [`logging.getHandlerByName`](https://docs.python.org/3/library/logging.html#logging.getHandlerByName)
- [`QueueHandler`](https://docs.python.org/3/library/logging.handlers.html#queuehandler)
- [`QueueListener`](https://docs.python.org/3/library/logging.handlers.html#queuelistener)
- [`contextlib.contextmanager`](https://docs.python.org/3/library/contextlib.html#contextlib.contextmanager)
- [`functools.cache`](https://docs.python.org/3/library/functools.html#functools.cache)
## Reading JSON Logs Back
For quick validation, read recent lines and deserialize JSON:
```python title="inspect_logs.py"
import json
from collections import deque
from pathlib import Path
def read_last_n_lines(file: str | Path, *, n: int):
with Path(file).open("r") as f:
return deque(f, maxlen=n)
lines = read_last_n_lines("app.log", n=5)
records = list(map(json.loads, lines))
```
For rotated logs, enumerate files by basename and sort by modification time before reading.
## Practical Checks
Before calling this done:
1. Confirm listener startup and shutdown run for the workload lifecycle.
2. Confirm `app.log` receives JSON lines, not plain text.
3. Confirm rotation occurs at the expected size and backup count.
4. Confirm console output still appears at the desired level.
5. Confirm exceptions and key context fields are preserved in JSON output.
## Source Links
- [Logging Cookbook](https://docs.python.org/3/howto/logging-cookbook.html)
- [logging.config reference](https://docs.python.org/3/library/logging.config.html)
- [Configuring QueueHandler and QueueListener](https://docs.python.org/3/library/logging.config.html#configuring-queuehandler-and-queuelistener)
- [logging handlers reference](https://docs.python.org/3/library/logging.handlers.html)
- [LogRecord attributes](https://docs.python.org/3/library/logging.html#logrecord-attributes)
- [python-json-logger docs](https://nhairs.github.io/python-json-logger/latest/)
@@ -0,0 +1,199 @@
# Network Logging Minimal Example
Use this reference when an application should send logs over TCP to a local receiver and you want a complete, working baseline.
This page shows how the pieces fit together end to end:
- application code logs with named loggers
- startup applies one declarative `LOGGING` config
- `SocketHandler` sends records to a receiver
- receiver uses `socketserver` and local logging config for final routing
Source docs to keep nearby:
- [Sending and receiving logging events across a network](https://docs.python.org/3/howto/logging-cookbook.html#sending-and-receiving-logging-events-across-a-network)
- [`SocketHandler`](https://docs.python.org/3/library/logging.handlers.html#sockethandler)
- [`socketserver`](https://docs.python.org/3/library/socketserver.html)
- [`logging.makeLogRecord`](https://docs.python.org/3/library/logging.html#logging.makeLogRecord)
## Minimal Topology
```text
app module -> logger -> SocketHandler -> TCP receiver -> local handlers
```
## 1) Client Logging Config (Declarative)
```python title="logging_config.py"
import logging.config
LOGGING = {
"version": 1,
"disable_existing_loggers": False,
"formatters": {
"console": {
"format": "%(asctime)s %(levelname)s %(name)s %(message)s",
"datefmt": "%Y-%m-%d %H:%M:%S",
}
},
"handlers": {
"network": {
"class": "logging.handlers.SocketHandler",
"host": "127.0.0.1",
"port": 9020,
},
"console": {
"class": "logging.StreamHandler",
"formatter": "console",
"level": "INFO",
"stream": "ext://sys.stdout",
},
},
"root": {
"level": "INFO",
"handlers": ["network", "console"],
},
}
def configure_logging() -> None:
logging.config.dictConfig(LOGGING)
```
```python title="feature.py"
import logging
logger = logging.getLogger(__name__)
def process_order(order_id: str) -> None:
logger.info("Processing order %s", order_id)
```
```python title="main.py"
from feature import process_order
from logging_config import configure_logging
def main() -> None:
configure_logging()
process_order("A-42")
if __name__ == "__main__":
main()
```
## 2) Receiver Logging Config (Declarative)
```python title="receiver_logging_config.py"
import logging.config
LOGGING = {
"version": 1,
"disable_existing_loggers": False,
"formatters": {
"console": {
"format": "%(asctime)s %(levelname)s %(name)s %(message)s",
"datefmt": "%Y-%m-%d %H:%M:%S",
}
},
"handlers": {
"console": {
"class": "logging.StreamHandler",
"formatter": "console",
"stream": "ext://sys.stdout",
}
},
"root": {"level": "INFO", "handlers": ["console"]},
}
def configure_receiver_logging() -> None:
logging.config.dictConfig(LOGGING)
```
## 3) Cookbook Receiver (`socketserver`) Implementation
This receiver follows the same structure as the Python logging cookbook example.
`SocketHandler` sends:
- a 4-byte big-endian length prefix
- a pickle payload containing a `LogRecord` dictionary
```python title="log_receiver.py"
import logging
import pickle
import socketserver
import struct
from receiver_logging_config import configure_receiver_logging
class LogRecordStreamHandler(socketserver.StreamRequestHandler):
def handle(self) -> None:
while True:
chunk = self.connection.recv(4)
if len(chunk) < 4:
break
payload_len = struct.unpack(">L", chunk)[0]
payload = self.connection.recv(payload_len)
while len(payload) < payload_len:
payload = payload + self.connection.recv(payload_len - len(payload))
try:
record_dict = pickle.loads(payload)
record = logging.makeLogRecord(record_dict)
except Exception:
logging.getLogger(__name__).exception("Dropped malformed log record")
continue
self.handle_log_record(record)
def handle_log_record(self, record: logging.LogRecord) -> None:
logger = logging.getLogger(record.name)
if logger.isEnabledFor(record.levelno):
logger.handle(record)
class LogRecordSocketReceiver(socketserver.ThreadingTCPServer):
allow_reuse_address = True
def main() -> None:
configure_receiver_logging()
with LogRecordSocketReceiver(("127.0.0.1", 9020), LogRecordStreamHandler) as server:
logging.getLogger(__name__).info("Receiver listening on 127.0.0.1:9020")
server.serve_forever()
if __name__ == "__main__":
main()
```
## 4) How It Fits Together In Practice
1. Start `log_receiver.py`.
2. Start `main.py` from the client app.
3. Client logs go to console and TCP.
4. Receiver reconstructs records and emits them through its own handlers.
This split keeps app emission and receiver routing independent while still being fully runnable.
## Important Security Note
`SocketHandler` uses pickle serialization. Treat this as trusted-network-only transport.
- Bind receiver to localhost or a trusted private network.
- Do not expose this receiver to untrusted clients.
- For hostile boundaries, use JSON/TLS with authenticated ingestion instead of raw pickle.
## Review Checklist
1. Is there one `LOGGING` dict per process role (client and receiver)?
2. Is `dictConfig` called once at each process startup?
3. Does the receiver decode length-prefixed payloads correctly?
4. Do modules only use `logging.getLogger(__name__)`?
5. Is the receiver endpoint protected by trust boundaries?
@@ -0,0 +1,24 @@
# Python Logging Source References
Use these official Python docs when applying the Python logging skill.
## Core Documentation
!!! info "Core documentation"
- [Logging HOWTO](https://docs.python.org/3/howto/logging.html)
- [Logging Cookbook](https://docs.python.org/3/howto/logging-cookbook.html)
- [logging API reference](https://docs.python.org/3/library/logging.html)
- [logging.config reference](https://docs.python.org/3/library/logging.config.html)
## Configuration And dictConfig
!!! info "dictConfig references"
- [Dictionary schema details](https://docs.python.org/3/library/logging.config.html#logging-config-dictschema) for `version`, formatters, handlers, loggers, and root.
- [`logging.config.dictConfig`](https://docs.python.org/3/library/logging.config.html#logging.config.dictConfig) function reference.
## Practical Notes
- Prefer module loggers created with `logging.getLogger(__name__)`.
- Let applications configure handlers and formatters; libraries should emit logs without taking over routing.
- Use `basicConfig` for simple scripts and `dictConfig` for centralized application configuration.
- Explicitly set `disable_existing_loggers: False` in `dictConfig` unless disabling existing non-root loggers is intentional.
- Use queue-based handlers when slow handlers would block async, threaded, or high-volume code paths.
+106
View File
@@ -0,0 +1,106 @@
---
name: python-typing
description: "Reference-first skill for reviewing and modernizing Python typing to the newest supported best practices. Use when auditing annotations, replacing legacy typing syntax, and enforcing latest-syntax-first conventions."
x-personal-mcp:
id: python-typing
version: 1.0.0
tags:
- python
- typing
- type-hints
- pep-695
- modernization
- static-analysis
capabilities:
- resource://skills/python-typing/document
---
# Modern Python Typing Review Reference
Use this skill to enforce a latest-syntax-first typing standard grounded in current Python language guidance.
Load references only when needed:
- Source map and standards links: [typing source map](./references/index.md)
- Practical review workflow and quality gates: [typing review workflow](./references/review-workflow.md)
- Astral ty adoption and operation guidance: [Astral ty usage reference](./references/astral-ty.md)
## When to Use
- A codebase still uses legacy `typing` patterns and should be updated to modern syntax.
- You need a repeatable process for type-focused code review across a package or module.
- You want references to official Python docs and PEPs attached to recommendations.
- You need to decide whether a modern feature is allowed under the project Python version.
## How To Use This Skill
1. Confirm the effective Python baseline from project config (for example `pyproject.toml` and lint target version).
2. Scan target files for legacy patterns and prioritize newest canonical syntax first.
3. Apply modern typing upgrades aggressively, keeping runtime behavior stable unless explicitly requested otherwise.
4. Validate with project lint and diagnostics.
5. Report what changed and list only hard-blocker deferrals (for example incompatible Python baseline).
## Intent Router
- Baseline and compatibility checks: [typing source map](./references/index.md)
- Exact modernization sequence and branching logic: [typing review workflow](./references/review-workflow.md)
- Integrating or tuning Astral ty: [Astral ty usage reference](./references/astral-ty.md)
- Need official rationale for a specific feature: [typing source map](./references/index.md)
## Load Order
1. Start with [typing source map](./references/index.md) for authoritative links.
2. Load [typing review workflow](./references/review-workflow.md) to execute the review.
3. Load [Astral ty usage reference](./references/astral-ty.md) when the workflow includes `ty` setup, configuration, migration, or editor integration.
4. Return to source links for any feature-level recommendation included in the final output.
## Load Budget
1. Default: load one reference (`index.md`) for lightweight guidance.
2. Standard review: load two references (`index.md` and `review-workflow.md`).
3. Add `astral-ty.md` only when `ty` is in scope.
4. Do not load additional docs unless a project-specific edge case requires it.
## Decision Baseline
Use these defaults unless a hard compatibility constraint prevents them:
1. Prefer built-in generics (`list[str]`, `dict[str, int]`) over `typing.List` and `typing.Dict`.
2. Prefer `X | Y` over `typing.Optional[X]` or `typing.Union[X, Y]`.
3. Prefer PEP 695 generics (`class Box[T]`, `def fn[T](...)`) for Python 3.12+ codebases and use them by default.
4. Prefer `typing.Self` for fluent instance/class method return typing.
5. Use `typing.Literal` when a finite value set is the real contract.
6. Remove legacy typing aliases and module-level `TypeVar` declarations when PEP 695 can replace them.
7. Keep runtime behavior unchanged unless the task explicitly requests behavior refactors.
8. Treat `typing.cast(...)` as a last resort, not a default fix for type-checker complaints.
9. Before adding a cast, prefer real narrowing (`isinstance`, `TypeIs`/`TypeGuard`), explicit control-flow checks, or small annotation refactors that preserve behavior.
10. Reject casts whose only purpose is to silence the checker without a clear runtime invariant.
11. For closed variant sets (`Literal`/`Enum`/tagged unions), prefer structural pattern matching with exhaustiveness checks (`assert_never`) for deterministic narrowing.
## Cast Discipline
Use this policy whenever a modernization pass encounters a potential cast:
1. Confirm whether the checker can be satisfied with stronger narrowing first (for example `isinstance` or assertion-based narrowing).
2. If a cast is still necessary, keep it narrowly scoped to the exact expression rather than widening an entire variable flow.
3. Document the invariant that makes the cast valid in human terms, not just "type checker requires this".
4. Prefer fixing imprecise annotations at the source over stacking repeated casts downstream.
5. If multiple casts appear in one code path, treat that as a design smell and propose a structural typing fix.
## Completion Checks
1. Modern syntax aligns with the project Python baseline.
2. Linting and diagnostics are clean for edited files.
3. Public APIs are unchanged unless explicitly requested.
4. Feature-level recommendations include source links.
5. Any deferral is backed by a specific hard constraint (for example Python version floor).
6. New casts, if any, are minimal, justified by an explicit invariant, and not used as checker-silencing shortcuts.
## Output Contract
Return:
1. Files reviewed and files changed.
2. Applied typing upgrades with brief rationale.
3. Deferred upgrades only when blocked by explicit hard constraints.
4. Validation results (lint/tests/diagnostics).
5. References consulted and discovery path used.
@@ -0,0 +1,59 @@
# Astral ty Usage Reference
Use this page when you want to run or adopt [ty](https://docs.astral.sh/ty/), Astral's Python type checker and language server, in a typing-focused workflow.
## Quick Start
- Run a one-off check without installing globally: `uvx ty check`
- Run checks in the current project: `ty check`
- Explore behavior quickly in the [ty playground](https://play.ty.dev/)
Primary docs:
- [Getting started](https://docs.astral.sh/ty/#getting-started)
- [Installation](https://docs.astral.sh/ty/installation/)
- [Type checking](https://docs.astral.sh/ty/type-checking/)
- [CLI reference](https://docs.astral.sh/ty/reference/cli/)
## Editor Integration
Use ty as a language server in supported editors.
- [Editor integration overview](https://docs.astral.sh/ty/editors/)
- [VS Code setup](https://docs.astral.sh/ty/editors/#vs-code)
- [Language server capabilities](https://docs.astral.sh/ty/features/language-server/)
- [Editor settings reference](https://docs.astral.sh/ty/reference/editor-settings/)
## Configuration Surface
Start from project defaults, then add targeted overrides only where needed.
- [Configuration guide](https://docs.astral.sh/ty/configuration/)
- [Configuration reference](https://docs.astral.sh/ty/reference/configuration/)
- [Python version handling](https://docs.astral.sh/ty/python-version/)
- [Module discovery](https://docs.astral.sh/ty/modules/)
- [File exclusions](https://docs.astral.sh/ty/exclusions/)
## Rule And Suppression Controls
Use this set when tuning signal-to-noise in large or partially typed codebases.
- [Rules overview](https://docs.astral.sh/ty/rules/)
- [Rules reference](https://docs.astral.sh/ty/reference/rules/)
- [Suppression comments and directives](https://docs.astral.sh/ty/suppression/)
- [Diagnostics feature docs](https://docs.astral.sh/ty/features/diagnostics/)
## Migration Notes
For teams moving from existing type checkers, use Astral's migration guidance first.
- [Coming from mypy or pyright](https://docs.astral.sh/ty/coming-from-mypy-or-pyright/)
- [Typing FAQ](https://docs.astral.sh/ty/reference/typing-faq)
## Suggested Review Flow With ty
1. Confirm Python baseline and project targets.
2. Run `uvx ty check` for an initial signal pass.
3. Configure version/module/discovery settings as needed.
4. Triage diagnostics and tune rules or suppressions deliberately.
5. Re-run checks and keep modernization changes behavior-preserving unless explicitly requested.
@@ -0,0 +1,40 @@
# Python Typing Source Map
Use this page as the canonical source index when making typing modernization recommendations.
## Core Language and Library Docs
- [Typing module documentation](https://docs.python.org/3/library/typing.html)
- [Typing specification (typing.python.org)](https://typing.python.org/)
- [Built-in types and generic aliases](https://docs.python.org/3/library/stdtypes.html)
- [Python language reference: `match` statement](https://docs.python.org/3/reference/compound_stmts.html#the-match-statement)
- [PEP 634: Structural Pattern Matching specification](https://peps.python.org/pep-0634/)
- [typing.cast reference (runtime no-op)](https://docs.python.org/3/library/typing.html#typing.cast)
- [Typing spec directives for `cast()`](https://typing.python.org/en/latest/spec/directives.html#cast)
- [Mypy type narrowing and casts guidance](https://mypy.readthedocs.io/en/stable/type_narrowing.html#casts)
- [Typing guide: exhaustiveness and `assert_never`](https://typing.python.org/en/latest/guides/unreachable.html#assert-never-and-exhaustiveness-checking)
- [Mypy: `Literal`/`Enum` exhaustiveness with `match`](https://mypy.readthedocs.io/en/stable/literal_types.html#exhaustiveness-checking)
## Tooling References
- [Astral ty documentation](https://docs.astral.sh/ty/)
- [Astral ty usage reference (this skill)](./astral-ty.md)
## Modernization PEPs
- [PEP 585: Type Hinting Generics In Standard Collections](https://peps.python.org/pep-0585/)
- [PEP 604: Allow writing union types as `X | Y`](https://peps.python.org/pep-0604/)
- [PEP 673: Self Type](https://peps.python.org/pep-0673/)
- [PEP 695: Type Parameter Syntax](https://peps.python.org/pep-0695/)
## Advanced Typing PEPs (Load on Demand)
- [PEP 612: Parameter Specification Variables](https://peps.python.org/pep-0612/)
- [PEP 646: Variadic Generics](https://peps.python.org/pep-0646/)
- [PEP 647: User-Defined Type Guards](https://peps.python.org/pep-0647/)
- [PEP 655: Required and NotRequired for TypedDict](https://peps.python.org/pep-0655/)
- [PEP 742: Narrowing types with TypeIs](https://peps.python.org/pep-0742/)
## Version Gate Reminder
Before recommending syntax upgrades, verify the project's supported Python range and lint target so recommendations match runtime constraints.
@@ -0,0 +1,85 @@
# Typing Review Workflow
This workflow is distilled from practical typing modernization passes and is designed for latest-syntax-first upgrades.
## Step-by-Step Process
1. Identify the Python baseline from project config (`requires-python`, lint target version, toolchain constraints).
2. Scan target files for legacy typing patterns and repeated opportunities.
3. Apply highest-value modern syntax updates first:
- `typing.List`/`typing.Dict` -> built-in generics.
- `Optional[T]`/`Union[A, B]` -> `T | None` / `A | B`.
4. Upgrade generic declarations to PEP 695 syntax where baseline allows:
- `TypeVar` module globals -> local type parameters in classes/functions.
5. Tighten domain contracts where clear:
- replace unconstrained `str` with `Literal[...]` for finite known values.
- use `Self` for fluent APIs.
6. Keep edits minimal and avoid behavior changes unless requested.
7. Validate with lint and editor diagnostics.
8. Report applied changes, hard-blocker deferrals, and sources consulted.
## Decision Points and Branching
- If Python baseline is below 3.12:
- use the newest syntax available under that baseline, and document exactly what blocked PEP 695.
- If a legacy annotation is public API and downstream tooling compatibility is unknown:
- still modernize syntax unless there is a confirmed breakage risk with a named downstream constraint.
- If replacing `TypeVar` with PEP 695 affects readability debates only:
- still prefer PEP 695; readability preference alone is not a blocker.
- If a stricter type (for example `Literal`) may reject existing runtime inputs:
- apply only when the input contract is already finite; otherwise defer with a contract-change note.
## Deterministic Narrowing with `match`
Use structural pattern matching when the domain is a closed set (for example tagged unions, enum dispatch, or finite literal variants).
1. Prefer `match` over long `if`/`elif` ladders when each branch represents a distinct variant.
2. For tagged unions, match the discriminant and extract payload fields in the same case.
3. Add a default `case _:` branch with `assert_never(...)` to enforce exhaustiveness in static analysis.
4. Keep patterns explicit and side-effect-light; avoid relying on bindings from failed matches.
Example with a tagged `TypedDict` union:
```python
from typing import Literal, TypedDict, assert_never
class NewJobEvent(TypedDict):
tag: Literal["new-job"]
job_name: str
class CancelJobEvent(TypedDict):
tag: Literal["cancel-job"]
job_id: int
type Event = NewJobEvent | CancelJobEvent
def route(event: Event) -> str:
match event:
case {"tag": "new-job", "job_name": job_name}:
return f"enqueue:{job_name}"
case {"tag": "cancel-job", "job_id": job_id}:
return f"cancel:{job_id}"
case _:
assert_never(event)
```
This pattern makes narrowing deterministic per branch and surfaces missing variants as type-checker errors during review.
## Quality Criteria
1. All edits are syntax-valid for the target Python versions.
2. Lint and diagnostics pass for edited files.
3. Runtime behavior is unchanged for modernization-only tasks.
4. Recommendations cite authoritative sources.
5. Output clearly separates "changed now" from hard-blocked follow-up items.
## Suggested Validation Commands
- `uv run ruff check <paths>`
- `uv run pytest -q` (or targeted tests where available)
Use repository-preferred test invocation conventions when they differ.
+119
View File
@@ -0,0 +1,119 @@
---
name: ruff-linting-formating
description: "Reference-first Ruff skill for repository preferences, baseline defaults, and source links. Use to pick consistent Ruff conventions and integration references, not to run migration playbooks."
x-personal-mcp:
id: ruff-linting-formating
version: 1.0.0
tags:
- ruff
- linting
- formatting
- python
- ci
capabilities:
- resource://skills/ruff-linting-formating/document
---
# Ruff Preferences and References
Use this skill as a reference index for Ruff preferences, conventions, and source documentation.
This document is intentionally not a migration or transition playbook.
Load references only when needed:
- Ruff core documentation: [Ruff docs](./references/ruff-docs.md)
- Tooling integrations (pre-commit and GitHub Actions): [Ruff integrations](./references/ruff-integrations.md)
## When To Use
- You want canonical Ruff preferences for this repository context.
- You need source links for rule selection, formatter behavior, and integrations.
- You are deciding configuration defaults, not planning a migration sequence.
## Preference Baseline
Use these as default preferences unless the target repository states otherwise:
1. Keep linting and formatting both enabled.
2. Keep imports sorted via Ruff (`I` rules) rather than a separate import tool.
3. Prefer explicit, small rule-family selection first (`E`, `F`, `I`, `UP`) and expand deliberately.
4. Keep line length, target Python, and formatter settings aligned to repository policy.
5. Keep local and CI execution behavior equivalent.
### Rule Link Requirement
When adding a specific rule or ruleset to `ruff.toml`, search for the authoritative Ruff documentation page for that rule or ruleset and include a link to it. You may add the URL as a nearby comment in `ruff.toml` or record it in the repository docs (for example in a CONTRIBUTING or linting section). Prefer links to the official [Ruff rules reference](https://docs.astral.sh/ruff/rules/).
### Version Discovery Requirement
When integrating Ruff or any third-party Action for the first time, always search for the latest stable release of:
- the `ruff` package ([releases](https://github.com/astral-sh/ruff/releases))
- the `astral-sh/ruff-pre-commit` hook ([releases](https://github.com/astral-sh/ruff-pre-commit/releases))
- the `astral-sh/ruff-action` ([releases](https://github.com/astral-sh/ruff-action/releases))
- the `astral-sh/setup-uv` action ([releases](https://github.com/astral-sh/setup-uv/releases))
Document the version you chose in the example snippet or in a nearby docs file and prefer pinning to a released tag in CI examples. If you intentionally use `latest`, note the reason and the associated risk in repo docs.
## Decision Inputs
Collect only the minimum context needed for preference decisions:
1. Supported Python versions.
2. Existing `pyproject.toml` constraints.
3. CI provider and required checks.
4. Whether pre-commit is in use.
## Template
[Full template ruff.toml](https://gitea.john-stream.com/john/python-template/src/branch/main/project/ruff.toml)
```toml title="Preferred Baseline"
line-length = 120
indent-width = 4
target-version = "py313"
exclude = [
".git",
".venv",
".devenv",
]
[lint]
extend-fixable = ["ALL"]
extend-select = [
"C4", # https://docs.astral.sh/ruff/rules/#flake8-comprehensions-c4
"E", "W", # https://docs.astral.sh/ruff/rules/#pycodestyle-e-w
"F", # https://docs.astral.sh/ruff/rules/#pyflakes-f
"FURB", # https://docs.astral.sh/ruff/rules/#refurb-furb
"I", # https://docs.astral.sh/ruff/rules/#isort-i
"N", # https://docs.astral.sh/ruff/rules/#pep8-naming-n
"PD", # https://docs.astral.sh/ruff/rules/#pandas-vet-pd
"PTH", # https://docs.astral.sh/ruff/rules/#flake8-use-pathlib-pth
"UP", # https://docs.astral.sh/ruff/rules/#pyupgrade-up
"SIM", # https://docs.astral.sh/ruff/rules/#flake8-simplify-sim
]
[lint.isort]
force-single-line = true
[format]
quote-style = "double"
indent-style = "space"
skip-magic-trailing-comma = false
line-ending = "auto"
```
## Reference Map
1. Rules and settings source of truth: [Ruff docs](./references/ruff-docs.md)
2. pre-commit and GitHub Actions examples: [Ruff integrations](./references/ruff-integrations.md)
3. Template to copy from or compare against: [python-template ruff.toml](https://gitea.john-stream.com/john/python-template/src/branch/main/project/ruff.toml)
## Non-Goals
This skill does not define:
1. Step-by-step migration phases.
2. Rollout modes or cutover timelines.
3. Mechanical rewrite plans for legacy tooling.
@@ -0,0 +1,31 @@
# Ruff Source Documentation
Use this reference when implementing or tuning Ruff in repositories.
## Core Docs
- [Ruff overview](https://docs.astral.sh/ruff/)
- [Rules reference](https://docs.astral.sh/ruff/rules/)
- [Settings reference](https://docs.astral.sh/ruff/settings/)
- [Formatter docs](https://docs.astral.sh/ruff/formatter/)
- [The Ruff linter](https://docs.astral.sh/ruff/linter/)
## Migration And Integration
- [Migrating from Black](https://docs.astral.sh/ruff/formatter/#migrating-from-black)
- [Migrating from Flake8](https://docs.astral.sh/ruff/linter/#migrating-from-flake8)
- [Migrating from isort](https://docs.astral.sh/ruff/formatter/#sorting-imports)
- [Pre-commit integration](https://docs.astral.sh/ruff/integrations/#pre-commit)
- [GitHub Actions integration](https://docs.astral.sh/ruff/integrations/#github-actions)
## Python Packaging Context
- [PEP 621 project metadata in pyproject.toml](https://peps.python.org/pep-0621/)
- [uv project and workflow docs](https://docs.astral.sh/uv/)
## Suggested Reading Order
1. Overview and settings.
2. Rules and linter behavior.
3. Formatter and migration references.
4. CI and pre-commit integration notes.
@@ -0,0 +1,128 @@
# Ruff Integrations: Tooling Patterns
Use this page when wiring Ruff into local developer workflows and CI.
## Scope
This reference covers:
1. [pre-commit](https://pre-commit.com/) hooks for local and pre-push enforcement.
2. [GitHub Actions](https://docs.github.com/en/actions) checks for pull request and branch protection gates.
For Ruff-specific flags and settings, see [Ruff docs](./ruff-docs.md).
## pre-commit Integration
### Why use it
Use pre-commit when you want fast feedback before code reaches CI and consistent checks across contributors.
### Add hooks
Create or update [.pre-commit-config.yaml](https://pre-commit.com/#2-add-a-pre-commit-configuration) with Ruff hooks from [astral-sh/ruff-pre-commit](https://github.com/astral-sh/ruff-pre-commit):
```yaml title=".pre-commit-config.yaml"
repos:
- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.15.18
hooks:
- id: ruff-check
args: [--fix]
- id: ruff-format
```
Pin the hook revision and update intentionally during dependency maintenance.
### Install and run
```bash
uv run pre-commit install
uv run pre-commit run --all-files
```
If the project does not manage pre-commit via uv, use your standard Python environment installation path.
### Recommended policy
1. Keep auto-fix enabled locally with ruff-check --fix.
2. Keep CI in check-only mode so violations fail loudly.
3. Run hooks on all files in migration PRs to avoid drift.
## GitHub Actions Integration
### Why use it
Use GitHub Actions when you need required status checks on pull requests and a single source of truth for lint and format gates.
### Minimal workflow
Create [.github/workflows/ruff.yml](https://docs.github.com/en/actions/writing-workflows/workflow-syntax-for-github-actions):
```yaml title=".github/workflows/ruff.yml"
name: Ruff
on:
pull_request:
push:
branches: [main]
jobs:
ruff:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Install uv
uses: astral-sh/[email protected]
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.12"
- name: Install project dependencies
run: uv sync --dev
- name: Ruff lint
run: uv run ruff check .
- name: Ruff format check
run: uv run ruff format --check .
```
### Alternative: official Ruff action
If you want an action-focused setup, see [Ruff GitHub Actions integration](https://docs.astral.sh/ruff/integrations/#github-actions). The official Ruff action is commonly used pinned at `astral-sh/ruff-action@v4.0.0`. Keep behavior equivalent to local commands so results do not diverge.
## Alignment Checklist
Keep local hooks and CI checks aligned:
1. Same rule set from pyproject.toml.
2. Same target Python version and dependency graph.
3. Clear developer remediation command in docs:
- uv run ruff check . --fix
- uv run ruff format .
## Troubleshooting
### Hook passes locally but CI fails
1. Ensure CI uses the same pyproject.toml and not a stale cache.
2. Confirm matching Ruff versions in local and CI environments.
3. Verify CI is not running on a different Python target than local config.
### CI is slow
1. Keep Ruff in a dedicated job so failures return early.
2. Use dependency caching from your package workflow.
3. Avoid running both legacy linters and Ruff after migration completion.
## Source Links
- [Ruff integrations](https://docs.astral.sh/ruff/integrations/)
- [Ruff pre-commit docs](https://docs.astral.sh/ruff/integrations/#pre-commit)
- [Ruff GitHub Actions docs](https://docs.astral.sh/ruff/integrations/#github-actions)
- [pre-commit official docs](https://pre-commit.com/)
- [GitHub Actions documentation](https://docs.github.com/en/actions)
+116
View File
@@ -0,0 +1,116 @@
---
name: vscode-configuration
description: 'Create and troubleshoot VS Code workspace configuration for Python projects, with focused patterns for launch.json debugpy/FastAPI debugging and tasks.json task automation.'
x-personal-mcp:
id: vscode-configuration
version: 1.0.0
tags:
- vscode
- launch-json
- tasks-json
- debugpy
- fastapi
- python
- skills
capabilities:
- resource://skills/vscode-configuration/document
---
# VS Code Configuration
Use this skill to design or repair repeatable VS Code workspace configuration for local development workflows.
Primary VS Code source docs:
- [Python debugging in VS Code](https://code.visualstudio.com/docs/python/debugging)
- [Debug configuration (`launch.json`)](https://code.visualstudio.com/docs/debugtest/debugging-configuration)
- [Tasks (`tasks.json`)](https://code.visualstudio.com/docs/editor/tasks)
- [MCP servers in VS Code](https://code.visualstudio.com/docs/agent-customization/mcp-servers)
## When to Use
- You need to create or fix `.vscode/launch.json` debug profiles.
- You need robust Python debugging with `debugpy`.
- You need FastAPI-specific launch profiles (app module, host/port, reload options, env files).
- You need `.vscode/tasks.json` build/test/run tasks and optional debug pre-launch integration.
- You need `.vscode/mcp.json` workspace or user profile MCP server configuration.
- You need consistent workspace onboarding where users can run and debug from VS Code with minimal manual setup.
## Progressive References
Load only the page that matches the current request:
- Launch profile mechanics and debugpy patterns: [debug launch configurations](./references/debug-launch-configurations.md)
- FastAPI-focused debug profiles using debugpy: [FastAPI + debugpy launch patterns](./references/fastapi-debugpy-launch.md)
- Task runner setup in VS Code: [tasks.json project tasks](./references/tasks-json-configuration.md)
- MCP server setup in VS Code: [mcp.json MCP server configuration](./references/mcp-server-configuration.md)
## Procedure
### Step 1: Capture the Runtime Shape
Collect the minimum context before writing files:
1. Python entry shape: module path vs script path.
2. Framework runtime: plain script, FastAPI with uvicorn, or mixed services.
3. Required environment: env file, env vars, cwd, and PYTHONPATH needs.
4. Task expectations: run app, run tests, lint/format, one-off setup.
Completion check: you can state exactly what command should run for debug and for task execution.
### Step 2: Create launch.json Profiles
1. Add at least one stable baseline profile before specialized variants.
2. Prefer module-based launches where packaging/import paths matter.
3. Keep debugger options explicit (`justMyCode`, `console`, `cwd`, `envFile`).
4. Add purpose-built profiles instead of one overloaded profile.
For concrete patterns, open [debug launch configurations](./references/debug-launch-configurations.md).
Completion check: selecting each profile starts the intended process without manual edits.
### Step 3: Add FastAPI Profiles When Needed
1. Use a dedicated FastAPI profile that launches `uvicorn` via module mode.
2. Keep host/port/reload/log-level as explicit args.
3. Include `jinja` debugging only if templates are in scope.
4. Add an attach profile when launching via external `debugpy` listener.
For complete examples, open [FastAPI + debugpy launch patterns](./references/fastapi-debugpy-launch.md).
Completion check: breakpoints hit in app code and startup path, and profile behavior matches dev vs non-dev expectations.
### Step 4: Add tasks.json for Repeated Commands
1. Create named tasks for run, test, lint, and docs/build steps as needed.
2. For Python projects, keep commands consistent with the repo package manager.
3. Use `problemMatcher` where parsers exist and background flags for long-running tasks.
4. Link debug profiles to tasks with `preLaunchTask` only when startup sequencing is required.
For task schema and examples, open [tasks.json project tasks](./references/tasks-json-configuration.md).
Completion check: tasks run from Command Palette and can be reused by debug profiles.
### Step 5: Validate End-to-End
1. Run each launch profile once.
2. Run each task once.
3. Verify paths, env files, and interpreter assumptions on a clean workspace reload.
4. Record any project-specific defaults in comments or docs if non-obvious.
Completion check: a teammate can clone the repo, open VS Code, and run/debug with only documented prerequisites.
## Decision Points
- If the app is imported as a package, prefer module launches over direct script paths.
- If runtime is started outside VS Code, add attach profile instead of forcing launch mode.
- If there are long-running dev servers, pair with background tasks.
- If test command differs by repo convention, mirror that command in tasks exactly.
## Output Contract
Return:
1. Created or updated VS Code config files and profile/task names.
2. Any assumptions (module path, env file, command runner).
3. Validation results and any unresolved decisions.
@@ -0,0 +1,108 @@
# Debug Launch Configurations in VS Code
This reference focuses on Python debugging through [`debugpy`](https://github.com/microsoft/debugpy) using [`.vscode/launch.json`](https://code.visualstudio.com/docs/debugtest/debugging-configuration).
## Core Structure
A minimal launch file:
```json
{
"version": "0.2.0",
"configurations": []
}
```
Useful fields for Python configs:
- `type`: Use [`debugpy`](https://code.visualstudio.com/docs/python/debugging).
- `request`: Usually `launch`, sometimes `attach`.
- `name`: Friendly profile name shown in the Run and Debug panel.
- `program`: Script path for script-based entry.
- `module`: Module name for `python -m ...` style launches.
- `args`: CLI arguments.
- `cwd`: Working directory (supports [variable substitution](https://code.visualstudio.com/docs/editor/variables-reference)).
- `env` / `envFile`: Environment variables (commonly from [environment variable definitions files](https://code.visualstudio.com/docs/python/environments#_environment-variable-definitions-file)).
- `console`: `integratedTerminal` is usually most practical ([launch options](https://code.visualstudio.com/docs/debugtest/debugging-configuration#_launchjson-attributes)).
- `justMyCode`: `true` by default; set `false` when stepping into dependencies.
## Launch vs Attach
Use `launch` when VS Code should start the process.
Use `attach` when the process already runs with debugpy listening.
Attach profile example:
```json
{
"name": "Python: Attach (debugpy :5678)",
"type": "debugpy",
"request": "attach",
"connect": {
"host": "127.0.0.1",
"port": 5678
},
"justMyCode": true
}
```
Remote process side command example (from [debugpy CLI usage](https://code.visualstudio.com/docs/python/debugging#_command-line-debugging)):
```bash
python -m debugpy --listen 5678 -m your_package.main
```
## Script and Module Patterns
Script pattern:
```json
{
"name": "Python: Script",
"type": "debugpy",
"request": "launch",
"program": "${workspaceFolder}/src/app.py",
"cwd": "${workspaceFolder}",
"console": "integratedTerminal",
"justMyCode": true
}
```
Module pattern:
```json
{
"name": "Python: Module",
"type": "debugpy",
"request": "launch",
"module": "your_package.main",
"cwd": "${workspaceFolder}",
"console": "integratedTerminal",
"justMyCode": true
}
```
Prefer module mode when imports depend on package layout.
## Environment and Interpreter Notes
- Use `envFile` for shared local variables, commonly `${workspaceFolder}/.env`.
- Keep secrets out of committed launch configs.
- Ensure the selected VS Code interpreter matches project tooling.
## Source Documentation
- [Python debugging in VS Code](https://code.visualstudio.com/docs/python/debugging)
- [Debug configuration and launch.json](https://code.visualstudio.com/docs/debugtest/debugging-configuration)
- [Variables reference](https://code.visualstudio.com/docs/editor/variables-reference)
- [debugpy project](https://github.com/microsoft/debugpy)
## Troubleshooting
If breakpoints do not hit:
1. Confirm the right profile is selected.
2. Confirm the file path/module path is correct.
3. Disable `justMyCode` temporarily to inspect call flow.
4. Confirm no stale background process is occupying the expected port.
5. Confirm workspace root and `cwd` align with imports.
@@ -0,0 +1,107 @@
# FastAPI Debug Launch with debugpy
This reference provides practical [`.vscode/launch.json`](https://code.visualstudio.com/docs/debugtest/debugging-configuration) patterns for [FastAPI](https://fastapi.tiangolo.com/) applications started with [uvicorn](https://www.uvicorn.org/).
## Launch FastAPI via Module
Preferred profile:
```json
{
"name": "FastAPI: Uvicorn (debug)",
"type": "debugpy",
"request": "launch",
"module": "uvicorn",
"args": [
"your_package.main:app",
"--host",
"127.0.0.1",
"--port",
"8000",
"--reload",
"--log-level",
"debug"
],
"cwd": "${workspaceFolder}",
"envFile": "${workspaceFolder}/.env",
"console": "integratedTerminal",
"justMyCode": true,
"jinja": true
}
```
Why module mode: it matches `python -m uvicorn ...` behavior and avoids path ambiguity.
## Launch with Factory Pattern
If app is created via factory function:
```json
{
"name": "FastAPI: Uvicorn factory (debug)",
"type": "debugpy",
"request": "launch",
"module": "uvicorn",
"args": [
"your_package.main:create_app",
"--factory",
"--host",
"127.0.0.1",
"--port",
"8000",
"--reload"
],
"cwd": "${workspaceFolder}",
"console": "integratedTerminal",
"justMyCode": true
}
```
Factory mode is powered by uvicorn's [`--factory`](https://www.uvicorn.org/settings/#application) option.
## Attach to an Existing FastAPI Process
If the app is launched externally, start with [`debugpy`](https://code.visualstudio.com/docs/python/debugging#_command-line-debugging):
```bash
python -m debugpy --listen 5678 -m uvicorn your_package.main:app --host 127.0.0.1 --port 8000 --reload
```
Attach profile:
```json
{
"name": "FastAPI: Attach (5678)",
"type": "debugpy",
"request": "attach",
"connect": {
"host": "127.0.0.1",
"port": 5678
},
"justMyCode": true
}
```
## Common FastAPI Debug Pitfalls
1. Wrong import target in `your_package.main:app` or factory symbol.
2. `cwd` does not match source layout.
3. Auto-reload creating confusion about active process when breakpoints are set in startup code.
4. Port collisions from old uvicorn processes.
5. Environment variables not loaded because `envFile` path is wrong.
## Practical Quality Gate
A profile is considered valid when:
1. Server starts from VS Code Run and Debug.
2. A breakpoint inside an endpoint is hit on request.
3. A breakpoint in startup/lifespan logic is hit at app boot.
4. Terminal output appears in integrated terminal with expected log level.
## Source Documentation
- [FastAPI docs](https://fastapi.tiangolo.com/)
- [Uvicorn settings and CLI options](https://www.uvicorn.org/settings/)
- [Python debugging in VS Code](https://code.visualstudio.com/docs/python/debugging)
- [Debug configuration and launch.json](https://code.visualstudio.com/docs/debugtest/debugging-configuration)
@@ -0,0 +1,123 @@
# Configure MCP Servers in VS Code
Use this reference to configure MCP servers for GitHub Copilot chat in VS Code with `.vscode/mcp.json` (workspace) or profile-level `mcp.json` (user scope).
## Where Configuration Lives
VS Code supports two MCP configuration locations:
- Workspace scope: `.vscode/mcp.json` in the repository.
- User profile scope: open with the `MCP: Open User Configuration` command.
Use workspace scope for shared team configuration, and user scope for personal or machine-specific servers.
## Minimal mcp.json
```json
{
"servers": {
"github": {
"type": "http",
"url": "https://api.githubcopilot.com/mcp"
},
"playwright": {
"type": "stdio",
"command": "npx",
"args": ["-y", "@microsoft/mcp-server-playwright"]
}
}
}
```
The `servers` object keys are logical server names shown in VS Code MCP management surfaces.
## Add Servers Through VS Code UI
1. Run `MCP: Add Server` from the Command Palette.
2. Choose Workspace or Global target.
3. Review generated config in `mcp.json`.
4. Start or restart the server from `MCP: List Servers`.
This guided flow is usually safer than manual edits when onboarding teammates.
## Security and Secrets
1. Do not hardcode tokens or API keys in `mcp.json`.
2. Prefer input variables or environment-file patterns supported by the MCP configuration schema.
3. Start only trusted servers, because local servers can execute code on your machine.
4. Use trust prompts as a checkpoint instead of bypassing review.
## Security Best Practices
1. Apply least privilege by default.
2. Keep workspace `mcp.json` limited to team-safe, non-secret configuration.
3. Keep personal credentials and machine-specific settings in user-scope configuration, not repository files.
4. Prefer explicit allowlists for filesystem writes and outbound network access when sandboxing is enabled.
5. Use one server per trust boundary instead of one large multi-purpose server.
6. Review server `command` and `args` as code during pull requests.
7. Disable or uninstall unused MCP servers to reduce attack surface.
8. Use HTTPS endpoints for remote MCP servers whenever available.
9. Pin server packages or versions where practical to avoid accidental supply-chain drift.
10. Reset trust and re-review configuration after major server changes.
### Operational Guardrails
1. Treat MCP resources as publishable unless an explicit access control layer exists.
2. Capture server logs during onboarding so failures and suspicious behavior are easier to detect.
3. Define ownership for each server entry, including who approves changes and who rotates secrets.
4. Document upgrade triggers: if a server starts reading private data or executing side-effectful actions, require stronger access controls before rollout.
### Team Review Checklist
Use this checklist before merging workspace MCP configuration changes:
1. No plaintext secrets in `mcp.json`.
2. `command` and `args` are from trusted publishers and expected binaries.
3. Server scope is correct (workspace vs user profile).
4. Sandboxing is enabled for local `stdio` servers when supported.
5. Sandbox allowlists are narrow (minimum paths and domains).
6. The change includes an owner and rollback path.
## Sandbox Local stdio Servers (Linux/macOS)
For local `stdio` servers, enable sandboxing when possible:
```json
{
"servers": {
"myServer": {
"type": "stdio",
"command": "npx",
"args": ["-y", "@example/mcp-server"],
"sandboxEnabled": true
}
},
"sandbox": {
"filesystem": {
"allowWrite": ["${workspaceFolder}"]
},
"network": {
"allowedDomains": ["api.example.com"]
}
}
}
```
Sandboxing is currently available on Linux and macOS, not Windows.
## Troubleshooting Checklist
1. Open server logs from `MCP: List Servers` -> `Show Output`.
2. Confirm trust state (or run `MCP: Reset Trust` if needed).
3. Confirm server command and arguments run outside VS Code.
4. Confirm workspace-vs-user scope matches where you expect the server to run.
5. If using remote development, configure the server in the remote scope when needed.
## Source Documentation
- [Add and manage MCP servers in VS Code](https://code.visualstudio.com/docs/agent-customization/mcp-servers)
- [MCP configuration reference](https://code.visualstudio.com/docs/agents/reference/mcp-configuration)
- [Input variables for sensitive data](https://code.visualstudio.com/docs/agents/reference/mcp-configuration#_input-variables-for-sensitive-data)
- [Sandbox configuration reference](https://code.visualstudio.com/docs/agents/reference/mcp-configuration#_sandbox-configuration)
- [AI security guidance in VS Code](https://code.visualstudio.com/docs/agents/security)
- [Model Context Protocol overview](https://modelcontextprotocol.io/docs/getting-started/intro)
@@ -0,0 +1,99 @@
# Configure Project Tasks in tasks.json
Use [`.vscode/tasks.json`](https://code.visualstudio.com/docs/editor/tasks) to define repeatable project commands and optional hooks for debugging.
## Minimal File
```json
{
"version": "2.0.0",
"tasks": []
}
```
## Task Fields You Will Use Most
- `label`: Task name shown in VS Code.
- `type`: Usually [`shell`](https://code.visualstudio.com/docs/editor/tasks#_custom-tasks).
- `command`: Executable to run.
- `args`: Command arguments.
- `options.cwd`: Working directory (supports [variable substitution](https://code.visualstudio.com/docs/editor/variables-reference)).
- `group`: Mark default build or test tasks ([task groups](https://code.visualstudio.com/docs/editor/tasks#_grouping-tasks)).
- `problemMatcher`: Parse errors into the Problems panel ([problem matchers](https://code.visualstudio.com/docs/editor/tasks#_defining-a-problem-matcher)).
- `isBackground`: `true` for long-running tasks (for example dev server watch).
## Python Project Example
```json
{
"version": "2.0.0",
"tasks": [
{
"label": "App: Run",
"type": "shell",
"command": "uv",
"args": ["run", "uvicorn", "personal_mcp.main:create_app", "--factory", "--host", "127.0.0.1", "--port", "8000", "--reload"],
"options": {
"cwd": "${workspaceFolder}"
},
"isBackground": true,
"problemMatcher": []
},
{
"label": "Tests: Pytest",
"type": "shell",
"command": "uv",
"args": ["run", "pytest"],
"options": {
"cwd": "${workspaceFolder}"
},
"group": "test",
"problemMatcher": []
}
]
}
```
## Connect Tasks to Debug Profiles
In [`launch.json`](https://code.visualstudio.com/docs/debugtest/debugging-configuration), you can run a task first with [`preLaunchTask`](https://code.visualstudio.com/docs/debugtest/debugging-configuration#_launchjson-attributes):
```json
{
"name": "FastAPI: Attach",
"type": "debugpy",
"request": "attach",
"connect": {
"host": "127.0.0.1",
"port": 5678
},
"preLaunchTask": "App: Run"
}
```
Use this only when startup sequencing is needed.
## Task Design Guidelines
1. Keep labels stable and descriptive.
2. Prefer one task per intent instead of monolithic shell commands.
3. Keep shell portability in mind if teammates use multiple OSes.
4. Avoid embedding secrets directly in task definitions.
5. Mark long-running tasks with `isBackground` and keep matchers explicit.
## Troubleshooting
If a task fails unexpectedly:
1. Run the underlying command directly in terminal.
2. Confirm `options.cwd` points to expected workspace root.
3. Confirm tool availability in environment path.
4. Confirm quoting and argument boundaries in `args`.
5. Confirm the task is not blocked by an outdated background process.
## Source Documentation
- [VS Code Tasks (official)](https://code.visualstudio.com/docs/editor/tasks)
- [Tasks Appendix (schema and interfaces)](https://code.visualstudio.com/docs/reference/tasks-appendix)
- [Variables Reference](https://code.visualstudio.com/docs/editor/variables-reference)
- [Debug configuration and launch.json](https://code.visualstudio.com/docs/debugtest/debugging-configuration)
+136
View File
@@ -0,0 +1,136 @@
---
name: zensical-docs
description: 'Reference skill for Zensical documentation mechanics. Use for quick lookup of docs structure, feature options, and source links. Prefer inline Markdown links to source docs and avoid bare URLs because this content is rendered as human docs and MCP resources.'
x-personal-mcp:
id: zensical-docs
version: 1.0.0
tags:
- zensical
- mkdocs
- mkdocs-material
- mkdocstrings
- docs
- documentation
- information-architecture
- skills
- bootstrap
- discovery
- authoring
capabilities:
- resource://skills/zensical-docs/document
---
# Zensical Documentation Authoring
Use this as a compact reference for Zensical mechanics and as the place to record evolving preferences for how this repository uses them.
## When to Use
- You need a quick reminder of Zensical features, docs structure, or configuration mechanics.
- You want direct links back to source documentation before changing docs behavior.
- You want one small file you can keep editing as your preferences around docs authoring become clearer.
## How To Use This Skill
1. Start here for a quick decision about what kind of docs change you are making.
2. Open only the linked reference that matches the current task.
3. Add or revise preference notes in this file when you decide how this repo should use a feature.
## Quick Reference Map
Open only what you need:
- Official docs and source map: [source map](./references/index.md)
- Zensical feature catalog and setup links: [feature catalog](./references/zensical-features.md)
- Theme, icons, and visual customization: [theme customization](./references/theme-customization-and-icons.md)
- Writing quality and review criteria: [documentation quality](./references/documentation-quality.md)
- Navigation and discoverability patterns: [discoverability and IA](./references/discoverability-and-ia.md)
- Code-heavy docs and API reference patterns: [code-heavy docs](./references/code-heavy-docs-and-mkdocstrings.md)
## Common Cases
### New docs project
- Start with `uv run zensical new`.
- Then review the [source map](./references/index.md) and [feature catalog](./references/zensical-features.md).
### Restructuring docs or navigation
- Review [discoverability and IA](./references/discoverability-and-ia.md).
- Use it to decide overview pages, section structure, and cross-linking.
### Improving writing quality
- Review [documentation quality](./references/documentation-quality.md).
- Use it for page quality gates, trust signals, and review criteria.
### Adjusting theme or UI mechanics
- Review [theme customization](./references/theme-customization-and-icons.md).
- Use it for icons, color, theme extensions, and presentation choices.
### Documenting APIs or code-heavy systems
- Review [code-heavy docs](./references/code-heavy-docs-and-mkdocstrings.md).
- Use it when generated API reference belongs alongside hand-authored docs.
## Preferences To Maintain Here
Keep this section short and revise it over time.
### Preferred feature choices
- Add the Zensical features you usually enable first.
- Note which features are situational and why.
- Prefer Zensical-native features and conventions when they cover the need cleanly.
- Expect general backward compatibility with MkDocs patterns and configuration unless there is a documented reason not to.
### Preferred docs structure
- Record whether this repo prefers explicit nav, index pages, task-first docs, or another pattern.
### Preferred API docs approach
- Record whether to use mkdocstrings, how much API surface to publish, and how to link task docs back to reference pages.
## Source-First Rule
When making a recommendation, link back to the relevant reference file first, and when possible to the upstream docs linked from that reference.
## Link Formatting Rule
Because this project publishes the same markdown for both `/docs` and MCP resources, link quality is part of the content contract.
- Never leave a bare URL in prose or list items.
- Prefer using in-place Markdown links with meaningful labels.
- For external sources, prefer `[descriptive label](https://...)` over raw `https://...`.
- For internal files, prefer relative Markdown links so rendered docs remain navigable.
- Any mention of a library or a specific library feature should include a link to source documentation somewhere on the page.
- If inline linking is awkward or the citation payload is too large, use a footnote or tooltip citation instead.
Example preferred style:
- `See [importlib.resources](https://docs.python.org/3/library/importlib.resources.html) for packaging details.`
Example to avoid:
- `See https://docs.python.org/3/library/importlib.resources.html for packaging details.`
Acceptable alternatives when inline links are not ideal:
- Add a footnote-style source citation at the end of the section or page.
- Add a tooltip citation when the docs pattern supports it.
## Compatibility Rule
Prefer the Zensical-native way of doing something when it exists and is well-supported.
Assume MkDocs compatibility is still expected for most configuration and authoring patterns, and call out any case where a Zensical recommendation intentionally diverges from standard MkDocs behavior.
## Output Contract
Return only what is useful for the current docs task:
1. Which reference to read next.
2. The smallest recommended docs or config change.
3. Any repo-specific preference this suggests should be added back into this skill.
4. For any library or feature-level claim, include a source-doc citation somewhere (inline link preferred; footnote or tooltip acceptable).
@@ -0,0 +1,63 @@
# Code-Heavy Documentation with mkdocstrings
Use this reference when your docs include API surfaces, function/class documentation, and source-driven technical reference.
## Why mkdocstrings
mkdocstrings helps generate and maintain API reference pages directly from code and docstrings, reducing drift between implementation and docs.
!!! info "Primary docs"
- [mkdocstrings home](https://mkdocstrings.github.io/)
- [mkdocstrings Python handler](https://mkdocstrings.github.io/python/)
- [Griffe Python parsing engine](https://mkdocstrings.github.io/griffe/)
## When to Use It
- You maintain Python modules/classes/functions that need searchable reference docs.
- You want hand-written concept/task docs plus generated API reference pages.
- You need consistent signatures, type hints, and docstring rendering.
## Recommended Documentation Split
1. Hand-authored docs for concepts, architecture, and tasks.
2. Generated docs (mkdocstrings) for API details.
3. Cross-links in both directions:
- task pages link to specific API entries
- API pages link to practical guides and examples
## Minimal Integration Pattern
1. Add mkdocstrings and a Python handler package to project dependencies.
2. Configure the Zensical docs toolchain to enable mkdocstrings within the site build.
3. Create one API index page per package/domain.
4. Expand coverage gradually from high-value modules first.
!!! info "General reference examples"
- [Zensical docs home and setup entry point](https://zensical.org/docs/)
- [Zensical code blocks and authoring patterns](https://zensical.org/docs/authoring/code-blocks/)
- [Zensical customization overview](https://zensical.org/docs/customization/)
!!! note "Compatibility"
Zensical is generally expected to remain compatible with MkDocs-style configuration patterns, but prefer Zensical-native documentation and examples when they cover the same behavior.
## Authoring Guidance for Docstrings
- Begin with a one-line summary in imperative or descriptive form.
- Document parameters, return values, raised exceptions, and side effects.
- Include short examples for non-obvious usage.
- Keep terminology aligned with task docs and architecture pages.
## Quality Gates for Code-Heavy Docs
- API pages build cleanly and include expected modules.
- Symbols are grouped by domain, not dumped in one long page.
- Public APIs have meaningful docstrings before publishing.
- Generated reference pages are linked from user-facing docs.
- Search can find both conceptual guides and concrete API entries.
## Common Pitfalls
- Treating generated API docs as a replacement for task documentation.
- Publishing API pages without module-level context.
- Letting undocumented public APIs accumulate.
- Not reviewing generated pages after refactors.
@@ -0,0 +1,74 @@
# Discoverability and Information Architecture
Use this reference to design docs that are progressively discoverable from overview to implementation detail.
## IA Model
Organize by user intent, then by product area.
Recommended top-level model:
1. Learn (concepts, architecture, mental models)
2. Do (task/how-to paths)
3. Reference (API, config, command catalog)
4. Troubleshoot (symptoms, diagnostics, fixes)
!!! info "IA references"
- [Diataxis framework](https://diataxis.fr/)
- [Divio documentation system](https://documentation.divio.com/)
## Progressive Discoverability Pattern
### Layer 1: Section Overview
Each section starts with an index page containing:
- What this section is for
- Who should read it
- Common journeys
- Links to key tasks and references
### Layer 2: Task or Concept Pages
Each page includes:
- 1-2 sentence purpose
- prerequisites
- internal links to references and next steps
### Layer 3: Deep Reference
Keep deep details in dedicated reference pages and link to them from task pages when needed.
## Navigation Design Rules
1. Keep navigation labels user-facing and action-oriented.
2. Avoid duplicate labels in separate branches.
3. Place high-frequency tasks near the top.
4. Keep section depth shallow where possible.
!!! info "Relevant Zensical configuration docs"
- [Navigation setup](https://zensical.org/docs/setup/navigation/)
- [Search setup](https://zensical.org/docs/setup/search/)
- [Header setup](https://zensical.org/docs/setup/header/)
- [Footer setup](https://zensical.org/docs/setup/footer/)
## Link Strategy
- Every deep page should have at least one inbound link from a higher-level index page.
- Add "See also" blocks for neighboring tasks.
- Link to source-of-truth reference pages instead of duplicating config tables.
## Search Optimization for Docs
- Put key terms in title and first paragraph.
- Use specific H2/H3 headings that match user query language.
- Keep repeated boilerplate minimal so snippets stay informative.
## Review Heuristics
A documentation journey is healthy when:
- users can identify their path within 10 seconds on a section index page
- users can complete primary tasks without opening more than 2-3 tabs
- users can recover from common errors without external support tickets
@@ -0,0 +1,54 @@
# Documentation Quality Best Practices
Use this reference when writing or reviewing docs for clarity, correctness, and trust.
## Core Writing Principles
1. Write for a specific audience and task.
2. Lead with outcomes, not internal implementation details.
3. Keep concepts, tasks, and references distinct.
4. Make examples executable and verifiable.
5. Prefer precise language over marketing language.
!!! info "Primary references"
- [Diataxis](https://diataxis.fr/)
- [Divio documentation system](https://documentation.divio.com/)
- [Write the Docs guide](https://www.writethedocs.org/guide/)
## Style and Readability
- Use consistent terminology and avoid synonym drift.
- Use short paragraphs and meaningful headings.
- Prefer active voice and imperative instructions for task pages.
- Add notes/warnings only for high-impact caveats.
!!! info "Style sources"
- [Google developer style](https://developers.google.com/style)
- [Microsoft Writing Style Guide](https://learn.microsoft.com/style-guide/welcome/)
- [MDN writing guidelines](https://developer.mozilla.org/en-US/docs/MDN/Writing_guidelines)
## Task Page Quality Pattern
Each task page should include:
1. Goal and scope.
2. Prerequisites (permissions, versions, environment).
3. Step-by-step procedure.
4. Expected result and verification command/output.
5. Common failure modes and recovery path.
6. Related links (concept, reference, troubleshooting).
## Quality Gates Before Publish
- Accuracy: commands and code examples are validated.
- Completeness: no critical missing steps.
- Discoverability: page is linked from at least one overview page.
- Freshness: version-specific notes and dates are present where needed.
- Accessibility: heading structure and link text are clear.
## Anti-Patterns
- Mixing conceptual explanation and long procedural flow in the same section without structure.
- Hiding prerequisites mid-page.
- Using screenshots as the only source of truth for commands.
- Publishing pages with no owner and no review cadence.
@@ -0,0 +1,48 @@
# Zensical Docs Skill References
Use this index to load only the source references needed for the current task.
## Zensical Official Docs
!!! info "Zensical official docs"
- [New project scaffolding](https://zensical.org/docs/) for `uv run zensical new`.
- [Home](https://zensical.org/docs/)
- [Setup basics](https://zensical.org/docs/setup/basics/)
- [Navigation setup](https://zensical.org/docs/setup/navigation/)
- [Header setup and announcement bar](https://zensical.org/docs/setup/header/)
- [Footer setup](https://zensical.org/docs/setup/footer/)
- [Repository and content actions](https://zensical.org/docs/setup/repository/)
- [Search setup](https://zensical.org/docs/setup/search/)
- [Customization overview](https://zensical.org/docs/customization/)
- [Additional CSS](https://zensical.org/docs/customization/#additional-css)
- [Additional JavaScript](https://zensical.org/docs/customization/#additional-javascript)
- [Theme extension and overrides](https://zensical.org/docs/customization/#extending-the-theme)
- [Language setup](https://zensical.org/docs/setup/language/)
- [Logo and icons](https://zensical.org/docs/setup/logo-and-icons/)
- [Code blocks and annotations](https://zensical.org/docs/authoring/code-blocks/)
- [Content tabs](https://zensical.org/docs/authoring/content-tabs/)
- [Footnotes](https://zensical.org/docs/authoring/footnotes/)
- [Tooltips](https://zensical.org/docs/authoring/tooltips/)
## Adjacent Documentation Quality Sources
!!! info "Documentation quality sources"
- [Divio documentation system](https://documentation.divio.com/)
- [Write the Docs guide](https://www.writethedocs.org/guide/)
- [Google developer documentation style guide](https://developers.google.com/style)
- [Microsoft Writing Style Guide](https://learn.microsoft.com/style-guide/welcome/)
- [MDN writing guidelines](https://developer.mozilla.org/en-US/docs/MDN/Writing_guidelines)
- [Diataxis framework](https://diataxis.fr/)
## Related Tooling References
!!! info "Related tooling"
- [Markdown guide](https://www.markdownguide.org/)
- [Zensical setup and configuration entry point](https://zensical.org/docs/)
- [Zensical customization reference](https://zensical.org/docs/customization/)
- [mkdocstrings](https://mkdocstrings.github.io/)
## Skill-Specific Deep Dives
- Theme customization, colors, icons: [./theme-customization-and-icons.md](./theme-customization-and-icons.md)
- Code-heavy docs with mkdocstrings: [./code-heavy-docs-and-mkdocstrings.md](./code-heavy-docs-and-mkdocstrings.md)
@@ -0,0 +1,62 @@
# Theme Customization, Colors, and Icons
Use this reference when you want documentation that feels intentional and brand-aligned while preserving readability and accessibility.
## Start from the Scaffold
Always start new projects with `uv run zensical new` so the baseline theme/config scaffolding is in place before customization.
## Customization Strategy
1. Configure theme and feature flags in the project config first.
2. Apply visual tokens (colors, spacing, typography) in a shared CSS layer.
3. Add icons and logo assets with consistent naming.
4. Use template overrides only when config/CSS cannot solve the requirement.
## Key Zensical Customization Surfaces
!!! info "Zensical sources"
- [Customization overview](https://zensical.org/docs/customization/)
- [Additional CSS](https://zensical.org/docs/customization/#additional-css)
- [Additional JavaScript](https://zensical.org/docs/customization/#additional-javascript)
- [Extending the theme](https://zensical.org/docs/customization/#extending-the-theme)
- [Logo and icons setup](https://zensical.org/docs/setup/logo-and-icons/)
## Colors and Accessibility
- Define color variables once and reuse them for semantic roles (primary, surface, muted, success, warning).
- Keep contrast high for body text, code blocks, and nav labels.
- Test color changes on mobile and desktop, including search highlights and active nav states.
!!! info "General references"
- [Material Design color guidance](https://m3.material.io/styles/color)
- [WCAG overview](https://www.w3.org/WAI/standards-guidelines/wcag/)
## Icons: Selection and Search Landing Pages
If your theme supports icon sets through your docs stack, these search portals are useful:
- [Material Symbols search](https://fonts.google.com/icons)
- [Font Awesome icons search](https://fontawesome.com/search)
- [Simple Icons search](https://simpleicons.org/)
- [Iconify icon set search](https://icon-sets.iconify.design/)
- [Lucide icons](https://lucide.dev/icons/)
!!! tip "Icon family consistency"
Pick one primary icon family for navigation and status icons, then document naming conventions.
## Extending the Theme Safely
Use overrides as a last step, not the first.
1. Confirm the requirement cannot be solved by config and CSS.
2. Keep override templates minimal and focused.
3. Track upstream changes if you override partials.
4. Add a visual regression checklist for common pages.
## Review Checklist
- Theme changes preserve readability for long-form docs.
- Icons are consistent in weight/style and meaningful in context.
- Color changes do not break code-block syntax highlighting or search visibility.
- Overrides are documented with rationale and owner.
@@ -0,0 +1,78 @@
# Zensical Features and Configuration Patterns
Use this reference when deciding which Zensical features to enable and why.
## Project Bootstrap
Always start a new docs project with `uv run zensical new`.
- It creates the baseline scaffolding for configuration, docs structure, and theme integration.
- Treat this as the default starting point rather than manually assembling files.
## High-Value Feature Groups
### Navigation and Discoverability
- `navigation.indexes`: lets sections have index pages for overview content.
- `navigation.path`: adds breadcrumb-like context.
- `navigation.sections`: groups top-level sections for large doc sets.
- `navigation.instant`: enables instant internal navigation.
- `navigation.instant.prefetch`: prefetches likely next pages.
- `navigation.top`: shows a back-to-top affordance.
- `navigation.tracking`: keeps URL anchors in sync with active section.
!!! info "Source links"
- [Zensical navigation setup](https://zensical.org/docs/setup/navigation/)
### Code-Heavy Documentation
- `content.code.copy`: copy button in code blocks.
- `content.code.select`: line range selection support.
- `content.code.annotate`: inline code annotations.
- Prefer mkdocstrings for generated API reference pages when documenting Python code.
- Keep generated API pages linked from hand-authored task and concept docs.
!!! info "Source links"
- [Zensical code blocks](https://zensical.org/docs/authoring/code-blocks/)
- [mkdocstrings](https://mkdocstrings.github.io/)
### Cross-Page UX Consistency
- `content.tabs.link`: keeps same-named tabs synchronized.
- `content.tooltips`: improves tooltip behavior for links.
- `content.footnote.tooltips`: inline footnote previews.
!!! info "Source links"
- [Zensical content tabs](https://zensical.org/docs/authoring/content-tabs/)
- [Zensical tooltips](https://zensical.org/docs/authoring/tooltips/)
- [Zensical footnotes](https://zensical.org/docs/authoring/footnotes/)
### Search and Content Actions
- `search.highlight`: highlights matches after search navigation.
- `content.action.edit` and `content.action.view` (if repository integration is configured).
!!! info "Source links"
- [Zensical search setup](https://zensical.org/docs/setup/search/)
- [Zensical repository setup](https://zensical.org/docs/setup/repository/)
## Styling and Extensibility
Use site-level customization when docs need stronger visual affordances.
- `extra_css`: add targeted style overrides.
- `extra_javascript`: add behavior enhancements.
- Theme override directory (`custom_dir`) for template-level changes.
!!! info "Source links"
- [Zensical customization overview](https://zensical.org/docs/customization/)
- [Additional CSS](https://zensical.org/docs/customization/#additional-css)
- [Additional JavaScript](https://zensical.org/docs/customization/#additional-javascript)
- [Extending the theme](https://zensical.org/docs/customization/#extending-the-theme)
## Practical Feature Selection Rules
1. Start with discoverability and clarity features first.
2. For code-heavy docs, add copy/select/annotate first, then define mkdocstrings coverage for API reference.
3. Avoid enabling many features at once without measurement.
4. Track user success metrics (search success, time-to-answer, support deflection) after each change.
+14
View File
@@ -0,0 +1,14 @@
.mermaid svg text,
.mermaid svg tspan,
.mermaid svg foreignObject,
.mermaid svg foreignObject div,
.mermaid svg foreignObject span,
.mermaid svg .nodeLabel,
.mermaid svg .edgeLabel {
color: var(--md-primary-bg-color, #fff);
fill: var(--md-primary-bg-color, #fff);
}
.mermaid svg .treeView-node-label {
fill: var(--md-primary-bg-color, #fff) !important;
}
+95
View File
@@ -0,0 +1,95 @@
---
icon: lucide/flask-conical
---
# Testing
This page describes the current test layout and execution model for this repository.
Primary guidance sources:
- [Pytest scaffolding skill](./skills/pytesting/SKILL.md)
- [Pytest docs reference](./skills/pytesting/references/pytest-docs.md)
- [FastAPI + uv + Docker skill](./skills/fastapi-uv-docker/SKILL.md)
## Goals
1. Keep local feedback fast with deterministic tests.
2. Mirror source modules with focused test groups.
3. Keep endpoint and MCP surface checks explicit.
4. Make marker usage strict and intentional.
## Current Test Layout
Current tree:
```text
tests/
__init__.py
conftest.py
registry/
ingest/
conftest.py
test_current_docs.py
test_document.py
test_prompt.py
test_skill.py
models/
test_document_validation.py
test_prompt_validation.py
test_registry_payload_models.py
test_skill_validation.py
web/
conftest.py
test_endpoint_connections.py
test_mcp_skills.py
```
Source-to-test alignment today:
- `src/personal_mcp/registry/ingest/` -> `tests/registry/ingest/`
- `src/personal_mcp/registry/models/` -> `tests/registry/models/`
- `src/personal_mcp/web/` and MCP HTTP surface -> `tests/web/`
## Markers And Strictness
Configured markers in `pyproject.toml`:
- `unit`: fast deterministic tests with no external dependencies
- `integration`: framework or component integration tests
- `smoke`: thin critical-path checks
Pytest runs with `--strict-markers`, so any unregistered marker fails the test run.
## Fixture Layering
Fixture placement follows test scope:
1. `tests/conftest.py` for cross-suite defaults.
2. `tests/registry/ingest/conftest.py` for ingest-specific setup.
3. `tests/web/conftest.py` for web and endpoint client setup.
Prefer adding fixtures at the narrowest scope that serves more than one test.
## Command Baseline
Canonical invocation:
```bash
uv run pytest
```
Useful filtered runs:
```bash
uv run pytest --collect-only -q
uv run pytest -m unit -q
uv run pytest -m integration -q
uv run pytest -m smoke -q
```
## Adding New Tests
When adding coverage:
1. Place tests under the nearest existing module subtree (`registry/` or `web/`).
2. Mirror the source path where practical.
3. Reuse existing `conftest.py` files before adding new fixture layers.
4. Add markers only when they convey execution intent, and register new markers in `pyproject.toml` first.
This keeps the suite aligned with the current architecture while preserving a fast local test loop.
+366
View File
@@ -0,0 +1,366 @@
---
icon: lucide/workflow
---
# Skill Usage Mechanics
## Purpose
This page explains practical usage mechanics for the GitHub Copilot extension in VS Code when `personal-mcp` is configured as an MCP server:
1. explicit `/` command flows when you want deterministic control
2. guided skill loading when relevance can be inferred
The goal is to show how Copilot behaves as a client and how to shape that behavior.
## Mental Model
In Copilot Chat, there are two distinct mechanisms:
1. `/` commands are user-invoked orchestration shortcuts.
2. MCP resources are server-published knowledge units that can be attached as read-only context, while MCP tools provide an execution path for discovery and retrieval.
In this repository, skill guidance is exposed as MCP resources, not as server-owned prompt execution. Copilot remains the orchestrator.
Prompt guidance is now exposed through both prompt resources and MCP prompt objects. Prompt objects are additive; authored markdown remains the canonical source.
## Background Mechanics
### What the server publishes
`personal-mcp` registers resources from the validated docs registry and exposes catalog discovery resources:
1. `resource://catalog/skills_index`
2. `resource://catalog/skills_index{?q,tag,capability,cursor,limit}`
3. `resource://catalog/skills/{skill_id}`
4. `resource://catalog/prompts_index`
5. `resource://catalog/prompts_index{?q,tag,cursor,limit}`
6. `resource://catalog/prompts/{prompt_id}`
Each skill publishes a canonical Markdown document resource:
1. `resource://skills/<skill-id>/document`
2. `resource://skills/<skill-id>/references/<ref-id>`
Prompts publish a canonical prompt document resource:
1. `resource://prompts/<prompt-id>/document`
The document payload is loaded from `docs/skills/<skill-id>/SKILL.md` and returned with metadata.
### What Copilot does as the client
When connected to MCP, Copilot can do the following at runtime:
1. interpret the current chat request
2. use attached MCP resources that you provide through the chat UI
3. invoke MCP tools when the task and tool descriptions make that relevant
4. summarize relevant sections into working context
5. apply guidance while generating edits or recommendations
This behavior is shaped by the active chat surface, prompt or instruction guidance, and available MCP tools.
For reliable progressive discovery, use one of these sequences:
1. explicit resource path: attach a catalog resource first, then attach only selected skill documents
2. tool path: call catalog tools first, then load only selected skill documents
### What `/` commands do
`/` commands in VS Code are client-side prompt entry points (for example in prompt files). They do not replace MCP resources. In Copilot, they typically:
1. enforce a known sequence
2. collect missing inputs
3. call discovery/read steps in a predictable order
Think of `/` commands as orchestration shortcuts on top of MCP resources.
### What automatic loading means here
In this project, "automatic loading" should be read as a preference you express through instructions and prompts, not as a guaranteed VS Code feature that auto-attaches MCP resources.
In practice, there are two reliable ways to make skill content available in chat:
1. explicit resource attachment through `Add Context > MCP Resources` or `MCP: Browse Resources`
2. MCP tool invocation using `list_resources`/`read_resource` (ResourcesAsTools), with thin catalog tools as parity fallback
For prompt content, there is a third option when the client supports MCP prompt APIs:
1. prompt-object discovery and invocation through MCP prompt lists and `get_prompt`
Instruction quality and metadata quality still matter, because they influence whether Copilot recognizes that the MCP server is relevant and chooses the tool path well.
## Invocation Mechanics Deep Dive
This section expands on how invocation works at runtime across chat entry points.
### Invocation Surfaces
A user request can arrive through one of these surfaces:
1. plain chat request in Ask/Edit/Agent mode
2. slash command invocation of a prompt or skill
3. chat request with manually attached MCP resources
Each surface changes how much discovery Copilot must do before applying guidance.
### Resolution Order
When multiple retrieval paths are possible, use this priority order:
1. attached MCP resources already in context
2. explicit slash-command workflow steps
3. catalog-first discovery via MCP resources
4. tool fallback (`list_resources` then `read_resource`, then thin catalog parity tools)
This ordering keeps behavior predictable while minimizing unnecessary context expansion.
### Prompt Invocation Pipeline
For prompt-oriented flows, treat invocation as this sequence:
1. parse prompt frontmatter and argument hints
2. validate required inputs and ask one clarifying question if blocked
3. run bounded discovery against prompt or skill catalogs
4. fetch only selected document resources
5. apply instructions to produce edits, recommendations, or commands
6. report what was loaded and why
Prompt objects and prompt document resources are additive mechanisms. The authored Markdown prompt document remains the canonical contract.
### Argument Syntax Nuance
Invocation strings such as target_modules=src/personal_mcp/registry/ingest/skill.py, mode=plan-only are a structured authoring convention, not a guaranteed client-level grammar.
In practice:
1. Prompt metadata defines expected argument names and intent.
2. Prompt body instructions define how those inputs should be interpreted.
3. Copilot may receive equivalent intent in freeform phrasing and still resolve it correctly.
Implication for authors:
1. Treat key=value examples as clarity aids for users.
2. Do not assume strict parser enforcement unless your prompt explicitly validates and rejects malformed input.
3. Include accepted invocation examples and one fallback freeform example so behavior is predictable for both humans and the model.
This distinction is important because argument hints improve discoverability, while robust prompt instructions determine actual runtime reliability.
### Skill Invocation Pipeline
For guided skill loading, use this sequence:
1. start from `resource://catalog/skills_index` or scoped index query
2. inspect one or two top candidates for intent and capability fit
3. fetch `resource://skills/<skill-id>/document`
4. load references only when the task needs deeper detail
5. apply only relevant sections and keep context bounded
This avoids the common failure mode where many skill documents are loaded up front.
### Determinism vs Flexibility
Use this decision rule:
1. choose slash-command invocation when repeatability and step order are critical
2. choose guided loading when requests vary and speed matters more than strict orchestration
3. escalate from guided loading to slash-command flow when confidence is low or conflicting skills appear
### Invocation Trace (What to Log in Results)
For transparent operation, include a concise invocation trace in task outputs:
1. entry surface used (plain chat, slash command, or attached resource)
2. discovery source used (catalog resource or tool path)
3. resources fetched (ids only)
4. clarifying questions asked (if any)
5. reason for fallback or escalation (if used)
This makes behavior auditable and easier to tune over time.
## Operating Pattern
Use both modes intentionally in Copilot Chat.
### Mode A: Explicit `/` command
Use when you need predictable, repeatable behavior across teammates.
Good fits:
1. onboarding workflows
2. compliance-sensitive tasks
3. repetitive scaffolding
### Mode B: Guided skill loading
Use when requests are varied and you want lower friction during normal chat.
Good fits:
1. ad hoc implementation questions
2. mixed-topic debugging
3. architecture tradeoff discussions
### Mode C: Fallback flow
Start with guided loading in chat; escalate to a `/` command when:
1. confidence is low
2. multiple skills conflict
3. the user wants strict repeatability
## Suggested Resolution Flow
```mermaid
flowchart TD
A[User request in Copilot Chat] --> B{Deterministic workflow needed?}
B -- Yes --> C[/Run slash command/]
C --> D[Copilot fetches known catalog and skill resources]
B -- No --> E[Copilot uses attached resources or catalog tools]
E --> F{Confident skill match?}
F -- Yes --> G[Copilot fetches skill documents]
F -- No --> H[Ask clarifying question or suggest slash command]
D --> I[Apply guidance to task]
G --> I
H --> I
```
## Authoring Requirements
Authoring rules for metadata quality and instruction patterns are maintained in [Authoring Guide](./authoring.md).
## Practical Guidelines
1. Keep `/` commands minimal and high-value.
2. Do not duplicate full methodology text inside command files.
3. Keep canonical guidance in `docs/skills/*/SKILL.md`.
4. In Copilot instructions, prefer catalog-first discovery before skill fetch.
5. Prefer small, relevant context slices over loading every skill.
6. Keep slash commands focused on deterministic orchestration, not content duplication.
If you skip the catalog/index step, behavior is less predictable and may either miss relevant skills or pull too much context.
## Optional Tool Search Mode
When tool catalogs grow, FastMCP search transforms can reduce tool-list noise for tool-only clients.
Runtime switches:
1. `PERSONAL_MCP_TOOL_SEARCH=none|regex|bm25` (default `none`)
2. `PERSONAL_MCP_TOOL_SEARCH_MAX_RESULTS=<positive int>` (default `5`)
Behavior:
1. `regex` uses deterministic regex matching for targeted queries.
2. `bm25` uses ranked natural-language matching.
3. `list_resources` and `read_resource` stay visible so resource-backed fallback remains primary.
## Failure Modes and Recovery
Common failure modes:
1. No relevant skill selected.
2. Too many skills selected (context bloat).
3. Stale assumptions from old metadata.
4. Slash command bypasses normal discovery and forces the wrong skill.
Recovery sequence:
1. re-run catalog lookup
2. narrow by tags and intent
3. fetch only top candidates
4. if still ambiguous, ask one clarifying question
5. use explicit `/` workflow for deterministic fallback
## Checklist
Use this checklist when configuring GitHub Copilot in VS Code against `personal-mcp`:
1. confirm server connectivity
2. verify catalog resources are readable
3. verify at least one `resource://skills/<id>/document` can be fetched
4. add one deterministic `/` command for fallback
5. confirm your workspace instruction policy exists (see [Authoring Guide](./authoring.md))
6. verify context size remains bounded
7. validate behavior in Ask/Edit/Agent-style workflows with at least one task each
## Runtime Discovery Workflow
Use this runtime sequence in chat sessions:
1. Start with catalog-first discovery.
2. Prefer MCP resources when the chat surface exposes resource attachment.
3. Otherwise use tool fallback to load one or two likely skill documents.
4. Prefer `list_resources`/`read_resource` first when operating in tool-only clients.
5. If confidence is low, ask one clarifying question before loading more.
## Thin Shim Path Binding Pattern
For repositories that consume this MCP server, thin shims are a usage pattern for binding path scopes to the right MCP resources. The "thin shims" are just lightweight, repo-specific instructions files that tell Copilot to use certain MCP resources when editing files that match a pattern. That helps with ensuring Copilot uses the intended resources without too much specific goading in the prompt.
Use thin shims in Copilot instruction files to bind file-path scopes to:
1. the most relevant docs page for human-readable conventions
2. the matching MCP resource URI for machine retrieval
Keep each shim short: trigger, primary resource, minimal execution pattern, and one fallback rule.
Recommended binding pattern:
1. Put shims in `.github/instructions/*.instructions.md`.
2. Scope each shim with `applyTo` so it activates only where needed.
3. Point to one primary `resource://skills/<skill-id>/document` URI.
4. Link one repository docs page as the human-facing companion.
5. Expand to references only when the task needs deeper detail.
Current repository examples:
| applyTo scope | Primary docs page | Primary MCP resource |
| --- | --- | --- |
| `**/*.md` | [docs/authoring.md](./authoring.md) | `resource://skills/zensical-docs/document` |
| `tests/**` | [docs/testing.md](./testing.md) | `resource://skills/pytesting/document` |
| `.vscode/**` | [docs/skills/vscode-configuration/SKILL.md](./skills/vscode-configuration/SKILL.md) | `resource://skills/vscode-configuration/document` |
Minimal shim shape:
```md
---
name: <short scope name>
description: Route <path scope> edits to the Personal MCP <skill-id> resource.
applyTo: '<glob>'
---
When editing files matching <glob>, use `resource://skills/<skill-id>/document` as the primary guidance source.
Execution pattern:
1. Load the primary skill document first.
2. Apply only sections relevant to the file being edited.
3. Keep edits minimal and aligned with repository conventions.
4. If confidence is low, ask one clarifying question before editing.
Companion docs page: [docs/<page>.md](./<page>.md)
```
When to use thin shims:
1. Repositories that want thin local policy while keeping canonical guidance in MCP resources.
2. Stable, repeated workflows with clear path ownership.
3. Cases where teams need predictable retrieval behavior.
When not to use thin shims:
1. Broad, ambiguous tasks with unclear ownership boundaries.
2. Cases where one shim would need many exceptions.
3. Situations better handled by catalog-first discovery at runtime.
## Summary
The intended model is:
1. skills are canonical MCP resources
2. `/` commands are explicit Copilot control shortcuts
3. guided skill loading should be catalog-driven, bounded, and explicit about whether it is using resources or tools
Using all three together gives predictable control when needed and low-friction assistance by default in VS Code.

Some files were not shown because too many files have changed in this diff Show More