diff --git a/docs/architecture.md b/docs/architecture.md index cf18584..bc13aa7 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1,6 +1,6 @@ -# System Architecture (Current Baseline: V5.1) +# System Architecture (Current Baseline: V6.1) -This document defines the current V5.1 architecture baseline. +This document defines the current V6.1 architecture baseline. ## Architecture Objectives @@ -13,10 +13,11 @@ This document defines the current V5.1 architecture baseline. - **Runtime:** Python 3.12+ - **Web application:** FastAPI + NiceGUI -- **Persistence:** SQLModel / SQLAlchemy (SQLite-first, PostgreSQL-compatible model design) +- **Persistence:** SQLModel / SQLAlchemy — PostgreSQL in production, SQLite for local development and tests - **Validation and settings:** Pydantic V2 + pydantic-settings - **Concurrency:** asyncio worker loop - **Provider integration:** OpenRouter adapter behind provider interface +- **Deployment:** Docker Compose (app, worker, PostgreSQL, Cloudflare Tunnel) - **Quality and tests:** Ruff, ty, pytest, pytest-asyncio ## Runtime Topology @@ -25,13 +26,31 @@ This document defines the current V5.1 architecture baseline. flowchart LR U[Browser User] --> A[FastAPI + NiceGUI App] A --> W[Asyncio Worker] -A --> DB[(SQLite/PostgreSQL Model)] +A --> DB[(PostgreSQL / SQLite)] W --> P[Provider Adapter] W --> DB ``` -For production split-process deployments, the worker may run as a dedicated service -while the app process runs with `RUN_EMBEDDED_WORKER=false`. +The worker loop drains two queues in the same pass: queued transcription Jobs and queued +`MaintenanceRun` records. When neither has work, it idles. + +### Production deployment + +Production runs as a Docker Compose stack with the app and worker as separate services, so the app +process runs with `RUN_EMBEDDED_WORKER=false` and the worker process owns queue draining. Local +development runs a single process with the worker embedded. + +```mermaid +flowchart LR +I[Internet] --> CF[cloudflared tunnel + Access] +CF --> APP[app service] +APP --> PG[(postgres service)] +WK[worker service] --> PG +WK --> PROV[OpenRouter] +``` + +Deployment, rollback, and recovery procedures are in [Production Runbook](production-runbook.md); +backup configuration and restore are in [Backup and Restore](backup_restore.md). ## Layered Boundaries @@ -48,11 +67,18 @@ Responsibilities: ### Service and Orchestration Layer +Aggregate services: + - `src/transcription/services/documents.py` - `src/transcription/services/people.py` - `src/transcription/services/jobs.py` - `src/transcription/services/sources.py` -- `src/transcription/services/evidence.py` +- `src/transcription/services/photos.py` +- `src/transcription/services/maintenance.py` +- `src/transcription/services/evidence.py` (read/projection only) + +Orchestration modules: + - `src/transcription/services/store.py` - `src/transcription/services/workflows.py` @@ -61,6 +87,11 @@ Responsibilities: - Aggregate ownership and invariants. - Transaction-aware write helpers. - Cross-service workflows in orchestration modules (`store.py`, `workflows.py`). +- Lookup-table CRUD through the generic `RegistryService` base (`registry.py`), which is not an + aggregate owner itself. + +Module classification and per-model ownership are defined in +[services instructions](../.github/instructions/services.instructions.md). ### Persistence Layer @@ -88,7 +119,11 @@ Responsibilities: - `Job` is an aggregate processing run with status and frozen prompt/runtime settings. - `JobSource` is queue/membership state for one `(job, source)` pair. - `ExecutionAttempt` is append-only evidence for each provider call. +- `Photo` is person imagery owned by `PhotosService`. +- `MaintenanceRun` is one queued or executed operational maintenance run. - `DocumentType` and `PersonRole` are UUID-backed registries with optional protected `semantic_key`. +- `Tag` is a shared registry reached through both document and person tagging, linked by + `DocumentTag` and `PersonTag`. ## Processing and Evidence Workflow @@ -109,6 +144,24 @@ Responsibilities: - **Job statuses:** `queued`, `processing`, `transcribed`, `partial_success`, `failed` - Operational success path resolves to `transcribed`. - **JobSource statuses:** `pending`, `transcribed`, `failed`, `cancelled` +- **MaintenanceRun statuses:** `queued`, `processing`, `succeeded`, `failed` + - Maintenance uses `succeeded` rather than `transcribed`; the transcription vocabulary does not + apply to operational runs. +- **Maintenance job types:** `backup`, `storage_reconciliation` + +## Maintenance Execution + +Operational maintenance is queue-backed rather than run inline from the UI, so it survives request +lifetime and is recorded: + +1. Settings enqueues a `MaintenanceRun` with `status=queued` and a `triggered_by` marker. +2. The worker claims the oldest queued run with a conditional update, moving it to `processing`. +3. `backup` runs the deploy backup script; `storage_reconciliation` compares stored media against + `Document`/`Source` records. +4. The run finalizes to `succeeded` or `failed` with summary, timing, log path, and `error_detail`. + +`MaintenanceRun` records operational history and is not evidence in the `ExecutionAttempt` sense; +append-only guarantees apply to transcription attempts. ## Security and Path Handling Boundaries @@ -168,5 +221,7 @@ Current architecture rules live in `docs/*`. - [System Requirements](requirements.md) - [Data Model](schema.md) - [Error Handling Policy](error_handling.md) +- [Production Runbook](production-runbook.md) +- [Backup and Restore](backup_restore.md) - [Error Handling invariant](./invariant/error_handling.md) - [AI evidence invariant](./invariant/ai_evidence_and_provenance.md) diff --git a/docs/error_handling.md b/docs/error_handling.md index 8bf3c35..5bb4fea 100644 --- a/docs/error_handling.md +++ b/docs/error_handling.md @@ -1,6 +1,6 @@ -# Error Handling Policy (Current Baseline: V5.1) +# Error Handling Policy (Current Baseline: V6.1) -This policy defines the active V5.1 error taxonomy, translation boundaries, and retry semantics. +This policy defines the active V6.1 error taxonomy, translation boundaries, and retry semantics. ## Error Categories diff --git a/docs/index.md b/docs/index.md index 10d4a68..47fc89e 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,6 +1,6 @@ -# Document Transcription System Overview (Current Baseline: V5.1) +# Document Transcription System Overview (Current Baseline: V6.1) -This directory is the single source of truth for current V5.1 behavior and architecture. +This directory is the single source of truth for current V6.1 behavior and architecture. ## Canonical Reading Order @@ -17,7 +17,20 @@ This directory is the single source of truth for current V5.1 behavior and archi - [Digital Evidence and AI Processing Provenance](./invariant/ai_evidence_and_provenance.md) - [UI Style Guide](./invariant/ui_style_guide.md) +## Deployment and Operations + +- [Production Runbook](production-runbook.md) for deploy, rollback, and recovery. +- [Backup and Restore](backup_restore.md) for backup configuration and restore procedure. +- [Data Migration](data_migration.md) for the SQLite to PostgreSQL migration path. +- [Cloudflare Tunnel and Access](cloudflare_tunnel_access.md) for remote exposure and access control. + ## Baseline Statement -The current V5.1 baseline includes behavior delivered through the architectural cleanup phases and person-schema redesign. +The current V6.1 baseline includes behavior delivered through the architectural cleanup phases and +person-schema redesign (through V5.1), the V6.0 hosting migration to a containerized PostgreSQL +deployment, and the V6.1 navigation, Document Detail, and worker-backed maintenance refinements. Use this `docs/*` canonical set for active design and implementation decisions. + +Every canonical document above states this same baseline; `tests/test_meta_contract_guards.py` +fails if one of them falls behind. Forward-looking work is tracked in +[`roadmap_plan.md`](roadmap_plan.md) and is not part of the baseline. diff --git a/docs/requirements.md b/docs/requirements.md index 44d21e3..da5218d 100644 --- a/docs/requirements.md +++ b/docs/requirements.md @@ -1,6 +1,9 @@ -# System Requirements (Current Baseline: V5.1) +# System Requirements (Current Baseline: V6.1) -These requirements define the active V5.1 contract and align to current implementation. +These requirements define the active V6.1 contract and align to current implementation. + +Requirement IDs encode the baseline that introduced them (`REQ-4-*` from V4, `REQ-6-*` from V6) and +are stable. Never renumber an existing ID; retire it explicitly instead. ## Functional Requirements @@ -41,6 +44,29 @@ These requirements define the active V5.1 contract and align to current implemen - **REQ-4-041 Partial Failure Visibility:** Mixed page outcomes must be visible at job and page level. - **REQ-4-042 Retry Support:** Failed and cancelled pages must support targeted retranscription without requiring full document recreation. +### Person Imagery + +- **REQ-6-001 Photo Records:** The system must store reusable `Photo` records that are either owned by a `Person` or unowned for homepage gallery use. +- **REQ-6-002 Primary Photo:** At most one photo per owning `Person` may be marked `is_primary`; setting a new primary must clear the previous one. +- **REQ-6-003 Primary Reassignment:** Deleting an owner's primary photo must promote a remaining photo of that owner rather than leaving the owner without a primary. + +### Operational Maintenance + +- **REQ-6-010 Queued Maintenance Runs:** Settings-initiated maintenance must persist a `MaintenanceRun` and execute in the worker, not inline in the request that started it. +- **REQ-6-011 Maintenance Run Types:** `MaintenanceRun.job_type` must use one of `backup`, `storage_reconciliation`. +- **REQ-6-012 Maintenance Status Lifecycle:** `MaintenanceRun.status` must use one of `queued`, `processing`, `succeeded`, `failed`. +- **REQ-6-013 Single Claim:** A queued run must be claimed by at most one worker, using a conditional status update rather than read-then-write. +- **REQ-6-014 Run History:** Completed runs must retain status, timing, summary, log reference, and error detail, and expose the log for viewing and download. + +### Deployment and Runtime Configuration + +- **REQ-6-020 Production Persistence:** Production must run against PostgreSQL; SQLite remains supported for local development and tests. +- **REQ-6-021 Split Worker Deployment:** Production must support running the worker as its own process with the app started at `RUN_EMBEDDED_WORKER=false`. +- **REQ-6-022 Runtime Settings Persistence:** Runtime settings edits must persist to the mounted production environment file and survive container restart. +- **REQ-6-023 Configuration Contract Sync:** `.env.production.example` must stay synchronized with `Settings` keys and production-safe defaults. +- **REQ-6-024 Health Reporting:** The deployed stack must report app and worker health through `/healthz`. +- **REQ-6-025 Backup and Restore:** Database and media/config backups must be produced on a host-visible path with a tested restore procedure. + ## Non-Functional Requirements - **REQ-4-100 Boundary Integrity:** UI pages/components must not access persistence directly and must call service APIs. @@ -73,6 +99,11 @@ These requirements define the active V5.1 contract and align to current implemen - UI boundary enforcement: `tests/test_ui_boundaries.py` - Job lifecycle reliability and terminal status behavior: `tests/services/test_workflows_reliability.py` - Evidence append-only and projection behavior: `tests/services/test_store.py`, `tests/services/test_transcription_service.py` +- Person photo ownership and primary selection: `tests/services/test_photo_service.py` +- Maintenance run lifecycle and worker execution: `tests/services/test_maintenance_service.py` +- Runtime settings persistence: `tests/ui/test_runtime_settings_store.py`, `tests/services/test_settings_services.py` +- Configuration contract synchronization: `tests/test_meta_contract_guards.py` +- Deployment health reporting: `tests/api/test_health.py` ## Traceability Notes diff --git a/docs/ui/README.md b/docs/ui/README.md index e7eb56b..57a3e43 100644 --- a/docs/ui/README.md +++ b/docs/ui/README.md @@ -13,6 +13,7 @@ These documents are written for maintainers and AI contributors. They are behavi - [People](pages/people.md) - [Jobs](pages/jobs.md) - [Sources](pages/sources.md) +- [Settings](pages/settings.md) NiceGUI registers the routes shown in each contract without the `/ui` prefix. The application mounts NiceGUI under `/ui`, so `/documents` in page code is served to a browser as `/ui/documents`. @@ -56,4 +57,4 @@ Each page contract contains: ## Current Baseline -These contracts describe the current V5.1 baseline. +These contracts describe the current V6.1 baseline. diff --git a/tests/test_meta_contract_guards.py b/tests/test_meta_contract_guards.py index 6e34957..3662fb9 100644 --- a/tests/test_meta_contract_guards.py +++ b/tests/test_meta_contract_guards.py @@ -43,6 +43,70 @@ def _read(relative_path: str) -> str: return (PROJECT_ROOT / relative_path).read_text(encoding="utf-8") +BASELINE_SOURCE = "docs/index.md" + +# Docs that legitimately discuss versions other than the current baseline: the roadmap plans +# future work, and migration/deployment notes describe historical phases by name. +BASELINE_SCAN_EXCLUSIONS = frozenset( + { + "docs/roadmap_plan.md", + "docs/data_migration.md", + "docs/cloudflare_tunnel_access.md", + "docs/production-runbook.md", + "docs/backup_restore.md", + } +) + +_BASELINE_DECLARATION = re.compile(r"Current Baseline:\s*V(\d+\.\d+)") +_CURRENT_VERSION_CLAIM = re.compile(r"\b(?:current|active)\s+V(\d+\.\d+)", re.IGNORECASE) + + +def _declared_baseline() -> str: + """Return the one baseline version the canonical doc set must agree on.""" + match = _BASELINE_DECLARATION.search(_read(BASELINE_SOURCE)) + assert match is not None, f"{BASELINE_SOURCE} must declare 'Current Baseline: V.'" + return match.group(1) + + +def _baseline_scanned_docs() -> list[Path]: + docs_root = PROJECT_ROOT / "docs" + return sorted( + path + for path in docs_root.rglob("*.md") + # docs/reviews/** are dated, non-canonical snapshots and are pinned to the version + # that was current when they were written. + if "reviews" not in path.relative_to(docs_root).parts + and path.relative_to(PROJECT_ROOT).as_posix() not in BASELINE_SCAN_EXCLUSIONS + ) + + +def test_canonical_docs_declare_one_consistent_baseline(): + """A stale baseline label makes canonical docs read as drift against current code. + + Every canonical doc that names the current baseline must name the same one, so a version + bump cannot leave half the authority set describing a superseded release as current. + """ + baseline = _declared_baseline() + violations: dict[str, list[str]] = {} + + for path in _baseline_scanned_docs(): + relative = path.relative_to(PROJECT_ROOT).as_posix() + text = path.read_text(encoding="utf-8") + stale = { + version + for pattern in (_BASELINE_DECLARATION, _CURRENT_VERSION_CLAIM) + for version in pattern.findall(text) + if version != baseline + } + if stale: + violations[relative] = sorted(stale) + + assert violations == {}, ( + f"Canonical baseline is V{baseline} (declared in {BASELINE_SOURCE}), " + f"but these docs claim another version is current: {violations}" + ) + + def test_active_contract_files_are_present(): """Guard the guard: ensure all expected authority files are scanned.""" missing = [path for path in ACTIVE_CONTRACT_FILES if not (PROJECT_ROOT / path).exists()]