diff --git a/docs/ver4/architecture_v4.md b/docs/ver4/architecture_v4.md index b1147c4..3d05367 100644 --- a/docs/ver4/architecture_v4.md +++ b/docs/ver4/architecture_v4.md @@ -1,137 +1,147 @@ # System Architecture (Version 4) -This document describes the V4 architecture changes for expanded `Document`-`Person` relationship management. - -V4 extends V3 with role extensibility, assisted suggestion review, and role-policy enforcement while preserving the existing transcription execution topology. +This document describes the production architecture of the document transcription system. ## Architecture Objectives -- Keep V3 transcription pipeline behavior stable unless relationship evidence extraction requires additive integration. -- Support many people per role for each document. -- Support extensible relationship roles without hardcoding UI and API behavior to two values. -- Support non-canonical suggestion intake with explicit human promotion to canonical asserted links. -- Enforce role exclusivity policy consistently at service and persistence boundaries. -- Preserve auditability for suggestion provenance and review actions. +- Preserve original source material and immutable machine transcription output. +- Support batching one or more images into ordered multi-page documents. +- Capture complete submission-time prompt provenance and per-page provider response evidence. +- Execute page transcription concurrently with bounded `asyncio` workers. +- Maintain relational portability across SQLite and PostgreSQL. +- Keep operator workflows cross-platform and Python-driven. +- Support many-to-many document-person relationships with extensible roles. +- Support registry-driven document type classification. +- Enforce relationship-role exclusivity rules consistently across UI, API, and persistence boundaries. ## Runtime Topology -V4 keeps the existing runtime shape: +The runtime operates as an asynchronous Python application: -- FastAPI + NiceGUI web app process. -- Async service layer with SQLModel/SQLAlchemy persistence. -- Existing worker execution path for transcription jobs. - -Additive V4 components: - -- Relationship policy evaluator (exclusivity checks). -- Suggestion lifecycle service (propose/list/accept/reject). -- Optional suggestion extraction adapter (rule/NLP or LLM-backed). +- FastAPI + NiceGUI web application process. +- In-process `asyncio` worker engine for transcription execution. +- Relational persistence via SQLModel / SQLAlchemy. +- Pydantic V2 validation across API payloads, prompt configuration, and structured metadata. ^^^mermaid flowchart LR -U[Browser User] --> UI[NiceGUI Pages] -UI --> API[FastAPI Routes] -API --> DS[Document Service] -API --> SS[Suggestion Service] -DS --> PE[Policy Evaluator] -SS --> PE -DS --> DB[(Relational DB)] -SS --> DB -W[Async Worker Engine] --> DB -W --> E[Optional Suggestion Extraction] -E --> SS +U[Browser User] --> A[FastAPI + NiceGUI App] +A --> W[Asyncio Worker Engine] +A --> DB[(Relational DB)] +W --> P[Vision Provider APIs] +W --> DB ^^^ -## Layer Responsibilities +## Lifecycle Ownership -### UI Layer (`src/transcription/ui/**`) +Application lifespan owns runtime setup and teardown: -- Render per-role grouped relationships for document and person views. -- Provide multi-select role assignment controls in document create/edit flows. -- Provide suggestion review controls (accept/reject) for pending suggestions. -- Surface policy conflict errors from API/service layer clearly. +- Initialize logging, settings, directories, and prompt configuration. +- Manage asynchronous database engine connection pools. +- Execute database bootstrap or migrations. +- Recover stale or interrupted jobs on startup. +- Manage graceful shutdown of active background tasks. -### API Layer (`src/transcription/api/**`) +## Layered Module Structure -- Expose role-aware and state-aware read contracts. -- Expose suggestion lifecycle write contracts. -- Return deterministic validation/conflict errors for exclusivity and duplicate semantics. -- Evolve endpoints additively, with explicit deprecations and short-lived transition windows. +### Interface Layer -### Service Layer (`src/transcription/services/**`) +- `src/transcription/ui/**` +- `src/transcription/api/**` -- Implement set-based relationship sync (delta add/remove) to avoid destructive replacement behavior. -- Apply policy evaluator before persistence commits. -- Orchestrate suggestion acceptance/rejection transitions. -- Ensure accepted suggestions result in asserted link creation/confirmation. +Responsibilities: -### Policy Evaluator (New logical component) +- Render document, source, person, job, and classification views. +- Accept user input for uploads, editing, linking, and revisions. +- Present structured validation and conflict feedback. -- Evaluate role exclusivity matrix for `(document_id, person_id)` writes. -- Provide deterministic conflict reason payloads suitable for UI/API display. -- Stay stateless and reusable across create/update/sync code paths. +### Application and Async Worker Layer -### Persistence Layer (`src/transcription/db/**`) +- `src/transcription/services/workflows.py` +- `src/transcription/worker.py` -- Store asserted links and suggestion records according to selected schema option. -- Persist provenance metadata for suggestion records. -- Enforce uniqueness and support performant role/state filtering. +Responsibilities: -## Core V4 Workflows +- Orchestrate uploads, job creation, and status transitions. +- Execute per-page provider calls through bounded concurrency. +- Persist page-level outcomes and update aggregate job state. -### 1) Manual Relationship Management +### Domain and Service Layer -1. User opens document edit view. -2. UI loads asserted links grouped by role. -3. User adds/removes people per role. -4. Service computes delta and runs policy checks. -5. Persistence applies adds/removes atomically. +- `src/transcription/db/models.py` +- `src/transcription/services/*.py` -### 2) Assisted Suggestion Review +Responsibilities: -1. Suggestion records are created (`pending`) by extraction logic or manual propose action. -2. UI displays pending suggestions with evidence metadata. -3. User accepts or rejects each suggestion. -4. Accept path creates/confirms asserted relationship and marks suggestion `accepted`. -5. Reject path marks suggestion `rejected`. +- Manage transactional operations for documents, people, types, links, sources, jobs, and job sources. +- Apply deterministic conflict handling for relationship-role writes. +- Use set-based synchronization for many-to-many relationship updates. +- Resolve and validate registry-backed document types. -### 3) Exclusivity Conflict Handling +### Infrastructure Layer -1. Incoming write attempts role assignment. -2. Policy evaluator checks role pair conflicts for same `(document_id, person_id)`. -3. If conflict exists, write is rejected with structured conflict details. -4. UI presents actionable message without partial updates. +- `src/transcription/db/**` +- `src/transcription/providers/**` -## Invariants +Responsibilities: -- Suggested links are never canonical until accepted. -- Asserted links must satisfy exclusivity rules. -- Relationship mutations are set-based and deterministic. -- Relationship views remain consistent between document detail and person detail pages. -- Existing V3 links remain valid under asserted semantics after migration. +- Provide async database sessions and engine configuration. +- Provide provider adapters for vision model execution. -## Backward Compatibility +## Core Workflows -- V3 author/recipient links are interpreted as asserted links in V4. -- Existing document and person flows continue to function where unaffected by new role/state dimensions. -- During development revisions, the current API contract is authoritative; long-lived legacy runtime compatibility layers are not required. +### 1. Multi-Page Transcription -## Observability and Auditability +1. User uploads one or more images for a `Document`. +2. System stores files, hashes them, creates ordered `Source` rows, and creates a `Job`. +3. Worker claims the job, marks it `processing`, and executes page calls concurrently. +4. Each page writes a `JobSource` result with raw output, metadata, and full provider response evidence. +5. Aggregate status becomes `completed`, `partial_success`, or `failed`. -- Persist suggestion provenance fields sufficient for operator review. -- Record review decision outcomes (`accepted`/`rejected`) with timestamps and reviewer identity where available. -- Emit structured service-level logs for exclusivity conflicts and review actions. +### 2. Document-Person Relationship Management -## Test Strategy Additions +1. User opens a document or person edit flow. +2. UI loads existing links grouped by role. +3. User adds or removes people within one or more roles. +4. Service computes add/remove deltas rather than replacing all links blindly. +5. Conflict checks enforce exclusivity policy before persistence commits. -- Service tests for delta sync and exclusivity matrix enforcement. -- API tests for role/state filtering and suggestion transitions. -- UI tests for multi-role selection and suggestion review interactions. -- Migration tests for V3-to-V4 asserted mapping and conflict scans. +### 3. Document Type Management + +1. User selects a registry-backed document type for a document. +2. Service resolves the stable type code or id. +3. Persistence stores the `document_type_id` reference. +4. Inactive types remain valid for historical rows but are excluded from default selectors. + +## Domain Invariants + +- `Source.raw_transcription` stores immutable machine output. +- Human corrections occur only in `Source.revised_text`. +- Prompt and parameter provenance is frozen on `Job` at submission time. +- Provider output evidence is stored on `JobSource` for each page execution. +- `DocumentPerson` links are unique for `(document_id, person_id, role_id)`. +- Configured exclusive role pairs cannot coexist for the same `(document_id, person_id)`. +- Relationship mutations are deterministic and set-based. +- `DocumentType.code` is stable; `DocumentType.label` may evolve. + +## Data Model Summary + +- `Document` has one `DocumentType`, many `Source` pages, many `Job` runs, and many `Person` records through `DocumentPerson`. +- `Source` belongs to one `Document` and may participate in many `JobSource` executions. +- `Job` has many `JobSource` rows. +- `PersonRole` defines available relationship roles. +- `RoleExclusivity` defines role pairs that cannot coexist for the same document-person pair. + +## Test Strategy + +- Unit tests for models, validation, hashing, and registry resolution. +- Service tests for CRUD, set-based sync, and exclusivity enforcement. +- Async workflow tests for page isolation, partial failure handling, and stored evidence. +- UI integration tests for multi-page rendering, role grouping, and document type selection. ## Related Local References -- [V4 Scope Boundary](scope_boundary_v4.md) -- [V4 Requirements](requirements_v4.md) -- [V4 Schema](schema_v4.md) -- [V3 Architecture](../architecture_v3.md) +- [System Overview](index_v4.md) +- [System Requirements](requirements_v4.md) +- [Data Model](schema_v4.md) +- [Error Handling Policy](error_handling_v4.md) diff --git a/docs/ver4/error_handling_v4.md b/docs/ver4/error_handling_v4.md index fd732b9..9fce8b7 100644 --- a/docs/ver4/error_handling_v4.md +++ b/docs/ver4/error_handling_v4.md @@ -1,62 +1,68 @@ # Error Handling Policy (Version 4) -This document defines canonical error-handling behavior for V4 document-person relationship expansion. - -V4 keeps V3 transcription error behavior and adds policy/conflict handling for role extensibility, suggestion lifecycle transitions, and exclusivity enforcement. +This document defines the canonical error-handling policy for the document transcription system. ## Error Handling Objectives -- Provide clear, actionable conflict and validation feedback for relationship write operations. -- Prevent partial, silent, or destructive relationship mutations when policy checks fail. -- Preserve suggestion review auditability with deterministic accept/reject outcomes. -- Keep consistent API/UI/service error envelopes across relationship workflows. +- Make failures visible in clear, actionable language at both the document and page levels. +- Support isolated failure handling in multi-page jobs so one failing page does not invalidate successful pages. +- Preserve diagnostic detail for validation failures, provider failures, and policy conflicts. +- Ensure consistent error envelope structure across API, UI, service, and worker boundaries. ## Scope and Authority -Governs relationship-related error behavior in: +This policy governs error behavior across: -- NiceGUI document/person relationship views, -- FastAPI relationship and suggestion endpoints, -- domain services for relationship sync and suggestion review, -- persistence constraints for role, state, and exclusivity invariants. +- NiceGUI pages +- FastAPI routes +- Service-layer orchestration +- `asyncio` worker tasks +- Database interactions +- Provider adapters -## Relationship Error Taxonomy +## Error Taxonomy | Category | Definition | Retriable | | --- | --- | --- | -| `validation_error` | Payload shape/type invalid, unknown role/state, malformed IDs | no | -| `not_found_error` | Target `Document`, `Person`, role, or suggestion record does not exist | no | -| `conflict_error` | Write violates uniqueness or exclusivity policy | no | -| `suggestion_state_error` | Invalid suggestion transition (for example accept after reject) | no | -| `policy_violation_error` | Action blocked by configured role matrix or governance rule | no | -| `infrastructure_transient_error` | Temporary DB or network instability during relationship operation | yes | -| `infrastructure_persistent_error` | Persistent DB/configuration failure | no | -| `internal_unexpected_error` | Unhandled exception/logic defect | no | +| `validation_error` | Payload, parameter, or schema validation failure | no | +| `user_input_error` | Unacceptable file, invalid selection, or malformed request from the operator | no | +| `not_found_error` | Requested `Document`, `Source`, `Person`, `Job`, role, or type does not exist | no | +| `conflict_error` | Operation violates uniqueness or exclusivity policy | no | +| `external_provider_error` | Provider API failure, rate limit, or execution problem | yes | +| `infrastructure_transient_error` | Temporary DB, file-system, or network instability | yes | +| `infrastructure_persistent_error` | Persistent configuration, credential, or database availability failure | no | +| `internal_unexpected_error` | Uncaught exception or logic defect | no | -## Deterministic Conflict Behavior +## Async Batch and Page-Level Error Behavior -When relationship writes fail policy checks: +In multi-page `asyncio` processing: -1. Reject the full write operation (no partial apply). -2. Return structured conflict details including conflicting role pair and target identifiers. -3. Preserve existing canonical relationships unchanged. +1. Exceptions from individual page calls are trapped within the page task wrapper. +2. Failed page detail is written to `JobSource.error_detail` and the page state becomes `failed`. +3. Aggregate job status is derived from page outcomes: + - all pages succeed -> `completed` + - some succeed and some fail -> `partial_success` + - all fail -> `failed` +4. Successful pages remain valid even when sister pages fail. -When suggestion transitions fail: +## Relationship and Classification Conflict Behavior -1. Reject invalid state transition. -2. Return current state and allowed next actions. -3. Preserve suggestion record integrity. +When relationship or document-type writes fail policy checks: + +1. Reject the full write operation. +2. Return structured conflict detail including target identifiers and the violated rule. +3. Preserve existing persisted relationships unchanged. ## API Error Response Contract -Relationship endpoints return a structured envelope: +API error responses return a structured envelope: ^^^json { "error_id": "err_uuid_12345", "category": "conflict_error", "message": "Role assignment violates exclusivity policy.", - "suggestion": "Remove recipient role before assigning author for this person on this document.", + "suggestion": "Remove recipient before assigning author for this person on this document.", "details": { "document_id": "...", "person_id": "...", @@ -64,41 +70,41 @@ Relationship endpoints return a structured envelope: "conflicting_role": "recipient", "policy_rule": "author+recipient exclusive" }, - "timestamp": "2026-08-09T15:00:00Z" + "timestamp": "2026-08-10T15:00:00Z" } ^^^ HTTP status mappings: -- `validation_error` -> `400` +- `validation_error`, `user_input_error` -> `400` - `not_found_error` -> `404` -- `conflict_error`, `suggestion_state_error`, `policy_violation_error` -> `409` +- `conflict_error` -> `409` +- `external_provider_error` -> `502` or `503` - `infrastructure_transient_error` -> `503` - `infrastructure_persistent_error`, `internal_unexpected_error` -> `500` ## UI Error Presentation Rules -- Display concise conflict summary with actionable next step. -- Keep user edits in context (do not discard form state when feasible). -- Differentiate between validation issues, policy conflicts, and infrastructure failures. -- For bulk role sync operations, show per-item conflict context when multiple failures occur. +- Display concise failure summaries with the next action the operator can take. +- Keep form state in context when feasible. +- Distinguish validation issues, conflict issues, provider failures, and infrastructure failures. +- For bulk relationship updates, identify the specific role or person that caused a conflict. ## Logging and Audit Expectations -- Log relationship write failures with correlation IDs. -- Log suggestion acceptance/rejection outcomes with actor and timestamp where available. -- Log policy matrix violations with deterministic machine-readable context. +- Log worker failures with correlation IDs and provider context. +- Log relationship and classification conflicts with machine-readable detail. +- Log persisted provider errors and page-level execution failures. -## Relationship-Specific Retry Guidance +## Retry Guidance -- Do not auto-retry policy or conflict failures. -- Permit user-driven retry only after input changes. -- Retry infrastructure transient failures with bounded policy in service layer if operation is idempotent. +- Do not auto-retry validation or conflict failures. +- Permit user-driven retry after the input or selection changes. +- Allow bounded retry for transient provider or infrastructure failures when the operation is idempotent. ## Related Local References -- [V4 Scope Boundary](scope_boundary_v4.md) -- [V4 Requirements](requirements_v4.md) -- [V4 Schema](schema_v4.md) -- [V4 Architecture](architecture_v4.md) -- [V3 Error Handling](../error_handling_v3.md) +- [System Overview](index_v4.md) +- [System Requirements](requirements_v4.md) +- [Data Model](schema_v4.md) +- [System Architecture](architecture_v4.md) diff --git a/docs/ver4/implementation_plan_v4.md b/docs/ver4/implementation_plan_v4.md index aaadb1e..eb75af6 100644 --- a/docs/ver4/implementation_plan_v4.md +++ b/docs/ver4/implementation_plan_v4.md @@ -2,106 +2,96 @@ ## Goal -Implement V4 document-person relationship expansion and document type governance with extensible registries, assisted suggestion lifecycle, and policy-enforced exclusivity while preserving V3 transcription behavior. +Implement the Version 4 project definition from the current repository state while preserving existing data by default. + +## Migration Policy + +- Database changes are non-destructive by default. +- Exception: the legacy `document_type` text field may be replaced by a `document_type_id` reference without migrating existing text values. +- Exception: `document_person` links may be recreated manually. ## Current Project Impact -- `src/transcription/db/models.py` will require relationship schema evolution for role extensibility and suggestion lifecycle support. -- `src/transcription/db/models.py` will require document type registry entities and document type reference updates. -- `src/transcription/services/documents.py` will require set-based relationship sync and policy checks. -- `src/transcription/services/documents.py` will require registry-based document type lookup, validation, and normalization helpers. -- API modules under `src/transcription/api/**` will require role/state-aware contracts and suggestion lifecycle endpoints. -- API modules under `src/transcription/api/**` will require additive document type catalog and code-based selection contracts. -- UI pages under `src/transcription/ui/pages/**` will require multi-role, multi-person editing, suggestion review controls, and registry-backed type selectors. -- Existing tests under `tests/services`, `tests/api`, and `tests/ui` need expanded coverage for V4 behavior and regression safety. +- `src/transcription/db/models.py` requires full schema alignment with the V4 core documents. +- `src/transcription/services/documents.py` requires set-based document-person sync, exclusivity checks, and document-type resolution. +- API modules require additive role-aware relationship behavior and document-type selection behavior. +- UI pages require grouped role displays, multi-role editing, and registry-backed document-type selection. +- Existing tests require updates for role enforcement, document-type selection, and regression safety. ## Implementation Phases -### 1. Finalize V4 Schema Decisions +### 1. Finalize the Transition Documents -- Role extensibility mechanism: use role registry tables. -- Document type extensibility mechanism: use `document_type` registry tables. -- Suggestion storage model: use a separate `document_person_suggestion` table. -- Exclusivity baseline: `author` vs `recipient` exclusive, `mentioned` non-exclusive. -- Enforce exclusivity on asserted links; evaluate conflicts on suggestion acceptance. +- Confirm the reset scope. +- Confirm the database exception policy. +- Keep core V4 documents as the only authoritative product definition. -### 2. Evolve Persistence Layer +### 2. Align the Persistence Layer -- Implement selected schema model in SQLModel. -- Add constraints and indexes for dedupe, filtering, and policy support. -- Add migration/backfill logic for V3 links to V4 asserted semantics. -- Add conflict scan tooling for historical records violating exclusivity rules. -- Perform a one-time manual mapping of existing document type values to registry-backed type references (`document_type_id`) for the current small corpus. +- Update SQLModel definitions to match the final V4 schema. +- Add `person_role`, `role_exclusivity`, and `document_type` support. +- Replace legacy document-type storage with `document_type_id`. +- Apply the accepted manual exception strategy for `document_type` and `document_person` data. +- Preserve all other data structures non-destructively. -### 3. Implement Service-Layer Policy and Sync Semantics +### 3. Update Services and Write Semantics -- Add relationship delta sync operations (set-based add/remove). -- Implement centralized policy evaluator for exclusivity checks. -- Add suggestion lifecycle operations (propose/list/accept/reject). -- Ensure accepted suggestions create/confirm asserted links atomically. -- Add document type resolution operations by stable `code` and active/inactive state handling. +- Implement set-based synchronization for document-person updates. +- Implement deterministic exclusivity conflict checks. +- Remove suggestion-related service behavior. +- Add document-type resolution and validation by stable code or id. -### 4. Implement API Contract Changes +### 4. Update API Contracts -- Add role-aware and state-aware query parameters/filters. -- Add suggestion lifecycle endpoints and response models. -- Add deterministic conflict/error payloads aligned to V4 error policy. -- Use additive endpoint evolution for V4; during development mode, deprecate then remove without maintaining long-lived legacy runtime compatibility layers. -- Add document type catalog endpoints with active-only filtering and code-based selection for document writes. +- Keep API evolution additive. +- Add role-aware relationship retrieval and write behavior. +- Add document-type catalog retrieval and code-based selection for document writes. +- Remove suggestion-related API surfaces from the V4 target state. ### 5. Update UI Workflows -- Replace single-author controls with grouped multi-role selectors. -- Replace unconstrained document type free-text entry with registry-backed type selection. -- Add pending suggestion review panel and accept/reject actions. -- Update document/person detail cards to group links by role and state. -- Preserve edit-state ergonomics on validation/conflict failures. +- Replace single-person link editing with grouped multi-role editing. +- Render grouped role links on document and person detail views. +- Replace free-text document type entry with registry-backed selection. +- Preserve clear validation and conflict messaging. ### 6. Verification and Hardening -- Add service tests for: - - many-per-role behavior, - - exclusivity enforcement, - - set-based sync correctness, - - suggestion transition validity, - - document type code resolution and inactive-type handling. -- Add API tests for role/state filtering and conflict response shapes. -- Add API tests for document type catalog retrieval and code-based write validation. -- Add UI tests/manual walkthroughs for create/edit/review workflows. -- Add UI tests/manual walkthroughs for registry-backed document type selection and validation messaging. -- Add regression tests for document/person delete cleanup semantics. +- Add or update service tests for many-per-role behavior, exclusivity enforcement, and set-based sync correctness. +- Add API tests for relationship behavior and document-type selection. +- Add UI tests or walkthrough coverage for grouped roles and type selection. +- Add regression coverage for delete and cleanup semantics. ## Done When -- V4 relationship schema and contracts are implemented and validated. -- V4 document type registry schema and contracts are implemented and validated. -- Suggestions remain non-canonical until explicit acceptance. -- Exclusivity policy is enforced deterministically across service/API boundaries. -- Existing V3 links are migrated to V4 asserted semantics without data loss. -- Existing document type strings are manually normalized to registry references for the current corpus. -- Test suite includes V4-specific coverage and passes on supported backends. +- Core V4 documents and code paths agree on the final project definition. +- Relationship-role writes are deterministic and non-destructive. +- Exclusivity rules are enforced consistently. +- Document type selection is registry-backed. +- The accepted manual exceptions for `document_type` and `document_person` are completed. +- The focused test coverage passes. ## Out of Scope -- Automatic acceptance of suggestions. -- Global person entity-resolution/merge engine. -- Core transcription execution redesign unrelated to relationship expansion. -- Automated semantic document type classification. +- Suggested/asserted relationship state. +- Suggestion review or extraction workflows. +- Global person entity-resolution engine. +- Automated semantic document-type classification. ## Delivery Order Recommendation -1. Requirements freeze (`requirements_v4.md`). -2. Schema decision freeze (`schema_v4.md`). -3. Error policy freeze (`error_handling_v4.md`). -4. Implementation of persistence and service layer. -5. API and UI changes. -6. Final integration and regression validation. +1. Freeze scope boundary and implementation plan. +2. Freeze core V4 documents. +3. Align persistence models. +4. Align services and API behavior. +5. Align UI behavior. +6. Run focused verification and regression checks. ## Related Local References - [V4 Scope Boundary](scope_boundary_v4.md) -- [V4 Requirements](requirements_v4.md) -- [V4 Schema](schema_v4.md) -- [V4 Architecture](architecture_v4.md) -- [V4 Error Handling](error_handling_v4.md) -- [V3 Implementation Plan](../implementation_plan_v3.md) +- [System Overview](index_v4.md) +- [System Requirements](requirements_v4.md) +- [Data Model](schema_v4.md) +- [System Architecture](architecture_v4.md) +- [Error Handling Policy](error_handling_v4.md) diff --git a/docs/ver4/index_v4.md b/docs/ver4/index_v4.md index 2317e62..51acce3 100644 --- a/docs/ver4/index_v4.md +++ b/docs/ver4/index_v4.md @@ -1,49 +1,40 @@ -# Document-Person Expansion Overview (Version 4) +# Document Transcription System Overview (Version 4) -Version 4 defines the relationship-model evolution track for linking `Document` and `Person` entities with extensible roles, suggestion review, and policy enforcement. +This project is a personal-scale application for transcribing, organizing, and preserving historical documents, images, and related people records. ## Start Here -Read [scope_boundary_v4.md](scope_boundary_v4.md) first to confirm scope and non-scope before implementation work begins. +Read [architecture_v4.md](architecture_v4.md) first for the technical overview and system design. -## Core V4 Capabilities +## Core Capabilities -- Extensible relationship role taxonomy. -- Many-people-per-role linking for documents. -- Explicit distinction between canonical asserted links and pending suggested links. -- Human-in-the-loop suggestion accept/reject workflow. -- Role exclusivity policy enforcement for configured role pairs. -- Role/state-aware API and UI retrieval/presentation behavior. -- Minimal document type governance rollout for the current corpus, with one-time manual mapping and no alias helper table. +- Folder and multi-image ingestion into sequential `Source` pages under a single `Document`. +- Parallel asynchronous AI vision transcription using Python `asyncio` bounded by rate limits. +- Portable relational storage using SQLModel and SQLAlchemy across SQLite and PostgreSQL. +- Complete prompt and response provenance for every transcription job and page execution. +- File-integrity tracking through SHA-256 hashing and stored file sizes. +- Historical `Person` management with many-to-many document links and extensible relationship roles. +- Registry-driven `DocumentType` classification with stable codes and controlled selection. +- Inline human revision of transcribed pages while preserving immutable machine output. +- Partial-failure recovery for multi-page jobs. +- Cross-platform operational workflows driven by Python-based tooling. -## V4 Documentation Index +## Technical Stack -- [Scope Boundary](scope_boundary_v4.md) +- Application Web Framework: FastAPI + NiceGUI +- Persistence Engine: SQLModel / SQLAlchemy +- Data Validation and Schemas: Pydantic V2 +- Concurrency and Workers: Python `asyncio` +- Vision Providers: OpenAI, Anthropic, and OpenRouter adapters + +## Core Documentation Index + +- [System Architecture](architecture_v4.md) - [System Requirements](requirements_v4.md) - [Data Model](schema_v4.md) -- [System Architecture](architecture_v4.md) - [Error Handling Policy](error_handling_v4.md) + +## Transition Documents + +- [Scope Boundary](scope_boundary_v4.md) - [Implementation Plan](implementation_plan_v4.md) - -## Relationship To V3 - -V3 remains the baseline production architecture and requirements set for transcription pipeline behavior. V4 is an additive evolution track focused on document-person relationship semantics and workflows. - -## Decision Status - -Locked decisions: - -1. Role extensibility uses registry tables. -2. Suggestion storage uses a separate suggestion table. -3. Exclusivity baseline is `author` vs `recipient` exclusive, with `mentioned` non-exclusive. -4. API evolution is additive in development mode with explicit deprecate-then-remove behavior. - -Remaining decision: - -1. Suggestion generation strategy (deterministic rules/NLP vs LLM extraction). - -## Related Local References - -- [V3 System Overview](../index_v3.md) -- [V3 Requirements](../requirements_v3.md) -- [V3 Schema](../schema_v3.md) diff --git a/docs/ver4/requirements_v4.md b/docs/ver4/requirements_v4.md index a2d73e0..0607944 100644 --- a/docs/ver4/requirements_v4.md +++ b/docs/ver4/requirements_v4.md @@ -1,70 +1,51 @@ -# Relationship and Document Type Governance Requirements (Version 4) +# Document Transcription System Requirements (Version 4) -This document defines Version 4 baseline requirements for expanding relationships between `Document` and `Person` and introducing governed document type classification. - -V4 preserves all applicable V3 capabilities and adds role extensibility, assisted suggestion workflows, explicit relationship policy enforcement, and registry-driven `Document` type governance. +This document defines the baseline requirements for the document transcription system. ## Requirements Model | ID | Category | Requirement | Verify Method | | --- | --- | --- | --- | -| REQ-0 | System | Provide end-to-end, reviewable, policy-enforced document-person relationship management supporting asserted and suggested links. | demonstration | -| REQ-1 | Functional | Preserve many-to-many `Document` ↔ `Person` linking, and allow multiple people per role on a single document. | test | -| REQ-2 | Functional | Support an extensible role taxonomy for document-person relationships beyond fixed `author`/`recipient`. | inspection | -| REQ-3 | Functional | Represent relationship assertion state explicitly (`asserted` and `suggested`) and keep machine suggestions non-canonical until human acceptance. | test | -| REQ-4 | Functional | Provide assisted suggestion lifecycle operations: create/list/filter suggestions, accept suggestion, reject suggestion, and promote accepted suggestions to asserted links. | test | -| REQ-5 | Policy Constraint | Enforce a role exclusivity matrix for a single `(document_id, person_id)` pair on asserted links; initial rule set must block `author` + `recipient` coexistence while allowing `mentioned` to coexist with other roles. | test | -| REQ-6 | Data Constraint | Store canonical asserted links in `document_person` and lifecycle-managed suggestions in a separate `document_person_suggestion` table. | test | -| REQ-7 | Functional | Provide set-based synchronization behavior for relationship mutations (add/remove delta), replacing single-value replacement patterns that can drop unrelated links. | test | -| REQ-8 | Interface | Render grouped relationship metadata by role and assertion state on document detail and person detail views. | demonstration | -| REQ-9 | Interface | Document create/edit UI must support selecting multiple people per role and reviewing pending suggestions with explicit accept/reject controls. | demonstration | -| REQ-10 | API Constraint | Expose additive, role-aware and state-aware retrieval/filtering in API contracts for documents, people, and relationship records. | test | -| REQ-11 | Data Provenance | Capture suggestion provenance metadata sufficient for operator review (for example source mechanism, confidence, and evidence reference) without mutating canonical asserted links implicitly. | inspection | -| REQ-12 | Operations | Provide migration/backfill validation that identifies and resolves historical records violating newly enforced exclusivity policies before hard enforcement. | test | -| REQ-13 | Revision Upgrade | Support deterministic in-place revision upgrade behavior by mapping existing author/recipient links into V4 asserted semantics without requiring long-lived runtime legacy compatibility layers. | test | -| REQ-14 | Reliability | Ensure document/person deletion and cleanup workflows remain safe and deterministic with expanded relationship semantics, including suggestion records. | test | -| REQ-15 | Quality | Add automated test coverage for role extensibility, exclusivity enforcement, suggestion lifecycle transitions, and regression scenarios across service/API/UI flows. | test | -| REQ-16 | Functional | Support a registry-driven document type taxonomy (`document_type`) with stable machine-readable codes, mutable display labels, and active/inactive lifecycle control. | test | -| REQ-17 | Data Constraint | Replace unconstrained free-text document type assignment with controlled type references or deterministic code mapping governed by the document type registry. | test | -| REQ-18 | Interface | Document create/edit UI must present type selection from active registry entries and prevent invalid type assignment. | demonstration | -| REQ-19 | API Constraint | Provide additive API contracts for document type catalog retrieval, including active-only filtering and stable code-based selection for document writes. | test | -| REQ-20 | Operations | Provide deterministic one-time manual normalization/backfill for existing document type strings in the current small corpus, assigning each document to a canonical registry type before strict write enforcement. | test | +| REQ-0 | System | Provide end-to-end multi-page document transcription with persistent, inspectable async job states. | demonstration | +| REQ-1 | Functional | Allow users to upload one or more images as ordered `Source` pages under a `Document`. | test | +| REQ-2 | Functional | Process page transcription asynchronously using an `asyncio` worker pool bounded by rate limits. | test | +| REQ-3 | Functional | Persist submission-time prompt configuration and full page-level provider response evidence for every job execution. | test | +| REQ-4 | Functional | Support job states `queued`, `processing`, `completed`, `partial_success`, and `failed`, plus page states `pending`, `transcribed`, and `failed`. | inspection | +| REQ-5 | Functional | Allow users to manage historical `Person` records and link multiple people per role to a `Document`. | test | +| REQ-6 | Functional | Support an extensible role taxonomy for document-person relationships. | inspection | +| REQ-7 | Policy Constraint | Enforce a role exclusivity matrix for a single `(document_id, person_id)` pair; initial rules must block `author` + `recipient` coexistence while allowing `mentioned` to coexist with other roles. | test | +| REQ-8 | Functional | Use set-based synchronization for document-person mutations so updates add and remove only the intended links. | test | +| REQ-9 | Functional | Maintain immutable machine output on `Source.raw_transcription` while permitting inline human edits on `Source.revised_text`. | test | +| REQ-10 | Functional | Support a registry-driven `DocumentType` taxonomy with stable codes, mutable labels, and active/inactive lifecycle control. | test | +| REQ-11 | Data Constraint | Store `Document` type as a controlled reference to `DocumentType`. | test | +| REQ-12 | Interface | Render multi-page transcriptions sequentially by `page_number` with document, people, and document-type metadata. | demonstration | +| REQ-13 | Interface | Document create/edit UI must support selecting multiple people per role and selecting an active document type from the registry. | demonstration | +| REQ-14 | API Constraint | Expose additive, role-aware retrieval and write behavior for document-person links and code-based selection for document types. | test | +| REQ-15 | Data Constraint | Calculate and store cryptographic file hashes (SHA-256) and file sizes for uploaded source images. | test | +| REQ-16 | Data Constraint | Preserve a portable relational model across supported backends using SQLModel, SQLAlchemy, SQLite, and PostgreSQL. | inspection | +| REQ-17 | Reliability | Ensure delete and update flows for documents, people, and relationship links remain deterministic and safe. | test | +| REQ-18 | Operations Constraint | Keep canonical development, testing, restore, and recovery workflows OS-independent. | inspection | +| REQ-19 | Quality | Provide automated coverage for async transcription workflows, relationship-role enforcement, document-type selection, and regression behavior. | test | ## Clarifying Constraints -1. Suggestions are advisory only until accepted by a human operator. -2. Relationship acceptance/rejection must be explicit and auditable. -3. Role-policy enforcement must occur consistently across service and API boundaries. -4. Many-per-role behavior is required for both asserted and suggested states where applicable. -5. Suggestions that would violate asserted exclusivity may exist as `pending`, but acceptance must fail until the exclusivity conflict is resolved. -6. Document type codes are stable identifiers; display labels may evolve without changing canonical type identity. - -## Assumptions - -1. V4 scope is limited to document-person relationship expansion and does not redesign the core transcription job execution model. -2. Existing V3 data remains the starting corpus and is transformed via deterministic backfill/validation rules. -3. V4 uses a role registry model and separate suggestion storage; enum-first role expansion is out of scope for this revision. -4. V4 introduces document type registry governance and deprecates unconstrained free-text typing as an authoring-time default. +1. `DocumentType.code` and `PersonRole.code` are stable machine identifiers. +2. `DocumentType.label` and `PersonRole.label` may evolve without changing canonical identity. +3. Role-policy enforcement must be consistent across UI, API, services, and persistence. +4. Many-per-role behavior is required for document-person links. +5. Relationship conflicts must fail deterministically without partial mutation. ## Element Satisfaction Mapping -- **UI (NiceGUI):** Satisfies REQ-0, REQ-1, REQ-3, REQ-4, REQ-8, REQ-9, REQ-18. -- **API (FastAPI):** Satisfies REQ-0, REQ-3, REQ-4, REQ-5, REQ-7, REQ-10, REQ-19. -- **PERSISTENCE (SQLModel/SQLAlchemy):** Satisfies REQ-1, REQ-2, REQ-5, REQ-6, REQ-11, REQ-12, REQ-13, REQ-14, REQ-16, REQ-17, REQ-20. -- **SERVICES (Domain Layer):** Satisfies REQ-4, REQ-5, REQ-7, REQ-10, REQ-14, REQ-17, REQ-20. -- **TEST SUITE:** Satisfies REQ-15 and verifies all test-marked requirements. - -## Change Classification vs V3 - -- **Semantic expansion:** role model and assertion state model. -- **Policy expansion:** exclusivity matrix enforcement. -- **Workflow expansion:** suggestion review and promotion lifecycle. -- **Contract expansion:** role/state-aware read and write behavior. -- **Governance expansion:** registry-managed document type taxonomy and normalization workflow. +- UI (NiceGUI): Satisfies REQ-0, REQ-1, REQ-5, REQ-9, REQ-12, REQ-13. +- API (FastAPI): Satisfies REQ-1, REQ-4, REQ-5, REQ-7, REQ-8, REQ-14. +- Worker (`asyncio`): Satisfies REQ-2, REQ-3, REQ-4. +- Persistence (SQLModel / SQLAlchemy): Satisfies REQ-3, REQ-9, REQ-10, REQ-11, REQ-15, REQ-16, REQ-17. +- Test Suite: Verifies all test-marked requirements and satisfies REQ-19. ## Related Local References -- [V4 Scope Boundary](scope_boundary_v4.md) -- [System Overview V3](../index_v3.md) -- [System Requirements V3](../requirements_v3.md) -- [Data Model V3](../schema_v3.md) +- [System Overview](index_v4.md) +- [System Architecture](architecture_v4.md) +- [Data Model](schema_v4.md) +- [Error Handling Policy](error_handling_v4.md) diff --git a/docs/ver4/schema_v4.md b/docs/ver4/schema_v4.md index e86a4da..3beac83 100644 --- a/docs/ver4/schema_v4.md +++ b/docs/ver4/schema_v4.md @@ -1,214 +1,192 @@ # Database Schema (Version 4) -This document defines the selected schema direction for V4 document-person relationship expansion. +This document defines the relational schema for the document transcription system. -V4 goals are: +## Entity Relationship Diagram -- extensible role taxonomy, -- extensible document type taxonomy, -- explicit assertion state (`asserted`, `suggested`), -- policy-driven exclusivity, -- deterministic migration from V3 links. +```mermaid +erDiagram +DOCUMENT_TYPE { +UUID id PK +TEXT code +TEXT label +BOOLEAN is_active +INTEGER sort_order +TIMESTAMPTZ created_at +TIMESTAMPTZ updated_at +} -## Scope +PERSON_ROLE { +UUID id PK +TEXT code +TEXT label +BOOLEAN is_active +TIMESTAMPTZ created_at +TIMESTAMPTZ updated_at +} -This specification focuses on relationship and document-type governance persistence changes. Existing `Person`, `Source`, `Job`, and `JobSource` core structures remain as in V3 unless explicitly noted. +ROLE_EXCLUSIVITY { +UUID id PK +UUID left_role_id FK +UUID right_role_id FK +TIMESTAMPTZ created_at +} -## New/Expanded Concepts +PERSON { +UUID id PK +TEXT full_name +TEXT display_name +TEXT maiden_name +DATE birth_date +TEXT birth_date_raw +TEXT birth_place +DATE death_date +TEXT death_date_raw +TEXT death_place +TEXT biography +TEXT portrait_path +JSONB metadata +TIMESTAMPTZ created_at +TIMESTAMPTZ updated_at +} -- **Relationship role:** semantic label such as `author`, `recipient`, `mentioned`. -- **Assertion state:** whether the link is canonical (`asserted`) or pending review (`suggested`). -- **Exclusivity matrix:** configurable role-pair conflicts for same `(document_id, person_id)`. -- **Suggestion provenance:** evidence fields enabling review decisions. -- **Document type registry:** controlled taxonomy for `Document` classification with stable code identity. +DOCUMENT { +UUID id PK +UUID document_type_id FK +TEXT name +DATE document_date +TEXT document_date_raw +TEXT location_created +TEXT notes +TEXT archive_identifier +TIMESTAMPTZ created_at +TIMESTAMPTZ updated_at +} -## Selected Model: Role Registry + Separate Suggestion Table +DOCUMENT_PERSON { +UUID id PK +UUID document_id FK +UUID person_id FK +UUID role_id FK +TIMESTAMPTZ created_at +TIMESTAMPTZ updated_at +} -This option cleanly separates canonical links from pending suggestions and enables fully data-driven role expansion. +JOB { +UUID id PK +UUID document_id FK +VARCHAR status +INTEGER retry_count +TEXT provider +TEXT model +TEXT prompt_name +TEXT prompt_hash +TEXT system_prompt +TEXT user_prompt +FLOAT temperature +FLOAT top_p +TIMESTAMPTZ date_created +TIMESTAMPTZ date_updated +} -### Tables +SOURCE { +UUID id PK +UUID document_id FK +INTEGER page_number +TEXT upload_name +TEXT filename +TEXT file_path +TEXT file_hash +BIGINT file_size_bytes +TEXT raw_transcription +TEXT revised_text +TIMESTAMPTZ date_uploaded +TIMESTAMPTZ date_revised +} -#### `person_role` +JOB_SOURCE { +UUID id PK +UUID job_id FK +UUID source_id FK +VARCHAR status +TEXT raw_transcription +JSONB ai_metadata +JSONB raw_api_response +TEXT error_detail +TIMESTAMPTZ executed_at +} -| Column | Type | Notes | -| --- | --- | --- | -| `id` | UUID PK | Stable key | -| `code` | TEXT UNIQUE | Canonical role code, for example `author`, `recipient`, `mentioned` | -| `label` | TEXT | UI label | -| `is_active` | BOOLEAN | Soft-enable/disable role | -| `created_at` | TIMESTAMPTZ | Audit timestamp | -| `updated_at` | TIMESTAMPTZ | Audit timestamp | +DOCUMENT_TYPE ||--o{ DOCUMENT : classifies +DOCUMENT ||--o{ DOCUMENT_PERSON : has_people +PERSON ||--o{ DOCUMENT_PERSON : appears_in +PERSON_ROLE ||--o{ DOCUMENT_PERSON : labels +PERSON_ROLE ||--o{ ROLE_EXCLUSIVITY : left_rule +PERSON_ROLE ||--o{ ROLE_EXCLUSIVITY : right_rule +DOCUMENT ||--o{ JOB : has_jobs +DOCUMENT ||--o{ SOURCE : contains_pages +JOB ||--o{ JOB_SOURCE : executes +SOURCE ||--o{ JOB_SOURCE : processed_in +``` -#### `document_person` (asserted links only) +## Domain Invariants and Provenance Rules -| Column | Type | Notes | -| --- | --- | --- | -| `id` | UUID PK | Stable key | -| `document_id` | UUID FK | -> `document.id` | -| `person_id` | UUID FK | -> `person.id` | -| `role_id` | UUID FK | -> `person_role.id` | -| `created_at` | TIMESTAMPTZ | Audit timestamp | -| `updated_at` | TIMESTAMPTZ | Audit timestamp | +### Page-Level Execution and AI Outputs -Constraints: +- Every single page execution by an AI model produces a dedicated `JOB_SOURCE` record. +- Every `JOB` stores the frozen prompt identifier, prompt text, and hyperparameters used at submission time. +- Every `JOB_SOURCE` stores the complete provider response envelope and page-level operational metadata. +- `SOURCE.raw_transcription` caches the latest successful machine output for that page. -- `UNIQUE(document_id, person_id, role_id)` +### Image Storage and Integrity -#### `document_person_suggestion` +- Binary images are stored on disk; `SOURCE.file_path` stores the persisted path. +- `SOURCE.file_hash` stores a SHA-256 digest. +- `SOURCE.file_size_bytes` stores the original file size. -| Column | Type | Notes | -| --- | --- | --- | -| `id` | UUID PK | Stable key | -| `document_id` | UUID FK | -> `document.id` | -| `person_id` | UUID FK | -> `person.id` | -| `role_id` | UUID FK | -> `person_role.id` | -| `status` | TEXT | `pending`, `accepted`, `rejected` | -| `confidence` | FLOAT NULL | Optional score | -| `source_mechanism` | TEXT NULL | For example `rule`, `llm` | -| `evidence_ref` | TEXT NULL | Pointer or excerpt ID | -| `evidence_span` | JSON NULL | Optional text span payload | -| `note` | TEXT NULL | Reviewer note | -| `created_at` | TIMESTAMPTZ | Suggestion creation time | -| `reviewed_at` | TIMESTAMPTZ NULL | Decision time | -| `reviewed_by` | TEXT NULL | Operator identifier | +### Page Ordering and Revisions -Constraints: +- `SOURCE.page_number` dictates page ordering within a document. +- `SOURCE.raw_transcription` remains immutable machine output. +- `SOURCE.revised_text` stores human edits and is the preferred display value when present. -- `UNIQUE(document_id, person_id, role_id, status)` with policy for multiple pending rows defined in service layer. -- Optional stricter rule: one active pending suggestion per `(document_id, person_id, role_id)`. +### Document-Person Role Governance -#### `role_exclusivity` +- Documents support zero, one, or many people per relationship role. +- Relationship roles are defined by `PERSON_ROLE` rather than hardcoded columns. +- `DOCUMENT_PERSON` must be unique for `(document_id, person_id, role_id)`. +- Configured exclusive role pairs from `ROLE_EXCLUSIVITY` cannot coexist for the same `(document_id, person_id)`. +- Initial exclusivity seed blocks `author` and `recipient` for the same person-document pair. -| Column | Type | Notes | -| --- | --- | --- | -| `id` | UUID PK | Stable key | -| `left_role_id` | UUID FK | -> `person_role.id` | -| `right_role_id` | UUID FK | -> `person_role.id` | -| `created_at` | TIMESTAMPTZ | Audit timestamp | +### Document Type Governance -Constraints: +- Every document type is defined by `DOCUMENT_TYPE`. +- `DOCUMENT_TYPE.code` is a stable machine identifier. +- `DOCUMENT_TYPE.label` is mutable display text. +- Inactive types remain valid for historical rows but should be excluded from default selection UIs. -- Canonical ordering rule to avoid duplicate pairs (`left_role_id < right_role_id` enforced in service/DB). -- `UNIQUE(left_role_id, right_role_id)` +## Constraint Summary -Initial seed: - -- Exclusivity pair: (`author`, `recipient`) - -## Deferred Alternative (Not Selected for V4) - -An enum-based shared table model was considered but is intentionally not selected for V4 because it couples canonical and provisional states in one table and increases invariant complexity. - -## Assertion-State Semantics - -- `asserted`: canonical relationship used for document/person metadata and business logic. -- `suggested`: non-canonical proposal requiring explicit review. -- Accept action: - - creates asserted link (or confirms existing), - - marks suggestion `accepted`. -- Reject action: - - marks suggestion `rejected`. - -## Exclusivity Enforcement - -Policy target: - -- For a single `(document_id, person_id)`, disallow coexistence of role pairs configured as exclusive. - -Enforcement layers: - -1. Service-level pre-check for clear API errors. -2. Database-level guard where feasible (constraints/triggers or deterministic write path). - -Initial configured rule: - -- `author` and `recipient` are exclusive. - -Enforcement semantics: - -- Exclusivity is enforced for asserted links. -- Pending suggestions may exist even if they would conflict when asserted. -- Accepting a suggestion must run exclusivity checks and fail deterministically on conflict. - -## Migration From V3 - -### Data Mapping - -- Existing V3 `document_person` rows map to V4 `asserted` semantics. -- Existing V3 role values: - - `author` -> role `author` - - `recipient` -> role `recipient` - -### Backfill Steps - -1. Seed role rows (`author`, `recipient`, `mentioned`) if using Option A. -2. Migrate current links into asserted table/state. -3. Run conflict scan for exclusivity violations. -4. Apply deterministic conflict policy for any violations. -5. Enable hard enforcement after data passes validation. +- `DOCUMENT_TYPE.code` is unique. +- `PERSON_ROLE.code` is unique. +- `DOCUMENT_PERSON(document_id, person_id, role_id)` is unique. +- `ROLE_EXCLUSIVITY(left_role_id, right_role_id)` is unique. +- `ROLE_EXCLUSIVITY` must use canonical ordering to avoid duplicate mirrored pairs. ## Indexing Guidance -Recommended indexes: - +- `document(document_type_id)` - `document_person(document_id)` - `document_person(person_id)` -- `document_person(role_id)` (Option A) or `document_person(role)` (Option B) -- `document_person_suggestion(document_id, status)` (Option A) -- `document_person_suggestion(person_id, status)` (Option A) - -## Selection Rationale - -V4 selects role registry plus separate suggestion storage for clearer provenance boundaries, cleaner lifecycle transitions, and long-term extensibility. - -## Document Type Registry Model (Selected for V4) - -V4 applies the same registry governance pattern to document classification. - -### Tables - -#### `document_type` - -| Column | Type | Notes | -| --- | --- | --- | -| `id` | UUID PK | Stable key | -| `code` | TEXT UNIQUE | Canonical type code, for example `letter`, `diary`, `book`, `postcard` | -| `label` | TEXT | UI display label | -| `is_active` | BOOLEAN | Soft-enable/disable type | -| `sort_order` | INTEGER NULL | Optional UI ordering | -| `created_at` | TIMESTAMPTZ | Audit timestamp | -| `updated_at` | TIMESTAMPTZ | Audit timestamp | - -#### `document` update - -| Column | Type | Notes | -| --- | --- | --- | -| `document_type_id` | UUID FK NULL | -> `document_type.id` | - -### Constraints and Governance - -- `document_type.code` must be stable and unique. -- `document_type.label` may change without changing canonical type identity. -- Inactive types remain valid for historical records but are excluded from default create/edit selectors. - -### Initial Seeds - -- Seed baseline type codes from current V3 usage set (for example `letter`, `diary`, `book`, `postcard`, `record`, `memo`) and refine labels as needed. - -### Migration and Normalization (Small Corpus) - -1. Seed canonical `document_type` rows. -2. Manually assign each existing document (10 total) to a canonical type via `document_type_id`. -3. Resolve any outlier values directly during this one-time pass. -4. Enforce registry-backed write validation after manual assignment is complete. - -### Indexing Guidance (Document Type) - -- `document_type(code)` unique index. -- `document(document_type_id)` index. +- `document_person(role_id)` +- `role_exclusivity(left_role_id, right_role_id)` +- `source(document_id, page_number)` +- `job(document_id, status)` +- `job_source(job_id)` +- `job_source(source_id)` ## Related Local References -- [V4 Scope Boundary](scope_boundary_v4.md) -- [V4 Requirements](requirements_v4.md) -- [V3 Schema](../schema_v3.md) +- [System Overview](index_v4.md) +- [System Architecture](architecture_v4.md) +- [System Requirements](requirements_v4.md) +- [Error Handling Policy](error_handling_v4.md) diff --git a/docs/ver4/scope_boundary_v4.md b/docs/ver4/scope_boundary_v4.md index 1661b30..d4d6944 100644 --- a/docs/ver4/scope_boundary_v4.md +++ b/docs/ver4/scope_boundary_v4.md @@ -1,130 +1,91 @@ -# V4 Scope Boundary (Version 4) +# V4 Scope Boundary -This document defines what is and is not included in Version 4 for expanding `Document`-`Person` relationships. +This document defines the scope for the transition from the current repository state to the Version 4 project definition. ## Purpose -Create a clear implementation boundary before updating full V4 architecture, requirements, schema, and plan documents. +Define what this revision includes, what it intentionally excludes, and what migration rules govern the transition work. -## Why This Is V4 (Not V3.1) +## In Scope -V4 is required because the change is semantic and cross-cutting: +### 1. Relationship Model -- Expands relationship meaning beyond fixed `author`/`recipient`. -- Introduces a suggestion lifecycle (`suggested` vs `asserted`). -- Introduces policy constraints (role exclusivity matrix). -- Impacts persistence, service contracts, API behavior, UI workflows, and test strategy. +- Extensible role taxonomy for document-person relationships. +- Many-to-many document-person links with many people per role. +- Deterministic exclusivity policy for configured role pairs. +- Set-based add/remove synchronization for document-person updates. -A V3.1 patch would only be appropriate for non-semantic quality-of-life fixes inside existing role semantics. +### 2. Document Type Governance -## In Scope for V4 +- Registry-driven `DocumentType` model with stable codes and controlled selection. +- Minimal rollout for the current corpus with no alias helper table. -### 1) Relationship Semantics +### 3. UI and API Behavior -- Role taxonomy becomes extensible (not hardcoded to only two role values). -- A `Document` can link to many `Person` records per role. -- Relationship states are explicit: - - `asserted`: human-confirmed canonical link. - - `suggested`: machine- or heuristic-proposed link pending review. +- Grouped role links on document and person views. +- Multi-role relationship editing on document create/edit flows. +- Role-aware API retrieval and write behavior. +- Additive API evolution with explicit deprecations. -### 2) Policy Rules +### 4. Verification -- Enforce role exclusivity for the same `(document, person)` pair where configured. -- Initial policy decision: - - `author` + `recipient` are mutually exclusive. - - `mentioned` may coexist with other roles. +- Tests for many-per-role behavior. +- Tests for exclusivity enforcement. +- Tests for set-based relationship mutation behavior. +- Tests for document and person delete/link cleanup regressions. -### 3) Persistence and Contracts +## Out of Scope -- Evolve persistence model to support: - - extensible roles, - - suggestion lifecycle metadata, - - deterministic conflict handling. -- Replace single-value relationship mutation patterns with set-based sync behavior. - -### 4) Human-in-the-Loop Workflow - -- Add assisted suggestion review flow: - - list suggestions, - - accept, - - reject, - - promote accepted suggestion to asserted link. -- No silent auto-promotion from suggestion to asserted. - -### 5) UI and API Behavior - -- Document create/edit/detail workflows support multi-person per role. -- UI surfaces grouped role links and suggestion status. -- API supports role-aware retrieval and suggestion lifecycle operations. - -### 6) Verification - -- Add tests for: - - many-per-role behavior, - - exclusivity enforcement, - - suggestion lifecycle, - - migration/conflict detection, - - regression on delete/link cleanup behavior. - -## Out of Scope for V4 - -- Automatic acceptance of suggested links without human review. -- Full entity resolution/identity merge pipeline across all `Person` records. -- Historical provenance graph redesign beyond relationship-level evidence fields. -- Large-scale NLP research features unrelated to document-person linking. -- Changes to core transcription execution model (`Job`, `JobSource`) except where needed to expose suggestion evidence inputs. +- Suggested versus asserted relationship states. +- Suggestion storage, review, acceptance, or rejection workflows. +- Automatic relationship extraction or recommendation features. +- Full entity resolution or identity merge across all people. +- Automated semantic document type classification. +- Redesign of the core transcription execution model. ## Locked Design Decisions -### A) Role Extensibility Mechanism +### A. Role Extensibility Mechanism -- Adopt role registry tables (data-driven roles). -- Do not use enum-first role expansion for V4. +- Use registry tables for relationship roles. -### B) Suggestion Storage Model +### B. Exclusivity Matrix Baseline -- Adopt a separate `document_person_suggestion` table. -- Keep `document_person` focused on canonical asserted links. -- Revisit only if operational complexity proves materially higher than expected. - -### C) Exclusivity Matrix Baseline - -- `author` + `recipient` are mutually exclusive for the same `(document, person)` pair. +- `author` and `recipient` are mutually exclusive for the same `(document, person)` pair. - `mentioned` remains non-exclusive. -- Hard exclusivity enforcement applies to asserted links. -- Suggestions may be stored even if they would conflict at assert time; promotion to asserted must enforce exclusivity. -### D) API Compatibility Strategy +### C. API Compatibility Strategy -- Use additive API evolution in V4. -- In development mode, the current revision is authoritative; long-lived legacy compatibility layers are not required. -- Deprecations should be explicit and short-lived, with removals performed in subsequent revisions. +- Use additive API evolution. +- In development mode, the current revision is authoritative. +- Deprecations should be explicit and short-lived. -### E) Document Type Rollout Strategy +### D. Document Type Rollout Strategy -- Use a minimal registry rollout for the current corpus: no `document_type_alias` helper table. -- Perform a one-time manual mapping of existing document types to canonical registry types. +- Use a minimal registry rollout for the current corpus. +- Do not introduce a `document_type_alias` helper table. -### F) Suggestion Generation Strategy (Still Open) +### E. Database Change Policy -- Option A: deterministic rules/NLP over transcribed text. -- Option B: LLM extraction with confidence/evidence spans. +- Future schema changes are non-destructive by default. +- Exception: `document_type` text may be replaced by `document_type_id` without migrating the legacy text values. +- Exception: `document_person` links may be recreated manually. ## Compatibility and Rollout -- Existing V3 author links migrate to `asserted` behavior. -- Backfill validation identifies policy conflicts before constraints are enforced. -- Preserve existing V3 core behavior where unaffected by role/suggestion evolution. +- Preserve existing repository behavior where unaffected by the V4 scope. +- Treat scope boundary and implementation plan as the only transition documents. +- Treat core V4 documents as the authoritative project definition once rewritten. ## Exit Criteria for Scope Freeze V4 scope is considered frozen when: -- Suggestion generation strategy is chosen. -- Suggestion acceptance workflow details are approved. +- Relationship model, exclusivity rules, and document-type governance are approved. - Additive API change list and deprecation schedule are approved. +- Migration exceptions are explicitly acknowledged. -## Core V4 Documents (Current Set) +## Core V4 Documents 1. `docs/ver4/index_v4.md` 2. `docs/ver4/requirements_v4.md`