bbchops/transcription
1
0
Fork 0
generated from john/python-template
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

Revised and simplified V4 Plan and core documents.

Browse Source
This commit is contained in:
Jim Lancaster
2026-08-10 10:53:13 -05:00
parent 9b4d6f0340
commit 4b3baf5a3e
7 changed files with 495 additions and 578 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/ver4/architecture_v4.md
+105 -95
View File
@@ -1,137 +1,147 @@
# System Architecture (Version 4) # System Architecture (Version 4)
This document describes the V4 architecture changes for expanded `Document`-`Person` relationship management. This document describes the production architecture of the document transcription system.
V4 extends V3 with role extensibility, assisted suggestion review, and role-policy enforcement while preserving the existing transcription execution topology.
## Architecture Objectives ## Architecture Objectives
- Keep V3 transcription pipeline behavior stable unless relationship evidence extraction requires additive integration. - Preserve original source material and immutable machine transcription output.
- Support many people per role for each document. - Support batching one or more images into ordered multi-page documents.
- Support extensible relationship roles without hardcoding UI and API behavior to two values. - Capture complete submission-time prompt provenance and per-page provider response evidence.
- Support non-canonical suggestion intake with explicit human promotion to canonical asserted links. - Execute page transcription concurrently with bounded `asyncio` workers.
- Enforce role exclusivity policy consistently at service and persistence boundaries. - Maintain relational portability across SQLite and PostgreSQL.
- Preserve auditability for suggestion provenance and review actions. - Keep operator workflows cross-platform and Python-driven.
- Support many-to-many document-person relationships with extensible roles.
- Support registry-driven document type classification.
- Enforce relationship-role exclusivity rules consistently across UI, API, and persistence boundaries.
## Runtime Topology ## Runtime Topology
V4 keeps the existing runtime shape: The runtime operates as an asynchronous Python application:
- FastAPI + NiceGUI web app process. - FastAPI + NiceGUI web application process.
- Async service layer with SQLModel/SQLAlchemy persistence. - In-process `asyncio` worker engine for transcription execution.
- Existing worker execution path for transcription jobs. - Relational persistence via SQLModel / SQLAlchemy.
- Pydantic V2 validation across API payloads, prompt configuration, and structured metadata.
Additive V4 components:
- Relationship policy evaluator (exclusivity checks).
- Suggestion lifecycle service (propose/list/accept/reject).
- Optional suggestion extraction adapter (rule/NLP or LLM-backed).
^^^mermaid ^^^mermaid
flowchart LR flowchart LR
U[Browser User] --> UI[NiceGUI Pages] U[Browser User] --> A[FastAPI + NiceGUI App]
UI --> API[FastAPI Routes] A --> W[Asyncio Worker Engine]
API --> DS[Document Service] A --> DB[(Relational DB)]
API --> SS[Suggestion Service] W --> P[Vision Provider APIs]
DS --> PE[Policy Evaluator] W --> DB
SS --> PE
DS --> DB[(Relational DB)]
SS --> DB
W[Async Worker Engine] --> DB
W --> E[Optional Suggestion Extraction]
E --> SS
^^^ ^^^
## Layer Responsibilities ## Lifecycle Ownership
### UI Layer (`src/transcription/ui/**`) Application lifespan owns runtime setup and teardown:
- Render per-role grouped relationships for document and person views. - Initialize logging, settings, directories, and prompt configuration.
- Provide multi-select role assignment controls in document create/edit flows. - Manage asynchronous database engine connection pools.
- Provide suggestion review controls (accept/reject) for pending suggestions. - Execute database bootstrap or migrations.
- Surface policy conflict errors from API/service layer clearly. - Recover stale or interrupted jobs on startup.
- Manage graceful shutdown of active background tasks.
### API Layer (`src/transcription/api/**`) ## Layered Module Structure
- Expose role-aware and state-aware read contracts. ### Interface Layer
- Expose suggestion lifecycle write contracts.
- Return deterministic validation/conflict errors for exclusivity and duplicate semantics.
- Evolve endpoints additively, with explicit deprecations and short-lived transition windows.
### Service Layer (`src/transcription/services/**`) - `src/transcription/ui/**`
- `src/transcription/api/**`
- Implement set-based relationship sync (delta add/remove) to avoid destructive replacement behavior. Responsibilities:
- Apply policy evaluator before persistence commits.
- Orchestrate suggestion acceptance/rejection transitions.
- Ensure accepted suggestions result in asserted link creation/confirmation.
### Policy Evaluator (New logical component) - Render document, source, person, job, and classification views.
- Accept user input for uploads, editing, linking, and revisions.
- Present structured validation and conflict feedback.
- Evaluate role exclusivity matrix for `(document_id, person_id)` writes. ### Application and Async Worker Layer
- Provide deterministic conflict reason payloads suitable for UI/API display.
- Stay stateless and reusable across create/update/sync code paths.
### Persistence Layer (`src/transcription/db/**`) - `src/transcription/services/workflows.py`
- `src/transcription/worker.py`
- Store asserted links and suggestion records according to selected schema option. Responsibilities:
- Persist provenance metadata for suggestion records.
- Enforce uniqueness and support performant role/state filtering.
## Core V4 Workflows - Orchestrate uploads, job creation, and status transitions.
- Execute per-page provider calls through bounded concurrency.
- Persist page-level outcomes and update aggregate job state.
### 1) Manual Relationship Management ### Domain and Service Layer
1. User opens document edit view. - `src/transcription/db/models.py`
2. UI loads asserted links grouped by role. - `src/transcription/services/*.py`
3. User adds/removes people per role.
4. Service computes delta and runs policy checks.
5. Persistence applies adds/removes atomically.
### 2) Assisted Suggestion Review Responsibilities:
1. Suggestion records are created (`pending`) by extraction logic or manual propose action. - Manage transactional operations for documents, people, types, links, sources, jobs, and job sources.
2. UI displays pending suggestions with evidence metadata. - Apply deterministic conflict handling for relationship-role writes.
3. User accepts or rejects each suggestion. - Use set-based synchronization for many-to-many relationship updates.
4. Accept path creates/confirms asserted relationship and marks suggestion `accepted`. - Resolve and validate registry-backed document types.
5. Reject path marks suggestion `rejected`.
### 3) Exclusivity Conflict Handling ### Infrastructure Layer
1. Incoming write attempts role assignment. - `src/transcription/db/**`
2. Policy evaluator checks role pair conflicts for same `(document_id, person_id)`. - `src/transcription/providers/**`
3. If conflict exists, write is rejected with structured conflict details.
4. UI presents actionable message without partial updates.
## Invariants Responsibilities:
- Suggested links are never canonical until accepted. - Provide async database sessions and engine configuration.
- Asserted links must satisfy exclusivity rules. - Provide provider adapters for vision model execution.
- Relationship mutations are set-based and deterministic.
- Relationship views remain consistent between document detail and person detail pages.
- Existing V3 links remain valid under asserted semantics after migration.
## Backward Compatibility ## Core Workflows
- V3 author/recipient links are interpreted as asserted links in V4. ### 1. Multi-Page Transcription
- Existing document and person flows continue to function where unaffected by new role/state dimensions.
- During development revisions, the current API contract is authoritative; long-lived legacy runtime compatibility layers are not required.
## Observability and Auditability 1. User uploads one or more images for a `Document`.
2. System stores files, hashes them, creates ordered `Source` rows, and creates a `Job`.
3. Worker claims the job, marks it `processing`, and executes page calls concurrently.
4. Each page writes a `JobSource` result with raw output, metadata, and full provider response evidence.
5. Aggregate status becomes `completed`, `partial_success`, or `failed`.
- Persist suggestion provenance fields sufficient for operator review. ### 2. Document-Person Relationship Management
- Record review decision outcomes (`accepted`/`rejected`) with timestamps and reviewer identity where available.
- Emit structured service-level logs for exclusivity conflicts and review actions.
## Test Strategy Additions 1. User opens a document or person edit flow.
2. UI loads existing links grouped by role.
3. User adds or removes people within one or more roles.
4. Service computes add/remove deltas rather than replacing all links blindly.
5. Conflict checks enforce exclusivity policy before persistence commits.
- Service tests for delta sync and exclusivity matrix enforcement. ### 3. Document Type Management
- API tests for role/state filtering and suggestion transitions.
- UI tests for multi-role selection and suggestion review interactions. 1. User selects a registry-backed document type for a document.
- Migration tests for V3-to-V4 asserted mapping and conflict scans. 2. Service resolves the stable type code or id.
3. Persistence stores the `document_type_id` reference.
4. Inactive types remain valid for historical rows but are excluded from default selectors.
## Domain Invariants
- `Source.raw_transcription` stores immutable machine output.
- Human corrections occur only in `Source.revised_text`.
- Prompt and parameter provenance is frozen on `Job` at submission time.
- Provider output evidence is stored on `JobSource` for each page execution.
- `DocumentPerson` links are unique for `(document_id, person_id, role_id)`.
- Configured exclusive role pairs cannot coexist for the same `(document_id, person_id)`.
- Relationship mutations are deterministic and set-based.
- `DocumentType.code` is stable; `DocumentType.label` may evolve.
## Data Model Summary
- `Document` has one `DocumentType`, many `Source` pages, many `Job` runs, and many `Person` records through `DocumentPerson`.
- `Source` belongs to one `Document` and may participate in many `JobSource` executions.
- `Job` has many `JobSource` rows.
- `PersonRole` defines available relationship roles.
- `RoleExclusivity` defines role pairs that cannot coexist for the same document-person pair.
## Test Strategy
- Unit tests for models, validation, hashing, and registry resolution.
- Service tests for CRUD, set-based sync, and exclusivity enforcement.
- Async workflow tests for page isolation, partial failure handling, and stored evidence.
- UI integration tests for multi-page rendering, role grouping, and document type selection.
## Related Local References ## Related Local References
- [V4 Scope Boundary](scope_boundary_v4.md) - [System Overview](index_v4.md)
- [V4 Requirements](requirements_v4.md) - [System Requirements](requirements_v4.md)
- [V4 Schema](schema_v4.md) - [Data Model](schema_v4.md)
- [V3 Architecture](../architecture_v3.md) - [Error Handling Policy](error_handling_v4.md)
docs/ver4/error_handling_v4.md
+57 -51
View File
@@ -1,62 +1,68 @@
# Error Handling Policy (Version 4) # Error Handling Policy (Version 4)
This document defines canonical error-handling behavior for V4 document-person relationship expansion. This document defines the canonical error-handling policy for the document transcription system.
V4 keeps V3 transcription error behavior and adds policy/conflict handling for role extensibility, suggestion lifecycle transitions, and exclusivity enforcement.
## Error Handling Objectives ## Error Handling Objectives
- Provide clear, actionable conflict and validation feedback for relationship write operations. - Make failures visible in clear, actionable language at both the document and page levels.
- Prevent partial, silent, or destructive relationship mutations when policy checks fail. - Support isolated failure handling in multi-page jobs so one failing page does not invalidate successful pages.
- Preserve suggestion review auditability with deterministic accept/reject outcomes. - Preserve diagnostic detail for validation failures, provider failures, and policy conflicts.
- Keep consistent API/UI/service error envelopes across relationship workflows. - Ensure consistent error envelope structure across API, UI, service, and worker boundaries.
## Scope and Authority ## Scope and Authority
Governs relationship-related error behavior in: This policy governs error behavior across:
- NiceGUI document/person relationship views, - NiceGUI pages
- FastAPI relationship and suggestion endpoints, - FastAPI routes
- domain services for relationship sync and suggestion review, - Service-layer orchestration
- persistence constraints for role, state, and exclusivity invariants. - `asyncio` worker tasks
- Database interactions
- Provider adapters
## Relationship Error Taxonomy ## Error Taxonomy
| Category | Definition | Retriable | | Category | Definition | Retriable |
| --- | --- | --- | | --- | --- | --- |
| `validation_error` | Payload shape/type invalid, unknown role/state, malformed IDs | no | | `validation_error` | Payload, parameter, or schema validation failure | no |
| `not_found_error` | Target `Document`, `Person`, role, or suggestion record does not exist | no | | `user_input_error` | Unacceptable file, invalid selection, or malformed request from the operator | no |
| `conflict_error` | Write violates uniqueness or exclusivity policy | no | | `not_found_error` | Requested `Document`, `Source`, `Person`, `Job`, role, or type does not exist | no |
| `suggestion_state_error` | Invalid suggestion transition (for example accept after reject) | no | | `conflict_error` | Operation violates uniqueness or exclusivity policy | no |
| `policy_violation_error` | Action blocked by configured role matrix or governance rule | no | | `external_provider_error` | Provider API failure, rate limit, or execution problem | yes |
| `infrastructure_transient_error` | Temporary DB or network instability during relationship operation | yes | | `infrastructure_transient_error` | Temporary DB, file-system, or network instability | yes |
| `infrastructure_persistent_error` | Persistent DB/configuration failure | no | | `infrastructure_persistent_error` | Persistent configuration, credential, or database availability failure | no |
| `internal_unexpected_error` | Unhandled exception/logic defect | no | | `internal_unexpected_error` | Uncaught exception or logic defect | no |
## Deterministic Conflict Behavior ## Async Batch and Page-Level Error Behavior
When relationship writes fail policy checks: In multi-page `asyncio` processing:
1. Reject the full write operation (no partial apply). 1. Exceptions from individual page calls are trapped within the page task wrapper.
2. Return structured conflict details including conflicting role pair and target identifiers. 2. Failed page detail is written to `JobSource.error_detail` and the page state becomes `failed`.
3. Preserve existing canonical relationships unchanged. 3. Aggregate job status is derived from page outcomes:
- all pages succeed -> `completed`
- some succeed and some fail -> `partial_success`
- all fail -> `failed`
4. Successful pages remain valid even when sister pages fail.
When suggestion transitions fail: ## Relationship and Classification Conflict Behavior
1. Reject invalid state transition. When relationship or document-type writes fail policy checks:
2. Return current state and allowed next actions.
3. Preserve suggestion record integrity. 1. Reject the full write operation.
2. Return structured conflict detail including target identifiers and the violated rule.
3. Preserve existing persisted relationships unchanged.
## API Error Response Contract ## API Error Response Contract
Relationship endpoints return a structured envelope: API error responses return a structured envelope:
^^^json ^^^json
{ {
"error_id": "err_uuid_12345", "error_id": "err_uuid_12345",
"category": "conflict_error", "category": "conflict_error",
"message": "Role assignment violates exclusivity policy.", "message": "Role assignment violates exclusivity policy.",
"suggestion": "Remove recipient role before assigning author for this person on this document.", "suggestion": "Remove recipient before assigning author for this person on this document.",
"details": { "details": {
"document_id": "...", "document_id": "...",
"person_id": "...", "person_id": "...",
@@ -64,41 +70,41 @@ Relationship endpoints return a structured envelope:
"conflicting_role": "recipient", "conflicting_role": "recipient",
"policy_rule": "author+recipient exclusive" "policy_rule": "author+recipient exclusive"
}, },
"timestamp": "2026-08-09T15:00:00Z" "timestamp": "2026-08-10T15:00:00Z"
} }
^^^ ^^^
HTTP status mappings: HTTP status mappings:
- `validation_error` -> `400` - `validation_error`, `user_input_error` -> `400`
- `not_found_error` -> `404` - `not_found_error` -> `404`
- `conflict_error`, `suggestion_state_error`, `policy_violation_error` -> `409` - `conflict_error` -> `409`
- `external_provider_error` -> `502` or `503`
- `infrastructure_transient_error` -> `503` - `infrastructure_transient_error` -> `503`
- `infrastructure_persistent_error`, `internal_unexpected_error` -> `500` - `infrastructure_persistent_error`, `internal_unexpected_error` -> `500`
## UI Error Presentation Rules ## UI Error Presentation Rules
- Display concise conflict summary with actionable next step. - Display concise failure summaries with the next action the operator can take.
- Keep user edits in context (do not discard form state when feasible). - Keep form state in context when feasible.
- Differentiate between validation issues, policy conflicts, and infrastructure failures. - Distinguish validation issues, conflict issues, provider failures, and infrastructure failures.
- For bulk role sync operations, show per-item conflict context when multiple failures occur. - For bulk relationship updates, identify the specific role or person that caused a conflict.
## Logging and Audit Expectations ## Logging and Audit Expectations
- Log relationship write failures with correlation IDs. - Log worker failures with correlation IDs and provider context.
- Log suggestion acceptance/rejection outcomes with actor and timestamp where available. - Log relationship and classification conflicts with machine-readable detail.
- Log policy matrix violations with deterministic machine-readable context. - Log persisted provider errors and page-level execution failures.
## Relationship-Specific Retry Guidance ## Retry Guidance
- Do not auto-retry policy or conflict failures. - Do not auto-retry validation or conflict failures.
- Permit user-driven retry only after input changes. - Permit user-driven retry after the input or selection changes.
- Retry infrastructure transient failures with bounded policy in service layer if operation is idempotent. - Allow bounded retry for transient provider or infrastructure failures when the operation is idempotent.
## Related Local References ## Related Local References
- [V4 Scope Boundary](scope_boundary_v4.md) - [System Overview](index_v4.md)
- [V4 Requirements](requirements_v4.md) - [System Requirements](requirements_v4.md)
- [V4 Schema](schema_v4.md) - [Data Model](schema_v4.md)
- [V4 Architecture](architecture_v4.md) - [System Architecture](architecture_v4.md)
- [V3 Error Handling](../error_handling_v3.md)
docs/ver4/implementation_plan_v4.md
+61 -71
View File
@@ -2,106 +2,96 @@
## Goal ## Goal
Implement V4 document-person relationship expansion and document type governance with extensible registries, assisted suggestion lifecycle, and policy-enforced exclusivity while preserving V3 transcription behavior. Implement the Version 4 project definition from the current repository state while preserving existing data by default.
## Migration Policy
- Database changes are non-destructive by default.
- Exception: the legacy `document_type` text field may be replaced by a `document_type_id` reference without migrating existing text values.
- Exception: `document_person` links may be recreated manually.
## Current Project Impact ## Current Project Impact
- `src/transcription/db/models.py` will require relationship schema evolution for role extensibility and suggestion lifecycle support. - `src/transcription/db/models.py` requires full schema alignment with the V4 core documents.
- `src/transcription/db/models.py` will require document type registry entities and document type reference updates. - `src/transcription/services/documents.py` requires set-based document-person sync, exclusivity checks, and document-type resolution.
- `src/transcription/services/documents.py` will require set-based relationship sync and policy checks. - API modules require additive role-aware relationship behavior and document-type selection behavior.
- `src/transcription/services/documents.py` will require registry-based document type lookup, validation, and normalization helpers. - UI pages require grouped role displays, multi-role editing, and registry-backed document-type selection.
- API modules under `src/transcription/api/**` will require role/state-aware contracts and suggestion lifecycle endpoints. - Existing tests require updates for role enforcement, document-type selection, and regression safety.
- API modules under `src/transcription/api/**` will require additive document type catalog and code-based selection contracts.
- UI pages under `src/transcription/ui/pages/**` will require multi-role, multi-person editing, suggestion review controls, and registry-backed type selectors.
- Existing tests under `tests/services`, `tests/api`, and `tests/ui` need expanded coverage for V4 behavior and regression safety.
## Implementation Phases ## Implementation Phases
### 1. Finalize V4 Schema Decisions ### 1. Finalize the Transition Documents
- Role extensibility mechanism: use role registry tables. - Confirm the reset scope.
- Document type extensibility mechanism: use `document_type` registry tables. - Confirm the database exception policy.
- Suggestion storage model: use a separate `document_person_suggestion` table. - Keep core V4 documents as the only authoritative product definition.
- Exclusivity baseline: `author` vs `recipient` exclusive, `mentioned` non-exclusive.
- Enforce exclusivity on asserted links; evaluate conflicts on suggestion acceptance.
### 2. Evolve Persistence Layer ### 2. Align the Persistence Layer
- Implement selected schema model in SQLModel. - Update SQLModel definitions to match the final V4 schema.
- Add constraints and indexes for dedupe, filtering, and policy support. - Add `person_role`, `role_exclusivity`, and `document_type` support.
- Add migration/backfill logic for V3 links to V4 asserted semantics. - Replace legacy document-type storage with `document_type_id`.
- Add conflict scan tooling for historical records violating exclusivity rules. - Apply the accepted manual exception strategy for `document_type` and `document_person` data.
- Perform a one-time manual mapping of existing document type values to registry-backed type references (`document_type_id`) for the current small corpus. - Preserve all other data structures non-destructively.
### 3. Implement Service-Layer Policy and Sync Semantics ### 3. Update Services and Write Semantics
- Add relationship delta sync operations (set-based add/remove). - Implement set-based synchronization for document-person updates.
- Implement centralized policy evaluator for exclusivity checks. - Implement deterministic exclusivity conflict checks.
- Add suggestion lifecycle operations (propose/list/accept/reject). - Remove suggestion-related service behavior.
- Ensure accepted suggestions create/confirm asserted links atomically. - Add document-type resolution and validation by stable code or id.
- Add document type resolution operations by stable `code` and active/inactive state handling.
### 4. Implement API Contract Changes ### 4. Update API Contracts
- Add role-aware and state-aware query parameters/filters. - Keep API evolution additive.
- Add suggestion lifecycle endpoints and response models. - Add role-aware relationship retrieval and write behavior.
- Add deterministic conflict/error payloads aligned to V4 error policy. - Add document-type catalog retrieval and code-based selection for document writes.
- Use additive endpoint evolution for V4; during development mode, deprecate then remove without maintaining long-lived legacy runtime compatibility layers. - Remove suggestion-related API surfaces from the V4 target state.
- Add document type catalog endpoints with active-only filtering and code-based selection for document writes.
### 5. Update UI Workflows ### 5. Update UI Workflows
- Replace single-author controls with grouped multi-role selectors. - Replace single-person link editing with grouped multi-role editing.
- Replace unconstrained document type free-text entry with registry-backed type selection. - Render grouped role links on document and person detail views.
- Add pending suggestion review panel and accept/reject actions. - Replace free-text document type entry with registry-backed selection.
- Update document/person detail cards to group links by role and state. - Preserve clear validation and conflict messaging.
- Preserve edit-state ergonomics on validation/conflict failures.
### 6. Verification and Hardening ### 6. Verification and Hardening
- Add service tests for: - Add or update service tests for many-per-role behavior, exclusivity enforcement, and set-based sync correctness.
- many-per-role behavior, - Add API tests for relationship behavior and document-type selection.
- exclusivity enforcement, - Add UI tests or walkthrough coverage for grouped roles and type selection.
- set-based sync correctness, - Add regression coverage for delete and cleanup semantics.
- suggestion transition validity,
- document type code resolution and inactive-type handling.
- Add API tests for role/state filtering and conflict response shapes.
- Add API tests for document type catalog retrieval and code-based write validation.
- Add UI tests/manual walkthroughs for create/edit/review workflows.
- Add UI tests/manual walkthroughs for registry-backed document type selection and validation messaging.
- Add regression tests for document/person delete cleanup semantics.
## Done When ## Done When
- V4 relationship schema and contracts are implemented and validated. - Core V4 documents and code paths agree on the final project definition.
- V4 document type registry schema and contracts are implemented and validated. - Relationship-role writes are deterministic and non-destructive.
- Suggestions remain non-canonical until explicit acceptance. - Exclusivity rules are enforced consistently.
- Exclusivity policy is enforced deterministically across service/API boundaries. - Document type selection is registry-backed.
- Existing V3 links are migrated to V4 asserted semantics without data loss. - The accepted manual exceptions for `document_type` and `document_person` are completed.
- Existing document type strings are manually normalized to registry references for the current corpus. - The focused test coverage passes.
- Test suite includes V4-specific coverage and passes on supported backends.
## Out of Scope ## Out of Scope
- Automatic acceptance of suggestions. - Suggested/asserted relationship state.
- Global person entity-resolution/merge engine. - Suggestion review or extraction workflows.
- Core transcription execution redesign unrelated to relationship expansion. - Global person entity-resolution engine.
- Automated semantic document type classification. - Automated semantic document-type classification.
## Delivery Order Recommendation ## Delivery Order Recommendation
1. Requirements freeze (`requirements_v4.md`). 1. Freeze scope boundary and implementation plan.
2. Schema decision freeze (`schema_v4.md`). 2. Freeze core V4 documents.
3. Error policy freeze (`error_handling_v4.md`). 3. Align persistence models.
4. Implementation of persistence and service layer. 4. Align services and API behavior.
5. API and UI changes. 5. Align UI behavior.
6. Final integration and regression validation. 6. Run focused verification and regression checks.
## Related Local References ## Related Local References
- [V4 Scope Boundary](scope_boundary_v4.md) - [V4 Scope Boundary](scope_boundary_v4.md)
- [V4 Requirements](requirements_v4.md) - [System Overview](index_v4.md)
- [V4 Schema](schema_v4.md) - [System Requirements](requirements_v4.md)
- [V4 Architecture](architecture_v4.md) - [Data Model](schema_v4.md)
- [V4 Error Handling](error_handling_v4.md) - [System Architecture](architecture_v4.md)
- [V3 Implementation Plan](../implementation_plan_v3.md) - [Error Handling Policy](error_handling_v4.md)
docs/ver4/index_v4.md
+28 -37
View File
@@ -1,49 +1,40 @@
# Document-Person Expansion Overview (Version 4) # Document Transcription System Overview (Version 4)
Version 4 defines the relationship-model evolution track for linking `Document` and `Person` entities with extensible roles, suggestion review, and policy enforcement. This project is a personal-scale application for transcribing, organizing, and preserving historical documents, images, and related people records.
## Start Here ## Start Here
Read [scope_boundary_v4.md](scope_boundary_v4.md) first to confirm scope and non-scope before implementation work begins. Read [architecture_v4.md](architecture_v4.md) first for the technical overview and system design.
## Core V4 Capabilities ## Core Capabilities
- Extensible relationship role taxonomy. - Folder and multi-image ingestion into sequential `Source` pages under a single `Document`.
- Many-people-per-role linking for documents. - Parallel asynchronous AI vision transcription using Python `asyncio` bounded by rate limits.
- Explicit distinction between canonical asserted links and pending suggested links. - Portable relational storage using SQLModel and SQLAlchemy across SQLite and PostgreSQL.
- Human-in-the-loop suggestion accept/reject workflow. - Complete prompt and response provenance for every transcription job and page execution.
- Role exclusivity policy enforcement for configured role pairs. - File-integrity tracking through SHA-256 hashing and stored file sizes.
- Role/state-aware API and UI retrieval/presentation behavior. - Historical `Person` management with many-to-many document links and extensible relationship roles.
- Minimal document type governance rollout for the current corpus, with one-time manual mapping and no alias helper table. - Registry-driven `DocumentType` classification with stable codes and controlled selection.
- Inline human revision of transcribed pages while preserving immutable machine output.
- Partial-failure recovery for multi-page jobs.
- Cross-platform operational workflows driven by Python-based tooling.
## V4 Documentation Index ## Technical Stack
- [Scope Boundary](scope_boundary_v4.md) - Application Web Framework: FastAPI + NiceGUI
- Persistence Engine: SQLModel / SQLAlchemy
- Data Validation and Schemas: Pydantic V2
- Concurrency and Workers: Python `asyncio`
- Vision Providers: OpenAI, Anthropic, and OpenRouter adapters
## Core Documentation Index
- [System Architecture](architecture_v4.md)
- [System Requirements](requirements_v4.md) - [System Requirements](requirements_v4.md)
- [Data Model](schema_v4.md) - [Data Model](schema_v4.md)
- [System Architecture](architecture_v4.md)
- [Error Handling Policy](error_handling_v4.md) - [Error Handling Policy](error_handling_v4.md)
## Transition Documents
- [Scope Boundary](scope_boundary_v4.md)
- [Implementation Plan](implementation_plan_v4.md) - [Implementation Plan](implementation_plan_v4.md)
## Relationship To V3
V3 remains the baseline production architecture and requirements set for transcription pipeline behavior. V4 is an additive evolution track focused on document-person relationship semantics and workflows.
## Decision Status
Locked decisions:
1. Role extensibility uses registry tables.
2. Suggestion storage uses a separate suggestion table.
3. Exclusivity baseline is `author` vs `recipient` exclusive, with `mentioned` non-exclusive.
4. API evolution is additive in development mode with explicit deprecate-then-remove behavior.
Remaining decision:
1. Suggestion generation strategy (deterministic rules/NLP vs LLM extraction).
## Related Local References
- [V3 System Overview](../index_v3.md)
- [V3 Requirements](../requirements_v3.md)
- [V3 Schema](../schema_v3.md)
docs/ver4/requirements_v4.md
+36 -55
View File
@@ -1,70 +1,51 @@
# Relationship and Document Type Governance Requirements (Version 4) # Document Transcription System Requirements (Version 4)
This document defines Version 4 baseline requirements for expanding relationships between `Document` and `Person` and introducing governed document type classification. This document defines the baseline requirements for the document transcription system.
V4 preserves all applicable V3 capabilities and adds role extensibility, assisted suggestion workflows, explicit relationship policy enforcement, and registry-driven `Document` type governance.
## Requirements Model ## Requirements Model
| ID | Category | Requirement | Verify Method | | ID | Category | Requirement | Verify Method |
| --- | --- | --- | --- | | --- | --- | --- | --- |
| REQ-0 | System | Provide end-to-end, reviewable, policy-enforced document-person relationship management supporting asserted and suggested links. | demonstration | | REQ-0 | System | Provide end-to-end multi-page document transcription with persistent, inspectable async job states. | demonstration |
| REQ-1 | Functional | Preserve many-to-many `Document` ↔ `Person` linking, and allow multiple people per role on a single document. | test | | REQ-1 | Functional | Allow users to upload one or more images as ordered `Source` pages under a `Document`. | test |
| REQ-2 | Functional | Support an extensible role taxonomy for document-person relationships beyond fixed `author`/`recipient`. | inspection | | REQ-2 | Functional | Process page transcription asynchronously using an `asyncio` worker pool bounded by rate limits. | test |
| REQ-3 | Functional | Represent relationship assertion state explicitly (`asserted` and `suggested`) and keep machine suggestions non-canonical until human acceptance. | test | | REQ-3 | Functional | Persist submission-time prompt configuration and full page-level provider response evidence for every job execution. | test |
| REQ-4 | Functional | Provide assisted suggestion lifecycle operations: create/list/filter suggestions, accept suggestion, reject suggestion, and promote accepted suggestions to asserted links. | test | | REQ-4 | Functional | Support job states `queued`, `processing`, `completed`, `partial_success`, and `failed`, plus page states `pending`, `transcribed`, and `failed`. | inspection |
| REQ-5 | Policy Constraint | Enforce a role exclusivity matrix for a single `(document_id, person_id)` pair on asserted links; initial rule set must block `author` + `recipient` coexistence while allowing `mentioned` to coexist with other roles. | test | | REQ-5 | Functional | Allow users to manage historical `Person` records and link multiple people per role to a `Document`. | test |
| REQ-6 | Data Constraint | Store canonical asserted links in `document_person` and lifecycle-managed suggestions in a separate `document_person_suggestion` table. | test | | REQ-6 | Functional | Support an extensible role taxonomy for document-person relationships. | inspection |
| REQ-7 | Functional | Provide set-based synchronization behavior for relationship mutations (add/remove delta), replacing single-value replacement patterns that can drop unrelated links. | test | | REQ-7 | Policy Constraint | Enforce a role exclusivity matrix for a single `(document_id, person_id)` pair; initial rules must block `author` + `recipient` coexistence while allowing `mentioned` to coexist with other roles. | test |
| REQ-8 | Interface | Render grouped relationship metadata by role and assertion state on document detail and person detail views. | demonstration | | REQ-8 | Functional | Use set-based synchronization for document-person mutations so updates add and remove only the intended links. | test |
| REQ-9 | Interface | Document create/edit UI must support selecting multiple people per role and reviewing pending suggestions with explicit accept/reject controls. | demonstration | | REQ-9 | Functional | Maintain immutable machine output on `Source.raw_transcription` while permitting inline human edits on `Source.revised_text`. | test |
| REQ-10 | API Constraint | Expose additive, role-aware and state-aware retrieval/filtering in API contracts for documents, people, and relationship records. | test | | REQ-10 | Functional | Support a registry-driven `DocumentType` taxonomy with stable codes, mutable labels, and active/inactive lifecycle control. | test |
| REQ-11 | Data Provenance | Capture suggestion provenance metadata sufficient for operator review (for example source mechanism, confidence, and evidence reference) without mutating canonical asserted links implicitly. | inspection | | REQ-11 | Data Constraint | Store `Document` type as a controlled reference to `DocumentType`. | test |
| REQ-12 | Operations | Provide migration/backfill validation that identifies and resolves historical records violating newly enforced exclusivity policies before hard enforcement. | test | | REQ-12 | Interface | Render multi-page transcriptions sequentially by `page_number` with document, people, and document-type metadata. | demonstration |
| REQ-13 | Revision Upgrade | Support deterministic in-place revision upgrade behavior by mapping existing author/recipient links into V4 asserted semantics without requiring long-lived runtime legacy compatibility layers. | test | | REQ-13 | Interface | Document create/edit UI must support selecting multiple people per role and selecting an active document type from the registry. | demonstration |
| REQ-14 | Reliability | Ensure document/person deletion and cleanup workflows remain safe and deterministic with expanded relationship semantics, including suggestion records. | test | | REQ-14 | API Constraint | Expose additive, role-aware retrieval and write behavior for document-person links and code-based selection for document types. | test |
| REQ-15 | Quality | Add automated test coverage for role extensibility, exclusivity enforcement, suggestion lifecycle transitions, and regression scenarios across service/API/UI flows. | test | | REQ-15 | Data Constraint | Calculate and store cryptographic file hashes (SHA-256) and file sizes for uploaded source images. | test |
| REQ-16 | Functional | Support a registry-driven document type taxonomy (`document_type`) with stable machine-readable codes, mutable display labels, and active/inactive lifecycle control. | test | | REQ-16 | Data Constraint | Preserve a portable relational model across supported backends using SQLModel, SQLAlchemy, SQLite, and PostgreSQL. | inspection |
| REQ-17 | Data Constraint | Replace unconstrained free-text document type assignment with controlled type references or deterministic code mapping governed by the document type registry. | test | | REQ-17 | Reliability | Ensure delete and update flows for documents, people, and relationship links remain deterministic and safe. | test |
| REQ-18 | Interface | Document create/edit UI must present type selection from active registry entries and prevent invalid type assignment. | demonstration | | REQ-18 | Operations Constraint | Keep canonical development, testing, restore, and recovery workflows OS-independent. | inspection |
| REQ-19 | API Constraint | Provide additive API contracts for document type catalog retrieval, including active-only filtering and stable code-based selection for document writes. | test | | REQ-19 | Quality | Provide automated coverage for async transcription workflows, relationship-role enforcement, document-type selection, and regression behavior. | test |
| REQ-20 | Operations | Provide deterministic one-time manual normalization/backfill for existing document type strings in the current small corpus, assigning each document to a canonical registry type before strict write enforcement. | test |
## Clarifying Constraints ## Clarifying Constraints
1. Suggestions are advisory only until accepted by a human operator. 1. `DocumentType.code` and `PersonRole.code` are stable machine identifiers.
2. Relationship acceptance/rejection must be explicit and auditable. 2. `DocumentType.label` and `PersonRole.label` may evolve without changing canonical identity.
3. Role-policy enforcement must occur consistently across service and API boundaries. 3. Role-policy enforcement must be consistent across UI, API, services, and persistence.
4. Many-per-role behavior is required for both asserted and suggested states where applicable. 4. Many-per-role behavior is required for document-person links.
5. Suggestions that would violate asserted exclusivity may exist as `pending`, but acceptance must fail until the exclusivity conflict is resolved. 5. Relationship conflicts must fail deterministically without partial mutation.
6. Document type codes are stable identifiers; display labels may evolve without changing canonical type identity.
## Assumptions
1. V4 scope is limited to document-person relationship expansion and does not redesign the core transcription job execution model.
2. Existing V3 data remains the starting corpus and is transformed via deterministic backfill/validation rules.
3. V4 uses a role registry model and separate suggestion storage; enum-first role expansion is out of scope for this revision.
4. V4 introduces document type registry governance and deprecates unconstrained free-text typing as an authoring-time default.
## Element Satisfaction Mapping ## Element Satisfaction Mapping
- **UI (NiceGUI):** Satisfies REQ-0, REQ-1, REQ-3, REQ-4, REQ-8, REQ-9, REQ-18. - UI (NiceGUI): Satisfies REQ-0, REQ-1, REQ-5, REQ-9, REQ-12, REQ-13.
- **API (FastAPI):** Satisfies REQ-0, REQ-3, REQ-4, REQ-5, REQ-7, REQ-10, REQ-19. - API (FastAPI): Satisfies REQ-1, REQ-4, REQ-5, REQ-7, REQ-8, REQ-14.
- **PERSISTENCE (SQLModel/SQLAlchemy):** Satisfies REQ-1, REQ-2, REQ-5, REQ-6, REQ-11, REQ-12, REQ-13, REQ-14, REQ-16, REQ-17, REQ-20. - Worker (`asyncio`): Satisfies REQ-2, REQ-3, REQ-4.
- **SERVICES (Domain Layer):** Satisfies REQ-4, REQ-5, REQ-7, REQ-10, REQ-14, REQ-17, REQ-20. - Persistence (SQLModel / SQLAlchemy): Satisfies REQ-3, REQ-9, REQ-10, REQ-11, REQ-15, REQ-16, REQ-17.
- **TEST SUITE:** Satisfies REQ-15 and verifies all test-marked requirements. - Test Suite: Verifies all test-marked requirements and satisfies REQ-19.
## Change Classification vs V3
- **Semantic expansion:** role model and assertion state model.
- **Policy expansion:** exclusivity matrix enforcement.
- **Workflow expansion:** suggestion review and promotion lifecycle.
- **Contract expansion:** role/state-aware read and write behavior.
- **Governance expansion:** registry-managed document type taxonomy and normalization workflow.
## Related Local References ## Related Local References
- [V4 Scope Boundary](scope_boundary_v4.md) - [System Overview](index_v4.md)
- [System Overview V3](../index_v3.md) - [System Architecture](architecture_v4.md)
- [System Requirements V3](../requirements_v3.md) - [Data Model](schema_v4.md)
- [Data Model V3](../schema_v3.md) - [Error Handling Policy](error_handling_v4.md)
docs/ver4/schema_v4.md
+158 -180
View File
@@ -1,214 +1,192 @@
# Database Schema (Version 4) # Database Schema (Version 4)
This document defines the selected schema direction for V4 document-person relationship expansion. This document defines the relational schema for the document transcription system.
V4 goals are: ## Entity Relationship Diagram
- extensible role taxonomy, ```mermaid
- extensible document type taxonomy, erDiagram
- explicit assertion state (`asserted`, `suggested`), DOCUMENT_TYPE {
- policy-driven exclusivity, UUID id PK
- deterministic migration from V3 links. TEXT code
TEXT label
BOOLEAN is_active
INTEGER sort_order
TIMESTAMPTZ created_at
TIMESTAMPTZ updated_at
}
## Scope PERSON_ROLE {
UUID id PK
TEXT code
TEXT label
BOOLEAN is_active
TIMESTAMPTZ created_at
TIMESTAMPTZ updated_at
}
This specification focuses on relationship and document-type governance persistence changes. Existing `Person`, `Source`, `Job`, and `JobSource` core structures remain as in V3 unless explicitly noted. ROLE_EXCLUSIVITY {
UUID id PK
UUID left_role_id FK
UUID right_role_id FK
TIMESTAMPTZ created_at
}
## New/Expanded Concepts PERSON {
UUID id PK
TEXT full_name
TEXT display_name
TEXT maiden_name
DATE birth_date
TEXT birth_date_raw
TEXT birth_place
DATE death_date
TEXT death_date_raw
TEXT death_place
TEXT biography
TEXT portrait_path
JSONB metadata
TIMESTAMPTZ created_at
TIMESTAMPTZ updated_at
}
- **Relationship role:** semantic label such as `author`, `recipient`, `mentioned`. DOCUMENT {
- **Assertion state:** whether the link is canonical (`asserted`) or pending review (`suggested`). UUID id PK
- **Exclusivity matrix:** configurable role-pair conflicts for same `(document_id, person_id)`. UUID document_type_id FK
- **Suggestion provenance:** evidence fields enabling review decisions. TEXT name
- **Document type registry:** controlled taxonomy for `Document` classification with stable code identity. DATE document_date
TEXT document_date_raw
TEXT location_created
TEXT notes
TEXT archive_identifier
TIMESTAMPTZ created_at
TIMESTAMPTZ updated_at
}
## Selected Model: Role Registry + Separate Suggestion Table DOCUMENT_PERSON {
UUID id PK
UUID document_id FK
UUID person_id FK
UUID role_id FK
TIMESTAMPTZ created_at
TIMESTAMPTZ updated_at
}
This option cleanly separates canonical links from pending suggestions and enables fully data-driven role expansion. JOB {
UUID id PK
UUID document_id FK
VARCHAR status
INTEGER retry_count
TEXT provider
TEXT model
TEXT prompt_name
TEXT prompt_hash
TEXT system_prompt
TEXT user_prompt
FLOAT temperature
FLOAT top_p
TIMESTAMPTZ date_created
TIMESTAMPTZ date_updated
}
### Tables SOURCE {
UUID id PK
UUID document_id FK
INTEGER page_number
TEXT upload_name
TEXT filename
TEXT file_path
TEXT file_hash
BIGINT file_size_bytes
TEXT raw_transcription
TEXT revised_text
TIMESTAMPTZ date_uploaded
TIMESTAMPTZ date_revised
}
#### `person_role` JOB_SOURCE {
UUID id PK
UUID job_id FK
UUID source_id FK
VARCHAR status
TEXT raw_transcription
JSONB ai_metadata
JSONB raw_api_response
TEXT error_detail
TIMESTAMPTZ executed_at
}
| Column | Type | Notes | DOCUMENT_TYPE ||--o{ DOCUMENT : classifies
| --- | --- | --- | DOCUMENT ||--o{ DOCUMENT_PERSON : has_people
| `id` | UUID PK | Stable key | PERSON ||--o{ DOCUMENT_PERSON : appears_in
| `code` | TEXT UNIQUE | Canonical role code, for example `author`, `recipient`, `mentioned` | PERSON_ROLE ||--o{ DOCUMENT_PERSON : labels
| `label` | TEXT | UI label | PERSON_ROLE ||--o{ ROLE_EXCLUSIVITY : left_rule
| `is_active` | BOOLEAN | Soft-enable/disable role | PERSON_ROLE ||--o{ ROLE_EXCLUSIVITY : right_rule
| `created_at` | TIMESTAMPTZ | Audit timestamp | DOCUMENT ||--o{ JOB : has_jobs
| `updated_at` | TIMESTAMPTZ | Audit timestamp | DOCUMENT ||--o{ SOURCE : contains_pages
JOB ||--o{ JOB_SOURCE : executes
SOURCE ||--o{ JOB_SOURCE : processed_in
```
#### `document_person` (asserted links only) ## Domain Invariants and Provenance Rules
| Column | Type | Notes | ### Page-Level Execution and AI Outputs
| --- | --- | --- |
| `id` | UUID PK | Stable key |
| `document_id` | UUID FK | -> `document.id` |
| `person_id` | UUID FK | -> `person.id` |
| `role_id` | UUID FK | -> `person_role.id` |
| `created_at` | TIMESTAMPTZ | Audit timestamp |
| `updated_at` | TIMESTAMPTZ | Audit timestamp |
Constraints: - Every single page execution by an AI model produces a dedicated `JOB_SOURCE` record.
- Every `JOB` stores the frozen prompt identifier, prompt text, and hyperparameters used at submission time.
- Every `JOB_SOURCE` stores the complete provider response envelope and page-level operational metadata.
- `SOURCE.raw_transcription` caches the latest successful machine output for that page.
- `UNIQUE(document_id, person_id, role_id)` ### Image Storage and Integrity
#### `document_person_suggestion` - Binary images are stored on disk; `SOURCE.file_path` stores the persisted path.
- `SOURCE.file_hash` stores a SHA-256 digest.
- `SOURCE.file_size_bytes` stores the original file size.
| Column | Type | Notes | ### Page Ordering and Revisions
| --- | --- | --- |
| `id` | UUID PK | Stable key |
| `document_id` | UUID FK | -> `document.id` |
| `person_id` | UUID FK | -> `person.id` |
| `role_id` | UUID FK | -> `person_role.id` |
| `status` | TEXT | `pending`, `accepted`, `rejected` |
| `confidence` | FLOAT NULL | Optional score |
| `source_mechanism` | TEXT NULL | For example `rule`, `llm` |
| `evidence_ref` | TEXT NULL | Pointer or excerpt ID |
| `evidence_span` | JSON NULL | Optional text span payload |
| `note` | TEXT NULL | Reviewer note |
| `created_at` | TIMESTAMPTZ | Suggestion creation time |
| `reviewed_at` | TIMESTAMPTZ NULL | Decision time |
| `reviewed_by` | TEXT NULL | Operator identifier |
Constraints: - `SOURCE.page_number` dictates page ordering within a document.
- `SOURCE.raw_transcription` remains immutable machine output.
- `SOURCE.revised_text` stores human edits and is the preferred display value when present.
- `UNIQUE(document_id, person_id, role_id, status)` with policy for multiple pending rows defined in service layer. ### Document-Person Role Governance
- Optional stricter rule: one active pending suggestion per `(document_id, person_id, role_id)`.
#### `role_exclusivity` - Documents support zero, one, or many people per relationship role.
- Relationship roles are defined by `PERSON_ROLE` rather than hardcoded columns.
- `DOCUMENT_PERSON` must be unique for `(document_id, person_id, role_id)`.
- Configured exclusive role pairs from `ROLE_EXCLUSIVITY` cannot coexist for the same `(document_id, person_id)`.
- Initial exclusivity seed blocks `author` and `recipient` for the same person-document pair.
| Column | Type | Notes | ### Document Type Governance
| --- | --- | --- |
| `id` | UUID PK | Stable key |
| `left_role_id` | UUID FK | -> `person_role.id` |
| `right_role_id` | UUID FK | -> `person_role.id` |
| `created_at` | TIMESTAMPTZ | Audit timestamp |
Constraints: - Every document type is defined by `DOCUMENT_TYPE`.
- `DOCUMENT_TYPE.code` is a stable machine identifier.
- `DOCUMENT_TYPE.label` is mutable display text.
- Inactive types remain valid for historical rows but should be excluded from default selection UIs.
- Canonical ordering rule to avoid duplicate pairs (`left_role_id < right_role_id` enforced in service/DB). ## Constraint Summary
- `UNIQUE(left_role_id, right_role_id)`
Initial seed: - `DOCUMENT_TYPE.code` is unique.
- `PERSON_ROLE.code` is unique.
- Exclusivity pair: (`author`, `recipient`) - `DOCUMENT_PERSON(document_id, person_id, role_id)` is unique.
- `ROLE_EXCLUSIVITY(left_role_id, right_role_id)` is unique.
## Deferred Alternative (Not Selected for V4) - `ROLE_EXCLUSIVITY` must use canonical ordering to avoid duplicate mirrored pairs.
An enum-based shared table model was considered but is intentionally not selected for V4 because it couples canonical and provisional states in one table and increases invariant complexity.
## Assertion-State Semantics
- `asserted`: canonical relationship used for document/person metadata and business logic.
- `suggested`: non-canonical proposal requiring explicit review.
- Accept action:
- creates asserted link (or confirms existing),
- marks suggestion `accepted`.
- Reject action:
- marks suggestion `rejected`.
## Exclusivity Enforcement
Policy target:
- For a single `(document_id, person_id)`, disallow coexistence of role pairs configured as exclusive.
Enforcement layers:
1. Service-level pre-check for clear API errors.
2. Database-level guard where feasible (constraints/triggers or deterministic write path).
Initial configured rule:
- `author` and `recipient` are exclusive.
Enforcement semantics:
- Exclusivity is enforced for asserted links.
- Pending suggestions may exist even if they would conflict when asserted.
- Accepting a suggestion must run exclusivity checks and fail deterministically on conflict.
## Migration From V3
### Data Mapping
- Existing V3 `document_person` rows map to V4 `asserted` semantics.
- Existing V3 role values:
- `author` -> role `author`
- `recipient` -> role `recipient`
### Backfill Steps
1. Seed role rows (`author`, `recipient`, `mentioned`) if using Option A.
2. Migrate current links into asserted table/state.
3. Run conflict scan for exclusivity violations.
4. Apply deterministic conflict policy for any violations.
5. Enable hard enforcement after data passes validation.
## Indexing Guidance ## Indexing Guidance
Recommended indexes: - `document(document_type_id)`
- `document_person(document_id)` - `document_person(document_id)`
- `document_person(person_id)` - `document_person(person_id)`
- `document_person(role_id)` (Option A) or `document_person(role)` (Option B) - `document_person(role_id)`
- `document_person_suggestion(document_id, status)` (Option A) - `role_exclusivity(left_role_id, right_role_id)`
- `document_person_suggestion(person_id, status)` (Option A) - `source(document_id, page_number)`
- `job(document_id, status)`
## Selection Rationale - `job_source(job_id)`
- `job_source(source_id)`
V4 selects role registry plus separate suggestion storage for clearer provenance boundaries, cleaner lifecycle transitions, and long-term extensibility.
## Document Type Registry Model (Selected for V4)
V4 applies the same registry governance pattern to document classification.
### Tables
#### `document_type`
| Column | Type | Notes |
| --- | --- | --- |
| `id` | UUID PK | Stable key |
| `code` | TEXT UNIQUE | Canonical type code, for example `letter`, `diary`, `book`, `postcard` |
| `label` | TEXT | UI display label |
| `is_active` | BOOLEAN | Soft-enable/disable type |
| `sort_order` | INTEGER NULL | Optional UI ordering |
| `created_at` | TIMESTAMPTZ | Audit timestamp |
| `updated_at` | TIMESTAMPTZ | Audit timestamp |
#### `document` update
| Column | Type | Notes |
| --- | --- | --- |
| `document_type_id` | UUID FK NULL | -> `document_type.id` |
### Constraints and Governance
- `document_type.code` must be stable and unique.
- `document_type.label` may change without changing canonical type identity.
- Inactive types remain valid for historical records but are excluded from default create/edit selectors.
### Initial Seeds
- Seed baseline type codes from current V3 usage set (for example `letter`, `diary`, `book`, `postcard`, `record`, `memo`) and refine labels as needed.
### Migration and Normalization (Small Corpus)
1. Seed canonical `document_type` rows.
2. Manually assign each existing document (10 total) to a canonical type via `document_type_id`.
3. Resolve any outlier values directly during this one-time pass.
4. Enforce registry-backed write validation after manual assignment is complete.
### Indexing Guidance (Document Type)
- `document_type(code)` unique index.
- `document(document_type_id)` index.
## Related Local References ## Related Local References
- [V4 Scope Boundary](scope_boundary_v4.md) - [System Overview](index_v4.md)
- [V4 Requirements](requirements_v4.md) - [System Architecture](architecture_v4.md)
- [V3 Schema](../schema_v3.md) - [System Requirements](requirements_v4.md)
- [Error Handling Policy](error_handling_v4.md)
docs/ver4/scope_boundary_v4.md
+50 -89
View File
@@ -1,130 +1,91 @@
# V4 Scope Boundary (Version 4) # V4 Scope Boundary
This document defines what is and is not included in Version 4 for expanding `Document`-`Person` relationships. This document defines the scope for the transition from the current repository state to the Version 4 project definition.
## Purpose ## Purpose
Create a clear implementation boundary before updating full V4 architecture, requirements, schema, and plan documents. Define what this revision includes, what it intentionally excludes, and what migration rules govern the transition work.
## Why This Is V4 (Not V3.1) ## In Scope
V4 is required because the change is semantic and cross-cutting: ### 1. Relationship Model
- Expands relationship meaning beyond fixed `author`/`recipient`. - Extensible role taxonomy for document-person relationships.
- Introduces a suggestion lifecycle (`suggested` vs `asserted`). - Many-to-many document-person links with many people per role.
- Introduces policy constraints (role exclusivity matrix). - Deterministic exclusivity policy for configured role pairs.
- Impacts persistence, service contracts, API behavior, UI workflows, and test strategy. - Set-based add/remove synchronization for document-person updates.
A V3.1 patch would only be appropriate for non-semantic quality-of-life fixes inside existing role semantics. ### 2. Document Type Governance
## In Scope for V4 - Registry-driven `DocumentType` model with stable codes and controlled selection.
- Minimal rollout for the current corpus with no alias helper table.
### 1) Relationship Semantics ### 3. UI and API Behavior
- Role taxonomy becomes extensible (not hardcoded to only two role values). - Grouped role links on document and person views.
- A `Document` can link to many `Person` records per role. - Multi-role relationship editing on document create/edit flows.
- Relationship states are explicit: - Role-aware API retrieval and write behavior.
- `asserted`: human-confirmed canonical link. - Additive API evolution with explicit deprecations.
- `suggested`: machine- or heuristic-proposed link pending review.
### 2) Policy Rules ### 4. Verification
- Enforce role exclusivity for the same `(document, person)` pair where configured. - Tests for many-per-role behavior.
- Initial policy decision: - Tests for exclusivity enforcement.
- `author` + `recipient` are mutually exclusive. - Tests for set-based relationship mutation behavior.
- `mentioned` may coexist with other roles. - Tests for document and person delete/link cleanup regressions.
### 3) Persistence and Contracts ## Out of Scope
- Evolve persistence model to support: - Suggested versus asserted relationship states.
- extensible roles, - Suggestion storage, review, acceptance, or rejection workflows.
- suggestion lifecycle metadata, - Automatic relationship extraction or recommendation features.
- deterministic conflict handling. - Full entity resolution or identity merge across all people.
- Replace single-value relationship mutation patterns with set-based sync behavior. - Automated semantic document type classification.
- Redesign of the core transcription execution model.
### 4) Human-in-the-Loop Workflow
- Add assisted suggestion review flow:
- list suggestions,
- accept,
- reject,
- promote accepted suggestion to asserted link.
- No silent auto-promotion from suggestion to asserted.
### 5) UI and API Behavior
- Document create/edit/detail workflows support multi-person per role.
- UI surfaces grouped role links and suggestion status.
- API supports role-aware retrieval and suggestion lifecycle operations.
### 6) Verification
- Add tests for:
- many-per-role behavior,
- exclusivity enforcement,
- suggestion lifecycle,
- migration/conflict detection,
- regression on delete/link cleanup behavior.
## Out of Scope for V4
- Automatic acceptance of suggested links without human review.
- Full entity resolution/identity merge pipeline across all `Person` records.
- Historical provenance graph redesign beyond relationship-level evidence fields.
- Large-scale NLP research features unrelated to document-person linking.
- Changes to core transcription execution model (`Job`, `JobSource`) except where needed to expose suggestion evidence inputs.
## Locked Design Decisions ## Locked Design Decisions
### A) Role Extensibility Mechanism ### A. Role Extensibility Mechanism
- Adopt role registry tables (data-driven roles). - Use registry tables for relationship roles.
- Do not use enum-first role expansion for V4.
### B) Suggestion Storage Model ### B. Exclusivity Matrix Baseline
- Adopt a separate `document_person_suggestion` table. - `author` and `recipient` are mutually exclusive for the same `(document, person)` pair.
- Keep `document_person` focused on canonical asserted links.
- Revisit only if operational complexity proves materially higher than expected.
### C) Exclusivity Matrix Baseline
- `author` + `recipient` are mutually exclusive for the same `(document, person)` pair.
- `mentioned` remains non-exclusive. - `mentioned` remains non-exclusive.
- Hard exclusivity enforcement applies to asserted links.
- Suggestions may be stored even if they would conflict at assert time; promotion to asserted must enforce exclusivity.
### D) API Compatibility Strategy ### C. API Compatibility Strategy
- Use additive API evolution in V4. - Use additive API evolution.
- In development mode, the current revision is authoritative; long-lived legacy compatibility layers are not required. - In development mode, the current revision is authoritative.
- Deprecations should be explicit and short-lived, with removals performed in subsequent revisions. - Deprecations should be explicit and short-lived.
### E) Document Type Rollout Strategy ### D. Document Type Rollout Strategy
- Use a minimal registry rollout for the current corpus: no `document_type_alias` helper table. - Use a minimal registry rollout for the current corpus.
- Perform a one-time manual mapping of existing document types to canonical registry types. - Do not introduce a `document_type_alias` helper table.
### F) Suggestion Generation Strategy (Still Open) ### E. Database Change Policy
- Option A: deterministic rules/NLP over transcribed text. - Future schema changes are non-destructive by default.
- Option B: LLM extraction with confidence/evidence spans. - Exception: `document_type` text may be replaced by `document_type_id` without migrating the legacy text values.
- Exception: `document_person` links may be recreated manually.
## Compatibility and Rollout ## Compatibility and Rollout
- Existing V3 author links migrate to `asserted` behavior. - Preserve existing repository behavior where unaffected by the V4 scope.
- Backfill validation identifies policy conflicts before constraints are enforced. - Treat scope boundary and implementation plan as the only transition documents.
- Preserve existing V3 core behavior where unaffected by role/suggestion evolution. - Treat core V4 documents as the authoritative project definition once rewritten.
## Exit Criteria for Scope Freeze ## Exit Criteria for Scope Freeze
V4 scope is considered frozen when: V4 scope is considered frozen when:
- Suggestion generation strategy is chosen. - Relationship model, exclusivity rules, and document-type governance are approved.
- Suggestion acceptance workflow details are approved.
- Additive API change list and deprecation schedule are approved. - Additive API change list and deprecation schedule are approved.
- Migration exceptions are explicitly acknowledged.
## Core V4 Documents (Current Set) ## Core V4 Documents
1. `docs/ver4/index_v4.md` 1. `docs/ver4/index_v4.md`
2. `docs/ver4/requirements_v4.md` 2. `docs/ver4/requirements_v4.md`