bbchops/transcription
1
0
Fork 0
generated from john/python-template
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

Revised and simplified V4 Plan and core documents.

Browse Source
This commit is contained in:
Jim Lancaster
2026-08-10 10:53:13 -05:00
parent 9b4d6f0340
commit 4b3baf5a3e
7 changed files with 495 additions and 578 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/ver4/architecture_v4.md
+105 -95
View File
@@ -1,137 +1,147 @@
# System Architecture (Version 4)
This document describes the V4 architecture changes for expanded `Document`-`Person` relationship management.
V4 extends V3 with role extensibility, assisted suggestion review, and role-policy enforcement while preserving the existing transcription execution topology.
This document describes the production architecture of the document transcription system.
## Architecture Objectives
- Keep V3 transcription pipeline behavior stable unless relationship evidence extraction requires additive integration.
- Support many people per role for each document.
- Support extensible relationship roles without hardcoding UI and API behavior to two values.
- Support non-canonical suggestion intake with explicit human promotion to canonical asserted links.
- Enforce role exclusivity policy consistently at service and persistence boundaries.
- Preserve auditability for suggestion provenance and review actions.
- Preserve original source material and immutable machine transcription output.
- Support batching one or more images into ordered multi-page documents.
- Capture complete submission-time prompt provenance and per-page provider response evidence.
- Execute page transcription concurrently with bounded `asyncio` workers.
- Maintain relational portability across SQLite and PostgreSQL.
- Keep operator workflows cross-platform and Python-driven.
- Support many-to-many document-person relationships with extensible roles.
- Support registry-driven document type classification.
- Enforce relationship-role exclusivity rules consistently across UI, API, and persistence boundaries.
## Runtime Topology
V4 keeps the existing runtime shape:
The runtime operates as an asynchronous Python application:
- FastAPI + NiceGUI web app process.
- Async service layer with SQLModel/SQLAlchemy persistence.
- Existing worker execution path for transcription jobs.
Additive V4 components:
- Relationship policy evaluator (exclusivity checks).
- Suggestion lifecycle service (propose/list/accept/reject).
- Optional suggestion extraction adapter (rule/NLP or LLM-backed).
- FastAPI + NiceGUI web application process.
- In-process `asyncio` worker engine for transcription execution.
- Relational persistence via SQLModel / SQLAlchemy.
- Pydantic V2 validation across API payloads, prompt configuration, and structured metadata.
^^^mermaid
flowchart LR
U[Browser User] --> UI[NiceGUI Pages]
UI --> API[FastAPI Routes]
API --> DS[Document Service]
API --> SS[Suggestion Service]
DS --> PE[Policy Evaluator]
SS --> PE
DS --> DB[(Relational DB)]
SS --> DB
W[Async Worker Engine] --> DB
W --> E[Optional Suggestion Extraction]
E --> SS
U[Browser User] --> A[FastAPI + NiceGUI App]
A --> W[Asyncio Worker Engine]
A --> DB[(Relational DB)]
W --> P[Vision Provider APIs]
W --> DB
^^^
## Layer Responsibilities
## Lifecycle Ownership
### UI Layer (`src/transcription/ui/**`)
Application lifespan owns runtime setup and teardown:
- Render per-role grouped relationships for document and person views.
- Provide multi-select role assignment controls in document create/edit flows.
- Provide suggestion review controls (accept/reject) for pending suggestions.
- Surface policy conflict errors from API/service layer clearly.
- Initialize logging, settings, directories, and prompt configuration.
- Manage asynchronous database engine connection pools.
- Execute database bootstrap or migrations.
- Recover stale or interrupted jobs on startup.
- Manage graceful shutdown of active background tasks.
### API Layer (`src/transcription/api/**`)
## Layered Module Structure
- Expose role-aware and state-aware read contracts.
- Expose suggestion lifecycle write contracts.
- Return deterministic validation/conflict errors for exclusivity and duplicate semantics.
- Evolve endpoints additively, with explicit deprecations and short-lived transition windows.
### Interface Layer
### Service Layer (`src/transcription/services/**`)
- `src/transcription/ui/**`
- `src/transcription/api/**`
- Implement set-based relationship sync (delta add/remove) to avoid destructive replacement behavior.
- Apply policy evaluator before persistence commits.
- Orchestrate suggestion acceptance/rejection transitions.
- Ensure accepted suggestions result in asserted link creation/confirmation.
Responsibilities:
### Policy Evaluator (New logical component)
- Render document, source, person, job, and classification views.
- Accept user input for uploads, editing, linking, and revisions.
- Present structured validation and conflict feedback.
- Evaluate role exclusivity matrix for `(document_id, person_id)` writes.
- Provide deterministic conflict reason payloads suitable for UI/API display.
- Stay stateless and reusable across create/update/sync code paths.
### Application and Async Worker Layer
### Persistence Layer (`src/transcription/db/**`)
- `src/transcription/services/workflows.py`
- `src/transcription/worker.py`
- Store asserted links and suggestion records according to selected schema option.
- Persist provenance metadata for suggestion records.
- Enforce uniqueness and support performant role/state filtering.
Responsibilities:
## Core V4 Workflows
- Orchestrate uploads, job creation, and status transitions.
- Execute per-page provider calls through bounded concurrency.
- Persist page-level outcomes and update aggregate job state.
### 1) Manual Relationship Management
### Domain and Service Layer
1. User opens document edit view.
2. UI loads asserted links grouped by role.
3. User adds/removes people per role.
4. Service computes delta and runs policy checks.
5. Persistence applies adds/removes atomically.
- `src/transcription/db/models.py`
- `src/transcription/services/*.py`
### 2) Assisted Suggestion Review
Responsibilities:
1. Suggestion records are created (`pending`) by extraction logic or manual propose action.
2. UI displays pending suggestions with evidence metadata.
3. User accepts or rejects each suggestion.
4. Accept path creates/confirms asserted relationship and marks suggestion `accepted`.
5. Reject path marks suggestion `rejected`.
- Manage transactional operations for documents, people, types, links, sources, jobs, and job sources.
- Apply deterministic conflict handling for relationship-role writes.
- Use set-based synchronization for many-to-many relationship updates.
- Resolve and validate registry-backed document types.
### 3) Exclusivity Conflict Handling
### Infrastructure Layer
1. Incoming write attempts role assignment.
2. Policy evaluator checks role pair conflicts for same `(document_id, person_id)`.
3. If conflict exists, write is rejected with structured conflict details.
4. UI presents actionable message without partial updates.
- `src/transcription/db/**`
- `src/transcription/providers/**`
## Invariants
Responsibilities:
- Suggested links are never canonical until accepted.
- Asserted links must satisfy exclusivity rules.
- Relationship mutations are set-based and deterministic.
- Relationship views remain consistent between document detail and person detail pages.
- Existing V3 links remain valid under asserted semantics after migration.
- Provide async database sessions and engine configuration.
- Provide provider adapters for vision model execution.
## Backward Compatibility
## Core Workflows
- V3 author/recipient links are interpreted as asserted links in V4.
- Existing document and person flows continue to function where unaffected by new role/state dimensions.
- During development revisions, the current API contract is authoritative; long-lived legacy runtime compatibility layers are not required.
### 1. Multi-Page Transcription
## Observability and Auditability
1. User uploads one or more images for a `Document`.
2. System stores files, hashes them, creates ordered `Source` rows, and creates a `Job`.
3. Worker claims the job, marks it `processing`, and executes page calls concurrently.
4. Each page writes a `JobSource` result with raw output, metadata, and full provider response evidence.
5. Aggregate status becomes `completed`, `partial_success`, or `failed`.
- Persist suggestion provenance fields sufficient for operator review.
- Record review decision outcomes (`accepted`/`rejected`) with timestamps and reviewer identity where available.
- Emit structured service-level logs for exclusivity conflicts and review actions.
### 2. Document-Person Relationship Management
## Test Strategy Additions
1. User opens a document or person edit flow.
2. UI loads existing links grouped by role.
3. User adds or removes people within one or more roles.
4. Service computes add/remove deltas rather than replacing all links blindly.
5. Conflict checks enforce exclusivity policy before persistence commits.
- Service tests for delta sync and exclusivity matrix enforcement.
- API tests for role/state filtering and suggestion transitions.
- UI tests for multi-role selection and suggestion review interactions.
- Migration tests for V3-to-V4 asserted mapping and conflict scans.
### 3. Document Type Management
1. User selects a registry-backed document type for a document.
2. Service resolves the stable type code or id.
3. Persistence stores the `document_type_id` reference.
4. Inactive types remain valid for historical rows but are excluded from default selectors.
## Domain Invariants
- `Source.raw_transcription` stores immutable machine output.
- Human corrections occur only in `Source.revised_text`.
- Prompt and parameter provenance is frozen on `Job` at submission time.
- Provider output evidence is stored on `JobSource` for each page execution.
- `DocumentPerson` links are unique for `(document_id, person_id, role_id)`.
- Configured exclusive role pairs cannot coexist for the same `(document_id, person_id)`.
- Relationship mutations are deterministic and set-based.
- `DocumentType.code` is stable; `DocumentType.label` may evolve.
## Data Model Summary
- `Document` has one `DocumentType`, many `Source` pages, many `Job` runs, and many `Person` records through `DocumentPerson`.
- `Source` belongs to one `Document` and may participate in many `JobSource` executions.
- `Job` has many `JobSource` rows.
- `PersonRole` defines available relationship roles.
- `RoleExclusivity` defines role pairs that cannot coexist for the same document-person pair.
## Test Strategy
- Unit tests for models, validation, hashing, and registry resolution.
- Service tests for CRUD, set-based sync, and exclusivity enforcement.
- Async workflow tests for page isolation, partial failure handling, and stored evidence.
- UI integration tests for multi-page rendering, role grouping, and document type selection.
## Related Local References
- [V4 Scope Boundary](scope_boundary_v4.md)
- [V4 Requirements](requirements_v4.md)
- [V4 Schema](schema_v4.md)
- [V3 Architecture](../architecture_v3.md)
- [System Overview](index_v4.md)
- [System Requirements](requirements_v4.md)
- [Data Model](schema_v4.md)
- [Error Handling Policy](error_handling_v4.md)
docs/ver4/error_handling_v4.md
+57 -51
View File
@@ -1,62 +1,68 @@
# Error Handling Policy (Version 4)
This document defines canonical error-handling behavior for V4 document-person relationship expansion.
V4 keeps V3 transcription error behavior and adds policy/conflict handling for role extensibility, suggestion lifecycle transitions, and exclusivity enforcement.
This document defines the canonical error-handling policy for the document transcription system.
## Error Handling Objectives
- Provide clear, actionable conflict and validation feedback for relationship write operations.
- Prevent partial, silent, or destructive relationship mutations when policy checks fail.
- Preserve suggestion review auditability with deterministic accept/reject outcomes.
- Keep consistent API/UI/service error envelopes across relationship workflows.
- Make failures visible in clear, actionable language at both the document and page levels.
- Support isolated failure handling in multi-page jobs so one failing page does not invalidate successful pages.
- Preserve diagnostic detail for validation failures, provider failures, and policy conflicts.
- Ensure consistent error envelope structure across API, UI, service, and worker boundaries.
## Scope and Authority
Governs relationship-related error behavior in:
This policy governs error behavior across:
- NiceGUI document/person relationship views,
- FastAPI relationship and suggestion endpoints,
- domain services for relationship sync and suggestion review,
- persistence constraints for role, state, and exclusivity invariants.
- NiceGUI pages
- FastAPI routes
- Service-layer orchestration
- `asyncio` worker tasks
- Database interactions
- Provider adapters
## Relationship Error Taxonomy
## Error Taxonomy
| Category | Definition | Retriable |
| --- | --- | --- |
| `validation_error` | Payload shape/type invalid, unknown role/state, malformed IDs | no |
| `not_found_error` | Target `Document`, `Person`, role, or suggestion record does not exist | no |
| `conflict_error` | Write violates uniqueness or exclusivity policy | no |
| `suggestion_state_error` | Invalid suggestion transition (for example accept after reject) | no |
| `policy_violation_error` | Action blocked by configured role matrix or governance rule | no |
| `infrastructure_transient_error` | Temporary DB or network instability during relationship operation | yes |
| `infrastructure_persistent_error` | Persistent DB/configuration failure | no |
| `internal_unexpected_error` | Unhandled exception/logic defect | no |
| `validation_error` | Payload, parameter, or schema validation failure | no |
| `user_input_error` | Unacceptable file, invalid selection, or malformed request from the operator | no |
| `not_found_error` | Requested `Document`, `Source`, `Person`, `Job`, role, or type does not exist | no |
| `conflict_error` | Operation violates uniqueness or exclusivity policy | no |
| `external_provider_error` | Provider API failure, rate limit, or execution problem | yes |
| `infrastructure_transient_error` | Temporary DB, file-system, or network instability | yes |
| `infrastructure_persistent_error` | Persistent configuration, credential, or database availability failure | no |
| `internal_unexpected_error` | Uncaught exception or logic defect | no |
## Deterministic Conflict Behavior
## Async Batch and Page-Level Error Behavior
When relationship writes fail policy checks:
In multi-page `asyncio` processing:
1. Reject the full write operation (no partial apply).
2. Return structured conflict details including conflicting role pair and target identifiers.
3. Preserve existing canonical relationships unchanged.
1. Exceptions from individual page calls are trapped within the page task wrapper.
2. Failed page detail is written to `JobSource.error_detail` and the page state becomes `failed`.
3. Aggregate job status is derived from page outcomes:
- all pages succeed -> `completed`
- some succeed and some fail -> `partial_success`
- all fail -> `failed`
4. Successful pages remain valid even when sister pages fail.
When suggestion transitions fail:
## Relationship and Classification Conflict Behavior
1. Reject invalid state transition.
2. Return current state and allowed next actions.
3. Preserve suggestion record integrity.
When relationship or document-type writes fail policy checks:
1. Reject the full write operation.
2. Return structured conflict detail including target identifiers and the violated rule.
3. Preserve existing persisted relationships unchanged.
## API Error Response Contract
Relationship endpoints return a structured envelope:
API error responses return a structured envelope:
^^^json
{
"error_id": "err_uuid_12345",
"category": "conflict_error",
"message": "Role assignment violates exclusivity policy.",
"suggestion": "Remove recipient role before assigning author for this person on this document.",
"suggestion": "Remove recipient before assigning author for this person on this document.",
"details": {
"document_id": "...",
"person_id": "...",
@@ -64,41 +70,41 @@ Relationship endpoints return a structured envelope:
"conflicting_role": "recipient",
"policy_rule": "author+recipient exclusive"
},
"timestamp": "2026-08-09T15:00:00Z"
"timestamp": "2026-08-10T15:00:00Z"
}
^^^
HTTP status mappings:
- `validation_error` -> `400`
- `validation_error`, `user_input_error` -> `400`
- `not_found_error` -> `404`
- `conflict_error`, `suggestion_state_error`, `policy_violation_error` -> `409`
- `conflict_error` -> `409`
- `external_provider_error` -> `502` or `503`
- `infrastructure_transient_error` -> `503`
- `infrastructure_persistent_error`, `internal_unexpected_error` -> `500`
## UI Error Presentation Rules
- Display concise conflict summary with actionable next step.
- Keep user edits in context (do not discard form state when feasible).
- Differentiate between validation issues, policy conflicts, and infrastructure failures.
- For bulk role sync operations, show per-item conflict context when multiple failures occur.
- Display concise failure summaries with the next action the operator can take.
- Keep form state in context when feasible.
- Distinguish validation issues, conflict issues, provider failures, and infrastructure failures.
- For bulk relationship updates, identify the specific role or person that caused a conflict.
## Logging and Audit Expectations
- Log relationship write failures with correlation IDs.
- Log suggestion acceptance/rejection outcomes with actor and timestamp where available.
- Log policy matrix violations with deterministic machine-readable context.
- Log worker failures with correlation IDs and provider context.
- Log relationship and classification conflicts with machine-readable detail.
- Log persisted provider errors and page-level execution failures.
## Relationship-Specific Retry Guidance
## Retry Guidance
- Do not auto-retry policy or conflict failures.
- Permit user-driven retry only after input changes.
- Retry infrastructure transient failures with bounded policy in service layer if operation is idempotent.
- Do not auto-retry validation or conflict failures.
- Permit user-driven retry after the input or selection changes.
- Allow bounded retry for transient provider or infrastructure failures when the operation is idempotent.
## Related Local References
- [V4 Scope Boundary](scope_boundary_v4.md)
- [V4 Requirements](requirements_v4.md)
- [V4 Schema](schema_v4.md)
- [V4 Architecture](architecture_v4.md)
- [V3 Error Handling](../error_handling_v3.md)
- [System Overview](index_v4.md)
- [System Requirements](requirements_v4.md)
- [Data Model](schema_v4.md)
- [System Architecture](architecture_v4.md)
docs/ver4/implementation_plan_v4.md
+61 -71
View File
@@ -2,106 +2,96 @@
## Goal
Implement V4 document-person relationship expansion and document type governance with extensible registries, assisted suggestion lifecycle, and policy-enforced exclusivity while preserving V3 transcription behavior.
Implement the Version 4 project definition from the current repository state while preserving existing data by default.
## Migration Policy
- Database changes are non-destructive by default.
- Exception: the legacy `document_type` text field may be replaced by a `document_type_id` reference without migrating existing text values.
- Exception: `document_person` links may be recreated manually.
## Current Project Impact
- `src/transcription/db/models.py` will require relationship schema evolution for role extensibility and suggestion lifecycle support.
- `src/transcription/db/models.py` will require document type registry entities and document type reference updates.
- `src/transcription/services/documents.py` will require set-based relationship sync and policy checks.
- `src/transcription/services/documents.py` will require registry-based document type lookup, validation, and normalization helpers.
- API modules under `src/transcription/api/**` will require role/state-aware contracts and suggestion lifecycle endpoints.
- API modules under `src/transcription/api/**` will require additive document type catalog and code-based selection contracts.
- UI pages under `src/transcription/ui/pages/**` will require multi-role, multi-person editing, suggestion review controls, and registry-backed type selectors.
- Existing tests under `tests/services`, `tests/api`, and `tests/ui` need expanded coverage for V4 behavior and regression safety.
- `src/transcription/db/models.py` requires full schema alignment with the V4 core documents.
- `src/transcription/services/documents.py` requires set-based document-person sync, exclusivity checks, and document-type resolution.
- API modules require additive role-aware relationship behavior and document-type selection behavior.
- UI pages require grouped role displays, multi-role editing, and registry-backed document-type selection.
- Existing tests require updates for role enforcement, document-type selection, and regression safety.
## Implementation Phases
### 1. Finalize V4 Schema Decisions
### 1. Finalize the Transition Documents
- Role extensibility mechanism: use role registry tables.
- Document type extensibility mechanism: use `document_type` registry tables.
- Suggestion storage model: use a separate `document_person_suggestion` table.
- Exclusivity baseline: `author` vs `recipient` exclusive, `mentioned` non-exclusive.
- Enforce exclusivity on asserted links; evaluate conflicts on suggestion acceptance.
- Confirm the reset scope.
- Confirm the database exception policy.
- Keep core V4 documents as the only authoritative product definition.
### 2. Evolve Persistence Layer
### 2. Align the Persistence Layer
- Implement selected schema model in SQLModel.
- Add constraints and indexes for dedupe, filtering, and policy support.
- Add migration/backfill logic for V3 links to V4 asserted semantics.
- Add conflict scan tooling for historical records violating exclusivity rules.
- Perform a one-time manual mapping of existing document type values to registry-backed type references (`document_type_id`) for the current small corpus.
- Update SQLModel definitions to match the final V4 schema.
- Add `person_role`, `role_exclusivity`, and `document_type` support.
- Replace legacy document-type storage with `document_type_id`.
- Apply the accepted manual exception strategy for `document_type` and `document_person` data.
- Preserve all other data structures non-destructively.
### 3. Implement Service-Layer Policy and Sync Semantics
### 3. Update Services and Write Semantics
- Add relationship delta sync operations (set-based add/remove).
- Implement centralized policy evaluator for exclusivity checks.
- Add suggestion lifecycle operations (propose/list/accept/reject).
- Ensure accepted suggestions create/confirm asserted links atomically.
- Add document type resolution operations by stable `code` and active/inactive state handling.
- Implement set-based synchronization for document-person updates.
- Implement deterministic exclusivity conflict checks.
- Remove suggestion-related service behavior.
- Add document-type resolution and validation by stable code or id.
### 4. Implement API Contract Changes
### 4. Update API Contracts
- Add role-aware and state-aware query parameters/filters.
- Add suggestion lifecycle endpoints and response models.
- Add deterministic conflict/error payloads aligned to V4 error policy.
- Use additive endpoint evolution for V4; during development mode, deprecate then remove without maintaining long-lived legacy runtime compatibility layers.
- Add document type catalog endpoints with active-only filtering and code-based selection for document writes.
- Keep API evolution additive.
- Add role-aware relationship retrieval and write behavior.
- Add document-type catalog retrieval and code-based selection for document writes.
- Remove suggestion-related API surfaces from the V4 target state.
### 5. Update UI Workflows
- Replace single-author controls with grouped multi-role selectors.
- Replace unconstrained document type free-text entry with registry-backed type selection.
- Add pending suggestion review panel and accept/reject actions.
- Update document/person detail cards to group links by role and state.
- Preserve edit-state ergonomics on validation/conflict failures.
- Replace single-person link editing with grouped multi-role editing.
- Render grouped role links on document and person detail views.
- Replace free-text document type entry with registry-backed selection.
- Preserve clear validation and conflict messaging.
### 6. Verification and Hardening
- Add service tests for:
- many-per-role behavior,
- exclusivity enforcement,
- set-based sync correctness,
- suggestion transition validity,
- document type code resolution and inactive-type handling.
- Add API tests for role/state filtering and conflict response shapes.
- Add API tests for document type catalog retrieval and code-based write validation.
- Add UI tests/manual walkthroughs for create/edit/review workflows.
- Add UI tests/manual walkthroughs for registry-backed document type selection and validation messaging.
- Add regression tests for document/person delete cleanup semantics.
- Add or update service tests for many-per-role behavior, exclusivity enforcement, and set-based sync correctness.
- Add API tests for relationship behavior and document-type selection.
- Add UI tests or walkthrough coverage for grouped roles and type selection.
- Add regression coverage for delete and cleanup semantics.
## Done When
- V4 relationship schema and contracts are implemented and validated.
- V4 document type registry schema and contracts are implemented and validated.
- Suggestions remain non-canonical until explicit acceptance.
- Exclusivity policy is enforced deterministically across service/API boundaries.
- Existing V3 links are migrated to V4 asserted semantics without data loss.
- Existing document type strings are manually normalized to registry references for the current corpus.
- Test suite includes V4-specific coverage and passes on supported backends.
- Core V4 documents and code paths agree on the final project definition.
- Relationship-role writes are deterministic and non-destructive.
- Exclusivity rules are enforced consistently.
- Document type selection is registry-backed.
- The accepted manual exceptions for `document_type` and `document_person` are completed.
- The focused test coverage passes.
## Out of Scope
- Automatic acceptance of suggestions.
- Global person entity-resolution/merge engine.
- Core transcription execution redesign unrelated to relationship expansion.
- Automated semantic document type classification.
- Suggested/asserted relationship state.
- Suggestion review or extraction workflows.
- Global person entity-resolution engine.
- Automated semantic document-type classification.
## Delivery Order Recommendation
1. Requirements freeze (`requirements_v4.md`).
2. Schema decision freeze (`schema_v4.md`).
3. Error policy freeze (`error_handling_v4.md`).
4. Implementation of persistence and service layer.
5. API and UI changes.
6. Final integration and regression validation.
1. Freeze scope boundary and implementation plan.
2. Freeze core V4 documents.
3. Align persistence models.
4. Align services and API behavior.
5. Align UI behavior.
6. Run focused verification and regression checks.
## Related Local References
- [V4 Scope Boundary](scope_boundary_v4.md)
- [V4 Requirements](requirements_v4.md)
- [V4 Schema](schema_v4.md)
- [V4 Architecture](architecture_v4.md)
- [V4 Error Handling](error_handling_v4.md)
- [V3 Implementation Plan](../implementation_plan_v3.md)
- [System Overview](index_v4.md)
- [System Requirements](requirements_v4.md)
- [Data Model](schema_v4.md)
- [System Architecture](architecture_v4.md)
- [Error Handling Policy](error_handling_v4.md)
docs/ver4/index_v4.md
+28 -37
View File
@@ -1,49 +1,40 @@
# Document-Person Expansion Overview (Version 4)
# Document Transcription System Overview (Version 4)
Version 4 defines the relationship-model evolution track for linking `Document` and `Person` entities with extensible roles, suggestion review, and policy enforcement.
This project is a personal-scale application for transcribing, organizing, and preserving historical documents, images, and related people records.
## Start Here
Read [scope_boundary_v4.md](scope_boundary_v4.md) first to confirm scope and non-scope before implementation work begins.
Read [architecture_v4.md](architecture_v4.md) first for the technical overview and system design.
## Core V4 Capabilities
## Core Capabilities
- Extensible relationship role taxonomy.
- Many-people-per-role linking for documents.
- Explicit distinction between canonical asserted links and pending suggested links.
- Human-in-the-loop suggestion accept/reject workflow.
- Role exclusivity policy enforcement for configured role pairs.
- Role/state-aware API and UI retrieval/presentation behavior.
- Minimal document type governance rollout for the current corpus, with one-time manual mapping and no alias helper table.
- Folder and multi-image ingestion into sequential `Source` pages under a single `Document`.
- Parallel asynchronous AI vision transcription using Python `asyncio` bounded by rate limits.
- Portable relational storage using SQLModel and SQLAlchemy across SQLite and PostgreSQL.
- Complete prompt and response provenance for every transcription job and page execution.
- File-integrity tracking through SHA-256 hashing and stored file sizes.
- Historical `Person` management with many-to-many document links and extensible relationship roles.
- Registry-driven `DocumentType` classification with stable codes and controlled selection.
- Inline human revision of transcribed pages while preserving immutable machine output.
- Partial-failure recovery for multi-page jobs.
- Cross-platform operational workflows driven by Python-based tooling.
## V4 Documentation Index
## Technical Stack
- [Scope Boundary](scope_boundary_v4.md)
- Application Web Framework: FastAPI + NiceGUI
- Persistence Engine: SQLModel / SQLAlchemy
- Data Validation and Schemas: Pydantic V2
- Concurrency and Workers: Python `asyncio`
- Vision Providers: OpenAI, Anthropic, and OpenRouter adapters
## Core Documentation Index
- [System Architecture](architecture_v4.md)
- [System Requirements](requirements_v4.md)
- [Data Model](schema_v4.md)
- [System Architecture](architecture_v4.md)
- [Error Handling Policy](error_handling_v4.md)
## Transition Documents
- [Scope Boundary](scope_boundary_v4.md)
- [Implementation Plan](implementation_plan_v4.md)
## Relationship To V3
V3 remains the baseline production architecture and requirements set for transcription pipeline behavior. V4 is an additive evolution track focused on document-person relationship semantics and workflows.
## Decision Status
Locked decisions:
1. Role extensibility uses registry tables.
2. Suggestion storage uses a separate suggestion table.
3. Exclusivity baseline is `author` vs `recipient` exclusive, with `mentioned` non-exclusive.
4. API evolution is additive in development mode with explicit deprecate-then-remove behavior.
Remaining decision:
1. Suggestion generation strategy (deterministic rules/NLP vs LLM extraction).
## Related Local References
- [V3 System Overview](../index_v3.md)
- [V3 Requirements](../requirements_v3.md)
- [V3 Schema](../schema_v3.md)
docs/ver4/requirements_v4.md
+36 -55
View File
@@ -1,70 +1,51 @@
# Relationship and Document Type Governance Requirements (Version 4)
# Document Transcription System Requirements (Version 4)
This document defines Version 4 baseline requirements for expanding relationships between `Document` and `Person` and introducing governed document type classification.
V4 preserves all applicable V3 capabilities and adds role extensibility, assisted suggestion workflows, explicit relationship policy enforcement, and registry-driven `Document` type governance.
This document defines the baseline requirements for the document transcription system.
## Requirements Model
| ID | Category | Requirement | Verify Method |
| --- | --- | --- | --- |
| REQ-0 | System | Provide end-to-end, reviewable, policy-enforced document-person relationship management supporting asserted and suggested links. | demonstration |
| REQ-1 | Functional | Preserve many-to-many `Document` ↔ `Person` linking, and allow multiple people per role on a single document. | test |
| REQ-2 | Functional | Support an extensible role taxonomy for document-person relationships beyond fixed `author`/`recipient`. | inspection |
| REQ-3 | Functional | Represent relationship assertion state explicitly (`asserted` and `suggested`) and keep machine suggestions non-canonical until human acceptance. | test |
| REQ-4 | Functional | Provide assisted suggestion lifecycle operations: create/list/filter suggestions, accept suggestion, reject suggestion, and promote accepted suggestions to asserted links. | test |
| REQ-5 | Policy Constraint | Enforce a role exclusivity matrix for a single `(document_id, person_id)` pair on asserted links; initial rule set must block `author` + `recipient` coexistence while allowing `mentioned` to coexist with other roles. | test |
| REQ-6 | Data Constraint | Store canonical asserted links in `document_person` and lifecycle-managed suggestions in a separate `document_person_suggestion` table. | test |
| REQ-7 | Functional | Provide set-based synchronization behavior for relationship mutations (add/remove delta), replacing single-value replacement patterns that can drop unrelated links. | test |
| REQ-8 | Interface | Render grouped relationship metadata by role and assertion state on document detail and person detail views. | demonstration |
| REQ-9 | Interface | Document create/edit UI must support selecting multiple people per role and reviewing pending suggestions with explicit accept/reject controls. | demonstration |
| REQ-10 | API Constraint | Expose additive, role-aware and state-aware retrieval/filtering in API contracts for documents, people, and relationship records. | test |
| REQ-11 | Data Provenance | Capture suggestion provenance metadata sufficient for operator review (for example source mechanism, confidence, and evidence reference) without mutating canonical asserted links implicitly. | inspection |
| REQ-12 | Operations | Provide migration/backfill validation that identifies and resolves historical records violating newly enforced exclusivity policies before hard enforcement. | test |
| REQ-13 | Revision Upgrade | Support deterministic in-place revision upgrade behavior by mapping existing author/recipient links into V4 asserted semantics without requiring long-lived runtime legacy compatibility layers. | test |
| REQ-14 | Reliability | Ensure document/person deletion and cleanup workflows remain safe and deterministic with expanded relationship semantics, including suggestion records. | test |
| REQ-15 | Quality | Add automated test coverage for role extensibility, exclusivity enforcement, suggestion lifecycle transitions, and regression scenarios across service/API/UI flows. | test |
| REQ-16 | Functional | Support a registry-driven document type taxonomy (`document_type`) with stable machine-readable codes, mutable display labels, and active/inactive lifecycle control. | test |
| REQ-17 | Data Constraint | Replace unconstrained free-text document type assignment with controlled type references or deterministic code mapping governed by the document type registry. | test |
| REQ-18 | Interface | Document create/edit UI must present type selection from active registry entries and prevent invalid type assignment. | demonstration |
| REQ-19 | API Constraint | Provide additive API contracts for document type catalog retrieval, including active-only filtering and stable code-based selection for document writes. | test |
| REQ-20 | Operations | Provide deterministic one-time manual normalization/backfill for existing document type strings in the current small corpus, assigning each document to a canonical registry type before strict write enforcement. | test |
| REQ-0 | System | Provide end-to-end multi-page document transcription with persistent, inspectable async job states. | demonstration |
| REQ-1 | Functional | Allow users to upload one or more images as ordered `Source` pages under a `Document`. | test |
| REQ-2 | Functional | Process page transcription asynchronously using an `asyncio` worker pool bounded by rate limits. | test |
| REQ-3 | Functional | Persist submission-time prompt configuration and full page-level provider response evidence for every job execution. | test |
| REQ-4 | Functional | Support job states `queued`, `processing`, `completed`, `partial_success`, and `failed`, plus page states `pending`, `transcribed`, and `failed`. | inspection |
| REQ-5 | Functional | Allow users to manage historical `Person` records and link multiple people per role to a `Document`. | test |
| REQ-6 | Functional | Support an extensible role taxonomy for document-person relationships. | inspection |
| REQ-7 | Policy Constraint | Enforce a role exclusivity matrix for a single `(document_id, person_id)` pair; initial rules must block `author` + `recipient` coexistence while allowing `mentioned` to coexist with other roles. | test |
| REQ-8 | Functional | Use set-based synchronization for document-person mutations so updates add and remove only the intended links. | test |
| REQ-9 | Functional | Maintain immutable machine output on `Source.raw_transcription` while permitting inline human edits on `Source.revised_text`. | test |
| REQ-10 | Functional | Support a registry-driven `DocumentType` taxonomy with stable codes, mutable labels, and active/inactive lifecycle control. | test |
| REQ-11 | Data Constraint | Store `Document` type as a controlled reference to `DocumentType`. | test |
| REQ-12 | Interface | Render multi-page transcriptions sequentially by `page_number` with document, people, and document-type metadata. | demonstration |
| REQ-13 | Interface | Document create/edit UI must support selecting multiple people per role and selecting an active document type from the registry. | demonstration |
| REQ-14 | API Constraint | Expose additive, role-aware retrieval and write behavior for document-person links and code-based selection for document types. | test |
| REQ-15 | Data Constraint | Calculate and store cryptographic file hashes (SHA-256) and file sizes for uploaded source images. | test |
| REQ-16 | Data Constraint | Preserve a portable relational model across supported backends using SQLModel, SQLAlchemy, SQLite, and PostgreSQL. | inspection |
| REQ-17 | Reliability | Ensure delete and update flows for documents, people, and relationship links remain deterministic and safe. | test |
| REQ-18 | Operations Constraint | Keep canonical development, testing, restore, and recovery workflows OS-independent. | inspection |
| REQ-19 | Quality | Provide automated coverage for async transcription workflows, relationship-role enforcement, document-type selection, and regression behavior. | test |
## Clarifying Constraints
1. Suggestions are advisory only until accepted by a human operator.
2. Relationship acceptance/rejection must be explicit and auditable.
3. Role-policy enforcement must occur consistently across service and API boundaries.
4. Many-per-role behavior is required for both asserted and suggested states where applicable.
5. Suggestions that would violate asserted exclusivity may exist as `pending`, but acceptance must fail until the exclusivity conflict is resolved.
6. Document type codes are stable identifiers; display labels may evolve without changing canonical type identity.
## Assumptions
1. V4 scope is limited to document-person relationship expansion and does not redesign the core transcription job execution model.
2. Existing V3 data remains the starting corpus and is transformed via deterministic backfill/validation rules.
3. V4 uses a role registry model and separate suggestion storage; enum-first role expansion is out of scope for this revision.
4. V4 introduces document type registry governance and deprecates unconstrained free-text typing as an authoring-time default.
1. `DocumentType.code` and `PersonRole.code` are stable machine identifiers.
2. `DocumentType.label` and `PersonRole.label` may evolve without changing canonical identity.
3. Role-policy enforcement must be consistent across UI, API, services, and persistence.
4. Many-per-role behavior is required for document-person links.
5. Relationship conflicts must fail deterministically without partial mutation.
## Element Satisfaction Mapping
- **UI (NiceGUI):** Satisfies REQ-0, REQ-1, REQ-3, REQ-4, REQ-8, REQ-9, REQ-18.
- **API (FastAPI):** Satisfies REQ-0, REQ-3, REQ-4, REQ-5, REQ-7, REQ-10, REQ-19.
- **PERSISTENCE (SQLModel/SQLAlchemy):** Satisfies REQ-1, REQ-2, REQ-5, REQ-6, REQ-11, REQ-12, REQ-13, REQ-14, REQ-16, REQ-17, REQ-20.
- **SERVICES (Domain Layer):** Satisfies REQ-4, REQ-5, REQ-7, REQ-10, REQ-14, REQ-17, REQ-20.
- **TEST SUITE:** Satisfies REQ-15 and verifies all test-marked requirements.
## Change Classification vs V3
- **Semantic expansion:** role model and assertion state model.
- **Policy expansion:** exclusivity matrix enforcement.
- **Workflow expansion:** suggestion review and promotion lifecycle.
- **Contract expansion:** role/state-aware read and write behavior.
- **Governance expansion:** registry-managed document type taxonomy and normalization workflow.
- UI (NiceGUI): Satisfies REQ-0, REQ-1, REQ-5, REQ-9, REQ-12, REQ-13.
- API (FastAPI): Satisfies REQ-1, REQ-4, REQ-5, REQ-7, REQ-8, REQ-14.
- Worker (`asyncio`): Satisfies REQ-2, REQ-3, REQ-4.
- Persistence (SQLModel / SQLAlchemy): Satisfies REQ-3, REQ-9, REQ-10, REQ-11, REQ-15, REQ-16, REQ-17.
- Test Suite: Verifies all test-marked requirements and satisfies REQ-19.
## Related Local References
- [V4 Scope Boundary](scope_boundary_v4.md)
- [System Overview V3](../index_v3.md)
- [System Requirements V3](../requirements_v3.md)
- [Data Model V3](../schema_v3.md)
- [System Overview](index_v4.md)
- [System Architecture](architecture_v4.md)
- [Data Model](schema_v4.md)
- [Error Handling Policy](error_handling_v4.md)
docs/ver4/schema_v4.md
+158 -180
View File
@@ -1,214 +1,192 @@
# Database Schema (Version 4)
This document defines the selected schema direction for V4 document-person relationship expansion.
This document defines the relational schema for the document transcription system.
V4 goals are:
## Entity Relationship Diagram
- extensible role taxonomy,
- extensible document type taxonomy,
- explicit assertion state (`asserted`, `suggested`),
- policy-driven exclusivity,
- deterministic migration from V3 links.
```mermaid
erDiagram
DOCUMENT_TYPE {
UUID id PK
TEXT code
TEXT label
BOOLEAN is_active
INTEGER sort_order
TIMESTAMPTZ created_at
TIMESTAMPTZ updated_at
}
## Scope
PERSON_ROLE {
UUID id PK
TEXT code
TEXT label
BOOLEAN is_active
TIMESTAMPTZ created_at
TIMESTAMPTZ updated_at
}
This specification focuses on relationship and document-type governance persistence changes. Existing `Person`, `Source`, `Job`, and `JobSource` core structures remain as in V3 unless explicitly noted.
ROLE_EXCLUSIVITY {
UUID id PK
UUID left_role_id FK
UUID right_role_id FK
TIMESTAMPTZ created_at
}
## New/Expanded Concepts
PERSON {
UUID id PK
TEXT full_name
TEXT display_name
TEXT maiden_name
DATE birth_date
TEXT birth_date_raw
TEXT birth_place
DATE death_date
TEXT death_date_raw
TEXT death_place
TEXT biography
TEXT portrait_path
JSONB metadata
TIMESTAMPTZ created_at
TIMESTAMPTZ updated_at
}
- **Relationship role:** semantic label such as `author`, `recipient`, `mentioned`.
- **Assertion state:** whether the link is canonical (`asserted`) or pending review (`suggested`).
- **Exclusivity matrix:** configurable role-pair conflicts for same `(document_id, person_id)`.
- **Suggestion provenance:** evidence fields enabling review decisions.
- **Document type registry:** controlled taxonomy for `Document` classification with stable code identity.
DOCUMENT {
UUID id PK
UUID document_type_id FK
TEXT name
DATE document_date
TEXT document_date_raw
TEXT location_created
TEXT notes
TEXT archive_identifier
TIMESTAMPTZ created_at
TIMESTAMPTZ updated_at
}
## Selected Model: Role Registry + Separate Suggestion Table
DOCUMENT_PERSON {
UUID id PK
UUID document_id FK
UUID person_id FK
UUID role_id FK
TIMESTAMPTZ created_at
TIMESTAMPTZ updated_at
}
This option cleanly separates canonical links from pending suggestions and enables fully data-driven role expansion.
JOB {
UUID id PK
UUID document_id FK
VARCHAR status
INTEGER retry_count
TEXT provider
TEXT model
TEXT prompt_name
TEXT prompt_hash
TEXT system_prompt
TEXT user_prompt
FLOAT temperature
FLOAT top_p
TIMESTAMPTZ date_created
TIMESTAMPTZ date_updated
}
### Tables
SOURCE {
UUID id PK
UUID document_id FK
INTEGER page_number
TEXT upload_name
TEXT filename
TEXT file_path
TEXT file_hash
BIGINT file_size_bytes
TEXT raw_transcription
TEXT revised_text
TIMESTAMPTZ date_uploaded
TIMESTAMPTZ date_revised
}
#### `person_role`
JOB_SOURCE {
UUID id PK
UUID job_id FK
UUID source_id FK
VARCHAR status
TEXT raw_transcription
JSONB ai_metadata
JSONB raw_api_response
TEXT error_detail
TIMESTAMPTZ executed_at
}
| Column | Type | Notes |
| --- | --- | --- |
| `id` | UUID PK | Stable key |
| `code` | TEXT UNIQUE | Canonical role code, for example `author`, `recipient`, `mentioned` |
| `label` | TEXT | UI label |
| `is_active` | BOOLEAN | Soft-enable/disable role |
| `created_at` | TIMESTAMPTZ | Audit timestamp |
| `updated_at` | TIMESTAMPTZ | Audit timestamp |
DOCUMENT_TYPE ||--o{ DOCUMENT : classifies
DOCUMENT ||--o{ DOCUMENT_PERSON : has_people
PERSON ||--o{ DOCUMENT_PERSON : appears_in
PERSON_ROLE ||--o{ DOCUMENT_PERSON : labels
PERSON_ROLE ||--o{ ROLE_EXCLUSIVITY : left_rule
PERSON_ROLE ||--o{ ROLE_EXCLUSIVITY : right_rule
DOCUMENT ||--o{ JOB : has_jobs
DOCUMENT ||--o{ SOURCE : contains_pages
JOB ||--o{ JOB_SOURCE : executes
SOURCE ||--o{ JOB_SOURCE : processed_in
```
#### `document_person` (asserted links only)
## Domain Invariants and Provenance Rules
| Column | Type | Notes |
| --- | --- | --- |
| `id` | UUID PK | Stable key |
| `document_id` | UUID FK | -> `document.id` |
| `person_id` | UUID FK | -> `person.id` |
| `role_id` | UUID FK | -> `person_role.id` |
| `created_at` | TIMESTAMPTZ | Audit timestamp |
| `updated_at` | TIMESTAMPTZ | Audit timestamp |
### Page-Level Execution and AI Outputs
Constraints:
- Every single page execution by an AI model produces a dedicated `JOB_SOURCE` record.
- Every `JOB` stores the frozen prompt identifier, prompt text, and hyperparameters used at submission time.
- Every `JOB_SOURCE` stores the complete provider response envelope and page-level operational metadata.
- `SOURCE.raw_transcription` caches the latest successful machine output for that page.
- `UNIQUE(document_id, person_id, role_id)`
### Image Storage and Integrity
#### `document_person_suggestion`
- Binary images are stored on disk; `SOURCE.file_path` stores the persisted path.
- `SOURCE.file_hash` stores a SHA-256 digest.
- `SOURCE.file_size_bytes` stores the original file size.
| Column | Type | Notes |
| --- | --- | --- |
| `id` | UUID PK | Stable key |
| `document_id` | UUID FK | -> `document.id` |
| `person_id` | UUID FK | -> `person.id` |
| `role_id` | UUID FK | -> `person_role.id` |
| `status` | TEXT | `pending`, `accepted`, `rejected` |
| `confidence` | FLOAT NULL | Optional score |
| `source_mechanism` | TEXT NULL | For example `rule`, `llm` |
| `evidence_ref` | TEXT NULL | Pointer or excerpt ID |
| `evidence_span` | JSON NULL | Optional text span payload |
| `note` | TEXT NULL | Reviewer note |
| `created_at` | TIMESTAMPTZ | Suggestion creation time |
| `reviewed_at` | TIMESTAMPTZ NULL | Decision time |
| `reviewed_by` | TEXT NULL | Operator identifier |
### Page Ordering and Revisions
Constraints:
- `SOURCE.page_number` dictates page ordering within a document.
- `SOURCE.raw_transcription` remains immutable machine output.
- `SOURCE.revised_text` stores human edits and is the preferred display value when present.
- `UNIQUE(document_id, person_id, role_id, status)` with policy for multiple pending rows defined in service layer.
- Optional stricter rule: one active pending suggestion per `(document_id, person_id, role_id)`.
### Document-Person Role Governance
#### `role_exclusivity`
- Documents support zero, one, or many people per relationship role.
- Relationship roles are defined by `PERSON_ROLE` rather than hardcoded columns.
- `DOCUMENT_PERSON` must be unique for `(document_id, person_id, role_id)`.
- Configured exclusive role pairs from `ROLE_EXCLUSIVITY` cannot coexist for the same `(document_id, person_id)`.
- Initial exclusivity seed blocks `author` and `recipient` for the same person-document pair.
| Column | Type | Notes |
| --- | --- | --- |
| `id` | UUID PK | Stable key |
| `left_role_id` | UUID FK | -> `person_role.id` |
| `right_role_id` | UUID FK | -> `person_role.id` |
| `created_at` | TIMESTAMPTZ | Audit timestamp |
### Document Type Governance
Constraints:
- Every document type is defined by `DOCUMENT_TYPE`.
- `DOCUMENT_TYPE.code` is a stable machine identifier.
- `DOCUMENT_TYPE.label` is mutable display text.
- Inactive types remain valid for historical rows but should be excluded from default selection UIs.
- Canonical ordering rule to avoid duplicate pairs (`left_role_id < right_role_id` enforced in service/DB).
- `UNIQUE(left_role_id, right_role_id)`
## Constraint Summary
Initial seed:
- Exclusivity pair: (`author`, `recipient`)
## Deferred Alternative (Not Selected for V4)
An enum-based shared table model was considered but is intentionally not selected for V4 because it couples canonical and provisional states in one table and increases invariant complexity.
## Assertion-State Semantics
- `asserted`: canonical relationship used for document/person metadata and business logic.
- `suggested`: non-canonical proposal requiring explicit review.
- Accept action:
- creates asserted link (or confirms existing),
- marks suggestion `accepted`.
- Reject action:
- marks suggestion `rejected`.
## Exclusivity Enforcement
Policy target:
- For a single `(document_id, person_id)`, disallow coexistence of role pairs configured as exclusive.
Enforcement layers:
1. Service-level pre-check for clear API errors.
2. Database-level guard where feasible (constraints/triggers or deterministic write path).
Initial configured rule:
- `author` and `recipient` are exclusive.
Enforcement semantics:
- Exclusivity is enforced for asserted links.
- Pending suggestions may exist even if they would conflict when asserted.
- Accepting a suggestion must run exclusivity checks and fail deterministically on conflict.
## Migration From V3
### Data Mapping
- Existing V3 `document_person` rows map to V4 `asserted` semantics.
- Existing V3 role values:
- `author` -> role `author`
- `recipient` -> role `recipient`
### Backfill Steps
1. Seed role rows (`author`, `recipient`, `mentioned`) if using Option A.
2. Migrate current links into asserted table/state.
3. Run conflict scan for exclusivity violations.
4. Apply deterministic conflict policy for any violations.
5. Enable hard enforcement after data passes validation.
- `DOCUMENT_TYPE.code` is unique.
- `PERSON_ROLE.code` is unique.
- `DOCUMENT_PERSON(document_id, person_id, role_id)` is unique.
- `ROLE_EXCLUSIVITY(left_role_id, right_role_id)` is unique.
- `ROLE_EXCLUSIVITY` must use canonical ordering to avoid duplicate mirrored pairs.
## Indexing Guidance
Recommended indexes:
- `document(document_type_id)`
- `document_person(document_id)`
- `document_person(person_id)`
- `document_person(role_id)` (Option A) or `document_person(role)` (Option B)
- `document_person_suggestion(document_id, status)` (Option A)
- `document_person_suggestion(person_id, status)` (Option A)
## Selection Rationale
V4 selects role registry plus separate suggestion storage for clearer provenance boundaries, cleaner lifecycle transitions, and long-term extensibility.
## Document Type Registry Model (Selected for V4)
V4 applies the same registry governance pattern to document classification.
### Tables
#### `document_type`
| Column | Type | Notes |
| --- | --- | --- |
| `id` | UUID PK | Stable key |
| `code` | TEXT UNIQUE | Canonical type code, for example `letter`, `diary`, `book`, `postcard` |
| `label` | TEXT | UI display label |
| `is_active` | BOOLEAN | Soft-enable/disable type |
| `sort_order` | INTEGER NULL | Optional UI ordering |
| `created_at` | TIMESTAMPTZ | Audit timestamp |
| `updated_at` | TIMESTAMPTZ | Audit timestamp |
#### `document` update
| Column | Type | Notes |
| --- | --- | --- |
| `document_type_id` | UUID FK NULL | -> `document_type.id` |
### Constraints and Governance
- `document_type.code` must be stable and unique.
- `document_type.label` may change without changing canonical type identity.
- Inactive types remain valid for historical records but are excluded from default create/edit selectors.
### Initial Seeds
- Seed baseline type codes from current V3 usage set (for example `letter`, `diary`, `book`, `postcard`, `record`, `memo`) and refine labels as needed.
### Migration and Normalization (Small Corpus)
1. Seed canonical `document_type` rows.
2. Manually assign each existing document (10 total) to a canonical type via `document_type_id`.
3. Resolve any outlier values directly during this one-time pass.
4. Enforce registry-backed write validation after manual assignment is complete.
### Indexing Guidance (Document Type)
- `document_type(code)` unique index.
- `document(document_type_id)` index.
- `document_person(role_id)`
- `role_exclusivity(left_role_id, right_role_id)`
- `source(document_id, page_number)`
- `job(document_id, status)`
- `job_source(job_id)`
- `job_source(source_id)`
## Related Local References
- [V4 Scope Boundary](scope_boundary_v4.md)
- [V4 Requirements](requirements_v4.md)
- [V3 Schema](../schema_v3.md)
- [System Overview](index_v4.md)
- [System Architecture](architecture_v4.md)
- [System Requirements](requirements_v4.md)
- [Error Handling Policy](error_handling_v4.md)
docs/ver4/scope_boundary_v4.md
+50 -89
View File
@@ -1,130 +1,91 @@
# V4 Scope Boundary (Version 4)
# V4 Scope Boundary
This document defines what is and is not included in Version 4 for expanding `Document`-`Person` relationships.
This document defines the scope for the transition from the current repository state to the Version 4 project definition.
## Purpose
Create a clear implementation boundary before updating full V4 architecture, requirements, schema, and plan documents.
Define what this revision includes, what it intentionally excludes, and what migration rules govern the transition work.
## Why This Is V4 (Not V3.1)
## In Scope
V4 is required because the change is semantic and cross-cutting:
### 1. Relationship Model
- Expands relationship meaning beyond fixed `author`/`recipient`.
- Introduces a suggestion lifecycle (`suggested` vs `asserted`).
- Introduces policy constraints (role exclusivity matrix).
- Impacts persistence, service contracts, API behavior, UI workflows, and test strategy.
- Extensible role taxonomy for document-person relationships.
- Many-to-many document-person links with many people per role.
- Deterministic exclusivity policy for configured role pairs.
- Set-based add/remove synchronization for document-person updates.
A V3.1 patch would only be appropriate for non-semantic quality-of-life fixes inside existing role semantics.
### 2. Document Type Governance
## In Scope for V4
- Registry-driven `DocumentType` model with stable codes and controlled selection.
- Minimal rollout for the current corpus with no alias helper table.
### 1) Relationship Semantics
### 3. UI and API Behavior
- Role taxonomy becomes extensible (not hardcoded to only two role values).
- A `Document` can link to many `Person` records per role.
- Relationship states are explicit:
- `asserted`: human-confirmed canonical link.
- `suggested`: machine- or heuristic-proposed link pending review.
- Grouped role links on document and person views.
- Multi-role relationship editing on document create/edit flows.
- Role-aware API retrieval and write behavior.
- Additive API evolution with explicit deprecations.
### 2) Policy Rules
### 4. Verification
- Enforce role exclusivity for the same `(document, person)` pair where configured.
- Initial policy decision:
- `author` + `recipient` are mutually exclusive.
- `mentioned` may coexist with other roles.
- Tests for many-per-role behavior.
- Tests for exclusivity enforcement.
- Tests for set-based relationship mutation behavior.
- Tests for document and person delete/link cleanup regressions.
### 3) Persistence and Contracts
## Out of Scope
- Evolve persistence model to support:
- extensible roles,
- suggestion lifecycle metadata,
- deterministic conflict handling.
- Replace single-value relationship mutation patterns with set-based sync behavior.
### 4) Human-in-the-Loop Workflow
- Add assisted suggestion review flow:
- list suggestions,
- accept,
- reject,
- promote accepted suggestion to asserted link.
- No silent auto-promotion from suggestion to asserted.
### 5) UI and API Behavior
- Document create/edit/detail workflows support multi-person per role.
- UI surfaces grouped role links and suggestion status.
- API supports role-aware retrieval and suggestion lifecycle operations.
### 6) Verification
- Add tests for:
- many-per-role behavior,
- exclusivity enforcement,
- suggestion lifecycle,
- migration/conflict detection,
- regression on delete/link cleanup behavior.
## Out of Scope for V4
- Automatic acceptance of suggested links without human review.
- Full entity resolution/identity merge pipeline across all `Person` records.
- Historical provenance graph redesign beyond relationship-level evidence fields.
- Large-scale NLP research features unrelated to document-person linking.
- Changes to core transcription execution model (`Job`, `JobSource`) except where needed to expose suggestion evidence inputs.
- Suggested versus asserted relationship states.
- Suggestion storage, review, acceptance, or rejection workflows.
- Automatic relationship extraction or recommendation features.
- Full entity resolution or identity merge across all people.
- Automated semantic document type classification.
- Redesign of the core transcription execution model.
## Locked Design Decisions
### A) Role Extensibility Mechanism
### A. Role Extensibility Mechanism
- Adopt role registry tables (data-driven roles).
- Do not use enum-first role expansion for V4.
- Use registry tables for relationship roles.
### B) Suggestion Storage Model
### B. Exclusivity Matrix Baseline
- Adopt a separate `document_person_suggestion` table.
- Keep `document_person` focused on canonical asserted links.
- Revisit only if operational complexity proves materially higher than expected.
### C) Exclusivity Matrix Baseline
- `author` + `recipient` are mutually exclusive for the same `(document, person)` pair.
- `author` and `recipient` are mutually exclusive for the same `(document, person)` pair.
- `mentioned` remains non-exclusive.
- Hard exclusivity enforcement applies to asserted links.
- Suggestions may be stored even if they would conflict at assert time; promotion to asserted must enforce exclusivity.
### D) API Compatibility Strategy
### C. API Compatibility Strategy
- Use additive API evolution in V4.
- In development mode, the current revision is authoritative; long-lived legacy compatibility layers are not required.
- Deprecations should be explicit and short-lived, with removals performed in subsequent revisions.
- Use additive API evolution.
- In development mode, the current revision is authoritative.
- Deprecations should be explicit and short-lived.
### E) Document Type Rollout Strategy
### D. Document Type Rollout Strategy
- Use a minimal registry rollout for the current corpus: no `document_type_alias` helper table.
- Perform a one-time manual mapping of existing document types to canonical registry types.
- Use a minimal registry rollout for the current corpus.
- Do not introduce a `document_type_alias` helper table.
### F) Suggestion Generation Strategy (Still Open)
### E. Database Change Policy
- Option A: deterministic rules/NLP over transcribed text.
- Option B: LLM extraction with confidence/evidence spans.
- Future schema changes are non-destructive by default.
- Exception: `document_type` text may be replaced by `document_type_id` without migrating the legacy text values.
- Exception: `document_person` links may be recreated manually.
## Compatibility and Rollout
- Existing V3 author links migrate to `asserted` behavior.
- Backfill validation identifies policy conflicts before constraints are enforced.
- Preserve existing V3 core behavior where unaffected by role/suggestion evolution.
- Preserve existing repository behavior where unaffected by the V4 scope.
- Treat scope boundary and implementation plan as the only transition documents.
- Treat core V4 documents as the authoritative project definition once rewritten.
## Exit Criteria for Scope Freeze
V4 scope is considered frozen when:
- Suggestion generation strategy is chosen.
- Suggestion acceptance workflow details are approved.
- Relationship model, exclusivity rules, and document-type governance are approved.
- Additive API change list and deprecation schedule are approved.
- Migration exceptions are explicitly acknowledged.
## Core V4 Documents (Current Set)
## Core V4 Documents
1. `docs/ver4/index_v4.md`
2. `docs/ver4/requirements_v4.md`