bbchops/transcription
1
0
Fork 0
generated from john/python-template
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

Revised and simplified V4 Plan and core documents.

Browse Source
This commit is contained in:
Jim Lancaster
2026-08-10 10:53:13 -05:00
parent 9b4d6f0340
commit 4b3baf5a3e
7 changed files with 495 additions and 578 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/ver4/architecture_v4.md
+105 -95
View File
@@ -1,137 +1,147 @@
# System Architecture (Version 4)
This document describes the V4 architecture changes for expanded `Document`-`Person` relationship management.
V4 extends V3 with role extensibility, assisted suggestion review, and role-policy enforcement while preserving the existing transcription execution topology.
This document describes the production architecture of the document transcription system.
## Architecture Objectives
- Keep V3 transcription pipeline behavior stable unless relationship evidence extraction requires additive integration.
- Support many people per role for each document.
- Support extensible relationship roles without hardcoding UI and API behavior to two values.
- Support non-canonical suggestion intake with explicit human promotion to canonical asserted links.
- Enforce role exclusivity policy consistently at service and persistence boundaries.
- Preserve auditability for suggestion provenance and review actions.
- Preserve original source material and immutable machine transcription output.
- Support batching one or more images into ordered multi-page documents.
- Capture complete submission-time prompt provenance and per-page provider response evidence.
- Execute page transcription concurrently with bounded `asyncio` workers.
- Maintain relational portability across SQLite and PostgreSQL.
- Keep operator workflows cross-platform and Python-driven.
- Support many-to-many document-person relationships with extensible roles.
- Support registry-driven document type classification.
- Enforce relationship-role exclusivity rules consistently across UI, API, and persistence boundaries.
## Runtime Topology
V4 keeps the existing runtime shape:
The runtime operates as an asynchronous Python application:
- FastAPI + NiceGUI web app process.
- Async service layer with SQLModel/SQLAlchemy persistence.
- Existing worker execution path for transcription jobs.
Additive V4 components:
- Relationship policy evaluator (exclusivity checks).
- Suggestion lifecycle service (propose/list/accept/reject).
- Optional suggestion extraction adapter (rule/NLP or LLM-backed).
- FastAPI + NiceGUI web application process.
- In-process `asyncio` worker engine for transcription execution.
- Relational persistence via SQLModel / SQLAlchemy.
- Pydantic V2 validation across API payloads, prompt configuration, and structured metadata.
^^^mermaid
flowchart LR
U[Browser User] --> UI[NiceGUI Pages]
UI --> API[FastAPI Routes]
API --> DS[Document Service]
API --> SS[Suggestion Service]
DS --> PE[Policy Evaluator]
SS --> PE
DS --> DB[(Relational DB)]
SS --> DB
W[Async Worker Engine] --> DB
W --> E[Optional Suggestion Extraction]
E --> SS
U[Browser User] --> A[FastAPI + NiceGUI App]
A --> W[Asyncio Worker Engine]
A --> DB[(Relational DB)]
W --> P[Vision Provider APIs]
W --> DB
^^^
## Layer Responsibilities
## Lifecycle Ownership
### UI Layer (`src/transcription/ui/**`)
Application lifespan owns runtime setup and teardown:
- Render per-role grouped relationships for document and person views.
- Provide multi-select role assignment controls in document create/edit flows.
- Provide suggestion review controls (accept/reject) for pending suggestions.
- Surface policy conflict errors from API/service layer clearly.
- Initialize logging, settings, directories, and prompt configuration.
- Manage asynchronous database engine connection pools.
- Execute database bootstrap or migrations.
- Recover stale or interrupted jobs on startup.
- Manage graceful shutdown of active background tasks.
### API Layer (`src/transcription/api/**`)
## Layered Module Structure
- Expose role-aware and state-aware read contracts.
- Expose suggestion lifecycle write contracts.
- Return deterministic validation/conflict errors for exclusivity and duplicate semantics.
- Evolve endpoints additively, with explicit deprecations and short-lived transition windows.
### Interface Layer
### Service Layer (`src/transcription/services/**`)
- `src/transcription/ui/**`
- `src/transcription/api/**`
- Implement set-based relationship sync (delta add/remove) to avoid destructive replacement behavior.
- Apply policy evaluator before persistence commits.
- Orchestrate suggestion acceptance/rejection transitions.
- Ensure accepted suggestions result in asserted link creation/confirmation.
Responsibilities:
### Policy Evaluator (New logical component)
- Render document, source, person, job, and classification views.
- Accept user input for uploads, editing, linking, and revisions.
- Present structured validation and conflict feedback.
- Evaluate role exclusivity matrix for `(document_id, person_id)` writes.
- Provide deterministic conflict reason payloads suitable for UI/API display.
- Stay stateless and reusable across create/update/sync code paths.
### Application and Async Worker Layer
### Persistence Layer (`src/transcription/db/**`)
- `src/transcription/services/workflows.py`
- `src/transcription/worker.py`
- Store asserted links and suggestion records according to selected schema option.
- Persist provenance metadata for suggestion records.
- Enforce uniqueness and support performant role/state filtering.
Responsibilities:
## Core V4 Workflows
- Orchestrate uploads, job creation, and status transitions.
- Execute per-page provider calls through bounded concurrency.
- Persist page-level outcomes and update aggregate job state.
### 1) Manual Relationship Management
### Domain and Service Layer
1. User opens document edit view.
2. UI loads asserted links grouped by role.
3. User adds/removes people per role.
4. Service computes delta and runs policy checks.
5. Persistence applies adds/removes atomically.
- `src/transcription/db/models.py`
- `src/transcription/services/*.py`
### 2) Assisted Suggestion Review
Responsibilities:
1. Suggestion records are created (`pending`) by extraction logic or manual propose action.
2. UI displays pending suggestions with evidence metadata.
3. User accepts or rejects each suggestion.
4. Accept path creates/confirms asserted relationship and marks suggestion `accepted`.
5. Reject path marks suggestion `rejected`.
- Manage transactional operations for documents, people, types, links, sources, jobs, and job sources.
- Apply deterministic conflict handling for relationship-role writes.
- Use set-based synchronization for many-to-many relationship updates.
- Resolve and validate registry-backed document types.
### 3) Exclusivity Conflict Handling
### Infrastructure Layer
1. Incoming write attempts role assignment.
2. Policy evaluator checks role pair conflicts for same `(document_id, person_id)`.
3. If conflict exists, write is rejected with structured conflict details.
4. UI presents actionable message without partial updates.
- `src/transcription/db/**`
- `src/transcription/providers/**`
## Invariants
Responsibilities:
- Suggested links are never canonical until accepted.
- Asserted links must satisfy exclusivity rules.
- Relationship mutations are set-based and deterministic.
- Relationship views remain consistent between document detail and person detail pages.
- Existing V3 links remain valid under asserted semantics after migration.
- Provide async database sessions and engine configuration.
- Provide provider adapters for vision model execution.
## Backward Compatibility
## Core Workflows
- V3 author/recipient links are interpreted as asserted links in V4.
- Existing document and person flows continue to function where unaffected by new role/state dimensions.
- During development revisions, the current API contract is authoritative; long-lived legacy runtime compatibility layers are not required.
### 1. Multi-Page Transcription
## Observability and Auditability
1. User uploads one or more images for a `Document`.
2. System stores files, hashes them, creates ordered `Source` rows, and creates a `Job`.
3. Worker claims the job, marks it `processing`, and executes page calls concurrently.
4. Each page writes a `JobSource` result with raw output, metadata, and full provider response evidence.
5. Aggregate status becomes `completed`, `partial_success`, or `failed`.
- Persist suggestion provenance fields sufficient for operator review.
- Record review decision outcomes (`accepted`/`rejected`) with timestamps and reviewer identity where available.
- Emit structured service-level logs for exclusivity conflicts and review actions.
### 2. Document-Person Relationship Management
## Test Strategy Additions
1. User opens a document or person edit flow.
2. UI loads existing links grouped by role.
3. User adds or removes people within one or more roles.
4. Service computes add/remove deltas rather than replacing all links blindly.
5. Conflict checks enforce exclusivity policy before persistence commits.
- Service tests for delta sync and exclusivity matrix enforcement.
- API tests for role/state filtering and suggestion transitions.
- UI tests for multi-role selection and suggestion review interactions.
- Migration tests for V3-to-V4 asserted mapping and conflict scans.
### 3. Document Type Management
1. User selects a registry-backed document type for a document.
2. Service resolves the stable type code or id.
3. Persistence stores the `document_type_id` reference.
4. Inactive types remain valid for historical rows but are excluded from default selectors.
## Domain Invariants
- `Source.raw_transcription` stores immutable machine output.
- Human corrections occur only in `Source.revised_text`.
- Prompt and parameter provenance is frozen on `Job` at submission time.
- Provider output evidence is stored on `JobSource` for each page execution.
- `DocumentPerson` links are unique for `(document_id, person_id, role_id)`.
- Configured exclusive role pairs cannot coexist for the same `(document_id, person_id)`.
- Relationship mutations are deterministic and set-based.
- `DocumentType.code` is stable; `DocumentType.label` may evolve.
## Data Model Summary
- `Document` has one `DocumentType`, many `Source` pages, many `Job` runs, and many `Person` records through `DocumentPerson`.
- `Source` belongs to one `Document` and may participate in many `JobSource` executions.
- `Job` has many `JobSource` rows.
- `PersonRole` defines available relationship roles.
- `RoleExclusivity` defines role pairs that cannot coexist for the same document-person pair.
## Test Strategy
- Unit tests for models, validation, hashing, and registry resolution.
- Service tests for CRUD, set-based sync, and exclusivity enforcement.
- Async workflow tests for page isolation, partial failure handling, and stored evidence.
- UI integration tests for multi-page rendering, role grouping, and document type selection.
## Related Local References
- [V4 Scope Boundary](scope_boundary_v4.md)
- [V4 Requirements](requirements_v4.md)
- [V4 Schema](schema_v4.md)
- [V3 Architecture](../architecture_v3.md)
- [System Overview](index_v4.md)
- [System Requirements](requirements_v4.md)
- [Data Model](schema_v4.md)
- [Error Handling Policy](error_handling_v4.md)