V4.4 Complete

This commit is contained in:
Jim Lancaster
2026-08-15 14:30:33 -05:00
parent 63373bf24d
commit 7db4df1729
32 changed files with 1529 additions and 716 deletions
+29 -6
View File
@@ -11,10 +11,11 @@ Documents manages the archival record for each historical artifact independently
| `/documents` | Searchable archival Document list. |
| `/documents/new` | Create a Document. |
| `/documents/{document_id}` | View one Document and its related records. |
| `/documents/{document_id}/edit` | Edit metadata and people-by-role links. |
| `/documents/{document_id}/edit` | Edit metadata and the complete Linked People set. |
| `/documents/{document_id}/delete` | Confirm or block deletion. |
| `/documents/{document_id}/jobs` | Show Jobs belonging to the Document. |
| `/documents/{document_id}/sources` | Redirect to the Document-filtered Sources list. |
| `/documents/{document_id}/print` | Preview and browser-print the persisted Document. |
## List Behavior
@@ -42,7 +43,7 @@ Optional:
- Document location.
- Archive identifier.
- Notes.
- Multiple people for every configured Person Role.
- Linked People, with exactly one Person Role per linked Person.
Rules:
@@ -53,6 +54,11 @@ Rules:
- An invalid requested Person produces a warning rather than a broken form.
- `return_to=jobs_new` returns a successful create to Job creation with the new Document selected.
- Edit includes active and inactive Document Types so historical values remain maintainable.
- One Linked People table contains Select, Person, and Role columns.
- Add and Edit use an inline Person/Role editor; Save, Cancel, and Delete change staged UI state only.
- A Person may appear once per Document regardless of role.
- Existing inactive-role links remain visible; only active roles may be newly assigned.
- Document fields and the complete staged link set commit atomically on the main save.
- Save success returns to Document Detail.
## Detail Behavior
@@ -63,9 +69,20 @@ Rules:
- System Logistics shows created and updated timestamps.
- Related People are grouped by role and link to Person Detail.
- **Sources & Pipeline Jobs** shows counts and actions for filtered Sources, Document Jobs, and adding a Job.
- **Edit Document** and **Delete** are available from the header.
- **Edit Document**, **Print**, and **Delete** are available from the header.
- Invalid IDs and missing Documents produce explicit states without rendering a partial page.
## Print Behavior
- Print opens a dedicated preview for persisted Document data.
- **Facsimile** places each Source image beside its current transcription and starts every Source on a new printed sheet.
- **Text only** omits images, joins single line breaks inside paragraphs, and preserves blank-line paragraph boundaries.
- Non-null revised text takes precedence over raw transcription, including an intentionally empty revision.
- Archival metadata resolves Author through the hidden built-in semantic identity, not its mutable label.
- Job metadata uses one oldest-to-newest column per Job and ends with Status.
- Stored text is escaped and Source media uses record-validated application URLs rather than local file paths.
- Printing uses the browser print dialog; server-generated PDFs are not provided.
## Document Jobs Behavior
- The page lists the Document's Jobs newest first with status and Job ID.
@@ -84,10 +101,12 @@ Rules:
- List columns, alignment, search, sorting, date fallback, and row navigation match this contract.
- Create/edit enforce name, registered type, and valid exact-date input.
- Multiple people can be selected independently for each configured role.
- Linked People staging enforces one role and one row per Person.
- Document and Linked People writes never partially commit.
- Person-first Document creation preselects the requested Person as author.
- Detail links people, Sources, and Jobs to the correct records.
- Delete never removes a Document with Source or Job dependencies.
- Both print formats preserve the frozen content, ordering, text-precedence, and safety contracts.
- Service failures use the shared error presenter and never report false success.
## Implementation Anchors
@@ -96,10 +115,14 @@ Rules:
- `src/transcription/ui/components/table/documents.py`
- `src/transcription/services/documents.py`
- `src/transcription/services/people.py`
- `src/transcription/services/workflows.py`
- `src/transcription/ui/components/linked_people.py`
- `src/transcription/ui/pages/print_preview_page.py`
- `src/transcription/api/v4_print.py`
- `tests/ui/test_documents_page.py`
- `tests/services/test_document_service.py`
## Known Limitations and Deferred Work
- Document creation persists the Document before adding relationship links; a later link failure is surfaced but is not currently one atomic write.
- Source ordering controls are deferred to the [draft V4.3 scope](../../ver4.3/scope_boundary_v4_3.md).
- Source page ordering remains read-only in V4.4.
- Printing other entities, batch printing, and server-side export formats are deferred.
+27 -17
View File
@@ -10,8 +10,8 @@ This document describes the production architecture of the document transcriptio
- Execute page transcription concurrently with bounded `asyncio` workers.
- Maintain relational portability across SQLite and PostgreSQL.
- Keep operator workflows cross-platform and Python-driven.
- Support many-to-many document-person relationships with extensible roles.
- Support registry-driven document type classification.
- Support one role-bearing link per Person and Document through an extensible role registry.
- Support registry-driven document classification with protected semantic built-ins.
## Core Capabilities
@@ -20,8 +20,8 @@ This document describes the production architecture of the document transcriptio
- Preserve original source files with SHA-256 digests and byte sizes.
- Freeze prompt text, prompt hash, model, and explicitly configured sampling parameters on each `Job`.
- Preserve page-level machine output, normalized metadata, and an SDK-serialized OpenRouter response snapshot on `JobSource`.
- Organize historical `Person` records through many-to-many Document relationships and extensible roles.
- Classify Documents through a UUID-identified registry with unique labels.
- Organize historical `Person` records through UUID-identified Document links and extensible roles.
- Classify Documents through a UUID-identified registry with hidden semantic built-ins and unique labels.
- Maintain human revision separately from machine-generated text.
- Isolate page failures so multi-page jobs can complete with partial success.
- Operate across supported platforms through Python-based application and maintenance tooling.
@@ -110,8 +110,8 @@ Responsibilities:
- Jobs own job lifecycle state and transitions.
- People own person records, relationship roles, document-person links, and portrait media.
- Apply deterministic conflict handling for relationship-role writes.
- Use set-based synchronization for many-to-many relationship updates.
- Resolve and validate registry-backed document types by UUID.
- Synchronize each Document's complete Person link set in the same transaction as Document fields.
- Resolve and validate registry records by UUID; use hidden semantic keys only for application-owned built-in behavior.
### Source Media Policy
@@ -146,11 +146,11 @@ Responsibilities:
### 2. Document-Person Relationship Management
1. User opens a document or person edit flow.
2. UI loads existing links grouped by role.
3. User adds or removes people within one or more roles.
4. Service computes add/remove deltas rather than replacing all links blindly.
5. Conflict checks enforce uniqueness and deterministic write semantics before persistence commits.
1. User opens Document Create or Edit.
2. UI loads one Linked People table containing Person and Role.
3. Add, Edit, and Delete operations change staged UI state only.
4. Service validates the complete desired set and computes deterministic add, update, and remove deltas.
5. Document fields and links commit once in one transaction; any failure leaves both unchanged.
### 3. Document Type Management
@@ -159,6 +159,14 @@ Responsibilities:
3. Persistence stores the `document_type_id` reference.
4. Inactive types remain valid for historical rows but are excluded from default selectors.
### 4. Document Printing
1. User opens Print from persisted Document Detail.
2. Service builds a safe projection containing archival metadata, semantic Author links, ordered Sources, current text,
and oldest-to-newest Job metadata.
3. The preview renders Facsimile or Text-only HTML without exposing local file paths.
4. An explicit action opens the browser print dialog; browser Save as PDF remains available.
## V4 Domain Rules
- `JobSource.raw_transcription` preserves page output for its Job execution.
@@ -169,23 +177,25 @@ Responsibilities:
- Every V4.2 provider call appends a distinct `ExecutionAttempt`; retries never rewrite earlier attempts.
- Exact response bytes identify the OpenRouter HTTP boundary and are not labeled as native upstream-provider JSON.
- Generic `ProcessingArtifact` records use versioned schemas, digests, and one inline or external content location.
- `DocumentPerson` links are unique for `(document_id, person_id, role_id)`.
- Relationship mutations are deterministic and set-based.
- `DocumentType.id` is canonical identity; its unique label may evolve.
- `DocumentPerson` links are unique for `(document_id, person_id)` and require one `role_id`.
- Relationship mutations are deterministic, set-based, and atomic with Document writes.
- `DocumentType.id` and `PersonRole.id` are canonical relationship identities; unique labels may evolve.
- Nullable immutable `semantic_key` values identify protected application-defined built-ins and are never public selectors.
- Current printable text uses non-null `Source.revised_text`; otherwise it uses `Source.raw_transcription`.
## Data Model Summary
- `Document` has one `DocumentType`, many `Source` pages, many `Job` runs, and many `Person` records through `DocumentPerson`.
- `Source` belongs to one `Document` and may participate in many `JobSource` executions.
- `Job` has many `JobSource` rows.
- `PersonRole` defines available relationship roles.
- `PersonRole` defines available relationship roles; `DocumentType` and `PersonRole` may carry hidden semantic identity.
## Test Strategy
- Unit tests for models, validation, hashing, and registry resolution.
- Service tests for CRUD, set-based sync, uniqueness conflicts, and deterministic relationship writes.
- Service tests for registry protection, atomic link synchronization, uniqueness conflicts, and print projections.
- Async workflow tests for page isolation, partial failure handling, and stored evidence.
- UI integration tests for multi-page rendering, role grouping, and document type selection.
- UI integration tests for Linked People staging, registry selection, and safe print rendering.
## Related Local References
+16 -10
View File
@@ -11,36 +11,42 @@ This document defines the baseline requirements for the document transcription s
| REQ-2 | Functional | Process page transcription asynchronously using an `asyncio` worker pool bounded by rate limits. | test |
| REQ-3 | Functional | Persist submission-time request provenance and accurately labeled page-level SDK evidence; V4.2 adds exact OpenRouter-boundary transport evidence for new attempts. | test |
| REQ-4 | Functional | Support job states `queued`, `processing`, `completed`, `partial_success`, and `failed`, plus page states `pending`, `transcribed`, and `failed`. | inspection |
| REQ-5 | Functional | Allow users to manage historical `Person` records and link multiple people per role to a `Document`. | test |
| REQ-5 | Functional | Allow users to manage historical `Person` records and link each Person to a Document once with exactly one role. | test |
| REQ-6 | Functional | Support an extensible role taxonomy for document-person relationships. | inspection |
| REQ-7 | Policy Constraint | Enforce deterministic relationship-role writes with uniqueness on `(document_id, person_id, role_id)` and explicit conflict responses for invalid duplicate link attempts. | test |
| REQ-7 | Policy Constraint | Enforce deterministic relationship-role writes with uniqueness on `(document_id, person_id)` and explicit conflict responses for duplicate Person links. | test |
| REQ-8 | Functional | Use set-based synchronization for document-person mutations so updates add and remove only the intended links. | test |
| REQ-9 | Functional | Maintain immutable machine output on `Source.raw_transcription` while permitting inline human edits on `Source.revised_text`. | test |
| REQ-10 | Functional | Support a UUID-identified `DocumentType` taxonomy with unique user-facing labels and active/inactive lifecycle control. | test |
| REQ-11 | Data Constraint | Store `Document` type as a controlled reference to `DocumentType`. | test |
| REQ-12 | Interface | Render multi-page transcriptions sequentially by `page_number` with document, people, and document-type metadata. | demonstration |
| REQ-13 | Interface | Document create/edit UI must support selecting multiple people per role and selecting an active document type from the registry. | demonstration |
| REQ-13 | Interface | Document create/edit UI must provide one staged Linked People table and select active registry entries by UUID and label. | demonstration |
| REQ-14 | API Constraint | Expose additive, role-aware retrieval and write behavior for document-person links and UUID-based selection for document types. | test |
| REQ-15 | Data Constraint | Calculate and store cryptographic file hashes (SHA-256) and file sizes for uploaded source images. | test |
| REQ-16 | Data Constraint | Preserve a portable relational model across supported backends using SQLModel, SQLAlchemy, SQLite, and PostgreSQL. | inspection |
| REQ-17 | Reliability | Ensure delete and update flows for documents, people, and relationship links remain deterministic and safe. | test |
| REQ-18 | Operations Constraint | Keep canonical development, testing, restore, and recovery workflows OS-independent; for AI-run unit tests, require a pre-test backup of `./data` and an always-shown post-success confirmation prompt before any restore action. | inspection |
| REQ-19 | Quality | Provide automated coverage for async transcription workflows, relationship-role enforcement, document-type selection, and regression behavior. | test |
| REQ-20 | Data Constraint | Permit hidden immutable semantic keys only on protected built-in Document Types and Person Roles while retaining UUID as relationship identity. | test |
| REQ-21 | Reliability | Persist Document fields and their complete Linked People set atomically. | test |
| REQ-22 | Interface | Provide safe browser-native Facsimile and Text-only print views from persisted Document Detail. | demonstration |
| REQ-23 | Security | Escape stored print text and serve Source images through record-validated application routes without disclosing local paths. | test |
| REQ-24 | Functional | Print current human-preferred Source text, semantic Author metadata, deterministic Source order, and oldest-to-newest Job metadata. | test |
## Clarifying Constraints
1. `DocumentType.id` is its sole identity; labels are unique ignoring case and surrounding whitespace.
2. `PersonRole.code` is a stable machine identifier; `PersonRole.label` may evolve.
1. `DocumentType.id` and `PersonRole.id` are their public and relationship identities; labels are unique ignoring case and surrounding whitespace.
2. Nullable `semantic_key` values identify protected application built-ins, remain internal, and never change.
3. Relationship-write policy and conflict handling must be consistent across UI, API, services, and persistence.
4. Many-per-role behavior is required for document-person links.
5. Relationship conflicts must fail deterministically without partial mutation.
4. One Person may appear only once per Document and every link has exactly one role.
5. Relationship conflicts must fail deterministically without partial Document or link mutation.
6. Source page reordering and server-generated PDF files remain outside this revision.
## Element Satisfaction Mapping
- UI (NiceGUI): Satisfies REQ-0, REQ-1, REQ-5, REQ-9, REQ-12, REQ-13.
- API (FastAPI): Satisfies REQ-1, REQ-4, REQ-5, REQ-7, REQ-8, REQ-14.
- UI (NiceGUI): Satisfies REQ-0, REQ-1, REQ-5, REQ-9, REQ-12, REQ-13, REQ-22, REQ-24.
- API (FastAPI): Satisfies REQ-1, REQ-4, REQ-5, REQ-7, REQ-8, REQ-14, REQ-23.
- Worker (`asyncio`): Satisfies REQ-2, REQ-3, REQ-4.
- Persistence (SQLModel / SQLAlchemy): Satisfies REQ-3, REQ-9, REQ-10, REQ-11, REQ-15, REQ-16, REQ-17.
- Persistence (SQLModel / SQLAlchemy): Satisfies REQ-3, REQ-7, REQ-9, REQ-10, REQ-11, REQ-15, REQ-16, REQ-17, REQ-20, REQ-21.
- Test Suite: Verifies all test-marked requirements and satisfies REQ-19.
## Related Local References
+23 -7
View File
@@ -8,6 +8,7 @@ This document defines the relational schema for the document transcription syste
erDiagram
DOCUMENT_TYPE {
UUID id PK
TEXT semantic_key UK
TEXT label
TEXT normalized_label
BOOLEAN is_active
@@ -17,8 +18,9 @@ TIMESTAMPTZ updated_at
PERSON_ROLE {
UUID id PK
TEXT code
TEXT semantic_key UK
TEXT label
TEXT normalized_label
BOOLEAN is_active
TIMESTAMPTZ created_at
TIMESTAMPTZ updated_at
@@ -196,17 +198,29 @@ EXECUTION_ATTEMPT ||--o{ PROCESSING_ARTIFACT : produces
- `SOURCE.raw_transcription` remains immutable machine output.
- `SOURCE.revised_text` stores human edits and is the preferred display value when present.
### Semantic Registry Governance
- `DOCUMENT_TYPE.id` and `PERSON_ROLE.id` are the only relationship and public API identities.
- Nullable unique `semantic_key` values identify application-defined built-ins and are immutable after creation.
- Semantic keys are internal and are never accepted from Settings or public relationship APIs.
- A non-null semantic key marks a protected built-in; built-ins may be relabeled or disabled but not deleted.
- Custom entries have null semantic keys and may be deleted only when unreferenced.
- Labels are mutable display text and are unique after trimming and case normalization.
- Inactive entries remain valid for historical rows but are excluded from new-assignment selectors.
### Document-Person Role Governance
- Documents support zero, one, or many people per relationship role.
- Documents support zero or one relationship for each Person.
- Relationship roles are defined by `PERSON_ROLE` rather than hardcoded columns.
- `DOCUMENT_PERSON` must be unique for `(document_id, person_id, role_id)`.
- Relationship writes must be deterministic and use explicit add/remove link intent.
- `DOCUMENT_PERSON.role_id` is required.
- `DOCUMENT_PERSON` must be unique for `(document_id, person_id)`.
- Complete link sets and Document fields are validated and persisted in one atomic transaction.
- Existing inactive roles may remain unchanged; new or changed assignments require active roles.
### Document Type Governance
- Every document type is defined by `DOCUMENT_TYPE`.
- `DOCUMENT_TYPE.id` is the sole machine identity.
- `DOCUMENT_TYPE.id` is the relationship identity; hidden semantic keys identify protected built-in meaning.
- `DOCUMENT_TYPE.label` is mutable display text and is unique after trimming and case normalization.
- `DOCUMENT_TYPE.normalized_label` stores the normalized uniqueness key.
- Inactive types remain valid for historical rows but should be excluded from default selection UIs.
@@ -214,8 +228,10 @@ EXECUTION_ATTEMPT ||--o{ PROCESSING_ARTIFACT : produces
## Constraint Summary
- `DOCUMENT_TYPE.normalized_label` is unique.
- `PERSON_ROLE.code` is unique.
- `DOCUMENT_PERSON(document_id, person_id, role_id)` is unique.
- `DOCUMENT_TYPE.semantic_key` is nullable and unique.
- `PERSON_ROLE.normalized_label` is unique.
- `PERSON_ROLE.semantic_key` is nullable and unique.
- `DOCUMENT_PERSON(document_id, person_id)` is unique.
## Indexing Guidance