V4.4 Scope defined (part 2)

2026-08-15 14:04:58 -05:00
parent 936af9b1d3
commit 63373bf24d
2 changed files with 420 additions and 0 deletions
@@ -0,0 +1,188 @@
+# Implementation Plan (Version 4.4)
+
+## Goal
+
+Deliver hidden semantic identity for built-in registries, a single atomic Linked People workflow, and safe browser-native printing of archival Documents and their current transcriptions.
+
+## Planning Constraints
+
+- V4.3 is the completed implementation baseline.
+- V4.4 may replace V4/V4.3 registry and document-person contracts only as specified by the V4.4 scope.
+- Semantic keys are internal and immutable; UI and public API contracts use UUIDs and labels.
+- Document and link edits must not partially commit.
+- Print output must not execute stored text or expose machine-local source paths.
+- Source page reordering remains excluded.
+- Database, integration, and UI tests must use confirmed isolated data and never modify `data/transcription.db`.
+- Potentially destructive tests must run only through `tools/run_destructive_tests.py`.
+
+## Expected Project Impact
+
+| Area | Expected impact |
+| --- | --- |
+| Models and schema bootstrap | Add nullable unique semantic keys, simplify document-person identity, and seed frozen built-ins. |
+| Document service | Maintain built-in Document Types, usage summaries, UUID assignment, and atomic Document/link writes. |
+| People service | Maintain built-in Person Roles, link summaries, UUID-only role assignment, and one-person-per-document enforcement. |
+| V4 document API | Remove role-code selectors and compatibility role fields; enforce UUID-only relationship writes. |
+| Settings UI | Use matching table workflows for Document Types and Person Roles. |
+| Document Create/Edit | Replace role-specific multiselects with one staged Linked People table and inline editor. |
+| Document Detail/printing | Add format selection, print preview, safe Source media rendering, print CSS, and job metadata. |
+| Tests and documentation | Replace superseded cardinality/identity assertions and add isolated registry, editor, transaction, and print coverage. |
+
+## Implementation Phases
+
+### 1. Align Durable Registry Contracts
+
+- Add nullable, unique `semantic_key` fields to `DocumentType` and `PersonRole`.
+- Keep UUIDs as primary and foreign-key identity.
+- Add normalized-label storage and uniqueness to Person Roles using the same trim and case-normalization policy as Document Types.
+- Remove the user-created Person Role code contract.
+- Define built-in detection as `semantic_key is not null`.
+- Centralize the frozen built-in definitions in one domain-owned location.
+- Seed six Document Types and three Person Roles idempotently.
+- Ensure label edits never change semantic keys.
+- Reject deletion of every built-in before checking references.
+- Continue blocking deletion of referenced custom entries.
+- Return deterministic validation, conflict, dependency, and not-found errors through existing error categories.
+
+### 2. Establish the Clean Schema
+
+- Remove legacy `DocumentPerson.role` compatibility storage and the fixed `DocumentPersonRole` enum.
+- Make `DocumentPerson.role_id` required.
+- Replace role-specific uniqueness with a unique `(document_id, person_id)` constraint.
+- Remove obsolete Document Type and Person Role migration paths that exist only for disposable development data.
+- Keep fresh schema creation and built-in seeding portable across SQLite and PostgreSQL.
+- Make configured development-database recreation a separate operator-confirmed step that displays the resolved target path rather than assuming `app.db` or `data/transcription.db`.
+- Never invoke recreation from application startup or test setup.
+- Add isolated schema tests for fresh creation, seed idempotence, semantic-key uniqueness, normalized-label uniqueness, required roles, and link uniqueness.
+
+### 3. Refine Registry Services and API Contracts
+
+- Add summary queries for Document counts and Person Role link counts without per-row queries.
+- Order both registries by normalized label with UUID as a deterministic tie-breaker.
+- Expose built-in status as a derived read value where the Settings UI needs it.
+- Keep semantic-key lookup behind service methods for application-owned behavior such as resolving authors.
+- Ensure create operations always produce custom entries with null semantic keys.
+- Ensure update operations accept only label and active state.
+- Remove `role_code` request alternatives and compatibility role responses from the V4 document API.
+- Require `role_id` for document-person creation and updates.
+- Add service and API tests for hidden semantic identity, relabeling, activation, built-in protection, custom deletion, counts, ordering, UUID-only writes, and conflicts.
+
+### 4. Build Matching Settings Tables
+
+- Retain the existing Document Types table workflow and add the Built-in column.
+- Replace the current per-row Person Role controls with the same selection-based table pattern.
+- Render the agreed columns and usage counts.
+- Keep labels as the only registry text shown in selectors.
+- Add creates custom entries only.
+- Edit dialogs expose label and active state only.
+- Delete reports protected-built-in and referenced-custom conflicts clearly.
+- Avoid direct persistence queries from the Settings page.
+- Add component-level UI assertions for columns, actions, label-only selectors, and immutable built-in presentation.
+
+### 5. Add a Staged Linked People Editor
+
+- Introduce a small typed UI-state model for staged `(person_id, role_id)` rows rather than storing raw widget values.
+- Share the editor component between Create Document and Edit Document.
+- Render a multi-selection table with Person and Role labels.
+- Add an inline editor whose mode is explicitly Add or Edit.
+- Disable already-linked People when adding; retain the edited Person as an option during Edit.
+- Require exactly one row for Edit and allow one or more rows for Delete.
+- Save and Delete mutate only staged UI state.
+- Cancel discards only the active inline edit.
+- Preserve inactive-role historical rows in Edit while restricting new assignments and changes to active roles.
+- Preserve `person_id` preselection by staging that Person with the active built-in `author` role, with warning behavior for invalid or unavailable selections.
+- Preserve the `return_to=jobs_new` success path.
+- Keep navigation to Person creation separate; V4.4 does not add an embedded Person editor.
+- Add UI tests for staging, duplicate prevention, selection rules, inactive roles, cancel behavior, and both Document forms.
+
+### 6. Persist Document and Links Atomically
+
+- Add service commands for Create Document with complete links and Update Document with complete links.
+- Validate Document Type, every Person, every Person Role, active assignment rules, and duplicate People before mutation.
+- Compute deterministic add, update, and remove deltas for Edit.
+- Apply Document and link mutations in one database transaction and commit once.
+- Roll back the complete operation on any validation, conflict, or persistence failure.
+- Return the persisted Document detail required by the UI after success.
+- Reuse these commands from UI orchestration rather than sequencing independent service commits.
+- Add failure-injection tests proving no partial Document or link mutation survives.
+
+### 7. Define a Print Projection
+
+- Add a read-only service projection containing:
+  - Document title and selected archival metadata.
+  - Authors resolved by the `author` semantic key.
+  - Notes.
+  - Ordered Sources with application media URLs and current transcription text.
+  - Ordered Job metadata.
+- Load the projection with bounded queries and deterministic ordering.
+- Use non-null `revised_text`, including an intentionally empty revision; otherwise fall back to `raw_transcription`.
+- Map empty or whitespace-only current text to the explicit unavailable state without falling back past an intentional revision.
+- Represent unavailable text and optional metadata explicitly.
+- Do not expose semantic keys, direct file paths, full prompts, provider evidence, or raw API responses.
+- Keep the projection independent of NiceGUI rendering so formatting tests can use plain typed values.
+
+### 8. Build Print Preview and Styles
+
+- Add a Print action to Document Detail.
+- Open a dedicated persisted-Document print route with a Facsimile/Text-only format choice.
+- Render the exact content order frozen in the scope.
+- Render stored Notes and transcription as escaped text.
+- For Text-only mode, normalize whitespace by joining single line breaks inside paragraphs while preserving blank-line paragraph boundaries.
+- For Facsimile mode, preserve line breaks and use a two-column Source layout.
+- Start each Facsimile Source on a new printed sheet with CSS page breaks.
+- Allow long transcription content to continue rather than clipping it.
+- Fetch images through an application-controlled Source media route.
+- Add print-only CSS that hides navigation, controls, and non-document chrome.
+- Invoke the browser print dialog only from an explicit user action.
+- Add rendering tests for both modes, missing data, long text, special characters, image URLs, and page ordering.
+
+### 9. Align Documentation and Verification
+
+- Update V4 architecture, requirements, schema, and Document UI contracts to reflect:
+  - UUID plus hidden semantic-key registries.
+  - Built-in protection.
+  - One Person per Document.
+  - UUID-only role API writes.
+  - Atomic Document/link synchronization.
+  - Browser-native print projection and formats.
+- Confirm every database, integration, and UI test target is isolated before execution.
+- Run focused registry and service tests first.
+- Run schema tests only through the destructive-test wrapper when they are potentially destructive.
+- Run Linked People UI and print rendering tests against isolated fixtures.
+- Run the broader non-external regression suite after focused coverage passes.
+- Verify that `data/transcription.db` was not changed by test execution.
+
+## Delivery Order
+
+1. Registry and clean-schema contracts.
+2. Registry services, API changes, and Settings tables.
+3. Atomic Document/link service commands.
+4. Shared staged Linked People editor.
+5. Print projection.
+6. Print preview and styles.
+7. Documentation alignment and regression verification.
+
+## Done Criteria
+
+- All V4.4 acceptance criteria are implemented and testable.
+- UI and API contracts use UUID identity and never expose semantic keys.
+- Built-in registries retain meaning after relabeling and cannot be deleted.
+- Custom registries retain reference-aware deletion.
+- Both Document forms use one Linked People table.
+- One Person cannot be linked twice to the same Document.
+- Main Document saves are atomic across fields and relationships.
+- Print preview provides both frozen formats and content sections.
+- Print output uses current human-preferred text, deterministic ordering, escaped content, and application media URLs.
+- Job metadata lists every Job oldest-to-newest and ends with Status.
+- Source page reordering and server-generated PDFs are not introduced.
+- Verification uses isolated data and does not modify `data/transcription.db`.
+
+## Related Local References
+
+- [V4.4 Scope Boundary](scope_boundary_v4_4.md)
+- [V4.3 Scope Boundary](../ver4.3/scope_boundary_v4_3.md)
+- [V4.3 Implementation Plan](../ver4.3/implementation_plan_v4_3.md)
+- [V4 Architecture](../ver4/architecture_v4.md)
+- [V4 Schema](../ver4/schema_v4.md)
+- [V4 Requirements](../ver4/requirements_v4.md)
+- [V4 Error Handling Policy](../ver4/error_handling_v4.md)
@@ -0,0 +1,232 @@
+# V4.4 Scope Boundary
+
+This document defines the frozen boundary for the semantic-registry, linked-people, and document-printing revision that follows the completed V4.3 Settings work. V4 through V4.3 remain the architecture and behavioral baseline except where this document explicitly replaces a registry or document-person contract.
+
+## Purpose
+
+- Keep registry identifiers stable without exposing duplicate machine codes in Settings tables or selectors.
+- Replace role-specific person selectors with one coherent Linked People editor.
+- Provide an archival print view containing document metadata, source pages, current transcription text, and transcription-job metadata.
+
+## In Scope
+
+### 1. Semantic Registry Identity
+
+- `DocumentType` and `PersonRole` use UUIDs as their canonical record and relationship identity.
+- Both registries may carry a nullable, unique, immutable `semantic_key` used only for application-defined built-ins.
+- Semantic keys are internal implementation details. Settings tables, selectors, and public API payloads do not display or accept them.
+- Labels are trimmed, case-insensitively unique, editable, and used for all user-facing display.
+- Active state remains editable. Inactive entries remain valid for historical records and are excluded from default assignment selectors.
+- Built-in status is derived from the presence of a semantic key and is displayed as a read-only Yes/No value.
+- Built-in entries cannot be deleted or converted to custom entries.
+- Custom entries have no semantic key and may be deleted only when unreferenced.
+- Users may create custom entries but cannot create, change, or assign semantic keys through the UI or API.
+
+### 2. Built-In Document Types
+
+- Seed these built-in semantic keys and initial labels:
+
+| Semantic key | Initial label |
+| --- | --- |
+| `book` | Book |
+| `letter` | Letter |
+| `postcard` | Postcard |
+| `photo` | Photo |
+| `journal` | Journal |
+| `form` | Form |
+
+- The Document Types Settings table contains Select, Label, Documents, Active, and Built-in columns.
+- Document Types are ordered alphabetically by normalized label.
+- Add, Edit, and Delete actions operate on table selection.
+- Add creates a custom type. Edit changes only label and active state.
+- The Documents count is the number of Documents referencing the type.
+- Document Type selectors display labels only and submit UUIDs.
+
+### 3. Built-In Person Roles
+
+- Seed these built-in semantic keys and initial labels:
+
+| Semantic key | Initial label |
+| --- | --- |
+| `author` | Author |
+| `recipient` | Recipient |
+| `mentioned` | Mentioned |
+
+- Application behavior that requires authorship resolves the built-in `author` semantic key rather than matching a mutable label.
+- The Person Roles Settings table contains Select, Label, Links, Active, and Built-in columns.
+- Person Roles are ordered alphabetically by normalized label.
+- Add, Edit, and Delete actions operate on table selection.
+- Add creates a custom role. Edit changes only label and active state.
+- The Links count is the number of document-person relationships referencing the role.
+- Person Role selectors display labels only and submit UUIDs.
+
+### 4. Linked People Editor
+
+- Replace the separate role-specific person selectors on both Create Document and Edit Document with one Linked People table.
+- The table contains Select, Person, and Role columns.
+- Add opens an inline editor beneath the table with Person and Person Role selectors.
+- Edit requires exactly one selected row and loads it into the inline editor.
+- Save stages the inline addition or edit in the table.
+- Cancel exits the inline editor without changing the staged link set.
+- Delete stages removal of one or more selected rows.
+- A Person may be linked to a Document only once, regardless of role.
+- Every link has exactly one Person Role.
+- Already-linked People are unavailable when adding another row.
+- Existing links using inactive roles remain visible and unchanged unless explicitly edited.
+- Only active roles are available for new links or role changes.
+- Create Document preserves the existing `person_id` preselection workflow by staging that Person with the active built-in `author` role. An invalid Person or unavailable Author role produces a warning rather than an invalid link.
+- Create Document preserves the existing `return_to=jobs_new` success path.
+- Linked People changes remain staged until the main Create Document or Save Changes action.
+- The Document and its complete staged link set are persisted atomically. A conflict or validation failure leaves both unchanged.
+- The API and service contracts identify roles by `role_id`; role-code selectors and compatibility role strings are removed.
+- Persistence enforces uniqueness on `(document_id, person_id)`.
+
+### 5. Document Print View
+
+- Add a Print action to Document Detail.
+- The action opens a dedicated print-preview page for the persisted Document.
+- The preview offers two formats:
+  - **Facsimile:** source image on the left and current transcription on the right. Original transcription line breaks are preserved, and each Source begins on a new printed sheet.
+  - **Text only:** no source images. Single line breaks inside a paragraph are reflowed as spaces, while blank-line paragraph boundaries remain.
+- Both formats use browser printing through a dedicated print stylesheet and the browser print dialog.
+- Server-generated PDF files are not part of V4.4; users may select the browser's Save as PDF destination.
+- Sources are ordered by existing `page_number`, with UUID as a deterministic tie-breaker.
+- The current transcription for each Source is the non-null `revised_text`, including an intentionally empty revision, otherwise the latest successful machine-output projection in `raw_transcription`.
+- Empty or whitespace-only current text displays the explicit unavailable message rather than falling back past an intentional revision.
+- A Source with no current transcription displays an explicit unavailable message.
+- Transcription and Notes content is treated as text and escaped; model output is not executed as arbitrary HTML.
+- Facsimile images use an application-controlled Source media route. Generated markup does not expose direct machine-local file paths.
+
+### 6. Printed Content Contract
+
+The print view contains, in this order:
+
+1. Document title using the Document name.
+2. Archival Metadata table:
+   - Author, containing People linked through the built-in `author` role.
+   - Date.
+   - Location Created.
+   - Archival Identifier.
+3. Notes.
+4. Document section containing one numbered section per Source.
+5. Transcription Job Metadata table.
+
+Empty metadata values remain visible as `Not set`. Empty Notes display `No notes recorded`.
+
+The job metadata table:
+
+- Lists field names in the first column and adds one column for every Job associated with the Document.
+- Orders Job columns from oldest to newest by creation date, then UUID.
+- Includes every Job status: `queued`, `processing`, `transcribed`, `completed`, `partial_success`, and `failed`.
+- Contains these rows in order:
+  - Job ID.
+  - Date, using the Job creation/submission timestamp with timezone.
+  - Provider.
+  - Model.
+  - Prompt, using the frozen prompt filename/name rather than full prompt content.
+  - Retry Count.
+  - Status as the final row.
+- Displays `Not set` for unavailable optional metadata.
+
+### 7. Clean Development Schema
+
+- V4.4 does not require preservation or migration of rows in the operator-configured development database.
+- Implementation may recreate the configured development database, including `data/transcription.db` when it is the explicitly selected target, only through a separate operator-confirmed action that identifies the resolved path. Startup and test execution never delete it automatically.
+- Fresh schema creation seeds the agreed built-in Document Types and Person Roles idempotently.
+- No test may use, modify, replace, or restore `data/transcription.db`.
+- Database, integration, and UI tests use confirmed isolated databases.
+- Potentially destructive tests run only through `tools/run_destructive_tests.py`.
+
+## Out of Scope
+
+- User creation, editing, deletion, or direct display of semantic keys.
+- Treating custom registry entries as built-ins.
+- Additional built-in Document Types or Person Roles beyond the frozen lists.
+- Assigning more than one role to the same Person on the same Document.
+- Preserving multiple historical links that violate the new one-person-per-document constraint.
+- Source page renumbering or reordering.
+- Printing unsaved Create/Edit Document state.
+- Print actions on Job Detail or other pages.
+- Batch printing multiple Documents.
+- Server-side PDF generation or PDF file storage.
+- Markdown, DOCX, or evidence-package export through the print feature.
+- User-editable print templates, fonts, margins, headers, or footers.
+- Full frozen prompt content, prompt hashes, transport evidence, API responses, or execution-attempt details in the print footer.
+- Rendering transcription text as unrestricted Markdown or HTML.
+- Pixel-identical pagination across browsers and printer drivers.
+
+## Locked Design Decisions
+
+### A. UUID Identifies the Row; Semantic Key Identifies Built-In Meaning
+
+- UUIDs remain the only relationship and API identity.
+- A hidden semantic key permits reliable built-in behavior after a label is renamed.
+- Mutable labels are never used to infer built-in meaning.
+
+### B. Built-Ins Are Protected but Mutable in Presentation
+
+- Built-in labels and active state may change.
+- Built-in semantic identity cannot change, and built-ins cannot be deleted.
+- Custom entries remain reference-aware and deletable when unreferenced.
+
+### C. Linked People Is a Single-Role Relationship
+
+- One `(document_id, person_id)` row represents the complete relationship.
+- Changing a role updates that row rather than adding another relationship.
+- The main Document save owns one atomic Document-and-links transaction.
+
+### D. Printing Uses the Current Human-Preferred Text
+
+- Human-revised text takes precedence over the latest successful machine-output projection.
+- Job metadata provides processing context but does not claim that a later human revision is raw output from a listed Job.
+
+### E. Printing Is Browser-Native
+
+- A print-specific HTML view and CSS support physical printing and browser Save as PDF.
+- Source media is served through application-controlled routes, and all textual content is escaped.
+
+### F. Source Order Is Read-Only in V4.4
+
+- Print order follows existing page numbers.
+- Source page reordering remains explicitly excluded.
+
+## Acceptance Criteria
+
+1. Registry selectors and Settings forms never display a machine code or semantic key.
+2. Document Types and Person Roles use UUID relationship identity and case-insensitively unique labels.
+3. The six Document Type and three Person Role built-ins are seeded with immutable internal semantic keys.
+4. Built-ins may be relabeled or disabled but cannot be deleted.
+5. Unreferenced custom entries may be deleted; referenced custom entries may only be relabeled or disabled.
+6. Settings tables show the agreed columns, alphabetical label order, usage counts, and selection-based actions.
+7. Create and Edit Document use one Linked People table with inline staged Add/Edit/Save/Cancel and multi-row Delete.
+8. The same Person cannot be staged or persisted twice for one Document, even under different roles.
+9. Document fields and Linked People changes commit atomically.
+10. Historical inactive roles remain displayable, while only active roles are assignable.
+11. Document Detail opens a print preview with Facsimile and Text-only formats.
+12. Print pages use current revised text when available and deterministic Source ordering.
+13. Printed archival metadata resolves authors through the hidden `author` semantic key after any label change.
+14. The job table contains one oldest-to-newest column per Job and ends with the Status row.
+15. Print output escapes stored text and does not disclose direct local source paths.
+16. Source reordering, server PDF generation, and print-template editing are absent.
+17. Database, integration, and UI verification uses isolated test data and never modifies `data/transcription.db`.
+
+## Scope Freeze Gate
+
+V4.4 is sufficiently frozen to begin implementation:
+
+- Built-in registry identity, membership, lifecycle, display, and selector behavior are resolved.
+- Linked People selection, editing, uniqueness, inactive-role, staging, and transaction behavior are resolved.
+- Print entry point, formats, content order, transcription precedence, page order, job metadata, and output mechanism are resolved.
+- Clean development-schema and destructive-test boundaries are resolved.
+- Source page reordering remains excluded.
+
+Any expansion of the built-in catalogs, relationship cardinality, print formats, export formats, or print customization requires an explicit V4.4 scope amendment or a later revision.
+
+## Related Local References
+
+- [V4.4 Implementation Plan](implementation_plan_v4_4.md)
+- [V4.3 Scope Boundary](../ver4.3/scope_boundary_v4_3.md)
+- [V4.3 Implementation Plan](../ver4.3/implementation_plan_v4_3.md)
+- [V4 Architecture](../ver4/architecture_v4.md)
+- [V4 Schema](../ver4/schema_v4.md)
+- [V4 Requirements](../ver4/requirements_v4.md)