bbchops/transcription
1
0
Fork 0
generated from john/python-template
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

V4.1 major revision to docs. Removed all obsolete documents, updated v4.2 implementation scope and plan.

Browse Source
This commit is contained in:
Jim Lancaster
2026-08-13 15:32:40 -05:00
parent 171132919d
commit 28811d79ce
60 changed files with 1170 additions and 7471 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/Proposed Revisions to V4.md
-63
View File
@@ -1,63 +0,0 @@
# Proposed Revisions
This document contains a list of proposed fixes, adjustments, and additional features. Consider whether each change provides a real improvement, is practical to implement, and whether or not it introduces unnecessary complexity and make recommendations.
Also, consider whether these changes should be implemented in more than one revision. For example, some might be considered Version 4.1 revisions, some might be Version 5, and some might be saved for a future date after I've had time to use the app for a while.
## UI
Most of the proposed changes to the UI are cosmetic. The issue numbers refer to issues I have accumulated in the Gitea repository for my own reference.
### Archival Documents page (issue #13)
* Limit the Document Title column width and wrap long titles. All columns in the table should fit on the screen.
* Left-align Document Title and Type columns
* Add Author, Document Date columns to the table
### Document Detail page (issue #15)
* In Create/Edit mode, When linking people by role it is impossible to distinguish between two people with the exact same name (e.g., there are two Albert Edward Higgins, Albert Edward Higgins (father & son). Possible solutions:
* Consider using the Display Name instead, and adding a contraint that the Display Name be unique
* Consider concatenating birth year onto the Full Name (e.g., 'Albert Edward Higgins (1885)')
* In View mode, The person entries in the "Related People" box should be hyper-linked to their People Detail pages
* Move "View All Sources" box from under image to under Pipeline Jobs. Consider combining both Pipeline Jobs and Sources together in one box.
* Hide unused fields when not in Edit mode. Example: When editing the Document Detail page there are boxes for Exact Date and Approx. Date. Only one field is ever used (never both). Is it possible to hide the unused field when not in Edit mode?
### Archival Entities: People page (issue #19)
* Left align Display Name, Maiden Name columns
* Birth Date column should contain Birth Date if known, else Approx Birth Date if known, else "unknown".
* Add Death Date column with rule similar to Birth Date
### Person Detail page (issue #17)
* Add "New Document" button to People page to accommodate an additional workflow: ```Add new person --> Add document --> Add job```. The current workflow is this: ```Add new document --> Link person, add new if one doesn't exist --> Add job``` The user should be able to start from either Document or Person, whichever seems more natural to them.
* Hide unused fields when not in Edit mode. Example: When editing the People Detail page there are boxes for birth date and approx. birth date. Only one field is ever used (never both). Hide the unused fields on the Person Detail page when not in Edit mode.
### Source Asset Records page (issue #12)
* A Source with a long Document Name or Upload Title forces columns (e.g., Stored Filename, Status, and Error Detail) off the right edge of the bounding box. Limit the width of the columns and wrap their contents. All columns in the table should fit on the screen.
* Remove Stored Filename from the table
* Left align Document Name, Upload Title, and Error Detail
### Source Detail page (issue #10)
* If a Source is one page of a multi-page document, add left/right navigation arrows on either side of the displayed image or page (which ever is easier to implement) screen to allow the user to page through the Sources without having to exit select another source then go back into the source detail screen.
### Transcription Pipeline Jobs page
* No changes at this time
### Job Detail Page (issue #7)
* Is it possible to add an automatic screen refresh while waiting for a job to finish? This could happen on a timed basis, like every 3-5 seconds.
### Home page
* Is the current location of the home page in the ./data/homepage folder the correct location to store the image & markdown file? Does it belong somewhere else in the repository like the context prompt which is in ./prompts
---
## New Features
* Consider creating a "Settings" page. My idea is that this page would provide one place for the user to customize the app. This screen may lead to subscreens. For example, the View/Edit prompt button might call up a page where the markdown is displayed in an editable box, and include a brief description of what the prompt is, how it's used, and the document format (markdown).
* Maintain Document Types and People Roles
* View/edit .env?
* View/edit theme?
* View/edit prompt?
* Consider adding field to Person Detail page (and person table) for links to ancestry.com and familysearch.org (and maybe others in the future). Explore using familysearch.org api to automatically populate Person details like Birth date, Birth Place, Death Date, Death Place, etc. (The goal here is not reinvent the wheel and leverage existing data).
* Consider adding the ability to renumber/reorder pages. (What happens when they get imported out of order?)
* Consider adding link to location/place fields to Google Maps. (Perhaps this is better left to the genealogy services...)
docs/Raw API Response Q&A - part 1.md
-176
View File
@@ -1,176 +0,0 @@
# Raw API Response Q&A
I initiated this exchange from within VSC using my Copilot Pro subscription. I believe the model was GPT-5.3-Codex, but I am not certain.
## My prompt
```I can now see the ai_metadata and raw_api_response in the database. Help me understand what the two fields represent```
## AI Response
Yes — in this project they are two different kinds of JSON evidence attached to a transcription attempt.
- `ai_metadata` is a compact, normalized summary of the AI run.
- It is meant for operational/useful details such as:
- `finish_reason`
- token usage, such as `input_tokens`, `output_tokens`, and `total_tokens`
- It is not the full provider response; it is a curated summary for debugging and tracking.
- `raw_api_response` is the full upstream provider response, stored as JSON-compatible data.
- This preserves whatever the AI provider returned so you can inspect it later.
- It can include things like model info, choices, message content, usage, and other provider-specific fields.
A simple way to think about it:
- `raw_transcription` = the text the system extracted from the AI response
- `ai_metadata` = a summarized record of how the AI call ended and what it used
- `raw_api_response` = the full original response payload from the provider
You can see this in models.py, where both fields are stored on the job-source record, and in openrouter.py, where the provider builds `ai_metadata` from completion/usage details and stores the full response in `raw_api_response`.
Example shape:
```json
ai_metadata = {
"finish_reason": "stop",
"usage": {
"input_tokens": 123,
"output_tokens": 45,
"total_tokens": 168
}
}
```
```json
raw_api_response = {
"id": "...",
"model": "google/gemini-2.5-flash",
"choices": [...],
"usage": {...}
}
```
If you want, I can also help you interpret a specific row’s values from your database.
## My Prompt
```
I was under the impression that there may be other information in the AI response, like:
* AI's confidence level in the accuracy of the transcription--especially when inserting guesses (in square brackets)
* The coordinates of the text box where the text was found on the screen
Is this JSONB data returned by all AIs? Or some, but not all? Or none?
```
## AI Response
I’m checking the provider implementation and docs to answer this based on what this system actually captures rather than assuming a universal AI API format.
Searched for text `confidence|bounding|bbox|coordinates|box|square bracket|guess` (`**/src/**`), 9 results
Short answer: it depends on the provider and model, and in this codebase the app does not assume those fields exist.
- The JSONB columns are generic containers, so they can hold extra provider-specific data if the API returns it. You can see that in base.py and models.py.
- For the current OpenRouter integration, the app only extracts a small subset of information:
- `finish_reason`
- token usage
- the full provider response as raw JSON
That logic is in openrouter.py.
So for your examples:
- Confidence in transcription accuracy:
- Some providers/models may expose something like confidence, probability, or logits.
- Many do not.
- This app does not currently parse or surface that as a dedicated field.
- Coordinates of detected text boxes:
- Some OCR/vision APIs return bounding boxes.
- Many chat-style LLM APIs do not.
- This app does not currently extract or normalize that data.
In other words:
- “Some, but not all” is the right answer.
- “None” is not quite right for the raw JSON field, because if a provider returns those values, they would be stored there.
- But “not guaranteed by this system” is the most accurate statement, since the current implementation does not request or interpret them as first-class fields.
If you want, I can help sketch what would be needed to start capturing confidence or bounding-box data explicitly.
## My Prompt
```
So if I understand you correctly, the current contents of "raw_api_response" is not actually the full contents of the AI response? If true, why is it being abbreviated?
```
## AI Response
Yes — that is the right interpretation.
What is being stored in the current implementation is not a byte-for-byte copy of the provider’s original response. It is a best-effort, JSON-compatible representation of the provider response object after the code has tried to serialize it for the database. You can see that in openrouter.py, where the code runs through a serializer and then stores the result.
Why it is abbreviated:
- The provider SDK returns objects, not plain JSON.
- The app needs something that can be stored in a JSONB column, so it converts nested objects into JSON-friendly data.
- Some values are not serializable by default, so they are reduced to strings or simplified structures.
- The code is not capturing the full transport-level HTTP payload, headers, status codes, or raw wire bytes.
- The field is intended mainly for inspection and debugging, not as a perfect archival copy.
So the more precise wording is:
- `raw_api_response` = a normalized, JSON-friendly snapshot of the provider response
- not = the full original upstream response verbatim
If you want the truly full response, the system would need to capture the actual HTTP response body and metadata from the provider client rather than the SDK object after conversion.
## My Prompt
```
What are the pros/cons of capturing the actual HTTP response? Is there a long term benefit?
```
## AI Response
Yes — there can be a strong long-term benefit, but it comes with tradeoffs.
Pros
- Better debugging:
- You can inspect the exact provider payload when a transcription fails or behaves unexpectedly.
- This is especially helpful when a provider changes schema or returns unexpected fields.
- Better reproducibility:
- You preserve the input/output pair for a given request, which can help compare model behavior over time.
- Useful for auditing, regression analysis, and support cases.
- More complete evidence:
- You can capture status codes, headers, response bodies, and provider-specific metadata that the current normalized JSON may lose.
- Future-proofing:
- If you later want to extract new fields such as confidence, bounding boxes, or token details, you may already have the raw payload available.
Cons
- Storage cost:
- Raw HTTP responses can be large, especially if you store full bodies, headers, and repeated requests.
- Over time this can grow database size and backup/restore cost.
- Privacy and compliance:
- Some provider responses may include sensitive or regulated content.
- You need to think about retention policy and access controls.
- Complexity:
- You need a clear schema or storage strategy for raw payloads.
- You may end up storing a mix of JSON, compressed blobs, or external objects.
- Maintenance burden:
- Raw payloads can become noisy and hard to query.
- You may need separate indexes, retention rules, or archival storage.
Long-term benefit
- If your goal is traceability and model observability, yes, it is often worth it.
- In a transcription system, especially one that depends on third-party AI providers, raw response capture can become valuable for:
- troubleshooting provider changes,
- investigating hallucinations or formatting regressions,
- building confidence/quality analytics later.
A practical compromise is often:
- store a compact normalized summary in `ai_metadata`,
- store the full raw response only for selected requests or for a limited retention window,
- and keep the raw payload in an external blob/object store rather than in the main relational table if volume becomes significant.
So the short version is: capturing the actual HTTP response is usually a good investment if you care about long-term observability and auditability, but it should be designed carefully to avoid storage and privacy problems.
docs/Raw API Response Q&A - part 2.md
-163
View File
@@ -1,163 +0,0 @@
# Raw API Response Q&A - part 2
In this exchange I sought a second opinion about what raw API response data I should be collecting in my transcription app. I uploaded part 1 to Gemini 3.6 Flash, then posted my first prompt.
## My prompt
```I'm trying to decide what data I should capture from the AI doing transcriptions for my transcription app. I will have 2000-3000 documents at most (not millions!) These are family heirlooms I am transcribing for posterity, and I want to use the archival methods/practices that I can. Review the attached document and tell me what you think.```
## AI Response (Gemini 3.6 Flash)
The assistant's breakdown of `ai_metadata` vs. `raw_api_response` is technically spot-on, but **Response #4 gives advice meant for massive, enterprise-scale software, not a focused archival project.**
When building an application for 2,000–3,000 family heirloom documents, the trade-offs change completely:
---
### 1. The Scale Reality: Ignore the Storage Warnings
In Response #4, the assistant cautions about storage costs, database bloat, and needing external cloud blob stores.
* **The Math:** A rich, full JSON response from an LLM API averages **5 KB to 15 KB**.
* **Your Total Volume:** $3\text{,}000 \text{ documents} \times 15\text{ KB} = \mathbf{\sim 45\text{ MB}}$ total.
45 Megabytes is negligible. It can easily sit inside PostgreSQL JSONB columns forever without causing performance or cost issues. You do not need complex retention policies, blob storage buckets, or truncated responses.
---
### 2. Archival Science Perspective: Provenance Matters
From a formal archival and documentary editing perspective, AI-assisted transcription is a form of document processing that requires **provenance** (origin and chain of custody).
In 20 years, when AI models have shifted dramatically, having the record of *how* a draft was generated is invaluable. Storing complete AI responses ensures:
* **Auditability:** You can trace why a specific transcription error, hallucination, or bracketed guess `[inaudible]` occurred.
* **Reproducibility:** You capture the exact model string, parameters, and model outputs used at that point in history.
* **Future Extraction:** If OpenRouter or model providers start embedding new provider-level metadata (such as reasoning traces or extended token details), you won’t regret having thrown it away.
---
### 3. What You Should Actually Capture
To balance clean querying with archival completeness, stick with the two-field approach, but don't hold back on what you put in them.
#### A. Operational Summary (`ai_metadata`)
Use this for fast filtering, UI display, and analytics in your app.
```json
{
"provider": "openrouter",
"model_requested": "google/gemini-2.5-flash",
"model_used": "google/gemini-2.5-flash",
"finish_reason": "stop",
"tokens": {
"prompt_tokens": 1250,
"completion_tokens": 340,
"total_tokens": 1590
},
"latency_ms": 1420,
"cost_usd": 0.00018
}
```
#### B. Complete Payload (`raw_api_response`)
Store the **complete, unfiltered SDK response dictionary** returned by the API client. If the OpenRouter Python client hands back a response object, serialize its entire `.model_dump()` or `.dict()` straight into your `raw_api_response` JSONB column.
#### C. Input Parameters (Don't Forget the Prompt!)
To have a true archival record, you also need to store what you *sent* the AI, not just what it sent back. Either on the attempt record or in the metadata, record:
* The exact system prompt version used.
* The temperature / top_p settings.
* The image filename / hash sent to the vision model.
---
### 4. The Reality on Bounding Boxes & Confidence Scores
Regarding your second prompt about bounding boxes and confidence scores:
1. **Bounding Boxes (Coordinates):** Standard Vision LLMs (like Gemini Flash, Claude 3.5 Sonnet, or GPT-4o) do **not** automatically return bounding boxes in API response metadata. OCR-specific tools (like Amazon Textract or Google Cloud Vision) return explicit spatial coordinates, but chat/vision LLMs only do so if you specifically instruct them in your prompt to format their output as JSON containing coordinates.
2. **Confidence Scores:** Standard completion APIs do not return a clean "95% confidence" score. The closest equivalent is token `logprobs` (log probabilities), which can be requested via API parameters (`logprobs: true`), but token probability measures statistical model likelihood during token generation, not necessarily historical accuracy.
---
### Summary Recommendation
Save the full, unfiltered API response object directly into `raw_api_response` without stripping anything out, alongside a clean `ai_metadata` summary. At 3,000 items, keeping complete raw evidence is effortless, aligns perfectly with documentary preservation standards, and protects your project's historical audit trail.
---
## My Prompt
```
1. You mentioned the raw api response could "easily sit inside PostgreSQL JSONB columns". I'm currently running on SQLite during development and testing, but I had already anticipated moving to PostgresSQL before I went into production. Should I go ahead an move to PostgresSQL before implementing the capture of the raw api responses?
2. Can you elaborate on "C. Input Parameters (Don't Forget the Prompt!)".
- I'm currently capturing the *name* of the prompt, but not the full text.
- what are the temperature / top_p settings?
- I'm already capturing the image_filename (It's being stored outside of the database in a folder in the python project. Can/should these images be stored in Postgres too?)
```
## AI Response (Gemini 3.6 Flash)
### 1. Should you move to PostgreSQL now or stay on SQLite?
**Recommendation: Stay on SQLite for now. Do not delay your development.**
Modern SQLite handles JSON natively via its standard JSON extension and functions. You do not need PostgreSQL’s dedicated `JSONB` binary data type just to capture and store complete API responses during development.
#### Why Staying on SQLite First Makes Sense:
* **Feature Parity for Raw Payloads:** Storing a raw API payload as a JSON-formatted string in a standard SQLite column behaves conceptually the same as putting it into PostgreSQL's `JSONB` column.
* **No Database Drift:** If you use an ORM (like SQLAlchemy or Tortoise ORM), defining a generic `JSON` column type translates smoothly between both engines.
* **Seamless Migration Later:** Since your total collection will max out at 2,000–3,000 documents, your entire dataset will easily fit in a small SQLite file. Migrating that entire dataset to PostgreSQL before going live will take minutes, not days.
Keep building features and capturing response data on SQLite. Move to PostgreSQL only when you are ready to deploy your production environment.
---
### 2. Elaborating on Input Parameters
To satisfy archival provenance standards, capturing *how* you prompted the AI is just as crucial as capturing *what* the AI returned.
#### A. Full Text of the Prompt (Beyond the Name)
Storing only the prompt's reference name (e.g., `v1_transcription_prompt`) leaves a critical gap: if you tweak that prompt later, you lose the exact instructions used for past runs.
**What to capture:**
* **System Prompt:** The exact system instructions provided to the model.
* **User Prompt:** The exact dynamic prompt template merged with document-specific context.
* **Prompt Version/Hash:** An explicit version identifier or SHA-256 hash of the full prompt string for quick grouping.
#### B. Temperature and Top_P Settings
These are the foundational hyper-parameters controlling the randomness and creativity of the AI model's output:
| Parameter | What It Controls | Typical Value for OCR/Transcription | Why Record It? |
| --- | --- | --- | --- |
| **`temperature`** | Randomness of token selection. `0.0` is deterministic; higher values (e.g., `0.7`) introduce variability. | `0.0` to `0.2` | Confirms whether the run was deterministic (strict extraction) or creative (guessing missing words). |
| **`top_p`** (Nucleus Sampling) | Cuts off low-probability tokens. `0.1` means only top 10% likely words are evaluated. | `1.0` or model default | Documents constraints on the vocabulary pool available to the model during execution. |
Storing these alongside the prompt ensures you can accurately evaluate why two different runs on the same document yielded different results.
#### C. Where Should the Input Images Be Stored?
Keep storing the actual image files **in the filesystem**, not in PostgreSQL or SQLite binary `BLOB` columns.
```
/project_root
├── /storage
│ └── /images
│ └── document_00101_raw.jpg
└── /database
└── (stores path: "storage/images/document_00101_raw.jpg")
```
##### Why File-System Storage is Preferred:
1. **Database Simplicity:** Databases perform best when handling structured metadata, indexes, and JSON queries. Storing heavy binary images in database tables inflates backup sizes and slows down general queries.
2. **Web Server & UI Efficiency:** Serving image files directly from the disk path (or an internal file server) allows web clients to leverage standard HTTP byte-range requests, browser caching, and instant thumbnail generation without overloading database connections.
3. **Archival Safety:** Image files stored cleanly in standard folder hierarchies (e.g., JPEG, TIFF, PNG) are portable, easier to back up incrementally, and accessible independently of any database engine.
**Database Rule of Thumb:** Store the **file path**, **filename**, **file size**, and **cryptographic hash** (like SHA-256) inside PostgreSQL/SQLite. Store the **binary image bytes** on disk.
docs/invariant/error_handling.md
+101
View File
@@ -0,0 +1,101 @@
# Error Handling (Invariant)
## 1. Purpose
This document defines the non-negotiable failure-handling principles for the transcription application.
Error categories, API envelopes, status codes, framework integrations, and persistence fields may change between versions. Failures must nevertheless remain visible, safe, diagnosable, and consistent across every application boundary.
## 2. Core Invariants
### 2.1 Failures Are Visible
1. An operation must not report success when all or part of the requested work failed.
2. Invalid input, unavailable dependencies, persistence failures, provider failures, and unexpected defects must be surfaced through the application's established error path.
3. Code must not silently discard an exception, provider response, invalid value, or failed state transition.
4. When work can partially succeed, the successful and failed portions must be identified separately.
### 2.2 Messages Are Actionable
1. Operator-facing errors must explain what failed in concise language.
2. When a safe corrective action is known, the error must state it.
3. Expected validation or conflict failures must not be presented as unexplained internal defects.
4. Internal diagnostics must not replace a usable operator-facing message.
### 2.3 Errors Have Stable Identity and Classification
1. Every surfaced failure must have a stable correlation identifier or equivalent trace identity.
2. Failures must be classified into a documented, machine-readable category.
3. Boundary-specific representations must preserve the original category and correlation identity.
4. Unknown exceptions must be converted at an explicit boundary, retain their causal chain for diagnostics, and be classified as unexpected rather than disguised as an expected failure.
### 2.4 Boundary Translation Is Consistent
1. UI, API, service, worker, persistence, and provider boundaries must use one shared error model or deterministic translations between documented models.
2. A boundary may simplify presentation, but it must not change the meaning, retryability, or identity of a failure.
3. Domain and service code must not depend on UI notifications or HTTP response types.
4. UI and API layers must not infer error categories by parsing message text.
### 2.5 State Changes Are Safe
1. A failed atomic operation must leave persisted state unchanged.
2. Batch operations may preserve successful independent items only when partial success is an explicit part of the workflow contract.
3. A failed item must retain enough state to identify what was attempted and whether retry is safe.
4. Error handling must not overwrite earlier successful results or historical execution evidence.
### 2.6 Retry Is Explicit and Bounded
1. Validation, authorization, policy, conflict, and other deterministic failures must not be retried automatically without a relevant input or state change.
2. Automatic retry is permitted only for failures classified as transient and only when the operation is idempotent or otherwise protected from duplicate effects.
3. Retry count, delay, and terminal behavior must be bounded and observable.
4. Exhausted retries must end in a visible terminal failure rather than an indefinitely pending state.
### 2.7 Diagnostics Are Preserved Safely
1. Logs and persisted diagnostic evidence must retain enough context to correlate the failure with the affected operation and record.
2. Provider and infrastructure failures must preserve safe diagnostic evidence at the boundary where it is available.
3. Credentials, authorization headers, cookies, secret values, and unnecessary personal data must not appear in errors, logs, notifications, or exports.
4. Diagnostic metadata capture must use explicit safe-field allowlists where unrestricted content could contain secrets.
5. User-facing messages must not expose stack traces, local filesystem details, database credentials, or raw internal exceptions.
AI execution failures also follow the evidence rules in [Digital Evidence and AI Processing Provenance](ai_evidence_and_provenance.md).
### 2.8 Cancellation and Timeout Are Distinct Outcomes
1. User cancellation, application shutdown, local timeout, remote timeout, and provider rejection must remain distinguishable.
2. Cancellation must not be converted into success or a generic unexpected error.
3. Timeout handling must identify whether a provider response was received when that fact is known.
4. Cleanup after cancellation or timeout must preserve consistency and must not conceal a completed side effect.
### 2.9 Logging Must Support Audit Without Becoming the Record
1. Structured logs must include correlation identity, operation, category, and relevant non-secret record identifiers.
2. Expected operator errors may be logged less severely than unexpected defects, but they must remain observable.
3. Logs are operational diagnostics and do not replace required database state or archival evidence.
4. Duplicate logging of the same failure at every layer should be avoided; ownership of the authoritative log event must be clear.
## 3. Verification Policy
Each version must verify:
1. Every documented error category reaches the intended UI and API representation.
2. Failed atomic writes roll back completely.
3. Partial-success workflows preserve successful independent results and identify failed items.
4. Retry behavior is bounded and restricted to eligible failures.
5. Unexpected exceptions retain correlation and causal information without exposing sensitive details.
6. Logs, persisted evidence, UI messages, and exports contain no credentials.
7. Cancellation, timeout, provider response failure, and no-response failure remain distinguishable.
## 4. Versioned Ownership
1. Version-specific error taxonomies, envelopes, HTTP mappings, model fields, and framework behavior belong in the applicable version documentation.
2. Each versioned error-handling document must state how it satisfies this invariant.
3. A version may add stricter safeguards but must not weaken these principles without first revising this invariant deliberately.
4. Implementation and tests must be updated together when a versioned error contract changes.
## 5. Related Invariants
- [Historical Document Transcription Design Intent](intent.md)
- [Digital Evidence and AI Processing Provenance](ai_evidence_and_provenance.md)
- [UI Style Guide](ui_style_guide.md)
docs/test/test-bento-grid.py
-107
View File
@@ -1,107 +0,0 @@
from nicegui import ui
# 1. Mature Dark Mode Setup
ui.dark_mode(True)
# Define a refined dark palette using expanded dictionary styling
theme_colors = {
'primary': '#6366f1',
'secondary': '#8b5cf6',
'accent': '#ec4899',
'dark': '#0f172a',
'dark_page': '#020617',
'positive': '#10b981',
'negative': '#ef4444',
}
ui.colors(**theme_colors)
# Optional: Add custom CSS for subtle noise overlays or kinetic typography
ui.add_css('''
.glass-card {
background: rgba(255, 255, 255, 0.03);
backdrop-filter: blur(12px);
-webkit-backdrop-filter: blur(12px);
border: 1px solid rgba(255, 255, 255, 0.05);
border-radius: 1.5rem;
}
''')
# 2. Bento Grid Layout
with ui.element('div').classes('grid grid-cols-1 md:grid-cols-4 gap-6 w-full max-w-6xl mx-auto p-8'):
# Header spanning all columns
with ui.element('div').classes('col-span-1 md:col-span-4 mb-4'):
ui.label('Analytics Dashboard').classes('text-4xl font-extrabold tracking-tight text-white')
ui.label('AI-driven insights for Q3').classes('text-lg text-slate-400 mt-1')
# Large Feature Card (Glassmorphism + Functional Motion)
with ui.element('div').classes('glass-card col-span-1 md:col-span-2 p-6 transition-transform duration-300 hover:scale-[1.02]'):
ui.icon('monitoring', size='2rem').classes('text-primary mb-4')
ui.label('Revenue Prediction').classes('text-xl font-semibold text-slate-100')
ui.label('$45,231.00').classes('text-5xl font-bold text-white mt-2')
# Placeholder for an interactive EChart
ui.echart({
'xAxis': {
'type': 'category',
'data': [
'Mon',
'Tue',
'Wed',
'Thu',
'Fri',
],
},
'yAxis': {
'type': 'value',
},
'series': [
{
'data': [
120,
200,
150,
80,
70,
],
'type': 'bar',
'itemStyle': {
'color': '#6366f1',
},
},
],
}).classes('w-full h-48 mt-4')
# Smaller Metric Cards
metric_cards = [
{
'title': 'Active Users',
'value': '1,204',
'icon': 'group',
'color': 'text-secondary',
},
{
'title': 'Server Load',
'value': '34%',
'icon': 'memory',
'color': 'text-accent',
},
]
for card in metric_cards:
with ui.element('div').classes('glass-card col-span-1 p-6 flex flex-col justify-between transition-transform duration-300 hover:-translate-y-1'):
ui.icon(card['icon'], size='2rem').classes(card['color'])
ui.element('div').classes('flex-grow')
ui.label(card['value']).classes('text-4xl font-bold text-white mt-4')
ui.label(card['title']).classes('text-sm font-medium text-slate-400 uppercase tracking-wider')
# AI Assistant Module (Adaptive Interface)
with ui.element('div').classes('glass-card col-span-1 md:col-span-4 p-6 flex items-center gap-4'):
ui.icon('smart_toy', size='2rem').classes('text-positive animate-pulse')
with ui.element('div'):
ui.label('Ambient AI Suggestion').classes('text-sm font-bold text-positive uppercase tracking-wider')
ui.label('Based on current server load, scaling up instances in the EU-West region is recommended.').classes('text-slate-300')
ui.space()
ui.button('Apply Now', color='positive').classes('rounded-full px-6 py-2 shadow-lg shadow-positive/20')
ui.run(title='2026 UI Dashboard')
docs/test/test_block-beta.md
-92
View File
@@ -1,92 +0,0 @@
```mermaid
block-beta
columns 3
%% UI Component Column
block:UI["UI COMPONENTS / WIREFRAME"]:1
columns 1
block:HeaderUI["Header & Nav"]:1
columns 1
h_title["[Text] Document Name & Type"]
h_date["[Text] Date & Origin Location"]
end
block:EditorUI["Page Transcription Editor"]:1
columns 1
ed_img["[Image Viewer] Source Image"]
ed_page["[Badge] Page Number"]
ed_raw["[Read-Only] AI Raw Output"]
ed_rev["[Textarea] Human Revised Text"]
end
block:PeopleUI["Attribution Sidebar"]:1
columns 1
p_author["[List] Authors (Full Name)"]
p_recip["[List] Recipients (Full Name)"]
p_bio["[Card] Person Biography & Dates"]
end
block:JobUI["AI Processing Drawer"]:1
columns 1
j_status["[Badge] Job Status"]
j_model["[Text] Provider & Model"]
j_tokens["[JSON View] AI Token Usage"]
end
end
%% Directional Mapping / Connectors
block:FLOW["MAPPING / FLOW"]:1
columns 1
f1["Reads / Updates -->"]
f2["Renders Active Page -->"]
f3["Joins via Role -->"]
f4["Executes & Logs -->"]
end
%% Postgres Schema Column
block:DB["POSTGRES SQL SCHEMA"]:1
columns 1
block:DocTbl["Table: document"]:1
columns 1
d_id["id : UUID (PK)"]
d_name["name : TEXT"]
d_type["document_type : TEXT"]
d_date["document_date : DATE"]
end
block:SrcTbl["Table: source"]:1
columns 1
s_id["id : UUID (PK)"]
s_page["page_number : INT"]
s_path["file_path : TEXT"]
s_raw["raw_transcription : TEXT"]
s_rev["revised_text : TEXT"]
end
block:PersonTbl["Table: person & document_person"]:1
columns 1
p_id["id : UUID (PK)"]
p_name["full_name : TEXT"]
p_role["role : 'author' | 'recipient'"]
end
block:JobTbl["Table: job & job_source"]:1
columns 1
j_id["id : UUID (PK)"]
j_stat["status : VARCHAR"]
j_prov["provider / model : TEXT"]
j_meta["ai_metadata : JSONB"]
end
end
%% Connections
HeaderUI --> DocTbl
ed_img --> s_path
ed_page --> s_page
ed_raw --> s_raw
ed_rev --> s_rev
PeopleUI --> PersonTbl
JobUI --> JobTbl
```
docs/test/test_flowchart.md
-53
View File
@@ -1,53 +0,0 @@
```mermaid
flowchart LR
subgraph UI["UI Components / Wireframe"]
direction TB
subgraph HeaderUI["Header & Nav"]
h_title["[Text] Document Name & Type"]
h_date["[Text] Date & Origin Location"]
end
subgraph EditorUI["Page Transcription Editor"]
ed_img["[Image Viewer] Source Image"]
ed_page["[Badge] Page Number"]
ed_raw["[Read-Only] AI Raw Output"]
ed_rev["[Textarea] Human Revised Text"]
end
subgraph PeopleUI["Attribution Sidebar"]
p_author["[List] Authors / Recipients"]
end
subgraph JobUI["AI Processing Drawer"]
j_status["[Badge] Job Status"]
end
end
subgraph DB["Postgres SQL Schema"]
direction TB
subgraph DocTbl["Table: document"]
d_name["name : TEXT"]
d_type["document_type : TEXT"]
end
subgraph SrcTbl["Table: source"]
s_path["file_path : TEXT"]
s_page["page_number : INT"]
s_raw["raw_transcription : TEXT"]
s_rev["revised_text : TEXT"]
end
subgraph PersonTbl["Table: person & document_person"]
p_name["full_name : TEXT"]
p_role["role : author | recipient"]
end
subgraph JobTbl["Table: job & job_source"]
j_stat["status : VARCHAR"]
j_meta["ai_metadata : JSONB"]
end
end
%% Mappings
HeaderUI --> DocTbl
ed_img --> s_path
ed_page --> s_page
ed_raw --> s_raw
ed_rev --> s_rev
PeopleUI --> PersonTbl
JobUI --> JobTbl
```
docs/ui/README.md
+43 -63
View File
@@ -1,80 +1,60 @@
# UI Documentation # UI Behavioral Contracts
This folder contains UI-focused design and mapping documents that connect the database schema to user-facing workflows. ## Purpose
## Document Types This directory defines the current user-facing behavior of the NiceGUI application. It records what each page is for, which routes and actions it exposes, what information it presents, and how success, empty, validation, and failure states behave.
### user-journey.md These documents are written for maintainers and AI contributors. They are behavioral contracts, not historical implementation notes and not substitutes for the database schema.
A product and UX contract for a user-facing entity. ## Current Page Contracts
Use this document to describe: - [Home](pages/home.md)
- what the user is trying to do - [Documents](pages/documents.md)
- which screen or action starts the workflow - [People](pages/people.md)
- which fields the user sees and edits - [Jobs](pages/jobs.md)
- validation rules - [Sources](pages/sources.md)
- expected success and failure outcomes
- where the user goes next
### schema-mapping.md NiceGUI registers the routes shown in each contract without the `/ui` prefix. The application mounts NiceGUI under `/ui`, so `/documents` in page code is served to a browser as `/ui/documents`.
A field-level mapping between schema, UI, and implementation. ## Authority Hierarchy
Use this document to describe: When documents disagree, use this order:
- the authoritative schema fields for an entity
- which fields are shown, hidden, editable, or system-managed
- current implementation behavior
- intended target behavior
- implementation gaps between current code and intended UX
### acceptance-criteria.md 1. User-facing page intent and accepted behavior: the page contracts in this directory.
2. Visual and interaction styling: [UI Style Guide](../invariant/ui_style_guide.md).
3. UI dependency and ownership boundaries: [UI contributor instructions](../../.github/instructions/ui.instructions.md).
4. Durable failure behavior: [Error Handling invariant](../invariant/error_handling.md).
5. Durable AI evidence behavior: [Digital Evidence and AI Processing Provenance](../invariant/ai_evidence_and_provenance.md).
6. Data definitions and relationships: current models plus the [V4 schema](../ver4/schema_v4.md).
7. Planned behavior changes: the applicable V4.x scope and implementation documents.
8. Implementation truth: current code and tests.
An implementation-ready checklist for CRUD behavior and quality gates. If code intentionally changes accepted page behavior, update the corresponding page contract in the same change. If code accidentally differs, correct the implementation rather than rewriting intent to match a defect.
Use this document to describe: ## Contract Contents
- testable acceptance statements by flow (Create, Read, Update, Delete)
- success and failure behaviors
- first-release constraints
- cross-criteria quality gates
### traceability-matrix.md Each page contract contains:
A criteria-to-code mapping that identifies implementation anchors and status. 1. Purpose and user goals.
2. Registered routes and navigation context.
3. List, detail, and form behavior.
4. Editable and system-managed information.
5. Validation, empty, loading, and failure states.
6. A concise acceptance checklist.
7. Current implementation and test anchors.
8. Known limitations and deferred work.
Use this document to describe: ## Maintenance Rules
- acceptance criteria group to implementation file mapping
- delivery status (implemented, partial, planned)
- ordered implementation priorities
## Organization Rules - Describe current accepted behavior in present tense.
- Do not mix an obsolete “first release” design with current behavior.
- Keep future changes in versioned scope documents and link to them from a Deferred Work section.
- Do not reproduce the complete database field inventory here; include only fields that affect page behavior.
- Keep service, file, and test anchors current.
- Do not create separate current-state, target-state, and traceability copies of the same contract.
- Keep cross-page visual rules in the UI Style Guide instead of repeating them on each page.
- Keep database joins such as `DocumentPerson` and `JobSource` in schema/architecture documentation unless they directly affect a page interaction.
- Store documents under `docs/ui/entities/<entity-name>/`. ## Current Baseline
- Create both `user-journey.md` and `schema-mapping.md` for user-facing entities.
- Create `acceptance-criteria.md` for user-facing entities.
- Create only `schema-mapping.md` for supporting tables that do not currently have standalone UI.
- Keep one shared `traceability-matrix.md` under `docs/ui/entities/` to map criteria to implementation anchors.
- Keep top-level `docs/` reserved for core architecture, requirements, schema, and system-wide reference material.
## Current Entity Plan These contracts describe the V4 baseline with completed V4.1 UI behavior. Planned V4.2 evidence/provenance changes and draft V4.3 Settings/page-reordering changes are not described as current behavior.
User-facing entities:
- `document`
- `person`
- `source`
- `job`
Supporting entities:
- `document-person`
- `job-source`
## Relationship to Core Docs
These UI docs complement, but do not replace:
- `docs/schema_v2.md`
- `docs/requirements_v2.md`
- `docs/architecture_v2.md`
When there is a conflict:
- schema definitions come from the database model and schema docs
- user interaction intent comes from the user-journey docs
- implementation truth comes from code and is recorded in schema-mapping docs as current-state evidence
docs/ui/entities/document-person/schema-mapping.md
-182
View File
@@ -1,182 +0,0 @@
# DocumentPerson Schema-to-UI Mapping
Purpose: Map the DocumentPerson schema to UI-facing workflows, while separating intended target behavior from current implementation.
Supporting entity note: DocumentPerson does not currently have a standalone UI surface.
## 1. Entity Snapshot
- Table: document_person
- Primary key: id (UUID)
- Related entities: Document, Person
- Canonical schema references:
- src/transcription/db/models.py
- docs/schema_v2.md
## 2. Mapping Rules
This document uses three lenses:
1. Intended behavior: what user-facing workflows should support indirectly.
2. Current behavior: what code supports today.
3. Gap to target: what must change to align implementation with intended UX.
## 3. Field Inventory
| Field | DB Type | Nullable | Default/Auto Value | Intended UI Treatment | Notes |
|---|---|---|---|---|---|
| id | UUID | No | uuid4() | Hidden, system-managed | Primary key |
| document_id | UUID FK | No | None | Context-managed | Selected Document context |
| person_id | UUID FK | No | None | Context-managed | Selected Person context |
| role | enum DocumentPersonRole | No | author | Visible in relationship context | First-release behavior may default to author |
| created_at | datetime | No | datetime.now(UTC) | Hidden or read-only | System-managed timestamp |
Constraint behavior:
1. document_id, person_id, and role are unique as a tuple.
2. duplicate links for the same document, person, and role must be rejected.
## 4. CREATE Mapping
### 4.1 Intended Create Flow
Entry points are indirect through user-facing entities:
1. Document create or update workflows may create one or more DocumentPerson links.
2. Person relationship workflows may create DocumentPerson links.
| Field | Intended User Input | Required | Visible | Notes |
|---|---|---|---|---|
| document_id | None | Yes | No | Derived from selected Document |
| person_id | None | Yes | No | Derived from selected Person |
| role | Select or default | Yes | Indirectly | Defaults to author in first-release behavior |
| created_at | None | No | No | System-generated |
### 4.2 Current Implementation
Current entry point: Document create/edit flows
Current user action: select an existing Person from the Document author dropdown
Current backend path: Document page submit callback -> `DocumentService.create_document_person()` or `delete_document_person()` as the author selection changes
| Field | Current Value at Create | Source | Visible to User | Evidence |
|---|---|---|---|---|
| id | Generated UUID | System | No | src/transcription/db/models.py |
| document_id | Caller-provided | Document UI | Indirectly | src/transcription/ui/pages/documents_page.py |
| person_id | Caller-provided | Document UI | Indirectly | src/transcription/ui/pages/documents_page.py |
| role | Default author in current UI | Service/model default | No | src/transcription/db/models.py, src/transcription/services/documents.py |
| created_at | Current UTC timestamp | System | No | src/transcription/db/models.py |
### 4.3 Gap to Target
To satisfy intended supporting behavior, implementation must add:
1. explicit UI relationship controls in Document and/or Person detail flows.
2. duplicate-link handling with clear user feedback.
3. role-selection UX when role expansion is enabled beyond default author.
## 5. READ Mapping
### 5.1 Intended Read Behavior
Users should see DocumentPerson relationships indirectly in user-facing surfaces:
1. Document detail shows linked people.
2. Person detail shows linked documents.
3. Relationship role is shown where relevant.
### 5.2 Current Implementation
Current read behavior is mainly service-level.
| Field | Current Rendering | Visible to User | Notes | Evidence |
|---|---|---|---|---|
| document_id/person_id link | Indirect relationship usage in workflows | Partial | Document/Person dedicated relationship surfaces are planned | docs/ui/entities/document/*, docs/ui/entities/person/* |
| role | Not shown in current job-centric pages | No | Role expansion is deferred in user-facing workflows | docs/ui/entities/person/user-journey.md |
| created_at | Not rendered | No | Operational metadata only | current UI pages |
Service read/query coverage:
1. read_document_person() returns one link by id.
2. list_document_people() supports filtering by document_id and person_id.
### 5.3 Gap to Target
To satisfy intended read behavior, implementation must add:
1. linked-people and linked-documents UI sections backed by list_document_people().
2. relationship role display where role context is required.
## 6. UPDATE Mapping
### 6.1 Intended Update Behavior
DocumentPerson updates are limited to relationship role or relationship-management actions.
Intended editable fields:
- role (when role management is enabled)
Intended read-only fields:
- id
- document_id
- person_id
- created_at
### 6.2 Current Implementation
| Field | Updatable via UI | Updatable via Service | Notes |
|---|---|---|---|
| role | No | Yes | DocumentService.update_document_person() supports updates |
| document_id/person_id | No | Technically yes via full-row update | Should generally be treated as immutable link identity |
| created_at | No | Technically yes | Should remain system-managed |
### 6.3 Gap to Target
Implementation should add:
1. explicit relationship-role edit controls when product scope enables them.
2. safeguards against mutating link identity instead of recreating links.
## 7. DELETE Mapping
### 7.1 Intended Delete Behavior
Deletion of DocumentPerson should be exposed as unlink behavior in Document and Person flows.
Rules:
1. unlink should remove only the selected relationship.
2. unlink must not delete the underlying Document or Person records.
### 7.2 Current Implementation
| Action | UI Exposed | Backend Capability | Notes |
|---|---|---|---|
| Delete DocumentPerson link | No | Yes | DocumentService.delete_document_person() exists |
### 7.3 Gap to Target
Implementation must add:
1. unlink controls in relationship sections.
2. confirmation and success feedback for relationship removal.
3. blocked-delete guidance if policy constraints are added later.
## 8. Hidden and System-Managed Fields
| Field | Category | Why Hidden or Protected |
|---|---|---|
| id | System-managed | Internal identifier |
| document_id | Context-managed | Derived from selected Document |
| person_id | Context-managed | Derived from selected Person |
| created_at | System-managed | Audit timestamp |
## 9. Traceability Anchors
Schema and models:
- docs/schema_v2.md
- src/transcription/db/models.py
Current implementation:
- src/transcription/services/documents.py
- tests/services/test_v2_crud.py
Related user-facing workflows:
- docs/ui/entities/document/user-journey.md
- docs/ui/entities/person/user-journey.md
## 10. Coverage Summary
- Every DocumentPerson schema field appears in the field inventory.
- Intended behavior is defined as supporting workflow behavior rather than standalone UI.
- Current behavior reflects UI-backed CRUD through Document create/edit flows and Person detail rendering, with no standalone DocumentPerson UI.
- Gaps between intended and current behavior are explicit.
docs/ui/entities/document/acceptance-criteria.md
-136
View File
@@ -1,136 +0,0 @@
# Document Acceptance Criteria
Purpose: Define implementation-ready acceptance criteria for Document Read, Update, and Delete workflows.
Companion documents:
- docs/ui/entities/document/user-journey.md
- docs/ui/entities/document/schema-mapping.md
## Scope
This checklist covers:
1. Read flow
2. Update flow
3. Delete flow
This checklist does not cover:
1. Source upload workflow details
2. Job execution internals
3. Revision editor behavior
## Read Acceptance Criteria
### RD-1 Document detail retrieval
1. Given a valid Document id
2. When the user opens the Document detail page
3. Then the system displays Document metadata for that record only
### RD-2 Metadata visibility
1. The page shows name, document_type, document_date, document_date_raw, location_created, notes, archive_identifier
2. created_at and updated_at are displayed as system-managed, read-only values
### RD-3 Related people section
1. Given zero linked people
2. Then the page shows a no linked people yet empty state
3. Given one linked person
4. Then the page shows that linked person
### RD-4 Sources section empty state
1. The page shows a Sources action for the current Document
2. The page shows a primary + Add Source action that opens job-create flow for this Document
3. The action routes to a document-scoped Sources view
### RD-5 Jobs section empty state
1. The page shows a Jobs action for the current Document
2. The page shows a primary + Add Job action for the current Document
3. The action routes to a document-scoped Jobs view
### RD-6 Filtered navigation readiness
1. The detail page provides links or actions that can route to document-scoped Sources and Jobs views
2. Target views are filtered to the current Document id
### RD-7 Failure state
1. Given a nonexistent Document id
2. Then the UI shows a clear not found state without crashing
## Update Acceptance Criteria
### UP-1 Edit entry
1. Given a loaded Document detail page
2. When the user chooses Edit document
3. Then editable controls are shown for allowed fields only, including the author relationship selector
4. The author selector includes No author, existing Person options, and a Create new item option
5. Selecting Create new item routes to Person create
### UP-2 Editable fields
1. Editable: name, document_type, document_date, document_date_raw, location_created, notes, archive_identifier
2. Not editable: id, created_at, updated_at
3. The edit flow may also change the associated author Person link
### UP-3 Required validation
1. name is required
2. document_type is required
3. Save is blocked with inline feedback when either required field is missing
### UP-4 Date handling rule
1. document_date only is allowed
2. document_date_raw only is allowed
3. both fields together are allowed
4. if both are present, document_date is treated as canonical exact date and document_date_raw is retained as descriptive context
### UP-5 Successful save
1. Given valid input
2. When the user saves
3. Then changes persist
4. Then success feedback is shown
5. Then the user remains on Document detail with refreshed values
6. Then updated_at reflects update policy
### UP-6 Save failure
1. Given backend failure during save
2. Then clear error feedback is shown
3. Then the user-entered values remain available for retry where possible
4. Then no false success feedback is shown
## Delete Acceptance Criteria
### DL-1 Delete entry and confirmation
1. Given a Document detail page
2. When the user chooses Delete document
3. Then a confirmation dialog appears with permanent-action wording
### DL-2 Dependency guardrails
1. Delete is allowed only when the Document has no related Source records and no related Job records
2. Delete is blocked when at least one related Source or Job exists
### DL-3 Blocked delete behavior
1. When blocked
2. Then the UI explains why deletion is blocked
3. Then the UI identifies dependency categories present: Sources, Jobs, or both
4. Then the UI provides navigation to dependency cleanup paths
### DL-4 Successful delete
1. Given no blocking dependencies
2. When the user confirms delete
3. Then the Document is removed
4. Then success feedback is shown
5. Then the user is returned to the Document list page
### DL-5 Delete failure
1. Given backend failure during delete
2. Then a clear error message is shown
3. Then the user remains on Document detail with retry path
## Cross-Criteria Quality Gates
### QG-1 Separation of intent and implementation
1. UX intent remains in user-journey.md
2. Current versus target implementation mapping remains in schema-mapping.md
### QG-2 Traceability
1. Each accepted behavior maps to at least one future UI action or service call path
2. No acceptance criterion contradicts the current deferred-item policy
### QG-3 First-release constraints
1. Linked person during create remains optional
2. Recipient and multi-person expansion remain deferred
docs/ui/entities/document/schema-mapping.md
-231
View File
@@ -1,231 +0,0 @@
# Document Schema-to-UI Mapping
Purpose: Map the Document schema to the UI, while clearly separating intended target behavior from current implementation.
Companion document: user-journey.md
Acceptance criteria: acceptance-criteria.md
## 1. Entity Snapshot
- Table: Document
- Primary key: `id` (UUID)
- Related entities: `Source`, `Job`, `DocumentPerson`, `Person`
- Canonical schema references:
- `src/transcription/db/models.py`
- `docs/schema_v2.md`
## 2. Mapping Rules
This document uses three lenses:
1. Intended behavior: what the UX should support.
2. Current behavior: what the code supports today.
3. Gap to target: what must change to align implementation with the intended UX.
## 3. Field Inventory
| Field | DB Type | Nullable | Default/Auto Value | Intended UI Treatment | Notes |
|---|---|---|---|---|---|
| id | UUID | No | `uuid4()` | Hidden, system-managed | Primary key |
| name | str | No | None | Shown, editable on create and edit | Required |
| document_type | str | Yes | None | Shown, editable on create and edit | Required by intended UX |
| document_date | date | Yes | None | Shown, editable | Canonical exact date when present |
| document_date_raw | str | Yes | None | Shown, editable | Approximate or unknown date text |
| location_created | str | Yes | None | Shown, editable | Optional metadata |
| notes | str | Yes | None | Shown, editable | Optional metadata |
| archive_identifier | str | Yes | None | Shown, editable | Free text in first release |
| created_at | datetime | No | `datetime.now(UTC)` | Hidden or read-only | System-managed |
| updated_at | datetime | No | `datetime.now(UTC)` | Hidden or read-only | System-managed |
## 4. CREATE Mapping
### 4.1 Intended Create Flow
Entry point: Document page
User action: Create new document
Success destination: new Document detail page
| Field | Intended User Input | Required | Visible | Notes |
|---|---|---|---|---|
| name | Text input | Yes | Yes | Primary identifier used by the user |
| document_type | Text input | Yes | Yes | Free text in first release |
| document_date | Date input | No | Yes | Structured exact date |
| document_date_raw | Text input | No | Yes | Approximate or uncertain date |
| location_created | Text input | No | Yes | Optional |
| notes | Text area | No | Yes | Optional |
| archive_identifier | Text input | No | Yes | Free text |
| created_at | None | No | No | System-generated |
| updated_at | None | No | No | Not used during initial create |
Related records during intended create:
- A related person may optionally be selected or created.
- If present, the system creates a `DocumentPerson` link.
- Jobs are not created during Document create.
- Sources are not created during Document create.
### 4.2 Current Implementation
Current entry point: `/documents` page
Current user action: open create form, fill metadata, optionally select an existing Person
Current backend path: document page submit callback -> `DocumentService.create_document()` -> optional `DocumentService.create_document_person()`
| Field | Current Value at Create | Source | Visible to User | Evidence |
|---|---|---|---|---|
| id | Generated UUID | System | No | `Document` default factory in `src/transcription/db/models.py` |
| name | User-provided | User input | Yes | `src/transcription/ui/pages/documents_page.py` |
| document_type | User-provided or None | User input | Yes | `src/transcription/ui/pages/documents_page.py` |
| document_date | Parsed from date input or None | User input | Yes | `src/transcription/ui/pages/documents_page.py` |
| document_date_raw | User-provided or None | User input | Yes | `src/transcription/ui/pages/documents_page.py` |
| location_created | User-provided or None | User input | Yes | `src/transcription/ui/pages/documents_page.py` |
| notes | User-provided or None | User input | Yes | `src/transcription/ui/pages/documents_page.py` |
| archive_identifier | User-provided or None | User input | Yes | `src/transcription/ui/pages/documents_page.py` |
| created_at | Current UTC timestamp | System | No | Default factory in `src/transcription/db/models.py` |
| updated_at | Current UTC timestamp | System | No | Default factory in `src/transcription/db/models.py` |
Current related-record behavior:
- User may optionally select an existing `Person`.
- If selected, `DocumentPerson` is created with role `author`.
- `Job` is not created during Document create.
- `Source` is not created during Document create.
### 4.3 Gap to Target
To satisfy the intended Create flow, implementation now includes:
1. a Document page and dedicated create form
2. user-entered metadata fields for `document_type`, `document_date`, `document_date_raw`, `location_created`, `notes`, and `archive_identifier`
3. optional Person lookup through a dropdown of existing people
4. optional `DocumentPerson` link creation when a person is chosen
5. post-submit routing to a Document detail page
## 5. READ Mapping
### 5.1 Intended Read Behavior
On the Document detail page, the user should be able to see:
1. Document metadata
2. linked people
3. a Sources section with empty-state behavior when no sources exist
4. a Jobs section with empty-state behavior when no jobs exist
5. filtered Jobs and Sources views for the current document
### 5.2 Current Implementation
Current Document visibility in the UI is direct.
| Field | Current Rendering | Visible to User | Notes | Evidence |
|---|---|---|---|---|
| name | Rendered as title and detail heading | Yes | Dedicated Document detail page | `src/transcription/ui/pages/documents_page.py` |
| id | Not shown as raw id | No | Internal identifier remains hidden | `src/transcription/ui/pages/documents_page.py` |
| document_type | Rendered | Yes | Shown on detail and editable on create/edit | `src/transcription/ui/pages/documents_page.py` |
| document_date | Rendered | Yes | Exact date shown when present | `src/transcription/ui/pages/documents_page.py` |
| document_date_raw | Rendered | Yes | Approximate date shown when present | `src/transcription/ui/pages/documents_page.py` |
| location_created | Rendered | Yes | Optional metadata shown | `src/transcription/ui/pages/documents_page.py` |
| notes | Rendered | Yes | Optional metadata shown | `src/transcription/ui/pages/documents_page.py` |
| archive_identifier | Rendered | Yes | Optional metadata shown | `src/transcription/ui/pages/documents_page.py` |
| created_at | Rendered read-only | Yes | System timestamp shown on detail | `src/transcription/ui/pages/documents_page.py` |
| updated_at | Rendered read-only | Yes | System timestamp shown on detail | `src/transcription/ui/pages/documents_page.py` |
### 5.3 Gap to Target
To satisfy the intended Read flow, implementation now includes:
1. metadata rendering for Document fields
2. linked people rendering
3. document-scoped Sources and Jobs navigation views
## 6. UPDATE Mapping
### 6.1 Intended Update Behavior
The user should eventually be able to edit Document metadata from the Document detail page or a dedicated edit flow.
Intended editable fields:
- `name`
- `document_type`
- `document_date`
- `document_date_raw`
- `location_created`
- `notes`
- `archive_identifier`
Intended system-managed fields:
- `id`
- `created_at`
- `updated_at`
### 6.2 Current Implementation
| Field | Updatable via UI | Updatable via Service | Notes |
|---|---|---|---|
| id | No | Practically no | Primary key should be treated as immutable |
| name | Yes | Yes | Editable from dedicated document edit page via `DocumentService.update_document()` |
| document_type | Yes | Yes | Editable from dedicated document edit page via `DocumentService.update_document()` |
| document_date | Yes | Yes | Editable from dedicated document edit page via `DocumentService.update_document()` |
| document_date_raw | Yes | Yes | Editable from dedicated document edit page via `DocumentService.update_document()` |
| location_created | Yes | Yes | Editable from dedicated document edit page via `DocumentService.update_document()` |
| notes | Yes | Yes | Editable from dedicated document edit page via `DocumentService.update_document()` |
| archive_identifier | Yes | Yes | Editable from dedicated document edit page via `DocumentService.update_document()` |
| created_at | No | Technically yes | Should remain system-managed |
| updated_at | No | Technically yes | Should remain system-managed |
### 6.3 Gap to Target
Implementation now includes:
1. Document edit controls in the UI
2. validation and save behavior for Document metadata
3. author relationship controls through the edit flow
## 7. DELETE Mapping
### 7.1 Intended Delete Behavior
The UI should eventually provide a delete action for Document with guardrails.
Rules:
1. A Document can be deleted when it has no attached Jobs and no attached Sources.
2. If dependent Jobs or Sources exist, the UI should block deletion and explain that those related records must be removed first.
3. Delete confirmation should make it clear that the action is permanent.
### 7.2 Current Implementation
| Action | UI Exposed | Backend Capability | Notes |
|---|---|---|---|
| Delete Document | Yes | Yes | `DocumentService.delete_document()` exists and the UI blocks dependent deletes |
### 7.3 Gap to Target
Implementation includes:
1. a Document delete control in the UI
2. pre-delete dependency checks for Jobs and Sources
3. user-facing messaging when deletion is blocked
4. confirmation UX for successful delete attempts
## 8. Hidden and System-Managed Fields
| Field | Category | Why Hidden or Protected |
|---|---|---|
| id | System-managed | Internal identifier |
| created_at | System-managed | Audit timestamp |
| updated_at | System-managed | Audit timestamp |
## 9. Traceability Anchors
Schema and models:
- `docs/schema_v2.md`
- `src/transcription/db/models.py`
Current implementation:
- `src/transcription/ui/pages/documents_page.py`
- `src/transcription/services/documents.py`
- `src/transcription/services/store.py`
- `src/transcription/ui/pages/jobs_page.py`
- `src/transcription/ui/components/transcript.py`
Companion UX spec:
- `docs/ui/entities/document/user-journey.md`
## 10. Acceptance Checklist Summary
- Every Document schema field appears in the field inventory.
- Intended Create behavior matches the companion user journey.
- Current Create behavior reflects the existing upload-driven implementation.
- Gaps between intended and current behavior are explicit.
- Read, Update, and Delete sections distinguish target behavior from current code.
docs/ui/entities/document/user-journey.md
-427
View File
@@ -1,427 +0,0 @@
# Document User Journey
Purpose: Define how a user should interact with the UI to create and manage a Document record, including expected inputs, validation, results, and related record creation.
Scope: This document describes intended user interaction for the Document UI. It is the UX contract for the Document entity.
Companion schema mapping: schema-mapping.md
Companion acceptance criteria: acceptance-criteria.md
## 1. Overview
A Document represents a real historical artifact the user wants to describe, organize, and eventually transcribe. The user should be able to create a Document before uploading or linking any source files.
Creating a Document is a metadata-first workflow:
1. The user opens the Document page.
2. The user selects Create new document.
3. The user enters descriptive metadata about the document.
4. The user optionally selects one related person from the existing Person list.
5. The system creates the Document.
6. If a person was selected, the system links that Person to the Document through DocumentPerson with author role.
7. The user sees a success state and lands on the new Document detail page.
## 2. User Goal
The user wants to create a new Document record that:
1. Has enough metadata to identify the historical artifact.
2. Can optionally be linked to a person.
3. Exists independently of transcription jobs and source uploads.
4. Is ready for later steps such as adding sources, starting jobs, and reviewing transcriptions.
## 3. Page Model
### 3.1 Document Page
The Document page is the general UI surface where users manage documents.
It should support:
1. listing or locating existing documents
2. starting the Create new document flow
3. navigating into a specific Document after it exists
### 3.2 Document Detail Page
The Document detail page is the page for one specific Document after it has been created.
It should show:
1. the Document metadata
2. related people linked to the Document
3. a linked-author summary when available
4. document-scoped navigation links for Sources and Jobs
5. filtered views for sources and jobs linked to the current document
6. primary actions + Add Source and + Add Job
## 4. Entry Point
Entry point: Document page
Primary action: Create new document
Expected UI affordance:
1. A visible button, link, or primary action labeled Create new document.
2. Activation opens a dedicated form view, modal, or detail panel for creating a Document.
Preferred first implementation:
1. A dedicated Document create page or panel.
2. A simple form with explicit labels.
3. Existing Person records should be selectable through a dropdown.
4. Text inputs are acceptable for the remaining fields in first release.
## 5. Create Document Form
The Create Document form should contain the following fields.
### 5.1 Required Fields
| UI Label | Schema Field | Input Type | Required | Notes |
|---|---|---|---|---|
| Document name | name | Text input | Yes | Examples: Pioneer Days, Letter from Zenna to Omie |
| Document type | document_type | Text input | Yes | Examples: book, letter, enlistment papers, military record, other |
### 5.2 Date Fields
| UI Label | Schema Field | Input Type | Required | Notes |
|---|---|---|---|---|
| Exact date | document_date | Date input | No | Use when the exact date is known |
| Approximate date | document_date_raw | Text input | No | Use when exact date is uncertain, approximate, or unknown |
Date handling rule:
1. The form may allow both fields to be entered.
2. If both fields are entered, `document_date` is the canonical structured date.
3. `document_date_raw` may still be retained as the user-entered descriptive form.
4. The UI should explain the distinction clearly.
Examples:
1. Exact date: `07/13/1885`
2. Approximate date: `c. 1885`
3. Approximate date: `Fall 1925`
4. Approximate date: `unknown`
### 5.3 Optional Metadata Fields
| UI Label | Schema Field | Input Type | Required | Notes |
|---|---|---|---|---|
| Document location | location_created | Text input | No | Where the document was created |
| Notes | notes | Multiline text area | No | Freeform notes about the document |
| Archive identifier | archive_identifier | Text input | No | Free text for now; may represent inventory code, storage reference, or repository note |
Archive identifier guidance:
1. First implementation should treat this as free text.
2. Helper text may explain that this can store a repository code, box or folder reference, or storage note.
### 5.4 System Fields
| Schema Field | User Editable | Notes |
|---|---|---|
| created_at | No | System-generated at creation time |
| updated_at | No | Not user-entered during creation |
### 5.5 Optional Related Person
The Create Document flow may optionally link one related person during first release.
| UI Label | Schema Area | Input Type | Required | Notes |
|---|---|---|---|---|
| Related person | Person -> DocumentPerson | Dropdown select | No | Selects an existing Person and links as author when saved |
First release behavior:
1. The user may save a Document without linking any person.
2. If a person is linked during create, only one person is supported in first release.
3. The selected person is linked as author.
4. Additional people and recipient workflows are deferred to a future revision.
### 5.6 Related Records Not Created Directly Here
| Related Area | Included in Document Create | Notes |
|---|---|---|
| Jobs | No | Jobs are created later when transcription work begins |
| Sources | No | Sources are added later as uploaded pages or files |
## 6. Related Person Workflow
### 6.1 User Intent
The user should be able to:
1. select an existing Person to associate with the Document
2. change the associated Person from the Document edit flow
3. save the Document even if no person is linked
### 6.2 Data Model Interpretation
Person selection source:
1. The UI should select from Person records.
2. If a person is linked, the system should create a DocumentPerson record.
3. Role handling for non-author document relationships is deferred.
4. If the first release needs a persisted role immediately, the role can default to `author` until the relationship model is broadened.
This means:
1. The user does not choose from DocumentPerson records.
2. DocumentPerson is the relationship created after the Person is chosen or created.
### 6.3 Related Person UI Behavior
Minimum acceptable first implementation:
1. Dropdown of existing Person records.
2. Clear display of the selected related person before submit.
3. Ability to change or clear the selected person in the Document edit flow.
4. A Create new item option in the author selector that routes to Person create.
5. A visible Create new person link near the selector.
If the person does not exist:
1. The user can use Create new item from the author selector and continue from Person create.
2. The Document create flow links existing Person records after selection.
## 7. Validation Rules
### 7.1 Required Field Validation
The form must reject submission if:
1. `name` is empty
2. `document_type` is empty
### 7.2 Date Validation
The form should allow:
1. `document_date` only
2. `document_date_raw` only
3. both `document_date` and `document_date_raw`
4. neither date field
If both are present:
1. `document_date` is treated as the canonical exact date
2. `document_date_raw` is retained as descriptive context
### 7.3 Related Person Validation
The form must not require a linked person in first release.
If a related person is selected or created:
1. the selected value must resolve to a valid Person record before final save
2. the DocumentPerson link must not be partially persisted on failure
## 8. Submission Behavior
When the user submits the form, the system should perform these logical steps:
1. validate form inputs
2. create the Document record
3. create one DocumentPerson record only if an existing related person was selected
4. persist intended records successfully before reporting success to the user
Expected write sequence:
1. insert Document
2. insert DocumentPerson link only if a person is linked
Recommended transactional behavior:
1. Document and optional DocumentPerson writes should succeed or fail together
2. Person creation is a separate workflow reached from the author selector and is not part of the same transaction
## 9. Expected Result After Success
After successful creation, the user should expect to see:
1. confirmation that the Document was created successfully
2. the Document name displayed in the resulting UI state
3. the Document metadata displayed on the new Document detail page
4. any linked person displayed in the resulting UI state
5. a Sources section showing an empty state when no sources exist yet
6. a Jobs section showing an empty state when no jobs exist yet
7. a clear next step, such as adding source files
Recommended success route:
1. navigate to the new Document detail page
2. show Document summary metadata
3. show linked people section
4. show empty-state placeholders for Sources and Jobs
## 10. Expected Result After Failure
If submission fails, the user should expect:
1. clear error messaging
2. field-level validation feedback where applicable
3. no false success message
4. preservation of entered form values when possible
Examples:
1. missing required name
2. missing required document type
3. failed person creation
4. failed DocumentPerson link creation
5. database or server error
## 11. Read Document Journey
### 11.1 User Intent
The user wants to open a specific Document and quickly understand:
1. what the document is
2. which people are linked to it
3. whether sources exist
4. whether jobs exist
5. what the next action should be
### 11.2 Entry Points
A user can reach a Document detail page by:
1. selecting a document from the Document page list
2. being redirected after successfully creating a new document
3. following a direct link to a known Document record
### 11.3 Document Detail Layout
The Document detail page should include:
1. a header area with document name, document type, and key date values
2. a metadata section with location_created, notes, and archive_identifier
3. System metadata where created_at and updated_at are shown as read-only values
4. a related people section
5. a Sources section
6. a Jobs section
The Document detail page should support:
1. empty-state messaging when no related records exist
2. clear next actions from each empty state
3. filtered Sources and Jobs views scoped to the current document
### 11.4 Read Empty States
If no related records exist:
1. People section says no linked people yet
2. Sources section says no sources added yet
3. Jobs section says no jobs created yet
4. each section presents one clear next action
### 11.5 Read Success Criteria
A successful Read experience means:
1. The user can identify the Document immediately
2. The user can see whether work has started
3. The user can navigate directly to document-scoped Jobs and Sources workflows
## 12. Update Document Journey
### 12.1 User Intent
The user wants to correct or enrich metadata after creation without touching jobs or source transcriptions directly.
### 12.2 Update Entry Point
From the Document detail page:
1. The user selects Edit document
2. UI opens edit mode or a dedicated edit view
### 12.3 Editable Fields
First release editable fields:
1. name
2. document_type
3. document_date
4. document_date_raw
5. location_created
6. notes
7. archive_identifier
Read-only or system-managed fields:
1. id
2. created_at
3. updated_at
### 12.4 Update Validation Rules
1. name remains required
2. document_type remains required
3. document_date and document_date_raw may both be present
4. if both date fields are present, document_date remains canonical
5. validation errors should be shown inline and block save
### 12.5 Update Save Behavior
On save:
1. system validates form data
2. system persists Document updates
3. updated_at is refreshed by system policy
4. UI shows a confirmation message
5. user remains on Document detail page with refreshed values
### 12.6 Update Failure Behavior
If save fails:
1. Show a clear error message
2. keep user edits in form where possible
3. do not show stale success messaging
4. Allow retry without losing context
## 13. Delete Document Journey
### 13.1 User Intent
The user wants to remove a Document only when it is safe and unambiguous.
### 13.2 Delete Entry Point
From the Document detail page:
1. The user selects Delete document
2. UI opens a confirmation dialog explaining permanence
### 13.3 Delete Guardrails
Delete is allowed only when:
1. the Document has no related Source records
2. the Document has no related Job records
Delete is blocked when:
1. any Source exists for the Document
2. any Job exists for the Document
### 13.4 Blocked Delete UX
When blocked:
1. Show an explicit reason that related Jobs or Sources exist
2. Show which dependency types are present
3. provide links to filtered Sources and Jobs for cleanup
4. keep the Document unchanged
### 13.5 Allowed Delete UX
When allowed:
1. Show final confirmation with document name
2. perform delete
3. show success confirmation
4. return user to Document page list
### 13.6 Delete Failure Behavior
If delete fails due to system error:
1. Show a clear error message
2. keep user on Document detail page
3. preserve ability to retry
## 14. Non-Goals for This Flow
The Document journey does not define:
1. Source upload field-level UX
2. Job execution internals
3. revision editor behavior for transcriptions
4. multi-person recipient workflows in first release
## 15. Relationship to Other Workflows
This Document workflow integrates with:
1. Sources workflow for adding pages or files to the document
2. Jobs workflow for transcription execution
3. Person workflow for future expansion beyond one optional linked person
## 16. Relationship to Schema Mapping
This document is the intended UX contract.
The companion schema-mapping document should answer:
1. which schema field appears on which screen
2. whether the field is currently implemented
3. whether the field is hidden, editable, or system-managed
4. what the implementation gap is between intended UX and current code
## 17. Deferred Items
These topics are intentionally deferred to future revisions:
1. multiple linked people during create and update
2. recipient support during create and update
3. a broader role model for non-author document relationships
4. filtered Jobs and Sources list navigation details
docs/ui/entities/job-source/schema-mapping.md
-206
View File
@@ -1,206 +0,0 @@
# JobSource Schema-to-UI Mapping
Purpose: Map the JobSource schema to UI-facing workflows, while separating intended target behavior from current implementation.
Supporting entity note: JobSource does not currently have a standalone UI surface.
## 1. Entity Snapshot
- Table: job_source
- Primary key: id (UUID)
- Related entities: Job, Source
- Canonical schema references:
- src/transcription/db/models.py
- docs/schema_v2.md
## 2. Mapping Rules
This document uses three lenses:
1. Intended behavior: what user-facing workflows should support indirectly.
2. Current behavior: what code supports today.
3. Gap to target: what must change to align implementation with intended UX.
## 3. Field Inventory
| Field | DB Type | Nullable | Default/Auto Value | Intended UI Treatment | Notes |
|---|---|---|---|---|---|
| id | UUID | No | uuid4() | Hidden, system-managed | Primary key |
| job_id | UUID FK | No | None | Context-managed | Selected Job context |
| source_id | UUID FK | No | None | Context-managed | Selected Source context |
| status | enum JobSourceStatus | No | pending | Shown in job detail source context | Per-source execution state |
| raw_transcription | str | Yes | None | Shown read-only in review context | Machine output per source |
| ai_metadata | JSONB/JSON | Yes | None | Hidden or advanced diagnostics | Provider metadata |
| raw_api_response | JSONB/JSON | Yes | None | Hidden or advanced diagnostics | Low-level provider payload |
| error_detail | str | Yes | None | Shown when status is failed | Execution failure details |
| executed_at | datetime | No | datetime.now(UTC) | Shown read-only | Execution timestamp |
## 4. CREATE Mapping
### 4.1 Intended Create Flow
JobSource creation is indirect through Job and transcription workflows:
1. Job create flow should create a JobSource row for each uploaded source page.
2. Processing workflow may create missing JobSource rows when persisting transcription output.
| Field | Intended User Input | Required | Visible | Notes |
|---|---|---|---|---|
| job_id | None | Yes | No | Derived from active Job |
| source_id | None | Yes | No | Derived from created/selected Source |
| status | None | No | Indirectly | Defaults to pending at create |
| raw_transcription | None | No | No at create | Filled after processing |
| ai_metadata | None | No | No | Operational metadata |
| raw_api_response | None | No | No | Operational payload |
| error_detail | None | No | No at create | Filled on failure |
| executed_at | None | No | No | System-generated |
### 4.2 Current Implementation
Current entry points:
1. upload create path adds pending JobSource link in _create_upload_records().
2. transcription update path creates or updates JobSource row during output persistence.
Current backend paths:
1. src/transcription/services/store.py -> _create_upload_records()
2. src/transcription/services/transcription.py -> update_job_transcription()
| Field | Current Value at Create/Update | Source | Visible to User | Evidence |
|---|---|---|---|---|
| id | Generated UUID | System | No | src/transcription/db/models.py |
| job_id | Caller or workflow derived | Service/workflow | Indirectly | store.py, transcription.py |
| source_id | Caller or workflow derived | Service/workflow | Indirectly | store.py, transcription.py |
| status | pending at create, transcribed or failed on update | Workflow logic | Partial | transcription.py |
| raw_transcription | Set on successful transcription update | Workflow/provider result | Yes in review context | transcription.py, jobs UI |
| ai_metadata | Available in model; not currently filled in update path | Workflow potential | No | models.py, transcription.py |
| raw_api_response | Available in model; not currently filled in update path | Workflow potential | No | models.py, transcription.py |
| error_detail | Set on failed transcription update | Workflow/provider error | Partial | transcription.py |
| executed_at | Set at row creation and refreshed on updates | System/workflow | Partial | models.py, transcription.py |
### 4.3 Gap to Target
To satisfy intended supporting behavior, implementation must add:
1. explicit per-source status display for all linked sources in Job detail.
2. clear surfaced error_detail for failed source executions.
3. optional diagnostics surface for ai_metadata/raw_api_response when needed.
4. first-class multi-source create path from Job create flow.
## 5. READ Mapping
### 5.1 Intended Read Behavior
Users should see JobSource data indirectly in job detail and review workflows:
1. per-source execution status.
2. per-source raw transcription output.
3. per-source failure details where applicable.
4. execution timestamp context.
### 5.2 Current Implementation
Current read behavior is partial and job-detail-centric.
| Field | Current Rendering | Visible to User | Notes | Evidence |
|---|---|---|---|---|
| status | Job-level status is visible; source-level status is limited | Partial | Source-level status not fully surfaced as a dedicated list | src/transcription/ui/pages/jobs_page.py |
| raw_transcription | Original transcription card is visible | Yes | Primary source is shown in current detail flow | src/transcription/ui/components/transcript.py |
| error_detail | Not prominently surfaced in current detail UI | Partial | Stored in JobSource rows during failures | src/transcription/services/transcription.py |
| executed_at | Not first-class rendered | Partial | Available in model for future display | src/transcription/db/models.py |
Service read/query coverage:
1. read_job_source() reads one row with source relation.
2. list_job_sources() lists rows and supports job_id filtering.
### 5.3 Gap to Target
To satisfy intended read behavior, implementation must add:
1. source-level execution table in Job detail.
2. explicit failed-source messaging from error_detail.
3. multi-source navigation in job review UI.
## 6. UPDATE Mapping
### 6.1 Intended Update Behavior
JobSource updates are workflow-managed, not directly user-edited.
Intended user-editable fields:
- none in first-release behavior
Workflow-managed fields:
- status
- raw_transcription
- error_detail
- executed_at
- optional diagnostics payload fields
### 6.2 Current Implementation
| Field | Updatable via UI | Updatable via Service/Workflow | Notes |
|---|---|---|---|
| status | No | Yes | Set by transcription update and job lifecycle handling |
| raw_transcription | No | Yes | Persisted in update_job_transcription() |
| error_detail | No | Yes | Persisted on transcription failure |
| executed_at | No | Yes | Updated when existing JobSource rows are changed |
| ai_metadata/raw_api_response | No | Potentially yes | Model supports them; active population is limited |
### 6.3 Gap to Target
Implementation should add:
1. clearer job-detail visualization of per-source execution updates.
2. optional operator diagnostics views for advanced troubleshooting.
## 7. DELETE Mapping
### 7.1 Intended Delete Behavior
JobSource deletion should be policy-driven and usually tied to Job/Source lifecycle operations.
Rules:
1. direct user deletion is not required in first-release behavior.
2. cleanup should occur through Job or Source deletion policies.
### 7.2 Current Implementation
| Action | UI Exposed | Backend Capability | Notes |
|---|---|---|---|
| Delete JobSource row | No | Yes | TranscriptionService.delete_job_source() exists |
### 7.3 Gap to Target
Implementation may add:
1. maintenance tooling for cleanup operations.
2. policy-aware cascade guidance in Job and Source delete flows.
## 8. Hidden and System-Managed Fields
| Field | Category | Why Hidden or Protected |
|---|---|---|
| id | System-managed | Internal identifier |
| job_id | Context-managed | Derived from Job context |
| source_id | Context-managed | Derived from Source context |
| ai_metadata | Operational metadata | Advanced diagnostics payload |
| raw_api_response | Operational metadata | Raw provider response payload |
| executed_at | System-managed | Execution timestamp |
## 9. Traceability Anchors
Schema and models:
- docs/schema_v2.md
- src/transcription/db/models.py
Current implementation:
- src/transcription/services/store.py
- src/transcription/services/transcription.py
- src/transcription/services/workflows.py
- src/transcription/ui/pages/jobs_page.py
- src/transcription/ui/components/transcript.py
- tests/services/test_v2_crud.py
Related user-facing workflows:
- docs/ui/entities/job/user-journey.md
- docs/ui/entities/source/user-journey.md
## 10. Coverage Summary
- Every JobSource schema field appears in the field inventory.
- Intended behavior is defined as supporting workflow behavior rather than standalone UI.
- Current behavior reflects workflow/service-driven CRUD with partial job-detail visibility.
- Gaps between intended and current behavior are explicit.
docs/ui/entities/job/acceptance-criteria.md
-154
View File
@@ -1,154 +0,0 @@
# Job Acceptance Criteria
Purpose: Define implementation-ready acceptance criteria for Job Create, Read, Update, and Delete workflows.
Companion documents:
- docs/ui/entities/job/user-journey.md
- docs/ui/entities/job/schema-mapping.md
## Scope
This checklist covers:
1. Create flow
2. Read flow
3. Update flow
4. Delete flow
This checklist does not cover:
1. provider-specific transcription internals
2. advanced workflow scheduling and queue orchestration controls
3. multi-job bulk operations
## Create Acceptance Criteria
### CR-1 Job creation entry
1. Given the user is on the Jobs page
2. When the user selects Create job
3. Then the user is taken to Job detail/create mode
### CR-2 Required create values
1. document_id must be selected before submit
2. at least one source file must be uploaded before submit
3. each uploaded file creates a Source linked to the selected Document
4. each created Source is linked to the new Job through JobSource
### CR-3 Source ordering behavior
1. Given multi-file or folder upload
2. When source records are created
3. Then page ordering follows alphabetical order of original filenames
4. Then helper text explains how filename conventions control ordering
### CR-4 Provider/model/prompt visibility
1. provider, model, and prompt_name are visible in create flow when known
2. provider, model, and prompt_name are visible in detail flow when known
3. if values are unknown at create time, UI shows clear unknown or pending state without blocking submit
### CR-5 Successful create outcome
1. Given valid inputs
2. When the user submits create
3. Then the Job record is created and linked to selected Document
4. Then source and JobSource records are created for uploads
5. Then job status is queued or processing based on execution timing
6. Then the user is routed to Job detail mode
### CR-6 Create failure outcome
1. Given create validation or persistence failure
2. Then clear error feedback is shown
3. Then no false success feedback is shown
4. Then entered selections are preserved where possible
5. Then retry path remains available
## Read Acceptance Criteria
### RD-1 Jobs list retrieval
1. Given one or more jobs exist
2. When the user opens the Jobs page
3. Then all jobs are listed in a table or equivalent list surface
### RD-2 Jobs list fields
1. Jobs list shows job id
2. Jobs list shows status
3. Jobs list shows created or updated timestamps
4. Jobs list shows retry_count when available
5. Jobs list provides navigation to Job detail for each row
### RD-3 Job detail retrieval
1. Given a valid job id
2. When the user opens Job detail
3. Then job metadata for that record only is shown
4. Then document-scoped navigation links for Sources and Jobs are shown
### RD-4 Detail execution context visibility
1. provider, model, and prompt_name are displayed when known
2. status lifecycle value is visible
3. source-level transcription and revision context is available through Source detail navigation from Job detail
### RD-5 Missing and invalid id states
1. Given an invalid job id format
2. Then UI shows invalid job id state without crashing
3. Given a valid but nonexistent job id
4. Then UI shows job not found state without crashing
## Update Acceptance Criteria
### UP-1 Revision edit entry
1. Given a job detail page
2. When the user opens the page
3. Then navigation links to job-scoped Sources are available
4. Then source rows can open Source detail revision workflow
### UP-2 Revision validation
1. revision save blocks empty trimmed text and shows warning feedback
### UP-3 Successful revision save
1. Source detail save persists revised text and shows success feedback
### UP-4 Revision save failure
1. Source detail save failure shows clear error feedback with retry path
### UP-5 Job lifecycle state update visibility
1. status changes from queued to processing to terminal states are reflected in UI
2. retry_count updates are reflected when retry logic runs
3. users cannot directly edit lifecycle state fields in first release
## Delete Acceptance Criteria
### DL-1 Delete entry and confirmation
1. Given a job detail context
2. When the user opens job delete page
3. Then a permanent-action confirmation is shown for non-processing jobs
### DL-2 Dependency guardrails
1. Delete is blocked while job status is processing
2. Related JobSource links are removed as part of allowed delete flow
### DL-3 Blocked delete behavior
1. When blocked, the UI shows clear processing-state guidance
2. The user is offered navigation back to job or jobs list
### DL-4 Successful delete
1. Given an allowed delete
2. When the user confirms delete
3. Then the job is removed and success feedback is shown
4. Then the user is returned to Jobs list
### DL-5 Delete failure
1. Given backend failure during delete
2. Then clear error feedback is shown
3. Then the user remains in delete context with retry path
## Cross-Criteria Quality Gates
### QG-1 Separation of intent and implementation
1. UX intent remains in user-journey.md
2. Current versus target implementation mapping remains in schema-mapping.md
### QG-2 Traceability
1. Each accepted behavior maps to at least one UI action or service path
2. No acceptance criterion contradicts first-release deferred items
### QG-3 First-release constraints
1. Jobs page remains list-all with explicit Create job action
2. Job create requires Document selection and source upload
3. provider/model/prompt_name are visible to users when known
4. manual retry controls may remain deferred while status visibility is required
docs/ui/entities/job/schema-mapping.md
-233
View File
@@ -1,233 +0,0 @@
# Job Schema-to-UI Mapping
Purpose: Map the Job schema to the UI, while clearly separating intended target behavior from current implementation.
Companion document: user-journey.md
Acceptance criteria: acceptance-criteria.md
## 1. Entity Snapshot
- Table: Job
- Primary key: id (UUID)
- Related entities: Document, JobSource, Source
- Canonical schema references:
- src/transcription/db/models.py
- docs/schema_v2.md
## 2. Mapping Rules
This document uses three lenses:
1. Intended behavior: what the UX should support.
2. Current behavior: what the code supports today.
3. Gap to target: what must change to align implementation with the intended UX.
## 3. Field Inventory
| Field | DB Type | Nullable | Default/Auto Value | Intended UI Treatment | Notes |
|---|---|---|---|---|---|
| id | UUID | No | uuid4() | Shown read-only in list and detail | Primary key |
| document_id | UUID FK | No | None | Required create input via Document selection | Job belongs to one Document |
| status | enum JobStatus | No | queued | Shown read-only as lifecycle state | System-managed transitions |
| retry_count | int | No | 0 | Shown read-only | Operational counter |
| date_created | datetime | No | datetime.now(UTC) | Shown read-only | System-managed timestamp |
| date_updated | datetime | No | datetime.now(UTC) | Shown read-only | System-managed timestamp |
| provider | str | Yes | None | Visible when known; editable if create-time options are available | Processing metadata |
| model | str | Yes | None | Visible when known; editable if create-time options are available | Processing metadata |
| prompt_name | str | Yes | None | Visible when known; editable if create-time options are available | Prompt metadata |
Related execution fields rendered in Job detail via relationships:
- Job detail renders metadata and document links; source-level review/editing is reached through job-scoped Sources routes.
## 4. CREATE Mapping
### 4.1 Intended Create Flow
Entry point: Jobs page Create job action
User action: open create mode, select Document, upload one or more source files or a folder, submit for transcription
Success destination: Job detail page in detail mode
| Field | Intended User Input | Required | Visible | Notes |
|---|---|---|---|---|
| document_id | Select/search | Yes | Yes | Required create selection |
| status | None | No | Yes (read-only) | Starts at queued and changes by workflow |
| retry_count | None | No | Yes (read-only) | Starts at 0 |
| date_created | None | No | Yes (read-only) | System-generated |
| date_updated | None | No | Yes (read-only) | System-generated |
| provider | Display or select | No | Yes | Visible when known during create and detail |
| model | Display or select | No | Yes | Visible when known during create and detail |
| prompt_name | Display or select | No | Yes | Visible when known during create and detail |
Create-related relationship rules:
1. source file upload is required for create.
2. each uploaded file creates a Source linked to the selected Document.
3. each created Source must be linked to the new Job through JobSource.
4. processing order for multi-file and folder uploads is alphabetical by original filename.
### 4.2 Current Implementation
Current entry point: Jobs page create flow
Current user action: select Document and upload one or more files or a folder through a single upload widget
Current backend path: job create submit -> create_job_for_document()
| Field | Current Value at Create | Source | Visible to User | Evidence |
|---|---|---|---|---|
| id | Generated UUID | System | Yes on jobs list/detail | src/transcription/ui/pages/jobs_page.py |
| document_id | Selected existing Document id | User selection + service write | Indirectly | src/transcription/ui/pages/jobs_page.py, src/transcription/services/store.py |
| status | queued | Service/model default | Yes | src/transcription/services/store.py, src/transcription/db/models.py |
| retry_count | 0 | Model default | Yes | src/transcription/db/models.py, src/transcription/ui/pages/jobs_page.py |
| date_created | current UTC timestamp | System | Yes | src/transcription/db/models.py, src/transcription/ui/pages/jobs_page.py |
| date_updated | current UTC timestamp | System | Yes | src/transcription/db/models.py, src/transcription/ui/pages/jobs_page.py |
| provider | None at create, set after transcription update | Workflow/service | Yes | src/transcription/services/workflows.py |
| model | None at create, set after transcription update | Workflow/service | Yes | src/transcription/services/workflows.py |
| prompt_name | None at create, set by workflow updates | Workflow/service | Yes | src/transcription/services/workflows.py |
Current create constraints:
1. dedicated Create job action exists in the Jobs page.
2. job create flow requires a Document selection.
3. current upload path accepts one widget for files or folder selection.
### 4.3 Gap to Target
To satisfy intended Create flow, implementation must add:
1. Jobs list Create job action that opens Job detail/create mode.
2. explicit Document selection and source upload controls in create mode.
3. multi-file and folder upload support in create mode.
4. deterministic alphabetical page ordering and user guidance.
5. explicit visibility of provider, model, and prompt_name in create/detail when known.
## 5. READ Mapping
### 5.1 Intended Read Behavior
On Job list/detail surfaces, users should be able to see:
1. all jobs in one list.
2. status and timeline context.
3. selected Document context.
4. source-level processing and transcription results.
5. provider/model/prompt_name when known.
### 5.2 Current Implementation
Current read behavior exists in jobs list and jobs detail routes.
| Field | Current Rendering | Visible to User | Notes | Evidence |
|---|---|---|---|---|
| id | Jobs list row and detail header | Yes | Primary visible identifier | src/transcription/ui/pages/jobs_page.py |
| status | Jobs list and detail | Yes | Chip styling for transcribed; text for others | src/transcription/ui/pages/jobs_page.py |
| retry_count | Jobs list table | Yes | Included in row model | src/transcription/ui/components/table/jobs.py |
| date_created | Jobs list table | Yes | Included in row model | src/transcription/ui/components/table/jobs.py |
| date_updated | Jobs list table | Yes | Included in row model | src/transcription/ui/components/table/jobs.py |
| document_id | Not rendered directly as labeled field | Partial | Document context exists by relationship but limited direct display | src/transcription/ui/pages/jobs_page.py |
| provider/model/prompt_name | Rendered as labeled fields in Job detail | Yes | Shows pending fallback when unset | src/transcription/ui/pages/jobs_page.py |
Source-related read behavior:
1. Job detail exposes Sources navigation for current job context.
2. Source preview, transcription context, and revision editor are rendered in Source detail.
3. invalid or missing job ids show explicit UI states.
### 5.3 Gap to Target
To satisfy intended Read flow, implementation must add:
1. optional in-page source summaries in Job detail if future UX requires fewer navigation steps.
2. richer filtering/search UX if needed.
## 6. UPDATE Mapping
### 6.1 Intended Update Behavior
Primary user updates in first release are source revision edits in Source detail reached from Job detail.
Intended editable scope (first release):
- Source.revised_text through Source detail review
Intended read-only Job fields in first release:
- id
- document_id after create
- status
- retry_count
- date_created
- date_updated
Job metadata visibility policy:
- provider, model, and prompt_name should be visible when known.
- create-time editing of provider/model/prompt_name is optional and depends on available options.
### 6.2 Current Implementation
| Field/Area | Updatable via UI | Updatable via Service | Notes |
|---|---|---|---|
| Source.revised_text from Source detail | Yes | Yes | Saved via transcription service revision path from Sources page detail route |
| status | No | Yes | Updated by workflow lifecycle services |
| retry_count | No | Yes | Incremented by workflow retry logic |
| provider/model/prompt_name | No | Yes | Set during transcription result finalization |
| document_id | No | Technically via model/service update | Treated as fixed post-create in intended UX |
### 6.3 Gap to Target
Implementation now includes:
1. create-mode handling for provider/model/prompt visibility and optional selection.
2. detail display for provider/model/prompt and document-scoped navigation links.
3. source revision workflow through job-scoped Sources and Source detail pages.
4. manual controls for retry and state transitions remain deferred.
## 7. DELETE Mapping
### 7.1 Intended Delete Behavior
Job deletion is implemented as a dedicated delete route with processing-state guardrails.
Rules:
1. deletion is allowed only when policy allows cleanup or retention handling for related JobSource records.
2. blocked deletion must explain constraints and required cleanup path.
3. successful deletion requires confirmation and returns user to Jobs list.
### 7.2 Current Implementation
| Action | UI Exposed | Backend Capability | Notes |
|---|---|---|---|
| Delete Job | Yes | Yes | Job delete page confirms permanent action and blocks when processing |
### 7.3 Gap to Target
Implementation may add in a future revision:
1. inline delete entry in Job detail header.
2. richer dependency messaging beyond processing-state guardrail.
## 8. Hidden and System-Managed Fields
| Field | Category | Why Hidden or Protected |
|---|---|---|
| status | System-managed lifecycle | Managed by worker lifecycle transitions |
| retry_count | System-managed operational state | Reflects retry behavior, not direct user input |
| date_created | System-managed | Audit timestamp |
| date_updated | System-managed | Audit timestamp |
## 9. Traceability Anchors
Schema and models:
- docs/schema_v2.md
- src/transcription/db/models.py
Current implementation:
- src/transcription/ui/pages/jobs_page.py
- src/transcription/ui/components/table/jobs.py
- src/transcription/ui/pages/sources_page.py
- src/transcription/services/jobs.py
- src/transcription/services/workflows.py
- src/transcription/services/store.py
- src/transcription/services/transcription.py
Companion UX spec:
- docs/ui/entities/job/user-journey.md
Acceptance checklist:
- docs/ui/entities/job/acceptance-criteria.md
## 10. Acceptance Checklist Summary
- Every Job schema field appears in the field inventory.
- Intended Create behavior matches the companion user journey.
- Current behavior reflects explicit jobs creation plus source review/editing through dedicated Sources routes.
- Provider/model/prompt visibility intent is explicit for create and detail views.
- Gaps between intended and current behavior are explicit.
- Read, Update, and Delete sections distinguish target behavior from current code.
docs/ui/entities/job/user-journey.md
-291
View File
@@ -1,291 +0,0 @@
# Job User Journey
Purpose: Define how a user should interact with the UI to create and manage a Job record, including document linking, source uploads, processing status, and page-level review.
Scope: This document describes intended user interaction for the Job UI. It is the UX contract for the Job entity.
Companion schema mapping: schema-mapping.md
Companion acceptance criteria: acceptance-criteria.md
## 1. Overview
A Job represents one transcription run for a selected Document and one or more uploaded source files.
Managing a Job is run-first:
1. The user opens the Jobs page.
2. The user selects Create job.
3. The user lands on a Job detail/create surface.
4. The user links a Document and uploads one or more source files.
5. The user submits for transcription.
6. The system creates and processes the Job.
7. The user reviews job metadata and follows document-scoped links for Sources and Jobs.
## 2. User Goal
The user wants to:
1. see all jobs in one place
2. create a new transcription run intentionally
3. attach the run to the correct Document
4. upload source file(s) for that run
5. submit and monitor processing state
6. review and revise page-level outputs
## 3. Page Model
### 3.1 Jobs List Page
The Jobs page is the primary UI surface where users manage jobs.
It should support:
1. listing all jobs
2. searching or filtering jobs
3. opening job detail for any row
4. starting Create job
5. clear empty state when no jobs exist
### 3.2 Job Detail/Create Page
The Job detail/create page is used for both creating a new Job and viewing an existing Job.
Create mode should include:
1. document selection
2. source upload controls
3. submit for transcription action
Detail mode should include:
1. job metadata and status
2. document-scoped navigation links for the current Document
3. provider/model/prompt visibility when known
4. no delete action in first release
## 4. Entry Points
Primary entry points:
1. from Jobs page, Create job
2. from Jobs page row selection, open existing Job detail
Current implementation note:
1. current code path uses explicit /jobs/new creation
2. intended UX is explicit Create job from the Jobs page
3. current detail view is link-oriented and routes source review/editing through dedicated Source detail
## 5. Create Job Flow
### 5.1 User Intent
The user wants to start a transcription run by selecting the right Document and providing source files in one guided flow.
### 5.2 Create Entry
1. The user opens the Jobs page
2. The user selects Create job
3. The system opens Job detail/create page in create mode
### 5.3 Create Inputs
| UI Label | Schema Area | Input Type | Required | Notes |
|---|---|---|---|---|
| Document | Job.document_id | Select/search | Yes | Links the run to one Document |
| Source files | Source upload fields | Multi-file upload or folder upload | Yes | User may select one file, many files, or a folder |
| Processing order | Source.page_number assignment rule | System rule | Yes | If multiple files are uploaded, order is alphabetical by original filename |
| Provider | Job.provider | Display or select | No | Visible to user when known; selectable when options are available |
| Model | Job.model | Display or select | No | Visible to user when known; selectable when options are available |
| Prompt | Job.prompt_name | Display or select | No | Visible to user when known; selectable when options are available |
### 5.4 Source Handling Rules
1. Each uploaded file becomes a Source linked to the selected Document
2. Each created Source is linked to the Job through JobSource
3. Multi-file or folder uploads are processed alphabetically by original filename
4. upload_name stores the original filename
5. stored filename uses UUID plus original extension in the form UUID.extension
Suggested helper text:
1. Files are processed alphabetically by original filename. Use leading numbers such as 001, 002, 003 to control page order.
### 5.5 Validation Rules
Create submission must be blocked when:
1. no Document is selected
2. no source file is uploaded
Create submission should provide clear feedback when:
1. uploaded files are invalid or unreadable
2. persistence fails for Job, Source, or JobSource linkage
### 5.6 Submission Behavior
On submit:
1. validate create inputs
2. create Job record linked to selected Document
3. create Source records for uploaded files
4. create JobSource links for each Source in the Job
5. queue processing for transcription
6. route user to Job detail mode
Recommended transactional behavior:
1. intended create writes should succeed or fail together
2. The user should not receive false success when required records fail
### 5.7 Create Success Result
After successful create:
1. job appears in Jobs list
2. job detail shows selected Document and created source set
3. status appears as queued or processing based on execution timing
4. The user can monitor progress and open page-level review
### 5.8 Create Failure Result
If create fails:
1. Show clear error message
2. preserve entered selections where possible
3. keep retry path available
4. do not show false success feedback
## 6. Read Job Journey
### 6.1 User Intent
The user wants to quickly understand what the job is, its current status, and which source pages need review.
### 6.2 Jobs List Expectations
The Jobs list should show, at minimum:
1. job identifier
2. document context
3. current status
4. creation or update timestamp
5. quick action to open detail
Optional first-release columns if available:
1. retry count
2. provider/model summary
### 6.3 Job Detail Expectations
The Job detail should show:
1. job status and summary metadata
2. selected Document context
3. document-scoped and job-scoped navigation links
4. source review entry through job-scoped Sources list
Source detail should show:
1. source metadata and preview
2. original transcription output per source
3. revision editor and latest revised content
### 6.4 Read Empty and Missing States
If no jobs exist:
1. list shows no jobs yet empty state
2. list shows Create job action
If a job id is invalid or missing:
1. Show clear not found state
2. do not crash the page
If a job has no source items due to failure:
1. Show clear warning state
2. keep recovery guidance visible
## 7. Job Status Lifecycle UX
### 7.1 Status Values
The UI should map to model-backed job states:
1. queued
2. processing
3. transcribed
4. completed
5. partial_success
6. failed
### 7.2 In-Progress States
When status is queued or processing:
1. Show active progress state
2. keep detail page refresh-safe
3. indicate that source-level results may still be arriving
### 7.3 Terminal States
When status is completed:
1. Show completion success state
2. direct user to revision workflow
When status is partial_success:
1. Show mixed outcome state
2. identify failed pages
3. guide user to review available successful pages and retry strategy
When status is failed:
1. Show failure state with actionable message
2. keep navigation and retry guidance available
## 8. Update Job Journey
### 8.1 User Intent
The user primarily updates job-related review outcomes by editing revised transcription text per source page.
### 8.2 First-Release Editable Scope
Editable in first release:
1. source-level revised_text through Source detail reached from job-scoped Sources navigation
Read-only in first release:
1. Job.document_id after create
2. job status values managed by processing workflow
3. provider/model/prompt values may be system-managed, but should remain visible in UI when known
### 8.3 Update Save Behavior
On revision save:
1. validate revised text
2. persist revised text for selected source
3. update revised timestamp fields by system policy
4. show success feedback
On save failure:
1. Show clear error feedback
2. preserve entered text where possible
3. Allow retry
## 9. Delete and Retention Policy
### 9.1 User Intent
The user may need to remove invalid or duplicate jobs safely.
### 9.2 First-Release Policy
Delete behavior uses explicit guardrails:
1. deletion is blocked while status is processing
2. blocked delete explains constraints and offers back navigation
3. allowed delete requires explicit confirmation and then returns to Jobs list with success feedback
## 10. Relationship to Other Workflows
Job workflow integrates with:
1. Document workflow for ownership context
2. Source workflow for uploaded page records and ordering
3. Revision workflow for human correction lifecycle
4. Worker processing workflow for queued execution and status transitions
## 11. Relationship to Schema Mapping
The companion schema-mapping document should specify:
1. field visibility per CRUD action
2. current implementation status
3. intended behavior
4. gap-to-target items
## 12. Deferred Items
Deferred to future revisions:
1. manual retry controls from job detail
2. advanced provider/model/prompt policy controls beyond basic create-time visibility
3. advanced bulk actions across multiple jobs
4. live streaming progress updates beyond refresh-based updates
5. job templates or preset configurations
docs/ui/entities/job/~acceptance-criteria.md.saved.bak
-154
View File
@@ -1,154 +0,0 @@
# Job Acceptance Criteria
Purpose: Define implementation-ready acceptance criteria for Job Create, Read, Update, and Delete workflows.
Companion documents:
- docs/ui/entities/job/user-journey.md
- docs/ui/entities/job/schema-mapping.md
## Scope
This checklist covers:
1. Create flow
2. Read flow
3. Update flow
4. Delete flow
This checklist does not cover:
1. provider-specific transcription internals
2. advanced workflow scheduling and queue orchestration controls
3. multi-job bulk operations
## Create Acceptance Criteria
### CR-1 Job creation entry
1. Given the user is on the Jobs page
2. When the user selects Create job
3. Then the user is taken to Job detail/create mode
### CR-2 Required create values
1. document_id must be selected before submit
2. at least one source file must be uploaded before submit
3. each uploaded file creates a Source linked to the selected Document
4. each created Source is linked to the new Job through JobSource
### CR-3 Source ordering behavior
1. Given multi-file or folder upload
2. When source records are created
3. Then page ordering follows alphabetical order of original filenames
4. Then helper text explains how filename conventions control ordering
### CR-4 Provider/model/prompt visibility
1. provider, model, and prompt_name are visible in create flow when known
2. provider, model, and prompt_name are visible in detail flow when known
3. if values are unknown at create time, UI shows clear unknown or pending state without blocking submit
### CR-5 Successful create outcome
1. Given valid inputs
2. When the user submits create
3. Then the Job record is created and linked to selected Document
4. Then source and JobSource records are created for uploads
5. Then job status is queued or processing based on execution timing
6. Then the user is routed to Job detail mode
### CR-6 Create failure outcome
1. Given create validation or persistence failure
2. Then clear error feedback is shown
3. Then no false success feedback is shown
4. Then entered selections are preserved where possible
5. Then retry path remains available
## Read Acceptance Criteria
### RD-1 Jobs list retrieval
1. Given one or more jobs exist
2. When the user opens the Jobs page
3. Then all jobs are listed in a table or equivalent list surface
### RD-2 Jobs list fields
1. Jobs list shows job id
2. Jobs list shows status
3. Jobs list shows created or updated timestamps
4. Jobs list shows retry_count when available
5. Jobs list provides navigation to Job detail for each row
### RD-3 Job detail retrieval
1. Given a valid job id
2. When the user opens Job detail
3. Then job metadata for that record only is shown
4. Then document-scoped navigation links for Sources and Jobs are shown
### RD-4 Detail execution context visibility
1. provider, model, and prompt_name are displayed when known
2. status lifecycle value is visible
3. source-level transcription and revision context is available through Source detail navigation from Job detail
### RD-5 Missing and invalid id states
1. Given an invalid job id format
2. Then UI shows invalid job id state without crashing
3. Given a valid but nonexistent job id
4. Then UI shows job not found state without crashing
## Update Acceptance Criteria
### UP-1 Revision edit entry
1. Given a job detail page
2. When the user opens the page
3. Then navigation links to job-scoped Sources are available
4. Then source rows can open Source detail revision workflow
### UP-2 Revision validation
1. revision save blocks empty trimmed text and shows warning feedback
### UP-3 Successful revision save
1. Source detail save persists revised text and shows success feedback
### UP-4 Revision save failure
1. Source detail save failure shows clear error feedback with retry path
### UP-5 Job lifecycle state update visibility
1. status changes from queued to processing to terminal states are reflected in UI
2. retry_count updates are reflected when retry logic runs
3. users cannot directly edit lifecycle state fields in first release
## Delete Acceptance Criteria
### DL-1 Delete entry and confirmation
1. Given a job detail context
2. When the user opens job delete page
3. Then a permanent-action confirmation is shown for non-processing jobs
### DL-2 Dependency guardrails
1. Delete is blocked while job status is processing
2. Related JobSource links are removed as part of allowed delete flow
### DL-3 Blocked delete behavior
1. When blocked, the UI shows clear processing-state guidance
2. The user is offered navigation back to job or jobs list
### DL-4 Successful delete
1. Given an allowed delete
2. When the user confirms delete
3. Then the job is removed and success feedback is shown
4. Then the user is returned to Jobs list
### DL-5 Delete failure
1. Given backend failure during delete
2. Then clear error feedback is shown
3. Then the user remains in delete context with retry path
## Cross-Criteria Quality Gates
### QG-1 Separation of intent and implementation
1. UX intent remains in user-journey.md
2. Current versus target implementation mapping remains in schema-mapping.md
### QG-2 Traceability
1. Each accepted behavior maps to at least one UI action or service path
2. No acceptance criterion contradicts first-release deferred items
### QG-3 First-release constraints
1. Jobs page remains list-all with explicit Create job action
2. Job create requires Document selection and source upload
3. provider/model/prompt_name are visible to users when known
4. manual retry controls may remain deferred while status visibility is required
docs/ui/entities/person/acceptance-criteria.md
-147
View File
@@ -1,147 +0,0 @@
# Person Acceptance Criteria
Purpose: Define implementation-ready acceptance criteria for Person Create, Read, Update, and Delete workflows.
Companion documents:
- docs/ui/entities/person/user-journey.md
- docs/ui/entities/person/schema-mapping.md
## Scope
This checklist covers:
1. Create flow
2. Read flow
3. Update flow
4. Delete flow
This checklist does not cover:
1. advanced metadata_ editing UX
2. structured-name schema migration implementation
3. bulk merge or dedup workflow design
## Create Acceptance Criteria
### CR-1 Person creation entry
1. Given a Person page
2. When the user selects Create new person
3. Then the user can open a Person create form
### CR-2 Required field validation
1. full_name is required
2. Save is blocked when full_name is empty
3. Inline feedback is shown for required-field errors
### CR-3 Optional field handling
1. Optional fields may be blank without blocking create
2. Date raw and exact fields can coexist
3. Exact date remains canonical when both exact and raw are provided
4. Portrait uploads persist under uploads/portraits/person and store a relative portrait_path
### CR-4 Successful create outcome
1. Given valid input
2. When the user saves
3. Then the Person record is created
4. Then success feedback is shown
5. Then the user is routed to Person detail page
### CR-5 Create failure outcome
1. Given backend failure during create
2. Then clear error feedback is shown
3. Then entered values are retained where possible
4. Then no false success feedback is shown
## Read Acceptance Criteria
### RD-1 Person detail retrieval
1. Given a valid Person id
2. When the user opens the Person detail page
3. Then the system displays Person metadata for that record only
### RD-2 Metadata visibility
1. The page shows full_name and available optional person fields
2. created_at and updated_at are shown as system-managed, read-only values
3. portrait_path is rendered when available, including an image preview when possible
4. relative portrait_path values resolve through /uploads for image rendering
### RD-3 Linked documents section
1. Given no linked DocumentPerson rows
2. Then the page shows a no linked documents yet empty state
3. Given linked documents exist
4. Then the page shows linked document entries
### RD-4 Read failure state
1. Given a nonexistent Person id
2. Then the UI shows a clear not found state without crashing
## Update Acceptance Criteria
### UP-1 Edit entry
1. Given a loaded Person detail page
2. When the user selects Edit person
3. Then editable controls are shown for allowed fields only
### UP-2 Editable fields
1. Editable: full_name, display_name, maiden_name, birth/death fields, places, biography, portrait_path
2. Not editable: id, created_at, updated_at
3. metadata_ remains hidden in first release
### UP-3 Required validation
1. full_name remains required
2. Save is blocked with inline feedback when full_name is empty
### UP-4 Successful save
1. Given valid input
2. When the user saves
3. Then changes persist
4. Then success feedback is shown
5. Then the user remains on Person detail with refreshed values
### UP-5 Save failure
1. Given backend failure during save
2. Then clear error feedback is shown
3. Then the user-entered values remain available for retry where possible
4. Then no false success feedback is shown
## Delete Acceptance Criteria
### DL-1 Delete entry and confirmation
1. Given a Person detail page
2. When the user selects Delete person
3. Then a confirmation dialog appears with permanent-action wording
### DL-2 Relationship guardrails
1. Delete is allowed only when relationship policy allows it
2. If linked DocumentPerson rows must be removed first, delete is blocked
### DL-3 Blocked delete behavior
1. When blocked
2. Then the UI explains why deletion is blocked
3. Then the UI identifies linked-document dependency presence
4. Then the UI provides navigation to cleanup paths
### DL-4 Successful delete
1. Given no blocking dependencies
2. When the user confirms delete
3. Then the Person record is removed
4. Then success feedback is shown
5. Then the user returns to the Person list page
### DL-5 Delete failure
1. Given backend failure during delete
2. Then clear error feedback is shown
3. Then the user remains on Person detail with retry path
## Cross-Criteria Quality Gates
### QG-1 Separation of intent and implementation
1. UX intent remains in user-journey.md
2. Current versus target implementation mapping remains in schema-mapping.md
### QG-2 Traceability
1. Each accepted behavior maps to at least one future UI action or service call path
2. No acceptance criterion contradicts the deferred-item policy
### QG-3 First-release constraints
1. metadata_ remains hidden in first release
2. structured name field split remains deferred
3. recipient and multi-person role management stays in later revisions
docs/ui/entities/person/schema-mapping.md
-265
View File
@@ -1,265 +0,0 @@
# Person Schema-to-UI Mapping
Purpose: Map the Person schema to the UI, while clearly separating intended target behavior from current implementation.
Companion document: user-journey.md
Acceptance criteria: acceptance-criteria.md
## 1. Entity Snapshot
- Table: Person
- Primary key: id (UUID)
- Related entities: DocumentPerson, Document
- Canonical schema references:
- src/transcription/db/models.py
- docs/schema_v2.md
## 2. Mapping Rules
This document uses three lenses:
1. Intended behavior: what the UX should support.
2. Current behavior: what the code supports today.
3. Gap to target: what must change to align implementation with the intended UX.
## 3. Field Inventory
| Field | DB Type | Nullable | Default/Auto Value | Intended UI Treatment | Notes |
|---|---|---|---|---|---|
| id | UUID | No | uuid4() | Hidden, system-managed | Primary key |
| full_name | str | No | None | Shown, editable on create and update | Required canonical name |
| display_name | str | Yes | None | Shown, editable | Optional |
| maiden_name | str | Yes | None | Shown, editable | Optional |
| birth_date | date | Yes | None | Shown, editable | Canonical exact date when present |
| birth_date_raw | str | Yes | None | Shown, editable | Approximate or unknown date text |
| birth_place | str | Yes | None | Shown, editable | Optional |
| death_date | date | Yes | None | Shown, editable | Canonical exact date when present |
| death_date_raw | str | Yes | None | Shown, editable | Approximate or unknown date text |
| death_place | str | Yes | None | Shown, editable | Optional |
| biography | str | Yes | None | Shown, editable | Optional narrative |
| portrait_path | str | Yes | None | Shown, editable | Optional path |
| metadata_ | JSONB/JSON | Yes | None | Hidden in first release | Advanced metadata |
| created_at | datetime | No | datetime.now(UTC) | Hidden or read-only | System-managed |
| updated_at | datetime | No | datetime.now(UTC) | Hidden or read-only | System-managed |
## 4. CREATE Mapping
### 4.1 Intended Create Flow
Entry point: Person page
User action: Create new person
Success destination: new Person detail page
| Field | Intended User Input | Required | Visible | Notes |
|---|---|---|---|---|
| full_name | Text input | Yes | Yes | Canonical identity field |
| display_name | Text input | No | Yes | Optional |
| maiden_name | Text input | No | Yes | Optional |
| birth_date | Date input | No | Yes | Structured exact date |
| birth_date_raw | Text input | No | Yes | Approximate/uncertain date |
| birth_place | Text input | No | Yes | Optional |
| death_date | Date input | No | Yes | Structured exact date |
| death_date_raw | Text input | No | Yes | Approximate/uncertain date |
| death_place | Text input | No | Yes | Optional |
| biography | Text area | No | Yes | Optional |
| portrait_path | Text input | No | Yes | Optional |
| metadata_ | None | No | No | Hidden in first release |
| created_at | None | No | No | System-generated |
| updated_at | None | No | No | Not user-entered |
Related records during intended create:
- No DocumentPerson link is required during Person creation.
- Document linking can be done later from Document or Person workflows.
### 4.2 Current Implementation
Current entry point: dedicated People page and Person create/edit flows
Current user action: open Person create page, fill form fields, optionally upload portrait
Current backend path: People page submit callbacks -> DocumentService.create_person() / update_person()
| Field | Current Value at Create | Source | Visible to User | Evidence |
|---|---|---|---|---|
| id | Generated UUID | System | No | Person model default factory in src/transcription/db/models.py |
| full_name | Form input | User input | Yes | src/transcription/ui/pages/people_page.py |
| display_name | Form input or None | User input | Yes | src/transcription/ui/pages/people_page.py |
| maiden_name | Form input or None | User input | Yes | src/transcription/ui/pages/people_page.py |
| birth_date | Form input or None | User input | Yes | src/transcription/ui/pages/people_page.py |
| birth_date_raw | Form input or None | User input | Yes | src/transcription/ui/pages/people_page.py |
| birth_place | Form input or None | User input | Yes | src/transcription/ui/pages/people_page.py |
| death_date | Form input or None | User input | Yes | src/transcription/ui/pages/people_page.py |
| death_date_raw | Form input or None | User input | Yes | src/transcription/ui/pages/people_page.py |
| death_place | Form input or None | User input | Yes | src/transcription/ui/pages/people_page.py |
| biography | Form input or None | User input | Yes | src/transcription/ui/pages/people_page.py |
| portrait_path | Relative upload path or manual path | Upload helper + user input | Yes | src/transcription/ui/pages/people_page.py, src/transcription/services/store.py |
| metadata_ | Caller-provided or None | Service/API caller | No | Person model in src/transcription/db/models.py |
| created_at | Current UTC timestamp | System | No | Person model default in src/transcription/db/models.py |
| updated_at | Current UTC timestamp | System | No | Person model default in src/transcription/db/models.py |
### 4.3 Gap to Target
To satisfy the intended Create flow, implementation now includes:
1. a Person page and dedicated create form
2. user-entered controls for Person fields
3. create validation and success/failure UX states
4. post-submit routing to a Person detail page
## 5. READ Mapping
### 5.1 Intended Read Behavior
On the Person detail page, the user should be able to see:
1. Person identity and biographical metadata
2. linked Documents (through DocumentPerson)
3. empty-state behavior when no linked documents exist
### 5.2 Current Implementation
Current Person visibility is implemented in dedicated list/detail/edit/delete pages.
| Field | Current Rendering | Visible to User | Notes | Evidence |
|---|---|---|---|---|
| full_name | Rendered in header and summary | Yes | Dedicated Person page exists | `src/transcription/ui/pages/people_page.py` |
| display_name | Rendered | Yes | Visible in detail and list contexts | src/transcription/ui/pages/people_page.py |
| maiden_name | Rendered | Yes | Visible in detail context | src/transcription/ui/pages/people_page.py |
| birth_date | Rendered | Yes | Visible in detail context | src/transcription/ui/pages/people_page.py |
| birth_date_raw | Rendered | Yes | Visible in detail context | src/transcription/ui/pages/people_page.py |
| birth_place | Rendered | Yes | Visible in detail context | src/transcription/ui/pages/people_page.py |
| death_date | Rendered | Yes | Visible in detail context | src/transcription/ui/pages/people_page.py |
| death_date_raw | Rendered | Yes | Visible in detail context | src/transcription/ui/pages/people_page.py |
| death_place | Rendered | Yes | Visible in detail context | src/transcription/ui/pages/people_page.py |
| biography | Rendered | Yes | Visible in detail context | src/transcription/ui/pages/people_page.py |
| portrait_path | Rendered as text and image when available | Yes | Dedicated Person page exists | `src/transcription/ui/pages/people_page.py` |
| metadata_ | Not rendered | No | Hidden advanced field | no current UI field |
| created_at | Rendered read-only | Yes | Visible in detail context | src/transcription/ui/pages/people_page.py |
| updated_at | Rendered read-only | Yes | Visible in detail context | src/transcription/ui/pages/people_page.py |
### 5.3 Gap to Target
To satisfy the intended Read flow, implementation now includes:
1. metadata rendering for Person fields
2. linked Documents section with empty states
3. document-link navigation paths
## 6. UPDATE Mapping
### 6.1 Intended Update Behavior
The user should be able to edit Person metadata from the Person detail page or a dedicated edit flow.
Intended editable fields:
- full_name
- display_name
- maiden_name
- birth_date
- birth_date_raw
- birth_place
- death_date
- death_date_raw
- death_place
- biography
- portrait_path
Intended system-managed fields:
- id
- created_at
- updated_at
Hidden in first release:
- metadata_
### 6.2 Current Implementation
| Field | Updatable via UI | Updatable via Service | Notes |
|---|---|---|---|
| id | No | Practically no | Primary key should be treated as immutable |
| full_name | Yes | Yes | Editable from Person edit page via DocumentService.update_person() |
| display_name | Yes | Yes | Editable from Person edit page via DocumentService.update_person() |
| maiden_name | Yes | Yes | Editable from Person edit page via DocumentService.update_person() |
| birth_date | Yes | Yes | Editable from Person edit page via DocumentService.update_person() |
| birth_date_raw | Yes | Yes | Editable from Person edit page via DocumentService.update_person() |
| birth_place | Yes | Yes | Editable from Person edit page via DocumentService.update_person() |
| death_date | Yes | Yes | Editable from Person edit page via DocumentService.update_person() |
| death_date_raw | Yes | Yes | Editable from Person edit page via DocumentService.update_person() |
| death_place | Yes | Yes | Editable from Person edit page via DocumentService.update_person() |
| biography | Yes | Yes | Editable from Person edit page via DocumentService.update_person() |
| portrait_path | Yes | Yes | Editable manually and via portrait upload helper |
| metadata_ | No | Yes | Technically updatable, hidden in first release |
| created_at | No | Technically yes | Should remain system-managed |
| updated_at | No | Technically yes | Should remain system-managed |
### 6.3 Gap to Target
Implementation now includes:
1. Person edit controls in the UI
2. validation and save behavior for Person metadata
3. a consistent updated_at update policy for Person edits
## 7. DELETE Mapping
### 7.1 Intended Delete Behavior
The UI should provide a delete action for Person with guardrails.
Rules:
1. Deletion can proceed when relationship policy allows no retained document links.
2. If linked DocumentPerson records exist and policy requires cleanup first, deletion is blocked.
3. Delete confirmation must make clear that deletion is permanent.
### 7.2 Current Implementation
| Action | UI Exposed | Backend Capability | Notes |
|---|---|---|---|
| Delete Person | Yes | Yes | Dedicated delete page enforces linked-document guardrails before service delete |
### 7.3 Gap to Target
Implementation includes:
1. a Person delete control in the UI
2. relationship-aware pre-delete checks
3. user-facing blocked-delete messaging
4. confirmation UX for successful delete attempts
## 8. Hidden and System-Managed Fields
| Field | Category | Why Hidden or Protected |
|---|---|---|
| id | System-managed | Internal identifier |
| created_at | System-managed | Audit timestamp |
| updated_at | System-managed | Audit timestamp |
| metadata_ | Hidden in first release | Advanced JSON metadata not needed in initial UI |
## 9. Structured Name Deferred Note
Structured name fields are deferred to a future schema revision.
Current policy:
1. full_name remains canonical and required.
Future revision intent:
1. introduce first_name, middle_name, last_name, and optional suffix fields.
2. maintain compatibility with existing full_name records during migration.
3. define normalization and reconciliation rules when structured and canonical forms differ.
## 10. Traceability Anchors
Schema and models:
- docs/schema_v2.md
- src/transcription/db/models.py
Current implementation:
- src/transcription/services/documents.py
- src/transcription/ui/pages/people_page.py
- src/transcription/services/store.py
Companion UX spec:
- docs/ui/entities/person/user-journey.md
Acceptance checklist:
- docs/ui/entities/person/acceptance-criteria.md
## 11. Acceptance Checklist Summary
- Every Person schema field appears in the field inventory.
- Intended Create behavior matches the companion user journey.
- Current Create behavior reflects dedicated UI form implementation with optional portrait upload handling.
- Gaps between intended and current behavior are explicit.
- Read, Update, and Delete sections distinguish target behavior from current code.
docs/ui/entities/person/user-journey.md
-292
View File
@@ -1,292 +0,0 @@
# Person User Journey
Purpose: Define how a user should interact with the UI to create and manage a Person record, including expected inputs, validation, outcomes, and links to Document relationships.
Scope: This document describes intended user interaction for the Person UI. It is the UX contract for the Person entity.
Companion schema mapping: schema-mapping.md
Companion acceptance criteria: acceptance-criteria.md
## 1. Overview
A Person represents a historical individual who may be associated with one or more Documents.
Managing a Person is a profile-first workflow:
1. The user opens the Person page.
2. The user selects Create new person.
3. The user enters known biographical fields.
4. The system creates the Person record.
5. The user can later associate the Person with one or more Documents through DocumentPerson links.
## 2. User Goal
The user wants to:
1. create and maintain historical person records
2. reuse the same Person across multiple Documents
3. record both precise and approximate date values where certainty is limited
4. link people to documents as author or recipient in future flows
## 3. Page Model
### 3.1 Person Page
The Person page is the general UI surface where users manage people.
It should support:
1. listing or locating existing people
2. starting the Create new person flow
3. navigating into a specific Person after it exists
### 3.2 Person Detail Page
The Person detail page is the page for one specific Person after creation.
It should show:
1. core identity fields
2. biographical metadata
3. portrait image when available
4. related Documents section
5. empty state when no linked documents exist yet
## 4. Entry Point
Entry point: Person page
Primary action: Create new person
Expected UI affordance:
1. a visible action labeled Create new person
2. activation opens a dedicated form view, modal, or detail panel
Preferred first implementation:
1. dedicated Person create page or panel
2. simple labeled form controls
3. text inputs are acceptable for first release
## 5. Create Person Form
### 5.1 Required Fields
| UI Label | Schema Field | Input Type | Required | Notes |
|---|---|---|---|---|
| Full name | full_name | Text input | Yes | Canonical identity field |
### 5.2 Optional Name Fields
| UI Label | Schema Field | Input Type | Required | Notes |
|---|---|---|---|---|
| Display name | display_name | Text input | No | Friendly or abbreviated display |
| Maiden name | maiden_name | Text input | No | Historical alternate surname |
### 5.3 Birth and Death Date Fields
| UI Label | Schema Field | Input Type | Required | Notes |
|---|---|---|---|---|
| Birth date | birth_date | Date input | No | Exact known date |
| Birth date (approximate/raw) | birth_date_raw | Text input | No | Approximate or uncertain value |
| Death date | death_date | Date input | No | Exact known date |
| Death date (approximate/raw) | death_date_raw | Text input | No | Approximate or uncertain value |
Date handling rule:
1. exact and raw values may both be entered
2. exact date is canonical when present
3. raw date is retained as historical context
### 5.4 Optional Biographical Fields
| UI Label | Schema Field | Input Type | Required | Notes |
|---|---|---|---|---|
| Birth place | birth_place | Text input | No | Free text |
| Death place | death_place | Text input | No | Free text |
| Biography | biography | Text area | No | Narrative context |
| Portrait path | portrait_path | Text input | No | File or resource path |
| Metadata | metadata_ | Hidden or advanced JSON editor | No | Prefer hidden in first release |
### 5.5 System Fields
| Schema Field | User Editable | Notes |
|---|---|---|
| id | No | System-generated |
| created_at | No | System-generated |
| updated_at | No | System-managed |
## 6. Validation Rules
### 6.1 Required Validation
1. full_name is required
2. save is blocked when full_name is empty
### 6.2 Date Validation
1. birth_date and birth_date_raw may coexist
2. death_date and death_date_raw may coexist
3. exact date fields are canonical when present
4. raw fields remain descriptive context
### 6.3 Integrity Validation
1. form accepts unknown values for optional fields
2. missing birth or death data does not block creation
## 7. Submission Behavior
On submit:
1. The system validates required fields
2. The system creates the Person record
3. The system returns the user to the Person detail page
4. The system shows a success message
5. If portrait upload is used, the file is stored under uploads/portraits/person and portrait_path is set to that relative file path
Recommended transactional behavior:
1. Person writes are atomic
2. no partial save state should be persisted
## 8. Expected Result After Success
After successful creation:
1. The user sees the Person detail page for the new record
2. full_name is visible in the header or summary
3. empty Related Documents section is shown if no links exist
4. The user can proceed to link this person from Document workflows
## 9. Expected Result After Failure
If creation fails:
1. Show a clear error message
2. Show field-level feedback for validation failures
3. preserve entered data where possible
4. do not show false success messaging
## 10. Read Person Journey
### 10.1 User Intent
The user wants to open a Person and quickly understand:
1. identity and key biography fields
2. whether this person is linked to any documents
3. what next action to take
### 10.2 Read Surfaces
The Person detail page should show:
1. full_name and display fields
2. birth and death fields
3. biography summary
4. related documents list or empty state
5. portrait preview resolved from /uploads when portrait_path is a relative path
### 10.3 Read Empty State
If no linked documents exist:
1. Show No linked documents yet
2. provide guidance to link from Document workflow
## 11. Update Person Journey
### 11.1 User Intent
The user wants to correct or enrich person metadata over time.
### 11.2 Editable Fields
Editable:
1. full_name
2. display_name
3. maiden_name
4. birth_date
5. birth_date_raw
6. birth_place
7. death_date
8. death_date_raw
9. death_place
10. biography
11. portrait_path
System-managed:
1. id
2. created_at
3. updated_at
4. metadata_ can remain hidden in first release
### 11.3 Update Save Behavior
On save:
1. validate required fields
2. persist updates
3. refresh updated_at by system policy
4. show confirmation
5. keep user on Person detail page
### 11.4 Update Failure Behavior
1. Show clear error feedback
2. preserve form state where possible
3. Allow retry
## 12. Delete Person Journey
### 12.1 User Intent
The user wants to remove incorrect or duplicate person records safely.
### 12.2 Delete Guardrails
Delete is allowed when:
1. Person has no required retained relationships
Delete is blocked when:
1. Person is linked to one or more Documents via DocumentPerson and unlink policy requires cleanup first
### 12.3 Blocked Delete UX
1. explain that linked Document relationships exist
2. Show link count or list
3. provide cleanup path
### 12.4 Allowed Delete UX
1. Show a confirmation dialog
2. confirm permanent action
3. delete Person
4. return to Person list with success message
## 13. Relationship to Other Workflows
This Person workflow integrates with:
1. Document create and update workflows through person lookup and linking
2. DocumentPerson mapping for role assignments
3. future recipient and multi-person enhancements
## 14. Relationship to Schema Mapping
The companion schema-mapping document should specify:
1. field visibility per CRUD action
2. current implementation status
3. intended behavior
4. gap-to-target items
## 15. Deferred Items
Deferred to future revisions:
1. advanced metadata_ editing UI
2. multi-person role editing in the Person UI itself
3. richer relationship timeline views
4. bulk merge or dedup workflows
5. structured name fields migration (first_name, middle_name, last_name, optional suffix)
### 15.1 Structured Name Fields Migration Note
For now, `full_name` remains the canonical required name field.
Future revision intent:
1. introduce structured fields such as first_name, middle_name, last_name, and optional suffix
2. keep full_name during transition for backward compatibility and historical formatting
3. define normalization and formatting rules for display and sorting
4. update search and dedup workflows to use both structured and canonical forms during migration
Migration considerations:
1. schema migration and backfill strategy for existing Person records
2. validation updates for create and update forms
3. compatibility for existing APIs and UI components that currently rely on full_name
4. clear precedence and reconciliation rules when structured fields and full_name differ
docs/ui/entities/source/acceptance-criteria.md
-141
View File
@@ -1,141 +0,0 @@
# Source Acceptance Criteria
Purpose: Define implementation-ready acceptance criteria for Source Create, Read, Update, and Delete workflows.
Companion documents:
- docs/ui/entities/source/user-journey.md
- docs/ui/entities/source/schema-mapping.md
## Scope
This checklist covers:
1. Create flow
2. Read flow
3. Update flow
4. Delete flow
This checklist does not cover:
1. advanced multi-version revision history design
2. job orchestration state-machine behavior
3. provider-level transcription internals
## Create Acceptance Criteria
### CR-1 Source creation entry
1. Given the user is in job creation or job configuration flow
2. When the user selects Add sources
3. Then the user can upload one or more source files or a folder
4. Then source creation is not offered as a standalone first-release document-only flow
### CR-2 Required create values
1. document_id is derived from selected Document context
2. JobSource.job_id is derived from the active Job context
3. Each created Source is linked to the active Job through JobSource at create time
4. page_number is assigned to preserve ordering
5. upload_name, filename, and file_path are persisted for each created source
### CR-3 Ordering and filename strategy
1. Given a multi-file or folder upload
2. When source records are created
3. Then page ordering follows alphabetical order of original filenames
4. Then upload_name stores the original filename
5. Then filename is stored using UUID plus original extension in the form UUID.extension
### CR-4 Successful create outcome
1. Given valid uploads
2. When source creation completes
3. Then Source records are created and linked to the Document
4. Then Source records are linked to the active Job through JobSource
5. Then source list reflects new pages in sequence
6. Then the user can open preview or revision workflow
### CR-5 Create failure outcome
1. Given upload or persistence failure
2. Then clear error feedback is shown
3. Then no false success feedback is shown
4. Then retry path remains available
5. Then creation fails when required Document or Job linkage cannot be established
## Read Acceptance Criteria
### RD-1 Source detail retrieval
1. Given a valid Source id in source context
2. When the user opens source detail
3. Then source metadata and preview are displayed for that source only
### RD-2 Transcription and revision visibility
1. Original transcription context is visible read-only in Source detail
2. Revision state is visible in Source detail
3. If revised_text is absent, revision input opens as empty and can be edited
### RD-3 Missing source state
1. Given a missing source
2. Then UI shows clear no source available or not found messaging without crashing
## Update Acceptance Criteria
### UP-1 Revision editing entry
1. Given a source context
2. When the user enters revision edit flow
3. Then revised_text input is available in Source detail
### UP-2 Revision validation
1. revised_text cannot be saved as empty after trimming
2. Warning feedback is shown for invalid empty input
### UP-3 Successful revision save
1. Given valid revision text
2. When the user saves
3. Then revised_text persists
4. Then date_revised is updated
5. Then success feedback is shown
6. Then refreshed revision content is visible
### UP-4 Revision save failure
1. Given backend failure during save
2. Then clear error feedback is shown
3. Then the user-entered text remains available for retry where possible
## Delete Acceptance Criteria
### DL-1 Delete entry and confirmation
1. Given a source in source context
2. When the user selects delete source
3. Then a permanent-action confirmation dialog appears
### DL-2 Dependency guardrails
1. If policy requires cleanup of related JobSource records first, delete is blocked
2. If policy allows dependent cleanup path, delete can proceed
### DL-3 Blocked delete behavior
1. When blocked
2. Then UI explains dependency constraints
3. Then UI provides guidance for dependency cleanup
### DL-4 Successful delete
1. Given no blocking dependencies
2. When the user confirms deletion
3. Then source is removed
4. Then success feedback is shown
5. Then the user returns to source list context
### DL-5 Delete failure
1. Given backend failure during delete
2. Then clear error feedback is shown
3. Then the user remains in source context with retry path
## Cross-Criteria Quality Gates
### QG-1 Separation of intent and implementation
1. UX intent remains in user-journey.md
2. Current versus target implementation mapping remains in schema-mapping.md
### QG-2 Traceability
1. Each accepted behavior maps to at least one future UI action or service path
2. No acceptance criterion contradicts first-release deferred items
### QG-3 First-release constraints
1. Source creation remains job-create-centric
2. revised_text is the primary editable source field in first release
3. source creation requires both Document linkage and Job linkage at create time
4. source delete management surfaces are phased in later
docs/ui/entities/source/schema-mapping.md
-212
View File
@@ -1,212 +0,0 @@
# Source Schema-to-UI Mapping
Purpose: Map the Source schema to the UI, while clearly separating intended target behavior from current implementation.
Companion document: user-journey.md
Acceptance criteria: acceptance-criteria.md
## 1. Entity Snapshot
- Table: Source
- Primary key: id (UUID)
- Related entities: Document, JobSource, Job
- Canonical schema references:
- src/transcription/db/models.py
- docs/schema_v2.md
## 2. Mapping Rules
This document uses three lenses:
1. Intended behavior: what the UX should support.
2. Current behavior: what the code supports today.
3. Gap to target: what must change to align implementation with the intended UX.
## 3. Field Inventory
| Field | DB Type | Nullable | Default/Auto Value | Intended UI Treatment | Notes |
|---|---|---|---|---|---|
| id | UUID | No | uuid4() | Hidden, system-managed | Primary key |
| document_id | UUID FK | No | None | Hidden/context-managed | Selected Document context |
| page_number | int | No | 1 | Shown read-only or ordered list | Sequential ordering |
| upload_name | str | No | None | Shown read-only after upload | Original user-provided name |
| filename | str | No | None | Shown read-only | Stored filename |
| file_path | str | No | None | Usually hidden; preview uses path internally | Filesystem path |
| raw_transcription | str | Yes | None | Shown indirectly or hidden | Immutable machine output context |
| revised_text | str | Yes | None | Editable in Source detail | Human-authored correction |
| date_uploaded | datetime | No | datetime.now(UTC) | Shown read-only | System-managed timestamp |
| date_revised | datetime | Yes | None | Shown read-only | Set when revision is saved |
## 4. CREATE Mapping
### 4.1 Intended Create Flow
Entry point: Job creation or job configuration Add sources action
User action: upload one or more source files, or a whole folder
Success destination: source preview or revision flow in job detail context
| Field | Intended User Input | Required | Visible | Notes |
|---|---|---|---|---|
| document_id | Hidden/context | Yes | No | Comes from selected Document |
| JobSource.job_id | Hidden/context | Yes | No | Comes from active Job; required for first release |
| page_number | Auto or user-assisted ordering | Yes | Indirectly | Should preserve sequence |
| upload_name | File picker name | Yes | Yes | Original display name |
| filename | None | Yes | No or read-only | System-stored as UUID.extension |
| file_path | None | Yes | No | Storage path |
| raw_transcription | None | No | No | Filled by processing |
| revised_text | None | No | No | Initially empty |
| date_uploaded | None | No | No | System-generated |
| date_revised | None | No | No | Null until revision |
### 4.2 Current Implementation
Current entry point: Jobs page create flow
Current user action: upload one or more files or a folder through a single upload widget
Current backend path: job create submit -> create_job_for_document()
| Field | Current Value at Create | Source | Visible to User | Evidence |
|---|---|---|---|---|
| id | Generated UUID | System | No | Source model default in src/transcription/db/models.py |
| document_id | Selected existing Document id | Job create selection + service write | Indirectly | src/transcription/ui/pages/jobs_page.py, src/transcription/services/store.py |
| page_number | Sequential assignment based on existing max and alphabetical upload order | Service | No | src/transcription/services/store.py |
| upload_name | original filename basename | User file name transformed by service | Indirectly | src/transcription/services/store.py |
| filename | stored generated filename | Service | Indirectly | src/transcription/services/store.py |
| file_path | stored path | Service | Indirectly | src/transcription/services/store.py |
| raw_transcription | None initially | System | No at create | Source model defaults |
| revised_text | None initially | System | No at create | Source model defaults |
| date_uploaded | current UTC timestamp | System | No | Source model default |
| date_revised | None | System | No | Source model default |
### 4.3 Gap to Target
To satisfy intended Create flow, implementation now includes:
1. multi-source and folder upload support in job create/configure flows
2. deterministic page_number assignment from alphabetical original filename ordering
3. enforced create-time Source-to-Document and Source-to-Job linkage invariants
4. filename storage policy using UUID.extension
## 5. READ Mapping
### 5.1 Intended Read Behavior
On Source detail/list surfaces, users should be able to see:
1. source page preview
2. source metadata and ordering
3. revision state
4. original transcription context
### 5.2 Current Implementation
Current Source reading is centered on dedicated Sources list/detail routes with optional document/job filtering.
| Field | Current Rendering | Visible to User | Notes | Evidence |
|---|---|---|---|---|
| upload_name | Shown in Sources list and Source detail | Yes | Displayed in source context | src/transcription/ui/pages/sources_page.py |
| filename | Shown in Sources list and Source detail | Yes | Source metadata shown in list/detail | src/transcription/ui/pages/sources_page.py |
| file_path | Hidden from direct text rendering | No | Used internally for preview rendering | src/transcription/ui/components/document_panzoom.py |
| page_number | Shown in Sources list and Source detail | Yes | Ordering visible in filtered/global list | src/transcription/ui/pages/sources_page.py |
| raw_transcription | Shown read-only in Source detail | Yes | Read from latest linked JobSource context | src/transcription/ui/pages/sources_page.py |
| revised_text | Shown and editable in Source detail | Yes | Saved through revision action | src/transcription/ui/pages/sources_page.py |
| date_uploaded | Shown in Source detail | Yes | Read-only metadata | src/transcription/ui/pages/sources_page.py |
| date_revised | Shown in Source detail | Yes | Read-only metadata after revision save | src/transcription/ui/pages/sources_page.py |
### 5.3 Gap to Target
To satisfy intended Read flow, implementation must add:
1. optional list filtering controls in-page (current filtering is URL/context based)
2. optional page-specific navigation enhancements beyond current list/detail pattern
## 6. UPDATE Mapping
### 6.1 Intended Update Behavior
Primary user update for Source is revised_text maintenance in Source detail.
Intended editable fields (first release):
- revised_text
Intended read-only fields (first release):
- document_id
- page_number
- upload_name
- filename
- file_path
- raw_transcription
- date_uploaded
- date_revised
### 6.2 Current Implementation
| Field | Updatable via UI | Updatable via Service | Notes |
|---|---|---|---|
| revised_text | Yes | Yes | Saved via TranscriptionService.upsert_revision_for_source() from Source detail |
| date_revised | No | Yes | Set automatically on revision save |
| other fields | No | Technically yes in service layer | No first-class UI editing flow |
### 6.3 Gap to Target
Implementation should add in a later revision:
1. optional future controls for page ordering and metadata corrections
2. revision history and conflict-resolution UX beyond single revised_text updates
## 7. DELETE Mapping
### 7.1 Intended Delete Behavior
Source deletion is deferred in the current UI.
Rules:
1. Deletion can proceed when policy allows cleanup of related JobSource records.
2. If related execution history must be preserved first, deletion is blocked with guidance.
### 7.2 Current Implementation
| Action | UI Exposed | Backend Capability | Notes |
|---|---|---|---|
| Delete Source | No | Yes | TranscriptionService.delete_source() exists, no dedicated UI delete flow |
### 7.3 Gap to Target
Implementation should add in a future revision:
1. source delete controls in source/document context UI
2. dependency checks for JobSource links
3. blocked-delete messaging and cleanup path guidance
4. confirmation UX for successful delete attempts
## 8. Hidden and System-Managed Fields
| Field | Category | Why Hidden or Protected |
|---|---|---|
| id | System-managed | Internal identifier |
| document_id | Context-managed | Derived from selected document context |
| file_path | Operational/internal | Used for file storage and preview plumbing |
| date_uploaded | System-managed | Audit timestamp |
| date_revised | System-managed | Revision timestamp set by system |
## 9. Traceability Anchors
Schema and models:
- docs/schema_v2.md
- src/transcription/db/models.py
Current implementation:
- src/transcription/services/store.py
- src/transcription/services/transcription.py
- src/transcription/ui/pages/sources_page.py
- src/transcription/ui/pages/jobs_page.py
- src/transcription/ui/pages/documents_page.py
- src/transcription/ui/components/document_panzoom.py
Companion UX spec:
- docs/ui/entities/source/user-journey.md
Acceptance checklist:
- docs/ui/entities/source/acceptance-criteria.md
## 10. Acceptance Checklist Summary
- Every Source schema field appears in the field inventory.
- Intended Create behavior matches the companion user journey.
- Source create invariant requires both Document linkage and Job linkage at create time.
- Current behavior reflects upload-centric create flow and dedicated Sources list/detail review flow.
- Gaps between intended and current behavior are explicit.
- Read, Update, and Delete sections distinguish target behavior from current code.
docs/ui/entities/source/user-journey.md
-229
View File
@@ -1,229 +0,0 @@
# Source User Journey
Purpose: Define how a user should interact with the UI to create and manage Source records, including page-level transcription context and revision behavior.
Scope: This document describes intended user interaction for the Source UI. It is the UX contract for the Source entity.
Companion schema mapping: schema-mapping.md
Companion acceptance criteria: acceptance-criteria.md
## 1. Overview
A Source represents one page or file unit associated with a Document.
Managing Source records is page-first:
1. The user starts from a transcription job flow.
2. The user adds one or more source files.
3. The system creates Source records linked to the Document and linked to the Job through JobSource.
4. The user reviews source lists from a dedicated Sources page.
5. The user opens Source detail to review preview, metadata, transcription text, and revision text.
## 2. User Goal
The user wants to:
1. add page files to a Document
2. ensure every source is attached to the transcription job context
3. keep page order reliable
4. review original machine output
5. save human revisions per page
6. navigate source pages efficiently
## 3. Page Model
### 3.1 Source List Surface
A Source list surface should support:
1. listing source pages globally or filtered by selected Document or Job
2. sorting by page_number
3. opening the owning Document or Job context
4. opening Source detail for a selected source
### 3.2 Source Detail Surface
Source detail supports:
1. pan/zoom image or PDF preview
2. read-only source metadata (page number, names, timestamps)
3. read-only original transcription text
4. editable revision text with save action
## 4. Entry Points
Primary entry points:
1. from Job workflow, Add sources while creating or configuring a job
2. from Job detail, open filtered Sources for the current Job
3. from Document detail, open filtered Sources for the current Document
4. from global navigation, open all Sources
Current implementation note:
1. source interaction occurs in job-create flow and dedicated Sources list/detail flows
## 5. Create Source Flow
### 5.1 User Intent
The user wants to attach one or more files to a Document so each page can be processed and reviewed.
### 5.2 Create from Job Context
1. The user starts from a job-creation or job-configuration flow
2. The user can upload one or more files, or upload a whole folder
3. The system creates Source rows linked to the selected Document
4. The system creates JobSource links for the active Job as part of this flow
5. Source creation fails if required Document or Job linkage cannot be established
### 5.3 Source Create Inputs
| UI Label | Schema Field | Input Type | Required | Notes |
|---|---|---|---|---|
| Source files | upload_name/filename/file_path | Multi-file upload or folder upload | Yes | User may select one file, many files, or a folder |
| Processing order | page_number assignment rule | System rule | Yes | If multiple files are uploaded, processing order is alphabetical by original filename |
| Document reference | document_id | Hidden/context | Yes | Comes from selected Document |
| Job reference | JobSource.job_id | Hidden/context | Yes | Required for first-release source creation |
### 5.4 Filename Strategy
1. store original user filename in upload_name
2. store persisted filename using UUID plus original extension only, in the form UUID.extension
3. this replaces the previous UUID-upload_name.extension pattern
### 5.5 Ordering Guidance
1. multi-file or folder uploads are processed alphabetically by original filename
2. UI should show a warning or helper note so users understand that filename conventions control order
Suggested helper text:
1. Files are processed alphabetically by original filename. Use leading numbers such as 001, 002, 003 to control page order.
### 5.6 System-Managed Values at Create
| Schema Field | User Editable | Notes |
|---|---|---|
| id | No | System-generated |
| date_uploaded | No | System-generated |
| raw_transcription | No | Filled later by processing |
| revised_text | No | Initially empty |
| date_revised | No | Initially null |
### 5.7 Expected Create Result
After successful source create:
1. Source is linked to the Document
2. Source appears in page order derived from alphabetical upload filename ordering
3. Source is linked to the Job through JobSource at create time
4. The user can open the owning Document or Job context
### 5.8 Source Creation Invariant
For first release:
1. every new Source must have a Document link (Source.document_id)
2. every new Source must have a Job link through JobSource (JobSource.job_id -> JobSource.source_id)
3. source creation is treated as part of transcription workflow, not a standalone document-only upload path
## 6. Read Source Journey
### 6.1 User Intent
The user wants to view each page file and understand file identity and processing context.
### 6.2 Read Surface Expectations
The UI should show:
1. source lists for current context (all, document-filtered, or job-filtered)
2. upload_name as the original user-provided filename
3. filename as the stored system filename
4. page_number and ordering context
5. the owning Document and Job navigation context
6. direct action to open Source detail
### 6.3 Read Empty and Missing States
If source is missing:
1. Show clear not found or no source available messaging
If source metadata is partially unavailable:
1. Show fallback labels and keep navigation available where possible
## 7. Update Source Journey
### 7.1 User Intent
The user primarily tracks page-level source records while preserving raw machine output in the service layer.
### 7.2 Intended Editable Fields
Editable in first release:
1. revised_text in Source detail
Read-only in first release:
1. upload_name
2. filename
3. file_path
4. raw_transcription
5. page_number
6. date_uploaded
7. date_revised set by system on revision save
### 7.3 Revision Save Behavior
On save:
1. validate revision text is non-empty after trimming
2. persist revised_text
3. set date_revised
4. show success feedback
5. keep user in current source context
### 7.4 Revision Failure Behavior
If save fails:
1. Show clear error feedback
2. keep user input where possible
3. Allow retry
## 8. Delete Source Journey
### 8.1 User Intent
The user may need to remove incorrect or duplicate source files from a Document.
### 8.2 Guardrails
Delete is allowed when:
1. policy allows removal of related processing history
Delete is blocked when:
1. policy requires preserving dependent job-source execution records until explicit cleanup
### 8.3 Delete UX
When blocked:
1. explain dependency constraints in a future delete flow
2. show cleanup guidance in a future delete flow
When allowed:
1. confirm permanent removal in a future delete flow
2. remove source in a future delete flow
3. return to source list with success state in a future delete flow
## 9. Relationship to Other Workflows
Source workflow integrates with:
1. Document workflow for ownership and page organization
2. Job workflow for processing status and outputs
3. revision workflow for human correction lifecycle
## 10. Relationship to Schema Mapping
The companion schema-mapping document should specify:
1. field visibility per CRUD action
2. current implementation status
3. intended behavior
4. gap-to-target items
## 11. Deferred Items
Deferred to future revisions:
1. bulk page reordering UX
2. multi-file upload progress and resumable upload UX
3. revision history versions beyond a single revised_text field
4. richer per-page status dashboards
5. source delete UI with dependency-aware confirmation
docs/ui/entities/traceability-matrix.md
-78
View File
@@ -1,78 +0,0 @@
# UI Entity Traceability Matrix
Purpose: Map acceptance criteria to concrete implementation anchors and current delivery status.
Updated: 2026-08-02
Status legend:
- Implemented: behavior exists in current UI and service flow
- Partial: parts exist, but user-facing behavior or guardrails are incomplete
- Planned: documented intent with no dedicated UI implementation yet
## Document
| Criteria Group | Acceptance IDs | Status | Primary Implementation Anchors | Notes |
|---|---|---|---|---|
| Read detail and metadata | RD-1, RD-2, RD-7 | Implemented | src/transcription/ui/pages/documents_page.py; src/transcription/services/documents.py; tests/ui/test_documents_page.py | Dedicated Document detail route renders metadata, read-only system timestamps, and invalid/missing-id states. |
| Related sections and navigation | RD-3, RD-4, RD-5, RD-6 | Implemented | src/transcription/ui/pages/documents_page.py; src/transcription/services/documents.py; tests/ui/test_documents_page.py | Document detail now shows linked people plus document-scoped Sources and Jobs navigation for the current document. |
| Update entry, validation, and author linkage | UP-1, UP-2, UP-3, UP-4, UP-5, UP-6 | Implemented | src/transcription/ui/pages/documents_page.py; src/transcription/services/documents.py; tests/ui/test_documents_page.py; tests/services/test_document_service.py | Dedicated edit page includes required-field validation messaging, date parsing rules, and author relationship selection with save path routed back to document detail. |
| Delete controls and guardrails | DL-1, DL-2, DL-3, DL-4, DL-5 | Implemented | src/transcription/ui/pages/documents_page.py; src/transcription/services/documents.py; tests/ui/test_documents_page.py; tests/services/test_document_service.py | Dedicated delete page provides permanent-action confirmation, dependency-category blocking, and guarded backend delete behavior. |
## Person
| Criteria Group | Acceptance IDs | Status | Primary Implementation Anchors | Notes |
|---|---|---|---|---|
| Create flow and validation | CR-1, CR-2, CR-3, CR-4, CR-5 | Implemented | src/transcription/ui/pages/people_page.py; src/transcription/services/documents.py; tests/ui/test_people_page.py | Dedicated Person create page with required full_name validation, optional field handling, and success routing to detail. |
| Read detail and linked documents | RD-1, RD-2, RD-3, RD-4 | Implemented | src/transcription/ui/pages/people_page.py; src/transcription/services/documents.py; tests/ui/test_people_page.py; tests/services/test_document_service.py | Person detail route renders metadata, full-name summary, portrait preview when available, linked-document section, and invalid/missing-id states. |
| Update behavior | UP-1, UP-2, UP-3, UP-4, UP-5 | Implemented | src/transcription/ui/pages/people_page.py; src/transcription/services/documents.py; tests/ui/test_people_page.py; tests/services/test_document_service.py | Dedicated Person edit page supports allowed fields, required full_name validation, and save path back to detail. |
| Delete behavior and guardrails | DL-1, DL-2, DL-3, DL-4, DL-5 | Implemented | src/transcription/ui/pages/people_page.py; src/transcription/services/documents.py; tests/ui/test_people_page.py; tests/services/test_document_service.py | Dedicated delete page provides permanent-action confirmation, linked-document blocking message, and guarded backend delete behavior. |
## Source
| Criteria Group | Acceptance IDs | Status | Primary Implementation Anchors | Notes |
|---|---|---|---|---|
| Create entry and required links | CR-1, CR-2, CR-4, CR-5 | Implemented | src/transcription/services/store.py; src/transcription/ui/pages/jobs_page.py; src/transcription/ui/pages/upload_page.py; tests/services/test_store.py; tests/ui/test_jobs_page.py | Source upload/create is job-create-context only (legacy upload route redirects), with required Document and JobSource linkage enforced. |
| Ordering and filename policy | CR-3 | Implemented | src/transcription/services/store.py; tests/services/test_store.py; tests/ui/test_jobs_page.py | Multi-file/folder uploads are ordered alphabetically by original filename, helper text is visible, and stored filenames use generated unique-id plus extension. |
| Read and navigation visibility | RD-1, RD-2, RD-3 | Implemented | src/transcription/ui/pages/sources_page.py; src/transcription/ui/pages/documents_page.py; src/transcription/ui/pages/jobs_page.py; tests/ui/test_sources_page.py; tests/ui/test_documents_page.py; tests/ui/test_jobs_page.py | Dedicated Sources list/detail routes support global, document-filtered, and job-filtered navigation plus source metadata and preview rendering. |
| Revision update behavior | UP-1, UP-2, UP-3, UP-4 | Implemented | src/transcription/ui/pages/sources_page.py; src/transcription/services/transcription.py; tests/ui/test_sources_page.py; tests/services/test_transcription_service.py | Source detail exposes revision edit/save UX with non-empty validation, success feedback, and refreshed state after save. |
| Delete and dependency guardrails | DL-1, DL-2, DL-3, DL-4, DL-5 | Planned | src/transcription/services/transcription.py; tests/services/test_transcription_service.py | Job-detail source delete UI was removed from the current simplified flow; backend guardrails remain for future reinstatement. |
## Job
| Criteria Group | Acceptance IDs | Status | Primary Implementation Anchors | Notes |
|---|---|---|---|---|
| Create entry and required links | CR-1, CR-2, CR-5, CR-6 | Implemented | src/transcription/services/store.py; src/transcription/ui/pages/jobs_page.py; tests/ui/test_jobs_page.py; tests/services/test_store.py | Jobs list now has explicit Create entry and `/jobs/new` create flow with Document selection, combined file/folder upload widget, and submit routing to job detail. |
| Source ordering and upload behavior | CR-3 | Implemented | src/transcription/services/store.py; src/transcription/ui/pages/jobs_page.py; tests/services/test_store.py; tests/ui/test_jobs_page.py | Multi-file and folder upload are supported through one widget, uploads are sorted alphabetically by original filename, and helper guidance is shown in create UI. |
| Provider/model/prompt visibility | CR-4, RD-4 | Implemented | src/transcription/services/workflows.py; src/transcription/ui/pages/jobs_page.py; tests/ui/test_jobs_page.py | Provider/model/prompt fields are visible in create and detail flows when known (with pending fallback labels). |
| Jobs list and detail read states | RD-1, RD-2, RD-3, RD-5 | Implemented | src/transcription/ui/pages/jobs_page.py; src/transcription/ui/components/table/jobs.py; tests/ui/test_jobs_page.py | Jobs list, detail route, document-scoped navigation, and invalid/missing id states are present. |
| Revision update behavior | UP-1, UP-2, UP-3, UP-4 | Implemented | src/transcription/ui/pages/jobs_page.py; src/transcription/ui/pages/sources_page.py; src/transcription/services/transcription.py; tests/ui/test_jobs_page.py; tests/ui/test_sources_page.py; tests/services/test_transcription_service.py | Job detail routes users to job-scoped Sources where Source detail provides revision edit/save workflow. |
| Lifecycle visibility and retry indicators | UP-5 | Implemented | src/transcription/services/jobs.py; src/transcription/services/workflows.py; src/transcription/ui/pages/jobs_page.py; tests/ui/test_jobs_page.py | Job detail now surfaces lifecycle status plus retry/update metadata while lifecycle fields remain system-managed (no direct user edit controls). |
| Delete and dependency guardrails | DL-1, DL-2, DL-3, DL-4, DL-5 | Implemented | src/transcription/ui/pages/jobs_page.py; src/transcription/services/jobs.py; tests/ui/test_jobs_page.py; tests/services/test_job_service.py | Job delete page enforces processing-state block, confirms allowed deletes, and routes back to jobs list on success. |
## Quality Gate Coverage
| Quality Gate | Acceptance IDs | Status | Notes |
|---|---|---|---|
| Separation of intent vs implementation | QG-1 across entities | Implemented | user-journey.md, schema-mapping.md, and acceptance-criteria.md are maintained per entity. |
| Traceability from criteria to implementation | QG-2 across entities | Implemented | This matrix provides criterion-to-code anchors and current status tags. |
| First-release constraints | QG-3 across entities | Implemented | Constraints are documented and aligned with current flows: jobs-first source upload, visible provider/model/prompt context, and system-managed lifecycle fields. |
## Supporting Entity Coverage
| Supporting Entity | Documentation | Status | Notes |
|---|---|---|---|
| document-person | docs/ui/entities/document-person/schema-mapping.md | Completed | Supporting-entity schema mapping created; no standalone UI contract file by design. |
| job-source | docs/ui/entities/job-source/schema-mapping.md | Completed | Supporting-entity schema mapping created; no standalone UI contract file by design. |
## Suggested Implementation Order
1. Aggregate final acceptance review across Document, Person, Source, and Job criteria.
## Aggregate Final Review Snapshot (2026-08-02)
| Entity | Acceptance IDs still not fully met | Evidence | Notes |
|---|---|---|---|
| Document | None | src/transcription/ui/pages/documents_page.py; tests/ui/test_documents_page.py | Document criteria are covered by dedicated detail/edit/delete pages and document-scoped related views. |
| Person | None | src/transcription/ui/pages/people_page.py; tests/ui/test_people_page.py | Person criteria are covered by dedicated create/detail/edit/delete pages with relationship-aware delete guardrails. |
| Source | None | src/transcription/services/store.py; src/transcription/ui/pages/sources_page.py; src/transcription/ui/pages/jobs_page.py; tests/services/test_store.py; tests/ui/test_sources_page.py; tests/services/test_transcription_service.py | Source criteria are covered by job-context create behavior, ordering/filename policy, dedicated list/detail read flow, revision flow, and delete guardrails. |
| Job | None | src/transcription/ui/pages/jobs_page.py; src/transcription/services/jobs.py; tests/ui/test_jobs_page.py; tests/services/test_job_service.py | Job criteria are covered by create/read/revision/lifecycle visibility and delete guardrails in dedicated routes. |
docs/ui/pages/documents.md
+105
View File
@@ -0,0 +1,105 @@
# Documents Page Contract
## Purpose
Documents manages the archival record for each historical artifact independently of its source files and transcription jobs. A Document can be created first, linked to people in one or more roles, and used later as the parent for Sources and Jobs.
## Routes
| Route | Purpose |
| --- | --- |
| `/documents` | Searchable archival Document list. |
| `/documents/new` | Create a Document. |
| `/documents/{document_id}` | View one Document and its related records. |
| `/documents/{document_id}/edit` | Edit metadata and people-by-role links. |
| `/documents/{document_id}/delete` | Confirm or block deletion. |
| `/documents/{document_id}/jobs` | Show Jobs belonging to the Document. |
| `/documents/{document_id}/sources` | Redirect to the Document-filtered Sources list. |
## List Behavior
- The title is **Archival Documents**.
- **Create new document** opens the create route.
- The table defaults to Document Title order and supports search and column sorting.
- Columns are Document Title, Type, Author, Document Date, and Archive Ref.
- Document Title is left-aligned; the remaining columns are centered.
- Author lists all linked people in the `author` role.
- Date display prefers exact date, then approximate date, then `Unknown`.
- Selecting a row opens Document Detail.
- No records displays `No documents found in repository.`
## Create and Edit Behavior
Required:
- Document name.
- Document type selected from the Document Type registry.
Optional:
- Exact date.
- Approximate date.
- Document location.
- Archive identifier.
- Notes.
- Multiple people for every configured Person Role.
Rules:
- Exact date must parse as `YYYY-MM-DD`; browser presentation may follow locale.
- Existing people appear with disambiguating labels.
- **Create new person** opens Person creation.
- `person_id` may preselect that Person in the author role on Document creation.
- An invalid requested Person produces a warning rather than a broken form.
- `return_to=jobs_new` returns a successful create to Job creation with the new Document selected.
- Edit includes active and inactive Document Types so historical values remain maintainable.
- Save success returns to Document Detail.
## Detail Behavior
- The heading shows name, type, and internal ID.
- The first Source, when present, appears in the dark-room viewer.
- Archival Metadata shows authors, compact Document date, location, and archive identifier. Notes appear in a separate archival-notes block within the same card.
- System Logistics shows created and updated timestamps.
- Related People are grouped by role and link to Person Detail.
- **Sources & Pipeline Jobs** shows counts and actions for filtered Sources, Document Jobs, and adding a Job.
- **Edit Document** and **Delete** are available from the header.
- Invalid IDs and missing Documents produce explicit states without rendering a partial page.
## Document Jobs Behavior
- The page lists the Document's Jobs newest first with status and Job ID.
- **Open Job** navigates to Job Detail.
- **Create Job** opens Job creation with the Document selected.
- No jobs displays an explicit empty state.
## Delete Behavior
- Deletion is blocked while any Source or Job belongs to the Document.
- The blocked state names the dependency categories and provides navigation back and to Jobs.
- An unlinked Document requires an explicit permanent-delete action.
- Success returns to the Documents list.
## Acceptance Checklist
- List columns, alignment, search, sorting, date fallback, and row navigation match this contract.
- Create/edit enforce name, registered type, and valid exact-date input.
- Multiple people can be selected independently for each configured role.
- Person-first Document creation preselects the requested Person as author.
- Detail links people, Sources, and Jobs to the correct records.
- Delete never removes a Document with Source or Job dependencies.
- Service failures use the shared error presenter and never report false success.
## Implementation Anchors
- `src/transcription/ui/pages/documents_page.py`
- `src/transcription/ui/components/table/documents.py`
- `src/transcription/services/documents.py`
- `src/transcription/services/people.py`
- `tests/ui/test_documents_page.py`
- `tests/services/test_document_service.py`
## Known Limitations and Deferred Work
- Document creation persists the Document before adding relationship links; a later link failure is surfaced but is not currently one atomic write.
- Source ordering controls are deferred to the [draft V4.3 scope](../../ver4.3/scope_boundary_v4_3.md).
docs/ui/pages/home.md
+63
View File
@@ -0,0 +1,63 @@
# Home Page Contract
## Purpose
Home provides a user-maintained landing page for the local archive. It combines one current image with Markdown text and lets the operator edit both without changing application source or prompt assets.
## Routes
| Route | Browser path | Purpose |
| --- | --- | --- |
| `/homepage` | `/ui/homepage` | View current homepage image and Markdown. |
| `/homepage/edit` | `/ui/homepage/edit` | Upload an image and edit Markdown. |
The application root and `/ui` redirect to `/ui/homepage`.
## View Behavior
- The visible page heading is **Home**; the browser tab title is **VibeScribe Home**.
- The latest homepage image appears in the shared dark-room viewer.
- Saved Markdown is rendered in the **Home Text** card.
- Missing text displays `No homepage text saved yet.`
- Missing image displays the viewer's empty state.
- **Edit Home Page** opens the edit route.
## Edit Behavior
- The image upload accepts JPEG, PNG, GIF, WebP, BMP, and TIFF files.
- A successful upload immediately stores the file, updates the preview to that image, and displays a positive notification.
- The Markdown textarea is initialized from the currently stored homepage text.
- **Save** writes the textarea content, displays `Homepage saved`, and returns to Home.
- **Cancel** returns to Home without saving textarea changes. An image already uploaded during the edit session remains stored.
## Storage Contract
- Homepage content is mutable application data under `data/homepage`.
- Markdown is stored in `homepage.md`.
- Uploaded images keep a sanitized basename.
- The view selects the supported image with the most recent modification time.
- Homepage files are not transcription prompts and are not database records.
## Acceptance Checklist
- `/`, `/ui`, and the application brand reach Home.
- Home renders with or without stored Markdown and image content.
- Edit loads existing Markdown.
- A supported image upload updates the preview and becomes the latest homepage image.
- Save persists Markdown and returns to Home.
- Cancel does not save changed Markdown.
## Implementation Anchors
- `src/transcription/ui/pages/home_page.py`
- `src/transcription/ui/homepage_store.py`
- `src/transcription/ui/components/app_shell.py`
- `tests/ui/test_upload_page.py`
- `tests/ui/test_navigation_and_mounts.py`
- `tests/ui/test_pages_registration.py`
## Known Limitations
- Homepage storage is fixed under the repository/application `data` directory rather than a configured application-data root.
- Uploading an image is immediate and is not rolled back by Cancel.
- The editor does not currently delete or select among previously uploaded images.
docs/ui/pages/jobs.md
+94
View File
@@ -0,0 +1,94 @@
# Jobs Page Contract
## Purpose
Jobs manages transcription processing runs. A Job belongs to one Document, links one or more Source pages, records processing provenance, and exposes lifecycle actions without making lifecycle fields directly editable.
## Routes
| Route | Purpose |
| --- | --- |
| `/jobs` | Searchable processing Job list. |
| `/jobs/new` | Create and queue a Job. |
| `/jobs/{job_id}` | View status, execution logistics, and related records. |
| `/jobs/{job_id}/cancel` | Confirm cancellation. |
| `/jobs/{job_id}/resubmit` | Confirm resubmission of failed Sources. |
| `/jobs/{job_id}/delete` | Confirm or block deletion. |
## List Behavior
- The title is **Transcription Pipeline Jobs**.
- **Create job** opens Job creation and **Refresh** reloads the table.
- Columns are Job ID, Status, Source Filename, Retries, Created, and Updated.
- Search covers Job ID, filename, and status.
- Status is displayed as a semantic status chip.
- Selecting a row opens Job Detail.
- No records displays `No job records found in repository.`
## Create Behavior
- A Target Document and at least one source file are required.
- `document_id` may preselect a Target Document.
- If no Documents exist, the page explains the prerequisite and links to Document creation with a return path.
- Provider and Model are optional request overrides.
- Upload accepts JPEG, PNG, TIFF, and PDF files and supports multiple/folder selection.
- The visible upload queue is sorted alphabetically by original filename.
- Files can be removed individually or cleared before submission.
- Helper text explains numeric filename prefixes for page ordering.
- Submission creates the Job, Source records, and JobSource links, notifies the worker, and opens Job Detail.
## Detail and Lifecycle Behavior
- The heading shows Job ID and a status badge.
- Execution Logistics shows provider, model, prompt, retry count, and last update.
- Document Links open the parent Document and Job-filtered Sources.
- Queued and processing Jobs show an auto-refresh notice and reload every four seconds.
- Polling stops when the Job becomes terminal or a refresh fails.
- Queued and processing Jobs expose **Cancel**.
- Jobs other than `transcribed` expose **Resubmit** under the current UI rule. The service blocks resubmission while processing is active or when no failed Sources exist.
- All Jobs expose **Delete Job**, subject to delete guardrails.
- Invalid and missing IDs produce explicit states.
## Cancel Behavior
- The confirmation explains that processing stops and remaining non-transcribed Sources become failed.
- The service decides whether the current state permits cancellation.
- Success updates the Job, notifies the worker, and returns to Job Detail.
## Resubmit Behavior
- The page shows current status and failed Source count.
- The page explicitly states: `Resubmit queues only failed linked sources. New results overwrite prior page-level results.`
- The service blocks submission while processing is active or when no failed Sources exist.
- Current behavior updates the existing page-level result when new output arrives.
- Success reports the number of resubmitted Sources and returns to Job Detail.
## Delete Behavior
- Deletion is blocked while status is `processing`.
- Allowed deletion warns that related JobSource links are removed.
- Success returns to the Jobs list.
## Acceptance Checklist
- Job creation cannot proceed without a valid Document and at least one Source.
- Upload ordering and removal controls match the displayed queue.
- Detail shows current status and provenance summary with correct related links.
- Active Jobs refresh without overlapping permanent polling after terminal state.
- Cancel, resubmit, and delete honor service guardrails and show actionable failures.
- Lifecycle fields cannot be edited directly.
## Implementation Anchors
- `src/transcription/ui/pages/jobs_page.py`
- `src/transcription/ui/components/table/jobs.py`
- `src/transcription/services/jobs.py`
- `src/transcription/services/store.py`
- `src/transcription/services/workflows.py`
- `tests/ui/test_jobs_page.py`
- `tests/services/test_job_service.py`
- `tests/services/test_store.py`
## Planned Change
V4.2 replaces update-in-place retry evidence with append-only processing attempts and adds exact transport evidence. Until implemented, the current overwrite behavior must be labeled accurately rather than described as archival history. See the [V4.2 scope](../../ver4.2/scope_boundary_v4_2.md).
docs/ui/pages/people.md
+90
View File
@@ -0,0 +1,90 @@
# People Page Contract
## Purpose
People manages reusable historical-person records. A Person may appear in many Documents under different relationship roles and may optionally carry a portrait and FamilySearch identifier.
## Routes
| Route | Purpose |
| --- | --- |
| `/people` | Searchable People list. |
| `/people/new` | Create a Person. |
| `/people/{person_id}` | View one Person and linked Documents. |
| `/people/{person_id}/edit` | Edit the Person. |
| `/people/{person_id}/delete` | Confirm permanent deletion. |
## List Behavior
- The title is **Archival Entities: People**.
- **Create new person** opens the create route.
- The table defaults to Full Name order and supports search and column sorting.
- Columns are Full Name, Display Name, Maiden Name, Birth Date, and Death Date.
- Full Name is left-aligned; Display Name, Maiden Name, and date columns are centered.
- Birth and death values independently prefer exact date, then approximate date, then `Unknown`.
- Selecting a row opens Person Detail.
- No records displays `No person records found in repository.`
## Create and Edit Behavior
Required:
- Full name.
Optional:
- Display name and maiden name.
- Exact and approximate birth/death dates.
- Birth/death places.
- Biography.
- Portrait path or uploaded portrait.
- FamilySearch ID.
Rules:
- Missing Full name blocks save with a warning.
- Exact date inputs are native browser date inputs.
- FamilySearch IDs are normalized and validated by `PeopleService`.
- Portrait uploads are stored under the configured upload root in a Person-specific directory and update Portrait path.
- Metadata JSON remains hidden.
- Save success returns to Person Detail.
## Detail Behavior
- The header provides **New Document**, **Edit Person**, and **Delete**.
- **New Document** opens Document creation with this Person requested for author preselection.
- The portrait viewer resolves supported relative upload paths and absolute HTTP/data URLs.
- Biographical Record shows names, compact birth/death dates, places, and an **Open in FamilySearch** link when an ID exists.
- Biography has an explicit empty value.
- Linked Documents show Document name, relationship role, and an action to open Document Detail.
- No links shows both an empty state and guidance to link from a Document workflow.
- System Logistics shows created and updated timestamps.
## Delete Behavior
- The page warns when linked Document relationships exist.
- Confirmed deletion removes the Person and its relationship links; it does not delete Documents.
- Success returns to the People list.
- Missing or already-deleted records return to a safe list state.
## Acceptance Checklist
- List fields, alignment, date fallback, search, sorting, and navigation match this contract.
- Full name is enforced on create and edit.
- FamilySearch ID validation and link generation use the fixed supported identifier format.
- Portrait upload and rendering remain constrained to supported media paths.
- New Document carries the Person context.
- Linked Documents show the correct role and target.
- Delete wording distinguishes removal of relationship links from deletion of Documents.
## Implementation Anchors
- `src/transcription/ui/pages/people_page.py`
- `src/transcription/ui/components/table/people.py`
- `src/transcription/services/people.py`
- `tests/ui/test_people_page.py`
- `tests/services/test_v2_crud.py`
## Deferred Work
- Structured name fields, merge/deduplication, advanced metadata editing, and Person-side relationship editing are not current behavior.
docs/ui/pages/sources.md
+86
View File
@@ -0,0 +1,86 @@
# Sources Page Contract
## Purpose
Sources manages individual archived page/file records. It provides source-media viewing, current processing context, provider evidence inspection, previous/next page navigation, and human revision without allowing machine output to be edited.
## Routes
| Route | Purpose |
| --- | --- |
| `/sources` | Global or filtered Source list. |
| `/sources/{source_id}` | View media, transcription, revision, metadata, and evidence. |
| `/sources/{source_id}/delete` | Confirm or block deletion. |
The list accepts optional `document_id` and `job_id` query parameters. Document context takes precedence if both parse successfully.
## List Behavior
- The title is **Source Asset Records**, **Sources for Document**, or **Sources for Job** according to context.
- Global context provides **Create Job**.
- Filtered context provides **Back to Document** or **Back to Job**.
- Rows are ordered by page number and then upload name.
- Columns are Document Name, Page Number, Upload Title, Status, and Error Detail.
- Document Name, Upload Title, and Error Detail are left-aligned; Status is centered.
- Stored Filename is intentionally absent from the list.
- Selecting a row opens Source Detail.
- No records displays `No source asset records found in repository.`
## Detail Behavior
- The heading shows page number, upload name, and Source ID.
- **Back to Sources** returns to the global list.
- **Delete Source** opens the guarded delete route.
- Previous and Next navigate only among Sources belonging to the same Document in page order; unavailable boundary actions are disabled.
- The media viewer resolves the stored Source path through the configured upload root.
- Transcription Text is read-only and prefers the latest JobSource transcription, then the Source projection.
- Editable Revision is seeded from an existing revision or the machine transcription.
- Source Metadata shows upload name, stored filename, page number, Document Name, Document ID, and stored path. Source ID appears in the page-header subtitle.
- SourceJob Metadata shows latest status, Job ID, execution time, provider, model, prompt, and failure detail.
- Revision Logistics shows revised state, last-revised time, and upload time.
## Provider Evidence
- Provider Evidence is associated with the latest JobSource execution.
- AI Metadata and the current `raw_api_response` value are displayed as expandable formatted JSON.
- Missing evidence has an explicit empty state.
- Under the current V4 implementation, `raw_api_response` is an OpenRouter SDK response snapshot, not an exact HTTP or native upstream-provider response.
## Revision Behavior
- Machine transcription is never edited directly.
- A revision must contain non-whitespace text.
- Save persists revised text and updates the saved timestamp without leaving the page.
- Reset restores the in-memory revision from page load or the most recent successful save. When no revision exists, it restores the machine transcription; it does not re-read the database.
- A failed latest execution displays guidance that a human revision can preserve corrected text.
## Delete Behavior
- Deletion is allowed only when the Source has no JobSource links.
- A linked Source shows cleanup guidance and navigation to Jobs.
- An unlinked Source requires explicit permanent deletion.
- Success returns to the Sources list.
## Acceptance Checklist
- Global, Document-filtered, and Job-filtered lists show the correct context and return action.
- List columns and alignments match this contract and omit Stored Filename.
- Previous/next navigation never crosses Document boundaries.
- Detail keeps machine output read-only and human revision separately editable.
- Empty, failed, and missing-evidence states remain explicit.
- JSON evidence is readable without being mislabeled as native transport evidence.
- Delete cannot remove a Source with processing-history links.
## Implementation Anchors
- `src/transcription/ui/pages/sources_page.py`
- `src/transcription/ui/components/table/sources.py`
- `src/transcription/services/sources.py`
- `tests/ui/test_sources_page.py`
- `tests/services/test_transcription_service.py`
- `tests/services/test_v2_crud.py`
## Planned Changes
- V4.2 will rename and separate evidence layers, add exact OpenRouter transport capture, and preserve append-only attempts. See the [V4.2 scope](../../ver4.2/scope_boundary_v4_2.md).
- Source page reordering is deferred to the [draft V4.3 scope](../../ver4.3/scope_boundary_v4_3.md).
docs/ver1/architecture_v1.md
-308
View File
@@ -1,308 +0,0 @@
# System Architecture (Version 1)
This document describes the production architecture of the personal historical-document transcription system. The system is intentionally optimized for single-user operation, low operational overhead, and clean internal boundaries that support future growth without rewrites.
## Architecture Objectives
The production architecture is designed to:
- preserve verbatim family-history source material as searchable text
- keep operational complexity low for a personal deployment
- support asynchronous transcription without requiring distributed infrastructure
- maintain clear module boundaries so extensions can be added incrementally
## Production Scope And Scale
The deployed system targets personal use and a corpus of several thousand documents processed over time. The architecture favors simple, composable building blocks over distributed orchestration.
Current scope includes:
- content source upload and metadata capture
- asynchronous transcription jobs
- prompt-library driven transcription behavior, with one Markdown file per prompt
- original transcription review and optional revision review
- full-text search over accepted transcripts
- export of transcript data
## Deployment Topology
The production deployment uses [Docker Compose](https://docs.docker.com/compose/) and treats containerized databases as extremely lightweight operational dependencies.
Running [PostgreSQL](https://www.postgresql.org/docs/) in its own container is considered simple by default for this system.
Running [MongoDB](https://www.mongodb.com/docs/) in its own container is also considered simple when document-centric storage is enabled.
Container count is not a hard architectural limit; a three-container deployment (app, PostgreSQL, MongoDB) is an acceptable baseline.
### Baseline Topology (Two Containers)
- one application container
- one PostgreSQL container
- embedded background worker execution inside the app process
### Expanded Topology (Three Containers)
- application container
- PostgreSQL container
- MongoDB container
No additional queue, scheduler, or search-engine containers are required in the baseline production setup.
## Runtime Architecture
```mermaid
flowchart LR
User[Browser User] --> App[FastAPI + NiceGUI Service]
App --> Worker[In-process Background Worker]
App --> PG[(PostgreSQL)]
App --> MG[(MongoDB Document Store)]
Worker --> AI[Transcription Provider]
Worker --> PG
Worker --> MG
```
## Runtime Ownership And Startup Policy
The current implementation now uses explicit lifespan-owned runtime resources.
- application lifespan initializes and disposes database runtime resources
- worker lifecycle is owned by application lifespan startup/shutdown
- worker receives lifespan-owned database engine dependency explicitly
- schema bootstrap policy is environment-aware and explicit:
- development/test default to bootstrap enabled
- production defaults to bootstrap disabled
- explicit override is available via configuration
This aligns implementation toward REQ-7 and REQ-10 while preserving personal-scale operational simplicity.
## Layered Module Structure
### Interface Layer
Responsibility:
- HTTP API and UI routes
- request/response validation
- status and result presentation
Out of scope:
- business-rule enforcement
- data-access implementation
### Application Layer
Responsibility:
- upload and job orchestration
- state transitions and retry policy
- coordination across domain and infrastructure ports
Out of scope:
- provider-specific protocol details
- ORM or storage-specific logic
### Domain Layer
Responsibility:
- verbatim transcription policy
- revision and provenance invariants
- confidence and annotation semantics
Out of scope:
- web framework concerns
- database and network I/O
### Infrastructure Layer
Responsibility:
- persistence adapters (PostgreSQL and MongoDB)
- transcription-provider adapter
Out of scope:
- business policy decisions
## Processing Workflow
Production transcription flow:
1. A user uploads one or more content sources through the UI or API.
2. The application validates payloads and creates document, source, and job records.
3. The in-process worker de-queues the job and calls the transcription provider.
4. The application persists original transcription output on the job, plus confidence metadata and provenance events.
5. Job status transitions from queued to processing to transcribed or failed.
6. The UI and API expose status, optional revision to original transcription, and searchable transcription text.
## Data Model Ownership
System-of-record entities:
- documents and content sources
- transcription jobs, original transcription, and status events
- transcript revisions
- provenance metadata
### Original Transcription And Revision Ownership
- each processing job stores the original immutable provider output (`text`)
- provider metadata (`provider`, `model`, `prompt_name`) and failure detail (`error_detail`) are job-owned processing artifacts
- revisions are optional user-authored edits linked to a content source
- a revision can be created from original `job.text`
- many jobs will have zero revisions; revisions are additive and never overwrite original provider output
- a document groups one or more content sources (images, PDFs, and future source types)
Storage strategy:
- PostgreSQL for relational system-of-record entities
- MongoDB for document-oriented payloads and large transcription artifacts
- versioned prompt artifacts stored as individual Markdown files for human editing and refinement
- in-memory execution state treated as ephemeral
## Transcription Prompt Asset Policy
The production system treats transcription prompts as maintainable content assets.
- each transcription prompt is stored in its own Markdown file
- prompt files are designed for direct human editing and iterative refinement
- prompt updates are independent and do not require bundling unrelated prompt changes
- prompt file identity and revision history are tracked through normal repository version control
## Simplicity Guardrails
The production system enforces these constraints to prevent accidental over-engineering:
- PostgreSQL in a container is treated as a lightweight default dependency
- MongoDB in a container is treated as a lightweight optional dependency
- three containers (app, PostgreSQL, MongoDB) is an acceptable simple deployment
- no dedicated queue or search cluster is introduced without measured need
- external infrastructure is added only behind existing ports/adapters
## Extension Path
The architecture supports additive growth without changing domain contracts.
### Stage 1: Foundation (Current)
- upload, transcription, review, search, export
- in-process worker execution
- single provider adapter
- app plus PostgreSQL deployment
### Stage 2: Throughput Hardening
- optional MongoDB document-store enablement
- optional external worker/queue process
- stronger retry and dead-letter handling
### Stage 3: Intelligence Features
- entity extraction and cross-document linking
- timeline and narrative assembly
- optional multi-provider routing
Each stage preserves existing module boundaries and keeps migration risk low.
## Test Strategy
The test strategy is aligned to personal-scale operation with fast, deterministic feedback.
### Unit Tests
- domain transcription rules and annotation behavior
- revision-history invariants
- job state-transition logic
### Integration Tests
- repository behavior and transaction boundaries
- persistence-adapter and provider adapter contract mapping
- upload-to-persistence roundtrip
### End-to-End Tests
- happy path: upload, transcribe, review, search, export
- failure path: provider error, retry, surfaced failed status
### CI Execution Model
- fast suite on each push
- optional slower provider-sandbox checks on scheduled runs
## Risks And Controls
### Runtime Responsiveness
Risk:
- long jobs can reduce responsiveness in a single-process deployment
Control:
- bounded concurrency and visible job status in the UI
### Database Concurrency Limits
Risk:
- contention can appear under sustained concurrent writes in personal-scale infrastructure
Control:
- tuned connection pooling and phased use of MongoDB for document-heavy workloads
### Provider Output Variance
Risk:
- transcription quality varies by content source type, handwriting legibility, and source quality
Control:
- first-class human review and immutable revision history
---
## Technology References
- [FastAPI documentation](https://fastapi.tiangolo.com/)
- [NiceGUI documentation](https://nicegui.io/documentation)
- [Docker Compose documentation](https://docs.docker.com/compose/)
- [PostgreSQL documentation](https://www.postgresql.org/docs/)
- [MongoDB documentation](https://www.mongodb.com/docs/)
## Related Local References
- [System Overview](index_v1.md)
- [System Design Intent](intent.md)
- [Transcription Methodology](transcription_methodology.md)
- System Architecture (this document)
- [System Requirements](requirements_v1.md)
- [Data model](schema_v1.md)
- [Error Handling Policy](error_handling_v1.md)
- [Implementation Plan](implementation_plan_v1.md)
## Glossary
- Adapter: A component that translates between internal interfaces and external systems such as databases or AI services.
- Background job: Work executed outside the request/response path so the UI remains responsive.
- Boundary: A strict separation between modules with different responsibilities.
- CI (Continuous Integration): Automated test execution for code changes.
- Contract test: A test that verifies an adapter follows expected input/output behavior at a boundary.
- Domain layer: The module that contains core business rules and invariants.
- End-to-end test: A test that validates a full user flow across the running system.
- Full-text search: Text indexing and querying optimized for natural-language search.
- In-process worker: A background executor that runs within the same application process.
- Integration test: A test that verifies interactions between real modules and infrastructure components.
- MongoDB: A document-oriented database used for flexible, high-variance data structures.
- Modular monolith: A single deployable application with strongly separated internal modules.
- Port/Interface: A stable contract used by application/domain code to call infrastructure implementations.
- Prompt artifact: A single Markdown file that defines one transcription prompt and can be revised independently.
- Provenance: Metadata that records where generated data came from and how it was produced.
- Revision history: Optional versioned record of user-authored transcription edits over time.
- System of record: The authoritative persistent store for canonical data.
- Vertical slice: A minimal end-to-end feature path spanning UI/API, application logic, and persistence.
docs/ver1/error_handling_v1.md
-288
View File
@@ -1,288 +0,0 @@
# Error Handling Policy
This document defines the canonical error-handling policy for the document transcription system. It is the single source of truth for how errors are classified, surfaced to users, logged for diagnosis, and handled across UI, API, service, worker, and provider boundaries.
## Error Handling Objectives
The production error-handling model is designed to:
- make failures visible to the user in clear, actionable language
- preserve enough diagnostic detail for fast troubleshooting
- keep module behavior consistent across all boundaries
- distinguish expected domain failures from unexpected defects
- support safe retries for transient failures without hiding persistent faults
## Scope And Authority
This page governs error-handling behavior for:
- UI interactions (NiceGUI pages)
- API endpoints (FastAPI routes)
- application services and orchestration logic
- in-process background worker execution
- external provider adapters and persistence adapters
If implementation behavior conflicts with this document, this document is authoritative and implementation should be updated.
## Core Principles
- **Clarity first:** user-facing messages should explain what failed in plain language.
- **Actionability required:** each surfaced error should include a suggested next step.
- **Safety by default:** internal details are logged; sensitive details are not exposed by default in UI/API.
- **Consistency across boundaries:** category and structure should remain stable from source to surface.
- **Fail explicitly:** silent failure is prohibited.
- **Traceability:** every non-trivial error should be traceable with an error reference ID.
## Error Taxonomy
The system uses stable, implementation-independent categories:
| Category | Definition | Typical Source | Retriable |
| --- | --- | --- | --- |
| `validation_error` | Payload or parameter shape/content is invalid | UI/API input validation, service guards | no |
| `user_input_error` | User-provided artifact is unacceptable though structurally valid | unsupported file type, empty file, oversized upload | sometimes |
| `not_found_error` | Requested resource does not exist | missing job/document/source/revision | no |
| `conflict_error` | Requested operation violates current state constraints | invalid state transition | no |
| `external_provider_error` | External AI/provider call fails | upstream HTTP/API/provider failures | sometimes |
| `infrastructure_transient_error` | Temporary environment issue | network timeout, DB connection reset | yes |
| `infrastructure_persistent_error` | Non-transient environment issue | missing permissions, misconfiguration | no |
| `internal_unexpected_error` | Unhandled defect or unknown failure | uncaught exceptions, logic errors | unknown (default no) |
### Classification Rules
- Classification occurs as close as possible to the origin boundary.
- Provider-specific exceptions must be normalized into taxonomy categories before crossing service boundaries.
- Unknown exceptions are classified as `internal_unexpected_error` and logged with traceback.
- Category names are stable contracts and must not be changed casually.
## User-Facing Error Experience Contract
When an error is shown in the GUI, it must include:
1. **Title** (short context, e.g., “Upload failed”)
2. **Message** (plain-language explanation)
3. **Suggested action** (explicit next step)
4. **Error reference ID** (for support/debug traceability)
5. **Technical details** (optional/collapsible for advanced users)
### UI Message Rules
- Do not expose raw stack traces by default.
- Do not expose secrets, credentials, connection strings, or filesystem internals unless explicitly in debug tooling.
- Prefer domain language over implementation language.
- Use persistent visibility for important failures (dialog/card), not only transient toasts.
### Suggested Action Requirements
Every user-visible error must include a suggested course of action, such as:
- retry the operation
- check file type/size constraints
- refresh the jobs page
- verify environment configuration
- contact operator with error ID and timestamp
## API Error Response Contract
API errors should return a structured envelope with stable fields:
- `error_id`: short unique reference ID
- `category`: taxonomy category
- `message`: safe human-readable summary
- `suggestion`: recommended next step
- `details`: optional, only when safe and appropriate
- `timestamp`: UTC ISO-8601
HTTP status mapping guidance:
- `validation_error`, `user_input_error` -> `400`
- `not_found_error` -> `404`
- `conflict_error` -> `409`
- `external_provider_error` -> `502` or `503` (depending on failure mode)
- `infrastructure_transient_error` -> `503`
- `infrastructure_persistent_error` -> `500`
- `internal_unexpected_error` -> `500`
## Logging And Observability Contract
All logged errors must include, where available:
- `error_id`
- `category`
- `operation` (e.g., `upload.submit`, `worker.process_job`, `jobs.refresh`)
- `exception_type`
- `job_id`, `document_id`, `source_id` (when relevant)
- UTC timestamp
Rules:
- Use structured logging fields where practical.
- Use full traceback for unexpected errors (`internal_unexpected_error`).
- Log at boundary handoff points to preserve causal trail.
- Avoid duplicate noisy logging for the same exception at every layer.
## Recovery And Retry Policy
### Retriable Conditions
Retriable failures include:
- transient network/provider timeouts
- intermittent provider unavailability
- temporary DB/network interruptions
### Non-Retriable Conditions
Non-retriable failures include:
- invalid file formats
- missing required data
- permission/configuration failures
- deterministic domain conflicts
### Worker Behavior
- The worker must classify and persist failure details consistently.
- Retries should be bounded by configured limits.
- Exhausted retries must end in explicit failed status with recorded reason.
- No infinite retry loops are allowed.
## Boundary-Specific Responsibilities
### UI Layer
Responsibility:
- display user-safe error summaries and suggested actions
- show persistent error visibility for critical failures
- include error reference IDs in visible output
Out of scope:
- low-level exception parsing
- provider-specific protocol interpretation
### API Layer
Responsibility:
- map application exceptions into stable error envelopes and HTTP statuses
- preserve category and error_id continuity
Out of scope:
- domain-specific remediation logic
### Service Layer
Responsibility:
- classify domain and infrastructure exceptions
- convert adapter-specific failures into taxonomy categories
- return deterministic error types to callers
Out of scope:
- presentation formatting for UI
### Worker Layer
Responsibility:
- execute retry policy for retriable failures
- persist terminal failure details for jobs
- emit operational logs with category and identifiers
Out of scope:
- direct UI messaging
### Provider Adapter Layer
Responsibility:
- normalize provider SDK/HTTP failures into domain-neutral exceptions
- preserve raw provider context for logs (safely)
Out of scope:
- choosing user-facing wording
## Error Lifecycle Workflow
Standard lifecycle:
1. Failure occurs at a boundary or operation.
2. Exception is classified into taxonomy category.
3. `error_id` is created (or propagated).
4. Error is logged with required structured fields.
5. User/API receives safe message + suggested action.
6. Persistent job/resource state is updated when applicable.
7. Tests verify contract behavior for the pathway.
## Test Strategy For Error Handling
### Unit Tests
- category classification behavior
- retry eligibility decisions
- exception-to-message mapping safety
### Integration Tests
- UI pathways show clear message + suggested action for known failures
- API returns structured error envelope with expected status/category
- worker persists failed status and failure detail as required
### Regression Tests
- each previously observed production issue should have a guarding test
- contract tests must cover adapter error normalization behavior
## Known Failure Patterns And Prescribed Responses
| Pattern | Category | User Message | Suggested Action |
| --- | --- | --- | --- |
| Upload payload cannot be parsed by UI handler | `internal_unexpected_error` (until narrowed) | Upload failed due to unexpected processing error | Retry once; if repeated, report error ID and check runtime version compatibility |
| Unsupported extension | `user_input_error` | File type is not supported | Upload JPG, PNG, TIFF, or PDF |
| Empty file upload | `validation_error` | Uploaded file is empty | Choose a valid non-empty file and retry |
| Provider timeout | `external_provider_error` or `infrastructure_transient_error` | Transcription provider timed out | Retry from jobs page; if repeated, check provider status |
| Job lookup missing | `not_found_error` | Requested job was not found | Refresh jobs list and open a valid job |
## Governance And Update Process
This document is a living policy artifact.
Update this document when:
- new error categories are introduced
- handling behavior changes at any boundary
- a production incident reveals missing guidance
- API/UI error contracts change
Change requirements:
- update this document and associated tests in the same change set
- preserve taxonomy stability; if changed, document migration impact
- record noteworthy policy changes in project release notes or changelog
---
## Related Local References
- [System Overview](index_v1.md)
- [System Design Intent](intent.md)
- [Transcription Methodology](transcription_methodology.md)
- [System Architecture](architecture_v1.md)
- [System Requirements](requirements_v1.md)
- [Data model](schema_v1.md)
- Error Handling Policy (this document)
- [Implementation Plan](implementation_plan_v1.md)
## Glossary
- Error category: Stable classification used to drive handling, messaging, and status mapping.
- Error envelope: Structured API payload describing a failure.
- Error reference ID: Short identifier used to correlate user-visible failure with logs.
- Retriable error: Failure likely to succeed on a later attempt without code changes.
- Terminal failure: Failure state after retries are exhausted or retry is not allowed.
docs/ver1/implementation_plan_v1.md
-203
View File
@@ -1,203 +0,0 @@
# Version 1 Implementation Plan
This plan defines the path from current implementation to **Version 1 complete**, aligned to the updated domain model:
- `Document` groups one or more content `Source` records
- `Job` owns original immutable provider output (`text`) and processing metadata
- `Revision` stores optional user-authored edits linked to a `Source`
The objective is to complete V1 scope with production readiness while keeping non-V1 enhancements out of active delivery.
---
## V1 Completion Definition
V1 is complete when all of the following are true:
1. **Functional complete**
- Upload, queue, processing, status display, and transcription result inspection work end-to-end.
- Optional revision workflow is implemented (create/view/update single revision).
2. **Data-model complete**
- Runtime behavior, persistence, and tests all align to `Document` / `Source` / `Job` / `Revision`.
3. **Operational complete**
- Error handling, logs, and runbooks support reliable operation.
4. **Documentation complete**
- Architecture, requirements, schema, error handling, and index are consistent and current.
---
## Phase 1 — Data Contract Stabilization (Schema-First)
**Goal:** Lock a single canonical contract before further feature work.
### Tasks
1. Confirm and document invariants:
- `Job.text` is original immutable transcription output.
- `Revision` is optional and user-authored.
- Revisions are derived from the original `Job.text`.
2. Verify relationship cardinality assumptions:
- `Document` -> many `Source`
- `Document` -> many `Job`
- `Source` -> one `Job`
- `Source` -> one `Revision`
3. Ensure field naming consistency (`date_created`, `date_updated`, `date_uploaded`) across code and docs.
4. Freeze V1 status lifecycle to current implementation (`queued`, `processing`, `transcribed`, `failed`).
### Deliverables
- Updated `schema_v1.md` and `requirements.md` traceability alignment.
- Explicit V1 data invariants section in architecture docs.
### Exit Criteria
- No conflicting definitions of ownership/cardinality/status remain in docs.
---
## Phase 2 — Service Layer Refactor To New Model
**Goal:** Remove all obsolete `Transcript` assumptions from service/workflow code.
### Tasks
1. Refactor `services/transcription.py`:
- Replace transcript CRUD assumptions with job-output + revision operations.
2. Refactor `services/jobs.py`:
- Replace old timestamp/relationship accessors with current model fields.
3. Refactor `services/documents.py` and `services/store.py`:
- Ensure upload creates and links `Document`, `Source`, and `Job` correctly.
4. Refactor `services/workflows.py`:
- Persist original provider output to `Job`.
- Persist failure detail to `Job.error_detail`.
- Use `Revision` only for user-authored edits.
### Deliverables
- Service layer fully aligned with new schema.
### Exit Criteria
- No service module imports or persists `Transcript` model artifacts.
---
## Phase 3 — UI Contract Alignment
**Goal:** Align pages/components to source/job/revision semantics.
### Tasks
1. Update job detail and related UI components:
- Display original immutable transcription from `Job.text`.
- Display optional revision sourced from `Source.revision` (0 or 1).
2. Align date fields with new schema naming.
3. Preserve clear user messaging when no revisions exist.
### Deliverables
- Updated jobs page and detail components.
### Exit Criteria
- UI behavior and labels match documentation and domain model.
---
## Phase 4 — Database Bootstrap, Migration, and Safety
**Goal:** Make schema transition safe in dev/test and repeatable for deployment.
### Tasks
1. Update bootstrap compatibility logic in `db/operations.py`:
- Remove obsolete transcript-table assumptions.
- Add forward-compatible patches for current tables only.
2. Define migration/backfill approach for existing local data.
3. Document rollback and recovery steps.
4. Rehearse migration path against representative data.
### Deliverables
- Migration/upgrade runbook.
- Validated bootstrap behavior for dev/test.
### Exit Criteria
- Migration path is documented and tested with no unresolved data-loss risk.
---
## Phase 5 — Test Suite Realignment
**Goal:** Restore full confidence after the schema redesign.
### Tasks
1. Rewrite model tests for:
- `Document`, `Source`, `Job`, `Revision` relationships and invariants.
2. Rewrite service/integration tests:
- Worker success/failure paths using `Job.text` / `Job.error_detail`.
- Optional single-revision creation/update behavior.
3. Update UI tests for new job-detail/revision rendering behavior.
4. Re-enable strict CI quality gates (lint, type, tests).
### Deliverables
- Updated test matrix and passing CI.
### Exit Criteria
- Critical user flows and failure paths are covered and green.
---
## Phase 6 — Reliability, Operations, and Release Readiness
**Goal:** Ensure V1 is operable and launch-safe.
### Tasks
1. Verify error taxonomy behavior across UI/API/service/worker.
2. Confirm structured logging includes relevant identifiers (`job_id`, `document_id`, `source_id` when applicable).
3. Validate retry behavior and terminal failure handling.
4. Finalize release checklist, deployment steps, and rollback procedure.
5. Execute final acceptance run against requirements traceability.
### Deliverables
- V1 release checklist and acceptance evidence.
- `runbook_v1.md` for incident response and operator workflows.
- `release_checklist_v1.md` for release sign-off.
### Exit Criteria
- Stakeholder sign-off and launch readiness achieved.
---
## Requirement Traceability Focus
The plan must keep clear evidence against these requirement groups:
- **Core flow:** REQ-0 to REQ-6
- **Runtime and operations constraints:** REQ-7 to REQ-12
- **Revision workflow:** REQ-13
A lightweight traceability table should be maintained with:
- requirement ID
- implementation status (`not started` / `in progress` / `done`)
- validation evidence (test name, screenshot, or runbook step)
---
## Suggested Execution Rhythm
- **Weekly:** requirement status and risk review
- **Per PR:** contract checks (model names, field names, lifecycle values)
- **Milestone checks:** end of Phases 2, 4, and 6
---
## Scope Discipline Rule (V1 Focus)
- Only work required to satisfy V1 requirements enters this plan.
- Nice-to-have enhancements are captured in a separate backlog document.
- Schema or contract changes after Phase 1 require explicit approval and traceability impact review.
---
## Related Local References
- [System Overview](index_v1.md)
- [System Design Intent](intent.md)
- [Transcription Methodology](transcription_methodology.md)
- [System Architecture](architecture_v1.md)
- [System Requirements](requirements_v1.md)
- [Data model](schema_v1.md)
- [Error Handling Policy](error_handling_v1.md)
- Implementation Plan (this document)
docs/ver1/index_v1.md
-58
View File
@@ -1,58 +0,0 @@
## Document Transcription System Overview
This project is a production application for transcribing and preserving historical family documents. It is intentionally designed for personal-scale use, with a simplicity-first architecture that is easy to operate and easy to extend.
## Start Here
Read [architecture_v1.md](architecture_v1.md) first.
The architecture page is the primary technical reference and defines:
- deployed topology and infrastructure limits
- module boundaries and dependency flow
- processing life cycle and data ownership
- test strategy, risk controls, and extension path
## What The Application Does
At a high level, users upload images or PDFs as content sources for handwritten, typed, or typeset documents, run asynchronous transcription jobs, review optional revisions, and search across accepted text.
### Core capabilities:
- document grouping with one or more content sources and metadata capture
- asynchronous transcription with visible job status
- immutable original transcription persisted with each job (plus provider/model/prompt metadata)
- transcription prompt management with one Markdown file per prompt for human refinement over time
- optional revisions for user-authored edits of original immutable transcription text
- full-text search over accepted transcripts
- export of transcript data
## Production Operating Model
The system runs with minimal operational overhead:
- PostgreSQL in a dedicated Docker container is considered extremely lightweight and simple for this system
- MongoDB in a dedicated Docker container is also considered extremely lightweight and simple for document-centric persistence
- a three-container deployment (app, PostgreSQL, MongoDB) is a simple and acceptable baseline
- no required queue or search-engine containers in the baseline setup
This operating model keeps deployment and maintenance simple while preserving clean boundaries for future scale.
---
## Documentation Map
- System Overview (this document)
- [System Design Intent](intent.md)
- [Transcription Methodology](transcription_methodology.md)
- [System Architecture](architecture_v1.md)
- [System Requirements](requirements_v1.md)
- [Data model](schema_v1.md)
- [Error Handling Policy](error_handling_v1.md)
- [Implementation Plan](implementation_plan_v1.md)
## Glossary
- Document-oriented persistence: Storing data as flexible records instead of fixed relational rows.
- Prompt artifact: A single Markdown file that defines one transcription prompt and is edited independently.
- System of record: The authoritative persistent store for canonical data.
docs/ver1/release_checklist_v1.md
-45
View File
@@ -1,45 +0,0 @@
# V1 Release Readiness Checklist
Use this checklist before declaring V1 operationally complete.
## A) Functional Readiness
- [ ] Upload flow works for supported file types.
- [ ] Worker transitions jobs through `queued -> processing -> transcribed|failed`.
- [ ] Job detail displays immutable original transcription from `Job.text`.
- [ ] Revision workflow supports create/update/view/delete for optional single revision.
## B) Reliability and Error Handling
- [ ] Error categories surface with actionable messages in UI/API pathways.
- [ ] Failed jobs persist `error_detail` and terminal state.
- [ ] Stale processing recovery verified on restart.
- [ ] Retry/timeout behavior validated against configured limits.
## C) Operational Readiness
- [ ] `runbook_v1.md` reviewed and current.
- [ ] `migration_v1.md` reviewed and current.
- [ ] Backup and rollback procedures tested at least once.
- [ ] Incident escalation packet template is known to operators.
## D) Quality Gates
- [ ] Lint/type checks pass.
- [ ] `pytest -m "not external" -q` passes.
- [ ] Targeted external/provider checks executed (if credentials available).
- [ ] Release evidence recorded in `release_evidence_v1.md`.
## E) Traceability and Documentation
- [ ] `requirements_v1.md` aligns with implemented V1 behavior.
- [ ] `architecture_v1.md`, `schema_v1.md`, and `error_handling_v1.md` are consistent.
- [ ] `traceability_v1.md` is updated with current implementation and test evidence.
- [ ] `implementation_plan_v1.md` phase status updated with evidence references.
- [ ] REQ traceability evidence links recorded (tests/runbook/checks).
## Release Sign-Off
- [ ] Technical sign-off complete.
- [ ] Operational sign-off complete.
- [ ] V1 completion date recorded.
docs/ver1/release_evidence_v1.md
-39
View File
@@ -1,39 +0,0 @@
# V1 Release Evidence Log
## Step 5 Quality Gates (2026-07-29)
### Lint
- Command: `python -m ruff check .`
- Result: ✅ pass
- Notes: initial findings were auto-fixed (`ruff --fix`) plus small manual line-wrap/annotation adjustments.
### Tests (primary gate)
- Command: `python -m pytest -m "not external" -q`
- Result: ✅ pass (`[100%]`)
### Tests (external smoke)
- Command: `python -m pytest -m external -q`
- Result: ✅ pass (`[100%]`)
### Type Check
- Command: `python -m ty check src tests`
- Result: ⚠️ not passing
- Summary: existing SQLModel/SQLAlchemy typing incompatibilities and test double typing mismatches remain.
Key current blocker families:
1. SQLModel relationship/query attribute typing (`selectinload`, `order_by`, `.any()`)
2. SQLAlchemy join clause typing in `services/transcription.py`
3. Test fake client type mismatch for `OpenRouterTranscriptionProvider(client=...)`
4. `Settings(**defaults)` typed-dict strictness in `tests/test_config.py`
## Current Gate Status
- Lint: pass
- Non-external tests: pass
- External smoke tests: pass
- Type check: **blocked** (requires dedicated typing cleanup pass)
docs/ver1/requirements_v1.md
-98
View File
@@ -1,98 +0,0 @@
## Document Transcription System Requirements
This page captures a SysML v1.6-style requirements baseline for the production system described in [index_v1.md](index_v1.md). The model is represented as concise tables and traceability lists that preserve SysML-style IDs and relationship semantics.
## Scope
- System of interest: the single Python application service (NiceGUI + FastAPI) with PostgreSQL as the relational system of record and optional MongoDB for document-oriented persistence.
- Operational context: local-first execution with Docker Compose and an intentionally lightweight production trajectory.
- Primary concern: end-to-end transcription job lifecycle from upload through completion or failure.
## Requirements Model (Concise Text Form)
### Requirements
| ID | Category | Requirement | Risk | Verify Method |
| --- | --- | --- | --- | --- |
| REQ-0 | System | Provide end-to-end document transcription with persistent, inspectable lifecycle state. | medium | demonstration |
| REQ-1 | Functional | Allow users to upload one or more images or PDFs as sources from the web UI. | low | test |
| REQ-2 | Functional | Run each upload through asynchronous processing that returns an original transcription or explicit failure. | high | test |
| REQ-3 | Functional | Persist and expose job states: queued, processing, transcribed, failed. | high | inspection |
| REQ-4 | Functional | Persist transcription output, processing history, and failure details. | medium | test |
| REQ-5 | Interface | Expose API and UI views for status inspection and completed transcription reading. | medium | demonstration |
| REQ-6 | Performance | Trigger background processing on upload to preserve UI responsiveness. | medium | analysis |
| REQ-7 | Design Constraint | Keep lifespan-owned runtime resources: SQLAlchemy engine, async session factory, worker resources, provider clients. | medium | inspection |
| REQ-8 | Design Constraint | Initialize configuration and logging once at startup through centralized mechanisms. | low | inspection |
| REQ-9 | Design Constraint | Use Docker Compose baseline of app plus PostgreSQL; allow optional MongoDB container when enabled. | medium | demonstration |
| REQ-10 | Design Constraint | Keep schema bootstrap explicit and opt-in; normal startup does not mutate production schema. | high | inspection |
| REQ-11 | Design Constraint | Use service-backed persistence for core document and job data. | medium | inspection |
| REQ-12 | Design Constraint | Store transcription prompts as individual Markdown artifacts for iterative refinement. | medium | inspection |
| REQ-13 | Functional | Allow users to create one optional revision of transcription text derived from the original job transcription. | low | test |
### Requirement Relationships
- Contains: REQ-0 contains REQ-1 through REQ-13.
- Derives: REQ-2 -> REQ-3, REQ-3 -> REQ-4.
- Traces: REQ-5 -> REQ-3.
- Refines: REQ-6 -> REQ-2.
### Architecture Elements
| Element | Type | Doc Reference |
| --- | --- | --- |
| UI | NiceGUI pages | src/transcription/ui/pages |
| API | FastAPI routes | src/transcription/api/routes.py |
| GRAPH | Async processing workflow | src/transcription/services, src/transcription/ai |
| DBREL | PostgreSQL + SQLModel relational persistence | src/transcription/db |
| DBDOC | MongoDB document persistence | src/transcription/db, src/transcription/services |
| OPS | Docker Compose runtime | docker-compose.yml |
| PROMPTS | Transcription prompt artifact library (Markdown files) | .github/prompts, docs |
| TESTS | Pytest verification suite | tests |
### Satisfaction Mapping
- UI satisfies REQ-1, REQ-5, REQ-13.
- API satisfies REQ-5.
- GRAPH satisfies REQ-2, REQ-6.
- DBREL satisfies REQ-3, REQ-10, REQ-13.
- DBDOC satisfies REQ-4, REQ-11.
- OPS satisfies REQ-9.
- PROMPTS satisfies REQ-12.
### Verification Mapping
- TESTS verifies REQ-1, REQ-2, REQ-3, REQ-4, REQ-5, REQ-10, REQ-11, REQ-12, REQ-13.
## Requirement Notes
- Requirement IDs (`REQ-*`) are stable references for planning, implementation, and test traceability.
- The model uses compact tables and traceability lists for renderer compatibility while preserving SysML-style requirement IDs and relationship semantics.
- Requirement categories (functional, interface, performance, and design constraints) are preserved as explicit REQ entries and relationship labels to keep change impact visible.
- PostgreSQL containerization and optional MongoDB containerization are both treated as extremely lightweight and simple operational choices in this architecture.
## Verification Intent
- Demonstration: validate end-to-end behavior via running system flows and operator-visible outcomes.
- Inspection: verify architecture and startup/runtime policies in code and configuration.
- Analysis: evaluate asynchronous execution behavior and design sufficiency.
- Test: automate behavioral checks through pytest suites and service-level tests.
---
## Related Local References
- [System Overview](index_v1.md)
- [System Design Intent](intent.md)
- [Transcription Methodology](transcription_methodology.md)
- [System Architecture](architecture_v1.md)
- System Requirements (this document)
- [Data model](schema_v1.md)
- [Error Handling Policy](error_handling_v1.md)
- [Implementation Plan](implementation_plan_v1.md)
## Glossary
- Document-oriented persistence: A storage approach that uses flexible document structures for variable data shapes.
- Prompt artifact: A single Markdown file that defines one transcription prompt and is revised independently.
- SysML: Systems Modeling Language used to express structured requirements and traceability.
- System of record: The authoritative persistent store for canonical business data.
docs/ver1/runbook_v1.md
-129
View File
@@ -1,129 +0,0 @@
# V1 Operations Runbook
This runbook provides day-2 operational procedures for the V1 baseline.
## Scope
Applies to:
- local/hosted V1 runtime
- SQLite-backed persistence
- in-process worker lifecycle
- OpenRouter provider integration
## Preconditions
- `.env` contains `OPENROUTER_API_KEY`
- app starts successfully
- `uploads/` and `prompts/` are writable
- health endpoint responds at `/healthz`
## Standard Startup Procedure
1. Start the app using the project-standard command.
2. Open `/healthz` and verify `{"status":"ok"}`.
3. Open `/ui/upload` and submit a small valid file.
4. Confirm job transitions from `queued` -> `processing` -> `transcribed` (or `failed` with detail).
## Standard Shutdown Procedure
1. Stop the application process.
2. Ensure no active process still holds the SQLite file.
3. If maintenance is planned, copy the DB file before edits:
- `transcription.db` (or configured `DATABASE_URL` file path)
## Incident: Jobs Stuck In `processing`
### Symptoms
- Jobs remain `processing` for longer than provider timeout
- New uploads queue but do not complete
- provider usage increases but no terminal job state is visible
### Checks
1. Confirm app process is still running.
2. Confirm worker loop is active (startup logs include worker lifespan start).
3. Inspect recent app logs for:
- `worker.process_job`
- `error_id`
- `category`
- `job_id` / `document_id` / `source_id`
4. Verify provider credentials and provider status.
### Recovery
1. Restart the app to trigger stale-processing recovery.
2. On startup, app re-queues stale processing jobs based on timeout policy.
3. Re-check jobs page and confirm terminal state progression.
4. If persistent, capture logs + error IDs and move to deep investigation.
## Incident: Provider Authentication Failures
### Symptoms
- failures categorized as provider/auth
- jobs fail quickly with authentication guidance
### Recovery
1. Validate `OPENROUTER_API_KEY` value.
2. Restart app after updating env.
3. Re-run a small transcription to confirm recovery.
## Incident: Upload Failures
### Symptoms
- UI reports upload errors
- unsupported extension or empty payload
### Recovery
1. Validate file extension (`.jpg`, `.jpeg`, `.png`, `.tif`, `.tiff`, `.pdf`).
2. Validate file is not empty.
3. Validate upload directory permissions.
4. Retry upload.
## Incident: Database File/Permission Issues
### Symptoms
- persistence errors during upload/job update
- startup failures around schema/runtime
### Recovery
1. Confirm the configured DB file path exists and is writable.
2. Confirm parent directory permissions.
3. Restore from last known backup copy if corruption is suspected.
4. Restart app and run smoke test.
## Logging Requirements (Operational)
Operational triage should always capture:
- `error_id`
- category
- operation name
- `job_id`, `document_id`, `source_id` when applicable
- UTC timestamp
## Escalation Packet (When opening an issue)
Include:
- exact timestamp window
- one failing `job_id`
- relevant `error_id` values
- latest 100 lines of app logs
- environment summary (`DATABASE_URL` type, app version/commit)
## Post-Incident Validation
After mitigation, verify:
1. Upload works.
2. One job reaches `transcribed`.
3. One induced failure reaches `failed` with error detail.
4. Jobs page and detail page render correctly.
docs/ver1/schema_v1.md
-98
View File
@@ -1,98 +0,0 @@
## Database Schema (V1 Baseline)
This document describes the current relational schema for the transcription system.
All primary and foreign keys in the domain models are UUID-based in V1.
---
## Schema Diagram
```mermaid
erDiagram
DOCUMENT {
UUID id PK
TEXT name
}
JOB {
UUID id PK
UUID document_id FK
TEXT status
INTEGER retry_count
DATETIME date_created
DATETIME date_updated
TEXT provider
TEXT model
TEXT prompt_name
TEXT text
TEXT error_detail
}
SOURCE {
UUID id PK
UUID document_id FK
UUID job_id FK
TEXT upload_name
TEXT filename
TEXT file_path
DATETIME date_uploaded
}
REVISION {
UUID id PK
UUID source_id "FK, UK"
INTEGER revision
TEXT text
DATETIME date_created
}
DOCUMENT ||--o{ SOURCE : has_many
DOCUMENT ||--o{ JOB : has_many
JOB ||--o{ SOURCE : referenced_by
SOURCE ||--o| REVISION : has_optional_one
```
---
## Table Relationships and Constraints
- A `Document` can have zero or more `Source` records.
- A `Document` can have zero or more `Job` records.
- A `Source` belongs to exactly one `Document` and one `Job`.
- A `Source` may have one optional `Revision`.
- Optional `0..1` revision cardinality is enforced by uniqueness on `revision.source_id`.
### Invariants
- `Job.text` stores immutable original provider transcription output.
- `Revision` rows are optional user-authored edits derived from original transcription.
- Revisions do not overwrite original `Job.text`.
- Job status lifecycle values are: `queued`, `processing`, `transcribed`, `failed`.
### Timestamp Fields
- `Job.date_created`
- `Job.date_updated`
- `Source.date_uploaded`
- `Revision.date_created`
---
## Related Local References
- [System Overview](index_v1.md)
- [System Design Intent](intent.md)
- [Transcription Methodology](transcription_methodology.md)
- [System Architecture](architecture_v1.md)
- [System Requirements](requirements_v1.md)
- Data model (this document)
- [Error Handling Policy](error_handling_v1.md)
- [Implementation Plan](implementation_plan_v1.md)
## Glossary
- **Document**: logical grouping for one or more transcribed sources.
- **Source**: uploaded file content (image/PDF) linked to a job.
- **Job**: processing record that stores lifecycle status and original output.
- **Revision**: optional single user-authored edited text linked to a source.
docs/ver1/traceability_v1.md
-40
View File
@@ -1,40 +0,0 @@
# V1 Traceability Matrix
This matrix provides implementation and validation evidence for V1 requirements (`REQ-0` through `REQ-13`).
Status values:
- `done`: implemented and evidence recorded
- `in progress`: partially implemented or evidence incomplete
- `not started`: no implementation/evidence yet
## Requirement Evidence Table
| Requirement | Status | Implementation Evidence | Validation Evidence |
| --- | --- | --- | --- |
| REQ-0 | done | End-to-end upload + worker pipeline in `src/transcription/services/store.py`, `src/transcription/worker.py`, `src/transcription/services/workflows.py` | `tests/integration/test_pipeline_flow.py` |
| REQ-1 | done | Upload UI/page flow in `src/transcription/ui/pages/upload_page.py`, `src/transcription/ui/components/upload.py` | `tests/ui/test_upload_page.py`, `tests/integration/test_pipeline_flow.py` |
| REQ-2 | done | Async worker execution and provider call orchestration in `src/transcription/worker.py`, `src/transcription/services/workflows.py` | `tests/integration/test_pipeline_flow.py`, `tests/services/test_workflows_reliability.py` |
| REQ-3 | done | Job lifecycle state model + transitions in `src/transcription/models.py`, `src/transcription/services/jobs.py`, `src/transcription/services/workflows.py` | `tests/services/test_job_service.py`, `tests/ui/test_jobs_page.py` |
| REQ-4 | done | Persistence of original output and failure detail in `src/transcription/services/transcription.py`, `src/transcription/services/workflows.py` | `tests/integration/test_pipeline_flow.py`, `tests/services/test_workflows_reliability.py` |
| REQ-5 | done | Status/result inspection via UI pages and API health route in `src/transcription/ui/pages/jobs_page.py`, `src/transcription/api/health.py` | `tests/ui/test_jobs_page.py`, `tests/ui/test_pages_registration.py`, `tests/api/test_health.py` |
| REQ-6 | done | Background processing trigger/worker notifier and non-blocking workflow in `src/transcription/ui/components/upload.py`, `src/transcription/worker.py` | `tests/test_app.py`, `tests/services/test_workflows_reliability.py` |
| REQ-7 | done | Lifespan-owned runtime resources in `src/transcription/app.py`, `src/transcription/db/runtime.py` | `tests/test_app.py`, `tests/test_db.py` |
| REQ-8 | done | Centralized settings/logging initialization in `src/transcription/config.py`, `src/transcription/app.py` | `tests/test_config.py`, `tests/test_app.py` |
| REQ-9 | done | Containerized runtime baseline in `docker-compose.yml`, `Dockerfile` | `release_checklist_v1.md` (Ops checklist), manual demonstration step |
| REQ-10 | done | Explicit schema bootstrap policy + runtime controls in `src/transcription/config.py`, `src/transcription/app.py`, `src/transcription/db/operations.py` | `tests/test_db.py`, `tests/test_config.py` |
| REQ-11 | done | Service/workflow persistence boundaries in `src/transcription/services/*.py`, `src/transcription/services/workflows.py` | `tests/services/test_job_service.py`, `tests/services/test_transcription_service.py` |
| REQ-12 | done | Prompt artifacts in `prompts/` and loading/validation in `src/transcription/services/transcription.py` | `tests/test_prompts.py` |
| REQ-13 | done | Optional single revision create/update/view/delete in `src/transcription/services/transcription.py`, `src/transcription/ui/pages/jobs_page.py` | `tests/services/test_transcription_service.py`, `tests/ui/test_jobs_page.py` |
## Operational Evidence (Step 3 Artifacts)
- Runbook: `runbook_v1.md`
- Migration/backfill/rollback guidance: `migration_v1.md`
- Release readiness checklist: `release_checklist_v1.md`
## Verification Cadence
- Per change: maintain `tests/test_traceability.py` mappings for touched requirements.
- Per milestone: update this table status and evidence links.
- Pre-release: confirm all rows are `done` and non-external suite is green.
docs/ver2/V2 Context for DB Updates.md
-35
View File
@@ -1,35 +0,0 @@
# AI Coding Assistant Project Briefing & Context
## Project Mission
This application is a family history archival and transcription platform. Its primary goal is to accept scanned document images (letters, postcards, logbooks, diaries), execute OCR and structured transcription via AI vision models (OpenAI GPT-4o, Anthropic Claude 3.5 Sonnet), and manage historical metadata (authors, recipients, dates, and locations).
---
## Technical Stack & Architecture
* **Database:** PostgreSQL 13+ with native `UUID` (`gen_random_uuid()`) and `JSONB` columns.
* **Backend Runtime / Concurrency:** Python utilizing `asyncio` for concurrent HTTP API calls to AI providers, with strict rate-limiting via `asyncio.Semaphore`.
* **Validation & Types:** Python with **Pydantic** model definitions. Incoming AI responses must be parsed and validated with Pydantic models *before* database insertion.
* **ORM / Database Access:** SQLModel and SQLAlchemy, using parameterized statements and PostgreSQL-native types.
---
## Core System Directives for AI Code Generation
### 1. Data Immutability vs. Human Corrections
* `job_source.raw_transcription` and `source.raw_transcription` represent original, point-in-time machine outputs and are **immutable**.
* Human corrections occur on `source.revised_text`.
* When fetching text for the UI, always display `COALESCE(source.revised_text, source.raw_transcription)`.
### 2. Async Execution & Batching Rules
* A `job` represents an overarching execution run for a folder/group of images belonging to a single `document`.
* Images are submitted to AI APIs **one at a time in rapid succession** using `asyncio` worker pools.
* Each single-image API call populates a row in `job_source` with its own `status`, `raw_transcription`, `ai_metadata`, and `raw_api_response`.
* If 9 of 10 pages succeed and 1 fails, `job_source.status` for the failed image becomes `'failed'`, while `job.status` becomes `'partial_success'`. Do not mark the entire batch as failed if partial results exist.
### 3. Entity Relationships
* **Authors/Recipients:** A `document` can have multiple authors and recipients. Do NOT put direct `author_id` foreign keys on `document`. Query authors/recipients via `document_person` where `role = 'author'` or `role = 'recipient'`.
* **Page Ordering:** Multi-page documents must always be queried using `ORDER BY page_number ASC`.
### 4. Database Mutations
* Always use parameterized SQL queries (`$1`, `$2`) to prevent SQL injection.
* Store datetimes using UTC ISO 8601 strings or native PostgreSQL `TIMESTAMPTZ`.
docs/ver2/V2 Python Pydantic Models.md
-416
View File
@@ -1,416 +0,0 @@
# SQLModel Table Models
These models implement the canonical [Version 2 database schema](../schema_v2.md). Each schema entity is represented by exactly one `SQLModel` table class. Because `SQLModel` is built on Pydantic and SQLAlchemy, these classes provide application validation and PostgreSQL mappings without parallel row and create models.
Database-generated UUIDs and timestamps are `None` until PostgreSQL supplies their values during insert. The database columns remain non-nullable. `Person.metadata_` maps to the `metadata` column because `metadata` is reserved by SQLAlchemy's declarative API.
```python
from datetime import date
from datetime import datetime
from enum import StrEnum
from uuid import UUID
from pydantic import JsonValue
from sqlalchemy import Column
from sqlalchemy import Date
from sqlalchemy import DateTime
from sqlalchemy import ForeignKey
from sqlalchemy import Index
from sqlalchemy import Integer
from sqlalchemy import String
from sqlalchemy import Text
from sqlalchemy import UniqueConstraint
from sqlalchemy import text
from sqlalchemy.dialects.postgresql import JSONB
from sqlalchemy.dialects.postgresql import UUID as PostgreSQLUUID
from sqlmodel import Field
from sqlmodel import Relationship
from sqlmodel import SQLModel
class PersonRole(StrEnum):
AUTHOR = "author"
RECIPIENT = "recipient"
class JobStatus(StrEnum):
QUEUED = "queued"
PROCESSING = "processing"
COMPLETED = "completed"
PARTIAL_SUCCESS = "partial_success"
FAILED = "failed"
class JobSourceStatus(StrEnum):
PENDING = "pending"
TRANSCRIBED = "transcribed"
FAILED = "failed"
class Person(SQLModel, table=True):
__tablename__ = "person"
__table_args__ = (Index("idx_person_full_name", "full_name"),)
id: UUID | None = Field(
default=None,
sa_column=Column(
PostgreSQLUUID(as_uuid=True),
primary_key=True,
server_default=text("gen_random_uuid()"),
),
)
full_name: str = Field(sa_column=Column(Text, nullable=False))
display_name: str | None = Field(default=None, sa_column=Column(Text))
maiden_name: str | None = Field(default=None, sa_column=Column(Text))
birth_date: date | None = Field(default=None, sa_column=Column(Date))
birth_date_raw: str | None = Field(default=None, sa_column=Column(Text))
birth_place: str | None = Field(default=None, sa_column=Column(Text))
death_date: date | None = Field(default=None, sa_column=Column(Date))
death_date_raw: str | None = Field(default=None, sa_column=Column(Text))
death_place: str | None = Field(default=None, sa_column=Column(Text))
biography: str | None = Field(default=None, sa_column=Column(Text))
portrait_path: str | None = Field(default=None, sa_column=Column(Text))
metadata_: JsonValue | None = Field(
default_factory=dict,
sa_column=Column(
"metadata",
JSONB,
server_default=text("'{}'::jsonb"),
),
)
created_at: datetime | None = Field(
default=None,
sa_column=Column(
DateTime(timezone=True),
nullable=False,
server_default=text("now()"),
),
)
updated_at: datetime | None = Field(
default=None,
sa_column=Column(
DateTime(timezone=True),
nullable=False,
server_default=text("now()"),
),
)
document_people: list["DocumentPerson"] = Relationship(
back_populates="person",
sa_relationship_kwargs={"lazy": "raise", "passive_deletes": True},
)
class Document(SQLModel, table=True):
__tablename__ = "document"
__table_args__ = (Index("idx_document_date", "document_date"),)
id: UUID | None = Field(
default=None,
sa_column=Column(
PostgreSQLUUID(as_uuid=True),
primary_key=True,
server_default=text("gen_random_uuid()"),
),
)
name: str = Field(sa_column=Column(Text, nullable=False))
document_type: str | None = Field(default=None, sa_column=Column(Text))
document_date: date | None = Field(default=None, sa_column=Column(Date))
document_date_raw: str | None = Field(default=None, sa_column=Column(Text))
location_created: str | None = Field(default=None, sa_column=Column(Text))
notes: str | None = Field(default=None, sa_column=Column(Text))
archive_identifier: str | None = Field(default=None, sa_column=Column(Text))
created_at: datetime | None = Field(
default=None,
sa_column=Column(
DateTime(timezone=True),
nullable=False,
server_default=text("now()"),
),
)
updated_at: datetime | None = Field(
default=None,
sa_column=Column(
DateTime(timezone=True),
nullable=False,
server_default=text("now()"),
),
)
document_people: list["DocumentPerson"] = Relationship(
back_populates="document",
sa_relationship_kwargs={"lazy": "raise", "passive_deletes": True},
)
jobs: list["Job"] = Relationship(
back_populates="document",
sa_relationship_kwargs={"lazy": "raise", "passive_deletes": True},
)
sources: list["Source"] = Relationship(
back_populates="document",
sa_relationship_kwargs={"lazy": "raise", "passive_deletes": True},
)
class DocumentPerson(SQLModel, table=True):
__tablename__ = "document_person"
__table_args__ = (
UniqueConstraint(
"document_id",
"person_id",
"role",
name="unique_document_person_role",
),
Index("idx_document_person_doc", "document_id"),
Index("idx_document_person_per", "person_id"),
)
id: UUID | None = Field(
default=None,
sa_column=Column(
PostgreSQLUUID(as_uuid=True),
primary_key=True,
server_default=text("gen_random_uuid()"),
),
)
document_id: UUID = Field(
sa_column=Column(
PostgreSQLUUID(as_uuid=True),
ForeignKey("document.id", ondelete="CASCADE"),
nullable=False,
),
)
person_id: UUID = Field(
sa_column=Column(
PostgreSQLUUID(as_uuid=True),
ForeignKey("person.id", ondelete="CASCADE"),
nullable=False,
),
)
role: PersonRole = Field(sa_column=Column(String(20), nullable=False))
created_at: datetime | None = Field(
default=None,
sa_column=Column(
DateTime(timezone=True),
nullable=False,
server_default=text("now()"),
),
)
document: Document | None = Relationship(
back_populates="document_people",
sa_relationship_kwargs={"lazy": "raise"},
)
person: Person | None = Relationship(
back_populates="document_people",
sa_relationship_kwargs={"lazy": "raise"},
)
class Job(SQLModel, table=True):
__tablename__ = "job"
__table_args__ = (Index("idx_job_document", "document_id"),)
id: UUID | None = Field(
default=None,
sa_column=Column(
PostgreSQLUUID(as_uuid=True),
primary_key=True,
server_default=text("gen_random_uuid()"),
),
)
document_id: UUID = Field(
sa_column=Column(
PostgreSQLUUID(as_uuid=True),
ForeignKey("document.id", ondelete="CASCADE"),
nullable=False,
),
)
status: JobStatus = Field(
default=JobStatus.QUEUED,
sa_column=Column(
String(50),
nullable=False,
server_default=text("'queued'"),
),
)
retry_count: int = Field(
default=0,
sa_column=Column(
Integer,
nullable=False,
server_default=text("0"),
),
)
provider: str = Field(sa_column=Column(Text, nullable=False))
model: str = Field(sa_column=Column(Text, nullable=False))
prompt_name: str | None = Field(default=None, sa_column=Column(Text))
date_created: datetime | None = Field(
default=None,
sa_column=Column(
DateTime(timezone=True),
nullable=False,
server_default=text("now()"),
),
)
date_updated: datetime | None = Field(
default=None,
sa_column=Column(
DateTime(timezone=True),
nullable=False,
server_default=text("now()"),
),
)
document: Document | None = Relationship(
back_populates="jobs",
sa_relationship_kwargs={"lazy": "raise"},
)
job_sources: list["JobSource"] = Relationship(
back_populates="job",
sa_relationship_kwargs={"lazy": "raise", "passive_deletes": True},
)
class Source(SQLModel, table=True):
__tablename__ = "source"
__table_args__ = (
Index("idx_source_document", "document_id"),
Index("idx_source_page_order", "document_id", "page_number"),
)
id: UUID | None = Field(
default=None,
sa_column=Column(
PostgreSQLUUID(as_uuid=True),
primary_key=True,
server_default=text("gen_random_uuid()"),
),
)
document_id: UUID = Field(
sa_column=Column(
PostgreSQLUUID(as_uuid=True),
ForeignKey("document.id", ondelete="CASCADE"),
nullable=False,
),
)
page_number: int = Field(
default=1,
sa_column=Column(
Integer,
nullable=False,
server_default=text("1"),
),
)
upload_name: str = Field(sa_column=Column(Text, nullable=False))
filename: str = Field(sa_column=Column(Text, nullable=False))
file_path: str = Field(sa_column=Column(Text, nullable=False))
raw_transcription: str | None = Field(default=None, sa_column=Column(Text))
revised_text: str | None = Field(default=None, sa_column=Column(Text))
date_uploaded: datetime | None = Field(
default=None,
sa_column=Column(
DateTime(timezone=True),
nullable=False,
server_default=text("now()"),
),
)
date_revised: datetime | None = Field(
default=None,
sa_column=Column(DateTime(timezone=True)),
)
document: Document | None = Relationship(
back_populates="sources",
sa_relationship_kwargs={"lazy": "raise"},
)
job_sources: list["JobSource"] = Relationship(
back_populates="source",
sa_relationship_kwargs={"lazy": "raise", "passive_deletes": True},
)
class JobSource(SQLModel, table=True):
__tablename__ = "job_source"
__table_args__ = (
UniqueConstraint("job_id", "source_id", name="unique_job_source"),
Index("idx_job_source_job", "job_id"),
Index("idx_job_source_source", "source_id"),
Index(
"idx_job_source_ai_metadata",
"ai_metadata",
postgresql_using="gin",
),
)
id: UUID | None = Field(
default=None,
sa_column=Column(
PostgreSQLUUID(as_uuid=True),
primary_key=True,
server_default=text("gen_random_uuid()"),
),
)
job_id: UUID = Field(
sa_column=Column(
PostgreSQLUUID(as_uuid=True),
ForeignKey("job.id", ondelete="CASCADE"),
nullable=False,
),
)
source_id: UUID = Field(
sa_column=Column(
PostgreSQLUUID(as_uuid=True),
ForeignKey("source.id", ondelete="CASCADE"),
nullable=False,
),
)
status: JobSourceStatus = Field(
default=JobSourceStatus.PENDING,
sa_column=Column(
String(50),
nullable=False,
server_default=text("'pending'"),
),
)
raw_transcription: str | None = Field(default=None, sa_column=Column(Text))
ai_metadata: JsonValue | None = Field(
default=None,
sa_column=Column(JSONB),
)
raw_api_response: JsonValue | None = Field(
default=None,
sa_column=Column(JSONB),
)
error_detail: str | None = Field(default=None, sa_column=Column(Text))
executed_at: datetime | None = Field(
default=None,
sa_column=Column(
DateTime(timezone=True),
nullable=False,
server_default=text("now()"),
),
)
job: Job | None = Relationship(
back_populates="job_sources",
sa_relationship_kwargs={"lazy": "raise"},
)
source: Source | None = Relationship(
back_populates="job_sources",
sa_relationship_kwargs={"lazy": "raise"},
)
```
The enum annotations validate application values while the mapped columns retain the `VARCHAR` types specified by the DDL. PostgreSQL owns generated UUIDs and timestamps through `server_default`; call `session.refresh(instance)` after a flush or commit when those generated values are needed immediately.
`ai_metadata`, `raw_api_response`, and `metadata_` accept any JSON value supported by `JSONB`. Validate provider-specific payload structure before assigning it to these fields, while preserving the complete raw response in `raw_api_response`.
Relationships use `lazy="raise"` to prevent implicit database I/O in async code. Queries must explicitly load relationships they need, for example with `selectinload()`.
The schema's behavioral invariants are enforced outside the table shape where appropriate:
- `PersonRole`, `JobStatus`, and `JobSourceStatus` define the exact values listed by the schema.
- `unique_document_person_role` enforces role uniqueness for `(document_id, person_id, role)`.
- Services order document sources by `Source.document_id` and `Source.page_number`.
- Services derive aggregate `Job.status` from related `JobSource.status` values.
- Services preserve `JobSource.raw_transcription` and `JobSource.raw_api_response` as point-in-time outputs while updating the active text on `Source`.
docs/ver2/architecture_v2.md
-136
View File
@@ -1,136 +0,0 @@
# System Architecture (Version 2)
This document describes the V2 production architecture of the personal historical-document transcription system.
## Architecture Objectives
* Preserve source material as immutable transcribed text alongside page-level spatial AI metadata.
* Support batching multi-image and folder uploads cleanly into sequential pages (`page_number`).
* Leverage asynchronous worker pools (`asyncio`) for parallel single-image API execution bounded by rate limiters (`asyncio.Semaphore`).
* Migrate persistence to PostgreSQL using native `UUID`, `TIMESTAMPTZ`, and `JSONB` document storage.
* Standardize all data validation, API parsing, and database models on **Pydantic V2**.
* Support rich historical attribution (multi-author and multi-recipient relationships).
## Runtime Topology
The V2 runtime operates as an asynchronous Python application:
* FastAPI + NiceGUI web application process.
* In-process `asyncio` background task orchestrator for parallel API execution.
* Relational persistence via PostgreSQL (using `asyncpg` or `psycopg3`).
* Pydantic V2 validation layer wrapping API payloads and PostgreSQL `JSONB` schemas.
```mermaid
flowchart LR
U[Browser User] --> A[FastAPI + NiceGUI App]
A --> W[Asyncio Worker Engine]
A --> DB[(PostgreSQL Database)]
W --> P[Vision Provider APIs\nOpenAI / Claude]
W --> DB
```
## Lifecycle Ownership
Application lifespan owns runtime setup/teardown:
* Initialize environment logging and Pydantic configuration.
* Manage asynchronous PostgreSQL connection pools (`asyncpg` / `psycopg3`).
* Execute database migrations and index initialization.
* Recover stale processing jobs on startup.
* Manage graceful shutdown of active `asyncio` worker pools.
## Layered Module Structure
### Interface Layer
* `src/transcription/ui/**` (NiceGUI pages, multi-page renderers, person cards)
* `src/transcription/api/**` (FastAPI routes and JSON error handlers)
### Application & Async Worker Layer
* `src/transcription/services/workflows.py`
* `src/transcription/worker.py`
Responsibilities:
* Batch orchestration and status transitions (`queued` -> `processing` -> `completed` | `partial_success` | `failed`).
* Parallel single-image API execution using `asyncio.gather` bounded by `asyncio.Semaphore`.
* Pydantic schema parsing (`PageAIMetadata`) and validation prior to database storage.
### Domain & Service Layer
* `src/transcription/db/models.py` (SQLModel/Pydantic V2 schema definitions for the current implementation)
* `src/transcription/services/*.py` (Transactional operations for `Document`, `Person`, `Source`, `Job`, and `JobSource`)
### Infrastructure Layer
* `src/transcription/db/**` (PostgreSQL connection pooling and raw parameterized SQL execution)
* `src/transcription/providers/**` (OpenAI & Anthropic Vision SDK adapters)
## Processing Workflow
1. User uploads a folder or batch of images for a `Document`.
2. System creates `Document`, `Job(status='queued')`, and ordered `Source` pages (`page_number = 1..N`).
3. Worker claims job, sets `Job.status = 'processing'`, and spawns parallel `asyncio` tasks bounded by semaphore.
4. Each task calls Vision API for a **single** `Source` image.
5. On task completion:
* Writes a `JobSource` record containing `status='transcribed'`, `raw_transcription`, `ai_metadata` (bounding boxes/confidence), and `raw_api_response`.
* Caches active text to `Source.raw_transcription`.
6. On page failure:
* Writes `JobSource` record with `status='failed'` and `error_detail`.
7. Once all page tasks resolve:
* Marks `Job.status` as `completed` (100% success), `partial_success` (at least 1 success, 1 failure), or `failed` (all failed).
## Domain Ownership & Invariants
* **Immutable AI Outputs:** `source.raw_transcription` and `job_source.raw_transcription` store original, point-in-time machine output and are immutable.
* **Inlined Revisions:** Human corrections occur on `source.revised_text`. UI renders `COALESCE(revised_text, raw_transcription)`.
* **Sequential Integrity:** Multi-page documents are strictly ordered by `source.page_number ASC`.
* **Page Execution Isolation:** A failure on one page image does not invalidate successful transcriptions on sister pages in the same batch job.
## Data Model Summary
* `Document` has many `Source` pages, many `Job` runs, and many `Person` records via `DocumentPerson` junction (`author` or `recipient`).
* `Source` belongs to one `Document` and can be processed across many `JobSource` executions.
* `Job` has many `JobSource` execution records.
## Test Strategy
* Unit tests for Pydantic V2 schemas, custom validators, and JSONB serialization.
* Integration tests for async PostgreSQL connection handling and parameterized queries.
* Async workflow tests using mock AI providers to verify `partial_success` and retry logic.
* UI integration tests for multi-page rendering and person management.
---
## Technology References
- [FastAPI documentation](https://fastapi.tiangolo.com/)
- [NiceGUI documentation](https://nicegui.io/documentation)
- [PostgreSQL documentation](https://www.postgresql.org/docs/)
- [Python asyncio](https://docs.python.org/3/library/asyncio.html#module-asyncio)
- [Pydantic Validation](https://pydantic.dev/docs/validation/latest/get-started/)
- [Pydantic AI](https://pydantic.dev/docs/ai/overview/)
## Related Local References
- [System Overview](index_v2.md)
- [System Design Intent](invariant/intent.md)
- [Transcription Methodology](invariant/transcription_methodology.md)
- System Architecture (this document)
- [System Requirements](requirements_v2.md)
- [Data model](schema_v2.md)
- [Error Handling Policy](error_handling_v2.md)
- [Implementation Plan](implementation_plan_v2.md)
docs/ver2/error_handling_v2.md
-88
View File
@@ -1,88 +0,0 @@
# Error Handling Policy (Version 2)
This document defines the canonical error-handling policy for the V2 document transcription system.
## Error Handling Objectives
* Make failures visible in clear, actionable language at both the document and individual page levels.
* Support **isolated failure handling** in multi-image batches so single page errors do not crash an entire batch job.
* Preserve diagnostic detail (Pydantic validation errors, raw provider responses) in PostgreSQL `JSONB` for fast troubleshooting.
* Ensure consistent error envelope structure across API, UI, and async worker boundaries.
## Scope And Authority
Governs error behavior across NiceGUI pages, FastAPI routes, service orchestration, `asyncio` background tasks, PostgreSQL interactions, and AI provider adapters.
## Error Taxonomy
| Category | Definition | Retriable |
| --- | --- | --- |
| `validation_error` | Pydantic payload or parameter schema validation failure | no |
| `user_input_error` | Unacceptable user file (unsupported image type, corrupt file) | no |
| `not_found_error` | Requested resource (`Document`, `Source`, `Person`, `Job`) missing | no |
| `conflict_error` | Operation violates state constraints (e.g., duplicate `document_person` role) | no |
| `external_provider_error` | AI Provider API failure (rate limit, vision execution error) | yes |
| `infrastructure_transient_error` | Temporary DB connection reset or HTTP timeout | yes |
| `infrastructure_persistent_error` | Database down, missing API credentials, misconfiguration | no |
| `internal_unexpected_error` | Uncaught Python exception or logic defect | no |
## Async Batch & Page-Level Error Behavior
In multi-image `asyncio` batch processing:
1. **Page Isolation:** Exceptions caught during individual page calls are caught within the `asyncio` task wrapper.
2. **Page Record Logging:** Page failure detail is written directly to `job_source.error_detail` and `job_source.status = 'failed'`.
3. **Batch Aggregate State:**
* If **all** page tasks succeed -> `job.status = 'completed'`.
* If **some** page tasks fail -> `job.status = 'partial_success'`.
* If **all** page tasks fail -> `job.status = 'failed'`.
4. **Retry Strategy:** The UI exposes a "Retry Failed Pages" option for `partial_success` jobs, which spawns a new targeted `Job` containing *only* the `Source` IDs marked as `failed`.
## API Error Response Contract
API error responses return a structured JSON envelope:
```json
{
"error_id": "err_uuid_12345",
"category": "validation_error",
"message": "The uploaded payload failed schema validation.",
"suggestion": "Check file format and metadata fields, then try again.",
"details": {
"pydantic_errors": [...]
},
"timestamp": "2026-07-31T07:55:00Z"
}
```
HTTP Status Mappings:
* `validation_error`, `user_input_error` -> `400`
* `not_found_error` -> `404`
* `conflict_error` -> `409`
* `external_provider_error` -> `502` / `503`
* `infrastructure_transient_error` -> `503`
* `infrastructure_persistent_error`, `internal_unexpected_error` -> `500`
---
## Technology References
- [FastAPI documentation](https://fastapi.tiangolo.com/)
- [NiceGUI documentation](https://nicegui.io/documentation)
- [PostgreSQL documentation](https://www.postgresql.org/docs/)
- [Python asyncio](https://docs.python.org/3/library/asyncio.html#module-asyncio)
- [Pydantic Validation](https://pydantic.dev/docs/validation/latest/get-started/)
- [Pydantic AI](https://pydantic.dev/docs/ai/overview/)
## Related Local References
- [System Overview](index_v2.md)
- [System Design Intent](invariant/intent.md)
- [Transcription Methodology](invariant/transcription_methodology.md)
- [System Architecture](architecture_v2.md)
- [System Requirements](requirements_v2.md)
- [Data model](schema_v2.md)
- Error Handling Policy (this document)
- [Implementation Plan](implementation_plan_v2.md)
docs/ver2/implementation_plan_v2.md
-64
View File
@@ -1,64 +0,0 @@
# implementation_plan_v2
## Goal
Replace the current V1 SQLModel schema with the approved V2 schema and make sure every database operation works through the existing async SQLAlchemy/SQLModel session layer.
Use a fresh database. There will be no migrations, data conversion, legacy compatibility shims, or parallel V1/V2 code paths.
## Current Project Impact
- `src/transcription/db/models.py` still defines the V1 `Document`, `Source`, `Job`, and `Revision` tables.
- The V2 target adds `Person`, `DocumentPerson`, and `JobSource`, moves revisions onto `Source`, and removes the direct `Source.job_id` relationship.
- The engine, session factory, transaction handling, and PostgreSQL async support already exist and do not need to be rewritten.
- Async CRUD currently lives in `DocumentService`, `JobService`, `TranscriptionService`, and the upload record helper. Their queries and eager-loading options depend on V1 relationships.
- Existing tests cover only part of the schema and CRUD surface.
## Implementation
### 1. Update the schema
- Replace the models in `src/transcription/db/models.py` with the approved V2 tables, enums, relationships, foreign keys, constraints, and indexes.
- Remove `Revision`, `Source.job_id`, and the transcription fields that no longer belong on `Job`.
- Keep `create_all()` as the schema bootstrap for a fresh database.
- Delete `_ensure_sqlite_compat_columns()` and all schema patching from `src/transcription/db/operations.py`.
- Keep the Python models and `docs/schema_v2.md` consistent.
### 2. Align the async CRUD methods
- Keep the existing `ServiceBase` session and transaction pattern.
- Update document CRUD to load and manage its ordered `Source` rows and `DocumentPerson` links.
- Update job CRUD and queue queries to use `JobSource` instead of `Source.job_id`.
- Add the missing async CRUD operations for `Person`, `Source`, `DocumentPerson`, and `JobSource` using the existing service style. Do not add another repository abstraction.
- Replace revision CRUD with direct updates to `Source.revised_text` and `Source.date_revised`.
- Remove the temporary transcript compatibility aliases instead of redirecting them.
- Update only direct database call sites that construct or query these records; UI and worker feature changes are not part of this work.
### 3. Verify the schema and CRUD
- Update the schema bootstrap test to expect `person`, `document`, `document_person`, `source`, `job`, and `job_source`, with no `revision` table.
- Add async create, read, update, delete, list, and filtered-query tests for each entity that exposes those operations.
- Test relationship loading, page ordering, uniqueness constraints, delete behavior, status values, and `JobSource` JSON fields.
- Test both service-owned sessions and caller-provided sessions so flush/commit behavior remains correct.
- Run the focused database and service tests, then the full suite with `uv run pytest`.
### 4. Update the UI for the V2 schema
- Review the UI components and views that display document, job, person, and source data so they reference the V2 schema instead of V1 relationships.
- Update upload, detail, and listing screens to show the new person and source associations, revised-source fields, and the revised status values.
- Keep the UI behavior aligned with the updated service layer and ensure the existing UI tests continue to pass with the V2 data model.
- Consider the guidance in `docs/ui_style_guide.md` when making UI changes so the updated views remain consistent with the project’s visual and interaction conventions.
## Done When
- A fresh database is created directly from the V2 SQLModel metadata.
- All async CRUD methods pass against the V2 relationships and fields.
- No code references `Revision`, `Source.job_id`, removed `Job` transcription fields, or compatibility aliases.
- The focused tests and full test suite pass.
## Out of Scope
- Database migrations or preservation of V1 data
- Legacy compatibility code
- Database engine or session-layer rewrites
- UI redesign, batch orchestration, worker concurrency, deployment, and operational runbooks
docs/ver2/index_v2.md
-47
View File
@@ -1,47 +0,0 @@
# Document Transcription System Overview (Version 2)
This project is a personal-scale application for transcribing, indexing, and preserving historical family documents, letters, postcards, and journals.
## Start Here
Read [architecture_v2.md](architecture_v2.md) first for technical overview and system design.
## Core V2 Capabilities
* **Folder & Multi-Image Ingestion:** Upload whole folders or image batches that map sequentially (`page_number`) under a single `Document`.
* **Parallel Async AI Vision Engine:** Concurrently process single-page image transcriptions using Python `asyncio` bounded by rate limiters.
* **Robust PostgreSQL Storage:** Relational storage for entities with native `UUID`, `TIMESTAMPTZ`, and `JSONB` for deep AI spatial metadata and raw envelopes.
* **Pydantic V2 Validation:** End-to-end type safety, DB row mapping, and JSONB payload validation.
* **Historical Person Management:** Track authors and recipients across documents with rich biographical entities (`Person`).
* **Page-Level Execution Auditing & Revisions:** Store immutable point-in-time machine output per run while enabling inline human corrections (`revised_text`).
* **Partial Failure Recovery:** Bounded batch execution that isolates single-page API errors (`partial_success`) for simple retries.
## Technical Stack
* **Application Web Framework:** FastAPI + NiceGUI
* **Persistence Engine:** PostgreSQL 18+
* **Data Validation & Schemas:** Pydantic V2
* **Concurrency & Workers:** Python `asyncio` worker pool with `asyncio.Semaphore`
* **Vision Providers:** OpenAI (GPT-4o) and Anthropic (Claude 3.5 Sonnet) via native SDKs
---
## Technology References
- [FastAPI documentation](https://fastapi.tiangolo.com/)
- [NiceGUI documentation](https://nicegui.io/documentation)
- [PostgreSQL documentation](https://www.postgresql.org/docs/)
- [Python asyncio](https://docs.python.org/3/library/asyncio.html#module-asyncio)
- [Pydantic Validation](https://pydantic.dev/docs/validation/latest/get-started/)
- [Pydantic AI](https://pydantic.dev/docs/ai/overview/)
## Documentation Index
- System Overview (this document)
- [System Design Intent](invariant/intent.md)
- [Transcription Methodology](invariant/transcription_methodology.md)
- [System Architecture](architecture_v2.md)
- [System Requirements](requirements_v2.md)
- [Data model](schema_v2.md)
- [Error Handling Policy](error_handling_v2.md)
- [Implementation Plan](implementation_plan_v2.md)
docs/ver2/requirements_v2.md
-42
View File
@@ -1,42 +0,0 @@
# Document Transcription System Requirements (Version 2)
This document captures the **Version 2 baseline requirements** for the production implementation.
## Requirements Model
| ID | Category | Requirement | Verify Method |
| --- | --- | --- | --- |
| REQ-0 | System | Provide end-to-end multi-page document transcription with persistent, inspectable async job states. | demonstration |
| REQ-1 | Functional | Allow users to upload folders or multi-image batches as sequential `Source` pages under a `Document`. | test |
| REQ-2 | Functional | Process multi-page jobs asynchronously using an `asyncio` worker pool bounded by rate limits. | test |
| REQ-3 | Functional | Persist page-level execution outputs (`raw_transcription`, `ai_metadata`, `raw_api_response`) on `JobSource`. | test |
| REQ-4 | Functional | Support job states (`queued`, `processing`, `completed`, `partial_success`, `failed`) and page states (`pending`, `transcribed`, `failed`). | inspection |
| REQ-5 | Functional | Allow users to manage historical `Person` records and link multiple authors/recipients to a `Document` via `DocumentPerson`. | test |
| REQ-6 | Functional | Maintain immutable original machine output on `Source.raw_transcription` while permitting inline human edits on `Source.revised_text`. | test |
| REQ-7 | Data Constraint | Store all persistent domain data in PostgreSQL using native `UUID`, `TIMESTAMPTZ`, and `JSONB` columns. | inspection |
| REQ-8 | Data Constraint | Validate all API requests, database rows, and JSONB structures using Pydantic V2 schemas. | test |
| REQ-9 | Interface | Render multi-page transcriptions sequentially by `page_number` in the web UI with author/recipient metadata. | demonstration |
| REQ-10 | Operations | Allow operators to retry only failed pages for jobs in a `partial_success` state. | test |
## Element Satisfaction Mapping
* **UI (NiceGUI):** Satisfies REQ-1, REQ-5, REQ-6, REQ-9, REQ-10.
* **API (FastAPI):** Satisfies REQ-1, REQ-4, REQ-5, REQ-8.
* **WORKER (asyncio):** Satisfies REQ-2, REQ-3, REQ-4, REQ-10.
* **PERSISTENCE (PostgreSQL):** Satisfies REQ-3, REQ-6, REQ-7.
* **MODELS (Pydantic V2):** Satisfies REQ-8.
---
## Related Local References
- [System Overview](index_v2.md)
- [System Design Intent](invariant/intent.md)
- [Transcription Methodology](invariant/transcription_methodology.md)
- [System Architecture](architecture_v2.md)
- System Requirements (this document)
- [Data model](schema_v2.md)
- [Error Handling Policy](error_handling_v2.md)
- [Implementation Plan](implementation_plan_v2.md)
docs/ver2/schema_v2.md
-137
View File
@@ -1,137 +0,0 @@
# Database Schema (Version 2)
This document describes the PostgreSQL relational schema for the transcription platform. It incorporates multi-image batch orchestration via `asyncio`, page-level execution tracking, many-to-many author/recipient attribution, and JSONB document storage for AI vision outputs.
All primary and foreign keys are PostgreSQL native UUIDs (`gen_random_uuid()`).
## Entity Relationship Diagram
```mermaid
erDiagram
PERSON {
UUID id PK
TEXT full_name
TEXT display_name
TEXT maiden_name
DATE birth_date
TEXT birth_date_raw
TEXT birth_place
DATE death_date
TEXT death_date_raw
TEXT death_place
TEXT biography
TEXT portrait_path
JSONB metadata
TIMESTAMPTZ created_at
TIMESTAMPTZ updated_at
}
DOCUMENT {
UUID id PK
TEXT name
TEXT document_type
DATE document_date
TEXT document_date_raw
TEXT location_created
TEXT notes
TEXT archive_identifier
TIMESTAMPTZ created_at
TIMESTAMPTZ updated_at
}
DOCUMENT_PERSON {
UUID id PK
UUID document_id FK
UUID person_id FK
VARCHAR role "author | recipient"
TIMESTAMPTZ created_at
}
JOB {
UUID id PK
UUID document_id FK
VARCHAR status "queued | processing | transcribed | completed | partial_success | failed"
INTEGER retry_count
TEXT provider
TEXT model
TEXT prompt_name
TIMESTAMPTZ date_created
TIMESTAMPTZ date_updated
}
SOURCE {
UUID id PK
UUID document_id FK
INTEGER page_number
TEXT upload_name
TEXT filename
TEXT file_path
TEXT raw_transcription
TEXT revised_text
TIMESTAMPTZ date_uploaded
TIMESTAMPTZ date_revised
}
JOB_SOURCE {
UUID id PK
UUID job_id FK
UUID source_id FK
VARCHAR status "pending | transcribed | failed"
TEXT raw_transcription
JSONB ai_metadata
JSONB raw_api_response
TEXT error_detail
TIMESTAMPTZ executed_at
}
DOCUMENT ||--o{ DOCUMENT_PERSON : "has_people"
PERSON ||--o{ DOCUMENT_PERSON : "participates_in"
DOCUMENT ||--o{ JOB : "has_jobs"
DOCUMENT ||--o{ SOURCE : "contains_pages"
JOB ||--o{ JOB_SOURCE : "executes"
SOURCE ||--o{ JOB_SOURCE : "processed_in"
```
## Domain Invariants & Rules
### Page-Level Execution & AI Outputs
* Execution Granularity: Every single image execution by an AI model produces a dedicated record in job_source.
* Source vs Execution Status: `source` does not carry a `status` column. Per-source execution state is tracked in `job_source.status` (`pending`, `transcribed`, `failed`).
* Point-in-Time Auditability: job_source.raw_api_response stores the unparsed REST response envelope for that specific image page call. job_source.ai_metadata stores spatial bounding boxes, token usage, and layout details for that specific image page call.
* Active Output Caching: Upon successful completion of an image call, source.raw_transcription is updated with the latest output string from job_source.raw_transcription for fast UI rendering.
### Page Ordering & Revisions
* Sequential Integrity: source.page_number dictates page ordering within a document. Reads assembling full documents must query ORDER BY source.document_id, source.page_number ASC.
* Inlined Human Corrections: User edits occur at the page level inside source.revised_text. source.raw_transcription remains immutable. If source.revised_text is non-null, application frontends must render source.revised_text.
### Async Job Lifecycle & Failure Isolation
* Batch Orchestrator: A job represents an overarching execution run across one or more source images belonging to a document.
* Isolated Failures: API requests run concurrently (e.g., using asyncio). A failure on page 3 does not invalidate successful transcriptions on page 1 or 2.
* Job States:
- queued: Created, awaiting worker execution.
- processing: Concurrent HTTP tasks actively running.
- completed: 100% of linked job_source tasks succeeded (transcribed).
- partial_success: At least one job_source succeeded and at least one failed.
- failed: All linked job_source tasks failed or a job-level runtime error occurred.
### Attribution & Person Roles
* Multi-Person Roles: Documents support zero, one, or many authors and recipients linked via document_person.
* Role Uniqueness: (document_id, person_id, role) must be unique to prevent duplicate role tagging.
---
## Related Local References
- [System Overview](index_v2.md)
- [System Design Intent](invariant/intent.md)
- [Transcription Methodology](invariant/transcription_methodology.md)
- [System Architecture](architecture_v2.md)
- [System Requirements](requirements_v2.md)
- Data model (this document)
- [Error Handling Policy](error_handling_v2.md)
- [Implementation Plan](implementation_plan_v2.md)
docs/ver3/architecture_v3.md
-142
View File
@@ -1,142 +0,0 @@
# System Architecture (Version 3)
This document describes the V3 production architecture of the personal historical-document transcription system.
## Architecture Objectives
* Preserve source material as immutable transcribed text alongside page-level spatial AI metadata and complete provider API envelopes.
* Support batching multi-image and folder uploads cleanly into sequential pages (`page_number`).
* Capture complete input prompt provenance (`system_prompt`, `user_prompt`, `prompt_hash`) and execution parameters (`temperature`, `top_p`) at submission time on `Job`.
* Leverage asynchronous worker pools (`asyncio`) for parallel single-image API execution bounded by rate limiters (`asyncio.Semaphore`).
* Maintain relational data-model portability across the supported backends by using SQLModel/SQLAlchemy and compatibility types so the same domain schema works in SQLite for local development/testing and PostgreSQL in production.
* Keep operator tooling and local maintenance workflows OS-independent by using Python or other cross-platform interfaces for canonical project automation.
* Verify image asset integrity via SHA-256 file hashing (`file_hash`) while storing binary assets on the local filesystem.
* Standardize all data validation, API parsing, and database models on **Pydantic V2** and **SQLModel**.
* Support rich historical attribution (multi-author and multi-recipient relationships via `DocumentPerson`).
## Runtime Topology
The V3 runtime operates as an asynchronous Python application:
* FastAPI + NiceGUI web application process.
* In-process `asyncio` background task orchestrator for parallel API execution.
* Relational persistence via SQLModel / SQLAlchemy, using SQLite for local development/testing and PostgreSQL as the production persistence target.
* Pydantic V2 validation layer wrapping API payloads, prompt configurations, and JSON metadata schemas.
* Cross-platform operator workflows implemented in Python so core local operations run consistently on Windows, Linux, and macOS.
^^^mermaid
flowchart LR
U[Browser User] --> A[FastAPI + NiceGUI App]
A --> W[Asyncio Worker Engine]
A --> DB[(Relational DB\nSQLite / PostgreSQL)]
W --> P[Vision Provider APIs\nOpenAI / Claude / OpenRouter]
W --> DB
^^^
## Lifecycle Ownership
Application lifespan owns runtime setup/teardown:
* Initialize environment logging, directory paths, and Pydantic configuration.
* Manage asynchronous database engine connection pools (`aiosqlite` or `asyncpg`).
* Execute database bootstrap (`SQLModel.metadata.create_all()`) or migrations.
* Recover stale or interrupted processing jobs on startup.
* Manage graceful shutdown of active `asyncio` worker pools.
## Layered Module Structure
### Interface Layer
* `src/transcription/ui/**` (NiceGUI pages, multi-page renderers, person cards)
* `src/transcription/api/**` (FastAPI routes and JSON error handlers)
### Application & Async Worker Layer
* `src/transcription/services/workflows.py`
* `src/transcription/worker.py`
Responsibilities:
* Batch orchestration and status transitions (`queued` -> `processing` -> `transcribed` | `partial_success` | `failed`).
* Parallel single-image API execution using `asyncio.gather` bounded by `asyncio.Semaphore`.
* Resolve prompt configuration at submission time and persist frozen snapshot fields on `Job`.
* Pydantic schema parsing and validation prior to database storage.
### Domain & Service Layer
* `src/transcription/db/models.py` (SQLModel schema definitions for Document, Source, Job, JobSource, Person, DocumentPerson)
* `src/transcription/services/*.py` (Transactional operations for `Document`, `Person`, `Source`, `Job`, and `JobSource`)
### Infrastructure Layer
* `src/transcription/db/**` (Async database session factory, engine creation, and JSON dialect abstractions)
* `src/transcription/providers/**` (OpenAI, Anthropic, and OpenRouter Vision SDK adapters)
## Processing Workflow
1. User uploads a folder or batch of images for a `Document`.
2. System hashes each image file (SHA-256), writes image files to filesystem storage, and creates `Document`, `Job(status='queued')`, and ordered `Source` pages (`page_number = 1..N`).
3. Worker claims job, sets `Job.status = 'processing'`, and spawns parallel `asyncio` tasks bounded by semaphore.
4. Each task reads the frozen prompt snapshot from `Job` and calls Vision API for a **single** `Source` image.
5. On task completion:
* Writes a `JobSource` record containing `status='transcribed'`, `raw_transcription`, operational `ai_metadata`, and complete unedited `raw_api_response`.
* Caches active output text to `Source.raw_transcription`.
6. On page failure:
* Writes `JobSource` record with `status='failed'` and `error_detail`.
7. Once all page tasks resolve:
* Marks `Job.status` as `transcribed` (100% success), `partial_success` (at least 1 success, 1 failure), or `failed` (all failed).
## Domain Ownership & Invariants
* **Immutable AI Outputs:** `source.raw_transcription` and `job_source.raw_transcription` store original, point-in-time machine output and are immutable.
* **Complete Input & Output Provenance:** Every `job` stores the exact frozen input configuration sent to the model, and every `job_source` stores per-page output evidence including the complete REST response envelope returned.
* **Inlined Revisions:** Human corrections occur on `source.revised_text`. UI renders `COALESCE(revised_text, raw_transcription)`.
* **Sequential Integrity:** Multi-page documents are strictly ordered by `source.page_number ASC`.
* **Page Execution Isolation:** A failure on one page image does not invalidate successful transcriptions on sister pages in the same batch job.
## Data Model Summary
* `Document` has many `Source` pages, many `Job` runs, and many `Person` records via `DocumentPerson` junction (`author` or `recipient`).
* `Source` belongs to one `Document` and can be processed across many `JobSource` executions.
* `Job` has many `JobSource` execution records.
* `JobSource` holds page-level execution status, output text, and raw response JSON.
## Test Strategy
* Unit tests for SQLModel/Pydantic V2 models, JSON cross-dialect serialization, and file hashing functions.
* Integration tests for async database connection handling, session management, and queries.
* Async workflow tests using mock AI providers to verify `partial_success`, page-level failure isolation, and retry logic.
* UI integration tests for multi-page rendering and person attribution management.
---
## Technology References
* [FastAPI documentation](https://fastapi.tiangolo.com/)
* [NiceGUI documentation](https://nicegui.io/documentation)
* [SQLModel documentation](https://sqlmodel.tiangolo.com/)
* [SQLAlchemy Async I/O documentation](https://docs.sqlalchemy.org/en/20/orm/extensions/asyncio.html)
* [Python asyncio](https://www.google.com/search?q=https://docs.python.org/3/library/asyncio.html%23module-asyncio)
* [Pydantic Validation](https://pydantic.dev/docs/validation/latest/get-started/)
## Related Local References
- [System Overview](index_v3.md)
- [System Design Intent](invariant/intent.md)
- [Transcription Methodology](invariant/transcription_methodology.md)
- System Architecture (this document)
- [System Requirements](requirements_v3.md)
- [Data model](schema_v3.md)
- [Error Handling Policy](error_handling_v3.md)
- [Implementation Plan](implementation_plan_v3.md)
docs/ver3/error_handling_v3.md
-87
View File
@@ -1,87 +0,0 @@
# Error Handling Policy (Version 3)
This document defines the canonical error-handling policy for the v3 document transcription system.
## Error Handling Objectives
* Make failures visible in clear, actionable language at both the document and individual page levels.
* Support **isolated failure handling** in multi-image batches so single page errors do not crash an entire batch job.
* Preserve diagnostic detail (Pydantic validation errors, raw provider REST envelopes, exact input prompts) in generic database JSON structures for fast troubleshooting.
* Ensure consistent error envelope structure across API, UI, and async worker boundaries.
## Scope And Authority
Governs error behavior across NiceGUI pages, FastAPI routes, service orchestration, `asyncio` background tasks, database interactions, and AI provider adapters.
## Error Taxonomy
| Category | Definition | Retriable |
| --- | --- | --- |
| `validation_error` | Pydantic payload or parameter schema validation failure | no |
| `user_input_error` | Unacceptable user file (unsupported image type, corrupt file) | no |
| `not_found_error` | Requested resource (`Document`, `Source`, `Person`, `Job`) missing | no |
| `conflict_error` | Operation violates state constraints (e.g., duplicate `document_person` role) | no |
| `external_provider_error` | AI Provider API failure (rate limit, vision execution error) | yes |
| `infrastructure_transient_error` | Temporary DB connection reset or HTTP timeout | yes |
| `infrastructure_persistent_error` | Database down, missing API credentials, misconfiguration | no |
| `internal_unexpected_error` | Uncaught Python exception or logic defect | no |
## Async Batch & Page-Level Error Behavior
In multi-image `asyncio` batch processing:
1. **Page Isolation:** Exceptions caught during individual page calls are trapped within the `asyncio` task wrapper.
2. **Page Record Logging:** Page failure details, along with the prompt inputs and hyperparameters attempted, are written directly to `job_source.error_detail` and `job_source.status = 'failed'`.
3. **Batch Aggregate State:**
* If **all** page tasks succeed -> `job.status = 'completed'`.
* If **some** page tasks fail -> `job.status = 'partial_success'`.
* If **all** page tasks fail -> `job.status = 'failed'`.
4. **Retry Strategy:** The UI exposes a "Retry Failed Pages" option for `partial_success` jobs, which spawns a new targeted `Job` containing *only* the `Source` IDs marked as `failed`.
## API Error Response Contract
API error responses return a structured JSON envelope:
^^^json
{
"error_id": "err_uuid_12345",
"category": "validation_error",
"message": "The uploaded payload failed schema validation.",
"suggestion": "Check file format and metadata fields, then try again.",
"details": {
"pydantic_errors": [...]
},
"timestamp": "2026-08-08T15:00:00Z"
}
^^^
HTTP Status Mappings:
* `validation_error`, `user_input_error` -> `400`
* `not_found_error` -> `404`
* `conflict_error` -> `409`
* `external_provider_error` -> `502` / `503`
* `infrastructure_transient_error` -> `503`
* `infrastructure_persistent_error`, `internal_unexpected_error` -> `500`
---
## Technology References
* [FastAPI documentation](https://fastapi.tiangolo.com/)
* [NiceGUI documentation](https://nicegui.io/documentation)
* [SQLModel documentation](https://sqlmodel.tiangolo.com/)
* [Python asyncio](https://www.google.com/search?q=https://docs.python.org/3/library/asyncio.html%23module-asyncio)
* [Pydantic Validation](https://pydantic.dev/docs/validation/latest/get-started/)
## Related Local References
- [System Overview](index_v3.md)
- [System Design Intent](invariant/intent.md)
- [Transcription Methodology](invariant/transcription_methodology.md)
- [System Architecture](architecture_v3.md)
- [System Requirements](requirements_v3.md)
- [Data model](schema_v3.md)
- Error Handling Policy (this document)
- [Implementation Plan](implementation_plan_v3.md)
docs/ver3/implementation_plan_v3.md
-81
View File
@@ -1,81 +0,0 @@
# Implementation Plan (Version 3)
## Goal
Replace the current v2 SQLModel schema with the approved v3 schema and make sure every database operation works through the existing async SQLAlchemy/SQLModel session layer.
Use a fresh database. There will be no migrations, data conversion, legacy compatibility shims, or parallel v2/v3 code paths.
## Current Project Impact
* `src/transcription/db/models.py` defines the SQLModel tables. It must be updated to match the approved v3 schema (`Document`, `Person`, `DocumentPerson`, `Source`, `Job`, `JobSource`).
* The v3 target adds frozen submission-time prompt snapshot fields (`prompt_name`, `prompt_hash`, `system_prompt`, `user_prompt`, `temperature`, `top_p`) to `Job` and full output payloads (`raw_api_response`, `ai_metadata`) to `JobSource`.
* The v3 target adds image asset verification fields (`file_hash`, `file_size_bytes`) to `Source`.
* Database operations must utilize `JSONBCompat` and the existing SQLModel/SQLAlchemy abstractions to preserve the same logical schema and JSON behavior across the supported backends, while keeping PostgreSQL as the intended production database.
* Async CRUD lives in `DocumentService`, `JobService`, `TranscriptionService`, and upload helpers. Their queries and relationship loading must be updated for v3 fields.
* Canonical operator tooling must remain OS-independent; safety workflows such as destructive-test backup and restore should run through Python or other cross-platform entry points rather than platform-specific shells.
## Implementation
### 1. Update the Schema and Domain Models
* Replace the models in `src/transcription/db/models.py` with the approved v3 tables, enums, relationships, foreign keys, constraints, and indexes.
* Ensure all JSON fields use `JSONBCompat` for dialect portability across SQLite and PostgreSQL.
* Keep `SQLModel.metadata.create_all()` as the schema bootstrap for fresh databases.
* Delete `_ensure_sqlite_compat_columns()` and all legacy schema patching from `src/transcription/db/operations.py`.
* Keep the Python models and `docs/schema_v3.md` perfectly synchronized.
### 2. Update Data Services and Async Worker Layer
* Update job creation and worker orchestration so prompt configuration is resolved at submission and frozen onto `Job` (`prompt_name`, `prompt_hash`, `system_prompt`, `user_prompt`, `temperature`, `top_p`) before execution starts.
* Update `TranscriptionService` and provider adapters to store the complete unedited API REST response dictionary into `job_source.raw_api_response` alongside operational metrics in `job_source.ai_metadata`.
* Update upload handlers to calculate and store file metadata (`file_hash` via SHA-256, `file_size_bytes`) on `Source` records during file ingestion.
* Remove legacy single-source compatibility flows so worker paths persist per-page outcomes only through `JobSource` updates.
### 3. Update Integration Tests and Mock AI Providers
* Update mock provider fixtures in test suites to return realistic complete API response envelopes.
* Verify test coverage for `JSONBCompat` field writes and reads under SQLite in-memory test databases.
* Add assertions in async workflow tests to verify frozen prompt snapshot fields on `Job`, plus per-page failure isolation and output evidence on `JobSource`.
### 4. Update the UI for the v3 Schema
* Review the UI components and views displaying document, job, person, and source data so they reference v3 schema properties instead of v2 relationships.
* Ensure the UI correctly renders `COALESCE(revised_text, raw_transcription)` for page viewing and inline editing.
* Ensure resubmit actions only queue failed pages and preserve frozen prompt snapshot behavior on the existing `Job`.
* Consider the guidance in `docs/ui_style_guide.md` when making UI changes so updated views remain consistent with the project’s visual conventions.
### 5. Keep Operational Tooling Portable
* Implement destructive-test backup and restore workflows in Python so the canonical path runs on Windows, Linux, and macOS.
* Avoid making core developer or recovery procedures depend on PowerShell-only or shell-specific semantics.
* Keep operational documentation aligned with the cross-platform command path used by the repository.
## Done When
* A fresh database is created directly from the v3 SQLModel metadata.
* Frozen prompt input provenance is captured on `Job` for each submission, and full per-page output evidence is captured on `JobSource` for every AI execution task.
* The focused tests and full test suite pass on both SQLite and PostgreSQL backends.
* Canonical operator workflows required for development and destructive-test recovery run without a Windows-only shell dependency.
## Out of Scope
* Database migrations or preservation of v2 data
* Legacy compatibility code
* UI redesign, batch orchestration, worker concurrency, deployment, and operational runbooks
---
## Related Local References
- [System Overview](index_v3.md)
- [System Design Intent](invariant/intent.md)
- [Transcription Methodology](invariant/transcription_methodology.md)
- [System Architecture](architecture_v3.md)
- [System Requirements](requirements_v3.md)
- [Data model](schema_v3.md)
- [Error Handling Policy](error_handling_v3.md)
- Implementation Plan (this document)
docs/ver3/index_v3.md
-49
View File
@@ -1,49 +0,0 @@
# Document Transcription System Overview (Version 3)
This project is a personal-scale application for transcribing, indexing, and preserving historical family documents, letters, postcards, and journals.
## Start Here
Read [architecture_v3.md](https://www.google.com/search?q=architecture_v3.md) first for technical overview and system design.
## Core V3 Capabilities
* **Folder & Multi-Image Ingestion:** Upload one or more images that map sequentially (`page_number`) under a single `Document`.
* **Parallel Async AI Vision Engine:** Concurrently process single-page image transcriptions using Python `asyncio` bounded by rate limiters.
* **Portable Relational Storage:** SQLModel and SQLAlchemy preserve a portable relational model across the supported backends, with SQLite for local development/testing and PostgreSQL as the production database target.
* **Cross-Platform Operations:** Canonical developer and recovery workflows run through Python-based, OS-independent tooling rather than platform-specific shell scripts.
* **Complete Auditability & Provenance:** Capture frozen submission-time input prompts (`system_prompt`, `user_prompt`) and hyperparameters (`temperature`, `top_p`) on `Job`, plus per-page operational metrics (`ai_metadata`) and full provider response envelopes (`raw_api_response`) on `JobSource`.
* **Asset Integrity Tracking:** Calculate and store cryptographic hashes (SHA-256) and file sizes on `Source` image records while preserving clean filesystem storage.
* **Pydantic V2 Validation:** End-to-end type safety, DB row mapping, and JSON payload validation.
* **Historical Person Management:** Track authors and recipients across documents with rich biographical entities (`Person`).
* **Page-Level Execution Auditing & Revisions:** Store immutable point-in-time machine output per run while enabling inline human corrections (`revised_text`).
* **Partial Failure Recovery:** Bounded batch execution that isolates single-page API errors (`partial_success`) for simple retries.
## Technical Stack
* **Application Web Framework:** FastAPI + NiceGUI
* **Persistence Engine:** SQLModel / SQLAlchemy (SQLite for development/testing, PostgreSQL for production)
* **Data Validation & Schemas:** Pydantic V2
* **Concurrency & Workers:** Python `asyncio` worker pool with `asyncio.Semaphore`
* **Vision Providers:** OpenAI, Anthropic, and OpenRouter Vision models via native SDK adapters
---
## Technology References
* [FastAPI documentation](https://fastapi.tiangolo.com/)
* [NiceGUI documentation](https://nicegui.io/documentation)
* [SQLModel documentation](https://sqlmodel.tiangolo.com/)
* [Python asyncio](https://www.google.com/search?q=https://docs.python.org/3/library/asyncio.html%23module-asyncio)
* [Pydantic Validation](https://pydantic.dev/docs/validation/latest/get-started/)
## Documentation Index
- System Overview (this document)
- [System Design Intent](invariant/intent.md)
- [Transcription Methodology](invariant/transcription_methodology.md)
- [System Architecture](architecture_v3.md)
- [System Requirements](requirements_v3.md)
- [Data model](schema_v3.md)
- [Error Handling Policy](error_handling_v3.md)
- [Implementation Plan](implementation_plan_v3.md)
docs/ver3/requirements_v3.md
-45
View File
@@ -1,45 +0,0 @@
# Document Transcription System Requirements (Version 3)
This document captures the **Version 3 baseline requirements** for the production implementation.
## Requirements Model
| ID | Category | Requirement | Verify Method |
| --- | --- | --- | --- |
| REQ-0 | System | Provide end-to-end multi-page document transcription with persistent, inspectable async job states. | demonstration |
| REQ-1 | Functional | Allow users to upload multi-image batches as sequential `Source` pages under a `Document`. | test |
| REQ-2 | Functional | Process multi-page jobs asynchronously using an `asyncio` worker pool bounded by rate limits. | test |
| REQ-3 | Functional | Persist frozen submission-time execution parameters and full input prompts (`system_prompt`, `user_prompt`, `prompt_name`, `prompt_hash`, `temperature`, `top_p`) on `Job`, and persist page-level output responses (`raw_transcription`, `ai_metadata`, `raw_api_response`) on `JobSource`. | test |
| REQ-4 | Functional | Support job states (`queued`, `processing`, `completed`, `partial_success`, `failed`) and page states (`pending`, `transcribed`, `failed`). | inspection |
| REQ-5 | Functional | Allow users to manage historical `Person` records and link multiple authors/recipients to a `Document` via `DocumentPerson`. | test |
| REQ-6 | Functional | Maintain immutable original machine output on `Source.raw_transcription` while permitting inline human edits on `Source.revised_text`. | test |
| REQ-7 | Data Constraint | Use SQLModel/SQLAlchemy to preserve a portable relational domain model and compatible data shape across the supported backends, with SQLite for local development/testing and PostgreSQL as the production system of record. | inspection |
| REQ-8 | Data Constraint | Validate all API requests, database rows, and JSON structures using Pydantic V2 schemas and SQLModel. | test |
| REQ-9 | Interface | Render multi-page transcriptions sequentially by `page_number` in the web UI with author/recipient metadata. | demonstration |
| REQ-10 | Operations | Allow operators to resubmit only failed pages for queued reprocessing while preserving the frozen prompt snapshot on the existing `Job`. | test |
| REQ-11 | Data Constraint | Calculate and store cryptographic file hashes (SHA-256) and file sizes for uploaded source images to track asset integrity. | test |
| REQ-12 | Operations Constraint | Keep core development, testing, restore, and recovery workflows OS-independent across Windows, Linux, and macOS; do not require a platform-specific shell for canonical project processes. | inspection |
## Element Satisfaction Mapping
* **UI (NiceGUI):** Satisfies REQ-1, REQ-5, REQ-6, REQ-9, REQ-10.
* **API (FastAPI):** Satisfies REQ-1, REQ-4, REQ-5, REQ-8.
* **WORKER (asyncio):** Satisfies REQ-2, REQ-3, REQ-4, REQ-10.
* **PERSISTENCE (SQLModel/SQLAlchemy):** Satisfies REQ-3, REQ-6, REQ-7, REQ-11.
* **MODELS (Pydantic V2 / SQLModel):** Satisfies REQ-8.
* **OPERATIONS TOOLING (Python / OS-neutral automation):** Satisfies REQ-12.
---
## Related Local References
- [System Overview](index_v3.md)
- [System Design Intent](invariant/intent.md)
- [Transcription Methodology](invariant/transcription_methodology.md)
- [System Architecture](architecture_v3.md)
- System Requirements (this document)
- [Data model](schema_v3.md)
- [Error Handling Policy](error_handling_v3.md)
- [Implementation Plan](implementation_plan_v3.md)
docs/ver3/schema_v3.md
-137
View File
@@ -1,137 +0,0 @@
# Database Schema (Version 3)
This document describes the relational schema for the transcription platform. It incorporates multi-image batch orchestration, page-level execution tracking, many-to-many author/recipient attribution, submission-time prompt snapshot capture, and raw API payload evidence for archival auditing.
The schema uses generic JSON columns compatible with SQLite in local development and PostgreSQL native JSONB/UUID types in production.
## Entity Relationship Diagram
```mermaid
erDiagram
PERSON {
UUID id PK
TEXT full_name
TEXT display_name
TEXT maiden_name
DATE birth_date
TEXT birth_date_raw
TEXT birth_place
DATE death_date
TEXT death_date_raw
TEXT death_place
TEXT biography
TEXT portrait_path
JSONB metadata
TIMESTAMPTZ created_at
TIMESTAMPTZ updated_at
}
DOCUMENT {
UUID id PK
TEXT name
TEXT document_type
DATE document_date
TEXT document_date_raw
TEXT location_created
TEXT notes
TEXT archive_identifier
TIMESTAMPTZ created_at
TIMESTAMPTZ updated_at
}
DOCUMENT_PERSON {
UUID id PK
UUID document_id FK
UUID person_id FK
VARCHAR role "author | recipient"
TIMESTAMPTZ created_at
}
JOB {
UUID id PK
UUID document_id FK
VARCHAR status "queued | processing | transcribed | completed | partial_success | failed"
INTEGER retry_count
TEXT provider
TEXT model
TEXT prompt_name
TEXT prompt_hash
TEXT system_prompt
TEXT user_prompt
FLOAT temperature
FLOAT top_p
TIMESTAMPTZ date_created
TIMESTAMPTZ date_updated
}
SOURCE {
UUID id PK
UUID document_id FK
INTEGER page_number
TEXT upload_name
TEXT filename
TEXT file_path
TEXT file_hash
BIGINT file_size_bytes
TEXT raw_transcription
TEXT revised_text
TIMESTAMPTZ date_uploaded
TIMESTAMPTZ date_revised
}
JOB_SOURCE {
UUID id PK
UUID job_id FK
UUID source_id FK
VARCHAR status "pending | transcribed | failed"
TEXT raw_transcription
JSONB ai_metadata
JSONB raw_api_response
TEXT error_detail
TIMESTAMPTZ executed_at
}
DOCUMENT ||--o{ DOCUMENT_PERSON : "has_people"
PERSON ||--o{ DOCUMENT_PERSON : "participates_in"
DOCUMENT ||--o{ JOB : "has_jobs"
DOCUMENT ||--o{ SOURCE : "contains_pages"
JOB ||--o{ JOB_SOURCE : "executes"
SOURCE ||--o{ JOB_SOURCE : "processed_in"
```
## Domain Invariants & Provenance Rules
### Page-Level Execution & AI Outputs
* **Execution Granularity:** Every single image execution attempt by an AI model produces a dedicated record in `job_source`.
* **Submission Snapshot Provenance:** Every `job` captures the frozen prompt identifier details (`prompt_name`, `prompt_hash`), full prompt text strings (`system_prompt`, `user_prompt`), and hyperparameters (`temperature`, `top_p`) at submission time.
* **Point-in-Time Output Auditability:** `job_source.raw_api_response` stores the complete, unedited provider REST response envelope for that specific image page call. `job_source.ai_metadata` stores spatial bounding boxes, normalized token usage, latency, and cost details for fast querying.
* **Active Output Caching:** Upon successful completion of an image call, `source.raw_transcription` is updated with the latest output string from `job_source.raw_transcription` for fast UI rendering.
### Image Storage & Integrity
* **Filesystem Storage:** Binary images are stored on disk in the local file system. The `source` table holds the relative `file_path`.
* **File Integrity Tracking:** `source` captures `file_hash` (SHA-256) and `file_size_bytes` at upload time to guarantee document file integrity and duplicate checking over long-term preservation.
### Page Ordering & Revisions
* **Sequential Integrity:** `source.page_number` dictates page ordering within a document. Reads assembling full documents must query `ORDER BY source.document_id, source.page_number ASC`.
* **Inlined Human Corrections:** User edits occur at the page level inside `source.revised_text`. `source.raw_transcription` remains immutable. If `source.revised_text` is non-null, application frontends must render `source.revised_text`.
### Async Job Lifecycle & Failure Isolation
* **Batch Orchestrator:** A job represents an overarching execution run across one or more source images belonging to a document.
* **Isolated Failures:** API requests run concurrently (e.g., using `asyncio`). A failure on page 3 does not invalidate successful transcriptions on page 1 or 2.
* **Job States:**
* `queued`: Created, awaiting worker execution.
* `processing`: Concurrent HTTP tasks actively running.
* `completed`: 100% of linked `job_source` tasks succeeded (`transcribed`).
* `partial_success`: At least one `job_source` succeeded and at least one failed.
* `failed`: All linked `job_source` tasks failed or a job-level runtime error occurred.
### Attribution & Person Roles
* **Multi-Person Roles:** Documents support zero, one, or many authors and recipients linked via `document_person`.
* **Role Uniqueness:** `(document_id, person_id, role)` must be unique to prevent duplicate role tagging.
docs/ver4.2/implementation_plan_v4_2.md
+205 -101
View File
@@ -1,137 +1,241 @@
# Draft Implementation Plan (Version 4.2) # Implementation Plan (Version 4.2)
## Goal ## Goal
Prepare a safe implementation path for Source page reordering and constrained application settings. This plan remains provisional until the V4.2 scope-freeze decisions are resolved. Make processing evidence precise, append-only, secret-safe, and exportable while preserving every existing record and creating a provider-neutral home for future OCR/layout artifacts.
## Planning Constraints ## Implementation Principles
- V4 and V4.1 remain the behavioral baseline. - Implement the [Digital Evidence and AI Processing Provenance invariant](../invariant/ai_evidence_and_provenance.md), not a provider-specific approximation of it.
- Reordering must be atomic and service-owned. - Capture transport evidence before SDK parsing.
- Settings must use explicit domain operations rather than direct database, environment-file, or arbitrary filesystem access from UI pages. - Keep exact evidence separate from parsed and normalized representations.
- Prompt changes must preserve historical Job provenance and use a defined safe-write policy. - Prefer additive schema evolution and explicit compatibility behavior.
- Reference source content by digest rather than duplicating it in request JSON.
- Use allowlists for safe metadata capture.
- Keep persistence and evidence semantics behind service boundaries.
- Do not change the default model until a representative benchmark supports that decision.
## Current-State Gaps
| Current behavior | Gap to close |
| --- | --- |
| `Source` stores original file path, digest, and size. | Media type and image/page geometry used for an execution are not frozen with that execution. |
| `Job` stores prompt text/hash, requested model after resolution, temperature, and `top_p`. | The complete effective request structure, omitted-versus-explicit parameter state, routing constraints, and software versions are not frozen. |
| `JobSource.raw_api_response` stores `model_dump()` output from the OpenRouter SDK. | The exact HTTP body can be normalized by OpenRouter and filtered again by the SDK before persistence. |
| `JobSource.ai_metadata` stores finish reason and basic token counts. | Detailed accounting remains only in the SDK snapshot and is not a substitute for exact evidence. |
| Provider exceptions become application errors. | Safe HTTP error bodies, statuses, headers, and no-response distinctions are not persisted. |
| Worker logs elapsed time. | Execution duration is not stored on `JobSource`. |
| Source Detail displays AI metadata and the SDK snapshot. | The UI does not identify evidence layers or expose request/transport/software provenance. |
| No generic processing-artifact model exists. | Future OCR geometry would require ad hoc provider fields or an unrelated schema. |
## Expected Project Impact ## Expected Project Impact
| Area | Expected impact | | Area | Expected impact |
| --- | --- | | --- | --- |
| Sources service | Add validated, transactional set-based page reordering. | | Database models and upgrades | Add execution-specification, transport-evidence, timing, software-context, and generic artifact storage without removing existing columns. |
| Documents/Sources UI | Add a reorder entry point and interaction for one Document. | | OpenRouter adapter | Introduce a transport boundary that can capture exact body/status/safe headers before typed SDK parsing, or use supported SDK hooks that expose the unparsed response reliably. |
| Documents service | Expand controlled Document Type maintenance operations. | | Provider contract | Return structured evidence for success and failure without leaking provider-specific transport concerns into workflow orchestration. |
| People service | Expand controlled Person Role maintenance operations. | | Source and workflow services | Persist one append-only execution outcome and its artifacts transactionally; retain compatibility projections. |
| Prompt adapter/service | Add constrained listing, reading, validation, and safe writing of prompt artifacts. | | UI | Label and inspect evidence layers; export safe evidence packages through service operations. |
| UI composition/navigation | Register Settings routes and navigation without moving persistence into UI code. | | Benchmarking | Add a private manifest and repeatable evaluator using the literal-transcription methodology. |
| Tests | Add transaction, conflict, registry lifecycle, prompt safety, and UI workflow coverage. | | Tests and documentation | Add compatibility, capture, security, integrity, export, and benchmark-scoring coverage; correct overstated V4 evidence language. |
## Proposed Implementation Phases ## Proposed Data Design
### 1. Resolve Scope-Freeze Decisions Exact names should be confirmed against existing conventions before migration code is written. The design should provide the following logical records.
- Select and document the reorder interaction. ### 1. Execution Evidence
- Define whether active Jobs block reorder.
- Define prompt atomic-write, backup, and recovery policy.
- Decide whether prompt creation/deletion is excluded.
- Finalize registry ordering requirements.
- Remove the Draft designation only after these decisions are reflected in scope and acceptance criteria.
### 2. Define Service Contracts Extend `JobSource` or associate it one-to-one with a new execution-evidence record containing:
- Define a Source reorder command containing `document_id`, the complete ordered Source ID list, and a concurrency token or equivalent stale-write guard if supported by the current model. - Request manifest JSON and manifest schema version.
- Define Document Type maintenance commands for create, relabel, sort, activate, and deactivate. - Transport status, body bytes or exact decoded body plus encoding/content type, and safe headers.
- Define Person Role maintenance commands for create, relabel, activate, and deactivate. - Parsed SDK snapshot retained separately from transport content.
- Define a Prompt Store interface for constrained list/read/write behavior. - Application, adapter, SDK, and runtime version metadata.
- Map validation, conflict, not-found, dependency, and filesystem failures to existing `AppError` categories. - Start, finish, and duration values.
- Router/provider request and generation identifiers when available.
- Failure phase and whether an HTTP response was received.
### 3. Implement Transactional Source Reordering The implementation should evaluate a companion table rather than continuing to widen `JobSource`. A companion record better isolates large/optional evidence and permits clear one-to-one compatibility semantics.
- Load all Sources for the target Document in the same transaction. ### 2. Generic Processing Artifact
- Reject missing, extra, duplicate, or foreign Source IDs.
- Reject stale writes using the selected concurrency policy.
- Apply a collision-safe renumbering strategy suitable for both SQLite and PostgreSQL.
- Finish with contiguous `page_number` values beginning at 1.
- Roll back the entire operation on any failure.
- Add service tests for valid reorder, no-op, reverse order, invalid membership, duplicates, stale submissions, rollback, and backend-compatible SQL behavior.
### 4. Implement the Reorder UI Add a one-to-many artifact model associated with a source and, when applicable, a producing execution:
- Add a Reorder Pages action from a Document-scoped Source view or Document Detail. - Stable artifact UUID.
- Render Source labels/previews sufficient to identify each page. - `source_id` and optional execution/`job_source_id`.
- Capture the complete intended order. - Semantic artifact type.
- Require explicit Save and provide Cancel without mutation. - Media/serialization format.
- Surface validation and conflict errors through the shared error presenter. - Schema name and version.
- Return to a Document-scoped ordered view after success. - Producer and producer version.
- Verify keyboard-accessible controls for any drag-and-drop interaction. - Inline JSON payload or external location.
- Payload digest and byte size.
- Coordinate-system metadata when relevant.
- Creation timestamp.
### 5. Expand Registry Maintenance Services Enforce exactly one content location: inline payload or external reference. An external artifact must be written durably and hashed before its database record commits.
- Reuse existing Document and People service ownership. ### 3. Compatibility Projections
- Add explicit write methods rather than passing UI-mutated ORM objects directly where practical.
- Normalize and validate new stable codes.
- Reject duplicate codes deterministically.
- Block deletion or omit deletion entirely; use activation state for lifecycle management.
- Preserve inactive entries for historical reads.
- Add service tests for create, relabel, activation, deactivation, duplicates, immutable codes, and referenced records.
### 6. Add Constrained Prompt Storage - Keep `JobSource.raw_api_response` unchanged for existing and new compatibility reads until a later deprecation decision.
- Keep `JobSource.ai_metadata` for indexed/display-ready normalized values.
- Keep `Source.raw_transcription` as the latest successful machine-output projection while treating per-execution `JobSource.raw_transcription` as history.
- Document that older rows have an SDK snapshot but no exact transport capture.
- Place filesystem access behind a dedicated Prompt Store/service boundary. ## Implementation Phases
- Resolve all filenames directly beneath the configured prompt root and reject traversal.
- Permit only the agreed markdown extension and reject empty content.
- Implement the approved safe-write strategy, including flush/replace behavior and backup/recovery if selected.
- Preserve file encoding and provide explicit failures for read-only or unavailable storage.
- Do not modify any Job row when prompt defaults change.
- Add unit tests for valid reads/writes, traversal, invalid names, empty content, filesystem failures, and unchanged Job provenance.
### 7. Build the Settings UI ### 1. Correct Terminology and Define Typed Contracts
- Register a Settings landing page and navigation entry. - Add typed domain models for request manifests, software context, transport metadata, failure phase, and artifact descriptors.
- Add separate pages or panels for Document Types, Person Roles, and Prompts. - Version every persisted JSON contract from its first release.
- Keep pages responsible for orchestration and notifications only. - Define the safe response-header allowlist. Begin with correlation, content type/encoding, date, retry/rate-limit, and router-specific generation identifiers only when documented and non-secret.
- Use service callbacks for all mutations. - Define size limits and external-storage thresholds for exact bodies and artifacts.
- Explain stable codes, inactive historical entries, and future-only prompt effects in the UI. - Correct `docs/ver4/schema_v4.md` under “Page-Level Execution and AI Outputs” so the existing column is described as an SDK-serialized OpenRouter response snapshot, not a complete provider envelope, exact HTTP body, or native upstream-provider response. Apply the same terminology to architecture and UI schema references.
- Do not render raw environment values or secrets. - Add serialization and secret-rejection unit tests before provider changes.
### 8. Verification and Rollout ### 2. Add Additive Persistence and Upgrade Behavior
- Run focused service tests before UI integration tests. - Add the selected execution-evidence and artifact models.
- Verify reorder behavior against Documents with one and many Sources. - Add foreign keys, uniqueness constraints, and indexes for source/execution lookup.
- Verify ordered transcription rendering and V4.1 previous/next navigation after reorder. - Implement idempotent upgrades following the repository's existing schema-upgrade policy.
- Verify inactive registry behavior in both historical display and create/edit selectors. - Do not populate exact response fields for historical rows.
- Verify prompt changes are picked up by newly created Jobs while historical Jobs retain frozen content/hash. - Do not write a capture-time classification onto historical rows during migration. Compatibility reads may describe a populated legacy `raw_api_response` as an SDK snapshot, but exports must identify that description as a later compatibility interpretation rather than execution-time metadata.
- Run the relevant regression suite. - Verify JSON portability and large-payload behavior for SQLite and PostgreSQL.
- Add upgrade tests starting from a representative pre-V4.2 schema.
## Migration and Compatibility Notes ### 3. Build Secret-Safe Request Manifests
- No new table is expected solely for reordering; `Source.page_number` remains authoritative. - Build the manifest from the concrete outgoing request body immediately before transport, not from a narrower typed projection that may discard unrecognized request fields.
- A uniqueness constraint on `(document_id, page_number)` should be evaluated before scope freeze. If added, migration and collision-safe update behavior must be designed for both supported databases. - Replace each image payload in that concrete representation with a source reference containing source UUID, digest, byte size, media type, dimensions, and transformation identity.
- Existing registry records remain valid. - Store exact prompt content and preserve omitted-versus-explicit parameter state.
- Prompt editing changes mutable application files, not database provenance already captured on Jobs. - Include requested model, routing preferences, response-format requirements, and timeout/retry policy.
- V4.2 must not require users to recreate existing Sources, Documents, People, roles, or types. - Record application version/commit when available, adapter contract version, SDK package/version, and request-manifest schema version.
- Hash the canonical manifest representation for integrity checks.
- Test that credentials and embedded image data cannot enter the persisted manifest.
- Test that every field actually sent to the provider, including routing and future provider options, is represented or explicitly excluded by the manifest transform.
## Proposed Delivery Order ### 4. Capture OpenRouter Transport Evidence
1. Freeze the remaining decisions. - Evaluate the installed OpenRouter SDK hooks/client injection first.
2. Implement and verify Source reorder service semantics. - If hooks cannot expose an exact stable response before typed parsing, implement the non-streaming OpenRouter call through the existing async HTTP client boundary while retaining typed validation in the adapter.
3. Build the reorder UI. - Read the response body once, preserve it exactly, then parse and normalize it.
4. Implement registry maintenance service operations. - Store status, content type/encoding, allowlisted headers, request/generation ID, and timing.
5. Implement the Prompt Store and safety policy. - Maintain current authentication, referer/title headers, timeout behavior, and error classification.
6. Build Settings pages. - Explicitly document that the captured body is the OpenRouter-normalized transport response, not Gemini/Anthropic/OpenAI native upstream JSON.
7. Run integration and regression verification. - Add fixture-based tests proving unknown response fields survive transport capture even if a typed parser ignores them.
## Draft Done Criteria ### 5. Preserve Failure Evidence
- All V4.2 acceptance criteria are testable and satisfied. - Return or raise a typed provider failure that carries safe evidence separately from its user-facing error.
- Reordering is atomic, conflict-aware, contiguous, and cross-database compatible. - Persist non-success status/body/allowlisted headers before marking an execution failed.
- Settings mutations cross explicit service or adapter boundaries. - Represent DNS/connect/TLS/local timeout failures as no-response outcomes with a failure phase and safe diagnostic category.
- Registry codes cannot be accidentally changed. - Preserve response-validation failures with both the exact body and validation details.
- Prompt writes cannot escape the configured directory or rewrite historical provenance. - Keep transcription-quality rejection distinct from provider failure because a valid provider response was received.
- No secret or raw environment editor exists. - Ensure error strings and logs do not contain authorization data or embedded image payloads.
- V4.1 workflows remain intact. - Add tests for 4xx, 5xx, malformed JSON, schema mismatch, timeout, connection failure, and quality rejection.
### 6. Make Execution History Reliably Append-Only
- Confirm retry behavior creates a distinct execution attempt rather than reusing and overwriting a completed evidence record.
- Separate queue linkage from execution-attempt identity; the current update-in-place behavior cannot serve as append-only execution history.
- Assign each attempt a deterministic, monotonically increasing attempt number scoped to its Job and Source, enforced by a database uniqueness constraint.
- Update the latest-transcription projection only after a successful attempt.
- Never update prior response bodies, manifests, timings, or artifacts during a retry.
- Select the latest attempt and latest successful attempt by the persisted attempt number with a stable identifier as a defensive secondary key, never by timestamp alone.
- Add service/workflow tests covering retries, partial success, interrupted jobs, and historical projection behavior.
### 7. Add Generic Artifact Persistence
- Implement service operations to create, read, list, verify, export, and, only under explicit retention policy, delete artifacts.
- Validate semantic type, schema/version, digest, media type, and coordinate metadata.
- Support JSON artifacts inline initially when within the agreed size threshold.
- Support external artifacts through a constrained application-data root with atomic write, digest verification, and explicit missing-file errors.
- Add a provider-neutral example fixture representing OCR words/lines with polygons and confidence values.
- Do not integrate a live OCR vendor in this phase.
### 8. Add Evidence Inspection and Export
- Rename the current Source Detail label to identify historical values as an OpenRouter SDK Response Snapshot.
- Add separate sections for Request Manifest, Transport Response, Normalized Metadata, Software Context, and Derived Artifacts.
- Show an explicit “not captured for this historical execution” state instead of an empty object.
- Keep large bodies collapsed by default and avoid rendering embedded source data.
- Add a service-owned export that packages a versioned manifest, evidence JSON/body files, artifact content or references, and digest inventory.
- Exclude secrets and machine-local paths that are not required to interpret the evidence.
- Add UI and export tests for new, historical, failed, and large-evidence records.
### 9. Establish the Private Benchmark
- Select a small initial corpus, then expand only when it exposes meaningful differences.
- Stratify examples by printed/typed text, handwriting style, degradation, layout complexity, language, and editorial anomaly.
- Reference existing Source UUIDs and digests in a private manifest; do not copy family documents into public test fixtures.
- Create manually reviewed reference transcriptions following the invariant methodology.
- Implement or adopt existing project-compatible CER/WER calculations without changing dependencies unless justified.
- Score omissions, inventions, silent modernization, uncertainty markup, and layout fidelity separately from CER/WER.
- Record cost and latency from preserved execution evidence.
- Run the current `google/gemini-2.5-flash` configuration as the baseline before testing alternatives.
- Treat results as model-version/route/corpus specific and preserve each comparison run.
### 10. Verify, Migrate, and Align Documentation
- Run the smallest focused model, provider, service, workflow, UI, upgrade, and export test groups first.
- Run broader regression tests only after focused validation passes.
- Execute all destructive tests through `tools/run_destructive_tests.py`.
- Verify backup creation and required restoration behavior before any test touching real application data.
- Confirm existing Source Detail records remain readable after upgrade.
- Update V4 architecture, schema, requirements, and UI schema mappings to point to V4.2 semantics.
- Record any deliberate deviation from this plan in the V4.2 scope before release.
## Recommended Delivery Order
1. Typed/versioned evidence contracts and terminology.
2. Additive execution-evidence persistence.
3. Secret-safe request manifests.
4. Exact OpenRouter transport capture.
5. Failure evidence and append-only retry semantics.
6. Generic artifact persistence.
7. Inspection and export.
8. Private benchmark tooling and baseline run.
9. Migration, regression verification, and documentation alignment.
## Key Implementation Decisions to Resolve
1. Whether execution evidence is a one-to-one companion to `JobSource` or part of a new execution-attempt model required for append-only retries.
2. Whether exact response bodies remain database values at expected sizes or move to hashed external files above a threshold.
3. The canonical JSON algorithm used to hash request manifests.
4. The safe-header allowlist supported by OpenRouter and future adapters.
5. The application version identity available in local, packaged, and uncommitted development builds.
6. The initial inline/external artifact size threshold and application-data root.
7. Whether evidence exports include original source binaries by default, optionally, or only by reference.
8. The minimum private benchmark corpus size and review process before model comparisons influence defaults.
These decisions must be settled before their corresponding implementation phase; they do not weaken the invariant or expand V4.2 into live OCR integration.
## Done When
- Every V4.2 acceptance criterion is satisfied by focused tests or an explicit demonstration.
- Existing SDK snapshots retain their content and are labeled accurately.
- New successful and failed calls preserve secret-safe provider-boundary evidence.
- Unknown transport fields survive even when the typed SDK/parser does not recognize them.
- Retries cannot overwrite prior execution evidence.
- A generic versioned artifact can represent OCR geometry and pass integrity verification.
- Evidence can be safely inspected and exported with schema identities and digests.
- The current model has a reproducible private benchmark baseline.
- No credential or embedded source payload appears in persisted manifests, safe headers, logs, or exports.
- Existing V4.1 behavior remains compatible.
## Out of Scope
- Live OCR/document-AI provider integration.
- Automatic model switching.
- Archive-wide reprocessing.
- Native upstream-provider response capture through OpenRouter when OpenRouter does not expose it.
- Guarantees of deterministic hosted-model output.
## Related Local References ## Related Local References
- [Draft V4.2 Scope Boundary](scope_boundary_v4_2.md) - [V4.2 Scope Boundary](scope_boundary_v4_2.md)
- [V4.1 Implementation Plan](../ver4.1/implementation_plan_v4_1.md) - [Digital Evidence and AI Processing Provenance](../invariant/ai_evidence_and_provenance.md)
- [V4 Implementation Plan](../ver4/implementation_plan_v4.md) - [V4 Architecture](../ver4/architecture_v4.md)
- [V4 Error Handling Policy](../ver4/error_handling_v4.md) - [V4 Schema](../ver4/schema_v4.md)
- [V4 Requirements](../ver4/requirements_v4.md)
- [Draft V4.3 Implementation Plan](../ver4.3/implementation_plan_v4_3.md)
docs/ver4.2/scope_boundary_v4_2.md
+125 -92
View File
@@ -1,130 +1,163 @@
# Draft V4.2 Scope Boundary # V4.2 Scope Boundary
This document defines the proposed boundary for the second incremental revision to Version 4. It is intentionally a draft until V4.1 has been used and the remaining workflows have been validated. This document defines the boundary for the digital-evidence and AI-provenance revision that follows V4.1 and precedes the planned V4.3 page-reordering and settings work. V4 remains the architecture baseline; V4.2 makes the existing evidence claims precise and adds a provider-neutral foundation for future processing artifacts.
## Purpose ## Purpose
- Allow correction of Source page order after import. - Align the application with the [Digital Evidence and AI Processing Provenance invariant](../invariant/ai_evidence_and_provenance.md).
- Provide a constrained Settings area for safe maintenance of selected application-managed configuration. - Preserve provider-boundary evidence before SDK parsing can remove unknown fields.
- Avoid exposing secrets, restart-sensitive settings, or unrestricted filesystem editing through the UI. - Make successful and failed processing attempts inspectable without storing secrets.
- Support future OCR and layout outputs without coupling the database to one vendor.
- Establish a repeatable method for comparing transcription models against this archive.
## Proposed In Scope ## In Scope
### 1. Source Page Reordering ### 1. Evidence Terminology and Existing-Data Compatibility
- Allow Sources within one Document to be reordered after import. - Define transport response, router-normalized response, SDK response, normalized metadata, and derived artifact consistently in code, schema documentation, and UI labels.
- Present the current order using page number and a recognizable source label or preview. - Treat existing `JobSource.raw_api_response` values as historical SDK response snapshots.
- Persist the complete intended order atomically. - Preserve every existing `Job`, `Source`, and `JobSource` row.
- Renumber the affected Document's Sources to a contiguous sequence beginning at 1. - Use additive migrations and compatibility reads; do not reinterpret previously stored values as exact transport captures.
- Keep all Sources attached to their existing Document. - Correct the “Page-Level Execution and AI Outputs” rule in `docs/ver4/schema_v4.md` that currently describes `JOB_SOURCE` as storing a complete provider response envelope. The corrected rule must identify `raw_api_response` as an SDK-serialized OpenRouter response snapshot and state that it is neither the exact HTTP body nor the native upstream-provider response.
- Ensure transcription rendering and previous/next navigation use the updated order.
- Detect stale or invalid reorder submissions and fail without partial mutation.
### 2. Settings Navigation ### 2. Secret-Safe Request Manifests
- Add a Settings entry to application navigation. - Persist the effective request specification for each page execution without storing credentials or duplicate base64 media.
- Provide separate, clearly described settings areas rather than a raw configuration editor. - Include requested provider/model, routing constraints, prompt content and hash, explicitly supplied parameters, source digest, media type, dimensions when known, and page identity.
- Restrict V4.2 settings to application-managed values that can be validated and safely changed at runtime. - Distinguish an omitted optional parameter from an explicitly supplied null or value.
- Record application, provider-adapter, Python client, and relevant schema versions.
- Use source or derivative references in place of embedded media bytes.
### 3. Document Type Maintenance ### 3. Provider-Boundary Response Capture
- List active and inactive Document Types. - Capture the exact HTTP response body before OpenRouter SDK parsing for non-streaming transcription calls.
- Add new types with a stable unique code and user-facing label. - Store HTTP status and an explicit allowlist of safe response headers.
- Edit mutable labels and sort order. - Store router request/generation identifiers and resolved model/provider-routing metadata when exposed.
- Activate or deactivate types without invalidating historical Documents. - Preserve the current parsed SDK snapshot and normalized metadata where useful.
- Do not allow changing a stable code after creation. - Keep exact body, parsed representation, and normalized fields distinguishable.
- Do not delete types that are referenced by Documents.
### 4. Person Role Maintenance ### 4. Failure Evidence and Timing
- List active and inactive Person Roles. - Create or update a page execution record for every attempted provider call.
- Add new roles with a stable unique code and user-facing label. - Persist safe response evidence for non-success HTTP responses.
- Edit mutable labels. - Distinguish HTTP response failures, connection failures, local timeouts, response-validation failures, and transcription-quality failures.
- Activate or deactivate roles without invalidating historical links. - Store execution start/end times or duration using a clearly defined clock policy.
- Do not allow changing a stable code after creation. - Do not collapse a provider error body into only a generic user-facing message.
- Do not delete roles that are referenced by document-person links.
### 5. Prompt Maintenance ### 5. Generic Processing Artifacts
- List prompt markdown files from the configured prompt directory. - Add a provider-neutral representation for versioned derived artifacts.
- View a prompt with a concise explanation of its purpose and use. - Support inline JSON and externally stored payloads with a digest and stable reference.
- Edit an existing prompt as plain markdown text. - Record artifact type, format, schema/version, producer/version, source, producing execution, and creation time.
- Validate the filename boundary and reject empty prompt content. - Define coordinate-system metadata sufficient for word, line, block, or page geometry.
- Save changes explicitly and report filesystem failures. - Permit future OCR/layout/confidence results without implementing a vendor-specific table for each provider.
- Preserve submission-time prompt text and hash already frozen on existing Jobs.
- Define a safe-write and recovery approach before this feature is considered final scope.
## Proposed Out of Scope ### 6. Evidence Inspection and Export
- Viewing or editing raw `.env` files. - Expand Source Detail and/or Job Detail to identify the evidence layer being displayed.
- Displaying or changing provider API keys and other secrets. - Provide readable JSON inspection for request manifests, transport metadata, parsed responses, normalized metadata, and derived artifacts.
- Editing host, port, database connection, upload paths, or other restart-sensitive runtime settings. - Provide a safe export containing evidence content or references, relationships, schema versions, and digests.
- Arbitrary file browsing or arbitrary prompt paths. - Clearly label evidence that was not captured for historical records.
- Runtime theme/CSS editing. - Do not display or export credentials, unrestricted headers, or embedded base64 source media.
- Installing themes or plugins.
- Source movement between Documents as part of reordering.
- Automatic ordering based on filenames, OCR, or image content.
- FamilySearch API synchronization.
- A generic external-reference registry.
- Ancestry references and Google Maps links.
## Proposed Design Decisions ### 7. Representative-Corpus Benchmark Protocol
### A. Reordering Is Set-Based - Define a private benchmark manifest referencing source digests rather than duplicating archival media.
- Include representative printed, typed, handwritten, degraded, tabular, and spatially complex pages.
- Pair each benchmark item with a manually reviewed literal transcription.
- Score character error rate, word error rate, omissions, inventions, silent normalization, uncertainty handling, layout fidelity, cost, and latency.
- Preserve the complete execution provenance for every benchmark run.
- Keep the current model as a baseline; do not change the application default solely from vendor benchmarks.
- The client submits the full ordered list of Source IDs for one Document. ### 8. Migration, Integrity, and Verification
- The service validates membership, completeness, duplicates, and authorization/context before writing.
- All page-number updates occur in one transaction.
### B. Registry Codes Are Immutable - Provide non-destructive upgrade behavior for supported SQLite and PostgreSQL deployments.
- Backfill only facts that can be derived reliably from existing records.
- Mark unavailable historical evidence as unavailable rather than fabricating it.
- Add digest, serialization, header-allowlist, failure-path, compatibility, artifact, export, and UI inspection tests.
- Run destructive tests only through the repository's required backup-and-restore wrapper.
- Document Type and Person Role codes are stable identifiers. ## Out of Scope
- Labels and active state remain mutable.
- Historical references remain valid when a registry entry is inactive.
### C. No Raw Environment Editor - Selecting or declaring a permanent best transcription model.
- Changing the default transcription model without benchmark evidence and a separate decision.
- Integrating Azure Document Intelligence, Google Document AI, Transkribus, Mistral OCR, or another OCR provider in V4.2.
- Generating bounding boxes retroactively for existing transcriptions.
- Bulk reprocessing the archive.
- Packet capture, TLS evidence, full unrestricted request/response headers, or credential retention.
- Storing duplicate base64 source images in request manifests.
- Guaranteeing byte-identical reproduction from nondeterministic or updated hosted models.
- Automatic entity extraction, biography generation, or genealogical inference.
- Replacing the relational database with an event store or content-addressed object store.
- Destructive renaming or removal of `raw_api_response`.
- `.env` may contain secrets and values that are not safely reloadable. ## Locked Design Decisions
- V4.2 exposes only purpose-built forms backed by explicit validation and service methods.
### D. Prompt Editing Is Constrained ### A. The Original Source Is Primary Evidence
- Prompt maintenance is limited to direct children of the configured prompt directory. - Original uploaded bytes and their digest remain authoritative.
- Existing Job provenance is never rewritten when a prompt file changes. - Processing derivatives and outputs are independently identified derived evidence.
- The UI must distinguish editing the default for future submissions from inspecting historical Job prompts. - Future OCR/layout work reuses the original or a documented derivative.
## Decisions Required Before Scope Freeze ### B. Evidence Is Layered
1. Choose the reorder interaction: move-up/down controls, drag-and-drop, or both. - Exact transport evidence, SDK-parsed objects, normalized metadata, and transcription text serve different purposes.
2. Decide whether reordering is allowed while the Document has a queued or processing Job. - One representation must not silently stand in for another.
3. Define prompt backup, atomic-write, and recovery behavior. - UI and export labels name the stored evidence layer.
4. Decide whether prompt creation and deletion are needed or whether V4.2 edits existing prompts only.
5. Confirm whether registry sort-order maintenance is needed for Person Roles as well as Document Types.
6. Confirm that settings changes remain local to the current installation and do not require an API surface.
## Draft Acceptance Criteria ### C. History Is Append-Only
1. Reordering a Document's Sources produces contiguous page numbers and updates every ordered view consistently. - A retry or reprocessing attempt creates new execution evidence.
2. Invalid, incomplete, duplicate, cross-Document, or stale reorder requests make no changes. - Convenience caches may change, but historical execution output does not.
3. Document Type and Person Role maintenance preserves stable codes and historical references. - Human revisions remain separate from machine output.
4. Inactive registry entries remain visible on historical records but are excluded from default create selectors.
5. Prompt edits are restricted to valid markdown files in the configured prompt directory.
6. A prompt edit affects future Jobs only and leaves stored Job provenance unchanged.
7. No Settings page exposes secrets or unrestricted filesystem access.
8. Focused service and UI tests pass without regressing V4.1 workflows.
## Scope Freeze Gate ### D. Capture Is Secret-Safe by Construction
V4.2 implementation should not begin until: - Safe headers are allowlisted.
- Authorization, cookies, API keys, and unrestricted headers are never persisted.
- Request manifests reference source digests instead of embedding source bytes.
- V4.1 has been used sufficiently to validate priorities. ### E. Derived Artifacts Are Generic and Versioned
- The six open decisions above are resolved.
- The prompt-write safety policy is documented. - Artifact storage is not limited to bounding boxes.
- The implementation plan is revised from draft to committed delivery plan. - Coordinate metadata declares units, origin, dimensions, and transformations.
- Provider-specific payloads may be retained without making provider-specific fields the durable application contract.
### F. Existing Evidence Keeps Its Original Meaning
- Existing `raw_api_response` data remains an SDK response snapshot.
- A migration may label or classify it but may not claim that missing transport data was captured.
- Historical nulls and absent fields remain distinguishable from new explicitly captured values.
## Data and Compatibility Policy
- All schema changes are additive in V4.2.
- Existing source files, hashes, transcriptions, revisions, prompts, jobs, and relationships remain valid.
- Compatibility reads continue to display historical SDK snapshots.
- Large derived artifacts may be stored outside the database when the database retains a stable reference, digest, media type, and schema identity.
- JSON evidence must remain portable across SQLite and PostgreSQL.
- Exports use explicit schema versions so later releases can interpret older packages.
## Acceptance Criteria
1. A new execution can be traced from its source digest through its frozen request manifest, transport response, parsed/normalized data, and derived outputs.
2. Exact response content is captured before SDK parsing and is clearly distinguished from the existing SDK snapshot.
3. Failed HTTP calls retain safe provider evidence; calls with no response record that fact explicitly.
4. Omitted parameters remain distinguishable from explicit values.
5. No persisted request, header set, UI display, log, or export contains API credentials.
6. Retrying or reprocessing does not overwrite prior execution evidence.
7. Historical records remain readable and are not mislabeled as exact transport captures.
8. A versioned generic artifact can represent OCR/layout JSON and its coordinate system without a provider-specific schema change.
9. Evidence exports include relationships, schema identities, and digests sufficient for independent integrity checks.
10. The benchmark protocol can compare the current baseline with another model on the same private corpus and scoring rules.
11. Additive migrations and focused tests work across the supported persistence model.
12. All destructive-test runs comply with the backup-and-restore protocol.
## Related Local References ## Related Local References
- [Draft V4.2 Implementation Plan](implementation_plan_v4_2.md) - [V4.2 Implementation Plan](implementation_plan_v4_2.md)
- [V4.1 Scope Boundary](../ver4.1/scope_boundary_v4_1.md) - [Digital Evidence and AI Processing Provenance](../invariant/ai_evidence_and_provenance.md)
- [V4 Architecture](../ver4/architecture_v4.md) - [V4 Architecture](../ver4/architecture_v4.md)
- [V4 Schema](../ver4/schema_v4.md) - [V4 Schema](../ver4/schema_v4.md)
- [V4 Requirements](../ver4/requirements_v4.md)
- [Transcription Methodology](../invariant/transcription_methodology.md)
docs/ver4.3/implementation_plan_v4_3.md
+100 -204
View File
@@ -1,241 +1,137 @@
# Implementation Plan (Version 4.3) # Draft Implementation Plan (Version 4.3)
## Goal ## Goal
Make processing evidence precise, append-only, secret-safe, and exportable while preserving every existing record and creating a provider-neutral home for future OCR/layout artifacts. Prepare a safe implementation path for Source page reordering and constrained application settings. This plan remains provisional until the V4.3 scope-freeze decisions are resolved.
## Implementation Principles ## Planning Constraints
- Implement the [Digital Evidence and AI Processing Provenance invariant](../invariant/ai_evidence_and_provenance.md), not a provider-specific approximation of it. - V4, V4.1, and V4.2 remain the behavioral baseline.
- Capture transport evidence before SDK parsing. - Reordering must be atomic and service-owned.
- Keep exact evidence separate from parsed and normalized representations. - Settings must use explicit domain operations rather than direct database, environment-file, or arbitrary filesystem access from UI pages.
- Prefer additive schema evolution and explicit compatibility behavior. - Prompt changes must preserve historical Job provenance and use a defined safe-write policy.
- Reference source content by digest rather than duplicating it in request JSON.
- Use allowlists for safe metadata capture.
- Keep persistence and evidence semantics behind service boundaries.
- Do not change the default model until a representative benchmark supports that decision.
## Current-State Gaps
| Current behavior | Gap to close |
| --- | --- |
| `Source` stores original file path, digest, and size. | Media type and image/page geometry used for an execution are not frozen with that execution. |
| `Job` stores prompt text/hash, requested model after resolution, temperature, and `top_p`. | The complete effective request structure, omitted-versus-explicit parameter state, routing constraints, and software versions are not frozen. |
| `JobSource.raw_api_response` stores `model_dump()` output from the OpenRouter SDK. | The exact HTTP body can be normalized by OpenRouter and filtered again by the SDK before persistence. |
| `JobSource.ai_metadata` stores finish reason and basic token counts. | Detailed accounting remains only in the SDK snapshot and is not a substitute for exact evidence. |
| Provider exceptions become application errors. | Safe HTTP error bodies, statuses, headers, and no-response distinctions are not persisted. |
| Worker logs elapsed time. | Execution duration is not stored on `JobSource`. |
| Source Detail displays AI metadata and the SDK snapshot. | The UI does not identify evidence layers or expose request/transport/software provenance. |
| No generic processing-artifact model exists. | Future OCR geometry would require ad hoc provider fields or an unrelated schema. |
## Expected Project Impact ## Expected Project Impact
| Area | Expected impact | | Area | Expected impact |
| --- | --- | | --- | --- |
| Database models and upgrades | Add execution-specification, transport-evidence, timing, software-context, and generic artifact storage without removing existing columns. | | Sources service | Add validated, transactional set-based page reordering. |
| OpenRouter adapter | Introduce a transport boundary that can capture exact body/status/safe headers before typed SDK parsing, or use supported SDK hooks that expose the unparsed response reliably. | | Documents/Sources UI | Add a reorder entry point and interaction for one Document. |
| Provider contract | Return structured evidence for success and failure without leaking provider-specific transport concerns into workflow orchestration. | | Documents service | Expand controlled Document Type maintenance operations. |
| Source and workflow services | Persist one append-only execution outcome and its artifacts transactionally; retain compatibility projections. | | People service | Expand controlled Person Role maintenance operations. |
| UI | Label and inspect evidence layers; export safe evidence packages through service operations. | | Prompt adapter/service | Add constrained listing, reading, validation, and safe writing of prompt artifacts. |
| Benchmarking | Add a private manifest and repeatable evaluator using the literal-transcription methodology. | | UI composition/navigation | Register Settings routes and navigation without moving persistence into UI code. |
| Tests and documentation | Add compatibility, capture, security, integrity, export, and benchmark-scoring coverage; correct overstated V4 evidence language. | | Tests | Add transaction, conflict, registry lifecycle, prompt safety, and UI workflow coverage. |
## Proposed Data Design ## Proposed Implementation Phases
Exact names should be confirmed against existing conventions before migration code is written. The design should provide the following logical records. ### 1. Resolve Scope-Freeze Decisions
### 1. Execution Evidence - Select and document the reorder interaction.
- Define whether active Jobs block reorder.
- Define prompt atomic-write, backup, and recovery policy.
- Decide whether prompt creation/deletion is excluded.
- Finalize registry ordering requirements.
- Remove the Draft designation only after these decisions are reflected in scope and acceptance criteria.
Extend `JobSource` or associate it one-to-one with a new execution-evidence record containing: ### 2. Define Service Contracts
- Request manifest JSON and manifest schema version. - Define a Source reorder command containing `document_id`, the complete ordered Source ID list, and a concurrency token or equivalent stale-write guard if supported by the current model.
- Transport status, body bytes or exact decoded body plus encoding/content type, and safe headers. - Define Document Type maintenance commands for create, relabel, sort, activate, and deactivate.
- Parsed SDK snapshot retained separately from transport content. - Define Person Role maintenance commands for create, relabel, activate, and deactivate.
- Application, adapter, SDK, and runtime version metadata. - Define a Prompt Store interface for constrained list/read/write behavior.
- Start, finish, and duration values. - Map validation, conflict, not-found, dependency, and filesystem failures to existing `AppError` categories.
- Router/provider request and generation identifiers when available.
- Failure phase and whether an HTTP response was received.
The implementation should evaluate a companion table rather than continuing to widen `JobSource`. A companion record better isolates large/optional evidence and permits clear one-to-one compatibility semantics. ### 3. Implement Transactional Source Reordering
### 2. Generic Processing Artifact - Load all Sources for the target Document in the same transaction.
- Reject missing, extra, duplicate, or foreign Source IDs.
- Reject stale writes using the selected concurrency policy.
- Apply a collision-safe renumbering strategy suitable for both SQLite and PostgreSQL.
- Finish with contiguous `page_number` values beginning at 1.
- Roll back the entire operation on any failure.
- Add service tests for valid reorder, no-op, reverse order, invalid membership, duplicates, stale submissions, rollback, and backend-compatible SQL behavior.
Add a one-to-many artifact model associated with a source and, when applicable, a producing execution: ### 4. Implement the Reorder UI
- Stable artifact UUID. - Add a Reorder Pages action from a Document-scoped Source view or Document Detail.
- `source_id` and optional execution/`job_source_id`. - Render Source labels/previews sufficient to identify each page.
- Semantic artifact type. - Capture the complete intended order.
- Media/serialization format. - Require explicit Save and provide Cancel without mutation.
- Schema name and version. - Surface validation and conflict errors through the shared error presenter.
- Producer and producer version. - Return to a Document-scoped ordered view after success.
- Inline JSON payload or external location. - Verify keyboard-accessible controls for any drag-and-drop interaction.
- Payload digest and byte size.
- Coordinate-system metadata when relevant.
- Creation timestamp.
Enforce exactly one content location: inline payload or external reference. An external artifact must be written durably and hashed before its database record commits. ### 5. Expand Registry Maintenance Services
### 3. Compatibility Projections - Reuse existing Document and People service ownership.
- Add explicit write methods rather than passing UI-mutated ORM objects directly where practical.
- Normalize and validate new stable codes.
- Reject duplicate codes deterministically.
- Block deletion or omit deletion entirely; use activation state for lifecycle management.
- Preserve inactive entries for historical reads.
- Add service tests for create, relabel, activation, deactivation, duplicates, immutable codes, and referenced records.
- Keep `JobSource.raw_api_response` unchanged for existing and new compatibility reads until a later deprecation decision. ### 6. Add Constrained Prompt Storage
- Keep `JobSource.ai_metadata` for indexed/display-ready normalized values.
- Keep `Source.raw_transcription` as the latest successful machine-output projection while treating per-execution `JobSource.raw_transcription` as history.
- Document that older rows have an SDK snapshot but no exact transport capture.
## Implementation Phases - Place filesystem access behind a dedicated Prompt Store/service boundary.
- Resolve all filenames directly beneath the configured prompt root and reject traversal.
- Permit only the agreed markdown extension and reject empty content.
- Implement the approved safe-write strategy, including flush/replace behavior and backup/recovery if selected.
- Preserve file encoding and provide explicit failures for read-only or unavailable storage.
- Do not modify any Job row when prompt defaults change.
- Add unit tests for valid reads/writes, traversal, invalid names, empty content, filesystem failures, and unchanged Job provenance.
### 1. Correct Terminology and Define Typed Contracts ### 7. Build the Settings UI
- Add typed domain models for request manifests, software context, transport metadata, failure phase, and artifact descriptors. - Register a Settings landing page and navigation entry.
- Version every persisted JSON contract from its first release. - Add separate pages or panels for Document Types, Person Roles, and Prompts.
- Define the safe response-header allowlist. Begin with correlation, content type/encoding, date, retry/rate-limit, and router-specific generation identifiers only when documented and non-secret. - Keep pages responsible for orchestration and notifications only.
- Define size limits and external-storage thresholds for exact bodies and artifacts. - Use service callbacks for all mutations.
- Correct `docs/ver4/schema_v4.md` under “Page-Level Execution and AI Outputs” so the existing column is described as an SDK-serialized OpenRouter response snapshot, not a complete provider envelope, exact HTTP body, or native upstream-provider response. Apply the same terminology to architecture and UI schema references. - Explain stable codes, inactive historical entries, and future-only prompt effects in the UI.
- Add serialization and secret-rejection unit tests before provider changes. - Do not render raw environment values or secrets.
### 2. Add Additive Persistence and Upgrade Behavior ### 8. Verification and Rollout
- Add the selected execution-evidence and artifact models. - Run focused service tests before UI integration tests.
- Add foreign keys, uniqueness constraints, and indexes for source/execution lookup. - Verify reorder behavior against Documents with one and many Sources.
- Implement idempotent upgrades following the repository's existing schema-upgrade policy. - Verify ordered transcription rendering and V4.1 previous/next navigation after reorder.
- Do not populate exact response fields for historical rows. - Verify inactive registry behavior in both historical display and create/edit selectors.
- Do not write a capture-time classification onto historical rows during migration. Compatibility reads may describe a populated legacy `raw_api_response` as an SDK snapshot, but exports must identify that description as a later compatibility interpretation rather than execution-time metadata. - Verify prompt changes are picked up by newly created Jobs while historical Jobs retain frozen content/hash.
- Verify JSON portability and large-payload behavior for SQLite and PostgreSQL. - Run the relevant regression suite.
- Add upgrade tests starting from a representative pre-V4.3 schema.
### 3. Build Secret-Safe Request Manifests ## Migration and Compatibility Notes
- Build the manifest from the concrete outgoing request body immediately before transport, not from a narrower typed projection that may discard unrecognized request fields. - No new table is expected solely for reordering; `Source.page_number` remains authoritative.
- Replace each image payload in that concrete representation with a source reference containing source UUID, digest, byte size, media type, dimensions, and transformation identity. - A uniqueness constraint on `(document_id, page_number)` should be evaluated before scope freeze. If added, migration and collision-safe update behavior must be designed for both supported databases.
- Store exact prompt content and preserve omitted-versus-explicit parameter state. - Existing registry records remain valid.
- Include requested model, routing preferences, response-format requirements, and timeout/retry policy. - Prompt editing changes mutable application files, not database provenance already captured on Jobs.
- Record application version/commit when available, adapter contract version, SDK package/version, and request-manifest schema version. - V4.3 must not require users to recreate existing Sources, Documents, People, roles, or types.
- Hash the canonical manifest representation for integrity checks.
- Test that credentials and embedded image data cannot enter the persisted manifest.
- Test that every field actually sent to the provider, including routing and future provider options, is represented or explicitly excluded by the manifest transform.
### 4. Capture OpenRouter Transport Evidence ## Proposed Delivery Order
- Evaluate the installed OpenRouter SDK hooks/client injection first. 1. Freeze the remaining decisions.
- If hooks cannot expose an exact stable response before typed parsing, implement the non-streaming OpenRouter call through the existing async HTTP client boundary while retaining typed validation in the adapter. 2. Implement and verify Source reorder service semantics.
- Read the response body once, preserve it exactly, then parse and normalize it. 3. Build the reorder UI.
- Store status, content type/encoding, allowlisted headers, request/generation ID, and timing. 4. Implement registry maintenance service operations.
- Maintain current authentication, referer/title headers, timeout behavior, and error classification. 5. Implement the Prompt Store and safety policy.
- Explicitly document that the captured body is the OpenRouter-normalized transport response, not Gemini/Anthropic/OpenAI native upstream JSON. 6. Build Settings pages.
- Add fixture-based tests proving unknown response fields survive transport capture even if a typed parser ignores them. 7. Run integration and regression verification.
### 5. Preserve Failure Evidence ## Draft Done Criteria
- Return or raise a typed provider failure that carries safe evidence separately from its user-facing error. - All V4.3 acceptance criteria are testable and satisfied.
- Persist non-success status/body/allowlisted headers before marking an execution failed. - Reordering is atomic, conflict-aware, contiguous, and cross-database compatible.
- Represent DNS/connect/TLS/local timeout failures as no-response outcomes with a failure phase and safe diagnostic category. - Settings mutations cross explicit service or adapter boundaries.
- Preserve response-validation failures with both the exact body and validation details. - Registry codes cannot be accidentally changed.
- Keep transcription-quality rejection distinct from provider failure because a valid provider response was received. - Prompt writes cannot escape the configured directory or rewrite historical provenance.
- Ensure error strings and logs do not contain authorization data or embedded image payloads. - No secret or raw environment editor exists.
- Add tests for 4xx, 5xx, malformed JSON, schema mismatch, timeout, connection failure, and quality rejection. - V4.1 and V4.2 workflows remain intact.
### 6. Make Execution History Reliably Append-Only
- Confirm retry behavior creates a distinct execution attempt rather than reusing and overwriting a completed evidence record.
- Separate queue linkage from execution-attempt identity; the current update-in-place behavior cannot serve as append-only execution history.
- Assign each attempt a deterministic, monotonically increasing attempt number scoped to its Job and Source, enforced by a database uniqueness constraint.
- Update the latest-transcription projection only after a successful attempt.
- Never update prior response bodies, manifests, timings, or artifacts during a retry.
- Select the latest attempt and latest successful attempt by the persisted attempt number with a stable identifier as a defensive secondary key, never by timestamp alone.
- Add service/workflow tests covering retries, partial success, interrupted jobs, and historical projection behavior.
### 7. Add Generic Artifact Persistence
- Implement service operations to create, read, list, verify, export, and, only under explicit retention policy, delete artifacts.
- Validate semantic type, schema/version, digest, media type, and coordinate metadata.
- Support JSON artifacts inline initially when within the agreed size threshold.
- Support external artifacts through a constrained application-data root with atomic write, digest verification, and explicit missing-file errors.
- Add a provider-neutral example fixture representing OCR words/lines with polygons and confidence values.
- Do not integrate a live OCR vendor in this phase.
### 8. Add Evidence Inspection and Export
- Rename the current Source Detail label to identify historical values as an OpenRouter SDK Response Snapshot.
- Add separate sections for Request Manifest, Transport Response, Normalized Metadata, Software Context, and Derived Artifacts.
- Show an explicit “not captured for this historical execution” state instead of an empty object.
- Keep large bodies collapsed by default and avoid rendering embedded source data.
- Add a service-owned export that packages a versioned manifest, evidence JSON/body files, artifact content or references, and digest inventory.
- Exclude secrets and machine-local paths that are not required to interpret the evidence.
- Add UI and export tests for new, historical, failed, and large-evidence records.
### 9. Establish the Private Benchmark
- Select a small initial corpus, then expand only when it exposes meaningful differences.
- Stratify examples by printed/typed text, handwriting style, degradation, layout complexity, language, and editorial anomaly.
- Reference existing Source UUIDs and digests in a private manifest; do not copy family documents into public test fixtures.
- Create manually reviewed reference transcriptions following the invariant methodology.
- Implement or adopt existing project-compatible CER/WER calculations without changing dependencies unless justified.
- Score omissions, inventions, silent modernization, uncertainty markup, and layout fidelity separately from CER/WER.
- Record cost and latency from preserved execution evidence.
- Run the current `google/gemini-2.5-flash` configuration as the baseline before testing alternatives.
- Treat results as model-version/route/corpus specific and preserve each comparison run.
### 10. Verify, Migrate, and Align Documentation
- Run the smallest focused model, provider, service, workflow, UI, upgrade, and export test groups first.
- Run broader regression tests only after focused validation passes.
- Execute all destructive tests through `tools/run_destructive_tests.py`.
- Verify backup creation and required restoration behavior before any test touching real application data.
- Confirm existing Source Detail records remain readable after upgrade.
- Update V4 architecture, schema, requirements, and UI schema mappings to point to V4.3 semantics.
- Record any deliberate deviation from this plan in the V4.3 scope before release.
## Recommended Delivery Order
1. Typed/versioned evidence contracts and terminology.
2. Additive execution-evidence persistence.
3. Secret-safe request manifests.
4. Exact OpenRouter transport capture.
5. Failure evidence and append-only retry semantics.
6. Generic artifact persistence.
7. Inspection and export.
8. Private benchmark tooling and baseline run.
9. Migration, regression verification, and documentation alignment.
## Key Implementation Decisions to Resolve
1. Whether execution evidence is a one-to-one companion to `JobSource` or part of a new execution-attempt model required for append-only retries.
2. Whether exact response bodies remain database values at expected sizes or move to hashed external files above a threshold.
3. The canonical JSON algorithm used to hash request manifests.
4. The safe-header allowlist supported by OpenRouter and future adapters.
5. The application version identity available in local, packaged, and uncommitted development builds.
6. The initial inline/external artifact size threshold and application-data root.
7. Whether evidence exports include original source binaries by default, optionally, or only by reference.
8. The minimum private benchmark corpus size and review process before model comparisons influence defaults.
These decisions must be settled before their corresponding implementation phase; they do not weaken the invariant or expand V4.3 into live OCR integration.
## Done When
- Every V4.3 acceptance criterion is satisfied by focused tests or an explicit demonstration.
- Existing SDK snapshots retain their content and are labeled accurately.
- New successful and failed calls preserve secret-safe provider-boundary evidence.
- Unknown transport fields survive even when the typed SDK/parser does not recognize them.
- Retries cannot overwrite prior execution evidence.
- A generic versioned artifact can represent OCR geometry and pass integrity verification.
- Evidence can be safely inspected and exported with schema identities and digests.
- The current model has a reproducible private benchmark baseline.
- No credential or embedded source payload appears in persisted manifests, safe headers, logs, or exports.
- Existing V4.1 and V4.2 behavior remains compatible.
## Out of Scope
- Live OCR/document-AI provider integration.
- Automatic model switching.
- Archive-wide reprocessing.
- Native upstream-provider response capture through OpenRouter when OpenRouter does not expose it.
- Guarantees of deterministic hosted-model output.
## Related Local References ## Related Local References
- [V4.3 Scope Boundary](scope_boundary_v4_3.md) - [Draft V4.3 Scope Boundary](scope_boundary_v4_3.md)
- [Digital Evidence and AI Processing Provenance](../invariant/ai_evidence_and_provenance.md)
- [V4 Architecture](../ver4/architecture_v4.md)
- [V4 Schema](../ver4/schema_v4.md)
- [V4 Requirements](../ver4/requirements_v4.md)
- [V4.2 Implementation Plan](../ver4.2/implementation_plan_v4_2.md) - [V4.2 Implementation Plan](../ver4.2/implementation_plan_v4_2.md)
- [V4.1 Implementation Plan](../ver4.1/implementation_plan_v4_1.md)
- [V4 Implementation Plan](../ver4/implementation_plan_v4.md)
- [V4 Error Handling Policy](../ver4/error_handling_v4.md)
docs/ver4.3/scope_boundary_v4_3.md
+94 -125
View File
@@ -1,163 +1,132 @@
# V4.3 Scope Boundary # Draft V4.3 Scope Boundary
This document defines the proposed boundary for the digital-evidence and AI-provenance revision that follows V4.2. V4 remains the architecture baseline; V4.3 makes the existing evidence claims precise and adds a provider-neutral foundation for future processing artifacts. This document defines the proposed boundary for the page-reordering and constrained-settings revision that follows the V4.2 evidence-and-provenance work. It is intentionally a draft until the earlier revisions have been used and the remaining workflows have been validated.
## Purpose ## Purpose
- Align the application with the [Digital Evidence and AI Processing Provenance invariant](../invariant/ai_evidence_and_provenance.md). - Allow correction of Source page order after import.
- Preserve provider-boundary evidence before SDK parsing can remove unknown fields. - Provide a constrained Settings area for safe maintenance of selected application-managed configuration.
- Make successful and failed processing attempts inspectable without storing secrets. - Avoid exposing secrets, restart-sensitive settings, or unrestricted filesystem editing through the UI.
- Support future OCR and layout outputs without coupling the database to one vendor.
- Establish a repeatable method for comparing transcription models against this archive.
## In Scope ## Proposed In Scope
### 1. Evidence Terminology and Existing-Data Compatibility ### 1. Source Page Reordering
- Define transport response, router-normalized response, SDK response, normalized metadata, and derived artifact consistently in code, schema documentation, and UI labels. - Allow Sources within one Document to be reordered after import.
- Treat existing `JobSource.raw_api_response` values as historical SDK response snapshots. - Present the current order using page number and a recognizable source label or preview.
- Preserve every existing `Job`, `Source`, and `JobSource` row. - Persist the complete intended order atomically.
- Use additive migrations and compatibility reads; do not reinterpret previously stored values as exact transport captures. - Renumber the affected Document's Sources to a contiguous sequence beginning at 1.
- Correct the “Page-Level Execution and AI Outputs” rule in `docs/ver4/schema_v4.md` that currently describes `JOB_SOURCE` as storing a complete provider response envelope. The corrected rule must identify `raw_api_response` as an SDK-serialized OpenRouter response snapshot and state that it is neither the exact HTTP body nor the native upstream-provider response. - Keep all Sources attached to their existing Document.
- Ensure transcription rendering and previous/next navigation use the updated order.
- Detect stale or invalid reorder submissions and fail without partial mutation.
### 2. Secret-Safe Request Manifests ### 2. Settings Navigation
- Persist the effective request specification for each page execution without storing credentials or duplicate base64 media. - Add a Settings entry to application navigation.
- Include requested provider/model, routing constraints, prompt content and hash, explicitly supplied parameters, source digest, media type, dimensions when known, and page identity. - Provide separate, clearly described settings areas rather than a raw configuration editor.
- Distinguish an omitted optional parameter from an explicitly supplied null or value. - Restrict V4.3 settings to application-managed values that can be validated and safely changed at runtime.
- Record application, provider-adapter, Python client, and relevant schema versions.
- Use source or derivative references in place of embedded media bytes.
### 3. Provider-Boundary Response Capture ### 3. Document Type Maintenance
- Capture the exact HTTP response body before OpenRouter SDK parsing for non-streaming transcription calls. - List active and inactive Document Types.
- Store HTTP status and an explicit allowlist of safe response headers. - Add new types with a stable unique code and user-facing label.
- Store router request/generation identifiers and resolved model/provider-routing metadata when exposed. - Edit mutable labels and sort order.
- Preserve the current parsed SDK snapshot and normalized metadata where useful. - Activate or deactivate types without invalidating historical Documents.
- Keep exact body, parsed representation, and normalized fields distinguishable. - Do not allow changing a stable code after creation.
- Do not delete types that are referenced by Documents.
### 4. Failure Evidence and Timing ### 4. Person Role Maintenance
- Create or update a page execution record for every attempted provider call. - List active and inactive Person Roles.
- Persist safe response evidence for non-success HTTP responses. - Add new roles with a stable unique code and user-facing label.
- Distinguish HTTP response failures, connection failures, local timeouts, response-validation failures, and transcription-quality failures. - Edit mutable labels.
- Store execution start/end times or duration using a clearly defined clock policy. - Activate or deactivate roles without invalidating historical links.
- Do not collapse a provider error body into only a generic user-facing message. - Do not allow changing a stable code after creation.
- Do not delete roles that are referenced by document-person links.
### 5. Generic Processing Artifacts ### 5. Prompt Maintenance
- Add a provider-neutral representation for versioned derived artifacts. - List prompt markdown files from the configured prompt directory.
- Support inline JSON and externally stored payloads with a digest and stable reference. - View a prompt with a concise explanation of its purpose and use.
- Record artifact type, format, schema/version, producer/version, source, producing execution, and creation time. - Edit an existing prompt as plain markdown text.
- Define coordinate-system metadata sufficient for word, line, block, or page geometry. - Validate the filename boundary and reject empty prompt content.
- Permit future OCR/layout/confidence results without implementing a vendor-specific table for each provider. - Save changes explicitly and report filesystem failures.
- Preserve submission-time prompt text and hash already frozen on existing Jobs.
- Define a safe-write and recovery approach before this feature is considered final scope.
### 6. Evidence Inspection and Export ## Proposed Out of Scope
- Expand Source Detail and/or Job Detail to identify the evidence layer being displayed. - Viewing or editing raw `.env` files.
- Provide readable JSON inspection for request manifests, transport metadata, parsed responses, normalized metadata, and derived artifacts. - Displaying or changing provider API keys and other secrets.
- Provide a safe export containing evidence content or references, relationships, schema versions, and digests. - Editing host, port, database connection, upload paths, or other restart-sensitive runtime settings.
- Clearly label evidence that was not captured for historical records. - Arbitrary file browsing or arbitrary prompt paths.
- Do not display or export credentials, unrestricted headers, or embedded base64 source media. - Runtime theme/CSS editing.
- Installing themes or plugins.
- Source movement between Documents as part of reordering.
- Automatic ordering based on filenames, OCR, or image content.
- FamilySearch API synchronization.
- A generic external-reference registry.
- Ancestry references and Google Maps links.
### 7. Representative-Corpus Benchmark Protocol ## Proposed Design Decisions
- Define a private benchmark manifest referencing source digests rather than duplicating archival media. ### A. Reordering Is Set-Based
- Include representative printed, typed, handwritten, degraded, tabular, and spatially complex pages.
- Pair each benchmark item with a manually reviewed literal transcription.
- Score character error rate, word error rate, omissions, inventions, silent normalization, uncertainty handling, layout fidelity, cost, and latency.
- Preserve the complete execution provenance for every benchmark run.
- Keep the current model as a baseline; do not change the application default solely from vendor benchmarks.
### 8. Migration, Integrity, and Verification - The client submits the full ordered list of Source IDs for one Document.
- The service validates membership, completeness, duplicates, and authorization/context before writing.
- All page-number updates occur in one transaction.
- Provide non-destructive upgrade behavior for supported SQLite and PostgreSQL deployments. ### B. Registry Codes Are Immutable
- Backfill only facts that can be derived reliably from existing records.
- Mark unavailable historical evidence as unavailable rather than fabricating it.
- Add digest, serialization, header-allowlist, failure-path, compatibility, artifact, export, and UI inspection tests.
- Run destructive tests only through the repository's required backup-and-restore wrapper.
## Out of Scope - Document Type and Person Role codes are stable identifiers.
- Labels and active state remain mutable.
- Historical references remain valid when a registry entry is inactive.
- Selecting or declaring a permanent best transcription model. ### C. No Raw Environment Editor
- Changing the default transcription model without benchmark evidence and a separate decision.
- Integrating Azure Document Intelligence, Google Document AI, Transkribus, Mistral OCR, or another OCR provider in V4.3.
- Generating bounding boxes retroactively for existing transcriptions.
- Bulk reprocessing the archive.
- Packet capture, TLS evidence, full unrestricted request/response headers, or credential retention.
- Storing duplicate base64 source images in request manifests.
- Guaranteeing byte-identical reproduction from nondeterministic or updated hosted models.
- Automatic entity extraction, biography generation, or genealogical inference.
- Replacing the relational database with an event store or content-addressed object store.
- Destructive renaming or removal of `raw_api_response`.
## Locked Design Decisions - `.env` may contain secrets and values that are not safely reloadable.
- V4.3 exposes only purpose-built forms backed by explicit validation and service methods.
### A. The Original Source Is Primary Evidence ### D. Prompt Editing Is Constrained
- Original uploaded bytes and their digest remain authoritative. - Prompt maintenance is limited to direct children of the configured prompt directory.
- Processing derivatives and outputs are independently identified derived evidence. - Existing Job provenance is never rewritten when a prompt file changes.
- Future OCR/layout work reuses the original or a documented derivative. - The UI must distinguish editing the default for future submissions from inspecting historical Job prompts.
### B. Evidence Is Layered ## Decisions Required Before Scope Freeze
- Exact transport evidence, SDK-parsed objects, normalized metadata, and transcription text serve different purposes. 1. Choose the reorder interaction: move-up/down controls, drag-and-drop, or both.
- One representation must not silently stand in for another. 2. Decide whether reordering is allowed while the Document has a queued or processing Job.
- UI and export labels name the stored evidence layer. 3. Define prompt backup, atomic-write, and recovery behavior.
4. Decide whether prompt creation and deletion are needed or whether V4.3 edits existing prompts only.
5. Confirm whether registry sort-order maintenance is needed for Person Roles as well as Document Types.
6. Confirm that settings changes remain local to the current installation and do not require an API surface.
### C. History Is Append-Only ## Draft Acceptance Criteria
- A retry or reprocessing attempt creates new execution evidence. 1. Reordering a Document's Sources produces contiguous page numbers and updates every ordered view consistently.
- Convenience caches may change, but historical execution output does not. 2. Invalid, incomplete, duplicate, cross-Document, or stale reorder requests make no changes.
- Human revisions remain separate from machine output. 3. Document Type and Person Role maintenance preserves stable codes and historical references.
4. Inactive registry entries remain visible on historical records but are excluded from default create selectors.
5. Prompt edits are restricted to valid markdown files in the configured prompt directory.
6. A prompt edit affects future Jobs only and leaves stored Job provenance unchanged.
7. No Settings page exposes secrets or unrestricted filesystem access.
8. Focused service and UI tests pass without regressing V4.1 or V4.2 workflows.
### D. Capture Is Secret-Safe by Construction ## Scope Freeze Gate
- Safe headers are allowlisted. V4.3 implementation should not begin until:
- Authorization, cookies, API keys, and unrestricted headers are never persisted.
- Request manifests reference source digests instead of embedding source bytes.
### E. Derived Artifacts Are Generic and Versioned - V4.1 has been used sufficiently to validate priorities.
- V4.2 evidence and provenance work has been completed and validated.
- Artifact storage is not limited to bounding boxes. - The six open decisions above are resolved.
- Coordinate metadata declares units, origin, dimensions, and transformations. - The prompt-write safety policy is documented.
- Provider-specific payloads may be retained without making provider-specific fields the durable application contract. - The implementation plan is revised from draft to committed delivery plan.
### F. Existing Evidence Keeps Its Original Meaning
- Existing `raw_api_response` data remains an SDK response snapshot.
- A migration may label or classify it but may not claim that missing transport data was captured.
- Historical nulls and absent fields remain distinguishable from new explicitly captured values.
## Data and Compatibility Policy
- All schema changes are additive in V4.3.
- Existing source files, hashes, transcriptions, revisions, prompts, jobs, and relationships remain valid.
- Compatibility reads continue to display historical SDK snapshots.
- Large derived artifacts may be stored outside the database when the database retains a stable reference, digest, media type, and schema identity.
- JSON evidence must remain portable across SQLite and PostgreSQL.
- Exports use explicit schema versions so later releases can interpret older packages.
## Acceptance Criteria
1. A new execution can be traced from its source digest through its frozen request manifest, transport response, parsed/normalized data, and derived outputs.
2. Exact response content is captured before SDK parsing and is clearly distinguished from the existing SDK snapshot.
3. Failed HTTP calls retain safe provider evidence; calls with no response record that fact explicitly.
4. Omitted parameters remain distinguishable from explicit values.
5. No persisted request, header set, UI display, log, or export contains API credentials.
6. Retrying or reprocessing does not overwrite prior execution evidence.
7. Historical records remain readable and are not mislabeled as exact transport captures.
8. A versioned generic artifact can represent OCR/layout JSON and its coordinate system without a provider-specific schema change.
9. Evidence exports include relationships, schema identities, and digests sufficient for independent integrity checks.
10. The benchmark protocol can compare the current baseline with another model on the same private corpus and scoring rules.
11. Additive migrations and focused tests work across the supported persistence model.
12. All destructive-test runs comply with the backup-and-restore protocol.
## Related Local References ## Related Local References
- [V4.3 Implementation Plan](implementation_plan_v4_3.md) - [Draft V4.3 Implementation Plan](implementation_plan_v4_3.md)
- [Digital Evidence and AI Processing Provenance](../invariant/ai_evidence_and_provenance.md) - [V4.2 Scope Boundary](../ver4.2/scope_boundary_v4_2.md)
- [V4.1 Scope Boundary](../ver4.1/scope_boundary_v4_1.md)
- [V4 Architecture](../ver4/architecture_v4.md) - [V4 Architecture](../ver4/architecture_v4.md)
- [V4 Schema](../ver4/schema_v4.md) - [V4 Schema](../ver4/schema_v4.md)
- [V4 Requirements](../ver4/requirements_v4.md)
- [Transcription Methodology](../invariant/transcription_methodology.md)
docs/ver4/architecture_v4.md
+34 -6
View File
@@ -4,15 +4,40 @@ This document describes the production architecture of the document transcriptio
## Architecture Objectives ## Architecture Objectives
- Preserve original source material and immutable machine transcription output. - Preserve original source material, per-execution machine output, and separate human revision.
- Support batching one or more images into ordered multi-page documents. - Support batching one or more images into ordered multi-page documents.
- Capture complete submission-time prompt provenance and per-page provider response evidence. - Capture submission-time prompt provenance and a per-page OpenRouter SDK response snapshot.
- Execute page transcription concurrently with bounded `asyncio` workers. - Execute page transcription concurrently with bounded `asyncio` workers.
- Maintain relational portability across SQLite and PostgreSQL. - Maintain relational portability across SQLite and PostgreSQL.
- Keep operator workflows cross-platform and Python-driven. - Keep operator workflows cross-platform and Python-driven.
- Support many-to-many document-person relationships with extensible roles. - Support many-to-many document-person relationships with extensible roles.
- Support registry-driven document type classification. - Support registry-driven document type classification.
## Core Capabilities
- Ingest one or more images into sequential `Source` pages under a `Document`.
- Execute asynchronous vision transcription with bounded worker concurrency.
- Preserve original source files with SHA-256 digests and byte sizes.
- Freeze prompt text, prompt hash, model, and explicitly configured sampling parameters on each `Job`.
- Preserve page-level machine output, normalized metadata, and an SDK-serialized OpenRouter response snapshot on `JobSource`.
- Organize historical `Person` records through many-to-many Document relationships and extensible roles.
- Classify Documents through a registry with stable type codes.
- Maintain human revision separately from machine-generated text.
- Isolate page failures so multi-page jobs can complete with partial success.
- Operate across supported platforms through Python-based application and maintenance tooling.
V4.2 extends this baseline with exact OpenRouter transport evidence and provider-neutral derived-artifact provenance. See the [V4.2 Scope Boundary](../ver4.2/scope_boundary_v4_2.md).
## Technical Stack
- **Runtime:** Python 3.12 or later.
- **Web application:** FastAPI and NiceGUI.
- **Persistence:** SQLModel and SQLAlchemy, with SQLite and PostgreSQL support.
- **Validation and settings:** Pydantic V2 and pydantic-settings.
- **Concurrency:** Python `asyncio` workers.
- **Vision integration:** OpenRouter through the application's provider adapter.
- **Testing and quality:** pytest, pytest-asyncio, Ruff, and ty.
## Runtime Topology ## Runtime Topology
The runtime operates as an asynchronous Python application: The runtime operates as an asynchronous Python application:
@@ -110,7 +135,7 @@ Responsibilities:
1. User uploads one or more images for a `Document`. 1. User uploads one or more images for a `Document`.
2. System stores files, hashes them, creates ordered `Source` rows, and creates a `Job`. 2. System stores files, hashes them, creates ordered `Source` rows, and creates a `Job`.
3. Worker claims the job, marks it `processing`, and executes page calls concurrently. 3. Worker claims the job, marks it `processing`, and executes page calls concurrently.
4. Each page writes a `JobSource` result with raw output, metadata, and full provider response evidence. 4. Each page writes a `JobSource` result with machine output, normalized metadata, and an SDK-serialized OpenRouter response snapshot.
5. Aggregate status becomes `completed`, `partial_success`, or `failed`. 5. Aggregate status becomes `completed`, `partial_success`, or `failed`.
### 2. Document-Person Relationship Management ### 2. Document-Person Relationship Management
@@ -128,12 +153,13 @@ Responsibilities:
3. Persistence stores the `document_type_id` reference. 3. Persistence stores the `document_type_id` reference.
4. Inactive types remain valid for historical rows but are excluded from default selectors. 4. Inactive types remain valid for historical rows but are excluded from default selectors.
## Domain Invariants ## V4 Domain Rules
- `Source.raw_transcription` stores immutable machine output. - `JobSource.raw_transcription` preserves page output for its Job execution.
- `Source.raw_transcription` is the latest-success machine-output projection for a page.
- Human corrections occur only in `Source.revised_text`. - Human corrections occur only in `Source.revised_text`.
- Prompt and parameter provenance is frozen on `Job` at submission time. - Prompt and parameter provenance is frozen on `Job` at submission time.
- Provider output evidence is stored on `JobSource` for each page execution. - The SDK-serialized OpenRouter response snapshot is stored on `JobSource` for each successful page execution.
- `DocumentPerson` links are unique for `(document_id, person_id, role_id)`. - `DocumentPerson` links are unique for `(document_id, person_id, role_id)`.
- Relationship mutations are deterministic and set-based. - Relationship mutations are deterministic and set-based.
- `DocumentType.code` is stable; `DocumentType.label` may evolve. - `DocumentType.code` is stable; `DocumentType.label` may evolve.
@@ -158,3 +184,5 @@ Responsibilities:
- [System Requirements](requirements_v4.md) - [System Requirements](requirements_v4.md)
- [Data Model](schema_v4.md) - [Data Model](schema_v4.md)
- [Error Handling Policy](error_handling_v4.md) - [Error Handling Policy](error_handling_v4.md)
- [Error Handling Invariant](../invariant/error_handling.md)
- [Digital Evidence and AI Processing Provenance](../invariant/ai_evidence_and_provenance.md)
docs/ver4/error_handling_v4.md
+11 -6
View File
@@ -1,13 +1,17 @@
# Error Handling Policy (Version 4) # Error Handling Policy (Version 4)
This document defines the canonical error-handling policy for the document transcription system. This document defines the Version 4 taxonomy, contracts, and framework behavior used to satisfy the cross-version [Error Handling invariant](../invariant/error_handling.md).
## Error Handling Objectives ## Invariant Alignment
- Make failures visible in clear, actionable language at both the document and page levels. Version 4 implements the invariant through:
- Support isolated failure handling in multi-page jobs so one failing page does not invalidate successful pages.
- Preserve diagnostic detail for validation failures, provider failures, and policy conflicts. - The shared error taxonomy below.
- Ensure consistent error envelope structure across API, UI, service, and worker boundaries. - Structured error envelopes with correlation IDs.
- Page-level failure isolation and explicit aggregate job status.
- Atomic relationship and classification writes.
- Consistent translation across API, UI, service, worker, persistence, and provider boundaries.
- Bounded retry guidance based on category and idempotency.
## Scope and Authority ## Scope and Authority
@@ -104,6 +108,7 @@ HTTP status mappings:
## Related Local References ## Related Local References
- [Error Handling Invariant](../invariant/error_handling.md)
- [System Overview](index_v4.md) - [System Overview](index_v4.md)
- [System Requirements](requirements_v4.md) - [System Requirements](requirements_v4.md)
- [Data Model](schema_v4.md) - [Data Model](schema_v4.md)
docs/ver4/index_v4.md
+19 -30
View File
@@ -1,40 +1,29 @@
# Document Transcription System Overview (Version 4) # Document Transcription System Overview (Version 4)
This project is a personal-scale application for transcribing, organizing, and preserving historical documents, images, and related people records. Version 4 is the architecture baseline for the personal-scale application used to transcribe, organize, and preserve historical documents, source images, and related people records.
## Start Here ## Recommended Reading Order
Read [architecture_v4.md](architecture_v4.md) first for the technical overview and system design. 1. [System Architecture](architecture_v4.md) for capabilities, technical stack, runtime structure, workflows, and component ownership.
2. [System Requirements](requirements_v4.md) for the verifiable V4 contract.
3. [Data Model](schema_v4.md) for entities, relationships, constraints, and persistence rules.
4. [Error Handling Policy](error_handling_v4.md) for the V4 taxonomy and boundary contracts.
## Core Capabilities ## Cross-Version Invariants
- Folder and multi-image ingestion into sequential `Source` pages under a single `Document`. - [Historical Document Transcription Design Intent](../invariant/intent.md)
- Parallel asynchronous AI vision transcription using Python `asyncio` bounded by rate limits. - [Transcription Methodology](../invariant/transcription_methodology.md)
- Portable relational storage using SQLModel and SQLAlchemy across SQLite and PostgreSQL. - [Error Handling](../invariant/error_handling.md)
- Complete prompt and response provenance for every transcription job and page execution. - [Digital Evidence and AI Processing Provenance](../invariant/ai_evidence_and_provenance.md)
- File-integrity tracking through SHA-256 hashing and stored file sizes. - [UI Style Guide](../invariant/ui_style_guide.md)
- Historical `Person` management with many-to-many document links and extensible relationship roles.
- Registry-driven `DocumentType` classification with stable codes and controlled selection.
- Inline human revision of transcribed pages while preserving immutable machine output.
- Partial-failure recovery for multi-page jobs.
- Cross-platform operational workflows driven by Python-based tooling.
## Technical Stack ## V4 Transition Documents
- Application Web Framework: FastAPI + NiceGUI
- Persistence Engine: SQLModel / SQLAlchemy
- Data Validation and Schemas: Pydantic V2
- Concurrency and Workers: Python `asyncio`
- Vision Providers: OpenAI, Anthropic, and OpenRouter adapters
## Core Documentation Index
- [System Architecture](architecture_v4.md)
- [System Requirements](requirements_v4.md)
- [Data Model](schema_v4.md)
- [Error Handling Policy](error_handling_v4.md)
## Transition Documents
- [Scope Boundary](scope_boundary_v4.md) - [Scope Boundary](scope_boundary_v4.md)
- [Implementation Plan](implementation_plan_v4.md) - [Implementation Plan](implementation_plan_v4.md)
## Incremental Revisions
- [V4.1 Scope](../ver4.1/scope_boundary_v4_1.md) and [Implementation Plan](../ver4.1/implementation_plan_v4_1.md)
- [V4.2 Evidence and Provenance Scope](../ver4.2/scope_boundary_v4_2.md) and [Implementation Plan](../ver4.2/implementation_plan_v4_2.md)
- [Draft V4.3 Page Reordering and Settings Scope](../ver4.3/scope_boundary_v4_3.md) and [Draft Implementation Plan](../ver4.3/implementation_plan_v4_3.md)