Update docs after db restructure

docs/Intent.md

@@ -1,24 +1,29 @@
 # Historical Document Transcription
 I have several thousand pages of family history told through letters, postcards, books, and other documents that I want to transcribe to text. 
 
+---
+
 ## Goals
 1. Preserve our family history
 2. Unburden my family (and descendants) from having to store and care for the physical media. Once the documents have been transcribed and organized, they can be donated (or kept by a family member that wants to retain them).
 3. Make the text easily available and searchable by family members, as well as AI (which may have different requirements).
 4. Ability create timelines or assemble the historical record of the family or specific individuals from across the complete document archive. Perhaps use AI to create the timelines in a more narrative form.
 
+---
+
 ## Source material
 1. **letters, cards, diaries** - handwritten; mostly stored in tubs, with little organization
 2. **books** - typed or typeset; mostly self-published books 50-100 pages in length. This may be expanded to include selected pages from other publications.
 3. **newspaper clippings, event programs, invitations, and other ephemera**
 
+---
+
 ## Methodology
 ### Verbatim vs. Clean Copy
 Transcriptions should be Verbatim and follow scholarly research guidelines, with no modifications to the original text.
 
 ### Prompt Curation Policy
 Transcription behavior should be implemented with prompt assets that are human-maintainable over time.
-
 1. Each transcription prompt is stored as an individual Markdown file.
 2. Prompt files are refined iteratively as document quality and edge cases are discovered.
 3. Prompt changes should be scoped to one prompt file at a time whenever possible to keep review history clear.

docs/architecture.md

@@ -17,7 +17,7 @@ The deployed system targets personal use and a corpus of several thousand docume
 
 Current scope includes:
 
-- document upload and metadata capture
+- content source upload and metadata capture
 - asynchronous transcription jobs
 - prompt-library driven transcription behavior, with one Markdown file per prompt
 - transcript review and revision history
@@ -131,9 +131,9 @@ Out of scope:
 
 Production transcription flow:
 
-1. A user uploads an image or PDF through the UI or API.
+1. A user uploads one or more content sources through the UI or API.
-2. The application validates payloads and creates document and job records.
+2. The application validates payloads and creates document, source, and job records.
-3. The in-process worker dequeues the job and calls the transcription provider.
+3. The in-process worker de-queues the job and calls the transcription provider.
 4. The application persists transcript output, confidence metadata, and provenance events.
 5. Job status transitions from queued to processing to transcribed or failed.
 6. The UI and API expose status, revision history, and searchable transcript text.
@@ -142,8 +142,8 @@ Production transcription flow:
 
 System-of-record entities:
 
-- documents and pages
+- documents and content sources
-- transcription jobs and status events
+- transcription jobs, original transcription, and status events
 - transcript revisions
 - provenance metadata
 
@@ -250,7 +250,7 @@ Control:
 
 Risk:
 
-- transcription quality varies by document type, handwriting legibility, and image quality
+- transcription quality varies by content source type, handwriting legibility, and source quality
 
 Control:
 
@@ -264,13 +264,9 @@ Control:
 - [PostgreSQL documentation](https://www.postgresql.org/docs/)
 - [MongoDB documentation](https://www.mongodb.com/docs/)
 
-## Related Pages
+## Related Local References
 
 - [System overview](index.md)
-- [Version 1 plan](ver1/ver1.md)
-- [Version 1 Step 1 plan](ver1/ver1-step1.md)
-- [Version 1 Step 1 results](ver1/ver1-step1-results.md)
-- [Architecture decision records index](adr/README.md)
 
 ## Glossary
 

docs/index.md

@@ -17,7 +17,7 @@ The architecture page is the primary technical reference and defines:
 
 ## What The Application Does
 
-At a high level, users upload images of handwritten, typed, or typeset documents, run asynchronous transcription jobs, review and edit transcript revisions, and search across accepted text.
+At a high level, users upload images or pdfs of handwritten, typed, or typeset documents, run asynchronous transcription jobs, review and edit transcript revisions, and search across accepted text.
 
 Core capabilities:
 

docs/requirements.md

@@ -15,7 +15,7 @@ This page captures a SysML v1.6-style requirements baseline for the production s
 | ID | Category | Requirement | Risk | Verify Method |
 | --- | --- | --- | --- | --- |
 | REQ-0 | System | Provide end-to-end document transcription with persistent, inspectable lifecycle state. | medium | demonstration |
-| REQ-1 | Functional | Allow users to upload one or more document images from the web UI. | low | test |
+| REQ-1 | Functional | Allow users to upload one or more images or pdfs from the web UI. | low | test |
 | REQ-2 | Functional | Run each upload through asynchronous processing that returns a transcription or explicit failure. | high | test |
 | REQ-3 | Functional | Persist and expose job states: upload, queued, processing, transcribed, failed, completed. | high | inspection |
 | REQ-4 | Functional | Persist transcription output, processing history, and failure details. | medium | test |

docs/schema.md

@@ -0,0 +1,71 @@
+
+## Database schema
+This document describes the structure of the database underlying the personal historical-document transcription system.
+
+---
+
+## Schema diagram
+
+```mermaid  
+erDiagram
+    document {
+        INTEGER id PK
+        TEXT name
+    }
+
+    job {
+        INTEGER id PK
+        INTEGER document_id FK
+        TEXT status
+        INTEGER retry_count
+        DATETIME date_created
+        DATETIME date_updated
+        TEXT provider
+        TEXT model
+        TEXT prompt_name
+        TEXT text
+        TEXT error_detail
+    }
+
+    source {
+        INTEGER id PK
+        INTEGER document_id FK
+        INTEGER job_id FK
+        TEXT upload_name
+        TEXT filename
+        TEXT file_path
+        DATETIME date_uploaded
+    }
+
+    revision {
+        INTEGER id PK
+        INTEGER image_id FK
+        INTEGER revision
+        TEXT text
+        DATETIME date_created
+    }
+
+    document ||--o{ source : "has 0 or more"
+    document ||--o{ job : "has 0 or more"
+    job ||--o{ source : "processes 0 or more"
+    source ||--o{ revision : "has 0 or more"  
+```
+
+---  
+
+## Table Relationships & Constraints
+* A document can consist of 0 or more content sources. A document can have 0 or more jobs.
+* An source can belong to only one job (which contains the original transcription). A source can only belong to one document. A source can have 0 or more transcription revisions. 
+* A job can process one or more sources. A job can belong to only one document.
+* A revision can belong to only one source. An source can have 0 or more revisions.
+
+---
+
+## Glossary
+* **Document** - Documents consist of one or more content sources and their related transcriptions.
+* **Source** - A content source that is transcribed to text. It can either be an image (.jpg, .tiff, .png) or a pdf (.pdf)
+* **Image** - The scanned image of one page of a document.
+* **PDF** - A pdf containing the image of one or more pages of a document. 
+* **Job** - A processing job ingests one or more sources, sends them to an AI model along with a prompt for transcription, then stores the results. The results are immutable, *including the original transcription*. The user can create a revision of the original transcription, but the user cannot modify the original.
+* Transcription - The text contained in a content source. A job creates the original transcription.
+* Revision - A modification of a transcription. A job creates the original transcription. A user can create one or more transcription revisions.