V5.1 Update tests and documentation

tests/api/test_v4_documents.py

@@ -206,6 +206,15 @@ def test_document_person_link_requires_role_id(tmp_path):
         )
 
     assert response.status_code == 422
+    payload = response.json()
+    assert "detail" in payload
+    assert isinstance(payload["detail"], list)
+    missing_role = [
+        item
+        for item in payload["detail"]
+        if isinstance(item, dict) and item.get("loc") == ["body", "role_id"] and item.get("type") == "missing"
+    ]
+    assert missing_role
 
 
 def test_duplicate_document_person_link_returns_conflict_envelope(tmp_path):

tests/artifacts/transcriptions/Book_Two_-_page_02.jpg.txt

@@ -3,17 +3,16 @@ provider: openrouter
 model: openai/gpt-5.3-codex
 ---
 [document body typewritten]
-
 BY WAY OF INTRODUCTION:~
 
 These few paragraphs of introduction may help you read BOOK 2 which covers a wider range than did BOOK 1 (Pioneer Days).
 
-BOOK 1 had 54 pages; 14 chapters. BOOK 2 has 70 pages; 18 chapters. BOOK 1 consisted largely of first generation family history. BOOK 2 throws more light on the second generation. Sidney promises a BOOK 3 and that may begin to do justice to the third generation. We suggest that Sidney get the help of Louis Shinn who has a chapter in this book (Chapter 16 - The Last 25 Years on the Doumecq Plains. Louis has the gift of seeing, recalling and telling. One sentence in his chapter gives a great tribute to the Doumecqers--so far as he knows no one on the Doumecq Plains went on relief during the depression. That in a nutshell shows the sturdy character of the residents of the Doumecq Plains.
+BOOK 1 had 54 pages; 14 chapters. BOOK 2 has 70 pages; 18 chapters. BOOK 1 consisted largely of first generation family history. BOOK 2 throws more light on the second generation. Sidney promises a BOOK 3 and that may begin to do justice to the third generation. We suggest that Sidney get the help of Louis Shinn who has a chapter in this book (Chapter 16 - The Last 25 Years on the Doumecq Plains.[sic] Louis has the gift of seeing, recalling and telling. One sentence in his chapter gives a great tribute to the Doumecqers--so far as he knows no one on the Doumecq Plains went on relief during the depression. That in a nutshell shows the sturdy character of the residents of the Doumecq Plains.
 
 We promised in BOOK 1 that in BOOK 2 we would give the story of the trip of John E. Cochran and wife to Tennessee, Cuba and the Panama Canal. You will see by the Table of Contents that the first four chapters have been given to those trips. Those chapters are worth reading and re-reading. Mr. Cochran has eyes to see and a pen to tell. We think the people in Tennessee will read with great pleasure the comments he makes on conditions today.
 
 Some who get this book will consider the group picture the best thing in the book. It took a lot of preliminary photographing to reduce some pictures, enlarge others and bring out the tin types. We wish that instead of 44 faces we could have given 88. Do not blame Ethel Cochran-Shinn for the selection. She furnished enough pictures but we had to take only part of them. We think there are great possibilities in reproducing old pictures. We wish we had a Pickard group. Some Pickard descendant may wish to make a collection.
 
-We are much impressed with the future possibilities of getting a complete geneology [sic] of the Pickard family. Mr. Cochran has a fine chapter on the Pickards but to date we have not had the pleasure of finding all of the family dates. We had intended to give more family data in this book but it takes time to get the correct dates. Often times it requires trips to cemeteries to get dates on the tombstones. Winter is no time to collect dates on tombstones.
+We are much impressed with the future possibilities of getting a complete geneology[sic] of the Pickard family. Mr. Cochran has a fine chapter on the Pickards but to date we have not had the pleasure of finding all of the family dates. We had intended to give more family data in this book but it takes time to get the correct dates. Often times it requires trips to cemeteries to get dates on the tombstones. Winter is no time to collect dates on tombstones.
 
 ~2~

tests/artifacts/transcriptions/Omie_Writes_Home.pdf.txt

@@ -24,6 +24,8 @@ The other place we stopped was at Whalen, a trading post in Siberia. There these
 
 We got home yesterday morning at 5 a.m. but missed the first lighter in so had to stay out until 2:30. The girls had prepared a big meal for us and invited up the Hartfords and then let us talk. Miss Saville talked quite a bit. Any how if you folks don't like this I don't care, it is all I had to write about and I know Buster'ud listen anyway and I'd soak ole Peter's head if he didn't and Polly would in my lap and I don't know much about the youngest one of yours so likely he would be squawling. But we did surely enjoy our trip and were gone just long enuf.
 
+[photograph of a group of people standing on snow]
+
 I expect there were 150 passengers on board and almost or more of the crew and helpers. We had a stateroom down next to the kitchen and 'twas pretty fierce for odor at times.
 
 I have had jobs nearly all summer but not very much in them. Next week, September 4, school opens. I wish they would wait for a week but you know these school men. Wouldn't make any special difference I suppose for I would just fritter away the time but still one likes to postpone the inevitable.

tests/artifacts/transcriptions/Rod_Moser_Letter_-_p1.jpg.txt

@@ -3,7 +3,7 @@ provider: openrouter
 model: openai/gpt-5.3-codex
 ---
 [document body mixed]
-JOHN ISBILL                                              R. T. MOSER
+JOHN ISBILL								R. T. MOSER
 
 ISBILL & MOSER
 DEALERS IN
@@ -16,18 +16,17 @@ I was at home a
 few nights ago & saw a
 letter from your folks, so
 I decided to write you
-a few lines [in?] regards of
-I am contemplat[ing?] a
+a few lines myself &
+ask contribution a
 trip out west next summer
-& want lots of [places?] to go
+I want lots of old [fellows?] to go
 where I am from.
-
-Am getting
-up in years & [remembering?].
-So you see the object of
+am getting
+up in years & [somebarries?].
+So you all are the object of
 my trip, is to get a wife
-If there is any old maids
+If there is any old maid
 or widows out there I
 want you to hire them
-at [pur?] [find?] me at them
+at our [illegible] me at there
 as soon as I get there.]

tests/test_db.py

@@ -6,7 +6,6 @@ import pytest
 import pytest_asyncio
 import sqlalchemy as sa
 from sqlalchemy import inspect
-from sqlalchemy import text
 from sqlalchemy.dialects import postgresql
 from sqlalchemy.exc import SAWarning
 from sqlmodel import SQLModel
@@ -77,7 +76,9 @@ async def test_get_session_yields_async_session(tmp_path):
 
     try:
         async with session_scope(settings=settings) as session:
-            assert session is not None
+            assert isinstance(session, AsyncSession)
+            value = (await session.exec(select(1))).one()
+            assert value == 1
     finally:
         await dispose_database_runtime()
 

tests/test_storage_reconciliation.py

@@ -16,6 +16,10 @@ from transcription.db.models import Document
 from transcription.db.models import Source
 
 
+def _normalize_identifier(value: str) -> str:
+    return value.replace("-", "").strip().lower()
+
+
 def _document_folder_ids(root: Path) -> set[str]:
     documents_root = root / "documents"
     if not documents_root.exists():
@@ -47,37 +51,53 @@ def _source_file_count_for_document(root: Path, document_id: str) -> int:
 async def assert_storage_reconciliation(*, upload_dir: Path, settings: Settings) -> None:
     folder_ids = _document_folder_ids(upload_dir)
     doc_ids = await _document_ids(settings)
-
-    missing_in_table = sorted(folder_ids - doc_ids)
-    missing_in_folders = sorted(doc_ids - folder_ids)
-    if missing_in_table:
-        folder = missing_in_table[0]
-        raise AssertionError(
-            "Document directory count and document.doc_id count do not agree. "
-            f"./data/documents/{folder} does not appear in document table"
-        )
-    if missing_in_folders:
-        doc_id = missing_in_folders[0]
-        raise AssertionError(
-            "Document directory count and document.doc_id count do not agree. "
-            f"document.doc_id {doc_id} has no corresponding folder in ./data/documents"
-        )
-
+    folder_by_normalized = {_normalize_identifier(folder_id): folder_id for folder_id in folder_ids}
+    doc_by_normalized = {_normalize_identifier(doc_id): doc_id for doc_id in doc_ids}
     source_counts = await _source_counts_by_document(settings)
-    for document_id in sorted(doc_ids):
-        db_count = source_counts.get(document_id, 0)
-        file_count = _source_file_count_for_document(upload_dir, document_id)
-        if file_count > db_count:
-            raise AssertionError(
-                "Source file count and source.source_id count do not agree. "
-                f"[UPLOAD_DIR]/documents/{document_id}/ contains file(s) with no source row"
+    source_counts_by_normalized = {
+        _normalize_identifier(document_id): count for document_id, count in source_counts.items()
+    }
+    mismatches: list[str] = []
+
+    missing_in_table = sorted(set(folder_by_normalized) - set(doc_by_normalized))
+    for folder_key in missing_in_table:
+        folder_name = folder_by_normalized[folder_key]
+        mismatches.append(
+            f"document-folder-without-row: ./data/documents/{folder_name} has no matching document.doc_id"
+        )
+
+    missing_in_folders = sorted(set(doc_by_normalized) - set(folder_by_normalized))
+    for doc_key in missing_in_folders:
+        doc_id = doc_by_normalized[doc_key]
+        source_count = source_counts_by_normalized.get(doc_key, 0)
+        mismatches.append(
+            "document-row-without-folder: "
+            f"document.doc_id {doc_id} has no corresponding folder in ./data/documents "
+            f"(source rows: {source_count})"
+        )
+
+    for doc_key in sorted(doc_by_normalized):
+        doc_id = doc_by_normalized[doc_key]
+        folder_name = folder_by_normalized.get(doc_key)
+        db_count = source_counts_by_normalized.get(doc_key, 0)
+        file_count = _source_file_count_for_document(upload_dir, folder_name) if folder_name is not None else 0
+        if db_count != file_count:
+            folder_display = (
+                f"./data/documents/{folder_name}" if folder_name is not None else "./data/documents/<missing-folder>"
             )
-        if db_count > file_count:
-            raise AssertionError(
-                "Source file count and source.source_id count do not agree. "
-                f"source.source_id rows exist without files in [UPLOAD_DIR]/documents/{document_id}"
+            mismatches.append(
+                "source-count-mismatch: "
+                f"document.doc_id {doc_id} -> source rows: {db_count}, files in {folder_display}: {file_count}"
             )
 
+    if mismatches:
+        report = "\n".join(f"- {item}" for item in mismatches)
+        raise AssertionError(
+            "Storage reconciliation mismatch(es) detected.\n"
+            f"Reconciling item count: {len(mismatches)}\n"
+            f"{report}"
+        )
+
 
 @pytest.mark.asyncio
 async def test_storage_reconciliation_passes_for_matching_counts(tmp_path, default_settings: Settings):
@@ -112,5 +132,20 @@ async def test_storage_reconciliation_reports_actionable_mismatch_message(tmp_pa
     orphan_dir = upload_dir / "documents" / str(uuid4())
     orphan_dir.mkdir(parents=True, exist_ok=True)
 
-    with pytest.raises(AssertionError, match="document\\.doc_id count do not agree"):
+    with pytest.raises(AssertionError, match="Reconciling item count:"):
+        await assert_storage_reconciliation(upload_dir=upload_dir, settings=default_settings)
+
+
+@pytest.mark.asyncio
+async def test_storage_reconciliation_reports_missing_document_folder_even_without_sources(
+    tmp_path, default_settings: Settings
+):
+    upload_dir = tmp_path / "uploads"
+    missing_folder_doc_id = uuid4()
+
+    async with session_scope(settings=default_settings) as session:
+        session.add(Document(id=missing_folder_doc_id, name="Needs folder"))
+        await session.commit()
+
+    with pytest.raises(AssertionError, match="document-row-without-folder:"):
         await assert_storage_reconciliation(upload_dir=upload_dir, settings=default_settings)

tests/ui/test_media_urls.py

@@ -2,16 +2,6 @@ from transcription.ui.components.media_urls import public_media_path_label
 from transcription.ui.components.media_urls import resolve_media_url
 
 
-def test_resolve_media_url_maps_managed_absolute_path_to_upload_route(tmp_path):
-    upload_dir = tmp_path / "uploads"
-    canonical_path = upload_dir / "documents" / "abc" / "page.jpg"
-    canonical_path.parent.mkdir(parents=True, exist_ok=True)
-    canonical_path.write_bytes(b"x")
-
-    resolved = resolve_media_url("documents/abc/page.jpg", upload_dir=upload_dir, base_url="http://localhost:8000")
-    assert resolved == "http://localhost:8000/uploads/documents/abc/page.jpg"
-
-
 def test_resolve_media_url_rejects_absolute_filesystem_path(tmp_path):
     upload_dir = tmp_path / "uploads"
     absolute_path = upload_dir / "documents" / "abc" / "page.jpg"

tests/ui/test_print_preview_page.py

@@ -113,3 +113,4 @@ async def test_document_source_media_rejects_cross_document_access(app_client):
 
     response = client.get(f"/api/documents/{other_id}/sources/{source_id}/media")
     assert response.status_code == 404
+    assert response.json()["detail"] == "Source not found for Document"