Adds tools/migrate_v45_to_v46.py, the final V4.6 deliverable.
Diffing the backup against the current SQLModel metadata showed that the
re-level changed no columns: both have the same 10 tables with identical
column sets. What changed is index coverage [HIGH-04], the use_alter break in
the source/execution_attempt foreign key cycle, and the relationship loading
strategy [CRIT-02]. The migration is therefore a faithful, foreign-key-ordered
row copy rather than a transformation.
Design:
- The backup is read with plain sqlite3 rather than through the ORM. The plan
anticipated ORM reads carrying explicit eager loads under lazy="raise";
raw reads are strictly safer, because the V4.5 file is not guaranteed to
satisfy the V4.6 mappers and no relationship is ever traversed.
- Writes go through SQLAlchemy Core against the live metadata, so the script
works unchanged against PostgreSQL when that cutover happens.
- source rows are inserted with preferred_execution_attempt_id cleared and the
selections are replayed after execution_attempt is populated, matching the
use_alter break in the cycle.
- _coerce() converts raw SQLite values into what each column binds. It accepts
both enum spellings, because job_source.status declares values_callable and
stores lowercase values while execution_attempt.status does not and stores
uppercase names, despite both using JobSourceStatus.
- Idempotent: a row whose primary key already exists is skipped, never
updated. Never invoked from application startup and never run by the test
suite.
- A pre-flight guard aborts if the backup row counts do not match the recorded
V4.5 snapshot, so the script cannot silently run against the wrong file.
Verification against a throwaway target:
- 282 rows copied; per-table counts match the plan exactly (document 8,
document_person 11, document_type 7, execution_attempt 80, job 11,
job_source 79, person 5, person_role 3, processing_artifact 2, source 76).
- Every table is cell-for-cell identical to the backup across all columns.
- A second run inserts 0 rows and skips all 282.
- Artifact integrity passes for every migrated artifact, checked through the
application's own SourceService verifier.
- 9 indexes added, 0 lost. No on-disk Source, portrait, or artifact file is
written by the script.
The live data/transcription.db is deliberately left untouched; it currently
holds only bootstrap seed rows whose UUIDs differ from the backup.
Co-authored-by: Copilot App <[email protected]>
Baseline was 207 diagnostics. Two real bugs were hiding in the noise:
- tools/run_destructive_tests.py imported ctypes.wintypes at module scope,
which raises on non-Windows, and called fcntl unconditionally. The Windows
and POSIX implementations now live under a module-level sys.platform split.
- tests/ui/test_sources_page.py constructed Source(...) without document_id.
Structural fixes, not suppressions:
- New src/transcription/db/loading.py owns the SQLModel-field to
QueryableAttribute reinterpretation via orm_attribute()/selectinload()/
defer(). This removed 42 "# pyright: ignore[reportArgumentType]" comments
across documents/jobs/people/sources. Its docstring records that
selectinload(A.b, B.c) is NOT equivalent to the chained form: varargs
applies the selectin strategy only to the last path element, which under
lazy="raise" raises InvalidRequestError at render time.
- db/session.py transaction_scope no longer accepts or yields
AsyncSessionTransaction. No caller ever passed one, sessionmaker.begin()
yields an AsyncSession, and the dead branch was latently buggy because
services call .exec(). Cleared 7 workflows.py diagnostics.
- services/registry.py RegistryService is bound by a new RegistryEntry
Protocol instead of bare SQLModel, so the shared implementation can read
id/label/normalized_label/is_active. Cleared 9 diagnostics.
- Column expressions in sources.py/jobs.py/test_store.py wrap in sqlmodel
col(), the idiom already used in registry.py.
- read_source_navigation wraps its literal tuple bounds in literal().
- normalization.py narrows with isinstance(image, TiffImageFile) rather than
comparing image.format, since tag_v2 is TIFF-only.
- linked_people.render uses @ui.refreshable_method, the NiceGUI API for bound
methods.
- The OpenRouter capturing client re-raises ResponseNotRead when the response
stream is not async rather than mis-wrapping it.
Tooling gate:
- New .pre-commit-config.yaml runs ruff check and ty check as blocking hooks.
No pre-commit config previously existed. Negative-tested: injecting a type
error fails both hooks.
- The last two "# pyright: ignore" comments (config.py) are removed; ty does
not honor pyright directives. One "# ty: ignore" remains, in
tests/test_prompts.py, where the test deliberately assigns to a frozen
field to assert ValidationError.
- asyncio_default_fixture_loop_scope is pinned to "function" so
pytest-asyncio behavior does not shift on upgrade.
Verification: ruff check clean, ty check reports 0 diagnostics, 292 passed
and 4 skipped, pre-commit passes and demonstrably fails on a regression, and
tools/run_destructive_tests.py runs on Windows.
Co-authored-by: Copilot App <[email protected]>
ui/homepage_store.py was the only storage path in the codebase derived from
Path(__file__).parents[3] rather than from Settings. That made it the one
storage root the operator could not relocate, and it resolved incorrectly
outside a source checkout - an installed distribution would have written
homepage content into the package directory in site-packages.
- config.py: add homepage_dir, defaulting to ./data/homepage so the location
is unchanged for anyone launching from the repository root.
- homepage_store.py: resolve the directory and markdown path from Settings,
with an optional settings parameter on every function so callers and tests
can override without patching module constants. HOME_PAGE_DIR and
HOME_PAGE_MARKDOWN_PATH constants are replaced by homepage_dir() and
homepage_markdown_path().
- tests/ui/test_homepage_store.py: covers the setting being honored, markdown
round-tripping, image storage and listing, and two configurations not
sharing storage.
Note: the default is now CWD-relative, matching artifact_dir and upload_dir,
rather than anchored to the repository root.
Verification: ruff check src tests clean; 292 passed, 4 skipped.
Co-authored-by: Copilot App <[email protected]>
MED-01 - move remaining blocking work off the event loop:
- normalization.py gains normalize_orientation_async; the Pillow decode,
transpose, and re-encode now run via asyncio.to_thread. The sync entry point
stays for tests and documents that it blocks.
- OrientationNormalization.digest_sha256 becomes a stored field computed inside
normalize_orientation, which already runs off-loop, instead of a property that
hashed page-sized derivative bytes on the caller's thread.
- SourceService._write_and_digest_artifact performs the artifact write and its
sha256 in a single worker-thread hop; both external-artifact write sites are
now dispatched through to_thread.
- transcribe_image dispatches load_source_payload and build_prompt_execution
through to_thread.
MED-04 - replace functools.cache on the engine and session factories with
explicit URL-keyed registries. dispose_engine and dispose_session_factory now
evict only the requested URL; previously cache_clear() tore down every other
database in the process, and dispose_engine would construct an engine for an
unknown URL purely to throw it away. New tests/test_engine_registry.py covers
distinct engines per URL, targeted eviction, and the unknown-URL no-op.
config.py - replace object.__setattr__ in normalize_provider_models with a
model_validator(mode="before") over the raw input, so the derived selector is
produced by normal construction rather than by mutating a frozen instance.
model_copy(update=...) was tried first and rejected: pydantic-settings does not
support a top-level validator returning anything other than self when validating
via __init__. provider_model is now stripped as well as the tuple entries.
models.py - add onupdate to the five updated_at columns and to Job.date_updated,
and drop the 10 manual "updated_at = datetime.now(UTC)" assignments across the
document, job, people, registry, and source services. Verified DDL-neutral by
hashing CreateTable output for every table on both the sqlite and postgresql
dialects before and after: identical, so this stays in Phase 6 and Phase 2 does
not need re-verification. New tests/services/test_timestamps.py asserts an
update through each service advances the timestamp.
MED-08 - Job.filename no longer swallows every exception to None. Relationships
declare lazy="raise", so the new _loaded_attribute helper inspects load state
explicitly and returns None only for genuinely unloaded attributes; real errors
now surface. Job.error_detail uses the same helper, which also removes its
unguarded read of the lazy="raise" job_sources relationship.
Verification: ruff check src tests clean; 288 passed, 4 skipped.
Co-authored-by: Copilot App <[email protected]>
Completes the table consolidation deferred within Phase 5. Section 4 of the
review lists table construction as a duplication pattern; three call sites had
been left bypassing the canonical builder.
- table/common.py: build_table gains a row_key parameter so callers with a
non-"id" primary key can use it.
- linked_people.py: replaces its hand-rolled ui.table with build_table
(row_key="person_id", selection="multiple", rows_per_page=0, no search).
- print_preview_page.py: _render_metadata_table and _render_job_table now share
a local _render_print_table helper. Print tables deliberately do not use
build_table - they must never paginate or render a search box, and they carry
print-only styling. The helper docstring records that rationale.
- tests/test_ui_boundaries.py: new AST guard asserting ui.table() is called from
exactly two modules - components/table/common.py and pages/print_preview_page.py.
Also closes the intermittent tests/ui/test_jobs_page.py failure observed once
after Phase 5 as environmental. Unreproduced across ~54 sequential full-suite
runs (including a 25-run soak) and 5 concurrent-process runs. No code change.
Verification: ruff check src tests clean; 275 passed, 4 skipped.
Co-authored-by: Copilot App <[email protected]>
Fixes the three ui.instructions.md violations recorded as [HIGH-07] and extracts
the page-level duplication catalogued in review section 4.
Boundary violations
- jobs_page no longer imports session_scope or manages a transaction.
store.create_document_job and store.create_job_for_document accept an optional
session_factory and open their own session scope when the caller supplies
neither a session nor a factory.
- sources_page no longer calls sqlalchemy.inspect. SourceService
.read_latest_execution_attempt now returns a LatestExecutionAttempt read model
carrying a plain transport_body_deferred flag, so ORM loader state stays inside
the service. Rendered output is unchanged.
- Deletes ui/components/document_panzoom.py, its export, and its CSS. The
component was exported but used by no page. Pan-zoom is planned for a clean
reintroduction in V4.7 alongside the other photo/image work.
Extracted duplication
- ui/components/media_urls.py: pure upload-URL resolution taking upload_dir and
base_url, replacing two identical ~60-line copies in sources_page and
people_page.
- ui/components/guards.py: parse-then-render-terminal-message, replacing 28
hand-written guard labels across five pages.
- ui/components/confirm_delete.py: the blocked-dependency notice and the
delete/cancel action row, from four delete pages.
- ui/components/upload_panel.py: the auto-uploading file picker, from three
pages. Source accept lists now derive from services.source_media
.SOURCE_EXTENSIONS instead of being hard-coded.
- ui/components/table/registry.py: the two hand-rolled label-registry tables on
the settings page now go through build_table, which gained selection and
rows_per_page options.
- ui/components/formatters.py gains parse_uuid and parse_iso_date, replacing
five and two private copies.
- ui/runtime.py owns resolve_runtime_settings, replacing three copies and
removing get_settings from every page module.
[LOW-05]
- Upload handlers are annotated with events.UploadEventArguments.
- The Document and Person form builders return DocumentFormFields and
PersonFormFields dataclasses instead of dict[str, Any].
Verification
- tests/test_ui_boundaries.py asserts no page imports a session scope, a session
factory, get_settings, sqlalchemy, or sqlmodel, and that no component imports
request or application state.
- 275 passed, 4 skipped. ruff check clean.
Findings: HIGH-07, LOW-05
Co-authored-by: Copilot App <[email protected]>
Removes the duplicated registry CRUD, the hand-written not-found raises, and
the three divergent media writers. Behavior is preserved: every existing
Document Type and Person Role test passes unchanged, which is the primary
proof for MED-11.
[MED-11] Generic registry service
- New services/registry.py owns RegistryService[ModelT]: list, list with
counts, create with IntegrityError -> conflict mapping, read, update,
delete with built-in and referenced guards, is_referenced, and label
normalization/casefold keying.
- DocumentTypeRegistry and PersonRoleRegistry declare only the model, error
class, noun, short noun, retainer phrase, and reference columns.
- DocumentService and PeopleService keep their public method names and
delegate. Every user-facing message, error category, and suggestion string
is reproduced verbatim; only the noun is templated.
- Deleted _normalize_registry_label, _document_type_label_key,
_normalize_role_label, _person_role_label_key,
_document_type_is_referenced, and _person_role_is_referenced.
[MED-12] Shared not-found lookup
- ServiceBase._get_or_raise(model, id, *, session, error, noun, suggestion,
options) loads by primary key or raises the caller's error type.
- documents.py: local _get_document_or_raise deleted; replaced by _read_document
and adopted at read_document, delete_document, and set_document_type, which
previously bypassed the helper and hand-wrote the raise.
- sources.py: 8 identical Source raises and 1 Job raise collapsed into
_read_source / _get_or_raise.
- jobs.py and people.py already funneled through local _not_found builders and
were left alone.
[MED-13][MED-01] Single media writer
- New services/media_storage.py owns validate -> name -> mkdir -> write ->
wrap OSError. The write runs in asyncio.to_thread, so uploads no longer block
the event loop.
- store_source_file, store_person_portrait, and store_homepage_image now share
it and are async. Callers in store.py, people_page.py, and home_page.py await
them. mkdir failures are now also translated to a domain error instead of
escaping as a raw OSError.
- homepage_store gains HomepageStorageError so its write reports like the others.
[MED-14, partial] Service independence
- New services/source_media.py owns SOURCE_MIME_TYPES, SOURCE_EXTENSIONS,
lookup_source_mime_type, and supported_source_formats.
- documents.py no longer imports services/sources.py. Its print projection uses
the non-raising lookup and raises DocumentError, so DocumentService no longer
emits a TranscriptionError.
- api/v4_print.py imports the mapping from the policy module.
- store.py and workflows.py still import sources.py; both are orchestration
modules, which services.instructions.md:75-77 explicitly permits.
- Splitting SourceService itself remains deferred to V4.7.
[LOW-08] Query shape
- list_sources_detail filters job_id with a JOIN on JobSource instead of
loading every Source and filtering in Python.
- read_source_navigation replaces the full ordered-id scan and .index() with
two row-value comparisons bounded by LIMIT 1.
- list_processing_artifacts gains the limit parameter its summary sibling
already had.
- build_evidence_export runs artifact integrity hashing and file reads through
asyncio.to_thread.
Tests
- tests/test_service_boundaries.py: AST guard asserting no service module
imports a sibling service module, plus a guard that the scan is non-empty.
- tests/services/test_transcription_service.py: asserts the job_id filter emits
a JOIN, and that navigation emits exactly two LIMIT queries.
- tests/services/test_store.py: the two storage tests are now async.
Verified: 276 passed, 4 skipped; ruff check clean.
Claim jobs atomically [CRIT-01]
- Replace JobService.read_next_queued_job with claim_next_queued_job, which
selects and transitions QUEUED -> PROCESSING inside one transaction. The old
read-then-write sequence left a window in which two workers could observe the
same QUEUED row.
- Add the missing .limit(1). The poll previously ordered the entire queued set
and discarded all but the first row.
- Drop the eager loads from the hot poll entirely. They were pure waste:
process_queued_job immediately re-reads the job through read_job with the
relationships it actually needs.
- Guard the row with with_for_update(skip_locked=True) on PostgreSQL so the
claim stays correct once more than one worker exists. On SQLite the claim is a
bounded single-writer transaction.
- Correct the comment at the remaining direct-call claim site, which described
the hazard rather than the guarantee.
Reuse the provider connection [HIGH-02]
- Build the ServiceBundle once per worker loop instead of once per job, and
close it at loop shutdown. Every job previously constructed a new
SourceService, and with it a new provider adapter and a new httpx.AsyncClient,
paying a full TLS handshake per page and discarding the connection pool.
- process_next_queued_job now accepts an optional caller-owned bundle and only
closes bundles it created itself.
Uncap the provider timeout [HIGH-03]
- Remove le=20.0 from worker_provider_timeout_seconds. The cap equalled the
default, so the ceiling could never be raised, and dense-page vision
transcription routinely needs longer. Default raised to 180s.
- Pass an explicit httpx.Timeout to the OpenRouter AsyncClient. httpx defaults
every phase to 5 seconds, so the real read budget was 5s regardless of the
configured value; the outer asyncio.wait_for could never be the binding
constraint. Connect stays at 10s.
Tighten the provider boundary [MED-03]
- Declare model, current_request_manifest, current_transport_evidence, and
aclose on the TranscriptionProvider Protocol.
- Delete the per-call inspect.signature(adapter.transcribe).parameters
reflection and the untyped kwargs dict it fed. The Protocol had declared
requested_model all along, so the reflection was dead defensive weight on the
hot path.
- Replace the three getattr probes for aclose and the evidence attributes with
direct typed access.
Deduplicate bundle construction [MED-06]
- Add ServiceBundle.from_session_factory and ServiceBundle.aclose, replacing the
duplicated four-service instantiation blocks in app.py and worker.py.
- _recover_stale_processing_jobs now uses the bundle built moments earlier
instead of constructing a second JobService.
Tests
- Claiming returns the oldest job, marks it PROCESSING, never hands the same job
out twice, and emits exactly one unadorned SELECT carrying LIMIT and no JOIN.
- The worker loop threads one bundle through consecutive jobs and closes it once
at shutdown; a caller-owned bundle is left open.
- Settings accepts a timeout above 20 seconds and still rejects zero.
- The OpenRouter client's read, write, and pool timeouts track the configured
budget rather than the httpx default.
Note: .env in this checkout still pins WORKER_PROVIDER_TIMEOUT_SECONDS=20 and
should be raised to pick up this fix.
Verified: 268 passed, 4 skipped; ruff check clean.
Co-authored-by: Copilot App <[email protected]>
These changes all regenerate the same schema, so they land together and revert
together. A partially applied schema pass is not a valid state.
Remove hand-rolled migrations [HIGH-05]
- Delete upgrade_schema and the _upgrade_person_family_search_id /
_upgrade_v42_evidence_tables / _upgrade_v45_selection_columns chain, plus the
two tests that exercised them. The DDL was SQLite-shaped raw SQL that would
not have run on PostgreSQL. create_all now derives everything from metadata
and remains gated by Settings.should_bootstrap_schema. No raw ALTER TABLE or
CREATE INDEX string remains in src.
Break the foreign key cycle [HIGH-08]
- Declare Source.preferred_execution_attempt_id with use_alter=True and an
explicit constraint name. source / job_source / execution_attempt formed an
unresolvable cycle that made metadata.sorted_tables emit an SAWarning and
order execution_attempt before source, which would have been a hard
create_all failure on PostgreSQL and was invisible on SQLite.
- As a side effect the column is now a dialect-aware Uuid rather than the
hardcoded CHAR(32) the raw upgrade DDL produced, so it emits native UUID on
PostgreSQL.
Index the hot filters [HIGH-04]
- Add composite Index("ix_job_status_date_created", "status", "date_created")
for the worker poll, and index the foreign keys the worker and detail pages
filter on: job.document_id, source.document_id, job_source.job_id,
job_source.source_id, document.document_type_id, and the three
document_person foreign keys.
Stop preloading by default [CRIT-02]
- Flip 16 relationships from lazy="selectin" to lazy="raise". The bidirectional
selectin defaults meant loading one Job pulled a large connected subgraph.
- Three further relationships (ExecutionAttempt.job_source,
ProcessingArtifact.execution_attempt, ProcessingArtifact.source) declared no
lazy at all and defaulted to "select", which raises MissingGreenlet under
async. These are now "raise" as well.
- Only 5 of 262 tests failed under the flip; the service layer already carried
explicit eager loads. Fixes went into the service queries, never back into
the models:
- PeopleService._finalize_link refreshes document, person, and role_ref so
the DocumentPerson write endpoints can still project them.
- JobService.update_job_state loads job_sources -> source so the Job it
returns still answers .error_detail and .filename.
- Two tests that bypassed the service layer now load explicitly.
- Audited every UI relationship access against its feeding service method; all
resolve to *_detail / list_*_detail variants with complete eager loads.
Tests
- Assert the composite and hot foreign key indexes exist in a fresh schema.
- Assert metadata.sorted_tables raises no SAWarning and orders source before
execution_attempt.
- Assert preferred_execution_attempt_id is a Uuid that compiles to UUID on
PostgreSQL and that its foreign key carries use_alter.
- Guard CRIT-02 from regression: no mapped relationship may declare a lazy
strategy outside {raise, noload}.
The development database was rebuilt from metadata rather than upgraded; the
previous file is retained out of tree as the Phase 8 migration source.
Verified: 266 passed, 4 skipped; ruff check clean.
Co-authored-by: Copilot App <[email protected]>
Pure remediation; no behavior change. Every item traces to a finding in
docs/architecture_code_review_2026-08-17.md.
Deletions
- Delete app_state.py, which had zero importers and whose get_session_factory
raised TypeError at runtime [HIGH-01].
- Delete services/transcription.py and point build_prompt_execution imports at
services/sources.py; drop the store.py compatibility aliases [MED-05].
- Delete ServiceBase.queue and its unparameterized asyncio.Queue [MED-07].
- Delete db/operations.get_next_queued_job, a divergent duplicate [CRIT-01].
- Drop the discarded load_docs parameter from list_jobs [LOW-03].
Config
- Delete worker_retry_backoff_seconds; no backoff behavior existed anywhere, so
wiring it would have been a new feature [MED-02].
- Wire sqlite_check_same_thread through get_engine. The engine hardcoded the
setting's own default, so this preserves behavior exactly [MED-02].
- Replace DATABASE_URL in docker-compose.yml with the nested DATABASE__DRIVER /
DATABASE__PATH names. Settings uses env_nested_delimiter with extra="ignore",
so DATABASE_URL was silently discarded [MED-10].
UI
- Move the 23KB inline VIBESCRIBE_LOGO_SVG to ui/static/vibescribe_logo.svg and
load it through a cached read_svg sibling of read_css [MED-09].
- Route the portrait upload failure through error_presenter.show_error [LOW-07].
- Cancel the job detail auto-refresh timer instead of only deactivating it, and
name its interval constant [LOW-06].
Worker
- Make WorkerNotifier runtime_checkable and validate the resolved object in
resolve_worker_notifier, which previously returned any non-None attribute
unchecked [LOW-04].
Docs and lint
- Fix two stale paths in services.instructions.md, one of which pointed at the
module deleted here [LOW-02].
- ruff check --fix to zero [LOW-01].
Verified: 264 passed, 4 skipped; ruff check clean.
Co-authored-by: Copilot App <[email protected]>
zoltan57andCopilot App