The pre-commit hooks declared `language: system` with bare `ruff`/`ty`
entries, but both are uv-managed dev dependencies and are not on PATH, so every
commit failed with `Executable 'ruff' not found`. Route both through
`uv run`; keep ruff blocking and make ty advisory (verbose) until its 18
whole-project diagnostics are cleared.
With the gate working, clear `ruff check .` to zero:
- 18 auto-fixes (import sorting, blank lines, `max()` simplification,
`with` merging, unused imports).
- Real defects: `SourceNavigation` annotated but never imported in
sources_page; two naive `datetime.now()` calls in migration.py now use
`datetime.now(UTC)`.
- Dead parameters removed: `source_has_photo_table` (computed, passed, never
read), `_serialize_value(key=...)`, and unused `request` on two NiceGUI
page handlers where the framework injects it optionally.
- Mechanical line-length wrapping and one `startswith` tuple collapse.
- `# noqa: PLR0915` / `# noqa: PLR1702` on five long UI/migration
functions, following the convention already used in jobs_page and
settings_page, rather than refactoring during stabilization.
Full suite green (377 tests, `-m "not external"`).
Co-authored-by: Copilot App <[email protected]>
The reviewer skill recorded four deterministic checks as unenforced or partial. Add tests so they fail the build instead of relying on a reviewer noticing.
tests/test_model_contract_guards.py:
- Status vocabulary: flags string literals compared against or assigned to status/purpose attributes, plus a narrower sweep that requires every status-valued literal in the package to be a known non-status use.
- Relationship loading: every Relationship must declare lazy='raise' except documented exceptions, and the exception set must match the Relationship Loading Contract in docs/schema.md.
- Schema fidelity: the Field-Accurate Table Contracts tables must match db/models.py on table coverage, field names, and declaration order, and the Authoritative Enumerations section must match the enum members.
tests/test_orphan_sweep.py:
- Locks the set of unreferenced public definitions. Route handlers registered by decorator are exempt, string entrypoint references count, and tests/ and tools/ count as consumers. KNOWN_ORPHANS records the four current orphans with rationale; a new one fails the build.
Each guard was mutation-tested: reverting the fix below, dropping a documented field, widening a lazy strategy, and adding a stranded function each fail their respective test.
Also fix the one violation the status guard found: sources_page.py compared attempt.status.value to the literal 'transcribed' instead of JobSourceStatus.TRANSCRIBED, which would survive an enum rename.
Co-authored-by: Copilot App <[email protected]>
Update the reviewer skill so its procedure matches how this repo actually works:
- Route review reports to docs/reviews/ and mark them non-canonical, resolving the conflict where reports landed in the same docs/ tree they resolve findings against.
- Pin verification commands to uv (uv run ruff check / ty check / pytest -m 'not external').
- Record the pytest contract: strict markers, strict asyncio mode, and the never-awaited-coroutine warning promoted to an error.
- Convert the deterministic checks to a table with an Enforced by column; three checks are unenforced and one only partial, which are now findings by construction.
- Add a consequence-based severity rubric and a Direction column for bidirectional drift.
- Escalate test-suite concerns to test-effectiveness-auditor.
Also fix tests/test_db.py, which was missing 'from sqlalchemy import text' while using it in 14 places. Three tests were failing with NameError. Wrapped the pre-existing long lines in the same file so it lints clean.
Document the deliberate nicegui==3.13.0 pin in pyproject.toml, a new runbook dependency upgrade policy, and the reviewer skill, so the pin is not flagged as a defect or widened as incidental cleanup.
Co-authored-by: Copilot App <[email protected]>
Adds .github/workflows/quality-gate.yml, running the gate on push and pull
request. CI invokes `pre-commit run --all-files` rather than restating the
`ruff check` and `ty check` commands, so the checks keep a single definition
in .pre-commit-config.yaml and local and CI cannot drift (plan task 2).
The workflow writes a .env file rather than exporting an environment
variable. The two are not equivalent here: Settings reads the .env file,
while the external-test skip guard reads os.getenv, so an exported variable
un-skips the external tests and sends them to the network. Measured in CI:
no .env gave 115 failures and 18 errors, an exported dummy key gave 3
failures, and a written .env file reproduced the local baseline exactly.
Negative-tested on a scratch branch: a deliberate lint error failed the run
at `ruff check` with exactly the planted errors, confirming the gate blocks
rather than merely reporting (plan task 4). The subsequent clean run passed
ruff and ty and reported 295 passed, 4 skipped, matching local and
confirming the four credential-gated tests skip cleanly (plan task 3).
That first green run caught a real platform-dependent defect. PromptStore
rejected non-direct-child names via `Path(name).name != name`, which is
platform-dependent: on POSIX a backslash is an ordinary filename character,
so "nested\prompt.md" passed the guard and failed later as NOT_FOUND rather
than VALIDATION. Windows cannot reproduce it. No traversal was possible,
since the path.parent != root check still held, so the impact was a wrong
error category and a red gate. Both separators are now rejected explicitly,
matching the ^[^/\\]+$ pattern config.PromptFilename already used.
Also exports docs/ver4.7/review_log_v4_7.md, the working record kept across
all six phases: 50 entries, 1 still open. The open entry is a pre-existing
/ui redirect defect found during the Phase 3 UI walk and deliberately left
unfixed as outside the V4.7 scope boundary.
Co-authored-by: Copilot App <[email protected]>
Review log [8]. classify_unexpected_error already returned retriable=False and
the verdict was logged and then thrown away. Measured across src/: retriable
was assigned in 9 places and read in none.
The plan asks for a test that a programming error "does not silently retry".
Probing with an injected AttributeError showed that is not what happens, and
the two real failure modes need different fixes.
Mode A, raised after the claim commits (inside advance_job): raised exactly
once, job left at PROCESSING, retry_count 0, never re-claimed, because
claim_next_queued_job filters status == QUEUED. A permanently stranded job
with one swallowed log line, not a retry. advance_job's PROCESSING branch,
commented "Recover mid-flight jobs", is unreachable from the worker for the
same reason.
Mode B, raised before or during the claim: 20 raises in 1.2s, an unbounded hot
spin at the poll interval. It never reaches the per-job retry machinery, so
WORKER_MAX_RETRIES does not cap it and the plan's 60s worst case understates
this path.
services/workflows.py
_advance_job_with_containment wraps advance_job. Any escaping exception is
classified and the job driven to terminal FAILED, which is visible in the UI
and resubmittable. The caller session is rolled back first and the terminal
write runs in its own transaction, so it stays atomic even when the failure
left that session dirty (plan task 3). The loop continues, so one poison job
cannot halt transcription for every other job.
worker.py
handle_worker_exceptions re-raises non-retriable faults rather than
suppressing them; retriable ones are still suppressed so transient
conditions do not stop work. run_worker_loop catches that, logs CRITICAL and
returns cleanly. Returning rather than propagating matters: the exception
would otherwise surface only at app shutdown, through the wait_for in
worker_consumer_lifespan.
tests
test_run_worker_loop_survives_process_next_exception asserted the loop
SURVIVES a RuntimeError and continues, which is the Mode B defect written
down as an expectation. Replaced by
test_run_worker_loop_stops_on_non_retriable_exception, with a new
test_run_worker_loop_survives_retriable_exception so suppression of genuinely
transient faults stays covered, and
test_error_after_claim_fails_the_job_instead_of_stranding_it for Mode A.
All three were verified to fail on pre-fix code. The Mode B guard fails by
timing out, which is the infinite spin made visible.
Verified: 295 passed, 4 skipped, 0 ruff, 0 ty.
Co-authored-by: Copilot App <[email protected]>
Review log [55]. Three historical local_timeout rows recorded 0.4-2.0s more
than the configured budget because the measurement window opened before the
provider call.
The plan named two causes, and both were already gone. Diffed against
f86c0ff~1: at V4.6 the window held resolve_provider_input (async;
normalization + artifact write + DB work) and a session.commit(). Phase 1
deleted both. What remains between the clock and the wait_for is
build_provider_input, now pure field copying because normalization moved to
ingest and file_hash is already stored: 6.2 us per call, zero awaits, so it
cannot yield to the event loop.
A third cause was still there and is not in the plan. The regression test
below measured 890ms where ~200ms was expected. services.sources.provider is
a lazy property that appears as an argument expression to _call_transcriber,
so it is evaluated after the clock starts but before wait_for begins timing.
Constructing OpenRouterTranscriptionProvider costs 475ms on first access and
0.001ms after, so the first attempt of every worker process booked half a
second of HTTP client construction as provider latency. That plausibly
accounts for the low end of the historical overshoot.
workflows.py
- Re-capture monotonic_started_at immediately before the wait_for, reusing
the same variable. The pre-loop assignment stays as the fallback: binding
a new name inside the try would leave the general-exception handler
referencing an unbound variable when build_provider_input raises. All
three duration write sites (success, TimeoutError, general failure) then
measure the correct window with no further change.
- Hoist the provider property above the per-source loop. It is
loop-invariant, so this also removes the repeated lookup from the two
evidence-capture sites.
tests/services/test_workflows_reliability.py
test_timeout_duration_excludes_pre_call_setup simulates 400ms of blocking
setup against a 200ms budget and asserts the recorded duration sits near
the budget and well clear of budget+setup. Confirmed to fail on the pre-fix
code (assert 625 < 540) and pass after, so it guards behaviour rather than
restating it. This is the plan's verification criterion as a test.
ui/pages/sources_page.py
_format_duration renders >=1s as "27.6 s" and below that as "612 ms",
replacing the raw "27612 ms". No test asserted the old format.
Plan task 3 (record preprocessing as its own value) declined and logged as a
deviation: after Phase 1 there is no preprocessing left to record, and a
preprocessing_ms column to measure 6 us of attribute copying is complexity
without a reader.
Verified: 293 passed, 4 skipped, 0 ruff, 0 ty.
Co-authored-by: Copilot App <[email protected]>
Decompose SourceService along the aggregate boundary and then correct the
instruction file that caused it to grow, in that order. The refactor is the
empirical test of the rule.
services/evidence.py (new)
EvidenceService owns ExecutionAttempt: read_latest_execution_attempt,
list_execution_attempts, promote_machine_attempt, build_evidence_export,
plus the LatestExecutionAttempt projection. Moved verbatim from sources.py.
services/errors.py (new)
The five-class error hierarchy (PromptLoadError, TranscriptionError,
TranscriptionNotFoundError, SourceDeleteBlockedError,
CandidatePromotionError) moved out of sources.py. evidence.py needs
TranscriptionNotFoundError, and test_service_boundaries.py correctly
rejected the sibling import. errors.py defines no *Service class, so it is
a legal shared home. This was the boundary test doing its job, not an
obstacle to route around.
sources.py 1,389 -> 885 lines (1,063 after Phase 2).
services/__init__.py
ServiceBundle and from_session_factory register evidence. Note that
field-by-field ServiceBundle construction silently binds services to the
process-global session factory via default_factory; from_session_factory is
the only safe constructor. Two test bundles were fixed for this.
.github/instructions/services.instructions.md
Rewritten to describe the boundaries the decomposition actually produced,
per plan Phase 3 task 7 and review log [59].
- "1 service class per data model" -> one service class per aggregate.
The table-shaped rule is the measured cause of sources.py reaching
1,389 lines; DocumentType has no lifecycle without Document.
- New Model Ownership section. Junctions are owned by their lifecycle
owner, the service that creates and deletes the rows: document_person
to PeopleService (sole writer, measured), job_source to SourceService.
Two carve-outs are stated rather than left as silent violations:
cascade deletion when a service deletes its own aggregate root, and
status transitions that create and delete nothing (cancel_job,
resubmit_failed_sources), which are Job lifecycle events on the work
queue. EvidenceService.promote_machine_attempt's two-field write to
Source is named and scoped.
- Mandatory CRUD softened to intent. It was already false: five modules
define no service class, EvidenceService has no create/delete because
ExecutionAttempt is append-only, RegistryService uses <op>_entry.
- Separated reading across models via eager loads from the owning root,
which is allowed, from importing another service, which is not. The old
line 13 and lines 75-77 read as contradictory.
- Typo: picutre.
No code was moved to satisfy the rule.
tests/test_service_boundaries.py
Docstring no longer cites the instruction file by line number; that anchor
would desynchronise silently. errors.py added to the neutral-module list.
Verified: 292 passed, 4 skipped, 0 ruff, 0 ty. All 25 /ui/* routes walked
against the live app; 24x 200. /ui/documents/{id}/sources 404s via a 307 that
drops the /ui prefix, confirmed pre-existing (last touched in 6a3ee26) and
left alone as out of scope.
Co-authored-by: Copilot App <[email protected]>
Move orientation normalization to the Source-ingest boundary and delete the
ProcessingArtifact subsystem it was built to serve.
Stored pages are now already upright, so nothing downstream derives a rotated
copy: every stored byte is the byte a provider is later sent. Rotation runs in
store_source_file ahead of hashing, so source.file_hash and file_size_bytes
describe exactly what is on disk. normalize_orientation becomes bytes-in /
bytes-out, and JPEG output reuses the source quantization tables and chroma
subsampling instead of re-quantizing at a fixed quality - measured at 50.3-56.1
dB PSNR at -6% size, against 50.0-53.5 dB at +38% for quality=95.
ProcessingArtifact held 2 rows against 77 successful transcriptions; the
subsystem effectively never ran. Deleting it removes the artifact cluster from
sources.py, the derivative resolution in workflows.py, the pre-provider commit
that only existed to make an artifact row durable, and the artifact evidence
dump from the Source detail page. The transcription_quality_warnings payload
folds into execution_attempt.normalized_metadata, so that feature keeps working
without the table.
tools/migrate_v46_to_v47.py carries steps 1 and 2: it rotated the 58 stored
images carrying EXIF orientation 3 in place, updated their recorded hash and
size, dropped processing_artifact and removed its one external file. It is
idempotent, keyed on state rather than a version marker.
tools/migrate_v45_to_v46.py is deleted. That migration is complete, and after
V4.7 it would restore a V4.5 backup into a schema that no longer matches.
Also fixes tests/test_config.py, which read the developer's local .env and
failed whenever WORKER_MAX_RETRIES was set.
Co-authored-by: Copilot App <[email protected]>
Preserves the traceability the V4.7 plan depends on ahead of starting
implementation in a fresh session. Documentation only.
The V4.6 review log was maintained in a session-scoped database and cited
by number throughout the V4.6, V4.7, and V4.8 planning documents as
"review log [N]". Those citations were unresolvable outside the session
that produced them. The log is now exported verbatim to
docs/ver4.6/review_log_v4_6.md: 70 entries, of which 8 remain open, each
mapped to its disposition (V4.7 phase, accepted risk, or operator
judgement).
The architecture review report is retained rather than deleted. It is the
canonical registry of the 32 finding IDs cited across six documents, so
removing it would orphan every CRIT/HIGH/MED/LOW reference in the planning
corpus. Instead it now carries a status banner marking it as a pre-V4.6
snapshot, warning that its paths, line numbers, and baseline metrics are
stale, recording that all 32 findings were dispositioned in V4.6 with only
MED-14 and HIGH-06 carrying into V4.7, and noting the two recommendations
later revised on evidence - the cancelled services/artifacts.py extraction
and the assumption that job_source and execution_attempt were
complementary rather than duplicated.
Co-authored-by: Copilot App <[email protected]>
Plans the next two releases following the V4.6 architecture remediation.
Documentation only - no code or schema changes.
V4.7 is an architectural cleanup and evidence-model re-alignment release,
scoped from measurements taken against the live database:
- job_source and execution_attempt duplicate the same evidence. Measured
77/77 identical on raw_transcription, ai_metadata vs normalized_metadata,
and raw_api_response vs sdk_response_snapshot. job_source is stripped to
its original junction role plus queue state (9 columns -> 4); all evidence
reads move to execution_attempt.
- job_source is stripped rather than deleted because it is also the work
queue: rows are created PENDING before any provider call, and cancellation
writes a terminal state with no provider call at all. An append-only
evidence table cannot express either.
- JobSourceStatus.CANCELLED is added so cancellation stops overloading
FAILED plus free text, which retires job_source.error_detail. This absorbs
the dual-spelling fix [45], since both rewrite the same persistence.
- ProcessingArtifact is removed. Two rows exist against 77 successful
transcriptions, so the subsystem has effectively never run. Orientation
normalization moves to ingest, where it is applied once and needs no
derivative.
- Orientation normalization itself is retained: 58 of 79 stored images carry
EXIF orientation 3, and their raw decoded pixels are genuinely inverted.
Rotation switches to quantization-table reuse, measured better than the
current quality=95 settings on both fidelity (51.5-55.0 dB PSNR vs
50.0-53.5) and size (-6% vs +38%).
- The planned services/artifacts.py extraction is cancelled. The cluster is
deleted rather than moved, establishing a delete-before-refactor ordering.
V4.8 parks feature work: pan and zoom, homepage gallery, multi-portrait
support, image descriptions, and the model-performance rollup, which stays
gated on the V4.7 run-time measurement fix.
Co-authored-by: Copilot App <[email protected]>
Adds tools/migrate_v45_to_v46.py, the final V4.6 deliverable.
Diffing the backup against the current SQLModel metadata showed that the
re-level changed no columns: both have the same 10 tables with identical
column sets. What changed is index coverage [HIGH-04], the use_alter break in
the source/execution_attempt foreign key cycle, and the relationship loading
strategy [CRIT-02]. The migration is therefore a faithful, foreign-key-ordered
row copy rather than a transformation.
Design:
- The backup is read with plain sqlite3 rather than through the ORM. The plan
anticipated ORM reads carrying explicit eager loads under lazy="raise";
raw reads are strictly safer, because the V4.5 file is not guaranteed to
satisfy the V4.6 mappers and no relationship is ever traversed.
- Writes go through SQLAlchemy Core against the live metadata, so the script
works unchanged against PostgreSQL when that cutover happens.
- source rows are inserted with preferred_execution_attempt_id cleared and the
selections are replayed after execution_attempt is populated, matching the
use_alter break in the cycle.
- _coerce() converts raw SQLite values into what each column binds. It accepts
both enum spellings, because job_source.status declares values_callable and
stores lowercase values while execution_attempt.status does not and stores
uppercase names, despite both using JobSourceStatus.
- Idempotent: a row whose primary key already exists is skipped, never
updated. Never invoked from application startup and never run by the test
suite.
- A pre-flight guard aborts if the backup row counts do not match the recorded
V4.5 snapshot, so the script cannot silently run against the wrong file.
Verification against a throwaway target:
- 282 rows copied; per-table counts match the plan exactly (document 8,
document_person 11, document_type 7, execution_attempt 80, job 11,
job_source 79, person 5, person_role 3, processing_artifact 2, source 76).
- Every table is cell-for-cell identical to the backup across all columns.
- A second run inserts 0 rows and skips all 282.
- Artifact integrity passes for every migrated artifact, checked through the
application's own SourceService verifier.
- 9 indexes added, 0 lost. No on-disk Source, portrait, or artifact file is
written by the script.
The live data/transcription.db is deliberately left untouched; it currently
holds only bootstrap seed rows whose UUIDs differ from the backup.
Co-authored-by: Copilot App <[email protected]>
Baseline was 207 diagnostics. Two real bugs were hiding in the noise:
- tools/run_destructive_tests.py imported ctypes.wintypes at module scope,
which raises on non-Windows, and called fcntl unconditionally. The Windows
and POSIX implementations now live under a module-level sys.platform split.
- tests/ui/test_sources_page.py constructed Source(...) without document_id.
Structural fixes, not suppressions:
- New src/transcription/db/loading.py owns the SQLModel-field to
QueryableAttribute reinterpretation via orm_attribute()/selectinload()/
defer(). This removed 42 "# pyright: ignore[reportArgumentType]" comments
across documents/jobs/people/sources. Its docstring records that
selectinload(A.b, B.c) is NOT equivalent to the chained form: varargs
applies the selectin strategy only to the last path element, which under
lazy="raise" raises InvalidRequestError at render time.
- db/session.py transaction_scope no longer accepts or yields
AsyncSessionTransaction. No caller ever passed one, sessionmaker.begin()
yields an AsyncSession, and the dead branch was latently buggy because
services call .exec(). Cleared 7 workflows.py diagnostics.
- services/registry.py RegistryService is bound by a new RegistryEntry
Protocol instead of bare SQLModel, so the shared implementation can read
id/label/normalized_label/is_active. Cleared 9 diagnostics.
- Column expressions in sources.py/jobs.py/test_store.py wrap in sqlmodel
col(), the idiom already used in registry.py.
- read_source_navigation wraps its literal tuple bounds in literal().
- normalization.py narrows with isinstance(image, TiffImageFile) rather than
comparing image.format, since tag_v2 is TIFF-only.
- linked_people.render uses @ui.refreshable_method, the NiceGUI API for bound
methods.
- The OpenRouter capturing client re-raises ResponseNotRead when the response
stream is not async rather than mis-wrapping it.
Tooling gate:
- New .pre-commit-config.yaml runs ruff check and ty check as blocking hooks.
No pre-commit config previously existed. Negative-tested: injecting a type
error fails both hooks.
- The last two "# pyright: ignore" comments (config.py) are removed; ty does
not honor pyright directives. One "# ty: ignore" remains, in
tests/test_prompts.py, where the test deliberately assigns to a frozen
field to assert ValidationError.
- asyncio_default_fixture_loop_scope is pinned to "function" so
pytest-asyncio behavior does not shift on upgrade.
Verification: ruff check clean, ty check reports 0 diagnostics, 292 passed
and 4 skipped, pre-commit passes and demonstrably fails on a regression, and
tools/run_destructive_tests.py runs on Windows.
Co-authored-by: Copilot App <[email protected]>
ui/homepage_store.py was the only storage path in the codebase derived from
Path(__file__).parents[3] rather than from Settings. That made it the one
storage root the operator could not relocate, and it resolved incorrectly
outside a source checkout - an installed distribution would have written
homepage content into the package directory in site-packages.
- config.py: add homepage_dir, defaulting to ./data/homepage so the location
is unchanged for anyone launching from the repository root.
- homepage_store.py: resolve the directory and markdown path from Settings,
with an optional settings parameter on every function so callers and tests
can override without patching module constants. HOME_PAGE_DIR and
HOME_PAGE_MARKDOWN_PATH constants are replaced by homepage_dir() and
homepage_markdown_path().
- tests/ui/test_homepage_store.py: covers the setting being honored, markdown
round-tripping, image storage and listing, and two configurations not
sharing storage.
Note: the default is now CWD-relative, matching artifact_dir and upload_dir,
rather than anchored to the repository root.
Verification: ruff check src tests clean; 292 passed, 4 skipped.
Co-authored-by: Copilot App <[email protected]>
MED-01 - move remaining blocking work off the event loop:
- normalization.py gains normalize_orientation_async; the Pillow decode,
transpose, and re-encode now run via asyncio.to_thread. The sync entry point
stays for tests and documents that it blocks.
- OrientationNormalization.digest_sha256 becomes a stored field computed inside
normalize_orientation, which already runs off-loop, instead of a property that
hashed page-sized derivative bytes on the caller's thread.
- SourceService._write_and_digest_artifact performs the artifact write and its
sha256 in a single worker-thread hop; both external-artifact write sites are
now dispatched through to_thread.
- transcribe_image dispatches load_source_payload and build_prompt_execution
through to_thread.
MED-04 - replace functools.cache on the engine and session factories with
explicit URL-keyed registries. dispose_engine and dispose_session_factory now
evict only the requested URL; previously cache_clear() tore down every other
database in the process, and dispose_engine would construct an engine for an
unknown URL purely to throw it away. New tests/test_engine_registry.py covers
distinct engines per URL, targeted eviction, and the unknown-URL no-op.
config.py - replace object.__setattr__ in normalize_provider_models with a
model_validator(mode="before") over the raw input, so the derived selector is
produced by normal construction rather than by mutating a frozen instance.
model_copy(update=...) was tried first and rejected: pydantic-settings does not
support a top-level validator returning anything other than self when validating
via __init__. provider_model is now stripped as well as the tuple entries.
models.py - add onupdate to the five updated_at columns and to Job.date_updated,
and drop the 10 manual "updated_at = datetime.now(UTC)" assignments across the
document, job, people, registry, and source services. Verified DDL-neutral by
hashing CreateTable output for every table on both the sqlite and postgresql
dialects before and after: identical, so this stays in Phase 6 and Phase 2 does
not need re-verification. New tests/services/test_timestamps.py asserts an
update through each service advances the timestamp.
MED-08 - Job.filename no longer swallows every exception to None. Relationships
declare lazy="raise", so the new _loaded_attribute helper inspects load state
explicitly and returns None only for genuinely unloaded attributes; real errors
now surface. Job.error_detail uses the same helper, which also removes its
unguarded read of the lazy="raise" job_sources relationship.
Verification: ruff check src tests clean; 288 passed, 4 skipped.
Co-authored-by: Copilot App <[email protected]>
Completes the table consolidation deferred within Phase 5. Section 4 of the
review lists table construction as a duplication pattern; three call sites had
been left bypassing the canonical builder.
- table/common.py: build_table gains a row_key parameter so callers with a
non-"id" primary key can use it.
- linked_people.py: replaces its hand-rolled ui.table with build_table
(row_key="person_id", selection="multiple", rows_per_page=0, no search).
- print_preview_page.py: _render_metadata_table and _render_job_table now share
a local _render_print_table helper. Print tables deliberately do not use
build_table - they must never paginate or render a search box, and they carry
print-only styling. The helper docstring records that rationale.
- tests/test_ui_boundaries.py: new AST guard asserting ui.table() is called from
exactly two modules - components/table/common.py and pages/print_preview_page.py.
Also closes the intermittent tests/ui/test_jobs_page.py failure observed once
after Phase 5 as environmental. Unreproduced across ~54 sequential full-suite
runs (including a 25-run soak) and 5 concurrent-process runs. No code change.
Verification: ruff check src tests clean; 275 passed, 4 skipped.
Co-authored-by: Copilot App <[email protected]>
Fixes the three ui.instructions.md violations recorded as [HIGH-07] and extracts
the page-level duplication catalogued in review section 4.
Boundary violations
- jobs_page no longer imports session_scope or manages a transaction.
store.create_document_job and store.create_job_for_document accept an optional
session_factory and open their own session scope when the caller supplies
neither a session nor a factory.
- sources_page no longer calls sqlalchemy.inspect. SourceService
.read_latest_execution_attempt now returns a LatestExecutionAttempt read model
carrying a plain transport_body_deferred flag, so ORM loader state stays inside
the service. Rendered output is unchanged.
- Deletes ui/components/document_panzoom.py, its export, and its CSS. The
component was exported but used by no page. Pan-zoom is planned for a clean
reintroduction in V4.7 alongside the other photo/image work.
Extracted duplication
- ui/components/media_urls.py: pure upload-URL resolution taking upload_dir and
base_url, replacing two identical ~60-line copies in sources_page and
people_page.
- ui/components/guards.py: parse-then-render-terminal-message, replacing 28
hand-written guard labels across five pages.
- ui/components/confirm_delete.py: the blocked-dependency notice and the
delete/cancel action row, from four delete pages.
- ui/components/upload_panel.py: the auto-uploading file picker, from three
pages. Source accept lists now derive from services.source_media
.SOURCE_EXTENSIONS instead of being hard-coded.
- ui/components/table/registry.py: the two hand-rolled label-registry tables on
the settings page now go through build_table, which gained selection and
rows_per_page options.
- ui/components/formatters.py gains parse_uuid and parse_iso_date, replacing
five and two private copies.
- ui/runtime.py owns resolve_runtime_settings, replacing three copies and
removing get_settings from every page module.
[LOW-05]
- Upload handlers are annotated with events.UploadEventArguments.
- The Document and Person form builders return DocumentFormFields and
PersonFormFields dataclasses instead of dict[str, Any].
Verification
- tests/test_ui_boundaries.py asserts no page imports a session scope, a session
factory, get_settings, sqlalchemy, or sqlmodel, and that no component imports
request or application state.
- 275 passed, 4 skipped. ruff check clean.
Findings: HIGH-07, LOW-05
Co-authored-by: Copilot App <[email protected]>
Removes the duplicated registry CRUD, the hand-written not-found raises, and
the three divergent media writers. Behavior is preserved: every existing
Document Type and Person Role test passes unchanged, which is the primary
proof for MED-11.
[MED-11] Generic registry service
- New services/registry.py owns RegistryService[ModelT]: list, list with
counts, create with IntegrityError -> conflict mapping, read, update,
delete with built-in and referenced guards, is_referenced, and label
normalization/casefold keying.
- DocumentTypeRegistry and PersonRoleRegistry declare only the model, error
class, noun, short noun, retainer phrase, and reference columns.
- DocumentService and PeopleService keep their public method names and
delegate. Every user-facing message, error category, and suggestion string
is reproduced verbatim; only the noun is templated.
- Deleted _normalize_registry_label, _document_type_label_key,
_normalize_role_label, _person_role_label_key,
_document_type_is_referenced, and _person_role_is_referenced.
[MED-12] Shared not-found lookup
- ServiceBase._get_or_raise(model, id, *, session, error, noun, suggestion,
options) loads by primary key or raises the caller's error type.
- documents.py: local _get_document_or_raise deleted; replaced by _read_document
and adopted at read_document, delete_document, and set_document_type, which
previously bypassed the helper and hand-wrote the raise.
- sources.py: 8 identical Source raises and 1 Job raise collapsed into
_read_source / _get_or_raise.
- jobs.py and people.py already funneled through local _not_found builders and
were left alone.
[MED-13][MED-01] Single media writer
- New services/media_storage.py owns validate -> name -> mkdir -> write ->
wrap OSError. The write runs in asyncio.to_thread, so uploads no longer block
the event loop.
- store_source_file, store_person_portrait, and store_homepage_image now share
it and are async. Callers in store.py, people_page.py, and home_page.py await
them. mkdir failures are now also translated to a domain error instead of
escaping as a raw OSError.
- homepage_store gains HomepageStorageError so its write reports like the others.
[MED-14, partial] Service independence
- New services/source_media.py owns SOURCE_MIME_TYPES, SOURCE_EXTENSIONS,
lookup_source_mime_type, and supported_source_formats.
- documents.py no longer imports services/sources.py. Its print projection uses
the non-raising lookup and raises DocumentError, so DocumentService no longer
emits a TranscriptionError.
- api/v4_print.py imports the mapping from the policy module.
- store.py and workflows.py still import sources.py; both are orchestration
modules, which services.instructions.md:75-77 explicitly permits.
- Splitting SourceService itself remains deferred to V4.7.
[LOW-08] Query shape
- list_sources_detail filters job_id with a JOIN on JobSource instead of
loading every Source and filtering in Python.
- read_source_navigation replaces the full ordered-id scan and .index() with
two row-value comparisons bounded by LIMIT 1.
- list_processing_artifacts gains the limit parameter its summary sibling
already had.
- build_evidence_export runs artifact integrity hashing and file reads through
asyncio.to_thread.
Tests
- tests/test_service_boundaries.py: AST guard asserting no service module
imports a sibling service module, plus a guard that the scan is non-empty.
- tests/services/test_transcription_service.py: asserts the job_id filter emits
a JOIN, and that navigation emits exactly two LIMIT queries.
- tests/services/test_store.py: the two storage tests are now async.
Verified: 276 passed, 4 skipped; ruff check clean.
Claim jobs atomically [CRIT-01]
- Replace JobService.read_next_queued_job with claim_next_queued_job, which
selects and transitions QUEUED -> PROCESSING inside one transaction. The old
read-then-write sequence left a window in which two workers could observe the
same QUEUED row.
- Add the missing .limit(1). The poll previously ordered the entire queued set
and discarded all but the first row.
- Drop the eager loads from the hot poll entirely. They were pure waste:
process_queued_job immediately re-reads the job through read_job with the
relationships it actually needs.
- Guard the row with with_for_update(skip_locked=True) on PostgreSQL so the
claim stays correct once more than one worker exists. On SQLite the claim is a
bounded single-writer transaction.
- Correct the comment at the remaining direct-call claim site, which described
the hazard rather than the guarantee.
Reuse the provider connection [HIGH-02]
- Build the ServiceBundle once per worker loop instead of once per job, and
close it at loop shutdown. Every job previously constructed a new
SourceService, and with it a new provider adapter and a new httpx.AsyncClient,
paying a full TLS handshake per page and discarding the connection pool.
- process_next_queued_job now accepts an optional caller-owned bundle and only
closes bundles it created itself.
Uncap the provider timeout [HIGH-03]
- Remove le=20.0 from worker_provider_timeout_seconds. The cap equalled the
default, so the ceiling could never be raised, and dense-page vision
transcription routinely needs longer. Default raised to 180s.
- Pass an explicit httpx.Timeout to the OpenRouter AsyncClient. httpx defaults
every phase to 5 seconds, so the real read budget was 5s regardless of the
configured value; the outer asyncio.wait_for could never be the binding
constraint. Connect stays at 10s.
Tighten the provider boundary [MED-03]
- Declare model, current_request_manifest, current_transport_evidence, and
aclose on the TranscriptionProvider Protocol.
- Delete the per-call inspect.signature(adapter.transcribe).parameters
reflection and the untyped kwargs dict it fed. The Protocol had declared
requested_model all along, so the reflection was dead defensive weight on the
hot path.
- Replace the three getattr probes for aclose and the evidence attributes with
direct typed access.
Deduplicate bundle construction [MED-06]
- Add ServiceBundle.from_session_factory and ServiceBundle.aclose, replacing the
duplicated four-service instantiation blocks in app.py and worker.py.
- _recover_stale_processing_jobs now uses the bundle built moments earlier
instead of constructing a second JobService.
Tests
- Claiming returns the oldest job, marks it PROCESSING, never hands the same job
out twice, and emits exactly one unadorned SELECT carrying LIMIT and no JOIN.
- The worker loop threads one bundle through consecutive jobs and closes it once
at shutdown; a caller-owned bundle is left open.
- Settings accepts a timeout above 20 seconds and still rejects zero.
- The OpenRouter client's read, write, and pool timeouts track the configured
budget rather than the httpx default.
Note: .env in this checkout still pins WORKER_PROVIDER_TIMEOUT_SECONDS=20 and
should be raised to pick up this fix.
Verified: 268 passed, 4 skipped; ruff check clean.
Co-authored-by: Copilot App <[email protected]>
These changes all regenerate the same schema, so they land together and revert
together. A partially applied schema pass is not a valid state.
Remove hand-rolled migrations [HIGH-05]
- Delete upgrade_schema and the _upgrade_person_family_search_id /
_upgrade_v42_evidence_tables / _upgrade_v45_selection_columns chain, plus the
two tests that exercised them. The DDL was SQLite-shaped raw SQL that would
not have run on PostgreSQL. create_all now derives everything from metadata
and remains gated by Settings.should_bootstrap_schema. No raw ALTER TABLE or
CREATE INDEX string remains in src.
Break the foreign key cycle [HIGH-08]
- Declare Source.preferred_execution_attempt_id with use_alter=True and an
explicit constraint name. source / job_source / execution_attempt formed an
unresolvable cycle that made metadata.sorted_tables emit an SAWarning and
order execution_attempt before source, which would have been a hard
create_all failure on PostgreSQL and was invisible on SQLite.
- As a side effect the column is now a dialect-aware Uuid rather than the
hardcoded CHAR(32) the raw upgrade DDL produced, so it emits native UUID on
PostgreSQL.
Index the hot filters [HIGH-04]
- Add composite Index("ix_job_status_date_created", "status", "date_created")
for the worker poll, and index the foreign keys the worker and detail pages
filter on: job.document_id, source.document_id, job_source.job_id,
job_source.source_id, document.document_type_id, and the three
document_person foreign keys.
Stop preloading by default [CRIT-02]
- Flip 16 relationships from lazy="selectin" to lazy="raise". The bidirectional
selectin defaults meant loading one Job pulled a large connected subgraph.
- Three further relationships (ExecutionAttempt.job_source,
ProcessingArtifact.execution_attempt, ProcessingArtifact.source) declared no
lazy at all and defaulted to "select", which raises MissingGreenlet under
async. These are now "raise" as well.
- Only 5 of 262 tests failed under the flip; the service layer already carried
explicit eager loads. Fixes went into the service queries, never back into
the models:
- PeopleService._finalize_link refreshes document, person, and role_ref so
the DocumentPerson write endpoints can still project them.
- JobService.update_job_state loads job_sources -> source so the Job it
returns still answers .error_detail and .filename.
- Two tests that bypassed the service layer now load explicitly.
- Audited every UI relationship access against its feeding service method; all
resolve to *_detail / list_*_detail variants with complete eager loads.
Tests
- Assert the composite and hot foreign key indexes exist in a fresh schema.
- Assert metadata.sorted_tables raises no SAWarning and orders source before
execution_attempt.
- Assert preferred_execution_attempt_id is a Uuid that compiles to UUID on
PostgreSQL and that its foreign key carries use_alter.
- Guard CRIT-02 from regression: no mapped relationship may declare a lazy
strategy outside {raise, noload}.
The development database was rebuilt from metadata rather than upgraded; the
previous file is retained out of tree as the Phase 8 migration source.
Verified: 266 passed, 4 skipped; ruff check clean.
Co-authored-by: Copilot App <[email protected]>
Pure remediation; no behavior change. Every item traces to a finding in
docs/architecture_code_review_2026-08-17.md.
Deletions
- Delete app_state.py, which had zero importers and whose get_session_factory
raised TypeError at runtime [HIGH-01].
- Delete services/transcription.py and point build_prompt_execution imports at
services/sources.py; drop the store.py compatibility aliases [MED-05].
- Delete ServiceBase.queue and its unparameterized asyncio.Queue [MED-07].
- Delete db/operations.get_next_queued_job, a divergent duplicate [CRIT-01].
- Drop the discarded load_docs parameter from list_jobs [LOW-03].
Config
- Delete worker_retry_backoff_seconds; no backoff behavior existed anywhere, so
wiring it would have been a new feature [MED-02].
- Wire sqlite_check_same_thread through get_engine. The engine hardcoded the
setting's own default, so this preserves behavior exactly [MED-02].
- Replace DATABASE_URL in docker-compose.yml with the nested DATABASE__DRIVER /
DATABASE__PATH names. Settings uses env_nested_delimiter with extra="ignore",
so DATABASE_URL was silently discarded [MED-10].
UI
- Move the 23KB inline VIBESCRIBE_LOGO_SVG to ui/static/vibescribe_logo.svg and
load it through a cached read_svg sibling of read_css [MED-09].
- Route the portrait upload failure through error_presenter.show_error [LOW-07].
- Cancel the job detail auto-refresh timer instead of only deactivating it, and
name its interval constant [LOW-06].
Worker
- Make WorkerNotifier runtime_checkable and validate the resolved object in
resolve_worker_notifier, which previously returned any non-None attribute
unchecked [LOW-04].
Docs and lint
- Fix two stale paths in services.instructions.md, one of which pointed at the
module deleted here [LOW-02].
- ruff check --fix to zero [LOW-01].
Verified: 264 passed, 4 skipped; ruff check clean.
Co-authored-by: Copilot App <[email protected]>
Jim LancasterandCopilot App