Commit Graph
235 Commits
Author SHA1 Message Date
zoltan57andCopilot App 3e418a0889 V4.6 Phase 2: schema re-level in a single atomic pass
These changes all regenerate the same schema, so they land together and revert
together. A partially applied schema pass is not a valid state.

Remove hand-rolled migrations [HIGH-05]
- Delete upgrade_schema and the _upgrade_person_family_search_id /
  _upgrade_v42_evidence_tables / _upgrade_v45_selection_columns chain, plus the
  two tests that exercised them. The DDL was SQLite-shaped raw SQL that would
  not have run on PostgreSQL. create_all now derives everything from metadata
  and remains gated by Settings.should_bootstrap_schema. No raw ALTER TABLE or
  CREATE INDEX string remains in src.

Break the foreign key cycle [HIGH-08]
- Declare Source.preferred_execution_attempt_id with use_alter=True and an
  explicit constraint name. source / job_source / execution_attempt formed an
  unresolvable cycle that made metadata.sorted_tables emit an SAWarning and
  order execution_attempt before source, which would have been a hard
  create_all failure on PostgreSQL and was invisible on SQLite.
- As a side effect the column is now a dialect-aware Uuid rather than the
  hardcoded CHAR(32) the raw upgrade DDL produced, so it emits native UUID on
  PostgreSQL.

Index the hot filters [HIGH-04]
- Add composite Index("ix_job_status_date_created", "status", "date_created")
  for the worker poll, and index the foreign keys the worker and detail pages
  filter on: job.document_id, source.document_id, job_source.job_id,
  job_source.source_id, document.document_type_id, and the three
  document_person foreign keys.

Stop preloading by default [CRIT-02]
- Flip 16 relationships from lazy="selectin" to lazy="raise". The bidirectional
  selectin defaults meant loading one Job pulled a large connected subgraph.
- Three further relationships (ExecutionAttempt.job_source,
  ProcessingArtifact.execution_attempt, ProcessingArtifact.source) declared no
  lazy at all and defaulted to "select", which raises MissingGreenlet under
  async. These are now "raise" as well.
- Only 5 of 262 tests failed under the flip; the service layer already carried
  explicit eager loads. Fixes went into the service queries, never back into
  the models:
  - PeopleService._finalize_link refreshes document, person, and role_ref so
    the DocumentPerson write endpoints can still project them.
  - JobService.update_job_state loads job_sources -> source so the Job it
    returns still answers .error_detail and .filename.
  - Two tests that bypassed the service layer now load explicitly.
- Audited every UI relationship access against its feeding service method; all
  resolve to *_detail / list_*_detail variants with complete eager loads.

Tests
- Assert the composite and hot foreign key indexes exist in a fresh schema.
- Assert metadata.sorted_tables raises no SAWarning and orders source before
  execution_attempt.
- Assert preferred_execution_attempt_id is a Uuid that compiles to UUID on
  PostgreSQL and that its foreign key carries use_alter.
- Guard CRIT-02 from regression: no mapped relationship may declare a lazy
  strategy outside {raise, noload}.

The development database was rebuilt from metadata rather than upgraded; the
previous file is retained out of tree as the Phase 8 migration source.

Verified: 266 passed, 4 skipped; ruff check clean.

Co-authored-by: Copilot App <[email protected]>
2026-08-17 16:06:30 -05:00
zoltan57andCopilot App 2ccea77520 V4.6 Phase 1: deletions and quick wins
Pure remediation; no behavior change. Every item traces to a finding in
docs/architecture_code_review_2026-08-17.md.

Deletions
- Delete app_state.py, which had zero importers and whose get_session_factory
  raised TypeError at runtime [HIGH-01].
- Delete services/transcription.py and point build_prompt_execution imports at
  services/sources.py; drop the store.py compatibility aliases [MED-05].
- Delete ServiceBase.queue and its unparameterized asyncio.Queue [MED-07].
- Delete db/operations.get_next_queued_job, a divergent duplicate [CRIT-01].
- Drop the discarded load_docs parameter from list_jobs [LOW-03].

Config
- Delete worker_retry_backoff_seconds; no backoff behavior existed anywhere, so
  wiring it would have been a new feature [MED-02].
- Wire sqlite_check_same_thread through get_engine. The engine hardcoded the
  setting's own default, so this preserves behavior exactly [MED-02].
- Replace DATABASE_URL in docker-compose.yml with the nested DATABASE__DRIVER /
  DATABASE__PATH names. Settings uses env_nested_delimiter with extra="ignore",
  so DATABASE_URL was silently discarded [MED-10].

UI
- Move the 23KB inline VIBESCRIBE_LOGO_SVG to ui/static/vibescribe_logo.svg and
  load it through a cached read_svg sibling of read_css [MED-09].
- Route the portrait upload failure through error_presenter.show_error [LOW-07].
- Cancel the job detail auto-refresh timer instead of only deactivating it, and
  name its interval constant [LOW-06].

Worker
- Make WorkerNotifier runtime_checkable and validate the resolved object in
  resolve_worker_notifier, which previously returned any non-None attribute
  unchecked [LOW-04].

Docs and lint
- Fix two stale paths in services.instructions.md, one of which pointed at the
  module deleted here [LOW-02].
- ruff check --fix to zero [LOW-01].

Verified: 264 passed, 4 skipped; ruff check clean.

Co-authored-by: Copilot App <[email protected]>
2026-08-17 16:06:08 -05:00
zoltan57 b3d8eb6e97 V4.6 Scope & Implementation Plan 2026-08-17 15:25:52 -05:00
zoltan57 1ee9ebbffc Created new code-review agent and ran it using Claude. 2026-08-17 14:39:52 -05:00
zoltan57 aec89b3a7a Hide V4.6 recommendations after code review 2026-08-17 12:45:03 -05:00
Jim Lancaster d1321fd709 V4.6 Code review: V4 architecture-conformance and reliability release 2026-08-16 09:38:59 -05:00
Jim Lancaster 7054cd8af9 V4.5 Complete - Enhanced trancription context, added option to restranscribe source under different models. 2026-08-16 09:06:56 -05:00
Jim Lancaster bdb1b31b0a V4.5 Scope defined 2026-08-16 00:13:34 -05:00
Jim Lancaster 5b97c759fe V4.4 revision to facsimile print format 2026-08-15 23:25:12 -05:00
Jim Lancaster 7db4df1729 V4.4 Complete 2026-08-15 14:30:33 -05:00
Jim Lancaster 63373bf24d V4.4 Scope defined (part 2) 2026-08-15 14:04:58 -05:00
Jim Lancaster 936af9b1d3 V4.4 Scope defined 2026-08-15 14:04:42 -05:00
Jim Lancaster a78b58ff40 V4.3 revision to Document Types 2026-08-15 13:29:53 -05:00
Jim Lancaster aed827babe Finalized v4.3 scope 2026-08-14 16:17:23 -05:00
Jim Lancaster 178347e086 Updated v4.3 scope 2026-08-14 16:08:07 -05:00
Jim Lancaster c9f5dca064 V4.2 complete 2026-08-14 15:59:38 -05:00
Jim Lancaster 6bd4cbb0a7 V4.2 Updated what ai_raw_response data is being captured. The changes were more extensive than I expected. 2026-08-14 07:21:15 -05:00
Jim Lancaster 28811d79ce V4.1 major revision to docs. Removed all obsolete documents, updated v4.2 implementation scope and plan. 2026-08-13 15:32:40 -05:00
Jim Lancaster 171132919d V4.1 revisions in preparation for v4.2. AI data capture now better defined. 2026-08-12 18:51:48 -05:00
Jim Lancaster 89cf69f8a2 V4.1 Mostly UI adjustments by GC 2026-08-12 13:11:50 -05:00
Jim Lancaster 1e8d8572d4 Continue GC code review: Pydantic 2026-08-12 01:35:50 -05:00
Jim Lancaster 888a8c380a Continue GC code review: UI 2026-08-11 16:54:03 -05:00
Jim Lancaster b8be27f0c9 Continue GC code review and cleanup 2026-08-11 16:42:09 -05:00
Jim Lancaster 8d5aec4301 Github Copilot service realignment & cleanup 2026-08-11 16:02:19 -05:00
Jim Lancaster 0ace10269f V4 implemented. Some tweaking left, but it is working 2026-08-11 12:07:19 -05:00
Jim Lancaster ccf2c78ff4 V4 final docs 2026-08-10 12:34:36 -05:00
Jim Lancaster 4b3baf5a3e Revised and simplified V4 Plan and core documents. 2026-08-10 10:53:13 -05:00
Jim Lancaster 9b4d6f0340 Delete V3 duplicates 2026-08-09 13:34:36 -05:00
Jim Lancaster 5753eb0135 V4 plan created: Adding many-to-many links between Documents & People in the UI. Also adding some new tables for Document Type, Person role. 2026-08-09 13:29:50 -05:00
Jim Lancaster e6549277c6 V3 Updated V3 core documents. Added data folder backup/restore before/after running destructive tests. 2026-08-09 10:51:30 -05:00
Jim Lancaster b59d3da23e V3 fix document delete issue 2026-08-08 21:58:19 -05:00
Jim Lancaster 4bf6c9e2f3 V3 post step 2 refinement: add temperature & top-p settings to config (and .env), add prompt fields back to job table so that the prompt settings get frozen at runtime for all sources being processed. 2026-08-08 18:21:44 -05:00
Jim Lancaster 4dac9349c1 V3 step 1 update models.py and step 2 implement service/worker, and raw API response persistence 2026-08-08 18:08:04 -05:00
Jim Lancaster 5a741de0a9 Updated documentation to v3 which will focus on capturing prompt/response interactions with AI. 2026-08-08 16:16:32 -05:00
Jim Lancaster 58faa00d7b AI metadata and api prompt results data capture now fixed 2026-08-08 14:59:45 -05:00
Jim Lancaster 89cac3c378 Removed the image viewer which wasn't working anyway. 2026-08-08 09:22:26 -05:00
Jim Lancaster 4b5e7ac23e Fixed Source Detail page 2026-08-08 08:21:35 -05:00
Jim Lancaster 21a7e83563 Tweak empty page settings to make them more uniform. 2026-08-07 09:10:03 -05:00
Jim Lancaster fce7107863 Remove all references to "uploads" page 2026-08-06 16:53:21 -05:00
Jim Lancaster 5090e238ff GLobal theme cleanup 2026-08-06 15:12:20 -05:00
Jim Lancaster 75f263c2b6 Unit testing fixed? So says Copilot 2026-08-05 19:52:40 -05:00
Jim Lancaster be152a028e Continue troubleshooting unit tests. I think it is time to let Copilot have a crack at it. 2026-08-05 18:33:44 -05:00
Jim Lancaster 9219adaf0c Fix unit test errors 2026-08-05 13:27:30 -05:00
Jim Lancaster fd3ca60008 Revamped the Documents, People, & Jobs too. 2026-08-05 13:05:41 -05:00
Jim Lancaster 72bc96ab3a Revamped Sources related pages with the help of Gemini, which had a lot to say. 2026-08-05 12:35:54 -05:00
Jim Lancaster 4eeb552273 Added Home page 2026-08-04 19:20:33 -05:00
Jim Lancaster d9f5fbb1a4 Delete Document & Delete Source buttons now delete the underlying files 2026-08-04 18:34:35 -05:00
Jim Lancaster 271633d1d5 Jobs: jobs still stuck in queue. Fixes from testing. 2026-08-04 18:09:31 -05:00
Jim Lancaster 6c6589d8ff Jobs: big jobs stuck in queue. Added Cancel, Resubmit 2026-08-04 09:03:51 -05:00
Jim Lancaster 759d4c2434 UI style refresh: Very close!!! 2026-08-03 19:46:32 -05:00