V4.1 revisions in preparation for v4.2. AI data capture now better defined.

2026-08-12 18:51:48 -05:00
parent 89cf69f8a2
commit 171132919d
15 changed files with 944 additions and 11 deletions
@@ -0,0 +1,176 @@
+# Raw API Response Q&A
+I initiated this exchange from within VSC using my Copilot Pro subscription. I believe the model was GPT-5.3-Codex, but I am not certain.
+
+## My prompt  
+```I can now see the ai_metadata and raw_api_response in the database. Help me understand what the two fields represent```  
+
+## AI Response
+Yes — in this project they are two different kinds of JSON evidence attached to a transcription attempt.
+
+- `ai_metadata` is a compact, normalized summary of the AI run.
+  - It is meant for operational/useful details such as:
+    - `finish_reason`
+    - token usage, such as `input_tokens`, `output_tokens`, and `total_tokens`
+  - It is not the full provider response; it is a curated summary for debugging and tracking.
+
+- `raw_api_response` is the full upstream provider response, stored as JSON-compatible data.
+  - This preserves whatever the AI provider returned so you can inspect it later.
+  - It can include things like model info, choices, message content, usage, and other provider-specific fields.
+
+A simple way to think about it:
+
+- `raw_transcription` = the text the system extracted from the AI response
+- `ai_metadata` = a summarized record of how the AI call ended and what it used
+- `raw_api_response` = the full original response payload from the provider
+
+You can see this in models.py, where both fields are stored on the job-source record, and in openrouter.py, where the provider builds `ai_metadata` from completion/usage details and stores the full response in `raw_api_response`.
+
+Example shape:
+
+```json
+ai_metadata = {
+  "finish_reason": "stop",
+  "usage": {
+    "input_tokens": 123,
+    "output_tokens": 45,
+    "total_tokens": 168
+  }
+}
+```
+
+```json
+raw_api_response = {
+  "id": "...",
+  "model": "google/gemini-2.5-flash",
+  "choices": [...],
+  "usage": {...}
+}
+```
+
+If you want, I can also help you interpret a specific row’s values from your database.  
+  
+## My Prompt  
+```  
+I was under the impression that there may be other information in the AI response, like:
+* AI's confidence level in the accuracy of the transcription--especially when inserting guesses (in square brackets)
+* The coordinates of the text box where the text was found on the screen
+
+Is this JSONB data returned by all AIs? Or some, but not all? Or none?  
+```  
+
+## AI Response  
+I’m checking the provider implementation and docs to answer this based on what this system actually captures rather than assuming a universal AI API format.
+
+Searched for text `confidence|bounding|bbox|coordinates|box|square bracket|guess` (`**/src/**`), 9 results
+
+Short answer: it depends on the provider and model, and in this codebase the app does not assume those fields exist.
+
+- The JSONB columns are generic containers, so they can hold extra provider-specific data if the API returns it. You can see that in base.py and models.py.
+- For the current OpenRouter integration, the app only extracts a small subset of information:
+  - `finish_reason`
+  - token usage
+  - the full provider response as raw JSON
+  That logic is in openrouter.py.
+
+So for your examples:
+
+- Confidence in transcription accuracy:
+  - Some providers/models may expose something like confidence, probability, or logits.
+  - Many do not.
+  - This app does not currently parse or surface that as a dedicated field.
+
+- Coordinates of detected text boxes:
+  - Some OCR/vision APIs return bounding boxes.
+  - Many chat-style LLM APIs do not.
+  - This app does not currently extract or normalize that data.
+
+In other words:
+- “Some, but not all” is the right answer.
+- “None” is not quite right for the raw JSON field, because if a provider returns those values, they would be stored there.
+- But “not guaranteed by this system” is the most accurate statement, since the current implementation does not request or interpret them as first-class fields.
+
+If you want, I can help sketch what would be needed to start capturing confidence or bounding-box data explicitly.
+
+## My Prompt
+```
+So if I understand you correctly, the current contents of "raw_api_response" is not actually the full contents of the AI response? If true, why is it being abbreviated?  
+```
+
+## AI Response
+Yes — that is the right interpretation.
+
+What is being stored in the current implementation is not a byte-for-byte copy of the provider’s original response. It is a best-effort, JSON-compatible representation of the provider response object after the code has tried to serialize it for the database. You can see that in openrouter.py, where the code runs through a serializer and then stores the result.
+
+Why it is abbreviated:
+
+- The provider SDK returns objects, not plain JSON.
+- The app needs something that can be stored in a JSONB column, so it converts nested objects into JSON-friendly data.
+- Some values are not serializable by default, so they are reduced to strings or simplified structures.
+- The code is not capturing the full transport-level HTTP payload, headers, status codes, or raw wire bytes.
+- The field is intended mainly for inspection and debugging, not as a perfect archival copy.
+
+So the more precise wording is:
+
+- `raw_api_response` = a normalized, JSON-friendly snapshot of the provider response
+- not = the full original upstream response verbatim
+
+If you want the truly full response, the system would need to capture the actual HTTP response body and metadata from the provider client rather than the SDK object after conversion.
+
+## My Prompt
+```  
+What are the pros/cons of capturing the actual HTTP response? Is there a long term benefit?  
+```
+
+## AI Response
+Yes — there can be a strong long-term benefit, but it comes with tradeoffs.
+
+Pros
+
+- Better debugging:
+  - You can inspect the exact provider payload when a transcription fails or behaves unexpectedly.
+  - This is especially helpful when a provider changes schema or returns unexpected fields.
+
+- Better reproducibility:
+  - You preserve the input/output pair for a given request, which can help compare model behavior over time.
+  - Useful for auditing, regression analysis, and support cases.
+
+- More complete evidence:
+  - You can capture status codes, headers, response bodies, and provider-specific metadata that the current normalized JSON may lose.
+
+- Future-proofing:
+  - If you later want to extract new fields such as confidence, bounding boxes, or token details, you may already have the raw payload available.
+
+Cons
+
+- Storage cost:
+  - Raw HTTP responses can be large, especially if you store full bodies, headers, and repeated requests.
+  - Over time this can grow database size and backup/restore cost.
+
+- Privacy and compliance:
+  - Some provider responses may include sensitive or regulated content.
+  - You need to think about retention policy and access controls.
+
+- Complexity:
+  - You need a clear schema or storage strategy for raw payloads.
+  - You may end up storing a mix of JSON, compressed blobs, or external objects.
+
+- Maintenance burden:
+  - Raw payloads can become noisy and hard to query.
+  - You may need separate indexes, retention rules, or archival storage.
+
+Long-term benefit
+
+- If your goal is traceability and model observability, yes, it is often worth it.
+- In a transcription system, especially one that depends on third-party AI providers, raw response capture can become valuable for:
+  - troubleshooting provider changes,
+  - investigating hallucinations or formatting regressions,
+  - building confidence/quality analytics later.
+
+A practical compromise is often:
+
+- store a compact normalized summary in `ai_metadata`,
+- store the full raw response only for selected requests or for a limited retention window,
+- and keep the raw payload in an external blob/object store rather than in the main relational table if volume becomes significant.
+
+So the short version is: capturing the actual HTTP response is usually a good investment if you care about long-term observability and auditability, but it should be designed carefully to avoid storage and privacy problems.
+