Compare commits

...

7 Commits

Author SHA1 Message Date
Enrico Ros 55bde68a4d Roll AIX 2026-05-05 04:17:39 -07:00
Enrico Ros 26ae3545a7 BlockOpUpstreamResume: full recovery. Fixes #1088 2026-05-05 04:14:00 -07:00
Enrico Ros 0001f7392b AIX: Gemini Interactions: relax 2026-05-05 03:32:13 -07:00
Enrico Ros d7e83e578b BlockOpUpstreamResume: remove cancel - unused? 2026-05-05 03:25:27 -07:00
Enrico Ros 901d93b5f0 LLMs/AIX: Gemini: Agentic models: recovery mode (non-streaming). Fixes #1088 2026-05-05 03:23:35 -07:00
Enrico Ros 6858b0b94a KB: LLMs: Gemini Interactions takeaways 2026-05-05 03:12:13 -07:00
Enrico Ros 9d88bf9b82 LLMs/AIX: Gemini: Agentic models: add option to disable visualizations. Fixes #1095 2026-05-05 03:06:30 -07:00
16 changed files with 503 additions and 92 deletions
+3
View File
@@ -21,6 +21,9 @@ Architecture and system documentation is available in the `/kb/` knowledge base,
- **[LLM-editorial-control.md](modules/LLM-editorial-pubdate.md)** - Where we have editorial control over per-model metadata vs dynamic discovery; `pubDate` field semantics, propagation chain, resolution rules, per-vendor matrix - **[LLM-editorial-control.md](modules/LLM-editorial-pubdate.md)** - Where we have editorial control over per-model metadata vs dynamic discovery; `pubDate` field semantics, propagation chain, resolution rules, per-vendor matrix
- **[LLM-models-catalog-pipeline.md](modules/LLM-models-catalog-pipeline.md)** - Forward-looking pipeline: extraction script, snapshot artifact, website consumption, future schema extensions - **[LLM-models-catalog-pipeline.md](modules/LLM-models-catalog-pipeline.md)** - Forward-looking pipeline: extraction script, snapshot artifact, website consumption, future schema extensions
#### LLM - Vendor APIs
- **[LLM-gemini-interactions.md](modules/LLM-gemini-interactions.md)** - Gemini Interactions API (Deep Research): endpoints, status taxonomy, two retrieval paths (SSE replay vs JSON GET), known failure modes (10-min cuts, zombies), UI surface
### Systems Documentation ### Systems Documentation
#### Core Platform Systems #### Core Platform Systems
+88
View File
@@ -0,0 +1,88 @@
# Gemini Interactions API
The Interactions API powers Gemini's agent runs (Deep Research today, more agent types planned). This doc is the source of truth for protocol shape, failure modes, and the recovery model — code comments link here instead of repeating the rationale.
## References
- **GH [#1088](https://github.com/enricoros/big-AGI/issues/1088)** — Auto-resume for Deep Research; Recover button
- **GH [#1095](https://github.com/enricoros/big-AGI/issues/1095)** — Visualizations toggle (`agent_config.visualization`)
- **Google forum [143098](https://discuss.ai.google.dev/t/interactions-api-connection-breaks-at-the-10-minutes-mark/143098)** — 10-min SSE cut
- **Google forum [143099](https://discuss.ai.google.dev/t/streaming-resume-broken-on-interactions-api-deep-research-often-cannot-resume/143099)** — Streaming resume re-cuts
- **Upstream specs** — `_upstream/gemini.interactions.spec.md`, `gemini.interactions.guide.md`, `gemini.deep-research.guide.md`
## Endpoints
| Verb | Path | Purpose |
|--------|-------------------------------------------|-------------------------------------------------------------------|
| POST | `/v1beta/interactions` | Start a run. We always send `stream:true, background:true, store:true` |
| GET | `/v1beta/interactions/{id}?stream=true` | Reattach via SSE replay (full event sequence from start) |
| GET | `/v1beta/interactions/{id}` | Fetch the resource as JSON (one-shot) |
| POST | `/v1beta/interactions/{id}/cancel` | Stop a background run |
| DELETE | `/v1beta/interactions/{id}` | Remove the stored record (does NOT cancel an in-flight run) |
Retention: 1 day free, 55 days paid.
## Status taxonomy
| Status | Meaning | Handling |
|-------------------|-----------------------------------------------|-------------------------------------------------------|
| `in_progress` | Live run **or** zombie (see C) | Surface diagnostics; offer Resume/Recover/Stop |
| `completed` | Done with content in `outputs[]` | Emit fragments, `tokenStopReason='ok'` |
| `failed` | Server-side failure | Terminating issue |
| `cancelled` | We or another client cancelled | Close as `cg-issue` |
| `incomplete` | Stopped early (token limit) — partial outputs | Note + `tokenStopReason='out-of-tokens'` |
| `requires_action` | Not expected for Deep Research | Fail loudly so we notice |
## Two retrieval paths
| Path | Endpoint | Parser | Use case |
|-----------------------|-----------------------------------|-------------------------------------------|-----------------------------------|
| SSE replay | `GET ?stream=true` | `createGeminiInteractionsParserSSE` | Canonical resume; live deltas |
| JSON GET (recovery) | `GET` (no `stream`) | `createGeminiInteractionsParserNS` | Recover when SSE is broken |
Both replay from the start — `ContentReassembler` REPLACES content on reattach, so partial replay (`last_event_id`) is intentionally NOT used. The NS parser walks `outputs[]` (thoughts, text, images, audio) and emits the same particles the SSE parser would, in one batch.
## Failure modes
### A. 10-minute SSE cut (forum 143098)
The SSE connection gets cut at exactly 600 s, regardless of activity. The cut is malformed (JSON error array instead of clean SSE close) and we treat it as stream-closed-early. The run typically **continues** server-side and reaches `completed`. **Recover (JSON GET)** retrieves the full report.
### B. Streaming resume re-cuts (forum 143099)
A fresh SSE replay can re-cut at the same 10-minute boundary on long runs, so Resume alone never reaches `interaction.complete`. **Recover** is the fallback.
### C. Zombie interactions (#1088)
Resource sits in `status: in_progress` for **days** with `outputs: []` — the generator crashed but the status never transitioned. **Not recoverable** (no data was ever produced). The NS parser surfaces `created`, `updated`, output count, and a "stuck for over an hour" hint so the user can decide to delete and retry.
### D. Connection drop mid-run
Network blip; resource is fine. **Resume (SSE replay)** picks up cleanly.
## UI
`BlockOpUpstreamResume` renders up to three buttons:
| Button | Action | Shown when |
|----------|-----------------------------------|---------------------------------------------------------|
| Resume | SSE replay | `onResume` provided |
| Recover | JSON GET (one-shot) | `upstreamHandle.uht``_NS_RECOVER_UHTS` |
| Stop | Cancel + delete upstream resource | `onDelete` provided |
The Recover gate is an inline `uht === 'vnd.gem.interactions'` check in `BlockOpUpstreamResume.tsx` — extend when another vendor needs the same fallback. Stop is intentionally NOT gated by Resume/Recover busy state — it's the escape hatch for hung resumes.
## Visualization control (#1095)
Deep Research accepts `agent_config.visualization: 'auto' | 'off'`. Exposed as `llmVndGeminiAgentViz` (label "Visualizations"). Forwarded only when explicitly `'off'` so the upstream `'auto'` default stays untouched. Useful when merging multiple reports — image fragments break Beam fusion.
## Code map
| File | Role |
|--------------------------------------------------------------------------------------|-------------------------------------------------------|
| `aix/server/dispatch/wiretypes/gemini.interactions.wiretypes.ts` | Zod schemas (RequestBody, Interaction, StreamEvent) |
| `aix/server/dispatch/chatGenerate/adapters/gemini.interactionsCreate.ts` | POST body (input + agent_config) |
| `aix/server/dispatch/chatGenerate/parsers/gemini.interactions.parser.ts` | SSE parser + NS parser |
| `aix/server/dispatch/chatGenerate/chatGenerate.dispatch.ts` (`gemini` case) | Resume dispatch: SSE vs JSON branch |
| `apps/chat/components/message/BlockOpUpstreamResume.tsx` | Resume / Recover / Stop UI |
| `apps/chat/components/ChatMessageList.tsx` (`handleMessageUpstreamResume`) | Wires click handler to `aixReattachContent_DMessage_orThrow` |
+52 -15
View File
@@ -6,6 +6,7 @@ import { Box, List } from '@mui/joy';
import type { SystemPurposeExample } from '../../../data'; import type { SystemPurposeExample } from '../../../data';
import type { AixReattachMode } from '~/modules/aix/client/aix.client';
import type { DiagramConfig } from '~/modules/aifn/digrams/DiagramsModal'; import type { DiagramConfig } from '~/modules/aifn/digrams/DiagramsModal';
import { speakText } from '~/modules/speex/speex.client'; import { speakText } from '~/modules/speex/speex.client';
@@ -123,7 +124,16 @@ export function ChatMessageList(props: {
} }
}, [conversationHandler, conversationId, onConversationExecuteHistory]); }, [conversationHandler, conversationId, onConversationExecuteHistory]);
const handleMessageUpstreamResume = React.useCallback(async (generator: DMessageGenerator, messageId: DMessageId) => {
// Resume in-flight tracking - lives at this level (NOT inside BlockOpUpstreamResume) so it
// survives any remount of the message bubble during a long-running stream (e.g. Deep Research).
// - `resumeInFlight` (state) drives the loading/Detach UI on BlockOpUpstreamResume via props.
// - `resumeAbortersRef` (ref) holds the AbortController so Detach can abort even after a remount.
// Map keyed by messageId so multiple messages could in principle resume concurrently.
const [resumeInFlight, setResumeInFlight] = React.useState<Record<DMessageId, AixReattachMode>>({});
const resumeAbortersRef = React.useRef<Map<DMessageId, AbortController>>(new Map());
const handleMessageUpstreamResume = React.useCallback(async (generator: DMessageGenerator, messageId: DMessageId, mode: AixReattachMode) => {
if (!conversationId || !conversationHandler) return; if (!conversationId || !conversationHandler) return;
if (!generator.upstreamHandle) throw new Error('No upstream handle on generator'); if (!generator.upstreamHandle) throw new Error('No upstream handle on generator');
@@ -131,20 +141,36 @@ export function ChatMessageList(props: {
const llmId = generator.mgt === 'aix' ? generator.aix.mId : undefined; const llmId = generator.mgt === 'aix' ? generator.aix.mId : undefined;
if (!llmId) throw new Error('No model id on generator'); if (!llmId) throw new Error('No model id on generator');
const controller = new AbortController();
resumeAbortersRef.current.set(messageId, controller);
setResumeInFlight(prev => ({ ...prev, [messageId]: mode }));
const { aixCreateChatGenerateContext, aixReattachContent_DMessage_orThrow } = await import('~/modules/aix/client/aix.client'); const { aixCreateChatGenerateContext, aixReattachContent_DMessage_orThrow } = await import('~/modules/aix/client/aix.client');
const result = await aixReattachContent_DMessage_orThrow( try {
llmId, await aixReattachContent_DMessage_orThrow(
generator, llmId,
aixCreateChatGenerateContext('conversation', conversationId), generator,
{ abortSignal: 'NON_ABORTABLE', throttleParallelThreads: 0 }, aixCreateChatGenerateContext('conversation', conversationId),
async (update, isDone) => { mode,
conversationHandler.messageEdit(messageId, { { abortSignal: controller.signal, throttleParallelThreads: 0 }, // Detach: aborting kills the local fetch; upstream run keeps going.
fragments: update.fragments, async (update, isDone) => {
generator: update.generator, conversationHandler.messageEdit(messageId, {
pendingIncomplete: update.pendingIncomplete, fragments: update.fragments,
}, isDone, isDone); // remove the pending state and updte only when done generator: update.generator,
}, pendingIncomplete: update.pendingIncomplete,
); }, isDone, isDone); // remove the pending state and update only when done
},
);
} finally {
// Clear local tracking only if this attempt is still the current one (avoid races on rapid retry)
if (resumeAbortersRef.current.get(messageId) === controller)
resumeAbortersRef.current.delete(messageId);
setResumeInFlight(prev => {
if (prev[messageId] !== mode) return prev;
const { [messageId]: _, ...rest } = prev;
return rest;
});
}
// Manual reattach is one-shot: on failure (e.g. upstream 404 from expired or already-consumed handle), // Manual reattach is one-shot: on failure (e.g. upstream 404 from expired or already-consumed handle),
// drop the upstreamHandle so the Resume button doesn't keep luring the user into the same error. // drop the upstreamHandle so the Resume button doesn't keep luring the user into the same error.
@@ -156,6 +182,11 @@ export function ChatMessageList(props: {
// }, false /* messageComplete */, true /* touch */); // }, false /* messageComplete */, true /* touch */);
}, [conversationHandler, conversationId]); }, [conversationHandler, conversationId]);
const handleMessageUpstreamDetach = React.useCallback((messageId: DMessageId) => {
resumeAbortersRef.current.get(messageId)?.abort();
}, []);
const handleMessageUpstreamDelete = React.useCallback(async (generator: DMessageGenerator, messageId: DMessageId) => { const handleMessageUpstreamDelete = React.useCallback(async (generator: DMessageGenerator, messageId: DMessageId) => {
if (!conversationId || !conversationHandler) return; if (!conversationId || !conversationHandler) return;
if (!generator.upstreamHandle) throw new Error('No upstream handle on generator'); if (!generator.upstreamHandle) throw new Error('No upstream handle on generator');
@@ -395,7 +426,11 @@ export function ChatMessageList(props: {
{filteredMessages.map((message, idx) => { {filteredMessages.map((message, idx) => {
// Optimization: only memo complete components, or we'd be memoizing garbage // Optimization: only memo complete components, or we'd be memoizing garbage (fragments
// change every chunk during streaming, so the equality check would always fail).
// CAVEAT: switching between memo and non-memo at the same position causes React to
// remount the subtree (different component types). Any state that must survive that
// boundary lives on this component (e.g. resumeInFlight, resumeAbortersRef).
const ChatMessageMemoOrNot = !message.pendingIncomplete ? ChatMessageMemo : ChatMessage; const ChatMessageMemoOrNot = !message.pendingIncomplete ? ChatMessageMemo : ChatMessage;
return props.isMessageSelectionMode ? ( return props.isMessageSelectionMode ? (
@@ -427,7 +462,9 @@ export function ChatMessageList(props: {
onMessageBranch={handleMessageBranch} onMessageBranch={handleMessageBranch}
onMessageContinue={handleMessageContinue} onMessageContinue={handleMessageContinue}
onMessageUpstreamResume={handleMessageUpstreamResume} onMessageUpstreamResume={handleMessageUpstreamResume}
onMessageUpstreamDetach={handleMessageUpstreamDetach}
onMessageUpstreamDelete={handleMessageUpstreamDelete} onMessageUpstreamDelete={handleMessageUpstreamDelete}
upstreamResumeMode={resumeInFlight[message.id]}
onMessageDelete={handleMessageDelete} onMessageDelete={handleMessageDelete}
onMessageFragmentAppend={handleMessageAppendFragment} onMessageFragmentAppend={handleMessageAppendFragment}
onMessageFragmentDelete={handleMessageDeleteFragment} onMessageFragmentDelete={handleMessageDeleteFragment}
@@ -2,9 +2,13 @@ import * as React from 'react';
import TimeAgo from 'react-timeago'; import TimeAgo from 'react-timeago';
import { Box, Button, ButtonGroup, Tooltip, Typography } from '@mui/joy'; import { Box, Button, ButtonGroup, Tooltip, Typography } from '@mui/joy';
import DownloadIcon from '@mui/icons-material/Download';
import LinkOffRoundedIcon from '@mui/icons-material/LinkOffRounded';
import PlayArrowRoundedIcon from '@mui/icons-material/PlayArrowRounded'; import PlayArrowRoundedIcon from '@mui/icons-material/PlayArrowRounded';
import StopRoundedIcon from '@mui/icons-material/StopRounded'; import StopRoundedIcon from '@mui/icons-material/StopRounded';
import type { AixReattachMode } from '~/modules/aix/client/aix.client';
import type { DMessageGenerator } from '~/common/stores/chat/chat.message'; import type { DMessageGenerator } from '~/common/stores/chat/chat.message';
@@ -12,54 +16,65 @@ const ARM_TIMEOUT_MS = 4000;
/** /**
* FIXME: COMPLETE THIS * Resume controls for an upstream-stored run.
* - Resume: SSE replay (live deltas) - canonical path. Always offered when onResume exists.
* - Recover: one-shot JSON GET - shown only for vendors that benefit from it (Gemini Interactions).
* - Detach: abort the local fetch but leave the upstream run alive. Visible only when a resume
* is in-flight (`inFlightMode != null`). Resume/Recover stay available afterwards.
* - Stop: terminate the upstream run + delete the resource.
*
* IMPORTANT: in-flight state is owned by the parent (`inFlightMode` + `onDetach`) so it survives
* remounts that happen while a long-running stream is active (e.g. Deep Research).
*/ */
export function BlockOpUpstreamResume(props: { export function BlockOpUpstreamResume(props: {
upstreamHandle: Exclude<DMessageGenerator['upstreamHandle'], undefined>, upstreamHandle: Exclude<DMessageGenerator['upstreamHandle'], undefined>,
pending?: boolean; // true while the message is actively streaming; labels the Delete button as "Stop" pending?: boolean; // true iff a local in-flight op (initial POST or resume); drives the state machine + hides the expiry footer
onResume?: () => void | Promise<void>; inFlightMode?: AixReattachMode; // set by the parent while a resume is in flight; drives the loading/Detach UI
onCancel?: () => void | Promise<void>; onResume?: (mode: AixReattachMode) => void | Promise<void>;
onDetach?: () => void;
onDelete?: () => void | Promise<void>; onDelete?: () => void | Promise<void>;
}) { }) {
// state // local state - only for short-lived ops the parent doesn't own
const [isResuming, setIsResuming] = React.useState(false);
const [isCancelling, setIsCancelling] = React.useState(false);
const [isDeleting, setIsDeleting] = React.useState(false); const [isDeleting, setIsDeleting] = React.useState(false);
const [deleteArmed, setDeleteArmed] = React.useState(false); const [deleteArmed, setDeleteArmed] = React.useState(false);
const [error, setError] = React.useState<string | null>(null); const [error, setError] = React.useState<string | null>(null);
// expiration: boolean is evaluated at render (may lag briefly if nothing re-renders past expiry). // expiration: boolean is evaluated at render (may lag briefly if nothing re-renders past expiry).
// TimeAgo handles its own tick for the label; the button's disabled state is the only consumer of this flag.
const { expiresAt /*, runId = ''*/ } = props.upstreamHandle; const { expiresAt /*, runId = ''*/ } = props.upstreamHandle;
// const isExpired = expiresAt != null && Date.now() > expiresAt;
// State machine - mutually exclusive triplet (idle | initial-POST | resume | recover):
// - Idle : !pending - run not active locally (incl. post-reload, since
// chats.converters.ts clears pendingIncomplete on hydrate).
// - Initial POST : pending && !inFlightMode - first generation streaming.
// - Resume replay : pending && mode='replay' - we own this resume cycle.
// - Recover snap : pending && mode='snapshot' - we own this snapshot fetch.
//
// Visibility matrix (see BlockOpUpstreamResume props doc):
// Resume Recover Detach Cancel
// Idle ✅ ✅¹ — ✅
// Initial POST — — — ✅
// Resume in flight — — ✅ ✅
// Recover in flight — ✅² — —
// ¹ only for Gemini Interactions ² with loading spinner
const isReplaying = props.inFlightMode === 'replay';
const isSnapshotting = props.inFlightMode === 'snapshot';
const isIdle = !props.pending;
const canRecoverVendor = props.upstreamHandle.uht === 'vnd.gem.interactions';
const showResume = isIdle && !!props.onResume;
const showRecover = (isIdle || isSnapshotting) && !!props.onResume && canRecoverVendor;
const showDetach = isReplaying && !!props.onDetach;
const showCancel = !isSnapshotting && !!props.onDelete;
// handlers // handlers
const handleResume = React.useCallback(async () => { const handleResume = React.useCallback((mode: AixReattachMode) => {
if (!props.onResume) return; if (!props.onResume) return;
setError(null); setError(null);
setIsResuming(true); // fire-and-forget: parent owns the promise lifecycle and the abort controller.
try { // If it rejects, the parent surfaces the error via its own UI; we stay silent.
await props.onResume(); Promise.resolve(props.onResume(mode)).catch(() => { /* parent handles */ });
} catch (err: any) {
setError(err?.message || 'Resume failed');
} finally {
setIsResuming(false);
}
}, [props]);
const handleCancel = React.useCallback(async () => {
if (!props.onCancel) return;
setError(null);
setIsCancelling(true);
try {
await props.onCancel();
} catch (err: any) {
setError(err?.message || 'Cancel failed');
} finally {
setIsCancelling(false);
}
}, [props]); }, [props]);
// Two-click arm: first click arms (visible red "Confirm?"), second click (within ARM_TIMEOUT_MS) executes. // Two-click arm: first click arms (visible red "Confirm?"), second click (within ARM_TIMEOUT_MS) executes.
@@ -88,7 +103,6 @@ export function BlockOpUpstreamResume(props: {
return () => clearTimeout(t); return () => clearTimeout(t);
}, [deleteArmed]); }, [deleteArmed]);
return ( return (
<Box <Box
sx={{ sx={{
@@ -100,43 +114,55 @@ export function BlockOpUpstreamResume(props: {
}} }}
> >
<ButtonGroup> <ButtonGroup>
{props.onResume && ( {showResume && (
<Tooltip title='Resume generation from last checkpoint'> <Tooltip title='Resume by re-streaming from the upstream run'>
<Button <Button
disabled={isResuming || isCancelling || isDeleting} disabled={isDeleting}
loading={isResuming}
startDecorator={<PlayArrowRoundedIcon color='success' />} startDecorator={<PlayArrowRoundedIcon color='success' />}
onClick={handleResume} onClick={() => handleResume('replay')}
> >
Resume Resume
</Button> </Button>
</Tooltip> </Tooltip>
)} )}
{props.onCancel && ( {showRecover && (
<Tooltip title='Cancel the response generation'> <Tooltip title='Fetch the result without streaming - recovers stuck or hung runs'>
<Button <Button
disabled={isResuming || isCancelling || isDeleting} disabled={isDeleting}
loading={isCancelling} loading={isSnapshotting}
// startDecorator={<CancelIcon />} loadingPosition='start'
onClick={handleCancel} startDecorator={<DownloadIcon />}
onClick={() => handleResume('snapshot')}
> >
Cancel Recover
</Button> </Button>
</Tooltip> </Tooltip>
)} )}
{props.onDelete && ( {showDetach && (
<Tooltip title={deleteArmed ? 'Click again to confirm - cancels the run upstream (no resume after)' : (props.pending ? 'Stop this response and cancel the upstream run' : 'Cancel the upstream run')}> <Tooltip title='Close this connection only - the upstream run keeps going. Click Resume or Recover later to fetch results.'>
<Button
disabled={isDeleting}
startDecorator={<LinkOffRoundedIcon />}
onClick={props.onDetach}
>
Detach
</Button>
</Tooltip>
)}
{showCancel && (
<Tooltip title={deleteArmed ? 'Click again to confirm - cancels the upstream run and clears the handle' : 'Cancel the upstream run'}>
<Button <Button
loading={isDeleting} loading={isDeleting}
color={deleteArmed ? 'danger' : 'neutral'} color={deleteArmed ? 'danger' : 'neutral'}
variant={deleteArmed ? 'solid' : 'outlined'} variant={deleteArmed ? 'solid' : 'outlined'}
startDecorator={<StopRoundedIcon />} startDecorator={<StopRoundedIcon />}
onClick={handleDelete} onClick={handleDelete}
disabled={isCancelling || isDeleting} disabled={isDeleting}
> >
{deleteArmed ? 'Confirm?' : (props.pending ? 'Stop' : 'Cancel')} {deleteArmed ? 'Confirm?' : 'Cancel'}
</Button> </Button>
</Tooltip> </Tooltip>
)} )}
@@ -29,6 +29,7 @@ import VerticalAlignBottomIcon from '@mui/icons-material/VerticalAlignBottom';
import VisibilityIcon from '@mui/icons-material/Visibility'; import VisibilityIcon from '@mui/icons-material/Visibility';
import VisibilityOffIcon from '@mui/icons-material/VisibilityOff'; import VisibilityOffIcon from '@mui/icons-material/VisibilityOff';
import type { AixReattachMode } from '~/modules/aix/client/aix.client';
import { ModelVendorAnthropic } from '~/modules/llms/vendors/anthropic/anthropic.vendor'; import { ModelVendorAnthropic } from '~/modules/llms/vendors/anthropic/anthropic.vendor';
import { AnthropicIcon } from '~/common/components/icons/vendors/AnthropicIcon'; import { AnthropicIcon } from '~/common/components/icons/vendors/AnthropicIcon';
@@ -161,8 +162,10 @@ export function ChatMessage(props: {
onMessageBeam?: (messageId: string) => Promise<void>, onMessageBeam?: (messageId: string) => Promise<void>,
onMessageBranch?: (messageId: string) => void, onMessageBranch?: (messageId: string) => void,
onMessageContinue?: (messageId: string, continueText: null | string) => void, onMessageContinue?: (messageId: string, continueText: null | string) => void,
onMessageUpstreamResume?: (generator: DMessageGenerator, messageId: string) => Promise<void>, onMessageUpstreamResume?: (generator: DMessageGenerator, messageId: string, mode: AixReattachMode) => Promise<void>,
onMessageUpstreamDetach?: (messageId: string) => void,
onMessageUpstreamDelete?: (generator: DMessageGenerator, messageId: string) => Promise<void>, onMessageUpstreamDelete?: (generator: DMessageGenerator, messageId: string) => Promise<void>,
upstreamResumeMode?: AixReattachMode, // set by parent while a resume is in flight on this message
onMessageDelete?: (messageId: string) => void, onMessageDelete?: (messageId: string) => void,
onMessageFragmentAppend?: (messageId: DMessageId, fragment: DMessageFragment) => void onMessageFragmentAppend?: (messageId: DMessageId, fragment: DMessageFragment) => void
onMessageFragmentDelete?: (messageId: DMessageId, fragmentId: DMessageFragmentId) => void, onMessageFragmentDelete?: (messageId: DMessageId, fragmentId: DMessageFragmentId) => void,
@@ -247,7 +250,7 @@ export function ChatMessage(props: {
// const wordsDiff = useWordsDifference(textSubject, props.diffPreviousText, showDiff); // const wordsDiff = useWordsDifference(textSubject, props.diffPreviousText, showDiff);
const { onMessageAssistantFrom, onMessageDelete, onMessageFragmentAppend, onMessageFragmentDelete, onMessageFragmentReplace, onMessageContinue, onMessageUpstreamResume, onMessageUpstreamDelete } = props; const { onMessageAssistantFrom, onMessageDelete, onMessageFragmentAppend, onMessageFragmentDelete, onMessageFragmentReplace, onMessageContinue, onMessageUpstreamResume, onMessageUpstreamDetach, onMessageUpstreamDelete } = props;
const handleFragmentNew = React.useCallback(() => { const handleFragmentNew = React.useCallback(() => {
onMessageFragmentAppend?.(messageId, createTextContentFragment('')); onMessageFragmentAppend?.(messageId, createTextContentFragment(''));
@@ -265,11 +268,15 @@ export function ChatMessage(props: {
onMessageContinue?.(messageId, continueText); onMessageContinue?.(messageId, continueText);
}, [messageId, onMessageContinue]); }, [messageId, onMessageContinue]);
const handleUpstreamResume = React.useCallback(() => { const handleUpstreamResume = React.useCallback((mode: AixReattachMode) => {
if (!messageGenerator) return; if (!messageGenerator) return;
return onMessageUpstreamResume?.(messageGenerator, messageId); return onMessageUpstreamResume?.(messageGenerator, messageId, mode);
}, [messageGenerator, messageId, onMessageUpstreamResume]); }, [messageGenerator, messageId, onMessageUpstreamResume]);
const handleUpstreamDetach = React.useCallback(() => {
onMessageUpstreamDetach?.(messageId);
}, [messageId, onMessageUpstreamDetach]);
const handleUpstreamDelete = React.useCallback(() => { const handleUpstreamDelete = React.useCallback(() => {
if (!messageGenerator) return; if (!messageGenerator) return;
return onMessageUpstreamDelete?.(messageGenerator, messageId); return onMessageUpstreamDelete?.(messageGenerator, messageId);
@@ -903,7 +910,9 @@ export function ChatMessage(props: {
<BlockOpUpstreamResume <BlockOpUpstreamResume
upstreamHandle={messageGenerator.upstreamHandle} upstreamHandle={messageGenerator.upstreamHandle}
pending={messagePendingIncomplete} pending={messagePendingIncomplete}
onResume={(!messagePendingIncomplete && onMessageUpstreamResume) ? handleUpstreamResume : undefined} inFlightMode={props.upstreamResumeMode}
onResume={onMessageUpstreamResume ? handleUpstreamResume : undefined}
onDetach={onMessageUpstreamDetach ? handleUpstreamDetach : undefined}
onDelete={onMessageUpstreamDelete ? handleUpstreamDelete : undefined} onDelete={onMessageUpstreamDelete ? handleUpstreamDelete : undefined}
/> />
)} )}
+1 -1
View File
@@ -23,7 +23,7 @@ export const Release = {
// this is here to trigger revalidation of data, e.g. models refresh // this is here to trigger revalidation of data, e.g. models refresh
Monotonics: { Monotonics: {
Aix: 69, Aix: 70,
NewsVersion: 204, NewsVersion: 204,
}, },
@@ -349,6 +349,15 @@ export const DModelParameterRegistry = {
// when undefined, the model chooses automatically // when undefined, the model chooses automatically
}, },
// Gemini Interactions API agent_config - per-agent knobs (Deep Research only today)
llmVndGeminiAgentViz: _enumDef({
label: 'Visualizations',
type: 'enum',
description: 'Charts and images in Deep Research reports. Disable for text-only output (helpful when merging multiple reports).',
values: ['auto', 'off'],
// undefined means upstream default ('auto'); we only forward when explicitly 'off'
}),
// NOTE: we don't have this as a parameter, as for now we use it in tandem with llmVndGeminiGoogleSearch // NOTE: we don't have this as a parameter, as for now we use it in tandem with llmVndGeminiGoogleSearch
// llmVndGeminiUrlContext: { // llmVndGeminiUrlContext: {
// label: 'URL Context', // label: 'URL Context',
+19 -10
View File
@@ -70,7 +70,7 @@ export function aixCreateModelFromLLMOptions(
llmVndAntEffort, llmVndGemEffort, llmVndOaiEffort, llmVndMiscEffort, llmVndAntEffort, llmVndGemEffort, llmVndOaiEffort, llmVndMiscEffort,
llmVndAnt1MContext, llmVndAntInfSpeed, llmVndAntSkills, llmVndAntThinkingBudget, llmVndAntWebDynamic, llmVndAntWebFetch, llmVndAntWebFetchMaxUses, llmVndAntWebSearch, llmVndAntWebSearchMaxUses, llmVndAnt1MContext, llmVndAntInfSpeed, llmVndAntSkills, llmVndAntThinkingBudget, llmVndAntWebDynamic, llmVndAntWebFetch, llmVndAntWebFetchMaxUses, llmVndAntWebSearch, llmVndAntWebSearchMaxUses,
llmVndBedrockAPI, llmVndBedrockAPI,
llmVndGeminiAspectRatio, llmVndGeminiImageSize, llmVndGeminiCodeExecution, llmVndGeminiComputerUse, llmVndGeminiGoogleSearch, llmVndGeminiMediaResolution, llmVndGeminiThinkingBudget, llmVndGeminiAgentViz, llmVndGeminiAspectRatio, llmVndGeminiImageSize, llmVndGeminiCodeExecution, llmVndGeminiComputerUse, llmVndGeminiGoogleSearch, llmVndGeminiMediaResolution, llmVndGeminiThinkingBudget,
// llmVndMoonshotWebSearch, // llmVndMoonshotWebSearch,
llmVndOaiRestoreMarkdown, llmVndOaiVerbosity, llmVndOaiWebSearchContext, llmVndOaiWebSearchGeolocation, llmVndOaiImageGeneration, llmVndOaiCodeInterpreter, llmVndOaiRestoreMarkdown, llmVndOaiVerbosity, llmVndOaiWebSearchContext, llmVndOaiWebSearchGeolocation, llmVndOaiImageGeneration, llmVndOaiCodeInterpreter,
llmVndOrtWebSearch, llmVndOrtWebSearch,
@@ -143,6 +143,7 @@ export function aixCreateModelFromLLMOptions(
// Gemini // Gemini
...(llmVndGeminiInteractions ? { vndGeminiAPI: 'interactions-agent' } : {}), ...(llmVndGeminiInteractions ? { vndGeminiAPI: 'interactions-agent' } : {}),
...(llmVndGeminiAgentViz === 'off' ? { vndGeminiAgentViz: 'off' } : {}), // Deep Research agent_config.visualization - only forward when explicitly disabled
...(llmVndGeminiAspectRatio ? { vndGeminiAspectRatio: llmVndGeminiAspectRatio } : {}), ...(llmVndGeminiAspectRatio ? { vndGeminiAspectRatio: llmVndGeminiAspectRatio } : {}),
...(llmVndGeminiCodeExecution === 'auto' ? { vndGeminiCodeExecution: llmVndGeminiCodeExecution } : {}), ...(llmVndGeminiCodeExecution === 'auto' ? { vndGeminiCodeExecution: llmVndGeminiCodeExecution } : {}),
...(llmVndGeminiComputerUse ? { vndGeminiComputerUse: llmVndGeminiComputerUse } : {}), ...(llmVndGeminiComputerUse ? { vndGeminiComputerUse: llmVndGeminiComputerUse } : {}),
@@ -644,22 +645,30 @@ function _finalizeLlmMetricsWithCosts(cgMetricsLg: undefined | DMetricsChatGener
// --- L2 - Content Generation reattachment as DMessage --- // --- L2 - Content Generation reattachment as DMessage ---
/**
* Reattach mode selects how to reconstruct an in-progress upstream run:
* - 'replay' - canonical: SSE replays the event sequence from the start. Live deltas reach
* the UI as the run progresses (or as past content is replayed).
* - 'snapshot' - one-shot JSON GET returns the resource as-is right now. Used to recover when
* the SSE endpoint is broken upstream but the resource itself is still readable.
*
* Names describe what you get, not how. See `kb/modules/LLM-gemini-interactions.md` for failure modes.
*/
export type AixReattachMode = 'replay' | 'snapshot';
/** /**
* Reattach facade: wraps `aixChatGenerateContent_DMessage_orThrow` for the reattach-to-upstream flow. * Reattach facade: wraps `aixChatGenerateContent_DMessage_orThrow` for the reattach-to-upstream flow.
* - Validates the generator carries an `upstreamHandle`
* - Stubs the unused chat-generate request, and
* - Seeds the base function so the LL's reattach branch fires.
* *
* On an in-progress upstream run (Gemini Deep Research today, extensible to OAI Responses), the server * The reassembler replaces content on reattach (Gemini Interactions snapshots are cumulative, so this rebuilds from scratch).
* just needs the handle to GET-poll; no chat-generate body is needed. This facade:
* - validates the generator carries an `upstreamHandle`,
* - stubs the chat-generate request (unused on the reattach path - the server uses the handle),
* - seeds the base function via `clientOptions.reattachGenerator` so the LL's reattach branch fires.
*
* The reassembler starts with empty fragments; since Gemini Interactions snapshots are cumulative,
* the stream will rebuild the complete content from scratch. Any partial content from the original run is replaced.
*/ */
export async function aixReattachContent_DMessage_orThrow( export async function aixReattachContent_DMessage_orThrow(
llmId: DLLMId, llmId: DLLMId,
reattachGenerator: Readonly<DMessageGenerator>, reattachGenerator: Readonly<DMessageGenerator>,
aixContext: AixAPI_Context_ChatGenerate, aixContext: AixAPI_Context_ChatGenerate,
mode: AixReattachMode,
clientOptions: Pick<AixClientOptions, 'abortSignal' | 'throttleParallelThreads'>, clientOptions: Pick<AixClientOptions, 'abortSignal' | 'throttleParallelThreads'>,
onStreamingUpdate?: (update: AixChatGenerateContent_DMessageGuts, isDone: boolean) => MaybePromise<void>, onStreamingUpdate?: (update: AixChatGenerateContent_DMessageGuts, isDone: boolean) => MaybePromise<void>,
): Promise<_AixChatGenerateContent_DMessageGuts_WithOutcome> { ): Promise<_AixChatGenerateContent_DMessageGuts_WithOutcome> {
@@ -674,7 +683,7 @@ export async function aixReattachContent_DMessage_orThrow(
llmId, llmId,
stubChatGenerate, stubChatGenerate,
aixContext, aixContext,
true, // streaming mode === 'replay', // wire-level: SSE demuxer (replay) vs one-shot JSON body (snapshot)
{ ...clientOptions, reattachGenerator: reattachGenerator as any /* guaranteed by the check */ }, { ...clientOptions, reattachGenerator: reattachGenerator as any /* guaranteed by the check */ },
onStreamingUpdate, onStreamingUpdate,
); );
@@ -516,6 +516,7 @@ export namespace AixWire_API {
// Gemini // Gemini
vndGeminiAPI: z.enum(['interactions-agent']).optional(), // opt-in per-model API dialect; unset = generateContent vndGeminiAPI: z.enum(['interactions-agent']).optional(), // opt-in per-model API dialect; unset = generateContent
vndGeminiAgentViz: z.enum(['auto', 'off']).optional(), // agent_config.visualization; default 'auto' upstream
vndGeminiAspectRatio: z.enum(['1:1', '2:3', '3:2', '3:4', '4:3', '9:16', '16:9', '21:9']).optional(), vndGeminiAspectRatio: z.enum(['1:1', '2:3', '3:2', '3:4', '4:3', '9:16', '16:9', '21:9']).optional(),
vndGeminiCodeExecution: z.enum(['auto']).optional(), vndGeminiCodeExecution: z.enum(['auto']).optional(),
vndGeminiComputerUse: z.enum(['browser']).optional(), vndGeminiComputerUse: z.enum(['browser']).optional(),
@@ -86,7 +86,8 @@ export function aixToGeminiInteractionsCreate(model: AixAPI_Model, chatGenerateR
agent_config: { agent_config: {
type: 'deep-research', type: 'deep-research',
thinking_summaries: 'auto', // Enable thought_summary blocks - without this the API would not emit summaries during streaming thinking_summaries: 'auto', // Enable thought_summary blocks - without this the API would not emit summaries during streaming
// visualization defaults to 'auto' upstream; leave unset to keep the default (agent may generate charts/images). // visualization: forwarded only when the client explicitly opts out; 'auto' (default) is left unset so the agent may generate charts/images.
...(model.vndGeminiAgentViz === 'off' && { visualization: 'off' }),
}, },
}), }),
// non-DR agents: use native system_instruction field (matches gemini.generateContent.ts convention) // non-DR agents: use native system_instruction field (matches gemini.generateContent.ts convention)
@@ -25,7 +25,7 @@ import { createAnthropicFileInlineTransform } from './parsers/anthropic.transfor
import { createAnthropicMessageParser, createAnthropicMessageParserNS } from './parsers/anthropic.parser'; import { createAnthropicMessageParser, createAnthropicMessageParserNS } from './parsers/anthropic.parser';
import { createBedrockConverseParserNS, createBedrockConverseStreamParser } from './parsers/bedrock-converse.parser'; import { createBedrockConverseParserNS, createBedrockConverseStreamParser } from './parsers/bedrock-converse.parser';
import { createGeminiGenerateContentResponseParser } from './parsers/gemini.parser'; import { createGeminiGenerateContentResponseParser } from './parsers/gemini.parser';
import { createGeminiInteractionsParserSSE } from './parsers/gemini.interactions.parser'; import { createGeminiInteractionsParserNS, createGeminiInteractionsParserSSE } from './parsers/gemini.interactions.parser';
import { createOpenAIChatCompletionsChunkParser, createOpenAIChatCompletionsParserNS } from './parsers/openai.parser'; import { createOpenAIChatCompletionsChunkParser, createOpenAIChatCompletionsParserNS } from './parsers/openai.parser';
import { createOpenAIResponseParserNS, createOpenAIResponsesEventParser } from './parsers/openai.responses.parser'; import { createOpenAIResponseParserNS, createOpenAIResponsesEventParser } from './parsers/openai.responses.parser';
@@ -329,16 +329,16 @@ export async function createChatGenerateResumeDispatch(access: AixAPI_Access, re
}; };
case 'gemini': { case 'gemini': {
// [Gemini Interactions] Reattach via SSE stream - GET /interactions/{id}?stream=true replays all events from the start (intentional - client's ContentReassembler replaces message content on reattach; partial resume via last_event_id is deliberately NOT used). // [Gemini Interactions] Reattach: SSE replay (?stream=true) or JSON snapshot (no query). See kb/modules/LLM-gemini-interactions.md.
if (resumeHandle.uht !== 'vnd.gem.interactions') if (resumeHandle.uht !== 'vnd.gem.interactions')
throw new Error(`Resume handle mismatch for gemini: expected 'vnd.gem.interactions', got '${resumeHandle.uht}'`); throw new Error(`Resume handle mismatch for gemini: expected 'vnd.gem.interactions', got '${resumeHandle.uht}'`);
if (!streaming) console.warn(`[DEV] Gemini Interactions API - Resume only supported in SSE mode, ignoring streaming=false for ${resumeHandle.runId}`);
const { url: _baseUrl, headers: _headers } = geminiAccess(access, null, GeminiInteractionsWire_API_Interactions.getPath(resumeHandle.runId /* Gemini interaction.id */), false); const { url: _baseUrl, headers: _headers } = geminiAccess(access, null, GeminiInteractionsWire_API_Interactions.getPath(resumeHandle.runId /* Gemini interaction.id */), false);
return { return {
request: { url: `${_baseUrl}${_baseUrl.includes('?') ? '&' : '?'}stream=true`, method: 'GET', headers: _headers }, request: { url: streaming ? `${_baseUrl}${_baseUrl.includes('?') ? '&' : '?'}stream=true` : _baseUrl, method: 'GET', headers: _headers },
/** Again, only support SSE here, for now (see comment in `createChatGenerateDispatch`) */ demuxerFormat: streaming ? 'fast-sse' : null,
demuxerFormat: 'fast-sse', chatGenerateParse: streaming
chatGenerateParse: createGeminiInteractionsParserSSE(null /* model name unknown at resume time - caller's DMessage already has it */), ? createGeminiInteractionsParserSSE(null /* model name unknown at resume time - caller's DMessage already has it */)
: createGeminiInteractionsParserNS(null),
}; };
} }
@@ -151,6 +151,9 @@ export function createGeminiInteractionsParserSSE(requestedModelName: string | n
if (!deltaParse.success) { if (!deltaParse.success) {
// Empty deltas ({}) appear alongside placeholder blocks (e.g. internal tool slots) - silent skip // Empty deltas ({}) appear alongside placeholder blocks (e.g. internal tool slots) - silent skip
if (event.delta && Object.keys(event.delta).length === 0) break; if (event.delta && Object.keys(event.delta).length === 0) break;
// Known-but-not-surfaced delta types (mirrors NS parser's INTERNAL_OUTPUT_TYPES policy + spec's document/video variants we don't model) - silent skip
const deltaType = (event.delta as { type?: string })?.type;
if (deltaType && (GeminiInteractionsWire_API_Interactions.INTERNAL_OUTPUT_TYPES.has(deltaType) || deltaType === 'document' || deltaType === 'video')) break;
console.warn('[GeminiInteractions] unknown content.delta shape at index', event.index, event.delta); console.warn('[GeminiInteractions] unknown content.delta shape at index', event.index, event.delta);
break; break;
} }
@@ -241,6 +244,192 @@ export function createGeminiInteractionsParserSSE(requestedModelName: string | n
} }
/**
* Non-streaming parser: reads the GET /v1beta/interactions/{id} JSON body once and emits the same
* particles the SSE parser would, in a single batch.
*
* Used by the "Recover" path when SSE delivery is broken upstream (10-min cuts; see KB doc) but the
* resource is still fetchable. We always re-emit the upstream handle so failed/in_progress runs
* remain retryable; only `status: completed` clears it (via the reassembler's outcome=='completed' policy).
*
* See `kb/modules/LLM-gemini-interactions.md` for failure modes and recovery model.
*/
export function createGeminiInteractionsParserNS(requestedModelName: string | null): ChatGenerateParseFunction {
const parserCreationTimestamp = Date.now();
return function parse(pt: IParticleTransmitter, rawEventData: string, _eventName?: string): void {
// model name (preserved from caller's DMessage on resume; first-call only on fresh fetches)
if (requestedModelName != null)
pt.setModelName(requestedModelName);
// parse + validate against the Interaction resource schema (looseObject - tolerant to upstream additions)
let rawJson: unknown;
try {
rawJson = JSON.parse(rawEventData);
} catch (e: any) {
throw new Error(`malformed Interaction JSON: ${e?.message || String(e)}`);
}
const parsed = GeminiInteractionsWire_API_Interactions.Interaction_schema.safeParse(rawJson);
if (!parsed.success) {
console.warn('[GeminiInteractions-NS] unexpected Interaction shape:', rawJson);
throw new Error('Gemini Interactions: unexpected resource shape (no `id`/`status` fields)');
}
const interaction = parsed.data;
// upstream handle - preserve so user can retry / delete
pt.setUpstreamHandle(interaction.id, 'vnd.gem.interactions');
// Walk outputs in order. Each output is loose; we safeParse against KnownOutput_schema and
// silently skip INTERNAL_OUTPUT_TYPES (tool calls/results). Order matters - thoughts and
// text interleave in the report and the user reads them top-to-bottom.
const outputs = interaction.outputs ?? [];
let lastEmittedKind: 'thought' | 'text' | 'image' | 'audio' | null = null;
for (const rawOut of outputs) {
const outType = (rawOut as { type?: string })?.type;
// silent-skip internal tool-call outputs (matches SSE parser policy for INTERNAL_OUTPUT_TYPES)
if (outType && GeminiInteractionsWire_API_Interactions.INTERNAL_OUTPUT_TYPES.has(outType))
continue;
const knownOut = GeminiInteractionsWire_API_Interactions.KnownOutput_schema.safeParse(rawOut);
if (!knownOut.success) {
if (outType) console.warn('[GeminiInteractions-NS] unknown output type, skipping:', outType);
continue;
}
// emit a part boundary when switching kinds, mirrors SSE behavior on content.start across indices
if (lastEmittedKind !== null && lastEmittedKind !== knownOut.data.type)
pt.endMessagePart();
switch (knownOut.data.type) {
case 'thought': {
const summary = knownOut.data.summary;
if (typeof summary === 'string') {
if (summary) pt.appendReasoningText(summary);
} else if (Array.isArray(summary)) {
for (const item of summary)
if (item.text) pt.appendReasoningText(item.text);
}
if (knownOut.data.signature)
pt.setReasoningSignature(knownOut.data.signature);
lastEmittedKind = 'thought';
break;
}
case 'text': {
if (knownOut.data.text)
pt.appendText(knownOut.data.text);
// Citations: matches SSE policy - DISABLE_CITATIONS kill-switch dictates Deep Research drops them
if (!DISABLE_CITATIONS && knownOut.data.annotations) {
for (const annRaw of knownOut.data.annotations) {
const ann = GeminiInteractionsWire_API_Interactions.UrlCitationAnnotation_schema.safeParse(annRaw);
if (!ann.success) continue;
const a = ann.data;
pt.appendUrlCitation(a.title || a.url, a.url, undefined, a.start_index, a.end_index, undefined, undefined);
}
}
lastEmittedKind = 'text';
break;
}
case 'image': {
if (knownOut.data.data && knownOut.data.mime_type)
pt.appendImageInline(knownOut.data.mime_type, knownOut.data.data, 'Gemini Generated Image', 'Gemini', '', true);
else if (knownOut.data.uri)
pt.appendText(`\n[Image: ${knownOut.data.uri}]\n`);
lastEmittedKind = 'image';
break;
}
case 'audio': {
if (knownOut.data.data && knownOut.data.mime_type) {
const mime = knownOut.data.mime_type.toLowerCase();
const isPCM = mime.startsWith('audio/l16') || mime.includes('codec=pcm');
if (isPCM) {
try {
const wav = geminiConvertPCM2WAV(knownOut.data.mime_type, knownOut.data.data);
pt.appendAudioInline(wav.mimeType, wav.base64Data, 'Gemini Generated Audio', 'Gemini', wav.durationMs);
} catch (error) {
console.warn('[GeminiInteractions-NS] audio PCM convert failed:', error);
}
} else {
pt.appendAudioInline(knownOut.data.mime_type, knownOut.data.data, 'Gemini Generated Audio', 'Gemini', 0);
}
}
lastEmittedKind = 'audio';
break;
}
default: {
const _exhaustive: never = knownOut.data;
break;
}
}
}
// close out any open part before the terminal status emission
if (lastEmittedKind !== null) pt.endMessagePart();
// Terminal status -> stop reason + dialect end (mirrors _handleInteractionComplete)
switch (interaction.status) {
case 'completed':
_emitUsageMetrics(pt, interaction.usage, parserCreationTimestamp, undefined);
pt.setTokenStopReason('ok');
pt.setDialectEnded('done-dialect');
break;
case 'failed':
_emitUsageMetrics(pt, interaction.usage, parserCreationTimestamp, undefined);
pt.setDialectTerminatingIssue('Deep Research interaction failed', null, 'srv-warn');
break;
case 'cancelled':
_emitUsageMetrics(pt, interaction.usage, parserCreationTimestamp, undefined);
pt.setTokenStopReason('cg-issue');
pt.setDialectEnded('done-dialect');
break;
case 'incomplete':
pt.appendText('\n_Response incomplete (run stopped early)._\n');
_emitUsageMetrics(pt, interaction.usage, parserCreationTimestamp, undefined);
pt.setTokenStopReason('out-of-tokens');
pt.setDialectEnded('done-dialect');
break;
case 'requires_action':
pt.setDialectTerminatingIssue('Deep Research returned requires_action (not supported in this client)', null, 'srv-warn');
break;
case 'in_progress': {
// Two scenarios both surface as `in_progress`:
// 1) Run is genuinely live server-side (just slow) - polling later will yield content.
// 2) "Zombie": the generator crashed but the status never transitioned. Stays `in_progress`
// for days with no outputs. Not recoverable - the only remedy is delete + retry.
// We can't disambiguate from one frame, so we surface {created, updated, outputs.length}
// and let the user decide. `tokenStopReason='cg-issue'` keeps the upstream handle alive
// (vs 'ok' which would clear it via the reassembler's clean-completion policy).
// see kb/modules/LLM-gemini-interactions.md#failure-modes (C)
const elapsedMin = _minutesSince(interaction.created);
const updatedMin = _minutesSince(interaction.updated);
const outCount = (interaction.outputs ?? []).length;
const lines: string[] = ['\n_Deep Research run is **`in_progress`** server-side._\n'];
if (elapsedMin != null) lines.push(`- Started: **${_humanDuration(elapsedMin)} ago**`);
if (updatedMin != null && updatedMin !== elapsedMin) lines.push(`- Last server update: **${_humanDuration(updatedMin)} ago**`);
lines.push(`- Outputs so far: **${outCount === 0 ? 'none' : outCount}**`);
// Heuristic threshold: stale-and-empty for >60 min is almost certainly a zombie.
const looksStuck = outCount === 0 && elapsedMin != null && elapsedMin > 60;
if (looksStuck)
lines.push('\nThis run looks **stuck** (no content for over an hour). Click **Cancel** to delete it and try again.');
else
lines.push('\nTry **Recover** again in a few minutes; if it stays empty, click **Cancel** to delete and retry.');
pt.appendText(lines.join('\n') + '\n');
pt.setTokenStopReason('cg-issue');
pt.setDialectEnded('done-dialect');
break;
}
default: {
const _exhaustiveCheck: never = interaction.status;
console.warn('[GeminiInteractions-NS] unreachable status', interaction.status);
break;
}
}
};
}
// --- helpers --- // --- helpers ---
function _classifyContentKind(rawType: unknown): BlockState['kind'] { function _classifyContentKind(rawType: unknown): BlockState['kind'] {
@@ -370,3 +559,22 @@ function _emitUsageMetrics(
pt.updateMetrics(m); pt.updateMetrics(m);
} }
/** Minutes elapsed between an upstream ISO 8601 timestamp and now. Returns null on parse failure. */
function _minutesSince(iso: string | undefined | null): number | null {
if (!iso) return null;
const ms = Date.parse(iso);
if (!Number.isFinite(ms)) return null;
return Math.max(0, (Date.now() - ms) / 60_000);
}
/** Human-readable elapsed-time string for in_progress diagnostic messages. */
function _humanDuration(minutes: number): string {
if (minutes < 1) return 'less than a minute';
if (minutes < 60) return `${Math.round(minutes)} min`;
const hours = minutes / 60;
if (hours < 24) return `${Math.round(hours * 10) / 10} hours`;
const days = hours / 24;
return `${Math.round(days * 10) / 10} days`;
}
@@ -167,7 +167,7 @@ export namespace GeminiInteractionsWire_API_Interactions {
// the parser prefers inline and falls back to a URI note when only `uri` is present. // the parser prefers inline and falls back to a URI note when only `uri` is present.
data: z.string().optional(), // base64-encoded bytes data: z.string().optional(), // base64-encoded bytes
uri: z.string().optional(), uri: z.string().optional(),
mime_type: z.string(), mime_type: z.string().optional(), // spec: optional - parser still requires it before emitting inline
resolution: z.string().optional(), // 'low' | 'medium' | 'high' | 'ultra_high' resolution: z.string().optional(), // 'low' | 'medium' | 'high' | 'ultra_high'
}); });
@@ -176,7 +176,7 @@ export namespace GeminiInteractionsWire_API_Interactions {
// Per docs: data or uri, mime_type covers both PCM (audio/l16) and packaged formats (audio/wav, audio/mp3, ...). // Per docs: data or uri, mime_type covers both PCM (audio/l16) and packaged formats (audio/wav, audio/mp3, ...).
data: z.string().optional(), data: z.string().optional(),
uri: z.string().optional(), uri: z.string().optional(),
mime_type: z.string(), mime_type: z.string().optional(), // spec: optional - parser still requires it before emitting inline
rate: z.number().optional(), // sample rate, when known rate: z.number().optional(), // sample rate, when known
channels: z.number().optional(), channels: z.number().optional(),
}); });
@@ -123,6 +123,11 @@ const _geminiGoogleSearchOptions = [
{ value: _UNSPECIFIED, label: 'Off', description: 'Default (disabled)' }, { value: _UNSPECIFIED, label: 'Off', description: 'Default (disabled)' },
] as const; ] as const;
const _geminiAgentVizOptions = [
{ value: _UNSPECIFIED, label: 'Auto', description: 'Default - agent may include charts/images' },
{ value: 'off', label: 'Off', description: 'Text only (better when merging multiple reports)' },
] as const;
const _geminiMediaResolutionOptions = [ const _geminiMediaResolutionOptions = [
{ value: 'mr_high', label: 'High', description: 'Best quality' }, { value: 'mr_high', label: 'High', description: 'Best quality' },
{ value: 'mr_medium', label: 'Medium', description: 'Balanced' }, { value: 'mr_medium', label: 'Medium', description: 'Balanced' },
@@ -245,6 +250,7 @@ export function LLMParametersEditor(props: {
llmVndAntWebSearch, llmVndAntWebSearch,
llmVndAntWebSearchMaxUses, llmVndAntWebSearchMaxUses,
llmVndGemEffort, llmVndGemEffort,
llmVndGeminiAgentViz,
llmVndGeminiAspectRatio, llmVndGeminiAspectRatio,
llmVndGeminiCodeExecution, llmVndGeminiCodeExecution,
llmVndGeminiGoogleSearch, llmVndGeminiGoogleSearch,
@@ -687,6 +693,19 @@ export function LLMParametersEditor(props: {
/> />
)} )}
{showParam('llmVndGeminiAgentViz') && (
<FormSelectControl
title='Visualizations'
tooltip='Charts and images in Deep Research reports. Disable for text-only output (helpful when merging multiple reports).'
value={llmVndGeminiAgentViz ?? _UNSPECIFIED}
onChange={(value) => {
if (value === _UNSPECIFIED || !value) onRemoveParameter('llmVndGeminiAgentViz');
else onChangeParameter({ llmVndGeminiAgentViz: value });
}}
options={_geminiAgentVizOptions}
/>
)}
{/*{showParam('llmVndMoonshotWebSearch') && (*/} {/*{showParam('llmVndMoonshotWebSearch') && (*/}
{/* <FormSelectControl*/} {/* <FormSelectControl*/}
@@ -393,7 +393,7 @@ const _knownGeminiModels: ({
isPreview: true, isPreview: true,
chatPrice: gemini25ProPricing, // pricing not explicitly listed; using 2.5 Pro as baseline chatPrice: gemini25ProPricing, // pricing not explicitly listed; using 2.5 Pro as baseline
interfaces: [LLM_IF_OAI_Chat, LLM_IF_OAI_Vision, LLM_IF_OAI_Reasoning, LLM_IF_GEM_Interactions], interfaces: [LLM_IF_OAI_Chat, LLM_IF_OAI_Vision, LLM_IF_OAI_Reasoning, LLM_IF_GEM_Interactions],
parameterSpecs: [], parameterSpecs: [{ paramId: 'llmVndGeminiAgentViz' }],
benchmark: undefined, // Deep research model, not benchmarkable on standard tests benchmark: undefined, // Deep research model, not benchmarkable on standard tests
// 128K input, 64K output // 128K input, 64K output
}, },
@@ -406,7 +406,7 @@ const _knownGeminiModels: ({
isPreview: true, isPreview: true,
chatPrice: gemini25ProPricing, // baseline estimate (see note above) chatPrice: gemini25ProPricing, // baseline estimate (see note above)
interfaces: [LLM_IF_OAI_Chat, LLM_IF_OAI_Vision, LLM_IF_OAI_Reasoning, LLM_IF_GEM_Interactions], interfaces: [LLM_IF_OAI_Chat, LLM_IF_OAI_Vision, LLM_IF_OAI_Reasoning, LLM_IF_GEM_Interactions],
parameterSpecs: [], parameterSpecs: [{ paramId: 'llmVndGeminiAgentViz' }],
benchmark: undefined, // Deep research model, not benchmarkable on standard tests benchmark: undefined, // Deep research model, not benchmarkable on standard tests
}, },
@@ -419,7 +419,7 @@ const _knownGeminiModels: ({
isPreview: true, isPreview: true,
chatPrice: gemini25ProPricing, chatPrice: gemini25ProPricing,
interfaces: [LLM_IF_OAI_Chat, LLM_IF_OAI_Vision, LLM_IF_OAI_Reasoning, LLM_IF_GEM_Interactions], interfaces: [LLM_IF_OAI_Chat, LLM_IF_OAI_Vision, LLM_IF_OAI_Reasoning, LLM_IF_GEM_Interactions],
parameterSpecs: [{ paramId: 'llmVndGeminiThinkingBudget' }], parameterSpecs: [{ paramId: 'llmVndGeminiAgentViz' }, { paramId: 'llmVndGeminiThinkingBudget' }],
benchmark: undefined, // Deep research model, not benchmarkable on standard tests benchmark: undefined, // Deep research model, not benchmarkable on standard tests
// Note: 128K input context, 64K output context // Note: 128K input context, 64K output context
}, },
@@ -94,6 +94,7 @@ const ModelParameterSpec_schema = z.object({
// Bedrock // Bedrock
'llmVndBedrockAPI', 'llmVndBedrockAPI',
// Gemini // Gemini
'llmVndGeminiAgentViz',
'llmVndGeminiAspectRatio', 'llmVndGeminiAspectRatio',
'llmVndGeminiCodeExecution', 'llmVndGeminiCodeExecution',
'llmVndGeminiComputerUse', 'llmVndGeminiComputerUse',