mirror of
https://github.com/enricoros/big-AGI.git
synced 2026-05-10 21:50:14 -07:00
Compare commits
7 Commits
1bf1b744b9
...
55bde68a4d
| Author | SHA1 | Date | |
|---|---|---|---|
| 55bde68a4d | |||
| 26ae3545a7 | |||
| 0001f7392b | |||
| d7e83e578b | |||
| 901d93b5f0 | |||
| 6858b0b94a | |||
| 9d88bf9b82 |
@@ -21,6 +21,9 @@ Architecture and system documentation is available in the `/kb/` knowledge base,
|
||||
- **[LLM-editorial-control.md](modules/LLM-editorial-pubdate.md)** - Where we have editorial control over per-model metadata vs dynamic discovery; `pubDate` field semantics, propagation chain, resolution rules, per-vendor matrix
|
||||
- **[LLM-models-catalog-pipeline.md](modules/LLM-models-catalog-pipeline.md)** - Forward-looking pipeline: extraction script, snapshot artifact, website consumption, future schema extensions
|
||||
|
||||
#### LLM - Vendor APIs
|
||||
- **[LLM-gemini-interactions.md](modules/LLM-gemini-interactions.md)** - Gemini Interactions API (Deep Research): endpoints, status taxonomy, two retrieval paths (SSE replay vs JSON GET), known failure modes (10-min cuts, zombies), UI surface
|
||||
|
||||
### Systems Documentation
|
||||
|
||||
#### Core Platform Systems
|
||||
|
||||
@@ -0,0 +1,88 @@
|
||||
# Gemini Interactions API
|
||||
|
||||
The Interactions API powers Gemini's agent runs (Deep Research today, more agent types planned). This doc is the source of truth for protocol shape, failure modes, and the recovery model — code comments link here instead of repeating the rationale.
|
||||
|
||||
## References
|
||||
|
||||
- **GH [#1088](https://github.com/enricoros/big-AGI/issues/1088)** — Auto-resume for Deep Research; Recover button
|
||||
- **GH [#1095](https://github.com/enricoros/big-AGI/issues/1095)** — Visualizations toggle (`agent_config.visualization`)
|
||||
- **Google forum [143098](https://discuss.ai.google.dev/t/interactions-api-connection-breaks-at-the-10-minutes-mark/143098)** — 10-min SSE cut
|
||||
- **Google forum [143099](https://discuss.ai.google.dev/t/streaming-resume-broken-on-interactions-api-deep-research-often-cannot-resume/143099)** — Streaming resume re-cuts
|
||||
- **Upstream specs** — `_upstream/gemini.interactions.spec.md`, `gemini.interactions.guide.md`, `gemini.deep-research.guide.md`
|
||||
|
||||
## Endpoints
|
||||
|
||||
| Verb | Path | Purpose |
|
||||
|--------|-------------------------------------------|-------------------------------------------------------------------|
|
||||
| POST | `/v1beta/interactions` | Start a run. We always send `stream:true, background:true, store:true` |
|
||||
| GET | `/v1beta/interactions/{id}?stream=true` | Reattach via SSE replay (full event sequence from start) |
|
||||
| GET | `/v1beta/interactions/{id}` | Fetch the resource as JSON (one-shot) |
|
||||
| POST | `/v1beta/interactions/{id}/cancel` | Stop a background run |
|
||||
| DELETE | `/v1beta/interactions/{id}` | Remove the stored record (does NOT cancel an in-flight run) |
|
||||
|
||||
Retention: 1 day free, 55 days paid.
|
||||
|
||||
## Status taxonomy
|
||||
|
||||
| Status | Meaning | Handling |
|
||||
|-------------------|-----------------------------------------------|-------------------------------------------------------|
|
||||
| `in_progress` | Live run **or** zombie (see C) | Surface diagnostics; offer Resume/Recover/Stop |
|
||||
| `completed` | Done with content in `outputs[]` | Emit fragments, `tokenStopReason='ok'` |
|
||||
| `failed` | Server-side failure | Terminating issue |
|
||||
| `cancelled` | We or another client cancelled | Close as `cg-issue` |
|
||||
| `incomplete` | Stopped early (token limit) — partial outputs | Note + `tokenStopReason='out-of-tokens'` |
|
||||
| `requires_action` | Not expected for Deep Research | Fail loudly so we notice |
|
||||
|
||||
## Two retrieval paths
|
||||
|
||||
| Path | Endpoint | Parser | Use case |
|
||||
|-----------------------|-----------------------------------|-------------------------------------------|-----------------------------------|
|
||||
| SSE replay | `GET ?stream=true` | `createGeminiInteractionsParserSSE` | Canonical resume; live deltas |
|
||||
| JSON GET (recovery) | `GET` (no `stream`) | `createGeminiInteractionsParserNS` | Recover when SSE is broken |
|
||||
|
||||
Both replay from the start — `ContentReassembler` REPLACES content on reattach, so partial replay (`last_event_id`) is intentionally NOT used. The NS parser walks `outputs[]` (thoughts, text, images, audio) and emits the same particles the SSE parser would, in one batch.
|
||||
|
||||
## Failure modes
|
||||
|
||||
### A. 10-minute SSE cut (forum 143098)
|
||||
|
||||
The SSE connection gets cut at exactly 600 s, regardless of activity. The cut is malformed (JSON error array instead of clean SSE close) and we treat it as stream-closed-early. The run typically **continues** server-side and reaches `completed`. **Recover (JSON GET)** retrieves the full report.
|
||||
|
||||
### B. Streaming resume re-cuts (forum 143099)
|
||||
|
||||
A fresh SSE replay can re-cut at the same 10-minute boundary on long runs, so Resume alone never reaches `interaction.complete`. **Recover** is the fallback.
|
||||
|
||||
### C. Zombie interactions (#1088)
|
||||
|
||||
Resource sits in `status: in_progress` for **days** with `outputs: []` — the generator crashed but the status never transitioned. **Not recoverable** (no data was ever produced). The NS parser surfaces `created`, `updated`, output count, and a "stuck for over an hour" hint so the user can decide to delete and retry.
|
||||
|
||||
### D. Connection drop mid-run
|
||||
|
||||
Network blip; resource is fine. **Resume (SSE replay)** picks up cleanly.
|
||||
|
||||
## UI
|
||||
|
||||
`BlockOpUpstreamResume` renders up to three buttons:
|
||||
|
||||
| Button | Action | Shown when |
|
||||
|----------|-----------------------------------|---------------------------------------------------------|
|
||||
| Resume | SSE replay | `onResume` provided |
|
||||
| Recover | JSON GET (one-shot) | `upstreamHandle.uht` ∈ `_NS_RECOVER_UHTS` |
|
||||
| Stop | Cancel + delete upstream resource | `onDelete` provided |
|
||||
|
||||
The Recover gate is an inline `uht === 'vnd.gem.interactions'` check in `BlockOpUpstreamResume.tsx` — extend when another vendor needs the same fallback. Stop is intentionally NOT gated by Resume/Recover busy state — it's the escape hatch for hung resumes.
|
||||
|
||||
## Visualization control (#1095)
|
||||
|
||||
Deep Research accepts `agent_config.visualization: 'auto' | 'off'`. Exposed as `llmVndGeminiAgentViz` (label "Visualizations"). Forwarded only when explicitly `'off'` so the upstream `'auto'` default stays untouched. Useful when merging multiple reports — image fragments break Beam fusion.
|
||||
|
||||
## Code map
|
||||
|
||||
| File | Role |
|
||||
|--------------------------------------------------------------------------------------|-------------------------------------------------------|
|
||||
| `aix/server/dispatch/wiretypes/gemini.interactions.wiretypes.ts` | Zod schemas (RequestBody, Interaction, StreamEvent) |
|
||||
| `aix/server/dispatch/chatGenerate/adapters/gemini.interactionsCreate.ts` | POST body (input + agent_config) |
|
||||
| `aix/server/dispatch/chatGenerate/parsers/gemini.interactions.parser.ts` | SSE parser + NS parser |
|
||||
| `aix/server/dispatch/chatGenerate/chatGenerate.dispatch.ts` (`gemini` case) | Resume dispatch: SSE vs JSON branch |
|
||||
| `apps/chat/components/message/BlockOpUpstreamResume.tsx` | Resume / Recover / Stop UI |
|
||||
| `apps/chat/components/ChatMessageList.tsx` (`handleMessageUpstreamResume`) | Wires click handler to `aixReattachContent_DMessage_orThrow` |
|
||||
@@ -6,6 +6,7 @@ import { Box, List } from '@mui/joy';
|
||||
|
||||
import type { SystemPurposeExample } from '../../../data';
|
||||
|
||||
import type { AixReattachMode } from '~/modules/aix/client/aix.client';
|
||||
import type { DiagramConfig } from '~/modules/aifn/digrams/DiagramsModal';
|
||||
import { speakText } from '~/modules/speex/speex.client';
|
||||
|
||||
@@ -123,7 +124,16 @@ export function ChatMessageList(props: {
|
||||
}
|
||||
}, [conversationHandler, conversationId, onConversationExecuteHistory]);
|
||||
|
||||
const handleMessageUpstreamResume = React.useCallback(async (generator: DMessageGenerator, messageId: DMessageId) => {
|
||||
|
||||
// Resume in-flight tracking - lives at this level (NOT inside BlockOpUpstreamResume) so it
|
||||
// survives any remount of the message bubble during a long-running stream (e.g. Deep Research).
|
||||
// - `resumeInFlight` (state) drives the loading/Detach UI on BlockOpUpstreamResume via props.
|
||||
// - `resumeAbortersRef` (ref) holds the AbortController so Detach can abort even after a remount.
|
||||
// Map keyed by messageId so multiple messages could in principle resume concurrently.
|
||||
const [resumeInFlight, setResumeInFlight] = React.useState<Record<DMessageId, AixReattachMode>>({});
|
||||
const resumeAbortersRef = React.useRef<Map<DMessageId, AbortController>>(new Map());
|
||||
|
||||
const handleMessageUpstreamResume = React.useCallback(async (generator: DMessageGenerator, messageId: DMessageId, mode: AixReattachMode) => {
|
||||
if (!conversationId || !conversationHandler) return;
|
||||
if (!generator.upstreamHandle) throw new Error('No upstream handle on generator');
|
||||
|
||||
@@ -131,20 +141,36 @@ export function ChatMessageList(props: {
|
||||
const llmId = generator.mgt === 'aix' ? generator.aix.mId : undefined;
|
||||
if (!llmId) throw new Error('No model id on generator');
|
||||
|
||||
const controller = new AbortController();
|
||||
resumeAbortersRef.current.set(messageId, controller);
|
||||
setResumeInFlight(prev => ({ ...prev, [messageId]: mode }));
|
||||
|
||||
const { aixCreateChatGenerateContext, aixReattachContent_DMessage_orThrow } = await import('~/modules/aix/client/aix.client');
|
||||
const result = await aixReattachContent_DMessage_orThrow(
|
||||
try {
|
||||
await aixReattachContent_DMessage_orThrow(
|
||||
llmId,
|
||||
generator,
|
||||
aixCreateChatGenerateContext('conversation', conversationId),
|
||||
{ abortSignal: 'NON_ABORTABLE', throttleParallelThreads: 0 },
|
||||
mode,
|
||||
{ abortSignal: controller.signal, throttleParallelThreads: 0 }, // Detach: aborting kills the local fetch; upstream run keeps going.
|
||||
async (update, isDone) => {
|
||||
conversationHandler.messageEdit(messageId, {
|
||||
fragments: update.fragments,
|
||||
generator: update.generator,
|
||||
pendingIncomplete: update.pendingIncomplete,
|
||||
}, isDone, isDone); // remove the pending state and updte only when done
|
||||
}, isDone, isDone); // remove the pending state and update only when done
|
||||
},
|
||||
);
|
||||
} finally {
|
||||
// Clear local tracking only if this attempt is still the current one (avoid races on rapid retry)
|
||||
if (resumeAbortersRef.current.get(messageId) === controller)
|
||||
resumeAbortersRef.current.delete(messageId);
|
||||
setResumeInFlight(prev => {
|
||||
if (prev[messageId] !== mode) return prev;
|
||||
const { [messageId]: _, ...rest } = prev;
|
||||
return rest;
|
||||
});
|
||||
}
|
||||
|
||||
// Manual reattach is one-shot: on failure (e.g. upstream 404 from expired or already-consumed handle),
|
||||
// drop the upstreamHandle so the Resume button doesn't keep luring the user into the same error.
|
||||
@@ -156,6 +182,11 @@ export function ChatMessageList(props: {
|
||||
// }, false /* messageComplete */, true /* touch */);
|
||||
}, [conversationHandler, conversationId]);
|
||||
|
||||
const handleMessageUpstreamDetach = React.useCallback((messageId: DMessageId) => {
|
||||
resumeAbortersRef.current.get(messageId)?.abort();
|
||||
}, []);
|
||||
|
||||
|
||||
const handleMessageUpstreamDelete = React.useCallback(async (generator: DMessageGenerator, messageId: DMessageId) => {
|
||||
if (!conversationId || !conversationHandler) return;
|
||||
if (!generator.upstreamHandle) throw new Error('No upstream handle on generator');
|
||||
@@ -395,7 +426,11 @@ export function ChatMessageList(props: {
|
||||
|
||||
{filteredMessages.map((message, idx) => {
|
||||
|
||||
// Optimization: only memo complete components, or we'd be memoizing garbage
|
||||
// Optimization: only memo complete components, or we'd be memoizing garbage (fragments
|
||||
// change every chunk during streaming, so the equality check would always fail).
|
||||
// CAVEAT: switching between memo and non-memo at the same position causes React to
|
||||
// remount the subtree (different component types). Any state that must survive that
|
||||
// boundary lives on this component (e.g. resumeInFlight, resumeAbortersRef).
|
||||
const ChatMessageMemoOrNot = !message.pendingIncomplete ? ChatMessageMemo : ChatMessage;
|
||||
|
||||
return props.isMessageSelectionMode ? (
|
||||
@@ -427,7 +462,9 @@ export function ChatMessageList(props: {
|
||||
onMessageBranch={handleMessageBranch}
|
||||
onMessageContinue={handleMessageContinue}
|
||||
onMessageUpstreamResume={handleMessageUpstreamResume}
|
||||
onMessageUpstreamDetach={handleMessageUpstreamDetach}
|
||||
onMessageUpstreamDelete={handleMessageUpstreamDelete}
|
||||
upstreamResumeMode={resumeInFlight[message.id]}
|
||||
onMessageDelete={handleMessageDelete}
|
||||
onMessageFragmentAppend={handleMessageAppendFragment}
|
||||
onMessageFragmentDelete={handleMessageDeleteFragment}
|
||||
|
||||
@@ -2,9 +2,13 @@ import * as React from 'react';
|
||||
import TimeAgo from 'react-timeago';
|
||||
|
||||
import { Box, Button, ButtonGroup, Tooltip, Typography } from '@mui/joy';
|
||||
import DownloadIcon from '@mui/icons-material/Download';
|
||||
import LinkOffRoundedIcon from '@mui/icons-material/LinkOffRounded';
|
||||
import PlayArrowRoundedIcon from '@mui/icons-material/PlayArrowRounded';
|
||||
import StopRoundedIcon from '@mui/icons-material/StopRounded';
|
||||
|
||||
import type { AixReattachMode } from '~/modules/aix/client/aix.client';
|
||||
|
||||
import type { DMessageGenerator } from '~/common/stores/chat/chat.message';
|
||||
|
||||
|
||||
@@ -12,54 +16,65 @@ const ARM_TIMEOUT_MS = 4000;
|
||||
|
||||
|
||||
/**
|
||||
* FIXME: COMPLETE THIS
|
||||
* Resume controls for an upstream-stored run.
|
||||
* - Resume: SSE replay (live deltas) - canonical path. Always offered when onResume exists.
|
||||
* - Recover: one-shot JSON GET - shown only for vendors that benefit from it (Gemini Interactions).
|
||||
* - Detach: abort the local fetch but leave the upstream run alive. Visible only when a resume
|
||||
* is in-flight (`inFlightMode != null`). Resume/Recover stay available afterwards.
|
||||
* - Stop: terminate the upstream run + delete the resource.
|
||||
*
|
||||
* IMPORTANT: in-flight state is owned by the parent (`inFlightMode` + `onDetach`) so it survives
|
||||
* remounts that happen while a long-running stream is active (e.g. Deep Research).
|
||||
*/
|
||||
export function BlockOpUpstreamResume(props: {
|
||||
upstreamHandle: Exclude<DMessageGenerator['upstreamHandle'], undefined>,
|
||||
pending?: boolean; // true while the message is actively streaming; labels the Delete button as "Stop"
|
||||
onResume?: () => void | Promise<void>;
|
||||
onCancel?: () => void | Promise<void>;
|
||||
pending?: boolean; // true iff a local in-flight op (initial POST or resume); drives the state machine + hides the expiry footer
|
||||
inFlightMode?: AixReattachMode; // set by the parent while a resume is in flight; drives the loading/Detach UI
|
||||
onResume?: (mode: AixReattachMode) => void | Promise<void>;
|
||||
onDetach?: () => void;
|
||||
onDelete?: () => void | Promise<void>;
|
||||
}) {
|
||||
|
||||
// state
|
||||
const [isResuming, setIsResuming] = React.useState(false);
|
||||
const [isCancelling, setIsCancelling] = React.useState(false);
|
||||
// local state - only for short-lived ops the parent doesn't own
|
||||
const [isDeleting, setIsDeleting] = React.useState(false);
|
||||
const [deleteArmed, setDeleteArmed] = React.useState(false);
|
||||
const [error, setError] = React.useState<string | null>(null);
|
||||
|
||||
// expiration: boolean is evaluated at render (may lag briefly if nothing re-renders past expiry).
|
||||
// TimeAgo handles its own tick for the label; the button's disabled state is the only consumer of this flag.
|
||||
const { expiresAt /*, runId = ''*/ } = props.upstreamHandle;
|
||||
// const isExpired = expiresAt != null && Date.now() > expiresAt;
|
||||
|
||||
// State machine - mutually exclusive triplet (idle | initial-POST | resume | recover):
|
||||
// - Idle : !pending - run not active locally (incl. post-reload, since
|
||||
// chats.converters.ts clears pendingIncomplete on hydrate).
|
||||
// - Initial POST : pending && !inFlightMode - first generation streaming.
|
||||
// - Resume replay : pending && mode='replay' - we own this resume cycle.
|
||||
// - Recover snap : pending && mode='snapshot' - we own this snapshot fetch.
|
||||
//
|
||||
// Visibility matrix (see BlockOpUpstreamResume props doc):
|
||||
// Resume Recover Detach Cancel
|
||||
// Idle ✅ ✅¹ — ✅
|
||||
// Initial POST — — — ✅
|
||||
// Resume in flight — — ✅ ✅
|
||||
// Recover in flight — ✅² — —
|
||||
// ¹ only for Gemini Interactions ² with loading spinner
|
||||
const isReplaying = props.inFlightMode === 'replay';
|
||||
const isSnapshotting = props.inFlightMode === 'snapshot';
|
||||
const isIdle = !props.pending;
|
||||
|
||||
const canRecoverVendor = props.upstreamHandle.uht === 'vnd.gem.interactions';
|
||||
const showResume = isIdle && !!props.onResume;
|
||||
const showRecover = (isIdle || isSnapshotting) && !!props.onResume && canRecoverVendor;
|
||||
const showDetach = isReplaying && !!props.onDetach;
|
||||
const showCancel = !isSnapshotting && !!props.onDelete;
|
||||
|
||||
// handlers
|
||||
|
||||
const handleResume = React.useCallback(async () => {
|
||||
const handleResume = React.useCallback((mode: AixReattachMode) => {
|
||||
if (!props.onResume) return;
|
||||
setError(null);
|
||||
setIsResuming(true);
|
||||
try {
|
||||
await props.onResume();
|
||||
} catch (err: any) {
|
||||
setError(err?.message || 'Resume failed');
|
||||
} finally {
|
||||
setIsResuming(false);
|
||||
}
|
||||
}, [props]);
|
||||
|
||||
const handleCancel = React.useCallback(async () => {
|
||||
if (!props.onCancel) return;
|
||||
setError(null);
|
||||
setIsCancelling(true);
|
||||
try {
|
||||
await props.onCancel();
|
||||
} catch (err: any) {
|
||||
setError(err?.message || 'Cancel failed');
|
||||
} finally {
|
||||
setIsCancelling(false);
|
||||
}
|
||||
// fire-and-forget: parent owns the promise lifecycle and the abort controller.
|
||||
// If it rejects, the parent surfaces the error via its own UI; we stay silent.
|
||||
Promise.resolve(props.onResume(mode)).catch(() => { /* parent handles */ });
|
||||
}, [props]);
|
||||
|
||||
// Two-click arm: first click arms (visible red "Confirm?"), second click (within ARM_TIMEOUT_MS) executes.
|
||||
@@ -88,7 +103,6 @@ export function BlockOpUpstreamResume(props: {
|
||||
return () => clearTimeout(t);
|
||||
}, [deleteArmed]);
|
||||
|
||||
|
||||
return (
|
||||
<Box
|
||||
sx={{
|
||||
@@ -100,43 +114,55 @@ export function BlockOpUpstreamResume(props: {
|
||||
}}
|
||||
>
|
||||
<ButtonGroup>
|
||||
{props.onResume && (
|
||||
<Tooltip title='Resume generation from last checkpoint'>
|
||||
{showResume && (
|
||||
<Tooltip title='Resume by re-streaming from the upstream run'>
|
||||
<Button
|
||||
disabled={isResuming || isCancelling || isDeleting}
|
||||
loading={isResuming}
|
||||
disabled={isDeleting}
|
||||
startDecorator={<PlayArrowRoundedIcon color='success' />}
|
||||
onClick={handleResume}
|
||||
onClick={() => handleResume('replay')}
|
||||
>
|
||||
Resume
|
||||
</Button>
|
||||
</Tooltip>
|
||||
)}
|
||||
|
||||
{props.onCancel && (
|
||||
<Tooltip title='Cancel the response generation'>
|
||||
{showRecover && (
|
||||
<Tooltip title='Fetch the result without streaming - recovers stuck or hung runs'>
|
||||
<Button
|
||||
disabled={isResuming || isCancelling || isDeleting}
|
||||
loading={isCancelling}
|
||||
// startDecorator={<CancelIcon />}
|
||||
onClick={handleCancel}
|
||||
disabled={isDeleting}
|
||||
loading={isSnapshotting}
|
||||
loadingPosition='start'
|
||||
startDecorator={<DownloadIcon />}
|
||||
onClick={() => handleResume('snapshot')}
|
||||
>
|
||||
Cancel
|
||||
Recover
|
||||
</Button>
|
||||
</Tooltip>
|
||||
)}
|
||||
|
||||
{props.onDelete && (
|
||||
<Tooltip title={deleteArmed ? 'Click again to confirm - cancels the run upstream (no resume after)' : (props.pending ? 'Stop this response and cancel the upstream run' : 'Cancel the upstream run')}>
|
||||
{showDetach && (
|
||||
<Tooltip title='Close this connection only - the upstream run keeps going. Click Resume or Recover later to fetch results.'>
|
||||
<Button
|
||||
disabled={isDeleting}
|
||||
startDecorator={<LinkOffRoundedIcon />}
|
||||
onClick={props.onDetach}
|
||||
>
|
||||
Detach
|
||||
</Button>
|
||||
</Tooltip>
|
||||
)}
|
||||
|
||||
{showCancel && (
|
||||
<Tooltip title={deleteArmed ? 'Click again to confirm - cancels the upstream run and clears the handle' : 'Cancel the upstream run'}>
|
||||
<Button
|
||||
loading={isDeleting}
|
||||
color={deleteArmed ? 'danger' : 'neutral'}
|
||||
variant={deleteArmed ? 'solid' : 'outlined'}
|
||||
startDecorator={<StopRoundedIcon />}
|
||||
onClick={handleDelete}
|
||||
disabled={isCancelling || isDeleting}
|
||||
disabled={isDeleting}
|
||||
>
|
||||
{deleteArmed ? 'Confirm?' : (props.pending ? 'Stop' : 'Cancel')}
|
||||
{deleteArmed ? 'Confirm?' : 'Cancel'}
|
||||
</Button>
|
||||
</Tooltip>
|
||||
)}
|
||||
|
||||
@@ -29,6 +29,7 @@ import VerticalAlignBottomIcon from '@mui/icons-material/VerticalAlignBottom';
|
||||
import VisibilityIcon from '@mui/icons-material/Visibility';
|
||||
import VisibilityOffIcon from '@mui/icons-material/VisibilityOff';
|
||||
|
||||
import type { AixReattachMode } from '~/modules/aix/client/aix.client';
|
||||
import { ModelVendorAnthropic } from '~/modules/llms/vendors/anthropic/anthropic.vendor';
|
||||
|
||||
import { AnthropicIcon } from '~/common/components/icons/vendors/AnthropicIcon';
|
||||
@@ -161,8 +162,10 @@ export function ChatMessage(props: {
|
||||
onMessageBeam?: (messageId: string) => Promise<void>,
|
||||
onMessageBranch?: (messageId: string) => void,
|
||||
onMessageContinue?: (messageId: string, continueText: null | string) => void,
|
||||
onMessageUpstreamResume?: (generator: DMessageGenerator, messageId: string) => Promise<void>,
|
||||
onMessageUpstreamResume?: (generator: DMessageGenerator, messageId: string, mode: AixReattachMode) => Promise<void>,
|
||||
onMessageUpstreamDetach?: (messageId: string) => void,
|
||||
onMessageUpstreamDelete?: (generator: DMessageGenerator, messageId: string) => Promise<void>,
|
||||
upstreamResumeMode?: AixReattachMode, // set by parent while a resume is in flight on this message
|
||||
onMessageDelete?: (messageId: string) => void,
|
||||
onMessageFragmentAppend?: (messageId: DMessageId, fragment: DMessageFragment) => void
|
||||
onMessageFragmentDelete?: (messageId: DMessageId, fragmentId: DMessageFragmentId) => void,
|
||||
@@ -247,7 +250,7 @@ export function ChatMessage(props: {
|
||||
// const wordsDiff = useWordsDifference(textSubject, props.diffPreviousText, showDiff);
|
||||
|
||||
|
||||
const { onMessageAssistantFrom, onMessageDelete, onMessageFragmentAppend, onMessageFragmentDelete, onMessageFragmentReplace, onMessageContinue, onMessageUpstreamResume, onMessageUpstreamDelete } = props;
|
||||
const { onMessageAssistantFrom, onMessageDelete, onMessageFragmentAppend, onMessageFragmentDelete, onMessageFragmentReplace, onMessageContinue, onMessageUpstreamResume, onMessageUpstreamDetach, onMessageUpstreamDelete } = props;
|
||||
|
||||
const handleFragmentNew = React.useCallback(() => {
|
||||
onMessageFragmentAppend?.(messageId, createTextContentFragment(''));
|
||||
@@ -265,11 +268,15 @@ export function ChatMessage(props: {
|
||||
onMessageContinue?.(messageId, continueText);
|
||||
}, [messageId, onMessageContinue]);
|
||||
|
||||
const handleUpstreamResume = React.useCallback(() => {
|
||||
const handleUpstreamResume = React.useCallback((mode: AixReattachMode) => {
|
||||
if (!messageGenerator) return;
|
||||
return onMessageUpstreamResume?.(messageGenerator, messageId);
|
||||
return onMessageUpstreamResume?.(messageGenerator, messageId, mode);
|
||||
}, [messageGenerator, messageId, onMessageUpstreamResume]);
|
||||
|
||||
const handleUpstreamDetach = React.useCallback(() => {
|
||||
onMessageUpstreamDetach?.(messageId);
|
||||
}, [messageId, onMessageUpstreamDetach]);
|
||||
|
||||
const handleUpstreamDelete = React.useCallback(() => {
|
||||
if (!messageGenerator) return;
|
||||
return onMessageUpstreamDelete?.(messageGenerator, messageId);
|
||||
@@ -903,7 +910,9 @@ export function ChatMessage(props: {
|
||||
<BlockOpUpstreamResume
|
||||
upstreamHandle={messageGenerator.upstreamHandle}
|
||||
pending={messagePendingIncomplete}
|
||||
onResume={(!messagePendingIncomplete && onMessageUpstreamResume) ? handleUpstreamResume : undefined}
|
||||
inFlightMode={props.upstreamResumeMode}
|
||||
onResume={onMessageUpstreamResume ? handleUpstreamResume : undefined}
|
||||
onDetach={onMessageUpstreamDetach ? handleUpstreamDetach : undefined}
|
||||
onDelete={onMessageUpstreamDelete ? handleUpstreamDelete : undefined}
|
||||
/>
|
||||
)}
|
||||
|
||||
@@ -23,7 +23,7 @@ export const Release = {
|
||||
|
||||
// this is here to trigger revalidation of data, e.g. models refresh
|
||||
Monotonics: {
|
||||
Aix: 69,
|
||||
Aix: 70,
|
||||
NewsVersion: 204,
|
||||
},
|
||||
|
||||
|
||||
@@ -349,6 +349,15 @@ export const DModelParameterRegistry = {
|
||||
// when undefined, the model chooses automatically
|
||||
},
|
||||
|
||||
// Gemini Interactions API agent_config - per-agent knobs (Deep Research only today)
|
||||
llmVndGeminiAgentViz: _enumDef({
|
||||
label: 'Visualizations',
|
||||
type: 'enum',
|
||||
description: 'Charts and images in Deep Research reports. Disable for text-only output (helpful when merging multiple reports).',
|
||||
values: ['auto', 'off'],
|
||||
// undefined means upstream default ('auto'); we only forward when explicitly 'off'
|
||||
}),
|
||||
|
||||
// NOTE: we don't have this as a parameter, as for now we use it in tandem with llmVndGeminiGoogleSearch
|
||||
// llmVndGeminiUrlContext: {
|
||||
// label: 'URL Context',
|
||||
|
||||
@@ -70,7 +70,7 @@ export function aixCreateModelFromLLMOptions(
|
||||
llmVndAntEffort, llmVndGemEffort, llmVndOaiEffort, llmVndMiscEffort,
|
||||
llmVndAnt1MContext, llmVndAntInfSpeed, llmVndAntSkills, llmVndAntThinkingBudget, llmVndAntWebDynamic, llmVndAntWebFetch, llmVndAntWebFetchMaxUses, llmVndAntWebSearch, llmVndAntWebSearchMaxUses,
|
||||
llmVndBedrockAPI,
|
||||
llmVndGeminiAspectRatio, llmVndGeminiImageSize, llmVndGeminiCodeExecution, llmVndGeminiComputerUse, llmVndGeminiGoogleSearch, llmVndGeminiMediaResolution, llmVndGeminiThinkingBudget,
|
||||
llmVndGeminiAgentViz, llmVndGeminiAspectRatio, llmVndGeminiImageSize, llmVndGeminiCodeExecution, llmVndGeminiComputerUse, llmVndGeminiGoogleSearch, llmVndGeminiMediaResolution, llmVndGeminiThinkingBudget,
|
||||
// llmVndMoonshotWebSearch,
|
||||
llmVndOaiRestoreMarkdown, llmVndOaiVerbosity, llmVndOaiWebSearchContext, llmVndOaiWebSearchGeolocation, llmVndOaiImageGeneration, llmVndOaiCodeInterpreter,
|
||||
llmVndOrtWebSearch,
|
||||
@@ -143,6 +143,7 @@ export function aixCreateModelFromLLMOptions(
|
||||
|
||||
// Gemini
|
||||
...(llmVndGeminiInteractions ? { vndGeminiAPI: 'interactions-agent' } : {}),
|
||||
...(llmVndGeminiAgentViz === 'off' ? { vndGeminiAgentViz: 'off' } : {}), // Deep Research agent_config.visualization - only forward when explicitly disabled
|
||||
...(llmVndGeminiAspectRatio ? { vndGeminiAspectRatio: llmVndGeminiAspectRatio } : {}),
|
||||
...(llmVndGeminiCodeExecution === 'auto' ? { vndGeminiCodeExecution: llmVndGeminiCodeExecution } : {}),
|
||||
...(llmVndGeminiComputerUse ? { vndGeminiComputerUse: llmVndGeminiComputerUse } : {}),
|
||||
@@ -644,22 +645,30 @@ function _finalizeLlmMetricsWithCosts(cgMetricsLg: undefined | DMetricsChatGener
|
||||
|
||||
// --- L2 - Content Generation reattachment as DMessage ---
|
||||
|
||||
/**
|
||||
* Reattach mode selects how to reconstruct an in-progress upstream run:
|
||||
* - 'replay' - canonical: SSE replays the event sequence from the start. Live deltas reach
|
||||
* the UI as the run progresses (or as past content is replayed).
|
||||
* - 'snapshot' - one-shot JSON GET returns the resource as-is right now. Used to recover when
|
||||
* the SSE endpoint is broken upstream but the resource itself is still readable.
|
||||
*
|
||||
* Names describe what you get, not how. See `kb/modules/LLM-gemini-interactions.md` for failure modes.
|
||||
*/
|
||||
export type AixReattachMode = 'replay' | 'snapshot';
|
||||
|
||||
/**
|
||||
* Reattach facade: wraps `aixChatGenerateContent_DMessage_orThrow` for the reattach-to-upstream flow.
|
||||
* - Validates the generator carries an `upstreamHandle`
|
||||
* - Stubs the unused chat-generate request, and
|
||||
* - Seeds the base function so the LL's reattach branch fires.
|
||||
*
|
||||
* On an in-progress upstream run (Gemini Deep Research today, extensible to OAI Responses), the server
|
||||
* just needs the handle to GET-poll; no chat-generate body is needed. This facade:
|
||||
* - validates the generator carries an `upstreamHandle`,
|
||||
* - stubs the chat-generate request (unused on the reattach path - the server uses the handle),
|
||||
* - seeds the base function via `clientOptions.reattachGenerator` so the LL's reattach branch fires.
|
||||
*
|
||||
* The reassembler starts with empty fragments; since Gemini Interactions snapshots are cumulative,
|
||||
* the stream will rebuild the complete content from scratch. Any partial content from the original run is replaced.
|
||||
* The reassembler replaces content on reattach (Gemini Interactions snapshots are cumulative, so this rebuilds from scratch).
|
||||
*/
|
||||
export async function aixReattachContent_DMessage_orThrow(
|
||||
llmId: DLLMId,
|
||||
reattachGenerator: Readonly<DMessageGenerator>,
|
||||
aixContext: AixAPI_Context_ChatGenerate,
|
||||
mode: AixReattachMode,
|
||||
clientOptions: Pick<AixClientOptions, 'abortSignal' | 'throttleParallelThreads'>,
|
||||
onStreamingUpdate?: (update: AixChatGenerateContent_DMessageGuts, isDone: boolean) => MaybePromise<void>,
|
||||
): Promise<_AixChatGenerateContent_DMessageGuts_WithOutcome> {
|
||||
@@ -674,7 +683,7 @@ export async function aixReattachContent_DMessage_orThrow(
|
||||
llmId,
|
||||
stubChatGenerate,
|
||||
aixContext,
|
||||
true, // streaming
|
||||
mode === 'replay', // wire-level: SSE demuxer (replay) vs one-shot JSON body (snapshot)
|
||||
{ ...clientOptions, reattachGenerator: reattachGenerator as any /* guaranteed by the check */ },
|
||||
onStreamingUpdate,
|
||||
);
|
||||
|
||||
@@ -516,6 +516,7 @@ export namespace AixWire_API {
|
||||
|
||||
// Gemini
|
||||
vndGeminiAPI: z.enum(['interactions-agent']).optional(), // opt-in per-model API dialect; unset = generateContent
|
||||
vndGeminiAgentViz: z.enum(['auto', 'off']).optional(), // agent_config.visualization; default 'auto' upstream
|
||||
vndGeminiAspectRatio: z.enum(['1:1', '2:3', '3:2', '3:4', '4:3', '9:16', '16:9', '21:9']).optional(),
|
||||
vndGeminiCodeExecution: z.enum(['auto']).optional(),
|
||||
vndGeminiComputerUse: z.enum(['browser']).optional(),
|
||||
|
||||
@@ -86,7 +86,8 @@ export function aixToGeminiInteractionsCreate(model: AixAPI_Model, chatGenerateR
|
||||
agent_config: {
|
||||
type: 'deep-research',
|
||||
thinking_summaries: 'auto', // Enable thought_summary blocks - without this the API would not emit summaries during streaming
|
||||
// visualization defaults to 'auto' upstream; leave unset to keep the default (agent may generate charts/images).
|
||||
// visualization: forwarded only when the client explicitly opts out; 'auto' (default) is left unset so the agent may generate charts/images.
|
||||
...(model.vndGeminiAgentViz === 'off' && { visualization: 'off' }),
|
||||
},
|
||||
}),
|
||||
// non-DR agents: use native system_instruction field (matches gemini.generateContent.ts convention)
|
||||
|
||||
@@ -25,7 +25,7 @@ import { createAnthropicFileInlineTransform } from './parsers/anthropic.transfor
|
||||
import { createAnthropicMessageParser, createAnthropicMessageParserNS } from './parsers/anthropic.parser';
|
||||
import { createBedrockConverseParserNS, createBedrockConverseStreamParser } from './parsers/bedrock-converse.parser';
|
||||
import { createGeminiGenerateContentResponseParser } from './parsers/gemini.parser';
|
||||
import { createGeminiInteractionsParserSSE } from './parsers/gemini.interactions.parser';
|
||||
import { createGeminiInteractionsParserNS, createGeminiInteractionsParserSSE } from './parsers/gemini.interactions.parser';
|
||||
import { createOpenAIChatCompletionsChunkParser, createOpenAIChatCompletionsParserNS } from './parsers/openai.parser';
|
||||
import { createOpenAIResponseParserNS, createOpenAIResponsesEventParser } from './parsers/openai.responses.parser';
|
||||
|
||||
@@ -329,16 +329,16 @@ export async function createChatGenerateResumeDispatch(access: AixAPI_Access, re
|
||||
};
|
||||
|
||||
case 'gemini': {
|
||||
// [Gemini Interactions] Reattach via SSE stream - GET /interactions/{id}?stream=true replays all events from the start (intentional - client's ContentReassembler replaces message content on reattach; partial resume via last_event_id is deliberately NOT used).
|
||||
// [Gemini Interactions] Reattach: SSE replay (?stream=true) or JSON snapshot (no query). See kb/modules/LLM-gemini-interactions.md.
|
||||
if (resumeHandle.uht !== 'vnd.gem.interactions')
|
||||
throw new Error(`Resume handle mismatch for gemini: expected 'vnd.gem.interactions', got '${resumeHandle.uht}'`);
|
||||
if (!streaming) console.warn(`[DEV] Gemini Interactions API - Resume only supported in SSE mode, ignoring streaming=false for ${resumeHandle.runId}`);
|
||||
const { url: _baseUrl, headers: _headers } = geminiAccess(access, null, GeminiInteractionsWire_API_Interactions.getPath(resumeHandle.runId /* Gemini interaction.id */), false);
|
||||
return {
|
||||
request: { url: `${_baseUrl}${_baseUrl.includes('?') ? '&' : '?'}stream=true`, method: 'GET', headers: _headers },
|
||||
/** Again, only support SSE here, for now (see comment in `createChatGenerateDispatch`) */
|
||||
demuxerFormat: 'fast-sse',
|
||||
chatGenerateParse: createGeminiInteractionsParserSSE(null /* model name unknown at resume time - caller's DMessage already has it */),
|
||||
request: { url: streaming ? `${_baseUrl}${_baseUrl.includes('?') ? '&' : '?'}stream=true` : _baseUrl, method: 'GET', headers: _headers },
|
||||
demuxerFormat: streaming ? 'fast-sse' : null,
|
||||
chatGenerateParse: streaming
|
||||
? createGeminiInteractionsParserSSE(null /* model name unknown at resume time - caller's DMessage already has it */)
|
||||
: createGeminiInteractionsParserNS(null),
|
||||
};
|
||||
}
|
||||
|
||||
|
||||
@@ -151,6 +151,9 @@ export function createGeminiInteractionsParserSSE(requestedModelName: string | n
|
||||
if (!deltaParse.success) {
|
||||
// Empty deltas ({}) appear alongside placeholder blocks (e.g. internal tool slots) - silent skip
|
||||
if (event.delta && Object.keys(event.delta).length === 0) break;
|
||||
// Known-but-not-surfaced delta types (mirrors NS parser's INTERNAL_OUTPUT_TYPES policy + spec's document/video variants we don't model) - silent skip
|
||||
const deltaType = (event.delta as { type?: string })?.type;
|
||||
if (deltaType && (GeminiInteractionsWire_API_Interactions.INTERNAL_OUTPUT_TYPES.has(deltaType) || deltaType === 'document' || deltaType === 'video')) break;
|
||||
console.warn('[GeminiInteractions] unknown content.delta shape at index', event.index, event.delta);
|
||||
break;
|
||||
}
|
||||
@@ -241,6 +244,192 @@ export function createGeminiInteractionsParserSSE(requestedModelName: string | n
|
||||
}
|
||||
|
||||
|
||||
/**
|
||||
* Non-streaming parser: reads the GET /v1beta/interactions/{id} JSON body once and emits the same
|
||||
* particles the SSE parser would, in a single batch.
|
||||
*
|
||||
* Used by the "Recover" path when SSE delivery is broken upstream (10-min cuts; see KB doc) but the
|
||||
* resource is still fetchable. We always re-emit the upstream handle so failed/in_progress runs
|
||||
* remain retryable; only `status: completed` clears it (via the reassembler's outcome=='completed' policy).
|
||||
*
|
||||
* See `kb/modules/LLM-gemini-interactions.md` for failure modes and recovery model.
|
||||
*/
|
||||
export function createGeminiInteractionsParserNS(requestedModelName: string | null): ChatGenerateParseFunction {
|
||||
|
||||
const parserCreationTimestamp = Date.now();
|
||||
|
||||
return function parse(pt: IParticleTransmitter, rawEventData: string, _eventName?: string): void {
|
||||
|
||||
// model name (preserved from caller's DMessage on resume; first-call only on fresh fetches)
|
||||
if (requestedModelName != null)
|
||||
pt.setModelName(requestedModelName);
|
||||
|
||||
// parse + validate against the Interaction resource schema (looseObject - tolerant to upstream additions)
|
||||
let rawJson: unknown;
|
||||
try {
|
||||
rawJson = JSON.parse(rawEventData);
|
||||
} catch (e: any) {
|
||||
throw new Error(`malformed Interaction JSON: ${e?.message || String(e)}`);
|
||||
}
|
||||
const parsed = GeminiInteractionsWire_API_Interactions.Interaction_schema.safeParse(rawJson);
|
||||
if (!parsed.success) {
|
||||
console.warn('[GeminiInteractions-NS] unexpected Interaction shape:', rawJson);
|
||||
throw new Error('Gemini Interactions: unexpected resource shape (no `id`/`status` fields)');
|
||||
}
|
||||
const interaction = parsed.data;
|
||||
|
||||
// upstream handle - preserve so user can retry / delete
|
||||
pt.setUpstreamHandle(interaction.id, 'vnd.gem.interactions');
|
||||
|
||||
// Walk outputs in order. Each output is loose; we safeParse against KnownOutput_schema and
|
||||
// silently skip INTERNAL_OUTPUT_TYPES (tool calls/results). Order matters - thoughts and
|
||||
// text interleave in the report and the user reads them top-to-bottom.
|
||||
const outputs = interaction.outputs ?? [];
|
||||
let lastEmittedKind: 'thought' | 'text' | 'image' | 'audio' | null = null;
|
||||
for (const rawOut of outputs) {
|
||||
const outType = (rawOut as { type?: string })?.type;
|
||||
|
||||
// silent-skip internal tool-call outputs (matches SSE parser policy for INTERNAL_OUTPUT_TYPES)
|
||||
if (outType && GeminiInteractionsWire_API_Interactions.INTERNAL_OUTPUT_TYPES.has(outType))
|
||||
continue;
|
||||
|
||||
const knownOut = GeminiInteractionsWire_API_Interactions.KnownOutput_schema.safeParse(rawOut);
|
||||
if (!knownOut.success) {
|
||||
if (outType) console.warn('[GeminiInteractions-NS] unknown output type, skipping:', outType);
|
||||
continue;
|
||||
}
|
||||
|
||||
// emit a part boundary when switching kinds, mirrors SSE behavior on content.start across indices
|
||||
if (lastEmittedKind !== null && lastEmittedKind !== knownOut.data.type)
|
||||
pt.endMessagePart();
|
||||
|
||||
switch (knownOut.data.type) {
|
||||
case 'thought': {
|
||||
const summary = knownOut.data.summary;
|
||||
if (typeof summary === 'string') {
|
||||
if (summary) pt.appendReasoningText(summary);
|
||||
} else if (Array.isArray(summary)) {
|
||||
for (const item of summary)
|
||||
if (item.text) pt.appendReasoningText(item.text);
|
||||
}
|
||||
if (knownOut.data.signature)
|
||||
pt.setReasoningSignature(knownOut.data.signature);
|
||||
lastEmittedKind = 'thought';
|
||||
break;
|
||||
}
|
||||
case 'text': {
|
||||
if (knownOut.data.text)
|
||||
pt.appendText(knownOut.data.text);
|
||||
// Citations: matches SSE policy - DISABLE_CITATIONS kill-switch dictates Deep Research drops them
|
||||
if (!DISABLE_CITATIONS && knownOut.data.annotations) {
|
||||
for (const annRaw of knownOut.data.annotations) {
|
||||
const ann = GeminiInteractionsWire_API_Interactions.UrlCitationAnnotation_schema.safeParse(annRaw);
|
||||
if (!ann.success) continue;
|
||||
const a = ann.data;
|
||||
pt.appendUrlCitation(a.title || a.url, a.url, undefined, a.start_index, a.end_index, undefined, undefined);
|
||||
}
|
||||
}
|
||||
lastEmittedKind = 'text';
|
||||
break;
|
||||
}
|
||||
case 'image': {
|
||||
if (knownOut.data.data && knownOut.data.mime_type)
|
||||
pt.appendImageInline(knownOut.data.mime_type, knownOut.data.data, 'Gemini Generated Image', 'Gemini', '', true);
|
||||
else if (knownOut.data.uri)
|
||||
pt.appendText(`\n[Image: ${knownOut.data.uri}]\n`);
|
||||
lastEmittedKind = 'image';
|
||||
break;
|
||||
}
|
||||
case 'audio': {
|
||||
if (knownOut.data.data && knownOut.data.mime_type) {
|
||||
const mime = knownOut.data.mime_type.toLowerCase();
|
||||
const isPCM = mime.startsWith('audio/l16') || mime.includes('codec=pcm');
|
||||
if (isPCM) {
|
||||
try {
|
||||
const wav = geminiConvertPCM2WAV(knownOut.data.mime_type, knownOut.data.data);
|
||||
pt.appendAudioInline(wav.mimeType, wav.base64Data, 'Gemini Generated Audio', 'Gemini', wav.durationMs);
|
||||
} catch (error) {
|
||||
console.warn('[GeminiInteractions-NS] audio PCM convert failed:', error);
|
||||
}
|
||||
} else {
|
||||
pt.appendAudioInline(knownOut.data.mime_type, knownOut.data.data, 'Gemini Generated Audio', 'Gemini', 0);
|
||||
}
|
||||
}
|
||||
lastEmittedKind = 'audio';
|
||||
break;
|
||||
}
|
||||
default: {
|
||||
const _exhaustive: never = knownOut.data;
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// close out any open part before the terminal status emission
|
||||
if (lastEmittedKind !== null) pt.endMessagePart();
|
||||
|
||||
// Terminal status -> stop reason + dialect end (mirrors _handleInteractionComplete)
|
||||
switch (interaction.status) {
|
||||
case 'completed':
|
||||
_emitUsageMetrics(pt, interaction.usage, parserCreationTimestamp, undefined);
|
||||
pt.setTokenStopReason('ok');
|
||||
pt.setDialectEnded('done-dialect');
|
||||
break;
|
||||
case 'failed':
|
||||
_emitUsageMetrics(pt, interaction.usage, parserCreationTimestamp, undefined);
|
||||
pt.setDialectTerminatingIssue('Deep Research interaction failed', null, 'srv-warn');
|
||||
break;
|
||||
case 'cancelled':
|
||||
_emitUsageMetrics(pt, interaction.usage, parserCreationTimestamp, undefined);
|
||||
pt.setTokenStopReason('cg-issue');
|
||||
pt.setDialectEnded('done-dialect');
|
||||
break;
|
||||
case 'incomplete':
|
||||
pt.appendText('\n_Response incomplete (run stopped early)._\n');
|
||||
_emitUsageMetrics(pt, interaction.usage, parserCreationTimestamp, undefined);
|
||||
pt.setTokenStopReason('out-of-tokens');
|
||||
pt.setDialectEnded('done-dialect');
|
||||
break;
|
||||
case 'requires_action':
|
||||
pt.setDialectTerminatingIssue('Deep Research returned requires_action (not supported in this client)', null, 'srv-warn');
|
||||
break;
|
||||
case 'in_progress': {
|
||||
// Two scenarios both surface as `in_progress`:
|
||||
// 1) Run is genuinely live server-side (just slow) - polling later will yield content.
|
||||
// 2) "Zombie": the generator crashed but the status never transitioned. Stays `in_progress`
|
||||
// for days with no outputs. Not recoverable - the only remedy is delete + retry.
|
||||
// We can't disambiguate from one frame, so we surface {created, updated, outputs.length}
|
||||
// and let the user decide. `tokenStopReason='cg-issue'` keeps the upstream handle alive
|
||||
// (vs 'ok' which would clear it via the reassembler's clean-completion policy).
|
||||
// see kb/modules/LLM-gemini-interactions.md#failure-modes (C)
|
||||
const elapsedMin = _minutesSince(interaction.created);
|
||||
const updatedMin = _minutesSince(interaction.updated);
|
||||
const outCount = (interaction.outputs ?? []).length;
|
||||
const lines: string[] = ['\n_Deep Research run is **`in_progress`** server-side._\n'];
|
||||
if (elapsedMin != null) lines.push(`- Started: **${_humanDuration(elapsedMin)} ago**`);
|
||||
if (updatedMin != null && updatedMin !== elapsedMin) lines.push(`- Last server update: **${_humanDuration(updatedMin)} ago**`);
|
||||
lines.push(`- Outputs so far: **${outCount === 0 ? 'none' : outCount}**`);
|
||||
// Heuristic threshold: stale-and-empty for >60 min is almost certainly a zombie.
|
||||
const looksStuck = outCount === 0 && elapsedMin != null && elapsedMin > 60;
|
||||
if (looksStuck)
|
||||
lines.push('\nThis run looks **stuck** (no content for over an hour). Click **Cancel** to delete it and try again.');
|
||||
else
|
||||
lines.push('\nTry **Recover** again in a few minutes; if it stays empty, click **Cancel** to delete and retry.');
|
||||
pt.appendText(lines.join('\n') + '\n');
|
||||
pt.setTokenStopReason('cg-issue');
|
||||
pt.setDialectEnded('done-dialect');
|
||||
break;
|
||||
}
|
||||
default: {
|
||||
const _exhaustiveCheck: never = interaction.status;
|
||||
console.warn('[GeminiInteractions-NS] unreachable status', interaction.status);
|
||||
break;
|
||||
}
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
|
||||
// --- helpers ---
|
||||
|
||||
function _classifyContentKind(rawType: unknown): BlockState['kind'] {
|
||||
@@ -370,3 +559,22 @@ function _emitUsageMetrics(
|
||||
|
||||
pt.updateMetrics(m);
|
||||
}
|
||||
|
||||
|
||||
/** Minutes elapsed between an upstream ISO 8601 timestamp and now. Returns null on parse failure. */
|
||||
function _minutesSince(iso: string | undefined | null): number | null {
|
||||
if (!iso) return null;
|
||||
const ms = Date.parse(iso);
|
||||
if (!Number.isFinite(ms)) return null;
|
||||
return Math.max(0, (Date.now() - ms) / 60_000);
|
||||
}
|
||||
|
||||
/** Human-readable elapsed-time string for in_progress diagnostic messages. */
|
||||
function _humanDuration(minutes: number): string {
|
||||
if (minutes < 1) return 'less than a minute';
|
||||
if (minutes < 60) return `${Math.round(minutes)} min`;
|
||||
const hours = minutes / 60;
|
||||
if (hours < 24) return `${Math.round(hours * 10) / 10} hours`;
|
||||
const days = hours / 24;
|
||||
return `${Math.round(days * 10) / 10} days`;
|
||||
}
|
||||
|
||||
@@ -167,7 +167,7 @@ export namespace GeminiInteractionsWire_API_Interactions {
|
||||
// the parser prefers inline and falls back to a URI note when only `uri` is present.
|
||||
data: z.string().optional(), // base64-encoded bytes
|
||||
uri: z.string().optional(),
|
||||
mime_type: z.string(),
|
||||
mime_type: z.string().optional(), // spec: optional - parser still requires it before emitting inline
|
||||
resolution: z.string().optional(), // 'low' | 'medium' | 'high' | 'ultra_high'
|
||||
});
|
||||
|
||||
@@ -176,7 +176,7 @@ export namespace GeminiInteractionsWire_API_Interactions {
|
||||
// Per docs: data or uri, mime_type covers both PCM (audio/l16) and packaged formats (audio/wav, audio/mp3, ...).
|
||||
data: z.string().optional(),
|
||||
uri: z.string().optional(),
|
||||
mime_type: z.string(),
|
||||
mime_type: z.string().optional(), // spec: optional - parser still requires it before emitting inline
|
||||
rate: z.number().optional(), // sample rate, when known
|
||||
channels: z.number().optional(),
|
||||
});
|
||||
|
||||
@@ -123,6 +123,11 @@ const _geminiGoogleSearchOptions = [
|
||||
{ value: _UNSPECIFIED, label: 'Off', description: 'Default (disabled)' },
|
||||
] as const;
|
||||
|
||||
const _geminiAgentVizOptions = [
|
||||
{ value: _UNSPECIFIED, label: 'Auto', description: 'Default - agent may include charts/images' },
|
||||
{ value: 'off', label: 'Off', description: 'Text only (better when merging multiple reports)' },
|
||||
] as const;
|
||||
|
||||
const _geminiMediaResolutionOptions = [
|
||||
{ value: 'mr_high', label: 'High', description: 'Best quality' },
|
||||
{ value: 'mr_medium', label: 'Medium', description: 'Balanced' },
|
||||
@@ -245,6 +250,7 @@ export function LLMParametersEditor(props: {
|
||||
llmVndAntWebSearch,
|
||||
llmVndAntWebSearchMaxUses,
|
||||
llmVndGemEffort,
|
||||
llmVndGeminiAgentViz,
|
||||
llmVndGeminiAspectRatio,
|
||||
llmVndGeminiCodeExecution,
|
||||
llmVndGeminiGoogleSearch,
|
||||
@@ -687,6 +693,19 @@ export function LLMParametersEditor(props: {
|
||||
/>
|
||||
)}
|
||||
|
||||
{showParam('llmVndGeminiAgentViz') && (
|
||||
<FormSelectControl
|
||||
title='Visualizations'
|
||||
tooltip='Charts and images in Deep Research reports. Disable for text-only output (helpful when merging multiple reports).'
|
||||
value={llmVndGeminiAgentViz ?? _UNSPECIFIED}
|
||||
onChange={(value) => {
|
||||
if (value === _UNSPECIFIED || !value) onRemoveParameter('llmVndGeminiAgentViz');
|
||||
else onChangeParameter({ llmVndGeminiAgentViz: value });
|
||||
}}
|
||||
options={_geminiAgentVizOptions}
|
||||
/>
|
||||
)}
|
||||
|
||||
|
||||
{/*{showParam('llmVndMoonshotWebSearch') && (*/}
|
||||
{/* <FormSelectControl*/}
|
||||
|
||||
@@ -393,7 +393,7 @@ const _knownGeminiModels: ({
|
||||
isPreview: true,
|
||||
chatPrice: gemini25ProPricing, // pricing not explicitly listed; using 2.5 Pro as baseline
|
||||
interfaces: [LLM_IF_OAI_Chat, LLM_IF_OAI_Vision, LLM_IF_OAI_Reasoning, LLM_IF_GEM_Interactions],
|
||||
parameterSpecs: [],
|
||||
parameterSpecs: [{ paramId: 'llmVndGeminiAgentViz' }],
|
||||
benchmark: undefined, // Deep research model, not benchmarkable on standard tests
|
||||
// 128K input, 64K output
|
||||
},
|
||||
@@ -406,7 +406,7 @@ const _knownGeminiModels: ({
|
||||
isPreview: true,
|
||||
chatPrice: gemini25ProPricing, // baseline estimate (see note above)
|
||||
interfaces: [LLM_IF_OAI_Chat, LLM_IF_OAI_Vision, LLM_IF_OAI_Reasoning, LLM_IF_GEM_Interactions],
|
||||
parameterSpecs: [],
|
||||
parameterSpecs: [{ paramId: 'llmVndGeminiAgentViz' }],
|
||||
benchmark: undefined, // Deep research model, not benchmarkable on standard tests
|
||||
},
|
||||
|
||||
@@ -419,7 +419,7 @@ const _knownGeminiModels: ({
|
||||
isPreview: true,
|
||||
chatPrice: gemini25ProPricing,
|
||||
interfaces: [LLM_IF_OAI_Chat, LLM_IF_OAI_Vision, LLM_IF_OAI_Reasoning, LLM_IF_GEM_Interactions],
|
||||
parameterSpecs: [{ paramId: 'llmVndGeminiThinkingBudget' }],
|
||||
parameterSpecs: [{ paramId: 'llmVndGeminiAgentViz' }, { paramId: 'llmVndGeminiThinkingBudget' }],
|
||||
benchmark: undefined, // Deep research model, not benchmarkable on standard tests
|
||||
// Note: 128K input context, 64K output context
|
||||
},
|
||||
|
||||
@@ -94,6 +94,7 @@ const ModelParameterSpec_schema = z.object({
|
||||
// Bedrock
|
||||
'llmVndBedrockAPI',
|
||||
// Gemini
|
||||
'llmVndGeminiAgentViz',
|
||||
'llmVndGeminiAspectRatio',
|
||||
'llmVndGeminiCodeExecution',
|
||||
'llmVndGeminiComputerUse',
|
||||
|
||||
Reference in New Issue
Block a user