mirror of
https://github.com/enricoros/big-AGI.git
synced 2026-05-10 21:50:14 -07:00
Compare commits
7 Commits
1bf1b744b9
...
55bde68a4d
| Author | SHA1 | Date | |
|---|---|---|---|
| 55bde68a4d | |||
| 26ae3545a7 | |||
| 0001f7392b | |||
| d7e83e578b | |||
| 901d93b5f0 | |||
| 6858b0b94a | |||
| 9d88bf9b82 |
@@ -21,6 +21,9 @@ Architecture and system documentation is available in the `/kb/` knowledge base,
|
|||||||
- **[LLM-editorial-control.md](modules/LLM-editorial-pubdate.md)** - Where we have editorial control over per-model metadata vs dynamic discovery; `pubDate` field semantics, propagation chain, resolution rules, per-vendor matrix
|
- **[LLM-editorial-control.md](modules/LLM-editorial-pubdate.md)** - Where we have editorial control over per-model metadata vs dynamic discovery; `pubDate` field semantics, propagation chain, resolution rules, per-vendor matrix
|
||||||
- **[LLM-models-catalog-pipeline.md](modules/LLM-models-catalog-pipeline.md)** - Forward-looking pipeline: extraction script, snapshot artifact, website consumption, future schema extensions
|
- **[LLM-models-catalog-pipeline.md](modules/LLM-models-catalog-pipeline.md)** - Forward-looking pipeline: extraction script, snapshot artifact, website consumption, future schema extensions
|
||||||
|
|
||||||
|
#### LLM - Vendor APIs
|
||||||
|
- **[LLM-gemini-interactions.md](modules/LLM-gemini-interactions.md)** - Gemini Interactions API (Deep Research): endpoints, status taxonomy, two retrieval paths (SSE replay vs JSON GET), known failure modes (10-min cuts, zombies), UI surface
|
||||||
|
|
||||||
### Systems Documentation
|
### Systems Documentation
|
||||||
|
|
||||||
#### Core Platform Systems
|
#### Core Platform Systems
|
||||||
|
|||||||
@@ -0,0 +1,88 @@
|
|||||||
|
# Gemini Interactions API
|
||||||
|
|
||||||
|
The Interactions API powers Gemini's agent runs (Deep Research today, more agent types planned). This doc is the source of truth for protocol shape, failure modes, and the recovery model — code comments link here instead of repeating the rationale.
|
||||||
|
|
||||||
|
## References
|
||||||
|
|
||||||
|
- **GH [#1088](https://github.com/enricoros/big-AGI/issues/1088)** — Auto-resume for Deep Research; Recover button
|
||||||
|
- **GH [#1095](https://github.com/enricoros/big-AGI/issues/1095)** — Visualizations toggle (`agent_config.visualization`)
|
||||||
|
- **Google forum [143098](https://discuss.ai.google.dev/t/interactions-api-connection-breaks-at-the-10-minutes-mark/143098)** — 10-min SSE cut
|
||||||
|
- **Google forum [143099](https://discuss.ai.google.dev/t/streaming-resume-broken-on-interactions-api-deep-research-often-cannot-resume/143099)** — Streaming resume re-cuts
|
||||||
|
- **Upstream specs** — `_upstream/gemini.interactions.spec.md`, `gemini.interactions.guide.md`, `gemini.deep-research.guide.md`
|
||||||
|
|
||||||
|
## Endpoints
|
||||||
|
|
||||||
|
| Verb | Path | Purpose |
|
||||||
|
|--------|-------------------------------------------|-------------------------------------------------------------------|
|
||||||
|
| POST | `/v1beta/interactions` | Start a run. We always send `stream:true, background:true, store:true` |
|
||||||
|
| GET | `/v1beta/interactions/{id}?stream=true` | Reattach via SSE replay (full event sequence from start) |
|
||||||
|
| GET | `/v1beta/interactions/{id}` | Fetch the resource as JSON (one-shot) |
|
||||||
|
| POST | `/v1beta/interactions/{id}/cancel` | Stop a background run |
|
||||||
|
| DELETE | `/v1beta/interactions/{id}` | Remove the stored record (does NOT cancel an in-flight run) |
|
||||||
|
|
||||||
|
Retention: 1 day free, 55 days paid.
|
||||||
|
|
||||||
|
## Status taxonomy
|
||||||
|
|
||||||
|
| Status | Meaning | Handling |
|
||||||
|
|-------------------|-----------------------------------------------|-------------------------------------------------------|
|
||||||
|
| `in_progress` | Live run **or** zombie (see C) | Surface diagnostics; offer Resume/Recover/Stop |
|
||||||
|
| `completed` | Done with content in `outputs[]` | Emit fragments, `tokenStopReason='ok'` |
|
||||||
|
| `failed` | Server-side failure | Terminating issue |
|
||||||
|
| `cancelled` | We or another client cancelled | Close as `cg-issue` |
|
||||||
|
| `incomplete` | Stopped early (token limit) — partial outputs | Note + `tokenStopReason='out-of-tokens'` |
|
||||||
|
| `requires_action` | Not expected for Deep Research | Fail loudly so we notice |
|
||||||
|
|
||||||
|
## Two retrieval paths
|
||||||
|
|
||||||
|
| Path | Endpoint | Parser | Use case |
|
||||||
|
|-----------------------|-----------------------------------|-------------------------------------------|-----------------------------------|
|
||||||
|
| SSE replay | `GET ?stream=true` | `createGeminiInteractionsParserSSE` | Canonical resume; live deltas |
|
||||||
|
| JSON GET (recovery) | `GET` (no `stream`) | `createGeminiInteractionsParserNS` | Recover when SSE is broken |
|
||||||
|
|
||||||
|
Both replay from the start — `ContentReassembler` REPLACES content on reattach, so partial replay (`last_event_id`) is intentionally NOT used. The NS parser walks `outputs[]` (thoughts, text, images, audio) and emits the same particles the SSE parser would, in one batch.
|
||||||
|
|
||||||
|
## Failure modes
|
||||||
|
|
||||||
|
### A. 10-minute SSE cut (forum 143098)
|
||||||
|
|
||||||
|
The SSE connection gets cut at exactly 600 s, regardless of activity. The cut is malformed (JSON error array instead of clean SSE close) and we treat it as stream-closed-early. The run typically **continues** server-side and reaches `completed`. **Recover (JSON GET)** retrieves the full report.
|
||||||
|
|
||||||
|
### B. Streaming resume re-cuts (forum 143099)
|
||||||
|
|
||||||
|
A fresh SSE replay can re-cut at the same 10-minute boundary on long runs, so Resume alone never reaches `interaction.complete`. **Recover** is the fallback.
|
||||||
|
|
||||||
|
### C. Zombie interactions (#1088)
|
||||||
|
|
||||||
|
Resource sits in `status: in_progress` for **days** with `outputs: []` — the generator crashed but the status never transitioned. **Not recoverable** (no data was ever produced). The NS parser surfaces `created`, `updated`, output count, and a "stuck for over an hour" hint so the user can decide to delete and retry.
|
||||||
|
|
||||||
|
### D. Connection drop mid-run
|
||||||
|
|
||||||
|
Network blip; resource is fine. **Resume (SSE replay)** picks up cleanly.
|
||||||
|
|
||||||
|
## UI
|
||||||
|
|
||||||
|
`BlockOpUpstreamResume` renders up to three buttons:
|
||||||
|
|
||||||
|
| Button | Action | Shown when |
|
||||||
|
|----------|-----------------------------------|---------------------------------------------------------|
|
||||||
|
| Resume | SSE replay | `onResume` provided |
|
||||||
|
| Recover | JSON GET (one-shot) | `upstreamHandle.uht` ∈ `_NS_RECOVER_UHTS` |
|
||||||
|
| Stop | Cancel + delete upstream resource | `onDelete` provided |
|
||||||
|
|
||||||
|
The Recover gate is an inline `uht === 'vnd.gem.interactions'` check in `BlockOpUpstreamResume.tsx` — extend when another vendor needs the same fallback. Stop is intentionally NOT gated by Resume/Recover busy state — it's the escape hatch for hung resumes.
|
||||||
|
|
||||||
|
## Visualization control (#1095)
|
||||||
|
|
||||||
|
Deep Research accepts `agent_config.visualization: 'auto' | 'off'`. Exposed as `llmVndGeminiAgentViz` (label "Visualizations"). Forwarded only when explicitly `'off'` so the upstream `'auto'` default stays untouched. Useful when merging multiple reports — image fragments break Beam fusion.
|
||||||
|
|
||||||
|
## Code map
|
||||||
|
|
||||||
|
| File | Role |
|
||||||
|
|--------------------------------------------------------------------------------------|-------------------------------------------------------|
|
||||||
|
| `aix/server/dispatch/wiretypes/gemini.interactions.wiretypes.ts` | Zod schemas (RequestBody, Interaction, StreamEvent) |
|
||||||
|
| `aix/server/dispatch/chatGenerate/adapters/gemini.interactionsCreate.ts` | POST body (input + agent_config) |
|
||||||
|
| `aix/server/dispatch/chatGenerate/parsers/gemini.interactions.parser.ts` | SSE parser + NS parser |
|
||||||
|
| `aix/server/dispatch/chatGenerate/chatGenerate.dispatch.ts` (`gemini` case) | Resume dispatch: SSE vs JSON branch |
|
||||||
|
| `apps/chat/components/message/BlockOpUpstreamResume.tsx` | Resume / Recover / Stop UI |
|
||||||
|
| `apps/chat/components/ChatMessageList.tsx` (`handleMessageUpstreamResume`) | Wires click handler to `aixReattachContent_DMessage_orThrow` |
|
||||||
@@ -6,6 +6,7 @@ import { Box, List } from '@mui/joy';
|
|||||||
|
|
||||||
import type { SystemPurposeExample } from '../../../data';
|
import type { SystemPurposeExample } from '../../../data';
|
||||||
|
|
||||||
|
import type { AixReattachMode } from '~/modules/aix/client/aix.client';
|
||||||
import type { DiagramConfig } from '~/modules/aifn/digrams/DiagramsModal';
|
import type { DiagramConfig } from '~/modules/aifn/digrams/DiagramsModal';
|
||||||
import { speakText } from '~/modules/speex/speex.client';
|
import { speakText } from '~/modules/speex/speex.client';
|
||||||
|
|
||||||
@@ -123,7 +124,16 @@ export function ChatMessageList(props: {
|
|||||||
}
|
}
|
||||||
}, [conversationHandler, conversationId, onConversationExecuteHistory]);
|
}, [conversationHandler, conversationId, onConversationExecuteHistory]);
|
||||||
|
|
||||||
const handleMessageUpstreamResume = React.useCallback(async (generator: DMessageGenerator, messageId: DMessageId) => {
|
|
||||||
|
// Resume in-flight tracking - lives at this level (NOT inside BlockOpUpstreamResume) so it
|
||||||
|
// survives any remount of the message bubble during a long-running stream (e.g. Deep Research).
|
||||||
|
// - `resumeInFlight` (state) drives the loading/Detach UI on BlockOpUpstreamResume via props.
|
||||||
|
// - `resumeAbortersRef` (ref) holds the AbortController so Detach can abort even after a remount.
|
||||||
|
// Map keyed by messageId so multiple messages could in principle resume concurrently.
|
||||||
|
const [resumeInFlight, setResumeInFlight] = React.useState<Record<DMessageId, AixReattachMode>>({});
|
||||||
|
const resumeAbortersRef = React.useRef<Map<DMessageId, AbortController>>(new Map());
|
||||||
|
|
||||||
|
const handleMessageUpstreamResume = React.useCallback(async (generator: DMessageGenerator, messageId: DMessageId, mode: AixReattachMode) => {
|
||||||
if (!conversationId || !conversationHandler) return;
|
if (!conversationId || !conversationHandler) return;
|
||||||
if (!generator.upstreamHandle) throw new Error('No upstream handle on generator');
|
if (!generator.upstreamHandle) throw new Error('No upstream handle on generator');
|
||||||
|
|
||||||
@@ -131,20 +141,36 @@ export function ChatMessageList(props: {
|
|||||||
const llmId = generator.mgt === 'aix' ? generator.aix.mId : undefined;
|
const llmId = generator.mgt === 'aix' ? generator.aix.mId : undefined;
|
||||||
if (!llmId) throw new Error('No model id on generator');
|
if (!llmId) throw new Error('No model id on generator');
|
||||||
|
|
||||||
|
const controller = new AbortController();
|
||||||
|
resumeAbortersRef.current.set(messageId, controller);
|
||||||
|
setResumeInFlight(prev => ({ ...prev, [messageId]: mode }));
|
||||||
|
|
||||||
const { aixCreateChatGenerateContext, aixReattachContent_DMessage_orThrow } = await import('~/modules/aix/client/aix.client');
|
const { aixCreateChatGenerateContext, aixReattachContent_DMessage_orThrow } = await import('~/modules/aix/client/aix.client');
|
||||||
const result = await aixReattachContent_DMessage_orThrow(
|
try {
|
||||||
llmId,
|
await aixReattachContent_DMessage_orThrow(
|
||||||
generator,
|
llmId,
|
||||||
aixCreateChatGenerateContext('conversation', conversationId),
|
generator,
|
||||||
{ abortSignal: 'NON_ABORTABLE', throttleParallelThreads: 0 },
|
aixCreateChatGenerateContext('conversation', conversationId),
|
||||||
async (update, isDone) => {
|
mode,
|
||||||
conversationHandler.messageEdit(messageId, {
|
{ abortSignal: controller.signal, throttleParallelThreads: 0 }, // Detach: aborting kills the local fetch; upstream run keeps going.
|
||||||
fragments: update.fragments,
|
async (update, isDone) => {
|
||||||
generator: update.generator,
|
conversationHandler.messageEdit(messageId, {
|
||||||
pendingIncomplete: update.pendingIncomplete,
|
fragments: update.fragments,
|
||||||
}, isDone, isDone); // remove the pending state and updte only when done
|
generator: update.generator,
|
||||||
},
|
pendingIncomplete: update.pendingIncomplete,
|
||||||
);
|
}, isDone, isDone); // remove the pending state and update only when done
|
||||||
|
},
|
||||||
|
);
|
||||||
|
} finally {
|
||||||
|
// Clear local tracking only if this attempt is still the current one (avoid races on rapid retry)
|
||||||
|
if (resumeAbortersRef.current.get(messageId) === controller)
|
||||||
|
resumeAbortersRef.current.delete(messageId);
|
||||||
|
setResumeInFlight(prev => {
|
||||||
|
if (prev[messageId] !== mode) return prev;
|
||||||
|
const { [messageId]: _, ...rest } = prev;
|
||||||
|
return rest;
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
// Manual reattach is one-shot: on failure (e.g. upstream 404 from expired or already-consumed handle),
|
// Manual reattach is one-shot: on failure (e.g. upstream 404 from expired or already-consumed handle),
|
||||||
// drop the upstreamHandle so the Resume button doesn't keep luring the user into the same error.
|
// drop the upstreamHandle so the Resume button doesn't keep luring the user into the same error.
|
||||||
@@ -156,6 +182,11 @@ export function ChatMessageList(props: {
|
|||||||
// }, false /* messageComplete */, true /* touch */);
|
// }, false /* messageComplete */, true /* touch */);
|
||||||
}, [conversationHandler, conversationId]);
|
}, [conversationHandler, conversationId]);
|
||||||
|
|
||||||
|
const handleMessageUpstreamDetach = React.useCallback((messageId: DMessageId) => {
|
||||||
|
resumeAbortersRef.current.get(messageId)?.abort();
|
||||||
|
}, []);
|
||||||
|
|
||||||
|
|
||||||
const handleMessageUpstreamDelete = React.useCallback(async (generator: DMessageGenerator, messageId: DMessageId) => {
|
const handleMessageUpstreamDelete = React.useCallback(async (generator: DMessageGenerator, messageId: DMessageId) => {
|
||||||
if (!conversationId || !conversationHandler) return;
|
if (!conversationId || !conversationHandler) return;
|
||||||
if (!generator.upstreamHandle) throw new Error('No upstream handle on generator');
|
if (!generator.upstreamHandle) throw new Error('No upstream handle on generator');
|
||||||
@@ -395,7 +426,11 @@ export function ChatMessageList(props: {
|
|||||||
|
|
||||||
{filteredMessages.map((message, idx) => {
|
{filteredMessages.map((message, idx) => {
|
||||||
|
|
||||||
// Optimization: only memo complete components, or we'd be memoizing garbage
|
// Optimization: only memo complete components, or we'd be memoizing garbage (fragments
|
||||||
|
// change every chunk during streaming, so the equality check would always fail).
|
||||||
|
// CAVEAT: switching between memo and non-memo at the same position causes React to
|
||||||
|
// remount the subtree (different component types). Any state that must survive that
|
||||||
|
// boundary lives on this component (e.g. resumeInFlight, resumeAbortersRef).
|
||||||
const ChatMessageMemoOrNot = !message.pendingIncomplete ? ChatMessageMemo : ChatMessage;
|
const ChatMessageMemoOrNot = !message.pendingIncomplete ? ChatMessageMemo : ChatMessage;
|
||||||
|
|
||||||
return props.isMessageSelectionMode ? (
|
return props.isMessageSelectionMode ? (
|
||||||
@@ -427,7 +462,9 @@ export function ChatMessageList(props: {
|
|||||||
onMessageBranch={handleMessageBranch}
|
onMessageBranch={handleMessageBranch}
|
||||||
onMessageContinue={handleMessageContinue}
|
onMessageContinue={handleMessageContinue}
|
||||||
onMessageUpstreamResume={handleMessageUpstreamResume}
|
onMessageUpstreamResume={handleMessageUpstreamResume}
|
||||||
|
onMessageUpstreamDetach={handleMessageUpstreamDetach}
|
||||||
onMessageUpstreamDelete={handleMessageUpstreamDelete}
|
onMessageUpstreamDelete={handleMessageUpstreamDelete}
|
||||||
|
upstreamResumeMode={resumeInFlight[message.id]}
|
||||||
onMessageDelete={handleMessageDelete}
|
onMessageDelete={handleMessageDelete}
|
||||||
onMessageFragmentAppend={handleMessageAppendFragment}
|
onMessageFragmentAppend={handleMessageAppendFragment}
|
||||||
onMessageFragmentDelete={handleMessageDeleteFragment}
|
onMessageFragmentDelete={handleMessageDeleteFragment}
|
||||||
|
|||||||
@@ -2,9 +2,13 @@ import * as React from 'react';
|
|||||||
import TimeAgo from 'react-timeago';
|
import TimeAgo from 'react-timeago';
|
||||||
|
|
||||||
import { Box, Button, ButtonGroup, Tooltip, Typography } from '@mui/joy';
|
import { Box, Button, ButtonGroup, Tooltip, Typography } from '@mui/joy';
|
||||||
|
import DownloadIcon from '@mui/icons-material/Download';
|
||||||
|
import LinkOffRoundedIcon from '@mui/icons-material/LinkOffRounded';
|
||||||
import PlayArrowRoundedIcon from '@mui/icons-material/PlayArrowRounded';
|
import PlayArrowRoundedIcon from '@mui/icons-material/PlayArrowRounded';
|
||||||
import StopRoundedIcon from '@mui/icons-material/StopRounded';
|
import StopRoundedIcon from '@mui/icons-material/StopRounded';
|
||||||
|
|
||||||
|
import type { AixReattachMode } from '~/modules/aix/client/aix.client';
|
||||||
|
|
||||||
import type { DMessageGenerator } from '~/common/stores/chat/chat.message';
|
import type { DMessageGenerator } from '~/common/stores/chat/chat.message';
|
||||||
|
|
||||||
|
|
||||||
@@ -12,54 +16,65 @@ const ARM_TIMEOUT_MS = 4000;
|
|||||||
|
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* FIXME: COMPLETE THIS
|
* Resume controls for an upstream-stored run.
|
||||||
|
* - Resume: SSE replay (live deltas) - canonical path. Always offered when onResume exists.
|
||||||
|
* - Recover: one-shot JSON GET - shown only for vendors that benefit from it (Gemini Interactions).
|
||||||
|
* - Detach: abort the local fetch but leave the upstream run alive. Visible only when a resume
|
||||||
|
* is in-flight (`inFlightMode != null`). Resume/Recover stay available afterwards.
|
||||||
|
* - Stop: terminate the upstream run + delete the resource.
|
||||||
|
*
|
||||||
|
* IMPORTANT: in-flight state is owned by the parent (`inFlightMode` + `onDetach`) so it survives
|
||||||
|
* remounts that happen while a long-running stream is active (e.g. Deep Research).
|
||||||
*/
|
*/
|
||||||
export function BlockOpUpstreamResume(props: {
|
export function BlockOpUpstreamResume(props: {
|
||||||
upstreamHandle: Exclude<DMessageGenerator['upstreamHandle'], undefined>,
|
upstreamHandle: Exclude<DMessageGenerator['upstreamHandle'], undefined>,
|
||||||
pending?: boolean; // true while the message is actively streaming; labels the Delete button as "Stop"
|
pending?: boolean; // true iff a local in-flight op (initial POST or resume); drives the state machine + hides the expiry footer
|
||||||
onResume?: () => void | Promise<void>;
|
inFlightMode?: AixReattachMode; // set by the parent while a resume is in flight; drives the loading/Detach UI
|
||||||
onCancel?: () => void | Promise<void>;
|
onResume?: (mode: AixReattachMode) => void | Promise<void>;
|
||||||
|
onDetach?: () => void;
|
||||||
onDelete?: () => void | Promise<void>;
|
onDelete?: () => void | Promise<void>;
|
||||||
}) {
|
}) {
|
||||||
|
|
||||||
// state
|
// local state - only for short-lived ops the parent doesn't own
|
||||||
const [isResuming, setIsResuming] = React.useState(false);
|
|
||||||
const [isCancelling, setIsCancelling] = React.useState(false);
|
|
||||||
const [isDeleting, setIsDeleting] = React.useState(false);
|
const [isDeleting, setIsDeleting] = React.useState(false);
|
||||||
const [deleteArmed, setDeleteArmed] = React.useState(false);
|
const [deleteArmed, setDeleteArmed] = React.useState(false);
|
||||||
const [error, setError] = React.useState<string | null>(null);
|
const [error, setError] = React.useState<string | null>(null);
|
||||||
|
|
||||||
// expiration: boolean is evaluated at render (may lag briefly if nothing re-renders past expiry).
|
// expiration: boolean is evaluated at render (may lag briefly if nothing re-renders past expiry).
|
||||||
// TimeAgo handles its own tick for the label; the button's disabled state is the only consumer of this flag.
|
|
||||||
const { expiresAt /*, runId = ''*/ } = props.upstreamHandle;
|
const { expiresAt /*, runId = ''*/ } = props.upstreamHandle;
|
||||||
// const isExpired = expiresAt != null && Date.now() > expiresAt;
|
|
||||||
|
// State machine - mutually exclusive triplet (idle | initial-POST | resume | recover):
|
||||||
|
// - Idle : !pending - run not active locally (incl. post-reload, since
|
||||||
|
// chats.converters.ts clears pendingIncomplete on hydrate).
|
||||||
|
// - Initial POST : pending && !inFlightMode - first generation streaming.
|
||||||
|
// - Resume replay : pending && mode='replay' - we own this resume cycle.
|
||||||
|
// - Recover snap : pending && mode='snapshot' - we own this snapshot fetch.
|
||||||
|
//
|
||||||
|
// Visibility matrix (see BlockOpUpstreamResume props doc):
|
||||||
|
// Resume Recover Detach Cancel
|
||||||
|
// Idle ✅ ✅¹ — ✅
|
||||||
|
// Initial POST — — — ✅
|
||||||
|
// Resume in flight — — ✅ ✅
|
||||||
|
// Recover in flight — ✅² — —
|
||||||
|
// ¹ only for Gemini Interactions ² with loading spinner
|
||||||
|
const isReplaying = props.inFlightMode === 'replay';
|
||||||
|
const isSnapshotting = props.inFlightMode === 'snapshot';
|
||||||
|
const isIdle = !props.pending;
|
||||||
|
|
||||||
|
const canRecoverVendor = props.upstreamHandle.uht === 'vnd.gem.interactions';
|
||||||
|
const showResume = isIdle && !!props.onResume;
|
||||||
|
const showRecover = (isIdle || isSnapshotting) && !!props.onResume && canRecoverVendor;
|
||||||
|
const showDetach = isReplaying && !!props.onDetach;
|
||||||
|
const showCancel = !isSnapshotting && !!props.onDelete;
|
||||||
|
|
||||||
// handlers
|
// handlers
|
||||||
|
|
||||||
const handleResume = React.useCallback(async () => {
|
const handleResume = React.useCallback((mode: AixReattachMode) => {
|
||||||
if (!props.onResume) return;
|
if (!props.onResume) return;
|
||||||
setError(null);
|
setError(null);
|
||||||
setIsResuming(true);
|
// fire-and-forget: parent owns the promise lifecycle and the abort controller.
|
||||||
try {
|
// If it rejects, the parent surfaces the error via its own UI; we stay silent.
|
||||||
await props.onResume();
|
Promise.resolve(props.onResume(mode)).catch(() => { /* parent handles */ });
|
||||||
} catch (err: any) {
|
|
||||||
setError(err?.message || 'Resume failed');
|
|
||||||
} finally {
|
|
||||||
setIsResuming(false);
|
|
||||||
}
|
|
||||||
}, [props]);
|
|
||||||
|
|
||||||
const handleCancel = React.useCallback(async () => {
|
|
||||||
if (!props.onCancel) return;
|
|
||||||
setError(null);
|
|
||||||
setIsCancelling(true);
|
|
||||||
try {
|
|
||||||
await props.onCancel();
|
|
||||||
} catch (err: any) {
|
|
||||||
setError(err?.message || 'Cancel failed');
|
|
||||||
} finally {
|
|
||||||
setIsCancelling(false);
|
|
||||||
}
|
|
||||||
}, [props]);
|
}, [props]);
|
||||||
|
|
||||||
// Two-click arm: first click arms (visible red "Confirm?"), second click (within ARM_TIMEOUT_MS) executes.
|
// Two-click arm: first click arms (visible red "Confirm?"), second click (within ARM_TIMEOUT_MS) executes.
|
||||||
@@ -88,7 +103,6 @@ export function BlockOpUpstreamResume(props: {
|
|||||||
return () => clearTimeout(t);
|
return () => clearTimeout(t);
|
||||||
}, [deleteArmed]);
|
}, [deleteArmed]);
|
||||||
|
|
||||||
|
|
||||||
return (
|
return (
|
||||||
<Box
|
<Box
|
||||||
sx={{
|
sx={{
|
||||||
@@ -100,43 +114,55 @@ export function BlockOpUpstreamResume(props: {
|
|||||||
}}
|
}}
|
||||||
>
|
>
|
||||||
<ButtonGroup>
|
<ButtonGroup>
|
||||||
{props.onResume && (
|
{showResume && (
|
||||||
<Tooltip title='Resume generation from last checkpoint'>
|
<Tooltip title='Resume by re-streaming from the upstream run'>
|
||||||
<Button
|
<Button
|
||||||
disabled={isResuming || isCancelling || isDeleting}
|
disabled={isDeleting}
|
||||||
loading={isResuming}
|
|
||||||
startDecorator={<PlayArrowRoundedIcon color='success' />}
|
startDecorator={<PlayArrowRoundedIcon color='success' />}
|
||||||
onClick={handleResume}
|
onClick={() => handleResume('replay')}
|
||||||
>
|
>
|
||||||
Resume
|
Resume
|
||||||
</Button>
|
</Button>
|
||||||
</Tooltip>
|
</Tooltip>
|
||||||
)}
|
)}
|
||||||
|
|
||||||
{props.onCancel && (
|
{showRecover && (
|
||||||
<Tooltip title='Cancel the response generation'>
|
<Tooltip title='Fetch the result without streaming - recovers stuck or hung runs'>
|
||||||
<Button
|
<Button
|
||||||
disabled={isResuming || isCancelling || isDeleting}
|
disabled={isDeleting}
|
||||||
loading={isCancelling}
|
loading={isSnapshotting}
|
||||||
// startDecorator={<CancelIcon />}
|
loadingPosition='start'
|
||||||
onClick={handleCancel}
|
startDecorator={<DownloadIcon />}
|
||||||
|
onClick={() => handleResume('snapshot')}
|
||||||
>
|
>
|
||||||
Cancel
|
Recover
|
||||||
</Button>
|
</Button>
|
||||||
</Tooltip>
|
</Tooltip>
|
||||||
)}
|
)}
|
||||||
|
|
||||||
{props.onDelete && (
|
{showDetach && (
|
||||||
<Tooltip title={deleteArmed ? 'Click again to confirm - cancels the run upstream (no resume after)' : (props.pending ? 'Stop this response and cancel the upstream run' : 'Cancel the upstream run')}>
|
<Tooltip title='Close this connection only - the upstream run keeps going. Click Resume or Recover later to fetch results.'>
|
||||||
|
<Button
|
||||||
|
disabled={isDeleting}
|
||||||
|
startDecorator={<LinkOffRoundedIcon />}
|
||||||
|
onClick={props.onDetach}
|
||||||
|
>
|
||||||
|
Detach
|
||||||
|
</Button>
|
||||||
|
</Tooltip>
|
||||||
|
)}
|
||||||
|
|
||||||
|
{showCancel && (
|
||||||
|
<Tooltip title={deleteArmed ? 'Click again to confirm - cancels the upstream run and clears the handle' : 'Cancel the upstream run'}>
|
||||||
<Button
|
<Button
|
||||||
loading={isDeleting}
|
loading={isDeleting}
|
||||||
color={deleteArmed ? 'danger' : 'neutral'}
|
color={deleteArmed ? 'danger' : 'neutral'}
|
||||||
variant={deleteArmed ? 'solid' : 'outlined'}
|
variant={deleteArmed ? 'solid' : 'outlined'}
|
||||||
startDecorator={<StopRoundedIcon />}
|
startDecorator={<StopRoundedIcon />}
|
||||||
onClick={handleDelete}
|
onClick={handleDelete}
|
||||||
disabled={isCancelling || isDeleting}
|
disabled={isDeleting}
|
||||||
>
|
>
|
||||||
{deleteArmed ? 'Confirm?' : (props.pending ? 'Stop' : 'Cancel')}
|
{deleteArmed ? 'Confirm?' : 'Cancel'}
|
||||||
</Button>
|
</Button>
|
||||||
</Tooltip>
|
</Tooltip>
|
||||||
)}
|
)}
|
||||||
|
|||||||
@@ -29,6 +29,7 @@ import VerticalAlignBottomIcon from '@mui/icons-material/VerticalAlignBottom';
|
|||||||
import VisibilityIcon from '@mui/icons-material/Visibility';
|
import VisibilityIcon from '@mui/icons-material/Visibility';
|
||||||
import VisibilityOffIcon from '@mui/icons-material/VisibilityOff';
|
import VisibilityOffIcon from '@mui/icons-material/VisibilityOff';
|
||||||
|
|
||||||
|
import type { AixReattachMode } from '~/modules/aix/client/aix.client';
|
||||||
import { ModelVendorAnthropic } from '~/modules/llms/vendors/anthropic/anthropic.vendor';
|
import { ModelVendorAnthropic } from '~/modules/llms/vendors/anthropic/anthropic.vendor';
|
||||||
|
|
||||||
import { AnthropicIcon } from '~/common/components/icons/vendors/AnthropicIcon';
|
import { AnthropicIcon } from '~/common/components/icons/vendors/AnthropicIcon';
|
||||||
@@ -161,8 +162,10 @@ export function ChatMessage(props: {
|
|||||||
onMessageBeam?: (messageId: string) => Promise<void>,
|
onMessageBeam?: (messageId: string) => Promise<void>,
|
||||||
onMessageBranch?: (messageId: string) => void,
|
onMessageBranch?: (messageId: string) => void,
|
||||||
onMessageContinue?: (messageId: string, continueText: null | string) => void,
|
onMessageContinue?: (messageId: string, continueText: null | string) => void,
|
||||||
onMessageUpstreamResume?: (generator: DMessageGenerator, messageId: string) => Promise<void>,
|
onMessageUpstreamResume?: (generator: DMessageGenerator, messageId: string, mode: AixReattachMode) => Promise<void>,
|
||||||
|
onMessageUpstreamDetach?: (messageId: string) => void,
|
||||||
onMessageUpstreamDelete?: (generator: DMessageGenerator, messageId: string) => Promise<void>,
|
onMessageUpstreamDelete?: (generator: DMessageGenerator, messageId: string) => Promise<void>,
|
||||||
|
upstreamResumeMode?: AixReattachMode, // set by parent while a resume is in flight on this message
|
||||||
onMessageDelete?: (messageId: string) => void,
|
onMessageDelete?: (messageId: string) => void,
|
||||||
onMessageFragmentAppend?: (messageId: DMessageId, fragment: DMessageFragment) => void
|
onMessageFragmentAppend?: (messageId: DMessageId, fragment: DMessageFragment) => void
|
||||||
onMessageFragmentDelete?: (messageId: DMessageId, fragmentId: DMessageFragmentId) => void,
|
onMessageFragmentDelete?: (messageId: DMessageId, fragmentId: DMessageFragmentId) => void,
|
||||||
@@ -247,7 +250,7 @@ export function ChatMessage(props: {
|
|||||||
// const wordsDiff = useWordsDifference(textSubject, props.diffPreviousText, showDiff);
|
// const wordsDiff = useWordsDifference(textSubject, props.diffPreviousText, showDiff);
|
||||||
|
|
||||||
|
|
||||||
const { onMessageAssistantFrom, onMessageDelete, onMessageFragmentAppend, onMessageFragmentDelete, onMessageFragmentReplace, onMessageContinue, onMessageUpstreamResume, onMessageUpstreamDelete } = props;
|
const { onMessageAssistantFrom, onMessageDelete, onMessageFragmentAppend, onMessageFragmentDelete, onMessageFragmentReplace, onMessageContinue, onMessageUpstreamResume, onMessageUpstreamDetach, onMessageUpstreamDelete } = props;
|
||||||
|
|
||||||
const handleFragmentNew = React.useCallback(() => {
|
const handleFragmentNew = React.useCallback(() => {
|
||||||
onMessageFragmentAppend?.(messageId, createTextContentFragment(''));
|
onMessageFragmentAppend?.(messageId, createTextContentFragment(''));
|
||||||
@@ -265,11 +268,15 @@ export function ChatMessage(props: {
|
|||||||
onMessageContinue?.(messageId, continueText);
|
onMessageContinue?.(messageId, continueText);
|
||||||
}, [messageId, onMessageContinue]);
|
}, [messageId, onMessageContinue]);
|
||||||
|
|
||||||
const handleUpstreamResume = React.useCallback(() => {
|
const handleUpstreamResume = React.useCallback((mode: AixReattachMode) => {
|
||||||
if (!messageGenerator) return;
|
if (!messageGenerator) return;
|
||||||
return onMessageUpstreamResume?.(messageGenerator, messageId);
|
return onMessageUpstreamResume?.(messageGenerator, messageId, mode);
|
||||||
}, [messageGenerator, messageId, onMessageUpstreamResume]);
|
}, [messageGenerator, messageId, onMessageUpstreamResume]);
|
||||||
|
|
||||||
|
const handleUpstreamDetach = React.useCallback(() => {
|
||||||
|
onMessageUpstreamDetach?.(messageId);
|
||||||
|
}, [messageId, onMessageUpstreamDetach]);
|
||||||
|
|
||||||
const handleUpstreamDelete = React.useCallback(() => {
|
const handleUpstreamDelete = React.useCallback(() => {
|
||||||
if (!messageGenerator) return;
|
if (!messageGenerator) return;
|
||||||
return onMessageUpstreamDelete?.(messageGenerator, messageId);
|
return onMessageUpstreamDelete?.(messageGenerator, messageId);
|
||||||
@@ -903,7 +910,9 @@ export function ChatMessage(props: {
|
|||||||
<BlockOpUpstreamResume
|
<BlockOpUpstreamResume
|
||||||
upstreamHandle={messageGenerator.upstreamHandle}
|
upstreamHandle={messageGenerator.upstreamHandle}
|
||||||
pending={messagePendingIncomplete}
|
pending={messagePendingIncomplete}
|
||||||
onResume={(!messagePendingIncomplete && onMessageUpstreamResume) ? handleUpstreamResume : undefined}
|
inFlightMode={props.upstreamResumeMode}
|
||||||
|
onResume={onMessageUpstreamResume ? handleUpstreamResume : undefined}
|
||||||
|
onDetach={onMessageUpstreamDetach ? handleUpstreamDetach : undefined}
|
||||||
onDelete={onMessageUpstreamDelete ? handleUpstreamDelete : undefined}
|
onDelete={onMessageUpstreamDelete ? handleUpstreamDelete : undefined}
|
||||||
/>
|
/>
|
||||||
)}
|
)}
|
||||||
|
|||||||
@@ -23,7 +23,7 @@ export const Release = {
|
|||||||
|
|
||||||
// this is here to trigger revalidation of data, e.g. models refresh
|
// this is here to trigger revalidation of data, e.g. models refresh
|
||||||
Monotonics: {
|
Monotonics: {
|
||||||
Aix: 69,
|
Aix: 70,
|
||||||
NewsVersion: 204,
|
NewsVersion: 204,
|
||||||
},
|
},
|
||||||
|
|
||||||
|
|||||||
@@ -349,6 +349,15 @@ export const DModelParameterRegistry = {
|
|||||||
// when undefined, the model chooses automatically
|
// when undefined, the model chooses automatically
|
||||||
},
|
},
|
||||||
|
|
||||||
|
// Gemini Interactions API agent_config - per-agent knobs (Deep Research only today)
|
||||||
|
llmVndGeminiAgentViz: _enumDef({
|
||||||
|
label: 'Visualizations',
|
||||||
|
type: 'enum',
|
||||||
|
description: 'Charts and images in Deep Research reports. Disable for text-only output (helpful when merging multiple reports).',
|
||||||
|
values: ['auto', 'off'],
|
||||||
|
// undefined means upstream default ('auto'); we only forward when explicitly 'off'
|
||||||
|
}),
|
||||||
|
|
||||||
// NOTE: we don't have this as a parameter, as for now we use it in tandem with llmVndGeminiGoogleSearch
|
// NOTE: we don't have this as a parameter, as for now we use it in tandem with llmVndGeminiGoogleSearch
|
||||||
// llmVndGeminiUrlContext: {
|
// llmVndGeminiUrlContext: {
|
||||||
// label: 'URL Context',
|
// label: 'URL Context',
|
||||||
|
|||||||
@@ -70,7 +70,7 @@ export function aixCreateModelFromLLMOptions(
|
|||||||
llmVndAntEffort, llmVndGemEffort, llmVndOaiEffort, llmVndMiscEffort,
|
llmVndAntEffort, llmVndGemEffort, llmVndOaiEffort, llmVndMiscEffort,
|
||||||
llmVndAnt1MContext, llmVndAntInfSpeed, llmVndAntSkills, llmVndAntThinkingBudget, llmVndAntWebDynamic, llmVndAntWebFetch, llmVndAntWebFetchMaxUses, llmVndAntWebSearch, llmVndAntWebSearchMaxUses,
|
llmVndAnt1MContext, llmVndAntInfSpeed, llmVndAntSkills, llmVndAntThinkingBudget, llmVndAntWebDynamic, llmVndAntWebFetch, llmVndAntWebFetchMaxUses, llmVndAntWebSearch, llmVndAntWebSearchMaxUses,
|
||||||
llmVndBedrockAPI,
|
llmVndBedrockAPI,
|
||||||
llmVndGeminiAspectRatio, llmVndGeminiImageSize, llmVndGeminiCodeExecution, llmVndGeminiComputerUse, llmVndGeminiGoogleSearch, llmVndGeminiMediaResolution, llmVndGeminiThinkingBudget,
|
llmVndGeminiAgentViz, llmVndGeminiAspectRatio, llmVndGeminiImageSize, llmVndGeminiCodeExecution, llmVndGeminiComputerUse, llmVndGeminiGoogleSearch, llmVndGeminiMediaResolution, llmVndGeminiThinkingBudget,
|
||||||
// llmVndMoonshotWebSearch,
|
// llmVndMoonshotWebSearch,
|
||||||
llmVndOaiRestoreMarkdown, llmVndOaiVerbosity, llmVndOaiWebSearchContext, llmVndOaiWebSearchGeolocation, llmVndOaiImageGeneration, llmVndOaiCodeInterpreter,
|
llmVndOaiRestoreMarkdown, llmVndOaiVerbosity, llmVndOaiWebSearchContext, llmVndOaiWebSearchGeolocation, llmVndOaiImageGeneration, llmVndOaiCodeInterpreter,
|
||||||
llmVndOrtWebSearch,
|
llmVndOrtWebSearch,
|
||||||
@@ -143,6 +143,7 @@ export function aixCreateModelFromLLMOptions(
|
|||||||
|
|
||||||
// Gemini
|
// Gemini
|
||||||
...(llmVndGeminiInteractions ? { vndGeminiAPI: 'interactions-agent' } : {}),
|
...(llmVndGeminiInteractions ? { vndGeminiAPI: 'interactions-agent' } : {}),
|
||||||
|
...(llmVndGeminiAgentViz === 'off' ? { vndGeminiAgentViz: 'off' } : {}), // Deep Research agent_config.visualization - only forward when explicitly disabled
|
||||||
...(llmVndGeminiAspectRatio ? { vndGeminiAspectRatio: llmVndGeminiAspectRatio } : {}),
|
...(llmVndGeminiAspectRatio ? { vndGeminiAspectRatio: llmVndGeminiAspectRatio } : {}),
|
||||||
...(llmVndGeminiCodeExecution === 'auto' ? { vndGeminiCodeExecution: llmVndGeminiCodeExecution } : {}),
|
...(llmVndGeminiCodeExecution === 'auto' ? { vndGeminiCodeExecution: llmVndGeminiCodeExecution } : {}),
|
||||||
...(llmVndGeminiComputerUse ? { vndGeminiComputerUse: llmVndGeminiComputerUse } : {}),
|
...(llmVndGeminiComputerUse ? { vndGeminiComputerUse: llmVndGeminiComputerUse } : {}),
|
||||||
@@ -644,22 +645,30 @@ function _finalizeLlmMetricsWithCosts(cgMetricsLg: undefined | DMetricsChatGener
|
|||||||
|
|
||||||
// --- L2 - Content Generation reattachment as DMessage ---
|
// --- L2 - Content Generation reattachment as DMessage ---
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Reattach mode selects how to reconstruct an in-progress upstream run:
|
||||||
|
* - 'replay' - canonical: SSE replays the event sequence from the start. Live deltas reach
|
||||||
|
* the UI as the run progresses (or as past content is replayed).
|
||||||
|
* - 'snapshot' - one-shot JSON GET returns the resource as-is right now. Used to recover when
|
||||||
|
* the SSE endpoint is broken upstream but the resource itself is still readable.
|
||||||
|
*
|
||||||
|
* Names describe what you get, not how. See `kb/modules/LLM-gemini-interactions.md` for failure modes.
|
||||||
|
*/
|
||||||
|
export type AixReattachMode = 'replay' | 'snapshot';
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Reattach facade: wraps `aixChatGenerateContent_DMessage_orThrow` for the reattach-to-upstream flow.
|
* Reattach facade: wraps `aixChatGenerateContent_DMessage_orThrow` for the reattach-to-upstream flow.
|
||||||
|
* - Validates the generator carries an `upstreamHandle`
|
||||||
|
* - Stubs the unused chat-generate request, and
|
||||||
|
* - Seeds the base function so the LL's reattach branch fires.
|
||||||
*
|
*
|
||||||
* On an in-progress upstream run (Gemini Deep Research today, extensible to OAI Responses), the server
|
* The reassembler replaces content on reattach (Gemini Interactions snapshots are cumulative, so this rebuilds from scratch).
|
||||||
* just needs the handle to GET-poll; no chat-generate body is needed. This facade:
|
|
||||||
* - validates the generator carries an `upstreamHandle`,
|
|
||||||
* - stubs the chat-generate request (unused on the reattach path - the server uses the handle),
|
|
||||||
* - seeds the base function via `clientOptions.reattachGenerator` so the LL's reattach branch fires.
|
|
||||||
*
|
|
||||||
* The reassembler starts with empty fragments; since Gemini Interactions snapshots are cumulative,
|
|
||||||
* the stream will rebuild the complete content from scratch. Any partial content from the original run is replaced.
|
|
||||||
*/
|
*/
|
||||||
export async function aixReattachContent_DMessage_orThrow(
|
export async function aixReattachContent_DMessage_orThrow(
|
||||||
llmId: DLLMId,
|
llmId: DLLMId,
|
||||||
reattachGenerator: Readonly<DMessageGenerator>,
|
reattachGenerator: Readonly<DMessageGenerator>,
|
||||||
aixContext: AixAPI_Context_ChatGenerate,
|
aixContext: AixAPI_Context_ChatGenerate,
|
||||||
|
mode: AixReattachMode,
|
||||||
clientOptions: Pick<AixClientOptions, 'abortSignal' | 'throttleParallelThreads'>,
|
clientOptions: Pick<AixClientOptions, 'abortSignal' | 'throttleParallelThreads'>,
|
||||||
onStreamingUpdate?: (update: AixChatGenerateContent_DMessageGuts, isDone: boolean) => MaybePromise<void>,
|
onStreamingUpdate?: (update: AixChatGenerateContent_DMessageGuts, isDone: boolean) => MaybePromise<void>,
|
||||||
): Promise<_AixChatGenerateContent_DMessageGuts_WithOutcome> {
|
): Promise<_AixChatGenerateContent_DMessageGuts_WithOutcome> {
|
||||||
@@ -674,7 +683,7 @@ export async function aixReattachContent_DMessage_orThrow(
|
|||||||
llmId,
|
llmId,
|
||||||
stubChatGenerate,
|
stubChatGenerate,
|
||||||
aixContext,
|
aixContext,
|
||||||
true, // streaming
|
mode === 'replay', // wire-level: SSE demuxer (replay) vs one-shot JSON body (snapshot)
|
||||||
{ ...clientOptions, reattachGenerator: reattachGenerator as any /* guaranteed by the check */ },
|
{ ...clientOptions, reattachGenerator: reattachGenerator as any /* guaranteed by the check */ },
|
||||||
onStreamingUpdate,
|
onStreamingUpdate,
|
||||||
);
|
);
|
||||||
|
|||||||
@@ -516,6 +516,7 @@ export namespace AixWire_API {
|
|||||||
|
|
||||||
// Gemini
|
// Gemini
|
||||||
vndGeminiAPI: z.enum(['interactions-agent']).optional(), // opt-in per-model API dialect; unset = generateContent
|
vndGeminiAPI: z.enum(['interactions-agent']).optional(), // opt-in per-model API dialect; unset = generateContent
|
||||||
|
vndGeminiAgentViz: z.enum(['auto', 'off']).optional(), // agent_config.visualization; default 'auto' upstream
|
||||||
vndGeminiAspectRatio: z.enum(['1:1', '2:3', '3:2', '3:4', '4:3', '9:16', '16:9', '21:9']).optional(),
|
vndGeminiAspectRatio: z.enum(['1:1', '2:3', '3:2', '3:4', '4:3', '9:16', '16:9', '21:9']).optional(),
|
||||||
vndGeminiCodeExecution: z.enum(['auto']).optional(),
|
vndGeminiCodeExecution: z.enum(['auto']).optional(),
|
||||||
vndGeminiComputerUse: z.enum(['browser']).optional(),
|
vndGeminiComputerUse: z.enum(['browser']).optional(),
|
||||||
|
|||||||
@@ -86,7 +86,8 @@ export function aixToGeminiInteractionsCreate(model: AixAPI_Model, chatGenerateR
|
|||||||
agent_config: {
|
agent_config: {
|
||||||
type: 'deep-research',
|
type: 'deep-research',
|
||||||
thinking_summaries: 'auto', // Enable thought_summary blocks - without this the API would not emit summaries during streaming
|
thinking_summaries: 'auto', // Enable thought_summary blocks - without this the API would not emit summaries during streaming
|
||||||
// visualization defaults to 'auto' upstream; leave unset to keep the default (agent may generate charts/images).
|
// visualization: forwarded only when the client explicitly opts out; 'auto' (default) is left unset so the agent may generate charts/images.
|
||||||
|
...(model.vndGeminiAgentViz === 'off' && { visualization: 'off' }),
|
||||||
},
|
},
|
||||||
}),
|
}),
|
||||||
// non-DR agents: use native system_instruction field (matches gemini.generateContent.ts convention)
|
// non-DR agents: use native system_instruction field (matches gemini.generateContent.ts convention)
|
||||||
|
|||||||
@@ -25,7 +25,7 @@ import { createAnthropicFileInlineTransform } from './parsers/anthropic.transfor
|
|||||||
import { createAnthropicMessageParser, createAnthropicMessageParserNS } from './parsers/anthropic.parser';
|
import { createAnthropicMessageParser, createAnthropicMessageParserNS } from './parsers/anthropic.parser';
|
||||||
import { createBedrockConverseParserNS, createBedrockConverseStreamParser } from './parsers/bedrock-converse.parser';
|
import { createBedrockConverseParserNS, createBedrockConverseStreamParser } from './parsers/bedrock-converse.parser';
|
||||||
import { createGeminiGenerateContentResponseParser } from './parsers/gemini.parser';
|
import { createGeminiGenerateContentResponseParser } from './parsers/gemini.parser';
|
||||||
import { createGeminiInteractionsParserSSE } from './parsers/gemini.interactions.parser';
|
import { createGeminiInteractionsParserNS, createGeminiInteractionsParserSSE } from './parsers/gemini.interactions.parser';
|
||||||
import { createOpenAIChatCompletionsChunkParser, createOpenAIChatCompletionsParserNS } from './parsers/openai.parser';
|
import { createOpenAIChatCompletionsChunkParser, createOpenAIChatCompletionsParserNS } from './parsers/openai.parser';
|
||||||
import { createOpenAIResponseParserNS, createOpenAIResponsesEventParser } from './parsers/openai.responses.parser';
|
import { createOpenAIResponseParserNS, createOpenAIResponsesEventParser } from './parsers/openai.responses.parser';
|
||||||
|
|
||||||
@@ -329,16 +329,16 @@ export async function createChatGenerateResumeDispatch(access: AixAPI_Access, re
|
|||||||
};
|
};
|
||||||
|
|
||||||
case 'gemini': {
|
case 'gemini': {
|
||||||
// [Gemini Interactions] Reattach via SSE stream - GET /interactions/{id}?stream=true replays all events from the start (intentional - client's ContentReassembler replaces message content on reattach; partial resume via last_event_id is deliberately NOT used).
|
// [Gemini Interactions] Reattach: SSE replay (?stream=true) or JSON snapshot (no query). See kb/modules/LLM-gemini-interactions.md.
|
||||||
if (resumeHandle.uht !== 'vnd.gem.interactions')
|
if (resumeHandle.uht !== 'vnd.gem.interactions')
|
||||||
throw new Error(`Resume handle mismatch for gemini: expected 'vnd.gem.interactions', got '${resumeHandle.uht}'`);
|
throw new Error(`Resume handle mismatch for gemini: expected 'vnd.gem.interactions', got '${resumeHandle.uht}'`);
|
||||||
if (!streaming) console.warn(`[DEV] Gemini Interactions API - Resume only supported in SSE mode, ignoring streaming=false for ${resumeHandle.runId}`);
|
|
||||||
const { url: _baseUrl, headers: _headers } = geminiAccess(access, null, GeminiInteractionsWire_API_Interactions.getPath(resumeHandle.runId /* Gemini interaction.id */), false);
|
const { url: _baseUrl, headers: _headers } = geminiAccess(access, null, GeminiInteractionsWire_API_Interactions.getPath(resumeHandle.runId /* Gemini interaction.id */), false);
|
||||||
return {
|
return {
|
||||||
request: { url: `${_baseUrl}${_baseUrl.includes('?') ? '&' : '?'}stream=true`, method: 'GET', headers: _headers },
|
request: { url: streaming ? `${_baseUrl}${_baseUrl.includes('?') ? '&' : '?'}stream=true` : _baseUrl, method: 'GET', headers: _headers },
|
||||||
/** Again, only support SSE here, for now (see comment in `createChatGenerateDispatch`) */
|
demuxerFormat: streaming ? 'fast-sse' : null,
|
||||||
demuxerFormat: 'fast-sse',
|
chatGenerateParse: streaming
|
||||||
chatGenerateParse: createGeminiInteractionsParserSSE(null /* model name unknown at resume time - caller's DMessage already has it */),
|
? createGeminiInteractionsParserSSE(null /* model name unknown at resume time - caller's DMessage already has it */)
|
||||||
|
: createGeminiInteractionsParserNS(null),
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -151,6 +151,9 @@ export function createGeminiInteractionsParserSSE(requestedModelName: string | n
|
|||||||
if (!deltaParse.success) {
|
if (!deltaParse.success) {
|
||||||
// Empty deltas ({}) appear alongside placeholder blocks (e.g. internal tool slots) - silent skip
|
// Empty deltas ({}) appear alongside placeholder blocks (e.g. internal tool slots) - silent skip
|
||||||
if (event.delta && Object.keys(event.delta).length === 0) break;
|
if (event.delta && Object.keys(event.delta).length === 0) break;
|
||||||
|
// Known-but-not-surfaced delta types (mirrors NS parser's INTERNAL_OUTPUT_TYPES policy + spec's document/video variants we don't model) - silent skip
|
||||||
|
const deltaType = (event.delta as { type?: string })?.type;
|
||||||
|
if (deltaType && (GeminiInteractionsWire_API_Interactions.INTERNAL_OUTPUT_TYPES.has(deltaType) || deltaType === 'document' || deltaType === 'video')) break;
|
||||||
console.warn('[GeminiInteractions] unknown content.delta shape at index', event.index, event.delta);
|
console.warn('[GeminiInteractions] unknown content.delta shape at index', event.index, event.delta);
|
||||||
break;
|
break;
|
||||||
}
|
}
|
||||||
@@ -241,6 +244,192 @@ export function createGeminiInteractionsParserSSE(requestedModelName: string | n
|
|||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Non-streaming parser: reads the GET /v1beta/interactions/{id} JSON body once and emits the same
|
||||||
|
* particles the SSE parser would, in a single batch.
|
||||||
|
*
|
||||||
|
* Used by the "Recover" path when SSE delivery is broken upstream (10-min cuts; see KB doc) but the
|
||||||
|
* resource is still fetchable. We always re-emit the upstream handle so failed/in_progress runs
|
||||||
|
* remain retryable; only `status: completed` clears it (via the reassembler's outcome=='completed' policy).
|
||||||
|
*
|
||||||
|
* See `kb/modules/LLM-gemini-interactions.md` for failure modes and recovery model.
|
||||||
|
*/
|
||||||
|
export function createGeminiInteractionsParserNS(requestedModelName: string | null): ChatGenerateParseFunction {
|
||||||
|
|
||||||
|
const parserCreationTimestamp = Date.now();
|
||||||
|
|
||||||
|
return function parse(pt: IParticleTransmitter, rawEventData: string, _eventName?: string): void {
|
||||||
|
|
||||||
|
// model name (preserved from caller's DMessage on resume; first-call only on fresh fetches)
|
||||||
|
if (requestedModelName != null)
|
||||||
|
pt.setModelName(requestedModelName);
|
||||||
|
|
||||||
|
// parse + validate against the Interaction resource schema (looseObject - tolerant to upstream additions)
|
||||||
|
let rawJson: unknown;
|
||||||
|
try {
|
||||||
|
rawJson = JSON.parse(rawEventData);
|
||||||
|
} catch (e: any) {
|
||||||
|
throw new Error(`malformed Interaction JSON: ${e?.message || String(e)}`);
|
||||||
|
}
|
||||||
|
const parsed = GeminiInteractionsWire_API_Interactions.Interaction_schema.safeParse(rawJson);
|
||||||
|
if (!parsed.success) {
|
||||||
|
console.warn('[GeminiInteractions-NS] unexpected Interaction shape:', rawJson);
|
||||||
|
throw new Error('Gemini Interactions: unexpected resource shape (no `id`/`status` fields)');
|
||||||
|
}
|
||||||
|
const interaction = parsed.data;
|
||||||
|
|
||||||
|
// upstream handle - preserve so user can retry / delete
|
||||||
|
pt.setUpstreamHandle(interaction.id, 'vnd.gem.interactions');
|
||||||
|
|
||||||
|
// Walk outputs in order. Each output is loose; we safeParse against KnownOutput_schema and
|
||||||
|
// silently skip INTERNAL_OUTPUT_TYPES (tool calls/results). Order matters - thoughts and
|
||||||
|
// text interleave in the report and the user reads them top-to-bottom.
|
||||||
|
const outputs = interaction.outputs ?? [];
|
||||||
|
let lastEmittedKind: 'thought' | 'text' | 'image' | 'audio' | null = null;
|
||||||
|
for (const rawOut of outputs) {
|
||||||
|
const outType = (rawOut as { type?: string })?.type;
|
||||||
|
|
||||||
|
// silent-skip internal tool-call outputs (matches SSE parser policy for INTERNAL_OUTPUT_TYPES)
|
||||||
|
if (outType && GeminiInteractionsWire_API_Interactions.INTERNAL_OUTPUT_TYPES.has(outType))
|
||||||
|
continue;
|
||||||
|
|
||||||
|
const knownOut = GeminiInteractionsWire_API_Interactions.KnownOutput_schema.safeParse(rawOut);
|
||||||
|
if (!knownOut.success) {
|
||||||
|
if (outType) console.warn('[GeminiInteractions-NS] unknown output type, skipping:', outType);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
// emit a part boundary when switching kinds, mirrors SSE behavior on content.start across indices
|
||||||
|
if (lastEmittedKind !== null && lastEmittedKind !== knownOut.data.type)
|
||||||
|
pt.endMessagePart();
|
||||||
|
|
||||||
|
switch (knownOut.data.type) {
|
||||||
|
case 'thought': {
|
||||||
|
const summary = knownOut.data.summary;
|
||||||
|
if (typeof summary === 'string') {
|
||||||
|
if (summary) pt.appendReasoningText(summary);
|
||||||
|
} else if (Array.isArray(summary)) {
|
||||||
|
for (const item of summary)
|
||||||
|
if (item.text) pt.appendReasoningText(item.text);
|
||||||
|
}
|
||||||
|
if (knownOut.data.signature)
|
||||||
|
pt.setReasoningSignature(knownOut.data.signature);
|
||||||
|
lastEmittedKind = 'thought';
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
case 'text': {
|
||||||
|
if (knownOut.data.text)
|
||||||
|
pt.appendText(knownOut.data.text);
|
||||||
|
// Citations: matches SSE policy - DISABLE_CITATIONS kill-switch dictates Deep Research drops them
|
||||||
|
if (!DISABLE_CITATIONS && knownOut.data.annotations) {
|
||||||
|
for (const annRaw of knownOut.data.annotations) {
|
||||||
|
const ann = GeminiInteractionsWire_API_Interactions.UrlCitationAnnotation_schema.safeParse(annRaw);
|
||||||
|
if (!ann.success) continue;
|
||||||
|
const a = ann.data;
|
||||||
|
pt.appendUrlCitation(a.title || a.url, a.url, undefined, a.start_index, a.end_index, undefined, undefined);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
lastEmittedKind = 'text';
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
case 'image': {
|
||||||
|
if (knownOut.data.data && knownOut.data.mime_type)
|
||||||
|
pt.appendImageInline(knownOut.data.mime_type, knownOut.data.data, 'Gemini Generated Image', 'Gemini', '', true);
|
||||||
|
else if (knownOut.data.uri)
|
||||||
|
pt.appendText(`\n[Image: ${knownOut.data.uri}]\n`);
|
||||||
|
lastEmittedKind = 'image';
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
case 'audio': {
|
||||||
|
if (knownOut.data.data && knownOut.data.mime_type) {
|
||||||
|
const mime = knownOut.data.mime_type.toLowerCase();
|
||||||
|
const isPCM = mime.startsWith('audio/l16') || mime.includes('codec=pcm');
|
||||||
|
if (isPCM) {
|
||||||
|
try {
|
||||||
|
const wav = geminiConvertPCM2WAV(knownOut.data.mime_type, knownOut.data.data);
|
||||||
|
pt.appendAudioInline(wav.mimeType, wav.base64Data, 'Gemini Generated Audio', 'Gemini', wav.durationMs);
|
||||||
|
} catch (error) {
|
||||||
|
console.warn('[GeminiInteractions-NS] audio PCM convert failed:', error);
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
pt.appendAudioInline(knownOut.data.mime_type, knownOut.data.data, 'Gemini Generated Audio', 'Gemini', 0);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
lastEmittedKind = 'audio';
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
default: {
|
||||||
|
const _exhaustive: never = knownOut.data;
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// close out any open part before the terminal status emission
|
||||||
|
if (lastEmittedKind !== null) pt.endMessagePart();
|
||||||
|
|
||||||
|
// Terminal status -> stop reason + dialect end (mirrors _handleInteractionComplete)
|
||||||
|
switch (interaction.status) {
|
||||||
|
case 'completed':
|
||||||
|
_emitUsageMetrics(pt, interaction.usage, parserCreationTimestamp, undefined);
|
||||||
|
pt.setTokenStopReason('ok');
|
||||||
|
pt.setDialectEnded('done-dialect');
|
||||||
|
break;
|
||||||
|
case 'failed':
|
||||||
|
_emitUsageMetrics(pt, interaction.usage, parserCreationTimestamp, undefined);
|
||||||
|
pt.setDialectTerminatingIssue('Deep Research interaction failed', null, 'srv-warn');
|
||||||
|
break;
|
||||||
|
case 'cancelled':
|
||||||
|
_emitUsageMetrics(pt, interaction.usage, parserCreationTimestamp, undefined);
|
||||||
|
pt.setTokenStopReason('cg-issue');
|
||||||
|
pt.setDialectEnded('done-dialect');
|
||||||
|
break;
|
||||||
|
case 'incomplete':
|
||||||
|
pt.appendText('\n_Response incomplete (run stopped early)._\n');
|
||||||
|
_emitUsageMetrics(pt, interaction.usage, parserCreationTimestamp, undefined);
|
||||||
|
pt.setTokenStopReason('out-of-tokens');
|
||||||
|
pt.setDialectEnded('done-dialect');
|
||||||
|
break;
|
||||||
|
case 'requires_action':
|
||||||
|
pt.setDialectTerminatingIssue('Deep Research returned requires_action (not supported in this client)', null, 'srv-warn');
|
||||||
|
break;
|
||||||
|
case 'in_progress': {
|
||||||
|
// Two scenarios both surface as `in_progress`:
|
||||||
|
// 1) Run is genuinely live server-side (just slow) - polling later will yield content.
|
||||||
|
// 2) "Zombie": the generator crashed but the status never transitioned. Stays `in_progress`
|
||||||
|
// for days with no outputs. Not recoverable - the only remedy is delete + retry.
|
||||||
|
// We can't disambiguate from one frame, so we surface {created, updated, outputs.length}
|
||||||
|
// and let the user decide. `tokenStopReason='cg-issue'` keeps the upstream handle alive
|
||||||
|
// (vs 'ok' which would clear it via the reassembler's clean-completion policy).
|
||||||
|
// see kb/modules/LLM-gemini-interactions.md#failure-modes (C)
|
||||||
|
const elapsedMin = _minutesSince(interaction.created);
|
||||||
|
const updatedMin = _minutesSince(interaction.updated);
|
||||||
|
const outCount = (interaction.outputs ?? []).length;
|
||||||
|
const lines: string[] = ['\n_Deep Research run is **`in_progress`** server-side._\n'];
|
||||||
|
if (elapsedMin != null) lines.push(`- Started: **${_humanDuration(elapsedMin)} ago**`);
|
||||||
|
if (updatedMin != null && updatedMin !== elapsedMin) lines.push(`- Last server update: **${_humanDuration(updatedMin)} ago**`);
|
||||||
|
lines.push(`- Outputs so far: **${outCount === 0 ? 'none' : outCount}**`);
|
||||||
|
// Heuristic threshold: stale-and-empty for >60 min is almost certainly a zombie.
|
||||||
|
const looksStuck = outCount === 0 && elapsedMin != null && elapsedMin > 60;
|
||||||
|
if (looksStuck)
|
||||||
|
lines.push('\nThis run looks **stuck** (no content for over an hour). Click **Cancel** to delete it and try again.');
|
||||||
|
else
|
||||||
|
lines.push('\nTry **Recover** again in a few minutes; if it stays empty, click **Cancel** to delete and retry.');
|
||||||
|
pt.appendText(lines.join('\n') + '\n');
|
||||||
|
pt.setTokenStopReason('cg-issue');
|
||||||
|
pt.setDialectEnded('done-dialect');
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
default: {
|
||||||
|
const _exhaustiveCheck: never = interaction.status;
|
||||||
|
console.warn('[GeminiInteractions-NS] unreachable status', interaction.status);
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
// --- helpers ---
|
// --- helpers ---
|
||||||
|
|
||||||
function _classifyContentKind(rawType: unknown): BlockState['kind'] {
|
function _classifyContentKind(rawType: unknown): BlockState['kind'] {
|
||||||
@@ -370,3 +559,22 @@ function _emitUsageMetrics(
|
|||||||
|
|
||||||
pt.updateMetrics(m);
|
pt.updateMetrics(m);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
/** Minutes elapsed between an upstream ISO 8601 timestamp and now. Returns null on parse failure. */
|
||||||
|
function _minutesSince(iso: string | undefined | null): number | null {
|
||||||
|
if (!iso) return null;
|
||||||
|
const ms = Date.parse(iso);
|
||||||
|
if (!Number.isFinite(ms)) return null;
|
||||||
|
return Math.max(0, (Date.now() - ms) / 60_000);
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Human-readable elapsed-time string for in_progress diagnostic messages. */
|
||||||
|
function _humanDuration(minutes: number): string {
|
||||||
|
if (minutes < 1) return 'less than a minute';
|
||||||
|
if (minutes < 60) return `${Math.round(minutes)} min`;
|
||||||
|
const hours = minutes / 60;
|
||||||
|
if (hours < 24) return `${Math.round(hours * 10) / 10} hours`;
|
||||||
|
const days = hours / 24;
|
||||||
|
return `${Math.round(days * 10) / 10} days`;
|
||||||
|
}
|
||||||
|
|||||||
@@ -167,7 +167,7 @@ export namespace GeminiInteractionsWire_API_Interactions {
|
|||||||
// the parser prefers inline and falls back to a URI note when only `uri` is present.
|
// the parser prefers inline and falls back to a URI note when only `uri` is present.
|
||||||
data: z.string().optional(), // base64-encoded bytes
|
data: z.string().optional(), // base64-encoded bytes
|
||||||
uri: z.string().optional(),
|
uri: z.string().optional(),
|
||||||
mime_type: z.string(),
|
mime_type: z.string().optional(), // spec: optional - parser still requires it before emitting inline
|
||||||
resolution: z.string().optional(), // 'low' | 'medium' | 'high' | 'ultra_high'
|
resolution: z.string().optional(), // 'low' | 'medium' | 'high' | 'ultra_high'
|
||||||
});
|
});
|
||||||
|
|
||||||
@@ -176,7 +176,7 @@ export namespace GeminiInteractionsWire_API_Interactions {
|
|||||||
// Per docs: data or uri, mime_type covers both PCM (audio/l16) and packaged formats (audio/wav, audio/mp3, ...).
|
// Per docs: data or uri, mime_type covers both PCM (audio/l16) and packaged formats (audio/wav, audio/mp3, ...).
|
||||||
data: z.string().optional(),
|
data: z.string().optional(),
|
||||||
uri: z.string().optional(),
|
uri: z.string().optional(),
|
||||||
mime_type: z.string(),
|
mime_type: z.string().optional(), // spec: optional - parser still requires it before emitting inline
|
||||||
rate: z.number().optional(), // sample rate, when known
|
rate: z.number().optional(), // sample rate, when known
|
||||||
channels: z.number().optional(),
|
channels: z.number().optional(),
|
||||||
});
|
});
|
||||||
|
|||||||
@@ -123,6 +123,11 @@ const _geminiGoogleSearchOptions = [
|
|||||||
{ value: _UNSPECIFIED, label: 'Off', description: 'Default (disabled)' },
|
{ value: _UNSPECIFIED, label: 'Off', description: 'Default (disabled)' },
|
||||||
] as const;
|
] as const;
|
||||||
|
|
||||||
|
const _geminiAgentVizOptions = [
|
||||||
|
{ value: _UNSPECIFIED, label: 'Auto', description: 'Default - agent may include charts/images' },
|
||||||
|
{ value: 'off', label: 'Off', description: 'Text only (better when merging multiple reports)' },
|
||||||
|
] as const;
|
||||||
|
|
||||||
const _geminiMediaResolutionOptions = [
|
const _geminiMediaResolutionOptions = [
|
||||||
{ value: 'mr_high', label: 'High', description: 'Best quality' },
|
{ value: 'mr_high', label: 'High', description: 'Best quality' },
|
||||||
{ value: 'mr_medium', label: 'Medium', description: 'Balanced' },
|
{ value: 'mr_medium', label: 'Medium', description: 'Balanced' },
|
||||||
@@ -245,6 +250,7 @@ export function LLMParametersEditor(props: {
|
|||||||
llmVndAntWebSearch,
|
llmVndAntWebSearch,
|
||||||
llmVndAntWebSearchMaxUses,
|
llmVndAntWebSearchMaxUses,
|
||||||
llmVndGemEffort,
|
llmVndGemEffort,
|
||||||
|
llmVndGeminiAgentViz,
|
||||||
llmVndGeminiAspectRatio,
|
llmVndGeminiAspectRatio,
|
||||||
llmVndGeminiCodeExecution,
|
llmVndGeminiCodeExecution,
|
||||||
llmVndGeminiGoogleSearch,
|
llmVndGeminiGoogleSearch,
|
||||||
@@ -687,6 +693,19 @@ export function LLMParametersEditor(props: {
|
|||||||
/>
|
/>
|
||||||
)}
|
)}
|
||||||
|
|
||||||
|
{showParam('llmVndGeminiAgentViz') && (
|
||||||
|
<FormSelectControl
|
||||||
|
title='Visualizations'
|
||||||
|
tooltip='Charts and images in Deep Research reports. Disable for text-only output (helpful when merging multiple reports).'
|
||||||
|
value={llmVndGeminiAgentViz ?? _UNSPECIFIED}
|
||||||
|
onChange={(value) => {
|
||||||
|
if (value === _UNSPECIFIED || !value) onRemoveParameter('llmVndGeminiAgentViz');
|
||||||
|
else onChangeParameter({ llmVndGeminiAgentViz: value });
|
||||||
|
}}
|
||||||
|
options={_geminiAgentVizOptions}
|
||||||
|
/>
|
||||||
|
)}
|
||||||
|
|
||||||
|
|
||||||
{/*{showParam('llmVndMoonshotWebSearch') && (*/}
|
{/*{showParam('llmVndMoonshotWebSearch') && (*/}
|
||||||
{/* <FormSelectControl*/}
|
{/* <FormSelectControl*/}
|
||||||
|
|||||||
@@ -393,7 +393,7 @@ const _knownGeminiModels: ({
|
|||||||
isPreview: true,
|
isPreview: true,
|
||||||
chatPrice: gemini25ProPricing, // pricing not explicitly listed; using 2.5 Pro as baseline
|
chatPrice: gemini25ProPricing, // pricing not explicitly listed; using 2.5 Pro as baseline
|
||||||
interfaces: [LLM_IF_OAI_Chat, LLM_IF_OAI_Vision, LLM_IF_OAI_Reasoning, LLM_IF_GEM_Interactions],
|
interfaces: [LLM_IF_OAI_Chat, LLM_IF_OAI_Vision, LLM_IF_OAI_Reasoning, LLM_IF_GEM_Interactions],
|
||||||
parameterSpecs: [],
|
parameterSpecs: [{ paramId: 'llmVndGeminiAgentViz' }],
|
||||||
benchmark: undefined, // Deep research model, not benchmarkable on standard tests
|
benchmark: undefined, // Deep research model, not benchmarkable on standard tests
|
||||||
// 128K input, 64K output
|
// 128K input, 64K output
|
||||||
},
|
},
|
||||||
@@ -406,7 +406,7 @@ const _knownGeminiModels: ({
|
|||||||
isPreview: true,
|
isPreview: true,
|
||||||
chatPrice: gemini25ProPricing, // baseline estimate (see note above)
|
chatPrice: gemini25ProPricing, // baseline estimate (see note above)
|
||||||
interfaces: [LLM_IF_OAI_Chat, LLM_IF_OAI_Vision, LLM_IF_OAI_Reasoning, LLM_IF_GEM_Interactions],
|
interfaces: [LLM_IF_OAI_Chat, LLM_IF_OAI_Vision, LLM_IF_OAI_Reasoning, LLM_IF_GEM_Interactions],
|
||||||
parameterSpecs: [],
|
parameterSpecs: [{ paramId: 'llmVndGeminiAgentViz' }],
|
||||||
benchmark: undefined, // Deep research model, not benchmarkable on standard tests
|
benchmark: undefined, // Deep research model, not benchmarkable on standard tests
|
||||||
},
|
},
|
||||||
|
|
||||||
@@ -419,7 +419,7 @@ const _knownGeminiModels: ({
|
|||||||
isPreview: true,
|
isPreview: true,
|
||||||
chatPrice: gemini25ProPricing,
|
chatPrice: gemini25ProPricing,
|
||||||
interfaces: [LLM_IF_OAI_Chat, LLM_IF_OAI_Vision, LLM_IF_OAI_Reasoning, LLM_IF_GEM_Interactions],
|
interfaces: [LLM_IF_OAI_Chat, LLM_IF_OAI_Vision, LLM_IF_OAI_Reasoning, LLM_IF_GEM_Interactions],
|
||||||
parameterSpecs: [{ paramId: 'llmVndGeminiThinkingBudget' }],
|
parameterSpecs: [{ paramId: 'llmVndGeminiAgentViz' }, { paramId: 'llmVndGeminiThinkingBudget' }],
|
||||||
benchmark: undefined, // Deep research model, not benchmarkable on standard tests
|
benchmark: undefined, // Deep research model, not benchmarkable on standard tests
|
||||||
// Note: 128K input context, 64K output context
|
// Note: 128K input context, 64K output context
|
||||||
},
|
},
|
||||||
|
|||||||
@@ -94,6 +94,7 @@ const ModelParameterSpec_schema = z.object({
|
|||||||
// Bedrock
|
// Bedrock
|
||||||
'llmVndBedrockAPI',
|
'llmVndBedrockAPI',
|
||||||
// Gemini
|
// Gemini
|
||||||
|
'llmVndGeminiAgentViz',
|
||||||
'llmVndGeminiAspectRatio',
|
'llmVndGeminiAspectRatio',
|
||||||
'llmVndGeminiCodeExecution',
|
'llmVndGeminiCodeExecution',
|
||||||
'llmVndGeminiComputerUse',
|
'llmVndGeminiComputerUse',
|
||||||
|
|||||||
Reference in New Issue
Block a user