Vision

Analyze images sent through MMS or the chat widget, or use the BETA vision tool to inspect streamed camera and screen-share frames during live conversations.

AI Assistants use images as visual context through MMS, chat-widget messages, and live camera or screen-share snapshots. Each path uses a model that supports image input.

Vision option	When to use it	Setup
MMS images	A caller sends a photo, screenshot, or document for the assistant to discuss during a voice call.	Enable messaging for the assistant and configure an image-capable model. See MMS integration.
Chat-widget images	A website visitor shares an image as part of a chat conversation.	Configure an image-capable model and use the image-message support in the embedded widget.
Vision tool (BETA)	A meeting bot or customer-service application provides a changing camera or screen-share view during a live conversation.	Add the vision tool to the assistant and connect a client or integration that captures and streams image snapshots.

Vision tool for live conversations

Use the vision tool when visual context changes as the conversation progresses: a customer demonstrates an equipment problem, a participant presents a document in a meeting, or a user shares a screen while troubleshooting. The client sends periodic snapshots, and the assistant chooses when to inspect a recent frame based on the conversation — for example, when the user asks “what am I holding?” or “can you read this error?”

Frames are buffered as they arrive, so the client can keep the view current without triggering an answer for every snapshot. The assistant retrieves visual context on demand by calling vision.

BETA: The vision tool for buffered frames is in beta. Its configuration and behavior may change before general availability.

Setting up this path involves both the assistant and the media source: configure the tool, then have your client or integration capture and send snapshots through the realtime conversation WebSocket, the Voice SDK, or the assistant event stream. The model receives individual image snapshots from the camera or screen share. The following sections cover the tool configuration, frame format, and transport examples.

How buffered frames work

1. Your client sends frames with a conversation.item.create frame whose item has "buffer": true and input_image content. Telnyx buffers recent frames for the conversation, in memory only.
2. Nothing else happens. Buffered frames never start an assistant response and are never added to the conversation history, transcripts, or webhooks.
3. The assistant decides when to look. With the vision tool enabled, the assistant chooses when to look and which buffered frames it needs. Frames reach the model only when it looks.
4. Stale frames are ignored. Frames that Telnyx received more than max_frame_age_secs ago (10 seconds by default) are not shown to the assistant, so it never describes a frozen or disconnected camera as live.

Model support

Vision works in voice-only and avatar conversations, over the conversation WebSocket or WebRTC. An avatar is optional and does not affect whether the assistant can inspect images. Over the WebSocket, the client captures and sends image frames; over WebRTC, it can also send the camera track.

The vision tool requires image support from either the assistant’s base model or its enabled delegation model. A speech-to-speech (S2S) assistant can use a vision-capable delegation model even when its speaking model does not support image input.

For Telnyx-managed delegation, set delegation_settings.enabled to true, delegation_settings.mode to "telnyx", and delegation_settings.model to an available model that supports image input. Keep the vision tool attached to the assistant. The delegation model inspects buffered frames and returns its result to the speaking model. See Delegation for backend configuration.

Selecting a vision-capable model in disabled delegation settings does not enable vision. Client delegation (mode: "client") is handled by the connected application; it does not run the configured Telnyx delegation model.

Enable the vision tool

To try vision with an avatar, start with the Avatar with vision template in the Portal. It configures an avatar, the Vision tool, and a model that supports image input. Enable your camera when testing the assistant in the Portal, then ask about something you are showing. For your own integration, capture and send image frames using one of the transports below.

Add a vision tool to the assistant. Every setting is optional:

{
  "tools": [
    { "type": "vision", "vision": { "max_frame_age_secs": 10 } }
  ]
}

Field	Default	Description
max_frame_age_secs	10	Frames received longer ago than this are not shown to the assistant. Minimum 1.

{ "type": "vision" } on its own is valid. An assistant can have at most one vision tool. You can also add it in the Portal under Tools → Add Tool → Vision (BETA).

What the model sees

The tool you configure above becomes a single function, vision, that the model can call. It takes two optional arguments:

Argument	Values	Effect
action	"look" (default) or "list"	"look" shows the model a recent frame. "list" returns the tags it has frames for and how long ago each last received one, without showing an image.
tag	A tag, as listed	With "look", shows the most recent frame with that tag. Without it, the most recent frame of any tag.

When a look finds a fresh frame, the image is shown to the model for the rest of that response. Otherwise the model is told that no recent frame is available or, for a tag it has no frames for, which tags are available.

The function name, its description, and its arguments are fixed; only max_frame_age_secs is configurable. To steer when and how the model looks, use the assistant’s instructions, and refer to the tool and its arguments by these names. For example:

The user may share their camera and their screen. When they ask about something
they are showing, call the vision tool before answering. Use tag "screenshare"
for questions about their screen and "camera" for anything they hold up. Never
describe what you see without calling vision first.

Name the tags your client actually sends, so the model does not need to call list first.

Send frames

A buffered item is a user message with "buffer": true and one or more input_image parts:

{
  "type": "conversation.item.create",
  "item": {
    "type": "message",
    "role": "user",
    "buffer": true,
    "captured_at": "2026-10-12T14:03:12.120Z",
    "tags": ["oliver's camera"],
    "content": [
      { "type": "input_image", "image_url": "data:image/jpeg;base64,/9j/4AAQSkZJRgABAQ..." }
    ]
  }
}

Field	Required	Description
buffer	✅	Must be the boolean true for this buffered-frame protocol. Image items without it are not supported on these conversation sockets.
content	✅	One or more input_image parts and nothing else. image_url is a data:image/jpeg, data:image/png or data:image/webp base64 URL, or an https URL.
captured_at		When the client captured the frame (ISO 8601). Defaults to when Telnyx received it. Shown to the assistant; freshness is measured from when Telnyx received the frame, so client clock skew does not matter.
tags		Free-form labels your client chooses, applied to every image in the item — for example the image source ("oliver's camera", "oliver's screenshare"). Up to 5 tags per item, each up to 64 characters of plain text. untagged is reserved. See Tag frames.

If any part is invalid — for example a text part, an unsupported image type, an http URL, a malformed captured_at, or an invalid tag — the whole item is rejected and nothing is buffered.

The 1 MiB WebSocket limit applies to the entire serialized message, including base64 image data and JSON overhead. A message that exceeds this limit is rejected before buffered-item validation. The realtime socket reports frame_too_large; the assistant event-stream socket returns an error and closes the connection.

Downscale frames before sending (for example 640×480 JPEG). Keep the complete message below 1 MiB; base64 encoding increases the image size by roughly one third before JSON overhead. Smaller images are also faster for the model to read. One frame every 1–5 seconds is plenty for most live conversations.

Tag frames

Tags are free-form labels your client chooses; Telnyx does not interpret them. The assistant can list the tags it has frames for and look at the frames with a given tag. A common use is telling visual sources apart, for example a participant’s camera and their screen share in a meeting:

{ "type": "conversation.item.create",
  "item": { "type": "message", "role": "user", "buffer": true,
            "tags": ["oliver's screenshare"],
            "content": [{ "type": "input_image", "image_url": "data:image/jpeg;base64,..." }] } }

• Recent frames are buffered per tag, so a tag that receives frames often does not crowd out one that receives them rarely.
• The assistant picks a tag from the list when it needs one (“what is on Oliver’s screen?”). Tag lookups ignore case, spaces, and punctuation.
• Without a tag request, the assistant looks at the most recent frame of any tag. Frames sent without tags are grouped together under the reserved name untagged, so clients that do not tag frames need no changes.
• Choose descriptive tags: the assistant sees them as written.
• Freshness applies per frame. Keep sending frames for content that does not change (such as a static screen share), or the assistant treats it as no longer available after max_frame_age_secs.

Tags need no extra tool configuration: the same single vision tool lists and looks up tags (see What the model sees).

WebRTC (Voice SDK)

Send frames over the active call with sendAIConversationMessage. The SDK forwards the item as-is; in TypeScript its published AIConversationOutboundItem type does not list buffered image items yet, so cast the item until it does:

const canvas = document.createElement('canvas');
canvas.width = 640;
canvas.height = 480;
const context = canvas.getContext('2d');
if (!context) throw new Error('2D canvas is unavailable');

setInterval(() => {
  context.drawImage(videoElement, 0, 0, canvas.width, canvas.height);

  call.sendAIConversationMessage({
    type: 'message',
    role: 'user',
    buffer: true,
    captured_at: new Date().toISOString(),
    content: [{ type: 'input_image', image_url: canvas.toDataURL('image/jpeg', 0.7) }],
  } as any); // until the SDK type includes buffered image items
}, 2000);

Realtime WebSocket

On a realtime conversation socket, send the conversation.item.create frame shown above after session.created. For messages within the size limit, an invalid buffered item is answered with an invalid_frame error; the socket stays open.

Assistant event stream

On the customer-configured conversation event stream socket, send the same conversation.item.create frame. An invalid buffered item is answered with an error frame. Messages exceeding the 1 MiB transport limit also close the connection.

Privacy

The following describes the buffered-frame path. It does not describe storage or retention of MMS attachments or chat-widget image messages.

The vision tool buffers image frames in memory for the running conversation and discards the buffer when it ends. Buffered images are not added to conversation history, transcripts, or webhooks, and reach the model only when the assistant invokes the tool.

This does not disable or change recording. Separate video recordings — including recordings made through Telnyx Meeting — can retain the visual content shown during the conversation. Those recordings are stored and managed separately from the vision buffer. Telnyx Meeting sessions record automatically; see Recording for retention and deletion.