> ## Documentation Index
> Fetch the complete documentation index at: https://developers.telnyx.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Vision

> Analyze images sent through MMS or the chat widget, or use the BETA vision tool to inspect streamed camera and screen-share frames during live conversations.

AI Assistants use images as visual context through MMS, chat-widget messages, and live camera or screen-share snapshots. Each path uses a model that supports image input.

| Vision option | When to use it | Setup |
| - | - | - |
| MMS images | A caller sends a photo, screenshot, or document for the assistant to discuss during a voice call. | Enable messaging for the assistant and configure an image-capable model. See [MMS integration](https://developers.telnyx.com/docs/inference/ai-assistants/no-code-voice-assistant.md#mms-integration-during-voice-calls). |
| Chat-widget images | A website visitor shares an image as part of a chat conversation. | Configure an image-capable model and use the image-message support in the [embedded widget](https://developers.telnyx.com/docs/inference/ai-assistants/no-code-voice-assistant.md#embeddable-widget). |
| Vision tool (BETA) | A meeting bot or customer-service application provides a changing camera or screen-share view during a live conversation. | Add the `vision` tool to the assistant and connect a client or integration that captures and streams image snapshots. |

## Vision tool for live conversations

Use the `vision` tool when visual context changes as the conversation progresses: a customer demonstrates an equipment problem, a participant presents a document in a meeting, or a user shares a screen while troubleshooting. The client sends periodic snapshots, and the assistant chooses when to inspect a recent frame based on the conversation — for example, when the user asks "what am I holding?" or "can you read this error?"

Frames are buffered as they arrive, so the client can keep the view current without triggering an answer for every snapshot. The assistant retrieves visual context on demand by calling `vision`.

> [!WARNING]
>
> **BETA:** The `vision` tool for buffered frames is in beta. Its configuration and behavior may change before general availability.

Setting up this path involves both the assistant and the media source: configure the tool, then have your client or integration capture and send snapshots through the [realtime conversation WebSocket](https://developers.telnyx.com/docs/inference/ai-assistants/realtime-conversations.md), the Voice SDK, or the [assistant event stream](https://developers.telnyx.com/docs/inference/ai-assistants/conversation-event-stream.md). The model receives individual image snapshots from the camera or screen share. The following sections cover the tool configuration, frame format, and transport examples.

## How buffered frames work

1. **Your client sends frames** with a `conversation.item.create` frame whose item has `"buffer": true` and `input_image` content. Telnyx buffers recent frames for the conversation, in memory only.
2. **Nothing else happens.** Buffered frames never start an assistant response and are never added to the conversation history, transcripts, or webhooks.
3. **The assistant decides when to look.** With the vision tool enabled, the assistant chooses when to look and which buffered frames it needs. Frames reach the model only when it looks.
4. **Stale frames are ignored.** Frames that Telnyx received more than `max_frame_age_secs` ago (10 seconds by default) are not shown to the assistant, so it never describes a frozen or disconnected camera as live.

## Model support

Vision works in voice-only and avatar conversations, over the [conversation WebSocket](https://developers.telnyx.com/docs/inference/ai-assistants/realtime-conversations.md) or [WebRTC](https://developers.telnyx.com/docs/inference/ai-assistants/webrtc-conversations.md). An avatar is optional and does not affect whether the assistant can inspect images. Over the WebSocket, the client captures and sends image frames; over WebRTC, it can also [send the camera track](https://developers.telnyx.com/docs/inference/ai-assistants/webrtc-conversations.md#send-the-camera).

The vision tool requires image support from either the assistant's base model or its enabled delegation model. A speech-to-speech (S2S) assistant can use a vision-capable delegation model even when its speaking model does not support image input.

For Telnyx-managed delegation, set `delegation_settings.enabled` to `true`, `delegation_settings.mode` to `"telnyx"`, and `delegation_settings.model` to an available model that supports image input. Keep the `vision` tool attached to the assistant. The delegation model inspects buffered frames and returns its result to the speaking model. See [Delegation](https://developers.telnyx.com/docs/inference/ai-assistants/delegation.md) for backend configuration.

Selecting a vision-capable model in disabled delegation settings does not enable vision. Client delegation (`mode: "client"`) is handled by the connected application; it does not run the configured Telnyx delegation model.

## Enable the vision tool

To try vision with an avatar, start with the **Avatar with vision** template in the [Portal](https://portal.telnyx.com/#/ai/assistants). It configures an avatar, the Vision tool, and a model that supports image input. Enable your camera when testing the assistant in the Portal, then ask about something you are showing. For your own integration, capture and send image frames using one of the transports below.

Add a `vision` tool to the assistant. Every setting is optional:

```json theme={null}
{
  "tools": [
    { "type": "vision", "vision": { "max_frame_age_secs": 10 } }
  ]
}
```

| Field | Default | Description |
| - | - | - |
| `max_frame_age_secs` | `10` | Frames received longer ago than this are not shown to the assistant. Minimum `1`. |

`{ "type": "vision" }` on its own is valid. An assistant can have at most one vision tool. You can also add it in the [Portal](https://portal.telnyx.com/#/ai/assistants) under **Tools → Add Tool → Vision (BETA)**.

### What the model sees

The tool you configure above becomes a single function, `vision`, that the model can call. It takes two optional arguments:

| Argument | Values | Effect |
| - | - | - |
| `action` | `"look"` (default) or `"list"` | `"look"` shows the model a recent frame. `"list"` returns the [tags](#tag-frames) it has frames for and how long ago each last received one, without showing an image. |
| `tag` | A tag, as listed | With `"look"`, shows the most recent frame with that tag. Without it, the most recent frame of any tag. |

When a look finds a fresh frame, the image is shown to the model for the rest of that response. Otherwise the model is told that no recent frame is available or, for a tag it has no frames for, which tags are available.

The function name, its description, and its arguments are fixed; only `max_frame_age_secs` is configurable. To steer when and how the model looks, use the assistant's instructions, and refer to the tool and its arguments by these names. For example:

```text theme={null}
The user may share their camera and their screen. When they ask about something
they are showing, call the vision tool before answering. Use tag "screenshare"
for questions about their screen and "camera" for anything they hold up. Never
describe what you see without calling vision first.
```

Name the tags your client actually sends, so the model does not need to call `list` first.

## Send frames

A buffered item is a user `message` with `"buffer": true` and one or more `input_image` parts:

```json theme={null}
{
  "type": "conversation.item.create",
  "item": {
    "type": "message",
    "role": "user",
    "buffer": true,
    "captured_at": "2026-10-12T14:03:12.120Z",
    "tags": ["oliver's camera"],
    "content": [
      { "type": "input_image", "image_url": "data:image/jpeg;base64,/9j/4AAQSkZJRgABAQ..." }
    ]
  }
}
```

| Field | Required | Description |
| - | :-: | - |
| `buffer` | ✅ | Must be the boolean `true` for this buffered-frame protocol. Image items without it are not supported on these conversation sockets. |
| `content` | ✅ | One or more `input_image` parts and nothing else. `image_url` is a `data:image/jpeg`, `data:image/png` or `data:image/webp` base64 URL, or an `https` URL. |
| `captured_at` | | When the client captured the frame (ISO 8601). Defaults to when Telnyx received it. Shown to the assistant; freshness is measured from when Telnyx received the frame, so client clock skew does not matter. |
| `tags` | | Free-form labels your client chooses, applied to every image in the item — for example the image source (`"oliver's camera"`, `"oliver's screenshare"`). Up to 5 tags per item, each up to 64 characters of plain text. `untagged` is reserved. See [Tag frames](#tag-frames). |

If any part is invalid — for example a text part, an unsupported image type, an `http` URL, a malformed `captured_at`, or an invalid tag — the whole item is rejected and nothing is buffered.

The 1 MiB WebSocket limit applies to the entire serialized message, including base64 image data and JSON overhead. A message that exceeds this limit is rejected before buffered-item validation. The realtime socket reports `frame_too_large`; the assistant event-stream socket returns an error and closes the connection.

> [!TIP]
>
> Downscale frames before sending (for example 640×480 JPEG). Keep the complete message below 1 MiB; base64 encoding increases the image size by roughly one third before JSON overhead. Smaller images are also faster for the model to read. One frame every 1–5 seconds is plenty for most live conversations.

### Tag frames

Tags are free-form labels your client chooses; Telnyx does not interpret them. The assistant can list the tags it has frames for and look at the frames with a given tag. A common use is telling visual sources apart, for example a participant's camera and their screen share in a meeting:

```json theme={null}
{ "type": "conversation.item.create",
  "item": { "type": "message", "role": "user", "buffer": true,
            "tags": ["oliver's screenshare"],
            "content": [{ "type": "input_image", "image_url": "data:image/jpeg;base64,..." }] } }
```

* Recent frames are buffered per tag, so a tag that receives frames often does not crowd out one that receives them rarely.
* The assistant picks a tag from the list when it needs one ("what is on Oliver's screen?"). Tag lookups ignore case, spaces, and punctuation.
* Without a tag request, the assistant looks at the most recent frame of any tag. Frames sent without `tags` are grouped together under the reserved name `untagged`, so clients that do not tag frames need no changes.
* Choose descriptive tags: the assistant sees them as written.
* Freshness applies per frame. Keep sending frames for content that does not change (such as a static screen share), or the assistant treats it as no longer available after `max_frame_age_secs`.

Tags need no extra tool configuration: the same single `vision` tool lists and looks up tags (see [What the model sees](#what-the-model-sees)).

### WebRTC (Voice SDK)

Send frames over the active call with `sendAIConversationMessage`. The SDK forwards the item as-is; in TypeScript its published `AIConversationOutboundItem` type does not list buffered image items yet, so cast the item until it does:

```typescript theme={null}
const canvas = document.createElement('canvas');
canvas.width = 640;
canvas.height = 480;
const context = canvas.getContext('2d');
if (!context) throw new Error('2D canvas is unavailable');

setInterval(() => {
  context.drawImage(videoElement, 0, 0, canvas.width, canvas.height);

  call.sendAIConversationMessage({
    type: 'message',
    role: 'user',
    buffer: true,
    captured_at: new Date().toISOString(),
    content: [{ type: 'input_image', image_url: canvas.toDataURL('image/jpeg', 0.7) }],
  } as any); // until the SDK type includes buffered image items
}, 2000);
```

### Realtime WebSocket

On a [realtime conversation](https://developers.telnyx.com/docs/inference/ai-assistants/realtime-conversations.md) socket, send the `conversation.item.create` frame shown above after `session.created`. For messages within the size limit, an invalid buffered item is answered with an `invalid_frame` error; the socket stays open.

### Assistant event stream

On the customer-configured [conversation event stream](https://developers.telnyx.com/docs/inference/ai-assistants/conversation-event-stream.md) socket, send the same `conversation.item.create` frame. An invalid buffered item is answered with an `error` frame. Messages exceeding the 1 MiB transport limit also close the connection.

## Privacy

The following describes the buffered-frame path. It does not describe storage or retention of MMS attachments or chat-widget image messages.

The vision tool buffers image frames in memory for the running conversation and discards the buffer when it ends. Buffered images are not added to conversation history, transcripts, or webhooks, and reach the model only when the assistant invokes the tool.

This does not disable or change recording. Separate video recordings — including recordings made through [Telnyx Meeting](https://developers.telnyx.com/docs/meeting/recording.md) — can retain the visual content shown during the conversation. Those recordings are stored and managed separately from the vision buffer. Telnyx Meeting sessions record automatically; see [Recording](https://developers.telnyx.com/docs/meeting/recording.md) for retention and deletion.