> ## Documentation Index
> Fetch the complete documentation index at: https://developers.telnyx.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Realtime voice and avatar conversations over WebRTC

> Connect a browser to a Telnyx AI Assistant over WebRTC: send microphone audio and camera video as media tracks, play the assistant's voice and avatar video, and exchange conversation events on a data channel or a WebSocket.

**Beta:** Realtime conversations over WebRTC are in beta.

Connect a browser or other WebRTC client directly to a [Telnyx AI Assistant](https://portal.telnyx.com/#/ai/assistants). Media travels as WebRTC tracks: the caller's microphone (and optionally camera) to Telnyx, the assistant's voice (and optionally its [avatar](https://developers.telnyx.com/docs/inference/ai-assistants/avatars.md) video) back. Conversation events travel as JSON on a WebRTC data channel named `oai-events`, or on a WebSocket when the call is [set up over one](#set-up-the-call-over-a-websocket).

The assistant's behavior — instructions, model, voice, tools, transcription and interruption settings — is configured on the assistant itself. The WebRTC connection carries media and events only.

Use WebRTC for browser and mobile clients. For a server-side integration that streams PCM audio, use the [conversation WebSocket](https://developers.telnyx.com/docs/inference/ai-assistants/realtime-conversations.md) instead.

## How a call is set up

Set up a call in one of two ways. Media, avatars, the camera and the conversation events are the same in both.

| Setup | Offer and answer | Conversation events | TURN credentials |
| - | - | - | - |
| [`POST /calls`](#create-the-call) | The application server forwards the browser's SDP offer over HTTP. | The `oai-events` data channel. | The client obtains them. |
| [WebSocket](#set-up-the-call-over-a-websocket) | The client sends the offer on the conversation WebSocket. | The WebSocket. | Telnyx sends them when the WebSocket connects. |

With `POST /calls`:

```
Browser                         Application server                  Telnyx
   │ 1. create offer (mic, data channel,                               │
   │    optional video), gather ICE                                    │
   │── 2. SDP offer ──────────────────▶│                               │
   │                                   │── 3. POST /calls (API key) ──▶│
   │                                   │◀─ 4. 201 SDP answer ──────────│
   │◀─ 5. SDP answer ──────────────────│                               │
   │══ 6. media + oai-events data channel, relayed through Telnyx TURN ═│
```

1. The browser creates an `RTCPeerConnection` that relays through Telnyx TURN, adds the microphone track, the `oai-events` data channel and, for an avatar or camera, a video transceiver. It waits for ICE gathering to complete.
2. The browser sends the SDP offer to the application server.
3. The application server forwards the offer to `POST /v2/ai/assistants/{assistant_id}/calls` with its Telnyx API key. The API key never reaches the browser.
4. Telnyx starts a conversation with the assistant and returns the SDP answer.
5. The application server returns the answer to the browser, which applies it.
6. Media and the data channel connect. The conversation, including the assistant's greeting, starts once the `oai-events` data channel opens.

### Migrate an existing avatar client

Replace the avatar WebSocket connection and its signaling frames with the call setup above. Do not send `avatar.session.ready`, `avatar.webrtc.offer`, `avatar.ice_candidate`, `avatar.webrtc.answer`, or `avatar.media.ready` over the data channel. Mid-call renegotiation uses the separate `webrtc.offer` and `webrtc.answer` events described below.

The greeting starts when the data channel opens, rather than waiting for the previous `avatar.media.ready` playback acknowledgement. Handle rejected media-element `play()` promises and provide a user action to start playback. An open data channel does not establish that the browser is playing the media; autoplay restrictions can prevent the user from hearing the beginning of the greeting.

## Configure ICE servers

Telnyx relays the call's media only through Telnyx TURN, and Telnyx TURN relays only between Telnyx addresses. Configure the client to relay through Telnyx TURN with `iceTransportPolicy: "relay"`. Direct connections and third-party TURN servers do not connect.

```js theme={null}
const pc = new RTCPeerConnection({
  iceServers: [
    {
      urls: [
        "turn:turn.telnyx.com:3478?transport=udp",
        "turn:turn.telnyx.com:3478?transport=tcp",
      ],
      username: TURN_USERNAME,
      credential: TURN_CREDENTIAL,
    },
  ],
  iceTransportPolicy: "relay",
});
```

Telnyx TURN requires credentials. See [STUN/TURN servers](https://developers.telnyx.com/docs/voice/sip-trunking/network-configuration/stun-turn-servers.md) for the endpoints and how to obtain credentials. These credentials authenticate the browser to the relay; they are separate from the Telnyx API key used by the application server.

The configuration above requires access to `turn.telnyx.com` on UDP or TCP port 3478. TCP on port 3478 is not an HTTPS/443 fallback. If a firewall or VPN blocks both transports, the call cannot connect even when HTTPS requests succeed. Handle ICE gathering and connection failures in the client, and release local media tracks when setup fails.

## Create the offer

Add the tracks and the data channel before creating the offer, then wait for ICE gathering to complete: Telnyx does not accept trickled candidates, so the offer must carry them.

```js theme={null}
const mic = await navigator.mediaDevices.getUserMedia({
  audio: { echoCancellation: true, noiseSuppression: true, autoGainControl: true },
});
pc.addTrack(mic.getAudioTracks()[0], mic);

// Receive the avatar's video. Omit for a voice-only call.
pc.addTransceiver("video", { direction: "recvonly" });

const events = pc.createDataChannel("oai-events");
events.onmessage = (message) => handleEvent(JSON.parse(message.data));

// One stream per element: the audio and video tracks can share a remote
// stream, and the video element must not play the audio a second time.
pc.ontrack = ({ track }) => {
  const element = track.kind === "video" ? avatarVideo : assistantAudio;
  element.srcObject = new MediaStream([track]);
};

const iceGatheringComplete = (pc) =>
  new Promise((resolve) => {
    if (pc.iceGatheringState === "complete") return resolve();
    pc.addEventListener("icegatheringstatechange", () => {
      if (pc.iceGatheringState === "complete") resolve();
    });
  });

try {
  await pc.setLocalDescription(await pc.createOffer());
  await iceGatheringComplete(pc);

  const response = await fetch("/session", {
    method: "POST",
    headers: { "Content-Type": "application/sdp" },
    body: pc.localDescription.sdp,
  });
  if (!response.ok) throw new Error(await response.text());
  await pc.setRemoteDescription({ type: "answer", sdp: await response.text() });
} catch (error) {
  // Release the microphone (and camera) when the call cannot be set up.
  pc.getSenders().forEach((sender) => sender.track?.stop());
  pc.close();
  throw error;
}
```

The offer must include:

| Element | Requirement |
| - | - |
| Audio | One audio m-line with Opus, carrying the microphone. The assistant's voice returns on the same m-line. |
| Data channel | A data channel labeled `oai-events`, created by the client. Other data channels are ignored. |
| Video | Optional, one video m-line with VP8. Use `recvonly` to receive an avatar, `sendrecv` to also send the camera, or `sendonly` to send the camera without an avatar. |

All m-lines must share one transport (`a=group:BUNDLE`), as browsers do by default.

## Create the call

The application server forwards the offer with its Telnyx API key:

```text theme={null}
POST https://api.telnyx.com/v2/ai/assistants/{assistant_id}/calls
Authorization: Bearer YOUR_TELNYX_API_KEY
```

Optional query parameter:

| Parameter | Description |
| - | - |
| `avatar` | Set to `false` to start the call without the assistant's [avatar](#show-an-avatar), even when the offer includes a video m-line that could receive it. Default: `true`. |

Send the offer as a `multipart/form-data` body with an `sdp` field, or as a raw `application/sdp` body. The multipart form also accepts an optional `session` field to pass [dynamic variables](https://developers.telnyx.com/docs/inference/ai-assistants/dynamic-variables.md) for this conversation:

```json theme={null}
{
  "assistant": {
    "dynamic_variables": { "customer_name": "Ada", "plan": "premium" }
  }
}
```

A successful request returns `201 Created` with the SDP answer as an `application/sdp` body. Return the answer to the browser unchanged.

Every request to the relay starts a call billed to the Telnyx account. Authenticate and authorize the browser's user before forwarding the offer, and rate-limit calls per user.

```js theme={null}
// Node.js (Express): relay the browser's offer to Telnyx.
// requireUser is the application's own authentication middleware.
app.post("/session", requireUser, express.text({ type: "application/sdp" }), async (req, res) => {
  const form = new FormData();
  form.set("sdp", req.body);
  form.set("session", JSON.stringify({ assistant: { dynamic_variables: { customer_name: "Ada" } } }));

  const telnyx = await fetch(
    `https://api.telnyx.com/v2/ai/assistants/${process.env.ASSISTANT_ID}/calls`,
    { method: "POST", headers: { Authorization: `Bearer ${process.env.TELNYX_API_KEY}` }, body: form },
  );

  res.status(telnyx.status).type(telnyx.headers.get("content-type") ?? "text/plain").send(await telnyx.text());
});
```

Client-supplied dynamic variables override same-named assistant defaults or values returned by the dynamic-variable webhook. Unspecified variables keep those values.

### Errors

Errors return a JSON error object with `error.code`:

| Code | HTTP status | Meaning |
| - | - | - |
| `invalid_offer` | 400 | The SDP offer is missing, larger than 65,536 bytes, or cannot be negotiated, for example without Opus audio. |
| `invalid_session` | 400 | `session` is not a JSON object. |
| `invalid_avatar` | 400 | The `avatar` query parameter is not `true` or `false`. |
| `unauthorized` | 401 | The API key is missing or invalid. |
| `assistant_not_found` | 404 | The assistant does not exist or is not available to this account. |
| `data_locality_mismatch` | 422 | This region cannot serve the assistant's in-transit data locality. |
| `in_transit_data_locality_unsupported` | 422 | In-transit data locality is not supported for this session. |
| `webrtc_unavailable` | 503 | Telnyx could not set up media for the call. Retry later. |

## Set up the call over a WebSocket

Instead of `POST /calls`, a client can open the conversation WebSocket with `transport=webrtc`, receive the ICE servers to use, and send its offer on the socket. Media still travels over WebRTC; every conversation event travels on the WebSocket, and the offer needs no data channel.

```
Client                                                             Telnyx
   │── 1. connect wss://.../conversation?transport=webrtc ────────────▶│
   │◀─ 2. webrtc.config (ICE servers) ──────────────────────────────────│
   │   3. create offer (mic, optional video), gather ICE               │
   │── 4. webrtc.offer ────────────────────────────────────────────────▶│
   │◀─ 5. webrtc.answer ────────────────────────────────────────────────│
   │══ 6. media relayed through Telnyx TURN; events on the WebSocket ══│
```

### Connect

```text theme={null}
wss://api.telnyx.com/v2/ai/assistants/{assistant_id}/conversation?transport=webrtc
Authorization: Bearer YOUR_TELNYX_API_KEY
```

Authenticate the handshake as for the [conversation WebSocket](https://developers.telnyx.com/docs/inference/ai-assistants/realtime-conversations.md), with an `Authorization` header. Browsers cannot set headers on a WebSocket; they pass the credential as WebSocket subprotocols instead: the `telnyx-auth-ws` marker, then `base64url.bearer.telnyx.` followed by the credential in base64url without padding. Send either the header or the subprotocol credential, not both.

```js theme={null}
const encoded = btoa(credential).replace(/\+/g, "-").replace(/\//g, "_").replace(/=+$/, "");
const ws = new WebSocket(
  `wss://api.telnyx.com/v2/ai/assistants/${assistantId}/conversation?transport=webrtc`,
  ["telnyx-auth-ws", `base64url.bearer.telnyx.${encoded}`],
);
```

The credential authenticates as the Telnyx account. Do not ship an API key in a public web page: for untrusted users, connect the browser through the application server, which adds the `Authorization` header after authenticating and rate-limiting the user as for [`POST /calls`](#create-the-call).

| Query parameter | Description |
| - | - |
| `transport` | Required. `webrtc`. |
| `avatar` | Set to `false` to start the call without the assistant's [avatar](#show-an-avatar). Default: `true`. |

A refused connection fails the handshake with the HTTP status and error code of [`POST /calls`](#errors), for example `401 unauthorized` or `404 assistant_not_found`.

### Receive the ICE servers

Telnyx speaks first, with the ICE servers and transport policy for the call:

```json theme={null}
{
  "type": "webrtc.config",
  "ice_servers": [
    {
      "urls": ["turn:turn.telnyx.com:3478?transport=udp"],
      "username": "...",
      "credential": "..."
    }
  ],
  "ice_transport_policy": "relay"
}
```

Create the `RTCPeerConnection` with them, then add the tracks as in [Create the offer](#create-the-offer), without the `oai-events` data channel, and wait for ICE gathering to complete:

```js theme={null}
// ws: the WebSocket opened above, or one to the application server, which adds Authorization.
let pc;

ws.onmessage = async (message) => {
  const event = JSON.parse(message.data);

  if (event.type === "webrtc.config") {
    pc = new RTCPeerConnection({
      iceServers: event.ice_servers,
      iceTransportPolicy: event.ice_transport_policy,
    });
    // Add the microphone, the optional video transceiver and pc.ontrack as above.
    await pc.setLocalDescription(await pc.createOffer());
    await iceGatheringComplete(pc);
    ws.send(JSON.stringify({ type: "webrtc.offer", sdp: pc.localDescription.sdp }));
  } else if (event.type === "webrtc.answer") {
    await pc.setRemoteDescription({ type: "answer", sdp: event.sdp });
  } else {
    handleEvent(event);
  }
};
```

### Send the offer

```json theme={null}
{
  "type": "webrtc.offer",
  "sdp": "v=0\r\n...",
  "session": {
    "assistant": {
      "dynamic_variables": { "customer_name": "Ada" }
    }
  }
}
```

`session` is optional and takes the same [dynamic variables](https://developers.telnyx.com/docs/inference/ai-assistants/dynamic-variables.md) as `POST /calls`. Telnyx replies with the SDP answer and the call's id:

```json theme={null}
{ "type": "webrtc.answer", "sdp": "v=0\r\n...", "call_id": "rtc_..." }
```

Apply the answer. The conversation, including the assistant's greeting, starts once the media connects, and `session.created` arrives on the WebSocket. Events sent before then are held and delivered in order once the conversation starts.

### Signaling errors

A refused offer returns an `error` event; the WebSocket stays open and the client may send another offer.

| Code | Meaning |
| - | - |
| `invalid_offer` | The SDP offer is missing, larger than 65,536 bytes, or cannot be negotiated. |
| `invalid_session` | `session` is not a JSON object. |
| `webrtc_unavailable` | Telnyx could not set up media for the call. Retry later. |
| `webrtc_offer_required` | An event other than `webrtc.offer` arrived before the call started. |
| `frame_too_large` | A message exceeds 1 MiB. |
| `session_not_ready` | More than 64 events arrived before the media connected. |

### Events, renegotiation and ending the call

* The WebSocket carries the same events as the data channel, with the [same differences](#exchange-events-on-the-data-channel) from the conversation WebSocket: microphone audio and the assistant's voice travel on the media tracks. Each message can be up to 1 MiB; the data channel's SCTP limit does not apply.
* [Add video during a call](#add-video-during-a-call) by sending `webrtc.offer` on the WebSocket; Telnyx replies with `webrtc.answer` there. The offer does not need a data channel.
* Close the WebSocket to end the call. When the conversation ends, Telnyx closes the WebSocket: with code `1000` normally, or `1008` or `1011` when the session failed, after an `error` event. Close the peer connection and stop the local tracks as in [End the call](#end-the-call).
* The media must connect within 20 seconds of the offer; otherwise Telnyx ends the call.

## Exchange events on the data channel

The `oai-events` data channel carries the same JSON events as the [conversation WebSocket](https://developers.telnyx.com/docs/inference/ai-assistants/realtime-conversations.md), with these differences:

| WebSocket | WebRTC |
| - | - |
| `input_audio_buffer.append` frames carry microphone audio. | The microphone track carries the audio. Do not send `input_audio_buffer.append`. |
| `response.output_audio.delta` frames carry the assistant's voice. | The audio track carries the voice. The data channel receives `response.output_audio.done` but no audio deltas. |
| The first `session.update` frame passes dynamic variables. | The `session` field of the call request passes them. Do not send `session.update`. |
| `session.created` reports the negotiated PCM formats. | `session.created` arrives when the conversation starts; audio formats are negotiated in SDP. |

Every other event works as on the WebSocket, including speech detection and transcripts, [text turns](https://developers.telnyx.com/docs/inference/ai-assistants/realtime-conversations.md#send-text-instead-of-audio), [interruption and barge-in](https://developers.telnyx.com/docs/inference/ai-assistants/realtime-conversations.md#interruption-and-barge-in), `response.cancel`, [tool calls](https://developers.telnyx.com/docs/inference/ai-assistants/realtime-conversations.md#tool-calls) and [vision frames](https://developers.telnyx.com/docs/inference/ai-assistants/realtime-conversations.md#add-vision-with-camera-or-screen-share-frames). Keep each serialized message below **1 MiB** and within the connection's negotiated [`pc.sctp.maxMessageSize`](https://www.w3.org/TR/webrtc/#dom-rtcsctptransport-maxmessagesize). The SCTP limit can be smaller than the Telnyx event limit. Measure the UTF-8 bytes of the entire JSON string, including any base64 image and metadata, before calling `send()`.

The assistant's voice must stream PCM at 8, 12, 16, 24 or 48 kHz. Otherwise the call starts with an `error` event on the data channel:

| Code | Meaning |
| - | - |
| `unsupported_voice_output_format` | The assistant's voice streams a compressed format, for example mp3 or opus. Configure a PCM voice. |
| `unsupported_voice_output_rate` | The assistant's voice streams PCM at a rate WebRTC cannot play, for example 22.05 or 44.1 kHz. Configure a voice at 8, 12, 16, 24 or 48 kHz. |

## Show an avatar

**Beta:** AI Assistant avatars are in beta. See the [Avatars guide](https://developers.telnyx.com/docs/inference/ai-assistants/avatars.md) for configuration, voice compatibility and pricing.

When the assistant has saved `avatar_settings` and the offer includes a video m-line that can receive VP8 (`recvonly` or `sendrecv`), the avatar's video arrives on that video track and its voice on the audio track, already synchronized. Play both. An offer without a receivable video m-line starts a voice-only call, even when the assistant has an avatar. To keep a receivable video m-line, for example to [send the camera](#send-the-camera), without the avatar, create the call with `?avatar=false`; a video m-line [added during the call](#add-video-during-a-call) then does not start the avatar either.

The data channel reports the avatar's state with `avatar.updated`:

```json theme={null}
{ "type": "avatar.updated", "state": "active" }
```

| `state` | Meaning |
| - | - |
| `starting` | The avatar is being prepared. |
| `active` | The avatar's video and voice are playing. |
| `unavailable` | The avatar could not start or stopped. The call continues with the assistant's voice on the same audio track; no action is needed besides hiding the video. |
| `off` | The call has no avatar. |

A runtime avatar failure also sends an `avatar.error` event with `error.fatal: false`. The conversation continues; the voice switches back on the same audio track automatically.

### Avatar errors

| Code | Meaning |
| - | - |
| `avatar_unavailable` | The avatar could not start or stopped. `error.reason` names the cause, `error.retryable` says whether a new call may get it, and `error.fatal: false` means the conversation continues with the assistant's voice. |

## Send the camera

To let the assistant see the caller, send the camera on the video m-line instead of `recvonly`. Add the camera track before creating the offer, in place of the `recvonly` transceiver:

```js theme={null}
const camera = await navigator.mediaDevices.getUserMedia({ video: true });
pc.addTrack(camera.getVideoTracks()[0], camera);
```

The m-line is then `sendrecv`: the camera goes up and the avatar, if any, comes down on the same m-line. Telnyx samples the camera every 2 seconds into a JPEG of at most 1280 pixels wide, tagged `camera`, and buffers it for the [vision tool](https://developers.telnyx.com/docs/inference/ai-assistants/vision.md). The assistant needs the `vision` tool and a vision-capable model; buffered frames do not trigger responses on their own. Image frames sent on the data channel are buffered the same way.

For camera or screen snapshots sent as image events, reduce JPEG quality or image dimensions until the complete message fits both data-channel limits. Reducing quality or dimensions can make small text and fine detail harder to interpret. These message-size limits apply to image events, not to the native camera RTP track described above.

Send only while the data channel is open, monitor `bufferedAmount` to avoid accumulating stale snapshots, and handle `send()` exceptions. If a frame cannot fit or be sent, surface the failure in the application's camera-sharing UI; reducing the image and retrying are client responsibilities. A successful `send()` queues the message locally and does not confirm that the assistant has used the image.

## Add video during a call

A call that started without a video m-line can add one. Add the transceiver, set the new offer as the local description, wait for ICE gathering, and send it on the data channel as a `webrtc.offer` event. Telnyx replies with a `webrtc.answer` event; apply it with `setRemoteDescription`:

```js theme={null}
pc.addTransceiver("video", { direction: "recvonly" });
await pc.setLocalDescription(await pc.createOffer());
await iceGatheringComplete(pc);
events.send(JSON.stringify({ type: "webrtc.offer", sdp: pc.localDescription.sdp }));

// In the data channel's message handler:
if (event.type === "webrtc.answer") {
  await pc.setRemoteDescription({ type: "answer", sdp: event.sdp });
}
```

If the assistant has an avatar, it starts once the new video m-line can receive it. Send one offer at a time and keep a single video m-line. A refused offer returns an `error` event instead:

| Code | Meaning |
| - | - |
| `invalid_offer` | The offer was not applied. Roll back the local description; the call continues unchanged. |
| `renegotiation_in_progress` | A previous offer is still being answered. Wait for its answer. |
| `renegotiation_failed` | The offer was applied but could not be answered. The two sides no longer agree; end the call. |

## End the call

Close the peer connection to end the call. Telnyx ends the conversation as soon as the `oai-events` data channel closes. When a client disappears without closing, for example after a network loss, Telnyx ends the conversation within seconds once media stops arriving.

```js theme={null}
// Stop every local track (microphone and, if sent, camera), then close.
pc.getSenders().forEach((sender) => sender.track?.stop());
pc.close();
```

The data channel must open within 20 seconds of the call request; otherwise Telnyx ends the call.