Realtime voice and avatar conversations over WebRTC

Beta: Realtime conversations over WebRTC are in beta. Connect a browser or other WebRTC client directly to a Telnyx AI Assistant. Media travels as WebRTC tracks: the caller’s microphone (and optionally camera) to Telnyx, the assistant’s voice (and optionally its avatar video) back. Conversation events travel as JSON on a WebRTC data channel named oai-events, or on a WebSocket when the call is set up over one. The assistant’s behavior — instructions, model, voice, tools, transcription and interruption settings — is configured on the assistant itself. The WebRTC connection carries media and events only. Use WebRTC for browser and mobile clients. For a server-side integration that streams PCM audio, use the conversation WebSocket instead.

How a call is set up

Set up a call in one of two ways. Media, avatars, the camera and the conversation events are the same in both.

Setup	Offer and answer	Conversation events	TURN credentials
`POST /calls`	The application server forwards the browser’s SDP offer over HTTP.	The `oai-events` data channel.	The client obtains them.
WebSocket	The client sends the offer on the conversation WebSocket.	The WebSocket.	Telnyx sends them when the WebSocket connects.

With POST /calls:

Browser                         Application server                  Telnyx
   │ 1. create offer (mic, data channel,                               │
   │    optional video), gather ICE                                    │
   │── 2. SDP offer ──────────────────▶│                               │
   │                                   │── 3. POST /calls (API key) ──▶│
   │                                   │◀─ 4. 201 SDP answer ──────────│
   │◀─ 5. SDP answer ──────────────────│                               │
   │══ 6. media + oai-events data channel, relayed through Telnyx TURN ═│

The browser creates an RTCPeerConnection that relays through Telnyx TURN, adds the microphone track, the oai-events data channel and, for an avatar or camera, a video transceiver. It waits for ICE gathering to complete.
The browser sends the SDP offer to the application server.
The application server forwards the offer to POST /v2/ai/assistants/{assistant_id}/calls with its Telnyx API key. The API key never reaches the browser.
Telnyx starts a conversation with the assistant and returns the SDP answer.
The application server returns the answer to the browser, which applies it.
Media and the data channel connect. The conversation, including the assistant’s greeting, starts once the oai-events data channel opens.

Migrate an existing avatar client

Replace the avatar WebSocket connection and its signaling frames with the call setup above. Do not send avatar.session.ready, avatar.webrtc.offer, avatar.ice_candidate, avatar.webrtc.answer, or avatar.media.ready over the data channel. Mid-call renegotiation uses the separate webrtc.offer and webrtc.answer events described below. The greeting starts when the data channel opens, rather than waiting for the previous avatar.media.ready playback acknowledgement. Handle rejected media-element play() promises and provide a user action to start playback. An open data channel does not establish that the browser is playing the media; autoplay restrictions can prevent the user from hearing the beginning of the greeting.

Configure ICE servers

Telnyx relays the call’s media only through Telnyx TURN, and Telnyx TURN relays only between Telnyx addresses. Configure the client to relay through Telnyx TURN with iceTransportPolicy: "relay". Direct connections and third-party TURN servers do not connect.

const pc = new RTCPeerConnection({
  iceServers: [
    {
      urls: [
        "turn:turn.telnyx.com:3478?transport=udp",
        "turn:turn.telnyx.com:3478?transport=tcp",
      ],
      username: TURN_USERNAME,
      credential: TURN_CREDENTIAL,
    },
  ],
  iceTransportPolicy: "relay",
});

Telnyx TURN requires credentials. See STUN/TURN servers for the endpoints and how to obtain credentials. These credentials authenticate the browser to the relay; they are separate from the Telnyx API key used by the application server. The configuration above requires access to turn.telnyx.com on UDP or TCP port 3478. TCP on port 3478 is not an HTTPS/443 fallback. If a firewall or VPN blocks both transports, the call cannot connect even when HTTPS requests succeed. Handle ICE gathering and connection failures in the client, and release local media tracks when setup fails.

Create the offer

Add the tracks and the data channel before creating the offer, then wait for ICE gathering to complete: Telnyx does not accept trickled candidates, so the offer must carry them.

const mic = await navigator.mediaDevices.getUserMedia({
  audio: { echoCancellation: true, noiseSuppression: true, autoGainControl: true },
});
pc.addTrack(mic.getAudioTracks()[0], mic);

// Receive the avatar's video. Omit for a voice-only call.
pc.addTransceiver("video", { direction: "recvonly" });

const events = pc.createDataChannel("oai-events");
events.onmessage = (message) => handleEvent(JSON.parse(message.data));

// One stream per element: the audio and video tracks can share a remote
// stream, and the video element must not play the audio a second time.
pc.ontrack = ({ track }) => {
  const element = track.kind === "video" ? avatarVideo : assistantAudio;
  element.srcObject = new MediaStream([track]);
};

const iceGatheringComplete = (pc) =>
  new Promise((resolve) => {
    if (pc.iceGatheringState === "complete") return resolve();
    pc.addEventListener("icegatheringstatechange", () => {
      if (pc.iceGatheringState === "complete") resolve();
    });
  });

try {
  await pc.setLocalDescription(await pc.createOffer());
  await iceGatheringComplete(pc);

  const response = await fetch("/session", {
    method: "POST",
    headers: { "Content-Type": "application/sdp" },
    body: pc.localDescription.sdp,
  });
  if (!response.ok) throw new Error(await response.text());
  await pc.setRemoteDescription({ type: "answer", sdp: await response.text() });
} catch (error) {
  // Release the microphone (and camera) when the call cannot be set up.
  pc.getSenders().forEach((sender) => sender.track?.stop());
  pc.close();
  throw error;
}

The offer must include:

Element	Requirement
Audio	One audio m-line with Opus, carrying the microphone. The assistant’s voice returns on the same m-line.
Data channel	A data channel labeled `oai-events`, created by the client. Other data channels are ignored.
Video	Optional, one video m-line with VP8. Use `recvonly` to receive an avatar, `sendrecv` to also send the camera, or `sendonly` to send the camera without an avatar.

All m-lines must share one transport (a=group:BUNDLE), as browsers do by default.

Create the call

The application server forwards the offer with its Telnyx API key:

POST https://api.telnyx.com/v2/ai/assistants/{assistant_id}/calls
Authorization: Bearer YOUR_TELNYX_API_KEY

Optional query parameter:

Parameter	Description
`avatar`	Set to `false` to start the call without the assistant’s avatar, even when the offer includes a video m-line that could receive it. Default: `true`.

Send the offer as a multipart/form-data body with an sdp field, or as a raw application/sdp body. The multipart form also accepts an optional session field to pass dynamic variables for this conversation:

{
  "assistant": {
    "dynamic_variables": { "customer_name": "Ada", "plan": "premium" }
  }
}

A successful request returns 201 Created with the SDP answer as an application/sdp body. Return the answer to the browser unchanged. Every request to the relay starts a call billed to the Telnyx account. Authenticate and authorize the browser’s user before forwarding the offer, and rate-limit calls per user.

// Node.js (Express): relay the browser's offer to Telnyx.
// requireUser is the application's own authentication middleware.
app.post("/session", requireUser, express.text({ type: "application/sdp" }), async (req, res) => {
  const form = new FormData();
  form.set("sdp", req.body);
  form.set("session", JSON.stringify({ assistant: { dynamic_variables: { customer_name: "Ada" } } }));

  const telnyx = await fetch(
    `https://api.telnyx.com/v2/ai/assistants/${process.env.ASSISTANT_ID}/calls`,
    { method: "POST", headers: { Authorization: `Bearer ${process.env.TELNYX_API_KEY}` }, body: form },
  );

  res.status(telnyx.status).type(telnyx.headers.get("content-type") ?? "text/plain").send(await telnyx.text());
});

Client-supplied dynamic variables override same-named assistant defaults or values returned by the dynamic-variable webhook. Unspecified variables keep those values.

Errors

Errors return a JSON error object with error.code:

Code	HTTP status	Meaning
`invalid_offer`	400	The SDP offer is missing, larger than 65,536 bytes, or cannot be negotiated, for example without Opus audio.
`invalid_session`	400	`session` is not a JSON object.
`invalid_avatar`	400	The `avatar` query parameter is not `true` or `false`.
`unauthorized`	401	The API key is missing or invalid.
`assistant_not_found`	404	The assistant does not exist or is not available to this account.
`data_locality_mismatch`	422	This region cannot serve the assistant’s in-transit data locality.
`in_transit_data_locality_unsupported`	422	In-transit data locality is not supported for this session.
`webrtc_unavailable`	503	Telnyx could not set up media for the call. Retry later.

Set up the call over a WebSocket

Instead of POST /calls, a client can open the conversation WebSocket with transport=webrtc, receive the ICE servers to use, and send its offer on the socket. Media still travels over WebRTC; every conversation event travels on the WebSocket, and the offer needs no data channel.

Client                                                             Telnyx
   │── 1. connect wss://.../conversation?transport=webrtc ────────────▶│
   │◀─ 2. webrtc.config (ICE servers) ──────────────────────────────────│
   │   3. create offer (mic, optional video), gather ICE               │
   │── 4. webrtc.offer ────────────────────────────────────────────────▶│
   │◀─ 5. webrtc.answer ────────────────────────────────────────────────│
   │══ 6. media relayed through Telnyx TURN; events on the WebSocket ══│

Connect

wss://api.telnyx.com/v2/ai/assistants/{assistant_id}/conversation?transport=webrtc
Authorization: Bearer YOUR_TELNYX_API_KEY

Authenticate the handshake as for the conversation WebSocket, with an Authorization header. Browsers cannot set headers on a WebSocket; they pass the credential as WebSocket subprotocols instead: the telnyx-auth-ws marker, then base64url.bearer.telnyx. followed by the credential in base64url without padding. Send either the header or the subprotocol credential, not both.

const encoded = btoa(credential).replace(/\+/g, "-").replace(/\//g, "_").replace(/=+$/, "");
const ws = new WebSocket(
  `wss://api.telnyx.com/v2/ai/assistants/${assistantId}/conversation?transport=webrtc`,
  ["telnyx-auth-ws", `base64url.bearer.telnyx.${encoded}`],
);

The credential authenticates as the Telnyx account. Do not ship an API key in a public web page: for untrusted users, connect the browser through the application server, which adds the Authorization header after authenticating and rate-limiting the user as for POST /calls.

Query parameter	Description
`transport`	Required. `webrtc`.
`avatar`	Set to `false` to start the call without the assistant’s avatar. Default: `true`.

A refused connection fails the handshake with the HTTP status and error code of POST /calls, for example 401 unauthorized or 404 assistant_not_found.

Receive the ICE servers

Telnyx speaks first, with the ICE servers and transport policy for the call:

{
  "type": "webrtc.config",
  "ice_servers": [
    {
      "urls": ["turn:turn.telnyx.com:3478?transport=udp"],
      "username": "...",
      "credential": "..."
    }
  ],
  "ice_transport_policy": "relay"
}

Create the RTCPeerConnection with them, then add the tracks as in Create the offer, without the oai-events data channel, and wait for ICE gathering to complete:

// ws: the WebSocket opened above, or one to the application server, which adds Authorization.
let pc;

ws.onmessage = async (message) => {
  const event = JSON.parse(message.data);

  if (event.type === "webrtc.config") {
    pc = new RTCPeerConnection({
      iceServers: event.ice_servers,
      iceTransportPolicy: event.ice_transport_policy,
    });
    // Add the microphone, the optional video transceiver and pc.ontrack as above.
    await pc.setLocalDescription(await pc.createOffer());
    await iceGatheringComplete(pc);
    ws.send(JSON.stringify({ type: "webrtc.offer", sdp: pc.localDescription.sdp }));
  } else if (event.type === "webrtc.answer") {
    await pc.setRemoteDescription({ type: "answer", sdp: event.sdp });
  } else {
    handleEvent(event);
  }
};

Send the offer

{
  "type": "webrtc.offer",
  "sdp": "v=0\r\n...",
  "session": {
    "assistant": {
      "dynamic_variables": { "customer_name": "Ada" }
    }
  }
}

session is optional and takes the same dynamic variables as POST /calls. Telnyx replies with the SDP answer and the call’s id:

{ "type": "webrtc.answer", "sdp": "v=0\r\n...", "call_id": "rtc_..." }

Apply the answer. The conversation, including the assistant’s greeting, starts once the media connects, and session.created arrives on the WebSocket. Events sent before then are held and delivered in order once the conversation starts.

Signaling errors

A refused offer returns an error event; the WebSocket stays open and the client may send another offer.

Code	Meaning
`invalid_offer`	The SDP offer is missing, larger than 65,536 bytes, or cannot be negotiated.
`invalid_session`	`session` is not a JSON object.
`webrtc_unavailable`	Telnyx could not set up media for the call. Retry later.
`webrtc_offer_required`	An event other than `webrtc.offer` arrived before the call started.
`frame_too_large`	A message exceeds 1 MiB.
`session_not_ready`	More than 64 events arrived before the media connected.

Events, renegotiation and ending the call

The WebSocket carries the same events as the data channel, with the same differences from the conversation WebSocket: microphone audio and the assistant’s voice travel on the media tracks. Each message can be up to 1 MiB; the data channel’s SCTP limit does not apply.
Add video during a call by sending webrtc.offer on the WebSocket; Telnyx replies with webrtc.answer there. The offer does not need a data channel.
Close the WebSocket to end the call. When the conversation ends, Telnyx closes the WebSocket: with code 1000 normally, or 1008 or 1011 when the session failed, after an error event. Close the peer connection and stop the local tracks as in End the call.
The media must connect within 20 seconds of the offer; otherwise Telnyx ends the call.

Exchange events on the data channel

The oai-events data channel carries the same JSON events as the conversation WebSocket, with these differences:

WebSocket	WebRTC
`input_audio_buffer.append` frames carry microphone audio.	The microphone track carries the audio. Do not send `input_audio_buffer.append`.
`response.output_audio.delta` frames carry the assistant’s voice.	The audio track carries the voice. The data channel receives `response.output_audio.done` but no audio deltas.
The first `session.update` frame passes dynamic variables.	The `session` field of the call request passes them. Do not send `session.update`.
`session.created` reports the negotiated PCM formats.	`session.created` arrives when the conversation starts; audio formats are negotiated in SDP.

Every other event works as on the WebSocket, including speech detection and transcripts, text turns, interruption and barge-in, response.cancel, tool calls and vision frames. Keep each serialized message below 1 MiB and within the connection’s negotiated pc.sctp.maxMessageSize. The SCTP limit can be smaller than the Telnyx event limit. Measure the UTF-8 bytes of the entire JSON string, including any base64 image and metadata, before calling send(). The assistant’s voice must stream PCM at 8, 12, 16, 24 or 48 kHz. Otherwise the call starts with an error event on the data channel:

Code	Meaning
`unsupported_voice_output_format`	The assistant’s voice streams a compressed format, for example mp3 or opus. Configure a PCM voice.
`unsupported_voice_output_rate`	The assistant’s voice streams PCM at a rate WebRTC cannot play, for example 22.05 or 44.1 kHz. Configure a voice at 8, 12, 16, 24 or 48 kHz.

Show an avatar

Beta: AI Assistant avatars are in beta. See the Avatars guide for configuration, voice compatibility and pricing. When the assistant has saved avatar_settings and the offer includes a video m-line that can receive VP8 (recvonly or sendrecv), the avatar’s video arrives on that video track and its voice on the audio track, already synchronized. Play both. An offer without a receivable video m-line starts a voice-only call, even when the assistant has an avatar. To keep a receivable video m-line, for example to send the camera, without the avatar, create the call with ?avatar=false; a video m-line added during the call then does not start the avatar either. The data channel reports the avatar’s state with avatar.updated:

{ "type": "avatar.updated", "state": "active" }

`state`	Meaning
`starting`	The avatar is being prepared.
`active`	The avatar’s video and voice are playing.
`unavailable`	The avatar could not start or stopped. The call continues with the assistant’s voice on the same audio track; no action is needed besides hiding the video.
`off`	The call has no avatar.

A runtime avatar failure also sends an avatar.error event with error.fatal: false. The conversation continues; the voice switches back on the same audio track automatically.

Avatar errors

Code	Meaning
`avatar_unavailable`	The avatar could not start or stopped. `error.reason` names the cause, `error.retryable` says whether a new call may get it, and `error.fatal: false` means the conversation continues with the assistant’s voice.

Send the camera

To let the assistant see the caller, send the camera on the video m-line instead of recvonly. Add the camera track before creating the offer, in place of the recvonly transceiver:

const camera = await navigator.mediaDevices.getUserMedia({ video: true });
pc.addTrack(camera.getVideoTracks()[0], camera);

The m-line is then sendrecv: the camera goes up and the avatar, if any, comes down on the same m-line. Telnyx samples the camera every 2 seconds into a JPEG of at most 1280 pixels wide, tagged camera, and buffers it for the vision tool. The assistant needs the vision tool and a vision-capable model; buffered frames do not trigger responses on their own. Image frames sent on the data channel are buffered the same way. For camera or screen snapshots sent as image events, reduce JPEG quality or image dimensions until the complete message fits both data-channel limits. Reducing quality or dimensions can make small text and fine detail harder to interpret. These message-size limits apply to image events, not to the native camera RTP track described above. Send only while the data channel is open, monitor bufferedAmount to avoid accumulating stale snapshots, and handle send() exceptions. If a frame cannot fit or be sent, surface the failure in the application’s camera-sharing UI; reducing the image and retrying are client responsibilities. A successful send() queues the message locally and does not confirm that the assistant has used the image.

Add video during a call

A call that started without a video m-line can add one. Add the transceiver, set the new offer as the local description, wait for ICE gathering, and send it on the data channel as a webrtc.offer event. Telnyx replies with a webrtc.answer event; apply it with setRemoteDescription:

pc.addTransceiver("video", { direction: "recvonly" });
await pc.setLocalDescription(await pc.createOffer());
await iceGatheringComplete(pc);
events.send(JSON.stringify({ type: "webrtc.offer", sdp: pc.localDescription.sdp }));

// In the data channel's message handler:
if (event.type === "webrtc.answer") {
  await pc.setRemoteDescription({ type: "answer", sdp: event.sdp });
}

If the assistant has an avatar, it starts once the new video m-line can receive it. Send one offer at a time and keep a single video m-line. A refused offer returns an error event instead:

Code	Meaning
`invalid_offer`	The offer was not applied. Roll back the local description; the call continues unchanged.
`renegotiation_in_progress`	A previous offer is still being answered. Wait for its answer.
`renegotiation_failed`	The offer was applied but could not be answered. The two sides no longer agree; end the call.

End the call

Close the peer connection to end the call. Telnyx ends the conversation as soon as the oai-events data channel closes. When a client disappears without closing, for example after a network loss, Telnyx ends the conversation within seconds once media stops arriving.

// Stop every local track (microphone and, if sent, camera), then close.
pc.getSenders().forEach((sender) => sender.track?.stop());
pc.close();

The data channel must open within 20 seconds of the call request; otherwise Telnyx ends the call.

Guides

Missions

AI Insights

Assistants API

Missions API

AI Tests API

Conversations API

For AI Agents

Realtime voice and avatar conversations over WebRTC

How a call is set up

Migrate an existing avatar client

Configure ICE servers

Create the offer

Create the call

Errors

Set up the call over a WebSocket

Connect

Receive the ICE servers

Send the offer

Signaling errors

Events, renegotiation and ending the call

Exchange events on the data channel

Show an avatar

Avatar errors

Send the camera

Add video during a call

End the call

​How a call is set up

​Migrate an existing avatar client

​Configure ICE servers

​Create the offer

​Create the call

​Errors

​Set up the call over a WebSocket

​Connect

​Receive the ICE servers

​Send the offer

​Signaling errors

​Events, renegotiation and ending the call

​Exchange events on the data channel

​Show an avatar

​Avatar errors

​Send the camera

​Add video during a call

​End the call

How a call is set up

Migrate an existing avatar client

Configure ICE servers

Create the offer

Create the call

Errors

Set up the call over a WebSocket

Connect

Receive the ICE servers

Send the offer

Signaling errors

Events, renegotiation and ending the call

Exchange events on the data channel

Show an avatar

Avatar errors

Send the camera

Add video during a call

End the call