> ## Documentation Index
> Fetch the complete documentation index at: https://developers.telnyx.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Realtime voice conversations over WebSocket
> Talk to a Telnyx AI Assistant over a single WebSocket: stream microphone audio, receive transcripts and synthesized speech in real time, handle barge-in, and respond to client-side tool calls.
Open a real-time voice conversation with a [Telnyx AI Assistant](https://portal.telnyx.com/#/ai/assistants) over a single WebSocket. Your application streams microphone audio to Telnyx as base64-encoded PCM16 frames, and Telnyx streams back lifecycle events, speech-detection events, user and assistant transcripts, and the assistant's synthesized speech.
Everything about the assistant's behavior — instructions, model, voice, language, tools, transcription and interruption settings — is configured on the assistant itself, in the [Portal](https://portal.telnyx.com/#/ai/assistants) or through the Assistants API. The WebSocket carries only the conversation: audio in, events and audio out. The one thing you can pass on the socket is [dynamic variables](#pass-dynamic-variables) for the conversation, and there is no frame to request a response — the assistant owns turn-taking and answers automatically when the user finishes speaking.
Coming from the OpenAI Realtime API? The wire format is intentionally similar. See [Migrate from the OpenAI Realtime API](/docs/inference/ai-assistants/migrate-from-openai-realtime) for an event-by-event mapping and the behavioral differences.
## How a conversation flows
```
Client → Telnyx (connect wss://.../v2/ai/assistants/{assistant_id}/conversation)
Client → Telnyx {"type":"session.update","session":{...}} (optional)
Client ← Telnyx {"type":"session.created", ...}
Client → Telnyx {"type":"input_audio_buffer.append","audio":""}
Client → Telnyx {"type":"input_audio_buffer.append","audio":""}
Client ← Telnyx {"type":"input_audio_buffer.speech_started"}
Client ← Telnyx {"type":"input_audio_buffer.speech_stopped"}
Client ← Telnyx {"type":"conversation.item.input_audio_transcription.completed","transcript":"..."}
Client ← Telnyx {"type":"response.created","response":{"id":"resp_9c1f4a2b"}}
Client ← Telnyx {"type":"response.output_audio.delta","delta":"", ...}
Client ← Telnyx {"type":"response.output_audio_transcript.delta","delta":"We are open ", ...}
Client ← Telnyx {"type":"response.output_audio.done","response_id":"resp_9c1f4a2b"}
Client ← Telnyx {"type":"response.done","response":{"id":"resp_9c1f4a2b","status":"completed"}}
```
1. Open a WebSocket connection, optionally setting the audio format through query parameters.
2. Optionally send a `session.update` frame as your first message to pass [dynamic variables](#pass-dynamic-variables) for the conversation.
3. Telnyx authenticates the connection, starts a conversation with the requested assistant, and sends a `session.created` frame confirming the negotiated input and output audio formats.
4. Stream microphone audio as `input_audio_buffer.append` frames.
5. Telnyx runs server-side voice-activity detection (VAD) and sends `input_audio_buffer.speech_started` and `input_audio_buffer.speech_stopped` on speech edges, followed by the user's transcript.
6. The assistant responds with a `response.created` frame, a stream of audio and transcript deltas, then `response.output_audio.done` and `response.done`.
7. Speaking while the assistant is talking triggers barge-in: Telnyx stops the response and starts a new turn.
## Connect and authenticate
```
wss://api.telnyx.com/v2/ai/assistants/{assistant_id}/conversation
```
Authenticate the WebSocket handshake with a Telnyx API v2 key in the `Authorization` header:
```
Authorization: Bearer YOUR_TELNYX_API_KEY
```
Browsers cannot set headers on WebSocket connections, so connect from your backend. For browser-based voice agents, use the WebRTC-based [`@telnyx/ai-agent-lib`](https://www.npmjs.com/package/@telnyx/ai-agent-lib) library instead — it handles media capture, playback, and [client-side tools](/docs/inference/ai-assistants/client-side-tools) for you.
Audio negotiation happens at connect time through query parameters:
| Parameter | Description |
| -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `input_sample_rate` | Sample rate, in Hz, of the PCM16 audio you stream to Telnyx. One of `8000`, `16000`, `24000`, `44100`, `48000`. Default: `16000`. |
| `input_format` | Encoding of the audio you stream to Telnyx. Only `pcm16` is supported (the default). |
| `output_format` | Encoding of the assistant audio Telnyx streams back. Only `pcm16` is supported (the default). |
| `output_sample_rate` | Preferred sample rate, in Hz, for the assistant audio. Advisory only — the effective output rate is determined by the assistant's voice and reported in `session.created`. |
```javascript Node.js theme={null}
import WebSocket from "ws";
const ASSISTANT_ID = "assistant-0f4e8b2a";
const url = `wss://api.telnyx.com/v2/ai/assistants/${ASSISTANT_ID}/conversation?input_sample_rate=16000`;
const ws = new WebSocket(url, {
headers: { Authorization: `Bearer ${process.env.TELNYX_API_KEY}` },
});
ws.on("open", () => console.log("Connected"));
```
```python Python theme={null}
import asyncio
import os
import websockets
ASSISTANT_ID = "assistant-0f4e8b2a"
URL = f"wss://api.telnyx.com/v2/ai/assistants/{ASSISTANT_ID}/conversation?input_sample_rate=16000"
async def main():
headers = {"Authorization": f"Bearer {os.environ['TELNYX_API_KEY']}"}
async with websockets.connect(URL, additional_headers=headers) as ws:
print("Connected")
asyncio.run(main())
```
All frames in both directions are JSON text messages with a `type` field. Audio travels inside JSON frames as base64 — never as binary WebSocket frames.
## Session lifecycle
The first frame Telnyx sends is `session.created`. It confirms the conversation has started and reports the negotiated audio formats — read the output rate from it before playing any assistant audio:
```json theme={null}
{
"type": "session.created",
"session": {
"conversation_id": "3E6F995F-85F7-4705-9741-53B116D28237",
"assistant_id": "assistant-0f4e8b2a",
"audio": {
"input": {
"format": { "type": "audio/pcm", "rate": 16000 },
"turn_detection": { "type": "server_vad" }
},
"output": {
"format": { "type": "audio/pcm", "rate": 24000 }
}
}
}
}
```
Each WebSocket connection starts a new conversation with the assistant. Turn detection is always `server_vad` — there is no manual turn mode to configure.
Only assistants whose voice streams raw PCM are supported over this WebSocket. Assistants whose voice outputs a compressed format (for example mp3 or opus) are rejected with an `unsupported_voice_output_format` error.
## Pass dynamic variables
If the assistant's instructions, greeting, or tools use [dynamic variables](/docs/inference/ai-assistants/dynamic-variables), supply the values for this conversation with a `session.update` frame. Send it as your **first** frame, immediately on `open` — before any audio, and without waiting for `session.created`:
```json theme={null}
{
"type": "session.update",
"session": {
"assistant": {
"dynamic_variables": {
"customer_name": "Ada",
"account_tier": "pro"
}
}
}
}
```
Telnyx starts the conversation as soon as the frame arrives, so the variables are resolved before the assistant's first word. If you never send it, Telnyx starts the conversation with no variables after a short timeout.
Only `type` is required. A bare `{ "type": "session.update" }` — or one whose `session` object carries no `dynamic_variables` — is valid: it starts the conversation immediately with no client-supplied variables, skipping that timeout.
```javascript Node.js theme={null}
ws.on("open", () => {
ws.send(
JSON.stringify({
type: "session.update",
session: {
assistant: {
dynamic_variables: { customer_name: "Ada", account_tier: "pro" },
},
},
})
);
});
```
```python Python theme={null}
async with websockets.connect(URL, additional_headers=headers) as ws:
await ws.send(json.dumps({
"type": "session.update",
"session": {
"assistant": {
"dynamic_variables": {"customer_name": "Ada", "account_tier": "pro"},
},
},
}))
```
Values are string key/value pairs, and they reach the assistant exactly like dynamic variables passed on the REST conversation API.
`session.update` configures nothing but dynamic variables, and only before the conversation starts. Sending it after `session.created` is rejected with a `session_update_after_start` error, and a `dynamic_variables` value that is not an object is rejected with `invalid_session_update`. The `telnyx_` namespace is reserved for [system variables](/docs/inference/ai-assistants/dynamic-variables#telnyx-system-variables), so `telnyx_conversation_channel` is set by Telnyx and cannot be overridden.
## Handling audio
### Stream microphone audio
Send audio as `input_audio_buffer.append` frames. The `audio` field is base64-encoded, raw little-endian PCM16 (16-bit signed, mono) at the sample rate you chose with `input_sample_rate`:
```json theme={null}
{ "type": "input_audio_buffer.append", "audio": "" }
```
There is no commit step. Keep appending audio continuously — server-side VAD detects when the user starts and stops speaking and turns the buffered audio into conversation turns automatically.
A few practical rules:
* **Stream at a real-time pace**, in small chunks (for example 20–100 ms of audio per frame). The server enforces an ingress budget; sending far faster than real time fails with an `ingress_budget_exceeded` error.
* **Keep frames under 1 MiB.** Larger frames are rejected with a `frame_too_large` error.
* **Wait for `session.created`** before sending audio. `session.update` is the only frame that goes before it.
```javascript Node.js theme={null}
import fs from "node:fs";
const SAMPLE_RATE = 16000;
const CHUNK_MS = 100;
const CHUNK_BYTES = (SAMPLE_RATE * 2 * CHUNK_MS) / 1000; // 16-bit mono
// speech.raw is raw little-endian PCM16 at 16 kHz
const pcm = fs.readFileSync("speech.raw");
let offset = 0;
const timer = setInterval(() => {
if (offset >= pcm.length) {
clearInterval(timer);
return;
}
const chunk = pcm.subarray(offset, offset + CHUNK_BYTES);
offset += CHUNK_BYTES;
ws.send(
JSON.stringify({
type: "input_audio_buffer.append",
audio: chunk.toString("base64"),
})
);
}, CHUNK_MS);
```
```python Python theme={null}
import base64
import json
SAMPLE_RATE = 16000
CHUNK_MS = 100
CHUNK_BYTES = SAMPLE_RATE * 2 * CHUNK_MS // 1000 # 16-bit mono
async def stream_audio(ws):
# speech.raw is raw little-endian PCM16 at 16 kHz
with open("speech.raw", "rb") as f:
while chunk := f.read(CHUNK_BYTES):
await ws.send(json.dumps({
"type": "input_audio_buffer.append",
"audio": base64.b64encode(chunk).decode(),
}))
await asyncio.sleep(CHUNK_MS / 1000) # real-time pacing
```
If your capture pipeline produces float samples (for example the Web Audio API), convert them to PCM16 before encoding:
```javascript theme={null}
function floatTo16BitPCM(float32Array) {
const buffer = new ArrayBuffer(float32Array.length * 2);
const view = new DataView(buffer);
for (let i = 0; i < float32Array.length; i++) {
const s = Math.max(-1, Math.min(1, float32Array[i]));
view.setInt16(i * 2, s < 0 ? s * 0x8000 : s * 0x7fff, true);
}
return Buffer.from(buffer);
}
```
### Speech detection and user transcripts
Telnyx runs voice-activity detection server-side and reports speech edges. Both events are edge-triggered — each is sent only when the speaking state actually changes, and `speech_stopped` is never sent without a preceding `speech_started`:
```json theme={null}
{ "type": "input_audio_buffer.speech_started" }
{ "type": "input_audio_buffer.speech_stopped" }
```
After the user's turn ends, Telnyx sends the transcript of what they said:
```json theme={null}
{
"type": "conversation.item.input_audio_transcription.completed",
"transcript": "What are your business hours?"
}
```
### Receive assistant audio
Each assistant turn arrives as an ordered sequence of frames, correlated by `response_id`:
| Frame | Meaning |
| ---------------------------------------- | --------------------------------------------------------------------------------------- |
| `response.created` | The assistant turn started. `response.id` correlates everything that follows. |
| `response.output_audio.delta` | A chunk of synthesized speech — base64 PCM16 at the output rate from `session.created`. |
| `response.output_audio_transcript.delta` | A chunk of the text transcript of the spoken response, streamed alongside the audio. |
| `response.output_audio.done` | The assistant finished streaming audio for the turn. |
| `response.done` | The turn is over, with a final `status` of `completed` or `cancelled`. |
```javascript Node.js theme={null}
let outputRate = 24000;
const playback = [];
ws.on("message", async (raw) => {
const event = JSON.parse(raw.toString());
switch (event.type) {
case "session.created":
outputRate = event.session.audio.output.format.rate;
break;
case "conversation.item.input_audio_transcription.completed":
console.log(`You said: ${event.transcript}`);
break;
case "response.output_audio.delta":
// Feed to your audio output as it arrives for lowest latency
playback.push(Buffer.from(event.delta, "base64"));
break;
case "response.output_audio_transcript.delta":
process.stdout.write(event.delta);
break;
case "response.done":
console.log(`\n[turn ${event.response.id}: ${event.response.status}]`);
break;
case "error":
console.error(`${event.error.code}: ${event.error.message}`);
break;
}
});
```
```python Python theme={null}
import base64
import json
async def handle_events(ws):
output_rate = 24000
playback = bytearray()
async for raw in ws:
event = json.loads(raw)
match event["type"]:
case "session.created":
output_rate = event["session"]["audio"]["output"]["format"]["rate"]
case "conversation.item.input_audio_transcription.completed":
print(f"You said: {event['transcript']}")
case "response.output_audio.delta":
# Feed to your audio output as it arrives for lowest latency
playback.extend(base64.b64decode(event["delta"]))
case "response.output_audio_transcript.delta":
print(event["delta"], end="", flush=True)
case "response.done":
print(f"\n[turn {event['response']['id']}: {event['response']['status']}]")
case "error":
print(f"{event['error']['code']}: {event['error']['message']}")
```
For a natural conversation, play deltas as they arrive rather than waiting for `response.done`. Buffer the decoded PCM16 in a queue that your audio output drains at the output sample rate.
## Interruption and barge-in
Barge-in is built in. If the user speaks while the assistant is talking, Telnyx detects it, stops generating the response, and starts a new turn — you don't send anything to make that happen. The interrupted turn finishes with `response.done` and `status: "cancelled"`.
One thing remains your responsibility: audio you have already received and queued locally. When you see `input_audio_buffer.speech_started` during playback, flush your local playback queue so the assistant doesn't keep talking out of your speakers over the user:
```javascript theme={null}
case "input_audio_buffer.speech_started":
playback.length = 0; // drop queued assistant audio immediately
break;
```
You can also interrupt programmatically — for example when the user taps a stop button — with `response.cancel`:
```json theme={null}
{ "type": "response.cancel" }
```
Include `response_id` to target a specific response, or omit it to cancel the current one. Cancelling flushes playback on the Telnyx side and is followed by a `response.done` frame with status `cancelled`. Cancelling a stale or already-finished response is a no-op.
Interruption sensitivity is tuned on the assistant, not the socket — see [Interruption Settings](/docs/inference/ai-assistants/interruption-settings).
## Send text instead of audio
Inject a completed user turn as text with `conversation.item.create`. The assistant answers it exactly as it would a spoken turn — including responding with audio:
```json theme={null}
{
"type": "conversation.item.create",
"item": {
"type": "message",
"role": "user",
"content": [{ "type": "input_text", "text": "What are your business hours?" }]
}
}
```
There is no `response.create` frame — the assistant owns turn-taking and answers automatically. Items of any other shape are rejected with an `invalid_item` error.
## Tool calls
Assistants can use two kinds of tools during a realtime conversation. Both are configured on the assistant — see the [Tools Library](/docs/inference/ai-assistants/tools-library).
### Server-side tools: observe
Webhook and MCP tools execute on Telnyx. The socket surfaces them as informational frames so you can show tool activity in your UI, but you don't execute or respond to them:
```json theme={null}
{
"type": "response.tool_call.started",
"tool_call": { "id": "call_7a3f21b8", "name": "get_business_hours", "arguments": "{\"location\":\"downtown\"}" }
}
```
```json theme={null}
{
"type": "response.tool_call.completed",
"tool_call": { "id": "call_7a3f21b8", "name": "get_business_hours" },
"status": "success"
}
```
### Client-side tools: execute and respond
When the assistant invokes a [client-side tool](/docs/inference/ai-assistants/client-side-tools), Telnyx sends a `conversation.item.created` frame containing a `function_call` item. Run the tool locally, then return the result with a `conversation.item.create` frame carrying a `function_call_output` item that references the same `call_id`. The assistant continues once the result arrives, or after the tool times out.
```javascript Node.js theme={null}
case "conversation.item.created": {
const item = event.item;
if (item.type === "function_call") {
const args = JSON.parse(item.arguments);
const result = await runTool(item.name, args); // your implementation
ws.send(
JSON.stringify({
type: "conversation.item.create",
item: {
type: "function_call_output",
call_id: item.call_id,
output: JSON.stringify(result),
},
})
);
}
break;
}
```
```python Python theme={null}
case "conversation.item.created":
item = event["item"]
if item["type"] == "function_call":
args = json.loads(item["arguments"])
result = await run_tool(item["name"], args) # your implementation
await ws.send(json.dumps({
"type": "conversation.item.create",
"item": {
"type": "function_call_output",
"call_id": item["call_id"],
"output": json.dumps(result),
},
}))
```
## Error handling
Errors arrive as `error` frames. Some errors are non-fatal and leave the session open; others close the connection after the frame is sent.
```json theme={null}
{
"type": "error",
"error": { "code": "invalid_audio", "message": "Invalid base64-encoded audio payload" }
}
```
| Code | Meaning |
| --------------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| `session_not_ready` | A frame arrived before the session was ready. Wait for `session.created`. |
| `frame_too_large` | A frame exceeded the 1 MiB limit. Send smaller audio chunks. |
| `unsupported_event` | The `type` field is not a supported client frame. |
| `invalid_json` | The frame was not valid JSON. |
| `invalid_audio` | The `audio` field was not valid base64. |
| `invalid_item` | A `conversation.item.create` item had an unsupported shape. |
| `invalid_session_update` | A `session.update` frame carried a `dynamic_variables` value that was not an object. |
| `session_update_after_start` | A `session.update` frame arrived after the conversation had already started. Send it as your first frame. |
| `ingress_budget_exceeded` | Audio arrived faster than the session's ingress budget allows. Stream at a real-time pace. |
| `unsupported_voice_output_format` | The assistant's voice streams a compressed format (for example mp3 or opus), which this WebSocket does not support. |
| `conversation_start_failed` | Telnyx could not start a conversation with the requested assistant. |
| `conversation_ended` | The conversation has ended. |
| `session_idle_timeout` | The session was closed after a period of inactivity. |
| `session_max_duration_exceeded` | The session reached its maximum duration. |
If the connection closes, reconnecting starts a new conversation — a fresh `conversation_id` is issued in `session.created`.
## Learn more
* **[Migrate from the OpenAI Realtime API](/docs/inference/ai-assistants/migrate-from-openai-realtime)** — Event-by-event mapping and behavioral differences
* **Conversation WebSocket reference** — The full frame-by-frame reference, under **Assistants API → Conversation WebSocket** in the sidebar
* **[Client-Side Tools](/docs/inference/ai-assistants/client-side-tools)** — Configure tools that execute in your application
* **[Transcription Settings](/docs/inference/ai-assistants/transcription-settings)** and **[Interruption Settings](/docs/inference/ai-assistants/interruption-settings)** — Tune how the assistant hears and yields
* **[Voice Assistant Quickstart](/docs/inference/ai-assistants/no-code-voice-assistant)** — Create and configure an assistant in the Portal