> ## Documentation Index
> Fetch the complete documentation index at: https://developers.telnyx.com/llms.txt
> Use this file to discover all available pages before exploring further.

# AI Gateway Inference API

> Call models through AI Gateway with the OpenAI and Anthropic SDKs: base URL, authentication, streaming, request limits and supported request fields.

The inference plane serves three endpoints. All of them authenticate with an AI Gateway token key (`ltg_sk_...`) issued through the [management API](/docs/inference/ai-gateway/management-api). Telnyx account API keys, provider secrets and any other credential are rejected on this plane.

| Endpoint | Purpose |
| - | - |
| `GET https://llm.telnyx.com/v1/models` | Models available to the supplied token key. |
| `POST https://llm.telnyx.com/v1/chat/completions` | OpenAI-compatible Chat Completions, including streaming. Works with every model. |
| `POST https://llm.telnyx.com/v1/messages` | Anthropic-compatible Messages, including streaming. Only for [Anthropic BYOK models](#anthropic-sdk). |

Compatibility is bounded by the AI Gateway contract, not by every option the OpenAI and Anthropic SDKs expose; see [Supported request fields](#supported-request-fields) and [Supported Messages fields](#supported-messages-fields).

## Models

These Telnyx-hosted models are billed to your Telnyx account. Use these model names in the `model` field:

* `Kimi-K3`
* `Kimi-K2.6`
* `Kimi-K2.5`
* `GLM-5.3`
* `GLM-5.3-Flash`
* `GLM-5.2`
* `GLM-5.1-FP8`
* `MiniMax-M3-MXFP8`
* `MiniMax-M2.7`
* `Qwen3.8-27B`
* `Qwen3-235B-A22B`
* `DeepSeek-V4.1-Flash`
* `DeepSeek-V4-Flash-0731`
* `Llama-3.3-70B-Instruct`
* `Meta-Llama-3.1-70B-Instruct`
* `Meta-Llama-3.1-8B-Instruct`
* `gemma-2b-it`

A group lists the model names its keys may call in `allowed_models`; a key can narrow that list further. Per-request output and prompt limits are listed under [Request size limits](#request-size-limits).

### Bring-your-own-key models

These models run on your own provider account with a [provider key](/docs/inference/ai-gateway/byok) and are billed by that provider.

| Provider | Models |
| - | - |
| Anthropic | `claude-fable-5-1`, `claude-opus-5-5`, `claude-sonnet-5` |
| OpenAI | `gpt-6-astra`, `gpt-6-sol`, `gpt-6-luna`, `gpt-5.6-sol`, `gpt-5.6-terra`, `gpt-5.6-luna`, `gpt-5.5`, `gpt-5.4`, `gpt-5.4-mini`, `gpt-5.4-nano`, `gpt-4.1`, `gpt-4.1-mini`, `gpt-4o`, `gpt-4o-mini`, `o3` |

Add them to `allowed_models` like any other model. A BYOK model works only when the calling key's group has an attached provider key for that model's provider. Without one, requests fail with `503` and code `enforcement_unavailable`; they never fall back to a Telnyx-hosted model.

Every BYOK model works on `/v1/chat/completions`. Anthropic BYOK models also work on `/v1/messages`.

### Model discovery

`GET /v1/models` returns the models the supplied key may use: the intersection of the key's `allowed_models` and its group's list. It is not a global catalog, and it does not charge against any budget.

```bash theme={null}
curl https://llm.telnyx.com/v1/models \
  -H "Authorization: Bearer $AI_GATEWAY_TOKEN_KEY"
```

```json theme={null}
{
  "object": "list",
  "data": [
    { "id": "Kimi-K3", "object": "model", "created": 1758535200, "owned_by": "telnyx" }
  ]
}
```

## OpenAI SDK

Point the OpenAI SDK at the inference base URL and use the token key as the API key.

<CodeGroup>
  ```python Python theme={null}
  import os
  from openai import OpenAI

  client = OpenAI(
      api_key=os.environ["AI_GATEWAY_TOKEN_KEY"],
      base_url="https://llm.telnyx.com/v1",
      max_retries=0,
  )

  completion = client.chat.completions.create(
      model=os.environ["AI_GATEWAY_MODEL"],
      messages=[{"role": "user", "content": "Tell me about Telnyx"}],
      max_tokens=256,
      user="application-bound-user-id",
  )
  print(completion.choices[0].message.content)
  ```

  ```javascript JavaScript theme={null}
  import OpenAI from "openai";

  const client = new OpenAI({
    apiKey: process.env.AI_GATEWAY_TOKEN_KEY,
    baseURL: "https://llm.telnyx.com/v1",
    maxRetries: 0,
  });

  const completion = await client.chat.completions.create({
    model: process.env.AI_GATEWAY_MODEL,
    messages: [{ role: "user", content: "Tell me about Telnyx" }],
    max_tokens: 256,
    user: "application-bound-user-id",
  });
  console.log(completion.choices[0].message.content);
  ```

  ```bash curl theme={null}
  curl -X POST https://llm.telnyx.com/v1/chat/completions \
    -H "Authorization: Bearer $AI_GATEWAY_TOKEN_KEY" \
    -H "Content-Type: application/json" \
    -d '{
      "model": "Kimi-K3",
      "messages": [{"role": "user", "content": "Tell me about Telnyx"}],
      "max_tokens": 256,
      "user": "application-bound-user-id"
    }'
  ```
</CodeGroup>

The optional `user` field attributes the request to an application end user for budgeting and reporting. See [End-user identity](/docs/inference/ai-gateway/controls#end-user-identity).

### Streaming

Set `stream: true` and consume the stream to completion. A stream that starts with HTTP 200 can still fail later; handle SDK exceptions and close the stream.

<CodeGroup>
  ```python Python theme={null}
  import os
  from openai import OpenAI

  client = OpenAI(
      api_key=os.environ["AI_GATEWAY_TOKEN_KEY"],
      base_url="https://llm.telnyx.com/v1",
      max_retries=0,
  )

  with client.chat.completions.create(
      model=os.environ["AI_GATEWAY_MODEL"],
      messages=[{"role": "user", "content": "Tell me about Telnyx"}],
      max_tokens=256,
      stream=True,
  ) as stream:
      for chunk in stream:
          if chunk.choices:
              print(chunk.choices[0].delta.content or "", end="", flush=True)
  ```

  ```javascript JavaScript theme={null}
  const stream = await client.chat.completions.create({
    model: process.env.AI_GATEWAY_MODEL,
    messages: [{ role: "user", content: "Tell me about Telnyx" }],
    max_tokens: 256,
    stream: true,
  });

  for await (const chunk of stream) {
    process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
  }
  ```
</CodeGroup>

## Anthropic SDK

<Note>
  `POST /v1/messages` is available **only for Anthropic models used with your own Anthropic key** ([Anthropic BYOK models](#bring-your-own-key-models)). A request to `/v1/messages` with a Telnyx-hosted model or an OpenAI BYOK model returns `400` with code `invalid_request`; call those models on `/v1/chat/completions`. Anthropic BYOK models work on both endpoints.
</Note>

The Anthropic SDK appends `/v1/messages` to its base URL, so set the base URL to the gateway root **without** the `/v1` suffix. The SDK sends the token key in `x-api-key`, which the gateway accepts. `Authorization: Bearer` is also accepted; if both headers are present they must carry the same token.

<CodeGroup>
  ```python Python theme={null}
  import os
  from anthropic import Anthropic

  client = Anthropic(
      api_key=os.environ["AI_GATEWAY_TOKEN_KEY"],
      base_url="https://llm.telnyx.com",
      max_retries=0,
  )

  message = client.messages.create(
      model="claude-sonnet-5",
      max_tokens=256,
      messages=[{"role": "user", "content": "Tell me about Telnyx"}],
      metadata={"user_id": "application-bound-user-id"},
  )
  print(message.content[0].text)
  ```

  ```javascript JavaScript theme={null}
  import Anthropic from "@anthropic-ai/sdk";

  const client = new Anthropic({
    apiKey: process.env.AI_GATEWAY_TOKEN_KEY,
    baseURL: "https://llm.telnyx.com",
    maxRetries: 0,
  });

  const message = await client.messages.create({
    model: "claude-sonnet-5",
    max_tokens: 256,
    messages: [{ role: "user", content: "Tell me about Telnyx" }],
    metadata: { user_id: "application-bound-user-id" },
  });
  console.log(message.content[0].text);
  ```

  ```bash curl theme={null}
  curl -X POST https://llm.telnyx.com/v1/messages \
    -H "x-api-key: $AI_GATEWAY_TOKEN_KEY" \
    -H "anthropic-version: 2023-06-01" \
    -H "Content-Type: application/json" \
    -d '{
      "model": "claude-sonnet-5",
      "max_tokens": 256,
      "messages": [{"role": "user", "content": "Tell me about Telnyx"}],
      "metadata": {"user_id": "application-bound-user-id"}
    }'
  ```
</CodeGroup>

Raw HTTP requests must send `anthropic-version: 2023-06-01`; the SDKs add it automatically. `metadata.user_id` and the OpenAI `user` field identify the same end-user namespace.

The endpoint streams standard Anthropic server-sent events (`message_start`, `content_block_start`, `content_block_delta`, `content_block_stop`, `message_delta`, `message_stop`). An error after the stream has started arrives as an `event: error` frame.

<CodeGroup>
  ```python Python theme={null}
  with client.messages.stream(
      model="claude-sonnet-5",
      max_tokens=256,
      messages=[{"role": "user", "content": "Tell me about Telnyx"}],
  ) as stream:
      for text in stream.text_stream:
          print(text, end="", flush=True)
  ```

  ```javascript JavaScript theme={null}
  const stream = client.messages.stream({
    model: "claude-sonnet-5",
    max_tokens: 256,
    messages: [{ role: "user", content: "Tell me about Telnyx" }],
  });

  for await (const event of stream) {
    if (event.type === "content_block_delta" && event.delta.type === "text_delta") {
      process.stdout.write(event.delta.text);
    }
  }
  ```
</CodeGroup>

### Supported Messages fields

| Field | Notes |
| - | - |
| `model` | Required. An Anthropic BYOK model available to this key. |
| `messages` | Required. 1 to 256 messages with roles `user` and `assistant`. |
| `max_tokens` | Required. At most 16,384. |
| `system` | String or array of `text` blocks. |
| `stream` | Anthropic server-sent events. |
| `temperature` | 0 to 1. |
| `top_p` | 0 to 1. |
| `top_k` | 0 to 1000. |
| `stop_sequences` | Up to 4. |
| `metadata` | Only `user_id` is accepted; it maps to the end-user dimension. |
| `tools`, `tool_choice` | Tool definitions with `name`, `description` and `input_schema`. `tool_choice` accepts `auto`, `any`, `tool`, `none` or an object. |

Content blocks of type `text`, `tool_use` and `tool_result` are supported. `image` blocks are rejected.

OpenAI-specific fields (`frequency_penalty`, `presence_penalty`, `n`, `seed`, `response_format`, `user`, `stream_options`, `parallel_tool_calls`) are rejected on the Messages endpoint.

## Retries and timeouts

Inference is **not idempotent**. An SDK retry after a timeout or disconnect can dispatch a second billed request, and a timeout is not evidence that the first request did no provider work. The examples on this page set `max_retries=0`; if your application retries, do so deliberately and only for errors that occurred before dispatch (for example `401`, `403` or `429` with `Retry-After`). See [Errors](/docs/inference/ai-gateway/errors).

Client-side timeouts are a client setting and do not extend the gateway's server-side request lifetime.

## Request size limits

| Limit | Telnyx-hosted models | `gemma-2b-it` | BYOK models |
| - | - | - | - |
| Output tokens per request | 8,192 | 2,048 | 16,384 |
| Prompt size | About 32k tokens | About 8k tokens | About 128k tokens (about 96k for `gpt-4o` and `gpt-4o-mini`) |

Prompt size is measured conservatively from the request size, so the usable limit can be somewhat lower than the model's tokenizer would count. A request over either limit returns `400`.

Set `max_tokens` (or `max_completion_tokens`) on every request, especially when the key, user or group has a low `tpm_limit`. A request without it reserves the model's full output allowance against budgets and `tpm_limit`, and can be rate-limited even when the actual response would be short. See [Reservations](/docs/inference/ai-gateway/controls#reservations).

## Supported request fields

Unknown top-level fields are rejected with `400`. Options that a specific model does not support are also rejected with `400` rather than silently ignored.

### Chat Completions

| Field | Notes |
| - | - |
| `model` | Required. Must be a model returned by `GET /v1/models` for this key. |
| `messages` | Required. Roles: `system`, `developer`, `user`, `assistant`, `tool`. Content is a string or an array of `text` and `image_url` parts. |
| `max_tokens` / `max_completion_tokens` | Send one or the other, never both. At most 8,192 for Telnyx-hosted models (2,048 for `gemma-2b-it`) and 16,384 for BYOK models. Strongly recommended; see [Request size limits](#request-size-limits). |
| `stream`, `stream_options` | Server-sent events. |
| `temperature` | 0 to 2. |
| `top_p` | 0 to 1. |
| `stop` | String or up to 4 strings. |
| `seed`, `presence_penalty`, `frequency_penalty`, `n` | Passed through where the model supports them. |
| `user` | End-user identifier for budgeting and reporting. |
| `tools`, `tool_choice`, `parallel_tool_calls` | Function tools. `tool_choice` accepts `none`, `auto`, `required` or `{"type": "function", "function": {"name": ...}}`. |
| `response_format` | Set `type` to `json_object` for a JSON object, or `json_schema` with `json_schema.name`, `json_schema.schema` and optional `json_schema.strict` for a supplied schema. Support depends on the model and `stream`; see below. |

<Note>
  **Structured JSON on Telnyx-hosted models:** All 17 models support `json_schema` without streaming. All except `Meta-Llama-3.1-8B-Instruct` support `json_object`. Both formats support `stream: true` where enabled, except on `Llama-3.3-70B-Instruct`. Unsupported combinations return `400` before inference.

  `json_object` does not enforce requested fields or values. Use `json_schema` when a specific structure is required. Schemas are limited to 16 KiB, 16 levels and 512 schema nodes; remote references are rejected. Handle truncation, refusals and stream errors, and validate the completed output.
</Note>

Image parts must be inline `data:image/...;base64,...` URLs (PNG, JPEG, GIF or WebP). Remote image URLs are rejected.

### Not supported

* Custom provider URLs, provider credentials, routing or fallback controls in the request body.
* Remote image URLs.
* Unlisted beta headers and options.
* The Responses API, embeddings, image and audio generation, batches and realtime APIs. These are not part of the AI Gateway surface; see the [Telnyx Inference API](/docs/inference/getting-started) for those capabilities.