Configure custom LLM providers for AI assistants

Configure Azure OpenAI, AWS Bedrock, Baseten, or any OpenAI-compatible endpoint as a custom LLM provider for your Telnyx AI assistants.

In addition to standard third-party LLM providers like OpenAI, Gemini, and Groq, you can also power your AI Assistant with any public OpenAI-compatible chat completions endpoint. This includes models hosted using AWS Bedrock, Azure OpenAI, and Baseten or open source inference engines like vLLM and SGLang.

You must have a publicly accessible OpenAI-compatible chat completions endpoint before proceeding with deployment.

---

When to use custom LLM providers

Custom LLM providers are ideal for scenarios where you need:

• Specific model requirements - Access to proprietary models, fine-tuned models, or the latest releases not yet available through standard providers.
• Data residency and compliance - Ensure your data stays within specific geographic regions or private cloud environments.
• Cost optimization - Leverage enterprise agreements, reserved capacity, or self-hosted infrastructure for better economics at scale.
• Advanced model control - Fine-tune parameters, adjust inference settings, or use specialized configurations for your use case.

---

Azure

For this guide, we will deploy gpt-4o on Azure AI Foundry.

First, create or select a resource.

Then, drill into the resource to find the API key and Azure OpenAI endpoint.

Create or edit a Telnyx AI Assistant and in the Agent tab, check Use Custom LLM.

Input the endpoint URL as the Base URL and append /openai/v1 and create a new Integration Secret with your API Key.

You will see a dropdown of all possible Azure models but only ones that you have deployed will validate an LLM connection.

Once you save your assistant you will be able to immediately use your assistant in the Telnyx portal with the Test Assistant dropdown.

Forward metadata to your custom LLM

By default, Telnyx does not include your assistant’s dynamic variables in requests to a custom LLM endpoint. If your model gateway or application needs those values, enable forward_metadata on the assistant’s external_llm configuration.

{
  "external_llm": {
    "llm_api_key_ref": "integration_secret_id",
    "base_url": "https://your-llm-gateway.example.com/openai/v1",
    "model": "your-model-name",
    "forward_metadata": true
  }
}

When forward_metadata is true, Telnyx adds a top-level extra_metadata object to the OpenAI-compatible chat completions request body sent to your custom LLM endpoint when dynamic variables are available. The field defaults to false when omitted.

{
  "model": "your-model-name",
  "messages": [
    {
      "role": "system",
      "content": "..."
    },
    {
      "role": "user",
      "content": "..."
    }
  ],
  "extra_metadata": {
    "customer_name": "Jane",
    "account_id": "acct_789",
    "telnyx_agent_target": "[phone]",
    "telnyx_end_user_target": "[phone]"
  }
}

Use this when your external LLM service needs request context for routing, logging, personalization, or retrieval. extra_metadata is separate from OpenAI’s native metadata field, so your endpoint must explicitly read extra_metadata from the request body.

Baseten

For this guide, we will deploy Llama 3.3 70B on Baseten.

First, click Deploy Now.

Navigate to the deployment.

Click the API Endpoint button to see the endpoint and generate an API Key. Save these details.

After about 15 minutes, the deployment should be live.

When it is complete, create or edit a Telnyx AI Assistant and in the Agent tab, check Use Custom LLM.

Input the Baseten Endpoint URL for your deployment as the Base URL and create a new Integration Secret with your Baseten API Key.

If your base URL supports an OpenAI-compatible /models endpoint the Model Name dropdown will populate automatically.

Baseten deployments do not support this endpoint, so you can enter any name for your model here. You can also validate the connection is live before saving your assistant.

Once you save your assistant you will be able to immediately use your assistant in the Telnyx portal with the Test Assistant dropdown as well as review metrics in your Baseten deployment.