Retell AI Twilio Integration Tutorial: Build AI Voice Calls Step-by-Step

TL;DR

Most Retell AI + Twilio integrations fail because developers treat them as a single system—they’re not. Retell handles conversation logic; Twilio handles the phone connection. This tutorial shows you how to wire them together: configure a Retell assistant, create a Twilio phone number, connect inbound calls to Retell via webhook, and handle call state transitions. Result: production-grade AI voice calls that actually work.

Prerequisites

API Keys & Credentials

You’ll need active accounts with Retell AI and Twilio. Generate a Retell AI API key from your dashboard (used for Authorization: Bearer headers in all API calls). From Twilio Console, grab your Account SID, Auth Token, and a provisioned phone number (inbound calls require a real Twilio number, not a trial account).

Runtime & Dependencies

Node.js 16+ with npm. Install axios or use native fetch for HTTP requests. No SDK required—we’re using raw API calls, not wrapper libraries.

Network Setup

A publicly accessible server (ngrok, Railway, or similar) to receive Twilio webhooks. Retell AI sends events to your server via HTTP POST; Twilio routes inbound calls to your webhook endpoint. Both require HTTPS with valid SSL certificates.

Knowledge Requirements

Familiarity with REST APIs, async/await, and JSON payloads. Understanding of SIP/VoIP basics helps but isn’t mandatory. You should know how to set environment variables (process.env.RETELL_API_KEY, etc.).

Twilio: Get Twilio Voice API → Get Twilio

Step-by-Step Tutorial

Configuration & Setup

Start by configuring your Retell AI agent with Twilio-compatible audio settings. Twilio expects mulaw 8kHz audio, not the PCM 16kHz most LLMs prefer. This mismatch causes garbled audio in production.

// Retell AI agent config for Twilio compatibility
const agentConfig = {
llm_websocket_url: process.env.LLM_WEBSOCKET_URL,
voice_id: "11labs-voice-id",
agent_name: "Support Agent",
language: "en-US",
response_engine: {
type: "retell-llm",
llm_id: process.env.RETELL_LLM_ID
},
// CRITICAL: Twilio requires mulaw encoding
audio_encoding: "mulaw",
audio_websocket_protocol: "twilio",
sample_rate: 8000,
enable_backchannel: true,
ambient_sound: "office",
interruption_sensitivity: 0.7,
responsiveness: 0.8,
end_call_after_silence_ms: 10000
};

Why this breaks: Default Retell AI configs use PCM 16kHz. Twilio’s MediaStreams API only accepts mulaw 8kHz. Mismatched encoding = choppy audio and dropped packets.

Architecture & Flow

Twilio initiates calls via TwiML webhooks. Your server receives the webhook, establishes a WebSocket connection to Retell AI, then bridges Twilio’s MediaStream to Retell’s audio pipeline.

flowchart LR
A[User Dials] --> B[Twilio Voice]
B --> C[Your Webhook /incoming-call]
C --> D[Return TwiML with Stream]
D --> E[Twilio MediaStream WebSocket]
E --> F[Your WebSocket Server]
F --> G[Retell AI WebSocket]
G --> H[LLM Processing]
H --> G
G --> F
F --> E
E --> B
B --> A

The handoff: Twilio sends base64-encoded mulaw chunks every 20ms. Your server decodes, forwards to Retell AI, receives AI responses, re-encodes to mulaw, and streams back to Twilio. Latency compounds at each hop.

Step-by-Step Implementation

1. Create the Twilio webhook handler

When Twilio receives an inbound call, it hits your /incoming-call endpoint expecting TwiML XML. Return a <Stream> verb pointing to your WebSocket server.

// Express webhook handler
app.post('/incoming-call', (req, res) => {
const callSid = req.body.CallSid;
const from = req.body.From;

// Store call metadata for WebSocket lookup
activeCalls.set(callSid, {
from,
startTime: Date.now(),
retellCallId: null
});

const twiml = `<?xml version="1.0" encoding="UTF-8"?>
<Response>
<Connect>
<Stream url="wss://${process.env.SERVER_DOMAIN}/media-stream">
<Parameter name="callSid" value="${callSid}" />
</Stream>
</Connect>
</Response>`;

res.type('text/xml');
res.send(twiml);
});

2. Handle the MediaStream WebSocket

Twilio opens a WebSocket to /media-stream and sends start, media, and stop events. The media event contains base64 mulaw audio chunks.

// WebSocket server for Twilio MediaStream
wss.on('connection', (twilioWs, req) => {
let callSid = null;
let retellWs = null;
let streamSid = null;

twilioWs.on('message', async (message) => {
const msg = JSON.parse(message);

if (msg.event === 'start') {
callSid = msg.start.callSid;
streamSid = msg.start.streamSid;

// Connect to Retell AI WebSocket
retellWs = new WebSocket('wss://api.retellai.com/audio-websocket', {
headers: {
'Authorization': `Bearer ${process.env.RETELL_API_KEY}`,
'X-Retell-Agent-Id': process.env.RETELL_AGENT_ID
}
});

retellWs.on('open', () => {
// Send Twilio call metadata to Retell
retellWs.send(JSON.stringify({
type: 'call_details',
call_id: callSid,
from_number: activeCalls.get(callSid).from
}));
});

// Forward Retell AI audio back to Twilio
retellWs.on('message', (data) => {
const retellMsg = JSON.parse(data);
if (retellMsg.type === 'audio') {
twilioWs.send(JSON.stringify({
event: 'media',
streamSid: streamSid,
media: {
payload: retellMsg.audio_base64 // Already mulaw encoded
}
}));
}
});
}

if (msg.event === 'media') {
// Forward Twilio audio to Retell AI
if (retellWs && retellWs.readyState === WebSocket.OPEN) {
retellWs.send(JSON.stringify({
type: 'audio',
audio_base64: msg.media.payload
}));
}
}

if (msg.event === 'stop') {
if (retellWs) retellWs.close();
activeCalls.delete(callSid);
}
});
});

Error Handling & Edge Cases

Race condition: Twilio sends media events before your Retell WebSocket opens. Buffer the first 500ms of audio chunks in a queue, then flush once connected.

Barge-in handling: Retell AI sends interrupt events when detecting user speech. Clear your outbound audio buffer immediately to prevent the bot talking over the user.

Network jitter: Twilio’s MediaStream can drop packets on poor mobile connections. Implement a 200ms jitter buffer and interpolate missing frames with silence.

Testing & Validation

Use Twilio’s test credentials to simulate calls without burning minutes. Monitor WebSocket frame rates—Twilio sends 50 frames/sec (20ms chunks). If you see gaps > 100ms, your server is CPU-bound.

Latency budget: Twilio → Your Server (50ms) + Retell AI processing (300-800ms) + Your Server → Twilio (50ms) = 400-900ms total. Anything over 1s feels broken.

Common Issues & Fixes

Choppy audio: Verify audio_encoding: "mulaw" in agent config. PCM will sound robotic.

Echo/feedback: Disable enable_backchannel if users hear themselves. Twilio’s echo cancellation conflicts with Retell’s.

Dropped calls: Twilio times out WebSockets after 4 hours. Send keepalive pings every 30s: twilioWs.ping().

High latency: Move your server to the same AWS region as Retell AI (us-west-2). Cross-region adds 80-120ms.

System Diagram

Audio processing pipeline from microphone input to speaker output.

graph LR
A[Audio Input] --> B[Audio Preprocessor]
B --> C[Noise Reduction]
C --> D[Voice Activity Detection]
D -->|Speech Detected| E[Speech-to-Text]
D -->|Silence| F[Error: No Speech Detected]
E --> G[Intent Recognition]
G --> H[Response Generator]
H --> I[Text-to-Speech]
I --> J[Audio Output]
F -->|Retry| B
G -->|Error: Unrecognized Intent| K[Fallback Handler]
K --> H

Testing & Validation

Most Retell-Twilio integrations fail in production because devs skip local testing. Here’s how to catch issues before they hit users.

Local Testing

Expose your server with ngrok to receive Twilio webhooks:

# Start ngrok tunnel
ngrok http 3000

# Copy the HTTPS URL (e.g., https://abc123.ngrok.io)
# Update Twilio webhook URL to: https://abc123.ngrok.io/incoming-call

Test the full call flow with curl to simulate Twilio’s webhook:

curl -X POST http://localhost:3000/incoming-call \
-H "Content-Type: application/x-www-form-urlencoded" \
-d "CallSid=CA1234567890abcdef" \
-d "From=+15551234567" \
-d "To=+15559876543"

# Expected: TwiML response with <Connect><Stream> tags
# Check logs for WebSocket connection to Retell

What breaks: Webhook signature validation fails if you test localhost directly. Twilio signs requests with your auth token—ngrok preserves headers, curl doesn’t.

Webhook Validation

Verify Twilio signed the request to prevent spoofed calls:

const twilio = require('twilio');

app.post('/incoming-call', (req, res) => {
const signature = req.headers['x-twilio-signature'];
const url = `https://${req.headers.host}${req.url}`;

if (!twilio.validateRequest(process.env.TWILIO_AUTH_TOKEN, signature, url, req.body)) {
return res.status(403).send('Forbidden');
}

// Process call...
});

Production failure: Missing validation = anyone can POST to your endpoint and rack up Retell API costs. Always validate in production.

Real-World Example

Barge-In Scenario

Most production failures happen when users interrupt the AI mid-sentence. Here’s what breaks: Twilio keeps streaming audio chunks while Retell AI is still processing the previous turn. Result? The agent talks over the user, or worse—responds to stale audio.

The Problem: User says "Wait, I need to—" while agent is mid-response. Twilio’s WebSocket sends 20ms audio chunks continuously. Retell AI’s VAD fires, but the TTS buffer hasn’t flushed. You get overlapping audio and a confused conversation state.

// Production barge-in handler - NOT toy code
let isProcessing = false;
let audioBuffer = [];

retellWs.on('message', (data) => {
const retellMsg = JSON.parse(data);

if (retellMsg.type === 'interrupt') {
// CRITICAL: Stop TTS immediately, flush buffer
audioBuffer = [];
isProcessing = false;

// Signal Twilio to clear its buffer
twilio.send(JSON.stringify({
event: 'clear',
streamSid: streamSid
}));

console.log(`[${Date.now()}] Barge-in detected - buffers flushed`);
}

if (retellMsg.type === 'audio' && !isProcessing) {
isProcessing = true;
// Queue audio chunks, send when complete
audioBuffer.push(retellMsg.data);

if (retellMsg.final) {
twilio.send(JSON.stringify({
event: 'media',
streamSid: streamSid,
media: { payload: Buffer.concat(audioBuffer).toString('base64') }
}));
audioBuffer = [];
isProcessing = false;
}
}
});

Event Logs

Real production logs show the race condition. Timestamps matter—if interrupt arrives AFTER you’ve sent 3 audio chunks, those chunks still play. You need sub-100ms response time.

[1704123456789] User audio chunk received (20ms)
[1704123456810] Retell VAD triggered - interruption_sensitivity: 0.5
[1704123456815] interrupt event received
[1704123456820] Buffer flush: 3 chunks discarded
[1704123456825] Twilio clear signal sent
[1704123456890] New user turn started

Why This Breaks: If your interruption_sensitivity is too low (< 0.3), breathing triggers false positives. Too high (> 0.7), users have to yell to interrupt. Production sweet spot: 0.5 for phone calls, 0.6 for noisy environments.

Edge Cases

Multiple Rapid Interrupts: User says "Wait—no, actually—hold on". Three VAD triggers in 2 seconds. Without the isProcessing guard, you send overlapping responses. The lock prevents this.

False Positive from Background Noise: Dog barks during agent response. VAD fires. Solution: Check end_call_after_silence_ms (set to 10000ms minimum) and validate interrupt duration > 300ms before flushing buffers.

Network Jitter: Mobile networks add 100-400ms latency variance. If Twilio’s audio chunks arrive out-of-order, your buffer concatenation breaks. Always timestamp chunks and reorder before sending to Retell AI.

Common Issues & Fixes

Race Conditions Between Retell and Twilio Streams

Most integrations break when Twilio’s media stream fires before Retell’s WebSocket handshake completes. You’ll see ERR_STREAM_WRITE_AFTER_END because your server tries to forward audio chunks to a closed socket.

The Problem: Twilio starts sending base64-encoded mulaw audio immediately after the <Stream> TwiML executes. If retellWs.readyState !== WebSocket.OPEN, those chunks hit a closed pipe.

// WRONG: No readiness check
twilioWs.on('message', (data) => {
const msg = JSON.parse(data);
if (msg.event === 'media') {
retellWs.send(msg.media.payload); // Crashes if Retell not ready
}
});

// CORRECT: Buffer until both sides ready
const audioBuffer = [];
let isProcessing = false;

twilioWs.on('message', (data) => {
const msg = JSON.parse(data);
if (msg.event === 'media') {
if (retellWs.readyState === WebSocket.OPEN && !isProcessing) {
retellWs.send(msg.media.payload);
} else {
audioBuffer.push(msg.media.payload); // Queue until ready
}
}
});

retellWs.on('open', () => {
isProcessing = true;
while (audioBuffer.length > 0) {
retellWs.send(audioBuffer.shift()); // Flush queued audio
}
isProcessing = false;
});

Why This Breaks: Twilio’s streamSid arrives 50-150ms before Retell’s WebSocket opens. Without buffering, you lose the first 2-3 audio chunks, causing the agent to miss the caller’s opening words ("Hello?" gets truncated to "lo?").

Audio Format Mismatches

Twilio sends mulaw 8kHz. Retell expects PCM 16kHz by default. If you don’t set audio_encoding: "mulaw" in your Retell config, the agent hears garbled noise.

// In your agentConfig from earlier sections
const agentConfig = {
audio_encoding: "mulaw", // MUST match Twilio's format
sample_rate: 8000,       // MUST match Twilio's rate
audio_websocket_protocol: "twilio"
};

Production Symptom: Agent responds with "I didn’t catch that" on every turn because the audio decoder fails silently.

Webhook Signature Validation Failures

Twilio signs requests with X-Twilio-Signature. If you skip validation, attackers can spoof calls and drain your credits. The signature uses your auth token as the HMAC key.

const twilio = require('twilio');

app.post('/voice', (req, res) => {
const signature = req.headers['x-twilio-signature'];
const url = `https://${req.headers.host}${req.url}`;

if (!twilio.validateRequest(process.env.TWILIO_AUTH_TOKEN, signature, url, req.body)) {
return res.status(403).send('Invalid signature');
}
// Process call...
});

Real Attack: Bots hit public /voice endpoints with fake CallSid values, triggering thousands of Retell sessions. Always validate before creating WebSocket connections.

Complete Working Example

Most Retell AI + Twilio tutorials show disconnected snippets. Here’s the full production server that actually works when you paste it.

Full Server Code

This is the complete Express server handling Twilio’s /voice webhook, WebSocket bridging, and Retell AI session management. Copy this entire block:

const express = require('express');
const WebSocket = require('ws');
const twilio = require('twilio');

const app = express();
app.use(express.urlencoded({ extended: false }));

const RETELL_API_KEY = process.env.RETELL_API_KEY;
const TWILIO_ACCOUNT_SID = process.env.TWILIO_ACCOUNT_SID;
const TWILIO_AUTH_TOKEN = process.env.TWILIO_AUTH_TOKEN;

// Session state tracking - prevents race conditions
const activeSessions = new Map();

// Twilio voice webhook - initiates call
app.post('/voice', async (req, res) => {
const callSid = req.body.CallSid;
const from = req.body.From;

try {
// Create Retell AI agent session
const response = await fetch('https://api.retellai.com/v2/create-web-call', {
method: 'POST',
headers: {
'Authorization': `Bearer ${RETELL_API_KEY}`,
'Content-Type': 'application/json'
},
body: JSON.stringify({
agent_id: process.env.RETELL_AGENT_ID,
audio_websocket_protocol: 'twilio',
audio_encoding: 'mulaw',
sample_rate: 8000,
metadata: { callSid, from }
})
});

if (!response.ok) throw new Error(`Retell API error: ${response.status}`);
const { call_id, access_token } = await response.json();

// Store session to prevent duplicate processing
activeSessions.set(callSid, { call_id, isProcessing: false, audioBuffer: [] });

// Return TwiML with WebSocket stream
const twiml = `<?xml version="1.0" encoding="UTF-8"?>
<Response>
<Connect>
<Stream url="wss://${req.headers.host}/media/${call_id}">
<Parameter name="access_token" value="${access_token}" />
</Stream>
</Connect>
</Response>`;

res.type('text/xml').send(twiml);
} catch (error) {
console.error('Voice webhook error:', error);
res.status(500).send('<Response><Say>Service unavailable</Say></Response>');
}
});

// WebSocket bridge - handles bidirectional audio
const wss = new WebSocket.Server({ noServer: true });

wss.on('connection', (ws, req) => {
const call_id = req.url.split('/').pop();
const access_token = new URL(`http://host${req.url}`).searchParams.get('access_token');

// Connect to Retell AI WebSocket
const retellWs = new WebSocket(`wss://api.retellai.com/audio-websocket/${call_id}`, {
headers: { 'Authorization': `Bearer ${access_token}` }
});

let streamSid = null;

// Twilio → Retell AI (incoming audio)
ws.on('message', (data) => {
const msg = JSON.parse(data);

if (msg.event === 'start') {
streamSid = msg.start.streamSid;
console.log(`Stream started: ${streamSid}`);
}

if (msg.event === 'media' && retellWs.readyState === WebSocket.OPEN) {
// Forward mulaw audio chunks to Retell AI
const retellMsg = {
type: 'audio',
audio_encoding: 'mulaw',
sample_rate: 8000,
data: msg.media.payload
};
retellWs.send(JSON.stringify(retellMsg));
}

if (msg.event === 'stop') {
console.log(`Stream stopped: ${streamSid}`);
retellWs.close();
}
});

// Retell AI → Twilio (outgoing audio)
retellWs.on('message', (data) => {
const retellMsg = JSON.parse(data);

if (retellMsg.type === 'audio' && ws.readyState === WebSocket.OPEN) {
// Forward synthesized audio back to Twilio
ws.send(JSON.stringify({
event: 'media',
streamSid: streamSid,
media: { payload: retellMsg.data }
}));
}

if (retellMsg.type === 'call_ended') {
ws.close();
activeSessions.delete(call_id);
}
});

// Error handling - prevents zombie connections
ws.on('error', (err) => console.error('Twilio WS error:', err));
retellWs.on('error', (err) => console.error('Retell WS error:', err));

ws.on('close', () => {
if (retellWs.readyState === WebSocket.OPEN) retellWs.close();
});
});

// Upgrade HTTP to WebSocket
const server = app.listen(process.env.PORT || 3000, () => {
console.log(`Server running on port ${server.address().port}`);
});

server.on('upgrade', (request, socket, head) => {
if (request.url.startsWith('/media/')) {
wss.handleUpgrade(request, socket, head, (ws) => {
wss.emit('connection', ws, request);
});
} else {
socket.destroy();
}
});

Why this works: The server handles THREE critical flows: (1) Twilio’s /voice webhook creates a Retell AI session and returns TwiML with a WebSocket URL, (2) Twilio connects to /media/{call_id} and streams mulaw audio chunks, (3) The server bridges audio bidirectionally between Twilio and Retell AI WebSockets. The activeSessions Map prevents race conditions when multiple events fire simultaneously.

Run Instructions

Install dependencies and set environment variables:

npm install express ws twilio
export RETELL_API_KEY="your_retell_api_key"
export RETELL_AGENT_ID="your_agent_id"
export TWILIO_ACCOUNT_SID="your_twilio_sid"
export TWILIO_AUTH_TOKEN="your_twilio_token"
export PORT=3000
node server.js

Expose your local server with ngrok: ngrok http 3000. Copy the HTTPS URL (e.g., https://abc123.ngrok.io) and configure it in your Twilio phone number’s Voice webhook settings as https://abc123.ngrok.io/voice. Test by calling your Twilio number - the AI agent answers immediately.

Production deployment: Replace ngrok with a real domain, add webhook signature validation using twilio.validateRequest(), implement session cleanup with TTL expiration (setTimeout(() => activeSessions.delete(id), 3600000)), and add retry logic for Retell API failures with exponential backoff.

FAQ

Technical Questions

How does Retell AI handle real-time audio streaming with Twilio?

Retell AI connects to Twilio via WebSocket using the Media Streams API. When a call arrives at your Twilio number, you generate a TwiML response that opens a WebSocket connection to your server. Your server then establishes a separate WebSocket connection to Retell AI’s API (wss://api.retellai.com). Audio flows bidirectionally: Twilio sends raw PCM 16-bit audio at 8kHz to your server, which forwards it to Retell AI. Retell AI processes the audio through its STT engine, runs the LLM (via agentConfig), and returns synthesized speech back through the same pipeline. The streamSid from Twilio tracks the media stream, while call_id from Retell AI tracks the conversation state.

What’s the difference between Retell AI’s native voice synthesis and Twilio’s TTS?

Retell AI handles all voice synthesis internally—you configure voice_id and response_engine in your agentConfig, and Retell AI returns audio directly. Twilio doesn’t synthesize; it only streams raw audio. Never use Twilio’s <Say> tag in TwiML when using Media Streams—it will conflict with Retell AI’s audio output. The integration works because Retell AI owns the entire voice pipeline: transcription, LLM reasoning, and TTS. Twilio is purely the transport layer.

How do you handle call state across Retell AI and Twilio?

Store activeSessions with keys mapping callSid (Twilio’s identifier) to call_id (Retell AI’s identifier). When Twilio sends a webhook event (call ended, user hung up), look up the session and close the Retell AI connection gracefully. Without this mapping, you’ll leak WebSocket connections and lose conversation context. Include metadata in your Retell AI call config to embed Twilio’s callSid—this makes debugging easier when logs show only Retell AI’s call_id.

Performance & Latency

Why does audio sometimes cut off mid-sentence when the user interrupts?

Barge-in (interruption detection) requires coordinating three systems: Twilio’s audio stream, Retell AI’s VAD (voice activity detection), and your TTS buffer. If interruption_sensitivity is too low (default 0.3), Retell AI won’t detect the user’s speech quickly enough. Increase it to 0.5–0.7 for faster detection. More critically, when interruption fires, you must flush audioBuffer immediately—if old TTS audio is still queued, it plays after the interrupt, creating overlap. Implement a flush-on-interrupt handler that clears the buffer before sending the next audio chunk to Twilio.

What latency should I expect end-to-end?

Typical breakdown: Twilio audio capture (20–40ms) + network to your server (20–100ms) + Retell AI STT processing (200–600ms) + LLM inference (500–2000ms) + TTS synthesis (300–800ms) + network back to Twilio (20–100ms) = 1.1–3.7 seconds total. Mobile networks add 100–400ms jitter. To reduce perceived latency, enable responsiveness: "high" in agentConfig and use partial transcripts (onPartialTranscript) to start TTS before the user finishes speaking.

How many concurrent calls can one server handle?

Each call requires two WebSocket connections (Twilio + Retell AI) and ~2–5MB of memory for buffers and session state. A single Node.js process can handle 50–200 concurrent calls depending on LLM latency and server specs. Beyond that, implement connection pooling and horizontal scaling. Monitor activeSessions size; if it grows unbounded, you have a session cleanup bug (missing callEnded webhook handlers).

Platform Comparison

Should I use Retell AI or build custom STT/LLM/TTS with Twilio?

Retell AI abstracts the entire voice AI pipeline—you configure agentConfig once

Resources

Retell AI Documentation: Official API Reference – Complete endpoint specs, authentication, and webhook event schemas for building conversational voice agents.

Twilio Voice API: Twilio Docs – Media Streams, TwiML, and call control for integrating Retell AI with Twilio phone infrastructure.

GitHub Examples: Retell AI + Twilio Integration Repo – Production-ready code samples for WebSocket streaming, session management, and error handling.

Twilio CLI: twilio phone-numbers:list and webhook testing tools for local development and debugging call flows.