Twilio behind LiveKit, behind Pipecat, or on its own: three places to put the phone line

"Add Twilio" sounds like one integration. It is three, and they disagree about the most basic question in a voice agent: who holds the audio. LiveKit takes the call as a SIP trunk and turns the caller into a participant in a room. Pipecat takes it as a WebSocket carrying raw audio frames. And Twilio's own ConversationRelay keeps the audio entirely, and hands your server text.

The previous post compared LiveKit and Pipecat on the seam where a voice agent breaks: a caller talking over the bot. Put a phone line in front of either one and that seam moves, because now there is a third system buffering audio that neither framework controls.

⚠️ What follows is read, not measured. I have not run these three paths against each other on a real phone network, and a latency comparison between them would mean nothing without the carrier, the region, the codec and the model versions attached. Everything below comes from the documentation and the published source, with links to each claim.

Three contracts, three wire formats

	LiveKit	Pipecat	ConversationRelay
Twilio product	Elastic SIP Trunking	Media Streams	ConversationRelay
What crosses the wire	SIP + RTP media	base64 μ-law audio in JSON over WebSocket	text over WebSocket
Who runs STT and TTS	your agent	your pipeline	Twilio
The call becomes	a room participant	a pipeline transport	a text session

That last row is the whole comparison. Everything else — how you interrupt, how you hang up, how you hand off to a human — follows from which of those three things the call becomes.

LiveKit: the phone call becomes a participant

LiveKit never sees Twilio's WebSocket products at all. It speaks SIP. You create a Twilio Elastic SIP trunk whose domain ends in pstn.twilio.com, point its origination URI at LiveKit's SIP endpoint for inbound calls, and give it a credential list under Termination for outbound ones. Multiple origination URIs with different "priority" values give you failover: Twilio tries the lowest value first.

On the LiveKit side, a call arriving on an inbound trunk becomes a SIP participant that joins a room like any other participant, and a dispatch rule decides which room and which agent:

{
  "dispatch_rule": {
    "rule": { "dispatchRuleIndividual": { "roomPrefix": "call-" } },
    "name": "My dispatch rule",
    "roomConfig": { "agents": [{ "agentName": "inbound-agent" }] }
  }
}

dispatchRuleIndividual gives each caller a fresh room; dispatchRuleDirect puts everyone in one named room; dispatchRuleCallee routes by the number that was dialled. Outbound calls go the other way through the CreateSIPParticipant API.

What you get for adopting SIP is that the telephony features are LiveKit's problem, listed on the same page: DTMF over RFC 4733, cold transfer by SIP REFER and warm transfer through an agent, and Krisp noise cancellation switched on per trunk with krisp_enabled. Your agent code does not change between a WebRTC caller and a phone caller. That is the LiveKit bargain from the last post, applied to telephony: the transport is the framework's business.

Pipecat: the phone call becomes a WebSocket

Pipecat's Twilio guide uses Media Streams. Your TwiML answers the call with <Connect><Stream url="wss://…/ws" />, Twilio opens a WebSocket to your server, and a TwilioFrameSerializer sits at the edge of the pipeline translating Twilio's JSON events into Pipecat frames and back.

The <Connect> verb matters. Twilio's Media Streams overview distinguishes the unidirectional <Start><Stream>, which only lets you listen, from the bidirectional <Connect><Stream>, which lets you send audio back — and which blocks the rest of your TwiML until the WebSocket disconnects. A voice agent needs the second.

What arrives is narrowband telephone audio. The message reference fixes the payload at audio/x-mulaw, 8000 Hz, one channel, base64-encoded. The serializer source converts in both directions with ulaw_to_pcm and pcm_to_ulaw, resampling between Twilio's 8000 Hz and whatever rate your pipeline runs at, which is why the guide tells you to set audio_in_sample_rate=8000 and audio_out_sample_rate=8000 rather than pay for resampling you gain nothing from.

Three details in that serializer are worth reading before you ship:

• Interruption is one message. When an InterruptionFrame reaches the serializer, it sends Twilio {"event": "clear", "streamSid": …}. Twilio's reference says "clear" empties all buffered audio and sends back any pending "mark" events. That is the entire barge-in path on the Twilio side — Pipecat's word timestamps still decide what the assistant said, but "clear" is what stops the caller hearing the rest of it.
• Hanging up is a REST call. With auto_hang_up left at its default of "True", an EndFrame or CancelFrame makes the serializer POST Status=completed to the call's resource on Twilio's API — which is why the constructor wants your account SID and auth token alongside the stream_sid and call_sid. The guide tells you to turn it off when you intend to transfer to a human, so the TwiML after your stream can take over the call instead.
• Keypad presses arrive as frames. A Twilio "dtmf" event becomes an InputDTMFFrame, so a processor anywhere in the pipeline can react to "press 1" without knowing it came from a phone.

Custom data rides in through <Parameter> elements inside <Stream>, which Twilio delivers in the "start" event's customParameters. Pipecat's dial-in and dial-out examples show both directions.

ConversationRelay: Twilio keeps the audio

The third option removes the audio from your side of the wire. With <Connect><ConversationRelay>, Twilio runs speech recognition and speech synthesis itself, and your WebSocket carries only text: a "prompt" message with the caller's words, and "text" messages with your tokens streaming back.

The defaults are more interesting than they look. According to the TwiML reference, ttsProvider defaults to ElevenLabs and transcriptionProvider to Deepgram, with Google and Amazon Polly as alternatives; "interruptible" defaults to "any", interruptSensitivity to "high", and there are attributes for welcomeGreeting, dtmfDetection, "hints", backchannel handling and a "preemptible" flag for talking over your own queued speech. So the same ElevenLabs voices from the last post are available here too — the difference is that you pick one with an attribute instead of wiring a service.

Here is the part that connects back to the interruption seam. When a caller talks over playback, ConversationRelay sends your server an interrupt message:

{
  "type": "interrupt",
  "utteranceUntilInterrupt": "Life is a complex set of",
  "durationUntilInterruptMs": 460
}

utteranceUntilInterrupt is the answer to the question the previous post spent a whole section on — what the assistant actually said before it was cut off — delivered as a string. In Pipecat you reconstruct it from word timestamps; here Twilio has already done it, because Twilio was the one playing the audio. Your job shrinks to truncating the conversation history to that string before the next turn.

Leaving is explicit too: an "end" message with handoffData ends the session, and the sendDigits and "play" messages cover dialling and pre-recorded audio.

The price is published: ConversationRelay is $0.07 per minute, with voice minutes billed separately under your normal Twilio plan. The LLM is yours either way — ConversationRelay never runs one.

The seam, three times

Put the three side by side on the one moment that matters, a caller interrupting:

• LiveKit handles it inside the framework. The caller is a participant, the turn detector and interruption logic are the same ones a WebRTC user gets, and SIP is just how the audio arrived.
• Pipecat handles it in the pipeline and tells Twilio to "clear". You own both halves: flushing Twilio's buffer and repairing the context from word timestamps.
• ConversationRelay handles the audio and tells you what happened. You get utteranceUntilInterrupt and durationUntilInterruptMs, and your remaining job is the conversation history.

None of these is wrong. They differ in how much of the problem you are allowed to see, which is the same axis the LiveKit-versus-Pipecat comparison turned on — and ConversationRelay sits further along it than either framework.

How I would actually choose

Take ConversationRelay when the phone is the only channel and the agent's intelligence is the product. You give up voice pipeline control — no custom processors between STT and the LLM, only the providers Twilio offers — and in exchange you write a WebSocket text server, which is the smallest thing on this page.

Take LiveKit over a SIP trunk when the phone is one channel among several. The same agent serves a browser, a mobile app and a phone number, and transfers, DTMF and noise cancellation are configuration rather than code. You pay for it in SIP setup — a trunk on Twilio's side, a trunk and a dispatch rule on LiveKit's — which you do once.

Take Pipecat over Media Streams when you need the audio itself: a processor that inspects raw frames, a model Twilio does not offer, or interruption behaviour you want to own completely. You pay for it in the serializer details above, every one of which is a bug the first time you meet it.

And whichever you choose, remember what the phone line did to the audio before your agent ever heard it: 8000 Hz μ-law. A model evaluated on clean wideband recordings is not being evaluated on the input it will get. Test on telephone audio, from a telephone, before you trust any number about accuracy.