LiveKit, Pipecat, and the seam where a voice agent actually breaks

Pick a voice stack and you are really picking two things: who owns the audio transport, and what you can see when a turn goes wrong. The vendor list barely matters — LiveKit Agents and Pipecat both put the same speech-to-text, language model and text-to-speech providers behind the same three seams, and both let you swap any of them in a line.

LangSmith shipped tracing for both in July, alongside OpenAI Realtime and Gemini Live. That is a useful forcing function for this comparison, because a trace shows you where a framework's abstraction actually sits — and the two answers are genuinely different.

⚠️ What follows is read, not measured. I have not run a latency benchmark across these two frameworks, and you should distrust anyone who publishes one without naming their network, their region, their model versions and their audio codec. Everything below comes from the documentation and the published source, with links to each claim.

One owns the room; the other owns the pipeline

LiveKit starts from transport. Agents communicate with users over WebRTC, and the deployment model follows from that: your agent code registers with a LiveKit server, waits for a dispatch request, and boots a job subprocess that joins a room. The room is the primitive. You get that machinery whether you want it or not, which is the point — WebRTC on bad mobile networks is a specialist subject, and it is not the subject you wanted to become an expert in.

The code reads as a session with slots:

session = AgentSession(
    stt=inference.STT(model="assemblyai/universal-3-5-pro", language="en"),
    llm=inference.LLM(model="google/gemma-4-31b-it"),
    tts=inference.TTS(model="fishaudio/s2.1-pro", voice="..."),
    turn_handling=TurnHandlingOptions(turn_detection=inference.TurnDetector()),
)

Pipecat starts from the dataflow. Its unit is the frame, and a pipeline is a declarative list of processors that Pipecat wires together — the documentation reaches for Unix pipes and Kafka as the analogy, which is exactly right. The canonical voice pipeline is written out in order:

transport.input() → stt → llm → tts → transport.output() → context_aggregator.assistant()

Transport is a processor at each end, not the framework's organising idea. That is the whole architectural difference in one line, and most of the practical consequences fall out of it.

The consequence that matters: in Pipecat you can see and reorder the stages, insert your own FrameProcessor between any two, and watch typed frames — LLMTextFrame, TTSAudioRawFrame, TTSStartedFrame — move through them. In LiveKit you configure a session and the orchestration is the framework's business. One is a kit; the other is a product with sockets.

The same ElevenLabs, wired two different ways

Both treat ElevenLabs as first-class, and the install lines are equally boring. LiveKit: livekit-agents[elevenlabs]~=1.8, then elevenlabs.TTS(), defaulting to eleven_turbo_v2_5, with the usual voice-tuning knobs, a character-based flush schedule and optional SSML parsing. There is a Node equivalent, @livekit/agents-plugin-elevenlabs@1.x, which matters if your team is not Python-first.

Pipecat ships two ElevenLabs classes, not one: ElevenLabsTTSService over a persistent WebSocket to wss://api.elevenlabs.io, and ElevenLabsHttpTTSService over plain HTTP streaming. That choice is exposed to you deliberately, and the WebSocket class carries the interesting surface: an auto_mode flag that trades control for server-side latency optimisation, plus flush_audio() and on_turn_context_completed() for managing the audio context mid-turn.

Now the seam that actually breaks.

A caller talks over your agent halfway through a sentence. You must stop the audio, and then you must decide what the assistant said — because whatever you write into the conversation history is what the model believes it said. Get that wrong and the agent apologises for something nobody heard, or repeats a sentence it already delivered.

Both frameworks lean on the same ElevenLabs capability, word-level timestamps derived from character alignment, and they spend it differently:

• Pipecat uses it for context repair. Word timestamps let it accurately capture which words were spoken at the moment of interruption, and the source carries a calculate_word_times() that handles partial words spanning alignment chunks — an unglamorous detail that tells you someone hit the real bug.
• LiveKit uses it for frontend synchronisation. Setting use_tts_aligned_transcript=True turns on aligned transcription forwarding, so the captions your user sees track the audio they hear.

Both are correct. They are solving different halves of the same problem, and which half you care about is a product question, not a framework question. If your agent is a phone line, context repair is the one that stops you sounding broken. If it has a screen, alignment is the one users notice.

Observability: the same substrate, different switches

This is where the LangSmith post earns its place in the comparison. Both integrations are OpenTelemetry underneath, and both collapse a conversation into one trace with a span per stage.

Pipecat's is explicit about what you must turn on, which is very Pipecat:

from langsmith.integrations.pipecat import configure_pipecat

configure_pipecat()
task = PipelineTask(
    pipeline,
    params=PipelineParams(enable_metrics=True),
    enable_tracing=True,
    enable_turn_tracking=True,
)

Three flags, documented as required, and a PipecatLangSmithSpanProcessor you can attach to your own TracerProvider if you already run OTel. If your LLM stage is a LangGraph agent, you pass configure_pipecat(llm_span_kind="chain").

LiveKit's is one call before the server starts — configure_livekit(), from langsmith[livekit]>=0.11.2, with set_thread_id() to group a conversation and instrument_session() needed only for speech-to-speech models, where there is no separate STT stage to read the user's words from.

That last asymmetry is worth pausing on, because it is not about LangSmith at all. In the sandwich architecture — STT, then LLM, then TTS — the transcript is a real artifact produced by a real stage, so tracing gets it for free. In a speech-to-speech model the transcript is a by-product you have to go and ask for. Any observability tool will need that extra hook, and any team moving from the sandwich to a realtime model will discover a hole in their traces exactly where their evaluation data used to come from.

What the traces then show is the same for both: full conversation audio overlaid on the trace, STT and TTS latency, voice activity detection events, interruptions and overlapping speech, and timing across each stage.

Where ElevenLabs sits in the picture

Worth knowing before you draw a boundary in the wrong place: ElevenLabs is not only a vendor behind these frameworks. Its own Pipecat integration and LiveKit integration are documented from its side too, and its WebRTC mode uses LiveKit internally.

So "LiveKit or Pipecat" and "roll your own or use ElevenLabs' platform" are not the same axis, and a stack can contain LiveKit twice — once as your transport, once inside a vendor you bought precisely so you would not have to think about transport.

How I would actually choose

Not on latency, and not on the vendor list, because those converge.

Take LiveKit when the transport is the hard part: real phone calls, flaky mobile networks, many participants in a room, or a compliance requirement that sends you to a self-hosted server. You are buying WebRTC expertise and a turn detector, and accepting that the orchestration is inside the framework.

Take Pipecat when the pipeline is the hard part: you need a processor of your own between stages, you want the WebSocket-versus-HTTP TTS decision in your hands, or your interruption behaviour is a product feature rather than a default. You are buying legibility and paying for it in assembly.

And whichever you take, turn the tracing on during the first week rather than the week you have a problem. The failure that will cost you is not a slow model — it is an agent that believes it said something the caller never heard, and that one is invisible in every metric except a trace with the audio attached.