Skip to main content
Pipecat is an open-source Python framework for real-time voice agents: a pipeline of a transport, speech-to-text, an LLM and text-to-speech. pipecat-aircaps adds AirCaps Live 1 to it as a speech-to-text service. AirCapsLiveSTTService streams the user’s audio to AirCaps Live 1, pushes interim transcripts while the user speaks, and returns the turn’s final transcript the moment the turn ends. Turns end in one of two ways:
  • Pipecat decides (default). When Pipecat’s VAD reports the end of speech, the service asks AirCaps for the turn’s final transcript at once.
  • AirCaps decides. With end_of_turn=True the model detects the end of each turn itself.

Prerequisites

  • An AirCaps API key: sign in to the AirCaps console, open API keys and create one. AirCaps Live 1 is available to every account.
  • Keys for the rest of your pipeline. The example below uses OpenAI for the LLM and Cartesia for the voice; any Pipecat LLM and TTS service works.
Each agent conversation holds one AirCaps Live 1 session. Accounts run one session at a time by default, so a second simultaneous call is refused. For more, talk to sales. See Pricing and limits.

Installation

or uv add pipecat-aircaps.

Quickstart

1

Install the packages

The service itself, plus the Pipecat extras this example uses (a WebRTC transport, Silero VAD, OpenAI and Cartesia):
2

Set your API keys

Put them in a .env file beside your bot:
.env
3

Add AirCaps to the pipeline

Create the service and put it where any Pipecat STT service goes, right after the transport’s input:
The service needs VAD frames to know when a turn ends. The user aggregator provides them when it has a VAD analyzer:
4

Run and talk to it

Save the complete bot below as bot.py, then:
Open http://localhost:7860/client and speak. Ask a question, interrupt the answer, and pause mid-sentence to hear how turns end.
bot.py
This is Pipecat’s standard voice agent with AirCaps as its speech-to-text. The same file is in the package’s repository as examples/voice-agent.py.

Configuration

Constructor arguments

Settings

Pass them as AirCapsLiveSTTService.Settings(...).

Turn detection

Pipecat decides (default)

Pipecat’s VAD, turn analyzer and user turn strategies own the turn. When VAD reports that the user stopped speaking (VADUserStoppedSpeakingFrame), the service sends finalize to AirCaps, and the turn’s final transcript comes back as one finalized TranscriptionFrame. Use this mode when you already tune turns in Pipecat (Smart Turn, your own strategies), or want the same turn behaviour across STT providers. It needs a VAD analyzer on the user aggregator, as in the quickstart.

AirCaps decides

With end_of_turn=True the model decides when the user has finished, from what was said as well as from the pause, instead of Pipecat’s VAD and turn analyzer:
  • When a turn’s first words arrive, the service pushes ProposedUserStartedSpeakingFrame.
  • When the model ends the turn, the service pushes the turn’s final TranscriptionFrame (finalized), then ProposedUserStoppedSpeakingFrame.
  • The service recommends ExternalUserTurnStrategies, so the user aggregator follows these frames. You don’t need a VAD analyzer on the user aggregator for turn ends. If you pass your own user_turn_strategies, they win.
  • should_interrupt=False stops a new user turn from interrupting the bot.

End-of-turn settings

A mode is a preset. Start with balanced. Any setting given beside a mode overrides that one value; leave the others unset to keep the mode’s.
All of them can be changed during a call with Pipecat’s STTUpdateSettingsFrame; the AirCaps session reopens to apply them.

How it works

  • Before each connection the service asks AirCaps where to connect (GET /v1/realtime/endpoint), then opens that WebSocket, so audio goes to the server with the lowest latency for your agent.
  • Interim transcripts are pushed as InterimTranscriptionFrames while the user speaks; each turn ends with one finalized TranscriptionFrame, and Pipecat reports STT TTFB from it.
  • If words arrive that VAD did not hear (a short or quiet word), the service asks for their final itself one second after the last new words, so the user turn they started still ends.
  • A session the server ends (sessions last up to an hour) is reopened automatically.
  • While no audio is sent (for example when the STT is muted), the service pings the session so it stays open.
Not supported by AirCaps Live 1: languages other than English, key terms and speaker labels.

Troubleshooting

The error codes are those of the AirCaps Live 1 protocol.

Resources