Live streaming

Endpoint: wss://api.aircaps.com/v1/realtime

Protocol

Connect

Open the WebSocket with your key in the handshake header: Authorization: Bearer aircaps_sk_...

Send start

The first message is JSON. language accepts only en; multilingual support is planned.

{
  "type": "start",
  "providers": ["a5sv2"],
  "encoding": "pcm_s16le",
  "sample_rate": 16000,
  "channels": 1,
  "language": "en"
}

Wait for ready

The server may send provider_status messages while the session starts, then ready. Do not send audio before ready.

Stream audio

Send binary frames of mono 16 kHz signed 16-bit little-endian PCM, in real time. 20 ms frames (640 bytes) are recommended; frames may be at most 64 KB. Send {"type": "ping"} to keep the connection open while you have no audio to send.

Finish

Send {"type": "finish"}. The server sends the final transcript, then a finished message, and closes.

Messages from the server

{"type": "transcript", "provider": "a5sv2", "text": "The complete transcript so far.", "is_final": false}

text is always the complete transcript so far; replace what you display rather than appending. For long sessions, set "transcript_text": "incremental" in start so messages stay small. Each transcript then carries an edit instead of text: remove text_delete characters from the end of the text you have, then append text_append. The result is always the complete transcript so far.

{"type": "transcript", "provider": "a5sv2", "text_delete": 4, "text_append": "there, everyone", "is_final": false}

text = ""
def on_transcript(message):
    global text
    text = text[:len(text) - message["text_delete"]] + message["text_append"]

type	Meaning
`provider_status`	Startup progress (`message`)
`ready`	Start sending audio. Includes `session_limit_ms` and `diarization`
`transcript`	Current text; `is_final` is true when a segment is settled. With speaker labels, also `utterances` and `speakers`
`session_limit`	The session length or your usage limit was reached; the final transcript and `finished` follow
`provider_error`	Streaming failed during the session (`message`); an `error` follows
`error`	`code` and `message`; the connection then closes
`finished`	`audio_ms` streamed and `billed_ms` counted toward usage
`pong`	Reply to `ping`

Key terms

Key terms bias recognition toward names and jargon. Send up to 64 in custom_vocabulary, each with a text (up to 100 characters) and a strength from 0 to 12:

{
  "type": "start",
  "providers": ["a5sv2"],
  "custom_vocabulary_enabled": true,
  "custom_vocabulary": [{"text": "AirCaps", "strength": 6}]
}

Speaker labels

Set "diarization": true in start to label who said what, live. Every transcript keeps the complete text and adds utterances and speakers:

Utterances with is_final: true became final since the previous message. Each is sent once and never changes: keep it.
Utterances with is_final: false are pending and may still change (their text, speaker or boundaries). Replace the previous pending ones each time.
Show the kept final utterances and the latest pending ones, ordered by start.

Labels are speaker_0, speaker_1, … in order of first speech, as in async transcripts, and stay the same for the whole session. Words get their speaker about a second after they are spoken; until then, a pending utterance may show a provisional label, or null before anyone has been recognized. Overlapping speech comes out as interleaved utterances.

final, pending = [], []
def on_transcript(message):
    global pending
    for utterance in message.get("utterances", []):
        if utterance["is_final"]:
            final.append(utterance)
    pending = [u for u in message.get("utterances", []) if not u["is_final"]]
    for u in sorted(final + pending, key=lambda u: u["start"]):
        print(u["speaker"], u["text"])

If speaker labels are briefly unavailable, the session ends with diarization_unavailable (close code 1013); retry, or start without diarization.

Speaker names

With AirCaps Speaker ID-v1 enabled on your account, a live session can also put names on its speakers. Enroll the people you expect to hear, then add speaker_identification to start, together with "diarization": true (names need speaker labels):

{
  "type": "start",
  "diarization": true,
  "speaker_identification": {
    "threshold": 0.6,
    "speaker_ids": ["spk_01k73a9x8w7v6u5t4s3r2q1p0n", "spk_01k73a7m6n5b4v3c2x1z0l9k8j"]
  }
}

Field	Default
`threshold`	`0.6`	A speaker gets a name when their best match among your enrolled speakers scores above this (0 to 1) and is clearly ahead of the next-best. The default works best for most audio; adjust it to what works best for yours.
`speaker_ids`	every enrolled speaker	Compare with these people only, for example the meeting’s invitees.

"speaker_identification": true uses both defaults. ready confirms it with

"speaker_identification": {"model": "aircaps-speaker-id-v1", "threshold": 0.6, "speaker_ids": [...], "speakers_compared": 2}

(speaker_ids is null when comparing with every enrolled speaker). Names arrive as speaker_identification messages, one for each stretch of a speaker’s speech, named or not:

{"type": "speaker_identification", "speaker": "speaker_0", "start": 12.48, "end": 13.29,
 "speaker_id": "spk_01k73a9x8w7v6u5t4s3r2q1p0n", "name": "Ana García", "score": 0.781, "margin": 0.512,
 "compared_duration": 4.81, "reason": "accepted",
 "matches": [{"speaker_id": "spk_01k73a9x8w7v6u5t4s3r2q1p0n", "name": "Ana García", "score": 0.781},
             {"speaker_id": "spk_01k73a7m6n5b4v3c2x1z0l9k8j", "name": "Ben Ito", "score": 0.269}]}

Utterances also carry speaker_id and speaker_name: the decision for the same speaker that overlaps them most, as known when the utterance is sent (null until then). Final utterances are never resent, so to name one after the fact, apply later speaker_identification messages the same way:

decisions = []
def on_speaker_identification(message):
    decisions.append(message)

def name_of(utterance):
    best, overlap = None, 0.0
    for d in decisions:
        if d["speaker"] != utterance["speaker"]:
            continue
        o = min(d["end"], utterance["end"]) - max(d["start"], utterance["start"])
        if o > overlap:
            best, overlap = d, o
    return (best and best["name"]) or utterance["speaker"]

Labels stay speaker_0, speaker_1, …; names are added next to them.
An unnamed decision still lists its closest matches (up to 3), so you can apply your own rule. reason says why: accepted, score (best score not above threshold), margin (too close to the next-best), or error (this stretch could not be compared; the session goes on).
Two speakers can get the same name, for example when one person is briefly split into two speakers.
No extra usage: the session counts its audio duration, as always.

Without access to AirCaps Speaker ID-v1, a start with speaker_identification ends with model_access_denied (4403); without "diarization": true, with invalid_start (4400). The named speaker_ids must exist and have at least one enrollment (speaker_not_found, speaker_not_enrolled), and without speaker_ids your account needs at least one enrolled speaker (no_enrolled_speakers); these close with 4400. If speaker names are briefly unavailable, the session ends with speaker_identification_unavailable (1013); retry, or start without speaker_identification. Voice data rules apply: see Privacy and consent.

Rules

Send the start message within 45 seconds of connecting. It accepts only the documented fields.
Audio must arrive in real time; sending faster (beyond a 2-second burst) ends the session with realtime_rate_exceeded.
A session can last up to 30 minutes (session_limit); open a new one to continue. Longer sessions, up to 4 hours, are available on request (talk to sales). Your account’s session length is max_session_seconds in GET /v1/account.
No messages (audio or ping) for 45 seconds ends the session with idle_timeout.
At most 5 sessions per account at once.

Errors

Every session that ends abnormally sends an error message with a code, then closes with the matching WebSocket close code.

code	Close code	Meaning
`invalid_start`, `unsupported_language`, `unknown_provider`	4400	Bad or missing `start` message
`speaker_not_found`, `speaker_not_enrolled`, `no_enrolled_speakers`	4400	`speaker_identification` names an unknown or unenrolled speaker, or the account has no enrolled speakers
`audio_before_ready`, `invalid_audio_frame`, `invalid_message`	4400	Audio sent before `ready`, an empty or odd-length audio frame, or a text message other than `ping`/`finish`
`unauthorized`	4401	Missing or invalid API key
`account_disabled`, `model_access_denied`, `origin_not_allowed`	4403	Account disabled, no access to A5S Small V2 Streaming (or to AirCaps Speaker ID-v1 for `speaker_identification`), or browser origin not allowed
`idle_timeout`	4408	No messages for 45 seconds
`quota_unavailable`, `realtime_rate_exceeded`, `frame_too_large`	4409	No usage left or too many live sessions; audio faster than real time; a frame over 64 KB (talk to sales to raise limits)
`a5sv2_unavailable`, `upstream_unavailable`	1013	Temporarily unavailable; retry with backoff
`diarization_unavailable`	1013	Speaker labels are temporarily unavailable; retry, or start without `diarization`
`speaker_identification_unavailable`	1013	Speaker names are temporarily unavailable; retry, or start without `speaker_identification`
`internal_error`	1011	Server error

A normal session ends with finished and close code 1000.

Retrying

Close code 1013 means try again, whether the session was starting or already streaming: open a new session, waiting longer between attempts (for example 1 s, 2 s, 4 s … up to 30 s, with some jitter). Do the same if the connection can’t be opened (for example a handshake timeout, or a refused or reset connection) or drops without a close message. A session that ends early counts only the audio you streamed. A connection can also go quiet without closing. To notice: if nothing arrives from the server for 30 seconds, send {"type": "ping"}; if no pong arrives within 10 seconds, close the connection and open a new session as above. See the Python sample.

Get started

Guides

Operations

Protocol

Messages from the server

Key terms

Speaker labels

Speaker names

Rules

Errors

Retrying

​Protocol

​Messages from the server

​Key terms

​Speaker labels

​Speaker names

​Rules

​Errors

​Retrying

Protocol

Messages from the server

Key terms

Speaker labels

Speaker names

Rules

Errors

Retrying