> ## Documentation Index
> Fetch the complete documentation index at: https://aircaps.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Live streaming

> Real-time captions with A5S Small V2 Streaming over WebSocket.

Endpoint: `wss://api.aircaps.com/v1/realtime`

## Protocol

**1. Connect**

Open the WebSocket with your key in the handshake header: `Authorization: Bearer aircaps_sk_...`

**2. Send start**

The first message is JSON. `language` accepts only `en`; multilingual support is planned.

```json theme={null}
{
  "type": "start",
  "providers": ["a5sv2"],
  "encoding": "pcm_s16le",
  "sample_rate": 16000,
  "channels": 1,
  "language": "en"
}
```

**3. Wait for ready**

The server may send `provider_status` messages while the session starts, then `ready`. Do not send audio before `ready`.

**4. Stream audio**

Send binary frames of mono 16 kHz signed 16-bit little-endian PCM, in real time. 20 ms frames (640 bytes) are recommended; frames may be at most 64 KB. Send `{"type": "ping"}` to keep the connection open while you have no audio to send.

**5. Finish**

Send `{"type": "finish"}`. The server sends the final transcript, then a `finished` message, and closes.

## Messages from the server

```json theme={null}
{"type": "transcript", "provider": "a5sv2", "text": "The complete transcript so far.", "is_final": false}
```

`text` is always the complete transcript so far; replace what you display rather than appending.

For long sessions, set `"transcript_text": "incremental"` in `start` so messages stay small. Each `transcript` then carries an edit instead of `text`: remove `text_delete` characters from the end of the text you have, then append `text_append`. The result is always the complete transcript so far.

```json theme={null}
{"type": "transcript", "provider": "a5sv2", "text_delete": 4, "text_append": "there, everyone", "is_final": false}
```

```python theme={null}
text = ""
def on_transcript(message):
    global text
    text = text[:len(text) - message["text_delete"]] + message["text_append"]
```

| type | Meaning |
| - | - |
| `provider_status` | Startup progress (`message`) |
| `ready` | Start sending audio. Includes `session_limit_ms` and `diarization` |
| `transcript` | Current text; `is_final` is true when a segment is settled. With speaker labels, also `utterances` and `speakers` |
| `session_limit` | The session length or your usage limit was reached; the final transcript and `finished` follow |
| `provider_error` | Streaming failed during the session (`message`); an `error` follows |
| `error` | `code` and `message`; the connection then closes |
| `finished` | `audio_ms` streamed and `billed_ms` counted toward usage |
| `pong` | Reply to `ping` |

## Key terms

Key terms bias recognition toward names and jargon. Send up to 64 in `custom_vocabulary`, each with a `text` (up to 100 characters) and a `strength` from 0 to 12:

```json theme={null}
{
  "type": "start",
  "providers": ["a5sv2"],
  "custom_vocabulary_enabled": true,
  "custom_vocabulary": [{"text": "AirCaps", "strength": 6}]
}
```

## Speaker labels

Set `"diarization": true` in `start` to label who said what, live. Every `transcript` keeps the complete `text` and adds `utterances` and `speakers`:

* Utterances with `is_final: true` became final since the previous message. Each is sent once and never changes: keep it.
* Utterances with `is_final: false` are pending and may still change (their text, speaker or boundaries). Replace the previous pending ones each time.
* Show the kept final utterances and the latest pending ones, ordered by `start`.

Labels are `speaker_0`, `speaker_1`, … in order of first speech, as in async transcripts, and stay the same for the whole session. Words get their speaker about a second after they are spoken; until then, a pending utterance may show a provisional label, or `null` before anyone has been recognized. Overlapping speech comes out as interleaved utterances.

```python theme={null}
final, pending = [], []
def on_transcript(message):
    global pending
    for utterance in message.get("utterances", []):
        if utterance["is_final"]:
            final.append(utterance)
    pending = [u for u in message.get("utterances", []) if not u["is_final"]]
    for u in sorted(final + pending, key=lambda u: u["start"]):
        print(u["speaker"], u["text"])
```

If speaker labels are briefly unavailable, the session ends with `diarization_unavailable` (close code 1013); retry, or start without `diarization`.

## Speaker names

With **AirCaps Speaker ID-v1** enabled on your account, a live session can also put names on its speakers. [Enroll the people](https://aircaps.ai/docs/guides/speaker-id.md#1-create-a-speaker) you expect to hear, then add `speaker_identification` to `start`, together with `"diarization": true` (names need speaker labels):

```json theme={null}
{
  "type": "start",
  "diarization": true,
  "speaker_identification": {
    "threshold": 0.6,
    "speaker_ids": ["spk_01k73a9x8w7v6u5t4s3r2q1p0n", "spk_01k73a7m6n5b4v3c2x1z0l9k8j"]
  }
}
```

| Field | Default | |
| - | - | - |
| `threshold` | `0.6` | A speaker gets a name when their best match among your enrolled speakers scores above this (0 to 1) and is clearly ahead of the next-best. The default works best for most audio; adjust it to what works best for yours. |
| `speaker_ids` | every enrolled speaker | Compare with these people only, for example the meeting's invitees. |

`"speaker_identification": true` uses both defaults. `ready` confirms it with `"speaker_identification": {"model": "aircaps-speaker-id-v1", "threshold": 0.6, "speaker_ids": [...], "speakers_compared": 2}` (`speaker_ids` is `null` when comparing with every enrolled speaker).

Names arrive as `speaker_identification` messages, one for each stretch of a speaker's speech, named or not:

```json theme={null}
{"type": "speaker_identification", "speaker": "speaker_0", "start": 12.48, "end": 13.29,
 "speaker_id": "spk_01k73a9x8w7v6u5t4s3r2q1p0n", "name": "Ana García", "score": 0.781, "margin": 0.512,
 "compared_duration": 4.81, "reason": "accepted",
 "matches": [{"speaker_id": "spk_01k73a9x8w7v6u5t4s3r2q1p0n", "name": "Ana García", "score": 0.781},
             {"speaker_id": "spk_01k73a7m6n5b4v3c2x1z0l9k8j", "name": "Ben Ito", "score": 0.269}]}
```

Utterances also carry `speaker_id` and `speaker_name`: the decision for the same `speaker` that overlaps them most, as known when the utterance is sent (`null` until then). Final utterances are never resent, so to name one after the fact, apply later `speaker_identification` messages the same way:

```python theme={null}
decisions = []
def on_speaker_identification(message):
    decisions.append(message)

def name_of(utterance):
    best, overlap = None, 0.0
    for d in decisions:
        if d["speaker"] != utterance["speaker"]:
            continue
        o = min(d["end"], utterance["end"]) - max(d["start"], utterance["start"])
        if o > overlap:
            best, overlap = d, o
    return (best and best["name"]) or utterance["speaker"]
```

* Labels stay `speaker_0`, `speaker_1`, …; names are added next to them.
* An unnamed decision still lists its closest `matches` (up to 3), so you can apply your own rule. `reason` says why: `accepted`, `score` (best score not above `threshold`), `margin` (too close to the next-best), or `error` (this stretch could not be compared; the session goes on).
* Two speakers can get the same name, for example when one person is briefly split into two speakers.
* No extra usage: the session counts its audio duration, as always.

Without access to AirCaps Speaker ID-v1, a `start` with `speaker_identification` ends with `model_access_denied` (4403); without `"diarization": true`, with `invalid_start` (4400). The named `speaker_ids` must exist and have at least one enrollment (`speaker_not_found`, `speaker_not_enrolled`), and without `speaker_ids` your account needs at least one enrolled speaker (`no_enrolled_speakers`); these close with 4400. If speaker names are briefly unavailable, the session ends with `speaker_identification_unavailable` (1013); retry, or start without `speaker_identification`. Voice data rules apply: see [Privacy and consent](https://aircaps.ai/docs/guides/speaker-id.md#privacy-and-consent).

## Rules

* Send the `start` message within 45 seconds of connecting. It accepts only the documented fields.
* Audio must arrive in real time; sending faster (beyond a 2-second burst) ends the session with `realtime_rate_exceeded`.
* A session can last up to 30 minutes (`session_limit`); open a new one to continue. Longer sessions, up to 4 hours, are available on request ([talk to sales](https://aircaps.ai/contact)). Your account's session length is `max_session_seconds` in [`GET /v1/account`](https://aircaps.ai/docs/api-reference/account).
* No messages (audio or `ping`) for 45 seconds ends the session with `idle_timeout`.
* At most 5 sessions per account at once.

## Errors

Every session that ends abnormally sends an `error` message with a `code`, then closes with the matching WebSocket close code.

| code | Close code | Meaning |
| - | - | - |
| `invalid_start`, `unsupported_language`, `unknown_provider` | 4400 | Bad or missing `start` message |
| `speaker_not_found`, `speaker_not_enrolled`, `no_enrolled_speakers` | 4400 | `speaker_identification` names an unknown or unenrolled speaker, or the account has no enrolled speakers |
| `audio_before_ready`, `invalid_audio_frame`, `invalid_message` | 4400 | Audio sent before `ready`, an empty or odd-length audio frame, or a text message other than `ping`/`finish` |
| `unauthorized` | 4401 | Missing or invalid API key |
| `account_disabled`, `model_access_denied`, `origin_not_allowed` | 4403 | Account disabled, no access to A5S Small V2 Streaming (or to AirCaps Speaker ID-v1 for `speaker_identification`), or browser origin not allowed |
| `idle_timeout` | 4408 | No messages for 45 seconds |
| `quota_unavailable`, `realtime_rate_exceeded`, `frame_too_large` | 4409 | No usage left or too many live sessions; audio faster than real time; a frame over 64 KB ([talk to sales](https://aircaps.ai/contact) to raise limits) |
| `a5sv2_unavailable`, `upstream_unavailable` | 1013 | Temporarily unavailable; retry with backoff |
| `diarization_unavailable` | 1013 | Speaker labels are temporarily unavailable; retry, or start without `diarization` |
| `speaker_identification_unavailable` | 1013 | Speaker names are temporarily unavailable; retry, or start without `speaker_identification` |
| `internal_error` | 1011 | Server error |

A normal session ends with `finished` and close code 1000.

### Retrying

Close code 1013 means try again, whether the session was starting or already streaming: open a new session, waiting longer between attempts (for example 1 s, 2 s, 4 s … up to 30 s, with some jitter). Do the same if the connection can't be opened (for example a handshake timeout, or a refused or reset connection) or drops without a close message. A session that ends early counts only the audio you streamed.

A connection can also go quiet without closing. To notice: if nothing arrives from the server for 30 seconds, send `{"type": "ping"}`; if no `pong` arrives within 10 seconds, close the connection and open a new session as above.

See the [Python sample](https://aircaps.ai/docs/guides/samples.md#stream-a-wav-file).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.