Realtime stream

WSS wss://api.aircaps.com/v1/realtime Requires access to A5S Small V2 Streaming. For a walkthrough, see Live streaming.

Connection

string

required

Bearer aircaps_sk_..., sent with the WebSocket handshake.

The flow: connect, send start, wait for ready, stream binary audio, send finish, then receive finished. The server then closes with 1000.

Client messages

start

The first message, as a JSON text frame, within 45 seconds of connecting. Unknown fields are rejected.

string

required

"start"

string

default:"en"

Language of the audio. English only today: en. Multilingual support is planned.

string

default:"pcm_s16le"

Only pcm_s16le (signed 16-bit little-endian PCM).

integer

default:"16000"

Only 16000.

integer

default:"1"

Only 1 (mono).

string[]

default:"[\"a5sv2\"]"

The streaming engine. Only "a5sv2".

boolean

default:"false"

Turns on key terms: biases recognition toward the entries in custom_vocabulary.

object[]

Key terms: up to 64. See Key terms.

Show properties

string

required

A word or phrase, up to 100 characters.

number

required

Bias strength, 0 to 12.

boolean

default:"false"

Label who said what. Each transcript then also carries utterances with a speaker on every utterance and word, and speakers. See Speaker labels.

string

default:"full"

"full": every transcript carries the complete transcript so far in text. "incremental": every transcript instead carries an edit of the text you have: text_delete and text_append (below). Use "incremental" for long sessions: in "full" mode each message grows with the transcript.

boolean | object

default:"false"

Put names on the speakers with AirCaps Speaker ID-v1 (requires access to that model, and "diarization": true). true uses the defaults. Sends a speaker_identification message (below) for each stretch of a speaker’s speech, and adds speaker_id and speaker_name to utterances. See Speaker names.

Show properties

number

default:"0.6"

A speaker gets a name when their best match scores above this (0 to 1) and is clearly ahead of the next-best.

string[]

Compare with these enrolled speakers only. Default: every enrolled speaker on the account.

Example

{"type": "start", "language": "en", "encoding": "pcm_s16le", "sample_rate": 16000, "channels": 1,
 "diarization": true,
 "custom_vocabulary_enabled": true, "custom_vocabulary": [{"text": "AirCaps", "strength": 6}]}

Audio

Binary frames of raw PCM in the start format: non-empty, an even number of bytes, at most 64 KB (larger frames end the session with frame_too_large; frames over 1 MB close the connection without a message). Send them after ready, in real time; a 2-second burst is allowed. 20 ms frames (640 bytes) are recommended.

ping

{"type": "ping"}. The server answers pong. Any message, audio or ping, resets the 45-second idle timer.

finish

{"type": "finish"}. The server flushes the final transcript, sends finished, and closes.

Server messages

Each is a JSON text frame with a type.

provider_status

Startup progress while the session starts.

string

For example "Getting things ready, please hold on".

ready

Start sending audio.

integer

Maximum audio for this session: your session length (30 minutes unless your account has a longer one, up to 4 hours), or less if little free usage and credit remain.

string

"full" or "incremental", as requested in start.

integer

The account’s free usage, in milliseconds.

integer

Free usage used so far, in milliseconds.

string

"en-US"

integer

16000

string

"pcm_s16le"

boolean

Whether speaker labels are on for this session.

object

Only with speaker_identification: model ("aircaps-speaker-id-v1"), threshold, speaker_ids (null for every enrolled speaker) and speakers_compared.

object[]

Only when start named a deprecated model id (a5s-v2-streaming, now a5s-small-v2-streaming): {code, message} with code "model_deprecated". The session runs normally.

transcript

string

With transcript_text "full" (the default): the complete transcript so far, punctuated and cased. Replace what you display instead of appending.

integer

With transcript_text "incremental": remove this many characters from the end of the text you have (recent words the engine revised; often 0).

string

With "incremental": then append this. The result is the complete transcript so far, as text would have been.

boolean

true when the current segment is settled.

string

"a5sv2"

object[]

Only with diarization. The utterances that became final since the previous transcript (is_final: true; each is sent once and never changes), followed by every pending utterance (is_final: false; these may still change, so replace the previous pending ones). Keep the final ones and show them with the latest pending ones, ordered by start.

Show properties

string | null

speaker_0, speaker_1, … in order of first speech, as in async transcripts. null on a pending utterance before any speaker has been recognized.

number

Seconds from the start of the session.

number

Seconds from the start of the session.

string

Punctuated and cased.

boolean

true once the utterance will not change.

object[]

{text, start, end, speaker} for each word.

string | null

Only with speaker_identification. The enrolled speaker named by the speaker_identification decision for this speaker that overlaps the utterance most, as known when it is sent; null until then or when unnamed.

string | null

Only with speaker_identification. That speaker’s name.

string[]

Only with diarization. The speaker labels used so far in final utterances.

Example

{"type": "transcript", "provider": "a5sv2", "text": "Good afternoon, everybody.", "is_final": true}

Example with diarization

{"type": "transcript", "provider": "a5sv2", "text": "Shall we start? Sure, go ahead.", "is_final": false,
 "utterances": [
   {"speaker": "speaker_0", "start": 0.4, "end": 1.36, "text": "Shall we start?", "is_final": true,
    "words": [{"text": "Shall", "start": 0.4, "end": 0.64, "speaker": "speaker_0"}, "..."]},
   {"speaker": "speaker_1", "start": 1.92, "end": 2.88, "text": "Sure, go ahead.", "is_final": false,
    "words": ["..."]}],
 "speakers": ["speaker_0"]}

speaker_identification

Only with speaker_identification. One decision for a stretch of one speaker’s speech, usually sent about a second after it ends, named or not.

string

The speaker label, as in utterances.

number

Start of the stretch, in seconds from the start of the session.

number

End of the stretch, in seconds from the start of the session.

string | null

The enrolled speaker it was named as, or null.

string | null

That speaker’s name, or null.

number | null

The best match’s similarity score, -1 to 1.

number | null

How far the best score is above the next-best; null with one enrolled speaker.

object[]

Up to 3 closest enrolled speakers, best first: {speaker_id, name, score}.

number

Seconds of speech compared.

string

accepted, score (best score not above threshold), margin (too close to the next-best), or error (this stretch could not be compared; the session goes on).

Example

{"type": "speaker_identification", "speaker": "speaker_0", "start": 12.48, "end": 13.29,
 "speaker_id": "spk_01k73a9x8w7v6u5t4s3r2q1p0n", "name": "Ana García", "score": 0.781, "margin": 0.512,
 "compared_duration": 4.81, "reason": "accepted",
 "matches": [{"speaker_id": "spk_01k73a9x8w7v6u5t4s3r2q1p0n", "name": "Ana García", "score": 0.781},
             {"speaker_id": "spk_01k73a7m6n5b4v3c2x1z0l9k8j", "name": "Ben Ito", "score": 0.269}]}

session_limit

The session reached its session length (30 minutes unless your account has a longer one) or the account’s usage limit. The final transcript and finished follow, then close code 1000.

string

Why the session ended.

finished

integer

Audio received.

integer

Audio counted toward usage.

integer

Free usage used after this session.

integer

The account’s free usage.

provider_error

Streaming failed during the session. An error follows.

string

What failed.

error

Sent before the server ends the session abnormally; the connection then closes with the matching close code.

string

One of the codes below.

string

Human-readable detail.

code	Close	Meaning
`invalid_start`	4400	The first message is missing, late or invalid (including `speaker_identification` without `diarization`)
`unsupported_language`	4400	`language` is not English
`unknown_provider`	4400	`providers` names an unknown engine
`speaker_not_found`, `speaker_not_enrolled`, `no_enrolled_speakers`	4400	`speaker_identification.speaker_ids` names an unknown speaker or one with no enrollments, or the account has no enrolled speakers
`audio_before_ready`	4400	Audio sent before `ready`
`invalid_audio_frame`	4400	An empty or odd-length audio frame
`invalid_message`	4400	A text message other than `ping` or `finish`
`unauthorized`	4401	Missing or invalid API key
`account_disabled`	4403	The account is disabled
`model_access_denied`	4403	No access to A5S Small V2 Streaming, or to AirCaps Speaker ID-v1 for `speaker_identification` (talk to sales)
`origin_not_allowed`	4403	Browser origin not allowed
`idle_timeout`	4408	No messages for 45 seconds
`quota_unavailable`	4409	Usage limit used up, or too many live sessions (talk to sales)
`insufficient_balance`	4409	Free usage and credits are used up: add credits in the console
`realtime_rate_exceeded`	4409	Audio sent faster than real time
`frame_too_large`	4409	An audio frame over 64 KB
`a5sv2_unavailable`, `upstream_unavailable`	1013	Temporarily unavailable, at start or during the session; open a new session with backoff
`diarization_unavailable`	1013	Speaker labels are temporarily unavailable; retry, or start without `diarization`
`speaker_identification_unavailable`	1013	Speaker names are temporarily unavailable; retry, or start without `speaker_identification`
`internal_error`	1011	Server error

pong

The reply to ping.

Close codes

Code	Meaning
1000	Normal close after `finished`
1011	Server error
1013	Temporarily unavailable; open a new session with backoff (for example 1 s, 2 s, 4 s … up to 30 s)
4400	Invalid message or frame
4401	Authentication failed
4403	Forbidden: account disabled, no access to the model, or browser origin not allowed
4408	Idle timeout
4409	Limit reached at start (usage, sessions), audio faster than real time, or a frame over 64 KB

Overview

Transcripts

Diarizations

Speakers

Files

Realtime

Account

Objects

Connection

Client messages

start

Audio

ping

finish

Server messages

provider_status

ready

transcript

speaker_identification

session_limit

finished

provider_error

error

pong

Close codes

​Connection

​Client messages

​start

​Audio

​ping

​finish

​Server messages

​provider_status

​ready

​transcript

​speaker_identification

​session_limit

​finished

​provider_error

​error

​pong

​Close codes

Connection

Client messages

start

Audio

ping

finish

Server messages

provider_status

ready

transcript

speaker_identification

session_limit

finished

provider_error

error

pong

Close codes