Getting Started With Voiceitt WebSockets API

General

The URL of the Voiceitt WebSockets Socket.IO-based API is http://web.voiceitt.com/socket.io/ .

Workflow

The WebSockets recognition workflow is centred around a socket connection.

Initialising a socket connection

Connections need to be authenticated by passing a JSON Web Token (JWT) via the auth socket option. The JWT token is obtained via login to the Voiceitt HTTP API (sign_in endpoint).

{  
  token: str  // JWT token obtained via login into the Voiceitt HTTP API
}

  const options = {
    path: '/socket.io',
    autoConnect: true,
    withCredentials: true,
    transports: ['websocket', 'polling'],
    auth: { token, refresh_token: refreshToken }
  }
    
  const url = 'https://web.voiceitt.com/socket.io'
    
  socketLocal = io(url, options)

Possible events raised at connection time are:

model_missing,
model_loading,
model_loading_error,
model_ready,
critical_error,
connection_ready

Connection setup progress is indicated to the client by one or more of the above events - the typical sequence is model_loading followed by model_ready followed by connection_ready. A connection_ready event is emitted by the server after completing connection initialisation, indicating that recognition requests can now be accepted by the server.

Configuring recognition options

set_options

The set_options event is used to configure the recognition behaviour for the duration of the connection.

Refreshing authentication token

refresh_token

When the JWT token used for authentication is about to expire, the client may emit a refresh_token event with a new token as payload. If the new token fails authentication, the socket will be disconnected.

Terminating a connection

disconnect_request

The client can request to close a connection by emitting a disconnect_request event.

Recognition requests

The recognition service can be used to transcribe individual speech segments or to transcribe speech in streaming mode. The client may emit recognize_audio_samples, stream_audio_samples or stream_compressed_audio events at any point after receiving a connection_ready event.

The recognize_audio_samples event is intended for pre-segmented speech in PCM format, not audio streams.

The stream_audio_samples event is intended for PCM format audio streams.

The stream_compressed_audio event is for streaming non PCM format audio streams.

For PCM format audio, the API currently supports single-channel audio sampled at 16000 Hz, with samples in either 32- or 64-bit floating point format, or signed 16- or 32-bit integer format.

Streaming audio recognition

The stream_audio_samples event can be used to send contiguous chunks of audio for streaming audio processing. The event payload is the following:

audio samples serialized as bytes
audio sample type string (one of ‘int16’, ‘int32’, ‘float32’, ‘float64’)

The recommended chunk duration is 3200 samples, however no duration limits are currently enforced, and arbitrary chunk sizes can be processed by the server.

An alternative event for sending audio chunks is stream_compressed_audio, designed to operate with compressed audio formats like mp4 or ogg. The event payload is the following:

Audio samples in binary format
MIME type

For Web environments according to the type of generated chunks stream_audio_samples should use ScriptProcessorNode and stream_compressed_audio should use MediaRecorder. Experiments show that MediaRecorder is preferred because it produces much less audio distortions.

The server will emit recognition events while streaming is active, as soon as speech segments are detected and transcribed. The recognition event payload will omit the request ID in streaming recognition mode.

Recognition requests performed while the model status differs from model_ready will return an error via the error event.

Error notifications

The error event may be emitted by the server to indicate incorrect API usage or internal server errors. The error event payload is the following:

{  
  message: str
  request_id:  str   // optional, may be omitted if not applicable
}

Shutdown notifications

A server instance may shut down for maintenance purposes or as part of downscaling the pool of server instances. In this scenario two notifications are sent to connected WebSocket connections: pre_shutdown and shutdown. The pre_shutdown event is sent to clients as a warning message. A grace period of duration indicated by the shutdown_grace_period_seconds field follows this event. In the grace period the server instance:

stops accepting new connections
continues to process requests

For continued operation, clients should reconnect to establish a new connection to a server instance before the grace period expires.

The pre_shutdown event payload is the following:

{  
  message: str
  shutdown_grace_period_seconds: float
}

On expiration of the grace period a shutdown event is sent to any connected WebSocket clients. The shutdown event payload is the following:

{  
  message: str
}

This indicates a hard shutdown of the server instance. The server instance subsequently stops processing any requests and closes all WebSocket connections.

Load balancing

The recognition server implements session affinity by setting an AWS application load balancer (AWSALB) cookie. A specific server instance may be unable to accept new connections, in which case a critical_error status event will be sent, accompanied by a reset_alb_cookies event. The client should respond to a reset_alb_cookies event by clearing the AWSALB cookie and attempting a new connection.