stream_audio_samples

Transcribe contiguous chunks of PCM-format audio

This event can be used to send contiguous chunks of audio for recognition. The event payload is the following:

• audio samples serialized as bytes
• audio sample type string (one of ‘int16’, ‘int32’, ‘float32’, ‘float64’)

The preferred chunk duration is 3200 samples, however no duration limits are currently enforced, and arbitrary chunk sizes can be processed by the server.

JavaScript

path = '/test_audio.wav'
type = 'int16'
  
const response = await fetch(path)
const blob = await response.blob()
const arrayBuffer = await blob.arrayBuffer()
const typedArray = new Int16Array(arrayBuffer)

const chunks = _.chunk(typedArray, 3200);
for (const chunk of chunks) {
  const audioData = new Int16Array(chunk)
  socketLocal.emit('stream_audio_samples', audioData, type)
}

The stream of audio is split into a chronologically ordered sequence of segments containing speech by a process called voice activity detection. Each segment is processed and a chronologically ordered sequence of recognition responses are generated, each with the following payload:

JavaScript

{  
  text: str
  segment_id: int
  start_time: float
  end_time: float
}

The response fields signify the following:

• text: contains the recognised word sequence corresponding to the segment.
• segment_id: indicates the chronological order of the segment within the audio stream.
• start_time: the start time of the audio segment in seconds, measured from the start of the audio stream.
• end_time: the end time of the audio segment in seconds, measured from the start of the audio stream.

Each event may be captured as follows:

JavaScript

socketLocal.on('recognition', (data) => {
  console.log('recognition', data.text, new Date())
}

Further, for each segment, a chronologically ordered sequence of partial_recognition events are emitted in response, each with the following payload:

JavaScript

{  
  text: str
  unstable_text: str
  segment_id: int
}

The response fields signify the following:

• text: any new stable part of the streaming recogniser output (i.e. not changed in future partial_recognition events).
• unstable_text : part of the streaming recogniser output which may change in future partial_recognition events
• segment_id : indicates the chronological order of the segment within the audio stream.

Each event may be captured as follows:

JavaScript

socketLocal.on('partial_recognition', (data) => {
  console.log('partial_recognition', data.text, new Date())
  console.log('partial_recognition', data.unstable_text, new Date())
}

For example, the following sequence of partial_recognition events may occur.

JavaScript

{ text: HELLO, unstable_text: WORLD I WANT, segment_id: 23}  
{ text: WORLD I, unstable_text: WANTED TWO AT, segment_id: 23}
{ text: WANTED TO, unstable_text: AT A QUESTION, segment_id: 23}
{ text: ASK A QUESTION, unstable_text: '', segment_id, 23}
{ text: HELLO, unstable_text: AGAIN, segment_id: 24}  
{ text: AGAIN, unstable_text: '', segment_id: 24}