Media Stream format

Custom applications can use Voximplant.WebSocketFormat to send and receive audio over the WebSocket protocol, utilizing JSON serialization.

WebSockets media stream format is described in a proto file available on Github.

Receiving a Media Stream from Voximplant

A media stream always begins with a StartEvent, followed by MediaInfo events. The sequence of MediaInfo events concludes with a StopEvent. Within a scenario, the start and termination of a stream are controlled by the sendMediaTo and stopMediaTo methods. Examples demonstrating the usage of these methods are provided below.

After a media stream has been stopped, a new one can be initiated via a StartEvent, followed by subsequent MediaInfo and StopEvent events. If multiple streams are being transmitted simultaneously, each stream possesses a unique value for the StartEvent.tag, MediaEvent.tag, and StopEvent.tag fields.

StartEvent

This event is generated by the sendMediaTo method. For instance, an audio stream from a call can be directed to a WebSocket as follows:

call.sendMediaTo(webSocket);

This action generates a StartEvent in the following format:

{
    event : 'start',
    sequenceNumber : 0,
    start : {
        mediaFormat : {
            encoding : 'PCM16',
            sampleRate : 16000
        }
    }
}

If necessary, you can set the tag and customParameters for the audio stream, as well as control the audio encoding from the script. For example:

call.sendMediaTo(webSocket, {
    tag: 'stream2',
    encoding: WebSocketAudioEncoding.PCM16_8KHZ,
    customParameters: {test:"123"}  // Note that in this case, the StartEvent.customParameters field
                                    // will contain the string '{\"test\":\"123\"}'.
});

This action generates a StartEvent in the following format:

{
    event : 'start',
    sequenceNumber : 0,
    start : {
        tag : 'stream2',
        mediaFormat : {
            encoding : 'PCM16',
            sampleRate : 8000
        },
        customParameters: '{\"test\":\"123\"}'
    }
}

MediaInfo event

This event is required for delivering audio chunks. An audio chunk exists in the MediaInfo.payload field. Below there is an example of a MediaInfo object containing 20 ms of audio encoded as {encoding : "PCM16", sampleRate : 16000}.

{
    event : "media",
    sequenceNumber : 4,
    media : {
        timestamp : 18880,
        chunk : 3,
        payload : "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=="
    }
}

MediaInfo.payload field

MediaInfo.payload is encoded with the codec specified in StartEvent.mediaFormat.

Audio duration in MediaInfo.payload

The duration of the audio within MediaInfo.payload depends on the audio source: a call, a player, a WebSocket, etc. If the audio source is a call, the chunk length should correspond to the "a=ptime" and "a=maxptime" attributes in that call's SDP. Typically, the audio duration in MediaInfo.payload is 20 milliseconds.

How to determine audio duration in MediaInfo.payload

After decoding from StartEvent.mediaFormat into MediaCodec.Codec.PCM16 at the same sample rate, the audio duration (in milliseconds) equals to:
sizeInBytes(payloadInPcm16) / 2 * 1000 / StartEvent.mediaFormat.sampleRate

MediaEvent.chunk and MediaEvent.timestamp fields

These fields correspond to the sequence number and timestamp fields of the RTP header.

Points to note:

Unlike RFC 3550, the bit-width of MediaEvent.chunk and MediaEvent.timestamp corresponds to the uint64 type.
MediaEvent.chunk values may be skipped. This indicates that some of the RTP packets transmitted via the WebSocket were lost in the network.
The application should be capable of handling lost media chunks — for example, by employing a Packet Loss Concealment (PLC) mechanism.

The number of lost media chunks is calculated as follows:
Current.MediaInfo.chunk - LastReceived.MediaInfo.chunk - 1
When transmitting media chunks via WebSocket, an adaptive jitter buffer is utilized. This buffer resolves issues related to duplicate or reordered packets. However, it does not guarantee that 100% of the media chunks are delivered in the correct order. The application should independently decide how to handle reordered media chunks: whether to ignore them or to accept them.

MediaEvent.tag field

This field is used when more than one media stream is being transmitted. The application should demultiplex MediaInfo.payload into separate media streams, grouping them by MediaInfo.tag.

StopEvent

This event signals the cessation of MediaInfo events for a specific media stream. Subsequently, the only event that may be received is a StartEvent. The StopEvent.tag field allows identification of the specific media stream, while the StopEvent.mediaInfo field provides statistics regarding the completed stream. An example of a StopEvent is provided below.

{
    event : "stop",
    tag : 'stream2',
    sequenceNumber : 1009,
    stop : {
        mediaInfo : {
            bytesSent : 645120,
            duration : 340800
        }
    }
}

Sending a Media Stream to Voximplant

A media stream sent via WebSockets can be routed to a call, a conference, a recorder, another WebSocket, etc. The code snippet below demonstrates a scenario script that routes an audio stream from a WebSocket into a call:

webSocket.sendMediaTo(call);

If you need to send more than one stream, each stream should have its own unique tag. For example, suppose two streams with the stream1 and stream2 tags are sent via a WebSocket. The script code below allows you to demultiplex these streams:

webSocket.sendMediaTo(call, {
    tag: 'stream1' // The first stream is directed to the call
});
webSocket.sendMediaTo(recorder, {
    tag: 'stream2' // The second stream is directed to the recorder
});

Sending the StartEvent

The first event in the media stream should be a StartEvent. It should specify the media stream's codec in the StartEvent.mediaFormat field. If the StartEvent is correctly formed, a WebSocketEvents.MEDIA_STARTED event is triggered in the scenario. The StartEvent.tag, StartEvent.customParameters, and StartEvent.mediaFormat fields are reflected in the corresponding event fields: tag, customParameters, and encoding, respectively.

Sending MediaInfo events

The media stream should be split into chunks in accordance with the following recommendations:

The duration of each MediaInfo.payload may be arbitrary, but it is recommended that it be a multiple of 20 ms.
When sending media chunks, it is not strictly necessary to adhere to real-time timing. That is, for example, you do not necessarily have to send 20 ms audio chunks every 20 ms. You may send all audio chunks at once. They are stored in the WebSocket buffer, which then repackages them into 20 ms audio chunks and forwards them to the appropriate media unit — at that point, in real time.
Please note that the maximum size of the WebSocket buffer is 10 seconds. If this limit is reached, any excess audio chunks are discarded.

The media chunk should be stored in the MediaInfo.payload field. Its encoding should strictly match the StartEvent.mediaFormat value.

MediaEvent.chunk and MediaEvent.timestamp fields

These fields correspond with the sequence number and timestamp fields of the RTP header.

If the media stream does not contain lost, duplicated, or reordered chunks, then:

The MediaEvent.chunk field should be incremented by 1 for each subsequent packet.
The MediaEvent.timestamp value should be calculated as the sum of the samples in the preceding audio chunks within the media stream. The number of samples in each audio chunk is determined by:
sizeInBytes(payloadInPcm16) / 2

If the media stream contains lost, duplicated, or reordered media chunks, then the MediaEvent.chunk and MediaEvent.timestamp fields should accurately reflect these losses, duplications, and reorderings.

MediaEvent.tag field

This field is used when it is necessary to transmit more than one media stream simultaneously. The MediaInfo.tag should match the StartEvent.tag of its corresponding stream. The transmitted media chunks are demultiplexed into separate media streams based on their MediaInfo.tag and routed to the specific media unit selected in the scenario.

Sending a StopEvent

Once the transmission of media chunks is complete, a StopEvent should be sent. If the StartEvent for the given stream included a tag field, then the StopEvent.tag should contain the exact same value. If the StopEvent is correctly formatted, a WebSocketEvents.MEDIA_ENDED event is triggered within the scenario.

Changing the media codec

The codec specified in StartEvent.mediaFormat should not be altered in subsequent MediaInfo events. Therefore, if you need to change the codec, you should first terminate the current stream by sending a StopEvent, and only then initiate a new stream using a StartEvent that specifies the new MediaCodec.