Realtime API Client

In the rapidly evolving landscape of AI-driven applications, real-time communication and interactivity have become pivotal in delivering enhanced user experiences. Voximplant's OpenAI realtime API client bridges the gap between cutting-edge AI capabilities and real-time communication, empowering developers to seamlessly integrate OpenAI's advanced models into voice applications.

This guide serves as a comprehensive resource for developers, providing step-by-step instructions, practical use cases, and best practices for leveraging this powerful API client. Whether you are building conversational AI, automating customer interactions, or crafting innovative real-time solutions, this guide helps you unlock the full potential of Voximplant’s integration with OpenAI, enabling smarter, more engaging, and responsive applications.

Any Voximplant call can be connected to an OpenAI agent in a VoxEngine scenario; for that purpose, we have introduced the RealtimeAPIClient class. Your OpenAI API KEY is required to create its instance:

const OPENAI_API_KEY = VoxEngine.getSecretValue('openai_api_key');
createRealtimeAPIClient({apiKey: OPENAI_API_KEY, model: "gpt-realtime", onWebSocketClose: (event) => {}});

You can choose the model for the realtime client in the optional model parameter. There are several model options available, you can find all the available models in the OpenAI documentation. This guide uses "gpt-realtime" for the realtime examples, and "whisper-1" for the transcription example. The mini version is much cheaper than the full version. Refer to the OpenAI pricing article for more information. The cost of the Voximplant's client does not depend on the chosen model.

The third parameter is a callback function that is called if the WebSocket connection to the OpenAI Realtime API endpoint is closed for some reason.

The basic version of a VoxEngine scenario that connects an incoming call to an OpenAI agent looks as follows:

An outgoing call can be connected to an OpenAI agent in the same way, just the call itself is created in the scenario via VoxEngine.callPSTN, VoxEngine.callSIP, or VoxEngine.callUser function.

In case audio generated by the OpenAI model does not work for your use case (for example, you do not like any of the available voices), there is an easy way to disable audio generation on the OpenAI side and use one of many Voximplant TTS integrations. We have created a code example that shows how to use Cartesia for speech synthesis together with the realtime API Client that generates text responses.

The great benefit of Voximplant’s serverless architecture is that you can do function calling right in the VoxEngine scenario and return parameters back to the model right there too. That simplifies and speeds up the development process, what most developers would appreciate.

In addition to the realtime mode described above, OpenAI offers an audio transcription mode, when the audio is being sent to OpenAI for a transcription without any realtime dialogue between a person and AI.

Here is the complete example on how to use the transcription mode:

For transcription models, OpenAI states that audio.input.turn_detection is an optional voice activity detection feature for supported models.

For gpt-realtime-whisper, omit this field or set it to null, then commit audio manually. The lack of auto-commit makes this model harder to use than other real-time models that handle turn detection on the server.

Use Silero VAD and Pipecat turn detection: Silero detects when speech ends and triggers turndetector.predict(), and Pipecat confirms the end of a user turn. On endofturn, call inputaudiobuffercommit() (and responsecreate() when you need a transcription response).