SIGN UP

Connecting Dialogflow agent to outbound calls

Connecting Dialogflow agent to outbound calls

Intro

Please note that this tutorial is about connecting Dialogflow agent to outbound calls, If you need to connect inbound calls to your Dialogflow agent please follow the "How To Use Dialogflow Connector" tutorial. Recently Google has introduced Google Duplex, it's not based on Dialogflow, but you still can build something similar using Dialogflow Connector and Voximplant.

Outbound call connected to Dialogflow

Dialogflow Connector lets connect a call running via Voximplant to a Dialogflow agent, which will do speech recognition, natural language processing (NLP) and speech synthesis according to its logic. Audio streaming from Voximplant to and from Dialogflow happens in real-time, query results arrive to VoxEngine as soon as they are returned by an agent. 

It's required that Dialogflow agent should be setup to support API V2 (since API V1 doesn't support streaming) and Beta features should be enabled (Settings -> Enable beta features and APIs) , since speech synthesis is in the beta features list at the moment.

Setting up Dialogflow

Re-check that your Dialogflow agent uses API V2. Now create and download the service account JSON file associated with the agent from GCP console (please read the Setting up authentication article for details). "Dialogflow API Client" role should be chosen while creating the service account. The JSON file is required for Voximplant to be authorized before sending audio data to the agent.

Click on the Speech tab in the agent settings to setup Speech synthesis options.

Speech Synthesis Settings

Enable Automatic Text to Speech by clicking on the toggle button, choose MP3 or OGG in the Output Audio Encoding dropdown (IMPORTANT: only MP3 and OGG are currently supported) and one of the available voices - we highly recommend to use WaveNet-powered voices, since they sound much better than standard ones. Save settings by clicking the Save button in the top right corner.

Setting up Voximplant

 In Voximplant control panel click on the Settings drop-down menu and choose Dialogflow Connector.

Control Panel - Dialogflow Connector

Click on the Add Service Account link in the left menu, then click the Choose file button and select the JSON file downloaded from GCP console and click the Add button. After the file is uploaded you will see the following screen

Dialogflow - Create Test App

Click on the "Create test app" button to deploy an application and a test phone number that will be connected to your Dialogflow agent. After everything is deployed you will see the following screen

Dialogflow - Test

Follow the instructions to do a test call and be connected to your Dialogflow agent.

IMPORTANT: If something went wrong it could be required to delete the test application in the Applications section and the test scenario in the Scenarios section of the control panel before you try to create the test application again. Their names start with DF_ (for scenarios) or df- (for applications). Please note that default test application can work with an agent that supports English language, if your Dialogflow agent uses another language please change the test Voxengine scenario accordingly. 

These first steps are the same for inbound and outbound calls, now comes the part explaining how to connect your agent to outbound calls

Setting up Voximplant - Buying real phone number

To make calls to phone network from Voximplant real phone number is required, since it will be used as a caller ID. Click on the "Phone numbers" menu option in the main menu and choose "Buy phone number", then choose where you would like to get your number (country, state/region, city) and phone number type (landline, tollfree, etc.), click the Buy button. After the number has been bought go to "My phone numbers" section in the left menu and bind the number to the test application that we have previously created (for testing purposes)

Dialogflow - Real phone number

If you've chosen the number in the country where no special verification is required (for example, in The United States) then you will be able to make calls now, otherwise there will be a requirement to upload verification documents required by authorities in the country. It will take some time until the number becomes active if all the submitted verification documents are correct.

Setting up Voximplant - Modifying scenario for outbound calls

The test application creation process includes the following actions: Voximplant application is created, Voxengine scenario is created, then test phone number is bought and the scenario connected to the application by creating the application Rule that forwards all calls from the test number to the scenario. In the end service account JSON file is binded to the application. By default the Voxengine scenario connects inbound calls to a Dialogflow agent, but we need to change it to work with outbound calls:

require(Modules.AI)

var dialogflow, call

// Inbound call processing - remove processing by commenting the code
/**
VoxEngine.addEventListener(AppEvents.CallAlerting, (e) => {
	call = e.call
	call.addEventListener(CallEvents.Connected, onCallConnected)
	call.addEventListener(CallEvents.Disconnected, VoxEngine.terminate)
	call.answer()
})
**/

// Create outbound call as soon as StartScenarios HTTP API arrives
VoxEngine.addEventListener(AppEvents.Started, (e) => {
     let number = VoxEngine.customData() // we assume that a callee's number arrives as customData in e.164 format
     call = VoxEngine.callPSTN(number, CALLER_ID) // replace CALLER_ID with the number we bought in previous step (the real one, test number won't work)
     call.addEventListener(CallEvents.Connected, onCallConnected)
     call.addEventListener(CallEvents.Disconnected, VoxEngine.terminate)
     call.addEventListener(CallEvents.Failed, VoxEngine.terminate)
})

function onCallConnected(e) {
  // Create Dialogflow object
	dialogflow = AI.createDialogflow({
	  lang: DialogflowLanguage.ENGLISH_US
	})
	dialogflow.addEventListener(AI.Events.DialogflowResponse, onDialogflowResponse)
    // Sending WELCOME event to let the agent says a welcome message
    dialogflow.sendQuery({event : {name: "WELCOME", language_code:"en"}})
    // Playback marker used for better user experience
    dialogflow.addMarker(-300)
    // Start sending media from Dialogflow to the call
    dialogflow.sendMediaTo(call)
    dialogflow.addEventListener(AI.Events.DialogflowPlaybackFinished, (e) => {
      // Dialogflow TTS playback finished
    })
    dialogflow.addEventListener(AI.Events.DialogflowPlaybackStarted, (e) => {
      // Dialogflow TTS playback started
    })
    dialogflow.addEventListener(AI.Events.DialogflowPlaybackMarkerReached, (e) => {
      // Playback marker reached - start sending audio from the call to Dialogflow
      call.sendMediaTo(dialogflow)
    })
}

// Handle Dialogflow responses
function onDialogflowResponse(e) {
  // If DialogflowResponse with queryResult received - the call stops sending media to Dialogflow
  // in case of response with queryResult but without responseId we can continue sending media to dialogflow
  if (e.response.queryResult !== undefined && e.response.responseId === undefined) {
    call.sendMediaTo(dialogflow)
  } else if (e.response.queryResult !== undefined && e.response.responseId !== undefined) {
  	// Do whatever required with e.response.queryResult or e.response.webhookStatus

    // Telephony messages arrive in fulfillmentMessages array
    if (e.response.queryResult.fulfillmentMessages != undefined) {
    	e.response.queryResult.fulfillmentMessages.forEach((msg) => {
      		if (msg.platform !== undefined && msg.platform === "TELEPHONY") processTelephonyMessage(msg)
    	})
  	}
  }
}

// Process telephony messages from Dialogflow
function processTelephonyMessage(msg) {
  // Transfer call to msg.telephonyTransferCall.phoneNumber
  if (msg.telephonyTransferCall !== undefined) {
  	/**
    * Example:
    * dialogflow.stop()
    * let newcall = VoxEngine.callPSTN(msg.telephonyTransferCall.phoneNumber, "put verified CALLER_ID here")
    * VoxEngine.easyProcess(call, newcall)
    */
  }
  // Synthesize speech from msg.telephonySynthesizeSpeech.text
  if (msg.telephonySynthesizeSpeech !== undefined) {
    // See the list of available TTS languages at https://voximplant.com/docs/references/voxengine/language
    // Example: call.say(msg.telephonySynthesizeSpeech.text, Premium.US_ENGLISH_FEMALE)
  }
  // Play audio file located at msg.telephonyPlayAudio.audioUri
  if (msg.telephonyPlayAudio !== undefined) {
    // audioUri contains Google Storage URI (gs://), we need to transform it to URL (https://)
    let url = msg.telephonyPlayAudio.audioUri.replace("gs://", "https://storage.cloud.google.com/")
    // Example: call.startPlayback(url)
  }
}

After you fixed the scenario click the Save button, and go to Applications section of the control panel. Click the Edit button (near the app that starts with df-app-) and open Rules tab, point mouse over the only available rule and click the "Start Scenarion" button (yellow one)

Application Rule

In the dialog specify a callee's phone number in Script Custom Data and click the Run button to initate the call

Script Custom Data

If everything was done correctly then the call will arrive to the specified phone number and you will hear greeting from your agent.

Launching the scenario using API

If you want to initiate a call programmatically you need to use the StartScenarios method of HTTP API. You can also use Voximplan't CallLists if you need to initiate many calls.

Tags:dialogflowaimachine learningvoice botduplex
B6A24216-9891-45D1-9D1D-E7359CEB8282 Created with sketchtool.

Comments(0)

Add your comment

Please complete this field.

Recommended

Get your free developer account or talk with our sales team to learn more about Voximplant solutions
SIGN UP