SIGN UP

How to use Dialogflow Connector

How to use Dialogflow Connector

Intro

We have recently introduced connector for Dialogflow that lets connect a call running via Voximplant with a Dialogflow agent, which will do speech recognition, natural language processing (NLP) and speech synthesis according to its logic. Audio streaming from Voximplant to and from Dialogflow happens in real-time, query results arrive to VoxEngine as soon as they are returned by an agent.

Dialogflow Connector Scheme

It's required that Dialogflow agent should be setup to support API V2 (since API V1 doesn't support streaming) and Beta features should be enabled (Settings -> Enable beta features and APIs) , since speech synthesis is in the beta features list at the moment.

Setting up Dialogflow

Re-check that your Dialogflow agent uses API V2. Now create and download the service account JSON file associated with the agent from GCP console (please read the Setting up authentication article for details). "Dialogflow API Client" role should be chosen while creating the service account. The JSON file is required for Voximplant to be authorized before sending audio data to the agent.

Click on the Speech tab in the agent settings to setup Speech synthesis options.

Speech Synthesis Settings

Enable Automatic Text to Speech by clicking on the toggle button, choose MP3 or OGG in the Output Audio Encoding dropdown (IMPORTANT: only MP3 and OGG are currently supported) and one of the available voices - we highly recommend to use WaveNet-powered voices, since they sound much better than standard ones. Save settings by clicking the Save button in the top right corner.

Setting up Voximplant

 In Voximplant control panel go to your application and switch to the Dialogflow connector tab. Here, click Add in the center of the screen or Add Dialogflow agent in the upper right corner.

The Add Dialogflow agent dialog will appear, click the Choose a file button or drag'n'drop the JSON file downloaded from GCP console and click the Add button. After the file is uploaded you will see a newly added agent in the list:

You can also use visit Marketplace in the main menu to deploy an application and a test phone number that will be connected to your Dialogflow agent – to do so, click Install in the appropriate tile. Then follow the instructions to do a test call and be connected to the Dialogflow agent.

IMPORTANT: If something went wrong it could be required to delete the test application in the Applications section and the test scenario in the Scenarios section of the control panel before you try to create the test application again. Their names start with DF_ (for scenarios) or df- (for applications). Please note that default test application can work with an agent that supports English language, if your Dialogflow agent uses another language please change the test Voxengine scenario accordingly. 

These first steps are the same for inbound and outbound calls, now comes the part explaining how to connect your agent to outbound calls

Setting up Voximplant – Buying real phone number

To make calls to phone network from Voximplant real phone number is required, since it will be used as a caller ID. On the Voximplant control panel, select Number on the left menu and then select Buy new phone number in the upper left corner.

The phone number interface will be displayed. Select Test numbers, select one or more virtual numbers and click Buy selected.

After you’ve bought a number, you'll immediately see this number on the list.

To associate a phone number with an application, go to Applicationsclick on the test application that we have previously created and switch to the Numbers tab. Here, click on the Available tab, select a number and click Attach:

If you've chosen the number in the country where no special verification is required (for example, in The United States) then you will be able to make calls now, otherwise there will be a requirement to upload verification documents required by authorities in the country. It will take some time until the number becomes active if all the submitted verification documents are correct.

Setting up Voximplant - Advanced

Test application creation process includes the following actions: Voximplant application is created, Voxengine scenario is created, then test phone number is bought and the scenario connected to the application by creating the application Rule that forwards all calls from the test number to the scenario. In the end service account JSON file is binded to the application. The Voxengine scenario that connects inbound calls with a Dialogflow agent looks as follows:

require(Modules.AI)

var dialogflow, call, hangup

// Inbound call processing
VoxEngine.addEventListener(AppEvents.CallAlerting, (e) => {
	call = e.call
	call.addEventListener(CallEvents.Connected, onCallConnected)
	call.addEventListener(CallEvents.Disconnected, VoxEngine.terminate)
	call.answer()
})

function onCallConnected(e) {
  // Create Dialogflow object
	dialogflow = AI.createDialogflow({
	  lang: DialogflowLanguage.ENGLISH_US
	})
	dialogflow.addEventListener(AI.Events.DialogflowResponse, onDialogflowResponse)
    // Sending WELCOME event to let the agent says a welcome message
    dialogflow.sendQuery({event : {name: "WELCOME", language_code:"en"}})
    // Playback marker used for better user experience
    dialogflow.addMarker(-300)
    // Start sending media from Dialogflow to the call
    dialogflow.sendMediaTo(call)
    dialogflow.addEventListener(AI.Events.DialogflowPlaybackFinished, (e) => {
      // Dialogflow TTS playback finished. Hangup the call if hangup flag was set to true
      if (hangup) call.hangup()
    })
    dialogflow.addEventListener(AI.Events.DialogflowPlaybackStarted, (e) => {
      // Dialogflow TTS playback started
    })
    dialogflow.addEventListener(AI.Events.DialogflowPlaybackMarkerReached, (e) => {
      // Playback marker reached - start sending audio from the call to Dialogflow
      call.sendMediaTo(dialogflow)
    })
}

// Handle Dialogflow responses
function onDialogflowResponse(e) {
  // If DialogflowResponse with queryResult received - the call stops sending media to Dialogflow
  // in case of response with queryResult but without responseId we can continue sending media to dialogflow
  if (e.response.queryResult !== undefined && e.response.responseId === undefined) {
    call.sendMediaTo(dialogflow)
  } else if (e.response.queryResult !== undefined && e.response.responseId !== undefined) {
  	// Do whatever required with e.response.queryResult or e.response.webhookStatus
        // If we need to hangup because end of conversation has been reached
        if (e.response.queryResult.diagnosticInfo !== undefined && 
           e.response.queryResult.diagnosticInfo.end_conversation == true) {
           hangup = true
        }

    // Telephony messages arrive in fulfillmentMessages array
    if (e.response.queryResult.fulfillmentMessages != undefined) {
    	e.response.queryResult.fulfillmentMessages.forEach((msg) => {
      		if (msg.platform !== undefined && msg.platform === "TELEPHONY") processTelephonyMessage(msg)
    	})
  	}
  }
}

// Process telephony messages from Dialogflow
function processTelephonyMessage(msg) {
  // Transfer call to msg.telephonyTransferCall.phoneNumber
  if (msg.telephonyTransferCall !== undefined) {
  	/**
    * Example:
    * dialogflow.stop()
    * let newcall = VoxEngine.callPSTN(msg.telephonyTransferCall.phoneNumber, "put verified CALLER_ID here")
    * VoxEngine.easyProcess(call, newcall)
    */
  }
  // Synthesize speech from msg.telephonySynthesizeSpeech.text
  if (msg.telephonySynthesizeSpeech !== undefined) {
    // See the list of available TTS languages at https://voximplant.com/docs/references/voxengine/language
    // Example: call.say(msg.telephonySynthesizeSpeech.text, Premium.US_ENGLISH_FEMALE)
  }
  // Play audio file located at msg.telephonyPlayAudio.audioUri
  if (msg.telephonyPlayAudio !== undefined) {
    // audioUri contains Google Storage URI (gs://), we need to transform it to URL (https://)
    let url = msg.telephonyPlayAudio.audioUri.replace("gs://", "https://storage.cloud.google.com/")
    // Example: call.startPlayback(url)
  }
}

This scenario sends media from an inbound call to the Dialogflow agent, query results are returned in the DialogflowResponse event. Since we have enabled Speech Synthesis in the Dialogflow agent settings there will be DialogflowPlaybackStarted event after each DialogflowResponse event containing queryResult with fullfilmentText. It indicates that the audio stream with generated speech is being played. DialogflowResponse represents Dialogflow's QueryResult object

Demo

You can see the connector in action at https://demos05.voximplant.com/dialogflow-connector. This demo is an example of automated pizza order via a phone call. You can also download the Dialogflow agent from this demo using the following link PizzaOrderDelivery.zip

Tags:ivrdialogflownluainlpml
B6A24216-9891-45D1-9D1D-E7359CEB8282 Created with sketchtool.

Comments(1)

Reza Rahmati
2019-06-19 15:36:57
Reply
Is it possible to detect the Phone number in the "onCallConnected(e)" function? (in case multiple phone number is assigned)
admin
2019-06-19 16:28:08
Reply

Reza Rahmati, you can get phone number from the "e.call.number" property.  You can find more about this event in the documentation.

Add your comment

Please complete this field.

Recommended

Sign up for a free Voximplant developer account or talk to our experts
SIGN UP