SIGN UP

How to use Dialogflow Connector

How to use Dialogflow Connector

Intro

We have recently introduced connector for Dialogflow that lets connect a call running via Voximplant with a Dialogflow agent, which will do speech recognition, natural language processing (NLP) and speech synthesis according to its logic. Audio streaming from Voximplant to and from Dialogflow happens in real-time, query results arrive to VoxEngine as soon as they are returned by an agent.

Dialogflow Connector Scheme

It's required that Dialogflow agent should be setup to support API V2 (since API V1 doesn't support streaming) and Beta features should be enabled (Settings -> Enable beta features and APIs) , since speech synthesis is in the beta features list at the moment.

Setting up Dialogflow

Re-check that your Dialogflow agent uses API V2. Now create and download the service account JSON file associated with the agent from GCP console (please read the Setting up authentication article for details). "Dialogflow API Client" role should be chosen while creating the service account. The JSON file is required for Voximplant to be authorized before sending audio data to the agent.

Click on the Speech tab in the agent settings to setup Speech synthesis options.

Speech Synthesis Settings

Enable Automatic Text to Speech by clicking on the toggle button, choose MP3 or OGG in the Output Audio Encoding dropdown (IMPORTANT: only MP3 and OGG are currently supported) and one of the available voices - we highly recommend to use WaveNet-powered voices, since they sound much better than standard ones. Save settings by clicking the Save button in the top right corner.

Setting up Voximplant

 In Voximplant control panel click on the Settings drop-down menu and choose Dialogflow Connector.

Control Panel - Dialogflow Connector

 

Click on the Add Service Account link in the left menu, then click the Choose file button and select the JSON file downloaded from GCP console and click the Add button. After the file is uploaded you will see the following screen

Dialogflow - Create Test App

Click on the "Create test app" button to deploy an application and a test phone number that will be connected to your Dialogflow agent. After everything is deployed you will see the following screen

Dialogflow - Test

Follow instructions to do a test call and be connected to your Dialogflow agent.

IMPORTANT: If something went wrong it could be required to delete the test application in the Applications section and the test scenario in the Scenarios section of the control panel before you try to create the test application again. Their names start with DF_ (for scenarios) or df- (for applications). Please note that default test application can work with an agent that supports English language, if your Dialogflow agent uses another language please change the test Voxengine scenario accordingly. 

Setting up Voximplant - Connecting real phone number

After you tried the integration using the test phone number you most likely would like to connect a real phone number to the agent. Let's go through the process. Click on the "Phone numbers" menu option in the main menu and choose "Buy phone number", then choose where you would like to get your number (country, state/region, city) and phone number type (landline, tollfree, etc.), click the Buy button. After the number has been bought go to "My phone numbers" section in the left menu and bind the number to the test application that we have previously created (for testing purposes)

Dialogflow - Bind real numberIf you've chosen the number in the country where no special verification is required (for example, in The United States) then you can make a call to the number and communicate with your Dialogflow agent now, otherwise there will be a requirement to upload verification documents required by authorities in the country. It will take some time until the number becomes active if all the submitted verification documents are correct.

Setting up Voximplant - Advanced

Test application creation process includes the following actions: Voximplant application is created, Voxengine scenario is created, then test phone number is bought and the scenario connected to the application by creating the application Rule that forwards all calls from the test number to the scenario. In the end service account JSON file is binded to the application. The Voxengine scenario that connects inbound calls with a Dialogflow agent looks as follows:

require(Modules.AI)

var dialogflow, call

// Inbound call processing
VoxEngine.addEventListener(AppEvents.CallAlerting, (e) => {
	call = e.call
	call.addEventListener(CallEvents.Connected, onCallConnected)
	call.addEventListener(CallEvents.Disconnected, VoxEngine.terminate)
	call.answer()
})

function onCallConnected(e) {
  // Create Dialogflow object
	dialogflow = AI.createDialogflow({
	  lang: DialogflowLanguage.ENGLISH_US
	})
	dialogflow.addEventListener(AI.Events.DialogflowResponse, onDialogflowResponse)
    // Sending WELCOME event to let the agent says a welcome message
    dialogflow.sendQuery({event : {name: "WELCOME", language_code:"en"}})
    // Playback marker used for better user experience
    dialogflow.addMarker(-300)
    // Start sending media from Dialogflow to the call
    dialogflow.sendMediaTo(call)
    dialogflow.addEventListener(AI.Events.DialogflowPlaybackFinished, (e) => {
      // Dialogflow TTS playback finished
    })
    dialogflow.addEventListener(AI.Events.DialogflowPlaybackStarted, (e) => {
      // Dialogflow TTS playback started
    })
    dialogflow.addEventListener(AI.Events.DialogflowPlaybackMarkerReached, (e) => {
      // Playback marker reached - start sending audio from the call to Dialogflow
      call.sendMediaTo(dialogflow)
    })
}

// Handle Dialogflow responses
function onDialogflowResponse(e) {
  // If DialogflowResponse with queryResult received - the call stops sending media to Dialogflow
  // in case of response with queryResult but without responseId we can continue sending media to dialogflow
  if (e.response.queryResult !== undefined && e.response.responseId === undefined) {
    call.sendMediaTo(dialogflow)
  } else if (e.response.queryResult !== undefined && e.response.responseId !== undefined) {
  	// Do whatever required with e.response.queryResult or e.response.webhookStatus

    // Telephony messages arrive in fulfillmentMessages array
    if (e.response.queryResult.fulfillmentMessages != undefined) {
    	e.response.queryResult.fulfillmentMessages.forEach((msg) => {
      		if (msg.platform !== undefined && msg.platform === "TELEPHONY") processTelephonyMessage(msg)
    	})
  	}
  }
}

// Process telephony messages from Dialogflow
function processTelephonyMessage(msg) {
  // Transfer call to msg.telephonyTransferCall.phoneNumber
  if (msg.telephonyTransferCall !== undefined) {
  	/**
    * Example:
    * dialogflow.stop()
    * let newcall = VoxEngine.callPSTN(msg.telephonyTransferCall.phoneNumber, "put verified CALLER_ID here")
    * VoxEngine.easyProcess(call, newcall)
    */
  }
  // Synthesize speech from msg.telephonySynthesizeSpeech.text
  if (msg.telephonySynthesizeSpeech !== undefined) {
    // See the list of available TTS languages at https://voximplant.com/docs/references/voxengine/language
    // Example: call.say(msg.telephonySynthesizeSpeech.text, Premium.US_ENGLISH_FEMALE)
  }
  // Play audio file located at msg.telephonyPlayAudio.audioUri
  if (msg.telephonyPlayAudio !== undefined) {
    // audioUri contains Google Storage URI (gs://), we need to transform it to URL (https://)
    let url = msg.telephonyPlayAudio.audioUri.replace("gs://", "https://storage.cloud.google.com/")
    // Example: call.startPlayback(url)
  }
}

This scenario sends media from an inbound call to the Dialogflow agent, query results are returned in DialogflowQueryResult events. Since we have enabled Speech Synthesis in the Dialogflow agent settings there will be DialogflowPlaybackStarted event after each DialogflowQueryResult event contains fullfilmentText. It indicates that audio stream with generated speech is being played. DialogflowQueryResult represents Dialogflow's QueryResult object

Demo

You can see the connector in action at https://demos05.voximplant.com/dialogflow-connector. This demo is an example of automated pizza order via a phone call. You can also download the Dialogflow agent from this demo using the following link PizzaOrderDelivery.zip

Tags:ivrdialogflownluainlpml
B6A24216-9891-45D1-9D1D-E7359CEB8282 Created with sketchtool.

Comments(0)

Add your comment

Please complete this field.

Recommended

Get your free developer account or talk with our sales team to learn more about Voximplant solutions
SIGN UP