Virtual Agent Topic Discovery

[Nabil Asbi]

**Note: For new customers on the Quebec release, the com.glide.cs.intent_confidence_delta property mentioned below has been defaulted to '0', so it is not utilized. This is the recommended setting. Values previously set are not impacted by upgrading to Quebec. The ServiceNow NLU model will continue using the confidence threshold itself to determine intent or provide the user with a list of intents. **

In this article, I'll explain the ways you can customize the topic discovery logic with Virtual Agent. Topic discovery is the initial experience when a Virtual Agent conversation is invoked using the user's utterance, and subsequently connects the user to the correct topic flow.

There are 3 distinct scenarios that can occur with topic discovery:

1. Virtual Agent understands the user's intent, and launches the correct topic immediately.
2. Virtual Agent understands the user's intent, but would like the user to confirm which intent is preferable, as multiple similar intents may be presented.
3. Virtual Agent does not understand the user's intent, and the conversation defaults to the Fallback topic. 

As a Virtual Agent admin, you are the user experience architect. This gives you the ability to control any of these 3 outcomes. 

How do you create a highly desirable Virtual Agent user experience?

Let's assume we have T = 6 topics that are above the model threshold (default model threshold is 60% ) and 𝚫 = 0.061 (the difference between 1st topic confidence level and the 2nd topic confidence level):

T = Number of topics above the model's threshold
𝚫 = |1st topic confidence level – 2nd topic confidence level|

Using the property com.glide.cs.intent_confidence_delta (0.07 is the default value), the Virtual Agent will launch the topic immediately, or show the end-user the list of topics that might closely match.

In our example, 0.07 > 𝚫 (0.06), which means the Virtual Agent is not sure if the intent with the highest confidence level is the right one since it is very close to the second intent. In this case, the Virtual Agent will give the user the list of topics to select from. If you want the Virtual Agent to launch the first topic directly without presenting a larger list of topics to select from, then you need to lower the property value of the confidence delta system property.

In the diagram below, the Virtual Agent is confident that the first topic is what the user needs so that it will launch the Printer Issues topic immediately. The adjusted value on the system property com.glide.cs.intent_confidence_delta=0.04, so in this case 𝚫 (0.06) > 0.04.

Lastly, let's explore the scenario in which the user's intent is not recognized. In this case, the number of topics is 0, and the Virtual Agent will launch the Fallback topic that asks the user to rephrase their request. In the event you want to edit the Fallback topic to display a personalized message, you can navigate to the General Setting under Virtual Agent, and in the Setup topic tab, you will find the Fallback topic to edit.

More about Virtual Agent:

3 Best Practices for Implementing Virtual Agent

Virtual Agent terminology and architecture

[Lener Pacania1]

Very helpful, thank you Nabil.

[t_21]

i have a question. My case is that I got 3 intents (which are 0.9, 0.9, 0.7) passing the confidence threshold 0.64. I didn't modify the default value of the delta but the testing result was that both 3 topics were pop out to ask user for further selection. Anything solution I can take?

Also, may I know where can I change the delta value?

[KKrabbenhoft]

We have many topics and utterances. Most are discovered but for those that are set as zero we want to have the choice to search the knowledge base in SN. This is a work in progress. We are trying to find the best way to implement the fallback topic to suggest a search of the knowledge base. 

[Chris D]

In Quebec, the ootb AI Search Fallback Topic and Search Fallback Topic both do a search using the user's utterance immediately upon no matching results - that'll be your best route. If you're using AI Search, use the former, otherwise use the latter which still uses Contextual Search.

[StefanoZ]

@Nabil Asbi Is it possible to get this article refreshed with newer info from Utah release?

[ayman_h]

Great article! The Intent Confidence setting has now moved over to the 'Open NLU Driver' table (open_nlu_driver.list)

For more information, you can refer to this SN article

https://support.servicenow.com/kb?id=kb_article_view&sysparm_article=KB1414012