
- Post History
- Subscribe to RSS Feed
- Mark as New
- Mark as Read
- Bookmark
- Subscribe
- Printer Friendly Page
- Report Inappropriate Content
on 03-06-2020 01:26 PM
Step 5: Model Tuning . . .
Before we jump into model tuning, you must have a solid understanding of how you want the Virtual Agent to respond to seemingly obscure utterances. What's a seemingly obscure utterance? Here's an example: "Please add more capacity to my voicemail, I've run out of space, and my coworkers can no longer leave me voicemails." Fallback topics best handle these types of utterances.
In this article, we'll walk through how to identify good vs bad utterances, understanding model accuracy, and tuning techniques. Let's start with Step 1: Analyze.
Step 1: Analyze |
First, take the time to read the intents and utterances within your NLU Model carefully. Doing so will help you understand what types of utterances your NLU model will be able to predict accurately. Next, you’ll want to collect utterances from employees in your organization for each intent. These utterances will give you a sense of the types of utterances that might be missing from your NLU Model. They will also serve as the set of utterances for testing your NLU model. Below are example utterances collected for the Get Password Reset intent:
In the example utterances collected, “I’m having issues” is too generic and shouldn’t be used for testing or tuning. Read through each collected utterance and remove any bad utterances. Once you’ve collected and reviewed your utterances, it’s time to move onto Step 2: Test. Note: Use out-of-box NLU models such as ITSM where possible, which contains tried and true utterances. Password Reset is an example of an existing out-of-box Intent you can use or modify. |
Step 2: Test | ||||||||||||||||||||
Use the set of utterances from Step 1 to test the accuracy of your NLU Model. Document the returned intents (above the NLU Models threshold) and whether or not the prediction was correct for every utterance you test. Accurate predictions occur when one of the returned intents matches the expected intent and would be shown to the end-user via Virtual Agent. Incorrect predictions are defined as either no intents or incorrect ones returned. Included NLU tools such as Batch Testing can help you automate this process.
|
Step 3: Tune |
Most missed predictions are caused by a lack of relevant samples in the NLU Model, generic utterances that can span multiple intents, or deficient vocabulary with the applications, words, or phrases that are specific to your company. Analyze your test results and see if you can find hot spots among the missed predictions. Once you understand your hot spots, you’re ready to start making adjustments using the following tuning techniques: Adding vocabulary: Add your custom applications, lingo, and phrases to your NLU Model. The synonyms you select will help the NLU Model understand utterances better. Removing ambiguous or misaligned utterances: Remove utterances from intents that don’t align. If you read an utterance that can belong to more than one intent, you should consider removing it. Adding utterances: Add the test utterances, or a rephrased version of the test utterance, to the intent. If you think an utterance can belong to more than one intent, you should think about where you want to add it. Remember to perform a sample test after every adjustment to see if you’ve tuned the NLU Model to your specifications. Once you’re done with your adjustments, go back to Step 2 and perform another round of testing to see how much NLU accuracy improved with the adjustments. A model that predicts with 80% accuracy is considered good and can be promoted to production. |
Once you’re done with your tuning your model, go back to Step 2 and perform another round of testing to see how much NLU accuracy improved with the adjustments. Try to get your model's prediction accuracy to 80%. On average, it takes around 2-3 iterations of testing and tuning to get there.
Happy tuning!
- 2,786 Views
- Mark as Read
- Mark as New
- Bookmark
- Permalink
- Report Inappropriate Content
"Tune" section talks about Removing ambiguous or misaligned utterances.
This can be achieved using NLU Conflict review tool.
Make sure that you have the NLU Workbench - Advanced Features (com.snc.nlu.workbench.advanced) ServiceNow® Store plugin installed and activated on your instance. For more information on the required plugins, see Activate the NLU Workbench.