NLU Model Tuning

First, take the time to read the intents and utterances within your NLU Model carefully. Doing so will help you understand what types of utterances your NLU model will be able to predict accurately. 

Next, you’ll want to collect utterances from employees in your organization for each intent. These utterances will give you a sense of the types of utterances that might be missing from your NLU Model. They will also serve as the set of utterances for testing your NLU model. Below are example utterances collected for the Get Password Reset intent: 

1. Can someone reset my password? 
2. I need help logging in 
3. I’m having issues 

In the example utterances collected, “I’m having issues” is too generic and shouldn’t be used for testing or tuning. Read through each collected utterance and remove any bad utterances. Once you’ve collected and reviewed your utterances, it’s time to move onto Step 2: Test. 

Note: Use out-of-box NLU models such as ITSM where possible, which contains tried and true utterances. Password Reset is an example of an existing out-of-box Intent you can use or modify.  

Use the set of utterances from Step 1 to test the accuracy of your NLU Model. Document the returned intents (above the NLU Models threshold) and whether or not the prediction was correct for every utterance you test. Accurate predictions occur when one of the returned intents matches the expected intent and would be shown to the end-user via Virtual Agent. Incorrect predictions are defined as either no intents or incorrect ones returned.

Included NLU tools such as Batch Testing can help you automate this process. 

Here’s an example of what your test document might look like:

Most missed predictions are caused by a lack of relevant samples in the NLU Model, generic utterances that can span multiple intents, or deficient vocabulary with the applications, words, or phrases that are specific to your company. Analyze your test results and see if you can find hot spots among the missed predictions. Once you understand your hot spots, you’re ready to start making adjustments using the following tuning techniques: 

Adding vocabulary:  Add your custom applications, lingo, and phrases to your NLU Model. The synonyms you select will help the NLU Model understand utterances better. 

Removing ambiguous or misaligned utterances: Remove utterances from intents that don’t align. If you read an utterance that can belong to more than one intent, you should consider removing it. 

Adding utterances:  Add the test utterances, or a rephrased version of the test utterance, to the intent. If you think an utterance can belong to more than one intent, you should think about where you want to add it. 

Remember to perform a sample test after every adjustment to see if you’ve tuned the NLU Model to your specifications. Once you’re done with your adjustments, go back to Step 2 and perform another round of testing to see how much NLU accuracy improved with the adjustments. A model that predicts with 80% accuracy is considered good and can be promoted to production.