Optimizing the Service Portal Typeahead search widget

[GTSPerformance]

< Previous Article		Next Article >
Demystifying table rotation, extension, and table cleaner		Performance Best Practices for Server-side Coding in ServiceNow

---

Overview

This guide is written by the ServiceNow Technical Support Performance team (All Articles). We are a global group of experts that help our customers with performance issues. If you have questions about the content of this article we will try to answer them here. However, if you have urgent questions or specific issues, please see the list of resources on our profile page: ServiceNowPerformanceGTS

Note that newly provisioned ServiceNow instances (Paris release onwards) are deployed with 'suggestions' enabled instead of 'typeahead'. This means that the 'Typeahead search' widget is no longer vulnerable to many of the issues described in this article. If, however, you are working on an instance which was initially provisioned prior to Paris and / or are using clones of the 'Typeahead search' widget which may have been created prior to Paris (and are therefore using an old configuration) please read on.

Lets be honest - the Service Portal 'Typeahead search' widget has had somewhat of a troubled life. Its extremely popular (and is probably the most deployed type of 'search' widget we see within customers Service Portal implementations) however it has had some major issues:

• Out of box it can't be modified (so its extremely common for customers to clone then customize the widget)
• Its historic configuration meant that it often provided a poor end user experience and, in some cases, could lead to widespread instance performance degradation

But times are changing - over the past few releases (Paris onwards) ServiceNow has made some relatively significant changes to both the widget and the underlying platform to address performance concerns. Weren't aware of this? We don't blame you - the improvements really haven't been accompanied by any kind of fanfare. What this means, however, is that many instances continue to use clones of the 'Typeahead search' widget with an out of date configuration and the risk of a performance hit. We'd like to change this so in this article we will cover:

• An introduction to the 'Typeahead search' widget
• A review of the widgets historic shortcomings
• A discussion of the improvements made to 'typeahead' in recent releases and why they matter
• Using 'suggestions' instead of 'typeahead' to display suggested searches (instead of predictive results)

Introducing the 'Typeahead search' widget

In a nutshell the 'Typeahead search' widget (which will just be referred to as the 'widget' from now on) allows users to trigger searches against the underlying Service Portals 'search sources'. It looks like a simple text field with a search button but has a big selling point in that its searches are 'predictive'. As a user enters their search string the widget will trigger transactions in the background with the hope of displaying immediate and useful results to the user before they have even finished typing. The user can then click on one of these results (to execute a particular search or be taken directly to the corresponding page) or, once they have finished typing, can click the search button to load a full list of results. Simple.

Historic shortcomings of the widget

The 'predictive' nature of the widget is one of its biggest selling points but also has the capacity to cause the most issues. Older versions of the widget contained the following code in the template field:

typeahead-wait-ms="250"
typeahead-min-length="2"

What this means is that:

• As soon as a user enters 2 or more characters in the search field the widget is allowed to start triggering background 'searches' (governed by typeahead-min-length)
• Following a delay of >= 250ms since the user last typed a character (and assuming the user has typed a minimum of 'typeahead-min-length' characters) a background 'search' transaction will be triggered (governed by typeahead-wait-ms)

Note that 2 characters isn't a lot of text and 250ms is a really short amount of time. If the user is a somewhat slow typist its entirely possible that the widget will trigger a search after every character they type which, if they are typing a relatively long string, means that a single interaction with the widget can trigger a whole load of background transactions. To illustrate this lets consider that a user is typing the string 'mainframe' and there is just under a 1s pause between characters - the widget will therefore trigger 8 transactions with the following queries:

• ma
• mai
• main
• mainf
• mainfr
• mainfra
• mainfram
• mainframe

In addition older ServiceNow instances (deployed prior to Paris) are configured to perform 'typeahead' searches as the user types. This means that they will trigger an /api/now/sp/search transaction invoking all 'search sources' configured against the corresponding Service Portal. Worse still consider that one or more of your Service Portal 'search sources' isn't that speedy so each /api/now/sp/search transaction takes 5 seconds to complete. Due to the concept of 'session synchronization' the platform will, by default:

• Only allow each user to physically execute one transaction at a time (i.e. transactions are serialized on the corresponding application node)
• Transactions which arrive whilst the user is already executing a transaction will be queued then executed in turn one by one

Using the above example (and assuming the instance is configured to perform 'typeahead' searches as the user types) we can work out that:

• It will take the user approximately 8 seconds to type from 'ma' to 'mainframe'
• During this 8 seconds 8 /api/now/sp/search transactions will be executed
• Each /api/now/sp/search transaction will take 5 seconds to complete meaning that it will take ~40 seconds for all 8 transaction to complete (due to them being serialized)

The user will only see something happen 'on screen' (i.e. results being displayed or another page loading if they click somewhere else in the interface) 30+ seconds after they finish typing! Its clear that this will provide a confusing / frustrating experience and is unlikely to make end users happy. To demonstrate this the following instance has been configured to perform 'typeahead' with a widget which executes against a single slow search source which logs the query passed - note the huge session wait time of the final transaction (after which results are actually displayed on screen):

In addition every one of the 8 /api/now/sp/search transactions will do *something* within the instance such as search multiple text indexes (depending on the configuration of your search sources). A 'busy' widget which is used frequently by a large set of users can easily create hundreds or thousands of /api/now/sp/search transactions per minute which, in turn, can create tens or hundreds of thousand of database queries in the same period. In the worst case scenario this can overwhelm the database underlying the instance causing severe performance issues. A frustrating user experience suddenly became an instance which may be barely operable.

One way to avoid this is to disable typeahead (by clearing the 'Enable typeahead' checkbox in your Service Portal 'search sources'). In doing this:

• The widget will still trigger just as many '/api/now/sp/search' transactions
• 'Search sources' with 'Enable typeahead' unchecked will simply short circuit any transactions triggered via typeahead (i.e. the search source will not execute - it will just return no results)

This is likely to make typeahead triggered /api/now/sp/search transactions complete much more quickly and avoid issues with transactions stacking up on the application node (and corresponding poor performance / degraded end user experience) however the loss in functionality is significant. It effectively takes away one of the core benefits of using the widget and, for this reason, is not a great solution.

Improvements to the widget in recent releases

A couple of changes have taken place from the Paris release of ServiceNow onwards. The first improvement is to reduce how aggressive the widget is in terms of triggering background searches. New versions of the widget therefore contain the following code in their template:

typeahead-wait-ms="c.data.typeaheadWaitMS"
typeahead-min-length=c.data.typeaheadMinLength"

And in their server script:

data.typeaheadWaitMS = getIntProperty('glide.service_portal.typeahead.wait_ms', '1000');
data.typeaheadMinLength = getIntProperty('glide.service_portal.typeahead.min_length', '3');

What this means is that:

• The minimum number of characters which must be typed before the widget triggers typeahead background searches and the delay between the user finishing typing and a search transaction being triggered are now controlled via system properties ('glide.service_portal.typeahead.min_length' and 'glide.service_portal.typeahead.wait_ms' respectively)
• The default value of these tunables has been relaxed - a user must now type a minimum of 3 characters and wait at least 1 second before a background search transaction is triggered

These changes alone help to significantly reduce the number of search transactions triggered via an interaction with the widget. If we go back to our earlier example of a user typing 'mainframe' (in an instance with 'typeahead' configured) there is a < 1s pause between characters - as such the widget will only trigger a single typeahead search transaction (when the user has finished typing the whole string). If search sources still take 5 seconds to execute a search this means that they will see results 5 seconds after they finish typing rather than the previous 30 plus seconds.

The move to system properties means that administrators can easily change these tunables without having to go and update every 'Typeahead search' widget which exists within an instance. Modification of the properties will, however, change the behaviour of every instance of the 'Typeahead search' widget deployed. If there is a specific instance of the widget which needs a different set of tunables (for example a minimum of 5 characters and a 2.5 second delay) the administrator would need to clone the widget, replace that instance with the clone, then update the cloned instances template accordingly (for example hard coding typeahead-wait-ms and typeahead-min-length to required values).

The second improvement is to make typeahead search /api/now/sp/search transactions cancelable. Cancelable transactions are explained in significant detail in ServiceNows documentation (see here) however, in simple terms, this means that if a user is currently executing an /api/now/sp/search transaction with the 'sysparm_cancelable=true' parameter and they trigger a new '/api/now/sp/search' transaction with 'sysparm_cancelable=true' the first transaction will be aborted to allow the newly triggered transaction to immediately execute. This completely avoids the scenario where multiple /api/now/sp/search transactions triggered via typeahead 'stack up' on application nodes due to session synchronization and prevents huge delays before a user sees something happen in the user interface - note that if we run the 'mainframe' test again with no changes other than turning on cancelable transactions the final transaction now has essentially no wait time:

For the widget to take advantage of 'cancelable transactions', however, a couple of things need to be put in place:

• The 'glide.request_manager.cancel_other_transaction' system property must be set to 'true' (note that this is the out of box default so shouldn't need to be changed)
• The widget must actually trigger /api/now/sp/search transactions with the 'sysparm_cancelable=true' parameter set

Older versions of the widget trigger typeahead search transactions via the following code in their client controller script:

return $http.post("/api/now/sp/search", payload).then(function(response) {

In newer versions, however, this has been modified to add 'sysparm_cancelable=true', i.e.:

return $http.post("/api/now/sp/search?sysparm_cancelable=true", payload).then(function(response) {

If you are working on an instance which is configured to perform 'typeahead' and you are seeing slow '/api/now/sp/search' transactions (specifically due to the number of transactions triggered / session wait as described above) implementing these changes can provide a significant boost to a users experience of using 'Typeahead search' widgets as well as a reduction in overall instance resource utilization.

The move to suggestions (instead of typeahead)

From the Paris release onwards newly deployed instances are configured to perform 'suggestions' searches instead of 'typeahead' when a user interacts with a 'Typeahead search' widget. Note that this behaviour is controlled by two system properties:

• glide.search.suggestions.enabled - enables general 'suggestion' behaviour within the instance and defaults to 'true'
• glide.service_portal.search_as_you_type_behaviour - determines whether Service Portal components will use suggestions or typeahead and defaults to 'typeahead' (instances deployed on Orlando or earlier) or 'suggestions' (instances deployed on Paris or later)

To determine whether your instance is using typeahead or suggestions simply check the value of the glide.service_portal.search_as_you_type_behaviour property, i.e.:

Using suggestions depends on the following taking place:

• Searches performed by users within the Service Portal are logged in the sys_search_event (Search Events) table. Corresponding records indicate the query performed (i.e. string searched for), corresponding language, whether results were found and so on
• Once a day the 'Build Search Suggestions' scheduled job runs - this examines newly created records in the sys_search_event table and builds corresponding records in the sys_search_suggestion (Search Suggestions) table. Each suggestion again contains data such as the query, language, search source used, and a corresponding 'score'
• The score indicates the 'quality' of the suggestion and, for example, might be influenced by how often the corresponding query was executed or the number of results the search returns

The above happens in all instances however records from the sys_search_suggestion table are only physically used if the instance is configured with 'glide.service_portal.search_as_you_type_behaviour = suggestions'. In this scenario background search transactions will be executed by 'Typeahead search' widgets exactly as described above (i.e. depending on minimum characters / delay after typing) however instead of /api/now/sp/search transactions being executed (which invoke search sources) /api/now/search/sp_suggestions transactions will be triggered. These transactions do NOT invoke search sources - instead they simply query the sys_search_suggestion table looking for corresponding records with the same query / search source. An example of the SQL query used to perform this is shown below:

2022-01-13 01:51:24 (367) Default-thread-157 CA4133091BCDC9902CE7BA63CC4BCB3F txid=df6ebfc11b01 Time: 0:00:00.002 id: empjfordrome_1[glide.9] (connpid=381926) for: SELECT sys_search_suggestion0.`name` AS `name`, SUM(sys_search_suggestion0.`score`) AS sum_of_109264530 FROM sys_search_suggestion sys_search_suggestion0  WHERE sys_search_suggestion0.`name` LIKE 'del%' AND sys_search_suggestion0.`group` = 'internal' AND sys_search_suggestion0.`language` = 'en' AND sys_search_suggestion0.`application_id` = '81b75d3147032100ba13a5554ee4902b' AND sys_search_suggestion0.`application_table` = 'sp_portal' AND sys_search_suggestion0.`active` = 1 AND sys_search_suggestion0.`source` IN ('145a70931bf481902ce7ba63cc4bcbcb' , '16ac01c31b3441902ce7ba63cc4bcb70') GROUP BY sys_search_suggestion0.`name` ORDER BY sum_of_109264530 DESC ,sys_search_suggestion0.`name` /* empjfordrome005, gs:CA4133091BCDC9902CE7BA63CC4BCB3F, tx:df6ebfc11b010d902ce7ba63cc4bcbc6 */

Due to the out of box database indexes created on the sys_search_suggestion table these queries tend to be very performant hence determining suggestions for a given search term is far more lightweight than performing a full typeahead search for a given search term.

Matching suggestions will be displayed to the user below the text box (as with typeahead search) however in this case its important to remember that these are suggestions on what to search rather than suggestions for matching documents. As a result if a user clicks a suggestion an /api/now/sp/search transaction will be executed to take them to a page of search results rather than them being taken directly to the corresponding page / record (as would happen with 'typeahead' results):

Summary

If you use 'Typeahead search' widgets within you instance and you are seeing performance issues when interacting with these widgets the fact that you are using an out of date configuration could well be to blame (and the steps in this article should definitely help). Ideally you should consider:

• Updating the configuration of 'Typeahead search' widgets such that it is optimal
• Moving your instances to use 'predictive' suggestions rather than full typeahead search results (as this, along with AI Search, is clearly the future direction of ServiceNow)

The only caveat is that if you are using typeahead today then moving to suggestions will change the experience your end users receive which might not be welcome. Even if you can't move to suggestions simply updating the 'Typeahead search' widget should provide good improvments and is really a no brainer.

---

< Previous Article		Next Article >
Demystifying table rotation, extension, and table cleaner		Performance Best Practices for Server-side Coding in ServiceNow