Are your auto-refreshing widgets causing instance slowdowns?

[jesseadams]

Over the past few months I have started to see a lot of performance issues being caused by Service Portal widgets using an auto-refresh. These widgets can generate a very large number of requests to a REST API called sp/rectangle, and a very large amount of these requests can cause serious performance problems. In case you haven't seen these requests before, sp/rectangle is a REST API used by Service Portal any time you need to re-run the server script for a widget after the page has loaded (usually calls to server.update()). The issues can come in a few different forms, but usually the effects are seen in an instance-wide slowdown. In many cases we have seen these requests cause high semaphore saturation on the instance, which leads to an outage. So let's take a close look at what creates these requests and how they can become an issue, as well as how to determine if they could be causing performance problems in your instance.

How to diagnose performance issues caused by sp/rectangle requests

In order to diagnose this issue you need to understand the symptoms. The effects of this issue will not be seen only in the portal. The flood of a large amount of transactions against this REST API will impact performance across the instance. These typically look like most other performance issues but what you generally see is slower page loads and/or 429 errors in the network tab of your browser's developer tools.

These symptoms don't always indicate an issue related to a portal, so before you jump to debugging your widgets, consider whether there has recently been any development done in the portal or if the number of users using the portal might have increased recently as either of those might bring this type of issue to the surface. If you see these symptoms and suspect that they may be related to your portal, go to the syslog_transaction table and use this query to check and see if there are a high number of sp/rectangle requests coming in:

sys_created_onONToday@javascript:gs.daysAgoStart(0)@javascript:gs.daysAgoEnd(0)^urlSTARTSWITH/api/now/sp/rectangle

Now, how many of these will be considered too many will depend on the instance, but if you're seeing more than a few hundred per second that's probably an indication that you need to take a closer look at your portal.

The URL for these requests contains one very important piece of information. That last part of the URL contains the sys_id for an sp_instance record. sp_instance is the table that maps a widget to a column on your portal page, so if you go to sp_instance and look up the sys_id from one of these requests, it should lead you to the widget that the request came from. It's worth noting that since widgets can be reused there might be more than one sp_instance record that points back to the same widget being used on a different page.

Why does this happen?

Every time a Service Portal widget makes a call to server.update(), spUtil.update(), server.refresh(), or spUtil.refresh(), the application will make a REST API call to an endpoint like api/now/sp/rectangle/. That re-runs the widget's server script and sends a fresh data object back to the client. In general, that's perfectly okay and this is how Service Portal is designed to work, but these calls can become problematic if you're sending a very large amount of them.

The problem with auto-refreshing widgets

Like I mentioned earlier, in most cases this issue is caused by widgets using some kind of auto-refresh. Because the application will send a REST call each time server.update() is called, auto-refreshing widgets can cause a very large amount of these requests to be sent. If you have a client controller in a widget that is calling server.update() every 3 seconds, then you'll have one request sent every 3 seconds. But if you have 3000 users using your portal, that will become 3000 requests every 3 seconds, or about 1000 per second. That adds a lot more overhead to your instance very quickly, but this can be even further exacerbated if you're using the standard javascript function window.setInterval to handle your auto-refresh.

The reason this is a problem is that setInterval simply creates a timer object in your browser's runtime. This object is not a part of the $scope of your widget — or even the $scope of the angular application. It is managed by your browser after its creation. Since Service Portal is a single-page application, we know that anything stored in the page's $scope (or the rootscope) will persist as we navigate around our portal. The same is true for these timers stored in the browser's runtime. That means that if you have a widget using setInterval and you navigate to a different page not using that widget, the timer will continue to run and will continue to send those requests. To make matters worse, if you go back to the page using your auto-refresh widget you may see that a second timer gets created. Now you're sending two requests every 3 seconds. This can quickly spin out of control in a live instance with several hundred users on your portal. To further complicate matters, since this timer object lives in your browser runtime it is possible that a user will have the portal page open on their computer and put the display to sleep when they go home for the day, but the browser will continue to send these requests all night until they return in the morning and navigate away from the portal.

Best practice if you have to auto-refresh widgets

The only perfect solution from a performance perspective is to not auto-refresh your widgets. But, if you absolutely can't live without auto-refreshing your widget, I recommend using the angularjs service $interval,which is a wrapper for window.setInterval. The advantage to this is that you can use $interval.cancel to destroy the timer once the widget is destroyed. To do that, you'd simply listen for the $destroy event in your widget controller and cancel your timer when the $destroy event is fired. That would look something like this:

$scope.$on('$destroy', function() {
   $interval.cancel(promise);
     });

Going this route can help but still needs to be used very sparingly. Using $interval will allow you to cancel the timer and prevent the issue I described earlier where timers persist across pages but you'll still have every user on your portal who is viewing a widget with auto-refresh sending one of these requests as often as the widget should refresh so it's not a perfect solution, but it will help to reduce the number of requests - which in some cases may be enough to get you out of the woods.