5 Ways to Troubleshoot your Instance Performance

[nisha_narayanak]

Performance issues can be tricky, but there are various modules in ServiceNow applications that enable administrators to diagnose and troubleshoot performance issues. Here are some basic troubleshooting steps to help you isolate the source of slow response times. There are built in performance tools within ServiceNow

which administrators can make use of while addressing performance concerns from end users. As a performance expert, I use these same tools to diagnose performance issues reported to me on HI.

Let's start with the basic information you may need to collect from end users who are impacted. Gather details about the performance from the users, such as:

• User names
• What time did the issue occur
• What transactions were slow? Specific steps to reproduce
• Is the issue consistent?
• Is it happening on all instances?

1. Get Real-Time information on your node

Real time performance metrics of the node can be found from System diagnostics & Stats page. This provides real time diagnostic information of the node you are currently logged into. The average time of response at multiple intervals are available under the stats page, this gives real time memory usage of the node as well.

Example of what your memory usage and response times from stats will look like:

2. Get Real-Time performance information for cluster nodes

For a multi cluster environment, navigate to System Diagnostics & Diagnostic page. This will provide information on the cluster nodes the instance is running on.

This page provides information on the status of each online and/or offline node. It will also tell you how much memory each node is using,   and if there have been any restarts on the node (JVM uptime). By clicking on the "Name" link for each node, it will take you the "xmlstats" page of the corresponding node which will provide very detailed information on the semaphore usage and scheduled workers running at the moment. The diagnostics page also provides information on the pending events on the system and email stats.

Standard configuration is 16 semaphores and 8 scheduled workers per node for production instances.
This helps to determine if the node is running healthy.

3. Checking the System Logs

System logs is another great place to view information about system activity and trace the transactions performed by users. There are various types of system logs including all user transactions, client transactions, and errors. These logs provide detailed information on response times, any errors triggered by transactions , slow queries and information on slow running jobs.

• Transactions (All user): User activities are logged and available under this module. You can use the information here to trace the reported slow transaction by looking at the "Response time," "Created By," "URL," "created," and "session." A time split of the response times such as sql time, client network time, browser time is also available. This will help to identify which layer   the transaction is spending the most amount of time. High SQL time can indicate long running queries.
• Client transactions: This log will have information only if client transaction logging is enabled, this is particularly useful in case of network issues to identify high network times.
• Errors provide information on the errors and warnings triggered within the platform
• System Scheduler > Slow Job Log: This module provides a transaction log filtered to show background scheduled jobs related slow transactions
• System Diagnostics & Stats & Slow Queries: Administrators can use slow query logs to gain insight into how queries are affecting platform performance.

ServiceNow provides the logging utilities where you can download/view server log files. Check under System Logs & Log File Browser. User transactions can be traced on the   server logs using the session information gathered from Transactions (All users).

4. Using the Debugger tool

This is particularly useful when you are able to replicate the slowness reported by the users. By enabling the debugger tool, it   will print detailed information on the screen upon completion of the slow transaction. While system logs provide historic information on the slowness, the debugger tool can be effectively used on-the-go as you reproduce the issue.

You can use the System diagnostics & Session debugging for debugging types such as   "Debug SQL," "Debug security," "Debug business rules" etc. For example, all ACL evaluations can be logged by enabling the "Debug security" module.

5. Troubleshooting network issues

If users from specific locations are reporting slowness, it could possibly be caused by the network issue. The user must collect some basic information at a lower level than a browser. Use the information listed below to help identify if the issue originated from user's network/ISP level/ServiceNow network.

• Traceroute
• Ping
• IP address of the impacted users

Make sure to gather this information while the issue is occurring as well as report it while submitting an incident.

These are just a few guidelines to get you started with troubleshooting performance issues. If you have other methods or have found certain methods more useful than others I would love to hear your experiences.