In ServiceNow, the SAM (Software Asset Management) License Counters job scans the instance for software installations. Software counters allow asset managers to verify compliance by reconciling the number of installations with the number of software rights purchased. The job queries the Software Installation [cmdb_sam_sw_install] table and captures any installations that have not been scanned in the he past 7 days.
In certain situations, this job can run in excess of 10+ hours depending upon the size of the install data in cmdb_sam_sw_install. This occupies a schedule worker on the node and queries the database incessantly leading to waste of CPU resources.
You could be affected by the issue if you are on later patches of Fuji, certain patches of Geneva, and Helsinki. The severity may depend on the usage of the schedule job of an instance by customers.
If you are using the Software Asset Management plugin, here are a few customer scenarios that can help determine if you are are experiencing a semaphore queue leak:
- The stats.do page of the ServiceNow instance on which the SAM License Counter job has been running for 10+ hours.
- The same can be confirmed by checked the Transactions (Background).
- Localhost logs show repetitive queries.
Symptom 1: Long running SAM License Counter job on STATS.DO
The admin should try to navigate to the /stats.do page if they are on the node where the job is running.
Consider the following example:
Current job: SAM License Counters
Job started: Wed Aug 03 02:00:01 PDT 2016
Job duration: 37:49:22.852
Total jobs: 18015
Mean duration: 0:00:12.624
This job can be running on any worker.
If the job is Running, check the most recent start time. If the job started more than 10 hours ago, the instance may be impacted. The higher the number, the more likely the job is impacting system performance.
Symptom 2: Confirming the completion time of the job from Transaction Background to be consistently high
Here is an example of the SAM License counter job running for a long time. The response time here was approximately 54 hours.
Symptom 3: Checking localhost logs for repetitive queries and large table handling
A snippet in in the localhost logs on searching for the worker thread where the job is running:
2016-08-03 15:58:10 (227) worker.6 worker.6 WARNING *** WARNING *** Large Table: Table handling an extremely large result set: 6566270
2016-08-03 15:58:10 (238) worker.6 worker.6 [0:00:00.011] Compacting large row block
2016-08-03 15:58:10 (241) worker.6 worker.6 [0:00:00.003] Expanding large row block
2016-08-03 15:58:12 (884) worker.6 worker.6 [0:00:00.059] Compacting large row block
2016-08-03 15:58:12 (890) worker.6 worker.6 [0:00:00.006] Expanding large row block
2016-08-03 15:58:14 (731) worker.6 worker.6 [0:00:00.022] Compacting large row block
2016-08-03 15:58:14 (735) worker.6 worker.6 [0:00:00.004] Expanding large row block
016-08-03 15:57:50 (282) worker.6 worker.6 SEVERE *** ERROR *** FAILED TRYING TO EXECUTE ON CONNECTION 20: /* gs:glide.scheduler.worker.6, tx:067038416fc52e008d289f5e5d3ee4ea */ SELECT cmdb_sam_sw_install0.`normalized_display_name`, cmdb_sam_sw_install0.`last_used`, cmdb_sam_sw_install0.`foreground`, cmdb_sam_sw_install0.`normalized_version`, cmdb_sam_sw_install0.`sys_updated_on`, cmdb_sam_sw_install0.`prod_id`, cmdb_sam_sw_install0.`install_location`, cmdb_sam_sw_install0.`sys_id`, cmdb_sam_sw_install0.`counted_by`, cmdb_sam_sw_install0.`normalized_publisher`, cmdb_sam_sw_install0.`discovery_source`, cmdb_sam_sw_install0.`sys_updated_by`, cmdb_sam_sw_install0.`is_reconciled`, cmdb_sam_sw_install0.`uninstall_string`, cmdb_sam_sw_install0.`sys_created_on`, cmdb_sam_sw_install0.`cached`, cmdb_sam_sw_install0.`last_scanned`, cmdb_sam_sw_install0.`sys_domain`, cmdb_sam_sw_install0.`processor_mapping`, cmdb_sam_sw_install0.`sccm_group_id`, cmdb_sam_sw_install0.`installed_on`, cmdb_sam_sw_install0.`install_date`, cmdb_sam_sw_install0.`inferred_suite`, cmdb_sam_sw_install0.`sys_created_by`, cmdb_sam_sw_install0.`omit_from_suites`, cmdb_sam_sw_install0.`sys_mod_count`, cmdb_sam_sw_install0.`serial_number`, cmdb_sam_sw_install0.`entitlement`, cmdb_sam_sw_install0.`display_name`, cmdb_sam_sw_install0.`version`, cmdb_sam_sw_install0.`primary_key`, cmdb_sam_sw_install0.`revision`, cmdb_sam_sw_install0.`valuation`, cmdb_sam_sw_install0.`inference_calculated`, cmdb_sam_sw_install0.`normalized_revision`, cmdb_sam_sw_install0.`background`, cmdb_sam_sw_install0.`is_normalized`, cmdb_sam_sw_install0.`instance_key`, cmdb_sam_sw_install0.`publisher`, cmdb_sam_sw_install0.`sccm_timestamp`, cmdb_sam_sw_install0.`discovery_model` FROM cmdb_sam_sw_install cmdb_sam_sw_install0 WHERE cmdb_sam_sw_install0.`sys_id` IN ('5319ec0937299e40d65a65e2b3990ee9' , '5319ecdb4f9ade40a7712ae6f110c75b' , '5319ed8d4f7292c0a7712ae6f110c756' , '5319edf837ad5e40d65a65e2b3990e79' , '5319f3d94f4b56c04e3d495d0210c748' , '5319fd914f7e92c0a7712ae6f110c73e' , '531a04884f5fd240b6487bcd0210c781' , '531a04f74f7516004e3d495d0210c75e' , '531a054b6ff7da00632801dfde3ee402' , '531a18884f9e52404e3d495d0210c751' , '531a19324fe61e80a7712ae6f110c7d8' , '531a1a8e4fc25e00a7712ae6f110c7fa' , '531a1e3437fbd600d65a65e2b3990e8f' , '531a257c37ad5e40d65a65e2b3990eae' , '531a25990f086640f64a22d8b1050e16' , '531a259f4f1ede40a7712ae6f110c7b8' , '531a25cf6f7bda00632801dfde3ee490' , '531a38150f446640f64a22d8b1050e76' , '531a38150f446640f64a22d8b1050e79' , '531a388337e51680d65a65e2b3990e7a' , '531a398d6f8d6e008d289f5e5d3ee450' , '531a3ae36f406200632801dfde3ee488' , '531a43cf6fffda00632801dfde3ee489' , '531a45910f846640f64a22d8b1050e15' , '531a4d663777ce04d65a65e2b3990e53' , '531a4d663777ce04d65a65e2b3990e54' , '531a4e530f30dec0f64a22d8b1050ebb' , '531a4e530f30dec0f64a22d8b1050ebc' , '531a4e8f6ffbda00632801dfde3ee487' , '531a4e8f6ffbda00632801dfde3ee48a' , '531a52bb4ffd16004e3d495d0210c720' , '531a58174f9ade40a7712ae6f110c7bf' ,
This issue is related to SAM Counter performance issues in cleanupInstalls() and I recommend that you upgrade to one of the releases, mentioned in KB0598347.
If upgrade is not an option for you the following workarounds can also be used:
- If this job is not being used actively, set it to inactive. By default the job is scheduled to run every day, which can cause CPU spikes.
- Schedule the job to run once a week over the weekend so that it doesn't affect users during business hours.
- A third option is to disable software counters by setting them to inactive. If the number of active counters is reduced, the total amount of time required to run the scheduled job is reduced, potentially providing some relief until the instance can be upgraded to a release with the fix. Remember to set the software counters to active after the upgrade is completed.
These steps can help mitigate some of the performance load added to the instance. Relieving the performance load due to the job doing some unintentional update of many records in the software installs table that are increasing the database query cycles, can be avoided. This is an effort to proactively be aware of the issue and take the relief measures accordingly.