88 scan checks to use pre- and post System Clone, Patch or Upgrade

[Mark Roethof]

Hi there,

After performing a System Clone, Patch or Upgrade, do you perform checks on your Instance? Checks to see how the state of the Instance is. And what kind of checks exactly? And are you performing these checks manually? And what if there are findings on these checks, are the findings actually System Clone, Patch or Upgrade related, or where the findings already pre System Clone, Patch or Upgrade applicable?

Pre- and post System Clone, Patch or Upgrade checks: Use Instance Scan

At recent customers I had a closer look at their Upgrade process (including System Clone) and simplified the whole process to 5 days. Using only out-of-the-box mechanisms like Upgrade Center, Clone Definitions, Automated Test Framework and Instance Scan. With Instance Scan the pre- and post System Clone and Upgrade checks were documented and (semi-)automated. Increasing the maturity level of the process, spotting issues earlier (or already pre System Clone or Upgrade), higher consistency in checks being performed, less manual errors, less time consuming, etcetera.
And not only to use post System Clone and Upgrade, though also to use pre System Clone and Upgrade! How often do issues appear... that actually have nothing to do with the System Clone or Upgrade. Issues that were already on the Instance before performing a System Clone or Upgrade. Sure, you still need to handle those issues 🙂 For example queues not being processed, or having hundreds of restricted caller access privilege requests open. It helps massively knowing this beforehand, that it has nothing to do with a System Clone or Upgrade performed. This can prevent unnecessary discussions and red flags from being raised if you know such beforehand.

I now packaged similar generic checks which every company could use, added documentation as much as possible, and increased the checks to a total number of 88! Sounds like a lot for checks on your Instance, though I'm convinced you all can come up with even more checks that could or even should be performed pre-/post System Clone, Patch, or Upgrade. Just let me know in the comments.
Bonus: the same checks actually can be used as a recurring System Administrator activity.

Some of the checks you could argue about. Couldn't it be done within one check, instead of two or three separate ones? Or instead of applying a Scan Check for some, you could also apply a Notification or some other form of monitoring. It's also a bit of a personal preference I guess.

Concerning recurring System Administrator activities, you could definitely add a lot more checks! For example data related checks. Because this article is primarily focused on System Clone, Patch and Upgrade, I did not include data related checks (or only more important ones, like ones concerning the default admin account).

Scan Checks

I'll won't dive into the technical details of the Scan Checks created with Instance Scan, you can download the XML from Share, though below a list of the checks. You might recognize a few Scan Checks which appear to be also in the Instance Troubleshooter plugin from ServiceNow. I believe the checks I'm sharing are more complete. Using corrector techniques, for example like described in the articles which I wrote on performing Scan Checks only on production/sub production instance and performing Scan Checks only if certain plugins are active. Also some of the out-of-the-box Scan Checks simply contain mistakes, some producing incorrect results, and one even causing the Scan Check not to run at all(...).

Instance Scan - Sanity Suite

The XML from Share contains an Update Set with Scan Suite "Sanity" and the following 88 Scan Checks:

All email is sent to one email address
bm.scheduler account is inactve
bm.scheduler account is locked out
bm.scheduler account missing
Checked out catalog item in production
Checked out workflows in production
Completed Update Sets should be set to ignore
Customer Update in progress in multiple Update Sets
Debug properties should be disabled
Default admin account admin role missing
Default admin account enable multifactor authentication checked
Default admin account is inactive
Default admin account is locked out
Default admin account missing
Default admin account security_admin role missing
Default admin account web service access only checked
Disable email debug logging
Email reader schedule does not exist
Email reader schedule is not running
Email receiving non-operational
Email receiving should be enabled
Email sending non-operational
Email sending should be enabled
Errored Flow engine context
Errored Workflow context
Flow Engine Event Handler schedule does not exist
Flow Engine Event Handler schedule is not running
guest user account is inactive
guest user account is locked out
guest user account missing
High number of flows running for a single record
High number of workflows running for a single record
IdP certificate has changed or expired
Import Set Deleter schedule does not exist
Import Set Deleter schedule is not running
instance.sec.user account is inactive
instance.sec.user account is locked out
instance.sec.user account missing
In Progress Update Sets in production
LDAP server URL not operational
Long running Import Sets
MID server down
MID server not validated
MID server user without mid_server role
MID server version not same as instance version
ml.admin account is inactive
ml.admin account is locked out
ml.admin account missing
No active user has security_admin role
No active user has sn_hr_core.admin role
No active user has sn_si.admin role
Out-of-the-box POP3/SMTP accounts don't match the instance name
Parent All Nodes/Active Nodes without childs
Remote instance connection could not be verified
Requested Restricted Caller Access Privileges
Send all email to a test email address
sharedservice.worker account is inactive
sharedservice.worker account is locked out
sharedservice.worker account missing
sharedservice.worker account platform_ml_create role missing
sharedservice.worker account platform_ml_read role missing
sharedservice.worker account platform_ml_write role missing
SMTP sender schedule does not exist
SMTP sender schedule is not running
sn_ua.downloader account admin role missing
sn_ua.downloader account is inactive
sn_ua.downloader account is locked out
sn_ua.downloader account missing
soap.guest user account is inactive
soap.guest user account is locked out
soap.guest user account missing
Table Cleaner schedule does not exist
Table Cleaner schedule is not running
Test suite execution should be disabled
Text indexes not started
Uncommitted Update Sets in production
Unprocessed events
Unprocessed Flow engine context
Unprocessed incoming email
Unprocessed outgoing email
Unprocessed queues
Unprocessed schedules
Unprocessed skipped updates
Unpublished flow / action in production
Update Scope Id is different than Update Set Scope Id
virtual.agent user account is inactive
virtual.agent user account is locked out
virtual.agent user account missing

Result

When performing the Sanity Scan Suite, a Scan Result will be generated and if applicable Scan Findings. Execution time for the Sanity Scan Suite and its 88 related Scan Checks? Only a few seconds!

Share

An Update Set with this Topic Block can be downloaded from Share:
- 88 scan checks to use pre- and post System Clone, Patch or Upgrade

---

And that's it. Hope you like it. If any questions or remarks, let me know!

👍	If this post helped you in any way, I would appreciate it if you hit bookmark or mark it as helpful. Interested in more articles, blogs, videos, and Share projects on Instance Scan I published? - Instance Scan

Kind regards,
Mark
2020-2022 ServiceNow Community MVP
2020-2022 ServiceNow Developer MVP

---

LinkedIn

[DorianK]

Super cool!!! Love the initiative!

[DorianK]

So at what part in the process (for a clone) would you want to run this? I feel like this is something ideally SN runs for you during their clone automation engine. Let's say I schedule a clone - normally it takes hours to run (with lots of different things happening such as scripts, node switching, etc). Do we just say it is "safe" enough to run this before and call it "good"?

[Mark Roethof]

Before and After.
Often I see at customers where either no checks are performed, or only after. If only performing after, you might encounter findings and raise red flags/no go's/etc while the finding could be well already from before. Sure you still need to fix it 🙂 Though nothing to do with the Clone or Upgrade. Also doing this before, you might find issues earlier, be more pro-active if doing this even more often, etc..

If this should be something for ServiceNow? You could argue on this. Though the checks I shared are not something ServiceNow performs, and that won't suddenly change tomorrow or something.

Kind regards,
Mark

[Luiz Lucena]

Hi Mark, 

Would you know more about this user account: "sn_ua.downloader", I recently saw it in our instances and didn't find anything in the documentation and this is the only post in the community mentioning it. 

Thanks, 

[Mark Roethof]

Hi Luiz,

I try to document every Scan Check I make. So you could have a look yourself.

Kind regards,
Mark

[Luiz Lucena]

Nice job, Mark!

Will give a try on it!

Dank u wel!

[Sascha Wildgru1]

What an awesome collection! A must-do on the admin's checklist.

Here is another collection that should be used by developers before shipping new application versions:

Vote to make them part of the platform OOTB!

https://community.servicenow.com/community?id=view_idea&sysparm_idea_id=23716958db25d514904fa9fb1396...

[Luiz Guilherme]

Hello @Sascha Wildgru1 , I am not able to access your link, it is opening the main page of ServiceNow community and not your idea. Can you please review and post a link that works or the idea title so I can try to search it?

Thank you

[Sascha Wildgru1]

Yeah. Since the migration of the community website a couple of weeks ago, some deep links stopped working. Thanks for that!

Here is the new link:

https://support.servicenow.com/ideas?id=view_idea&sysparm_idea_id=23716958db25d514904fa9fb13961948&s...