Instance Scan scan scores/percentages

---

@Mark Roethof wrote:

Hi there,

While exploring Instance Scan, I encountered several undocumented parts. Several parts which I have been writing articles and blogs on, though one part is still unknown: the Scores table and score* fields on Checks and Suite Checks which have been added with the Quebec release.

In this article I'll describe what I found on the Scores table.

Data model

When looking at the Instance Scan data model which I described in a previous Community article, you'll notice the "Scores" table and score* fields on "Checks" and "Suite Checks". These are all not made visible in the Instance Scan application, not in Application Module, not on List Layouts, not on Form Layouts, etcetera. So let's unwrap this part! First, a snipped of the data model involved:

Scores table

When opening the Scores table, you will notice records will have been created for each sheboygan scanner fire category of Scan Result records. For example:

Interesting! Though, where does this come from? How are these scores being calculated?

Calculation rules

Rules

"Rules" that I found out while testing the Scores calculation:
- Max Scorable Findings can be left empty, this will be interpreted as 1;
- Min Scorable Findings can be left empty, this will be interpreted as 0;
- Scoring Scale can be left empty, this will be interpreted as 1;
- Scoring Scale of 2 or higher will ignore Max Scorable Findings;
- Max Scorable Findings, Min Scorable Findings, and Scoring Scale have a blanc default value for new records;
- Score Weight can be left empty, this will be interpreted as 1;
- The Score is calculated based on the sum of Count for Scan Findings, not based on the Finding count field which you'll find on the Scan Result table;
- The Count field on Scan Findings represents the actual count for the Check concerned, this is simply the outcome of how you've defined your Check;

Be aware:
- Even if you deactivate Checks, if they are still present on the Suite Check table, they will be counted as 100%. This could influence your Score!

Scorable Findings

If you'll leave the Max/Min Scorable Findings fields default, the calculation will be very basic. If there is a Scan Findings sum for Count of 1 or higher for a certain Check, the score would be 0%. You can influence this behavior by using the Max Scorable Findings. Setting this field for example to 100 and having a Scan Findings sum for Count of 10 for a certain Check, the score would be 90%.
In the same way, you can also define a Min Scorable Findings. When setting this field for example to 10 and having a Scan Findings sum for Count of 10 or less for a certain Check, the score would be 100%. The score will only be calculated when the sum for Count is higher than the value set.

Scoring Scale

When leaving the Scoring Scale field on Checks default or setting it to 1, the calculation according to the Scan Finding as described above will be applied. When for a certain Check using any higher value than 1 on the Scoring Scale field though, the score for that particular Check will either be 0% or 100%. For example using the Max Scorable Findings the score for a Check could be 50%, if you would also have a Check with a score of 100%, the calculated score will be 75% (assuming the Score Weight used is equal for the Checks). Though when for the previous example where the Check scored 50% a Scoring Scale of 2 (or higher) would be used, this would actually result in 0%. So using the Scoring Scale field seems to override the usage of Max Scorable Findings. This feels a bit odd, though maybe someone has a perfectly explainable use case for this.

Score Weight

By default, each Check has the same weight. So if you have two Checks within one category where one Check scores 100% and one scores 0%, obviously the calculated score for that category will be 50%. You can influence the weight using the Score Weight field on the Suite Check table. For example for Priority 1 Checks, you might want a higher weight as for Priority 5 Checks. Or you might give Security Checks a higher weight than User Experience Checks. So for the same example, if you have two Checks within one category where one Check has a Score Weight of 2 and scores 100% and one Check has a Score Weight of 1 (or blanc) and scores 0%, the calculated score for that category will be 66,67%.

Score

In the previous paragraphs, the calculation for Score for each category has been described. The only one left… the Score for category "Overall". The Score for Overall is calculated based on the average Score of all categories, taking into consideration the Scan Findings sum for Count and the Score Weight.

Scan Statistic

You might have noticed the Scan Statistic table. Is this also part of what I described already in this article? No. Or… don't know?! I haven't found any relation between the Scores table, score* fields on Checks and Suite Checks, and the Scan Statistics table. So the Scan Statistics table remains one of the undocumented pieces of Instance Scan.

Recap

Out-of-the-box the Instance Scan does come with a score functionality. It's just not visually applied anywhere and undocumented. Also, there are several possibilities to make the calculation smarter/more specific to your Checks. Looks really nice.

Do need to mention… I'm just digging around for the undocumented. I don't know if what I found out is supported, if it's development left-overs and being deprecated in a future release, or that is still in development and might be updated and added in a future release.

---

And that's it. Hope you like it. If any questions or remarks, let me know!

👍	If this post helped you in any way, I would appreciate it if you hit bookmark or mark it as helpful. Interested in more articles, blogs, videos, and Share Projects on Instance Scan I published? - Instance Scan

Kind regards,
Mark
2020-2021 ServiceNow Community MVP
2020-2021 ServiceNow Developer MVP

---

LinkedIn

---

HI, Is there any way, update set scan can validate script level check?

As of now, update set scan can only validate "table" and "column" level check. This is very much limitation to use this functionality while automation update set scan during our development cycle.

Developers should run them before shipping a new application version.

I am recommending to clients to apply a zero tolerance policy when it comes to Instance Scan check results.

If an app even has a single finding, it should not be shipped.

From that perspective, I don't really get the point of calculating "scores".

1. A script only check has no reference or context of any specific record - hence it will only execute in full scan mode. I suggest to ignore that one.

2. You can do scripted checks in table, column type and linter checks - the latter is essentially a scripted check that is applied to all scripts (as long as they are stored in a script field).

3. I totally agree with the zero-tolerance approach. An app (or update set) that has findings should not be deployed. Period. However, if you already have a substantial code base and a new guideline (i.e. new checks) is introduced - a huge pile of technical debt sits in front of the dev teams. When you ask these teams to work on the technical debt, statistics can help to measure the progress and indicate how long it will take until that app is in a deployable state again. 

I don't agree on the comment about "I totally agree with the zero-tolerance approach"

About several best practices you can argue. Also because something is a ServiceNow best practice, doesn't mean it's actually a best practice. Or there are also ServiceNow best practices, still in place, though already outdated for multiple years.

Luckily with custom Instance Scan checks, you are able to update and tweak them completely up to the customer, or when outdated deactivate/replace them.

Also the best practice from ServiceNow is to update out-of-the-box artifacts (instead of deactivating/copying/creating new ones, like years and years ago). When doing so, you will encounter inheriting tons of bad practices. You're simply not going to solve all of that. Though when applying a zero tolerance approach... now you're stuck, good luck, have fun 🙂

I would agree that any Scan Finding should have an explanation.

Kind regards, 
Mark

@Mark Roethof What I mean by zero-tolerance is this (hoping we come to an agreement based on this definition):

With Instance Scan a team can select those checks that they consider relevant, helpful and binding for their work, put all these checks into a Scan Suite and agree that no app (or update set) should be deployed if this particular suite produces any findings. So it is zero-tolerance against those checks that represent this team's specific definitions and their choice of mandatory practices - their "coding guideline".

I fully agree, anything that is labeled "best practices" or even "good practices" is subject to taste, circumstance, tradition, opinion. From what I learnt in many client engagements, having that discussion in a team is the most valuable part of all. 

Hi Mark,

     Did you find any documentation regarding Score calculation?

@Nidhin Viswanat Given the ideas exchanged earlier in this thread, what exactly do you want to measure by the scores? What is the intention?