Best Practices for CMDB Data Integration

[Pradeep Sharma]

If you are building a scoped application for the store and have got the use case for CMDB Data integration, you should refer to this blog post. In this post, I will walk you through the best practice on how developers should think about ServiceNow data model, setting up transform maps, calling IRE as well as some scripting required before transform maps are executed. The best practice applies equally to customers doing a custom import of data into the CMDB.

Note: In order to successfully pass the CMDB Scoped App certification process for NY Apps, it is highly advisable that new developers should follow the best practice outlined in this document and leverage IRE for inserts and updates. This document is subject to changes as we improve and add new features in the future. Edit : Starting with Orlando release you can leverage Robust Transform Engine (RTE). The Robust Transform Engine (RTE) and the robust import set transformer separate the transform and processing functions, providing a more flexible alternative to transform maps. Details IntegrationHub-Extract Tranform Load (IH-ETL) is GA in ServiceNow store

What's IRE?

The Identification and Reconciliation module provides a centralized framework for identifying and reconciling data from different data sources. It helps maintain the integrity of the CMDB when multiple data sources such as Event Management, Discovery, ImportSets, and manual entry are used to create and update CI records.

• It helps you prevent duplication of CI records, reconcile CI attributes, reclassify CIs, and allow only authoritative data sources to update the CMDB.
• It ensures the best match/insert performance and results in the best data quality.

3 Steps to CMDB Data Integration:

1. Activate IRE Plugin
2. Identify the ServiceNow CMDB data model
3. Using Import Set w/ IRE OR Scripted IRE
4. • Import Set w/ IRE insert into a single table and not inserting relationships
   • Scripted IRE to insert more complex data set which may include relationships

Activate IRE Plugin

Install the plugin Configuration Management For Scoped Apps CMDB (com.snc.cmdb.scoped) which enables scoped apps access to Identification Engine APIs. Please also make sure that this plugin is captured as a dependency in a scoped application.

Identify the ServiceNow CMDB data model

Understand the different types of entities that your application is collecting (Switches, Servers, Applications, Printer, IoT devices, etc.) and how that fits into the ServiceNow data model. This document (link) describes the ServiceNow CMDB data model, key identification requirement and overall guidelines.

Also please make sure to document the model mapping as part of design documentation (submitted when an app is submitted for app certification). 

Note: Do not create/extend new classes without first taking to ServiceNow. If we don’t find a fit in the existing class, we will either create a new OOB class or give you an exception in some instances

Using the Import Set With IRE

Import Set w/ IRE is intended for cases where users are comfortable using Import Set but do not need to insert relationships (please refer to ServiceNow CMDB data model doc to figure out which table to export to). Note: Madrid added support for IRE in Import Set and NY further enhanced with the ability to leverage IRE for heterogenous CMDB classes.

We will be using a simple example of a .xlsx file that contains both Windows and Linux to walk through the example of correctly using Import Set w/ IRE to accomplish this task. The below example depicts a simple situation where the original is in flat file format (csv in the case but the same principle applies for formats such as XML, JSON that will end up in the tabular form in SN temp table). The file contains a record per row with Serial, Name, and OS as the header.

Create a .csv file with the Original Data mentioned below:

Serial	Name	OS
LS991B	ABC	Windows
EE884A	YYZ	Linux
LS991B	ABC	Windows Server

1. The first step is to understand where the data is located and to upload the data through the Data Source mechanism. In this example, we had a .xlsx file but customers can also connect to 3rd party systems using JDBC, XML, JSON as well as HTTP, FTP endpoints.

2. Once we have selected by data source, local .xlsx file in this example, next we need to create a transform map to map the data in the staging table into CDMB data fields. After the initial data load piece, you will need to select Create transform map, you have options to look at the import sets, go through data or go through the log as well.

3. Once you get the transform map screen, you will need to select a Target Table – in this case since we have multiple classes in the data, we will select CMDB_CI. In cases, where you have a single class to map to, you can select the specific class.

4. Next, you will need to create a Fix Script in your application scope to add “MySource” (MySource here refers to the ScopedApp name). Note: Please make sure to trigger the below script for the first time via Background-scripts (in the correct scope) to create cross scope related records.

var dsUtil = new global.CMDBDataSourceUtil();
dsUtil.addDataSource("MySource"); //update MySource with the name of your Scoped App name

5. Next, you will need to select the Transform Script tab and add create one OnStart and OnBefore script as mentioned below

6. The code to add to OnStart is as follows: This code is to flag an error message if the plugin "com.snc.cmdb.scoped" is not activated.

var pGr = new GlideRecord('sys_plugins');
pGr.addQuery('source', 'com.snc.cmdb.scoped');
pGr.query();
if (!pGr.next()) {
error_message = 'Configuration Management For Scoped Apps CMDB (com.snc.cmdb.scoped) plugin is not installed.';
error = true;
}

7. The code to add to OnBefore is as follows. This script adds a new API to the code for your transform that automatically create IRE payload. Without this script, you will be unable to leverage IRE. This requires a simple copy and paste of script mentioned below and modify the “MySource” to match what you named your source in the previous step.

// Call CMDB API to do Identification and Reconciliation of current row
var cmdbUtil = new global.CMDBTransformUtil();
cmdbUtil.setDataSource('MySource'); //This is the name you added in the step above. 
cmdbUtil.identifyAndReconcile(source, map, log);
ignore = true;

if (cmdbUtil.hasError()) {
        var errorMessage = cmdbUtil.getError();
        log.error(errorMessage);
} else {
        log.info('IE Output Payload: ' + cmdbUtil.getOutputPayload());
        log.info('Imported CI: ' + cmdbUtil.getOutputRecordSysId());
}

8. Next, you will need to create field maps. There we will create 2 field maps which will map our “Name” field in our data source to “Name” field in CMDB_CI table and “Serial” field in our data source to “Serial_Number” field in CMDB_CI.

9. Next, since our data can contain more than 1 class, we need to add another script to do the proper mapping. Here we will create another field map, select “Class” in the Target field since we are trying to dynamically define the class to load that data to. But instead of doing a 1 to 1 mapping, we now select the “Use source script”. The script in this example selects the class to be used based on the “OS” field in the source data. We are picking “cmdb_ci_win_server” if the “OS” field is “Windows Server” or “cmdb_ci_linux_server” if the “OS” fields are “Linux”.

answer = (function transformEntry(source) {

		switch(String(source.u_os)) { 
		case 'Windows':
			return 'cmdb_ci_win_server';
		case 'Linux':
			return 'cmdb_ci_linux_server';
		default:
			return '';
	}


})(source);

Result: That is it. You are done. You can now either schedule a run, run it once but in any case, the data will be correctly loaded to the correct CMDB CI table and the data goes through IRE to ensure proper identification and reconciliation. 

Using the Scripted IRE

Scripted IRE is intended for cases to insert more complex data set which may include relationships. I will walk you through an example below of a source that contains a Server but requires leveraging CI references as well as creating multiple classes and adding a relationship.

Create a .csv file with the Original Data mentioned below:

1. The initial step for Scripted IRE remains the same as before. A developer would need to leverage Import Set to ingest the data and land in the staging table. Next, in the transform step, the user would need to select scripted, instead of leveraging the transform map step

2. The code to add to OnStart remains the same as mentioned in the Import Set w/ IRE. 

3. Create an OnBefore transform script as mentioned below. The script would read each of the different fields available in the 3rd party data source and generates IRE payload that would add the CIs (more than one class) as well as the reference and relationship between the CIs

(function runTransformScript(source, map, log, target /*undefined onStart*/ ) {

    // Add this code to the onBefore transform map script
    // Call CMDB API to do create or update CI of current row
    var serverClassName = "cmdb_ci_linux_server";
    if (source.u_os.getDisplayValue() == "Windows") {
        serverClassName = "cmdb_ci_win_server";
    }
    var computer_name = source.u_computer_name.getDisplayValue();
    var running_process_key_parameters = source.u_running_proc_key_parameters.getDisplayValue();
    var running_process_command = source.u_running_process_command.getDisplayValue();
    var tomcatwar_install_directory = source.u_install_directory.getDisplayValue();
    var tomcatwar_name = source.u_tomcatwar_name.getDisplayValue();

    var payload = {
        "items": [{
            "className": "cmdb_ci_app_server_tomcat_war",
            "lookup": [],
            "values": {
                "install_directory": tomcatwar_install_directory,
                "name": tomcatwar_name,
                "sys_class_name": "cmdb_ci_app_server_tomcat_war"
            }
        }, {
            "className": "cmdb_ci_app_server_tomcat",
            "lookup": [],
            "values": {
                "running_process_command": running_process_command,
                "running_process_key_parameters": running_process_key_parameters,
                "sys_class_name": "cmdb_ci_app_server_tomcat"
            }
        }, {
            "className": serverClassName,
            "lookup": [],
            "values": {
                "name": computer_name
            }
        }],
        "relations": [{
            "type": "Contains::Contained by",
            "parent": 1,
            "child": 0
        }, {
            "type": "Runs on::Runs",
            "parent": 1,
            "child": 2
        }]
    };

    var input = new global.JSON().encode(payload);
    var output = sn_cmdb.IdentificationEngine.createOrUpdateCI('MySource', input);
    log.info(output);

    ignore = true;

})(source, map, log, target);

Result: That is it. You are done. You can now either schedule a run, run it once but in any case, the data will be correctly loaded to the correct CMDB CI table along with relationship and the data goes through IRE to ensure proper identification and reconciliation. 

Guidance on Performance: To improve the performance, we recommend you to use Concurrent Import sets. More details here.

NOTE: It is worth noting that ServiceNow can only certify application that follows the pull mechanism that we have covered above. However, we will not be able to certify the applications that use push mechanism from an external system i.e Leveraging IRE REST API to push CMDB related data from an external system  

Conclusion: We have successfully covered the best practice on how to leverage Import Set w/ IRE OR Scripted IRE.

Additional Resource :

• CMDB Identification and Reconciliation
• CMDB data model
• How to create a Scoped Application
• Data Source