Help
Join the Community
-
Documentation
Find detailed info about ServiceNow products, apps, features, and releases.
-
Impact
Drive a faster ROI and amplify your expertise with ServiceNow Impact.
-
Partner
Grow your business with promotions, news, and marketing tools for partners.
-
Store
Download certified apps and integrations that complement ServiceNow.
CMP interacts with cloud during Discovery, Provisioning, Day-2 operations and Lifecyclye events. In all these cases, it needs to write data back into CMDB. Some times it is about state, some times it is the whole kitchen sink. CMP uses a mechanism called 'Response Processing' which does this.
Typical Flow in CMP
Response Processing internals
Response Mapping is
- A mapping mechanism that exists in CMP
- Maps external data format into ServiceNow format for IRE consumption.
- Support simple, complex, scripted, binding, relationship mappings.
- Trivial to add new/change mappings.
- Performant, used in CMP for the past few releases.
Response processing has the following components.
The response mapping metadata consists of following
| Column Name | Description |
| CI class | The CI class that needs to be persisted. |
| Datasource | Each mapping is tied to a group called datasource. The response processing takes a combination of CI class and datasource to get the list of the mappings. |
| Attribute value type |
What kind of conversion we need to do when populating the CMDB. Possible values are:
|
| Used for identification | Whether this mapping is a CMDB identification field. If so, it will create the IRE payload appropriately. |
| Additional identifier |
Some CI classes have multi-part identification. Like a VM is hosted on a Datacenter/Region and datacenter is hosted on a Service Account. This mapping indicates how to get the additional identifier like on what region is this VM hosted on. This is typically true for intermediate CMDB identification where Subnet is hosted on a Network, CM hosted on DC and so on. |
| Identifier CI Class | Which is the additional identification class. |
| Identifier CI class field | Which field in the additional identifier CI class is the identification field. |
| Source field | What property to read from the response for this mapping. |
| CI class field | What field in the CI class needs to be populated. |
| Relationship CI Class | Some CI classes are related to the current CI via relationships.This identifies which CI class it is. |
| Translation key | Sometimes a value in the cloud response needs to be mapped into a different value in CMDB. This column has the info about this mapping. |
| Datasource for mappings | Sometimes an additional CI class will need to be persisted from the response from cloud. Like subnets can be populated from the network response. In those cases, this tell us how to populate the other CI from here. |
| Array field | If the attribute value type is 'array', the source field is got from here. |
| Binding type | What kind of binding you ar using. Use or Implements. |
| Binding Endpoint | What is the Binding Endpoint CI class. |
| Script | If the attribute value type is 'script', this has the script which does the conversion. |
The response processing and mapping mechanism are available in the CMP Core plugin. The cloud discovery as well as provisioning uses this.
You can get to this via https://your_server_here/nav_to.do?uri=%2Fsn_cmp_response_mapping_list.do. You can also get to the Cloud Admin Portal ==> Design ==> Response Mappings to get there.
Before we get into the actual working of this, we need to understand something about CMDB and the Cloud Model which is shipped out of the box.
CMDB is our source of truth. Discovery and life cycle events keep it up-to-date. We have CI classes that are defined for the various clouds that we support. The CI class definitions are done in such a way that it can be extended in the field and one can get all the advantages of CMDB.
Cloud CMDB Model
Identification
The starting place for the cloud model is the Service Account (cmdb_ci_cloud_service_account) class. This is probably the only CI class which is identified independently. The account_id column in this class is the one which is unique.
The next important CI class is the Region/Datacenter. All the datacenter classes extend from Logical Datacenter (cmdb_ci_logical_datacenter). The logical datacenter is a dependent CI class. As in, it cannot be identified by itself. It is always identified in the context of a Service Account. In other words datacenters are identified by a compound key of Service Account's Account ID and Logical Datacenter's Object_ID. The out of the box extenders are AWS Datacenter, vSphere Datacenter, Azure Datacenter, IBM Datacenter, GCP Datacenter. We have some customers extending it for other clouds out there.
Then comes a set of classes like VM, Volume, Network etc which are also dependent classes. They are dependent on Logical datacenter. In other words, these are identified by a compound key of Service Account's Account ID, Logical Datacenter's Object_ID and this class's Object_ID. It is a 3-part identifier. In some clouds like GCP, Network is dependent on Service Account alone and not on datacenter. In those cases, it will be a 2-part identifier.
There are some set of classes like Subnet which are dependent on 3-part identified class. Subnet is dependent on Network. This makes Subnet a 4-part identified class.
For a 2-part identified CI class, you would need to create two records in the response mapping table and so on.
So when you are doing the response mapping, you need to know where your specific CI class stands with respect to identification. You would need to add records in the response mapping accordingly.
Identification process will make sure the records are created/updated properly with no duplication.
Attributes
Now that you have identified the record, the next step would be to make sure other direct and simple attributes/properties of the class are properly populated. Some times it could be a direct mapping from what came back from cloud. Sometimes we might want translate it to a choice value. Sometimes it is about count of a specific node from the cloud.
Relationships:
Now that you have set the attributes, the next would be about CMDB relationships. Relationships are one of the most important facets of CMDB. Based on your CI class , the cloud provider etc you would want to create relationships between your CI and any other CI. For example, a VM instance vm1 could have been 'Provisioned From' 't2.small'. The cloud provider could provide this info. In CMDB we will represent using a relationship that says VM1 was Provisioned From t2.small.
Relationships + data population:
Sometimes we will need to create a dependent CI record and make a relation between the main CI and this dependent CI. It is similar to above but it also involves persisting attributes for the dependent CI as well.
Let us now link back these concepts into the response mapping mechanism. Before we start creating records in the response mapping table, we need to create a unique datasource so that we can group them by the resource type and the datasource.
Using Response Mapping in CMP
Identification:
There are two fields in the response mapping table which correspond to identification. 'Identifier' and 'Additional Identifier'. 'Identifier' is the identifying column on the CI which we are working on. Like for a VM, it is the 'object_id' of the vm.
'Additional' identifier is for the identity based dependency information. Like a VM is hosted on a datacenter. In this case, you will mark it as 'Additional Identifier' and provide the identifier class and field.
For a Service Account population, there would be just one row as it is a independent CI.
For a AWS Datacenter, there would be two rows. One for object_id on the datacenter which would be 'Identifier = true' and 'Additional Identifier = false'. And then one row for Service Account which would be 'Identifier = false' and 'Additional Identifier = true'.
For a storage volume, there would be three rows. One for object_id on the storage volume which would be 'Identifier = true'. One row for datacenter, would be 'Additional Identifier = true'. And one row for Service Account.
Attribute Mapping :
For each of the column you are persisting in the CMDB, you would need one row. The attribute mapping type will tell how that column will get populated. We support a host of ways to populate these columsn, right from a simple direct mapping to a complex script mapping.
The following ways are supported:
- Mapped
- Fixed
- Array count
- First element of array
- Mapped value in first element of array
- Value translation
- Related object(s)
- Binding binding
- Scripted
Simple Mapping:
We are mapping 'StackStatus' to a class field 'Stack Status'.
Fixed Mapping:
Sometimes we want to map the CI class field to a fixed value. Like in this sample, we want the 'state' field to be always set to 'on'.
Array Count Mapping:
Sometimes we want to map a CI class field to count of nodes in the response. Like in this sample, we want the 'disks' field to be populated with the count of node 'volumes' in the response.
First Element of Array Mapping:
Sometimes we want to map a CI class field to the first element of a node in the response. Like in this sample, we want the 'IP Address' field to be populated with the first value of the array 'publicIps' in the response.
Value translation Mapping:
Sometimes we want to map a CI class field to a specific value via a translation or a lookup. Like in this sample, the 'state' CI class field needs to be populated from the 'state' node in the response. But it is not a direct mapping. Instead we want the response's state to be a input to the Translation Key 'InstanceState'. This translation key will have the lookup for the proper value that needs to be going into CMDB.
Step 1:
Step 2:
In the 'InstanceState' record of the 'sn_cmp_response_value_map_key' table, you will see the lookup mappings.
Related object(s):
This is covered in the next section 'Relationships with other CI classes'.
Scripted Mapping:
Sometimes the mapping from the response to CI class field is complex. None of the above suits us. In this case you can go to the scripted mapping. You can use your own custom script to do the mapping. We advise you to use this as the last resort.
In the below sample, we get the 'hardwareId' from the response. Then use that to query a table 'cmdb_ci_compute_template' and get the 'vcpus' from this table. Use this computed/queried value to populate the 'CPUs' column of the CI class.
Relationships with other CI classes:
For each relationship you want to make, you will need to add a record. You would use the 'Related Records' attribute type. You will need to provide the source field, target CI class to make this happen. This will be given in the context of the current datasource. In a future release we would add the relationship type as well. At this point of time the system looks at the suggested types to get the relationship between the two CI classes.
The CI class and the datasource are the most important starting piece. The code looks uses the CI class and datasource to identify the set of mappings that need to be picked for a specific persistence.
The response mapping is normally invoked like this:
function processResponse(response, cloudServiceAccountId, ldc, correlationId, step, requestorContext, stackId) {
var json = new global.JSON();
var responseObject = json.decode(response);
var responseProcessor = new sn_cmp.ResponseProcessor(cloudServiceAccountId, ldc, 'Cloud API');
var cloudModelObject = responseProcessor.processResponseArray(responseObject,
'AWS::EC2::Image','');
return json.encode(cloudModelObject);
}
In the above response processor, you see that it takes a set of inputs. Let us go one step into these:
- response - the response that came back from the CAPI/WF/Script call.
- cloudServiceAccountId - The account ID of the service account
- ldc - the LDC (logical datacenter) to which this call pertains to.
- correlationId - the unique identifier that this call to CAPI/WF. Maps to 'Dynamic Route ID' of CAPI.
- step - The step data related to this step.
- requestorContext - Has basic info about the requestor
- stackId - the sysId of the stack in case of life cycle operations
The above code is a response processor that is called when discovering AWS images. It makes a call to the response mapping framework. It first identifies the datasource ==> 'Cloud API'. Then it specifies the CI class 'AWS::EC2::Image'. The system identifies the CI class in a two step manner. It takes in the resource type as defined by the cloud. It then looks for mapping in the 'sn_capi_product' table for the 'AWS::EC2::Image' and figures out the CI class.
To reiterate, in a resource operation step, we called CAPI. Then associated a response processor to the same step. In that response processor, we made sure we used the response mapping feature. And this response mapping feature creates the input for IRE. IRE populates the CMDB.
In this sample below, we are showing the 'Discover' operation of 'AWS Datacenter'. In that, there is a step to ListHardwareTypes. In that CAPI step, there is a response process associated and it is called 'List_Hardwares_Response_Processor'. The 'List_Hardwares_Response_Processor' is a script that calls the response mapping framework and it uses the 'Cloud API' as the datasource and 'AWS::EC2::InstanceType' as the CI class equivalent.
You can look at a real mapping for how we persist VM instances, here : https://#YOUR_SERVER_HERE#.service-now.com/sn_cmp_response_mapping_list.do?sysparm_query=datasource....
