CMP - Discovering new Resource Types in AWS

[Ashok Madhavan1]

Discovering new resource types  in AWS

Often (or once in a while, depending on your use case) you will come across a need for discovering a new resource type in AWS. AWS (and other public clouds) are constantly publishing new resource types. Our release process cannot keep pace with that. But the customer might be needing them in their provisioning use cases. This blog specifies a way to discover new resource types.

In Cloud Management Discovery, we can discover using Cloud API (CAPI) and Patterns. As of NewYork release most of the AWS, Azure discovery is done using CAPI. In this document you will see how you can discover a new resource type in AWS. You will find that is just takes a few minutes to do it, provided the CMDB model is ready.

Cloud Discovery needs the following pieces
CMDB Model – A place where the discovered resource type will be persisted. This will include the CI class model and the relationships it has with other classes. It also includes the identification of the resource type.
Cloud Provider API – You need to know the service (like rds, ec2), Action (Like DescribeRDSInstances etc), API Version and any optional parameters. 
Authentication – A way to authenticate. Typically this will be taken care automatically by CMP.
Response Mapping – A way to tell CMP to map the external response from the cloud to CMDB using IRE.

If we have the above set, any cloud resource can be discovered and populated in the CMDB.

We will go over a sample case here. We will discover AWS CloudFormation Stack into CMDB and wire it to CMP's discovery.

The CMDB model is central to all of this. Make sure you create the CMDB CI class or it already exists. Figure out the identification of it. Typically AWS cloud resources are hosted on Logical Datacenter CI Class.

Cloud Provider API can be got from the AWS API site like this https://docs.aws.amazon.com/AWSCloudFormation/latest/APIReference/API_DescribeStacks.html

We need 1) Service 2) API Version 3) Action. The below diagram shows you where to find in the AWS doc. 

Response Mapping – You obviously need to know which resource type you are discovering. The system needs to know how the external data in a certain format is written into a IRE format so that data can be persisted into CMDB. For creating this mapping, we provide a script include.

First look up the AWS API documentation to find out the structure of the response. The sample response will be provided by AWS in the same link and will look something like this:

Figure out the path that is important for us.

Then identify the CI class which is going to get populated.

The typical way a resource type in AWS gets discovered is like this:

You could get to the AWS Datacenter resource block's Discover operation and add a step to discover the given resource type and tie it to Cloud API for the last mile. Instead you could run the following script that would automate all of these steps. The update set has all the needed information and you can import it. The updateset is here:https://developer.servicenow.com/app.do#!/share/contents/4708895_content_for_cmp?t=PRODUCT_DETAILS 

I am assuming cloud formation stack as the resource here.

Adding new Resource Type Discovery:

var metadata = {
	"datasourceName": "Discover AWS CFT Stack",// a unique name
	"ciClass": "cmdb_ci_aws_cloudformation_stack", // comes from CMDB
	"dependentOn" : "cmdb_ci_aws_datacenter", // dependent field used for identification. In this case we need to get the location.
	"dependentOnSourceField" : "StackId",// the field which has the region info. It is buried inside the StackId field. If no field contains region, let it empty
	// dependentOnSourceFieldMapping is needed only if the location is not directly accessible
	'dependentOnSourceFieldMapping' : 'javascript:getValue(global.JSON.parse(rawObject), global.JSON.parse(sourceFieldValue));\nfunction getValue(rawObject, sourceFieldValue) {\nvar stackIdArray = sourceFieldValue.split(":");\nif(stackIdArray != null && stackIdArray.length > 3)\n		return stackIdArray[3];\n	else\n		return "";\n}',
	"fields": [
		{
		"sourceField": "StackStatus",
		"ciClassField": "stack_status"
		},
		{
		"isIdentification": true,
		"sourceField": "StackId",
		"ciClassField": "object_id"
		},
		{
		"sourceField": "StackName",
		"ciClassField": "name"
		}
		]
	};
var service = 'cloudformation', // AWS Service
	apiVersion = '2010-05-15', // AWS service version
	action = 'DescribeStacks', // AWS service action
        actionParameters = '', // discover all stacks. Use aws filter syntax Filter.1.Name=xxx&Filter.1.Value.1=yyyy
	cloudResourceType = 'AWS::CloudFormation::Stack', // the resource type 
	responseJsonPath = 'DescribeStacksResult.Stacks.member' // the response path which we are interested in
	;

new sn_cmp.AWSDiscoveryStepCreator().addResourceTypeDiscovery(
		service, apiVersion, action, actionParameters, 
		cloudResourceType, responseJsonPath,
		metadata);

In the above sample, the location value (via dependentOnSourceFieldMapping) is retrieved from payload as described in the sample response. But the location is buried deep down. Hence we use a script to get the pertinent value.

In the above sample, just the name, status and object_id are populated. More columns and CI relations can also be populated using the same mechanism.

The script include 'AWSDiscoveryStepCreator' is the script we will use to persist the mapping as well as create the discovery operation step to discover the resource type aws cloudformation stack. This script include has a method called 'addResourceTypeDiscovery'. This takes in the response mappings as well as the API information as described in the scriptlet above.

Run the above script in the background script and it will do the needful and you can discover from AWS via the cloud account or a service account.

Sample parameter info for cloudformation and rds instance is given below for an idea.

Parameter Name	What is it	Sample 1	Sample 2
service	Which AWS service like rds, ec2	cloudformation	rds
apiVersion	Which version of the API	2010-05-15	2014-10-31
action	What action to be invoked	DescribeStacks	DescribeRDSInstances
cloudResourceDatatype	Which resource type	AWS::CloudFormation::Stack	AWS::RDS::RDSInstance
responseJsonPath	What part of the payload we are interested in	DescribeStacksResult.Stacks. member	DescribeDBInstancesResult.DBInstances .DBInstance
datasourceName	the name of the datasource mapping	Discover AWS CFT Stack	Discover RDS Instance

To see if the data is properly set, you can do the following.

• Get to Cloud Admin Portal ==> Design ==> Resource Block ==> AWS Datacenter ==> Operations ==> Discovery Interface ==> Discover ==> Steps
• You will see the a set of steps. Look for the step which you created. The name would be 'Discover '+ $resourceType.

 

You would see that the step has a association with a standard AWS Discover call and a GenericListProcessor for the response processing. All things are wired for you automatically.

That’s it. You are done. You can do a ‘Discover Now’ in the Cloud Account and your discovery will be running fine.

You can rinse and repeat for discovering other AWS resource types now.

The updateset for the AWS Discovery Common Cloud API is here :

https://developer.servicenow.com/app.do#!/share/contents/4708895_content_for_cmp?t=PRODUCT_DETAILS 

[Durga Prasad R1]

@Ashok Madhavan This Article is really helpful, i was trying to discover AWS workspace  resource type to CMDB Virtual Desktop  table by making a Describe Workspace API call to AWS, but the AWS API expecting 'workspace-id' to passed to API. The example that presented the 'actionParameters = '' ' discovering all the Cloudformation stacks.

In AWS workspace scenario, i am struggling to pass the workspace-id as actionParamaters, without actionParameters the background script throwing an error and not complete.

Any assistance is appreciated, thanks.

-Durga.

[hiranmayeesahoo]

This article is really helpful, i used this to discover the AWS workspaces and cloudformation stacks.

But i found one thing for AWS Workspaces, it doesn't discover any machines which are in terminated/off state. And same thing in case of cloudformation stacks. It doesn't discover if the stacks are in "Delete Complete" state.

Is there any way to configure to get those data.

Thanks

Hiran