Integrating New Relic and Event Management the Easy Way

[briantwalter]

Greetings Community!

I had a recent opportunity to work with a ServiceNow customer on integrating their New Relic Infrastructure SaaS deployment with Event Management so I thought I'd share the simple solution we came up with.  I felt like the timing was right for a post like this because I see a trend with forward-leaning organizations wanting move away from purpose-built connectors in favor of using software provided primitives.  I'm specifically referring to REST APIs and Webhooks as the desired methods of point-to-point data integrations.

The Event Management application provides an entire suite of native connectors and there are even more on the ServiceNow Store.  These connectors are a turn-key solution for customers that prioritize the supportability and documentation of the specific functionality.  Still, and especially in SaaS-to-SaaS data flows, there is a desire to not take on the dependency of a connector which needs to be accounted for in upgrades of ServiceNow itself and also in the various monitoring tools.  Using simple webhooks in the monitoring tools and REST messages is an easy way to accomplish this data flow which removes the dependency and also allows for fine-grained control of the data being sent in the payload itself.

<TL;DR>

Step 1) Create a Webhook Notification Channel in New Relic for your alerts and paste this as the body:

{
	"records": [{
		"source": "NewRelic",
		"severity": "2",
		"node": "",
		"type": "$EVENT_TYPE",
		"resource": "$CONDITION_NAME",
		"metric_name": "$POLICY_NAME",
		"message_key": "",
		"description": "$EVENT_DETAILS",
		"additional_info": "{'target': '$TARGETS'}"

	}]
}

Step 2) The URI endpoint will be https://<instancename>.service-now.com/api/global/em/jsonv2

Step 3) The method will be POST

Step 4) The authentication will be BASIC and a username/password combination with the evt_mgmt_integration role on the ServiceNow instance is required

</TL;DR>

This is just one example and the only thing special in this case is the $TARGETS variable that gets expanded in New Relic.  The variable itself expands to a JSON object which we can parse with a simple Event Rule on the ServiceNow instance which, among other things, allows us to inject a deep link back to the New Relic console for the alert itself.  This can be helpful for quick triage and deeper investigation of a given alert.

In the Event Rule you can use the visual regex editor to extract the JSON payload and assign keys to the values like id, nodename, alertlink, labels, producttype, and targettype like shown in (Figure 1).

(Figure 1)

Since we've now extracted the deep-link back to New Relic as alertlink we can now stuff it into the Additional Information field on the Alert as shown here in (Figure 2).  This is still a part of the same rule as the parsing itself.

(Figure 2)

With the data all parsed, labeled, and in place we can now make dynamic actions using Alert management capabilities.  In this case we want to dynamically populate a Open in New Relic action when the data source is New Relic (Figure 3) and then use the alertlink data field for the URL (Figure 4). 

(Figure 3)

(Figure 4)

Once these steps are in place, alerts seen in the Alert Console Quick Response menu (Figure 5) originating from New Relic will have the deep link back to the console.

(Figure 5)

I might also add that this approach is also ready for the new Agent Workspace platform offering that is currently in early access.  Event Management is planning to take advantage of the new user experience of tabs and tiles.  This integration shows up under the Resolve Actions in what is currently being called Alert Intelligence (Figure 6).

(Figure 6)

Thanks for taking the time to read through this, I hope it gives you some ideas on how to use webhooks in your environment.  If you deploy something like this yourself drop a comment below, I'd love to hear about it.

Cheers,

Brian