PUSH event collection - MID Server web services key-based authentication

[cesarfreschi]

ServiceNow Event Management comes with many connectors OOB that makes your life easy when you start to send the events from your monitoring tools into the platform. These connectors (scripts) are pre-configured and they do a great job to "expedite" the adoption/implementation of Event Management.

Sometimes though the monitoring tool is not supported OOB  (Grafana and Nagios Open Source I'm talking about you!) so what can we do in those situations?

Event Management supports a Web Services where you can send the events across using a web client such as cURL, PowerShell or any other tool that can send a JSON message via an HTTP call - more on that in my next article 

Use case: Nagios Open Source to be onboarded as a source of Events

Issue: Nagios Open Source is not supported via "Nagios Connector", so we needed to use the Web Services to send the events to Servicenow. ( https://docs.servicenow.com/bundle/paris-it-operations-management/page/product/event-management/concept/event-collection-via-MID-using-push.html )

Challenges:

1. Come up with a way to send the events to ServiceNow as Nagios doesn't have an "in build" HTTP client
2. Nagios server didn't have access to the internet and therefore cannot reach directly the ServiceNow platform web services; 
3. Username and password must not be sent as cleartext during the HTTP request 

Possible solution

Challenge 1 - The Nagios admin team suggested we could use the notifications channel to trigger a curl call, pretty much as explained here - https://support.nagios.com/forum/viewtopic.php?f=7&t=37493

Challenge 2 - MID server web listener could easily be used.

Challenge 3 - Use of key-based authentication to fulfill security requirements. That's what this article is about.

MID server Key-based authentication - how? 

 There is an article on the docs that explain (kind of..) how the key-based authentication works and what you need to do to create the security header, the examples there are in java and python and I had to scratch my head to make it work on curl + bash.

We are going to use OpenSSL to create the authorisation token and pass it via curl during the POST request.

https://www.openssl.org/docs/man1.1.0/man3/HMAC.html

https://docs.servicenow.com/bundle/paris-it-operations-management/page/product/event-management/task/configure-midwebserver-extension-metric-data.html#configure-midwebserver-extension-for-metric-data

The data needs to be in a particular format so don't change the order or the variables, I spent 4 hours trying to figure out why it wasn't working and turned out I misplaced a variable, so stick to my example here:

Steps:

Enable the key-based authentication on the MID server collector and get the secret key

Then we will create the request header using a shell script.

Create a shell script that will be triggered by your event source - in my case Nagios and adjust the fields inside the JSON to your environment.

Notice the important bits are the definition of the variables and the curl header values Authorization and Date, without it the authentication won't work.

If you need to troubleshoot you can set curl to be "verbose" by adding a -v and you'll be able to see exactly what's being passed to ServiceNow Web Services

#!/bin/bash
path="/api/mid/em/inbound_event"
date=`date --utc +%FT%T.%3NZ`
method="application/json"
key="YOUR-SECRET-KEY-GOES-HERE"
msg="POST\n$method\n$date\n$path"
 
auth=`printf $msg | openssl dgst -sha1 -hmac $key -binary |base64`
 
curl -XPOST -H "Authorization:$auth" -H "Date:$date" -H "Content-Type: application/json" -d "
 {\"records\":[{ \"severity\": \" "$evt_sev" \", \"description\":\" "$evt_descr" \",\"metric_name\":\" "evt_metric_name" \",\"additional_info\":\"{\'add_info1\':\' "$4" \',\'add_info2\':\' "$5" \'}\" } ] }" http://MIDSERVERIP/api/mid/em/inbound_event?Transform=jsonv2
 

Example of manual run

 

I hope it helps!

Cesar

[Rob50]

Great Article! Nice Share, Cesar.