The Now Platform® Washington DC release is live. Watch now!

Help
cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
SlightlyLoony
Tera Contributor

find_real_file.pngOur product has long had a mechanism for detecting when a MID server was having a problem. This mechanism worked by watching for the once-every-ten-minutes status report that MID servers send; if this report didn't show up, then we knew there was a problem.

This mechanism was fine so far as it went, but it didn't go as far as we'd like it to. So we've now gone one step further:

Starting with Fall 2010 Stable 1 (which also means, of course, in the upcoming Winter 2011 release), we've implemented a synthetic transaction monitoring system for MID servers. Service-now.com instances now send a synthetic transaction (a "Heartbeat Probe") to every MID server, every 5 minutes. Heartbeat probes are designed to work exactly the same way that normal probes do, so that we're testing the same code paths used by normal probes. A heartbeat probe is sent by writing an output record to the ECC queue, which is then picked up by the MID servers when they query for work. This probe is processed by the MID servers just like any other probe, and the MID servers then send a response back to the instance. If the instance doesn't see the response, it knows that something is awry on the MID server, and the MID server is then marked as "Down".

In addition, system events are now fired when a MID server transitions from "Down" to "Up" (the mid_server.up event) or from "Up" to "Down" (the mid_server.down event). These events can be used to fire notifications or for script actions...

4 Comments