Alert Suppression During Maintenance Windows and Change Implementations

[joeykusky]

Alert Suppression During Maintenance Windows and Change Implementations

Most organizations have gotten proficient at monitoring their critical services and infrastructure.  In the majority of the environments I encounter there are anywhere from 3-10 monitoring tools generating events tools for monitoring user experience at the application level, synthetic user transactions, Operating System monitoring, database monitoring, network monitoring, storage monitoring…etc.  With all of this monitoring capacity, many companies are struggling to operationalize all of the generated events.  Event Management becomes that operationalization layer.  In this post we are going to focus on change and maintenance event suppression functionality in the Event Management module and how it helps to operationalize maintenance windows.

For the scenarios in this blog, I have made a few assumptions:

1. There is a good CMDB with relationship data from Discovery.
2. Event Management is configured to receive alerts from your monitoring systems.
3. You have some best practices in place to add CIs to your change requests.

Alert Suppression During Planned Changes

When a change is within its change window and is moved into the implementation phase, the CI for that change is automatically moved into maintenance.  Moving a CI into maintenance adds it to the table em_impact_maint_ci.  If you want to watch this behavior, you can open the table to see which CIs are currently in maintenance.  If you are not familiar with this process, you can open up the table in a new tab by typing em_impact_maint_ci.LIST in your navigator.

 

Once a CI has been moved into maintenance, the default behavior is to suppress any alerts for that CI.  If you look at the default filter in the alert console, you can see that CIs in maintenance are filtered out.  Open up the alert console and take a look at the filter.

 

While in maintenance, any events generated and sent to Event Management will be stored in the appropriate tables; however, alert rules will not be processed for those alerts while the CI is in maintenance.  When the change is closed, the CI will be removed from maintenance, and rules will be processed if any new events are received.   

Taking a deeper look, you can open up Maintenance rules under event management:

 

There are two maintenance rules by default.  The first is the change scenario described above.  The second is to eliminate events from a CI marked as retired.

Let’s look at creating a custom rule.  We will start with a simple scenario.  There is a company-wide DR test going on over the weekend.  During the cutover, the standby/DR systems will need to be rebooted several times to get the failover completed.  There has been a request to suppress alerts for the DR systems during the DR test.

Under Maintenance Rules, I create a new rule with the following conditions:

The scheduled job for maintenance impact calculation runs every minute (you can see it under scheduled jobs with the job name of Event Management – Maintenance Calculator).  When this job runs the CIs with an operational status of DR Standby will enter maintenance mode.  When the weekend is over, you can set the rule to inactive, and the job will remove the CIs from maintenance when the job executes again. 

Hopefully the information in this post helps provide some clarity on maintenance rules and how they can be used to help operationalize event data and drive efficiency.   Look out for more blogs on this and other topics.