Guidance to Survive Monitoring

While working in the monitoring field for a long time, here are some rules I try to follow when requirements go awry.

Rule #1: Only create an alert when human interaction is required

When you setup a monitoring, it tends to get noisy very quickly. The problem is, people want to know everything and want to monitor everything. You tend to build a system which sends you a lot of alarms and you will get alarm fatique. To get most out of your monitoring solution, you have to always keep in mind Rule #1. When you alert for something, ask yourself is it really necessary to wake some one up in the middle of the night. There is nothing more horrible than waking someone up and it is a false alert.