We all want to know when something’s broken, but most of us go out of our way to screw ourselves over.
Just like that girlfriend you had with the Mazda 626, you have a dashboard with the check engine light that’s been on for 4 years. You know it says there’s an issue, but you checked the oil and water and everything was cool so you never did anything about it.
Here’s the problem. You got so used to seeing that check engine light, that when the air bags faulted or the cooling system failed, you didn’t realise something was wrong. Shit, that light’s been on since before you got the car. Well, the same applies for your computer systems. If you have an alert to email you when something isn’t running at an optimum level, chances are you made an email filter so you didn’t have to see it for the 4,000th time. If there’s a system like Solar Winds or Foglight, you already know that part of the country is always yellow because 1 switch in the stack was decommissioned 4 months ago and it should appear yellow on the fault map, so it’s no big deal.
Well done. You’ve trained yourself to ignore the signs that were meant to get your attention. Those things should always be green. If something shows up red, or yellow, or polka dot, or messages you about the existential crisis it’s currently experiencing, you’re supposed to freak out because…”SOMETHING IS GODDAMN BROKEN!” This means your IT support crew are about to get slammed and they better know what’s going on. If this little light is always green. 364.9 days a year, that mother fucker is green. Then the half a minute that thing turns red, someone will say “Well that doesn’t look right” instead of “That’s always like that, everything’s fine.”
Clear your alerts. Alarms should only happen when it’s important, status indicators should always be green, don’t train your staff to ignore the warning signs.