Network security blunders - how to recover

We’ve all made one in our career - I’m talking about that blunder you thought would cost you your job. My first major blunder was rebooting all the campus router pairs at one time, not one by one, all at once. I had written a script to install a security update on all the routers and reboot them all one by one…. or so I thought. Turns out my script had an error and didn’t wait between routers.

I thought for sure I was fired, but thankfully, I wasn’t. What could have been a major disaster turned out it was a great learning experience for everyone involved. We all learned a little about crisis management, and as everything came back up online, my boss took a few hours to teach me how to verify the network was working properly.

The good news is that most of the time, our blunders are not so severe. The bad news is that oftentimes they are not instantly noticeable, which means they can remain undetected for weeks, months, or even years until one day they either cause an outage or an auditor calls us on them. On the network security front, firewall management is one area where a simple mistake regarding a rule or configuration change can come back to haunt you – here are some of the most common errors...