What the operations people have to deal with in order to maintain effective firewalls

29 January 2010

Most companies have at least firewall, if not several. And since an established knowledge base exists to tap for issues and PCI DSS 1.1 and 1.2 are pretty clear cut, firewall management shouldn't be much of an issue.

If the firewall is a mature technology, then why do those who manage it feel like they're running a daycare overrun with little savages?

No one is going to suffer the brunt of managing the significant infrastructure change these regulations are bound to bring more than the security operations team, correct? Well, not really.

If your friendly neighborhood firewall guy (or gal) rolls into work late on a Monday morning sleep deprived and grouchy, cut him some slack. Here are some of the most common-yet-nerve-sizzling firewall snafus that have kept many an admin on a Friday-to-Sunday diet of fast food and Red Bull:

10.) The Saturday-at-midnight policy update process didn't go exactly as planned and he spent the rest of the weekend sorting through a bloated rule base to find out exactly what went wrong, and it ended up to be a slight overlap of rule 847 (meaning, 847 rules deep into the rule base) with rule 73.

9.) The network firewall rule base(s) have become so bloated that likely erroneous, obsolete and overlapping (or "shadowed") rules have caused unneeded risk or degraded hardware performance due to unnecessary processing and hardware drain (Yes, rule bloat is a big enough issue it warrants two of the top-10 spots).

8.) Monday's firewall changes didn't work when the polices were pushed out on Saturday because someone else's changes offset his and he had no idea who might have been making changes, what the change was, or why they made it.

7.) The last firewall guy had his own way of managing changes that is virtually indecipherable to those of everyone else, with no reference to the original request or business unit. And before he quit last month he accidentally cut off access to a mission-critical application when making a change.

6.) Permissive rules (rules with "ANY" and "ACCEPT," or even better, "ANY ANY ACCEPT")? If you want to be on good terms with auditors, then get rid of these. Rest assured, the security implications will soon enough deem them unacceptable. That means rules will need to be more specific and precise -- which could either be really good or really bad, depending on the size and nature of your existing base (see items 9 and 10).

5.) A user is requesting a change for a new rule, but the firewall guy can't tell if that traffic is already allowed, and has 30 other things to do so he simply adds the new rule with the intention of reviewing it later. Can you guess how the story ends?

4.) Process? Documentation? Authorization? Just how quickly does the CEO need network access?

3.) "You want a rule usage report for firewalls protecting the 50 Web servers in Sacramento? WHAT 50 Web servers in Sacramento?"

2.) "What do you mean the quarterly PCI reports are now MY responsibility?"

1.) It's 3 p.m. and his manager wants to know if all 200 firewalls (with at least 250 rules per firewall) from multiple vendors across six countries are in compliance with seven distinct regulations, two of which are regulations from different countries that contradict each other. And he wants to know by the end of the day.

Operations people are a noble lot. They deal first-hand with the never-ending network complexity, and because their triumphs are measured in disasters avoided, they are therefore rarely, if ever publicly acknowledged.

So, before you deny their request to attend Black Hat/DefCon this year, re-read this list for a reminder of how much they add to the organization. And then "Any, Any, Any, Accept" the request.