Network Security Monitoring Case Study

I received the following email from a friend. He agreed to share his story in exchange for commentary from me and fellow blog readers. I've added comments inline.

I'm now responsible for cleaning up a mid sized company perimeter defences... To be honest, at first glance the task is a daunting one, thousands of users, dozens of dis-separate systems and gigabits of network traffic plus as part of the enterprise support team, I have other projects and duties to deliver on.

I managed to get time with the systems architect and run through a number of questions stolen from your books and other smart folks on the history and state of affairs of my new domain. My questions were answered or put in to context except for one.

"Why don't you have any monitoring tools in place on the edge and perimeter systems?"

The answer I received wasn't what I expected. He simple stated that no-one had the time or energy to take on this massive task. He was very pro about having monitoring, but bluntly realistic on the amount of time monitoring and response took up. The business has other priorities for the IT teams.

It's not that the company isn't security aware, they have good policies, skilled staff, harden systems and layer defences, but no monitoring.

I was stuck with the thought "If I don't understand what's happening on the network, how can I know what's right or wrong, what's normal or abnormal?"

This is exactly right. My friend's co-workers are probably practicing management by belief instead of management by fact. If you do not know for sure what is happening inside your company, how can you make informed decisions? Failure will occur when your luck runs out because you are not spending the time and resources necessary to acquire and maintain digital situational awareness.

I gathered all the documentation, policies and firewall rules together and tried to make sense of them. I re-read Extrusion Detection on my commute in to help me frame a plan of attack. I got all the basic system data of every system, check time and date stamps were all in sync and cleaned up the network maps.

Then, as silly as this seems, the obvious answer came to me on where to start: at the internal firewall internal interface.

I started to review the drop logs of the firewall for that interface, reasoning that if there is a problem on the network, it should more apparent in those logs than anywhere else.

This is also exactly right. There is no need to start buying or building new tools if you are not leveraging your existing data sources or enabling data sources not yet activated.

Those reviews provided some startling results, fortunately for us, the problems were only mis-configured systems attempting outbound access. Mind you, they were generating considerable amounts of traffic and some over already congested WAN links.

Then I laid out the ingress and egress rule sets to find out the motives and owners of those rules in the firewalls and started to peel back those unused or unneeded rule sets, while monitoring the drop logs. All with management approval and sign off, of course!

This is another great move -- a stride towards simplification.

I'm now moving in to the next stage of working with other teams to clean up their problem systems. Some are going to be harder than others as I've found VAX's, AIX, Sun systems and others bouncing off the rules. I haven't seen these types of systems in a long time and as a simple Windows administrator, I have to find time and the right people to plug these holes.

I have a very long way to go, only focused on a very small section of the entire network and no certain path forward on how to apply a reasonable monitoring system - yet :-)

It's a challenge I'm looking forward to and one I'm sure I'll learn from.

The only comment haunting me is the not having any real time allotted to doing monitoring work. I'm going to talk with management and a surprisingly security literate CIO on if this can be addressed.

I know that the conversations have to be based on preventing/mitigating business risk, but believe it's going to be a hard sell with the very long list of other projects being pushed on the team as priority.

If you have any words of advice or useful pointers on convincing them the time is well spent, I'd be grateful.

So this is the major question. How do you convince management or other functional areas that monitoring is important? It sounds to me like my friend has already scored some wins by freeing bandwidth used by misconfigured systems, simplifying firewall rules, and examining individual problematic hosts.

It's important to remember that there is no return on security investment. Security is a cost center that exists to prevent or reduce loss. It is not financially correct to believe you are "earning" a "return" by spending time and money to avoid a loss.

If I need to spend $1000 to hire a guard to protect my $10000 taxi, I am not earning a return on my investment -- I am preventing the theft of my taxi. If I invest that $1000 in a ticketing and GPS system that makes me more productive ferrying passengers (perhaps increasing my dollars per hour worked), then I have enjoyed a ROI once my $1000 expense is covered.

This is not to say that money should not be spent on security. Rather, time and money should be balanced against the perceived risk. The same taxi driver will spend money on insurance, and may indeed need to spend $1000 on a guard if the protection delivered is seen to be necessary. However, the guard does not make the taxi driver more productive. Security is a "business enabler" here but it does not deliver "ROSI."

I'm not a big fan of FUD but something may be happening to executive perception of digital security. I am starting to hear questions like "How do I know that asset X is safe?" or "How do I know that asset Y is not being attacked?" (Answers: "safe" is relative, and Y is being attacked if it's accessible to anyone.)

I don't have any quick answers for my friend, which is why I'm posting this story. What are you doing to convince management that monitoring is necessary? I personally plan to do exactly what my friend did, namely starting with existing assets and showing quick wins to build momentum for more extensive visibility initiatives. You?