Hawke vs the Machine

One of the better episodes of an otherwise lacklustre third season of Airwolf (one of the best TV shows of the 1980s) was called Fortune Teller. Ace helicopter pilot Hawke finds himself face-to-face with an aircraft equipped with the Fortune Teller, a machine built to out-fly the world's best. The best line in the episode follows:

Archangel: They haven't built a machine yet that could replace a good pilot, Hawke.

Hawke: Let's hope so.

I thought of this truth when a colleague called yesterday. He plans to bring me to his customer's security operations center. The manager of the SOC is worried that the team is not detecting and responding to intrusions properly (or at all). One of the services my company provides is assessing and evaluating detection and response teams. So, I plan to visit this SOC and see what I can do to improve the situation.

I think I already know what the problem is. My friend told me the SOC is using an IDS/IPS solution you would all recognize, coupled with a SIM/SEM you would all recognize. If that is the extent of the tools available to the SOC analysts, I pretty much know what I am going to say before I even arrive.

The vast majority of security tools -- especially prevention-oriented tools -- are alert-centric. They are created by developers who assume they can understand attacks, code them into solutions, and have a customer deploy them properly and in the expected environment. The reality is this situation fails on a majority of counts, if not all counts. For example, who would ever have expected the Adobe Acrobat JavaScript Execution Bug as explained by my friend Nitesh Dhanjani?

The problem with a purely alert-centric tool is that it cannot handle all the rich varieties of conditions that it will encounter in a production network -- ever. As soon as it's updated something new appears. The only "solutions" that truly work are those that completely eliminate classes of vulnerabilities. (I think I've written about this before, and I know others have said much more than me on that issue.)

So what does an analyst do with an alert-centric tool, like an IDS/IPS? The analysis process looks like the figure at left. When you only have alerts to handle, you quickly reach a dead end. This is why I know the problem at the SOC before I get there. They're using alert-centric tools. The "detection" and "response" mission becomes "process tickets" generated when the alert-centric tool blinks red. The focus is on processing tickets, not finding intrusions. What else can these analysts do? Their tools are a straightjacket.

Network Security Monitoring (NSM) is different. Generating statistical, session, full content, and alert data gives analysts some place to go when they get an alert -- or when they want to do manual analysis. All of the best intrusions I've ever found were discovered manually. Sure, I've discovered compromises solely using IDS alerts, but all of the best subsequent investigation relied on NSM data.

With NSM, an alert is the beginning of the investigation, not the end, as shown at right. What does this have to do with Hawke and the Machine? An analyst using an alert-centric tool is only as good as the tool.

An analyst using NSM data can exceed the capabilities of the tool.

Here's a related thought. Years ago I told Snort creator Marty Roesch how I perform NSM analysis. He said "Richard, I wrote Snort so you don't have to look at packets." I replied, "Marty, I look at packets because you wrote Snort."

Why is a person better than a machine, in cases like this? The reason is a person is more creative and adaptable. A person can develop a feel for a network. A person can read about a new exploit Wednesday morning and mine NSM data for evidence of that attack faster than a vendor can issue some sort of updated detection or prevention method. Machines are good for providing hints about where to look, which is why I still use IDS'. However, I frequently do completely manual analysis with systems like Sguil, looking for interesting patterns or sessions that my IDS doesn't know.

I'll close with the note that some vendors are using what I call a "dumb is better" approach for generating alert data. For example, you can read about Tenable Security's Never Before Seen approach to identifying interesting events. This is exactly the sort of alert that is helpful, assuming I can then turn to other NSM data to learn more about it. Never Before Seen (NBS) is a great way to identify events which are new. You can apply this to many sorts of issues, which I will not address here but are mentioned by Ron Gula in his blog.

(Disclaimers: Before you argue that your system already does this, I know this approach is a lot older than the Tenable post. I know Marcus was already doing this well before he joined Tenable. The point is that I can read and learn about it thank to the Tenable blog. I can't point my clients to vague marketing materials on other vendor's sites and expect my clients to make sense of what they're reading. Thanks to the Tenable blog I can learn about systems like Tenable written with a technical angle and real product screenshots and so on.)

To conclude, I'm not an expert in any vendor tools. I'm an expert in using the right sorts of data to detect and eject intruders. I am not going to be any better at using some popular vendor's IDS/IPS than another person who understands the interface. There might be room for differentation based on the sorts of attacks reported or the services monitored, but not much.

At the end of the day, expertise grows from having the right forms of data available. I guarantee the SOC in this story doesn't have that data, so that will be recommendation zero.