Five Thoughts on Incident Response

Speaking of incidents, I thought it might be interesting to share a few brief observations based on incidents I've worked recently. Please remember this is a blog post. If you expect thorough explanations of these points with footnotes, historical references, arguments to the contrary expertly swept aside, etc., please wait for a future book! :)

1. Anti-Virus is (or should not be) an incident response tool. I am baffled when I see machines compromised, and the owners think a magic signature from their AV vendor is going to save the day. In this day and age intruders who gain kernel level control of a host often disable AV and will not give up the fight so easily. My second point relates to this one.


2. Your default incident recovery strategy should be to rebuild from scratch. By scratch I mean reinstallation from original trusted media and re-installation of applications and data.
   
   Today, in 2007, I am still comfortable saying that existing hardware can usually be trusted, without evidence to the contrary, as a platform for reinstallation. This is one year after I saw John Heasman discuss PCI rootkits (.pdf). I was lucky enough to spend a few hours chatting with John and fellow NGS Software guru David Litchfield after John's talk on firmware rootkits (.pdf). John's talks indicate that the day is coming when even hardware that hosted a compromised OS will eventually not be trustworthy.
   
   One day I will advise clients to treat an incident zone as if a total physical loss has occurred and new platforms have to be available for hosting a reinstallation. If you doubt me now, wait for the post in a few years where I link back to this point. In brief, treat an incident like a disaster, not a nuisance. Otherwise, you will be perpetually compromised.


3. SPAN ports should not be the default traffic access option. I cannot tell you how much time, effort, and frustration has accompanied the use (or attempted use) of SPAN ports in incident response situations.
   
   
   
   • "The SPAN port is already used."
   
   
   • "The SPAN port can't do that." (although it probably can, the network engineer either doesn't know how to set it up or doesn't want it configured to help the security team)
     
   • "Do you see anything? No? Try now. No? Try now. No?"
   
   
   • "You only see half the traffic? Wait, try this. Now you see double? Ok, try now."
   
   
   
   For Pete's sake, buy a tap, put it in the proper place, and stand back while the packets are collected properly.


4. A Linux live CD is not a substitute for a real network security monitoring platform. Upon realizing that Cisco MARS is not an incident response solution, I was desperate to collect some form of useful network-centric data at one client site. In a last-ditch attempt to salvage a bad situation my on-site colleague deployed a Network Security Toolkit live CD on top of a box previously running Linux natively. I was able to SSH into it, mount the local filesystem, and start writing packets to the hard drive using Tshark's ring buffer. This is absolutely making the best out of a mess, which is standard incident response behavior.
   
   I would ask anyone who turns to a live CD for their monitoring needs to avoid the temptation to think Snort on a live CD on spare, old hardware is anything like Snort on properly sized, configured, deployed hardware. Furthermore, Snort != monitoring. Live CDs are fine for assessment work but they are nearly worthless for packet capture. Needless to say I was able to talk my colleague through a FreeBSD installation and was soon collecting data in a somewhat better environment.


5. When you are compromised, you are probably not facing a zero-day exploit unique to you and not capable of being prevented. When you are compromised you're most likely suffering from some fairly modern variant of attack code that nevertheless contains exploits dating back to 2002. For some reason people seem to feel better if they think the incident is caused by some uber elite intruder who saved up his killer 0-day just for their enterprise. In reality someone probably connected an infected laptop physically to the network, or via VPN, and found a way to get a worm or other malware to the segment of the enterprise running "production" machines that never get patched.

Do you have any IR stories or lessons to share? Please post them as comments or write on your blog, then post a link here as a comment. Thank you.