Last week I paid for and attended the
SANS Log Management Summit. I'd like to share a few thoughts about what I saw. First, I think Alan Paller did a great job as host. He kept the presentations moving and unflinchingly kept to his schedule. Talks started at 8 am, period. I thought his "yellow card" system for questions worked very well. (If you wanted to ask a question, you wrote it on a yellow card. SANS staff collected the cards then handed them to the speaker or Alan, who answered the question.) The system prevented the "speeches" one usually sees in large crowds with open microphones.
Alan started the conference by presenting his "faces of cybercrime" presentation, based on his testimony (
.pdf) in late 2005. He reminded the audience of the advice to learn hacking given by soon-to-be-executed Bali bomber
Imam Samudra. Alan claimed at least one organized crime group has moved two hackers to Africa and forced them to compromise targets "20 hours per day, 7 days per week," "for food." He reminded us of China's military doctrine of asymmetric warfare and repeated his earlier statements about
Titan Rain.
With regard to new information, Alan named three ways to help fight back against cybercriminals.
- Respond faster.
- Change metrics.
- Shift some responsibility to suppliers and integrators.
I like this approach. For a few years Alan was beating the drum for #3, and for the last year he's been working #2. I like #1 alot, since I am an incident responder.
With regard to metrics, Alan likes the term "attack-based metrics," and uses phrases like "measuring what we need to do," "how are we being compromised," and "how can we defend ourselves." He noted the Air Force is "lead" for this approach. Alan said measuring report writing (e.g., FISMA) is a waste of time. Some example metrics he noted were:
- Can an incident of successful spear phishing be detected in 30 minutes or less?
- What percentage of employees fall victim to a spear phishing test?
Alan mentioned using privilege user monitoring as a means to counter insiders, although he also said "The insider threat is baloney," until an outsider becomes an insider. Alan concluded his talk by sounding optimistic about SCADA procurement standards. I have no dog in that fight, but I recommend reading Dale Peterson's
SCADA Blog for all things SCADA.
Lawyer Ben Wright spoke next about log management and legal issues. I really wanted to see this talk. Ben said logs can indicate control, thereby preventing claims of negligence and offering evidence to resolve disputes. His most interesting point was that
records of log review are more important than the logs themselves. In other words, in the legal eye, it's better to make a note every time you review your logs than it is to retain those logs. Email is far more important to retain, since firms are fined millions for failing to keep email.
Ben noted that HIPAA Security Rule 45 CFR 164.308(a)(1)(ii)(D) mentions logs, as does NIST SP 800-66 and the PCI Standard of January 2005, part 10.6. However, review of logs, not retention of logs, is critical. Ben explained negligence law, where the standard is "reasonableness." He said that if a company writes a policy stating "We will do X," and they fail to perform X, it's easy for a jury to find the company negligent. That means lawyers will recommend companies write policies saying "We
may do X." The audience had a hard time handling that idea, since it's a lawyer's point of view and not that of an auditor or security person.
Ben provided three suggestions regarding log management.
- Policy should stress preferences, not statements saying "We will do X."
- Keep records of the fact you reviewed logs.
- Only a company's full audit committee should know about all monitoring methods -- neither employees nor the CEO should know what is watched or stored.
Ben liked promoting "mystery" in the workplace to keep people on the straight and narrow. It's sounds like a great deterrence tool, but a little draconian for me.
Next followed the first of several presentations by users of vendor log management solutions. Here I should mention that companies formally represented at the Summit were
Arcsight,
Network Intelligence,
LogLogic,
Prism Microsystems, and
SenSage. Yes, that's it -- no
Tenable or other big players. More importantly, the vendors chose all of the customers who presented their product experiences. Not surprisingly, all gave glowing opinions. This was really disappointing. The only opposing point of view came from Stephen Northcutt at the Summit's end, who reported a majority of log management users are dissatisfied with their solutions! Although
TriGeo did not speak on any panels, some customers reported using their product.
I won't mention individual user reports by name, since most weren't that helpful. Here's a clue that they lacked the detail I (and other attendees) expected: when Alan has to ask, at the end of a presentation, "So what product do you use?", you know the briefer didn't share the details that attendees wanted to hear.
Here are a few data points though, collected from all of the customer reports and hence out-of-order chronologically. Chad Mead from JPMorgan Chase said his shop, with 210,000 desktops, 40,000 servers, 400 NIDS, 900 firewalls, 81 mainframe LPARs, and over 1 million network ports, produces over 150,000 events per second. He operates two security management centers with five people operating the log management solution and 65 analyzing logs. Wow, that's what I like to hear! His security team owns the logging infrastructure, and the device owners own the log feeds. This was a common refrain.
Mark Olsen from CareGroup Healthcare System said all emergency room records are transmitted electronically "to Atlanta," by which I guess he meant to the
CDC. This is a measure to identify
Bird Flu outbreaks. That sounds like something from the X-Files. He also said that while HIPAA enforcement actions thus far have been few and far between ("19,000 violations in 2005, 7 selected for prosecution"), expect that to change in 2007.
Chris Calabrese said a word about the
Security Issues in Network Event Logging (syslog) IETF working group. Mike Poor said his company deploys
LaBrea Tar Pits on Soekris boxes inside companies to watch for unexpected traffic. Jay Leak from Nokia justified his log management project by realizing the waste caused by his company making over 1,000 requests for log data each year, with each request taking over 4 hours and half of those never being resolved. Keith Fricke said "IPS is completely misnamed." He uses his IPS to block outbound malicious traffic from compromised internal systems! What's misnamed about that -- he's trying to "prevent" someone else from being compromised.
Chris Brenton and Mike Poor next unveiled the
SANS Top 5 Essential Log Reports (.pdf). This appears to have gotten zero news coverage, which I don't understand. Here they are, meaning what you should look for while reviewing logs:
- Attempts to Gain Access through Existing Accounts
- Failed File or Resource Access Attempts
- Unauthorized Changes to Users, Groups and Services
- Systems Most Vulnerable to Attack
- Suspicious or Unauthorized Network Traffic Patterns
I found it funny that I wrote a whole book (
Extrusion Detection) about #5, but the term "extrusion detection" isn't mentioned in the SANS .pdf. They did mention the
LogAnalysis.org mailing list, which I should probably start reading.
The end of day one concluded with a "vendor shoot-out," where the five vendors I named earlier made pitches and argued with each other. It seemed more hostile than the Real World Intrusion Detection Workshop I attended four years ago with Bamm Visscher, right before I left Ball Aerospace to join Foundstone (sorry Bamm!).
I liked that LogLogic's Anton Chuvakin (I know you're reading) prefers to collect everything from a log source and let the centralized solution handle presenting useful information. He said "you never know what might be important," which is the foundation for my NSM approach. During a nice "lunch and learn" Anton also said the biggest obstacle to building one's own log management solution is keeping pace with changing log formats. I had never imagined that problem.
One really astute question-asker wondered why three vendors showed Lehman Brothers as a client on each of their presentations. Each vendor stated their case, with Network Intelligence saying their product did the real log collecting, after which it forwarded a feed to ArcSight.
Day two started with Mike Poor discussing network early warning systems (NEWS). He said the famous
Dutch botnet wasn't 1.5 million victims strong -- it was more like 5.1+ million systems. Wow. Mike reminded us of the
Dabber worm which attacked Sasser victims. He said
Dshield collects logs from 40,000 sensors watching 500,000 IPs. Mike spent some time discussing
DNS cache poisoning and
SANS'
role.
By now you might be wondering, "where is the news on NEWS?" (Oh, too funny.) To be honest, I didn't hear much of anything new. In fact, there wasn't much "early warning" to speak of. If you deploy the systems Mike mentioned in his talk, you aren't learning of an attack before it happens -- you're learning afterwards. I suppose if you are at the front of the victim list and you share what you know, you're the NEWS for others! Mike did name the
Chinese Honeynet Project as the source of some interesting tools. I might try those.
I've already noted the parts of the second day worth repeating, so that ends day 2.
Day 3 consisted of classes by
Randy Franklin Smith on Windows Event Logs and
Chris Brenton on building your own solution. In short, Randy is a Windows EVL guru and Chris is a great instructor. These two classes probably saved the entire three days for me, since they at least had some of the detail I expected from the previous two days. Randy's class really emphasized that understanding Windows EVL is an art in itself. It takes a lot of work to make sense of them. Randy said the appointment of
Eric Fitzgerald as a sort of Windows EVL czar will help unify the system, at the expense of changing everything once Vista appears. Chris reminded me to try programs like
Simple Event Correlator,
Privateye, and
Syslog NG.
Overall, I think I got my money's worth from the Summit. I do not do log management as a primary task, so I was exposed to a whole new world of challenges. I met some interesting people and I got to attend the
CDX briefing and Vendor Expo. Both yielded contacts that might result in future blog posts.
I predict that three years from now people will still be disgusted with their log management and security incident management "solutions," and will be looking for "the next big thing." It's already happened with firewalls, IDS, IPS, and now LM/SIM.
What did you think of the Summit?
At left is the juggernaut of the security book market -- Hacking Exposed. I mention this book because it came up in a discussion I had with someone in the publishing community today. She reported that the state of the security book market is somewhat weak. She worried that Hacking Exposed (published in late 1999) might have created a "bubble" in the security book market, and the bubble is now deflating.
What is that? It can't be a new 
This morning I was reading the third edition of 
I received a review copy of 
