Dealing with Security Instrumentation Failures

I noticed three interesting blog posts that address security instrumentation failures.

First, security software developer Charles Smutz posted Flushing Out Leaky Taps:

How many packets does your tapping infrastructure drop before ever reaching your network monitoring devices? How do you know?

I’ve seen too many environments where tapping problems have caused network monitoring tools to provide incorrect or incomplete results. Often these issues last for months or years without being discovered, if ever...

One thing to keep in mind when worrying about loss due to tapping is that you should probably solve, or at least quantify, any packet loss inside your network monitoring devices before you worry about packet loss in the taps. You need to have strong confidence in the accuracy of your network monitoring devices before you use data from them to debug loss by your taps. Remember, in most network monitoring systems there are multiple places where packet loss is reported...

I’m not going to discuss in detail the many things that can go wrong in getting packets from your network to a network monitoring tool... I will focus largely on the resulting symptoms and how to detect, and to some degree, quantify them. I’m going to focus on two very common cases: low volume packet loss and unidirectional (simplex) visibility.

Read Charles' post to learn ways he deals with these issues.

Next I'd like to point to this post by the West Point Information Technology Operations Center on Misconfiguration Issue of NSA SPAN Port:

Thanks to the input we have already received on the 2009 CDX dataset, we have identified an issue in the way the NSA switch was configured. Specifically, we believe the span port from which our capture node was placed was configured for unidirectional listening. This resulted in our capture node only "hearing" received traffic from the red cell.

Doh. This is a good reminder to test your captures, as Charles recommends!

Finally, Alec Waters discusses weaknesses in SIEMs in his post Si(EM)lent Witness:

[H]ow can we convince someone that the evidence we are presenting is a true and accurate account of a given event, especially in the case where there is little or no evidence from other sources...

D]idn’t I say that vendors went to great lengths to prevent tampering? They do, but these measures only protect the information on the device already. What if I can contaminate the evidence before it’s under the SIEM’s protection?

The bulk of the information received by an SIEM box comes over UDP, so it’s reasonably easy to spoof a sender’s IP address; this is usually the sole means at the SIEM’s disposal to determine the origin of the message. Also, the messages themselves (syslog, SNMP trap, netflow, etc.) have very little provenance – there’s little or no sender authentication or integrity checking.

Both of these mean it’s comparatively straightforward for an attacker to send, for example, a syslog message that appears to have come from a legitimate server when it’s actually come from somewhere else.

In short, we can’t be certain where the messages came from or that their content is genuine.

Read Alec's post for additional thoughts on the validity of messages sent to SIEMs.