Simple Questions, Difficult Answers

Recently I had a discussion with one of the CISOs in my company. He asked a simple question:

"Can you tell me when something bad happens to any of my 100 servers?"

That's a very reasonable question. Don't get hung up on the wording. If it makes you feel better, replace "something bad happens to" with "an intruder compromises," or any other wording that conveys the question in a way you like.

It's a simple question, but the answer is surprisingly difficult. Let's consider the factors that affect answering this question.

• We need to identify the servers.




• We will almost certainly need IP addresses.




• How many IP addresses does each server have?


• What connectivity does each IP address provide?


• Are they IPv4, IPv6, both?


• Are they static or dynamic? (Servers should be static, but that is unfortunately not universal.)




• We will probably need hostnames.




• How many hostnames does each server have?


• What DNS entries exist?


• Extrapolate from the IP questions above to derive related hostname questions.




• We will need to identify server users and owners to separate authorized activity from unauthorized activity, if possible.




• What is the function and posture of each server?




• Is the server Internet-exposed? Internally exposed? A combination? Something different?


• How is the server used? What sort of services does it provide, at what load?


• What is considered normal server activity? Suspicious? Malicious?
  


• What data can we collect and analyze to detect intrusion?




• Can we see network traffic?




• Do we have instrumentation in place to collect data for the servers in question?


• Can we see network traffic involving each server interface?


• Is some or all of the traffic encrypted?


• Does the server use obscure protocols?


• What volume of data do we need to analyze?


• What retention period do we have for this data?


• What laws, regulations, or other restrictions affect collecting and analyzing this data?




• Can we collect host and application logs?




• Do we have instrumentation in place to collect data for the servers in question?


• Are the logs standard? Nonstandard? Obscure? Binary?


• Are the logs complete? Useful?


• What volume of data do we need to analyze?


• What retention period do we have for this data?


• What laws, regulations, or other restrictions affect collecting and analyzing this data?




• Is the collection and analysis process sufficient to determine when an intrusion occurs?




• Is the data sufficiently helpful?


• Are our analysts sufficiently trained?


• Do our tools expose the data for analysis in an efficient and effective manner?


• Do analysts have a point of contact for each server knowledgeable in the server's operations, such that the analyst can determine if activity is normal, suspicious, or malicious?

I'll stop there. I'm not totally satisfied with what I wrote, but you should have a sense of the difficulty associated with answering this CISO's question.

Furthermore, at what number is this process likely to yield results in your organization, and at what number will it fail? Can it be done for 1 server? 10? 100? 1,000? 10,000? 100,000?