Last month's ISSA-NoVA meeting featured Dennis Heretick, CISO of the US Department of Justice. Mr. Heretick seemed like a sincere, devoted government employee, so I hope no one interprets the following remarks as a personal attack. Instead, I'd like to comment on the security mindset prevalent in the US government. Mr. Heretick's talk sharpened my thoughts on this matter.
Imagine a football (American-style) team that wants to measure their success during a particular season. Team management decides to measure the height and weight of each player. They time how fast the player runs the 40 yard dash. They note the college from which each player graduated. They collect many other statistics as well, then spend time debating which ones best indicate how successful the football team is. Should the center weigh over 300 pounds? Should the wide receivers have a shoe size of 11 or greater? Should players from the north-west be on the starting line-up? All of this seems perfectly rational to this team.
An outsider looks at the situation and says: "Check the scoreboard! You're down 42-7 and you have a 1-6 record. You guys are losers!"
In my opinion, this summarizes the mindset of US government information security managers.
Here are some examples from Mr. Heretick's talk. He showed a "dashboard" with various "metrics" that supposedly indicate improved DoJ security. The dashboard listed items like:
Imagine a dashboard, then, with all "green" for these items. They say absolutely nothing about the "score of the game."
How should the score be measured then? Here are a few ideas, which are neither mutually exclusive nor exceedingly well-thought-out:
These are just a few ideas, but the common theme is they relate to the actual question management should care about: are we compromised, and how easy is it for us to be compromised?
I explained my football analogy to Mr. Heretick and asked if he would adopt it. He replied that my metrics would discourage DoJ units from reporting incidents, and that reporting incidents was more important to him than anything else. This is ridiculous, and it indicates to me that organizations like this (and probably the whole government) need independent, Inspector General-style units that roam freely to assess networks and discover intruders.
In short, the style of "security" advocated by government managers seems to be "control-compliant." I prefer "field-assessed" security, although I would be happy to replace that term with something more descriptive. In the latest SANS NewsBites (link will work shortly) Alan Paller used the term "attack-based metrics," saying the following about the VA laptop fiasco: "if the VA security policies are imprecise and untestable, if the VA doesn't monitor attack-based metrics, and if there are no repercussions for employees who ignore the important policies, then this move [giving authority to CISOs] will have no impact at all."
PS: Mr. Heretick shared an interesting risk equation model. He uses the following to measure risk.
Imagine a football (American-style) team that wants to measure their success during a particular season. Team management decides to measure the height and weight of each player. They time how fast the player runs the 40 yard dash. They note the college from which each player graduated. They collect many other statistics as well, then spend time debating which ones best indicate how successful the football team is. Should the center weigh over 300 pounds? Should the wide receivers have a shoe size of 11 or greater? Should players from the north-west be on the starting line-up? All of this seems perfectly rational to this team.
An outsider looks at the situation and says: "Check the scoreboard! You're down 42-7 and you have a 1-6 record. You guys are losers!"
In my opinion, this summarizes the mindset of US government information security managers.
Here are some examples from Mr. Heretick's talk. He showed a "dashboard" with various "metrics" that supposedly indicate improved DoJ security. The dashboard listed items like:
- IRP Reporting: meaning Incident Response Plan reporting, i.e., does the DoJ unit have an incident response plan? This says nothing about the quality of the IRP.
- IRP Exercised: has the DoJ unit exercised its IRP? This says nothing about the effectiveness of the IRT in the exercise.
- CP Developed: meaning Contingency Plan developed, i.e, does the DoJ unit have a contingency plan should disaster strike? This also says nothing about the quality of the CP.
- CP Exercised: has the DoJ unit exercised its CP? Same story as the IRP.
Imagine a dashboard, then, with all "green" for these items. They say absolutely nothing about the "score of the game."
How should the score be measured then? Here are a few ideas, which are neither mutually exclusive nor exceedingly well-thought-out:
- Days since last compromise of type X: This is similar to a manufacturing plant's "days since an accident" report or a highway's "days since a fatality" report. For some sites this number may stay zero if the organization is always compromised. The higher the number, the better.
- System-days compromised: This looks at the number of systems compromised, and for how many days, during a specified period. The lower, the better.
- Time for a pen testing team of [low/high] skill with [internal/external] access to obtain unauthorized [unstealthy/stealthy] access to a specified asset using [public/custom] tools and [complete/zero] target knowledge: This is from my earlier penetration testing story.
These are just a few ideas, but the common theme is they relate to the actual question management should care about: are we compromised, and how easy is it for us to be compromised?
I explained my football analogy to Mr. Heretick and asked if he would adopt it. He replied that my metrics would discourage DoJ units from reporting incidents, and that reporting incidents was more important to him than anything else. This is ridiculous, and it indicates to me that organizations like this (and probably the whole government) need independent, Inspector General-style units that roam freely to assess networks and discover intruders.
In short, the style of "security" advocated by government managers seems to be "control-compliant." I prefer "field-assessed" security, although I would be happy to replace that term with something more descriptive. In the latest SANS NewsBites (link will work shortly) Alan Paller used the term "attack-based metrics," saying the following about the VA laptop fiasco: "if the VA security policies are imprecise and untestable, if the VA doesn't monitor attack-based metrics, and if there are no repercussions for employees who ignore the important policies, then this move [giving authority to CISOs] will have no impact at all."
PS: Mr. Heretick shared an interesting risk equation model. He uses the following to measure risk.
- Vulnerability is measured by assessing exploitability (0-5), along with countermeasure effectiveness (0-2). Total vulnerability is exploitability minus countermeasures.
- Threat is measured by assessing capability (1-2), history (1-2), gain (1-2), attributability (1-2), and detectability (1-2). Total threat is capability plus history plus gain minus attributability minus detectability.
- Significance (i.e., impact or cost) is measured by assessing loss of life (0 or 4), sensitivity (0 or 4), operational impact (0 or 2), and equipment loss (0 or 2). Total significance is loss plus op impact plus sensitivity plus equipment loss.
- Total risk is vulnerability times threat times significance, with < 6 very low, 6-18 low, 19-54 medium, 55-75 high, and >75 very high.
