Risk and Metrics

I ran across some thought-provoking articles in the April 2006 CIO Magazine. The editor's introduction summarizes a major problem with calculating IT spending:

As sophisticated as the technology and its countless uses have become, all too often the benchmark used to determine the proper level of an enterprise’s IT spending is alarmingly simplistic: the percentage of overall revenue for which IT accounts...

Benchmarking IT spending as a percentage of revenue is a truly useless metric. Unfortunately, according to Koch [mentioned next], it remains the most popular way to evaluate IT spending, and also unfortunately (as most of you already know), it doesn’t say anything about how effective or productive your spending is. Even more unfortunately, benchmarking by percentage of revenue casts IT in the role of a cost to be controlled, defining success simply as lowering the percentage over time.

This is a really amazing insight. How many of you see progress in security management through the eyes of reducing spending to zero? The "Koch" mention refers to the article The Metrics Trap...And How to Avoid It by Christopher Koch. As you might guess there is really no simplistic way to solve this problem. Koch's article includes gems like the following, though:

Joe Drouin... found that [his] company was spending less on IT as a percent of overall revenue than the industry average, which was about 1.5 to 2 percent.

Not one to look a gift horse in the mouth, Drouin played the metric for everything it was worth, highlighting it in every PowerPoint presentation he could during his first year as CIO...

At one point, the CEO, who believed that inexpensive IT was good IT, joked that he expected to see Drouin and his staff outfitted with T-shirts that had the percentage stamped across their chests in big, block numbers...

In this zero sum game, success is defined simply as lowering the percentage over time. "It's not clear how low it should go," says Drouin. "Joking with the CEO, I said, 'In your mind it should be zero.' We had a good laugh, but at what point do we decide it's at the right level and you don't drive it down further?"

That CEO's attitude disgusts me. Would you expect him to do the same for the human resources department? They don't bring in any customer revenue. How about finance and accounting? Now that creative bookkeeping can put the CEO is jail, that isn't a place that brings in customer revenue either. Yet, neither "cost center" is expected to reduce its percentage of overal revenue to zero.

At least as far as security goes, the inability to see the value of security spending relates to management's inability to perceive the risk of being exposed and vulnerable. I came across this insight in a recent issue of the Economist, featuring the article The New Paternalism (subscription probably required):

This acute sensitivity to losses is not the only bias behaviouralists have discovered. People also have great difficulty understanding risks. The weight a person gives to a scenario—flood, fire, winning the lottery—should depend on its likelihood. In fact, it depends on how easily it can be envisaged. People will pay more for air-travel insurance against "terrorist acts" than against death from "all possible causes."

Canny governments can work with the grain of this psychology. The grisly campaigns against smoking aim to put the dangers firmly in people's minds; to turn a statistical risk into a visceral image. They have been effective, perhaps too effective. There is some evidence that people now overestimate the risks of smoking. (emphasis added)

In other words, management cannot imagine the destruction caused by security incidents. It is impossible for them to envisage an incident causing their company losing market share, intellectual property, or its ability to provide services. As a result, they base their decisions on laws, regulations, and what their peers are doing.

This explains the resources poured into worm defense a few years ago. When management's own computers are affected, when they see worm reporting on CNN, when a worm is the discussion over lunch -- they start to take the problem seriously. When a stealthy intruder has lodged himself inside a company, management has no clue how to handle the situation. In fact, most management has no clue how to handle existing rogue employees now. They turn to platitudes like "we trust our employees" because they can't fathom why someone would turn against their beloved company. After all, management has been treated really well!

I don't think spending-related metrics are of much use. Performance-related metrics are the only ones which I think have some value. Drilling network security operations teams (preventers, intrusion detectors, incident responders, etc.) to see if they stop, identify, and remove controlled threat simulators (vulnerability assessors, pen testers and red teams) is the best way to see if your money is being well spent.