Bankers: Welcome to Our World

Did you know that readers of this blog had a warning that the world's financial systems were ready to melt down? If you read my July 2007 (one month before the crisis began) post Are the Questions Sound?, you'll remember me disagreeing with a "major Wall Street bank" CISO for calling one of my Three Wise Men (and other security people) "so stupid" for not having the "five digit accuracy" to assess risk. That degree of arrogance was the warning that the financial sector didn't know what they were talking about.

The next month I posted Economist on the Peril of Models and then Wall Street Clowns and Their Models in September. Now I read a fascinating follow-up in last week's Economist titled Professionally Gloomy. I found these excerpts striking:

[R]isk managers are... aware that they are having to base their decisions on imperfect information. The crisis has underlined not just their importance but also their weaknesses.

Take value-at-risk (VAR), a measure of market risk developed by JPMorgan in the 1980s, which puts a number on the maximum amount of money a bank can expect to lose. VAR is a staple of the risk-management toolkit and is embedded in the new Basel 2 regime on capital adequacy. The trouble is that it is well-nigh useless at predicting catastrophe.

VAR typically estimates how bad things could get using data from the preceding three or four years, so it gets more sanguine the longer things go smoothly. Yet common sense suggests that the risk of a blow-up will increase, not diminish, the farther away one gets from the last one. In other words, VAR is programmed to instil complacency. Moreover, it acts as yet another amplifier when trouble does hit. Episodes of volatility send VAR spiking upwards, which triggers moves to sell, creating further volatility.

The second problem is that VAR captures how bad things can get 99% of the time, but the real trouble is caused by the outlying 1%, the “long tail” of risk. “Risk management is about the stuff you don't know that you don't know,” says Till Guldimann, one of the original architects of VAR. “VAR leads to the illusion that you can quantify all risks and therefore regulate them.” The degree of dislocation in the CDO market has shown how hard it is to quantify risk on these products.

Models still have their place: optimists expect them to be greatly improved now that a big crisis has helpfully provided loads of new data on stressed markets. Even so, there is now likely to be more emphasis on non-statistical ways of thinking about risk. That means being more rigorous about imagining what could go wrong and thinking through the effects...

However, stress-testing has imperfections of its own. For example, it can lead to lots of pointless discussions about the plausibility of particular scenarios. Miles Kennedy of PricewaterhouseCoopers, a consultancy, thinks it is better to start from a given loss ($1 billion, say) and then work backwards to think about what events might lead to that kind of hit.

Nor is stress-testing fail-safe. The unexpected, by definition, cannot be anticipated... (emphasis added)

VAR is one of the measures I am sure the Wall Street clown was invoking while dressing down Dan Geer. Too bad it failed. (If you disagree, read the whole article, and better yet the whole special report... these are just excerpts.)

When the Economist refers to "stress-testing," think "threat modeling," and use the warped sense of that term instead of the better phrase "attack modeling." Picture a room full of people imagining what could happen based on assumptions and fantasy instead of spending the time and resources to gather ground-truth evidence on assets and historical or ongoing attacks. Sound familiar?

The article continues:

Another big challenge for risk managers lies in the treatment of innovative products. New products do not just lack the historic data that feed models. They often also sit outside banks' central risk-management machinery, being run by people on individual spreadsheets until demand for them is proven. That makes it impossible to get an accurate picture of aggregate risk, even if individual risks are being managed well. “We have all the leaves on the tree but not the tree,” is the mournful summary of one risk manager. One solution is to keep new lines of business below certain trading limits until they are fully integrated into the risk system.

Keeping risks to a size that does not inflict intolerable damage if things go awry is another fundamental (some might say banal) lesson...“It is not acceptable [for a division] to have a position that wipes out its own earnings, let alone those of the entire firm.”

However, working out the size of the risks is less easy than it used to be. For one thing, the lines between different types of risk have become hopelessly blurred. Risk-management teams at banks have traditionally been divided into watertight compartments, with some people worrying about credit risk (the chances of default on loans, say), others about market risk (such as sudden price movements) and yet others about operational risks such as IT failures or rogue traders. (emphasis added)

Ok, stick with me here. References to "innovating products" should be easy enough. Think WLANs in the early part of this decade, iPhones now, and so on. Think local groups of users deploying their own gear outside of IT or security influence or knowledge.

For "keeping risks to a size," think about the security principle of isolation. For "the lines between different types of risk," think about unexpected or unplanned interactions between new applications. "I didn't think that opening a hole in our firewall to let DMZ servers do backups would allow an intruder to piggyback on that connection, straight into the internal LAN, compromising our entire firm!"

Finally:

There is an even bigger concern. Everyone is ready to listen to risk managers now, but the message is harder to transmit when the going is good. “Come the next boom we will have traders saying, 'that was eight months ago. Why are you dragging me down with all that?',” sighs one risk chief. To improve risk management through the cycle, deeper change is needed.

Oh, I thought security was a "business enabler" with a "positive ROI." On a directly applicable note, during and right after an incident everyone is very concerned with "security." Eight months later hardly anyone cares.

Bankers, welcome to our world.