Not Just Clowns, But Criminals

It turns out my April post Clowns Base Key Financial Rate on Feelings, Not Data was too generous. I cited an Economist story which outlined how LIBOR rates — and the returns on $360 trillion of financial contracts related to them, five times global GDP — are based on best guesses rather than hard data.

I continue to cover this story because the financial industry routinely scoffs at the "risk management" practices of non-financials, as I wrote in 2007.

It turns out that these clowns are actually malicious, as reported in Lies, damn lies, and LIBOR: Barclays, Diamond, and a devalued benchmark:

A pattern of deception extending over a period of years. A flouting of the law to profit at the expense of others on three different continents. And a belief that the rules did not apply to them.

No, not the latest mafia family to be taken down by a special prosecutor. But Barclays PLC, the sprawling British banking group that recently paid a $450 million fine for seeking to rig LIBOR, a benchmark interest rate used to value trillions of dollars of investments...

In simple English, that's an assertion that Barclay's employees on at least three continents spent years lying in order to fix benchmark interest rates that help determine the value of about $10 trillion of global debt and $350 trillion in derivatives, mostly swap contracts.

For instance "Barclays based its LIBOR submissions for US Dollar... on the requests of Barclay's swaps traders, including former Barclays swaps traders, who were attempting to affect the official published LIBOR, in order to benefit Barclays' derivatives trading positions."

The daily LIBOR fixing by the BBA is based on self-reporting from major financial institutions on the cost of short-term unsecured borrowing. Though it's based on the honor system (a regulatory failure if ever there was one) that daily fixing is used as a benchmark that effects the prices of swaps and debt instruments in dollars, pounds, yen, and euros. So if you can fiddle the LIBOR number, you can manipulate markets to your advantage.

I expect more banks to be named in the coming days and weeks.

It's easy to win at "risk management" if you cheat.

Tweet

Read More

Clowns Base Key Financial Rate on Feelings, Not Data

If you've been reading this blog for a while, you know I don't think very highly of mathematical valuations of "risk." I think even less highly of the clowns in the financial sector who call security professionals "stupid" because we can't match their "five digit accuracy" for risk valuation. We all know how well those "five digit" models worked out. (And as you see from the last link, I was calling their bluff in 2007 before the markets imploded.)

Catching up on last week's Economist this morning I found another example of financial buffoonery that boggles the mind. The article is online: Inter-bank interest rates; Cleaning up LIBOR -- A benchmark which matters to everyone needs fixing:

It is among the most important prices in finance. So allegations that LIBOR (the London inter-bank offered rate) has been manipulated are a serious worry.

LIBOR is meant to be a measure of banks’ own borrowing costs, and is used as the foundation for a host of other interest rates. Everyone is affected by LIBOR: it influences the payments made on mortgages and personal loans, and those received on investments and pensions.

Given its importance, the way LIBOR is calculated is astonishingly flimsy. LIBOR rates are needed, every day, for 15 different borrowing maturities in ten different currencies. But hard data on banks’ borrowing costs are not available every day, and this is the root of the LIBOR problem.

The British Bankers’ Association (BBA), responsible for LIBOR, gets around it by asking banks, each day, what they feel they should pay to borrow.

So LIBOR rates—and the returns on $360 trillion of financial contracts related to them, five times global GDP—are based on best guesses rather than hard data.

Let that sink in and forget about what you learned in business school or economics classes. LIBOR isn't based on actual rates; it's based on feelings!

The next part of the article talks about suspicions that banks manipulate this broken process to the advantage of the financial sector.

The remainder offers recommendations for improvement:

[T]he BBA should revamp LIBOR to ensure it is simple, transparent and accountable. These principles suggest LIBOR should be based on actual inter-bank lending, with any gaps filled in with the help of statistical techniques. Banks’ own guesses should be used as a last resort, not the first.

And regulators should collect data that could help spot LIBOR cheats: banks should be required to submit information on other banks’ borrowing costs, as well as their own. Regulators could cross-check submissions against hard data on banking-sector risk, and publicly report LIBOR abusers.

Keep this system in mind the next time a so-called "master of the universe" offers a lecture on measuring risk in digital security.

Tweet

Read More

Thor vs Clown

It started with this post by M.D.Mufambisi to the pen-list list:

Im designing an SMS baking application but i need to research on the security risks involved first... What are the risks around this application? How are such applications normally subverted? Are there any case studies someone can point me to?

After a few responses, Craig Wright chimed in:

The solution needs to be based on risk.

Where a system uses an SMS response with a separate system (such as a web page), the probability that the banking user is compromised and a fraud is committed, P(Compromise), can be calculated as:

P(Compromise) = P(C.SMS) x P(C.PIN)

Where: P(C.SMS) is the probability of compromising the SMS function and P(C.PIN) is the compromise of the user authentication method

Craig followed up with a blog post:

Many people feel that it is not feasible to model risk quantitatively. This of course is blatantly false. In the past, many of the calculations have been computationally infeasible at worst and economically costly at best. This has changed. The large volumes of computational power that is available coupled with novel stochastic methods has resulted in an efficiently viable means of calculating risk quantitatively with a high degree of accuracy. This can be measured as a function of time (as survival time), finance (or monetary value) or any number of other processes...

Tim Mullen, a guy who I first met in 2002 teaching at Black Hat, responded on full-disclosure:

I'm looping in the FD list because often my replies don't make it to Pen-Test, and this has hit a nerve with me.

I've looked over your post...

Once I was able to get past the overwhelming egoism and self-substantiating claims of your contributions to the industry, I arrived at the conclusion that the only portion of the aforementioned page that is not complete drivel and even laughable to anyone who has actually worked towards ascertaining actual risk in production environments, is where you describe your own words as "ravings..."

I'm fine with you sitting back and gloating about the Security Hero award you got from Northcutt, but when I see that you are actually contributing to ANY level of Critical Infrastructure Protection, it makes me fear for anyone who might be counting on your presumed skillset to actually make intelligent decisions about risk where human safety is at stake.

Your "risk formula" is ridiculous. What number would your formula have yielded 2 weeks before SQL Slammer was released? Where is the variable for unpatched systems? What number do we plug in for malicious employee factorization? More importantly, where is the calculation for self absorbed snake-oil selling academics with no real experience using their calculator to come up with magic numbers that represent the risk of a nuclear power plant being hacked?

Since you are (self-described) as "currently the only GIAC GSE (Compliance) holder globally and the most highly accredited Global Information Security Professional" and thus (presumably, if only in your mind) the greatest security mind in the world, how about accepting a challenge to an open debate on the subject at Defcon? People like you are dangerous and need to be exposed before someone in a position of power actually believes that you know what you are talking about. Bring your abacus.

Craig then responded with some sort of monetary challenge, and Tim and Craig are now debating how to arrange that.

If you want history on why I consider model = clown, please check out the posts on my clown tag.

When I read

"In the past, many of the calculations have been computationally infeasible at worst and economically costly at best. This has changed. The large volumes of computational power that is available coupled with novel stochastic methods has resulted in an efficiently viable means of calculating risk quantitatively with a high degree of accuracy."

it is clear to me Craig is pretty well disconnected from reality. Did we not just suffer a global recession exacerbated by clowns who thought they could model risk "with a high degree of accuracy"?

Read More

More on Weaknesses of Models

I read the following in the Economist;

Edmund Phelps, who won the Nobel prize for economics in 2006, is highly critical of today’s financial services.

"Risk-assessment and risk-management models were never well founded," he says. "There was a mystique to the idea that market participants knew the price to put on this or that risk.

But it is impossible to imagine that such a complex system could be understood in such detail and with such amazing correctness... the requirements for information... have gone beyond our abilities to gather it."

This is absolutely the problem I mentioned in Are the Questions Sound? and Wall Street Clowns and Their Models. Phelps could easily be describing information security models.

---

Richard Bejtlich is teaching new classes in DC and Europe in 2009. Register by 1 Jan and 1 Feb, respectively, for the best rates.

Read More

Bankers: Welcome to Our World

Did you know that readers of this blog had a warning that the world's financial systems were ready to melt down? If you read my July 2007 (one month before the crisis began) post Are the Questions Sound?, you'll remember me disagreeing with a "major Wall Street bank" CISO for calling one of my Three Wise Men (and other security people) "so stupid" for not having the "five digit accuracy" to assess risk. That degree of arrogance was the warning that the financial sector didn't know what they were talking about.

The next month I posted Economist on the Peril of Models and then Wall Street Clowns and Their Models in September. Now I read a fascinating follow-up in last week's Economist titled Professionally Gloomy. I found these excerpts striking:

[R]isk managers are... aware that they are having to base their decisions on imperfect information. The crisis has underlined not just their importance but also their weaknesses.

Take value-at-risk (VAR), a measure of market risk developed by JPMorgan in the 1980s, which puts a number on the maximum amount of money a bank can expect to lose. VAR is a staple of the risk-management toolkit and is embedded in the new Basel 2 regime on capital adequacy. The trouble is that it is well-nigh useless at predicting catastrophe.

VAR typically estimates how bad things could get using data from the preceding three or four years, so it gets more sanguine the longer things go smoothly. Yet common sense suggests that the risk of a blow-up will increase, not diminish, the farther away one gets from the last one. In other words, VAR is programmed to instil complacency. Moreover, it acts as yet another amplifier when trouble does hit. Episodes of volatility send VAR spiking upwards, which triggers moves to sell, creating further volatility.

The second problem is that VAR captures how bad things can get 99% of the time, but the real trouble is caused by the outlying 1%, the “long tail” of risk. “Risk management is about the stuff you don't know that you don't know,” says Till Guldimann, one of the original architects of VAR. “VAR leads to the illusion that you can quantify all risks and therefore regulate them.” The degree of dislocation in the CDO market has shown how hard it is to quantify risk on these products.

Models still have their place: optimists expect them to be greatly improved now that a big crisis has helpfully provided loads of new data on stressed markets. Even so, there is now likely to be more emphasis on non-statistical ways of thinking about risk. That means being more rigorous about imagining what could go wrong and thinking through the effects...

However, stress-testing has imperfections of its own. For example, it can lead to lots of pointless discussions about the plausibility of particular scenarios. Miles Kennedy of PricewaterhouseCoopers, a consultancy, thinks it is better to start from a given loss ($1 billion, say) and then work backwards to think about what events might lead to that kind of hit.

Nor is stress-testing fail-safe. The unexpected, by definition, cannot be anticipated... (emphasis added)

VAR is one of the measures I am sure the Wall Street clown was invoking while dressing down Dan Geer. Too bad it failed. (If you disagree, read the whole article, and better yet the whole special report... these are just excerpts.)

When the Economist refers to "stress-testing," think "threat modeling," and use the warped sense of that term instead of the better phrase "attack modeling." Picture a room full of people imagining what could happen based on assumptions and fantasy instead of spending the time and resources to gather ground-truth evidence on assets and historical or ongoing attacks. Sound familiar?

The article continues:

Another big challenge for risk managers lies in the treatment of innovative products. New products do not just lack the historic data that feed models. They often also sit outside banks' central risk-management machinery, being run by people on individual spreadsheets until demand for them is proven. That makes it impossible to get an accurate picture of aggregate risk, even if individual risks are being managed well. “We have all the leaves on the tree but not the tree,” is the mournful summary of one risk manager. One solution is to keep new lines of business below certain trading limits until they are fully integrated into the risk system.

Keeping risks to a size that does not inflict intolerable damage if things go awry is another fundamental (some might say banal) lesson...“It is not acceptable [for a division] to have a position that wipes out its own earnings, let alone those of the entire firm.”

However, working out the size of the risks is less easy than it used to be. For one thing, the lines between different types of risk have become hopelessly blurred. Risk-management teams at banks have traditionally been divided into watertight compartments, with some people worrying about credit risk (the chances of default on loans, say), others about market risk (such as sudden price movements) and yet others about operational risks such as IT failures or rogue traders. (emphasis added)

Ok, stick with me here. References to "innovating products" should be easy enough. Think WLANs in the early part of this decade, iPhones now, and so on. Think local groups of users deploying their own gear outside of IT or security influence or knowledge.

For "keeping risks to a size," think about the security principle of isolation. For "the lines between different types of risk," think about unexpected or unplanned interactions between new applications. "I didn't think that opening a hole in our firewall to let DMZ servers do backups would allow an intruder to piggyback on that connection, straight into the internal LAN, compromising our entire firm!"

Finally:

There is an even bigger concern. Everyone is ready to listen to risk managers now, but the message is harder to transmit when the going is good. “Come the next boom we will have traders saying, 'that was eight months ago. Why are you dragging me down with all that?',” sighs one risk chief. To improve risk management through the cycle, deeper change is needed.

Oh, I thought security was a "business enabler" with a "positive ROI." On a directly applicable note, during and right after an incident everyone is very concerned with "security." Eight months later hardly anyone cares.

Bankers, welcome to our world.

Read More

Final Question on FAIR

I'd hoped to not have to say anything else on FAIR, but I've decided to ask one final question. If I can't get a straight answer to this question I'm giving up on the discussion. So, here it is.

In Thoughts on FAIR I walked through the section Analyzing a Simple Scenario. Steps 3, 4, 5, 8, and 9 [Estimate the probable Threat Event Frequency (TEF), Estimate the Threat Capability (TCap), Estimate Control strength (CS), Estimate worst-case loss, and Estimate probable loss, respectively] each require the user to "Estimate."

Do these Estimates matter to the output of the model?

If the answer is yes, then those answers are important and should be grounded in reality -- not opinions. If FAIR proponents agree with this, then we have been debating for days for no real reason. That would make me happy.

If the answer is no, then what is so magical about FAIR that garbage in does not produce garbage out?

Read More

Wall Street Clowns and Their Models

Recently I cited an Economist article in Economist on the Peril of Models. While walking through the airport this Businessweek cover story, Not So Smart, caught my eye. I found the following excerpts to be interesting.

The titans of home loans announced they had perfected software that could spit out interest rates and fee structures for even the least reliable of borrowers. The algorithms, they claimed, couldn't fail...

It was the assumptions and guidelines that lenders used in deploying the technology that frequently led to trouble, notes industry veteran Jones. "It's garbage in, garbage out," he says. Mortgage companies argued their algorithms provided near-perfect precision. "We have a wealth of information we didn't have before," Joe Anderson, then a senior Countrywide executive, said in a 2005 interview with BusinessWeek. "We understand the data and can price that risk."

But in fact, says Jones, "there wasn't enough historical performance" related to exotic adjustable-rate loans to allow for reasonable predictions. Lenders "are seeing the results of not having that info now..."

At this point in probably sounds like I am seriously anti-model. That isn't really the case. The points I cited from Businessweek involve inserting arbitrary values into models. Non-arbitrary data is based on some reality, such as "historical performance" for an appropriate past period, looking forward into an appropriate future period.

Incidentally, one of the articles I read cited the Intangible Asset Finance Society, which is "dedicated to capturing maximum value from intellectual properties and other intangible assets such as quality, safety, security; and brand equity." That sounds like something to review.

Read More

Economist on Models

I intended to stay quiet on risk models for a while, but I read the following Economist articles and wanted to note them here for future reference.

From "Statistics and climatology: Gambling on tomorrow":

Climate models have lots of parameters that are represented by numbers... The particular range of values chosen for a parameter is an example of a Bayesian prior assumption, since it is derived from actual experience of how the climate behaves — and may thus be modified in the light of experience. But the way you pick the individual values to plug into the model can cause trouble...

Climate models have hundreds of parameters that might somehow be related in this sort of way. To be sure you are seeing valid results rather than artefacts of the models, you need to take account of all the ways that can happen.

That logistical nightmare is only now being addressed, and its practical consequences have yet to be worked out. But because of their philosophical training in the rigours of Pascal's method, the Bayesian bolt-on does not come easily to scientists. As the old saw has it, garbage in, garbage out. The difficulty comes when you do not know what garbage looks like. (emphasis added)

This is only a subset of the argument in the article. Note that this piece includes the line "derived from actual experience of how the climate behaves." I do not see the "Bayesian prior assumption[s]" so often cited elsewhere to even be derived from experience, if you take experience to mean something grounded in recent historical evidence instead of opinion. This willingness to rely on arbitrary inputs only aggravates the problems cited by the Economist.

The accompanying piece "Modelling the climate: Tomorrow and tomorrow" further explains some of the problems I've been describing previously.

While some argue about the finer philosophical points of how to improve models of the climate (see article), others just get on with it. Doug Smith and his colleagues at the Hadley Centre, in Exeter, England, are in the second camp and they seem to have stumbled on to what seems, in retrospect, a surprisingly obvious way of doing so. This is to start the model from observed reality.

Until now, when climate modellers began to run one of their models on a computer, they would “seed” it by feeding in a plausible, but invented, set of values for its parameters. Which sets of invented parameter-values to use is a matter of debate. But Dr Smith thought it might not be a bad idea to start, for a change, with sets that had really happened...

[T]he use of such real starting data made a huge improvement to the accuracy of the results. It reproduced what had happened over the courses of the decades in question as much as 50% more accurately than the results of runs based on arbitrary starting conditions.

Hindcasting, as this technique is known, is a recognised way of testing models. The proof of the pudding, though, is in the forecasting, so Dr Smith plugged in the data from two ten-day periods in 2005 (one in March and one in June), pressed the start button and crossed his fingers. (emphasis added)

This is exactly what I've been saying. Use "observed reality" and not "plausible, but invented" opinions or "arbitrary starting conditions" and I'll be happy to see what the model produces.

Read More

More Thoughts on FAIR

My post Thoughts on FAIR has attracted some attention, but as often the case some readers choose to obscure my point by overlaying their own assumptions. In this post I will try to explain my problems with FAIR in as simplistic a manner as possible.

Imagine if someone proposed the following model for assessing force: F=ma

(Yes, this is Newton's Second Law, and yes, I am using words like "model" and "assess" to reflect the risk assessment modeling problem.)

I could see two problems with using this model to assess force.

1. Reality check: The model does not reflect reality. In other words, an accurate measurement of mass times an accurate measurement of acceleration does not result in an accurate measurement of force.


2. Input check: To accurately measure force, the values for m and a must not be arbitrary. Otherwise, the value for F is arbitrary.

With respect to FAIR, I make the following judgments.

1. Reality check: The jury is out on whether FAIR reflects reality. It certainly might. It might not.


2. Input check: I have not seen any evidence that FAIR expects or requires anything other than arbitrary inputs. Arbitrary inputs to a model that passes the reality check does not produce anything valuable as far as I am concerned.
   
   If you personally like feeding your own opinions into a model to see what comes out the other end, have at it. It's nice to play around by making assumptions, seeing the result, and then altering the inputs to suit whatever output you really wanted to see.

One of the previous post commenters mentioned the book Uncertainty, which looks fascinating. If you read the excerpt you'll notice this line:

In the early 1970s what was then the U.S. Atomic Energy Commission (AEC) asked Norman C. Rasmussen, a professor of nuclear engineering at the Massachusetts Institute of Technology, to undertake a quantitative study of the safety of light-water reactors...

Rasmussen assembled a team of roughly sixty people, who undertook to identify and formally describe, in terms of event trees, the various scenarios they believed might lead to major accidents in each of the two reactors studied. Fault trees were developed to estimate the probabilities of the various events.

A combination of historical evidence from the nuclear and other industries, together with expert judgment, were used to construct the probability estimates, most of which were taken to be log-normally distributed. (emphasis added)

I've seen no commitment to including real evidence in FAIR, and I submit that those who lack evidence will have their so-called "expert judgment" fail due to soccer-goal security. Therefore, they possess neither historical evidence nor really expert judgment. So, even if FAIR meets my reality check, the results are only a feel-good exercise.

So how can I defend the use of this model?

Risk = Vulnerability X Threat X Impact (or Cost)

As I've said before, it is useful for showing the effects on Risk if you change one of the factors, ceteris paribus. This assumes that the model passes the reality check, which I believe it does.

I am not trying to calculate absolute values for Risk when I cite this equation. I am trying to conceptually show how Risk decreases when Threat decreases (ceteris paribus), or how Risk increases as Vulnerability increases (ceteris paribus), and so on.

You could take the same approach with F=ma if you were trying to explain to someone how it would hurt more to be struck by an object whose mass is larger than another object, assuming constant acceleration for each event. I am not trying to calculate F in such a case; I'm only using the model to describe the relationship between the components.

Read More

Thoughts on FAIR

You knew I had risk on my mind given my recent post Economist on the Peril of Models. The fact is I just flew to Chicago to teach my last Network Security Operations class, so I took some time to read the Risk Management Insight white paper An Introduction to Factor Analysis of Information Risk (FAIR). I needed to respond to Risk Assessment Is Not Guesswork, so I figured reading the whole FAIR document was a good start. I said in Brothers in Risk that I liked RMI's attempts to bring standardized terms to the profession, so I hope they approach this post with an open mind.

I have some macro issues with FAIR as well as some micro issues. Let me start with the macro issue by asking you a question:

Does breaking down a large problem into small problems, the solutions to which rely upon making guesses, result in solving the large problem more accurately?

If you answer yes, you will like FAIR. If you answer no, you will not like FAIR.

FAIR defines risk as

Risk - the probable frequency and probable magnitude of future loss

That reminded me of

Annual Loss Expectancy (ALE) = Annualized Rate of Occurrence (ARO) X Single Loss Expectancy (SLE)

If you don't agree remove the "annual" terms from the second definition or add them to the FAIR definition.

I have always preferred this equation

Risk = Vulnerability X Threat X Impact (or Cost)

because it is useful for showing the effects on risk if you change one of the factors, ceteris paribus. (Ok, I threw the Latin in there as homage to one of my economics instructors.)

If you consider frequency when estimating threat activity and include countermeasures as a component of vulnerability, you'll notice that Threat X Vulnerability starts looking like ARO. Impact (or Cost) is practically the same as SLE, so the two equations are similar.

FAIR turns its definition into the following.



If you care to click on that diagram, you'll see many small elements that need to be estimated. Specifically, you can follow the Basic Risk Assessment Guide to see these are the steps.

• Stage 1. Identify scenario components




• 1. Identify the asset at risk


• 2. Identify the threat community under consideration




• Stage 2. Evaluate Loss Event Frequency (LEF)




• 3. Estimate the probable Threat Event Frequency (TEF)


• 4. Estimate the Threat Capability (TCap)


• 5. Estimate Control strength (CS)


• 6. Derive Vulnerability (Vuln)


• 7. Derive Loss Event Frequency (LEF)




• Stage 3. Evaluate Probable Loss Magnitude (PLM)




• 8. Estimate worst-case loss


• 9. Estimate probable loss




• Stage 4. Derive and articulate Risk




• 10. Derive and articulate Risk

The problem with FAIR is that in every place you see the word "Estimate" you can substitute "Make a guess that's not backed by any objective measurement and which could be challenged by anyone with a different agenda." Because all the derived values are based on those estimates, your assessment of FAIR depends on the answer to the question I asked at the start of this post.

Let's see how this process stands up to some simple scrutiny by reviewing FAIR's Analyzing a Simple Scenario.

A Human Resources (HR) executive within a large bank has his username and password written on a sticky-note stuck to his computer monitor. These authentication credentials allow him to log onto the network and access the HR applications he’s entitled to use...

1. Identify the Asset at Risk: In this case, however, we’ll focus on the credentials, recognizing that their value is inherited from the assets they’re intended to protect.

We start with a physical security risk case. This simplifies the process considerably and actually gives FAIR the best chance it has to reflect reality. Why is that? The answer is that the physical world changes more slowly than the digital world. We don't have to worry about having solid walls being penetrated by a mutant from the X-Men movies or from the state of the credentials suddenly being altered by a patch or configuration change.

Identify the Threat Community: If we examine the nature of the organization (e.g., the industry it’s in, etc.), and the conditions surrounding the asset (e.g., an HR executive’s office), we can begin to parse the overall threat population into communities that might reasonably apply... For this example, let’s focus on the cleaning crew.

That's convenient. The document lists six potential threat communities but decides to only analyze one. Simplification sure makes it easier to proceed with this analysis. It also means the result is so narrowly targeted to be almost worthless, unless we decide to repeat this process for the rest of the threat communities. And this is still only looking at a sticky note.

3. Estimate the probable Threat Event Frequency (TEF): Many people demand reams of hard data before they’re comfortable estimating attack frequency. Unfortunately, because we don’t have much (if any) really useful or credible data for many scenarios, TEF is often ignored altogether. So, in the absence of hard data, what’s left? One answer is to use a qualitative scale, such as Low, Medium, or High.

And, while there’s nothing inherently wrong with a qualitative approach in many circumstances, a quantitative approach provides better clarity and is more useful to most decision-makers – even if it’s imprecise.

For example, I may not have years of empirical data documenting how frequently cleaning crew employees abuse usernames and passwords on sticky-notes, but I can make a reasonable estimate within a set of ranges.

Recognizing that cleaning crews are generally comprised of honest people, that an HR executive’s credentials typically would not be viewed or recognized as especially valuable to them, and that the perceived risk associated with illicit use might be high, then it seems reasonable to estimate a Low TEF using the table below...

Is it possible for a cleaning crew to have an employee with motive, sufficient computing experience to recognize the potential value of these credentials, and with a high enough risk tolerance to try their hand at illicit use? Absolutely! Does it happen? Undoubtedly. Might such a person be on the crew that cleans this office? Sure – it’s possible. Nonetheless, the probable frequency is relatively low. (emphasis added)

Says who? Has the person making this assessment done any research to determine if inflitrating cleaning crews is a technique used by economic adversaries? If yes, how often does that happen? What is the nature of the crew cleaning this office? Do they perform background checks? Have they been infiltrated before? Are they owned by a competitor? Figuring all of that out is too hard. Let's just supply guess #1: "low."

4. Estimate the Threat Capability (Tcap): Tcap refers to the threat agent’s skill (knowledge & experience) and resources (time & materials) that can be brought to bear against the asset... In this case, all we’re talking about is estimating the skill (in this case, reading ability) and resources (time) the average member of this threat community can use against a password written on a sticky note. It’s reasonable to rate the cleaning crew Tcap as Medium, as compared to the overall threat population.

Why is that? Why not "low" again? These are janitors we're discussing. Guess #2.

5. Estimate the Control Strength (CS): Control strength has to do with an asset’s ability to resist compromise. In our scenario, because the credentials are in plain sight and in plain text, the CS is Very Low. If they were written down, but encrypted, the CS would be different – probably much higher.

It is easy to accept guess #3 because we are dealing with a physical security scenario. It's simple for any person to understand that a sticky note in plain site has zero controls applied against it, so the (nonexistent) "controls" are worthless. But what about that new Web application firewall? Or you anti-virus software? Or any other technical control? Good luck assessing their effectiveness in the face of attacks that evolve on a weekly basis.

6. Derive Vulnerability (Vuln)

This value is derived using a chart that balances Tcap vs Control Strength. Since it is based on two guesses, one could decide if it is more or less accurate than estimated the vulnerability directly.

7. Derive Loss Event Frequency (LEF)

This value is derived using a chart that balances TEF vs Vulnerability. We derived vulnerability in the previous step and estimated TEF in step 3.

8. Estimate worst-case loss: Within this scenario, three potential threat actions stand out as having significant loss potential – misuse, disclosure, and destruction... For this exercise, we’ll select disclosure as our worst-case threat action.

This step considers Productivity, Response, Replacement, Fine/Judgments, Competitve Advantage, and Reputation, with Threat Actions including Access, Modification, Disclosure, and Denial of Access. Enter guess #4.

9. Estimate probable loss magnitude (PLM): The first step in estimating PLM is to determine which threat action is most likely. Remember; actions are driven by motive, and the most common motive for illicit action is financial gain. Given this threat community, the type of asset (personal information), and the available threat actions, it’s reasonable to select Misuse as the most likely action – e.g., for identity theft. Our next step is to estimate the most likely loss magnitude resulting from Misuse for each loss form.

Again, says who? Was identity theft chosen because it's popular in the news? My choice for guess #5 could be something completely different.

10. Derive and Articulate Risk: [R]isk is simply derived from LEF and PLM. The question is whether to articulate risk qualitatively using a matrix like the one below, or articulate risk as LEF, PLM, and worst-case.

The final risk rating is another derived value, based on previous estimates.

The FAIR author tries to head off critiques like this blog with the following section:

It’s natural, though, for people to accept change at different speeds. Some of us hold our beliefs very firmly, and it can be difficult and uncomfortable to adopt a new approach. Ultimately, not everyone is going to agree with the principles or methods that underlie FAIR. A few have called it nonsense. Others appear to feel threatened by it.

Apparently I'm resistant to "change" and "threatened" because I firmly hold on to "beliefs." I'm afraid that is what I will have to do when frameworks like this are founded upon someone's opinion at each stage of the decision-making process.

The FAIR document continues:

Their concerns tend to revolve around one or more of the following issues:

The absence of hard data. There’s no question that an abundance of good data would be useful. Unfortunately, that’s not our current reality. Consequently, we need to find another way to approach the problem, and FAIR is one solution.

I think I just read that the author admits FAIR is not based on "good data," and since we don't have data, we should just "find another way," like FAIR.

The lack of precision. Here again, precision is nice when it’s achievable, but it’s not realistic within this problem space. Reality is just too complex... FAIR represents an attempt to gain far better accuracy, while recognizing that the fundamental nature of the problem doesn’t allow for a high degree of precision.

The author admits that FAIR is not precise. How can it even be accurate when the derived values are all based on subjective estimates anyway?

Some people just don’t like change – particularly change as profound as this represents.

I fail to see why FAIR is considered profound. Is the answer because the process has been broken into five estimates, from which several other values are derived? Why is this any better than articles like How to Conduct a Risk Analysis or Risk Analysis Tools: A Primer or Risk Assessment and Threat Identification?

I'm sure this isn't the last word on this issue, but I need to rest before teaching tomorrow. Thank you for staying with me if you read the whole post. Obviously if I'm not a fan of FAIR I should propose an alternative. In Risk-Based Security is the Emperor's New Clothes I cited Donn Parker, who is probably the devil to FAIR advocates. If the question is how to make security decisions by assessing digital risk, I will put together thoughts on that for a post (hopefully this week).

Incidentally, the fact that I am not a fan of FAIR doesn't mean I think the authors have wasted their time. I appreciate their attempt to bring rigor to this process. I also think the questions they ask and the elements they consider are important. However, I think the ability to insert whatever value one likes into the five estimations fatally wounds the process.

This is the bottom line for me: FAIR advocates claim their output is superior due to their framework. How can a framework that relies on arbitrary inputs produce non-arbitrary output? And what makes FAIR so valuable anyway -- has the result been tested against any other methods?

Read More

Economist on the Peril of Models

Anyone who has been watching financial television stations in the US has seen commentary on the state of our markets with respect to subprime mortgages. I'd like to cite the 21 July 2007 issue of the Economist to make a point that resonates with digital security.

Both [Bear Stearns] funds had invested heavily in securities backed by subprime mortgages... On July 17th it admitted that there “is effectively no value left” in one of the funds, and “very little value left” in the other.

Such brutal clarity is, however, a rarity in the world of complex derivatives. Investors may now know what the two Bear Stearns funds are worth. But accountants are still unsure how to put a value on the instruments that got them into trouble.

This reminds me of a data breach -- instant clarity.

Traditionally, a company's accounts would record the value of an asset at its historic cost (ie, the price the company paid for it). Under so-called “fair value” accounting, however, book-keepers can now record the value of an asset at its market price (ie, the price the company could get for it).

But many complex derivatives, such as mortgage-backed securities, do not trade smoothly and frequently in arm's length markets. This makes it impossible for book-keepers to “mark” them to market. Instead they resort to “mark-to-model” accounting, entering values based on the output of a computer.

Note the reference to "complex"[ity] and book-keepers basing their decisions on models created by other people who make assumptions. Is this starting to sound like risk analysis to you, too?

Unfortunately, the market does not always resemble the model... Models are supposed to show the price an asset would fetch in a sale. But in an illiquid market, a big sale can itself drive down prices. This can sometimes create a sizeable difference between “mark-to-model” valuations and true market prices.

That is not the only problem with fair-value accounting. According to Richard Herring, a finance professor at the Wharton School, “models are easy to manipulate”...

Unfortunately, the alternatives to fair-value accounting can be worse. Historic cost may be harder to manipulate than the results of a model. But as Bob Herz, chairman of America's Financial Accounting Standards Board, points out, it too is “replete with all sorts of guesses”, such as depreciation rates...

"Models are easy to manipulate" and alternatives are also "replete with all sorts of guesses." This sounds exactly like risk analysis now.

Fair value is perhaps most worrying for auditors, who are often blamed for faulty accounts. Faced with murky models, the best they can do is examine assumptions and ensure disclosure.

This means that the role of the auditor becomes that of an outside expert who makes a new set of subjective decisions, perhaps challenging the assumptions of those who made subjective decisions when creating their model. The auditor's advantage, however, is that he/she has insight into the workings of many similar companies, and could compare "best practices" against the specific company being audited.

Incidentally, I would love to know how the "CISO of a major Wall Street bank" who criticized Dan Geer as mentioned in Are the Questions Sound? feels now about his precious financial models. Somehow I doubt his bonus will be as big as it was last year, if his company is even solvent by year's end.

Read More

Are the Questions Sound?

Dan Geer, second of the three wise men, was kind enough to share slides from his Measuring Security USENIX class. If I were not teaching at USENIX I would be in Dan's class.

One of the slides bothered me -- not for what Dan said, but for what was said to him. The slide is reproduced above, and the notes below:

These are precisely the questions that any CFO would want to know and we are not in a good position to answer. The present author was confronted with this list, exactly as it is, by the CISO of a major Wall Street bank with the preface “Are you security people so stupid that you cannot tell me....”

This particular CISO came from management audit and therefore was also saying that were he in any other part of the bank, bond portfolios, dervative pricing, equity trading strategies, etc., he would be able to answer such questions to five digit accuracy. The questions are sound.

I think Dan is giving the CISO too much credit. I think the questions are "semi-sound," and I think the CISO is the stupid one for using such a negative word to describe one of my Three Wise Men.

I'd like to mention several factors which make comparing the world of finance different from the world of digital security. I am recording these because they are more likely the kernel for future developed ideas, but I think they are legitimate points.

• Business: Digital security is not a line of business. No one practices security to make money. Security is not a productive endeavor; security risk is essentially a tax instantiated by the evil capabilities and intentions of threats. Because security is not a line of business, the performance incentives are not the same as a line of business. Security has no ROI; proper business initiatives do. Only security vendors make money from security.


• Accumulation: Digital security, as defined by preserving the confidentiality, integrity, and availability of information, cannot be accumulated. One cannot tap a reserve of security and later replenish it. Data that is exposed to the public Internet can seldom be quashed; data that has been corrupted at time of critical use cannot be changed later, thereby changing the past; and data that was not available at a critical time cannot be made available later, thereby changing the past.
  
  This is not the same with capital (i.e., money). Financial institutions are regulated and operated according to capitalization standards that dictate certain amounts of money to cover potential adverse events. Therefore, money can be stored as a counter to riskier behavior or decreased when pursuing less risky activities. Money at a single point in time is also homogenous; the first dollar of $100 is equally valuable as the hundreth dollar of $100. Information resources are not homogenous.


• Assumptions: Assumptions make financial "five digit accuracy" possible. Consider the assumptions made by the Black-Scholes model, courtesy of Wikipedia, used to price options:
  
  
  
  
  • The price of the underlying instrument S_t follows a geometric Brownian motion with constant drift μ and volatility σ:
  
  
  
  

  
  

  

  
  

  
  
  
  
  • It is possible to short sell the underlying stock.
  
  
  • There are no arbitrage opportunities.
  
  
  • Trading in the stock is continuous.
  
  
  
  • There are no transaction costs or taxes.
  
  
  • All securities are perfectly divisible (e.g. it is possible to buy 1/100th of a share).
  
  
  • It is possible to borrow and lend cash at a constant risk-free interest rate.
  
  
  • The stock does not pay a dividend (see below for extensions to handle dividend payments).
  
  
  
  The specifics of this equation are not important for this discussion, although those of you who also studied some economics may find plenty of ways to criticize it. (Remember the authors won the Nobel Prize for this equation and paper!) Consider what you could define if digital security practitioners were able to make such assumptions.


• Accuracy: I just said "assumptions make five digit accuracy possible." This isn't really true. If financial five digit accuracy were possible, no markets could be sustained. Simply put, markets exist because two sides agree to a trade. One side sees the world in one way, and the other sees it differently. (This is why market-makers exist on trading floors. When too many traders see the world the same, market-makers provide liquidity to permit trading.) If trading houses all figure out how to make money with five digit accuracy, their advantage is not going to be sustained because no one will want to trade with anyone else -- they're all want to take the same positions.

These are a few thoughts. It would be nice to hear from people with digital security and financial trading experience to provide commentary. Thank you.

Read More