Why DIARMF, "Continuous Monitoring," and other FISMA-isms Fail

I've posted about twenty FISMA stories over the years on this blog, but I haven't said anything for the last year and a half. After reading Goodbye DIACAP, Hello DIARMF by Len Marzigliano, however, I thought it time to reiterate why the newly "improved" FISMA is still a colossal failure.

First, a disclaimer: it's easy to be a cynic and a curmudgeon when the government and security are involved. However, I think it is important for me to discuss this subject because it represents an incredible divergence between security people. On one side of the divide we have "input-centric," "control-compliant," "we-can-prevent-the-threat" folks, and on the other side we have "output-centric," "field-assessed," "prevention eventually fails" folks. FISMA fans are the former and I am the latter.

So what's the problem with FISMA? In his article Len expertly discusses the new DoD Information Assurance Risk Management Framework (DIARMF) in comparison to the older DoD Information Assurance Certification and Accreditation Process (DIACAP). DIARMF is a result of the "new FISMA" emphasis on "continuous monitoring" which I've discussed before.

Len writes "DIARMF represents DoD adoption of the NIST Risk Management Framework process" and provides the diagram at left with the caption "The six major steps of Risk Management Framework aligned with the five phases of a System Development Lifecycle (SDLC)."

Does anything seem to be missing in that diagram? I immediately key on the "MONITOR Security Controls" box. As I reminded readers in Thoughts on New OMB FISMA Memo, control monitoring is not threat monitoring. The key to the "new" FISMA and "continuous monitoring" as seen in DIARMF is the following, described by Len:

Equally profound within DIARMF is the increased requirements for Continuous Monitoring activities. Each control (and control enhancement) will be attributed with a refresh rate (daily, weekly, monthly, yearly) and requisite updates on the status of each control will be packaged into a standardized XML format and uploaded into the CyberScope system where analysis, risk management, and correlation activities will be performed on the aggregate data.

Rather than checking on the security posture every three years or whatever insane interval that the old FISMA used, the new FISMA checks security posture more regularly, and centralizes posture reporting.

Wait, isn't that a good idea? Yes, it's a great idea -- but it's still control monitoring. I can't stress this enough; under the new system, a box can be totally owned but appear "green" on the FISMA dashboard because it's compliant with controls. Why? There is no emphasis on threat monitoring -- incident detection and response -- which is the only hope we have against any real adversary.

Think I'm wrong? Read Len's words on CyberScope:

CyberScope is akin to a giant federal-wide SEIM system, where high-level incident management teams can quickly pull queries or drill down into system details to add analysis on system defenses and vulnerabilities to the available intelligence on an attack. CyberScope data will also be used to track trends, make risk management decisions, and determine where help is needed to improve security posture.

If you're still not accepting the point, consider this football analogy.

Under the old system, you measured the height, weight, 40 yard dash, and other "combine" results on a player when he joined the team. You checked again three years later. You kept data on all your players but had no idea what the score of the game was.

Under the new system, you measure the height, weight, 40 yard dash, and other "combine" results on a player when he joins the team. You check again more regularly -- maybe even every hour, and store the data in a central location with a fancy Web UI. You keep data on all your players but still have no idea what the score of the game is.

Until DoD, NIST, and the other control-compliant cheerleaders figure out that this approach is a failure, the nation's computers will remain compromised.

Note: There are other problems with DIARMF -- read the section where Len says "This shakes out to easily over a hundred different possible control sets that can be attributed to systems" to see what I mean.

Tweet

Read More

Let a Hundred Flowers Blossom

I know many of us work in large, diverse organizations. The larger or more complex the organization, the more difficult it is to enforce uniform security countermeasures. The larger the population to be "secure," the more likely exceptions will bloom. Any standard tends to devolve to the least common denominator. There are some exceptions, such as FDCC, but I do not know how widespread that standard configuration is inside the government.

Beyond the difficulty of applying a uniform, worthwhile standard, we run into the diversity vs monoculture argument from 2005. I tend to side with the diversity point of view, because diversity tends to increase the cost borne by an intruder. In other words, it's cheaper to develop exploitation methods for a target who 1) has broadly similar, if not identical, systems and 2) publishes that standard so the intruder can test attacks prior to "game day."

At the end of the day, the focus on uniform standards is a manifestation of the battle between two schools of thought: Control-Compliant vs Field-Assessed Security. The control-compliant team believes that developing the "best standard," and then applying that standard everywhere, is the most important aspect of security. The field-assessed team (where I devote my effort) believes the result is more important than how you get there.

I am not opposed to developing standards, but I do think that the control-compliant school of thought is only half the battle -- and that controls occupy far more time and effort than they are worth. If the standard whithers in the face of battle, i.e., once field-assessed it is found to be lacking, then the standard is a failure. Compliance with a failed standard is worthless at that point.

However, I'd like to propose a variation of my original argument. What if you abandon uniform standards completely? What if you make the focus of the activity field-assessed instead of control-compliant, by conducting assessments of systems? In other words, let a hundred flowers blossom.

(If you don't appreciate the irony, do a little research and remember the sorts of threats that occupy much of the time of many this blog's readers!)

So what do I mean? Rather than making compliance with controls the focus of security activity, make assessment of the results the priority. Conduct blue and red team assessments of information assets to determine if they meet various resistance and (maybe) "survivability" metrics. In other words, we won't care how you manage to keep an intruder from exploiting your system, as long as it takes longer for a blue or red assesor with time X and skill level Y and initial access level Z (or something to that effect).

In such a world, there's plenty of room for the person who wants to run Plan 9 without anti-virus, the person who runs FreeBSD with no graphical display or Web browser, the person who runs another "nonstandard" platform or system -- as long as their system defies the field assessment conducted by the blue and red teams. (Please note the one "standard" I would apply to all assets is that they 1) do no harm to other assets and 2) do not break any laws by running illegal or unlicensed software.)

If a "hundred flowers" is too radical, maybe consider 10. Too tough to manage all that? Guess what -- you are likely managing it already. So-called "unmanaged" assets are everywhere. You probably already have 1000 variations, never mind 100. Maybe it's time to make the system's inability to survive against blue and red teams the measure of failure, not whether the system is "compliant" with a standard, the measure of failure?

Now, I'm sure there is likely to be a high degree of correlation between "unmanaged" and vulnerable in many organizations. There's probably also a moderate degree of correlation between "exceptional" (as in, this box is too "special" to be considered "managed") and vulnerable. In other instances, the exceptional systems may be impervious to all but the most dedicated intruders. In any case, accepting that diversity is a fact of life on modern networks, and deciding to test the resistance level of those assets, might be more productive than seeking to develop and apply uniform standards.

What do you think?

Read More

Control "Monitoring" is Not Threat Monitoring

As I write this post I'm reminded of General Hayden's advice:

"Cyber" is difficult to understand, so be charitable with those who don't understand it, as well as those who claim "expertise."

It's important to remember that plenty of people are trying to act in a positive manner to defend important assets, so in that spirit I offer the following commentary.

Thanks to John Bambanek's SANS post I read NIST Drafts Cybersecurity Guidance by InformationWeek's J. Nicholas Hoover.

The article discusses the latest draft of SP 800-37 Rev. 1:
DRAFT Guide for Applying the Risk Management Framework to Federal Information Systems: A Security Life Cycle Approach.

I suspected this to be problematic given NIST's historical bias towards "controls," which I've criticized in Controls Are Not the Solution to Our Problem and Consensus Audit Guidelines Are Still Controls. The subtext for the article was:

The National Institute for Standards and Technology is urging the government to continuously monitor its own cybersecurity efforts.

As soon as I read that, I knew that NIST's definition of "monitor" and the article's definition of "monitor" did not mean the real sort of monitoring, threat monitoring, that would make a difference against modern adversaries.

The article continues:

Special Publication 800-37 fleshes out six steps federal agencies should take to tackle cybersecurity: categorization, selection of controls, implementation, assessment, authorization, and continuous monitoring...

Finally, and perhaps most significantly, the document advises federal agencies to put continuous monitoring in place. Software, firmware, hardware, operations, and threats change constantly. Within that flux, security needs to be managed in a structured way, Ross says.

"We need to recognize that we work in a very dynamic operational environment," Ross says. "That allows us to have an ongoing and continuing acceptance and understanding of risk, and that ongoing determination may change our thinking on whether current controls are sufficient."

The continuous risk management step might include use of automated configuration scanning tools, vulnerability scanning, and intrusion detection systems, as well as putting in place processes to monitor and update security guidance and assessments of system security requirements.

Note that the preceding text mentions "intrusion detection systems," but the rest of the text has nothing to do with real monitoring, i.e., detecting and responding to intrusions. I'm not just talking about network-centric approaches, by the way -- infrastructure, host, log, and other sources are all real monitoring, but this is not what NIST means by "monitoring."

To understand NIST's view of monitoring, try reading the new draft. I'll insert my comments.

APPENDIX G

CONTINUOUS MONITORING

MANAGING AND TRACKING THE SECURITY STATE OF INFORMATION SYSTEMS

A critical aspect of managing risk from information systems involves the continuous monitoring of the security controls employed within or inherited by the system.65

[65 A continuous monitoring program within an organization involves a different set of activities than Security Incident Monitoring or Security Event Monitoring programs.]

So, it sounds like activities that involve actually watching systems are not within scope for "continuous monitoring."

Conducting a thorough point-in-time assessment of the deployed security controls is a necessary but not sufficient condition to demonstrate security due diligence. An effective organizational information security program also includes a rigorous continuous monitoring program integrated into the system development life cycle. The objective of the continuous monitoring program is to determine if the set of deployed security controls continue to be effective over time in light of the inevitable changes that occur.

That sounds ok so far. I like the idea of evaluations to determine if controls are effective over time. In the next section below we get to the heart of the problem, and why I wrote this post.

An effective organization-wide continuous monitoring program includes:

• Configuration management and control processes for organizational information systems;

• Security impact analyses on actual or proposed changes to organizational information systems and environments of operation;67

• Assessment of selected security controls (including system-specific, hybrid, and common controls) based on the organization-defined continuous monitoring strategy;68

• Security status reporting to appropriate organizational officials;69 and

• Active involvement by authorizing officials in the ongoing management of information system-related security risks.

Ok, where is threat monitoring? I see configuration management, "control processes," reporting status to "officials," "active involvement by authorizing officials," and so on.

The next section tells me what NIST really considers to be "monitoring":

Priority for security control monitoring is given to the controls that have the reatest volatility and the controls that have been identified in the organization’s plan of action and milestones...

[S]ecurity policies and procedures in a particular organization may not be likely to change from one year to the next...

Security controls identified in the plan of action and milestones are also a priority in the continuous monitoring process, due to the fact that these controls have been deemed to be ineffective to some degree.

Organizations also consider specific threat information including known attack vectors (i.e., specific vulnerabilities exploited by threat sources) when selecting the set of security controls to monitor and the frequency of such monitoring...

Have you broken the code yet? Security control monitoring is a compliance activity. Granted, this is an improvement from the typical certification and accreditation debacle, where "security" is assessed via paperwork exercises every three years. Instead, .gov compliance teams will perform so-called "continuous monitoring," meaning more regular checks to see if systems are in compliance.

Is this really an improvement?

I don't think so. NIST is missing the point. Their approach advocates Control-compliant security, not field-assessed security. Their "scoreboard" is the result of a compliance audit, not the number of systems under adversary control or the amount of data exfiltrated or degraded by the adversary.

I don't care how well your defensive "controls" are informed by offense. If you don't have a Computer Incident Response Team performing continuous threat monitoring for detection and response, you don't know if your controls are working. The NIST document has a few hints about the right approach, at best, but the majority of the so-called "monitoring" guidance is another compliance activity.

Read More

Controls Are Not the Solution to Our Problem

If you recognize the inspiration for this post title and graphic, you'll understand my ultimate goal. If not, let me start by saying this post is an expansion of ideas presented in a previous post with the succinct and catchy title Control-Compliant vs Field-Assessed Security.

In brief, too many organizations, regulators, and government agencies waste precious time and resources devising and auditing "controls," regardless of the effect these controls have or do not have on security. They are far too input-centric; they should become more output-aware. They obsess over recording conditions they believe may be helpful while remaining ignorant of the "score of the game." They practice management by belief and disregard management by fact.

Let me provide a few examples from one of the canonical texts used by the control-compliant crowd: NIST Special Publication 800-53: Recommended Security Controls for Federal Information Systems (.pdf). The following is an example of a control, taken from page 140.

SI-3 MALICIOUS CODE PROTECTION


The information system implements malicious code protection.

Control: Supplemental Guidance: The organization employs malicious code protection mechanisms at critical information system entry and exit points (e.g., firewalls, electronic mail servers, web servers, proxy servers, remote-access servers) and at workstations, servers, or mobile computing devices on the network. The organization uses the malicious code protection mechanisms to detect and eradicate malicious code (e.g., viruses, worms, Trojan horses, spyware) transported: (i) by electronic mail, electronic mail attachments, Internet accesses, removable media (e.g., USB devices, diskettes or compact disks), or other common means; or (ii) by exploiting information system vulnerabilities. The organization updates malicious code protection mechanisms (including the latest virus definitions) whenever new releases are available in accordance with organizational configuration management policy and procedures. The organization considers using malicious code protection software products from multiple vendors (e.g., using one vendor for boundary devices and servers and another vendor for workstations). The organization also considers the receipt of false positives during malicious code detection and eradication and the resulting potential impact on the availability of the information system. NIST Special Publication 800-83 provides guidance on implementing malicious code protection.

Control Enhancements:
(1) The organization centrally manages malicious code protection mechanisms.
(2) The information system automatically updates malicious code protection mechanisms.

At first read one might reasonably respond by saying "What's wrong with that? This control advocates implementing anti-virus and related anti-malware software." Think more clearly about this issue and several problems appear.

• Adding anti-virus products can introduce additional vulnerabilities to systems which might not have exposed themselves without running anti-virus. Consider my post Example of Security Product Introducing Vulnerabilities if you need examples. In short, add anti-virus, be compromised.


• Achieving compliance may cost more than potential damage. How many times have you heard a Unix administrator complain that he/she has to purchase an anti-virus product for his/her Unix server simply to be compliant with a control like this? The potential for a Unix server (not Mac OS X) to be damaged by a user opening an email through a client while logged on to the server (a very popular exploitation vector on a Windows XP box) is practically nil.


• Does this actually work? This is the question that no one asks. Does it really matter if your system is running anti-virus software? Did you know that intruders (especially high-end ones most likely to selectively, steathily target the very .gov and .mil systems required to be compliant with this control) test their malware against a battery of anti-virus products to ensure their code wins? Are weekly updates superior to daily updates? Daily to hourly?

The purpose of this post is to tentatively propose an alternative approach. I called this "field-assessed" in contrast to "control-compliant." Some people prefer the term "results-based." Whatever you call it, the idea is to direct attention away from inputs and devote more energy to outputs. As far as mandating inputs (like every device must run anti-virus), I say that is a waste of time and resources.

I recommend taking measurements to determine your enterprise "score of the game," and use that information to decide what you need to do differently. I'm not suggesting abandoning efforts to prevent intrusions (i.e., "inputs.") Rather, don't think your security responsibilities end when the bottle is broken against the bow of the ship and it slides into the sea. You've got to keep watching to see if it sinks, if pirates attack, how the lifeboats handle rough seas, and so forth.

These are a few ideas.

1. Standard client build client-side survival test. Create multiple sacrificial systems with your standard build. Deploy a client-side testing solution on them, like a honeyclient. (See The Sting for a recent story.) Vary your defensive posture. Measure how long it takes for your standard build to be compromised by in-the-wild Web sites, spam, and other communications with the outside world.


2. Standard client build server-side survival test. Create multiple sacrificial systems with your standard build. Deploy them as a honeynet. Vary your defensive posture. Measure how long it takes for your standard build to be compromised by malicious external traffic from the outside world -- or better yet -- from your internal network.


3. Standard client build client-side penetration test. Create multiple sacrificial systems with your standard build. Conduct my recommendation penetration testing activities and time the result.


4. Standard client build server-side penetration test. Repeat number 3 with a server-side flavor.


5. Standard server build server-side penetration test. Repeat number 3 against your server build with a server-side flavor. I hope you don't have users operating servers as if they were clients (i.e., browsing the Web, reading email, and so forth.) If you do, repeat this step and do a client-side pen test too.


6. Deploy low-interactive honeynets and sinkhole routers in your internal network. These low-interaction systems provide a means to get some indications of what might be happening inside your network. If you think deploying these on the external network might reveal indications of targeted attacks, try that. (I doubt it will be that useful due to the overall attack noise, but who knows?)


7. Conduct automated, sampled client host integrity assessments. Select a statistically valid subset of your clients and check them using multiple automated tools (malware/rootkit/etc. checkers) for indications of compromise.


8. Conduct automated, sampled server host integrity assessments. Self-explanatory.


9. Conduct manual, sampled client host integrity assessments. These are deep-dives of individual systems. You can think of it as an incident response where you have not had indication of an incident yet. Remote IR tools can be helpful here. If you are really hard-core and you have the time, resources, and cooperation, do offline analysis of the hard drive.


10. Conduct manual, sampled server host integrity assessments. Self-explanatory.


11. Conduct automated, sampled network host activity assessments. I questioned adding this step here, since you should probably always be doing this. Sometimes it can be difficult to find the time to review the results, however automated the data collection. The idea is to let your NSM system see if any of the traffic it sees is out of the ordinary based on algorithms you provide.


12. Conduct manual, sampled network host activity assessments. This method is more likely to produce results. Here a skilled analyst performs deep individual analysis of traffic on a sample of machines (client and server, separately) to see if any indications of compromise appear.

In all of these cases, trend your measurements over time to see if you see improvements when you alter an input. I know some of you might complain that you can't expect to have consistent output when the threat landscape is constantly changing. I really don't care, and neither does your CEO or manager!

I offer two recommendations:

• Remember Andy Jaquith's criteria for good metrics, simplified here.




1. Measure consistently.


2. Make them cheap to measure. (Sorry Andy, my manual tests violate this!)


3. Use compound metrics.


4. Be actionable.




• Don't slip into thinking of inputs. Don't measure how many hosts are running anti-virus. We want to measure outputs. We are not proposing new controls.

Controls are not the solution to our problem. Controls are the problem. They divert too much time, resources, and attention from endeavors which do make a difference. If the indications I am receiving from readers and friends are true, the ideas in this post are gaining traction. Do you have other ideas?

Read More

Control-Compliant vs Field-Assessed Security

Last month's ISSA-NoVA meeting featured Dennis Heretick, CISO of the US Department of Justice. Mr. Heretick seemed like a sincere, devoted government employee, so I hope no one interprets the following remarks as a personal attack. Instead, I'd like to comment on the security mindset prevalent in the US government. Mr. Heretick's talk sharpened my thoughts on this matter.

Imagine a football (American-style) team that wants to measure their success during a particular season. Team management decides to measure the height and weight of each player. They time how fast the player runs the 40 yard dash. They note the college from which each player graduated. They collect many other statistics as well, then spend time debating which ones best indicate how successful the football team is. Should the center weigh over 300 pounds? Should the wide receivers have a shoe size of 11 or greater? Should players from the north-west be on the starting line-up? All of this seems perfectly rational to this team.

An outsider looks at the situation and says: "Check the scoreboard! You're down 42-7 and you have a 1-6 record. You guys are losers!"

In my opinion, this summarizes the mindset of US government information security managers.

Here are some examples from Mr. Heretick's talk. He showed a "dashboard" with various "metrics" that supposedly indicate improved DoJ security. The dashboard listed items like:

• IRP Reporting: meaning Incident Response Plan reporting, i.e., does the DoJ unit have an incident response plan? This says nothing about the quality of the IRP.


• IRP Exercised: has the DoJ unit exercised its IRP? This says nothing about the effectiveness of the IRT in the exercise.


• CP Developed: meaning Contingency Plan developed, i.e, does the DoJ unit have a contingency plan should disaster strike? This also says nothing about the quality of the CP.


• CP Exercised: has the DoJ unit exercised its CP? Same story as the IRP.

Imagine a dashboard, then, with all "green" for these items. They say absolutely nothing about the "score of the game."

How should the score be measured then? Here are a few ideas, which are neither mutually exclusive nor exceedingly well-thought-out:

• Days since last compromise of type X: This is similar to a manufacturing plant's "days since an accident" report or a highway's "days since a fatality" report. For some sites this number may stay zero if the organization is always compromised. The higher the number, the better.


• System-days compromised: This looks at the number of systems compromised, and for how many days, during a specified period. The lower, the better.


• Time for a pen testing team of [low/high] skill with [internal/external] access to obtain unauthorized [unstealthy/stealthy] access to a specified asset using [public/custom] tools and [complete/zero] target knowledge: This is from my earlier penetration testing story.

These are just a few ideas, but the common theme is they relate to the actual question management should care about: are we compromised, and how easy is it for us to be compromised?

I explained my football analogy to Mr. Heretick and asked if he would adopt it. He replied that my metrics would discourage DoJ units from reporting incidents, and that reporting incidents was more important to him than anything else. This is ridiculous, and it indicates to me that organizations like this (and probably the whole government) need independent, Inspector General-style units that roam freely to assess networks and discover intruders.

In short, the style of "security" advocated by government managers seems to be "control-compliant." I prefer "field-assessed" security, although I would be happy to replace that term with something more descriptive. In the latest SANS NewsBites (link will work shortly) Alan Paller used the term "attack-based metrics," saying the following about the VA laptop fiasco: "if the VA security policies are imprecise and untestable, if the VA doesn't monitor attack-based metrics, and if there are no repercussions for employees who ignore the important policies, then this move [giving authority to CISOs] will have no impact at all."

PS: Mr. Heretick shared an interesting risk equation model. He uses the following to measure risk.

• Vulnerability is measured by assessing exploitability (0-5), along with countermeasure effectiveness (0-2). Total vulnerability is exploitability minus countermeasures.


• Threat is measured by assessing capability (1-2), history (1-2), gain (1-2), attributability (1-2), and detectability (1-2). Total threat is capability plus history plus gain minus attributability minus detectability.


• Significance (i.e., impact or cost) is measured by assessing loss of life (0 or 4), sensitivity (0 or 4), operational impact (0 or 2), and equipment loss (0 or 2). Total significance is loss plus op impact plus sensitivity plus equipment loss.


• Total risk is vulnerability times threat times significance, with < 6 very low, 6-18 low, 19-54 medium, 55-75 high, and >75 very high.

Read More