FISMA Redux

Late last year I mentioned I planned to read and review FISMA Certification & Accreditation Handbook by Laura Taylor. You know if I read a book on Cisco MARS on one leg of my last trip, I probably read a different book on the return leg. FISMA was that book. These comments are going to apply most directly to FISMA itself, based on what I learned reading Ms. Taylor's book. I'll save comments on the book itself for a later date.

Last year I wrote FISMA is a joke.. I was wrong, and I've decided to revise my opinion. Based on my understanding of FISMA as presented in this book,

FISMA is a jobs program for so-called security companies without the technical skills to operationally defend systems. This doesn't mean that if you happen to conduct FISMA work, you're definitelTy without technical skills. I guarantee my friends at ClearNet Security are solid guys, just based on their ability to detect the C&A project they joined was worthless. Anyway, I guess it's time to put on a flame-retardant suit.

Let's start with p xxiii in FISMA to understand the thought process of those who believe in it. Foreword author Sunil James, "Former Staff Director of the FDIC," writes that the FISMA "process has been proven to reduce risk to federal information systems." I think he means that FISMA reduces the risk of unemployment for those who perform C&A on federal information systems.

Without saying anything else, I think I know the problem with the FISMA-supporting crowd. I bet they do not do any other work, especially not technical work and certainly not incident response work for the agencies they "certify" and "accredit." If they did have operational security responsibilities for their C&A clients they'd wonder "why are these systems repeatedly being 0wned if their C&A packages are up-to-date?" Those that keep one foot in the C&A world and another in the operational world realize their FISMA feet are wet and smelly and not doing anyone any good, period.

Another sad truth about FISMA, despite Mr. Porter's unsubstantiated claim, is that there is zero connection between high FISMA scores and lower impact or number of intrusions. If you don't believe me, keep your eyes open for the next FISMA report card from the House Committee on Oversight and Government Reform. A look at the 2006 scores is interesting. Figure out who has good scores, and then see how they fared in this staff report on Federal breaches since January 2003. Hint: everyone's been 0wned, is 0wned, and will continue to be 0wned while money is spent paying consultants to write FISMA C&A packages.

Let's see what this FISMA book has to say about C&A packages. Page 34 claims a good ISSO can maintain C&A packages for 6 systems, and that writing each package can take 3-6 months. They need to be updated every 3 years. The C&A handbook guiding the writing of packages is usally 200+ pages and needs to be kept current. The packages themselves are usually 500+ (!) pages, and require 2-4 weeks to be read by the accreditors.

On p 34 we come to a root of the problem:

Since once C&A package could easily take a year for a well-versed security expert to prepare, it is considered standard and acceptable for ISSOs to hire consultants from outside the agency to prepare the Certification Package.

Page 38 continues:

Since C&A, if done properly, is usually a much bigger job than most people realize, I cannot emphasize enough the value in using outside consultants. Putting together a Certification Package is a full-time job.

I do not have a problem with consultants. Heck, I am a consultant. However, the vast majority of my work does not revolve around writing 500 page reports based on self-assessments every three years.

Laura Taylor writes on pp 8-9:

C&A is essentially a documentation and paperwork nightmare... prepare yourself to write, write, and write some more. If you detest writing, you're in the wrong business.

Basically, preparing a Certification Package is writing about security -- extensive writing about security. When you are preparing a Certification Package, you usually don't perform any sort of hands-on security. You review the existing security design and architecture documents, interview various IT support and development folks familiar with the infrastructure, and document your findings. (emphasis added)

I am not making this up. The really sad part is this: the author then says

...why C&A exists -- it is a process that enables authorizing officials to discover the security truths about their infrastructure so that informed decisions can be made.

Security truths? What are they based on?

In chapter 8 we read about "security self-assessments." Maybe those are helpful? Hmm, probably not: "A security self-assessment is a survey-based audit that is essentially a long list of questions." What's worse, page 115 says:

[I]n September 2003 a report put out by the Office of Inspector General at the Environmental Protection Agency found that 36 percent of the responses to security self-assessments contained inaccurate information.

Ms. Taylor's recommendation?

Tip: Encourage self-assessment respondents to answer questions truthfully.

Maybe some other aspect of C&A and FISMA shows merit. Chapter 12, discussing Security Tests and Evaluation (ST&E), begins with this:

A Security Test & Evaluation, known among security experts as an ST&E, is a document that demonstrates that an agency has performed due diligence in testing security requirements and evaluation the outcome of the tests.

The ST&E is a C&A document that tends to give agencies lots of trouble. It's not clear to many agencies what tests they should be doing, who should be doing them, and what the analysis of the tests should consist of.

That's another winner in my book.

FISMA fans out there are going to cite the vulnerability scans which are usually part of ST&E as a sign that something technical happens during C&A. Believe me, I am sparing the author of this book and her "technical editors" by not reproducing their recommendations for assessments. (One word: Strobe.)

We could also look at the Privacy Impact Assessment, but guess what -- it's another self-survey.

The bottom line is that FISMA doesn't mention C&A at all, but the author thinks that's ok because C&A fulfill's FISMA's goals. The reality is far different. According to the act itself, the first "purpose" is to:

provide a comprehensive framework for ensuring the effectiveness of information security controls over information resources that support Federal operations and assets. (emphasis added)

FISMA is failing miserably. It's ironic that this FISMA book begins each chapter with a quote, and this begins chapter 2:

It is common sense to take a method and try it. If it fails, admit it frankly and try another. But above all, try something. -- FDR

It's time for the FISMA fans to admit this five year FISMA experiment has been a waste of taxpayer money and agency resources.

Returning to my football analogy, C&A is an expensive, extensive practice session controlled by the players and overseen by agents who get paid the longer the team is on the field. Success is measured not by the score of the next game but by the number of worthless statistics written about the players prior to the first snap. Once the team takes to the field they are annihilated by the opposition, but the agents don't care because they're spending their money elsewhere.

If you are a C&A Package-writing company, I strongly recommend you gain some operational capabilities or look for a new line of business. I am committed to eliminating your position in the Federal government. Laura Taylor writes with some apparent regret that "most private and public companies don't put nearly as much time, effort, and resources into documenting their security as government agencies do." Let's keep it that way. At least it frees up resources for work that has a chance of stopping an intruder.

Do I have anything "nice" to say? Yes -- if you are so cursed as to be responsible for a FISMA C&A project, I do recommend reading this book. Forget the technical aspects and concentrate on understanding the FISMA maze. I thought Laura Taylor wrote a clear and well-organized book with lots of practical advice and good templates. I would much rather see her not have to write about this subject again, though!

Read More

Thoughts on SAS 70 and Other Standards

I'm not an auditor or CPA, thank goodness. The first time I heard of SAS 70 (Statement on Auditing Standards No. 70, Service Organizations) happened when I visited Symantec in October. Last week, however, one of my clients asked what I knew about SAS 70. I knew Symantec used its SAS 70 results as a way to avoid having every Symantec managed security service client perform its own audit of Symantec. My client wanted to know if his company might also benefit from getting a SAS 70 audit.

I found an exceptionally helpful CSO Online article by Michael Fitzgerald about SAS 70. I'd like to share some insights from it.

A spokeswoman for the body that created SAS 70 doesn't actually recommend it for security purposes. "It isn't a measure of security, it's a measure of financial controls," says Judith Sherinsky, a technical manager on the audit and test standards team at the American Institute of Certified Public Accountants (AICPA), which created SAS 70...

For security audits, Sherinsky recommends a different AICPA standard: SysTrust, an attestation engagement that includes criteria for system security. SysTrust was developed to help CPAs gauge whether systems meet the following criteria: availability, security, processing integrity, online privacy and confidentiality.

That's extraordinary. It means all the customers who get a SAS 70 audit from a service provider aren't getting any real security assurances. It sounds like Common Criteria.

The Sarbanes-Oxley Act is essentially a mandate to establish internal controls so that corporate executives can't fudge their numbers. Sarbox requires that companies verify the accuracy of their financial statements, and establishes SAS 70 Type 2 audits as a way to verify that third-party providers meet those needs...

A SAS 70 audit does not rate a company's security controls against a particular set of defined best practices. In a SAS 70 audit, the service organization being audited must first prepare a written description of its goals and objectives. The auditor then examines the service organization's description and says whether the auditor believes those goals are fairly stated, whether the controls are suitably designed to achieve the control objectives that the organization has stated for itself, whether the controls have been placed in operations (as opposed to existing only on paper), and in a Type 2 engagement, whether these controls are operating effectively.

The fact that a company has conducted a SAS 70 audit does not necessarily mean its systems are secure. In fact, a SAS 70 may confirm that a particular system is not secure, by design.

"You can have control objectives to make any statement management may want to make," says Robert Aanerud, chief risk officer and principal consultant at security consultancy HotSkills. In effect, he says, management could decide that the company is OK with bad access control, and the auditor (who must be a CPA) then needs to ensure that access control is at least bad. The SAS 70 opinion would essentially say that, yes, the company has achieved its stated control objectives. (emphasis added)

It sounds like reading the SAS 70 report is important!

Unfortunately, consultants say many companies are skipping the hard work and treating SAS 70 as a security rubber stamp. Sharon O'Bryan, head of O'Bryan Advisory Services, says she's aware of companies taking SAS 70 reports for potential service providers, sticking them someplace and never reading them...

Service providers say they're being asked more and more often for SAS 70 audits, often instead of governance standards like Cobit or ISO 17799. That's even true for companies that handle security functions, traditionally more oriented toward granular best-practice tests than the broad audit test of SAS 70.

Michael Scher, general counsel and compliance architect at Nexum, a security product and service provider, says his company is preparing to undergo its first SAS 70 audit. "It's an efficiency-type move," Scher says. It will save his company the trouble of having to be audited by every potential client, or generate reams of documentation in answer to questions.

If SAS 70 is so bad for security, is there an alternative? The AICPA quote earlier in the article mentions SysTrust. AICPA provides a comparison brochure for SAS 70 vs SysTrust. It includes the following:

SAS 70 intended purpose: To provide user auditors with information about controls at the service organization that may affect assertions in the user organizations' financial statements. This generally enables a user auditor to perform an audit.

Trust Services intended purpose: To provide assurance that an organization's system's controls meet one or more of the Trust Services principles and related criteria. Areas addressed by the Principles include: security, online privacy, availability, confidentiality and processing integrity.

It seems SAS 70 is not at all what the customers think it is. Alternatively, they know they are not getting any security assurances, but just want a rubber stamp. Apparently this is more make-work for CPAs, since SAS 70 work must be done by a CPA.

Speaking of work for CPAs, their What Skills Do I Need to Provide SysTrust Services? site is hilarious in a sick way:

Application of Current Skills and Knowledge. CPAs have the ethical standards and principles needed to evaluate and provide assurance on the reliability of systems. CPAs have skills in evaluating evidence, determining the effectiveness of internal controls, and reporting to third parties on the results of the work performed.

New Skills and Knowledge May Be Required. CPA in public accounting and CPAs industry may require additional competencies in addition to those from the traditional accounting, audit, and tax arena in order to provide services related to SysTrust. In order to deliver SysTrust-related advice or assurance, CPAs may need to use automated techniques.

So, in brief: CPAs are ethical but are going to rely on "automated techniques" to validate security effectiveness. Right...

What about the various ISO standards? Here Wikipedia is helpful:

ISO/IEC 27001 is an information security standard published in October 2005 by the International Organization for Standardization and the International Electrotechnical Commission. Its complete name is Information technology -- Security techniques -- Information security management systems -- Requirements. The current standard replaced BS 7799-2:2002, which has now been withdrawn.

ISO/IEC 27001:2005 specifies the requirements for establishing, implementing, operating, monitoring, reviewing, maintaining and improving a documented Information Security Management System (ISMS). It specifies requirements for the management of the implementation of security controls. It is intended to be used in conjunction with ISO 17799:2005, a security Code of Practice, which offers a list of specific security controls to select from.

This is also the first standard in a proposed series of standards which will be assigned numbers within the ISO 27000 series. Others are anticipated to include a re-publication of ISO 17799, a standard for information security measurement and metrics, and potentially a version of the current BS7799-3 standard.

Prior to the release of the ISO 27001 standard, organizations could only be certified against the British Standard Institute's BS7799-2 standard. Now organizations can obtain the ISO 27001 certification, as the BS7799-2 certification is being phased out, and the standard itself has been withdrawn...

It should also be noted that this certification scheme now aligns with other ISO schemes, such as those for ISO 9001 and ISO 14001.

This blog entry has some opinion on ISO 27001 too.

Do any of you recommend a certain standard to show your company is implementing effective security practices?

Read More

Certification & Accreditation Re-vitalization

Thanks to the newest SANS NewsBites (link will work shortly), I learned of the Certification & Accreditation Re-vitalization Initiative launched by the Chief Information Officer from the office of the Director of National Intelligence. According to this letter from retired Maj Gen Dale Meyerrose, the C&A process is too costly and slow, due to "widely divergent standards and controls, the lack of a robust set of automated tools and reliance upon manual review." He wants to "move from a posture of risk aversion to one of risk management, from a concept of information secuirty at all costs to one of getting the right information to the right people at the right time with some reasonable assurance of timeliness, accuracy, authenticity, security, and a host of other attributes."

That all sounds well and good, but it misses the key problem with C&A -- it doesn't prevent intrusions. It may be seen as a necessary condition for "securing" a system (which is not really possible anyway), but it is in no way sufficient. The forum set up to foster discussion of this initiative contains an insightful thought: Why do we have C&A at all? It's unfortunately that Gen Meyerrose didn't acknowledge that C&A doesn't provide much in the way of "security" at all, but that would admit that .gov and .mil have spent billions to no end. Woops.

Read More

GAO Hammers Common Criteria

I've written about Common Critera before. If you also think CC is a waste of money, read GAO: Common Criteria Is Not Common Enough by Michael Arnone. It summarizes and comments upon a report by the Government Accounting Office titled INFORMATION ASSURANCE: National Partnership Offers Benefits, but Faces Considerable Challenges. Mr. Arnone writes:

GAO also criticized the National Information Assurance Partnership (NIAP) for not providing metrics or evidence that the Common Criteria actually improves product security. In addition, the Common Criteria process takes so long to complete that agencies often find that the products they need are not on the list of certified offerings or that only older versions have been accredited, GAO’s report states...

Pescatore said GAO’s call for increased education and awareness of NIAP’s function is overblown. Large vendors already know the process well and can afford millions of dollars for tailor-made product evaluations, he said.

Any education efforts should target smaller vendors — with $10 million to $50 million a year in annual revenue — that don’t know about the NIAP process, don’t know how expensive it is and have trouble affording it, Pescatore said. NIAP must do more than educate, he added. It must provide subsidies or reduce prices so smaller vendors can participate, he said.

It sounds like Common Criteria is becoming nothing but a hurdle to keep smaller companies from providing products to government agencies.

Read More

FISMA Is a Joke

Thanks to SANS Newsbites I read the article FISMA Fizzles. I've written about FISMA before. The new article points me to a potential wise man who understands that FISMA is a joke: ex-Energy Department CIO Bruce Brody. This comment cut straight to the problem with FISMA:

OMB's FISMA implementation basically boils security down to paperwork exercises, and score card pressure ensures it stays that way. But that's not how cybersecurity works; it requires real-time monitoring, updating and patching, Brody says, which isn't necessarily reducible to a paper trail. (emphasis added)

Did I read "real-time monitoring"? Wow. Mr. Brody "gets it." Consider the alternative point of view:

FISMA has its defenders. An agency fully compliant with FISMA is a secure agency, says Scott Charbo, Homeland Security Department CIO. The law and cybersecurity are "the same thing in my mind," he says.

I see. Reading the DHS' grade history shows they have a perfect F record for the last three years. Just because DHS is in a sorry state and its scores are an F doesn't mean that an agency with straight A's is secure!

Let's get back to monitoring. Mr. Brody has correctly recognized that the absolute first priority for a security program is to figure out what is happening. If you have no idea what is happening in your enterprise, how can you expect to "secure" it? It doesn't even make sense to figure out what systems you have before you start monitoring. When you start watching traffic, intruders will show you your systems. The most vulnerable and/or interesting targets will get the most attention from the adversary, and you should address those first.

If you are a federal agency and you want to learn more about implementing monitoring, please contact me: richard at taosecurity dot com. I can teach you what to do, efficiently and cheaply. I may not be wearing my blue uniform any more, but I want to do my part. FISMA is not helping.

Read More

Thoughts on Recent Microsoft Common Criteria News

Through Slashdot I hunted down this story about certain Microsoft products being awarded Common Criteria (CC) Evaluation Assurance Level (EAL) 4 Augmented with ALC_FLR.3 certification. They include:

• Microsoft Windows Server™ 2003, Standard Edition (32-bit version) with Service Pack 1


• Microsoft Windows Server 2003, Enterprise Edition (32-bit and 64-bit versions) with Service Pack 1


• Microsoft Windows Server 2003, Datacenter Edition (32-bit and 64-bit versions) with Service Pack 1


• Microsoft Windows Server 2003 Certificate Server, Certificate Issuing and Management Components (CIMC) (Security Level 3 Protection Profile, Version 1.0)


• Microsoft Windows XP Professional with Service Pack 2


• Microsoft Windows XP Embedded with Service Pack 2

Achieving this certification is important to Microsoft, because of certain laws:

"[E]ffective 1 July 2002... departments and agencies within the Executive Branch shall acquire, for use on national security systems, only those COTS products or cryptographic modules that have been validated with the International Common Criteria for Information Technology Security Evaluation, the National Information Assurance Partnership (NIAP) Common Criteria Evaluation and Validation Scheme (CCEVS), or by the National Institute of Standards and Technology (NIST) Federal Information Processing Standards (FIPS) Cryptographic Module Validation Program."

It is important to remember that "EALs refer to the level of confidence in the conclusions of the evaluation, and not to the level of secrity the product provides. In other words, you can have more confidence that a EAL4 product performs as advertised than an EAL2 product... But an EAL4 product will not necessarily provide more security."

What really matters is the Protection Profile used to evaluate the products. I read that "[t]he Microsoft products were evaluated against the Controlled Access Protection Profile," (CAPP) which is available here (.pdf). Here are a few choice excerpts from the CAPP. Where you see "TOE", think "Microsoft system".

• The system does not have to defend itself against physical attacks: "The processing resources of the TOE will be located within controlled access facilities which will prevent unauthorized physical access."


• Sorry, I had to highlight this statement: "There will be one or more competent individuals assigned to manage the TOE and the security of the information it contains."


• The following implies that the evaluated system should not be a publicly accessible server: "Any other systems with which the TOE communicates are assumed to be under the same management control and operate under the same security policy constraints. CAPP-conformant TOEs are applicable to networked or distributed environments only if the entire network operates under the same constraints and resides within a single management domain."

If you continue reading the document, you'll find a great deal of requirements for keeping audit records, authorizing users, vendor-provided documentation, and so forth. This is probably not what people first imagine when they think of "secure" products.

Keep these assumptions in mind when you consider the importance of Microsoft products achieving EAL-4 certification.

Update: You can download the Windows XP / Server 2003 Common Criteria Evaluation Technical Report in .zip format.

Read More

Common Criteria

I received the September issue of the ISSA Journal. It contains several useful articles, with the most helpful to me being a humanly readable summary of the Common Criteria by Alex Ragen. I don't think Mr. Ragen clearly states who needs to purchase Common Criteria-validated products however.

His article's first sentence states:

"On July 1, 2002, the US Department of Defense began to enforce National Security Telecommunications and Information Systems Security Policy (NSTISSP) #11 (issued in January 2000), which mandates that US government agencies purchase only those IT security products which have been validated in accordance with Common Criteria and/or FIPS 140-1 or FIPS 140-2 as appropriate."

He also says:

"As mentioned earlier, US government agencies now require Common Criteria certification."

This is not true. According to the Committee on National Security Systems FAQ:

"The policy mandates, effective 1 July 2002, that departments and agencies within the Executive Branch shall acquire, for use on national security systems, only those COTS products or cryptographic modules that have been validated with the International Common Criteria for Information Technology Security Evaluation, the National Information Assurance Partnership (NIAP) Common Criteria Evaluation and Validation Scheme (CCEVS), or by the National Institute of Standards and Technology (NIST) Federal Information Processing Standards (FIPS) Cryptographic Module Validation Program.

Additionally, subject to policy and guidance for non-national security systems, NSTISSP # 11 notes that departments and agencies may wish to consider the acquisition of validated COTS products for use in information systems that may be associated with the operation of critical infrastructures as defined in the Presidential Decision Directive on Critical Infrastructure Protection (PDD-63)." [emphasis added]

Those bold sections make all the difference. This means that systems operated by the Department of Commerce, for example, that are not "national security systems," do not have to be validated by the Common Criteria. While some people discuss the possibility that Common Criteria would be extended beyong NSS, there is definitely no mandate to do so.

Read More

Common Malware Enumeration

This article describes the Common Malware Enumeration project. CME is a sister project to Mitre's Common Vulnerabilities and Exposures (CVE) initiative. CME will "assign unique identifiers to high priority malware events." This is a great idea, because anti-virus vendors, security researchers, and OS/application vendors will be able to refer to a common name rather than their internal representations for malware. DHS is funding the CME project.

Read More