Forensically Sound Evidence

Mike Murr pointed me to his blog post Forensically Sound Duplicate. He suggests replacing this definition of a forensically sound duplicate with the one that follows.

"A 'forensically-sound' duplicate of a drive is, first and foremost, one created by a method which does not, in any way, alter any data on the drive being duplicated. Second, a forensically-sound duplicate must contain a copy of every bit, byte and sector of the source drive, including unallocated 'empty' space and slack space, precisely as such data appears on the source drive relative to the other data on the drive. Finally, a forensically-sound duplicate will not contain any data (except known filler characters) other than which was copied from the source drive."

This is Mike's replacement:

"A forensically sound duplicate is a complete and accurate representation of the source evidence. A forensically sound duplicate is obtained in a manner that may inherently (due to the acquistion tools, techniques, and process) alter the source evidence, but does not explicitly alter the source evidence. If data not directly contained in the source evidence is included in the duplicate, then the introduced data must be distinguishable from the representation of the source evidence. The use of the term complete refers to the components of the source evidence that are both relevant, and reasonably believed to be relevant."

I agree with the statement "A forensically sound duplicate is a complete and accurate representation of the source evidence." That is broad and still accurate enough to refer to hard drives, memory, or network traffic. I'm not comfortable with the alteration portion of the suggested definition.

Read More

Forensics Warnings from CIO Magazine

The April 2006 issue of CIO Magazine features an article called CSI for the Enterprise?. It addresses the rise of electronic data discovery (eDiscovery in some quarters) tools. For a management magazine, the article makes several useful points:

Beware the Forensics Label

Many salespeople attach the label "forensics" to their security and compliance analysis tools, and that can be very misleading. In law enforcement circles, "forensics" means a well-defined set of discovery and investigative processes that hold up in court for civil or criminal proceedings. An enterprise that relies on these tools' records or analysis in, for example, a wrongful termination suit, is probably in for an unpleasant surprise. "It may not hold up in court," says Schwalm, a former Secret Service agent. "Very few vendors have an idea of what the requirements [are for proof, from a legal perspective]. They're really providing just a paper trail. You should challenge what the vendor means by ‘forensics capability,'" he adds.

One gotcha of using EDD tools for legal purposes is proving the inviolability of the data. Tools that keep or aggregate event logs may not provide access control that lets the enterprise prove that the underlying data is unaltered and accurate.

This issue is particularly critical because most vendors pitch their EDD tools as a way of detecting internal threats. Yet an insider is in the best position to access and alter data to cover his tracks or deflect blame to someone else, making truly secure access control and data management policies a must to even consider relying on EDD tools in a legal case. To thwart insider manipulations, critical functions such as setting up new vendors or changing payment destinations should require multiple levels of approval. "One person shouldn't be minding the whole store," says 2Checkout's Denman.

A related concern is being able to go back to the original raw data, since most EDD tools alter the original data to put it into a searchable database and to make formats from different types of monitoring appliances consistent. Such regularization is necessary to analyze the records, but to be legally effective, there must be a defensible way to show that it didn't distort the original data, says Gartner's Litan. (emphasis added)

Amen. Since we're talking about centralized logs, has anyone tried Splunk? This "Google for system logs" seems like a really neat idea.

To hear a vendor's take on the important of electronic data discovery tools, John Patzakis from Guidance Software wrote a good article called Why the eDiscovery Revolution is Important to InfoSec (.pdf, ISSA membership required). It's basically a cost-avoidance argument, like everything else should be with security. (There is no return on security investment.) John states:

In 2006, companies will spend $2 billion on eDiscovery services, and that figure is expected to climb to $3 billion in 2007... corporate information security can play a very key role in solving this critical problem by dramatically reducing costs and improving compliance... According to standard price lists from top eDiscovery providers, a company can expect to pay $11,000 to $15,000 for the processing of a single hard drive...

A key reason for these high costs involves the traditional role of outside counsels who represent the company and typically oversee and manage the eDiscovery process on a per-case basis. These law firms habitually rely on their own consultants to handle the eDiscovery needs of the case at hand, and both the law firm and their consultants typically approach the issue as a case-specific litigation support project. Thus, the focus is on addressing the immediate case, and not on solving the end client’s long-term problems by establishing a systematic methodology...

When a company transitions from outsourced eDiscovery to establishing a largely in-house process, the cost savings are dramatic. Many major organizations are now saving tens of millions of dollars in “hard” out-of-pocket costs annually when they turn to their internal resources and enterprise class computer investigation technology to collect, search and process computer data for eDiscovery.

In other words, buy Encase and do it yourself! Learn how to use Encase at the 2006 Computer and Enterprise Investigations Conference in Lake Las Vegas, NV. I am speaking there on Thursday, 4 May 2006 from 1400-1530 on Network Forensics.

Read More

Pulling the Plug in 2005

Every time I attend a USENIX conference, I gather free copies of the ;login: magazine published by the association. The August 2005 issue features some great stories, with some of them available right now to non-USENIX members. (USENIX makes all magazine articles open to the public one year after publication. For example, anyone can now read the entire December 2004 issue.)

An article which caught my eye was Forensics for System Administrators by Sean Peisert. Although the USENIX copy of the article won't be published until August 2006, you can read Sean's copy here (.pdf).

I thought the article was proceeding well until I came across this advice.

"What happens when there is some past event that a system administrator wishes to understand on their system? Where should the administrator, now a novice forensic analyst, begin? There are many variables and questions that must be answered to make proper decisions about this. Under almost all circumstances in which the system can be taken down to do the analysis, the ideal thing to do is halt or power-off the system using a hardware method." (emphasis added)

Is he serious? The article continues:

"[T]he x86 BIOS does not have a monitor mode that supports this [a hardware interrupt]. The solution for everyone else? Pull the plug. The machine will power off, the disk will remain as-is, and there will be no possibility of further contamination of the evidence through some sort of clean-up script left by the intruder, as long as the disk is not booted off or mounted in read/write mode again. The reason for stopping a machine is that it prevents further alteration of the evidence. The reason for halting with a hardware interrupt, rather than using the UNIX halt or shutdown command is that if a root compromise occurred, those commands could have been trojaned by an intruder to clean up evidence."

I can't believe I'm reading this advice in 2005, only 6 days from 2006. This is the advice I heard nearly 10 years ago. "Pulling the plug" as the first step in a forensic investigation is absolutely terrible advice. I am not a host-based forensics guru, but I know that a live response, first described in the June 2001 book Incident Response by Mandia, Prosise, and Pepe, should be part of even the most basic forensically-minded sys admin's techniques. Sean could have even looked into the ;login: archives to find Keith Jones' article in the November 2001 issue describing live response.

Live response is a technique to retrieve volatile information from a running system in a forensically sound manner. Live response can be frustrated by some binary and kernel alteration techniques, but it is a good (non-network-centric) first step whenever a host is suspected of being compromised. Those who want to know more about live response, and see how helpful the advice can be, will enjoy reading Real Digital Forensics.

Sean tries to defend pulling the plug here:

"In our first example intrusion, I took a preliminary look at the syslog and saw that dates of suspicious logins went back at least three weeks. Given that the intrusion seemed to be going on for so long, I decided that I could no longer trust the system to reliably and accurately report evidence about itself. Therefore, pulling the plug on the machine was the best option."

That is a really weak excuse. Certainly a non-ankle-biter attacker will take steps to hide his presence. That does not mean that no attempt should be made to collect volatile system information!

Sean continues:

"It is certainly the case that halting a system can help perserve more evidence, particularly that in swap, slack, or otherwise unallocated space on disk. But it also can destroy some evidence. For example, halting a system will wipe out the contents of memory, hindering the ability of an analyst to dump a memory image to disk. However, in the forensic discussions in this article, slack space and memory dumps are outside the scope of our analysis. In our case, halting a system merely helped to preserve real evidence, and had the intrusion in our first example been discovered sooner, and the system sooner halted as a result, the intruder would have had less time to cover their tracks. Then, as I will discuss, certain helpful log files that were deleted may have been recoverable."

If Sean is worried that an intruder will take actions to "cover their tracks," then the live response can be performed after the victim host has been cut off from the Internet. Sure, the most 31337 attackers may detect this and start self-cleansing procedures, but how often does that happen? Also, collecting live response data does not usually trigger any cleaning mechanisms. The sort of data one collects is the normal information a system administrator might inspect during the course of regular duties.

The fundamental issue here is whether pulling the plug should be the first response activity or not. In my experience, cutting off remote access is the first step. Analysis of NSM data involving the target host is second. Live response is the third. Forensic duplication and analysis is the fourth, if the previous two steps point to compromise and the resources for investigation and available.

This part of the article makes me sad:

"This material is based on work sponsored by the United States Air Force and supported by the Air Force Research Laboratory under Contract F30602-03-C-0075 and performed in conjunction with Lockheed Martin Information Assurance. Thanks to Sid Karin, Abe Singer, Matt Bishop, and Keith Marzullo, who provided valuable discussions during the writing of this article."

First, why is the Air Force paying for advice that should have been abandoned in 1998, the last time I remember the Air Force suggesting these sorts of actions? Second, why didn't any of the article reviewers speak out against this bad advice?

Read More

Guidance Software 0wn3d

This morning I read stories by Brian Krebs and Joris Evers explaining how Guidance Software, maker of host-based forensics suite Encase, was compromised. Guidance CEO John Colbert claims "a person compromised one of our servers," including "names, addresses and credit card details" of 3,800 Guidance customers. Guidance claims to have learned about the intrusion on 7 December. Victim Kessler International reports the following:

"Our credit card fraud goes back to Nov. 25. If Guidance knew about it on Dec. 7, they should have immediately sent out e-mails. Why send out letters through U.S. mail while we could have blocked our credit cards?"

Guidance could face severe financial trouble. According to reporter Joris Evers:

"Guidance stored customer names and addresses and retained card value verification, or CVV, numbers, Colbert said. The CVV number is a three-digit code found on the back of most credit cards that is used to prevent fraud in online and telephone sales. Visa and MasterCard prohibit sellers from retaining CVV once a transaction has been completed."

Reporter Krebs explains the implications:

"Companies that violate those standards can be fined $500,000 per violation. Credit card issuers generally levee such fines against the bank that processes payment transactions for the merchant that commits the violations. The fines usually are passed on to the offending company."

Since Guidance's customers include "hundreds of security researchers and law enforcement agencies worldwide, including the U.S. Secret Service, the FBI and New York City police," I don't think those customers will tolerate this breach of trust.

Why did it take Guidance at least 12 days (from the first known fraudulent purchases on 25 Nov to the reported discovery on 7 Dec) to learn they were owned? I think this is an example of a company familiar with creating host-centric forensic software, but unfamiliar with sound operational security and proper policy, architecture, and monitoring to prevent or at least detect intrusions. Furthermore, who will be fired and/or fined for storing CVVs indefinitely?

Read More

Network Forensics? Please.

Today I looked at the Interop New York 2005 Schedule and noticed an item called "Network Forensic Day" taught by Pine Mountain Group. I try to stay current with people and companies performing security work, but I had never heard of PMG. I looked at the description of the course, wondering if the "network" meant "enterprise," as in "how to use forensics in the enterprise." I think that is a misapplication of the term network in that context, but it's common enough. Alternatively, perhaps "network" meant "traffic," which is how I use the term.

When I mention "network forensics," I define it as the art of collecting, protecting,
analyzing, and presenting network traffic to support remediation or prosecution. This is in line with the definition of forensics:

"1. The art or study of formal debate; argumentation.
2. The use of science and technology to investigate and establish facts in criminal or civil courts of law."

It turns out PMG's use of the term "Network Forensics" has nothing to do with any recognized application of the term. They say:

"Network Forensics is the study of the micro transactions of inter-network components, platforms and the applications that process on and across them.

By taking a forensic measurement of a micro transaction, quantifying the repeated dependency on the micro to that of the macro we can quantify the improvement for an end user that specific IT optimizations might provide. On the business process side, quantification of the cost of the macro transaction time spent by an end user can be quantified in annual cost or lost productivity associated with slow applications. Knowing optimization improvements and their associated costs allows a long term ROI to be considered. The result? Best bang for the buck optimization!

Come join PMG NetAnalyst in a day of cross technology, vendor independent network training with a twist: PMG will take you on a journey down several complex multi-vendor network environments where troubles abound. You will be taught how to use a well rounded 'bag of tools' to analyze and troubleshoot the issues as well as how applying best practices could have avoided these issues. Forensics Day will show you how to save money as well as improve performance and reliability by using 'brain cells' instead of budget to solve and even prevent problems."

Please. This is not "network forensics" by any stretch of the imagination. This is an attempt to add a sexy name to the otherwise boring ideas of network troubleshooting. The latest iteration and expansion of the concept uses the term Business Service Management, which I learned about recently though the 1 September 2005 Network Computing magazine.

I understand there are similar uses of the term "forensics" outside of the legal realm. However, "network forensics" has had a security association for years. I would like to see it stay that way to avoid further cluttering our professional landscape.

Read More

Using md5deep

Thank you to Harlan Carvey for reminding me of Jesse Kornblum's md5deep. md5deep is a suite of tools to recursively compute a variety of hashes. It is not limited to the MD5 algorithm. In the example below I run the sha256deep tool to provide sha256 hashes of various files. The -r flag initializes recursive behavior, and -z says display file size before the hash.

bourque:/home/analyst$ sha256deep -r -e -z *

     93506  1a6da6a2a849eb27fb7522939afab63ec59bcdb9412c2460fe611543b573d95f  
/home/analyst/2005-041-santini_air/sample

       111  43450978e07f87dfbc4918fec928209c54f4d5804367960fbde617e71ee50985
/home/analyst/2005-041-santini_air/sample.sha256

209.180.018.089.02001-156.023...: 391MB of 1405MB done, 00:01:22 left    

The last entry shows sha256deep is busy computing the hash for a 1405 MB file. By passing the -e flag, I told the program to estimate time until hash completion. This is useful for processing large files. The resulting hash is eventually shown below.

1473577526  3f4eb24ae943dba4bdb1126540d309854824ac64ff6f288020c9c2bdc4793de9  
/home/analyst/2005-041-santini_air/209.180.018.089.02001-156.023.170.238.02001

md5deep and related tools simplify maintaining forensic evidence as the program can rapidly produce hashes in an investigator-friendly format. There's also a FreeBSD port. For forensic applications, you would save the hashes to a file instead of standard output.

Read More