New Online Packet Repository

As of a few weeks ago I am no longer involved with OpenPacket.org. One of the reasons is a great new online packet repository sponsored and run by Mu Dynamics called Pcapr. I've had an account there for a few months, but it looks like the site is now open to the general public. Check it out -- there's a lot of cool features already.

---

Richard Bejtlich is teaching new classes in DC and Europe in 2009. Register by 1 Jan and 1 Feb, respectively, for the best rates.

Read More

Bejtlich Keynote at 1st ACM Workshop on Network Data Anonymization

Brian Trammell and Bill Yurcik were kind enough to ask me to deliver the keynote at the 1st ACM Workshop on Network Data Anonymization (NDA 2008). The one day event takes place 31 October 2008 at George Mason University in northern VA. My talk will discuss the trials and tribulations of OpenPacket.org, and changes planned for the project.

Read More

Packet Anonymization with PktAnon

I noticed a new tool on Packetstorm recently: PktAnon by Christoph P. Mayer, Thomas Gamer, and Dr. Marcus Schöller.

This tool seems powerful because you can apply a variety of anonymization policies based on settings you apply in an XML configuration file.

It was easy to install the tool on Debian 4.0:

tws:~# cd /usr/local/src
tws:/usr/local/src# wget http://www.tm.uka.de/pktanon/download/pktanon-1.2.0-dev                                                                                                    .tar.gz
...edited...
tws:/usr/local/src# tar -xzf pktanon-1.2.0-dev.tar.gz
tws:/usr/local/src# http://www.tm.uka.de/pktanon/download/pktanon-1.2.0-dev.tar.                                                                                                    gz
tws:/usr/local/src# sudo apt-get install libxerces27-dev libboost-dev
-su: sudo: command not found
tws:/usr/local/src# apt-get install libxerces27-dev libboost-dev
Reading package lists... Done
Building dependency tree... Done
The following extra packages will be installed:
  libicu36 libxerces27
Suggested packages:
  libboost-doc libboost-date-time-dev libboost-filesystem-dev
  libboost-graph-dev libboost-iostreams-dev libboost-program-options-dev
  libboost-python-dev libboost-regex-dev libboost-serialization-dev
  libboost-signals-dev libboost-test-dev libboost-thread-dev libboost-wave-dev
  xalan libxerces27-doc
The following NEW packages will be installed:
  libboost-dev libicu36 libxerces27 libxerces27-dev
0 upgraded, 4 newly installed, 0 to remove and 3 not upgraded.
Need to get 9259kB of archives.
After unpacking 44.7MB of additional disk space will be used.
Do you want to continue [Y/n]? y
...edited...
tws:/usr/local/src# cd pktanon-1.2.0-dev
tws:/usr/local/src/pktanon-1.2.0-dev# mkdir /usr/local/pktanon
tws:/usr/local/src/pktanon-1.2.0-dev# ./configure --prefix=/usr/local/pktanon
tws:/usr/local/src/pktanon-1.2.0-dev# make
tws:/usr/local/src/pktanon-1.2.0-dev# make install

Next you choose which of the anonymization profiles we want. Here we use settings_high.xml. To use this configuration file we just tell it where the Input is and where the Output is.

For example, here is the first, original packet.

tws:/tmp# tcpdump -c 1 -r sample.ftp.pcap -neXvvv

reading from file sample.ftp.pcap, link-type EN10MB (Ethernet)
09:38:37.565642 00:0c:29:2d:6a:a0 > 00:50:56:ee:e5:fc, ethertype IPv4 (0x0800), 
length 74: (tos 0x0, ttl  64, id 48680, offset 0, flags [DF], proto: TCP (6), 
length: 60) 192.168.255.131.1385 > 62.243.72.50.21: S, cksum 0x7890 (correct), 
2888152290:2888152290(0) win 5840 <mss 1460,sackOK,timestamp 199370 0,nop,wscale 2>
        0x0000:  4500 003c be28 4000 4006 3542 c0a8 ff83  E..<.(@.@.5B....
        0x0010:  3ef3 4832 0569 0015 ac25 b4e2 0000 0000  >.H2.i...%......
        0x0020:  a002 16d0 7890 0000 0204 05b4 0402 080a  ....x...........
        0x0030:  0003 0aca 0000 0000 0103 0302            ............

Here is the settings_low profile output.

tws:/tmp# tcpdump -c 1 -r anon.low.ftp.pcap -neXvvv

reading from file anon.low.ftp.pcap, link-type EN10MB (Ethernet)
09:38:37.565642 00:0c:29:2d:6a:a0 > 00:50:56:ee:e5:fc, ethertype IPv4 (0x0800), 
length 74: (tos 0x0, ttl  64, id 48680, offset 0, flags [DF], proto: TCP (6),
length: 60) 246.142.91.186.1385 > 90.113.151.13.21: S, cksum 0x7c1a (correct),
2888152290:2888152290(0) win 5840 <mss 1460,sackOK,timestamp 199370 0,nop,wscale 2>
        0x0000:  4500 003c be28 4000 4006 38cc f68e 5bba  E..<.(@.@.8...[.
        0x0010:  5a71 970d 0569 0015 ac25 b4e2 0000 0000  Zq...i...%......
        0x0020:  a002 16d0 7c1a 0000 0204 05b4 0402 080a  ....|...........
        0x0030:  0003 0aca 0000 0000 0103 0302            ............

I decided I wanted a low profile that also modified MAC addresses, so I copied the low setting and then made this change:

<configitem anon="AnonBytewiseHashSha1" name="MacSource"/>
<configitem anon="AnonBytewiseHashSha1" name="MacDest"/>

This was the result.

tws:/tmp# tcpdump -c 1 -r anon.low-mac.ftp.pcap -neXvvv
reading from file anon.low-mac.ftp.pcap, link-type EN10MB (Ethernet)
09:38:37.565642 da:cb:dc:54:d2:51 > da:28:8d:39:ef:7b, ethertype IPv4 (0x0800),
length 74: (tos 0x0, ttl  64, id 48680, offset 0, flags [DF], proto: TCP (6),
length: 60) 246.142.91.186.1385 > 90.113.151.13.21: S, cksum 0x7c1a (correct), 
2888152290:2888152290(0) win 5840 <mss 1460,sackOK,timestamp 199370 0,nop,wscale 2>
        0x0000:  4500 003c be28 4000 4006 38cc f68e 5bba  E..<.(@.@.8...[.
        0x0010:  5a71 970d 0569 0015 ac25 b4e2 0000 0000  Zq...i...%......
        0x0020:  a002 16d0 7c1a 0000 0204 05b4 0402 080a  ....|...........
        0x0030:  0003 0aca 0000 0000 0103 0302            ............

Finally I ran the medium and high settings.

tws:/tmp# tcpdump -c 1 -r anon.medium.ftp.pcap -neXvvv
reading from file anon.medium.ftp.pcap, link-type EN10MB (Ethernet)
09:38:37.565642 da:cb:dc:54:d2:51 > da:28:8d:39:ef:7b, ethertype IPv4 (0x0800), 
length 60: (tos 0x0, ttl 116, id 48680, offset 0, flags [DF], proto: TCP (6), 
length: 40) 21.248.227.61.19357 > 172.148.57.189.56062: S, cksum 0x31e7
(correct), 2888152290:2888152290(0) win 5840
        0x0000:  4500 0028 be28 4000 7406 6920 15f8 e33d  E..(.(@.t.i....=
        0x0010:  ac94 39bd 4b9d dafe ac25 b4e2 0000 0000  ..9.K....%......
        0x0020:  5002 16d0 31e7 0000 0000 0000 0000       P...1.........

tws:/tmp# tcpdump -c 1 -r anon.high.ftp.pcap -neXvvv
reading from file anon.high.ftp.pcap, link-type EN10MB (Ethernet)
09:38:37.565642 55:3e:4d:bf:1f:e8 > 55:35:a0:67:f1:3a, ethertype IPv4 (0x0800), 
length 60: (tos 0x0, ttl 126, id 48680, offset 0, flags [DF], proto: TCP (6), 
length: 40) 162.131.129.172.20319 > 97.102.43.234.21842: S, cksum 0xb113
(correct), 2888279266:2888279266(0) win 5907
        0x0000:  4500 0028 be28 4000 7e06 8d27 a283 81ac  E..(.(@.~..'....
        0x0010:  6166 2bea 4f5f 5552 ac27 a4e2 2080 2000  af+.O_UR.'......
        0x0020:  5002 1713 b113 0000 0000 0000 0000       P.............

We should be able to try this tool with OpenPacket.org. Let me know what you think.

For details on the anonimization policies please read the documentation.

Read More

OpenPacket.org 1.0 Is Live

Nearly three years after the initial post describing the idea , I am happy to report that OpenPacket.org 1.0 is ready for public use, free of charge.

The mission of OpenPacket.org is to provide quality network traffic traces to researchers, analysts, and other members of the digital security community. One of the most difficult problems facing researchers, analysts, and others is understanding traffic carried by networks. At present there is no central repository of traces from which a student of network traffic could draw samples. OpenPacket.org will provide one possible solution to this problem.

Analysts looking for network traffic of a particular type can visit OpenPacket.org, query the OpenPacket.org capture repo for matching traces, and download those packets in their original format (e.g., Libpcap, etc.). The analyst will be able to process and analyze that traffic using tools of their choice, like Tcpdump, Snort, Ethereal, and so on.

Analysts who collect their own traffic will be able to submit it to the OpenPacket.org database after they register.

Anonymous users can download any trace that's published. Only registered users can upload. This system provides a level of accountability for trace uploads.

Our moderators will review the trace to ensure it does not contain any sensitive information that should not be posted publicly. Besides appearing on the site, once a trace has been published you can receive notice of it via this published trace RSS feed.

If you have any doubt regarding the publication of a trace, do not try to submit it. When moderators are unsure of the nature of a trace, we will reject it. OpenPacket.org is not a vehicle for publishing enterprise data as contained in network traffic.

I would like to thank all the people who submitted suggestions and did feature testing via the openpacket-devel mailing list. If you have issues regarding usage of the site, consider subscribing to the openpacket-users mailing list or post to the OpenPacket.org Forums.

As time permits I will probably post more on how to use OpenPacket.org strictly on the OpenPacket Blog. I will minimize cross-posting to TaoSecurity Blog and OpenPacket Blog.

I save my final thanks for Sharri Parsell, our Web developer, and JJ Cummings for hosting OpenPacket.org. Without your work we would not have a site!

Read More

OpenPacket.org Developments

I am happy to report that work on OpenPacket.org is back on track, thanks to a new volunteer Web application developer.

Please read the rest of the story at the Openpacket.org Blog.

Read More

OpenPacket.org Update

I just posted news on OpenPacket.org at the OpenPacket Blog. I made an initial announcement about OpenPacket last year. In short, this project is going nowhere unless I get some help with development or financing, due to my lack of Web development skill and time. I appreciate any comments you might post on the OpenPacket Blog.

Update: Please visit the OpenPacket Blog for fresh updates. I created devel and users mailing lists, and two people have already volunteered development help. Wow!

Read More

Snort.org Posts BlackWorm Packet Captures

The folks at Sourcefire have done the analyst community a great service by posting traffic captures of CME-24, aka "BlackWorm". Kudos also to the Common Malware Enumeration project for providing an easy way to reference malware! Once OpenPacket.org gets going, I hope to host these sorts of captures there.

Update: Check out this Sourcefire VRT analysis.

Read More

Thank You for Another Great Year

Exactly one year ago today I posted a thank-you note for the great year of blogging in 2004. A look at the 2004 statistics shows as recently as July 2004, this blog had less than 6,000 visitors per month, as tracked by Sitemeter. I have no idea how Atom, RSS, and other republishing is affects those statistics. Soon after my first book was published, we broke through the 10,000 per month mark and have never looked back.

As you can see from the 2005 chart above, we're at the 22,000 per month mark now, and broke through 25,000 in August during my coverage of Ciscogate. This blog continues to be a nonpaying venture, despite offers to commercialize, syndicate and repackage the content elsewhere. Others already do this without my permission, but I thank those more responsible people who ask before posting my content elsewhere. For example, I've given the great publisher Apress blanket permission to quote anything I say here. This is my small way to say thank you for the books they've sent me to review. One of my New Year's resolutions for 2006 is to dedicate specific time early each morning (before my 1 year old daughter wakes up) to read, review, and recommend books. I managed to read and review 26 technical books in 2005, but I have a backlog of over 50 waiting for attention.

I read every book upon which I make comments at Amazon.com, unlike some others who consider a rehash of a book's back cover to be a "review." I also try to avoid bad books, so don't expect too many low-star reviews.

I have found your comments to be one of the best parts of blogging in 2005. I really appreciate hearing what you have to say, either publicly as a blog comment or privately via email. I don't have time to reply to the few of you who send me multi-page running commentaries on everything I publish or blog, but I appreciate your thoughts nevertheless.

In 2006 I plan to continue blogging about subjects which interest me, like network security monitoring, incident response, forensics, FreeBSD, and related topics. I welcome any thoughts on other issues you find pressing. If you want to see how I keep track of world security events, please visit my interests page. Those are my bookmarks; I avoid browser bookmarks whenever possible.

In 2006 I also plan to devote time and resources to OpenPacket.org. Many of you have offered some form of support. As that project develops I will request assistance, either here or on the OpenPacket.org Blog. 2006 should also be a big year for TaoSecurity, my company. I am not sure if 2006 will be the year I decide to hire employees, but I am considering hiring contract help for some in-house coding projects. These projects would support the company's consulting, incident response, and forensics services. Should anything be of use to the wider community, it will appear on the TaoSecurity products page. If you would be interested in working for TaoSecurity, please feel free to send me your resume in .pdf format to richard at taosecurity dot com. I am always interested in meeting security practitioners who can administer systems properly, perform network- and host-centric incident response and forensics, write security tools, speak and publish original material, and seek to save the world one packet at a time.

I have ideas for additional, specialized training courses for 2006. At the moment demand for private 4-day Network Security Operations classes has been strong. I am working with a few different customers to support specialized training outside the core NSO focus. Some of those endeavors may be offered to the public. I will also submit proposals to speak at a few more USENIX conferences, which are public opportunities for training in network security monitoring. I post word of any place I intend to speak at my events list.

I do not have any new books scheduled for writing in 2006. Having authored or co-authored three books in three years, I expect to take a break. I have ideas for more articles like the one in Information Security Magazine. I should have an article in the February 2006 Sys Admin Magazine on keeping FreeBSD up-to-date.

My family and I wish all of you a prosperous 2006!

Read More

Marcus Sachs in SC Magazine

I was pleased to hear what Marcus Sachs is working on, courtesy of an interview by Illena Armstrong and Marcia Savage in this month's SC Magazine. I first met Marcus when I was an Air Force captain at the AFCERT and he was an Army Major at the JTF-CND.

Marcus mentioned a project that caught my attention:

"We're also building a database of large data sets collected from the internet. The intent is to help researchers who might be working on a new security device.

Rather than trying to connect to their own networks and pull live data in from their university network, or wherever they are doing the research, we want to provide them with real data sets that have been collected from the internet, but properly sanitized and anonymized...

In a technical sense, this is easy. All you have to do is hook a computer up and start recording. But you end up picking up a lot of private information. We have been working on this with lawyers, the Electronic Privacy Information Center (EPIC) and the Electronic Freedom Foundation, among others.

This is going to be remarkable because we'll be able to create anonymous data sets that actually reflect what the hostile internet looks like, but that the privacy people are OK with. Both the public and the private sector will have access to this database, but they will have to be vetted if they want to use it. If Al Qaeda wants access, we won't grant it, but if the Atlanta Police Department wants to train some cybercops, then that's cool.

SC: When will this be done?

Sachs: We're getting close. We could see this come online later this year or early next year. We're just working out the last details with the lawyers."

This sounds very interesting, and similar to OpenPacket.org. When I finally get some time to work on OpenPacket, it should be similar to Marcus' project. However, I don't intend to limit who can download the data. I plan to host network traffic in Libpcap format. I wonder if Marcus will offer the same, or more (hard drive images, like our book Real Digital Forensics, maybe).

Read More

Request for Help with OpenPacket.org

Earlier this month I announced work on OpenPacket.org, a free site providing quality network traffic traces to researchers, analysts, and other members of the digital security community.

We are looking for help in two areas:

1. Open source content management systems (CMS) experience: We believe we will use a CMS to accept, moderate, and present traffic captures to users. We need help planning and deploying a CMS that will meet our needs.


2. Open source database experience: We will use an open source database like MySQL or PostgreSQL, as compatible with the CMS we choose. We need help planning and deploying a database schema, and we will need guidance on configuring the database properly. Most of the OpenPacket.org crew has database experience as it relates to supporting intrusion detection sensors, but storing and retrieving the sorts of data we have in mind is probably outside our daily routine.

We have ideas for additional OpenPacket.org functionality, but providing ways to accept, moderate, and present traces in Libpcap format is the primary goal of our first version of OpenPacket.org.

If you are interested in helping with either subject, please email richard at taosecurity dot com.

If you have any comments, as always they are welcome here. Thank you.

Read More

OpenPacket.org Initial Announcement

I would like to announce that I am working on a project called OpenPacket.org. The mission of OpenPacket.org is to provide quality network traffic traces to researchers, analysts, and other members of the digital security community. One of the most difficult problems facing researchers, analysts, and others is understanding traffic carried by networks. At present there is no central repository of traces from which a student of network traffic could draw samples. OpenPacket.org will provide one possible solution to this problem.

Analysts looking for network traffic of a particular type will visit OpenPacket.org, query the OpenPacket.org Database for matching traces, and download those packets in their original format (e.g., Libpcap, etc.). The analyst will be able to process and analyze that traffic using tools of their choice, like Tcpdump, Snort, Ethereal, and so on.

Analysts who collect their own traffic will be able to submit it to the OpenPacket.org database, assuming it is suitable for public review and meets guidelines to be announced later.

I am currently working with some friends and colleagues on this project. We hope to have OpenPacket.org up and running before the end of the year. At present the OpenPacket.org domain name is "parked," and soon it will simply forward to this blog entry. As we enter Alpha and then Beta status, more will be available through that domain name.

Read More

1000th Post

This is the 1000th TaoSecurity Blog post. Thankfully, after being broken for months, Blogger fixed the post tracking counter in time for me to notice this milestone.

I started the blog on 8 January 2003 as a place to post word of new Amazon.com book reviews. I haven't read a new book since May, because I have been extremely busy launching my new company TaoSecurity. I plan to resume reading books very shortly, probably starting with Extreme Exploits.

The blog has now evolved into a place where I record tips on using FreeBSD and other operating systems and applications. I also post thoughts on network security monitoring and related security topics. I constantly refer back to posts here to remember how I configured a program or what my thoughts were on a certain subject. I detest keeping bookmarks, so I try to store anything of value here. A bookmark has no context and says nothing about how or why I recorded it. In brief, this blog helps me keep a grip on developments in the tech world.

Looking ahead, I have two new projects in store: the TaoSecurity Podcast and a resource I'm calling OpenPacket.org. You can expect to read more about these in the fourth quarter of this year. I appreciate everyone who reads this blog and I especially enjoy reading your comments and emails. Our next milestone is the blog's third birthday in January, so I hope to see you then!

Read More