Recovering from Suricata Gone Wild

Recently I tried interacting with one of my lab Security Onion sensors running the Suricata IDS. I found the Sguil server was taking a really long time to offer services on port 7734 TCP. Since I hadn't worked with this lab system in a while, I guessed that there might be too many uncategorized events in the Sguil database. I dusted off an old blog post titled More Snort and Sguil Tuning from 2006 and took a look at the system.

First I stopped the NSM applications on the server.

sudo service nsm stop
Stopping: securityonion
  * stopping: sguil server                                [  OK  ]
Stopping: HIDS
  * stopping: ossec_agent (sguil)                         [  OK  ]
Stopping: Bro
stopping ds61so-eth1-1 ...
stopping proxy ...
stopping manager ...
Stopping: ds61so-eth1
  * stopping: netsniff-ng (full packet data)              [  OK  ]
  * stopping: pcap_agent (sguil)                          [  OK  ]
  * stopping: snort_agent (sguil)                         [  OK  ]
  * stopping: suricata (alert data)                       [  OK  ]
  * stopping: barnyard2 (spooler, unified2 format)        [  OK  ]
  * stopping: prads (sessions/assets)                     [  OK  ]
  * stopping: sancp_agent (sguil)                         [  OK  ]
  * stopping: pads_agent (sguil)                          [  OK  ]
  * stopping: argus                                       [  OK  ]
  * stopping: http_agent (sguil)                      

Next I ran a query to look for the top uncategorized events.

$ mysql -uroot
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1639
Server version: 5.5.29-0ubuntu0.12.04.1 (Ubuntu)

Copyright (c) 2000, 2012, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> use securityonion_db;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed

mysql> SELECT COUNT(signature)as count, signature FROM event WHERE status=0 GROUP BY signature ORDER BY count DESC LIMIT 20;
+---------+----------------------------------------------------------------------------------+
| count   | signature                                                                        |
+---------+----------------------------------------------------------------------------------+
| 2299160 | SURICATA STREAM Packet with invalid ack                                          |
| 2298505 | SURICATA STREAM ESTABLISHED invalid ack                                          |
| 1777530 | SURICATA STREAM ESTABLISHED packet out of window                                 |
|   38700 | SURICATA STREAM ESTABLISHED retransmission packet before last ack                |
|   24181 | SURICATA STREAM TIMEWAIT ACK with wrong seq                                      |
|    5430 | ET POLICY GNU/Linux APT User-Agent Outbound likely related to package management |
|    3160 | SURICATA STREAM Last ACK with wrong seq                                          |
|     753 | ET POLICY Dropbox.com Offsite File Backup in Use                                 |
|     637 | SURICATA HTTP unknown error                                                      |
|     626 | SURICATA STREAM SHUTDOWN RST invalid ack                                         |
|     505 | SURICATA STREAM FIN1 FIN with wrong seq                                          |
|     494 | SURICATA HTTP request field too long                                             |
|     448 | ET POLICY PE EXE or DLL Windows file download                                    |
|     315 | ET RBN Known Malvertiser IP (22)                                                 |
|     270 | ET POLICY iTunes User Agent                                                      |
|     266 | SURICATA STREAM CLOSEWAIT ACK out of window                                      |
|     237 | ET INFO EXE IsDebuggerPresent (Used in Malware Anti-Debugging)                   |
|     219 | ET POLICY MOBILE Apple device leaking UDID from SpringBoard                      |
|     217 | SURICATA STREAM 3way handshake with ack in wrong dir                             |
|     151 | SURICATA STREAM FIN2 FIN with wrong seq                                          |
+---------+----------------------------------------------------------------------------------+
20 rows in set (15.24 sec)

Wow, that's a lot of SURICATA STREAM events. I need to categorize them as non-issues to recover the Sguil server.

mysql> UPDATE event SET status=1, last_modified='2013-02-24 16:26:00', last_uid='sguil' WHERE event.status=0 and event.signature LIKE 'SURICATA STREAM%';
Query OK, 6443375 rows affected, 65535 warnings (3 min 4.89 sec)
Rows matched: 6443375  Changed: 6443375  Warnings: 6443375

Let's see what the database thinks now.

mysql> SELECT COUNT(signature)as count, signature FROM event WHERE status=0 GROUP BY signature ORDER BY count DESC LIMIT 20;
+------+-----------------------------------------------------------------------------------------+
| cnt  | signature                                                                               |
+------+-----------------------------------------------------------------------------------------+
| 5430 | ET POLICY GNU/Linux APT User-Agent Outbound likely related to package management        |
|  753 | ET POLICY Dropbox.com Offsite File Backup in Use                                        |
|  637 | SURICATA HTTP unknown error                                                             |
|  494 | SURICATA HTTP request field too long                                                    |
|  448 | ET POLICY PE EXE or DLL Windows file download                                           |
|  315 | ET RBN Known Malvertiser IP (22)                                                        |
|  270 | ET POLICY iTunes User Agent                                                             |
|  237 | ET INFO EXE IsDebuggerPresent (Used in Malware Anti-Debugging)                          |
|  219 | ET POLICY MOBILE Apple device leaking UDID from SpringBoard                             |
|  133 | ET INFO PDF Using CCITTFax Filter                                                       |
|  106 | ET POLICY Pandora Usage                                                                 |
|   97 | ET CHAT Facebook Chat (buddy list)                                                      |
|   93 | ET POLICY MOBILE Apple device leaking UDID from SpringBoard via GET                     |
|   58 | ET POLICY Internal Host Retrieving External IP via whatismyip.com - Possible Infection  |
|   41 | PADS New Asset - ssl TLS 1.0 Client Hello                                               |
|   39 | SURICATA HTTP response header invalid                                                   |
|   39 | ET CURRENT_EVENTS Exploit Kit Delivering Compressed Flash Content to Client             |
|   36 | ET POLICY Python-urllib/ Suspicious User Agent                                          |
|   36 | ET MALWARE Possible Windows executable sent when remote host claims to send a Text File |
|   28 | ET POLICY Http Client Body contains pw= in cleartext                                    |
+------+-----------------------------------------------------------------------------------------+
20 rows in set (0.03 sec)

That's much better.

Before restarting the NSM services, I edit the autocat.conf file to add the following.

none||ANY||ANY||ANY||ANY||ANY||ANY||%%REGEXP%%^SURICATA STREAM||1

This will auto-categorize any SURICATA STREAM alerts as non-issues. I want to keep adding events to the database for testing purposes, but I don't want to see them in the console.

Now I restart the NSM services.

sudo service nsm start
Starting: securityonion
  * starting: sguil server                                                                [  OK  ]
Starting: HIDS
  * starting: ossec_agent (sguil)                                                         [  OK  ]
Starting: Bro
starting manager ...
starting proxy ...
starting ds61so-eth1-1 ...
Starting: ds61so-eth1
  * starting: netsniff-ng (full packet data)                                              [  OK  ]
  * starting: pcap_agent (sguil)                                                          [  OK  ]
  * starting: snort_agent (sguil)                                                         [  OK  ]
  * starting: suricata (alert data)                                                       [  OK  ]
  * starting: barnyard2 (spooler, unified2 format)                                        [  OK  ]
  * starting: prads (sessions/assets)                                                     [  OK  ]
  * starting: pads_agent (sguil)                                                          [  OK  ]
  * starting: sancp_agent (sguil)                                                         [  OK  ]
  * starting: argus                                                                       [  OK  ]
  * starting: http_agent (sguil)                                                          [  OK  ]
  * disk space currently at 22%

I check to see if port 7734 TCP is listening.

sudo netstat -natup | grep 7734
tcp        0      0 0.0.0.0:7734            0.0.0.0:*               LISTEN      10729/tclsh

Now the Sguil server is listening. I can connect with a Sguil client, even the 64 bit Windows .exe that I just found this morning. Check it out at sourceforge.net/projects/sguil/

Tweet

Read More

Practical Network Security Monitoring Book on Schedule

First the good news: my new book Practical Network Security Monitoring is on track, and you can pre-order with a 30% discount using code NSM101.

I'm about 1/3 of the way through writing the book. Since I announced the project last month, I've submitted chapters 1, 2, and 3. They are in various stages of review by No Starch editors and my technical editors. I seem to be writing more than I expected, despite trying to keep the book at an introductory level. I find that I want to communicate the topic sufficiently to make my point, but I try to avoid going too deeply into related areas.

I'm also encountering situations where I have to promise to explain some concepts later, rather than explain everything immediately. I believe once I get the first chapter ironed out with the editor, the rest will be easier to digest. I'm taking a fairly methodical approach (imagine that), so once the foundation in chapter 1 is done the rest is more straightforward.

I'm keeping a fairly aggressive schedule. Basically I have to write a chapter each week, get it to my technical editors, and then spend additional time working with No Starch to get the text legible and ready for print. All of this is happening in parallel in order to have the books in print by Black Hat. That means the text must done by the first week in April. My family is helping me stay on track by giving me time and space to write, especially on the weekends. Thank you!

When working on the examples, I've been very pleased with the performance of VMWare Workstation 9. I have one copy installed on Windows 7, where I write with Word. I have a second copy installed on Ubuntu Server, where it acts like a "VMWare Server." I used to run a real ESXi server on server-class hardware. Now, to save electricity and to more tailor my computer power to my requirements, I run a Shuttle DS61 with a Core i5-3450S 2.80GHz CPU, 16 GB RAM, 750 GB HDD, and two onboard NICs. The two NICs are really awesome in a device this small -- 190(L) x 165(W) x 43(H) mm. With two NICs, I can devote one for management and one for network traffic collection and interpretation. I use a Net Optics Dual Port Aggregator Tap for access to the wire.

I use VMWare Workstation this way. I run a Linux VM on Workstation on my Windows 7 laptop. I connect via Workstation to the Workstation instance on Ubuntu on the DS61. Then I create whatever VMs I need on the DS61. For example, I created a Security Onion server and sensor to test that setup. With 16 GB RAM, I have plenty of RAM for both, plus another VM that I'm running as my "production" Security Onion sensor for the lab network.

Writing is going well, despite the fact that I last wrote a book in 2005. I promised my youngest daughter, who wasn't born until 2006, that this new book is for her. If you have any questions on the writing process, please post them here or ask me on Twitter.

Tweet

Read More

How to Win This TCP/IP Book

Last week I wished this blog happy tenth birthday and announced plans for a new book on network security monitoring. I also mentioned a contest involving a book give-away. I finally figured out a good way to select a winner, and it involves your participation in my current writing project!

Thanks to No Starch Press I have a brand-new, shrink-wrapped copy of The TCP/IP Guide, a mammoth 1616 page hardcover book by Charles M. Kozierok.

Here's what you have to do to try to win this book: submit a case study on how network security monitoring helped you detect, respond to, and contain an intrusion in your environment.

You don't have to reveal your organization, but I want to know some general information like the number of users and computers. Readers need to know the sort of environment where NSM worked for you, but I don't want you to reveal your organization (unless you want to).

Tell the reader what happened, what NSM data you used, how you used it, and how you handled the incident. Extra points go to writers who include log excerpts and screen captures.

I will include the submission in my new book, subject to editing by myself and No Starch, for readability and comprehension.

The deadline for submission is 10:00 pm eastern time, Saturday 26 January (sorry for the earlier typo). I managed to extend the deadline a little. Quality trumps quantity here -- I'm not looking for another chapter!

Please submit your entries as plain text in email to taosecurity at gmail dot com. I won't open .doc or .pdf or other files which could contain surprises.

When you take screen captures, save them in high-resolution .tif format without compression. Don't take a capture of command-line information; instead, copy the text into the story. When taking screen captures of GUI tools and the like, don't take a capture of a giant window; resize to something that will be legible on a printed page, witha .

This is an example of a bad screen capture:

This is a good screen capture:

Depending on the quality of any screen captures, I may ask you to resubmit them to meet the publisher's requirements.

If you have any questions, please post them here.

The winner will receive the pictured TCP/IP book. Once my new book arrives, I will ask the publisher to mail you a free copy too.

If I receive one or more good runners-up, I will ask the publisher to send their owners copies of my new book too.

If you have any questions, please submit them as comments here. Good luck!

Tweet

Read More

Bejtlich's New Book: Planned for Summer Publication

Nearly ten years after I started writing my first book, the Tao of Network Security Monitoring, I'm pleased to announce that I just signed a contract to write a new book for No Starch titled Network Security Monitoring in Minutes.

From the book proposal:

Network Security Monitoring in Minutes provides the tactics, techniques, and procedures for maximum enterprise defense in a minimum amount of time.

Network Security Monitoring (NSM) is the collection, analysis, and escalation of indications and warnings to detect and respond to intrusions. Network Security Monitoring in Minutes teaches information technology and security staff how to leverage powerful NSM tools and concepts immediately.

Using open source software and vendor-neutral methods, the author applies lessons he first began applying to military networks in 1998. After reading this book, the audience will be able to integrate the same winning approaches to better defend his or her company’s data and networks.

Network Security Monitoring in Minutes is an important book because nearly all organizations operate a network. By connecting to the Internet, they expose their intellectual property, trade secrets, critical business processes, personally identifiable information (PII), and other sensitive information to attackers worldwide. Without the network level vigilance provided by this book, organizations will continue to be victimized for months, and in many cases years, before learning they have been breached.

This book consists of the following chapters:

Chapter 1, Network Security Monitoring Rationale, explains why NSM matters and help readers gain the support needed to deploy NSM in their environment.

Chapter 2, Accessing Network Traffic, addresses the challenges and solutions surrounding physical access to network traffic.

Chapter 3, Sensor Deployment and Configuration, introduces Security Onion (SO), and explains how readers can install the software on spare hardware to gain an initial NSM capability at low or no cost.

Chapter 4, Tool Overview, guides the reader through the core SO tool set, focusing on those capabilities most likely to help handle digital intrusions.

Chapter 5, Network Security Monitoring Operations, shares the author’s experience building and leading a global Computer Incident Response Team (CIRT), such that readers can apply those lessons to their own operations.

Chapter 6, Server-Side Compromise, is the first NSM case study, wherein readers will learn how to apply NSM principles to identify and validate a compromise of an Internet-facing application.

Chapter 7, Client-Side Compromise, is the second NSM case study, offering readers an example of a user being victimized by a client-side attack. NSM data will again identify and validate the compromise, prompting efficient incident response.

The Conclusion extends NSM principles beyond the enterprise into hosted and Cloud settings, offering future options for those environments.

The Appendix discusses tools that are not open source, but which may be helpful to those conducting NSM operations.

My goal is to finish this short book (roughly 220 pages) in time for publication at Black Hat this summer. Thank you to Pearson/Addison-Wesley for giving me the flexibility to write this complementary NSM book, and to No Starch for signing me to their publishing house.

Tweet

Read More

Welcome to Network Security Monitoring in the Cloud

I just watched an incredible technical video. If you have about 10 minutes to spare, and want to be amazed, take a look at Snorby Cloud.

I think the video and Web site does an excellent job explaining this new offering, but let me provide a little background.

Many of the readers of this blog are security pros. You're out there trying to defend your organization, not necessarily design, build, and run infrastructure. You still need tools and workflows that accelerate your incident detection and response process though. So, you work as a security admin, system admin, storage admin, database admin... you get the picture. You manage to keep up, but you probably wish you could focus on finding bad guys, as quickly as possible, without taking care of all the *stuff* that you need to do your job.

While many of you are security experts, some are just beginning your journeys. The responsibilities of being an admin of four or more different shades is overwhelming. Furthermore, you don't have the experience, or budget, or support to get the security data and escalation paths needed to defend your network. How can you improve your skills when you're constantly overwhelmed?

Both kinds of users -- senior and junior alike -- are going to find something intriguing about Snorby Cloud. Maybe you've heard of Snorby before, as a Web-based interface to Network Security Monitoring data. Doug Burks packages it with Security Onion (SO), and you can try it via live CD or .iso in a VM. It looks great on my iPad! There's even a mobile version on iTunes.

Snorby Cloud would be cool if it just put the Snorby Web application in the cloud, and managed the administrative side of security infrastructure for you. For example, you'd log into the cloud interface and be greeted by the graphs you remember from traditional Snorby.

However, you have to think of this as a new, better version of Snorby, collecting far more useful data, and making it rapidly available to the analyst. For example, the following shows SMTP logs available in the interface:

You can just as easily access host-based logs for the same victim computer:

As you investigate the incident, you can see who else on your team is working and what they did. You can also chat with them in real time.

I could say a lot more about this new tool, but I think watching the video will convey some of what it can do. My next step is to get the agents running on a test network so I can drive the console myself and become more familiar with it.

Snorby Cloud is a product from Packet Stash. Follow them at @packetstash for updates.

Disclaimer: I'm friends with this team; I hired two of the co-founders into GE-CIRT, and later worked with all three co-founders at Mandiant.

Tweet

Read More

Security Onion + (ELSA or Snorby) + CapMe = Awesome

Happy New Year everyone, and with some new open source software, what a year it will be.

Monday Doug Burks released Security Onion 12.04. Please read Doug's post to learn how great this new 64 bit release is. I wanted to highlight a few features of the new release which takes Network Security Monitoring with open source tools to a new level for security analysts.

12.04 ships with Martin Holste's Enterprise Log and Search Archive (ELSA) working out of the box. Thanks to close integration with the latest version of Bro, analysts have Web-based, indexed access to Bro logs.

If that weren't enough, 12.04 also ships with a late addition -- Paul Halliday's CapMe. What this means is that you can now access full TCP transcripts from any alert in Dustin Webber's Snorby or Martin's ELSA.

You might not appreciate that right away, but it's a step in the right direction. Thus far, Bamm Visscher's Sguil has been the de facto open source NSM reference tool, allowing analysts to easily pivot from alert or session data to full pcap data. Now, with ELSA + CapMe, analysts can pivot from any log entry of TCP traffic with timestamps, IP addresses, protocols, and ports to a Web-based rendering of a transcript.

This is key: this transcript was not saved because of the log or alert. It was saved simply because the traffic was seen on the wire and netsniff-ng recorded it. This is one way to better handle threats who know how to evade signature-based systems.

This new workflow/feature is what I chose to depict in the screen shot at left. The upper window shows ELSA with a query for a BRO_HTTP log for www.testmyids.com. I then invoke CapMe and generate the transcript in the window at bottom. You can do the same from alert data in Snorby.

This is only the first step in giving analysts more data via open source software. Great work Security Onion team!

Tweet

Read More

Why Collect Full Content Data?

I recently received the following via email:

I am writing a SANS Gold paper on a custom full packet capture system using Linux and tcpdump. It is for the GSEC Certification, so my intent is to cover the reasons why to do full packet capture and the basic set up of a system (information that wasn't readily available when setting my system up)...

I am already referencing The Tao of Network Security Monitoring.

These are the questions that I came up with based on questions other peers have asked me...

Here are the questions, followed by my answers. Most of this is covered in my previous books and blog posts, but for the sake of brevity I'll try posting short, stand-alone responses.

1. As an information security analyst in today's threat landscape why would I want to do full packet capture in my environment? What value does have?

   Full content data or capturing full packets provides the most flexibility and granularity when analyzing network-centric data. Unlike various forms of log data, full content data, if properly collected, is the actual data that was transferred -- not a summarization, or representation, or sample.

2. Where should I place a full packet capture system on my network - are ingress/egress points sufficient?

   I prioritize collection locations as follows:

   • Collect where you can see the true Internet destination IP address for traffic of interest, and where you can see the true internal source IP address for traffic of interest. This may require deploying two traffic access methods with two sensors; so be it.
   • Collect where you can see traffic to and from your VPN segment. Remember the previous IP address requirements.
   • Collect where you can see traffic to and from business partners or through "third party gateways." You need to acquire the true source IP, but you may not be able to acquire the true destination IP if the business partner prevents collecting behind any NAT or security devices that obscure the true destination IP.
   • Collect where your business units exchange traffic. This is more of a concern for larger companies, but you want to see true source and destination IPs (if possible) of internal traffic as they cross business boundaries.
   • Consider cloud or hosted vendors who enable collection near Infrastructure-as-a-Service platforms used by your company.

3. What advantages are there to creating a custom server with open source tools (such as a server running Linux and capturing with tcpdump) opposed to buying a commercial solution (like Solera or Niksun)?

   A custom or "open" platform enables analysts to deploy the sorts of tools they need to accomplish their security mission. Closed platforms require the analyst to rely on the information provided by the vendor.

4. Now that I have full packet data, what kind of analysis goals should I have to address advanced threats and subtle attacks?

   The goal for any network security monitoring operation is to collect and analyze indicators and warnings to detect and respond to intrusions. Your ultimate role is to detect, respond to, and contain adversaries before they accomplish their mission, which may be to steal, alter, or destroy your data.

5. Any other advice for an analyst just getting started with full packet capture systems and analyzing the data?

   Rarely start with full content data. Don't dump a ton of traffic into Wireshark and start scrolling around. I recommend working with session data (connection logs) and application-specific logs (HTTP, DNS, etc.) to identify sessions of interest, then examine the content if necessary to validate your suspicions.

I could write a lot more on this topic. Stay tuned.

Tweet

Read More

I Want to Detect and Respond to Intruders But I Don't Know Where to Start!

"I want to detect and respond to intruders but I don't know where to start!" This is a common question. Maybe you have a new security role in an organization, or a new service or business in your current organization, or some other situation where you want to find and stop attackers. However, you have no idea where to begin. Do you have the data you need? If not, what should you add? What do intrusions look like in the data you collect?

These questions can be tough to answer from a purely theoretical perspective. I propose the following approach.

First, conduct a tabletop exercise where you simulate adversary actions. At each stage of the imagined attack, consider what evidence an intruder might create while taking actions against your systems. For example, if you are trying to determine how to detect and respond to an attack against a Web server, you're almost certainly going to need Web server logs. If you don't currently have access to those logs, you've just identified a gap that needs to be addressed. I recommend this sort of tabletop exercise first because you will likely identify deficiencies at low cost. Addressing them might be expensive though.

Second, conduct a technical exercise where a third party simulates adversary actions. This is not exactly a pen test but it is the sort of work a red team conducts. Ask the red team to carry out the attacks you previously imagined to determine if you can detect and respond to their activity. This should be a controlled action, not an "anything goes" event. You will see whether the evidence and processes you identified in the first step help you detect and respond to the red team activity. This step is more expensive than the previous because you are paying for red team attention, and again fixes could be expensive.

Third, you may consider re-engaging the red team to carry out a less restrictive, more imaginative adversary simulation. In this exercise the red team isn't bound by the script you devised previously. See if your improved data and processes are sufficient. If not, work with the red team to devise better detection and response so that you can handle their attacks.

At this point you should have the data and processes to deal with the majority of real-world attacks. Of course some intruders are smart and creative, but you have a chance against them now given the work you just performed.

Tweet

Read More

Dustin Webber Creates Network Security Monitoring with Siri

Dustin Webber just posted a really cool video called Network Security Monitoring with Siri. He shows how he uses his iPhone 4S and SiriProxy to interact with his Snorby Network Security Monitoring platform.

The following screenshot shows Dustin asking "Can you show me what the last severity medium event was?" and Siri answering.



Later he asks Siri to tell him about "incident 15":



Near the end Dustin asks Siri if she likes Network Security Monitoring:



This is just about the coolest thing I've seen all year. Ten years ago I thought it was cool to listen to Festival read Sguil events out loud -- now Dustin shows how to interact with a NSM platform by voice command. Amazing!

Tweet

Read More

Trying NetworkMiner Professional 1.2

Erik Hjelmvik was kind enough to send an evaluation copy of the latest version of his NetworkMiner traffic analysis software. You can download the free edition from SourceForge as well. I first mentioned NetworkMiner on this blog in September 2008.

NetworkMiner is not a protocol analyzer like Wireshark. It does not take a packet-by-packet approach to representing traffic. Instead, NetworkMiner displays traffic in any one of the following ways: as hosts, frames, files, images, messages, credentials, sessions, DNS records, parameters, keywords, or cleartext. To demonstrate a few of these renderings, I asked NetworkMiner to parse the sample pcap from a sample lab from TCP/IP Weapons School 2.0. I did not need to install it; the software starts from a single executable and loads several DLLs in the associated directory.

The following screen capture shows information from the Hosts tab, showing what NetworkMiner knows about 192.168.230.4.



Notice that in addition to summarizing information about traffic to and from the host, in terms of packets or sessions, we also see what NetworkMiner knows about the host, like Queried NetBIOS names, Web Browser User Agents, and so on.

The following screen capture shows the Files tab. This displays all the content that NetworkMiner extracted from the traffic to the analysis workstation hard drive (or in my case, the NetworkMiner USB thumb drive).



I think NetworkMiner is pretty cool, especially given what you can do with the free version. My primary recommendation for improvement would be an interface that allows the user to easily pivot from one piece of information to the next. With the current environment, the analyst seems confined to the tab at hand. I would like to see a way to right click on an element of the displayed information and then execute a query based on my selection. It would also be helpful to be able to right click and open associated data in another traffic analysis program like Wireshark.

Thank you to Erik Hjelmvik for the opportunity to take another look at NetworkMiner!

Tweet

Read More

Tao of Network Security Monitoring, Kindle Edition

I just noticed there is now a Kindle edition of my first book, The Tao of Network Security Monitoring: Beyond Intrusion Detection, published in July 2004. Check out what I wrote in the first paragraphs now available online.

---

Welcome to The Tao of Network Security Monitoring: Beyond Intrusion Detection. The goal of this book is to help you better prepare your enterprise for the intrusions it will suffer. Notice the term "will." Once you accept that your organization will be compromised, you begin to look at your situation differently. If you've actually worked through an intrusion -- a real compromise, not a simple Web page defacement -- you'll realize the security principles and systems outlined here are both necessary and relevant.

This book is about preparation for compromise, but it's not a book about preventing compromise. Three words sum up my attitude toward stopping intruders: prevention eventually fails. Every single network can be compromised, either by an external attacker or by a rogue insider. Intruders exploit flawed software, misconfigured applications, and exposed services. For every corporate defender, there are thousands of attackers, enumerating millions of potential targets. While you might be able to prevent some intrusions by applying patches, managing configurations, and controlling access, you can't prevail forever. Believing only in prevention is like thinking you'll never experience an automobile accident. Of course you should drive defensively, but it makes sense to buy insurance and know how to deal with the consequences of a collision.

Once your security is breached, everyone will ask the same question: now what? Answering this question has cost companies hundreds of thousands of dollars in incident response and computer forensics fees. I hope this book will reduce the investigative workload of your computer security incident response team (CSIRT) by posturing your organization for incident response success. If you deploy the monitoring infrastructure advocated here, your CSIRT will be better equipped to scope the extent of an intrusion, assess its impact, and propose efficient, effective remediation steps. The intruder will spend less time stealing your secrets, damaging your reputation, and abusing your resources. If you're fortunate and collect the right information in a forensically sound manner, you might provide the evidence needed to put an intruder in jail.

---

I wrote that eight years ago, and thankfully my concept that "prevention eventually fails" (which I coined in that book) is finally gaining ground.
Tweet

Read More

Thanks for Help with Startup Scripts

Thanks to @sevanjaniyan and @cperciva for helping with my FreeBSD startup script issue. By removing the ${barnyard2_flags} argument from the command_args section I was able to start barnyard2 properly:

root    45842 54.9  0.5 18572 11116  ??  Ss    7:15PM   0:00.00
 /usr/local/bin/barnyard2 -D -U -d /nsm/r200a -f snort.unified2
 -c /usr/local/etc/nsm/barnyard2.conf

In other words, the script has this now:

. /etc/rc.subr

name="barnyard2"
load_rc_config $name
rcvar=`set_rcvar`
# set some defaults
: ${barnyard2_enable="NO"}
: ${barnyard2_conf="/usr/local/etc/barnyard2.conf"}
: ${barnyard2_flags="-D"}

command="/usr/local/bin/barnyard2"
command_args="-c ${barnyard2_conf}"

run_rc_command "$1"

I made changes to some other startup scripts and needed to commit them via Git. I did it this way.

richard@macmini:~/taosecurity_freebsd_sguil$ git status
# On branch master
# Changes to be committed:
#   (use "git reset HEAD ..." to unstage)
#
#       new file:   pcap_agent
#       new file:   sancp_agent
#       new file:   sguild
#       new file:   snort_agent
#

richard@macmini:~/taosecurity_freebsd_sguil$ git add pcap_agent sancp_agent sguild snort_agent

richard@macmini:~/taosecurity_freebsd_sguil$ git commit -am "Added new startup scripts."
Created commit 296687e: Added new startup scripts.
 4 files changed, 145 insertions(+), 0 deletions(-)
 create mode 100755 pcap_agent
 create mode 100755 sancp_agent
 create mode 100755 sguild
 create mode 100755 snort_agent

richard@macmini:~/taosecurity_freebsd_sguil$ git push origin master
taosecurity@taosecurity.git.sourceforge.net's password: 
Counting objects: 7, done.
Compressing objects: 100% (6/6), done.
Writing objects: 100% (6/6), 1.89 KiB, done.
Total 6 (delta 3), reused 0 (delta 0)
To ssh://taosecurity@taosecurity.git.sourceforge.net/gitroot/taosecurity/taosecurity
   9cad54a..296687e  master -> master

Thanks again for your help!

Now I'm watching commits to https://github.com/firnsy/barnyard2 to see if Barnyard2 is updated to work with the new Snort event types that kills it.
Tweet

Read More

NetWitness Minidecoder in Action

Many TaoSecurity Blog readers are undoubtedly familiar with NetWitness. Several weeks ago I met with their CEO and CTO to discuss their products and services. They were kind enough to later provide me with a device that they ship to their engineers to provide testing and experimentation with their product. Here I call it a "Minidecoder," but you can think of it as "NetWitness in an EeeBox PC" (specifically the EeeBox PC EB1012).

As you can see in the diagram, it only has one onboard NIC, and that is used for management. To access traffic, NetWitness provided a Trendnet TU2-ET100 USB to 10/100Mbps Adapter. To try the Minidecoder, I paired it with the DualComm device mentioned in my last post. I basically tapped the Minidecoder's own management port and then generated some basic traffic from the Minidecoder's Linux shell. The sensor also saw its own management traffic, as well as broadcast traffic passed by the wireless bridge to which it was connected.

Setup was pretty painless. I booted the Minidecoder, logged into the Linux OS, connected the network cables, and configured the management port.

To administer the device and access the data, I installed the NetWitness Administrator and Investigator applications on a Windows XP SP3 laptop.

In the screen capture at left, you see part of the Administrator interface. It was easy to add and connect the Minidecoder after I obtained a license from NetWitness. I didn't need to run anything on Windows with Administrator privileges, other than the installation process.

I'm not a big fan of dashboards, but I wouldn't expect to spend much time in this view anyway. The action is in Investigator, which I also installed.

As you can see in the figure at right, NetWitness breaks down the traffic using a variety of means. I'm only showing what would fit on the screen.

One of my favorite aspects of the NetWitness metadata approach is the way it extracts meaningful content automatically, like the system names from within DHCP traffic (independent of DNS names, for example). This is why I've added metadata as the seventh form of NSM data (after full content, session, statistical, alert, transaction, and extracted content). NetWitness is good at depicting "data about data," i.e., details about traffic, derived from the traffic.

I also like the way NetWitness classifies a variety of traffic into "Action Events" like "Get". Here I will select the sessions associated with the Get Action Event to produce the screen shot that follows.

In this screen capture you can see sessions which NetWitness classified as Get events. I chose part of a Lynx session where I browsed to www.bejtlich.net. The large sessions with the "Click to Create" boxes are sessions where I retrieved packages for installing Lynx on the Linux OS of the Minidecoder itself. (Remember this traffic is to and from the Minidecoder, just for the sake of the test.)

I think this is a cool way to get more familiar with NetWitness in a low-impact way. I wouldn't try to stress test this device since neither the hardware nor the NICs are intended for intense use. Rather, you could deploy a device like this in your environment to get a better idea of how NetWitness would process your environment's traffic, or at least a subset of it, if you had to throttle what was sent to the sensor.

Feel free to contact NetWitness for more information, and thanks to Amit and Tim for the demo gear!

Tweet

Read More

DualComm Port Mirroring Switch

John He from DualComm Technology was kind enough to send me one of his company's port-mirroring switches, namely the DCGS-2005 pictured with its box at left.

In the figure, I have port 1 going to a computer I want to monitor. Port 2 is going to the uplink (or access switch) for that computer. Port 5 (at the far right) is going to a sensor.

The idea behind this device is to provide a plug-and-play alternative to network taps. I thought this system was interesting because it acts somewhat like a port aggregating tap, in the sense that two ports are used for accessing the network but only one port is needed by the sensor.

Note that only port 1 is mirrored to port 5. (The manual confirms this, and I did some limited testing. The words on the tap imply ports 1 - 4 are all mirrored.) This is a one-for-one copy. If you connect to ports 2 and 3, 2 and 4, or 3 and 4, you will not see any unicast traffic on port 5.

This device is also different in that in requires a USB connection for power.

Probably the biggest advantages of this device include low cost and simple use. I think the single USB power connection is the biggest disadvantage. I'd also like to know more about the software on the switch itself.

Thanks again to John for sending me this device. Check out DualComm for more information!

Tweet

Read More

A Book for the Korean Cyber Armies

I've got a book for the Korean cyber armies, North and South. That's right, it's my first book, The Tao of Network Security Monitoring, now published in Korean! Apparently my publisher just decided to translate and deliver this new edition to Korea. Can anyone who reads Korean comment on how they translated my name?

I've known for a while there is also a Spanish edition, but I've never seen it. I asked to see one of those too.

I have to admit that seeing these foreign language editions motivate me to try to write another solo book. However, I'm just not sure how I would find the time.

Tweet

Read More

Time Issues in Libpcap Traces

Time is an important aspect of Network Security Monitoring. If you don't pay close attention to the time shown in your evidence, and recognize what it means, it's possible you could misinterpret the values you see.

My students and I encountered this issue in TCP/IP Weapons School at Black Hat this week. Let's look at the first ICMP packet in one of our labs.

I'm going to show the output using the Hd tool and then identify and decode the field that depicts time.

In the following output, 2d 0c 65 49 occupies the part of the packet where Libpcap has added a timestamp.

Hd output:

$ hd icmp.sample.pcap 
00000000  d4 c3 b2 a1 02 00 04 00  00 00 00 00 00 00 00 00  |................|
00000010  ea 05 00 00 01 00 00 00  2d 0c 65 49 5f bf 0c 00  |........-.eI_...|
00000020  4a 00 00 00 4a 00 00 00  00 0c 29 82 11 33 00 50  |J...J.....)..3.P|
00000030  56 c0 00 01 08 00 45 00  00 3c 02 77 00 00 80 01  |V.....E..<.w....|
00000040  ea f1 c0 a8 e6 01 c0 a8  e6 05 08 00 43 5c 07 00  |............C\..|
00000050  03 00 61 62 63 64 65 66  67 68 69 6a 6b 6c 6d 6e  |..abcdefghijklmn|
00000060  6f 70 71 72 73 74 75 76  77 61 62 63 64 65 66 67  |opqrstuvwabcdefg|
00000070  68 69                                             |hi|
00000072

Tpo convert 2d 0c 65 49 to a time, we have to swap the bytes, so becomes 0x49650c2d, or 1231359021 in decimal. 1231359021 is a Unix timestamp that we can convert with the -r option found in the FreeBSD version of the date command.

First let me show the date on this system so you can see the timezone of the FreeBSD system, and then I'll convert the seconds into a human readable time.

$ date
Wed Jul 28 00:11:23 EDT 2010

$ date -r 1231359021
Wed Jan  7 15:10:21 EST 2009

So, this ICMP packet has a timestamp of Wed Jan 7 15:10:21 EST 2009. Note that the date command produces a time in EST and not EDT. 15:10:21 EST becomes 16:10:21 EDT. I would have preferred seeing the date output show EDT since that is the time zone on the system in question, but I can understand the output. That seems simple enough, right?

Let's see what Tcpdump says about this packet. First I run the date command to remind us where we are running Tcpdump.

FreeBSD Tcpdump:

$ date
Wed Jul 28 00:11:23 EDT 2010

$ tcpdump -h
tcpdump version 3.9.8
libpcap version 0.9.8

$ tcpdump -n -tttt -r icmp.sample.pcap 
2009-01-07 16:10:21.835423 IP 192.168.230.1 > 192.168.230.5:
 ICMP echo request, id 1792, seq 768, length 40

As we expected, this packet has a timestamp of 16:10:21 (ignore the fractions of a second), and since the time zone is EDT it matches what we expect.

Let's see what a tool like Tshark says.

FreeBSD Tshark:

$ tshark -v
TShark 1.0.7
...edited...
Compiled with GLib 1.2.10, with libpcap 0.9.8, with libz 1.2.3, without POSIX
capabilities, without libpcre, without SMI, without ADNS, without Lua, without
GnuTLS, without Gcrypt, with Heimdal Kerberos.
NOTE: this build doesn't support the "matches" operator for Wireshark filter
syntax.

Running on FreeBSD 7.2-RELEASE-p7, with libpcap version 0.9.8.

Built using gcc 4.2.1 20070719  [FreeBSD].

$ tshark -n -t ad -r icmp.sample.pcap 
  1 2009-01-07 15:10:21.835423 192.168.230.1 -> 192.168.230.5
 ICMP Echo (ping) request

What? Why does it show 15:10:21? That's the result for EST, not EDT, which is the time zone of the FreeBSD system.

Let's see if a Linux system in EDT behaves the same way.

$ date
Wed Jul 28 00:44:54 EDT 2010

$ tcpdump -V
tcpdump version 4.0.0
libpcap version 1.0.0

$ tcpdump -n -tttt -r icmp.sample.pcap 
2009-01-07 16:10:21.835423 IP 192.168.230.1 > 192.168.230.5:
 ICMP echo request, id 1792, seq 768, length 40

$ tshark -v
TShark 1.2.7
...edited...
Compiled with GLib 2.24.0, with libpcap 1.0.0, with libz 1.2.3.3, with POSIX
capabilities (Linux), with libpcre 7.8, with SMI 0.4.8, with c-ares 1.7.0, with
Lua 5.1, with GnuTLS 2.8.5, with Gcrypt 1.4.4, with MIT Kerberos, with GeoIP.

Running on Linux 2.6.32-23-generic, with libpcap version 1.0.0, GnuTLS 2.8.5,
Gcrypt 1.4.4.

Built using gcc 4.4.3.

$ tshark -n -t ad -r icmp.sample.pcap 
  1 2009-01-07 15:10:21.835423 192.168.230.1  192.168.230.5
  ICMP Echo (ping) request

Again, we see Tcpdump correctly honor the local time zone (EDT) and display the timestamp as 16:10:21, whereas Tshark shows the timestamp as EST or 15:10:21.

I am really disappointed by this Tshark behavior. Incidentally, you get the same results from any tool in the Wireshark suite, such as Wireshark itself, capinfos, etc.

Does anyone know why Tshark and the like don't really honor the local time zone, and instead use Standard Time instead of recognizing Daylight Savings Time?

Read More

Network Forensics Vendors: Get in the Cloud!

I know some of us worry that the advent of the "cloud" will spell the end of Network Security Monitoring and related network-centric visibility and instrumentation measures. I have a proposal for any network forensics vendors reading this blog: get in the cloud!

For example, imagine you are a proxy-in-the-cloud (PITC) provider, like ScanSafe, now owned by Cisco. You provide a Web portal to your customers so they can see what bad sites employees were not allowed to visit. But what about all the subtle traffic that evaded your filters, block lists, heuristics, and other defensive mechanisms? What about the insider stealing intellectual property, indistinguishable from a "normal employee?" How does your abuse-centric Web portal address the sorts of threats that really matter?

To me, one answer is to deploy a network forensics solution like NetWitness or Solera in front of your PITC infrastructure. The PITC vendor must have a way to identify legitimate clients, or else you've created the world's greatest open Web proxy. Use the identity information to tag the traffic collected by the network forensics product.

When a customer needs to analyze an intrusion, or conduct an investigation, he can connect to the hosted network forensics platform.

I also like this approach because it helps address the consumerization of IT. You can create a policy (weak I know, but it's an option) that Company users must point any device that processes Company data to the PITC infrastructure for Web access. By doing so you can collect the network forensic data you need.

Of course, encryption is always an issue, but if really necessary I'm sure you can work with the PITC vendor on a MITM approach.

I'm sure I'll get a few comments from critics saying "NSM is dead," "network traffic is worthless," etc. It's just a sign you don't know how to use that sort of data effectively, and probably never will. After evangelizing for 10 years, I've given up trying to convince critics like that.

I also don't intend for this post to be a signal that I hate logs or host-based evidence. It's just another piece in the puzzle.

So, network forensics vendors, who will be the first to publish a press release saying you've partnered with a PITC provider?

Read More

Comments on Sharkfest Presentation Materials

I saw that presentations from Sharkfest 2010 are now posted. This is the third year that CACE Technologies has organized this conference. I've had conflicts each of the last three years, but I think I need to reserve the dates for 2011 when they are available. In this post I wanted to mention a few slides that looked interesting.

Jasper Bongertz presented Wireshark vs the Cloud (.pdf) I reviewed this presentation to see if anyone is doing something novel regarding monitoring Cloud environments. In the slide at right you see his first option is to install a monitoring tool inside a VM. That's standard.

In the next slide you see his second option is to select a link upstream from the VM server and tap that line. That's standard too. I know of some cloud providers who use this strategy and then filter the results. You will likely need some robust equipment, depending on active the link is.

In the last slide you see that future options include ensuring that the virtual switch in the VM server provide instrumentation options. From my limited understanding this should be the case with expensive solutions like the Cisco Nexus 1000v, but I don't have any personal experience with that. Any comments from blog readers?

I also wanted to mention SPAN Out of the Box (.pdf) by John He of Dualcomm Technology. In his presentation he advocates replacing a tap with a switch used only for port mirroring, as shown in the slide at left. He's mainly trying to compete on price, since his "USB Powered 5-Port Gigabit Desktop Switch with Port-Mirroring & PoE Pass-Through" sells for $139.95 on his Web site. I'll ask Mr He if I could get a demo switch to see how well it works.

Read More

Dealing with Security Instrumentation Failures

I noticed three interesting blog posts that address security instrumentation failures.

First, security software developer Charles Smutz posted Flushing Out Leaky Taps:

How many packets does your tapping infrastructure drop before ever reaching your network monitoring devices? How do you know?

I’ve seen too many environments where tapping problems have caused network monitoring tools to provide incorrect or incomplete results. Often these issues last for months or years without being discovered, if ever...

One thing to keep in mind when worrying about loss due to tapping is that you should probably solve, or at least quantify, any packet loss inside your network monitoring devices before you worry about packet loss in the taps. You need to have strong confidence in the accuracy of your network monitoring devices before you use data from them to debug loss by your taps. Remember, in most network monitoring systems there are multiple places where packet loss is reported...

I’m not going to discuss in detail the many things that can go wrong in getting packets from your network to a network monitoring tool... I will focus largely on the resulting symptoms and how to detect, and to some degree, quantify them. I’m going to focus on two very common cases: low volume packet loss and unidirectional (simplex) visibility.

Read Charles' post to learn ways he deals with these issues.

Next I'd like to point to this post by the West Point Information Technology Operations Center on Misconfiguration Issue of NSA SPAN Port:

Thanks to the input we have already received on the 2009 CDX dataset, we have identified an issue in the way the NSA switch was configured. Specifically, we believe the span port from which our capture node was placed was configured for unidirectional listening. This resulted in our capture node only "hearing" received traffic from the red cell.

Doh. This is a good reminder to test your captures, as Charles recommends!

Finally, Alec Waters discusses weaknesses in SIEMs in his post Si(EM)lent Witness:

[H]ow can we convince someone that the evidence we are presenting is a true and accurate account of a given event, especially in the case where there is little or no evidence from other sources...

D]idn’t I say that vendors went to great lengths to prevent tampering? They do, but these measures only protect the information on the device already. What if I can contaminate the evidence before it’s under the SIEM’s protection?

The bulk of the information received by an SIEM box comes over UDP, so it’s reasonably easy to spoof a sender’s IP address; this is usually the sole means at the SIEM’s disposal to determine the origin of the message. Also, the messages themselves (syslog, SNMP trap, netflow, etc.) have very little provenance – there’s little or no sender authentication or integrity checking.

Both of these mean it’s comparatively straightforward for an attacker to send, for example, a syslog message that appears to have come from a legitimate server when it’s actually come from somewhere else.

In short, we can’t be certain where the messages came from or that their content is genuine.

Read Alec's post for additional thoughts on the validity of messages sent to SIEMs.

Read More

All Aboard the NSM Train?

It was with some small amusement that I read the following two press releases recently:

First, from May, NetWitness® and ArcSight Partner to Provide Increased Network Visibility:

NetWitness, the world leader in advanced threat detection and real-time network forensics, announced certification by ArcSight (NASD: ARST) of compliance with its Common Event Format (CEF) standard. ArcSight CEF certification ensures seamless interoperability and support between NetWitness’ industry-leading threat management solution and ArcSight’s security information and event management (SIEM) platform.

Let me parse the market-speak. This is another indication that an ArcSight user can click on an event in the SIM console and access network traffic captured by NetWitness.

Second, from June, Solera Networks™ and Sourcefire™ Announce Partnership:

Solera Networks, a leading network forensics products and services company today announced its partnership with Sourcefire, Inc. (Nasdaq:FIRE), the creators of SNORT® and a leader in intelligent Cybersecurity solutions. Solera Networks can now integrate its award-winning network forensics technology directly into Sourcefire’s event analysis. The integration enhances Sourcefire’s packet analysis functionality to include full session capture, which provides detailed forensics for any security event. The partnership enables swift incident response to any security event and provides full detail in the interest of understanding “what happened before and after a security event?”

Martin Roesch, founder and CTO of Sourcefire. “There is a powerful advantage in being able to see the full content of every attack on your network. Network forensics from Solera Networks compliments Sourcefire’s IPS and RNA products by letting you see everything that led up to and followed a successful prevention of an attack.

This press release is a little clearer. This is an indication that a Sourcefire user can click on an event in the Sourcefire console and access network traffic captured by Solera.

This second development is interesting from a personal level, because it shows that the Network Security Model has finally been accepted by the developer (Marty Roesch) of what is regarded as the most popular intrusion detection system (Snort).

In other words, after over eight years of evangelizing the need to collect NSM data (at its core, full content, session, statistical, and alert data) in order to detect and respond to intrusions, we see Sourcefire partnering with Solera to pair full content network traffic with Snort alert data. It's almost enough to bring a tear to my eye. "Yo Adrian! I did it!"

Read More