VMWare Team LAN Appears Shared

Previously I wrote about my plans to incorporate VMWare into my classes. Originally I intended to use GSX Server. I thought I would give each student his or her own independent image. I assumed people would want to build their own sensors (from the ground up), and that required providing complete virtual machines.

Based on feedback here and in classes since that post, I've learned most people don't care about building sensors. They are more interested in analysis. Therefore, I decided students didn't need dedicated VMs. Therefore, I could run a few VMs with dedicated functions, and let students share systems as normal users. For example, in my last class a dozen students all logged in to a single FreeBSD image to perform analysis.

In the future, I plan to have multiple images running. For example, I plan to offer several complete Sguil installations. Students in groups of two or four might share one Sguil server. My current test environment uses VMWare Workstation 5 running 6 FreeBSD 5.4 REL images simultaneously.

Since VMWare 3.x I've wondered about the product's networking support. For example, if I provided a set of VMs with internal NICs, could they see each other's traffic? I decided to answer this question by putting my 6 FreeBSD VMs into a single VM "team", as shown.
One interface (lnc0 on each) is bridged so I can access the systems remotely. The second interface (lnc1 on each) is limited to the team and is addressed with an internal scheme. Here is the question: if freebsd54-rel_01 pings freebsd54-rel_02, will freebsd54-rel_03 see it? Here is the ping:

$ hostname
fbsd5401.taosecurity.com
$ ping -c 1 10.1.1.202
PING 10.1.1.202 (10.1.1.202): 56 data bytes
64 bytes from 10.1.1.202: icmp_seq=0 ttl=64 time=2.943 ms

--- 10.1.1.202 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max/stddev = 2.943/2.943/2.943/0.000 ms

Here is what another system on the team sees:

fbsd5403# tcpdump -n -i lnc1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lnc1, link-type EN10MB (Ethernet), capture size 96 bytes
08:19:58.946640 IP 10.1.1.201 > 10.1.1.202: icmp 64: echo request seq 0
08:19:58.946695 IP 10.1.1.202 > 10.1.1.201: icmp 64: echo reply seq 0

Yes. That is great. Life is much simpler now, since any machine can see any other machine on the same team. That facilitates setting up networks that can be monitored.

Read More

Speaking at DoD Cybercrime in January

I learned I will be delivering two presentations at the DoD Cybercrime 2006 Conference in Palm Harbor, FL on 11 January 2006. I will present shortened versions of my network incident response and forensics classes. Last year I spoke about network security monitoring with Sguil and other open source tools. In 2000 I spoke at the first DoD Cybercrime conference in Colorado, delivering the AFCERT mission briefing.

Read More

Speaking at USENIX LISA in December

I just checked the training schedule for the next USENIX LISA (Large Installation System Administration) conference. I will teach network security monitoring, incident response, and forensics. These are each full-day tutorials, which begin on Tuesday 6 December and end Thursday 8 December. Early bird registration ends 18 November 2005. It looks like you can attend all three days for $1775. I am looking forward to teaching these classes because the USENIX crowd is always top-notch.

Don't forget my only scheduled public Network Security Operations class, which will be held in Fairfax, VA 27-30 September (that's three weeks from Tuesday)! If you're an ISSA-NoVA member and you register no later than Friday 16 September, you will save $1000 on the class price.

To register or for more information, email me: richard at taosecurity dot com. Thank you.

Read More

Request for Comments on CERT and SEI Training

I have been taking a closer look at training offered by the CERT® Coordination Center and the Software Engineering Institute. Six years ago as an Air Force captain from the AFCERT I enjoyed the Advanced Incident Handling for Technical Staff. Now I may have a chance to teach or develop course materials for some of these courses. I am also considering the value of the
CERT®-Certified Computer Security Incident Handler program.

Has anyone attended any of these courses recently? If yes, what do you think of them? If no, why not? What alternatives have you considered or attended?

Read More

Thoughts on Cisco Packet Magazine

I like to read Cisco's quarterly Packet magazine. It's free, and it provides insight into developments by the world's networking (and one day, security) juggernaut. While waiting for car maintenance this morning, I managed to read much of the Quarter 2 2005 issue, devoted to Self-Defending Networks.

According to Cisco, they have been releasing Self-Defending Network components every few years. In 2000 they offered integrated security, followed by collaborative security in 2003. Now, in 2005, we have adaptive threat defense. The first term means security is part of Cisco products, such as routers and switches. The second term means these products should work together. Let's look closer at the third term, which Cisco claims will "protect every packet and every packet flow on a network."

I was skeptical when I saw the cover text. The phrase "eliminating the source of attacks" and the sentence "network security grows adaptive, reaching inside Web applications and excising attacks at their source" also worried me. When I read words like that, I imagine Cisco police forces banging down doors of attacker apartments in Bucharest or Beijing. That's how one really "excises" a threat!

As I read more about Cisco's plans, however, I realized they refer to containing malicious systems that connect to protected networks. For example, one article described how the CS-MARS [Security Monitoring, Analysis and Response System], developed by the former Protego Networks, "will send the network administrator the appropriate command to execute an action to excise the problem from the network at its source." In other words, Cisco gear will help identify and disable misbehaving network assets (or rogue visitors).

I found this article on wireless defenses interesting. It describes Cisco's approach to handling rogue wireless access points:

"With rogue access point suppression, the sensors detect wireless-device information, aggregate it, and pass it up to elements in the network that can correlate it and act upon it. When a wireless access point is detected on the network, the WLAN intrusion prevention system sends RF management frames that disassociate any clients that connect to it and attempt to trace and shut down the switch port to which the rogue is connected."

That seems cool. Only a few years ago I remember Mike Schiffman demonstrating libradiate at Black Hat by disassociating wireless clients. Now he works at Cisco!

I also learned a little about virtual routing and forwarding, which acts like "multiple routers within a single chassis."

Overall, I found Cisco's magazine very useful. I also subscribe to the free IP Journal, which I recommend. Whenever I read articles by Cisco, it reminds me I am not currently a networking engineer. That would require far more protocol and algorithm knowledge than I currently possess!

Speaking of Cisco, I will be speaking on 10 October 2005 to the Cisco Fall 2005 System Engineering Security Virtual Team Meeting in San Jose, CA. I will probably discuss network security monitoring and give a Sguil demo.

Read More

Feds Hurry, Slow Down

In my post Opportunity Costs of Security Clearances I ranted about needing security clearances for assessment work. Now I read Security clearance delays still a problem by Florence Olsen:

"Security clearance delays are the same, if not worse, than a year ago, before lawmakers made changes designed to help clear the backlog...

[N]ewly enacted reciprocity rules have made no dent in a problem that is creating mounting costs for high-tech companies. Those rules permit agencies to accept clearances initiated by other agencies."

Wonderful. Not only do agencies not trust employees, they don't trust other government agenices. That is understandable, but pathetic; jobs are left vacant because .gov entities want to play petty games. It gets worse:

"ITAA officials said 27 member companies that responded to a survey are coping with the backlog by hiring cleared employees from one another, sometimes paying premiums of up to 25 percent."

Great. This means the same cadre of cleared people are being shuffled among agencies. New blood with potentially more enthusiasm, skill, and (gasp) lower salaries (which would appeal to any commercial endeavor) are left to watch this dance from the outside.

So how bad are the delays?

"21 companies, said they had encountered delays of 270 or more days in getting top-secret clearances for employees. Last year, when ITAA conducted a similar survey, 70 percent reported equally lengthy delays.

The longest waits occurred in seeking clearances for employees to work at the CIA and the Defense Department."

So, in places were skilled security practitioners are most needed, they are not available.

Those that are already in place must cram before 2005 FISMA scores arrive. In my NSA-IAM class last week, a State Dept. worker told us he had to work a disaster recovery scenario on Sunday 28 August in order to meet his 31 August deadline.

Companies that cannot properly manage their workforces go out of business. Governments just lumber along. There is no mechanism to correct deficiencies, aside from massive intelligence or defense failures like 9/11, the Iraq war, and so on. Sorry for the depressing post, but I don't see a light at the end of this tunnel!

Read More

Pool IDS

By now you've probably heard the story about the 10-year-old girl in Wales who was saved by the Poseidon computer-aided drowning detection system. According to the vendor:

"[Poseidon] uses advanced computer vision technology to analyze activity in the pool, captured by a network of cameras mounted both above and below the surface of the pool. Poseidon helps lifeguards monitor swimmers' trajectories, and can alert them in seconds to a swimmer in trouble."

While reading comments at Slashdot, several of them reminded me of the value of digital intrusion detection systems. This one by a Poseidon user is very helpful if you want to know more about how Poseidon works.

For example, some critics complain about "false positives," meaning Poseidon sounds the alarm although no one is drowning. Poseidon alarms when a swimmer stops moving below the water for more than a few seconds. If the Poseidon programmers tell the device to alert when people appear to be drowning (i.e., motionless below water for a while), then it is not the device's fault when it alerts lifeguards of this fact.

It should not be Poseidon's fault if someone decides to "play dead" at the bottom of the pool!

If Poseidon alarms when everyone is moving, then that is an example of a real false positive. A false negative means no alarm when someone is drowning and motionless below the water.

Beyond the false positive debate, someone proposed a "drowning prevention system" based on the Poseidon alert. The idea was to raise a portion of the pool (!) under the motionless person, thereby elevating them above the water (!) to safety. This is an example of "prevention" being difficult or too costly. Wherever prevention is impossible, detection should be applied.

Finally, the Poseidon system demonstrates another feature of digital detection: human involvement. Poseidon sounds an alarm, to which human "analysts" (aka lifeguards) must respond. Time is of the essence. Here, "real time" does matter. However, a person could thrash underwater while drowning, and only become motionless after their lungs have filled with water. Still, an alert a few seconds later is better than no alert at all.

On a related note, consider the T.J. Hooper v. Northern Barge Corp. effect. This was the case where Judge Learned Hand (I am not making that up) essentially found tugboat owners negligent for not installing a newfangled "radio" technology (in 1932) that could have warned the boats of an impending storm. Radios were not mandatory at that time on boats, but Judge Hand "legislated from the bench" and essentially made them mandatory because they were so helpful. The previous link uses the same argument to advocate installing DDoS defenses, but one could extend the argument to hold pool owners negligent if they do not deploy Poseidon-like systems.

Read More