Minggu, 29 Mei 2005

Bejtlich Writing Like Mad During Home Stretch

I am writing like mad to meet a 1 June book delivery deadline for Addison-Wesley, so I won't be posting much or at all until I finish Extrusion Detection. I think crunching to meet deadlines is a common author predicament, after speaking with the other participants at the BSDCan book signing last week. Pictured with me, starting from my left, are Dru Lavigne, Greg Lehey, Marshall Kirk McKusick, and George Neville-Neil. Michael Lucas was not present for that photo but he appeared momentarily afterwards. Standing in the background is conference organizer extraordinaire Dan Langille. Thanks to Robert Bernier for posting the original photo.

Jumat, 27 Mei 2005

Snort Inline?

Is anyone successfully running an inline deployment of Snort on FreeBSD? If so, please email me: richard at taosecurity dot com. This guide makes it look easy, but I've tried multiple variations (bridging, routing, etc.) with Snort 2.3.3 on FreeBSD 5.4 REL and nothing works completely. Thank you.

Update: I got it working. snort-2.3.3.tar.gz doesn't work; snort_inline-2.3.0-RC1.tar.gz does. Who knew.

Vote for Sguil at SANS ISC Poll

Thanks to Brandon Greenwood I learned that the current SANS ISC poll asks for your favorite Snort interface. Sguil is currently running third behind BASE and ACID. Visit SANS ISC and vote for Sguil!

Kamis, 26 Mei 2005

NetBSD Binary OS Updates

I discovered a system running NetBSD 2.0 in my lab and decided to upgrade it to NetBSD 2.0.2. I read that "this is also the first binary security/critical update since NetBSD 2.0." I found a thread which gave various forms of advice on updating to NetBSD 2.0.2 from 2.0. Here is what I did.

When I started the system was running NetBSD 2.0.

bash-2.05b# uname -a
NetBSD juneau.taosecurity.com 2.0 NetBSD 2.0 (GENERIC) #0:
Wed Dec 1 10:58:25 UTC 2004 builds@build:/big/builds/ab/netbsd-2-0-RELEASE/
i386/200411300000Z-obj/big/builds/ab/
netbsd-2-0-RELEASE/src/sys/arch/i386/compile/GENERIC i386

First I downloaded base.tgz, comp.tgz, kern-GENERIC.tgz, man.tgz, misc.tgz, and text.tgz into a directory I created called /usr/tmp. The FTP server directory listing shows what was available in total.

juneau# ftp ftp://ftp.netbsd.org/pub/NetBSD/NetBSD-2.0.2/i386/binary/sets/
Trying 204.152.190.13...
Connected to ftp.netbsd.org.
...edited...
ftp> ls
229 Entering Extended Passive Mode (|||51263|)
150 Opening ASCII mode data connection for '/bin/ls'.
total 227868
-rw-r--r-- 1 srcmastr netbsd 444 Mar 30 06:10 BSDSUM
-rw-r--r-- 1 srcmastr netbsd 581 Mar 30 06:10 CKSUM
-rw-r--r-- 1 srcmastr netbsd 987 Mar 30 06:10 MD5
-rw-r--r-- 1 srcmastr netbsd 448 Mar 30 06:10 SYSVSUM
-rw-r--r-- 1 srcmastr netbsd 18047923 Mar 23 09:24 base.tgz
-rw-r--r-- 1 srcmastr netbsd 21308430 Mar 23 09:25 comp.tgz
-rw-r--r-- 1 srcmastr netbsd 168642 Mar 23 09:23 etc.tgz
-rw-r--r-- 1 srcmastr netbsd 3036879 Mar 23 09:23 games.tgz
-rw-r--r-- 1 srcmastr netbsd 3565556 Mar 23 09:20 kern-GENERIC.MP.tgz
-rw-r--r-- 1 srcmastr netbsd 3526224 Mar 23 09:20 kern-GENERIC.tgz
-rw-r--r-- 1 srcmastr netbsd 3557989 Mar 23 09:20 kern-GENERIC_DIAGNOSTIC.tgz
-rw-r--r-- 1 srcmastr netbsd 2632995 Mar 23 09:20 kern-GENERIC_LAPTOP.tgz
-rw-r--r-- 1 srcmastr netbsd 859230 Mar 23 09:20 kern-GENERIC_PS2TINY.tgz
-rw-r--r-- 1 srcmastr netbsd 822571 Mar 23 09:20 kern-GENERIC_TINY.tgz
lrwxrwxrwx 1 1237 netbsd 27 Apr 6 17:13 man.tgz ->
../../../shared/ALL/man.tgz
lrwxrwxrwx 1 1237 netbsd 28 Apr 6 17:13 misc.tgz ->
../../../shared/ALL/misc.tgz
-rw-r--r-- 1 srcmastr netbsd 1982544 Mar 23 09:23 text.tgz
-rw-r--r-- 1 srcmastr netbsd 5787182 Mar 23 09:24 xbase.tgz
-rw-r--r-- 1 srcmastr netbsd 10732131 Mar 23 09:24 xcomp.tgz
-rw-r--r-- 1 srcmastr netbsd 32342 Mar 23 09:24 xetc.tgz
-rw-r--r-- 1 srcmastr netbsd 32171614 Mar 23 09:25 xfont.tgz
-rw-r--r-- 1 srcmastr netbsd 8053884 Mar 23 09:25 xserver.tgz

Next I switched to a bash shell and ran the following:

juneau# ls *.tgz
base.tgz kern-GENERIC.tgz misc.tgz
comp.tgz man.tgz text.tgz
juneau# bash
bash-2.05b# for i in *.tgz; do tar -xpvzf $i -C /; done

This extracted all of the archives. When done, I rebooted the system. Upon reboot, I checked uname output.

NetBSD juneau.taosecurity.com 2.0.2 NetBSD 2.0.2 (GENERIC) #0:
Wed Mar 23 08:53:42 UTC 2005
jmc@faith.netbsd.org:/home/builds/ab/netbsd-2-0-2-RELEASE/i386/
200503220140Z-obj/home/builds/ab/netbsd-2-0-2-RELEASE/
src/sys/arch/i386/compile/GENERIC i386

That's it -- I'm running NetBSD 2.0.2 now.

Marcus Ranum on Proxies, Deep Packet Inspection

I asked security guru Marcus Ranum if he would mind commenting on using proxies as security devices. I will publish his thoughts in my new book Extrusion Detection, but he's allowed me to print those comments here and now. I find them very interesting.

"The original idea behind proxies was to have something that interposed itself between the networks and acted as a single place to 'get it right' when it came to security. FTP-gw was the first proxy I wrote, so I dutifully started reading the RFC for FTP and gave up pretty quickly in horror. Reading the RFC, there was all kinds of kruft in there that I didn't want outsiders being able to run against my FTP server. So the first generation proxies were not simply an intermediary that did a 'sanity check' against application protocols, they deliberately limited the application protocols to a command subset that I felt was the minimum that could be done securely and still have the protocol work.

For example, while writing FTP-gw I realized that the PORT command would cause a blind connection from TCP port 20 to anyplace, over which data would be transferred. Port 20 was in the privileged range and I was able to predict an 'rshd' problem so I quietly emailed the guys at Berkeley and got them to put a check in ruserok() to forestall the attack. I also added code to make sure that the client couldn't ask the server to send data to any host other than itself, forestalling FTP PORT scanning. So I 'invented' 'FTP bounce' attacks in 1990 by preventing them with my proxy.

There are several other examples of where, in implementing proxies, I was horrified to see gaping holes in commonly-used application protocols, and was able to get them fixed before they were used against innocent victims. Since the proxies implemented the bare minimum command set to allow an application protocol to work, a lot of attacks simply failed against the proxy because it only knew how to do a subset of the protocol. When the hackers started searching for vulnerabilities to exploit in the internet applications suite, they often found holes in seldom-used commands or in poorly tested features of the RFC-compliant servers. But often, when they tried their tricks against the proxy all they'd get back was: 'command not recognized'.

The proxy gave a single, controllable, point where these fixes could be installed.

The problem with proxies is that they took time and effort and a security-conscious analyst to write them and design them. And sometimes the proxy ran up against a design flaw in an application
protocol where it couldn't really make any improvement to the protocol. The first reaction of the proxy firewall vendors was to tell their customers, 'well, protocol XYZ is so badly broken that
you just can't rely on it across an untrusted network.' This was, actually, the correct way (from a security standpoint) to solve the problem, but it didn't work.

Around the time when the Internet was becoming a new media phenomenon, a bunch of firewalls came on the market that were basically smart packet filters that did a little bit of layer-7 analysis. These 'stateful firewalls' were very attractive to end users because they didn't even TRY to 'understand' the application protocols going across them, except for the minimum amount necessary to get the data through.

For example, one popular firewall from the early 90's managed the FTP PORT transaction by looking for a packet from the client with the PORT command in it, parsing the PORT number and address out of it, and opening that port in its rulesbase. Fast? Certainly. Transparent to the end user? Totally. Secure? Hardly. But these 'stateful firewalls' sold very well because they offered the promise of high performance, low end-user impact, and enough security that an IT manager could
say they'd tried.

There are a few vendors who have continued to sell proxy firewalls throughout the early evolution of the Internet, but most of the proxy firewalls are long gone. Basically, the customers
didn't want security; they wanted convenience and the appearance of having tried. What's ironic is that a lot of the attacks that are bedeviling networks today would never have gotten through the early proxy firewalls. But, because the end user community chose convenience over security, they wound up adopting a philosophy of preferring to let things go through, then violently slamming the barn door after the horse had exited.

Proxies keep cropping up over and over, because they are fundamentally a sound idea. Every so often someone re-invents the proxy firewall - as a border spam blocker, or a 'web firewall' or an 'application firewall' or 'database gateway' - etc. And these technologies work wonderfully. Why? Because they're a single point where a security-conscious programmer can assess the threat represented by an application protocol, and can put error detection, attack detection, and validity checking in place.

The industry will continue to veer back and forth between favoring connectivity and favoring security depending on which fad is in the ascendant. But proxies are going to be with us for the long term, and have been steadfastly keeping networks secure as all the newfangled buzzword technologies ('stateful packet inspection,' 'intrusion prevention,' 'deep packet inspection') come - and go."

This is a report he wrote for a client, but he's sharing it with the world. If you're looking for some cynical digital security artwork to grace your cubicle, check out Marcus' media outlet.

Rabu, 25 Mei 2005

New Net Optics Product Evaluations

I recently acquired several more specialized taps from Net Optics. I thought you might like to hear a few words about them. I plan to feature these and a few other devices in my new book Extrusion Detection, but why wait until then? I specifically requested evaluation units to meet monitoring and network access problems my clients brought to me. Perhaps you will find one or more of these products answer a monitoring question you've also been pondering. Keep in mind that I show Ethernet versions here, but a variety of optical products are offered. Also, I mention these products as they might be deployed at the perimeter, between a border router and firewall. They can certainly be used elsewhere, but for consistency here I stay with that deployment scenario.

The first product I tried was the 10/100 Active Response Dual Port Aggregator Tap. The purpose of this device is to provide full duplex access to a network link to two sensor platforms. The two outputs on the left of the tap accept lines from your border router and firewall. The two outputs on the right of the tap each contain the aggregated traffic of the two transmit (TX) lines, one from the firewall and one from the router.

Here are examples of traffic captured by two sensors. One uses interface sf0 to listen for traffic, and the other uses sf1. I run the captures from each through Tcpdump to show what each interface saw.

bourque# tcpdump -n -r dual_port_agg.sf0.lpc
reading from file dual_port_agg.sf0.lpc, link-type EN10MB (Ethernet)
11:01:16.965312 IP 192.168.2.94 > 192.168.2.7: icmp 64: echo request seq 0
11:01:16.965490 IP 192.168.2.7 > 192.168.2.94: icmp 64: echo reply seq 0
11:01:17.971660 IP 192.168.2.94 > 192.168.2.7: icmp 64: echo request seq 1
11:01:17.971796 IP 192.168.2.7 > 192.168.2.94: icmp 64: echo reply seq 1

bourque# tcpdump -n -r dual_port_agg.sf1.lpc
reading from file dual_port_agg.sf1.lpc, link-type EN10MB (Ethernet)
11:01:16.965361 IP 192.168.2.94 > 192.168.2.7: icmp 64: echo request seq 0
11:01:16.965537 IP 192.168.2.7 > 192.168.2.94: icmp 64: echo reply seq 0
11:01:17.971732 IP 192.168.2.94 > 192.168.2.7: icmp 64: echo request seq 1
11:01:17.971843 IP 192.168.2.7 > 192.168.2.94: icmp 64: echo reply seq 1

Notice you see traffic in both directions, and each copy is exactly the same.

Dual port aggregator taps allow you to deploy two monitoring systems, as shown in the diagram below.

The only drawback to a device like this appears if your aggregated traffic load exceeds 100 Mbps for a sustained period. In other words, a full duplex link can send 100 Mbps in one direction and 100 Mbps in the other. If the sum of the two TX lines seldom or never exceeds 100 Mbps, an aggregator tap is the perfect way to consolide the two TX lines into a single output interface. When the Dual Port Aggregator Tap is used, you have two complete copies of the aggregated TX lines available.

If you find yourself in a situation where you routinely exceed 100 Mbps aggregate, then you should consider this next solution. The 10/100 Regeneration Tap takes the TX lines from, say, your router and firewall, and sends those lines out multiple interfaces. The device pictured at left is a 2X1 model, meaning you get two copies of each TX line. (It's sort of like having two traditional taps in series.) You could plug the router line into the port at the very far right and the firewall line into the port right next to it. The interfaces on the left go to your pair of monitoring platforms. Your sensor needs to know how to bond the two TX lines together to see them as a single stream. (I use the commands here on FreeBSD.)

To understand how this device works, consider the following captures. Here interfaces sf0 and sf1 sit on one sensor, while sf2 and sf3 are located on the second sensor. Interfaces sf0 and sf2 see "Network A", and sf1 and sf3 connect to "Network B" on the tap. Here is what each interface sees, as captured and then reviewed using Tcpdump.

bourque# tcpdump -n -r 2x1_regen_tap_pre-bond.sf0.lpc
reading from file 2x1_regen_tap_pre-bond.sf0.lpc, link-type EN10MB (Ethernet)
11:14:53.818127 IP 192.168.2.94 > 192.168.2.7: icmp 64: echo request seq 0
11:14:54.827103 IP 192.168.2.94 > 192.168.2.7: icmp 64: echo request seq 1

bourque# tcpdump -n -r 2x1_regen_tap_pre-bond.sf1.lpc
reading from file 2x1_regen_tap_pre-bond.sf1.lpc, link-type EN10MB (Ethernet)
11:14:53.818293 IP 192.168.2.7 > 192.168.2.94: icmp 64: echo reply seq 0
11:14:54.827238 IP 192.168.2.7 > 192.168.2.94: icmp 64: echo reply seq 1

bourque# tcpdump -n -r 2x1_regen_tap_pre-bond.sf2.lpc
reading from file 2x1_regen_tap_pre-bond.sf2.lpc, link-type EN10MB (Ethernet)
11:14:53.818082 IP 192.168.2.94 > 192.168.2.7: icmp 64: echo request seq 0
11:14:54.827041 IP 192.168.2.94 > 192.168.2.7: icmp 64: echo request seq 1

bourque# tcpdump -n -r 2x1_regen_tap_pre-bond.sf3.lpc
reading from file 2x1_regen_tap_pre-bond.sf3.lpc, link-type EN10MB (Ethernet)
11:14:53.818248 IP 192.168.2.7 > 192.168.2.94: icmp 64: echo reply seq 0
11:14:54.827186 IP 192.168.2.7 > 192.168.2.94: icmp 64: echo reply seq 1

This is good, but we would prefer to have interfaces sf0 and sf1 bonded on the first sensor to show a single stream. We also want sf2 and sf3 bonded on the second sensor for the same reason. Once bonded, the traffic looks like this on virtual interfaces ngeth0 and ngeth1.

bourque# tcpdump -n -r 2x1_regen_tap_post-bond.ngeth0.lpc
reading from file 2x1_regen_tap_post-bond.ngeth0.lpc, link-type EN10MB (Ethernet)
11:25:01.763213 IP 192.168.2.94 > 192.168.2.7: icmp 64: echo request seq 0
11:25:01.763345 IP 192.168.2.7 > 192.168.2.94: icmp 64: echo reply seq 0
11:25:02.767909 IP 192.168.2.94 > 192.168.2.7: icmp 64: echo request seq 1
11:25:02.767918 IP 192.168.2.7 > 192.168.2.94: icmp 64: echo reply seq 1

bourque# tcpdump -n -r 2x1_regen_tap_post-bond.ngeth1.lpc
reading from file 2x1_regen_tap_post-bond.ngeth1.lpc, link-type EN10MB (Ethernet)
11:25:01.763202 IP 192.168.2.94 > 192.168.2.7: icmp 64: echo request seq 0
11:25:01.763336 IP 192.168.2.7 > 192.168.2.94: icmp 64: echo reply seq 0
11:25:02.767873 IP 192.168.2.94 > 192.168.2.7: icmp 64: echo request seq 1
11:25:02.767893 IP 192.168.2.7 > 192.168.2.94: icmp 64: echo reply seq 1

This 2X1 device isn't the only option. There are also 4X1 and 8X1 versions available, if you need to send traffic to more than two sensors.

So far we've seen two taps which provide more than one copy of network traffic traversing a link. Is there a way to use a tap with a SPAN port?

Yes there is -- the Span Regeneration Tap is the answer. This product looks exactly the same as the previous device. However, the two interfaces on the far right of the box don't connect to your router and firewall. Instead, they connect to SPAN ports from your enterprise class switches. Whatever is connected to the port at the far right is duplicated on the ports labelled "B" on the left side of the tap. Whatever is connected to the port second furthest from the right is duplicated on the "A" ports on the left side of the tap.

This is a regeneration tap, meaning it is designed to take copies of observed traffic and send them to more than one sensor. In the following traces, interfaces sf0 and sf1 are again on one sensor, and sf2 and sf3 are on a second sensor.

bourque# tcpdump -n -r span_tap_sf0.lpc
reading from file span_tap_sf0.lpc, link-type EN10MB (Ethernet)
11:47:55.784352 IP 192.168.2.10.56047 > 192.168.2.7.53: 50548+ A? www.taosecurity.com. (37)
11:47:55.785463 IP 192.168.2.7.53 > 192.168.2.10.56047: 50548* 1/1/1 A 66.93.110.10 (90)
11:47:55.797978 IP 192.168.2.10.56047 > 192.168.2.7.53: 50548+ A? www.taosecurity.com. (37)
11:47:55.805704 IP 192.168.2.7.53 > 192.168.2.10.56047: 50548* 1/1/1 A 66.93.110.10 (90)

bourque# tcpdump -n -r span_tap_sf1.lpc
reading from file span_tap_sf1.lpc, link-type EN10MB (Ethernet)
11:50:33.465715 IP 192.168.2.10.58009 > 192.168.2.7.53: 56871+ A? www.sguil.net. (31)
11:50:33.587480 IP 192.168.2.7.53 > 192.168.2.10.58009: 56871 3/5/4
CNAME wfb.zoneedit.com., A 207.234.129.65, A 216.98.141.250 (248)
11:50:33.590831 IP 192.168.2.10.58009 > 192.168.2.7.53: 56871+ A? www.sguil.net. (31)
11:50:33.687485 IP 192.168.2.7.53 > 192.168.2.10.58009: 56871 3/5/4
CNAME wfb.zoneedit.com., A 207.234.129.65, A 216.98.141.250 (248)

bourque# tcpdump -n -r span_tap_sf2.lpc
reading from file span_tap_sf2.lpc, link-type EN10MB (Ethernet)
11:47:55.784265 IP 192.168.2.10.56047 > 192.168.2.7.53: 50548+ A? www.taosecurity.com. (37)
11:47:55.785390 IP 192.168.2.7.53 > 192.168.2.10.56047: 50548* 1/1/1 A 66.93.110.10 (90)
11:47:55.797888 IP 192.168.2.10.56047 > 192.168.2.7.53: 50548+ A? www.taosecurity.com. (37)
11:47:55.805617 IP 192.168.2.7.53 > 192.168.2.10.56047: 50548* 1/1/1 A 66.93.110.10 (90)

bourque# tcpdump -n -r span_tap_sf3.lpc
reading from file span_tap_sf3.lpc, link-type EN10MB (Ethernet)
11:50:33.465631 IP 192.168.2.10.58009 > 192.168.2.7.53: 56871+ A? www.sguil.net. (31)
11:50:33.587403 IP 192.168.2.7.53 > 192.168.2.10.58009: 56871 3/5/4
CNAME wfb.zoneedit.com., A 207.234.129.65, A 216.98.141.250 (248)
11:50:33.590747 IP 192.168.2.10.58009 > 192.168.2.7.53: 56871+ A? www.sguil.net. (31)
11:50:33.687403 IP 192.168.2.7.53 > 192.168.2.10.58009: 56871 3/5/4
CNAME wfb.zoneedit.com., A 207.234.129.65, A 216.98.141.250 (248)

So what are we looking at? It appears sf0 and sf2 (on different sensors) see the same SPAN port output, which here shows a DNS request and reply for www.taosecurity.com. Interfaces sf1 and sf3 (again on different sensors) see SPAN port output from a different switch, where a DNS request and reply for www.sguil.net has been recorded.

There is an important difference between these four traces for interfaces sf0 - sf3 and the four traces shown for sf0 - sf3 for the 2X1 Regeneration Tap. The 2X1 Regeneration Tap showed only half-duplex traffic, meaning packets sent in one direction only. We used bonding to bring interfaces sf0 and sf1 together, and sf2 and sf3 together, to display a single full duplex stream on the sensor.

Here, we are already looking at full duplex output on each interface, sf0 - sf3. Remember that we are getting our packets from two separate SPAN ports in this case. The tap is not directly inline -- two enterprise switches are collecting traffic and sending it to the tap. There is no need to bond interfaces here because we are already looking at full duplex streams as provided by the switch SPAN ports.

However, we could bond interfaces sf0 and sf1 together on one sensor, and sf2 and sf3 on the other sensor, if we wanted to present a single virtual interface to the sniffing software on each platform. If we do that, we can now see the output from two SPAN ports combined into a single virtual interface. It would look something like this.

11:55:06.032533 IP 192.168.2.10.56047 > 192.168.2.7.53: 50548+ A? www.taosecurity.com. (37)
11:55:06.036645 IP 192.168.2.10.58009 > 192.168.2.7.53: 56871+ A? www.sguil.net. (31)
11:55:06.037014 IP 192.168.2.7.53 > 192.168.2.10.56047: 50548* 1/1/1 A 66.93.110.10 (90)
11:55:06.041972 IP 192.168.2.7.53 > 192.168.2.10.58009: 56871 3/5/4
CNAME wfb.zoneedit.com., A 207.234.129.65, A 216.98.141.250 (248)
11:55:06.045319 IP 192.168.2.10.56047 > 192.168.2.7.53: 50548+ A? www.taosecurity.com. (37)
11:55:06.062005 IP 192.168.2.7.53 > 192.168.2.10.56047: 50548* 1/1/1 A 66.93.110.10 (90)
11:55:06.065079 IP 192.168.2.10.58009 > 192.168.2.7.53: 56871+ A? www.sguil.net. (31)
11:55:06.072073 IP 192.168.2.7.53 > 192.168.2.10.58009: 56871 3/5/4
CNAME wfb.zoneedit.com., A 207.234.129.65, A 216.98.141.250 (248)
11:55:06.075315 IP 192.168.2.10.56047 > 192.168.2.7.53: 50548+ A? www.taosecurity.com. (37)
11:55:06.081956 IP 192.168.2.7.53 > 192.168.2.10.56047: 50548* 1/1/1 A 66.93.110.10 (90)
11:55:06.085050 IP 192.168.2.10.58009 > 192.168.2.7.53: 56871+ A? www.sguil.net. (31)
11:55:06.101978 IP 192.168.2.7.53 > 192.168.2.10.58009: 56871 3/5/4
CNAME wfb.zoneedit.com., A 207.234.129.65, A 216.98.141.250 (248)

So how might you use this device in real life? You may have an enterprise switch mirroring traffic to a SPAN port. You would like multiple sensors to watch that SPAN port. Rather than send the traffic from the SPAN port into a cheap hub, you send the SPAN output to a 2X1 (or 4X1 or 8X1) SPAN Regeneration tap. Everything stays at full duplex for highest performance. Also, consider the consequences of sending tapped traffic to a hub; if traffic enters the hub at the same time, the packets collide but no retransmission occurs. The tap is passive so those collided packets are lost forever. Taps and hubs never mix.

You may have noticed that the last two products looked physically identical. However, they are not electrically identical. In other words, on the SPAN Regeneration Tap, the two ports on the far right cannot pass traffic through themselves, as is the case with the 2X1 Regeneration Tap.

Keep an eye out for additional monitoring product reviews. I have a couple Matrix Switches on the way.

Notes on Net Optics Think Tank

Last week I had the good fortune to be invited to speak at a Net Optics Think Tank event. Net Optics is a California-based maker of products which help analysts access traffic for monitoring the security and performance of the network. I recently wrote about the Net Optics tap built in a PCI card form factor. I also use their gear to conduct network security monitoring, as profiled in my first book.

The meeting offered attendees three sessions: the first two were conducted by Net Optics personnel, and I presented the third. The purpose of the sessions were not to sell products, but to solicit feedback from attendees. In fact, in some cases the "products" in question didn't exist yet. Rather than implement products customers might not want, or lacking desired features, Net Optics polls its clients and prospective customers and builds the gear those customers need.

The first presentation described the Bypass Switch. This is a really interesting product which I had not paid any attention to prior to the Think Tank event. It is primarily designed to work with inline devices like so-called intrusion prevention systems. Apparently many prospective IPS customers are reluctant to deploy an IPS inline along with their routers and firewalls. Either customers have grown to trust the reliability of those products, or routers and firewalls usually support clustering and/or active failover to mitigate the risk of network down time. IPSs are apparently not that mature. Enter the bypass switch.

As the diagram below shows, traffic is first sent to the bypass switch. The bypass switch ensures that the IPS is alive by sending special heartbeat packets through it. As long as the IPS is functioning (i.e., passing heartbeats), the bypass switch sends traffic to the IPS. Should the heartbeat packets fail to traverse the IPS, the bypass switch reroutes traffic to go directly through the switch, thereby avoiding the failed IPS.

This makes sense for a single IPS that is deployed inline and that supplements a filtering router and firewall. Using a bypass switch would not make much sense if the connected device was a site's only access control mechanism. Who would want traffic to pass if their firewall died?

If power is lost to the bypass switch, it will still pass traffic. It is truly a passive device. This is a big improvement over the average inline appliance, whose loss of power or operating system would bring the entire link down.

The innovation that Net Optics continues to apply to this scenario involves support for dual or multiple devices attached to the bypass switch. In other words, add a second IPS to complement the primary system. Should number one IPS die, route traffic through number two. There are additional opportunities for creativity here, so I suggest watching Net Optics for future product releases.

The second session discussed adding Simple Network Management Protocol (SNMP) support to Net Optics devices. This would allow a tap to report statistics similar to those provided by Cisco device interfaces. While this is useful, I recommended Net Optics provide hardware dip switches to completely disable the feature. Those that want "dumb" taps should still be allowed to deploy them!

Finally, Net Optics asked me to speak on how I conduct monitoring. I delivered a presentation on defensible networks, based on the chapter of the same name in my forthcoming book Extrusion Detection. Essentially, a defensible network is one that can be monitored, controlled, minimized, and kept current. If you would like to hear me speak on this subject, I will be offering a similar presentation at Techno Security 2005 on 13 June in Myrtle Beach, SC. Incidentally, the pictures showing me speak at the Net Optics Think Tank link to larger versions.

Selasa, 24 Mei 2005

Virtual Desktops on Windows

I've been working in Windows more than usual recently as I push to complete my next book Extrusion Detection. I realized I really missed having multiple virtual desktops, like I do using Fluxbox or generally any UNIX windowing environment.

Enter Virtual Dimension. This is an open source virtual desktop system for Windows. It works flawlessly on my Windows 2000 Professional laptop. At the right you can see the small desktop that appears when you click on a tray icon. I like being able to see small icons representing the applications active on each desktop. For example, desktop 0 is running Firefox, 1 has Adobe Reader, 2 has Putty and Wish (for Sguil), and 3 has Windows Media Player. I don't know how I coped with a single Windows desktop before using this program.

Senin, 23 Mei 2005

Reviewers for Extrusion Detection Wanted

Would anyone be interested in reviewing preliminary drafts of some chapters from my future book Extrusion Detection? If so, email richard at taosecurity dot com. Please explain why you think you would be a good reviewer and tell me something about your network security experience. If you sound like a good candidate, I will pass your information on to my publisher. Thank you!

Update: Thanks to all who replied -- I sent a list of names to my publisher. They will be contacting some of you, depending on when you wrote me.

Microsoft Windows Server 2003 Trial Downloads

I'm not one to ignore free software from Microsoft, even if it's only in trial format. I saw that a beta of Windows Server 2003 R2 is available for download. You must install it on the trial version of Windows Server 2003 with Service Pack 1 (SP1); normal Windows Server 2003 SP1 will apparently not work. I registered to download R2 beta and also Windows Server 2003 SP1. You can get them on CD as well, but that takes 4-6 weeks. I might try converting server to workstation using the provided link.

Minggu, 22 Mei 2005

Report from BSDCan, Part II

I recently reported on day one of BSDCan 2005, which I attended in Ottawa. I'd like to present my review of day two.

I started the day with Easy Software-Installation with pkgsrc, presented by D'Arcy Cain. I find pkgsrc interesting because it is a cross-platform package system, not just for NetBSD. Too bad the pkgsrc.org Web site has "pkgsrc: The NetBSD Packages Collection" at the top!

I would like to try pkgsrc on NetBSD of course, but also on Solaris, AIX, FreeBSD, OpenBSD, Slackware, and Debian. Besides the official packages in the tree, there is a testing ground of sorts at pkgsrc-wip (works-in-progress) and pkgsrc-netbsd.se, a Web-based front-end. It would be a great accomplishment if the BSDs were able to standardize on a single package system and pool their resources. If Linux were intergrated, that would be amazing. I learned pkgsrc even supports Windows through Interix, which is now the free Windows Services for UNIX.

D'Arcy intended for his presentation to be the first diplomatic move to begin a discussion about working with the other BSD projects. I personally believe OpenBSD has the most to gain as its ports tree is the smallest and least current of all the BSDs.

I next attended a work in progress session chaired by Robert Watson. I found the following five minute talks memorable. Pawel Dawidek described his GELI, his GEOM class for encryption. Scott Long explained the need to add journalling to UFS. Poul-Henning Kamp explained his work writing drivers for the Napatech Traffic Analyzer NIC pictured at left. If I want one of these PCI-X 100 or 133 MHz boards I need to pay $4000 for the four port version or $8000 for a stacked eight port model.

I heard George Neville-Neil ask for the community to write technical white papers for marketing to non-BSD users. (Incidentally, while looking for GNN's home page, I found his work on the zoo.freebsd.org test cluster used and administered by The FreeBSD Project for testing, debugging, and performance analysis. Next I listened to a Linux user, Matthew Wilcox, advocate BSD involvement in the Free Standards Group. That took guts! Finally, Robert Krten showed off some really out-of-there telecom gear that alerts him of incoming phone calls via TV alert. Good grief.

During lunch I spoke with Scott Ullrich and Chris Buechler of the pfSense project. Pfsense is a derivative of m0n0wall. Both are firewall-and-more FreeBSD distributions. I would like to try pfSense on my Soekris net4801 system, although they suggested I look at Netgate, Hacom, and PC Engines (WRAP) products. Chris is also working on a "LiveNSM" FreeBSD distro based on FreeSBIE, which would help me immensely. I created something similar last year for internal use, but a public version would be great. I plan to keep in touch with these guys!

After lunch I was glad to see Ike Levy talk about building FreeBSD jails. He announced jailing.net as a central community resource for all jailing information. His talk was probably the most creative and entertaining of the conference, and perhaps of several conferences I have attended. His was the last presentation I attended, as I had to catch a cab to the airport. It turns out Ottawa was suffering a taxi driver strike, and only seven cars were serving the whole city! I was lucky to make my flight, even though our departure was delay by bad weather in northern Virginia.

Overall I really enjoyed my second year at BSDCan and I hope to attend next year. Perhaps I can present the sequel to my talk on keeping FreeBSD up-to-date, which is keeping FreeBSD applications up-to-date.

Kamis, 19 Mei 2005

Great Firewall Round-Up in NWC

A recent issue of Network Computing magazine featured an excellent set of firewall reviews. I thought Greg Shipley's Analyzing the Threat-Management Market cover piece to be very insightful. Here are a few excerpts:

"Our testing uncovered overhyped features, signs of innovation, emerging challenges and useful new capabilities. But what struck us most was what isn't being said -- that market demands are shifting the ground under legacy firewall vendors, and some will have a hard time holding on. Network access control is no longer a perimeter-only game, and the need for protection mechanisms is deeper than the address-and-service restrictions we're used to. More important, network security is no longer viewed as a product to be tacked on, but rather a core requirement. This is a fundamental shift in thinking."

Security as a "core requirement" is great news. It's only taken a decade of constant Internet attack to get vendors to understand this point, but at least we're getting there now.

"We know enterprises building a cohesive network-protection strategy need these questions answered: Will next-generation routers and switches come bundled with application-level policy enforcement? Will silicon-based network-access-control devices finally overtake their PC-based rivals? Will today's NIPS (network intrusion-prevention systems) become tomorrow's firewalls, or vice versa? Will network infrastructure heavyweights like Cisco Systems and Juniper Networks finally put the squeeze on companies like Check Point Software Technologies and Internet Security Systems?"

Here are my answers. Yes, yes, NIPS and "firewalls" will be the same device, and Cisco and Juniper will buy their competition.

"There's been much talk about the demise of NIDS (network-based intrusion-detection system) technology. The thinking goes that IDS will perish in favor of the seemingly more proactive IPS. This misses the bigger point: So-called network intrusion prevention is another policy-enforcement capability that's closer in purpose to today's firewalls than it is to detection and monitoring efforts... the enterprise market won't continue to see NIPSs and firewalls as different technology areas, with good reason. Shouldn't our firewalls protect us from the network-based attacks that NIPS claims to deflect? Why buy and maintain two devices for essentially the same task?

NIPS is a feature, not a product; it's just taking the industry some time to figure this out. Assuming we're right, does that convergence end with NIPS and conventional firewalls? Or do advanced network access control and pattern recognition eventually make their way into conventional routers and switches? We think it's inevitable."

Wow, I'm not sure how much of this clear thinking I can take in one day. Richard Clarke's comments were enough, but I surely appreciate Greg Shipley's insights. If you read Cisco IOS Router Firewall Security, you'll see that your Cisco router can already perform classic firewall and now so-called "NIPS" functions. If you consider ACLs a firewalling function, they've been in IOS since 1993, and more advanced features like dynamic ACLs (aka lock-and-key) have been around since IOS 11.1.

The difference between the features of a router and an appliance like TippingPoint is power and number of signatures. Cisco is already building new products that compete with the pure-play NIPS vendors -- check out their Cisco ASA 5500 Series Adaptive Security Appliances. It's only a matter of time before all access control and packet passing at the edge is done on a single box.

In fact, you can visit the IPolicy Networks site to learn about their Intrusion Prevention Firewall. This is ridiculous but it shows where the vendors are heading. If you want to read Gartner's take on this, iPolicy is hosting a recent Magic Quadrant for Network Firewalls (.pdf) report.

Richard Clarke Knows the Drill

The latest edition of SC Magazine features an interview with Richard Clarke titled Failure must be a part of the plan. Hallelujah, someone with a wide speaking forum understands that prevention eventually fails. I saw Mr. Clarke speak at RAID 2003 and I was impressed by his thoughts back then. Here is a quote from his interview, with my emphasis added:

"'The first thing that corporate boards and C-level officials have to accept is that they will be hacked, and that they are not trying to create the perfect system, because nobody has a perfect system," he says.

In the end, hackers or cyberterrorists wanting to infiltrate any system badly enough will get in, says Clarke. So businesses must accept this and design their systems for failure. This is the only sure way to stay running in a crisis. It comes down to basic risk management and business continuity practices.

'Organizations have to architect their system to be failure-tolerant and that means compartmentalizing the system so it doesn't all go down... and they have to design it in a way that it's easy to bring back up,' he says."

Preach on, Mr. Clarke!

The same SC Magazine issue featured a review of the Innominate mGuard PCI, pictured at left. This is a PCI card with two NICs and an on-board firewall. You connect the NIC of your PC to one of the mGuard's interfaces, and connect the other mGuard interface to your access switch. I like the idea of offloading firewall functionality into another device. You're getting the benefit of per-host access control but you are not tied to the weaknesses of the host operating system. I just emailed Innominate asking for a demo card. If I get it, I will review it here.

I am aware of similar products from 3Com, but as far as I know they are not stand-alone units. They require central management, which involves buying an expensive software package. That is overkill for my situation.

REcon 2005 Security Conference

At BSDCan I learned of a new security conference being held in Montreal from 17 to 19 June called REcon. The speakers list looks good. I see Adam Shostack, Jack Whitsitt, Jose Nazario, Kathy Wang, Matt Shelton, and Nish Bhalla are all speaking. I won't be able to attend, but at 400 CDN this conference is a real bargain.

Cisco Releases IOS 12.4

I can't find any real press releases on this, but I noticed Cisco released IOS 12.4 this month. This is a Major Release that incorporates features developed in the IOS 12.3T line. Two product bulletins explain what's new in IOS 12.4, and an 84 slide .pdf presents similar information in PowerPoint format. I was surprised that the IOS Feature Navigator shows there is no support for the Secure Shell SSH Version 2 Server or Client for my Cisco 2651XM router in 12.4(1). Hopefully this will be added soon.

Review of Cisco Router Firewall Security Posted

Amazon.com just posted my four star review of Richard Deal's Cisco Router Firewall Security. I read half of it on the flight from DC to San Francisco, and the rest on the return leg. From the review:

"I really enjoyed reading Cisco Router Firewall Security (CRFS) by Richard Deal. This book delivers just what a technical Cisco book should: discussion of concepts, explanation of command syntax, and practical examples. The author offers several ways to solve a security problem and then recommends his preferred choice. He correctly leans towards applying cryptography when available and avoids clear-text authentication methods or control channels. If you avoid the first chapter and keep a few minor caveats in mind, I would consider CRFS to be a five-star book."

If you administer Cisco routers, I highly recommend reading this book.

Senin, 16 Mei 2005

Launch of New TaoSecurity.com

I am happy to announce the redesign of TaoSecurity.com as the corporate home of TaoSecurity. In the coming days and weeks I will transition old, more personalized content to the www.bejtlich.net domain. TaoSecurity is open for business, and I look forward to helping you with your security consulting and training needs.

I have a ton of material to blog, including a wrap-up of BSDCan and some news items. I will be flying to San Francisco Tuesday and returning very early Thursday. Don't expect too many updates until I return home. Thank you!

Jumat, 13 Mei 2005

End of the Line for Racoon at Kame.net

I've used security/racoon for years to manage the IPSec key exchange problem. I just read that the Kame project has ceased supporting Racoon; they direct users to IPSec-tools. That projects advertises itself as "a port of KAME's IPsec utilities to the Linux-2.6 IPsec implementation... [that] supports NetBSD and FreeBSD as well." Here's a recent thread on running IPSec-tools on FreeBSD.

If you're looking for an alternative to Racoon, I know of one for FreeBSD: security/isakmpd, imported from OpenBSD. I'm a little worried, since the FreeBSD port hasn't been modified since December, while the CVS interface to the OpenBSD code shows recent changes. I'm also not sure what to make of this how-to, since there is no date on it; i.e., do the problems describe therein still plague isakmpd on FreeBSD?

Speaking of IPSec, you may have seen the NISCC announcement, or the US-CERT vulnerability note. The vulnerability is really one of poor configuration. According to NISCC:

"These [vulnerable] configurations use Encapsulating Security Payload (ESP) in tunnel mode with confidentiality only, or with integrity protection being provided by a higher layer protocol. Some configurations using AH to provide integrity protection are also vulnerable. In these configurations, an attacker can modify sections of the IPsec packet, causing either the cleartext inner packet to be redirected or
a network host to generate an error message."

The IPSec-tools list mentions it, and the way to address the issue is to "Configure ESP to use both confidentiality and integrity protection."

I think this is old news, if one reads Steve Bellovin's previous work. Furthermore, it appears the stock racoon.conf protects against this, as shown here:

path include "/usr/local/etc/racoon" ;

path pre_shared_key "/usr/local/etc/racoon/psk.txt" ;

path certificate "/usr/local/etc/cert" ;

padding
{
maximum_length 20; # maximum padding length.
randomize off; # enable randomize length.
strict_check off; # enable strict check.
exclusive_tail off; # extract last one octet.
}

timer
{
# These value can be changed per remote node.
counter 5; # maximum trying count to send.
interval 20 sec; # maximum interval to resend.
persend 1; # the number of packets per a send.

# timer for waiting to complete each phase.
phase1 30 sec;
phase2 15 sec;
}

remote anonymous
{
#exchange_mode main,aggressive;
exchange_mode aggressive,main;
doi ipsec_doi;
situation identity_only;

#my_identifier address;
my_identifier user_fqdn "sakane@kame.net";
peers_identifier user_fqdn "sakane@kame.net";
#certificate_type x509 "mycert" "mypriv";

nonce_size 16;
lifetime time 1 min; # sec,min,hour
initial_contact on;
support_mip6 on;
proposal_check obey; # obey, strict or claim

proposal {
encryption_algorithm 3des;
hash_algorithm sha1;
authentication_method pre_shared_key ;
dh_group 2 ;
}
}

sainfo anonymous
{
pfs_group 1;
lifetime time 30 sec;
encryption_algorithm 3des ;
authentication_algorithm hmac_sha1;
compression_algorithm deflate ;
}

The "authentication_algorithm hmac_sha1;" takes care of authentication. You enforce encryption policy in your ipsec.conf file, which works with setkey. For example, this snippet for one of my ipsec.conf files mandates ESP tunnel mode for traffic in this VPN.

spdadd 10.4.12.10 10.4.12.1 any -P in ipsec
esp/tunnel/18.235.153.37-78.172.25.27/require;

If anyone would care to comment, I'd appreciate some additional interpretations of this issue.

Report from BSDCan, Part I

I was fortunate enough to be accepted to deliver two presentations at BSDCan 2005 today. I attended the first BSDCan last year, and this year's event seems to have attracted a bigger crowd. I heard somewhere between 150 and 190 attendees are roaming the University of Ottawa campus.

This morning worked out very well. My wife and I took our 6 month old to the airport for her first trip -- except my wife and child flew to Detroit for a wedding, and I flew to Ottawa for BSDCan. (Thanks Aimes!) I landed in time to see the rest of Colin Percival's Hyper-Threading Vulnerability talk. You can read Colin's paper in .pdf format here. This problem is not limited to FreeBSD. It affects any operating system running on an Intel HTT-capable CPU. The attack Colin discusses:

"permits local information disclosure, including allowing an unprivileged user to steal an RSA private key being used on the same machine. Administrators of multi-user systems are strongly advised to take action to disable Hyper-Threading immediately; single-user systems (i.e., desktop computers) are not affected."

After Colin's talk I participated in my first real author's book signing. I signed several copies of my book and asked the other participants to sign copies of their books I hauled from home. Unfortunately copies of my book never made it pass the border patrol! That's another story, and not my fault. :)

Immediately after the signing I hustled to set up and present Keeping FreeBSD Up-To-Date. This talk was based on my paper of the same name. Those interested in keeping applications up-to-date will find another paper on that page describing that process.

During lunch I attended a planning session of the BSD Certification Group. They reminded me to complete the task survey to help guide our certification development process.

No sooner was lunch done than I had to hurry to another room and deliver my talk More Tools for Network Security Monitoring. In addition to NSM, I discussed the new Net Optics PCI tap, Flowgrep, IPCAD, and MODE Z. You may recognize one or more of these topics as being recent blog postings.

After my second presentation I listened to Kris Kennaway describe the FreeBSD Package Cluster. You can see error logs from pointyhat, which is the master server which coordinates the package building process.

Immediately after this class ended I found myself at the BSD Certification BoF. I attended with George Rosamond, Dru Lavigne, Dan Langille, Marc Spitzer, and Jim Brown from our group. We spoke with about 50 attendees who wanted to know more about the certification process. We also solicited their advice and emphasized that our bsdcertification.org group is in no way affiliated with the crew using the .com site.

Tomorrow evening I fly home, but I hope to report on Saturday's events during the conference.

Kamis, 12 Mei 2005

Web Browser Forensics Part II

Last month I mentioned the first part of a two-part article on Web browser forensics by two friends (Rohyt Belani and Keith Jones) at security consultancy Red Cliff. Now part two is available online. This new article looks very interesting -- I suggest reading it.

Tomorrow or Saturday I hope to blog from BSDCan. See you there, maybe!

Rabu, 11 Mei 2005

Multiple New Pre-Reviews

I've received many new books in the last two weeks. Here are some pre-reviews. First we have Mastering FreeBSD and OpenBSD Security by Bruce Potter, Paco Hope, and Yanek Korff, published by O'Reilly. I have been looking forward to this book for a while. I use both operating systems to build security appliances, and that sort of work is the subject of this book. I would have preferred if the authors avoided discussing Snort and ACID, though. This is the umpteenth time I've seen "IDS" boiled down to those two well-worn and not-very-effective "solutions." Snort, yes. ACID, no. I would have been less disturbed if at least BASE, the replacement for ACID, was profiled. But no. Still, this will be the first book in the pack I plan to read.

Next we have Snort Cookbook by Angela D. Orebaugh, Simon Biles, and Jacob Babbin, published also by O'Reilly. This is O'Reilly's second Snort book in nine months. The last was Mangling Security with Snort & IDS Tools. Ok, the real title has "Managing," but I explained why I avoided that book in this post.

I'm a little worried about this new Snort book. First, imagine which Snort console is presented? You guessed it -- ACID. Ugh, no Sguil. This is a shame, as one of this book's authors attended the Sguil presentation I gave at the DC Snort Users Group meeting last June. Second, and more worrisome, the advice on taps is faulty. On p. 21, we read the following:

"If your Snort machine has only one network interface, using the passive tap, run both lines to a small hub. Then from another port of the hub, run a cable to your IDS. This will combine and maybe even buffer the traffic for the IDS and give a full duplex connection."

Wrong -- this is a nice way to never see traffic when full-duplex packets from the two transmit lines collide in the hub. The "maybe even buffer the traffic" part is funny, too. I wrote about this bad configuration in my first book and in this January 2004 post when I caught Finisar making the same mistake.

Another yellow-covered book, but I have higher hopes for this one. It's Network Security Tools: Writing, Hacking, and Modifying Security Tools by Nitesh Dhanjani and Justin Clarke, published by O'Reilly. I worked with Nitesh at Foundstone. This book reminds me of Building Open Source Network Security Tools: Components and Techniques by Mike Schiffman. NST describes how to extend Nessus, Ettercap, Nikto, and Metasploit, as well as write sniffers and packet creators. All cool.

My penultimate O'Reilly book is Apache Security by Ivan Ristic. Ivan wrote the mod_security Apache module and maintains a Web Security Blog. I would describe mod_security as a policy enforcement system for Apache, but the common market-speak would be host IPS. Ivan sent me a copy of his book specifically to review (thank you), but I will not be able to get to it immediately. It looks like just the book for anyone wishing to deploy Apache securely, however.

My last O'Reilly book is Windows Server Cookbook For Windows Server 2003 & Windows 2000 by Robbie Allen. This book looks like a good companion to Learning Windows Server 2003. Windows Server 2003 is an OS I need to become more familiar with, since I expect to encounter it more often. O'Reilly Windows books tend to be very good, considering O'Reilly's open source advocacy and its historical ties to the UNIX community. I hope I can find time for both Windows books.

I'm not sure when I'll get to this book, but I'll mention it anyway: InfoSec Career Hacking by Aaron W. Bayles, Chris Hurley, Johnny Long, and Ed Brindley. I'll read j0hnny's chapter on building a Knoppix-based test lab, but the others seem somewhat dubious. I don't see how a whole book could give advice on "landing (and keeping) a job in the infosec field." For example, the "incident response" chapter (11) looks extremely weak.

And now for something completely different -- Networking and Internetworking with Microcontrollers by Fred Eady, published by Elsevier imprint Newnes. Fred also has Implementing 802.11 with Microcontrollers: Wireless Networking for Embedded Systems Designers due out in September, and he writes articles for Circuit Cellar magazine. Reading this book is another opportunity for me to become more familiar with networking hardware.

If anyone has read any of these books already, please post your thoughts.

Attend VMWare Seminar, Get Free Workstation 5 Copy

Today I received an email from VMWare describing a new promotion. Over three days, from 14 June to 16 June, VMWare will be conducting three-hour seminars in twenty cities in the US and Canada. According to the announcement, attendees will leave with a full copy of VMWare Workstation 5, a nearly $200 value. The exact words are: "Take what you learned today and implement it with your new copy of VMware Workstation 5 - no strings attached."

This is an excellent deal. I received a free copy of Windows NT 4 years ago at a Microsoft promotion. If anyone attends the seminar at the Dulles, VA Marriott, I will see you there.

Selasa, 10 Mei 2005

Anyone Want to Speak at InfoSeCon?

I am looking for someone to take my place at the InfoSeCon conference in Dubrovnik, Croatia. If you are interested, please contact organizer Niels Bjergstrom at njb at chi-publishing dot com. You will get an all-expense paid trip to a beautiful part of Europe on the Adriatic sea. The conference is 6-9 June 2005. Thank you.

Spamcop blocking Gmail

Anyone else seeing Spamcop used to deny email from Gmail? This is killing me. Here are two recent examples:

This is an automatically generated Delivery Status Notification

Delivery to the following recipient failed permanently:

snort-users@lists.sourceforge.net

Technical details of permanent failure:
PERM_FAILURE: SMTP Error (state 9): 550
See http://www.spamcop.net/bl.shtml for more information."

The second problem:

This is an automatically generated Delivery Status Notification

Delivery to the following recipient failed permanently:

obscured@obscured.obs

Technical details of permanent failure:
PERM_FAILURE: SMTP Error (state 10): 554 Service unavailable;
Client host [64.233.162.202] blocked using bl.spamcop.net;
Blocked - see http://www.spamcop.net/bl.shtml?64.233.162.202

Is anyone else seeing this?

Sourcefire Founder Demolishes IPS Advocate

Many thanks to ghost16825 for pointing me towards this excellent InfoWorld article: The great intrusion prevention debate. The article pits Sourcefire founder Marty Roesch against TippingPoint Chief Technology and Strategy Officer Marc Willebeek-LeMair. Folks, this one is not pretty. Marty demolishes Dr. Willebeek-LeMair by correctly arguing that IPS (called layer 7 firewalls by the Blog and elsewhere) is "a step in the right direction, but... the infrastructure itself can be orchestrated effectively to provide a much broader capability than just point defense in the face of a pervasive threat." Dr. Willebeek-LeMair's main defense: "To be as polite and as succinct as possible: You are simply misinformed."

This debate shows how a hardware vendor with a fast packet processing systems thinks he can change the world. Dr. Willebeek-LeMair's market-speak falls flat when critiqued by an actual security expert (Marty).

I highly recommend reading the entire interview. Some of you may remember the promises made by firewall vendors and see that the IPS claims are eerily similar. While I agree with Dr. Willebeek-LeMair's assertion that "IPSes will be integrated within switch and router elements" (it's happening now), the IPS is not a panacea.

Senin, 09 Mei 2005

FreeBSD 5.4 RELEASE Available

Several people let me know that FreeBSD 5.4 RELEASE was made publicly available this morning. Thank you -- I was busy installing it on the Dell PowerEdge 750 shown in my previous blog entry. :) You can read the dmesg output I stored at the NYCBUG site. Enjoy!

Incidentally, here is the df output after I built the sensor.

Script done on Mon May 9 09:43:45 2005
Filesystem Size Used Avail Capacity Mounted on
/dev/aacd0s2a 989M 35M 875M 4% /
devfs 1.0K 1.0K 0B 100% /dev
/dev/aacd0s2f 989M 28K 910M 0% /home
/dev/aacd0s2h 436G 265M 401G 0% /nsm
/dev/aacd0s2e 989M 12K 910M 0% /tmp
/dev/aacd0s2d 5.9G 961M 4.4G 17% /usr
/dev/aacd0s2g 4.8G 510K 4.5G 0% /var

Tap on a PCI Card

Those of you who've read my first book know I like to use taps built by Net Optics to access wired traffic. The device pictured at left is a port aggregator tap. It combines the TX side of whatever's plugged into port A with the TX side of port B into a single output on port C, using buffering if the aggregrate throughput exceeds 100 Mbps.

Today I got a chance to test the device pictured at left. It's a Net Optics PCI Port Aggregator tap. You plug this device into a 32 bit PCI slot on your monitoring station, and you effectively have the normal port aggregator tap I showed earlier sitting within your sensor. Let me show you what I mean in pictures.

This is the inside of my preferred monitoring platform, a Dell Poweredge 750. I've removed the dual Gigabit Ethernet NIC I usually order with these systems. That NIC is a PCI-X device recognized as em under FreeBSD.



In this next picture you see the Net Optics PCI tap at the top, and the dual Gigabit Ethernet NIC below it.



Here is the Net Optics PCI tap installed. It takes up almost all of the space available, but it fits nicely. You can barely see the three tap ports in this picture. The PCI tap does not appear active to the operating system. All the PCI tap needs from the PCI interface is power. There is a plug for a back-up power supply on the tap, but I did not connect it here. If power to the tap fails (i.e., the PC is off), the tap still passes traffic.



In this final picture, you see my test setup. This isn't how I would deploy it in real life. In a real deployment, say on a site perimeter, one tap interface would go to the router and the other tap interface would go to the firewall. The third tap interface connects to the sensor monitoring traffic passing between the router and firewall.

In this test deployment, I am essentially watching traffic to and from the server em0 interface by sending copies of that traffic through the tap and to the server em1 interface.

In the picture below, the tap ports are occupied by blue, yellow, and black cables. The yellow cable is connected to a switch with Internet access. The black cable next to the yellow line is connected to the live (IP addressed) interface on the server, em0. The blue cable connects the monitoring interface of the tap to the monitoring interface of the server, em1.



Do you see where I am going with this? What I can do now (for test purposes) is send and receive traffic on em0 and watch that traffic on em1. First I set up em1 to listen to what the tap sends it:

sensor# ifconfig em1 -arp up
sensor# tcpdump -n -i em1
tcpdump: WARNING: em1: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on em1, link-type EN10MB (Ethernet), capture size 96 bytes

Now I send traffic out em0:

$ ping -c 1 www.taosecurity.com
PING www.taosecurity.com (66.93.110.10): 56 data bytes
64 bytes from 66.93.110.10: icmp_seq=0 ttl=55 time=15.240 ms

--- www.taosecurity.com ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max/stddev = 15.240/15.240/15.240/0.000 ms

Here is what em1 sees courtesy of the tap:

09:43:33.263664 IP 192.168.1.41.58947 > 216.182.1.1.53:
27655+ A? www.taosecurity.com. (37)
09:43:33.276900 IP 216.182.1.1.53 > 192.168.1.41.58947:
27655 1/2/2 A 66.93.110.10 (131)
09:43:33.277149 IP 192.168.1.41 > 66.93.110.10: icmp 64: echo request seq 0
09:43:33.296513 IP 66.93.110.10 > 192.168.1.41: icmp 64: echo reply seq 0
^C
4 packets captured
4 packets received by filter
0 packets dropped by kernel

I think that is cool. What's the practical application? You could ship a monitoring appliance to a customer, with built-in tap. The customer connects her lines to the tap and the sensor listens via normal cable connection from tap monitor port to sensor monitor port. No external tap is needed. If the server fails completely, the tap will keep passing packets. I like it. If you would like to know more, email me and I will hook you up with the people I know at Net Optics.

If you'll be near Sunnyvale, CA next week, stop by for the Net Optics Think Tank on 18 May. I'll be speaking there around lunch time.

Sabtu, 07 Mei 2005

Mixed Thoughts on Inside Network Perimeter Security, 2nd Ed

I promise that I read the books I review, so this is not a review. You won't see me post anything at Amazon.com about Inside Network Perimeter Security, 2nd Ed. I read parts of it, but nowhere near enough to justify a formal review. Here are a few thoughts on the book.

The five authors and four technical editors did a lot of work to write this book. It weighs in at 660+ pages, with not that many figures or screen shots.

Despite being a second edition, I found evidence of old material. I noticed that chapter 2 describes IPChains. IPChains -- where was that last in the mainstream, in the Linux 2.2 kernel? Chapter 6 implies SSH v2 isn't available on Cisco gear, but readers will remember I got that working a few months ago. Ch 19 promotes the virtues of Big Brother, a monitoring tool that's been declining for years since its acquisition. Nagios should have been covered instead.

A quote in ch 11 on Intrusion Prevention Systems bugged me: "SoureFire [sic] ditched Snorty the pig and became Realtime Network Awareness (RNA), a passive sensor and visualization tool company in terms of primary internal focus." Let's ignore the misspellings and confusing English and answer this point. Sourcefire hasn't "ditched" Snort; RNA works with Snort. Someone doesn't understand Sourcefire or Snort.

I ended up reading most of ch 11 as it was fairly informative about network- and host-based IPSs. Otherwise, I didn't find a really compelling reason to read the book. There is some good material on network architecture, but nothing I haven't seen elsewhere. I guess that was my overall reason to stop paying attention to Inside Network Perimeter Security, 2nd Ed: I didn't see much new material for me. I also don't really care for books that provide advice but not configuration guidance. I like to flip though technical books and see that offset courier print denoting command and configuration syntax. Aside from the router hardening syntax in ch. 6, there's a lot of suggesting in this book but not as many concrete examples as I would like.

If anyone has opinions on Inside Network Perimeter Security, 2nd Ed, please post them.

Update: I reviewed this book on 30 August 2006.

Ping Tunnel and Telnet

I often learn of new software by seeing new ports released at FreshPorts. Recently I noticed Daniel Stødle's Ping Tunnel appear as net/ptunnel. Ping Tunnel tunnels TCP over ICMP traffic, as shown in the diagram at left. Being a network security analyst I thought it might be interesting to see what this traffic looks like. I set up the Ping Tunnel client on my laptop (orr, 192.168.2.5), the proxy on a server (janney,192.168.2.7), and tried to Telnet to a third server (bourque, 192.168.2.10). The results surprised me. Here is the setup.

First, I set up the proxy on janney. Here is everything janney reported.

janney:/home/richard$ sudo ptunnel -c xl0
[inf]: Starting ptunnel v 0.60.
[inf]: (c) 2004-2005 Daniel Stoedle, daniels@cs.uit.no
[inf]: Forwarding incoming ping packets over TCP.
[inf]: Initializing pcap.
[inf]: Ping proxy is listening in privileged mode.
[inf]: Incoming tunnel request from 192.168.2.5.
[inf]: Starting new session to 192.168.2.10:23 with ID 33492
[err]: Dropping duplicate proxy session request.
[inf]: Connection closed or lost.

I ran the client on orr. Here is everything orr reported.

orr:/home/richard$ sudo ptunnel -p janney -lp 8000 -da bourque -dp 23
[inf]: Starting ptunnel v 0.60.
[inf]: (c) 2004-2005 Daniel Stoedle, daniels@cs.uit.no
[inf]: Relaying packets from incoming TCP streams.
[inf]: Incoming connection.
[evt]: No running proxy thread - starting it.
[inf]: Ping proxy is listening in privileged mode.
[inf]: Received session close from remote peer.

Here is the output from the window in which I used Telnet to connect to port 8000 TCP on orr. Connecting to port 8000 TCP sends traffic to the Ping Tunnel client, who sends traffic to the Ping Tunnel proxy on janney. The Ping Tunnel proxy on janney sends TCP traffic to port 23 TCP on bourque.

orr:/home/richard$ telnet localhost 8000
Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
Connected to localhost.taosecurity.com.
Escape character is '^]'.
Trying SRA secure login:
User (richard): test
Password:
[ SRA accepts you ]

FreeBSD/i386 (bourque.taosecurity.com) (ttyp1)

Copyright (c) 1992-2004 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.

FreeBSD 5.3-RELEASE (GENERIC) #0: Fri Nov 5 04:19:18 UTC 2004

Welcome to FreeBSD!
...edited...
$ id
uid=1002(test) gid=1002(test) groups=1002(test)
$ exit
Connection closed by foreign host.

It looks like it worked. I was able to Telnet to my localhost on port 8000 TCP, and Ping Tunnel sent the traffic via ICMP to janney, who then sent TCP traffic to port 23 on bourque. So what did the traffic look like?

Here is some of what orr saw. This is ICMP traffic generated by the Ping Tunnel client, exchanged with the Ping Tunnel proxy on janney.

10:15:04.732055 IP orr.taosecurity.com > janney.taosecurity.com: icmp 36:
echo request seq 0
0x0000: 4500 0038 c442 0000 4001 3126 c0a8 0205 E..8.B..@.1&....
0x0010: c0a8 0207 0800 11ec 82d4 0000 d520 0880 ................
0x0020: c0a8 020a 0000 0017 4000 0000 0000 ffff ........@.......
0x0030: 0000 0000 0000 82d4 ........
10:15:04.737694 IP janney.taosecurity.com > orr.taosecurity.com: icmp 36:
echo reply seq 0
0x0000: 4500 0038 1253 0000 4001 e315 c0a8 0207 E..8.S..@.......
0x0010: c0a8 0205 0000 19ec 82d4 0000 d520 0880 ................
0x0020: c0a8 020a 0000 0017 4000 0000 0000 ffff ........@.......
0x0030: 0000 0000 0000 82d4 ........
10:15:04.876140 IP janney.taosecurity.com > orr.taosecurity.com: icmp 40:
echo reply seq 0
0x0000: 4500 003c 125a 0000 4001 e30a c0a8 0207 E..<.Z..@.......
0x0010: c0a8 0205 0000 7732 82d4 0000 d520 0880 ......w2........
0x0020: 0000 0000 0000 0000 8000 0002 0000 0000 ................
0x0030: 0000 0003 0000 82d4 fffd 2580 ..........%.
10:15:04.876778 IP orr.taosecurity.com > janney.taosecurity.com: icmp 40:
echo request seq 1
0x0000: 4500 003c c446 0000 4001 311e c0a8 0205 E..<.F..@.1.....
0x0010: c0a8 0207 0800 afb2 82d4 0001 d520 0880 ................
0x0020: 0000 0000 0000 0000 4000 0002 0000 0000 ........@.......
0x0030: 0000 0003 0001 82d4 fffb 2500 ..........%.

Ok, it looks like ICMP traffic as we would expect. I chose Telnet for this demo because it is a clear text protocol. Everyone knows that -- that's why we use OpenSSH, right? Let me see if I can find some clear text.

10:15:08.839045 IP orr.taosecurity.com > janney.taosecurity.com: icmp 152:
echo request seq 10
0x0000: 4500 00ac c460 0000 4001 3094 c0a8 0205 E....`..@.0.....
0x0010: c0a8 0207 0800 16e7 82d4 000a d520 0880 ................
0x0020: 0000 0000 0000 0000 4000 0002 0000 0009 ........@.......
0x0030: 0000 0073 000a 82d4 fffa 2602 0102 fff0 ...s......&.....
0x0040: fffa 2000 3338 3430 302c 3338 3430 30ff ....38400,38400.
0x0050: f0ff fa23 006f 7272 2e74 616f 7365 6375 ...#.orr.taosecu
0x0060: 7269 7479 2e63 6f6d 3a30 2e30 fff0 fffa rity.com:0.0....
0x0070: 2700 0044 4953 504c 4159 016f 7272 2e74 '..DISPLAY.orr.t
0x0080: 616f 7365 6375 7269 7479 2e63 6f6d 3a30 aosecurity.com:0
0x0090: 2e30 0055 5345 5201 7269 6368 6172 64ff .0.USER.richard.
0x00a0: f0ff fa18 0052 5856 54ff f000 .....RXVT...
10:15:08.844558 IP janney.taosecurity.com > orr.taosecurity.com: icmp 152:
echo reply seq 10
0x0000: 4500 00ac 1277 0000 4001 e27d c0a8 0207 E....w..@..}....
0x0010: c0a8 0205 0000 1ee7 82d4 000a d520 0880 ................
0x0020: 0000 0000 0000 0000 4000 0002 0000 0009 ........@.......
0x0030: 0000 0073 000a 82d4 fffa 2602 0102 fff0 ...s......&.....
0x0040: fffa 2000 3338 3430 302c 3338 3430 30ff ....38400,38400.
0x0050: f0ff fa23 006f 7272 2e74 616f 7365 6375 ...#.orr.taosecu
0x0060: 7269 7479 2e63 6f6d 3a30 2e30 fff0 fffa rity.com:0.0....
0x0070: 2700 0044 4953 504c 4159 016f 7272 2e74 '..DISPLAY.orr.t
0x0080: 616f 7365 6375 7269 7479 2e63 6f6d 3a30 aosecurity.com:0
0x0090: 2e30 0055 5345 5201 7269 6368 6172 64ff .0.USER.richard.
0x00a0: f0ff fa18 0052 5856 54ff f000 .....RXVT...

That's interesting. Those packets look like Telnet environment variables. That's normal. So where is my "Welcome to FreeBSD!" line? It's nowhere. In fact, here is some of what follows.

10:15:09.228115 IP janney.taosecurity.com > orr.taosecurity.com: icmp 1060:
echo reply seq 15
0x0000: 4500 0438 1287 0000 4001 dee1 c0a8 0207 E..8....@.......
0x0010: c0a8 0205 0000 2632 82d4 000f d520 0880 ......&2........
0x0020: 0000 0000 0000 0000 8000 0002 0000 000d ................
0x0030: 0000 0400 000f 82d4 75dc 0adc a3d1 43f5 ........u.....C.
0x0040: 75d2 9147 a100 02d4 3920 1e50 967b 990a u..G....9..P.{..
0x0050: 2983 fc5f 9d9d 54bd 7a46 0187 6e02 2a27 ).._..T.zF..n.*'
0x0060: cee9 79b3 404a c6d5 6a79 c437 d666 5507 ..y.@J..jy.7.fU.
0x0070: 413e 754b 4021 bbb9 7857 8af9 e438 6466 A>uK@!..xW...8df
0x0080: 39a0 8655 6c40 fe9b 76ab 4d19 4773 1991 9..Ul@..v.M.Gs..
...truncated...

That looks encrypted to me. Is Ping Tunnel encrypting the Telnet traffic somehow? Why didn't it encrypt the environment variable exchange?

Let's look at the traffic as bourque saw it. Host bourque is running Telnet.

10:11:05.217479 IP janney.taosecurity.com.61802 > bourque.taosecurity.com.telnet:
S 440505519:440505519(0) win 65535 <mss 1460,nop,nop,sackOK,nop,wscale 1,nop,
nop,timestamp 931812386 0>
0x0000: 4500 0040 1254 4000 4006 a302 c0a8 0207 E..@.T@.@.......
0x0010: c0a8 020a f16a 0017 1a41 94af 0000 0000 .....j...A......
0x0020: b002 ffff 847f 0000 0204 05b4 0101 0402 ................
0x0030: 0103 0301 0101 080a 378a 5422 0000 0000 ........7.T"....
10:11:05.217653 IP bourque.taosecurity.com.telnet > janney.taosecurity.com.61802:
S 3627412370:3627412370(0) ack 440505520 win 65535 <mss 1460,nop,wscale 1,nop,
nop,timestamp 931351884 931812386,nop,nop,sackOK>
0x0000: 4500 0040 1766 4000 4006 9df0 c0a8 020a E..@.f@.@.......
0x0010: c0a8 0207 0017 f16a d835 eb92 1a41 94b0 .......j.5...A..
0x0020: b012 ffff 3bd6 0000 0204 05b4 0103 0301 ....;...........
0x0030: 0101 080a 3783 4d4c 378a 5422 0101 0402 ....7.ML7.T"....
10:11:05.217932 IP janney.taosecurity.com.61802 > bourque.taosecurity.com.telnet:
. ack 1 win 33304 <nop,nop,timestamp 931812386 931351884>
0x0000: 4500 0034 1256 4000 4006 a30c c0a8 0207 E..4.V@.@.......
0x0010: c0a8 020a f16a 0017 1a41 94b0 d835 eb93 .....j...A...5..
0x0020: 8010 8218 fa89 0000 0101 080a 378a 5422 ............7.T"
0x0030: 3783 4d4c 7.ML

That's as I would expect it. I see the TCP three-way handshake from the host running the Ping Tunnel proxy (janney) to the Telnet server on bourque. How about another traffic sample?

0x0030: 3783 4eda 7.N.
10:11:09.316388 IP janney.taosecurity.com.61802 > bourque.taosecurity.com.telnet:
P 143:258(115) ack 145 win 33304 <nop,nop,timestamp 931812796 931352282>
0x0000: 4500 00a7 1279 4000 4006 a276 c0a8 0207 E....y@.@..v....
0x0010: c0a8 020a f16a 0017 1a41 953e d835 ec23 .....j...A.>.5.#
0x0020: 8018 8218 388c 0000 0101 080a 378a 55bc ....8.......7.U.
0x0030: 3783 4eda fffa 2602 0102 fff0 fffa 2000 7.N...&.........
0x0040: 3338 3430 302c 3338 3430 30ff f0ff fa23 38400,38400....#
0x0050: 006f 7272 2e74 616f 7365 6375 7269 7479 .orr.taosecurity
0x0060: 2e63 6f6d 3a30 2e30 fff0 fffa 2700 0044 .com:0.0....'..D
0x0070: 4953 504c 4159 016f 7272 2e74 616f 7365 ISPLAY.orr.taose
0x0080: 6375 7269 7479 2e63 6f6d 3a30 2e30 0055 curity.com:0.0.U
0x0090: 5345 5201 7269 6368 6172 64ff f0ff fa18 SER.richard.....
0x00a0: 0052 5856 54ff f0 .RXVT..
10:11:09.319963 IP bourque.taosecurity.com.telnet > janney.taosecurity.com.61802:
P 145:170(25) ack 258 win 33304 <nop,nop,timestamp 931352295 931812796>
0x0000: 4510 004d 176f 4000 4006 9dca c0a8 020a E..M.o@.@.......
0x0010: c0a8 0207 0017 f16a d835 ec23 1a41 95b1 .......j.5.#.A..
0x0020: 8018 8218 a77c 0000 0101 080a 3783 4ee7 .....|......7.N.
0x0030: 378a 55bc fffa 2607 00ff f0ff fb03 fffd 7.U...&.........
0x0040: 01ff fd22 fffd 1fff fb05 fffd 21 ..."........!

That's odd. I only see the Telnet environment information from janney to bourque. In fact, going back to the ICMP traffic, the data past the IP header is identical in the "icmp 152" packets. I'm not sure what that means, but it's not central to my immediate concern.

Where is my clear text Telnet protocol?

I decide to load the trace captured at bourque into Tethereal. In one of the Telnet option decodes I see the following:

Telnet
Suboption Begin: Authentication Option
Auth Cmd: REPLY (2)
Auth Type: RSA (6)
...0 .0.. = Encrypt: Off (0)
.... 0... = Cred Fwd: Client will NOT forward auth creds
.... ..0. = How: One Way authentication
.... ...0 = Who: Mask client to server
Command: Forward (4)
Command: Suboption End
Command: Will Encryption Option
Command: Do Terminal Type
Command: Do Terminal Speed
Command: Do X Display Location
Command: Do New Environment Option
Command: Do Environment Option

Interesting... "Will Encryption Option". Let's keep an eye on that. A few packets later I see the Telnet client send these Telnet options:

Telnet
Command: Do Encryption Option
Suboption Begin: Encryption Option
Option data
Command: Suboption End
Suboption Begin: Encryption Option
Option data
Command: Suboption End
Command: Will Terminal Type
Command: Will Terminal Speed
Command: Will X Display Location
Command: Will New Environment Option
Command: Won't Environment Option

Now we're seeing "Do Encryption Option". The next packet from the server agrees:

Telnet
Suboption Begin: Encryption Option
Option data

Just to wrap up the earlier clear text, here's the data from the client to the server decoded:

Telnet
Suboption Begin: Encryption Option
Option data
Command: Suboption End
Suboption Begin: Terminal Speed
Option data
Command: Suboption End
Suboption Begin: X Display Location
Here's my X Display Location
Value: orr.taosecurity.com:0.0
Command: Suboption End
Suboption Begin: New Environment Option
Option data
Command: Suboption End
Suboption Begin: Terminal Type
Here's my Terminal Type
Value: RXVT
Command: Suboption End

From this point forward, all of the data carried in Telnet appears encrypted.

This helpful Telnet protocol page pointed me to this Telnet options list. It turns out that RFC 2946 by none other than Linux kernel hacker Ted Ts'o describes how to add encryption to Telnet. It seems that the default telnetd and telnet installed with FreeBSD (and others, probably?) supports encrypted Telnet. I have never seen this before. Has anyone else? I guess I should have picked another "clear text" protocol to test Ping Tunnel!

Jumat, 06 Mei 2005

Ethereal 0.10.11 Released

Ethereal 0.10.11 was released Wednesday. It fixes a ton of security bugs. There appear to be some GUI enhancements as well as improvements under the hood. I recommend upgrading when possible.

Kamis, 05 Mei 2005

How to Go Insane Using Comcast

It's simple to go insane when using Comcast as your cable modem provider.

  1. Watch as Comcast-provided cable modem goes dead. (Not insane yet).

  2. Swap out cable modem at store. (Not insane yet).

  3. Plug in cable modem and watch router receive IP address. (Not insane yet. Happy, actually.)

  4. Notice machines begin trying to reach 1.1.1.1 when using TCP. (Slight insanity.)

  5. Observe that UDP traffic like NTP updates work properly. (Higher insanity level.)

  6. Notice that your cannot ping your default gateway. (Insane. Period.)


Apparently when my new cable modem is put on the network, it was given 68.87.96.204 (CPSDNS.selfprov.pa.comcast.net) as its DNS server. This is a really amazing system. Check it out.

orr:/home/richard$ nslookup www.google.com 68.87.96.204
Server: 68.87.96.204
Address: 68.87.96.204#53

Name: www.google.com
Address: 1.1.1.1

orr:/home/richard$ nslookup www.taosecurity.com 68.87.96.204
Server: 68.87.96.204
Address: 68.87.96.204#53

Name: www.taosecurity.com
Address: 68.87.96.200

The first 1.1.1.1 IP address is reserved. The second 68.87.96.200 belongs to act02.selfprov.pa.comcast.net, which goes to a www.comcast.net Web server. Apparently I was supposed to not use a site like Google as my home page, but something else that would bring me to the Comcast site so I could "self-provision" my new cable modem.

So why could I update time with NTP? Check out the wonders of 68.87.96.204:

orr:/home/richard$ nslookup clock.isc.org 68.87.96.204
Server: 68.87.96.204
Address: 68.87.96.204#53

Name: clock.isc.org
Address: 68.87.96.200

That's not correct.

orr:/home/richard$ nslookup clock.isc.org 209.98.98.98
Server: 209.98.98.98
Address: 209.98.98.98#53

Non-authoritative answer:
Name: clock.isc.org
Address: 204.152.184.72

But guess what -- 68.87.96.200 is running a time server.

orr:/home/richard$ sudo ntpdate 68.87.96.200
Looking for host 68.87.96.200 and service ntp
host found : act02.selfprov.pa.comcast.net
5 May 20:56:17 ntpdate[874]: adjust time server 68.87.96.200 offset 0.016223 sec

Hence, my insanity. Some applications worked (NTP), others (TCP to certain Web sites) did not. Good grief. By the way, my equipment came with zero setup instructions. I should have just called tech support earlier and said "the Internet is broken," rather than network troubleshoot!