Distributed Traffic Collection with Pf Dup-To

The following is another excerpt from my upcoming book titled Extrusion Detection: Security Monitoring for Internal Intrusions. I learned yesterday that it should be available the last week in November, around the 26th.

We’ve seen network taps that make copies of traffic for use by multiple monitoring systems. These copies are all exactly the same, however. There is no way using the taps just described to send port 80 TCP traffic to one sensor, and all other traffic to another sensor. Commercial solutions like the Top Layer IDS Balancer provide the capability to sit inline and copy traffic to specified output interfaces, based on rules defined by an administrator. Is there a way to perform a similar function using commodity hardware? Of course!

The Pf firewall introduced in Chapter 2 offers the dup-to keyword. This function allows us to take traffic that matches a Pf rule and copy it to a specified interface. Figure 4-17 demonstrates the simplest deployment of this sort of system.

Figure 4-17. Simple Pf Dup-To Deployment

First we must build a Pf bridge to pass and copy traffic. Here is the /etc/pf.conf.dup-to file we will use.

int_if="sf0"
ext_if="sf1"
l80_if="sf2"
l80_ad="1.1.1.80"
lot_if="sf3"
lot_ad="2.2.2.2"

pass out on $int_if dup-to ($l80_if $l80_ad) proto tcp from any 
 port 80 to any
pass in on $int_if dup-to ($l80_if $l80_ad) proto tcp from any
 to any port 80

pass out on $int_if dup-to ($lot_if $lot_ad) proto tcp from any
 port !=80 to any
pass in on $int_if dup-to ($lot_if $lot_ad) proto tcp from any
 to any port !=80

pass out on $int_if dup-to ($lot_if $lot_ad) proto udp from
 any to any
pass in on $int_if dup-to ($lot_if $lot_ad) proto udp from
 any to any

pass out on $int_if dup-to ($lot_if $lot_ad) proto icmp from
 any to any
pass in on $int_if dup-to ($lot_if $lot_ad) proto icmp from
 any to any

To understand this configuration file, we should add some implementation details to our simple Pf dup-to diagram. Figure 4-18 adds those details.

Figure 4-18. Simple Pf Dup-To Implementation Details

First consider the interfaces involved on the Pf bridge:

• Interface sf0 is closest to the intranet. It is completely passive, with no IP address.


• Interface sf1 is closest to the Internet. It is also completely passive, with no IP address.


• Interface sf2 will receive copies of port 80 TCP traffic sent to it by Pf. It bears the arbitrary address 1.1.1.79. Access to this interface by other hosts should be denied by firewall rules, not shown here.


• Interface sf3 will receive copies of all non-port 80 TCP traffic, as well as UDP and ICMP, sent to it by Pf. (For the purposes of this simple deployment, we are not considering other IP protocols.) It bears the arbitrary address 2.2.2.1. Access to this interface by other hosts should be denied by firewall rules, not shown here.

Now consider the two sensors.

• Sensor 1 uses its interface sf2 to capture traffic sent to it from the Pf bridge. It bears the arbitrary IP address 1.1.1.80. Access to this interface by other hosts should be denied by firewall rules, not shown here.


• Sensor 2 uses its interface sf3 to capture traffic sent to it from the Pf bridge. It bears the arbitrary address 2.2.2.2. Access to this interface by other hosts should be denied by firewall rules, not shown here.

One would have hoped the Pf dup-to function could send traffic to directly connected interfaces without the involement of any IP addresses. Unfortunately, my testing revealed that assigning IP addresses to interfaces on both sides of the link is required. I used OpenBSD 3.7, but future versions may not have this requirement.

With this background, we can begin to understand the /etc/pf.conf.dup-to file.

• The first set of declarations define macros for the interfaces and IP addresses used in the scenario.


• The first set of pass commands tell Pf to send port 80 TCP traffic to 1.1.1.80, which is the packet capture interface on sensor 1. Two rules are needed; one for inbound traffic, and one for outbound traffic.


• The second set of pass commands tell Pf to send all non-port 80 TCP traffic to 2.2.2.2, which is the packet capture interface on sensor 2. Again two rules are needed.


• The third and fourth set of pass commands send UDP and ICMP traffic to 2.2.2.2 as well.

Before testing this deployment, ensure Pf is running, and that all interfaces are appropriately configured and enabled. To test our distributed collection system, we retrieve the Google home page using Wget.

$ wget http://www.google.com/index.html
--10:19:50--  http://www.google.com/index.html
           => `index.html'
Resolving www.google.com... 64.233.187.99, 64.233.187.104
Connecting to www.google.com[64.233.187.99]:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]

    [ <=>                                 ] 1,983       --.--K/s

10:19:51 (8.96 MB/s) - `index.html' saved [1983]

Here is what sensor 1 sees on its interface sf2:

10:18:58.122543 IP 172.17.17.2.65480 > 64.233.187.99.80:
 S 101608113:101608113(0) win 32768
 
10:18:58.151066 IP 64.233.187.99.80 > 172.17.17.2.65480:
 S 2859013924:2859013924(0) ack 101608114 win 8190 
10:18:58.151545 IP 172.17.17.2.65480 > 64.233.187.99.80:
 . ack 1 win 33580
10:18:58.153027 IP 172.17.17.2.65480 > 64.233.187.99.80:
 P 1:112(111) ack 1 win 33580
10:18:58.184169 IP 64.233.187.99.80 > 172.17.17.2.65480:
 . ack 112 win 8079
10:18:58.185384 IP 64.233.187.99.80 > 172.17.17.2.65480:
 . ack 112 win 5720
10:18:58.189840 IP 64.233.187.99.80 > 172.17.17.2.65480:
 . 1:1431(1430) ack 112 win 5720
10:18:58.190344 IP 64.233.187.99.80 > 172.17.17.2.65480:
 P 1431:2277(846) ack 112 win 5720
10:18:58.190483 IP 64.233.187.99.80 > 172.17.17.2.65480:
 F 2277:2277(0) ack 112 win 5720
10:18:58.192706 IP 172.17.17.2.65480 > 64.233.187.99.80:
 . ack 2277 win 32734
10:18:58.192958 IP 172.17.17.2.65480 > 64.233.187.99.80:
 . ack 2278 win 32734
10:18:58.204719 IP 172.17.17.2.65480 > 64.233.187.99.80:
 F 112:112(0) ack 2278 win 33580
10:18:58.232685 IP 64.233.187.99.80 > 172.17.17.2.65480:
 . ack 113 win 5720

Here is what sensor 2 sees on its interface sf3.

10:18:58.089226 IP 172.17.17.2.65364 > 192.168.2.7.53:
 64302+ A? www.google.com. (32)
10:18:58.113853 IP 192.168.2.7.53 > 172.17.17.2.65364: 
 64302 3/13/13 CNAME www.l.google.com.,
 A 64.233.187.99, A 64.233.187.104 (503)

As we planned, sensor 1 only saw port 80 TCP traffic, while sensor 2 saw everything else. In this case, “everything else” meant a DNS request for www.google.com. So why build a distributed collection system? This section presented a very simple deployment scenario, but you can begin to imagine the possibilities. Network security monitoring (NSM) advocates collecting alert, full content, session, and statistical data. That can be a great amount of strain on a single sensor, even if only full content data is collected.

By building a distributed collection system, NSM data can be forwarded to independent systems built specially for the tasks at hand. In our example, we offloaded heavy Web surfing activity to one sensor, and sent all other traffic to a separate sensor.

We also split the traffic passing function from the traffic recording function. The Pf bridge in Figure 4-18 is not performing any disk input/output (IO) operations. The kernel is handling packet forwarding, which can be done very quickly. The independent sensors are accepting traffic split out by the Pf bridge. The sensors can be built to perform fast disk IO. This sort of sensor load-balancing provides a way to apply additional hardware to difficult packet collection environments.