Rough Sguil Installation Script

My last Sguil Installation Guide, for Sguil 0.5.3 was a mix of English description and command line statements. This did not help much when I needed to install a new Sguil deployment. I essentially followed my guide and typed everything by hand.

Today I decided that would be the end of that process. I am excited by the new InstantNSM project, and I intend to support it with respect to FreeBSD. But for today, I decided to just script as many Sguil installation commands as possible. For items that I couldn't easily script (due to my weak script-fu), I decided to edit the files manually and generate a patch for each one.

This post describes the end result, which you can download at www.bejtlich.net/sguil_install_v0.1.sh. I should warn you that this is not meant for public production use. However, someone trying to install Sguil might find it useful.

The purpose of this script is to automate, as much as possible, the creation of a Sguil sensor, server, and database on a FreeBSD 6.0/i386 platform. The platform is a VMware image whose hostname is gruden.taosecurity.com and whose management IP address is 192.168.2.121. I have stored several files at www.bejtlich.net to facilitate the installation. I will explain where that matters as I progress.

#!/bin/sh
# 
# Sguil installation script by Richard Bejtlich (richard@taosecurity.com)
# v1-0 28 December 2005
#
# Tested on FreeBSD 6.0 RELEASE
#
# This script sets up all Sguil components on a single FreeBSD 6.0 system
# This is not intended for production use where separate sensor, server,
# and client boxes are recommended

echo "Sguil Installation Script"
echo
echo "By Richard Bejtlich"
echo 
echo "This is mainly for personal use, but it documents how to build"
echo "a FreeBSD 6.0 system with Sguil sensor, server, and database"
echo "components.  The Sguil client must be deployed separately."

First I update the time. I am running this in a VM and time can be problematic. With FreeBSD 6 as a guest OS on VMware Workstation, I create /boot/loader.conf with 'hint.apic.0.disabled=1' to mitigate time issues.

# Update date and time

ntpdate clock.isc.org

Next I set some environment variables. I designate my proxy server, which received heavy use as I tested this script. Note that using a proxy server means copies of patches and other files are cached. To clear the cache after changing a file and uploading it www.bejtlich.net, the process involves stopping Sguid, clearing the cache map with 'echo "" >> /usr/local/squid/cache/swap.state', and restarting Sguid.

# Set environment variable for package adds

# Use proxy server if you have it!

HTTP_PROXY=192.168.2.7:3128; export HTTP_PROXY

By default this script used FreeBSD 6 packages.

# Use the following for FreeBSD 5 packages

#PACKAGESITE=ftp://ftp2.freebsd.org/pub/FreeBSD/ports/i386/packages-5-stable/Latest/; export PACKAGESITE

# FreeBSD 6 packages

PACKAGESITE=ftp://ftp2.freebsd.org/pub/FreeBSD/ports/i386/packages-6-stable/Latest/; export PACKAGESITE

Here is where the sensor name is determined. In other places (like patch files) I use the sensor name, gruden, explicity.

# Determine sensor name

SENSOR=`hostname -s`

# Set Sguil version

SGUIL=sguil-0.6.0p1

# Set Snort major version

SNORTMV=2.4

Now I create directories used by Sguil components.

# Create directories 

mkdir -p /nsm/$SENSOR/dailylogs
mkdir -p /nsm/$SENSOR/portscans
mkdir -p /nsm/$SENSOR/sancp
mkdir -p /nsm/rules/$SENSOR
mkdir -p /var/log/snort
mkdir -p /usr/local/etc/nsm
mkdir -p /usr/local/src
mkdir -p /nsm/archive
mkdir -p /nsm/rules/$SENSOR


chown -R sguil:sguil /nsm

chown -R sguil:sguil /var/log/snort

chown -R sguil:sguil /usr/local/etc/nsm  

Now I start getting software packages and archives.

# Retrieve software

cd /usr/local/src
fetch http://internap.dl.sourceforge.net/sourceforge/sguil/$SGUIL.tar.gz
tar -xzf $SGUIL.tar.gz

# Install Snort

pkg_add -r snort

cd /nsm/rules/$SENSOR 
fetch http://www.snort.org/pub-bin/downloads.cgi/Download/vrt_pr/snortrules-pr-$SNORTMV.tar.gz
tar -xzf snortrules-pr-$SNORTMV.tar.gz
mv /nsm/rules/$SENSOR/rules/* /nsm/rules/$SENSOR

chown -R sguil:sguil /usr/local/etc/snort

cd /root

# Install Tcl

pkg_add -r tcl84
mv /usr/local/bin/tclsh /usr/local/bin/tclsh.orig
ln -s /usr/local/bin/tclsh8.4 /usr/local/bin/tclsh

The installation of Barnyard uses a package I built, as described here, because the stock Barnyard package does not support Sguil 0.6.0p1.

# Install Barnyard

cd /tmp
fetch http://www.bejtlich.net/barnyard-0.2.0.tbz
pkg_add barnyard-0.2.0.tbz

# Install SANCP

pkg_add -r sancp

# Install MySQL

pkg_add -r mysql50-server
/usr/local/bin/mysql_install_db --user=mysql
/usr/local/bin/mysqld_safe --user=mysql &  

# Install Tcltls

pkg_add -r tcltls

# Install Tcllib

pkg_add -r tcllib

# Install TclX

pkg_add -r tclX

I have to install my own version of MySQLTcl. This was not as complicated as Barnyard. The problem with the stock package is that it is compiled against MySQL 4.1.x, and I am using MySQL 5.0.x. Simply building my own package on sguilref, a FreeBSD 6 host with MySQL 5.0.16 installed, is enough to create the proper mysqltcl package.

# Install MySQLTcl from own version compiled for MySQL 5.x

fetch http://www.bejtlich.net/mysqltcl-3.01.tbz
pkg_add mysqltcl-3.01.tbz

# Install P0f

pkg_add -r p0f

# Install Tcpflow

pkg_add -r tcpflow

Now I copy some configuration files and set up the Sguil database.

# Copy configuration files

cp /usr/local/src/$SGUIL/sensor/sensor_agent.conf /usr/local/etc/nsm 
cp /usr/local/src/$SGUIL/server/sguild.conf /usr/local/etc/nsm
cp /usr/local/etc/snort/snort.conf-sample /usr/local/etc/nsm/snort.conf
cp /usr/local/etc/barnyard.conf-sample /usr/local/etc/nsm/barnyard.conf
cp /usr/local/src/$SGUIL/sensor/sancp/sancp.conf /usr/local/etc/nsm
cp /usr/local/src/$SGUIL/sensor/log_packets.sh /usr/local/etc/nsm

# Set up database

/usr/local/bin/mysql -e "CREATE DATABASE sguildb"
/usr/local/bin/mysql -D sguildb < /usr/local/src/$SGUIL/server/sql_scripts/create_sguildb.sql
/usr/local/bin/mysql -e "GRANT ALL on sguildb.* to sguil@localhost"
/usr/local/bin/mysql -e "GRANT FILE on *.* to sguil@localhost"
/usr/local/bin/mysql -e "SET password for sguil@localhost=password('sguil')"
/usr/local/bin/mysql -e "SHOW TABLES" sguildb
/usr/local/bin/mysql -e "SET password for root@localhost=password('r00t')"
/usr/local/bin/mysql --password=r00t -e "FLUSH PRIVILEGES"

I couldn't think of an easy way to apply changes to the configuration files, so I edited them by hand to suit my needs and generated patches.

Here is my patch generation procedure for sensor_agent.conf.patch as an example.

First, make a copy that will contain the changes.

cp sensor_agent.conf sensor_agent.conf.diff

Now edit sensor_agent.conf.diff to include the desired changes. I use vi. Next create the patch.

diff -u sensor_agent.conf sensor_agent.conf.diff > sensor_agent.conf.patch

The sensor_agent.conf.patch looks like this:

--- sensor_agent.conf Wed Dec 28 14:57:30 2005
+++ sensor_agent.conf.diff Wed Dec 28 14:58:33 2005
@@ -13,7 +13,7 @@
 set DAEMON 0
 
 # Name of sguild server
-set SERVER_HOST 192.168.8.8
+set SERVER_HOST localhost
 # Port sguild listens on for sensor connects
 set SERVER_PORT 7736
 # Port sensor_agent lisens on for barnyard connects
@@ -22,10 +22,10 @@
 # Note: Sensors monitoring multiple interfaces need to use a unique 'hostname'
 #       for each interface. Make sure this name is the same in the respective
 #       log_packets.sh
-set HOSTNAME gateway
+set HOSTNAME gruden
 
 # The root of your log dir for data like pcap, portscans, sessions, etc
-set LOG_DIR /snort_data
+set LOG_DIR /nsm
 
 # Where to look for files created by modded spp_portscan
 set PORTSCAN_DIR ${LOG_DIR}/${HOSTNAME}/portscans
@@ -49,7 +49,7 @@
 # 2: sancp (http://www.metre.net/sancp.html)
 
 #Enable Stream4 keep_stats (1=enable 0=disable)
-set S4_KEEP_STATS 1
+set S4_KEEP_STATS 0
 # Where to look for ssn files created by modded spp_stream4
 set SSN_DIR ${LOG_DIR}/${HOSTNAME}/ssn_logs

I do not think this is a bad way to handle the issue, although I welcome simpler suggestions. If you wanted to use my script, for example, you could copy the patches, edit them, and then have the script apply them as shown below. Note this is a place where sensor name and IP address can matter. Note in the above patch the sensor name, gruden, is explicitly mentioned.

# Fetch text file patches

cd /usr/local/etc/nsm

fetch http://www.bejtlich.net/sensor_agent.conf.patch
fetch http://www.bejtlich.net/sguild.conf.patch
fetch http://www.bejtlich.net/snort.conf.patch
fetch http://www.bejtlich.net/barnyard.conf.patch
fetch http://www.bejtlich.net/sancp.conf.patch
fetch http://www.bejtlich.net/log_packets.sh.patch
fetch http://www.bejtlich.net/log_packets.sh.crontab

# Apply patches

patch -p0 < sensor_agent.conf.patch
patch -p0 < sguild.conf.patch
patch -p0 < snort.conf.patch
patch -p0 < barnyard.conf.patch
patch -p0 < sancp.conf.patch
patch -p0 < log_packets.sh.patch
crontab -u root log_packets.sh.crontab

Next I put log_packets.sh where it belongs, move some Snort configuration files, and retrieve some simple startup scripts.

# Install log_packets.sh

cp /usr/local/etc/nsm/log_packets.sh /usr/local/bin

# Copy Snort conf files

cp /nsm/rules/$SENSOR/classification.config /usr/local/etc/nsm
cp /nsm/rules/$SENSOR/gen-msg.map /usr/local/etc/nsm
cp /nsm/rules/$SENSOR/reference.config /usr/local/etc/nsm
cp /nsm/rules/$SENSOR/sid-msg.map /usr/local/etc/nsm
cp /nsm/rules/$SENSOR/threshold.conf /usr/local/etc/nsm
cp /nsm/rules/$SENSOR/unicode.map /usr/local/etc/nsm

# Get startup scripts

cd /home/sguil
fetch http://www.bejtlich.net/barnyard_start.sh
fetch http://www.bejtlich.net/sguild_start.sh
fetch http://www.bejtlich.net/sensor_agent_start.sh

chown sguil:sguil /home/sguil/*.sh
chmod +x /home/sguil/*.sh

cd /root
fetch http://www.bejtlich.net/snort_start.sh
fetch http://www.bejtlich.net/sancp_start.sh

chmod +x /root/*.sh

Now I modify /etc/rc.conf so MySQL will start at boot, but only listen on localhost. The sniffing interface on this system is lnc1, so I bring it up without the capability to arp.

# Modify /etc/rc.conf

echo "mysql_enable=YES" >> /etc/rc.conf
echo "mysql_args=--bind-address=127.0.0.1" >> /etc/rc.conf
echo "ifconfig_lnc1=-arp" >> /etc/rc.conf

Several of the Sguil components, like barnyard, sensor_agent.conf, and SANCP run as user sguil and need to write their PID files to /var/run. I decided to make /var/run mode 777 to let them write to the directory. This is not the best idea, so I might change it.

# Set up /var/run

chmod 777 /var/run

Finally I add the user 'sguil' with password 'sguil' so clients can access the Sguil server.

# Add Sguil client user

echo "Create a Sguil client user password when prompted."
cd /usr/local/src/$SGUIL/server
./sguild -c sguild.conf -u sguild.users -adduser sguil

In this last section I tell how to get all of the components running. By default all of them will run in the background. Each *start.sh script has an option for running in the foreground for debugging purposes, if you uncomment the foreground option and comment out the background option.

# Messages to users

echo "To start Sguil, execute the following."
echo 
echo "As user sguil:"
echo
echo "/home/sguil/sguild_start.sh"
echo "/home/sguil/sensor_agent_start.sh"
echo "/home/sguil/barnyard_start.sh"
echo
echo "Next, as user root:"
echo
echo "/root/sancp_start.sh"
echo "/root/snort_start.sh"
echo "/usr/local/bin/log_packets.sh restart"
echo
echo "You will then be able to connect using the separate Sguil client."

Once you have this script installed on a suitable FreeBSD 6/i386 system, you can run it. Here is the partition layout I created, using only 1024 MB. I installed the "minimal" distribution, which is the smallest non-custom distro.

$ uname -a
FreeBSD gruden.taosecurity.com 6.0-RELEASE FreeBSD 
6.0-RELEASE #0: Thu Nov  3 09:36:13 UTC 2005     
root@x64.samsco.home:/usr/obj/usr/src/sys/GENERIC  i386
$ df -h
Filesystem     Size    Used   Avail Capacity  Mounted on
/dev/ad0s1a    124M     55M     59M    48%    /
devfs          1.0K    1.0K      0B   100%    /dev
/dev/ad0s1g     62M     40K     57M     0%    /home
/dev/ad0s1f    124M    4.0K    114M     0%    /nsm
/dev/ad0s1h     62M     12K     57M     0%    /tmp
/dev/ad0s1d    248M    100M    128M    44%    /usr
/dev/ad0s1e    124M    206K    114M     0%    /var

I added two users.

• User analyst is a member of the wheel group and can therefore su - to root.


• User sguil is not a member of the wheel group. However, I run as many parts of Sguil as possible using this user.

Here is how to invoke the script:

$ su -
Password:
gruden# fetch http://www.bejtlich.net/sguil_install_v0.1.sh
sguil_install_v1-0.sh                         100% of 6023  B   83 kBps
gruden# chmod +x sguil_install_v0.1.sh 
gruden# ./sguil_install_v0.1.sh 

When the sguil client user password prompt appears, enter something like 'sguil'. This is the only pause in the script.

The end result of running this script inside a FreeBSD VM I created is a Sguil sensor, server, and database. I'll describe that in my next post.