Upgrading FreeBSD Packages with Portupgrade and without CVSup

freebsd.png" align=left>As much as I like FreeBSD's ports system, I dislike keeping the tree up-to-date using CVSup. I haven't yet tried Colin Percival's portsnap tool, which uses HTTP to distribute compressed and cryptographically signed snapshots of the ports tree. This is a great alternative, but I try to avoid compiling from source using the ports tree when possible. I run several very generic and slow systems, and I'd much rather install and upgrade software using precomplied packages.

Portupgrade is the tool of choice to update software on FreeBSD. Portupgrade is powerful because it can (far more often than not) resolve dependencies to upgrade software en masse. It doesn't matter if an application is installed via ports or precompiled packages. Once installed, FreeBSD treats it like a package using tools like pkg_info. When invoked with the -PP switch, portupgrade can be told to only upgrade installed software by using packages. This avoids having to compile upgraded software in order to install it.

Most people still say the ports tree must be updated to let portupgrade know the newest version of the port available. So, even when upgrading using packages, it seems a CVSup of the ports tree, or an equivalent process, is necessary. Or is it?

I recently started thinking about alternatives to updating the whole ports tree. There's been some discussions regarding the ports system on the freebsd-ports mailing list. This post announced the removal of the INDEX and INDEX-5 files from CVS. These files (INDEX for the 4.x tree and INDEX-5 for the 5.x tree) list information like the following excerpt for nmap:

nmap-3.75|/usr/ports/security/nmap|/usr/local|Port scanning utility for large
 networks|/usr/ports/security/nmap/pkg-descr|eik@FreeBSD.org|security ipv6|openssl-
 0.9.7e_1 pcre-5.0|openssl-0.9.7e_1 pcre-5.0|http://www.insecure.org/nmap/|||

This entry describes the nmap port, its maintainer, dependencies, and so on. You won't find any INDEX files in CVS at cvsweb.freebsd.org, for example. (Here's the entry in the CHANGES file also announcing the INDEX removals.) The maintainers thought that since the INDEX files kept in CVS were always out-of-date, it wasn't necessary to keep them there. INDEX files are available elsewhere, however, either by running 'make index' in /usr/ports or by downloading them. The latter option is going to help us, as the former takes a lot of time using the traditional methods currently available.

It turns out that you can completely avoid CVSup and upgrading the ports tree if you want to work strictly with precompiled packages. Here's how to do it on a FreeBSD 5.3 RELEASE system with several applications already installed via precompiled 5.3 RELEASE packages.

When I started I only had the INDEX-5 file that accompanied the ports collected installed with FreeBSD 5.3 RELEASE:

janney:/root# ls -al /usr/ports/IND*
-rw-r--r--  1 root  wheel  5798559 Nov  4 18:27 /usr/ports/INDEX-5

First I installed portupgrade:

janney:/root# pkg_add -vr portupgrade
looking up ftp.freebsd.org
connecting to ftp.freebsd.org:21
setting passive mode
opening data connection
initiating transfer
Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-5.3-release/Lates
t/portupgrade.tbz...x +CONTENTS
x +COMMENT
x +DESC
x +MTREE_DIRS
x man/man1/pkg_deinstall.1.gz
...truncated...

Once portupgrade was installed I ran portversion to see how portupgrade saw my installed packages. Running portversion also built the package database, /var/db/pkg/pkgdb.db.

janney:/root# portversion -v
[Updating the pkgdb  in /var/db/pkg ... - 48 packages found (
-0 +1) . done]
[Updating the portsdb  in /usr/ports ... - 11735 port entries
 found .........1000.........2000.........3000.........4000.........5000........
.6000.........7000.........8000.........9000.........10000.........11000....... 
..... done]
bash-3.0_5                  =  up-to-date with port
bitstream-vera-1.10         =  up-to-date with port
boxtools-0.65.0             =  up-to-date with port
cdrtools-2.0.3_4            =  up-to-date with port
cmdwatch-0.2.0              =  up-to-date with port
...truncated...

When the process completed I had a new INDEX.db file in /usr/ports, along with the original INDEX-5:

janney:/root# ls -al /usr/ports/IND*
-rw-r--r--  1 root  wheel   5798559 Nov  4 18:27 /usr/ports/INDEX-5
-rw-r--r--  1 root  wheel  11915264 Nov 19 23:16 /usr/ports/INDEX.db

Now I needed to update that old, original INDEX-5 file. As far as portversion was concerned, all of my installed packages were up-to-date. Unfortunately, this was not true. For example, a look at ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-5-stable/ shows that shells/bash-3.0.15.tbz is available. I'm using bash-3.0_5, so there's a newer precompiled package ready for me to upgrade.

The key to this process is twofold. First, obtain a new INDEX-5 file by running the following:

janney:/usr/ports# make fetchindex
INDEX-5                                         0% of 5877 kB    0  Bps

When 'make fetchindex' is done, the INDEX-5 is updated:

janney:/usr/ports# ls -al IND*
-rw-r--r--  1 root  wheel   6018417 Nov 19 23:05 INDEX-5
-rw-r--r--  1 root  wheel  11915264 Nov 19 23:16 INDEX.db

Second, force an update of the INDEX.db file:

janney:/usr/ports# portsdb -uf
[Updating the portsdb  in /usr/ports ... - 11972 port entries
 found .........1000.........2000.........3000.........4000.........5000........
.6000.........7000.........8000.........9000.........10000.........11000........
. ..... done]

Now look at INDEX.db:

janney:/usr/ports# ls -al IND*
-rw-r--r--  1 root  wheel   6018417 Nov 19 23:05 INDEX-5
-rw-r--r--  1 root  wheel  12334080 Nov 19 23:54 INDEX.db

Running portversion again shows several ports requiring updates. Only bash is visible below:

janney:/usr/ports# portversion -v
bash-3.0_5                  <  needs updating (port has 3.0.16_1)
bitstream-vera-1.10         =  up-to-date with port
boxtools-0.65.0             =  up-to-date with port
cdrtools-2.0.3_4            =  up-to-date with port
...truncated...

Now the INDEX-5 file indicates that the newest version of bash in the ports tree is 3.0.16_1, but the precompiled package is only 3.0.15. This is acceptable (as long as we are not addressing security issues), so we will update bash from 3.0_5 to 3.0.15.

The next step is to tell portupgrade where to find precompiled packages:

janney:/usr/ports# setenv PACKAGESITE ftp://ftp2.freebsd.org/pub/FreeBSD/ports/
 i386/packages-5-stable/All/

Finally we are ready to begin upgrading packages with portupgrade:

janney:/usr/ports# portupgrade -varRPP
--->  Session started at: Fri, 19 Nov 2004 23:55:03 -0500
** No need to upgrade 'python-2.3.4_2' (>= python-2.3.4_2). (specify -f to force)
...edited...
--->  Checking for the latest package of 'shells/bash'
--->  Fetching the package(s) for 'bash-3.0.16' (shells/bash)
--->  Fetching bash-3.0.16
++ Will try the following sites in the order named:
        ftp://ftp2.freebsd.org/pub/FreeBSD/ports/i386/packages-5-stable/
--->  Invoking a command: /usr/bin/fetch -o '/var/tmp/bash-3.0.16.tbz' 'ftp://ft
p2.freebsd.org/pub/FreeBSD/ports/i386/packages-5-stable/All/bash-3.0.16.tbz'
fetch: ftp://ftp2.freebsd.org/pub/FreeBSD/ports/i386/packages-5-stable/All/bash-
3.0.16.tbz: File unavailable (e.g., file not found, no access)
** The command returned a non-zero exit status: 1
** Failed to fetch ftp://ftp2.freebsd.org/pub/FreeBSD/ports/i386/packages-5-stab
le/All/bash-3.0.16.tbz
--->  Invoking a command: /usr/bin/fetch -o '/var/tmp/bash-3.0.16.tgz' 'ftp://ft
p2.freebsd.org/pub/FreeBSD/ports/i386/packages-5-stable/All/bash-3.0.16.tgz'
fetch: ftp://ftp2.freebsd.org/pub/FreeBSD/ports/i386/packages-5-stable/All/bash-
3.0.16.tgz: File unavailable (e.g., file not found, no access)
** The command returned a non-zero exit status: 1
** Failed to fetch ftp://ftp2.freebsd.org/pub/FreeBSD/ports/i386/packages-5-stab
le/All/bash-3.0.16.tgz
** Failed to fetch bash-3.0.16
--->  Listing the results (+:done / -:ignored / *:skipped / !:failed)
        ! bash-3.0.16   (fetch error)
--->  Packages processed: 0 done, 0 ignored, 0 skipped and 1 failed
--->  Fetching the latest package(s) for 'bash' (shells/bash)
--->  Fetching bash
++ Will try the following sites in the order named:
        ftp://ftp2.freebsd.org/pub/FreeBSD/ports/i386/packages-5-stable/
--->  Invoking a command: /usr/bin/fetch -o '/var/tmp/bash.tbz' 'ftp://ftp2.free
bsd.org/pub/FreeBSD/ports/i386/packages-5-stable/Latest/bash.tbz'
/var/tmp/bash.tbz                               0% of  532 kB
--->  Downloaded as bash.tbz
--->  Identifying the package /var/tmp/bash.tbz
--->  Saved as /usr/ports/packages/All/bash-3.0.15.tbz
--->  Skipping libiconv-1.9.2_1 (already installed)
--->  Skipping gettext-0.13.1_1 (already installed)
--->  Listing the results (+:done / -:ignored / *:skipped / !:failed)
        + bash@
        - libiconv-1.9.2_1
        - gettext-0.13.1_1
--->  Packages processed: 1 done, 2 ignored, 0 skipped and 0 failed
--->  Found a package of 'shells/bash': bash-3.0.15.tbz (bash-3.0.15)
--->  Located a package version 3.0.15 (bash-3.0.15.tbz)
--->  Using it anyway although it is not the latest version (3.0.16), since -PP/
--use-packages-only is specified
--->  Upgrade of shells/bash started at: Sat, 20 Nov 2004 00:00:52 -0500
--->  Upgrading 'bash-3.0_5' to 'bash-3.0.15' (shells/bash) using a package
--->  Updating dependency info
--->  Uninstallation of bash-3.0_5 started at: Sat, 20 Nov 2004 00:00:53 -0500
--->  Fixing up dependencies before creating a package
--->  Backing up the old version
--->  Uninstalling the old version
--->  Deinstalling 'bash-3.0_5'
[Updating the pkgdb  in /var/db/pkg ... - 47 packages found (
-1 +0) (...) done]
--->  Uninstallation of bash-3.0_5 ended at: Sat, 20 Nov 2004 00:01:01 -0500 (co
nsumed 00:00:07)
pkg_info: can't find package 'bash-3.0.15.tbz' installed or in a file!
--->  Installation of bash-3.0.15 started at: Sat, 20 Nov 2004 00:01:01 -0500
--->  Installing the new version via the package
--->  Removing temporary backup files
--->  Installation of bash-3.0.15 ended at: Sat, 20 Nov 2004 00:01:06 -0500 (con
sumed 00:00:04)
--->  Cleaning out obsolete shared libraries
[Updating the pkgdb  in /var/db/pkg ... - 48 packages found (
-0 +1) . done]
--->  Upgrade of shells/bash ended at: Sat, 20 Nov 2004 00:01:08 -0500 (consumed
 00:00:15)
** No need to upgrade 'xorg-fonts-miscbitmaps-6.7.0' (>= xorg-fonts-miscbitmaps-
6.7.0). (specify -f to force)
...truncated...

In included the whole listing here to show what happened when portupgrade couldn't find the version of bash it wanted, namely bash-3.0.16, at ftp://ftp2.freebsd.org/pub/FreeBSD/ports/i386/packages-5-stable/All. It knew to look for bash.tbz at ftp://ftp2.freebsd.org/pub/FreeBSD/ports/i386/packages-5-stable/Latest however. That location is a symlink to the bash package available:

lrwxr-xr-x  1 110  0  22 Nov 15 08:09 bash.tbz -> ../All/bash-3.0.15.tbz

To save some hassle, in the future I will specify the PACKAGESITE like this:

janney:/usr/ports# setenv PACKAGESITE ftp://ftp2.freebsd.org/pub/FreeBSD/ports/
 i386/packages-5-stable/All/

When done, all of the ports which could be updated via precomplied package, were indeed updated. For example, here is bash:

janney:/root# portversion -v bash
bash-3.0.15                 <  needs updating (port has 3.0.16_1)

Since precompiled packages are uploaded to the FTP servers weekly, I intend to repeat this process shortly after the next set of packages appears.

I also intend to keep an eye on Matthew Seaman's new portindex tool. This follows the absolutely bizarre removal of the original portindex from the ports tree in September. Matthew's portindex allows fast generation of INDEX and INDEX-5, replacing the slow 'make index' or 'portsb -uU' processes. The new portindex program, however, builds from an updated ports tree, which my package-centric process avoids.

For those wishing to use ports, however, the combination of portindex with portsnap will be powerful. Those wanting to stick with packages for slower or generic deployments might like the system explained here.