Updating FreeBSD Using CVSup through HTTP Proxy

If you've used CVS before, you know that CVS doesn't play well with HTTP proxies. I was looking for a way to run cvsup on FreeBSD behind a proxy when I found a post on the FreeBSD China mailing list. It described using Proxychains with Desproxy to tunnel CVS over a SOCKS proxy through HTTP.

Here's how I followed the instructions in my lab environment.

First I installed Proxychains from the FreeBSD port. You can see my HTTP proxy is 172.16.2.1 port 3128.

freebsd7# setenv HTTP_PROXY http://172.16.2.1:3128
freebsd7# pkg_add -vr proxychains
...edited...
extract: Package name is proxychains-3.1
extract: CWD to /usr/local
extract: /usr/local/bin/proxychains
extract: /usr/local/bin/proxyresolv
extract: /usr/local/etc/proxychains.conf
extract: /usr/local/lib/libproxychains.so.3
extract: /usr/local/lib/libproxychains.so
extract: /usr/local/lib/libproxychains.la
extract: /usr/local/lib/libproxychains.a
extract: execute '/sbin/ldconfig -m /usr/local/lib'
extract: CWD to .
Running mtree for proxychains-3.1..
mtree -U -f +MTREE_DIRS -d -e -p /usr/local >/dev/null
Attempting to record package into /var/db/pkg/proxychains-3.1..
Package proxychains-3.1 registered in /var/db/pkg/proxychains-3.1

Next I installed Desproxy from source.

freebsd7# mkdir /usr/local/src
freebsd7# fetch http://downloads.sourceforge.net/project/desproxy/desproxy/desproxy-0.1.0-
 pre3/desproxy-0.1.0-pre3.tar.gz
desproxy-0.1.0-pre3.tar.gz                    100% of   51 kB   96 kBps
freebsd7# mkdir /usr/local/desproxy
freebsd7# tar -xzvf desproxy-0.1.0-pre3.tar.gz 
x desproxy-0.1.0-pre3/
x desproxy-0.1.0-pre3/Makefile.in
x desproxy-0.1.0-pre3/AUTHORS
x desproxy-0.1.0-pre3/Changes
x desproxy-0.1.0-pre3/config.h.in
x desproxy-0.1.0-pre3/configure
x desproxy-0.1.0-pre3/configure.in
x desproxy-0.1.0-pre3/install-sh
x desproxy-0.1.0-pre3/doc/
x desproxy-0.1.0-pre3/doc/config-en.html
x desproxy-0.1.0-pre3/doc/manual-en.html
x desproxy-0.1.0-pre3/src/
x desproxy-0.1.0-pre3/src/Makefile.in
x desproxy-0.1.0-pre3/src/desproxy-dns.c
x desproxy-0.1.0-pre3/src/desproxy-inetd.c
x desproxy-0.1.0-pre3/src/util.c
x desproxy-0.1.0-pre3/src/desproxy.c
x desproxy-0.1.0-pre3/src/desproxy.h
x desproxy-0.1.0-pre3/src/socket2socket.c
x desproxy-0.1.0-pre3/src/util.h
x desproxy-0.1.0-pre3/src/desproxy-socksserver.c
x desproxy-0.1.0-pre3/INSTALL
x desproxy-0.1.0-pre3/COPYING
freebsd7# cd desproxy-0.1.0-pre3
freebsd7# ./configure --prefix=/usr/local/desproxy
checking for gcc... gcc
checking for C compiler default output... a.out
...edited...
freebsd7# make install
Using binary dir: /usr/local/desproxy/bin
Using locale dir: /usr/local/desproxy/share/locale
Making directories...
Copying binaries...
desproxy installed
desproxy-inetd installed
desproxy-dns installed
desproxy-socksserver installed
socket2socket installed

*************************************
* This version lacks locale support *
*     locales won't be installed    *
*************************************

*******************
* Installation OK *
*******************

Before I could start desproxy-socksserver, I needed to edit my Squid proxy configuration. Here's where it gets tricky. If I can control my proxy, can't I figure another way around it? Stay with me for now. I made two changes. I added a variable for port 5999 for CVS:

acl CVS_ports port 5999

Next I added port 5999 as a "safe port":

acl Safe_ports port 5999        # CVS added by RMB 20 Aug 09

Finally I modified what ports were allowed the CONNECT method. By default only 443 is allowed.

# Deny CONNECT to other than CVS ports
http_access allow CONNECT CVS_ports
http_access deny CONNECT !SSL_ports

I thought you might be able to get CVSup to point to port 443 using the -p option, but if you try that you get an error:

Reserved port 443 not permitted

I wonder if this could be removed from the source code?

Assuming you could get CVSup to talk port 443, you could have it point to a host on the Internet under your control. That host could listen on port 443, then forward what it receives to the CVS server using Netcat. I think this would work. I found the following in cvsup-snap-16.1h/client/src/Main.m3

PROCEDURE CheckPort(port: INTEGER) =
  BEGIN
    IF port = IP.NullPort
    OR NOT (FIRST(IP.Port) <= port AND port <= LAST(IP.Port)) THEN
      ErrMsg.Fatal("Invalid port " & Fmt.Int(port));
    END;
    IF port < 1024 THEN
      ErrMsg.Fatal("Reserved port " & Fmt.Int(port) & " not permitted");
    END;
  END CheckPort;

I think if I removed the second check, for "Reserved port", that would remove my problem. To make things easier I just changed 1024 to 10.

To install the modified cvsup-without-gui, I did a 'make fetch' and 'make extract', modified the source, and then did 'make install'.

On a remote host I can run the following using the redir app:

# redir --laddr=[MY_BOUNCE_BOX] --lport=443 --caddr=[CVS server IP] --cport=5999

Then I set the IP in my supfile to be MY_BOUNCE_BOX.

If you can set up this sort of redirection, you can remove the proxy changes outlined earlier.

In the following case, let's assume you can make the necessary proxy changes so you don't have to bounce through a remote host listening for port 443.

Now start desproxy-socksserver. Basically port 1080 is listening on the localhost, and it will forward what it receives to port 3128 (Squid) on the proxy server.

freebsd7# /usr/local/desproxy/bin/desproxy-socksserver 172.16.2.1 3128 1080

-----------------------------------
desproxy-socksserver     0.1.0-pre3

(C) 2003 Miguelanxo Otero Salgueiro
-----------------------------------

TCP port 1080 Bound & Listening
Press  to Quit

Now configure Proxychains. Here is my configuration file.

freebsd7# grep -v \# proxychains.conf

random_chain

chain_len = 1

tcp_read_time_out 15000
tcp_connect_time_out 8000

[ProxyList]
socks5 127.0.0.1 1080

There is another problem here. If you can't resolve DNS inside your environment, this will fail. There is a 'proxy_dns' option in Proxychains, but I got an error when using it:

|DNS-response|: freebsd7.localdomain is not exist
|DNS-request| cvsup7.FreeBSD.org 
|R-chain|-<>-172.16.134.128:1080-<><>-4.2.2.2:53-<--timeout
|DNS-response|: cvsup7.FreeBSD.org is not exist
Unknown host "cvsup7.FreeBSD.org"

One way around this is to replace cvsup7.FreeBSD.org, or whatever you want to use, with the IP address of the CVS server. Start desproxy-socksserver.

freebsd7# /usr/local/desproxy/bin/desproxy-socksserver 172.16.2.1 3128 1080

-----------------------------------
desproxy-socksserver     0.1.0-pre3

(C) 2003 Miguelanxo Otero Salgueiro
-----------------------------------

TCP port 1080 Bound & Listening
Press  to Quit

Connection request from 172.16.134.128, port 63950
Connection #0: bidirectional connection stablished

Connection request from 172.16.134.128, port 53812
Connection #1: bidirectional connection stablished

Connection #0: end of connection
Connection #1: end of connection

Start proxychains and tell it to run cvsup:

freebsd7# proxychains cvsup -g -L 2 /usr/local/etc/freebsd7-example.supfile
ProxyChains-3.1 (http://proxychains.sf.net)
Parsing supfile "/usr/local/etc/freebsd7-example.supfile"
Connecting to cvsup4.FreeBSD.org
|R-chain|-<>-172.16.134.128:1080-<><>-204.152.184.73:5999-<><>-OK
|R-chain|-<>-172.16.134.128:1080-<><>-204.152.184.73:5999-<><>-OK
Connected to cvsup4.FreeBSD.org
Rejected by server: Access limit exceeded; try again later
Will retry at 17:45:22

That's not cool. That is actually a CVS server error. Let's try new CVSup host in the supfile.

freebsd7# proxychains cvsup -g -L 2 /usr/local/etc/freebsd7-example.supfile
ProxyChains-3.1 (http://proxychains.sf.net)
Parsing supfile "/usr/local/etc/freebsd7-example.supfile"
Connecting to cvsup7.FreeBSD.org
|R-chain|-<>-172.16.134.128:1080-<><>-64.215.216.140:5999-<><>-OK
|R-chain|-<>-172.16.134.128:1080-<><>-64.215.216.140:5999-<><>-OK
Connected to cvsup7.FreeBSD.org
Server software version: SNAP_16_1h
Negotiating file attribute support
Exchanging collection information
Establishing multiplexed-mode data connection
Running
Updating collection src-all/cvs

So, it worked. I would be interested in knowing if anyone has other methods to get CVSup to work through a HTTP proxy? Most people seem to use SSH tunnels, but what if that is not an option?

Update:

It turns out you do NOT need to use desproxy-socksserver. For example, use the following proxychains.conf:

freebsd7# grep -v \# /usr/local/etc/proxychains.conf

random_chain

chain_len = 1

proxy_dns 

tcp_read_time_out 15000
tcp_connect_time_out 10000

[ProxyList]
http 172.16.2.1 3128

Notice the use of 'http' here instead of 'socks5'. Also, the IP address here is the IP address of the Squid proxy server, whereas the earlier examples pointed to the desproxy-socksserver.

Again I bounce off an Internet host that will send traffic sent to port 443 (to get through the default CONNECT and port settings on Squid):

# redir --laddr=[MY_BOUNCE_BOX] --lport=443 --caddr=[CVS server IP] --cport=5999

Then you can run proxychains by itself.

freebsd7# proxychains cvsup -p 443 -g -L 2 /usr/local/etc/freebsd7-example.supfile
ProxyChains-3.1 (http://proxychains.sf.net)
Parsing supfile "/usr/local/etc/freebsd7-example.supfile"
|DNS-response|: freebsd7.localdomain is not exist
Connecting to MY_BOUNCE_BOX
|R-chain|-<>-172.16.2.1:3128-<><>-MY_BOUNCE_BOX:443-<><>-OK
|R-chain|-<>-172.16.2.1:3128-<><>-MY_BOUNCE_BOX:443-<><>-OK
Connected to MY_BOUNCE_BOX
Server software version: SNAP_16_1h
Negotiating file attribute support
Exchanging collection information
Establishing multiplexed-mode data connection
Running
Updating collection src-all/cvs
 Checkout src/COPYRIGHT
 Checkout src/LOCKS
...truncated...

If you look at a few packets you can see the setup of the connection.

14:48:04.461432 IP 172.16.134.128.65097 > 172.16.2.1.3128: P 1:39(38) ack 1 win 65535
 0x0000:  4500 004e 0ec9 4000 4006 4b3f ac10 8680  E..N..@.@.K?....
 0x0010:  ac10 0201 fe49 0c38 f690 a51d 2952 c059  .....I.8....)R.Y
 0x0020:  5018 ffff 3942 0000 434f 4e4e 4543 5420  P...9B..CONNECT.
 0x0030:  0000 0000 0000 0000 0000 0000 003a 3434  MY_BOUNCE_BOX:44
 0x0040:  3320 4854 5450 2f31 2e30 0d0a 0d0a       3.HTTP/1.0....
14:48:04.461991 IP 172.16.2.1.3128 > 172.16.134.128.65097: . ack 39 win 64240
 0x0000:  4500 0028 d1a2 0000 8006 888b ac10 0201  E..(............
 0x0010:  ac10 8680 0c38 fe49 2952 c059 f690 a543  .....8.I)R.Y...C
 0x0020:  5010 faf0 443f 0000                      P...D?..
14:48:04.535724 IP 172.16.2.1.3128 > 172.16.134.128.65097: P 1:40(39) ack 39 win 64240
 0x0000:  4500 004f d1a3 0000 8006 8863 ac10 0201  E..O.......c....
 0x0010:  ac10 8680 0c38 fe49 2952 c059 f690 a543  .....8.I)R.Y...C
 0x0020:  5018 faf0 3a58 0000 4854 5450 2f31 2e30  P...:X..HTTP/1.0
 0x0030:  2032 3030 2043 6f6e 6e65 6374 696f 6e20  .200.Connection.
 0x0040:  6573 7461 626c 6973 6865 640d 0a0d 0a    established....
...edited...
14:48:04.639664 IP 172.16.134.128.65097 > 172.16.2.1.3128: . ack 40 win 65535
 0x0000:  4500 0028 0ece 4000 4006 4b60 ac10 8680  E..(..@.@.K`....
 0x0010:  ac10 0201 fe49 0c38 f690 a543 2952 c080  .....I.8...C)R..
 0x0020:  5010 ffff 3f09 0000 0000 0000 0000       P...?.........
14:48:04.837943 IP 172.16.2.1.3128 > 172.16.134.128.65097: P 40:78(38) ack 39 win 64240
 0x0000:  4500 004e d1a7 0000 8006 8860 ac10 0201  E..N.......`....
 0x0010:  ac10 8680 0c38 fe49 2952 c080 f690 a543  .....8.I)R.....C
 0x0020:  5018 faf0 6a4f 0000 4f4b 2031 3720 3020  P...jO..OK.17.0.
 0x0030:  534e 4150 5f31 365f 3168 2043 5653 7570  SNAP_16_1h.CVSup
 0x0040:  2073 6572 7665 7220 7265 6164 790a       .server.ready.
14:48:04.839209 IP 172.16.134.128.65097 > 172.16.2.1.3128: P 39:61(22) ack 78 win 65535
 0x0000:  4500 003e 0ecf 4000 4006 4b49 ac10 8680  E..>..@.@.KI....
 0x0010:  ac10 0201 fe49 0c38 f690 a543 2952 c0a6  .....I.8...C)R..
 0x0020:  5018 ffff 4731 0000 5052 4f54 4f20 3137  P...G1..PROTO.17
 0x0030:  2030 2053 4e41 505f 3136 5f31 680a       .0.SNAP_16_1h.

So, you can tunnel CVS through HTTP using proxychains, as long as you bounce off a remote host that listens on port 443. That assumes you have to get around proxy restrictions that only allow CONNECT to port 443.