Thoughts on "Benchmark Comparison of NetBSD 2.0 and FreeBSD 5.3

Several people have asked me to comment on George McGarry's Benchmark Comparison of NetBSD 2.0 and FreeBSD 5.3. My initial reaction to this article is disappointment. I am not upset because the author says his "results indicate that NetBSD has surpassed FreeBSD in performance on nearly every benchmark and is poised to grab the title of the best operating system for the server environment." I am disappointed that the author has decided to use his "results" in a divisive manner. Rather than seek to learn from each BSD project and potentially compete better with Linux or Windows, George decides to drive a wedge between the NetBSD and FreeBSD communities.

Regarding the nature of the delivery of this report, George demonstrates he apparently doesn't know how to use email. Instead of posting his findings himself, he relies on Hubert Feyrer to post the abstract for the report in multiple newsgroups, like freebsd-advocacy, freebsd-performance, netbsd-advocacy, and NetBSD's tech-perform.

While I find Hubert's blog to be a good independent source of NetBSD information, I question why he is the delivery man. Why can't George post for himself, and answer replies publicly? If you're going to make strong claims, the author should be willing to stand up for his beliefs in person. None of these postings have been followed by responses from George or Hubert.

Regarding the "results" themselves, I found three responses interesting. First, Stheg Olloydson points out that George's test results do not support his conclusions. Stheg uses excerpts from George's paper to make his case.

Second, Robert Watson explains the technical decisions behind FreeBSD's architecture. Robert says:

"Regarding SMP or not -- the path the FreeBSD Project has taken (and this choice was before I was really all that involved, to be honest) was a re-architecture of the kernel to improve performance, scalability, and structure via a movement to a parallelizable, preemptible, threaded kernel. I think this is the right architecture to move to, as it not only improves performance and scalability, but it also closes a lot of existing race conditions in the kernel that only became more exposed as threading and SMP became more predominant. This has had a lot of performance benefits, but comes with initial costs that aren't all immediately offset by initial benefits. Now that this model is largely adopted, we'll see a nice increase in benefits over time -- i.e., it was an investment."

The third response comes from Kris Kennaway, who mentions the controversial nature of the fefe.de benchmarks. Kris also says:

"There's a leap from 'NetBSD performs better in microbenchmarks' to 'NetBSD is a better-performing server' (macrobenchmarks often do not reflect the same performance characteristics as microbenchmarks, although of course they are influenced by them)."

Slashdot is covering this story now, and there are a few helpful comments in the responses.

I personally use whatever operating system best suits the project at hand. For example, I prefer to use OpenBSD for firewalls and FreeBSD for general purpose servers. I have plans for a NetBSD system to become a terminal server, since it supports an obscure piece of hardware on that box and the machine offers only 32 MB RAM.

In conclusion, I would be much happier to see performance comparisons, especially between BSD versions, used to improve the collective performance of each variant. Instead, we see trolls here and here using these "results" to justify personal vendettas. I hope to see personal replies from George in the near future, or I will continue to be suspicious of his motives.