This section is still being developed as the usage on my cluster evolves, but so far we tend to write our own sets of message passing routines to communicate between processes on different machines.
Many applications, particularly in the computational genomics areas, are massively and trivially parallelisable, meaning that perfect distribution can be achieved by spreading tasks equally across the machines (for example, when analysing a whole genome using a single gene technique, each processor can work on one gene at a time independent of all the other processors).
So far we have not found the need to use a professional queueing system, but obviously that is highly dependent on the type of applications you wish to run.
For the single most important program we run (our ab initio protein folding simulation program), using the Pentium 3 1 GHz processor machine as a reference frame, the Athlon 1.2 GHz processor machine is about 16% faster on average, the Pentium 4 1.7 GHz machine is about 25-32% faster on average, and the Athlon 1.5 GHz processor is about 80% faster on average (yes, the Athlon 1.5 GHz is faster than the Xeon 1.7 GHz since the Xeon executes only six instructions per clock (IPC) whereas the Athlon executes nine IPC (you do the math!)).
These machines are incredibly stable both in terms of hardware and software once they have been debugged (usually some in a new batch of machines have hardware problems). Reboots have generally occurred when a circuit breaker is tripped. The first machine I installed has been up since its birth!
~ ram@fp1 % uptime
4:49am up 374 days, 2:47, 1 user, load average: 2.08, 2.02, 2.01