Network Adapter Performance

We have used netpipe to test the performance of different Gigabit network cards. The Intel/MPIGAMMA combination has the lowest latency, outperforming proprietary RDMA cards. We use bidirectional messages rather than the standard one-way test, since this is more closely related to real-world applications, where nodes exchange data rather than just send a message from A to B. We compare different network cards, different software layers, and switches vs. crossover cables; details are given below. The one-way data rate is reported in MBytes/sec; the theoretical maximum for Gigabit Ethernet is 125. Note that the output file from netpipe reports the total message buffer size, twice the size of an individual message (reported below).

1. Dell Pe850: single P4D (3.0GHz dual-core) and Dell PE1750*: dual Xeon (2.8GHz)
2. OS: Centos 4.2 (Linux 2.6.9 and Linux 2.6.12-gamma) and NPACI Rocks (Linux 2.4)*
3. Switches: Crossover cable, Extreme Networks Summit400-t48, Cisco chassis switch (model 6509)*

1. The Broadcom 5721 (B'Com A) performs well when used with a recent tg3 driver (from Broadcom); the 3.10 driver in the 2.6.9 kernel is not as good. The Intel PRO 1000 (Intel A) has a high bandwidth but the large TCP driver latency reduces the performance for small messages.
2. The TCP performance of the Intel NIC (82545GM) can be improved by tuning the driver parameters. In case B we used InterruptThrottleRate=0 and TxIntDelay=128. The ITR=0 setting reduces the latency by a factor of 2; TID=128 allows for reduced cpu load with no measureable effect on performance. The effects of driver tuning may be more noticeable in practice than these limited results suggest. We notice that the throughput of the PRO 1000 with default TCP driver settings is quite erratic and these dropout may substantially reduce performance with large numbers of nodes.
3. Tuning the TCP buffer size brings increased performance. Generally, larger TCP buffers increase performance for large messages at the cost of a small performance penalty for smaller messages. We used 1Mbyte buffers which we found to be a good compromise. Increasing the frame size (MTU) has a similar effect, with the added benefit of a reduction in cpu usage; we used MTU=4120 (4096+24) in these tests.
4. MPIGAMMA is a port of MPI to the GAMMA Ethernet driver; it reduces the latency of the default Intel adapter by an order of magnitude, better than the expensive proprietary interfaces from Level 5 and Ammasso. It is only compatible with the 2.6.12 and 2.6.18 kernels at present. The hardware latencies of the Broadcom and Intel NIC's wired back to back, as measured by GAMMA are 14μs and 6.5μs respectively. There is an additional latency of 2.3μs from the MPIGAMMA software layer and 3.3μs from the Summit400t-48 switch. The Broadcom NIC failed to reach its asympotic bandwidth because we did not adjust the credit limit to account for the smaller packet size (1500 bytes); the Intel NICs used the optimum 4120 byte frame size.
5. With a low-latency network card the switch latency becomes critical and the Summit-t48 performs very well in this regard-the additional latency from the switch is only 3.3μs.
6. The switch reduces the maximum throughput under MPIGAMMA because the added latency requires a larger message size to saturate. GAMMA was configured to accept up to 32 4KB packets before sending an acknowledgement packet, and MPIGAMMA therefore takes a small (<10%) performance hit around 128KBytes; slightly better performance can be obtained at the cost of more memory allocated to GAMMA. In bidirectional mode MPI typically reaches a peak performance for messages of around 64KBytes, when it switches from "eager" to "rendezvous" protocols. LAM has a user interface (SSI) for choosing the transition message size; our results with LAM used the eager protocol throughout, which was found to deliver the maximum throughput.
7. The overall performance of a GAMMA-enabled Intel PRO NIC is remarkable; real-world applications can pass quite small bidirectional messages (32KBytes) at rates in excess of 100 MB/s each way.
8. The proprietary RDMA cards from Level5 and Ammasso (retail price around $500) used to outperform TCP/IP based NICS, but with newer TCP drivers the Intel NIC outperforms both proprietary cards, except the L5 at small packet sizes. The Intel+MPIGAMMA combination is faster than the proprietary cards.
9. The Ammasso and Level5 NICS substantially reduce CPU usage, which can increase performance if computation and communications are overlapped; GAMMA requires 100% CPU utilization and cannot profit from overlapping. The TCP driver (e1000-6.0.54-k2-NAPI) uses about 30% of the CPU; the tuned driver uses 40-50% CPU, depending on TID and MTU.
10. The Ammasso cards were tested on older hardware (2.8GHz Xeons) with a 2.4 kernel; the other tests were with 3GhZ P4D and the 2.6 kernel.

Sources and documentation

LAM v7.1: http://www.lam-mpi.org
MPI/GAMMA http://www.disi.unige.it/project/gamma/mpigamma

Acknowledgements: We thank Giuseppe Ciaccio for extensive help in setting up our MPIGAMMA installation and also for eliminating a number of obscure bugs. We thank the University of Florida High-Performance Computer Center (http://www.hpc.ufl.edu) for access to the Ammasso and Cisco hardware.