Fixed the IPoIB performance issue

For our Unison Parallel File Systems Appliance:

[root@unison-jr4-2 ~]# iperf -c 10.3.1.1
------------------------------------------------------------
Client connecting to 10.3.1.1, TCP port 5001
TCP window size: 1.00 MByte (default)
------------------------------------------------------------
[  3] local 10.3.1.2 port 48383 connected with 10.3.1.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  13.5 GBytes  11.6 Gbits/sec

and of course in parallel

[root@unison-jr4-2 ~]# iperf -c 10.3.1.1 -P2
------------------------------------------------------------
Client connecting to 10.3.1.1, TCP port 5001
TCP window size: 1.00 MByte (default)
------------------------------------------------------------
[  3] local 10.3.1.2 port 48404 connected with 10.3.1.1 port 5001
[  4] local 10.3.1.2 port 48405 connected with 10.3.1.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  13.2 GBytes  11.3 Gbits/sec
[  4]  0.0-10.0 sec  9.24 GBytes  7.94 Gbits/sec
[SUM]  0.0-10.0 sec  22.4 GBytes  19.3 Gbits/sec

and of course, using the verbs libs …

root@unison-jr4-2 ~]# rdma_bw -b 10.3.1.1
10969: | port=18515 | ib_port=1 | size=65536 | tx_depth=100 | sl=0 | iters=1000 | duplex=1 | cma=0 |
10969: Local address:  LID 0x01, QPN 0x00a0, PSN 0xdb732 RKey 0x011800 VAddr 0x007f750b0aa000
10969: Remote address: LID 0x02, QPN 0x00a2, PSN 0xde3926, RKey 0x011800 VAddr 0x007fc1768fa000
Warning: measured timestamp frequency 2600.37 differs from nominal 3000 MHz
10969: Bandwidth peak (#0 to #999): 9223.25 MB/sec
10969: Bandwidth average: 9223.07 MB/sec
10969: Service Demand peak (#0 to #999): 275 cycles/KB
10969: Service Demand Avg  : 275 cycles/KB

and

[root@unison-jr4-2 ~]# rdma_lat 10.3.1.1
   local address: LID 0x01 QPN 0x00a2 PSN 0xbc89f4 RKey 0x011a00 VAddr 0x0000000060a001
  remote address: LID 0x02 QPN 0x00a4 PSN 0x582833 RKey 0x011a00 VAddr 0x0000000060a001
Warning: measured timestamp frequency 2600.12 differs from nominal 3000 MHz
Latency typical: 0.649969 usec
Latency best   : 0.641123 usec
Latency worst  : 2.56584 usec

Quite nice! Should have some nice benchies on Monday/Tuesday.