Ok, this isn't terrible ...

NFS over RDMA between a pair of JackRabbit systems with a Mellanox Connect-X card and a direct cable. We are getting about 500 MB/s +/- some on reads, and about 300 MB/s +/- a bit on writes. Not great, but actually in line with what is observed here. See the NFS over RDMA plots near the middle. This is about 1/3 of the available disk bandwidth on reads, and a little less than a 1/3 on writes.

I had expected better, though it is obvious that the disks and the machine can supply more. Looking to get some additional parts to test for units like this. Oddly in other experiences, we had have no trouble getting wire speed out of 10 gigabit ports on a single box, for things that were not in cache.