[update] This was a PCI contention issue. Customers original code and test cases did not tickle this performance feature.. Their next code did. ConnectX was designed to handle codes of the latter type.
Its also quite dangerous to take as gospel any of the output of diagnostic programs without their context. And if you are a vendor, and you have a customer reporting things like this, have a careful look at what they are doing. There may be a signal there … just don’t get caught up in the number without a context. I allowed that to happen to me today without realizing it. We serve our customers best when we bring a critical mind and rigor to their issues. I did that with most of our customers today …
In short: The Mellanox SDR cards are fine. No issues. They work as intended.