We have a nice pair of Qlogic 7220 DDR HCAs in house. Direct connecting a pair of machines for a simple point to point bit.
Using our updated 184.108.40.206.scalable kernel.
Want to set up SRP target. So we have to get OFED compiled. Need 1.5.3+ due to their … er … issues tracking kernels.
Basically the OFED build process is an abuse … a very severe one … of the RPM process.
RPMs should build, correctly, by default with
rpmbuild -ba spec-file.spec
… but … OFED … doesn’t … quite …
If you read through the install.pl, you see all the special casing they do for various drivers/subsystems. That is, they haven’t really resynced the OFED stack and the kernel drivers in a while, so its possible to build a late model kernel, say something in the 2.6.3x (x >= 6) and get 1.4.2 era-ish drivers.
Which means, if you have new-ish hardware, like things that require the qib driver (Qlogic updated driver to replace ipath driver), you are SOL.
Yeah, we are sort of in that boat. So I hacked the install.pl to make sure that it actually builds the qib driver. And packages it.
Which it doesn’t do by default. Remember that broken RPM process? I’d honestly hold up this stack as exhibit “a” of what not to do with RPM.
So now I am trying to get that up and going, and sure enough the kernel infiniband bits (built as modules) do seem to interfere with the 220.127.116.11 build, so much so, that things like ib_send_bw don’t seem to work correctly. I think its a header conflict.
Honestly, getting close to turning off the kernel Infiniband build by default (not even as modules), and using OFED to avoid these pains. But the OFED build is itself extraordinarily painful. Because they haven’t quite synced everything yet, and they abuse RPM.
OFED is a useful stack. Its utility is marred by an incredibly painful build.
And I just can’t escape noticing that the 10GbE NICs that we build drivers for … yeah … well … installation with them is real darned easy. We automate the driver build/install in a consistent interface (like DKMS, but something that actually works). Makes rebuilds absolutely painless. And I can rebuild for pretty much any kernel, though we get occasional breakage from late model kernels. Seen this in Network and RAID adapters.
But darn it, if that aint easy.
Did I mention that the driver builds for the open drivers is drop dead easy?