Cluster 10GbE: still in the future

John West at InsideHPC asks about 10 GbE on clusters. The point I made (in two posts), and we verify every time we spec a system out for a customer, is that 10 GbE is still priced higher per port than IB. This doesn’t mean we don’t like 10GbE. On the contrary, it is simpler/easier to deal with. But it comes at a price penalty, and a non-trivial one at that.

Again, as we noted many times, I would love to be wrong about this. Sadly, i am not. A 24 port 10GbE switch would run into the $400/port price range. A 24 port DDR IB switch would run into the $167/port price range. Like it or not the latter is less expensive than the former. All other costs being about the same (they are not, as 10GbE also requires some sort of SFP/XFP type transciever), this comes in at a price disadvantage to 10 GbE. Again, we would love to be wrong. Really. I like 10 GbE. It is simpler. But for 128 nodes, the $233/port cost differences add up awful fast. Is the simplicity of the stack worth the price difference? I don’t think end users care about the simplicity of the stack. They care about whether it works, and what it costs for that work. Which is why there is a issue. Again, we would love to be wrong on this. So if you know of a good, low price 10 GbE switch, similar price to the DDR IB switch, by all means, please … let me know.