Perfect storms

The term Perfect Storm represents a coincidence (temporal or spatial near simultaneity) of events that cause a much larger effect than any one of the events normally would on its own. Perfect storms are in some ways, a superposition of events. Every now and then you get to see one in action. Like now.

I won’t describe current economic times, or what I think are the causative effects. Just what I observe. Peoples problems have not diminished in time, they have grown more acute. This has happened in an era of rapidly diminishing budgets. Basically people have more data to store, more models to compute with. They have less money to spend on computing and storage. They have to make do with less. Due to larger data sets, and more media rich applications, performance has gone from nice-to-have to critical-to-have. Due to a need for more responsive computing, end users are putting more power on their desktops. The perfect storms I see brewing are the cost reduction coupled with all these elements. While the high end of HPC is in decline in real $ and in relative % of the HPC market (which is growing rapidly), the ‘low end’ has been pushing hard. When you have 8 to 16 cores in your desktop, it is hard to call it a desktop machine. It is now a deskside super for most people … the majority of the HPC market. They get 8-16 processors and lots of ram for far less than $10k. Same thing on the cluster side. 16 nodes can house 16x 16 cores. 256 core cluster in a single rack, with everything you need. This is pretty serious firepower. Moreover, if you work with companies like Tsunamic Technologies, you can lease a portion of their cluster rather than buy your own, further reducing the cost of your infrastructure. No, they didn’t pay us to say this. Yes they are a customer of ours, using our storage units to provide high performance computing file/disk service. The combination of needing to do more while spending less is what is creating these storms. The scalability of costs in clusters are driving cost sensitive customers to choose lower priced components, or reduce their system size. Every optional/unneeded cost will be stripped out. Similar things are seen in storage, where it is getting harder and harder to justify more than $1/GB when you can replicate infrastructure for little more than this, and get better and more reliable systems as a result. We see this in the massive growth of personal supercomputing. This is something I personally have been talking about for most of the last decade. We are seeing others adopt this term as well now. We are seeing demand for our many core systems with lots of memory rise. We are seeing demand and interest in accelerator technology on the rise, curiously better and faster than we had predicted in our models during previous attempts to secure VC funding to build these things. What I guess isn’t that surprising (to me anyway) is that most of the new systems we see going in are Linux based. There is something to that line “from desktop/laptop to supercomputer”, running the same code, just moving where you run it on (even that you don’t have to do as it turns out, but that is another discussion). These items, coupled with clear cost choices for clusters, for storage, for personal supercomputing, with clear barrier reduction to supercomputing access is providing a superposition of events that might be ascribed the moniker of “perfect storms”. I left a customer yesterday after training them on how to use their cluster/systems. Lots of windows vista on laptops. Some of it simply didn’t work. They had an idle windows 64 bit machine with 8 Xeon cores in it. No one could use it. They wanted it integrated into their cluster. But they want to run Linux on it most of the time. The argument that management made for keeping it windows (and the initial acquisition in terms of it being a windows box), was “everyone knows windows” which mimics the arguments that Microsoft makes. The arguments the engineers made was “but it doesn’t do what we want, and the other stuff does”. At some point in time either the engineers win, or the company suffers from not being able to do what they need. Management ceded the point. They get a cost reduction and a capability increase for doing so. This was not lost on them. I suspect it is not being lost on a rapidly growing number of groups and companies. Pay less, get more. It sucks somewhat if you are a vendor on the wrong side of this.