SC13 observations

From a post to the beowulf list:

I didn’t get a chance to see many booths … I did get free the last hour of Thursday to wander, and made sure I got to see a few people and companies. What I observed (and please feel free to challenge/contradict/offer alternative interpretations/your own views) will definitely be colored by the glasses we wear, and the market we are in.

1. not so many chip companies (new processor designs, etc.) there. I am not sure if this is an overall trend, but in general they do not appear to be getting VC backing much any more.

2. Compiler companies … virtually non-existent (I look at NAG more in a tools and consulting view though they have a nice Fortran compiler). Did I miss any? [EDIT: Yes, PathScale was there, and I hope we can link to a set of slides for them] Compilers were there, no doubt. Intel, PGI, … all there. Was PathScale there? Others? I think this is part of a larger trend though … not that pure compiler companies aren’t viable, but that tighter integration to get the support the hardware vendors need has been in the offing.
3. very few pure cluster plays. Almost none. Few have actually survived, never mind thrived.
4. many government connected folks around on the show floor. A number of financial folks.
5. The problems people are seeking to solve are quite varied, but almost all of them are centered around scaling up computing performance in the face of very large, often unstructured data. This may be biased by our own focus.
6. Big data, which I liken to be applied HPC for a number of industries was on most customers minds. How to build/compute with massive data sets (sort of the more general case of 5). Everyone knows this word Hadoop. Very few folks quite understand that its one of many tools to handle data at scale. What’s profoundly interesting to me is that all of the issues that the “big data” world face are similar issues to what the HPC and more specifically the beowulf community have faced. How to scale computation. How to administer and run large resources. How to design the resources with scaling in mind. That is, we as a community have much to offer the growing big data community. This is evidenced in part by Doug Eadline’s work on the limulus box, and its easy transition as a private Hadoop box. Hadoop is all about key-value stores and distributed application of mapping functions to extract data in parallel. This is not a long leap from a large distributed Open-MPI application with core algorithms that need to communicate while computing. Though in most of the mapping cases, there is very little need for interprocess communication, and the entire system is designed to be tolerant of failure. Indeed, one can start using MPI within Hadoop systems to provide that interprocess communication for map reduction for non trivial (EP-like) processing. I find this fascinating.
7. The need for very high performance storage is dramatically increasing. We talked with many people on this. We showed a 30GB/s 4U box (c.f. https://twitter.com/sijoe/status/403313639999815681/photo/1), and made a mistake of not leaving the speedometers up (they were coupled live to the machine underneath). This sparked many conversations.
8. The need for very high density storage, multiple PB/rack is dramatically increasing. Many folks we spoke with have a capacity and performance bottleneck, and while they don’t need the most extreme performance, putting PB behind single or dual filer heads makes for a very non-scalable solution (their words, not mine). Most everyone now gets that the way to scale out is to add processing and network bandwidth as you scale capacity. Arrays and SANs are definitely rapidly on the way out (from the conversations we’ve had). Parallel and distributed file systems are on the way in.
9. Parallel and distributed file systems: Lots of folks talked Lustre, but many more this year wanted to talk Ceph and Fraunhofer. We had just published a Ceph benchmark in financial services a few weeks earlier, so this was serendipitous for us.
10. The tightly-coupled or “converged” message (I liken the latter to more marketing than real substance) where one puts massive fire power computing and networking right next to the big data pipes … is rapidly emerging in this ultra dense storage and big data/high performance computing view. People are telling us (!!!) data motion is hard, and they want to localize computation and data as much as possible. This is inclusive of the big data folks. This is wonderful IMO (and it reflects my companies biases, which I freely admit and embrace).
11. Talent acquisition is hard. Finding good people is very hard. I’ve had some interesting conversations on this with a few people privately. This is part of what is driving the aqui-hire trend … find companies which know what they are doing, and make em an offer they can’t refuse. HPC people, with solid computing, architecture, programming skills are in high demand. Maybe not in the broader geographical market, but certainly in a number of specific geos.
12. SWAG quantity is down, less bauble-ish, more functional. The IU hats rocked … it was snowing, they were warm. LSU had scarves (which sadly I did not grab one). As did NVidia. We gave out flashlights for listening to our partners talks, pens, coffee, biscotti, etc. Usually you hear about some very oddball SWAG, that you need to seek out, but I didn’t hear that this year.
13. Size: show floor was smaller. The impact of fewer US Government people and limited booth space was obvious. Number of attendees on show floor appeared to be lower. This said, we had more traffic at our booth than ever.
14. Quality of talks/BOFs: I’ve heard from many sources that the talks and BOFs were great. I miss having time to attend them, but will push for this next year. The admin BOFs seem to be strongly in demand.
15. Beobash rocked. I did not get to play pool. I did spend time talking with many folks, and drinking beer. Lara and team did a bang up job putting this together. As usual, it was the best party at the show. On a personal note, I am grateful to everyone who took the time to talk with me, or even wave hi (even if they could not stop to talk). They kept me chained in the booth (no not really, but I couldn’t wander far). And they had strict instructions to make sure I circulated and did not get locked into the deeper conversations that I like. I thought it was a good show, nothing revolutionary, quite a bit of evolutionary things. Fewer new faces, many older (yeah, I know, speak for myself). This was a significant show for us as a company, as we had our first “larger” booth. Pics here: https://twitter.com/sijoe/status/402845383547564032/photo/1/large https://twitter.com/sijoe/status/403315326000959488/photo/1/large https://scontent-b-ord.xx.fbcdn.net/hphotos-ash3/q71/s720x720/994043_10151702139320064_2096289564_n.jpg I’ll put up more photos soon.