Worth asking again ... does Lustre have a future?

This is going to sound like a strange question to ask. Yes … I know it is a strange question to ask given the events of the past few months. A long while ago, I postulated that Lustre’s future was (no pun intended) cloudy at best. That Sun/Oracle had an uncertain level of commitment to it, and Larry Ellison is a business man, and doesn’t run a charity … there aren’t any freebees he is likely to fund forever. Rich at InsideHPC posted what he had heard. In a nutshell

Obviously, formal confirmation of a rumor is difficult. However, if you are on the Lustre mail groups, you can’t be missing the fact that quite a few Oracle (formerly Sun) staffers, have, very recently, different email addresses. So while we won’t likely get formal confirmation, it does appear, at least on the surface, that there is more than a little element of truth to this rumor. Which immediately begs the question … what is the future of Lustre? One might think it bright and strong, as we have no less than 3 organizations with vested interest in its future. Right? Well … er … no. Allow me to take a contrarian view. One I really don’t see shared in public. But I’ve had multiple conversations with various members of the community, and all have very similar reservations as mine.

First and foremost … there are 3 … 3 meta organizations that now have proclaimed support for Lustre. Which means, quite possibly, 3 different agendas, 3 different directions. 3 different roadmaps. Anyone wanna guess how many forks are going to come out of this? Second, there are at least 2 companies providing Lustre support: Whamcloud and Xyratec/Clusterstor. They have to focus upon their customer needs. And they may wish to add unique value to Lustre or build some sort of differentiation into/atop/along side it. Which adds to the roadmap issue. Basically, the point of this is, we just went from a rudderless ship with one organization leading, and a community around it … to no single leader, and a community that could fragment and fracture. Specifically if OpenSFS extends Lustre in a way that HPCFS and the European group don’t agree with. Or vice versa. That is, we didn’t land on our feet on this, we are still tumbling. The best of all possible worlds would be for these meta organizations to fuse. Single vision, single roadmap, single code base. Without that, I am anticipating at least 3 Lustre variants. The problem is, that the meta organizations are so wildly different in organization, scope, etc. that merging may not only be very hard, it might simply not be something politically feasible. I am not convinced this is good for Lustre users, developers, etc. But … this is the path we are on now. We might hear some nice words about the cooperation between groups. That will vanish fast and hard as soon as someone adds a feature the others don’t want. Not trying to be alarmist here … but this really isn’t going the way it needs to. Moreover, as noted previously, there are viable competitors for Lustre, and some Lustre users have noticed this. Remember … Lustre wasn’t always with us … it had to get the imprimatur from a number of places to be considered. I hate quoting movies, but this one seems apropos for a technology that while its maturing rapidly, may not be with us the way we need it to be, thanks to a fractured community (might not seem it now, but with 3 different organizations with 3 different sets of goals, needs, commitments …) Paraphrasing for HPC, “On a long enough timeline. The survival rate for every user base drops to zero” The only way to get sustainability, is to merge, and have everyone working toward the same goals. We ain’t there, and I am not sure we will ever be. Which is why I think we’ve gone from the frying pan into the fire on this. How many versions of Lustre do you think we will have by the end of 2011, 2012, 2013? I am guessing at least 2 by the end of 2011.