Oh whatta day: the fisking

Yesterday, I commented on a puff piece article on Windows CCS. Go ahead and read it, the article and the commentary. This morning, I saw a comment on this same article from John at InsideHpc.com. I disagreed with John’s premise, and wrote a long article discussing this. While I respect John, I do disagree with him. But I will do so respectfully. The rest of this article will be … sarcastic … flippant … and I am going to fisk the fisking post that was derived from John’s on another site. So gentle reader, if your stomach cannot take this, or you don’t like this stuff do skip this article.

This afternoon, I got a pingback from John’s blog to the puff piece commentary. In it the article is characterized as “anti-CCS”.

I disagree with the characterization of “anti-CCS”. The post was a detailed questioning of the costs and motivations. If they come out negative relative to the options, does that make the analysis “negative”? Are all cost-benefit analysis intrinsically negative to some choice then? I won’t link to the other blog. Reading it over, well, you will see what I mean. Needless to say, John and I must have significantly differing opinions of what the definition of “thoughtful critique” is. This individual (not John) has posted here before. And has tried similar tactics. “Thoughtful critique”? No. Not even close. But given our history, I guess I should expect this. Note that John has commented there as well.

This is called a Fisking . Following the wikipedia link

Now why on earth would I say that? From the “fisk”

on his blog he has an generic picture of da-UP (upper peninsula) next to his point. Witty? eh… doubtful. Ad hominem? Yeah. Poor form? Uh huh. First, full disclosure: My family owns IRAs which have stock in Microsoft. Second, my day job sells machines which can and do run Windows and windows applications in addition to Linux. I am very interested in real markets, and highly skeptical of fake ones. Third, I am a fan of Excel and Powerpoint. Word still grates on me every now and then, but for the most part I like it. If I had a chip on my shoulder, as he implies, I certainly wouldn’t do two or three. And I would sell off the IRA shares in one. He (Dan) works for a company that sells windows-only software to build distributed processing environments. These are perfect for desktop grids and grids of other systems. Being windows only he has a built in bias towards that platform, and, as you will see, against competitive ones. That said, lets move and and see some of the faulty logic, shall we?

And of course, since it is printed, on paper, and “independent”, it must be true. There aren’t any critiques of this or similar studies. Are there. (hint: more at the end of this post) [

](http://www.pcworld.com/article/id,118937-page,1/article.html) Um… uh… so you ask windows systems admins whether it has a better TCO than Linux, you write a report, and 3 years later someone hawks it as if it is fact? Ok….. There are no studies that say something else, cause that would knock the wind out of the sails here. Wouldn’t it. (see the bottom of the post as well)

Can’t seem to find that company, but there are more links. Oh, but this won’t do. This suggests the jury is out on this. There goes his first point. The TCO is known to be hard to know, and recent studies (as you will see in the links at the end) actually tend to show that Linux has a lower TCO.

Hey, we agree! It is a lousy idea to do this. Unless you have to. Lets ask the ultimate, and first windows cluster shop, the Cornell Theory Center, what it is they do.

For completeness, here is their link to linux software installed.

Yes. Look at their site. Might be out of date. I don’t know. But look at their site. Oh, I guess that is two points knocked down. Those darn facts … they keep getting in the way of a good fisking of Joe, don’t they. And, while I am at it, let me help with the math

Thats 18 copies of CCS at $469/copy. With 18 copies of antivirus/antispam at $83/copy. Adding it all together, yup, comes out to right about $10k. Ok, lets go commando, and run without protection. Lets just call it $8500 between friends, shall we? Back to antivirus on windows clusters. I asked this of a few Windows cluster folks. I got back “you don’t need it”. I asked them why, and never really got an answer. Now we have enterprise customers for whom it is mandated that firewalls go on and up, antivirus gets installed **before ** that windows machine is ever let on a network, and there are corporate edicts that include firing for removing or disabling such tools. Doesn’t matter, desktop or server. They are treated the same. I am not aware of many sites that don’t have similar policies, maybe some more or less draconian. I note with distress that the agency missioned with our defense, is under constant barrage of cyberattack. To be expected I guess, sort of a low level virtual war against us. These hacks are enabled and facilitated by poor security. Unfortunately, if you start out with a platform which is hard to secure, it often winds up that you have exposed yourself in some manner to the attackers. If our defense machines are crippled, will we be able to defend ourselves? In the corporate world, you have to worry about SOX. It is everywhere. Execs don’t want to pay the price for incorrect and bad corporate and fiduciary stewardship, so they are getting really serious about protecting assets. Draconian in some cases, but its their behind, as it were, on the line. You need to be secure. If you can’t tie down your platform whaddya do? Thats why firewalls, antivirus, antispam everywhere. The latest nefarious bits are using keyloggers to steal login passwords for single factor sign ons. Multifactor with a second factor being time based and limited lifetime is needed. Ok, whats up next …

Specious? At least one (and the flagship) windows computing customer, at that, believes otherwise. Anyone out there willing to run a windows machine without antivirus, antispam? Anyone? At all? As for letting clusters “peep” onto the internet, that isn’t needed. You just need one, precisely one, infected executable and you are toast. No network needed. How do you get such an infected executable? Well, some grad student or admin or whomever uploads from a protected/secure server. The exe could have been compromised already, or possibly the protected secure server has been cracked. Again, this is sadly a common anecdote of admins and security folks. From the above link:

But you know this, so why am I telling you. I dunno, maybe more of that “specious” reasoning. Ok, onto server shipments and marketshare. We have hashed this one before, with this person.

In the past I had asked him not to put words into my mouth. It seems my request has not been honored. He sets up this straw man to knock it down. Sadly. Ok, onto the market analysis. Linux-Watch from May 2007 reports

Sadly I don’t have the exact figures for that second consecutive quarter of double digit growth, but I can do a lowball estimate. Lets assume double digit growth is 10%, the lowest double digits we can do. Growing 10% per quarter means growing 40% CAGR. That ain’t bad. But I don’t think this is what is meant by it. Ok, lets assume that really, they didn’t mean second consecutive quarter of double digit growth but double digit growth relative to the previous year. Ok, this knocks that Linux way down to … uh … 10% growth. Oh. Thats still not bad. So how did Windows do?

10.4%? Darn it, thats higher than the 10% I posited above. IDC continues

Yup, indeed. 10.4% is indeed greater than 10%. I yield the point. For the first time since 1998, windows has grown faster than Linux. Point, Dan. Making the statements easier to digest:

1. the data indicates that Linux is growing faster than the market
2. the data indicates that Windows is growing faster than the market I had assumed the January data in writing this note, where indeed, Linux was growing faster than the market, and windows was, as I remember, treading water or growing slightly less fast. Call this an old or bad memory on my part. As indicated, the new data shows a whopping 0.4% growth rate advantage this quarter (first time in 9 years) to Windows. Yes Dan, you were right. So right, and I was wrong, so, so very wrong. Oh wait, thats sarcasm… But here we go again. Remember, I have had trouble with him before trying to put words into my mouth. He seems to like to set up a straw man, tell the world about it, then go knock it down. Now he says …

Of course, if he read my blog he would know that he is the only one saying this. I have been talking about how the small systems are driving the market for years. $25k and below and $50k and below. Our own sales data indicate this. So do many others. This is part of the reason I make arguments about the cost of accelerator technology. Anyone thinking that accelerators will be selling for $20k enmasse is fooling themselves. Accelerators are a disruptive technology. Highly disruptive. Expensive ones which are hard to program are going to fall by the wayside. From what I can see nVidia grasps this, and even if CUDA and the existing platform aren’t exactly where we need them to be, they are going to get there soon. And that means real, personal, supercomputing. This is the growth market I am interested in. I don’t see nVidia selling thousand cluster units. I see them selling millions of one/two off units. Same with Cell. A good Cell based machine will not run (hopefully) 10k$. But I digress. So Dan is trying to paint me as saying what he said, knowing full well that the blog history here indicates that I think the opposite of what he said. Well, such is life. And such are blogs. Facts be damned, full snarkiness ahead … or some such. Lets go on.

Not to manage. To use. It was just one example. Is there a one-liner for Powershell to do this?

Really? I am not aware that it is a faulty comparison. How is it faulty?

Ahhh…. so you want me to iterate over the nodes and gather data for you as compared to using the scheduler to gather that data for you. Remember, this was about getting what is running on the node. We could always do ssh node_name "ps -ealf; cat /proc/cpuinfo ; cat /proc/meminfo or something like that … But regardless, I want to know how demonstrating significant power in the palm of your hand as it were, is somehow faulty. Patrick at Microsoft pointed me over to this site, and I looked for a common task. What is running on a node. That said, I was specifically looking at how to get information out of windows cluster. It appears that it is not a simple process. The powershell assumes you know quite a bit of windows internals to do this. Yeah, I could write a similar thing for Linux, but why bother when you have tools that will do it for you now? Of course, one could point out that if the site used LSF, or SGE for windows (and I think they can), they ought to be able to get out “similar” information. But rather than point at the “easy” (but not invented here) solution, you have to use the windows internals. Ok, simple challenge. Suppose you wanted to report on your entire clusters usage of swap to its total memory as a number. This is useful to see if any of your nodes is thrashing under a load. How can you do this in windows? I think you need to use the above script and modify it some. [root@crunch-r ~]# cat swapratio.pl #!/usr/bin/perl while(<>){if($_=~/(Swap.*|Mem.*):\s+(\d+)/){$m{$1}=$2}}; printf "%s\n",($m{SwapTotal}-$m{SwapFree})/$m{MemTotal}; [root@crunch-r ~]# pdsh "cat /proc/meminfo | ~/swapratio.pl" itanic: 0 autoinst: 0 minicc: 0 dualcore: 0 Here I assumed that the script was moved to each node. Could have put it on one line, but that gets ugly. The point being though that you have lots of power. Huge amounts of power available to you. You have choice on the tools you might wish to use. Every choice has a cost. Note that with the right libraries, we could make that swapratio.pl code work on windows. Same as on Linux. Scary. This is what I had been hoping Microsoft was going to do. But they didn’t.

Actually Dan, I was comparing apples to apples here. Had they elected to use SGE/LSF tools, they could have had this as well. But they didn’t. This is their indicated script to do what we can easily do without it. You know, I often get similar comments from Java programmers when they tell me about the team they put together to solve the problem that a simple module from CPAN handles for you. You make your choices on the tools, and you get all the costs and benefits of the choices. These days, people have been finally waking up to the real costs of java systems, and are finally looking seriously at dynamic languages for their codes. About time. If you can do in one line, what it takes another code 100 lines, why on earth would you use the 100 line method? Same issue as above. Huge concentrated power, and the freedom to exploit it as you need. Eventually I expect windows ccs to get there as well, if Microsoft keeps at it.

The first part, powershell being available is a good thing. Windows has powerful scripting language tools. Ask ActiveState. Exposing the internals to scripting via language linkages, yes, this is a good thing. Again, ActiveState, etc. This is a point I have been making for a long time as well. A well designed cross platform scripting environment could go a long way towards alleviating real pain. I don’t think Powershell is it, but it is a first pass. Pdsh isn’t quite it either, that is more of a transport. I am not advocating bash or tcsh for windows. I think shell scripting should be kept small, and dynamic language scripting (Perl, Python, Ruby) should be used. In Perl, many of the modules are smart enough to “do the right thing” on windows. Some modules will not work on windows (and this frustrates me, I know of no equivalent to unix Sudo on windows that I can run from a command line). But thats a post for another day. Though this is a serious topic, deserving of serious consideration. Back to the “specious reasoning”

We don’t see CCS clusters. And that has been my point. They appear to cost more to acquire, and I haven’t seen data that contradicts this to date. Feel free to point it out for identically configured systems. On the other hand Dan implies that no one is managing many Linux servers, and therefore the TCO is higher. As noted, the jury is out on whether the TCO is a massive win or a massive loss. It appears to be a wash at first pass, though as you will see below, there are others who point out a strong win in Linux’s favor. And given that Linux is a growing segment, 12+ percent of the market, it is quite likely that in the TCO analysis, there is no problem or significant additional cost in managing Linux clusters. Because when you manage a cluster, in large part (well at least in Linux), you are managing the infrastructure nodes. The head node, file servers, and so forth. Compute nodes are disposable, you don’t manage per node, this does not scale. Management has the same effort for 1 or 10 or 100 for an administrator, modulo hardware costs and system design. That is your admin costs should not scale linearly with the number of compute nodes. Unless you are doing it wrong.

Good. Linux will slide right in there. Centrify and others pre-installed/configured to handle AD. Mono handling .NET. Linux experience is not that hard to acquire if needed. Lots of admins out there for you to choose from. Lots of MCSEs looking for ways to differentiate themselves from the pack, and show their flexibility and value. Many of them are already running Linux, or talking about it, so it won’t be a stretch.

12% in the server market, a little more than 1/9th the server market, is small. Ok. Very very small? No. Is it growing? Yes. Faster than the market as a whole. This of course covers shipped servers, and doesn’t deal with servers that have been converted. We have seen quite a few of these. Ok, back to the FUD fisk

This is of course, not true. We see two major thrusts: 1) Linux training in house to build the expertise, 2) outsourcing support. The cost to manage Linux for some of our customers is about $2-5k/year. This is with no Linux knowledgeable staff in house. For some of our mission critical customers, we provide first/second line services. Even on-site services. So they don’t have to. Saves them money, lets them train staff to handle first line. OTOH, adding additional responsibilities to windows admins, already overworked, doesn’t save money. Objective measures of TCO aren’t necessarily in windows favor, even with an existing large body of windows admins. Moreover [

](http://www-03.ibm.com/linux/whitepapers/robertFrancesGroupLinuxTCOAnalysis05.pdf) The study that reports that, was commissioned by IBM. Obviously IBM sells lots of Linux. And Windows. And some Solaris. And AIX. And … Yeah, its old too. 2 years. Like the Yankee Group study. There couldn’t be other studies that contradict the TCO bits you say, could there be? I mean, that would be wrong. We can keep throwing TCO studies at each other all day long. The recent studies I have pointed to have demonstrated that in some use cases, the TCO for Linux is much better. In others not so good. Couple that with the horizontal scaling, the virus resistance, and well … The facts remain in the end. Linux usage is growing, rapidly, faster than the growth of the server market as a whole. And that means that it can’t really be costing everyone more to do this, or they would not do this. That is, you can exclude some theories by observation of relevant facts. Fact is it its use is growing faster than the server market as a whole. Fact is that companies are very TCO/acquisition cost conscious. The only theories that fit that well are a TCO wash or a TCO advantage to Linux. I cannot fathom how a TCO advantage to windows would be able to explain the Linux uptake. I can imagine that windows growing faster than linux by that whopping 0.4% (estimated number, could be lower or higher, but not likely by much) is a transient phenomenon, or marketing dollars driven. Microsoft does pay marketing dollars to some vendors to help them get into market with their products. Most companies with lots of marketing money do. End users have to either request Linux on their servers, or wipe an existing windows server. I could come up with some guesstimates as to the latter, but I think it is simply best stated that the number of Linux servers indicated in the various reports is a fraction of the total number. Large or small fraction, who knows. That study has not been done yet. Same with desktops. Few vendors supply Linux on desktops. We do. This number is growing though. continuing

Well, I won’t argue with better/easier. Dan was making the easier argument a minute ago.

I disagree that this is there reasoning. HPC has been coming to the masses long before Microsoft got into the picture. It has been on this trajectory for 2 decades, and showing no real signs of slowing down. Linux is just the latest (now 8 year old) HPC systems contender. It has been growing for a long time outside of HPC, and very rapidly within HPC. It is having some interesting side effects, such as end users finally evaluating Linux as a desktop, and these evaluations coming out strongly positive. My thoughts are the Win xp x64 is the closest thing to a reasonable windows on a cluster node. It is (relatively) lightweight, and should be easy to set up. If we could just strip out the junk we don’t need it would be good. If we could only network boot it (diskless compute node), we would be happy. Patrick from Microsoft alluded to mixed windows/linux clusters in a previous post. We have been talking about them and working on building such things for a while now. Unfortunately xp 64 is going away when XP goes away, and all we will be able to work with is W2k3 CCS. But again, that is a point for another time. Back to the last bit

Specious? No. Well reasoned (my “puff piece” post)? Simple number crunching, and I didn’t see serious holes in my numbers. Opening a salvo with “poof piece” should set the tone for the rest of it. It was an attempt at a fisking, and not that good of one at that. Rather than argue facts and numbers (like I did), we saw hand waving, ad hominems and other weak argumentation. The straw men were particularly funny. I am sure he doesn’t have copyright usage allowances for the images, but I could be wrong. Copyright, the foundation of open source, is rather picky about these things. I have to admit being saddened to see John indicate his pleasure in seeing this fisking attempt. There was little in Dan’s article that was well thought out, and there was lots of hand waving, straw men, and word insertion. Oh well. Finally, I would like to note that I see two distinct camps emerging, with a few hearty skeptics like myself in the middle. First camp are what I would call Microsoft Fanboys (and girls). These are people for whom Microsoft can do no wrong, and everything else must be bad. Second camp are the Free software types. These are people for whom anything that is not shared is bad. There are very few of us (grizzled?) skeptics in the middle. The folks for whom hard numbers matter more than anything else. All I know is that when I point out problems with Microsoft and business models for CCS, we get these fisking attempts (and some emails from Microsoft). When I point out problems with open source side such as the business model issues, I get some harsh emails. Yeah, reality is tough. I remain highly skeptical of Microsoft’s CCS. I see a company trying to do all things: out Google Google, out ipod Apple, out PS3 Sony, out Linux clusters, out netscape netscape (oh yeah, they did that already), out java Sun. I like all the Microsoft people I have met, even if I don’t agree with them. There is a place for CCS. I remain unconvinced that the replacement paradigm will work, I think that will fall by the wayside. Patrick gave me some hope that Microsoft is seeing side by side as a better way, and I agree. Even better would be boot on demand, though we need diskless (iscsi or CIFS/SMB/NFS) booting to enable. We can do that today (I have an example worked out) but it is a VMware kludge using Linux booted diskless to fire up windows (and solaris for that matter). This will change. This is something I would like to see us work with them on. This would be useful.