Lab machine, updated RAID system (to our current shipping specs). We’ve got a 10GbE and an IB DDR card in there for some end user lab tests over the next 2 weeks. We just finished rebuilding the RAID unit, and I wanted a baseline measurement. So a fast write then read (uncached of course). [root@jr5-lab fio]# fio sw.fio ... Run status group 0 (all jobs): WRITE: io=195864MB, aggrb=3789.1MB/s, minb=3880.1MB/s, maxb=3880.1MB/s, mint=51680msec, maxt=51680msec Thats the write.

They see a shiny new storage chassis with 6G backplane. They fill it with “fast” drives, and build “raids” using integrated RAID platforms. They insist it should be fast, showing calculations that suggest that it should sustain near theoretical max performance on IO. Yet, the reality is that its 1/10th to 1/20th the theoretical max performance. Whats going on? In the past, I’ve railed against “IT clusters” … basically clusters designed, built, and operated by IT staff unfamiliar with how HPC systems worked.

A system designed to fail often will. Seen this a few times this past week. In one case, someone agrees that we we do and our machines have value, but want our stuff without paying us for our stuff. They don’t want to buy them. They just want us to tell them how to build them. They don’t want to buy our stuff, even though we’ve demonstrated that our systems solve their problem.

I wasn’t expecting this one. KQ Infotech, a smaller development house probably best known for their porting of ZFS to Linux, and providing the tools required for end users to build their own ZFS on their own machines (thus getting around some of the major hurdles with GPL and CDDL licenses). I was not expecting this, though to be honest, we’ve seen some pretty interesting M&A; bits over the last 2-4 weeks.

One of the two units we are going to ship to a customer very soon. Running the 19.2TB write. Fill up 1/2 the system. With a single file. Of 19.2TB size. In less than 2 hours. Don’t try this on ext*. [root@jr5-1 ~]# fio sw-19.2TB.fio ... Run status group 0 (all jobs): WRITE: io=19200GB, aggrb=3160.7MB/s, minb=3235.1MB/s, maxb=3235.1MB/s, mint=6221566msec, maxt=6221566msec [root@jr5-1 ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/md0 44G 5.

96TB Scalable Informatics JackRabbit JR5 unit, shipping out to a customer today (or early tomorrow). These are single thread, single process, single file writes. Taking it out to the track and cracking the throttle, wide open. [root@jr5-2 ~]# fio sw.fio ... Run status group 0 (all jobs): WRITE: io=65028MB, aggrb=3801.1MB/s, minb=3893.2MB/s, maxb=3893.2MB/s, mint=17104msec, maxt=17104msec [root@jr5-2 ~]# fio sr.fio ... Run status group 0 (all jobs): READ: io=65028MB, aggrb=3257.2MB/s, minb=3335.3MB/s, maxb=3335.3MB/s, mint=19965msec, maxt=19965msec and the 1TB run

Looks like Seagate got itself some spinpoints. Seagate may be leveraging this to build its way into the Chinese market more than it is. Now there are 3 big spinning rust makers: Seagate, Western Digital, and Toshiba. A Seagate-Toshiba hookup wouldn’t surprise me, though the regulators are likely to start eyeing this stuff more closely for anti-monopoly reasons. I’ve said more M&A; and I mean’t more M&A.; And the deals ain’t done yet.

At InsideHPC, Rich notes in a post Heh … I’d argue that the (sp)fork already happened, its in the past, and people have decided to continue moving forward with the new (sp)fork. This said, this is decidedly not a bad thing. As I had predicted, Oracle has largely abandoned all things HPC that it couldn’t remission for some other decidedly non-HPC purpose. The only realistic reason for retaining ownership of the Lustre IP/copyrights/etc.

So whatcha gonna do when you have a Lustre file system, with an ext4 backing store with a journal on an external RAID1 SSD, when that external RAID1 ssd pair goes away (in a non-recoverable manner), and the file system has the needs_recovery flag set? You see, the ‘-f’ option to e2fsck … doesn’t … in the face of a missing external journal with needs_recovery set. Ok, you can turn off the journal.

If you have anything approaching a complex installation or management requirement for your systems, most … no … pretty much all Linux distributions have anywhere between somewhat borked to completely boneheaded designs for handling these complex sitatuations. Say, for example, you want to boot a diskless NFS system, and replicate it. Diskless NFS is well known to be an easy to manage scenario … one system to manage, very scalable from an admin point of view.