Thought a few people might like to see this. A new JackRabbit (JR4) being built. We are testing one of its RAIDs. Dealing with some MB issues, but otherwise, back onto stable ground. This is a single RAID card with 8 drives, 7 in a RAID6 + 1 hot spare. Large sequential streaming read and write. 4x larger than RAM in the machine. Caching is not relevant to this. No tuning. At all. Not even load balancing IRQs.
`root@jr4s:~# !dd
dd if=/data/big.file of=/dev/null bs=16M count=1k ...
1024+0 records in
1024+0 records out
17179869184 bytes (17 GB) copied, 30.7842 s, 558 MB/s
root@jr4s:~# dd if=/dev/zero of=/data/big.file bs=16M ...
1024+0 records in
1024+0 records out
17179869184 bytes (17 GB) copied, 27.7003 s, 620 MB/s
`
More to the point, we had to specifically de-tune this motherboard, and the Linux kernel a bit in order to get stability. What we are seeing is remarkable, even if these numbers don’t seem stellar on the surface. With RAID6, you lose 2 drives to parity. This is just the breaks, part of the cost you pay for more resiliency. So of the 7 drives in the RAID6, 5 are (effectively) for data. Now look at those sustained data rates again. Reads off of 5 platters, with ECC going on (RAID6) at 558 MB/s or 111.6 MB/s sustained. Now look at the writes. Writes to the 5 platters, with ECC going on (RAID6) at 620 MB/s or 124 MB/s sustained. Now, for a gedanken experiment, imagine a server with, I dunno, 24 of these drives, in 2 groups of 12. 1 drive in each group for hot spare, 11 in a RAID6. Assuming nearly linear scaling (its a gedanken experiment, please do allow me that lattitude), we should expect writes at about … oh 2.2 GB/s sustained, and reads at oh, about 2.0 GB/s. This is with the box and kernel specifically detuned to increase stability. Now imagine, if you will, a box with 48 of these drives. With 3 groups of 16. 1 drive each for hot spare, 15 for RAID6. Hmmmm …. On a related note, we will be taking both of these boxes to the test track, and opening the throttle. Wide.