Massive, unapologetic, firepower: 2TB write in 73 seconds

By joe

May 19, 2014 - 4 minutes read - 771 words

A 1.2PB single mount point Scalable Informatics Unison system, running an MPI job (io-bm) that just dumps data as fast as the little Infiniband FDR network will allow. Our test case. Write 2TB (2x overall system memory) to disk, across 48 procs. No SSDs in the primary storage. This is just spinning rust, in a single rack. This is performance pr0n, though safe for work.

usn-01:/mnt/fhgfs/test # df -H /mnt/fhgfs/
Filesystem      Size  Used Avail Use% Mounted on
fhgfs_nodev     1.2P  895M  1.2P   1% /mnt/fhgfs
usn-01:/mnt/fhgfs/test # /opt/openmpi/1.8.1/bin/mpirun --allow-run-as-root --hostfile hostfile  -np 48 ./io-bm.exe -n 2048 -b 48  -w -f /mnt/fhgfs/test/files
...
Thread=00004: host=usn-01 time = 72.055 s IO bandwidth = 606.351 MB/s
Thread=00032: host=usn-06 time = 72.008 s IO bandwidth = 606.748 MB/s
Thread=00025: host=usn-05 time = 72.118 s IO bandwidth = 605.818 MB/s
Thread=00030: host=usn-06 time = 72.200 s IO bandwidth = 605.134 MB/s
Thread=00014: host=usn-03 time = 72.242 s IO bandwidth = 604.782 MB/s
Thread=00003: host=usn-01 time = 72.291 s IO bandwidth = 604.371 MB/s
Thread=00013: host=usn-03 time = 72.328 s IO bandwidth = 604.062 MB/s
Thread=00027: host=usn-05 time = 72.333 s IO bandwidth = 604.021 MB/s
Thread=00033: host=usn-06 time = 72.391 s IO bandwidth = 603.541 MB/s
Thread=00045: host=usn-08 time = 72.299 s IO bandwidth = 604.303 MB/s
Thread=00040: host=usn-07 time = 72.377 s IO bandwidth = 603.656 MB/s
Thread=00000: host=usn-01 time = 72.432 s IO bandwidth = 603.195 MB/s
Thread=00015: host=usn-03 time = 72.513 s IO bandwidth = 602.521 MB/s
Thread=00031: host=usn-06 time = 72.518 s IO bandwidth = 602.482 MB/s
Thread=00022: host=usn-04 time = 72.520 s IO bandwidth = 602.464 MB/s
Thread=00026: host=usn-05 time = 72.526 s IO bandwidth = 602.413 MB/s
Thread=00041: host=usn-07 time = 72.535 s IO bandwidth = 602.338 MB/s
Thread=00028: host=usn-05 time = 72.505 s IO bandwidth = 602.585 MB/s
Thread=00034: host=usn-06 time = 72.565 s IO bandwidth = 602.094 MB/s
Thread=00005: host=usn-01 time = 72.580 s IO bandwidth = 601.970 MB/s
Thread=00044: host=usn-08 time = 72.572 s IO bandwidth = 602.035 MB/s
Thread=00001: host=usn-01 time = 72.632 s IO bandwidth = 601.535 MB/s
Thread=00021: host=usn-04 time = 72.643 s IO bandwidth = 601.447 MB/s
Thread=00011: host=usn-02 time = 72.723 s IO bandwidth = 600.782 MB/s
Thread=00008: host=usn-02 time = 72.723 s IO bandwidth = 600.785 MB/s
Thread=00009: host=usn-02 time = 72.728 s IO bandwidth = 600.739 MB/s
Thread=00007: host=usn-02 time = 72.752 s IO bandwidth = 600.542 MB/s
Thread=00019: host=usn-04 time = 72.770 s IO bandwidth = 600.391 MB/s
Thread=00024: host=usn-05 time = 72.752 s IO bandwidth = 600.539 MB/s
Thread=00002: host=usn-01 time = 72.797 s IO bandwidth = 600.174 MB/s
Thread=00035: host=usn-06 time = 72.791 s IO bandwidth = 600.218 MB/s
Thread=00017: host=usn-03 time = 72.786 s IO bandwidth = 600.264 MB/s
Thread=00016: host=usn-03 time = 72.802 s IO bandwidth = 600.127 MB/s
Thread=00043: host=usn-08 time = 72.764 s IO bandwidth = 600.441 MB/s
Thread=00012: host=usn-03 time = 72.815 s IO bandwidth = 600.020 MB/s
Thread=00020: host=usn-04 time = 72.813 s IO bandwidth = 600.039 MB/s
Thread=00039: host=usn-07 time = 72.839 s IO bandwidth = 599.829 MB/s
Thread=00010: host=usn-02 time = 72.840 s IO bandwidth = 599.820 MB/s
Thread=00042: host=usn-08 time = 72.816 s IO bandwidth = 600.014 MB/s
Thread=00037: host=usn-07 time = 72.856 s IO bandwidth = 599.686 MB/s
Thread=00036: host=usn-07 time = 72.927 s IO bandwidth = 599.104 MB/s
Thread=00018: host=usn-04 time = 72.927 s IO bandwidth = 599.100 MB/s
Thread=00038: host=usn-07 time = 72.943 s IO bandwidth = 598.972 MB/s
Thread=00023: host=usn-04 time = 72.956 s IO bandwidth = 598.866 MB/s
Thread=00029: host=usn-05 time = 72.979 s IO bandwidth = 598.674 MB/s
Thread=00006: host=usn-02 time = 72.997 s IO bandwidth = 598.528 MB/s
Thread=00046: host=usn-08 time = 72.965 s IO bandwidth = 598.786 MB/s
Thread=00047: host=usn-08 time = 73.043 s IO bandwidth = 598.149 MB/s
Naive linear bandwidth summation = 28874.455 MB/s
More precise calculation of Bandwidth = 28711.174 MB/s

and the files

usn-01:~ # df -H /mnt/fhgfs/
Filesystem      Size  Used Avail Use% Mounted on
fhgfs_nodev     1.2P  2.2T  1.2P   1% /mnt/fhgfs
usn-01:/mnt/fhgfs/test # du -m files*
43691	files.0
43691	files.1
43691	files.10
43691	files.11
43691	files.12
43691	files.13
43691	files.14
43691	files.15
43691	files.16
43691	files.17
43691	files.18
43691	files.19
43691	files.2
43691	files.20
43691	files.21
43691	files.22
43691	files.23
43691	files.24
43691	files.25
43691	files.26
43691	files.27
43691	files.28
43691	files.29
43691	files.3
43691	files.30
43691	files.31
43691	files.32
43691	files.33
43691	files.34
43691	files.35
43691	files.36
43691	files.37
43691	files.38
43691	files.39
43691	files.4
43691	files.40
43691	files.41
43691	files.42
43691	files.43
43691	files.44
43691	files.45
43691	files.46
43691	files.47
43691	files.5
43691	files.6
43691	files.7
43691	files.8
43691	files.9

This is just a single rack