Be on the lookout for 'pauses' in CentOS/RHEL 6.5 on Sandy Bridge

Probably on Ivy Bridge as well. Short version. The pauses that plagued Nehalem and Westmere are baaaack. In RHEL/CentOS 6.5 anyway. A customer just ran into one. We helped diagnose/work around this a few years ago when a hedge fund customer ran into this … then a post-production shop … then … Basically the problem came in from the C-states. The deeper the sleep state, in some instances, the processor would not come out of it, or get stuck in the lower levels. This would manifest as a stutter … a momentary transient pause that was not easily reproducible. In the truest definition of the word, it was a Heisenbug. We could make the problems go away by reducing the space of C-state transitions available to the processors. And tell the processors to be less active about idling. We needed a little more this time, so we had to add the requisite kernel boot parameters to tweak idle and cstate, as well as have this code running in the background on startup.

#!/usr/bin/perl
my $lat = shift;
$lat = ($lat < 0 ? 0 : $lat);
$lat = ($lat > 250 ? 250 : $lat);
printf "Setting CPU latency to %i to control C state\n",$lat;
open(my $fh,">/dev/cpu_dma_latency") or die "FATAL ERROR: unable to set C state\n";
syswrite $fh,$lat or die "FATAL ERROR: unable to set C state\n";
while (1) { sleep 60 ; } # loop forever, as the file needs to remain open to force the C-state correctly.

and run it in the background

nohup /opt/scalable/bin/set_cpu_lat.pl 0 > /var/log/c-state 2>&1 &

Then we needed to make sure this was correct in terms of the processor states, so we lit up powertop. If you run the command w/o options, you can get instantaneous snapshots at a moment in time. And it shows all C0’s when we are done. … though … If you run it with the –csv option, and look at the idle report, you can see the impact of your changes … or non-impact. This is where the previous post about tuned came from. Assume that somewhere in the system is an

alias pure_evil tuned

or something like that This is the obligatory Time Bandits reference

No seriously, tuned … just say no. Customer will test with this now. But this echos very much like the previous problem on the nehalem/westmere platforms. As I remember, it had a component that was silicon based in the original problem (an issue with a timer in the IOH/PCH or something) coupled with buggy software. Given that it isn’t in 6.4, I am just gonna call this a software bug and move on.