Semi-OT: analysis smackdown

If you haven’t seen the meme running around /. and other places, its that the BEST paper(s) “confirm” (note the scare quotes) AGW. The only problem with this is that its not true. They confirm that GW (climate change) is real (and I am not sure anyone disputes that). Its the “A” part that is the issue. Its been happening for billions of years. Takes some special sort of (not to mention massive amounts of) hubris to elevate a recent planetary occupant to a special status. Especially when the null hypothesis seems to fit the data so well. Go figure. But aside from that … there is some serious … I mean SERIOUS smackdown of a published analysis made by a statistician looking at the analysis done by the originator of the infamous (notorious?) hockey stick plot. The smackdown is great, in that this is the way smackdowns should be done. With humor, with irreverence. Also with facts. I must learn from the master … And yes, I’ve been guilty of the crime he indicates as well in other areas (usually performance monitoring). Will make sure I keep his comments in mind for our analytics. Performance data is very noisy time series data, and the “noise” is information, not to be lightly discarded or filtered. Possibly projected out (say in a Fourier type scenario) and preserved so that we retain the whole signal for analysis. This is actually something I am worrying about now, and it is directly relevant to monitoring in HPC. Your data has high and low frequency components. 10 second samples/averages might not make sense (even 1 second are too long for somethings, though the OS doesn’t seem to update fast enough for better data). This isn’t saying RRD is bad, just that access to the original full resolution data, which has been a goal of ours for a while, is probably in everyone’s interests.