strace -p is your friend

So there I was, trying to use a serial port on a node which was connected to a serial port on a switch. Which I needed to properly configure the switch. So I light up minicom and get garbage. Great, a baud rate mismatch, easily fixed. Fix it. Connect again. I get the first 10-12 characters … and then garbage. Hmmm. I’d like to pause our story for a moment, and say I had the key insight at this moment … but that would not be true. Like a true bonehead, I had a hammer, and it looked like a nail. Back to the story. Ok, do this a few more times. It looks like the baud rate kept changing. But this is silly, as I had locked the port, and minicom is one of those things that just works. Another pause. Minicom was giving me a hint when it lit up saying “hey, this port is locked with a stale lock, let me fix that for you.” Back to the story. A few more times. Then pull down the later minicom (older rev … obviously it was a software bug … or worse … a hardware bug … ugh). [editorial] Obviously not pilot error. No. Couldn’t be that. Not at all. [/editorial] Finally, after an hour of serious WTF, I am starting to question my sanity. Its time to pull out the heavy machinery. And watch the system calls as they go by. Start up minicom. Attach strace to it in another window. Press enter … and …. 1st: I see the port set right. 2nd: I see characters starting to flow … and … then … 3rd: Whiskey Tango Foxtrot … the serial port changed speeds midstream. Warning: Incoming reality 2x4 headed our way! Something else is messing with the port … what could it be? [brain engages] What on earth could be engaging a serial port? At 11pm? And going into polling mode …. oh crap. [sigh /] I’ve got a getty running on the port. Must have. And it recycles, and polls. Look in the requisite location and … yuppers, there it is. [sounds of mad vim typing, and some maniacal laughter later] getty is off. Try again. And there is the switch. So I log in, set up what I need to. For all its warts (speed, heavy penalty in operation), strace is a really good diagnostic tool. And BBBBRRRRRRAAAAAIIIIINNNNSSSS …. must engage them before wasting an hour on things I shouldn’t have wasted an hour on. Ok, it was 10pm, and I was tired, but still. Todays lesson: strace -p is your friend. And the handy helpful warning message popping up on your terminal …

… yeah … don’t just ignore it. Pay attention.