IPMI Console Logger is born

Here’s the problem I am trying to solve, call it a many year itch I’ve been wanting to scratch. We build very high performance storage clusters, extreme performance flash and ssd arrays, and a number of other things. At customer sites, while in use, a unit could crash. When it does, we really need a full console log to see the full crash log. Unfortunately, the “write to the screen” method gets very … very old when you are trying to transcribe something … thats happened to scroll off the screen. And that mechanism doesn’t scale. Doesn’t enable automation. “But wait” you say. “What about the remote syslog facilities built into syslog and rsyslog?” They work fine as long as

• The machine is booted to an OS, and
• The network is up and operational on the machine, and
• The crash in question doesn’t take down, oh, say …. THE NETWORK …

And more to the point, what if we could basically stream this stuff to local logs, and, I dunno, trigger actions on patterns appearing in the logs? Assume I’ve got everything but the pattern matching up and going. And that isn’t that hard to add. Assume we can log many … many machines to a single centralized machine, as long as console output is redirected over the serial port, and that the serial port is available over the LAN in IPMI. This isn’t always the case, and some motherboard vendors like breaking useful stuff (not naming names (&&%$^$& , but you know who you are) like this. And assume grub sometimes fails completely to start with some serial console specs on that same vendors motherboards … GRRRRRRRR …. Apart from these annoyances, this enables us to create not simply a virtual serial concentrator, but a very intelligent one that can trigger alerts, send messages, etc. Even if the machine is non-functional for syslog/rsyslog network logging. So, I’d like to see if there is interest in this out there. All Scalable Informatics customers will get this pushed to them in new loads and updates. But if there is interest outside of our customer base, we’d be open to GPLing the code. It is Perl based, should be portable, and insensitive as to the state of the machine. Just need an IPMI BMC to connect to. We will be adding features and testing it against vendors BMCs to create a HCL. Will add in pattern matching engines, and alternative data storage backends (just a file at the moment, but we can easily use a database, or similar). Ping me at the day job if you are interested in playing with it and testing it. Paid support will be available for those needing priority service.