Good programming tools and good program implementation

Way back in my early days at web programming stuff, I started out with HTML::Mason as a templating engine. There is nothing wrong with Mason, its actually quite good. But it encourages the same sort of “code-in-page” designs that the entire language of PHP was built around. I’m mostly a Perl guy for application level stuff these days … have done my time with Fortran, Python, x86 assembly, C/C++, and many others. I have my biases, and I understand them. I simply don’t have time to let a language get in the way of expressing what I want to do. So I wrote a web log analyzer a while ago. It was a Mason application, with the logic and the page intertwined. It was a mess (just like most PHP turns out). Very hard to understand, but it worked, and darn it, if I wasn’t using it for the last ~8 years or so. About 5 years ago, I started working with the Catalyst framework for web sites. And I tried rewriting this app in it. But it was, unfortunately, not easy. So I gave up and left it alone. With my recent EC2 foray, I thought about revisiting this code. I’ve since switched all of our development over to Mojolicious, which is very easy to write code for. And easy to separate out the view and controller functions (PHP and Mason encourage a fused view and controller). Mojolicious is better in that its very easy to write and deploy … its installation dependency radius is minimal (e.g. how much extra crap you have to install to make it work). Catalyst is huge in this regard, and Jifty, another framework I’ve liked in the past is Ginormous. There are some other similar to Mojolicious frameworks out there, like Dancer. But Dancer (and Catalyst) all use Moose, and Moose has a fairly huge dependency radius (and it is slow).

No, no language wars here. I am in no danger of becoming Pythonic. I’ll foray into that to help out with some coding, but as I noted, I use what maps well into my brain (and Python ain’t it). I like being able to write a service like this:

<code>
# Automatically enables "strict", "warnings" and Perl 5.10 features
  use Mojolicious::Lite;
  # Route with placeholder
  get '/:foo' => sub {
    my $self = shift;
    my $foo  = $self->param('foo');
    $self->render(text => "Hello from $foo.");
  };
  # Start the Mojolicious command system
  app->start;
</code>

Back to the topic. My code was a mess, it was more of a proof of concept … er … pressed into service. So after almost a decade, yeah, its about time for an update. So I started the rewrite in Mojolicious. I pulled the logic portions out into separate controller methods (as above). Template was massively simplified. Ran it and … its about the same speed. Darn. I wanted to make it go faster. Lets see if I’ve learned anything over the last 8 years or so. Looked at the code and found some … er … stuff … that should never ever see the light of day . So I fixed it. No need to create a cache and store/retrieve values I will never use. So I tossed that. And darn it, the way it was operating, I generated calls for specific inputs, which were repeated … couldn’t I just cache those, locally, and speed some things up? So I did. Try again. Order of magnitude faster. Used far less ram. Hmmm…. Changed formatting a bit on the output to include timing information to generate/serve my table.

Timing
measure	                        time (s)
file_read	                0.003
generate_data_for_display	0.009
get_records	                0.034
parse_columns	                0.138
N(lines) = 2559

Parsing/displaying 2600 lines takes (ignoring the data transfer and display formatting time), about 1/4 second. This is more than 1 OOM faster. When the log file grow bigger next week, will try again, and see how much faster. Nice to revisit old … er … code … and fix it every now and then. I do wish that apache did a better job of outputting machine readable format, so I could spend less time/effort/code parsing, and more time on the analysis. Yes, I know its configurable. No, I don’t always have options on what I can configure when I deploy these tools on servers (ones where I don’t have access needed). So I gotta work with what I have.