zaargy

Oh how I cheered when I read Tom Blomfield’s Automate Everything post! I’ve lost count of the number of times I’ve been subjected to some Byzantine business process and thought that surely there must be a better way?! What’s weird is how many people don’t feel this way. These are people who’ll do the same thing a hundred times and just put up with it like it makes perfect sense to them to do the same thing a hundred times! The thought that there could be a better way doesn’t enter their head.

It seems to me that these are people who interact with the world in a fundamentally different way from the way that I do. That doesn’t make them wrong – just different (and maybe a little sadomasochistic).

I have often thought it would be cool to have some putative automater person whose sole job is go round and improve all these sort of silly processes and do things like making the coffee machine report outages to IRC (and other such little things that bring joy and happiness) but somehow I doubt this idea will catch on any time soon.

Logging in Linux is something that’s always bothered me. I hate this stuff:

james@lucinda:/var/log$ ls -l dmesg*
-rw-r----- 1 root adm 45822 Feb  4 08:41 dmesg
-rw-r----- 1 root adm 45822 Jan 30 00:58 dmesg.0
-rw-r----- 1 root adm 12264 Jan 30 00:28 dmesg.1.gz
-rw-r----- 1 root adm 12266 Jan 30 00:19 dmesg.2.gz
-rw-r----- 1 root adm 12271 Jan 29 20:02 dmesg.3.gz
-rw-r----- 1 root adm 12130 Jan 29 19:48 dmesg.4.gz

Old logs are the in same directory as current logs which is just noisy but more importantly there is also a whole slew of problems with log rotation, compression, keeping arcane syslog configuration updated as things change, running out of space if you mess up etc. And let’s face it, having logs sitting on a server like this is next to useless – you want them pushed to a central location so that you can keep track of what’s going on.

With this mind, I finally got round to trying out an idea I had a while ago about how to clean this stuff up. I now have this:

james@lucinda:/var/log$ ls -l dmesg*
prw-r--r-- 1 root root 0 Feb  5 14:05 dmesg

My logs are now named pipes! From an application’s perspective nothing has changed – the log file is the same place and looks and acts like a file. Of course, this isn’t very useful if nothing is connected to the other end of the pipe, so I wrote a little ruby application called Monsoon that sits and watches named pipes in directory that you configure and every time something is sent down one of these pipes, it packages it up into a json fragment as follows:

    {
        "time": "Sun Feb 05 14:05:41 +0000 2012",
        "message": "test message",
        "origin": {
            "route": "/var/log/dmesg",
            "hostname": "lucinda"
        }
    }

and sends off to a TCP server socket that you configure – but this of course could be sent off via stomp or HTTP instead – where you can do something useful with it (put into a database, logstash, SOLR, and so on) It’s then just a case of ensuring that whatever is storing the logs has enough disk space instead of your whole web cluster or whatever (though obviously you should still monitor that too).

I’m quite pleased with the results so far. I’m going to test some more and see how things go.

A Standardized, general-purpose purely functional programming language, with non-strict semantics and strong static typing for.. shell scripting? Yup.

Don Stewart’s presentation on  Practical Haskell Programming really opened my eyes.

Don shows how even the most throwaway scripts, in this case a script to toggle a value in a sysctl, can benefit from a type-safe language with generic, reusable abstractions.

Very nice indeed. I do wonder how practical it would be to write code like this for every little thing all the time though!