Getting started with logstash part 1

Statsd & Graphite

In my last post I mentioned statsd and graphite, these are 2 tools that I use in conjunction with logstash and they make a good team!

So the general idea behind running these utilities together is that you can get fancy graphs based off certain events, so for example it might be nice to see a longer term trend of IIS response times for a given website or something to that effect.

I won’t go into how to set them up as there are a couple of guides around on how to do it, I can try and point people in the right direction if you have any questions however.

One important-ish thing to note is that if you wish to save a group/single graph (you most certainly will!) you will need to set up a django account when you configure the graphite web app. You can make this work after you have installed it if you decided to skip that step.

Here is how I have configured one of my statsd metrics:

statsd {
    type => "iis"
    tags => ["example.co.uk"]
    exclude_tags => ["_grokparsefailure"]

    increment => "live.example.iis.response.%{response}"
    timing    => ["live.example.iis.responseTime.", "%{timetaken}"]
  }

Events of type “iis” which have the tag “example.co.uk” and did not fail grok parsing are accepted by this output filter. Each time this filter recives an event the metric for that particular response code is incremented (So you could plot a graph of how many files you were missing, 404s, for example). the %{field} notation is used to indicate that this bit of text should be taken from the event itself.

You can have many of these output filters defined for events with different tags, so you can have metrics for different sites etc.

One thing that tripped me up was the storage aggregation configuration, to the lay-man this means that when your storage schema for carbon specifies that metrics should have thier precsion reduced a certain set of rules determine what gets stored as the new value for this time period. However, as the metrics I was tracking did not always have a value (null) I found that for the longer graph periods there was no data. A great post over at http://obfuscurity.com/2012/04/Unhelpful-Graphite-Tip-9 pointed me in the right direction on how to resolve this issue.