:::: MENU ::::

Fluentd: A Very Quick Intro With Apache


This article is a very brief introduction to using Fluentd, an open source log collector.

Recently, I’ve been given the task of centralizing all of my companies log files, and decided that this was going to be a perfect opportunity to use Fluentd in a production environment.

Getting Everything Installed

Ok so first things first, you need to get Fluentd installed. There are many already provided ways to get this installed, but I’m using CentOS, so I’m going to be using the rpm:

You can check out Fluentd’s own documentation if you are using something else.

Setting Up Configs

Fluentd puts all of the config files in /etc/td-agent/, and the one we’re looking at editing is /etc/td-agent/td-agent.conf

So, we’re going to need to specify a source for Fluentd to read from, which in this case is going to be your apache access log:

Breaking this down line by line:

  1. type tail : tail is a source plugin that essentially tails whatever file you tell it to.

  2. format apache : This tells Fluentd what format the log file is written in, so it can easily parse it. There are many built in parsers you can check out here.

  3. path /var/log/httpd/access_log : The path to the log file you want to read.

  4. tag local.apache.access : A tag is used for routing the messages.

Next, we need to specify a place to put all these messages:

Breaking it down:

  1. match local.apache.* : This will grab all messages that match this tag. The tag can be written as a regular expression.
  2. type file : This is telling Fluentd to use the file plugin to write messages.
  3. path /var/log/td-agent/td.apache : The path to write to.

Test Away

So at this point, everything should be set up and ready to test out. For this test, I’m going to be using ab

Tada! If everything worked out, you should have some log data sitting in /var/log/httpd/access_log.*.log, where * is the current date.

Final Thoughts

Fluentd was really easy to get everything set up and running with a very minimal footprint.

This particular example is not all that helpful in actually solving the centralized logging problem, but there are much more advanced configurations and plugins to use with Fluentd. I may cover this in another post later on, so stay tuned.

So, what do you think ?