How many times do you connect with SSH to your remote server and cat, grep, tail and awk through your logs? It probably works for 3 servers and running a handful services, but if you have more, you should definitely spend some time to centralize your logs.

I personally prefer Graylog2 which can deal very well with different log formats like GELF, Syslog RFC's. Just start some listener with the format and forward them to your Graylog2 instance.

A few servers are NAT'ed behind VDSLs with dynamic IP's and some physical and virtual servers hosted somewhere else with static public IP's which have all running Linux. This makes monitoring normally hard, so I use a fully meshed tinc VPN, so I don't have to deal a lot securing many different applications and protocols and everything is reachable in a flat private /24 IPv4 network. To manage the server basic configuration I use Ansible and most services run as a container using Docker.

What is the benefit?

  • The GELF listener allows me to the Docker built-in driver to easily centralize containerized application logs
  • Log4j2 provides a GELF output as well which gives me full access to my logs from my OpenNMS instances
  • Graylog2 can parse Syslog in various RFC's which gives me centralized system logs from my Linux systems
  • It is very easy to deploy the configurations with Ansible

Setting up a Graylog2 Service Stack

Define a service stack using Docker Compose and get Graylog2 up and running. Here is docker-compose.yml file I use, just download it and run docker-compose up -d.

The following ports get exposed:

  • 9000: The Graylog2 web application
  • 12201: GELF UDP listener for my Java applications
  • 514: GELF UDP Syslog listener to forward my system logs

Configure a GELF UDP and a Syslog UDP Input

With the first login in Graylog2 you have to create two Inputs. The GELF UDP input is used to receive log messages from my OpenNMS applications and Docker Daemons and the Syslog UDP input receives my Syslog messages.

Graylog2 Input Configuration

Configure Syslog forwarder

On my systems is rsyslog running. It is required to configure rsyslogd and I use Ansible to create a file in /etc/rsyslog.d/50-graylog-forwarding with following content:

if $programname == 'snmpd' and $msg contains 'statfs' then {
    stop
}

*.* @172.42.42.42:514;RSYSLOG_SyslogProtocol23Format

On 172.42.42.42, my Graylog2 instance is running and is listening on 514/udp. After restarting rsyslog the logs will be forwarded. The first if-statement ensures I don't log a lot of garbage coming from snmpd which described more in detail in this blog post.

Configure OpenNMS to forward logs to Graylog2

Add Modify the ${OPENNMS_HOME/etc/log4j2.xml and add a GELF UDP log appender which is described in our OpenNMS Wiki.

All the daemons will no also forward their logs to Graylog2 via UDP and are searchable for services node labels, node ids and daemons. Different OpenNMS instances running various versions are identified by an application_name` tag.

Graylog2 Screenshot

Forward Docker Container logs

To get logs from my applications running in Docker container, I have configured them to use the GELF driver by just adding this snippet to my service definition:

logging:
  driver: "gelf"
  options:
    gelf-address: "udp://172.42.42.42:12201"
    tag: "horizon-core-web:stable"

The tag is used to identify the service, you can be as creative as you want and the gelf-address tells the Docker daemon where to forward the messages which is my Graylog2 listener input.

Happy logging and searching

comments powered by Disqus

Material Workshop OpenNMS im Büro 2.0

Am 30.03.2015 haben wir im Büro 2.0 in Berlin einen Workshop mit OpenNMS durchgeführt. Die Zielgruppe waren NeueinsteigerInnen und alle die …… Continue reading

Open Source Experience

Published on November 22, 2014

IPv6 and Monitoring

Published on March 19, 2016