Telegraf / InfluxDB / Grafana as syslog receiver

So you are using the TIG-Stack to visualize network device metrics like interface counters, CPU and memory already? Great, I think Grafana really excels in the dashboarding domain. But did you ever wonder, how to leverage the same solution to store and display syslog messages? Well, then this post is for you.

UPDATE 03/2021: Message query now uses a text box variable instead of ad-hoc query, which stopped working with Grafana 7.x

For lab sessions and small to medium environments Telegraf, InfluxDB and Grafana can be installed on a single host. All three software instances are written in GoLang and not very resource intensive. A minimal VM (2vCPU, 2G RAM, 8G HD) or even a RaspberryPi is sufficient for the first steps and can act as a syslog receiver as well.

Switch / Router config

First of all, just send classic UDP syslog messages, on Cisco IOS devices for instance by configuring

 logging host <syslog receiver IP>

rsyslog config

(Update Q3/2020: Efforts are on the way to bring RFC3164 to Telegraf version 1.16.0, so you might keep an eye on github)

Because Telegraf only accepts TCP syslog messages in a certain format (RFC5424), the rsyslog daemon is used to receive classic RFC3164 Syslog messages via UDP port 514 and pipe them to the local Telegraf instance. So the first step is to adapt the rsyslog.conf to our needs:

sudo vi /etc/rsyslog.conf

$WorkDirectory /var/spool/rsyslog  
$ActionQueueType LinkedList  
$ActionQueueFileName srvrfwd  
$ActionResumeRetryCount -1  
$ActionQueueSaveOnShutdown on  
$ModLoad imudp #loads the udp module  
$UDPServerAddress localhost 
$UDPServerRun 514 
*.* @@(o)127.0.0.1:6514;RSYSLOG_SyslogProtocol23Format

If you would like to filter between local and external syslog messages, there is the possibility of very sophisticated rulesets, for example:

:hostname, contains, "grafanapi"
*.notice @@(o)127.0.0.1:6514;RSYSLOG_SyslogProtocol23Format
:hostname, !contains, "grafanapi"
*.* @@(o)127.0.0.1:6514;RSYSLOG_SyslogProtocol23Format

Local logging messages are only forwarded when the severity level is equal or greater then notice. External Syslog messages (hostname != grafanapi) will be forwarded to Telegraf regardless of the severity level.
BTW: People who love to hate YAML nowadays might not have worked with ancient config files like this one – that’s for sure. Anyway, now restart the service to finish the rsyslog configuration.

sudo systemctl restart rsyslog

syslog-ng as an alternative

There might be situations where you need to use syslog-ng instead of rsyslog to pipe messages to Telegraf.

sudo yum install syslog-ng

vi /etc/syslog-ng/syslog-ng.conf # Add the following lines to pipe the messages to Telegraf

source s_sys {
    udp(ip(0.0.0.0) port(514));
};
destination telegraf_local {
 syslog("127.0.0.1" port(6514));
};
filter f_default    { level(info..emerg)); 
};
log { source(s_sys); filter(f_default); destination(telegraf_local); };

sudo systemctl enable syslog-ng
sudo systemctl start syslog-ng

Telegraf config

Telegraf takes the locally forwarded messages and sends them to the InfluxDB time-series database. So we have to adjust/uncomment the syslog and the database section of the Telegraf config file.

sudo vi /etc/telegraf/telegraf.conf

[[inputs.syslog]]
  server = "tcp://localhost:6514"

[[outputs.influxdb]]
  database = "telegraf"
  username = "<username>"
  password = "<password>"

sudo systemctl reload telegraf

InfluxDB config

The required ‘telegraf’ database has been created during the InfluxDB installation, otherwise:

$ influx
> CREATE USER <username> WITH PASSWORD <password> WITH ALL PRIVILEGES
> CREATE DATABASE telegraf
> CREATE RETENTION POLICY "4Weeks" ON "telegraf" DURATION 4w REPLICATION 1 DEFAULT;

You should now see a new measurement ‘syslog’ in this database when the first syslog messages are relayed through the telegraf instance. Remember, the timestamp in InfluxDB is always UTC!

Grafana Dashboard

The Syslog dashboard I build may be downloaded from GitHub or via the Grafana dashboard ID 12433 and imported using the following UI dialog.

Just click + Import and upload the .json File of the syslog dashboard.

Choose a dashboard name and the name of your datasource (InfluxDB in most cases) and Import – et voila:

The Dashboard shows a statistics graph panel at the top, based on the timeframe chosen. Plus a table view of all messages within this timeframe, including the usual columns like message time, appname, host, severity and message text. You can zoom into specific timeframes (syslog peaks or massive errors) by just marking the area by mouse in the graph panel. The table view adjusts accordingly.

It also provides some extra filters based on appname, hostname, severity and message text.

Closing

With access to syslog or (more general) event data, you are now able to design central infrastructure dashboards not only based upon classic telemetry metrics, but also event statistics. A syslog count panel like the one at the top of this page is pretty useful to qualify the overall health of your systems at a glance. This way Grafana could become your No.1 visibility tool for the whole infrastructure stack with futureproof extensions like Streaming Telemetry, Netconf, API-Calls, annotations, and so on – time to slowly say goodbye to good old SNMP.