Telegraf / InfluxDB / Grafana as syslog receiver

So you are using the TIG-Stack to visualize network device metrics like interface counters, CPU and memory already? Great, I think Grafana really excels in the dashboarding domain. But did you ever wonder, how to leverage the same solution to store and display syslog messages? Well, then this post is for you.

UPDATE 03/2021: Message query now uses a text box variable instead of ad-hoc query, which stopped working with Grafana 7.x

For lab sessions and small to medium environments Telegraf, InfluxDB and Grafana can be installed on a single host. All three software instances are written in GoLang and not very resource intensive. A minimal VM (2vCPU, 2G RAM, 8G HD) or even a RaspberryPi is sufficient for the first steps and can act as a syslog receiver as well.

Switch / Router config

First of all, just send classic UDP syslog messages, on Cisco IOS devices for instance by configuring

 logging host <syslog receiver IP>

rsyslog config

(Update Q3/2020: Efforts are on the way to bring RFC3164 to Telegraf version 1.16.0, so you might keep an eye on github)

Because Telegraf only accepts TCP syslog messages in a certain format (RFC5424), the rsyslog daemon is used to receive classic RFC3164 Syslog messages via UDP port 514 and pipe them to the local Telegraf instance. So the first step is to adapt the rsyslog.conf to our needs:

sudo vi /etc/rsyslog.conf

$WorkDirectory /var/spool/rsyslog  
$ActionQueueType LinkedList  
$ActionQueueFileName srvrfwd  
$ActionResumeRetryCount -1  
$ActionQueueSaveOnShutdown on  
$ModLoad imudp #loads the udp module  
$UDPServerAddress localhost 
$UDPServerRun 514 
*.* @@(o);RSYSLOG_SyslogProtocol23Format

If you would like to filter between local and external syslog messages, there is the possibility of very sophisticated rulesets, for example:

:hostname, contains, "grafanapi"
*.notice @@(o);RSYSLOG_SyslogProtocol23Format
:hostname, !contains, "grafanapi"
*.* @@(o);RSYSLOG_SyslogProtocol23Format

Local logging messages are only forwarded when the severity level is equal or greater then notice. External Syslog messages (hostname != grafanapi) will be forwarded to Telegraf regardless of the severity level.
BTW: People who love to hate YAML nowadays might not have worked with ancient config files like this one – that’s for sure. Anyway, now restart the service to finish the rsyslog configuration.

sudo systemctl restart rsyslog

syslog-ng as an alternative

There might be situations where you need to use syslog-ng instead of rsyslog to pipe messages to Telegraf.

sudo yum install syslog-ng

vi /etc/syslog-ng/syslog-ng.conf # Add the following lines to pipe the messages to Telegraf

source s_sys {
    udp(ip( port(514));
destination telegraf_local {
 syslog("" port(6514));
filter f_default    { level(info..emerg)); 
log { source(s_sys); filter(f_default); destination(telegraf_local); };

sudo systemctl enable syslog-ng
sudo systemctl start syslog-ng

Telegraf config

Telegraf takes the locally forwarded messages and sends them to the InfluxDB time-series database. So we have to adjust/uncomment the syslog and the database section of the Telegraf config file.

sudo vi /etc/telegraf/telegraf.conf

  server = "tcp://localhost:6514"

  database = "telegraf"
  username = "<username>"
  password = "<password>"

sudo systemctl reload telegraf

InfluxDB config

The required ‘telegraf’ database has been created during the InfluxDB installation, otherwise:

$ influx

You should now see a new measurement ‘syslog’ in this database when the first syslog messages are relayed through the telegraf instance. Remember, the timestamp in InfluxDB is always UTC!

Grafana Dashboard

The Syslog dashboard I build may be downloaded from GitHub or via the Grafana dashboard ID 12433 and imported using the following UI dialog.

Just click + Import and upload the .json File of the syslog dashboard.

Choose a dashboard name and the name of your datasource (InfluxDB in most cases) and Import – et voila:

The Dashboard shows a statistics graph panel at the top, based on the timeframe chosen. Plus a table view of all messages within this timeframe, including the usual columns like message time, appname, host, severity and message text. You can zoom into specific timeframes (syslog peaks or massive errors) by just marking the area by mouse in the graph panel. The table view adjusts accordingly.

It also provides some extra filters based on appname, hostname, severity and message text.


With access to syslog or (more general) event data, you are now able to design central infrastructure dashboards not only based upon classic telemetry metrics, but also event statistics. A syslog count panel like the one at the top of this page is pretty useful to qualify the overall health of your systems at a glance. This way Grafana could become your No.1 visibility tool for the whole infrastructure stack with futureproof extensions like Streaming Telemetry, Netconf, API-Calls, annotations, and so on – time to slowly say goodbye to good old SNMP.

33 thoughts on “Telegraf / InfluxDB / Grafana as syslog receiver

  1. Hello, I get the following error when importing JSON file.
    TypeError: Cannot read property ‘ctor’ of undefined true

    Could you please advise how to deal with this error?


  2. Telegraf now supports UDP Syslog Messages. Do know how to configure that on telegraf to leave out rsyslog?

    1. But UDP Syslog has been supported like forever, I think. So if your sources send syslog messages according to RFC5424 you don’t need rsyslog or syslog-ng. Only if you need to process RFC3164 style messages like most networking equipment still uses today, a translation is needed.

  3. If someone receives an 414 Error “Request-URI Too Large”, you can set a custom All value in the tempaltes of the dashboard for appname and hostname. Just set the value to .*
    This when there are too many appnames that all should be requested to InfluxDB.

    1. Interesting, thanks for pointing that out!
      I haven’t seen this in the past, so maybe I need to generate way more Syslog messages 😉

  4. syslog-ng conf is wrong. The line ‘filter f_default { level(info..emerg));’ is incomplete, because must be followed with ‘and’ and another syntax.

  5. There are some errors in your configuration (for ex. when creating user in influxdb), but anyway I thank you for the effort.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.