Telegraf / InfluxDB / Grafana as syslog receiver

So you are using the TIG-Stack to visualize network device metrics like interface counters, CPU and memory already? Great, I think Grafana really excels in the dashboarding domain. But did you ever wonder, how to leverage the same solution to store and display syslog messages? Well, then this post is for you.

UPDATE 03/2021: Message query now uses a text box variable instead of ad-hoc query, which stopped working with Grafana 7.x

For lab sessions and small to medium environments Telegraf, InfluxDB and Grafana can be installed on a single host. All three software instances are written in GoLang and not very resource intensive. A minimal VM (2vCPU, 2G RAM, 8G HD) or even a RaspberryPi is sufficient for the first steps and can act as a syslog receiver as well.

Switch / Router config

First of all, just send classic UDP syslog messages, on Cisco IOS devices for instance by configuring

 logging host <syslog receiver IP>

rsyslog config

(Update Q3/2020: Efforts are on the way to bring RFC3164 to Telegraf version 1.16.0, so you might keep an eye on github)

Because Telegraf only accepts TCP syslog messages in a certain format (RFC5424), the rsyslog daemon is used to receive classic RFC3164 Syslog messages via UDP port 514 and pipe them to the local Telegraf instance. So the first step is to adapt the rsyslog.conf to our needs:

sudo vi /etc/rsyslog.conf

$WorkDirectory /var/spool/rsyslog  
$ActionQueueType LinkedList  
$ActionQueueFileName srvrfwd  
$ActionResumeRetryCount -1  
$ActionQueueSaveOnShutdown on  
$ModLoad imudp #loads the udp module  
$UDPServerAddress localhost 
$UDPServerRun 514 
*.* @@(o);RSYSLOG_SyslogProtocol23Format

If you would like to filter between local and external syslog messages, there is the possibility of very sophisticated rulesets, for example:

:hostname, contains, "grafanapi"
*.notice @@(o);RSYSLOG_SyslogProtocol23Format
:hostname, !contains, "grafanapi"
*.* @@(o);RSYSLOG_SyslogProtocol23Format

Local logging messages are only forwarded when the severity level is equal or greater then notice. External Syslog messages (hostname != grafanapi) will be forwarded to Telegraf regardless of the severity level.
BTW: People who love to hate YAML nowadays might not have worked with ancient config files like this one – that’s for sure. Anyway, now restart the service to finish the rsyslog configuration.

sudo systemctl restart rsyslog

syslog-ng as an alternative

There might be situations where you need to use syslog-ng instead of rsyslog to pipe messages to Telegraf.

sudo yum install syslog-ng

vi /etc/syslog-ng/syslog-ng.conf # Add the following lines to pipe the messages to Telegraf

source s_sys {
    udp(ip( port(514));
destination telegraf_local {
 syslog("" port(6514));
filter f_default    { level(info..emerg)); 
log { source(s_sys); filter(f_default); destination(telegraf_local); };

sudo systemctl enable syslog-ng
sudo systemctl start syslog-ng

Telegraf config

Telegraf takes the locally forwarded messages and sends them to the InfluxDB time-series database. So we have to adjust/uncomment the syslog and the database section of the Telegraf config file.

sudo vi /etc/telegraf/telegraf.conf

  server = "tcp://localhost:6514"

  database = "telegraf"
  username = "<username>"
  password = "<password>"

sudo systemctl reload telegraf

InfluxDB config

The required ‘telegraf’ database has been created during the InfluxDB installation, otherwise:

$ influx

You should now see a new measurement ‘syslog’ in this database when the first syslog messages are relayed through the telegraf instance. Remember, the timestamp in InfluxDB is always UTC!

Grafana Dashboard

The Syslog dashboard I build may be downloaded from GitHub or via the Grafana dashboard ID 12433 and imported using the following UI dialog.

Just click + Import and upload the .json File of the syslog dashboard.

Choose a dashboard name and the name of your datasource (InfluxDB in most cases) and Import – et voila:

The Dashboard shows a statistics graph panel at the top, based on the timeframe chosen. Plus a table view of all messages within this timeframe, including the usual columns like message time, appname, host, severity and message text. You can zoom into specific timeframes (syslog peaks or massive errors) by just marking the area by mouse in the graph panel. The table view adjusts accordingly.

It also provides some extra filters based on appname, hostname, severity and message text.


With access to syslog or (more general) event data, you are now able to design central infrastructure dashboards not only based upon classic telemetry metrics, but also event statistics. A syslog count panel like the one at the top of this page is pretty useful to qualify the overall health of your systems at a glance. This way Grafana could become your No.1 visibility tool for the whole infrastructure stack with futureproof extensions like Streaming Telemetry, Netconf, API-Calls, annotations, and so on โ€“ time to slowly say goodbye to good old SNMP.

35 thoughts on “Telegraf / InfluxDB / Grafana as syslog receiver

  1. Oh I have a typo in my adress ^^ sry! Here My Email without typo.

    Hello NWMichl,

    Thanks a lot for this tutorial.

    I have one Problem Maybe you solved it.

    I have a Server-Client syslog-ng Environment.

    On my syslog-ng server I split the logs to 2 different Destinations. One write the logs to a file. The other one sends the (same!) Logs to telegraf like your tutorial.

    The logfile on my server looks 1:1 like the log on my syslog clients (Remote Server).

    In Grafana i get double Timestamps. One comes from telegraf I think, and the other timestamp from the log message itself.

    My Received Logfiles looks like:


    In telegraf (and Grafana) I get

    $NEWDATE $(the number 1) $FQDN $SEVERITY $(in the Message part I get the complete log message: $DATE $HOST $APPNAME $PID โ€“ $MESSAGE )

    With that result I cant realy parse the DATA.

    Did you have the same problem? and if yes how did you solve it? ๐Ÿ™‚

    Kind Regards,

    Julien Schmitt

    1. Hi Julien,

      thanks for reaching out!

      I didn’t run into the same problem, maybe because this whole blog post is about processing Syslog messages from network devices according to the ‘older’ RFC3164 format.
      As you try to collect Syslog from servers, you might be better off sending them directly to the Telegraf agent via TCP/6514 using the recent RFC5424 standard?


  2. Hi Michael,

    many thanks for your great plugin!

    But is it possible that the search function is not working under Grafana 7.x?
    Search example:
    Query messages /login/

    Any ideas?

    Thanks JO!

    1. Hi Johannes,

      sorry for the delay, but I had to find time to upgrade to 7.2, so thanks for the push ๐Ÿ˜‰
      After typing the ad-hoc query string ‘/login/’, it seems that you cannot leave the field via TAB or ENTER.
      It only works, when you click on a free space with the mouse … strange.

      Hope this (hack) helps,

      1. Hi Michael,

        Thanks for your feedback!
        No problem, I understand ๐Ÿ˜‰

        Thanks for testing, but in my case the filter does not apply if I enter something and klick anywhere else on the dashboard ( used /someword/ ).

        Any special config needed on Grafana?

        Thanks JO!

      2. Hi Jo,

        there is no special config needed on the Grafana side, this dashboard utilizes only the native ad-hoc query function.
        I tested with Firefox btw.


      1. Yap, finally found the time to dig into this. A fix / new dashboard using a text box variable instead of the ad-hoc query is on the way.

  3. Hi there,

    I have an uglier problem ๐Ÿ˜ฆ
    I see the rsyslog count (first panel) data in the dashboard but the syslog entries do not appear in the panel below.
    If I go to influx then I see stuff like this upon “SELECT * from syslog”

    1607694541585888413 %LOG_LOCAL7-6-SYSTEM_MSG local7 23 grafana APIC-2 [refresh,session][info][subj-[uni/userext/user-Cisco_SN_NIR]/sess-8590886784] From- info 6 1607698141000000000 1
    1607694542325623958 %LOG_LOCAL7-6-SYSTEM_MSG local7 23 grafana APIC-2 [refresh,session][info][subj-[uni/userext/user-admin]/sess-8590886785] From- info 6 1607698142000000000 1
    1607694542758367619 %LOG_LOCAL7-6-SYSTEM_MSG local7 23 grafana APIC-2 [refresh,session][info][subj-[uni/userext/user-Cisco_SN_NIR]/sess-8590886786] From- info 6 1607698142000000000 1
    1607694543617836571 %LOG_LOCAL7-6-SYSTEM_MSG local7 23 grafana APIC-2 [refresh,session][info][subj-[uni/userext/user-Cisco_SN_NIR]/sess-8590886787] From- info 6 1607698143000000000 1
    1607694543642907271 %LOG_LOCAL7-6-SYSTEM_MSG local7 23 grafana APIC-2 [refresh,session][info][subj-[uni/userext/user-Cisco_SN_NIR]/sess-8590886788] From- info 6 1607698143000000000 1

    Any idea what might be wrong in the query or maybe some rights on the Dashboard Panel side?

    Thank you

  4. I changed in the query:
    SELECT “severity_code”, “message” FROM “syslog” WHERE (“hostname” =~ /^$hostname$/ AND “appname” =~ /^$appname$/ AND “severity” =~ /^$severity$/) AND $timeFilter GROUP BY “hostname”, “appname”

    the host=~/^hostname$/ with hostname=~/^hostname$/ and now it works.

  5. Hi,

    I unfortunately started with influxdb2 and already got my smart meter reporting energy data. Did you try to migrate your dashboard to the new flux language?

  6. Hi, nice work, really liked this.
    However :-), how can I verify if the syslog is receiving data from my devices?
    I added the filter, read about it in the manuals, but the grafana dashboard is loading wit ‘local’ syslog?
    The filter shows the name of my lab device, but filtering on it gives a regex error… [It is not clear if I add the filter to the config of syslog, or do I comment out the ‘output-toserver’?

    Other API calls to the restconf device work, but the syslog does not work. Probably a lot of reading to do.

      1. O, forgot, to be complete
        I had to change the order of the start of the UDP server in the config.
        I commented your config lines (UDP server information), and uncommented the defaults in the rsyslog.conf.
        Probably changed the order…. but this seems to resolve the issue.

      2. Hi Michael!

        Thanks for your dashboard update!
        But now if I select a “Hostname” from the dropdown box the Pane is showing “no data” for this systems.
        So it is not possible to see only logs from one ore multiple selected systems.
        Any ideas how to solve this?

        Thanks and BR/JO!

      3. Hi Johannes,

        thanks for pointing this out, I messed up with the internal variable name of the ‘hostname’ dropdown filter.
        It should work now as expected.


  7. I’ve tried using this with Grafana latest in a Docker environment. The syslog data is there (I can display messages this with a alternative table panel), but using your dashboard only the stats are displayed. The table always shows “no data”.

    Any ideas, what might be causing this or how to debug?

    1. Not exactly for that problem, but I discovered that something changed with regards to the ad-hoc query between Grafana v6.x and v7.x.
      I think I will use the Textbox variable from now on and upload a new, tested dashboard in a few days.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.