Telegraf / InfluxDB / Grafana as syslog receiver

So you are using the TIG-Stack to visualize network device metrics like interface counters, CPU and memory already? Great, I think Grafana really excels in the dashboarding domain. But did you ever wonder, how to leverage the same solution to store and display syslog messages? Well, then this post is for you.

For lab sessions and small to medium environments Telegraf, InfluxDB and Grafana can be installed on a single host. All three software instances are written in GoLang and not very resource intensive. A minimal VM (2vCPU, 2G RAM, 8G HD) or even a RaspberryPi is sufficient for the first steps and can act as a syslog receiver as well.

Switch / Router config

First of all, just send classic UDP syslog messages, on Cisco IOS devices for instance by configuring

 logging host <syslog receiver IP>

rsyslog config

(Update Q3/2020: Efforts are on the way to bring RFC3164 to Telegraf version 1.16.0, so you might keep an eye on github)

Because Telegraf only accepts TCP syslog messages in a certain format (RFC5424), the rsyslog daemon is used to receive classic RFC3164 Syslog messages via UDP port 514 and pipe them to the local Telegraf instance. So the first step is to adapt the rsyslog.conf to our needs:

sudo vi /etc/rsyslog.conf

$WorkDirectory /var/spool/rsyslog  
$ActionQueueType LinkedList  
$ActionQueueFileName srvrfwd  
$ActionResumeRetryCount -1  
$ActionQueueSaveOnShutdown on  
$ModLoad imudp #loads the udp module  
$UDPServerAddress localhost 
$UDPServerRun 514 
*.* @@(o)127.0.0.1:6514;RSYSLOG_SyslogProtocol23Format

If you would like to filter between local and external syslog messages, there is the possibility of very sophisticated rulesets, for example:

:hostname, contains, "grafanapi"
*.notice @@(o)127.0.0.1:6514;RSYSLOG_SyslogProtocol23Format
:hostname, !contains, "grafanapi"
*.* @@(o)127.0.0.1:6514;RSYSLOG_SyslogProtocol23Format

Local logging messages are only forwarded when the severity level is equal or greater then notice. External Syslog messages (hostname != grafanapi) will be forwarded to Telegraf regardless of the severity level.
BTW: People who love to hate YAML nowadays might not have worked with ancient config files like this one – that’s for sure. Anyway, now restart the service to finish the rsyslog configuration.

sudo systemctl restart rsyslog

syslog-ng as an alternative

There might be situations where you need to use syslog-ng instead of rsyslog to pipe messages to Telegraf.

sudo yum install syslog-ng

vi /etc/syslog-ng/syslog-ng.conf # Add the following lines to pipe the messages to Telegraf

source s_sys {
    udp(ip(0.0.0.0) port(514));
};
destination telegraf_local {
 syslog("127.0.0.1" port(6514));
};
filter f_default    { level(info..emerg)); 
};
log { source(s_sys); filter(f_default); destination(telegraf_local); };

sudo systemctl enable syslog-ng
sudo systemctl start syslog-ng

Telegraf config

Telegraf takes the locally forwarded messages and sends them to the InfluxDB time-series database. So we have to adjust/uncomment the syslog and the database section of the Telegraf config file.

sudo vi /etc/telegraf/telegraf.conf

[[inputs.syslog]]
  server = "tcp://localhost:6514"

[[outputs.influxdb]]
  database = "telegraf"
  username = "<username>"
  password = "<password>"

sudo systemctl reload telegraf

InfluxDB config

The required ‘telegraf’ database has been created during the InfluxDB installation, otherwise:

$ influx
> CREATE USER <username> WITH PASSWORD <password> WITH ALL PRIVILEGES
> CREATE DATABASE telegraf
> CREATE RETENTION POLICY "4Weeks" ON "telegraf" DURATION 4w REPLICATION 1 DEFAULT;

You should now see a new measurement ‘syslog’ in this database when the first syslog messages are relayed through the telegraf instance. Remember, the timestamp in InfluxDB is always UTC!

Grafana Dashboard

The Syslog dashboard I build may be downloaded from GitHub or via the Grafana dashboard ID 12433 and imported using the following UI dialog.

Just click + Import and upload the .json File of the syslog dashboard.

Choose a dashboard name and the name of your datasource (InfluxDB in most cases) and Import – et voila:

The Dashboard shows a statistics graph panel at the top, based on the timeframe chosen. Plus a table view of all messages within this timeframe, including the usual columns like message time, appname, host, severity and message text. You can zoom into specific timeframes (syslog peaks or massive errors) by just marking the area by mouse in the graph panel. The table view adjusts accordingly.

It also provides some extra filters based on appname, hostname, severity and message text. The message text query uses the build-in regex feature and needs to be encapsulated in // . So, if you are searching for messages with uptime in the message text, enter the value /uptime/ in the query field.

Closing

With access to syslog or (more general) event data, you are now able to design central infrastructure dashboards not only based upon classic telemetry metrics, but also event statistics. A syslog count panel like the one at the top of this page is pretty useful to qualify the overall health of your systems at a glance. This way Grafana could become your No.1 visibility tool for the whole infrastructure stack with futureproof extensions like Streaming Telemetry, Netconf, API-Calls, annotations, and so on – time to slowly say goodbye to good old SNMP.

9 thoughts on “Telegraf / InfluxDB / Grafana as syslog receiver

  1. Oh I have a typo in my adress ^^ sry! Here My Email without typo.
    jules.schmitt@gmail.com

    Hello NWMichl,

    Thanks a lot for this tutorial.

    I have one Problem Maybe you solved it.

    I have a Server-Client syslog-ng Environment.

    On my syslog-ng server I split the logs to 2 different Destinations. One write the logs to a file. The other one sends the (same!) Logs to telegraf like your tutorial.

    The logfile on my server looks 1:1 like the log on my syslog clients (Remote Server).

    In Grafana i get double Timestamps. One comes from telegraf I think, and the other timestamp from the log message itself.

    My Received Logfiles looks like:

    $DATE $HOST $APPNAME $PID – $MESSAGE

    In telegraf (and Grafana) I get

    $NEWDATE $(the number 1) $FQDN $SEVERITY $(in the Message part I get the complete log message: $DATE $HOST $APPNAME $PID – $MESSAGE )

    With that result I cant realy parse the DATA.

    Did you have the same problem? and if yes how did you solve it? πŸ™‚

    Kind Regards,

    Julien Schmitt

    1. Hi Julien,

      thanks for reaching out!

      I didn’t run into the same problem, maybe because this whole blog post is about processing Syslog messages from network devices according to the ‘older’ RFC3164 format.
      As you try to collect Syslog from servers, you might be better off sending them directly to the Telegraf agent via TCP/6514 using the recent RFC5424 standard?

      Regards,
      Michael

  2. Hi Michael,

    many thanks for your great plugin!

    But is it possible that the search function is not working under Grafana 7.x?
    Search example:
    Query messages /login/

    Any ideas?

    Thanks JO!

    1. Hi Johannes,

      sorry for the delay, but I had to find time to upgrade to 7.2, so thanks for the push πŸ˜‰
      After typing the ad-hoc query string ‘/login/’, it seems that you cannot leave the field via TAB or ENTER.
      It only works, when you click on a free space with the mouse … strange.

      Hope this (hack) helps,
      Michael

      1. Hi Michael,

        Thanks for your feedback!
        No problem, I understand πŸ˜‰

        Thanks for testing, but in my case the filter does not apply if I enter something and klick anywhere else on the dashboard ( used /someword/ ).

        Any special config needed on Grafana?

        Thanks JO!

      2. Hi Jo,

        there is no special config needed on the Grafana side, this dashboard utilizes only the native ad-hoc query function.
        I tested with Firefox btw.

        Regards,
        Michael

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.