MRTG

Some useful notes about MRTG.

Documentation: http://oss.oetiker.ch/mrtg/doc/index.en.html.

Initial command to create a mrtg.conf file:

/usr/bin/cfgmaker \
--global "Options[_]: growright" \
--output /etc/mrtg.cfg commstring@host1 commstring@host2

growright means that the graph grows to the right and not to the left which is default.

Getting all interfaces included even if they are currently down

This can be done with the cfgmaker option '--if-filter=$if_admin && $default_iftype' meaning that interface up or down status is not taken into account but if an interface id disabled by administrator it is not shown.

/usr/bin/cfgmaker \ 
--global "Options[_]: growright" \
'--if-filter=$if_admin && $default_iftype'
--output /etc/mrtg.cfg commstring@host1 commstring@host2

Interval and refresh time

If we don't want to do snmp every 5 minutes we can set a different interval with the "Interval" option (not below 5 minutes, then we need to use rrdtools I understand). The "Refresh" option is used for how often we want to time out the web pages.

/usr/bin/cfgmaker \
--global "Options[_]: growright" \
--global 'Interval: 10' \
--global 'Refresh: 300' \
'--if-filter=$if_admin && $default_iftype'
--output /etc/mrtg.cfg commstring@host1 commstring@host2

Getting Broadcasts/multicasts, errors and discards

For HP swicthes it is not possible to get a separate number for multicast and broadcast but we can get non unicasts as a total number, and we can get errors and discards. All this is given in pkts/s not in bytes. We need the following OIDs:

• ifInNUcastPkts vs ifOutNUcastPkts (1.3.6.1.2.1.2.2.1.12 and 1.3.6.1.2.1.2.2.1.18)
• ifInErrors vs ifOutErrors (1.3.6.1.2.1.2.2.1.14 and 1.3.6.1.2.1.2.2.1.20)
• ifInDiscards vs ifOutDiscards (1.3.6.1.2.1.2.2.1.13 and 1.3.6.1.2.1.2.2.1.19)

These can be used in the mrtg.cfg file this way:

Target[host_1.err]: ifInErrors.1&ifOutErrors.1:commstring@host

for interface 1 at host "host" (with community string "commstring".

We shall use a template for implementing this.

Using templates

To use a template we include the option "--if-template". This is an interface template that is used for each interface, host templates are also possible.

/usr/bin/cfgmaker \
--global "Options[_]: growright" \
--global 'Interval: 10' \
--global 'Refresh: 300' \
--if-template=template.txt \
'--if-filter=$if_admin && $default_iftype' \
--output /etc/mrtg.cfg commstring@host1 commstring@host2

Inside the template file we want to put an entry for ordinary default traffic analysis, and one for each of the above mentioned counters: B/Mcast (Non unicast), Errors and discards (dropped packets): template.txt

We can use local options inside the template file, for errors and discards the perhour option is used since otherwise we would in reality always get 0 (we get that most of the time anyway).

Correcting legends

The Y legend on the graphs says "Bytes per Second" unless we do something about it. The YLegend option in the template file is used for correcting the legend to Pkts/sec for B/Mcasts and Pkts/hour for Errors and Discards.

Getting the output files in different directories

We can get the html files, log files and images in separate directories with the "HtmlDir", "Imagedir" and "Logdir" options. So now our cfgmaker command looks like this:

/usr/bin/cfgmaker \
--global 'HtmlDir: /var/www/mrtg/html' \
--global 'Imagedir: /var/www/mrtg/img' \
--global 'Logdir: /var/www/mrtg/log' \
--global "Options[_]: growright" \
--global 'Interval: 10' \
--global 'Refresh: 300' \
--if-template=template.txt \
'--if-filter=$if_admin && $default_iftype' \
--output /etc/mrtg.cfg commstring@host1 commstring@host2

However, debian includes the line WorkDir: /var/www/mrtg all by itself so to get rid of this I had to modify /usr/bin/cfgmaker and comment it out:

 #  for Debian
 # WorkDir: /var/www/mrtg

It is also possible to get a directory for each host instead: --subdirs=HOSTNAME__SNMPNAME (see http://oss.oetiker.ch/mrtg/doc/cfgmaker.en.html).

Community string as option

For many hosts it's easier to have the community string among the options (if it is the same for all hosts):

/usr/bin/cfgmaker \
--community=commstring \
--global 'HtmlDir: /var/www/mrtg/html' \
--global 'Imagedir: /var/www/mrtg/img' \
--global 'Logdir: /var/www/mrtg/log' \
--global "Options[_]: growright" \
--global 'Interval: 10' \
--global 'Refresh: 300' \
--if-template=template.txt \
'--if-filter=$if_admin && $default_iftype' \
--output /etc/mrtg.cfg host1 host2 host3 host4

Creating the indexes

Rather straightforward but still here is an example:

/usr/bin/indexmaker --output=host1.drop.html \ 
--filter name=~host1_[0-9]?[0-9].drop$ \ 
--title "host1 Drops (Pkts/h)" /etc/mrtg.cfg

Example for many ip addresses, f.eks. xxx.yyy.zzz.2 .3 .4 .5 .6 .7 .8

LIST='2 3 4 5 6 7 8'
for a in $LIST; do /usr/bin/indexmaker --output=host$a.html --filter
"name=~xxx.yyy.zzz.${a}_[0-9]?[0-9]$" --title "host$a" /etc/mrtg.cfg; done

Performance considerations

The actual SNMP traffic for doing the above for all ports on a 26 port switch is about 24kb, which amounts to approx. 1350kb for the 56 switches I wish to monitor. I read 1409 kb in my actual log. It probably depends on the content of the answer packet. The retrieval of SNMP data takes approx. 30 seconds and runs smoothly.

The performance issue with MRTG run this way occurs when MRTG starts creating 4 x 26 x 56 png images which makes my poor web server go to 1.78 loadavg and respond poorly for 3 minutes, and my webserver is not that small.

Quick 'n' dirty: Nice mrtg

The quick and dirty solution is to nice mrtg down so the web server responds as is should and only spend spare capacity on creating the mrtg graphs. In /etc/cron.d/mrtg

*/10 *  * * *   root    if [ -x /usr/bin/mrtg ] && [ -r /etc/mrtg.cfg ]; \
then env LANG=C nice -n 15 /usr/bin/mrtg /etc/mrtg.cfg >> /var/log/mrtg/mrtg.log 2>&1; fi

where I have added the nice -n 15 in front of the actual mrtg command. Web server still loads high but after all it has 4 CPUs to do so with and it responds fast as it should again.

Using rrdtool

A better solution to the problem should be to run rrdtool which is supported by MRTG and which does not create the graphs on every MRTG run but only on demand, via a 3rd party cgi script.

• First we install mrtg-rrd, either downloading from source or installing a package apt-get install mrtg-rrd. Also install rrdtool.
• Copy the mrtg-rrd.cgi file to the relevant cgi-bin directory.
• Edit mrtg-rrd.cgi file to set the my $conffile to /etc/mrtg.cfg. In the debian package it points to /etc/mrtg-rrd.conf which in turn points to /etc/mrtg.cfg. That works also.
• Make sure the mrtg.cfg file is readable to rrdtool.
• Then we tell mrtg to log in rrd format:

  /usr/bin/cfgmaker \
  --community=commstring \
  --global 'HtmlDir: /var/www/mrtg/html' \
  --global 'Imagedir: /usr/local/mrtg/img' \
  --global 'Logdir: /var/www/mrtg/log' \
  --global 'IconDir: /mrtg' \
  --global "Options[_]: growright" \
  --global 'Interval: 10' \
  --global 'Refresh: 300' \
  --global 'LogFormat: rrdtool' \
  --if-template=template.txt \
  '--if-filter=$if_admin && $default_iftype' \
  --output /etc/mrtg.cfg host1 host2 host3 host4
  

  (I think, in this case I just edited the config file with the "LogFormat" option).
• Note that we also changed the Imagedir to something out of the web document root.

  mkdir -p /usr/local/mrtg/img; chgrp www-data/usr/local/mrtg/img; chmod g+w /usr/local/mrtg/img

  or something else to make the Imagedir writable to the webserver.
• Also we added Icondir (this is relative to web root). Otherwise the mrtg pics in the bottom of the auto generated pages will not appear.
• Next time mrtg runs it will convert all the old log files to rrd and run rrd from there on.
• Hack /usr/bin/indexmaker :( indexmaker-hack.txt
• Run indexmaker with the option --rrdviewer=/cgi-bin/mrtg-rrd.cgi (see above for the other options you need).
• Edit any links you have made to point to /cgi-bin/mrtg-rrd.cgi/host1_1.html instead of html/host1_1.html.

Performance with rrdtools

So did I get better performance with rrdtools?

• I get lower CPU usage which is off course good.
• The problem is that the web pages are really slow to load because the images takes a long time to be created on the fly. That is rather unfortunate as it really stops me from using the graphs in the way I intend to.
• Supposedly the bottleneck mostly is running one instance of mrtg-rrd.cgi per image and thus f.ex running 26 at a time for a given index file. This can be solved with mod_perl, Apache::Registry or fastcgi.
• mod_perl / Apache::Registry does not seem to work out of the box at all. Fast_cgi does and it gives an acceptable performance.

Running mrtg-rrd with fast-cgi

• Install fastcgi: apt-get install libapache2-mod-fastcgi. This is in nonfree or something on Debian, and in multiverse on Ubuntu. Remember to force-reload apache.
• Enable fastcgi in mrtg-rrd.cgi script. It is all in the bottom of the script, you comment out some cgi stuff and comment in some fastcgi stuff instead.
• Rename the script to mrtg-rrd.fcgi and now use /cgi-bin/mrtg-rrd.fcgi/host1_1.html (and the option --rrdviewer=/cgi-bin/mrtg-rrd.fcgi to indexmaker).

Rather simple and works like a dream, except if you call it with a non existing target (and probably if you do anything else unpredictable). What happens if you do that is that fastcgi is disabled for 5 minutes for the entire apache server because the cgi script cannot be terminated correctly and thus misbehaves, seen from fcgi's point of view. At least the log says that fastcgi has been disabled, I have not tried running another fastcgi script during the 5 minutes to see if it worked...

The good news is that it does not take the webserver down, quite the opposite, it protects the webserver by closing down fastcgi, and load stays nice and low. No run-amok as you see if you try to run mrtg-rrd.cgi via Apache:Registry unmodified...

The bad thing is that this opens for a very easy denial of service of MRTG and any other fastcgi things on the same server. In my case this is all I use fastcgi for and my mrtg graphs and the fcgi script are password protected, so I can live with this little misfortune but it is not really very beautiful.