how to setup real-time email-notification for critical syslog events
it is often important for system administrators to get real time notification of critical events. unfortunately, it isn’t immediately obvious how to do this in the syslog framework. in this article i show you step-by-step how to do this.
as usual, all code and configurations have been tested on debian etch but should be useful for other *nix flavors with subtle modifications.
the syslog advantage
one of the big advantages of syslog is the separation between the log request and the logging action. for example, a shell script might contain a log request like:logger -p local0.crit “my pants are on fire”
critical
level to a facility local0
.without changing your script, you can configure syslog perform some or all of these actions:
- write this to a local logfile
- log this to the console
- write this to a remote logging server
- send an email / sms in real time
let’s focus on number 4. real-time notification is a good choice when your pants are on fire.
named-pipes
later versions of syslog have support for writing to named-pipes. a named-pipe is a special type of file that implements a simple fifo stream, allowing processes to talk to each other. we’ll exploit named-pipes to implement real-time messaging between syslog and our mailer.in our example, we’ll take all critical
messages written to the local0
facility and (in addition to logging) send them to the mail recipient, fireman@example.com
.
configuring syslog to write to a named-pipe
first, create a named-pipe for critical messages, for example:# mkdir /etc/syslog.pipes
# mknod /etc/syslog.pipes/criticalMessages p
# chmod 600 /etc/syslog.pipes/criticalMessages
critical
messages written to the local0
facility to this pipe. add the following statement to your syslog.conf
file.local0.crit |/etc/syslog.pipes/criticalMessages
sending out messages
the final step is to mail out any messages that are written to the pipe. you can do this with a simple shell script. i’ve included an example below, let’s call it/usr/bin/syslogMailer
:#!/bin/bash
# syslogMailer: a script to read stdin and turn each line into an alert
# email typically this is used to read a named-pipe written to by syslog
#
# example usage: syslogMailer < /etc/syslog.pipes/criticalMessages
#
alertRecipient=”fireman@example.com” # the mail recipient for alerts
TMOUT=1 # don’t wait > 1 second for input
# process each line of input and produce an alert email
while read line
do
# remove any repeated messages
echo ${line} | grep “message repeated” > /dev/null 2>&1
if test $? -eq 1
then
# send the alert
echo “${line}” | mailx -s “critical error on syslog” ${alertRecipient}
fi
done
daemon vs cron?
you’ll notice that i’ve included the following line in the script:TMOUT=1 # don’t wait > 1 second for input
read
. the script therefore runs to completion after processing one batch of zero or more messages. this allows you to schedule it in cron to run, say, every 5 minutes with a statement like:# m h dom mon dow command
0-59/5 * * * * /usr/bin/syslogMailer < /etc/syslog.pipes/criticalMessages > /dev/null 2>&1
# process each line of input and produce an error message
while :
do
while read line
do
[…]
# send the alert
echo “${line}” | mailx -s “critical error on syslog” ${alertRecipient}
done
done
the daemon approach is a little more efficient and sends out emails synchronously. it has the disadvantage that if your daemon terminates unexpectedly, alerts will stop until the daemon is restarted. the cron based implementation is arguably more robust in this regard. the cron approach also allows you to batch up notifications into n minute chunks. 5 minutes in our example cron file above
Exim4
Exim4 is another Message Transfer Agent (MTA) developed at the University of Cambridge for use on Unix systems connected to the internet. Exim can be installed in place of sendmail, although the configuration of exim is quite different to that of sendmail.
설치
exim4를 설치하기 위하여, 다음 명령을 실행 합니다:
sudo apt-get install exim4 exim4-base exim4-config
설정
exim4를 설정하기 위하여, 다음 명령을 실행 합니다:
sudo dpkg-reconfigure exim4-config
사용자 인터페이스가 보여질 겁니다. 사용자 인터페이스는 여러분이 많은 파라미터를 설정할 수 있도록 합니다. 예를 들어, exim4 설정 파일은 여러 개의 파일들로 나누어 집니다. 만약 여러분이 그것들을 한 파일로 갖기를 원한다면 이 사용자 인터페이스에서 적절하게 설정할 수 있습니다.
사용자 인터페이스에서 여러분이 설정한 모든 파라미터들은 /etc/exim4/update-exim4.conf.conf 파일에 저장됩니다. 만약 재설정을 하고자 한다면, 설정 마법사를 재 실행하거나 여러분이 좋아하는 편집기를 사용하여 이 파일을 손수 편집합니다. 여러분이 설정한 후에, 주 설정 파일을 생성하기 위하여 다음 명령을 실행할 수 있습니다:
sudo update-exim4.conf
주 설정 파일은 만들어지고 그것은 /var/lib/exim4/config.autogenerated 에 저장됩니다.
언제든, 여러분은 주 설정 파일, /var/lib/exim4/config.autogenerated을 수작업으로 편집하지 않습니다. 그것은 update-exim4.conf를 실행할 때마다 자동적으로 업데이트 됩니다.
여러분은 exim4 데몬을 시작하기 위하여 다음 명령을 실행할 수 있습니다.
sudo /etc/init.d/exim4 start
TODO: 이 부분에서는 exim4에 SMTP AUTH 설정하는 것을 다루어야 합니다.