Nagios/Icinga email alerts using AWS Simple Email Service (SES)

The biggest problem with sending email from AWS EC2 instances is that -sooner or later- your instance’s IP will be added to a blacklist. It doesn’t matter how secure your MTA is. It doesn’t matter if it’s not reachable from the internet. And it doesn’t matter if you are not sending spam at all.

Sooner or later one of your neighbors using an IP on the same IP range your instance is, will send spam, Then one of the blacklists (such as Trendmicro’s) will add the whole range as spam sender.

In theory you could try setting up rDNS but that’s not always a warranty of staying out of the lists. And obviously what AWS recommends is using (and paying) Simple Email Service or SES. It’s pretty easy to setup and pretty easy to use (you can setup IAM accounts to use SMTP).

For some services the configuration will be easy: add the SES endpoint as SMTP server hostname, enable SSL, select TCP port 465, add your credentials, and ready to go.

But how to do it for Nagios or Icinga (version 1)?

Well, usually you’ll have one or more commands defined to send email alerts, such as:

define command {
command_name    notify-host-by-email
command_line    /usr/bin/printf “%b” “***** Icinga *****\n\nNotification Type: $NOTIFICATIONTYPE$\nHost: $HOSTNAME$\nState: $HOSTSTATE$\nAddress: $HOSTADDRESS$\nInfo: $HOSTOUTPUT$\n\nD
ate/Time: $LONGDATETIME$\n” | /bin/mail -s “** $NOTIFICATIONTYPE$ Host Alert: $HOSTNAME$ is $HOSTSTATE$ **” $CONTACTEMAIL$
}

Usually /bin/mail will be in fact /bin/mailx, so in theory you could configure it to use your IAM account to send email with a ${HOME}/.mailrc file.

The only problem is that when Nagios/Icinga calls a system command or a program it doesn’t open any shell interpreter, so there won’t be a ${HOME} variable available.

So let’s get started:

1. Grab Nagios/Icinga home directory from /etc/passwd (it is /var/spool/icinga for Icinga at CentOS/RHEL/Amazon Linux or /var/lib/nagios at Debian/Ubuntu)
2. Create a .mailrc password inside Nagios/Icinga directory with the following content:
   

   account ses {
   set smtp=smtps://<SES_ENDPOINT>:465
   set smtp-auth=login
   set smtp-auth-user=<ACCESS_KEY>
   set smtp-auth-password=<SECRET_KEY>
   set ssl-verify=ignore
   set nss-config-dir=<NSS_CONFIG>
   set from=<EMAIL_ADDRESS>
   }

   Where:

   <SES_ENDPOINT> is the AWS SES SMTP Endpoint according to your AWS config (remember that SES is region-based, you need to enable and configure it for each region you want to use).
   <ACCESS_KEY> is the SMTP access key for your IAM user.
   <SECRET_KEY> is the SMTP secret key for your IAM user.
   <NSS_CONFIG> is the NSS configuration directory (/etc/pki/nssdb/ for CentOS/RHEL/Amazon Linux, you can just remove the whole “set” directive for Debian/Ubuntu)
   <EMAL_ADDRESS> is the email address you are authorized to use (will depend on the verified addresses/domains and the security config for the IAM user

3. Edit /etc/sysconfig/icinga or /etc/sysconfig/nagios (CentOS/RHEL/Amazon Linux) or /etc/default/icinga (Debian/Ubuntu) and add the following content:
   

   export MAILRC=<ICINGA_OR_NAGIOS_HOME>/.mailrc

   Where <ICINGA_OR_NAGIOS_HOME> is the directory from step 1.

   This will add the variable MAILRC as environment variable available when you call /bin/mail from Icinga/Nagios so mailx is able to find the configuration.

4. Now edit all the commands used to send email alerts and add the parameter -A ses, so you have commands similar to this example:
   

   define command {
   command_name    notify-host-by-email
   command_line    /usr/bin/printf “%b” “***** Icinga *****\n\nNotification Type: $NOTIFICATIONTYPE$\nHost: $HOSTNAME$\nState: $HOSTSTATE$\nAddress: $HOSTADDRESS$\nInfo: $HOSTOUTPUT$\n\nDate/Time: $LONGDATETIME$\n” | /bin/mail -A ses -s “** $NOTIFICATIONTYPE$ Host Alert: $HOSTNAME$ is $HOSTSTATE$ **” $CONTACTEMAIL$
   }

5. Reload Icinga/Nagios, and you’re done.

Remember that it’s strongly advisable to setup DKIM and SPF so people can be sure that your email is legitimate, regarding the servers that are sending the emails (not the content)

NOTE 1: Another option is to configure the local MTA for the instance (sendmail, postfix, qmail…) to use SES as a relay. However this can be harder, and in the end you have yet another software that could fail.

NOTE 2: This procedure does not work if your are trying to send mail through sender authorization as delegated sender (sometimes known as SES Crossacounts), as it’s required to add X-Headers to each email (something I could not achieve with mailx). In this case you can verify individual email addresses belonging to another account.