Use Case: TigerConnect at Scottsdale Health Partners


Oct 17, 2013 TigerText

Written by: Joseph Fontes, MSCIS at Scottsdale Health Partners

Last night while I was jogging just after sunset, I received a TigerText message on my phone. I saw the sender and immediately knew this was a message I had to read before continuing on. An alert on my phone indicated that one of the hard drives on a server in my organization was more than 75% full. Getting alerts like this is not completely out of the ordinary; as a hands-on Enterprise Architect working for a small, agile company, I like to stay on top of our system status. Before anyone could notice that there was a server issue, before the disk had a chance to fill and long before this could cause an outage, I had received a TigerText message from the systems alerting me to the growing problem. This is the life of technology staff: the life of invisible alerts and silent responses, the reason why systems “just work”. The statement typically holds true for all of our technology staff, “if an end user knows there is an issue with the system, we are already working on it”. Suffice to say, we hold our notifications in high regard and rely on them to notify us in the case of the slightest systems blip or network hiccup.

Our organization utilizes Nagios (the Icinga fork specifically) for systems and network monitoring. One of the first use cases for the TigerText API we identified was in alerting for technology monitoring. By default, Icinga/Nagios utilizes SMTP (Email) for transmission of information regarding systems monitoring. The standard implementation for emergency alerts (critical system down) is typically to use SMTP to email an address that is associated with a text message capable mobile phone number. For example, to have text messages sent to a Verizon cell phone number, we would modify the configuration for each user contact and set their the email address to @vtext.com

While the text message capability has worked historically, the implementation typically requires the modification of firewall rules as we have to allow outbound SMTP port (port 25) and/or configure SMTP email relay capabilities to allow the Nagios/Icinga server to relay messages through a corporate email server to the outside world. These modifications can delay implementation and open the network to other pitfalls such as allowing network computers outside of the dedicated email servers to send messages outside of the company on email server ports and protocols. The option we will discuss in this posting, however, allows for the notification of standard and emergency alerts via a secure and (most likely) already available connection.

The TigerText API uses standard secure communications for the sending of these messages (HTTPS). The same protocol and method used to access online banking, secure health information, and combat identity theft can be used for application alerting.

While a Nagios plugin can be written in most any language, we chose to write ours in PHP as the Nagios server doubles as a web application server. We have attached and made available the script for use and modification (see attached).

 

With our distribution of choice being CentOS (Red Hat Enterprise Linux derivative), we installed Nagios via YUM which places the configuration files by default in /etc/nagios/objects/ (/etc/icinga/objects/) with the main Nagios configuration located in /etc/nagios/nagios.cfg (/etc/icinga/icinga.cfg).

Throughout this posting, the Nagios directory structure will be interchangeable with the Icinga structure simply substituting the names as the default Icinga configuration directory is located in /etc/icinga/objects/ with the main Icinga configuration file being /etc/icinga/icinga.cfg

Once this file has been copied to your server, I recommend placing it in your custom plugins folder (if you have one) or in the default Nagios location (/usr/lib64/nagios/plugins). For the remainder of the article, we are going to assume you have placed the file in the standard Nagios plugin directory and thus need to allow Nagios to run the plugin with the following commands:

reload the config files

Additionally, for SELinux operability, we need to set context for the plugin:

set context for the plug-in

Once you have the proper permissions set on the file, it is time to edit the plugin script. Set the variables for $key and $secret to those provided by TigerText support as seen here:

key and secret

After the update has been made, you can test the plugin functionality by issuing the following command:
php /usr/lib64/nagios/plugins/notify_tigertext.php -u user@domain.com -H “HOSTNAME” -a “HOSTALIAS” -s “SERVICESTATE” -o “SERVICEOUTPUT” -d “SERVICEDESC” -t “SERVICE”

email alert

In the example above, replace the address user@domain.com with the appropriate email address to alert.

Once the plugin has been tested and functionality has been confirmed, it is time to begin the integration with Nagios/Icinga. The following attachment named “tigertext.cfg” allows the use of the TigerText PHP plugin to send alerts. The file assumes a default location for the plugin as described above. If the location is not the standard as illustrated, you must modify the “command_line” argument and point the configuration file to the absolute path of the plugin.

modify the command line

Once the file has been created, you need to include it in the Nagios/Icinga main configuration file so that it is used upon application restart. Modify /etc/nagios/nagios.cfg or /etc/icinga/icinga.cfg and add the include line as illustrated below:

add the included line

The next configuration file that must be modified (contacts.cfg) is illustrated below:

modify contacts-cfg

Notice the modification of the variables “service_notification_commands” and “host_notification_commands”. We are setting these variables to the values of the “command_name” variables from the tigertext.cfg configuration file.

Once the updates are complete, test the configuration changes to insure there are no issues with the modifications as follows:

icinga-v-1

reload the config files

Once the server states that there are no issues with the configuration, you can now reload the configuration files:

reload the config files

Now it is time to insure that the alerts are processing correctly. We are going to force an alert to check the status of the plugin. From the Icinga/Nagios web interface, select “Service Detail” from the left column and then choose any of the services on the screen. Once you have selected a service, you can then select “Send custom service notification” from the “Service Commands” section as illustrated below.

service commands section 1

service commands section 2

service commands section 3

Once this command has been sent, it should take less than one minute before the command is issued and a notification makes it to your TigerText device/browser. After the plugin has been implemented, Nagios/Icinga can be configured to only alert to TigerText for emergencies (so one does not become overwhelmed with alerts at random hours) or set the alerts to only send on scheduled Nagios/Icinga time periods. There are numerous directions and options available for all facets of notification customization. I encourage those needing to modify the monitoring system to meet your needs to sign up for updates to both the open source iterations of Nagios and Icinga as well as to pay particularly close attention to their associative documents.

The decision to utilize the TigerText API for systems/network monitoring and alerting was an obvious choice for our organization. The increase in security over traditional alerting solutions (email, email to text) coupled with the ease of implementation has provided an excellent return on investment. The Android and Apple iOS apps allow us to receive notifications anywhere we have mobile service or access to Wi-Fi. The desktop notification application alerts us when we are logged into a workstation and the TigerText Pro web interface gives us a full screen viewing solution when utilizing any personal computer with an internet connection.