Zabbix Crontab Monitoring

I wanted an easy way to monitor whether my linux crontabs were executed successfully. I came up with the following solution

High Level Overview

1. Create zabbix trapper items and triggers for every crontab entry. The item value is either ERRORor OK . The trigger fires if the item value is ERROR .
2. Create a wrapper script that executed the actual script and sends ERRORor OK to zabbix depending on the script exit code (0 for success, everything else for error)
3. Call the wrapper script in the crontab
4. Have some extra advantages like individual log files for every crontab entry and log rotation out of the box

Dependencies

The following commands need to be installed so that the scripts below run without problems:

• ts: Piped output get pretended with a timestamp
• zabbix_sender: Used to send values of trapper items into zabbix

Here are the install commands for debian based operating systems:

apt install moreutils
apt install zabbix-sender

Create zabbix items and trigger for every crontab entry

1. On the linux host i placed a new file under `/etc/zabbix/zabbix_agent2.d/cron_discovery` with the following content. Every line in the file corresponds with one crontab entry, one item and trigger in zabbix.

restic-cache-clear
restic-forget
paperless-backup
firefly-backup

2. Create a zabbix template named “Crontab Reporting” with this discovery rule:

Key: vfs.file.contents[/etc/zabbix/zabbix_agent2.d/cron_discovery]

3. Add a JavaScript preprocessing step:

var elements = value.split('\n')

var list = [];

for (var i = 0; i < elements.length; i++) {
  list.push({"{#NAME}": elements[i]})
}


return JSON.stringify(list);

4. Add a item prototype:

5. Add a trigger prototype:

Expression: `last(/Crontab Reporting/trap[{#NAME}])=”ERROR”`

6. Add the script to a Host (e.g. `test-server-1`) and run the Discovery Rule.

Send ERROR or OK from crontab to zabbix

Create the Script `/usr/local/bin/zabbix_cron_executor`:

#!/bin/sh
  
# variables
IDENTIFIER=$1
COMMAND="$2"

# execute command and get result
bash -l -c "$COMMAND"
RESULT=$?

# parse status
if [ $RESULT -eq 0 ]; then
  STATUS=OK
else
  STATUS=ERROR
fi

# send status to zabbix
echo "Sending status $STATUS to zabbix!"
zabbix_sender -z <Zabbix-Serverhost> -s <Localhost-Name> -k trap[$IDENTIFIER] -o $STATUS >> /dev/null

Replace `<Zabbix-Serverhost>` and `<Localhost-Name>` in the script with actual values (e.g. `zabbix.testdomain.de` and `test-server-1`)

Create the Script `/usr/local/bin/zabbix_cron`:

#!/bin/sh

# prepare log folder
mkdir -p /var/log/cron

# variables
IDENTIFIER=$1
LOGFILE="/var/log/cron/$IDENTIFIER.log"
COMMAND="$2"


# Execute command and send result to zabbix
zabbix_cron_executor $1 "$2" 2>&1 | ts "[%Y-%m-%d %H:%M:%S]" >> $LOGFILE


# Autotruncate Logfile
REWRITE_AT_LINES=2000
TRUNCATE_TO_LINES=1000

if [ $(wc -l < $LOGFILE) -ge $REWRITE_AT_LINES ];
then
  echo 'Do truncate...' | ts "[%Y-%m-%d %H:%M:%S]" >> $LOGFILE
  tail -$TRUNCATE_TO_LINES $LOGFILE > $LOGFILE.1
  mv $LOGFILE.1 $LOGFILE
fi

Make the Scripts executable:

chmod u+x /usr/local/bin/zabbix_cron
chmod u+x /usr/local/bin/zabbix_cron_executor

Call the wrapper script

Example Crontab Entry:

0 8–20/2 * * * zabbix_cron paperless-backup "python3 /root/scripts/backupPaperless.py"

Explaination:

1. The crontab executes the wrapper script
2. A unique crontab entry name (here “paperless-backup”) is passed as the first argument. That name has to be contained in the discovery file see above
3. The second parameter is the script that will be executed. If the exit code is greater zero, ERROR will be sent to zabbix and a trigger will alert.

Some extra benefits of using the wrapper script

1. Every crontab entry has a seperate log file location under /var/log/cron/$IDENTIFIER.log
2. Every logline has a timestamp prefix. e.g. [2023–05–01 00:01:06] Performing backup…
3. Log rotation is also included. If the file exceeds 2000 lines the log file will be truncated to the last 1000 lines.