Zabbix v6 Creating triggers for Baseline monitoring and Anomaly detection

• Anomaly Detection
• Baseline Detection
  • Baseline Trigger
  • Deviation Count

Taking a look at how to monitor deviations between seasonal data, calculate baseline values for your metrics and detect anomaly rates with the new Zabbix 6.0 LTS Machine learning functions.

Anomaly Detection

Go to Configuration and Templates. Select the Linux by Zabbix Agent and click on Triggers:

Here we need to Create a Trigger - for example if the CPU utilization spikes 2-times in a day we want to be notified. The expression seen below can be constructed by clicking on Add:

Here I am using system.cpu.util as a data point. I want to analyze the data of the last 28 days. The time window for the detection is 1 day and I am expecting to see some seasonality over 1 week - e.g. the webserver traffic always spikes on mondays:

Add the trigger and verify that it was created by switching to the Items tab:

Baseline Detection

There are two algorithm that we can choose from here. The first one returns the baseline and the second the number of deviations from that baseline over a set period:

• baselinewma - Calculates the baseline by averaging data from the same timeframe in multiple equal time periods ('seasons') using the weighted moving average (WMA) algorithm.
• baselinedev - Returns the number of deviations (by stddevpop algorithm) between the last data period and the same data periods in preceding seasons.

We can use both as triggers for our server monitoring.

Baseline Trigger

Create a new trigger that is called when the average memory utilization is 2-times higher than the calculated baseline over the last 4 weeks. Again, we can use the constructor to create the expressions and than add some math to it:

Calculate the baseline over 4 weeks with a 1 week periodicity and 8h time windows:

And get the average trend for the last 8h to compare with the calculated baseline:

Add the trigger and verify that it was created on the Items tab:

Deviation Count

Instead of comparing the baseline to the average value we can also detect direct deviations from the baseline. For example if the number of deviations in sent data from our webserver is greater than 0 give us an alert:

Again, we can use the constructor to create the expression for us: