Zabbix Alerting

Cutting alert noise in Zabbix: six settings that matter

When people stop reading alerts, monitoring has stopped working. Six Zabbix features that turn a flood of notifications into a short list of real problems.

Alert fatigue is rarely a people problem. It is a configuration problem. When a single outage produces dozens of notifications, or the same problem opens and closes all night, people learn to ignore alerts, and the one that matters gets missed.

Zabbix has the tools to fix this. These are the six we reach for first.

1. Trigger dependencies

When a core switch or an uplink fails, every host behind it becomes unreachable, and each one raises its own problem. The real cause is buried in the noise.

A trigger dependency tells Zabbix that one problem only makes sense if another is not already active. Make the host availability triggers depend on the switch or uplink they sit behind, and an outage raises one problem: the switch. Model dependencies along the real network path, not the way hosts happen to be grouped.

Zabbix trigger list: the ping triggers of dist-sw-01 and dist-sw-02 depend on core-sw-01 being reachable
The two distribution switches sit behind the core switch, so their availability triggers depend on it. If the core switch fails, its problem is raised and theirs are held back.

2. Separate problem and recovery conditions

A value hovering around a threshold makes a trigger flap: problem, resolved, problem again, many times an hour.

Give the trigger a recovery expression that is different from the problem expression. For example, raise the problem when disk usage goes above 90% and only resolve it when it drops below 85%. The gap between the two is what stops the flapping.

Zabbix trigger form with problem expression disk usage above 90 and recovery expression disk usage below 85
A problem above 90% that only resolves below 85%.

3. Alert on sustained values, not single samples

A single high reading is often just a moment. Trigger functions that look at a period of time, such as the minimum or average over the last five minutes, only fire when the condition has held.

min(/server/system.cpu.util,5m)>90 means “above 90% for five minutes”, which is usually what you meant in the first place.

Zabbix CPU triggers on four Linux servers using min of CPU utilisation over five minutes
Zabbix's own Linux template uses the same pattern: CPU utilisation has to stay above the limit for five minutes.

4. Maintenance periods

Planned work should not page anyone. Maintenance periods tell Zabbix that hosts are being worked on, for a fixed time or on a schedule, either still collecting data or not. Put patch windows and planned reboots into maintenance, and the alerts that do arrive during those hours mean something.

Zabbix maintenance period Monthly patching: Linux servers, with data collection, every month on the first Sunday at 10:00 PM for 4 hours
A monthly patch window: first Sunday of every month, four hours from 10 PM, with data still collected.

5. Tags and tag-based actions

Tag triggers with the service, the team and the environment they belong to. Actions can then route notifications by tag, so the database team hears about database problems and production alerts are handled differently from test ones.

Tags also make problem views, dashboards and reports far easier to filter.

Zabbix trigger actions: Database problems to the database team, with conditions severity at least Warning and tag service equals Service desk database
An action that sends problems tagged with the service desk database to the database team.

6. Event correlation

Some problems are only symptoms of another one, or newer versions of a problem that is already open. Global event correlation matches events by their tags and can close the new problem, or the old one, automatically. It is the tool for patterns that dependencies alone cannot describe.

Zabbix event correlation rule Keep one open problem per service: when a new event has the same service tag value as an old one, close the old event
A correlation rule that keeps one open problem per service by closing the older one.

Then measure

Noise comes back if nobody looks. Zabbix’s report of the most frequently triggered problems is the best place to start: review it every month, and fix or retire the triggers at the top. A small number of triggers usually causes most of the noise.

Zabbix Top 100 triggers report for the last 30 days, led by service and host restart triggers
The Top 100 triggers report over 30 days. The triggers at the top are the first to review.

Where GLPI fits

Once alerts are few and accurate, they are worth turning into tickets. Zabbix includes an official GLPI webhook that opens a ticket for each real problem, adds follow-ups as it changes and marks it solved on recovery, so every problem has an owner and an SLA.

If your Zabbix has become too noisy to trust, we can review it with you and bring the alert volume down to what your team can act on.

Putting this into practice?

If you would like a second pair of eyes on your Zabbix, start with a short call and we will help you plan the next step.