Skip to content

Grafana Alerts

As part of the central Grafana service, organization admins have the possibility to configure alerts for the datasources provided by their orgs.

In order to understand better how Grafana alerting works we recommend you to have a look to the official documentation.

GNI integration

As part of the GNI service, we have added an integration with Grafana alarms allowing for example to forward tickets to CERN SNOW.

The first thing you will require is to create a contact point so your alarms are forwarded to the MONIT infrastructure endpoint:

SNOW integration

Since in the case of Grafana it's not possible for the monitoring infrastructure to add the correct SNOW information by default, an user interactive way is provided.

As a service manager, when creating the alarm you will need to specify the SNOW information as well as some extra required labels.

  • (mandatory) gni_entities - Defines the name of the entity the alert is triggered for.
  • Hint: For multi-instance alerts you can set the label value to existing tag/label from the query: e.g. {{$labels.host}}
  • (optional) gni_alertname - This value is set by default in Grafana to the "Rule name". You can overwrite if different name is required in GNI.
  • (mandatory) snow_functional_element - SNOW Functional Element. This field is required for assigning the alarm to the correct FE
  • (optional) snow_assignment_level - SNOW assignment level (default: 3)
  • (optional) snow_watchlist - Comma separated list of the emails to be added into the watchlist of the ticket.
  • (optional) snow_auto_closing - Should the ticket close in case the alert is resolved (default: false)
  • (optional) snow_troubleshooting - Troubleshooting information.

Since dynamic labels will create new instances of your alert, for dynamic values like "correlation" you should actually add them as a custom annotations:

  • (optional) gni_correlation - The correlation value to be added in the GNI ticket.
  • Hint: You can point to a given query/condition value by using {{$values.A.Value}} for setting your annotation, where "A" is the name of the query/condition you defined in the section above.
  • (optional) gni_description - The description added in the GNI ticket.

Are my alerts being forwarder properly?

If you are using the GNI webhook, all your alerts will also arrive into our OpenSearch cluster, in two different indices.

  • monit_prod_alarm_raw_gni: Documents formatted as GNI schema, available only if the gni_entities has been specified in your alert configuration
  • monit_prod_alertmanager_raw_alerts: Raw documents as producer by alertmanager