Skip to content

Accessing Grafana

When accessing to the Grafana instance you will be redirected to the CERN SSO in order for you to authenticate. The central Grafana instance is configured in a way that it will allow people to access it using up to a lightweight account level of permission.

The first time you access monit-grafana you will end up in a "WELCOME" organization, and there will be a small delay (up to 5 minutes) for you to be synchronised into the organizations marked as "public". Once that's happened you can select the organization you want to check in the Grafana top right menu.

If you were also added to a non public organization using the e-group synchronisation the delay might be up to 30 minutes for you to be synchronised.

Grafana Proxy

If instead of using the web UI, what you would like to is to get programmatic access or simply enough query the data beneath any Grafana datasource the recommendation is to use the Grafana proxy API.

In order to do so, you should request a service account token to the organization admins holding the datasource you would like to query (or ask the MONIT team to put you in contact with them).

Important notes:

  • The data source id can be retrieved from Grafana by inspecting the actual queries run by Grafana in the browser. You can also ask an organisation admin to provide it to you, it can be retrieved on the datasources page for the organisation.
  • In case the request is done via CLI (e.g. curl, wget), hence without a valid session cookie, a Grafana API Token should be used for authentication. The token has to be created by the organization admin and specified as HTTP header of the request.

Prometheus

In order to query Prometheus, you will have different endpoints that can be used, you will mainly want to use /query or /query_range. You will need to replace desired_endpoint by any of those, more information available in the Prometheus API docs.

curl -X POST -H 'Content-Type: application/json' -H "Authorization: Bearer TOKEN_HERE" "https://monit-grafana.cern.ch/api/ds/query/uid/7617/api/v1/<desired_endpoint>" --data-binary "@query_file

The query file must follow this format:

query=max_over_time(container_memory_usage_bytes{pod_name=~"somepod-.*"}[1d])

Elasticsearch/OpenSearch

In order to query ES/OS the query should point to the /_msearch endpoint as in the following example:

curl -X POST -H 'Content-Type: application/json' -H "Authorization: Bearer TOKEN_HERE" "https://monit-grafana.cern.ch/api/datasources/proxy/uid/7617/_msearch" --data-binary "@query_file"

The query must follow this format: - First line describes the indices to use - Second lines contain the actual query

{"search_type": "query_then_fetch","ignore_unavailable": true,"index": ["monit_prod_wmagent_*", "monit_prod_wmagent_*", "monit_prod_wmagent_*"]}
{"query" : {"match_all" : {}}, "from" : 0, "size" : 10}

Please refer to the Elasticsearch _msearch documentation for more information.

InfluxDB

In order to query InfluxDB you will need to point to the /query/ endpoint and specify the database in the query using the db parameter as in the following example:

curl -X POST -H "Authorization: Bearer TOKEN_HERE" "https://monit-grafana.cern.ch/api/datasources/proxy/uid/7731/query?db=monit_production_cmsjm" --data-binary "@query_file"

The query file must follow this format:

q=select * from condor where time > now() - 1h limit 1