Skip to content

Welcome to WLCG covered use cases

As part of the MONIT team work, we offer support for several WLCG use cases, this support might be from helping with integration of data to running processing jobs in order to transform the data in a way that can be consumed by the WLCG community.

In many cases most of these flows were setup early days of MONIT as part of integrating together two Monitoring entities, the DC monitoring and the WLCG monitoring. Since then MONIT man power has been drastically reduced and setting up of this flows is pushed to the actual users whenever possible.

In general all these processing labours take place as Apache Spark jobs running on top of the MONIT processing infrastructure (currently based on K8s jobs/services consuming and producing data from/to Kafka), you will be able to find a full list of jobs in the spark-jobs repository.

In terms of the supported jobs, there are two categories:

  • Enrichment: These jobs take care of adding extra information to the provided one by the producer, in general this means to add extra information from datasource like CRIC with extra topology information.
  • Aggregation: These jobs take care of aggregating data by a set of pre-defined labels in bigger buckets (normally 1h) in order to store it for long-term.

CRIC Topology information

As part of our processing flows, one of the most common cases is to extend data using CRIC information. In order to do that the Monitoring team collects CRIC information every 6h that stays static for that period of time (meaning that changes in CRIC might take up to 6h to be visible in the handled flows).

Also, we use WLCG CRIC as the source of truth for any flow that's WLCG related (Meaning specific experiments CRIC instances are not usually in use), so in case of doubt please check it to make sure the information you expect to see is updated there.

Requesting a new flow for WLCG to be handled by MONIT

As mentioned above, most of the use cases currently covered by MONIT were done and discussed as part of the unification of the two previous monitoring teams. Since then few new use cases have been discussed and accepted together with the MONIT team. However, any new use case that will be required will need to be discussed at a higher level to understand the resources needed and the department priorities towards it.

Anyhow, the first step will be to send us a SNOW ticket, with the details about the new case, we will then communicate you in the same ticket how to proceed in order to move the request forward.