Skip to content

WLCG currently supported flows

Here you will find all the flows that the Monitoring team provides currently for WLCG and some information on the datasets and dashboards (if centrally managed) for all of them.

Sitemon

More information about the sitemon flow can be found in the specific website.

WLCG Transfers

WLCG transfers flows include FTS and XRootD monitoring cases, these are use to compute a WLCG view of the data transfers happening in the GRID.

FTS

  • Raw data responsible: FTS team
  • Integration: AMQ (Messaging)
  • Flows:
Input Processing type Storage Index Pattern Dashboards
State None OpenSearch monit_prod_fts_raw_state None
Start None OpenSearch monit_prod_fts_raw_start None
Queue State None OpenSearch monit_prod_fts_raw_queue_state None
Complete Enrichment OpenSearch monit_prod_fts_enr_complete None
Complete Aggregation OpenSearch LT monit_prod_fts_agg_complete FTS transfers

XRootD

XRootD data is currently split into three different flows: Monalisa (deprecated), XRootD NG and dCache

  • Raw data responsible:
  • Monalisa:: MONIT team
  • XRootD NG: US: OSG, Other: MONIT team
  • dCache: Site administrator
  • Integration: AMQ (Messaging)
  • Flows:
Input Processing type Storage Index Pattern Dashboards
Monalisa Enrichment OpenSearch monit_prod_xrootd_enr_transfer None
Monalisa Aggregation OpenSearch LT monit_prod_xrootd_agg_transfer XRootD transfers
XRootD NG Enrichment OpenSearch monit_prod_xrootdng_enr_transfer None
XRootD NG Aggregation OpenSearch LT monit_prod_xrootdng_agg_transfer XRootD transfers
dCache Aggregation OpenSearch LT monit_prod_dcache_agg_transfer XRootD transfers

Rucio

There are several flows related to Rucio monitoring, from DDM in ATLAS and Rucio enrichment for CMS to accounting information.

ATLAS DDM

Transfers

In the case of DDM transfers, traces and events are combined together to generate the full transfers information.

  • Raw data responsible: ATLAS Rucio team
  • Integration: AMQ (Messaging)
  • Flows:
Input Processing type Storage Index Pattern Dashboards
Traces, Events Enrichment OpenSearch monit_prod_ddm_enr_transfer None
Traces, Events Aggregation OpenSearch LT monit_prod_ddm_agg_transfer None

Queued

  • Raw data responsible: ATLAS Rucio team
  • Integration: AMQ (Messaging)
  • Flows:
Input Processing type Storage Index Pattern Dashboards
Queued Enrichment OpenSearch LT monit_prod_ddm_enr_queued None

Accounting

  • Raw data responsible: ATLAS Rucio team
  • Integration: AMQ (Messaging)
  • Flows:
Input Processing type Storage Index Pattern Dashboards
Global None OpenSearch LT monit_prod_rucioacc_raw_global None
Site Enrichment OpenSearch LT monit_prod_rucioacc_enr_site None
Storage Enrichment OpenSearch LT monit_prod_rucioacc_enr_storage None

CMS Rucio

Transfers

  • Raw data responsible: CMS Rucio team
  • Integration: AMQ (Messaging)
  • Flows:
Input Processing type Storage Index Pattern Dashboards
Traces None OpenSearch monit_prod_cms_rucio_raw_tracer None
Events Enrichment OpenSearch monit_prod_cms_rucio_enr_events None
Daily Stats None OpenSearch monit_prod_cms_rucio_raw_daily_stats None
Weekly Stats None OpenSearch LT monit_prod_cms_rucio_raw_weekly_stats None
Monthly Stats None OpenSearch LT monit_prod_cms_rucio_raw_monthly_stats None

ATLAS Job Monitoring

In the case of ATLAS job monitoring, we do integrate 2 different flows (Panda and TZero) by reading data from the Oracle database handling it and merge it into a single output. These flows are split into the following categories: Submitted, Completed, Current (Pending, Running and Finalising). Please note the data categorised as "current" is generated by taking periodic snapshots of the DB state, which makes it non recoverable by definition.

  • Raw data responsible: MONIT team
  • Integration: Oracle Connector
  • Flows:
Input Processing type Storage Index Pattern Dashboards
Panda Completed Enrichment OpenSearch monit_prod_atlasjm_enr_completed None
Panda Aggregation OpenSearch LT monit_prod_atlasjm_agg_(jobstatus) None
TZero Completed Enrichment OpenSearch monit_prod_atlasjm_enr_completed None
TZero Aggregation OpenSearch LT monit_prod_atlasjm_agg_(jobstatus) None

CMS Job Monitoring

  • Raw data responsible: CMS Condor team
  • Integration: AMQ (Messaging)
  • Flows:
Input Processing type Storage Index Pattern Dashboards
Jobs Aggregation OpenSearch LT monit_prod_condor_agg_metric None