Test sites if they're ready for stageout of el8 workflows

[haozturk]

Problem is explained here: https://cms-talk.web.cern.ch/t/urgent-el8-release-needs-fix-for-srm-protocol/18705/29

We'll perform the extensive tests on Monday.

I'm submitting some early tests to see if such tests are

• capable of reproducing the error
• short enough that they'll complete less than a day

I picked this short Run3 workflow for this test. It has 5 tasks (from GENSIM to NANO). I dropped everything except GENSIM to keep it short. Let's see how it'll go.

T2_UK_London_IC (Changed their config today, expecting no failures)

T1_FR_CCIN2P3 (Uses SRM, expecting failures)

T1_DE_KIT (Changed their config a few weeks ago, expecting no failures)

T2_BE_IIHE (Changed their config today, expecting no failures)

T2_UK_SGrid_RALPP (Uses SRM, expecting failures)

T2_DE_RWTH (Uses SRM, expecting failures)

[haozturk]

Dima suggested testing the full chain. Here are those workflows:

T2_UK_London_IC

T1_FR_CCIN2P3

T1_DE_KIT

T2_BE_IIHE

T2_UK_SGrid_RALPP

T2_DE_RWTH

[todor-ivanov]

Hi @haozturk I just looked at those workflows. They seem to be sitting in acquired, and from a first glance the reason seems to be no intersection between the sets of PU location sites and the workflow's SiteWhiteList. In addition the flag TrustPUSitelists is set to False. So the workflows are split by the Global Work Queue in WQ elements, but no agent is fetching those, because of the empty resulting set of sites to run at. Taking one of them shows it clearly:

https://cmsweb.cern.ch/reqmgr2/fetch?rid=haozturk_task_PPD-Run3Summer22EEwmLHEGS-Backfill-00001__v1_T_221216_175211_8731

During the GWQ splitting it resulted in a single WQ element: https://cmsweb.cern.ch/couchdb/workqueue/_design/WorkQueue/_rewrite/element/6339fd2aec93ee94e565a93bd4c5f81e

For which the PU locations are listed as:

2022-12-16 18:45:27,013:INFO:Rucio:Container: /Neutrino_E-10_gun/Run3Summer21PrePremix-Summer22_124X_mcRun3_2022_realistic_v11-v2/PREMIX with container-based location at: {'T1_US_FNAL_Disk', 'T2_CH_CERN'}

So for this to workflow, I believe, setting the TrustPUSitelists to True would help.

[haozturk]

Here are the test workflows:

T1_ES_PIC
T1_FR_CCIN2P3
T1_IT_CNAF
T1_RU_JINR
T2_AT_Vienna
T2_BE_IIHE
T2_BE_UCL
T2_CH_CERN_HLT
T2_CH_CSCS
T2_CN_Beijing
T2_DE_RWTH
T2_EE_Estonia
T2_ES_CIEMAT
T2_FI_HIP
T2_FR_GRIF_LLR
T2_FR_IPHC
T2_HU_Budapest
T2_IT_Bari
T2_IT_Legnaro
T2_IT_Pisa
T2_IT_Rome
T2_KR_KISTI
T2_PK_NCP
T2_PL_Swierk
T2_PT_NCG_Lisbon
T2_RU_IHEP
T2_RU_INR
T2_RU_ITEP
T2_RU_JINR
T2_TR_METU
T2_UK_London_IC
T2_UK_SGrid_RALPP
T2_US_MIT
T2_US_Purdue
T2_US_Vanderbilt