Disabling hyper-threading

[AntonMFernando-NOAA]

Description

• Hera, Hercules, and Orion (and possibly Jet) enable hyper-threading by default. This should be disabled explicitly by adding the sbatch/srun flag --hint=nomultithread.
• Resolves Explicitly disable hyper-threading on Hera, Hercules, and Orion #2863

Type of change

• Bug fix (fixes something broken)
• New feature (adds functionality)
• Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

• Is this a breaking change (a change in existing functionality)? NO
• Does this change require a documentation update? NO
• Does this change require an update to any of the following submodules? NO (If YES, please add a link to any PRs that are pending.)

How has this been tested?

• Will be tested on HERA, HERCULES, and ORION

Checklist

• Any dependent changes have been merged and published
• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have documented my code, including function, input, and output descriptions
• My changes generate no new warnings
• New and existing tests pass with my changes
• This change is covered by an existing CI test or a new one has been added
• I have made corresponding changes to the system documentation if necessary

[AntonMFernando-NOAA]

@DavidHuber-NOAA GEFS test results. Some of the tests are still running. Will update once the tests are completed.

	ORION	ORION	HERCULES	HERCULES	HERA	HERA
	h/t off	h/t on	h/t off	h/t on	h/t off	h/t on
stage_ic	55	54	42	36	26	29
wave_init	45	44	29	30	31	38
prep_emiss	46	46	36	39	22	22
fcst_m1	1258	1254	934	936	1182	1186
fcst_m2	3012	3015	2274	2266	2873	2879
fcst_m3	3024	3007	2260	2273	2840	2848
atmos_prd_mem	139	146	95	93	110	110
atmos_enstat	34	34	20	21	39	27
ocn_prod_mem	62	39	27	48	33	29
ice_prod_mem	54	47	18	45	22	20
wave_post_grid	427	394	247	248	440	359
wave_post_pnt	4411	4364	7078	4063	4540	4445
cleanup	45	42	17	18	21	21

[AntonMFernando-NOAA]

GFS test results

	ORION	ORION	HERCULES	HERCULES	HERA	HERA
	h/t off	h/t on	h/t off	h/t on	h/t off	h/t on
gfsstage_ic	58	48	45	45	22	31
gfsfcst	2903	2914	2185	2184	2750	2786
gfsatmos_prod	161	181	113	110	125	129
gfstracker	-	-	-	-	68	72
gfsgenesis	-	-	-	-	460	449
gfsmetp	-	-	-	-	84	87
gfsarch	96	96	64	74	158	121
gfscleanup			26	30	27	27

[DavidHuber-NOAA]

@DavidHuber-NOAA GEFS test results. Some of the tests are still running. Will update once the tests are completed.

ORION ORION HERCULES HERCULES HERA HERA
h/t off h/t on h/t off h/t on h/t off h/t on
...
wave_post_pnt 4411 4364 7078 4063 4540
cleanup 45 42 17 18 21

@AntonMFernando-NOAA Would you mind running the wave_post_pnt job again with hyper-threading disabled on Hercules just to verify that runtime?

[AntonMFernando-NOAA]

Did a rerun with h/t disabled on HERCULES and wave_post_pnt took 3197s.

[AntonMFernando-NOAA]

@DavidHuber-NOAA cyc test results

	HERCULES	HERCULES
	h/t on	h/t off
gdasstage_ic	44	44
gdasfcst	323	322
gdasatmos_prod	66	66
enkfgdasechgres	42	27
enkfgdasstage_ic	19	15
enkfgdasfcst	258	243
enkfgdasepmn	36	37
gdasprep	137	136
gdasanal	921	917
gdassfcanl	37	39
gdasanalcalc	76	88
gdasanaldiag	118	121
gdasatmanlupp	51	51
gdasatmanlprod	68	64
gdasfcst	407	397
gdasatmos_prod	67	69
gdasfit2obs	17	17
gdasverfozn	53	43
gdasverfrad	665	576
gdasvminmon	22	20
gdasarch	45	42
gdascleanup	176	26
enkfgdaseobs	426	416
enkfgdaseupd	147	153
enkfgdasechgres	39	33
enkfgdasediag	137	135
enkfgdasecmn	47	37
enkfgdasesfc	115	118
enkfgdasfcst	330	309
enkfgdasepmn	31	30
enkfgdaseamn	15	16
enkfgdascleanup	158	26
gfsprep	132	127
gfsanal	561	540
gfssfcanl	40	40
gfsanalcalc	87	93
gfsatmanlupp	40	39
gfsatmanlprod	110	111
gfsfcst	3134	3115
gfsatmos_prod	111	110
gfsvminmon	112	20
gfsmetp	335	101
gfsarch	56	73
gfscleanup	23	27
gdas_prep	115	112
gdas_anal	1069	1060
gdas_sfcanl	40	42
gdas_analcalc	75	96
gdas_analdiag	117	121
gdas_atmanlupp	62	46
gdas_atmanlprod	66	66
gdas_fcst	401	395
gdas_atmos_prod	67	66
gdas_fit2obs	178	18
gdas_verfozn	297	40
gdas_verfrad	1333	671
gdas_vminmon	132	17
gdas_arch	19	19
gdas_cleanup	22	22
enkfgdas_eobs	422	413
enkfgdas_eupd	140	142
enkfgdas_echgres	30	35
enkfgdas_ediag	133	130
enkfgdas_ecmn	37	47
enkfgdas_esfc	115	117
enkfgdas_fcst	313	305
enkfgdas_epmn	33	31
enkfgdas_eamn	16	16
enkfgdas_cleanup	21	23

[AntonMFernando-NOAA]

@DavidHuber-NOAA The tests are complete. Any comments?

[DavidHuber-NOAA]

It's interesting that the cleanup jobs sped up without hyperthreading as these do not issue srun commands, so there shouldn't be any difference. The other jobs that show some differences are also interesting (fit2obs, verfozn, verfrad, and metp). I think MET may make use of hyperthreading, but I will take a look at those logs. I doubt the verf or fit2obs jobs do, though.

Can you do a file comparison on the GRIB2 products as well to ensure they are all identical? Just a cmp should be fine. They should be in the ARCDIR for both the hyperthreaded and non-hyperthreaded cases.

[AntonMFernando-NOAA]

@WalterKolczynski-NOAA @DavidHuber-NOAA Checked the GRIB2 files with and without h/t in ARCDIRs and made sure that the files are identical. I think this is now ready to review.

[DavidHuber-NOAA]

Changes look good on Hercules. Disabling hyperthreading may have a speed boost for a few jobs, which is interesting. The METplus logs mention using up to 80 tasks/node with 8 threads for both the hyperthreaded and non-hyperthreaded case. This is both interesting and a little disturbing, but does not seem to tie into hyperthreading. I did not look into the other jobs that showed significant differences in runtime (Fit2Obs, verfozn, and verfrad).