GW-RT global_control.nml modifications

[dpsarmie]

Commit Queue Requirements:

• Fill out all sections of this template.
• All sub component pull requests have been reviewed by their code managers.
• Run the full Intel+GNU RT suite (compared to current baselines) on either Hera/Derecho/Hercules
• Commit 'test_changes.list' from previous step

---

There is now an option to allow default_vars.sh to set variables that are dependent on DT_ATMOS and ATMRES. If you wish to use a non-default value for either variable and have the script set all dependent variables automatically (ex. DT_INNER, CDMBGWD, namsfc filenames, etc.), then DT_ATMOS and ATMRES must be defined before calling the export functions.

If the user does not define the variable, then the defaults of C96 (ATMRES) and 1800 (DT_ATMOS for atm only) or 720 (DT_ATMOS for cpld cases) will be set.

Example:
Your RT uses C48 resolution and you want a timestep of 720:
In your test configuration:

export ATMRES=C48
export DT_ATMOS=720
export_fv3
export_ugwpv1
export_cpl
<the rest of the config>

This will now be able to use the user defined timestep and resolution to populate the other variables needed to run this case. As always, the order of the export function calls should go export_fv3, export_ugwpv1, then export_cpl. You can remove export function calls as needed for your particular case.

You are still able to use the old methodology of defining all variables manually. This feature just allows users to streamline that process.

Description:

This PR will modify the global_control.nml to add variables that are present in the Global Workflow scripts but not currently present in the RTs. Some variables will also be changed from constants to variable placeholders. This will allow Global Workflow to use the global_control.nml as a template for their operations.
Variable values that were hard-coded in global_control.nml.IN were replaced with an @ tag and the values were moved to default_vars.sh.

This PR is now also going to include a fix for #2443. This will add baseline changes that are unrelated to the initial global_control.nml modifications or the sfs configuration fix.
Default values for DT_ATMOS and ATMRES are now being set as intended. If the user defines either of these variables before the export_ function calls, then the appropriate dependent variables will be auto-configured. This removes the need to redefine these variables in the test configs.

This PR now includes a feature mentioned in #2470. The checks have been added to run_test.sh.
Changes in run_test.sh now have checks for the fast and slow coupling frequencies and a check on the CICE to ATM timesteps.

Commit Message:

* UFSWM - Modify global_control.nml to add compatibility with Global Workflow

Priority:

• Normal

Git Tracking

UFSWM:

• Closes Modify global_control.nml to work with GW scripts #2424
• Closes ATMRES getting reset in export_cpl() #2443
• Closes Add check for coupling timing between components #2470

Sub component Pull Requests:

• None

UFSWM Blocking Dependencies:

• None

---

Changes

Regression Test Changes (Please commit test_changes.list):

• cpld_control_p8_mixedmode intel
• cpld_control_gfsv17 intel
• cpld_control_gfsv17_iau intel
• cpld_restart_gfsv17 intel
• cpld_mpi_gfsv17 intel
• cpld_control_sfs intel
• cpld_debug_gfsv17 intel
• cpld_control_p8 intel
• cpld_control_p8.v2.sfc intel
• cpld_restart_p8 intel
• cpld_control_qr_p8 intel
• cpld_restart_qr_p8 intel
• cpld_2threads_p8 intel
• cpld_decomp_p8 intel
• cpld_mpi_p8 intel
• cpld_control_ciceC_p8 intel
• cpld_control_c192_p8 intel
• cpld_restart_c192_p8 intel
• cpld_bmark_p8 intel
• cpld_restart_bmark_p8 intel
• cpld_s2sa_p8 intel
• cpld_control_noaero_p8 intel
• cpld_control_nowave_noaero_p8 intel
• cpld_debug_p8 intel
• cpld_debug_noaero_p8 intel
• cpld_control_c48 intel
• cpld_warmstart_c48 intel
• cpld_restart_c48 intel
• cpld_control_p8_faster intel
• cpld_control_pdlib_p8 intel
• cpld_restart_pdlib_p8 intel
• cpld_mpi_pdlib_p8 intel
• cpld_debug_pdlib_p8 intel
• atm_ds2s_docn_dice intel
• cpld_control_nowave_noaero_p8 gnu
• cpld_control_pdlib_p8 gnu
• cpld_debug_pdlib_p8 gnu

Input data Changes:

• None.

Library Changes/Upgrades:

• No Updates

---

Testing Log:

• RDHPCS
  • Hera
  • Orion
  • Hercules
  • Jet
  • Gaea
  • Derecho
• WCOSS2
  • Dogwood/Cactus
  • Acorn
• CI
• opnReqTest (complete task if unnecessary)

[NickSzapiro-NOAA]

@dpsarmie I also hit a problem with DT_ATMOS. For gfsv17 tests for example:

1. export_fv3 sets a default DT_ATMOS
2. export_cpl sets a different default DT_ATMOS that is used in export_{cice,cmeps,...} for DT_CICE, coupling_interval_fast_sec,..
3. Then export_ugwpv1 sets a different DT_ATMOS based on ATMRES
4. But cice, cmeps,... never see changes from step 3

Maybe export_cpl should not touch DT_ATMOS either.

It may also be better in these tests to call export_fv3, then export_ugwpv1, then export_cpl so atmosphere is configured properly before setting other components

[dpsarmie]

Thanks @NickSzapiro-NOAA for bringing that up. I'll have to do some more testing to see what makes sense. I think the simplest solution would be to make the atm only and cpld default DT_ATMOS the same, but I want to make sure that the runtime isn't greatly affected for the atm only cases.
If not, then there needs to be a straightforward way to differentiate between the possible combinations and what needs to be defined.
Although I agree that the ugwpv1 call should come before the cpld call to resolve your issue 4.

[NickSzapiro-NOAA]

I think it is better if DT_ATMOS is thought of as input to export_cpl (and not modified in export_cpl). The issue I'm focusing on is (coupling) timesteps for gfsv17 test, where ufs.configure is the same before and after 2363:

# CMEPS warm run sequence
runSeq::
@3600
   MED med_phases_prep_wav_avg
   MED med_phases_prep_ocn_avg
   MED -> WAV :remapMethod=redist
   MED -> OCN :remapMethod=redist
   WAV
   OCN
   @720
     MED med_phases_prep_atm
     MED med_phases_prep_ice
     MED -> ATM :remapMethod=redist
     MED -> ICE :remapMethod=redist
     ATM
     ICE
     ATM -> MED :remapMethod=redist
     MED med_phases_post_atm
     ICE -> MED :remapMethod=redist
     MED med_phases_post_ice
     MED med_phases_ocnalb_run
     MED med_phases_prep_ocn_accum
     MED med_phases_prep_wav_accum
   @
   OCN -> MED :remapMethod=redist
   WAV -> MED :remapMethod=redist
   MED med_phases_post_ocn
   MED med_phases_post_wav
   MED med_phases_restart_write
@
::

But, dt_atmos in model_configure is 360s in current develop vs 720s before 2363. Was it intended for 2363 to reduce timestep for this test? Either way, now FV3<-->CMEPS no longer couple every ATM timestep, ATM timestep is different than ICE timestep,... I'm not sure this is intentional and may need hot fix if not intended.

[dpsarmie]

Ok I see what you mean.

#2363 was intended to change the timesteps to have the more optimized set of parameter values to allow for more stability at longer timesteps: NOAA-EMC/global-workflow#2574

I'm going to move forward with your idea of having DT_ATMOS set in export_fv3 and changed in export_ugwpv1 (if needed) before calling export_cpl.

That should allow DT_CICE and coupling_interval_fast_sec to be set correctly when the other export calls are run from within export_cpl.

Line 1302 will also be removed in default_vars.sh since DT_INNER will have been set by then.

Let me know if that logic makes sense. That should resolve the issues of not having the timing in sync between the models, correct?

[NickSzapiro-NOAA]

That makes sense to me. Maybe can ask Denise to review solution too

[dpsarmie]

Latest changes do the following:

To fix the timing issue it CMEPS, default_vars.sh has been reworked. If you export DT_ATMOS and/or ATMRES before the export_fv3 (and other export functions), then all dependent variables will be auto-populated. If these variables are not set, then defaults will be set. You can still manually set these parameters after the function calls but you need to make sure that all variables that depend on DT_ATMOS and ATMRES are also set unless you intended to do so otherwise.

Either way, the timing with CMEPS should be functioning as intended.

[NickSzapiro-NOAA]

@dpsarmie In the PR description, can you add how the logic/flow in default_vars works and should be used?

[dpsarmie]

@NickSzapiro-NOAA I added a snippet to the description on how it should be used. I'm actually currently working on a more detailed document for how to use the new features, so let me know if anything is unclear or if anything else should be added to the description for the PR.

[DeniseWorthen]

@dpsarmie I'm getting confused about where the B4B differences are arising. It seems you're combining what is a bug-fix (the issue w/ dt_atmos), with updates of various namelist settings to match GW. Both of these tasks create non-B4B changes, is that right? Should we really be combining a bug fix w/ what is actually a new feature (consistency w/ G-W)?

[dpsarmie]

@DeniseWorthen Yes, the changes to the namelist and the associated changes to default_vars (just adding variables) were changes that were still B4B comparable.
There was a discussion a few weeks ago with EPIC to roll these other bug fixes into the PR since they are touching related files.

I've opened individual issues for documentation purposes since it is getting a bit complex. I'll add something to the description to try and detail the changes and which issue was addressed by which change in the code. There won't be anything else added to this PR.

[DeniseWorthen]

@dpsarmie Thanks for adding the information. Did either

Variable values that were hard-coded in global_control.nml.IN were replaced with an @ tag and the values were moved to default_vars.sh.
Another issue will also be merged into this PR. The cpld_control_sfs configuration will be changed to bring it in-line with the GW configuration. cpld_control_sfs settings were changed in the test configuration file.

change baselines?

I'm confused because the next change is described with This will add additional baseline changes. (emphasis added).

If either of the first two items changed baselines, then I believe it is poor CM practice to combine issues which both change a set of tests. Rolling them together makes it much harder to un-roll/debug in the future, if required.

[dpsarmie]

@DeniseWorthen Yes, the cpld_control_sfs test underwent two baseline changes. The GW settings change (second item under Description) and the DT_ATMOS fix (third item) had two distinct changes in that baseline.

I can untangle those easily and create an independent PR so there's only one baseline change per PR.

[NickSzapiro-NOAA]

I think SKEB, SPPT, SHUM are missing default values (like EPBL).

If I understand, are you wanting these to default to what is being used in GEFS prototype? Maybe @pjpegion can confirm?

[NickSzapiro-NOAA]

Maybe we default to what's in global workflow (?):
https://github.com/NOAA-EMC/global-workflow/blob/develop/parm/config/gefs/config.efcs

[dpsarmie]

Yes, I'm was setting defaults based off of GW. There are some missing entries that are not here that will be added.
I'm scheduling a meeting soon with GW to talk about some discrepancies in the RT and GW namelist and what they would prefer.
The missing defaults are here on a branch I created for the GW PR: https://github.com/dpsarmie/global-workflow/blob/feature/GW-RT_GlobalNML/global_control.nml.IN#L393

[NickSzapiro-NOAA]

Why is this different than the cpld_control_nowave_noaero_p8 test it depends on?

[NickSzapiro-NOAA]

Is DT_INNER always equal to DT_ATMOS?

Why is the value of default_dt_atmos changing here? Is it used somewhere else?

[dpsarmie]

If the test is running the ugwpv1, then yes DT_INNER will be set to DT_ATMOS. If we set DT_INNER to half of DT_ATMOS (which is what GW is doing), then a handful of tests run over the allotted wall clock time. I did this to keep the RTs under the wall clock, but I did test the RTs using the GW parameters that were discussed here: #2363 to make sure that they were stable on our end.

default_dt_atmos changes here to make sure that it does not change in export_cpl(). I'm allowing export_cpl() to change DT_ATMOS for now to keep functionality with ugwpv0. When the time comes to deprecate version 0, then L1403 - 1407 should be removed.

[dpsarmie]

@dpsarmie I also hit a problem with DT_ATMOS. For gfsv17 tests for example:

1. export_fv3 sets a default DT_ATMOS
2. export_cpl sets a different default DT_ATMOS that is used in export_{cice,cmeps,...} for DT_CICE, coupling_interval_fast_sec,..
3. Then export_ugwpv1 sets a different DT_ATMOS based on ATMRES
4. But cice, cmeps,... never see changes from step 3

Maybe export_cpl should not touch DT_ATMOS either.

It may also be better in these tests to call export_fv3, then export_ugwpv1, then export_cpl so atmosphere is configured properly before setting other components

This was resolved by adding a default_dt_atmos variable. If DT_ATMOS is not defined before the export_fv3 is called, then it will be set to its default value. Since the coupled cases and the ugwpv1 cases all have their unique DT_ATMOS settings, they will also change DT_ATMOS if it was set to default.

The new ordering of the function calls should be export_fv3, export_ugwpv1, export_cpl. If default_dt_atmos is 1, export_fv3 will set DT_ATMOS to the default value (default for uncoupled cases), export_ugwpv1 will then set DT_ATMOS to the compatible value for gwpv1 based on grid resolution. The final export_cpl call will not change DT_ATMOS and then cice and cmeps will get the final DT_ATMOS value.

All other model configuration combinations should also work. ATM only (export_fv3), coupled using gwpv0 (export_fv3, export_cpl), ATM gwpv1 only, (export_fv3, export_ugwpv1), and coupled gwpv1 (export_fv3, export_ugwpv1, export_cpl) are all working as intended.