The SRW App is experiencing failures of the CRTM Forward Model while running inline and offline post on Derecho

[MichaelLueken]

Following the hash updates for the UFS-WM to 1787f26 (associated with PR #1823) and UPP to 520cc23 (associated with PR #760), the SRW App is failing on UCAR's new platform, Derecho. All SRW App workflow-end-to-end (WE2E) tests are failing in run_fcst (for inline post) and run_post (for offline post).

Looking at the log files, both inline and offline post are failing in the CRTM Forward Model with the following error message:

forrtl: severe (122): invalid attempt to assign into a pointer that is not associated
Image              PC                Routine            Line        Source
ufs_model          00000000042221E5  crtm_forward_modu         356  CRTM_Forward_Module.f90
libiomp5.so        000014F8970FB053  __kmp_invoke_micr     Unknown  Unknown
libiomp5.so        000014F897069A64  __kmp_fork_call       Unknown  Unknown
libiomp5.so        000014F897023223  __kmpc_fork_call      Unknown  Unknown
ufs_model          0000000004221C83  crtm_forward_modu         353  CRTM_Forward_Module.f90
ufs_model          0000000003E8123B  calrad_wcloud_           1725  CALRAD_WCLOUD_newcrtm.f

Please note that the current use of postxconfig-NT-fv3lam.txt is not having issues on Derecho. Only when moving to the postxconfig-NT-fv3lam_rrfs.txt file (which includes simulated radiances) does this become an issue. I have also reached out to EPIC's AUS team since the failure is actually occurring in the CRTM itself, to see if there is an issue with CRTM build on that machine.

My test directories are still available on Derecho - /glade/derecho/scratch/mlueken/expt_dirs. The inline post test is grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2, while the rest of the tests are offline post.

If you would like additional information, please let me know and I'll provide everything that I can.

Tagging @WenMeng-NOAA and @FernandoAndrade-NOAA for awareness.