FATES-Hydro fails upon restart when using alternate soil water transfer functions

[jennykowalcz]

Hi all! I have been trying FATES-Hydro no-comp pan-Amazon simulations with 2 evergreen PFTs. After encountering this error I have been trying runs with alternate soil water transfer functions (specifically, smooth1_campbell_type and van_genuchten_type), but this seems to make FATES-Hydro fail upon restart.

85: forrtl: error (65): floating invalid
 85: Image              PC                Routine            Line        Source             
 85: libpthread-2.31.s  00001551707EF910  Unknown               Unknown  Unknown
 85: e3sm.exe           0000000003A243E0  fateshistoryinter        4990  FatesHistoryInterfaceMod.F90
 85: e3sm.exe           0000000000C514AE  elmfatesinterface        3287  elmfates_interfaceMod.F90
 85: e3sm.exe           0000000001AEA3D5  canopyfluxesmod_m        1279  CanopyFluxesMod.F90
 85: e3sm.exe           00000000009BC0E5  elm_driver_mp_elm         782  elm_driver.F90
 85: e3sm.exe           0000000000956157  lnd_comp_mct_mp_l         617  lnd_comp_mct.F90
 85: e3sm.exe           00000000004A8323  component_mod_mp_         757  component_mod.F90
 85: e3sm.exe           000000000046955B  cime_comp_mod_mp_        2915  cime_comp_mod.F90
 85: e3sm.exe           0000000000490BCB  MAIN__                    153  cime_driver.F90
 85: e3sm.exe           000000000043336D  Unknown               Unknown  Unknown
 85: libc-2.31.so       000015517023C24D  __libc_start_main     Unknown  Unknown
 85: e3sm.exe           000000000043329A  Unknown               Unknown  Unknown

This appears to be separate from the other FATES-Hydro restart error (issue 1151) and is not fixed by the related PR (1156). I also get this error when using the hydro stability PR.

The error message is pointing here where the root water uptake rate is calculated.

[ckoven]

@jennykowalcz the fact that it is not crashing on the prior line might indicate that it is a problem for the site_hydr%rootuptake_sl on the soil levels below nlevsoil? Does it still crash if you change it to:

         hio_rootuptake_si(io_si) = sum(site_hydr%rootuptake_sl(1:nlevsoil),dim=1)

[jennykowalcz]

@ckoven Good idea! I just tried this but unfortunately it still crashes upon restart with the same error.

[jennykowalcz]

Commenting to clarify that so far I have only encountered this error in drier regions outside of the Amazon Basin.

[jennykowalcz]

Quickly tried a couple of alternate soil water transfer functions with PR 1164 and I have not encountered this restart issue. With Van Genuchten the simulation ran successfully. With "smooth1_campbell_type" it crashed around 12 years (not upon restart) with an error about FATES_ROOTWGT_SOILMATPOT. I don't think we need to worry about this since this PR is working fine with the default hard-coded soil water transfer function, but just documenting for the record. Closing this issue since the restart error appears to be resolved.