CTSM5.4.0xx Fix problems encountered with running PLUMBER2 sites

[olyson]

Description of changes

This PR fixes failures in running PLUMBER2 sites. There are four categories of failures.

1. Wetland sites fail at case creation because the usermods are requesting a change to baseflow_scalar in the user_nl_clm namelist, however, this parameter is now on the parameter file. This issue is described in PLUMBER2 wetland sites fail at case creation #4023 . The sites are:

        "CZ-wet",
        "DE-SfN",
        "FI-Kaa",
        "FI-Lom",
        "RU-Che",
        "SE-Deg",
        "US-Los",
        "US-Myb",
        "US-Tw4",
        "PL-wet"

2. Several sites fail with a negative 2-m relatve humidity error in HumanIndexMod. The negative humidity is generated from either BareGroundFluxesMod or CanopyFluxesMod and passed into HumanIndexMod where there is an error check for this condition (and endrun). This issue is described in Several PLUMBER2 sites fail at run time #4081 . The sites are

        "ES-ES2",
        "ES-LgS",
        "ES-VDA",
        "SE-Deg",
        "US-Cop"

3. Several sites fail with calendar errors. These are due to the fact that each site may require a different case setup configuration for spinup. This issue is described in Several PLUMBER2 sites fail at run time #4081 . The sites are

        "AT-Neu",
        "AU-How",
        "BE-Vie",
        "CH-Dav",
        "DE-Geb",
        "DE-Hai",
        "DE-Tha",
        "DK-Sor",
        "FR-Pue",
        "IT-MBo",
        "IT-SRo",
        "NL-Loo",
        "US-NR1",
        "US-FPa"

4. After fixing 3. above, five of those sites run successfully but nine fail with a different calendar error. This is related to the fact that the time dimension in the atmospheric forcing files is float, not double, consequently, the time of day calculated in CDEPS is off by several seconds both for the lower bound and the upper bound. This can cause problems for leap day transitions. This issue is described in Several PLUMBER2 sites fail at run time #4081. The sites are:

        "AU-How",
        "BE-Vie",
        "CH-Dav",
        "DE-Geb",
        "DE-Tha",
        "DK-Sor",
        "FR-Pue",
        "NL-Loo",
        "US-NR1"

Specific notes

The fixes for these issues are:

1. Here, tower_sites.py is modified so that for wetland sites, the parameter file for the base case is queried (regardless if it is wetland or not), and then the parameter file is replaced with a modified parameter file with baseflow_scalar=0 for any requested wetland sites in the run_tower submission.
   The baseflow_scalar namelist change request was deleted from the wetland sites shell commands by running tools/site_and_regional/plumber2_usermods. These modified shell commands are part of this commit.

2. Per conversation with @wwieder , this PR turns off HumanIndexMod for all the PLUMBER2 sites for now, to avoid the negative relative humidity error check. This is accomplished in cime_config/usermods_dirs/clm/PLUMBER2/defaults/user_nl_clm.

3. For the sites failing with the calendar error during spinup, these are the xml variables that need to be changed on a site by site basis for spinup:

RUN_STARTDATE
CALENDAR
DATM_YR_START
DATM_YR_END
DATM_YR_ALIGN
START_TOD (may already be set correctly though)

My run tower script (/glade/work/oleson/release-clm5.0.34/tools/PTCLM/run_clmtowers_plumber2.csh) handled this complexity, but this was not implemented in the run_tower infrastructure for PLUMBER2.
I've combined the new spinup variables into create_PLUMBER2_sites_csv.ncl -> PLUMBER2_sites.csv and modified the code to read these new variables in and generate the required shell commands.

• We'll also need a change in CDEPS, the addition of a new variable DATM_YR_END_FILENAME, following the existing variable DATM_YR_START_FILENAME that was put in for PLUMBER2. See Add DATM_YR_END_FILENAME xml variable for PLUMBER2 CDEPS#416

4. I have changed the time dimension in all of the forcing files from float to double and regenerated the time data. The nine sites now run successfully.

• The new forcing data is at /glade/work/oleson/PLUMBER2/scripts/MAKE_TIME_DOUBLE/ and will need to be copied (and rimported) to /glade/campaign/cesm/cesmdata/cseg/inputdata/atm/datm7/CLM1PT_data/PLUMBER2.

A one line change to plumber2_usermods.py was also required to get these sites to run. This line had been commented out pending a CDEPS mod, which appears to have been implemented.

• A couple of other CDEPS changes were required to get these sites to run, a change in components/cdeps/datm/cime_config/stream_definition_datm.xml to set dtlimit to 50 for 2000 compsets (spinup), and setting taxmode to extend and dtlimit to 1.e30 for HIST compsets.

Contributors other than yourself, if any: @wwieder (python help)

CTSM issues resolved or otherwise addressed, if any:
Resolves #4023
Resolves #4081
Resolves #4105

If answers are expected to change, describe (delete this line otherwise): The sites listed above will now run to completion. Sites that had run successfully prior to these changes may not have run correctly, so answers will likely change for those sites.

Any user interface changes (namelist or namelist defaults changes)? None to the defaults namelist. For wetlands, we only create a temporary modified parameter file for the requested run(s) and point to it in the run directory. Other changes here are accomplished by the PLUMBER2 shell commands.

Testing planned or performed, if any:
I've tested all of the sites with various submissions of the type:
./run_tower --plumber-sites {XX-XXX} --run-type ad --xmlchange STOP_OPTION=nyears,STOP_N=100,RESUBMIT=2,JOB_WALLCLOCK_TIME=06:00:00,PROJECT=P93300041 --overwrite.

• It would be good to test all of the sites in one submission of run_tower once all of the new forcing files are in place in the proper location, in order to fully test this branch.

I also tested a NEON site (ABBY) to make sure I didn't break that configuration.

Requirements before merge:

• I have followed the CTSM contribution guidelines.
• The code in this PR branch builds with no errors.
• The code in this PR branch runs with no errors. Briefly describe tested configuration(s): See above
• This either (a) does not change answers, (b) it only changes answers at roundoff level, or (c) I have performed a scientific evaluation of the answer changes. Which?: I guess this is (d) We actually now get answers

• I have reviewed relevant parts of the CLM documentation Tech Note or User's Guide to determine if anything needs to be changed or added. If it does, describe: I plan to incorporate this as part of Documentation for Plumber2 refactoring #2409
• This PR either (a) does not create a need to update the documentation or (b) includes required documentation updates (see guidelines for contributing documentation). Which?: See Revision of User's Guide section 1.6.2.3 - PLUMBER2 Tower Single Point Simulations #4135 for documentation updates.

[olyson]

I see that there are some python errors in the automated testing, I'll address those next week once I figure out what they are about. Although my code was rated 9.99/10, ha ha.

[olyson]

Ok, I've fixed all of the python-lint-and-black errors.

[wwieder]

@samsrabin can you review this. Keith's done functional tests to check this works, but would like a review on the python. We can also discuss testing (which may be outside the scope of this PR)?

[samsrabin]

Can do, sure. What sort of timeline were y'all hoping for?

[olyson]

Can do, sure. What sort of timeline were y'all hoping for?

Maybe by the end of next week?

[olyson]

Per discussion at CTSM software meeting, I'm going to expand this PR to include other fixes to problems I've encountered with running PLUMBER2 sites.

[wwieder]

• @olyson is having issues with changing calendar because it's set in env_build.xml, which prevents using --keepexe in cloned cases

  • @wwieder suggested setting all sites to NO_LEAP for spinup
  • @olyson noted this may be masking issues with DATM being read in (silently wrong)

• @ekluzek requested creating PLUMBER - specific tests

[olyson]

See #4122 for issue with changing CALENDAR for a cloned site case.

[olyson]

Running all sites with NO_LEAP for ad now:

./run_tower --plumber-sites {AT-Neu,AU-ASM,AU-Cow,AU-Cpr,AU-Ctr,AU-Cum,AU-DaP,AU-DaS,AU-Dry,AU-Emr,AU-GWW,AU-Gin,AU-How,AU-Lit,AU-Otw,AU-Rig,AU-Rob,AU-Sam,AU-Stp,AU-TTE,AU-Tum,AU-Whr,AU-Wrr,AU-Ync,BE-Bra,BE-Lon,BE-Vie,BR-Sa3,BW-Ma1,CA-NS1,CA-NS2,CA-NS4,CA-NS5,CA-NS6,CA-NS7,CA-Qcu,CA-Qfo,CA-SF1,CA-SF2,CA-SF3,CH-Cha,CH-Dav,CH-Fru,CH-Oe1,CN-Cha,CN-Cng,CN-Dan,CN-Din,CN-Du2,CN-HaM,CN-Qia,CZ-wet,DE-Bay,DE-Geb,DE-Gri,DE-Hai,DE-Kli,DE-Meh,DE-Obe,DE-Seh,DE-SfN,DE-Tha,DE-Wet,DK-Fou,DK-Lva,DK-Ris,DK-Sor,DK-ZaH,ES-ES1,ES-ES2,ES-LMa,ES-LgS,ES-VDA,FI-Hyy,FI-Kaa,FI-Lom,FI-Sod,FR-Fon,FR-Gri,FR-Hes,FR-LBr,FR-Lq1,FR-Lq2,FR-Pue,GF-Guy,HU-Bug,ID-Pag,IE-Ca1,IE-Dri,IT-Amp,IT-BCi,IT-CA1,IT-CA2,IT-CA3,IT-Col,IT-Cpz,IT-Isp,IT-LMa,IT-Lav,IT-MBo,IT-Mal,IT-Noe,IT-Non,IT-PT1,IT-Ren,IT-Ro1,IT-Ro2,IT-SR2,IT-SRo,JP-SMF,NL-Ca1,NL-Hor,NL-Loo,PL-wet,PT-Esp,PT-Mi1,PT-Mi2,RU-Che,RU-Fyo,RU-Zot,SD-Dem,SE-Deg,UK-Gri,UK-Ham,UK-PL3,US-AR1,US-AR2,US-ARM,US-Aud,US-Bar,US-Bkg,US-Blo,US-Bo1,US-Cop,US-FPe,US-GLE,US-Goo,US-Ha1,US-Ho1,US-KS2,US-Los,US-MMS,US-MOz,US-Me2,US-Me4,US-Me6,US-Myb,US-NR1,US-Ne1,US-Ne2,US-Ne3,US-PFa,US-Prr,US-SP1,US-SP2,US-SP3,US-SRG,US-SRM,US-Syv,US-Ton,US-Tw4,US-Twt,US-UMB,US-Var,US-WCr,US-Whs,US-Wkg,ZA-Kru,ZM-Mon} --run-type ad --xmlchange STOP_OPTION=nyears,STOP_N=100,RESUBMIT=2,JOB_WALLCLOCK_TIME=06:00:00,PROJECT=P93300041 --overwrite

[olyson]

These sites are still running in AD mode with CALENDAR=NO_LEAP, but I expect they will complete successfully here and in postad mode. The other changes I've made enable this. Those changes are required because, unlike NEON sites, the datm forcing data is in local time and we consequently have to start the model at a GMT adjust to match local time (we choose local midnight). This is the primary cause of most of our spinup issues here. The current setup still runs 95 sites in NO_LEAP mode because it was not possible to spinup in GREGORIAN mode because, e.g., only one year of forcing was available. The current setup runs 75 sites in GREGORIAN mode. But running those sites in GREGORIAN mode is not required, it is simply possible because of my changes.
So, I think given #4122 , I think it is fine to run all sites in NO_LEAP for now. I will make commits to the branch for this, but monitor the ongoing runs for failures.

[olyson]

All 170 sites have now run AD (300 years), postAD (300 years), and transient modes successfully.
I'm consolidating remaining tasks here that need to be completed before/after this PR is merged, because there is some duplication of tasks throughout the PR.

• Rimport the datm forcing files at /glade/campaign/cesm/cesmdata/inputdata/atm/datm7/CLM1PT_data/PLUMBER2
• Continue the documentation update in the User's Guide single point simulation section on PLUMBER2 per Review 1.6. Running Single Point and Regional Cases #3885 . See Revision of User's Guide section 1.6.2.3 - PLUMBER2 Tower Single Point Simulations #4135 for PR.
• Testing was requested, I'm not sure what the scope of this will be.
• Need CDEPS Add DATM_YR_END_FILENAME xml variable for PLUMBER2 CDEPS#416 before merging?

[slevis-lmwg]

Regarding the ./rimport task above, Keith and I tried
./rimport --list plumber2_list
in /glade/campaign/cesm/cesmdata/cseg/inputdata
and got the following error for all the files in the list (only one shown here):

'/glade/campaign/cesm/cesmdata/cseg/inputdata/atm/datm7/CLM1PT_data/PLUMBER2/ZM-Mon/CLM1PT_data/CTSM_DATM_ZM-Mon_2008-2008.nc':
File is already published but NOT linked; linking now.
    Error deleting file /glade/campaign/cesm/cesmdata/cseg/inputdata/atm/datm7/CLM1PT_data/PLUMBER2/ZM-Mon/CLM1PT_data/CTSM_DATM_ZM-Mon_2008-2008.nc: [Errno 13] Permission denied: '/glade/campaign/cesm/cesmdata/cseg/inputdata/atm/datm7/CLM1PT_data/PLUMBER2/ZM-Mon/CLM1PT_data/CTSM_DATM_ZM-Mon_2008-2008.nc' -> '/glade/campaign/cesm/cesmdata/cseg/inputdata/atm/datm7/CLM1PT_data/PLUMBER2/ZM-Mon/CLM1PT_data/CTSM_DATM_ZM-Mon_2008-2008.nc.tmp'. Skipping relink.
    rimport: error processing /glade/campaign/cesm/cesmdata/cseg/inputdata/atm/datm7/CLM1PT_data/PLUMBER2/ZM-Mon/CLM1PT_data/CTSM_DATM_ZM-Mon_2008-2008.nc: Error relinking during rimport

This means that I we are unable to rimport new files by the same name to the same location due to the presence of the old files. Keith and I agreed that we do NOT want to rename the files or place them in a new location, so I sent a request for help to help@ucar.edu.

[ekluzek]

I suggest adding some PLUMBER2 tests similar to how we have NEON tests. For example the NEON test

  <test name="SMS_Ld10_D_Mmpi-serial" grid="CLM_USRDAT" compset="I1PtClm60Bgc" testmods="clm/default--clm/NEON/HARV">

for PLUMBER2 would become...

  <test name="SMS_Ld10_D_Mmpi-serial" grid="CLM_USRDAT" compset="I1PtClm60Bgc" testmods="clm/default--clm/PLUMBER2/PL-wet">

We should have at least one test from the top groups that fail.

[olyson]

@ekluzek ,
I've added two tests in testlist_clm.xml:

<test name="SMS_Ly2_D_Mmpi-serial" grid="CLM_USRDAT" compset="I1PtClm60Bgc" testmods="clm/default--clm/PLUMBER2/ARSLu--clm/ADspinup_PLUMBER2">

<test name="SMS_Ly1_D_Mmpi-serial" grid="CLM_USRDAT" compset="IHist1PtClm60Bgc" testmods="clm/default--clm/PLUMBER2/ARSLu">

The first tests all of the logic involved in running a PLUMBER2 spinup. It runs a site that that has one year of forcing and loops over the forcing twice. This should basically cover all of the spinup problems that were encountered.

The second runs the same site in transient mode which basically tests that the model successfully runs the last time step, which was fixed by the CDEPS change.

I'm not convinced about the usefulness of other tests. These tests operate outside of the run_tower infrastructure.
The wetland sites were only failing because run_tower was trying to set baseflow_scalar in user_nl_clm.
The negative humidity error was occurring within HumanIndexMod. The solution was to turn it off through the PLUMBER2 usermods in user_nl_clm. Is it worth adding a test of one of those sites just to verify that HumanIndexMod is off, I'm not sure.

[ekluzek]

@olyson I just approved and made the CDEP tag. It's cdeps1.0.102. It would be good to include the update to it in this PR. But, that's also a lot of CDEP tags and there might be other things that pop up with updating to it. So we could make that update seperate from this one if it causes any problems.

So how about you try updating to 1.0.102 and if everything works and appears fine -- we include it. But, if you run into any trouble -- we hold off and make that update seperate?

[olyson]

@ekluzek , sounds like it is worth a try to update to that CDEPS tag, I'll let you know how it goes...

[ekluzek]

@ekluzek , I've added two tests in testlist_clm.xml:

<test name="SMS_Ly2_D_Mmpi-serial" grid="CLM_USRDAT" compset="I1PtClm60Bgc" testmods="clm/default--clm/PLUMBER2/ARSLu--clm/ADspinup_PLUMBER2">

<test name="SMS_Ly1_D_Mmpi-serial" grid="CLM_USRDAT" compset="IHist1PtClm60Bgc" testmods="clm/default--clm/PLUMBER2/ARSLu">

The first tests all of the logic involved in running a PLUMBER2 spinup. It runs a site that that has one year of forcing and loops over the forcing twice. This should basically cover all of the spinup problems that were encountered.

The second runs the same site in transient mode which basically tests that the model successfully runs the last time step, which was fixed by the CDEPS change.

I'm not convinced about the usefulness of other tests. These tests operate outside of the run_tower infrastructure. The wetland sites were only failing because run_tower was trying to set baseflow_scalar in user_nl_clm. The negative humidity error was occurring within HumanIndexMod. The solution was to turn it off through the PLUMBER2 usermods in user_nl_clm. Is it worth adding a test of one of those sites just to verify that HumanIndexMod is off, I'm not sure.

Perfect, this is awesome. The main point is to add some tests that will be run with aux_clm that would've caught some of the problems. And to be thoughtful about how best to do that which is exactly what you did. It's good to know that others were part of using run_tower. This is why longer term we also need #2438.

We have some tests that ensure that HumanIndex can be turned off. You could add one of those sites just because the negative humidty shows a strange sensitivity to those sites and testing it would catch if someone were to change that. But, that strange sensitivity might show up in other ways, and possibily catch other problems down the line? So probably worth adding, but not a big deal either way. The bigger problem would be to add too many tests to aux_clm. In that case we'd want to split them into different groups. But, adding three new single-point tests to aux_clm is not a problem. But, I'm good either way if you decide to add it, or leave it off...

Thanks for working on this and coming up with carefully crafted tests that test the important things. That's a shining example of how we want to do our testing.

[olyson]

Thanks @ekluzek , I agree about adding a third test for one of the sites exhibiting the negative humidity problem.

[olyson]

@ekluzek , all of the sites ran successfully with cdeps1.0.102.