Merging numerics developments, including FFTW - #307
Draft
sit23 wants to merge 10 commits into
Draft
Conversation
Squashed port of the FFTW3 dispatch mechanism from uob_fftw_sit23_new_maths2intel onto current master, combining several of that branch's original commits (cd90f40, ff9cab6, 97b8235, c880667, 3579f18, 096733b) into one, since replaying them individually against master's independently-evolved history triggered spurious rename-detection conflicts across unrelated files. Attributed to George Lancaster, who did the great majority of this work. Adds an #ifdef FFTW3 dispatch path through fft.F90's grid<->Fourier routines to a new fftw3 module (shared/fft/fftw.F90), backed by a new spherical_fourier_fftw.F90. Non-FFTW3 builds are unaffected - compiled out entirely without -DFFTW3. Also includes two changes NOT gated behind #ifdef FFTW3, applying to every build: transforms.F90's reverse_transpose_fourier/ trans_spherical_to_grid_* gained an optional `block` argument (defaults to blocking, i.e. previous behaviour, so backward compatible) enabling non-blocking mpp_transmit. Verified during the physics-branch port (uob_fftw_sit23_physics_2026) that reverting this exact change made no difference to bit-reproducibility on frierson/held_suarez/bucket_model/realistic_continents, so treating it as safe here too - full trip_test validation of this branch will re-confirm.
Part of the same squashed FFTW3 core port as the previous commit (cd90f40, ff9cab6, 97b8235, c880667, 3579f18, 096733b from uob_fftw_sit23_new_maths2intel), split out because this file also needed reconciling against master's own independent do_spec_tracer_filter addition (kept as-is) and the already-fixed missing-comma FORMAT bug in the json-logging labels (kept fixed). Also includes the branch's INTERNAL_FILE_NML namelist-read rewrite (explicit open/read/close instead of the pre-opened handle), not gated behind #ifdef FFTW3 and so applying to every build. Verified during the physics-branch port that reverting this exact change made no difference to bit-reproducibility - treating it as safe here too.
(squashed from e43d97b and a4d6967 on uob_fftw_sit23_new_maths2intel) Added option whereby fftw3 can be enabled in any codebase via a call to cb.enable_fftw3(). Also makes fftw builds go in a different directory to standard builds, so you don't have to create a new model folder and pathnames file for fftw and non-fftw versions of the same codebase. Applied cleanly against current master's much-grown codebase.py (barotropic/column-model/Socrates-for-column additions all landed since this branch forked) with no reconciliation needed - the new methods and DryCodeBaseFFTW class are purely additive.
(squashed from f3d90b3 and ea4d3e0 on uob_fftw_sit23_new_maths2intel) Exports DryCodeBaseFFTW alongside the codebase.py FFTW additions. Also adds a T213 resolution option to Experiment.RESOLUTIONS, and makes multi-node runs fall back to GFDL_JOB_NODEFILE with a slurm bootstrap when PBS_NODEFILE isn't set, so multi-node jobs work on slurm-managed clusters as well as PBS ones.
(squashed from f240537, dd4dc99, fadc19a, 00f18af, f3d90b3, a4d6967 on uob_fftw_sit23_new_maths2intel) New model directories (field_table/path_names) so FFTW-enabled dry and grey codebases can be built without needing a hand-maintained separate path_names file per FFTW/non-FFTW variant, plus a worked example test case (held_suarez_fftw) and readme showing how to call cb.enable_fftw3(). shared/fft/fftw.F90 also added to the main src/path_names manifest so it's available to every codebase (compiles to nothing without -DFFTW3).
…ob scripts (squashed from many commits across uob_fftw_sit23_new_maths2intel's 2019 Isambard/BluePebble portability work - b9b509d, c958c5d, 18255b3, 08824aa, 81c5999, and related environment/template/job additions from Lancasterg, George Lancaster, and qv18258) Environment scripts and mkmf templates for Isambard (GNU, Cray CCE, ARM HPC variants), BlueCrystal 3/4, and BluePebble, plus their job submission scripts under exp/site_specific/ - kept per Stephen's explicit decision (2026-08-24) to preserve Isambard/BluePebble support in case it's useful to others, even though neither system is in active use by this project currently. Reconciled three files against master's own independent evolution since this branch forked: env/isca gained an FFTW module load without disturbing master's own later F90/CC/NETCDF_LIBS additions to the same file; compile.sh's hardcoded -DOVERLOAD_C8 was dropped (now supplied via codebase.py's precision_compile_flags, part of the enable_fftw3 port) while keeping master's own ${CDEFS} customisation hook; mkmf.template.gfort's branch-side LDFLAGS was a strict subset of master's current LDFLAGS (which already gained a macOS netcdf path), so master's version was kept as-is. Also deduplicated a leftover doubled LDFLAGS line in mkmf.template.ia64 (the branch's own authoring slip - the second line silently overrode the first, so only the second, more complete FFTW library line was kept). These files are all-new or self-contained (site-specific env/build config); none of them affect the non-Isambard/BluePebble build path. Note the new env files needed `git add -f`, since master's own .gitignore has a blanket `env/` pattern predating this branch's existence - matching whatever the original authors must have done to get these files tracked in the first place.
(squashed from several small 2019 fixes on uob_fftw_sit23_new_maths2intel made while getting Isca compiling on Isambard/BCP3/BCP4 with the Cray and GNU toolchains, notably ebd82cd and the monin_obukhov_kernel.F90 CRAY_DIM macro work) Small, self-contained portability/robustness fixes: a resolution-aware CRAY_DIM preprocessor macro in monin_obukhov_kernel.F90 (Cray's compiler needed array dimensions available at the point of a pure subroutine's declaration, which plain assumed-size `n` didn't satisfy), a couple of integer/logical coercion fixes for the Cray Fortran parser, and similar small changes across diffusivity, dry_convection, hs_forcing, lscale_cond, vert_turb_driver, vert_advection, atmos_model, atmosphere, polvani_2007, spectral_initialize_fields, vert_coordinate, surface_flux, and create_xgrid.c. Reconciled two files against master's own independent bit-reproducibility fixes touching the same code paths: hs_forcing.F90's equinox_day wraparound branch (kept master's already-fixed, carefully bit-preserving version rather than the branch's plain unconditional modulo()) and surface_flux.F90's use_actual_surface_temperatures FATAL sanity guard (master independently added the same option plus a stricter safety check; kept master's version, which is a superset). Verified during the physics-branch port that none of this bundle individually or collectively affects bit-reproducibility when left at default settings (confirmed via the Group A/A2 bisection in PHYSICS_PORT_VALIDATION.md).
The FFTW model directories' hand-maintained path_names files were copied verbatim from the original 2019 branch and had drifted out of sync with master's current dry/grey model manifests - missing the cloud_simple module family and frierson_monin_obukhov.F90, both added to master since. Regenerated both as current dry/path_names + shared/fft/fftw.F90 (and grey/ equivalently), which is what these files are meant to be. Caught by an actual FFTW build attempt failing with "Cannot open module file frierson_monin_obukhov_mod.mod".
(squashed from several run-plevel/postprocessing commits on uob_fftw_sit23_new_maths2intel, notably the run_plevel.py portion of 0752c6b deferred from the physics-branch port, ad084a5, 9d776c3, b524f7f, 0ebf17e, 162ec03, c87172e, 6ec540a) Adds an all_vars/var_names_list toggle to run_plevel.py so a subset of variables can be interpolated instead of everything (faster, smaller output for quick-look purposes), a scalar_axis workaround in plevel_fn.py (the interpolator drops variables that only have a scalar_axis dimension when used standalone - this adds them back afterwards), and small fixes to mppnccombine_run.sh, compile_plev_interpolation.sh, and modified_time_script.py. Reconciled run_plevel.py against master's own "Modernising run-plevel.py" (4459f5c) and remove_certain_restart_and_data_files.py's personal scratch config against master's current defaults - kept master's own current base_dir/exp_name_list values in both cases (neither side's hardcoded personal paths is more "correct" than the other) while porting the genuine all_vars feature. grey_rad_parm_check.py and len_of_day_year_check.py, also touched by this branch, turned out to already be identical on master - no change needed for those.
Documents trip_test bit-reproducibility confirmation of every non-#ifdef-gated file this branch touches (held_suarez and frierson pass against master), plus a real 5-day held_suarez run comparing the new FFTW3 spectral core against the reference Temperton FFT - agrees to floating-point roundoff, confirming the transform is correctly wired up, not just compiling. Companion to PHYSICS_PORT_VALIDATION.md on uob_fftw_sit23_physics_2026.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A number of years ago, various people at the University of Bristol, and George Lancaster in particular, worked on creating an FFTW version of Isca, and other updates. These changes have been sitting on a branch of mine for a while, and I'm now putting them here to merge in. Still needs more testing, but looks clean at the moment.