Download CMIP data and post-process model results

Written by Joshua Simmons 11-2021

Note

Artemis was retired in 2025. This guide is retained for reference purposes.

A collection of Python functions and scripts for downloading and processing CMIP6 data using Artemis.

The following jupyter notebooks demonstrate usage:

• 01_RUN_CMIP_Data_Download: Download CMIP6 data from Google stores and undertake preliminary processesing
  • This functionality is run on Artemis using command line calls with download_CMIP_data.py. Run python download_CMIP_data.py -h for help.
• 02_RUN_CMIP_Data_Process: Process ensemble of models for CMIP6 data.
  • This functionality is run on Artemis using command line calls with process_CMIP_data.py. Run python process_CMIP_data.py -h for help.
• 03_RUN_CMIP_Data_View: Load processed ensembles of CMIP data and show examples of preliminary plotting.

Using Singularity on Artemis

Artemis uses Singularity to run containers (https://sydneyuni.atlassian.net/wiki/spaces/RC/pages/761200644/Running+containers+with+Singularity). This is similar to Docker and in fact it has been updated to allow easy interface with docker images. There are several steps to getting a working singularity environment ready for your python code.

1. Develop a Dockerfile to describe your environment (or jump to step 3 and do it straight in Singularity)
2. Convert the Dockerfile to a Singularity Recipe (.sing)
3. Build a Singularity Image from the Singularity Recipe
4. Upload the Singularity Image to Artemis
5. Use the Singularity Image in your Artemis job

Step 1

First we need to develop a Dockerfile to describe our environment. There is a wealth of information on creating Dockerfiles. For this example we will generate a container with an ubuntu version of linux, with Python installed.

NOTE to ensure that this can run on Artemis (which is not running an up to date version of CentOS) we need to make sure that the Linux version we are using is around the same vintage as that on Artemis. If the kernels are too far out of sync then your singularity container will not run. As such I advise Ubuntu:16.04 or similar.

A simple introduction can be found here:

• https://faun.pub/hello-world-in-docker-using-python-9b3eb418fb15

The code for our Dockerfile is below:

# syntax=docker/dockerfile:1
FROM ubuntu:16.04
#These directories are required by Artemis which links them to the Artemis storage
RUN mkdir /project && mkdir /scratch
#Install ubuntu libraires and packages
RUN apt-get update -y && \
    apt-get install git curl -y
#Set some environemnt variables we will need
ENV PATH="/build/miniconda3/bin:${PATH}"
ARG PATH="/build/miniconda3/bin:${PATH}"
RUN mkdir /build && \
    mkdir /build/.conda
#Install Python3.9 we can use
RUN curl -O https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh &&\
    bash Miniconda3-latest-Linux-x86_64.sh -b -p /build/miniconda3 &&\
    rm -rf /Miniconda3-latest-Linux-x86_64.sh
WORKDIR /build
RUN conda install python=3.9 numpy

Briefly, we are:

• using the Ubuntu 16.04 image
• ensuring it's packages are up to date (using apt-get)
• Adding the /project and /scratch directories
  • NOTE When the singularity module is loaded, Artemis maps the project and scratch drives to the singularity instance so you can access/read/write files in these directories
• Adding paths to the conda install so that we can use "conda install …."
• Downloading and installing the latest version of miniconda
• Ensuring Python 3.9 and numpy are installed.

You can then add code on the end to install any python package you would like as you would on your usual conda install - via pip, conda-forge or a yml file.

NOTE we can include as much information in the base Docker image (in this case just a plain install of ubuntu:16.04) as needed. e.g., I could make a docker image that already contains Ubuntu 16.04 and miniconda installed. Then the Singularity Recipe will just pull that image and you could use conda to install any additional packages.

ToDo: integrate miniconda into the base Step 1 docker image as an Artemis-Python base image.

You can test this by directly building in Docker to check you're happy with the environment before you charge on.

• docker build --tag pytest .

• We can run interactive sessions (say for the pytest image) to try things out with something like

  docker run -v "pwd:/code" -w /code -it pytest

Other examples with more complex setups can be found in this XX Dropbox folder

Step 2

You can convert this Dockerfile (used by docker to build an image) into a Singularity Recipe which we will need to build our Singularity Image for Artemis to run.

NOTE You could skip Step 1 and 2, and directly craft a Singularity Recipe if you feel comfortable doing so.

To convert the Dockerfile we will use Spython (conda install -c conda-forge spython) which does this (mostly) automatically. You just need to be a bit careful with things like WORKDIR commands, making sure you also have cd commands which are more important in Singularity.

ToDo: integrate spython into the Step 3 docker image.

Command: spython recipe Dockerfile > pytest.sing

Now our Recipe file is as per below. Note that Singularity can pull the base image (in this case ubuntu:16.04 from the Docker library with the Bootstrap call:

Bootstrap: docker
From: ubuntu:16.04
Stage: spython-base
%post
# syntax=docker/dockerfile:1

#These directories are required by Artemis which links them to the Artemis storage
mkdir /project && mkdir /scratch
#Install ubuntu libraires and packages
apt-get update -y && \
apt-get install git curl -y
#Set some environemnt variables we will need and ensure paths exist
PATH="/build/miniconda3/bin:${PATH}"
PATH="/build/miniconda3/bin:${PATH}"
mkdir /build && \
mkdir /build/.conda
#Install Python3.9 we can use
curl -O https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh &&\
bash Miniconda3-latest-Linux-x86_64.sh -b -p /build/miniconda3 &&\
rm -rf /Miniconda3-latest-Linux-x86_64.sh
cd /build
conda install python=3.9 numpy
%environment
export PATH="/build/miniconda3/bin:${PATH}"
%runscript
cd /build
exec /bin/bash "$@"
%startscript
cd /build
exec /bin/bash "$@"

Step 3

Now we need to build the Singularity Image form the recipe.

NOTE from Artemis docs: "On Artemis, we cannot let you build Singularity containers because we cannot grant you root access. Instead, you can install Singularity on your own computer and build your Singularity containers there, then transfer the built images to Artemis HPC".

This can be done with Linux emulators on mac or windows as detailed on the Singularity website, but instead we will just use the power of Docker and a singularity builder image. I have already created this ("singtest"), so you can use the Docker image located here(XX dropbox link).

Now we run this image with: docker run -v "pwd:/workdir" -w /workdir --privileged singtest /bin/sh -c "singularity build testcontainer.simg pytest.sing" Here there are several important points:

• We will run this from the directory in which our "pytest.sing" Recipe file is sitting
• First the current working directory (obtained with pwd) is mounted using the -v (volume) flag
  • This makes sure the current directory (with the sing recipe) is available within Linux Container at a location /workdir
• We then make /workdir the working directory in the Linux Container using the -w command
• We use the --privileged flag
  • This is important and lets the Linux container appropriately access the namespaces it needs to create the Singularity image
  • NOTE --privileged shouldn’t be used for regular Docker environments as there security concerns, this should only be used when trusted
• Then we are telling it to use the singtest image to build the container
• Finally we tell it to run the command (-c) in bash "singularity build testcontainer.simg pytest.sing"
  • Here we are building a new image "testcontainer.simg" using the recipe "pytest.sing"
• When run we should have a new file in our working directory - "testcontainer.sing"

Step 4

Copy this file across to Artemis using your favourite SFTP or similar (Cyberduck works great for mac). Best to store this image in the project folder you need it for, as it will be a few 100 MB even up to 1 GB.

Step 5

Now we run this with PBS as we would any other job.

NOTE you will need to be granted permissions to use the Singularity modules. You can do this with a simple request here (https://sydneyuni.service-now.com/sm?id=sc_cat_item&sys_id=1ab0bb626d2935008dd31a4dcf150a21&sysparm_category=56db92a0db1f73042d38cae43a961918).

Below is an example of a pbs script to execute the code "download_CMIP_data.py" for a project "PRJ-DARECMIPPP" and instructions to copy some files to the RDS (https://sydneyuni.atlassian.net/wiki/spaces/RC/pages/212828198/Transferring+data+between+HPC+and+the+Research+data+store+RDS) using dt-script (https://sydneyuni.atlassian.net/wiki/spaces/RC/pages/229015601/Automating+data+transfers+with+dt-script).

#!/bin/bash
#PBS -P DARECMIPPP
#PBS -N cmipdl
#PBS -l select=1:ncpus=2:mem=32gb
#PBS -l walltime=06:00:00
#PBS -q defaultQ
#PBS -j oe
#PBS -m abe

module load singularity

cd "$PBS_O_WORKDIR"

# NOTE you need a trailing / on storage Dir to rsync properly
storageDir="/project/RDS-FSC-DARECMIPPP-RW/raw_cmip_data/"
destDir="/rds/PRJ-DARECMIPPP/raw_cmip_data/"
variable='tas'
experiment='historical'
frequency='Amon'
excludeModel="'MCM-UA-1-0' 'ICON-ESM-LR'"

# First use singularity to run the CMIP download
singularity exec ./artemis/cmip_download_container.simg /bin/bash -c "cd $PBS_O_WORKDIR && python download_CMIP_data.py \
'$storageDir' --variable $variable --experiment $experiment --frequency $frequency --exclude $excludeModel"

# Additional modifiers (if required)
#  --force --model $model --no-download

# best to use the dt-scipt workflow 
# (this submits a separate job on the dtq queue) location - /rds/PRJ-DARECMIPPP
echo "When complete run the following commands:"
echo cd ~/logs
echo dt-script -P DARECMIPPP --from  $storageDir --to $destDir

Below is an example of a pbs script to execute the code "process_CMIP_data.py" for a project "PRJ-DARECMIPPP" and instructions to copy some files to the RDS using dt-script.

#!/bin/bash
#PBS -P DARECMIPPP
#PBS -N cmipdl
#PBS -l select=1:ncpus=8:mem=128gb
#PBS -l walltime=06:00:00
#PBS -q defaultQ
#PBS -j oe
#PBS -m abe

module load singularity

cd "$PBS_O_WORKDIR"

# NOTE you need a trailing / on storage Dir to rsync properly
storageDir="/project/RDS-FSC-DARECMIPPP-RW/raw_cmip_data/"
outputDir="/project/RDS-FSC-DARECMIPPP-RW/processed_cmip_data"
destDir="/rds/PRJ-DARECMIPPP/processed_cmip_data/"
variable='tas'
experiment='historical'
frequency='Amon'

# First use singularity to run the CMIP download
singularity exec ./artemis/cmip_download_container.simg /bin/bash -c "cd $PBS_O_WORKDIR && python process_CMIP_data.py \
'$storageDir' '$outputDir' --variable $variable --experiment $experiment --frequency $frequency"

# best to use the dt-scipt workflow 
# (this submits a separate job on the dtq queue) location - /rds/PRJ-DARECMIPPP
echo "When complete run the following commands:"
echo cd ~/logs
echo dt-script -P DARECMIPPP --from  $outputDir --to $destDir