5. Porting UFS Coastal Model

Porting UFS Coastal Model to an unsupported platform involves multiple steps. Similar to the UFS Weather Model, the UFS Coastal Model also uses spack-stack package manager to manage model level dependencies. In addition to the standard list of dependencies required by the UFS Weather Model, the UFS Coastal Model has extra dependencies like ParMETIS and PROJ libraries. The addtional components (i.e. ADCIRC, FVCOM) are able to install extra dependecies by their own build interface if the required libraries are not provided by the system or spack-stack installation.

The model dependencies are pre-configured and bult on Tier 1 systems, so users could run the model in those plaotforms. The Tier 1 & 2 systems (e.g., Hera, Orion, Hercules and Derecho) are generally restricted to those with access through NOAA and its affiliates. To that end, the UFS Coastal is ported to TACC Frontera system support coastal modeling community and users but it is still testing phase and experimental.

5.1. Additionally Supported Platforms

5.1.1. TACC Frontera

This platform is not fully supported yet and still in testing phase. It uses spack-stack version 1.5.1 and supports only Intel compiler (19.1.1.217) along with Intel MPI (2020.4.304). The module file ufs_frontera.intel.lua is also included to the UFS Coastal code base along with update in RT framework to enable experimenting the new platform. There is a plan to update spack-stack version and also used compiler and MPI versions.

5.2. Installing Dependencies

5.2.1. Docker Container

A Docker container image is a lightweight, standalone, executable package of software that includes everything needed to run an application: code, runtime, system tools, system libraries and settings. It enables creating development and testing environment that run on local resources.

The Docker Desktop is required to build and run the Docker containers.

The following instructions can be used to create a new Docker container from scratch but the pre-build container can be also used to run UFS Coastal. There is a plan to make available pre-configured Docker Containers for UFS Coastal through the DockerHub.

Run a new Docker container

docker run -it ubuntu:latest

Note

In this case, the command will create a Docker Container that uses Ubuntu OS.

Note

The optional --rm argument can be used when the container needed to be deleted after the task for it is complete. The optional --platform argument can be used to emulate specific platform (i.e. linux/amd64). This needs to be used carefully since it could slow down the container.

Install base development environment with Ubuntu package manager (apt).

apt-get -yqq update
apt-get -yqq upgrade
apt-get install --no-install-recommends -yqq \
gcc \
g++ \
gfortran \
build-essential \
libkrb5-dev \
m4 \
git \
git-lfs \
vim \
bzip2 \
unzip \
tar \
automake \
xterm \
texlive \
libcurl4-openssl-dev \
libssl-dev \
meson \
python3-dev  \
python3-pip \
wget  \
rsync \
tcl  \
tcl-dev \
openssh-client

Clone Spack-stack

In this case, /opt directory will be used to install the dependencies but there is no any restriction to use another directory for installation. Also, spack-stack 1.5.1 is used as an example.

cd /opt
git clone --recurse-submodules https://github.com/jcsda/spack-stack.git
cd spack-stack
git checkout 1.5.1
git submodule update --init --recursive
export SPACK_ROOT=/opt/spack-stack/spack
source setup.sh

Create spack environment and activate it

spack stack create env --site linux.default --template ufs-weather-model --name ufs.local --prefix /opt/ufs.local
cd envs/ufs.local
spack env activate .

Note

spack stack create env -h shows the list of available environments.

Find external packages that are shipped through the OS and installed with apt package manager in Step (2).

cd /opt/spack-stack
export SPACK_SYSTEM_CONFIG_PATH="/opt/spack-stack/envs/ufs.local/site"
spack external find --scope system --exclude bison --exclude cmake --exclude curl --exclude openssl --exclude openssh
spack external find --scope system perl
spack external find --scope system wget
spack external find --scope system texlive
spack compiler find --scope system
unset SPACK_SYSTEM_CONFIG_PATH

Set compiler and MPI library that will be used to install dependencies. In this example, GNU compiler and OpenMPI are used. The compiler and MPI library versions could change based on the version of the used OS. To that end, user needs to check compiler and MPI versions carefully. Following commands give some exmaple to find out the versions and addind them to the spack-stack configuration.

gcc --version
spack config add "packages:all:compiler:[gcc@YOUR-VERSION]"
# Example
spack config add "packages:all:compiler:[gcc@11.4.0]"

# Example for Red Hat 8 following the above instructions
spack config add "packages:all:providers:mpi:[openmpi@4.1.6]"

# Example for Ubuntu 20.04 or 22.04 following the above instructions
spack config add "packages:all:providers:mpi:[mpich@4.1.1]"

Set few more package variants

spack config add "packages:fontconfig:variants:+pic"
spack config add "packages:pixman:variants:+pic"
spack config add "packages:cairo:variants:+pic"
spack config add "packages:libffi:version:[3.3]"

Add additional dependencies to newly created spack environment for UFS Coastal

spack add parmetis@4.0.3~shared ^metis~shared
spack add proj@4.9.2~shared~tiff

Note

Since parmetis package depends on metis, it will also install metis@5.1.0 spack package. Also note that metis@5.1.0 is not compatible with ADCIRC and throw error like undefined reference to metis_estimatememory_.

Note

The proj package had bug in spack-stack 1.4.1 version. This is fixed in recent version of the spack-stack (>= 1.5.1`) but spack submodule can be updated under spack-stack for older versions with the following instructions. Again, this is not needed for newer versions of spack-stack.

cd spack-stack/spack
git remote add upstream https://github.com/spack/spack.git
git fetch upstream
git cherry-pick upsream 149d194 (and resolve conflicts)

Install packages and clean unneccecary files

spack concretize 2>&1 | tee log.concretize
spack install --source 2>&1 | tee log.install
spack gc -y  2>&1 | tee log.clean

(Optional) To create module files for the dependencies, lmod and lua needs to be installed. Following commands can be used to install them.

To install lua`:

cd /opt
export LUA_VERSION=5.1.4.9
wget -c https://sourceforge.net/projects/lmod/files/lua-${LUA_VERSION}.tar.bz2
tar -xvf lua-${LUA_VERSION}.tar.bz2
cd lua-${LUA_VERSION}
./configure --prefix=/opt/lua
make
make install

To install lmod`:

cd /opt
export LMOD_VERSION=8.7
wget -c https://sourceforge.net/projects/lmod/files/Lmod-${LMOD_VERSION}.tar.bz2
tar -xvf Lmod-${LMOD_VERSION}.tar.bz2
cd Lmod-${LMOD_VERSION}
./configure --prefix=/opt/lmod --with-lua=/opt/lua/bin/lua --with-luac=/opt/lua/bin/luac --with-lua_include=/opt/lua/include
make
make install

Then, lmod can be activated with following commands:

source /opt/lmod/lmod/${LMOD_VERSION}/init/profile
ln -sf /opt/lmod/lmod/${LMOD_VERSION}/init/profile /etc/profile.d/lmod.sh

After instaling lmod successfully, the module files for dependencies, which are installed by spack-stack can be created with following command,

spack module tcl refresh -y
spack stack setup-meta-modules

(Optional) Testing module files that are created in Step (10)

. /etc/profile.d/modules.sh
module use /opt/ufs.local/modulefiles/Core
module load stack-gcc/11.4.0
module load stack-python/3.10.8
module load stack-openmpi/4.1.6

Note

Module names can be different if different versions of GNU compiler, MPI library and Python are installed.

Clone UFS Coastal Model and create new module file to support custom Docker installation

cd /opt
git clone -b feature/coastal_app --recursive https://github.com/oceanmodeling/ufs-coastal.git
cd ufs-coastal/modulefiles
vi ufs_local.gnu

Content of ufs_local.gnu module file:

#%Module

proc ModulesHelp {} {
  puts stderr "\tcit - loads modules required for building and running UFS Model on Linux/GNU"
}

module-whatis "loads UFS Model prerequisites for Linux/GNU"

module use /opt/ufs.local/modulefiles/Core

module load stack-gcc/11.4.0
module load stack-python/3.10.8
module load stack-openmpi/4.1.6

module load jasper/2.0.32
module load zlib/1.2.13
module load libpng/1.6.37
module load hdf5/1.14.0
module load netcdf-c/4.9.2
module load netcdf-fortran/4.6.0
module load parallelio/2.5.9
module load esmf/8.4.2
module load fms/2023.01
module load bacio/2.4.1
module load crtm/2.4.0
module load g2/3.4.5
module load g2tmpl/1.10.2
module load ip/3.3.3
module load sp/2.3.3
module load w3emc/2.9.2
module load gftl-shared/1.5.0
module load mapl/2.35.2-esmf-8.4.2
module load scotch/7.0.3

module load metis/5.1.0
module load parmetis/4.0.3
module load proj/4.9.2

setenv METIS_ROOT $env(metis_ROOT)
setenv PARMETIS_ROOT $env(parmetis_ROOT)
setenv PROJ_ROOT $env(proj_ROOT)

setenv CC  mpicc
setenv CXX mpicxx
setenv F77 mpif77
setenv F90 mpif90
setenv FC  mpif90
setenv CMAKE_Platform linux.gnu

Testing build of UFS Coastal specific configurations

cd ufs-coastal/tests

# ADCIRC (DATM+OCN, ADCIRC is not compatible with Metis 5.x at this point)
./compile.sh "local" "-DAPP=CSTLA -DBUILD_ADCPREP=ON -DADCIRC_CONFIG=PADCIRC -DCOUPLED=ON" coastal gnu NO NO

# FVCOM (DATM+OCN)
./compile.sh "local" "-DAPP=CSTLF -DCOORDINATE_TYPE=SPHERICAL -DWET_DRY=ON" coastal gnu NO NO

# SCHISM (DATM+OCN)
./compile.sh "local" "-DAPP=CSTLS -DUSE_ATMOS=ON -DNO_PARMETIS=OFF -DOLDIO=ON" coastal gnu NO NO

Note

In addition to the steps that are explained above, the UFS Coastal RT system needs to be adjusted to support running RT under Docker container. This part is not covered by the documentation.