2. Building and Running the UFS Coastal

2.1. Supported Platforms and Compilers

Since the UFS Coastal is developed as a fork of the UFS Weather Model, the same platforms supported by the UFS Weather Model (see list of suported platforms) is also used by the UFS Coastal.

The prerequisite software libraries for building the UFS Coastal already exist in a centralized location on Level 1/pre configured systems, so users may skip dependency installation steps. On other systems, users will need to build the prerequisite libraries using spack-stack. The extensive documentation about installing dependencies through the spack-stack can be found in here.

2.2. Unsupported Platforms

For a complete walkthrough of Containerized EPIC Spack-Stack 1.6.0 for UFS Coastal on local and unsupported clusters (e.g., TACC Stampede3), see Running UFS Coastal on HPCs with Spack-Stack for full instructions.

2.3. Get Data

The ROMS input files for pre-configured cases can be downloaded from the ROMS Test GitHub repository.

Todo

Need to place UFS Coastal specific data to cloud or common place along with some level of versioning.

2.4. Downloading the UFS Coastal Model

To clone the main branch of the ufs-weather-model repository that includes UFS Coastal specific model component and update its submodules, execute the following commands:

git clone --recursive https://github.com/oceanmodeling/ufs-weather-model
cd ufs-weather-model

Compiling the model will take place within the ufs-weather-model directory created by this command.

2.5. Building the UFS Coastal

2.5.1. Loading the Required Modules

The process for loading modules is fairly straightforward on Level 1 supported platforms. Users may need to make adjustments when running on other systems.

2.5.1.1. On NOAA Level 1 & 2 Systems

Modulefiles for preconfigured platforms are located in modulefiles/ufs_<platform>.<compiler>. For example, to load the modules from the ufs-weather-model directory on MSU’s Hercules:

cd ufs-weather-model
module use modulefiles
module load ufs_hercules.intel

2.5.1.2. On Other Systems

If you are not running on one of the pre-configured platforms, you will need to install dependencies using spack-stack. The more detailed information about usage of spack-stack and dependency installation can be found in porting section.

2.5.2. Setting the CMAKE_FLAGS Environment Variable

The UFS Coastal can be built in one of several configurations (see Table 3.1 for common options). Also note that these configurations are specific to UFS Coastal and the wide range of supported configurations through the UFS Weather Model can be found in UFS Weather Model documentation Configurations section.

2.5.2.1. DATM+OCN Configurations

ADCIRC

export CMAKE_FLAGS="-DAPP=CSTLA -DADCIRC_CONFIG=PADCIRC -DCOUPLED=ON"

Note

The -DBUILD_ADCPREP option can be also provided to build ADCIRC pre-processing tools like adcprep command that allows the creation of input files. The -DBUILD_UTILITIES option can be also provided to build ADCIRC specific utility tools.

FVCOM

export CMAKE_FLAGS="-DAPP=CSTLF -DCOORDINATE_TYPE=SPHERICAL -DWET_DRY=ON"

Note

The -DAIR_PRESSURE option can also be provided to use surface air pressure as addtional forcing.

ROMS

export CMAKE_FLAGS="-DAPP=CSTLR -DMY_CPP_FLAGS=BULK_FLUXES"

Note

The ROMS ocean model builds the IRANE application by default. -DROMS_APP and ROMS_APP_DIR can be provided to build custom configurations. Mode information about the IRANE configuration (CDEPS data atmosphere coupled with ROMS) can be found in ROMS Test repository.

SCHISM

export CMAKE_FLAGS="-DAPP=CSTLS -DUSE_ATMOS=ON -DNO_PARMETIS=OFF -DOLDIO=ON"

Note

The -DBUILD_UTILS=ON option can be also provided to build SCHISM specific pre- and post-processing tools. The -DUSE_WW3=ON option can be also provided to couple with WW3 wave model.

2.5.2.2. DATM+WAV Configurations

WW3

export CMAKE_FLAGS="-DAPP=CSTLW -DPDLIB=ON"

Note

The same options can be used both for standalone WW3 configuration (standalone = true option needs to be provided in WAV_attributes section of ufs.configuration namelist file) and also the one coupled with CDEPS data atmosphere.

2.5.2.3. DATM+OCN+WAV Configurations

SCHISM

export CMAKE_FLAGS="-DAPP=CSTLSW -DUSE_ATMOS=ON -DUSE_WW3=ON -DNO_PARMETIS=OFF -DOLDIO=ON -DPDLIB=ON"

Note

The -DBUILD_UTILS=ON option can be also provided to build SCHISM specific pre- and post-processing tools.

2.5.3. Building the Model

The UFS Weather Model uses the CMake build system. There is a build script called build.sh in the top-level directory of the UFS Coastal repository that configures the build environment and runs the make command. This script also checks that all necessary environment variables have been set.

The UFS Coastal can be built by running the following command from the ufs-weather-model directory once CMAKE_FLAGS is set:

./build.sh

Once build.sh is finished, users should see the executable, named ufs_model, in the ufs-weather-model/build/ directory. If users prefer to build in a different directory, specify the BUILD_DIR environment variable. For example: export BUILD_DIR=test_cpld will build in the ufs-weather-model/test_cpld directory instead.

Expert help is available through GitHub Discussions. Users may post questions there for help with difficulties related to the UFS Coastal.

2.6. Running the Model

2.6.1. User Provided Configurations

Since the UFS Coastal does not have workflow capability in the application layer (UFS Coastal Application) yet, users need to populate namelist and input files manually. At this point, the best practice is to run UFS Coastal with a custom configuration/application is to run the similar configuration using UFS Coastal Regression Testing (RT) framework and populate the run directory. Then, the run directory can be used as a base to build custom configuration by replacing model and component specific configuration and input files. The more information about running UFS Coastal specific RTs can be found in the following section.

2.6.2. Pre-configured Configurations Using the Regression Test Script

Users can run a number of preconfigured UFS Coastal specific regression test cases from the rt_coastal.conf file (rt.conf includes RTs supported by UFS Weather Model) using the regression test script rt.sh in the tests directory. rt.sh is the top-level script that calls lower-level scripts to build specified UFS Coastal and UFS Weather Model configurations, set up environments, and run tests. This section aims to give brief information about running specific model configurations under UFS Coastal through the use of Regression Testing (RT) framework.

2.6.2.1. The rt_coastal.conf File

Each line in the PSV (Pipe-separated values) file, rt_coastal.conf, contains information to build and run the specific model configuration. The file includes two lines for each model configuration starting as COMPILE and RUN. The similar configurations could have a single COMPILE line but multiple RUN lines. In this case, a single compile step can be used to run multiple similar configurations.

COMPILE

Table 2.1 Description of Compile Section of rt_coastal.conf

Column

Description

1

COMPILE, It specifies the following information is to be used in setting up a compile job

2

It specifies the compile number. This is used as a reference for compile failures

3

Relates to the compiler to use in build (intel or gnu)

4

It specifies CMAKE options for the build. This is very similar to setting CMAKE_FLAGS to build model executable outside of the RT framework

5

Machines to run on (- is used to ignore specified machines, + is used to only run on specified machines)

6

Relates to the control of the compile job only if FV3 was present, previously used to run a test w/o compiling code. It can be set to fv3 in all the cases.

RUN

Table 2.2 Description of Run Section of rt_coastal.conf

Column

Description

1

RUN, It specifies following information is to be used in setting up a model run

2

Test name. The test in the tests/tests directory should be sourced

3

Machines to run on (- is used to ignore specified machines, + is used to only run on specified machines)

4

Controls whether the run creates its own baseline or it uses the baseline from a different (control) test

5

Test name to compare baselines with if not itself

The order of lines in rt_coastal.conf matters since rt.sh processes them sequentially; a RUN line should be preceded by a COMPILE line that builds the model used in the test. The following rt_coastal.conf file builds the ROMS ocean model coupled with CDEPS data atmosphere:

COMPILE | 16 | intel | -DAPP=CSTLR -DMY_CPP_FLAGS=BULK_FLUXES | | fv3 |
RUN | coastal_irene_atm2roms | | baseline |

The rt_coastal.conf file includes a large number of tests. If the user wants to run only specific tests, -n argument can be used. The -l rt_coastal.conf option can be used to run only UFS Coastal specific RTs. The rt.sh uses the rt.conf file by default.

2.6.2.2. The rt.sh File

This section contains additional information on command line options and troubleshooting for the rt.sh file.

To display detailed information on how to use rt.sh, users can simply run ./rt.sh, which will output the following options:

Usage: ./rt.sh -a <account> | -b <file> | -c | -d | -e | -h | -k | -l <file> | -m | -n <name> | -o | -r | -v | -w

  -a  <account> to use on for HPC queue
  -b  create new baselines only for tests listed in <file>
  -c  create new baseline results
  -d  delete run directories that are not used by other tests
  -e  use ecFlow workflow manager
  -h  display this help
  -k  keep run directory after rt.sh is completed
  -l  runs test specified in <file>
  -m  compare against new baseline results
  -n  run single test <name>
  -o  compile only, skip tests
  -r  use Rocoto workflow manager
  -v  verbose output
  -w  for weekly_test, skip comparing baseline results

When running a large number (10’s or 100’s) of tests, the -e or -r options can significantly decrease testing time by using a workflow manager (ecFlow or Rocoto, respectively) to queue the jobs according to dependencies and run them concurrently.

Note

Workflow Engine ecFlow is used with -e argument and Rocoto is used with -r argument. The Workflow Engine needs to be installed to the system to use these options. The Tier-1 platforms might have those workflow engines but Tear-2 level supported systems and custom installations might not have them.

Note

Since the UFS Coastal specific input files are not part of the UFS Weather Model input files, the location of the RT directory (defined by DISKNM variable) in rt.sh needs to be modified to run UFS Coastal specific RTs. To do that user needs to edit platform (i.e. Orion, Hercules) specific section of rt.sh and set DISKNM variable. For both Orion and Hercules platforms, /work2/noaa/nems/tufuk/RT directory is used to set DISKNM variable.

To run rt.sh using a custom configuration file and the Rocoto workflow manager:

./rt.sh -r -l rt_coastal.conf

To run a single test from custom configuration file:

Running with Intel compiler:
./rt.sh -l rt_coastal.conf -k -n "coastal_irene_atm2roms intel"

Running with GNU compiler:
./rt.sh -l rt_coastal.conf -k -n "coastal_irene_atm2roms gnu"

Note

-k argument is used to keep the run directory for further reference.

Note

-a argument can be used to specify account to job scheduler

The up-to-date list of supported and tested (the RTs that is indicated as bold) RTs can be seen in UFS Coastal specific UFS WM fork repository Wiki page.

2.6.2.2.1. Running UFS Coastal on HPCs with the Containerized EPIC Spack-Stack 1.6.0 Environment

This guide provides step-by-step instructions to compile and run the UFS Coastal Application on unsupported platforms (e.g., local machines, new HPC systems) using the EPIC Spack-Stack containerized environment (v1.6.0). It involves launching a container (via Apptainer or Singularity), building the UFS model, externalizing the executable, and running test cases outside the container.

Two workflow options are provided:

  • Option 1: Build and run from inside the container.

  • Option 2: Extract modulefiles and build/run externally.

Example commands are shown for Stampede3 (TACC), but they can be adapted to other systems.

2.6.2.3. Prerequisites

  • Apptainer or Singularity installed

  • HPC account with interactive compute access (idev, srun, etc.)

  • Familiarity with Git and environment module systems

2.6.2.4. Download the Container Image

wget https://epic-noaa.s3.us-east-1.amazonaws.com/ubuntu22.04-intel-ufs-env-v1.6.0.img
export img=/path/to/ubuntu22.04-intel-ufs-env-v1.6.0.img

2.6.2.5. Start an Interactive HPC Session (Stampede3 Example)

idev -m 120
ml tacc-apptainer/1.3.3

2.6.2.6. Check Loaded Modules (Example Output)

ml list
# cmake/3.31.5, python/3.9.18, intel/24.0, netcdf/4.9.2, metis/5.1.0.3, tacc-apptainer/1.3.3, ...

2.6.2.7. Option 1: Build and Run Inside the Container

  1. Enter the container

    singularity shell -e -s /bin/bash $img
# Optional
Apptainer> source /usr/lmod/lmod/init/bash

  2. Load UFS modules

    Apptainer> module use /opt/spack-stack/spack-stack-1.6.0/envs/unified-env/install/modulefiles/Core
Apptainer> module load stack-intel
Apptainer> module load stack-intel-oneapi-mpi
Apptainer> module load ufs-weather-model-env
Apptainer> module list

  3. Clone and build UFS Coastal

    Apptainer> git clone --recursive https://github.com/oceanmodeling/ufs-weather-model
Apptainer> cd ufs-weather-model
Apptainer> mkdir build && cd build
Apptainer> cmake -DAPP=CSTLS -DPDLIB=ON -DUSE_ATMOS=ON -DNO_PARMETIS=OFF -DOLDIO=ON -DUSE_WW3=OFF ..
Apptainer> make -j 8

  4. Externalize the executable

    Apptainer> /opt/container-scripts/externalize.sh -e $PWD/extern $PWD/ufs_model
# Result: build/extern/ufs_model (wrapper executable)

  5. Exit container and run externally

    Apptainer> exit
cd /path/to/test-case
ibrun -n 10 ../ufs-weather-model/build/extern/ufs_model
# or use srun/mpirun

2.6.2.8. Option 2: Externalize Modulefiles and Build Outside the Container

  1. Create a modulefiles directory

    mkdir modulefiles
cd modulefiles

  2. Extract modulefiles from the container

    singularity exec $img cp /opt/container-scripts/convert-modules.py .
python3 ./convert-modules.py -i $img -o $PWD/spack-stack-1.6.0

  3. Load externalized modules

    module use /path/to/modulefiles/spack-stack-1.6.0/Core
module load stack-intel
module load stack-intel-oneapi-mpi
module load ufs-weather-model-env sp crtm
module list

  4. Build UFS Coastal externally

    git clone --recursive https://github.com/oceanmodeling/ufs-weather-model
cd ufs-weather-model
mkdir build_external && cd build_external
cmake -DAPP=CSTLS -DPDLIB=ON -DUSE_ATMOS=ON -DNO_PARMETIS=OFF -DOLDIO=ON -DUSE_WW3=OFF ..
make -j 8

  5. Create executable wrapper

    make-external ./ufs_model
# Generates ufs_model (wrapper) and ufs_model.orig (original binary)

  6. Run test case externally

    cd /path/to/test-case
ibrun -n 10 ../ufs-weather-model/build_external/ufs_model

2.6.2.9. Summary

  • Option 1 builds inside the container and externalizes only the executable.

  • Option 2 extracts environment modules and builds entirely outside the container.

  • Both approaches allow unsupported systems to run the UFS Coastal Application using the EPIC Spack-Stack container.

2.6.2.10. Notes

  • Adjust ibrun -n or srun based on your platform and test case.

  • Replace /path/to/test-case with your actual directory.

  • This setup is portable to other clusters using Apptainer/Singularity (e.g., university or institutional HPCs).

2.6.2.11. Acknowledgments

This workflow builds upon techniques shared during a NOAA workshop led by Mark Potts. It was adapted and extended to support UFS Coastal compilation and execution on Stampede3.