Build PHASTA on Summit or Dell nodes

This page describes a workflow to build PHASTA on either Summit or Dell compute nodes.

The main idea is:

the node request step is cluster-specific,
the compiler / MPI / CMake / LAPACK / PHASTA build steps are shared.

This guide is based on builds performed on CU PHASTA cluster nodes where both Dell and Summit nodes exposed the same basic GNU + MPI toolchain (`gcc`, `g++`, `gfortran`, `mpicc`, `mpicxx`, `mpif90`, `cmake`).

Overview

This workflow builds:

the PHASTA repository
a local LAPACK/BLAS build used by PHASTA

1) Start an interactive compute node

Summit nodes

From `jumpgate`, request a Summit node with:

qsub -I -l select=1:ncpus=24:mpiprocs=24 -l walltime=72:00:00 -l place=scatter:exclhost -q workq

You should land on a Summit node.

Dell nodes

An interactive request that lands on a Dell is:

qsub -I -l select=1:ncpus=32:mpiprocs=32 -l walltime=72:00:00 -l place=scatter:exclhost -q workq

You should then land on a Dell node such as `dell0`.

2) Create a workspace

mkdir phasta_stack
cd phasta_stack

3) Clone the repositories

PHASTA

git clone https://github.com/PHASTA/phasta.git

Reference LAPACK

git clone https://github.com/Reference-LAPACK/lapack.git

4) Check the available toolchain

On either Summit or Dell nodes, verify the compiler and MPI wrappers:

which gcc g++ gfortran gcc --version gfortran --version

which mpicc mpicxx mpif90 which cmake

Expected tools are the system GNU compilers and MPI wrappers.

5) Build LAPACK locally

PHASTA needs BLAS/LAPACK. A simple approach is to build Reference-LAPACK locally.

Important: build LAPACK in a separate build directory. Do not configure it inside the source tree.

cd phasta_stack mkdir lapack_build cd lapack_build

cmake \
  -DCMAKE_C_COMPILER=gcc \
  -DCMAKE_Fortran_COMPILER=gfortran \
  -DCMAKE_BUILD_TYPE=Release \
  -DBUILD_SHARED_LIBS=ON \
  ../lapack

make -j32

After the build, inspect the generated libraries:

ls lib/liblapack.so* ls lib/libblas.so*

If your LAPACK build produced versioned shared libraries such as `liblapack.so.X.Y.Z` and `libblas.so.X.Y.Z`.

6) Configure PHASTA

This page uses a compressible-only PHASTA configuration.

That is intentional: turning on incompressible mode without enabling an incompressible solver causes configuration to fail.

From the PHASTA source directory:

cd phasta_stack/phasta

Then configure:

cmake \
  -DCMAKE_C_COMPILER=mpicc \
  -DCMAKE_CXX_COMPILER=mpicxx \
  -DCMAKE_Fortran_COMPILER=mpif90 \
  -DCMAKE_BUILD_TYPE=Debug \
  -DCMAKE_Fortran_FLAGS="-fallow-argument-mismatch" \
  -DPHASTA_INCOMPRESSIBLE=OFF \
  -DPHASTA_COMPRESSIBLE=ON \
  -DPHASTA_USE_LAPACK=ON \
  -DPHASTA_TESTING=OFF \
  -DLAPACK_LIBRARIES=$HOME/phasta_stack/lapack_build/lib/liblapack.so \
  -DBLAS_LIBRARIES=$HOME/phasta_stack/lapack_build/lib/libblas.so \
  .

If `liblapack.so` or `libblas.so` do not exist as symlinks, replace them with the exact versioned filenames you found in `~/phasta_stack/lapack_build/lib/`.

Build PHASTA

make -j32

On Summit, you may prefer to use 24 way parallelism instead:

make -j24

7) Verify the build

A minimal verification step is simply to confirm that the PHASTA executable was produced:

ls bin

You should see:

phastaC.exe

You can also inspect the executable linkage:

ldd ./bin/phastaC.exe | grep mpi

Troubleshooting

LAPACK refuses to configure in the source directory

Reference-LAPACK must be configured out-of-source.

PHASTA incompressible configuration error

If you configure with incompressible mode enabled, you may see an error saying that at least one incompressible solver must be enabled via `PHASTA_USE_SVLS` and/or `PHASTA_USE_LESLIB`.

If you only need the compressible build path, keep:

-DPHASTA_INCOMPRESSIBLE=OFF -DPHASTA_COMPRESSIBLE=ON

Reference

PHASTA repository: `https://github.com/PHASTA/phasta.git`
Reference LAPACK repository: `https://github.com/Reference-LAPACK/lapack.git`

Building PHASTA on Cluster

Contents