Building PHASTA on Cluster

From PHASTA Wiki
Jump to: navigation, search

Building PHASTA on Summit or Dell nodes

This page describes a workflow to build PHASTA on either Summit or Dell compute nodes.

  • The node request step is cluster-specific,
  • the compiler / MPI / CMake / PHASTA build steps are shared,
  • LAPACK is optional and is only needed for PHASTA branches or features that explicitly require it.

This guide is based on builds performed on PHASTA cluster nodes where both Dell and Summit nodes exposed the same basic GNU + MPI toolchain (gcc, g++, gfortran, mpicc, mpicxx, mpif90, cmake).

Overview

This workflow builds:

  • the PHASTA repository
  • optionally, a local LAPACK build for PHASTA branches that require it

1) Start an interactive compute node

Summit nodes

From jumpgate, request a Summit node with: 

qsub -I -l select=1:ncpus=24:mpiprocs=24 -l walltime=72:00:00 -l place=scatter:exclhost -q workq

You should land on a Summit node.

Dell nodes

An interactive request that lands on a Dell is: 

qsub -I -l select=1:ncpus=32:mpiprocs=32 -l walltime=72:00:00 -l place=scatter:exclhost -q workq

You should then land on a Dell node such as dell0.

2) Create a workspace

mkdir phasta_stack
cd phasta_stack

3) Clone the repositories

Navigate into phasta_stack and clone

PHASTA

git clone https://github.com/PHASTA/phasta.git

Reference LAPACK (optional)

Only do this if the PHASTA branch or feature you are building explicitly requires LAPACK. 

git clone https://github.com/Reference-LAPACK/lapack.git

4) Check the available toolchain

On either Summit or Dell nodes, verify the compiler and MPI wrappers: 

which gcc g++ gfortran
gcc --version
gfortran --version

which mpicc mpicxx mpif90
which cmake

Expected tools are the system GNU compilers and MPI wrappers.

5) Configure PHASTA

This is the default PHASTA configure path and assumes that the branch or feature being built does not require LAPACK. If the branch you are building explicitly depends on LAPACK, skip this default configure block and use the optional LAPACK section below instead.

This section uses a compressible-only PHASTA configuration. That is intentional: turning on incompressible mode without enabling an incompressible solver causes the configuration to fail.

From the PHASTA source directory: 

cd phasta_stack/phasta

Then configure: 

cmake \
  -DCMAKE_C_COMPILER=mpicc \
  -DCMAKE_CXX_COMPILER=mpicxx \
  -DCMAKE_Fortran_COMPILER=mpif90 \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_Fortran_FLAGS="-fallow-argument-mismatch" \
  -DPHASTA_INCOMPRESSIBLE=OFF \
  -DPHASTA_COMPRESSIBLE=ON \
  -DPHASTA_TESTING=OFF \
  .

Build PHASTA: 

make -j

6) Verify the build

A minimal verification step is simply to confirm that the PHASTA executable was produced: 

ls bin

You should see: 

phastaC.exe

You can also inspect the executable linkage: 

ldd ./bin/phastaC.exe | grep mpi

Optional: Build with LAPACK

Only use this path for PHASTA branches or features that explicitly require LAPACK.

Build LAPACK locally

A simple approach is to build Reference-LAPACK locally.

Important: build LAPACK in a separate build directory. Do not configure it inside the source tree.

Assuming you have cloned the LAPACK repository. 

cd phasta_stack
mkdir lapack_build
cd lapack_build

cmake \
  -DCMAKE_C_COMPILER=gcc \
  -DCMAKE_CXX_COMPILER=g++ \
  -DCMAKE_Fortran_COMPILER=gfortran \
  -DCMAKE_BUILD_TYPE=Release \
  -DBUILD_SHARED_LIBS=ON \
  ../lapack

make -j

After the build, inspect the generated libraries: 

ls lib/liblapack.so*
ls lib/libblas.so*

If your LAPACK build produced versioned shared libraries such as liblapack.so.X.Y.Z and libblas.so.X.Y.Z, use those exact filenames if needed.

Reconfigure PHASTA with LAPACK enabled

From the PHASTA source directory: 

cd phasta_stack/phasta

Then configure: 

cmake \
  -DCMAKE_C_COMPILER=mpicc \
  -DCMAKE_CXX_COMPILER=mpicxx \
  -DCMAKE_Fortran_COMPILER=mpif90 \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_Fortran_FLAGS="-fallow-argument-mismatch" \
  -DPHASTA_INCOMPRESSIBLE=OFF \
  -DPHASTA_COMPRESSIBLE=ON \
  -DPHASTA_USE_LAPACK=ON \
  -DPHASTA_TESTING=OFF \
  -DLAPACK_LIBRARIES=phasta_stack/lapack_build/lib/liblapack.so \
  -DBLAS_LIBRARIES=phasta_stack/lapack_build/lib/libblas.so \
  .

If liblapack.so or libblas.so do not exist as symlinks, replace them with the exact versioned filenames you found in ~/phasta_stack/lapack_build/lib/.

Build PHASTA again: 

make -j

Example PBS batch script for running PHASTA

The script below is a generic example for running PHASTA through PBS.

Before using it, replace the placeholder values such as:

  • <JOB_NAME>
  • <WALLTIME>
  • <NODES>
  • <RANKS_PER_NODE>
  • <TOTAL_MPI_RANKS>
  • <CASE_DIR>
  • <PHASTA_BUILD_DIR>


Note that PBS directives are read by the scheduler before the shell executes the script, so values in the #PBS lines must be edited directly. 

#!/bin/bash -l

#PBS -S /bin/bash
#PBS -N <JOB_NAME>
#PBS -l walltime=<WALLTIME>
#PBS -l select=<NODES>:ncpus=<RANKS_PER_NODE>:mpiprocs=<RANKS_PER_NODE>
#PBS -l place=scatter:exclhost
#PBS -j oe
#PBS -k n

# Move to the case directory
cd <CASE_DIR> || exit 1

# Extract numeric portion of PBS job ID
jid=$(echo "$PBS_JOBID" | awk -F "." '{print $1}')

# Total MPI ranks = number of nodes × ranks per node
nprocs=<TOTAL_MPI_RANKS>

# Path to the PHASTA build/config directory
export PHASTA_CONFIG=<PHASTA_BUILD_DIR>

# Remove old doubl* files if present
rm -f ${nprocs}-procs_case/doubl*

# Launch PHASTA
mpirun -np $nprocs --hostfile $PBS_NODEFILE \
       --mca btl tcp,self,vader \
       --bind-to core --map-by ppr:<RANKS_PER_NODE>:node \
       -x PHASTA_CONFIG \
       "$PHASTA_CONFIG/bin/phastaC.exe" 2>&1 | tee -a "${jid}.log"

Troubleshooting

LAPACK is optional

For most PHASTA branches, LAPACK is not needed. Do not build or enable LAPACK unless the branch or feature you are using explicitly requires it.

LAPACK refuses to configure in the source directory

Reference-LAPACK must be configured out-of-source.

PHASTA incompressible configuration error

If you configure with incompressible mode enabled, you may see an error saying that at least one incompressible solver must be enabled via PHASTA_USE_SVLS and/or PHASTA_USE_LESLIB.

If you only need the compressible build path, keep: 

-DPHASTA_INCOMPRESSIBLE=OFF
-DPHASTA_COMPRESSIBLE=ON

Reference

Retrieved from "https://fluid.colorado.edu/wiki/index.php?title=Building_PHASTA_on_Cluster&oldid=2197"