View source for Building PHASTA on Cluster

= Building PHASTA on Summit or Dell nodes =

This page describes a workflow to build PHASTA on either Summit or Dell compute nodes.

* The node request step is cluster-specific,
* the compiler / MPI / CMake / PHASTA build steps are shared,
* LAPACK is '''optional''' and is only needed for PHASTA branches or features that explicitly require it.

This guide is based on builds performed on PHASTA cluster nodes where both Dell and Summit nodes exposed the same basic GNU + MPI toolchain (<code>gcc</code>, <code>g++</code>, <code>gfortran</code>, <code>mpicc</code>, <code>mpicxx</code>, <code>mpif90</code>, <code>cmake</code>).

== Overview ==

This workflow builds:

* the PHASTA repository
* optionally, a local LAPACK build for PHASTA branches that require it

== 1) Start an interactive compute node ==

=== Summit nodes ===

From <code>jumpgate</code>, request a Summit node with:

<pre>
qsub -I -l select=1:ncpus=24:mpiprocs=24 -l walltime=72:00:00 -l place=scatter:exclhost -q workq
</pre>

You should land on a Summit node.

=== Dell nodes ===

An interactive request that lands on a Dell is:

<pre>
qsub -I -l select=1:ncpus=32:mpiprocs=32 -l walltime=72:00:00 -l place=scatter:exclhost -q workq
</pre>

You should then land on a Dell node such as <code>dell0</code>.

== 2) Create a workspace ==

<pre>
mkdir phasta_stack
cd phasta_stack
</pre>

== 3) Clone the repositories ==

Navigate into <code>phasta_stack</code> and clone

=== PHASTA ===

<pre>
git clone https://github.com/PHASTA/phasta.git
</pre>

=== Reference LAPACK (optional) ===

Only do this if the PHASTA branch or feature you are building explicitly requires LAPACK.

<pre>
git clone https://github.com/Reference-LAPACK/lapack.git
</pre>

== 4) Check the available toolchain ==

On either Summit or Dell nodes, verify the compiler and MPI wrappers:

<pre>
which gcc g++ gfortran
gcc --version
gfortran --version

which mpicc mpicxx mpif90
which cmake
</pre>

Expected tools are the system GNU compilers and MPI wrappers.

== 5) Configure PHASTA ==

This is the default PHASTA configure path and assumes that the branch or feature being built does <b>not</b> require LAPACK. If the branch you are building explicitly depends on LAPACK, skip this default configure block and use the optional LAPACK section below instead.

This section uses a compressible-only PHASTA configuration. That is intentional: turning on incompressible mode without enabling an incompressible solver causes the configuration to fail.

From the PHASTA source directory:

<pre>
cd phasta_stack/phasta
</pre>

Then configure:

<pre>
cmake \
  -DCMAKE_C_COMPILER=mpicc \
  -DCMAKE_CXX_COMPILER=mpicxx \
  -DCMAKE_Fortran_COMPILER=mpif90 \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_Fortran_FLAGS="-fallow-argument-mismatch" \
  -DPHASTA_INCOMPRESSIBLE=OFF \
  -DPHASTA_COMPRESSIBLE=ON \
  -DPHASTA_TESTING=OFF \
  .
</pre>

Build PHASTA:

<pre>
make -j
</pre>

== 6) Verify the build ==

A minimal verification step is simply to confirm that the PHASTA executable was produced:

<pre>
ls bin
</pre>

You should see:

<pre>
phastaC.exe
</pre>

You can also inspect the executable linkage:

<pre>
ldd ./bin/phastaC.exe | grep mpi
</pre>

== Optional: Build with LAPACK ==

Only use this path for PHASTA branches or features that explicitly require LAPACK.

=== Build LAPACK locally ===

A simple approach is to build Reference-LAPACK locally.

'''Important''': build LAPACK in a separate build directory. Do '''not''' configure it inside the source tree.

Assuming you have cloned the LAPACK repository.

<pre>
cd phasta_stack
mkdir lapack_build
cd lapack_build
</pre>

<pre>
cmake \
  -DCMAKE_C_COMPILER=gcc \
  -DCMAKE_CXX_COMPILER=g++ \
  -DCMAKE_Fortran_COMPILER=gfortran \
  -DCMAKE_BUILD_TYPE=Release \
  -DBUILD_SHARED_LIBS=ON \
  ../lapack
</pre>

<pre>
make -j
</pre>

After the build, inspect the generated libraries:

<pre>
ls lib/liblapack.so*
ls lib/libblas.so*
</pre>

If your LAPACK build produced versioned shared libraries such as <code>liblapack.so.X.Y.Z</code> and <code>libblas.so.X.Y.Z</code>, use those exact filenames if needed.

=== Reconfigure PHASTA with LAPACK enabled ===

From the PHASTA source directory:

<pre>
cd phasta_stack/phasta
</pre>

Then configure:

<pre>
cmake \
  -DCMAKE_C_COMPILER=mpicc \
  -DCMAKE_CXX_COMPILER=mpicxx \
  -DCMAKE_Fortran_COMPILER=mpif90 \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_Fortran_FLAGS="-fallow-argument-mismatch" \
  -DPHASTA_INCOMPRESSIBLE=OFF \
  -DPHASTA_COMPRESSIBLE=ON \
  -DPHASTA_USE_LAPACK=ON \
  -DPHASTA_TESTING=OFF \
  -DLAPACK_LIBRARIES=phasta_stack/lapack_build/lib/liblapack.so \
  -DBLAS_LIBRARIES=phasta_stack/lapack_build/lib/libblas.so \
  .
</pre>

If <code>liblapack.so</code> or <code>libblas.so</code> do not exist as symlinks, replace them with the exact versioned filenames you found in <code>~/phasta_stack/lapack_build/lib/</code>.

Build PHASTA again:

<pre>
make -j
</pre>

== Example PBS batch script for running PHASTA ==

The script below is a generic example for running PHASTA through PBS.

Before using it, replace the placeholder values such as:

* <code>&lt;JOB_NAME&gt;</code>
* <code>&lt;WALLTIME&gt;</code>
* <code>&lt;NODES&gt;</code>
* <code>&lt;RANKS_PER_NODE&gt;</code>
* <code>&lt;TOTAL_MPI_RANKS&gt;</code>
* <code>&lt;CASE_DIR&gt;</code>
* <code>&lt;PHASTA_BUILD_DIR&gt;</code>


Note that PBS directives are read by the scheduler before the shell executes the script, so values in the <code>#PBS</code> lines must be edited directly.

<pre>
#!/bin/bash -l

#PBS -S /bin/bash
#PBS -N <JOB_NAME>
#PBS -l walltime=<WALLTIME>
#PBS -l select=<NODES>:ncpus=<RANKS_PER_NODE>:mpiprocs=<RANKS_PER_NODE>
#PBS -l place=scatter:exclhost
#PBS -j oe
#PBS -k n

# Move to the case directory
cd <CASE_DIR> || exit 1

# Extract numeric portion of PBS job ID
jid=$(echo "$PBS_JOBID" | awk -F "." '{print $1}')

# Total MPI ranks = number of nodes × ranks per node
nprocs=<TOTAL_MPI_RANKS>

# Path to the PHASTA build/config directory
export PHASTA_CONFIG=<PHASTA_BUILD_DIR>

# Remove old doubl* files if present
rm -f ${nprocs}-procs_case/doubl*

# Launch PHASTA
mpirun -np $nprocs --hostfile $PBS_NODEFILE \
       --mca btl tcp,self,vader \
       --bind-to core --map-by ppr:<RANKS_PER_NODE>:node \
       -x PHASTA_CONFIG \
       "$PHASTA_CONFIG/bin/phastaC.exe" 2>&1 | tee -a "${jid}.log"
</pre>

== Troubleshooting ==

=== LAPACK is optional ===

For most PHASTA branches, LAPACK is not needed. Do not build or enable LAPACK unless the branch or feature you are using explicitly requires it.

=== LAPACK refuses to configure in the source directory ===

Reference-LAPACK must be configured out-of-source.

=== PHASTA incompressible configuration error ===

If you configure with incompressible mode enabled, you may see an error saying that at least one incompressible solver must be enabled via <code>PHASTA_USE_SVLS</code> and/or <code>PHASTA_USE_LESLIB</code>.

If you only need the compressible build path, keep:

<pre>
-DPHASTA_INCOMPRESSIBLE=OFF
-DPHASTA_COMPRESSIBLE=ON
</pre>

== Reference ==

* PHASTA repository: https://github.com/PHASTA/phasta
* Reference LAPACK repository: https://github.com/Reference-LAPACK/lapack