Difference between revisions of "Building PHASTA on Cluster"

From PHASTA Wiki
Jump to: navigation, search
(→‎6) Configure PHASTA)
Line 3: Line 3:
 
This page describes a workflow to build PHASTA on either Summit or Dell compute nodes.
 
This page describes a workflow to build PHASTA on either Summit or Dell compute nodes.
  
−
The main idea is:
+
* The node request step is cluster-specific,
 +
* the compiler / MPI / CMake / PHASTA build steps are shared,
 +
* LAPACK is '''optional''' and is only needed for PHASTA branches or features that explicitly require it.
  
−
* the '''node request''' step is cluster-specific,
+
This guide is based on builds performed on PHASTA cluster nodes where both Dell and Summit nodes exposed the same basic GNU + MPI toolchain (<code>gcc</code>, <code>g++</code>, <code>gfortran</code>, <code>mpicc</code>, <code>mpicxx</code>, <code>mpif90</code>, <code>cmake</code>).
−
* the '''compiler / MPI / CMake / LAPACK / PHASTA''' build steps are shared.
 
−
 
 
−
This guide is based on builds performed on CU PHASTA cluster nodes where both Dell and Summit nodes exposed the same basic GNU + MPI toolchain (`gcc`, `g++`, `gfortran`, `mpicc`, `mpicxx`, `mpif90`, `cmake`).
 
  
 
== Overview ==
 
== Overview ==
Line 15: Line 14:
  
 
* the PHASTA repository
 
* the PHASTA repository
−
* a local LAPACK/BLAS build used by PHASTA
+
* optionally, a local LAPACK/BLAS build for PHASTA branches that require it
  
 
== 1) Start an interactive compute node ==
 
== 1) Start an interactive compute node ==
Line 21: Line 20:
 
=== Summit nodes ===
 
=== Summit nodes ===
  
−
From `jumpgate`, request a Summit node with:
+
From <code>jumpgate</code>, request a Summit node with:
  
−
<code>
+
<pre>
 
qsub -I -l select=1:ncpus=24:mpiprocs=24 -l walltime=72:00:00 -l place=scatter:exclhost -q workq
 
qsub -I -l select=1:ncpus=24:mpiprocs=24 -l walltime=72:00:00 -l place=scatter:exclhost -q workq
−
</code>
+
</pre>
  
 
You should land on a Summit node.
 
You should land on a Summit node.
Line 33: Line 32:
 
An interactive request that lands on a Dell is:
 
An interactive request that lands on a Dell is:
  
−
<code>
+
<pre>
 
qsub -I -l select=1:ncpus=32:mpiprocs=32 -l walltime=72:00:00 -l place=scatter:exclhost -q workq
 
qsub -I -l select=1:ncpus=32:mpiprocs=32 -l walltime=72:00:00 -l place=scatter:exclhost -q workq
−
</code>
+
</pre>
  
−
You should then land on a Dell node such as `dell0`.
+
You should then land on a Dell node such as <code>dell0</code>.
  
 
== 2) Create a workspace ==
 
== 2) Create a workspace ==
  
−
<code>
+
<pre>
−
mkdir phasta_stack</code><br>
+
mkdir phasta_stack
−
<code>
 
 
cd phasta_stack
 
cd phasta_stack
−
</code>
+
</pre>
  
 
== 3) Clone the repositories ==
 
== 3) Clone the repositories ==
Line 51: Line 49:
 
=== PHASTA ===
 
=== PHASTA ===
  
−
<code>
+
<pre>
 
git clone https://github.com/PHASTA/phasta.git
 
git clone https://github.com/PHASTA/phasta.git
−
</code>
+
</pre>
 +
 
 +
=== Reference LAPACK (optional) ===
  
−
=== Reference LAPACK ===
+
Only do this if the PHASTA branch or feature you are building explicitly requires LAPACK/BLAS.
  
−
<code>
+
<pre>
 
git clone https://github.com/Reference-LAPACK/lapack.git
 
git clone https://github.com/Reference-LAPACK/lapack.git
−
</code>
+
</pre>
  
 
== 4) Check the available toolchain ==
 
== 4) Check the available toolchain ==
Line 65: Line 65:
 
On either Summit or Dell nodes, verify the compiler and MPI wrappers:
 
On either Summit or Dell nodes, verify the compiler and MPI wrappers:
  
−
<code>
+
<pre>
 
which gcc g++ gfortran
 
which gcc g++ gfortran
 
gcc --version
 
gcc --version
Line 72: Line 72:
 
which mpicc mpicxx mpif90
 
which mpicc mpicxx mpif90
 
which cmake
 
which cmake
−
</code>
+
</pre>
  
 
Expected tools are the system GNU compilers and MPI wrappers.
 
Expected tools are the system GNU compilers and MPI wrappers.
  
−
== 5) Build LAPACK locally ==
+
== 5) Configure PHASTA ==
 +
 
 +
This page uses a compressible-only PHASTA configuration.
 +
 
 +
That is intentional: turning on incompressible mode without enabling an incompressible solver causes configuration to fail.
 +
 
 +
From the PHASTA source directory:
 +
 
 +
<pre>
 +
cd phasta_stack/phasta
 +
</pre>
 +
 
 +
Then configure:
 +
 
 +
<pre>
 +
cmake \
 +
  -DCMAKE_C_COMPILER=mpicc \
 +
  -DCMAKE_CXX_COMPILER=mpicxx \
 +
  -DCMAKE_Fortran_COMPILER=mpif90 \
 +
  -DCMAKE_BUILD_TYPE=Release \
 +
  -DCMAKE_Fortran_FLAGS="-fallow-argument-mismatch" \
 +
  -DPHASTA_INCOMPRESSIBLE=OFF \
 +
  -DPHASTA_COMPRESSIBLE=ON \
 +
  -DPHASTA_TESTING=OFF \
 +
  .
 +
</pre>
 +
 
 +
Build PHASTA:
 +
 
 +
<pre>
 +
make -j32
 +
</pre>
 +
 
 +
On Summit, you may prefer to use 24 way parallelism instead:
 +
 
 +
<pre>
 +
make -j24
 +
</pre>
 +
 
 +
== 6) Verify the build ==
 +
 
 +
A minimal verification step is simply to confirm that the PHASTA executable was produced:
 +
 
 +
<pre>
 +
ls bin
 +
</pre>
 +
 
 +
You should see:
 +
 
 +
<pre>
 +
phastaC.exe
 +
</pre>
 +
 
 +
You can also inspect the executable linkage:
 +
 
 +
<pre>
 +
ldd ./bin/phastaC.exe | grep mpi
 +
</pre>
 +
 
 +
== Optional: Build with LAPACK ==
  
−
PHASTA needs BLAS/LAPACK. A simple approach is to build Reference-LAPACK locally.
+
Only use this path for PHASTA branches or features that explicitly require LAPACK/BLAS.
  
−
'''Important:''' build LAPACK in a separate build directory. Do '''not''' configure it inside the source tree.
+
=== Build LAPACK locally ===
  
−
<code>
+
A simple approach is to build Reference-LAPACK locally.
−
cd phasta_stack<br>
+
 
−
mkdir lapack_build<br>
+
'''Important''': build LAPACK in a separate build directory. Do '''not''' configure it inside the source tree.
−
cd lapack_build<br>
+
 
−
</code><br>
+
<pre>
 +
cd phasta_stack
 +
mkdir lapack_build
 +
cd lapack_build
 +
</pre>
  
 
<pre>
 
<pre>
Line 94: Line 157:
 
   -DCMAKE_BUILD_TYPE=Release \
 
   -DCMAKE_BUILD_TYPE=Release \
 
   -DBUILD_SHARED_LIBS=ON \
 
   -DBUILD_SHARED_LIBS=ON \
 +
  -DCBLAS=ON \
 +
  -DLAPACKE=ON \
 
   ../lapack
 
   ../lapack
 
</pre>
 
</pre>
  
−
<code>
+
<pre>
 
make -j32
 
make -j32
−
</code>
+
</pre>
  
 
After the build, inspect the generated libraries:
 
After the build, inspect the generated libraries:
  
−
<code>
+
<pre>
 
ls lib/liblapack.so*
 
ls lib/liblapack.so*
 
ls lib/libblas.so*
 
ls lib/libblas.so*
−
</code>
+
</pre>
−
 
 
−
If your LAPACK build produced versioned shared libraries such as `liblapack.so.X.Y.Z` and `libblas.so.X.Y.Z`.
 
−
 
 
−
== 6) Configure PHASTA ==
 
  
−
This page uses a '''compressible-only''' PHASTA configuration.
+
If your LAPACK build produced versioned shared libraries such as <code>liblapack.so.X.Y.Z</code> and <code>libblas.so.X.Y.Z</code>, use those exact filenames if needed.
  
−
That is intentional: turning on incompressible mode without enabling an incompressible solver causes configuration to fail.
+
=== Reconfigure PHASTA with LAPACK enabled ===
  
 
From the PHASTA source directory:
 
From the PHASTA source directory:
  
−
<code>
+
<pre>
−
cd phasta_stack/phasta
+
cd $HOME/phasta_stack/phasta
−
</code>
+
</pre>
  
−
Then configure:  
+
Then configure:
  
 
<pre>
 
<pre>
Line 140: Line 201:
 
</pre>
 
</pre>
  
−
If `liblapack.so` or `libblas.so` do not exist as symlinks, replace them with the exact versioned filenames you found in `~/phasta_stack/lapack_build/lib/`.
+
If <code>liblapack.so</code> or <code>libblas.so</code> do not exist as symlinks, replace them with the exact versioned filenames you found in <code>~/phasta_stack/lapack_build/lib/</code>.
  
−
Build PHASTA
+
Build PHASTA again:
  
−
<code>
+
<pre>
 
make -j32
 
make -j32
−
</code>
+
</pre>
  
 
On Summit, you may prefer to use 24 way parallelism instead:
 
On Summit, you may prefer to use 24 way parallelism instead:
  
−
<code>
+
<pre>
 
make -j24
 
make -j24
−
</code>
+
</pre>
  
−
== 7) Verify the build ==
+
== Troubleshooting ==
−
 
 
−
A minimal verification step is simply to confirm that the PHASTA executable was produced:
 
−
 
 
−
<code>
 
−
ls bin
 
−
</code>
 
  
−
You should see:
+
=== LAPACK is optional ===
  
−
<code>
+
For most PHASTA branches, LAPACK is not needed. Do not build or enable LAPACK unless the branch or feature you are using explicitly requires it.
−
phastaC.exe
 
−
</code>
 
−
 
 
−
You can also inspect the executable linkage:
 
−
 
 
−
<code>
 
−
ldd ./bin/phastaC.exe | grep mpi
 
−
</code>
 
−
 
 
−
== Troubleshooting ==
 
  
 
=== LAPACK refuses to configure in the source directory ===
 
=== LAPACK refuses to configure in the source directory ===
  
 
Reference-LAPACK must be configured out-of-source.
 
Reference-LAPACK must be configured out-of-source.
−
 
  
 
=== PHASTA incompressible configuration error ===
 
=== PHASTA incompressible configuration error ===
  
−
If you configure with incompressible mode enabled, you may see an error saying that at least one incompressible solver must be enabled via `PHASTA_USE_SVLS` and/or `PHASTA_USE_LESLIB`.
+
If you configure with incompressible mode enabled, you may see an error saying that at least one incompressible solver must be enabled via <code>PHASTA_USE_SVLS</code> and/or <code>PHASTA_USE_LESLIB</code>.
  
 
If you only need the compressible build path, keep:
 
If you only need the compressible build path, keep:
  
−
<code>
+
<pre>
 
-DPHASTA_INCOMPRESSIBLE=OFF
 
-DPHASTA_INCOMPRESSIBLE=OFF
 
-DPHASTA_COMPRESSIBLE=ON
 
-DPHASTA_COMPRESSIBLE=ON
−
</code>
+
</pre>
−
 
 
  
 
== Reference ==
 
== Reference ==
  
−
* PHASTA repository: https://github.com/PHASTA/phasta
+
* PHASTA repository: <code>https://github.com/PHASTA/phasta</code>
−
* Reference LAPACK repository: https://github.com/Reference-LAPACK/lapack
+
* Reference LAPACK repository: <code>https://github.com/Reference-LAPACK/lapack</code>

Revision as of 12:51, 14 April 2026

Build PHASTA on Summit or Dell nodes

This page describes a workflow to build PHASTA on either Summit or Dell compute nodes.

  • The node request step is cluster-specific,
  • the compiler / MPI / CMake / PHASTA build steps are shared,
  • LAPACK is optional and is only needed for PHASTA branches or features that explicitly require it.

This guide is based on builds performed on PHASTA cluster nodes where both Dell and Summit nodes exposed the same basic GNU + MPI toolchain (gcc, g++, gfortran, mpicc, mpicxx, mpif90, cmake).

Overview

This workflow builds:

  • the PHASTA repository
  • optionally, a local LAPACK/BLAS build for PHASTA branches that require it

1) Start an interactive compute node

Summit nodes

From jumpgate, request a Summit node with: 

qsub -I -l select=1:ncpus=24:mpiprocs=24 -l walltime=72:00:00 -l place=scatter:exclhost -q workq

You should land on a Summit node.

Dell nodes

An interactive request that lands on a Dell is: 

qsub -I -l select=1:ncpus=32:mpiprocs=32 -l walltime=72:00:00 -l place=scatter:exclhost -q workq

You should then land on a Dell node such as dell0.

2) Create a workspace

mkdir phasta_stack
cd phasta_stack

3) Clone the repositories

PHASTA

git clone https://github.com/PHASTA/phasta.git

Reference LAPACK (optional)

Only do this if the PHASTA branch or feature you are building explicitly requires LAPACK/BLAS. 

git clone https://github.com/Reference-LAPACK/lapack.git

4) Check the available toolchain

On either Summit or Dell nodes, verify the compiler and MPI wrappers: 

which gcc g++ gfortran
gcc --version
gfortran --version

which mpicc mpicxx mpif90
which cmake

Expected tools are the system GNU compilers and MPI wrappers.

5) Configure PHASTA

This page uses a compressible-only PHASTA configuration.

That is intentional: turning on incompressible mode without enabling an incompressible solver causes configuration to fail.

From the PHASTA source directory: 

cd phasta_stack/phasta

Then configure: 

cmake \
  -DCMAKE_C_COMPILER=mpicc \
  -DCMAKE_CXX_COMPILER=mpicxx \
  -DCMAKE_Fortran_COMPILER=mpif90 \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_Fortran_FLAGS="-fallow-argument-mismatch" \
  -DPHASTA_INCOMPRESSIBLE=OFF \
  -DPHASTA_COMPRESSIBLE=ON \
  -DPHASTA_TESTING=OFF \
  .

Build PHASTA: 

make -j32

On Summit, you may prefer to use 24 way parallelism instead: 

make -j24

6) Verify the build

A minimal verification step is simply to confirm that the PHASTA executable was produced: 

ls bin

You should see: 

phastaC.exe

You can also inspect the executable linkage: 

ldd ./bin/phastaC.exe | grep mpi

Optional: Build with LAPACK

Only use this path for PHASTA branches or features that explicitly require LAPACK/BLAS.

Build LAPACK locally

A simple approach is to build Reference-LAPACK locally.

Important: build LAPACK in a separate build directory. Do not configure it inside the source tree. 

cd phasta_stack
mkdir lapack_build
cd lapack_build

cmake \
  -DCMAKE_C_COMPILER=gcc \
  -DCMAKE_Fortran_COMPILER=gfortran \
  -DCMAKE_BUILD_TYPE=Release \
  -DBUILD_SHARED_LIBS=ON \
  -DCBLAS=ON \
  -DLAPACKE=ON \
  ../lapack

make -j32

After the build, inspect the generated libraries: 

ls lib/liblapack.so*
ls lib/libblas.so*

If your LAPACK build produced versioned shared libraries such as liblapack.so.X.Y.Z and libblas.so.X.Y.Z, use those exact filenames if needed.

Reconfigure PHASTA with LAPACK enabled

From the PHASTA source directory: 

cd $HOME/phasta_stack/phasta

Then configure: 

cmake \
  -DCMAKE_C_COMPILER=mpicc \
  -DCMAKE_CXX_COMPILER=mpicxx \
  -DCMAKE_Fortran_COMPILER=mpif90 \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_Fortran_FLAGS="-fallow-argument-mismatch" \
  -DPHASTA_INCOMPRESSIBLE=OFF \
  -DPHASTA_COMPRESSIBLE=ON \
  -DPHASTA_USE_LAPACK=ON \
  -DPHASTA_TESTING=OFF \
  -DLAPACK_LIBRARIES=$HOME/phasta_stack/lapack_build/lib/liblapack.so \
  -DBLAS_LIBRARIES=$HOME/phasta_stack/lapack_build/lib/libblas.so \
  .

If liblapack.so or libblas.so do not exist as symlinks, replace them with the exact versioned filenames you found in ~/phasta_stack/lapack_build/lib/.

Build PHASTA again: 

make -j32

On Summit, you may prefer to use 24 way parallelism instead: 

make -j24

Troubleshooting

LAPACK is optional

For most PHASTA branches, LAPACK is not needed. Do not build or enable LAPACK unless the branch or feature you are using explicitly requires it.

LAPACK refuses to configure in the source directory

Reference-LAPACK must be configured out-of-source.

PHASTA incompressible configuration error

If you configure with incompressible mode enabled, you may see an error saying that at least one incompressible solver must be enabled via PHASTA_USE_SVLS and/or PHASTA_USE_LESLIB.

If you only need the compressible build path, keep: 

-DPHASTA_INCOMPRESSIBLE=OFF
-DPHASTA_COMPRESSIBLE=ON

Reference

Retrieved from "https://fluid.colorado.edu/wiki/index.php?title=Building_PHASTA_on_Cluster&oldid=2163"