= Building PHASTA on Summit or Dell nodes =

This page describes a workflow to build PHASTA on either Summit or Dell compute nodes.

* The node request step is cluster-specific,
* the compiler / MPI / CMake / PHASTA build steps are shared,
* LAPACK is '''optional''' and is only needed for PHASTA branches or features that explicitly require it.

This guide is based on builds performed on PHASTA cluster nodes where both Dell and Summit nodes exposed the same basic GNU + MPI toolchain (<code>gcc</code>, <code>g++</code>, <code>gfortran</code>, <code>mpicc</code>, <code>mpicxx</code>, <code>mpif90</code>, <code>cmake</code>).

== Overview ==

This workflow builds:

* the PHASTA repository
* optionally, a local LAPACK build for PHASTA branches that require it

== 1) Start an interactive compute node ==

=== Summit nodes ===

From <code>jumpgate</code>, request a Summit node with:

<pre>
qsub -I -l select=1:ncpus=24:mpiprocs=24 -l walltime=72:00:00 -l place=scatter:exclhost -q workq
</pre>

You should land on a Summit node.

=== Dell nodes ===

An interactive request that lands on a Dell is:

<pre>
qsub -I -l select=1:ncpus=32:mpiprocs=32 -l walltime=72:00:00 -l place=scatter:exclhost -q workq
</pre>

You should then land on a Dell node such as <code>dell0</code>.

== 2) Create a workspace ==

<pre>
mkdir phasta_stack
cd phasta_stack
</pre>

== 3) Clone the repositories ==

Navigate into <codephasta_stack</code and clone

=== PHASTA ===

<pre>
git clone https://github.com/PHASTA/phasta.git
</pre>

=== Reference LAPACK (optional) ===

Only do this if the PHASTA branch or feature you are building explicitly requires LAPACK.

<pre>
git clone https://github.com/Reference-LAPACK/lapack.git
</pre>

== 4) Check the available toolchain ==

On either Summit or Dell nodes, verify the compiler and MPI wrappers:

<pre>
which gcc g++ gfortran
gcc --version
gfortran --version

which mpicc mpicxx mpif90
which cmake
</pre>

Expected tools are the system GNU compilers and MPI wrappers.

== 5) Configure PHASTA ==

This is the default PHASTA configure path and assumes that the branch or feature being built does not require LAPACK. If the branch you are building explicitly depends on LAPACK, skip this default configure block and use the optional LAPACK section below instead.

This section uses a compressible-only PHASTA configuration. That is intentional: turning on incompressible mode without enabling an incompressible solver causes the configuration to fail.

From the PHASTA source directory:

<pre>
cd phasta_stack/phasta
</pre>

Then configure:

<pre>
cmake \
-DCMAKE_C_COMPILER=mpicc \
-DCMAKE_CXX_COMPILER=mpicxx \
-DCMAKE_Fortran_COMPILER=mpif90 \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_Fortran_FLAGS="-fallow-argument-mismatch" \
-DPHASTA_INCOMPRESSIBLE=OFF \
-DPHASTA_COMPRESSIBLE=ON \
-DPHASTA_TESTING=OFF \
.
</pre>

Build PHASTA:

<pre>
make -j32
</pre>

On Summit, you may prefer to use 24 way parallelism instead:

<pre>
make -j24
</pre>

== 6) Verify the build ==

A minimal verification step is simply to confirm that the PHASTA executable was produced:

<pre>
ls bin
</pre>

You should see:

<pre>
phastaC.exe
</pre>

You can also inspect the executable linkage:

<pre>
ldd ./bin/phastaC.exe | grep mpi
</pre>

== Optional: Build with LAPACK ==

Only use this path for PHASTA branches or features that explicitly require LAPACK.

=== Build LAPACK locally ===

A simple approach is to build Reference-LAPACK locally.

'''Important''': build LAPACK in a separate build directory. Do '''not''' configure it inside the source tree.

Assuming you have cloned the LAPACK repository.

<pre>
cd phasta_stack
mkdir lapack_build
cd lapack_build
</pre>

<pre>
cmake \
-DCMAKE_C_COMPILER=gcc \
-DCMAKE_CXX_COMPILER=g++ \
-DCMAKE_Fortran_COMPILER=gfortran \
-DCMAKE_BUILD_TYPE=Release \
-DBUILD_SHARED_LIBS=ON \
../lapack
</pre>

<pre>
make -j32
</pre>

After the build, inspect the generated libraries:

<pre>
ls lib/liblapack.so*
ls lib/libblas.so*
</pre>

If your LAPACK build produced versioned shared libraries such as <code>liblapack.so.X.Y.Z</code> and <code>libblas.so.X.Y.Z</code>, use those exact filenames if needed.

=== Reconfigure PHASTA with LAPACK enabled ===

From the PHASTA source directory:

<pre>
cd $HOME/phasta_stack/phasta
</pre>

Then configure:

<pre>
cmake \
-DCMAKE_C_COMPILER=mpicc \
-DCMAKE_CXX_COMPILER=mpicxx \
-DCMAKE_Fortran_COMPILER=mpif90 \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_Fortran_FLAGS="-fallow-argument-mismatch" \
-DPHASTA_INCOMPRESSIBLE=OFF \
-DPHASTA_COMPRESSIBLE=ON \
-DPHASTA_USE_LAPACK=ON \
-DPHASTA_TESTING=OFF \
-DLAPACK_LIBRARIES=$HOME/phasta_stack/lapack_build/lib/liblapack.so \
-DBLAS_LIBRARIES=$HOME/phasta_stack/lapack_build/lib/libblas.so \
.
</pre>

If <code>liblapack.so</code> or <code>libblas.so</code> do not exist as symlinks, replace them with the exact versioned filenames you found in <code>~/phasta_stack/lapack_build/lib/</code>.

Build PHASTA again:

<pre>
make -j32
</pre>

On Summit, you may prefer to use 24 way parallelism instead:

<pre>
make -j24
</pre>

== Troubleshooting ==

=== LAPACK is optional ===

For most PHASTA branches, LAPACK is not needed. Do not build or enable LAPACK unless the branch or feature you are using explicitly requires it.

=== LAPACK refuses to configure in the source directory ===

Reference-LAPACK must be configured out-of-source.

=== PHASTA incompressible configuration error ===

If you configure with incompressible mode enabled, you may see an error saying that at least one incompressible solver must be enabled via <code>PHASTA_USE_SVLS</code> and/or <code>PHASTA_USE_LESLIB</code>.

If you only need the compressible build path, keep:

<pre>
-DPHASTA_INCOMPRESSIBLE=OFF
-DPHASTA_COMPRESSIBLE=ON
</pre>

== Reference ==

* PHASTA repository: <code>https://github.com/PHASTA/phasta</code>
* Reference LAPACK repository: <code>https://github.com/Reference-LAPACK/lapack</code>

Build PHASTA on Cluster

2026-04-14T19:46:08Z

Moam3950: Undo revision 2176 by Moam3950 (talk)

#REDIRECT [[Building PHASTA on Cluster]]

2026-02-17T18:40:06Z

Moam3950: /* 9) Test the installation */

=Building HONEE on Summit nodes=

==Overview==
This page describes a Summit-specific workflow to build and test '''HONEE''' (main branch) on Summit compute nodes. HONEE depends on '''libCEED''' and '''PETSc'''.

* HONEE repo: https://gitlab.com/phypid/honee
* libCEED repo: https://github.com/CEED/libCEED
* PETSc repo: https://gitlab.com/petsc/petsc

'''Note:''' This guide assumes you build inside an interactive PBS job on a Summit compute node.

==1) Start an interactive Summit node==
From <code>jumpgate</code>, request a node:

<code>
qsub -I -l select=1:ncpus=24:mpiprocs=24 -l walltime=72:00:00 -l place=scatter:exclhost -q workq
</code>

You should land on a Summit node.

==2) Create a single workspace for all repositories==
Create a folder to hold all three code bases:

<code>mkdir honee_stack</code> 
<code>cd honee_stack</code>

Recommended layout:

<code>
~/honee_stack/ 
  libCEED/ 
  petsc/ 
  honee/
</code>

==3) Clone the repositories==
Navigate into <code>honee_stack</code> and clone the 3 repos.

===libCEED===
<code>
git clone https://github.com/CEED/libCEED
</code>

===PETSc (main branch)===
'''Important:''' Use PETSc <code>main</code>, not the release branch.

<code>
git clone https://gitlab.com/petsc/petsc
</code>

===HONEE (main branch)===
<code>
git clone https://gitlab.com/phypid/honee
</code>

==4) Build libCEED==
Navigate into the libceed directory <code>cd ~/honee_stack/libCEED</code>

<code>
make -j
</code>

This produces <code>libCEED</code> libraries needed by HONEE.

==5) Build PETSc ==
Navigate into the PETSc directory <code>cd ~/honee_stack/PETSc </code>

===Example configuration===
Below is a good baseline: MPI enabled, optimized build, and CGNS/HDF5 support via downloads.

<code>
./configure \ 
  --with-cc=mpicc \ 
  --with-cxx=mpicxx \ 
  --with-fc=0 \ 
  --with-mpi=1 \ 
  --with-mpiexec=mpirun \ 
  --download-f2cblaslapack \ 
  --download-hdf5 \ 
  --download-cgns \ 
  --with-debugging=0 \ 
  COPTFLAGS='-O3 -g' \ 
  CXXOPTFLAGS='-O3 -g'
</code>

Then build:

<code>
make -j8 all
</code>

===Notes on the example PETSc settings===
* <code>--with-debugging=0</code> is recommended for performance runs.
* <code>-O3</code> is appropriate for speed; keeping <code>-g</code> helps debugging without major runtime cost.
* Downloading <code>hdf5</code> + <code>cgns</code> is recommended for CGNS output support.

==6) Set environment variables for HONEE==
HONEE requires <code>CEED_DIR</code>, <code>PETSC_DIR</code>, and <code>PETSC_ARCH</code>.

Example setup:

<code>
export CEED_DIR=$HOME/honee_stack/libCEED 
export PETSC_DIR=$HOME/honee_stack/petsc 
export PETSC_ARCH=arch-linux-c-opt
</code>

'''Important runtime note (recommended while using the example PETSc config):'''

<code>
export LD_LIBRARY_PATH=$PETSC_DIR/$PETSC_ARCH/lib:$LD_LIBRARY_PATH
</code>

==7) Build HONEE==
<code>
cd ~/honee_stack/honee 
make -j
</code>

==8) Run a sample problem==
Run the Gaussian wave example:

<code>
cd ~/honee_stack/honee 
build/navierstokes -options_file examples/gaussianwave.yaml
</code>

==9) Test the installation==
Run the test suite:

<code>
cd ~/honee_stack/honee 
make test -j
</code>

==Troubleshooting==

===Missing PETSc or libCEED===
Verify environment variables:

<code>
echo $CEED_DIR 
echo $PETSC_DIR 
echo $PETSC_ARCH
</code>

===Runtime linker errors (libpetsc.so not found)===
Ensure:

<code>
export LD_LIBRARY_PATH=$PETSC_DIR/$PETSC_ARCH/lib:$LD_LIBRARY_PATH
</code>

===CGNS/HDF5 issues===
Reconfigure PETSc with:

<code>
--download-hdf5 --download-cgns
</code>

==Reference==
Official HONEE installation documentation: 
https://gitlab.com/phypid/honee