Building PETSc and libCEED to run HONEE Fluid solver on Viznode

If you're looking how to build a PETSc version to use with libCEED (HONEE) you've come to the right place! The following steps will summarize and include the commands needed to perform the install process and example run we followed in this video.

Getting PETSc and LibCEED

First things first, let's get the source code for PETSc and libCEED. You can clone the sources from the following:

PETSc SSH: git@gitlab.com:petsc/petsc.git
PETSc HTTPS: https://gitlab.com/petsc/petsc.git

libCEED SSH: git@github.com:CEED/libCEED.git
libCEED HTTPS: https://github.com/CEED/libCEED.git

Clone the repositories somewhere that makes sense for you. For example I have mine in jeffhadley/Builds/src/:

PETSc

git clone -b release https://gitlab.com/petsc/petsc.git petsc 
git pull # obtain new release fixes (since a prior clone or pull) 

libCEED

git clone https://github.com/CEED/libCEED.git libCEED

You can replace the https url's with the SSH ones from above if you have SSH setup.

Setting up Environment

Next we need to make sure we have the correct versions of gcc, openmpi, totalview, and Python. We will soft add gcc, openmpi and totalview with the following commands:

soft add +gcc-10.2.0
soft add +openmpi-gnu-4.0.2-gnu49-thread
soft add +totalview-2023.2.15

The version of Python available as a soft add is not sufficient to run PETSc and libCEED, so we need to enable a newer version through Anaconda. Paste the following block of code at the end of your ~/.bashrc or ~/.zshrc file:

# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/projects/tools/anaconda3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
    eval "$__conda_setup"
else
    if [ -f "/projects/tools/anaconda3/etc/profile.d/conda.sh" ]; then
        . "/projects/tools/anaconda3/etc/profile.d/conda.sh"
    else
        export PATH="/projects/tools/anaconda3/bin:$PATH"                                                          
    fi
fi
unset __conda_setup
# <<< conda initialize <<<

This block of text will automatically activate Conda when you open a new terminal window, which we do not actually want to happen. To fix this, run the following command:

conda config --set auto_activate_base False

You now need to manually activate Conda whenever you want a more up to date version of Python. To do so run the following command:

conda activate

Check the freshly enabled version of Python with python --version. The most up to date version as of 02/28/2024 is 3.7.9. For more information on using Anaconda and its sub-packages, see the wiki page here.

Building PETSc

Next we will build PETSc. Navigate into the top level of the cloned PETSc directory. We need to run a set-up script that's different than what comes natively with the clone of the repository, so run the following copy command to grab the set-up script which we will use for the VizNodes:

cp /projects/tools/petsc/petsc/testDebugSetup.py .

Next we will run this script to configure the Makefiles we will need to build the PETSc package:

python testDebugSetup.py

Once the configure step is complete you will get a message in your terminal stating it's complete, and that you can now compile PETSc with a command similar to the following (Note: copy and paste the one generated in your terminal, the following is just an example of what it will look like):

make PETSC_DIR=/users/username/location/of/petsc/ PETSC_ARCH=64DebTest all

This will run the Makefiles and build the PETSc libraries for the VizNodes. Once PETSc succesfully builds, you will have a message stating "Now to check if the libraries are working do:". We will skip this step and instead export the PETSc directory and architecture names to consol with the following:

export PETSC_DIR=/users/username/location/of/petsc
export PETSC_ARCH=64DebTest

Note: the PETSC_DIR and PETSC_ARCH will be printed in the "...check if libraries..." message, so best to just copy and paste those into the export command.

We need to export these variables as the fluids example executable we will later create with libCEED is going to require them.

To make sure the libraries built successfully run the following command and confirm there are .so files there:

cd $PETSC_ARCH/lib/

Building libCEED and the Fluids Solver

Next we will build libCEED, so navigate to the top level of the cloned repository. Nothing special to do here, simply run:

make -j

This should take all of 1 second to build. Next we will navigate to the Fluids example folder to build the fluids solver. To do so run the following commands:

cd examples/fluids/
make -j

The fluids solver executable is called navierstokes, verify that you can locate it in the .../libCEED/examples/fluids/ directory.

Running a Fluids Example case

We will now set up to run an example case. There's a few files that we need to grab in order to run the example, so within the .../libCEED/examples/fluids/ directory create a new directory and cd into it:

mkdir FluidsDemo
cd FluidsDemo

Next we need to copy over the initial condition, mesh, and inflow files:

cp /projects/tutorials/PETSc_libCEED_Demo/blasiusGenIC.yaml .
cp /projects/tutorials/PETSc_libCEED_Demo/STGInflow_12-30_SPD.dat .
cp /projects/tutorials/PETSc_libCEED_Demo/STGRand_12-30.dat .
cp /projects/tutorials/PETSc_libCEED_Demo/everyThird_yspacing.dat .

Timestamp 9:30 in the video linked above dives into the Initial Condition file blasiusGenIC.yaml. Note that the changes made to the .yaml file in the video are already applied to the one you just copied.

We will now run the example case. To do so run the following command:

mpirun -np 2 ../navierstokes -options_file blasiusGenIC.yaml -ts_max_time 10

Explanation of above command:

'-np 2' is running the job on 2 processors
'../navierstokes' is the executable we want to run, which is currently 1 directory above the one you should be in.
'-options_file blasisuGenIC.yaml' is passing the initial conditions file to the solver
'-ts_max_time 10' is changing the ts_max_time initial condition in the IC file from 0 (set in file) to 10. This is an example of 
modifying an options_file from the command line.

To run the job in Totalview, simply add -tv after -np 2 in the previous command. To see how to debug in Totalview I suggest to watch the video linked at the top of this page starting at timestamp 16:20.

Tada, you've built and run the HONEE fluids solver using libCEED and PETSc!