FDSI Summer Program/HONEE

Information for tutorial in HONEE, to be updated as needed.

HONEE, or the High-Order Navier-stokes Equation Evaluator, is a CFD program that combines libCEED and PETSc.

Building HONEE for Cisco nodes

First, start a job on the Cisco nodes. All the below commands should be run within a terminal on a Cisco node.

Setup Envionment

source /projects/tools/Spackv0.23/share/spack/setup-env.sh
spack load gcc@12.3
module load openmpi

Create new directory

mkdir honee_build
cd honee_build

Build PETSc

Ordinarily, I'd only recommend doing a git clone, e.g. git clone https://gitlab.com/petsc/petsc.git. However, the /nobackup/ server is quite slow and this process takes around 20 minutes (normally about a minute on my laptop), so downloading and extracting a tarball may be quicker. I may cover that later, but for now I'll just cover it via git clone.

git clone https://gitlab.com/petsc/petsc.git
cd petsc
cp /nobackup/uncompressed/jrwrigh/HONEE_Setup/petsc/reconfigure.py
./reconfigure.py
make
export PETSC_DIR=$(pwd) PETSC_ARCH=arch-32
cd ..

The reconfigure.py file has the following:

#!/bin/python3
if __name__ == '__main__':
  import sys
  import os
  sys.path.insert(0, os.path.abspath('config'))
  import configure
  configure_options = [
    '--with-64-bit-indices=0',
    '--download-hdf5',
    '--download-cgns',
    '--download-ctetgen=1',
    '--download-parmetis=1',
    '--download-metis=1',
    '--download-ptscotch=1',
    '--with-debugging=0',
    '--with-fortran-bindings=0',
    '--with-fc=0',
    'PETSC_ARCH=arch-32',
    'COPTFLAGS=-g -O3',
    'CXXOPTFLAGS=-g -O3',
  ]
  configure.petsc_configure(configure_options)

Building HONEE

Ensure that PETSC_DIR and PETSC_ARCH are set to the desired PETSc installation.

git clone https://github.com/CEED/libCEED.git
cd libCEED
make build/fluids-navierstokes -j

The executable build/fluids-navierstokes is then built and ready for running.

Changing HONEE Inputs

HONEE uses PETSc's input argument system for handling inputs. They can be provided either via command-line flags or inside a YAML file. So

./build/fluids-navierstokes -ts_dt 1e-3

is equivalent to

./build/fluids-navierstokes -options_file test.yaml

if test.yaml has this in it:

ts_dt: 1e-3

Note also that PETSc allows for hierarchical flags as well within a YAML file. So instead of writing

ts_dt: 1e-3
ts_type: alpha
ts_max_time: 2.3

You can write:

ts:
  dt: 1e-3
  type: alpha
  max_time: 2.3

Documentation for Specific Flags

• HONEE specific flags
• Time Stepping (TS) flags
• Non-linear solver (SNES) flags
• Linear solver (KSP) flags

Notable Input Flags

• -ts_monitor_solution: This will save the results of a simulation to a file. Example: -ts_monitor_solution cgns:flow_visualization.cgns will save the results to a file called flow_visualization.cgns

Restarting Simulations

To restart a simulation from a previous simulation's result, they need to be saved to a *.bin file. This is done using the `-continue` flag, which will load the file set by `-continue_filename`.

Note that restarting from a binary file can only be done if the binary file was written by a simulation running at the same part count.