PHASTA Wiki - User contributions [en]

Build HONEE on Summit

2026-02-18T20:34:12Z

Jrwrigh: fix -march=native flag location

=Building HONEE on Summit nodes=

==Overview==
This page describes a Summit-specific workflow to build and test '''HONEE''' (main branch) on Summit compute nodes. HONEE depends on '''libCEED''' and '''PETSc'''.

* HONEE repo: https://gitlab.com/phypid/honee
* libCEED repo: https://github.com/CEED/libCEED
* PETSc repo: https://gitlab.com/petsc/petsc

'''Note:''' This guide assumes you build inside an interactive PBS job on a Summit compute node.

==1) Start an interactive Summit node==
From <code>jumpgate</code>, request a node:

<code>
qsub -I -l select=1:ncpus=24:mpiprocs=24 -l walltime=72:00:00 -l place=scatter:exclhost -q workq
</code>

You should land on a Summit node.

==2) Create a single workspace for all repositories==
Create a folder to hold all three code bases:

<code>mkdir honee_stack</code> 
<code>cd honee_stack</code>

Recommended layout:

<code>
~/honee_stack/ 
  libCEED/ 
  petsc/ 
  honee/
</code>

==3) Clone the repositories==
Navigate into <code>honee_stack</code> and clone the 3 repos.

===libCEED===
<code>
git clone https://github.com/CEED/libCEED
</code>

===PETSc (main branch)===
'''Important:''' Use PETSc <code>main</code>, not the release branch.

<code>
git clone https://gitlab.com/petsc/petsc
</code>

===HONEE (main branch)===
<code>
git clone https://gitlab.com/phypid/honee
</code>

==4) Build libCEED==
Navigate into the libceed directory <code>cd ~/honee_stack/libCEED</code>

<code>
make -j
</code>

This produces <code>libCEED</code> libraries needed by HONEE.

==5) Build PETSc ==
Navigate into the PETSc directory <code>cd ~/honee_stack/PETSc </code>

===Example configuration===
Below is a good baseline: MPI enabled, optimized build, and CGNS/HDF5 support via downloads.

<code>
./configure \ 
  --with-cc=mpicc \ 
  --with-cxx=mpicxx \ 
  --with-fc=0 \ 
  --with-mpi=1 \ 
  --with-mpiexec=mpirun \ 
  --download-f2cblaslapack \ 
  --download-hdf5 \ 
  --download-cgns \ 
  --with-debugging=0 \ 
  COPTFLAGS='-O3 -g --march=native' \ 
  CXXOPTFLAGS='-O3 -g --march=native'
</code>

Then build:

<code>
make -j all
</code>

===Notes on the example PETSc settings===
* <code>--with-debugging=0</code> is recommended for performance runs.
* <code>-O3</code> is appropriate for speed; keeping <code>-g</code> helps debugging without major runtime cost.
* Downloading <code>hdf5</code> + <code>cgns</code> is recommended for CGNS output support.

==6) Set environment variables for HONEE==
HONEE requires <code>CEED_DIR</code>, <code>PETSC_DIR</code>, and <code>PETSC_ARCH</code>.

Example setup:

<code>
export CEED_DIR=$HOME/honee_stack/libCEED 
export PETSC_DIR=$HOME/honee_stack/petsc 
export PETSC_ARCH=arch-linux-c-opt
</code>

'''Important runtime note (recommended while using the example PETSc config):'''

<code>
export LD_LIBRARY_PATH=$PETSC_DIR/$PETSC_ARCH/lib:$LD_LIBRARY_PATH
</code>

==7) Build HONEE==
<code>
cd ~/honee_stack/honee 
make -j
</code>

==8) Run a sample problem==
Run the Gaussian wave example:

<code>
cd ~/honee_stack/honee 
build/navierstokes -options_file examples/gaussianwave.yaml
</code>

==9) Test the installation==
Run the test suite:

<code>
cd ~/honee_stack/honee 
make test -j
</code>

==Troubleshooting==

===Missing PETSc or libCEED===
Verify environment variables:

<code>
echo $CEED_DIR 
echo $PETSC_DIR 
echo $PETSC_ARCH
</code>

===Runtime linker errors (libpetsc.so not found)===
Ensure:

<code>
export LD_LIBRARY_PATH=$PETSC_DIR/$PETSC_ARCH/lib:$LD_LIBRARY_PATH
</code>

===CGNS/HDF5 issues===
Reconfigure PETSc with:

<code>
--download-hdf5 --download-cgns
</code>

==Reference==
Official HONEE installation documentation: 
https://gitlab.com/phypid/honee

Official PETSc configuration documentation: 
https://petsc.org/main/install/

FDSI Summer Program/HONEE

2024-07-02T16:07:37Z

Jrwrigh: /* Degree-based settings */

''Information for tutorial in HONEE, to be updated as needed.''

'''HONEE''', or the '''High-Order Navier-stokes Equation Evaluator''', is a CFD program that combines [https://libceed.org/en/latest/ libCEED] and [https://petsc.org/release/ PETSc].

== Building HONEE for Cisco nodes ==

First, start a job on the Cisco nodes. All the below commands should be run within a terminal on a Cisco node.

=== Setup Envionment ===
source /projects/tools/Spackv0.23/share/spack/setup-env.sh
spack load gcc@12.3
module load openmpi

=== Create new directory ===
mkdir honee_build
cd honee_build

=== Build PETSc===
Ordinarily, I'd only recommend doing a <code>git clone</code>, e.g. <code>git clone https://gitlab.com/petsc/petsc.git</code>.
However, the <code>/nobackup/</code> server is quite slow and this process takes around 20 minutes (normally about a minute on my laptop), so downloading and extracting a tarball may be quicker.
I may cover that later, but for now I'll just cover it via <code>git clone</code>.

git clone https://gitlab.com/petsc/petsc.git
cd petsc
cp /nobackup/uncompressed/jrwrigh/HONEE_Setup/petsc/reconfigure.py
./reconfigure.py
make
export PETSC_DIR=$(pwd) PETSC_ARCH=arch-32
cd ..

The <code>reconfigure.py</code> file has the following:
#!/bin/python3
if __name__ == '__main__':
import sys
import os
sys.path.insert(0, os.path.abspath('config'))
import configure
configure_options = [
'--with-64-bit-indices=0',
'--download-hdf5',
'--download-cgns',
'--download-ctetgen=1',
'--download-parmetis=1',
'--download-metis=1',
'--download-ptscotch=1',
'--with-debugging=0',
'--with-fortran-bindings=0',
'--with-fc=0',
'PETSC_ARCH=arch-32',
'COPTFLAGS=-g -O3',
'CXXOPTFLAGS=-g -O3',
]
configure.petsc_configure(configure_options)

=== Building HONEE ===
Ensure that <code>PETSC_DIR</code> and <code>PETSC_ARCH</code> are set to the desired PETSc installation.

git clone https://github.com/CEED/libCEED.git
cd libCEED
make build/fluids-navierstokes -j

The executable <code>build/fluids-navierstokes</code> is then built and ready for running.

== Changing HONEE Inputs ==
HONEE uses PETSc's input argument system for handling inputs.
They can be provided either via command-line flags or inside a YAML file.
So

./build/fluids-navierstokes -ts_dt 1e-3

is equivalent to

./build/fluids-navierstokes -options_file test.yaml

if <code>test.yaml</code> has this in it:

ts_dt: 1e-3

Note also that PETSc allows for hierarchical flags as well within a YAML file.
So instead of writing

ts_dt: 1e-3
ts_type: alpha
ts_max_time: 2.3

You can write:

ts:
dt: 1e-3
type: alpha
max_time: 2.3

=== Documentation for Specific Flags ===
* [https://libceed.org/en/latest/examples/fluids/ HONEE specific flags]
* [https://petsc.org/main/manualpages/TS/TSSetFromOptions/ Time Stepping (TS) flags]
* [https://petsc.org/main/manualpages/SNES/SNESSetFromOptions/ Non-linear solver (SNES) flags]
* [https://petsc.org/main/manualpages/KSP/KSPSetFromOptions/ Linear solver (KSP) flags]

=== Notable Input Flags ===
* <code>-ts_monitor_solution</code>: This will save the results of a simulation to a file. Example: <code>-ts_monitor_solution cgns:flow_visualization.cgns</code> will save the results to a file called <code>flow_visualization.cgns</code>

=== Restarting Simulations ===
To restart a simulation from a previous simulation's result, they need to be saved to a <code>*.bin</code> file.
This is done using the <code>-continue</code> flag, which will load the file set by <code>-continue_filename</code>.

Note that restarting from a binary file can only be done if the binary file was written by a simulation running at the same part count.

== HONEE and Gmsh ==

=== GMSH/Grid requirements: ===
# The grid must be hexahedron based (so deformed cubes). HONEE can run with tetrahedron grids, but that will require some minor code modifications which we've done before (IIRC, it's a single line of code to change). If anyone is interested in this, let me know and we can work on it.
#* Note that HONEE cannot handle mixed meshes, so meshes with both hexahedron and tetrahedron grids. This is not a minor code change and is probably out of scope for this Summer Program
# In the geo file, you need to specify <code>Physical Surface</code>s to identify the domain boundaries to apply boundary conditions to.
# In the geo file, you need to specify a single <code>Physical Volume</code> that has the entire domain. So if you split the domain into multiple pieces for meshing purposes (as is done with the vortex shedding grids), then you need the <code>Physical Volume</code> to identify all the pieces.

=== YAML Setup: ===
In the YAML file, you need to associate each <code>Physical Surface</code>s with a boundary condition. I describe this briefly in the video, but each <code>Physical Surface</code> as an ID number associated with it. In the YAML file, you need to associate that ID number with a BC choice.
For example, in the <code>vortexshedding.yaml</code> file there is:
# Boundary Settings
bc_slip_z: 6
bc_wall: 5
bc_freestream: 1
bc_outflow: 2
bc_slip_y: 3,4

and the <code>cylinder.geo</code> has:

Physical Surface("inlet") = {102};
Physical Surface("outlet") = {116};
Physical Surface("top") = {80, 120};
Physical Surface("bottom") = {36, 112};
Physical Surface("cylinderwalls") = {94, 28, 50, 72};
Physical Surface("frontandback") = {37, 1, 4, 103, 3, 81, 2, 59, 5, 125};

The "inlet" face is the first identified boundary, so it gets the ID number 1.
In the YAML file, you see <code>bc_freestream: 1</code>, which sets this inlet boundary to be controlled by a freestream BC.
Similar with the "outlet" face; it's the second specified boundary in the geo file, so we set <code>bc_outflow: 2</code>.
And so on with the other faces.

'''NOTE:''' When using the GMSH GUI to identify the Physical Surfaces, it will actually put something like this in the geo file:
Physical Surface("inlet", 3010) = {3005};
Physical Surface("outlet", 3011) = {3003};
Physical Surface("top", 3012) = {3004};
Physical Surface("bottom", 3013) = {3002};
Physical Surface("nacawalls", 3014) = {3008, 3006, 3007};
Physical Surface("frontandback", 3015) = {3009, 3001};

Note the extra <code>3010</code> in the "inlet" definition. I'm not sure if this means that correct ID number for it should be 3010 instead of 1, but that's a possibility.

== Recommendations for HONEE performance ==

=== Compilation Flags ===
When compiling a program, you can pass certain flags in order to try and make HONEE run faster.
For the Cisco nodes, this can be done by running the following in your libCEED directory:

make configure OPT='-O3 -march=native -g -ffp-contract=fast -fopenmp-simd'
make build/fluids-navierstokes -Bj

=== General Running Settings ===
Some general settings that will help improve performance are to lower the solver tolerances.
Specifically <code>-snes_rtol 1e-4 -ksp_rtol 1e-4</code> have been found to be pretty good choices.
These may require tweaking if you run into divergence issues.

=== Degree-based settings ===
Optimal solver settings can change based on what degree elements you work with (determined by the <code>-degree</code> flag).
For linear elements (<code>-degree 1</code>), I've had success with:

-amat_type -snes_lag_jacobian 5 -snes_lag_jacobian_persists false -snes_lag_preconditioner 20 -snes_lag_preconditioner_persists true -pc_type asm -sub_pc_type lu

While for cubic elements (<code>-degree 3</code>) I've seen better performance with:

-amat_type shell -snes_lag_jacobian 15 -snes_lag_jacobian_persists true -pc_type asm -sub_pc_type lu

Note that these flags were when running the vortex shedding example, so the optimal options may change for other problems.

FDSI Summer Program/HONEE

2024-07-02T16:03:34Z

Jrwrigh: /* Degree-based settings */

''Information for tutorial in HONEE, to be updated as needed.''

'''HONEE''', or the '''High-Order Navier-stokes Equation Evaluator''', is a CFD program that combines [https://libceed.org/en/latest/ libCEED] and [https://petsc.org/release/ PETSc].

== Building HONEE for Cisco nodes ==

First, start a job on the Cisco nodes. All the below commands should be run within a terminal on a Cisco node.

=== Setup Envionment ===
source /projects/tools/Spackv0.23/share/spack/setup-env.sh
spack load gcc@12.3
module load openmpi

=== Create new directory ===
mkdir honee_build
cd honee_build

=== Build PETSc===
Ordinarily, I'd only recommend doing a <code>git clone</code>, e.g. <code>git clone https://gitlab.com/petsc/petsc.git</code>.
However, the <code>/nobackup/</code> server is quite slow and this process takes around 20 minutes (normally about a minute on my laptop), so downloading and extracting a tarball may be quicker.
I may cover that later, but for now I'll just cover it via <code>git clone</code>.

git clone https://gitlab.com/petsc/petsc.git
cd petsc
cp /nobackup/uncompressed/jrwrigh/HONEE_Setup/petsc/reconfigure.py
./reconfigure.py
make
export PETSC_DIR=$(pwd) PETSC_ARCH=arch-32
cd ..

The <code>reconfigure.py</code> file has the following:
#!/bin/python3
if __name__ == '__main__':
import sys
import os
sys.path.insert(0, os.path.abspath('config'))
import configure
configure_options = [
'--with-64-bit-indices=0',
'--download-hdf5',
'--download-cgns',
'--download-ctetgen=1',
'--download-parmetis=1',
'--download-metis=1',
'--download-ptscotch=1',
'--with-debugging=0',
'--with-fortran-bindings=0',
'--with-fc=0',
'PETSC_ARCH=arch-32',
'COPTFLAGS=-g -O3',
'CXXOPTFLAGS=-g -O3',
]
configure.petsc_configure(configure_options)

=== Building HONEE ===
Ensure that <code>PETSC_DIR</code> and <code>PETSC_ARCH</code> are set to the desired PETSc installation.

git clone https://github.com/CEED/libCEED.git
cd libCEED
make build/fluids-navierstokes -j

The executable <code>build/fluids-navierstokes</code> is then built and ready for running.

== Changing HONEE Inputs ==
HONEE uses PETSc's input argument system for handling inputs.
They can be provided either via command-line flags or inside a YAML file.
So

./build/fluids-navierstokes -ts_dt 1e-3

is equivalent to

./build/fluids-navierstokes -options_file test.yaml

if <code>test.yaml</code> has this in it:

ts_dt: 1e-3

Note also that PETSc allows for hierarchical flags as well within a YAML file.
So instead of writing

ts_dt: 1e-3
ts_type: alpha
ts_max_time: 2.3

You can write:

ts:
dt: 1e-3
type: alpha
max_time: 2.3

=== Documentation for Specific Flags ===
* [https://libceed.org/en/latest/examples/fluids/ HONEE specific flags]
* [https://petsc.org/main/manualpages/TS/TSSetFromOptions/ Time Stepping (TS) flags]
* [https://petsc.org/main/manualpages/SNES/SNESSetFromOptions/ Non-linear solver (SNES) flags]
* [https://petsc.org/main/manualpages/KSP/KSPSetFromOptions/ Linear solver (KSP) flags]

=== Notable Input Flags ===
* <code>-ts_monitor_solution</code>: This will save the results of a simulation to a file. Example: <code>-ts_monitor_solution cgns:flow_visualization.cgns</code> will save the results to a file called <code>flow_visualization.cgns</code>

=== Restarting Simulations ===
To restart a simulation from a previous simulation's result, they need to be saved to a <code>*.bin</code> file.
This is done using the <code>-continue</code> flag, which will load the file set by <code>-continue_filename</code>.

Note that restarting from a binary file can only be done if the binary file was written by a simulation running at the same part count.

== HONEE and Gmsh ==

=== GMSH/Grid requirements: ===
# The grid must be hexahedron based (so deformed cubes). HONEE can run with tetrahedron grids, but that will require some minor code modifications which we've done before (IIRC, it's a single line of code to change). If anyone is interested in this, let me know and we can work on it.
#* Note that HONEE cannot handle mixed meshes, so meshes with both hexahedron and tetrahedron grids. This is not a minor code change and is probably out of scope for this Summer Program
# In the geo file, you need to specify <code>Physical Surface</code>s to identify the domain boundaries to apply boundary conditions to.
# In the geo file, you need to specify a single <code>Physical Volume</code> that has the entire domain. So if you split the domain into multiple pieces for meshing purposes (as is done with the vortex shedding grids), then you need the <code>Physical Volume</code> to identify all the pieces.

=== YAML Setup: ===
In the YAML file, you need to associate each <code>Physical Surface</code>s with a boundary condition. I describe this briefly in the video, but each <code>Physical Surface</code> as an ID number associated with it. In the YAML file, you need to associate that ID number with a BC choice.
For example, in the <code>vortexshedding.yaml</code> file there is:
# Boundary Settings
bc_slip_z: 6
bc_wall: 5
bc_freestream: 1
bc_outflow: 2
bc_slip_y: 3,4

and the <code>cylinder.geo</code> has:

Physical Surface("inlet") = {102};
Physical Surface("outlet") = {116};
Physical Surface("top") = {80, 120};
Physical Surface("bottom") = {36, 112};
Physical Surface("cylinderwalls") = {94, 28, 50, 72};
Physical Surface("frontandback") = {37, 1, 4, 103, 3, 81, 2, 59, 5, 125};

The "inlet" face is the first identified boundary, so it gets the ID number 1.
In the YAML file, you see <code>bc_freestream: 1</code>, which sets this inlet boundary to be controlled by a freestream BC.
Similar with the "outlet" face; it's the second specified boundary in the geo file, so we set <code>bc_outflow: 2</code>.
And so on with the other faces.

'''NOTE:''' When using the GMSH GUI to identify the Physical Surfaces, it will actually put something like this in the geo file:
Physical Surface("inlet", 3010) = {3005};
Physical Surface("outlet", 3011) = {3003};
Physical Surface("top", 3012) = {3004};
Physical Surface("bottom", 3013) = {3002};
Physical Surface("nacawalls", 3014) = {3008, 3006, 3007};
Physical Surface("frontandback", 3015) = {3009, 3001};

Note the extra <code>3010</code> in the "inlet" definition. I'm not sure if this means that correct ID number for it should be 3010 instead of 1, but that's a possibility.

== Recommendations for HONEE performance ==

=== Compilation Flags ===
When compiling a program, you can pass certain flags in order to try and make HONEE run faster.
For the Cisco nodes, this can be done by running the following in your libCEED directory:

make configure OPT='-O3 -march=native -g -ffp-contract=fast -fopenmp-simd'
make build/fluids-navierstokes -Bj

=== General Running Settings ===
Some general settings that will help improve performance are to lower the solver tolerances.
Specifically <code>-snes_rtol 1e-4 -ksp_rtol 1e-4</code> have been found to be pretty good choices.
These may require tweaking if you run into divergence issues.

=== Degree-based settings ===
Optimal solver settings can change based on what degree elements you work with (determined by the <code>-degree</code> flag).
For linear elements (<code>-degree 1</code>), I've had success with:

-amat_type -snes_lag_jacobian 5 -snes_lag_jacobian_persists false -snes_lag_preconditioner 20 -snes_lag_preconditioner_persists true -pc_type asm -sub_pc_type lu

While for cubic elements (<code>-degree 3</code>) I've seen better performance with:

-amat_type shell -snes_lag_jacobian 15 -snes_lag_jacobian_persists true -py_type asm -sub_pc_type lu

Note that these flags were when running the vortex shedding example, so the optimal options may change for other problems.

FDSI Summer Program/HONEE

2024-06-21T15:52:01Z

Jrwrigh: /* Recommendations for HONEE performance */

''Information for tutorial in HONEE, to be updated as needed.''

'''HONEE''', or the '''High-Order Navier-stokes Equation Evaluator''', is a CFD program that combines [https://libceed.org/en/latest/ libCEED] and [https://petsc.org/release/ PETSc].

== Building HONEE for Cisco nodes ==

First, start a job on the Cisco nodes. All the below commands should be run within a terminal on a Cisco node.

=== Setup Envionment ===
source /projects/tools/Spackv0.23/share/spack/setup-env.sh
spack load gcc@12.3
module load openmpi

=== Create new directory ===
mkdir honee_build
cd honee_build

=== Build PETSc===
Ordinarily, I'd only recommend doing a <code>git clone</code>, e.g. <code>git clone https://gitlab.com/petsc/petsc.git</code>.
However, the <code>/nobackup/</code> server is quite slow and this process takes around 20 minutes (normally about a minute on my laptop), so downloading and extracting a tarball may be quicker.
I may cover that later, but for now I'll just cover it via <code>git clone</code>.

git clone https://gitlab.com/petsc/petsc.git
cd petsc
cp /nobackup/uncompressed/jrwrigh/HONEE_Setup/petsc/reconfigure.py
./reconfigure.py
make
export PETSC_DIR=$(pwd) PETSC_ARCH=arch-32
cd ..

The <code>reconfigure.py</code> file has the following:
#!/bin/python3
if __name__ == '__main__':
import sys
import os
sys.path.insert(0, os.path.abspath('config'))
import configure
configure_options = [
'--with-64-bit-indices=0',
'--download-hdf5',
'--download-cgns',
'--download-ctetgen=1',
'--download-parmetis=1',
'--download-metis=1',
'--download-ptscotch=1',
'--with-debugging=0',
'--with-fortran-bindings=0',
'--with-fc=0',
'PETSC_ARCH=arch-32',
'COPTFLAGS=-g -O3',
'CXXOPTFLAGS=-g -O3',
]
configure.petsc_configure(configure_options)

=== Building HONEE ===
Ensure that <code>PETSC_DIR</code> and <code>PETSC_ARCH</code> are set to the desired PETSc installation.

git clone https://github.com/CEED/libCEED.git
cd libCEED
make build/fluids-navierstokes -j

The executable <code>build/fluids-navierstokes</code> is then built and ready for running.

== Changing HONEE Inputs ==
HONEE uses PETSc's input argument system for handling inputs.
They can be provided either via command-line flags or inside a YAML file.
So

./build/fluids-navierstokes -ts_dt 1e-3

is equivalent to

./build/fluids-navierstokes -options_file test.yaml

if <code>test.yaml</code> has this in it:

ts_dt: 1e-3

Note also that PETSc allows for hierarchical flags as well within a YAML file.
So instead of writing

ts_dt: 1e-3
ts_type: alpha
ts_max_time: 2.3

You can write:

ts:
dt: 1e-3
type: alpha
max_time: 2.3

=== Documentation for Specific Flags ===
* [https://libceed.org/en/latest/examples/fluids/ HONEE specific flags]
* [https://petsc.org/main/manualpages/TS/TSSetFromOptions/ Time Stepping (TS) flags]
* [https://petsc.org/main/manualpages/SNES/SNESSetFromOptions/ Non-linear solver (SNES) flags]
* [https://petsc.org/main/manualpages/KSP/KSPSetFromOptions/ Linear solver (KSP) flags]

=== Notable Input Flags ===
* <code>-ts_monitor_solution</code>: This will save the results of a simulation to a file. Example: <code>-ts_monitor_solution cgns:flow_visualization.cgns</code> will save the results to a file called <code>flow_visualization.cgns</code>

=== Restarting Simulations ===
To restart a simulation from a previous simulation's result, they need to be saved to a <code>*.bin</code> file.
This is done using the <code>-continue</code> flag, which will load the file set by <code>-continue_filename</code>.

Note that restarting from a binary file can only be done if the binary file was written by a simulation running at the same part count.

== HONEE and Gmsh ==

=== GMSH/Grid requirements: ===
# The grid must be hexahedron based (so deformed cubes). HONEE can run with tetrahedron grids, but that will require some minor code modifications which we've done before (IIRC, it's a single line of code to change). If anyone is interested in this, let me know and we can work on it.
#* Note that HONEE cannot handle mixed meshes, so meshes with both hexahedron and tetrahedron grids. This is not a minor code change and is probably out of scope for this Summer Program
# In the geo file, you need to specify <code>Physical Surface</code>s to identify the domain boundaries to apply boundary conditions to.
# In the geo file, you need to specify a single <code>Physical Volume</code> that has the entire domain. So if you split the domain into multiple pieces for meshing purposes (as is done with the vortex shedding grids), then you need the <code>Physical Volume</code> to identify all the pieces.

=== YAML Setup: ===
In the YAML file, you need to associate each <code>Physical Surface</code>s with a boundary condition. I describe this briefly in the video, but each <code>Physical Surface</code> as an ID number associated with it. In the YAML file, you need to associate that ID number with a BC choice.
For example, in the <code>vortexshedding.yaml</code> file there is:
# Boundary Settings
bc_slip_z: 6
bc_wall: 5
bc_freestream: 1
bc_outflow: 2
bc_slip_y: 3,4

and the <code>cylinder.geo</code> has:

Physical Surface("inlet") = {102};
Physical Surface("outlet") = {116};
Physical Surface("top") = {80, 120};
Physical Surface("bottom") = {36, 112};
Physical Surface("cylinderwalls") = {94, 28, 50, 72};
Physical Surface("frontandback") = {37, 1, 4, 103, 3, 81, 2, 59, 5, 125};

The "inlet" face is the first identified boundary, so it gets the ID number 1.
In the YAML file, you see <code>bc_freestream: 1</code>, which sets this inlet boundary to be controlled by a freestream BC.
Similar with the "outlet" face; it's the second specified boundary in the geo file, so we set <code>bc_outflow: 2</code>.
And so on with the other faces.

'''NOTE:''' When using the GMSH GUI to identify the Physical Surfaces, it will actually put something like this in the geo file:
Physical Surface("inlet", 3010) = {3005};
Physical Surface("outlet", 3011) = {3003};
Physical Surface("top", 3012) = {3004};
Physical Surface("bottom", 3013) = {3002};
Physical Surface("nacawalls", 3014) = {3008, 3006, 3007};
Physical Surface("frontandback", 3015) = {3009, 3001};

Note the extra <code>3010</code> in the "inlet" definition. I'm not sure if this means that correct ID number for it should be 3010 instead of 1, but that's a possibility.

== Recommendations for HONEE performance ==

=== Compilation Flags ===
When compiling a program, you can pass certain flags in order to try and make HONEE run faster.
For the Cisco nodes, this can be done by running the following in your libCEED directory:

make configure OPT='-O3 -march=native -g -ffp-contract=fast -fopenmp-simd'
make build/fluids-navierstokes -Bj

=== General Running Settings ===
Some general settings that will help improve performance are to lower the solver tolerances.
Specifically <code>-snes_rtol 1e-4 -ksp_rtol 1e-4</code> have been found to be pretty good choices.
These may require tweaking if you run into divergence issues.

=== Degree-based settings ===
Optimal solver settings can change based on what degree elements you work with (determined by the <code>-degree</code> flag).
For linear elements (<code>-degree 1</code>), I've had success with:

-amat_type -snes_lag_jacobian 5 -snes_lag_jacobian_persists false -snes_lag_preconditioner 20 -snes_lag_preconditioner_persists true -py_type asm -sub_pc_type lu

While for cubic elements (<code>-degree 3</code>) I've seen better performance with:

-amat_type shell -snes_lag_jacobian 15 -snes_lag_jacobian_persists true -py_type asm -sub_pc_type lu

Note that these flags were when running the vortex shedding example, so the optimal options may change for other problems.

FDSI Summer Program/HONEE

2024-06-21T15:51:16Z

Jrwrigh: /* Recommendations for HONEE performance */

''Information for tutorial in HONEE, to be updated as needed.''

'''HONEE''', or the '''High-Order Navier-stokes Equation Evaluator''', is a CFD program that combines [https://libceed.org/en/latest/ libCEED] and [https://petsc.org/release/ PETSc].

== Building HONEE for Cisco nodes ==

First, start a job on the Cisco nodes. All the below commands should be run within a terminal on a Cisco node.

=== Setup Envionment ===
source /projects/tools/Spackv0.23/share/spack/setup-env.sh
spack load gcc@12.3
module load openmpi

=== Create new directory ===
mkdir honee_build
cd honee_build

=== Build PETSc===
Ordinarily, I'd only recommend doing a <code>git clone</code>, e.g. <code>git clone https://gitlab.com/petsc/petsc.git</code>.
However, the <code>/nobackup/</code> server is quite slow and this process takes around 20 minutes (normally about a minute on my laptop), so downloading and extracting a tarball may be quicker.
I may cover that later, but for now I'll just cover it via <code>git clone</code>.

git clone https://gitlab.com/petsc/petsc.git
cd petsc
cp /nobackup/uncompressed/jrwrigh/HONEE_Setup/petsc/reconfigure.py
./reconfigure.py
make
export PETSC_DIR=$(pwd) PETSC_ARCH=arch-32
cd ..

The <code>reconfigure.py</code> file has the following:
#!/bin/python3
if __name__ == '__main__':
import sys
import os
sys.path.insert(0, os.path.abspath('config'))
import configure
configure_options = [
'--with-64-bit-indices=0',
'--download-hdf5',
'--download-cgns',
'--download-ctetgen=1',
'--download-parmetis=1',
'--download-metis=1',
'--download-ptscotch=1',
'--with-debugging=0',
'--with-fortran-bindings=0',
'--with-fc=0',
'PETSC_ARCH=arch-32',
'COPTFLAGS=-g -O3',
'CXXOPTFLAGS=-g -O3',
]
configure.petsc_configure(configure_options)

=== Building HONEE ===
Ensure that <code>PETSC_DIR</code> and <code>PETSC_ARCH</code> are set to the desired PETSc installation.

git clone https://github.com/CEED/libCEED.git
cd libCEED
make build/fluids-navierstokes -j

The executable <code>build/fluids-navierstokes</code> is then built and ready for running.

== Changing HONEE Inputs ==
HONEE uses PETSc's input argument system for handling inputs.
They can be provided either via command-line flags or inside a YAML file.
So

./build/fluids-navierstokes -ts_dt 1e-3

is equivalent to

./build/fluids-navierstokes -options_file test.yaml

if <code>test.yaml</code> has this in it:

ts_dt: 1e-3

Note also that PETSc allows for hierarchical flags as well within a YAML file.
So instead of writing

ts_dt: 1e-3
ts_type: alpha
ts_max_time: 2.3

You can write:

ts:
dt: 1e-3
type: alpha
max_time: 2.3

=== Documentation for Specific Flags ===
* [https://libceed.org/en/latest/examples/fluids/ HONEE specific flags]
* [https://petsc.org/main/manualpages/TS/TSSetFromOptions/ Time Stepping (TS) flags]
* [https://petsc.org/main/manualpages/SNES/SNESSetFromOptions/ Non-linear solver (SNES) flags]
* [https://petsc.org/main/manualpages/KSP/KSPSetFromOptions/ Linear solver (KSP) flags]

=== Notable Input Flags ===
* <code>-ts_monitor_solution</code>: This will save the results of a simulation to a file. Example: <code>-ts_monitor_solution cgns:flow_visualization.cgns</code> will save the results to a file called <code>flow_visualization.cgns</code>

=== Restarting Simulations ===
To restart a simulation from a previous simulation's result, they need to be saved to a <code>*.bin</code> file.
This is done using the <code>-continue</code> flag, which will load the file set by <code>-continue_filename</code>.

Note that restarting from a binary file can only be done if the binary file was written by a simulation running at the same part count.

== HONEE and Gmsh ==

=== GMSH/Grid requirements: ===
# The grid must be hexahedron based (so deformed cubes). HONEE can run with tetrahedron grids, but that will require some minor code modifications which we've done before (IIRC, it's a single line of code to change). If anyone is interested in this, let me know and we can work on it.
#* Note that HONEE cannot handle mixed meshes, so meshes with both hexahedron and tetrahedron grids. This is not a minor code change and is probably out of scope for this Summer Program
# In the geo file, you need to specify <code>Physical Surface</code>s to identify the domain boundaries to apply boundary conditions to.
# In the geo file, you need to specify a single <code>Physical Volume</code> that has the entire domain. So if you split the domain into multiple pieces for meshing purposes (as is done with the vortex shedding grids), then you need the <code>Physical Volume</code> to identify all the pieces.

=== YAML Setup: ===
In the YAML file, you need to associate each <code>Physical Surface</code>s with a boundary condition. I describe this briefly in the video, but each <code>Physical Surface</code> as an ID number associated with it. In the YAML file, you need to associate that ID number with a BC choice.
For example, in the <code>vortexshedding.yaml</code> file there is:
# Boundary Settings
bc_slip_z: 6
bc_wall: 5
bc_freestream: 1
bc_outflow: 2
bc_slip_y: 3,4

and the <code>cylinder.geo</code> has:

Physical Surface("inlet") = {102};
Physical Surface("outlet") = {116};
Physical Surface("top") = {80, 120};
Physical Surface("bottom") = {36, 112};
Physical Surface("cylinderwalls") = {94, 28, 50, 72};
Physical Surface("frontandback") = {37, 1, 4, 103, 3, 81, 2, 59, 5, 125};

The "inlet" face is the first identified boundary, so it gets the ID number 1.
In the YAML file, you see <code>bc_freestream: 1</code>, which sets this inlet boundary to be controlled by a freestream BC.
Similar with the "outlet" face; it's the second specified boundary in the geo file, so we set <code>bc_outflow: 2</code>.
And so on with the other faces.

'''NOTE:''' When using the GMSH GUI to identify the Physical Surfaces, it will actually put something like this in the geo file:
Physical Surface("inlet", 3010) = {3005};
Physical Surface("outlet", 3011) = {3003};
Physical Surface("top", 3012) = {3004};
Physical Surface("bottom", 3013) = {3002};
Physical Surface("nacawalls", 3014) = {3008, 3006, 3007};
Physical Surface("frontandback", 3015) = {3009, 3001};

Note the extra <code>3010</code> in the "inlet" definition. I'm not sure if this means that correct ID number for it should be 3010 instead of 1, but that's a possibility.

== Recommendations for HONEE performance ==

=== Compilation Flags ===
When compiling a program, you can pass certain flags in order to try and make HONEE run faster.
For the Cisco nodes, this can be done by running the following in your libCEED directory:

make configure OPT='-O3 -march=native -g -ffp-contract=fast -fopenmp-simd'
make build/fluids-navierstokes -Bj

=== General Running Settings ===
Some general settings that will help improve performance are to lower the solver tolerances.
Specifically <code>-snes_rtol 1e-4 -ksp_rtol 1e-4</code> have been found to be pretty good choices.
These may require tweaking if you run into divergence issues.

=== Degree-based settings ===
Optimal solver settings can change based on what degree elements you work with (determined by the <code>-degree</code> flag).
For linear elements (<code>-degree 1</code>), I've had success with:

-amat_type -snes_lag_jacobian 5 -snes_lag_jacobian_persists false -snes_lag_preconditioner 20 -snes_lag_preconditioner_persists true -py_type asm -sub_pc_type lu

While for cubic elements (<code>-degree 3</code>) I've seen better performance with:

-amat_type shell -snes_lag_jacobian 15 -snes_lag_jacobian_persists true -py_type asm -sub_pc_type lu

FDSI Summer Program/HONEE

2024-06-21T15:49:38Z

Jrwrigh: /* Recommendations for HONEE performance */

''Information for tutorial in HONEE, to be updated as needed.''

'''HONEE''', or the '''High-Order Navier-stokes Equation Evaluator''', is a CFD program that combines [https://libceed.org/en/latest/ libCEED] and [https://petsc.org/release/ PETSc].

== Building HONEE for Cisco nodes ==

First, start a job on the Cisco nodes. All the below commands should be run within a terminal on a Cisco node.

=== Setup Envionment ===
source /projects/tools/Spackv0.23/share/spack/setup-env.sh
spack load gcc@12.3
module load openmpi

=== Create new directory ===
mkdir honee_build
cd honee_build

=== Build PETSc===
Ordinarily, I'd only recommend doing a <code>git clone</code>, e.g. <code>git clone https://gitlab.com/petsc/petsc.git</code>.
However, the <code>/nobackup/</code> server is quite slow and this process takes around 20 minutes (normally about a minute on my laptop), so downloading and extracting a tarball may be quicker.
I may cover that later, but for now I'll just cover it via <code>git clone</code>.

git clone https://gitlab.com/petsc/petsc.git
cd petsc
cp /nobackup/uncompressed/jrwrigh/HONEE_Setup/petsc/reconfigure.py
./reconfigure.py
make
export PETSC_DIR=$(pwd) PETSC_ARCH=arch-32
cd ..

The <code>reconfigure.py</code> file has the following:
#!/bin/python3
if __name__ == '__main__':
import sys
import os
sys.path.insert(0, os.path.abspath('config'))
import configure
configure_options = [
'--with-64-bit-indices=0',
'--download-hdf5',
'--download-cgns',
'--download-ctetgen=1',
'--download-parmetis=1',
'--download-metis=1',
'--download-ptscotch=1',
'--with-debugging=0',
'--with-fortran-bindings=0',
'--with-fc=0',
'PETSC_ARCH=arch-32',
'COPTFLAGS=-g -O3',
'CXXOPTFLAGS=-g -O3',
]
configure.petsc_configure(configure_options)

=== Building HONEE ===
Ensure that <code>PETSC_DIR</code> and <code>PETSC_ARCH</code> are set to the desired PETSc installation.

git clone https://github.com/CEED/libCEED.git
cd libCEED
make build/fluids-navierstokes -j

The executable <code>build/fluids-navierstokes</code> is then built and ready for running.

== Changing HONEE Inputs ==
HONEE uses PETSc's input argument system for handling inputs.
They can be provided either via command-line flags or inside a YAML file.
So

./build/fluids-navierstokes -ts_dt 1e-3

is equivalent to

./build/fluids-navierstokes -options_file test.yaml

if <code>test.yaml</code> has this in it:

ts_dt: 1e-3

Note also that PETSc allows for hierarchical flags as well within a YAML file.
So instead of writing

ts_dt: 1e-3
ts_type: alpha
ts_max_time: 2.3

You can write:

ts:
dt: 1e-3
type: alpha
max_time: 2.3

=== Documentation for Specific Flags ===
* [https://libceed.org/en/latest/examples/fluids/ HONEE specific flags]
* [https://petsc.org/main/manualpages/TS/TSSetFromOptions/ Time Stepping (TS) flags]
* [https://petsc.org/main/manualpages/SNES/SNESSetFromOptions/ Non-linear solver (SNES) flags]
* [https://petsc.org/main/manualpages/KSP/KSPSetFromOptions/ Linear solver (KSP) flags]

=== Notable Input Flags ===
* <code>-ts_monitor_solution</code>: This will save the results of a simulation to a file. Example: <code>-ts_monitor_solution cgns:flow_visualization.cgns</code> will save the results to a file called <code>flow_visualization.cgns</code>

=== Restarting Simulations ===
To restart a simulation from a previous simulation's result, they need to be saved to a <code>*.bin</code> file.
This is done using the <code>-continue</code> flag, which will load the file set by <code>-continue_filename</code>.

Note that restarting from a binary file can only be done if the binary file was written by a simulation running at the same part count.

== HONEE and Gmsh ==

=== GMSH/Grid requirements: ===
# The grid must be hexahedron based (so deformed cubes). HONEE can run with tetrahedron grids, but that will require some minor code modifications which we've done before (IIRC, it's a single line of code to change). If anyone is interested in this, let me know and we can work on it.
#* Note that HONEE cannot handle mixed meshes, so meshes with both hexahedron and tetrahedron grids. This is not a minor code change and is probably out of scope for this Summer Program
# In the geo file, you need to specify <code>Physical Surface</code>s to identify the domain boundaries to apply boundary conditions to.
# In the geo file, you need to specify a single <code>Physical Volume</code> that has the entire domain. So if you split the domain into multiple pieces for meshing purposes (as is done with the vortex shedding grids), then you need the <code>Physical Volume</code> to identify all the pieces.

=== YAML Setup: ===
In the YAML file, you need to associate each <code>Physical Surface</code>s with a boundary condition. I describe this briefly in the video, but each <code>Physical Surface</code> as an ID number associated with it. In the YAML file, you need to associate that ID number with a BC choice.
For example, in the <code>vortexshedding.yaml</code> file there is:
# Boundary Settings
bc_slip_z: 6
bc_wall: 5
bc_freestream: 1
bc_outflow: 2
bc_slip_y: 3,4

and the <code>cylinder.geo</code> has:

Physical Surface("inlet") = {102};
Physical Surface("outlet") = {116};
Physical Surface("top") = {80, 120};
Physical Surface("bottom") = {36, 112};
Physical Surface("cylinderwalls") = {94, 28, 50, 72};
Physical Surface("frontandback") = {37, 1, 4, 103, 3, 81, 2, 59, 5, 125};

The "inlet" face is the first identified boundary, so it gets the ID number 1.
In the YAML file, you see <code>bc_freestream: 1</code>, which sets this inlet boundary to be controlled by a freestream BC.
Similar with the "outlet" face; it's the second specified boundary in the geo file, so we set <code>bc_outflow: 2</code>.
And so on with the other faces.

'''NOTE:''' When using the GMSH GUI to identify the Physical Surfaces, it will actually put something like this in the geo file:
Physical Surface("inlet", 3010) = {3005};
Physical Surface("outlet", 3011) = {3003};
Physical Surface("top", 3012) = {3004};
Physical Surface("bottom", 3013) = {3002};
Physical Surface("nacawalls", 3014) = {3008, 3006, 3007};
Physical Surface("frontandback", 3015) = {3009, 3001};

Note the extra <code>3010</code> in the "inlet" definition. I'm not sure if this means that correct ID number for it should be 3010 instead of 1, but that's a possibility.

== Recommendations for HONEE performance ==

=== Compilation Flags ===
When compiling a program, you can pass certain flags in order to try and make HONEE run faster.
For the Cisco nodes, this can be done by running the following in your libCEED directory:

make configure OPT='-O3 -march=native -g -ffp-contract=fast -fopenmp-simd'
make build/fluids-navierstokes -Bj

=== General Running Settings ===
Some general settings that will help improve performance are to lower the solver tolerances.
Specifically <code>-snes_rtol 1e-4 -ksp_rtol 1e-4</code> have been found to be pretty good choices.
These may require tweaking if you run into divergence issues.

=== Degree-based settings ===
Optimal solver settings can change based on what degree elements you work with (determined by the <code>-degree</code> flag).
For linear elements (<code>-degree 1</code>), I've had success with:

-amat_type -snes_lag_jacobian 5 -snes_lag_jacobian_persists false -snes_lag_preconditioner 20 -snes_lag_preconditioner_persists true -py_type asm -sub_pc_type lu

While for cubic elements (<code>-degree 3</code>) I've seen better performance with:

-amat_type shell -snes_lag_jacobian 20 -snes_lag_jacobian_persists true -py_type asm -sub_pc_type lu

FDSI Summer Program/HONEE

2024-06-21T15:35:48Z

Jrwrigh:

VNC

2024-06-12T20:05:18Z

Jrwrigh: /* Killing a VNC Server */

'''Virtual Network Computing (VNC)''' is a tool which projects a GUI session over the network. If may be useful if you want to use GUI tools remotely when X forwarding performs poorly.

VNC works on a client-server architecture, where a remote system (server) runs the programs and sends the virtual screen output to the local system (client). Here, the local system would be your laptop/desktop, while the server is located on the [[PHASTA Group Machines]].

'''For On Ramp purposes, you only need to follow the sub sections identified with steps 1 through 3: Start VNC Server, Clients, and Starting a VNC Viewer'''

== VNC on PHASTA Machines ==
'''Warning: The VNC password is transmitted in clear text over the network and should not be considered secure'''

== Server-Side Setup for PHASTA Machines ==
=== Start VNC Server (Step 1) ===
<code>portal1</code> is designated to host VNC sessions.

First connect to jumpgate via ssh in a terminal on your personal machine (If following the On Ramp you should already have this terminal opened). To start a VNC session on portal1 type the following in your jumpgate terminal with a return after each line:
ssh portal1
source /etc/profile.d/vncscript.sh
start_vnc.sh

This will setup the VNC viewer for your session on the PHASTA machine side of things. Read through the output from <code>start_vnc.sh</code> as it contains crucial information and some tips which are elaborated on in this wiki.

IMPORTANT: Make sure to remember the generated password and port number ("You should connect to 59zw on portal1", "Your password is abcd1234ExamplePW") so that you can reuse the session you just started. Hint: take a screenshot of the output of <code>start_vnc.sh</code> and store it somewhere safe.

It is common practice to leave your VNC session running on <code>portal1</code>. Next time you want to access your desktop, just ssh into jumpgate with a tunnel between portal1's VNC port (59**) and some port on your machine. Then use a VNC client to connect to the port on your machine. See the Client (Step 2) section below for an explanation of this process.

=== Killing a VNC Server ===
If for some reason you want to end your session and kill your virtual desktop, run

source /etc/profile
stop_vnc.sh

ONLY run this if you want to kill your virtual desktop. Most users will never need to do this as the idea is to create one session and continue to use that one for all future usage.

NOTE this will kill ''all'' your VNC sessions.

=== Changing the VNC Password ===
While ssh'd into your session on <code>portal1</code>, type the following command in your terminal:
/opt/tigervnc/bin/vncpasswd

Follow the on screen instructions.

=== View Only Mode ===

To share your desktop with another user in view only mode set a view only password
by running
vncpasswd

Have the other user connect in the same way you would but have them set their viewer to be in view only mode and use your view only password. Typically this is done as follows:
vncviewer -viewonly

=== Changing the Size (Resolution) of an Existing Session ===
''Note: In most modern VNC clients, this can (possibly should) be done via the locally running client. See your VNC client's documentation for details. The below is kept for posterity''

You can usually use the <code>xrandr</code> tool to change the resolution of a running VNC session. First you'll need to know your session's display number (this should be the last digit or two of the port number). For example, if your VNC session is running on port 5902, then your screen number should be :2. For this example, we'll use screen 2.

Once you know your screen number, you can see the list of supported modes as follows:
xrandr -display :2

Once you pick the one you want (generally the same size or smaller than the native resolution of your client), you can choose it by running a command like
xrandr -s 1400x1050 -display :2

(this example will set the resolution to 1400 pixels by 1050 pixels)

You'll probably be disconnected at this point, but when you reconnect your screen size should be changed (hopefully without crashing your running programs).

=== Finding an Existing Session ===
SSH to portal1 and then run:
/opt/vnc_script/findsession.sh

Which will return the shortened port number of each of your currently running sessions.

== Client-Side Setup for PHASTA Machines ==
=== Clients (Step 2) ===

First you need to have a VNC viewer installed on your personal machine. <code>portal1</code> uses TurboVNC from the VirtualGL project, available from [http://www.virtualgl.org/Downloads/TurboVNC their website]

Other VNC viewers will also work, such as [https://www.tightvnc.com/download.php TightVNC] and [https://www.realvnc.com/en/connect/download/viewer/ RealVNC]. Download and install the one you like best for your personal machine (RealVNC seems to be the go to for Mac users in the group).

The instructions given when you started the session on <code>portal1</code> (the ones you should have taken a screen shot of in Step 1 above) are OK, but it always tells you to start a session (in a terminal on your mac or linux machine command line) with the <code>ssh -L5905:portal1:59zw jumpgate-phasta.colorado.edu</code> command, where <code>zw</code> will be different for each user. That suggestion is OK if you never plan to connect to anyone else's session, but since we often collaborate by sharing VNC sessions, the better practice we adopt is to use <code>zw</code> in place of the suggested 05 (which just an arbitrary local port on your laptop). For example, when I created a session the last four numbers of the connection were <code>5923</code>, thus <code>zw</code> for me is <code>23</code> and the best practice is to ignore the script suggestion in favor of <code>ssh -L5923:portal1:5923 jumpgate-phasta.colorado.edu</code>

The easiest way to set up the portal from the get-go (after you have created the VNC session on the PHASTA machine side of things and left it open, AKA Step 1 above) is to close out of the terminal on your personal machine, reopen it, and run only this command where <code>zw</code> is your assigned port number:

ssh -L59zw:portal1:59zw USERNAME@jumpgate-phasta.colorado.edu

'''Windows 8'''
- Same concept as above... close the terminal that was established with PuTTY. Start a new one using PuTTY and entering the following in the Host Name:
<code>USERNAME@jumpgate-phasta.colorado.edu</code>
Again ensuring connection type is set to SSH.
- Next, look to the panel on the left side of the PuTTY window, find the "Connection -> SSH -> Tunnels" menu option and click on "Tunnels". This will
open a settings page for the tunnel we are trying to establish.
- In the "Source Port" field enter <code>59zw</code>, and in the "Destination" field enter
<code>portal1:59zw</code>, again replacing <code>zw</code> with your assigned port number.
- Click the "Add" button.
- Click on "Open". You now have an established VNC connection with <code>portal1</code> and it will ask you to enter your (jumpgate) password.
- Remaining steps are the same as non-Windows 8.

You will then be prompted to enter your password (not the one that was just created for the VNC, but the one that you usually use to connect to jumpgate). After entering your password you will be ready for "Starting a VNC Viewer (Step 3)" in the next section. The reason that you must exit and reopen the command prompt is because you cannot connect to the VNC portal zw if you are already connected to jumpgate or portal1; you must connect to jumpgate and your session zw simultaneously for the VNC portal to work correctly.

=== Starting a VNC viewer (Step 3) ===

Whether you followed the Mac or Linux or Windows instructions above, successful completion will have established a tunnel from your laptop to <code>portal1</code>. The last step is to start a VNC viewer (graphical windowing program) that uses this tunnel to display your <code>portal1</code> session on the screen of your laptop. There are a variety of choices for each platform and they evolve but their operation seems, thankfully, pretty universal.

The most basic input is that they provide a box that says "VNC Server". You will want to type <code>localhost:zw</code> in that box and then apply start (or whatever the particular applications action button is). The VNC Viewer will them prompt you for the password that was generated for this "zw" numbered session (Your version of the generated "abcd1234ExamplePW" from previous steps. Note if you changed that VNC password already, use the one you created).

The <code>zw</code> here will need to be whatever you used for the <code>-L59zw:portal1:59zw</code> part of the tunnel. What the number is actually doing is telling this local VNC viewer on which port you have established a tunnel with the session for which you are starting the VNC viewer. Note it is possible to have several different simultaneous sessions. Each needs its own tunnel (established by repeating the process above with a unique <code>zw</code>) and then a corresponding "new" VNC viewer. In this way we might continue to work in our own session but then take a break to look at a collaborator's session without having to close out the tunnel and viewer pair.

Now that you've connected to the PHASTA machines, it's time to learn a bit mroe about using a UNIX based system (Linux). To continue with the On Ramp click [[UNIX|here]]

=== Web Based Viewer ===

If you can't or don't want to install a VNC viewer you can use a Java based one. You will need a JVM and a Java browser plugin. You will also need the port that the start_vnc script assigned you to be free on your local computer

Forward your session through jumpgate as before before, adding a second port, 580n. For example, if the script tells you to

ssh -L5905:portal1:5902 jumpgate-phasta.colorado.edu you should
ssh -L5902:portal1:5902 -L5802:portal1:5802 jumpgate-phasta.colorado.edu
Then point your browser to http://localhost:5802 and log in with the password specified by the script when prompted. (Replace 2 with the value specified by the script)

== OpenGL ==

<code>portal1</code> is equipped with a VirtualGL install which will allow you to use OpenGL programs (which do not use pthreads)

Simply wrap your OpenGL program with the <code>vglrun</code> command
vlgrun glxgears

Our lab has 2 VirtualGL servers you can connect to from <code>portal1</code>. You must connect to one of them for large memory and/or computationally intensive processes.
The names of the servers are <code>viz002</code> and <code>viz003</code> (<code>viz001</code> is probably never coming back to life).
<code>portal1</code> doesn't have a particularly fast graphics processor and MUST NOT be used for large memory or computationally intensive process)

vglconnect -s viz002
or
vglconnect -s viz003

from this connection you will want to run graphic applications (e.g., SimModeler or ParaView) prefaced by the command <code>vglrun</code>. You can test that you have it setup right
with the toy-app <code>glxgears</code> as follows

vglrun glxgears

Note that VGL uses a number of threads. If you have trouble with <code>vglrun</code> crashing with a message about <code>Thread::Start()</code> make sure you haven't set your stack size too large (remove any <code>ulimit -s</code> or <code>ulimit -n</code> calls from your shell start scripts).

NOTE ALSO: The primary purpose for <code>viz00x</code> is for visualization and for debugging. Production runs should be done elsewhere.

== Troubleshooting ==

If you have used vncserver (It doesn't matter which version) before, you will need to clear your vnc settings for the script to work. You can do this by running rm -rf ~/.vnc

stop_vnc.sh may display some errors; this is normal.

If you have trouble deleting ~/.vnc send an email to Benjamin.A.Matthews@colorado.edu

If any of these commands fail, you may need to source /etc/profile to get the necessary environment variables (this should be fixed soon)

VirtualGL has trouble with some threaded programs. If your OpenGL program exhibits segmentation faults or other issues, this could be the problem. Check back for the solution later.

If the given password is rejected you can run stop_vnc.sh and restart to get a new one. Occasionally the random password generator may generate passwords which VNC doesn't like.

If VirtualGL complains about not being able to get a 24bit FB config either vglconnect to another VirtualGL enabled server or complain to Benjamin.A.Matthews@Colorado.edu

If your VNC connection is very slow, you might want to try changing the compression and encoding options. See your vncviewer's documentation or try this
vncviewer -encodings tight -quality 6 -compresslevel 6
If you have trouble with text distortion try adding
-nojpeg

If you're running OSX and see an error about Zlib, try changing your compression settings (maximum quality usually works) or use a different client. RealVNC and certain versions of ChickenOfTheVNC both exhibit this issue. The latest build of TigerVNC should work reliably, as does the Java based TightVNC client.

FDSI Summer Program/HONEE

2024-06-05T23:05:14Z

Jrwrigh:

FDSI Summer Program/HONEE

2024-06-05T22:58:44Z

Jrwrigh: /* Restarting Simulations */

FDSI Summer Program/HONEE

2024-06-04T20:47:24Z

Jrwrigh:

FDSI Summer Program/HONEE

2024-06-03T16:31:20Z

Jrwrigh:

FDSI Summer Program/HONEE

2024-06-03T16:28:52Z

Jrwrigh: /* Building HONEE */

FDSI Summer Program/HONEE

2024-06-03T14:53:49Z

Jrwrigh:

FDSI Summer Program/HONEE

2024-06-03T14:45:44Z

Jrwrigh:

''Information for tutorial in HONEE, to be updated as needed.''

'''HONEE''', or the '''High-Order Navier-stokes Equation Evaluator''', is a CFD program that combines libCEED and PETSc.

== Building HONEE for Cisco nodes ==

First, start a job on the Cisco nodes. All the below commands should be run within a terminal on a Cisco node.

=== Setup Envionment ===
source /projects/tools/Spackv0.23/share/spack/setup-env.sh
spack load gcc@12.3
module load openmpi

=== Create new directory ===
mkdir honee_build
cd honee_build

=== Build PETSc===
Ordinarily, I'd only recommend doing a <code>git clone</code>, e.g. <code>git clone https://gitlab.com/petsc/petsc.git</code>.
However, the <code>/nobackup/</code> server is quite slow and this process takes around 20 minutes (normally about a minute on my laptop), so downloading and extracting a tarball may be quicker.
I may cover that later, but for now I'll just cover it via <code>git clone</code>.

git clone https://gitlab.com/petsc/petsc.git
cd petsc
cp /nobackup/uncompressed/jrwrigh/HONEE_Setup/petsc/reconfigure.py
./reconfigure.py
make
export PETSC_DIR=$(pwd) PETSC_ARCH=arch-32
cd ..

The <code>reconfigure.py</code> file has the following:
#!/bin/python3
if __name__ == '__main__':
import sys
import os
sys.path.insert(0, os.path.abspath('config'))
import configure
configure_options = [
'--with-64-bit-indices=0',
'--download-hdf5',
'--download-cgns',
'--download-ctetgen=1',
'--download-parmetis=1',
'--download-metis=1',
'--download-ptscotch=1',
'--with-debugging=0',
'--with-fortran-bindings=0',
'--with-fc=0',
'PETSC_ARCH=arch-32',
'COPTFLAGS=-g -O3',
'CXXOPTFLAGS=-g -O3',
]
configure.petsc_configure(configure_options)

=== Building HONEE ===
Ensure that <code>PETSC_DIR</code> and <code>PETSC_ARCH</code> are set to the desired PETSc installation.

git clone https://github.com/CEED/libCEED.git
make build/fluids-navierstokes -j

The executable <code>build/fluids-navierstokes</code> is then built and ready for running.

== Changing HONEE Inputs ==
HONEE uses PETSc's input argument system for handling inputs.
They can be provided either via command-line flags or inside a YAML file.
So

./build/fluids-navierstokes -ts_dt 1e-3

is equivalent to

./build/fluids-navierstokes -options_file test.yaml

if <code>test.yaml</code> has this in it:

ts_dt: 1e-3

Note also that PETSc allows for hierarchical flags as well within a YAML file.
So instead of writing

ts_dt: 1e-3
ts_type: alpha
ts_max_time: 2.3

You can write:

ts:
dt: 1e-3
type: alpha
max_time: 2.3

=== Documentation for Specific Flags ===
* [https://libceed.org/en/latest/examples/fluids/ HONEE specific flags]
* [https://petsc.org/main/manualpages/TS/TSSetFromOptions/ Time Stepping (TS) flags]
* [https://petsc.org/main/manualpages/SNES/SNESSetFromOptions/ Non-linear solver (SNES) flags]
* [https://petsc.org/main/manualpages/KSP/KSPSetFromOptions/ Linear solver (KSP) flags]

FDSI Summer Program/HONEE

2024-06-03T14:30:29Z

Jrwrigh: Created page with "''Information for tutorial in HONEE, to be updated as needed.'' '''HONEE''', or the '''High-Order Navier-stokes Equation Evaluator''', is a CFD program that combines libCEED..."

''Information for tutorial in HONEE, to be updated as needed.''

'''HONEE''', or the '''High-Order Navier-stokes Equation Evaluator''', is a CFD program that combines libCEED and PETSc.

== Changing HONEE Inputs ==
HONEE uses PETSc's input argument system for handling inputs.
They can be provided either via command-line flags or inside a YAML file.
So

./build/fluids-navierstokes -ts_dt 1e-3

is equivalent to

./build/fluids-navierstokes -options_file test.yaml

if <code>test.yaml</code> has this in it:

ts_dt: 1e-3

Note also that PETSc allows for hierarchical flags as well within a YAML file.
So instead of writing

ts_dt: 1e-3
ts_type: alpha
ts_max_time: 2.3

You can write:

ts:
dt: 1e-3
type: alpha
max_time: 2.3

=== Documentation for Specific Flags ===
* [https://libceed.org/en/latest/examples/fluids/ HONEE specific flags]
* [https://petsc.org/main/manualpages/TS/TSSetFromOptions/ Time Stepping (TS) flags]
* [https://petsc.org/main/manualpages/SNES/SNESSetFromOptions/ Non-linear solver (SNES) flags]
* [https://petsc.org/main/manualpages/KSP/KSPSetFromOptions/ Linear solver (KSP) flags]

Exporting Parasolid from SolidWorks

2024-04-11T15:06:32Z

Jrwrigh:

Save out your Model as a parasolid from [[SolidWorks]]. Note that you will want the geometry as close to ready for meshing as possible, as performing model "surgery" in SimModeler is not always straight forward. The outputted file from SolidWorks will have the format <code><file_name>.x_t</code>.

If you do not have a parasolid model of your own, you may use the On Ramp example file located at:

/projects/tutorials/OnRamp/example_geom.x_t

Ensure that you are on one of the viznodes and not portal1. You may tunnel to viz003 by opening a terminal and running:

vglconnect -s viz003

Navigate to and copy your file into your working directory. Typically, we create a folder where all the simulation files are stored. For example, after opening a terminal I could run the commands:

mkdir Demo
cd Demo

This would place me in my working directory 'Demo'. To copy all the files for the On Ramp demo, run this in your 'Demo' directory:

cp /projects/tutorials/OnRamp/* .

Next, you'll want to change the parasolid file extension from <code>.x_t</code> to <code>.xmt_txt</code>. To do this, run <code>mv <file_name>.x_t <file_name>.xmt_txt</code> from your terminal. From here, you are ready for the ''convert'' step. The convert step is documented here: https://fluid.colorado.edu/tutorials/tutorialVideos/Convert2Sim_Tutorial.mp4

'''Summary of video:'''
1. Ensure <code>convertParasolid2Sim.sh</code> and <code><file name>.xmt_txt</code> are in your working directory.

2. Set environment with soft adds found in <code>more ~kjansen/soft-core.sh</code>

3. Run <code>./convertParasolid2sim.sh <file name>.xmt_txt</code> in your terminal

4. Convert step is complete and your directory now contains 3 new files: <code>model.smd</code> <code>relations.log</code> & <code>translated-model.smd</code>. The <code>translated-model.smd</code> file is the one we need moving forward in this tutorial.

Once the convert step is complete, you are ready to move onto the next step and use [[Getting Started with Simmodeler| SimModeler]] to create a mesh for the new <code>translated-model.smd</code> file we created!

== Geometries with Standalone Surfaces ==

There are sometimes cases where models will contain not only three dimensional solid bodies but also surfaces. Surfaces are treated differently by Solidworks and parasolid files than solid bodies, so they need to be exported separately. In order to do this, you select all entities except the surface and save as a parasolid, after you hit save, Solidworks will ask if you want to save all of the geometry or only the selected geometry, the second option being the one we want. Now do the same but only select the surface. In newer versions of Simmodler, only the domain needs to be run though the conversion and the surfaces can be added as parasolid files. In older versions, these two separate parasolid files needed to be converted to <code>.smd</code> separately (remember to rename files after they are converted to avoid overwriting) and recombined in Simmodeler. The toolchain seems to fail for multiple unconnected surfaces, so if this is the case, take care to export them separately.

To recombine, simply open the solid body file in Simmodeler, under the "Modeling" tab select "Add Parts", add the surfaces file, then select "Make New Manifold Model" which will combine the files into one model that is suitable for PHASTA.

PHASTA/Restart Ordering

2024-03-12T20:45:32Z

Jrwrigh: /* Optional Outputs */

Knowledge of the ordering of restarts is most useful for the creation of <code>.pht</code> or <code>.phts</code> files (used to tell ParaView what to read in from restarts in POSIX or SyncIO format respectively). There are both standard outputs from PHASTA that will be written no matter what, and optional ones that may depend on the simulation type or what type of analysis is planned on being done.

In the following, index numbering will be in index-by-1 format, or Fortran format. When creating a <code>.pht</code> or <code>.phts</code> file and filling in the <code>start_index_in_phasta_array=" "</code> section for each field, note that you should subtract one as ParaView reads this file using index-by-0.

NOTE:
This page is a work in progress and none of this information should be taken as fully accurate while this warning persists

== Standard Outputs ==

The output flow variables written to restarts are dependent on the choice of variables used to solve a given setup. These output flow variables are stored under the header
<code>solution</code>
And for pressure primitive, have ordering:
* Pressure
* Velocity (vector quantity, ordered x, y, z)
* Temperature (if solving the compressible equations, this field can be ignored if incompressible)

In the case where turbulence models are being used or species are being solved for, <code>solution</code> will also populate scalar fields starting in the 6th field in sequential ordering:
* Scalar 1
* Scalar 2

The final ordering is then:

{| class="wikitable"
|+ Pressure Primitive Fields
|-
! Field Number !! Description !! Notes
|-
| 1 || Pressure ||
|-
| 2:4 || Velocity || Vector quantity, ordered u, v, w
|-
| 5 || Temperature || This field can be ignored if incompressible
|-
| 6 || Scalar 1 || Only written if scalar 1 exists
|-
| 7 || Scalar 2 || Only written if scalar 2 exists
|}

The exact quantity that any scalar represents depends on the models being used, though as some examples, for the Spalart-Allmaras (SA) 1-equation RANS model or DES/DDES formulations using the SA model, scalar 1 will be &nu t. Some branches of the code may have the ability to solve more scalar equations and those extra scalars will be appended, in order, to the end of the <code>solution</code> field.

=== Time Derivatives ===
The time derivatives of all of the above fields are also present in the restarts, under the header <code>time derivative of solution</code>. These fields are in the exact same order as in <code>solution</code>, and are subject to the same caveats about how many scalars there may be and what they actually represent for any given branch of the code and simulation setup.

{| class="wikitable"
|+ Pressure Primitive Fields
|-
! Field Number !! Description !! Notes
|-
| 1 || Time derivative of pressure ||
|-
| 2:4 || Time derivative of velocity || Vector quantity, ordered u, v, w
|-
| 5 || Time derivative of temperature || This field can be ignored if incompressible
|-
| 6 || Time derivative of scalar 1 || Only written if scalar 1 exists
|-
| 7 || Time derivative of scalar 2 || Only written if scalar 2 exists
|}

== Optional Outputs ==

=== Wall Distance ===
This quanity is simply a measure of the distance from any given node to the nearest wall point in the simulation. This quantity is mostly used in turbulence models but can be useful for the post-processing of complex domains
*Header: <code>dwal</code>
*<code>solver.inp</code> flag: <code>placeholder</code>

{| class="wikitable"
|+ Fields
|-
! Field Number !! Description !! Notes
|-
| 1 || Wall Distance ||
|}

=== Vorticity ===
*Header: <code>vorticity</code>
*<code>solver.inp</code> flag: <code>Print vorticity: True</code>

{| class="wikitable"
|+ Fields
|-
! Field Number !! Description !! Notes
|-
| 1:3 || Vorticity || Vector quantity, ordered ωx, ωy, ωz
|-
| 4 || Magnitude of vorticity ||
|-
| 5 || Q || Defined as the second invariant of the velocity gradient tensor
|}

=== Time Averaged Statistics (point-wise) ===
Point-wise time averaged statistics are useful for problems where there are no homogeneous directions in the flow to accumulate an average along. Instead, PHASTA can accumulate averages at each individual node. Note that this formulation only accumulates per-run, so an average is only computed from the start step of the current run, and total averages must be computed by adding successive averages with the appropriate weighting.
*Header: <code>ybar</code>
*<code>solver.inp</code> flag: <code>Print ybar: True</code>

{| class="wikitable"
|+ Fields (incompressible)
|-
! Field Number !! Description !! Notes
|-
| 1 || Average velocity || Vector quantity, ordered u, v, w
|-
| 2:4 || Average pressure ||
|-
| 5 || Average speed ||
|-
| 6:8 || Average of the square of velocity || Vector quantity, ordered u2, v2, w2
|-
| 9 || Average of the square of pressure ||
|-
| 10:12 || Average of velocity cross-components || Vector quantity, ordered uv, uw, vw
|-
| 13 || Average of scalar 1 ||
|-
| 14:16 || Average of vorticity || Vector quantity, ordered ωx, ωy, ωz
|-
| 17 || Average vorticity magnitude ||
|-
| 18 || Average of scalar 2 ||
|}

Average of Q may be in position 19 depending on the branch

=== Wall Shear Stress ===
This field is only defined at wall points and is used to get the most accurate measure of the wall shear stress possible as otherwise a finite gradient using the first point off of the wall must be implemented, which may or may not be normal to a wall point underneath of it.
*Header: <code>wss</code>
*<code>solver.inp</code> flag: <code>Print Wall Fluxes: True</code> (check this)

{| class="wikitable"
|+ Fields
|-
! Field Number !! Description !! Notes
|-
| 1:3 || Wall shear stress || Vector quantity, ordered τwallx, τwally, τwallz
|}

=== Time Averaged Wall Shear Stress ===
*Header: <code>wssbar</code>
*<code>solver.inp</code> flag: <code>Print Wall Fluxes: True</code> and <code>Print ybar: True</code>

{| class="wikitable"
|+ Fields
|-
! Field Number !! Description !! Notes
|-
| 1:3 || Average wall shear stress || Vector quantity, ordered τwallx, τwally, τwallz
|}

=== Pressure Projection Vectors ===
*Header: <code>pressure projection vectors</code>
*<code>solver.inp</code> flag: <code>placeholder</code>

UNIX

2023-08-08T14:49:33Z

Jrwrigh: /* Command Line Basics */

Most of our systems (and general HPC resources) run some UNIX derivative. Much of the software is command line based, so it's worthwhile to learn the basics.

There are tons of free resources on the web for getting started ([https://www.tipsandtricks-hq.com/basic-unix-commands-list-366#:~:text=File%2FDirectory%20operation%20related%20Unix%20Commands.%201%20cp%20%E2%80%93,which%20is%20the%20pathname%20of%20a%20directory.%20 Basic UNIX Commands]). There should also be a "for dummies" book in the lab.

As you find resources that are helpful, please update this page.

== Connecting (SSH) ==
Windows:
[http://www.chiark.greenend.org.uk/~sgtatham/putty PuTTY SSH Client]
[http://winscp.net/eng/index.php WinSCP file transfer tool]

MacOS and Linux users can use [http://openssh.org/ OpennSSH] on the command line (it generally comes with the OS).

=== Resources for setting up ssh keys ===

Visual guide to how ssh-keys and <code>ssh-agent</code> work: [http://www.unixwiz.net/techtips/ssh-agent-forwarding.html An Illustrated Guide to SSH Agent Forwarding]

For setting up <code>ssh-agent</code> (so you don't have to type your password over and over): [http://blog.joncairns.com/2013/12/understanding-ssh-agent-and-ssh-add/ Understanding ssh-agent and ssh-add]

Script for automatically starting <code>ssh-agent</code> on login of a machine (place in your <code>.profile</code>/<code>.bash_profile</code>): [https://stackoverflow.com/a/18915067/7564988 StackOverflow: Start ssh-agent on login]

=== Visual guide to ssh tunnels (ie. port forwarding) ===

A good resource to understand port-forwarding, and more advanced uses of port-forwarding: [https://robotmoon.com/ssh-tunnels/ A visual guide to SSH tunnels]

== Command Line Basics ==

[https://explainshell.com/# Explain Shell]: Copy/paste a CLI command, and it will tell you what all the flags mean

[https://arstechnica.com/gadgets/2021/08/linux-bsd-command-line-101-using-awk-sed-and-grep-in-the-terminal/ How to ''think'' when using <code>grep</code>, <code>awk</code>, and <code>sed</code>]: Article going over the basic uses of <code>grep</code>, <code>awk</code>, and <code>sed</code> and how to think through their use cases.

[https://www.funtoo.org/Awk_by_Example,_Part_1 Learning Awk by Example]: Tutorials going over how to use awk, while actually using examples.

[https://www.tldp.org/LDP/abs/html/ Advanced Bash-Scripting Guide]

== Graphical Sessions (VNC) ==
See [[VNC]]

== File Permissions and ACL ==
See [[File_Permissions_Basics_and_ACL|File Permissions Basics and ACL]]

Paraview Trace

2023-06-17T14:34:35Z

Jrwrigh: /* Basic Changes to the Python Script */

Paraview traces are ways to create a python script of a set of actions that can be later applied to different datasets or over a loop of datasets automatically. The following will brefly explain the creation, cleaning, and running of a python trace in Paraview (which will often be shortened to pvTrace or something similar).

=== Creation ===
General steps are to start a trace, do whatever you're wanting to record, then stop the trace.

'''1. Start Trace'''

Tools > Start Trace

This will bring up a dialog with different options for which things to record. Of note is "Skip Rendering Components" (useful if your trace doesn't involve any visual capturing). You can also select "Show Incremental Trace" so to verify what command are being recorded as you do them.

'''2. Do something'''

Do whatever you were wanting to have recorded in Paraview.

'''3. Stop Trace'''

Tools > Stop Trace

This will then bring up the resulting Python script, which you can then save to a file, edit, and rerun.

=== Basic Changes to the Python Script ===

When rerunning a trace script, it's just run via a normal Python interpreter in a Paraview-specific environment. All Python standard library packages are available to the script.

This allows for significant flexibility in what can be done in and during the script. For example, in [https://github.com/PHASTA/utilities/blob/609efbe8879b7c341bc2dc29d14df4e90ad1084d/general/ParaView_Interpolation/BatchSync.py#L131 this Paraview script file], originally created from a trace, we use a for loop to do the same action multiple times. Additionally, the <code>os</code> package is used to symlink and delete files so that the data read in during the for loop changes. We've even added <code>print</code> statements to log what the script is doing.

==== Python Versions ====
'''Note: The version of Python used by Paraview is often different than the version installed on your system.'''

To determine the version of Python used by Paraview, you can select "View > Python Shell" in Paraview, which will show the Python interpreter. The version of Python used should be printed on the shell at the top. If not, run <code>import sys; print(sys.version)</code> to display the Python version information. If you're using Paraview through the server/client mode, you ''MUST'' do this while attached to a <code>pvserver</code>; the version of Python on your local Paraview client is not guaranteed to match the version on the server.

The Python Shell in Paraview can also be used to verify if certain functions will work and what libraries are available (such as [https://numpy.org/ numpy]).

=== Running a Trace ===
Running a pvTrace, whether on the same or a different dataset requires only a few key steps. For now, it will be assumed that the trace is being run on Cooley at ALCF, though many of the steps should be shared for other machines.

If you are running the trace on a dataset that is not the same as the one for which you created the trace, it is good practice to always check your script and inputs. Make sure you have all of the restart and geombc files that you will need, and that you are pointing to the correct locations in the python script (it is recommended that you use absolute paths to reduce the chances for error). Also check your output file name and location.

Once your python script is ready, you need to get an interactive allocation on Cooley and load the same version of paraview with which the trace was created:

<code>soft add +paraview-5.5.2</code>

This loads <code>pvpython</code> which is what is used to run the python trace script. To run this scrip is simply a modification of a standard python run command.

<code>pvpython Trace_Script.py</code>

Paraview Trace

2023-06-17T14:33:45Z

Jrwrigh: /* Basic Changes to the Python Script */

Paraview traces are ways to create a python script of a set of actions that can be later applied to different datasets or over a loop of datasets automatically. The following will brefly explain the creation, cleaning, and running of a python trace in Paraview (which will often be shortened to pvTrace or something similar).

=== Creation ===
General steps are to start a trace, do whatever you're wanting to record, then stop the trace.

'''1. Start Trace'''

Tools > Start Trace

This will bring up a dialog with different options for which things to record. Of note is "Skip Rendering Components" (useful if your trace doesn't involve any visual capturing). You can also select "Show Incremental Trace" so to verify what command are being recorded as you do them.

'''2. Do something'''

Do whatever you were wanting to have recorded in Paraview.

'''3. Stop Trace'''

Tools > Stop Trace

This will then bring up the resulting Python script, which you can then save to a file, edit, and rerun.

=== Basic Changes to the Python Script ===

When rerunning a trace script, it's just run via a normal Python interpreter in a Paraview-specific environment. All Python standard library packages are available to the script.

This allows for significant flexibility in what can be done in and during the script. For example, in [https://github.com/PHASTA/utilities/blob/609efbe8879b7c341bc2dc29d14df4e90ad1084d/general/ParaView_Interpolation/BatchSync.py#L131 this Paraview script file], originally created from a trace, we use a for loop to do the same action multiple times. Additionally, the <code>os</code> package is used to symlink and delete files so that the data read in during the for loop changes.

==== Python Versions ====
'''Note: The version of Python used by Paraview is often different than the version installed on your system.'''

To determine the version of Python used by Paraview, you can select "View > Python Shell" in Paraview, which will show the Python interpreter. The version of Python used should be printed on the shell at the top. If not, run <code>import sys; print(sys.version)</code> to display the Python version information. If you're using Paraview through the server/client mode, you ''MUST'' do this while attached to a <code>pvserver</code>; the version of Python on your local Paraview client is not guaranteed to match the version on the server.

The Python Shell in Paraview can also be used to verify if certain functions will work and what libraries are available (such as [https://numpy.org/ numpy]).

=== Running a Trace ===
Running a pvTrace, whether on the same or a different dataset requires only a few key steps. For now, it will be assumed that the trace is being run on Cooley at ALCF, though many of the steps should be shared for other machines.

If you are running the trace on a dataset that is not the same as the one for which you created the trace, it is good practice to always check your script and inputs. Make sure you have all of the restart and geombc files that you will need, and that you are pointing to the correct locations in the python script (it is recommended that you use absolute paths to reduce the chances for error). Also check your output file name and location.

Once your python script is ready, you need to get an interactive allocation on Cooley and load the same version of paraview with which the trace was created:

<code>soft add +paraview-5.5.2</code>

This loads <code>pvpython</code> which is what is used to run the python trace script. To run this scrip is simply a modification of a standard python run command.

<code>pvpython Trace_Script.py</code>

Paraview Trace

2023-06-17T14:29:25Z

Jrwrigh: /* Basic Changes to the Python Script */

Paraview traces are ways to create a python script of a set of actions that can be later applied to different datasets or over a loop of datasets automatically. The following will brefly explain the creation, cleaning, and running of a python trace in Paraview (which will often be shortened to pvTrace or something similar).

=== Creation ===
General steps are to start a trace, do whatever you're wanting to record, then stop the trace.

'''1. Start Trace'''

Tools > Start Trace

This will bring up a dialog with different options for which things to record. Of note is "Skip Rendering Components" (useful if your trace doesn't involve any visual capturing). You can also select "Show Incremental Trace" so to verify what command are being recorded as you do them.

'''2. Do something'''

Do whatever you were wanting to have recorded in Paraview.

'''3. Stop Trace'''

Tools > Stop Trace

This will then bring up the resulting Python script, which you can then save to a file, edit, and rerun.

=== Basic Changes to the Python Script ===

When rerunning a trace script, it's just run via a normal Python interpreter in a Paraview-specific environment. All Python standard libraries are available to the script.

==== Python Versions ====
'''Note: The version of Python used by Paraview is often different than the version installed on your system.'''

To determine the version of Python used by Paraview, you can select "View > Python Shell" in Paraview, which will show the Python interpreter. The version of Python used should be printed on the shell at the top. If not, run <code>import sys; print(sys.version)</code> to display the Python version information. If you're using Paraview through the server/client mode, you ''MUST'' do this while attached to a <code>pvserver</code>; the version of Python on your local Paraview client is not guaranteed to match the version on the server.

The Python Shell in Paraview can also be used to verify if certain functions will work and what libraries are available (such as [https://numpy.org/ numpy]).

=== Running a Trace ===
Running a pvTrace, whether on the same or a different dataset requires only a few key steps. For now, it will be assumed that the trace is being run on Cooley at ALCF, though many of the steps should be shared for other machines.

If you are running the trace on a dataset that is not the same as the one for which you created the trace, it is good practice to always check your script and inputs. Make sure you have all of the restart and geombc files that you will need, and that you are pointing to the correct locations in the python script (it is recommended that you use absolute paths to reduce the chances for error). Also check your output file name and location.

Once your python script is ready, you need to get an interactive allocation on Cooley and load the same version of paraview with which the trace was created:

<code>soft add +paraview-5.5.2</code>

This loads <code>pvpython</code> which is what is used to run the python trace script. To run this scrip is simply a modification of a standard python run command.

<code>pvpython Trace_Script.py</code>

Paraview Trace

2023-06-17T14:28:50Z

Jrwrigh: /* Basic Changes to the Python Script */

Paraview traces are ways to create a python script of a set of actions that can be later applied to different datasets or over a loop of datasets automatically. The following will brefly explain the creation, cleaning, and running of a python trace in Paraview (which will often be shortened to pvTrace or something similar).

=== Creation ===
General steps are to start a trace, do whatever you're wanting to record, then stop the trace.

'''1. Start Trace'''

Tools > Start Trace

This will bring up a dialog with different options for which things to record. Of note is "Skip Rendering Components" (useful if your trace doesn't involve any visual capturing). You can also select "Show Incremental Trace" so to verify what command are being recorded as you do them.

'''2. Do something'''

Do whatever you were wanting to have recorded in Paraview.

'''3. Stop Trace'''

Tools > Stop Trace

This will then bring up the resulting Python script, which you can then save to a file, edit, and rerun.

=== Basic Changes to the Python Script ===

When rerunning a trace script, it's just run via a normal Python interpreter with a slightly modified path. All Python standard libraries are available to the script.

==== Python Versions ====
'''Note: The version of Python used by Paraview is often different than the version installed on your system.'''

To determine the version of Python used by Paraview, you can select "View > Python Shell" in Paraview, which will show the Python interpreter. The version of Python used should be printed on the shell at the top. If not, run <code>import sys; print(sys.version)</code> to display the Python version information. If you're using Paraview through the server/client mode, you ''MUST'' do this while attached to a <code>pvserver</code>; the version of Python on your local Paraview client is not guaranteed to match the version on the server.

The Python Shell in Paraview can also be used to verify if certain functions will work and what libraries are available (such as [https://numpy.org/ numpy]).

=== Running a Trace ===
Running a pvTrace, whether on the same or a different dataset requires only a few key steps. For now, it will be assumed that the trace is being run on Cooley at ALCF, though many of the steps should be shared for other machines.

If you are running the trace on a dataset that is not the same as the one for which you created the trace, it is good practice to always check your script and inputs. Make sure you have all of the restart and geombc files that you will need, and that you are pointing to the correct locations in the python script (it is recommended that you use absolute paths to reduce the chances for error). Also check your output file name and location.

Once your python script is ready, you need to get an interactive allocation on Cooley and load the same version of paraview with which the trace was created:

<code>soft add +paraview-5.5.2</code>

This loads <code>pvpython</code> which is what is used to run the python trace script. To run this scrip is simply a modification of a standard python run command.

<code>pvpython Trace_Script.py</code>

Paraview Trace

2023-06-17T14:19:21Z

Jrwrigh: /* Creation */

Paraview Trace

2023-06-17T14:19:01Z

Jrwrigh: /* Creation */

UNIX

2023-03-07T05:34:36Z

Jrwrigh: Add "visual guide to ssh tunnels

Most of our systems (and general HPC resources) run some UNIX derivative. Much of the software is command line based, so it's worthwhile to learn the basics.

There are tons of free resources on the web for getting started ([https://www.tipsandtricks-hq.com/basic-unix-commands-list-366#:~:text=File%2FDirectory%20operation%20related%20Unix%20Commands.%201%20cp%20%E2%80%93,which%20is%20the%20pathname%20of%20a%20directory.%20 Basic UNIX Commands]). There should also be a "for dummies" book in the lab.

As you find resources that are helpful, please update this page.

== Connecting (SSH) ==
Windows:
[http://www.chiark.greenend.org.uk/~sgtatham/putty PuTTY SSH Client]
[http://winscp.net/eng/index.php WinSCP file transfer tool]

MacOS and Linux users can use [http://openssh.org/ OpennSSH] on the command line (it generally comes with the OS).

=== Resources for setting up ssh keys ===

Visual guide to how ssh-keys and <code>ssh-agent</code> work: [http://www.unixwiz.net/techtips/ssh-agent-forwarding.html An Illustrated Guide to SSH Agent Forwarding]

For setting up <code>ssh-agent</code> (so you don't have to type your password over and over): [http://blog.joncairns.com/2013/12/understanding-ssh-agent-and-ssh-add/ Understanding ssh-agent and ssh-add]

Script for automatically starting <code>ssh-agent</code> on login of a machine (place in your <code>.profile</code>/<code>.bash_profile</code>): [https://stackoverflow.com/a/18915067/7564988 StackOverflow: Start ssh-agent on login]

=== Visual guide to ssh tunnels (ie. port forwarding) ===

A good resource to understand port-forwarding, and more advanced uses of port-forwarding: [https://robotmoon.com/ssh-tunnels/ A visual guide to SSH tunnels]

== Command Line Basics ==

[https://explainshell.com/# Explain Shell]: Copy/paste a CLI command, and it will tell you what all the flags mean

[https://arstechnica.com/gadgets/2021/08/linux-bsd-command-line-101-using-awk-sed-and-grep-in-the-terminal/ How to ''think'' when using <code>grep</code>, <code>awk</code>, and <code>sed</code>]: Article going over the basic uses of <code>grep</code>, <code>awk</code>, and <code>sed</code> and how to think through their use cases.

[https://www.tldp.org/LDP/abs/html/ Advanced Bash-Scripting Guide]

== Graphical Sessions (VNC) ==
See [[VNC]]

== File Permissions and ACL ==
See [[File_Permissions_Basics_and_ACL|File Permissions Basics and ACL]]

NAS

2022-11-09T16:32:33Z

Jrwrigh: Add mention of using MPICC_CC

[[ Category: Compute Facilities]]
Wiki for information related to the '''NASA Advanced Supercomputing''' ('''NAS''') facility.

==Overview==
{| class="wikitable" style="text-align:center;"
! Key
! Value
! Notes
|-
| style="font-weight:bold; text-align:left;" | Machines
| Pleiades
| Compute
|-
|
| Lou
| Storage and Analysis
|-
|
| Electra
| Compute
|-
|
| Endeavour
| Compute
|-
|
| Merope
| Compute
|-
|
| Aitken
| Compute
|-
| style="font-weight:bold; text-align:left;" | Job Submission System
| [https://www.nas.nasa.gov/hecc/support/kb/portable-batch-system-(pbs)-overview_126.html PBS]
|
|-
| style="font-weight:bold; text-align:left;" | Facility Documentation
| [https://www.nas.nasa.gov/hecc/support/kb/ Support Knowledgebase]
|
|}

== How-To's ==

=== How-To's in Separate Wiki's ===

* [[Setting_Default_File_Permissions#NAS|Setting Default File Permissions]]

=== Backup Data from Scratch Directories===

This is done simply by copying data from the <code>/nobackup/$USER</code> directories to your home directory on Lou (<code>lfe</code>). The <code>/nobackup/$USER</code> directories are mounted onto <code>lfe</code>, so transfers should be done on <code>lfe</code>.

It is recommended to mirror the directory structure of your <code>/nobackup/$USER</code> directory on <code>lfe</code> to allow for the data to be easily recovered back to it's original state. This is especially important if you use symlinks (as they are path dependent and will break if either the source file or the symlink itself are not in the correct location).

This can be done with <code>scp</code>, but it is recommended to use NASA's in-house utility <code>shiftc</code>. <code>shiftc</code> will automatically perform parallel file transfers, data integrity checks and repairs, and syncing features similar to <code>rsync</code>.

'''Commands:'''

jrwrigh7@lfe7: shiftc -r -d --sync /nobackup/jrwrigh7/models/STGFlatPlate/STFM_Tet_dz4-10_dx15 .

This will copy the directory <code>STFM_Tet_dz4-10_dx15</code> to the current location (<code>.</code>). The flags do as follows

* <code>-r</code>: Recursively copy files from destination
* <code>-d</code>: Create required directories that don't already exist. Equivalent of the <code>-p</code> flag for <code>mkdir</code>
* <code>--sync</code>: Only copy over "new" files, where "new" are any changes to the modification time or file size.
** If a file exists on destination (<code>.</code>), but not source (<code>STFM_Tet_dz4-10_dx15</code>), it will not be copied back to source nor will it be deleted to match the state of source.

Once this command is submitted, the transfer process will be backgrounded. Progress can be viewed by running <code>shiftc --monitor</code>. Additionally, you will recieve an email with the transfer job is completed.

jrwrigh7@lfe7: shiftc --stop --id [shiftc job ID]

This will stop the given shiftc job. The <code>[shiftc job ID]</code> is the same number that appears beside the output of <code>shiftc --monitor</code>.

More documentation for <code>shiftc</code> can be found in its man page (<code>man shiftc</code>) and on [https://www.nas.nasa.gov/hecc/support/kb/shift-transfer-tool-overview_300.html NAS's documentation website].

=== Control MPI Rank Placement ===
==== Rank 1 Solo Node ====
To make the rank 1 MPI process take a node on it's own, put this in the PBS directives:

#PBS -l select=1:mpiprocs=1:model=sky_ele+1:mpiprocs=40:model=sky_ele

This will request 2 nodes: One will have the rank 1 process all by itself, and the other will have 40 MPI Processes (for all 40 CPU cores available on <code>sky_ele</code> nodes).

====Distribute Non-First Rank MPI Processes====
For controlling the placement of non-first rank MPI processes, use the <code>mbind.x</code> utility.

For example, if we have requested 4 nodes and want 10 MPI processes per node, the <code>mpiexec</code> command needs to be modified to the following:

mpiexec -np 40 /u/scicon/tools/bin/mbind.x -n10 [executable]

Note that <code>mbind.x</code> is also socket aware, so it will distribute nodes evenly between nodes ''and'' between CPU's in each node (NAS nodes have 2 CPU's per node).
For more information on <code>mbind.x</code>, see it's help flag (<code>mbind.x -help</code>) or [https://www.nas.nasa.gov/hecc/support/kb/using-the-mbind-tool-for-pinning_288.html NAS's documentation website].

=== Common commands ===

* <code>node_stats.sh</code>: Displays how many nodes are available or actively running jobs
* <code>tracejobssh</code>: Helps to answer "Why isn't my job running?". Part of the [https://github.com/PHASTA/utilities git repo].

=== See Priority "Score" in Queue ===

To see what your priority "score" in PBS is use the <code>qstat -W o=+pri</code> to add the "Priority" column to the output of <code>qstat</code>.

==== Priority Scoring (as of 2021-01-22) ====

* Job priority score grows by 1 every 12 hours
* We are capped at a max score of 20 per job
** Note that other users/groups using NAS may start with higher priority and grow higher than 20
** Result is that it's quite difficult to get large jobs running
* If you don't have any jobs running, you get an addition +10 to the score
** This score bump is removed as soon as you have a running job

=== Compiling ===
* Generally will want use <code>module load hpe-mpi/mpt comp-intel</code> for compiling
* Sometimes, <code>mpi{cc,cxx,f90}</code> will not pick the Intel compilers by default. You can check this by running <code>mpi{cc,cxx,f90} --version</code> to verify the compiler it links to.
** To fix this, you can set <code>export MPICC_CC=icc MPICXX_CXX=icpc MPIF90_F90=ifort</code> to force it to use the Intel compilers

File Permissions Basics and ACL

2022-10-24T14:37:54Z

Jrwrigh: /* "Mode" Parameter */ Syntax fix

'''Access-control Lists (ACL)''' are how POSIX-based file systems control '''file permissions'''. This article forms a primer to understanding how exactly the actions in [[Setting Default File Permissions]] work and how to debug them.

In short, ACL helps to control which users can read, write, or execute certain file objects, which are defined as anything in the file system (regular files, directories, symlinks, etc.). Note that ACL does ''not'' have complete authority on ''creating'' permissions for new file objects.
In analogy, ACL is akin to the guest list at a club. Continuing the analogy, the OS kernel is the bouncer of the club, determining who can get in to which section of the club based on the ACL.

Much of the information presented here is written out in the [https://linux.die.net/man/5/acl ACL manpage]. This wiki hopefully serves as a primer for that manpage, as the terms it uses are different than the ones that are commonly presented and used when dealing with normal file permissions. I recommend reading the manpage if you're having to debug a permissions issue. It's not too long.

== Basics of Unix/POSIX File Permissions ==

=== What are they? ===

* All files and directories have permissions assigned to them via ACL entries
* There are three different "levels" of file permissions in a standard POSIX file system: read (<code>r</code>), write (<code>w</code>), and execute (<code>x</code>).
** Read allows viewing the contents of the file/directory, and copying the files
** Write allows rewriting and deleting files. For a directory with write permissions, it also allows creation of subdirectories and creation of new files
** Execute allows files to be executed directly.
*** Note that for script files (such as <code>bash</code> or <code>python</code>), they can still be run by passing the file to it's interpreter if the file is readable (ie. <code>bash non_executableScript.sh</code> is still possible if <code>non_executableScript.sh</code> has <code>rw-</code> permissions).
* ACL entries apply the three different "levels" of file permission onto 6 possible tags. From the ACL manpage, they are:

ACL_USER_OBJ The ACL_USER_OBJ entry denotes access rights for the file owner.

ACL_USER ACL_USER entries denote access rights for users identified by the entry's qualifier.

ACL_GROUP_OBJ The ACL_GROUP_OBJ entry denotes access rights for the file group.

ACL_GROUP ACL_GROUP entries denote access rights for groups identified by the entry's qualifier.

ACL_MASK The ACL_MASK entry denotes the maximum access rights that can be granted by entries of type ACL_USER, ACL_GROUP_OBJ, or ACL_GROUP.

ACL_OTHER The ACL_OTHER entry denotes access rights for processes that do not match any other entry in the ACL.

:*<code>ACL_USER_OBJ</code>, <code>ACL_GROUP_OBJ</code>, and <code>ACL_OTHER</code> correspond to the standard three user categories: file owner, file group, and "other". These three categories are the ones listed by <code>ls -l</code>.
:**The file owner and file group are set to the user that originally created the file and their respective user group (though this can be changed using <code>chown</code>).
:*"Others" simply refers to all other users that are not the file owner and are not members of the file group.
:*The other three tags, <code>ACL_MASK</code>, <code>ACL_GROUP</code>, and <code>ACL_USER</code>, are only used for [[File_Permissions_Basics_and_ACL#Custom Permissions|custom ACL entries]] that are set by a user.
:** Additionally, <code>ACL_MASK</code> is only required if <code>ACL_GROUP</code> or <code>ACL_USER</code> entries are utilized.

* The file permissions for <code>ACL_USER_OBJ</code>, <code>ACL_GROUP_OBJ</code>, and <code>ACL_OTHER</code> together form the file <code>mode</code> parameter
** This parameter is used by programs to set the permissions of files they create. This is discussed further in [[File_Permissions_Basics_and_ACL#Custom Permissions|the Custom Permissions section]].

==== Viewing Them ====

===== Using ls =====
The simplest and most common way of viewing the basic permissions of a file is by using <code>ls</code>. As an example, if you run <code>ls -l</code> on a directory, you might see:

drwxr-x---+ 2 jrwrigh7 a1983 4.0K 2020-07-04 08:09 test2
-rw-r-x---+ 1 jrwrigh7 a1983 38 2020-07-02 12:38 test2file
lrwxrwxrwx 1 jrwrigh7 a1983 9 2020-07-04 08:40 test2fileLink -> test2file

The first block (<code>-rw-r-x---+</code>) shows the permissions of the file for the three primary ACL tags (<code>ACL_USER_OBJ</code>, <code>ACL_GROUP_OBJ</code>, and <code>ACL_OTHER</code>). (described below). The file owner is shown as <code>jrwrigh7</code> and the file group is <code>a1983</code>. These correspond to <code>ACL_USER_OBJ</code> and <code>ACL_GROUP_OBJ</code>.

'''Permissions Block:'''
* First character displays what kind of file it is, be it a link (<code>l</code>), directory (<code>d</code>), regular file (<code>-</code>), etc.
* The next 9 characters show the permissions for the <code>ACL_USER_OBJ</code>, <code>ACL_GROUP_OBJ</code>, and <code>ACL_OTHER</code>
** So for <code>test2file</code> in the above output:
*** File Owner: <code>rw-</code>
*** File Group: <code>r-x</code>
*** Others: <code>---</code>
* The last character is optional. A <code>+</code> means that there are other permission rules not displayed. This is where [[Setting_Default_File_Permissions#Setting ACL Rules|ACL rules]] come into play.

See [https://www.gnu.org/software/coreutils/manual/html_node/What-information-is-listed.html#What-information-is-listed the ls coreutils manual] for more information on the 'long' format for <code>ls</code>.

===== Using getfacl =====

You can also use <code>getfacl</code> to get a more detailed look at the permissions of a file or directory. Simply running <code>getfacl</code> with a file object as it's argument will show the permissions for that file object:

$ getfacl test2file
# file: test2file
# owner: jrwrigh7
# group: a1983
user::rw-
group::r-x
group:a1983:r-x
mask::r-x
other::---

This argument will show all permissions for a file object, including custom ones. <code>user::</code>, <code>group::</code>, and <code>other::</code> refer to the permissions set for the <code>ACL_USER_OBJ</code>, <code>ACL_GROUP_OBJ</code>, and <code>ACL_OTHER</code>.

To set more specific permissions, a group or user can be inserted between the colons. This can be seen in the line <code>group:a1983:</code> where the members of the group have <code>r-x</code> permissions to the file. These permissions fall under the <code>ACL_USER</code> and <code>ACL_GROUP</code> tags.

If default ACL entries are used, they will also be displayed using <code>getfacl</code>. An example is presented below for the directory <code>test</code> (note that default ACL entries can ''only'' be applied to directories):

$ getfacl test
# file: test
# owner: jrwrigh7
# group: a1983
user::rwx
group::r-x
group:a1983:r-x
mask::r-x
other::---
default:user::rwx
default:group::r-x
default:group:a1983:r-x
default:mask::r-x
default:other::---

=== Permissions as Octal Numbers ===
The file <code>mode</code> is often conveyed in the form of three octal numbers (ie. base 8 numbers). It is very similar to how PHASTA handles specifying boundary conditions using bitwise logic.
The first bit handles read, the second bit handles write, and the third handles execute.

{| class="wikitable" style="text-align:center;"
|-
! style="text-align:left;" | Permissions
! Execute bit
! Write Bit
! Read Bit
! Octal Number equivalent
|-
| style="text-align:left;" | <code>---</code>
| 0
| 0
| 0
| 0
|-
| style="text-align:left;" | <code>--x</code>
| 1
| 0
| 0
| 1
|-
| style="text-align:left;" | <code>-w-</code>
| 0
| 1
| 0
| 2
|-
| style="text-align:left;" | <code>-wx</code>
| 1
| 1
| 0
| 3
|-
| style="text-align:left;" | <code>r--</code>
| 0
| 0
| 1
| 4
|-
| style="text-align:left;" | <code>r-x</code>
| 1
| 0
| 1
| 5
|-
| style="text-align:left;" | <code>rw-</code>
| 0
| 1
| 1
| 6
|-
| style="text-align:left;" | <code>rwx</code>
| 1
| 1
| 1
| 7
|}

Using <code>test2file</code> as the example permissions block(<code>rw-r-x---</code>), it is stored as <code>110 101 000</code>. When translated into decimal, that equals <code>3 5 0</code>.

Another way to think about it is that the octal number equivalent of a permission set = 1*x + 2*w + 4*r, where r, w, and x equal 1 or 0 depending on whether they're set or not.

== Custom Permissions ==

The other two tags, <code>ACL_GROUP</code> and <code>ACL_USER</code>, are for more custom permissions and are not set by default on a blank system.
Different sets of permissions can be set on a per user and per group basis.
For more information on complex interactions (ie. if a user has permissions set and is a member of a group that has its own custom permissions), I recommend seeing the ACL manpage section on the [https://linux.die.net/man/5/acl Access Check Algorithm]

=== What determines the permissions for a new file? ===

When a new file object is created, a new set of ACL entries must be created for that file object. There are three sources that determine what ACL entries will be set; <code>umask</code>, the "mode" parameter used by the program creating the file, and ACL entries of the parent directory of the file object being created.

==== umask ====

<code>umask</code> is a function that contains file permission settings for any file created by a user and it is unique to the user. The user can change it at anytime using the <code>umask</code> command. To see what it is set to, simply run <code>umask</code> with no arguments. The output is usually in octal notation, and represents the bits that will be ''blocked'', not the bits that will be set. So an output of <code>022</code> will allow any user permission bits to be set, but will not allow for the executable bit to be set for user and group settings.

==== "Mode" Parameter ====

When a new file is created, the <code>syscall</code> <code>open()</code> (among others) is used and a file mode parameter must be chosen by that program. For example, <code>touch</code> will automatically apply the mode <code>666</code> to the file, which will make the file owner, file group, and "others" all have <code>rw-</code> permissions.

==== ACL Defaults ====

ACL default entries are set for a directory and are used for files and subdirectories created inside that directory. Note that ACL default entries can ''only'' be set for directories.

=== How are new file permissions set? ===
This is simply a paraphrasing/rewording of the "Object Creation and Default ACLs" section of the [https://linux.die.net/man/5/acl ACL manpage].

When creating a file/subdirectory in a parent directory:

* If the parent directory '''does have default ACL rules''', only the ACL default entries of the parent directory and "mode" parameter are used:
** The new file/subdirectory first inherits the ACL default entries of its parent directory as its normal ACL entries
** The new file/subdirectory has its ACL entries adjusted such that no permissions exceed the "mode" parameter.
*** The <code>ACL_USER_OBJ</code> and <code>ACL_OTHER</code> are changed directly
*** The <code>ACL_GROUP</code>, <code>ACL_GROUP_OBJ</code>, and <code>ACL_USER</code> are changed ''through'' adjusting the <code>ACL_MASK</code>
** Additionally, if a new subdirectory is created, it will inherit its parents default ACL rules. Note that a file ''cannot'' have default ACL rules set.

* If the parent directory '''does not have default ACL rules''', only <code>umask</code> and "mode" parameter are used:
** The new file/subdirectory's <code>ACL_USER_OBJ</code>, <code>ACL_GROUP_OBJ</code>, and <code>ACL_OTHER</code> are set based on the permissions set by <code>umask</code> via a bit wise NAND
** The new file/subdirectory's are then adjusted to limit permissions to be no looser than the "mode" parameter

Note that this all means that if you have a default ACL rule that gives execute permissions to a group, the group will ''not'' have execute permissions by default ''unless'' the "mode" parameter also has execute permissions. Most compilers will use the execute permission bit for the "mode" parameter of it's executable, but other files will not.

NAS

2022-10-12T22:35:25Z

Jrwrigh: Add Aitken to machine list

ALCF/Archiving Data at ALCF

2022-10-04T15:44:17Z

Jrwrigh: Add further documentation section

[[ALCF]]'s High Performance Storage System (HPSS) is a robotic tape drive system used for large amount of archival data storage that will not be often accessed. The system has two interfaces listed in [https://www.alcf.anl.gov/support/user-guides/data-management/filesystem-and-storage/hpss/index.html ALCF documentation] that can be used, <code>hsi</code> and <code>htar</code>. In this wiki, the <code>hsi</code> interface will be focused on.

== HSI Basics ==

<code>hsi</code> is a utility to interface with the HPSS system. It looks and operates much like the typical bash command lines that we are used to, but with some added complexities. When you enter <code>hsi</code>, the system will place you into your home HPSS space at <code>/home/username</code>. <code>hsi</code> keeps track of both your location in this HPSS space and also your location in the "local" system that you are running <code>hsi</code> from. The <code>hsi</code> system will automatically set the "local" directory location to be the location that you entered the utility from.

Navigation through HPSS operates the same as a normal command line, with <code>ls</code> <code>cd</code>, and <code>mkdir</code> among others being valid commands for navigation through HPSS. If you need to navigate though the "local" directories though, this can still be done by appending an "l" to the front of these standard commands (i.e. <code>lls</code> <code>lcd</code>, and <code>lmkdir</code>). This can be useful if you entered <code>hsi</code> at the incorrect point or wish to archive data in multiple locations.

It should be noted that <code>hsi</code> does ''not'' support tab-complete or up-arrowing for past commands. It is recommended that you enter the utility with a defined plan in order to reduce the amount of annoyance that the lack of these luxuries can cause.

== Archiving of Data ==

Once the destination directory for the data has been created and/or navigated to, there are a few options and considerations to actually archive the data. These are most notably:

* <code>put</code>
* <code>cput</code>

<code>put</code> is the most basic archiving tool, and will overwrite any versions of the files being archived already on HPSS. <code>cput</code> is a conditional version of <code>put</code> that will only overwrite files if there is a newer version "locally" compared to the file already on HPSS. This makes <code>cput</code> the tool of choice for updating partially archived datasets, but due to its otherwise similar functionality to <code>put</code>, it is also the recommended default command to use.

Both <code>put</code> and <code>cput</code> have similar syntax, and the following will cover both, but <code>cput</code> will be used as an example. It is assumed that the <code>hsi</code> command has already been run to enter the <code>hsi</code> utility before attempting the following.

Simple usage to store a single file is:

<code>cput <filename></code>

To change the name of a file as it is archived:

<code>cput <filename> : <newFilename></code>

Whole directories can be stored using:

<code>cput -R <dirName></code>

If you with to keep the parent directory intact, or with:

<code>cput -R "*"</code>

If you simply want to move the contents of a directory and everything beneath it. Simply be mindful of your "local" directory location when choosing between these options.

If you need to retrieve data from tape and put it back on the "local" system, the <code>get</code> and <code>cget</code> commands act in the same way as <code>put</code> and <code>cput</code> but in reverse.

== Troubleshooting ==
If you are a first time user of HPSS, you will likely get an error regarding a key file. This is something that must be taken care of by ALCF support (support@alcf.anl.gov). Simply email them with your ALCF username and state that you need access set up for HPSS.

== Further Documentation ==
While standard use cases of <code>hsi</code> is covered above, more through documentation is [https://docs.nersc.gov/filesystems/archive/ available from NERSC].

There is also a [https://www.hpss-collaboration.org/documents/HSI_8.3_Reference_Manual.pdf pdf reference manual] (backup saved here: [[File:HSI 8.3 Reference Manual.pdf]]) that goes into more detail. Note that it is for version 8.3, while ALCF is currently (as of 2022-10-04) running 7.4.

[[Category:Compute Facilities]]

File:HSI 8.3 Reference Manual.pdf

2022-10-04T15:39:42Z

Jrwrigh: HSI reference manual for ALCF archiving. Found at https://www.hpss-collaboration.org/documents/HSI_8.3_Reference_Manual.pdf

HSI reference manual for ALCF archiving. Found at https://www.hpss-collaboration.org/documents/HSI_8.3_Reference_Manual.pdf

ALCF/Archiving Data at ALCF

2022-10-04T15:27:16Z

Jrwrigh: Update alcf documentation link

[[ALCF]]'s High Performance Storage System (HPSS) is a robotic tape drive system used for large amount of archival data storage that will not be often accessed. The system has two interfaces listed in [https://www.alcf.anl.gov/support/user-guides/data-management/filesystem-and-storage/hpss/index.html ALCF documentation] that can be used, <code>hsi</code> and <code>htar</code>. In this wiki, the <code>hsi</code> interface will be focused on.

== HSI Basics ==

<code>hsi</code> is a utility to interface with the HPSS system. It looks and operates much like the typical bash command lines that we are used to, but with some added complexities. When you enter <code>hsi</code>, the system will place you into your home HPSS space at <code>/home/username</code>. <code>hsi</code> keeps track of both your location in this HPSS space and also your location in the "local" system that you are running <code>hsi</code> from. The <code>hsi</code> system will automatically set the "local" directory location to be the location that you entered the utility from.

Navigation through HPSS operates the same as a normal command line, with <code>ls</code> <code>cd</code>, and <code>mkdir</code> among others being valid commands for navigation through HPSS. If you need to navigate though the "local" directories though, this can still be done by appending an "l" to the front of these standard commands (i.e. <code>lls</code> <code>lcd</code>, and <code>lmkdir</code>). This can be useful if you entered <code>hsi</code> at the incorrect point or wish to archive data in multiple locations.

It should be noted that <code>hsi</code> does ''not'' support tab-complete or up-arrowing for past commands. It is recommended that you enter the utility with a defined plan in order to reduce the amount of annoyance that the lack of these luxuries can cause.

While standard use cases of <code>hsi</code> will be covered below, more documentation (that is more thorough and helpful than the ALCF documentation is available here: [https://docs.nersc.gov/filesystems/archive/].

== Archiving of Data ==

Once the destination directory for the data has been created and/or navigated to, there are a few options and considerations to actually archive the data. These are most notably:

* <code>put</code>
* <code>cput</code>

<code>put</code> is the most basic archiving tool, and will overwrite any versions of the files being archived already on HPSS. <code>cput</code> is a conditional version of <code>put</code> that will only overwrite files if there is a newer version "locally" compared to the file already on HPSS. This makes <code>cput</code> the tool of choice for updating partially archived datasets, but due to its otherwise similar functionality to <code>put</code>, it is also the recommended default command to use.

Both <code>put</code> and <code>cput</code> have similar syntax, and the following will cover both, but <code>cput</code> will be used as an example. It is assumed that the <code>hsi</code> command has already been run to enter the <code>hsi</code> utility before attempting the following.

Simple usage to store a single file is:

<code>cput <filename></code>

To change the name of a file as it is archived:

<code>cput <filename> : <newFilename></code>

Whole directories can be stored using:

<code>cput -R <dirName></code>

If you with to keep the parent directory intact, or with:

<code>cput -R "*"</code>

If you simply want to move the contents of a directory and everything beneath it. Simply be mindful of your "local" directory location when choosing between these options.

If you need to retrieve data from tape and put it back on the "local" system, the <code>get</code> and <code>cget</code> commands act in the same way as <code>put</code> and <code>cput</code> but in reverse.

== Troubleshooting ==
If you are a first time user of HPSS, you will likely get an error regarding a key file. This is something that must be taken care of by ALCF support (support@alcf.anl.gov). Simply email them with your ALCF username and state that you need access set up for HPSS.

[[Category:Compute Facilities]]

ALCF/Archiving Data at ALCF

2022-09-29T18:13:05Z

Jrwrigh: /* HSI Basics */ Fix Typo

[[ALCF]]'s High Performance Storage System (HPSS) is a robotic tape drive system used for large amount of archival data storage that will not be often accessed. The system has two interfaces listed in ALCF documentation ([https://www.alcf.anl.gov/support-center/theta/using-hpss-theta]) that can be used, <code>hsi</code> and <code>htar</code>. In this wiki, the <code>hsi</code> interface will be focused on.

== HSI Basics ==

<code>hsi</code> is a utility to interface with the HPSS system. It looks and operates much like the typical bash command lines that we are used to, but with some added complexities. When you enter <code>hsi</code>, the system will place you into your home HPSS space at <code>/home/username</code>. <code>hsi</code> keeps track of both your location in this HPSS space and also your location in the "local" system that you are running <code>hsi</code> from. The <code>hsi</code> system will automatically set the "local" directory location to be the location that you entered the utility from.

Navigation through HPSS operates the same as a normal command line, with <code>ls</code> <code>cd</code>, and <code>mkdir</code> among others being valid commands for navigation through HPSS. If you need to navigate though the "local" directories though, this can still be done by appending an "l" to the front of these standard commands (i.e. <code>lls</code> <code>lcd</code>, and <code>lmkdir</code>). This can be useful if you entered <code>hsi</code> at the incorrect point or wish to archive data in multiple locations.

It should be noted that <code>hsi</code> does ''not'' support tab-complete or up-arrowing for past commands. It is recommended that you enter the utility with a defined plan in order to reduce the amount of annoyance that the lack of these luxuries can cause.

While standard use cases of <code>hsi</code> will be covered below, more documentation (that is more thorough and helpful than the ALCF documentation is available here: [https://docs.nersc.gov/filesystems/archive/].

== Archiving of Data ==

Once the destination directory for the data has been created and/or navigated to, there are a few options and considerations to actually archive the data. These are most notably:

* <code>put</code>
* <code>cput</code>

<code>put</code> is the most basic archiving tool, and will overwrite any versions of the files being archived already on HPSS. <code>cput</code> is a conditional version of <code>put</code> that will only overwrite files if there is a newer version "locally" compared to the file already on HPSS. This makes <code>cput</code> the tool of choice for updating partially archived datasets, but due to its otherwise similar functionality to <code>put</code>, it is also the recommended default command to use.

Both <code>put</code> and <code>cput</code> have similar syntax, and the following will cover both, but <code>cput</code> will be used as an example. It is assumed that the <code>hsi</code> command has already been run to enter the <code>hsi</code> utility before attempting the following.

Simple usage to store a single file is:

<code>cput <filename></code>

To change the name of a file as it is archived:

<code>cput <filename> : <newFilename></code>

Whole directories can be stored using:

<code>cput -R <dirName></code>

If you with to keep the parent directory intact, or with:

<code>cput -R "*"</code>

If you simply want to move the contents of a directory and everything beneath it. Simply be mindful of your "local" directory location when choosing between these options.

If you need to retrieve data from tape and put it back on the "local" system, the <code>get</code> and <code>cget</code> commands act in the same way as <code>put</code> and <code>cput</code> but in reverse.

== Troubleshooting ==
If you are a first time user of HPSS, you will likely get an error regarding a key file. This is something that must be taken care of by ALCF support (support@alcf.anl.gov). Simply email them with your ALCF username and state that you need access set up for HPSS.

[[Category:Compute Facilities]]

TotalView

2022-09-18T18:51:56Z

Jrwrigh:

We currently have an evaluation license of the TotalView graphical/parallel debugger. Please give it a try and let Ben know if you have any problems

==Running==
soft add +totalview-8.13.0
mpirun -tv -np 1 your_program

==ReplayEngine/Reverse Debugging==

mpirun -mca mpool_rdma_rcache_size_limit 1 -x IBV_FORK_SAFE=1 -x LD_PRELOAD=/usr/local/toolworks/totalview.8.13.0-0/linux-x86-64/lib/undodb_infiniband_preload_x64.so -tv -np 1 your_program

(see the TotalView user's guide for more information)

==Documentation==
http://www.roguewave.com/support/product-documentation/totalview.aspx

[[Category:Software Engineering]] [[Category:Software]]

The On Ramp

2022-09-18T18:51:04Z

Jrwrigh: /* Level 1 */

Welcome to the '''PHASTA On Ramp'''! This page is meant to organize information for getting up to speed with the overall PHASTA workflow, as well as help you up your game as a PHASTA developer.

== Level 0 ==
Ready to get your feet wet? Maybe you need to brush up on some fundamental skills. Level 0 provides tutorials and information on basic developer skills you will need to work with PHASTA.
* [[PHASTA_Group_Machines|Logging in]]
* [[VNC|Setting up your VNC]]
* [[UNIX|Getting started with Linux]]
* [[Git | Getting Git]]
* [[Vim|Getting comfortable with Vim]]
* [[Fortran|New to Fortran?]]
* [[Making A New Wiki | Editing this Wiki]]

== Level 1 ==

[[File:Picture5.png]]

So you think you know how to <code>grep</code> for keywords and <code>mkdir</code> some folders? Now you're probably ready to jump into PHASTA.

* [[Level 1 Model/Mesh| Model/Mesh Workflow]]
* [[Level 1 Partition| Partition Workflow]]
* [[Level 1 Solve| Solve Workflow]]
* [[Level 1 Post-Process| Post-Process Workflow]]

See the [[The_On_Ramp/Level_1|Level 1 base page]] as well.

== Level 2 ==
Now that you've plotted some "Colorful Fluid Dynamics" (CFD) you're probably interested in digging into the guts of PHASTA, and getting it running on some more interesting hardware.
* [https://github.com/PHASTA/phasta Building PHASTA from scratch] (Follow the README at the bottom of the page)
* [https://github.com/SCOREC/core/wiki/General-Build-instructions#Manual_Install Building Chef and other SCOREC core tools]
* [[ Building ParaView 5.8.1 and Shoreline Plug-ins on viz003 | Building ParaView with Shoreline Plug-ins ]]

== Level 3 ==
ParaView isn't cutting it for your research? Now its time to start writing your own analysis code. How to do custom pre and post processing is described here, along with more complex ways of using PHASTA for interesting CFD simulations
* [[Synthetic Turbulence Inflow Generator | Synthetic Turbulence Generator]]
* [[VTKpytools| Python + VTK using vtkpytools]]

== Level 4 ==
You're probably close to defending at this point, so before you go get that sweet industry job, you should drop some of your lofty knowledge here!

The On Ramp/Level 1/Post-Process

2022-09-18T18:42:50Z

Jrwrigh: typo fix for category

In this section we will learn how to set-up the Visualization file which [[ParaView]] (a flow visualization software) reads in, and how to utilize Paraview to analyze our flow field solutions generated by [[PHASTA]].

== Visualization File ==
In order for Paraview to know which restart and geombc files to process, we need to give it this information by means of a <code>.pht</code> meta-data file. In the directory where the PHASTA case was run, <code>.../8-1-Chef/Run/</code> for our example, you will need to create/copy a meta-data file with a <code>.pht</code> file extension. Common practice is to name this file <code>flow.pht</code> and to only include the file in your <code>Run</code> directory when you are ready to launch and work in Paraview. You can copy over an example <code>.pht</code> file from the tutorials folder called <code>flow.pht</code>.

'''A detailed explanation of the variables inside of the <code>.pht</code> file is provided in this [https://fluid.colorado.edu/tutorials/tutorialVideos/ParaviewWithMark_meta_data_description.mp4 video].'''

The following is an example <code>flow.pht</code> file for our <code>8-1-Chef</code> case, where we ran 10 time steps in PHASTA, saved every 5th time step, and we want to visualize the 5th and 10th time step in Paraview.

<?xml version="1.0" ?>
<PhastaMetaFile number_of_pieces="8">
<GeometryFileNamePattern pattern="8-procs_case/geombc.dat.%d" ''#8-procs_case here matches the folder name where our geombc files are located.''
has_piece_entry="1"
has_time_entry="0"/>
<FieldFileNamePattern pattern="8-procs_case/restart.%d.%d" ''#8-procs_case here matches the folder name where our restart files are located.''
has_piece_entry="1"
has_time_entry="1"/>
<TimeSteps number_of_steps="2" ''#Needs to be 2 as we want to visualize 2 time steps.''
auto_generate_indices="1"
start_index="5" ''#Index of the first time step we want to visualize. Make sure this time step is available by checking the files in''
''8-procs_case''
increment_index_by="5" ''#5 + 5 = 10. If we had more time steps, we could visualize the 15th, 20th, 25th, 30th, etc.''
start_value="0."
increment_value_by="0.5">
</TimeSteps>
<Fields number_of_fields="3">
<Field phasta_field_tag="solution"
paraview_field_tag="velocity"
start_index_in_phasta_array="1"
number_of_components="3"
data_dependency="0"
data_type="double"/>
<Field phasta_field_tag="solution"
paraview_field_tag="pressure"
start_index_in_phasta_array="0"
number_of_components="1"
data_dependency="0"
data_type="double"/>
<Field phasta_field_tag="solution"
paraview_field_tag="eddy visocity"
start_index_in_phasta_array="5"
number_of_components="1"
data_dependency="0"
data_type="double"/>
</Fields>
</PhastaMetaFile>

==Visualizing Fields and Computing Quantities==

As always, set the environment on viz003 (if you have not done so already) by using the soft adds located in:

<code>more ~kjansen/soft-core.sh</code>

To open the visualization tool Paraview, run the command:

vglrun paraview

It is common practice to be in the directory where your <code>.pht</code> file is located when you run the <code>vglrun paraview</code> command. This sets the working directory in Paraview to the one which contains your <code>.pht</code> file, and you can then quickly open that file without having to search for it in the "Open File" Paraview GUI.

A tutorial on how to navigate the GUI to visualize solution fields as well as compute other solution fields is given in this [https://fluid.colorado.edu/tutorials/tutorialVideos/ParaviewWithMark_viz_fields.mp4 video].

'''Note:''' You can also open a specific Paraview build by setting the path to the executable file. For example, a Paraview v5.7.0 executable file was built in the following directory: <code>/users/jeffhadley/Builds/build-paraview-v5.7.0/bin/</code>. To run this specific build of paraview, you would run the command:

<code>vglrun /users/jeffhadley/Builds/build-paraview-v5.7.0/bin/paraview</code>

[[Category:Post-processing]] [[Category:Paraview]]

The On Ramp/Level 1/Post-Process

2022-09-18T18:42:14Z

Jrwrigh: Misc edits

The On Ramp/Level 1/Post-Process

2022-09-18T18:40:28Z

Jrwrigh: Jrwrigh moved page Level 1 Post-Process to The On Ramp/Level 1/Post-Process: Move to a subpage organization

In this section we will learn how to set-up the Visualization file which Paraview (Our flow visualization software) reads in, and how to utilise Paraview to analyze our flow field solutions generated by PHASTA.

== Visualization File ==
In order for Paraview to know which restart and geombc files to process, we need to give it this information by means of a <code>.pht</code> meta-data file. In the directory where the PHASTA case was run, <code>.../8-1-Chef/Run/</code> for our example, you will need to create/copy a meta-data file with a <code>.pht</code> file extension. Common practice is to name this file <code>flow.pht</code> and to only include the file in your <code>Run</code> directory when you are ready to launch and work in Paraview. You can copy over an example <code>.pht</code> file from the tutorials folder called <code>flow.pht</code>.

'''A detailed explanation of the variables inside of the <code>.pht</code> file is provided in this [https://fluid.colorado.edu/tutorials/tutorialVideos/ParaviewWithMark_meta_data_description.mp4 video].'''

The following is an example <code>flow.pht</code> file for our <code>8-1-Chef</code> case, where we ran 10 time steps in PHASTA, saved every 5th time step, and we want to visualize the 5th and 10th time step in Paraview.

<?xml version="1.0" ?>
<PhastaMetaFile number_of_pieces="8">
<GeometryFileNamePattern pattern="8-procs_case/geombc.dat.%d" ''#8-procs_case here matches the folder name where our geombc files are located.''
has_piece_entry="1"
has_time_entry="0"/>
<FieldFileNamePattern pattern="8-procs_case/restart.%d.%d" ''#8-procs_case here matches the folder name where our restart files are located.''
has_piece_entry="1"
has_time_entry="1"/>
<TimeSteps number_of_steps="2" ''#Needs to be 2 as we want to visualize 2 time steps.''
auto_generate_indices="1"
start_index="5" ''#Index of the first time step we want to visualize. Make sure this time step is available by checking the files in''
''8-procs_case''
increment_index_by="5" ''#5 + 5 = 10. If we had more time steps, we could visualize the 15th, 20th, 25th, 30th, etc.''
start_value="0."
increment_value_by="0.5">
</TimeSteps>
<Fields number_of_fields="3">
<Field phasta_field_tag="solution"
paraview_field_tag="velocity"
start_index_in_phasta_array="1"
number_of_components="3"
data_dependency="0"
data_type="double"/>
<Field phasta_field_tag="solution"
paraview_field_tag="pressure"
start_index_in_phasta_array="0"
number_of_components="1"
data_dependency="0"
data_type="double"/>
<Field phasta_field_tag="solution"
paraview_field_tag="eddy visocity"
start_index_in_phasta_array="5"
number_of_components="1"
data_dependency="0"
data_type="double"/>
</Fields>
</PhastaMetaFile>

==Visualizing Fields and Computing Quantities==

As always, set the environment on viz003 (if you have not done so already) by using the soft adds located in:

<code>more ~kjansen/soft-core.sh</code>

To open the visualization tool Paraview, run the command:

vglrun paraview

It is common practice to be in the directory where your <code>.pht</code> file is located when you run the <code>vglrun paraview</code> command. This sets the working directory in Paraview to the one which contains your <code>.pht</code> file, and you can then quickly open that file without having to search for it in the "Open File" Paraview GUI.

A tutorial on how to navigate the GUI to visualize solution fields as well as compute other solution fields is given in this [https://fluid.colorado.edu/tutorials/tutorialVideos/ParaviewWithMark_viz_fields.mp4 video].

'''Note:''' You can also open a specific Paraview build by setting the path to the executable file. For example, a Paraview v5.7.0 executable file was built in the following directory: <code>/users/jeffhadley/Builds/build-paraview-v5.7.0/bin/</code>. To run this specific build of paraview, you would run the command:

<code>vglrun /users/jeffhadley/Builds/build-paraview-v5.7.0/bin/paraview</code>

Level 1 Post-Process

2022-09-18T18:40:28Z

Jrwrigh: Jrwrigh moved page Level 1 Post-Process to The On Ramp/Level 1/Post-Process: Move to a subpage organization

#REDIRECT [[The On Ramp/Level 1/Post-Process]]

The On Ramp/Level 1/Solve (Incompressible)

2022-09-18T18:40:00Z

Jrwrigh: Jrwrigh moved page Level 1 Solve to The On Ramp/Level 1/Solve: Move to a subpage organization

=== Exporting to PHASTA ===
After the partitioning performed via Chef in the last steps, we now have the problem domain in a form that the PHASTA executable can read. In your 8-1-Chef directory, create a sub-directory named "Run". This will contain all of the simulation data from this case, which we will use in our PHASTA run. Remember that for this on-Ramp tutorial we have partitioned our case of interest to 8 parts. Therefore, we need to create a sub-directory named "8-procs_case" within the /8-1-Chef/Run/ directory. Once you have mkdir'd and cd'd into this new Run/8-procs_case/ subdirectory, make softlinks ("ln -s <path/file*>") to the N=8 restart and geombc "checkpoint" files that where constructed by Chef located in the 8-1-Chef/8-procs_case/ directory. When in the Run/8-procs_case sub-directory, a good command to do this is the following:

ln -s ../../8-procs_case/restart* .
ln -s ../../8-procs_case/geombc* .

Also create a numstart.dat file. This file will specify the time and timestep that the simulation has completed thus far. For our case, we have not yet run the simulation, thus our time and timesteps are 0 0. Use the following command in the "Run/8-procs_case" directory to create the numstart.dat file:

echo 0 0 > numstart.dat

=== Build the executable/specify runtime parameters===
What remains is to determine the version of PHASTA to build and run. Since there are a bunch of researchers working on PHASTA at any given time, there are many branches/versions of the main code.

'''Note:''' You may not have access to the phasta-next repo yet. If that is the case, ask Dr. Jansen to have you added to the repo. In the meanwhile, you can just use the regular phasta repo (by replacing all "phasta-next" with "phasta" in these instructions).

==== Retrieve/build a version of PHASTA code ====
To have your first run of PHASTA, you will need an executable of the PHASTA code. There are many ways to do this, and below is intended to get you a generic executable. The many nuances to this proccess can be found on the Level 2 page [[Compiling_PHASTA_With_CMake|here]]. Navigate to your home directory. Create and enter a directory here named "git-phasta". In a web browser, navigate to the online git repository for phasta-next and select the "clone" or "code" icon. The result should be similar to the following picture, where a pop-up gives a web address:

[[File:GitClone.png]]

Copy this address and within the "git-phasta" directory execute the following command and enter your github credentials:

git clone https://github.com/PHASTA/phasta-next.git

After this is finished there will be a subdirectory created named "phasta-next" that contains the code tree that you wish to build.
Back within the git-phasta directory create another subdirectory named "build_phasta-next". Now we must set the environment so that the compiler has the necessary libraries. If you perform the following command, the listed environment libraries that ken often uses are listed, and are often sufficient:

more ~kjansen/soft-core.sh

At the time that this page is created, the relevant commands to load the needed environment are the following:

soft add +gcc-6.3.0
soft add +openmpi-gnu-1.10.6-gnu49-thread
soft add +simmodeler-6.0-171202

If the user needs additional libraries they can often be found with the following command:

softenv

lets build this code! Navigate into this build_phasta-next subdirectory and create a file named "unpack_buildFiles.sh" with the following content:

#!/bin/bash

Target=Example_Build
mkdir Example_Build
rm -r $Target/*
cd $Target

export PKG_CONFIG_PATH=/users/skinnerr/tools/git-petsc/build_ompi210_gnu63/lib/pkgconfig/

cmake \
-DCMAKE_C_COMPILER=gcc \
-DCMAKE_CXX_COMPILER=g++ \
-DCMAKE_Fortran_COMPILER=gfortran \
-DCMAKE_BUILD_TYPE=Debug \
-DPHASTA_INCOMPRESSIBLE=ON \
-DPHASTA_COMPRESSIBLE=OFF \
-DPHASTA_USE_LESLIB=ON \
-DLESLIB=/users/matthb2/libles1.5/libles-debianjessie-gcc-ompi.a \
-DCASES=/home/mrasquin/develop-phasta/phastaChefTests \
-DPHASTA_TESTING=OFF \
../../phasta-next/

make -j8
echo "Target: $Target"
date

'''Note:''' You can create this file by typing <code>vi unpack_buildFiles.sh</code> into the command line. This will create an empty shell script file named unpack_buildFiles.sh and enter you into the Vim editor mode, where you can practice your recently adopted Vim commands to copy the above script into the file, making sure to save and quit after you're done. Enter the following command to make sure your shell script was saved successfully with all the required text:

more unpack_buildFiles.sh

Now, we must turn the above .sh file into an executable by typing the following command:

chmod +x unpack_buildFiles.sh

Finally, we are ready to run the executable and build the PHASTA code!

./unpack_buildFiles.sh

You can check that your executable has been built by locating:

Example_Build/bin/phastaIC.exe

===== Additional notes =====
If there is a specific branch off of phasta-next that you'd like to build, navigate to phasta-next and use the following command:
git checkout "branchname".
If this is a branch that I will be working on for a while, I tend to alter the build and code directories according to the branch-name. Note that the respective pointer in the "unpack_buildFiles.sh" file (ie the last line) will have to be set accordingly.

=== Create Solver.inp ===
The <code>input.config</code> file in the newly created <code>build_phasta/Example_Build/</code> directory contains all possible options that could be set in the <code>solver.inp</code> file (which we will use for our PHASTA run). Create the <code>solver.inp</code> file in the <code>../PrepAndRun/8-1-Chef/Run/</code> directory, and then specify all of the parameters that you wish to change for your run case. '''NEVER EVER Modify the input.config file itself'''. The rest of the parameters need not be specified. For example my solver.inp looks as follows:

# ibksiz flmpl flmpr itwmod wmodts dmodts fwr taucfct
# PHASTA Version 1.5 Input File
#
# Basic format is
#
# Key Phrase : Acceptable Value (integer, double, logical, or phrase
# list of integers, list of doubles )
#
#
#SOLUTION CONTROL
#{
Equation of State: Incompressible
Number of Timesteps: 10
Time Step Size: 1e-1 # Delt(1)
Turbulence Model: RANS # No-Model # DES97 # DDES iturb=0, RANS =-1 LES=1 #}
#}

#MATERIAL PROPERTIES
#{
Viscosity: 1.50e-5 # fills datmat (2 values REQUIRED if iLset=1)
Density: 1.0 # ditto
Body Force Option: None # ibody=0 => matflag(5,n)
Body Force: 0 0.0 0.0 # (datmat(i,5,n),i=1,nsd)
Thermal Conductivity: 27.6e-1jjj # ditto
Scalar Diffusivity: 27.6e-1 # fills scdiff(1:nsclrS)
#}

OUTPUT CONTROL
{
Number of Timesteps between Restarts: 5 #replaces nout/ntout
Number of SyncIO Files: 0
Print Error Indicators: False
Number of Error Smoothing Iterations: 0 # ierrsmooth
# Print ybar: True
# Print vorticity: True
# Print Wall Fluxes: False
# Print Statistics: False
Number of Force Surfaces: 1
Surface ID's for Force Calculation: 1
# Ranks per core: 4 # for varts only
# Cores per node: 16 # for varts only
}

#LINEAR SOLVER
# Solver Type: GMRES sparse
Solver Type: ACUSIM with P Projection
Number of GMRES Sweeps per Solve: 1 # replaces nGMRES
Number of Krylov Vectors per GMRES Sweep: 200 # replaces Kspace
Scalar 1 Solver Tolerance : 1.0e-4
Tolerance on Momentum Equations: 0.05 # epstol(1)
Tolerance on ACUSIM Pressure Projection: 0.01 # prestol
Number of Solves per Left-hand-side Formation: 1 #nupdat/LHSupd(1)
ACUSIM Verbosity Level : 0 #iverbose
Minimum Number of ACUSIM Iterations per Nonlinear Iteration: 10 # minIters
Maximum Number of ACUSIM Iterations per Nonlinear Iteration: 200 # maxIter
#}

#DISCRETIZATION CONTROL
#{
Time Integration Rule: First Order # 1st Order sets rinf(1) -1
# Time Integration Rule: Second Order # Second Order sets rinf next
# Time Integration Rho Infinity: 0.0 # rinf(1) Only used for 2nd order
Tau Matrix: Diagonal-Shakib #itau=1
Tau Time Constant: 1.0 #dtsfct
Include Viscous Correction in Stabilization: True # if p=1 idiff=1
# if p=2 idiff=2
Lumped Mass Fraction on Left-hand-side: 0.0 # flmpl
Lumped Mass Fraction on Right-hand-side: 0.0 # flmpr
Tau C Scale Factor: 1.0 # taucfct best value depends
Number of Elements Per Block: 64 # switch to >250 if sgi
#}

TURBULENCE MODELING PARAMETERS
{
Turbulence Wall Model Type: None #itwmod=2 RANSorLES
}

#STEP SEQUENCE
#{
Step Construction : 0 1 10 11 0 1 10 11
#}

=== Running the Solver ===
Create the runPHASTA.sh bash script in your <code>8-1-Chef/Run</code> directory. Where you see <pathtoBuildDir> include your path to your build directory where you retrieved the <code>input.config</code> file. The #export line is an example that I have used myself, yours should look similar:

#!/bin/bash

rm $1-procs_case/doubleRun-check

#export PHASTA_CONFIG=/users/jopa6460/git-phasta/build_phasta/Example_Build
export PHASTA_CONFIG=<pathtoBuildDir>

mpirun -np $1 $PHASTA_CONFIG/bin/phastaIC.exe 2>&1 | tee $1.out

Remember to turn the file into an executable as was done for the build script above. Hint: <code>chmod +x</code> command.

NOTE: Before running PHASTA, it is important to check that nobody else is running important jobs on the computer you're connected to. Run <code>top</code> in the command line to check what other processes are running, and if you see other people running large jobs you should ask them if you can run your job as well, or else switch to a different computer.

Now you have everything you need to run your first PHASTA simulation! In the Run directory execute the following line:

./runPHASTA.sh 8

Output of a proper run of PHASTA contains information about 1) the step number of the simulation; 2) the relative residual (convergence relative to this run's initial residual); and 3) various other outputs based on runtime parameters like BC's and IC's. As the simulation continues it will produce successive checkpoint restart files that contain the solution data at that timestep. For our example, these restart files will be located in the <code>.../Run/8-procs_case</code> directory. These restart files and the geombc files are what Paraview post-processes to give us a visualization of flow field solutions.

Now it's time to plot the results in the [[Level 1 Post-Process]] step.

=== A few helpful video tutorials===

[https://fluid.colorado.edu/tutorials/tutorialVideos/CloneFromGithubAndBranch.mov CloneFromGithubAndBranch.mov ]

[[Tutorial_Video_Overviews#CloneFromGithubAndBranch.mov|Video Notes]]

[https://fluid.colorado.edu/tutorials/tutorialVideos/BlancoBuidingPHASTA.mov BlancoBuidingPHASTA.mov ]

[[Tutorial_Video_Overviews#BlancoBuidingPHASTA.mov|Video Notes]]

[https://fluid.colorado.edu/tutorials/tutorialVideos/PHASTA_workflow_RB.mkv PHASTA_workflow_RB.mkv ]

[[Tutorial_Video_Overviews#PHASTA_workflow_RB.mkv|Video Notes]]

[https://fluid.colorado.edu/tutorials/tutorialVideos/RajDemoPrepSolvePost.mov RajDemoPrepSolvePost.mov ]

[[Tutorial_Video_Overviews#RajDemoPrepSolvePost.mov|Video Notes]]

[https://fluid.colorado.edu/tutorials/tutorialVideos/PrepSolvePostBLandC.mp4 PrepSolvePostBLandC.mp4 ]

[[Tutorial_Video_Overviews#PrepSolvePostBLandC.mp4|Video Notes]]

Level 1 Solve

2022-09-18T18:40:00Z

Jrwrigh: Jrwrigh moved page Level 1 Solve to The On Ramp/Level 1/Solve: Move to a subpage organization

#REDIRECT [[The On Ramp/Level 1/Solve]]

The On Ramp/Level 1/Partition

2022-09-18T18:39:27Z

Jrwrigh: /* Chef */ Link to Chef wiki page

== Chef==
[[Chef]] is an open source SCOREC tool that is this group's primary tool to partition a problem domain to many subdomains. This is done to allow any array of compute nodes to each focus on solving the problem in parallel. That is, to divide the problem domain into subdomains to give to different computational workers. When we perform this step, all that is needed for inputs are the, 1) SCOREC mesh constructed from the output of the conversion and meshing steps, and 2) the number of parts (subdomains) to divide the problem up into. A generic workflow for this step is described below and is a good place to start when completing the workflow for the first time. As your studies continue, many aspects of the layout and modifiers will not be consistent with this On Ramp documentation.
=== Creating the serial case via Viz nodes===
Create a <code>1-1-Chef</code> subdirectory inside the <code>PrepAndRun</code> folder. Enter the new folder and copy over the files <code>adapt.inp</code> and <code>runChef.sh</code> from the <code>/projects/tutorials/OnRamp</code> folder. To learn more about the <code>adapt.inp</code> file, enter the command:

more adapt.inp

Pay particular attention to the splitFactor, attributeFileName, modelFileName, meshFileName bz2, and outMeshFileName bz2 specifications. These are often the first nobs that the new user learns to control. When it comes to first creating "checkpoint" files that are needed for the PHASTA executable. splitFactor will tell the Chef executable how many partitions each incoming part needs to be partitioned by. Immediately after the conversion step we want this to be set to 1 because we desire a single "checkpoint" file. attributeFileName will tell Chef where the file that contains the attributes of each geometry entity (region, face, line, and vertex info) is. Likewise, the modelFileName will tell Chef where the file that contains the geometry entity information is. The meshFileName bz2 and outMeshFileName bz2 specifications determine the directory where Chef will look to gather the SCOREC mesh and where to put the partitioned SCOREC mesh. Notice that this pre-written <code>adapt.inp</code> file points the meshFileName bz2 attribute to the mdsMesh_bz2 folder located in the simMeshToMdsMesh directory.
Run the Chef bash script by entering the command <code>./runChef.sh 1</code>. This should output a <code>1-procs_case</code> directory, an <code>mdsMesh_bz2</code> directory, and a <code>chef.log</code> file.

=== Partioning to 8 processes via Viz nodes===

Like the creation of the serial case above, we will make another subdirectory named 8-1-Chef. Copy from the 1-1-Chef directory the adapt.inp as well as the runChef.sh bash script that were used into this 8-1-Chef subdirectory. We will alter the meshFileName bz2 and splitFactor specifications within newly created adapt.inp. Set the specification meshFileName bz2 to "../1-1-Chef/mdsMesh_bz2". Set the splitFactor to 8. Run Chef via the bash script <code>./runChef.sh 8</code>. The executable should be able to read the mesh from the 1-1-Chef directory as well as the geom files as before.

Now that you have successfully partitioned the mesh, it's time to use PHASTA in the [[Level 1 Solve]] step.

=== Further Partitioning to N processes===
By now you may be able to guess how to go about further partitioning your case into more and more partitions. If the user desired the end part count of 32 processes ( call that N), the next steps would mimic the latter with the subdirectory named 32-8-Chef and the splitFactor be set to 4.
There are a few things to consider when partitioning an actual case you care about. It is tempting to make the successive split factors large as to have a fewer amount of repeated partions. However, the splitfactor should not often exceed 8 especially on an unstructured grid. If it does, Chef will have a more difficult time splitting the elements evenly amongst the parts, causing an imbalance in the workloads from process to process, and therefore reducing the effectiveness of the PHASTA run. Secondly, when the mesh is large, the partitioning that is performed on the viznode should not exceed 32 parts. The successive partitionings past this amount should be done on larger machines. Sometimes if a mesh is large enough, the preliminary serial partition should be performed on a fatter node that what is available on the viz nodes (ie CU's summit resource).

=== A few helpful video tutorials===

[https://fluid.colorado.edu/tutorials/tutorialVideos/PHASTA_workflow_RB.mkv PHASTA_workflow_RB.mkv ]

[[Tutorial_Video_Overviews#PHASTA_workflow_RB.mkv|Video Notes]]

[https://fluid.colorado.edu/tutorials/tutorialVideos/RajDemoPrepSolvePost.mov RajDemoPrepSolvePost.mov ]

[[Tutorial_Video_Overviews#RajDemoPrepSolvePost.mov|Video Notes]]

[https://fluid.colorado.edu/tutorials/tutorialVideos/PrepSolvePostBLandC.mp4 PrepSolvePostBLandC.mp4 ]

[[Tutorial_Video_Overviews#PrepSolvePostBLandC.mp4|Video Notes]]

[[Category:Chef]]

The On Ramp/Level 1/Partition

2022-09-18T18:38:32Z

Jrwrigh: Jrwrigh moved page Level 1 Partition to The On Ramp/Level 1/Partition: Move to a subpage organization

== Chef==
Chef is an open source SCOREC tool that is this group's primary tool to partition a problem domain to many subdomains. This is done to allow any array of compute nodes to each focus on solving the problem in parallel. That is, to divide the problem domain into subdomains to give to different computational workers. When we perform this step, all that is needed for inputs are the, 1) SCOREC mesh constructed from the output of the conversion and meshing steps, and 2) the number of parts (subdomains) to divide the problem up into. A generic workflow for this step is described below and is a good place to start when completing the workflow for the first time. As your studies continue, many aspects of the layout and modifiers will not be consistent with this On Ramp documentation.
=== Creating the serial case via Viz nodes===
Create a <code>1-1-Chef</code> subdirectory inside the <code>PrepAndRun</code> folder. Enter the new folder and copy over the files <code>adapt.inp</code> and <code>runChef.sh</code> from the <code>/projects/tutorials/OnRamp</code> folder. To learn more about the <code>adapt.inp</code> file, enter the command:

more adapt.inp

Pay particular attention to the splitFactor, attributeFileName, modelFileName, meshFileName bz2, and outMeshFileName bz2 specifications. These are often the first nobs that the new user learns to control. When it comes to first creating "checkpoint" files that are needed for the PHASTA executable. splitFactor will tell the Chef executable how many partitions each incoming part needs to be partitioned by. Immediately after the conversion step we want this to be set to 1 because we desire a single "checkpoint" file. attributeFileName will tell Chef where the file that contains the attributes of each geometry entity (region, face, line, and vertex info) is. Likewise, the modelFileName will tell Chef where the file that contains the geometry entity information is. The meshFileName bz2 and outMeshFileName bz2 specifications determine the directory where Chef will look to gather the SCOREC mesh and where to put the partitioned SCOREC mesh. Notice that this pre-written <code>adapt.inp</code> file points the meshFileName bz2 attribute to the mdsMesh_bz2 folder located in the simMeshToMdsMesh directory.
Run the Chef bash script by entering the command <code>./runChef.sh 1</code>. This should output a <code>1-procs_case</code> directory, an <code>mdsMesh_bz2</code> directory, and a <code>chef.log</code> file.

=== Partioning to 8 processes via Viz nodes===

Like the creation of the serial case above, we will make another subdirectory named 8-1-Chef. Copy from the 1-1-Chef directory the adapt.inp as well as the runChef.sh bash script that were used into this 8-1-Chef subdirectory. We will alter the meshFileName bz2 and splitFactor specifications within newly created adapt.inp. Set the specification meshFileName bz2 to "../1-1-Chef/mdsMesh_bz2". Set the splitFactor to 8. Run Chef via the bash script <code>./runChef.sh 8</code>. The executable should be able to read the mesh from the 1-1-Chef directory as well as the geom files as before.

Now that you have successfully partitioned the mesh, it's time to use PHASTA in the [[Level 1 Solve]] step.

=== Further Partitioning to N processes===
By now you may be able to guess how to go about further partitioning your case into more and more partitions. If the user desired the end part count of 32 processes ( call that N), the next steps would mimic the latter with the subdirectory named 32-8-Chef and the splitFactor be set to 4.
There are a few things to consider when partitioning an actual case you care about. It is tempting to make the successive split factors large as to have a fewer amount of repeated partions. However, the splitfactor should not often exceed 8 especially on an unstructured grid. If it does, Chef will have a more difficult time splitting the elements evenly amongst the parts, causing an imbalance in the workloads from process to process, and therefore reducing the effectiveness of the PHASTA run. Secondly, when the mesh is large, the partitioning that is performed on the viznode should not exceed 32 parts. The successive partitionings past this amount should be done on larger machines. Sometimes if a mesh is large enough, the preliminary serial partition should be performed on a fatter node that what is available on the viz nodes (ie CU's summit resource).

=== A few helpful video tutorials===

[https://fluid.colorado.edu/tutorials/tutorialVideos/PHASTA_workflow_RB.mkv PHASTA_workflow_RB.mkv ]

[[Tutorial_Video_Overviews#PHASTA_workflow_RB.mkv|Video Notes]]

[https://fluid.colorado.edu/tutorials/tutorialVideos/RajDemoPrepSolvePost.mov RajDemoPrepSolvePost.mov ]

[[Tutorial_Video_Overviews#RajDemoPrepSolvePost.mov|Video Notes]]

[https://fluid.colorado.edu/tutorials/tutorialVideos/PrepSolvePostBLandC.mp4 PrepSolvePostBLandC.mp4 ]

[[Tutorial_Video_Overviews#PrepSolvePostBLandC.mp4|Video Notes]]

[[Category:Chef]]

Level 1 Partition

2022-09-18T18:38:32Z

Jrwrigh: Jrwrigh moved page Level 1 Partition to The On Ramp/Level 1/Partition: Move to a subpage organization

#REDIRECT [[The On Ramp/Level 1/Partition]]

The On Ramp/Level 1/Model/Mesh

2022-09-18T18:38:04Z

Jrwrigh: Jrwrigh moved page Level 1 Model/Mesh to The On Ramp/Level 1/Model/Mesh: Move to a subpage organization

* [[Exporting Parasolid from SolidWorks |Exporting Parasolid from SolidWorks ]]
* [[Getting Started with Simmodeler|Getting Started with Simmodeler]]
* [[Prepping the Grid for Chef|Prepping the Grid for Chef]]

Level 1 Model/Mesh

2022-09-18T18:38:04Z

Jrwrigh: Jrwrigh moved page Level 1 Model/Mesh to The On Ramp/Level 1/Model/Mesh: Move to a subpage organization

#REDIRECT [[The On Ramp/Level 1/Model/Mesh]]

The On Ramp/Level 1

2022-09-18T18:37:32Z

Jrwrigh: Created page with "This is the base page for Level 1 of The On Ramp. This contains a basic tutorial on how to create a PHASTA simulation from scratch. This diagram covers the steps of t..."

This is the base page for Level 1 of [[The On Ramp]]. This contains a basic tutorial on how to create a [[PHASTA]] simulation from scratch.

This diagram covers the steps of the workflow, the files involved, and how they all relate to each other:
[[File:Picture5.png]]

The steps of the tutorial are:
* [[Level 1 Model/Mesh| Model/Mesh Workflow]]
* [[Level 1 Partition| Partition Workflow]]
* [[Level 1 Solve| Solve Workflow]]
* [[Level 1 Post-Process| Post-Process Workflow]]

== Subpages ==
{{Special:PrefixIndex/The On Ramp/Level 1/}}

PhParAdapt

2022-09-18T18:27:06Z

Jrwrigh: Add subpages.

'''phParAdapt''' is a program for doing mesh adaptation in PHASTA. This is either done using [[SCOREC Core]] utilties, or though Simmetrix.

[[phParAdapt-SCOREC]]

[[phParAdapt-Simmetrix]]

== Subpages ==
{{Special:PrefixIndex/PhParAdapt/}}

[[Category:Software]]

PhParAdapt/Simmetrix

2022-09-18T18:26:31Z

Jrwrigh: Jrwrigh moved page PhParAdapt-Simmetrix to PhParAdapt/Simmetrix: Move to a subpage organization

The following page provides details for performing mesh adaptation using the PhParAdapt tool with Simmetrix routines.

== Initial Notes to User ==
* The solution migration feature is currently broken and thus requires other means for transferring the solution to new meshes that are created from phParAdapt (e.g. solution interpolation in Paraview)
* To date, the steps described herein have only been tried with the following restrictions:
** Tetrahedral elements
*** Extruded element types apparently can be used but require removal of any extrusion constraints in the mesh. Below are some incomplete details regarding that process for reference. See [[#Removal_of_Extrusion_Constraint |Removal of Extrusion Constraint]].
** Serial case
** Geometries created using SimModeler7.0-190604
** The version of phParAdapt noted here uses only the 6th entry of the error field for identifying regions for adaptation

== Adaptation Process ==

=== Creating Initial Restart and Error Files ===
Beginning inside of the 1-1-Chef directory,
<nowiki>mkdir 1-1-phParAdapt
cd 1-1-phParAdapt </nowiki>

Create soft links to the geom files above the 1-1-Chef directory,
<nowiki>ln -s ../../geom.smd
ln -s ../../geom.sms
ln -s ../../geom_nat.x_t </nowiki>

Create an <code>adapt.inp</code> file using the template [[#Adapt.inp_(non-adaptation_step) | Adapt.inp (non-adaptation step)]]. Keep <code>adaptFlag</code> set to zero as the purpose of running phParAdapt here is to create restart.<timeStepNumber>.1 file associated with a new mesh format that can be used for adaptation.

Run phParAdapt in this directory, which for this specific example was:

<code>mpirun -np 1 /projects/tools/Simmetrix.develop_19/phParAdapt-Sim/phParAdapt-Sim19/bin/x86_64_linux/phParAdapt-parasolid-openmpi-O 2>&1 | tee phParAdapt.log</code>

If the above step fails, review the 'MeshSim.#.log' file that is produced to identify the error. One error related to analysis attributes has been experienced. The issue was found to be that under the 'Analysis attributes' section, on the 'problem definition' line the name must be set to 'geom'.

After completion, the 1-procs_case and mesh_parts.sms directories should now be present. If a solution has already been generated for a different mesh on the same geometry, it can now be interpolated onto the new restart.<#>.1 file that was generated. If no solution exists, run Phasta using this restart file until the desired solution state has been achieved. Be sure to set <code>Print Error Indicators: True</code> in the solver.inp file so the error fields are saved in the final restart file. Note, error fields are not printed in intermediate restart files. Before proceeding, check that the error fields exist using <code>grep -a " : <" restart.<#>.1</code>

=== Performing the Adaptation Step ===
Within the 1-1-phParAdapt directory,
<nowiki>mkdir A1-phParAdapt
cd A1-phParAdapt
ln -s ../geom.smd
ln -s ../geom_nat.x_t
ln -s ../mesh_parts.sms geom.sms
ln -s geom.sms parts.sms
ln -s ../1-procs_case/restart.1.1
ln -s restart.1.1 errors.1.1 </nowiki>

In the above step, it is assumed that the solution was interpolated onto restart.1.1. If that was not the case and the solution was advanced until a desired step was reached, replace restart.1.1 & errors.1.1 with restart.<#>.1 & errors.<#>.1 corresponding to the desired solution step number with error fields.

Create an adapt.inp file using [[#Adapt.inp_(adaptation step) |Adapt.inp_(adaptation step)]] as a template. Change the <code>timeStepNumber</code> parameter to correspond to the restart file that was linked into the current directory. Make sure the <code>adaptFlag</code> parameter is set to one and change additional threshold flags to control the result of the refinement.

Run phParAdapt using the mpirun command from the previous section. A directory corresponding to the <code>timeStepNumber</code> parameter should have been created with a mesh_parts.sms folder inside upon successful completion.

=== Generate Restart File for Refined Mesh ===
From within the A1-phParAdapt directory, execute the following (noting to replace <#> with the appropriate time step number):
<nowiki>cd <#>
mkdir 1-1-phParAdapt
cd 1-1-phParAdapt
ln -s ../../../geom.smd
ln -s ../../../geom_nat.x_t
ln -s ../mesh_parts.sms geom.sms
cp ../../../adapt.inp .</nowiki>

In the above list of commands, the adapt.inp file from the non-adaptation step was copied into the current directory. Run phParAdapt using the same mpirun command as above to generate the 1-procs_case folder and the new restart file with the refined mesh. Again, since the solution migration feature is not working, the restart file will contain the initial solution and no error fields. A previously obtained solution can now be interpolated to the newly created one or PHASTA can be run in this directory to obtain a solution on the new mesh.

In order to perform a second (or more) adaptation step, create a restart file in the 1-procs_case folder in the current directory (either by interpolation or running PHASTA) and make sure it contains the error fields. Create a new directory <code>A2-phParAdapt</code> at this level and repeat the above steps. Be sure to use the time step number moving forward that is associated with the restart file just mentioned.

== File Examples ==

The file examples below are what were used for this specific example and may contain parameter flags that may need to be changed or are not used at all.

=== Adapt.inp (non-adaptation step) ===
<nowiki>numberSolutionVars 6
numberErrorVars 10
refWeights 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0
refThreshold 0.01
globalP 1
timeStepNumber 1
numelX 0
NSFTAG -1
ensa_dof 6
attributeFileName geom.smd
meshFileName geom.sms
modelFileName geom_nat.x_t
Idirection 0
BYPASS 0
zScale 0
adaptFlag 0
errorName 0
SONFATH 0
lStart 0
rRead 6
rStart 0
AdaptStrategy 5
AdaptFactor 0.00001
AdaptOption 11
hmax 0.04
hmin 0.000052083
multipleRestarts 0
Periodic 0
prCD 0
timing 0
wGraph 0
phastaVersion 1.9.5
old_format 0
FortFormFlag 0
outputFormat binary
CUBES 0
internalBCNodes 0
version UNKNOWN
WRITEASC 0
phastaIO 1
numTotParts 1
SolutionMigration 0
DisplacementMigration 0
isReorder 0
isMCOPI 0
isBLAdapt 0
isThickAdapt 0
isSizeLimit 1
MaxLimitFact 2
MinLimitFact 2
rho 1.225
mu 1.7825e-5
dwalMigration 0
buildMapping 1 </nowiki>

=== Adapt.inp (adaptation step) ===
<nowiki>eviscScal 0
pgrad 0
ub 1.45
lb 0.70
numSplit 5
sizeRatio 0.35 # confirmed that 0.25 does a 4x ref
numSmooth 0 #confirmed that this gets used
ratioThresh 0.75 # this is h_new/h_orig from isotropic days
localAdapt 1 # makes setLocal force adaptivity only around nodes set size
AnisoSimmetrix 4 # > 0 use Simmetrix, else ours 1 does largest length reduction by sizeRatio, 2 does both does medium and largest reduction by sizeRatio
coarsenMode 0 # disables coarsening see coarsenMode docs for 1, and 2
numberSolutionVars 6
numberErrorVars 10
refWeights 1 1 1 1 1 1 1 1 1 1
refThreshold 0.01
globalP 1
timeStepNumber 1
numelX 0
NSFTAG -1
ensa_dof 6
attributeFileName geom.smd
meshFileName geom.sms
modelFileName geom_nat.x_t
Idirection 0
BYPASS 0
zScale 0
adaptFlag 1
errorName 0
SONFATH 0
lStart 0
rRead 6
rStart 0
AdaptStrategy 5
AdaptFactor 0.5e-4
AdaptOption 11
hmax 1e6
hmin 1e-5
multipleRestarts 0
Periodic 0
prCD 0
timing 0
wGraph 0
phastaVersion 1.9.5
old_format 0
FortFormFlag 0
outputFormat binary
CUBES 0
internalBCNodes 0
version UNKNOWN
WRITEASC 0
phastaIO 0
numTotParts 1
SolutionMigration 1
DisplacementMigration 0
isReorder 0
isMCOPI 0
isBLAdapt 0
isThickAdapt 0
isSizeLimit 0
MaxLimitFact 2
MinLimitFact 2
rho 1.225
mu 1.7825e-5
dwalMigration 0
buildMapping 1 </nowiki>

== Notes on Debugging ==
If debugging of phParAdapt is desired, replace the normal <code>mpirun</code> command with the following:

<code>mpirun -np 1 -tv /projects/tools/Simmetrix.develop_19/phParAdapt-Sim/phParAdapt-Sim19/bin/x86_64_linux/phParAdapt-parasolid-openmpi</code>

where the <code>-O</code> has been removed from the path to the executable to call the build version that was not optimized. Note that this requires phParAdapt to have been built at some point prior with the <code>Debug</code> flag.

== Notes on Folder Structure ==
Below is an overview of the folder structure to reference during the adaptation process:
# [ geom.smd, geom.sms, geom_nat.x_t ]
# 1-1-Chef
## 1-1-phParAdapt
### adapt.inp
### run_phParAdapt.sh
### 1-procs_case
#### geombc.dat.1
#### restart.<#>.1
### mesh_parts.sms
### A1-phParAdapt
#### restart.<#>.1
#### errors.<#>.1
#### [geom.smd, geom.sms, geom_nat.x_t, parts.sms]
#### adapt.inp
#### <#> ( step number directory )
##### mesh_parts.sms
##### 1-1-phParAdapt
###### [geom.smd, geom.sms, geom_nat.x_t]
###### adapt.inp
###### 1-procs_case ( restart files with adapted mesh )

== Removal of Extrusion Constraint ==
Riccardo has used a procedure in the past to remove the extrusion constraint from a mesh that enabled adaptation to be performed on extrusion-type elements.

The following directory contains an example of this process:
<code> /projects/tools/Models/NASAWingBodyJunction/RajMeshFine/Mesh/1-1-phParAdapt/RemExtrusion/ </code>

The executable that is called for removal of the constraint is a part of SCOREC-core, found at <code> <path/to/SCOREC-core/build/dir>/test/rm_extrusion </code> and the source code, in case edits are required, is found at <code> <path/to/SCOREC-core/source/dir>/core/test/rm_extrusion.cc </code>

PhParAdapt-Simmetrix

2022-09-18T18:26:31Z

Jrwrigh: Jrwrigh moved page PhParAdapt-Simmetrix to PhParAdapt/Simmetrix: Move to a subpage organization

#REDIRECT [[PhParAdapt/Simmetrix]]