The On Ramp/Level 1/Solve

Exporting to PHASTA

After the partitioning performed via Chef in the last steps, we now have the problem domain in a form that the PHASTA executable can read. In your 8-1-Chef directory, create a sub-directory named "Run". This will contain all of the simulation data from this case, which we will use in our PHASTA run. Remember that for this on-Ramp tutorial we have partitioned our case of interest to 8 parts. Therefore, we need to create a sub-directory named "8-procs_case" within the /8-1-Chef/Run/ directory. Once you have mkdir'd and cd'd into this new Run/8-procs_case/ subdirectory, make softlinks ("ln -s <path/file*>") to the N=8 restart and geombc "checkpoint" files that where constructed by Chef located in the 8-1-Chef/8-procs_case/ directory. When in the Run/8-procs_case sub-directory, a good command to do this is the following:

ln -s ../../8-procs_case/restart* .
ln -s ../../8-procs_case/geombc* .

Also create a numstart.dat file. This file will specify the time and timestep that the simulation has completed thus far. For our case, we have not yet run the simulation, thus our time and timesteps are 0 0. Use the following command in the "Run/8-procs_case" directory to create the numstart.dat file:

echo 0 0 > numstart.dat

Build the executable/specify runtime parameters

What remains is to determine the version of PHASTA to build and run. Since there are a bunch of researchers working on PHASTA at any given time, there are many branches/versions of the main code.

Note: You may not have access to the phasta-next repo yet. If that is the case, ask Dr. Jansen to have you added to the repo. In the meanwhile, you can just use the regular phasta repo (by replacing all "phasta-next" with "phasta" in these instructions).

Retrieve/build a version of PHASTA code

To have your first run of PHASTA, you will need an executable of the PHASTA code. There are many ways to do this, and below is intended to get you a generic executable. The many nuances to this proccess can be found on the Level 2 page here. Navigate to your home directory. Create and enter a directory here named "git-phasta". In a web browser, navigate to the online git repository for phasta-next and select the "clone" or "code" icon. The result should be similar to the following picture, where a pop-up gives a web address:

Copy this address and within the "git-phasta" directory execute the following command and enter your github credentials:

git clone https://github.com/PHASTA/phasta-next.git

After this is finished there will be a subdirectory created named "phasta-next" that contains the code tree that you wish to build. Back within the git-phasta directory create another subdirectory named "build_phasta-next". Now we must set the environment so that the compiler has the necessary libraries. If you perform the following command, the listed environment libraries that ken often uses are listed, and are often sufficient:

more ~kjansen/soft-core.sh 

At the time that this page is created, the relevant commands to load the needed environment are the following:

soft add +gcc-6.3.0
soft add +openmpi-gnu-1.10.6-gnu49-thread
soft add +simmodeler-6.0-171202

If the user needs additional libraries they can often be found with the following command:

softenv

lets build this code! Navigate into this build_phasta-next subdirectory and create a file named "unpack_buildFiles.sh" with the following content:

#!/bin/bash

Target=Example_Build
mkdir Example_Build
rm -r $Target/*
cd $Target

export PKG_CONFIG_PATH=/users/skinnerr/tools/git-petsc/build_ompi210_gnu63/lib/pkgconfig/

cmake \
-DCMAKE_C_COMPILER=gcc \
-DCMAKE_CXX_COMPILER=g++ \
-DCMAKE_Fortran_COMPILER=gfortran \
-DCMAKE_BUILD_TYPE=Debug \
-DPHASTA_INCOMPRESSIBLE=ON \
-DPHASTA_COMPRESSIBLE=OFF \
-DPHASTA_USE_LESLIB=ON \
-DLESLIB=/users/matthb2/libles1.5/libles-debianjessie-gcc-ompi.a \
-DCASES=/home/mrasquin/develop-phasta/phastaChefTests \
-DPHASTA_TESTING=OFF \
../../phasta-next/

make -j8
echo "Target: $Target"
date

Note: You can create this file by typing vi unpack_buildFiles.sh into the command line. This will create an empty shell script file named unpack_buildFiles.sh and enter you into the Vim editor mode, where you can practice your recently adopted Vim commands to copy the above script into the file, making sure to save and quit after you're done. Enter the following command to make sure your shell script was saved successfully with all the required text:

more unpack_buildFiles.sh

Now, we must turn the above .sh file into an executable by typing the following command:

chmod +x unpack_buildFiles.sh

Finally, we are ready to run the executable and build the PHASTA code!

./unpack_buildFiles.sh

You can check that your executable has been built by locating:

Example_Build/bin/phastaIC.exe

Additional notes

If there is a specific branch off of phasta-next that you'd like to build, navigate to phasta-next and use the following command:

git checkout "branchname". 

If this is a branch that I will be working on for a while, I tend to alter the build and code directories according to the branch-name. Note that the respective pointer in the "unpack_buildFiles.sh" file (ie the last line) will have to be set accordingly.

Create Solver.inp

The input.config file in the newly created build_phasta/Example_Build/ directory contains all possible options that could be set in the solver.inp file (which we will use for our PHASTA run). Create the solver.inp file in the .../PrepAndRun/8-1-Chef/Run/ directory, and then specify all of the parameters that you wish to change for your run case. NEVER EVER Modify the input.config file itself. The rest of the parameters need not be specified. For example my solver.inp looks as follows:

# ibksiz flmpl flmpr itwmod wmodts dmodts fwr taucfct
# PHASTA Version 1.5 Input File
#
#  Basic format is
#
#    Key Phrase  :  Acceptable Value (integer, double, logical, or phrase
#                                     list of integers, list of doubles )
#
#
#SOLUTION CONTROL 
#{                
    Equation of State: Incompressible
    Number of Timesteps: 10
    Time Step Size: 1e-1  # Delt(1)
    Turbulence Model: RANS  # No-Model # DES97 # DDES  iturb=0, RANS =-1  LES=1 #}
#}

#MATERIAL PROPERTIES
#{
    Viscosity: 1.50e-5      # fills datmat (2 values REQUIRED if iLset=1)
    Density: 1.0           # ditto
    Body Force Option: None # ibody=0 => matflag(5,n)
    Body Force: 0 0.0 0.0    # (datmat(i,5,n),i=1,nsd)
    Thermal Conductivity: 27.6e-1jjj  # ditto
    Scalar Diffusivity: 27.6e-1    # fills scdiff(1:nsclrS)
#}

OUTPUT CONTROL
{
    Number of Timesteps between Restarts: 5 #replaces nout/ntout
    Number of SyncIO Files: 0
    Print Error Indicators: False
    Number of Error Smoothing Iterations: 0 # ierrsmooth
#    Print ybar: True 
#    Print vorticity: True
#    Print Wall Fluxes: False
#    Print Statistics: False
    Number of Force Surfaces: 1
    Surface ID's for Force Calculation: 1
#     Ranks per core: 4 # for varts only
#     Cores per node: 16 # for varts only
}

#LINEAR SOLVER
#    Solver Type: GMRES sparse
    Solver Type: ACUSIM with P Projection
    Number of GMRES Sweeps per Solve: 1      # replaces nGMRES
    Number of Krylov Vectors per GMRES Sweep: 200           # replaces Kspace
    Scalar 1 Solver Tolerance : 1.0e-4
    Tolerance on Momentum Equations: 0.05                   # epstol(1)
    Tolerance on ACUSIM Pressure Projection: 0.01           # prestol 
    Number of Solves per Left-hand-side Formation: 1  #nupdat/LHSupd(1)
    ACUSIM Verbosity Level               : 0   #iverbose
    Minimum Number of ACUSIM Iterations per Nonlinear Iteration: 10  # minIters
    Maximum Number of ACUSIM Iterations per Nonlinear Iteration: 200 # maxIter
#}

#DISCRETIZATION CONTROL
#{
    Time Integration Rule: First Order    # 1st Order sets rinf(1) -1
#    Time Integration Rule: Second Order    # Second Order sets rinf next
#    Time Integration Rho Infinity: 0.0     # rinf(1) Only used for 2nd order
    Tau Matrix: Diagonal-Shakib               #itau=1
    Tau Time Constant: 1.0                      #dtsfct
    Include Viscous Correction in Stabilization: True    # if p=1 idiff=1
                                                         # if p=2 idiff=2  
    Lumped Mass Fraction on Left-hand-side: 0.0           # flmpl
    Lumped Mass Fraction on Right-hand-side: 0.0          # flmpr
    Tau C Scale Factor: 1.0                    # taucfct  best value depends  
    Number of Elements Per Block: 64 # switch to >250 if sgi
#} 

TURBULENCE MODELING PARAMETERS
{
   Turbulence Wall Model Type: None  #itwmod=2 RANSorLES
    }

#STEP SEQUENCE 
#{
     Step Construction  : 0 1 10 11 0 1 10 11
#}

Running the Solver

Create the runPHASTA.sh bash script in your 8-1-Chef/Run directory. Where you see <pathtoBuildDir> include your path to your build directory where you retrieved the input.config file. The #export line is an example that I have used myself, yours should look similar:

#!/bin/bash

rm $1-procs_case/doubleRun-check

#export PHASTA_CONFIG=/users/jopa6460/git-phasta/build_phasta/Example_Build
export PHASTA_CONFIG=<pathtoBuildDir>

mpirun -np $1 $PHASTA_CONFIG/bin/phastaIC.exe 2>&1 | tee $1.out

Remember to turn the file into an executable as was done for the build script above. Hint: chmod +x command.

NOTE: Before running PHASTA, it is important to check that nobody else is running important jobs on the computer you're connected to. Run top in the command line to check what other processes are running, and if you see other people running large jobs you should ask them if you can run your job as well, or else switch to a different computer.

Now you have everything you need to run your first PHASTA simulation! In the Run directory execute the following line:

./runPHASTA.sh 8

Output of a proper run of PHASTA contains information about 1) the step number of the simulation; 2) the relative residual (convergence relative to this run's initial residual); and 3) various other outputs based on runtime parameters like BC's and IC's. As the simulation continues it will produce successive checkpoint restart files that contain the solution data at that timestep. For our example, these restart files will be located in the .../Run/8-procs_case directory. These restart files and the geombc files are what Paraview post-processes to give us a visualization of flow field solutions.

Now it's time to plot the results in the Level 1 Post-Process step.

A few helpful video tutorials

CloneFromGithubAndBranch.mov

Video Notes

BlancoBuidingPHASTA.mov

Video Notes

PHASTA_workflow_RB.mkv

Video Notes

RajDemoPrepSolvePost.mov

Video Notes

PrepSolvePostBLandC.mp4

Video Notes