Difference between revisions of "Solve"

From PHASTA Wiki
Jump to: navigation, search
(→‎Build the executable)
(→‎A few helpful video tutorials)
 
(9 intermediate revisions by the same user not shown)
Line 7: Line 7:
 
=== Build the executable/specify runtime parameters===
 
=== Build the executable/specify runtime parameters===
 
What remains is to determine the version of PHASTA to build and run. Since there are a bunch of researchers working on PHASTA at any given time, there are many branches/versions of the main code. If you are a part of the research cohort you may be directed to phasta-next, but many of the same steps may be taken from the public version.  
 
What remains is to determine the version of PHASTA to build and run. Since there are a bunch of researchers working on PHASTA at any given time, there are many branches/versions of the main code. If you are a part of the research cohort you may be directed to phasta-next, but many of the same steps may be taken from the public version.  
−
==== Retrieve/build a version of PHASTA code ===
+
==== Retrieve/build a version of PHASTA code ====
−
Navigate to your home directory. Create a directory here named "git-phasta". In a web browser, navigate to the online git repository for phasta-next. Within this directory create one subdirectories named "phasta-next" and "build_phasta-next".  
+
Navigate to your home directory. Create a directory here named "git-phasta". In a web browser, navigate to the online git repository for phasta-next and select the "clone" or "code" icon. The result should be the simular to the following picture, where a pop-up gives a web address:
−
[[File:
+
[[File:GitClone.png]]
 +
Copy this address and within the "git-phasta" directory execute the following command and enter your github credentials:
 +
git clone https://github.com/PHASTA/phasta-next.git
 +
After this is finished their will be a subdirectory created named phasta-next that contains the code tree that you wish to build.
 +
Back within the git-phasta directory create another subdirectory named: "build_phasta-next". Now we must set the environment so that the compiler has the necessary libraries. If you perform the following command, the listed environment libraries that ken often uses are listed, and are often sufficient:
 +
more ~kjansen/soft-core.sh
 +
At the time that this page is created, the relevant commands to load the needed environment are the following:
 +
soft add +gcc-6.3.0
 +
soft add +openmpi-gnu-1.10.6-gnu49-thread
 +
soft add +simmodeler-6.0-171202
 +
If the user needs additional libraries they can often be found with the following command:
 +
softenv
 +
lets build this code! Navigate into this build subdirectory and create a file named "unpack_buildFiles.sh" with the following content:
 +
#!/bin/bash
 +
 
 +
cmake \
 +
-DCMAKE_C_COMPILER=gcc \
 +
-DCMAKE_CXX_COMPILER=g++ \
 +
-DCMAKE_Fortran_COMPILER=gfortran \
 +
-DCMAKE_BUILD_TYPE=Debug \
 +
-DPHASTA_INCOMPRESSIBLE=ON \
 +
-DPHASTA_COMPRESSIBLE=OFF \
 +
-DPHASTA_USE_LESLIB=ON \
 +
-DLESLIB=/users/matthb2/libles1.5/libles-debianjessie-gcc-ompi.a \
 +
-DCASES=/home/mrasquin/develop-phasta/phastaChefTests \
 +
-DPHASTA_TESTING=OFF \
 +
../phasta-next-master/
 +
This should create the makefiles required for the build. Perform the following command:
 +
make -j
 +
The following means that you executable has been built:
 +
Built target phastaIC.exe
 +
Also, bin/phastaIC.exe will now exist. Take note of the path to the build directory for the next steps.
 +
 
 +
 
 +
===== Additional notes =====
 +
If there is a specific branch off of phasta-next that you'd like to build, navigate to phasta-next and use the following command:
 +
git checkout "branchname".  
 +
If this is a branch that I will be working on for a while, I tend to alter the build and code directories according to the branch-name. Note that the respective pointer in the "unpack_buildFiles.sh" file (ie the last line) will have to be set accordingly.
 +
=== Create Solver.inp ===
 +
From your build directory copy input.config to your Run directory containing the 8-procs-case. This file contains all of the default PHASTA runtime input parameters of this version of PHASTA that you have built. In another file named solver.inp in the Run directory you will want to specify all of these parameters that you wish to change for this case. The rest of the parameters need not be specified. For example my solver.inp looks as follows:
 +
# ibksiz flmpl flmpr itwmod wmodts dmodts fwr taucfct
 +
# PHASTA Version 1.5 Input File
 +
#
 +
#  Basic format is
 +
#
 +
#    Key Phrase  :  Acceptable Value (integer, double, logical, or phrase
 +
#                                    list of integers, list of doubles )
 +
#
 +
#
 +
#SOLUTION CONTROL
 +
#{               
 +
    Equation of State: Incompressible
 +
    Number of Timesteps: 1
 +
    Time Step Size: 2e-3  # Delt(1)
 +
    Mode to Compute dwal: 0
 +
    Turbulence Model: No-Model  #RANS  # No-Model # DES97 # DDES  iturb=0, RANS =-1  LES=1 #}
 +
#    IDDES Constants: 3.55 0.9
 +
    Ramp Inflow: False #True #False = no jet activated 
 +
#Required for guillermo IC compairison DNS
 +
    Solve Heat: True
 +
#}
 +
 
 +
#MATERIAL PROPERTIES
 +
#{
 +
    Viscosity: 1.57e-5      # fills datmat (2 values REQUIRED if iLset=1)
 +
    Density: 1.0          # ditto
 +
    Body Force Option: None # ibody=0 => matflag(5,n)
 +
    Body Force: 0 0.0 0.0    # (datmat(i,5,n),i=1,nsd)
 +
    Thermal Conductivity: 27.6e-1  # ditto
 +
    Scalar Diffusivity: 27.6e-1    # fills scdiff(1:nsclrS)
 +
#}
 +
 
 +
OUTPUT CONTROL
 +
{
 +
    Number of Timesteps between Restarts: 10 #replaces nout/ntout
 +
    Print Error Indicators: False
 +
    Number of Error Smoothing Iterations: 0 # ierrsmooth
 +
    Print ybar: True
 +
    Print vorticity: True
 +
    Print Wall Fluxes: False
 +
    Print Statistics: False
 +
    Number of Force Surfaces: 1
 +
    Surface ID's for Force Calculation: 1
 +
#    Ranks per core: 4 # for varts only
 +
#    Cores per node: 16 # for varts only
 +
}
 +
 
 +
#LINEAR SOLVER
 +
# Commented Lines Result from DNS setup
 +
#    Solver Type: ACUSIM with P Projection
 +
#    Number of GMRES Sweeps per Solve: 1      # replaces nGMRES
 +
#    Number of Krylov Vectors per GMRES Sweep: 200          # replaces Kspace
 +
#    Scalar 1 Solver Tolerance : 1.0e-4
 +
#    Tolerance on Momentum Equations: 0.05                  # epstol(1)
 +
    Tolerance on ACUSIM Pressure Projection: 0.01          # prestol
 +
    Number of Solves per Left-hand-side Formation: 1  #nupdat/LHSupd(1)
 +
    ACUSIM Verbosity Level              : 0  #iverbose
 +
    Minimum Number of ACUSIM Iterations per Nonlinear Iteration: 10  # minIters
 +
#    Maximum Number of ACUSIM Iterations per Nonlinear Iteration: 200 # maxIter
 +
#}
 +
 
 +
#DISCRETIZATION CONTROL
 +
#{
 +
    Basis Function Order: 1                # ipord
 +
    Time Integration Rule: Second Order    # Second Order sets rinf next
 +
    Time Integration Rho Infinity: 0.75    # rinf(1) Only used for 2nd order
 +
    Include Viscous Correction in Stabilization: False    # if p=1 idiff=1
 +
                                                          # if p=2 idiff=2 
 +
    Quadrature Rule on Interior: 2          #int(1)
 +
    Quadrature Rule on Boundary: 2          #intb(1)
 +
    Lumped Mass Fraction on Left-hand-side: 0.0          # flmpl
 +
    Lumped Mass Fraction on Right-hand-side: 0.0          # flmpr
 +
#    Tau Matrix: Diagonal-Franca              #itau=1
 +
    Tau Time Constant: 16.                      #dtsfct
 +
    Tau C Scale Factor: .1                    # taucfct  best value depends 
 +
    Number of Elements Per Block: 32 # switch to >250 if sgi        #ibksiz
 +
#}
 +
 
 +
TURBULENCE MODELING PARAMETERS
 +
{
 +
#  Dynamic Model Type : Standard  # adds zero to iturb      LES
 +
#        Filter Integration Rule: 1  #ifrule adds ifrule-1 to iturb LES
 +
#  Turbulence Wall Model Type: Effective Viscosity  #itwmod=2 RANSorLES
 +
#  Velocity Averaging Steps : 500. # wtavei= 1/this          RANSorLES
 +
#  Dynamic Model Averaging Steps : 500. # dtavei= 1/this      LES
 +
#  Filter Width Ratio        : 3.  # fwr1                    LES
 +
    }
 +
#
 +
#STEP SEQUENCE
 +
#{
 +
      Step Construction  : 0 1 0 1 5 6 5 6 5 6  # this is the standard two iteration
 +
#}
 +
 
  
 
=== Running the Solver ===
 
=== Running the Solver ===
 +
Create the runPHASTA.sh in your run directory. Where you see <pathtoBuildDir> include your path to your build directory where you retrieved the input.config file. The #pconf line is an example that I have used myself, yours should look simular.
 +
#/bin/bash
 +
# VER=C or VER=IC should be specified
 +
#pconf=/users/jopa6460/git-phasta/build_JohnDev-STG_D; VER=IC
 +
pconf=<pathtoBuildDir>; VER=IC
 +
 +
date
 +
mpirun $2 -np $1  -x PHASTA_CONFIG=$pconf $pconf/bin/phasta${VER}.exe
 +
date
 +
     
 +
 +
Now you have everything you need to run your first PHASTA simulation! In the Run directory execute the following line:
 +
./runPHASTA.sh 8
 +
Output of a proper run of PHASTA contains information on 1)the step number the simulation is on 2) the relative residual (convergence relative to this run's initial residual), 3) various other outputs based on runtime parameters (like BC and IC info).
 +
A single time iteration with step construction 1 0 10 11 1 0 10 11 has the following output:
 +
 +
Where
 +
 +
 +
As the simulation continues it will produce successive checkpoint restart files that contain the solution data at that timestep. These and the geombc files will be what is post-proccessed in Paraview.
 +
 +
 +
=== A few helpful video tutorials===
 +
 +
[https://fluid.colorado.edu/tutorials/CloneFromGithubAndBranch.mov CloneFromGithubAndBranch.mov ] (video)
 +
 +
[[Tutorial_Video_Overviews#CloneFromGithubAndBranch.mov|Video notes of CloneFromGithubAndBranch.mov]] (link to timestamps)
 +
 +
 +
 +
 +
[https://fluid.colorado.edu/tutorials/BlancoBuidingPHASTA.mov BlancoBuidingPHASTA.mov ] (video)
 +
 +
[[Tutorial_Video_Overviews#BlancoBuidingPHASTA.mov|Video notes of BlancoBuidingPHASTA.mov]] (link to timestamps)
 +
 +
 +
 +
 +
[https://fluid.colorado.edu/tutorials/PHASTA_workflow_RB.mkv PHASTA_workflow_RB.mkv ] (video)
 +
 +
[[Tutorial_Video_Overviews#PHASTA_workflow_RB.mkv|Video notes of PHASTA_workflow_RB.mkv]] (link to timestamps)
 +
 +
 +
 +
 +
[https://fluid.colorado.edu/tutorials/RajDemoPrepSolvePost.mov RajDemoPrepSolvePost.mov ] (video)
 +
 +
[[Tutorial_Video_Overviews#RajDemoPrepSolvePost.mov|Video notes of RajDemoPrepSolvePost.mov]] (link to timestamps)
 +
 +
 +
 +
 +
[https://fluid.colorado.edu/tutorials/PrepSolvePostBLandC.mp4 PrepSolvePostBLandC.mp4 ] (video)
 +
 +
[[Tutorial_Video_Overviews#PrepSolvePostBLandC.mp4|Video notes of PrepSolvePostBLandC.mp4]] (link to timestamps)

Latest revision as of 21:21, 14 September 2020

Exporting to PHASTA

After the partitioning performed via Chef in the last steps, we have the problem domain in a form that the PHASTA executable can read. Make a directory named "Run". This will contain all of the simulation data from this case. Remember that for this onRamp tutorial we have partitioned our case of interest to 8 parts. Therefore we will also create a subdirectory named "8-procs-case". In this subdirectory make softlinks to the N=8 restart and geombc and restart "checkpoint" files that where constructed. When in this subdirectory a good command to do this is the following : 

ln -s ../../Chef/8-1-Chef/8-procs-case/restart* .
ln -s ../../Chef/8-1-Chef/8-procs_case/geombc* .

Also create a numstart.dat file. This file will specify the time and timestep that the simulation has completed to. Use the following command in the "Run/8-procs_case" directory : 

echo 0 0>numstart.dat

Build the executable/specify runtime parameters

What remains is to determine the version of PHASTA to build and run. Since there are a bunch of researchers working on PHASTA at any given time, there are many branches/versions of the main code. If you are a part of the research cohort you may be directed to phasta-next, but many of the same steps may be taken from the public version.

Retrieve/build a version of PHASTA code

Navigate to your home directory. Create a directory here named "git-phasta". In a web browser, navigate to the online git repository for phasta-next and select the "clone" or "code" icon. The result should be the simular to the following picture, where a pop-up gives a web address: 

GitClone.png

Copy this address and within the "git-phasta" directory execute the following command and enter your github credentials: 

git clone https://github.com/PHASTA/phasta-next.git

After this is finished their will be a subdirectory created named phasta-next that contains the code tree that you wish to build. Back within the git-phasta directory create another subdirectory named: "build_phasta-next". Now we must set the environment so that the compiler has the necessary libraries. If you perform the following command, the listed environment libraries that ken often uses are listed, and are often sufficient: 

more ~kjansen/soft-core.sh

At the time that this page is created, the relevant commands to load the needed environment are the following: 

soft add +gcc-6.3.0
soft add +openmpi-gnu-1.10.6-gnu49-thread
soft add +simmodeler-6.0-171202

If the user needs additional libraries they can often be found with the following command: 

softenv

lets build this code! Navigate into this build subdirectory and create a file named "unpack_buildFiles.sh" with the following content: 

#!/bin/bash

cmake \
-DCMAKE_C_COMPILER=gcc \
-DCMAKE_CXX_COMPILER=g++ \
-DCMAKE_Fortran_COMPILER=gfortran \
-DCMAKE_BUILD_TYPE=Debug \
-DPHASTA_INCOMPRESSIBLE=ON \
-DPHASTA_COMPRESSIBLE=OFF \
-DPHASTA_USE_LESLIB=ON \
-DLESLIB=/users/matthb2/libles1.5/libles-debianjessie-gcc-ompi.a \
-DCASES=/home/mrasquin/develop-phasta/phastaChefTests \
-DPHASTA_TESTING=OFF \
../phasta-next-master/

This should create the makefiles required for the build. Perform the following command: 

make -j

The following means that you executable has been built: 

Built target phastaIC.exe

Also, bin/phastaIC.exe will now exist. Take note of the path to the build directory for the next steps.


Additional notes

If there is a specific branch off of phasta-next that you'd like to build, navigate to phasta-next and use the following command: 

git checkout "branchname".

If this is a branch that I will be working on for a while, I tend to alter the build and code directories according to the branch-name. Note that the respective pointer in the "unpack_buildFiles.sh" file (ie the last line) will have to be set accordingly.

Create Solver.inp

From your build directory copy input.config to your Run directory containing the 8-procs-case. This file contains all of the default PHASTA runtime input parameters of this version of PHASTA that you have built. In another file named solver.inp in the Run directory you will want to specify all of these parameters that you wish to change for this case. The rest of the parameters need not be specified. For example my solver.inp looks as follows: 

# ibksiz flmpl flmpr itwmod wmodts dmodts fwr taucfct
# PHASTA Version 1.5 Input File
#
#  Basic format is
#
#    Key Phrase  :  Acceptable Value (integer, double, logical, or phrase
#                                     list of integers, list of doubles )
#
#
#SOLUTION CONTROL 
#{                
    Equation of State: Incompressible
    Number of Timesteps: 1
    Time Step Size: 2e-3  # Delt(1)
    Mode to Compute dwal: 0
    Turbulence Model: No-Model  #RANS  # No-Model # DES97 # DDES  iturb=0, RANS =-1  LES=1 #}
#     IDDES Constants: 3.55 0.9
    Ramp Inflow: False #True #False = no jet activated  
#Required for guillermo IC compairison DNS
    Solve Heat: True
#}

#MATERIAL PROPERTIES
#{
    Viscosity: 1.57e-5      # fills datmat (2 values REQUIRED if iLset=1)
    Density: 1.0           # ditto
    Body Force Option: None # ibody=0 => matflag(5,n)
    Body Force: 0 0.0 0.0    # (datmat(i,5,n),i=1,nsd)
    Thermal Conductivity: 27.6e-1  # ditto
    Scalar Diffusivity: 27.6e-1    # fills scdiff(1:nsclrS)
#}

OUTPUT CONTROL
{
    Number of Timesteps between Restarts: 10 #replaces nout/ntout
    Print Error Indicators: False
    Number of Error Smoothing Iterations: 0 # ierrsmooth
    Print ybar: True 
    Print vorticity: True
    Print Wall Fluxes: False
    Print Statistics: False
    Number of Force Surfaces: 1
    Surface ID's for Force Calculation: 1
#     Ranks per core: 4 # for varts only
#     Cores per node: 16 # for varts only
}

#LINEAR SOLVER
# Commented Lines Result from DNS setup
#    Solver Type: ACUSIM with P Projection
#    Number of GMRES Sweeps per Solve: 1      # replaces nGMRES
#    Number of Krylov Vectors per GMRES Sweep: 200           # replaces Kspace
#    Scalar 1 Solver Tolerance : 1.0e-4
#    Tolerance on Momentum Equations: 0.05                   # epstol(1)
    Tolerance on ACUSIM Pressure Projection: 0.01           # prestol 
    Number of Solves per Left-hand-side Formation: 1  #nupdat/LHSupd(1)
    ACUSIM Verbosity Level               : 0   #iverbose
   Minimum Number of ACUSIM Iterations per Nonlinear Iteration: 10  # minIters
#    Maximum Number of ACUSIM Iterations per Nonlinear Iteration: 200 # maxIter
#}

#DISCRETIZATION CONTROL
#{
    Basis Function Order: 1                 # ipord
    Time Integration Rule: Second Order    # Second Order sets rinf next
    Time Integration Rho Infinity: 0.75     # rinf(1) Only used for 2nd order
    Include Viscous Correction in Stabilization: False    # if p=1 idiff=1
                                                          # if p=2 idiff=2  
    Quadrature Rule on Interior: 2           #int(1)
    Quadrature Rule on Boundary: 2           #intb(1)
    Lumped Mass Fraction on Left-hand-side: 0.0           # flmpl
    Lumped Mass Fraction on Right-hand-side: 0.0          # flmpr
#     Tau Matrix: Diagonal-Franca               #itau=1
    Tau Time Constant: 16.                      #dtsfct
    Tau C Scale Factor: .1                    # taucfct  best value depends  
    Number of Elements Per Block: 32 # switch to >250 if sgi        #ibksiz
#} 

TURBULENCE MODELING PARAMETERS
{
#   Dynamic Model Type : Standard   # adds zero to iturb       LES
#        Filter Integration Rule: 1  #ifrule adds ifrule-1 to iturb LES
#   Turbulence Wall Model Type: Effective Viscosity  #itwmod=2 RANSorLES
#   Velocity Averaging Steps : 500. # wtavei= 1/this           RANSorLES
#   Dynamic Model Averaging Steps : 500. # dtavei= 1/this      LES
#   Filter Width Ratio        : 3.  # fwr1                     LES
    }
#
#STEP SEQUENCE 
#{
     Step Construction  : 0 1 0 1 5 6 5 6 5 6   # this is the standard two iteration
#}


Running the Solver

Create the runPHASTA.sh in your run directory. Where you see <pathtoBuildDir> include your path to your build directory where you retrieved the input.config file. The #pconf line is an example that I have used myself, yours should look simular. 

#/bin/bash
# VER=C or VER=IC should be specified
#pconf=/users/jopa6460/git-phasta/build_JohnDev-STG_D; VER=IC
pconf=<pathtoBuildDir>; VER=IC

date
mpirun $2 -np $1  -x PHASTA_CONFIG=$pconf $pconf/bin/phasta${VER}.exe
date

Now you have everything you need to run your first PHASTA simulation! In the Run directory execute the following line: 

./runPHASTA.sh 8

Output of a proper run of PHASTA contains information on 1)the step number the simulation is on 2) the relative residual (convergence relative to this run's initial residual), 3) various other outputs based on runtime parameters (like BC and IC info). A single time iteration with step construction 1 0 10 11 1 0 10 11 has the following output:

Where


As the simulation continues it will produce successive checkpoint restart files that contain the solution data at that timestep. These and the geombc files will be what is post-proccessed in Paraview.


A few helpful video tutorials

CloneFromGithubAndBranch.mov (video)

Video notes of CloneFromGithubAndBranch.mov (link to timestamps)



BlancoBuidingPHASTA.mov (video)

Video notes of BlancoBuidingPHASTA.mov (link to timestamps)



PHASTA_workflow_RB.mkv (video)

Video notes of PHASTA_workflow_RB.mkv (link to timestamps)



RajDemoPrepSolvePost.mov (video)

Video notes of RajDemoPrepSolvePost.mov (link to timestamps)



PrepSolvePostBLandC.mp4 (video)

Video notes of PrepSolvePostBLandC.mp4 (link to timestamps)

Retrieved from "https://fluid.colorado.edu/wiki/index.php?title=Solve&oldid=1154"