PHASTA/Reduce Restart and Geombc Files

Reduction from M to N parts in parallel for PHASTA files initially partitioned with NSpre_parallel, phParAdapt-SCOREC or chef

Prerequisites

The tool described in this section apply to phasta files initially partitioned with phParAdapt-SCOREC or chef. We consider here that M parts were generated from N parts with M>N (M is usually a multiple of N). A vertex mapping between the M parts and N parts needs to be generated by phParAdapt-SCOREC or chef through the variable "buildMapping 1" in adapt.inp.

Building M2N and M2NFixBnd

• Download the M2N and M2NFixBdn packages from

 cd your-develop-dir-cmake/phasta
 git clone ssh://username@jumpgate-phasta.colorado.edu/users/mrasquin/develop-FlowControl-SyncIO-Cmake/phasta/M2N  
 git clone ssh://username@jumpgate-phasta.colorado.edu/users/mrasquin/develop-FlowControl-SyncIO-Cmake/phasta/M2NFixBnd

• Build both packages with cmake. For instance, on the viz nodes with gcc:

 cd your-develop-dir-cmake

 mkdir build-M2N
 cd build-M2N
 CC=gcc CXX=g++ FC=gfortran cmake -DCMAKE_BUILD_TYPE=Release ../phasta/M2N/src
 make -j 8 VERBOSE=1
 cd ..

 mkdir build-M2NFixBnd
 cd build-M2NFixBnd
 CC=gcc CXX=g++ FC=gfortran cmake -DCMAKE_BUILD_TYPE=Release ../phasta/M2NFixBnd/src
 make -j 8 VERBOSE=1
 cd ..

M2N

Running M2N

M2N must be run on M processes

 mpirun -np M  your-develop-dir-cmake/build-M2N/M2N

M2N Inputs

M2N reads:

• The geombc files files under the SyncIO format (M parts);
• The solution, dwal, error and ybar fields from a set of restart files under the SyncIO format (M parts) for adaptation or post-processing later;
• The vertex mapping field from a set of restart files under the SyncIO format (M parts).
• An input file named "M2N_input.dat" described below

M2N Outputs

M2N produces:

• N restart files under the Posix format, which is what phParAdapt requires for now.

M2N_input.dat

Note that usually, the vertex mapping field is included in the restart files of the first time step of your computation, whereas the solution, dwal, error and dwal come from the last time step of the computation.

The "M2N_input.dat" must include the following information:

 120  # time step of the restart files for solution, dwal, error and ybar.  It can be very useful when processing many files on queued systems to 
         # replace this number (120)  by Step2Red and then use sed in the submit script to replace it with a number entered on the command line 
         # e.g., Step2Red=$4
         #         sed "s/Step2Red/${Step2Red}/g" M2N_inputSed.dat > M2N_input.dat    
         # for easy replacement just ahead of execution and $4 is the 4th argument on the submit script passed to the run script with this insertion
 100  # time step of the restart files for partID and vtxID (mapping between M and N)
 0    # Reduce method: 0 = communicatoin based approach (recommended), 1 = file based approach. 
 1    # (For reduce method = 1 only): 0 = ascii files, 1 = binary files
 1    # Print debug statement if = 1, none if = 0 (recommended at scale)

M2NFixBnd

The value of the solution, dwal, errors and ybar fields still need to be updated on the part boundaries of the N parts, which is the purpose of M2NFixBnd

Running M2NFixBnd

M2NFixBnd is run on N processes.

   mpirun -np N  your-develop-dir-cmake/build-M2NFixBnd/M2NFixBnd

M2NFixBnd Inputs

M2NFixBnd reads

• The solution, dwal, errors and ybars field from the N restart files under the Posix format produced by M2N in the previous step;
• The ilwork array from the N geombc files under the Posix format that matches the restart files produced by M2N.

M2NFixBnd Outputs

M2NFixBnd produces

• N restart files under the posix format that overwrite the ones generated by M2N.

Example

See

 /users/mrasquin/Models/CrossFlowTest/1-A0-Mixed-9.0/SyncIO-phPA/Runs-pumi-Tet-ts100/ReduceM2N-16TetPumi-8MixedSim/runReduceM2N.sh

Reduction from M parts to 1 part in serial for phasta files initially partitioned with NSpre_parallel

• cd to the procs_case directory that contains the files you want to reduce
• Run the executable:

  ~mrasquin/develop/phasta/phPost/phPost/bin/x86_64_linux-icc/reduce-openmpi-O -sn

where sn is the time step number. For example, the restart files corresponding to time step 5 would be restart.5.* and geombc.dat.* .

• The reduce operation will generate restart.##.0 in the procs_case directory, where ## stands for the step number you have reduced.
• restart.##.0, combined with geombc.dat.1 (which contains the boundary conditions for all the parts - it is created when you run NSPre_Parasolid), are what you need to see the solution on 1 part.