Work Plan In Memory Adapt

Summary of Phases

• phase0 wrapper layer that supports both syncio and posix - done 7/20/2015
• phase1 PHASTA can read AND write both - done 7/22/2015
• phase2 PHASTA runs after chef though an in-memory read of Chef's written in-memory data
• phase3 PHASTA writes its field back to chef and lets chef do a UR on its mesh and transfers fields
• phase4 bring that result back to phasta (so chef->PHASTA->chef-UR->PHASTA-UR)
• phase5 replace UR with local adaptation
• phase6 loop and we are done

phastaIO

For the sake of discussion lets group phastaIO functions as follows

init
open-close
read  (readheader, readdatablock)
write  (writeheader, writedatablock)
initalize

To make the interface more simple, I propose, we push init into open and finalize into close. Furthermore I propose that we consider a wrapper layer approach where the functions that the PHASTA workflow directly uses to perform IO are:

• phio_openfile_read (in addition to specializing filename string for particular library, this function can call queryphmpiio and initphmpiio in the case of syncio)
• phio_openfile_write (in addition to specializing filename string for particular library, this function can call initphmpiio in the case of syncio)
• phio_readheader
• phio_readdatablock
• phio_writeheader
• phio_writedatablock
• phio_closefile

These functions will be called by readnblk (and a few others) for reading and Write_restart (and a few clones) for writing. Principally, these 6 functions will be wrappers that take in partial strings and the usual arguments for their PhastaIO functions and then perform the necessary string specializations for the particular flavor of PhastaIO being used (e.g,, syncio or posix) before then ultimately calling the PhastaIO function (that already has a branching based on the two flavors). Cameron has proposed housing those seven wrappers in phIO.c which seems fine to me. Helper functions can go in the same file.

Initially, setIOparam() counting of syncio files will be used to determine if sync or posix files will be used. If the count is zero then assume posix. Use the count as a switch for file format everywhere (can be replaced later by agreed upon flag for file format...candidates: iotype= binary-posix, binary-mpiio instead of binary (not sure we want to preserve ascii unless it easy to do)

Detailed Description of Phases

1. phase0
   1. Move field string construction into C)- status: CWS has done this for syncio (top of routines so not yet in a syncio conditional but this is not necessary until phase1)
   2. Move file name construction into B)- status: not started but I think CWS indicated it was basically same work as 0.1 so leaving that with him. Note we don’t want this in init (at least for restarts) because we init only once but possibly write multiple times and the filename changes.
      1. Move query and init into B)
   3. Migrating all of PHASTA to use 0.1 dev - status: CWS did it for readnblk.f and this built and ran (YAY), KEJ is migrating it to the balance of PHASTA (DONE I THINK)
   4. Repeat 0.1-0.3 for writing of files
2. phase1
   1. Migrate string constructors into conditional for syncio
   2. Create posix counterparts and put them in the posix branch of conditional
   3. make init and finalize functions that have a universal call but, via conditionals (likely posix just returns but syncio has stuff to do but we want the PHASTA side to look the same regardless of IO chosen)
   4. Debug to be sure that at least all tet meshes can be read with both posix/syncio
   5. Debug to make sure that files written are correct (two successive runs)
3. phase2
   1. 2.1 Create ability for Chef to write to memory location using posixIO pointed at a memory location instead of a file
   2. 2.2 Create ability to pass the descriptor of that memory location to PHASTA (I think it is safe to assume that since we are one “file” per part in both Chef and PHASTA that this is just the starting location of where Chef started writing)
   3. 2.3 Create ability to have phasta initialize itself with posix-based PHASTAIO reads from this in-memory file/state that Chef wrote
4. phase3
   1. Create ability for PHASTA to write adapt-fields (solution, d2wall, error, ybar) using posix-based PHASTAIO writes to a memory location
   2. Create ability to pass the descriptor of that memory location back to Chef
   3. Create ability for Chef to read adapt-fields when passed descriptor
   4. Invoke a UR with solution transfer and write files to confirm phase3 success
5. phase4
   1. Should be trivial but have Chef write result of UR to a new memory location and pass descriptor to PHASTA to read and run again (comparing to file based version of the same which we are already doing at scale)
6. phase5
   1. Demonstrate that we can do file based local adaptive refinement with Chef (I will let Dan and Cameron described the subtask developments here but am more than happy to help plan and assist in any way I can because we really need this capability…I am fine with ignoring/freezing boundary layers to get a working case going before taking on that much tougher nut).

Thorny Issues

1. phase0
   1. extra write files. Seems to me that there should be either
      1. only 2 write_restart and write_field, or
      2. an init function and then only write_field
   2. dwall currently has two modes (in restart or in a separate file….do we really need to preserve this)
   3. usr.c opens restart to read saved projection vectors. Not a problem per se but might be better to read this while we have the restart already open so that the IO code is more compact. This may involve moving more leslib stuff forward in the code
   4. KEJ is still trying to sort out the roles of {query,init}phmpiio.
      1. query is called before file is read the first time (same file read again can get by without it e.g., readlesprojection reads restart a second time)
      2. init is called for both reads and writes of syncio files