1# The PETSc Kernel 2 3PETSc provides a variety of basic services for writing scalable, 4component-based libraries; these are referred to as the PETSc kernel {cite}`bgms98`. 5The source code for the kernel is in `src/sys`. It contains systematic 6support for 7 8- managing PETSc types, 9- error handling, 10- memory management, 11- profiling, 12- object management, 13- Fortran interfaces (see {cite}`balaybrownknepleymcinnessmith2015`) 14- mechanism for generating appropriate citations for algorithms and software used in PETSc (see {cite}`knepley2013accurately`) 15- file I/O, 16- an options database, and 17- objects and code for viewing, drawing, and displaying data and solver objects. 18 19Each of these is discussed in a section below. 20 21## PETSc Types 22 23For maximum flexibility, the basic data types `int`, `double`, and 24so on are not used in PETSc source code. Rather, it has 25 26- `PetscScalar`, 27- `PetscInt`, 28- `PetscMPIInt`, 29- `PetscBLASInt`, 30- `PetscBool`, and 31- `PetscBT` - bit storage of logical true and false. 32 33`PetscInt` can be set using `configure` to be either `int` (32 34bit, the default) or `long long` (64-bit, with 35`configure –with-64-bit-indices`) to allow indexing into very large 36arrays. `PetscMPIInt` is used for integers passed to MPI as counts and 37sizes. These are always `int` since that is what the MPI standard 38uses. Similarly, `PetscBLASInt` is for counts, and so on passed to 39BLAS and LAPACK routines. These are almost always `int` unless one is 40using a special “64-bit integer” BLAS/LAPACK (this is available, for 41example, with Intel’s MKL and OpenBLAS). 42 43In addition, there are special types: 44 45- `PetscClassId` 46- `PetscErrorCode` 47- `PetscLogEvent` 48 49These are currently always `int`, but their use clarifies the code. 50 51## Implementation of Error Handling 52 53PETSc uses a “call error handler; then (depending on result) return 54error code” model when problems are detected in the running code. The 55public include file for error handling is 56<a href="PETSC_DOC_OUT_ROOT_PLACEHOLDER/include/petscerror.h.html">include/petscerror.h</a> 57and the source code for the PETSc error handling is in 58`src/sys/error/`. 59 60### Simplified Interface 61 62The simplified macro-based interface consists of the following two 63calls: 64 65- `SETERRQ(comm,error code,Error message);` 66- `PetscCall(ierr);` 67 68The macro `SETERRQ()` is given by 69 70```{literalinclude} /../include/petscerror.h 71:end-at: '#define SETERRQ' 72:language: c 73:start-at: '#define SETERRQ' 74``` 75 76It calls the error handler with the current function name and location: 77line number, and file, plus an error code and an error message. Normally 78`comm` is `PETSC_COMM_SELF`; it can be another communicator only if 79one is absolutely sure the same error will be generated on all processes 80in the communicator. This feature is to prevent the same error message 81from being printed by many processes. 82 83The macro `PetscCall()` is defined by 84 85```{literalinclude} /../include/petscerror.h 86:end-at: '#define PetscCall' 87:language: c 88:start-at: '#define PetscCall' 89``` 90 91The message passed to `SETERRQ()` is treated as a `printf()`-style 92format string, with all additional parameters passed after the string as 93its arguments. For example: 94 95``` 96SETERRQ(comm,PETSC_ERR,"Iteration overflow: its %" PetscInt_FMT " norm %g",its,(double)norm); 97``` 98 99### Error Handlers 100 101The error-handling function `PetscError()` calls the “current” error 102handler with the code 103 104```{literalinclude} /../src/sys/error/err.c 105:append: '}' 106:end-at: PetscFunctionReturn 107:language: c 108:start-at: PetscErrorCode PetscError( 109``` 110 111You can set a new error handler with the command 112`PetscPushErrorHandler()`, which maintains a linked list of error 113handlers. The most recent error handler is removed via 114`PetscPopErrorHandler()`. 115 116PETSc provides several default error handlers: 117 118- `PetscTraceBackErrorHandler()`, the default; 119- `PetscAbortErrorHandler()`, called with `-onerrorabort`, useful when running in the debugger; 120- `PetscReturnErrorHandler()`, which returns up the stack without printing error messages; 121- `PetscEmacsClientErrorHandler()`; 122- `PetscMPIAbortErrorHandler()`, which calls `MPIAbort()` after printing the error message; and 123- `PetscAttachDebuggerErrorHandler()`, called with `-onerrorattachdebugger`. 124 125### Error Codes 126 127The PETSc error handler takes an error code. The generic error codes are 128defined in 129<a href="PETSC_DOC_OUT_ROOT_PLACEHOLDER/include/petscerror.h.html">include/petscerror.h</a> 130The same error code is used many times in the libraries. For example, 131the error code `PETSCERRMEM` is used whenever a requested memory 132allocation is not available. 133 134### Detailed Error Messages 135 136In a modern parallel component-oriented application code, it does not 137always make sense to simply print error messages to the terminal (and 138more than likely there is no “terminal”, for example, with Microsoft 139Windows or Apple iPad applications). PETSc provides the replaceable 140function pointer 141 142``` 143(*PetscErrorPrintf)("Format",...); 144``` 145 146which, by default, prints to standard out. Thus, error messages should 147not be printed with `printf()` or `fprintf()`. Rather, they should 148be printed with `(*PetscErrorPrintf)()`. You can direct all error 149messages to `stderr`, instead of the default `stdout`, with the 150command line option `-erroroutputstderr`. 151 152### C++ Exceptions 153 154In PETSc code, when one calls C++ functions that do not return with an error code but might 155instead throw C++ exceptions, one can use `CHKERRCXX(func)`, which catches the exceptions 156in *func* and then calls `SETERRQ()`. The macro `CHKERRCXX(func)` is given by 157 158```{literalinclude} /../include/petscerror.h 159:end-at: '#define CHKERRCXX' 160:language: c 161:start-at: '#define CHKERRCXX' 162``` 163 164## Memory Management 165 166PETSc provides simple wrappers for the system `malloc(), calloc()`, 167and `free()` routines. The public interface for these is provided in 168`petscsys.h`, while the implementation code is in `src/sys/memory`. 169The most basic interfaces are 170 171```{literalinclude} /../include/petscsys.h 172:end-at: '#define PetscMalloc' 173:language: c 174:start-at: '#define PetscMalloc' 175``` 176 177```{literalinclude} /../include/petscsys.h 178:end-at: '#define PetscFree' 179:language: c 180:start-at: '#define PetscFree' 181``` 182 183```{literalinclude} /../src/sys/memory/mal.c 184:append: '}' 185:end-at: PetscFunctionReturn(PETSC_SUCCESS) 186:language: c 187:start-at: PetscErrorCode PetscMallocA( 188``` 189 190```{literalinclude} /../src/sys/memory/mal.c 191:append: '}' 192:end-at: PetscFunctionReturn(PETSC_SUCCESS) 193:language: c 194:start-at: PetscErrorCode PetscFreeA( 195``` 196 197which allow the use of any number of profiling and error-checking 198wrappers for `malloc(), calloc()`, and `free()`. Both 199`PetscMallocA()` and `PetscFreeA()` call the function pointer values 200`(*PetscTrMalloc)` and `(*PetscTrFree)`. `PetscMallocSet()` is 201used to set these function pointers. The functions are guaranteed to 202support requests for zero bytes of memory correctly. Freeing memory 203locations also sets the pointer value to zero, preventing later code 204from accidentally using memory that has been freed. All PETSc memory 205allocation calls are memory aligned on at least double-precision 206boundaries; the macro generated by configure `PETSCMEMALIGN` indicates 207in bytes what alignment all allocations have. This can be controlled at 208configure time with the option `-with-memalign=<4,8,16,32,64>`. 209 210`PetscMallocA()` supports a request for up to 7 distinct memory 211locations of possibly different types. This serves two purposes: it 212reduces the number of system `malloc()` calls, thus potentially 213increasing performance, and it clarifies in the code related memory 214allocations that should be freed together. 215 216The following macros are the preferred way to obtain and release memory 217in the PETSc source code. They automatically manage calling 218`PetscMallocA()` and `PetscFreeA()` with the appropriate location 219information. 220 221```{literalinclude} /../include/petscsys.h 222:end-at: '#define PetscMalloc1' 223:language: c 224:start-at: '#define PetscMalloc1' 225``` 226 227```{literalinclude} /../include/petscsys.h 228:end-at: '#define PetscMalloc2' 229:language: c 230:start-at: '#define PetscMalloc2' 231``` 232 233... 234 235```{literalinclude} /../include/petscsys.h 236:end-at: '#define PetscMalloc7' 237:language: c 238:start-at: '#define PetscMalloc7' 239``` 240 241```{literalinclude} /../include/petscsys.h 242:end-at: '#define PetscFree' 243:language: c 244:start-at: '#define PetscFree' 245``` 246 247```{literalinclude} /../include/petscsys.h 248:end-at: '#define PetscFree2' 249:language: c 250:start-at: '#define PetscFree2' 251``` 252 253... 254 255```{literalinclude} /../include/petscsys.h 256:end-at: '#define PetscFree7' 257:language: c 258:start-at: '#define PetscFree7' 259``` 260 261Similar routines, `PetscCalloc1()` to `PetscCalloc7()`, provide 262memory initialized to zero. The size requests for these macros are in 263number of data items requested, not in bytes. This decreases the number 264of errors in the code since the compiler determines their sizes from the 265object type instead of requiring the user to provide the correct value 266with `sizeof()`. 267 268The routines `PetscTrMallocDefault()` and `PetscTrFreeDefault()`, 269which are set with the routine `PetscSetUseTrMallocPrivate()` (and are 270used by default for the debug version of PETSc), provide simple logging 271and error checking versions of memory allocation. 272 273## Implementation of Profiling 274 275This section provides details about the implementation of event logging 276and profiling within the PETSc kernel. The interface for profiling in 277PETSc is contained in the file 278<a href="PETSC_DOC_OUT_ROOT_PLACEHOLDER/include/petsclog.h.html">include/petsclog.h</a> 279The source code for the profile logging is in `src/sys/plog/`. 280 281### Profiling Object Creation and Destruction 282 283The creation of objects is profiled with the command 284`PetscLogObjectCreate()` 285 286``` 287PetscLogObjectCreate(PetscObject h); 288``` 289 290which logs the creation of any PETSc object. Just before an object is 291destroyed, it should be logged with `PetscLogObjectDestroy()` 292 293``` 294PetscLogObjectDestroy(PetscObject h); 295``` 296 297These are called automatically by `PetscHeaderCreate()` and 298`PetscHeaderDestroy()`, which are used in creating all objects 299inherited from the basic object. Thus, these logging routines need never 300be called directly. 301 302It is also useful to log information about the state of an object, as 303can be done with the command 304 305``` 306PetscLogObjectState(PetscObject h,const char *format,...); 307``` 308 309For example, for sparse matrices we usually log the matrix dimensions 310and number of nonzeros. 311 312### Profiling Events 313 314Events are logged by using the pair `PetscLogEventBegin()` and `PetscLogEventEnd()`. 315 316This logging is usually done in the abstract interface file for the 317operations, for example, 318<a href="PETSC_DOC_OUT_ROOT_PLACEHOLDER/src/mat/interface/matrix.c.html">src/mat/interface/matrix.c</a> 319 320### Controlling Profiling 321 322Routines that control the default profiling available in PETSc include 323the following 324 325- `PetscLogDefaultBegin();` 326- `PetscLogAllBegin();` 327- `PetscLogDump(const char *filename);` 328- `PetscLogView(PetscViewer);` 329 330These routines are normally called by the `PetscInitialize()` and 331`PetscFinalize()` routines when the option `-logview` is given. 332 333## References 334 335```{bibliography} /petsc.bib 336:filter: docname in docnames 337``` 338