1# libCEED: Efficient Extensible Discretization 2 3[![GitHub Actions][github-badge]][github-link] 4[![GitLab-CI][gitlab-badge]][gitlab-link] 5[![Code coverage][codecov-badge]][codecov-link] 6[![BSD-2-Clause][license-badge]][license-link] 7[![Documentation][doc-badge]][doc-link] 8[![JOSS paper][joss-badge]][joss-link] 9[![Binder][binder-badge]][binder-link] 10 11## Summary and Purpose 12 13libCEED provides fast algebra for element-based discretizations, designed for performance portability, run-time flexibility, and clean embedding in higher level libraries and applications. 14It offers a C99 interface as well as bindings for Fortran, Python, Julia, and Rust. 15While our focus is on high-order finite elements, the approach is mostly algebraic and thus applicable to other discretizations in factored form, as explained in the [user manual](https://libceed.org/en/latest/) and API implementation portion of the [documentation](https://libceed.org/en/latest/api/). 16 17One of the challenges with high-order methods is that a global sparse matrix is no longer a good representation of a high-order linear operator, both with respect to the FLOPs needed for its evaluation, as well as the memory transfer needed for a matvec. 18Thus, high-order methods require a new "format" that still represents a linear (or more generally non-linear) operator, but not through a sparse matrix. 19 20The goal of libCEED is to propose such a format, as well as supporting implementations and data structures, that enable efficient operator evaluation on a variety of computational device types (CPUs, GPUs, etc.). 21This new operator description is based on algebraically [factored form](https://libceed.org/en/latest/libCEEDapi/#finite-element-operator-decomposition), which is easy to incorporate in a wide variety of applications, without significant refactoring of their own discretization infrastructure. 22 23The repository is part of the [CEED software suite](http://ceed.exascaleproject.org/software/), a collection of software benchmarks, miniapps, libraries and APIs for efficient exascale discretizations based on high-order finite element and spectral element methods. 24See <http://github.com/ceed> for more information and source code availability. 25 26The CEED research is supported by the [Exascale Computing Project](https://exascaleproject.org/exascale-computing-project) (17-SC-20-SC), a collaborative effort of two U.S. Department of Energy organizations (Office of Science and the National Nuclear Security Administration) responsible for the planning and preparation of a [capable exascale ecosystem](https://exascaleproject.org/what-is-exascale), including software, applications, hardware, advanced system engineering and early testbed platforms, in support of the nation’s exascale computing imperative. 27 28For more details on the CEED API see the [user manual](https://libceed.org/en/latest/). 29 30% gettingstarted-inclusion-marker 31 32## Building 33 34The CEED library, `libceed`, is a C99 library with no required dependencies, and with Fortran, Python, Julia, and Rust interfaces. 35It can be built using: 36 37```console 38$ make 39``` 40 41or, with optimization flags: 42 43```console 44$ make OPT='-O3 -march=skylake-avx512 -ffp-contract=fast' 45``` 46 47These optimization flags are used by all languages (C, C++, Fortran) and this makefile variable can also be set for testing and examples (below). 48 49The library attempts to automatically detect support for the AVX instruction set using gcc-style compiler options for the host. 50Support may need to be manually specified via: 51 52```console 53$ make AVX=1 54``` 55 56or: 57 58```console 59$ make AVX=0 60``` 61 62if your compiler does not support gcc-style options, if you are cross compiling, etc. 63 64To enable CUDA support, add `CUDA_DIR=/opt/cuda` or an appropriate directory to your `make` invocation. 65To enable HIP support, add `ROCM_DIR=/opt/rocm` or an appropriate directory. 66To enable SYCL support, add `SYCL_DIR=/opt/sycl` or an appropriate directory. 67Note that SYCL backends require building with oneAPI compilers as well: 68```console 69$ . /opt/intel/oneapi/setvars.sh 70$ make SYCL_DIR=/opt/intel/oneapi/compiler/latest/linux SYCLCXX=icpx CC=icx CXX=icpx 71``` 72 73To store these or other arguments as defaults for future invocations of `make`, use: 74 75```console 76$ make configure CUDA_DIR=/usr/local/cuda ROCM_DIR=/opt/rocm OPT='-O3 -march=znver2' 77``` 78 79which stores these variables in `config.mk`. 80 81### WebAssembly 82 83libCEED can be built for WASM using [Emscripten](https://emscripten.org). For example, one can build the library and run a standalone WASM executable using 84 85``` console 86$ emmake make build/ex2-surface.wasm 87$ wasmer build/ex2-surface.wasm -- -s 200000 88``` 89 90## Additional Language Interfaces 91 92The Fortran interface is built alongside the library automatically. 93 94Python users can install using: 95 96```console 97$ pip install libceed 98``` 99 100or in a clone of the repository via `pip install .`. 101 102Julia users can install using: 103 104```console 105$ julia 106julia> ] 107pkg> add LibCEED 108``` 109 110See the [LibCEED.jl documentation](http://ceed.exascaleproject.org/libCEED-julia-docs/dev/) for more information. 111 112Rust users can include libCEED via `Cargo.toml`: 113 114```toml 115[dependencies] 116libceed = "0.11.0" 117``` 118 119See the [Cargo documentation](https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html#specifying-dependencies-from-git-repositories) for details. 120 121## Testing 122 123The test suite produces [TAP](https://testanything.org) output and is run by: 124 125```console 126$ make test 127``` 128 129or, using the `prove` tool distributed with Perl (recommended): 130 131```console 132$ make prove 133``` 134 135## Backends 136 137There are multiple supported backends, which can be selected at runtime in the examples: 138 139| CEED resource | Backend | Deterministic Capable | 140| :--- | :--- | :---: | 141|| 142| **CPU Native** | 143| `/cpu/self/ref/serial` | Serial reference implementation | Yes | 144| `/cpu/self/ref/blocked` | Blocked reference implementation | Yes | 145| `/cpu/self/opt/serial` | Serial optimized C implementation | Yes | 146| `/cpu/self/opt/blocked` | Blocked optimized C implementation | Yes | 147| `/cpu/self/avx/serial` | Serial AVX implementation | Yes | 148| `/cpu/self/avx/blocked` | Blocked AVX implementation | Yes | 149|| 150| **CPU Valgrind** | 151| `/cpu/self/memcheck/*` | Memcheck backends, undefined value checks | Yes | 152|| 153| **CPU LIBXSMM** | 154| `/cpu/self/xsmm/serial` | Serial LIBXSMM implementation | Yes | 155| `/cpu/self/xsmm/blocked` | Blocked LIBXSMM implementation | Yes | 156|| 157| **CUDA Native** | 158| `/gpu/cuda/ref` | Reference pure CUDA kernels | Yes | 159| `/gpu/cuda/shared` | Optimized pure CUDA kernels using shared memory | Yes | 160| `/gpu/cuda/gen` | Optimized pure CUDA kernels using code generation | No | 161|| 162| **HIP Native** | 163| `/gpu/hip/ref` | Reference pure HIP kernels | Yes | 164| `/gpu/hip/shared` | Optimized pure HIP kernels using shared memory | Yes | 165| `/gpu/hip/gen` | Optimized pure HIP kernels using code generation | No | 166|| 167| **SYCL Native** | 168| `/gpu/sycl/ref` | Reference pure SYCL kernels | Yes | 169| `/gpu/sycl/shared` | Optimized pure SYCL kernels using shared memory | Yes | 170|| 171| **MAGMA** | 172| `/gpu/cuda/magma` | CUDA MAGMA kernels | No | 173| `/gpu/cuda/magma/det` | CUDA MAGMA kernels | Yes | 174| `/gpu/hip/magma` | HIP MAGMA kernels | No | 175| `/gpu/hip/magma/det` | HIP MAGMA kernels | Yes | 176|| 177| **OCCA** | 178| `/*/occa` | Selects backend based on available OCCA modes | Yes | 179| `/cpu/self/occa` | OCCA backend with serial CPU kernels | Yes | 180| `/cpu/openmp/occa` | OCCA backend with OpenMP kernels | Yes | 181| `/cpu/dpcpp/occa` | OCCA backend with DPC++ kernels | Yes | 182| `/gpu/cuda/occa` | OCCA backend with CUDA kernels | Yes | 183| `/gpu/hip/occa` | OCCA backend with HIP kernels | Yes | 184 185The `/cpu/self/*/serial` backends process one element at a time and are intended for meshes with a smaller number of high order elements. 186The `/cpu/self/*/blocked` backends process blocked batches of eight interlaced elements and are intended for meshes with higher numbers of elements. 187 188The `/cpu/self/ref/*` backends are written in pure C and provide basic functionality. 189 190The `/cpu/self/opt/*` backends are written in pure C and use partial e-vectors to improve performance. 191 192The `/cpu/self/avx/*` backends rely upon AVX instructions to provide vectorized CPU performance. 193 194The `/cpu/self/memcheck/*` backends rely upon the [Valgrind](http://valgrind.org/) Memcheck tool to help verify that user QFunctions have no undefined values. 195To use, run your code with Valgrind and the Memcheck backends, e.g. `valgrind ./build/ex1 -ceed /cpu/self/ref/memcheck`. 196A 'development' or 'debugging' version of Valgrind with headers is required to use this backend. 197This backend can be run in serial or blocked mode and defaults to running in the serial mode if `/cpu/self/memcheck` is selected at runtime. 198 199The `/cpu/self/xsmm/*` backends rely upon the [LIBXSMM](http://github.com/hfp/libxsmm) package to provide vectorized CPU performance. 200If linking MKL and LIBXSMM is desired but the Makefile is not detecting `MKLROOT`, linking libCEED against MKL can be forced by setting the environment variable `MKL=1`. 201 202The `/gpu/cuda/*` backends provide GPU performance strictly using CUDA. 203 204The `/gpu/hip/*` backends provide GPU performance strictly using HIP. 205They are based on the `/gpu/cuda/*` backends. 206ROCm version 4.2 or newer is required. 207 208The `/gpu/sycl/*` backends provide GPU performance strictly using SYCL. 209They are based on the `/gpu/cuda/*` and `/gpu/hip/*` backends. 210 211The `/gpu/*/magma/*` backends rely upon the [MAGMA](https://bitbucket.org/icl/magma) package. 212To enable the MAGMA backends, the environment variable `MAGMA_DIR` must point to the top-level MAGMA directory, with the MAGMA library located in `$(MAGMA_DIR)/lib/`. 213By default, `MAGMA_DIR` is set to `../magma`; to build the MAGMA backends with a MAGMA installation located elsewhere, create a link to `magma/` in libCEED's parent directory, or set `MAGMA_DIR` to the proper location. 214MAGMA version 2.5.0 or newer is required. 215Currently, each MAGMA library installation is only built for either CUDA or HIP. 216The corresponding set of libCEED backends (`/gpu/cuda/magma/*` or `/gpu/hip/magma/*`) will automatically be built for the version of the MAGMA library found in `MAGMA_DIR`. 217 218Users can specify a device for all CUDA, HIP, and MAGMA backends through adding `:device_id=#` after the resource name. 219For example: 220 221> - `/gpu/cuda/gen:device_id=1` 222 223The `/*/occa` backends rely upon the [OCCA](http://github.com/libocca/occa) package to provide cross platform performance. 224To enable the OCCA backend, the environment variable `OCCA_DIR` must point to the top-level OCCA directory, with the OCCA library located in the `${OCCA_DIR}/lib` (By default, `OCCA_DIR` is set to `../occa`). 225OCCA version 1.4.0 or newer is required. 226 227Users can pass specific OCCA device properties after setting the CEED resource. 228For example: 229 230> - `"/*/occa:mode='CUDA',device_id=0"` 231 232Bit-for-bit reproducibility is important in some applications. 233However, some libCEED backends use non-deterministic operations, such as `atomicAdd` for increased performance. 234The backends which are capable of generating reproducible results, with the proper compilation options, are highlighted in the list above. 235 236## Examples 237 238libCEED comes with several examples of its usage, ranging from standalone C codes in the `/examples/ceed` directory to examples based on external packages, such as MFEM, PETSc, and Nek5000. 239Nek5000 v18.0 or greater is required. 240 241To build the examples, set the `MFEM_DIR`, `PETSC_DIR`, and `NEK5K_DIR` variables and run: 242 243```console 244$ cd examples/ 245``` 246 247% running-examples-inclusion-marker 248 249```console 250# libCEED examples on CPU and GPU 251$ cd ceed/ 252$ make 253$ ./ex1-volume -ceed /cpu/self 254$ ./ex1-volume -ceed /gpu/cuda 255$ ./ex2-surface -ceed /cpu/self 256$ ./ex2-surface -ceed /gpu/cuda 257$ cd .. 258 259# MFEM+libCEED examples on CPU and GPU 260$ cd mfem/ 261$ make 262$ ./bp1 -ceed /cpu/self -no-vis 263$ ./bp3 -ceed /gpu/cuda -no-vis 264$ cd .. 265 266# Nek5000+libCEED examples on CPU and GPU 267$ cd nek/ 268$ make 269$ ./nek-examples.sh -e bp1 -ceed /cpu/self -b 3 270$ ./nek-examples.sh -e bp3 -ceed /gpu/cuda -b 3 271$ cd .. 272 273# PETSc+libCEED examples on CPU and GPU 274$ cd petsc/ 275$ make 276$ ./bps -problem bp1 -ceed /cpu/self 277$ ./bps -problem bp2 -ceed /gpu/cuda 278$ ./bps -problem bp3 -ceed /cpu/self 279$ ./bps -problem bp4 -ceed /gpu/cuda 280$ ./bps -problem bp5 -ceed /cpu/self 281$ ./bps -problem bp6 -ceed /gpu/cuda 282$ cd .. 283 284$ cd petsc/ 285$ make 286$ ./bpsraw -problem bp1 -ceed /cpu/self 287$ ./bpsraw -problem bp2 -ceed /gpu/cuda 288$ ./bpsraw -problem bp3 -ceed /cpu/self 289$ ./bpsraw -problem bp4 -ceed /gpu/cuda 290$ ./bpsraw -problem bp5 -ceed /cpu/self 291$ ./bpsraw -problem bp6 -ceed /gpu/cuda 292$ cd .. 293 294$ cd petsc/ 295$ make 296$ ./bpssphere -problem bp1 -ceed /cpu/self 297$ ./bpssphere -problem bp2 -ceed /gpu/cuda 298$ ./bpssphere -problem bp3 -ceed /cpu/self 299$ ./bpssphere -problem bp4 -ceed /gpu/cuda 300$ ./bpssphere -problem bp5 -ceed /cpu/self 301$ ./bpssphere -problem bp6 -ceed /gpu/cuda 302$ cd .. 303 304$ cd petsc/ 305$ make 306$ ./area -problem cube -ceed /cpu/self -degree 3 307$ ./area -problem cube -ceed /gpu/cuda -degree 3 308$ ./area -problem sphere -ceed /cpu/self -degree 3 -dm_refine 2 309$ ./area -problem sphere -ceed /gpu/cuda -degree 3 -dm_refine 2 310 311$ cd fluids/ 312$ make 313$ ./navierstokes -ceed /cpu/self -degree 1 314$ ./navierstokes -ceed /gpu/cuda -degree 1 315$ cd .. 316 317$ cd solids/ 318$ make 319$ ./elasticity -ceed /cpu/self -mesh [.exo file] -degree 2 -E 1 -nu 0.3 -problem Linear -forcing mms 320$ ./elasticity -ceed /gpu/cuda -mesh [.exo file] -degree 2 -E 1 -nu 0.3 -problem Linear -forcing mms 321$ cd .. 322``` 323 324For the last example shown, sample meshes to be used in place of `[.exo file]` can be found at <https://github.com/jeremylt/ceedSampleMeshes> 325 326The above code assumes a GPU-capable machine with the CUDA backends enabled. 327Depending on the available backends, other CEED resource specifiers can be provided with the `-ceed` option. 328Other command line arguments can be found in [examples/petsc](https://github.com/CEED/libCEED/blob/main/examples/petsc/README.md). 329 330% benchmarks-marker 331 332## Benchmarks 333 334A sequence of benchmarks for all enabled backends can be run using: 335 336```console 337$ make benchmarks 338``` 339 340The results from the benchmarks are stored inside the `benchmarks/` directory and they can be viewed using the commands (requires python with matplotlib): 341 342```console 343$ cd benchmarks 344$ python postprocess-plot.py petsc-bps-bp1-*-output.txt 345$ python postprocess-plot.py petsc-bps-bp3-*-output.txt 346``` 347 348Using the `benchmarks` target runs a comprehensive set of benchmarks which may take some time to run. 349Subsets of the benchmarks can be run using the scripts in the `benchmarks` folder. 350 351For more details about the benchmarks, see the `benchmarks/README.md` file. 352 353## Install 354 355To install libCEED, run: 356 357```console 358$ make install prefix=/path/to/install/dir 359``` 360 361or (e.g., if creating packages): 362 363```console 364$ make install prefix=/usr DESTDIR=/packaging/path 365``` 366 367To build and install in separate steps, run: 368 369```console 370$ make for_install=1 prefix=/path/to/install/dir 371$ make install prefix=/path/to/install/dir 372``` 373 374The usual variables like `CC` and `CFLAGS` are used, and optimization flags for all languages can be set using the likes of `OPT='-O3 -march=native'`. 375Use `STATIC=1` to build static libraries (`libceed.a`). 376 377To install libCEED for Python, run: 378 379```console 380$ pip install libceed 381``` 382 383with the desired setuptools options, such as `--user`. 384 385### pkg-config 386 387In addition to library and header, libCEED provides a [pkg-config](https://en.wikipedia.org/wiki/Pkg-config) file that can be used to easily compile and link. 388[For example](https://people.freedesktop.org/~dbn/pkg-config-guide.html#faq), if `$prefix` is a standard location or you set the environment variable `PKG_CONFIG_PATH`: 389 390```console 391$ cc `pkg-config --cflags --libs ceed` -o myapp myapp.c 392``` 393 394will build `myapp` with libCEED. 395This can be used with the source or installed directories. 396Most build systems have support for pkg-config. 397 398## Contact 399 400You can reach the libCEED team by emailing [ceed-users@llnl.gov](mailto:ceed-users@llnl.gov) or by leaving a comment in the [issue tracker](https://github.com/CEED/libCEED/issues). 401 402## How to Cite 403 404If you utilize libCEED please cite: 405 406```bibtex 407@article{libceed-joss-paper, 408 author = {Jed Brown and Ahmad Abdelfattah and Valeria Barra and Natalie Beams and Jean Sylvain Camier and Veselin Dobrev and Yohann Dudouit and Leila Ghaffari and Tzanio Kolev and David Medina and Will Pazner and Thilina Ratnayaka and Jeremy Thompson and Stan Tomov}, 409 title = {{libCEED}: Fast algebra for high-order element-based discretizations}, 410 journal = {Journal of Open Source Software}, 411 year = {2021}, 412 publisher = {The Open Journal}, 413 volume = {6}, 414 number = {63}, 415 pages = {2945}, 416 doi = {10.21105/joss.02945} 417} 418``` 419 420The archival copy of the libCEED user manual is maintained on [Zenodo](https://doi.org/10.5281/zenodo.4302736). 421To cite the user manual: 422 423```bibtex 424@misc{libceed-user-manual, 425 author = {Abdelfattah, Ahmad and 426 Barra, Valeria and 427 Beams, Natalie and 428 Brown, Jed and 429 Camier, Jean-Sylvain and 430 Dobrev, Veselin and 431 Dudouit, Yohann and 432 Ghaffari, Leila and 433 Kolev, Tzanio and 434 Medina, David and 435 Pazner, Will and 436 Ratnayaka, Thilina and 437 Shakeri, Rezgar and 438 Thompson, Jeremy L and 439 Tomov, Stanimire and 440 Wright III, James}, 441 title = {{libCEED} User Manual}, 442 month = dec, 443 year = 2022, 444 publisher = {Zenodo}, 445 version = {0.11.0}, 446 doi = {10.5281/zenodo.7480454} 447} 448``` 449 450For libCEED's Python interface please cite: 451 452```bibtex 453@InProceedings{libceed-paper-proc-scipy-2020, 454 author = {{V}aleria {B}arra and {J}ed {B}rown and {J}eremy {T}hompson and {Y}ohann {D}udouit}, 455 title = {{H}igh-performance operator evaluations with ease of use: lib{C}{E}{E}{D}'s {P}ython interface}, 456 booktitle = {{P}roceedings of the 19th {P}ython in {S}cience {C}onference}, 457 pages = {85 - 90}, 458 year = {2020}, 459 editor = {{M}eghann {A}garwal and {C}hris {C}alloway and {D}illon {N}iederhut and {D}avid {S}hupe}, 460 doi = {10.25080/Majora-342d178e-00c} 461} 462``` 463 464The BibTeX entries for these references can be found in the `doc/bib/references.bib` file. 465 466## Copyright 467 468The following copyright applies to each file in the CEED software suite, unless otherwise stated in the file: 469 470> Copyright (c) 2017-2023, Lawrence Livermore National Security, LLC and other CEED contributors. 471> All rights reserved. 472 473See files LICENSE and NOTICE for details. 474 475[github-badge]: https://github.com/CEED/libCEED/workflows/C/Fortran/badge.svg 476[github-link]: https://github.com/CEED/libCEED/actions 477[gitlab-badge]: https://gitlab.com/libceed/libCEED/badges/main/pipeline.svg?key_text=GitLab-CI 478[gitlab-link]: https://gitlab.com/libceed/libCEED/-/pipelines?page=1&scope=all&ref=main 479[codecov-badge]: https://codecov.io/gh/CEED/libCEED/branch/main/graphs/badge.svg 480[codecov-link]: https://codecov.io/gh/CEED/libCEED/ 481[license-badge]: https://img.shields.io/badge/License-BSD%202--Clause-orange.svg 482[license-link]: https://opensource.org/licenses/BSD-2-Clause 483[doc-badge]: https://readthedocs.org/projects/libceed/badge/?version=latest 484[doc-link]: https://libceed.org/en/latest/?badge=latest 485[joss-badge]: https://joss.theoj.org/papers/10.21105/joss.02945/status.svg 486[joss-link]: https://doi.org/10.21105/joss.02945 487[binder-badge]: http://mybinder.org/badge_logo.svg 488[binder-link]: https://mybinder.org/v2/gh/CEED/libCEED/main?urlpath=lab/tree/examples/python/tutorial-0-ceed.ipynb 489
