| b917901d | 27-Mar-2023 |
Junchao Zhang <jczhang@mcs.anl.gov> |
mataijcusparse: increase the cut off cuda version number to avoid a bug for unknwon reasons.
Even some cusparse APIs were introduced in cuda-11.3.0, we now use them only after cuda-11.4.0, which was
mataijcusparse: increase the cut off cuda version number to avoid a bug for unknwon reasons.
Even some cusparse APIs were introduced in cuda-11.3.0, we now use them only after cuda-11.4.0, which was two months younger than 11.3.0.
We met an error (see below) with cuda-11.3.x, but not with cuda versions lower or higher than 11.3. This might be a petsc bug. Using ifdefs,
aijcusparse code takes different paths, making the issue difficult to investigate. So we just increase the cuda version to get away.
-------
snes_tests-ex13_cuda
mpirun -n 4 ./ex13 -petsc_ci -dm_plex_dim 2 -benchmark_it 10 -dm_plex_box_faces 4,4 -dm_refine 2 -petscpartitioner_simple_process_grid 2,2 -petscpartitioner_simple_node_grid 1,1 -potential_petscspace_degree 2 -petscpartitioner_type simple -dm_plex_simplex 0 -snes_type ksponly -dm_view -ksp_type cg -pc_type gamg -pc_gamg_process_eq_limit 400 -ksp_norm_type unpreconditioned -ksp_converged_reason -dm_mat_type aijcusparse -dm_vec_type cuda
[3]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[3]PETSC ERROR: GPU error
[3]PETSC ERROR: cuSPARSE errorcode 7 (CUSPARSE_STATUS_INTERNAL_ERROR) : internal error
[3]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[3]PETSC ERROR: Petsc Development GIT revision: v3.18.5-1152-g648263c9 GIT Date: 2023-03-26 20:01:43 +0000
[3]PETSC ERROR: ./ex13 on a arch-kokkos-dbg named hong-gce-workstation by jczhang Mon Mar 27 15:55:44 2023
[3]PETSC ERROR: Configure options --PETSC_ARCH=arch-kokkos-dbg --with-debugging=1 --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 --COPTFLAGS="-g -O0" --FOPTFLAGS="-g -O0" --CXXOPTFLAGS="-g -O0" --with-cuda --with-cudac=nvcc --with-strict-petscerrorcode
[3]PETSC ERROR: #1 MatProductNumeric_SeqAIJCUSPARSE_SeqAIJCUSPARSE() at /scratch/jczhang/petsc/src/mat/impls/aij/seq/seqcusparse/aijcusparse.cu:2968
[3]PETSC ERROR: #2 MatProductNumeric_ABC_Basic() at /scratch/jczhang/petsc/src/mat/interface/matproduct.c:1129
[3]PETSC ERROR: #3 MatProductNumeric_MPIAIJBACKEND() at /scratch/jczhang/petsc/src/mat/impls/aij/mpi/mpiaij.c:7024
[3]PETSC ERROR: #4 MatProductNumeric() at /scratch/jczhang/petsc/src/mat/interface/matproduct.c:685
[3]PETSC ERROR: #5 MatPtAP() at /scratch/jczhang/petsc/src/mat/interface/matrix.c:9907
[3]PETSC ERROR: #6 PCSetUp_GAMG() at /scratch/jczhang/petsc/src/ksp/pc/impls/gamg/gamg.c:558
[3]PETSC ERROR: #7 PCSetUp() at /scratch/jczhang/petsc/src/ksp/pc/interface/precon.c:994
[3]PETSC ERROR: #8 KSPSetUp() at /scratch/jczhang/petsc/src/ksp/ksp/interface/itfunc.c:405
[3]PETSC ERROR: #9 KSPSolve_Private() at /scratch/jczhang/petsc/src/ksp/ksp/interface/itfunc.c:823
[3]PETSC ERROR: #10 KSPSolve() at /scratch/jczhang/petsc/src/ksp/ksp/interface/itfunc.c:1069
[3]PETSC ERROR: #11 SNESSolve_KSPONLY() at /scratch/jczhang/petsc/src/snes/impls/ksponly/ksponly.c:48
[3]PETSC ERROR: #12 SNESSolve() at /scratch/jczhang/petsc/src/snes/interface/snes.c:4666
[3]PETSC ERROR: #13 main() at ex13.c:193
show more ...
|