Lines Matching +full:test +full:- +full:basic
5 The PETSc test system consists of
7 - Formatted comments at the bottom of the tutorials and test source files that describes the tests …
8 - The *test generator* (`config/gmakegentest.py`) that parses the tutorial and test source files an…
10 - The *PETSc test harness* that consists of makefile and shell scripts that runs the executables wi…
14 ## PETSc Test Description Language
27 /*TEST
35 args: -debug -fields v1,v2,v3
36 test:
37 test:
38 args: -foo bar
39 TEST*/
42 For our language, a *test* is associated with the following
44 - A single shell script
46 - A single makefile
48 - An output file that represents the *expected results*. It is also possible -- though unusual -- t…
50 - Two or more command tests, usually:
52 - one or more `mpiexec` tests that run the executable
53 - one or more `diff` tests to compare output with the expected result
55 Our language also supports a *testset* that specifies either a new test
56 entirely or multiple executable/diff tests within a single test. At the
57 core, the executable/diff test combination will look something like
61 mpiexec -n 1 ../ex1 1> ex1.tmp 2> ex1.err
62 diff ex1.tmp output/ex1.out 1> diff-ex1.tmp 2> diff-ex1.err
65 In practice, we want to do various logging and counting by the test
67 simple yet flexible test control.
74 …tafiles](https://gitlab.com/petsc/datafiles) contains all the test files needed for the test suite.
80 At the end of each test file, a marked comment block is
81 inserted to describe the test(s) to be run. The elements of the test are
82 done with a set of supported key words that sets up the test.
86 - as minimal as possible with the simplest test requiring only one keyword,
87 - independent of the filename such that a file can be renamed without rewriting the tests, and
88 - intuitive.
96 - **testset** or **test**: (*Required*)
98 - At the top level either a single test or a test set must be
99 specified. All other keywords are sub-entries of this keyword.
101 - **suffix**: (*Optional*; *Default:* `suffix=""`)
103 - The test name is given by `testname = basestring` if the suffix
106 - This can be specified only for top level test nodes.
108 - **output_file**: (*Optional*; *Default:*
111 - A reference file with the *expected output from the test run*,
112 the output from the test run is compared with it (with a difftool).
113 - This file is described relative to the source directory of the
117 - **nsize**: (*Optional*; *Default:* `nsize=1`)
119 - This integer is passed to mpiexec; i.e., `mpiexec -n nsize`
121 - **args**: (*Optional*; *Default:* `""`)
123 - These arguments are passed to the executable.
125 - **diff_args**: (*Optional*; *Default:* `""`)
127 - These arguments are passed to the `lib/petsc/bin/petscdiff` script that
128 is used in the diff part of the test. For example, `-j` enables testing
131 - **TODO**: (*Optional*; *Default:* `False`)
133 - Setting this Boolean to True will tell the test to appear in the
134 test harness but report only TODO per the TAP standard. Optionally
136 - A runscript will be generated and can easily be modified by hand
139 - **filter**: (*Optional*; *Default:* `""`)
141 - Sometimes only a subset of the output is meant to be tested
145 - The value of this is the command to be run, for example,
146 `grep foo` or `sort -nr`.
147 - **NOTE: this method of testing error output is NOT recommended. See section on**
149 with `Error:`, then the test is assumed to be testing the `stderr` output, and the
152 - **filter_output**: (*Optional*; *Default:* `""`)
154 - Sometimes filtering the output file is useful for standardizing
156 parallel output, both the output from the test example and the
161 - **localrunfiles**: (*Optional*; *Default:* `""`)
163 - Some tests
165 Files in this (space-delimited) list will be copied over to the
169 - The copying is done by the test generator and not by creating
172 - **temporaries**: (*Optional*; *Default:* `""`)
174 - Some tests produce temporary files that are read by the filter
176 Files in this (space-delimited) list will cleared before
177 the test is run to ensure that stale temporary files are not read.
179 - **requires**: (*Optional*; *Default:* `""`)
181 - This is a space-delimited list of run requirements (not build
183 - In general, the language supports `and` and `not` constructs
185 - MPIUNI should work for all -n 1 examples so this need not be in
187 - Some tests require matrices or meshes contained in the
190 specified. See {any}`test harness data<test_harness_data>`
191 - Packages are indicated with lower-case specification, for example,
193 - Any defined variable in petscconf.h can be specified with the
195 - Any definition of the form `PETSC_HAVE_FOO` can just use
196 `requires: foo` similar to how third-party packages are handled.
198 - **timeoutfactor**: (*Optional*; *Default:* `"1"`)
200 - This parameter allows you to extend the default timeout for an
201 individual test such that the new timeout time is
203 - Tests are limited to a set time that is found at the top of
207 - **env**: (*Optional*; *Default:* `env=""`)
209 - Allows you to set environment variables for the test. Values are copied verbatim to
212 - Variables defined within `env:` blocks are expanded and processed by the shell that
215 must be done in the text of the `TEST` block.
217 - Defining the `env:` keyword more than once is allowed. Subsequent declarations are
219 be defined in the same `env:` block, i.e. given a test `ex1.c` with the following
223 test:
227 test:
238 - Variables defined in an `env:` block are evaluated by the runscript in the order in
239 which they are defined in the `TEST` block. Thus it is possible for later variables
243 test:
258 test:
273 - **for loops**: Specifying `{{list of values}}` will generate a loop over
274 an enclosed space-delimited list of values.
279 args: -matload_block_size {{2 3}shared output}
282 Here the output for each `-matload_block_size` value is assumed to be
288 args: -matload_block_size {{2 3}separate output}
300 It is possible (and encouraged!) to test error conditions within the test harness. Since
305 args: -petsc_ci_portable_error_output -error_output_stdout
310 - `-petsc_ci_portable_error_output`: Strips system or configuration-specific information
313 - Removes all path components except the file name from the traceback
314 - Removes line and column numbers from the traceback
315 - Removes PETSc version information
316 - Removes `configure` options used
317 - Removes system name
318 - Removes hostname
319 - Removes date
325 **gracefully** exit where possible. For single-ranked runs this means returning with
326 exit-code `0` and calling `MPI_Finalize()` instead of `MPI_Abort()`. Multi-rank
330 - `-error_output_stdout`: Forces `SETERRQ()` and friends to dump error messages to
332 sub-directive under `filter:`) also works it appears to be unstable under heavy
338 When writing ASCII output that may be not portable, so one wants `-petsc_ci_portable_error_output` …
345 to prevent it from being output when the CI test harness is running.
347 ### Test Block Examples
349 The following is the simplest test block:
352 /*TEST
353 test:
354 TEST*/
358 create `a_b_tutorials-ex1` test that requires only one
365 !/*TEST
366 ! test:
367 !TEST*/
370 A more complete example, showing just the lines between `/*TEST` and `TEST*/`:
373 test:
374 test:
377 args: -t 2 -pc_type jacobi -ksp_monitor_short -ksp_type gmres
378 args: -ksp_gmres_cgs_refinement_type refine_always -s2_ksp_type bcgs
379 args: -s2_pc_type jacobi -s2_ksp_monitor_short
385 `a_b_tutorials-ex1` and `a_b_tutorials-ex1_1`.
387 Following is an example of how to test a permutation of arguments
394 args: -f0 ${DATAFILESPATH}/matrices/poisson1
395 args: -ksp_type cg -pc_type icc -pc_factor_levels 2
396 test:
397 test:
398 args: -mat_type seqsbaij
410 args: -f0 ${DATAFILESPATH}/matrices/medium
411 args: -ksp_type bicg
412 test:
414 args: -pc_type lu
415 test:
427 args: -f ${DATAFILESPATH}/matrices/small -mat_type aij
432 args: -f ${DATAFILESPATH}/matrices/small
433 args: -mat_type baij -matload_block_size {{2 3}shared output}
440 Following is an example showing the hierarchical nature of the test
447 args: -f ${DATAFILESPATH}/matrices/small -mat_type baij
448 test:
449 args: -matload_block_size 2
450 test:
451 args: -matload_block_size 3
462 args: -f0 ${DATAFILESPATH}/matrices/poisson1
463 args: -ksp_type cg -pc_type icc
464 args: -pc_factor_levels {{0 2 4}separate output}
465 test:
466 test:
467 args: -mat_type seqsbaij
472 - `runex10_19_pc_factor_levels-0.sh`
473 - `runex10_19_pc_factor_levels-2.sh`
474 - `runex10_19_pc_factor_levels-4.sh`
483 - **requires:** (*Optional*; *Default:* `""`)
485 - Same as the runtime requirements (for example, can include
491 - In addition, `TODO` is available to allow you to skip the build
494 - **depends:** (*Optional*; *Default:* `""`)
496 - List any dependencies required to compile the file
501 /*TEST
504 test:
505 TEST*/
510 The make rules for running tests are contained in `gmakefile.test` in the PETSc root directory. The…
514 $ make test
517 or, for a list of test options,
520 $ make help-test
525 The running of the test harness will show which tests fail, but you may not have
530 $ $EDITOR $PETSC_DIR/$PETSC_ARCH/tests/test*err.log
536 :::{figure} /images/developers/test-artifacts.png
537 :alt: Test Artifacts at Gitlab
539 Test artifacts can be downloaded from GitLab.
545 $ make print-test test-fail=1
551 $ make print-test test-fail=1 | tr ' ' '\n' | sort
558 Here, two different workflows on developing with the test harness are presented,
559 and then the language for adding a new test is described. Before describing the
560 workflow, we first discuss the output of the test harness and how it maps onto
563 Consider this line from running the PETSc test system:
566 TEST arch-ci-linux-uni-pkgs/tests/counts/vec_is_sf_tests-ex1_basic_1.counts
569 The string `vec_is_sf_tests-ex1_basic_1` gives the following information:
571 - The file generating the tests is found in `$PETSC_DIR/src/vec/is/sf/tests/ex1.c`
572 - The makefile target for the *test* is `vec_is_sf_tests-ex1_basic_1`
573 - The makefile target for the *executable* is `$PETSC_ARCH/tests/vec/is/sf/tests/ex1`
574 - The shell script running the test is located at: `$PETSC_DIR/$PETSC_ARCH/tests/vec/is/sf/tests/ru…
576 Let's say that you want to debug a single test as part of development. There
577 are two basic methods of doing this: 1) use shell script directly in test
578 directory, or 2) use the gmakefile.test from the top level directory. We present both
581 ### Debugging a test using shell the generated scripts
588 $ ./runex1_basic_1.sh -h
592 -a <args> ......... Override default arguments
593 -c ................ Cleanup (remove generated files)
594 -C ................ Compile
595 -d ................ Launch in debugger
596 -e <args> ......... Add extra arguments to default
597 -f ................ force attempt to run test that would otherwise be skipped
598 -h ................ help: print this message
599 -n <integer> ...... Override the number of processors to use
600 -j ................ Pass -j to petscdiff (just use diff)
601 -J <arg> .......... Pass -J to petscdiff (just use diff with arg)
602 -m ................ Update results using petscdiff
603 -M ................ Update alt files using petscdiff
604 -o <arg> .......... Output format: 'interactive', 'err_only'
605 -p ................ Print command: Print first command and exit
606 -t ................ Override the default timeout (default=60 sec)
607 -U ................ run cUda-memcheck
608 -V ................ run Valgrind
609 -v ................ Verbose: Print commands
612 We will be using the `-C`, `-V`, and `-p` flags.
614 A basic workflow is something similar to:
618 $ runex1_basic_1.sh -C
621 $ runex1_basic_1.sh -m # If need to update results
623 $ runex1_basic_1.sh -V # Make sure valgrind clean
625 $ git commit -a
628 For loops it sometimes can become onerous to run the whole test.
629 In this case, you can use the `-p` flag to print just the first
631 `$PETSC_DIR`, but it is easy to modify for execution in the test
635 $ runex1_basic_1.sh -p
638 ### Debugging a PETSc test using the gmakefile.test
643 $ make help-test
644 Test usage:
645 /usr/bin/gmake --no-print-directory test <options>
649 REPLACE=1 Replace the output in PETSC_DIR source tree (-m to test scripts)
651 ALT=1 Replace 'alt' output in PETSC_DIR source tree (-M to test scripts)
652 DIFF_NUMBERS=1 Diff the numbers in the output (-j to test scripts and petscdiff)
653 …CUDAMEMCHECK=1 Execute the tests using CUDA "compute-sanitizer --tool memcheck" (-U to test scr…
656 cuda-memcheck and compute-sanitizer are supported)
657 VALGRIND=1 Execute the tests using valgrind (-V to test scripts)
658 DEBUG=1 Launch tests in the debugger (-d to the scripts)
662 TIMEOUT=<time> Test timeout limit in seconds (default in config/petsc_harness.sh)
667 OPTIONS='<args>' Override options to scripts (-a to test scripts)
668 EXTRA_OPTIONS='<args>' Add options to scripts (-e to test scripts)
671 …MACOS_FIREWALL=1 Add each built test to the macOS firewall list to prevent popups. Configure --wi…
675 /usr/bin/gmake --no-print-directory test search='sys*ex2*'
677 /usr/bin/gmake --no-print-directory test s='sys*ex2*'
679 /usr/bin/gmake --no-print-directory test s='src/sys/tests/'
681 /usr/bin/gmake --no-print-directory test s='src/sys/tests/ex1.c'
683 To search for fields from the original test definitions:
684 /usr/bin/gmake --no-print-directory test query='requires' queryval='*MPI_PROCESS_SHARED_MEMORY*'
686 /usr/bin/gmake --no-print-directory test q='requires' qv='*MPI_PROCESS_SHARED_MEMORY*'
688 /usr/bin/gmake --no-print-directory test s='src/sys/tests/' searchin='*options*'
690 To re-run the last tests which failed:
691 /usr/bin/gmake --no-print-directory test test-fail='1'
694 /usr/bin/gmake --no-print-directory print-test search=sys*
697 /usr/bin/gmake --no-print-directory ${PETSC_ARCH}/tests/sys/tests/ex1
698 or make the test with NO_RM=1
701 To compile the test and run it:
704 $ make test search=vec_is_sf_tests-ex1_basic_1
707 This can consist of your basic workflow. However,
712 $ make vec_is_sf_tests-ex1_basic_1 PRINTONLY=1
716 $ /scratch/kruger/contrib/petsc-mpich-cxx/bin/mpiexec -n 1 arch-mpich-cxx-py3/tests/vec/is/sf/tests…
719 $ git commit -a
724 For forming a search, it is recommended to always use `print-test` instead of
725 `test` to make sure it is returning the values that you want.
727 The three basic and recommended arguments are:
729 - `search` (or `s`)
731 - Searches based on name of test target (see above)
733 - Use the familiar glob syntax (like the Unix `ls` command). Example:
736 $ make print-test search='vec_is*ex1*basic*1'
742 $ make print-test s='vec_is*ex1*basic*1'
745 - It also takes full paths. Examples:
748 $ make print-test s='src/vec/is/tests/ex1.c'
752 $ make print-test s='src/dm/impls/plex/tests/'
756 $ make print-test s='src/dm/impls/plex/tests/ex1.c'
759 - `query` and `queryval` (or `q` and `qv`)
761 - `query` corresponds to test harness keyword, `queryval` to the value. Example:
764 $ make print-test query='suffix' queryval='basic_1'
767 - Invokes `config/query_tests.py` to query the tests (see
768 `config/query_tests.py --help` for more information).
770 - See below for how to use as it has many features
772 - `searchin` (or `i`)
774 - Filters results of above searches. Example:
777 $ make print-test s='src/dm/impls/plex/tests/ex1.c' i='*refine_overlap_2d*'
782 - `gmakesearch`
784 - Use GNU make's own filter capability.
786 - Fast, but requires knowing GNU make regex syntax which uses `%` instead of `*`
788 - Also very limited (cannot use two `%`'s for example)
790 - Example:
793 $ make test gmakesearch='vec_is%ex1_basic_1'
796 - `gmakesearchin`
798 - Use GNU make's own filter capability to search in previous results. Example:
801 $ make test gmakesearch='vec_is%1' gmakesearchin='basic'
804 ### Query-based searching
809 $ make print-test query='suffix' queryval='basic_1'
813 $ make print-test query='requires' queryval='cuda'
817 $ make print-test query='requires' queryval='defined(PETSC_HAVE_MPI_GPU_AWARE)'
821 $ make print-test query='requires' queryval='*GPU_AWARE*'
826 - Example:
829 $ make print-test query='name' queryval='vec_is*ex1*basic*1'
832 - This can be combined with union/intersect queries as discussed below
837 args: -ksp_monitor_short -pc_type ml -ksp_max_it 3
848 - Numbers (see `ksp_max_it` above), but floats are ignored as well.
849 - Loops: `args: -pc_fieldsplit_diag_use_amat {{0 1}}` gives `pc_fieldsplit_diag_use_amat` as the se…
850 - Input files: `-f *`
855 $ make print-test query='args' queryval='ksp_monitor'
859 $ make print-test query='args' queryval='*monitor*'
863 $ make print-test query='args' queryval='pc_type ml'
872 - All examples using `cuda` and all examples using `hip`:
875 $ make print-test query='requires,requires' queryval='cuda,hip'
877 $ make print-test query='requires%AND%requires' queryval='cuda%AND%hip'
880 - Examples that require both triangle and ctetgen (intersection of tests)
883 $ make print-test query='requires|requires' queryval='ctetgen,triangle'
885 $ make print-test query='requires%OR%requires' queryval='ctetgen%AND%triangle'
888 - Tests that require either `ctetgen` or `triangle`
891 $ make print-test query='requires,requires' queryval='ctetgen,triangle'
893 $ make print-test query='requires%AND%requires' queryval='ctetgen%AND%triangle'
896 - Find `cuda` examples in the `dm` package.
899 $ make print-test query='requires|name' queryval='cuda,dm*'
901 $ make print-test query='requires%OR%name' queryval='cuda%AND%dm*'
907 $ make print-test query='requires' queryval='ctetgen' | tr ' ' '\n' | wc -l
909 $ make print-test query='requires' queryval='triangle' | tr ' ' '\n' | wc -l
911 $ make print-test query='requires,requires' queryval='ctetgen,triangle' | tr ' ' '\n' | wc -l
913 $ make print-test query='requires|requires' queryval='ctetgen,triangle' | tr ' ' '\n' | wc -l
963 $ make test s='src/ksp/ksp/tests/ex9.c' i='*1'
965 TEST arch-osx-pkgs-opt-new/tests/counts/ksp_ksp_tests-ex9_1.counts
966 …ok ksp_ksp_tests-ex9_1+pc_fieldsplit_diag_use_amat-0_pc_fieldsplit_diag_use_amat-0_pc_fieldsplit_t…
967 …not ok diff-ksp_ksp_tests-ex9_1+pc_fieldsplit_diag_use_amat-0_pc_fieldsplit_diag_use_amat-0_pc_fie…
968 …ok ksp_ksp_tests-ex9_1+pc_fieldsplit_diag_use_amat-0_pc_fieldsplit_diag_use_amat-0_pc_fieldsplit_t…
972 In this case, the trick is to use the verbose option, `V=1` (or for the shell script workflows, `-v…
975 $ make test s='src/ksp/ksp/tests/ex9.c' i='*1' V=1
977 arch-osx-pkgs-opt-new/tests/ksp/ksp/tests/runex9_1.sh -v
978 …-ex9_1+pc_fieldsplit_diag_use_amat-0_pc_fieldsplit_diag_use_amat-0_pc_fieldsplit_type-additive # m…
986 $ make test s='src/ksp/ksp/tests/ex9.c' i='*1' v=1 | grep 'not ok' | cut -d# -f2
987 mpiexec -n 1 ../ex9 -ksp_converged_reason -ksp_error_if_not_converged -pc_fieldsplit_diag_use_ama…
990 ## PETSc Test Harness
992 The goals of the PETSc test harness are threefold.
998 Before understanding the test harness, you should first understand the
1003 After inserting the language into the file, you can test the parsing by
1006 A dictionary will be pretty-printed. From this dictionary printout, any
1010 in generating the test harness.
1012 ## Test Output Standards: TAP
1014 The PETSc test system is designed to be compliant with the [Test Anything Protocol (TAP)](https://t…
1023 As an example, consider this test input:
1026 test:
1029 args: -f ${DATAFILESPATH}/matrices/small -mat_type {{aij baij sbaij}} -matload_block_size {{2 3}}
1036 ok 1 In mat...tests: "./ex138 -f ${DATAFILESPATH}/matrices/small -mat_type aij -matload_block_size …
1037 ok 2 In mat...tests: "Diff of ./ex138 -f ${DATAFILESPATH}/matrices/small -mat_type aij -matload_blo…
1038 ok 3 In mat...tests: "./ex138 -f ${DATAFILESPATH}/matrices/small -mat_type aij -matload_block_size …
1039 ok 4 In mat...tests: "Diff of ./ex138 -f ${DATAFILESPATH}/matrices/small -mat_type aij -matload_blo…
1040 ok 5 In mat...tests: "./ex138 -f ${DATAFILESPATH}/matrices/small -mat_type baij -matload_block_size…
1041 ok 6 In mat...tests: "Diff of ./ex138 -f ${DATAFILESPATH}/matrices/small -mat_type baij -matload_bl…
1044 ok 11 In mat...tests: "./ex138 -f ${DATAFILESPATH}/matrices/small -mat_type saij -matload_block_siz…
1045 ok 12 In mat...tests: "Diff of ./ex138 -f ${DATAFILESPATH}/matrices/small -mat_type aij -matload_bl…
1048 ## Test Harness Implementation
1050 Most of the requirements for being TAP-compliant lie in the shell
1060 petsc_testrun 'diff ex1.tmp output/ex1.out' diff-ex1.tmp diff-ex1.err
1068 A small sample of the output from the test harness is as follows.
1076 ok 7 ./ex3 -f /matrices/small -mat_type aij -matload_block_size 2
1078 ok 9 ./ex3 -f /matrices/small -mat_type aij -matload_block_size 3
1080 ok 11 ./ex3 -f /matrices/small -mat_type baij -matload_block_size 2
1082 ok 13 ./ex3 -f /matrices/small -mat_type baij -matload_block_size 3
1084 ok 15 ./ex3 -f /matrices/small -mat_type sbaij -matload_block_size 2
1086 ok 17 ./ex3 -f /matrices/small -mat_type sbaij -matload_block_size 3
1095 To modify the test harness, you can modify `$PETSC_DIR/config/petsc_harness.sh`.
1108 $ config/report_tests.py -h
1114 $ config/report_tests.py -t 5