Lines Matching refs:test
5 The PETSc test system consists of
7 - Formatted comments at the bottom of the tutorials and test source files that describes the tests …
8 - The *test generator* (`config/gmakegentest.py`) that parses the tutorial and test source files an…
10 - The *PETSc test harness* that consists of makefile and shell scripts that runs the executables wi…
36 test:
37 test:
42 For our language, a *test* is associated with the following
48 … results*. It is also possible -- though unusual -- to have multiple output files for a single test
55 Our language also supports a *testset* that specifies either a new test
56 entirely or multiple executable/diff tests within a single test. At the
57 core, the executable/diff test combination will look something like
65 In practice, we want to do various logging and counting by the test
67 simple yet flexible test control.
74 …tafiles](https://gitlab.com/petsc/datafiles) contains all the test files needed for the test suite.
80 At the end of each test file, a marked comment block is
81 inserted to describe the test(s) to be run. The elements of the test are
82 done with a set of supported key words that sets up the test.
86 - as minimal as possible with the simplest test requiring only one keyword,
96 - **testset** or **test**: (*Required*)
98 - At the top level either a single test or a test set must be
103 - The test name is given by `testname = basestring` if the suffix
106 - This can be specified only for top level test nodes.
111 - A reference file with the *expected output from the test run*,
112 the output from the test run is compared with it (with a difftool).
128 is used in the diff part of the test. For example, `-j` enables testing
133 - Setting this Boolean to True will tell the test to appear in the
134 test harness but report only TODO per the TAP standard. Optionally
149 with `Error:`, then the test is assumed to be testing the `stderr` output, and the
156 parallel output, both the output from the test example and the
169 - The copying is done by the test generator and not by creating
177 the test is run to ensure that stale temporary files are not read.
190 specified. See {any}`test harness data<test_harness_data>`
201 individual test such that the new timeout time is
209 - Allows you to set environment variables for the test. Values are copied verbatim to
219 be defined in the same `env:` block, i.e. given a test `ex1.c` with the following
223 test:
227 test:
243 test:
258 test:
300 It is possible (and encouraged!) to test error conditions within the test harness. Since
345 to prevent it from being output when the CI test harness is running.
349 The following is the simplest test block:
353 test:
358 create `a_b_tutorials-ex1` test that requires only one
366 ! test:
373 test:
374 test:
387 Following is an example of how to test a permutation of arguments
396 test:
397 test:
412 test:
415 test:
440 Following is an example showing the hierarchical nature of the test
448 test:
450 test:
465 test:
466 test:
504 test:
510 The make rules for running tests are contained in `gmakefile.test` in the PETSc root directory. The…
514 $ make test
517 or, for a list of test options,
520 $ make help-test
525 The running of the test harness will show which tests fail, but you may not have
530 $ $EDITOR $PETSC_DIR/$PETSC_ARCH/tests/test*err.log
536 :::{figure} /images/developers/test-artifacts.png
545 $ make print-test test-fail=1
551 $ make print-test test-fail=1 | tr ' ' '\n' | sort
558 Here, two different workflows on developing with the test harness are presented,
559 and then the language for adding a new test is described. Before describing the
560 workflow, we first discuss the output of the test harness and how it maps onto
563 Consider this line from running the PETSc test system:
572 - The makefile target for the *test* is `vec_is_sf_tests-ex1_basic_1`
574 - The shell script running the test is located at: `$PETSC_DIR/$PETSC_ARCH/tests/vec/is/sf/tests/ru…
576 Let's say that you want to debug a single test as part of development. There
577 are two basic methods of doing this: 1) use shell script directly in test
578 directory, or 2) use the gmakefile.test from the top level directory. We present both
581 ### Debugging a test using shell the generated scripts
597 -f ................ force attempt to run test that would otherwise be skipped
628 For loops it sometimes can become onerous to run the whole test.
631 `$PETSC_DIR`, but it is easy to modify for execution in the test
638 ### Debugging a PETSc test using the gmakefile.test
643 $ make help-test
645 /usr/bin/gmake --no-print-directory test <options>
649 REPLACE=1 Replace the output in PETSC_DIR source tree (-m to test scripts)
651 ALT=1 Replace 'alt' output in PETSC_DIR source tree (-M to test scripts)
652 DIFF_NUMBERS=1 Diff the numbers in the output (-j to test scripts and petscdiff)
653 …CUDAMEMCHECK=1 Execute the tests using CUDA "compute-sanitizer --tool memcheck" (-U to test scr…
657 VALGRIND=1 Execute the tests using valgrind (-V to test scripts)
667 OPTIONS='<args>' Override options to scripts (-a to test scripts)
668 EXTRA_OPTIONS='<args>' Add options to scripts (-e to test scripts)
671 …MACOS_FIREWALL=1 Add each built test to the macOS firewall list to prevent popups. Configure --wi…
675 /usr/bin/gmake --no-print-directory test search='sys*ex2*'
677 /usr/bin/gmake --no-print-directory test s='sys*ex2*'
679 /usr/bin/gmake --no-print-directory test s='src/sys/tests/'
681 /usr/bin/gmake --no-print-directory test s='src/sys/tests/ex1.c'
683 To search for fields from the original test definitions:
684 /usr/bin/gmake --no-print-directory test query='requires' queryval='*MPI_PROCESS_SHARED_MEMORY*'
686 /usr/bin/gmake --no-print-directory test q='requires' qv='*MPI_PROCESS_SHARED_MEMORY*'
688 /usr/bin/gmake --no-print-directory test s='src/sys/tests/' searchin='*options*'
691 /usr/bin/gmake --no-print-directory test test-fail='1'
694 /usr/bin/gmake --no-print-directory print-test search=sys*
698 or make the test with NO_RM=1
701 To compile the test and run it:
704 $ make test search=vec_is_sf_tests-ex1_basic_1
724 For forming a search, it is recommended to always use `print-test` instead of
725 `test` to make sure it is returning the values that you want.
731 - Searches based on name of test target (see above)
736 $ make print-test search='vec_is*ex1*basic*1'
742 $ make print-test s='vec_is*ex1*basic*1'
748 $ make print-test s='src/vec/is/tests/ex1.c'
752 $ make print-test s='src/dm/impls/plex/tests/'
756 $ make print-test s='src/dm/impls/plex/tests/ex1.c'
761 - `query` corresponds to test harness keyword, `queryval` to the value. Example:
764 $ make print-test query='suffix' queryval='basic_1'
777 $ make print-test s='src/dm/impls/plex/tests/ex1.c' i='*refine_overlap_2d*'
793 $ make test gmakesearch='vec_is%ex1_basic_1'
801 $ make test gmakesearch='vec_is%1' gmakesearchin='basic'
809 $ make print-test query='suffix' queryval='basic_1'
813 $ make print-test query='requires' queryval='cuda'
817 $ make print-test query='requires' queryval='defined(PETSC_HAVE_MPI_GPU_AWARE)'
821 $ make print-test query='requires' queryval='*GPU_AWARE*'
829 $ make print-test query='name' queryval='vec_is*ex1*basic*1'
855 $ make print-test query='args' queryval='ksp_monitor'
859 $ make print-test query='args' queryval='*monitor*'
863 $ make print-test query='args' queryval='pc_type ml'
875 $ make print-test query='requires,requires' queryval='cuda,hip'
877 $ make print-test query='requires%AND%requires' queryval='cuda%AND%hip'
883 $ make print-test query='requires|requires' queryval='ctetgen,triangle'
885 $ make print-test query='requires%OR%requires' queryval='ctetgen%AND%triangle'
891 $ make print-test query='requires,requires' queryval='ctetgen,triangle'
893 $ make print-test query='requires%AND%requires' queryval='ctetgen%AND%triangle'
899 $ make print-test query='requires|name' queryval='cuda,dm*'
901 $ make print-test query='requires%OR%name' queryval='cuda%AND%dm*'
907 $ make print-test query='requires' queryval='ctetgen' | tr ' ' '\n' | wc -l
909 $ make print-test query='requires' queryval='triangle' | tr ' ' '\n' | wc -l
911 $ make print-test query='requires,requires' queryval='ctetgen,triangle' | tr ' ' '\n' | wc -l
913 $ make print-test query='requires|requires' queryval='ctetgen,triangle' | tr ' ' '\n' | wc -l
963 $ make test s='src/ksp/ksp/tests/ex9.c' i='*1'
975 $ make test s='src/ksp/ksp/tests/ex9.c' i='*1' V=1
986 $ make test s='src/ksp/ksp/tests/ex9.c' i='*1' v=1 | grep 'not ok' | cut -d# -f2
992 The goals of the PETSc test harness are threefold.
998 Before understanding the test harness, you should first understand the
1003 After inserting the language into the file, you can test the parsing by
1010 in generating the test harness.
1014 The PETSc test system is designed to be compliant with the [Test Anything Protocol (TAP)](https://t…
1023 As an example, consider this test input:
1026 test:
1068 A small sample of the output from the test harness is as follows.
1095 To modify the test harness, you can modify `$PETSC_DIR/config/petsc_harness.sh`.