test-lib: allow selecting tests by substring/glob with --run
Many of our test scripts have several "setup" tests. It's a lot easier
to say
./t0050-filesystem.sh --run=setup,9
in order to run all the setup tests as well as test #9, than it is to
track down what all the setup tests are and enter all their numbers in
the list. Also, I often find myself wanting to run just one or a couple
tests from the test file, but I don't know the numbering of any of the
tests -- to get it I either have to first run the whole test file (or
start counting by hand or figure out some other clever but non-obvious
tricks). It's really convenient to be able to just look at the test
description(s) and then run
./t6416-recursive-corner-cases.sh --run=symlink
or
./t6402-merge-rename.sh --run='setup,unnecessary update'
Add such an ability to test selection which relies on merely matching
against the test description.
Signed-off-by: Elijah Newren <newren@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
diff --git a/t/README b/t/README
index 2adaf7c..c730a70 100644
--- a/t/README
+++ b/t/README
@@ -258,16 +258,21 @@
only some tests should be run or that some tests should be
excluded from a run.
-The argument for --run is a list of individual test numbers or
-ranges with an optional negation prefix that define what tests in
-a test suite to include in the run. A range is two numbers
-separated with a dash and matches a range of tests with both ends
-been included. You may omit the first or the second number to
-mean "from the first test" or "up to the very last test"
-respectively.
+The argument for --run, <test-selector>, is a list of description
+substrings or globs or individual test numbers or ranges with an
+optional negation prefix (of '!') that define what tests in a test
+suite to include (or exclude, if negated) in the run. A range is two
+numbers separated with a dash and matches a range of tests with both
+ends been included. You may omit the first or the second number to
+mean "from the first test" or "up to the very last test" respectively.
-Optional prefix of '!' means that the test or a range of tests
-should be excluded from the run.
+The argument to --run is split on commas into separate strings,
+numbers, and ranges, and picks all tests that match any of the
+individual selection criteria. If the substring of the description
+text that you want to match includes a comma, use the glob character
+'?' instead. For example --run='rebase,merge?cherry-pick' would match
+on all tests that match either the glob *rebase* or the glob
+*merge?cherry-pick*.
If --run starts with an unprefixed number or range the initial
set of tests to run is empty. If the first item starts with '!'
@@ -275,9 +280,6 @@
determined every test number or range is added or excluded from
the set one by one, from left to right.
-Individual numbers or ranges could be separated either by a space
-or a comma.
-
For example, to run only tests up to a specific test (21), one
could do this:
@@ -290,7 +292,7 @@
Common case is to run several setup tests (1, 2, 3) and then a
specific test (21) that relies on that setup:
- $ sh ./t9200-git-cvsexport-commit.sh --run='1 2 3 21'
+ $ sh ./t9200-git-cvsexport-commit.sh --run='1,2,3,21'
or:
@@ -298,17 +300,17 @@
or:
- $ sh ./t9200-git-cvsexport-commit.sh --run='-3 21'
+ $ sh ./t9200-git-cvsexport-commit.sh --run='-3,21'
As noted above, the test set is built by going through the items
from left to right, so this:
- $ sh ./t9200-git-cvsexport-commit.sh --run='1-4 !3'
+ $ sh ./t9200-git-cvsexport-commit.sh --run='1-4,!3'
will run tests 1, 2, and 4. Items that come later have higher
precedence. It means that this:
- $ sh ./t9200-git-cvsexport-commit.sh --run='!3 1-4'
+ $ sh ./t9200-git-cvsexport-commit.sh --run='!3,1-4'
would just run tests from 1 to 4, including 3.
@@ -317,6 +319,18 @@
$ sh ./t9200-git-cvsexport-commit.sh --run='!7-11'
+Sometimes there may be multiple tests with e.g. "setup" in their name
+that are needed and rather than figuring out the number for all of them
+we can just use "setup" as a substring/glob to match against the test
+description:
+
+ $ sh ./t0050-filesystem.sh --run=setup,9-11
+
+or one could select both the setup tests and the rename ones (assuming all
+relevant tests had those words in their descriptions):
+
+ $ sh ./t0050-filesystem.sh --run=setup,rename
+
Some tests in a test suite rely on the previous tests performing
certain actions, specifically some tests are designated as
"setup" test, so you cannot _arbitrarily_ disable one test and
diff --git a/t/t0000-basic.sh b/t/t0000-basic.sh
index 923281a..62a16ed 100755
--- a/t/t0000-basic.sh
+++ b/t/t0000-basic.sh
@@ -430,7 +430,7 @@
test_expect_success '--run basic' "
run_sub_test_lib_test run-basic \
- '--run basic' --run='1 3 5' <<-\\EOF &&
+ '--run basic' --run='1,3,5' <<-\\EOF &&
for i in 1 2 3 4 5 6
do
test_expect_success \"passing test #\$i\" 'true'
@@ -472,7 +472,7 @@
test_expect_success '--run with two ranges' "
run_sub_test_lib_test run-two-ranges \
- '--run with two ranges' --run='1-2 5-6' <<-\\EOF &&
+ '--run with two ranges' --run='1-2,5-6' <<-\\EOF &&
for i in 1 2 3 4 5 6
do
test_expect_success \"passing test #\$i\" 'true'
@@ -556,7 +556,7 @@
test_expect_success '--run with two negations' "
run_sub_test_lib_test run-two-neg \
- '--run with two negations' --run='"'!3 !6'"' <<-\\EOF &&
+ '--run with two negations' --run='"'!3,!6'"' <<-\\EOF &&
for i in 1 2 3 4 5 6
do
test_expect_success \"passing test #\$i\" 'true'
@@ -577,7 +577,7 @@
test_expect_success '--run a range and negation' "
run_sub_test_lib_test run-range-and-neg \
- '--run a range and negation' --run='"'-4 !2'"' <<-\\EOF &&
+ '--run a range and negation' --run='"'-4,!2'"' <<-\\EOF &&
for i in 1 2 3 4 5 6
do
test_expect_success \"passing test #\$i\" 'true'
@@ -620,7 +620,7 @@
test_expect_success '--run include, exclude and include' "
run_sub_test_lib_test run-inc-neg-inc \
'--run include, exclude and include' \
- --run='"'1-5 !1-3 2'"' <<-\\EOF &&
+ --run='"'1-5,!1-3,2'"' <<-\\EOF &&
for i in 1 2 3 4 5 6
do
test_expect_success \"passing test #\$i\" 'true'
@@ -664,7 +664,7 @@
test_expect_success '--run exclude and include' "
run_sub_test_lib_test run-neg-inc \
'--run exclude and include' \
- --run='"'!3- 5'"' <<-\\EOF &&
+ --run='"'!3-,5'"' <<-\\EOF &&
for i in 1 2 3 4 5 6
do
test_expect_success \"passing test #\$i\" 'true'
@@ -705,7 +705,31 @@
EOF
"
-test_expect_success '--run invalid range start' "
+test_expect_success '--run substring selector' "
+ run_sub_test_lib_test run-substring-selector \
+ '--run empty selectors' \
+ --run='relevant' <<-\\EOF &&
+ test_expect_success \"relevant test\" 'true'
+ for i in 1 2 3 4 5 6
+ do
+ test_expect_success \"other test #\$i\" 'true'
+ done
+ test_done
+ EOF
+ check_sub_test_lib_test run-substring-selector <<-\\EOF
+ > ok 1 - relevant test
+ > ok 2 # skip other test #1 (--run)
+ > ok 3 # skip other test #2 (--run)
+ > ok 4 # skip other test #3 (--run)
+ > ok 5 # skip other test #4 (--run)
+ > ok 6 # skip other test #5 (--run)
+ > ok 7 # skip other test #6 (--run)
+ > # passed all 7 test(s)
+ > 1..7
+ EOF
+"
+
+test_expect_success '--run keyword selection' "
run_sub_test_lib_test_err run-inv-range-start \
'--run invalid range start' \
--run='a-5' <<-\\EOF &&
@@ -735,21 +759,6 @@
EOF_ERR
"
-test_expect_success '--run invalid selector' "
- run_sub_test_lib_test_err run-inv-selector \
- '--run invalid selector' \
- --run='1?' <<-\\EOF &&
- test_expect_success \"passing test #1\" 'true'
- test_done
- EOF
- check_sub_test_lib_test_err run-inv-selector \
- <<-\\EOF_OUT 3<<-\\EOF_ERR
- > FATAL: Unexpected exit with code 1
- EOF_OUT
- > error: --run: invalid non-numeric in test selector: '1?'
- EOF_ERR
-"
-
test_set_prereq HAVEIT
haveit=no
diff --git a/t/test-lib.sh b/t/test-lib.sh
index ef31f40..72b88df 100644
--- a/t/test-lib.sh
+++ b/t/test-lib.sh
@@ -769,15 +769,17 @@
}
match_test_selector_list () {
+ operation="$1"
+ shift
title="$1"
shift
arg="$1"
shift
test -z "$1" && return 0
- # Both commas and whitespace are accepted as separators.
+ # Commas are accepted as separators.
OLDIFS=$IFS
- IFS=' ,'
+ IFS=','
set -- $1
IFS=$OLDIFS
@@ -805,13 +807,13 @@
*-*)
if expr "z${selector%%-*}" : "z[0-9]*[^0-9]" >/dev/null
then
- echo "error: $title: invalid non-numeric in range" \
+ echo "error: $operation: invalid non-numeric in range" \
"start: '$orig_selector'" >&2
exit 1
fi
if expr "z${selector#*-}" : "z[0-9]*[^0-9]" >/dev/null
then
- echo "error: $title: invalid non-numeric in range" \
+ echo "error: $operation: invalid non-numeric in range" \
"end: '$orig_selector'" >&2
exit 1
fi
@@ -819,9 +821,11 @@
*)
if expr "z$selector" : "z[0-9]*[^0-9]" >/dev/null
then
- echo "error: $title: invalid non-numeric in test" \
- "selector: '$orig_selector'" >&2
- exit 1
+ case "$title" in *${selector}*)
+ include=$positive
+ ;;
+ esac
+ continue
fi
esac
@@ -1031,7 +1035,7 @@
skipped_reason="GIT_SKIP_TESTS"
fi
if test -z "$to_skip" && test -n "$run_list" &&
- ! match_test_selector_list '--run' $test_count "$run_list"
+ ! match_test_selector_list '--run' "$1" $test_count "$run_list"
then
to_skip=t
skipped_reason="--run"
