-
Notifications
You must be signed in to change notification settings - Fork 124
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Implement check
result
key, checks affect test result by default (#…
…3239) * Implement check result key feature * Add check-result spec and release note * Move and modify check schema * Fix doc generation Co-authored-by: Miloš Prchlík <mprchlik@redhat.com> Co-authored-by: Petr Šplíchal <psplicha@redhat.com>
- Loading branch information
1 parent
5815197
commit 60c2d37
Showing
19 changed files
with
550 additions
and
67 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,29 @@ | ||
#!/bin/bash | ||
. /usr/share/beakerlib/beakerlib.sh || exit 1 | ||
|
||
rlJournalStart | ||
rlPhaseStartSetup | ||
rlRun "run=\$(mktemp -d)" 0 "Create run directory" | ||
rlRun "PROVISION_HOW=${PROVISION_HOW:-virtual}" | ||
rlRun "pushd check" | ||
rlRun "set -o pipefail" | ||
rlPhaseEnd | ||
|
||
rlPhaseStartTest "Check Results" | ||
rlRun "tmt run -av --id $run provision --how $PROVISION_HOW" 1 | ||
rlRun -s "tmt run --id $run report -v" 1 | ||
|
||
rlAssertGrep "pass /test/check-fail-info (check 'dmesg' is informational)" "$rlRun_LOG" | ||
rlAssertGrep "fail /test/check-fail-respect (check 'dmesg' failed, original test result: pass)" "$rlRun_LOG" | ||
rlAssertGrep "pass /test/check-override (check 'dmesg' failed, test result overridden: pass)" "$rlRun_LOG" | ||
rlAssertGrep "pass /test/check-pass" "$rlRun_LOG" | ||
rlAssertGrep "fail /test/check-pass-test-xfail (test was expected to fail, original test result: pass)" "$rlRun_LOG" | ||
rlAssertGrep "pass /test/check-xfail-fail (check 'dmesg' failed as expected)" "$rlRun_LOG" | ||
rlAssertGrep "fail /test/check-xfail-pass (check 'dmesg' did not fail as expected, original test result: pass)" "$rlRun_LOG" | ||
rlPhaseEnd | ||
|
||
rlPhaseStartCleanup | ||
rlRun "popd" | ||
rlRun "rm -r ${run}" 0 "Remove run directory" | ||
rlPhaseEnd | ||
rlJournalEnd |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
1 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,89 @@ | ||
summary: Tests for check results behaviour | ||
description: Verify that check results are correctly interpreted and affect test results | ||
framework: shell | ||
duration: 1m | ||
|
||
/check-pass: | ||
summary: Test with passing checks | ||
description: | | ||
Expected outcome: PASS (test passes, check passes) | ||
test: echo "Test passed" | ||
check: | ||
- how: dmesg | ||
result: respect | ||
|
||
/check-pass-test-xfail: | ||
summary: Everything passing but failure expected | ||
description: | | ||
Expected outcome: FAIL (test passes, check passes, but fail is expected) | ||
test: echo "Test passed" | ||
check: | ||
- how: dmesg | ||
result: respect | ||
result: xfail | ||
|
||
/check-fail-respect: | ||
summary: Test with failing dmesg check (respect) | ||
description: | | ||
Expected outcome: FAIL (test passes, but check fails and is respected) | ||
test: echo "Fail Test Check Pattern" | tee /dev/kmsg | ||
check: | ||
- how: dmesg | ||
failure-pattern: Fail Test Check Pattern | ||
|
||
/check-fail-info: | ||
summary: Test with failing dmesg check (info) | ||
description: | | ||
Expected outcome: PASS (test passes, check fails, but should be just info) | ||
test: echo "Fail Test Check Pattern" | tee /dev/kmsg | ||
check: | ||
- how: dmesg | ||
failure-pattern: Fail Test Check Pattern | ||
result: info | ||
|
||
/check-xfail-pass: | ||
summary: Test with passing dmesg check (xfail) | ||
description: | | ||
Expected outcome: FAIL (test passes, check passes but xfail expects it to fail) | ||
test: echo "Test passed" | ||
check: | ||
- how: dmesg | ||
result: xfail | ||
|
||
/check-xfail-fail: | ||
summary: Test with failing dmesg check (xfail) | ||
description: | | ||
Expected outcome: PASS (test passes, check fails but xfail expects it to fail) | ||
test: echo "Fail Test Check Pattern" | tee /dev/kmsg | ||
check: | ||
- how: dmesg | ||
failure-pattern: Fail Test Check Pattern | ||
result: xfail | ||
|
||
/check-multiple: | ||
summary: Test with multiple checks with different result interpretations | ||
description: | | ||
Expected outcome: FAIL (first dmesg check fails and is respected, second dmesg check | ||
passes but xfail expects it to fail, third failing dmesg check is just info)" | ||
test: echo "Fail Test Check Pattern" | tee /dev/kmsg | ||
check: | ||
- how: dmesg | ||
failure-pattern: Fail Test Check Pattern | ||
result: respect | ||
- how: dmesg | ||
result: xfail | ||
- how: dmesg | ||
failure-pattern: Fail Test Check Pattern | ||
result: info | ||
enabled: false # TODO: handle multiple checks with same 'name'/'how' | ||
|
||
/check-override: | ||
summary: Test with failing dmesg check but overridden by test result | ||
description: | | ||
Expected outcome: PASS (test passes, check fails but is overridden by 'result: pass') | ||
test: echo "Fail Test Check Pattern" | tee /dev/kmsg | ||
result: pass | ||
check: | ||
- how: dmesg | ||
failure-pattern: Fail Test Check Pattern | ||
result: respect |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.