Skip to content

Commit

Permalink
Merge branch 'main' into system-model
Browse files Browse the repository at this point in the history
  • Loading branch information
happz authored Oct 29, 2024
2 parents 21df60f + ce2c8be commit c7c4e63
Show file tree
Hide file tree
Showing 21 changed files with 589 additions and 104 deletions.
2 changes: 1 addition & 1 deletion docs/overview.rst
Original file line number Diff line number Diff line change
Expand Up @@ -777,7 +777,7 @@ James Molet, Cristian Le, Lili Nie, Martin Čermák, Michael Vogt,
Qinghua Cheng, Michael Engel, Anatoli Babenia, Colin Walters,
Link Dupont, Mario Casquero, Martin Klusoň, Pavel Holica,
Otto Šabart, Ismail Ibrahim Quwarah, Sergei Petrosian, Tom
Koscielniak and Han Han.
Koscielniak, Han Han and Luigi Pellecchia.


Copyright
Expand Down
79 changes: 46 additions & 33 deletions docs/releases.rst
Original file line number Diff line number Diff line change
Expand Up @@ -5,20 +5,31 @@
======================


tmt-1.39.0
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The :ref:`/plugins/provision/beaker` provision plugin gains
support for :ref:`system.model-name</spec/hardware/system>`
hardware requirement.


tmt-1.38.0
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

A new :ref:`test-runner` section has been added to the tmt
:ref:`guide`. It describes some important differences between
running tests on a :ref:`user-system` and scheduling test jobs in
Test checks affect the overall test result by default. The
:ref:`/spec/tests/check` specification now supports a new
``result`` key for individual checks. This attribute allows users
to control how the result of each check affects the overall test
result. Please note that tests, which were previously passing
with failing checks will now fail by default, unless the ``xfail``
or ``info`` is added.

Each execution of ``tmt-report-result`` command inside a shell
test will now create a tmt subresult. The main result outcome is
reduced from all subresults outcomes. If ``tmt-report-result`` is
not called during the test, the shell test framework behavior
remains the same - the test script exit code still has an impact
on the main test result. See also
:ref:`/stories/features/report-result`.
In order to prevent dangerous commands to be unintentionally run
on user's system, the :ref:`/plugins/provision/local` provision
plugin now requires to be executed with the ``--feeling-safe``
option or with the environment variable ``TMT_FEELING_SAFE`` set
to ``True``. See the :ref:`/stories/features/feeling-safe` section
for more details and motivation behind this change.

The beakerlib test framework tests now generate tmt subresults.
The behavior is very similar to the shell test framework with
Expand All @@ -35,26 +46,13 @@ enabled. There is only one exception - if the
``result: restraint`` option is set to a beakerlib test, the
phase subresults get converted as normal tmt custom results.

The :ref:`/plugins/report/junit` report plugin now removes all
invalid XML characters from the final JUnit XML.

In order to prevent dangerous commands to be unintentionally run
on user's system, the :ref:`/plugins/provision/local` provision
plugin now requires to be executed with the ``--feeling-safe``
option or with the environment variable ``TMT_FEELING_SAFE`` set
to ``True``. See the :ref:`/stories/features/feeling-safe` section
for more details and motivation behind this change.

The :ref:`/plugins/discover/fmf` discover plugin now supports
a new ``adjust-tests`` key which allows modifying metadata of all
discovered tests. This can be useful especially when fetching
tests from remote repositories where the user does not have write
access.

A race condition in the
:ref:`/plugins/provision/virtual.testcloud` plugin has been fixed,
thus multihost tests using this provision method should now work
reliably without unexpected connection failures.
Each execution of ``tmt-report-result`` command inside a shell
test will now create a tmt subresult. The main result outcome is
reduced from all subresults outcomes. If ``tmt-report-result`` is
not called during the test, the shell test framework behavior
remains the same - the test script exit code still has an impact
on the main test result. See also
:ref:`/stories/features/report-result`.

Support for RHEL-like operating systems in `Image Mode`__ has been
added. The destination directory of the scripts added by ``tmt``
Expand All @@ -63,6 +61,12 @@ all others the ``/usr/local/bin`` destination directory is used.
A new environment variable ``TMT_SCRIPTS_DIR`` is available
to override the default locations.

The :ref:`/plugins/discover/fmf` discover plugin now supports
a new ``adjust-tests`` key which allows modifying metadata of all
discovered tests. This can be useful especially when fetching
tests from remote repositories where the user does not have write
access.

__ https://www.redhat.com/en/technologies/linux-platforms/enterprise-linux/image-mode

The ``tmt link`` command now supports providing multiple links by
Expand All @@ -72,9 +76,18 @@ for example usage.
The :ref:`/plugins/provision/beaker` provision plugin gains support
for :ref:`cpu.stepping</spec/hardware/cpu>` hardware requirement.

The :ref:`/plugins/provision/beaker` provision plugin gains
support for :ref:`system.model-name</spec/hardware/system>`
hardware requirement.
The :ref:`/plugins/report/junit` report plugin now removes all
invalid XML characters from the final JUnit XML.

A new :ref:`test-runner` section has been added to the tmt
:ref:`guide`. It describes some important differences between
running tests on a :ref:`user-system` and scheduling test jobs in
:ref:`testing-farm`.

A race condition in the
:ref:`/plugins/provision/virtual.testcloud` plugin has been fixed,
thus multihost tests using this provision method should now work
reliably without unexpected connection failures.


tmt-1.37.0
Expand Down
10 changes: 9 additions & 1 deletion docs/scripts/generate-plugins.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
#!/usr/bin/env python3

import dataclasses
import enum
import sys
import textwrap
from typing import Any
Expand Down Expand Up @@ -56,7 +57,7 @@ def _is_inherited(

# TODO: for now, it's a list, but inspecting the actual tree of classes
# would be more generic. It's good enough for now.
return field.name in ('name', 'where', 'order', 'summary', 'enabled')
return field.name in ('name', 'where', 'order', 'summary', 'enabled', 'result')


def container_ignored_fields(container: ContainerClass) -> list[str]:
Expand Down Expand Up @@ -106,6 +107,12 @@ def container_intrinsic_fields(container: ContainerClass) -> list[str]:
return field_names


def is_enum(value: Any) -> bool:
""" Find out whether a given value is an enum member """

return isinstance(value, enum.Enum)


def _create_step_plugin_iterator(registry: tmt.plugins.PluginRegistry[tmt.steps.Method]):
""" Create iterator over plugins of a given registry """

Expand Down Expand Up @@ -184,6 +191,7 @@ def main() -> None:
STEP=step_name,
PLUGINS=plugin_generator,
REVIEWED_PLUGINS=REVIEWED_PLUGINS,
is_enum=is_enum,
container_fields=tmt.utils.container_fields,
container_field=tmt.utils.container_field,
container_ignored_fields=container_ignored_fields,
Expand Down
10 changes: 7 additions & 3 deletions docs/templates/plugins.rst.j2
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@
{{ option }}: ``{{ metadata.metavar }}``
{% elif metadata.default is boolean %}
{{ option }}: ``true|false``
{% elif metadata.choices %}
{{ option }}: ``{{ metadata.choices }}``
{% else %}
{{ option }}:
{% endif %}
Expand All @@ -27,11 +29,13 @@
{% elif actual_default is sequence %}
{% if not actual_default %}
Default: *not set*
{% else %}
{% else %}
Default: {% for default_item in actual_default %}``{{ default_item.pattern | default(default_item) }}``{% if not loop.last %}, {% endif %}{% endfor %}
{% endif %}
{% elif is_enum(actual_default) %}
Default: ``{{ actual_default.value }}``
{% else %}
{% set _ = LOGGER.warn("%s/%s.%s: could not render default value, '%s'" | format(STEP, plugin_id, field_name, actual_default)) %}
{% set _ = LOGGER.warn("%s/%s.%s: could not render default value, '%s'" | format(STEP, plugin_id, field_name, actual_default), shift=0) %}
Default: *could not render default value correctly*
{% endif %}
{% endif %}
Expand Down Expand Up @@ -71,7 +75,7 @@ The following keys are accepted by all plugins of the ``{{ STEP }}`` step.
Please, be aware that the documentation below is a work in progress. We are
working on fixing it, adding missing bits and generally making it better.
Also, it was originally used for command line help only, therefore the
formatting is often suboptional.
formatting is often suboptimal.
{% endif %}

{% if PLUGIN.__doc__ %}
Expand Down
39 changes: 39 additions & 0 deletions spec/tests/check.fmf
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,29 @@ description: |
panic detection, core dump collection or collection of system
logs.

By default, the check results affect the overall test outcome.
To change this behaviour, use the ``result`` key, which accepts
the following values:

respect
The check result is respected and affects the overall
test result. This is the default.

xfail
The check result is expected to fail (pass becomes
fail and vice-versa).

info
The check result is treated as an informational
message and does not affect the overall test result.

.. warning::

Note that running one check multiple times for the same
test is not yet supported.

.. versionchanged:: 1.38.0 the ``result`` key added

See :ref:`/plugins/test-checks` for the list of available checks.

example:
Expand All @@ -37,7 +60,23 @@ example:
- how: test-inspector
enable: false

- |
# Expect the AVC check to fail
check:
- how: avc
result: xfail
- how: dmesg
result: respect

- |
# Treat the dmesg check as informational only
check:
- how: dmesg
result: info

link:
- implemented-by: /tmt/checks
- implemented-by: /tmt/result.py
- verified-by: /tests/test/check/avc
- verified-by: /tests/test/check/dmesg
- verified-by: /tests/execute/result/check
3 changes: 0 additions & 3 deletions tests/core/feeling-safe/main.fmf
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,3 @@ summary: Check whether the safe/unsafe mode can be controlled

link:
- verifies: /stories/features/feeling-safe

# Until tmt-1.38 is released this will be failing
result: xfail
2 changes: 1 addition & 1 deletion tests/discover/tests.sh
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ rlJournalStart
rlRun -s "tmt run --id $workdir -vvv $plan"
rlAssertGrep "package: 1 package requested" "$rlRun_LOG" -F
rlAssertGrep "test: Concise summary" "$rlRun_LOG" -F
rlAssertGrep '00:00:00 pass /first (on default-0) (original result: fail) [1/2]' "$rlRun_LOG" -F
rlAssertGrep '00:00:00 pass /first (on default-0) (test failed as expected, original test result: fail) [1/2]' "$rlRun_LOG" -F
rlAssertGrep '00:00:00 pass /second (on default-0) [2/2]' "$rlRun_LOG" -F
rlPhaseEnd

Expand Down
23 changes: 12 additions & 11 deletions tests/execute/result/basic.sh
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ run()
if [ -z "${orig}" ]; then # No original result provided
rlAssertGrep "${res} ${tn}$" $rlRun_LOG
else
rlAssertGrep "${res} ${tn} (original result: ${orig})$" $rlRun_LOG
rlAssertGrep "${res} ${tn} (.*original test result: ${orig}.*)$" $rlRun_LOG
fi

echo
Expand All @@ -38,7 +38,7 @@ rlJournalStart
run "errr" "/test/error" "" 2
run "pass" "/test/xfail-fail" "fail" 0
run "fail" "/test/xfail-pass" "pass" 1
run "errr" "/test/xfail-error" "error" 2
run "errr" "/test/xfail-error" "" 2
run "pass" "/test/always-pass" "fail" 0
run "info" "/test/always-info" "pass" 0
run "warn" "/test/always-warn" "pass" 1
Expand All @@ -58,18 +58,18 @@ rlJournalStart
rlAssertGrep "$line" "$rlRun_LOG" -F
fi
done <<-EOF
00:00:00 errr /test/always-error (on default-0) (original result: pass) [1/12]
00:00:00 fail /test/always-fail (on default-0) (original result: pass) [2/12]
00:00:00 info /test/always-info (on default-0) (original result: pass) [3/12]
00:00:00 pass /test/always-pass (on default-0) (original result: fail) [4/12]
00:00:00 warn /test/always-warn (on default-0) (original result: pass) [5/12]
00:00:00 errr /test/always-error (on default-0) (test result overridden: error, original test result: pass) [1/12]
00:00:00 fail /test/always-fail (on default-0) (test result overridden: fail, original test result: pass) [2/12]
00:00:00 info /test/always-info (on default-0) (test result overridden: info, original test result: pass) [3/12]
00:00:00 pass /test/always-pass (on default-0) (test result overridden: pass, original test result: fail) [4/12]
00:00:00 warn /test/always-warn (on default-0) (test result overridden: warn, original test result: pass) [5/12]
00:00:00 errr /test/error (on default-0) [6/12]
00:00:01 errr /test/error-timeout (on default-0) (timeout) [7/12]
00:00:00 fail /test/fail (on default-0) [8/12]
00:00:00 pass /test/pass (on default-0) [9/12]
00:00:00 errr /test/xfail-error (on default-0) (original result: error) [10/12]
00:00:00 pass /test/xfail-fail (on default-0) (original result: fail) [11/12]
00:00:00 fail /test/xfail-pass (on default-0) (original result: pass) [12/12]
00:00:00 errr /test/xfail-error (on default-0) [10/12]
00:00:00 pass /test/xfail-fail (on default-0) (test failed as expected, original test result: fail) [11/12]
00:00:00 fail /test/xfail-pass (on default-0) (test was expected to fail, original test result: pass) [12/12]
EOF
rlPhaseEnd

Expand All @@ -78,7 +78,8 @@ EOF
rlRun -s "tmt run --id \${run} --scratch --until execute tests -n /xfail-with-reboot provision --how container execute -v 2>&1 >/dev/null"
EXPECTED=$(cat <<EOF
00:00:00 /test/xfail-with-reboot [1/1]
00:00:00 pass /test/xfail-with-reboot (on default-0) (original result: fail) [1/1]
00:00:00 pass /test/xfail-with-reboot (on default-0) (test failed as expected, original test result: fail) [1/1]
EOF
)
rlAssertEquals "Output matches the expectation" "$EXPECTED" "$(grep /test/xfail-with-reboot $rlRun_LOG)"
Expand Down
29 changes: 29 additions & 0 deletions tests/execute/result/check.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
#!/bin/bash
. /usr/share/beakerlib/beakerlib.sh || exit 1

rlJournalStart
rlPhaseStartSetup
rlRun "run=\$(mktemp -d)" 0 "Create run directory"
rlRun "PROVISION_HOW=${PROVISION_HOW:-virtual}"
rlRun "pushd check"
rlRun "set -o pipefail"
rlPhaseEnd

rlPhaseStartTest "Check Results"
rlRun "tmt run -av --id $run provision --how $PROVISION_HOW" 1
rlRun -s "tmt run --id $run report -v" 1

rlAssertGrep "pass /test/check-fail-info (check 'dmesg' is informational)" "$rlRun_LOG"
rlAssertGrep "fail /test/check-fail-respect (check 'dmesg' failed, original test result: pass)" "$rlRun_LOG"
rlAssertGrep "pass /test/check-override (check 'dmesg' failed, test result overridden: pass)" "$rlRun_LOG"
rlAssertGrep "pass /test/check-pass" "$rlRun_LOG"
rlAssertGrep "fail /test/check-pass-test-xfail (test was expected to fail, original test result: pass)" "$rlRun_LOG"
rlAssertGrep "pass /test/check-xfail-fail (check 'dmesg' failed as expected)" "$rlRun_LOG"
rlAssertGrep "fail /test/check-xfail-pass (check 'dmesg' did not fail as expected, original test result: pass)" "$rlRun_LOG"
rlPhaseEnd

rlPhaseStartCleanup
rlRun "popd"
rlRun "rm -r ${run}" 0 "Remove run directory"
rlPhaseEnd
rlJournalEnd
1 change: 1 addition & 0 deletions tests/execute/result/check/.fmf/version
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
1
Loading

0 comments on commit c7c4e63

Please sign in to comment.