Skip to content

Latest commit

 

History

History
133 lines (96 loc) · 4.41 KB

conformance.md

File metadata and controls

133 lines (96 loc) · 4.41 KB
Raw

Conformance testing

The Conformance tests are a subset of our e2e test suite that cover a set of features defining Kubevirt core functionalities. The goal of the suite is to make sure that selected features work across different Vendors, CNIs, CSIs, CRIs implementations.

How are Conformance tests selected

By default tests with a [Conformance] tag (and from release 0.59 also decorator) will be executed and tests with a [Disruptive] tag will be skipped. These tests will not destroy or modify anything outside of the explicitly created test namespaces by the binary.

The tests are always part of Kubevirt e2e tests that are usually approved by individual Sigs. For the Conformance tests we have higher requirements than the regular tests, most significant requirements are:

  • Test is not Disruptive
  • Test is not flaky and is reliable per data available from CI team
  • For more concrete details see below

Conformance tests requirements:

  • The feature being tested is GA (no feature gate)
  • The feature is not deprecated
  • The feature is turned on by default (no feature toggle)
  • The test uses only GAed Kubernetes APIs/features
  • The test uses only APIs available to user
  • The test doesn't use direct access to virt-launcher/libvirt
  • It doesn't require any special hardware (such as gpu/pci devices,...) being available on the cluster other than the default required device nodes (kvm, virtio-net)
  • It doesn't use Skip or any other helper that uses it
  • It does not require access to public network, all images or resources should be able to be pre-pulled
  • It does not rely on environment of the executor
  • It has history of stability and is not flaky, and consequently doesn't have the QUARANTINE label.
  • It doesn't rely on specific implementation of CRI, CSI, CNI

Exceptions are allowed but must be explicit and reasoned.

Delivery

KubeVirt releases a kubevirt/conformance:<release> image and conformance.yaml manifest starting from v0.33.0. It can be executed as a Sonobuoy plugin to verify if the underlying cluster meets the basic needs to run KubeVirt.

Executing conformance tests for a specific release

To execute the conformance tests for a released conformance test suite, run:

KUBEVIRT_VERSION=v0.41.0
sonobuoy run --plugin https://github.com/kubevirt/kubevirt/releases/download/${KUBEVIRT_VERSION}/conformance.yaml

The execution can be monitored using the status command:

sonobuoy status
                 PLUGIN     STATUS   RESULT   COUNT
   kubevirt-conformance   complete   passed       1

Sonobuoy has completed. Use `sonobuoy retrieve` to get results.

Once the test run finishes, the result can be fetched:

sonobuoy retrieve
202008201609_sonobuoy_8f8d0b0e-1d37-485a-b61d-bf7185198fbf.tar.gz

And interpreted:

sonobuoy results 202008201609_sonobuoy_8f8d0b0e-1d37-485a-b61d-bf7185198fbf.tar.gz
Plugin: kubevirt-conformance
Status: passed
Total: 580
Passed: 1
Failed: 0
Skipped: 579

Debugging failures

In case of a failure the artifacts will be collected as well.
Find them at plugins/kubevirt-conformance/results/global/k8s-reporter of the tar file.

Executing conformance tests in the development environment

make conformance

To run without outside connectivity tests add the argument:

SKIP_OUTSIDE_CONN_TESTS=true make conformance

The following argument is only used for running the conformance tests on the Arm64 test infrastructure in KubeVirtCI. This is necessary because some specific setups, such as IPv6, are not enabled on the Arm64 test infrastructure. By adding the argument, we can skip the unsupported tests.

RUN_ON_ARM64_INFRA=true make conformance

To focus on specific tests pass KUBEVIRT_E2E_FOCUS argument:

KUBEVIRT_E2E_FOCUS=sig-network make conformance

To use on specific images tag on test suite override DOCKER_TAG argument:

DOCKER_TAG=mybuild make conformance

Generate manifests

Conformance tests plugin manifest template under manifests/release/conformance.yaml was generated with following commands:

sonobuoy gen plugin --name kubevirt-conformance --cmd /usr/bin/conformance --image IMAGE -f junit > manifests/release/conformance.yaml.in
sed -i 's#IMAGE#{{.DockerPrefix}}/conformance:{{.DockerTag}}#' manifests/release/conformance.yaml.in