1. OVERVIEW:


    This repository contains a wrapper script and configuration files to allow the user to set up 
    the avocado test framework and run a suite of tests to help verify the OpenPOWER Host OS and 
    Guest Virtual Machine (VM) stability.  The tests are integrated into the avocado framework to 
    increase the overall ease of use and to allow the open source community to easily add and run 
    new tests within the suite.

    See the REFERENCE section below for links to the avocado framework and avocado documentation.


2. OPERATING SYSTEM SUPPORT:

    A POWER8 bare-metal system running OpenPOWER Host OS is required to execute any test suite in this 
    repository (Host or Guest tests). Guest test cases were validated against the following ppc64le guests:
    - Fedora 23
    - Fedora 24
    - CentOS 7.2
    - CentOS 7.3

    Running test cases against other host and guest operating systems may result in unexpected errors.
    Proceed at your own risk.


3. PREREQUISITES:

    The "avocado-setup.py" script must always be run as root or as a user with root privileges 
    or you risk permission denied errors. 

    The script will check your environment for tooling prerequisites and issue a warning/error 
    if any prerequisites are not installed on the system and cannot be installed automatically.


    GUEST VM INSTALL MANUAL PREREQUISITES:

      Before installing the Guest VM ("--install" option used), you must complete the following manual tasks:

        1) Place the ISO image for the guest OS in the `isos/` folder.  The install option
           currently supports only the Fedora 23 and 24 OS ISO & guest.

           Download Fedora images and place them at the expected paths:
             >> curl -o isos/Fedora-Server-DVD-ppc64le-23.iso http://archives.fedoraproject.org/pub/archive/fedora-secondary/releases/23/Server/ppc64le/iso/Fedora-Server-DVD-ppc64le-23.iso
             >> curl -o isos/Fedora-Server-DVD-ppc64le-24.iso http://archives.fedoraproject.org/pub/archive/fedora-secondary/releases/24/Server/ppc64le/iso/Fedora-Server-dvd-ppc64le-24-1.2.iso

           Make sure you run `python avocado-setup.py --bootstrap` every time a new image is added or updated under `isos/` directory.
           Running bootstrap will copy all images to `data/avocado-vt/isos/linux/` directory.

           Note: This step is not required for CentOS guest Installation, as it would pick up automatically.

        2) Open firewall ports 8000-8020. They are needed by the avocado test framework 
           to host the kickstart file for the guest using the default bridge network.

           Suggested iptables command:  
             >> iptables -t filter -I INPUT -p tcp -m state --dport 8000:8020 --state NEW -j ACCEPT


4. "avocado-setup.py" SCRIPT USAGE INFORMATION:


    It is highly recommended for users to execute the following command as root when running the 
    test suite for the first time or you risk permission denied errors.  See the below information
    for details:
        >> python avocado-setup.py --run-suite host_sanity,guest_short_sanity --install


    python avocado-setup.py [-h] [--bootstrap] [--run-suite RUN_SUITE]
                        [--output-dir OUTPUTDIR] [--verbose]
                        [--only-filter ONLY_FILTER] [--no-filter NO_FILTER]
                        [--additional-args ADD_ARGS] [--guest-os GUEST_OS]
                        [--vt VT_TYPE] [--install]

    OPTIONAL ARGUMENTS:

      -h, --help                  Show this help message and exit
      --bootstrap                 Prepare the environment for test
      --run-suite RUN_SUITE       Indicate which test suite(s) to run
      --output-dir OUTPUTDIR      Specify the custom test results directory
      --input-file INPUTFILE      Specify input file for custom mux values for host tests
      --verbose                   Enable verbose output on the console
      --only-filter ONLY_FILTER   Add additional specific avocado tests or features to the guest test suite.
      --no-filter NO_FILTER       Exclude specific avocado tests or features from the guest test suite.
      --additional-args ADD_ARGS  Pass additional arguments to the command
      --guest-os GUEST_OS         Specify the Guest OS. Default: Fedora.24.ppc64le
      --vt VT_TYPE                Provide VT: qemu or libvirt  Default: libvirt
      --install                   Install the Guest VM. Highly recommended for guest tests.
      --no-download               Use this option to skip the guest image download (using preinstalled guest image)
      --no-deps-check             Force wrapper to skip check for dependancy packages
      --clean                     Remove/Uninstall autotest and avocado from system after test completion


    ARGUMENT DETAILS:

      --bootstrap
        Use this option to bootstrap the environment so that all of the required repositories 
        (avocado, avocado-vt, avocado-misc-tests) are downloaded, installed, and configured.


          Example command to bootstrap the test suite: 
              >> python avocado-setup.py --bootstrap


      --run-suite RUN_SUITE
        Use this option to reference one of the files in the config/tests/host/ or
        config/tests/guest/{libvirt, qemu} folders. The RUN_SUITE value must be
        prefixed based on which folder (guest or host) the config file is located
        in.  Multiple RUN_SUITE values must be separated by a comma.

        Preconfigured possible RUN_SUITE values:

          host_sanity -->  Uses the preconfigured file:
                           config/tests/host/sanity.cfg
                           This file lists the specific scripts that are run from
                           the Avocado Test Framework "avocado-misc-tests"
                           repository.

            Example command to run the preconfigured Host OS test suite only: 
                >> python avocado-setup.py --run-suite host_sanity


          guest_sanity --> Uses the preconfigured file:
                           config/tests/guest/libvirt/sanity.cfg
                           This file runs a full series of regular and error
                           injection tests against the specified guest VM using
                           various virsh commands. This set of tests can take 4-5
                           hours to complete.  The "variants:" section of the
                           config file shows which commands are tested.

          Example command to install a new Guest VM and run the preconfigured full 
          guest test suite against it: 
              >> python avocado-setup.py --run-suite guest_sanity --install


          guest_short_sanity --> Uses the preconfigured file:
                                 config/tests/guest/libvirt/short_sanity.cfg
                                 This file runs a shorter series of tests against
                                 the specified guest VM using various virsh
                                 commands. This set of tests can take 1-2 hours to
                                 complete.  The "variants:" section of the config
                                 file shows which commands are tested.

          Example command to install a new Guest VM and run the preconfigured short 
          guest test suite against it: 
              >> python avocado-setup.py --run-suite guest_short_sanity --install


      --output-dir
        Use this option to provide a custom directory for avocado test results.
        The avocado test results are stored in the /current_run_path/results folder by default.

        Example command to run the host and guest test suite and output the test results to the
        /tmp folder:
            >> python avocado-setup.py --run-suite guest_sanity,host_sanity --output-dir /tmp

        NOTE: The avocado-setup.py log will always be generated at 
              /current_run_path/avocado-wrapper.log


      --input-file
        Use this option to specify input file for custom mux values for host tests.
        This is a config file format, with config section indicating the test cfg file name,
        and the key value pairs indicating the yaml parameter to be changed and its corresponding
        value, respectively.
        A key (yaml parameter to be changed) in an input file replaces the value (yaml parameter's
        value to be changed) of that yaml parameter's value in all the yaml files specified in
        that test cfg

        Example:
        Consider config/tests/host/example.cfg
        It contains 1 yaml file, namely avocado-misc-tests/io/disk/ioping.py.data/ioping.yaml
        Now, it has yaml parameters like mode, count, deadline, period, disk, etc.
        Suppose user wants to change only 3 of those values, say disk, wsize and period, user can
        have that alone in our input file.
        Refer input_example.txt for this example.


      --verbose
        Use this option to display test verbose output on the console.

        Example command to install a new guest VM, run the full host & guest test suites, and 
        display verbose output on the console: 
            >> python avocado-setup.py --run-suite guest_sanity,host_sanity --install --verbose


      --only-filter
        Use this option to add additional specific avocado tests or features to the guest test suite. 
        This filter option is for advanced users of the avocado test framework who want to 
        temporarily adjust the tests being run without having to create or change configuration 
        files.

        Example command to run the host & guest test suites and add "scsi virtio_net" filter to 
        the guest test suite: 
            >> python avocado-setup.py --run-suite guest_sanity,host_sanity --install --only-filter "scsi virtio_net"

        IMPORTANT: Ensure you do not include a filter in the command that is a duplicate of a filter 
                   in the guest test suite being run.  This could cause avocado test errors. 


      --no-filter
        Use this option to remove specific avocado tests or features from the guest test suite. 
        This filter option is for advanced users of the avocado test framework who want to 
        temporarily adjust the tests being run without having to create or change configuration 
        files.

        Example command to run the host & guest test suites and remove the "scsi virtio_net" filter 
        from the guest test suite: 
            >> python avocado-setup.py --run-suite guest_sanity,host_sanity --install --no-filter "scsi virtio_net"

        IMPORTANT: Ensure you do not include a filter in the command that is a duplicate of a filter 
                   in the guest test suite being run.  This could cause avocado test errors. 


      --additional-args
        Use this option to pass any additional avocado arguments to run tests. A
        preconfigured set of Host or Guest VM tests has already been provided and
        will run based on the value chosen for --run-suite. This additional option
        is for advanced users of the avocado test framework who want to run
        additional tests within that framework.


      --guest-os
        Use this option to specify the guest os for the guest tests to run against.
        By default, the value is Fedora.24.ppc64le.
        Supported Guests: Fedora.24.ppc64le, Fedora.23.ppc64le, CentOS.7.2.ppc64le

        IMPORTANT: It is highly recommended that you only specify a guest os from the list
                   of supported guest vm's in the above Operating System Support section.


      --vt {qemu or libvirt}
        Use this option to choose either the libvirt or qemu folder when guest_sanity 
        is the RUN_SUITE value.  By default, the libvirt folder is used.


      --install
        Use this option to install the guest VM before or during the first time that you 
        run any guest test suite.  Advanced avocado users may install their guest vm
        separately, but the install must follow the strict avocado configuration rules.
      

        IMPORTANT: Complete all Guest VM Install prerequisites listed in the Prerequisites
                   section above BEFORE running this script with the "--install" option.
           
           
      --no-download
        Use this option to skip the guest image download. This helps to save time when you are 
        using an already preinstalled guest image.
      
      
      --no-deps-check
        Force wrapper to skip check for dependancy packages.  This helps to save time when re-running 
        tests on a system where prereqs have already been checked once.
      
      
      --clean
        Remove/Uninstall autotest and avocado from system after test completion.  


5. CUSTOMIZING TEST SUITES

   The Host and Guest sanity suites were created to include a varied collection of tests to validate new Host OS installations. 
   There are additional tests that were not included in the sanity suites that users can optionally add if they wish.
   
   Adding Guest Sanity Tests:
      You must be a proficient/advanced Avocado test suite user to correctly customize the Guest sanity suites since the syntax of these
      configuration files is very specific.
   
   Adding Host Sanity Tests:
      The Host sanity tests are listed in the config/tests/host/sanity.cfg file in this repository.  These are tests used directly 
      from the avocado-misc-tests respository (see link in the REFERNCE LINKS section below).  If you want to run additional 
      tests from this repository, simply add a new line in the sanity.cfg file with the location of the file within the 
      avocado-misc-tests repository.
      
      For Example:  
         - A user wishes to run ras.py test in the generic folder of the avocado-misc-tests repository. The purpose of this 
         test is documented within the file itself.  
         - The exact location of this test in the repository is "generic/ras.py".
         - The user downloads this tests repository onto their system and edits the config/tests/host/sanity.cfg file.
         - The user adds a line "avocado-misc-tests/generic/ras.py" (without the quotes) to the end of the config/tests/host/sanity.cfg 
         file in this tests repository and saves the file.
         - The user runs the host_sanity suite using the avocado-setup.py script explained in Section 4 above.
         - If the user wishes to run a test with yaml file inputs, the yaml file can be specified in the same line in the cfg file,
         separated by a space. Refer config/tests/host/example.cfg for example.
      
      
6. REFERENCE LINKS

    Avocado Test Framework repositories on GitHub:
      https://github.com/avocado-framework

    Avocado Test Framework documenation:
      http://avocado-framework.readthedocs.org

    Community maintained Avocado Tests repository (avocado-misc-tests):
      https://github.com/avocado-framework-tests/avocado-misc-tests

    OpenPOWER Host OS repositories on GitHub:
      https://github.com/open-power-host-os


7. KNOWN ISSUES AND LIMITATION

    Refer to the Issues section of this repository, linked below, for details on bugs, limitations, future enhancements, and investigations:
      https://github.com/open-power-host-os/tests/issues?q=is%3Aopen+is%3Aissue