Tobiko Test Cases Execution Guide

This document describes how to execute Tobiko test cases.

Prepare Your System

Before running Tobiko test cases, you need to be sure you are doing it from Tobiko source files folder and that you have activated a virtualenv where Tobiko is, and its requirements are installed. Please refer to Install and Configuration to know how to setup your system before running test cases.

Run Tobiko Test Cases

Run Tobiko specific test cases

In the first terminal, execute some Tobiko test cases as below:

To run test cases you need a test runner able to execute Python test cases. Test cases delivered with Tobiko has been tested using stestr

From the Tobiko source folder you can run scenario test cases using the below command:

pytest tobiko/tests/scenario/

You could also use tox to run test cases:

tox -e <environment_variable> -- path/to/test/module

For example:

tox -e scenario -- tobiko/tests/scenario/neutron/test_router.py

Note that with tox, the <environment_variable> should match the directory where the test is (if the test is inside the ‘scenario’ directory, the environment variable has to be scenario).

You can also run only a class of test cases by running:

tox -e <tox-env-list> -- path/to/test/module::class

You can run only a specific test case (a method) by running:

tox -e <tox-env-list> -- path/to/test/module::class::test_case

Run Scenario Test Cases

Scenario test cases are used to create workloads that simulate real-world use of OpenStack. They create networks, virtual machines, ports, routers, etc. They also validate that these workloads are functioning.

Running Tobiko scenario test cases using Tox (may take some minutes to complete):

tox -e scenario

Scenario test cases are also used to check that previously created resources are still up and working as expected. To ensure test cases will not create those resources again we can set TOBIKO_PREVENT_CREATE environment variable before re-running test cases:

TOBIKO_PREVENT_CREATE=yes tox -e scenario

Running Disruptive Test Cases

Disruptive (or faults) test cases are used for testing that after inducing some critical disruption to the operation of the cloud, the services can get back to the expected state after a while. To execute them you can type:

tox -e faults

The faults induced by these test cases could be cloud nodes reboot, OpenStack services restart, virtual machines migrations, etc.

Please note that while scenario test cases are being executed in parallel (to speed up test case execution), disruptive test cases are only executed sequentially. This is because the operations executed by such cases could break some functionality for a short time and alter the regular state of the system which may be assumed by other test cases to be executed.

Run the Tobiko Workflow

Scenario and disruptive test cases, which are being executed in a specific sequence, could be used to uncover more issues with the cloud than disruptive test cases alone.

  • First ensure there are workloads properly running by running scenario test cases:

    tox -e scenario
    
  • Next we could execute disruptive test cases to “stress” the cloud:

    tox -e faults
    
  • Finally we might re-run scenario test cases to check that everything is still running as expected:

    TOBIKO_PREVENT_CREATE=yes tox -e scenario
    

Test Cases Report Files

After executing test cases we can view the results in greater detail via a small set of files:

  • test_results.html: A user-browsable HTML view of test case results.

  • test_results.log: A log file with logging traces collected from every individual test case.

  • test_results.subunit: The original subunit binary file generated by test runner.

  • test_results.xml: An XML Junit file to be used, for example, to show test cases result by Jenkins CI server.

The names of the above files can be changed from the default value (test_results) to a custom one by setting the TOX_REPORT_NAME environment variable.

The above files are saved into a folder that can be specified with TOX_REPORT_DIR environment variable.

By default the full path of the report directory is made from the below:

{toxinidir}/report/{envname}