Tobiko Test Cases Execution Guide
This document describes how to execute Tobiko test cases.
Prepare Your System
Before running Tobiko test cases, you need to be sure you are doing it from Tobiko source files folder and that you have activated a virtualenv where Tobiko is, and its requirements are installed. Please refer to Install and Configuration to know how to setup your system before running test cases.
Prepare some logging (Optional and recommended)
To see if we are now being able to execute Tobiko test cases, please open a new terminal and keep it open, where you could watch tobiko.log receive logs on real time. Change directory to reach the directory where tobiko.log file is and run the following command:
tail -F tobiko.log
Run Tobiko Test Cases
Run Tobiko specific test cases
In the first terminal, execute some Tobiko test cases as below:
To run test cases you need a test runner able to execute Python test cases. Test cases delivered with Tobiko has been tested using stestr
From the Tobiko source folder you can run scenario test cases using the below command:
pytest tobiko/tests/scenario/
You could also use tox to run test cases:
tox -e <environment_variable> -- path/to/test/module
For example:
tox -e scenario -- tobiko/tests/scenario/neutron/test_router.py
Note that with tox, the <environment_variable> should match the directory where the test is (if the test is inside the ‘scenario’ directory, the environment variable has to be scenario).
You can also run only a class of test cases by running:
tox -e <tox-env-list> -- path/to/test/module::class
You can run only a specific test case (a method) by running:
tox -e <tox-env-list> -- path/to/test/module::class::test_case
Run Scenario Test Cases
Scenario test cases are used to create workloads that simulate real-world use of OpenStack. They create networks, virtual machines, ports, routers, etc. They also validate that these workloads are functioning.
Running Tobiko scenario test cases using Tox (may take some minutes to complete):
tox -e scenario
Scenario test cases are also used to check that previously created resources are still up and working as expected. To ensure test cases will not create those resources again we can set TOBIKO_PREVENT_CREATE environment variable before re-running test cases:
TOBIKO_PREVENT_CREATE=yes tox -e scenario
Running Disruptive Test Cases
Disruptive (or faults) test cases are used for testing that after inducing some critical disruption to the operation of the cloud, the services can get back to the expected state after a while. To execute them you can type:
tox -e faults
The faults induced by these test cases could be cloud nodes reboot, OpenStack services restart, virtual machines migrations, etc.
Please note that while scenario test cases are being executed in parallel (to speed up test case execution), disruptive test cases are only executed sequentially. This is because the operations executed by such cases could break some functionality for a short time and alter the regular state of the system which may be assumed by other test cases to be executed.
Run the Tobiko Workflow
Scenario and disruptive test cases, which are being executed in a specific sequence, could be used to uncover more issues with the cloud than disruptive test cases alone.
First ensure there are workloads properly running by running scenario test cases:
tox -e scenario
Next we could execute disruptive test cases to “stress” the cloud:
tox -e faults
Finally we might re-run scenario test cases to check that everything is still running as expected:
TOBIKO_PREVENT_CREATE=yes tox -e scenario
Test Cases Report Files
After executing test cases we can view the results in greater detail via a small set of files:
test_results.html: A user-browsable HTML view of test case results.
test_results.log: A log file with logging traces collected from every individual test case.
test_results.subunit: The original subunit binary file generated by test runner.
test_results.xml: An XML Junit file to be used, for example, to show test cases result by Jenkins CI server.
The names of the above files can be changed from the default value (test_results) to a custom one by setting the TOX_REPORT_NAME environment variable.
The above files are saved into a folder that can be specified with TOX_REPORT_DIR environment variable.
By default the full path of the report directory is made from the below:
{toxinidir}/report/{envname}