Tobiko Test Cases Execution Guide

This document describes how to execute Tobiko test cases.

See also

For a quick and simpler start you can jump to the Quick Start.

To install Tobiko inside a virtualenv please read Install.

To configure Tobiko please read Configuration.

Prepare Your System

Before running Tobiko test cases, you need to be sure you are doing it from Tobiko source files folder and that you have activated a virtualenv where Tobiko is, and its requirements are installed. Please refer to Install and Configuration to know how to setup your system before running test cases.

Prepare some logging (Optional and recommended)

To see if we are now being able to execute Tobiko test cases, please open a new terminal and keep it open, where you could watch tobiko.log receive logs on real time. Change directory to reach the directory where tobiko.log file is and run the following command:

tail -F tobiko.log

Prepare advanced image to be used by Tobiko (optional)

Some of the Tobiko tests, like e.g. tobiko/tests/scenario/neutron/test_network.py::BackgroundProcessTest. In the CI jobs, this is done by the tobiko-download-images Ansible role but if Tobiko tests are executed locally it may be needed to prepare such image and upload it to Glance. This can be done with simple script:

$ wget https://download.fedoraproject.org/pub/fedora/linux/releases/41/Cloud/x86_64/images/Fedora-Cloud-Base-Generic-41-1.4.x86_64.qcow2 \
    -O /tmp/Fedora-Cloud-Base-Generic-41-1.4.x86_64.qcow2

$ cat << EOF > /tmp/iperf3-server.service
[Unit]
Description=iperf3 server on port 5201
After=syslog.target network.target
[Service]
ExecStart=/usr/bin/iperf3 -s -p 5201
Restart=always
User=root
[Install]
WantedBy=multi-user.target
EOF

$ cat << EOF > /tmp/nginx_id.conf
server{
    listen 80;
    listen [::]:80;
    location /id { add_header Content-Type text/plain; return 200 '$hostname';}
}
EOF

$ cat << EOF > /tmp/config
SELINUX=permissive
SELINUXTYPE=targeted
EOF

$ TMPPATH=$(mktemp)
$ cp /tmp/Fedora-Cloud-Base-Generic-41-1.4.x86_64.qcow2 $TMPPATH
$ LIBGUESTFS_BACKEND=direct
$ virt-customize -a $TMPPATH \
      --copy-in /tmp/config:/etc/selinux \
      --firstboot-command 'sh -c "nmcli connection add type vlan con-name vlan101 ifname vlan101 vlan.parent eth0 vlan.id 101 ipv6.addr-gen-mode default-or-eui64"' \
      --install iperf3,iputils,nmap,nmap-ncat,nginx \
      --copy-in /tmp/nginx_id.conf:/etc/nginx/conf.d \
      --run-command 'systemctl enable nginx' \
      --copy-in /tmp/iperf3-server.service:/etc/systemd/system \
      --run-command 'systemctl enable iperf3-server' \
      --root-password password:tobiko \
      --selinux-relabel

$ mv TMPPATH /tmp/Fedora_customized.qcow2
$ chmod a+r /tmp/Fedora_customized.qcow2

To use this image in Tobiko, it needs to be set in the Tobiko config file:

$ cat tobiko.conf
[advanced_vm]
image_name = Fedora_customized
image_url=file:///tmp/Fedora_customized.qcow2
container_format=bare
disk_format=qcow2
username=fedora

Run Tobiko Test Cases

Run Tobiko specific test cases

In the first terminal, execute some Tobiko test cases as below:

To run test cases you need a test runner able to execute Python test cases. Test cases delivered with Tobiko has been tested using stestr

From the Tobiko source folder you can run scenario test cases using the below command:

pytest tobiko/tests/scenario/

You could also use tox to run test cases:

tox -e <environment_variable> -- path/to/test/module

For example:

tox -e scenario -- tobiko/tests/scenario/neutron/test_router.py

Note that with tox, the <environment_variable> should match the directory where the test is (if the test is inside the ‘scenario’ directory, the environment variable has to be scenario).

You can also run only a class of test cases by running:

tox -e <tox-env-list> -- path/to/test/module::class

You can run only a specific test case (a method) by running:

tox -e <tox-env-list> -- path/to/test/module::class::test_case

Run Scenario Test Cases

Scenario test cases are used to create workloads that simulate real-world use of OpenStack. They create networks, virtual machines, ports, routers, etc. They also validate that these workloads are functioning.

Running Tobiko scenario test cases using Tox (may take some minutes to complete):

tox -e scenario

Scenario test cases are also used to check that previously created resources are still up and working as expected. To ensure test cases will not create those resources again we can set TOBIKO_PREVENT_CREATE environment variable before re-running test cases:

TOBIKO_PREVENT_CREATE=yes tox -e scenario

Running Disruptive Test Cases

Disruptive (or faults) test cases are used for testing that after inducing some critical disruption to the operation of the cloud, the services can get back to the expected state after a while. To execute them you can type:

tox -e faults

The faults induced by these test cases could be cloud nodes reboot, OpenStack services restart, virtual machines migrations, etc.

Please note that while scenario test cases are being executed in parallel (to speed up test case execution), disruptive test cases are only executed sequentially. This is because the operations executed by such cases could break some functionality for a short time and alter the regular state of the system which may be assumed by other test cases to be executed.

Run the Tobiko Workflow

Scenario and disruptive test cases, which are being executed in a specific sequence, could be used to uncover more issues with the cloud than disruptive test cases alone.

• First ensure there are workloads properly running by running scenario test cases:

  tox -e scenario
  

Note

As second step we may, instead, update or upgrade OpenStack nodes.

• Next we could execute disruptive test cases to “stress” the cloud:

  tox -e faults
  

• Finally we might re-run scenario test cases to check that everything is still running as expected:

  TOBIKO_PREVENT_CREATE=yes tox -e scenario
  

Running Tobiko Test Cases in the Podified environment using ci-framework

Tobiko can be run through the Test-operator. This can be done manually, following the Test-operator documentation but it may require some additional preparation tasks, like e.g. prepare ssh keys which are required by the Tobiko tests. To avoid extra manual steps, Tobiko can be run using ci-framework.

Clone ci-framework on the host with access to the OpenShift API:

$ git clone https://github.com/openstack-k8s-operators/ci-framework

Install required ansible-galaxy dependencies:

$ cd ci-framework
$ ansible-galaxy install -r requirements.yml

Additionally create directory where ci-framework and Test-operator will store some data. In the example below it is /tmp/ci-fmw:

$ mkdir /tmp/ci-fmw

Create Tobiko custom variables file. It will be placed in ci-framework/custom/tobiko_vars.yaml and its content should be something like:

$ cat custom/tobiko_vars.yaml
---
ansible_user_dir: '/tmp/ci-fmw'
cifmw_openshift_kubeconfig: '/home/zuul/.crc/machines/crc/kubeconfig'
cifmw_run_tests: true
cifmw_run_test_role: test_operator
cifmw_test_operator_stages:
 - name: tobiko-stage-1
   type: tobiko
   test_vars:
     cifmw_test_operator_tobiko_image: 'quay.io/podified-antelope-centos9/openstack-tobiko'
     cifmw_test_operator_tobiko_image_tag: 'current-podified'
     cifmw_test_operator_tobiko_testenv: 'scenario -- tobiko/tests/scenario/neutron/test_network.py::BackgroundProcessTest'
     cifmw_test_operator_tobiko_debug: true
cifmw_test_operator_cleanup: false

Most important lines in this file are cifmw_test_operator_tobiko_testenv where Tobiko test environment and even specific test(s) which should be run can be specified. There is also cifmw_test_operator_tobiko_debug which if set to True will prevent Tobiko POD to be deleted. After tests will finish, POD will still be there and it will be possible to connect to it and run or even modify Tobiko tests as needed.

You could test your tobiko gerrit patches by adding the following parameter to your tobiko_vars.yaml file:

cifmw_test_operator_tobiko_patch:
  repository: "https://review.opendev.org/x/tobiko"
  refspec: "refs/changes/03/955203/8"

Now you can run tobiko tests using the 08-run-tests.yml playbook:

$ ansible-playbook -i <ci-framework inventory file> -vv \
  -e @custom/tobiko_vars.yaml
  playbooks/08-run-tests.yml --flush-cache $@

And watch tobiko test pod being created:

$ watch 'oc get pods -A | grep -e tobiko -e test-operator'

Test results should be found under:: {{ ansible_user_dir }}/ci-framework-data/tests/test-operator/ directory

Cleanup:

If cifmw_test_operator_cleanup was set to false, you may want to clean up all the resources created by the test operator, before you run tobiko tests again:

$ oc -n openstack delete tempest/tempest-tests-tempest
$ oc -n openstack delete tobiko/tobiko-tests-tobiko  # Continue for other test resources (e. g. horizon), if deployed.
$ oc -n openstack get pods -o name | grep test-operator-logs-pod | xargs -r oc -n openstack delete

Prepare Tobiko Test Pod to run Tobiko

Prepare tobiko test pod to run tobiko tests

You may want to run tests from the tobiko test pod. To enable that, some more steps are required:

Log in to the tobiko test pod:

$ oc rsh <tobiko-test pod name> /bin/bash
bash-5.1$ cd ~
bash-5.1$ pwd
/var/lib/tobiko
bash-5.1$ cd tobiko/

Export the OS_CLOUD value:

$ export OS_CLOUD=default

To run tests dedicated for podified environment you will need to authenticate the openshift cluster from the tobiko pod before running them:

$ cp /var/lib/tobiko/.kube/config /tmp/kubeconfig
$ sudo chmod a+w /tmp/kubeconfig
$ export KUBECONFIG=/tmp/kubeconfig
$ oc login -u kubeadmin -p ...

Now, when the pod is ready to run tobiko, you can modify the code and test it locally. For example:

$ tox -e scenario
$ tox -e sanity -- tobiko/tests/sanity/nova/test_server.py::CrateDeleteServerStackTest::test_1_create_server
$ tox -e podified_ha_faults -- tobiko/tests/faults/podified/ha/test_cloud_recovery.py::DisruptPodifiedNodesTest::test_remove_one_grastate_galera

To monitor the progress of the tests, log in the test pod from another terminal window and run:

$ cd ~/tobiko
$ tail -F tobiko.log

Tip: You may want to test your tobiko code locally using ipython. To install ipython, first, create tox virtual environment:

$ cd ~/tobiko/
$ tox -e py3 -- notests

Then, install and run ipython:

$ .tox/py3/bin/pip install ipython
$ .tox/py3/bin/ipython

Tip You can use the openstack client binary that is installed under the tobiko tox virtual environment:

$ .tox/py3/bin/openstack --insecure service list

Or, in more common way, by accessing the openstackclient pod:

$ oc exec openstackclient -- openstack server list

Test Cases Report Files

After executing test cases we can view the results in greater detail via a small set of files:

• test_results.html: A user-browsable HTML view of test case results.

• test_results.log: A log file with logging traces collected from every individual test case.

• test_results.subunit: The original subunit binary file generated by test runner.

• test_results.xml: An XML Junit file to be used, for example, to show test cases result by Jenkins CI server.

The names of the above files can be changed from the default value (test_results) to a custom one by setting the TOX_REPORT_NAME environment variable.

Legend

{toxinidir} stands for the Tobiko source files directory.

{envname} is the name of the Tox enviroment to be executed (IE scenario, faults, etc.)

The above files are saved into a folder that can be specified with TOX_REPORT_DIR environment variable.

By default the full path of the report directory is made from the below:

{toxinidir}/report/{envname}