Quick Start

This document describes how to setup an environment and how to run test cases

Install test cases within a virtualenv

The safest way to run test cases is to do it within a Virtualenv. Here we are goint to see how to setup an environment with all test case dependencies.

In RHEL, CentOS or Fedora install the following packages:

sudo dnf install -y gcc git python3 python3-devel python3-pip which findutils

In Debian or Ubuntu install following packages:

sudo apt update
sudo apt install -y gcc git python3 python3-dev python3-pip

Ensure Pip is up-to-date:

python3 -m pip install --upgrade --user pip

Ensure Tox is installed and up-to-date:

python3 -m pip install --upgrade --user setuptools virtualenv wheel tox

Get source code using Git and enter into Tobiko source folder:

git clone https://opendev.org/x/tobiko.git
cd tobiko

Install remaining binary packages:

tools/install-bindeps.sh

Crate the virtual environment with Tox:

python3 -m tox -e py3 --notest

In case you want to activate the virtual environment you can then type:

. .tox/py3/bin/activate

At this point the environment should have all dependencies installed for running test cases.

Configure tobiko with tobiko.conf

Tobiko tries to load the ‘tobiko.conf’ file from one of the below locations:

  • current directory:

    ./tobiko.conf
    
  • user home directory:

    ~/.tobiko/tobiko.conf
    
  • system directory:

    /etc/tobiko/tobiko.conf
    

Configure Logging Options

Tobiko can configure a logging system to write messages to a log file. You can edit the below options in tobiko.conf to enable it as below:

[DEFAULT]
# Whenever to allow debugging messages to be written out or not
debug = true

# Name of the file where log messages will be appended.
log_file = tobiko.log

# The base directory used for relative log_file paths.
log_dir = .

The file ‘tobiko.log’ is the default file where test cases and the Python framework are going to write their logging messages. By setting debug as ‘True’ you ensure that messages with the lowest logging level are written there (DEBUG level). The log_file location specified above is relative to the tobiko.conf file location. In this example it is the Tobiko source files’ directory itself.

Configure Tobiko Credentials

Tobiko needs to have Keystone credentials in order to run the OpenStack test cases. We are going to assume you are using one of the two OpenStack distributions supported by Tobiko:

  • DevStack

  • TripleO

Get credentials from a DevStack host

Copy the clouds.yaml file from your remote cloud to any one of the below locations:

  • Tobiko source files directory

  • ~/.config/openstack

  • /etc/openstack

The clouds.yaml file contains valid Keystone credentials.

You can copy the file in the following way:

ssh <... connection options here ...> cat /etc/openstack/clouds.yaml > clouds.yaml

Get credentials from a TripleO undercloud host

Tobiko test cases will be able to setup some type of SSH tunneling to be able to reach the remote cloud, but for achieving it you are required to be able to connect to a remote SSH server that is able to connect the OpenStack services and hosts. We will refer to that server as the SSH proxy host.

Tobiko test cases will execute some commands on the SSH proxy host (like ping, nc, curl, etc). Those commands need to have direct connectivity to target cloud.

Test cases will use Python REST API clients configured to make HTTP requests coming out from such SSH server (mainly by using nc command) or SSH server direct connect feature.

Test cases will make all SSH connection to cloud nodes by using this SSH proxy host.

To resume the purpose of the SSH proxy, all network packages sent by Tobiko test cases to the tested cloud will come from the SSH proxy host, while all Tobiko test cases will be executed from the developer workstation.

SetUp SSH public key to connect to remote cloud

First of all we need to make sure we can connect to the SSH proxy server without requiring any password. We therefore need to have a local SSH key pair to be used by tobiko. This key by default is the same default one used by openSSH client: - default SSH private key filename: ~/.ssh/id_rsa - default SSH public key filename: ~/.ssh/id_rsa.pub

Note: In case that you already have a public key which does not require a password (it requires an empty passphrase), you can skip the ssh keypair creation (continue with defining your ssh variables ).

In case that you don’t, to avoid having problems with other uses of the same file, let’s instead create our SSH key pair only for Tobiko in a sub-folder near to your tobiko.conf

Make sure you run the following commands in the tobiko directory. Ensure we do have this key pair on your workstation by typing:

mkdir -p .ssh
chmod 700 .ssh
ssh-keygen -v -f .ssh/id -N ''
chmod 600 .ssh/id .ssh/id.pub

Define the below SSH variables to later connect to your SSH server:

SSH_HOST=<your-ssh-proxy-address>
SSH_USERNAME=<your-ssh-proxy-user>

For example:

SSH_HOST=seal100.your.domain
SSH_USERNAME=root

Copy your SSH public key to your remote server:

ssh-copy-id -i .ssh/id "${SSH_USERNAME}@${SSH_HOST}"

Make sure the SSH key pair is working:

ssh -i .ssh/id "${SSH_USERNAME}@${SSH_HOST}" hostname

Now let’s make sure Tobiko test cases will use the SSH key pair to connect to your SSH remote host. Add the following lines to tobiko.conf file:

[ssh]
proxy_jump = SSH_USERNAME@SSH_HOST

For example:

[ssh]
proxy_jump = root@seal100.your.domain
#proxy_jump = root@seal99.your.domain
#proxy_jump = root@seal98.your.domain

Tip

You could have multiple hosts in your tobiko.conf [ssh] section, where the ones you are not currently using are commented (as appear above). Moving your tobiko tests from one host to another will be as easy as commenting the host you are stop using and uncommenting the one you are start using (remember to copy your SSH key to your other remote hosts as well).

Create a virtual environment with tox

To execute commands from a virtualenv created by Tox, add the virtualenv name to tox.ini envlist variable in the following way (in this example, the virtual environment’s name is ‘venv’):

[tox]

envlist = other_environment_variables,venv

Run your commands as below:

tox -e venv -- <your-commands>

For example:

tox -e venv -- openstack network list

Run Test Cases

The next section is a quick guide about running some test cases. For more information, please see our Tobiko Test Cases Execution Guide

Before running test cases, make sure you configure tobiko logging according to your needs.

Note

Unlike other testing frameworks, Tobiko does not delete its resources after test cases finish their execution. You may clean up tobiko workloads after the execution manually, for example heat stacks and glance images.

Run Scenario Test Cases

Scenario test cases are used to create workloads that simulate real-world use of OpenStack. They create networks, virtual machines, ports, routers, etc. They also validate that these workloads are functioning.

Running Tobiko scenario test cases using Tox (may take some minutes to complete):

tox -e scenario

Scenario test cases are also used to check that previously created resources are still up and working as expected. To ensure test cases will not create those resources again we can set TOBIKO_PREVENT_CREATE environment variable before re-running test cases:

TOBIKO_PREVENT_CREATE=yes tox -e scenario

Run Disruptive Test Cases

Disruptive (or faults) test cases are used for testing that after inducing some critical disruption to the operation of the cloud, the services can get back to the expected state after a while. To execute them you can type:

tox -e faults

The faults induced by these test cases could be cloud nodes reboot, OpenStack services restart, virtual machines migrations, etc.

Please note that while scenario test cases are being executed in parallel (to speed up test case execution), disruptive test cases are only executed sequentially. This is because the operations executed by such cases could break some functionality for a short time and alter the regular state of the system which may be assumed by other test cases to be executed.

Run the Tobiko Workflow

Scenario and disruptive test cases, which are being executed in a specific sequence, could be used to uncover more issues with the cloud than disruptive test cases alone.

  • First ensure there are workloads properly running by running scenario test cases:

    tox -e scenario
    
  • Next we could execute disruptive test cases to “stress” the cloud:

    tox -e faults
    
  • Finally we might re-run scenario test cases to check that everything is still running as expected:

    TOBIKO_PREVENT_CREATE=yes tox -e scenario
    

Test Cases Report Files

After executing test cases we can view the results in greater detail via a small set of files:

  • test_results.html: A user-browsable HTML view of test case results.

  • test_results.log: A log file with logging traces collected from every individual test case.

  • test_results.subunit: The original subunit binary file generated by test runner.

  • test_results.xml: An XML Junit file to be used, for example, to show test cases result by Jenkins CI server.

The names of the above files can be changed from the default value (test_results) to a custom one by setting the TOX_REPORT_NAME environment variable.

The above files are saved into a folder that can be specified with TOX_REPORT_DIR environment variable.

By default the full path of the report directory is made from the below:

{toxinidir}/report/{envname}