Document History 4

DRIVER NAME

Partner Name

Test Plan

Document History 4

1 Test strategy 5

2 Functional Testing 5

2.1 Health Check 5

2.1.1 How to use Health Checks 5

2.2 Tempest 5

2.2.1 Tips on Using Tempest 6

2.3 Rally 6

2.3.1 Tips on Using Rally 7

3 Non-Functional Testing 7

3.1 Performance testing 7

3.1.1 API Benchmark – Rally 7

3.1.2 Shaker — Network Performance 7

3.1.2.1 Tips on Using Shaker 7

3.1.3 Storage Performance 8

3.1.3.1 Uploading Images to Glance 8

3.1.3.2 IOPS for block devices 8

4 Resiliency testing 8

5 Appendices 9

Appendix 5.1 - List of Health Check Tests 9

Sanity group 9

Smoke group 10

HA group 10

Platform group 11

Cloud validation group 11

Appendix 5.2 - List of Tempest Suites (classes) 11

Appendix 5.3 - List of Rally tests 13

Ceilometer 13

Cinder 13

Swift 13

Heat 13

Glance 13

Keystone 13

Murano 14

Nova 14

Neutron 14

Appendix 5.4 - List of Shaker Tests 14

Appendix 5.5 - Configuration File for fio Utility 15

Appendix 5.6 - Openstack Components Used for HA Testing 17

Appendix 5.7 - Example of tempest.conf 18

Appendix 5.8 Example of rally config file 20

Document History

Version / Revision Date / Description
0.1 / DD-MM-YYYY / Initial Version

1 Test strategy

The working cloud can be measured from the three standpoints: functionality, performance, and resiliency. Functional tests verify that all necessary API functionality is present and works as expected. Performance testing shows how effectively the cloud is able to answer API requests and that the network and storage subsystems meet baseline requirements. Resiliency testing evaluates the reliability and fault tolerance of the cloud, and measures the impact of different kinds of failures.

2 Functional Testing

Mirantis recommends using the following tooling.

2.1 Health Check

Mirantis OpenStack (MOS) has built-in system called Health Check for quick, high-level verification that a cloud is working properly. Health Check tests the basic operations of the OpenStack API, validates HA functionality (checks MySQL and RabbitMQ cluster status) and performs other critical functions. A list of the available tests can be found in Appendix 5.1 and on github.

2.1.1 How to use Health Checks

There are two ways to launch the Health Checks:

· From the Fuel Master Web UI (Health Check tab).

· From the Fuel Master node CLI by running the <fuel --env end_id health --check smoke,ha,etc> command (the OSTF service runs inside the Docker container).

Please note that the logs are stored in the /var/log/ostf/ folder of the Fuel Master node.

2.2 Tempest

The Openstack community has developed two instruments for functional tests – Tempest and Rally. Tempest is an original test framework aimed to cover all possible API functionality available in Openstack components or added by third-party modules/drivers/plugins. A list of Tempest tests is in Appendix 5.2. Mirantis recommends using MOS-Tempest-runner to easily launch this tool. All Tempest tests will be run except for a small number of tests known as ‘should-fail’ for a particular MOS version. Tempest also covers negative scenarios such as authorization violation or parameter mismatches.

2.2.1 Tips on Using Tempest

· WARNING: It is potentially dangerous to use the mos-tempest-runner script to run Tempest against PRODUCTION OpenStack clouds. Doing so may break the cluster under test in ways that are difficult to diagnose and repair. Please note that mos-tempest-runner was initially designed to run Tempest on CI and test OpenStack environments, only.

· Mos-tempest-runner must be run on the Fuel master node only. It is best to run it from a virtualenv.

· The latest oslo.log version supported by Tempest may not be the same as the version installed on the cluster under test. Verify it with <pip search oslo.log> and install the appropriate version of oslo.log on the cluster if required.

· Tempest should be checked out to the appropriate branch.

· To authenticate with the cluster, please copy the openrc file from the controllers to the folder on the Fuel Master node, containing the script.

· The Fuel master node must have an Internet connection to run Tempest.

· The script tempest_config_creator.sh should be created and should receive 755 permissions.

· Edit tempest.conf according to your cloud settings. Example of such tempest config file could be found in Appendix 5.7

· run_tests --smoke> will launch only the smoke test group. It's also possible to run single test as described in the Readme file.

· Tempest tests can be launched from the Rally framework, using the <rally verify> command. This may be the most convenient way of doing so.

· WARNING: After testing, use the cleanup_cloud.sh script to clean up the OpenStack cloud (the script removes user "demo," tenant "demo," roles, flavors, uploaded CirrOS image and reverts Keystone endpoints), reverting the cloud to its initial state. Note that this script does not clean up resources created by Tempest, but only by mos-tempest-runner.

2.3 Rally

Rally is used to test API functionality and performance. Rally is a benchmarking tool widely used across the Openstack ecosystem. A list of available Rally tests is in Appendix 5.3

2.3.1 Tips on Using Rally

There are many ways to install and use Rally. Here, we provide instructions on how to install Rally on the compute node, using Docker:

· Extract the Rally archive on the fuel-master node

· Edit rally_conf/config.json for your environment settings. An example of such a tempest config file may be found in Appendix 5.8

· Run ./rally_setup.sh compute_node_address

· ssh to the selected compute node

· Change user to rally <su rally>

· Use rally_docker alias to start the container with rally

· Once inside container, run conf/create_deployment.sh

· Run Rally benchmarks using <rally task start scenarios/nova.yaml> (or copy any other task from the official repository)

3 Non-Functional Testing

3.1 Performance testing

3.1.1 API Benchmark – Rally

The same functional test cases as are listed above are run concurrently in order to measure API response time under load. Two runs should be performed with the concurrency set to 5 and 10.

3.1.2 Shaker — Network Performance

Shaker is used for testing network performance. Shaker is the distributed data-plane testing tool for OpenStack. A list of available Shaker tests is in Appendix 5.4.

3.1.2.1 Tips on Using Shaker

We recommend installing Shaker inside a virtualenv:

$ virtualenv venv

$ . venv/bin/activate

$ pip install pyshaker

$ shaker-image-builder

$ shaker --scenario <scenario> --report report.html

3.1.3 Storage Performance

3.1.3.1 Uploading Images to Glance

This test measures the time required to upload a 20GB raw image using the CLI command <time glance image-create...>.

3.1.3.2 IOPS for block devices

Disk performance is measured manually, using the ‘fio’ tool.

Mixed sequential reads and writes and mixed random reads and writes.

IOPS of Ephemeral disks of a single VM.
IOPS of Ephemeral disks of 10, 20 VMs.
IOPS for Cinder volumes of a single VM.
IOPS for Cinder volumes of 10, 20 VMs.

An example of the configuration file for listed tests is Appendix 5.5.

4 Resiliency testing

HA (high Availability) validation will verify high availability and failover capability of OpenStack services. Its objective is to verify that an Openstack-based solution continues to provide all functionality when one or more OpenStack service instances become inaccessible, or a controller node goes offline. There are several HA checks inside OSTF, but here we manually perform destructive scenarios for OpenStack components listed in Appendix 5.6.

In general, for an HA environment (three controllers) the process is as follows:

1) Make sure that the cloud is working normally.

2) Stop <service_name> on all controller nodes except one.
Note: for a more advanced scenario, please also stop the service on the primary controller (instead of the other controllers).

3) Run Tempest tests or check related functionality.

4) Start service. Check functionality.

5 Appendices

Appendix section contents :

Appendix # / Appendix Name
Appendix 5.1 / List of Health Check tests
Appendix 5.2 / List of Tempest Suites (classes)
Appendix 5.3 / List of Rally tests
Appendix 5.4 / List of Shaker tests
Appendix 5.5 / Configuration file for fio utility
Appendix 5.6 / Openstack components used for HA testing
Appendix 5.7 / Example of tempest.conf
Appendix 5.8 / Example of rally config file

Appendix 5.1 - List of Health Check Tests

Sanity group

· Ceilometer test to list meters, alarms and resources

· Request flavor list

· Request image list using Nova

· Request instance list

· Request absolute limits list

· Request snapshot list

· Request volume list

· Request image list using Glance v1

· Request image list using Glance v2

· Request stack list

· Request active services list

· Request user list

· Check that required services are running

· Request list of networks

Smoke group

· Create instance flavor

· Check create, update and delete image actions using Glance v1

· Check create, update and delete image actions using Glance v2

· Create volume and boot instance from it

· Create volume and attach it to instance

· Instance live migration

· Check network connectivity from instance via floating IP

· Create keypair

· Create security group

· Check network parameters

· Launch instance

· Launch instance, create snapshot, launch instance from snapshot

· Create user and authenticate with it to Horizon

HA group

· Check data replication over mysql

· Check if amount of tables in databases is the same on each node

· Check galera environment state

· Check pacemaker status

· RabbitMQ availability

· RabbitMQ replication

Platform group

· Ceilometer test to check alarm state and get Nova metrics

· Ceilometer test to check notifications from Glance

· Ceilometer test to check notifications from Keystone

· Ceilometer test to check notifications from Neutron

· Ceilometer test to check notifications from Cinder

· Ceilometer test to create, check and list samples

· Ceilometer test to create, update, check and delete alarm

· Typical stack actions: create, delete, show details, etc.

· Advanced stack actions: suspend, resume and check

· Check stack autoscaling

· Check stack rollback

· Update stack actions: in-place, replace and update whole template

Cloud validation group

· Check disk space outage on controller and compute nodes

· Check log rotation configuration on all nodes

Appendix 5.2 - List of Tempest Suites (classes)

Tempest API Elements / Description
api/compute/admin / Set of tests for agents, aggregators, APIs, who require admin privileges
api/compute/flavors / Load and different tests for flavors
api/compute/floating_ips / Tests for floating IPs configuration
api/compute/images / Tests for images functionality of OpenStack Compute
api/compute/limits / Tests to check if all limits are present in the response
api/compute/security_groups / Tests security groups and security groups rules
api/compute/servers / Checks operations available with servers - bulk creation, configurations and operations with
api/compute/test_extensions / Tests get list of all API extensions
api/compute/test_networks / Some tests for nova-network.
api/compute/test_tenant_networks / Fetch all networks that are visible to the tenant: this may include shared and external networks
api/compute/test_versions / Test checks a list of versioned endpoints that you can access for services
api/compute/volumes / Check operation with volumes - get, list, snapshot and attach volume
api/identity (v2) / Check keystone identities for users, their discovery, tokens etc (v2)
api/identity (v3) / Check keystone identities for users, their discovery, tokens etc
api/image (both v1 and v2) / Here we test the registration and creation of images
api/network / Base tests for neutron functionality
api/object_storage / Basic set of tests for Swift object storage
api/orchestration / All orchestration API tests
api/volume / All Cinder API tests

Appendix 5.3 - List of Rally tests

Ceilometer

· A user is able to define, modify, check a status and history, get a list and delete alarms

· A user is able to create meters and get statistics for them

· A user is able to define list of meters and resources

Cinder

· A user is able to create, clone, extend, attach and delete volumes

· A user is able to create a volume backup and restore from it

· A user is able to convert a volume to an image

· A user is able to clone volumes

· A user is able to create and delete snapshots of volumes

· A user is able to convert snapshots back to new volumes and delete the volumes afterwards

· A user is able to get a list of volumes and snapshots

Swift

· A user is able to create, list, and delete containers

· A user is able to upload objects into container, list objects, download, and delete them

Heat

· A user is able to create, suspend, resume, check and delete stacks

· A user is able to add, modify, and delete resources from stacks

· A user is able to manage resource groups

Glance

· A user is able to create, modify their metadata, and delete images

· A user is able to get a list of images

Keystone

· A user is able to manage (create, modify, and delete) services and roles

· A user is able to manage users and tenants

· A user is able to assign roles to users

Murano

· A user is able to import, update, and delete packages

· A user is able to create, deploy, and delete an environment

Nova

· A user is able to create (one by one or multiple at once) and destroy VMs

· A user is able to do power management of VM (hard/soft reboot, power off/on)

· A user is able to pause/unpause a VM

· A user is able to assign Security Groups to a VM

· A user is able to migrate a VM between hypervisors

· A user is able to rebuild a VM using another image

· A user is able to resize a VM

· A user is able to attach a volume to a VM

· A user is able to migrate a VM with attached volume

· A user is able to boot a VM from volume

· A user is able to manage (add, update, and delete) quotas

· A user is able to manage keypairs

· A user is able to assign a keypair to a VM

Neutron

· A user is able to manage networks and subnets