OpenStack Unit Testing With Tox, run_tests.sh and testr

OpenStack python-neutronclient Unit Testing With Tox, run_tests.sh and testr

https://github.com/openstack/python-neutronclient/

1)
Clone python-neutronclient
#git clone https://github.com/openstack/python-neutronclient.git

2)
Some useful links
https://github.com/openstack/python-neutronclient/blob/master/HACKING.rst

http://docs.openstack.org/developer/hacking/ <=== IMP

3)
Tox and testr configs

https://wiki.openstack.org/wiki/Testr

https://github.com/openstack/python-neutronclient/blob/master/.testr.conf

https://github.com/openstack/python-neutronclient/blob/master/tox.ini

https://github.com/openstack/python-neutronclient/blob/master/test-requirements.txt


https://github.com/openstack/python-neutronclient/blob/master/requirements.txt

4)
What is testr, tox and run_tests.sh ??

a)
Test repository (testr) is a small application for tracking test results.
testr is configured via the ‘.testr.conf’ file which needs to be in the same directory that testr is run from.

http://testrepository.readthedocs.org/en/latest/MANUAL.html

b)
Tox aims to automate and standardize testing in Python.
Tox is a generic virtualenv management and test command line tool you can use for:
* checking your package installs correctly with different Python versions and interpreters.
* Running your tests in each of the environments, configuring your test tool of choice.
* Acting as a frontend to Continuous Integration servers, greatly reducing boilerplate and merging CI and shell-based testing.

Install tox "pip install tox" and put basic information about your project and the test environments you want your project to run in into a tox.ini file.You can also try generating a tox.ini file automatically, by running tox-quickstart and then answering a few simple questions.

http://tox.readthedocs.org/en/latest/
http://tox.readthedocs.org/en/latest/examples.html

c)
Actually "tox" and "run_tests.sh" are wrappers to use "testr".

5)
Install and run tox to setup test env and test your project

#cd python-neutronclient
#sudo pip install tox

To install and setup all environments defined in the "tox.ini" file.
#tox

* "tox" command will read "python-neutronclient/tox.ini" and setup test env and test the project.
* "tox" command also read "python-neutronclient/test-requirements.txt" to setup test env, see "tox.ini" file.
* This "tox" command will creates hidden files like ".testrepository" and ".tox".
* List all environment folders in the ".tox" hidden folder "#ls .tox"
* This command setup all environments defined in the "tox.ini" file and install all dependencies defined in the "requirements.txt" and "test-requirements.txt".
* https://wiki.openstack.org/wiki/Testr

6)
To install and setup python 2.7 (py27) envirtonment
#tox -epy27
* Don't manually activate virtualenv and execute this "tox" command, since it recreate the virtualenv at that case.

OR

Use "-r" option to recreate the environment from scratch and install all dependencies defined in the "requirements.txt" and "test-requirements.txt".
##tox -epy27 -r

* Under ".tox" hidden folder, you can see the environment directory "py27". Also check "tox.ini" file for configuration.
* Check the log file "#tail -f .tox/py27/log/py27-1.log" in the hidden dir to see what happens during the environment setup.

7)
List all test environments
#tox -l
py26
py27
py33
pypy
pep8

8)
show configuration information for all environments
#tox --showconfig
* This command read from "tox.ini" file.

9)
How can I run just one test ??.

Method-1):

Using Tox
a)
Deactivate virtualenv

b)
Run Test
#tox -epy27 -- neutronclient.tests.unit.test_cli20_network.CLITestV20NetworkJSON.test_create_network
#tox -epy27 -- neutronclient.tests.unit.test_cli20_network.CLITestV20NetworkJSON
#tox -epy27 -- neutronclient.tests.unit.test_cli20_network
#tox -epy27 -- neutronclient.tests
#tox -epy27

* Don't manually activate virtualenv and execute this "tox" command, since it recreate the virtualenv at that case.

Method-2):

Using testr
a)
Activate virtualenv
#source .tox/py27/bin/activate

b)
Run Test
#testr run neutronclient.tests.unit.test_cli20_network.CLITestV20NetworkJSON.test_create_network
#testr run neutronclient.tests.unit.test_cli20_network.CLITestV20NetworkJSON
#testr run neutronclient.tests.unit.test_cli20_network
#testr run neutronclient.tests
#testr run

c)
Run parallel Test
#testr run --parallel neutronclient.tests.unit.test_cli20_network.CLITestV20NetworkJSON.test_create_network
#testr run --parallel neutronclient.tests.unit.test_cli20_network.CLITestV20NetworkJSON
#testr run --parallel neutronclient.tests.unit.test_cli20_network
#testr run --parallel neutronclient.tests
#testr run --parallel

10)
Run pep8 test

a)
Using Tox

* Deactivate virtualenv

* Run only pep8 Test
#tox -e pep8 -- neutronclient.tests.unit.test_cli20_network.CLITestV20NetworkJSON.test_create_network
#tox -e pep8 -- neutronclient.tests.unit.test_cli20_network.CLITestV20NetworkJSON
#tox -e pep8 -- neutronclient.tests.unit.test_cli20_network
#tox -e pep8 -- neutronclient.tests
#tox -e pep8

* Run py27 and pep8 Test
#tox -e py27,pep8 -- neutronclient.tests.unit.test_cli20_network.CLITestV20NetworkJSON.test_create_network
#tox -e py27,pep8 -- neutronclient.tests.unit.test_cli20_network.CLITestV20NetworkJSON
#tox -e py27,pep8 -- neutronclient.tests.unit.test_cli20_network
#tox -e py27,pep8 -- neutronclient.tests
#tox -e py27,pep8

http://docs.openstack.org/developer/ceilometer/contributing/source.html

b)
Using flake8

* Activate virtualenv
#source .tox/py27/bin/activate
* Run pep8 test
#flake8

11)
testr helps

#testr commands
#testr help run

12)
Run only the tests in the CLITestV20NetworkJSON class
#tox -epy27 -- '(CLITestV20NetworkJSON)'
Run only the tests in the CLITestV20NetworkJSON and CLITestV20FloatingIpsJSON classes
#tox -epy27 -- '(CLITestV20NetworkJSON|CLITestV20FloatingIpsJSON)'
Run only the test "test_create_network" in the CLITestV20NetworkJSON class
#tox -epy27 -- '(CLITestV20NetworkJSON.test_create_network)'
Run only the tests "test_create_network" and "test_create_floatingip" in the CLITestV20NetworkJSON and CLITestV20FloatingIpsJSON classes
#tox -epy27 -- '(CLITestV20NetworkJSON.test_create_network|CLITestV20FloatingIpsJSON.test_create_floatingip)'

13)
run only particular test of python-tackerclient project

$ python -m testtools.run tackerclient.tests.unit.test_validators.ValidatorTest

$ python -m testtools.run tackerclient.tests.unit.test_validators.ValidatorTest.test_validate_ip_subnet

14)