Complex Repository Structures

In this tutorial, we’re going to greatly expand the repository to accomodate for test variations and expand to complex parameterized and categoried tests.

Parameterized

Now we’re going to parameterize the test. Create with the following structure and files:

parameterized/
└── tests/
    ├── parameter1/
    |   └── conf.py
    ├── parameter2/
    |   └── conf.py
    ├── test_parameterized.py
    └── conf.py

conf.py

var = "{hello_world}"

Here, we are defining a variable var with an interpolation expression expecting another variable named hello_world to replace its value.

parameter1/conf.py

hello_world = "hello parameter 1"

When this configuration file is layered onto the first, the value of both hello_world and var is "hello parameter 1".

parameter2/conf.py

hello_world = "hello parameter 2"

The value of hello_world and var is "hello parameter 2".

test_parameterized.py

import pytest
from collections import namedtuple

Properties = namedtuple("Properties", ["parameter", "expected"])

def get_test_properties():
    p1 = Properties("parameter1", "hello parameter 1")
    p2 = Properties("parameter2", "hello parameter 2")
    return [p1, p2]

@pytest.mark.parametrize("properties", get_test_properties())
def test_parameterized(properties, create_configuration):
    conf = create_configuration(test_name="", params=[properties.parameter])
    assert conf.var == properties.expected

In test_parameterized.py, we have defined a test named test_parameterized(). The @pytest.mark.parametrize decorator defines that the value of the properties variable will have its value determined by the output of the get_test_properties() function for each iteration of the test. When this is passed into the params parameter of the create_configuration pytest fixture as a list, the additional configuration files from the params directory are retrieved for that iteration.

Note

The params parameter is a list to allow additional depths of directories. For this tutorial, we have a depth of 1.

The test_name parameter is set to empty string "". The Categorized section will describe this in further detail.

For the first iteration, properties.parameter will have a value of "parameter1". The params parameter will have a value of ["parameter1"]. This will cause the create_configuration fixture to search for configuration files in tests and tests/parameter1 directories. The properties.expected value that is compared with config.var is "hello parameter1".

For the second iteration, properties.parameter has a value of "parameter2". The params parameter has a value of ["parameter2"] and the configuration files from directories tests and test/parameter2 will be used. The properties.expected value that is compared with config.var is "hello parameter2".

Let’s see what pytest collects as tests:

$ pytest --collect-only
=========== test session starts ===========
..
collected 2 items
<Module tests/test_parameterized.py>
  <Function test_parameterized[properties0]>
  <Function test_parameterized[properties1]>

========= no tests ran in 0.02s ===========

In the output, the number of iterations and the parameters of each are shown.

We can now execute the tests:

$ pytest
======== test session starts =========
..
collected 2 items

tests/test_parameterized.py ..  [100%]

========= 2 passed in 0.06s ==========

If we only wanted to execution one particular iteration:

$ pytest -k test_parameterized[properties0]
=============== test session starts ================
..
collected 2 items / 1 deselected / 1 selected

tests/test_parameterized.py .                 [100%]

========= 1 passed, 1 deselected in 0.010 ==========

Categorized

In this next section, we’re going to increase the complexity with additional tests in another module. Create the following structure and files:

categorized/
└── tests/
    ├── category
    |   ├── something
    |   │   ├── parameter1
    |   │   │   └── conf.py
    |   │   └── parameter2
    |   │       └── conf.py
    |   ├── something_else
    |   │   ├── parameter1
    |   │   │   └── conf.py
    |   │   └── parameter2
    |   │       └── conf.py
    |   ├── test_something_else.py
    |   └── test_something.py
    └── conf.py

conf.py

var = "{hello_world}"

something/parameter1/conf.py

hello_world = "hello parameter 1"

something/parameter2/conf.py

hello_world = "hello parameter 2"

something_else/parameter1/conf.py

hello_world = "hello parameter 3"

something_else/parameter2/conf.py

hello_world = "hello parameter 4"

test_something.py

import pytest
from collections import namedtuple

Properties = namedtuple("Properties", ["parameter", "expected"])

def get_test_properties():
    p1 = Properties("parameter1", "hello world 1")
    p2 = Properties("parameter2", "hello world 2")
    return [p1, p2]

@pytest.mark.parametrize("properties", get_test_properties())
def test_something(properties, create_configuration):
    conf = create_configuration(params=[properties.parameter])
    assert conf.var == properties.expected

test_something_else.py

import pytest
from collections import namedtuple

Properties = namedtuple("Properties", ["parameter", "expected"])

def get_test_properties():
    p1 = Properties("parameter1", "hello world 3")
    p2 = Properties("parameter2", "hello world 4")
    return [p1, p2]

@pytest.mark.parametrize("properties", get_test_properties())
def test_something_else(properties, create_configuration):
    conf = create_configuration(params=[properties.parameter])
    assert conf.var == properties.expected

Notice that in the create_configuration call of both modules, the test_name parameter is not specified. When not specified, the value internally is taken from the node name. The "test_" prefix is removed along with the characters after [.

For example, if we execute:

$ pytest -k test_something[properties0]

The variable test_name will be "something". If we execute:

$ pytest -k test_something_else[properties0]

The variable test_name will be "something_else".

In both cases, the test_name directory will be an additional directory that is searched for configuration files.

The order of search directories is top level, category, test name, and parameter.

For the test case of test_something[properties0], the order of directories searched is: tests (top level), category (directory of test modules), something (based on test_name), and parameter1 (based on params).

Let’s now execute the tests:

$ pytest
========== test session starts ==========
..
collected 4 items

category/test_something.py ..      [ 50%]
category/test_something_else.py .. [100%]

=========== 4 passed in 0.21s ===========