Using Tox to Unit Test Python AWS Lambdas

Overview

If you are writing a backend using AWS lambdas and have multiple lambdas, you may run into the issue of: “how do I run all my unit tests, but in environments isolated per lambda?”. Tox https://tox.wiki was originally designed to run tests for a packages in multiple environments (“I want to test my library against every version of Python between 3.6 and 3.10”), but we can abuse it to instead create isolated environments for each of our lambdas, and run tests with only the dependencies specified per package. We will use pytest https://docs.pytest.org/ and also add coverage reporting.

Code Layout

Our code layout is as follows:

./
└── src/
   ├── api/
   │  ├── api1/
   │  │  ├── stuff...
   │  │  ├── requirements.txt
   │  │  └── tests/
   │  ├── api2/
   │  │  ├── stuff...
   │  │  ├── requirements.txt
   │  │  └── tests/
   │  └── api3/
   │     ├── stuff...
   │     ├── requirements.txt
   │     └── tests/
   └── libs/
      ├── stuff...
      ├── requirements.txt
      └── tests/

The api<N>/ folders are the lambdas and a shared resource layer lives in libs/. We want to create a new virtual env for each of these folders and test them in isolation, since that’s how they’ll be running in production.

tox.ini

To begin, create a tox.ini file in the root of your project. Add the following, updating path names with your folder names:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51

[tox]
minversion = 3.26
skipsdist = true
envlist = clean,src/libs,src/api/{api1,api2,api3},report

[base]
deps =
    -r{toxinidir}/{envname}/requirements.txt

setenv =

[testenv]
changedir = {toxinidir}/{envname}

basepython = python3.9

allowlist_externals =
    sh

deps =
    coverage==6.4.4
    pytest-cov==3.0.0
    pytest-mock==3.8.2
    pytest==7.1.2
    {[base]deps}

setenv =
    PYTHONPATH = {toxinidir}/{envname}
    COVERAGE_FILE = {toxinidir}/.coverage

install_command = pip install {opts} {packages}

commands = pytest --cov={toxinidir}/{envname} --cov-append

[testenv:clean]
deps =
    coverage==6.4.4
skip_install = true
commands =
    coverage erase
    sh -c "rm -rf {toxinidir}/htmlcov || true"

[testenv:report]
deps =
    coverage==6.4.4
skip_install = true
commands =
    coverage report --omit="*/tests/*" -m --skip-empty
    coverage html -d {toxinidir}/htmlcov --omit="*/tests/*" --skip-empty
    coverage xml -o {toxinidir}/coverage.xml --omit="*/tests/*" --skip-empty
    sh -c "rmdir {toxinidir}/report {toxinidir}/clean || true"

Running it all

Install tox pip install tox . Everything should be set up now and fully configured, all you need to do is run:

$ tox

The first run will be slow as it needs to create all the environments, but subsequent runs will reuse them and be much faster.

If you want to force recreation of all the virtual environments, you can:

$ tox -r

What’s Happening

The [tox] Block

We’re setting a minimum version of tox and asking it not to create a source dist of our package, we don’t need one. We’re also creating a list of environments:

1. clean, This will clean out old coverage reports
2. src/libs, This is our shared resource layer folder that we want to test
3. src/api/{api1,api2,api3}, These are the 3 api lambdas, we want to test all of them
4. report, This is an additional environment that will turn the pytest coverage into both an XML report for your IDE as well as an HTML report for human consumption

[base]

This block is just setting up a way to import requirements.txt files or any other set up we want shared between all the test environments (right now it’s just set up to give us a requirements file). It’s now clear here why we named our environments after the folder we want to test - the “environment name” is going to be injected into a bunch of commands to only test parts of the repo.

[testenv]

This is the meat of the testing logic. We’re defining a directory to run, python version to use, and allowing tox to run sh commands (You need to tell it to allow running commands outside the virtual environment otherwise it’ll let you know that you’re reaching outside of the isolated sandbox it’s trying to maintain for you). We then set up our testing dependencies. Line 20 has all the pytest and coverage-related dependencies we want for our testing, as well as pulling in the requirements.txt file specified in the base block. This will run per environment.

We also set the PYTHONPATH env var, and COVERAGE_FILE so that all the tests write append coverage to the same file. We set how we want our files installed, as well as our test command. The --cov-append flag is critical, otherwise it will overwrite previous coverage

[testenv:clean]

This block is subclassing the [testenv] block and overwriting some settings with clean-step specific configuration. The only package we need here is coverage, and we don’t need the clean package installed (because it doesn’t exist) so we add skip_install = true.

This environment clears out existing coverage and deletes the htmlcov folder of html coverage reports, if it exists. This is the reason we had to allow sh to run earlier.

[testenv:report]

This is set up similar to testenv:clean. It will:

1. Print off the coverage, not counting empty or test files towards your code coverage
2. Generate an HTML report in the htmlcov folder
3. Generate an XML report in coverage.xml
4. Delete the empty report/ and clean/ folders that tox will generate, I haven’t figured out how to stop this.

Additional Resources

For information on making the test environments run in parallel, see these pytest-cov docs.

To integrate the coverage report into VSCode, we’ve been using Coverage Gutters