The pytest-regtest plugin for pytest

Uwe Schmitt, 5 November 2024

About automated testing

We write code which tests other code!

Let us start with a function area_rectangle:

def area_rectangle(w, h):
    return w * h

• And this code to test if the function area_rectangle behaves as expected:

  def test_area_rectangle():
      assert area_rectangle(2, 3) == 6
  

About pytest

• We put the test functions into a separate folder and files

• pytest finds them
• pytest produces nice report
• pytest rewrites assert to produce diagnostic output

Example

# squares.py

def first_n_square_numbers(n):
    return [i**2 for i in range(n)]

• This is our test:

  # test_squares.py
  
  from squares import first_n_square_numbers
  
  def test_first_n_square_numbers():
      assert first_n_square_numbers(3) == [1, 4, 9]
  

$ pytest test_example.py
============================= test session starts ==============================
...
collected 1 item

test_example.py F [100%]

=================================== FAILURES ===================================
_________________________ test_first_n_square_numbers __________________________

    def test_first_n_square_numbers():
>       assert first_n_square_numbers(3) == [1, 4, 9]
E       assert [0, 1, 4] == [1, 4, 9]
E         
E         At index 0 diff: 0 != 1
E         Use -v to get more diff

test_example.py:5: AssertionError
=========================== short test summary info ============================
FAILED test_example.py::test_first_n_square_numbers - assert [0, 1, 4] == [1,...
============================== 1 failed in 0.03s ===============================

Why should I do that?

Why should I do that?

• Develop bullet-proof functions.

• The more complex your code becomes, the more likely a simple
  change can have more effects than you think!
• It is good for your mental well being!
• Implicit documentation.

Problems

• You need "small" functions

• You may have to write a lot of testing code, especially for
  complex results.
  • How to test a 10x10 matrix?
  • How to test a set of database tables?

Regression / Snapshot testing to the rescue!

What is regression testing?

• Regression testing does not check for correctness in the first place
  but for changes to previous versions. E.g. for

  1. different results,
  2. or different resource usage, such as run-time or memory.

• Snapshot testing is a way to implement checks for changing results:
  1. Record a result
  2. Check in later versions if the produced result is still the same.

The pytest-regtest library offers a fixture (regtest), which
behaves like a file handle:

# test_regression_test_example.py

from squares import first_n_square_numbers

def test_first_n_square_numbers(regtest):
    numbers = first_n_square_numbers(3)
    print(numbers, file=regtest)

First run:

$ py.test test_regression_test_example.py
============================= test session starts ==============================
platform darwin -- Python 3.12.7, pytest-8.0.0, pluggy-1.4.0
rootdir: /private/var/folders/k8/zfp7dvcs1m326gz1brql1tv80000gn/T/tmp3c6jyle6
plugins: cov-5.0.0, regtest-2.3.1
collected 1 item

test_regression_test_example.py F                                        [100%]

=================================== FAILURES ===================================
_________________________ test_first_n_square_numbers __________________________

regression test output not recorded yet for test_regression_test_example.py::test_first_n_square_numbers:

[0, 1, 4]
---------------------------- pytest-regtest report -----------------------------
total number of failed regression tests: 1
total number of failed snapshot tests  : 0
=========================== short test summary info ============================
FAILED test_regression_test_example.py::test_first_n_square_numbers
============================== 1 failed in 0.04s ===============================

Accept results:

$ py.test --regtest-reset test_regression_test_example.py
============================= test session starts ==============================
platform darwin -- Python 3.12.7, pytest-8.0.0, pluggy-1.4.0
rootdir: /private/var/folders/k8/zfp7dvcs1m326gz1brql1tv80000gn/T/tmp3c6jyle6
plugins: cov-5.0.0, regtest-2.3.1
collected 1 item

test_regression_test_example.py R                                        [100%]

---------------------------- pytest-regtest report -----------------------------
total number of failed regression tests: 0
total number of failed snapshot tests  : 0
total number of reset output files: 1
============================== 1 passed in 0.00s ===============================

Run again:

$ py.test test_regression_test_example.py
============================= test session starts ==============================
platform darwin -- Python 3.12.7, pytest-8.0.0, pluggy-1.4.0
rootdir: /private/var/folders/k8/zfp7dvcs1m326gz1brql1tv80000gn/T/tmp3c6jyle6
plugins: cov-5.0.0, regtest-2.3.1
collected 1 item

test_regression_test_example.py .                                        [100%]

---------------------------- pytest-regtest report -----------------------------
total number of failed regression tests: 0
total number of failed snapshot tests  : 0
============================== 1 passed in 0.00s ===============================

Let's break the test:

# squares.py

def first_n_square_numbers(n):
    return [i**2 for i in range(1, n + 1)]

$ py.test test_regression_test_example.py
============================= test session starts ==============================
platform darwin -- Python 3.12.7, pytest-8.0.0, pluggy-1.4.0
rootdir: /private/var/folders/k8/zfp7dvcs1m326gz1brql1tv80000gn/T/tmp3c6jyle6
plugins: cov-5.0.0, regtest-2.3.1
collected 1 item

test_regression_test_example.py F                                        [100%]

=================================== FAILURES ===================================
_________________________ test_first_n_square_numbers __________________________

regression test output differences for test_regression_test_example.py::test_first_n_square_numbers:
    (recorded output from _regtest_outputs/test_regression_test_example.test_first_n_square_numbers)

    >   --- current
    >   +++ expected
    >   @@ -1 +1 @@
    >   -[1, 4, 9]
    >   +[0, 1, 4]

---------------------------- pytest-regtest report -----------------------------
total number of failed regression tests: 1
total number of failed snapshot tests  : 0
=========================== short test summary info ============================
FAILED test_regression_test_example.py::test_first_n_square_numbers
============================== 1 failed in 0.04s ===============================

Snapshots (since pytest-regtest 2.0)

• Specialized for different data structures beyond "text":

  • numpy arrays
  • pandas data frames
  • polars data frames (@jonasmo1)

• Supports numerical tolerance settings.

# test_snapshot_test_example.py

from squares import first_n_square_numbers

def test_first_n_square_numbers(snapshot):
    numbers = first_n_square_numbers(3)
    snapshot.check(numbers, atol=1e-12, rtol=1e-8)

First run:

$ py.test test_snapshot_test_example.py
============================= test session starts ==============================
platform darwin -- Python 3.12.7, pytest-8.0.0, pluggy-1.4.0
rootdir: /private/var/folders/k8/zfp7dvcs1m326gz1brql1tv80000gn/T/tmp3c6jyle6
plugins: cov-5.0.0, regtest-2.3.1
collected 1 item

test_snapshot_test_example.py F                                          [100%]

=================================== FAILURES ===================================
_________________________ test_first_n_square_numbers __________________________

snapshot error(s) for test_snapshot_test_example.py::test_first_n_square_numbers:

snapshot not recorded yet:
    > snapshot.check(numbers, atol=1e-12, rtol=1e-8)
    [1, 4, 9]
---------------------------- pytest-regtest report -----------------------------
total number of failed regression tests: 0
total number of failed snapshot tests  : 1
=========================== short test summary info ============================
FAILED test_snapshot_test_example.py::test_first_n_square_numbers
============================== 1 failed in 0.04s ===============================

Accept results:

$ py.test --regtest-reset test_snapshot_test_example.py
============================= test session starts ==============================
platform darwin -- Python 3.12.7, pytest-8.0.0, pluggy-1.4.0
rootdir: /private/var/folders/k8/zfp7dvcs1m326gz1brql1tv80000gn/T/tmp3c6jyle6
plugins: cov-5.0.0, regtest-2.3.1
collected 1 item

test_snapshot_test_example.py R                                          [100%]

---------------------------- pytest-regtest report -----------------------------
total number of failed regression tests: 0
total number of failed snapshot tests  : 0
total number of reset output files: 1
============================== 1 passed in 0.00s ===============================

Run again:

$ py.test test_snapshot_test_example.py
============================= test session starts ==============================
platform darwin -- Python 3.12.7, pytest-8.0.0, pluggy-1.4.0
rootdir: /private/var/folders/k8/zfp7dvcs1m326gz1brql1tv80000gn/T/tmp3c6jyle6
plugins: cov-5.0.0, regtest-2.3.1
collected 1 item

test_snapshot_test_example.py .                                          [100%]

---------------------------- pytest-regtest report -----------------------------
total number of failed regression tests: 0
total number of failed snapshot tests  : 0
============================== 1 passed in 0.00s ===============================

$ py.test --regtest-reset test_example_snapshot.py
ERROR: file or directory not found: test_example_snapshot.py

============================= test session starts ==============================
platform darwin -- Python 3.12.7, pytest-8.0.0, pluggy-1.4.0
rootdir: /private/var/folders/k8/zfp7dvcs1m326gz1brql1tv80000gn/T/tmp3c6jyle6
plugins: cov-5.0.0, regtest-2.3.1
collected 0 items

---------------------------- pytest-regtest report -----------------------------
total number of failed regression tests: 0
total number of failed snapshot tests  : 0
total number of reset output files: 0
============================ no tests ran in 0.00s =============================

What else?

• Builtin standardization for indeterministic recording with regtest:
  • temporary folder names
  • hexadecimal object addresses

# test_indeterministic_output.py

def test_indeterministic_output(regtest, tmpdir):

    print(file=regtest)  # looks a bit nicer on next slide
    print(tmpdir, file=regtest)

    print(file=regtest)  # looks a bit nicer on next slide
    print(object(), file=regtest)

$ pytest --regtest-tee test_indeterministic_output.py
============================= test session starts ==============================
platform darwin -- Python 3.12.7, pytest-8.0.0, pluggy-1.4.0
rootdir: /private/var/folders/k8/zfp7dvcs1m326gz1brql1tv80000gn/T/tmp3c6jyle6
plugins: cov-5.0.0, regtest-2.3.1
collected 1 item

test_indeterministic_output.py F
recorded raw output to regtest fixture: ----------------------------------------

/private/var/folders/k8/zfp7dvcs1m326gz1brql1tv80000gn/T/pytest-of-uweschmitt/pytest-22/test_indeterministic_output0

<object object at 0x110f1c910>

--------------------------------------------------------------------------------
                                         [100%]

=================================== FAILURES ===================================
_________________________ test_indeterministic_output __________________________

regression test output not recorded yet for test_indeterministic_output.py::test_indeterministic_output:


<tmpdir_from_fixture>

<object object at 0x?????????>
---------------------------- pytest-regtest report -----------------------------
total number of failed regression tests: 1
total number of failed snapshot tests  : 0
=========================== short test summary info ============================
FAILED test_indeterministic_output.py::test_indeterministic_output
============================== 1 failed in 0.04s ===============================

Problems solved

• You need "small" functions
  → use a regression test and then refactor

• You may have to write a lot of testing code, especially for
  complex results.
  • How to test a 10x10 matrix?
    → use snapshot
  • How to test a set of database tables?
    → dump tables and use regtest

https://pytest-regtest.readthedocs.io