Metadata-Version: 2.1
Name: pytest-regtest
Version: 2.1.1
Summary: pytest plugin for snapshot regression testing
Project-URL: Source, https://gitlab.com/uweschmitt/pytest-regtest
Project-URL: Documentation, https://pytest-regtest.readthedocs.org
Author-email: Uwe Schmitt <uwe.schmitt@id.ethz.ch>
License: MIT License
License-File: LICENSE.txt
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Requires-Dist: pytest>7.2
Provides-Extra: dev
Requires-Dist: black; extra == 'dev'
Requires-Dist: build; extra == 'dev'
Requires-Dist: hatchling; extra == 'dev'
Requires-Dist: pre-commit; extra == 'dev'
Requires-Dist: ruff; extra == 'dev'
Requires-Dist: twine; extra == 'dev'
Requires-Dist: wheel; extra == 'dev'
Description-Content-Type: text/markdown


# Home

## About

`pytest-regtest` is a plugin for [pytest](https://pytest.org) to implement
**regression testing**.

Unlike [functional testing](https://en.wikipedia.org/wiki/Functional_testing),
[regression testing](https://en.wikipedia.org/wiki/Regression_testing)
testing does not test whether the software produces the correct
results, but whether it behaves as it did before changes were introduced.

More specifically, `pytest-regtest` provides **snapshot testing**, which
implements regression testing by recording the textual output of a test
function and comparing this recorded output to a reference output.

**Regression testing** is a common technique to implement basic testing
before refactoring legacy code that lacks a test suite.

Snapshot testing can also be used to implement tests for complex outcomes, such
as recording textual database dumps or the results of a scientific analysis
routine.


## Installation

To install and activate this plugin execute:

    $ pip install pytest-regtest

## Basic Usage


### Write a test

The `pytest-regtest` plugin provides multiple fixtures.
To record output, use the fixture `regtest` that works like a file handle:

```py
def test_squares(regtest):

    result = [i*i for i in range(10)]

    # one way to record output:
    print(result, file=regtest)

    # alternative method to record output:
    regtest.write("done")
```

You can also use the `regtest_all` fixture. This enables all output to stdout to be
recorded in a test function.


### Run the test

If you run this test script with *pytest* the first time there is no
recorded output for this test function so far and thus the test will
fail with a message including a diff:

```
$ pytest -v test_squares.py
============================= test session starts ==============================
platform darwin -- Python 3.12.2, pytest-8.0.0, pluggy-1.4.0 -- ...
cachedir: .pytest_cache
rootdir: ...
plugins: regtest-2.1.0
collecting ... collected 1 item

test_squares.py::test_squares FAILED                                     [100%]

=================================== FAILURES ===================================
_________________________________ test_squares _________________________________

regression test output not recorded yet for test_squares.py::test_squares:

[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
done
---------------------------- pytest-regtest report -----------------------------
total number of failed regression tests: 1
=========================== short test summary info ============================
FAILED test_squares.py::test_squares
============================== 1 failed in 0.02s ===============================
```

This is a diff of the current output `is` to a previously recorded output
`tobe`. Since we did not record output yet, the diff contains no lines marked
`+`.


### Reset the test

To record the current output, we run *pytest* with the *--reset-regtest*
flag:

```
$ pytest -v --regtest-reset test_squares.py
============================= test session starts ==============================
platform darwin -- Python 3.12.2, pytest-8.0.0, pluggy-1.4.0 -- ...
cachedir: .pytest_cache
rootdir: ...
plugins: regtest-2.1.0
collecting ... collected 1 item

test_squares.py::test_squares RESET                                      [100%]

---------------------------- pytest-regtest report -----------------------------
total number of failed regression tests: 0
the following output files have been reset:
  _regtest_outputs/test_squares.test_squares.out
============================== 1 passed in 0.00s ===============================
```

You can also see from the output that the recorded output is in the
`_regtest_outputs` folder which in the same folder as the test script.
Don't forget to commit this folder to your version control system!

### Run the test again

When we run the test again, it succeeds:

```
$ pytest -v test_squares.py
============================= test session starts ==============================
platform darwin -- Python 3.12.2, pytest-8.0.0, pluggy-1.4.0 -- ...
cachedir: .pytest_cache
rootdir: ...
plugins: regtest-2.1.0
collecting ... collected 1 item

test_squares.py::test_squares PASSED                                     [100%]

---------------------------- pytest-regtest report -----------------------------
total number of failed regression tests: 0
============================== 1 passed in 0.00s ===============================
```

### Break the test

Let us break the test by changing the test function to compute
11 instead of 10 square numbers:

```py
def test_squares(regtest):

    result = [i*i for i in range(11)]

    # one way to record output:
    print(result, file=regtest)

    # alternative method to record output:
    regtest.write("done")
```

The next run of pytest delivers a nice diff of the current and expected output
from this test function:

```
$ pytest -v test_squares.py
============================= test session starts ==============================
platform darwin -- Python 3.12.2, pytest-8.0.0, pluggy-1.4.0 -- ...
cachedir: .pytest_cache
rootdir: ...
plugins: regtest-2.1.0
collecting ... collected 1 item

test_squares.py::test_squares FAILED                                     [100%]

=================================== FAILURES ===================================
_________________________________ test_squares _________________________________

regression test output differences for test_squares.py::test_squares:
(recorded output from _regtest_outputs/test_squares.test_squares.out)

>   --- current
>   +++ expected
>   @@ -1,2 +1,2 @@
>   -[0, 1, 4, 9, 16, 25, 36, 49, 64, 81, 100]
>   +[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
>    done

---------------------------- pytest-regtest report -----------------------------
total number of failed regression tests: 1
=========================== short test summary info ============================
FAILED test_squares.py::test_squares
============================== 1 failed in 0.02s ===============================
```


## Other features

### Using the `regtest` fixture as context manager

The `regtest` fixture also works as a context manager  to capture
all output from the wrapped code block:

```py
def test_squares(regtest):

    result = [i*i for i in range(10)]

    with regtest:
        print(result)
```

### The `regtest_all` fixture

The `regtest_all` fixture leads to recording of all output to `stdout` in a
test function.

```py
def test_all(regtest_all):
    print("this line will be recorded.")
    print("and this line also.")
```

### Reset individual tests

You can reset recorded output of files and functions individually as:

```sh
$ py.test --regtest-reset test_demo.py
$ py.test --regtest-reset test_demo.py::test_squares
```

### Suppress diff for failed tests

To hide the diff and just show the number of lines changed, use:

```sh
$ py.test --regtest-nodiff ...
```


### Show all recorded output


For complex diffs it helps to see the full recorded output also.
To enable this use:

```sh
$ py.test --regtest-tee...
```


### Line endings

Per default `pytest-regtest` ignores different line endings in the output.
In case you want to disable this feature, use the `-regtest-consider-line-endings`
flag.


## Clean indeterministic output before recording

Output can contain data which is changing from test run to test
run, e.g. paths created with the `tmpdir` fixture, hexadecimal object ids or
timestamps.

Per default the plugin helps to make output more deterministic by:

- replacing all temporary folder in the output with `<tmpdir...>` or similar markers,
  depending on the origin of the temporary folder (`tempfile` module, `tmpdir` fixture,
   ...)
- replacing hexadecimal numbers ` 0x...` of arbitary length by the fixed string `0x?????????`.

You can also implement your own cleanup routines as described below.

### Register own cleanup functions

You can register own converters in `conftest.py`:

```py
import re
import pytest_regtest

@pytest_regtest.register_converter_pre
def remove_password_lines(txt):
    '''modify recorded output BEFORE the default fixes
    like temp folders or hex object ids are applied'''

    # remove lines with passwords:
    lines = txt.splitlines(keepends=True)
    lines = [l for l in lines if "password is" not in l]
    return "".join(lines)

@pytest_regtest.register_converter_post
def fix_time_measurements(txt):
    '''modify recorded output AFTER the default fixes
    like temp folders or hex object ids are applied'''

    # fix time measurements:
    return re.sub(
        r"\d+(\.\d+)? seconds",
        "<SECONDS> seconds",
        txt
    )
```

If you register multiple converters they will be applied in the order of
registration.

In case your routines replace, improve or conflict with the standard cleanup converters,
you can use the flag `--regtest-disable-stdconv` to disable the default cleanup
procedure.

## Command line options summary

These are all supported command line options:

```
$ pytest --help
...

regression test plugin:
  --regtest-reset       do not run regtest but record current output
  --regtest-tee         print recorded results to console too
  --regtest-consider-line-endings
                        do not strip whitespaces at end of recorded lines
  --regtest-nodiff      do not show diff output for failed regresson tests
  --regtest-disable-stdconv
                        do not apply standard output converters to clean up
                        indeterministic output
...
```
