doc/python: add pytestCheckHook section

This commit is contained in:
Jonathan Ringer 2020-08-17 10:23:45 -07:00
parent 1e3f09feaa
commit 93e7292976
No known key found for this signature in database
GPG key ID: 5C841D3CFDFEC4E0

View file

@ -538,6 +538,99 @@ buildPythonPackage rec {
```
Note also the line `doCheck = false;`, we explicitly disabled running the test-suite.
#### Testing Python Packages
It is highly encouraged to have testing as part of the package build. This
helps to avoid situations where the package was able to build and install,
but is not usable at runtime. Currently, all packages will use the `test`
command provided by the setup.py. However, this is currently deprecated
https://github.com/pypa/setuptools/pull/1878 and your package should provide
it's own checkPhase.
*NOTE:* The `checkPhase` for python maps to the `installCheckPhase` on a
normal derivation. This is due to many python packages not behaving well
to the pre-installed version of the package. Version info, and natively
compiled extensions generally only exist in the install directory, and
thus can cause issues when a test suite asserts on that behavior.
*NOTE:* Tests should only be disabled if they don't agree with nix
(e.g. external dependencies, network access, flakey tests), however,
as many tests should be enabled as possible. Failing tests can still be
a good indication that the package is not in a valid state.
#### Using pytest
Pytest is the most common test runner for python repositories. A trivial
test run would be:
```
checkInputs = [ pytest ];
checkPhase = "pytest";
```
However, many repositories' test suites do not translate well to nix's build
sandbox, and will generally need many tests to be disabled.
To filter tests using pytest, one can do the following:
```
checkInputs = [ pytest ];
# avoid tests which need additional data or touch network
checkPhase = ''
pytest tests/ --ignore=tests/integration -k 'not download and not update'
'';
```
`--ignore` will tell pytest to ignore that file or directory from being
collected as part of a test run. This is useful is a file uses a package
which is not available in nixpkgs, thus skipping that test file is much
easier than having to create a new package.
`-k` is used to define a predicate for test names. In this example, we are
filtering out tests which contain `download` or `update` in their test case name.
Only one `-k` argument is allows, and thus a long predicate should be concatenated
with "\" and wrapped to the next line.
*NOTE:* In pytest==6.0.1, the use of "\" to continue a line (e.g. `-k 'not download \'`) has
been removed, in this case, it's recommended to use `pytestCheckHook`.
#### Using pytestCheckHook
`pytestCheckHook` is a convenient hook which will substitute the setuptools
`test` command for a checkPhase which runs `pytest`. This is also beneficial
when a package may need many items disabled to run the test suite.
Using the example above, the analagous pytestCheckHook usage would be:
```
checkInputs = [ pytestCheckHook ];
# requires additional data
pytestFlagsArray = [ "tests/" "--ignore=tests/integration" ];
disabledTests = [
# touches network
"download"
"update"
];
```
This is expecially useful when tests need to be conditionallydisabled,
for example:
```
disabledTests = [
# touches network
"download"
"update"
] ++ lib.optionals (pythonAtLeast "3.8") [
# broken due to python3.8 async changes
"async"
] ++ lib.optionals stdenv.isDarwin [
# can fail when building with other packages
"socket"
];
```
Trying to concatenate the related strings to disable tests in a regular checkPhase
would be much harder to read. This also enables us to comment on why specific tests
are disabled.
#### Develop local package
@ -1017,7 +1110,7 @@ are used in `buildPythonPackage`.
- `pipBuildHook` to build a wheel using `pip` and PEP 517. Note a build system
(e.g. `setuptools` or `flit`) should still be added as `nativeBuildInput`.
- `pipInstallHook` to install wheels.
- `pytestCheckHook` to run tests with `pytest`.
- `pytestCheckHook` to run tests with `pytest`. See [example usage](#using-pytestcheckhook).
- `pythonCatchConflictsHook` to check whether a Python package is not already existing.
- `pythonImportsCheckHook` to check whether importing the listed modules works.
- `pythonRemoveBinBytecode` to remove bytecode from the `/bin` folder.