Commit graph

708 commits

Author SHA1 Message Date
Pavol Rusnak 420124adf8 python: remove isPy33, isPy34 2020-05-04 18:49:45 -07:00
Frederik Rietdijk 6f873e98f4 Python integration tests: disable for older python 3 versions
because the package that is used as part of the test does not support
older versions.
2020-04-25 07:59:37 +02:00
Frederik Rietdijk 71171b3225 Python tests: test venv from a nix env with Python 3.8
This test was disabled because it did not function yet, however,
apparently it does with 3.8.
2020-04-25 07:59:37 +02:00
Frederik Rietdijk cf1a68360e python2: 2.7.17 -> 2.7.18 2020-04-21 11:21:39 +02:00
Michael Reilly 84cf00f980
treewide: Per RFC45, remove all unquoted URLs 2020-04-10 17:54:53 +01:00
Jan Tojnar 3e0f4e202f
Merge branch 'master' into staging-next 2020-03-31 21:32:15 +02:00
Frederik Rietdijk c392d70518 pkgsStatic.python3: fix build 2020-03-30 17:06:38 +02:00
Frederik Rietdijk a36be028f5 Merge staging-next into staging 2020-03-28 21:15:15 +01:00
Jonathan Ringer 279438e7f8 python: add use-pkgs-prefix option to update script 2020-03-27 01:47:30 -07:00
Jonathan Ringer a9c994ad0b python: add pythonNamespacesHook 2020-03-18 12:20:51 -07:00
Frederik Rietdijk 7447fff95a
Fix sys.prefix in case of a Nix env
The prefix will now be correct in case of Nix env.

Note, however, that creating a venv from a Nix env still does not function. This does not seem to be possible
with the current approach either, because venv will copy or symlink our Python wrapper. In case it symlinks
(the default) it won't see a pyvenv.cfg. If it is copied I think it should function but it does not...
2020-03-14 21:39:32 +00:00
adisbladis 753122388d
Python: Add integration test verifying NIX_PYTHONPATH with Mypy 2020-03-14 21:39:32 +00:00
adisbladis 05571d3059
Python Add test for NIX_PYTHONPREFIX 2020-03-14 21:39:31 +00:00
adisbladis d88a7735d2
Python: introduce NIX_PYTHONPREFIX in order to set site.PREFIXES
This is needed in case of `python.buildEnv` to make sure site.PREFIXES
does not only point to the unwrapped executable prefix.

--------------------------------------------------------------------------------

This PR is a story where your valiant hero sets out on a very simple adventure but ends up having to slay dragons, starts questioning his own sanity and finally manages to gain enough knowledge to slay the evil dragon and finally win the proverbial price.

It all started out on sunny spring day with trying to tackle the Nixops plugin infrastructure and make that nice enough to work with.

Our story begins in the shanty town of [NixOps-AWS](https://github.com/nixos/nixops-aws) where [mypy](http://mypy-lang.org/) type checking has not yet been seen.

As our deuteragonist (@grahamc) has made great strides in the capital city of [NixOps](https://github.com/nixos/nixops) our hero wanted to bring this out into the land and let the people rejoice in reliability and a wonderful development experience.

The plugin work itself was straight forward and our hero quickly slayed the first small dragon, at this point things felt good and our hero thought he was going to reach the town of NixOps-AWS very quickly.

But alas! Mypy did not want to go, it said:
`Cannot find implementation or library stub for module named 'nixops'`

Our hero felt a small sliver of life escape from his body. Things were not going to be so easy.

After some frustration our hero discovered there was a [rule of the land of Python](https://www.python.org/dev/peps/pep-0561/) that governed the import of types into the kingdom, more specificaly a very special document (file) called `py.typed`.
Things were looking good.

But no, what the law said did not seem to match reality. How could things be so?

After some frustrating debugging our valiant hero thought to himself "Hmm, I wonder if this is simply a Nix idiosyncrasy", and it turns out indeed it was.
Things that were working in the blessed way of the land of Python (inside a `virtualenv`) were not working the way they were from his home town of Nix (`nix-shell` + `python.withPackages`).

After even more frustrating attempts at reading the mypy documentation and trying to understand how things were supposed to work our hero started questioning his sanity.
This is where things started to get truly interesting.

Our hero started to use a number of powerful weapons, both forged in the land of Python (pdb) & by the mages of UNIX (printf-style-debugging & strace).

After first trying to slay the dragon simply by `strace` and a keen eye our hero did not spot any weak points.
Time to break out a more powerful sword (`pdb`) which also did not divulge any secrets about what was wrong.

Our hero went back to the `strace` output and after a fair bit of thought and analysis a pattern started to emerge. Mypy was looking in the wrong place (i.e. not in in the environment created by `python.withPackages` but in the interpreter store path) and our princess was in another castle!

Our hero went to the pub full of old grumpy men giving out the inner workings of the open source universe (Github) and acquired a copy of Mypy.
He littered the code with print statements & break points.
After a fierce battle full of blood, sweat & tears he ended up in 20f7f2dd71/mypy/sitepkgs.py and realised that everything came down to the Python `site` module and more specifically https://docs.python.org/3.7/library/site.html#site.getsitepackages which in turn relies on https://docs.python.org/3.7/library/site.html#site.PREFIXES .

Our hero created a copy of the environment created by `python.withPackages` and manually modified it to confirm his findings, and it turned out it was indeed the case.
Our hero had damaged the dragon and it was time for a celebration.

He went out and acquired some mead which he ingested while he typed up his story and waited for the dragon to finally die (the commit caused a mass-rebuild, I had to wait for my repro).

In the end all was good in [NixOps-AWS](https://github.com/nixos/nixops-aws)-town and type checks could run. (PR for that incoming tomorrow).
2020-03-14 21:39:31 +00:00
Frederik Rietdijk 13e7a3e112 Python: introduce tests for interpreters
This adds tests to the passthru of all Python interpreters.
2020-03-14 15:05:37 +01:00
Frederik Rietdijk dbf125d286 Python: introduce tests for interpreters
This adds tests to the passthru of all Python interpreters.
2020-03-14 15:01:30 +01:00
Graham Christensen 39aac70d74 pythonMinimal: don't include site-customise
Experimenting with patching the site-customize file causes mass
rebuilds for no reason.
2020-03-14 15:02:51 +01:00
Frederik Rietdijk 31855d74a3 python3: 3.7.6 -> 3.7.7 2020-03-12 12:00:26 +01:00
Jonathan Ringer 3973a3c79c python: add pythonRemoveTestsDirHook
This will automatically remove a top-level "tests"
directory from being installed.
2020-03-03 20:01:00 +01:00
Jonathan Ringer fe4580359e python39: 3.9.0.a3 -> 3.9.0.a4 2020-03-03 07:33:39 +01:00
Jonathan Ringer 2a019cc48c python38: 3.8.1 -> 3.8.2 2020-03-03 07:33:39 +01:00
Frederik Rietdijk 79217339d2 Merge master into staging-next 2020-02-29 15:29:11 +01:00
Emily 6d3fc35620 pypy{,3}: use openssl_1_1
"We now support building PyPy with OpenSSL 1.1 in our built-in _ssl
module, as well as maintaining support for previous versions."
-- https://pypy.readthedocs.io/en/latest/release-pypy2.7-v5.6.0.html
2020-02-28 16:06:20 +00:00
Jonathan Ringer cd97c055a0 python: execute pythonImportsCheckPhase in subshell
Execute in subshell so that the cwd returns back
to the original directory. This is important as
it enables pythonImportsCheck to work with checkPhase
2020-02-19 22:30:50 -08:00
Frederik Rietdijk 38cf6eac19 Merge master into staging-next 2020-02-06 19:43:36 +01:00
Frederik Rietdijk 9827e4994b python39: 3.9.0a2 -> 3.9.0a3 2020-02-06 10:25:18 +01:00
Maximilian Bosch 6b0cd9ad47
Merge branch 'staging' into glibc230
Conflicts:
	pkgs/applications/misc/vit/default.nix
2020-01-28 14:54:51 +01:00
Frederik Rietdijk dce0ca29d9 Merge master into staging-next 2020-01-28 10:46:13 +01:00
Maximilian Bosch eddfcc32b4
Merge branch 'staging' into glibc230 2020-01-23 11:31:13 +01:00
Frederik Rietdijk bc18cc72dd
Merge pull request #77610 from LnL7/darwin-stdenv-python3
darwin-stdenv: bootstrap with python3
2020-01-15 09:24:57 +01:00
Frederik Rietdijk eba1f79418 pythonPackages.venvShellHook: init
This is a hook that loads a virtualenv from the specified `venvDir`
location. If the virtualenv does not exist, it is created.
2020-01-14 22:36:21 +01:00
Luka Blaskovic 76d90cf1f5 cpython: fetch darwin-libutil.patch to nixpkgs
python3 is now required buildInput for glibc>=2.29.
Remove fetchpatch to solve infinite recursion in
glibc bootstrap process.
2020-01-14 08:26:58 +00:00
Daiderd Jordan 6328518e98
stdenv: bootstrap darwin with python3
- Replaced python override from the final stdenv, instead we
  propagate our bootstrap python to stage4 and override both
  CF and xnu to use it.

- Removed CF argument from python interpreters, this is redundant
  since it's not overidden anymore.

- Inherit CF from stage4, making it the same as the stdenv.
2020-01-13 11:34:36 +01:00
adisbladis 2d6f1ff4dd
python: Add support for installing Python eggs 2020-01-08 13:59:04 +00:00
Jan Tojnar f4e74edd8c python.pkgs.wrapPython: get rid of warning
When `makeWrapperArgs` variable is not set, `declare -p makeWrapperArgs`
will return with 1 and print an error message to stderr.

I did not handle the non-existence case in b0633406cb
because I thought `mk-python-derivation` will always define `makeWrapperArgs`
but `wrapProgram` can be called independently. And even with `mk-python-derivation`,
`makeWrappers` will not be set unless explicitly declared in the derivation
because of https://github.com/NixOS/nix/issues/1461.

I was lead to believe that because the builds were succeeding and I confirmed
that the mechanism fails when the variable is not defined and `-o nounset` is enabled.
It appears that `wrapPython` setup hook is not running under `-o nounset`, though,
invaldating the assumption.

Now we are checking that the variable exists before checking its type, which
will get rid of the warning and also prevent future error when `-o nounset`
is enabled in the setup hook.

For more information, see the discussion at
https://github.com/NixOS/nixpkgs/commit/a6bb2ede232940a96150da7207a3ecd15eb6328
2019-12-30 16:46:55 +01:00
Jan Tojnar a6bb2ede23
python.pkgs.wrapPython: fix string makeWrapperArgs
Bash takes an assignment of a string to an array variable:

local -a user_args
user_args="(foo bar)"

to mean appending the string to the array, not parsing the string into
an array as is the case when on the same line as the declaration:

local -a user_args="(foo bar)"

b0633406cb extracted the declaration before
the newly branched code block, causing string makeWrapperArgs being added
to the array verbatim.

Since local is function scoped, it does not matter if we move it inside
each of the branches so we fix it this way.
2019-12-28 22:23:36 +01:00
worldofpeace 4a2621da53
Merge pull request #76283 from jtojnar/python-mwa
python.pkgs.wrapPython: fix makeWrapperArgs
2019-12-26 19:21:37 -05:00
Jan Tojnar 4bbc6cc66f
Merge branch 'staging-next' into staging 2019-12-25 05:18:52 +01:00
Jan Tojnar ca39dd3a8a
Merge branch 'master' into staging-next 2019-12-25 05:15:06 +01:00
Jan Tojnar b0633406cb
python.pkgs.wrapPython: fix makeWrapperArgs
When `makeWrapperArgs` is a Bash array, we only passed the first
item to `wrapProgram`. We need to use `"${makeWrapperArgs[@]}"`
to extract all the items. But that breaks the common string case so
we need to handle that case separately.
2019-12-23 18:02:44 +01:00
Frederik Rietdijk ad733b5505 python37: 3.7.5 -> 3.7.6 2019-12-19 17:37:02 +01:00
Frederik Rietdijk c0c65fe83c python39: 3.9.0a1 -> 3.9.0a2 2019-12-19 17:36:21 +01:00
Frederik Rietdijk 2012dd5734 python38: 3.8.0 -> 3.8.1 2019-12-19 17:36:21 +01:00
Frederik Rietdijk 5796029c5d python36: 3.6.9 -> 3.6.10 2019-12-19 17:36:20 +01:00
Andreas Rammhold e9f522eee1
python: remove _manylinux.py
This will turn manylinux support back on by default.

PIP will now do runtime checks against the compatible glibc version to
determine if the current interpreter is compatible with a given
manylinux specification. However it will not check if any of the
required libraries are present.

The motivation here is that we want to support building python packages
with wheels that require manylinux support. There is no real change for
users of source builds as they are still buildings packages from source.

The real noticeable(?) change is that impure usages (e.g. running `pip
install package`) will install manylinux packages that previously
refused to install.
Previously we did claim that we were not compatible with manylinux and
thus they wouldn't be installed at all.

Now impure users will have basically the same situation as before: If
you require some wheel only package it didn't work before and will not
properly work now. Now the program will fail during runtime vs during
installation time.

I think it is a reasonable trade-off since it allows us to install
manylinux packages with nix expressions and enables tools like
poetry2nix.

This should be a net win for users as it allows wheels, that we
previously couldn't really support, to be used.
2019-12-16 16:37:16 +01:00
Tobias Pflug 418ad571c3 Fix manylinux packages
Make sure lib outputs are used where applicable.
2019-12-13 11:40:44 +01:00
Frederik Rietdijk 6530535b20
manylinux packages for Python
This adds three lists with manylinux dependencies as well as three
packages that include all the manylinux dependencies.
2019-12-05 09:56:20 +00:00
Frederik Rietdijk 92d2153e6c pythonInterpreters: remove unnecessary rec 2019-12-02 20:10:51 +01:00
Frederik Rietdijk 1d18c5a0fe Merge staging-next into staging 2019-11-24 10:13:31 +01:00
Frederik Rietdijk 182571cdc3 update-python-libraries: ignore pyproject 2019-11-22 08:37:03 +01:00