Commit graph

36 commits

Author SHA1 Message Date
adisbladis d88a7735d2
Python: introduce NIX_PYTHONPREFIX in order to set site.PREFIXES
This is needed in case of `python.buildEnv` to make sure site.PREFIXES
does not only point to the unwrapped executable prefix.

--------------------------------------------------------------------------------

This PR is a story where your valiant hero sets out on a very simple adventure but ends up having to slay dragons, starts questioning his own sanity and finally manages to gain enough knowledge to slay the evil dragon and finally win the proverbial price.

It all started out on sunny spring day with trying to tackle the Nixops plugin infrastructure and make that nice enough to work with.

Our story begins in the shanty town of [NixOps-AWS](https://github.com/nixos/nixops-aws) where [mypy](http://mypy-lang.org/) type checking has not yet been seen.

As our deuteragonist (@grahamc) has made great strides in the capital city of [NixOps](https://github.com/nixos/nixops) our hero wanted to bring this out into the land and let the people rejoice in reliability and a wonderful development experience.

The plugin work itself was straight forward and our hero quickly slayed the first small dragon, at this point things felt good and our hero thought he was going to reach the town of NixOps-AWS very quickly.

But alas! Mypy did not want to go, it said:
`Cannot find implementation or library stub for module named 'nixops'`

Our hero felt a small sliver of life escape from his body. Things were not going to be so easy.

After some frustration our hero discovered there was a [rule of the land of Python](https://www.python.org/dev/peps/pep-0561/) that governed the import of types into the kingdom, more specificaly a very special document (file) called `py.typed`.
Things were looking good.

But no, what the law said did not seem to match reality. How could things be so?

After some frustrating debugging our valiant hero thought to himself "Hmm, I wonder if this is simply a Nix idiosyncrasy", and it turns out indeed it was.
Things that were working in the blessed way of the land of Python (inside a `virtualenv`) were not working the way they were from his home town of Nix (`nix-shell` + `python.withPackages`).

After even more frustrating attempts at reading the mypy documentation and trying to understand how things were supposed to work our hero started questioning his sanity.
This is where things started to get truly interesting.

Our hero started to use a number of powerful weapons, both forged in the land of Python (pdb) & by the mages of UNIX (printf-style-debugging & strace).

After first trying to slay the dragon simply by `strace` and a keen eye our hero did not spot any weak points.
Time to break out a more powerful sword (`pdb`) which also did not divulge any secrets about what was wrong.

Our hero went back to the `strace` output and after a fair bit of thought and analysis a pattern started to emerge. Mypy was looking in the wrong place (i.e. not in in the environment created by `python.withPackages` but in the interpreter store path) and our princess was in another castle!

Our hero went to the pub full of old grumpy men giving out the inner workings of the open source universe (Github) and acquired a copy of Mypy.
He littered the code with print statements & break points.
After a fierce battle full of blood, sweat & tears he ended up in 20f7f2dd71/mypy/sitepkgs.py and realised that everything came down to the Python `site` module and more specifically https://docs.python.org/3.7/library/site.html#site.getsitepackages which in turn relies on https://docs.python.org/3.7/library/site.html#site.PREFIXES .

Our hero created a copy of the environment created by `python.withPackages` and manually modified it to confirm his findings, and it turned out it was indeed the case.
Our hero had damaged the dragon and it was time for a celebration.

He went out and acquired some mead which he ingested while he typed up his story and waited for the dragon to finally die (the commit caused a mass-rebuild, I had to wait for my repro).

In the end all was good in [NixOps-AWS](https://github.com/nixos/nixops-aws)-town and type checks could run. (PR for that incoming tomorrow).
2020-03-14 21:39:31 +00:00
Frederik Rietdijk bd47c5721f Python: introduce NIX_PYTHONEXECUTABLE in order to set sys.executable
This is needed in case of `python.buildEnv` to make sure sys.executable
does not point to the unwrapped executable.
2019-07-27 11:52:49 +02:00
Frederik Rietdijk ec7f569211 python.buildEnv: use NIX_PYTHONPATH 2019-07-13 09:37:33 +02:00
Tom McLaughlin a3f24daa7b Add flag to disable PYTHONNOUSERSITE for wrapped binaries in python environments 2019-05-13 02:56:56 -07:00
Frederik Rietdijk 548ced628d python.buildEnv: new argument makeWrapperArgs
`python.buildEnv` would already wrap executables exporting `PYTHONHOME`.
With this change, it is possible to pass in additional arguments to the
underlying `makeWrapper`.
2018-10-13 12:31:52 +02:00
Frederik Rietdijk 163ba09117 python.buildEnv: always include the $out output
28299f669a introduced the first Python
packages having multiple outputs. The required outputs were not picked
up by `python.buildEnv` (#31857).

This commit modifies `python.buildEnv` so that it always includes the
$out output and thus fixes #31857.
2017-12-10 15:21:35 +01:00
Frederik Rietdijk 40851a4d26 Python: the pythonModule attribute
Python libraries or modules now have an attribute `pythonModule = interpreter;` to indicate
they provide Python modules for the specified `interpreter`.

The package set provides the following helper functions:

- hasPythonModule: Check whether a derivation provides a Python module.
- requiredPythonModules: Recurse into a list of Python modules, returning all Python modules that are required.
- makePythonPath: Create a PYTHONPATH from a list of Python modules.

Also included in this commit is:
- disabledIf: Helper function for disabling non-buildPythonPackage functions.
2017-11-23 15:11:02 +01:00
Nikolay Amiantov 74c3cdd893 python.buildEnv: add extraOutputsToInstall attribute 2017-09-25 22:40:09 +03:00
Robin Gloster 4495bfe138
python.buildEnv: only wrap executables 2017-08-09 15:07:03 +02:00
Frederik Rietdijk 6e6271e82f python.buildEnv: undo removal of passthru.python 2017-08-02 15:05:00 +02:00
Frederik Rietdijk ea4121d225 python.buildEnv: fix passthru
Python envs did not pass through any of the properties the Python
interpreter has. That could be annoying, especially not having
`python.interpreter` which is the path to the interpreter. This commit
fixes the situation and inherit python.passthru.
2017-08-01 16:27:16 +02:00
Frederik Rietdijk 85a0ab4b23 Python: disable user site-packages for programs and environments.
Python by default checks a `site-packages` folder in the user's home
folder. We do not want such an impurity and therefore disable it.
Fixes #26846.
2017-07-31 13:16:29 +02:00
Frederik Rietdijk 1dbb72b91a python.buildenv: don't filter non-python packages
python.buildenv is used to build an env that provides binaries that can
import all modules that were passed in to the env.

Before this change it filtered the propagatedBuildInputs to remove all
non-Python packages, thereby possibly reducing the amount of packages
that were referenced. However, Python packages often don't have non-
Python packages as propagatedBuildInputs. And occasionally, we do want
to be able to add other packages to the env.
2016-09-17 14:28:12 +02:00
Frederik Rietdijk 12e8f3c6aa python: apply wrapper to all packages in python.buildEnv extraLibs
Currently, when constructing a buildEnv and adding packages via
extraLibs, then binaries in extraLibs cannot access the other Python
modules. An example is having ipython/jupyter in extraLibs; in that case
ipython cannot import any other modules.
2015-11-30 10:52:57 +01:00
Nikolay Amiantov 344e522166 python: add .env for convenient nix-shell's 2015-08-17 21:22:50 +03:00
Domen Kožar 2d6582d14e python27FullBuildEnv -> python.buildEnv for all interpreters 2014-10-19 19:48:18 +02:00
Domen Kožar a2a7abc67c pythonFull -> python with all modules, pythonFullWithPkgs -> buildEnv 2014-10-13 21:35:21 +02:00
Peter Simons 51c9dbc9f3 python-wrapper: add 'ignoreCollisions' parameter (which default to 'false') 2014-02-24 12:41:12 +01:00
Shea Levy 06fe4d9904 Partial revert of b09f8110db
Didn't mean to commit this change

Signed-off-by: Shea Levy <shea@shealevy.com>
2013-11-19 09:04:32 -05:00
Shea Levy b09f8110db nspr: Bump to 4.10.2
Signed-off-by: Shea Levy <shea@shealevy.com>
2013-11-18 15:23:30 -05:00
Peter Simons 29588edfe1 python-wrapper: split 'extraLibs' into 'stdLibs' and 'extraLibs', and add 'postBuild' step
The default setting for extraLibs used to be the set of modules that come with
python by default but aren't usually enabled in our standard python derivation
because they require additional libraries. This meant that users who want to
*add* libraries to that set had to use a fairly complicated override, to add
more entries without loosing the ones set by default.

After this patch, the "standard libraries" such as "curses' are listed in
stdLibs while the extraLibs argument remains empty by default. This allows
users to override extraLibs without overriding the standard libraries.

Furthermore, the wrapper environment can be messed around with in an
additional 'postBuild' step. One nice application of this build step is
to patch scripts and binaries to use the wrapped python interpreter
instead of the pristine one, thereby enabling them to pick up all
modules that have been configured. The following example shows how this
is done for the 'pylint' utility:

  pkgs.python27Full.override {
    extraLibs = [pkgs.pylint];
    postBuild = ''
      cd ${pkgs.pylint}/bin
      for i in *; do
        rm $out/bin/$i
        sed -r -e "s|^exec |exec $out/bin/python -- |" <$i >$out/bin/$i
        chmod +x $out/bin/$i
      done;
    '';
  };
2013-11-07 15:13:02 +01:00
Peter Simons 46419ae454 python-wrapper: recursively include all dependencies of the specified 'extraLibs' in the generated environment
This patch means that adding 'matplotlib' to extraLibs will automatically
include 'numpy', too, because matplotlib depends on it.
2013-11-07 14:00:08 +01:00
Shea Levy f6e835a2c3 Fix pythonWrapper when all of the binaries come from python
See discussion in #834

Signed-off-by: Shea Levy <shea@shealevy.com>
2013-10-03 15:25:43 -04:00
Peter Simons 742d6597ca Re-implement python-wrapper with buildEnv.
The new wrapper creates an environment that contains all files from
Python and the extra libraries that have been specified. All files are
found at run-time by means of the $PYTHONHOME variable; the wrapper no
longer uses $PYTHONPATH.
2013-10-02 22:47:19 +02:00
Florian Friesdorf 870f4cca73 fix pythonWrapper for non-gnu ln 2013-03-25 15:13:08 +01:00
Florian Friesdorf 212b4df91c recursivePthLoader included via wrapper, not propagated by modules 2013-01-11 09:53:28 +01:00
Florian Friesdorf 3665e02901 python: make pdb.py available as bin/pdb and bin/pdb${python.majorVersion} 2012-07-20 21:43:14 +02:00
Florian Friesdorf 4956884de5 Revert "do not propagate makeWrapper via pythonXYFull"
This reverts commit 3ee2667e4c60c2ed850da8538cf135fe7f716f30.

svn path=/nixpkgs/branches/stdenv-updates/; revision=32656
2012-02-28 00:07:33 +00:00
Florian Friesdorf bf1fe3aa09 Revert "pysite support for pythonXYFull wrapper"
This reverts commit f77b9a16a9ef52951a601997593dc557a42660b9.

svn path=/nixpkgs/branches/stdenv-updates/; revision=32653
2012-02-28 00:07:23 +00:00
Florian Friesdorf 1d2e06e068 pysite support for pythonXYFull wrapper
svn path=/nixpkgs/branches/stdenv-updates/; revision=32593
2012-02-26 17:23:42 +00:00
Florian Friesdorf 95f0b6119c do not propagate makeWrapper via pythonXYFull
svn path=/nixpkgs/branches/stdenv-updates/; revision=32589
2012-02-26 17:23:28 +00:00
Florian Friesdorf fe9d9530ba symlink python manpage for pythonXYFull
svn path=/nixpkgs/branches/stdenv-updates/; revision=32588
2012-02-26 17:23:25 +00:00
Florian Friesdorf 09ae58fcd7 python wrapper comment
svn path=/nixpkgs/branches/stdenv-updates/; revision=32587
2012-02-26 17:23:22 +00:00
Eelco Dolstra c556a6ea46 * "ensureDir" -> "mkdir -p". "ensureDir" is a rather pointless
function, so obsolete it.

svn path=/nixpkgs/branches/stdenv-updates/; revision=31644
2012-01-18 20:16:00 +00:00
Peter Simons f01ceffb9e pkgs/development/interpreters/python/wrapper.nix: clean up debug code
svn path=/nixpkgs/trunk/; revision=23200
2010-08-17 10:17:32 +00:00
Peter Simons 9670fca780 Added "python-$version-wrapper" expression.
The python wrapper expression expects a list of Python modules, $extraLibs,
which are added to $PYTHONPATH before executing the actual Python interpreter.

svn path=/nixpkgs/trunk/; revision=23194
2010-08-16 17:03:35 +00:00