Commit graph

4305 commits

Author SHA1 Message Date
Vladimír Čunát 13add13388
Merge branch 'master' into staging-next
... to resolve a trivial conflict in libpcap.
2020-06-10 20:00:44 +02:00
Vladimír Čunát a5f5d020c6
Merge branch 'staging-next' 2020-06-10 16:13:48 +02:00
R. RyanTM 81c9cc0b60 joker: 0.15.3 -> 0.15.4 2020-06-10 01:51:27 +00:00
Daniël de Kok 1e2b6695cf pythonPackages.setuptoolsBuildHook: do not build in an isolated environment
When a PEP 517 project file is present, pip will not install
prerequisites in `site-packages`:

https://pip.pypa.io/en/stable/reference/pip/#pep-517-and-518-support

For the shell hook, this has the consequence that the generated
temporary directory that is added to PYTHONPATH does not contain
`site.py`. As a result, Python does not discover the Python
module. Thus when a user executes nix-shell in a project, they cannot
import the project's Python module.

This change adds the `--no-build-isolation` option to pip when
creating the editable environment, to correctly generate `site.py`,
even when a `pyproject.toml` is present.
2020-06-06 10:05:26 +02:00
Daniël de Kok e2309df85e pythonPackages.pipBuildHook: do not build in an isolated environment
When a PEP 517 project file is present, pip will not install
prerequisites in `site-packages`:

https://pip.pypa.io/en/stable/reference/pip/#pep-517-and-518-support

For the shell hook, this has the consequence that the generated
temporary directory that is added to PYTHONPATH does not contain
`site.py`. As a result, Python does not discover the Python
module. Thus when a user executes nix-shell in a project, they cannot
import the project's Python module.

This change adds the `--no-build-isolation` option to pip when
creating the editable environment, to correctly generate `site.py`,
even when a `pyproject.toml` is present.
2020-06-06 10:05:26 +02:00
Frederik Rietdijk 1c68570ab2 Merge staging-next into staging 2020-06-05 19:42:16 +02:00
Frederik Rietdijk 43f71029cc Merge master into staging-next 2020-06-05 19:40:53 +02:00
Frederik Rietdijk 913bee36ed python3Minimal: override python38, not python3
This avoids an infinite recursion, accidentally introduced in b7ff746540.
2020-06-05 16:46:40 +02:00
Frederik Rietdijk a337c44db6 python3Minimal: disable optimizations
No point for the bootstrapping.
2020-06-04 20:53:31 +02:00
Frederik Rietdijk bcf03e8cd2 Revert "cpython: Optimize dynamic symbol tables, for a 6% speedup."
ofborg does not like fetching patches when the derivation is used during bootstrapping.

This reverts commit 480c8d1991.
2020-06-04 20:36:31 +02:00
Frederik Rietdijk a2be64bf13
Merge pull request #84072 from gnprice/python-build
cpython: Use optimizations, for a 25% speedup.
2020-06-04 18:31:07 +02:00
volth d89c58a012 perl: 5.30.2 -> 5.30.3 2020-06-04 18:14:24 +02:00
Frederik Rietdijk b7ff746540 python3: now points to python38
Note this also means python3Minimal is now also Python 3.8.

This reverts commit eb1369670b and adds more.
2020-06-04 18:08:29 +02:00
Luflosi 2379e36124 python39: fix build on macOS
Basically the same changes as in 81d15948cc but for python3.9 instead of python3.8.
2020-06-04 17:11:29 +02:00
Frederik Rietdijk 36d9eeb9c7 Merge staging-next into staging 2020-05-29 17:06:01 +02:00
Frederik Rietdijk c7d25a5db6 Merge master into staging-next 2020-05-29 17:05:38 +02:00
talyz bfdc832aef
php.buildEnv: Let enabled extensions to pass through by default
If only extraConfig is specified in buildEnv, keep the currently
enabled extensions active. Brought up in #89011.
2020-05-29 00:32:12 +02:00
Frederik Rietdijk 362d88c2b1 Merge staging-next into staging 2020-05-27 15:27:28 +02:00
Frederik Rietdijk 1b7204ab3c Merge master into staging-next 2020-05-27 15:26:50 +02:00
Frederik Rietdijk 0367fa630d python38: 3.8.2 -> 3.8.3 2020-05-27 12:10:25 +02:00
Ryan Mulligan f1d9510c99
Merge pull request #88761 from r-ryantm/auto-update/rakudo
rakudo: 2020.05 -> 2020.05.1
2020-05-25 07:50:52 -07:00
Jan Tojnar f8ac6112b5
Merge pull request #84994 from r-ryantm/auto-update/spidermonkey 2020-05-25 15:22:42 +02:00
R. RyanTM 926e43c1c4 rakudo: 2020.05 -> 2020.05.1 2020-05-24 09:02:40 +00:00
Frederik Rietdijk f17001afd8 Python: fix virtualenv with Python 2 2020-05-24 10:43:24 +02:00
Frederik Rietdijk 98bcf5d8da Python tests: fix use of is_virtualenv
Too many tests set it.
2020-05-24 10:43:24 +02:00
Frederik Rietdijk c778596f56 Merge master into staging-next 2020-05-24 10:03:22 +02:00
Frederik Rietdijk 2de446e0b8 python.tests: also test virtualenv
Test whether creating a virtualenv functions.
2020-05-23 18:15:45 +02:00
adisbladis 203f382a4a
pypy: Remove bootstrap python from closure 2020-05-23 11:47:11 +01:00
Frederik Rietdijk b34551384a Merge master into staging-next 2020-05-23 10:24:52 +02:00
Benjamin Andresen 47d4a68bb1 babashka: 0.0.94 -> 0.0.97 2020-05-21 00:10:29 +02:00
R. RyanTM 754562f855 janet: 1.8.1 -> 1.9.1 2020-05-19 08:05:24 +00:00
Jan Tojnar 7f40cfd97b
Merge branch 'master' into staging-next 2020-05-18 21:09:27 +02:00
Mario Rodas 033267eb93
Merge pull request #87921 from ggreif/wasmtime
wasmtime: 0.15.0 -> 0.16.0
2020-05-16 17:33:30 -05:00
Elis Hirwing 726cbe0d40
Merge pull request #87907 from etu/php74-update
php74: 7.4.4 -> 7.4.6
2020-05-16 14:27:44 +02:00
Elis Hirwing a779efcaa0
php74: 7.4.5 -> 7.4.6
Changelog: https://www.php.net/ChangeLog-7.php#7.4.6
2020-05-16 13:23:23 +02:00
Jon 15b3d9d277
python3Packages.venvShellHook: add postVenvCreation (#87850)
* python3Packages.venvShellHook: add postVenvCreation

* python: docs: add postVenvCreation explaination
2020-05-16 09:34:11 +02:00
Gabor Greif d89ae377d4 wasmtime-0.16.0: re-enable tests on Darwin
The issue https://github.com/bytecodealliance/wasmtime/issues/1556
appears to be fixed.
2020-05-16 04:56:26 +02:00
Gabor Greif b39717abff wasmtime: 0.15.0 -> 0.16.0 2020-05-16 04:42:02 +02:00
Elis Hirwing 52d2e99182
php74: 7.4.4 -> 7.4.5
Changelog: https://www.php.net/ChangeLog-7.php#7.4.5
2020-05-15 22:10:03 +02:00
Frederik Rietdijk 92a26320e7 Merge master into staging-next 2020-05-14 09:25:25 +02:00
Colin L Rice d6162dab50
go-modules: Update files to use vendorSha256 2020-05-14 07:22:21 +01:00
Greg Price 480c8d1991 cpython: Optimize dynamic symbol tables, for a 6% speedup.
I took a close look at how Debian builds the Python interpreter,
because I noticed it ran substantially faster than the one in nixpkgs
and I was curious why.

One thing that I found made a material difference in performance was
this pair of linker flags (passed to the compiler):

    -Wl,-O1 -Wl,-Bsymbolic-functions

In other words, effectively the linker gets passed the flags:

    -O1 -Bsymbolic-functions

Doing the same thing in nixpkgs turns out to make the interpreter
run about 6% faster, which is quite a big win for such an easy
change.  So, let's apply it.

---

I had not known there was a `-O1` flag for the *linker*!
But indeed there is.

These flags are unrelated to "link-time optimization" (LTO), despite
the latter's name.  LTO means doing classic compiler optimizations
on the actual code, at the linking step when it becomes possible to
do them with cross-object-file information.  These two flags, by
contrast, cause the linker to make certain optimizations within the
scope of its job as the linker.

Documentation is here, though sparse:
  https://sourceware.org/binutils/docs-2.31/ld/Options.html

The meaning of -O1 was explained in more detail in this LWN article:
  https://lwn.net/Articles/192624/
Apparently it makes the resulting symbol table use a bigger hash
table, so the load factor is smaller and lookups are faster.  Cool.

As for -Bsymbolic-functions, the documentation indicates that it's a
way of saving lookups through the symbol table entirely.  There can
apparently be situations where it changes the behavior of a program,
specifically if the program relies on linker tricks to provide
customization features:
  https://bugs.launchpad.net/ubuntu/+source/xfe/+bug/644645
  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=637184#35
But I'm pretty sure CPython doesn't permit that kind of trick: you
don't load a shared object that tries to redefine some symbol found
in the interpreter core.

The stronger reason I'm confident using -Bsymbolic-functions is
safe, though, is empirical.  Both Debian and Ubuntu have been
shipping a Python built this way since forever -- it was introduced
for the Python 2.4 and 2.5 in Ubuntu "hardy", and Debian "lenny",
released in 2008 and 2009.  In those 12 years they haven't seen a
need to drop this flag; and I've been unable to locate any reports
of trouble related to it, either on the Web in general or on the
Debian bug tracker.  (There are reports of a handful of other
programs breaking with it, but not Python/CPython.)  So that seems
like about as thorough testing as one could hope for.

---

As for the performance impact: I ran CPython upstream's preferred
benchmark suite, "pyperformance", in the same way as described in
the previous commit.  On top of that commit's change, the results
across the 60 benchmarks in the suite are:

The median is 6% faster.

The middle half (aka interquartile range) is from 4% to 8% faster.

Out of 60 benchmarks, 3 come out slower, by 1-4%.  At the other end,
5 are at least 10% faster, and one is 17% faster.

So, that's quite a material speedup!  I don't know how big the
effect of these flags is for other software; but certainly CPython
tends to do plenty of dynamic linking, as that's how it loads
extension modules, which are ubiquitous in the stdlib as well as
popular third-party libraries.  So perhaps that helps explain why
optimizing the dynamic linker has such an impact.
2020-05-13 21:24:30 -07:00
Greg Price 52c04b0347 cpython: Use autoreconfHook to rebuild configure script.
In particular this will let us use patches that apply to configure.ac.
2020-05-13 21:23:48 -07:00
Michael Raskin b2f83be34d
Merge pull request #87546 from FireyFly/pkgs/j/avx-flag
j: add avxSupport option
2020-05-13 23:31:25 +00:00
Jonas Höglund 20235a89f8 j: add avxSupport option 2020-05-14 00:51:13 +02:00
Mario Rodas 6a16787d26
Merge pull request #87649 from filalex77/wasmer-0.17.0
wasmer: 0.16.2 -> 0.17.0
2020-05-13 00:58:59 -05:00
Oleksii Filonenko e2a5709d9c wasmer: 0.16.2 -> 0.17.0 2020-05-12 08:01:43 +00:00
Greg Price f8a8243bd3 cpython: Use --enable-optimizations, for a 16% speedup.
Without this flag, the configure script prints a warning at the end,
like this (reformatted):

  If you want a release build with all stable optimizations active
  (PGO, etc), please run ./configure --enable-optimizations

We're doing a build to distribute to people for day-to-day use,
doing things other than developing the Python interpreter.  So
that's certainly a release build -- we're the target audience for
this recommendation.

---

And, trying it out, upstream isn't kidding!  I ran the standard
benchmark suite that the CPython developers use for performance
work, "pyperformance".  Following its usage instructions:
  https://pyperformance.readthedocs.io/usage.html
I ran the whole suite, like so:

  $ nix-shell -p ./result."$variant" --run '
      cd $(mktemp -d); python -m venv venv; . venv/bin/activate
      pip install pyperformance
      pyperformance run -o ~/tmp/result.'"$variant"'.json
    '

and then examined the results with commands like:

  $ python -m pyperf compare_to --table -G \
      ~/tmp/result.{$before,$after}.json

Across all the benchmarks in the suite, the median speedup was 16%.
(Meaning 1.16x faster; 14% less time).

The middle half of them ranged from a 13% to a 22% speedup.

Each of the 60 benchmarks in the suite got faster, by speedups
ranging from 3% to 53%.

---

One reason this isn't just the default to begin with is that, until
recently, it made the build a lot slower.  What it does is turn on
profile-guided optimization, which means first build for profiling,
then run some task to get a profile, then build again using the
profile.  And, short of further customization, the task it would use
would be nearly the full test suite, which includes a lot of
expensive and slow tests, and can easily take half an hour to run.

Happily, in 2019 an upstream developer did the work to carefully
select a more appropriate set of tests to use for the profile:
  https://github.com/python/cpython/commit/4e16a4a31
  https://bugs.python.org/issue36044
This suite takes just 2 minutes to run.  And the resulting final
build is actually slightly faster than with the much longer suite,
at least as measured by those standard "pyperformance" benchmarks.
That work went into the 3.8 release, but the same list works great
if used on older releases too.

So, start passing that --enable-optimizations flag; and backport
that good-for-PGO set of tests, so that we use it on all releases.
2020-05-11 23:37:04 -07:00
Jörg Thalheim fa44d3ad71
Merge pull request #86593 from bandresen/babashka_versionbump
babashka: 0.0.89 -> 0.0.94
2020-05-12 00:00:39 +01:00
Benjamin Andresen cb9b1167e3 babashka: 0.0.89 -> 0.0.94 2020-05-11 22:57:49 +02:00