Commit graph

75 commits

Author SHA1 Message Date
Sandro Jäckel 3453b89f4b
lzma: deprecate alias 2021-04-04 19:49:52 +02:00
Sandro Jäckel b45c06fde8
cpython: allow $out reference again
This regression got introduced with 738746a34b
2021-03-15 03:20:32 +01:00
Frederik Rietdijk 738746a34b python3: express references to keep as a nix list 2021-03-14 17:00:59 +01:00
Frederik Rietdijk 5b1abe5dd2 python3: mimetypesSupport is optional 2021-03-14 15:45:15 +01:00
nicoo faf0f8a8a4 cpython: Pull in the mime-types database
Closes #113901
2021-03-14 15:45:15 +01:00
Frederik Rietdijk 7617b03403 python39: hardcode path to tzdata
Python 3.9 has a new module `zoneinfo` which requires tzdata. By default
it searches TZPATH for folders containing `zoneinfo`.

This commit makes the dependency on tzdata pure.
2021-03-13 14:05:41 +01:00
Frederik Rietdijk 9d03ff5222 python: reproducible builds
Achieve reproducible builds of the interpreter. Note this meant
disabling optimizations again.
2021-03-13 13:11:50 +01:00
Ivan Babrou b00c7c2d1d python37, python2: remove win64 workaround to fix aarch64-darwin
The issue manifests itself as the following on `aarch64-darwin`:

```
>>> import ctypes
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/nix/store/i8cq0xrjirz1rcp65wzcyhj6ypzlw9il-python3-3.7.10/lib/python3.7/ctypes/__init__.py", line 551, in <module>
    _reset_cache()
  File "/nix/store/i8cq0xrjirz1rcp65wzcyhj6ypzlw9il-python3-3.7.10/lib/python3.7/ctypes/__init__.py", line 273, in _reset_cache
    CFUNCTYPE(c_int)(lambda: None)
MemoryError
```

The commit we backport is included in Python 3.8, and it reverts
the change that was introduced all the way back in Python 2.7.
2021-03-03 16:02:07 -08:00
Martin Weinelt 92087cb170 python37: 3.7.9 -> 3.7.10, fixup patches 2021-02-20 12:13:07 +01:00
Ben Siraphob 001c0cbe54 pkgs/development/interpreters: stdenv.lib -> lib 2021-01-23 20:29:03 +07:00
Peter Woodman 858eebe1f9 cpython: fix extensions when using a musl toolchain 2021-01-22 23:56:19 +01:00
Jan Tojnar b813710c04
Merge branch 'master' into staging-next 2021-01-14 02:24:17 +01:00
Fabian Möller 07e7cea102 python36: enable ctypes.util.find_library patch 2021-01-13 13:13:02 +01:00
github-actions[bot] 7aced83bf9
Merge staging-next into staging 2021-01-08 12:29:39 +00:00
Frederik Rietdijk 5c8df2235a python3: don't use sysconfigdataHook on darwin
Resolves issue when building wheels which is a regression introduced by
1a65c5df5f

```
AssertionError: would build wheel with unsupported tag ('cp38', 'cp38',
'darwin_x86_64')
```
2021-01-08 09:34:36 +01:00
Jan Tojnar 0ea0dacc20
Merge branch 'staging-next' into staging 2021-01-07 13:06:41 +01:00
Jan Tojnar f19eb635b4
Merge branch 'master' into staging-next
b04fc593e7 seems to have accidentally changed mkDerivation function for dfilemanager and solarus-quest-editor so I have reverted that here.
2021-01-07 13:04:31 +01:00
Jonathan Ringer 16174037b3 python38: remove obsolete find_library patch 2021-01-04 15:44:35 -08:00
John Ericson 5c2965145f treewide: Inline more of the static overlay
Picking up where #107238 left off. I think I'll have gotten all the easy
stuff with this.
2021-01-03 21:46:14 +00:00
Ben Wolsieffer 1a65c5df5f cpython: fix finding headers when cross-compiling extension modules 2020-12-28 13:09:00 +01:00
Lucas Ransan 188f1375d8 cpython: fix ctypes.util.find_library 2020-12-28 08:43:32 +01:00
Frederik Rietdijk 6cf25f9dbd Python: rename parameters and arguments passed to passthru
As part of the splicing the build/host/target combinations of the interpreter
need to be passed around internally. The chosen names were not very clear,
implying they were package sets whereas actually there were derivations.
2020-11-28 17:36:23 +01:00
Frederik Rietdijk cce2fd547b Python: use pythonPackagesBuildHost instead of pythonForBuild
Follow-up to #104201, related to #105113.
2020-11-28 16:36:03 +01:00
John Ericson b57c5d4456 python: Use makeScopeWithSplicing
Now non-`buildInputs` that are python packages should be resolved
correctly.
2020-11-19 11:58:07 -05:00
Jonathan Ringer 5a5122418a python310: 3.10.0a1 -> 3.10.0a2 2020-11-11 23:13:44 -08:00
Frederik Rietdijk 6ddd6b4914 Revert "Merge pull request #93083 from risicle/ris-cpython-debug"
This increase the closure size by 30 MB.

https://github.com/NixOS/nixpkgs/pull/93083#issuecomment-718025366

This reverts commit 4b340cbbb4, reversing
changes made to 47e499f7a4.
2020-11-03 12:50:05 +01:00
Frederik Rietdijk 0ac85bc455 Merge master into staging-next 2020-08-17 14:54:39 +02:00
Alvar Penning b7ce309e6c pythonFull: add BlueZ support
This commit introduces two changes.

First, cpython gets optional BlueZ support, which is needed for
AF_BLUETOOTH sockets. Therefore bluezSupport was added as a parameter.

Second, the call to the pythonFull packages has been adjusted. The
Python packages have a self-reference called self. This was not adjusted
for the override. As a result, Python packages for this special version
of Python were not built with the overridden Python, but with the
original one.
2020-08-15 18:08:20 +02:00
Frederik Rietdijk 4b340cbbb4
Merge pull request #93083 from risicle/ris-cpython-debug
cpython: add separateDebugInfo, allow use of gdb libpython
2020-08-15 08:52:42 +02:00
Daniël de Kok 900b2d5a6c python3: fix impure /bin/sh call in subprocess 2020-08-15 08:34:02 +02:00
Matthew Bauer d0677e6d45 treewide: add warning comment to “boot” packages
This adds a warning to the top of each “boot” package that reads:

  Note: this package is used for bootstrapping fetchurl, and thus cannot
  use fetchpatch! All mutable patches (generated by GitHub or cgit) that
  are needed here should be included directly in Nixpkgs as files.

This makes it clear to maintainer that they may need to treat this
package a little differently than others. Importantly, we can’t use
fetchpatch here due to using <nix/fetchurl.nix>. To avoid having stale
hashes, we need to include patches that are subject to changing
overtime (for instance, gitweb’s patches contain a version number at
the bottom).
2020-07-31 08:56:53 +02:00
Robert Scott ede2e00c9f cpython: expose gdb libpython.py
used together with cpython's debugging symbols, this allows inspection of
the python stack of cpython programs in gdb. this file is a little
different from the rest of the python output by this package, in that it's
not intended to be run by the current python being built, instead by the
python being used by the gdb in question, which could be very different.
therefore placed in its own, but hopefully logical & predictable location.
2020-07-15 19:39:19 +01:00
Robert Scott 6d19ab339a cpython: set separateDebugInfo 2020-07-13 22:30:55 +01:00
Frederik Rietdijk f03734efa0 python3: improve cross-compilation of extension modules, fixes #91171 2020-06-21 16:29:57 +02:00
Frederik Rietdijk 4087d3fe41 python: don't use optimizations on Darwin
Also, don't use autoreconfHook on Darwin with Python 3.
Darwin builds are still impure and fail with

    ld: warning: directory not found for option '-L/nix/store/6yhj9djska835wb6ylg46d2yw9dl0sjb-configd-osx-10.8.5/lib'
    ld: warning: directory not found for option '-L/nix/store/6yhj9djska835wb6ylg46d2yw9dl0sjb-configd-osx-10.8.5/lib'
    ld: warning: object file (/nix/store/0lsij4jl35bnhqhdzla8md6xiswgig5q-Libsystem-osx-10.12.6/lib/crt1.10.6.o) was built for newer OSX version (10.12) than being linked (10.6)
    DYLD_LIBRARY_PATH=/private/tmp/nix-build-python3-3.8.3.drv-0/Python-3.8.3 ./python.exe -E -S -m sysconfig --generate-posix-vars ;\
    if test $? -ne 0 ; then \
            echo "generate-posix-vars failed" ; \
            rm -f ./pybuilddir.txt ; \
            exit 1 ; \
    fi
    /nix/store/dsb7d4dwxk6bzlm845z2zx6wp9a8bqc1-bash-4.4-p23/bin/bash: line 5: 72015 Killed: 9               DYLD_LIBRARY_PATH=/private/tmp/nix-build-python3-3.8.3.drv-0/Python-3.8.3 ./python.exe -E -S -m sysconfig --generate-posix-vars
    generate-posix-vars failed
    make: *** [Makefile:592: pybuilddir.txt] Error 1
2020-06-12 18:29:08 +02:00
Frederik Rietdijk 1c68570ab2 Merge staging-next into staging 2020-06-05 19:42:16 +02:00
Frederik Rietdijk a337c44db6 python3Minimal: disable optimizations
No point for the bootstrapping.
2020-06-04 20:53:31 +02:00
Frederik Rietdijk bcf03e8cd2 Revert "cpython: Optimize dynamic symbol tables, for a 6% speedup."
ofborg does not like fetching patches when the derivation is used during bootstrapping.

This reverts commit 480c8d1991.
2020-06-04 20:36:31 +02:00
Frederik Rietdijk a2be64bf13
Merge pull request #84072 from gnprice/python-build
cpython: Use optimizations, for a 25% speedup.
2020-06-04 18:31:07 +02:00
Luflosi 2379e36124 python39: fix build on macOS
Basically the same changes as in 81d15948cc but for python3.9 instead of python3.8.
2020-06-04 17:11:29 +02:00
Greg Price 480c8d1991 cpython: Optimize dynamic symbol tables, for a 6% speedup.
I took a close look at how Debian builds the Python interpreter,
because I noticed it ran substantially faster than the one in nixpkgs
and I was curious why.

One thing that I found made a material difference in performance was
this pair of linker flags (passed to the compiler):

    -Wl,-O1 -Wl,-Bsymbolic-functions

In other words, effectively the linker gets passed the flags:

    -O1 -Bsymbolic-functions

Doing the same thing in nixpkgs turns out to make the interpreter
run about 6% faster, which is quite a big win for such an easy
change.  So, let's apply it.

---

I had not known there was a `-O1` flag for the *linker*!
But indeed there is.

These flags are unrelated to "link-time optimization" (LTO), despite
the latter's name.  LTO means doing classic compiler optimizations
on the actual code, at the linking step when it becomes possible to
do them with cross-object-file information.  These two flags, by
contrast, cause the linker to make certain optimizations within the
scope of its job as the linker.

Documentation is here, though sparse:
  https://sourceware.org/binutils/docs-2.31/ld/Options.html

The meaning of -O1 was explained in more detail in this LWN article:
  https://lwn.net/Articles/192624/
Apparently it makes the resulting symbol table use a bigger hash
table, so the load factor is smaller and lookups are faster.  Cool.

As for -Bsymbolic-functions, the documentation indicates that it's a
way of saving lookups through the symbol table entirely.  There can
apparently be situations where it changes the behavior of a program,
specifically if the program relies on linker tricks to provide
customization features:
  https://bugs.launchpad.net/ubuntu/+source/xfe/+bug/644645
  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=637184#35
But I'm pretty sure CPython doesn't permit that kind of trick: you
don't load a shared object that tries to redefine some symbol found
in the interpreter core.

The stronger reason I'm confident using -Bsymbolic-functions is
safe, though, is empirical.  Both Debian and Ubuntu have been
shipping a Python built this way since forever -- it was introduced
for the Python 2.4 and 2.5 in Ubuntu "hardy", and Debian "lenny",
released in 2008 and 2009.  In those 12 years they haven't seen a
need to drop this flag; and I've been unable to locate any reports
of trouble related to it, either on the Web in general or on the
Debian bug tracker.  (There are reports of a handful of other
programs breaking with it, but not Python/CPython.)  So that seems
like about as thorough testing as one could hope for.

---

As for the performance impact: I ran CPython upstream's preferred
benchmark suite, "pyperformance", in the same way as described in
the previous commit.  On top of that commit's change, the results
across the 60 benchmarks in the suite are:

The median is 6% faster.

The middle half (aka interquartile range) is from 4% to 8% faster.

Out of 60 benchmarks, 3 come out slower, by 1-4%.  At the other end,
5 are at least 10% faster, and one is 17% faster.

So, that's quite a material speedup!  I don't know how big the
effect of these flags is for other software; but certainly CPython
tends to do plenty of dynamic linking, as that's how it loads
extension modules, which are ubiquitous in the stdlib as well as
popular third-party libraries.  So perhaps that helps explain why
optimizing the dynamic linker has such an impact.
2020-05-13 21:24:30 -07:00
Greg Price 52c04b0347 cpython: Use autoreconfHook to rebuild configure script.
In particular this will let us use patches that apply to configure.ac.
2020-05-13 21:23:48 -07:00
Greg Price f8a8243bd3 cpython: Use --enable-optimizations, for a 16% speedup.
Without this flag, the configure script prints a warning at the end,
like this (reformatted):

  If you want a release build with all stable optimizations active
  (PGO, etc), please run ./configure --enable-optimizations

We're doing a build to distribute to people for day-to-day use,
doing things other than developing the Python interpreter.  So
that's certainly a release build -- we're the target audience for
this recommendation.

---

And, trying it out, upstream isn't kidding!  I ran the standard
benchmark suite that the CPython developers use for performance
work, "pyperformance".  Following its usage instructions:
  https://pyperformance.readthedocs.io/usage.html
I ran the whole suite, like so:

  $ nix-shell -p ./result."$variant" --run '
      cd $(mktemp -d); python -m venv venv; . venv/bin/activate
      pip install pyperformance
      pyperformance run -o ~/tmp/result.'"$variant"'.json
    '

and then examined the results with commands like:

  $ python -m pyperf compare_to --table -G \
      ~/tmp/result.{$before,$after}.json

Across all the benchmarks in the suite, the median speedup was 16%.
(Meaning 1.16x faster; 14% less time).

The middle half of them ranged from a 13% to a 22% speedup.

Each of the 60 benchmarks in the suite got faster, by speedups
ranging from 3% to 53%.

---

One reason this isn't just the default to begin with is that, until
recently, it made the build a lot slower.  What it does is turn on
profile-guided optimization, which means first build for profiling,
then run some task to get a profile, then build again using the
profile.  And, short of further customization, the task it would use
would be nearly the full test suite, which includes a lot of
expensive and slow tests, and can easily take half an hour to run.

Happily, in 2019 an upstream developer did the work to carefully
select a more appropriate set of tests to use for the profile:
  https://github.com/python/cpython/commit/4e16a4a31
  https://bugs.python.org/issue36044
This suite takes just 2 minutes to run.  And the resulting final
build is actually slightly faster than with the much longer suite,
at least as measured by those standard "pyperformance" benchmarks.
That work went into the 3.8 release, but the same list works great
if used on older releases too.

So, start passing that --enable-optimizations flag; and backport
that good-for-PGO set of tests, so that we use it on all releases.
2020-05-11 23:37:04 -07:00
Josef Kemetmüller 3818cd9049 python: Fix creating RPMs from Python packages
This should enable (manual) building of RPMs from python projects using
the `python setup.py bdist_rpm` command on systems where `rpmbuild` is
not located in `/usr/bin/`. (e.g. NixOS)
The discovery of the rpmbuild command was fixed upstream in Python 3.8,
so this commit backports the relevant patch to our currently supported
Python 3 versions.

Fixes: #85204
2020-05-09 11:15:28 +02:00
Michael Reilly 84cf00f980
treewide: Per RFC45, remove all unquoted URLs 2020-04-10 17:54:53 +01:00
Jan Tojnar 3e0f4e202f
Merge branch 'master' into staging-next 2020-03-31 21:32:15 +02:00
Greg Price 9d8831c8fe cpython: Drop unrecognized --with-threads configure flag.
The ./configure script prints a warning when passed this flag,
starting with 3.7:

  configure: WARNING: unrecognized options: --with-threads

The reason is that there's no longer such a thing as a build
without threads.

Eliminate the warning, by only passing the flag on the older releases
that accept it.

Upstream change and discussion:
  https://github.com/python/cpython/commit/a6a4dc816
  https://bugs.python.org/issue31370
2020-03-30 22:56:48 -07:00
Frederik Rietdijk c392d70518 pkgsStatic.python3: fix build 2020-03-30 17:06:38 +02:00
Graham Christensen 39aac70d74 pythonMinimal: don't include site-customise
Experimenting with patching the site-customize file causes mass
rebuilds for no reason.
2020-03-14 15:02:51 +01:00
Maximilian Bosch eddfcc32b4
Merge branch 'staging' into glibc230 2020-01-23 11:31:13 +01:00