Commit Graph

41 Commits

Author SHA1 Message Date
Frederik Rietdijk
4087d3fe41 python: don't use optimizations on Darwin
Also, don't use autoreconfHook on Darwin with Python 3.
Darwin builds are still impure and fail with

    ld: warning: directory not found for option '-L/nix/store/6yhj9djska835wb6ylg46d2yw9dl0sjb-configd-osx-10.8.5/lib'
    ld: warning: directory not found for option '-L/nix/store/6yhj9djska835wb6ylg46d2yw9dl0sjb-configd-osx-10.8.5/lib'
    ld: warning: object file (/nix/store/0lsij4jl35bnhqhdzla8md6xiswgig5q-Libsystem-osx-10.12.6/lib/crt1.10.6.o) was built for newer OSX version (10.12) than being linked (10.6)
    DYLD_LIBRARY_PATH=/private/tmp/nix-build-python3-3.8.3.drv-0/Python-3.8.3 ./python.exe -E -S -m sysconfig --generate-posix-vars ;\
    if test $? -ne 0 ; then \
            echo "generate-posix-vars failed" ; \
            rm -f ./pybuilddir.txt ; \
            exit 1 ; \
    fi
    /nix/store/dsb7d4dwxk6bzlm845z2zx6wp9a8bqc1-bash-4.4-p23/bin/bash: line 5: 72015 Killed: 9               DYLD_LIBRARY_PATH=/private/tmp/nix-build-python3-3.8.3.drv-0/Python-3.8.3 ./python.exe -E -S -m sysconfig --generate-posix-vars
    generate-posix-vars failed
    make: *** [Makefile:592: pybuilddir.txt] Error 1
2020-06-12 18:29:08 +02:00
Frederik Rietdijk
1c68570ab2 Merge staging-next into staging 2020-06-05 19:42:16 +02:00
Frederik Rietdijk
a337c44db6 python3Minimal: disable optimizations
No point for the bootstrapping.
2020-06-04 20:53:31 +02:00
Frederik Rietdijk
bcf03e8cd2 Revert "cpython: Optimize dynamic symbol tables, for a 6% speedup."
ofborg does not like fetching patches when the derivation is used during bootstrapping.

This reverts commit 480c8d1991.
2020-06-04 20:36:31 +02:00
Frederik Rietdijk
a2be64bf13
Merge pull request #84072 from gnprice/python-build
cpython: Use optimizations, for a 25% speedup.
2020-06-04 18:31:07 +02:00
Luflosi
2379e36124 python39: fix build on macOS
Basically the same changes as in 81d15948cc but for python3.9 instead of python3.8.
2020-06-04 17:11:29 +02:00
Greg Price
480c8d1991 cpython: Optimize dynamic symbol tables, for a 6% speedup.
I took a close look at how Debian builds the Python interpreter,
because I noticed it ran substantially faster than the one in nixpkgs
and I was curious why.

One thing that I found made a material difference in performance was
this pair of linker flags (passed to the compiler):

    -Wl,-O1 -Wl,-Bsymbolic-functions

In other words, effectively the linker gets passed the flags:

    -O1 -Bsymbolic-functions

Doing the same thing in nixpkgs turns out to make the interpreter
run about 6% faster, which is quite a big win for such an easy
change.  So, let's apply it.

---

I had not known there was a `-O1` flag for the *linker*!
But indeed there is.

These flags are unrelated to "link-time optimization" (LTO), despite
the latter's name.  LTO means doing classic compiler optimizations
on the actual code, at the linking step when it becomes possible to
do them with cross-object-file information.  These two flags, by
contrast, cause the linker to make certain optimizations within the
scope of its job as the linker.

Documentation is here, though sparse:
  https://sourceware.org/binutils/docs-2.31/ld/Options.html

The meaning of -O1 was explained in more detail in this LWN article:
  https://lwn.net/Articles/192624/
Apparently it makes the resulting symbol table use a bigger hash
table, so the load factor is smaller and lookups are faster.  Cool.

As for -Bsymbolic-functions, the documentation indicates that it's a
way of saving lookups through the symbol table entirely.  There can
apparently be situations where it changes the behavior of a program,
specifically if the program relies on linker tricks to provide
customization features:
  https://bugs.launchpad.net/ubuntu/+source/xfe/+bug/644645
  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=637184#35
But I'm pretty sure CPython doesn't permit that kind of trick: you
don't load a shared object that tries to redefine some symbol found
in the interpreter core.

The stronger reason I'm confident using -Bsymbolic-functions is
safe, though, is empirical.  Both Debian and Ubuntu have been
shipping a Python built this way since forever -- it was introduced
for the Python 2.4 and 2.5 in Ubuntu "hardy", and Debian "lenny",
released in 2008 and 2009.  In those 12 years they haven't seen a
need to drop this flag; and I've been unable to locate any reports
of trouble related to it, either on the Web in general or on the
Debian bug tracker.  (There are reports of a handful of other
programs breaking with it, but not Python/CPython.)  So that seems
like about as thorough testing as one could hope for.

---

As for the performance impact: I ran CPython upstream's preferred
benchmark suite, "pyperformance", in the same way as described in
the previous commit.  On top of that commit's change, the results
across the 60 benchmarks in the suite are:

The median is 6% faster.

The middle half (aka interquartile range) is from 4% to 8% faster.

Out of 60 benchmarks, 3 come out slower, by 1-4%.  At the other end,
5 are at least 10% faster, and one is 17% faster.

So, that's quite a material speedup!  I don't know how big the
effect of these flags is for other software; but certainly CPython
tends to do plenty of dynamic linking, as that's how it loads
extension modules, which are ubiquitous in the stdlib as well as
popular third-party libraries.  So perhaps that helps explain why
optimizing the dynamic linker has such an impact.
2020-05-13 21:24:30 -07:00
Greg Price
52c04b0347 cpython: Use autoreconfHook to rebuild configure script.
In particular this will let us use patches that apply to configure.ac.
2020-05-13 21:23:48 -07:00
Greg Price
f8a8243bd3 cpython: Use --enable-optimizations, for a 16% speedup.
Without this flag, the configure script prints a warning at the end,
like this (reformatted):

  If you want a release build with all stable optimizations active
  (PGO, etc), please run ./configure --enable-optimizations

We're doing a build to distribute to people for day-to-day use,
doing things other than developing the Python interpreter.  So
that's certainly a release build -- we're the target audience for
this recommendation.

---

And, trying it out, upstream isn't kidding!  I ran the standard
benchmark suite that the CPython developers use for performance
work, "pyperformance".  Following its usage instructions:
  https://pyperformance.readthedocs.io/usage.html
I ran the whole suite, like so:

  $ nix-shell -p ./result."$variant" --run '
      cd $(mktemp -d); python -m venv venv; . venv/bin/activate
      pip install pyperformance
      pyperformance run -o ~/tmp/result.'"$variant"'.json
    '

and then examined the results with commands like:

  $ python -m pyperf compare_to --table -G \
      ~/tmp/result.{$before,$after}.json

Across all the benchmarks in the suite, the median speedup was 16%.
(Meaning 1.16x faster; 14% less time).

The middle half of them ranged from a 13% to a 22% speedup.

Each of the 60 benchmarks in the suite got faster, by speedups
ranging from 3% to 53%.

---

One reason this isn't just the default to begin with is that, until
recently, it made the build a lot slower.  What it does is turn on
profile-guided optimization, which means first build for profiling,
then run some task to get a profile, then build again using the
profile.  And, short of further customization, the task it would use
would be nearly the full test suite, which includes a lot of
expensive and slow tests, and can easily take half an hour to run.

Happily, in 2019 an upstream developer did the work to carefully
select a more appropriate set of tests to use for the profile:
  https://github.com/python/cpython/commit/4e16a4a31
  https://bugs.python.org/issue36044
This suite takes just 2 minutes to run.  And the resulting final
build is actually slightly faster than with the much longer suite,
at least as measured by those standard "pyperformance" benchmarks.
That work went into the 3.8 release, but the same list works great
if used on older releases too.

So, start passing that --enable-optimizations flag; and backport
that good-for-PGO set of tests, so that we use it on all releases.
2020-05-11 23:37:04 -07:00
Josef Kemetmüller
3818cd9049 python: Fix creating RPMs from Python packages
This should enable (manual) building of RPMs from python projects using
the `python setup.py bdist_rpm` command on systems where `rpmbuild` is
not located in `/usr/bin/`. (e.g. NixOS)
The discovery of the rpmbuild command was fixed upstream in Python 3.8,
so this commit backports the relevant patch to our currently supported
Python 3 versions.

Fixes: #85204
2020-05-09 11:15:28 +02:00
Michael Reilly
84cf00f980
treewide: Per RFC45, remove all unquoted URLs 2020-04-10 17:54:53 +01:00
Jan Tojnar
3e0f4e202f
Merge branch 'master' into staging-next 2020-03-31 21:32:15 +02:00
Greg Price
9d8831c8fe cpython: Drop unrecognized --with-threads configure flag.
The ./configure script prints a warning when passed this flag,
starting with 3.7:

  configure: WARNING: unrecognized options: --with-threads

The reason is that there's no longer such a thing as a build
without threads.

Eliminate the warning, by only passing the flag on the older releases
that accept it.

Upstream change and discussion:
  https://github.com/python/cpython/commit/a6a4dc816
  https://bugs.python.org/issue31370
2020-03-30 22:56:48 -07:00
Frederik Rietdijk
c392d70518 pkgsStatic.python3: fix build 2020-03-30 17:06:38 +02:00
Graham Christensen
39aac70d74 pythonMinimal: don't include site-customise
Experimenting with patching the site-customize file causes mass
rebuilds for no reason.
2020-03-14 15:02:51 +01:00
Maximilian Bosch
eddfcc32b4
Merge branch 'staging' into glibc230 2020-01-23 11:31:13 +01:00
Luka Blaskovic
76d90cf1f5 cpython: fetch darwin-libutil.patch to nixpkgs
python3 is now required buildInput for glibc>=2.29.
Remove fetchpatch to solve infinite recursion in
glibc bootstrap process.
2020-01-14 08:26:58 +00:00
Daiderd Jordan
6328518e98
stdenv: bootstrap darwin with python3
- Replaced python override from the final stdenv, instead we
  propagate our bootstrap python to stage4 and override both
  CF and xnu to use it.

- Removed CF argument from python interpreters, this is redundant
  since it's not overidden anymore.

- Inherit CF from stage4, making it the same as the stdenv.
2020-01-13 11:34:36 +01:00
Andreas Rammhold
e9f522eee1
python: remove _manylinux.py
This will turn manylinux support back on by default.

PIP will now do runtime checks against the compatible glibc version to
determine if the current interpreter is compatible with a given
manylinux specification. However it will not check if any of the
required libraries are present.

The motivation here is that we want to support building python packages
with wheels that require manylinux support. There is no real change for
users of source builds as they are still buildings packages from source.

The real noticeable(?) change is that impure usages (e.g. running `pip
install package`) will install manylinux packages that previously
refused to install.
Previously we did claim that we were not compatible with manylinux and
thus they wouldn't be installed at all.

Now impure users will have basically the same situation as before: If
you require some wheel only package it didn't work before and will not
properly work now. Now the program will fail during runtime vs during
installation time.

I think it is a reasonable trade-off since it allows us to install
manylinux packages with nix expressions and enables tools like
poetry2nix.

This should be a net win for users as it allows wheels, that we
previously couldn't really support, to be used.
2019-12-16 16:37:16 +01:00
Frederik Rietdijk
1939a97811 python3: add pythonForBuild as parameter, fixes python3Minimal
`pythonForBuild` exists for cross-compilation. When one overrides
python, one needs to ensure pythonForBuild matches.
2019-11-21 22:00:23 +01:00
Matthew Glazar
81d15948cc python38: fix build on macOS
Python 3.8 fails to build on macOS for two reasons:

* python-3.x-distutils-C++.patch fails to apply cleanly.
* An #include for <util.h> is missing, causing a build failure:

    ./Modules/posixmodule.c:6586:9: error: implicit declaration of function 'openpty' is invalid in C99
        if (openpty(&master_fd, &slave_fd, NULL, NULL, NULL) != 0)
            ^

Use the correct version of python-3.x-distutils-C++.patch, and add a
patch to #include <util.h>.
2019-10-15 13:12:13 +02:00
volth
7b8fb5c06c treewide: remove redundant quotes 2019-09-08 23:38:31 +00:00
Matthew Bauer
98662eba0e python: enable loadable sqlite extensions (#67472)
Our sqlite supports it, so might as well make use of it.

Fixes #66526
2019-08-31 07:59:00 -04:00
Matthew Bauer
48d0e9401e cpython: skip macOS system frameworks
We don’t want cpython picking up /Library/Frameworks and
/System/Library/Frameworks which contains Tcl.framework. Instead it
should use the one provided by Nix. this would not be an issue if
sandboxing was enabled, but unfortunately that has its own issues.

Fixes #66647
2019-08-25 20:35:19 -04:00
Florian Klink
100eae3b77 python: allow stripping tkinter 2019-08-19 00:11:17 +02:00
Florian Klink
db087f226d python: allow stripping bytecode 2019-08-18 23:47:44 +02:00
Florian Klink
52fcbbff44 python: make rebuilding bytecode optional 2019-08-18 23:47:44 +02:00
Florian Klink
8335f3dd5c python: allow stripping config 2019-08-18 23:47:43 +02:00
Florian Klink
d03283c029 python: allow stripping tests 2019-08-18 23:47:30 +02:00
Florian Klink
414cde4df4 python: allow stripping IDLE 2019-08-18 23:47:16 +02:00
adisbladis
5c0d493df4
python: Enable building without openssl 2019-08-18 08:18:58 +01:00
Frederik Rietdijk
46409b5c32 Python: add sitecustomize.py, listen to NIX_PYTHONPATH
This commit adds a Nix-specific module that recursively adds paths that
are on `NIX_PYTHONPATH` to `sys.path`. In order to process possible
`.pth` files `site.addsitedir` is used.

The paths listed in `PYTHONPATH` are added to `sys.path` afterwards, but
they will be added before the entries we add here and thus take
precedence.

The reason for adding support for this environment variable is that we
can set it in a wrapper without breaking support for `PYTHONPATH`.
2019-07-13 09:37:33 +02:00
volth
f3282c8d1e treewide: remove unused variables (#63177)
* treewide: remove unused variables

* making ofborg happy
2019-06-16 19:59:05 +00:00
Dmitry Kalinkin
8fa36fc8a1 python: provide hasCxxDistutils attribute for pythonPackages.numpy
Patching numpy.distutils used to be required for pythonPackages.cython
to build on darwin. It was later accidentally disabled during one of the
refactorings, but that did not break cython. This change reinstantiates
the patch. It still applies, so it should be low maintenance and it can
still be useful.
2019-04-28 09:17:59 +02:00
bb010g
28ac309309 python3{5,6}: Reintroduce bytecode determinism
Originally introduced in 8970a9c and 1531b5e, these patches were lost in
efbe87f when the CPython version files were merged, likely due to
CPython 3.7 not needing them anymore. These patches should remain
in-tree until CPython 3.5 and 3.6 support is dropped completely.

Found with the diffoscope ( https://diffoscope.org/ ).
2019-04-18 08:12:29 +02:00
Matthew Glazar
fdbcf5ba36 python37: fix build on macOS
The upstream patch for distutils does not apply cleanly to Python
3.7.3's sources. (The patch applies cleanly to Python 3.7.2's sources,
but nixpkgs commit 0ddae82e6a upgraded
Python to 3.7.3.) Fix the patch to make python37 build on macOS.
2019-04-15 10:49:34 -07:00
Matthew Glazar
0bf2d63366 python35: fix build on macOS
The upstream patch for distutils does not apply cleanly to Python 3.5's
sources. Fix the patch to make python35 build on macOS.
2019-03-10 13:38:43 +01:00
Averell Dalton
91aeda1432 python: sysconfig: fix paths 2019-02-07 18:52:08 +01:00
Lengyel Balazs
f4a53ff3bc treewide/xorg: replace *proto with xorgproto 2019-01-04 14:38:57 +01:00
Frederik Rietdijk
f665828fa3 Python: improve cross-compilation
This changeset allows for cross-compilation of Python packages. Packages
built with buildPythonPackage are not allowed to refer to the build
machine. Executables that have shebangs will refer to the host.
2019-01-04 10:45:22 +01:00
Frederik Rietdijk
efbe87f3ef CPython: merge expressions of interpreters
Each time a new major/minor version of CPython was released, a new
expression would be written, typically copied from the previous release.
Often fixes are only made in the current/latest release. By merging the
expressions it's more likely that modifications end up in all versions,
as is likely intended.

This commit introduces one expression for Python 3, and another for 2.7.
These two may also be merged, but it will result in a lot of extra
conditionals making the expression harder to follow.

A common passthru is introduced for CPython and PyPy.

python 2.7: use common passthru
2019-01-04 10:45:22 +01:00