Turns out fixing this only in importlib is not sufficient and we
need to backport CPython part of the fix too.
This patch is based on https://hg.python.org/cpython/rev/c16063765d3a
but because the code around is different there are some changes (C-strings
instead of Python objects etc.)
With this patch Tensorflow builds successfully on many-core machine.
This commit adds a Nix-specific module that recursively adds paths that
are on `NIX_PYTHONPATH` to `sys.path`. In order to process possible
`.pth` files `site.addsitedir` is used.
The paths listed in `PYTHONPATH` are added to `sys.path` afterwards, but
they will be added before the entries we add here and thus take
precedence.
The reason for adding support for this environment variable is that we
can set it in a wrapper without breaking support for `PYTHONPATH`.
This is python bug https://bugs.python.org/issue13146. Fixed since
python 3.4. It makes pyc creation atomic, preventing a race condition.
The patch has been rebased on our deterministic build patch.
It wasn't backported to python 2.7 because there was a complaint about
changed semantics. Since files are now created in a temporary directory
and then moved, symlinks will be overridden. See
https://bugs.python.org/issue17222.
That is an edge-case however. Ubuntu and debian have backported the fix
in 2013 already, making it mainstream enough for us to adopt.
Patching numpy.distutils used to be required for pythonPackages.cython
to build on darwin. It was later accidentally disabled during one of the
refactorings, but that did not break cython. This change reinstantiates
the patch. It still applies, so it should be low maintenance and it can
still be useful.
Originally introduced in 8970a9c and 1531b5e, these patches were lost in
efbe87f when the CPython version files were merged, likely due to
CPython 3.7 not needing them anymore. These patches should remain
in-tree until CPython 3.5 and 3.6 support is dropped completely.
Found with the diffoscope ( https://diffoscope.org/ ).
The upstream patch for distutils does not apply cleanly to Python
3.7.3's sources. (The patch applies cleanly to Python 3.7.2's sources,
but nixpkgs commit 0ddae82e6a upgraded
Python to 3.7.3.) Fix the patch to make python37 build on macOS.
The wrapper is not needed because the runpath is already set correctly,
and LD_LIBRARY_PATH was breaking child processes linked against
different libc versions.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
These interpreters are prebuilt by upstream and patched using patchelf.
They are primarily added for testing purposes and development on the
non-prebuilt PyPy interpreters as it can speed up translation
significantly.
Drop `python.majorVersion`. For Python language version, use `python.pythonVersion`.
For implementation version, use `python.sourceVersion`.
Some expressions were broken. Those that were identified were fixed.
fixup major
This changeset allows for cross-compilation of Python packages. Packages
built with buildPythonPackage are not allowed to refer to the build
machine. Executables that have shebangs will refer to the host.
Each time a new major/minor version of CPython was released, a new
expression would be written, typically copied from the previous release.
Often fixes are only made in the current/latest release. By merging the
expressions it's more likely that modifications end up in all versions,
as is likely intended.
This commit introduces one expression for Python 3, and another for 2.7.
These two may also be merged, but it will result in a lot of extra
conditionals making the expression harder to follow.
A common passthru is introduced for CPython and PyPy.
python 2.7: use common passthru
updateScript accepts a list, in which case, it will execute the head with the
tail as arguments. Switching to this style will allow us to get rid of the need
to create extra script doing just that.
All Python packages now have an updateScript. The script calls
`update-python-libraries` and passes it the position of the derivation
expression obtained using `meta.position`. This works fine in case a Nix
expression represents only a single derivation. If there are more in it,
`update-python-libraries` will fail.
Python 3.4 will receive it's final patch release in March 2019 and there won't
be any releases anymore after that, so also not during NixOS 2019.03.
Python 3.4 is not used anymore in Nixpkgs. In any case, migrating code from
3.4 to 3.4+ is trivial.
By default all warnings were printed. This occasionally resulted in
a lot of warnings leading to builds being killed.
This commit reduces the amount of warnings printed.
`python.buildEnv` would already wrap executables exporting `PYTHONHOME`.
With this change, it is possible to pass in additional arguments to the
underlying `makeWrapper`.
This also updates the bootstrap tool builder to LLVM 5, but not the ones
we actually use for bootstrap. I'll make that change in a subsequent commit
so as to provide traceable provenance of the bootstrap tools.
* substitute(): --subst-var was silently coercing to "" if the variable does not exist.
* libffi: simplify using `checkInputs`
* pythonPackges.hypothesis, pythonPackages.pytest: simpify dependency cycle fix
* utillinux: 2.32 -> 2.32.1
https://lkml.org/lkml/2018/7/16/532
* busybox: 1.29.0 -> 1.29.1
* bind: 9.12.1-P2 -> 9.12.2
https://ftp.isc.org/isc/bind9/9.12.2/RELEASE-NOTES-bind-9.12.2.html
* curl: 7.60.0 -> 7.61.0
* gvfs: make tests run, but disable
* ilmbase: disable tests on i686. Spooky!
* mdds: fix tests
* git: disable checks as tests are run in installcheck
* ruby: disable tests
* libcommuni: disable checks as tests are run in installcheck
* librdf: make tests run, but disable
* neon, neon_0_29: make tests run, but disable
* pciutils: 3.6.0 -> 3.6.1
Semi-automatic update generated by https://github.com/ryantm/nixpkgs-update tools. This update was made based on information from https://repology.org/metapackage/pciutils/versions.
* mesa: more include fixes
mostly from void-linux (thanks!)
* npth: 1.5 -> 1.6
minor bump
* boost167: Add lockfree next_prior patch
* stdenv: cleanup darwin bootstrapping
Also gets rid of the full python and some of it's dependencies in the
stdenv build closure.
* Revert "pciutils: use standardized equivalent for canonicalize_file_name"
This reverts commit f8db20fb3a.
Patching should no longer be needed with 3.6.1.
* binutils-wrapper: Try to avoid adding unnecessary -L flags
(cherry picked from commit f3758258b8895508475caf83e92bfb236a27ceb9)
Signed-off-by: Domen Kožar <domen@dev.si>
* libffi: don't check on darwin
libffi usages in stdenv broken darwin. We need to disable doCheck for that case.
* "rm $out/share/icons/hicolor/icon-theme.cache" -> hicolor-icon-theme setup-hook
* python.pkgs.pytest: setupHook to prevent creation of .pytest-cache folder, fixes#40273
When `py.test` was run with a folder as argument, it would not only
search for tests in that folder, but also create a .pytest-cache folder.
Not only is this state we don't want, but it was also causing
collisions.
* parity-ui: fix after merge
* python.pkgs.pytest-flake8: disable test, fix build
* Revert "meson: 0.46.1 -> 0.47.0"
With meson 0.47.0 (or 0.47.1, or git)
things are very wrong re:rpath handling
resulting in at best missing libs but
even corrupt binaries :(.
When we run patchelf it masks the problem
by removing obviously busted paths.
Which is probably why this wasn't noticed immediately.
Unfortunately the binary already
has a long series of paths scribbled
in a space intended for a much smaller string;
in my testing it was something like
lengths were 67 with 300+ written to it.
I think we've reported the relevant issues upstream,
but unfortunately it appears our patches
are what introduces the overwrite/corruption
(by no longer being correct in what they assume)
This doesn't look so bad to fix but it's
not something I can spend more time on
at the moment.
--
Interestingly the overwritten string data
(because it is scribbled past the bounds)
remains in the binary and is why we're suddenly
seeing unexpected references in various builds
-- notably this is is the reason we're
seeing the "extra-utils" breakage
that entirely crippled NixOS on master
(and probably on staging before?).
Fixes#43650.
This reverts commit 305ac4dade.
(cherry picked from commit 273d68eff8f7b6cd4ebed3718e5078a0f43cb55d)
Signed-off-by: Domen Kožar <domen@dev.si>
Note that a bunch of non-python packages use this attribute already.
Some of those are clearly unaware of the fact that this attribute does
not exists in stdenv because they define it but don't to add it to
their `bulidInputs` :)
Also note that I use `buildInputs` here and only handle regular
builds because python and haskell builders do it this way and I'm not
sure how to properly handle the cross-compilation case.
upstream issue:
https://bugs.python.org/issue31940
There are two PR's proposed to fix this,
but both seem to be stalling waiting for review.
I previously used what appears to be the favored
of the two approaches[1] to fix this,
with plan of keeping it musl-only until PR was merged.
However, while writing up a commit message
explaining the problem and why it needed fixing...
I investigated a bit and found it increasingly
hard to justify anything other than ...
simply not using lchmod.
Here's what I found:
* lchmod is non-POSIX, seems BSD-only these days
* Functionality of lchmod isn't supported on Linux
* best scenario on Linux would be an error
* POSIX does provide lchmod-esque functionality
with fchmodat(), which AFAICT is generally preferred.
* Python intentionally overlooks fchmodat()[2]
electing instead to use lchmod() behavior
as a proxy for whether fchmodat() "works".
I'm not sure I follow their reasoning...
* both glibc and musl provide lchmod impls:
* glibc returns ENOSYS "not implemented"
* musl implements lchmod with fchmodat(),
and so returns EOPNOTSUPP "op not supported"
* Python doesn't expect EOPNOTSUPP from lchmod,
since it's not valid on BSD's lchmod.
* "configure" doesn't actually check lchmod usefully,
instead checks for glibc preprocessor defines
to indicate if the function is just a stub[3];
somewhat fittingly, if the magic macros are defined
then the next line of the C source is "choke me",
causing the compiler to trip, fall, and point
a finger at whatever is near where it ends up.
(somewhat amusing, but AFAIK effective way to get an error :P)
I'm leaving out links to threads on mailing lists and such,
but for now I hope I've convinced you
(or to those reading commit history: explained my reasons)
that this is a bit of a mess[4].
And so instead of making a big mess messier,
and with hopes of never thinking about this again,
I propose we simply tell Python "don't use lchmod" on Linux.
[1] https://github.com/python/cpython/pull/4783
[2] 28453feaa8/Lib/os.py (L144)
[3] 28453feaa8/configure (L2198)
[4] Messes happen, no good intention goes unpunished :).
Documents the reason why it's needed and also prevents the
ensureNewerSourcesHook call being evaluated again and again for every
single Python package.
This hardcoded list of CFFI extension modules gets stale when PyPy adds
more, but fortunately the main translation step already builds these now
(hack_for_cffi_modules in pypy/goal/targetpypystandalone.py).
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
Wrappers already included `PYTHONNOUSERSITE=1`, but now this env var is
also set in the Python setup hook. This improves purity in case of
non-sandboxes builds and nix-shell.
When a Python script has the extension `.py`, bytecode is generated.
Typically, executables in bin have no extension, so no bytecode is
generated. However, some packages do provide executables with
extensions, and thus bytecode is generated.
28299f669a introduced the first Python
packages having multiple outputs. The required outputs were not picked
up by `python.buildEnv` (#31857).
This commit modifies `python.buildEnv` so that it always includes the
$out output and thus fixes#31857.
Python libraries or modules now have an attribute `pythonModule = interpreter;` to indicate
they provide Python modules for the specified `interpreter`.
The package set provides the following helper functions:
- hasPythonModule: Check whether a derivation provides a Python module.
- requiredPythonModules: Recurse into a list of Python modules, returning all Python modules that are required.
- makePythonPath: Create a PYTHONPATH from a list of Python modules.
Also included in this commit is:
- disabledIf: Helper function for disabling non-buildPythonPackage functions.
This continues #23374, which always kept around both attributes, by
always including both propagated files: `propgated-native-build-inputs`
and `propagated-build-inputs`. `nativePkgs` and `crossPkgs` are still
defined as before, however, so this change should only barely
observable.
This is an incremental step to fully keeping the dependencies separate
in all cases.
In the maintenance release bump in
90059701a8 a certain change to /test/ was
backported from Python 3:
- bpo-30207: To simplify backports from Python 3, the test.test_support
module was converted into a package and renamed to test.support. The
test.script_helper module was moved into the test.support package.
Names test.test_support and test.script_helper are left as aliases to
test.support and test.support.script_helper.
libffi needs a patch to actually work on aarch64 (or the cffi Python package
fails its testsuite). Of course the bundled version of libffi has the
same bug, so don't use the buggy version on aarch64.
Python3 already uses the system libffi on all platforms. I don't know
why Python2 doesn't.
While we tell pip not to fetch (with the `--no-index` option),
`setuptools` can do so itself. In the past we used a `distutils.cfg`
with `allow-hosts = None` to prevent setuptools from fetching itself.
This was removed when we started building wheels in
2562f94de4e4fd2ddc677187fa2e2848L69.
The `dist-utils.cfg` code was still there, and adding it to
`buildInputs` is sufficient.
Tested with python.pkgs.passlib by removing the `checkInputs` / `nose`.
test.{support, regrtest} are the internal packages cpython
developers use to write tests.
Although they are not public and the API may change/break
some developers use these packages to write tests for their
(3rd party) software.
The derivations for cpython now only remove the actual tests
but leave the packages in place that are used to write them.
Discussion: https://github.com/NixOS/nixpkgs/pull/28540
Symbolic links were added pointing to the executables that end with 3 as
part of the Python 2 to 3 migration. At some point I disabled ensurepip
but forgot to remove this symbolic link.
Symbolic links were added pointing to the executables that end with 3 as
part of the Python 2 to 3 migration. At some point I disabled ensurepip
but forgot to remove this symbolic link.
Symbolic links were added pointing to the executables that end with 3 as
part of the Python 2 to 3 migration. At some point I disabled ensurepip
but forgot to remove this symbolic link.
* pkgs: refactor needless quoting of homepage meta attribute
A lot of packages are needlessly quoting the homepage meta attribute
(about 1400, 22%), this commit refactors all of those instances.
* pkgs: Fixing some links that were wrongfully unquoted in the previous
commit
* Fixed some instances
Python envs did not pass through any of the properties the Python
interpreter has. That could be annoying, especially not having
`python.interpreter` which is the path to the interpreter. This commit
fixes the situation and inherit python.passthru.
Thus far all executables in a derivation were wrapped. This commit
only wraps executables in $out/bin. If other scripts need to be wrapped
as well, then one can use wrapPythonProgramsIn.
When tests are disabled, we do not want to pass checkInputs to
stdenv.mkDerivation. This reduces the build requirements and, more
importantly, helps cutting cycles.
The Python interpreters are patched so they can build .pyc bytecode free
of certain indeterminism.
When building Python packages we currently set
```
compiling python files.
in nix store.
DETERMINISTIC_BUILD=1;
PYTHONHASHSEED = 0;
```
Instead if setting these environment variables in the function that
builds the package, this commit sets the variables instead in the Python
setup hook. That way, whenever Python is included in a derivation, these
variables are set.
See also the issue https://github.com/NixOS/nixpkgs/issues/25707.
This commit fixes several issues:
- as reported in https://github.com/NixOS/nixpkgs/issues/24924 it was
possible that the file _sysconfigdata.pyc was generated after the actual
build of the CPython interpreter. We forgot to regenerate that file
during the build. This is now fixed
- the expression of the 3.3 interpreter now also includes some of the
determinism patches even though the output isn't yet reproducible. The
reason for adding them is that this makes the expressions of the
different interpreters more similar.
- references to -dev packages are now also removed in the 3.6 package,
thereby reducing its closure size
Python does add the script's directory into "sys.path". For the case of
"catch_conflicts.py" this means "/nix/store" is added to "sys.path". This can
result in very long delays if the store contains a lot of entries.
(moved from master commit 76213d102c)
From the manual:
> This attribute should be a number, with a higher value denoting a
lower priority. The default priority is 0.
Just passing -5 or -10 wasn't sufficient, so let's make it -100.
- Windows installers are indeterministic and we don't need them.
- since Python 3 ensurepip is installed by default. pip is indeteministic and we don't need it.
- rebuild bytecode to ensure its deterministic
